[==========] Running 8 tests from 2 test suites.
[----------] Global test environment set-up.
[----------] 6 tests from LeaderRebalancerTest
[ RUN      ] LeaderRebalancerTest.FunctionalTestForDivided
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20260709 12:19:12.122532 16501 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.16.29.126:43845
I20260709 12:19:12.124596 16501 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260709 12:19:12.125830 16501 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:19:12.150159 16508 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:19:12.150362 16510 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:19:12.153139 16501 server_base.cc:1061] running on GCE node
W20260709 12:19:12.150189 16507 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:19:13.446116 16501 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:19:13.446388 16501 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:19:13.446626 16501 hybrid_clock.cc:648] HybridClock initialized: now 1783599553446604 us; error 0 us; skew 500 ppm
I20260709 12:19:13.447559 16501 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:19:13.453449 16501 webserver.cc:533] Webserver started at http://127.16.29.126:36271/ using document root <none> and password file <none>
I20260709 12:19:13.454661 16501 fs_manager.cc:362] Metadata directory not provided
I20260709 12:19:13.454928 16501 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:19:13.455441 16501 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:19:13.462440 16501 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783599551912677-16501-0/minicluster-data/master-0-root/instance:
uuid: "d2fba92c59de4f3893f9baebc856a51c"
format_stamp: "Formatted at 2026-07-09 12:19:13 on dist-test-slave-5n9v"
I20260709 12:19:13.472505 16501 fs_manager.cc:696] Time spent creating directory manager: real 0.009s	user 0.006s	sys 0.005s
I20260709 12:19:13.478381 16516 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:19:13.479943 16501 fs_manager.cc:730] Time spent opening block manager: real 0.004s	user 0.007s	sys 0.000s
I20260709 12:19:13.480378 16501 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783599551912677-16501-0/minicluster-data/master-0-root
uuid: "d2fba92c59de4f3893f9baebc856a51c"
format_stamp: "Formatted at 2026-07-09 12:19:13 on dist-test-slave-5n9v"
I20260709 12:19:13.480769 16501 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783599551912677-16501-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783599551912677-16501-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783599551912677-16501-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:19:13.540055 16501 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:19:13.541750 16501 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260709 12:19:13.542308 16501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:19:13.631608 16567 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.29.126:43845 every 8 connection(s)
I20260709 12:19:13.631603 16501 rpc_server.cc:307] RPC server started. Bound to: 127.16.29.126:43845
I20260709 12:19:13.640774 16568 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:19:13.666054 16568 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2fba92c59de4f3893f9baebc856a51c: Bootstrap starting.
I20260709 12:19:13.673611 16568 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d2fba92c59de4f3893f9baebc856a51c: Neither blocks nor log segments found. Creating new log.
I20260709 12:19:13.675689 16568 log.cc:826] T 00000000000000000000000000000000 P d2fba92c59de4f3893f9baebc856a51c: Log is configured to *not* fsync() on all Append() calls
I20260709 12:19:13.681236 16568 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2fba92c59de4f3893f9baebc856a51c: No bootstrap required, opened a new log
I20260709 12:19:13.704490 16568 raft_consensus.cc:359] T 00000000000000000000000000000000 P d2fba92c59de4f3893f9baebc856a51c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2fba92c59de4f3893f9baebc856a51c" member_type: VOTER }
I20260709 12:19:13.705066 16568 raft_consensus.cc:385] T 00000000000000000000000000000000 P d2fba92c59de4f3893f9baebc856a51c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:19:13.705396 16568 raft_consensus.cc:740] T 00000000000000000000000000000000 P d2fba92c59de4f3893f9baebc856a51c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2fba92c59de4f3893f9baebc856a51c, State: Initialized, Role: FOLLOWER
I20260709 12:19:13.706316 16568 consensus_queue.cc:260] T 00000000000000000000000000000000 P d2fba92c59de4f3893f9baebc856a51c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2fba92c59de4f3893f9baebc856a51c" member_type: VOTER }
I20260709 12:19:13.706830 16568 raft_consensus.cc:399] T 00000000000000000000000000000000 P d2fba92c59de4f3893f9baebc856a51c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260709 12:19:13.707130 16568 raft_consensus.cc:493] T 00000000000000000000000000000000 P d2fba92c59de4f3893f9baebc856a51c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260709 12:19:13.707641 16568 raft_consensus.cc:3060] T 00000000000000000000000000000000 P d2fba92c59de4f3893f9baebc856a51c [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:19:13.713153 16568 raft_consensus.cc:515] T 00000000000000000000000000000000 P d2fba92c59de4f3893f9baebc856a51c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2fba92c59de4f3893f9baebc856a51c" member_type: VOTER }
I20260709 12:19:13.714046 16568 leader_election.cc:304] T 00000000000000000000000000000000 P d2fba92c59de4f3893f9baebc856a51c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d2fba92c59de4f3893f9baebc856a51c; no voters: 
I20260709 12:19:13.715926 16568 leader_election.cc:290] T 00000000000000000000000000000000 P d2fba92c59de4f3893f9baebc856a51c [CANDIDATE]: Term 1 election: Requested vote from peers 
I20260709 12:19:13.716535 16571 raft_consensus.cc:2804] T 00000000000000000000000000000000 P d2fba92c59de4f3893f9baebc856a51c [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:19:13.719625 16571 raft_consensus.cc:697] T 00000000000000000000000000000000 P d2fba92c59de4f3893f9baebc856a51c [term 1 LEADER]: Becoming Leader. State: Replica: d2fba92c59de4f3893f9baebc856a51c, State: Running, Role: LEADER
I20260709 12:19:13.720535 16571 consensus_queue.cc:237] T 00000000000000000000000000000000 P d2fba92c59de4f3893f9baebc856a51c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2fba92c59de4f3893f9baebc856a51c" member_type: VOTER }
I20260709 12:19:13.720846 16568 sys_catalog.cc:565] T 00000000000000000000000000000000 P d2fba92c59de4f3893f9baebc856a51c [sys.catalog]: configured and running, proceeding with master startup.
I20260709 12:19:13.733443 16573 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2fba92c59de4f3893f9baebc856a51c [sys.catalog]: SysCatalogTable state changed. Reason: New leader d2fba92c59de4f3893f9baebc856a51c. Latest consensus state: current_term: 1 leader_uuid: "d2fba92c59de4f3893f9baebc856a51c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2fba92c59de4f3893f9baebc856a51c" member_type: VOTER } }
I20260709 12:19:13.734712 16572 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2fba92c59de4f3893f9baebc856a51c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d2fba92c59de4f3893f9baebc856a51c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2fba92c59de4f3893f9baebc856a51c" member_type: VOTER } }
I20260709 12:19:13.735278 16573 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2fba92c59de4f3893f9baebc856a51c [sys.catalog]: This master's current role is: LEADER
I20260709 12:19:13.735312 16572 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2fba92c59de4f3893f9baebc856a51c [sys.catalog]: This master's current role is: LEADER
I20260709 12:19:13.739261 16580 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260709 12:19:13.753443 16580 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260709 12:19:13.755476 16501 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260709 12:19:13.773744 16580 catalog_manager.cc:1383] Generated new cluster ID: 312671978b1b4dd78285f8ef302ab177
I20260709 12:19:13.774085 16580 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260709 12:19:13.793704 16580 catalog_manager.cc:1406] Generated new certificate authority record
I20260709 12:19:13.795292 16580 catalog_manager.cc:1540] Loading token signing keys...
I20260709 12:19:13.816711 16580 catalog_manager.cc:6070] T 00000000000000000000000000000000 P d2fba92c59de4f3893f9baebc856a51c: Generated new TSK 0
I20260709 12:19:13.817831 16580 catalog_manager.cc:1550] Initializing in-progress tserver states...
I20260709 12:19:13.888023 16501 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:19:13.898502 16590 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:19:13.900836 16591 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:19:13.904035 16593 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:19:13.904712 16501 server_base.cc:1061] running on GCE node
I20260709 12:19:13.905709 16501 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:19:13.905987 16501 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:19:13.906239 16501 hybrid_clock.cc:648] HybridClock initialized: now 1783599553906214 us; error 0 us; skew 500 ppm
I20260709 12:19:13.907135 16501 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:19:13.911867 16501 webserver.cc:533] Webserver started at http://127.16.29.65:41709/ using document root <none> and password file <none>
I20260709 12:19:13.912762 16501 fs_manager.cc:362] Metadata directory not provided
I20260709 12:19:13.913025 16501 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:19:13.913388 16501 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:19:13.915009 16501 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783599551912677-16501-0/minicluster-data/ts-0-root/instance:
uuid: "49e15cd7009e4d6198f9d9352a27e9d1"
format_stamp: "Formatted at 2026-07-09 12:19:13 on dist-test-slave-5n9v"
I20260709 12:19:13.922184 16501 fs_manager.cc:696] Time spent creating directory manager: real 0.006s	user 0.009s	sys 0.000s
I20260709 12:19:13.928609 16598 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:19:13.929756 16501 fs_manager.cc:730] Time spent opening block manager: real 0.004s	user 0.002s	sys 0.002s
I20260709 12:19:13.930140 16501 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783599551912677-16501-0/minicluster-data/ts-0-root
uuid: "49e15cd7009e4d6198f9d9352a27e9d1"
format_stamp: "Formatted at 2026-07-09 12:19:13 on dist-test-slave-5n9v"
I20260709 12:19:13.930547 16501 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783599551912677-16501-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783599551912677-16501-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783599551912677-16501-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:19:13.976143 16501 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:19:13.977573 16501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:19:13.979627 16501 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:19:13.984201 16501 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:19:13.984513 16501 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:19:13.984787 16501 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:19:13.984984 16501 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:19:14.094118 16501 rpc_server.cc:307] RPC server started. Bound to: 127.16.29.65:44833
I20260709 12:19:14.094874 16660 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.29.65:44833 every 8 connection(s)
I20260709 12:19:14.102978 16501 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:19:14.122728 16666 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:19:14.128798 16665 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:19:14.131807 16501 server_base.cc:1061] running on GCE node
W20260709 12:19:14.134370 16669 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:19:14.136557 16501 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:19:14.136842 16501 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:19:14.137058 16501 hybrid_clock.cc:648] HybridClock initialized: now 1783599554137036 us; error 0 us; skew 500 ppm
I20260709 12:19:14.137859 16501 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:19:14.142031 16661 heartbeater.cc:344] Connected to a master server at 127.16.29.126:43845
I20260709 12:19:14.142532 16661 heartbeater.cc:461] Registering TS with master...
I20260709 12:19:14.143324 16501 webserver.cc:533] Webserver started at http://127.16.29.66:35505/ using document root <none> and password file <none>
I20260709 12:19:14.143870 16501 fs_manager.cc:362] Metadata directory not provided
I20260709 12:19:14.143798 16661 heartbeater.cc:507] Master 127.16.29.126:43845 requested a full tablet report, sending...
I20260709 12:19:14.145258 16501 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:19:14.145746 16501 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:19:14.147511 16501 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783599551912677-16501-0/minicluster-data/ts-1-root/instance:
uuid: "49598ea210614f1fbb942c1fb77c7932"
format_stamp: "Formatted at 2026-07-09 12:19:14 on dist-test-slave-5n9v"
I20260709 12:19:14.147939 16533 ts_manager.cc:194] Registered new tserver with Master: 49e15cd7009e4d6198f9d9352a27e9d1 (127.16.29.65:44833)
I20260709 12:19:14.151151 16533 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:41806
I20260709 12:19:14.156114 16501 fs_manager.cc:696] Time spent creating directory manager: real 0.008s	user 0.005s	sys 0.001s
I20260709 12:19:14.161139 16673 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:19:14.162155 16501 fs_manager.cc:730] Time spent opening block manager: real 0.004s	user 0.002s	sys 0.002s
I20260709 12:19:14.162473 16501 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783599551912677-16501-0/minicluster-data/ts-1-root
uuid: "49598ea210614f1fbb942c1fb77c7932"
format_stamp: "Formatted at 2026-07-09 12:19:14 on dist-test-slave-5n9v"
I20260709 12:19:14.162786 16501 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783599551912677-16501-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783599551912677-16501-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783599551912677-16501-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:19:14.178176 16501 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:19:14.180020 16501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:19:14.181667 16501 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:19:14.185214 16501 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:19:14.185411 16501 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:19:14.185712 16501 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:19:14.185878 16501 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:19:14.295639 16501 rpc_server.cc:307] RPC server started. Bound to: 127.16.29.66:35871
I20260709 12:19:14.295771 16735 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.29.66:35871 every 8 connection(s)
I20260709 12:19:14.304153 16501 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:19:14.315425 16739 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:19:14.319968 16740 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:19:14.323395 16736 heartbeater.cc:344] Connected to a master server at 127.16.29.126:43845
I20260709 12:19:14.323990 16736 heartbeater.cc:461] Registering TS with master...
I20260709 12:19:14.325147 16736 heartbeater.cc:507] Master 127.16.29.126:43845 requested a full tablet report, sending...
I20260709 12:19:14.326177 16501 server_base.cc:1061] running on GCE node
W20260709 12:19:14.326828 16742 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:19:14.327903 16533 ts_manager.cc:194] Registered new tserver with Master: 49598ea210614f1fbb942c1fb77c7932 (127.16.29.66:35871)
I20260709 12:19:14.328617 16501 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:19:14.328905 16501 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:19:14.329121 16501 hybrid_clock.cc:648] HybridClock initialized: now 1783599554329103 us; error 0 us; skew 500 ppm
I20260709 12:19:14.329810 16501 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:19:14.329707 16533 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:41808
I20260709 12:19:14.333766 16501 webserver.cc:533] Webserver started at http://127.16.29.67:36031/ using document root <none> and password file <none>
I20260709 12:19:14.334497 16501 fs_manager.cc:362] Metadata directory not provided
I20260709 12:19:14.334770 16501 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:19:14.335110 16501 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:19:14.336853 16501 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783599551912677-16501-0/minicluster-data/ts-2-root/instance:
uuid: "3e772c31b50d4c6693644565ef92de04"
format_stamp: "Formatted at 2026-07-09 12:19:14 on dist-test-slave-5n9v"
I20260709 12:19:14.342295 16501 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.001s	sys 0.005s
I20260709 12:19:14.346628 16747 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:19:14.347931 16501 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.000s
I20260709 12:19:14.348495 16501 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783599551912677-16501-0/minicluster-data/ts-2-root
uuid: "3e772c31b50d4c6693644565ef92de04"
format_stamp: "Formatted at 2026-07-09 12:19:14 on dist-test-slave-5n9v"
I20260709 12:19:14.348889 16501 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783599551912677-16501-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783599551912677-16501-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783599551912677-16501-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:19:14.365725 16501 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:19:14.367059 16501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:19:14.369859 16501 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:19:14.373584 16501 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:19:14.373796 16501 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:19:14.374121 16501 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:19:14.374270 16501 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:19:14.490425 16501 rpc_server.cc:307] RPC server started. Bound to: 127.16.29.67:35049
I20260709 12:19:14.491130 16809 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.29.67:35049 every 8 connection(s)
I20260709 12:19:14.510380 16810 heartbeater.cc:344] Connected to a master server at 127.16.29.126:43845
I20260709 12:19:14.510936 16810 heartbeater.cc:461] Registering TS with master...
I20260709 12:19:14.512226 16810 heartbeater.cc:507] Master 127.16.29.126:43845 requested a full tablet report, sending...
I20260709 12:19:14.514652 16533 ts_manager.cc:194] Registered new tserver with Master: 3e772c31b50d4c6693644565ef92de04 (127.16.29.67:35049)
I20260709 12:19:14.515618 16501 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.019284608s
I20260709 12:19:14.516463 16533 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:41824
I20260709 12:19:14.584296 16533 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41830:
name: "test-workload"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
  rows: "\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q""\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q"
  indirect_data: """"
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W20260709 12:19:14.589576 16533 catalog_manager.cc:7077] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260709 12:19:14.703397 16774 tablet_service.cc:1511] Processing CreateTablet for tablet 927ab821d939440ab1def38a7dbd0358 (DEFAULT_TABLE table=test-workload [id=a43ed59afe3b4cf6bb712ca478f41d4b]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260709 12:19:14.703620 16775 tablet_service.cc:1511] Processing CreateTablet for tablet 0edbf034be4a41d79cbce7d0c876132d (DEFAULT_TABLE table=test-workload [id=a43ed59afe3b4cf6bb712ca478f41d4b]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260709 12:19:14.705962 16775 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0edbf034be4a41d79cbce7d0c876132d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:19:14.711314 16774 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 927ab821d939440ab1def38a7dbd0358. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:19:14.712512 16773 tablet_service.cc:1511] Processing CreateTablet for tablet dd736bd544c746238c77320b4a5706e0 (DEFAULT_TABLE table=test-workload [id=a43ed59afe3b4cf6bb712ca478f41d4b]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260709 12:19:14.717190 16773 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd736bd544c746238c77320b4a5706e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:19:14.726521 16772 tablet_service.cc:1511] Processing CreateTablet for tablet 2ef5967933364223bef5315de3fa6bfe (DEFAULT_TABLE table=test-workload [id=a43ed59afe3b4cf6bb712ca478f41d4b]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260709 12:19:14.730824 16772 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ef5967933364223bef5315de3fa6bfe. 1 dirs total, 0 dirs full, 0 dirs failed
W20260709 12:19:14.744639 16581 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:19:14.752784 16626 tablet_service.cc:1511] Processing CreateTablet for tablet 0edbf034be4a41d79cbce7d0c876132d (DEFAULT_TABLE table=test-workload [id=a43ed59afe3b4cf6bb712ca478f41d4b]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260709 12:19:14.756937 16626 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0edbf034be4a41d79cbce7d0c876132d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:19:14.756258 16624 tablet_service.cc:1511] Processing CreateTablet for tablet dd736bd544c746238c77320b4a5706e0 (DEFAULT_TABLE table=test-workload [id=a43ed59afe3b4cf6bb712ca478f41d4b]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260709 12:19:14.754886 16625 tablet_service.cc:1511] Processing CreateTablet for tablet 927ab821d939440ab1def38a7dbd0358 (DEFAULT_TABLE table=test-workload [id=a43ed59afe3b4cf6bb712ca478f41d4b]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260709 12:19:14.772723 16624 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd736bd544c746238c77320b4a5706e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:19:14.772519 16622 tablet_service.cc:1511] Processing CreateTablet for tablet 0086bd356ad04503ad306b7217280c34 (DEFAULT_TABLE table=test-workload [id=a43ed59afe3b4cf6bb712ca478f41d4b]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260709 12:19:14.773619 16700 tablet_service.cc:1511] Processing CreateTablet for tablet 927ab821d939440ab1def38a7dbd0358 (DEFAULT_TABLE table=test-workload [id=a43ed59afe3b4cf6bb712ca478f41d4b]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260709 12:19:14.767207 16623 tablet_service.cc:1511] Processing CreateTablet for tablet 2ef5967933364223bef5315de3fa6bfe (DEFAULT_TABLE table=test-workload [id=a43ed59afe3b4cf6bb712ca478f41d4b]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260709 12:19:14.777426 16699 tablet_service.cc:1511] Processing CreateTablet for tablet dd736bd544c746238c77320b4a5706e0 (DEFAULT_TABLE table=test-workload [id=a43ed59afe3b4cf6bb712ca478f41d4b]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260709 12:19:14.785210 16771 tablet_service.cc:1511] Processing CreateTablet for tablet 0086bd356ad04503ad306b7217280c34 (DEFAULT_TABLE table=test-workload [id=a43ed59afe3b4cf6bb712ca478f41d4b]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260709 12:19:14.791359 16625 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 927ab821d939440ab1def38a7dbd0358. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:19:14.793164 16698 tablet_service.cc:1511] Processing CreateTablet for tablet 2ef5967933364223bef5315de3fa6bfe (DEFAULT_TABLE table=test-workload [id=a43ed59afe3b4cf6bb712ca478f41d4b]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260709 12:19:14.794344 16697 tablet_service.cc:1511] Processing CreateTablet for tablet 0086bd356ad04503ad306b7217280c34 (DEFAULT_TABLE table=test-workload [id=a43ed59afe3b4cf6bb712ca478f41d4b]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260709 12:19:14.790149 16701 tablet_service.cc:1511] Processing CreateTablet for tablet 0edbf034be4a41d79cbce7d0c876132d (DEFAULT_TABLE table=test-workload [id=a43ed59afe3b4cf6bb712ca478f41d4b]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260709 12:19:14.799528 16699 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd736bd544c746238c77320b4a5706e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:19:14.803191 16771 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0086bd356ad04503ad306b7217280c34. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:19:14.807180 16700 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 927ab821d939440ab1def38a7dbd0358. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:19:14.812208 16697 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0086bd356ad04503ad306b7217280c34. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:19:14.814643 16623 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ef5967933364223bef5315de3fa6bfe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:19:14.816107 16621 tablet_service.cc:1511] Processing CreateTablet for tablet 3692a4a710ef469aa412d9870427fb63 (DEFAULT_TABLE table=test-workload [id=a43ed59afe3b4cf6bb712ca478f41d4b]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260709 12:19:14.819473 16622 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0086bd356ad04503ad306b7217280c34. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:19:14.820886 16621 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3692a4a710ef469aa412d9870427fb63. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:19:14.822597 16701 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0edbf034be4a41d79cbce7d0c876132d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:19:14.827956 16698 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ef5967933364223bef5315de3fa6bfe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:19:14.855629 16696 tablet_service.cc:1511] Processing CreateTablet for tablet 3692a4a710ef469aa412d9870427fb63 (DEFAULT_TABLE table=test-workload [id=a43ed59afe3b4cf6bb712ca478f41d4b]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260709 12:19:14.865684 16696 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3692a4a710ef469aa412d9870427fb63. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:19:14.863817 16770 tablet_service.cc:1511] Processing CreateTablet for tablet 3692a4a710ef469aa412d9870427fb63 (DEFAULT_TABLE table=test-workload [id=a43ed59afe3b4cf6bb712ca478f41d4b]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260709 12:19:14.869825 16770 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3692a4a710ef469aa412d9870427fb63. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:19:14.886775 16828 tablet_bootstrap.cc:492] T dd736bd544c746238c77320b4a5706e0 P 3e772c31b50d4c6693644565ef92de04: Bootstrap starting.
I20260709 12:19:14.971583 16621 tablet_service.cc:1511] Processing CreateTablet for tablet 95f027ce38784f81bf5c3b35b3357173 (DEFAULT_TABLE table=test-workload [id=a43ed59afe3b4cf6bb712ca478f41d4b]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260709 12:19:14.972597 16829 tablet_bootstrap.cc:492] T 3692a4a710ef469aa412d9870427fb63 P 49e15cd7009e4d6198f9d9352a27e9d1: Bootstrap starting.
I20260709 12:19:14.972857 16830 tablet_bootstrap.cc:492] T 0086bd356ad04503ad306b7217280c34 P 49598ea210614f1fbb942c1fb77c7932: Bootstrap starting.
I20260709 12:19:14.974586 16621 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 95f027ce38784f81bf5c3b35b3357173. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:19:14.979988 16828 tablet_bootstrap.cc:654] T dd736bd544c746238c77320b4a5706e0 P 3e772c31b50d4c6693644565ef92de04: Neither blocks nor log segments found. Creating new log.
I20260709 12:19:14.984436 16771 tablet_service.cc:1511] Processing CreateTablet for tablet 95f027ce38784f81bf5c3b35b3357173 (DEFAULT_TABLE table=test-workload [id=a43ed59afe3b4cf6bb712ca478f41d4b]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260709 12:19:14.988627 16771 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 95f027ce38784f81bf5c3b35b3357173. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:19:14.992210 16626 tablet_service.cc:1511] Processing CreateTablet for tablet af674c5ffb4e44dab6f15ab84dbeb737 (DEFAULT_TABLE table=test-workload [id=a43ed59afe3b4cf6bb712ca478f41d4b]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260709 12:19:14.980513 16697 tablet_service.cc:1511] Processing CreateTablet for tablet 95f027ce38784f81bf5c3b35b3357173 (DEFAULT_TABLE table=test-workload [id=a43ed59afe3b4cf6bb712ca478f41d4b]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260709 12:19:14.993534 16626 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af674c5ffb4e44dab6f15ab84dbeb737. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:19:15.000103 16697 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 95f027ce38784f81bf5c3b35b3357173. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:19:15.002360 16701 tablet_service.cc:1511] Processing CreateTablet for tablet af674c5ffb4e44dab6f15ab84dbeb737 (DEFAULT_TABLE table=test-workload [id=a43ed59afe3b4cf6bb712ca478f41d4b]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260709 12:19:15.003677 16701 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af674c5ffb4e44dab6f15ab84dbeb737. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:19:15.013974 16830 tablet_bootstrap.cc:654] T 0086bd356ad04503ad306b7217280c34 P 49598ea210614f1fbb942c1fb77c7932: Neither blocks nor log segments found. Creating new log.
I20260709 12:19:15.015130 16770 tablet_service.cc:1511] Processing CreateTablet for tablet af674c5ffb4e44dab6f15ab84dbeb737 (DEFAULT_TABLE table=test-workload [id=a43ed59afe3b4cf6bb712ca478f41d4b]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260709 12:19:15.016471 16770 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af674c5ffb4e44dab6f15ab84dbeb737. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:19:15.027662 16696 tablet_service.cc:1511] Processing CreateTablet for tablet 5c3e9fafdd8142eea33e771756209147 (DEFAULT_TABLE table=test-workload [id=a43ed59afe3b4cf6bb712ca478f41d4b]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260709 12:19:15.030305 16774 tablet_service.cc:1511] Processing CreateTablet for tablet 5c3e9fafdd8142eea33e771756209147 (DEFAULT_TABLE table=test-workload [id=a43ed59afe3b4cf6bb712ca478f41d4b]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260709 12:19:15.031929 16696 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5c3e9fafdd8142eea33e771756209147. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:19:15.032096 16774 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5c3e9fafdd8142eea33e771756209147. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:19:15.023488 16625 tablet_service.cc:1511] Processing CreateTablet for tablet 5c3e9fafdd8142eea33e771756209147 (DEFAULT_TABLE table=test-workload [id=a43ed59afe3b4cf6bb712ca478f41d4b]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260709 12:19:15.035780 16829 tablet_bootstrap.cc:654] T 3692a4a710ef469aa412d9870427fb63 P 49e15cd7009e4d6198f9d9352a27e9d1: Neither blocks nor log segments found. Creating new log.
I20260709 12:19:15.048580 16625 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5c3e9fafdd8142eea33e771756209147. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:19:15.091565 16830 tablet_bootstrap.cc:492] T 0086bd356ad04503ad306b7217280c34 P 49598ea210614f1fbb942c1fb77c7932: No bootstrap required, opened a new log
I20260709 12:19:15.092144 16830 ts_tablet_manager.cc:1403] T 0086bd356ad04503ad306b7217280c34 P 49598ea210614f1fbb942c1fb77c7932: Time spent bootstrapping tablet: real 0.122s	user 0.015s	sys 0.023s
I20260709 12:19:15.106828 16830 raft_consensus.cc:359] T 0086bd356ad04503ad306b7217280c34 P 49598ea210614f1fbb942c1fb77c7932 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } } peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } }
I20260709 12:19:15.108103 16830 raft_consensus.cc:385] T 0086bd356ad04503ad306b7217280c34 P 49598ea210614f1fbb942c1fb77c7932 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:19:15.108609 16830 raft_consensus.cc:740] T 0086bd356ad04503ad306b7217280c34 P 49598ea210614f1fbb942c1fb77c7932 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49598ea210614f1fbb942c1fb77c7932, State: Initialized, Role: FOLLOWER
I20260709 12:19:15.110425 16830 consensus_queue.cc:260] T 0086bd356ad04503ad306b7217280c34 P 49598ea210614f1fbb942c1fb77c7932 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } } peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } }
I20260709 12:19:15.111457 16828 tablet_bootstrap.cc:492] T dd736bd544c746238c77320b4a5706e0 P 3e772c31b50d4c6693644565ef92de04: No bootstrap required, opened a new log
I20260709 12:19:15.111992 16828 ts_tablet_manager.cc:1403] T dd736bd544c746238c77320b4a5706e0 P 3e772c31b50d4c6693644565ef92de04: Time spent bootstrapping tablet: real 0.231s	user 0.020s	sys 0.040s
I20260709 12:19:15.115259 16828 raft_consensus.cc:359] T dd736bd544c746238c77320b4a5706e0 P 3e772c31b50d4c6693644565ef92de04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } } peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } }
I20260709 12:19:15.116068 16828 raft_consensus.cc:385] T dd736bd544c746238c77320b4a5706e0 P 3e772c31b50d4c6693644565ef92de04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:19:15.116385 16828 raft_consensus.cc:740] T dd736bd544c746238c77320b4a5706e0 P 3e772c31b50d4c6693644565ef92de04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e772c31b50d4c6693644565ef92de04, State: Initialized, Role: FOLLOWER
I20260709 12:19:15.119679 16830 ts_tablet_manager.cc:1434] T 0086bd356ad04503ad306b7217280c34 P 49598ea210614f1fbb942c1fb77c7932: Time spent starting tablet: real 0.027s	user 0.012s	sys 0.003s
I20260709 12:19:15.120672 16736 heartbeater.cc:499] Master 127.16.29.126:43845 was elected leader, sending a full tablet report...
I20260709 12:19:15.120409 16828 consensus_queue.cc:260] T dd736bd544c746238c77320b4a5706e0 P 3e772c31b50d4c6693644565ef92de04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } } peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } }
I20260709 12:19:15.121294 16829 tablet_bootstrap.cc:492] T 3692a4a710ef469aa412d9870427fb63 P 49e15cd7009e4d6198f9d9352a27e9d1: No bootstrap required, opened a new log
I20260709 12:19:15.121729 16829 ts_tablet_manager.cc:1403] T 3692a4a710ef469aa412d9870427fb63 P 49e15cd7009e4d6198f9d9352a27e9d1: Time spent bootstrapping tablet: real 0.154s	user 0.024s	sys 0.022s
I20260709 12:19:15.125610 16830 tablet_bootstrap.cc:492] T 927ab821d939440ab1def38a7dbd0358 P 49598ea210614f1fbb942c1fb77c7932: Bootstrap starting.
I20260709 12:19:15.133277 16830 tablet_bootstrap.cc:654] T 927ab821d939440ab1def38a7dbd0358 P 49598ea210614f1fbb942c1fb77c7932: Neither blocks nor log segments found. Creating new log.
I20260709 12:19:15.141201 16810 heartbeater.cc:499] Master 127.16.29.126:43845 was elected leader, sending a full tablet report...
I20260709 12:19:15.141657 16828 ts_tablet_manager.cc:1434] T dd736bd544c746238c77320b4a5706e0 P 3e772c31b50d4c6693644565ef92de04: Time spent starting tablet: real 0.029s	user 0.018s	sys 0.007s
I20260709 12:19:15.142618 16828 tablet_bootstrap.cc:492] T 0edbf034be4a41d79cbce7d0c876132d P 3e772c31b50d4c6693644565ef92de04: Bootstrap starting.
I20260709 12:19:15.149088 16828 tablet_bootstrap.cc:654] T 0edbf034be4a41d79cbce7d0c876132d P 3e772c31b50d4c6693644565ef92de04: Neither blocks nor log segments found. Creating new log.
I20260709 12:19:15.140439 16829 raft_consensus.cc:359] T 3692a4a710ef469aa412d9870427fb63 P 49e15cd7009e4d6198f9d9352a27e9d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } } peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } }
I20260709 12:19:15.152942 16829 raft_consensus.cc:385] T 3692a4a710ef469aa412d9870427fb63 P 49e15cd7009e4d6198f9d9352a27e9d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:19:15.153381 16829 raft_consensus.cc:740] T 3692a4a710ef469aa412d9870427fb63 P 49e15cd7009e4d6198f9d9352a27e9d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49e15cd7009e4d6198f9d9352a27e9d1, State: Initialized, Role: FOLLOWER
I20260709 12:19:15.154755 16661 heartbeater.cc:499] Master 127.16.29.126:43845 was elected leader, sending a full tablet report...
I20260709 12:19:15.154651 16829 consensus_queue.cc:260] T 3692a4a710ef469aa412d9870427fb63 P 49e15cd7009e4d6198f9d9352a27e9d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } } peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } }
I20260709 12:19:15.172175 16829 ts_tablet_manager.cc:1434] T 3692a4a710ef469aa412d9870427fb63 P 49e15cd7009e4d6198f9d9352a27e9d1: Time spent starting tablet: real 0.050s	user 0.009s	sys 0.002s
I20260709 12:19:15.173447 16829 tablet_bootstrap.cc:492] T dd736bd544c746238c77320b4a5706e0 P 49e15cd7009e4d6198f9d9352a27e9d1: Bootstrap starting.
I20260709 12:19:15.174232 16830 tablet_bootstrap.cc:492] T 927ab821d939440ab1def38a7dbd0358 P 49598ea210614f1fbb942c1fb77c7932: No bootstrap required, opened a new log
I20260709 12:19:15.174610 16828 tablet_bootstrap.cc:492] T 0edbf034be4a41d79cbce7d0c876132d P 3e772c31b50d4c6693644565ef92de04: No bootstrap required, opened a new log
I20260709 12:19:15.174726 16830 ts_tablet_manager.cc:1403] T 927ab821d939440ab1def38a7dbd0358 P 49598ea210614f1fbb942c1fb77c7932: Time spent bootstrapping tablet: real 0.049s	user 0.018s	sys 0.017s
I20260709 12:19:15.175174 16828 ts_tablet_manager.cc:1403] T 0edbf034be4a41d79cbce7d0c876132d P 3e772c31b50d4c6693644565ef92de04: Time spent bootstrapping tablet: real 0.033s	user 0.017s	sys 0.000s
I20260709 12:19:15.177912 16830 raft_consensus.cc:359] T 927ab821d939440ab1def38a7dbd0358 P 49598ea210614f1fbb942c1fb77c7932 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } } peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } }
I20260709 12:19:15.178189 16828 raft_consensus.cc:359] T 0edbf034be4a41d79cbce7d0c876132d P 3e772c31b50d4c6693644565ef92de04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } } peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } }
I20260709 12:19:15.180001 16830 raft_consensus.cc:385] T 927ab821d939440ab1def38a7dbd0358 P 49598ea210614f1fbb942c1fb77c7932 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:19:15.180080 16828 raft_consensus.cc:385] T 0edbf034be4a41d79cbce7d0c876132d P 3e772c31b50d4c6693644565ef92de04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:19:15.180505 16828 raft_consensus.cc:740] T 0edbf034be4a41d79cbce7d0c876132d P 3e772c31b50d4c6693644565ef92de04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e772c31b50d4c6693644565ef92de04, State: Initialized, Role: FOLLOWER
I20260709 12:19:15.180692 16829 tablet_bootstrap.cc:654] T dd736bd544c746238c77320b4a5706e0 P 49e15cd7009e4d6198f9d9352a27e9d1: Neither blocks nor log segments found. Creating new log.
I20260709 12:19:15.180531 16830 raft_consensus.cc:740] T 927ab821d939440ab1def38a7dbd0358 P 49598ea210614f1fbb942c1fb77c7932 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49598ea210614f1fbb942c1fb77c7932, State: Initialized, Role: FOLLOWER
I20260709 12:19:15.181334 16828 consensus_queue.cc:260] T 0edbf034be4a41d79cbce7d0c876132d P 3e772c31b50d4c6693644565ef92de04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } } peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } }
I20260709 12:19:15.183892 16828 ts_tablet_manager.cc:1434] T 0edbf034be4a41d79cbce7d0c876132d P 3e772c31b50d4c6693644565ef92de04: Time spent starting tablet: real 0.008s	user 0.001s	sys 0.005s
I20260709 12:19:15.182008 16830 consensus_queue.cc:260] T 927ab821d939440ab1def38a7dbd0358 P 49598ea210614f1fbb942c1fb77c7932 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } } peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } }
I20260709 12:19:15.184959 16828 tablet_bootstrap.cc:492] T 2ef5967933364223bef5315de3fa6bfe P 3e772c31b50d4c6693644565ef92de04: Bootstrap starting.
I20260709 12:19:15.186261 16830 ts_tablet_manager.cc:1434] T 927ab821d939440ab1def38a7dbd0358 P 49598ea210614f1fbb942c1fb77c7932: Time spent starting tablet: real 0.011s	user 0.008s	sys 0.000s
I20260709 12:19:15.186666 16834 raft_consensus.cc:493] T 0086bd356ad04503ad306b7217280c34 P 49598ea210614f1fbb942c1fb77c7932 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:15.187208 16830 tablet_bootstrap.cc:492] T 2ef5967933364223bef5315de3fa6bfe P 49598ea210614f1fbb942c1fb77c7932: Bootstrap starting.
I20260709 12:19:15.187089 16834 raft_consensus.cc:515] T 0086bd356ad04503ad306b7217280c34 P 49598ea210614f1fbb942c1fb77c7932 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } } peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } }
I20260709 12:19:15.190096 16834 leader_election.cc:290] T 0086bd356ad04503ad306b7217280c34 P 49598ea210614f1fbb942c1fb77c7932 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49e15cd7009e4d6198f9d9352a27e9d1 (127.16.29.65:44833), 3e772c31b50d4c6693644565ef92de04 (127.16.29.67:35049)
I20260709 12:19:15.195287 16830 tablet_bootstrap.cc:654] T 2ef5967933364223bef5315de3fa6bfe P 49598ea210614f1fbb942c1fb77c7932: Neither blocks nor log segments found. Creating new log.
I20260709 12:19:15.201908 16829 tablet_bootstrap.cc:492] T dd736bd544c746238c77320b4a5706e0 P 49e15cd7009e4d6198f9d9352a27e9d1: No bootstrap required, opened a new log
I20260709 12:19:15.202184 16828 tablet_bootstrap.cc:654] T 2ef5967933364223bef5315de3fa6bfe P 3e772c31b50d4c6693644565ef92de04: Neither blocks nor log segments found. Creating new log.
I20260709 12:19:15.202577 16829 ts_tablet_manager.cc:1403] T dd736bd544c746238c77320b4a5706e0 P 49e15cd7009e4d6198f9d9352a27e9d1: Time spent bootstrapping tablet: real 0.029s	user 0.000s	sys 0.016s
I20260709 12:19:15.205824 16829 raft_consensus.cc:359] T dd736bd544c746238c77320b4a5706e0 P 49e15cd7009e4d6198f9d9352a27e9d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } } peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } }
I20260709 12:19:15.206600 16829 raft_consensus.cc:385] T dd736bd544c746238c77320b4a5706e0 P 49e15cd7009e4d6198f9d9352a27e9d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:19:15.206897 16829 raft_consensus.cc:740] T dd736bd544c746238c77320b4a5706e0 P 49e15cd7009e4d6198f9d9352a27e9d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49e15cd7009e4d6198f9d9352a27e9d1, State: Initialized, Role: FOLLOWER
I20260709 12:19:15.207687 16830 tablet_bootstrap.cc:492] T 2ef5967933364223bef5315de3fa6bfe P 49598ea210614f1fbb942c1fb77c7932: No bootstrap required, opened a new log
I20260709 12:19:15.208263 16830 ts_tablet_manager.cc:1403] T 2ef5967933364223bef5315de3fa6bfe P 49598ea210614f1fbb942c1fb77c7932: Time spent bootstrapping tablet: real 0.021s	user 0.013s	sys 0.004s
I20260709 12:19:15.207746 16829 consensus_queue.cc:260] T dd736bd544c746238c77320b4a5706e0 P 49e15cd7009e4d6198f9d9352a27e9d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } } peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } }
I20260709 12:19:15.210379 16829 ts_tablet_manager.cc:1434] T dd736bd544c746238c77320b4a5706e0 P 49e15cd7009e4d6198f9d9352a27e9d1: Time spent starting tablet: real 0.007s	user 0.009s	sys 0.000s
I20260709 12:19:15.211351 16829 tablet_bootstrap.cc:492] T 2ef5967933364223bef5315de3fa6bfe P 49e15cd7009e4d6198f9d9352a27e9d1: Bootstrap starting.
I20260709 12:19:15.211679 16830 raft_consensus.cc:359] T 2ef5967933364223bef5315de3fa6bfe P 49598ea210614f1fbb942c1fb77c7932 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } } peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } }
I20260709 12:19:15.212695 16830 raft_consensus.cc:385] T 2ef5967933364223bef5315de3fa6bfe P 49598ea210614f1fbb942c1fb77c7932 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:19:15.213063 16830 raft_consensus.cc:740] T 2ef5967933364223bef5315de3fa6bfe P 49598ea210614f1fbb942c1fb77c7932 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49598ea210614f1fbb942c1fb77c7932, State: Initialized, Role: FOLLOWER
I20260709 12:19:15.213900 16830 consensus_queue.cc:260] T 2ef5967933364223bef5315de3fa6bfe P 49598ea210614f1fbb942c1fb77c7932 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } } peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } }
I20260709 12:19:15.223276 16830 ts_tablet_manager.cc:1434] T 2ef5967933364223bef5315de3fa6bfe P 49598ea210614f1fbb942c1fb77c7932: Time spent starting tablet: real 0.015s	user 0.008s	sys 0.000s
I20260709 12:19:15.224483 16830 tablet_bootstrap.cc:492] T 0edbf034be4a41d79cbce7d0c876132d P 49598ea210614f1fbb942c1fb77c7932: Bootstrap starting.
I20260709 12:19:15.230655 16829 tablet_bootstrap.cc:654] T 2ef5967933364223bef5315de3fa6bfe P 49e15cd7009e4d6198f9d9352a27e9d1: Neither blocks nor log segments found. Creating new log.
I20260709 12:19:15.237392 16830 tablet_bootstrap.cc:654] T 0edbf034be4a41d79cbce7d0c876132d P 49598ea210614f1fbb942c1fb77c7932: Neither blocks nor log segments found. Creating new log.
I20260709 12:19:15.240321 16636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0086bd356ad04503ad306b7217280c34" candidate_uuid: "49598ea210614f1fbb942c1fb77c7932" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" is_pre_election: true
W20260709 12:19:15.242379 16674 leader_election.cc:343] T 0086bd356ad04503ad306b7217280c34 P 49598ea210614f1fbb942c1fb77c7932 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 49e15cd7009e4d6198f9d9352a27e9d1 (127.16.29.65:44833): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:19:15.245002 16828 tablet_bootstrap.cc:492] T 2ef5967933364223bef5315de3fa6bfe P 3e772c31b50d4c6693644565ef92de04: No bootstrap required, opened a new log
I20260709 12:19:15.245471 16828 ts_tablet_manager.cc:1403] T 2ef5967933364223bef5315de3fa6bfe P 3e772c31b50d4c6693644565ef92de04: Time spent bootstrapping tablet: real 0.061s	user 0.007s	sys 0.009s
I20260709 12:19:15.247700 16828 raft_consensus.cc:359] T 2ef5967933364223bef5315de3fa6bfe P 3e772c31b50d4c6693644565ef92de04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } } peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } }
I20260709 12:19:15.248416 16828 raft_consensus.cc:385] T 2ef5967933364223bef5315de3fa6bfe P 3e772c31b50d4c6693644565ef92de04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:19:15.248703 16828 raft_consensus.cc:740] T 2ef5967933364223bef5315de3fa6bfe P 3e772c31b50d4c6693644565ef92de04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e772c31b50d4c6693644565ef92de04, State: Initialized, Role: FOLLOWER
I20260709 12:19:15.249488 16828 consensus_queue.cc:260] T 2ef5967933364223bef5315de3fa6bfe P 3e772c31b50d4c6693644565ef92de04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } } peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } }
W20260709 12:19:15.251452 16811 tablet.cc:2406] T 0edbf034be4a41d79cbce7d0c876132d P 3e772c31b50d4c6693644565ef92de04: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260709 12:19:15.251768 16829 tablet_bootstrap.cc:492] T 2ef5967933364223bef5315de3fa6bfe P 49e15cd7009e4d6198f9d9352a27e9d1: No bootstrap required, opened a new log
I20260709 12:19:15.252434 16829 ts_tablet_manager.cc:1403] T 2ef5967933364223bef5315de3fa6bfe P 49e15cd7009e4d6198f9d9352a27e9d1: Time spent bootstrapping tablet: real 0.041s	user 0.005s	sys 0.011s
I20260709 12:19:15.251248 16785 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0086bd356ad04503ad306b7217280c34" candidate_uuid: "49598ea210614f1fbb942c1fb77c7932" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e772c31b50d4c6693644565ef92de04" is_pre_election: true
I20260709 12:19:15.254810 16835 raft_consensus.cc:493] T dd736bd544c746238c77320b4a5706e0 P 3e772c31b50d4c6693644565ef92de04 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:19:15.254843 16674 leader_election.cc:343] T 0086bd356ad04503ad306b7217280c34 P 49598ea210614f1fbb942c1fb77c7932 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3e772c31b50d4c6693644565ef92de04 (127.16.29.67:35049): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:19:15.255582 16674 leader_election.cc:304] T 0086bd356ad04503ad306b7217280c34 P 49598ea210614f1fbb942c1fb77c7932 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 49598ea210614f1fbb942c1fb77c7932; no voters: 3e772c31b50d4c6693644565ef92de04, 49e15cd7009e4d6198f9d9352a27e9d1
I20260709 12:19:15.255505 16835 raft_consensus.cc:515] T dd736bd544c746238c77320b4a5706e0 P 3e772c31b50d4c6693644565ef92de04 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } } peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } }
I20260709 12:19:15.257860 16840 raft_consensus.cc:2749] T 0086bd356ad04503ad306b7217280c34 P 49598ea210614f1fbb942c1fb77c7932 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:19:15.258594 16835 leader_election.cc:290] T dd736bd544c746238c77320b4a5706e0 P 3e772c31b50d4c6693644565ef92de04 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49e15cd7009e4d6198f9d9352a27e9d1 (127.16.29.65:44833), 49598ea210614f1fbb942c1fb77c7932 (127.16.29.66:35871)
I20260709 12:19:15.260412 16828 ts_tablet_manager.cc:1434] T 2ef5967933364223bef5315de3fa6bfe P 3e772c31b50d4c6693644565ef92de04: Time spent starting tablet: real 0.015s	user 0.003s	sys 0.005s
I20260709 12:19:15.261452 16828 tablet_bootstrap.cc:492] T 927ab821d939440ab1def38a7dbd0358 P 3e772c31b50d4c6693644565ef92de04: Bootstrap starting.
I20260709 12:19:15.261853 16830 tablet_bootstrap.cc:492] T 0edbf034be4a41d79cbce7d0c876132d P 49598ea210614f1fbb942c1fb77c7932: No bootstrap required, opened a new log
I20260709 12:19:15.262444 16830 ts_tablet_manager.cc:1403] T 0edbf034be4a41d79cbce7d0c876132d P 49598ea210614f1fbb942c1fb77c7932: Time spent bootstrapping tablet: real 0.038s	user 0.012s	sys 0.005s
I20260709 12:19:15.266007 16830 raft_consensus.cc:359] T 0edbf034be4a41d79cbce7d0c876132d P 49598ea210614f1fbb942c1fb77c7932 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } } peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } }
I20260709 12:19:15.266850 16830 raft_consensus.cc:385] T 0edbf034be4a41d79cbce7d0c876132d P 49598ea210614f1fbb942c1fb77c7932 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:19:15.267194 16830 raft_consensus.cc:740] T 0edbf034be4a41d79cbce7d0c876132d P 49598ea210614f1fbb942c1fb77c7932 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49598ea210614f1fbb942c1fb77c7932, State: Initialized, Role: FOLLOWER
I20260709 12:19:15.268458 16828 tablet_bootstrap.cc:654] T 927ab821d939440ab1def38a7dbd0358 P 3e772c31b50d4c6693644565ef92de04: Neither blocks nor log segments found. Creating new log.
I20260709 12:19:15.270539 16830 consensus_queue.cc:260] T 0edbf034be4a41d79cbce7d0c876132d P 49598ea210614f1fbb942c1fb77c7932 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } } peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } }
I20260709 12:19:15.278760 16829 raft_consensus.cc:359] T 2ef5967933364223bef5315de3fa6bfe P 49e15cd7009e4d6198f9d9352a27e9d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } } peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } }
I20260709 12:19:15.279652 16829 raft_consensus.cc:385] T 2ef5967933364223bef5315de3fa6bfe P 49e15cd7009e4d6198f9d9352a27e9d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:19:15.282752 16830 ts_tablet_manager.cc:1434] T 0edbf034be4a41d79cbce7d0c876132d P 49598ea210614f1fbb942c1fb77c7932: Time spent starting tablet: real 0.020s	user 0.005s	sys 0.001s
I20260709 12:19:15.283290 16829 raft_consensus.cc:740] T 2ef5967933364223bef5315de3fa6bfe P 49e15cd7009e4d6198f9d9352a27e9d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49e15cd7009e4d6198f9d9352a27e9d1, State: Initialized, Role: FOLLOWER
I20260709 12:19:15.284700 16830 tablet_bootstrap.cc:492] T 3692a4a710ef469aa412d9870427fb63 P 49598ea210614f1fbb942c1fb77c7932: Bootstrap starting.
I20260709 12:19:15.286550 16829 consensus_queue.cc:260] T 2ef5967933364223bef5315de3fa6bfe P 49e15cd7009e4d6198f9d9352a27e9d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } } peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } }
I20260709 12:19:15.299722 16636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd736bd544c746238c77320b4a5706e0" candidate_uuid: "3e772c31b50d4c6693644565ef92de04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" is_pre_election: true
I20260709 12:19:15.300838 16636 raft_consensus.cc:2468] T dd736bd544c746238c77320b4a5706e0 P 49e15cd7009e4d6198f9d9352a27e9d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e772c31b50d4c6693644565ef92de04 in term 0.
I20260709 12:19:15.302976 16748 leader_election.cc:304] T dd736bd544c746238c77320b4a5706e0 P 3e772c31b50d4c6693644565ef92de04 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e772c31b50d4c6693644565ef92de04, 49e15cd7009e4d6198f9d9352a27e9d1; no voters: 
I20260709 12:19:15.304282 16848 raft_consensus.cc:2804] T dd736bd544c746238c77320b4a5706e0 P 3e772c31b50d4c6693644565ef92de04 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:19:15.304859 16848 raft_consensus.cc:493] T dd736bd544c746238c77320b4a5706e0 P 3e772c31b50d4c6693644565ef92de04 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:15.305220 16848 raft_consensus.cc:3060] T dd736bd544c746238c77320b4a5706e0 P 3e772c31b50d4c6693644565ef92de04 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:19:15.305454 16829 ts_tablet_manager.cc:1434] T 2ef5967933364223bef5315de3fa6bfe P 49e15cd7009e4d6198f9d9352a27e9d1: Time spent starting tablet: real 0.053s	user 0.009s	sys 0.000s
I20260709 12:19:15.306654 16829 tablet_bootstrap.cc:492] T 0086bd356ad04503ad306b7217280c34 P 49e15cd7009e4d6198f9d9352a27e9d1: Bootstrap starting.
I20260709 12:19:15.313987 16829 tablet_bootstrap.cc:654] T 0086bd356ad04503ad306b7217280c34 P 49e15cd7009e4d6198f9d9352a27e9d1: Neither blocks nor log segments found. Creating new log.
I20260709 12:19:15.322479 16830 tablet_bootstrap.cc:654] T 3692a4a710ef469aa412d9870427fb63 P 49598ea210614f1fbb942c1fb77c7932: Neither blocks nor log segments found. Creating new log.
I20260709 12:19:15.333917 16828 tablet_bootstrap.cc:492] T 927ab821d939440ab1def38a7dbd0358 P 3e772c31b50d4c6693644565ef92de04: No bootstrap required, opened a new log
I20260709 12:19:15.334455 16828 ts_tablet_manager.cc:1403] T 927ab821d939440ab1def38a7dbd0358 P 3e772c31b50d4c6693644565ef92de04: Time spent bootstrapping tablet: real 0.073s	user 0.023s	sys 0.012s
I20260709 12:19:15.337909 16830 tablet_bootstrap.cc:492] T 3692a4a710ef469aa412d9870427fb63 P 49598ea210614f1fbb942c1fb77c7932: No bootstrap required, opened a new log
I20260709 12:19:15.337913 16711 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd736bd544c746238c77320b4a5706e0" candidate_uuid: "3e772c31b50d4c6693644565ef92de04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49598ea210614f1fbb942c1fb77c7932" is_pre_election: true
I20260709 12:19:15.338780 16830 ts_tablet_manager.cc:1403] T 3692a4a710ef469aa412d9870427fb63 P 49598ea210614f1fbb942c1fb77c7932: Time spent bootstrapping tablet: real 0.054s	user 0.009s	sys 0.010s
W20260709 12:19:15.340401 16749 leader_election.cc:343] T dd736bd544c746238c77320b4a5706e0 P 3e772c31b50d4c6693644565ef92de04 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 49598ea210614f1fbb942c1fb77c7932 (127.16.29.66:35871): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:19:15.340137 16828 raft_consensus.cc:359] T 927ab821d939440ab1def38a7dbd0358 P 3e772c31b50d4c6693644565ef92de04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } } peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } }
I20260709 12:19:15.342536 16828 raft_consensus.cc:385] T 927ab821d939440ab1def38a7dbd0358 P 3e772c31b50d4c6693644565ef92de04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:19:15.342981 16828 raft_consensus.cc:740] T 927ab821d939440ab1def38a7dbd0358 P 3e772c31b50d4c6693644565ef92de04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e772c31b50d4c6693644565ef92de04, State: Initialized, Role: FOLLOWER
I20260709 12:19:15.344117 16838 raft_consensus.cc:493] T 2ef5967933364223bef5315de3fa6bfe P 49e15cd7009e4d6198f9d9352a27e9d1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:15.344633 16838 raft_consensus.cc:515] T 2ef5967933364223bef5315de3fa6bfe P 49e15cd7009e4d6198f9d9352a27e9d1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } } peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } }
I20260709 12:19:15.344740 16830 raft_consensus.cc:359] T 3692a4a710ef469aa412d9870427fb63 P 49598ea210614f1fbb942c1fb77c7932 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } } peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } }
I20260709 12:19:15.345818 16830 raft_consensus.cc:385] T 3692a4a710ef469aa412d9870427fb63 P 49598ea210614f1fbb942c1fb77c7932 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:19:15.346253 16830 raft_consensus.cc:740] T 3692a4a710ef469aa412d9870427fb63 P 49598ea210614f1fbb942c1fb77c7932 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49598ea210614f1fbb942c1fb77c7932, State: Initialized, Role: FOLLOWER
I20260709 12:19:15.349120 16838 leader_election.cc:290] T 2ef5967933364223bef5315de3fa6bfe P 49e15cd7009e4d6198f9d9352a27e9d1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49598ea210614f1fbb942c1fb77c7932 (127.16.29.66:35871), 3e772c31b50d4c6693644565ef92de04 (127.16.29.67:35049)
I20260709 12:19:15.348856 16830 consensus_queue.cc:260] T 3692a4a710ef469aa412d9870427fb63 P 49598ea210614f1fbb942c1fb77c7932 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } } peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } }
I20260709 12:19:15.349349 16828 consensus_queue.cc:260] T 927ab821d939440ab1def38a7dbd0358 P 3e772c31b50d4c6693644565ef92de04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } } peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } }
I20260709 12:19:15.348520 16848 raft_consensus.cc:515] T dd736bd544c746238c77320b4a5706e0 P 3e772c31b50d4c6693644565ef92de04 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } } peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } }
I20260709 12:19:15.358202 16830 ts_tablet_manager.cc:1434] T 3692a4a710ef469aa412d9870427fb63 P 49598ea210614f1fbb942c1fb77c7932: Time spent starting tablet: real 0.018s	user 0.009s	sys 0.000s
I20260709 12:19:15.362952 16830 tablet_bootstrap.cc:492] T dd736bd544c746238c77320b4a5706e0 P 49598ea210614f1fbb942c1fb77c7932: Bootstrap starting.
I20260709 12:19:15.365496 16840 raft_consensus.cc:493] T 2ef5967933364223bef5315de3fa6bfe P 49598ea210614f1fbb942c1fb77c7932 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:15.366240 16840 raft_consensus.cc:515] T 2ef5967933364223bef5315de3fa6bfe P 49598ea210614f1fbb942c1fb77c7932 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } } peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } }
I20260709 12:19:15.374351 16829 tablet_bootstrap.cc:492] T 0086bd356ad04503ad306b7217280c34 P 49e15cd7009e4d6198f9d9352a27e9d1: No bootstrap required, opened a new log
I20260709 12:19:15.374889 16829 ts_tablet_manager.cc:1403] T 0086bd356ad04503ad306b7217280c34 P 49e15cd7009e4d6198f9d9352a27e9d1: Time spent bootstrapping tablet: real 0.069s	user 0.015s	sys 0.000s
I20260709 12:19:15.378491 16829 raft_consensus.cc:359] T 0086bd356ad04503ad306b7217280c34 P 49e15cd7009e4d6198f9d9352a27e9d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } } peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } }
I20260709 12:19:15.379222 16829 raft_consensus.cc:385] T 0086bd356ad04503ad306b7217280c34 P 49e15cd7009e4d6198f9d9352a27e9d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:19:15.379546 16829 raft_consensus.cc:740] T 0086bd356ad04503ad306b7217280c34 P 49e15cd7009e4d6198f9d9352a27e9d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49e15cd7009e4d6198f9d9352a27e9d1, State: Initialized, Role: FOLLOWER
I20260709 12:19:15.380290 16829 consensus_queue.cc:260] T 0086bd356ad04503ad306b7217280c34 P 49e15cd7009e4d6198f9d9352a27e9d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } } peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } }
I20260709 12:19:15.382642 16829 ts_tablet_manager.cc:1434] T 0086bd356ad04503ad306b7217280c34 P 49e15cd7009e4d6198f9d9352a27e9d1: Time spent starting tablet: real 0.007s	user 0.006s	sys 0.000s
I20260709 12:19:15.384136 16829 tablet_bootstrap.cc:492] T 0edbf034be4a41d79cbce7d0c876132d P 49e15cd7009e4d6198f9d9352a27e9d1: Bootstrap starting.
I20260709 12:19:15.388971 16840 leader_election.cc:290] T 2ef5967933364223bef5315de3fa6bfe P 49598ea210614f1fbb942c1fb77c7932 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49e15cd7009e4d6198f9d9352a27e9d1 (127.16.29.65:44833), 3e772c31b50d4c6693644565ef92de04 (127.16.29.67:35049)
I20260709 12:19:15.392357 16636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ef5967933364223bef5315de3fa6bfe" candidate_uuid: "49598ea210614f1fbb942c1fb77c7932" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" is_pre_election: true
I20260709 12:19:15.393438 16838 raft_consensus.cc:493] T 3692a4a710ef469aa412d9870427fb63 P 49e15cd7009e4d6198f9d9352a27e9d1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:15.393491 16636 raft_consensus.cc:2468] T 2ef5967933364223bef5315de3fa6bfe P 49e15cd7009e4d6198f9d9352a27e9d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49598ea210614f1fbb942c1fb77c7932 in term 0.
I20260709 12:19:15.391788 16828 ts_tablet_manager.cc:1434] T 927ab821d939440ab1def38a7dbd0358 P 3e772c31b50d4c6693644565ef92de04: Time spent starting tablet: real 0.057s	user 0.012s	sys 0.000s
I20260709 12:19:15.395378 16848 leader_election.cc:290] T dd736bd544c746238c77320b4a5706e0 P 3e772c31b50d4c6693644565ef92de04 [CANDIDATE]: Term 1 election: Requested vote from peers 49e15cd7009e4d6198f9d9352a27e9d1 (127.16.29.65:44833), 49598ea210614f1fbb942c1fb77c7932 (127.16.29.66:35871)
I20260709 12:19:15.394100 16838 raft_consensus.cc:515] T 3692a4a710ef469aa412d9870427fb63 P 49e15cd7009e4d6198f9d9352a27e9d1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } } peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } }
I20260709 12:19:15.394099 16711 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ef5967933364223bef5315de3fa6bfe" candidate_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49598ea210614f1fbb942c1fb77c7932" is_pre_election: true
I20260709 12:19:15.397046 16828 tablet_bootstrap.cc:492] T 0086bd356ad04503ad306b7217280c34 P 3e772c31b50d4c6693644565ef92de04: Bootstrap starting.
I20260709 12:19:15.396883 16785 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ef5967933364223bef5315de3fa6bfe" candidate_uuid: "49598ea210614f1fbb942c1fb77c7932" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e772c31b50d4c6693644565ef92de04" is_pre_election: true
I20260709 12:19:15.400048 16710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd736bd544c746238c77320b4a5706e0" candidate_uuid: "3e772c31b50d4c6693644565ef92de04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49598ea210614f1fbb942c1fb77c7932"
I20260709 12:19:15.401029 16785 raft_consensus.cc:2468] T 2ef5967933364223bef5315de3fa6bfe P 3e772c31b50d4c6693644565ef92de04 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49598ea210614f1fbb942c1fb77c7932 in term 0.
I20260709 12:19:15.401530 16838 leader_election.cc:290] T 3692a4a710ef469aa412d9870427fb63 P 49e15cd7009e4d6198f9d9352a27e9d1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49598ea210614f1fbb942c1fb77c7932 (127.16.29.66:35871), 3e772c31b50d4c6693644565ef92de04 (127.16.29.67:35049)
I20260709 12:19:15.401628 16830 tablet_bootstrap.cc:654] T dd736bd544c746238c77320b4a5706e0 P 49598ea210614f1fbb942c1fb77c7932: Neither blocks nor log segments found. Creating new log.
I20260709 12:19:15.401898 16840 raft_consensus.cc:493] T 927ab821d939440ab1def38a7dbd0358 P 49598ea210614f1fbb942c1fb77c7932 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:15.394654 16635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd736bd544c746238c77320b4a5706e0" candidate_uuid: "3e772c31b50d4c6693644565ef92de04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49e15cd7009e4d6198f9d9352a27e9d1"
I20260709 12:19:15.400800 16674 leader_election.cc:304] T 2ef5967933364223bef5315de3fa6bfe P 49598ea210614f1fbb942c1fb77c7932 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49598ea210614f1fbb942c1fb77c7932, 49e15cd7009e4d6198f9d9352a27e9d1; no voters: 
I20260709 12:19:15.399106 16711 raft_consensus.cc:2468] T 2ef5967933364223bef5315de3fa6bfe P 49598ea210614f1fbb942c1fb77c7932 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49e15cd7009e4d6198f9d9352a27e9d1 in term 0.
W20260709 12:19:15.404600 16749 leader_election.cc:343] T dd736bd544c746238c77320b4a5706e0 P 3e772c31b50d4c6693644565ef92de04 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 49598ea210614f1fbb942c1fb77c7932 (127.16.29.66:35871): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:19:15.406018 16834 raft_consensus.cc:2804] T 2ef5967933364223bef5315de3fa6bfe P 49598ea210614f1fbb942c1fb77c7932 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:19:15.403715 16840 raft_consensus.cc:515] T 927ab821d939440ab1def38a7dbd0358 P 49598ea210614f1fbb942c1fb77c7932 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } } peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } }
I20260709 12:19:15.406417 16834 raft_consensus.cc:493] T 2ef5967933364223bef5315de3fa6bfe P 49598ea210614f1fbb942c1fb77c7932 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:15.406801 16834 raft_consensus.cc:3060] T 2ef5967933364223bef5315de3fa6bfe P 49598ea210614f1fbb942c1fb77c7932 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:19:15.406744 16711 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3692a4a710ef469aa412d9870427fb63" candidate_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49598ea210614f1fbb942c1fb77c7932" is_pre_election: true
I20260709 12:19:15.407431 16711 raft_consensus.cc:2468] T 3692a4a710ef469aa412d9870427fb63 P 49598ea210614f1fbb942c1fb77c7932 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49e15cd7009e4d6198f9d9352a27e9d1 in term 0.
I20260709 12:19:15.407594 16635 raft_consensus.cc:3060] T dd736bd544c746238c77320b4a5706e0 P 49e15cd7009e4d6198f9d9352a27e9d1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:19:15.409914 16636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "927ab821d939440ab1def38a7dbd0358" candidate_uuid: "49598ea210614f1fbb942c1fb77c7932" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" is_pre_election: true
I20260709 12:19:15.413906 16840 leader_election.cc:290] T 927ab821d939440ab1def38a7dbd0358 P 49598ea210614f1fbb942c1fb77c7932 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49e15cd7009e4d6198f9d9352a27e9d1 (127.16.29.65:44833), 3e772c31b50d4c6693644565ef92de04 (127.16.29.67:35049)
W20260709 12:19:15.418045 16674 leader_election.cc:343] T 927ab821d939440ab1def38a7dbd0358 P 49598ea210614f1fbb942c1fb77c7932 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 49e15cd7009e4d6198f9d9352a27e9d1 (127.16.29.65:44833): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:19:15.418711 16785 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "927ab821d939440ab1def38a7dbd0358" candidate_uuid: "49598ea210614f1fbb942c1fb77c7932" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e772c31b50d4c6693644565ef92de04" is_pre_election: true
I20260709 12:19:15.415098 16600 leader_election.cc:304] T 2ef5967933364223bef5315de3fa6bfe P 49e15cd7009e4d6198f9d9352a27e9d1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49598ea210614f1fbb942c1fb77c7932, 49e15cd7009e4d6198f9d9352a27e9d1; no voters: 
I20260709 12:19:15.420214 16785 raft_consensus.cc:2468] T 927ab821d939440ab1def38a7dbd0358 P 3e772c31b50d4c6693644565ef92de04 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49598ea210614f1fbb942c1fb77c7932 in term 0.
I20260709 12:19:15.420815 16838 raft_consensus.cc:2804] T 2ef5967933364223bef5315de3fa6bfe P 49e15cd7009e4d6198f9d9352a27e9d1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:19:15.421779 16838 raft_consensus.cc:493] T 2ef5967933364223bef5315de3fa6bfe P 49e15cd7009e4d6198f9d9352a27e9d1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:15.422139 16838 raft_consensus.cc:3060] T 2ef5967933364223bef5315de3fa6bfe P 49e15cd7009e4d6198f9d9352a27e9d1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:19:15.422214 16674 leader_election.cc:304] T 927ab821d939440ab1def38a7dbd0358 P 49598ea210614f1fbb942c1fb77c7932 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3e772c31b50d4c6693644565ef92de04, 49598ea210614f1fbb942c1fb77c7932; no voters: 49e15cd7009e4d6198f9d9352a27e9d1
I20260709 12:19:15.423923 16600 leader_election.cc:304] T 3692a4a710ef469aa412d9870427fb63 P 49e15cd7009e4d6198f9d9352a27e9d1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49598ea210614f1fbb942c1fb77c7932, 49e15cd7009e4d6198f9d9352a27e9d1; no voters: 
I20260709 12:19:15.425014 16840 raft_consensus.cc:2804] T 927ab821d939440ab1def38a7dbd0358 P 49598ea210614f1fbb942c1fb77c7932 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:19:15.426605 16840 raft_consensus.cc:493] T 927ab821d939440ab1def38a7dbd0358 P 49598ea210614f1fbb942c1fb77c7932 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:15.428526 16785 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ef5967933364223bef5315de3fa6bfe" candidate_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e772c31b50d4c6693644565ef92de04" is_pre_election: true
I20260709 12:19:15.428071 16840 raft_consensus.cc:3060] T 927ab821d939440ab1def38a7dbd0358 P 49598ea210614f1fbb942c1fb77c7932 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:19:15.430871 16784 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3692a4a710ef469aa412d9870427fb63" candidate_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e772c31b50d4c6693644565ef92de04" is_pre_election: true
I20260709 12:19:15.429324 16785 raft_consensus.cc:2468] T 2ef5967933364223bef5315de3fa6bfe P 3e772c31b50d4c6693644565ef92de04 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49e15cd7009e4d6198f9d9352a27e9d1 in term 0.
I20260709 12:19:15.441300 16829 tablet_bootstrap.cc:654] T 0edbf034be4a41d79cbce7d0c876132d P 49e15cd7009e4d6198f9d9352a27e9d1: Neither blocks nor log segments found. Creating new log.
I20260709 12:19:15.441880 16635 raft_consensus.cc:2468] T dd736bd544c746238c77320b4a5706e0 P 49e15cd7009e4d6198f9d9352a27e9d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e772c31b50d4c6693644565ef92de04 in term 1.
I20260709 12:19:15.443693 16748 leader_election.cc:304] T dd736bd544c746238c77320b4a5706e0 P 3e772c31b50d4c6693644565ef92de04 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3e772c31b50d4c6693644565ef92de04, 49e15cd7009e4d6198f9d9352a27e9d1; no voters: 49598ea210614f1fbb942c1fb77c7932
I20260709 12:19:15.445508 16848 raft_consensus.cc:2804] T dd736bd544c746238c77320b4a5706e0 P 3e772c31b50d4c6693644565ef92de04 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:19:15.446159 16848 raft_consensus.cc:697] T dd736bd544c746238c77320b4a5706e0 P 3e772c31b50d4c6693644565ef92de04 [term 1 LEADER]: Becoming Leader. State: Replica: 3e772c31b50d4c6693644565ef92de04, State: Running, Role: LEADER
I20260709 12:19:15.447240 16848 consensus_queue.cc:237] T dd736bd544c746238c77320b4a5706e0 P 3e772c31b50d4c6693644565ef92de04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } } peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } }
I20260709 12:19:15.452075 16838 raft_consensus.cc:515] T 2ef5967933364223bef5315de3fa6bfe P 49e15cd7009e4d6198f9d9352a27e9d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } } peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } }
I20260709 12:19:15.457056 16834 raft_consensus.cc:515] T 2ef5967933364223bef5315de3fa6bfe P 49598ea210614f1fbb942c1fb77c7932 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } } peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } }
W20260709 12:19:15.460601 16599 leader_election.cc:343] T 3692a4a710ef469aa412d9870427fb63 P 49e15cd7009e4d6198f9d9352a27e9d1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3e772c31b50d4c6693644565ef92de04 (127.16.29.67:35049): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:19:15.461416 16835 raft_consensus.cc:493] T 0edbf034be4a41d79cbce7d0c876132d P 3e772c31b50d4c6693644565ef92de04 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:15.462261 16835 raft_consensus.cc:515] T 0edbf034be4a41d79cbce7d0c876132d P 3e772c31b50d4c6693644565ef92de04 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } } peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } }
I20260709 12:19:15.463092 16834 leader_election.cc:290] T 2ef5967933364223bef5315de3fa6bfe P 49598ea210614f1fbb942c1fb77c7932 [CANDIDATE]: Term 1 election: Requested vote from peers 49e15cd7009e4d6198f9d9352a27e9d1 (127.16.29.65:44833), 3e772c31b50d4c6693644565ef92de04 (127.16.29.67:35049)
I20260709 12:19:15.465212 16711 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0edbf034be4a41d79cbce7d0c876132d" candidate_uuid: "3e772c31b50d4c6693644565ef92de04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49598ea210614f1fbb942c1fb77c7932" is_pre_election: true
I20260709 12:19:15.465636 16636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0edbf034be4a41d79cbce7d0c876132d" candidate_uuid: "3e772c31b50d4c6693644565ef92de04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" is_pre_election: true
I20260709 12:19:15.465914 16711 raft_consensus.cc:2468] T 0edbf034be4a41d79cbce7d0c876132d P 49598ea210614f1fbb942c1fb77c7932 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e772c31b50d4c6693644565ef92de04 in term 0.
I20260709 12:19:15.465231 16635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ef5967933364223bef5315de3fa6bfe" candidate_uuid: "49598ea210614f1fbb942c1fb77c7932" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49e15cd7009e4d6198f9d9352a27e9d1"
W20260709 12:19:15.467662 16748 leader_election.cc:343] T 0edbf034be4a41d79cbce7d0c876132d P 3e772c31b50d4c6693644565ef92de04 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 49e15cd7009e4d6198f9d9352a27e9d1 (127.16.29.65:44833): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:19:15.468987 16635 raft_consensus.cc:2393] T 2ef5967933364223bef5315de3fa6bfe P 49e15cd7009e4d6198f9d9352a27e9d1 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 49598ea210614f1fbb942c1fb77c7932 in current term 1: Already voted for candidate 49e15cd7009e4d6198f9d9352a27e9d1 in this term.
I20260709 12:19:15.464497 16835 leader_election.cc:290] T 0edbf034be4a41d79cbce7d0c876132d P 3e772c31b50d4c6693644565ef92de04 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49598ea210614f1fbb942c1fb77c7932 (127.16.29.66:35871), 49e15cd7009e4d6198f9d9352a27e9d1 (127.16.29.65:44833)
I20260709 12:19:15.469023 16749 leader_election.cc:304] T 0edbf034be4a41d79cbce7d0c876132d P 3e772c31b50d4c6693644565ef92de04 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3e772c31b50d4c6693644565ef92de04, 49598ea210614f1fbb942c1fb77c7932; no voters: 49e15cd7009e4d6198f9d9352a27e9d1
I20260709 12:19:15.471989 16835 raft_consensus.cc:2804] T 0edbf034be4a41d79cbce7d0c876132d P 3e772c31b50d4c6693644565ef92de04 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:19:15.472653 16835 raft_consensus.cc:493] T 0edbf034be4a41d79cbce7d0c876132d P 3e772c31b50d4c6693644565ef92de04 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:15.473235 16835 raft_consensus.cc:3060] T 0edbf034be4a41d79cbce7d0c876132d P 3e772c31b50d4c6693644565ef92de04 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:19:15.478901 16840 raft_consensus.cc:515] T 927ab821d939440ab1def38a7dbd0358 P 49598ea210614f1fbb942c1fb77c7932 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } } peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } }
I20260709 12:19:15.480289 16838 leader_election.cc:290] T 2ef5967933364223bef5315de3fa6bfe P 49e15cd7009e4d6198f9d9352a27e9d1 [CANDIDATE]: Term 1 election: Requested vote from peers 49598ea210614f1fbb942c1fb77c7932 (127.16.29.66:35871), 3e772c31b50d4c6693644565ef92de04 (127.16.29.67:35049)
I20260709 12:19:15.489159 16785 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ef5967933364223bef5315de3fa6bfe" candidate_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e772c31b50d4c6693644565ef92de04"
I20260709 12:19:15.498435 16635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "927ab821d939440ab1def38a7dbd0358" candidate_uuid: "49598ea210614f1fbb942c1fb77c7932" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49e15cd7009e4d6198f9d9352a27e9d1"
I20260709 12:19:15.496140 16785 raft_consensus.cc:3060] T 2ef5967933364223bef5315de3fa6bfe P 3e772c31b50d4c6693644565ef92de04 [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:19:15.501097 16674 leader_election.cc:343] T 927ab821d939440ab1def38a7dbd0358 P 49598ea210614f1fbb942c1fb77c7932 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 49e15cd7009e4d6198f9d9352a27e9d1 (127.16.29.65:44833): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:19:15.496631 16838 raft_consensus.cc:2804] T 3692a4a710ef469aa412d9870427fb63 P 49e15cd7009e4d6198f9d9352a27e9d1 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20260709 12:19:15.497181 16600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:35871, user_credentials={real_user=slave}} blocked reactor thread for 73740.7us
I20260709 12:19:15.500665 16840 leader_election.cc:290] T 927ab821d939440ab1def38a7dbd0358 P 49598ea210614f1fbb942c1fb77c7932 [CANDIDATE]: Term 1 election: Requested vote from peers 49e15cd7009e4d6198f9d9352a27e9d1 (127.16.29.65:44833), 3e772c31b50d4c6693644565ef92de04 (127.16.29.67:35049)
I20260709 12:19:15.506620 16835 raft_consensus.cc:515] T 0edbf034be4a41d79cbce7d0c876132d P 3e772c31b50d4c6693644565ef92de04 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } } peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } }
I20260709 12:19:15.504123 16838 raft_consensus.cc:493] T 3692a4a710ef469aa412d9870427fb63 P 49e15cd7009e4d6198f9d9352a27e9d1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:15.508860 16838 raft_consensus.cc:3060] T 3692a4a710ef469aa412d9870427fb63 P 49e15cd7009e4d6198f9d9352a27e9d1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:19:15.511482 16829 tablet_bootstrap.cc:492] T 0edbf034be4a41d79cbce7d0c876132d P 49e15cd7009e4d6198f9d9352a27e9d1: No bootstrap required, opened a new log
I20260709 12:19:15.512014 16829 ts_tablet_manager.cc:1403] T 0edbf034be4a41d79cbce7d0c876132d P 49e15cd7009e4d6198f9d9352a27e9d1: Time spent bootstrapping tablet: real 0.128s	user 0.023s	sys 0.010s
I20260709 12:19:15.507614 16783 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "927ab821d939440ab1def38a7dbd0358" candidate_uuid: "49598ea210614f1fbb942c1fb77c7932" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e772c31b50d4c6693644565ef92de04"
I20260709 12:19:15.514937 16711 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ef5967933364223bef5315de3fa6bfe" candidate_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49598ea210614f1fbb942c1fb77c7932"
I20260709 12:19:15.515520 16783 raft_consensus.cc:3060] T 927ab821d939440ab1def38a7dbd0358 P 3e772c31b50d4c6693644565ef92de04 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:19:15.515036 16785 raft_consensus.cc:2468] T 2ef5967933364223bef5315de3fa6bfe P 3e772c31b50d4c6693644565ef92de04 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49e15cd7009e4d6198f9d9352a27e9d1 in term 1.
I20260709 12:19:15.518620 16711 raft_consensus.cc:2393] T 2ef5967933364223bef5315de3fa6bfe P 49598ea210614f1fbb942c1fb77c7932 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 49e15cd7009e4d6198f9d9352a27e9d1 in current term 1: Already voted for candidate 49598ea210614f1fbb942c1fb77c7932 in this term.
I20260709 12:19:15.521239 16835 leader_election.cc:290] T 0edbf034be4a41d79cbce7d0c876132d P 3e772c31b50d4c6693644565ef92de04 [CANDIDATE]: Term 1 election: Requested vote from peers 49598ea210614f1fbb942c1fb77c7932 (127.16.29.66:35871), 49e15cd7009e4d6198f9d9352a27e9d1 (127.16.29.65:44833)
I20260709 12:19:15.521471 16828 tablet_bootstrap.cc:654] T 0086bd356ad04503ad306b7217280c34 P 3e772c31b50d4c6693644565ef92de04: Neither blocks nor log segments found. Creating new log.
I20260709 12:19:15.517138 16829 raft_consensus.cc:359] T 0edbf034be4a41d79cbce7d0c876132d P 49e15cd7009e4d6198f9d9352a27e9d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } } peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } }
I20260709 12:19:15.523070 16599 leader_election.cc:304] T 2ef5967933364223bef5315de3fa6bfe P 49e15cd7009e4d6198f9d9352a27e9d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e772c31b50d4c6693644565ef92de04, 49e15cd7009e4d6198f9d9352a27e9d1; no voters: 
I20260709 12:19:15.523450 16710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0edbf034be4a41d79cbce7d0c876132d" candidate_uuid: "3e772c31b50d4c6693644565ef92de04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49598ea210614f1fbb942c1fb77c7932"
I20260709 12:19:15.524273 16635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0edbf034be4a41d79cbce7d0c876132d" candidate_uuid: "3e772c31b50d4c6693644565ef92de04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49e15cd7009e4d6198f9d9352a27e9d1"
I20260709 12:19:15.524425 16710 raft_consensus.cc:3060] T 0edbf034be4a41d79cbce7d0c876132d P 49598ea210614f1fbb942c1fb77c7932 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:19:15.525513 16858 raft_consensus.cc:2804] T 2ef5967933364223bef5315de3fa6bfe P 49e15cd7009e4d6198f9d9352a27e9d1 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:19:15.525163 16838 raft_consensus.cc:515] T 3692a4a710ef469aa412d9870427fb63 P 49e15cd7009e4d6198f9d9352a27e9d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } } peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } }
I20260709 12:19:15.524506 16829 raft_consensus.cc:385] T 0edbf034be4a41d79cbce7d0c876132d P 49e15cd7009e4d6198f9d9352a27e9d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:19:15.526665 16829 raft_consensus.cc:740] T 0edbf034be4a41d79cbce7d0c876132d P 49e15cd7009e4d6198f9d9352a27e9d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49e15cd7009e4d6198f9d9352a27e9d1, State: Initialized, Role: FOLLOWER
I20260709 12:19:15.528189 16829 consensus_queue.cc:260] T 0edbf034be4a41d79cbce7d0c876132d P 49e15cd7009e4d6198f9d9352a27e9d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } } peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } }
I20260709 12:19:15.529654 16635 raft_consensus.cc:3060] T 0edbf034be4a41d79cbce7d0c876132d P 49e15cd7009e4d6198f9d9352a27e9d1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:19:15.529800 16858 raft_consensus.cc:697] T 2ef5967933364223bef5315de3fa6bfe P 49e15cd7009e4d6198f9d9352a27e9d1 [term 1 LEADER]: Becoming Leader. State: Replica: 49e15cd7009e4d6198f9d9352a27e9d1, State: Running, Role: LEADER
I20260709 12:19:15.531302 16858 consensus_queue.cc:237] T 2ef5967933364223bef5315de3fa6bfe P 49e15cd7009e4d6198f9d9352a27e9d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } } peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } }
I20260709 12:19:15.533048 16830 tablet_bootstrap.cc:492] T dd736bd544c746238c77320b4a5706e0 P 49598ea210614f1fbb942c1fb77c7932: No bootstrap required, opened a new log
I20260709 12:19:15.534711 16635 raft_consensus.cc:2468] T 0edbf034be4a41d79cbce7d0c876132d P 49e15cd7009e4d6198f9d9352a27e9d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e772c31b50d4c6693644565ef92de04 in term 1.
I20260709 12:19:15.535017 16830 ts_tablet_manager.cc:1403] T dd736bd544c746238c77320b4a5706e0 P 49598ea210614f1fbb942c1fb77c7932: Time spent bootstrapping tablet: real 0.173s	user 0.013s	sys 0.019s
I20260709 12:19:15.534391 16838 leader_election.cc:290] T 3692a4a710ef469aa412d9870427fb63 P 49e15cd7009e4d6198f9d9352a27e9d1 [CANDIDATE]: Term 1 election: Requested vote from peers 49598ea210614f1fbb942c1fb77c7932 (127.16.29.66:35871), 3e772c31b50d4c6693644565ef92de04 (127.16.29.67:35049)
I20260709 12:19:15.536031 16711 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3692a4a710ef469aa412d9870427fb63" candidate_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49598ea210614f1fbb942c1fb77c7932"
I20260709 12:19:15.536953 16711 raft_consensus.cc:3060] T 3692a4a710ef469aa412d9870427fb63 P 49598ea210614f1fbb942c1fb77c7932 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:19:15.539249 16785 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3692a4a710ef469aa412d9870427fb63" candidate_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e772c31b50d4c6693644565ef92de04"
W20260709 12:19:15.541172 16599 leader_election.cc:343] T 3692a4a710ef469aa412d9870427fb63 P 49e15cd7009e4d6198f9d9352a27e9d1 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3e772c31b50d4c6693644565ef92de04 (127.16.29.67:35049): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:19:15.546528 16829 ts_tablet_manager.cc:1434] T 0edbf034be4a41d79cbce7d0c876132d P 49e15cd7009e4d6198f9d9352a27e9d1: Time spent starting tablet: real 0.034s	user 0.008s	sys 0.001s
I20260709 12:19:15.547091 16783 raft_consensus.cc:2468] T 927ab821d939440ab1def38a7dbd0358 P 3e772c31b50d4c6693644565ef92de04 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49598ea210614f1fbb942c1fb77c7932 in term 1.
I20260709 12:19:15.547567 16829 tablet_bootstrap.cc:492] T 927ab821d939440ab1def38a7dbd0358 P 49e15cd7009e4d6198f9d9352a27e9d1: Bootstrap starting.
I20260709 12:19:15.548219 16674 leader_election.cc:304] T 927ab821d939440ab1def38a7dbd0358 P 49598ea210614f1fbb942c1fb77c7932 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3e772c31b50d4c6693644565ef92de04, 49598ea210614f1fbb942c1fb77c7932; no voters: 49e15cd7009e4d6198f9d9352a27e9d1
I20260709 12:19:15.548979 16840 raft_consensus.cc:2804] T 927ab821d939440ab1def38a7dbd0358 P 49598ea210614f1fbb942c1fb77c7932 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:19:15.549439 16840 raft_consensus.cc:697] T 927ab821d939440ab1def38a7dbd0358 P 49598ea210614f1fbb942c1fb77c7932 [term 1 LEADER]: Becoming Leader. State: Replica: 49598ea210614f1fbb942c1fb77c7932, State: Running, Role: LEADER
I20260709 12:19:15.545511 16830 raft_consensus.cc:359] T dd736bd544c746238c77320b4a5706e0 P 49598ea210614f1fbb942c1fb77c7932 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } } peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } }
I20260709 12:19:15.550796 16710 raft_consensus.cc:2468] T 0edbf034be4a41d79cbce7d0c876132d P 49598ea210614f1fbb942c1fb77c7932 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e772c31b50d4c6693644565ef92de04 in term 1.
I20260709 12:19:15.550257 16840 consensus_queue.cc:237] T 927ab821d939440ab1def38a7dbd0358 P 49598ea210614f1fbb942c1fb77c7932 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } } peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } }
I20260709 12:19:15.551028 16711 raft_consensus.cc:2468] T 3692a4a710ef469aa412d9870427fb63 P 49598ea210614f1fbb942c1fb77c7932 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49e15cd7009e4d6198f9d9352a27e9d1 in term 1.
I20260709 12:19:15.551520 16830 raft_consensus.cc:385] T dd736bd544c746238c77320b4a5706e0 P 49598ea210614f1fbb942c1fb77c7932 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:19:15.552583 16830 raft_consensus.cc:740] T dd736bd544c746238c77320b4a5706e0 P 49598ea210614f1fbb942c1fb77c7932 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49598ea210614f1fbb942c1fb77c7932, State: Initialized, Role: FOLLOWER
I20260709 12:19:15.554428 16749 leader_election.cc:304] T 0edbf034be4a41d79cbce7d0c876132d P 3e772c31b50d4c6693644565ef92de04 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e772c31b50d4c6693644565ef92de04, 49598ea210614f1fbb942c1fb77c7932; no voters: 
I20260709 12:19:15.555464 16835 raft_consensus.cc:2804] T 0edbf034be4a41d79cbce7d0c876132d P 3e772c31b50d4c6693644565ef92de04 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:19:15.554385 16830 consensus_queue.cc:260] T dd736bd544c746238c77320b4a5706e0 P 49598ea210614f1fbb942c1fb77c7932 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } } peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } }
I20260709 12:19:15.556035 16835 raft_consensus.cc:697] T 0edbf034be4a41d79cbce7d0c876132d P 3e772c31b50d4c6693644565ef92de04 [term 1 LEADER]: Becoming Leader. State: Replica: 3e772c31b50d4c6693644565ef92de04, State: Running, Role: LEADER
I20260709 12:19:15.557574 16828 tablet_bootstrap.cc:492] T 0086bd356ad04503ad306b7217280c34 P 3e772c31b50d4c6693644565ef92de04: No bootstrap required, opened a new log
I20260709 12:19:15.557658 16835 consensus_queue.cc:237] T 0edbf034be4a41d79cbce7d0c876132d P 3e772c31b50d4c6693644565ef92de04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } } peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } }
I20260709 12:19:15.557592 16600 leader_election.cc:304] T 3692a4a710ef469aa412d9870427fb63 P 49e15cd7009e4d6198f9d9352a27e9d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 49598ea210614f1fbb942c1fb77c7932, 49e15cd7009e4d6198f9d9352a27e9d1; no voters: 3e772c31b50d4c6693644565ef92de04
I20260709 12:19:15.561604 16838 raft_consensus.cc:2804] T 3692a4a710ef469aa412d9870427fb63 P 49e15cd7009e4d6198f9d9352a27e9d1 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:19:15.560937 16828 ts_tablet_manager.cc:1403] T 0086bd356ad04503ad306b7217280c34 P 3e772c31b50d4c6693644565ef92de04: Time spent bootstrapping tablet: real 0.164s	user 0.015s	sys 0.006s
I20260709 12:19:15.562835 16838 raft_consensus.cc:697] T 3692a4a710ef469aa412d9870427fb63 P 49e15cd7009e4d6198f9d9352a27e9d1 [term 1 LEADER]: Becoming Leader. State: Replica: 49e15cd7009e4d6198f9d9352a27e9d1, State: Running, Role: LEADER
I20260709 12:19:15.564373 16838 consensus_queue.cc:237] T 3692a4a710ef469aa412d9870427fb63 P 49e15cd7009e4d6198f9d9352a27e9d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } } peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } }
I20260709 12:19:15.566833 16830 ts_tablet_manager.cc:1434] T dd736bd544c746238c77320b4a5706e0 P 49598ea210614f1fbb942c1fb77c7932: Time spent starting tablet: real 0.031s	user 0.006s	sys 0.003s
I20260709 12:19:15.567350 16828 raft_consensus.cc:359] T 0086bd356ad04503ad306b7217280c34 P 3e772c31b50d4c6693644565ef92de04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } } peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } }
I20260709 12:19:15.569653 16828 raft_consensus.cc:385] T 0086bd356ad04503ad306b7217280c34 P 3e772c31b50d4c6693644565ef92de04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:19:15.569981 16828 raft_consensus.cc:740] T 0086bd356ad04503ad306b7217280c34 P 3e772c31b50d4c6693644565ef92de04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e772c31b50d4c6693644565ef92de04, State: Initialized, Role: FOLLOWER
I20260709 12:19:15.572155 16830 tablet_bootstrap.cc:492] T af674c5ffb4e44dab6f15ab84dbeb737 P 49598ea210614f1fbb942c1fb77c7932: Bootstrap starting.
I20260709 12:19:15.579250 16828 consensus_queue.cc:260] T 0086bd356ad04503ad306b7217280c34 P 3e772c31b50d4c6693644565ef92de04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } } peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } }
I20260709 12:19:15.580585 16829 tablet_bootstrap.cc:654] T 927ab821d939440ab1def38a7dbd0358 P 49e15cd7009e4d6198f9d9352a27e9d1: Neither blocks nor log segments found. Creating new log.
I20260709 12:19:15.586355 16784 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ef5967933364223bef5315de3fa6bfe" candidate_uuid: "49598ea210614f1fbb942c1fb77c7932" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e772c31b50d4c6693644565ef92de04"
I20260709 12:19:15.587544 16784 raft_consensus.cc:2393] T 2ef5967933364223bef5315de3fa6bfe P 3e772c31b50d4c6693644565ef92de04 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 49598ea210614f1fbb942c1fb77c7932 in current term 1: Already voted for candidate 49e15cd7009e4d6198f9d9352a27e9d1 in this term.
I20260709 12:19:15.611570 16531 catalog_manager.cc:5697] T 927ab821d939440ab1def38a7dbd0358 P 49598ea210614f1fbb942c1fb77c7932 reported cstate change: term changed from 0 to 1, leader changed from <none> to 49598ea210614f1fbb942c1fb77c7932 (127.16.29.66). New cstate: current_term: 1 leader_uuid: "49598ea210614f1fbb942c1fb77c7932" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } health_report { overall_health: UNKNOWN } } }
I20260709 12:19:15.617053 16674 leader_election.cc:304] T 2ef5967933364223bef5315de3fa6bfe P 49598ea210614f1fbb942c1fb77c7932 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 49598ea210614f1fbb942c1fb77c7932; no voters: 3e772c31b50d4c6693644565ef92de04, 49e15cd7009e4d6198f9d9352a27e9d1
I20260709 12:19:15.618788 16840 raft_consensus.cc:2749] T 2ef5967933364223bef5315de3fa6bfe P 49598ea210614f1fbb942c1fb77c7932 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 12:19:15.639181 16838 raft_consensus.cc:493] T 0086bd356ad04503ad306b7217280c34 P 49e15cd7009e4d6198f9d9352a27e9d1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:15.648622 16828 ts_tablet_manager.cc:1434] T 0086bd356ad04503ad306b7217280c34 P 3e772c31b50d4c6693644565ef92de04: Time spent starting tablet: real 0.085s	user 0.005s	sys 0.003s
I20260709 12:19:15.647789 16838 raft_consensus.cc:515] T 0086bd356ad04503ad306b7217280c34 P 49e15cd7009e4d6198f9d9352a27e9d1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } } peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } }
I20260709 12:19:15.654901 16828 tablet_bootstrap.cc:492] T 3692a4a710ef469aa412d9870427fb63 P 3e772c31b50d4c6693644565ef92de04: Bootstrap starting.
I20260709 12:19:15.655754 16532 catalog_manager.cc:5697] T dd736bd544c746238c77320b4a5706e0 P 3e772c31b50d4c6693644565ef92de04 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3e772c31b50d4c6693644565ef92de04 (127.16.29.67). New cstate: current_term: 1 leader_uuid: "3e772c31b50d4c6693644565ef92de04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } health_report { overall_health: HEALTHY } } }
I20260709 12:19:15.659977 16711 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0086bd356ad04503ad306b7217280c34" candidate_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49598ea210614f1fbb942c1fb77c7932" is_pre_election: true
I20260709 12:19:15.660851 16711 raft_consensus.cc:2468] T 0086bd356ad04503ad306b7217280c34 P 49598ea210614f1fbb942c1fb77c7932 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49e15cd7009e4d6198f9d9352a27e9d1 in term 0.
I20260709 12:19:15.661968 16600 leader_election.cc:304] T 0086bd356ad04503ad306b7217280c34 P 49e15cd7009e4d6198f9d9352a27e9d1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49598ea210614f1fbb942c1fb77c7932, 49e15cd7009e4d6198f9d9352a27e9d1; no voters: 
I20260709 12:19:15.665518 16861 raft_consensus.cc:2804] T 0086bd356ad04503ad306b7217280c34 P 49e15cd7009e4d6198f9d9352a27e9d1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:19:15.665683 16784 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0086bd356ad04503ad306b7217280c34" candidate_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e772c31b50d4c6693644565ef92de04" is_pre_election: true
I20260709 12:19:15.666150 16861 raft_consensus.cc:493] T 0086bd356ad04503ad306b7217280c34 P 49e15cd7009e4d6198f9d9352a27e9d1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:15.666503 16784 raft_consensus.cc:2468] T 0086bd356ad04503ad306b7217280c34 P 3e772c31b50d4c6693644565ef92de04 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49e15cd7009e4d6198f9d9352a27e9d1 in term 0.
I20260709 12:19:15.666622 16861 raft_consensus.cc:3060] T 0086bd356ad04503ad306b7217280c34 P 49e15cd7009e4d6198f9d9352a27e9d1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:19:15.664505 16838 leader_election.cc:290] T 0086bd356ad04503ad306b7217280c34 P 49e15cd7009e4d6198f9d9352a27e9d1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49598ea210614f1fbb942c1fb77c7932 (127.16.29.66:35871), 3e772c31b50d4c6693644565ef92de04 (127.16.29.67:35049)
I20260709 12:19:15.679100 16830 tablet_bootstrap.cc:654] T af674c5ffb4e44dab6f15ab84dbeb737 P 49598ea210614f1fbb942c1fb77c7932: Neither blocks nor log segments found. Creating new log.
I20260709 12:19:15.680452 16533 catalog_manager.cc:5697] T 2ef5967933364223bef5315de3fa6bfe P 49e15cd7009e4d6198f9d9352a27e9d1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 49e15cd7009e4d6198f9d9352a27e9d1 (127.16.29.65). New cstate: current_term: 1 leader_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } health_report { overall_health: UNKNOWN } } }
I20260709 12:19:15.686666 16828 tablet_bootstrap.cc:654] T 3692a4a710ef469aa412d9870427fb63 P 3e772c31b50d4c6693644565ef92de04: Neither blocks nor log segments found. Creating new log.
I20260709 12:19:15.689545 16829 tablet_bootstrap.cc:492] T 927ab821d939440ab1def38a7dbd0358 P 49e15cd7009e4d6198f9d9352a27e9d1: No bootstrap required, opened a new log
I20260709 12:19:15.690122 16829 ts_tablet_manager.cc:1403] T 927ab821d939440ab1def38a7dbd0358 P 49e15cd7009e4d6198f9d9352a27e9d1: Time spent bootstrapping tablet: real 0.143s	user 0.023s	sys 0.048s
I20260709 12:19:15.690486 16861 raft_consensus.cc:515] T 0086bd356ad04503ad306b7217280c34 P 49e15cd7009e4d6198f9d9352a27e9d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } } peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } }
I20260709 12:19:15.698549 16829 raft_consensus.cc:359] T 927ab821d939440ab1def38a7dbd0358 P 49e15cd7009e4d6198f9d9352a27e9d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } } peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } }
I20260709 12:19:15.699450 16829 raft_consensus.cc:385] T 927ab821d939440ab1def38a7dbd0358 P 49e15cd7009e4d6198f9d9352a27e9d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:19:15.699791 16829 raft_consensus.cc:740] T 927ab821d939440ab1def38a7dbd0358 P 49e15cd7009e4d6198f9d9352a27e9d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49e15cd7009e4d6198f9d9352a27e9d1, State: Initialized, Role: FOLLOWER
I20260709 12:19:15.707015 16711 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0086bd356ad04503ad306b7217280c34" candidate_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49598ea210614f1fbb942c1fb77c7932"
I20260709 12:19:15.707710 16711 raft_consensus.cc:3060] T 0086bd356ad04503ad306b7217280c34 P 49598ea210614f1fbb942c1fb77c7932 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:19:15.708398 16784 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0086bd356ad04503ad306b7217280c34" candidate_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e772c31b50d4c6693644565ef92de04"
I20260709 12:19:15.709406 16784 raft_consensus.cc:3060] T 0086bd356ad04503ad306b7217280c34 P 3e772c31b50d4c6693644565ef92de04 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:19:15.715919 16711 raft_consensus.cc:2468] T 0086bd356ad04503ad306b7217280c34 P 49598ea210614f1fbb942c1fb77c7932 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49e15cd7009e4d6198f9d9352a27e9d1 in term 1.
I20260709 12:19:15.716900 16829 consensus_queue.cc:260] T 927ab821d939440ab1def38a7dbd0358 P 49e15cd7009e4d6198f9d9352a27e9d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } } peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } }
I20260709 12:19:15.720757 16600 leader_election.cc:304] T 0086bd356ad04503ad306b7217280c34 P 49e15cd7009e4d6198f9d9352a27e9d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49598ea210614f1fbb942c1fb77c7932, 49e15cd7009e4d6198f9d9352a27e9d1; no voters: 
I20260709 12:19:15.722471 16858 raft_consensus.cc:2804] T 0086bd356ad04503ad306b7217280c34 P 49e15cd7009e4d6198f9d9352a27e9d1 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:19:15.723114 16858 raft_consensus.cc:697] T 0086bd356ad04503ad306b7217280c34 P 49e15cd7009e4d6198f9d9352a27e9d1 [term 1 LEADER]: Becoming Leader. State: Replica: 49e15cd7009e4d6198f9d9352a27e9d1, State: Running, Role: LEADER
I20260709 12:19:15.724237 16858 consensus_queue.cc:237] T 0086bd356ad04503ad306b7217280c34 P 49e15cd7009e4d6198f9d9352a27e9d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } } peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } }
I20260709 12:19:15.734259 16829 ts_tablet_manager.cc:1434] T 927ab821d939440ab1def38a7dbd0358 P 49e15cd7009e4d6198f9d9352a27e9d1: Time spent starting tablet: real 0.041s	user 0.006s	sys 0.002s
I20260709 12:19:15.740926 16829 tablet_bootstrap.cc:492] T 95f027ce38784f81bf5c3b35b3357173 P 49e15cd7009e4d6198f9d9352a27e9d1: Bootstrap starting.
I20260709 12:19:15.736533 16861 leader_election.cc:290] T 0086bd356ad04503ad306b7217280c34 P 49e15cd7009e4d6198f9d9352a27e9d1 [CANDIDATE]: Term 1 election: Requested vote from peers 49598ea210614f1fbb942c1fb77c7932 (127.16.29.66:35871), 3e772c31b50d4c6693644565ef92de04 (127.16.29.67:35049)
I20260709 12:19:15.737552 16828 tablet_bootstrap.cc:492] T 3692a4a710ef469aa412d9870427fb63 P 3e772c31b50d4c6693644565ef92de04: No bootstrap required, opened a new log
I20260709 12:19:15.739636 16784 raft_consensus.cc:2468] T 0086bd356ad04503ad306b7217280c34 P 3e772c31b50d4c6693644565ef92de04 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49e15cd7009e4d6198f9d9352a27e9d1 in term 1.
I20260709 12:19:15.740072 16830 tablet_bootstrap.cc:492] T af674c5ffb4e44dab6f15ab84dbeb737 P 49598ea210614f1fbb942c1fb77c7932: No bootstrap required, opened a new log
I20260709 12:19:15.744364 16828 ts_tablet_manager.cc:1403] T 3692a4a710ef469aa412d9870427fb63 P 3e772c31b50d4c6693644565ef92de04: Time spent bootstrapping tablet: real 0.090s	user 0.022s	sys 0.019s
I20260709 12:19:15.745466 16830 ts_tablet_manager.cc:1403] T af674c5ffb4e44dab6f15ab84dbeb737 P 49598ea210614f1fbb942c1fb77c7932: Time spent bootstrapping tablet: real 0.174s	user 0.017s	sys 0.012s
I20260709 12:19:15.747615 16828 raft_consensus.cc:359] T 3692a4a710ef469aa412d9870427fb63 P 3e772c31b50d4c6693644565ef92de04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } } peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } }
I20260709 12:19:15.748410 16828 raft_consensus.cc:385] T 3692a4a710ef469aa412d9870427fb63 P 3e772c31b50d4c6693644565ef92de04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:19:15.748157 16830 raft_consensus.cc:359] T af674c5ffb4e44dab6f15ab84dbeb737 P 49598ea210614f1fbb942c1fb77c7932 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } } peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } }
I20260709 12:19:15.748694 16828 raft_consensus.cc:740] T 3692a4a710ef469aa412d9870427fb63 P 3e772c31b50d4c6693644565ef92de04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e772c31b50d4c6693644565ef92de04, State: Initialized, Role: FOLLOWER
I20260709 12:19:15.749964 16828 consensus_queue.cc:260] T 3692a4a710ef469aa412d9870427fb63 P 3e772c31b50d4c6693644565ef92de04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } } peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } }
I20260709 12:19:15.752720 16829 tablet_bootstrap.cc:654] T 95f027ce38784f81bf5c3b35b3357173 P 49e15cd7009e4d6198f9d9352a27e9d1: Neither blocks nor log segments found. Creating new log.
I20260709 12:19:15.752846 16828 ts_tablet_manager.cc:1434] T 3692a4a710ef469aa412d9870427fb63 P 3e772c31b50d4c6693644565ef92de04: Time spent starting tablet: real 0.008s	user 0.005s	sys 0.001s
I20260709 12:19:15.753989 16828 tablet_bootstrap.cc:492] T 95f027ce38784f81bf5c3b35b3357173 P 3e772c31b50d4c6693644565ef92de04: Bootstrap starting.
I20260709 12:19:15.748883 16830 raft_consensus.cc:385] T af674c5ffb4e44dab6f15ab84dbeb737 P 49598ea210614f1fbb942c1fb77c7932 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:19:15.755996 16830 raft_consensus.cc:740] T af674c5ffb4e44dab6f15ab84dbeb737 P 49598ea210614f1fbb942c1fb77c7932 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49598ea210614f1fbb942c1fb77c7932, State: Initialized, Role: FOLLOWER
I20260709 12:19:15.756912 16830 consensus_queue.cc:260] T af674c5ffb4e44dab6f15ab84dbeb737 P 49598ea210614f1fbb942c1fb77c7932 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } } peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } }
I20260709 12:19:15.760159 16830 ts_tablet_manager.cc:1434] T af674c5ffb4e44dab6f15ab84dbeb737 P 49598ea210614f1fbb942c1fb77c7932: Time spent starting tablet: real 0.014s	user 0.004s	sys 0.003s
I20260709 12:19:15.761168 16830 tablet_bootstrap.cc:492] T 95f027ce38784f81bf5c3b35b3357173 P 49598ea210614f1fbb942c1fb77c7932: Bootstrap starting.
I20260709 12:19:15.761953 16828 tablet_bootstrap.cc:654] T 95f027ce38784f81bf5c3b35b3357173 P 3e772c31b50d4c6693644565ef92de04: Neither blocks nor log segments found. Creating new log.
I20260709 12:19:15.764405 16835 raft_consensus.cc:493] T 3692a4a710ef469aa412d9870427fb63 P 3e772c31b50d4c6693644565ef92de04 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:15.764966 16835 raft_consensus.cc:515] T 3692a4a710ef469aa412d9870427fb63 P 3e772c31b50d4c6693644565ef92de04 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } } peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } }
I20260709 12:19:15.767259 16835 leader_election.cc:290] T 3692a4a710ef469aa412d9870427fb63 P 3e772c31b50d4c6693644565ef92de04 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49e15cd7009e4d6198f9d9352a27e9d1 (127.16.29.65:44833), 49598ea210614f1fbb942c1fb77c7932 (127.16.29.66:35871)
I20260709 12:19:15.773370 16635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3692a4a710ef469aa412d9870427fb63" candidate_uuid: "3e772c31b50d4c6693644565ef92de04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" is_pre_election: true
I20260709 12:19:15.774391 16830 tablet_bootstrap.cc:654] T 95f027ce38784f81bf5c3b35b3357173 P 49598ea210614f1fbb942c1fb77c7932: Neither blocks nor log segments found. Creating new log.
W20260709 12:19:15.768790 16581 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:19:15.782147 16711 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3692a4a710ef469aa412d9870427fb63" candidate_uuid: "3e772c31b50d4c6693644565ef92de04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49598ea210614f1fbb942c1fb77c7932" is_pre_election: true
I20260709 12:19:15.783267 16711 raft_consensus.cc:2393] T 3692a4a710ef469aa412d9870427fb63 P 49598ea210614f1fbb942c1fb77c7932 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3e772c31b50d4c6693644565ef92de04 in current term 1: Already voted for candidate 49e15cd7009e4d6198f9d9352a27e9d1 in this term.
I20260709 12:19:15.785669 16749 leader_election.cc:304] T 3692a4a710ef469aa412d9870427fb63 P 3e772c31b50d4c6693644565ef92de04 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3e772c31b50d4c6693644565ef92de04; no voters: 49598ea210614f1fbb942c1fb77c7932, 49e15cd7009e4d6198f9d9352a27e9d1
I20260709 12:19:15.785256 16532 catalog_manager.cc:5697] T 0edbf034be4a41d79cbce7d0c876132d P 3e772c31b50d4c6693644565ef92de04 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3e772c31b50d4c6693644565ef92de04 (127.16.29.67). New cstate: current_term: 1 leader_uuid: "3e772c31b50d4c6693644565ef92de04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } health_report { overall_health: UNKNOWN } } }
I20260709 12:19:15.789938 16835 raft_consensus.cc:3060] T 3692a4a710ef469aa412d9870427fb63 P 3e772c31b50d4c6693644565ef92de04 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:19:15.806983 16829 tablet_bootstrap.cc:492] T 95f027ce38784f81bf5c3b35b3357173 P 49e15cd7009e4d6198f9d9352a27e9d1: No bootstrap required, opened a new log
I20260709 12:19:15.811198 16829 ts_tablet_manager.cc:1403] T 95f027ce38784f81bf5c3b35b3357173 P 49e15cd7009e4d6198f9d9352a27e9d1: Time spent bootstrapping tablet: real 0.070s	user 0.009s	sys 0.022s
I20260709 12:19:15.810725 16533 catalog_manager.cc:5697] T 3692a4a710ef469aa412d9870427fb63 P 49e15cd7009e4d6198f9d9352a27e9d1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 49e15cd7009e4d6198f9d9352a27e9d1 (127.16.29.65). New cstate: current_term: 1 leader_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } health_report { overall_health: UNKNOWN } } }
I20260709 12:19:15.812983 16533 catalog_manager.cc:5697] T 0086bd356ad04503ad306b7217280c34 P 49e15cd7009e4d6198f9d9352a27e9d1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 49e15cd7009e4d6198f9d9352a27e9d1 (127.16.29.65). New cstate: current_term: 1 leader_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } health_report { overall_health: UNKNOWN } } }
I20260709 12:19:15.806938 16835 raft_consensus.cc:2749] T 3692a4a710ef469aa412d9870427fb63 P 3e772c31b50d4c6693644565ef92de04 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:19:15.818856 16828 tablet_bootstrap.cc:492] T 95f027ce38784f81bf5c3b35b3357173 P 3e772c31b50d4c6693644565ef92de04: No bootstrap required, opened a new log
I20260709 12:19:15.819416 16828 ts_tablet_manager.cc:1403] T 95f027ce38784f81bf5c3b35b3357173 P 3e772c31b50d4c6693644565ef92de04: Time spent bootstrapping tablet: real 0.066s	user 0.010s	sys 0.013s
I20260709 12:19:15.821273 16830 tablet_bootstrap.cc:492] T 95f027ce38784f81bf5c3b35b3357173 P 49598ea210614f1fbb942c1fb77c7932: No bootstrap required, opened a new log
I20260709 12:19:15.821946 16830 ts_tablet_manager.cc:1403] T 95f027ce38784f81bf5c3b35b3357173 P 49598ea210614f1fbb942c1fb77c7932: Time spent bootstrapping tablet: real 0.061s	user 0.024s	sys 0.001s
I20260709 12:19:15.823700 16829 raft_consensus.cc:359] T 95f027ce38784f81bf5c3b35b3357173 P 49e15cd7009e4d6198f9d9352a27e9d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } } peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } }
I20260709 12:19:15.825002 16829 raft_consensus.cc:385] T 95f027ce38784f81bf5c3b35b3357173 P 49e15cd7009e4d6198f9d9352a27e9d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:19:15.825414 16830 raft_consensus.cc:359] T 95f027ce38784f81bf5c3b35b3357173 P 49598ea210614f1fbb942c1fb77c7932 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } } peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } }
I20260709 12:19:15.826078 16830 raft_consensus.cc:385] T 95f027ce38784f81bf5c3b35b3357173 P 49598ea210614f1fbb942c1fb77c7932 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:19:15.825439 16829 raft_consensus.cc:740] T 95f027ce38784f81bf5c3b35b3357173 P 49e15cd7009e4d6198f9d9352a27e9d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49e15cd7009e4d6198f9d9352a27e9d1, State: Initialized, Role: FOLLOWER
I20260709 12:19:15.826452 16830 raft_consensus.cc:740] T 95f027ce38784f81bf5c3b35b3357173 P 49598ea210614f1fbb942c1fb77c7932 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49598ea210614f1fbb942c1fb77c7932, State: Initialized, Role: FOLLOWER
I20260709 12:19:15.826963 16829 consensus_queue.cc:260] T 95f027ce38784f81bf5c3b35b3357173 P 49e15cd7009e4d6198f9d9352a27e9d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } } peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } }
I20260709 12:19:15.827219 16830 consensus_queue.cc:260] T 95f027ce38784f81bf5c3b35b3357173 P 49598ea210614f1fbb942c1fb77c7932 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } } peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } }
I20260709 12:19:15.829681 16830 ts_tablet_manager.cc:1434] T 95f027ce38784f81bf5c3b35b3357173 P 49598ea210614f1fbb942c1fb77c7932: Time spent starting tablet: real 0.007s	user 0.008s	sys 0.000s
I20260709 12:19:15.830554 16830 tablet_bootstrap.cc:492] T 5c3e9fafdd8142eea33e771756209147 P 49598ea210614f1fbb942c1fb77c7932: Bootstrap starting.
I20260709 12:19:15.834497 16829 ts_tablet_manager.cc:1434] T 95f027ce38784f81bf5c3b35b3357173 P 49e15cd7009e4d6198f9d9352a27e9d1: Time spent starting tablet: real 0.023s	user 0.008s	sys 0.000s
I20260709 12:19:15.835491 16829 tablet_bootstrap.cc:492] T af674c5ffb4e44dab6f15ab84dbeb737 P 49e15cd7009e4d6198f9d9352a27e9d1: Bootstrap starting.
I20260709 12:19:15.835215 16828 raft_consensus.cc:359] T 95f027ce38784f81bf5c3b35b3357173 P 3e772c31b50d4c6693644565ef92de04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } } peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } }
I20260709 12:19:15.836268 16828 raft_consensus.cc:385] T 95f027ce38784f81bf5c3b35b3357173 P 3e772c31b50d4c6693644565ef92de04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:19:15.836824 16830 tablet_bootstrap.cc:654] T 5c3e9fafdd8142eea33e771756209147 P 49598ea210614f1fbb942c1fb77c7932: Neither blocks nor log segments found. Creating new log.
I20260709 12:19:15.836697 16828 raft_consensus.cc:740] T 95f027ce38784f81bf5c3b35b3357173 P 3e772c31b50d4c6693644565ef92de04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e772c31b50d4c6693644565ef92de04, State: Initialized, Role: FOLLOWER
I20260709 12:19:15.837751 16828 consensus_queue.cc:260] T 95f027ce38784f81bf5c3b35b3357173 P 3e772c31b50d4c6693644565ef92de04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } } peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } }
I20260709 12:19:15.841100 16828 ts_tablet_manager.cc:1434] T 95f027ce38784f81bf5c3b35b3357173 P 3e772c31b50d4c6693644565ef92de04: Time spent starting tablet: real 0.021s	user 0.009s	sys 0.000s
I20260709 12:19:15.842427 16828 tablet_bootstrap.cc:492] T af674c5ffb4e44dab6f15ab84dbeb737 P 3e772c31b50d4c6693644565ef92de04: Bootstrap starting.
I20260709 12:19:15.842898 16829 tablet_bootstrap.cc:654] T af674c5ffb4e44dab6f15ab84dbeb737 P 49e15cd7009e4d6198f9d9352a27e9d1: Neither blocks nor log segments found. Creating new log.
I20260709 12:19:15.846114 16840 raft_consensus.cc:493] T af674c5ffb4e44dab6f15ab84dbeb737 P 49598ea210614f1fbb942c1fb77c7932 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:15.846935 16840 raft_consensus.cc:515] T af674c5ffb4e44dab6f15ab84dbeb737 P 49598ea210614f1fbb942c1fb77c7932 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } } peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } }
I20260709 12:19:15.849516 16840 leader_election.cc:290] T af674c5ffb4e44dab6f15ab84dbeb737 P 49598ea210614f1fbb942c1fb77c7932 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49e15cd7009e4d6198f9d9352a27e9d1 (127.16.29.65:44833), 3e772c31b50d4c6693644565ef92de04 (127.16.29.67:35049)
I20260709 12:19:15.853307 16784 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af674c5ffb4e44dab6f15ab84dbeb737" candidate_uuid: "49598ea210614f1fbb942c1fb77c7932" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e772c31b50d4c6693644565ef92de04" is_pre_election: true
I20260709 12:19:15.858696 16635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af674c5ffb4e44dab6f15ab84dbeb737" candidate_uuid: "49598ea210614f1fbb942c1fb77c7932" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" is_pre_election: true
W20260709 12:19:15.860271 16674 leader_election.cc:343] T af674c5ffb4e44dab6f15ab84dbeb737 P 49598ea210614f1fbb942c1fb77c7932 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 49e15cd7009e4d6198f9d9352a27e9d1 (127.16.29.65:44833): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:19:15.861657 16674 leader_election.cc:343] T af674c5ffb4e44dab6f15ab84dbeb737 P 49598ea210614f1fbb942c1fb77c7932 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3e772c31b50d4c6693644565ef92de04 (127.16.29.67:35049): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:19:15.862268 16674 leader_election.cc:304] T af674c5ffb4e44dab6f15ab84dbeb737 P 49598ea210614f1fbb942c1fb77c7932 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 49598ea210614f1fbb942c1fb77c7932; no voters: 3e772c31b50d4c6693644565ef92de04, 49e15cd7009e4d6198f9d9352a27e9d1
I20260709 12:19:15.861588 16828 tablet_bootstrap.cc:654] T af674c5ffb4e44dab6f15ab84dbeb737 P 3e772c31b50d4c6693644565ef92de04: Neither blocks nor log segments found. Creating new log.
I20260709 12:19:15.868199 16840 raft_consensus.cc:2749] T af674c5ffb4e44dab6f15ab84dbeb737 P 49598ea210614f1fbb942c1fb77c7932 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:19:15.875587 16830 tablet_bootstrap.cc:492] T 5c3e9fafdd8142eea33e771756209147 P 49598ea210614f1fbb942c1fb77c7932: No bootstrap required, opened a new log
I20260709 12:19:15.876102 16830 ts_tablet_manager.cc:1403] T 5c3e9fafdd8142eea33e771756209147 P 49598ea210614f1fbb942c1fb77c7932: Time spent bootstrapping tablet: real 0.046s	user 0.011s	sys 0.006s
I20260709 12:19:15.876663 16829 tablet_bootstrap.cc:492] T af674c5ffb4e44dab6f15ab84dbeb737 P 49e15cd7009e4d6198f9d9352a27e9d1: No bootstrap required, opened a new log
I20260709 12:19:15.877081 16829 ts_tablet_manager.cc:1403] T af674c5ffb4e44dab6f15ab84dbeb737 P 49e15cd7009e4d6198f9d9352a27e9d1: Time spent bootstrapping tablet: real 0.042s	user 0.015s	sys 0.017s
I20260709 12:19:15.877342 16828 tablet_bootstrap.cc:492] T af674c5ffb4e44dab6f15ab84dbeb737 P 3e772c31b50d4c6693644565ef92de04: No bootstrap required, opened a new log
I20260709 12:19:15.877866 16828 ts_tablet_manager.cc:1403] T af674c5ffb4e44dab6f15ab84dbeb737 P 3e772c31b50d4c6693644565ef92de04: Time spent bootstrapping tablet: real 0.036s	user 0.012s	sys 0.004s
I20260709 12:19:15.878645 16830 raft_consensus.cc:359] T 5c3e9fafdd8142eea33e771756209147 P 49598ea210614f1fbb942c1fb77c7932 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } } peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } }
I20260709 12:19:15.879307 16830 raft_consensus.cc:385] T 5c3e9fafdd8142eea33e771756209147 P 49598ea210614f1fbb942c1fb77c7932 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:19:15.879644 16830 raft_consensus.cc:740] T 5c3e9fafdd8142eea33e771756209147 P 49598ea210614f1fbb942c1fb77c7932 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49598ea210614f1fbb942c1fb77c7932, State: Initialized, Role: FOLLOWER
I20260709 12:19:15.879376 16829 raft_consensus.cc:359] T af674c5ffb4e44dab6f15ab84dbeb737 P 49e15cd7009e4d6198f9d9352a27e9d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } } peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } }
I20260709 12:19:15.880070 16829 raft_consensus.cc:385] T af674c5ffb4e44dab6f15ab84dbeb737 P 49e15cd7009e4d6198f9d9352a27e9d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:19:15.880354 16829 raft_consensus.cc:740] T af674c5ffb4e44dab6f15ab84dbeb737 P 49e15cd7009e4d6198f9d9352a27e9d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49e15cd7009e4d6198f9d9352a27e9d1, State: Initialized, Role: FOLLOWER
I20260709 12:19:15.880498 16830 consensus_queue.cc:260] T 5c3e9fafdd8142eea33e771756209147 P 49598ea210614f1fbb942c1fb77c7932 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } } peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } }
I20260709 12:19:15.881012 16829 consensus_queue.cc:260] T af674c5ffb4e44dab6f15ab84dbeb737 P 49e15cd7009e4d6198f9d9352a27e9d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } } peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } }
I20260709 12:19:15.881202 16828 raft_consensus.cc:359] T af674c5ffb4e44dab6f15ab84dbeb737 P 3e772c31b50d4c6693644565ef92de04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } } peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } }
I20260709 12:19:15.882091 16828 raft_consensus.cc:385] T af674c5ffb4e44dab6f15ab84dbeb737 P 3e772c31b50d4c6693644565ef92de04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:19:15.882452 16828 raft_consensus.cc:740] T af674c5ffb4e44dab6f15ab84dbeb737 P 3e772c31b50d4c6693644565ef92de04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e772c31b50d4c6693644565ef92de04, State: Initialized, Role: FOLLOWER
I20260709 12:19:15.883404 16828 consensus_queue.cc:260] T af674c5ffb4e44dab6f15ab84dbeb737 P 3e772c31b50d4c6693644565ef92de04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } } peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } }
I20260709 12:19:15.887053 16829 ts_tablet_manager.cc:1434] T af674c5ffb4e44dab6f15ab84dbeb737 P 49e15cd7009e4d6198f9d9352a27e9d1: Time spent starting tablet: real 0.010s	user 0.006s	sys 0.000s
I20260709 12:19:15.888206 16829 tablet_bootstrap.cc:492] T 5c3e9fafdd8142eea33e771756209147 P 49e15cd7009e4d6198f9d9352a27e9d1: Bootstrap starting.
I20260709 12:19:15.891799 16828 ts_tablet_manager.cc:1434] T af674c5ffb4e44dab6f15ab84dbeb737 P 3e772c31b50d4c6693644565ef92de04: Time spent starting tablet: real 0.014s	user 0.006s	sys 0.000s
I20260709 12:19:15.892840 16828 tablet_bootstrap.cc:492] T 5c3e9fafdd8142eea33e771756209147 P 3e772c31b50d4c6693644565ef92de04: Bootstrap starting.
I20260709 12:19:15.898788 16829 tablet_bootstrap.cc:654] T 5c3e9fafdd8142eea33e771756209147 P 49e15cd7009e4d6198f9d9352a27e9d1: Neither blocks nor log segments found. Creating new log.
I20260709 12:19:15.900861 16840 raft_consensus.cc:493] T 5c3e9fafdd8142eea33e771756209147 P 49598ea210614f1fbb942c1fb77c7932 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:15.901383 16840 raft_consensus.cc:515] T 5c3e9fafdd8142eea33e771756209147 P 49598ea210614f1fbb942c1fb77c7932 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } } peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } }
I20260709 12:19:15.903457 16828 tablet_bootstrap.cc:654] T 5c3e9fafdd8142eea33e771756209147 P 3e772c31b50d4c6693644565ef92de04: Neither blocks nor log segments found. Creating new log.
I20260709 12:19:15.905465 16635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c3e9fafdd8142eea33e771756209147" candidate_uuid: "49598ea210614f1fbb942c1fb77c7932" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" is_pre_election: true
I20260709 12:19:15.906070 16861 raft_consensus.cc:493] T 927ab821d939440ab1def38a7dbd0358 P 49e15cd7009e4d6198f9d9352a27e9d1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:15.906586 16861 raft_consensus.cc:515] T 927ab821d939440ab1def38a7dbd0358 P 49e15cd7009e4d6198f9d9352a27e9d1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } } peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } }
W20260709 12:19:15.907644 16674 leader_election.cc:343] T 5c3e9fafdd8142eea33e771756209147 P 49598ea210614f1fbb942c1fb77c7932 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 49e15cd7009e4d6198f9d9352a27e9d1 (127.16.29.65:44833): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:19:15.908845 16840 leader_election.cc:290] T 5c3e9fafdd8142eea33e771756209147 P 49598ea210614f1fbb942c1fb77c7932 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49e15cd7009e4d6198f9d9352a27e9d1 (127.16.29.65:44833), 3e772c31b50d4c6693644565ef92de04 (127.16.29.67:35049)
I20260709 12:19:15.910409 16784 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c3e9fafdd8142eea33e771756209147" candidate_uuid: "49598ea210614f1fbb942c1fb77c7932" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e772c31b50d4c6693644565ef92de04" is_pre_election: true
W20260709 12:19:15.911855 16674 leader_election.cc:343] T 5c3e9fafdd8142eea33e771756209147 P 49598ea210614f1fbb942c1fb77c7932 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3e772c31b50d4c6693644565ef92de04 (127.16.29.67:35049): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:19:15.912312 16674 leader_election.cc:304] T 5c3e9fafdd8142eea33e771756209147 P 49598ea210614f1fbb942c1fb77c7932 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 49598ea210614f1fbb942c1fb77c7932; no voters: 3e772c31b50d4c6693644565ef92de04, 49e15cd7009e4d6198f9d9352a27e9d1
I20260709 12:19:15.913352 16840 raft_consensus.cc:2749] T 5c3e9fafdd8142eea33e771756209147 P 49598ea210614f1fbb942c1fb77c7932 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:19:15.913439 16711 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "927ab821d939440ab1def38a7dbd0358" candidate_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49598ea210614f1fbb942c1fb77c7932" is_pre_election: true
I20260709 12:19:15.915361 16830 ts_tablet_manager.cc:1434] T 5c3e9fafdd8142eea33e771756209147 P 49598ea210614f1fbb942c1fb77c7932: Time spent starting tablet: real 0.039s	user 0.006s	sys 0.000s
I20260709 12:19:15.915278 16784 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "927ab821d939440ab1def38a7dbd0358" candidate_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e772c31b50d4c6693644565ef92de04" is_pre_election: true
I20260709 12:19:15.916172 16784 raft_consensus.cc:2393] T 927ab821d939440ab1def38a7dbd0358 P 3e772c31b50d4c6693644565ef92de04 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 49e15cd7009e4d6198f9d9352a27e9d1 in current term 1: Already voted for candidate 49598ea210614f1fbb942c1fb77c7932 in this term.
I20260709 12:19:15.917614 16599 leader_election.cc:304] T 927ab821d939440ab1def38a7dbd0358 P 49e15cd7009e4d6198f9d9352a27e9d1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 49e15cd7009e4d6198f9d9352a27e9d1; no voters: 3e772c31b50d4c6693644565ef92de04, 49598ea210614f1fbb942c1fb77c7932
I20260709 12:19:15.918596 16840 raft_consensus.cc:493] T dd736bd544c746238c77320b4a5706e0 P 49598ea210614f1fbb942c1fb77c7932 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:15.919026 16858 raft_consensus.cc:3060] T 927ab821d939440ab1def38a7dbd0358 P 49e15cd7009e4d6198f9d9352a27e9d1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:19:15.919129 16840 raft_consensus.cc:515] T dd736bd544c746238c77320b4a5706e0 P 49598ea210614f1fbb942c1fb77c7932 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } } peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } }
I20260709 12:19:15.920178 16828 tablet_bootstrap.cc:492] T 5c3e9fafdd8142eea33e771756209147 P 3e772c31b50d4c6693644565ef92de04: No bootstrap required, opened a new log
I20260709 12:19:15.920702 16828 ts_tablet_manager.cc:1403] T 5c3e9fafdd8142eea33e771756209147 P 3e772c31b50d4c6693644565ef92de04: Time spent bootstrapping tablet: real 0.028s	user 0.015s	sys 0.006s
I20260709 12:19:15.921339 16861 leader_election.cc:290] T 927ab821d939440ab1def38a7dbd0358 P 49e15cd7009e4d6198f9d9352a27e9d1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49598ea210614f1fbb942c1fb77c7932 (127.16.29.66:35871), 3e772c31b50d4c6693644565ef92de04 (127.16.29.67:35049)
I20260709 12:19:15.923090 16840 leader_election.cc:290] T dd736bd544c746238c77320b4a5706e0 P 49598ea210614f1fbb942c1fb77c7932 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49e15cd7009e4d6198f9d9352a27e9d1 (127.16.29.65:44833), 3e772c31b50d4c6693644565ef92de04 (127.16.29.67:35049)
I20260709 12:19:15.924046 16635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd736bd544c746238c77320b4a5706e0" candidate_uuid: "49598ea210614f1fbb942c1fb77c7932" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" is_pre_election: true
I20260709 12:19:15.924917 16635 raft_consensus.cc:2393] T dd736bd544c746238c77320b4a5706e0 P 49e15cd7009e4d6198f9d9352a27e9d1 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 49598ea210614f1fbb942c1fb77c7932 in current term 1: Already voted for candidate 3e772c31b50d4c6693644565ef92de04 in this term.
I20260709 12:19:15.926920 16784 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd736bd544c746238c77320b4a5706e0" candidate_uuid: "49598ea210614f1fbb942c1fb77c7932" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e772c31b50d4c6693644565ef92de04" is_pre_election: true
I20260709 12:19:15.928560 16674 leader_election.cc:304] T dd736bd544c746238c77320b4a5706e0 P 49598ea210614f1fbb942c1fb77c7932 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 49598ea210614f1fbb942c1fb77c7932; no voters: 3e772c31b50d4c6693644565ef92de04, 49e15cd7009e4d6198f9d9352a27e9d1
I20260709 12:19:15.930022 16840 raft_consensus.cc:3060] T dd736bd544c746238c77320b4a5706e0 P 49598ea210614f1fbb942c1fb77c7932 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:19:15.933192 16858 raft_consensus.cc:2749] T 927ab821d939440ab1def38a7dbd0358 P 49e15cd7009e4d6198f9d9352a27e9d1 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:19:15.934393 16828 raft_consensus.cc:359] T 5c3e9fafdd8142eea33e771756209147 P 3e772c31b50d4c6693644565ef92de04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } } peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } }
I20260709 12:19:15.935087 16828 raft_consensus.cc:385] T 5c3e9fafdd8142eea33e771756209147 P 3e772c31b50d4c6693644565ef92de04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:19:15.935356 16828 raft_consensus.cc:740] T 5c3e9fafdd8142eea33e771756209147 P 3e772c31b50d4c6693644565ef92de04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e772c31b50d4c6693644565ef92de04, State: Initialized, Role: FOLLOWER
I20260709 12:19:15.937525 16840 raft_consensus.cc:2749] T dd736bd544c746238c77320b4a5706e0 P 49598ea210614f1fbb942c1fb77c7932 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:19:15.937243 16828 consensus_queue.cc:260] T 5c3e9fafdd8142eea33e771756209147 P 3e772c31b50d4c6693644565ef92de04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } } peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } }
I20260709 12:19:15.938944 16829 tablet_bootstrap.cc:492] T 5c3e9fafdd8142eea33e771756209147 P 49e15cd7009e4d6198f9d9352a27e9d1: No bootstrap required, opened a new log
I20260709 12:19:15.939498 16829 ts_tablet_manager.cc:1403] T 5c3e9fafdd8142eea33e771756209147 P 49e15cd7009e4d6198f9d9352a27e9d1: Time spent bootstrapping tablet: real 0.051s	user 0.014s	sys 0.016s
I20260709 12:19:15.943127 16829 raft_consensus.cc:359] T 5c3e9fafdd8142eea33e771756209147 P 49e15cd7009e4d6198f9d9352a27e9d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } } peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } }
I20260709 12:19:15.944052 16829 raft_consensus.cc:385] T 5c3e9fafdd8142eea33e771756209147 P 49e15cd7009e4d6198f9d9352a27e9d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:19:15.944430 16829 raft_consensus.cc:740] T 5c3e9fafdd8142eea33e771756209147 P 49e15cd7009e4d6198f9d9352a27e9d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49e15cd7009e4d6198f9d9352a27e9d1, State: Initialized, Role: FOLLOWER
I20260709 12:19:15.945245 16829 consensus_queue.cc:260] T 5c3e9fafdd8142eea33e771756209147 P 49e15cd7009e4d6198f9d9352a27e9d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } } peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } }
I20260709 12:19:15.945667 16848 consensus_queue.cc:1048] T dd736bd544c746238c77320b4a5706e0 P 3e772c31b50d4c6693644565ef92de04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:19:15.948426 16829 ts_tablet_manager.cc:1434] T 5c3e9fafdd8142eea33e771756209147 P 49e15cd7009e4d6198f9d9352a27e9d1: Time spent starting tablet: real 0.009s	user 0.006s	sys 0.001s
I20260709 12:19:15.951115 16848 consensus_queue.cc:1048] T dd736bd544c746238c77320b4a5706e0 P 3e772c31b50d4c6693644565ef92de04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20260709 12:19:15.953974 16835 consensus_queue.cc:1048] T 0edbf034be4a41d79cbce7d0c876132d P 3e772c31b50d4c6693644565ef92de04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.006s
I20260709 12:19:15.961192 16828 ts_tablet_manager.cc:1434] T 5c3e9fafdd8142eea33e771756209147 P 3e772c31b50d4c6693644565ef92de04: Time spent starting tablet: real 0.040s	user 0.004s	sys 0.002s
I20260709 12:19:16.001679 16887 consensus_queue.cc:1048] T 0edbf034be4a41d79cbce7d0c876132d P 3e772c31b50d4c6693644565ef92de04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:19:16.008832 16858 consensus_queue.cc:1048] T 3692a4a710ef469aa412d9870427fb63 P 49e15cd7009e4d6198f9d9352a27e9d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:19:16.031647 16858 raft_consensus.cc:493] T 95f027ce38784f81bf5c3b35b3357173 P 49e15cd7009e4d6198f9d9352a27e9d1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:16.032341 16858 raft_consensus.cc:515] T 95f027ce38784f81bf5c3b35b3357173 P 49e15cd7009e4d6198f9d9352a27e9d1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } } peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } }
I20260709 12:19:16.035969 16858 leader_election.cc:290] T 95f027ce38784f81bf5c3b35b3357173 P 49e15cd7009e4d6198f9d9352a27e9d1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49598ea210614f1fbb942c1fb77c7932 (127.16.29.66:35871), 3e772c31b50d4c6693644565ef92de04 (127.16.29.67:35049)
I20260709 12:19:16.036496 16858 raft_consensus.cc:493] T af674c5ffb4e44dab6f15ab84dbeb737 P 49e15cd7009e4d6198f9d9352a27e9d1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:16.036945 16858 raft_consensus.cc:515] T af674c5ffb4e44dab6f15ab84dbeb737 P 49e15cd7009e4d6198f9d9352a27e9d1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } } peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } }
I20260709 12:19:16.037460 16784 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f027ce38784f81bf5c3b35b3357173" candidate_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e772c31b50d4c6693644565ef92de04" is_pre_election: true
I20260709 12:19:16.038322 16784 raft_consensus.cc:2468] T 95f027ce38784f81bf5c3b35b3357173 P 3e772c31b50d4c6693644565ef92de04 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49e15cd7009e4d6198f9d9352a27e9d1 in term 0.
I20260709 12:19:16.035694 16838 consensus_queue.cc:1048] T 2ef5967933364223bef5315de3fa6bfe P 49e15cd7009e4d6198f9d9352a27e9d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:19:16.040959 16709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f027ce38784f81bf5c3b35b3357173" candidate_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49598ea210614f1fbb942c1fb77c7932" is_pre_election: true
I20260709 12:19:16.041697 16709 raft_consensus.cc:2468] T 95f027ce38784f81bf5c3b35b3357173 P 49598ea210614f1fbb942c1fb77c7932 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49e15cd7009e4d6198f9d9352a27e9d1 in term 0.
I20260709 12:19:16.045117 16600 leader_election.cc:304] T 95f027ce38784f81bf5c3b35b3357173 P 49e15cd7009e4d6198f9d9352a27e9d1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49598ea210614f1fbb942c1fb77c7932, 49e15cd7009e4d6198f9d9352a27e9d1; no voters: 
I20260709 12:19:16.053085 16838 raft_consensus.cc:2804] T 95f027ce38784f81bf5c3b35b3357173 P 49e15cd7009e4d6198f9d9352a27e9d1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:19:16.053519 16838 raft_consensus.cc:493] T 95f027ce38784f81bf5c3b35b3357173 P 49e15cd7009e4d6198f9d9352a27e9d1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:16.045389 16711 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af674c5ffb4e44dab6f15ab84dbeb737" candidate_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49598ea210614f1fbb942c1fb77c7932" is_pre_election: true
I20260709 12:19:16.056275 16783 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af674c5ffb4e44dab6f15ab84dbeb737" candidate_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e772c31b50d4c6693644565ef92de04" is_pre_election: true
I20260709 12:19:16.056931 16783 raft_consensus.cc:2468] T af674c5ffb4e44dab6f15ab84dbeb737 P 3e772c31b50d4c6693644565ef92de04 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49e15cd7009e4d6198f9d9352a27e9d1 in term 0.
I20260709 12:19:16.053880 16838 raft_consensus.cc:3060] T 95f027ce38784f81bf5c3b35b3357173 P 49e15cd7009e4d6198f9d9352a27e9d1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:19:16.049595 16858 leader_election.cc:290] T af674c5ffb4e44dab6f15ab84dbeb737 P 49e15cd7009e4d6198f9d9352a27e9d1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49598ea210614f1fbb942c1fb77c7932 (127.16.29.66:35871), 3e772c31b50d4c6693644565ef92de04 (127.16.29.67:35049)
I20260709 12:19:16.055871 16711 raft_consensus.cc:2468] T af674c5ffb4e44dab6f15ab84dbeb737 P 49598ea210614f1fbb942c1fb77c7932 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49e15cd7009e4d6198f9d9352a27e9d1 in term 0.
I20260709 12:19:16.060240 16599 leader_election.cc:304] T af674c5ffb4e44dab6f15ab84dbeb737 P 49e15cd7009e4d6198f9d9352a27e9d1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e772c31b50d4c6693644565ef92de04, 49e15cd7009e4d6198f9d9352a27e9d1; no voters: 
I20260709 12:19:16.059577 16861 consensus_queue.cc:1048] T 3692a4a710ef469aa412d9870427fb63 P 49e15cd7009e4d6198f9d9352a27e9d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.005s
I20260709 12:19:16.066906 16895 raft_consensus.cc:2804] T af674c5ffb4e44dab6f15ab84dbeb737 P 49e15cd7009e4d6198f9d9352a27e9d1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:19:16.067407 16895 raft_consensus.cc:493] T af674c5ffb4e44dab6f15ab84dbeb737 P 49e15cd7009e4d6198f9d9352a27e9d1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:16.068280 16895 raft_consensus.cc:3060] T af674c5ffb4e44dab6f15ab84dbeb737 P 49e15cd7009e4d6198f9d9352a27e9d1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:19:16.083118 16838 raft_consensus.cc:515] T 95f027ce38784f81bf5c3b35b3357173 P 49e15cd7009e4d6198f9d9352a27e9d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } } peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } }
I20260709 12:19:16.086858 16784 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f027ce38784f81bf5c3b35b3357173" candidate_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e772c31b50d4c6693644565ef92de04"
I20260709 12:19:16.087934 16784 raft_consensus.cc:3060] T 95f027ce38784f81bf5c3b35b3357173 P 3e772c31b50d4c6693644565ef92de04 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:19:16.093953 16838 leader_election.cc:290] T 95f027ce38784f81bf5c3b35b3357173 P 49e15cd7009e4d6198f9d9352a27e9d1 [CANDIDATE]: Term 1 election: Requested vote from peers 49598ea210614f1fbb942c1fb77c7932 (127.16.29.66:35871), 3e772c31b50d4c6693644565ef92de04 (127.16.29.67:35049)
I20260709 12:19:16.098042 16784 raft_consensus.cc:2468] T 95f027ce38784f81bf5c3b35b3357173 P 3e772c31b50d4c6693644565ef92de04 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49e15cd7009e4d6198f9d9352a27e9d1 in term 1.
I20260709 12:19:16.099344 16599 leader_election.cc:304] T 95f027ce38784f81bf5c3b35b3357173 P 49e15cd7009e4d6198f9d9352a27e9d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e772c31b50d4c6693644565ef92de04, 49e15cd7009e4d6198f9d9352a27e9d1; no voters: 
I20260709 12:19:16.100981 16838 raft_consensus.cc:2804] T 95f027ce38784f81bf5c3b35b3357173 P 49e15cd7009e4d6198f9d9352a27e9d1 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:19:16.102001 16711 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f027ce38784f81bf5c3b35b3357173" candidate_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49598ea210614f1fbb942c1fb77c7932"
I20260709 12:19:16.106676 16895 raft_consensus.cc:515] T af674c5ffb4e44dab6f15ab84dbeb737 P 49e15cd7009e4d6198f9d9352a27e9d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } } peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } }
I20260709 12:19:16.107488 16711 raft_consensus.cc:3060] T 95f027ce38784f81bf5c3b35b3357173 P 49598ea210614f1fbb942c1fb77c7932 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:19:16.110555 16709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af674c5ffb4e44dab6f15ab84dbeb737" candidate_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49598ea210614f1fbb942c1fb77c7932"
I20260709 12:19:16.111174 16709 raft_consensus.cc:3060] T af674c5ffb4e44dab6f15ab84dbeb737 P 49598ea210614f1fbb942c1fb77c7932 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:19:16.106127 16838 raft_consensus.cc:697] T 95f027ce38784f81bf5c3b35b3357173 P 49e15cd7009e4d6198f9d9352a27e9d1 [term 1 LEADER]: Becoming Leader. State: Replica: 49e15cd7009e4d6198f9d9352a27e9d1, State: Running, Role: LEADER
I20260709 12:19:16.118515 16895 leader_election.cc:290] T af674c5ffb4e44dab6f15ab84dbeb737 P 49e15cd7009e4d6198f9d9352a27e9d1 [CANDIDATE]: Term 1 election: Requested vote from peers 49598ea210614f1fbb942c1fb77c7932 (127.16.29.66:35871), 3e772c31b50d4c6693644565ef92de04 (127.16.29.67:35049)
I20260709 12:19:16.118826 16840 consensus_queue.cc:1048] T 927ab821d939440ab1def38a7dbd0358 P 49598ea210614f1fbb942c1fb77c7932 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:19:16.118288 16838 consensus_queue.cc:237] T 95f027ce38784f81bf5c3b35b3357173 P 49e15cd7009e4d6198f9d9352a27e9d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } } peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } }
I20260709 12:19:16.119993 16784 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af674c5ffb4e44dab6f15ab84dbeb737" candidate_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e772c31b50d4c6693644565ef92de04"
I20260709 12:19:16.121544 16784 raft_consensus.cc:3060] T af674c5ffb4e44dab6f15ab84dbeb737 P 3e772c31b50d4c6693644565ef92de04 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:19:16.131289 16784 raft_consensus.cc:2468] T af674c5ffb4e44dab6f15ab84dbeb737 P 3e772c31b50d4c6693644565ef92de04 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49e15cd7009e4d6198f9d9352a27e9d1 in term 1.
I20260709 12:19:16.133142 16599 leader_election.cc:304] T af674c5ffb4e44dab6f15ab84dbeb737 P 49e15cd7009e4d6198f9d9352a27e9d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e772c31b50d4c6693644565ef92de04, 49e15cd7009e4d6198f9d9352a27e9d1; no voters: 
I20260709 12:19:16.134910 16895 raft_consensus.cc:2804] T af674c5ffb4e44dab6f15ab84dbeb737 P 49e15cd7009e4d6198f9d9352a27e9d1 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:19:16.135653 16895 raft_consensus.cc:697] T af674c5ffb4e44dab6f15ab84dbeb737 P 49e15cd7009e4d6198f9d9352a27e9d1 [term 1 LEADER]: Becoming Leader. State: Replica: 49e15cd7009e4d6198f9d9352a27e9d1, State: Running, Role: LEADER
I20260709 12:19:16.136354 16711 raft_consensus.cc:2468] T 95f027ce38784f81bf5c3b35b3357173 P 49598ea210614f1fbb942c1fb77c7932 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49e15cd7009e4d6198f9d9352a27e9d1 in term 1.
I20260709 12:19:16.129248 16709 raft_consensus.cc:2468] T af674c5ffb4e44dab6f15ab84dbeb737 P 49598ea210614f1fbb942c1fb77c7932 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49e15cd7009e4d6198f9d9352a27e9d1 in term 1.
I20260709 12:19:16.138545 16858 consensus_queue.cc:1048] T 2ef5967933364223bef5315de3fa6bfe P 49e15cd7009e4d6198f9d9352a27e9d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:19:16.141127 16895 consensus_queue.cc:237] T af674c5ffb4e44dab6f15ab84dbeb737 P 49e15cd7009e4d6198f9d9352a27e9d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } } peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } }
I20260709 12:19:16.166043 16895 consensus_queue.cc:1048] T 0086bd356ad04503ad306b7217280c34 P 49e15cd7009e4d6198f9d9352a27e9d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:19:16.167052 16858 consensus_queue.cc:1048] T 0086bd356ad04503ad306b7217280c34 P 49e15cd7009e4d6198f9d9352a27e9d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.007s
I20260709 12:19:16.177372 16898 consensus_queue.cc:1048] T 927ab821d939440ab1def38a7dbd0358 P 49598ea210614f1fbb942c1fb77c7932 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:19:16.187525 16532 catalog_manager.cc:5697] T 95f027ce38784f81bf5c3b35b3357173 P 49e15cd7009e4d6198f9d9352a27e9d1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 49e15cd7009e4d6198f9d9352a27e9d1 (127.16.29.65). New cstate: current_term: 1 leader_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } health_report { overall_health: UNKNOWN } } }
I20260709 12:19:16.241999 16533 catalog_manager.cc:5697] T af674c5ffb4e44dab6f15ab84dbeb737 P 49e15cd7009e4d6198f9d9352a27e9d1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 49e15cd7009e4d6198f9d9352a27e9d1 (127.16.29.65). New cstate: current_term: 1 leader_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } health_report { overall_health: UNKNOWN } } }
I20260709 12:19:16.309152 16887 raft_consensus.cc:493] T 5c3e9fafdd8142eea33e771756209147 P 3e772c31b50d4c6693644565ef92de04 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:16.310012 16887 raft_consensus.cc:515] T 5c3e9fafdd8142eea33e771756209147 P 3e772c31b50d4c6693644565ef92de04 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } } peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } }
I20260709 12:19:16.313645 16887 leader_election.cc:290] T 5c3e9fafdd8142eea33e771756209147 P 3e772c31b50d4c6693644565ef92de04 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49e15cd7009e4d6198f9d9352a27e9d1 (127.16.29.65:44833), 49598ea210614f1fbb942c1fb77c7932 (127.16.29.66:35871)
I20260709 12:19:16.314124 16636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c3e9fafdd8142eea33e771756209147" candidate_uuid: "3e772c31b50d4c6693644565ef92de04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" is_pre_election: true
I20260709 12:19:16.314767 16709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c3e9fafdd8142eea33e771756209147" candidate_uuid: "3e772c31b50d4c6693644565ef92de04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49598ea210614f1fbb942c1fb77c7932" is_pre_election: true
I20260709 12:19:16.314908 16636 raft_consensus.cc:2468] T 5c3e9fafdd8142eea33e771756209147 P 49e15cd7009e4d6198f9d9352a27e9d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e772c31b50d4c6693644565ef92de04 in term 0.
I20260709 12:19:16.315459 16709 raft_consensus.cc:2468] T 5c3e9fafdd8142eea33e771756209147 P 49598ea210614f1fbb942c1fb77c7932 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e772c31b50d4c6693644565ef92de04 in term 0.
I20260709 12:19:16.316412 16748 leader_election.cc:304] T 5c3e9fafdd8142eea33e771756209147 P 3e772c31b50d4c6693644565ef92de04 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e772c31b50d4c6693644565ef92de04, 49e15cd7009e4d6198f9d9352a27e9d1; no voters: 
I20260709 12:19:16.317345 16887 raft_consensus.cc:2804] T 5c3e9fafdd8142eea33e771756209147 P 3e772c31b50d4c6693644565ef92de04 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:19:16.317812 16887 raft_consensus.cc:493] T 5c3e9fafdd8142eea33e771756209147 P 3e772c31b50d4c6693644565ef92de04 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:16.318364 16887 raft_consensus.cc:3060] T 5c3e9fafdd8142eea33e771756209147 P 3e772c31b50d4c6693644565ef92de04 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:19:16.328267 16887 raft_consensus.cc:515] T 5c3e9fafdd8142eea33e771756209147 P 3e772c31b50d4c6693644565ef92de04 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } } peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } }
I20260709 12:19:16.330909 16887 leader_election.cc:290] T 5c3e9fafdd8142eea33e771756209147 P 3e772c31b50d4c6693644565ef92de04 [CANDIDATE]: Term 1 election: Requested vote from peers 49e15cd7009e4d6198f9d9352a27e9d1 (127.16.29.65:44833), 49598ea210614f1fbb942c1fb77c7932 (127.16.29.66:35871)
I20260709 12:19:16.331635 16636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c3e9fafdd8142eea33e771756209147" candidate_uuid: "3e772c31b50d4c6693644565ef92de04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49e15cd7009e4d6198f9d9352a27e9d1"
I20260709 12:19:16.332051 16709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c3e9fafdd8142eea33e771756209147" candidate_uuid: "3e772c31b50d4c6693644565ef92de04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49598ea210614f1fbb942c1fb77c7932"
I20260709 12:19:16.332280 16636 raft_consensus.cc:3060] T 5c3e9fafdd8142eea33e771756209147 P 49e15cd7009e4d6198f9d9352a27e9d1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:19:16.332621 16709 raft_consensus.cc:3060] T 5c3e9fafdd8142eea33e771756209147 P 49598ea210614f1fbb942c1fb77c7932 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:19:16.339429 16709 raft_consensus.cc:2468] T 5c3e9fafdd8142eea33e771756209147 P 49598ea210614f1fbb942c1fb77c7932 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e772c31b50d4c6693644565ef92de04 in term 1.
I20260709 12:19:16.339433 16636 raft_consensus.cc:2468] T 5c3e9fafdd8142eea33e771756209147 P 49e15cd7009e4d6198f9d9352a27e9d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e772c31b50d4c6693644565ef92de04 in term 1.
I20260709 12:19:16.341006 16748 leader_election.cc:304] T 5c3e9fafdd8142eea33e771756209147 P 3e772c31b50d4c6693644565ef92de04 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e772c31b50d4c6693644565ef92de04, 49e15cd7009e4d6198f9d9352a27e9d1; no voters: 
I20260709 12:19:16.342324 16887 raft_consensus.cc:2804] T 5c3e9fafdd8142eea33e771756209147 P 3e772c31b50d4c6693644565ef92de04 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:19:16.342823 16887 raft_consensus.cc:697] T 5c3e9fafdd8142eea33e771756209147 P 3e772c31b50d4c6693644565ef92de04 [term 1 LEADER]: Becoming Leader. State: Replica: 3e772c31b50d4c6693644565ef92de04, State: Running, Role: LEADER
I20260709 12:19:16.343678 16887 consensus_queue.cc:237] T 5c3e9fafdd8142eea33e771756209147 P 3e772c31b50d4c6693644565ef92de04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } } peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } }
I20260709 12:19:16.352952 16531 catalog_manager.cc:5697] T 5c3e9fafdd8142eea33e771756209147 P 3e772c31b50d4c6693644565ef92de04 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3e772c31b50d4c6693644565ef92de04 (127.16.29.67). New cstate: current_term: 1 leader_uuid: "3e772c31b50d4c6693644565ef92de04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } health_report { overall_health: HEALTHY } } }
I20260709 12:19:16.452359 16785 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "0edbf034be4a41d79cbce7d0c876132d"
dest_uuid: "3e772c31b50d4c6693644565ef92de04"
mode: GRACEFUL
new_leader_uuid: "49e15cd7009e4d6198f9d9352a27e9d1"
 from {username='slave'} at 127.0.0.1:52756
I20260709 12:19:16.453001 16785 raft_consensus.cc:606] T 0edbf034be4a41d79cbce7d0c876132d P 3e772c31b50d4c6693644565ef92de04 [term 1 LEADER]: Received request to transfer leadership to TS 49e15cd7009e4d6198f9d9352a27e9d1
I20260709 12:19:16.472651 16709 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "927ab821d939440ab1def38a7dbd0358"
dest_uuid: "49598ea210614f1fbb942c1fb77c7932"
mode: GRACEFUL
new_leader_uuid: "49e15cd7009e4d6198f9d9352a27e9d1"
 from {username='slave'} at 127.0.0.1:33770
I20260709 12:19:16.473290 16709 raft_consensus.cc:606] T 927ab821d939440ab1def38a7dbd0358 P 49598ea210614f1fbb942c1fb77c7932 [term 1 LEADER]: Received request to transfer leadership to TS 49e15cd7009e4d6198f9d9352a27e9d1
I20260709 12:19:16.479609 16785 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "dd736bd544c746238c77320b4a5706e0"
dest_uuid: "3e772c31b50d4c6693644565ef92de04"
mode: GRACEFUL
new_leader_uuid: "49e15cd7009e4d6198f9d9352a27e9d1"
 from {username='slave'} at 127.0.0.1:52756
I20260709 12:19:16.480209 16785 raft_consensus.cc:606] T dd736bd544c746238c77320b4a5706e0 P 3e772c31b50d4c6693644565ef92de04 [term 1 LEADER]: Received request to transfer leadership to TS 49e15cd7009e4d6198f9d9352a27e9d1
I20260709 12:19:16.499809 16636 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "0086bd356ad04503ad306b7217280c34"
dest_uuid: "49e15cd7009e4d6198f9d9352a27e9d1"
mode: GRACEFUL
new_leader_uuid: "49598ea210614f1fbb942c1fb77c7932"
 from {username='slave'} at 127.0.0.1:34778
I20260709 12:19:16.500403 16636 raft_consensus.cc:606] T 0086bd356ad04503ad306b7217280c34 P 49e15cd7009e4d6198f9d9352a27e9d1 [term 1 LEADER]: Received request to transfer leadership to TS 49598ea210614f1fbb942c1fb77c7932
I20260709 12:19:16.506866 16636 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "3692a4a710ef469aa412d9870427fb63"
dest_uuid: "49e15cd7009e4d6198f9d9352a27e9d1"
mode: GRACEFUL
new_leader_uuid: "49598ea210614f1fbb942c1fb77c7932"
 from {username='slave'} at 127.0.0.1:34778
I20260709 12:19:16.507396 16636 raft_consensus.cc:606] T 3692a4a710ef469aa412d9870427fb63 P 49e15cd7009e4d6198f9d9352a27e9d1 [term 1 LEADER]: Received request to transfer leadership to TS 49598ea210614f1fbb942c1fb77c7932
I20260709 12:19:16.513036 16636 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "95f027ce38784f81bf5c3b35b3357173"
dest_uuid: "49e15cd7009e4d6198f9d9352a27e9d1"
mode: GRACEFUL
new_leader_uuid: "49598ea210614f1fbb942c1fb77c7932"
 from {username='slave'} at 127.0.0.1:34778
I20260709 12:19:16.513599 16636 raft_consensus.cc:606] T 95f027ce38784f81bf5c3b35b3357173 P 49e15cd7009e4d6198f9d9352a27e9d1 [term 1 LEADER]: Received request to transfer leadership to TS 49598ea210614f1fbb942c1fb77c7932
I20260709 12:19:16.519371 16636 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "af674c5ffb4e44dab6f15ab84dbeb737"
dest_uuid: "49e15cd7009e4d6198f9d9352a27e9d1"
mode: GRACEFUL
new_leader_uuid: "49598ea210614f1fbb942c1fb77c7932"
 from {username='slave'} at 127.0.0.1:34778
I20260709 12:19:16.519964 16636 raft_consensus.cc:606] T af674c5ffb4e44dab6f15ab84dbeb737 P 49e15cd7009e4d6198f9d9352a27e9d1 [term 1 LEADER]: Received request to transfer leadership to TS 49598ea210614f1fbb942c1fb77c7932
I20260709 12:19:16.580956 16835 raft_consensus.cc:993] T dd736bd544c746238c77320b4a5706e0 P 3e772c31b50d4c6693644565ef92de04: : Instructing follower 49e15cd7009e4d6198f9d9352a27e9d1 to start an election
I20260709 12:19:16.582566 16887 raft_consensus.cc:1081] T dd736bd544c746238c77320b4a5706e0 P 3e772c31b50d4c6693644565ef92de04 [term 1 LEADER]: Signalling peer 49e15cd7009e4d6198f9d9352a27e9d1 to start an election
I20260709 12:19:16.584833 16636 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "dd736bd544c746238c77320b4a5706e0"
dest_uuid: "49e15cd7009e4d6198f9d9352a27e9d1"
 from {username='slave'} at 127.0.0.1:34774
I20260709 12:19:16.585551 16636 raft_consensus.cc:493] T dd736bd544c746238c77320b4a5706e0 P 49e15cd7009e4d6198f9d9352a27e9d1 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:19:16.586323 16636 raft_consensus.cc:3060] T dd736bd544c746238c77320b4a5706e0 P 49e15cd7009e4d6198f9d9352a27e9d1 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:19:16.593552 16636 raft_consensus.cc:515] T dd736bd544c746238c77320b4a5706e0 P 49e15cd7009e4d6198f9d9352a27e9d1 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } } peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } }
I20260709 12:19:16.599462 16861 raft_consensus.cc:993] T 3692a4a710ef469aa412d9870427fb63 P 49e15cd7009e4d6198f9d9352a27e9d1: : Instructing follower 49598ea210614f1fbb942c1fb77c7932 to start an election
I20260709 12:19:16.600168 16895 raft_consensus.cc:1081] T 3692a4a710ef469aa412d9870427fb63 P 49e15cd7009e4d6198f9d9352a27e9d1 [term 1 LEADER]: Signalling peer 49598ea210614f1fbb942c1fb77c7932 to start an election
I20260709 12:19:16.602375 16785 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd736bd544c746238c77320b4a5706e0" candidate_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "3e772c31b50d4c6693644565ef92de04"
I20260709 12:19:16.602977 16711 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "3692a4a710ef469aa412d9870427fb63"
dest_uuid: "49598ea210614f1fbb942c1fb77c7932"
 from {username='slave'} at 127.0.0.1:33766
I20260709 12:19:16.603513 16636 leader_election.cc:290] T dd736bd544c746238c77320b4a5706e0 P 49e15cd7009e4d6198f9d9352a27e9d1 [CANDIDATE]: Term 2 election: Requested vote from peers 49598ea210614f1fbb942c1fb77c7932 (127.16.29.66:35871), 3e772c31b50d4c6693644565ef92de04 (127.16.29.67:35049)
I20260709 12:19:16.603749 16711 raft_consensus.cc:493] T 3692a4a710ef469aa412d9870427fb63 P 49598ea210614f1fbb942c1fb77c7932 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:19:16.603106 16785 raft_consensus.cc:3055] T dd736bd544c746238c77320b4a5706e0 P 3e772c31b50d4c6693644565ef92de04 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:19:16.604579 16785 raft_consensus.cc:740] T dd736bd544c746238c77320b4a5706e0 P 3e772c31b50d4c6693644565ef92de04 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3e772c31b50d4c6693644565ef92de04, State: Running, Role: LEADER
I20260709 12:19:16.605095 16709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd736bd544c746238c77320b4a5706e0" candidate_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "49598ea210614f1fbb942c1fb77c7932"
I20260709 12:19:16.605744 16709 raft_consensus.cc:3060] T dd736bd544c746238c77320b4a5706e0 P 49598ea210614f1fbb942c1fb77c7932 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:19:16.605652 16785 consensus_queue.cc:260] T dd736bd544c746238c77320b4a5706e0 P 3e772c31b50d4c6693644565ef92de04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } } peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } }
I20260709 12:19:16.604215 16711 raft_consensus.cc:3060] T 3692a4a710ef469aa412d9870427fb63 P 49598ea210614f1fbb942c1fb77c7932 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:19:16.607308 16785 raft_consensus.cc:3060] T dd736bd544c746238c77320b4a5706e0 P 3e772c31b50d4c6693644565ef92de04 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:19:16.614279 16861 consensus_queue.cc:1048] T af674c5ffb4e44dab6f15ab84dbeb737 P 49e15cd7009e4d6198f9d9352a27e9d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:19:16.617429 16785 raft_consensus.cc:2468] T dd736bd544c746238c77320b4a5706e0 P 3e772c31b50d4c6693644565ef92de04 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49e15cd7009e4d6198f9d9352a27e9d1 in term 2.
I20260709 12:19:16.621006 16887 raft_consensus.cc:993] T 0edbf034be4a41d79cbce7d0c876132d P 3e772c31b50d4c6693644565ef92de04: : Instructing follower 49e15cd7009e4d6198f9d9352a27e9d1 to start an election
I20260709 12:19:16.621572 16887 raft_consensus.cc:1081] T 0edbf034be4a41d79cbce7d0c876132d P 3e772c31b50d4c6693644565ef92de04 [term 1 LEADER]: Signalling peer 49e15cd7009e4d6198f9d9352a27e9d1 to start an election
I20260709 12:19:16.623919 16635 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "0edbf034be4a41d79cbce7d0c876132d"
dest_uuid: "49e15cd7009e4d6198f9d9352a27e9d1"
 from {username='slave'} at 127.0.0.1:34774
I20260709 12:19:16.624826 16635 raft_consensus.cc:493] T 0edbf034be4a41d79cbce7d0c876132d P 49e15cd7009e4d6198f9d9352a27e9d1 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:19:16.625244 16635 raft_consensus.cc:3060] T 0edbf034be4a41d79cbce7d0c876132d P 49e15cd7009e4d6198f9d9352a27e9d1 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:19:16.625314 16599 leader_election.cc:304] T dd736bd544c746238c77320b4a5706e0 P 49e15cd7009e4d6198f9d9352a27e9d1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e772c31b50d4c6693644565ef92de04, 49e15cd7009e4d6198f9d9352a27e9d1; no voters: 
I20260709 12:19:16.627470 16900 raft_consensus.cc:2804] T dd736bd544c746238c77320b4a5706e0 P 49e15cd7009e4d6198f9d9352a27e9d1 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:19:16.628038 16900 raft_consensus.cc:697] T dd736bd544c746238c77320b4a5706e0 P 49e15cd7009e4d6198f9d9352a27e9d1 [term 2 LEADER]: Becoming Leader. State: Replica: 49e15cd7009e4d6198f9d9352a27e9d1, State: Running, Role: LEADER
I20260709 12:19:16.628963 16900 consensus_queue.cc:237] T dd736bd544c746238c77320b4a5706e0 P 49e15cd7009e4d6198f9d9352a27e9d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } } peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } }
I20260709 12:19:16.638654 16709 raft_consensus.cc:2468] T dd736bd544c746238c77320b4a5706e0 P 49598ea210614f1fbb942c1fb77c7932 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49e15cd7009e4d6198f9d9352a27e9d1 in term 2.
I20260709 12:19:16.639034 16533 catalog_manager.cc:5697] T dd736bd544c746238c77320b4a5706e0 P 49e15cd7009e4d6198f9d9352a27e9d1 reported cstate change: term changed from 1 to 2, leader changed from 3e772c31b50d4c6693644565ef92de04 (127.16.29.67) to 49e15cd7009e4d6198f9d9352a27e9d1 (127.16.29.65). New cstate: current_term: 2 leader_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } health_report { overall_health: UNKNOWN } } }
I20260709 12:19:16.640450 16895 consensus_queue.cc:1048] T 95f027ce38784f81bf5c3b35b3357173 P 49e15cd7009e4d6198f9d9352a27e9d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:19:16.649417 16838 consensus_queue.cc:1048] T af674c5ffb4e44dab6f15ab84dbeb737 P 49e15cd7009e4d6198f9d9352a27e9d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:19:16.634855 16635 raft_consensus.cc:515] T 0edbf034be4a41d79cbce7d0c876132d P 49e15cd7009e4d6198f9d9352a27e9d1 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } } peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } }
I20260709 12:19:16.651568 16635 leader_election.cc:290] T 0edbf034be4a41d79cbce7d0c876132d P 49e15cd7009e4d6198f9d9352a27e9d1 [CANDIDATE]: Term 2 election: Requested vote from peers 3e772c31b50d4c6693644565ef92de04 (127.16.29.67:35049), 49598ea210614f1fbb942c1fb77c7932 (127.16.29.66:35871)
I20260709 12:19:16.656312 16708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0edbf034be4a41d79cbce7d0c876132d" candidate_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "49598ea210614f1fbb942c1fb77c7932"
I20260709 12:19:16.657312 16708 raft_consensus.cc:3060] T 0edbf034be4a41d79cbce7d0c876132d P 49598ea210614f1fbb942c1fb77c7932 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:19:16.656968 16784 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0edbf034be4a41d79cbce7d0c876132d" candidate_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "3e772c31b50d4c6693644565ef92de04"
I20260709 12:19:16.664906 16784 raft_consensus.cc:3055] T 0edbf034be4a41d79cbce7d0c876132d P 3e772c31b50d4c6693644565ef92de04 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:19:16.670954 16784 raft_consensus.cc:740] T 0edbf034be4a41d79cbce7d0c876132d P 3e772c31b50d4c6693644565ef92de04 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3e772c31b50d4c6693644565ef92de04, State: Running, Role: LEADER
I20260709 12:19:16.673249 16784 consensus_queue.cc:260] T 0edbf034be4a41d79cbce7d0c876132d P 3e772c31b50d4c6693644565ef92de04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } } peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } }
I20260709 12:19:16.675536 16784 raft_consensus.cc:3060] T 0edbf034be4a41d79cbce7d0c876132d P 3e772c31b50d4c6693644565ef92de04 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:19:16.677161 16708 raft_consensus.cc:2468] T 0edbf034be4a41d79cbce7d0c876132d P 49598ea210614f1fbb942c1fb77c7932 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49e15cd7009e4d6198f9d9352a27e9d1 in term 2.
I20260709 12:19:16.674340 16711 raft_consensus.cc:515] T 3692a4a710ef469aa412d9870427fb63 P 49598ea210614f1fbb942c1fb77c7932 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } } peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } }
I20260709 12:19:16.678843 16600 leader_election.cc:304] T 0edbf034be4a41d79cbce7d0c876132d P 49e15cd7009e4d6198f9d9352a27e9d1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49598ea210614f1fbb942c1fb77c7932, 49e15cd7009e4d6198f9d9352a27e9d1; no voters: 
I20260709 12:19:16.680195 16861 raft_consensus.cc:2804] T 0edbf034be4a41d79cbce7d0c876132d P 49e15cd7009e4d6198f9d9352a27e9d1 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:19:16.680274 16635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3692a4a710ef469aa412d9870427fb63" candidate_uuid: "49598ea210614f1fbb942c1fb77c7932" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "49e15cd7009e4d6198f9d9352a27e9d1"
I20260709 12:19:16.680964 16635 raft_consensus.cc:3055] T 3692a4a710ef469aa412d9870427fb63 P 49e15cd7009e4d6198f9d9352a27e9d1 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:19:16.681300 16635 raft_consensus.cc:740] T 3692a4a710ef469aa412d9870427fb63 P 49e15cd7009e4d6198f9d9352a27e9d1 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 49e15cd7009e4d6198f9d9352a27e9d1, State: Running, Role: LEADER
I20260709 12:19:16.681524 16861 raft_consensus.cc:697] T 0edbf034be4a41d79cbce7d0c876132d P 49e15cd7009e4d6198f9d9352a27e9d1 [term 2 LEADER]: Becoming Leader. State: Replica: 49e15cd7009e4d6198f9d9352a27e9d1, State: Running, Role: LEADER
I20260709 12:19:16.682112 16635 consensus_queue.cc:260] T 3692a4a710ef469aa412d9870427fb63 P 49e15cd7009e4d6198f9d9352a27e9d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } } peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } }
I20260709 12:19:16.682346 16861 consensus_queue.cc:237] T 0edbf034be4a41d79cbce7d0c876132d P 49e15cd7009e4d6198f9d9352a27e9d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } } peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } }
I20260709 12:19:16.682565 16783 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3692a4a710ef469aa412d9870427fb63" candidate_uuid: "49598ea210614f1fbb942c1fb77c7932" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "3e772c31b50d4c6693644565ef92de04"
I20260709 12:19:16.683259 16635 raft_consensus.cc:3060] T 3692a4a710ef469aa412d9870427fb63 P 49e15cd7009e4d6198f9d9352a27e9d1 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:19:16.683579 16783 raft_consensus.cc:3060] T 3692a4a710ef469aa412d9870427fb63 P 3e772c31b50d4c6693644565ef92de04 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:19:16.691546 16635 raft_consensus.cc:2468] T 3692a4a710ef469aa412d9870427fb63 P 49e15cd7009e4d6198f9d9352a27e9d1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49598ea210614f1fbb942c1fb77c7932 in term 2.
I20260709 12:19:16.692853 16674 leader_election.cc:304] T 3692a4a710ef469aa412d9870427fb63 P 49598ea210614f1fbb942c1fb77c7932 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49598ea210614f1fbb942c1fb77c7932, 49e15cd7009e4d6198f9d9352a27e9d1; no voters: 
I20260709 12:19:16.700379 16531 catalog_manager.cc:5697] T 0edbf034be4a41d79cbce7d0c876132d P 49e15cd7009e4d6198f9d9352a27e9d1 reported cstate change: term changed from 1 to 2, leader changed from 3e772c31b50d4c6693644565ef92de04 (127.16.29.67) to 49e15cd7009e4d6198f9d9352a27e9d1 (127.16.29.65). New cstate: current_term: 2 leader_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } health_report { overall_health: HEALTHY } } }
I20260709 12:19:16.703393 16916 raft_consensus.cc:2804] T 3692a4a710ef469aa412d9870427fb63 P 49598ea210614f1fbb942c1fb77c7932 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:19:16.704607 16916 raft_consensus.cc:697] T 3692a4a710ef469aa412d9870427fb63 P 49598ea210614f1fbb942c1fb77c7932 [term 2 LEADER]: Becoming Leader. State: Replica: 49598ea210614f1fbb942c1fb77c7932, State: Running, Role: LEADER
I20260709 12:19:16.694752 16711 leader_election.cc:290] T 3692a4a710ef469aa412d9870427fb63 P 49598ea210614f1fbb942c1fb77c7932 [CANDIDATE]: Term 2 election: Requested vote from peers 49e15cd7009e4d6198f9d9352a27e9d1 (127.16.29.65:44833), 3e772c31b50d4c6693644565ef92de04 (127.16.29.67:35049)
I20260709 12:19:16.712258 16783 raft_consensus.cc:2468] T 3692a4a710ef469aa412d9870427fb63 P 3e772c31b50d4c6693644565ef92de04 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49598ea210614f1fbb942c1fb77c7932 in term 2.
I20260709 12:19:16.719043 16861 consensus_queue.cc:1048] T 95f027ce38784f81bf5c3b35b3357173 P 49e15cd7009e4d6198f9d9352a27e9d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:19:16.705550 16916 consensus_queue.cc:237] T 3692a4a710ef469aa412d9870427fb63 P 49598ea210614f1fbb942c1fb77c7932 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } } peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } }
I20260709 12:19:16.725085 16784 raft_consensus.cc:2468] T 0edbf034be4a41d79cbce7d0c876132d P 3e772c31b50d4c6693644565ef92de04 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49e15cd7009e4d6198f9d9352a27e9d1 in term 2.
I20260709 12:19:16.730181 16840 raft_consensus.cc:993] T 927ab821d939440ab1def38a7dbd0358 P 49598ea210614f1fbb942c1fb77c7932: : Instructing follower 49e15cd7009e4d6198f9d9352a27e9d1 to start an election
I20260709 12:19:16.730928 16840 raft_consensus.cc:1081] T 927ab821d939440ab1def38a7dbd0358 P 49598ea210614f1fbb942c1fb77c7932 [term 1 LEADER]: Signalling peer 49e15cd7009e4d6198f9d9352a27e9d1 to start an election
I20260709 12:19:16.733902 16635 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "927ab821d939440ab1def38a7dbd0358"
dest_uuid: "49e15cd7009e4d6198f9d9352a27e9d1"
 from {username='slave'} at 127.0.0.1:34760
I20260709 12:19:16.734697 16635 raft_consensus.cc:493] T 927ab821d939440ab1def38a7dbd0358 P 49e15cd7009e4d6198f9d9352a27e9d1 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:19:16.735198 16635 raft_consensus.cc:3060] T 927ab821d939440ab1def38a7dbd0358 P 49e15cd7009e4d6198f9d9352a27e9d1 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:19:16.741963 16900 raft_consensus.cc:993] T af674c5ffb4e44dab6f15ab84dbeb737 P 49e15cd7009e4d6198f9d9352a27e9d1: : Instructing follower 49598ea210614f1fbb942c1fb77c7932 to start an election
I20260709 12:19:16.742736 16900 raft_consensus.cc:1081] T af674c5ffb4e44dab6f15ab84dbeb737 P 49e15cd7009e4d6198f9d9352a27e9d1 [term 1 LEADER]: Signalling peer 49598ea210614f1fbb942c1fb77c7932 to start an election
I20260709 12:19:16.749001 16533 catalog_manager.cc:5697] T 3692a4a710ef469aa412d9870427fb63 P 49598ea210614f1fbb942c1fb77c7932 reported cstate change: term changed from 1 to 2, leader changed from 49e15cd7009e4d6198f9d9352a27e9d1 (127.16.29.65) to 49598ea210614f1fbb942c1fb77c7932 (127.16.29.66). New cstate: current_term: 2 leader_uuid: "49598ea210614f1fbb942c1fb77c7932" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } health_report { overall_health: UNKNOWN } } }
I20260709 12:19:16.752375 16710 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "af674c5ffb4e44dab6f15ab84dbeb737"
dest_uuid: "49598ea210614f1fbb942c1fb77c7932"
 from {username='slave'} at 127.0.0.1:33766
I20260709 12:19:16.753064 16710 raft_consensus.cc:493] T af674c5ffb4e44dab6f15ab84dbeb737 P 49598ea210614f1fbb942c1fb77c7932 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:19:16.753479 16710 raft_consensus.cc:3060] T af674c5ffb4e44dab6f15ab84dbeb737 P 49598ea210614f1fbb942c1fb77c7932 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:19:16.813228 16835 consensus_queue.cc:1048] T 5c3e9fafdd8142eea33e771756209147 P 3e772c31b50d4c6693644565ef92de04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:19:16.814056 16838 raft_consensus.cc:993] T 0086bd356ad04503ad306b7217280c34 P 49e15cd7009e4d6198f9d9352a27e9d1: : Instructing follower 49598ea210614f1fbb942c1fb77c7932 to start an election
I20260709 12:19:16.814513 16838 raft_consensus.cc:1081] T 0086bd356ad04503ad306b7217280c34 P 49e15cd7009e4d6198f9d9352a27e9d1 [term 1 LEADER]: Signalling peer 49598ea210614f1fbb942c1fb77c7932 to start an election
I20260709 12:19:16.816882 16708 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "0086bd356ad04503ad306b7217280c34"
dest_uuid: "49598ea210614f1fbb942c1fb77c7932"
 from {username='slave'} at 127.0.0.1:33766
I20260709 12:19:16.817456 16708 raft_consensus.cc:493] T 0086bd356ad04503ad306b7217280c34 P 49598ea210614f1fbb942c1fb77c7932 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:19:16.817811 16708 raft_consensus.cc:3060] T 0086bd356ad04503ad306b7217280c34 P 49598ea210614f1fbb942c1fb77c7932 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:19:16.776036 16635 raft_consensus.cc:515] T 927ab821d939440ab1def38a7dbd0358 P 49e15cd7009e4d6198f9d9352a27e9d1 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } } peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } }
I20260709 12:19:16.824227 16895 raft_consensus.cc:993] T 95f027ce38784f81bf5c3b35b3357173 P 49e15cd7009e4d6198f9d9352a27e9d1: : Instructing follower 49598ea210614f1fbb942c1fb77c7932 to start an election
I20260709 12:19:16.825065 16895 raft_consensus.cc:1081] T 95f027ce38784f81bf5c3b35b3357173 P 49e15cd7009e4d6198f9d9352a27e9d1 [term 1 LEADER]: Signalling peer 49598ea210614f1fbb942c1fb77c7932 to start an election
I20260709 12:19:16.829109 16709 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "95f027ce38784f81bf5c3b35b3357173"
dest_uuid: "49598ea210614f1fbb942c1fb77c7932"
 from {username='slave'} at 127.0.0.1:33766
I20260709 12:19:16.829895 16709 raft_consensus.cc:493] T 95f027ce38784f81bf5c3b35b3357173 P 49598ea210614f1fbb942c1fb77c7932 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:19:16.830382 16709 raft_consensus.cc:3060] T 95f027ce38784f81bf5c3b35b3357173 P 49598ea210614f1fbb942c1fb77c7932 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:19:16.833391 16635 leader_election.cc:290] T 927ab821d939440ab1def38a7dbd0358 P 49e15cd7009e4d6198f9d9352a27e9d1 [CANDIDATE]: Term 2 election: Requested vote from peers 49598ea210614f1fbb942c1fb77c7932 (127.16.29.66:35871), 3e772c31b50d4c6693644565ef92de04 (127.16.29.67:35049)
I20260709 12:19:16.824075 16710 raft_consensus.cc:515] T af674c5ffb4e44dab6f15ab84dbeb737 P 49598ea210614f1fbb942c1fb77c7932 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } } peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } }
I20260709 12:19:16.837965 16785 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "927ab821d939440ab1def38a7dbd0358" candidate_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "3e772c31b50d4c6693644565ef92de04"
I20260709 12:19:16.838528 16785 raft_consensus.cc:3060] T 927ab821d939440ab1def38a7dbd0358 P 3e772c31b50d4c6693644565ef92de04 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:19:16.833290 16707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "927ab821d939440ab1def38a7dbd0358" candidate_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "49598ea210614f1fbb942c1fb77c7932"
I20260709 12:19:16.842206 16835 consensus_queue.cc:1048] T 5c3e9fafdd8142eea33e771756209147 P 3e772c31b50d4c6693644565ef92de04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:19:16.842737 16707 raft_consensus.cc:3055] T 927ab821d939440ab1def38a7dbd0358 P 49598ea210614f1fbb942c1fb77c7932 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:19:16.845052 16707 raft_consensus.cc:740] T 927ab821d939440ab1def38a7dbd0358 P 49598ea210614f1fbb942c1fb77c7932 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 49598ea210614f1fbb942c1fb77c7932, State: Running, Role: LEADER
I20260709 12:19:16.845911 16707 consensus_queue.cc:260] T 927ab821d939440ab1def38a7dbd0358 P 49598ea210614f1fbb942c1fb77c7932 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } } peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } }
I20260709 12:19:16.849308 16707 raft_consensus.cc:3060] T 927ab821d939440ab1def38a7dbd0358 P 49598ea210614f1fbb942c1fb77c7932 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:19:16.849625 16784 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af674c5ffb4e44dab6f15ab84dbeb737" candidate_uuid: "49598ea210614f1fbb942c1fb77c7932" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "3e772c31b50d4c6693644565ef92de04"
I20260709 12:19:16.850229 16784 raft_consensus.cc:3060] T af674c5ffb4e44dab6f15ab84dbeb737 P 3e772c31b50d4c6693644565ef92de04 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:19:16.850883 16635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af674c5ffb4e44dab6f15ab84dbeb737" candidate_uuid: "49598ea210614f1fbb942c1fb77c7932" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "49e15cd7009e4d6198f9d9352a27e9d1"
I20260709 12:19:16.851524 16635 raft_consensus.cc:3055] T af674c5ffb4e44dab6f15ab84dbeb737 P 49e15cd7009e4d6198f9d9352a27e9d1 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:19:16.851915 16635 raft_consensus.cc:740] T af674c5ffb4e44dab6f15ab84dbeb737 P 49e15cd7009e4d6198f9d9352a27e9d1 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 49e15cd7009e4d6198f9d9352a27e9d1, State: Running, Role: LEADER
I20260709 12:19:16.857983 16710 leader_election.cc:290] T af674c5ffb4e44dab6f15ab84dbeb737 P 49598ea210614f1fbb942c1fb77c7932 [CANDIDATE]: Term 2 election: Requested vote from peers 49e15cd7009e4d6198f9d9352a27e9d1 (127.16.29.65:44833), 3e772c31b50d4c6693644565ef92de04 (127.16.29.67:35049)
I20260709 12:19:16.876258 16784 raft_consensus.cc:2468] T af674c5ffb4e44dab6f15ab84dbeb737 P 3e772c31b50d4c6693644565ef92de04 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49598ea210614f1fbb942c1fb77c7932 in term 2.
I20260709 12:19:16.872630 16709 raft_consensus.cc:515] T 95f027ce38784f81bf5c3b35b3357173 P 49598ea210614f1fbb942c1fb77c7932 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } } peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } }
I20260709 12:19:16.877585 16674 leader_election.cc:304] T af674c5ffb4e44dab6f15ab84dbeb737 P 49598ea210614f1fbb942c1fb77c7932 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e772c31b50d4c6693644565ef92de04, 49598ea210614f1fbb942c1fb77c7932; no voters: 
I20260709 12:19:16.879199 16917 raft_consensus.cc:2804] T af674c5ffb4e44dab6f15ab84dbeb737 P 49598ea210614f1fbb942c1fb77c7932 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:19:16.879987 16917 raft_consensus.cc:697] T af674c5ffb4e44dab6f15ab84dbeb737 P 49598ea210614f1fbb942c1fb77c7932 [term 2 LEADER]: Becoming Leader. State: Replica: 49598ea210614f1fbb942c1fb77c7932, State: Running, Role: LEADER
I20260709 12:19:16.881242 16917 consensus_queue.cc:237] T af674c5ffb4e44dab6f15ab84dbeb737 P 49598ea210614f1fbb942c1fb77c7932 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } } peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } }
I20260709 12:19:16.884804 16708 raft_consensus.cc:515] T 0086bd356ad04503ad306b7217280c34 P 49598ea210614f1fbb942c1fb77c7932 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } } peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } }
I20260709 12:19:16.868316 16635 consensus_queue.cc:260] T af674c5ffb4e44dab6f15ab84dbeb737 P 49e15cd7009e4d6198f9d9352a27e9d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } } peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } }
I20260709 12:19:16.888118 16785 raft_consensus.cc:2468] T 927ab821d939440ab1def38a7dbd0358 P 3e772c31b50d4c6693644565ef92de04 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49e15cd7009e4d6198f9d9352a27e9d1 in term 2.
I20260709 12:19:16.896142 16635 raft_consensus.cc:3060] T af674c5ffb4e44dab6f15ab84dbeb737 P 49e15cd7009e4d6198f9d9352a27e9d1 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:19:16.900151 16599 leader_election.cc:304] T 927ab821d939440ab1def38a7dbd0358 P 49e15cd7009e4d6198f9d9352a27e9d1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e772c31b50d4c6693644565ef92de04, 49e15cd7009e4d6198f9d9352a27e9d1; no voters: 
I20260709 12:19:16.901734 16634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f027ce38784f81bf5c3b35b3357173" candidate_uuid: "49598ea210614f1fbb942c1fb77c7932" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "49e15cd7009e4d6198f9d9352a27e9d1"
I20260709 12:19:16.903650 16634 raft_consensus.cc:3055] T 95f027ce38784f81bf5c3b35b3357173 P 49e15cd7009e4d6198f9d9352a27e9d1 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:19:16.904018 16634 raft_consensus.cc:740] T 95f027ce38784f81bf5c3b35b3357173 P 49e15cd7009e4d6198f9d9352a27e9d1 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 49e15cd7009e4d6198f9d9352a27e9d1, State: Running, Role: LEADER
I20260709 12:19:16.904071 16895 raft_consensus.cc:2804] T 927ab821d939440ab1def38a7dbd0358 P 49e15cd7009e4d6198f9d9352a27e9d1 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:19:16.904647 16895 raft_consensus.cc:697] T 927ab821d939440ab1def38a7dbd0358 P 49e15cd7009e4d6198f9d9352a27e9d1 [term 2 LEADER]: Becoming Leader. State: Replica: 49e15cd7009e4d6198f9d9352a27e9d1, State: Running, Role: LEADER
I20260709 12:19:16.908442 16633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0086bd356ad04503ad306b7217280c34" candidate_uuid: "49598ea210614f1fbb942c1fb77c7932" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "49e15cd7009e4d6198f9d9352a27e9d1"
I20260709 12:19:16.910022 16633 raft_consensus.cc:3055] T 0086bd356ad04503ad306b7217280c34 P 49e15cd7009e4d6198f9d9352a27e9d1 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:19:16.910434 16633 raft_consensus.cc:740] T 0086bd356ad04503ad306b7217280c34 P 49e15cd7009e4d6198f9d9352a27e9d1 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 49e15cd7009e4d6198f9d9352a27e9d1, State: Running, Role: LEADER
I20260709 12:19:16.911216 16633 consensus_queue.cc:260] T 0086bd356ad04503ad306b7217280c34 P 49e15cd7009e4d6198f9d9352a27e9d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } } peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } }
I20260709 12:19:16.914523 16633 raft_consensus.cc:3060] T 0086bd356ad04503ad306b7217280c34 P 49e15cd7009e4d6198f9d9352a27e9d1 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:19:16.914723 16708 leader_election.cc:290] T 0086bd356ad04503ad306b7217280c34 P 49598ea210614f1fbb942c1fb77c7932 [CANDIDATE]: Term 2 election: Requested vote from peers 49e15cd7009e4d6198f9d9352a27e9d1 (127.16.29.65:44833), 3e772c31b50d4c6693644565ef92de04 (127.16.29.67:35049)
I20260709 12:19:16.904826 16634 consensus_queue.cc:260] T 95f027ce38784f81bf5c3b35b3357173 P 49e15cd7009e4d6198f9d9352a27e9d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } } peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } }
I20260709 12:19:16.907194 16895 consensus_queue.cc:237] T 927ab821d939440ab1def38a7dbd0358 P 49e15cd7009e4d6198f9d9352a27e9d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } } peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } }
I20260709 12:19:16.915802 16785 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0086bd356ad04503ad306b7217280c34" candidate_uuid: "49598ea210614f1fbb942c1fb77c7932" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "3e772c31b50d4c6693644565ef92de04"
I20260709 12:19:16.916409 16785 raft_consensus.cc:3060] T 0086bd356ad04503ad306b7217280c34 P 3e772c31b50d4c6693644565ef92de04 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:19:16.915589 16634 raft_consensus.cc:3060] T 95f027ce38784f81bf5c3b35b3357173 P 49e15cd7009e4d6198f9d9352a27e9d1 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:19:16.919154 16784 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f027ce38784f81bf5c3b35b3357173" candidate_uuid: "49598ea210614f1fbb942c1fb77c7932" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "3e772c31b50d4c6693644565ef92de04"
I20260709 12:19:16.923799 16784 raft_consensus.cc:3060] T 95f027ce38784f81bf5c3b35b3357173 P 3e772c31b50d4c6693644565ef92de04 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:19:16.928375 16531 catalog_manager.cc:5697] T af674c5ffb4e44dab6f15ab84dbeb737 P 49598ea210614f1fbb942c1fb77c7932 reported cstate change: term changed from 1 to 2, leader changed from 49e15cd7009e4d6198f9d9352a27e9d1 (127.16.29.65) to 49598ea210614f1fbb942c1fb77c7932 (127.16.29.66). New cstate: current_term: 2 leader_uuid: "49598ea210614f1fbb942c1fb77c7932" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } health_report { overall_health: UNKNOWN } } }
I20260709 12:19:16.931815 16532 catalog_manager.cc:5697] T 927ab821d939440ab1def38a7dbd0358 P 49e15cd7009e4d6198f9d9352a27e9d1 reported cstate change: term changed from 1 to 2, leader changed from 49598ea210614f1fbb942c1fb77c7932 (127.16.29.66) to 49e15cd7009e4d6198f9d9352a27e9d1 (127.16.29.65). New cstate: current_term: 2 leader_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } health_report { overall_health: UNKNOWN } } }
I20260709 12:19:16.918363 16709 leader_election.cc:290] T 95f027ce38784f81bf5c3b35b3357173 P 49598ea210614f1fbb942c1fb77c7932 [CANDIDATE]: Term 2 election: Requested vote from peers 49e15cd7009e4d6198f9d9352a27e9d1 (127.16.29.65:44833), 3e772c31b50d4c6693644565ef92de04 (127.16.29.67:35049)
I20260709 12:19:16.942529 16785 raft_consensus.cc:2468] T 0086bd356ad04503ad306b7217280c34 P 3e772c31b50d4c6693644565ef92de04 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49598ea210614f1fbb942c1fb77c7932 in term 2.
I20260709 12:19:16.944034 16674 leader_election.cc:304] T 0086bd356ad04503ad306b7217280c34 P 49598ea210614f1fbb942c1fb77c7932 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e772c31b50d4c6693644565ef92de04, 49598ea210614f1fbb942c1fb77c7932; no voters: 
I20260709 12:19:16.944206 16707 raft_consensus.cc:2468] T 927ab821d939440ab1def38a7dbd0358 P 49598ea210614f1fbb942c1fb77c7932 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49e15cd7009e4d6198f9d9352a27e9d1 in term 2.
I20260709 12:19:16.944938 16917 raft_consensus.cc:2804] T 0086bd356ad04503ad306b7217280c34 P 49598ea210614f1fbb942c1fb77c7932 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:19:16.945415 16917 raft_consensus.cc:697] T 0086bd356ad04503ad306b7217280c34 P 49598ea210614f1fbb942c1fb77c7932 [term 2 LEADER]: Becoming Leader. State: Replica: 49598ea210614f1fbb942c1fb77c7932, State: Running, Role: LEADER
I20260709 12:19:16.946193 16917 consensus_queue.cc:237] T 0086bd356ad04503ad306b7217280c34 P 49598ea210614f1fbb942c1fb77c7932 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } } peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } }
I20260709 12:19:16.951699 16634 raft_consensus.cc:2468] T 95f027ce38784f81bf5c3b35b3357173 P 49e15cd7009e4d6198f9d9352a27e9d1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49598ea210614f1fbb942c1fb77c7932 in term 2.
I20260709 12:19:16.956647 16674 leader_election.cc:304] T 95f027ce38784f81bf5c3b35b3357173 P 49598ea210614f1fbb942c1fb77c7932 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49598ea210614f1fbb942c1fb77c7932, 49e15cd7009e4d6198f9d9352a27e9d1; no voters: 
I20260709 12:19:16.958472 16917 raft_consensus.cc:2804] T 95f027ce38784f81bf5c3b35b3357173 P 49598ea210614f1fbb942c1fb77c7932 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:19:16.959161 16917 raft_consensus.cc:697] T 95f027ce38784f81bf5c3b35b3357173 P 49598ea210614f1fbb942c1fb77c7932 [term 2 LEADER]: Becoming Leader. State: Replica: 49598ea210614f1fbb942c1fb77c7932, State: Running, Role: LEADER
I20260709 12:19:16.960006 16917 consensus_queue.cc:237] T 95f027ce38784f81bf5c3b35b3357173 P 49598ea210614f1fbb942c1fb77c7932 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } } peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } }
I20260709 12:19:16.960867 16635 raft_consensus.cc:2468] T af674c5ffb4e44dab6f15ab84dbeb737 P 49e15cd7009e4d6198f9d9352a27e9d1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49598ea210614f1fbb942c1fb77c7932 in term 2.
I20260709 12:19:16.961102 16533 catalog_manager.cc:5697] T 0086bd356ad04503ad306b7217280c34 P 49598ea210614f1fbb942c1fb77c7932 reported cstate change: term changed from 1 to 2, leader changed from 49e15cd7009e4d6198f9d9352a27e9d1 (127.16.29.65) to 49598ea210614f1fbb942c1fb77c7932 (127.16.29.66). New cstate: current_term: 2 leader_uuid: "49598ea210614f1fbb942c1fb77c7932" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } health_report { overall_health: UNKNOWN } } }
I20260709 12:19:16.954362 16784 raft_consensus.cc:2468] T 95f027ce38784f81bf5c3b35b3357173 P 3e772c31b50d4c6693644565ef92de04 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49598ea210614f1fbb942c1fb77c7932 in term 2.
I20260709 12:19:16.966840 16633 raft_consensus.cc:2468] T 0086bd356ad04503ad306b7217280c34 P 49e15cd7009e4d6198f9d9352a27e9d1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49598ea210614f1fbb942c1fb77c7932 in term 2.
I20260709 12:19:16.976070 16533 catalog_manager.cc:5697] T 95f027ce38784f81bf5c3b35b3357173 P 49598ea210614f1fbb942c1fb77c7932 reported cstate change: term changed from 1 to 2, leader changed from 49e15cd7009e4d6198f9d9352a27e9d1 (127.16.29.65) to 49598ea210614f1fbb942c1fb77c7932 (127.16.29.66). New cstate: current_term: 2 leader_uuid: "49598ea210614f1fbb942c1fb77c7932" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } health_report { overall_health: UNKNOWN } } }
I20260709 12:19:17.108493 16784 raft_consensus.cc:1275] T 0edbf034be4a41d79cbce7d0c876132d P 3e772c31b50d4c6693644565ef92de04 [term 2 FOLLOWER]: Refusing update from remote peer 49e15cd7009e4d6198f9d9352a27e9d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:19:17.110162 16895 consensus_queue.cc:1048] T 0edbf034be4a41d79cbce7d0c876132d P 49e15cd7009e4d6198f9d9352a27e9d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:19:17.120254 16707 raft_consensus.cc:1275] T 0edbf034be4a41d79cbce7d0c876132d P 49598ea210614f1fbb942c1fb77c7932 [term 2 FOLLOWER]: Refusing update from remote peer 49e15cd7009e4d6198f9d9352a27e9d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:19:17.122953 16895 consensus_queue.cc:1048] T 0edbf034be4a41d79cbce7d0c876132d P 49e15cd7009e4d6198f9d9352a27e9d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:19:17.125372 16784 raft_consensus.cc:1275] T dd736bd544c746238c77320b4a5706e0 P 3e772c31b50d4c6693644565ef92de04 [term 2 FOLLOWER]: Refusing update from remote peer 49e15cd7009e4d6198f9d9352a27e9d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:19:17.128129 16900 consensus_queue.cc:1048] T dd736bd544c746238c77320b4a5706e0 P 49e15cd7009e4d6198f9d9352a27e9d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:19:17.139729 16707 raft_consensus.cc:1275] T dd736bd544c746238c77320b4a5706e0 P 49598ea210614f1fbb942c1fb77c7932 [term 2 FOLLOWER]: Refusing update from remote peer 49e15cd7009e4d6198f9d9352a27e9d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:19:17.147534 16838 consensus_queue.cc:1048] T dd736bd544c746238c77320b4a5706e0 P 49e15cd7009e4d6198f9d9352a27e9d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:19:17.279102 16635 raft_consensus.cc:1275] T af674c5ffb4e44dab6f15ab84dbeb737 P 49e15cd7009e4d6198f9d9352a27e9d1 [term 2 FOLLOWER]: Refusing update from remote peer 49598ea210614f1fbb942c1fb77c7932: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:19:17.280753 16917 consensus_queue.cc:1048] T af674c5ffb4e44dab6f15ab84dbeb737 P 49598ea210614f1fbb942c1fb77c7932 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:19:17.289870 16784 raft_consensus.cc:1275] T af674c5ffb4e44dab6f15ab84dbeb737 P 3e772c31b50d4c6693644565ef92de04 [term 2 FOLLOWER]: Refusing update from remote peer 49598ea210614f1fbb942c1fb77c7932: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:19:17.292663 16916 consensus_queue.cc:1048] T af674c5ffb4e44dab6f15ab84dbeb737 P 49598ea210614f1fbb942c1fb77c7932 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:19:17.333897 16635 raft_consensus.cc:1275] T 3692a4a710ef469aa412d9870427fb63 P 49e15cd7009e4d6198f9d9352a27e9d1 [term 2 FOLLOWER]: Refusing update from remote peer 49598ea210614f1fbb942c1fb77c7932: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:19:17.335475 16917 consensus_queue.cc:1048] T 3692a4a710ef469aa412d9870427fb63 P 49598ea210614f1fbb942c1fb77c7932 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:19:17.345311 16784 raft_consensus.cc:1275] T 3692a4a710ef469aa412d9870427fb63 P 3e772c31b50d4c6693644565ef92de04 [term 2 FOLLOWER]: Refusing update from remote peer 49598ea210614f1fbb942c1fb77c7932: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:19:17.346719 16916 consensus_queue.cc:1048] T 3692a4a710ef469aa412d9870427fb63 P 49598ea210614f1fbb942c1fb77c7932 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:19:17.392397 16784 raft_consensus.cc:1275] T 95f027ce38784f81bf5c3b35b3357173 P 3e772c31b50d4c6693644565ef92de04 [term 2 FOLLOWER]: Refusing update from remote peer 49598ea210614f1fbb942c1fb77c7932: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:19:17.392741 16785 raft_consensus.cc:1275] T 0086bd356ad04503ad306b7217280c34 P 3e772c31b50d4c6693644565ef92de04 [term 2 FOLLOWER]: Refusing update from remote peer 49598ea210614f1fbb942c1fb77c7932: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:19:17.393883 16916 consensus_queue.cc:1048] T 95f027ce38784f81bf5c3b35b3357173 P 49598ea210614f1fbb942c1fb77c7932 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:19:17.394220 16929 consensus_queue.cc:1048] T 0086bd356ad04503ad306b7217280c34 P 49598ea210614f1fbb942c1fb77c7932 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:19:17.404410 16635 raft_consensus.cc:1275] T 95f027ce38784f81bf5c3b35b3357173 P 49e15cd7009e4d6198f9d9352a27e9d1 [term 2 FOLLOWER]: Refusing update from remote peer 49598ea210614f1fbb942c1fb77c7932: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:19:17.405915 16927 consensus_queue.cc:1048] T 95f027ce38784f81bf5c3b35b3357173 P 49598ea210614f1fbb942c1fb77c7932 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:19:17.410141 16635 raft_consensus.cc:1275] T 0086bd356ad04503ad306b7217280c34 P 49e15cd7009e4d6198f9d9352a27e9d1 [term 2 FOLLOWER]: Refusing update from remote peer 49598ea210614f1fbb942c1fb77c7932: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:19:17.411657 16916 consensus_queue.cc:1048] T 0086bd356ad04503ad306b7217280c34 P 49598ea210614f1fbb942c1fb77c7932 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:19:17.459564 16707 raft_consensus.cc:1275] T 927ab821d939440ab1def38a7dbd0358 P 49598ea210614f1fbb942c1fb77c7932 [term 2 FOLLOWER]: Refusing update from remote peer 49e15cd7009e4d6198f9d9352a27e9d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:19:17.461405 16900 consensus_queue.cc:1048] T 927ab821d939440ab1def38a7dbd0358 P 49e15cd7009e4d6198f9d9352a27e9d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:19:17.471175 16784 raft_consensus.cc:1275] T 927ab821d939440ab1def38a7dbd0358 P 3e772c31b50d4c6693644565ef92de04 [term 2 FOLLOWER]: Refusing update from remote peer 49e15cd7009e4d6198f9d9352a27e9d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:19:17.473971 16900 consensus_queue.cc:1048] T 927ab821d939440ab1def38a7dbd0358 P 49e15cd7009e4d6198f9d9352a27e9d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:19:19.559422 16501 auto_leader_rebalancer-test.cc:294] The leader distribution is 
I20260709 12:19:19.560322 16501 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:19:19.584160 16707 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "0086bd356ad04503ad306b7217280c34"
dest_uuid: "49598ea210614f1fbb942c1fb77c7932"
mode: GRACEFUL
new_leader_uuid: "3e772c31b50d4c6693644565ef92de04"
 from {username='slave'} at 127.0.0.1:33772
I20260709 12:19:19.584851 16707 raft_consensus.cc:606] T 0086bd356ad04503ad306b7217280c34 P 49598ea210614f1fbb942c1fb77c7932 [term 2 LEADER]: Received request to transfer leadership to TS 3e772c31b50d4c6693644565ef92de04
I20260709 12:19:19.596392 16633 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "dd736bd544c746238c77320b4a5706e0"
dest_uuid: "49e15cd7009e4d6198f9d9352a27e9d1"
mode: GRACEFUL
new_leader_uuid: "3e772c31b50d4c6693644565ef92de04"
 from {username='slave'} at 127.0.0.1:34784
I20260709 12:19:19.596863 16633 raft_consensus.cc:606] T dd736bd544c746238c77320b4a5706e0 P 49e15cd7009e4d6198f9d9352a27e9d1 [term 2 LEADER]: Received request to transfer leadership to TS 3e772c31b50d4c6693644565ef92de04
I20260709 12:19:19.598065 16501 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260709 12:19:19.598627 16501 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:19:19.662259 16940 raft_consensus.cc:993] T dd736bd544c746238c77320b4a5706e0 P 49e15cd7009e4d6198f9d9352a27e9d1: : Instructing follower 3e772c31b50d4c6693644565ef92de04 to start an election
I20260709 12:19:19.662806 16940 raft_consensus.cc:1081] T dd736bd544c746238c77320b4a5706e0 P 49e15cd7009e4d6198f9d9352a27e9d1 [term 2 LEADER]: Signalling peer 3e772c31b50d4c6693644565ef92de04 to start an election
I20260709 12:19:19.664600 16784 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "dd736bd544c746238c77320b4a5706e0"
dest_uuid: "3e772c31b50d4c6693644565ef92de04"
 from {username='slave'} at 127.0.0.1:52752
I20260709 12:19:19.665287 16784 raft_consensus.cc:493] T dd736bd544c746238c77320b4a5706e0 P 3e772c31b50d4c6693644565ef92de04 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:19:19.665611 16784 raft_consensus.cc:3060] T dd736bd544c746238c77320b4a5706e0 P 3e772c31b50d4c6693644565ef92de04 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:19:19.670559 16784 raft_consensus.cc:515] T dd736bd544c746238c77320b4a5706e0 P 3e772c31b50d4c6693644565ef92de04 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } } peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } }
I20260709 12:19:19.672942 16784 leader_election.cc:290] T dd736bd544c746238c77320b4a5706e0 P 3e772c31b50d4c6693644565ef92de04 [CANDIDATE]: Term 3 election: Requested vote from peers 49e15cd7009e4d6198f9d9352a27e9d1 (127.16.29.65:44833), 49598ea210614f1fbb942c1fb77c7932 (127.16.29.66:35871)
I20260709 12:19:19.673497 16633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd736bd544c746238c77320b4a5706e0" candidate_uuid: "3e772c31b50d4c6693644565ef92de04" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "49e15cd7009e4d6198f9d9352a27e9d1"
I20260709 12:19:19.673659 16707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd736bd544c746238c77320b4a5706e0" candidate_uuid: "3e772c31b50d4c6693644565ef92de04" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "49598ea210614f1fbb942c1fb77c7932"
I20260709 12:19:19.674247 16633 raft_consensus.cc:3055] T dd736bd544c746238c77320b4a5706e0 P 49e15cd7009e4d6198f9d9352a27e9d1 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:19:19.674358 16707 raft_consensus.cc:3060] T dd736bd544c746238c77320b4a5706e0 P 49598ea210614f1fbb942c1fb77c7932 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:19:19.674664 16633 raft_consensus.cc:740] T dd736bd544c746238c77320b4a5706e0 P 49e15cd7009e4d6198f9d9352a27e9d1 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 49e15cd7009e4d6198f9d9352a27e9d1, State: Running, Role: LEADER
I20260709 12:19:19.675611 16633 consensus_queue.cc:260] T dd736bd544c746238c77320b4a5706e0 P 49e15cd7009e4d6198f9d9352a27e9d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } } peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } }
I20260709 12:19:19.676833 16633 raft_consensus.cc:3060] T dd736bd544c746238c77320b4a5706e0 P 49e15cd7009e4d6198f9d9352a27e9d1 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:19:19.680074 16707 raft_consensus.cc:2468] T dd736bd544c746238c77320b4a5706e0 P 49598ea210614f1fbb942c1fb77c7932 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e772c31b50d4c6693644565ef92de04 in term 3.
I20260709 12:19:19.681409 16749 leader_election.cc:304] T dd736bd544c746238c77320b4a5706e0 P 3e772c31b50d4c6693644565ef92de04 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e772c31b50d4c6693644565ef92de04, 49598ea210614f1fbb942c1fb77c7932; no voters: 
I20260709 12:19:19.682346 16835 raft_consensus.cc:2804] T dd736bd544c746238c77320b4a5706e0 P 3e772c31b50d4c6693644565ef92de04 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:19:19.682539 16633 raft_consensus.cc:2468] T dd736bd544c746238c77320b4a5706e0 P 49e15cd7009e4d6198f9d9352a27e9d1 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e772c31b50d4c6693644565ef92de04 in term 3.
I20260709 12:19:19.683720 16835 raft_consensus.cc:697] T dd736bd544c746238c77320b4a5706e0 P 3e772c31b50d4c6693644565ef92de04 [term 3 LEADER]: Becoming Leader. State: Replica: 3e772c31b50d4c6693644565ef92de04, State: Running, Role: LEADER
I20260709 12:19:19.684643 16835 consensus_queue.cc:237] T dd736bd544c746238c77320b4a5706e0 P 3e772c31b50d4c6693644565ef92de04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } } peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } }
I20260709 12:19:19.692147 16532 catalog_manager.cc:5697] T dd736bd544c746238c77320b4a5706e0 P 3e772c31b50d4c6693644565ef92de04 reported cstate change: term changed from 2 to 3, leader changed from 49e15cd7009e4d6198f9d9352a27e9d1 (127.16.29.65) to 3e772c31b50d4c6693644565ef92de04 (127.16.29.67). New cstate: current_term: 3 leader_uuid: "3e772c31b50d4c6693644565ef92de04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } health_report { overall_health: HEALTHY } } }
I20260709 12:19:19.813117 16916 raft_consensus.cc:993] T 0086bd356ad04503ad306b7217280c34 P 49598ea210614f1fbb942c1fb77c7932: : Instructing follower 3e772c31b50d4c6693644565ef92de04 to start an election
I20260709 12:19:19.813635 16936 raft_consensus.cc:1081] T 0086bd356ad04503ad306b7217280c34 P 49598ea210614f1fbb942c1fb77c7932 [term 2 LEADER]: Signalling peer 3e772c31b50d4c6693644565ef92de04 to start an election
I20260709 12:19:19.815373 16784 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "0086bd356ad04503ad306b7217280c34"
dest_uuid: "3e772c31b50d4c6693644565ef92de04"
 from {username='slave'} at 127.0.0.1:52742
I20260709 12:19:19.816004 16784 raft_consensus.cc:493] T 0086bd356ad04503ad306b7217280c34 P 3e772c31b50d4c6693644565ef92de04 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:19:19.816435 16784 raft_consensus.cc:3060] T 0086bd356ad04503ad306b7217280c34 P 3e772c31b50d4c6693644565ef92de04 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:19:19.821947 16784 raft_consensus.cc:515] T 0086bd356ad04503ad306b7217280c34 P 3e772c31b50d4c6693644565ef92de04 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } } peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } }
I20260709 12:19:19.823743 16784 leader_election.cc:290] T 0086bd356ad04503ad306b7217280c34 P 3e772c31b50d4c6693644565ef92de04 [CANDIDATE]: Term 3 election: Requested vote from peers 49e15cd7009e4d6198f9d9352a27e9d1 (127.16.29.65:44833), 49598ea210614f1fbb942c1fb77c7932 (127.16.29.66:35871)
I20260709 12:19:19.824801 16635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0086bd356ad04503ad306b7217280c34" candidate_uuid: "3e772c31b50d4c6693644565ef92de04" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "49e15cd7009e4d6198f9d9352a27e9d1"
I20260709 12:19:19.825150 16707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0086bd356ad04503ad306b7217280c34" candidate_uuid: "3e772c31b50d4c6693644565ef92de04" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "49598ea210614f1fbb942c1fb77c7932"
I20260709 12:19:19.825472 16635 raft_consensus.cc:3060] T 0086bd356ad04503ad306b7217280c34 P 49e15cd7009e4d6198f9d9352a27e9d1 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:19:19.825832 16707 raft_consensus.cc:3055] T 0086bd356ad04503ad306b7217280c34 P 49598ea210614f1fbb942c1fb77c7932 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:19:19.826197 16707 raft_consensus.cc:740] T 0086bd356ad04503ad306b7217280c34 P 49598ea210614f1fbb942c1fb77c7932 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 49598ea210614f1fbb942c1fb77c7932, State: Running, Role: LEADER
I20260709 12:19:19.826968 16707 consensus_queue.cc:260] T 0086bd356ad04503ad306b7217280c34 P 49598ea210614f1fbb942c1fb77c7932 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } } peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } }
I20260709 12:19:19.828043 16707 raft_consensus.cc:3060] T 0086bd356ad04503ad306b7217280c34 P 49598ea210614f1fbb942c1fb77c7932 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:19:19.830847 16635 raft_consensus.cc:2468] T 0086bd356ad04503ad306b7217280c34 P 49e15cd7009e4d6198f9d9352a27e9d1 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e772c31b50d4c6693644565ef92de04 in term 3.
I20260709 12:19:19.832458 16748 leader_election.cc:304] T 0086bd356ad04503ad306b7217280c34 P 3e772c31b50d4c6693644565ef92de04 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e772c31b50d4c6693644565ef92de04, 49e15cd7009e4d6198f9d9352a27e9d1; no voters: 
I20260709 12:19:19.833369 16835 raft_consensus.cc:2804] T 0086bd356ad04503ad306b7217280c34 P 3e772c31b50d4c6693644565ef92de04 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:19:19.833636 16707 raft_consensus.cc:2468] T 0086bd356ad04503ad306b7217280c34 P 49598ea210614f1fbb942c1fb77c7932 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e772c31b50d4c6693644565ef92de04 in term 3.
I20260709 12:19:19.833842 16835 raft_consensus.cc:697] T 0086bd356ad04503ad306b7217280c34 P 3e772c31b50d4c6693644565ef92de04 [term 3 LEADER]: Becoming Leader. State: Replica: 3e772c31b50d4c6693644565ef92de04, State: Running, Role: LEADER
I20260709 12:19:19.834867 16835 consensus_queue.cc:237] T 0086bd356ad04503ad306b7217280c34 P 3e772c31b50d4c6693644565ef92de04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } } peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } }
I20260709 12:19:19.843087 16532 catalog_manager.cc:5697] T 0086bd356ad04503ad306b7217280c34 P 3e772c31b50d4c6693644565ef92de04 reported cstate change: term changed from 2 to 3, leader changed from 49598ea210614f1fbb942c1fb77c7932 (127.16.29.66) to 3e772c31b50d4c6693644565ef92de04 (127.16.29.67). New cstate: current_term: 3 leader_uuid: "3e772c31b50d4c6693644565ef92de04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } health_report { overall_health: HEALTHY } } }
I20260709 12:19:20.110785 16635 raft_consensus.cc:1275] T dd736bd544c746238c77320b4a5706e0 P 49e15cd7009e4d6198f9d9352a27e9d1 [term 3 FOLLOWER]: Refusing update from remote peer 3e772c31b50d4c6693644565ef92de04: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:19:20.112430 16835 consensus_queue.cc:1048] T dd736bd544c746238c77320b4a5706e0 P 3e772c31b50d4c6693644565ef92de04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:19:20.123970 16707 raft_consensus.cc:1275] T dd736bd544c746238c77320b4a5706e0 P 49598ea210614f1fbb942c1fb77c7932 [term 3 FOLLOWER]: Refusing update from remote peer 3e772c31b50d4c6693644565ef92de04: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:19:20.125972 16951 consensus_queue.cc:1048] T dd736bd544c746238c77320b4a5706e0 P 3e772c31b50d4c6693644565ef92de04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:19:20.398612 16707 raft_consensus.cc:1275] T 0086bd356ad04503ad306b7217280c34 P 49598ea210614f1fbb942c1fb77c7932 [term 3 FOLLOWER]: Refusing update from remote peer 3e772c31b50d4c6693644565ef92de04: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:19:20.399979 16951 consensus_queue.cc:1048] T 0086bd356ad04503ad306b7217280c34 P 3e772c31b50d4c6693644565ef92de04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:19:20.408501 16635 raft_consensus.cc:1275] T 0086bd356ad04503ad306b7217280c34 P 49e15cd7009e4d6198f9d9352a27e9d1 [term 3 FOLLOWER]: Refusing update from remote peer 3e772c31b50d4c6693644565ef92de04: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:19:20.410104 16956 consensus_queue.cc:1048] T 0086bd356ad04503ad306b7217280c34 P 3e772c31b50d4c6693644565ef92de04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:19:20.599663 16501 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:19:20.604393 16501 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:19:20.604725 16501 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:19:21.605599 16501 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:19:21.611536 16501 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:19:21.612015 16501 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:19:22.613507 16501 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:19:22.620137 16501 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:19:22.620651 16501 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
W20260709 12:19:23.895262 16657 debug-util.cc:398] Leaking SignalData structure 0x7b08000e5060 after lost signal to thread 16504
W20260709 12:19:23.901372 16657 debug-util.cc:398] Leaking SignalData structure 0x7b08000e5740 after lost signal to thread 16567
W20260709 12:19:23.902824 16657 debug-util.cc:398] Leaking SignalData structure 0x7b08001f3a60 after lost signal to thread 16660
W20260709 12:19:23.904422 16657 debug-util.cc:398] Leaking SignalData structure 0x7b0800196660 after lost signal to thread 16735
W20260709 12:19:23.906245 16657 debug-util.cc:398] Leaking SignalData structure 0x7b08001f10a0 after lost signal to thread 16809
I20260709 12:19:24.407512 16501 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:19:24.413631 16501 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:19:24.414124 16501 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
W20260709 12:19:24.897619 16732 debug-util.cc:398] Leaking SignalData structure 0x7b080012a060 after lost signal to thread 16504
W20260709 12:19:24.898499 16732 debug-util.cc:398] Leaking SignalData structure 0x7b080012a740 after lost signal to thread 16567
W20260709 12:19:24.899464 16732 debug-util.cc:398] Leaking SignalData structure 0x7b0800114c40 after lost signal to thread 16660
W20260709 12:19:24.900425 16732 debug-util.cc:398] Leaking SignalData structure 0x7b0800197da0 after lost signal to thread 16735
W20260709 12:19:24.901252 16732 debug-util.cc:398] Leaking SignalData structure 0x7b08001e2e40 after lost signal to thread 16809
I20260709 12:19:25.415279 16501 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:19:25.423986 16501 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:19:25.424588 16501 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:19:26.425833 16501 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:19:26.433820 16501 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:19:26.434369 16501 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:19:27.435461 16501 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:19:27.441648 16501 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:19:27.442111 16501 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:19:28.991266 16501 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:19:28.996778 16501 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:19:28.997427 16501 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
W20260709 12:19:29.853998 16806 debug-util.cc:398] Leaking SignalData structure 0x7b080016d060 after lost signal to thread 16504
W20260709 12:19:29.854794 16806 debug-util.cc:398] Leaking SignalData structure 0x7b080016d740 after lost signal to thread 16567
W20260709 12:19:29.855623 16806 debug-util.cc:398] Leaking SignalData structure 0x7b08001e27e0 after lost signal to thread 16660
W20260709 12:19:29.856434 16806 debug-util.cc:398] Leaking SignalData structure 0x7b08001ed1e0 after lost signal to thread 16735
W20260709 12:19:29.857218 16806 debug-util.cc:398] Leaking SignalData structure 0x7b08001cd7c0 after lost signal to thread 16809
I20260709 12:19:29.998409 16501 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:19:30.004281 16501 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:19:30.004743 16501 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:19:31.040027 16501 auto_leader_rebalancer-test.cc:313] The leader distribution is 
I20260709 12:19:31.047077 16633 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "0edbf034be4a41d79cbce7d0c876132d"
dest_uuid: "49e15cd7009e4d6198f9d9352a27e9d1"
mode: GRACEFUL
new_leader_uuid: "49598ea210614f1fbb942c1fb77c7932"
 from {username='slave'} at 127.0.0.1:34778
I20260709 12:19:31.047933 16633 raft_consensus.cc:606] T 0edbf034be4a41d79cbce7d0c876132d P 49e15cd7009e4d6198f9d9352a27e9d1 [term 2 LEADER]: Received request to transfer leadership to TS 49598ea210614f1fbb942c1fb77c7932
I20260709 12:19:31.054118 16633 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "927ab821d939440ab1def38a7dbd0358"
dest_uuid: "49e15cd7009e4d6198f9d9352a27e9d1"
mode: GRACEFUL
new_leader_uuid: "49598ea210614f1fbb942c1fb77c7932"
 from {username='slave'} at 127.0.0.1:34778
I20260709 12:19:31.054623 16633 raft_consensus.cc:606] T 927ab821d939440ab1def38a7dbd0358 P 49e15cd7009e4d6198f9d9352a27e9d1 [term 2 LEADER]: Received request to transfer leadership to TS 49598ea210614f1fbb942c1fb77c7932
I20260709 12:19:31.061544 16783 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "dd736bd544c746238c77320b4a5706e0"
dest_uuid: "3e772c31b50d4c6693644565ef92de04"
mode: GRACEFUL
new_leader_uuid: "49598ea210614f1fbb942c1fb77c7932"
 from {username='slave'} at 127.0.0.1:52756
I20260709 12:19:31.062304 16783 raft_consensus.cc:606] T dd736bd544c746238c77320b4a5706e0 P 3e772c31b50d4c6693644565ef92de04 [term 3 LEADER]: Received request to transfer leadership to TS 49598ea210614f1fbb942c1fb77c7932
I20260709 12:19:31.070602 16982 raft_consensus.cc:993] T 0edbf034be4a41d79cbce7d0c876132d P 49e15cd7009e4d6198f9d9352a27e9d1: : Instructing follower 49598ea210614f1fbb942c1fb77c7932 to start an election
I20260709 12:19:31.071812 16976 raft_consensus.cc:1081] T 0edbf034be4a41d79cbce7d0c876132d P 49e15cd7009e4d6198f9d9352a27e9d1 [term 2 LEADER]: Signalling peer 49598ea210614f1fbb942c1fb77c7932 to start an election
I20260709 12:19:31.071868 16633 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "2ef5967933364223bef5315de3fa6bfe"
dest_uuid: "49e15cd7009e4d6198f9d9352a27e9d1"
mode: GRACEFUL
new_leader_uuid: "49598ea210614f1fbb942c1fb77c7932"
 from {username='slave'} at 127.0.0.1:34778
I20260709 12:19:31.072965 16633 raft_consensus.cc:606] T 2ef5967933364223bef5315de3fa6bfe P 49e15cd7009e4d6198f9d9352a27e9d1 [term 1 LEADER]: Received request to transfer leadership to TS 49598ea210614f1fbb942c1fb77c7932
I20260709 12:19:31.074602 16710 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "0edbf034be4a41d79cbce7d0c876132d"
dest_uuid: "49598ea210614f1fbb942c1fb77c7932"
 from {username='slave'} at 127.0.0.1:33766
I20260709 12:19:31.075250 16710 raft_consensus.cc:493] T 0edbf034be4a41d79cbce7d0c876132d P 49598ea210614f1fbb942c1fb77c7932 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:19:31.076151 16710 raft_consensus.cc:3060] T 0edbf034be4a41d79cbce7d0c876132d P 49598ea210614f1fbb942c1fb77c7932 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:19:31.081008 16783 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "0086bd356ad04503ad306b7217280c34"
dest_uuid: "3e772c31b50d4c6693644565ef92de04"
mode: GRACEFUL
new_leader_uuid: "49598ea210614f1fbb942c1fb77c7932"
 from {username='slave'} at 127.0.0.1:52756
I20260709 12:19:31.081713 16783 raft_consensus.cc:606] T 0086bd356ad04503ad306b7217280c34 P 3e772c31b50d4c6693644565ef92de04 [term 3 LEADER]: Received request to transfer leadership to TS 49598ea210614f1fbb942c1fb77c7932
I20260709 12:19:31.085331 16710 raft_consensus.cc:515] T 0edbf034be4a41d79cbce7d0c876132d P 49598ea210614f1fbb942c1fb77c7932 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } } peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } }
I20260709 12:19:31.089371 16710 leader_election.cc:290] T 0edbf034be4a41d79cbce7d0c876132d P 49598ea210614f1fbb942c1fb77c7932 [CANDIDATE]: Term 3 election: Requested vote from peers 3e772c31b50d4c6693644565ef92de04 (127.16.29.67:35049), 49e15cd7009e4d6198f9d9352a27e9d1 (127.16.29.65:44833)
I20260709 12:19:31.090204 16783 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0edbf034be4a41d79cbce7d0c876132d" candidate_uuid: "49598ea210614f1fbb942c1fb77c7932" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "3e772c31b50d4c6693644565ef92de04"
I20260709 12:19:31.091033 16783 raft_consensus.cc:3060] T 0edbf034be4a41d79cbce7d0c876132d P 3e772c31b50d4c6693644565ef92de04 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:19:31.099457 16783 raft_consensus.cc:2468] T 0edbf034be4a41d79cbce7d0c876132d P 3e772c31b50d4c6693644565ef92de04 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49598ea210614f1fbb942c1fb77c7932 in term 3.
I20260709 12:19:31.100689 16633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0edbf034be4a41d79cbce7d0c876132d" candidate_uuid: "49598ea210614f1fbb942c1fb77c7932" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "49e15cd7009e4d6198f9d9352a27e9d1"
I20260709 12:19:31.102077 16633 raft_consensus.cc:3055] T 0edbf034be4a41d79cbce7d0c876132d P 49e15cd7009e4d6198f9d9352a27e9d1 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:19:31.102525 16633 raft_consensus.cc:740] T 0edbf034be4a41d79cbce7d0c876132d P 49e15cd7009e4d6198f9d9352a27e9d1 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 49e15cd7009e4d6198f9d9352a27e9d1, State: Running, Role: LEADER
I20260709 12:19:31.103600 16633 consensus_queue.cc:260] T 0edbf034be4a41d79cbce7d0c876132d P 49e15cd7009e4d6198f9d9352a27e9d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } } peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } }
I20260709 12:19:31.104959 16633 raft_consensus.cc:3060] T 0edbf034be4a41d79cbce7d0c876132d P 49e15cd7009e4d6198f9d9352a27e9d1 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:19:31.111766 16674 leader_election.cc:304] T 0edbf034be4a41d79cbce7d0c876132d P 49598ea210614f1fbb942c1fb77c7932 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e772c31b50d4c6693644565ef92de04, 49598ea210614f1fbb942c1fb77c7932; no voters: 
I20260709 12:19:31.112725 16973 raft_consensus.cc:2804] T 0edbf034be4a41d79cbce7d0c876132d P 49598ea210614f1fbb942c1fb77c7932 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:19:31.113862 16973 raft_consensus.cc:697] T 0edbf034be4a41d79cbce7d0c876132d P 49598ea210614f1fbb942c1fb77c7932 [term 3 LEADER]: Becoming Leader. State: Replica: 49598ea210614f1fbb942c1fb77c7932, State: Running, Role: LEADER
I20260709 12:19:31.114058 16633 raft_consensus.cc:2468] T 0edbf034be4a41d79cbce7d0c876132d P 49e15cd7009e4d6198f9d9352a27e9d1 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49598ea210614f1fbb942c1fb77c7932 in term 3.
I20260709 12:19:31.114859 16973 consensus_queue.cc:237] T 0edbf034be4a41d79cbce7d0c876132d P 49598ea210614f1fbb942c1fb77c7932 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } } peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } }
I20260709 12:19:31.125074 16532 catalog_manager.cc:5697] T 0edbf034be4a41d79cbce7d0c876132d P 49598ea210614f1fbb942c1fb77c7932 reported cstate change: term changed from 2 to 3, leader changed from 49e15cd7009e4d6198f9d9352a27e9d1 (127.16.29.65) to 49598ea210614f1fbb942c1fb77c7932 (127.16.29.66). New cstate: current_term: 3 leader_uuid: "49598ea210614f1fbb942c1fb77c7932" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } health_report { overall_health: UNKNOWN } } }
I20260709 12:19:31.268318 16989 raft_consensus.cc:993] T dd736bd544c746238c77320b4a5706e0 P 3e772c31b50d4c6693644565ef92de04: : Instructing follower 49598ea210614f1fbb942c1fb77c7932 to start an election
I20260709 12:19:31.268893 16972 raft_consensus.cc:1081] T dd736bd544c746238c77320b4a5706e0 P 3e772c31b50d4c6693644565ef92de04 [term 3 LEADER]: Signalling peer 49598ea210614f1fbb942c1fb77c7932 to start an election
I20260709 12:19:31.270397 16710 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "dd736bd544c746238c77320b4a5706e0"
dest_uuid: "49598ea210614f1fbb942c1fb77c7932"
 from {username='slave'} at 127.0.0.1:33756
I20260709 12:19:31.270849 16710 raft_consensus.cc:493] T dd736bd544c746238c77320b4a5706e0 P 49598ea210614f1fbb942c1fb77c7932 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:19:31.271153 16710 raft_consensus.cc:3060] T dd736bd544c746238c77320b4a5706e0 P 49598ea210614f1fbb942c1fb77c7932 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:19:31.275600 16710 raft_consensus.cc:515] T dd736bd544c746238c77320b4a5706e0 P 49598ea210614f1fbb942c1fb77c7932 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } } peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } }
I20260709 12:19:31.277403 16710 leader_election.cc:290] T dd736bd544c746238c77320b4a5706e0 P 49598ea210614f1fbb942c1fb77c7932 [CANDIDATE]: Term 4 election: Requested vote from peers 49e15cd7009e4d6198f9d9352a27e9d1 (127.16.29.65:44833), 3e772c31b50d4c6693644565ef92de04 (127.16.29.67:35049)
I20260709 12:19:31.278182 16633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd736bd544c746238c77320b4a5706e0" candidate_uuid: "49598ea210614f1fbb942c1fb77c7932" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "49e15cd7009e4d6198f9d9352a27e9d1"
I20260709 12:19:31.278398 16783 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd736bd544c746238c77320b4a5706e0" candidate_uuid: "49598ea210614f1fbb942c1fb77c7932" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "3e772c31b50d4c6693644565ef92de04"
I20260709 12:19:31.278860 16633 raft_consensus.cc:3060] T dd736bd544c746238c77320b4a5706e0 P 49e15cd7009e4d6198f9d9352a27e9d1 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:19:31.278975 16783 raft_consensus.cc:3055] T dd736bd544c746238c77320b4a5706e0 P 3e772c31b50d4c6693644565ef92de04 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:19:31.279351 16783 raft_consensus.cc:740] T dd736bd544c746238c77320b4a5706e0 P 3e772c31b50d4c6693644565ef92de04 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 3e772c31b50d4c6693644565ef92de04, State: Running, Role: LEADER
I20260709 12:19:31.280146 16783 consensus_queue.cc:260] T dd736bd544c746238c77320b4a5706e0 P 3e772c31b50d4c6693644565ef92de04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } } peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } }
I20260709 12:19:31.281051 16783 raft_consensus.cc:3060] T dd736bd544c746238c77320b4a5706e0 P 3e772c31b50d4c6693644565ef92de04 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:19:31.285729 16783 raft_consensus.cc:2468] T dd736bd544c746238c77320b4a5706e0 P 3e772c31b50d4c6693644565ef92de04 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49598ea210614f1fbb942c1fb77c7932 in term 4.
I20260709 12:19:31.286494 16633 raft_consensus.cc:2468] T dd736bd544c746238c77320b4a5706e0 P 49e15cd7009e4d6198f9d9352a27e9d1 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49598ea210614f1fbb942c1fb77c7932 in term 4.
I20260709 12:19:31.286805 16674 leader_election.cc:304] T dd736bd544c746238c77320b4a5706e0 P 49598ea210614f1fbb942c1fb77c7932 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e772c31b50d4c6693644565ef92de04, 49598ea210614f1fbb942c1fb77c7932; no voters: 
I20260709 12:19:31.287602 16973 raft_consensus.cc:2804] T dd736bd544c746238c77320b4a5706e0 P 49598ea210614f1fbb942c1fb77c7932 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:19:31.288115 16973 raft_consensus.cc:697] T dd736bd544c746238c77320b4a5706e0 P 49598ea210614f1fbb942c1fb77c7932 [term 4 LEADER]: Becoming Leader. State: Replica: 49598ea210614f1fbb942c1fb77c7932, State: Running, Role: LEADER
I20260709 12:19:31.289057 16973 consensus_queue.cc:237] T dd736bd544c746238c77320b4a5706e0 P 49598ea210614f1fbb942c1fb77c7932 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } } peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } }
I20260709 12:19:31.297516 16532 catalog_manager.cc:5697] T dd736bd544c746238c77320b4a5706e0 P 49598ea210614f1fbb942c1fb77c7932 reported cstate change: term changed from 3 to 4, leader changed from 3e772c31b50d4c6693644565ef92de04 (127.16.29.67) to 49598ea210614f1fbb942c1fb77c7932 (127.16.29.66). New cstate: current_term: 4 leader_uuid: "49598ea210614f1fbb942c1fb77c7932" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } health_report { overall_health: UNKNOWN } } }
I20260709 12:19:31.402307 16982 raft_consensus.cc:993] T 2ef5967933364223bef5315de3fa6bfe P 49e15cd7009e4d6198f9d9352a27e9d1: : Instructing follower 49598ea210614f1fbb942c1fb77c7932 to start an election
I20260709 12:19:31.403762 16976 raft_consensus.cc:1081] T 2ef5967933364223bef5315de3fa6bfe P 49e15cd7009e4d6198f9d9352a27e9d1 [term 1 LEADER]: Signalling peer 49598ea210614f1fbb942c1fb77c7932 to start an election
I20260709 12:19:31.406133 16707 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "2ef5967933364223bef5315de3fa6bfe"
dest_uuid: "49598ea210614f1fbb942c1fb77c7932"
 from {username='slave'} at 127.0.0.1:33766
I20260709 12:19:31.406715 16707 raft_consensus.cc:493] T 2ef5967933364223bef5315de3fa6bfe P 49598ea210614f1fbb942c1fb77c7932 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:19:31.407096 16707 raft_consensus.cc:3060] T 2ef5967933364223bef5315de3fa6bfe P 49598ea210614f1fbb942c1fb77c7932 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:19:31.407334 16976 raft_consensus.cc:993] T 927ab821d939440ab1def38a7dbd0358 P 49e15cd7009e4d6198f9d9352a27e9d1: : Instructing follower 49598ea210614f1fbb942c1fb77c7932 to start an election
I20260709 12:19:31.407860 16976 raft_consensus.cc:1081] T 927ab821d939440ab1def38a7dbd0358 P 49e15cd7009e4d6198f9d9352a27e9d1 [term 2 LEADER]: Signalling peer 49598ea210614f1fbb942c1fb77c7932 to start an election
I20260709 12:19:31.410629 16710 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "927ab821d939440ab1def38a7dbd0358"
dest_uuid: "49598ea210614f1fbb942c1fb77c7932"
 from {username='slave'} at 127.0.0.1:33766
I20260709 12:19:31.411120 16710 raft_consensus.cc:493] T 927ab821d939440ab1def38a7dbd0358 P 49598ea210614f1fbb942c1fb77c7932 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:19:31.411435 16710 raft_consensus.cc:3060] T 927ab821d939440ab1def38a7dbd0358 P 49598ea210614f1fbb942c1fb77c7932 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:19:31.415645 16710 raft_consensus.cc:515] T 927ab821d939440ab1def38a7dbd0358 P 49598ea210614f1fbb942c1fb77c7932 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } } peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } }
I20260709 12:19:31.415513 16707 raft_consensus.cc:515] T 2ef5967933364223bef5315de3fa6bfe P 49598ea210614f1fbb942c1fb77c7932 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } } peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } }
I20260709 12:19:31.418052 16707 leader_election.cc:290] T 2ef5967933364223bef5315de3fa6bfe P 49598ea210614f1fbb942c1fb77c7932 [CANDIDATE]: Term 2 election: Requested vote from peers 49e15cd7009e4d6198f9d9352a27e9d1 (127.16.29.65:44833), 3e772c31b50d4c6693644565ef92de04 (127.16.29.67:35049)
I20260709 12:19:31.418608 16783 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "927ab821d939440ab1def38a7dbd0358" candidate_uuid: "49598ea210614f1fbb942c1fb77c7932" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "3e772c31b50d4c6693644565ef92de04"
I20260709 12:19:31.419205 16783 raft_consensus.cc:3060] T 927ab821d939440ab1def38a7dbd0358 P 3e772c31b50d4c6693644565ef92de04 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:19:31.419430 16633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "927ab821d939440ab1def38a7dbd0358" candidate_uuid: "49598ea210614f1fbb942c1fb77c7932" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "49e15cd7009e4d6198f9d9352a27e9d1"
I20260709 12:19:31.419988 16633 raft_consensus.cc:3055] T 927ab821d939440ab1def38a7dbd0358 P 49e15cd7009e4d6198f9d9352a27e9d1 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:19:31.420296 16633 raft_consensus.cc:740] T 927ab821d939440ab1def38a7dbd0358 P 49e15cd7009e4d6198f9d9352a27e9d1 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 49e15cd7009e4d6198f9d9352a27e9d1, State: Running, Role: LEADER
I20260709 12:19:31.420248 16634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ef5967933364223bef5315de3fa6bfe" candidate_uuid: "49598ea210614f1fbb942c1fb77c7932" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "49e15cd7009e4d6198f9d9352a27e9d1"
I20260709 12:19:31.421346 16634 raft_consensus.cc:3055] T 2ef5967933364223bef5315de3fa6bfe P 49e15cd7009e4d6198f9d9352a27e9d1 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:19:31.421738 16634 raft_consensus.cc:740] T 2ef5967933364223bef5315de3fa6bfe P 49e15cd7009e4d6198f9d9352a27e9d1 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 49e15cd7009e4d6198f9d9352a27e9d1, State: Running, Role: LEADER
I20260709 12:19:31.421535 16633 consensus_queue.cc:260] T 927ab821d939440ab1def38a7dbd0358 P 49e15cd7009e4d6198f9d9352a27e9d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } } peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } }
I20260709 12:19:31.422387 16785 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ef5967933364223bef5315de3fa6bfe" candidate_uuid: "49598ea210614f1fbb942c1fb77c7932" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "3e772c31b50d4c6693644565ef92de04"
I20260709 12:19:31.422765 16633 raft_consensus.cc:3060] T 927ab821d939440ab1def38a7dbd0358 P 49e15cd7009e4d6198f9d9352a27e9d1 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:19:31.422942 16785 raft_consensus.cc:3060] T 2ef5967933364223bef5315de3fa6bfe P 3e772c31b50d4c6693644565ef92de04 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:19:31.422571 16634 consensus_queue.cc:260] T 2ef5967933364223bef5315de3fa6bfe P 49e15cd7009e4d6198f9d9352a27e9d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } } peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } }
I20260709 12:19:31.423996 16634 raft_consensus.cc:3060] T 2ef5967933364223bef5315de3fa6bfe P 49e15cd7009e4d6198f9d9352a27e9d1 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:19:31.429251 16633 raft_consensus.cc:2468] T 927ab821d939440ab1def38a7dbd0358 P 49e15cd7009e4d6198f9d9352a27e9d1 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49598ea210614f1fbb942c1fb77c7932 in term 3.
I20260709 12:19:31.430433 16674 leader_election.cc:304] T 927ab821d939440ab1def38a7dbd0358 P 49598ea210614f1fbb942c1fb77c7932 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49598ea210614f1fbb942c1fb77c7932, 49e15cd7009e4d6198f9d9352a27e9d1; no voters: 
I20260709 12:19:31.431216 16973 raft_consensus.cc:2804] T 927ab821d939440ab1def38a7dbd0358 P 49598ea210614f1fbb942c1fb77c7932 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:19:31.431721 16973 raft_consensus.cc:697] T 927ab821d939440ab1def38a7dbd0358 P 49598ea210614f1fbb942c1fb77c7932 [term 3 LEADER]: Becoming Leader. State: Replica: 49598ea210614f1fbb942c1fb77c7932, State: Running, Role: LEADER
I20260709 12:19:31.432907 16785 raft_consensus.cc:2468] T 2ef5967933364223bef5315de3fa6bfe P 3e772c31b50d4c6693644565ef92de04 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49598ea210614f1fbb942c1fb77c7932 in term 2.
I20260709 12:19:31.432662 16973 consensus_queue.cc:237] T 927ab821d939440ab1def38a7dbd0358 P 49598ea210614f1fbb942c1fb77c7932 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } } peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } }
I20260709 12:19:31.417608 16710 leader_election.cc:290] T 927ab821d939440ab1def38a7dbd0358 P 49598ea210614f1fbb942c1fb77c7932 [CANDIDATE]: Term 3 election: Requested vote from peers 49e15cd7009e4d6198f9d9352a27e9d1 (127.16.29.65:44833), 3e772c31b50d4c6693644565ef92de04 (127.16.29.67:35049)
I20260709 12:19:31.433351 16783 raft_consensus.cc:2468] T 927ab821d939440ab1def38a7dbd0358 P 3e772c31b50d4c6693644565ef92de04 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49598ea210614f1fbb942c1fb77c7932 in term 3.
I20260709 12:19:31.434152 16674 leader_election.cc:304] T 2ef5967933364223bef5315de3fa6bfe P 49598ea210614f1fbb942c1fb77c7932 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e772c31b50d4c6693644565ef92de04, 49598ea210614f1fbb942c1fb77c7932; no voters: 
I20260709 12:19:31.435254 16983 raft_consensus.cc:2804] T 2ef5967933364223bef5315de3fa6bfe P 49598ea210614f1fbb942c1fb77c7932 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:19:31.436021 16634 raft_consensus.cc:2468] T 2ef5967933364223bef5315de3fa6bfe P 49e15cd7009e4d6198f9d9352a27e9d1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49598ea210614f1fbb942c1fb77c7932 in term 2.
I20260709 12:19:31.439163 16983 raft_consensus.cc:697] T 2ef5967933364223bef5315de3fa6bfe P 49598ea210614f1fbb942c1fb77c7932 [term 2 LEADER]: Becoming Leader. State: Replica: 49598ea210614f1fbb942c1fb77c7932, State: Running, Role: LEADER
I20260709 12:19:31.440157 16983 consensus_queue.cc:237] T 2ef5967933364223bef5315de3fa6bfe P 49598ea210614f1fbb942c1fb77c7932 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } } peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } }
I20260709 12:19:31.443603 16532 catalog_manager.cc:5697] T 927ab821d939440ab1def38a7dbd0358 P 49598ea210614f1fbb942c1fb77c7932 reported cstate change: term changed from 2 to 3, leader changed from 49e15cd7009e4d6198f9d9352a27e9d1 (127.16.29.65) to 49598ea210614f1fbb942c1fb77c7932 (127.16.29.66). New cstate: current_term: 3 leader_uuid: "49598ea210614f1fbb942c1fb77c7932" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } health_report { overall_health: UNKNOWN } } }
I20260709 12:19:31.455111 16532 catalog_manager.cc:5697] T 2ef5967933364223bef5315de3fa6bfe P 49598ea210614f1fbb942c1fb77c7932 reported cstate change: term changed from 1 to 2, leader changed from 49e15cd7009e4d6198f9d9352a27e9d1 (127.16.29.65) to 49598ea210614f1fbb942c1fb77c7932 (127.16.29.66). New cstate: current_term: 2 leader_uuid: "49598ea210614f1fbb942c1fb77c7932" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } health_report { overall_health: UNKNOWN } } }
I20260709 12:19:31.485749 16989 raft_consensus.cc:993] T 0086bd356ad04503ad306b7217280c34 P 3e772c31b50d4c6693644565ef92de04: : Instructing follower 49598ea210614f1fbb942c1fb77c7932 to start an election
I20260709 12:19:31.486330 16972 raft_consensus.cc:1081] T 0086bd356ad04503ad306b7217280c34 P 3e772c31b50d4c6693644565ef92de04 [term 3 LEADER]: Signalling peer 49598ea210614f1fbb942c1fb77c7932 to start an election
I20260709 12:19:31.487851 16710 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "0086bd356ad04503ad306b7217280c34"
dest_uuid: "49598ea210614f1fbb942c1fb77c7932"
 from {username='slave'} at 127.0.0.1:33756
I20260709 12:19:31.488543 16710 raft_consensus.cc:493] T 0086bd356ad04503ad306b7217280c34 P 49598ea210614f1fbb942c1fb77c7932 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:19:31.488962 16710 raft_consensus.cc:3060] T 0086bd356ad04503ad306b7217280c34 P 49598ea210614f1fbb942c1fb77c7932 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:19:31.496539 16710 raft_consensus.cc:515] T 0086bd356ad04503ad306b7217280c34 P 49598ea210614f1fbb942c1fb77c7932 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } } peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } }
I20260709 12:19:31.498740 16710 leader_election.cc:290] T 0086bd356ad04503ad306b7217280c34 P 49598ea210614f1fbb942c1fb77c7932 [CANDIDATE]: Term 4 election: Requested vote from peers 49e15cd7009e4d6198f9d9352a27e9d1 (127.16.29.65:44833), 3e772c31b50d4c6693644565ef92de04 (127.16.29.67:35049)
I20260709 12:19:31.499636 16634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0086bd356ad04503ad306b7217280c34" candidate_uuid: "49598ea210614f1fbb942c1fb77c7932" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "49e15cd7009e4d6198f9d9352a27e9d1"
I20260709 12:19:31.499845 16783 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0086bd356ad04503ad306b7217280c34" candidate_uuid: "49598ea210614f1fbb942c1fb77c7932" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "3e772c31b50d4c6693644565ef92de04"
I20260709 12:19:31.500443 16783 raft_consensus.cc:3055] T 0086bd356ad04503ad306b7217280c34 P 3e772c31b50d4c6693644565ef92de04 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:19:31.500590 16634 raft_consensus.cc:3060] T 0086bd356ad04503ad306b7217280c34 P 49e15cd7009e4d6198f9d9352a27e9d1 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:19:31.500847 16783 raft_consensus.cc:740] T 0086bd356ad04503ad306b7217280c34 P 3e772c31b50d4c6693644565ef92de04 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 3e772c31b50d4c6693644565ef92de04, State: Running, Role: LEADER
I20260709 12:19:31.501783 16783 consensus_queue.cc:260] T 0086bd356ad04503ad306b7217280c34 P 3e772c31b50d4c6693644565ef92de04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } } peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } }
I20260709 12:19:31.503108 16783 raft_consensus.cc:3060] T 0086bd356ad04503ad306b7217280c34 P 3e772c31b50d4c6693644565ef92de04 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:19:31.507916 16634 raft_consensus.cc:2468] T 0086bd356ad04503ad306b7217280c34 P 49e15cd7009e4d6198f9d9352a27e9d1 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49598ea210614f1fbb942c1fb77c7932 in term 4.
I20260709 12:19:31.509167 16674 leader_election.cc:304] T 0086bd356ad04503ad306b7217280c34 P 49598ea210614f1fbb942c1fb77c7932 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49598ea210614f1fbb942c1fb77c7932, 49e15cd7009e4d6198f9d9352a27e9d1; no voters: 
I20260709 12:19:31.509965 16983 raft_consensus.cc:2804] T 0086bd356ad04503ad306b7217280c34 P 49598ea210614f1fbb942c1fb77c7932 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:19:31.510164 16783 raft_consensus.cc:2468] T 0086bd356ad04503ad306b7217280c34 P 3e772c31b50d4c6693644565ef92de04 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49598ea210614f1fbb942c1fb77c7932 in term 4.
I20260709 12:19:31.510528 16983 raft_consensus.cc:697] T 0086bd356ad04503ad306b7217280c34 P 49598ea210614f1fbb942c1fb77c7932 [term 4 LEADER]: Becoming Leader. State: Replica: 49598ea210614f1fbb942c1fb77c7932, State: Running, Role: LEADER
I20260709 12:19:31.511767 16983 consensus_queue.cc:237] T 0086bd356ad04503ad306b7217280c34 P 49598ea210614f1fbb942c1fb77c7932 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } } peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } }
I20260709 12:19:31.520599 16532 catalog_manager.cc:5697] T 0086bd356ad04503ad306b7217280c34 P 49598ea210614f1fbb942c1fb77c7932 reported cstate change: term changed from 3 to 4, leader changed from 3e772c31b50d4c6693644565ef92de04 (127.16.29.67) to 49598ea210614f1fbb942c1fb77c7932 (127.16.29.66). New cstate: current_term: 4 leader_uuid: "49598ea210614f1fbb942c1fb77c7932" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } health_report { overall_health: UNKNOWN } } }
I20260709 12:19:31.528604 16634 raft_consensus.cc:1275] T 0edbf034be4a41d79cbce7d0c876132d P 49e15cd7009e4d6198f9d9352a27e9d1 [term 3 FOLLOWER]: Refusing update from remote peer 49598ea210614f1fbb942c1fb77c7932: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:19:31.530099 16983 consensus_queue.cc:1048] T 0edbf034be4a41d79cbce7d0c876132d P 49598ea210614f1fbb942c1fb77c7932 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:19:31.548292 16783 raft_consensus.cc:1275] T 0edbf034be4a41d79cbce7d0c876132d P 3e772c31b50d4c6693644565ef92de04 [term 3 FOLLOWER]: Refusing update from remote peer 49598ea210614f1fbb942c1fb77c7932: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:19:31.551277 16983 consensus_queue.cc:1048] T 0edbf034be4a41d79cbce7d0c876132d P 49598ea210614f1fbb942c1fb77c7932 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:19:31.716250 16634 raft_consensus.cc:1275] T dd736bd544c746238c77320b4a5706e0 P 49e15cd7009e4d6198f9d9352a27e9d1 [term 4 FOLLOWER]: Refusing update from remote peer 49598ea210614f1fbb942c1fb77c7932: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:19:31.717481 16973 consensus_queue.cc:1048] T dd736bd544c746238c77320b4a5706e0 P 49598ea210614f1fbb942c1fb77c7932 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:19:31.728901 16785 raft_consensus.cc:1275] T dd736bd544c746238c77320b4a5706e0 P 3e772c31b50d4c6693644565ef92de04 [term 4 FOLLOWER]: Refusing update from remote peer 49598ea210614f1fbb942c1fb77c7932: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:19:31.732434 16992 consensus_queue.cc:1048] T dd736bd544c746238c77320b4a5706e0 P 49598ea210614f1fbb942c1fb77c7932 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:19:31.923075 16785 raft_consensus.cc:1275] T 927ab821d939440ab1def38a7dbd0358 P 3e772c31b50d4c6693644565ef92de04 [term 3 FOLLOWER]: Refusing update from remote peer 49598ea210614f1fbb942c1fb77c7932: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:19:31.924558 16992 consensus_queue.cc:1048] T 927ab821d939440ab1def38a7dbd0358 P 49598ea210614f1fbb942c1fb77c7932 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:19:31.939116 16634 raft_consensus.cc:1275] T 927ab821d939440ab1def38a7dbd0358 P 49e15cd7009e4d6198f9d9352a27e9d1 [term 3 FOLLOWER]: Refusing update from remote peer 49598ea210614f1fbb942c1fb77c7932: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:19:31.940994 16973 consensus_queue.cc:1048] T 927ab821d939440ab1def38a7dbd0358 P 49598ea210614f1fbb942c1fb77c7932 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:19:31.968107 16633 raft_consensus.cc:1275] T 0086bd356ad04503ad306b7217280c34 P 49e15cd7009e4d6198f9d9352a27e9d1 [term 4 FOLLOWER]: Refusing update from remote peer 49598ea210614f1fbb942c1fb77c7932: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:19:31.969336 16973 consensus_queue.cc:1048] T 0086bd356ad04503ad306b7217280c34 P 49598ea210614f1fbb942c1fb77c7932 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:19:31.979468 16785 raft_consensus.cc:1275] T 0086bd356ad04503ad306b7217280c34 P 3e772c31b50d4c6693644565ef92de04 [term 4 FOLLOWER]: Refusing update from remote peer 49598ea210614f1fbb942c1fb77c7932: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:19:31.982568 16785 raft_consensus.cc:1275] T 2ef5967933364223bef5315de3fa6bfe P 3e772c31b50d4c6693644565ef92de04 [term 2 FOLLOWER]: Refusing update from remote peer 49598ea210614f1fbb942c1fb77c7932: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:19:31.984825 16973 consensus_queue.cc:1048] T 0086bd356ad04503ad306b7217280c34 P 49598ea210614f1fbb942c1fb77c7932 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:19:31.985056 16992 consensus_queue.cc:1048] T 2ef5967933364223bef5315de3fa6bfe P 49598ea210614f1fbb942c1fb77c7932 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:19:32.005800 16633 raft_consensus.cc:1275] T 2ef5967933364223bef5315de3fa6bfe P 49e15cd7009e4d6198f9d9352a27e9d1 [term 2 FOLLOWER]: Refusing update from remote peer 49598ea210614f1fbb942c1fb77c7932: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:19:32.007784 16992 consensus_queue.cc:1048] T 2ef5967933364223bef5315de3fa6bfe P 49598ea210614f1fbb942c1fb77c7932 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:19:34.143460 16501 auto_leader_rebalancer-test.cc:328] The leader distribution is 
I20260709 12:19:34.144481 16501 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:19:34.152976 16710 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "0edbf034be4a41d79cbce7d0c876132d"
dest_uuid: "49598ea210614f1fbb942c1fb77c7932"
mode: GRACEFUL
new_leader_uuid: "3e772c31b50d4c6693644565ef92de04"
 from {username='slave'} at 127.0.0.1:33772
I20260709 12:19:34.153846 16710 raft_consensus.cc:606] T 0edbf034be4a41d79cbce7d0c876132d P 49598ea210614f1fbb942c1fb77c7932 [term 3 LEADER]: Received request to transfer leadership to TS 3e772c31b50d4c6693644565ef92de04
I20260709 12:19:34.156994 16710 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "3692a4a710ef469aa412d9870427fb63"
dest_uuid: "49598ea210614f1fbb942c1fb77c7932"
mode: GRACEFUL
new_leader_uuid: "3e772c31b50d4c6693644565ef92de04"
 from {username='slave'} at 127.0.0.1:33772
I20260709 12:19:34.157704 16710 raft_consensus.cc:606] T 3692a4a710ef469aa412d9870427fb63 P 49598ea210614f1fbb942c1fb77c7932 [term 2 LEADER]: Received request to transfer leadership to TS 3e772c31b50d4c6693644565ef92de04
I20260709 12:19:34.160499 16710 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "927ab821d939440ab1def38a7dbd0358"
dest_uuid: "49598ea210614f1fbb942c1fb77c7932"
mode: GRACEFUL
new_leader_uuid: "49e15cd7009e4d6198f9d9352a27e9d1"
 from {username='slave'} at 127.0.0.1:33772
I20260709 12:19:34.161038 16710 raft_consensus.cc:606] T 927ab821d939440ab1def38a7dbd0358 P 49598ea210614f1fbb942c1fb77c7932 [term 3 LEADER]: Received request to transfer leadership to TS 49e15cd7009e4d6198f9d9352a27e9d1
I20260709 12:19:34.163674 16710 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "95f027ce38784f81bf5c3b35b3357173"
dest_uuid: "49598ea210614f1fbb942c1fb77c7932"
mode: GRACEFUL
new_leader_uuid: "49e15cd7009e4d6198f9d9352a27e9d1"
 from {username='slave'} at 127.0.0.1:33772
I20260709 12:19:34.164391 16710 raft_consensus.cc:606] T 95f027ce38784f81bf5c3b35b3357173 P 49598ea210614f1fbb942c1fb77c7932 [term 2 LEADER]: Received request to transfer leadership to TS 49e15cd7009e4d6198f9d9352a27e9d1
I20260709 12:19:34.166903 16710 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "dd736bd544c746238c77320b4a5706e0"
dest_uuid: "49598ea210614f1fbb942c1fb77c7932"
mode: GRACEFUL
new_leader_uuid: "49e15cd7009e4d6198f9d9352a27e9d1"
 from {username='slave'} at 127.0.0.1:33772
I20260709 12:19:34.167418 16710 raft_consensus.cc:606] T dd736bd544c746238c77320b4a5706e0 P 49598ea210614f1fbb942c1fb77c7932 [term 4 LEADER]: Received request to transfer leadership to TS 49e15cd7009e4d6198f9d9352a27e9d1
I20260709 12:19:34.168697 16501 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260709 12:19:34.169224 16501 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:19:34.289606 17012 raft_consensus.cc:993] T 927ab821d939440ab1def38a7dbd0358 P 49598ea210614f1fbb942c1fb77c7932: : Instructing follower 49e15cd7009e4d6198f9d9352a27e9d1 to start an election
I20260709 12:19:34.290249 17012 raft_consensus.cc:1081] T 927ab821d939440ab1def38a7dbd0358 P 49598ea210614f1fbb942c1fb77c7932 [term 3 LEADER]: Signalling peer 49e15cd7009e4d6198f9d9352a27e9d1 to start an election
I20260709 12:19:34.291702 16633 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "927ab821d939440ab1def38a7dbd0358"
dest_uuid: "49e15cd7009e4d6198f9d9352a27e9d1"
 from {username='slave'} at 127.0.0.1:34760
I20260709 12:19:34.292191 16633 raft_consensus.cc:493] T 927ab821d939440ab1def38a7dbd0358 P 49e15cd7009e4d6198f9d9352a27e9d1 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:19:34.292464 16633 raft_consensus.cc:3060] T 927ab821d939440ab1def38a7dbd0358 P 49e15cd7009e4d6198f9d9352a27e9d1 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:19:34.297351 16633 raft_consensus.cc:515] T 927ab821d939440ab1def38a7dbd0358 P 49e15cd7009e4d6198f9d9352a27e9d1 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } } peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } }
I20260709 12:19:34.299702 16633 leader_election.cc:290] T 927ab821d939440ab1def38a7dbd0358 P 49e15cd7009e4d6198f9d9352a27e9d1 [CANDIDATE]: Term 4 election: Requested vote from peers 49598ea210614f1fbb942c1fb77c7932 (127.16.29.66:35871), 3e772c31b50d4c6693644565ef92de04 (127.16.29.67:35049)
I20260709 12:19:34.301138 16783 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "927ab821d939440ab1def38a7dbd0358" candidate_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "3e772c31b50d4c6693644565ef92de04"
I20260709 12:19:34.301069 16710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "927ab821d939440ab1def38a7dbd0358" candidate_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "49598ea210614f1fbb942c1fb77c7932"
I20260709 12:19:34.302022 16783 raft_consensus.cc:3060] T 927ab821d939440ab1def38a7dbd0358 P 3e772c31b50d4c6693644565ef92de04 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:19:34.302054 16710 raft_consensus.cc:3055] T 927ab821d939440ab1def38a7dbd0358 P 49598ea210614f1fbb942c1fb77c7932 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:19:34.302557 16710 raft_consensus.cc:740] T 927ab821d939440ab1def38a7dbd0358 P 49598ea210614f1fbb942c1fb77c7932 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 49598ea210614f1fbb942c1fb77c7932, State: Running, Role: LEADER
I20260709 12:19:34.303671 16710 consensus_queue.cc:260] T 927ab821d939440ab1def38a7dbd0358 P 49598ea210614f1fbb942c1fb77c7932 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } } peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } }
I20260709 12:19:34.305083 16710 raft_consensus.cc:3060] T 927ab821d939440ab1def38a7dbd0358 P 49598ea210614f1fbb942c1fb77c7932 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:19:34.307590 16783 raft_consensus.cc:2468] T 927ab821d939440ab1def38a7dbd0358 P 3e772c31b50d4c6693644565ef92de04 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49e15cd7009e4d6198f9d9352a27e9d1 in term 4.
I20260709 12:19:34.309276 16599 leader_election.cc:304] T 927ab821d939440ab1def38a7dbd0358 P 49e15cd7009e4d6198f9d9352a27e9d1 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e772c31b50d4c6693644565ef92de04, 49e15cd7009e4d6198f9d9352a27e9d1; no voters: 
I20260709 12:19:34.310398 16710 raft_consensus.cc:2468] T 927ab821d939440ab1def38a7dbd0358 P 49598ea210614f1fbb942c1fb77c7932 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49e15cd7009e4d6198f9d9352a27e9d1 in term 4.
I20260709 12:19:34.311120 17013 raft_consensus.cc:2804] T 927ab821d939440ab1def38a7dbd0358 P 49e15cd7009e4d6198f9d9352a27e9d1 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:19:34.312350 17013 raft_consensus.cc:697] T 927ab821d939440ab1def38a7dbd0358 P 49e15cd7009e4d6198f9d9352a27e9d1 [term 4 LEADER]: Becoming Leader. State: Replica: 49e15cd7009e4d6198f9d9352a27e9d1, State: Running, Role: LEADER
I20260709 12:19:34.313350 17013 consensus_queue.cc:237] T 927ab821d939440ab1def38a7dbd0358 P 49e15cd7009e4d6198f9d9352a27e9d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } } peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } }
I20260709 12:19:34.320573 16532 catalog_manager.cc:5697] T 927ab821d939440ab1def38a7dbd0358 P 49e15cd7009e4d6198f9d9352a27e9d1 reported cstate change: term changed from 3 to 4, leader changed from 49598ea210614f1fbb942c1fb77c7932 (127.16.29.66) to 49e15cd7009e4d6198f9d9352a27e9d1 (127.16.29.65). New cstate: current_term: 4 leader_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } health_report { overall_health: UNKNOWN } } }
I20260709 12:19:34.329048 16992 raft_consensus.cc:993] T 0edbf034be4a41d79cbce7d0c876132d P 49598ea210614f1fbb942c1fb77c7932: : Instructing follower 3e772c31b50d4c6693644565ef92de04 to start an election
I20260709 12:19:34.329711 17012 raft_consensus.cc:1081] T 0edbf034be4a41d79cbce7d0c876132d P 49598ea210614f1fbb942c1fb77c7932 [term 3 LEADER]: Signalling peer 3e772c31b50d4c6693644565ef92de04 to start an election
I20260709 12:19:34.331967 16783 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "0edbf034be4a41d79cbce7d0c876132d"
dest_uuid: "3e772c31b50d4c6693644565ef92de04"
 from {username='slave'} at 127.0.0.1:52742
I20260709 12:19:34.332527 16783 raft_consensus.cc:493] T 0edbf034be4a41d79cbce7d0c876132d P 3e772c31b50d4c6693644565ef92de04 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:19:34.332810 16783 raft_consensus.cc:3060] T 0edbf034be4a41d79cbce7d0c876132d P 3e772c31b50d4c6693644565ef92de04 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:19:34.338361 16783 raft_consensus.cc:515] T 0edbf034be4a41d79cbce7d0c876132d P 3e772c31b50d4c6693644565ef92de04 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } } peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } }
I20260709 12:19:34.340407 16783 leader_election.cc:290] T 0edbf034be4a41d79cbce7d0c876132d P 3e772c31b50d4c6693644565ef92de04 [CANDIDATE]: Term 4 election: Requested vote from peers 49598ea210614f1fbb942c1fb77c7932 (127.16.29.66:35871), 49e15cd7009e4d6198f9d9352a27e9d1 (127.16.29.65:44833)
I20260709 12:19:34.341428 16710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0edbf034be4a41d79cbce7d0c876132d" candidate_uuid: "3e772c31b50d4c6693644565ef92de04" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "49598ea210614f1fbb942c1fb77c7932"
I20260709 12:19:34.341536 16633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0edbf034be4a41d79cbce7d0c876132d" candidate_uuid: "3e772c31b50d4c6693644565ef92de04" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "49e15cd7009e4d6198f9d9352a27e9d1"
I20260709 12:19:34.342049 16710 raft_consensus.cc:3055] T 0edbf034be4a41d79cbce7d0c876132d P 49598ea210614f1fbb942c1fb77c7932 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:19:34.342109 16633 raft_consensus.cc:3060] T 0edbf034be4a41d79cbce7d0c876132d P 49e15cd7009e4d6198f9d9352a27e9d1 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:19:34.342389 16710 raft_consensus.cc:740] T 0edbf034be4a41d79cbce7d0c876132d P 49598ea210614f1fbb942c1fb77c7932 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 49598ea210614f1fbb942c1fb77c7932, State: Running, Role: LEADER
I20260709 12:19:34.343266 16710 consensus_queue.cc:260] T 0edbf034be4a41d79cbce7d0c876132d P 49598ea210614f1fbb942c1fb77c7932 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } } peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } }
I20260709 12:19:34.344290 16710 raft_consensus.cc:3060] T 0edbf034be4a41d79cbce7d0c876132d P 49598ea210614f1fbb942c1fb77c7932 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:19:34.346948 16633 raft_consensus.cc:2468] T 0edbf034be4a41d79cbce7d0c876132d P 49e15cd7009e4d6198f9d9352a27e9d1 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e772c31b50d4c6693644565ef92de04 in term 4.
I20260709 12:19:34.348178 16748 leader_election.cc:304] T 0edbf034be4a41d79cbce7d0c876132d P 3e772c31b50d4c6693644565ef92de04 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e772c31b50d4c6693644565ef92de04, 49e15cd7009e4d6198f9d9352a27e9d1; no voters: 
I20260709 12:19:34.349350 17007 raft_consensus.cc:2804] T 0edbf034be4a41d79cbce7d0c876132d P 3e772c31b50d4c6693644565ef92de04 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:19:34.350214 16710 raft_consensus.cc:2468] T 0edbf034be4a41d79cbce7d0c876132d P 49598ea210614f1fbb942c1fb77c7932 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e772c31b50d4c6693644565ef92de04 in term 4.
I20260709 12:19:34.350837 17007 raft_consensus.cc:697] T 0edbf034be4a41d79cbce7d0c876132d P 3e772c31b50d4c6693644565ef92de04 [term 4 LEADER]: Becoming Leader. State: Replica: 3e772c31b50d4c6693644565ef92de04, State: Running, Role: LEADER
I20260709 12:19:34.351984 17007 consensus_queue.cc:237] T 0edbf034be4a41d79cbce7d0c876132d P 3e772c31b50d4c6693644565ef92de04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } } peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } }
I20260709 12:19:34.353940 16992 raft_consensus.cc:993] T dd736bd544c746238c77320b4a5706e0 P 49598ea210614f1fbb942c1fb77c7932: : Instructing follower 49e15cd7009e4d6198f9d9352a27e9d1 to start an election
I20260709 12:19:34.354372 17012 raft_consensus.cc:1081] T dd736bd544c746238c77320b4a5706e0 P 49598ea210614f1fbb942c1fb77c7932 [term 4 LEADER]: Signalling peer 49e15cd7009e4d6198f9d9352a27e9d1 to start an election
I20260709 12:19:34.356272 16633 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "dd736bd544c746238c77320b4a5706e0"
dest_uuid: "49e15cd7009e4d6198f9d9352a27e9d1"
 from {username='slave'} at 127.0.0.1:34760
I20260709 12:19:34.356992 16633 raft_consensus.cc:493] T dd736bd544c746238c77320b4a5706e0 P 49e15cd7009e4d6198f9d9352a27e9d1 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:19:34.357401 16633 raft_consensus.cc:3060] T dd736bd544c746238c77320b4a5706e0 P 49e15cd7009e4d6198f9d9352a27e9d1 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:19:34.361269 16532 catalog_manager.cc:5697] T 0edbf034be4a41d79cbce7d0c876132d P 3e772c31b50d4c6693644565ef92de04 reported cstate change: term changed from 3 to 4, leader changed from 49598ea210614f1fbb942c1fb77c7932 (127.16.29.66) to 3e772c31b50d4c6693644565ef92de04 (127.16.29.67). New cstate: current_term: 4 leader_uuid: "3e772c31b50d4c6693644565ef92de04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } health_report { overall_health: UNKNOWN } } }
I20260709 12:19:34.363845 16633 raft_consensus.cc:515] T dd736bd544c746238c77320b4a5706e0 P 49e15cd7009e4d6198f9d9352a27e9d1 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } } peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } }
I20260709 12:19:34.365737 16633 leader_election.cc:290] T dd736bd544c746238c77320b4a5706e0 P 49e15cd7009e4d6198f9d9352a27e9d1 [CANDIDATE]: Term 5 election: Requested vote from peers 49598ea210614f1fbb942c1fb77c7932 (127.16.29.66:35871), 3e772c31b50d4c6693644565ef92de04 (127.16.29.67:35049)
I20260709 12:19:34.367070 16783 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd736bd544c746238c77320b4a5706e0" candidate_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "3e772c31b50d4c6693644565ef92de04"
I20260709 12:19:34.367666 16783 raft_consensus.cc:3060] T dd736bd544c746238c77320b4a5706e0 P 3e772c31b50d4c6693644565ef92de04 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:19:34.367108 16710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd736bd544c746238c77320b4a5706e0" candidate_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "49598ea210614f1fbb942c1fb77c7932"
I20260709 12:19:34.368481 16710 raft_consensus.cc:3055] T dd736bd544c746238c77320b4a5706e0 P 49598ea210614f1fbb942c1fb77c7932 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:19:34.368808 16710 raft_consensus.cc:740] T dd736bd544c746238c77320b4a5706e0 P 49598ea210614f1fbb942c1fb77c7932 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 49598ea210614f1fbb942c1fb77c7932, State: Running, Role: LEADER
I20260709 12:19:34.369668 16710 consensus_queue.cc:260] T dd736bd544c746238c77320b4a5706e0 P 49598ea210614f1fbb942c1fb77c7932 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } } peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } }
I20260709 12:19:34.371421 16710 raft_consensus.cc:3060] T dd736bd544c746238c77320b4a5706e0 P 49598ea210614f1fbb942c1fb77c7932 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:19:34.375488 16783 raft_consensus.cc:2468] T dd736bd544c746238c77320b4a5706e0 P 3e772c31b50d4c6693644565ef92de04 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49e15cd7009e4d6198f9d9352a27e9d1 in term 5.
I20260709 12:19:34.377060 16599 leader_election.cc:304] T dd736bd544c746238c77320b4a5706e0 P 49e15cd7009e4d6198f9d9352a27e9d1 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e772c31b50d4c6693644565ef92de04, 49e15cd7009e4d6198f9d9352a27e9d1; no voters: 
I20260709 12:19:34.378000 17013 raft_consensus.cc:2804] T dd736bd544c746238c77320b4a5706e0 P 49e15cd7009e4d6198f9d9352a27e9d1 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:19:34.378532 17013 raft_consensus.cc:697] T dd736bd544c746238c77320b4a5706e0 P 49e15cd7009e4d6198f9d9352a27e9d1 [term 5 LEADER]: Becoming Leader. State: Replica: 49e15cd7009e4d6198f9d9352a27e9d1, State: Running, Role: LEADER
I20260709 12:19:34.379530 17013 consensus_queue.cc:237] T dd736bd544c746238c77320b4a5706e0 P 49e15cd7009e4d6198f9d9352a27e9d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } } peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } }
I20260709 12:19:34.379977 16710 raft_consensus.cc:2468] T dd736bd544c746238c77320b4a5706e0 P 49598ea210614f1fbb942c1fb77c7932 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49e15cd7009e4d6198f9d9352a27e9d1 in term 5.
I20260709 12:19:34.387547 16992 raft_consensus.cc:993] T 95f027ce38784f81bf5c3b35b3357173 P 49598ea210614f1fbb942c1fb77c7932: : Instructing follower 49e15cd7009e4d6198f9d9352a27e9d1 to start an election
I20260709 12:19:34.388099 17012 raft_consensus.cc:1081] T 95f027ce38784f81bf5c3b35b3357173 P 49598ea210614f1fbb942c1fb77c7932 [term 2 LEADER]: Signalling peer 49e15cd7009e4d6198f9d9352a27e9d1 to start an election
I20260709 12:19:34.387815 16532 catalog_manager.cc:5697] T dd736bd544c746238c77320b4a5706e0 P 49e15cd7009e4d6198f9d9352a27e9d1 reported cstate change: term changed from 4 to 5, leader changed from 49598ea210614f1fbb942c1fb77c7932 (127.16.29.66) to 49e15cd7009e4d6198f9d9352a27e9d1 (127.16.29.65). New cstate: current_term: 5 leader_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } health_report { overall_health: UNKNOWN } } }
I20260709 12:19:34.390045 16633 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "95f027ce38784f81bf5c3b35b3357173"
dest_uuid: "49e15cd7009e4d6198f9d9352a27e9d1"
 from {username='slave'} at 127.0.0.1:34760
I20260709 12:19:34.390659 16633 raft_consensus.cc:493] T 95f027ce38784f81bf5c3b35b3357173 P 49e15cd7009e4d6198f9d9352a27e9d1 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:19:34.391069 16633 raft_consensus.cc:3060] T 95f027ce38784f81bf5c3b35b3357173 P 49e15cd7009e4d6198f9d9352a27e9d1 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:19:34.396677 16633 raft_consensus.cc:515] T 95f027ce38784f81bf5c3b35b3357173 P 49e15cd7009e4d6198f9d9352a27e9d1 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } } peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } }
I20260709 12:19:34.398485 16633 leader_election.cc:290] T 95f027ce38784f81bf5c3b35b3357173 P 49e15cd7009e4d6198f9d9352a27e9d1 [CANDIDATE]: Term 3 election: Requested vote from peers 49598ea210614f1fbb942c1fb77c7932 (127.16.29.66:35871), 3e772c31b50d4c6693644565ef92de04 (127.16.29.67:35049)
I20260709 12:19:34.399492 16710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f027ce38784f81bf5c3b35b3357173" candidate_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "49598ea210614f1fbb942c1fb77c7932"
I20260709 12:19:34.399616 16783 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f027ce38784f81bf5c3b35b3357173" candidate_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "3e772c31b50d4c6693644565ef92de04"
I20260709 12:19:34.400017 16710 raft_consensus.cc:3055] T 95f027ce38784f81bf5c3b35b3357173 P 49598ea210614f1fbb942c1fb77c7932 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:19:34.400235 16783 raft_consensus.cc:3060] T 95f027ce38784f81bf5c3b35b3357173 P 3e772c31b50d4c6693644565ef92de04 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:19:34.400398 16710 raft_consensus.cc:740] T 95f027ce38784f81bf5c3b35b3357173 P 49598ea210614f1fbb942c1fb77c7932 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 49598ea210614f1fbb942c1fb77c7932, State: Running, Role: LEADER
I20260709 12:19:34.401270 16710 consensus_queue.cc:260] T 95f027ce38784f81bf5c3b35b3357173 P 49598ea210614f1fbb942c1fb77c7932 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } } peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } }
I20260709 12:19:34.402246 16710 raft_consensus.cc:3060] T 95f027ce38784f81bf5c3b35b3357173 P 49598ea210614f1fbb942c1fb77c7932 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:19:34.404569 16783 raft_consensus.cc:2468] T 95f027ce38784f81bf5c3b35b3357173 P 3e772c31b50d4c6693644565ef92de04 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49e15cd7009e4d6198f9d9352a27e9d1 in term 3.
I20260709 12:19:34.405689 16599 leader_election.cc:304] T 95f027ce38784f81bf5c3b35b3357173 P 49e15cd7009e4d6198f9d9352a27e9d1 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e772c31b50d4c6693644565ef92de04, 49e15cd7009e4d6198f9d9352a27e9d1; no voters: 
I20260709 12:19:34.406667 17013 raft_consensus.cc:2804] T 95f027ce38784f81bf5c3b35b3357173 P 49e15cd7009e4d6198f9d9352a27e9d1 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:19:34.406772 16710 raft_consensus.cc:2468] T 95f027ce38784f81bf5c3b35b3357173 P 49598ea210614f1fbb942c1fb77c7932 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49e15cd7009e4d6198f9d9352a27e9d1 in term 3.
I20260709 12:19:34.407166 17013 raft_consensus.cc:697] T 95f027ce38784f81bf5c3b35b3357173 P 49e15cd7009e4d6198f9d9352a27e9d1 [term 3 LEADER]: Becoming Leader. State: Replica: 49e15cd7009e4d6198f9d9352a27e9d1, State: Running, Role: LEADER
I20260709 12:19:34.408030 17013 consensus_queue.cc:237] T 95f027ce38784f81bf5c3b35b3357173 P 49e15cd7009e4d6198f9d9352a27e9d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } } peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } }
I20260709 12:19:34.414718 16532 catalog_manager.cc:5697] T 95f027ce38784f81bf5c3b35b3357173 P 49e15cd7009e4d6198f9d9352a27e9d1 reported cstate change: term changed from 2 to 3, leader changed from 49598ea210614f1fbb942c1fb77c7932 (127.16.29.66) to 49e15cd7009e4d6198f9d9352a27e9d1 (127.16.29.65). New cstate: current_term: 3 leader_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } health_report { overall_health: UNKNOWN } } }
I20260709 12:19:34.485550 16992 raft_consensus.cc:993] T 3692a4a710ef469aa412d9870427fb63 P 49598ea210614f1fbb942c1fb77c7932: : Instructing follower 3e772c31b50d4c6693644565ef92de04 to start an election
I20260709 12:19:34.486008 17012 raft_consensus.cc:1081] T 3692a4a710ef469aa412d9870427fb63 P 49598ea210614f1fbb942c1fb77c7932 [term 2 LEADER]: Signalling peer 3e772c31b50d4c6693644565ef92de04 to start an election
I20260709 12:19:34.487450 16783 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "3692a4a710ef469aa412d9870427fb63"
dest_uuid: "3e772c31b50d4c6693644565ef92de04"
 from {username='slave'} at 127.0.0.1:52742
I20260709 12:19:34.487928 16783 raft_consensus.cc:493] T 3692a4a710ef469aa412d9870427fb63 P 3e772c31b50d4c6693644565ef92de04 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:19:34.488276 16783 raft_consensus.cc:3060] T 3692a4a710ef469aa412d9870427fb63 P 3e772c31b50d4c6693644565ef92de04 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:19:34.492751 16783 raft_consensus.cc:515] T 3692a4a710ef469aa412d9870427fb63 P 3e772c31b50d4c6693644565ef92de04 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } } peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } }
I20260709 12:19:34.494151 16783 leader_election.cc:290] T 3692a4a710ef469aa412d9870427fb63 P 3e772c31b50d4c6693644565ef92de04 [CANDIDATE]: Term 3 election: Requested vote from peers 49e15cd7009e4d6198f9d9352a27e9d1 (127.16.29.65:44833), 49598ea210614f1fbb942c1fb77c7932 (127.16.29.66:35871)
I20260709 12:19:34.495352 16633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3692a4a710ef469aa412d9870427fb63" candidate_uuid: "3e772c31b50d4c6693644565ef92de04" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "49e15cd7009e4d6198f9d9352a27e9d1"
I20260709 12:19:34.495699 16710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3692a4a710ef469aa412d9870427fb63" candidate_uuid: "3e772c31b50d4c6693644565ef92de04" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "49598ea210614f1fbb942c1fb77c7932"
I20260709 12:19:34.495935 16633 raft_consensus.cc:3060] T 3692a4a710ef469aa412d9870427fb63 P 49e15cd7009e4d6198f9d9352a27e9d1 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:19:34.496209 16710 raft_consensus.cc:3055] T 3692a4a710ef469aa412d9870427fb63 P 49598ea210614f1fbb942c1fb77c7932 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:19:34.496483 16710 raft_consensus.cc:740] T 3692a4a710ef469aa412d9870427fb63 P 49598ea210614f1fbb942c1fb77c7932 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 49598ea210614f1fbb942c1fb77c7932, State: Running, Role: LEADER
I20260709 12:19:34.497195 16710 consensus_queue.cc:260] T 3692a4a710ef469aa412d9870427fb63 P 49598ea210614f1fbb942c1fb77c7932 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } } peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } }
I20260709 12:19:34.498261 16710 raft_consensus.cc:3060] T 3692a4a710ef469aa412d9870427fb63 P 49598ea210614f1fbb942c1fb77c7932 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:19:34.501891 16633 raft_consensus.cc:2468] T 3692a4a710ef469aa412d9870427fb63 P 49e15cd7009e4d6198f9d9352a27e9d1 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e772c31b50d4c6693644565ef92de04 in term 3.
I20260709 12:19:34.503049 16748 leader_election.cc:304] T 3692a4a710ef469aa412d9870427fb63 P 3e772c31b50d4c6693644565ef92de04 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e772c31b50d4c6693644565ef92de04, 49e15cd7009e4d6198f9d9352a27e9d1; no voters: 
I20260709 12:19:34.503830 17007 raft_consensus.cc:2804] T 3692a4a710ef469aa412d9870427fb63 P 3e772c31b50d4c6693644565ef92de04 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:19:34.503847 16710 raft_consensus.cc:2468] T 3692a4a710ef469aa412d9870427fb63 P 49598ea210614f1fbb942c1fb77c7932 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e772c31b50d4c6693644565ef92de04 in term 3.
I20260709 12:19:34.504467 17007 raft_consensus.cc:697] T 3692a4a710ef469aa412d9870427fb63 P 3e772c31b50d4c6693644565ef92de04 [term 3 LEADER]: Becoming Leader. State: Replica: 3e772c31b50d4c6693644565ef92de04, State: Running, Role: LEADER
I20260709 12:19:34.505245 17007 consensus_queue.cc:237] T 3692a4a710ef469aa412d9870427fb63 P 3e772c31b50d4c6693644565ef92de04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } } peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } }
I20260709 12:19:34.511955 16532 catalog_manager.cc:5697] T 3692a4a710ef469aa412d9870427fb63 P 3e772c31b50d4c6693644565ef92de04 reported cstate change: term changed from 2 to 3, leader changed from 49598ea210614f1fbb942c1fb77c7932 (127.16.29.66) to 3e772c31b50d4c6693644565ef92de04 (127.16.29.67). New cstate: current_term: 3 leader_uuid: "3e772c31b50d4c6693644565ef92de04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 } health_report { overall_health: HEALTHY } } }
I20260709 12:19:34.726249 16710 raft_consensus.cc:1275] T 927ab821d939440ab1def38a7dbd0358 P 49598ea210614f1fbb942c1fb77c7932 [term 4 FOLLOWER]: Refusing update from remote peer 49e15cd7009e4d6198f9d9352a27e9d1: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:19:34.727764 17013 consensus_queue.cc:1048] T 927ab821d939440ab1def38a7dbd0358 P 49e15cd7009e4d6198f9d9352a27e9d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:19:34.734057 16783 raft_consensus.cc:1275] T 927ab821d939440ab1def38a7dbd0358 P 3e772c31b50d4c6693644565ef92de04 [term 4 FOLLOWER]: Refusing update from remote peer 49e15cd7009e4d6198f9d9352a27e9d1: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:19:34.736105 17013 consensus_queue.cc:1048] T 927ab821d939440ab1def38a7dbd0358 P 49e15cd7009e4d6198f9d9352a27e9d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:19:34.770099 16710 raft_consensus.cc:1275] T dd736bd544c746238c77320b4a5706e0 P 49598ea210614f1fbb942c1fb77c7932 [term 5 FOLLOWER]: Refusing update from remote peer 49e15cd7009e4d6198f9d9352a27e9d1: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260709 12:19:34.771693 17014 consensus_queue.cc:1048] T dd736bd544c746238c77320b4a5706e0 P 49e15cd7009e4d6198f9d9352a27e9d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:19:34.781399 16783 raft_consensus.cc:1275] T dd736bd544c746238c77320b4a5706e0 P 3e772c31b50d4c6693644565ef92de04 [term 5 FOLLOWER]: Refusing update from remote peer 49e15cd7009e4d6198f9d9352a27e9d1: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260709 12:19:34.784133 17013 consensus_queue.cc:1048] T dd736bd544c746238c77320b4a5706e0 P 49e15cd7009e4d6198f9d9352a27e9d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:19:34.886765 16710 raft_consensus.cc:1275] T 3692a4a710ef469aa412d9870427fb63 P 49598ea210614f1fbb942c1fb77c7932 [term 3 FOLLOWER]: Refusing update from remote peer 3e772c31b50d4c6693644565ef92de04: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:19:34.888206 17007 consensus_queue.cc:1048] T 3692a4a710ef469aa412d9870427fb63 P 3e772c31b50d4c6693644565ef92de04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:19:34.891776 16783 raft_consensus.cc:1275] T 95f027ce38784f81bf5c3b35b3357173 P 3e772c31b50d4c6693644565ef92de04 [term 3 FOLLOWER]: Refusing update from remote peer 49e15cd7009e4d6198f9d9352a27e9d1: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:19:34.893245 17013 consensus_queue.cc:1048] T 95f027ce38784f81bf5c3b35b3357173 P 49e15cd7009e4d6198f9d9352a27e9d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e772c31b50d4c6693644565ef92de04" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 35049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:19:34.898317 16633 raft_consensus.cc:1275] T 3692a4a710ef469aa412d9870427fb63 P 49e15cd7009e4d6198f9d9352a27e9d1 [term 3 FOLLOWER]: Refusing update from remote peer 3e772c31b50d4c6693644565ef92de04: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:19:34.900136 17007 consensus_queue.cc:1048] T 3692a4a710ef469aa412d9870427fb63 P 3e772c31b50d4c6693644565ef92de04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:19:34.910998 16710 raft_consensus.cc:1275] T 95f027ce38784f81bf5c3b35b3357173 P 49598ea210614f1fbb942c1fb77c7932 [term 3 FOLLOWER]: Refusing update from remote peer 49e15cd7009e4d6198f9d9352a27e9d1: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:19:34.912671 17013 consensus_queue.cc:1048] T 95f027ce38784f81bf5c3b35b3357173 P 49e15cd7009e4d6198f9d9352a27e9d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:19:34.919059 16633 raft_consensus.cc:1275] T 0edbf034be4a41d79cbce7d0c876132d P 49e15cd7009e4d6198f9d9352a27e9d1 [term 4 FOLLOWER]: Refusing update from remote peer 3e772c31b50d4c6693644565ef92de04: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:19:34.921011 17034 consensus_queue.cc:1048] T 0edbf034be4a41d79cbce7d0c876132d P 3e772c31b50d4c6693644565ef92de04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49e15cd7009e4d6198f9d9352a27e9d1" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:19:34.933946 16710 raft_consensus.cc:1275] T 0edbf034be4a41d79cbce7d0c876132d P 49598ea210614f1fbb942c1fb77c7932 [term 4 FOLLOWER]: Refusing update from remote peer 3e772c31b50d4c6693644565ef92de04: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:19:34.936249 17007 consensus_queue.cc:1048] T 0edbf034be4a41d79cbce7d0c876132d P 3e772c31b50d4c6693644565ef92de04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49598ea210614f1fbb942c1fb77c7932" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 35871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:19:35.170548 16501 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:19:35.176800 16501 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:19:35.177209 16501 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:19:36.178143 16501 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:19:36.182942 16501 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:19:36.183341 16501 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:19:37.184787 16501 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:19:37.190063 16501 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:19:37.190480 16501 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:19:38.191583 16501 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:19:38.197369 16501 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:19:38.197801 16501 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:19:39.198958 16501 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:19:39.205183 16501 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:19:39.205649 16501 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:19:40.206763 16501 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:19:40.212422 16501 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:19:40.212885 16501 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:19:41.213925 16501 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:19:41.218863 16501 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:19:41.219267 16501 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:19:42.243261 16501 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:19:42.260878 16501 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:19:42.261452 16501 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:19:43.262900 16501 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:19:43.269305 16501 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:19:43.269832 16501 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:19:44.299633 16501 auto_leader_rebalancer-test.cc:340] The leader distribution is 
I20260709 12:19:44.300243 16501 tablet_server.cc:179] TabletServer@127.16.29.65:0 shutting down...
I20260709 12:19:44.334894 16501 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:19:44.335594 16501 tablet_replica.cc:333] T 5c3e9fafdd8142eea33e771756209147 P 49e15cd7009e4d6198f9d9352a27e9d1: stopping tablet replica
I20260709 12:19:44.336551 16501 raft_consensus.cc:2243] T 5c3e9fafdd8142eea33e771756209147 P 49e15cd7009e4d6198f9d9352a27e9d1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:19:44.337215 16501 raft_consensus.cc:2272] T 5c3e9fafdd8142eea33e771756209147 P 49e15cd7009e4d6198f9d9352a27e9d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:19:44.340253 16501 tablet_replica.cc:333] T 95f027ce38784f81bf5c3b35b3357173 P 49e15cd7009e4d6198f9d9352a27e9d1: stopping tablet replica
I20260709 12:19:44.340855 16501 raft_consensus.cc:2243] T 95f027ce38784f81bf5c3b35b3357173 P 49e15cd7009e4d6198f9d9352a27e9d1 [term 3 LEADER]: Raft consensus shutting down.
I20260709 12:19:44.341835 16501 raft_consensus.cc:2272] T 95f027ce38784f81bf5c3b35b3357173 P 49e15cd7009e4d6198f9d9352a27e9d1 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:19:44.344053 16501 tablet_replica.cc:333] T 0edbf034be4a41d79cbce7d0c876132d P 49e15cd7009e4d6198f9d9352a27e9d1: stopping tablet replica
I20260709 12:19:44.344635 16501 raft_consensus.cc:2243] T 0edbf034be4a41d79cbce7d0c876132d P 49e15cd7009e4d6198f9d9352a27e9d1 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 12:19:44.345167 16501 raft_consensus.cc:2272] T 0edbf034be4a41d79cbce7d0c876132d P 49e15cd7009e4d6198f9d9352a27e9d1 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:19:44.347056 16501 tablet_replica.cc:333] T 0086bd356ad04503ad306b7217280c34 P 49e15cd7009e4d6198f9d9352a27e9d1: stopping tablet replica
I20260709 12:19:44.347682 16501 raft_consensus.cc:2243] T 0086bd356ad04503ad306b7217280c34 P 49e15cd7009e4d6198f9d9352a27e9d1 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 12:19:44.348198 16501 raft_consensus.cc:2272] T 0086bd356ad04503ad306b7217280c34 P 49e15cd7009e4d6198f9d9352a27e9d1 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:19:44.350155 16501 tablet_replica.cc:333] T af674c5ffb4e44dab6f15ab84dbeb737 P 49e15cd7009e4d6198f9d9352a27e9d1: stopping tablet replica
I20260709 12:19:44.350666 16501 raft_consensus.cc:2243] T af674c5ffb4e44dab6f15ab84dbeb737 P 49e15cd7009e4d6198f9d9352a27e9d1 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:19:44.351233 16501 raft_consensus.cc:2272] T af674c5ffb4e44dab6f15ab84dbeb737 P 49e15cd7009e4d6198f9d9352a27e9d1 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:19:44.353094 16501 tablet_replica.cc:333] T 927ab821d939440ab1def38a7dbd0358 P 49e15cd7009e4d6198f9d9352a27e9d1: stopping tablet replica
I20260709 12:19:44.353634 16501 raft_consensus.cc:2243] T 927ab821d939440ab1def38a7dbd0358 P 49e15cd7009e4d6198f9d9352a27e9d1 [term 4 LEADER]: Raft consensus shutting down.
I20260709 12:19:44.354710 16501 raft_consensus.cc:2272] T 927ab821d939440ab1def38a7dbd0358 P 49e15cd7009e4d6198f9d9352a27e9d1 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:19:44.356729 16501 tablet_replica.cc:333] T 2ef5967933364223bef5315de3fa6bfe P 49e15cd7009e4d6198f9d9352a27e9d1: stopping tablet replica
I20260709 12:19:44.357352 16501 raft_consensus.cc:2243] T 2ef5967933364223bef5315de3fa6bfe P 49e15cd7009e4d6198f9d9352a27e9d1 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:19:44.357821 16501 raft_consensus.cc:2272] T 2ef5967933364223bef5315de3fa6bfe P 49e15cd7009e4d6198f9d9352a27e9d1 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:19:44.360306 16501 tablet_replica.cc:333] T dd736bd544c746238c77320b4a5706e0 P 49e15cd7009e4d6198f9d9352a27e9d1: stopping tablet replica
I20260709 12:19:44.360927 16501 raft_consensus.cc:2243] T dd736bd544c746238c77320b4a5706e0 P 49e15cd7009e4d6198f9d9352a27e9d1 [term 5 LEADER]: Raft consensus shutting down.
I20260709 12:19:44.361809 16501 raft_consensus.cc:2272] T dd736bd544c746238c77320b4a5706e0 P 49e15cd7009e4d6198f9d9352a27e9d1 [term 5 FOLLOWER]: Raft consensus is shut down!
I20260709 12:19:44.363972 16501 tablet_replica.cc:333] T 3692a4a710ef469aa412d9870427fb63 P 49e15cd7009e4d6198f9d9352a27e9d1: stopping tablet replica
I20260709 12:19:44.364451 16501 raft_consensus.cc:2243] T 3692a4a710ef469aa412d9870427fb63 P 49e15cd7009e4d6198f9d9352a27e9d1 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:19:44.364923 16501 raft_consensus.cc:2272] T 3692a4a710ef469aa412d9870427fb63 P 49e15cd7009e4d6198f9d9352a27e9d1 [term 3 FOLLOWER]: Raft consensus is shut down!
W20260709 12:19:44.370574 16674 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:19:44.374804 16674 consensus_peers.cc:597] T 2ef5967933364223bef5315de3fa6bfe P 49598ea210614f1fbb942c1fb77c7932 -> Peer 49e15cd7009e4d6198f9d9352a27e9d1 (127.16.29.65:44833): Couldn't send request to peer 49e15cd7009e4d6198f9d9352a27e9d1. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:19:44.377008 16674 consensus_peers.cc:597] T af674c5ffb4e44dab6f15ab84dbeb737 P 49598ea210614f1fbb942c1fb77c7932 -> Peer 49e15cd7009e4d6198f9d9352a27e9d1 (127.16.29.65:44833): Couldn't send request to peer 49e15cd7009e4d6198f9d9352a27e9d1. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:19:44.464798 16674 consensus_peers.cc:597] T 0086bd356ad04503ad306b7217280c34 P 49598ea210614f1fbb942c1fb77c7932 -> Peer 49e15cd7009e4d6198f9d9352a27e9d1 (127.16.29.65:44833): Couldn't send request to peer 49e15cd7009e4d6198f9d9352a27e9d1. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:44833: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:19:44.481163 16501 tablet_server.cc:196] TabletServer@127.16.29.65:0 shutdown complete.
W20260709 12:19:44.487118 16748 consensus_peers.cc:597] T 3692a4a710ef469aa412d9870427fb63 P 3e772c31b50d4c6693644565ef92de04 -> Peer 49e15cd7009e4d6198f9d9352a27e9d1 (127.16.29.65:44833): Couldn't send request to peer 49e15cd7009e4d6198f9d9352a27e9d1. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:44833: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:19:44.502195 16501 tablet_server.cc:179] TabletServer@127.16.29.66:0 shutting down...
I20260709 12:19:44.533190 16501 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:19:44.533856 16501 tablet_replica.cc:333] T 5c3e9fafdd8142eea33e771756209147 P 49598ea210614f1fbb942c1fb77c7932: stopping tablet replica
I20260709 12:19:44.534552 16501 raft_consensus.cc:2243] T 5c3e9fafdd8142eea33e771756209147 P 49598ea210614f1fbb942c1fb77c7932 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:19:44.535037 16501 raft_consensus.cc:2272] T 5c3e9fafdd8142eea33e771756209147 P 49598ea210614f1fbb942c1fb77c7932 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:19:44.536891 16501 tablet_replica.cc:333] T dd736bd544c746238c77320b4a5706e0 P 49598ea210614f1fbb942c1fb77c7932: stopping tablet replica
I20260709 12:19:44.537439 16501 raft_consensus.cc:2243] T dd736bd544c746238c77320b4a5706e0 P 49598ea210614f1fbb942c1fb77c7932 [term 5 FOLLOWER]: Raft consensus shutting down.
I20260709 12:19:44.537875 16501 raft_consensus.cc:2272] T dd736bd544c746238c77320b4a5706e0 P 49598ea210614f1fbb942c1fb77c7932 [term 5 FOLLOWER]: Raft consensus is shut down!
I20260709 12:19:44.539537 16501 tablet_replica.cc:333] T 3692a4a710ef469aa412d9870427fb63 P 49598ea210614f1fbb942c1fb77c7932: stopping tablet replica
I20260709 12:19:44.540071 16501 raft_consensus.cc:2243] T 3692a4a710ef469aa412d9870427fb63 P 49598ea210614f1fbb942c1fb77c7932 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:19:44.540515 16501 raft_consensus.cc:2272] T 3692a4a710ef469aa412d9870427fb63 P 49598ea210614f1fbb942c1fb77c7932 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:19:44.542848 16501 tablet_replica.cc:333] T 95f027ce38784f81bf5c3b35b3357173 P 49598ea210614f1fbb942c1fb77c7932: stopping tablet replica
I20260709 12:19:44.543357 16501 raft_consensus.cc:2243] T 95f027ce38784f81bf5c3b35b3357173 P 49598ea210614f1fbb942c1fb77c7932 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:19:44.543813 16501 raft_consensus.cc:2272] T 95f027ce38784f81bf5c3b35b3357173 P 49598ea210614f1fbb942c1fb77c7932 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:19:44.545801 16501 tablet_replica.cc:333] T 0086bd356ad04503ad306b7217280c34 P 49598ea210614f1fbb942c1fb77c7932: stopping tablet replica
I20260709 12:19:44.546324 16501 raft_consensus.cc:2243] T 0086bd356ad04503ad306b7217280c34 P 49598ea210614f1fbb942c1fb77c7932 [term 4 LEADER]: Raft consensus shutting down.
I20260709 12:19:44.547258 16501 raft_consensus.cc:2272] T 0086bd356ad04503ad306b7217280c34 P 49598ea210614f1fbb942c1fb77c7932 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:19:44.549146 16501 tablet_replica.cc:333] T 0edbf034be4a41d79cbce7d0c876132d P 49598ea210614f1fbb942c1fb77c7932: stopping tablet replica
I20260709 12:19:44.549656 16501 raft_consensus.cc:2243] T 0edbf034be4a41d79cbce7d0c876132d P 49598ea210614f1fbb942c1fb77c7932 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 12:19:44.549985 16501 raft_consensus.cc:2272] T 0edbf034be4a41d79cbce7d0c876132d P 49598ea210614f1fbb942c1fb77c7932 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:19:44.551548 16501 tablet_replica.cc:333] T af674c5ffb4e44dab6f15ab84dbeb737 P 49598ea210614f1fbb942c1fb77c7932: stopping tablet replica
I20260709 12:19:44.552016 16501 raft_consensus.cc:2243] T af674c5ffb4e44dab6f15ab84dbeb737 P 49598ea210614f1fbb942c1fb77c7932 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:19:44.552675 16501 raft_consensus.cc:2272] T af674c5ffb4e44dab6f15ab84dbeb737 P 49598ea210614f1fbb942c1fb77c7932 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:19:44.554153 16501 tablet_replica.cc:333] T 2ef5967933364223bef5315de3fa6bfe P 49598ea210614f1fbb942c1fb77c7932: stopping tablet replica
I20260709 12:19:44.554541 16501 raft_consensus.cc:2243] T 2ef5967933364223bef5315de3fa6bfe P 49598ea210614f1fbb942c1fb77c7932 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:19:44.555197 16501 raft_consensus.cc:2272] T 2ef5967933364223bef5315de3fa6bfe P 49598ea210614f1fbb942c1fb77c7932 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:19:44.556815 16501 tablet_replica.cc:333] T 927ab821d939440ab1def38a7dbd0358 P 49598ea210614f1fbb942c1fb77c7932: stopping tablet replica
I20260709 12:19:44.557180 16501 raft_consensus.cc:2243] T 927ab821d939440ab1def38a7dbd0358 P 49598ea210614f1fbb942c1fb77c7932 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 12:19:44.557603 16501 raft_consensus.cc:2272] T 927ab821d939440ab1def38a7dbd0358 P 49598ea210614f1fbb942c1fb77c7932 [term 4 FOLLOWER]: Raft consensus is shut down!
W20260709 12:19:44.571928 16749 consensus_peers.cc:597] T 5c3e9fafdd8142eea33e771756209147 P 3e772c31b50d4c6693644565ef92de04 -> Peer 49598ea210614f1fbb942c1fb77c7932 (127.16.29.66:35871): Couldn't send request to peer 49598ea210614f1fbb942c1fb77c7932. Status: Network error: recv got EOF from unknown peer (error 108). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:19:44.606596 16501 tablet_server.cc:196] TabletServer@127.16.29.66:0 shutdown complete.
W20260709 12:19:44.621235 16749 consensus_peers.cc:597] T 3692a4a710ef469aa412d9870427fb63 P 3e772c31b50d4c6693644565ef92de04 -> Peer 49598ea210614f1fbb942c1fb77c7932 (127.16.29.66:35871): Couldn't send request to peer 49598ea210614f1fbb942c1fb77c7932. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.66:35871: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:19:44.627729 16501 tablet_server.cc:179] TabletServer@127.16.29.67:0 shutting down...
W20260709 12:19:44.637564 16748 consensus_peers.cc:597] T 5c3e9fafdd8142eea33e771756209147 P 3e772c31b50d4c6693644565ef92de04 -> Peer 49e15cd7009e4d6198f9d9352a27e9d1 (127.16.29.65:44833): Couldn't send request to peer 49e15cd7009e4d6198f9d9352a27e9d1. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:44833: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:19:44.653092 16501 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:19:44.653839 16501 tablet_replica.cc:333] T 5c3e9fafdd8142eea33e771756209147 P 3e772c31b50d4c6693644565ef92de04: stopping tablet replica
I20260709 12:19:44.654824 16501 raft_consensus.cc:2243] T 5c3e9fafdd8142eea33e771756209147 P 3e772c31b50d4c6693644565ef92de04 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:19:44.655766 16501 raft_consensus.cc:2272] T 5c3e9fafdd8142eea33e771756209147 P 3e772c31b50d4c6693644565ef92de04 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:19:44.657686 16501 tablet_replica.cc:333] T 3692a4a710ef469aa412d9870427fb63 P 3e772c31b50d4c6693644565ef92de04: stopping tablet replica
I20260709 12:19:44.658135 16501 raft_consensus.cc:2243] T 3692a4a710ef469aa412d9870427fb63 P 3e772c31b50d4c6693644565ef92de04 [term 3 LEADER]: Raft consensus shutting down.
I20260709 12:19:44.659085 16501 raft_consensus.cc:2272] T 3692a4a710ef469aa412d9870427fb63 P 3e772c31b50d4c6693644565ef92de04 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:19:44.661090 16501 tablet_replica.cc:333] T 95f027ce38784f81bf5c3b35b3357173 P 3e772c31b50d4c6693644565ef92de04: stopping tablet replica
I20260709 12:19:44.661612 16501 raft_consensus.cc:2243] T 95f027ce38784f81bf5c3b35b3357173 P 3e772c31b50d4c6693644565ef92de04 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:19:44.662075 16501 raft_consensus.cc:2272] T 95f027ce38784f81bf5c3b35b3357173 P 3e772c31b50d4c6693644565ef92de04 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:19:44.663852 16501 tablet_replica.cc:333] T 0edbf034be4a41d79cbce7d0c876132d P 3e772c31b50d4c6693644565ef92de04: stopping tablet replica
I20260709 12:19:44.664450 16501 raft_consensus.cc:2243] T 0edbf034be4a41d79cbce7d0c876132d P 3e772c31b50d4c6693644565ef92de04 [term 4 LEADER]: Raft consensus shutting down.
I20260709 12:19:44.665210 16501 raft_consensus.cc:2272] T 0edbf034be4a41d79cbce7d0c876132d P 3e772c31b50d4c6693644565ef92de04 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:19:44.666949 16501 tablet_replica.cc:333] T 0086bd356ad04503ad306b7217280c34 P 3e772c31b50d4c6693644565ef92de04: stopping tablet replica
I20260709 12:19:44.667404 16501 raft_consensus.cc:2243] T 0086bd356ad04503ad306b7217280c34 P 3e772c31b50d4c6693644565ef92de04 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 12:19:44.667788 16501 raft_consensus.cc:2272] T 0086bd356ad04503ad306b7217280c34 P 3e772c31b50d4c6693644565ef92de04 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:19:44.669550 16501 tablet_replica.cc:333] T af674c5ffb4e44dab6f15ab84dbeb737 P 3e772c31b50d4c6693644565ef92de04: stopping tablet replica
I20260709 12:19:44.670017 16501 raft_consensus.cc:2243] T af674c5ffb4e44dab6f15ab84dbeb737 P 3e772c31b50d4c6693644565ef92de04 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:19:44.670414 16501 raft_consensus.cc:2272] T af674c5ffb4e44dab6f15ab84dbeb737 P 3e772c31b50d4c6693644565ef92de04 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:19:44.672302 16501 tablet_replica.cc:333] T 927ab821d939440ab1def38a7dbd0358 P 3e772c31b50d4c6693644565ef92de04: stopping tablet replica
I20260709 12:19:44.672739 16501 raft_consensus.cc:2243] T 927ab821d939440ab1def38a7dbd0358 P 3e772c31b50d4c6693644565ef92de04 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 12:19:44.673117 16501 raft_consensus.cc:2272] T 927ab821d939440ab1def38a7dbd0358 P 3e772c31b50d4c6693644565ef92de04 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:19:44.674819 16501 tablet_replica.cc:333] T 2ef5967933364223bef5315de3fa6bfe P 3e772c31b50d4c6693644565ef92de04: stopping tablet replica
I20260709 12:19:44.675237 16501 raft_consensus.cc:2243] T 2ef5967933364223bef5315de3fa6bfe P 3e772c31b50d4c6693644565ef92de04 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:19:44.675606 16501 raft_consensus.cc:2272] T 2ef5967933364223bef5315de3fa6bfe P 3e772c31b50d4c6693644565ef92de04 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:19:44.677261 16501 tablet_replica.cc:333] T dd736bd544c746238c77320b4a5706e0 P 3e772c31b50d4c6693644565ef92de04: stopping tablet replica
I20260709 12:19:44.677673 16501 raft_consensus.cc:2243] T dd736bd544c746238c77320b4a5706e0 P 3e772c31b50d4c6693644565ef92de04 [term 5 FOLLOWER]: Raft consensus shutting down.
I20260709 12:19:44.678030 16501 raft_consensus.cc:2272] T dd736bd544c746238c77320b4a5706e0 P 3e772c31b50d4c6693644565ef92de04 [term 5 FOLLOWER]: Raft consensus is shut down!
I20260709 12:19:44.723347 16501 tablet_server.cc:196] TabletServer@127.16.29.67:0 shutdown complete.
I20260709 12:19:44.742592 16501 master.cc:562] Master@127.16.29.126:43845 shutting down...
I20260709 12:19:44.890280 16501 raft_consensus.cc:2243] T 00000000000000000000000000000000 P d2fba92c59de4f3893f9baebc856a51c [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:19:44.890913 16501 raft_consensus.cc:2272] T 00000000000000000000000000000000 P d2fba92c59de4f3893f9baebc856a51c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:19:44.891265 16501 tablet_replica.cc:333] T 00000000000000000000000000000000 P d2fba92c59de4f3893f9baebc856a51c: stopping tablet replica
I20260709 12:19:44.912846 16501 master.cc:584] Master@127.16.29.126:43845 shutdown complete.
3e772c31b50d4c6693644565ef92de04  1
49598ea210614f1fbb942c1fb77c7932  4
49e15cd7009e4d6198f9d9352a27e9d1  4
3e772c31b50d4c6693644565ef92de04  3
49598ea210614f1fbb942c1fb77c7932  3
49e15cd7009e4d6198f9d9352a27e9d1  3
3e772c31b50d4c6693644565ef92de04  1
49598ea210614f1fbb942c1fb77c7932  8
49e15cd7009e4d6198f9d9352a27e9d1  0
3e772c31b50d4c6693644565ef92de04  3
49598ea210614f1fbb942c1fb77c7932  3
49e15cd7009e4d6198f9d9352a27e9d1  3
[       OK ] LeaderRebalancerTest.FunctionalTestForDivided (32840 ms)
[ RUN      ] LeaderRebalancerTest.FunctionalTestForNotDivided
I20260709 12:19:44.957351 16501 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.16.29.126:37041
I20260709 12:19:44.958865 16501 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:19:44.964524 17059 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:19:44.965441 17060 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:19:44.966014 16501 server_base.cc:1061] running on GCE node
W20260709 12:19:44.965898 17062 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:19:44.967093 16501 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:19:44.967315 16501 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:19:44.967470 16501 hybrid_clock.cc:648] HybridClock initialized: now 1783599584967453 us; error 0 us; skew 500 ppm
I20260709 12:19:44.968043 16501 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:19:44.970387 16501 webserver.cc:533] Webserver started at http://127.16.29.126:39007/ using document root <none> and password file <none>
I20260709 12:19:44.970916 16501 fs_manager.cc:362] Metadata directory not provided
I20260709 12:19:44.971125 16501 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:19:44.971392 16501 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:19:44.972895 16501 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783599551912677-16501-0/minicluster-data/master-0-root/instance:
uuid: "8d41903a1faf49aaa6fbd5b36d9afa79"
format_stamp: "Formatted at 2026-07-09 12:19:44 on dist-test-slave-5n9v"
I20260709 12:19:44.977595 16501 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.005s	sys 0.000s
I20260709 12:19:44.981025 17067 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:19:44.981913 16501 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.001s
I20260709 12:19:44.982185 16501 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783599551912677-16501-0/minicluster-data/master-0-root
uuid: "8d41903a1faf49aaa6fbd5b36d9afa79"
format_stamp: "Formatted at 2026-07-09 12:19:44 on dist-test-slave-5n9v"
I20260709 12:19:44.982437 16501 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783599551912677-16501-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783599551912677-16501-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783599551912677-16501-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:19:44.993396 16501 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:19:44.994560 16501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:19:45.033138 16501 rpc_server.cc:307] RPC server started. Bound to: 127.16.29.126:37041
I20260709 12:19:45.033288 17118 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.29.126:37041 every 8 connection(s)
I20260709 12:19:45.037012 17119 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:19:45.046634 17119 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d41903a1faf49aaa6fbd5b36d9afa79: Bootstrap starting.
I20260709 12:19:45.051409 17119 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8d41903a1faf49aaa6fbd5b36d9afa79: Neither blocks nor log segments found. Creating new log.
I20260709 12:19:45.055642 17119 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d41903a1faf49aaa6fbd5b36d9afa79: No bootstrap required, opened a new log
I20260709 12:19:45.057883 17119 raft_consensus.cc:359] T 00000000000000000000000000000000 P 8d41903a1faf49aaa6fbd5b36d9afa79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d41903a1faf49aaa6fbd5b36d9afa79" member_type: VOTER }
I20260709 12:19:45.058314 17119 raft_consensus.cc:385] T 00000000000000000000000000000000 P 8d41903a1faf49aaa6fbd5b36d9afa79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:19:45.058501 17119 raft_consensus.cc:740] T 00000000000000000000000000000000 P 8d41903a1faf49aaa6fbd5b36d9afa79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d41903a1faf49aaa6fbd5b36d9afa79, State: Initialized, Role: FOLLOWER
I20260709 12:19:45.058983 17119 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8d41903a1faf49aaa6fbd5b36d9afa79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d41903a1faf49aaa6fbd5b36d9afa79" member_type: VOTER }
I20260709 12:19:45.059398 17119 raft_consensus.cc:399] T 00000000000000000000000000000000 P 8d41903a1faf49aaa6fbd5b36d9afa79 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260709 12:19:45.059592 17119 raft_consensus.cc:493] T 00000000000000000000000000000000 P 8d41903a1faf49aaa6fbd5b36d9afa79 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260709 12:19:45.059798 17119 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 8d41903a1faf49aaa6fbd5b36d9afa79 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:19:45.064603 17119 raft_consensus.cc:515] T 00000000000000000000000000000000 P 8d41903a1faf49aaa6fbd5b36d9afa79 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d41903a1faf49aaa6fbd5b36d9afa79" member_type: VOTER }
I20260709 12:19:45.065148 17119 leader_election.cc:304] T 00000000000000000000000000000000 P 8d41903a1faf49aaa6fbd5b36d9afa79 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8d41903a1faf49aaa6fbd5b36d9afa79; no voters: 
I20260709 12:19:45.066323 17119 leader_election.cc:290] T 00000000000000000000000000000000 P 8d41903a1faf49aaa6fbd5b36d9afa79 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20260709 12:19:45.066749 17122 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 8d41903a1faf49aaa6fbd5b36d9afa79 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:19:45.068101 17122 raft_consensus.cc:697] T 00000000000000000000000000000000 P 8d41903a1faf49aaa6fbd5b36d9afa79 [term 1 LEADER]: Becoming Leader. State: Replica: 8d41903a1faf49aaa6fbd5b36d9afa79, State: Running, Role: LEADER
I20260709 12:19:45.068917 17122 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8d41903a1faf49aaa6fbd5b36d9afa79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d41903a1faf49aaa6fbd5b36d9afa79" member_type: VOTER }
I20260709 12:19:45.070053 17119 sys_catalog.cc:565] T 00000000000000000000000000000000 P 8d41903a1faf49aaa6fbd5b36d9afa79 [sys.catalog]: configured and running, proceeding with master startup.
I20260709 12:19:45.074167 17123 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d41903a1faf49aaa6fbd5b36d9afa79 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8d41903a1faf49aaa6fbd5b36d9afa79" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d41903a1faf49aaa6fbd5b36d9afa79" member_type: VOTER } }
I20260709 12:19:45.074476 17124 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d41903a1faf49aaa6fbd5b36d9afa79 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8d41903a1faf49aaa6fbd5b36d9afa79. Latest consensus state: current_term: 1 leader_uuid: "8d41903a1faf49aaa6fbd5b36d9afa79" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d41903a1faf49aaa6fbd5b36d9afa79" member_type: VOTER } }
I20260709 12:19:45.075245 17123 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d41903a1faf49aaa6fbd5b36d9afa79 [sys.catalog]: This master's current role is: LEADER
I20260709 12:19:45.075398 17124 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d41903a1faf49aaa6fbd5b36d9afa79 [sys.catalog]: This master's current role is: LEADER
I20260709 12:19:45.078754 17128 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260709 12:19:45.084515 17128 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260709 12:19:45.088769 16501 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260709 12:19:45.094416 17128 catalog_manager.cc:1383] Generated new cluster ID: 95ff0d2fbd7e43d1bcf960302e3e03ae
I20260709 12:19:45.094861 17128 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260709 12:19:45.118656 17128 catalog_manager.cc:1406] Generated new certificate authority record
I20260709 12:19:45.119943 17128 catalog_manager.cc:1540] Loading token signing keys...
I20260709 12:19:45.134193 17128 catalog_manager.cc:6070] T 00000000000000000000000000000000 P 8d41903a1faf49aaa6fbd5b36d9afa79: Generated new TSK 0
I20260709 12:19:45.134806 17128 catalog_manager.cc:1550] Initializing in-progress tserver states...
I20260709 12:19:45.155263 16501 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:19:45.161823 17140 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:19:45.163661 17141 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:19:45.164829 17143 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:19:45.165702 16501 server_base.cc:1061] running on GCE node
I20260709 12:19:45.168216 16501 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:19:45.168414 16501 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:19:45.168552 16501 hybrid_clock.cc:648] HybridClock initialized: now 1783599585168541 us; error 0 us; skew 500 ppm
I20260709 12:19:45.169016 16501 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:19:45.171355 16501 webserver.cc:533] Webserver started at http://127.16.29.65:39069/ using document root <none> and password file <none>
I20260709 12:19:45.171851 16501 fs_manager.cc:362] Metadata directory not provided
I20260709 12:19:45.172122 16501 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:19:45.172341 16501 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:19:45.173448 16501 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783599551912677-16501-0/minicluster-data/ts-0-root/instance:
uuid: "30f3ca7002424f919b063ee77bc473de"
format_stamp: "Formatted at 2026-07-09 12:19:45 on dist-test-slave-5n9v"
I20260709 12:19:45.177639 16501 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.005s	sys 0.000s
I20260709 12:19:45.180847 17148 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:19:45.181599 16501 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.003s	sys 0.000s
I20260709 12:19:45.181854 16501 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783599551912677-16501-0/minicluster-data/ts-0-root
uuid: "30f3ca7002424f919b063ee77bc473de"
format_stamp: "Formatted at 2026-07-09 12:19:45 on dist-test-slave-5n9v"
I20260709 12:19:45.182104 16501 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783599551912677-16501-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783599551912677-16501-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783599551912677-16501-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:19:45.195658 16501 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:19:45.196904 16501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:19:45.198310 16501 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:19:45.200721 16501 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:19:45.200919 16501 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:19:45.201105 16501 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:19:45.201320 16501 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:19:45.244925 16501 rpc_server.cc:307] RPC server started. Bound to: 127.16.29.65:35609
I20260709 12:19:45.245042 17210 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.29.65:35609 every 8 connection(s)
I20260709 12:19:45.250139 16501 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:19:45.258514 17215 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:19:45.259608 17216 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:19:45.263962 16501 server_base.cc:1061] running on GCE node
W20260709 12:19:45.265230 17218 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:19:45.265795 17211 heartbeater.cc:344] Connected to a master server at 127.16.29.126:37041
I20260709 12:19:45.266261 17211 heartbeater.cc:461] Registering TS with master...
I20260709 12:19:45.266361 16501 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:19:45.266633 16501 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:19:45.266839 16501 hybrid_clock.cc:648] HybridClock initialized: now 1783599585266822 us; error 0 us; skew 500 ppm
I20260709 12:19:45.267212 17211 heartbeater.cc:507] Master 127.16.29.126:37041 requested a full tablet report, sending...
I20260709 12:19:45.267590 16501 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:19:45.269618 17084 ts_manager.cc:194] Registered new tserver with Master: 30f3ca7002424f919b063ee77bc473de (127.16.29.65:35609)
I20260709 12:19:45.270336 16501 webserver.cc:533] Webserver started at http://127.16.29.66:41043/ using document root <none> and password file <none>
I20260709 12:19:45.270860 16501 fs_manager.cc:362] Metadata directory not provided
I20260709 12:19:45.271054 16501 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:19:45.271368 16501 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:19:45.271680 17084 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:49502
I20260709 12:19:45.272702 16501 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783599551912677-16501-0/minicluster-data/ts-1-root/instance:
uuid: "71eca61062e040d48b2398afb3b0b510"
format_stamp: "Formatted at 2026-07-09 12:19:45 on dist-test-slave-5n9v"
I20260709 12:19:45.277809 16501 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.005s	sys 0.000s
I20260709 12:19:45.281463 17223 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:19:45.282351 16501 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.001s
I20260709 12:19:45.282930 16501 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783599551912677-16501-0/minicluster-data/ts-1-root
uuid: "71eca61062e040d48b2398afb3b0b510"
format_stamp: "Formatted at 2026-07-09 12:19:45 on dist-test-slave-5n9v"
I20260709 12:19:45.283447 16501 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783599551912677-16501-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783599551912677-16501-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783599551912677-16501-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:19:45.311861 16501 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:19:45.313282 16501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:19:45.314884 16501 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:19:45.317240 16501 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:19:45.317484 16501 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:19:45.317752 16501 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:19:45.317930 16501 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:19:45.358904 16501 rpc_server.cc:307] RPC server started. Bound to: 127.16.29.66:44237
I20260709 12:19:45.359014 17285 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.29.66:44237 every 8 connection(s)
I20260709 12:19:45.363510 16501 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:19:45.370314 17289 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:19:45.370561 17290 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:19:45.374750 17292 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:19:45.374946 16501 server_base.cc:1061] running on GCE node
I20260709 12:19:45.375761 17286 heartbeater.cc:344] Connected to a master server at 127.16.29.126:37041
I20260709 12:19:45.376169 17286 heartbeater.cc:461] Registering TS with master...
I20260709 12:19:45.376279 16501 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:19:45.376554 16501 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:19:45.376753 16501 hybrid_clock.cc:648] HybridClock initialized: now 1783599585376735 us; error 0 us; skew 500 ppm
I20260709 12:19:45.377084 17286 heartbeater.cc:507] Master 127.16.29.126:37041 requested a full tablet report, sending...
I20260709 12:19:45.377513 16501 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:19:45.380023 17084 ts_manager.cc:194] Registered new tserver with Master: 71eca61062e040d48b2398afb3b0b510 (127.16.29.66:44237)
I20260709 12:19:45.380596 16501 webserver.cc:533] Webserver started at http://127.16.29.67:44111/ using document root <none> and password file <none>
I20260709 12:19:45.381049 16501 fs_manager.cc:362] Metadata directory not provided
I20260709 12:19:45.381204 16501 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:19:45.381392 16501 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:19:45.381413 17084 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:49516
I20260709 12:19:45.382880 16501 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783599551912677-16501-0/minicluster-data/ts-2-root/instance:
uuid: "78285c60669649f6a5f13c1b65f9419e"
format_stamp: "Formatted at 2026-07-09 12:19:45 on dist-test-slave-5n9v"
I20260709 12:19:45.388398 16501 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.004s	sys 0.003s
I20260709 12:19:45.391809 17297 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:19:45.392724 16501 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.001s
I20260709 12:19:45.393026 16501 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783599551912677-16501-0/minicluster-data/ts-2-root
uuid: "78285c60669649f6a5f13c1b65f9419e"
format_stamp: "Formatted at 2026-07-09 12:19:45 on dist-test-slave-5n9v"
I20260709 12:19:45.393373 16501 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783599551912677-16501-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783599551912677-16501-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783599551912677-16501-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:19:45.431921 16501 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:19:45.433110 16501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:19:45.434597 16501 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:19:45.436735 16501 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:19:45.436965 16501 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:19:45.437217 16501 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:19:45.437392 16501 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:19:45.477210 16501 rpc_server.cc:307] RPC server started. Bound to: 127.16.29.67:44055
I20260709 12:19:45.477319 17359 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.29.67:44055 every 8 connection(s)
I20260709 12:19:45.493109 17360 heartbeater.cc:344] Connected to a master server at 127.16.29.126:37041
I20260709 12:19:45.493521 17360 heartbeater.cc:461] Registering TS with master...
I20260709 12:19:45.494342 17360 heartbeater.cc:507] Master 127.16.29.126:37041 requested a full tablet report, sending...
I20260709 12:19:45.496445 17084 ts_manager.cc:194] Registered new tserver with Master: 78285c60669649f6a5f13c1b65f9419e (127.16.29.67:44055)
I20260709 12:19:45.496639 16501 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.016332701s
I20260709 12:19:45.498279 17084 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:49528
I20260709 12:19:45.522578 17084 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49544:
name: "test-workload"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
  rows: "\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s""\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s"
  indirect_data: """"
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W20260709 12:19:45.525614 17084 catalog_manager.cc:7077] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260709 12:19:45.599454 17251 tablet_service.cc:1511] Processing CreateTablet for tablet 0e08c6dc97d34c998b47e5207487ef31 (DEFAULT_TABLE table=test-workload [id=37330d8d797743f9b268b745b9fcdad2]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260709 12:19:45.599511 17250 tablet_service.cc:1511] Processing CreateTablet for tablet 9c4d5228c0fc4198aaf94191977ad479 (DEFAULT_TABLE table=test-workload [id=37330d8d797743f9b268b745b9fcdad2]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260709 12:19:45.601198 17251 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0e08c6dc97d34c998b47e5207487ef31. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:19:45.601234 17176 tablet_service.cc:1511] Processing CreateTablet for tablet 0e08c6dc97d34c998b47e5207487ef31 (DEFAULT_TABLE table=test-workload [id=37330d8d797743f9b268b745b9fcdad2]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260709 12:19:45.602844 17176 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0e08c6dc97d34c998b47e5207487ef31. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:19:45.604054 17174 tablet_service.cc:1511] Processing CreateTablet for tablet 0762726dc14a430cb8f27b2393feec0e (DEFAULT_TABLE table=test-workload [id=37330d8d797743f9b268b745b9fcdad2]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260709 12:19:45.605371 17174 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0762726dc14a430cb8f27b2393feec0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:19:45.606734 17249 tablet_service.cc:1511] Processing CreateTablet for tablet 0762726dc14a430cb8f27b2393feec0e (DEFAULT_TABLE table=test-workload [id=37330d8d797743f9b268b745b9fcdad2]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260709 12:19:45.608197 17172 tablet_service.cc:1511] Processing CreateTablet for tablet eecaa10d24c045d3921a84ecb5501242 (DEFAULT_TABLE table=test-workload [id=37330d8d797743f9b268b745b9fcdad2]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260709 12:19:45.609077 17248 tablet_service.cc:1511] Processing CreateTablet for tablet c55fea8d70354a8fa064ca624fb7a880 (DEFAULT_TABLE table=test-workload [id=37330d8d797743f9b268b745b9fcdad2]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260709 12:19:45.602861 17175 tablet_service.cc:1511] Processing CreateTablet for tablet 9c4d5228c0fc4198aaf94191977ad479 (DEFAULT_TABLE table=test-workload [id=37330d8d797743f9b268b745b9fcdad2]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260709 12:19:45.611163 17173 tablet_service.cc:1511] Processing CreateTablet for tablet c55fea8d70354a8fa064ca624fb7a880 (DEFAULT_TABLE table=test-workload [id=37330d8d797743f9b268b745b9fcdad2]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260709 12:19:45.613971 17247 tablet_service.cc:1511] Processing CreateTablet for tablet eecaa10d24c045d3921a84ecb5501242 (DEFAULT_TABLE table=test-workload [id=37330d8d797743f9b268b745b9fcdad2]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260709 12:19:45.616349 17246 tablet_service.cc:1511] Processing CreateTablet for tablet ce6646ae2dfe43c681b0b1a3ed8cecd4 (DEFAULT_TABLE table=test-workload [id=37330d8d797743f9b268b745b9fcdad2]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260709 12:19:45.617372 17171 tablet_service.cc:1511] Processing CreateTablet for tablet ce6646ae2dfe43c681b0b1a3ed8cecd4 (DEFAULT_TABLE table=test-workload [id=37330d8d797743f9b268b745b9fcdad2]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260709 12:19:45.618474 17249 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0762726dc14a430cb8f27b2393feec0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:19:45.619611 17170 tablet_service.cc:1511] Processing CreateTablet for tablet 103a303e4bce44cca4e34b3c391dbdac (DEFAULT_TABLE table=test-workload [id=37330d8d797743f9b268b745b9fcdad2]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260709 12:19:45.620725 17248 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c55fea8d70354a8fa064ca624fb7a880. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:19:45.622349 17169 tablet_service.cc:1511] Processing CreateTablet for tablet ec3f5327156e49e48ae8c88da0db0522 (DEFAULT_TABLE table=test-workload [id=37330d8d797743f9b268b745b9fcdad2]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260709 12:19:45.624259 17247 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eecaa10d24c045d3921a84ecb5501242. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:19:45.624136 17168 tablet_service.cc:1511] Processing CreateTablet for tablet 58783bb92efb47c2b521006941944a9d (DEFAULT_TABLE table=test-workload [id=37330d8d797743f9b268b745b9fcdad2]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260709 12:19:45.628808 17250 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9c4d5228c0fc4198aaf94191977ad479. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:19:45.632422 17246 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ce6646ae2dfe43c681b0b1a3ed8cecd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:19:45.632294 17167 tablet_service.cc:1511] Processing CreateTablet for tablet 9f65da5ec1234214840536f7d27a5499 (DEFAULT_TABLE table=test-workload [id=37330d8d797743f9b268b745b9fcdad2]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260709 12:19:45.609503 17172 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eecaa10d24c045d3921a84ecb5501242. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:19:45.642807 17245 tablet_service.cc:1511] Processing CreateTablet for tablet 103a303e4bce44cca4e34b3c391dbdac (DEFAULT_TABLE table=test-workload [id=37330d8d797743f9b268b745b9fcdad2]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260709 12:19:45.644146 17245 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 103a303e4bce44cca4e34b3c391dbdac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:19:45.644245 17170 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 103a303e4bce44cca4e34b3c391dbdac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:19:45.645687 17171 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ce6646ae2dfe43c681b0b1a3ed8cecd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:19:45.646322 17175 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9c4d5228c0fc4198aaf94191977ad479. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:19:45.646984 17173 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c55fea8d70354a8fa064ca624fb7a880. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:19:45.648486 17169 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ec3f5327156e49e48ae8c88da0db0522. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:19:45.648444 17244 tablet_service.cc:1511] Processing CreateTablet for tablet ec3f5327156e49e48ae8c88da0db0522 (DEFAULT_TABLE table=test-workload [id=37330d8d797743f9b268b745b9fcdad2]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260709 12:19:45.649822 17244 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ec3f5327156e49e48ae8c88da0db0522. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:19:45.649971 17168 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 58783bb92efb47c2b521006941944a9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:19:45.652392 17243 tablet_service.cc:1511] Processing CreateTablet for tablet 58783bb92efb47c2b521006941944a9d (DEFAULT_TABLE table=test-workload [id=37330d8d797743f9b268b745b9fcdad2]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260709 12:19:45.654105 17243 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 58783bb92efb47c2b521006941944a9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:19:45.656538 17324 tablet_service.cc:1511] Processing CreateTablet for tablet 9c4d5228c0fc4198aaf94191977ad479 (DEFAULT_TABLE table=test-workload [id=37330d8d797743f9b268b745b9fcdad2]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260709 12:19:45.657878 17324 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9c4d5228c0fc4198aaf94191977ad479. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:19:45.661808 17242 tablet_service.cc:1511] Processing CreateTablet for tablet 9f65da5ec1234214840536f7d27a5499 (DEFAULT_TABLE table=test-workload [id=37330d8d797743f9b268b745b9fcdad2]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260709 12:19:45.665201 17323 tablet_service.cc:1511] Processing CreateTablet for tablet 0762726dc14a430cb8f27b2393feec0e (DEFAULT_TABLE table=test-workload [id=37330d8d797743f9b268b745b9fcdad2]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260709 12:19:45.666543 17323 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0762726dc14a430cb8f27b2393feec0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:19:45.669682 17322 tablet_service.cc:1511] Processing CreateTablet for tablet c55fea8d70354a8fa064ca624fb7a880 (DEFAULT_TABLE table=test-workload [id=37330d8d797743f9b268b745b9fcdad2]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260709 12:19:45.670987 17322 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c55fea8d70354a8fa064ca624fb7a880. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:19:45.655283 17325 tablet_service.cc:1511] Processing CreateTablet for tablet 0e08c6dc97d34c998b47e5207487ef31 (DEFAULT_TABLE table=test-workload [id=37330d8d797743f9b268b745b9fcdad2]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260709 12:19:45.684880 17319 tablet_service.cc:1511] Processing CreateTablet for tablet 103a303e4bce44cca4e34b3c391dbdac (DEFAULT_TABLE table=test-workload [id=37330d8d797743f9b268b745b9fcdad2]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260709 12:19:45.687218 17325 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0e08c6dc97d34c998b47e5207487ef31. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:19:45.688208 17319 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 103a303e4bce44cca4e34b3c391dbdac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:19:45.688010 17318 tablet_service.cc:1511] Processing CreateTablet for tablet ec3f5327156e49e48ae8c88da0db0522 (DEFAULT_TABLE table=test-workload [id=37330d8d797743f9b268b745b9fcdad2]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260709 12:19:45.690318 17320 tablet_service.cc:1511] Processing CreateTablet for tablet ce6646ae2dfe43c681b0b1a3ed8cecd4 (DEFAULT_TABLE table=test-workload [id=37330d8d797743f9b268b745b9fcdad2]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260709 12:19:45.682835 17321 tablet_service.cc:1511] Processing CreateTablet for tablet eecaa10d24c045d3921a84ecb5501242 (DEFAULT_TABLE table=test-workload [id=37330d8d797743f9b268b745b9fcdad2]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260709 12:19:45.693152 17242 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9f65da5ec1234214840536f7d27a5499. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:19:45.680895 17167 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9f65da5ec1234214840536f7d27a5499. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:19:45.691747 17320 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ce6646ae2dfe43c681b0b1a3ed8cecd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:19:45.701762 17318 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ec3f5327156e49e48ae8c88da0db0522. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:19:45.704320 17321 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eecaa10d24c045d3921a84ecb5501242. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:19:45.712070 17317 tablet_service.cc:1511] Processing CreateTablet for tablet 58783bb92efb47c2b521006941944a9d (DEFAULT_TABLE table=test-workload [id=37330d8d797743f9b268b745b9fcdad2]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260709 12:19:45.713516 17317 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 58783bb92efb47c2b521006941944a9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:19:45.730865 17316 tablet_service.cc:1511] Processing CreateTablet for tablet 9f65da5ec1234214840536f7d27a5499 (DEFAULT_TABLE table=test-workload [id=37330d8d797743f9b268b745b9fcdad2]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260709 12:19:45.732883 17316 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9f65da5ec1234214840536f7d27a5499. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:19:45.735976 17375 tablet_bootstrap.cc:492] T 0e08c6dc97d34c998b47e5207487ef31 P 71eca61062e040d48b2398afb3b0b510: Bootstrap starting.
I20260709 12:19:45.837597 17377 tablet_bootstrap.cc:492] T 103a303e4bce44cca4e34b3c391dbdac P 78285c60669649f6a5f13c1b65f9419e: Bootstrap starting.
I20260709 12:19:45.840654 17376 tablet_bootstrap.cc:492] T 58783bb92efb47c2b521006941944a9d P 30f3ca7002424f919b063ee77bc473de: Bootstrap starting.
I20260709 12:19:45.860031 17375 tablet_bootstrap.cc:654] T 0e08c6dc97d34c998b47e5207487ef31 P 71eca61062e040d48b2398afb3b0b510: Neither blocks nor log segments found. Creating new log.
I20260709 12:19:45.902361 17376 tablet_bootstrap.cc:654] T 58783bb92efb47c2b521006941944a9d P 30f3ca7002424f919b063ee77bc473de: Neither blocks nor log segments found. Creating new log.
I20260709 12:19:45.912469 17377 tablet_bootstrap.cc:654] T 103a303e4bce44cca4e34b3c391dbdac P 78285c60669649f6a5f13c1b65f9419e: Neither blocks nor log segments found. Creating new log.
I20260709 12:19:45.918229 17375 tablet_bootstrap.cc:492] T 0e08c6dc97d34c998b47e5207487ef31 P 71eca61062e040d48b2398afb3b0b510: No bootstrap required, opened a new log
I20260709 12:19:45.918655 17375 ts_tablet_manager.cc:1403] T 0e08c6dc97d34c998b47e5207487ef31 P 71eca61062e040d48b2398afb3b0b510: Time spent bootstrapping tablet: real 0.183s	user 0.006s	sys 0.034s
I20260709 12:19:45.921003 17375 raft_consensus.cc:359] T 0e08c6dc97d34c998b47e5207487ef31 P 71eca61062e040d48b2398afb3b0b510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 } } peers { permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 } } peers { permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 } }
I20260709 12:19:45.921550 17375 raft_consensus.cc:385] T 0e08c6dc97d34c998b47e5207487ef31 P 71eca61062e040d48b2398afb3b0b510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:19:45.921794 17375 raft_consensus.cc:740] T 0e08c6dc97d34c998b47e5207487ef31 P 71eca61062e040d48b2398afb3b0b510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71eca61062e040d48b2398afb3b0b510, State: Initialized, Role: FOLLOWER
I20260709 12:19:45.922497 17375 consensus_queue.cc:260] T 0e08c6dc97d34c998b47e5207487ef31 P 71eca61062e040d48b2398afb3b0b510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 } } peers { permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 } } peers { permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 } }
I20260709 12:19:45.922963 17377 tablet_bootstrap.cc:492] T 103a303e4bce44cca4e34b3c391dbdac P 78285c60669649f6a5f13c1b65f9419e: No bootstrap required, opened a new log
I20260709 12:19:45.923435 17377 ts_tablet_manager.cc:1403] T 103a303e4bce44cca4e34b3c391dbdac P 78285c60669649f6a5f13c1b65f9419e: Time spent bootstrapping tablet: real 0.086s	user 0.017s	sys 0.000s
I20260709 12:19:45.923908 17376 tablet_bootstrap.cc:492] T 58783bb92efb47c2b521006941944a9d P 30f3ca7002424f919b063ee77bc473de: No bootstrap required, opened a new log
I20260709 12:19:45.924381 17376 ts_tablet_manager.cc:1403] T 58783bb92efb47c2b521006941944a9d P 30f3ca7002424f919b063ee77bc473de: Time spent bootstrapping tablet: real 0.084s	user 0.007s	sys 0.010s
I20260709 12:19:45.924944 17286 heartbeater.cc:499] Master 127.16.29.126:37041 was elected leader, sending a full tablet report...
I20260709 12:19:45.925372 17375 ts_tablet_manager.cc:1434] T 0e08c6dc97d34c998b47e5207487ef31 P 71eca61062e040d48b2398afb3b0b510: Time spent starting tablet: real 0.006s	user 0.007s	sys 0.000s
I20260709 12:19:45.926260 17375 tablet_bootstrap.cc:492] T c55fea8d70354a8fa064ca624fb7a880 P 71eca61062e040d48b2398afb3b0b510: Bootstrap starting.
I20260709 12:19:45.926311 17377 raft_consensus.cc:359] T 103a303e4bce44cca4e34b3c391dbdac P 78285c60669649f6a5f13c1b65f9419e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 } } peers { permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 } } peers { permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 } }
I20260709 12:19:45.927110 17377 raft_consensus.cc:385] T 103a303e4bce44cca4e34b3c391dbdac P 78285c60669649f6a5f13c1b65f9419e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:19:45.927490 17377 raft_consensus.cc:740] T 103a303e4bce44cca4e34b3c391dbdac P 78285c60669649f6a5f13c1b65f9419e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78285c60669649f6a5f13c1b65f9419e, State: Initialized, Role: FOLLOWER
I20260709 12:19:45.927173 17376 raft_consensus.cc:359] T 58783bb92efb47c2b521006941944a9d P 30f3ca7002424f919b063ee77bc473de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 } } peers { permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 } } peers { permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 } }
I20260709 12:19:45.927920 17376 raft_consensus.cc:385] T 58783bb92efb47c2b521006941944a9d P 30f3ca7002424f919b063ee77bc473de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:19:45.928689 17376 raft_consensus.cc:740] T 58783bb92efb47c2b521006941944a9d P 30f3ca7002424f919b063ee77bc473de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30f3ca7002424f919b063ee77bc473de, State: Initialized, Role: FOLLOWER
I20260709 12:19:45.928869 17377 consensus_queue.cc:260] T 103a303e4bce44cca4e34b3c391dbdac P 78285c60669649f6a5f13c1b65f9419e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 } } peers { permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 } } peers { permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 } }
I20260709 12:19:45.929410 17376 consensus_queue.cc:260] T 58783bb92efb47c2b521006941944a9d P 30f3ca7002424f919b063ee77bc473de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 } } peers { permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 } } peers { permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 } }
I20260709 12:19:45.931030 17381 raft_consensus.cc:493] T 0e08c6dc97d34c998b47e5207487ef31 P 71eca61062e040d48b2398afb3b0b510 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:45.931567 17381 raft_consensus.cc:515] T 0e08c6dc97d34c998b47e5207487ef31 P 71eca61062e040d48b2398afb3b0b510 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 } } peers { permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 } } peers { permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 } }
I20260709 12:19:45.933159 17375 tablet_bootstrap.cc:654] T c55fea8d70354a8fa064ca624fb7a880 P 71eca61062e040d48b2398afb3b0b510: Neither blocks nor log segments found. Creating new log.
I20260709 12:19:45.938205 17211 heartbeater.cc:499] Master 127.16.29.126:37041 was elected leader, sending a full tablet report...
I20260709 12:19:45.938638 17376 ts_tablet_manager.cc:1434] T 58783bb92efb47c2b521006941944a9d P 30f3ca7002424f919b063ee77bc473de: Time spent starting tablet: real 0.014s	user 0.006s	sys 0.007s
I20260709 12:19:45.939615 17376 tablet_bootstrap.cc:492] T 0762726dc14a430cb8f27b2393feec0e P 30f3ca7002424f919b063ee77bc473de: Bootstrap starting.
I20260709 12:19:45.943578 17375 tablet_bootstrap.cc:492] T c55fea8d70354a8fa064ca624fb7a880 P 71eca61062e040d48b2398afb3b0b510: No bootstrap required, opened a new log
I20260709 12:19:45.944167 17375 ts_tablet_manager.cc:1403] T c55fea8d70354a8fa064ca624fb7a880 P 71eca61062e040d48b2398afb3b0b510: Time spent bootstrapping tablet: real 0.018s	user 0.010s	sys 0.005s
I20260709 12:19:45.946281 17376 tablet_bootstrap.cc:654] T 0762726dc14a430cb8f27b2393feec0e P 30f3ca7002424f919b063ee77bc473de: Neither blocks nor log segments found. Creating new log.
I20260709 12:19:45.947305 17375 raft_consensus.cc:359] T c55fea8d70354a8fa064ca624fb7a880 P 71eca61062e040d48b2398afb3b0b510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 } } peers { permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 } } peers { permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 } }
I20260709 12:19:45.948172 17375 raft_consensus.cc:385] T c55fea8d70354a8fa064ca624fb7a880 P 71eca61062e040d48b2398afb3b0b510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:19:45.948483 17375 raft_consensus.cc:740] T c55fea8d70354a8fa064ca624fb7a880 P 71eca61062e040d48b2398afb3b0b510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71eca61062e040d48b2398afb3b0b510, State: Initialized, Role: FOLLOWER
I20260709 12:19:45.949337 17375 consensus_queue.cc:260] T c55fea8d70354a8fa064ca624fb7a880 P 71eca61062e040d48b2398afb3b0b510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 } } peers { permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 } } peers { permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 } }
I20260709 12:19:45.958483 17381 leader_election.cc:290] T 0e08c6dc97d34c998b47e5207487ef31 P 71eca61062e040d48b2398afb3b0b510 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30f3ca7002424f919b063ee77bc473de (127.16.29.65:35609), 78285c60669649f6a5f13c1b65f9419e (127.16.29.67:44055)
I20260709 12:19:45.960253 17375 ts_tablet_manager.cc:1434] T c55fea8d70354a8fa064ca624fb7a880 P 71eca61062e040d48b2398afb3b0b510: Time spent starting tablet: real 0.016s	user 0.010s	sys 0.004s
I20260709 12:19:45.961593 17375 tablet_bootstrap.cc:492] T ce6646ae2dfe43c681b0b1a3ed8cecd4 P 71eca61062e040d48b2398afb3b0b510: Bootstrap starting.
I20260709 12:19:45.969102 17375 tablet_bootstrap.cc:654] T ce6646ae2dfe43c681b0b1a3ed8cecd4 P 71eca61062e040d48b2398afb3b0b510: Neither blocks nor log segments found. Creating new log.
I20260709 12:19:45.970775 17360 heartbeater.cc:499] Master 127.16.29.126:37041 was elected leader, sending a full tablet report...
I20260709 12:19:45.973722 17377 ts_tablet_manager.cc:1434] T 103a303e4bce44cca4e34b3c391dbdac P 78285c60669649f6a5f13c1b65f9419e: Time spent starting tablet: real 0.050s	user 0.010s	sys 0.000s
I20260709 12:19:45.981657 17186 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e08c6dc97d34c998b47e5207487ef31" candidate_uuid: "71eca61062e040d48b2398afb3b0b510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30f3ca7002424f919b063ee77bc473de" is_pre_election: true
W20260709 12:19:45.983184 17361 tablet.cc:2406] T 103a303e4bce44cca4e34b3c391dbdac P 78285c60669649f6a5f13c1b65f9419e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260709 12:19:45.984903 17377 tablet_bootstrap.cc:492] T 0762726dc14a430cb8f27b2393feec0e P 78285c60669649f6a5f13c1b65f9419e: Bootstrap starting.
W20260709 12:19:45.985258 17227 leader_election.cc:343] T 0e08c6dc97d34c998b47e5207487ef31 P 71eca61062e040d48b2398afb3b0b510 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30f3ca7002424f919b063ee77bc473de (127.16.29.65:35609): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:19:45.987294 17376 tablet_bootstrap.cc:492] T 0762726dc14a430cb8f27b2393feec0e P 30f3ca7002424f919b063ee77bc473de: No bootstrap required, opened a new log
I20260709 12:19:45.987850 17376 ts_tablet_manager.cc:1403] T 0762726dc14a430cb8f27b2393feec0e P 30f3ca7002424f919b063ee77bc473de: Time spent bootstrapping tablet: real 0.048s	user 0.030s	sys 0.013s
I20260709 12:19:45.991309 17376 raft_consensus.cc:359] T 0762726dc14a430cb8f27b2393feec0e P 30f3ca7002424f919b063ee77bc473de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 } } peers { permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 } } peers { permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 } }
I20260709 12:19:45.992060 17376 raft_consensus.cc:385] T 0762726dc14a430cb8f27b2393feec0e P 30f3ca7002424f919b063ee77bc473de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:19:45.992456 17376 raft_consensus.cc:740] T 0762726dc14a430cb8f27b2393feec0e P 30f3ca7002424f919b063ee77bc473de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30f3ca7002424f919b063ee77bc473de, State: Initialized, Role: FOLLOWER
I20260709 12:19:45.993235 17377 tablet_bootstrap.cc:654] T 0762726dc14a430cb8f27b2393feec0e P 78285c60669649f6a5f13c1b65f9419e: Neither blocks nor log segments found. Creating new log.
I20260709 12:19:45.993324 17376 consensus_queue.cc:260] T 0762726dc14a430cb8f27b2393feec0e P 30f3ca7002424f919b063ee77bc473de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 } } peers { permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 } } peers { permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 } }
I20260709 12:19:45.998060 17376 ts_tablet_manager.cc:1434] T 0762726dc14a430cb8f27b2393feec0e P 30f3ca7002424f919b063ee77bc473de: Time spent starting tablet: real 0.010s	user 0.005s	sys 0.002s
I20260709 12:19:45.999017 17376 tablet_bootstrap.cc:492] T 0e08c6dc97d34c998b47e5207487ef31 P 30f3ca7002424f919b063ee77bc473de: Bootstrap starting.
I20260709 12:19:46.006381 17376 tablet_bootstrap.cc:654] T 0e08c6dc97d34c998b47e5207487ef31 P 30f3ca7002424f919b063ee77bc473de: Neither blocks nor log segments found. Creating new log.
I20260709 12:19:46.011834 17375 tablet_bootstrap.cc:492] T ce6646ae2dfe43c681b0b1a3ed8cecd4 P 71eca61062e040d48b2398afb3b0b510: No bootstrap required, opened a new log
I20260709 12:19:46.012528 17375 ts_tablet_manager.cc:1403] T ce6646ae2dfe43c681b0b1a3ed8cecd4 P 71eca61062e040d48b2398afb3b0b510: Time spent bootstrapping tablet: real 0.051s	user 0.021s	sys 0.007s
I20260709 12:19:46.015683 17377 tablet_bootstrap.cc:492] T 0762726dc14a430cb8f27b2393feec0e P 78285c60669649f6a5f13c1b65f9419e: No bootstrap required, opened a new log
I20260709 12:19:46.016238 17377 ts_tablet_manager.cc:1403] T 0762726dc14a430cb8f27b2393feec0e P 78285c60669649f6a5f13c1b65f9419e: Time spent bootstrapping tablet: real 0.032s	user 0.010s	sys 0.005s
I20260709 12:19:46.015794 17375 raft_consensus.cc:359] T ce6646ae2dfe43c681b0b1a3ed8cecd4 P 71eca61062e040d48b2398afb3b0b510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 } } peers { permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 } } peers { permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 } }
I20260709 12:19:46.016494 17376 tablet_bootstrap.cc:492] T 0e08c6dc97d34c998b47e5207487ef31 P 30f3ca7002424f919b063ee77bc473de: No bootstrap required, opened a new log
I20260709 12:19:46.016628 17375 raft_consensus.cc:385] T ce6646ae2dfe43c681b0b1a3ed8cecd4 P 71eca61062e040d48b2398afb3b0b510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:19:46.017001 17376 ts_tablet_manager.cc:1403] T 0e08c6dc97d34c998b47e5207487ef31 P 30f3ca7002424f919b063ee77bc473de: Time spent bootstrapping tablet: real 0.018s	user 0.013s	sys 0.004s
I20260709 12:19:46.016999 17375 raft_consensus.cc:740] T ce6646ae2dfe43c681b0b1a3ed8cecd4 P 71eca61062e040d48b2398afb3b0b510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71eca61062e040d48b2398afb3b0b510, State: Initialized, Role: FOLLOWER
I20260709 12:19:46.018374 17375 consensus_queue.cc:260] T ce6646ae2dfe43c681b0b1a3ed8cecd4 P 71eca61062e040d48b2398afb3b0b510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 } } peers { permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 } } peers { permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 } }
I20260709 12:19:46.019038 17377 raft_consensus.cc:359] T 0762726dc14a430cb8f27b2393feec0e P 78285c60669649f6a5f13c1b65f9419e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 } } peers { permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 } } peers { permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 } }
I20260709 12:19:46.019734 17377 raft_consensus.cc:385] T 0762726dc14a430cb8f27b2393feec0e P 78285c60669649f6a5f13c1b65f9419e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:19:46.020097 17377 raft_consensus.cc:740] T 0762726dc14a430cb8f27b2393feec0e P 78285c60669649f6a5f13c1b65f9419e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78285c60669649f6a5f13c1b65f9419e, State: Initialized, Role: FOLLOWER
I20260709 12:19:46.019819 17376 raft_consensus.cc:359] T 0e08c6dc97d34c998b47e5207487ef31 P 30f3ca7002424f919b063ee77bc473de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 } } peers { permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 } } peers { permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 } }
I20260709 12:19:46.020543 17375 ts_tablet_manager.cc:1434] T ce6646ae2dfe43c681b0b1a3ed8cecd4 P 71eca61062e040d48b2398afb3b0b510: Time spent starting tablet: real 0.008s	user 0.004s	sys 0.003s
I20260709 12:19:46.020619 17376 raft_consensus.cc:385] T 0e08c6dc97d34c998b47e5207487ef31 P 30f3ca7002424f919b063ee77bc473de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:19:46.021013 17377 consensus_queue.cc:260] T 0762726dc14a430cb8f27b2393feec0e P 78285c60669649f6a5f13c1b65f9419e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 } } peers { permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 } } peers { permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 } }
I20260709 12:19:46.021025 17376 raft_consensus.cc:740] T 0e08c6dc97d34c998b47e5207487ef31 P 30f3ca7002424f919b063ee77bc473de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30f3ca7002424f919b063ee77bc473de, State: Initialized, Role: FOLLOWER
I20260709 12:19:46.022503 17376 consensus_queue.cc:260] T 0e08c6dc97d34c998b47e5207487ef31 P 30f3ca7002424f919b063ee77bc473de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 } } peers { permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 } } peers { permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 } }
I20260709 12:19:46.021569 17375 tablet_bootstrap.cc:492] T 0762726dc14a430cb8f27b2393feec0e P 71eca61062e040d48b2398afb3b0b510: Bootstrap starting.
I20260709 12:19:46.024740 17376 ts_tablet_manager.cc:1434] T 0e08c6dc97d34c998b47e5207487ef31 P 30f3ca7002424f919b063ee77bc473de: Time spent starting tablet: real 0.007s	user 0.007s	sys 0.000s
I20260709 12:19:46.024806 17335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e08c6dc97d34c998b47e5207487ef31" candidate_uuid: "71eca61062e040d48b2398afb3b0b510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78285c60669649f6a5f13c1b65f9419e" is_pre_election: true
I20260709 12:19:46.025892 17376 tablet_bootstrap.cc:492] T c55fea8d70354a8fa064ca624fb7a880 P 30f3ca7002424f919b063ee77bc473de: Bootstrap starting.
W20260709 12:19:46.026424 17225 leader_election.cc:343] T 0e08c6dc97d34c998b47e5207487ef31 P 71eca61062e040d48b2398afb3b0b510 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 78285c60669649f6a5f13c1b65f9419e (127.16.29.67:44055): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:19:46.026893 17225 leader_election.cc:304] T 0e08c6dc97d34c998b47e5207487ef31 P 71eca61062e040d48b2398afb3b0b510 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 71eca61062e040d48b2398afb3b0b510; no voters: 30f3ca7002424f919b063ee77bc473de, 78285c60669649f6a5f13c1b65f9419e
I20260709 12:19:46.027957 17381 raft_consensus.cc:2749] T 0e08c6dc97d34c998b47e5207487ef31 P 71eca61062e040d48b2398afb3b0b510 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:19:46.030305 17375 tablet_bootstrap.cc:654] T 0762726dc14a430cb8f27b2393feec0e P 71eca61062e040d48b2398afb3b0b510: Neither blocks nor log segments found. Creating new log.
I20260709 12:19:46.033430 17377 ts_tablet_manager.cc:1434] T 0762726dc14a430cb8f27b2393feec0e P 78285c60669649f6a5f13c1b65f9419e: Time spent starting tablet: real 0.017s	user 0.007s	sys 0.000s
I20260709 12:19:46.034608 17377 tablet_bootstrap.cc:492] T ce6646ae2dfe43c681b0b1a3ed8cecd4 P 78285c60669649f6a5f13c1b65f9419e: Bootstrap starting.
I20260709 12:19:46.033555 17376 tablet_bootstrap.cc:654] T c55fea8d70354a8fa064ca624fb7a880 P 30f3ca7002424f919b063ee77bc473de: Neither blocks nor log segments found. Creating new log.
I20260709 12:19:46.042419 17375 tablet_bootstrap.cc:492] T 0762726dc14a430cb8f27b2393feec0e P 71eca61062e040d48b2398afb3b0b510: No bootstrap required, opened a new log
I20260709 12:19:46.042797 17377 tablet_bootstrap.cc:654] T ce6646ae2dfe43c681b0b1a3ed8cecd4 P 78285c60669649f6a5f13c1b65f9419e: Neither blocks nor log segments found. Creating new log.
I20260709 12:19:46.042943 17375 ts_tablet_manager.cc:1403] T 0762726dc14a430cb8f27b2393feec0e P 71eca61062e040d48b2398afb3b0b510: Time spent bootstrapping tablet: real 0.022s	user 0.013s	sys 0.003s
I20260709 12:19:46.045913 17375 raft_consensus.cc:359] T 0762726dc14a430cb8f27b2393feec0e P 71eca61062e040d48b2398afb3b0b510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 } } peers { permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 } } peers { permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 } }
I20260709 12:19:46.046643 17375 raft_consensus.cc:385] T 0762726dc14a430cb8f27b2393feec0e P 71eca61062e040d48b2398afb3b0b510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:19:46.046952 17375 raft_consensus.cc:740] T 0762726dc14a430cb8f27b2393feec0e P 71eca61062e040d48b2398afb3b0b510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71eca61062e040d48b2398afb3b0b510, State: Initialized, Role: FOLLOWER
I20260709 12:19:46.047780 17375 consensus_queue.cc:260] T 0762726dc14a430cb8f27b2393feec0e P 71eca61062e040d48b2398afb3b0b510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 } } peers { permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 } } peers { permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 } }
I20260709 12:19:46.051991 17375 ts_tablet_manager.cc:1434] T 0762726dc14a430cb8f27b2393feec0e P 71eca61062e040d48b2398afb3b0b510: Time spent starting tablet: real 0.009s	user 0.009s	sys 0.000s
I20260709 12:19:46.052906 17375 tablet_bootstrap.cc:492] T ec3f5327156e49e48ae8c88da0db0522 P 71eca61062e040d48b2398afb3b0b510: Bootstrap starting.
I20260709 12:19:46.053440 17377 tablet_bootstrap.cc:492] T ce6646ae2dfe43c681b0b1a3ed8cecd4 P 78285c60669649f6a5f13c1b65f9419e: No bootstrap required, opened a new log
I20260709 12:19:46.053910 17377 ts_tablet_manager.cc:1403] T ce6646ae2dfe43c681b0b1a3ed8cecd4 P 78285c60669649f6a5f13c1b65f9419e: Time spent bootstrapping tablet: real 0.020s	user 0.007s	sys 0.010s
I20260709 12:19:46.057354 17376 tablet_bootstrap.cc:492] T c55fea8d70354a8fa064ca624fb7a880 P 30f3ca7002424f919b063ee77bc473de: No bootstrap required, opened a new log
I20260709 12:19:46.056763 17377 raft_consensus.cc:359] T ce6646ae2dfe43c681b0b1a3ed8cecd4 P 78285c60669649f6a5f13c1b65f9419e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 } } peers { permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 } } peers { permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 } }
I20260709 12:19:46.058010 17377 raft_consensus.cc:385] T ce6646ae2dfe43c681b0b1a3ed8cecd4 P 78285c60669649f6a5f13c1b65f9419e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:19:46.058300 17376 ts_tablet_manager.cc:1403] T c55fea8d70354a8fa064ca624fb7a880 P 30f3ca7002424f919b063ee77bc473de: Time spent bootstrapping tablet: real 0.033s	user 0.012s	sys 0.008s
I20260709 12:19:46.058564 17377 raft_consensus.cc:740] T ce6646ae2dfe43c681b0b1a3ed8cecd4 P 78285c60669649f6a5f13c1b65f9419e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78285c60669649f6a5f13c1b65f9419e, State: Initialized, Role: FOLLOWER
I20260709 12:19:46.059842 17375 tablet_bootstrap.cc:654] T ec3f5327156e49e48ae8c88da0db0522 P 71eca61062e040d48b2398afb3b0b510: Neither blocks nor log segments found. Creating new log.
I20260709 12:19:46.059990 17377 consensus_queue.cc:260] T ce6646ae2dfe43c681b0b1a3ed8cecd4 P 78285c60669649f6a5f13c1b65f9419e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 } } peers { permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 } } peers { permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 } }
I20260709 12:19:46.062268 17388 raft_consensus.cc:493] T 103a303e4bce44cca4e34b3c391dbdac P 78285c60669649f6a5f13c1b65f9419e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:46.062496 17376 raft_consensus.cc:359] T c55fea8d70354a8fa064ca624fb7a880 P 30f3ca7002424f919b063ee77bc473de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 } } peers { permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 } } peers { permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 } }
I20260709 12:19:46.062813 17388 raft_consensus.cc:515] T 103a303e4bce44cca4e34b3c391dbdac P 78285c60669649f6a5f13c1b65f9419e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 } } peers { permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 } } peers { permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 } }
I20260709 12:19:46.065392 17377 ts_tablet_manager.cc:1434] T ce6646ae2dfe43c681b0b1a3ed8cecd4 P 78285c60669649f6a5f13c1b65f9419e: Time spent starting tablet: real 0.011s	user 0.008s	sys 0.000s
I20260709 12:19:46.066138 17375 tablet_bootstrap.cc:492] T ec3f5327156e49e48ae8c88da0db0522 P 71eca61062e040d48b2398afb3b0b510: No bootstrap required, opened a new log
I20260709 12:19:46.066447 17377 tablet_bootstrap.cc:492] T 58783bb92efb47c2b521006941944a9d P 78285c60669649f6a5f13c1b65f9419e: Bootstrap starting.
I20260709 12:19:46.066665 17375 ts_tablet_manager.cc:1403] T ec3f5327156e49e48ae8c88da0db0522 P 71eca61062e040d48b2398afb3b0b510: Time spent bootstrapping tablet: real 0.014s	user 0.008s	sys 0.005s
I20260709 12:19:46.069772 17375 raft_consensus.cc:359] T ec3f5327156e49e48ae8c88da0db0522 P 71eca61062e040d48b2398afb3b0b510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 } } peers { permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 } } peers { permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 } }
I20260709 12:19:46.071133 17375 raft_consensus.cc:385] T ec3f5327156e49e48ae8c88da0db0522 P 71eca61062e040d48b2398afb3b0b510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:19:46.071661 17375 raft_consensus.cc:740] T ec3f5327156e49e48ae8c88da0db0522 P 71eca61062e040d48b2398afb3b0b510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71eca61062e040d48b2398afb3b0b510, State: Initialized, Role: FOLLOWER
I20260709 12:19:46.072490 17375 consensus_queue.cc:260] T ec3f5327156e49e48ae8c88da0db0522 P 71eca61062e040d48b2398afb3b0b510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 } } peers { permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 } } peers { permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 } }
I20260709 12:19:46.074188 17377 tablet_bootstrap.cc:654] T 58783bb92efb47c2b521006941944a9d P 78285c60669649f6a5f13c1b65f9419e: Neither blocks nor log segments found. Creating new log.
I20260709 12:19:46.074783 17388 leader_election.cc:290] T 103a303e4bce44cca4e34b3c391dbdac P 78285c60669649f6a5f13c1b65f9419e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30f3ca7002424f919b063ee77bc473de (127.16.29.65:35609), 71eca61062e040d48b2398afb3b0b510 (127.16.29.66:44237)
I20260709 12:19:46.063172 17376 raft_consensus.cc:385] T c55fea8d70354a8fa064ca624fb7a880 P 30f3ca7002424f919b063ee77bc473de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:19:46.075464 17376 raft_consensus.cc:740] T c55fea8d70354a8fa064ca624fb7a880 P 30f3ca7002424f919b063ee77bc473de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30f3ca7002424f919b063ee77bc473de, State: Initialized, Role: FOLLOWER
I20260709 12:19:46.076551 17376 consensus_queue.cc:260] T c55fea8d70354a8fa064ca624fb7a880 P 30f3ca7002424f919b063ee77bc473de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 } } peers { permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 } } peers { permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 } }
I20260709 12:19:46.078893 17376 ts_tablet_manager.cc:1434] T c55fea8d70354a8fa064ca624fb7a880 P 30f3ca7002424f919b063ee77bc473de: Time spent starting tablet: real 0.020s	user 0.005s	sys 0.000s
I20260709 12:19:46.079838 17376 tablet_bootstrap.cc:492] T ce6646ae2dfe43c681b0b1a3ed8cecd4 P 30f3ca7002424f919b063ee77bc473de: Bootstrap starting.
I20260709 12:19:46.087148 17375 ts_tablet_manager.cc:1434] T ec3f5327156e49e48ae8c88da0db0522 P 71eca61062e040d48b2398afb3b0b510: Time spent starting tablet: real 0.020s	user 0.007s	sys 0.000s
W20260709 12:19:46.087293 17132 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:19:46.088653 17375 tablet_bootstrap.cc:492] T 58783bb92efb47c2b521006941944a9d P 71eca61062e040d48b2398afb3b0b510: Bootstrap starting.
I20260709 12:19:46.088914 17376 tablet_bootstrap.cc:654] T ce6646ae2dfe43c681b0b1a3ed8cecd4 P 30f3ca7002424f919b063ee77bc473de: Neither blocks nor log segments found. Creating new log.
I20260709 12:19:46.107510 17375 tablet_bootstrap.cc:654] T 58783bb92efb47c2b521006941944a9d P 71eca61062e040d48b2398afb3b0b510: Neither blocks nor log segments found. Creating new log.
I20260709 12:19:46.116216 17377 tablet_bootstrap.cc:492] T 58783bb92efb47c2b521006941944a9d P 78285c60669649f6a5f13c1b65f9419e: No bootstrap required, opened a new log
I20260709 12:19:46.120368 17377 ts_tablet_manager.cc:1403] T 58783bb92efb47c2b521006941944a9d P 78285c60669649f6a5f13c1b65f9419e: Time spent bootstrapping tablet: real 0.054s	user 0.009s	sys 0.028s
I20260709 12:19:46.117230 17186 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103a303e4bce44cca4e34b3c391dbdac" candidate_uuid: "78285c60669649f6a5f13c1b65f9419e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30f3ca7002424f919b063ee77bc473de" is_pre_election: true
I20260709 12:19:46.119254 17375 tablet_bootstrap.cc:492] T 58783bb92efb47c2b521006941944a9d P 71eca61062e040d48b2398afb3b0b510: No bootstrap required, opened a new log
I20260709 12:19:46.121053 17375 ts_tablet_manager.cc:1403] T 58783bb92efb47c2b521006941944a9d P 71eca61062e040d48b2398afb3b0b510: Time spent bootstrapping tablet: real 0.033s	user 0.015s	sys 0.005s
I20260709 12:19:46.121371 17261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103a303e4bce44cca4e34b3c391dbdac" candidate_uuid: "78285c60669649f6a5f13c1b65f9419e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71eca61062e040d48b2398afb3b0b510" is_pre_election: true
I20260709 12:19:46.123229 17376 tablet_bootstrap.cc:492] T ce6646ae2dfe43c681b0b1a3ed8cecd4 P 30f3ca7002424f919b063ee77bc473de: No bootstrap required, opened a new log
I20260709 12:19:46.123771 17376 ts_tablet_manager.cc:1403] T ce6646ae2dfe43c681b0b1a3ed8cecd4 P 30f3ca7002424f919b063ee77bc473de: Time spent bootstrapping tablet: real 0.044s	user 0.016s	sys 0.008s
W20260709 12:19:46.123145 17301 leader_election.cc:343] T 103a303e4bce44cca4e34b3c391dbdac P 78285c60669649f6a5f13c1b65f9419e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30f3ca7002424f919b063ee77bc473de (127.16.29.65:35609): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:19:46.123329 17377 raft_consensus.cc:359] T 58783bb92efb47c2b521006941944a9d P 78285c60669649f6a5f13c1b65f9419e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 } } peers { permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 } } peers { permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 } }
W20260709 12:19:46.124572 17300 leader_election.cc:343] T 103a303e4bce44cca4e34b3c391dbdac P 78285c60669649f6a5f13c1b65f9419e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 71eca61062e040d48b2398afb3b0b510 (127.16.29.66:44237): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:19:46.124855 17375 raft_consensus.cc:359] T 58783bb92efb47c2b521006941944a9d P 71eca61062e040d48b2398afb3b0b510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 } } peers { permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 } } peers { permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 } }
I20260709 12:19:46.125137 17300 leader_election.cc:304] T 103a303e4bce44cca4e34b3c391dbdac P 78285c60669649f6a5f13c1b65f9419e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78285c60669649f6a5f13c1b65f9419e; no voters: 30f3ca7002424f919b063ee77bc473de, 71eca61062e040d48b2398afb3b0b510
I20260709 12:19:46.125600 17375 raft_consensus.cc:385] T 58783bb92efb47c2b521006941944a9d P 71eca61062e040d48b2398afb3b0b510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:19:46.125911 17375 raft_consensus.cc:740] T 58783bb92efb47c2b521006941944a9d P 71eca61062e040d48b2398afb3b0b510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71eca61062e040d48b2398afb3b0b510, State: Initialized, Role: FOLLOWER
I20260709 12:19:46.126080 17388 raft_consensus.cc:2749] T 103a303e4bce44cca4e34b3c391dbdac P 78285c60669649f6a5f13c1b65f9419e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:19:46.124606 17377 raft_consensus.cc:385] T 58783bb92efb47c2b521006941944a9d P 78285c60669649f6a5f13c1b65f9419e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:19:46.126513 17377 raft_consensus.cc:740] T 58783bb92efb47c2b521006941944a9d P 78285c60669649f6a5f13c1b65f9419e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78285c60669649f6a5f13c1b65f9419e, State: Initialized, Role: FOLLOWER
I20260709 12:19:46.126801 17375 consensus_queue.cc:260] T 58783bb92efb47c2b521006941944a9d P 71eca61062e040d48b2398afb3b0b510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 } } peers { permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 } } peers { permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 } }
I20260709 12:19:46.127713 17376 raft_consensus.cc:359] T ce6646ae2dfe43c681b0b1a3ed8cecd4 P 30f3ca7002424f919b063ee77bc473de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 } } peers { permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 } } peers { permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 } }
I20260709 12:19:46.127847 17377 consensus_queue.cc:260] T 58783bb92efb47c2b521006941944a9d P 78285c60669649f6a5f13c1b65f9419e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 } } peers { permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 } } peers { permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 } }
I20260709 12:19:46.128450 17376 raft_consensus.cc:385] T ce6646ae2dfe43c681b0b1a3ed8cecd4 P 30f3ca7002424f919b063ee77bc473de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:19:46.130506 17377 ts_tablet_manager.cc:1434] T 58783bb92efb47c2b521006941944a9d P 78285c60669649f6a5f13c1b65f9419e: Time spent starting tablet: real 0.010s	user 0.006s	sys 0.000s
I20260709 12:19:46.131145 17375 ts_tablet_manager.cc:1434] T 58783bb92efb47c2b521006941944a9d P 71eca61062e040d48b2398afb3b0b510: Time spent starting tablet: real 0.010s	user 0.005s	sys 0.001s
I20260709 12:19:46.131479 17377 tablet_bootstrap.cc:492] T c55fea8d70354a8fa064ca624fb7a880 P 78285c60669649f6a5f13c1b65f9419e: Bootstrap starting.
I20260709 12:19:46.132149 17375 tablet_bootstrap.cc:492] T 9f65da5ec1234214840536f7d27a5499 P 71eca61062e040d48b2398afb3b0b510: Bootstrap starting.
I20260709 12:19:46.134342 17376 raft_consensus.cc:740] T ce6646ae2dfe43c681b0b1a3ed8cecd4 P 30f3ca7002424f919b063ee77bc473de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30f3ca7002424f919b063ee77bc473de, State: Initialized, Role: FOLLOWER
I20260709 12:19:46.135344 17376 consensus_queue.cc:260] T ce6646ae2dfe43c681b0b1a3ed8cecd4 P 30f3ca7002424f919b063ee77bc473de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 } } peers { permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 } } peers { permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 } }
I20260709 12:19:46.137955 17377 tablet_bootstrap.cc:654] T c55fea8d70354a8fa064ca624fb7a880 P 78285c60669649f6a5f13c1b65f9419e: Neither blocks nor log segments found. Creating new log.
I20260709 12:19:46.139911 17376 ts_tablet_manager.cc:1434] T ce6646ae2dfe43c681b0b1a3ed8cecd4 P 30f3ca7002424f919b063ee77bc473de: Time spent starting tablet: real 0.016s	user 0.006s	sys 0.000s
I20260709 12:19:46.140259 17375 tablet_bootstrap.cc:654] T 9f65da5ec1234214840536f7d27a5499 P 71eca61062e040d48b2398afb3b0b510: Neither blocks nor log segments found. Creating new log.
I20260709 12:19:46.140993 17376 tablet_bootstrap.cc:492] T ec3f5327156e49e48ae8c88da0db0522 P 30f3ca7002424f919b063ee77bc473de: Bootstrap starting.
I20260709 12:19:46.148455 17381 raft_consensus.cc:493] T c55fea8d70354a8fa064ca624fb7a880 P 71eca61062e040d48b2398afb3b0b510 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:46.148937 17381 raft_consensus.cc:515] T c55fea8d70354a8fa064ca624fb7a880 P 71eca61062e040d48b2398afb3b0b510 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 } } peers { permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 } } peers { permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 } }
I20260709 12:19:46.151065 17381 leader_election.cc:290] T c55fea8d70354a8fa064ca624fb7a880 P 71eca61062e040d48b2398afb3b0b510 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30f3ca7002424f919b063ee77bc473de (127.16.29.65:35609), 78285c60669649f6a5f13c1b65f9419e (127.16.29.67:44055)
I20260709 12:19:46.157786 17186 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c55fea8d70354a8fa064ca624fb7a880" candidate_uuid: "71eca61062e040d48b2398afb3b0b510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30f3ca7002424f919b063ee77bc473de" is_pre_election: true
I20260709 12:19:46.157966 17335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c55fea8d70354a8fa064ca624fb7a880" candidate_uuid: "71eca61062e040d48b2398afb3b0b510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78285c60669649f6a5f13c1b65f9419e" is_pre_election: true
I20260709 12:19:46.158550 17186 raft_consensus.cc:2468] T c55fea8d70354a8fa064ca624fb7a880 P 30f3ca7002424f919b063ee77bc473de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71eca61062e040d48b2398afb3b0b510 in term 0.
I20260709 12:19:46.159468 17375 tablet_bootstrap.cc:492] T 9f65da5ec1234214840536f7d27a5499 P 71eca61062e040d48b2398afb3b0b510: No bootstrap required, opened a new log
W20260709 12:19:46.159364 17225 leader_election.cc:343] T c55fea8d70354a8fa064ca624fb7a880 P 71eca61062e040d48b2398afb3b0b510 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 78285c60669649f6a5f13c1b65f9419e (127.16.29.67:44055): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:19:46.160063 17375 ts_tablet_manager.cc:1403] T 9f65da5ec1234214840536f7d27a5499 P 71eca61062e040d48b2398afb3b0b510: Time spent bootstrapping tablet: real 0.028s	user 0.005s	sys 0.010s
I20260709 12:19:46.160545 17382 raft_consensus.cc:493] T 0e08c6dc97d34c998b47e5207487ef31 P 30f3ca7002424f919b063ee77bc473de [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:46.161052 17382 raft_consensus.cc:515] T 0e08c6dc97d34c998b47e5207487ef31 P 30f3ca7002424f919b063ee77bc473de [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 } } peers { permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 } } peers { permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 } }
I20260709 12:19:46.161803 17227 leader_election.cc:304] T c55fea8d70354a8fa064ca624fb7a880 P 71eca61062e040d48b2398afb3b0b510 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30f3ca7002424f919b063ee77bc473de, 71eca61062e040d48b2398afb3b0b510; no voters: 78285c60669649f6a5f13c1b65f9419e
I20260709 12:19:46.162993 17381 raft_consensus.cc:2804] T c55fea8d70354a8fa064ca624fb7a880 P 71eca61062e040d48b2398afb3b0b510 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:19:46.162729 17375 raft_consensus.cc:359] T 9f65da5ec1234214840536f7d27a5499 P 71eca61062e040d48b2398afb3b0b510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 } } peers { permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 } } peers { permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 } }
I20260709 12:19:46.163342 17381 raft_consensus.cc:493] T c55fea8d70354a8fa064ca624fb7a880 P 71eca61062e040d48b2398afb3b0b510 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:46.163455 17375 raft_consensus.cc:385] T 9f65da5ec1234214840536f7d27a5499 P 71eca61062e040d48b2398afb3b0b510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:19:46.163790 17375 raft_consensus.cc:740] T 9f65da5ec1234214840536f7d27a5499 P 71eca61062e040d48b2398afb3b0b510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71eca61062e040d48b2398afb3b0b510, State: Initialized, Role: FOLLOWER
I20260709 12:19:46.163765 17381 raft_consensus.cc:3060] T c55fea8d70354a8fa064ca624fb7a880 P 71eca61062e040d48b2398afb3b0b510 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:19:46.164705 17375 consensus_queue.cc:260] T 9f65da5ec1234214840536f7d27a5499 P 71eca61062e040d48b2398afb3b0b510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 } } peers { permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 } } peers { permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 } }
I20260709 12:19:46.165297 17382 leader_election.cc:290] T 0e08c6dc97d34c998b47e5207487ef31 P 30f3ca7002424f919b063ee77bc473de [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 71eca61062e040d48b2398afb3b0b510 (127.16.29.66:44237), 78285c60669649f6a5f13c1b65f9419e (127.16.29.67:44055)
I20260709 12:19:46.169713 17375 ts_tablet_manager.cc:1434] T 9f65da5ec1234214840536f7d27a5499 P 71eca61062e040d48b2398afb3b0b510: Time spent starting tablet: real 0.009s	user 0.004s	sys 0.002s
I20260709 12:19:46.170686 17375 tablet_bootstrap.cc:492] T 9c4d5228c0fc4198aaf94191977ad479 P 71eca61062e040d48b2398afb3b0b510: Bootstrap starting.
I20260709 12:19:46.174610 17376 tablet_bootstrap.cc:654] T ec3f5327156e49e48ae8c88da0db0522 P 30f3ca7002424f919b063ee77bc473de: Neither blocks nor log segments found. Creating new log.
I20260709 12:19:46.177026 17375 tablet_bootstrap.cc:654] T 9c4d5228c0fc4198aaf94191977ad479 P 71eca61062e040d48b2398afb3b0b510: Neither blocks nor log segments found. Creating new log.
I20260709 12:19:46.177835 17377 tablet_bootstrap.cc:492] T c55fea8d70354a8fa064ca624fb7a880 P 78285c60669649f6a5f13c1b65f9419e: No bootstrap required, opened a new log
I20260709 12:19:46.178407 17377 ts_tablet_manager.cc:1403] T c55fea8d70354a8fa064ca624fb7a880 P 78285c60669649f6a5f13c1b65f9419e: Time spent bootstrapping tablet: real 0.047s	user 0.014s	sys 0.003s
I20260709 12:19:46.182713 17387 raft_consensus.cc:493] T ce6646ae2dfe43c681b0b1a3ed8cecd4 P 71eca61062e040d48b2398afb3b0b510 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:46.183399 17387 raft_consensus.cc:515] T ce6646ae2dfe43c681b0b1a3ed8cecd4 P 71eca61062e040d48b2398afb3b0b510 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 } } peers { permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 } } peers { permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 } }
I20260709 12:19:46.182873 17377 raft_consensus.cc:359] T c55fea8d70354a8fa064ca624fb7a880 P 78285c60669649f6a5f13c1b65f9419e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 } } peers { permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 } } peers { permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 } }
I20260709 12:19:46.185428 17377 raft_consensus.cc:385] T c55fea8d70354a8fa064ca624fb7a880 P 78285c60669649f6a5f13c1b65f9419e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:19:46.185863 17377 raft_consensus.cc:740] T c55fea8d70354a8fa064ca624fb7a880 P 78285c60669649f6a5f13c1b65f9419e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78285c60669649f6a5f13c1b65f9419e, State: Initialized, Role: FOLLOWER
I20260709 12:19:46.186337 17387 leader_election.cc:290] T ce6646ae2dfe43c681b0b1a3ed8cecd4 P 71eca61062e040d48b2398afb3b0b510 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30f3ca7002424f919b063ee77bc473de (127.16.29.65:35609), 78285c60669649f6a5f13c1b65f9419e (127.16.29.67:44055)
I20260709 12:19:46.186833 17377 consensus_queue.cc:260] T c55fea8d70354a8fa064ca624fb7a880 P 78285c60669649f6a5f13c1b65f9419e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 } } peers { permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 } } peers { permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 } }
I20260709 12:19:46.189414 17377 ts_tablet_manager.cc:1434] T c55fea8d70354a8fa064ca624fb7a880 P 78285c60669649f6a5f13c1b65f9419e: Time spent starting tablet: real 0.011s	user 0.004s	sys 0.005s
I20260709 12:19:46.189311 17335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce6646ae2dfe43c681b0b1a3ed8cecd4" candidate_uuid: "71eca61062e040d48b2398afb3b0b510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78285c60669649f6a5f13c1b65f9419e" is_pre_election: true
I20260709 12:19:46.190157 17335 raft_consensus.cc:2468] T ce6646ae2dfe43c681b0b1a3ed8cecd4 P 78285c60669649f6a5f13c1b65f9419e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71eca61062e040d48b2398afb3b0b510 in term 0.
I20260709 12:19:46.190526 17377 tablet_bootstrap.cc:492] T 9c4d5228c0fc4198aaf94191977ad479 P 78285c60669649f6a5f13c1b65f9419e: Bootstrap starting.
I20260709 12:19:46.192013 17186 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce6646ae2dfe43c681b0b1a3ed8cecd4" candidate_uuid: "71eca61062e040d48b2398afb3b0b510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30f3ca7002424f919b063ee77bc473de" is_pre_election: true
I20260709 12:19:46.192781 17186 raft_consensus.cc:2468] T ce6646ae2dfe43c681b0b1a3ed8cecd4 P 30f3ca7002424f919b063ee77bc473de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71eca61062e040d48b2398afb3b0b510 in term 0.
I20260709 12:19:46.194449 17227 leader_election.cc:304] T ce6646ae2dfe43c681b0b1a3ed8cecd4 P 71eca61062e040d48b2398afb3b0b510 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30f3ca7002424f919b063ee77bc473de, 71eca61062e040d48b2398afb3b0b510; no voters: 
I20260709 12:19:46.194890 17382 raft_consensus.cc:493] T 0762726dc14a430cb8f27b2393feec0e P 30f3ca7002424f919b063ee77bc473de [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:46.195397 17382 raft_consensus.cc:515] T 0762726dc14a430cb8f27b2393feec0e P 30f3ca7002424f919b063ee77bc473de [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 } } peers { permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 } } peers { permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 } }
I20260709 12:19:46.196880 17387 raft_consensus.cc:493] T 58783bb92efb47c2b521006941944a9d P 71eca61062e040d48b2398afb3b0b510 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:46.197677 17382 leader_election.cc:290] T 0762726dc14a430cb8f27b2393feec0e P 30f3ca7002424f919b063ee77bc473de [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 71eca61062e040d48b2398afb3b0b510 (127.16.29.66:44237), 78285c60669649f6a5f13c1b65f9419e (127.16.29.67:44055)
I20260709 12:19:46.197551 17387 raft_consensus.cc:515] T 58783bb92efb47c2b521006941944a9d P 71eca61062e040d48b2398afb3b0b510 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 } } peers { permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 } } peers { permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 } }
I20260709 12:19:46.201699 17335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58783bb92efb47c2b521006941944a9d" candidate_uuid: "71eca61062e040d48b2398afb3b0b510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78285c60669649f6a5f13c1b65f9419e" is_pre_election: true
I20260709 12:19:46.202168 17377 tablet_bootstrap.cc:654] T 9c4d5228c0fc4198aaf94191977ad479 P 78285c60669649f6a5f13c1b65f9419e: Neither blocks nor log segments found. Creating new log.
I20260709 12:19:46.202517 17335 raft_consensus.cc:2468] T 58783bb92efb47c2b521006941944a9d P 78285c60669649f6a5f13c1b65f9419e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71eca61062e040d48b2398afb3b0b510 in term 0.
I20260709 12:19:46.207005 17225 leader_election.cc:304] T 58783bb92efb47c2b521006941944a9d P 71eca61062e040d48b2398afb3b0b510 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71eca61062e040d48b2398afb3b0b510, 78285c60669649f6a5f13c1b65f9419e; no voters: 
I20260709 12:19:46.209355 17387 leader_election.cc:290] T 58783bb92efb47c2b521006941944a9d P 71eca61062e040d48b2398afb3b0b510 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30f3ca7002424f919b063ee77bc473de (127.16.29.65:35609), 78285c60669649f6a5f13c1b65f9419e (127.16.29.67:44055)
I20260709 12:19:46.210578 17387 raft_consensus.cc:2804] T 58783bb92efb47c2b521006941944a9d P 71eca61062e040d48b2398afb3b0b510 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:19:46.215799 17387 raft_consensus.cc:493] T 58783bb92efb47c2b521006941944a9d P 71eca61062e040d48b2398afb3b0b510 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:46.216204 17387 raft_consensus.cc:3060] T 58783bb92efb47c2b521006941944a9d P 71eca61062e040d48b2398afb3b0b510 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:19:46.216069 17377 tablet_bootstrap.cc:492] T 9c4d5228c0fc4198aaf94191977ad479 P 78285c60669649f6a5f13c1b65f9419e: No bootstrap required, opened a new log
I20260709 12:19:46.217398 17377 ts_tablet_manager.cc:1403] T 9c4d5228c0fc4198aaf94191977ad479 P 78285c60669649f6a5f13c1b65f9419e: Time spent bootstrapping tablet: real 0.027s	user 0.015s	sys 0.004s
I20260709 12:19:46.214838 17376 tablet_bootstrap.cc:492] T ec3f5327156e49e48ae8c88da0db0522 P 30f3ca7002424f919b063ee77bc473de: No bootstrap required, opened a new log
I20260709 12:19:46.209692 17410 raft_consensus.cc:2804] T ce6646ae2dfe43c681b0b1a3ed8cecd4 P 71eca61062e040d48b2398afb3b0b510 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:19:46.216061 17381 raft_consensus.cc:515] T c55fea8d70354a8fa064ca624fb7a880 P 71eca61062e040d48b2398afb3b0b510 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 } } peers { permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 } } peers { permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 } }
I20260709 12:19:46.220118 17410 raft_consensus.cc:493] T ce6646ae2dfe43c681b0b1a3ed8cecd4 P 71eca61062e040d48b2398afb3b0b510 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:46.220235 17376 ts_tablet_manager.cc:1403] T ec3f5327156e49e48ae8c88da0db0522 P 30f3ca7002424f919b063ee77bc473de: Time spent bootstrapping tablet: real 0.080s	user 0.020s	sys 0.000s
I20260709 12:19:46.220523 17410 raft_consensus.cc:3060] T ce6646ae2dfe43c681b0b1a3ed8cecd4 P 71eca61062e040d48b2398afb3b0b510 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:19:46.221222 17377 raft_consensus.cc:359] T 9c4d5228c0fc4198aaf94191977ad479 P 78285c60669649f6a5f13c1b65f9419e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 } } peers { permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 } } peers { permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 } }
I20260709 12:19:46.221846 17381 leader_election.cc:290] T c55fea8d70354a8fa064ca624fb7a880 P 71eca61062e040d48b2398afb3b0b510 [CANDIDATE]: Term 1 election: Requested vote from peers 30f3ca7002424f919b063ee77bc473de (127.16.29.65:35609), 78285c60669649f6a5f13c1b65f9419e (127.16.29.67:44055)
I20260709 12:19:46.222011 17377 raft_consensus.cc:385] T 9c4d5228c0fc4198aaf94191977ad479 P 78285c60669649f6a5f13c1b65f9419e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:19:46.222470 17377 raft_consensus.cc:740] T 9c4d5228c0fc4198aaf94191977ad479 P 78285c60669649f6a5f13c1b65f9419e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78285c60669649f6a5f13c1b65f9419e, State: Initialized, Role: FOLLOWER
I20260709 12:19:46.225418 17377 consensus_queue.cc:260] T 9c4d5228c0fc4198aaf94191977ad479 P 78285c60669649f6a5f13c1b65f9419e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 } } peers { permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 } } peers { permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 } }
I20260709 12:19:46.223395 17335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c55fea8d70354a8fa064ca624fb7a880" candidate_uuid: "71eca61062e040d48b2398afb3b0b510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78285c60669649f6a5f13c1b65f9419e"
I20260709 12:19:46.228318 17335 raft_consensus.cc:3060] T c55fea8d70354a8fa064ca624fb7a880 P 78285c60669649f6a5f13c1b65f9419e [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:19:46.228287 17387 raft_consensus.cc:515] T 58783bb92efb47c2b521006941944a9d P 71eca61062e040d48b2398afb3b0b510 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 } } peers { permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 } } peers { permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 } }
I20260709 12:19:46.230465 17387 leader_election.cc:290] T 58783bb92efb47c2b521006941944a9d P 71eca61062e040d48b2398afb3b0b510 [CANDIDATE]: Term 1 election: Requested vote from peers 30f3ca7002424f919b063ee77bc473de (127.16.29.65:35609), 78285c60669649f6a5f13c1b65f9419e (127.16.29.67:44055)
I20260709 12:19:46.231617 17377 ts_tablet_manager.cc:1434] T 9c4d5228c0fc4198aaf94191977ad479 P 78285c60669649f6a5f13c1b65f9419e: Time spent starting tablet: real 0.014s	user 0.007s	sys 0.003s
I20260709 12:19:46.232306 17376 raft_consensus.cc:359] T ec3f5327156e49e48ae8c88da0db0522 P 30f3ca7002424f919b063ee77bc473de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 } } peers { permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 } } peers { permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 } }
I20260709 12:19:46.233035 17376 raft_consensus.cc:385] T ec3f5327156e49e48ae8c88da0db0522 P 30f3ca7002424f919b063ee77bc473de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:19:46.233242 17377 tablet_bootstrap.cc:492] T 0e08c6dc97d34c998b47e5207487ef31 P 78285c60669649f6a5f13c1b65f9419e: Bootstrap starting.
I20260709 12:19:46.240747 17377 tablet_bootstrap.cc:654] T 0e08c6dc97d34c998b47e5207487ef31 P 78285c60669649f6a5f13c1b65f9419e: Neither blocks nor log segments found. Creating new log.
I20260709 12:19:46.233337 17376 raft_consensus.cc:740] T ec3f5327156e49e48ae8c88da0db0522 P 30f3ca7002424f919b063ee77bc473de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30f3ca7002424f919b063ee77bc473de, State: Initialized, Role: FOLLOWER
I20260709 12:19:46.241113 17334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58783bb92efb47c2b521006941944a9d" candidate_uuid: "71eca61062e040d48b2398afb3b0b510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78285c60669649f6a5f13c1b65f9419e"
I20260709 12:19:46.241771 17334 raft_consensus.cc:3060] T 58783bb92efb47c2b521006941944a9d P 78285c60669649f6a5f13c1b65f9419e [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:19:46.241864 17376 consensus_queue.cc:260] T ec3f5327156e49e48ae8c88da0db0522 P 30f3ca7002424f919b063ee77bc473de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 } } peers { permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 } } peers { permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 } }
I20260709 12:19:46.245237 17410 raft_consensus.cc:515] T ce6646ae2dfe43c681b0b1a3ed8cecd4 P 71eca61062e040d48b2398afb3b0b510 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 } } peers { permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 } } peers { permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 } }
I20260709 12:19:46.245589 17376 ts_tablet_manager.cc:1434] T ec3f5327156e49e48ae8c88da0db0522 P 30f3ca7002424f919b063ee77bc473de: Time spent starting tablet: real 0.025s	user 0.009s	sys 0.000s
I20260709 12:19:46.247120 17376 tablet_bootstrap.cc:492] T eecaa10d24c045d3921a84ecb5501242 P 30f3ca7002424f919b063ee77bc473de: Bootstrap starting.
W20260709 12:19:46.250270 17227 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:35609, user_credentials={real_user=slave}} blocked reactor thread for 55942.5us
I20260709 12:19:46.252262 17186 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58783bb92efb47c2b521006941944a9d" candidate_uuid: "71eca61062e040d48b2398afb3b0b510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30f3ca7002424f919b063ee77bc473de" is_pre_election: true
I20260709 12:19:46.239764 17335 raft_consensus.cc:2468] T c55fea8d70354a8fa064ca624fb7a880 P 78285c60669649f6a5f13c1b65f9419e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71eca61062e040d48b2398afb3b0b510 in term 1.
I20260709 12:19:46.252921 17186 raft_consensus.cc:2468] T 58783bb92efb47c2b521006941944a9d P 30f3ca7002424f919b063ee77bc473de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71eca61062e040d48b2398afb3b0b510 in term 0.
I20260709 12:19:46.253214 17260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0762726dc14a430cb8f27b2393feec0e" candidate_uuid: "30f3ca7002424f919b063ee77bc473de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71eca61062e040d48b2398afb3b0b510" is_pre_election: true
I20260709 12:19:46.253890 17260 raft_consensus.cc:2468] T 0762726dc14a430cb8f27b2393feec0e P 71eca61062e040d48b2398afb3b0b510 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30f3ca7002424f919b063ee77bc473de in term 0.
I20260709 12:19:46.255426 17185 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c55fea8d70354a8fa064ca624fb7a880" candidate_uuid: "71eca61062e040d48b2398afb3b0b510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30f3ca7002424f919b063ee77bc473de"
I20260709 12:19:46.256085 17185 raft_consensus.cc:3060] T c55fea8d70354a8fa064ca624fb7a880 P 30f3ca7002424f919b063ee77bc473de [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:19:46.258611 17333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0762726dc14a430cb8f27b2393feec0e" candidate_uuid: "30f3ca7002424f919b063ee77bc473de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78285c60669649f6a5f13c1b65f9419e" is_pre_election: true
I20260709 12:19:46.259323 17333 raft_consensus.cc:2468] T 0762726dc14a430cb8f27b2393feec0e P 78285c60669649f6a5f13c1b65f9419e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30f3ca7002424f919b063ee77bc473de in term 0.
I20260709 12:19:46.260407 17184 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce6646ae2dfe43c681b0b1a3ed8cecd4" candidate_uuid: "71eca61062e040d48b2398afb3b0b510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30f3ca7002424f919b063ee77bc473de"
I20260709 12:19:46.261082 17184 raft_consensus.cc:3060] T ce6646ae2dfe43c681b0b1a3ed8cecd4 P 30f3ca7002424f919b063ee77bc473de [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:19:46.261024 17150 leader_election.cc:304] T 0762726dc14a430cb8f27b2393feec0e P 30f3ca7002424f919b063ee77bc473de [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30f3ca7002424f919b063ee77bc473de, 78285c60669649f6a5f13c1b65f9419e; no voters: 
I20260709 12:19:46.262216 17335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e08c6dc97d34c998b47e5207487ef31" candidate_uuid: "30f3ca7002424f919b063ee77bc473de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78285c60669649f6a5f13c1b65f9419e" is_pre_election: true
W20260709 12:19:46.263537 17150 leader_election.cc:343] T 0e08c6dc97d34c998b47e5207487ef31 P 30f3ca7002424f919b063ee77bc473de [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 78285c60669649f6a5f13c1b65f9419e (127.16.29.67:44055): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:19:46.263749 17376 tablet_bootstrap.cc:654] T eecaa10d24c045d3921a84ecb5501242 P 30f3ca7002424f919b063ee77bc473de: Neither blocks nor log segments found. Creating new log.
I20260709 12:19:46.264819 17186 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58783bb92efb47c2b521006941944a9d" candidate_uuid: "71eca61062e040d48b2398afb3b0b510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30f3ca7002424f919b063ee77bc473de"
I20260709 12:19:46.265581 17186 raft_consensus.cc:3060] T 58783bb92efb47c2b521006941944a9d P 30f3ca7002424f919b063ee77bc473de [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:19:46.253903 17261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e08c6dc97d34c998b47e5207487ef31" candidate_uuid: "30f3ca7002424f919b063ee77bc473de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71eca61062e040d48b2398afb3b0b510" is_pre_election: true
I20260709 12:19:46.267796 17261 raft_consensus.cc:2468] T 0e08c6dc97d34c998b47e5207487ef31 P 71eca61062e040d48b2398afb3b0b510 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30f3ca7002424f919b063ee77bc473de in term 0.
I20260709 12:19:46.264051 17382 raft_consensus.cc:2804] T 0762726dc14a430cb8f27b2393feec0e P 30f3ca7002424f919b063ee77bc473de [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:19:46.268957 17382 raft_consensus.cc:493] T 0762726dc14a430cb8f27b2393feec0e P 30f3ca7002424f919b063ee77bc473de [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:46.269450 17382 raft_consensus.cc:3060] T 0762726dc14a430cb8f27b2393feec0e P 30f3ca7002424f919b063ee77bc473de [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:19:46.272814 17335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce6646ae2dfe43c681b0b1a3ed8cecd4" candidate_uuid: "71eca61062e040d48b2398afb3b0b510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78285c60669649f6a5f13c1b65f9419e"
I20260709 12:19:46.273552 17335 raft_consensus.cc:3060] T ce6646ae2dfe43c681b0b1a3ed8cecd4 P 78285c60669649f6a5f13c1b65f9419e [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:19:46.264456 17410 leader_election.cc:290] T ce6646ae2dfe43c681b0b1a3ed8cecd4 P 71eca61062e040d48b2398afb3b0b510 [CANDIDATE]: Term 1 election: Requested vote from peers 30f3ca7002424f919b063ee77bc473de (127.16.29.65:35609), 78285c60669649f6a5f13c1b65f9419e (127.16.29.67:44055)
I20260709 12:19:46.281366 17410 raft_consensus.cc:493] T ec3f5327156e49e48ae8c88da0db0522 P 71eca61062e040d48b2398afb3b0b510 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:46.281867 17410 raft_consensus.cc:515] T ec3f5327156e49e48ae8c88da0db0522 P 71eca61062e040d48b2398afb3b0b510 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 } } peers { permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 } } peers { permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 } }
I20260709 12:19:46.283397 17375 tablet_bootstrap.cc:492] T 9c4d5228c0fc4198aaf94191977ad479 P 71eca61062e040d48b2398afb3b0b510: No bootstrap required, opened a new log
I20260709 12:19:46.283939 17375 ts_tablet_manager.cc:1403] T 9c4d5228c0fc4198aaf94191977ad479 P 71eca61062e040d48b2398afb3b0b510: Time spent bootstrapping tablet: real 0.113s	user 0.048s	sys 0.054s
I20260709 12:19:46.284175 17410 leader_election.cc:290] T ec3f5327156e49e48ae8c88da0db0522 P 71eca61062e040d48b2398afb3b0b510 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30f3ca7002424f919b063ee77bc473de (127.16.29.65:35609), 78285c60669649f6a5f13c1b65f9419e (127.16.29.67:44055)
I20260709 12:19:46.284417 17376 tablet_bootstrap.cc:492] T eecaa10d24c045d3921a84ecb5501242 P 30f3ca7002424f919b063ee77bc473de: No bootstrap required, opened a new log
I20260709 12:19:46.285038 17376 ts_tablet_manager.cc:1403] T eecaa10d24c045d3921a84ecb5501242 P 30f3ca7002424f919b063ee77bc473de: Time spent bootstrapping tablet: real 0.038s	user 0.015s	sys 0.001s
I20260709 12:19:46.286922 17375 raft_consensus.cc:359] T 9c4d5228c0fc4198aaf94191977ad479 P 71eca61062e040d48b2398afb3b0b510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 } } peers { permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 } } peers { permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 } }
I20260709 12:19:46.287717 17375 raft_consensus.cc:385] T 9c4d5228c0fc4198aaf94191977ad479 P 71eca61062e040d48b2398afb3b0b510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:19:46.288017 17375 raft_consensus.cc:740] T 9c4d5228c0fc4198aaf94191977ad479 P 71eca61062e040d48b2398afb3b0b510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71eca61062e040d48b2398afb3b0b510, State: Initialized, Role: FOLLOWER
I20260709 12:19:46.288657 17375 consensus_queue.cc:260] T 9c4d5228c0fc4198aaf94191977ad479 P 71eca61062e040d48b2398afb3b0b510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 } } peers { permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 } } peers { permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 } }
I20260709 12:19:46.288964 17376 raft_consensus.cc:359] T eecaa10d24c045d3921a84ecb5501242 P 30f3ca7002424f919b063ee77bc473de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 } } peers { permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 } } peers { permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 } }
I20260709 12:19:46.289876 17376 raft_consensus.cc:385] T eecaa10d24c045d3921a84ecb5501242 P 30f3ca7002424f919b063ee77bc473de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:19:46.290257 17376 raft_consensus.cc:740] T eecaa10d24c045d3921a84ecb5501242 P 30f3ca7002424f919b063ee77bc473de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30f3ca7002424f919b063ee77bc473de, State: Initialized, Role: FOLLOWER
I20260709 12:19:46.290656 17375 ts_tablet_manager.cc:1434] T 9c4d5228c0fc4198aaf94191977ad479 P 71eca61062e040d48b2398afb3b0b510: Time spent starting tablet: real 0.006s	user 0.003s	sys 0.000s
I20260709 12:19:46.290464 17382 raft_consensus.cc:515] T 0762726dc14a430cb8f27b2393feec0e P 30f3ca7002424f919b063ee77bc473de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 } } peers { permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 } } peers { permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 } }
I20260709 12:19:46.291623 17375 tablet_bootstrap.cc:492] T eecaa10d24c045d3921a84ecb5501242 P 71eca61062e040d48b2398afb3b0b510: Bootstrap starting.
I20260709 12:19:46.296908 17335 raft_consensus.cc:2468] T ce6646ae2dfe43c681b0b1a3ed8cecd4 P 78285c60669649f6a5f13c1b65f9419e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71eca61062e040d48b2398afb3b0b510 in term 1.
I20260709 12:19:46.298382 17334 raft_consensus.cc:2468] T 58783bb92efb47c2b521006941944a9d P 78285c60669649f6a5f13c1b65f9419e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71eca61062e040d48b2398afb3b0b510 in term 1.
I20260709 12:19:46.272030 17225 leader_election.cc:304] T c55fea8d70354a8fa064ca624fb7a880 P 71eca61062e040d48b2398afb3b0b510 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71eca61062e040d48b2398afb3b0b510, 78285c60669649f6a5f13c1b65f9419e; no voters: 
I20260709 12:19:46.298488 17151 leader_election.cc:304] T 0e08c6dc97d34c998b47e5207487ef31 P 30f3ca7002424f919b063ee77bc473de [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30f3ca7002424f919b063ee77bc473de, 71eca61062e040d48b2398afb3b0b510; no voters: 78285c60669649f6a5f13c1b65f9419e
I20260709 12:19:46.300173 17410 raft_consensus.cc:2804] T c55fea8d70354a8fa064ca624fb7a880 P 71eca61062e040d48b2398afb3b0b510 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:19:46.300211 17335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0762726dc14a430cb8f27b2393feec0e" candidate_uuid: "30f3ca7002424f919b063ee77bc473de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78285c60669649f6a5f13c1b65f9419e"
I20260709 12:19:46.300431 17225 leader_election.cc:304] T ce6646ae2dfe43c681b0b1a3ed8cecd4 P 71eca61062e040d48b2398afb3b0b510 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71eca61062e040d48b2398afb3b0b510, 78285c60669649f6a5f13c1b65f9419e; no voters: 
I20260709 12:19:46.300977 17335 raft_consensus.cc:3060] T 0762726dc14a430cb8f27b2393feec0e P 78285c60669649f6a5f13c1b65f9419e [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:19:46.301548 17408 raft_consensus.cc:2804] T ce6646ae2dfe43c681b0b1a3ed8cecd4 P 71eca61062e040d48b2398afb3b0b510 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:19:46.302109 17408 raft_consensus.cc:697] T ce6646ae2dfe43c681b0b1a3ed8cecd4 P 71eca61062e040d48b2398afb3b0b510 [term 1 LEADER]: Becoming Leader. State: Replica: 71eca61062e040d48b2398afb3b0b510, State: Running, Role: LEADER
I20260709 12:19:46.302315 17382 leader_election.cc:290] T 0762726dc14a430cb8f27b2393feec0e P 30f3ca7002424f919b063ee77bc473de [CANDIDATE]: Term 1 election: Requested vote from peers 71eca61062e040d48b2398afb3b0b510 (127.16.29.66:44237), 78285c60669649f6a5f13c1b65f9419e (127.16.29.67:44055)
I20260709 12:19:46.302745 17375 tablet_bootstrap.cc:654] T eecaa10d24c045d3921a84ecb5501242 P 71eca61062e040d48b2398afb3b0b510: Neither blocks nor log segments found. Creating new log.
I20260709 12:19:46.302898 17382 raft_consensus.cc:2804] T 0e08c6dc97d34c998b47e5207487ef31 P 30f3ca7002424f919b063ee77bc473de [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:19:46.302938 17333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ec3f5327156e49e48ae8c88da0db0522" candidate_uuid: "71eca61062e040d48b2398afb3b0b510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78285c60669649f6a5f13c1b65f9419e" is_pre_election: true
I20260709 12:19:46.303395 17382 raft_consensus.cc:493] T 0e08c6dc97d34c998b47e5207487ef31 P 30f3ca7002424f919b063ee77bc473de [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:46.303063 17408 consensus_queue.cc:237] T ce6646ae2dfe43c681b0b1a3ed8cecd4 P 71eca61062e040d48b2398afb3b0b510 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 } } peers { permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 } } peers { permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 } }
I20260709 12:19:46.296613 17184 raft_consensus.cc:2468] T ce6646ae2dfe43c681b0b1a3ed8cecd4 P 30f3ca7002424f919b063ee77bc473de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71eca61062e040d48b2398afb3b0b510 in term 1.
W20260709 12:19:46.304893 17225 leader_election.cc:343] T ec3f5327156e49e48ae8c88da0db0522 P 71eca61062e040d48b2398afb3b0b510 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 78285c60669649f6a5f13c1b65f9419e (127.16.29.67:44055): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:19:46.291514 17376 consensus_queue.cc:260] T eecaa10d24c045d3921a84ecb5501242 P 30f3ca7002424f919b063ee77bc473de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 } } peers { permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 } } peers { permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 } }
I20260709 12:19:46.307830 17376 ts_tablet_manager.cc:1434] T eecaa10d24c045d3921a84ecb5501242 P 30f3ca7002424f919b063ee77bc473de: Time spent starting tablet: real 0.022s	user 0.006s	sys 0.000s
I20260709 12:19:46.308975 17387 raft_consensus.cc:493] T 9f65da5ec1234214840536f7d27a5499 P 71eca61062e040d48b2398afb3b0b510 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:46.303808 17382 raft_consensus.cc:3060] T 0e08c6dc97d34c998b47e5207487ef31 P 30f3ca7002424f919b063ee77bc473de [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:19:46.309986 17387 raft_consensus.cc:515] T 9f65da5ec1234214840536f7d27a5499 P 71eca61062e040d48b2398afb3b0b510 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 } } peers { permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 } } peers { permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 } }
I20260709 12:19:46.309134 17225 leader_election.cc:304] T 58783bb92efb47c2b521006941944a9d P 71eca61062e040d48b2398afb3b0b510 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71eca61062e040d48b2398afb3b0b510, 78285c60669649f6a5f13c1b65f9419e; no voters: 
I20260709 12:19:46.311661 17381 raft_consensus.cc:2804] T 58783bb92efb47c2b521006941944a9d P 71eca61062e040d48b2398afb3b0b510 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:19:46.312098 17387 leader_election.cc:290] T 9f65da5ec1234214840536f7d27a5499 P 71eca61062e040d48b2398afb3b0b510 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30f3ca7002424f919b063ee77bc473de (127.16.29.65:35609), 78285c60669649f6a5f13c1b65f9419e (127.16.29.67:44055)
I20260709 12:19:46.296494 17186 raft_consensus.cc:2468] T 58783bb92efb47c2b521006941944a9d P 30f3ca7002424f919b063ee77bc473de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71eca61062e040d48b2398afb3b0b510 in term 1.
I20260709 12:19:46.314493 17376 tablet_bootstrap.cc:492] T 9f65da5ec1234214840536f7d27a5499 P 30f3ca7002424f919b063ee77bc473de: Bootstrap starting.
I20260709 12:19:46.315353 17375 tablet_bootstrap.cc:492] T eecaa10d24c045d3921a84ecb5501242 P 71eca61062e040d48b2398afb3b0b510: No bootstrap required, opened a new log
I20260709 12:19:46.315817 17375 ts_tablet_manager.cc:1403] T eecaa10d24c045d3921a84ecb5501242 P 71eca61062e040d48b2398afb3b0b510: Time spent bootstrapping tablet: real 0.024s	user 0.010s	sys 0.005s
I20260709 12:19:46.317838 17382 raft_consensus.cc:515] T 0e08c6dc97d34c998b47e5207487ef31 P 30f3ca7002424f919b063ee77bc473de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 } } peers { permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 } } peers { permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 } }
I20260709 12:19:46.318457 17375 raft_consensus.cc:359] T eecaa10d24c045d3921a84ecb5501242 P 71eca61062e040d48b2398afb3b0b510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 } } peers { permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 } } peers { permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 } }
I20260709 12:19:46.319087 17375 raft_consensus.cc:385] T eecaa10d24c045d3921a84ecb5501242 P 71eca61062e040d48b2398afb3b0b510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:19:46.319357 17375 raft_consensus.cc:740] T eecaa10d24c045d3921a84ecb5501242 P 71eca61062e040d48b2398afb3b0b510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71eca61062e040d48b2398afb3b0b510, State: Initialized, Role: FOLLOWER
I20260709 12:19:46.319373 17410 raft_consensus.cc:697] T c55fea8d70354a8fa064ca624fb7a880 P 71eca61062e040d48b2398afb3b0b510 [term 1 LEADER]: Becoming Leader. State: Replica: 71eca61062e040d48b2398afb3b0b510, State: Running, Role: LEADER
I20260709 12:19:46.320271 17375 consensus_queue.cc:260] T eecaa10d24c045d3921a84ecb5501242 P 71eca61062e040d48b2398afb3b0b510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 } } peers { permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 } } peers { permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 } }
I20260709 12:19:46.320609 17410 consensus_queue.cc:237] T c55fea8d70354a8fa064ca624fb7a880 P 71eca61062e040d48b2398afb3b0b510 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 } } peers { permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 } } peers { permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 } }
I20260709 12:19:46.321173 17334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e08c6dc97d34c998b47e5207487ef31" candidate_uuid: "30f3ca7002424f919b063ee77bc473de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78285c60669649f6a5f13c1b65f9419e"
I20260709 12:19:46.322490 17375 ts_tablet_manager.cc:1434] T eecaa10d24c045d3921a84ecb5501242 P 71eca61062e040d48b2398afb3b0b510: Time spent starting tablet: real 0.006s	user 0.004s	sys 0.001s
I20260709 12:19:46.322526 17333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f65da5ec1234214840536f7d27a5499" candidate_uuid: "71eca61062e040d48b2398afb3b0b510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78285c60669649f6a5f13c1b65f9419e" is_pre_election: true
I20260709 12:19:46.323504 17375 tablet_bootstrap.cc:492] T 103a303e4bce44cca4e34b3c391dbdac P 71eca61062e040d48b2398afb3b0b510: Bootstrap starting.
I20260709 12:19:46.324371 17335 raft_consensus.cc:2468] T 0762726dc14a430cb8f27b2393feec0e P 78285c60669649f6a5f13c1b65f9419e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30f3ca7002424f919b063ee77bc473de in term 1.
W20260709 12:19:46.324894 17150 leader_election.cc:343] T 0e08c6dc97d34c998b47e5207487ef31 P 30f3ca7002424f919b063ee77bc473de [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 78285c60669649f6a5f13c1b65f9419e (127.16.29.67:44055): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:19:46.325482 17382 leader_election.cc:290] T 0e08c6dc97d34c998b47e5207487ef31 P 30f3ca7002424f919b063ee77bc473de [CANDIDATE]: Term 1 election: Requested vote from peers 71eca61062e040d48b2398afb3b0b510 (127.16.29.66:44237), 78285c60669649f6a5f13c1b65f9419e (127.16.29.67:44055)
I20260709 12:19:46.325881 17150 leader_election.cc:304] T 0762726dc14a430cb8f27b2393feec0e P 30f3ca7002424f919b063ee77bc473de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30f3ca7002424f919b063ee77bc473de, 78285c60669649f6a5f13c1b65f9419e; no voters: 
W20260709 12:19:46.326489 17225 leader_election.cc:343] T 9f65da5ec1234214840536f7d27a5499 P 71eca61062e040d48b2398afb3b0b510 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 78285c60669649f6a5f13c1b65f9419e (127.16.29.67:44055): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:19:46.327112 17382 raft_consensus.cc:2804] T 0762726dc14a430cb8f27b2393feec0e P 30f3ca7002424f919b063ee77bc473de [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:19:46.327783 17382 raft_consensus.cc:697] T 0762726dc14a430cb8f27b2393feec0e P 30f3ca7002424f919b063ee77bc473de [term 1 LEADER]: Becoming Leader. State: Replica: 30f3ca7002424f919b063ee77bc473de, State: Running, Role: LEADER
I20260709 12:19:46.328858 17382 consensus_queue.cc:237] T 0762726dc14a430cb8f27b2393feec0e P 30f3ca7002424f919b063ee77bc473de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 } } peers { permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 } } peers { permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 } }
I20260709 12:19:46.329535 17376 tablet_bootstrap.cc:654] T 9f65da5ec1234214840536f7d27a5499 P 30f3ca7002424f919b063ee77bc473de: Neither blocks nor log segments found. Creating new log.
I20260709 12:19:46.329979 17375 tablet_bootstrap.cc:654] T 103a303e4bce44cca4e34b3c391dbdac P 71eca61062e040d48b2398afb3b0b510: Neither blocks nor log segments found. Creating new log.
I20260709 12:19:46.334607 17186 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ec3f5327156e49e48ae8c88da0db0522" candidate_uuid: "71eca61062e040d48b2398afb3b0b510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30f3ca7002424f919b063ee77bc473de" is_pre_election: true
I20260709 12:19:46.335280 17186 raft_consensus.cc:2468] T ec3f5327156e49e48ae8c88da0db0522 P 30f3ca7002424f919b063ee77bc473de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71eca61062e040d48b2398afb3b0b510 in term 0.
I20260709 12:19:46.336591 17227 leader_election.cc:304] T ec3f5327156e49e48ae8c88da0db0522 P 71eca61062e040d48b2398afb3b0b510 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30f3ca7002424f919b063ee77bc473de, 71eca61062e040d48b2398afb3b0b510; no voters: 78285c60669649f6a5f13c1b65f9419e
I20260709 12:19:46.337230 17186 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f65da5ec1234214840536f7d27a5499" candidate_uuid: "71eca61062e040d48b2398afb3b0b510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30f3ca7002424f919b063ee77bc473de" is_pre_election: true
I20260709 12:19:46.339015 17185 raft_consensus.cc:2468] T c55fea8d70354a8fa064ca624fb7a880 P 30f3ca7002424f919b063ee77bc473de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71eca61062e040d48b2398afb3b0b510 in term 1.
I20260709 12:19:46.342625 17377 tablet_bootstrap.cc:492] T 0e08c6dc97d34c998b47e5207487ef31 P 78285c60669649f6a5f13c1b65f9419e: No bootstrap required, opened a new log
I20260709 12:19:46.343189 17377 ts_tablet_manager.cc:1403] T 0e08c6dc97d34c998b47e5207487ef31 P 78285c60669649f6a5f13c1b65f9419e: Time spent bootstrapping tablet: real 0.110s	user 0.011s	sys 0.011s
I20260709 12:19:46.346771 17377 raft_consensus.cc:359] T 0e08c6dc97d34c998b47e5207487ef31 P 78285c60669649f6a5f13c1b65f9419e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 } } peers { permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 } } peers { permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 } }
I20260709 12:19:46.352288 17377 raft_consensus.cc:385] T 0e08c6dc97d34c998b47e5207487ef31 P 78285c60669649f6a5f13c1b65f9419e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:19:46.352816 17377 raft_consensus.cc:740] T 0e08c6dc97d34c998b47e5207487ef31 P 78285c60669649f6a5f13c1b65f9419e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78285c60669649f6a5f13c1b65f9419e, State: Initialized, Role: FOLLOWER
I20260709 12:19:46.354324 17376 tablet_bootstrap.cc:492] T 9f65da5ec1234214840536f7d27a5499 P 30f3ca7002424f919b063ee77bc473de: No bootstrap required, opened a new log
I20260709 12:19:46.354846 17376 ts_tablet_manager.cc:1403] T 9f65da5ec1234214840536f7d27a5499 P 30f3ca7002424f919b063ee77bc473de: Time spent bootstrapping tablet: real 0.041s	user 0.011s	sys 0.018s
I20260709 12:19:46.354089 17377 consensus_queue.cc:260] T 0e08c6dc97d34c998b47e5207487ef31 P 78285c60669649f6a5f13c1b65f9419e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 } } peers { permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 } } peers { permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 } }
I20260709 12:19:46.357501 17376 raft_consensus.cc:359] T 9f65da5ec1234214840536f7d27a5499 P 30f3ca7002424f919b063ee77bc473de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 } } peers { permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 } } peers { permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 } }
I20260709 12:19:46.358180 17376 raft_consensus.cc:385] T 9f65da5ec1234214840536f7d27a5499 P 30f3ca7002424f919b063ee77bc473de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:19:46.358438 17376 raft_consensus.cc:740] T 9f65da5ec1234214840536f7d27a5499 P 30f3ca7002424f919b063ee77bc473de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30f3ca7002424f919b063ee77bc473de, State: Initialized, Role: FOLLOWER
I20260709 12:19:46.354327 17381 raft_consensus.cc:697] T 58783bb92efb47c2b521006941944a9d P 71eca61062e040d48b2398afb3b0b510 [term 1 LEADER]: Becoming Leader. State: Replica: 71eca61062e040d48b2398afb3b0b510, State: Running, Role: LEADER
I20260709 12:19:46.359210 17376 consensus_queue.cc:260] T 9f65da5ec1234214840536f7d27a5499 P 30f3ca7002424f919b063ee77bc473de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 } } peers { permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 } } peers { permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 } }
I20260709 12:19:46.361485 17376 ts_tablet_manager.cc:1434] T 9f65da5ec1234214840536f7d27a5499 P 30f3ca7002424f919b063ee77bc473de: Time spent starting tablet: real 0.006s	user 0.007s	sys 0.000s
I20260709 12:19:46.362367 17376 tablet_bootstrap.cc:492] T 103a303e4bce44cca4e34b3c391dbdac P 30f3ca7002424f919b063ee77bc473de: Bootstrap starting.
I20260709 12:19:46.364851 17377 ts_tablet_manager.cc:1434] T 0e08c6dc97d34c998b47e5207487ef31 P 78285c60669649f6a5f13c1b65f9419e: Time spent starting tablet: real 0.021s	user 0.003s	sys 0.005s
I20260709 12:19:46.366255 17377 tablet_bootstrap.cc:492] T ec3f5327156e49e48ae8c88da0db0522 P 78285c60669649f6a5f13c1b65f9419e: Bootstrap starting.
I20260709 12:19:46.368425 17376 tablet_bootstrap.cc:654] T 103a303e4bce44cca4e34b3c391dbdac P 30f3ca7002424f919b063ee77bc473de: Neither blocks nor log segments found. Creating new log.
I20260709 12:19:46.370718 17387 raft_consensus.cc:2804] T ec3f5327156e49e48ae8c88da0db0522 P 71eca61062e040d48b2398afb3b0b510 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:19:46.371119 17387 raft_consensus.cc:493] T ec3f5327156e49e48ae8c88da0db0522 P 71eca61062e040d48b2398afb3b0b510 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:46.371531 17387 raft_consensus.cc:3060] T ec3f5327156e49e48ae8c88da0db0522 P 71eca61062e040d48b2398afb3b0b510 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:19:46.383993 17376 tablet_bootstrap.cc:492] T 103a303e4bce44cca4e34b3c391dbdac P 30f3ca7002424f919b063ee77bc473de: No bootstrap required, opened a new log
I20260709 12:19:46.384480 17376 ts_tablet_manager.cc:1403] T 103a303e4bce44cca4e34b3c391dbdac P 30f3ca7002424f919b063ee77bc473de: Time spent bootstrapping tablet: real 0.022s	user 0.013s	sys 0.004s
I20260709 12:19:46.382467 17081 catalog_manager.cc:5697] T c55fea8d70354a8fa064ca624fb7a880 P 71eca61062e040d48b2398afb3b0b510 reported cstate change: term changed from 0 to 1, leader changed from <none> to 71eca61062e040d48b2398afb3b0b510 (127.16.29.66). New cstate: current_term: 1 leader_uuid: "71eca61062e040d48b2398afb3b0b510" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 } health_report { overall_health: UNKNOWN } } }
W20260709 12:19:46.380482 17227 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:35609, user_credentials={real_user=slave}} blocked reactor thread for 43990.2us
W20260709 12:19:46.386721 17227 leader_election.cc:343] T 9f65da5ec1234214840536f7d27a5499 P 71eca61062e040d48b2398afb3b0b510 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30f3ca7002424f919b063ee77bc473de (127.16.29.65:35609): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:19:46.387125 17227 leader_election.cc:304] T 9f65da5ec1234214840536f7d27a5499 P 71eca61062e040d48b2398afb3b0b510 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 71eca61062e040d48b2398afb3b0b510; no voters: 30f3ca7002424f919b063ee77bc473de, 78285c60669649f6a5f13c1b65f9419e
I20260709 12:19:46.387161 17376 raft_consensus.cc:359] T 103a303e4bce44cca4e34b3c391dbdac P 30f3ca7002424f919b063ee77bc473de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 } } peers { permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 } } peers { permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 } }
I20260709 12:19:46.386968 17081 catalog_manager.cc:5697] T ce6646ae2dfe43c681b0b1a3ed8cecd4 P 71eca61062e040d48b2398afb3b0b510 reported cstate change: term changed from 0 to 1, leader changed from <none> to 71eca61062e040d48b2398afb3b0b510 (127.16.29.66). New cstate: current_term: 1 leader_uuid: "71eca61062e040d48b2398afb3b0b510" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 } health_report { overall_health: UNKNOWN } } }
I20260709 12:19:46.387847 17376 raft_consensus.cc:385] T 103a303e4bce44cca4e34b3c391dbdac P 30f3ca7002424f919b063ee77bc473de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:19:46.388240 17376 raft_consensus.cc:740] T 103a303e4bce44cca4e34b3c391dbdac P 30f3ca7002424f919b063ee77bc473de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30f3ca7002424f919b063ee77bc473de, State: Initialized, Role: FOLLOWER
I20260709 12:19:46.388620 17410 raft_consensus.cc:2749] T 9f65da5ec1234214840536f7d27a5499 P 71eca61062e040d48b2398afb3b0b510 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:19:46.388886 17376 consensus_queue.cc:260] T 103a303e4bce44cca4e34b3c391dbdac P 30f3ca7002424f919b063ee77bc473de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 } } peers { permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 } } peers { permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 } }
I20260709 12:19:46.390669 17376 ts_tablet_manager.cc:1434] T 103a303e4bce44cca4e34b3c391dbdac P 30f3ca7002424f919b063ee77bc473de: Time spent starting tablet: real 0.006s	user 0.004s	sys 0.000s
I20260709 12:19:46.391273 17376 tablet_bootstrap.cc:492] T 9c4d5228c0fc4198aaf94191977ad479 P 30f3ca7002424f919b063ee77bc473de: Bootstrap starting.
W20260709 12:19:46.381951 17151 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:44237, user_credentials={real_user=slave}} blocked reactor thread for 83722.6us
I20260709 12:19:46.390899 17084 catalog_manager.cc:5697] T 0762726dc14a430cb8f27b2393feec0e P 30f3ca7002424f919b063ee77bc473de reported cstate change: term changed from 0 to 1, leader changed from <none> to 30f3ca7002424f919b063ee77bc473de (127.16.29.65). New cstate: current_term: 1 leader_uuid: "30f3ca7002424f919b063ee77bc473de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 } health_report { overall_health: UNKNOWN } } }
I20260709 12:19:46.393923 17387 raft_consensus.cc:515] T ec3f5327156e49e48ae8c88da0db0522 P 71eca61062e040d48b2398afb3b0b510 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 } } peers { permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 } } peers { permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 } }
I20260709 12:19:46.384948 17416 raft_consensus.cc:493] T eecaa10d24c045d3921a84ecb5501242 P 71eca61062e040d48b2398afb3b0b510 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:46.359620 17381 consensus_queue.cc:237] T 58783bb92efb47c2b521006941944a9d P 71eca61062e040d48b2398afb3b0b510 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 } } peers { permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 } } peers { permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 } }
I20260709 12:19:46.394763 17261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0762726dc14a430cb8f27b2393feec0e" candidate_uuid: "30f3ca7002424f919b063ee77bc473de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71eca61062e040d48b2398afb3b0b510"
I20260709 12:19:46.397327 17261 raft_consensus.cc:3060] T 0762726dc14a430cb8f27b2393feec0e P 71eca61062e040d48b2398afb3b0b510 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:19:46.397565 17387 leader_election.cc:290] T ec3f5327156e49e48ae8c88da0db0522 P 71eca61062e040d48b2398afb3b0b510 [CANDIDATE]: Term 1 election: Requested vote from peers 30f3ca7002424f919b063ee77bc473de (127.16.29.65:35609), 78285c60669649f6a5f13c1b65f9419e (127.16.29.67:44055)
I20260709 12:19:46.398134 17335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ec3f5327156e49e48ae8c88da0db0522" candidate_uuid: "71eca61062e040d48b2398afb3b0b510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78285c60669649f6a5f13c1b65f9419e"
W20260709 12:19:46.399305 17225 leader_election.cc:343] T ec3f5327156e49e48ae8c88da0db0522 P 71eca61062e040d48b2398afb3b0b510 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 78285c60669649f6a5f13c1b65f9419e (127.16.29.67:44055): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:19:46.402925 17260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e08c6dc97d34c998b47e5207487ef31" candidate_uuid: "30f3ca7002424f919b063ee77bc473de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71eca61062e040d48b2398afb3b0b510"
I20260709 12:19:46.403549 17260 raft_consensus.cc:3060] T 0e08c6dc97d34c998b47e5207487ef31 P 71eca61062e040d48b2398afb3b0b510 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:19:46.405165 17185 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ec3f5327156e49e48ae8c88da0db0522" candidate_uuid: "71eca61062e040d48b2398afb3b0b510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30f3ca7002424f919b063ee77bc473de"
I20260709 12:19:46.406706 17185 raft_consensus.cc:3060] T ec3f5327156e49e48ae8c88da0db0522 P 30f3ca7002424f919b063ee77bc473de [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:19:46.395028 17416 raft_consensus.cc:515] T eecaa10d24c045d3921a84ecb5501242 P 71eca61062e040d48b2398afb3b0b510 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 } } peers { permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 } } peers { permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 } }
I20260709 12:19:46.412591 17377 tablet_bootstrap.cc:654] T ec3f5327156e49e48ae8c88da0db0522 P 78285c60669649f6a5f13c1b65f9419e: Neither blocks nor log segments found. Creating new log.
I20260709 12:19:46.414675 17335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eecaa10d24c045d3921a84ecb5501242" candidate_uuid: "71eca61062e040d48b2398afb3b0b510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78285c60669649f6a5f13c1b65f9419e" is_pre_election: true
I20260709 12:19:46.415247 17186 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eecaa10d24c045d3921a84ecb5501242" candidate_uuid: "71eca61062e040d48b2398afb3b0b510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30f3ca7002424f919b063ee77bc473de" is_pre_election: true
I20260709 12:19:46.416270 17186 raft_consensus.cc:2468] T eecaa10d24c045d3921a84ecb5501242 P 30f3ca7002424f919b063ee77bc473de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71eca61062e040d48b2398afb3b0b510 in term 0.
W20260709 12:19:46.416761 17225 leader_election.cc:343] T eecaa10d24c045d3921a84ecb5501242 P 71eca61062e040d48b2398afb3b0b510 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 78285c60669649f6a5f13c1b65f9419e (127.16.29.67:44055): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:19:46.417686 17227 leader_election.cc:304] T eecaa10d24c045d3921a84ecb5501242 P 71eca61062e040d48b2398afb3b0b510 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30f3ca7002424f919b063ee77bc473de, 71eca61062e040d48b2398afb3b0b510; no voters: 78285c60669649f6a5f13c1b65f9419e
I20260709 12:19:46.418633 17387 raft_consensus.cc:2804] T eecaa10d24c045d3921a84ecb5501242 P 71eca61062e040d48b2398afb3b0b510 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:19:46.419679 17387 raft_consensus.cc:493] T eecaa10d24c045d3921a84ecb5501242 P 71eca61062e040d48b2398afb3b0b510 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:46.420406 17387 raft_consensus.cc:3060] T eecaa10d24c045d3921a84ecb5501242 P 71eca61062e040d48b2398afb3b0b510 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:19:46.414352 17416 leader_election.cc:290] T eecaa10d24c045d3921a84ecb5501242 P 71eca61062e040d48b2398afb3b0b510 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30f3ca7002424f919b063ee77bc473de (127.16.29.65:35609), 78285c60669649f6a5f13c1b65f9419e (127.16.29.67:44055)
I20260709 12:19:46.419435 17185 raft_consensus.cc:2468] T ec3f5327156e49e48ae8c88da0db0522 P 30f3ca7002424f919b063ee77bc473de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71eca61062e040d48b2398afb3b0b510 in term 1.
I20260709 12:19:46.418570 17260 raft_consensus.cc:2468] T 0e08c6dc97d34c998b47e5207487ef31 P 71eca61062e040d48b2398afb3b0b510 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30f3ca7002424f919b063ee77bc473de in term 1.
I20260709 12:19:46.423108 17151 leader_election.cc:304] T 0e08c6dc97d34c998b47e5207487ef31 P 30f3ca7002424f919b063ee77bc473de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30f3ca7002424f919b063ee77bc473de, 71eca61062e040d48b2398afb3b0b510; no voters: 78285c60669649f6a5f13c1b65f9419e
I20260709 12:19:46.424252 17375 tablet_bootstrap.cc:492] T 103a303e4bce44cca4e34b3c391dbdac P 71eca61062e040d48b2398afb3b0b510: No bootstrap required, opened a new log
I20260709 12:19:46.424769 17375 ts_tablet_manager.cc:1403] T 103a303e4bce44cca4e34b3c391dbdac P 71eca61062e040d48b2398afb3b0b510: Time spent bootstrapping tablet: real 0.101s	user 0.028s	sys 0.043s
I20260709 12:19:46.426122 17413 raft_consensus.cc:2804] T 0e08c6dc97d34c998b47e5207487ef31 P 30f3ca7002424f919b063ee77bc473de [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:19:46.427227 17377 tablet_bootstrap.cc:492] T ec3f5327156e49e48ae8c88da0db0522 P 78285c60669649f6a5f13c1b65f9419e: No bootstrap required, opened a new log
I20260709 12:19:46.426708 17227 leader_election.cc:304] T ec3f5327156e49e48ae8c88da0db0522 P 71eca61062e040d48b2398afb3b0b510 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30f3ca7002424f919b063ee77bc473de, 71eca61062e040d48b2398afb3b0b510; no voters: 78285c60669649f6a5f13c1b65f9419e
I20260709 12:19:46.426708 17413 raft_consensus.cc:697] T 0e08c6dc97d34c998b47e5207487ef31 P 30f3ca7002424f919b063ee77bc473de [term 1 LEADER]: Becoming Leader. State: Replica: 30f3ca7002424f919b063ee77bc473de, State: Running, Role: LEADER
I20260709 12:19:46.428175 17410 raft_consensus.cc:2804] T ec3f5327156e49e48ae8c88da0db0522 P 71eca61062e040d48b2398afb3b0b510 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:19:46.428813 17410 raft_consensus.cc:697] T ec3f5327156e49e48ae8c88da0db0522 P 71eca61062e040d48b2398afb3b0b510 [term 1 LEADER]: Becoming Leader. State: Replica: 71eca61062e040d48b2398afb3b0b510, State: Running, Role: LEADER
I20260709 12:19:46.429392 17377 ts_tablet_manager.cc:1403] T ec3f5327156e49e48ae8c88da0db0522 P 78285c60669649f6a5f13c1b65f9419e: Time spent bootstrapping tablet: real 0.063s	user 0.012s	sys 0.007s
I20260709 12:19:46.429903 17410 consensus_queue.cc:237] T ec3f5327156e49e48ae8c88da0db0522 P 71eca61062e040d48b2398afb3b0b510 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 } } peers { permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 } } peers { permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 } }
I20260709 12:19:46.433552 17377 raft_consensus.cc:359] T ec3f5327156e49e48ae8c88da0db0522 P 78285c60669649f6a5f13c1b65f9419e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 } } peers { permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 } } peers { permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 } }
I20260709 12:19:46.434520 17377 raft_consensus.cc:385] T ec3f5327156e49e48ae8c88da0db0522 P 78285c60669649f6a5f13c1b65f9419e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:19:46.434907 17261 raft_consensus.cc:2468] T 0762726dc14a430cb8f27b2393feec0e P 71eca61062e040d48b2398afb3b0b510 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30f3ca7002424f919b063ee77bc473de in term 1.
I20260709 12:19:46.434924 17377 raft_consensus.cc:740] T ec3f5327156e49e48ae8c88da0db0522 P 78285c60669649f6a5f13c1b65f9419e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78285c60669649f6a5f13c1b65f9419e, State: Initialized, Role: FOLLOWER
I20260709 12:19:46.428479 17413 consensus_queue.cc:237] T 0e08c6dc97d34c998b47e5207487ef31 P 30f3ca7002424f919b063ee77bc473de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 } } peers { permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 } } peers { permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 } }
I20260709 12:19:46.436519 17377 consensus_queue.cc:260] T ec3f5327156e49e48ae8c88da0db0522 P 78285c60669649f6a5f13c1b65f9419e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 } } peers { permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 } } peers { permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 } }
I20260709 12:19:46.438786 17375 raft_consensus.cc:359] T 103a303e4bce44cca4e34b3c391dbdac P 71eca61062e040d48b2398afb3b0b510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 } } peers { permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 } } peers { permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 } }
I20260709 12:19:46.439492 17375 raft_consensus.cc:385] T 103a303e4bce44cca4e34b3c391dbdac P 71eca61062e040d48b2398afb3b0b510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:19:46.439788 17375 raft_consensus.cc:740] T 103a303e4bce44cca4e34b3c391dbdac P 71eca61062e040d48b2398afb3b0b510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71eca61062e040d48b2398afb3b0b510, State: Initialized, Role: FOLLOWER
I20260709 12:19:46.440631 17375 consensus_queue.cc:260] T 103a303e4bce44cca4e34b3c391dbdac P 71eca61062e040d48b2398afb3b0b510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 } } peers { permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 } } peers { permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 } }
I20260709 12:19:46.443663 17377 ts_tablet_manager.cc:1434] T ec3f5327156e49e48ae8c88da0db0522 P 78285c60669649f6a5f13c1b65f9419e: Time spent starting tablet: real 0.014s	user 0.007s	sys 0.000s
I20260709 12:19:46.444928 17377 tablet_bootstrap.cc:492] T 9f65da5ec1234214840536f7d27a5499 P 78285c60669649f6a5f13c1b65f9419e: Bootstrap starting.
I20260709 12:19:46.435529 17387 raft_consensus.cc:515] T eecaa10d24c045d3921a84ecb5501242 P 71eca61062e040d48b2398afb3b0b510 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 } } peers { permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 } } peers { permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 } }
I20260709 12:19:46.458376 17375 ts_tablet_manager.cc:1434] T 103a303e4bce44cca4e34b3c391dbdac P 71eca61062e040d48b2398afb3b0b510: Time spent starting tablet: real 0.033s	user 0.007s	sys 0.001s
I20260709 12:19:46.461798 17185 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eecaa10d24c045d3921a84ecb5501242" candidate_uuid: "71eca61062e040d48b2398afb3b0b510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30f3ca7002424f919b063ee77bc473de"
I20260709 12:19:46.462460 17185 raft_consensus.cc:3060] T eecaa10d24c045d3921a84ecb5501242 P 30f3ca7002424f919b063ee77bc473de [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:19:46.462819 17335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eecaa10d24c045d3921a84ecb5501242" candidate_uuid: "71eca61062e040d48b2398afb3b0b510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78285c60669649f6a5f13c1b65f9419e"
I20260709 12:19:46.463286 17376 tablet_bootstrap.cc:654] T 9c4d5228c0fc4198aaf94191977ad479 P 30f3ca7002424f919b063ee77bc473de: Neither blocks nor log segments found. Creating new log.
W20260709 12:19:46.466068 17225 leader_election.cc:343] T eecaa10d24c045d3921a84ecb5501242 P 71eca61062e040d48b2398afb3b0b510 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 78285c60669649f6a5f13c1b65f9419e (127.16.29.67:44055): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:19:46.464080 17387 leader_election.cc:290] T eecaa10d24c045d3921a84ecb5501242 P 71eca61062e040d48b2398afb3b0b510 [CANDIDATE]: Term 1 election: Requested vote from peers 30f3ca7002424f919b063ee77bc473de (127.16.29.65:35609), 78285c60669649f6a5f13c1b65f9419e (127.16.29.67:44055)
I20260709 12:19:46.472119 17377 tablet_bootstrap.cc:654] T 9f65da5ec1234214840536f7d27a5499 P 78285c60669649f6a5f13c1b65f9419e: Neither blocks nor log segments found. Creating new log.
I20260709 12:19:46.474428 17185 raft_consensus.cc:2468] T eecaa10d24c045d3921a84ecb5501242 P 30f3ca7002424f919b063ee77bc473de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71eca61062e040d48b2398afb3b0b510 in term 1.
I20260709 12:19:46.475828 17227 leader_election.cc:304] T eecaa10d24c045d3921a84ecb5501242 P 71eca61062e040d48b2398afb3b0b510 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30f3ca7002424f919b063ee77bc473de, 71eca61062e040d48b2398afb3b0b510; no voters: 78285c60669649f6a5f13c1b65f9419e
I20260709 12:19:46.476593 17376 tablet_bootstrap.cc:492] T 9c4d5228c0fc4198aaf94191977ad479 P 30f3ca7002424f919b063ee77bc473de: No bootstrap required, opened a new log
I20260709 12:19:46.477238 17376 ts_tablet_manager.cc:1403] T 9c4d5228c0fc4198aaf94191977ad479 P 30f3ca7002424f919b063ee77bc473de: Time spent bootstrapping tablet: real 0.086s	user 0.012s	sys 0.007s
I20260709 12:19:46.477268 17387 raft_consensus.cc:2804] T eecaa10d24c045d3921a84ecb5501242 P 71eca61062e040d48b2398afb3b0b510 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:19:46.478130 17387 raft_consensus.cc:697] T eecaa10d24c045d3921a84ecb5501242 P 71eca61062e040d48b2398afb3b0b510 [term 1 LEADER]: Becoming Leader. State: Replica: 71eca61062e040d48b2398afb3b0b510, State: Running, Role: LEADER
I20260709 12:19:46.478842 17387 consensus_queue.cc:237] T eecaa10d24c045d3921a84ecb5501242 P 71eca61062e040d48b2398afb3b0b510 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 } } peers { permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 } } peers { permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 } }
I20260709 12:19:46.481818 17377 tablet_bootstrap.cc:492] T 9f65da5ec1234214840536f7d27a5499 P 78285c60669649f6a5f13c1b65f9419e: No bootstrap required, opened a new log
I20260709 12:19:46.482312 17377 ts_tablet_manager.cc:1403] T 9f65da5ec1234214840536f7d27a5499 P 78285c60669649f6a5f13c1b65f9419e: Time spent bootstrapping tablet: real 0.038s	user 0.008s	sys 0.009s
I20260709 12:19:46.484359 17376 raft_consensus.cc:359] T 9c4d5228c0fc4198aaf94191977ad479 P 30f3ca7002424f919b063ee77bc473de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 } } peers { permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 } } peers { permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 } }
I20260709 12:19:46.485642 17376 raft_consensus.cc:385] T 9c4d5228c0fc4198aaf94191977ad479 P 30f3ca7002424f919b063ee77bc473de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:19:46.486045 17376 raft_consensus.cc:740] T 9c4d5228c0fc4198aaf94191977ad479 P 30f3ca7002424f919b063ee77bc473de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30f3ca7002424f919b063ee77bc473de, State: Initialized, Role: FOLLOWER
I20260709 12:19:46.487078 17376 consensus_queue.cc:260] T 9c4d5228c0fc4198aaf94191977ad479 P 30f3ca7002424f919b063ee77bc473de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 } } peers { permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 } } peers { permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 } }
I20260709 12:19:46.489478 17376 ts_tablet_manager.cc:1434] T 9c4d5228c0fc4198aaf94191977ad479 P 30f3ca7002424f919b063ee77bc473de: Time spent starting tablet: real 0.010s	user 0.007s	sys 0.000s
I20260709 12:19:46.486608 17377 raft_consensus.cc:359] T 9f65da5ec1234214840536f7d27a5499 P 78285c60669649f6a5f13c1b65f9419e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 } } peers { permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 } } peers { permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 } }
I20260709 12:19:46.491734 17377 raft_consensus.cc:385] T 9f65da5ec1234214840536f7d27a5499 P 78285c60669649f6a5f13c1b65f9419e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:19:46.492640 17377 raft_consensus.cc:740] T 9f65da5ec1234214840536f7d27a5499 P 78285c60669649f6a5f13c1b65f9419e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78285c60669649f6a5f13c1b65f9419e, State: Initialized, Role: FOLLOWER
I20260709 12:19:46.493607 17377 consensus_queue.cc:260] T 9f65da5ec1234214840536f7d27a5499 P 78285c60669649f6a5f13c1b65f9419e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 } } peers { permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 } } peers { permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 } }
I20260709 12:19:46.497195 17377 ts_tablet_manager.cc:1434] T 9f65da5ec1234214840536f7d27a5499 P 78285c60669649f6a5f13c1b65f9419e: Time spent starting tablet: real 0.014s	user 0.007s	sys 0.000s
I20260709 12:19:46.498548 17377 tablet_bootstrap.cc:492] T eecaa10d24c045d3921a84ecb5501242 P 78285c60669649f6a5f13c1b65f9419e: Bootstrap starting.
I20260709 12:19:46.506528 17377 tablet_bootstrap.cc:654] T eecaa10d24c045d3921a84ecb5501242 P 78285c60669649f6a5f13c1b65f9419e: Neither blocks nor log segments found. Creating new log.
I20260709 12:19:46.505754 17081 catalog_manager.cc:5697] T 0e08c6dc97d34c998b47e5207487ef31 P 30f3ca7002424f919b063ee77bc473de reported cstate change: term changed from 0 to 1, leader changed from <none> to 30f3ca7002424f919b063ee77bc473de (127.16.29.65). New cstate: current_term: 1 leader_uuid: "30f3ca7002424f919b063ee77bc473de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 } health_report { overall_health: UNKNOWN } } }
I20260709 12:19:46.508003 17388 raft_consensus.cc:493] T 9f65da5ec1234214840536f7d27a5499 P 78285c60669649f6a5f13c1b65f9419e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:46.508451 17388 raft_consensus.cc:515] T 9f65da5ec1234214840536f7d27a5499 P 78285c60669649f6a5f13c1b65f9419e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 } } peers { permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 } } peers { permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 } }
I20260709 12:19:46.510510 17388 leader_election.cc:290] T 9f65da5ec1234214840536f7d27a5499 P 78285c60669649f6a5f13c1b65f9419e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30f3ca7002424f919b063ee77bc473de (127.16.29.65:35609), 71eca61062e040d48b2398afb3b0b510 (127.16.29.66:44237)
I20260709 12:19:46.512854 17185 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f65da5ec1234214840536f7d27a5499" candidate_uuid: "78285c60669649f6a5f13c1b65f9419e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30f3ca7002424f919b063ee77bc473de" is_pre_election: true
I20260709 12:19:46.513707 17185 raft_consensus.cc:2468] T 9f65da5ec1234214840536f7d27a5499 P 30f3ca7002424f919b063ee77bc473de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78285c60669649f6a5f13c1b65f9419e in term 0.
I20260709 12:19:46.514928 17261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f65da5ec1234214840536f7d27a5499" candidate_uuid: "78285c60669649f6a5f13c1b65f9419e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71eca61062e040d48b2398afb3b0b510" is_pre_election: true
I20260709 12:19:46.515700 17261 raft_consensus.cc:2468] T 9f65da5ec1234214840536f7d27a5499 P 71eca61062e040d48b2398afb3b0b510 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78285c60669649f6a5f13c1b65f9419e in term 0.
I20260709 12:19:46.517323 17300 leader_election.cc:304] T 9f65da5ec1234214840536f7d27a5499 P 78285c60669649f6a5f13c1b65f9419e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71eca61062e040d48b2398afb3b0b510, 78285c60669649f6a5f13c1b65f9419e; no voters: 
I20260709 12:19:46.518468 17388 raft_consensus.cc:2804] T 9f65da5ec1234214840536f7d27a5499 P 78285c60669649f6a5f13c1b65f9419e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:19:46.518817 17388 raft_consensus.cc:493] T 9f65da5ec1234214840536f7d27a5499 P 78285c60669649f6a5f13c1b65f9419e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:46.519150 17388 raft_consensus.cc:3060] T 9f65da5ec1234214840536f7d27a5499 P 78285c60669649f6a5f13c1b65f9419e [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:19:46.520035 17377 tablet_bootstrap.cc:492] T eecaa10d24c045d3921a84ecb5501242 P 78285c60669649f6a5f13c1b65f9419e: No bootstrap required, opened a new log
I20260709 12:19:46.520479 17377 ts_tablet_manager.cc:1403] T eecaa10d24c045d3921a84ecb5501242 P 78285c60669649f6a5f13c1b65f9419e: Time spent bootstrapping tablet: real 0.022s	user 0.012s	sys 0.006s
I20260709 12:19:46.522927 17377 raft_consensus.cc:359] T eecaa10d24c045d3921a84ecb5501242 P 78285c60669649f6a5f13c1b65f9419e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 } } peers { permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 } } peers { permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 } }
I20260709 12:19:46.523588 17377 raft_consensus.cc:385] T eecaa10d24c045d3921a84ecb5501242 P 78285c60669649f6a5f13c1b65f9419e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:19:46.523847 17377 raft_consensus.cc:740] T eecaa10d24c045d3921a84ecb5501242 P 78285c60669649f6a5f13c1b65f9419e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78285c60669649f6a5f13c1b65f9419e, State: Initialized, Role: FOLLOWER
I20260709 12:19:46.524571 17377 consensus_queue.cc:260] T eecaa10d24c045d3921a84ecb5501242 P 78285c60669649f6a5f13c1b65f9419e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 } } peers { permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 } } peers { permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 } }
I20260709 12:19:46.524505 17081 catalog_manager.cc:5697] T 58783bb92efb47c2b521006941944a9d P 71eca61062e040d48b2398afb3b0b510 reported cstate change: term changed from 0 to 1, leader changed from <none> to 71eca61062e040d48b2398afb3b0b510 (127.16.29.66). New cstate: current_term: 1 leader_uuid: "71eca61062e040d48b2398afb3b0b510" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 } health_report { overall_health: UNKNOWN } } }
I20260709 12:19:46.525976 17395 raft_consensus.cc:493] T 0e08c6dc97d34c998b47e5207487ef31 P 78285c60669649f6a5f13c1b65f9419e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:46.526456 17395 raft_consensus.cc:515] T 0e08c6dc97d34c998b47e5207487ef31 P 78285c60669649f6a5f13c1b65f9419e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 } } peers { permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 } } peers { permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 } }
I20260709 12:19:46.526310 17388 raft_consensus.cc:515] T 9f65da5ec1234214840536f7d27a5499 P 78285c60669649f6a5f13c1b65f9419e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 } } peers { permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 } } peers { permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 } }
I20260709 12:19:46.526222 17081 catalog_manager.cc:5697] T eecaa10d24c045d3921a84ecb5501242 P 71eca61062e040d48b2398afb3b0b510 reported cstate change: term changed from 0 to 1, leader changed from <none> to 71eca61062e040d48b2398afb3b0b510 (127.16.29.66). New cstate: current_term: 1 leader_uuid: "71eca61062e040d48b2398afb3b0b510" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 } health_report { overall_health: UNKNOWN } } }
I20260709 12:19:46.528652 17395 leader_election.cc:290] T 0e08c6dc97d34c998b47e5207487ef31 P 78285c60669649f6a5f13c1b65f9419e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 71eca61062e040d48b2398afb3b0b510 (127.16.29.66:44237), 30f3ca7002424f919b063ee77bc473de (127.16.29.65:35609)
I20260709 12:19:46.528106 17081 catalog_manager.cc:5697] T ec3f5327156e49e48ae8c88da0db0522 P 71eca61062e040d48b2398afb3b0b510 reported cstate change: term changed from 0 to 1, leader changed from <none> to 71eca61062e040d48b2398afb3b0b510 (127.16.29.66). New cstate: current_term: 1 leader_uuid: "71eca61062e040d48b2398afb3b0b510" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 } health_report { overall_health: UNKNOWN } } }
I20260709 12:19:46.529098 17388 leader_election.cc:290] T 9f65da5ec1234214840536f7d27a5499 P 78285c60669649f6a5f13c1b65f9419e [CANDIDATE]: Term 1 election: Requested vote from peers 30f3ca7002424f919b063ee77bc473de (127.16.29.65:35609), 71eca61062e040d48b2398afb3b0b510 (127.16.29.66:44237)
I20260709 12:19:46.532681 17377 ts_tablet_manager.cc:1434] T eecaa10d24c045d3921a84ecb5501242 P 78285c60669649f6a5f13c1b65f9419e: Time spent starting tablet: real 0.012s	user 0.006s	sys 0.000s
I20260709 12:19:46.533152 17186 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f65da5ec1234214840536f7d27a5499" candidate_uuid: "78285c60669649f6a5f13c1b65f9419e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30f3ca7002424f919b063ee77bc473de"
I20260709 12:19:46.533735 17186 raft_consensus.cc:3060] T 9f65da5ec1234214840536f7d27a5499 P 30f3ca7002424f919b063ee77bc473de [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:19:46.533857 17185 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e08c6dc97d34c998b47e5207487ef31" candidate_uuid: "78285c60669649f6a5f13c1b65f9419e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30f3ca7002424f919b063ee77bc473de" is_pre_election: true
I20260709 12:19:46.539002 17261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e08c6dc97d34c998b47e5207487ef31" candidate_uuid: "78285c60669649f6a5f13c1b65f9419e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71eca61062e040d48b2398afb3b0b510" is_pre_election: true
I20260709 12:19:46.539789 17261 raft_consensus.cc:2393] T 0e08c6dc97d34c998b47e5207487ef31 P 71eca61062e040d48b2398afb3b0b510 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 78285c60669649f6a5f13c1b65f9419e in current term 1: Already voted for candidate 30f3ca7002424f919b063ee77bc473de in this term.
I20260709 12:19:46.540781 17260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f65da5ec1234214840536f7d27a5499" candidate_uuid: "78285c60669649f6a5f13c1b65f9419e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71eca61062e040d48b2398afb3b0b510"
I20260709 12:19:46.541036 17300 leader_election.cc:304] T 0e08c6dc97d34c998b47e5207487ef31 P 78285c60669649f6a5f13c1b65f9419e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78285c60669649f6a5f13c1b65f9419e; no voters: 30f3ca7002424f919b063ee77bc473de, 71eca61062e040d48b2398afb3b0b510
I20260709 12:19:46.541354 17260 raft_consensus.cc:3060] T 9f65da5ec1234214840536f7d27a5499 P 71eca61062e040d48b2398afb3b0b510 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:19:46.544234 17186 raft_consensus.cc:2468] T 9f65da5ec1234214840536f7d27a5499 P 30f3ca7002424f919b063ee77bc473de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78285c60669649f6a5f13c1b65f9419e in term 1.
I20260709 12:19:46.545169 17301 leader_election.cc:304] T 9f65da5ec1234214840536f7d27a5499 P 78285c60669649f6a5f13c1b65f9419e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30f3ca7002424f919b063ee77bc473de, 78285c60669649f6a5f13c1b65f9419e; no voters: 
I20260709 12:19:46.545832 17395 raft_consensus.cc:3060] T 0e08c6dc97d34c998b47e5207487ef31 P 78285c60669649f6a5f13c1b65f9419e [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:19:46.547066 17260 raft_consensus.cc:2468] T 9f65da5ec1234214840536f7d27a5499 P 71eca61062e040d48b2398afb3b0b510 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78285c60669649f6a5f13c1b65f9419e in term 1.
I20260709 12:19:46.548242 17388 raft_consensus.cc:2804] T 9f65da5ec1234214840536f7d27a5499 P 78285c60669649f6a5f13c1b65f9419e [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:19:46.549407 17388 raft_consensus.cc:697] T 9f65da5ec1234214840536f7d27a5499 P 78285c60669649f6a5f13c1b65f9419e [term 1 LEADER]: Becoming Leader. State: Replica: 78285c60669649f6a5f13c1b65f9419e, State: Running, Role: LEADER
I20260709 12:19:46.538524 17434 raft_consensus.cc:493] T eecaa10d24c045d3921a84ecb5501242 P 78285c60669649f6a5f13c1b65f9419e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:46.550254 17388 consensus_queue.cc:237] T 9f65da5ec1234214840536f7d27a5499 P 78285c60669649f6a5f13c1b65f9419e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 } } peers { permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 } } peers { permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 } }
I20260709 12:19:46.550227 17434 raft_consensus.cc:515] T eecaa10d24c045d3921a84ecb5501242 P 78285c60669649f6a5f13c1b65f9419e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 } } peers { permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 } } peers { permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 } }
I20260709 12:19:46.550879 17395 raft_consensus.cc:2749] T 0e08c6dc97d34c998b47e5207487ef31 P 78285c60669649f6a5f13c1b65f9419e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:19:46.554199 17186 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eecaa10d24c045d3921a84ecb5501242" candidate_uuid: "78285c60669649f6a5f13c1b65f9419e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30f3ca7002424f919b063ee77bc473de" is_pre_election: true
I20260709 12:19:46.554504 17260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eecaa10d24c045d3921a84ecb5501242" candidate_uuid: "78285c60669649f6a5f13c1b65f9419e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71eca61062e040d48b2398afb3b0b510" is_pre_election: true
I20260709 12:19:46.554976 17186 raft_consensus.cc:2393] T eecaa10d24c045d3921a84ecb5501242 P 30f3ca7002424f919b063ee77bc473de [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 78285c60669649f6a5f13c1b65f9419e in current term 1: Already voted for candidate 71eca61062e040d48b2398afb3b0b510 in this term.
I20260709 12:19:46.556272 17434 leader_election.cc:290] T eecaa10d24c045d3921a84ecb5501242 P 78285c60669649f6a5f13c1b65f9419e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30f3ca7002424f919b063ee77bc473de (127.16.29.65:35609), 71eca61062e040d48b2398afb3b0b510 (127.16.29.66:44237)
I20260709 12:19:46.556234 17301 leader_election.cc:304] T eecaa10d24c045d3921a84ecb5501242 P 78285c60669649f6a5f13c1b65f9419e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78285c60669649f6a5f13c1b65f9419e; no voters: 30f3ca7002424f919b063ee77bc473de, 71eca61062e040d48b2398afb3b0b510
I20260709 12:19:46.557087 17434 raft_consensus.cc:3060] T eecaa10d24c045d3921a84ecb5501242 P 78285c60669649f6a5f13c1b65f9419e [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:19:46.563375 17434 raft_consensus.cc:2749] T eecaa10d24c045d3921a84ecb5501242 P 78285c60669649f6a5f13c1b65f9419e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:19:46.565552 17081 catalog_manager.cc:5697] T 9f65da5ec1234214840536f7d27a5499 P 78285c60669649f6a5f13c1b65f9419e reported cstate change: term changed from 0 to 1, leader changed from <none> to 78285c60669649f6a5f13c1b65f9419e (127.16.29.67). New cstate: current_term: 1 leader_uuid: "78285c60669649f6a5f13c1b65f9419e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 } health_report { overall_health: HEALTHY } } }
I20260709 12:19:46.587843 17387 raft_consensus.cc:493] T 9c4d5228c0fc4198aaf94191977ad479 P 71eca61062e040d48b2398afb3b0b510 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:46.588388 17387 raft_consensus.cc:515] T 9c4d5228c0fc4198aaf94191977ad479 P 71eca61062e040d48b2398afb3b0b510 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 } } peers { permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 } } peers { permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 } }
I20260709 12:19:46.590044 17387 leader_election.cc:290] T 9c4d5228c0fc4198aaf94191977ad479 P 71eca61062e040d48b2398afb3b0b510 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30f3ca7002424f919b063ee77bc473de (127.16.29.65:35609), 78285c60669649f6a5f13c1b65f9419e (127.16.29.67:44055)
I20260709 12:19:46.590938 17186 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c4d5228c0fc4198aaf94191977ad479" candidate_uuid: "71eca61062e040d48b2398afb3b0b510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30f3ca7002424f919b063ee77bc473de" is_pre_election: true
I20260709 12:19:46.590996 17335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c4d5228c0fc4198aaf94191977ad479" candidate_uuid: "71eca61062e040d48b2398afb3b0b510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78285c60669649f6a5f13c1b65f9419e" is_pre_election: true
I20260709 12:19:46.591563 17186 raft_consensus.cc:2468] T 9c4d5228c0fc4198aaf94191977ad479 P 30f3ca7002424f919b063ee77bc473de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71eca61062e040d48b2398afb3b0b510 in term 0.
I20260709 12:19:46.591598 17335 raft_consensus.cc:2468] T 9c4d5228c0fc4198aaf94191977ad479 P 78285c60669649f6a5f13c1b65f9419e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71eca61062e040d48b2398afb3b0b510 in term 0.
I20260709 12:19:46.592869 17227 leader_election.cc:304] T 9c4d5228c0fc4198aaf94191977ad479 P 71eca61062e040d48b2398afb3b0b510 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30f3ca7002424f919b063ee77bc473de, 71eca61062e040d48b2398afb3b0b510; no voters: 
I20260709 12:19:46.593595 17387 raft_consensus.cc:2804] T 9c4d5228c0fc4198aaf94191977ad479 P 71eca61062e040d48b2398afb3b0b510 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:19:46.593911 17387 raft_consensus.cc:493] T 9c4d5228c0fc4198aaf94191977ad479 P 71eca61062e040d48b2398afb3b0b510 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:46.594201 17387 raft_consensus.cc:3060] T 9c4d5228c0fc4198aaf94191977ad479 P 71eca61062e040d48b2398afb3b0b510 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:19:46.598664 17387 raft_consensus.cc:515] T 9c4d5228c0fc4198aaf94191977ad479 P 71eca61062e040d48b2398afb3b0b510 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 } } peers { permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 } } peers { permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 } }
I20260709 12:19:46.600207 17387 leader_election.cc:290] T 9c4d5228c0fc4198aaf94191977ad479 P 71eca61062e040d48b2398afb3b0b510 [CANDIDATE]: Term 1 election: Requested vote from peers 30f3ca7002424f919b063ee77bc473de (127.16.29.65:35609), 78285c60669649f6a5f13c1b65f9419e (127.16.29.67:44055)
I20260709 12:19:46.601044 17186 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c4d5228c0fc4198aaf94191977ad479" candidate_uuid: "71eca61062e040d48b2398afb3b0b510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30f3ca7002424f919b063ee77bc473de"
I20260709 12:19:46.601267 17335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c4d5228c0fc4198aaf94191977ad479" candidate_uuid: "71eca61062e040d48b2398afb3b0b510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78285c60669649f6a5f13c1b65f9419e"
I20260709 12:19:46.601569 17186 raft_consensus.cc:3060] T 9c4d5228c0fc4198aaf94191977ad479 P 30f3ca7002424f919b063ee77bc473de [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:19:46.601907 17335 raft_consensus.cc:3060] T 9c4d5228c0fc4198aaf94191977ad479 P 78285c60669649f6a5f13c1b65f9419e [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:19:46.606366 17186 raft_consensus.cc:2468] T 9c4d5228c0fc4198aaf94191977ad479 P 30f3ca7002424f919b063ee77bc473de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71eca61062e040d48b2398afb3b0b510 in term 1.
I20260709 12:19:46.607034 17335 raft_consensus.cc:2468] T 9c4d5228c0fc4198aaf94191977ad479 P 78285c60669649f6a5f13c1b65f9419e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71eca61062e040d48b2398afb3b0b510 in term 1.
I20260709 12:19:46.607475 17227 leader_election.cc:304] T 9c4d5228c0fc4198aaf94191977ad479 P 71eca61062e040d48b2398afb3b0b510 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30f3ca7002424f919b063ee77bc473de, 71eca61062e040d48b2398afb3b0b510; no voters: 
I20260709 12:19:46.608496 17387 raft_consensus.cc:2804] T 9c4d5228c0fc4198aaf94191977ad479 P 71eca61062e040d48b2398afb3b0b510 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:19:46.609238 17387 raft_consensus.cc:697] T 9c4d5228c0fc4198aaf94191977ad479 P 71eca61062e040d48b2398afb3b0b510 [term 1 LEADER]: Becoming Leader. State: Replica: 71eca61062e040d48b2398afb3b0b510, State: Running, Role: LEADER
I20260709 12:19:46.609973 17387 consensus_queue.cc:237] T 9c4d5228c0fc4198aaf94191977ad479 P 71eca61062e040d48b2398afb3b0b510 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 } } peers { permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 } } peers { permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 } }
I20260709 12:19:46.617568 17081 catalog_manager.cc:5697] T 9c4d5228c0fc4198aaf94191977ad479 P 71eca61062e040d48b2398afb3b0b510 reported cstate change: term changed from 0 to 1, leader changed from <none> to 71eca61062e040d48b2398afb3b0b510 (127.16.29.66). New cstate: current_term: 1 leader_uuid: "71eca61062e040d48b2398afb3b0b510" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 } health_report { overall_health: UNKNOWN } } }
I20260709 12:19:46.648303 17387 raft_consensus.cc:493] T 103a303e4bce44cca4e34b3c391dbdac P 71eca61062e040d48b2398afb3b0b510 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:46.648844 17387 raft_consensus.cc:515] T 103a303e4bce44cca4e34b3c391dbdac P 71eca61062e040d48b2398afb3b0b510 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 } } peers { permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 } } peers { permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 } }
I20260709 12:19:46.650656 17387 leader_election.cc:290] T 103a303e4bce44cca4e34b3c391dbdac P 71eca61062e040d48b2398afb3b0b510 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30f3ca7002424f919b063ee77bc473de (127.16.29.65:35609), 78285c60669649f6a5f13c1b65f9419e (127.16.29.67:44055)
I20260709 12:19:46.651815 17335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103a303e4bce44cca4e34b3c391dbdac" candidate_uuid: "71eca61062e040d48b2398afb3b0b510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78285c60669649f6a5f13c1b65f9419e" is_pre_election: true
I20260709 12:19:46.651822 17186 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103a303e4bce44cca4e34b3c391dbdac" candidate_uuid: "71eca61062e040d48b2398afb3b0b510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30f3ca7002424f919b063ee77bc473de" is_pre_election: true
I20260709 12:19:46.652544 17335 raft_consensus.cc:2468] T 103a303e4bce44cca4e34b3c391dbdac P 78285c60669649f6a5f13c1b65f9419e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71eca61062e040d48b2398afb3b0b510 in term 0.
I20260709 12:19:46.652706 17186 raft_consensus.cc:2468] T 103a303e4bce44cca4e34b3c391dbdac P 30f3ca7002424f919b063ee77bc473de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71eca61062e040d48b2398afb3b0b510 in term 0.
I20260709 12:19:46.653738 17225 leader_election.cc:304] T 103a303e4bce44cca4e34b3c391dbdac P 71eca61062e040d48b2398afb3b0b510 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71eca61062e040d48b2398afb3b0b510, 78285c60669649f6a5f13c1b65f9419e; no voters: 
I20260709 12:19:46.654512 17387 raft_consensus.cc:2804] T 103a303e4bce44cca4e34b3c391dbdac P 71eca61062e040d48b2398afb3b0b510 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:19:46.654834 17387 raft_consensus.cc:493] T 103a303e4bce44cca4e34b3c391dbdac P 71eca61062e040d48b2398afb3b0b510 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:46.655097 17387 raft_consensus.cc:3060] T 103a303e4bce44cca4e34b3c391dbdac P 71eca61062e040d48b2398afb3b0b510 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:19:46.660358 17387 raft_consensus.cc:515] T 103a303e4bce44cca4e34b3c391dbdac P 71eca61062e040d48b2398afb3b0b510 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 } } peers { permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 } } peers { permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 } }
I20260709 12:19:46.662060 17387 leader_election.cc:290] T 103a303e4bce44cca4e34b3c391dbdac P 71eca61062e040d48b2398afb3b0b510 [CANDIDATE]: Term 1 election: Requested vote from peers 30f3ca7002424f919b063ee77bc473de (127.16.29.65:35609), 78285c60669649f6a5f13c1b65f9419e (127.16.29.67:44055)
I20260709 12:19:46.662998 17186 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103a303e4bce44cca4e34b3c391dbdac" candidate_uuid: "71eca61062e040d48b2398afb3b0b510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30f3ca7002424f919b063ee77bc473de"
I20260709 12:19:46.663056 17335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103a303e4bce44cca4e34b3c391dbdac" candidate_uuid: "71eca61062e040d48b2398afb3b0b510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78285c60669649f6a5f13c1b65f9419e"
I20260709 12:19:46.663589 17186 raft_consensus.cc:3060] T 103a303e4bce44cca4e34b3c391dbdac P 30f3ca7002424f919b063ee77bc473de [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:19:46.663678 17335 raft_consensus.cc:3060] T 103a303e4bce44cca4e34b3c391dbdac P 78285c60669649f6a5f13c1b65f9419e [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:19:46.669662 17335 raft_consensus.cc:2468] T 103a303e4bce44cca4e34b3c391dbdac P 78285c60669649f6a5f13c1b65f9419e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71eca61062e040d48b2398afb3b0b510 in term 1.
I20260709 12:19:46.669515 17186 raft_consensus.cc:2468] T 103a303e4bce44cca4e34b3c391dbdac P 30f3ca7002424f919b063ee77bc473de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71eca61062e040d48b2398afb3b0b510 in term 1.
I20260709 12:19:46.670825 17225 leader_election.cc:304] T 103a303e4bce44cca4e34b3c391dbdac P 71eca61062e040d48b2398afb3b0b510 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71eca61062e040d48b2398afb3b0b510, 78285c60669649f6a5f13c1b65f9419e; no voters: 
I20260709 12:19:46.671550 17387 raft_consensus.cc:2804] T 103a303e4bce44cca4e34b3c391dbdac P 71eca61062e040d48b2398afb3b0b510 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:19:46.672019 17387 raft_consensus.cc:697] T 103a303e4bce44cca4e34b3c391dbdac P 71eca61062e040d48b2398afb3b0b510 [term 1 LEADER]: Becoming Leader. State: Replica: 71eca61062e040d48b2398afb3b0b510, State: Running, Role: LEADER
I20260709 12:19:46.672794 17387 consensus_queue.cc:237] T 103a303e4bce44cca4e34b3c391dbdac P 71eca61062e040d48b2398afb3b0b510 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 } } peers { permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 } } peers { permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 } }
I20260709 12:19:46.679703 17081 catalog_manager.cc:5697] T 103a303e4bce44cca4e34b3c391dbdac P 71eca61062e040d48b2398afb3b0b510 reported cstate change: term changed from 0 to 1, leader changed from <none> to 71eca61062e040d48b2398afb3b0b510 (127.16.29.66). New cstate: current_term: 1 leader_uuid: "71eca61062e040d48b2398afb3b0b510" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 } health_report { overall_health: UNKNOWN } } }
I20260709 12:19:46.687093 17387 consensus_queue.cc:1048] T ce6646ae2dfe43c681b0b1a3ed8cecd4 P 71eca61062e040d48b2398afb3b0b510 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:19:46.698545 17410 consensus_queue.cc:1048] T ce6646ae2dfe43c681b0b1a3ed8cecd4 P 71eca61062e040d48b2398afb3b0b510 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:19:46.712049 17413 consensus_queue.cc:1048] T 0762726dc14a430cb8f27b2393feec0e P 30f3ca7002424f919b063ee77bc473de [LEADER]: Connected to new peer: Peer: permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:19:46.723258 17413 consensus_queue.cc:1048] T 0762726dc14a430cb8f27b2393feec0e P 30f3ca7002424f919b063ee77bc473de [LEADER]: Connected to new peer: Peer: permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:19:46.741871 17434 raft_consensus.cc:493] T ec3f5327156e49e48ae8c88da0db0522 P 78285c60669649f6a5f13c1b65f9419e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:46.742365 17434 raft_consensus.cc:515] T ec3f5327156e49e48ae8c88da0db0522 P 78285c60669649f6a5f13c1b65f9419e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 } } peers { permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 } } peers { permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 } }
I20260709 12:19:46.744109 17434 leader_election.cc:290] T ec3f5327156e49e48ae8c88da0db0522 P 78285c60669649f6a5f13c1b65f9419e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30f3ca7002424f919b063ee77bc473de (127.16.29.65:35609), 71eca61062e040d48b2398afb3b0b510 (127.16.29.66:44237)
I20260709 12:19:46.744882 17186 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ec3f5327156e49e48ae8c88da0db0522" candidate_uuid: "78285c60669649f6a5f13c1b65f9419e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30f3ca7002424f919b063ee77bc473de" is_pre_election: true
I20260709 12:19:46.744987 17260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ec3f5327156e49e48ae8c88da0db0522" candidate_uuid: "78285c60669649f6a5f13c1b65f9419e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71eca61062e040d48b2398afb3b0b510" is_pre_election: true
I20260709 12:19:46.745529 17186 raft_consensus.cc:2393] T ec3f5327156e49e48ae8c88da0db0522 P 30f3ca7002424f919b063ee77bc473de [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 78285c60669649f6a5f13c1b65f9419e in current term 1: Already voted for candidate 71eca61062e040d48b2398afb3b0b510 in this term.
I20260709 12:19:46.746421 17301 leader_election.cc:304] T ec3f5327156e49e48ae8c88da0db0522 P 78285c60669649f6a5f13c1b65f9419e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78285c60669649f6a5f13c1b65f9419e; no voters: 30f3ca7002424f919b063ee77bc473de, 71eca61062e040d48b2398afb3b0b510
I20260709 12:19:46.747040 17434 raft_consensus.cc:3060] T ec3f5327156e49e48ae8c88da0db0522 P 78285c60669649f6a5f13c1b65f9419e [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:19:46.751698 17434 raft_consensus.cc:2749] T ec3f5327156e49e48ae8c88da0db0522 P 78285c60669649f6a5f13c1b65f9419e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:19:46.810313 17260 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "9c4d5228c0fc4198aaf94191977ad479"
dest_uuid: "71eca61062e040d48b2398afb3b0b510"
mode: GRACEFUL
new_leader_uuid: "30f3ca7002424f919b063ee77bc473de"
 from {username='slave'} at 127.0.0.1:45576
I20260709 12:19:46.810858 17260 raft_consensus.cc:606] T 9c4d5228c0fc4198aaf94191977ad479 P 71eca61062e040d48b2398afb3b0b510 [term 1 LEADER]: Received request to transfer leadership to TS 30f3ca7002424f919b063ee77bc473de
I20260709 12:19:46.818634 17260 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "c55fea8d70354a8fa064ca624fb7a880"
dest_uuid: "71eca61062e040d48b2398afb3b0b510"
mode: GRACEFUL
new_leader_uuid: "30f3ca7002424f919b063ee77bc473de"
 from {username='slave'} at 127.0.0.1:45576
I20260709 12:19:46.819083 17260 raft_consensus.cc:606] T c55fea8d70354a8fa064ca624fb7a880 P 71eca61062e040d48b2398afb3b0b510 [term 1 LEADER]: Received request to transfer leadership to TS 30f3ca7002424f919b063ee77bc473de
I20260709 12:19:46.824404 17260 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "eecaa10d24c045d3921a84ecb5501242"
dest_uuid: "71eca61062e040d48b2398afb3b0b510"
mode: GRACEFUL
new_leader_uuid: "30f3ca7002424f919b063ee77bc473de"
 from {username='slave'} at 127.0.0.1:45576
I20260709 12:19:46.824889 17260 raft_consensus.cc:606] T eecaa10d24c045d3921a84ecb5501242 P 71eca61062e040d48b2398afb3b0b510 [term 1 LEADER]: Received request to transfer leadership to TS 30f3ca7002424f919b063ee77bc473de
I20260709 12:19:46.854373 17413 consensus_queue.cc:1048] T 0e08c6dc97d34c998b47e5207487ef31 P 30f3ca7002424f919b063ee77bc473de [LEADER]: Connected to new peer: Peer: permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:19:46.866528 17413 consensus_queue.cc:1048] T 0e08c6dc97d34c998b47e5207487ef31 P 30f3ca7002424f919b063ee77bc473de [LEADER]: Connected to new peer: Peer: permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:19:46.873063 17381 consensus_queue.cc:1048] T c55fea8d70354a8fa064ca624fb7a880 P 71eca61062e040d48b2398afb3b0b510 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:19:46.877038 17381 consensus_queue.cc:1048] T ec3f5327156e49e48ae8c88da0db0522 P 71eca61062e040d48b2398afb3b0b510 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:19:46.877084 17410 consensus_queue.cc:1048] T c55fea8d70354a8fa064ca624fb7a880 P 71eca61062e040d48b2398afb3b0b510 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:19:46.879637 17408 consensus_queue.cc:1048] T 58783bb92efb47c2b521006941944a9d P 71eca61062e040d48b2398afb3b0b510 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:19:46.897877 17408 raft_consensus.cc:993] T c55fea8d70354a8fa064ca624fb7a880 P 71eca61062e040d48b2398afb3b0b510: : Instructing follower 30f3ca7002424f919b063ee77bc473de to start an election
I20260709 12:19:46.898073 17387 consensus_queue.cc:1048] T eecaa10d24c045d3921a84ecb5501242 P 71eca61062e040d48b2398afb3b0b510 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:19:46.898361 17408 raft_consensus.cc:1081] T c55fea8d70354a8fa064ca624fb7a880 P 71eca61062e040d48b2398afb3b0b510 [term 1 LEADER]: Signalling peer 30f3ca7002424f919b063ee77bc473de to start an election
I20260709 12:19:46.903759 17186 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "c55fea8d70354a8fa064ca624fb7a880"
dest_uuid: "30f3ca7002424f919b063ee77bc473de"
 from {username='slave'} at 127.0.0.1:53972
I20260709 12:19:46.905087 17186 raft_consensus.cc:493] T c55fea8d70354a8fa064ca624fb7a880 P 30f3ca7002424f919b063ee77bc473de [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:19:46.905455 17186 raft_consensus.cc:3060] T c55fea8d70354a8fa064ca624fb7a880 P 30f3ca7002424f919b063ee77bc473de [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:19:46.916127 17416 consensus_queue.cc:1048] T 58783bb92efb47c2b521006941944a9d P 71eca61062e040d48b2398afb3b0b510 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:19:46.917559 17186 raft_consensus.cc:515] T c55fea8d70354a8fa064ca624fb7a880 P 30f3ca7002424f919b063ee77bc473de [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 } } peers { permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 } } peers { permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 } }
I20260709 12:19:46.924832 17186 leader_election.cc:290] T c55fea8d70354a8fa064ca624fb7a880 P 30f3ca7002424f919b063ee77bc473de [CANDIDATE]: Term 2 election: Requested vote from peers 71eca61062e040d48b2398afb3b0b510 (127.16.29.66:44237), 78285c60669649f6a5f13c1b65f9419e (127.16.29.67:44055)
I20260709 12:19:46.926762 17260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c55fea8d70354a8fa064ca624fb7a880" candidate_uuid: "30f3ca7002424f919b063ee77bc473de" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "71eca61062e040d48b2398afb3b0b510"
I20260709 12:19:46.925864 17333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c55fea8d70354a8fa064ca624fb7a880" candidate_uuid: "30f3ca7002424f919b063ee77bc473de" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "78285c60669649f6a5f13c1b65f9419e"
I20260709 12:19:46.927560 17260 raft_consensus.cc:3055] T c55fea8d70354a8fa064ca624fb7a880 P 71eca61062e040d48b2398afb3b0b510 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:19:46.927814 17333 raft_consensus.cc:3060] T c55fea8d70354a8fa064ca624fb7a880 P 78285c60669649f6a5f13c1b65f9419e [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:19:46.928092 17260 raft_consensus.cc:740] T c55fea8d70354a8fa064ca624fb7a880 P 71eca61062e040d48b2398afb3b0b510 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 71eca61062e040d48b2398afb3b0b510, State: Running, Role: LEADER
I20260709 12:19:46.934564 17260 consensus_queue.cc:260] T c55fea8d70354a8fa064ca624fb7a880 P 71eca61062e040d48b2398afb3b0b510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 } } peers { permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 } } peers { permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 } }
I20260709 12:19:46.934908 17408 consensus_queue.cc:1048] T ec3f5327156e49e48ae8c88da0db0522 P 71eca61062e040d48b2398afb3b0b510 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:19:46.935915 17260 raft_consensus.cc:3060] T c55fea8d70354a8fa064ca624fb7a880 P 71eca61062e040d48b2398afb3b0b510 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:19:46.941879 17408 consensus_queue.cc:1048] T eecaa10d24c045d3921a84ecb5501242 P 71eca61062e040d48b2398afb3b0b510 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:19:46.948665 17333 raft_consensus.cc:2468] T c55fea8d70354a8fa064ca624fb7a880 P 78285c60669649f6a5f13c1b65f9419e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30f3ca7002424f919b063ee77bc473de in term 2.
I20260709 12:19:46.951858 17260 raft_consensus.cc:2468] T c55fea8d70354a8fa064ca624fb7a880 P 71eca61062e040d48b2398afb3b0b510 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30f3ca7002424f919b063ee77bc473de in term 2.
I20260709 12:19:46.953617 17151 leader_election.cc:304] T c55fea8d70354a8fa064ca624fb7a880 P 30f3ca7002424f919b063ee77bc473de [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30f3ca7002424f919b063ee77bc473de, 71eca61062e040d48b2398afb3b0b510; no voters: 
I20260709 12:19:46.954339 17413 raft_consensus.cc:2804] T c55fea8d70354a8fa064ca624fb7a880 P 30f3ca7002424f919b063ee77bc473de [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:19:46.954803 17413 raft_consensus.cc:697] T c55fea8d70354a8fa064ca624fb7a880 P 30f3ca7002424f919b063ee77bc473de [term 2 LEADER]: Becoming Leader. State: Replica: 30f3ca7002424f919b063ee77bc473de, State: Running, Role: LEADER
I20260709 12:19:46.955523 17413 consensus_queue.cc:237] T c55fea8d70354a8fa064ca624fb7a880 P 30f3ca7002424f919b063ee77bc473de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 } } peers { permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 } } peers { permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 } }
I20260709 12:19:46.967182 17408 raft_consensus.cc:993] T eecaa10d24c045d3921a84ecb5501242 P 71eca61062e040d48b2398afb3b0b510: : Instructing follower 30f3ca7002424f919b063ee77bc473de to start an election
I20260709 12:19:46.967655 17408 raft_consensus.cc:1081] T eecaa10d24c045d3921a84ecb5501242 P 71eca61062e040d48b2398afb3b0b510 [term 1 LEADER]: Signalling peer 30f3ca7002424f919b063ee77bc473de to start an election
I20260709 12:19:46.969338 17185 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "eecaa10d24c045d3921a84ecb5501242"
dest_uuid: "30f3ca7002424f919b063ee77bc473de"
 from {username='slave'} at 127.0.0.1:53972
I20260709 12:19:46.969967 17185 raft_consensus.cc:493] T eecaa10d24c045d3921a84ecb5501242 P 30f3ca7002424f919b063ee77bc473de [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:19:46.970384 17185 raft_consensus.cc:3060] T eecaa10d24c045d3921a84ecb5501242 P 30f3ca7002424f919b063ee77bc473de [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:19:46.973784 17083 catalog_manager.cc:5697] T c55fea8d70354a8fa064ca624fb7a880 P 30f3ca7002424f919b063ee77bc473de reported cstate change: term changed from 1 to 2, leader changed from 71eca61062e040d48b2398afb3b0b510 (127.16.29.66) to 30f3ca7002424f919b063ee77bc473de (127.16.29.65). New cstate: current_term: 2 leader_uuid: "30f3ca7002424f919b063ee77bc473de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 } health_report { overall_health: UNKNOWN } } }
I20260709 12:19:46.978042 17185 raft_consensus.cc:515] T eecaa10d24c045d3921a84ecb5501242 P 30f3ca7002424f919b063ee77bc473de [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 } } peers { permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 } } peers { permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 } }
I20260709 12:19:46.980671 17185 leader_election.cc:290] T eecaa10d24c045d3921a84ecb5501242 P 30f3ca7002424f919b063ee77bc473de [CANDIDATE]: Term 2 election: Requested vote from peers 71eca61062e040d48b2398afb3b0b510 (127.16.29.66:44237), 78285c60669649f6a5f13c1b65f9419e (127.16.29.67:44055)
I20260709 12:19:46.980841 17260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eecaa10d24c045d3921a84ecb5501242" candidate_uuid: "30f3ca7002424f919b063ee77bc473de" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "71eca61062e040d48b2398afb3b0b510"
I20260709 12:19:46.981488 17260 raft_consensus.cc:3055] T eecaa10d24c045d3921a84ecb5501242 P 71eca61062e040d48b2398afb3b0b510 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:19:46.981801 17333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eecaa10d24c045d3921a84ecb5501242" candidate_uuid: "30f3ca7002424f919b063ee77bc473de" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "78285c60669649f6a5f13c1b65f9419e"
I20260709 12:19:46.981834 17260 raft_consensus.cc:740] T eecaa10d24c045d3921a84ecb5501242 P 71eca61062e040d48b2398afb3b0b510 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 71eca61062e040d48b2398afb3b0b510, State: Running, Role: LEADER
I20260709 12:19:46.982504 17333 raft_consensus.cc:3060] T eecaa10d24c045d3921a84ecb5501242 P 78285c60669649f6a5f13c1b65f9419e [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:19:46.982976 17260 consensus_queue.cc:260] T eecaa10d24c045d3921a84ecb5501242 P 71eca61062e040d48b2398afb3b0b510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 } } peers { permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 } } peers { permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 } }
I20260709 12:19:46.984277 17260 raft_consensus.cc:3060] T eecaa10d24c045d3921a84ecb5501242 P 71eca61062e040d48b2398afb3b0b510 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:19:46.989810 17333 raft_consensus.cc:2468] T eecaa10d24c045d3921a84ecb5501242 P 78285c60669649f6a5f13c1b65f9419e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30f3ca7002424f919b063ee77bc473de in term 2.
I20260709 12:19:46.990952 17150 leader_election.cc:304] T eecaa10d24c045d3921a84ecb5501242 P 30f3ca7002424f919b063ee77bc473de [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30f3ca7002424f919b063ee77bc473de, 78285c60669649f6a5f13c1b65f9419e; no voters: 
I20260709 12:19:46.991806 17445 raft_consensus.cc:2804] T eecaa10d24c045d3921a84ecb5501242 P 30f3ca7002424f919b063ee77bc473de [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:19:46.991748 17260 raft_consensus.cc:2468] T eecaa10d24c045d3921a84ecb5501242 P 71eca61062e040d48b2398afb3b0b510 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30f3ca7002424f919b063ee77bc473de in term 2.
I20260709 12:19:46.992478 17445 raft_consensus.cc:697] T eecaa10d24c045d3921a84ecb5501242 P 30f3ca7002424f919b063ee77bc473de [term 2 LEADER]: Becoming Leader. State: Replica: 30f3ca7002424f919b063ee77bc473de, State: Running, Role: LEADER
I20260709 12:19:46.993398 17445 consensus_queue.cc:237] T eecaa10d24c045d3921a84ecb5501242 P 30f3ca7002424f919b063ee77bc473de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 } } peers { permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 } } peers { permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 } }
I20260709 12:19:47.001435 17083 catalog_manager.cc:5697] T eecaa10d24c045d3921a84ecb5501242 P 30f3ca7002424f919b063ee77bc473de reported cstate change: term changed from 1 to 2, leader changed from 71eca61062e040d48b2398afb3b0b510 (127.16.29.66) to 30f3ca7002424f919b063ee77bc473de (127.16.29.65). New cstate: current_term: 2 leader_uuid: "30f3ca7002424f919b063ee77bc473de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 } health_report { overall_health: UNKNOWN } } }
I20260709 12:19:47.026031 17434 consensus_queue.cc:1048] T 9f65da5ec1234214840536f7d27a5499 P 78285c60669649f6a5f13c1b65f9419e [LEADER]: Connected to new peer: Peer: permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:19:47.039131 17434 consensus_queue.cc:1048] T 9f65da5ec1234214840536f7d27a5499 P 78285c60669649f6a5f13c1b65f9419e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:19:47.085902 17408 consensus_queue.cc:1048] T 103a303e4bce44cca4e34b3c391dbdac P 71eca61062e040d48b2398afb3b0b510 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:19:47.097543 17410 consensus_queue.cc:1048] T 103a303e4bce44cca4e34b3c391dbdac P 71eca61062e040d48b2398afb3b0b510 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:19:47.134552 17408 consensus_queue.cc:1048] T 9c4d5228c0fc4198aaf94191977ad479 P 71eca61062e040d48b2398afb3b0b510 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:19:47.143215 17387 raft_consensus.cc:993] T 9c4d5228c0fc4198aaf94191977ad479 P 71eca61062e040d48b2398afb3b0b510: : Instructing follower 30f3ca7002424f919b063ee77bc473de to start an election
I20260709 12:19:47.143745 17387 raft_consensus.cc:1081] T 9c4d5228c0fc4198aaf94191977ad479 P 71eca61062e040d48b2398afb3b0b510 [term 1 LEADER]: Signalling peer 30f3ca7002424f919b063ee77bc473de to start an election
I20260709 12:19:47.145715 17184 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "9c4d5228c0fc4198aaf94191977ad479"
dest_uuid: "30f3ca7002424f919b063ee77bc473de"
 from {username='slave'} at 127.0.0.1:53972
I20260709 12:19:47.146291 17184 raft_consensus.cc:493] T 9c4d5228c0fc4198aaf94191977ad479 P 30f3ca7002424f919b063ee77bc473de [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:19:47.146658 17184 raft_consensus.cc:3060] T 9c4d5228c0fc4198aaf94191977ad479 P 30f3ca7002424f919b063ee77bc473de [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:19:47.148437 17410 consensus_queue.cc:1048] T 9c4d5228c0fc4198aaf94191977ad479 P 71eca61062e040d48b2398afb3b0b510 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:19:47.153180 17184 raft_consensus.cc:515] T 9c4d5228c0fc4198aaf94191977ad479 P 30f3ca7002424f919b063ee77bc473de [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 } } peers { permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 } } peers { permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 } }
I20260709 12:19:47.154372 17185 raft_consensus.cc:1240] T 9c4d5228c0fc4198aaf94191977ad479 P 30f3ca7002424f919b063ee77bc473de [term 2 FOLLOWER]: Rejecting Update request from peer 71eca61062e040d48b2398afb3b0b510 for earlier term 1. Current term is 2. Ops: []
I20260709 12:19:47.156905 17260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c4d5228c0fc4198aaf94191977ad479" candidate_uuid: "30f3ca7002424f919b063ee77bc473de" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "71eca61062e040d48b2398afb3b0b510"
I20260709 12:19:47.157526 17260 raft_consensus.cc:3055] T 9c4d5228c0fc4198aaf94191977ad479 P 71eca61062e040d48b2398afb3b0b510 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:19:47.157910 17260 raft_consensus.cc:740] T 9c4d5228c0fc4198aaf94191977ad479 P 71eca61062e040d48b2398afb3b0b510 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 71eca61062e040d48b2398afb3b0b510, State: Running, Role: LEADER
I20260709 12:19:47.158777 17260 consensus_queue.cc:260] T 9c4d5228c0fc4198aaf94191977ad479 P 71eca61062e040d48b2398afb3b0b510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 } } peers { permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 } } peers { permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 } }
I20260709 12:19:47.159515 17333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c4d5228c0fc4198aaf94191977ad479" candidate_uuid: "30f3ca7002424f919b063ee77bc473de" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "78285c60669649f6a5f13c1b65f9419e"
I20260709 12:19:47.159869 17260 raft_consensus.cc:3060] T 9c4d5228c0fc4198aaf94191977ad479 P 71eca61062e040d48b2398afb3b0b510 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:19:47.160174 17333 raft_consensus.cc:3060] T 9c4d5228c0fc4198aaf94191977ad479 P 78285c60669649f6a5f13c1b65f9419e [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:19:47.164038 17184 leader_election.cc:290] T 9c4d5228c0fc4198aaf94191977ad479 P 30f3ca7002424f919b063ee77bc473de [CANDIDATE]: Term 2 election: Requested vote from peers 71eca61062e040d48b2398afb3b0b510 (127.16.29.66:44237), 78285c60669649f6a5f13c1b65f9419e (127.16.29.67:44055)
I20260709 12:19:47.167587 17333 raft_consensus.cc:2468] T 9c4d5228c0fc4198aaf94191977ad479 P 78285c60669649f6a5f13c1b65f9419e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30f3ca7002424f919b063ee77bc473de in term 2.
I20260709 12:19:47.168447 17260 raft_consensus.cc:2468] T 9c4d5228c0fc4198aaf94191977ad479 P 71eca61062e040d48b2398afb3b0b510 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30f3ca7002424f919b063ee77bc473de in term 2.
I20260709 12:19:47.168874 17150 leader_election.cc:304] T 9c4d5228c0fc4198aaf94191977ad479 P 30f3ca7002424f919b063ee77bc473de [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30f3ca7002424f919b063ee77bc473de, 78285c60669649f6a5f13c1b65f9419e; no voters: 
I20260709 12:19:47.169716 17445 raft_consensus.cc:2804] T 9c4d5228c0fc4198aaf94191977ad479 P 30f3ca7002424f919b063ee77bc473de [term 2 FOLLOWER]: Leader election won for term 2
W20260709 12:19:47.159667 17410 consensus_queue.cc:1175] T 9c4d5228c0fc4198aaf94191977ad479 P 71eca61062e040d48b2398afb3b0b510 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "30f3ca7002424f919b063ee77bc473de" responder_term: 2 status { last_received { term: 1 index: 1 } last_committed_idx: 0 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 71eca61062e040d48b2398afb3b0b510 for earlier term 1. Current term is 2. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260709 12:19:47.170239 17445 raft_consensus.cc:697] T 9c4d5228c0fc4198aaf94191977ad479 P 30f3ca7002424f919b063ee77bc473de [term 2 LEADER]: Becoming Leader. State: Replica: 30f3ca7002424f919b063ee77bc473de, State: Running, Role: LEADER
I20260709 12:19:47.171077 17445 consensus_queue.cc:237] T 9c4d5228c0fc4198aaf94191977ad479 P 30f3ca7002424f919b063ee77bc473de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 } } peers { permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 } } peers { permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 } }
I20260709 12:19:47.177732 17081 catalog_manager.cc:5697] T 9c4d5228c0fc4198aaf94191977ad479 P 30f3ca7002424f919b063ee77bc473de reported cstate change: term changed from 1 to 2, leader changed from 71eca61062e040d48b2398afb3b0b510 (127.16.29.66) to 30f3ca7002424f919b063ee77bc473de (127.16.29.65). New cstate: current_term: 2 leader_uuid: "30f3ca7002424f919b063ee77bc473de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 } health_report { overall_health: UNKNOWN } } }
I20260709 12:19:47.375227 17260 raft_consensus.cc:1275] T eecaa10d24c045d3921a84ecb5501242 P 71eca61062e040d48b2398afb3b0b510 [term 2 FOLLOWER]: Refusing update from remote peer 30f3ca7002424f919b063ee77bc473de: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:19:47.376590 17445 consensus_queue.cc:1048] T eecaa10d24c045d3921a84ecb5501242 P 30f3ca7002424f919b063ee77bc473de [LEADER]: Connected to new peer: Peer: permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:19:47.385219 17333 raft_consensus.cc:1275] T eecaa10d24c045d3921a84ecb5501242 P 78285c60669649f6a5f13c1b65f9419e [term 2 FOLLOWER]: Refusing update from remote peer 30f3ca7002424f919b063ee77bc473de: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:19:47.386828 17413 consensus_queue.cc:1048] T eecaa10d24c045d3921a84ecb5501242 P 30f3ca7002424f919b063ee77bc473de [LEADER]: Connected to new peer: Peer: permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:19:47.422981 17333 raft_consensus.cc:1275] T c55fea8d70354a8fa064ca624fb7a880 P 78285c60669649f6a5f13c1b65f9419e [term 2 FOLLOWER]: Refusing update from remote peer 30f3ca7002424f919b063ee77bc473de: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:19:47.424397 17413 consensus_queue.cc:1048] T c55fea8d70354a8fa064ca624fb7a880 P 30f3ca7002424f919b063ee77bc473de [LEADER]: Connected to new peer: Peer: permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:19:47.433845 17260 raft_consensus.cc:1275] T c55fea8d70354a8fa064ca624fb7a880 P 71eca61062e040d48b2398afb3b0b510 [term 2 FOLLOWER]: Refusing update from remote peer 30f3ca7002424f919b063ee77bc473de: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:19:47.436102 17413 consensus_queue.cc:1048] T c55fea8d70354a8fa064ca624fb7a880 P 30f3ca7002424f919b063ee77bc473de [LEADER]: Connected to new peer: Peer: permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:19:47.568562 17261 raft_consensus.cc:1275] T 9c4d5228c0fc4198aaf94191977ad479 P 71eca61062e040d48b2398afb3b0b510 [term 2 FOLLOWER]: Refusing update from remote peer 30f3ca7002424f919b063ee77bc473de: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:19:47.569799 17413 consensus_queue.cc:1048] T 9c4d5228c0fc4198aaf94191977ad479 P 30f3ca7002424f919b063ee77bc473de [LEADER]: Connected to new peer: Peer: permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:19:47.578043 17333 raft_consensus.cc:1275] T 9c4d5228c0fc4198aaf94191977ad479 P 78285c60669649f6a5f13c1b65f9419e [term 2 FOLLOWER]: Refusing update from remote peer 30f3ca7002424f919b063ee77bc473de: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:19:47.580116 17465 consensus_queue.cc:1048] T 9c4d5228c0fc4198aaf94191977ad479 P 30f3ca7002424f919b063ee77bc473de [LEADER]: Connected to new peer: Peer: permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:19:49.876984 16501 auto_leader_rebalancer-test.cc:365] The leader distribution is 
I20260709 12:19:49.877758 16501 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:19:49.899037 17184 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "0e08c6dc97d34c998b47e5207487ef31"
dest_uuid: "30f3ca7002424f919b063ee77bc473de"
mode: GRACEFUL
new_leader_uuid: "78285c60669649f6a5f13c1b65f9419e"
 from {username='slave'} at 127.0.0.1:53982
I20260709 12:19:49.899550 17184 raft_consensus.cc:606] T 0e08c6dc97d34c998b47e5207487ef31 P 30f3ca7002424f919b063ee77bc473de [term 1 LEADER]: Received request to transfer leadership to TS 78285c60669649f6a5f13c1b65f9419e
I20260709 12:19:49.902208 17184 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "eecaa10d24c045d3921a84ecb5501242"
dest_uuid: "30f3ca7002424f919b063ee77bc473de"
mode: GRACEFUL
new_leader_uuid: "78285c60669649f6a5f13c1b65f9419e"
 from {username='slave'} at 127.0.0.1:53982
I20260709 12:19:49.902745 17184 raft_consensus.cc:606] T eecaa10d24c045d3921a84ecb5501242 P 30f3ca7002424f919b063ee77bc473de [term 2 LEADER]: Received request to transfer leadership to TS 78285c60669649f6a5f13c1b65f9419e
I20260709 12:19:49.903839 16501 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260709 12:19:49.904457 16501 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:19:49.963542 17466 raft_consensus.cc:993] T eecaa10d24c045d3921a84ecb5501242 P 30f3ca7002424f919b063ee77bc473de: : Instructing follower 78285c60669649f6a5f13c1b65f9419e to start an election
I20260709 12:19:49.964067 17413 raft_consensus.cc:1081] T eecaa10d24c045d3921a84ecb5501242 P 30f3ca7002424f919b063ee77bc473de [term 2 LEADER]: Signalling peer 78285c60669649f6a5f13c1b65f9419e to start an election
I20260709 12:19:49.965677 17333 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "eecaa10d24c045d3921a84ecb5501242"
dest_uuid: "78285c60669649f6a5f13c1b65f9419e"
 from {username='slave'} at 127.0.0.1:38522
I20260709 12:19:49.966179 17333 raft_consensus.cc:493] T eecaa10d24c045d3921a84ecb5501242 P 78285c60669649f6a5f13c1b65f9419e [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:19:49.966506 17333 raft_consensus.cc:3060] T eecaa10d24c045d3921a84ecb5501242 P 78285c60669649f6a5f13c1b65f9419e [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:19:49.971236 17333 raft_consensus.cc:515] T eecaa10d24c045d3921a84ecb5501242 P 78285c60669649f6a5f13c1b65f9419e [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 } } peers { permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 } } peers { permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 } }
I20260709 12:19:49.972994 17333 leader_election.cc:290] T eecaa10d24c045d3921a84ecb5501242 P 78285c60669649f6a5f13c1b65f9419e [CANDIDATE]: Term 3 election: Requested vote from peers 30f3ca7002424f919b063ee77bc473de (127.16.29.65:35609), 71eca61062e040d48b2398afb3b0b510 (127.16.29.66:44237)
I20260709 12:19:49.974045 17184 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eecaa10d24c045d3921a84ecb5501242" candidate_uuid: "78285c60669649f6a5f13c1b65f9419e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "30f3ca7002424f919b063ee77bc473de"
I20260709 12:19:49.974184 17261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eecaa10d24c045d3921a84ecb5501242" candidate_uuid: "78285c60669649f6a5f13c1b65f9419e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "71eca61062e040d48b2398afb3b0b510"
I20260709 12:19:49.974581 17184 raft_consensus.cc:3055] T eecaa10d24c045d3921a84ecb5501242 P 30f3ca7002424f919b063ee77bc473de [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:19:49.974814 17261 raft_consensus.cc:3060] T eecaa10d24c045d3921a84ecb5501242 P 71eca61062e040d48b2398afb3b0b510 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:19:49.974937 17184 raft_consensus.cc:740] T eecaa10d24c045d3921a84ecb5501242 P 30f3ca7002424f919b063ee77bc473de [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 30f3ca7002424f919b063ee77bc473de, State: Running, Role: LEADER
I20260709 12:19:49.975929 17184 consensus_queue.cc:260] T eecaa10d24c045d3921a84ecb5501242 P 30f3ca7002424f919b063ee77bc473de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 } } peers { permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 } } peers { permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 } }
I20260709 12:19:49.977187 17184 raft_consensus.cc:3060] T eecaa10d24c045d3921a84ecb5501242 P 30f3ca7002424f919b063ee77bc473de [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:19:49.982280 17261 raft_consensus.cc:2468] T eecaa10d24c045d3921a84ecb5501242 P 71eca61062e040d48b2398afb3b0b510 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78285c60669649f6a5f13c1b65f9419e in term 3.
I20260709 12:19:49.984354 17300 leader_election.cc:304] T eecaa10d24c045d3921a84ecb5501242 P 78285c60669649f6a5f13c1b65f9419e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71eca61062e040d48b2398afb3b0b510, 78285c60669649f6a5f13c1b65f9419e; no voters: 
I20260709 12:19:49.984812 17184 raft_consensus.cc:2468] T eecaa10d24c045d3921a84ecb5501242 P 30f3ca7002424f919b063ee77bc473de [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78285c60669649f6a5f13c1b65f9419e in term 3.
I20260709 12:19:49.985198 17434 raft_consensus.cc:2804] T eecaa10d24c045d3921a84ecb5501242 P 78285c60669649f6a5f13c1b65f9419e [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:19:49.986436 17434 raft_consensus.cc:697] T eecaa10d24c045d3921a84ecb5501242 P 78285c60669649f6a5f13c1b65f9419e [term 3 LEADER]: Becoming Leader. State: Replica: 78285c60669649f6a5f13c1b65f9419e, State: Running, Role: LEADER
I20260709 12:19:49.987299 17434 consensus_queue.cc:237] T eecaa10d24c045d3921a84ecb5501242 P 78285c60669649f6a5f13c1b65f9419e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 } } peers { permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 } } peers { permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 } }
I20260709 12:19:49.994632 17083 catalog_manager.cc:5697] T eecaa10d24c045d3921a84ecb5501242 P 78285c60669649f6a5f13c1b65f9419e reported cstate change: term changed from 2 to 3, leader changed from 30f3ca7002424f919b063ee77bc473de (127.16.29.65) to 78285c60669649f6a5f13c1b65f9419e (127.16.29.67). New cstate: current_term: 3 leader_uuid: "78285c60669649f6a5f13c1b65f9419e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 } health_report { overall_health: HEALTHY } } }
I20260709 12:19:50.317278 17466 raft_consensus.cc:993] T 0e08c6dc97d34c998b47e5207487ef31 P 30f3ca7002424f919b063ee77bc473de: : Instructing follower 78285c60669649f6a5f13c1b65f9419e to start an election
I20260709 12:19:50.317773 17413 raft_consensus.cc:1081] T 0e08c6dc97d34c998b47e5207487ef31 P 30f3ca7002424f919b063ee77bc473de [term 1 LEADER]: Signalling peer 78285c60669649f6a5f13c1b65f9419e to start an election
I20260709 12:19:50.319363 17334 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "0e08c6dc97d34c998b47e5207487ef31"
dest_uuid: "78285c60669649f6a5f13c1b65f9419e"
 from {username='slave'} at 127.0.0.1:38522
I20260709 12:19:50.319832 17334 raft_consensus.cc:493] T 0e08c6dc97d34c998b47e5207487ef31 P 78285c60669649f6a5f13c1b65f9419e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:19:50.320145 17334 raft_consensus.cc:3060] T 0e08c6dc97d34c998b47e5207487ef31 P 78285c60669649f6a5f13c1b65f9419e [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:19:50.324749 17334 raft_consensus.cc:515] T 0e08c6dc97d34c998b47e5207487ef31 P 78285c60669649f6a5f13c1b65f9419e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 } } peers { permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 } } peers { permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 } }
I20260709 12:19:50.326395 17334 leader_election.cc:290] T 0e08c6dc97d34c998b47e5207487ef31 P 78285c60669649f6a5f13c1b65f9419e [CANDIDATE]: Term 2 election: Requested vote from peers 71eca61062e040d48b2398afb3b0b510 (127.16.29.66:44237), 30f3ca7002424f919b063ee77bc473de (127.16.29.65:35609)
I20260709 12:19:50.327380 17261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e08c6dc97d34c998b47e5207487ef31" candidate_uuid: "78285c60669649f6a5f13c1b65f9419e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "71eca61062e040d48b2398afb3b0b510"
I20260709 12:19:50.327440 17184 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e08c6dc97d34c998b47e5207487ef31" candidate_uuid: "78285c60669649f6a5f13c1b65f9419e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "30f3ca7002424f919b063ee77bc473de"
I20260709 12:19:50.328140 17261 raft_consensus.cc:3060] T 0e08c6dc97d34c998b47e5207487ef31 P 71eca61062e040d48b2398afb3b0b510 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:19:50.328189 17184 raft_consensus.cc:3055] T 0e08c6dc97d34c998b47e5207487ef31 P 30f3ca7002424f919b063ee77bc473de [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:19:50.328642 17184 raft_consensus.cc:740] T 0e08c6dc97d34c998b47e5207487ef31 P 30f3ca7002424f919b063ee77bc473de [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 30f3ca7002424f919b063ee77bc473de, State: Running, Role: LEADER
I20260709 12:19:50.329342 17184 consensus_queue.cc:260] T 0e08c6dc97d34c998b47e5207487ef31 P 30f3ca7002424f919b063ee77bc473de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 } } peers { permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 } } peers { permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 } }
I20260709 12:19:50.330271 17184 raft_consensus.cc:3060] T 0e08c6dc97d34c998b47e5207487ef31 P 30f3ca7002424f919b063ee77bc473de [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:19:50.333181 17261 raft_consensus.cc:2468] T 0e08c6dc97d34c998b47e5207487ef31 P 71eca61062e040d48b2398afb3b0b510 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78285c60669649f6a5f13c1b65f9419e in term 2.
I20260709 12:19:50.334219 17300 leader_election.cc:304] T 0e08c6dc97d34c998b47e5207487ef31 P 78285c60669649f6a5f13c1b65f9419e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71eca61062e040d48b2398afb3b0b510, 78285c60669649f6a5f13c1b65f9419e; no voters: 
I20260709 12:19:50.335058 17434 raft_consensus.cc:2804] T 0e08c6dc97d34c998b47e5207487ef31 P 78285c60669649f6a5f13c1b65f9419e [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:19:50.335318 17184 raft_consensus.cc:2468] T 0e08c6dc97d34c998b47e5207487ef31 P 30f3ca7002424f919b063ee77bc473de [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78285c60669649f6a5f13c1b65f9419e in term 2.
I20260709 12:19:50.335477 17434 raft_consensus.cc:697] T 0e08c6dc97d34c998b47e5207487ef31 P 78285c60669649f6a5f13c1b65f9419e [term 2 LEADER]: Becoming Leader. State: Replica: 78285c60669649f6a5f13c1b65f9419e, State: Running, Role: LEADER
I20260709 12:19:50.336426 17434 consensus_queue.cc:237] T 0e08c6dc97d34c998b47e5207487ef31 P 78285c60669649f6a5f13c1b65f9419e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 } } peers { permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 } } peers { permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 } }
I20260709 12:19:50.343142 17083 catalog_manager.cc:5697] T 0e08c6dc97d34c998b47e5207487ef31 P 78285c60669649f6a5f13c1b65f9419e reported cstate change: term changed from 1 to 2, leader changed from 30f3ca7002424f919b063ee77bc473de (127.16.29.65) to 78285c60669649f6a5f13c1b65f9419e (127.16.29.67). New cstate: current_term: 2 leader_uuid: "78285c60669649f6a5f13c1b65f9419e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 } health_report { overall_health: HEALTHY } } }
I20260709 12:19:50.397135 17261 raft_consensus.cc:1275] T eecaa10d24c045d3921a84ecb5501242 P 71eca61062e040d48b2398afb3b0b510 [term 3 FOLLOWER]: Refusing update from remote peer 78285c60669649f6a5f13c1b65f9419e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:19:50.398486 17434 consensus_queue.cc:1048] T eecaa10d24c045d3921a84ecb5501242 P 78285c60669649f6a5f13c1b65f9419e [LEADER]: Connected to new peer: Peer: permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:19:50.411804 17184 raft_consensus.cc:1275] T eecaa10d24c045d3921a84ecb5501242 P 30f3ca7002424f919b063ee77bc473de [term 3 FOLLOWER]: Refusing update from remote peer 78285c60669649f6a5f13c1b65f9419e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:19:50.413156 17469 consensus_queue.cc:1048] T eecaa10d24c045d3921a84ecb5501242 P 78285c60669649f6a5f13c1b65f9419e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:19:50.781991 17261 raft_consensus.cc:1275] T 0e08c6dc97d34c998b47e5207487ef31 P 71eca61062e040d48b2398afb3b0b510 [term 2 FOLLOWER]: Refusing update from remote peer 78285c60669649f6a5f13c1b65f9419e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:19:50.783854 17469 consensus_queue.cc:1048] T 0e08c6dc97d34c998b47e5207487ef31 P 78285c60669649f6a5f13c1b65f9419e [LEADER]: Connected to new peer: Peer: permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:19:50.795353 17184 raft_consensus.cc:1275] T 0e08c6dc97d34c998b47e5207487ef31 P 30f3ca7002424f919b063ee77bc473de [term 2 FOLLOWER]: Refusing update from remote peer 78285c60669649f6a5f13c1b65f9419e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:19:50.797156 17434 consensus_queue.cc:1048] T 0e08c6dc97d34c998b47e5207487ef31 P 78285c60669649f6a5f13c1b65f9419e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:19:50.905445 16501 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:19:50.911155 16501 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:19:50.911638 16501 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:19:51.912599 16501 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:19:51.919054 16501 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:19:51.919493 16501 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:19:52.920917 16501 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:19:52.927709 16501 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:19:52.928187 16501 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:19:53.929342 16501 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:19:53.935159 16501 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:19:53.935643 16501 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:19:54.936604 16501 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:19:54.941819 16501 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:19:54.942247 16501 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:19:55.943288 16501 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:19:55.948575 16501 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:19:55.949028 16501 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:19:56.950193 16501 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:19:56.957228 16501 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:19:56.957988 16501 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:19:57.959200 16501 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:19:57.965435 16501 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:19:57.965785 16501 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:19:58.966792 16501 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:19:58.973752 16501 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:19:58.974293 16501 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:20:00.007716 16501 auto_leader_rebalancer-test.cc:384] The leader distribution is 
I20260709 12:20:00.023350 17334 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "0e08c6dc97d34c998b47e5207487ef31"
dest_uuid: "78285c60669649f6a5f13c1b65f9419e"
mode: GRACEFUL
new_leader_uuid: "30f3ca7002424f919b063ee77bc473de"
 from {username='slave'} at 127.0.0.1:37398
I20260709 12:20:00.024014 17334 raft_consensus.cc:606] T 0e08c6dc97d34c998b47e5207487ef31 P 78285c60669649f6a5f13c1b65f9419e [term 2 LEADER]: Received request to transfer leadership to TS 30f3ca7002424f919b063ee77bc473de
I20260709 12:20:00.033774 17493 raft_consensus.cc:993] T 0e08c6dc97d34c998b47e5207487ef31 P 78285c60669649f6a5f13c1b65f9419e: : Instructing follower 30f3ca7002424f919b063ee77bc473de to start an election
I20260709 12:20:00.034359 17487 raft_consensus.cc:1081] T 0e08c6dc97d34c998b47e5207487ef31 P 78285c60669649f6a5f13c1b65f9419e [term 2 LEADER]: Signalling peer 30f3ca7002424f919b063ee77bc473de to start an election
I20260709 12:20:00.035987 17184 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "0e08c6dc97d34c998b47e5207487ef31"
dest_uuid: "30f3ca7002424f919b063ee77bc473de"
 from {username='slave'} at 127.0.0.1:53978
I20260709 12:20:00.036643 17184 raft_consensus.cc:493] T 0e08c6dc97d34c998b47e5207487ef31 P 30f3ca7002424f919b063ee77bc473de [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:20:00.037133 17184 raft_consensus.cc:3060] T 0e08c6dc97d34c998b47e5207487ef31 P 30f3ca7002424f919b063ee77bc473de [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:20:00.038218 17334 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "eecaa10d24c045d3921a84ecb5501242"
dest_uuid: "78285c60669649f6a5f13c1b65f9419e"
mode: GRACEFUL
new_leader_uuid: "30f3ca7002424f919b063ee77bc473de"
 from {username='slave'} at 127.0.0.1:37398
I20260709 12:20:00.038714 17334 raft_consensus.cc:606] T eecaa10d24c045d3921a84ecb5501242 P 78285c60669649f6a5f13c1b65f9419e [term 3 LEADER]: Received request to transfer leadership to TS 30f3ca7002424f919b063ee77bc473de
I20260709 12:20:00.044489 17184 raft_consensus.cc:515] T 0e08c6dc97d34c998b47e5207487ef31 P 30f3ca7002424f919b063ee77bc473de [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 } } peers { permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 } } peers { permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 } }
I20260709 12:20:00.045502 17260 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "ce6646ae2dfe43c681b0b1a3ed8cecd4"
dest_uuid: "71eca61062e040d48b2398afb3b0b510"
mode: GRACEFUL
new_leader_uuid: "30f3ca7002424f919b063ee77bc473de"
 from {username='slave'} at 127.0.0.1:45576
I20260709 12:20:00.045925 17260 raft_consensus.cc:606] T ce6646ae2dfe43c681b0b1a3ed8cecd4 P 71eca61062e040d48b2398afb3b0b510 [term 1 LEADER]: Received request to transfer leadership to TS 30f3ca7002424f919b063ee77bc473de
I20260709 12:20:00.046931 17184 leader_election.cc:290] T 0e08c6dc97d34c998b47e5207487ef31 P 30f3ca7002424f919b063ee77bc473de [CANDIDATE]: Term 3 election: Requested vote from peers 71eca61062e040d48b2398afb3b0b510 (127.16.29.66:44237), 78285c60669649f6a5f13c1b65f9419e (127.16.29.67:44055)
I20260709 12:20:00.047806 17260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e08c6dc97d34c998b47e5207487ef31" candidate_uuid: "30f3ca7002424f919b063ee77bc473de" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "71eca61062e040d48b2398afb3b0b510"
I20260709 12:20:00.048549 17260 raft_consensus.cc:3060] T 0e08c6dc97d34c998b47e5207487ef31 P 71eca61062e040d48b2398afb3b0b510 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:20:00.051472 17261 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "103a303e4bce44cca4e34b3c391dbdac"
dest_uuid: "71eca61062e040d48b2398afb3b0b510"
mode: GRACEFUL
new_leader_uuid: "30f3ca7002424f919b063ee77bc473de"
 from {username='slave'} at 127.0.0.1:45576
I20260709 12:20:00.052105 17261 raft_consensus.cc:606] T 103a303e4bce44cca4e34b3c391dbdac P 71eca61062e040d48b2398afb3b0b510 [term 1 LEADER]: Received request to transfer leadership to TS 30f3ca7002424f919b063ee77bc473de
I20260709 12:20:00.054047 17260 raft_consensus.cc:2468] T 0e08c6dc97d34c998b47e5207487ef31 P 71eca61062e040d48b2398afb3b0b510 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30f3ca7002424f919b063ee77bc473de in term 3.
I20260709 12:20:00.055341 17151 leader_election.cc:304] T 0e08c6dc97d34c998b47e5207487ef31 P 30f3ca7002424f919b063ee77bc473de [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30f3ca7002424f919b063ee77bc473de, 71eca61062e040d48b2398afb3b0b510; no voters: 
I20260709 12:20:00.055397 17334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e08c6dc97d34c998b47e5207487ef31" candidate_uuid: "30f3ca7002424f919b063ee77bc473de" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "78285c60669649f6a5f13c1b65f9419e"
I20260709 12:20:00.056213 17334 raft_consensus.cc:3055] T 0e08c6dc97d34c998b47e5207487ef31 P 78285c60669649f6a5f13c1b65f9419e [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:20:00.056559 17334 raft_consensus.cc:740] T 0e08c6dc97d34c998b47e5207487ef31 P 78285c60669649f6a5f13c1b65f9419e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 78285c60669649f6a5f13c1b65f9419e, State: Running, Role: LEADER
I20260709 12:20:00.056560 17489 raft_consensus.cc:2804] T 0e08c6dc97d34c998b47e5207487ef31 P 30f3ca7002424f919b063ee77bc473de [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:20:00.057574 17334 consensus_queue.cc:260] T 0e08c6dc97d34c998b47e5207487ef31 P 78285c60669649f6a5f13c1b65f9419e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 } } peers { permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 } } peers { permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 } }
I20260709 12:20:00.058022 17260 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "ec3f5327156e49e48ae8c88da0db0522"
dest_uuid: "71eca61062e040d48b2398afb3b0b510"
mode: GRACEFUL
new_leader_uuid: "30f3ca7002424f919b063ee77bc473de"
 from {username='slave'} at 127.0.0.1:45576
I20260709 12:20:00.058127 17489 raft_consensus.cc:697] T 0e08c6dc97d34c998b47e5207487ef31 P 30f3ca7002424f919b063ee77bc473de [term 3 LEADER]: Becoming Leader. State: Replica: 30f3ca7002424f919b063ee77bc473de, State: Running, Role: LEADER
I20260709 12:20:00.058678 17260 raft_consensus.cc:606] T ec3f5327156e49e48ae8c88da0db0522 P 71eca61062e040d48b2398afb3b0b510 [term 1 LEADER]: Received request to transfer leadership to TS 30f3ca7002424f919b063ee77bc473de
I20260709 12:20:00.059039 17334 raft_consensus.cc:3060] T 0e08c6dc97d34c998b47e5207487ef31 P 78285c60669649f6a5f13c1b65f9419e [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:20:00.059409 17489 consensus_queue.cc:237] T 0e08c6dc97d34c998b47e5207487ef31 P 30f3ca7002424f919b063ee77bc473de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 } } peers { permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 } } peers { permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 } }
I20260709 12:20:00.064908 17334 raft_consensus.cc:2468] T 0e08c6dc97d34c998b47e5207487ef31 P 78285c60669649f6a5f13c1b65f9419e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30f3ca7002424f919b063ee77bc473de in term 3.
I20260709 12:20:00.072317 17081 catalog_manager.cc:5697] T 0e08c6dc97d34c998b47e5207487ef31 P 30f3ca7002424f919b063ee77bc473de reported cstate change: term changed from 2 to 3, leader changed from 78285c60669649f6a5f13c1b65f9419e (127.16.29.67) to 30f3ca7002424f919b063ee77bc473de (127.16.29.65). New cstate: current_term: 3 leader_uuid: "30f3ca7002424f919b063ee77bc473de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:00.121661 17410 raft_consensus.cc:993] T 103a303e4bce44cca4e34b3c391dbdac P 71eca61062e040d48b2398afb3b0b510: : Instructing follower 30f3ca7002424f919b063ee77bc473de to start an election
I20260709 12:20:00.122201 17500 raft_consensus.cc:1081] T 103a303e4bce44cca4e34b3c391dbdac P 71eca61062e040d48b2398afb3b0b510 [term 1 LEADER]: Signalling peer 30f3ca7002424f919b063ee77bc473de to start an election
I20260709 12:20:00.123708 17184 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "103a303e4bce44cca4e34b3c391dbdac"
dest_uuid: "30f3ca7002424f919b063ee77bc473de"
 from {username='slave'} at 127.0.0.1:53972
I20260709 12:20:00.124423 17184 raft_consensus.cc:493] T 103a303e4bce44cca4e34b3c391dbdac P 30f3ca7002424f919b063ee77bc473de [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:20:00.124831 17184 raft_consensus.cc:3060] T 103a303e4bce44cca4e34b3c391dbdac P 30f3ca7002424f919b063ee77bc473de [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:00.129873 17184 raft_consensus.cc:515] T 103a303e4bce44cca4e34b3c391dbdac P 30f3ca7002424f919b063ee77bc473de [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 } } peers { permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 } } peers { permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 } }
I20260709 12:20:00.131253 17184 leader_election.cc:290] T 103a303e4bce44cca4e34b3c391dbdac P 30f3ca7002424f919b063ee77bc473de [CANDIDATE]: Term 2 election: Requested vote from peers 71eca61062e040d48b2398afb3b0b510 (127.16.29.66:44237), 78285c60669649f6a5f13c1b65f9419e (127.16.29.67:44055)
I20260709 12:20:00.132216 17260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103a303e4bce44cca4e34b3c391dbdac" candidate_uuid: "30f3ca7002424f919b063ee77bc473de" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "71eca61062e040d48b2398afb3b0b510"
I20260709 12:20:00.132360 17334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103a303e4bce44cca4e34b3c391dbdac" candidate_uuid: "30f3ca7002424f919b063ee77bc473de" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "78285c60669649f6a5f13c1b65f9419e"
I20260709 12:20:00.132696 17260 raft_consensus.cc:3055] T 103a303e4bce44cca4e34b3c391dbdac P 71eca61062e040d48b2398afb3b0b510 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:20:00.132894 17334 raft_consensus.cc:3060] T 103a303e4bce44cca4e34b3c391dbdac P 78285c60669649f6a5f13c1b65f9419e [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:00.133029 17260 raft_consensus.cc:740] T 103a303e4bce44cca4e34b3c391dbdac P 71eca61062e040d48b2398afb3b0b510 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 71eca61062e040d48b2398afb3b0b510, State: Running, Role: LEADER
I20260709 12:20:00.133984 17260 consensus_queue.cc:260] T 103a303e4bce44cca4e34b3c391dbdac P 71eca61062e040d48b2398afb3b0b510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 } } peers { permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 } } peers { permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 } }
I20260709 12:20:00.134871 17260 raft_consensus.cc:3060] T 103a303e4bce44cca4e34b3c391dbdac P 71eca61062e040d48b2398afb3b0b510 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:00.137874 17334 raft_consensus.cc:2468] T 103a303e4bce44cca4e34b3c391dbdac P 78285c60669649f6a5f13c1b65f9419e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30f3ca7002424f919b063ee77bc473de in term 2.
I20260709 12:20:00.139138 17150 leader_election.cc:304] T 103a303e4bce44cca4e34b3c391dbdac P 30f3ca7002424f919b063ee77bc473de [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30f3ca7002424f919b063ee77bc473de, 78285c60669649f6a5f13c1b65f9419e; no voters: 
I20260709 12:20:00.139241 17260 raft_consensus.cc:2468] T 103a303e4bce44cca4e34b3c391dbdac P 71eca61062e040d48b2398afb3b0b510 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30f3ca7002424f919b063ee77bc473de in term 2.
I20260709 12:20:00.140035 17494 raft_consensus.cc:2804] T 103a303e4bce44cca4e34b3c391dbdac P 30f3ca7002424f919b063ee77bc473de [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:20:00.140496 17494 raft_consensus.cc:697] T 103a303e4bce44cca4e34b3c391dbdac P 30f3ca7002424f919b063ee77bc473de [term 2 LEADER]: Becoming Leader. State: Replica: 30f3ca7002424f919b063ee77bc473de, State: Running, Role: LEADER
I20260709 12:20:00.141260 17494 consensus_queue.cc:237] T 103a303e4bce44cca4e34b3c391dbdac P 30f3ca7002424f919b063ee77bc473de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 } } peers { permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 } } peers { permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 } }
I20260709 12:20:00.148952 17081 catalog_manager.cc:5697] T 103a303e4bce44cca4e34b3c391dbdac P 30f3ca7002424f919b063ee77bc473de reported cstate change: term changed from 1 to 2, leader changed from 71eca61062e040d48b2398afb3b0b510 (127.16.29.66) to 30f3ca7002424f919b063ee77bc473de (127.16.29.65). New cstate: current_term: 2 leader_uuid: "30f3ca7002424f919b063ee77bc473de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:00.304384 17493 raft_consensus.cc:993] T eecaa10d24c045d3921a84ecb5501242 P 78285c60669649f6a5f13c1b65f9419e: : Instructing follower 30f3ca7002424f919b063ee77bc473de to start an election
I20260709 12:20:00.304879 17487 raft_consensus.cc:1081] T eecaa10d24c045d3921a84ecb5501242 P 78285c60669649f6a5f13c1b65f9419e [term 3 LEADER]: Signalling peer 30f3ca7002424f919b063ee77bc473de to start an election
I20260709 12:20:00.306613 17184 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "eecaa10d24c045d3921a84ecb5501242"
dest_uuid: "30f3ca7002424f919b063ee77bc473de"
 from {username='slave'} at 127.0.0.1:53978
I20260709 12:20:00.307075 17184 raft_consensus.cc:493] T eecaa10d24c045d3921a84ecb5501242 P 30f3ca7002424f919b063ee77bc473de [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:20:00.307332 17184 raft_consensus.cc:3060] T eecaa10d24c045d3921a84ecb5501242 P 30f3ca7002424f919b063ee77bc473de [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:20:00.312727 17184 raft_consensus.cc:515] T eecaa10d24c045d3921a84ecb5501242 P 30f3ca7002424f919b063ee77bc473de [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 } } peers { permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 } } peers { permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 } }
I20260709 12:20:00.314211 17184 leader_election.cc:290] T eecaa10d24c045d3921a84ecb5501242 P 30f3ca7002424f919b063ee77bc473de [CANDIDATE]: Term 4 election: Requested vote from peers 71eca61062e040d48b2398afb3b0b510 (127.16.29.66:44237), 78285c60669649f6a5f13c1b65f9419e (127.16.29.67:44055)
I20260709 12:20:00.315140 17334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eecaa10d24c045d3921a84ecb5501242" candidate_uuid: "30f3ca7002424f919b063ee77bc473de" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "78285c60669649f6a5f13c1b65f9419e"
I20260709 12:20:00.315508 17260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eecaa10d24c045d3921a84ecb5501242" candidate_uuid: "30f3ca7002424f919b063ee77bc473de" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "71eca61062e040d48b2398afb3b0b510"
I20260709 12:20:00.315733 17334 raft_consensus.cc:3055] T eecaa10d24c045d3921a84ecb5501242 P 78285c60669649f6a5f13c1b65f9419e [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:20:00.316068 17260 raft_consensus.cc:3060] T eecaa10d24c045d3921a84ecb5501242 P 71eca61062e040d48b2398afb3b0b510 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:20:00.316102 17334 raft_consensus.cc:740] T eecaa10d24c045d3921a84ecb5501242 P 78285c60669649f6a5f13c1b65f9419e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 78285c60669649f6a5f13c1b65f9419e, State: Running, Role: LEADER
I20260709 12:20:00.317116 17334 consensus_queue.cc:260] T eecaa10d24c045d3921a84ecb5501242 P 78285c60669649f6a5f13c1b65f9419e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 } } peers { permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 } } peers { permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 } }
I20260709 12:20:00.318341 17334 raft_consensus.cc:3060] T eecaa10d24c045d3921a84ecb5501242 P 78285c60669649f6a5f13c1b65f9419e [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:20:00.323122 17260 raft_consensus.cc:2468] T eecaa10d24c045d3921a84ecb5501242 P 71eca61062e040d48b2398afb3b0b510 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30f3ca7002424f919b063ee77bc473de in term 4.
I20260709 12:20:00.324457 17151 leader_election.cc:304] T eecaa10d24c045d3921a84ecb5501242 P 30f3ca7002424f919b063ee77bc473de [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30f3ca7002424f919b063ee77bc473de, 71eca61062e040d48b2398afb3b0b510; no voters: 
I20260709 12:20:00.324687 17334 raft_consensus.cc:2468] T eecaa10d24c045d3921a84ecb5501242 P 78285c60669649f6a5f13c1b65f9419e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30f3ca7002424f919b063ee77bc473de in term 4.
I20260709 12:20:00.325269 17494 raft_consensus.cc:2804] T eecaa10d24c045d3921a84ecb5501242 P 30f3ca7002424f919b063ee77bc473de [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:20:00.325803 17494 raft_consensus.cc:697] T eecaa10d24c045d3921a84ecb5501242 P 30f3ca7002424f919b063ee77bc473de [term 4 LEADER]: Becoming Leader. State: Replica: 30f3ca7002424f919b063ee77bc473de, State: Running, Role: LEADER
I20260709 12:20:00.326671 17494 consensus_queue.cc:237] T eecaa10d24c045d3921a84ecb5501242 P 30f3ca7002424f919b063ee77bc473de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 } } peers { permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 } } peers { permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 } }
I20260709 12:20:00.334654 17081 catalog_manager.cc:5697] T eecaa10d24c045d3921a84ecb5501242 P 30f3ca7002424f919b063ee77bc473de reported cstate change: term changed from 3 to 4, leader changed from 78285c60669649f6a5f13c1b65f9419e (127.16.29.67) to 30f3ca7002424f919b063ee77bc473de (127.16.29.65). New cstate: current_term: 4 leader_uuid: "30f3ca7002424f919b063ee77bc473de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:00.456656 17334 raft_consensus.cc:1275] T 0e08c6dc97d34c998b47e5207487ef31 P 78285c60669649f6a5f13c1b65f9419e [term 3 FOLLOWER]: Refusing update from remote peer 30f3ca7002424f919b063ee77bc473de: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:20:00.458051 17494 consensus_queue.cc:1048] T 0e08c6dc97d34c998b47e5207487ef31 P 30f3ca7002424f919b063ee77bc473de [LEADER]: Connected to new peer: Peer: permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:20:00.470319 17260 raft_consensus.cc:1275] T 0e08c6dc97d34c998b47e5207487ef31 P 71eca61062e040d48b2398afb3b0b510 [term 3 FOLLOWER]: Refusing update from remote peer 30f3ca7002424f919b063ee77bc473de: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:20:00.472632 17505 consensus_queue.cc:1048] T 0e08c6dc97d34c998b47e5207487ef31 P 30f3ca7002424f919b063ee77bc473de [LEADER]: Connected to new peer: Peer: permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:20:00.477988 17500 raft_consensus.cc:993] T ec3f5327156e49e48ae8c88da0db0522 P 71eca61062e040d48b2398afb3b0b510: : Instructing follower 30f3ca7002424f919b063ee77bc473de to start an election
I20260709 12:20:00.478538 17500 raft_consensus.cc:1081] T ec3f5327156e49e48ae8c88da0db0522 P 71eca61062e040d48b2398afb3b0b510 [term 1 LEADER]: Signalling peer 30f3ca7002424f919b063ee77bc473de to start an election
I20260709 12:20:00.480511 17184 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "ec3f5327156e49e48ae8c88da0db0522"
dest_uuid: "30f3ca7002424f919b063ee77bc473de"
 from {username='slave'} at 127.0.0.1:53972
I20260709 12:20:00.481122 17184 raft_consensus.cc:493] T ec3f5327156e49e48ae8c88da0db0522 P 30f3ca7002424f919b063ee77bc473de [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:20:00.481472 17184 raft_consensus.cc:3060] T ec3f5327156e49e48ae8c88da0db0522 P 30f3ca7002424f919b063ee77bc473de [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:00.487111 17184 raft_consensus.cc:515] T ec3f5327156e49e48ae8c88da0db0522 P 30f3ca7002424f919b063ee77bc473de [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 } } peers { permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 } } peers { permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 } }
I20260709 12:20:00.488903 17184 leader_election.cc:290] T ec3f5327156e49e48ae8c88da0db0522 P 30f3ca7002424f919b063ee77bc473de [CANDIDATE]: Term 2 election: Requested vote from peers 71eca61062e040d48b2398afb3b0b510 (127.16.29.66:44237), 78285c60669649f6a5f13c1b65f9419e (127.16.29.67:44055)
I20260709 12:20:00.489825 17260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ec3f5327156e49e48ae8c88da0db0522" candidate_uuid: "30f3ca7002424f919b063ee77bc473de" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "71eca61062e040d48b2398afb3b0b510"
I20260709 12:20:00.489985 17334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ec3f5327156e49e48ae8c88da0db0522" candidate_uuid: "30f3ca7002424f919b063ee77bc473de" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "78285c60669649f6a5f13c1b65f9419e"
I20260709 12:20:00.490373 17260 raft_consensus.cc:3055] T ec3f5327156e49e48ae8c88da0db0522 P 71eca61062e040d48b2398afb3b0b510 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:20:00.490473 17334 raft_consensus.cc:3060] T ec3f5327156e49e48ae8c88da0db0522 P 78285c60669649f6a5f13c1b65f9419e [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:00.490710 17260 raft_consensus.cc:740] T ec3f5327156e49e48ae8c88da0db0522 P 71eca61062e040d48b2398afb3b0b510 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 71eca61062e040d48b2398afb3b0b510, State: Running, Role: LEADER
I20260709 12:20:00.491442 17260 consensus_queue.cc:260] T ec3f5327156e49e48ae8c88da0db0522 P 71eca61062e040d48b2398afb3b0b510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 } } peers { permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 } } peers { permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 } }
I20260709 12:20:00.492600 17260 raft_consensus.cc:3060] T ec3f5327156e49e48ae8c88da0db0522 P 71eca61062e040d48b2398afb3b0b510 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:00.496920 17334 raft_consensus.cc:2468] T ec3f5327156e49e48ae8c88da0db0522 P 78285c60669649f6a5f13c1b65f9419e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30f3ca7002424f919b063ee77bc473de in term 2.
I20260709 12:20:00.498061 17150 leader_election.cc:304] T ec3f5327156e49e48ae8c88da0db0522 P 30f3ca7002424f919b063ee77bc473de [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30f3ca7002424f919b063ee77bc473de, 78285c60669649f6a5f13c1b65f9419e; no voters: 
I20260709 12:20:00.498693 17505 raft_consensus.cc:2804] T ec3f5327156e49e48ae8c88da0db0522 P 30f3ca7002424f919b063ee77bc473de [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:20:00.498637 17260 raft_consensus.cc:2468] T ec3f5327156e49e48ae8c88da0db0522 P 71eca61062e040d48b2398afb3b0b510 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30f3ca7002424f919b063ee77bc473de in term 2.
I20260709 12:20:00.499254 17505 raft_consensus.cc:697] T ec3f5327156e49e48ae8c88da0db0522 P 30f3ca7002424f919b063ee77bc473de [term 2 LEADER]: Becoming Leader. State: Replica: 30f3ca7002424f919b063ee77bc473de, State: Running, Role: LEADER
I20260709 12:20:00.500171 17505 consensus_queue.cc:237] T ec3f5327156e49e48ae8c88da0db0522 P 30f3ca7002424f919b063ee77bc473de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 } } peers { permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 } } peers { permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 } }
I20260709 12:20:00.508097 17083 catalog_manager.cc:5697] T ec3f5327156e49e48ae8c88da0db0522 P 30f3ca7002424f919b063ee77bc473de reported cstate change: term changed from 1 to 2, leader changed from 71eca61062e040d48b2398afb3b0b510 (127.16.29.66) to 30f3ca7002424f919b063ee77bc473de (127.16.29.65). New cstate: current_term: 2 leader_uuid: "30f3ca7002424f919b063ee77bc473de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:00.509485 17410 raft_consensus.cc:993] T ce6646ae2dfe43c681b0b1a3ed8cecd4 P 71eca61062e040d48b2398afb3b0b510: : Instructing follower 30f3ca7002424f919b063ee77bc473de to start an election
I20260709 12:20:00.509985 17500 raft_consensus.cc:1081] T ce6646ae2dfe43c681b0b1a3ed8cecd4 P 71eca61062e040d48b2398afb3b0b510 [term 1 LEADER]: Signalling peer 30f3ca7002424f919b063ee77bc473de to start an election
I20260709 12:20:00.511458 17184 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "ce6646ae2dfe43c681b0b1a3ed8cecd4"
dest_uuid: "30f3ca7002424f919b063ee77bc473de"
 from {username='slave'} at 127.0.0.1:53972
I20260709 12:20:00.511921 17184 raft_consensus.cc:493] T ce6646ae2dfe43c681b0b1a3ed8cecd4 P 30f3ca7002424f919b063ee77bc473de [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:20:00.512219 17184 raft_consensus.cc:3060] T ce6646ae2dfe43c681b0b1a3ed8cecd4 P 30f3ca7002424f919b063ee77bc473de [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:00.517223 17184 raft_consensus.cc:515] T ce6646ae2dfe43c681b0b1a3ed8cecd4 P 30f3ca7002424f919b063ee77bc473de [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 } } peers { permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 } } peers { permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 } }
I20260709 12:20:00.518637 17184 leader_election.cc:290] T ce6646ae2dfe43c681b0b1a3ed8cecd4 P 30f3ca7002424f919b063ee77bc473de [CANDIDATE]: Term 2 election: Requested vote from peers 71eca61062e040d48b2398afb3b0b510 (127.16.29.66:44237), 78285c60669649f6a5f13c1b65f9419e (127.16.29.67:44055)
I20260709 12:20:00.519642 17260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce6646ae2dfe43c681b0b1a3ed8cecd4" candidate_uuid: "30f3ca7002424f919b063ee77bc473de" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "71eca61062e040d48b2398afb3b0b510"
I20260709 12:20:00.519727 17334 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce6646ae2dfe43c681b0b1a3ed8cecd4" candidate_uuid: "30f3ca7002424f919b063ee77bc473de" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "78285c60669649f6a5f13c1b65f9419e"
I20260709 12:20:00.520196 17260 raft_consensus.cc:3055] T ce6646ae2dfe43c681b0b1a3ed8cecd4 P 71eca61062e040d48b2398afb3b0b510 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:20:00.520336 17334 raft_consensus.cc:3060] T ce6646ae2dfe43c681b0b1a3ed8cecd4 P 78285c60669649f6a5f13c1b65f9419e [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:00.520483 17260 raft_consensus.cc:740] T ce6646ae2dfe43c681b0b1a3ed8cecd4 P 71eca61062e040d48b2398afb3b0b510 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 71eca61062e040d48b2398afb3b0b510, State: Running, Role: LEADER
I20260709 12:20:00.521157 17260 consensus_queue.cc:260] T ce6646ae2dfe43c681b0b1a3ed8cecd4 P 71eca61062e040d48b2398afb3b0b510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 } } peers { permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 } } peers { permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 } }
I20260709 12:20:00.522010 17260 raft_consensus.cc:3060] T ce6646ae2dfe43c681b0b1a3ed8cecd4 P 71eca61062e040d48b2398afb3b0b510 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:00.525879 17334 raft_consensus.cc:2468] T ce6646ae2dfe43c681b0b1a3ed8cecd4 P 78285c60669649f6a5f13c1b65f9419e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30f3ca7002424f919b063ee77bc473de in term 2.
I20260709 12:20:00.526863 17150 leader_election.cc:304] T ce6646ae2dfe43c681b0b1a3ed8cecd4 P 30f3ca7002424f919b063ee77bc473de [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30f3ca7002424f919b063ee77bc473de, 78285c60669649f6a5f13c1b65f9419e; no voters: 
I20260709 12:20:00.527000 17260 raft_consensus.cc:2468] T ce6646ae2dfe43c681b0b1a3ed8cecd4 P 71eca61062e040d48b2398afb3b0b510 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30f3ca7002424f919b063ee77bc473de in term 2.
I20260709 12:20:00.527607 17505 raft_consensus.cc:2804] T ce6646ae2dfe43c681b0b1a3ed8cecd4 P 30f3ca7002424f919b063ee77bc473de [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:20:00.528117 17505 raft_consensus.cc:697] T ce6646ae2dfe43c681b0b1a3ed8cecd4 P 30f3ca7002424f919b063ee77bc473de [term 2 LEADER]: Becoming Leader. State: Replica: 30f3ca7002424f919b063ee77bc473de, State: Running, Role: LEADER
I20260709 12:20:00.529139 17505 consensus_queue.cc:237] T ce6646ae2dfe43c681b0b1a3ed8cecd4 P 30f3ca7002424f919b063ee77bc473de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 } } peers { permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 } } peers { permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 } }
I20260709 12:20:00.535522 17083 catalog_manager.cc:5697] T ce6646ae2dfe43c681b0b1a3ed8cecd4 P 30f3ca7002424f919b063ee77bc473de reported cstate change: term changed from 1 to 2, leader changed from 71eca61062e040d48b2398afb3b0b510 (127.16.29.66) to 30f3ca7002424f919b063ee77bc473de (127.16.29.65). New cstate: current_term: 2 leader_uuid: "30f3ca7002424f919b063ee77bc473de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:00.606684 17260 raft_consensus.cc:1275] T 103a303e4bce44cca4e34b3c391dbdac P 71eca61062e040d48b2398afb3b0b510 [term 2 FOLLOWER]: Refusing update from remote peer 30f3ca7002424f919b063ee77bc473de: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:20:00.607826 17505 consensus_queue.cc:1048] T 103a303e4bce44cca4e34b3c391dbdac P 30f3ca7002424f919b063ee77bc473de [LEADER]: Connected to new peer: Peer: permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:20:00.616446 17334 raft_consensus.cc:1275] T 103a303e4bce44cca4e34b3c391dbdac P 78285c60669649f6a5f13c1b65f9419e [term 2 FOLLOWER]: Refusing update from remote peer 30f3ca7002424f919b063ee77bc473de: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:20:00.618062 17489 consensus_queue.cc:1048] T 103a303e4bce44cca4e34b3c391dbdac P 30f3ca7002424f919b063ee77bc473de [LEADER]: Connected to new peer: Peer: permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:20:00.852536 17334 raft_consensus.cc:1275] T eecaa10d24c045d3921a84ecb5501242 P 78285c60669649f6a5f13c1b65f9419e [term 4 FOLLOWER]: Refusing update from remote peer 30f3ca7002424f919b063ee77bc473de: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:20:00.853766 17489 consensus_queue.cc:1048] T eecaa10d24c045d3921a84ecb5501242 P 30f3ca7002424f919b063ee77bc473de [LEADER]: Connected to new peer: Peer: permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:20:00.863667 17260 raft_consensus.cc:1275] T eecaa10d24c045d3921a84ecb5501242 P 71eca61062e040d48b2398afb3b0b510 [term 4 FOLLOWER]: Refusing update from remote peer 30f3ca7002424f919b063ee77bc473de: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:20:00.865573 17494 consensus_queue.cc:1048] T eecaa10d24c045d3921a84ecb5501242 P 30f3ca7002424f919b063ee77bc473de [LEADER]: Connected to new peer: Peer: permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:20:00.974673 17333 raft_consensus.cc:1275] T ce6646ae2dfe43c681b0b1a3ed8cecd4 P 78285c60669649f6a5f13c1b65f9419e [term 2 FOLLOWER]: Refusing update from remote peer 30f3ca7002424f919b063ee77bc473de: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:20:00.976056 17494 consensus_queue.cc:1048] T ce6646ae2dfe43c681b0b1a3ed8cecd4 P 30f3ca7002424f919b063ee77bc473de [LEADER]: Connected to new peer: Peer: permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:20:00.976820 17260 raft_consensus.cc:1275] T ce6646ae2dfe43c681b0b1a3ed8cecd4 P 71eca61062e040d48b2398afb3b0b510 [term 2 FOLLOWER]: Refusing update from remote peer 30f3ca7002424f919b063ee77bc473de: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:20:00.978066 17489 consensus_queue.cc:1048] T ce6646ae2dfe43c681b0b1a3ed8cecd4 P 30f3ca7002424f919b063ee77bc473de [LEADER]: Connected to new peer: Peer: permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:20:00.978700 17334 raft_consensus.cc:1275] T ec3f5327156e49e48ae8c88da0db0522 P 78285c60669649f6a5f13c1b65f9419e [term 2 FOLLOWER]: Refusing update from remote peer 30f3ca7002424f919b063ee77bc473de: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:20:00.980765 17489 consensus_queue.cc:1048] T ec3f5327156e49e48ae8c88da0db0522 P 30f3ca7002424f919b063ee77bc473de [LEADER]: Connected to new peer: Peer: permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:20:00.982184 17260 raft_consensus.cc:1275] T ec3f5327156e49e48ae8c88da0db0522 P 71eca61062e040d48b2398afb3b0b510 [term 2 FOLLOWER]: Refusing update from remote peer 30f3ca7002424f919b063ee77bc473de: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:20:00.984268 17489 consensus_queue.cc:1048] T ec3f5327156e49e48ae8c88da0db0522 P 30f3ca7002424f919b063ee77bc473de [LEADER]: Connected to new peer: Peer: permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:20:03.117149 16501 auto_leader_rebalancer-test.cc:399] The leader distribution is 
I20260709 12:20:03.118290 16501 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:20:03.128835 17184 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "0e08c6dc97d34c998b47e5207487ef31"
dest_uuid: "30f3ca7002424f919b063ee77bc473de"
mode: GRACEFUL
new_leader_uuid: "71eca61062e040d48b2398afb3b0b510"
 from {username='slave'} at 127.0.0.1:53982
I20260709 12:20:03.129527 17184 raft_consensus.cc:606] T 0e08c6dc97d34c998b47e5207487ef31 P 30f3ca7002424f919b063ee77bc473de [term 3 LEADER]: Received request to transfer leadership to TS 71eca61062e040d48b2398afb3b0b510
I20260709 12:20:03.132151 17184 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "103a303e4bce44cca4e34b3c391dbdac"
dest_uuid: "30f3ca7002424f919b063ee77bc473de"
mode: GRACEFUL
new_leader_uuid: "78285c60669649f6a5f13c1b65f9419e"
 from {username='slave'} at 127.0.0.1:53982
I20260709 12:20:03.132591 17184 raft_consensus.cc:606] T 103a303e4bce44cca4e34b3c391dbdac P 30f3ca7002424f919b063ee77bc473de [term 2 LEADER]: Received request to transfer leadership to TS 78285c60669649f6a5f13c1b65f9419e
I20260709 12:20:03.134865 17184 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "ce6646ae2dfe43c681b0b1a3ed8cecd4"
dest_uuid: "30f3ca7002424f919b063ee77bc473de"
mode: GRACEFUL
new_leader_uuid: "71eca61062e040d48b2398afb3b0b510"
 from {username='slave'} at 127.0.0.1:53982
I20260709 12:20:03.135356 17184 raft_consensus.cc:606] T ce6646ae2dfe43c681b0b1a3ed8cecd4 P 30f3ca7002424f919b063ee77bc473de [term 2 LEADER]: Received request to transfer leadership to TS 71eca61062e040d48b2398afb3b0b510
I20260709 12:20:03.137740 17184 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "ec3f5327156e49e48ae8c88da0db0522"
dest_uuid: "30f3ca7002424f919b063ee77bc473de"
mode: GRACEFUL
new_leader_uuid: "78285c60669649f6a5f13c1b65f9419e"
 from {username='slave'} at 127.0.0.1:53982
I20260709 12:20:03.138172 17184 raft_consensus.cc:606] T ec3f5327156e49e48ae8c88da0db0522 P 30f3ca7002424f919b063ee77bc473de [term 2 LEADER]: Received request to transfer leadership to TS 78285c60669649f6a5f13c1b65f9419e
I20260709 12:20:03.139214 16501 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 4
I20260709 12:20:03.139633 16501 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:20:03.497025 17522 raft_consensus.cc:993] T ce6646ae2dfe43c681b0b1a3ed8cecd4 P 30f3ca7002424f919b063ee77bc473de: : Instructing follower 71eca61062e040d48b2398afb3b0b510 to start an election
I20260709 12:20:03.497627 17519 raft_consensus.cc:1081] T ce6646ae2dfe43c681b0b1a3ed8cecd4 P 30f3ca7002424f919b063ee77bc473de [term 2 LEADER]: Signalling peer 71eca61062e040d48b2398afb3b0b510 to start an election
I20260709 12:20:03.499267 17261 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "ce6646ae2dfe43c681b0b1a3ed8cecd4"
dest_uuid: "71eca61062e040d48b2398afb3b0b510"
 from {username='slave'} at 127.0.0.1:45568
I20260709 12:20:03.499763 17261 raft_consensus.cc:493] T ce6646ae2dfe43c681b0b1a3ed8cecd4 P 71eca61062e040d48b2398afb3b0b510 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:20:03.500160 17261 raft_consensus.cc:3060] T ce6646ae2dfe43c681b0b1a3ed8cecd4 P 71eca61062e040d48b2398afb3b0b510 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:20:03.506387 17261 raft_consensus.cc:515] T ce6646ae2dfe43c681b0b1a3ed8cecd4 P 71eca61062e040d48b2398afb3b0b510 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 } } peers { permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 } } peers { permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 } }
I20260709 12:20:03.508689 17261 leader_election.cc:290] T ce6646ae2dfe43c681b0b1a3ed8cecd4 P 71eca61062e040d48b2398afb3b0b510 [CANDIDATE]: Term 3 election: Requested vote from peers 30f3ca7002424f919b063ee77bc473de (127.16.29.65:35609), 78285c60669649f6a5f13c1b65f9419e (127.16.29.67:44055)
I20260709 12:20:03.509737 17184 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce6646ae2dfe43c681b0b1a3ed8cecd4" candidate_uuid: "71eca61062e040d48b2398afb3b0b510" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "30f3ca7002424f919b063ee77bc473de"
I20260709 12:20:03.510046 17333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce6646ae2dfe43c681b0b1a3ed8cecd4" candidate_uuid: "71eca61062e040d48b2398afb3b0b510" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "78285c60669649f6a5f13c1b65f9419e"
I20260709 12:20:03.510690 17184 raft_consensus.cc:3055] T ce6646ae2dfe43c681b0b1a3ed8cecd4 P 30f3ca7002424f919b063ee77bc473de [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:20:03.510743 17333 raft_consensus.cc:3060] T ce6646ae2dfe43c681b0b1a3ed8cecd4 P 78285c60669649f6a5f13c1b65f9419e [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:20:03.511138 17184 raft_consensus.cc:740] T ce6646ae2dfe43c681b0b1a3ed8cecd4 P 30f3ca7002424f919b063ee77bc473de [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 30f3ca7002424f919b063ee77bc473de, State: Running, Role: LEADER
I20260709 12:20:03.512188 17184 consensus_queue.cc:260] T ce6646ae2dfe43c681b0b1a3ed8cecd4 P 30f3ca7002424f919b063ee77bc473de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 } } peers { permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 } } peers { permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 } }
I20260709 12:20:03.513603 17184 raft_consensus.cc:3060] T ce6646ae2dfe43c681b0b1a3ed8cecd4 P 30f3ca7002424f919b063ee77bc473de [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:20:03.519111 17333 raft_consensus.cc:2468] T ce6646ae2dfe43c681b0b1a3ed8cecd4 P 78285c60669649f6a5f13c1b65f9419e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71eca61062e040d48b2398afb3b0b510 in term 3.
I20260709 12:20:03.520692 17225 leader_election.cc:304] T ce6646ae2dfe43c681b0b1a3ed8cecd4 P 71eca61062e040d48b2398afb3b0b510 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71eca61062e040d48b2398afb3b0b510, 78285c60669649f6a5f13c1b65f9419e; no voters: 
I20260709 12:20:03.521354 17184 raft_consensus.cc:2468] T ce6646ae2dfe43c681b0b1a3ed8cecd4 P 30f3ca7002424f919b063ee77bc473de [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71eca61062e040d48b2398afb3b0b510 in term 3.
I20260709 12:20:03.521672 17500 raft_consensus.cc:2804] T ce6646ae2dfe43c681b0b1a3ed8cecd4 P 71eca61062e040d48b2398afb3b0b510 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:20:03.522828 17500 raft_consensus.cc:697] T ce6646ae2dfe43c681b0b1a3ed8cecd4 P 71eca61062e040d48b2398afb3b0b510 [term 3 LEADER]: Becoming Leader. State: Replica: 71eca61062e040d48b2398afb3b0b510, State: Running, Role: LEADER
I20260709 12:20:03.523844 17500 consensus_queue.cc:237] T ce6646ae2dfe43c681b0b1a3ed8cecd4 P 71eca61062e040d48b2398afb3b0b510 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 } } peers { permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 } } peers { permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 } }
I20260709 12:20:03.530706 17083 catalog_manager.cc:5697] T ce6646ae2dfe43c681b0b1a3ed8cecd4 P 71eca61062e040d48b2398afb3b0b510 reported cstate change: term changed from 2 to 3, leader changed from 30f3ca7002424f919b063ee77bc473de (127.16.29.65) to 71eca61062e040d48b2398afb3b0b510 (127.16.29.66). New cstate: current_term: 3 leader_uuid: "71eca61062e040d48b2398afb3b0b510" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:03.546680 17522 raft_consensus.cc:993] T 0e08c6dc97d34c998b47e5207487ef31 P 30f3ca7002424f919b063ee77bc473de: : Instructing follower 71eca61062e040d48b2398afb3b0b510 to start an election
I20260709 12:20:03.547339 17519 raft_consensus.cc:1081] T 0e08c6dc97d34c998b47e5207487ef31 P 30f3ca7002424f919b063ee77bc473de [term 3 LEADER]: Signalling peer 71eca61062e040d48b2398afb3b0b510 to start an election
I20260709 12:20:03.549106 17261 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "0e08c6dc97d34c998b47e5207487ef31"
dest_uuid: "71eca61062e040d48b2398afb3b0b510"
 from {username='slave'} at 127.0.0.1:45568
I20260709 12:20:03.549683 17261 raft_consensus.cc:493] T 0e08c6dc97d34c998b47e5207487ef31 P 71eca61062e040d48b2398afb3b0b510 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:20:03.549974 17261 raft_consensus.cc:3060] T 0e08c6dc97d34c998b47e5207487ef31 P 71eca61062e040d48b2398afb3b0b510 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:20:03.554631 17261 raft_consensus.cc:515] T 0e08c6dc97d34c998b47e5207487ef31 P 71eca61062e040d48b2398afb3b0b510 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 } } peers { permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 } } peers { permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 } }
I20260709 12:20:03.556219 17261 leader_election.cc:290] T 0e08c6dc97d34c998b47e5207487ef31 P 71eca61062e040d48b2398afb3b0b510 [CANDIDATE]: Term 4 election: Requested vote from peers 30f3ca7002424f919b063ee77bc473de (127.16.29.65:35609), 78285c60669649f6a5f13c1b65f9419e (127.16.29.67:44055)
I20260709 12:20:03.557294 17333 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e08c6dc97d34c998b47e5207487ef31" candidate_uuid: "71eca61062e040d48b2398afb3b0b510" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "78285c60669649f6a5f13c1b65f9419e"
I20260709 12:20:03.557361 17184 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e08c6dc97d34c998b47e5207487ef31" candidate_uuid: "71eca61062e040d48b2398afb3b0b510" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "30f3ca7002424f919b063ee77bc473de"
I20260709 12:20:03.557844 17333 raft_consensus.cc:3060] T 0e08c6dc97d34c998b47e5207487ef31 P 78285c60669649f6a5f13c1b65f9419e [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:20:03.557912 17184 raft_consensus.cc:3055] T 0e08c6dc97d34c998b47e5207487ef31 P 30f3ca7002424f919b063ee77bc473de [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:20:03.558360 17184 raft_consensus.cc:740] T 0e08c6dc97d34c998b47e5207487ef31 P 30f3ca7002424f919b063ee77bc473de [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 30f3ca7002424f919b063ee77bc473de, State: Running, Role: LEADER
I20260709 12:20:03.559119 17184 consensus_queue.cc:260] T 0e08c6dc97d34c998b47e5207487ef31 P 30f3ca7002424f919b063ee77bc473de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 } } peers { permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 } } peers { permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 } }
I20260709 12:20:03.560122 17184 raft_consensus.cc:3060] T 0e08c6dc97d34c998b47e5207487ef31 P 30f3ca7002424f919b063ee77bc473de [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:20:03.562897 17333 raft_consensus.cc:2468] T 0e08c6dc97d34c998b47e5207487ef31 P 78285c60669649f6a5f13c1b65f9419e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71eca61062e040d48b2398afb3b0b510 in term 4.
I20260709 12:20:03.564059 17225 leader_election.cc:304] T 0e08c6dc97d34c998b47e5207487ef31 P 71eca61062e040d48b2398afb3b0b510 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71eca61062e040d48b2398afb3b0b510, 78285c60669649f6a5f13c1b65f9419e; no voters: 
I20260709 12:20:03.564682 17500 raft_consensus.cc:2804] T 0e08c6dc97d34c998b47e5207487ef31 P 71eca61062e040d48b2398afb3b0b510 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:20:03.564906 17184 raft_consensus.cc:2468] T 0e08c6dc97d34c998b47e5207487ef31 P 30f3ca7002424f919b063ee77bc473de [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71eca61062e040d48b2398afb3b0b510 in term 4.
I20260709 12:20:03.565060 17500 raft_consensus.cc:697] T 0e08c6dc97d34c998b47e5207487ef31 P 71eca61062e040d48b2398afb3b0b510 [term 4 LEADER]: Becoming Leader. State: Replica: 71eca61062e040d48b2398afb3b0b510, State: Running, Role: LEADER
I20260709 12:20:03.566039 17500 consensus_queue.cc:237] T 0e08c6dc97d34c998b47e5207487ef31 P 71eca61062e040d48b2398afb3b0b510 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 } } peers { permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 } } peers { permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 } }
I20260709 12:20:03.572576 17083 catalog_manager.cc:5697] T 0e08c6dc97d34c998b47e5207487ef31 P 71eca61062e040d48b2398afb3b0b510 reported cstate change: term changed from 3 to 4, leader changed from 30f3ca7002424f919b063ee77bc473de (127.16.29.65) to 71eca61062e040d48b2398afb3b0b510 (127.16.29.66). New cstate: current_term: 4 leader_uuid: "71eca61062e040d48b2398afb3b0b510" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:03.593945 17522 raft_consensus.cc:993] T ec3f5327156e49e48ae8c88da0db0522 P 30f3ca7002424f919b063ee77bc473de: : Instructing follower 78285c60669649f6a5f13c1b65f9419e to start an election
I20260709 12:20:03.594563 17519 raft_consensus.cc:1081] T ec3f5327156e49e48ae8c88da0db0522 P 30f3ca7002424f919b063ee77bc473de [term 2 LEADER]: Signalling peer 78285c60669649f6a5f13c1b65f9419e to start an election
I20260709 12:20:03.596243 17333 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "ec3f5327156e49e48ae8c88da0db0522"
dest_uuid: "78285c60669649f6a5f13c1b65f9419e"
 from {username='slave'} at 127.0.0.1:38522
I20260709 12:20:03.596716 17333 raft_consensus.cc:493] T ec3f5327156e49e48ae8c88da0db0522 P 78285c60669649f6a5f13c1b65f9419e [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:20:03.597040 17333 raft_consensus.cc:3060] T ec3f5327156e49e48ae8c88da0db0522 P 78285c60669649f6a5f13c1b65f9419e [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:20:03.601475 17333 raft_consensus.cc:515] T ec3f5327156e49e48ae8c88da0db0522 P 78285c60669649f6a5f13c1b65f9419e [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 } } peers { permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 } } peers { permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 } }
I20260709 12:20:03.603668 17333 leader_election.cc:290] T ec3f5327156e49e48ae8c88da0db0522 P 78285c60669649f6a5f13c1b65f9419e [CANDIDATE]: Term 3 election: Requested vote from peers 30f3ca7002424f919b063ee77bc473de (127.16.29.65:35609), 71eca61062e040d48b2398afb3b0b510 (127.16.29.66:44237)
I20260709 12:20:03.604679 17184 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ec3f5327156e49e48ae8c88da0db0522" candidate_uuid: "78285c60669649f6a5f13c1b65f9419e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "30f3ca7002424f919b063ee77bc473de"
I20260709 12:20:03.604967 17261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ec3f5327156e49e48ae8c88da0db0522" candidate_uuid: "78285c60669649f6a5f13c1b65f9419e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "71eca61062e040d48b2398afb3b0b510"
I20260709 12:20:03.605232 17184 raft_consensus.cc:3055] T ec3f5327156e49e48ae8c88da0db0522 P 30f3ca7002424f919b063ee77bc473de [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:20:03.605588 17261 raft_consensus.cc:3060] T ec3f5327156e49e48ae8c88da0db0522 P 71eca61062e040d48b2398afb3b0b510 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:20:03.605592 17184 raft_consensus.cc:740] T ec3f5327156e49e48ae8c88da0db0522 P 30f3ca7002424f919b063ee77bc473de [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 30f3ca7002424f919b063ee77bc473de, State: Running, Role: LEADER
I20260709 12:20:03.606491 17184 consensus_queue.cc:260] T ec3f5327156e49e48ae8c88da0db0522 P 30f3ca7002424f919b063ee77bc473de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 } } peers { permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 } } peers { permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 } }
I20260709 12:20:03.607546 17184 raft_consensus.cc:3060] T ec3f5327156e49e48ae8c88da0db0522 P 30f3ca7002424f919b063ee77bc473de [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:20:03.610620 17261 raft_consensus.cc:2468] T ec3f5327156e49e48ae8c88da0db0522 P 71eca61062e040d48b2398afb3b0b510 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78285c60669649f6a5f13c1b65f9419e in term 3.
I20260709 12:20:03.611804 17300 leader_election.cc:304] T ec3f5327156e49e48ae8c88da0db0522 P 78285c60669649f6a5f13c1b65f9419e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71eca61062e040d48b2398afb3b0b510, 78285c60669649f6a5f13c1b65f9419e; no voters: 
I20260709 12:20:03.612038 17184 raft_consensus.cc:2468] T ec3f5327156e49e48ae8c88da0db0522 P 30f3ca7002424f919b063ee77bc473de [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78285c60669649f6a5f13c1b65f9419e in term 3.
I20260709 12:20:03.612708 17487 raft_consensus.cc:2804] T ec3f5327156e49e48ae8c88da0db0522 P 78285c60669649f6a5f13c1b65f9419e [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:20:03.617237 17487 raft_consensus.cc:697] T ec3f5327156e49e48ae8c88da0db0522 P 78285c60669649f6a5f13c1b65f9419e [term 3 LEADER]: Becoming Leader. State: Replica: 78285c60669649f6a5f13c1b65f9419e, State: Running, Role: LEADER
I20260709 12:20:03.618281 17487 consensus_queue.cc:237] T ec3f5327156e49e48ae8c88da0db0522 P 78285c60669649f6a5f13c1b65f9419e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 } } peers { permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 } } peers { permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 } }
I20260709 12:20:03.625875 17083 catalog_manager.cc:5697] T ec3f5327156e49e48ae8c88da0db0522 P 78285c60669649f6a5f13c1b65f9419e reported cstate change: term changed from 2 to 3, leader changed from 30f3ca7002424f919b063ee77bc473de (127.16.29.65) to 78285c60669649f6a5f13c1b65f9419e (127.16.29.67). New cstate: current_term: 3 leader_uuid: "78285c60669649f6a5f13c1b65f9419e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 } health_report { overall_health: HEALTHY } } }
I20260709 12:20:03.633652 17522 raft_consensus.cc:993] T 103a303e4bce44cca4e34b3c391dbdac P 30f3ca7002424f919b063ee77bc473de: : Instructing follower 78285c60669649f6a5f13c1b65f9419e to start an election
I20260709 12:20:03.634163 17519 raft_consensus.cc:1081] T 103a303e4bce44cca4e34b3c391dbdac P 30f3ca7002424f919b063ee77bc473de [term 2 LEADER]: Signalling peer 78285c60669649f6a5f13c1b65f9419e to start an election
I20260709 12:20:03.635689 17333 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "103a303e4bce44cca4e34b3c391dbdac"
dest_uuid: "78285c60669649f6a5f13c1b65f9419e"
 from {username='slave'} at 127.0.0.1:38522
I20260709 12:20:03.636173 17333 raft_consensus.cc:493] T 103a303e4bce44cca4e34b3c391dbdac P 78285c60669649f6a5f13c1b65f9419e [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:20:03.636422 17333 raft_consensus.cc:3060] T 103a303e4bce44cca4e34b3c391dbdac P 78285c60669649f6a5f13c1b65f9419e [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:20:03.640479 17333 raft_consensus.cc:515] T 103a303e4bce44cca4e34b3c391dbdac P 78285c60669649f6a5f13c1b65f9419e [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 } } peers { permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 } } peers { permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 } }
I20260709 12:20:03.641851 17333 leader_election.cc:290] T 103a303e4bce44cca4e34b3c391dbdac P 78285c60669649f6a5f13c1b65f9419e [CANDIDATE]: Term 3 election: Requested vote from peers 30f3ca7002424f919b063ee77bc473de (127.16.29.65:35609), 71eca61062e040d48b2398afb3b0b510 (127.16.29.66:44237)
I20260709 12:20:03.642792 17261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103a303e4bce44cca4e34b3c391dbdac" candidate_uuid: "78285c60669649f6a5f13c1b65f9419e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "71eca61062e040d48b2398afb3b0b510"
I20260709 12:20:03.642772 17184 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103a303e4bce44cca4e34b3c391dbdac" candidate_uuid: "78285c60669649f6a5f13c1b65f9419e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "30f3ca7002424f919b063ee77bc473de"
I20260709 12:20:03.643447 17261 raft_consensus.cc:3060] T 103a303e4bce44cca4e34b3c391dbdac P 71eca61062e040d48b2398afb3b0b510 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:20:03.643481 17184 raft_consensus.cc:3055] T 103a303e4bce44cca4e34b3c391dbdac P 30f3ca7002424f919b063ee77bc473de [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:20:03.643946 17184 raft_consensus.cc:740] T 103a303e4bce44cca4e34b3c391dbdac P 30f3ca7002424f919b063ee77bc473de [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 30f3ca7002424f919b063ee77bc473de, State: Running, Role: LEADER
I20260709 12:20:03.644656 17184 consensus_queue.cc:260] T 103a303e4bce44cca4e34b3c391dbdac P 30f3ca7002424f919b063ee77bc473de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 } } peers { permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 } } peers { permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 } }
I20260709 12:20:03.645802 17184 raft_consensus.cc:3060] T 103a303e4bce44cca4e34b3c391dbdac P 30f3ca7002424f919b063ee77bc473de [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:20:03.649699 17261 raft_consensus.cc:2468] T 103a303e4bce44cca4e34b3c391dbdac P 71eca61062e040d48b2398afb3b0b510 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78285c60669649f6a5f13c1b65f9419e in term 3.
I20260709 12:20:03.651096 17300 leader_election.cc:304] T 103a303e4bce44cca4e34b3c391dbdac P 78285c60669649f6a5f13c1b65f9419e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71eca61062e040d48b2398afb3b0b510, 78285c60669649f6a5f13c1b65f9419e; no voters: 
I20260709 12:20:03.651827 17487 raft_consensus.cc:2804] T 103a303e4bce44cca4e34b3c391dbdac P 78285c60669649f6a5f13c1b65f9419e [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:20:03.651827 17184 raft_consensus.cc:2468] T 103a303e4bce44cca4e34b3c391dbdac P 30f3ca7002424f919b063ee77bc473de [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78285c60669649f6a5f13c1b65f9419e in term 3.
I20260709 12:20:03.652457 17487 raft_consensus.cc:697] T 103a303e4bce44cca4e34b3c391dbdac P 78285c60669649f6a5f13c1b65f9419e [term 3 LEADER]: Becoming Leader. State: Replica: 78285c60669649f6a5f13c1b65f9419e, State: Running, Role: LEADER
I20260709 12:20:03.653436 17487 consensus_queue.cc:237] T 103a303e4bce44cca4e34b3c391dbdac P 78285c60669649f6a5f13c1b65f9419e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 } } peers { permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 } } peers { permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 } }
I20260709 12:20:03.659610 17083 catalog_manager.cc:5697] T 103a303e4bce44cca4e34b3c391dbdac P 78285c60669649f6a5f13c1b65f9419e reported cstate change: term changed from 2 to 3, leader changed from 30f3ca7002424f919b063ee77bc473de (127.16.29.65) to 78285c60669649f6a5f13c1b65f9419e (127.16.29.67). New cstate: current_term: 3 leader_uuid: "78285c60669649f6a5f13c1b65f9419e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 } health_report { overall_health: HEALTHY } } }
I20260709 12:20:03.974925 17184 raft_consensus.cc:1275] T 0e08c6dc97d34c998b47e5207487ef31 P 30f3ca7002424f919b063ee77bc473de [term 4 FOLLOWER]: Refusing update from remote peer 71eca61062e040d48b2398afb3b0b510: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:20:03.976197 17500 consensus_queue.cc:1048] T 0e08c6dc97d34c998b47e5207487ef31 P 71eca61062e040d48b2398afb3b0b510 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:20:03.986564 17333 raft_consensus.cc:1275] T 0e08c6dc97d34c998b47e5207487ef31 P 78285c60669649f6a5f13c1b65f9419e [term 4 FOLLOWER]: Refusing update from remote peer 71eca61062e040d48b2398afb3b0b510: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:20:03.988945 17500 consensus_queue.cc:1048] T 0e08c6dc97d34c998b47e5207487ef31 P 71eca61062e040d48b2398afb3b0b510 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:20:04.001508 17333 raft_consensus.cc:1275] T ce6646ae2dfe43c681b0b1a3ed8cecd4 P 78285c60669649f6a5f13c1b65f9419e [term 3 FOLLOWER]: Refusing update from remote peer 71eca61062e040d48b2398afb3b0b510: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:20:04.002315 17261 raft_consensus.cc:1275] T ec3f5327156e49e48ae8c88da0db0522 P 71eca61062e040d48b2398afb3b0b510 [term 3 FOLLOWER]: Refusing update from remote peer 78285c60669649f6a5f13c1b65f9419e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:20:04.002839 17500 consensus_queue.cc:1048] T ce6646ae2dfe43c681b0b1a3ed8cecd4 P 71eca61062e040d48b2398afb3b0b510 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78285c60669649f6a5f13c1b65f9419e" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:20:04.003571 17487 consensus_queue.cc:1048] T ec3f5327156e49e48ae8c88da0db0522 P 78285c60669649f6a5f13c1b65f9419e [LEADER]: Connected to new peer: Peer: permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:20:04.013602 17184 raft_consensus.cc:1275] T ce6646ae2dfe43c681b0b1a3ed8cecd4 P 30f3ca7002424f919b063ee77bc473de [term 3 FOLLOWER]: Refusing update from remote peer 71eca61062e040d48b2398afb3b0b510: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:20:04.014946 17500 consensus_queue.cc:1048] T ce6646ae2dfe43c681b0b1a3ed8cecd4 P 71eca61062e040d48b2398afb3b0b510 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:20:04.022074 17185 raft_consensus.cc:1275] T ec3f5327156e49e48ae8c88da0db0522 P 30f3ca7002424f919b063ee77bc473de [term 3 FOLLOWER]: Refusing update from remote peer 78285c60669649f6a5f13c1b65f9419e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:20:04.023816 17530 consensus_queue.cc:1048] T ec3f5327156e49e48ae8c88da0db0522 P 78285c60669649f6a5f13c1b65f9419e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:20:04.046271 17261 raft_consensus.cc:1275] T 103a303e4bce44cca4e34b3c391dbdac P 71eca61062e040d48b2398afb3b0b510 [term 3 FOLLOWER]: Refusing update from remote peer 78285c60669649f6a5f13c1b65f9419e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:20:04.048738 17530 consensus_queue.cc:1048] T 103a303e4bce44cca4e34b3c391dbdac P 78285c60669649f6a5f13c1b65f9419e [LEADER]: Connected to new peer: Peer: permanent_uuid: "71eca61062e040d48b2398afb3b0b510" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 44237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:20:04.060777 17185 raft_consensus.cc:1275] T 103a303e4bce44cca4e34b3c391dbdac P 30f3ca7002424f919b063ee77bc473de [term 3 FOLLOWER]: Refusing update from remote peer 78285c60669649f6a5f13c1b65f9419e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:20:04.062394 17543 consensus_queue.cc:1048] T 103a303e4bce44cca4e34b3c391dbdac P 78285c60669649f6a5f13c1b65f9419e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30f3ca7002424f919b063ee77bc473de" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:20:04.222766 16501 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:20:04.758141 16501 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:20:04.759079 16501 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
W20260709 12:20:05.348728 17207 debug-util.cc:398] Leaking SignalData structure 0x7b0800147a20 after lost signal to thread 16504
W20260709 12:20:05.349576 17207 debug-util.cc:398] Leaking SignalData structure 0x7b0800149140 after lost signal to thread 17118
W20260709 12:20:05.350467 17207 debug-util.cc:398] Leaking SignalData structure 0x7b080019f760 after lost signal to thread 17210
W20260709 12:20:05.351215 17207 debug-util.cc:398] Leaking SignalData structure 0x7b08000ec040 after lost signal to thread 17285
W20260709 12:20:05.351963 17207 debug-util.cc:398] Leaking SignalData structure 0x7b0800117140 after lost signal to thread 17359
I20260709 12:20:05.760267 16501 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:20:05.766117 16501 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:20:05.766578 16501 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:20:06.767571 16501 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:20:06.773468 16501 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:20:06.773921 16501 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:20:07.775087 16501 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:20:07.785063 16501 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:20:07.785804 16501 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:20:08.786890 16501 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:20:08.792464 16501 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:20:08.792953 16501 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:20:09.793936 16501 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:20:09.800747 16501 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:20:09.801333 16501 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:20:10.802441 16501 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:20:10.808986 16501 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:20:10.809468 16501 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:20:11.810467 16501 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:20:11.815716 16501 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:20:11.816147 16501 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:20:12.817584 16501 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:20:12.824934 16501 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:20:12.825431 16501 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:20:13.862623 16501 auto_leader_rebalancer-test.cc:410] The leader distribution is 
I20260709 12:20:13.863543 16501 tablet_server.cc:179] TabletServer@127.16.29.65:0 shutting down...
W20260709 12:20:13.884222 17301 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 6 similar messages]
W20260709 12:20:13.898698 17301 consensus_peers.cc:597] T ec3f5327156e49e48ae8c88da0db0522 P 78285c60669649f6a5f13c1b65f9419e -> Peer 30f3ca7002424f919b063ee77bc473de (127.16.29.65:35609): Couldn't send request to peer 30f3ca7002424f919b063ee77bc473de. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:13.912909 16501 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:20:13.913786 16501 tablet_replica.cc:333] T 9c4d5228c0fc4198aaf94191977ad479 P 30f3ca7002424f919b063ee77bc473de: stopping tablet replica
I20260709 12:20:13.914739 16501 raft_consensus.cc:2243] T 9c4d5228c0fc4198aaf94191977ad479 P 30f3ca7002424f919b063ee77bc473de [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:20:13.916002 16501 raft_consensus.cc:2272] T 9c4d5228c0fc4198aaf94191977ad479 P 30f3ca7002424f919b063ee77bc473de [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:13.918141 16501 tablet_replica.cc:333] T 103a303e4bce44cca4e34b3c391dbdac P 30f3ca7002424f919b063ee77bc473de: stopping tablet replica
I20260709 12:20:13.918726 16501 raft_consensus.cc:2243] T 103a303e4bce44cca4e34b3c391dbdac P 30f3ca7002424f919b063ee77bc473de [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:13.919296 16501 raft_consensus.cc:2272] T 103a303e4bce44cca4e34b3c391dbdac P 30f3ca7002424f919b063ee77bc473de [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:13.921223 16501 tablet_replica.cc:333] T 9f65da5ec1234214840536f7d27a5499 P 30f3ca7002424f919b063ee77bc473de: stopping tablet replica
I20260709 12:20:13.921681 16501 raft_consensus.cc:2243] T 9f65da5ec1234214840536f7d27a5499 P 30f3ca7002424f919b063ee77bc473de [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:13.922056 16501 raft_consensus.cc:2272] T 9f65da5ec1234214840536f7d27a5499 P 30f3ca7002424f919b063ee77bc473de [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:13.924103 16501 tablet_replica.cc:333] T ec3f5327156e49e48ae8c88da0db0522 P 30f3ca7002424f919b063ee77bc473de: stopping tablet replica
I20260709 12:20:13.924568 16501 raft_consensus.cc:2243] T ec3f5327156e49e48ae8c88da0db0522 P 30f3ca7002424f919b063ee77bc473de [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:13.925037 16501 raft_consensus.cc:2272] T ec3f5327156e49e48ae8c88da0db0522 P 30f3ca7002424f919b063ee77bc473de [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:13.926718 16501 tablet_replica.cc:333] T ce6646ae2dfe43c681b0b1a3ed8cecd4 P 30f3ca7002424f919b063ee77bc473de: stopping tablet replica
I20260709 12:20:13.927162 16501 raft_consensus.cc:2243] T ce6646ae2dfe43c681b0b1a3ed8cecd4 P 30f3ca7002424f919b063ee77bc473de [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:13.927661 16501 raft_consensus.cc:2272] T ce6646ae2dfe43c681b0b1a3ed8cecd4 P 30f3ca7002424f919b063ee77bc473de [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:13.929724 16501 tablet_replica.cc:333] T 0e08c6dc97d34c998b47e5207487ef31 P 30f3ca7002424f919b063ee77bc473de: stopping tablet replica
I20260709 12:20:13.930253 16501 raft_consensus.cc:2243] T 0e08c6dc97d34c998b47e5207487ef31 P 30f3ca7002424f919b063ee77bc473de [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:13.930768 16501 raft_consensus.cc:2272] T 0e08c6dc97d34c998b47e5207487ef31 P 30f3ca7002424f919b063ee77bc473de [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:13.932752 16501 tablet_replica.cc:333] T 0762726dc14a430cb8f27b2393feec0e P 30f3ca7002424f919b063ee77bc473de: stopping tablet replica
I20260709 12:20:13.933184 16501 raft_consensus.cc:2243] T 0762726dc14a430cb8f27b2393feec0e P 30f3ca7002424f919b063ee77bc473de [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:20:13.934033 16501 raft_consensus.cc:2272] T 0762726dc14a430cb8f27b2393feec0e P 30f3ca7002424f919b063ee77bc473de [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:13.935784 16501 tablet_replica.cc:333] T c55fea8d70354a8fa064ca624fb7a880 P 30f3ca7002424f919b063ee77bc473de: stopping tablet replica
I20260709 12:20:13.936311 16501 raft_consensus.cc:2243] T c55fea8d70354a8fa064ca624fb7a880 P 30f3ca7002424f919b063ee77bc473de [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:20:13.937073 16501 raft_consensus.cc:2272] T c55fea8d70354a8fa064ca624fb7a880 P 30f3ca7002424f919b063ee77bc473de [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:13.939050 16501 tablet_replica.cc:333] T eecaa10d24c045d3921a84ecb5501242 P 30f3ca7002424f919b063ee77bc473de: stopping tablet replica
I20260709 12:20:13.939565 16501 raft_consensus.cc:2243] T eecaa10d24c045d3921a84ecb5501242 P 30f3ca7002424f919b063ee77bc473de [term 4 LEADER]: Raft consensus shutting down.
I20260709 12:20:13.940446 16501 raft_consensus.cc:2272] T eecaa10d24c045d3921a84ecb5501242 P 30f3ca7002424f919b063ee77bc473de [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:13.942242 16501 tablet_replica.cc:333] T 58783bb92efb47c2b521006941944a9d P 30f3ca7002424f919b063ee77bc473de: stopping tablet replica
I20260709 12:20:13.942725 16501 raft_consensus.cc:2243] T 58783bb92efb47c2b521006941944a9d P 30f3ca7002424f919b063ee77bc473de [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:13.943192 16501 raft_consensus.cc:2272] T 58783bb92efb47c2b521006941944a9d P 30f3ca7002424f919b063ee77bc473de [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 12:20:13.949306 17227 consensus_peers.cc:597] T ce6646ae2dfe43c681b0b1a3ed8cecd4 P 71eca61062e040d48b2398afb3b0b510 -> Peer 30f3ca7002424f919b063ee77bc473de (127.16.29.65:35609): Couldn't send request to peer 30f3ca7002424f919b063ee77bc473de. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:14.001688 16501 tablet_server.cc:196] TabletServer@127.16.29.65:0 shutdown complete.
I20260709 12:20:14.022480 16501 tablet_server.cc:179] TabletServer@127.16.29.66:0 shutting down...
I20260709 12:20:14.056463 16501 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:20:14.057438 16501 tablet_replica.cc:333] T 103a303e4bce44cca4e34b3c391dbdac P 71eca61062e040d48b2398afb3b0b510: stopping tablet replica
I20260709 12:20:14.058493 16501 raft_consensus.cc:2243] T 103a303e4bce44cca4e34b3c391dbdac P 71eca61062e040d48b2398afb3b0b510 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:14.059087 16501 raft_consensus.cc:2272] T 103a303e4bce44cca4e34b3c391dbdac P 71eca61062e040d48b2398afb3b0b510 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:14.061632 16501 tablet_replica.cc:333] T 9c4d5228c0fc4198aaf94191977ad479 P 71eca61062e040d48b2398afb3b0b510: stopping tablet replica
I20260709 12:20:14.062376 16501 raft_consensus.cc:2243] T 9c4d5228c0fc4198aaf94191977ad479 P 71eca61062e040d48b2398afb3b0b510 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:14.063095 16501 raft_consensus.cc:2272] T 9c4d5228c0fc4198aaf94191977ad479 P 71eca61062e040d48b2398afb3b0b510 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:14.065294 16501 tablet_replica.cc:333] T 9f65da5ec1234214840536f7d27a5499 P 71eca61062e040d48b2398afb3b0b510: stopping tablet replica
I20260709 12:20:14.066013 16501 raft_consensus.cc:2243] T 9f65da5ec1234214840536f7d27a5499 P 71eca61062e040d48b2398afb3b0b510 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260709 12:20:14.066493 17227 consensus_peers.cc:597] T 0e08c6dc97d34c998b47e5207487ef31 P 71eca61062e040d48b2398afb3b0b510 -> Peer 30f3ca7002424f919b063ee77bc473de (127.16.29.65:35609): Couldn't send request to peer 30f3ca7002424f919b063ee77bc473de. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:35609: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:14.066550 16501 raft_consensus.cc:2272] T 9f65da5ec1234214840536f7d27a5499 P 71eca61062e040d48b2398afb3b0b510 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:14.068572 16501 tablet_replica.cc:333] T eecaa10d24c045d3921a84ecb5501242 P 71eca61062e040d48b2398afb3b0b510: stopping tablet replica
I20260709 12:20:14.069124 16501 raft_consensus.cc:2243] T eecaa10d24c045d3921a84ecb5501242 P 71eca61062e040d48b2398afb3b0b510 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:14.069660 16501 raft_consensus.cc:2272] T eecaa10d24c045d3921a84ecb5501242 P 71eca61062e040d48b2398afb3b0b510 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:14.071586 16501 tablet_replica.cc:333] T 58783bb92efb47c2b521006941944a9d P 71eca61062e040d48b2398afb3b0b510: stopping tablet replica
I20260709 12:20:14.072166 16501 raft_consensus.cc:2243] T 58783bb92efb47c2b521006941944a9d P 71eca61062e040d48b2398afb3b0b510 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:20:14.073182 16501 raft_consensus.cc:2272] T 58783bb92efb47c2b521006941944a9d P 71eca61062e040d48b2398afb3b0b510 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:14.075585 16501 tablet_replica.cc:333] T ec3f5327156e49e48ae8c88da0db0522 P 71eca61062e040d48b2398afb3b0b510: stopping tablet replica
I20260709 12:20:14.076578 16501 raft_consensus.cc:2243] T ec3f5327156e49e48ae8c88da0db0522 P 71eca61062e040d48b2398afb3b0b510 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:14.077040 16501 raft_consensus.cc:2272] T ec3f5327156e49e48ae8c88da0db0522 P 71eca61062e040d48b2398afb3b0b510 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:14.079243 16501 tablet_replica.cc:333] T 0e08c6dc97d34c998b47e5207487ef31 P 71eca61062e040d48b2398afb3b0b510: stopping tablet replica
I20260709 12:20:14.079733 16501 raft_consensus.cc:2243] T 0e08c6dc97d34c998b47e5207487ef31 P 71eca61062e040d48b2398afb3b0b510 [term 4 LEADER]: Raft consensus shutting down.
I20260709 12:20:14.080631 16501 raft_consensus.cc:2272] T 0e08c6dc97d34c998b47e5207487ef31 P 71eca61062e040d48b2398afb3b0b510 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:14.082777 16501 tablet_replica.cc:333] T ce6646ae2dfe43c681b0b1a3ed8cecd4 P 71eca61062e040d48b2398afb3b0b510: stopping tablet replica
I20260709 12:20:14.083375 16501 raft_consensus.cc:2243] T ce6646ae2dfe43c681b0b1a3ed8cecd4 P 71eca61062e040d48b2398afb3b0b510 [term 3 LEADER]: Raft consensus shutting down.
I20260709 12:20:14.084301 16501 raft_consensus.cc:2272] T ce6646ae2dfe43c681b0b1a3ed8cecd4 P 71eca61062e040d48b2398afb3b0b510 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:14.086759 16501 tablet_replica.cc:333] T c55fea8d70354a8fa064ca624fb7a880 P 71eca61062e040d48b2398afb3b0b510: stopping tablet replica
I20260709 12:20:14.087332 16501 raft_consensus.cc:2243] T c55fea8d70354a8fa064ca624fb7a880 P 71eca61062e040d48b2398afb3b0b510 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:14.087788 16501 raft_consensus.cc:2272] T c55fea8d70354a8fa064ca624fb7a880 P 71eca61062e040d48b2398afb3b0b510 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:14.089679 16501 tablet_replica.cc:333] T 0762726dc14a430cb8f27b2393feec0e P 71eca61062e040d48b2398afb3b0b510: stopping tablet replica
I20260709 12:20:14.090169 16501 raft_consensus.cc:2243] T 0762726dc14a430cb8f27b2393feec0e P 71eca61062e040d48b2398afb3b0b510 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:14.090615 16501 raft_consensus.cc:2272] T 0762726dc14a430cb8f27b2393feec0e P 71eca61062e040d48b2398afb3b0b510 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:14.123730 16501 tablet_server.cc:196] TabletServer@127.16.29.66:0 shutdown complete.
I20260709 12:20:14.145346 16501 tablet_server.cc:179] TabletServer@127.16.29.67:0 shutting down...
I20260709 12:20:14.173282 16501 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:20:14.174127 16501 tablet_replica.cc:333] T 9f65da5ec1234214840536f7d27a5499 P 78285c60669649f6a5f13c1b65f9419e: stopping tablet replica
I20260709 12:20:14.174806 16501 raft_consensus.cc:2243] T 9f65da5ec1234214840536f7d27a5499 P 78285c60669649f6a5f13c1b65f9419e [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:20:14.175812 16501 raft_consensus.cc:2272] T 9f65da5ec1234214840536f7d27a5499 P 78285c60669649f6a5f13c1b65f9419e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:14.177788 16501 tablet_replica.cc:333] T ec3f5327156e49e48ae8c88da0db0522 P 78285c60669649f6a5f13c1b65f9419e: stopping tablet replica
I20260709 12:20:14.178314 16501 raft_consensus.cc:2243] T ec3f5327156e49e48ae8c88da0db0522 P 78285c60669649f6a5f13c1b65f9419e [term 3 LEADER]: Raft consensus shutting down.
I20260709 12:20:14.179149 16501 raft_consensus.cc:2272] T ec3f5327156e49e48ae8c88da0db0522 P 78285c60669649f6a5f13c1b65f9419e [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:14.180932 16501 tablet_replica.cc:333] T 9c4d5228c0fc4198aaf94191977ad479 P 78285c60669649f6a5f13c1b65f9419e: stopping tablet replica
I20260709 12:20:14.181452 16501 raft_consensus.cc:2243] T 9c4d5228c0fc4198aaf94191977ad479 P 78285c60669649f6a5f13c1b65f9419e [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:14.181856 16501 raft_consensus.cc:2272] T 9c4d5228c0fc4198aaf94191977ad479 P 78285c60669649f6a5f13c1b65f9419e [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:14.183588 16501 tablet_replica.cc:333] T c55fea8d70354a8fa064ca624fb7a880 P 78285c60669649f6a5f13c1b65f9419e: stopping tablet replica
I20260709 12:20:14.184160 16501 raft_consensus.cc:2243] T c55fea8d70354a8fa064ca624fb7a880 P 78285c60669649f6a5f13c1b65f9419e [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:14.184576 16501 raft_consensus.cc:2272] T c55fea8d70354a8fa064ca624fb7a880 P 78285c60669649f6a5f13c1b65f9419e [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:14.186199 16501 tablet_replica.cc:333] T eecaa10d24c045d3921a84ecb5501242 P 78285c60669649f6a5f13c1b65f9419e: stopping tablet replica
I20260709 12:20:14.186733 16501 raft_consensus.cc:2243] T eecaa10d24c045d3921a84ecb5501242 P 78285c60669649f6a5f13c1b65f9419e [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:14.187155 16501 raft_consensus.cc:2272] T eecaa10d24c045d3921a84ecb5501242 P 78285c60669649f6a5f13c1b65f9419e [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:14.188868 16501 tablet_replica.cc:333] T 58783bb92efb47c2b521006941944a9d P 78285c60669649f6a5f13c1b65f9419e: stopping tablet replica
I20260709 12:20:14.189342 16501 raft_consensus.cc:2243] T 58783bb92efb47c2b521006941944a9d P 78285c60669649f6a5f13c1b65f9419e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:14.189729 16501 raft_consensus.cc:2272] T 58783bb92efb47c2b521006941944a9d P 78285c60669649f6a5f13c1b65f9419e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:14.191308 16501 tablet_replica.cc:333] T 0e08c6dc97d34c998b47e5207487ef31 P 78285c60669649f6a5f13c1b65f9419e: stopping tablet replica
I20260709 12:20:14.191788 16501 raft_consensus.cc:2243] T 0e08c6dc97d34c998b47e5207487ef31 P 78285c60669649f6a5f13c1b65f9419e [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:14.192278 16501 raft_consensus.cc:2272] T 0e08c6dc97d34c998b47e5207487ef31 P 78285c60669649f6a5f13c1b65f9419e [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:14.193826 16501 tablet_replica.cc:333] T ce6646ae2dfe43c681b0b1a3ed8cecd4 P 78285c60669649f6a5f13c1b65f9419e: stopping tablet replica
I20260709 12:20:14.194306 16501 raft_consensus.cc:2243] T ce6646ae2dfe43c681b0b1a3ed8cecd4 P 78285c60669649f6a5f13c1b65f9419e [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:14.194738 16501 raft_consensus.cc:2272] T ce6646ae2dfe43c681b0b1a3ed8cecd4 P 78285c60669649f6a5f13c1b65f9419e [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:14.196887 16501 tablet_replica.cc:333] T 0762726dc14a430cb8f27b2393feec0e P 78285c60669649f6a5f13c1b65f9419e: stopping tablet replica
I20260709 12:20:14.197365 16501 raft_consensus.cc:2243] T 0762726dc14a430cb8f27b2393feec0e P 78285c60669649f6a5f13c1b65f9419e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:14.197746 16501 raft_consensus.cc:2272] T 0762726dc14a430cb8f27b2393feec0e P 78285c60669649f6a5f13c1b65f9419e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:14.199357 16501 tablet_replica.cc:333] T 103a303e4bce44cca4e34b3c391dbdac P 78285c60669649f6a5f13c1b65f9419e: stopping tablet replica
I20260709 12:20:14.199803 16501 raft_consensus.cc:2243] T 103a303e4bce44cca4e34b3c391dbdac P 78285c60669649f6a5f13c1b65f9419e [term 3 LEADER]: Raft consensus shutting down.
I20260709 12:20:14.200562 16501 raft_consensus.cc:2272] T 103a303e4bce44cca4e34b3c391dbdac P 78285c60669649f6a5f13c1b65f9419e [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:14.229650 16501 tablet_server.cc:196] TabletServer@127.16.29.67:0 shutdown complete.
I20260709 12:20:14.251487 16501 master.cc:562] Master@127.16.29.126:37041 shutting down...
I20260709 12:20:14.275619 16501 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 8d41903a1faf49aaa6fbd5b36d9afa79 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:20:14.276242 16501 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 8d41903a1faf49aaa6fbd5b36d9afa79 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:14.276525 16501 tablet_replica.cc:333] T 00000000000000000000000000000000 P 8d41903a1faf49aaa6fbd5b36d9afa79: stopping tablet replica
I20260709 12:20:14.296687 16501 master.cc:584] Master@127.16.29.126:37041 shutdown complete.
30f3ca7002424f919b063ee77bc473de  5
71eca61062e040d48b2398afb3b0b510  4
78285c60669649f6a5f13c1b65f9419e  1
30f3ca7002424f919b063ee77bc473de  3
71eca61062e040d48b2398afb3b0b510  4
78285c60669649f6a5f13c1b65f9419e  3
30f3ca7002424f919b063ee77bc473de  8
71eca61062e040d48b2398afb3b0b510  1
78285c60669649f6a5f13c1b65f9419e  1
30f3ca7002424f919b063ee77bc473de  4
71eca61062e040d48b2398afb3b0b510  3
78285c60669649f6a5f13c1b65f9419e  3
[       OK ] LeaderRebalancerTest.FunctionalTestForNotDivided (29385 ms)
[ RUN      ] LeaderRebalancerTest.AddTserver
I20260709 12:20:14.342226 16501 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.16.29.126:39149
I20260709 12:20:14.343355 16501 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:20:14.349431 17562 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:20:14.349800 17563 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:20:14.350293 17565 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:20:14.351197 16501 server_base.cc:1061] running on GCE node
I20260709 12:20:14.352160 16501 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:20:14.352372 16501 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:20:14.352489 16501 hybrid_clock.cc:648] HybridClock initialized: now 1783599614352479 us; error 0 us; skew 500 ppm
I20260709 12:20:14.352964 16501 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:20:14.355511 16501 webserver.cc:533] Webserver started at http://127.16.29.126:44203/ using document root <none> and password file <none>
I20260709 12:20:14.356336 16501 fs_manager.cc:362] Metadata directory not provided
I20260709 12:20:14.356499 16501 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:20:14.356729 16501 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:20:14.357810 16501 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599551912677-16501-0/minicluster-data/master-0-root/instance:
uuid: "2193af31cb894ec9b3d65d939a2a6851"
format_stamp: "Formatted at 2026-07-09 12:20:14 on dist-test-slave-5n9v"
I20260709 12:20:14.362262 16501 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.005s	sys 0.000s
I20260709 12:20:14.365654 17570 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:20:14.366591 16501 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.001s	sys 0.000s
I20260709 12:20:14.367087 16501 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599551912677-16501-0/minicluster-data/master-0-root
uuid: "2193af31cb894ec9b3d65d939a2a6851"
format_stamp: "Formatted at 2026-07-09 12:20:14 on dist-test-slave-5n9v"
I20260709 12:20:14.367538 16501 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599551912677-16501-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599551912677-16501-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599551912677-16501-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:20:14.386618 16501 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:20:14.387871 16501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:20:14.427305 16501 rpc_server.cc:307] RPC server started. Bound to: 127.16.29.126:39149
I20260709 12:20:14.427418 17621 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.29.126:39149 every 8 connection(s)
I20260709 12:20:14.431308 17622 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:14.441136 17622 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2193af31cb894ec9b3d65d939a2a6851: Bootstrap starting.
I20260709 12:20:14.445765 17622 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2193af31cb894ec9b3d65d939a2a6851: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:14.450160 17622 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2193af31cb894ec9b3d65d939a2a6851: No bootstrap required, opened a new log
I20260709 12:20:14.452374 17622 raft_consensus.cc:359] T 00000000000000000000000000000000 P 2193af31cb894ec9b3d65d939a2a6851 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2193af31cb894ec9b3d65d939a2a6851" member_type: VOTER }
I20260709 12:20:14.452783 17622 raft_consensus.cc:385] T 00000000000000000000000000000000 P 2193af31cb894ec9b3d65d939a2a6851 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:14.453045 17622 raft_consensus.cc:740] T 00000000000000000000000000000000 P 2193af31cb894ec9b3d65d939a2a6851 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2193af31cb894ec9b3d65d939a2a6851, State: Initialized, Role: FOLLOWER
I20260709 12:20:14.453657 17622 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2193af31cb894ec9b3d65d939a2a6851 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2193af31cb894ec9b3d65d939a2a6851" member_type: VOTER }
I20260709 12:20:14.454109 17622 raft_consensus.cc:399] T 00000000000000000000000000000000 P 2193af31cb894ec9b3d65d939a2a6851 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260709 12:20:14.454370 17622 raft_consensus.cc:493] T 00000000000000000000000000000000 P 2193af31cb894ec9b3d65d939a2a6851 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260709 12:20:14.454645 17622 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 2193af31cb894ec9b3d65d939a2a6851 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:14.458745 17622 raft_consensus.cc:515] T 00000000000000000000000000000000 P 2193af31cb894ec9b3d65d939a2a6851 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2193af31cb894ec9b3d65d939a2a6851" member_type: VOTER }
I20260709 12:20:14.459282 17622 leader_election.cc:304] T 00000000000000000000000000000000 P 2193af31cb894ec9b3d65d939a2a6851 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2193af31cb894ec9b3d65d939a2a6851; no voters: 
I20260709 12:20:14.460539 17622 leader_election.cc:290] T 00000000000000000000000000000000 P 2193af31cb894ec9b3d65d939a2a6851 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20260709 12:20:14.460928 17625 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 2193af31cb894ec9b3d65d939a2a6851 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:14.462280 17625 raft_consensus.cc:697] T 00000000000000000000000000000000 P 2193af31cb894ec9b3d65d939a2a6851 [term 1 LEADER]: Becoming Leader. State: Replica: 2193af31cb894ec9b3d65d939a2a6851, State: Running, Role: LEADER
I20260709 12:20:14.462913 17625 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2193af31cb894ec9b3d65d939a2a6851 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2193af31cb894ec9b3d65d939a2a6851" member_type: VOTER }
I20260709 12:20:14.463757 17622 sys_catalog.cc:565] T 00000000000000000000000000000000 P 2193af31cb894ec9b3d65d939a2a6851 [sys.catalog]: configured and running, proceeding with master startup.
I20260709 12:20:14.469619 17627 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2193af31cb894ec9b3d65d939a2a6851 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2193af31cb894ec9b3d65d939a2a6851. Latest consensus state: current_term: 1 leader_uuid: "2193af31cb894ec9b3d65d939a2a6851" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2193af31cb894ec9b3d65d939a2a6851" member_type: VOTER } }
I20260709 12:20:14.470384 17627 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2193af31cb894ec9b3d65d939a2a6851 [sys.catalog]: This master's current role is: LEADER
I20260709 12:20:14.471583 17626 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2193af31cb894ec9b3d65d939a2a6851 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2193af31cb894ec9b3d65d939a2a6851" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2193af31cb894ec9b3d65d939a2a6851" member_type: VOTER } }
I20260709 12:20:14.472818 17626 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2193af31cb894ec9b3d65d939a2a6851 [sys.catalog]: This master's current role is: LEADER
I20260709 12:20:14.473603 17633 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260709 12:20:14.479980 17633 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260709 12:20:14.481220 16501 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260709 12:20:14.489684 17633 catalog_manager.cc:1383] Generated new cluster ID: e00b0ddb18fc450d97e1b79cde2ecf60
I20260709 12:20:14.490155 17633 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260709 12:20:14.505425 17633 catalog_manager.cc:1406] Generated new certificate authority record
I20260709 12:20:14.506775 17633 catalog_manager.cc:1540] Loading token signing keys...
I20260709 12:20:14.521843 17633 catalog_manager.cc:6070] T 00000000000000000000000000000000 P 2193af31cb894ec9b3d65d939a2a6851: Generated new TSK 0
I20260709 12:20:14.522504 17633 catalog_manager.cc:1550] Initializing in-progress tserver states...
I20260709 12:20:14.548435 16501 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:20:14.555120 17643 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:20:14.557082 17644 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:20:14.557942 17646 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:20:14.558566 16501 server_base.cc:1061] running on GCE node
I20260709 12:20:14.559476 16501 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:20:14.559679 16501 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:20:14.559935 16501 hybrid_clock.cc:648] HybridClock initialized: now 1783599614559865 us; error 0 us; skew 500 ppm
I20260709 12:20:14.560554 16501 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:20:14.563052 16501 webserver.cc:533] Webserver started at http://127.16.29.65:43205/ using document root <none> and password file <none>
I20260709 12:20:14.563606 16501 fs_manager.cc:362] Metadata directory not provided
I20260709 12:20:14.563817 16501 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:20:14.564193 16501 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:20:14.565356 16501 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599551912677-16501-0/minicluster-data/ts-0-root/instance:
uuid: "0da169ff82674dc1947055ce43de68a3"
format_stamp: "Formatted at 2026-07-09 12:20:14 on dist-test-slave-5n9v"
I20260709 12:20:14.570463 16501 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.006s	sys 0.000s
I20260709 12:20:14.574087 17651 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:20:14.574963 16501 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.000s
I20260709 12:20:14.575378 16501 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599551912677-16501-0/minicluster-data/ts-0-root
uuid: "0da169ff82674dc1947055ce43de68a3"
format_stamp: "Formatted at 2026-07-09 12:20:14 on dist-test-slave-5n9v"
I20260709 12:20:14.575729 16501 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599551912677-16501-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599551912677-16501-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599551912677-16501-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:20:14.584985 16501 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:20:14.586265 16501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:20:14.587746 16501 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:20:14.590075 16501 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:20:14.590323 16501 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:20:14.590572 16501 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:20:14.590764 16501 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:20:14.631278 16501 rpc_server.cc:307] RPC server started. Bound to: 127.16.29.65:33071
I20260709 12:20:14.631392 17713 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.29.65:33071 every 8 connection(s)
I20260709 12:20:14.636272 16501 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:20:14.644755 17718 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:20:14.646306 17719 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:20:14.646982 17721 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:20:14.649052 16501 server_base.cc:1061] running on GCE node
I20260709 12:20:14.650193 17714 heartbeater.cc:344] Connected to a master server at 127.16.29.126:39149
I20260709 12:20:14.650614 17714 heartbeater.cc:461] Registering TS with master...
I20260709 12:20:14.650693 16501 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:20:14.650977 16501 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:20:14.651153 16501 hybrid_clock.cc:648] HybridClock initialized: now 1783599614651138 us; error 0 us; skew 500 ppm
I20260709 12:20:14.651482 17714 heartbeater.cc:507] Master 127.16.29.126:39149 requested a full tablet report, sending...
I20260709 12:20:14.651782 16501 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:20:14.654091 17587 ts_manager.cc:194] Registered new tserver with Master: 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071)
I20260709 12:20:14.654913 16501 webserver.cc:533] Webserver started at http://127.16.29.66:40685/ using document root <none> and password file <none>
I20260709 12:20:14.655551 16501 fs_manager.cc:362] Metadata directory not provided
I20260709 12:20:14.655798 16501 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:20:14.656167 16501 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:20:14.656741 17587 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:35958
I20260709 12:20:14.657891 16501 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599551912677-16501-0/minicluster-data/ts-1-root/instance:
uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
format_stamp: "Formatted at 2026-07-09 12:20:14 on dist-test-slave-5n9v"
I20260709 12:20:14.662693 16501 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.005s	sys 0.001s
I20260709 12:20:14.666648 17726 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:20:14.667531 16501 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.000s	sys 0.002s
I20260709 12:20:14.667778 16501 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599551912677-16501-0/minicluster-data/ts-1-root
uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
format_stamp: "Formatted at 2026-07-09 12:20:14 on dist-test-slave-5n9v"
I20260709 12:20:14.668100 16501 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599551912677-16501-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599551912677-16501-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599551912677-16501-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:20:14.679605 16501 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:20:14.681017 16501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:20:14.682709 16501 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:20:14.685503 16501 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:20:14.685694 16501 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:20:14.686278 16501 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:20:14.686466 16501 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:20:14.728925 16501 rpc_server.cc:307] RPC server started. Bound to: 127.16.29.66:39651
I20260709 12:20:14.729066 17788 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.29.66:39651 every 8 connection(s)
I20260709 12:20:14.733732 16501 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:20:14.741606 17792 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:20:14.743589 17793 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:20:14.745667 17795 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:20:14.746924 17789 heartbeater.cc:344] Connected to a master server at 127.16.29.126:39149
I20260709 12:20:14.747277 17789 heartbeater.cc:461] Registering TS with master...
I20260709 12:20:14.747978 16501 server_base.cc:1061] running on GCE node
I20260709 12:20:14.748322 17789 heartbeater.cc:507] Master 127.16.29.126:39149 requested a full tablet report, sending...
I20260709 12:20:14.749384 16501 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:20:14.749719 16501 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:20:14.749930 16501 hybrid_clock.cc:648] HybridClock initialized: now 1783599614749913 us; error 0 us; skew 500 ppm
I20260709 12:20:14.750602 16501 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:20:14.751142 17587 ts_manager.cc:194] Registered new tserver with Master: e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:20:14.753407 16501 webserver.cc:533] Webserver started at http://127.16.29.67:32773/ using document root <none> and password file <none>
I20260709 12:20:14.753417 17587 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:35962
I20260709 12:20:14.754060 16501 fs_manager.cc:362] Metadata directory not provided
I20260709 12:20:14.754293 16501 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:20:14.754573 16501 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:20:14.756122 16501 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599551912677-16501-0/minicluster-data/ts-2-root/instance:
uuid: "e063541542a94969898868bf6eb5ce1b"
format_stamp: "Formatted at 2026-07-09 12:20:14 on dist-test-slave-5n9v"
I20260709 12:20:14.760788 16501 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.002s	sys 0.001s
I20260709 12:20:14.765271 17800 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:20:14.766479 16501 fs_manager.cc:730] Time spent opening block manager: real 0.004s	user 0.004s	sys 0.000s
I20260709 12:20:14.766891 16501 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599551912677-16501-0/minicluster-data/ts-2-root
uuid: "e063541542a94969898868bf6eb5ce1b"
format_stamp: "Formatted at 2026-07-09 12:20:14 on dist-test-slave-5n9v"
I20260709 12:20:14.767216 16501 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599551912677-16501-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599551912677-16501-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599551912677-16501-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:20:14.783026 16501 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:20:14.784556 16501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:20:14.786078 16501 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:20:14.788734 16501 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:20:14.788995 16501 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:20:14.789242 16501 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:20:14.789417 16501 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:20:14.835755 16501 rpc_server.cc:307] RPC server started. Bound to: 127.16.29.67:36837
I20260709 12:20:14.835861 17862 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.29.67:36837 every 8 connection(s)
I20260709 12:20:14.852329 17863 heartbeater.cc:344] Connected to a master server at 127.16.29.126:39149
I20260709 12:20:14.852763 17863 heartbeater.cc:461] Registering TS with master...
I20260709 12:20:14.853576 17863 heartbeater.cc:507] Master 127.16.29.126:39149 requested a full tablet report, sending...
I20260709 12:20:14.855707 17587 ts_manager.cc:194] Registered new tserver with Master: e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:14.856804 16501 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.017410281s
I20260709 12:20:14.857524 17587 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:35966
I20260709 12:20:14.887344 17587 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35976:
name: "test-workload"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
  rows: "\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}""\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}"
  indirect_data: """"
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W20260709 12:20:14.893585 17587 catalog_manager.cc:7077] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260709 12:20:15.115769 17679 tablet_service.cc:1511] Processing CreateTablet for tablet 217734cc2eb24a7a8d92bfef30ee5c52 (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260709 12:20:15.117362 17679 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 217734cc2eb24a7a8d92bfef30ee5c52. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.118304 17752 tablet_service.cc:1511] Processing CreateTablet for tablet 5f52bc21096e4519acb00ed1ea68c343 (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260709 12:20:15.118817 17751 tablet_service.cc:1511] Processing CreateTablet for tablet 8b56346a04a14c5d9f73c5205b74cf2d (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260709 12:20:15.121057 17673 tablet_service.cc:1511] Processing CreateTablet for tablet c64345b0a2614ba4a382b83feb235cad (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260709 12:20:15.122210 17750 tablet_service.cc:1511] Processing CreateTablet for tablet 044ba5aee1d6499a975c6b29d932fd95 (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260709 12:20:15.123696 17675 tablet_service.cc:1511] Processing CreateTablet for tablet 044ba5aee1d6499a975c6b29d932fd95 (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260709 12:20:15.123998 17747 tablet_service.cc:1511] Processing CreateTablet for tablet e4dcc70202ad4cd6880749fbb3a6d0c6 (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20260709 12:20:15.125934 17753 tablet_service.cc:1511] Processing CreateTablet for tablet 7cb3768d215e409fb9f5baf18d124442 (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260709 12:20:15.126678 17748 tablet_service.cc:1511] Processing CreateTablet for tablet c64345b0a2614ba4a382b83feb235cad (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260709 12:20:15.127157 17673 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c64345b0a2614ba4a382b83feb235cad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.128144 17674 tablet_service.cc:1511] Processing CreateTablet for tablet a5732e71b9d54f8786b8c511d69943bf (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20260709 12:20:15.129463 17674 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a5732e71b9d54f8786b8c511d69943bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.129127 17749 tablet_service.cc:1511] Processing CreateTablet for tablet a5732e71b9d54f8786b8c511d69943bf (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20260709 12:20:15.131291 17675 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 044ba5aee1d6499a975c6b29d932fd95. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.131625 17678 tablet_service.cc:1511] Processing CreateTablet for tablet 7cb3768d215e409fb9f5baf18d124442 (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260709 12:20:15.133580 17678 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7cb3768d215e409fb9f5baf18d124442. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.136184 17677 tablet_service.cc:1511] Processing CreateTablet for tablet 5f52bc21096e4519acb00ed1ea68c343 (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260709 12:20:15.137704 17677 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f52bc21096e4519acb00ed1ea68c343. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.141497 17754 tablet_service.cc:1511] Processing CreateTablet for tablet 217734cc2eb24a7a8d92bfef30ee5c52 (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260709 12:20:15.144057 17676 tablet_service.cc:1511] Processing CreateTablet for tablet 8b56346a04a14c5d9f73c5205b74cf2d (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260709 12:20:15.145529 17676 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b56346a04a14c5d9f73c5205b74cf2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.121058 17672 tablet_service.cc:1511] Processing CreateTablet for tablet e4dcc70202ad4cd6880749fbb3a6d0c6 (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20260709 12:20:15.148377 17828 tablet_service.cc:1511] Processing CreateTablet for tablet 217734cc2eb24a7a8d92bfef30ee5c52 (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260709 12:20:15.119724 17752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f52bc21096e4519acb00ed1ea68c343. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.150195 17672 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e4dcc70202ad4cd6880749fbb3a6d0c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.150226 17747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e4dcc70202ad4cd6880749fbb3a6d0c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.150588 17828 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 217734cc2eb24a7a8d92bfef30ee5c52. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.151036 17748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c64345b0a2614ba4a382b83feb235cad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.153241 17754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 217734cc2eb24a7a8d92bfef30ee5c52. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.155488 17827 tablet_service.cc:1511] Processing CreateTablet for tablet 7cb3768d215e409fb9f5baf18d124442 (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260709 12:20:15.156474 17749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a5732e71b9d54f8786b8c511d69943bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.157356 17827 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7cb3768d215e409fb9f5baf18d124442. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.159314 17751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b56346a04a14c5d9f73c5205b74cf2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.160225 17753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7cb3768d215e409fb9f5baf18d124442. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.164171 17750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 044ba5aee1d6499a975c6b29d932fd95. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.166736 17826 tablet_service.cc:1511] Processing CreateTablet for tablet 5f52bc21096e4519acb00ed1ea68c343 (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260709 12:20:15.168572 17826 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f52bc21096e4519acb00ed1ea68c343. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.203176 17878 tablet_bootstrap.cc:492] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3: Bootstrap starting.
I20260709 12:20:15.204955 17825 tablet_service.cc:1511] Processing CreateTablet for tablet 8b56346a04a14c5d9f73c5205b74cf2d (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260709 12:20:15.206364 17825 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b56346a04a14c5d9f73c5205b74cf2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.207309 17671 tablet_service.cc:1511] Processing CreateTablet for tablet 3ed3f6bd4d904d42a3a48e346ed9227d (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260709 12:20:15.208767 17671 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3ed3f6bd4d904d42a3a48e346ed9227d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.209889 17878 tablet_bootstrap.cc:654] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:15.210784 17824 tablet_service.cc:1511] Processing CreateTablet for tablet 044ba5aee1d6499a975c6b29d932fd95 (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260709 12:20:15.211603 17822 tablet_service.cc:1511] Processing CreateTablet for tablet c64345b0a2614ba4a382b83feb235cad (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260709 12:20:15.212230 17824 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 044ba5aee1d6499a975c6b29d932fd95. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.212936 17822 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c64345b0a2614ba4a382b83feb235cad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.216334 17821 tablet_service.cc:1511] Processing CreateTablet for tablet e4dcc70202ad4cd6880749fbb3a6d0c6 (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20260709 12:20:15.217705 17821 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e4dcc70202ad4cd6880749fbb3a6d0c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.225737 17823 tablet_service.cc:1511] Processing CreateTablet for tablet a5732e71b9d54f8786b8c511d69943bf (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20260709 12:20:15.225869 17746 tablet_service.cc:1511] Processing CreateTablet for tablet 3ed3f6bd4d904d42a3a48e346ed9227d (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260709 12:20:15.221143 17820 tablet_service.cc:1511] Processing CreateTablet for tablet 3ed3f6bd4d904d42a3a48e346ed9227d (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260709 12:20:15.227272 17823 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a5732e71b9d54f8786b8c511d69943bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.227373 17746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3ed3f6bd4d904d42a3a48e346ed9227d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.228025 17820 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3ed3f6bd4d904d42a3a48e346ed9227d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.231600 17673 tablet_service.cc:1511] Processing CreateTablet for tablet d522f70beb1f417cb98940d90b594823 (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260709 12:20:15.242448 17673 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d522f70beb1f417cb98940d90b594823. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.243672 17745 tablet_service.cc:1511] Processing CreateTablet for tablet d522f70beb1f417cb98940d90b594823 (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260709 12:20:15.245198 17745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d522f70beb1f417cb98940d90b594823. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.254393 17819 tablet_service.cc:1511] Processing CreateTablet for tablet d522f70beb1f417cb98940d90b594823 (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260709 12:20:15.254487 17677 tablet_service.cc:1511] Processing CreateTablet for tablet a9aeb14eca4a43dcbeaccffdf7e28995 (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260709 12:20:15.256114 17819 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d522f70beb1f417cb98940d90b594823. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.256958 17677 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a9aeb14eca4a43dcbeaccffdf7e28995. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.274943 17878 tablet_bootstrap.cc:492] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3: No bootstrap required, opened a new log
I20260709 12:20:15.280056 17878 ts_tablet_manager.cc:1403] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3: Time spent bootstrapping tablet: real 0.077s	user 0.013s	sys 0.011s
I20260709 12:20:15.283020 17878 raft_consensus.cc:359] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:15.283743 17878 raft_consensus.cc:385] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:15.305879 17878 raft_consensus.cc:740] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Initialized, Role: FOLLOWER
I20260709 12:20:15.306635 17878 consensus_queue.cc:260] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:15.324018 17714 heartbeater.cc:499] Master 127.16.29.126:39149 was elected leader, sending a full tablet report...
I20260709 12:20:15.324115 17748 tablet_service.cc:1511] Processing CreateTablet for tablet a9aeb14eca4a43dcbeaccffdf7e28995 (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260709 12:20:15.325564 17748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a9aeb14eca4a43dcbeaccffdf7e28995. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.325829 17825 tablet_service.cc:1511] Processing CreateTablet for tablet a9aeb14eca4a43dcbeaccffdf7e28995 (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260709 12:20:15.327090 17825 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a9aeb14eca4a43dcbeaccffdf7e28995. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.333660 17880 tablet_bootstrap.cc:492] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb: Bootstrap starting.
I20260709 12:20:15.350423 17878 ts_tablet_manager.cc:1434] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3: Time spent starting tablet: real 0.070s	user 0.009s	sys 0.002s
I20260709 12:20:15.352236 17679 tablet_service.cc:1511] Processing CreateTablet for tablet 4092aabb101d411798b0354f88a5dd2b (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20260709 12:20:15.357004 17882 raft_consensus.cc:493] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:15.357617 17882 raft_consensus.cc:515] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:15.368183 17679 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4092aabb101d411798b0354f88a5dd2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.370599 17878 tablet_bootstrap.cc:492] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3: Bootstrap starting.
I20260709 12:20:15.382001 17881 tablet_bootstrap.cc:492] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b: Bootstrap starting.
W20260709 12:20:15.400941 17571 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 51631.9us
I20260709 12:20:15.414310 17878 tablet_bootstrap.cc:654] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:15.416038 17677 tablet_service.cc:1511] Processing CreateTablet for tablet 3de52d85935746e9a1e770fef3529988 (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20260709 12:20:15.426923 17677 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3de52d85935746e9a1e770fef3529988. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.428375 17882 leader_election.cc:290] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:15.433648 17826 tablet_service.cc:1511] Processing CreateTablet for tablet 4092aabb101d411798b0354f88a5dd2b (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20260709 12:20:15.436540 17881 tablet_bootstrap.cc:654] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:15.423830 17880 tablet_bootstrap.cc:654] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:15.439025 17820 tablet_service.cc:1511] Processing CreateTablet for tablet 3de52d85935746e9a1e770fef3529988 (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20260709 12:20:15.440382 17826 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4092aabb101d411798b0354f88a5dd2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.437165 17747 tablet_service.cc:1511] Processing CreateTablet for tablet 3de52d85935746e9a1e770fef3529988 (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20260709 12:20:15.442392 17747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3de52d85935746e9a1e770fef3529988. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.442848 17820 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3de52d85935746e9a1e770fef3529988. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.424615 17753 tablet_service.cc:1511] Processing CreateTablet for tablet 4092aabb101d411798b0354f88a5dd2b (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20260709 12:20:15.446772 17753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4092aabb101d411798b0354f88a5dd2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.458282 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:20:15.459975 17652 leader_election.cc:343] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:15.465711 17881 tablet_bootstrap.cc:492] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b: No bootstrap required, opened a new log
I20260709 12:20:15.466357 17881 ts_tablet_manager.cc:1403] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b: Time spent bootstrapping tablet: real 0.085s	user 0.008s	sys 0.012s
I20260709 12:20:15.468326 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:20:15.469902 17652 leader_election.cc:343] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:15.470402 17652 leader_election.cc:304] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:20:15.471539 17882 raft_consensus.cc:2749] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:15.472059 17878 tablet_bootstrap.cc:492] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3: No bootstrap required, opened a new log
I20260709 12:20:15.472604 17878 ts_tablet_manager.cc:1403] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3: Time spent bootstrapping tablet: real 0.102s	user 0.009s	sys 0.011s
I20260709 12:20:15.472893 17825 tablet_service.cc:1511] Processing CreateTablet for tablet 28062a50142448aa97994888b4765920 (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20260709 12:20:15.474294 17825 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 28062a50142448aa97994888b4765920. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.476630 17881 raft_consensus.cc:359] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:15.476724 17674 tablet_service.cc:1511] Processing CreateTablet for tablet 84421f094faa40f39b30b4c3d0737639 (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
W20260709 12:20:15.477187 17635 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:20:15.477615 17881 raft_consensus.cc:385] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:15.477991 17881 raft_consensus.cc:740] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Initialized, Role: FOLLOWER
I20260709 12:20:15.478223 17674 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 84421f094faa40f39b30b4c3d0737639. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.482628 17881 consensus_queue.cc:260] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:15.483034 17878 raft_consensus.cc:359] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:15.483733 17878 raft_consensus.cc:385] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:15.484076 17878 raft_consensus.cc:740] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Initialized, Role: FOLLOWER
I20260709 12:20:15.477736 17748 tablet_service.cc:1511] Processing CreateTablet for tablet 84421f094faa40f39b30b4c3d0737639 (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20260709 12:20:15.488687 17748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 84421f094faa40f39b30b4c3d0737639. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.472132 17676 tablet_service.cc:1511] Processing CreateTablet for tablet 28062a50142448aa97994888b4765920 (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20260709 12:20:15.492633 17676 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 28062a50142448aa97994888b4765920. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.481549 17827 tablet_service.cc:1511] Processing CreateTablet for tablet 84421f094faa40f39b30b4c3d0737639 (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20260709 12:20:15.493777 17675 tablet_service.cc:1511] Processing CreateTablet for tablet 45397041e3ac43c8901c5cb03f27c7b9 (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20260709 12:20:15.495414 17827 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 84421f094faa40f39b30b4c3d0737639. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.495409 17675 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 45397041e3ac43c8901c5cb03f27c7b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.480682 17880 tablet_bootstrap.cc:492] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb: No bootstrap required, opened a new log
I20260709 12:20:15.498251 17679 tablet_service.cc:1511] Processing CreateTablet for tablet 0f940f29438f475294357324452abe76 (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20260709 12:20:15.499511 17679 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0f940f29438f475294357324452abe76. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.500944 17819 tablet_service.cc:1511] Processing CreateTablet for tablet 45397041e3ac43c8901c5cb03f27c7b9 (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20260709 12:20:15.502192 17819 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 45397041e3ac43c8901c5cb03f27c7b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.498656 17880 ts_tablet_manager.cc:1403] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb: Time spent bootstrapping tablet: real 0.165s	user 0.014s	sys 0.004s
I20260709 12:20:15.471199 17750 tablet_service.cc:1511] Processing CreateTablet for tablet 28062a50142448aa97994888b4765920 (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20260709 12:20:15.510352 17863 heartbeater.cc:499] Master 127.16.29.126:39149 was elected leader, sending a full tablet report...
I20260709 12:20:15.514133 17677 tablet_service.cc:1511] Processing CreateTablet for tablet 44ad5363c5954da4a1bff5857780fd5c (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20260709 12:20:15.515453 17677 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 44ad5363c5954da4a1bff5857780fd5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.495517 17878 consensus_queue.cc:260] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:15.517328 17878 ts_tablet_manager.cc:1434] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3: Time spent starting tablet: real 0.044s	user 0.007s	sys 0.000s
I20260709 12:20:15.523038 17750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 28062a50142448aa97994888b4765920. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.523324 17752 tablet_service.cc:1511] Processing CreateTablet for tablet 44ad5363c5954da4a1bff5857780fd5c (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20260709 12:20:15.510375 17880 raft_consensus.cc:359] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:15.524600 17752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 44ad5363c5954da4a1bff5857780fd5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.524652 17880 raft_consensus.cc:385] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:15.525019 17880 raft_consensus.cc:740] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Initialized, Role: FOLLOWER
I20260709 12:20:15.525882 17880 consensus_queue.cc:260] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:15.531284 17878 tablet_bootstrap.cc:492] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3: Bootstrap starting.
I20260709 12:20:15.533602 17747 tablet_service.cc:1511] Processing CreateTablet for tablet 45397041e3ac43c8901c5cb03f27c7b9 (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20260709 12:20:15.534919 17747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 45397041e3ac43c8901c5cb03f27c7b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.536005 17826 tablet_service.cc:1511] Processing CreateTablet for tablet 0f940f29438f475294357324452abe76 (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20260709 12:20:15.540659 17821 tablet_service.cc:1511] Processing CreateTablet for tablet 44ad5363c5954da4a1bff5857780fd5c (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20260709 12:20:15.541931 17881 ts_tablet_manager.cc:1434] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b: Time spent starting tablet: real 0.075s	user 0.007s	sys 0.005s
I20260709 12:20:15.539193 17826 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0f940f29438f475294357324452abe76. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.542737 17821 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 44ad5363c5954da4a1bff5857780fd5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.542855 17753 tablet_service.cc:1511] Processing CreateTablet for tablet 0f940f29438f475294357324452abe76 (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20260709 12:20:15.556751 17753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0f940f29438f475294357324452abe76. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.559388 17878 tablet_bootstrap.cc:654] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:15.543025 17881 tablet_bootstrap.cc:492] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b: Bootstrap starting.
I20260709 12:20:15.568799 17671 tablet_service.cc:1511] Processing CreateTablet for tablet 11896555f5d7458dbb02a3203b27f1cf (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20260709 12:20:15.570384 17671 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11896555f5d7458dbb02a3203b27f1cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.576134 17746 tablet_service.cc:1511] Processing CreateTablet for tablet 11896555f5d7458dbb02a3203b27f1cf (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20260709 12:20:15.577437 17746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11896555f5d7458dbb02a3203b27f1cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.575781 17820 tablet_service.cc:1511] Processing CreateTablet for tablet 11896555f5d7458dbb02a3203b27f1cf (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20260709 12:20:15.584770 17673 tablet_service.cc:1511] Processing CreateTablet for tablet 8e417cb980e44a77962d2e3dfd58aad0 (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20260709 12:20:15.586007 17820 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11896555f5d7458dbb02a3203b27f1cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.586346 17673 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e417cb980e44a77962d2e3dfd58aad0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.589157 17678 tablet_service.cc:1511] Processing CreateTablet for tablet e5dabc1347544bd8b4e81fa5903d4a8b (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20260709 12:20:15.589555 17881 tablet_bootstrap.cc:654] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:15.590389 17678 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e5dabc1347544bd8b4e81fa5903d4a8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.591531 17749 tablet_service.cc:1511] Processing CreateTablet for tablet 8e417cb980e44a77962d2e3dfd58aad0 (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20260709 12:20:15.592779 17749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e417cb980e44a77962d2e3dfd58aad0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.625249 17878 tablet_bootstrap.cc:492] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3: No bootstrap required, opened a new log
I20260709 12:20:15.624986 17827 tablet_service.cc:1511] Processing CreateTablet for tablet e5dabc1347544bd8b4e81fa5903d4a8b (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20260709 12:20:15.625821 17878 ts_tablet_manager.cc:1403] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3: Time spent bootstrapping tablet: real 0.095s	user 0.014s	sys 0.007s
I20260709 12:20:15.626595 17827 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e5dabc1347544bd8b4e81fa5903d4a8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.635104 17880 ts_tablet_manager.cc:1434] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb: Time spent starting tablet: real 0.131s	user 0.000s	sys 0.010s
I20260709 12:20:15.635491 17789 heartbeater.cc:499] Master 127.16.29.126:39149 was elected leader, sending a full tablet report...
I20260709 12:20:15.636200 17880 tablet_bootstrap.cc:492] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb: Bootstrap starting.
I20260709 12:20:15.641201 17824 tablet_service.cc:1511] Processing CreateTablet for tablet 19dad6ab92894d708ec8eb7e8ac28185 (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260709 12:20:15.642517 17824 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 19dad6ab92894d708ec8eb7e8ac28185. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.602795 17751 tablet_service.cc:1511] Processing CreateTablet for tablet 19dad6ab92894d708ec8eb7e8ac28185 (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260709 12:20:15.643586 17751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 19dad6ab92894d708ec8eb7e8ac28185. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.644325 17748 tablet_service.cc:1511] Processing CreateTablet for tablet e5dabc1347544bd8b4e81fa5903d4a8b (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20260709 12:20:15.645673 17748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e5dabc1347544bd8b4e81fa5903d4a8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.649268 17679 tablet_service.cc:1511] Processing CreateTablet for tablet 3219dcbfd86340ecbde507517060ff81 (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20260709 12:20:15.650523 17679 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3219dcbfd86340ecbde507517060ff81. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.652788 17675 tablet_service.cc:1511] Processing CreateTablet for tablet 19dad6ab92894d708ec8eb7e8ac28185 (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260709 12:20:15.654110 17675 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 19dad6ab92894d708ec8eb7e8ac28185. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.657815 17823 tablet_service.cc:1511] Processing CreateTablet for tablet 3219dcbfd86340ecbde507517060ff81 (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20260709 12:20:15.659106 17823 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3219dcbfd86340ecbde507517060ff81. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.661656 17880 tablet_bootstrap.cc:654] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:15.668951 17752 tablet_service.cc:1511] Processing CreateTablet for tablet 3219dcbfd86340ecbde507517060ff81 (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20260709 12:20:15.670565 17752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3219dcbfd86340ecbde507517060ff81. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.677698 17881 tablet_bootstrap.cc:492] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b: No bootstrap required, opened a new log
I20260709 12:20:15.678215 17881 ts_tablet_manager.cc:1403] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b: Time spent bootstrapping tablet: real 0.135s	user 0.019s	sys 0.011s
I20260709 12:20:15.679718 17878 raft_consensus.cc:359] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:15.680469 17878 raft_consensus.cc:385] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:15.680768 17878 raft_consensus.cc:740] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Initialized, Role: FOLLOWER
I20260709 12:20:15.681490 17878 consensus_queue.cc:260] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:15.684347 17826 tablet_service.cc:1511] Processing CreateTablet for tablet 8e417cb980e44a77962d2e3dfd58aad0 (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20260709 12:20:15.689687 17878 ts_tablet_manager.cc:1434] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3: Time spent starting tablet: real 0.064s	user 0.008s	sys 0.000s
I20260709 12:20:15.691169 17754 tablet_service.cc:1511] Processing CreateTablet for tablet b5462347eb74482b8cd11beb1d39b8a9 (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20260709 12:20:15.705648 17878 tablet_bootstrap.cc:492] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3: Bootstrap starting.
I20260709 12:20:15.692651 17825 tablet_service.cc:1511] Processing CreateTablet for tablet b5462347eb74482b8cd11beb1d39b8a9 (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20260709 12:20:15.705438 17671 tablet_service.cc:1511] Processing CreateTablet for tablet b5462347eb74482b8cd11beb1d39b8a9 (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20260709 12:20:15.713238 17671 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b5462347eb74482b8cd11beb1d39b8a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.693861 17826 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e417cb980e44a77962d2e3dfd58aad0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.716064 17754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b5462347eb74482b8cd11beb1d39b8a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.716542 17825 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b5462347eb74482b8cd11beb1d39b8a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.695637 17881 raft_consensus.cc:359] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:15.718251 17881 raft_consensus.cc:385] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:15.718530 17881 raft_consensus.cc:740] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Initialized, Role: FOLLOWER
I20260709 12:20:15.719267 17881 consensus_queue.cc:260] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:15.721365 17881 ts_tablet_manager.cc:1434] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b: Time spent starting tablet: real 0.043s	user 0.006s	sys 0.000s
I20260709 12:20:15.724534 17878 tablet_bootstrap.cc:654] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:15.735172 17878 tablet_bootstrap.cc:492] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3: No bootstrap required, opened a new log
I20260709 12:20:15.735713 17878 ts_tablet_manager.cc:1403] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3: Time spent bootstrapping tablet: real 0.030s	user 0.010s	sys 0.003s
I20260709 12:20:15.737859 17880 tablet_bootstrap.cc:492] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb: No bootstrap required, opened a new log
I20260709 12:20:15.738366 17880 ts_tablet_manager.cc:1403] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent bootstrapping tablet: real 0.102s	user 0.007s	sys 0.006s
I20260709 12:20:15.738624 17878 raft_consensus.cc:359] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:15.739372 17878 raft_consensus.cc:385] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:15.739670 17878 raft_consensus.cc:740] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Initialized, Role: FOLLOWER
I20260709 12:20:15.740425 17878 consensus_queue.cc:260] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:15.741470 17880 raft_consensus.cc:359] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:15.742252 17880 raft_consensus.cc:385] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:15.742624 17880 raft_consensus.cc:740] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Initialized, Role: FOLLOWER
I20260709 12:20:15.742893 17881 tablet_bootstrap.cc:492] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b: Bootstrap starting.
I20260709 12:20:15.743592 17880 consensus_queue.cc:260] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:15.744912 17878 ts_tablet_manager.cc:1434] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3: Time spent starting tablet: real 0.009s	user 0.005s	sys 0.000s
I20260709 12:20:15.745904 17878 tablet_bootstrap.cc:492] T 217734cc2eb24a7a8d92bfef30ee5c52 P 0da169ff82674dc1947055ce43de68a3: Bootstrap starting.
I20260709 12:20:15.748245 17880 ts_tablet_manager.cc:1434] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent starting tablet: real 0.010s	user 0.007s	sys 0.000s
I20260709 12:20:15.750608 17880 tablet_bootstrap.cc:492] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb: Bootstrap starting.
I20260709 12:20:15.750648 17881 tablet_bootstrap.cc:654] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:15.751799 17882 raft_consensus.cc:493] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:15.752290 17878 tablet_bootstrap.cc:654] T 217734cc2eb24a7a8d92bfef30ee5c52 P 0da169ff82674dc1947055ce43de68a3: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:15.752336 17882 raft_consensus.cc:515] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:15.756182 17882 leader_election.cc:290] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:15.758517 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:20:15.760257 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:20:15.764400 17748 tablet_service.cc:1511] Processing CreateTablet for tablet d11ed0a8f1094765abe028dc60aa8bed (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20260709 12:20:15.766666 17748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d11ed0a8f1094765abe028dc60aa8bed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.767107 17673 tablet_service.cc:1511] Processing CreateTablet for tablet bd3425997bcf4fe7abc0399fd06e54d9 (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20260709 12:20:15.768479 17673 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd3425997bcf4fe7abc0399fd06e54d9. 1 dirs total, 0 dirs full, 0 dirs failed
W20260709 12:20:15.769622 17652 leader_election.cc:343] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:20:15.770370 17652 leader_election.cc:343] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:15.770797 17652 leader_election.cc:304] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:20:15.772079 17678 tablet_service.cc:1511] Processing CreateTablet for tablet d11ed0a8f1094765abe028dc60aa8bed (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20260709 12:20:15.773401 17678 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d11ed0a8f1094765abe028dc60aa8bed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.780619 17882 raft_consensus.cc:2749] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:15.782830 17878 tablet_bootstrap.cc:492] T 217734cc2eb24a7a8d92bfef30ee5c52 P 0da169ff82674dc1947055ce43de68a3: No bootstrap required, opened a new log
I20260709 12:20:15.783613 17878 ts_tablet_manager.cc:1403] T 217734cc2eb24a7a8d92bfef30ee5c52 P 0da169ff82674dc1947055ce43de68a3: Time spent bootstrapping tablet: real 0.038s	user 0.010s	sys 0.005s
I20260709 12:20:15.787032 17881 tablet_bootstrap.cc:492] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b: No bootstrap required, opened a new log
I20260709 12:20:15.787384 17880 tablet_bootstrap.cc:654] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:15.787592 17881 ts_tablet_manager.cc:1403] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b: Time spent bootstrapping tablet: real 0.045s	user 0.012s	sys 0.004s
I20260709 12:20:15.788546 17889 raft_consensus.cc:493] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:15.789031 17889 raft_consensus.cc:515] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:15.790675 17881 raft_consensus.cc:359] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:15.791828 17881 raft_consensus.cc:385] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:15.792587 17881 raft_consensus.cc:740] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Initialized, Role: FOLLOWER
I20260709 12:20:15.791811 17751 tablet_service.cc:1511] Processing CreateTablet for tablet bd3425997bcf4fe7abc0399fd06e54d9 (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20260709 12:20:15.787168 17878 raft_consensus.cc:359] T 217734cc2eb24a7a8d92bfef30ee5c52 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:15.793977 17878 raft_consensus.cc:385] T 217734cc2eb24a7a8d92bfef30ee5c52 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:15.793605 17881 consensus_queue.cc:260] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:15.794337 17878 raft_consensus.cc:740] T 217734cc2eb24a7a8d92bfef30ee5c52 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Initialized, Role: FOLLOWER
I20260709 12:20:15.795284 17878 consensus_queue.cc:260] T 217734cc2eb24a7a8d92bfef30ee5c52 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:15.797823 17878 ts_tablet_manager.cc:1434] T 217734cc2eb24a7a8d92bfef30ee5c52 P 0da169ff82674dc1947055ce43de68a3: Time spent starting tablet: real 0.014s	user 0.006s	sys 0.000s
I20260709 12:20:15.799860 17751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd3425997bcf4fe7abc0399fd06e54d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.799639 17889 leader_election.cc:290] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:15.801427 17823 tablet_service.cc:1511] Processing CreateTablet for tablet bd3425997bcf4fe7abc0399fd06e54d9 (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20260709 12:20:15.801759 17677 tablet_service.cc:1511] Processing CreateTablet for tablet d48b7de191684318aa08f60a7a395387 (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20260709 12:20:15.803292 17823 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd3425997bcf4fe7abc0399fd06e54d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.803296 17677 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d48b7de191684318aa08f60a7a395387. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.807801 17881 ts_tablet_manager.cc:1434] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b: Time spent starting tablet: real 0.020s	user 0.005s	sys 0.002s
I20260709 12:20:15.824242 17881 tablet_bootstrap.cc:492] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b: Bootstrap starting.
I20260709 12:20:15.817612 17878 tablet_bootstrap.cc:492] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3: Bootstrap starting.
I20260709 12:20:15.831739 17826 tablet_service.cc:1511] Processing CreateTablet for tablet d11ed0a8f1094765abe028dc60aa8bed (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20260709 12:20:15.833067 17826 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d11ed0a8f1094765abe028dc60aa8bed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.844475 17878 tablet_bootstrap.cc:654] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:15.844696 17754 tablet_service.cc:1511] Processing CreateTablet for tablet d48b7de191684318aa08f60a7a395387 (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20260709 12:20:15.842403 17888 raft_consensus.cc:493] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:15.845958 17754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d48b7de191684318aa08f60a7a395387. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.846114 17888 raft_consensus.cc:515] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:15.872212 17889 raft_consensus.cc:493] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:15.872738 17889 raft_consensus.cc:515] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:15.876366 17671 tablet_service.cc:1511] Processing CreateTablet for tablet ea7da467817a4ba88aa5797c3f8f493f (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20260709 12:20:15.877730 17671 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea7da467817a4ba88aa5797c3f8f493f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.880036 17889 leader_election.cc:290] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:15.860961 17888 leader_election.cc:290] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:20:15.885551 17678 tablet_service.cc:1511] Processing CreateTablet for tablet 5ad2828ec58f490a9ea564d9805b56b7 (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20260709 12:20:15.885984 17748 tablet_service.cc:1511] Processing CreateTablet for tablet 5ad2828ec58f490a9ea564d9805b56b7 (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20260709 12:20:15.887280 17678 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ad2828ec58f490a9ea564d9805b56b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.887660 17748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ad2828ec58f490a9ea564d9805b56b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.888248 17880 tablet_bootstrap.cc:492] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb: No bootstrap required, opened a new log
I20260709 12:20:15.888757 17880 ts_tablet_manager.cc:1403] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent bootstrapping tablet: real 0.138s	user 0.010s	sys 0.011s
I20260709 12:20:15.889537 17881 tablet_bootstrap.cc:654] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:15.892921 17751 tablet_service.cc:1511] Processing CreateTablet for tablet ea7da467817a4ba88aa5797c3f8f493f (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20260709 12:20:15.894551 17751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea7da467817a4ba88aa5797c3f8f493f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.891711 17880 raft_consensus.cc:359] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:15.900537 17880 raft_consensus.cc:385] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:15.904722 17880 raft_consensus.cc:740] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Initialized, Role: FOLLOWER
I20260709 12:20:15.905405 17878 tablet_bootstrap.cc:492] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3: No bootstrap required, opened a new log
I20260709 12:20:15.905944 17878 ts_tablet_manager.cc:1403] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3: Time spent bootstrapping tablet: real 0.106s	user 0.011s	sys 0.002s
I20260709 12:20:15.905611 17880 consensus_queue.cc:260] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:15.909250 17878 raft_consensus.cc:359] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:15.916735 17878 raft_consensus.cc:385] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:15.917508 17878 raft_consensus.cc:740] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Initialized, Role: FOLLOWER
I20260709 12:20:15.918433 17878 consensus_queue.cc:260] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:15.922118 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:20:15.922472 17827 tablet_service.cc:1511] Processing CreateTablet for tablet ea7da467817a4ba88aa5797c3f8f493f (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20260709 12:20:15.922657 17825 tablet_service.cc:1511] Processing CreateTablet for tablet d48b7de191684318aa08f60a7a395387 (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20260709 12:20:15.923924 17827 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea7da467817a4ba88aa5797c3f8f493f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.924402 17880 ts_tablet_manager.cc:1434] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent starting tablet: real 0.035s	user 0.008s	sys 0.000s
I20260709 12:20:15.924569 17825 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d48b7de191684318aa08f60a7a395387. 1 dirs total, 0 dirs full, 0 dirs failed
W20260709 12:20:15.923769 17727 leader_election.cc:343] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:15.922631 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:20:15.925379 17880 tablet_bootstrap.cc:492] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb: Bootstrap starting.
I20260709 12:20:15.925490 17837 raft_consensus.cc:2468] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 0.
I20260709 12:20:15.928102 17727 leader_election.cc:304] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:20:15.931838 17889 raft_consensus.cc:2804] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:15.931764 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:20:15.932298 17889 raft_consensus.cc:493] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:15.932665 17889 raft_consensus.cc:3060] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:15.932577 17689 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 0.
I20260709 12:20:15.937949 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:20:15.939852 17728 leader_election.cc:343] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:15.940622 17728 leader_election.cc:304] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:20:15.942163 17889 raft_consensus.cc:515] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:15.945640 17878 ts_tablet_manager.cc:1434] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3: Time spent starting tablet: real 0.039s	user 0.006s	sys 0.000s
I20260709 12:20:15.945317 17823 tablet_service.cc:1511] Processing CreateTablet for tablet 5ad2828ec58f490a9ea564d9805b56b7 (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20260709 12:20:15.946662 17823 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ad2828ec58f490a9ea564d9805b56b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.946796 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
W20260709 12:20:15.950858 17728 leader_election.cc:343] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:15.953800 17880 tablet_bootstrap.cc:654] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:15.946728 17878 tablet_bootstrap.cc:492] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3: Bootstrap starting.
I20260709 12:20:15.963655 17882 raft_consensus.cc:493] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:15.964138 17882 raft_consensus.cc:515] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:15.966094 17882 leader_election.cc:290] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:15.959436 17889 leader_election.cc:290] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 1 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:15.977697 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:20:15.977620 17671 tablet_service.cc:1511] Processing CreateTablet for tablet 154e6b769a8d476ba26f7fe3a521fc8d (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20260709 12:20:15.978346 17754 tablet_service.cc:1511] Processing CreateTablet for tablet 154e6b769a8d476ba26f7fe3a521fc8d (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20260709 12:20:15.979131 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:20:15.979398 17652 leader_election.cc:343] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:15.979715 17754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 154e6b769a8d476ba26f7fe3a521fc8d. 1 dirs total, 0 dirs full, 0 dirs failed
W20260709 12:20:15.980722 17652 leader_election.cc:343] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:15.981196 17652 leader_election.cc:304] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:20:15.981873 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:20:15.982407 17671 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 154e6b769a8d476ba26f7fe3a521fc8d. 1 dirs total, 0 dirs full, 0 dirs failed
W20260709 12:20:15.983414 17802 leader_election.cc:343] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:20:15.984261 17790 tablet.cc:2406] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260709 12:20:15.984107 17824 tablet_service.cc:1511] Processing CreateTablet for tablet 154e6b769a8d476ba26f7fe3a521fc8d (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20260709 12:20:15.985545 17824 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 154e6b769a8d476ba26f7fe3a521fc8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.988065 17882 raft_consensus.cc:2749] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260709 12:20:15.987983 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 59960.4us
I20260709 12:20:15.989976 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:20:15.990098 17888 raft_consensus.cc:493] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:15.990546 17837 raft_consensus.cc:3060] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:15.990950 17889 raft_consensus.cc:2804] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:15.991127 17888 raft_consensus.cc:515] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:15.999032 17888 leader_election.cc:290] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:20:15.996624 17889 raft_consensus.cc:493] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:16.000936 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:20:16.001263 17889 raft_consensus.cc:3060] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:16.001592 17689 raft_consensus.cc:2468] T 217734cc2eb24a7a8d92bfef30ee5c52 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 0.
I20260709 12:20:16.001860 17673 tablet_service.cc:1511] Processing CreateTablet for tablet 1a4040191930435786e2eeea338bb3d3 (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20260709 12:20:16.002866 17802 leader_election.cc:304] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:20:16.003157 17673 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a4040191930435786e2eeea338bb3d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:16.003727 17888 raft_consensus.cc:2804] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:16.004105 17888 raft_consensus.cc:493] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:16.004441 17888 raft_consensus.cc:3060] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:16.006939 17878 tablet_bootstrap.cc:654] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:16.017290 17888 raft_consensus.cc:515] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:16.009306 17880 tablet_bootstrap.cc:492] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb: No bootstrap required, opened a new log
I20260709 12:20:16.009802 17752 tablet_service.cc:1511] Processing CreateTablet for tablet 1a4040191930435786e2eeea338bb3d3 (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20260709 12:20:16.018432 17880 ts_tablet_manager.cc:1403] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb: Time spent bootstrapping tablet: real 0.093s	user 0.008s	sys 0.017s
I20260709 12:20:16.014158 17677 tablet_service.cc:1511] Processing CreateTablet for tablet ab4d9071484d4dc0998db462cb01a10f (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20260709 12:20:16.020153 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:20:16.018942 17752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a4040191930435786e2eeea338bb3d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:16.020761 17689 raft_consensus.cc:3060] T 217734cc2eb24a7a8d92bfef30ee5c52 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:16.020778 17677 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ab4d9071484d4dc0998db462cb01a10f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:16.012769 17820 tablet_service.cc:1511] Processing CreateTablet for tablet 1a4040191930435786e2eeea338bb3d3 (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20260709 12:20:16.028858 17820 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a4040191930435786e2eeea338bb3d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:16.030193 17881 tablet_bootstrap.cc:492] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b: No bootstrap required, opened a new log
I20260709 12:20:16.030704 17881 ts_tablet_manager.cc:1403] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b: Time spent bootstrapping tablet: real 0.207s	user 0.010s	sys 0.015s
I20260709 12:20:16.031525 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:20:16.032451 17764 raft_consensus.cc:2393] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 1: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:20:16.029693 17880 raft_consensus.cc:359] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:16.033675 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:20:16.007809 17837 raft_consensus.cc:2468] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 1.
I20260709 12:20:16.035225 17801 leader_election.cc:304] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:20:16.035640 17727 leader_election.cc:304] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:20:16.033912 17880 raft_consensus.cc:385] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:16.036594 17763 raft_consensus.cc:2468] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 0.
I20260709 12:20:16.037153 17905 raft_consensus.cc:2804] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:16.033608 17881 raft_consensus.cc:359] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:16.038990 17881 raft_consensus.cc:385] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:16.039324 17881 raft_consensus.cc:740] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Initialized, Role: FOLLOWER
I20260709 12:20:16.037143 17880 raft_consensus.cc:740] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Initialized, Role: FOLLOWER
I20260709 12:20:16.048801 17689 raft_consensus.cc:2468] T 217734cc2eb24a7a8d92bfef30ee5c52 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 1.
I20260709 12:20:16.044723 17880 consensus_queue.cc:260] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:16.050160 17802 leader_election.cc:304] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:20:16.048509 17881 consensus_queue.cc:260] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:16.057968 17908 raft_consensus.cc:2749] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:16.061086 17908 raft_consensus.cc:2804] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:16.061645 17908 raft_consensus.cc:697] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 1 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:20:16.062053 17748 tablet_service.cc:1511] Processing CreateTablet for tablet ab4d9071484d4dc0998db462cb01a10f (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20260709 12:20:16.062655 17908 consensus_queue.cc:237] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:16.063540 17748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ab4d9071484d4dc0998db462cb01a10f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:16.071975 17905 raft_consensus.cc:697] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 1 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:20:16.072965 17905 consensus_queue.cc:237] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:16.064286 17888 leader_election.cc:290] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 1 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:20:16.065371 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:20:16.065547 17889 raft_consensus.cc:515] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:16.082798 17763 raft_consensus.cc:3060] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:16.096506 17881 ts_tablet_manager.cc:1434] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b: Time spent starting tablet: real 0.065s	user 0.004s	sys 0.015s
I20260709 12:20:16.097067 17823 tablet_service.cc:1511] Processing CreateTablet for tablet 6fcb5bdaa4be4d619bb945ffd44090ba (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20260709 12:20:16.097585 17881 tablet_bootstrap.cc:492] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b: Bootstrap starting.
I20260709 12:20:16.098471 17823 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6fcb5bdaa4be4d619bb945ffd44090ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:16.098953 17678 tablet_service.cc:1511] Processing CreateTablet for tablet 69716f6d5e7549418fdd8d54d7b722ce (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20260709 12:20:16.100207 17678 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69716f6d5e7549418fdd8d54d7b722ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:16.100616 17751 tablet_service.cc:1511] Processing CreateTablet for tablet 6fcb5bdaa4be4d619bb945ffd44090ba (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20260709 12:20:16.101967 17751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6fcb5bdaa4be4d619bb945ffd44090ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:16.103127 17888 raft_consensus.cc:493] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:16.103672 17888 raft_consensus.cc:515] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:16.105582 17889 leader_election.cc:290] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 1 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:16.105953 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:20:16.106523 17689 raft_consensus.cc:3060] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:16.106881 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:20:16.108110 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:20:16.108723 17727 leader_election.cc:343] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:20:16.109598 17802 leader_election.cc:343] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:16.109760 17888 leader_election.cc:290] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:20:16.108100 17880 ts_tablet_manager.cc:1434] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb: Time spent starting tablet: real 0.088s	user 0.013s	sys 0.001s
I20260709 12:20:16.114473 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:20:16.118232 17764 raft_consensus.cc:2468] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 0.
I20260709 12:20:16.118489 17752 tablet_service.cc:1511] Processing CreateTablet for tablet 69716f6d5e7549418fdd8d54d7b722ce (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20260709 12:20:16.119534 17801 leader_election.cc:304] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:20:16.119818 17752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69716f6d5e7549418fdd8d54d7b722ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:16.120350 17888 raft_consensus.cc:2804] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:16.121409 17888 raft_consensus.cc:493] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:16.121701 17880 tablet_bootstrap.cc:492] T c64345b0a2614ba4a382b83feb235cad P e2909cff1da44ad387f96a8ae3aa5afb: Bootstrap starting.
I20260709 12:20:16.121753 17888 raft_consensus.cc:3060] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:16.122709 17881 tablet_bootstrap.cc:654] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:16.121583 17827 tablet_service.cc:1511] Processing CreateTablet for tablet ab4d9071484d4dc0998db462cb01a10f (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20260709 12:20:16.126871 17827 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ab4d9071484d4dc0998db462cb01a10f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:16.129601 17689 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 1.
I20260709 12:20:16.134430 17882 raft_consensus.cc:493] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:16.140269 17882 raft_consensus.cc:515] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:16.142292 17882 leader_election.cc:290] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:16.144635 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:20:16.146064 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:20:16.147738 17652 leader_election.cc:343] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:20:16.148438 17652 leader_election.cc:343] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:16.148844 17652 leader_election.cc:304] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:20:16.149907 17882 raft_consensus.cc:2749] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:16.137339 17880 tablet_bootstrap.cc:654] T c64345b0a2614ba4a382b83feb235cad P e2909cff1da44ad387f96a8ae3aa5afb: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:16.153658 17673 tablet_service.cc:1511] Processing CreateTablet for tablet 6fcb5bdaa4be4d619bb945ffd44090ba (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20260709 12:20:16.156579 17673 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6fcb5bdaa4be4d619bb945ffd44090ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:16.120519 17763 raft_consensus.cc:2468] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 1.
I20260709 12:20:16.166872 17878 tablet_bootstrap.cc:492] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3: No bootstrap required, opened a new log
I20260709 12:20:16.167074 17824 tablet_service.cc:1511] Processing CreateTablet for tablet 69716f6d5e7549418fdd8d54d7b722ce (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20260709 12:20:16.167358 17728 leader_election.cc:304] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:20:16.170302 17754 tablet_service.cc:1511] Processing CreateTablet for tablet ac984176358d453a8800cc9e0e304061 (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20260709 12:20:16.172631 17825 tablet_service.cc:1511] Processing CreateTablet for tablet ac984176358d453a8800cc9e0e304061 (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20260709 12:20:16.174160 17677 tablet_service.cc:1511] Processing CreateTablet for tablet ac984176358d453a8800cc9e0e304061 (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20260709 12:20:16.179692 17754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac984176358d453a8800cc9e0e304061. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:16.185309 17826 tablet_service.cc:1511] Processing CreateTablet for tablet 0540d79e2ec2406588b295145aa6d54d (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20260709 12:20:16.180401 17677 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac984176358d453a8800cc9e0e304061. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:16.175141 17671 tablet_service.cc:1511] Processing CreateTablet for tablet 0540d79e2ec2406588b295145aa6d54d (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20260709 12:20:16.189410 17671 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0540d79e2ec2406588b295145aa6d54d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:16.178011 17878 ts_tablet_manager.cc:1403] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3: Time spent bootstrapping tablet: real 0.232s	user 0.007s	sys 0.014s
I20260709 12:20:16.180979 17824 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69716f6d5e7549418fdd8d54d7b722ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:16.193205 17826 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0540d79e2ec2406588b295145aa6d54d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:16.182826 17748 tablet_service.cc:1511] Processing CreateTablet for tablet 0540d79e2ec2406588b295145aa6d54d (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20260709 12:20:16.194612 17748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0540d79e2ec2406588b295145aa6d54d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:16.194420 17746 tablet_service.cc:1511] Processing CreateTablet for tablet 3418610c3e1a4e28824db2b86acff690 (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20260709 12:20:16.196157 17825 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac984176358d453a8800cc9e0e304061. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:16.197853 17585 catalog_manager.cc:5697] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b reported cstate change: term changed from 0 to 1, leader changed from <none> to e063541542a94969898868bf6eb5ce1b (127.16.29.67). New cstate: current_term: 1 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: HEALTHY } } }
I20260709 12:20:16.200127 17905 raft_consensus.cc:2804] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:16.200670 17905 raft_consensus.cc:697] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:20:16.202001 17878 raft_consensus.cc:359] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:16.202764 17878 raft_consensus.cc:385] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:16.203055 17878 raft_consensus.cc:740] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Initialized, Role: FOLLOWER
I20260709 12:20:16.169464 17888 raft_consensus.cc:515] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:16.201625 17905 consensus_queue.cc:237] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:16.203944 17878 consensus_queue.cc:260] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:16.206499 17679 tablet_service.cc:1511] Processing CreateTablet for tablet 13a48228b0d44d3ea21f474203dd02e3 (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20260709 12:20:16.210328 17675 tablet_service.cc:1511] Processing CreateTablet for tablet e55c231d5fd5471285b2d7879a38f38b (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20260709 12:20:16.216231 17878 ts_tablet_manager.cc:1434] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3: Time spent starting tablet: real 0.024s	user 0.007s	sys 0.000s
I20260709 12:20:16.217267 17878 tablet_bootstrap.cc:492] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3: Bootstrap starting.
I20260709 12:20:16.217541 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:20:16.218101 17764 raft_consensus.cc:3060] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:16.221565 17747 tablet_service.cc:1511] Processing CreateTablet for tablet 13a48228b0d44d3ea21f474203dd02e3 (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20260709 12:20:16.222769 17823 tablet_service.cc:1511] Processing CreateTablet for tablet 3418610c3e1a4e28824db2b86acff690 (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20260709 12:20:16.223687 17749 tablet_service.cc:1511] Processing CreateTablet for tablet e55c231d5fd5471285b2d7879a38f38b (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20260709 12:20:16.224146 17679 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 13a48228b0d44d3ea21f474203dd02e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:16.224251 17823 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3418610c3e1a4e28824db2b86acff690. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:16.228226 17675 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e55c231d5fd5471285b2d7879a38f38b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:16.224045 17888 leader_election.cc:290] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 1 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:20:16.234186 17881 tablet_bootstrap.cc:492] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b: No bootstrap required, opened a new log
I20260709 12:20:16.234047 17676 tablet_service.cc:1511] Processing CreateTablet for tablet 53e726e4ebbb4ca58b6a6102c9b0850f (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20260709 12:20:16.235144 17881 ts_tablet_manager.cc:1403] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b: Time spent bootstrapping tablet: real 0.138s	user 0.013s	sys 0.005s
I20260709 12:20:16.235039 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
W20260709 12:20:16.236714 17802 leader_election.cc:343] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:16.236565 17821 tablet_service.cc:1511] Processing CreateTablet for tablet 53e726e4ebbb4ca58b6a6102c9b0850f (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20260709 12:20:16.237903 17821 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 53e726e4ebbb4ca58b6a6102c9b0850f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:16.238957 17753 tablet_service.cc:1511] Processing CreateTablet for tablet 53e726e4ebbb4ca58b6a6102c9b0850f (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20260709 12:20:16.240139 17881 raft_consensus.cc:359] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:16.240865 17881 raft_consensus.cc:385] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:16.241147 17881 raft_consensus.cc:740] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Initialized, Role: FOLLOWER
I20260709 12:20:16.195866 17746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3418610c3e1a4e28824db2b86acff690. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:16.243547 17676 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 53e726e4ebbb4ca58b6a6102c9b0850f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:16.243745 17747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 13a48228b0d44d3ea21f474203dd02e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:16.244342 17749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e55c231d5fd5471285b2d7879a38f38b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:16.249189 17753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 53e726e4ebbb4ca58b6a6102c9b0850f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:16.253472 17764 raft_consensus.cc:2468] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 1.
I20260709 12:20:16.253733 17819 tablet_service.cc:1511] Processing CreateTablet for tablet f5f0dd0890f44e35bbb838da838faa9c (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20260709 12:20:16.254755 17801 leader_election.cc:304] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:20:16.254594 17881 consensus_queue.cc:260] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:16.255205 17819 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f5f0dd0890f44e35bbb838da838faa9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:16.255936 17888 raft_consensus.cc:2804] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:16.256461 17888 raft_consensus.cc:697] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 1 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:20:16.256994 17881 ts_tablet_manager.cc:1434] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b: Time spent starting tablet: real 0.021s	user 0.007s	sys 0.001s
I20260709 12:20:16.257323 17673 tablet_service.cc:1511] Processing CreateTablet for tablet f5f0dd0890f44e35bbb838da838faa9c (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20260709 12:20:16.258528 17827 tablet_service.cc:1511] Processing CreateTablet for tablet e55c231d5fd5471285b2d7879a38f38b (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20260709 12:20:16.259769 17827 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e55c231d5fd5471285b2d7879a38f38b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:16.260221 17750 tablet_service.cc:1511] Processing CreateTablet for tablet f5f0dd0890f44e35bbb838da838faa9c (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20260709 12:20:16.228835 17586 catalog_manager.cc:5697] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb reported cstate change: term changed from 0 to 1, leader changed from <none> to e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66). New cstate: current_term: 1 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:16.228163 17820 tablet_service.cc:1511] Processing CreateTablet for tablet 13a48228b0d44d3ea21f474203dd02e3 (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20260709 12:20:16.261947 17750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f5f0dd0890f44e35bbb838da838faa9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:16.262427 17820 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 13a48228b0d44d3ea21f474203dd02e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:16.257378 17888 consensus_queue.cc:237] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:16.258003 17881 tablet_bootstrap.cc:492] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b: Bootstrap starting.
I20260709 12:20:16.266184 17673 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f5f0dd0890f44e35bbb838da838faa9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:16.283454 17674 tablet_service.cc:1511] Processing CreateTablet for tablet 00a02ab39fcd4078ad49697693cf3be8 (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20260709 12:20:16.284060 17678 tablet_service.cc:1511] Processing CreateTablet for tablet 3418610c3e1a4e28824db2b86acff690 (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20260709 12:20:16.283984 17745 tablet_service.cc:1511] Processing CreateTablet for tablet 00a02ab39fcd4078ad49697693cf3be8 (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20260709 12:20:16.286111 17745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00a02ab39fcd4078ad49697693cf3be8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:16.288878 17822 tablet_service.cc:1511] Processing CreateTablet for tablet 00a02ab39fcd4078ad49697693cf3be8 (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20260709 12:20:16.296705 17822 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00a02ab39fcd4078ad49697693cf3be8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:16.285032 17674 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00a02ab39fcd4078ad49697693cf3be8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:16.302385 17678 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3418610c3e1a4e28824db2b86acff690. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:16.306217 17908 raft_consensus.cc:493] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:16.306792 17908 raft_consensus.cc:515] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:16.310694 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:20:16.310981 17908 leader_election.cc:290] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:20:16.311401 17689 raft_consensus.cc:2468] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 0.
I20260709 12:20:16.315037 17672 tablet_service.cc:1511] Processing CreateTablet for tablet 8cab6f01c6a2485cb5024322a1bd5462 (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20260709 12:20:16.316344 17672 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8cab6f01c6a2485cb5024322a1bd5462. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:16.318342 17802 leader_election.cc:304] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:20:16.319757 17908 raft_consensus.cc:2804] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:16.320129 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:20:16.321748 17801 leader_election.cc:343] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:16.321789 17820 tablet_service.cc:1511] Processing CreateTablet for tablet 8cab6f01c6a2485cb5024322a1bd5462 (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20260709 12:20:16.323737 17820 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8cab6f01c6a2485cb5024322a1bd5462. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:16.325249 17752 tablet_service.cc:1511] Processing CreateTablet for tablet 8cab6f01c6a2485cb5024322a1bd5462 (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20260709 12:20:16.326575 17752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8cab6f01c6a2485cb5024322a1bd5462. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:16.320164 17908 raft_consensus.cc:493] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:16.328366 17908 raft_consensus.cc:3060] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:16.341504 17878 tablet_bootstrap.cc:654] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:16.343202 17670 tablet_service.cc:1511] Processing CreateTablet for tablet 0e02889970bc40dc8ca2e81483f211e0 (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20260709 12:20:16.362851 17825 tablet_service.cc:1511] Processing CreateTablet for tablet 0e02889970bc40dc8ca2e81483f211e0 (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20260709 12:20:16.365223 17748 tablet_service.cc:1511] Processing CreateTablet for tablet 0e02889970bc40dc8ca2e81483f211e0 (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20260709 12:20:16.366500 17748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0e02889970bc40dc8ca2e81483f211e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:16.386418 17670 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0e02889970bc40dc8ca2e81483f211e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:16.397354 17881 tablet_bootstrap.cc:654] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:16.400767 17825 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0e02889970bc40dc8ca2e81483f211e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:16.370878 17908 raft_consensus.cc:515] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:16.440225 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:20:16.440371 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:20:16.440876 17689 raft_consensus.cc:3060] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:20:16.441847 17801 leader_election.cc:343] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:16.443980 17908 leader_election.cc:290] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 1 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:20:16.457927 17689 raft_consensus.cc:2468] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 1.
I20260709 12:20:16.459386 17802 leader_election.cc:304] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:20:16.460510 17908 raft_consensus.cc:2804] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:16.460975 17908 raft_consensus.cc:697] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 1 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:20:16.461771 17908 consensus_queue.cc:237] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:20:16.470235 17571 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 54960us
I20260709 12:20:16.479473 17880 tablet_bootstrap.cc:492] T c64345b0a2614ba4a382b83feb235cad P e2909cff1da44ad387f96a8ae3aa5afb: No bootstrap required, opened a new log
I20260709 12:20:16.479293 17675 tablet_service.cc:1511] Processing CreateTablet for tablet 1a698a75c7954dd7b24e7889cb19ae35 (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20260709 12:20:16.485015 17878 tablet_bootstrap.cc:492] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3: No bootstrap required, opened a new log
W20260709 12:20:16.485184 17635 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:20:16.485479 17878 ts_tablet_manager.cc:1403] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3: Time spent bootstrapping tablet: real 0.268s	user 0.044s	sys 0.038s
I20260709 12:20:16.488631 17878 raft_consensus.cc:359] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:16.489360 17878 raft_consensus.cc:385] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:16.489671 17878 raft_consensus.cc:740] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Initialized, Role: FOLLOWER
I20260709 12:20:16.481401 17675 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a698a75c7954dd7b24e7889cb19ae35. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:16.495492 17905 consensus_queue.cc:1048] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:16.496672 17673 tablet_service.cc:1511] Processing CreateTablet for tablet ea2f2b5792f84bbba2738f1ce1e90122 (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20260709 12:20:16.498299 17673 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea2f2b5792f84bbba2738f1ce1e90122. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:16.504509 17754 tablet_service.cc:1511] Processing CreateTablet for tablet ea2f2b5792f84bbba2738f1ce1e90122 (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20260709 12:20:16.504951 17676 tablet_service.cc:1511] Processing CreateTablet for tablet 9160d9f98b8c406db2771d9ea2e0995f (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20260709 12:20:16.510309 17754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea2f2b5792f84bbba2738f1ce1e90122. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:16.510886 17825 tablet_service.cc:1511] Processing CreateTablet for tablet ea2f2b5792f84bbba2738f1ce1e90122 (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20260709 12:20:16.512130 17825 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea2f2b5792f84bbba2738f1ce1e90122. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:16.514608 17827 tablet_service.cc:1511] Processing CreateTablet for tablet 1a698a75c7954dd7b24e7889cb19ae35 (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20260709 12:20:16.516561 17746 tablet_service.cc:1511] Processing CreateTablet for tablet 1a698a75c7954dd7b24e7889cb19ae35 (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20260709 12:20:16.516963 17827 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a698a75c7954dd7b24e7889cb19ae35. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:16.483472 17880 ts_tablet_manager.cc:1403] T c64345b0a2614ba4a382b83feb235cad P e2909cff1da44ad387f96a8ae3aa5afb: Time spent bootstrapping tablet: real 0.362s	user 0.029s	sys 0.044s
I20260709 12:20:16.525321 17888 raft_consensus.cc:493] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:16.526098 17888 raft_consensus.cc:515] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:16.508273 17878 consensus_queue.cc:260] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:16.528931 17674 tablet_service.cc:1511] Processing CreateTablet for tablet 12b1fcd35e12402e8d6df7504c578e01 (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20260709 12:20:16.531368 17878 ts_tablet_manager.cc:1434] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3: Time spent starting tablet: real 0.046s	user 0.007s	sys 0.001s
I20260709 12:20:16.532230 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:20:16.534296 17801 leader_election.cc:343] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:16.537979 17676 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9160d9f98b8c406db2771d9ea2e0995f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:16.539316 17881 tablet_bootstrap.cc:492] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b: No bootstrap required, opened a new log
I20260709 12:20:16.535450 17888 leader_election.cc:290] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:20:16.541746 17878 tablet_bootstrap.cc:492] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3: Bootstrap starting.
I20260709 12:20:16.546589 17674 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 12b1fcd35e12402e8d6df7504c578e01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:16.549969 17888 consensus_queue.cc:1048] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260709 12:20:16.553404 17728 consensus_peers.cc:597] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:16.556802 17881 ts_tablet_manager.cc:1403] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b: Time spent bootstrapping tablet: real 0.299s	user 0.016s	sys 0.008s
I20260709 12:20:16.569211 17750 tablet_service.cc:1511] Processing CreateTablet for tablet 9160d9f98b8c406db2771d9ea2e0995f (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20260709 12:20:16.575639 17878 tablet_bootstrap.cc:654] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:16.556468 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:20:16.579908 17825 tablet_service.cc:1511] Processing CreateTablet for tablet 9160d9f98b8c406db2771d9ea2e0995f (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20260709 12:20:16.578073 17746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a698a75c7954dd7b24e7889cb19ae35. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:16.581271 17825 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9160d9f98b8c406db2771d9ea2e0995f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:16.585441 17673 tablet_service.cc:1511] Processing CreateTablet for tablet 10f96c39fba3496696f64385105bfeaa (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20260709 12:20:16.586562 17673 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 10f96c39fba3496696f64385105bfeaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:16.586731 17880 raft_consensus.cc:359] T c64345b0a2614ba4a382b83feb235cad P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:16.586930 17819 tablet_service.cc:1511] Processing CreateTablet for tablet 12b1fcd35e12402e8d6df7504c578e01 (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20260709 12:20:16.587529 17880 raft_consensus.cc:385] T c64345b0a2614ba4a382b83feb235cad P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:16.583632 17689 raft_consensus.cc:2468] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 0.
I20260709 12:20:16.588245 17819 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 12b1fcd35e12402e8d6df7504c578e01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:16.597005 17750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9160d9f98b8c406db2771d9ea2e0995f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:16.596056 17802 leader_election.cc:304] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:20:16.589067 17752 tablet_service.cc:1511] Processing CreateTablet for tablet 12b1fcd35e12402e8d6df7504c578e01 (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20260709 12:20:16.614378 17908 raft_consensus.cc:2804] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:16.614766 17908 raft_consensus.cc:493] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:16.615099 17908 raft_consensus.cc:3060] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:16.617749 17752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 12b1fcd35e12402e8d6df7504c578e01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:16.595290 17882 raft_consensus.cc:493] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:16.618386 17882 raft_consensus.cc:515] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:16.618516 17749 tablet_service.cc:1511] Processing CreateTablet for tablet 10f96c39fba3496696f64385105bfeaa (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20260709 12:20:16.619829 17749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 10f96c39fba3496696f64385105bfeaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:16.622215 17908 raft_consensus.cc:515] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:16.626138 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:20:16.626674 17908 leader_election.cc:290] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 1 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:20:16.626694 17888 consensus_queue.cc:1048] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:16.628024 17905 consensus_queue.cc:1048] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:20:16.628815 17882 leader_election.cc:290] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:16.629890 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:20:16.630023 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:20:16.631053 17689 raft_consensus.cc:3060] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:16.632460 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
W20260709 12:20:16.634163 17801 leader_election.cc:343] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:16.630626 17837 raft_consensus.cc:2393] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 1: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:20:16.587865 17880 raft_consensus.cc:740] T c64345b0a2614ba4a382b83feb235cad P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Initialized, Role: FOLLOWER
W20260709 12:20:16.634853 17652 leader_election.cc:343] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:16.640735 17878 tablet_bootstrap.cc:492] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3: No bootstrap required, opened a new log
I20260709 12:20:16.649945 17878 ts_tablet_manager.cc:1403] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3: Time spent bootstrapping tablet: real 0.113s	user 0.024s	sys 0.000s
I20260709 12:20:16.598084 17881 raft_consensus.cc:359] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:16.662974 17652 leader_election.cc:304] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:20:16.663455 17670 tablet_service.cc:1511] Processing CreateTablet for tablet ab84345f9aa742a7b85389e9b6f2ce90 (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20260709 12:20:16.664808 17825 tablet_service.cc:1511] Processing CreateTablet for tablet 10f96c39fba3496696f64385105bfeaa (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20260709 12:20:16.665637 17825 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 10f96c39fba3496696f64385105bfeaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:16.659801 17881 raft_consensus.cc:385] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:16.683007 17881 raft_consensus.cc:740] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Initialized, Role: FOLLOWER
I20260709 12:20:16.699527 17880 consensus_queue.cc:260] T c64345b0a2614ba4a382b83feb235cad P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:16.758482 17878 raft_consensus.cc:359] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:16.759001 17878 raft_consensus.cc:385] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:16.759276 17878 raft_consensus.cc:740] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Initialized, Role: FOLLOWER
I20260709 12:20:16.764606 17753 tablet_service.cc:1511] Processing CreateTablet for tablet ab84345f9aa742a7b85389e9b6f2ce90 (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20260709 12:20:16.765946 17753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ab84345f9aa742a7b85389e9b6f2ce90. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:16.766256 17675 tablet_service.cc:1511] Processing CreateTablet for tablet e2bcfe1b94464b6d9620cc3b8a17a2ad (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
W20260709 12:20:16.767772 17802 consensus_peers.cc:597] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:16.764572 17878 consensus_queue.cc:260] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:16.776889 17878 ts_tablet_manager.cc:1434] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3: Time spent starting tablet: real 0.127s	user 0.009s	sys 0.000s
I20260709 12:20:16.778158 17878 tablet_bootstrap.cc:492] T 8b56346a04a14c5d9f73c5205b74cf2d P 0da169ff82674dc1947055ce43de68a3: Bootstrap starting.
W20260709 12:20:16.762265 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 99400.1us
I20260709 12:20:16.770721 17825 tablet_service.cc:1511] Processing CreateTablet for tablet ab84345f9aa742a7b85389e9b6f2ce90 (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20260709 12:20:16.781829 17825 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ab84345f9aa742a7b85389e9b6f2ce90. 1 dirs total, 0 dirs full, 0 dirs failed
W20260709 12:20:16.783130 17727 consensus_peers.cc:597] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb -> Peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837): Couldn't send request to peer e063541542a94969898868bf6eb5ce1b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:16.774887 17933 raft_consensus.cc:493] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:16.763036 17881 consensus_queue.cc:260] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:16.784235 17933 raft_consensus.cc:515] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:16.786672 17933 leader_election.cc:290] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:16.784197 17670 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ab84345f9aa742a7b85389e9b6f2ce90. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:16.786969 17747 tablet_service.cc:1511] Processing CreateTablet for tablet 8a208e793881438b975b83755dc05cfa (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20260709 12:20:16.788727 17747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a208e793881438b975b83755dc05cfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:16.790341 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:20:16.791414 17675 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e2bcfe1b94464b6d9620cc3b8a17a2ad. 1 dirs total, 0 dirs full, 0 dirs failed
W20260709 12:20:16.791749 17652 leader_election.cc:343] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:16.801946 17823 tablet_service.cc:1511] Processing CreateTablet for tablet 8a208e793881438b975b83755dc05cfa (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20260709 12:20:16.803830 17823 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a208e793881438b975b83755dc05cfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:16.773644 17689 raft_consensus.cc:2468] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 1.
I20260709 12:20:16.805493 17882 raft_consensus.cc:2749] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:16.816448 17802 leader_election.cc:304] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:20:16.819422 17888 raft_consensus.cc:2804] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:16.820304 17888 raft_consensus.cc:697] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 1 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:20:16.821501 17888 consensus_queue.cc:237] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:16.827592 17820 tablet_service.cc:1511] Processing CreateTablet for tablet e2bcfe1b94464b6d9620cc3b8a17a2ad (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20260709 12:20:16.828975 17820 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e2bcfe1b94464b6d9620cc3b8a17a2ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:16.829706 17678 tablet_service.cc:1511] Processing CreateTablet for tablet f8f6bfcb37be41c6917282a17239ad06 (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20260709 12:20:16.830950 17678 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f8f6bfcb37be41c6917282a17239ad06. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:16.830603 17748 tablet_service.cc:1511] Processing CreateTablet for tablet f8f6bfcb37be41c6917282a17239ad06 (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20260709 12:20:16.832211 17748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f8f6bfcb37be41c6917282a17239ad06. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:16.787328 17881 ts_tablet_manager.cc:1434] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b: Time spent starting tablet: real 0.213s	user 0.007s	sys 0.004s
I20260709 12:20:16.838567 17881 tablet_bootstrap.cc:492] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b: Bootstrap starting.
I20260709 12:20:16.844815 17673 tablet_service.cc:1511] Processing CreateTablet for tablet 8a208e793881438b975b83755dc05cfa (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20260709 12:20:16.846135 17673 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a208e793881438b975b83755dc05cfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:16.820299 17752 tablet_service.cc:1511] Processing CreateTablet for tablet e2bcfe1b94464b6d9620cc3b8a17a2ad (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20260709 12:20:16.890017 17752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e2bcfe1b94464b6d9620cc3b8a17a2ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:16.912870 17749 tablet_service.cc:1511] Processing CreateTablet for tablet 56b500bd1b914ebfb3f19afc8a330665 (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20260709 12:20:16.915367 17825 tablet_service.cc:1511] Processing CreateTablet for tablet f8f6bfcb37be41c6917282a17239ad06 (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20260709 12:20:16.916308 17749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 56b500bd1b914ebfb3f19afc8a330665. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:16.916496 17825 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f8f6bfcb37be41c6917282a17239ad06. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:16.900835 17676 tablet_service.cc:1511] Processing CreateTablet for tablet 56b500bd1b914ebfb3f19afc8a330665 (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20260709 12:20:16.904208 17878 tablet_bootstrap.cc:654] T 8b56346a04a14c5d9f73c5205b74cf2d P 0da169ff82674dc1947055ce43de68a3: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:16.912158 17908 consensus_queue.cc:1048] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:16.936647 17880 ts_tablet_manager.cc:1434] T c64345b0a2614ba4a382b83feb235cad P e2909cff1da44ad387f96a8ae3aa5afb: Time spent starting tablet: real 0.406s	user 0.018s	sys 0.016s
I20260709 12:20:16.938611 17586 catalog_manager.cc:5697] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 reported cstate change: term changed from 0 to 1, leader changed from <none> to e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66). New cstate: current_term: 1 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } }
I20260709 12:20:16.939404 17678 tablet_service.cc:1511] Processing CreateTablet for tablet 23c0cde5227f470388b8ed1fc0786119 (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20260709 12:20:16.945781 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:20:16.949651 17908 consensus_queue.cc:1048] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260709 12:20:16.952638 17678 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 23c0cde5227f470388b8ed1fc0786119. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:16.953950 17748 tablet_service.cc:1511] Processing CreateTablet for tablet 23c0cde5227f470388b8ed1fc0786119 (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20260709 12:20:16.954756 17748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 23c0cde5227f470388b8ed1fc0786119. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:16.947641 17880 tablet_bootstrap.cc:492] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb: Bootstrap starting.
W20260709 12:20:16.959306 17652 leader_election.cc:343] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:16.959787 17652 leader_election.cc:304] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:20:16.961030 17934 raft_consensus.cc:2749] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:16.957098 17908 raft_consensus.cc:493] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:16.964531 17673 tablet_service.cc:1511] Processing CreateTablet for tablet 22fa38faae58422e9cecc99973cab688 (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20260709 12:20:16.964576 17908 raft_consensus.cc:515] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:16.966830 17908 leader_election.cc:290] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:20:16.969059 17747 tablet_service.cc:1511] Processing CreateTablet for tablet 22fa38faae58422e9cecc99973cab688 (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20260709 12:20:16.969951 17881 tablet_bootstrap.cc:654] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:16.969812 17674 tablet_service.cc:1511] Processing CreateTablet for tablet 50f9ff8f24a14a079db34b1ba88fa278 (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20260709 12:20:16.980870 17673 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 22fa38faae58422e9cecc99973cab688. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:16.979595 17825 tablet_service.cc:1511] Processing CreateTablet for tablet 23c0cde5227f470388b8ed1fc0786119 (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20260709 12:20:16.975095 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:20:16.977216 17679 tablet_service.cc:1511] Processing CreateTablet for tablet 9439565c5dbb4f4ba3375aaef2c9add5 (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20260709 12:20:16.982023 17687 raft_consensus.cc:2468] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 0.
I20260709 12:20:16.974678 17819 tablet_service.cc:1511] Processing CreateTablet for tablet 22fa38faae58422e9cecc99973cab688 (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20260709 12:20:16.982595 17827 tablet_service.cc:1511] Processing CreateTablet for tablet 50f9ff8f24a14a079db34b1ba88fa278 (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20260709 12:20:16.981487 17676 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 56b500bd1b914ebfb3f19afc8a330665. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:16.983990 17747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 22fa38faae58422e9cecc99973cab688. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:16.984589 17750 tablet_service.cc:1511] Processing CreateTablet for tablet 9439565c5dbb4f4ba3375aaef2c9add5 (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20260709 12:20:16.985824 17750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9439565c5dbb4f4ba3375aaef2c9add5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:16.983494 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:20:16.992134 17674 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 50f9ff8f24a14a079db34b1ba88fa278. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:16.996275 17679 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9439565c5dbb4f4ba3375aaef2c9add5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:16.996601 17819 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 22fa38faae58422e9cecc99973cab688. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:16.997189 17827 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 50f9ff8f24a14a079db34b1ba88fa278. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:17.000077 17820 tablet_service.cc:1511] Processing CreateTablet for tablet 56b500bd1b914ebfb3f19afc8a330665 (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20260709 12:20:17.000537 17672 tablet_service.cc:1511] Processing CreateTablet for tablet 19a1253e500942cd8a1a0f568159cfa3 (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20260709 12:20:17.008849 17825 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 23c0cde5227f470388b8ed1fc0786119. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:17.014210 17748 tablet_service.cc:1511] Processing CreateTablet for tablet 19a1253e500942cd8a1a0f568159cfa3 (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20260709 12:20:17.015659 17748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 19a1253e500942cd8a1a0f568159cfa3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:17.016646 17672 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 19a1253e500942cd8a1a0f568159cfa3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:17.018615 17764 raft_consensus.cc:3060] T c64345b0a2614ba4a382b83feb235cad P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:17.039744 17881 tablet_bootstrap.cc:492] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b: No bootstrap required, opened a new log
I20260709 12:20:17.040439 17881 ts_tablet_manager.cc:1403] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b: Time spent bootstrapping tablet: real 0.202s	user 0.009s	sys 0.008s
I20260709 12:20:17.040822 17880 tablet_bootstrap.cc:654] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:17.043468 17878 tablet_bootstrap.cc:492] T 8b56346a04a14c5d9f73c5205b74cf2d P 0da169ff82674dc1947055ce43de68a3: No bootstrap required, opened a new log
I20260709 12:20:17.043973 17878 ts_tablet_manager.cc:1403] T 8b56346a04a14c5d9f73c5205b74cf2d P 0da169ff82674dc1947055ce43de68a3: Time spent bootstrapping tablet: real 0.266s	user 0.013s	sys 0.004s
I20260709 12:20:16.974045 17753 tablet_service.cc:1511] Processing CreateTablet for tablet 50f9ff8f24a14a079db34b1ba88fa278 (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20260709 12:20:17.048804 17753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 50f9ff8f24a14a079db34b1ba88fa278. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:17.056648 17585 catalog_manager.cc:5697] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b reported cstate change: term changed from 0 to 1, leader changed from <none> to e063541542a94969898868bf6eb5ce1b (127.16.29.67). New cstate: current_term: 1 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: HEALTHY } } }
I20260709 12:20:17.058593 17585 catalog_manager.cc:5697] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b reported cstate change: term changed from 0 to 1, leader changed from <none> to e063541542a94969898868bf6eb5ce1b (127.16.29.67). New cstate: current_term: 1 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: HEALTHY } } }
I20260709 12:20:17.021965 17802 leader_election.cc:304] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
W20260709 12:20:17.060361 17801 leader_election.cc:343] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:20:17.060598 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 57047.6us
I20260709 12:20:17.062855 17908 raft_consensus.cc:2804] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:17.063354 17908 raft_consensus.cc:493] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:17.063745 17908 raft_consensus.cc:3060] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:17.063567 17823 tablet_service.cc:1511] Processing CreateTablet for tablet 9439565c5dbb4f4ba3375aaef2c9add5 (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20260709 12:20:17.064280 17820 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 56b500bd1b914ebfb3f19afc8a330665. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:17.065034 17823 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9439565c5dbb4f4ba3375aaef2c9add5. 1 dirs total, 0 dirs full, 0 dirs failed
W20260709 12:20:17.062862 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 44517.2us
I20260709 12:20:17.069967 17912 consensus_queue.cc:1048] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:17.082432 17821 tablet_service.cc:1511] Processing CreateTablet for tablet 19a1253e500942cd8a1a0f568159cfa3 (DEFAULT_TABLE table=test-workload [id=ddc5b74250df4c65909b73b3742d615f]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20260709 12:20:17.083864 17821 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 19a1253e500942cd8a1a0f568159cfa3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:17.090387 17878 raft_consensus.cc:359] T 8b56346a04a14c5d9f73c5205b74cf2d P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:17.091137 17878 raft_consensus.cc:385] T 8b56346a04a14c5d9f73c5205b74cf2d P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:17.091435 17878 raft_consensus.cc:740] T 8b56346a04a14c5d9f73c5205b74cf2d P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Initialized, Role: FOLLOWER
I20260709 12:20:17.101634 17878 consensus_queue.cc:260] T 8b56346a04a14c5d9f73c5205b74cf2d P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:17.134095 17881 raft_consensus.cc:359] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:17.138262 17878 ts_tablet_manager.cc:1434] T 8b56346a04a14c5d9f73c5205b74cf2d P 0da169ff82674dc1947055ce43de68a3: Time spent starting tablet: real 0.094s	user 0.006s	sys 0.000s
I20260709 12:20:17.139555 17878 tablet_bootstrap.cc:492] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3: Bootstrap starting.
I20260709 12:20:17.136065 17881 raft_consensus.cc:385] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:17.148056 17881 raft_consensus.cc:740] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Initialized, Role: FOLLOWER
I20260709 12:20:17.153899 17881 consensus_queue.cc:260] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:17.162174 17940 raft_consensus.cc:493] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:17.162781 17940 raft_consensus.cc:515] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:17.166162 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:20:17.167681 17652 leader_election.cc:343] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:17.169312 17881 ts_tablet_manager.cc:1434] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b: Time spent starting tablet: real 0.089s	user 0.007s	sys 0.004s
I20260709 12:20:17.169592 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:20:17.171972 17837 raft_consensus.cc:2468] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 0.
I20260709 12:20:17.172204 17881 tablet_bootstrap.cc:492] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b: Bootstrap starting.
I20260709 12:20:17.173398 17652 leader_election.cc:304] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:20:17.167707 17940 leader_election.cc:290] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:17.174263 17934 raft_consensus.cc:2804] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:17.174782 17934 raft_consensus.cc:493] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:17.175109 17934 raft_consensus.cc:3060] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:17.179831 17881 tablet_bootstrap.cc:654] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:17.185858 17878 tablet_bootstrap.cc:654] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:17.182840 17908 raft_consensus.cc:515] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:17.196723 17908 leader_election.cc:290] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 1 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:20:17.187764 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 52531.4us
I20260709 12:20:17.198010 17934 raft_consensus.cc:515] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:17.204849 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:20:17.205991 17934 leader_election.cc:290] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:17.207510 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:20:17.208097 17837 raft_consensus.cc:3060] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:17.208648 17880 tablet_bootstrap.cc:492] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb: No bootstrap required, opened a new log
I20260709 12:20:17.209357 17880 ts_tablet_manager.cc:1403] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb: Time spent bootstrapping tablet: real 0.267s	user 0.017s	sys 0.019s
I20260709 12:20:17.211359 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:20:17.213279 17880 raft_consensus.cc:359] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:17.214272 17880 raft_consensus.cc:385] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:17.214654 17880 raft_consensus.cc:740] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Initialized, Role: FOLLOWER
W20260709 12:20:17.216284 17652 leader_election.cc:343] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:20:17.217170 17801 leader_election.cc:343] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:17.221925 17688 raft_consensus.cc:3060] T 8b56346a04a14c5d9f73c5205b74cf2d P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:17.215757 17880 consensus_queue.cc:260] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:17.229301 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:20:17.230139 17881 tablet_bootstrap.cc:492] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b: No bootstrap required, opened a new log
I20260709 12:20:17.230633 17881 ts_tablet_manager.cc:1403] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b: Time spent bootstrapping tablet: real 0.059s	user 0.021s	sys 0.011s
I20260709 12:20:17.231488 17687 raft_consensus.cc:3060] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:17.234062 17880 ts_tablet_manager.cc:1434] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb: Time spent starting tablet: real 0.024s	user 0.008s	sys 0.000s
I20260709 12:20:17.235237 17880 tablet_bootstrap.cc:492] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb: Bootstrap starting.
I20260709 12:20:17.240809 17687 raft_consensus.cc:2468] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 1.
I20260709 12:20:17.242388 17880 tablet_bootstrap.cc:654] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:17.246929 17762 raft_consensus.cc:3060] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:17.252730 17802 leader_election.cc:304] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:20:17.255693 17584 catalog_manager.cc:5697] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b reported cstate change: term changed from 0 to 1, leader changed from <none> to e063541542a94969898868bf6eb5ce1b (127.16.29.67). New cstate: current_term: 1 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: HEALTHY } } }
I20260709 12:20:17.246819 17881 raft_consensus.cc:359] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:17.220103 17837 raft_consensus.cc:2468] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 1.
I20260709 12:20:17.260102 17881 raft_consensus.cc:385] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:17.249233 17878 tablet_bootstrap.cc:492] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3: No bootstrap required, opened a new log
I20260709 12:20:17.260448 17951 raft_consensus.cc:2804] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:17.260548 17881 raft_consensus.cc:740] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Initialized, Role: FOLLOWER
I20260709 12:20:17.261089 17951 raft_consensus.cc:697] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 1 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:20:17.261476 17652 leader_election.cc:304] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:20:17.262310 17934 raft_consensus.cc:2804] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:17.262015 17951 consensus_queue.cc:237] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:17.262806 17878 ts_tablet_manager.cc:1403] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3: Time spent bootstrapping tablet: real 0.124s	user 0.018s	sys 0.004s
I20260709 12:20:17.262825 17934 raft_consensus.cc:697] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 1 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:20:17.263475 17881 consensus_queue.cc:260] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:17.263731 17934 consensus_queue.cc:237] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:17.265475 17878 raft_consensus.cc:359] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:17.266134 17878 raft_consensus.cc:385] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:17.266533 17878 raft_consensus.cc:740] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Initialized, Role: FOLLOWER
I20260709 12:20:17.267309 17878 consensus_queue.cc:260] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:17.268857 17881 ts_tablet_manager.cc:1434] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b: Time spent starting tablet: real 0.038s	user 0.010s	sys 0.000s
I20260709 12:20:17.270092 17881 tablet_bootstrap.cc:492] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b: Bootstrap starting.
I20260709 12:20:17.270162 17912 consensus_queue.cc:1048] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:20:17.270110 17878 ts_tablet_manager.cc:1434] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3: Time spent starting tablet: real 0.007s	user 0.006s	sys 0.000s
I20260709 12:20:17.271672 17912 consensus_queue.cc:1048] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:20:17.272015 17878 tablet_bootstrap.cc:492] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3: Bootstrap starting.
I20260709 12:20:17.282924 17881 tablet_bootstrap.cc:654] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:17.294793 17878 tablet_bootstrap.cc:654] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:17.295183 17880 tablet_bootstrap.cc:492] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb: No bootstrap required, opened a new log
I20260709 12:20:17.296653 17880 ts_tablet_manager.cc:1403] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent bootstrapping tablet: real 0.062s	user 0.011s	sys 0.015s
I20260709 12:20:17.306090 17880 raft_consensus.cc:359] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:17.306838 17880 raft_consensus.cc:385] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:17.307152 17880 raft_consensus.cc:740] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Initialized, Role: FOLLOWER
I20260709 12:20:17.307981 17880 consensus_queue.cc:260] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:17.312140 17880 ts_tablet_manager.cc:1434] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent starting tablet: real 0.010s	user 0.006s	sys 0.003s
I20260709 12:20:17.313501 17880 tablet_bootstrap.cc:492] T 7cb3768d215e409fb9f5baf18d124442 P e2909cff1da44ad387f96a8ae3aa5afb: Bootstrap starting.
I20260709 12:20:17.321064 17880 tablet_bootstrap.cc:654] T 7cb3768d215e409fb9f5baf18d124442 P e2909cff1da44ad387f96a8ae3aa5afb: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:17.323065 17586 catalog_manager.cc:5697] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0da169ff82674dc1947055ce43de68a3 (127.16.29.65). New cstate: current_term: 1 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:17.330389 17934 raft_consensus.cc:493] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:17.331058 17934 raft_consensus.cc:515] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:17.334096 17934 leader_election.cc:290] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:17.335263 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:20:17.335260 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:20:17.336398 17837 raft_consensus.cc:2468] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 0.
W20260709 12:20:17.336714 17652 leader_election.cc:343] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:17.337563 17652 leader_election.cc:304] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:20:17.338665 17934 raft_consensus.cc:2804] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:17.339016 17934 raft_consensus.cc:493] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:17.339340 17934 raft_consensus.cc:3060] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:17.340095 17880 tablet_bootstrap.cc:492] T 7cb3768d215e409fb9f5baf18d124442 P e2909cff1da44ad387f96a8ae3aa5afb: No bootstrap required, opened a new log
I20260709 12:20:17.340533 17880 ts_tablet_manager.cc:1403] T 7cb3768d215e409fb9f5baf18d124442 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent bootstrapping tablet: real 0.027s	user 0.015s	sys 0.005s
I20260709 12:20:17.338685 17881 tablet_bootstrap.cc:492] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b: No bootstrap required, opened a new log
I20260709 12:20:17.343801 17880 raft_consensus.cc:359] T 7cb3768d215e409fb9f5baf18d124442 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:17.344455 17881 ts_tablet_manager.cc:1403] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b: Time spent bootstrapping tablet: real 0.075s	user 0.013s	sys 0.002s
I20260709 12:20:17.344659 17880 raft_consensus.cc:385] T 7cb3768d215e409fb9f5baf18d124442 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:17.344996 17880 raft_consensus.cc:740] T 7cb3768d215e409fb9f5baf18d124442 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Initialized, Role: FOLLOWER
I20260709 12:20:17.345728 17880 consensus_queue.cc:260] T 7cb3768d215e409fb9f5baf18d124442 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:17.345467 17585 catalog_manager.cc:5697] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b reported cstate change: term changed from 0 to 1, leader changed from <none> to e063541542a94969898868bf6eb5ce1b (127.16.29.67). New cstate: current_term: 1 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: HEALTHY } } }
I20260709 12:20:17.346906 17881 raft_consensus.cc:359] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:17.347440 17881 raft_consensus.cc:385] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:17.347673 17881 raft_consensus.cc:740] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Initialized, Role: FOLLOWER
I20260709 12:20:17.347934 17880 ts_tablet_manager.cc:1434] T 7cb3768d215e409fb9f5baf18d124442 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent starting tablet: real 0.007s	user 0.008s	sys 0.000s
I20260709 12:20:17.348289 17881 consensus_queue.cc:260] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:17.349239 17880 tablet_bootstrap.cc:492] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e2909cff1da44ad387f96a8ae3aa5afb: Bootstrap starting.
I20260709 12:20:17.349923 17881 ts_tablet_manager.cc:1434] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b: Time spent starting tablet: real 0.005s	user 0.004s	sys 0.000s
I20260709 12:20:17.350705 17881 tablet_bootstrap.cc:492] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b: Bootstrap starting.
I20260709 12:20:17.355779 17880 tablet_bootstrap.cc:654] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e2909cff1da44ad387f96a8ae3aa5afb: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:17.364426 17878 tablet_bootstrap.cc:492] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3: No bootstrap required, opened a new log
I20260709 12:20:17.364287 17934 raft_consensus.cc:515] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:17.365052 17878 ts_tablet_manager.cc:1403] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3: Time spent bootstrapping tablet: real 0.093s	user 0.019s	sys 0.016s
I20260709 12:20:17.367079 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:20:17.367746 17878 raft_consensus.cc:359] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:20:17.368346 17652 leader_election.cc:343] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:17.368510 17878 raft_consensus.cc:385] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:17.368993 17880 tablet_bootstrap.cc:492] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e2909cff1da44ad387f96a8ae3aa5afb: No bootstrap required, opened a new log
I20260709 12:20:17.368888 17878 raft_consensus.cc:740] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Initialized, Role: FOLLOWER
I20260709 12:20:17.369746 17880 ts_tablet_manager.cc:1403] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent bootstrapping tablet: real 0.021s	user 0.016s	sys 0.002s
I20260709 12:20:17.370846 17878 consensus_queue.cc:260] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:17.373309 17881 tablet_bootstrap.cc:654] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:17.373886 17934 leader_election.cc:290] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:17.374917 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:20:17.378118 17837 raft_consensus.cc:3060] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:17.378976 17880 raft_consensus.cc:359] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:17.379722 17880 raft_consensus.cc:385] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:17.380065 17880 raft_consensus.cc:740] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Initialized, Role: FOLLOWER
I20260709 12:20:17.380800 17880 consensus_queue.cc:260] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:17.381615 17878 ts_tablet_manager.cc:1434] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3: Time spent starting tablet: real 0.016s	user 0.005s	sys 0.000s
I20260709 12:20:17.382494 17878 tablet_bootstrap.cc:492] T 4092aabb101d411798b0354f88a5dd2b P 0da169ff82674dc1947055ce43de68a3: Bootstrap starting.
I20260709 12:20:17.382942 17880 ts_tablet_manager.cc:1434] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent starting tablet: real 0.007s	user 0.004s	sys 0.000s
I20260709 12:20:17.384109 17880 tablet_bootstrap.cc:492] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb: Bootstrap starting.
I20260709 12:20:17.388286 17878 tablet_bootstrap.cc:654] T 4092aabb101d411798b0354f88a5dd2b P 0da169ff82674dc1947055ce43de68a3: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:17.388482 17880 tablet_bootstrap.cc:654] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:17.390867 17837 raft_consensus.cc:2468] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 1.
I20260709 12:20:17.392774 17652 leader_election.cc:304] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:20:17.396284 17934 raft_consensus.cc:2804] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:17.397436 17934 raft_consensus.cc:697] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 1 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:20:17.398638 17934 consensus_queue.cc:237] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:17.405709 17881 tablet_bootstrap.cc:492] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b: No bootstrap required, opened a new log
I20260709 12:20:17.406505 17881 ts_tablet_manager.cc:1403] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b: Time spent bootstrapping tablet: real 0.056s	user 0.012s	sys 0.000s
I20260709 12:20:17.414983 17880 tablet_bootstrap.cc:492] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb: No bootstrap required, opened a new log
I20260709 12:20:17.415486 17880 ts_tablet_manager.cc:1403] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent bootstrapping tablet: real 0.032s	user 0.018s	sys 0.011s
I20260709 12:20:17.411198 17881 raft_consensus.cc:359] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:17.417075 17881 raft_consensus.cc:385] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:17.417367 17881 raft_consensus.cc:740] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Initialized, Role: FOLLOWER
I20260709 12:20:17.417474 17878 tablet_bootstrap.cc:492] T 4092aabb101d411798b0354f88a5dd2b P 0da169ff82674dc1947055ce43de68a3: No bootstrap required, opened a new log
I20260709 12:20:17.418107 17878 ts_tablet_manager.cc:1403] T 4092aabb101d411798b0354f88a5dd2b P 0da169ff82674dc1947055ce43de68a3: Time spent bootstrapping tablet: real 0.036s	user 0.017s	sys 0.018s
I20260709 12:20:17.418327 17880 raft_consensus.cc:359] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:17.419410 17880 raft_consensus.cc:385] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:17.419768 17880 raft_consensus.cc:740] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Initialized, Role: FOLLOWER
I20260709 12:20:17.418071 17881 consensus_queue.cc:260] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:17.420493 17880 consensus_queue.cc:260] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:17.421844 17878 raft_consensus.cc:359] T 4092aabb101d411798b0354f88a5dd2b P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:17.422567 17881 ts_tablet_manager.cc:1434] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b: Time spent starting tablet: real 0.016s	user 0.007s	sys 0.000s
I20260709 12:20:17.422643 17878 raft_consensus.cc:385] T 4092aabb101d411798b0354f88a5dd2b P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:17.423209 17878 raft_consensus.cc:740] T 4092aabb101d411798b0354f88a5dd2b P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Initialized, Role: FOLLOWER
I20260709 12:20:17.423601 17880 ts_tablet_manager.cc:1434] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent starting tablet: real 0.008s	user 0.008s	sys 0.000s
I20260709 12:20:17.424192 17878 consensus_queue.cc:260] T 4092aabb101d411798b0354f88a5dd2b P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:17.424839 17880 tablet_bootstrap.cc:492] T 044ba5aee1d6499a975c6b29d932fd95 P e2909cff1da44ad387f96a8ae3aa5afb: Bootstrap starting.
I20260709 12:20:17.426780 17878 ts_tablet_manager.cc:1434] T 4092aabb101d411798b0354f88a5dd2b P 0da169ff82674dc1947055ce43de68a3: Time spent starting tablet: real 0.008s	user 0.004s	sys 0.002s
I20260709 12:20:17.427795 17878 tablet_bootstrap.cc:492] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3: Bootstrap starting.
I20260709 12:20:17.428109 17951 raft_consensus.cc:493] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:17.428619 17951 raft_consensus.cc:515] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:17.431382 17951 leader_election.cc:290] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:20:17.432946 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:20:17.433403 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:20:17.435439 17802 leader_election.cc:304] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:20:17.437002 17951 raft_consensus.cc:2749] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:17.437248 17878 tablet_bootstrap.cc:654] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:17.439585 17880 tablet_bootstrap.cc:654] T 044ba5aee1d6499a975c6b29d932fd95 P e2909cff1da44ad387f96a8ae3aa5afb: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:17.447820 17881 tablet_bootstrap.cc:492] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b: Bootstrap starting.
I20260709 12:20:17.451036 17584 catalog_manager.cc:5697] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0da169ff82674dc1947055ce43de68a3 (127.16.29.65). New cstate: current_term: 1 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:17.452610 17880 tablet_bootstrap.cc:492] T 044ba5aee1d6499a975c6b29d932fd95 P e2909cff1da44ad387f96a8ae3aa5afb: No bootstrap required, opened a new log
I20260709 12:20:17.453326 17880 ts_tablet_manager.cc:1403] T 044ba5aee1d6499a975c6b29d932fd95 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent bootstrapping tablet: real 0.029s	user 0.011s	sys 0.007s
I20260709 12:20:17.455814 17881 tablet_bootstrap.cc:654] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:17.456363 17878 tablet_bootstrap.cc:492] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3: No bootstrap required, opened a new log
I20260709 12:20:17.456177 17880 raft_consensus.cc:359] T 044ba5aee1d6499a975c6b29d932fd95 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:17.456836 17878 ts_tablet_manager.cc:1403] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3: Time spent bootstrapping tablet: real 0.029s	user 0.013s	sys 0.004s
I20260709 12:20:17.456959 17880 raft_consensus.cc:385] T 044ba5aee1d6499a975c6b29d932fd95 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:17.457285 17880 raft_consensus.cc:740] T 044ba5aee1d6499a975c6b29d932fd95 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Initialized, Role: FOLLOWER
I20260709 12:20:17.458046 17880 consensus_queue.cc:260] T 044ba5aee1d6499a975c6b29d932fd95 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:17.460386 17880 ts_tablet_manager.cc:1434] T 044ba5aee1d6499a975c6b29d932fd95 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent starting tablet: real 0.007s	user 0.006s	sys 0.001s
I20260709 12:20:17.459824 17878 raft_consensus.cc:359] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:17.460875 17878 raft_consensus.cc:385] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:17.461329 17878 raft_consensus.cc:740] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Initialized, Role: FOLLOWER
I20260709 12:20:17.462121 17880 tablet_bootstrap.cc:492] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb: Bootstrap starting.
I20260709 12:20:17.468174 17880 tablet_bootstrap.cc:654] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:17.462148 17878 consensus_queue.cc:260] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:17.471659 17878 ts_tablet_manager.cc:1434] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3: Time spent starting tablet: real 0.014s	user 0.007s	sys 0.001s
I20260709 12:20:17.472705 17878 tablet_bootstrap.cc:492] T 0f940f29438f475294357324452abe76 P 0da169ff82674dc1947055ce43de68a3: Bootstrap starting.
I20260709 12:20:17.479821 17881 tablet_bootstrap.cc:492] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b: No bootstrap required, opened a new log
I20260709 12:20:17.480517 17881 ts_tablet_manager.cc:1403] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b: Time spent bootstrapping tablet: real 0.033s	user 0.017s	sys 0.011s
I20260709 12:20:17.484911 17951 raft_consensus.cc:493] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:17.485405 17951 raft_consensus.cc:515] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:17.487491 17878 tablet_bootstrap.cc:654] T 0f940f29438f475294357324452abe76 P 0da169ff82674dc1947055ce43de68a3: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:17.487576 17951 leader_election.cc:290] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:20:17.488694 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:20:17.489007 17635 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:20:17.489352 17762 raft_consensus.cc:2468] T 7cb3768d215e409fb9f5baf18d124442 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 0.
I20260709 12:20:17.490221 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:20:17.490597 17801 leader_election.cc:304] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:20:17.490849 17688 raft_consensus.cc:2468] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 0.
I20260709 12:20:17.491416 17951 raft_consensus.cc:2804] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:17.491834 17951 raft_consensus.cc:493] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:17.492233 17951 raft_consensus.cc:3060] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:17.491462 17881 raft_consensus.cc:359] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:17.492671 17881 raft_consensus.cc:385] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:17.493076 17881 raft_consensus.cc:740] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Initialized, Role: FOLLOWER
I20260709 12:20:17.493913 17881 consensus_queue.cc:260] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:17.496426 17881 ts_tablet_manager.cc:1434] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b: Time spent starting tablet: real 0.016s	user 0.008s	sys 0.001s
I20260709 12:20:17.496443 17878 tablet_bootstrap.cc:492] T 0f940f29438f475294357324452abe76 P 0da169ff82674dc1947055ce43de68a3: No bootstrap required, opened a new log
I20260709 12:20:17.497273 17878 ts_tablet_manager.cc:1403] T 0f940f29438f475294357324452abe76 P 0da169ff82674dc1947055ce43de68a3: Time spent bootstrapping tablet: real 0.025s	user 0.009s	sys 0.003s
I20260709 12:20:17.497882 17880 tablet_bootstrap.cc:492] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb: No bootstrap required, opened a new log
I20260709 12:20:17.498322 17880 ts_tablet_manager.cc:1403] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent bootstrapping tablet: real 0.036s	user 0.012s	sys 0.012s
I20260709 12:20:17.499763 17878 raft_consensus.cc:359] T 0f940f29438f475294357324452abe76 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:17.500443 17878 raft_consensus.cc:385] T 0f940f29438f475294357324452abe76 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:17.500131 17951 raft_consensus.cc:515] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:17.500722 17878 raft_consensus.cc:740] T 0f940f29438f475294357324452abe76 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Initialized, Role: FOLLOWER
I20260709 12:20:17.501049 17880 raft_consensus.cc:359] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:17.501730 17880 raft_consensus.cc:385] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:17.501976 17878 consensus_queue.cc:260] T 0f940f29438f475294357324452abe76 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:17.503021 17881 tablet_bootstrap.cc:492] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b: Bootstrap starting.
I20260709 12:20:17.503073 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:20:17.503785 17688 raft_consensus.cc:3060] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:17.504068 17878 ts_tablet_manager.cc:1434] T 0f940f29438f475294357324452abe76 P 0da169ff82674dc1947055ce43de68a3: Time spent starting tablet: real 0.007s	user 0.006s	sys 0.000s
I20260709 12:20:17.501972 17880 raft_consensus.cc:740] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Initialized, Role: FOLLOWER
I20260709 12:20:17.505200 17880 consensus_queue.cc:260] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:17.507091 17880 ts_tablet_manager.cc:1434] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent starting tablet: real 0.008s	user 0.005s	sys 0.000s
I20260709 12:20:17.510113 17881 tablet_bootstrap.cc:654] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:17.513315 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:20:17.514197 17762 raft_consensus.cc:3060] T 7cb3768d215e409fb9f5baf18d124442 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:17.515648 17934 raft_consensus.cc:493] T 0f940f29438f475294357324452abe76 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:17.515682 17688 raft_consensus.cc:2468] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 1.
I20260709 12:20:17.516284 17934 raft_consensus.cc:515] T 0f940f29438f475294357324452abe76 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:17.517577 17802 leader_election.cc:304] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:20:17.518558 17934 leader_election.cc:290] T 0f940f29438f475294357324452abe76 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:17.519515 17881 tablet_bootstrap.cc:492] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b: No bootstrap required, opened a new log
I20260709 12:20:17.519976 17881 ts_tablet_manager.cc:1403] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b: Time spent bootstrapping tablet: real 0.017s	user 0.012s	sys 0.003s
I20260709 12:20:17.520607 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:20:17.522167 17652 leader_election.cc:343] T 0f940f29438f475294357324452abe76 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:17.523517 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:20:17.524174 17951 leader_election.cc:290] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 1 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:20:17.525314 17952 raft_consensus.cc:2804] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Leader election won for term 1
W20260709 12:20:17.526312 17652 leader_election.cc:343] T 0f940f29438f475294357324452abe76 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:17.526456 17952 raft_consensus.cc:697] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 1 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:20:17.526814 17652 leader_election.cc:304] T 0f940f29438f475294357324452abe76 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:20:17.527423 17952 consensus_queue.cc:237] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:17.528051 17934 raft_consensus.cc:2749] T 0f940f29438f475294357324452abe76 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:17.529510 17878 tablet_bootstrap.cc:492] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3: Bootstrap starting.
I20260709 12:20:17.529228 17881 raft_consensus.cc:359] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:17.529881 17881 raft_consensus.cc:385] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:17.530135 17881 raft_consensus.cc:740] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Initialized, Role: FOLLOWER
I20260709 12:20:17.530777 17881 consensus_queue.cc:260] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:17.532733 17881 ts_tablet_manager.cc:1434] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b: Time spent starting tablet: real 0.012s	user 0.006s	sys 0.000s
I20260709 12:20:17.533603 17881 tablet_bootstrap.cc:492] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b: Bootstrap starting.
I20260709 12:20:17.533989 17880 tablet_bootstrap.cc:492] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb: Bootstrap starting.
I20260709 12:20:17.525549 17762 raft_consensus.cc:2468] T 7cb3768d215e409fb9f5baf18d124442 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 1.
I20260709 12:20:17.541149 17880 tablet_bootstrap.cc:654] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:17.545897 17688 raft_consensus.cc:3060] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:17.547572 17976 raft_consensus.cc:493] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:17.548074 17976 raft_consensus.cc:515] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:17.550132 17878 tablet_bootstrap.cc:654] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:17.551146 17976 leader_election.cc:290] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:17.551365 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:20:17.550637 17584 catalog_manager.cc:5697] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b reported cstate change: term changed from 0 to 1, leader changed from <none> to e063541542a94969898868bf6eb5ce1b (127.16.29.67). New cstate: current_term: 1 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: HEALTHY } } }
I20260709 12:20:17.552469 17837 raft_consensus.cc:2393] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 1: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:20:17.555229 17881 tablet_bootstrap.cc:654] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:17.558511 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:20:17.560644 17728 leader_election.cc:304] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:20:17.561533 17976 raft_consensus.cc:3060] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:17.569487 17976 raft_consensus.cc:2749] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:17.575299 17878 tablet_bootstrap.cc:492] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3: No bootstrap required, opened a new log
I20260709 12:20:17.576195 17878 ts_tablet_manager.cc:1403] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3: Time spent bootstrapping tablet: real 0.047s	user 0.015s	sys 0.001s
I20260709 12:20:17.580004 17878 raft_consensus.cc:359] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:17.580901 17878 raft_consensus.cc:385] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:17.581180 17878 raft_consensus.cc:740] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Initialized, Role: FOLLOWER
I20260709 12:20:17.582046 17881 tablet_bootstrap.cc:492] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b: No bootstrap required, opened a new log
I20260709 12:20:17.582171 17880 tablet_bootstrap.cc:492] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb: No bootstrap required, opened a new log
I20260709 12:20:17.581974 17878 consensus_queue.cc:260] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:17.582674 17881 ts_tablet_manager.cc:1403] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b: Time spent bootstrapping tablet: real 0.049s	user 0.004s	sys 0.006s
I20260709 12:20:17.582759 17880 ts_tablet_manager.cc:1403] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb: Time spent bootstrapping tablet: real 0.049s	user 0.007s	sys 0.012s
I20260709 12:20:17.584465 17878 ts_tablet_manager.cc:1434] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3: Time spent starting tablet: real 0.008s	user 0.007s	sys 0.001s
I20260709 12:20:17.585420 17878 tablet_bootstrap.cc:492] T 28062a50142448aa97994888b4765920 P 0da169ff82674dc1947055ce43de68a3: Bootstrap starting.
I20260709 12:20:17.585626 17881 raft_consensus.cc:359] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:17.586289 17881 raft_consensus.cc:385] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:17.586093 17880 raft_consensus.cc:359] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:17.586589 17881 raft_consensus.cc:740] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Initialized, Role: FOLLOWER
I20260709 12:20:17.586753 17880 raft_consensus.cc:385] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:17.587280 17880 raft_consensus.cc:740] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Initialized, Role: FOLLOWER
I20260709 12:20:17.587368 17881 consensus_queue.cc:260] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:17.587998 17880 consensus_queue.cc:260] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:17.590032 17880 ts_tablet_manager.cc:1434] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb: Time spent starting tablet: real 0.007s	user 0.007s	sys 0.000s
I20260709 12:20:17.589906 17934 raft_consensus.cc:493] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:20:17.590561 17934 raft_consensus.cc:515] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:17.593655 17878 tablet_bootstrap.cc:654] T 28062a50142448aa97994888b4765920 P 0da169ff82674dc1947055ce43de68a3: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:17.593657 17880 tablet_bootstrap.cc:492] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb: Bootstrap starting.
I20260709 12:20:17.596025 17881 ts_tablet_manager.cc:1434] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b: Time spent starting tablet: real 0.013s	user 0.009s	sys 0.000s
I20260709 12:20:17.597009 17881 tablet_bootstrap.cc:492] T 84421f094faa40f39b30b4c3d0737639 P e063541542a94969898868bf6eb5ce1b: Bootstrap starting.
I20260709 12:20:17.601275 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:20:17.601920 17880 tablet_bootstrap.cc:654] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:17.603173 17881 tablet_bootstrap.cc:654] T 84421f094faa40f39b30b4c3d0737639 P e063541542a94969898868bf6eb5ce1b: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:17.615988 17881 tablet_bootstrap.cc:492] T 84421f094faa40f39b30b4c3d0737639 P e063541542a94969898868bf6eb5ce1b: No bootstrap required, opened a new log
I20260709 12:20:17.617448 17934 leader_election.cc:290] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:17.621546 17878 tablet_bootstrap.cc:492] T 28062a50142448aa97994888b4765920 P 0da169ff82674dc1947055ce43de68a3: No bootstrap required, opened a new log
I20260709 12:20:17.622072 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:20:17.624146 17652 leader_election.cc:304] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:20:17.625360 17934 raft_consensus.cc:2749] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:20:17.622072 17878 ts_tablet_manager.cc:1403] T 28062a50142448aa97994888b4765920 P 0da169ff82674dc1947055ce43de68a3: Time spent bootstrapping tablet: real 0.037s	user 0.021s	sys 0.011s
I20260709 12:20:17.629020 17878 raft_consensus.cc:359] T 28062a50142448aa97994888b4765920 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:17.629784 17878 raft_consensus.cc:385] T 28062a50142448aa97994888b4765920 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:17.630115 17878 raft_consensus.cc:740] T 28062a50142448aa97994888b4765920 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Initialized, Role: FOLLOWER
I20260709 12:20:17.630582 17881 ts_tablet_manager.cc:1403] T 84421f094faa40f39b30b4c3d0737639 P e063541542a94969898868bf6eb5ce1b: Time spent bootstrapping tablet: real 0.034s	user 0.015s	sys 0.003s
I20260709 12:20:17.630930 17878 consensus_queue.cc:260] T 28062a50142448aa97994888b4765920 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:17.634860 17878 ts_tablet_manager.cc:1434] T 28062a50142448aa97994888b4765920 P 0da169ff82674dc1947055ce43de68a3: Time spent starting tablet: real 0.008s	user 0.003s	sys 0.004s
I20260709 12:20:17.635130 17881 raft_consensus.cc:359] T 84421f094faa40f39b30b4c3d0737639 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:17.635831 17881 raft_consensus.cc:385] T 84421f094faa40f39b30b4c3d0737639 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:17.635831 17878 tablet_bootstrap.cc:492] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3: Bootstrap starting.
I20260709 12:20:17.636296 17881 raft_consensus.cc:740] T 84421f094faa40f39b30b4c3d0737639 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Initialized, Role: FOLLOWER
I20260709 12:20:17.637012 17881 consensus_queue.cc:260] T 84421f094faa40f39b30b4c3d0737639 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:17.638932 17881 ts_tablet_manager.cc:1434] T 84421f094faa40f39b30b4c3d0737639 P e063541542a94969898868bf6eb5ce1b: Time spent starting tablet: real 0.008s	user 0.001s	sys 0.004s
I20260709 12:20:17.642325 17880 tablet_bootstrap.cc:492] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb: No bootstrap required, opened a new log
I20260709 12:20:17.642838 17878 tablet_bootstrap.cc:654] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:17.642902 17880 ts_tablet_manager.cc:1403] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent bootstrapping tablet: real 0.049s	user 0.016s	sys 0.016s
I20260709 12:20:17.643966 17952 raft_consensus.cc:493] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:17.644423 17952 raft_consensus.cc:515] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:17.646473 17952 leader_election.cc:290] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:20:17.646173 17880 raft_consensus.cc:359] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:17.647060 17880 raft_consensus.cc:385] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:17.647426 17880 raft_consensus.cc:740] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Initialized, Role: FOLLOWER
I20260709 12:20:17.647643 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:20:17.647986 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:20:17.648346 17880 consensus_queue.cc:260] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:17.648613 17762 raft_consensus.cc:2468] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 0.
I20260709 12:20:17.648334 17688 raft_consensus.cc:2468] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 0.
I20260709 12:20:17.651062 17802 leader_election.cc:304] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:20:17.651412 17880 ts_tablet_manager.cc:1434] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent starting tablet: real 0.008s	user 0.006s	sys 0.000s
I20260709 12:20:17.652158 17952 raft_consensus.cc:2804] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:17.652431 17952 raft_consensus.cc:493] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:17.652705 17952 raft_consensus.cc:3060] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:17.653549 17880 tablet_bootstrap.cc:492] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb: Bootstrap starting.
I20260709 12:20:17.659189 17881 tablet_bootstrap.cc:492] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b: Bootstrap starting.
I20260709 12:20:17.660432 17880 tablet_bootstrap.cc:654] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:17.680300 17952 raft_consensus.cc:515] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:17.683131 17878 tablet_bootstrap.cc:492] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3: No bootstrap required, opened a new log
I20260709 12:20:17.683655 17878 ts_tablet_manager.cc:1403] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3: Time spent bootstrapping tablet: real 0.048s	user 0.027s	sys 0.014s
I20260709 12:20:17.684397 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:20:17.684993 17688 raft_consensus.cc:3060] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:17.686573 17880 tablet_bootstrap.cc:492] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb: No bootstrap required, opened a new log
I20260709 12:20:17.686754 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:20:17.687080 17880 ts_tablet_manager.cc:1403] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb: Time spent bootstrapping tablet: real 0.034s	user 0.025s	sys 0.006s
I20260709 12:20:17.687333 17762 raft_consensus.cc:3060] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:17.683732 17881 tablet_bootstrap.cc:654] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:17.687265 17878 raft_consensus.cc:359] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:17.688724 17878 raft_consensus.cc:385] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:17.688688 17952 leader_election.cc:290] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 1 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:20:17.689127 17878 raft_consensus.cc:740] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Initialized, Role: FOLLOWER
I20260709 12:20:17.689800 17878 consensus_queue.cc:260] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:17.690204 17880 raft_consensus.cc:359] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:17.690958 17880 raft_consensus.cc:385] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:17.691279 17880 raft_consensus.cc:740] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Initialized, Role: FOLLOWER
I20260709 12:20:17.692099 17878 ts_tablet_manager.cc:1434] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3: Time spent starting tablet: real 0.008s	user 0.006s	sys 0.000s
I20260709 12:20:17.692090 17880 consensus_queue.cc:260] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:17.692984 17688 raft_consensus.cc:2468] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 1.
I20260709 12:20:17.693334 17878 tablet_bootstrap.cc:492] T 11896555f5d7458dbb02a3203b27f1cf P 0da169ff82674dc1947055ce43de68a3: Bootstrap starting.
I20260709 12:20:17.694289 17802 leader_election.cc:304] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:20:17.696144 17950 raft_consensus.cc:2804] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:17.696601 17950 raft_consensus.cc:697] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 1 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:20:17.697376 17950 consensus_queue.cc:237] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:17.701300 17878 tablet_bootstrap.cc:654] T 11896555f5d7458dbb02a3203b27f1cf P 0da169ff82674dc1947055ce43de68a3: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:17.705227 17880 ts_tablet_manager.cc:1434] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb: Time spent starting tablet: real 0.018s	user 0.003s	sys 0.003s
I20260709 12:20:17.710932 17762 raft_consensus.cc:2468] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 1.
I20260709 12:20:17.714193 17880 tablet_bootstrap.cc:492] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb: Bootstrap starting.
I20260709 12:20:17.714582 17952 raft_consensus.cc:493] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:17.715025 17952 raft_consensus.cc:515] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:17.720973 17880 tablet_bootstrap.cc:654] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:17.724749 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:20:17.725694 17688 raft_consensus.cc:2468] T 4092aabb101d411798b0354f88a5dd2b P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 0.
I20260709 12:20:17.727308 17802 leader_election.cc:304] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:20:17.728508 17951 raft_consensus.cc:2804] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:17.728965 17951 raft_consensus.cc:493] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:17.729385 17951 raft_consensus.cc:3060] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:17.731360 17934 raft_consensus.cc:493] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:17.732046 17934 raft_consensus.cc:515] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:17.734730 17934 leader_election.cc:290] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:17.736198 17949 consensus_queue.cc:1048] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:20:17.737252 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:20:17.737578 17951 raft_consensus.cc:515] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:17.739657 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:20:17.739723 17652 leader_election.cc:343] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:20:17.740965 17652 leader_election.cc:343] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:17.741495 17880 tablet_bootstrap.cc:492] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb: No bootstrap required, opened a new log
I20260709 12:20:17.741358 17652 leader_election.cc:304] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:20:17.741926 17880 ts_tablet_manager.cc:1403] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent bootstrapping tablet: real 0.028s	user 0.011s	sys 0.006s
I20260709 12:20:17.742616 17934 raft_consensus.cc:2749] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:17.744752 17880 raft_consensus.cc:359] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:17.745910 17880 raft_consensus.cc:385] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:17.746201 17880 raft_consensus.cc:740] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Initialized, Role: FOLLOWER
I20260709 12:20:17.747498 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:20:17.747725 17763 raft_consensus.cc:3060] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:17.751370 17952 leader_election.cc:290] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:20:17.751499 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:20:17.753479 17687 raft_consensus.cc:3060] T 4092aabb101d411798b0354f88a5dd2b P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:17.756747 17951 leader_election.cc:290] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 1 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:20:17.757985 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:20:17.749408 17976 raft_consensus.cc:493] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:17.757794 17762 raft_consensus.cc:2468] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 0.
I20260709 12:20:17.759402 17976 raft_consensus.cc:515] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:17.759171 17586 catalog_manager.cc:5697] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b reported cstate change: term changed from 0 to 1, leader changed from <none> to e063541542a94969898868bf6eb5ce1b (127.16.29.67). New cstate: current_term: 1 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: HEALTHY } } }
I20260709 12:20:17.761616 17976 leader_election.cc:290] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:17.762470 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:20:17.763171 17688 raft_consensus.cc:2468] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 0.
I20260709 12:20:17.764534 17728 leader_election.cc:304] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:20:17.769656 17934 consensus_queue.cc:1048] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:20:17.770884 17951 consensus_queue.cc:1048] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:20:17.748378 17880 consensus_queue.cc:260] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:17.772183 17975 raft_consensus.cc:493] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:17.772563 17881 tablet_bootstrap.cc:492] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b: No bootstrap required, opened a new log
I20260709 12:20:17.773031 17881 ts_tablet_manager.cc:1403] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b: Time spent bootstrapping tablet: real 0.114s	user 0.031s	sys 0.041s
I20260709 12:20:17.772792 17975 raft_consensus.cc:515] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:17.773077 17935 raft_consensus.cc:2804] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:17.773584 17935 raft_consensus.cc:493] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:17.773797 17880 ts_tablet_manager.cc:1434] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent starting tablet: real 0.032s	user 0.006s	sys 0.001s
I20260709 12:20:17.773924 17935 raft_consensus.cc:3060] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:17.775010 17880 tablet_bootstrap.cc:492] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb: Bootstrap starting.
I20260709 12:20:17.777895 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:20:17.778648 17837 raft_consensus.cc:2468] T 84421f094faa40f39b30b4c3d0737639 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 0.
I20260709 12:20:17.788712 17878 tablet_bootstrap.cc:492] T 11896555f5d7458dbb02a3203b27f1cf P 0da169ff82674dc1947055ce43de68a3: No bootstrap required, opened a new log
I20260709 12:20:17.789179 17878 ts_tablet_manager.cc:1403] T 11896555f5d7458dbb02a3203b27f1cf P 0da169ff82674dc1947055ce43de68a3: Time spent bootstrapping tablet: real 0.096s	user 0.028s	sys 0.013s
I20260709 12:20:17.790831 17881 raft_consensus.cc:359] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:17.791491 17881 raft_consensus.cc:385] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:17.791781 17881 raft_consensus.cc:740] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Initialized, Role: FOLLOWER
I20260709 12:20:17.792716 17881 consensus_queue.cc:260] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:17.791514 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:20:17.795149 17881 ts_tablet_manager.cc:1434] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b: Time spent starting tablet: real 0.022s	user 0.004s	sys 0.001s
I20260709 12:20:17.796512 17881 tablet_bootstrap.cc:492] T 44ad5363c5954da4a1bff5857780fd5c P e063541542a94969898868bf6eb5ce1b: Bootstrap starting.
I20260709 12:20:17.787083 17687 raft_consensus.cc:2468] T 4092aabb101d411798b0354f88a5dd2b P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 1.
I20260709 12:20:17.806924 17802 leader_election.cc:304] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:20:17.808066 17952 raft_consensus.cc:2804] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:17.808346 17992 raft_consensus.cc:493] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:17.808846 17952 raft_consensus.cc:697] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 1 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:20:17.810281 17950 raft_consensus.cc:493] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:17.810959 17950 raft_consensus.cc:515] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:17.812597 17975 leader_election.cc:290] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:17.814405 17878 raft_consensus.cc:359] T 11896555f5d7458dbb02a3203b27f1cf P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:17.815088 17878 raft_consensus.cc:385] T 11896555f5d7458dbb02a3203b27f1cf P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:17.815502 17878 raft_consensus.cc:740] T 11896555f5d7458dbb02a3203b27f1cf P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Initialized, Role: FOLLOWER
I20260709 12:20:17.808995 17992 raft_consensus.cc:515] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:17.817114 17878 consensus_queue.cc:260] T 11896555f5d7458dbb02a3203b27f1cf P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:17.818627 17878 ts_tablet_manager.cc:1434] T 11896555f5d7458dbb02a3203b27f1cf P 0da169ff82674dc1947055ce43de68a3: Time spent starting tablet: real 0.029s	user 0.005s	sys 0.000s
I20260709 12:20:17.819542 17878 tablet_bootstrap.cc:492] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3: Bootstrap starting.
I20260709 12:20:17.819837 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:20:17.820469 17687 raft_consensus.cc:2468] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 0.
I20260709 12:20:17.820427 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:20:17.821161 17935 raft_consensus.cc:515] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:17.821542 17688 raft_consensus.cc:2468] T 0f940f29438f475294357324452abe76 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 0.
I20260709 12:20:17.821842 17728 leader_election.cc:304] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:20:17.822734 17975 raft_consensus.cc:2804] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:17.823339 17935 leader_election.cc:290] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 1 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:17.825860 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:20:17.824326 17975 raft_consensus.cc:493] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:17.831156 17976 raft_consensus.cc:493] T 044ba5aee1d6499a975c6b29d932fd95 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:17.831487 17975 raft_consensus.cc:3060] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:17.826823 17952 consensus_queue.cc:237] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:17.840520 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:20:17.839592 17976 raft_consensus.cc:515] T 044ba5aee1d6499a975c6b29d932fd95 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:17.840309 17950 leader_election.cc:290] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:20:17.840483 17881 tablet_bootstrap.cc:654] T 44ad5363c5954da4a1bff5857780fd5c P e063541542a94969898868bf6eb5ce1b: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:17.830947 17688 raft_consensus.cc:3060] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:17.827665 17992 leader_election.cc:290] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:17.844132 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:20:17.846140 17802 leader_election.cc:304] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
W20260709 12:20:17.848733 17801 leader_election.cc:343] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:17.848747 17950 raft_consensus.cc:2804] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:17.854686 17763 raft_consensus.cc:3060] T 044ba5aee1d6499a975c6b29d932fd95 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:17.849898 17950 raft_consensus.cc:493] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:17.859184 17940 consensus_queue.cc:1048] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:20:17.852231 17976 leader_election.cc:290] T 044ba5aee1d6499a975c6b29d932fd95 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:17.856932 17950 raft_consensus.cc:3060] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:17.874482 17878 tablet_bootstrap.cc:654] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:17.884809 17880 tablet_bootstrap.cc:654] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:17.895920 17950 raft_consensus.cc:515] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:17.921651 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:20:17.982340 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 61650.7us
I20260709 12:20:17.991263 17951 raft_consensus.cc:493] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:18.008960 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:20:17.998186 17940 consensus_queue.cc:1048] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.017s
I20260709 12:20:18.015554 17687 raft_consensus.cc:3060] T 0f940f29438f475294357324452abe76 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:18.001358 17975 raft_consensus.cc:515] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:18.024155 18000 raft_consensus.cc:493] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:18.024279 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:20:18.024838 17838 raft_consensus.cc:3060] T 84421f094faa40f39b30b4c3d0737639 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:18.025038 17950 leader_election.cc:290] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 1 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:20:18.024787 18000 raft_consensus.cc:515] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:18.026224 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
W20260709 12:20:18.027715 17801 leader_election.cc:343] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:18.022876 17837 raft_consensus.cc:3060] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:18.013049 17951 raft_consensus.cc:515] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:18.033352 17727 leader_election.cc:304] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:20:18.021613 18002 raft_consensus.cc:493] T 11896555f5d7458dbb02a3203b27f1cf P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:18.037177 17995 raft_consensus.cc:2749] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:20:18.037833 17995 raft_consensus.cc:493] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:18.038244 17995 raft_consensus.cc:515] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:18.038764 17881 tablet_bootstrap.cc:492] T 44ad5363c5954da4a1bff5857780fd5c P e063541542a94969898868bf6eb5ce1b: No bootstrap required, opened a new log
I20260709 12:20:18.039215 17881 ts_tablet_manager.cc:1403] T 44ad5363c5954da4a1bff5857780fd5c P e063541542a94969898868bf6eb5ce1b: Time spent bootstrapping tablet: real 0.243s	user 0.035s	sys 0.062s
I20260709 12:20:18.019796 18003 consensus_queue.cc:1048] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.027s
I20260709 12:20:18.048394 17995 leader_election.cc:290] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:18.053094 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:20:18.053778 17686 raft_consensus.cc:3060] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:18.054629 17951 leader_election.cc:290] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:20:18.055645 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:20:18.060460 17762 raft_consensus.cc:2393] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 1: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:20:18.056399 17688 raft_consensus.cc:2468] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 1.
I20260709 12:20:18.070050 17881 raft_consensus.cc:359] T 44ad5363c5954da4a1bff5857780fd5c P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:18.070760 17881 raft_consensus.cc:385] T 44ad5363c5954da4a1bff5857780fd5c P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:18.071065 17881 raft_consensus.cc:740] T 44ad5363c5954da4a1bff5857780fd5c P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Initialized, Role: FOLLOWER
I20260709 12:20:18.071748 17881 consensus_queue.cc:260] T 44ad5363c5954da4a1bff5857780fd5c P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:18.071911 17950 consensus_queue.cc:1048] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:18.073166 17878 tablet_bootstrap.cc:492] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3: No bootstrap required, opened a new log
I20260709 12:20:18.073575 17878 ts_tablet_manager.cc:1403] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3: Time spent bootstrapping tablet: real 0.254s	user 0.019s	sys 0.009s
I20260709 12:20:18.037174 18002 raft_consensus.cc:515] T 11896555f5d7458dbb02a3203b27f1cf P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:18.076951 17838 raft_consensus.cc:2468] T 84421f094faa40f39b30b4c3d0737639 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 1.
I20260709 12:20:18.078446 17727 leader_election.cc:304] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:20:18.079247 17995 raft_consensus.cc:2804] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:18.079706 17995 raft_consensus.cc:697] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:20:18.080546 17995 consensus_queue.cc:237] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:18.086620 18000 leader_election.cc:290] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:18.086813 18002 leader_election.cc:290] T 11896555f5d7458dbb02a3203b27f1cf P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:18.088797 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:20:18.089722 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:20:18.090271 17835 raft_consensus.cc:2468] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 0.
I20260709 12:20:18.090541 17685 raft_consensus.cc:2468] T 28062a50142448aa97994888b4765920 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 0.
I20260709 12:20:18.092653 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:20:18.100073 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:20:18.102571 17881 ts_tablet_manager.cc:1434] T 44ad5363c5954da4a1bff5857780fd5c P e063541542a94969898868bf6eb5ce1b: Time spent starting tablet: real 0.063s	user 0.016s	sys 0.000s
I20260709 12:20:18.102629 17880 tablet_bootstrap.cc:492] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb: No bootstrap required, opened a new log
I20260709 12:20:18.103432 17880 ts_tablet_manager.cc:1403] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent bootstrapping tablet: real 0.329s	user 0.015s	sys 0.006s
I20260709 12:20:18.107059 17880 raft_consensus.cc:359] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:18.108675 17880 raft_consensus.cc:385] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:18.111313 17880 raft_consensus.cc:740] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Initialized, Role: FOLLOWER
I20260709 12:20:18.097852 17687 raft_consensus.cc:2468] T 0f940f29438f475294357324452abe76 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 1.
I20260709 12:20:18.112323 17880 consensus_queue.cc:260] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:18.111143 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:20:18.117022 17880 ts_tablet_manager.cc:1434] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent starting tablet: real 0.013s	user 0.010s	sys 0.000s
I20260709 12:20:18.118175 17880 tablet_bootstrap.cc:492] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb: Bootstrap starting.
I20260709 12:20:18.120981 17878 raft_consensus.cc:359] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:18.121464 17878 raft_consensus.cc:385] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:18.121800 17878 raft_consensus.cc:740] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Initialized, Role: FOLLOWER
I20260709 12:20:18.123409 17878 consensus_queue.cc:260] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:18.116719 17686 raft_consensus.cc:2468] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 1.
I20260709 12:20:18.120358 17762 raft_consensus.cc:2468] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 0.
I20260709 12:20:18.124677 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:20:18.125299 17878 ts_tablet_manager.cc:1434] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3: Time spent starting tablet: real 0.052s	user 0.005s	sys 0.002s
I20260709 12:20:18.124908 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:20:18.124531 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:20:18.133072 17652 leader_election.cc:343] T 11896555f5d7458dbb02a3203b27f1cf P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:18.136515 17652 leader_election.cc:304] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:20:18.126914 17727 leader_election.cc:343] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:18.140319 17934 raft_consensus.cc:2804] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:18.146452 17802 leader_election.cc:304] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:20:18.146997 17934 raft_consensus.cc:493] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:18.147589 17934 raft_consensus.cc:3060] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:18.147835 17802 leader_election.cc:304] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:20:18.148795 17952 consensus_queue.cc:1048] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:20:18.150367 17952 raft_consensus.cc:3060] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:18.144343 17727 leader_election.cc:304] T 044ba5aee1d6499a975c6b29d932fd95 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
W20260709 12:20:18.142876 17652 leader_election.cc:343] T 11896555f5d7458dbb02a3203b27f1cf P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:18.152315 17652 leader_election.cc:304] T 11896555f5d7458dbb02a3203b27f1cf P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:20:18.155244 18008 raft_consensus.cc:2749] T 11896555f5d7458dbb02a3203b27f1cf P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:18.155324 17951 raft_consensus.cc:2804] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:18.156132 17951 raft_consensus.cc:697] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 1 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:20:18.161336 17951 consensus_queue.cc:237] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:18.155346 17728 leader_election.cc:304] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:20:18.165920 17728 leader_election.cc:304] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:20:18.166898 17878 tablet_bootstrap.cc:492] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3: Bootstrap starting.
I20260709 12:20:18.167666 17992 raft_consensus.cc:2804] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:18.168105 17992 raft_consensus.cc:493] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:18.168512 17992 raft_consensus.cc:3060] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:18.168915 17935 raft_consensus.cc:2804] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:18.172433 17995 raft_consensus.cc:2749] T 044ba5aee1d6499a975c6b29d932fd95 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:18.180651 17935 raft_consensus.cc:697] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:20:18.190434 17935 consensus_queue.cc:237] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:18.206252 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:20:18.206995 17835 raft_consensus.cc:2468] T 44ad5363c5954da4a1bff5857780fd5c P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 0.
I20260709 12:20:18.208223 17881 tablet_bootstrap.cc:492] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b: Bootstrap starting.
I20260709 12:20:18.210940 17975 leader_election.cc:290] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 1 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:18.202311 17952 raft_consensus.cc:2749] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:18.220921 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:20:18.237061 17881 tablet_bootstrap.cc:654] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:18.207270 17934 raft_consensus.cc:515] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:18.245739 17878 tablet_bootstrap.cc:654] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:18.248971 17880 tablet_bootstrap.cc:654] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:18.257550 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:20:18.253129 17934 leader_election.cc:290] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:18.258212 17835 raft_consensus.cc:3060] T 44ad5363c5954da4a1bff5857780fd5c P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:18.260960 17585 catalog_manager.cc:5697] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b reported cstate change: term changed from 0 to 1, leader changed from <none> to e063541542a94969898868bf6eb5ce1b (127.16.29.67). New cstate: current_term: 1 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: HEALTHY } } }
I20260709 12:20:18.249822 17950 consensus_queue.cc:1048] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.017s
I20260709 12:20:18.256758 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:20:18.264358 17992 raft_consensus.cc:515] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:18.281155 17992 leader_election.cc:290] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 1 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:18.282364 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:20:18.319525 17835 raft_consensus.cc:2468] T 44ad5363c5954da4a1bff5857780fd5c P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 1.
I20260709 12:20:18.320015 17834 raft_consensus.cc:2468] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 1.
I20260709 12:20:18.384656 17584 catalog_manager.cc:5697] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb reported cstate change: term changed from 0 to 1, leader changed from <none> to e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66). New cstate: current_term: 1 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:18.391592 17881 tablet_bootstrap.cc:492] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b: No bootstrap required, opened a new log
I20260709 12:20:18.278043 17763 raft_consensus.cc:3060] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:18.394361 17878 tablet_bootstrap.cc:492] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3: No bootstrap required, opened a new log
I20260709 12:20:18.395547 17762 raft_consensus.cc:3060] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:18.400590 17881 ts_tablet_manager.cc:1403] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b: Time spent bootstrapping tablet: real 0.193s	user 0.029s	sys 0.050s
I20260709 12:20:18.400849 17878 ts_tablet_manager.cc:1403] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3: Time spent bootstrapping tablet: real 0.234s	user 0.021s	sys 0.014s
I20260709 12:20:18.400943 17950 consensus_queue.cc:1048] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:20:18.281925 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:20:18.408121 17684 raft_consensus.cc:3060] T 28062a50142448aa97994888b4765920 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:18.409089 17881 raft_consensus.cc:359] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:18.410338 17881 raft_consensus.cc:385] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:18.410631 17881 raft_consensus.cc:740] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Initialized, Role: FOLLOWER
I20260709 12:20:18.411312 17881 consensus_queue.cc:260] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:20:18.411401 17727 leader_election.cc:343] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:18.416342 17951 consensus_queue.cc:1048] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:20:18.418849 17652 leader_election.cc:304] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:20:18.420594 18005 raft_consensus.cc:2804] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:18.422070 18005 raft_consensus.cc:697] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 1 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:20:18.422212 17881 ts_tablet_manager.cc:1434] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b: Time spent starting tablet: real 0.018s	user 0.003s	sys 0.003s
I20260709 12:20:18.423784 17881 tablet_bootstrap.cc:492] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b: Bootstrap starting.
I20260709 12:20:18.423223 18005 consensus_queue.cc:237] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:18.452812 17935 raft_consensus.cc:493] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:18.453120 17763 raft_consensus.cc:2468] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 1.
I20260709 12:20:18.453389 17935 raft_consensus.cc:515] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:18.454159 17880 tablet_bootstrap.cc:492] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb: No bootstrap required, opened a new log
I20260709 12:20:18.454661 17880 ts_tablet_manager.cc:1403] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent bootstrapping tablet: real 0.337s	user 0.010s	sys 0.017s
I20260709 12:20:18.455519 17935 leader_election.cc:290] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:18.455694 17881 tablet_bootstrap.cc:654] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:18.457828 17951 consensus_queue.cc:1048] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:20:18.458855 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:20:18.459681 17687 raft_consensus.cc:2393] T 0f940f29438f475294357324452abe76 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 1: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:20:18.460860 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:20:18.463378 17880 raft_consensus.cc:359] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:18.464375 17880 raft_consensus.cc:385] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:18.466485 17880 raft_consensus.cc:740] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Initialized, Role: FOLLOWER
I20260709 12:20:18.465826 17727 leader_election.cc:304] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:20:18.469105 17992 raft_consensus.cc:3060] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:18.469425 17880 consensus_queue.cc:260] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:18.473513 17684 raft_consensus.cc:2468] T 28062a50142448aa97994888b4765920 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 1.
I20260709 12:20:18.479991 17728 leader_election.cc:304] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:20:18.481069 17880 ts_tablet_manager.cc:1434] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent starting tablet: real 0.026s	user 0.010s	sys 0.000s
I20260709 12:20:18.488288 17878 raft_consensus.cc:359] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:18.489108 17878 raft_consensus.cc:385] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:18.489410 17878 raft_consensus.cc:740] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Initialized, Role: FOLLOWER
I20260709 12:20:18.491160 17880 tablet_bootstrap.cc:492] T 45397041e3ac43c8901c5cb03f27c7b9 P e2909cff1da44ad387f96a8ae3aa5afb: Bootstrap starting.
I20260709 12:20:18.490067 17878 consensus_queue.cc:260] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:18.500078 17935 raft_consensus.cc:2804] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:18.504889 17935 raft_consensus.cc:697] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:20:18.520908 17975 consensus_queue.cc:1048] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:18.523672 17940 raft_consensus.cc:493] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:18.524312 17940 raft_consensus.cc:515] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:18.581820 17878 ts_tablet_manager.cc:1434] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3: Time spent starting tablet: real 0.177s	user 0.013s	sys 0.000s
I20260709 12:20:18.583457 17878 tablet_bootstrap.cc:492] T 19dad6ab92894d708ec8eb7e8ac28185 P 0da169ff82674dc1947055ce43de68a3: Bootstrap starting.
I20260709 12:20:18.585021 17992 raft_consensus.cc:2749] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260709 12:20:18.581319 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 106011us
I20260709 12:20:18.598289 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:20:18.594087 17940 leader_election.cc:290] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:18.605402 17952 consensus_queue.cc:1048] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:20:18.606643 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:20:18.608219 17652 leader_election.cc:343] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:20:18.610710 17635 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
W20260709 12:20:18.639050 18023 log.cc:927] Time spent T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.071s	user 0.000s	sys 0.000s
W20260709 12:20:18.644620 17652 leader_election.cc:343] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:18.645139 17652 leader_election.cc:304] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:20:18.638680 17995 consensus_queue.cc:1048] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.007s
I20260709 12:20:18.595319 17935 consensus_queue.cc:237] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:18.646646 17940 raft_consensus.cc:2749] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:18.649814 17585 catalog_manager.cc:5697] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b reported cstate change: term changed from 0 to 1, leader changed from <none> to e063541542a94969898868bf6eb5ce1b (127.16.29.67). New cstate: current_term: 1 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: HEALTHY } } }
I20260709 12:20:18.657688 17881 tablet_bootstrap.cc:492] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b: No bootstrap required, opened a new log
I20260709 12:20:18.658109 17881 ts_tablet_manager.cc:1403] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b: Time spent bootstrapping tablet: real 0.235s	user 0.014s	sys 0.010s
I20260709 12:20:18.660799 17881 raft_consensus.cc:359] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:18.661590 17881 raft_consensus.cc:385] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:18.661882 17881 raft_consensus.cc:740] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Initialized, Role: FOLLOWER
I20260709 12:20:18.662523 17881 consensus_queue.cc:260] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:18.666616 17881 ts_tablet_manager.cc:1434] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b: Time spent starting tablet: real 0.008s	user 0.007s	sys 0.000s
I20260709 12:20:18.667518 17881 tablet_bootstrap.cc:492] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b: Bootstrap starting.
I20260709 12:20:18.670305 17586 catalog_manager.cc:5697] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0da169ff82674dc1947055ce43de68a3 (127.16.29.65). New cstate: current_term: 1 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:18.679270 17878 tablet_bootstrap.cc:654] T 19dad6ab92894d708ec8eb7e8ac28185 P 0da169ff82674dc1947055ce43de68a3: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:18.701123 17880 tablet_bootstrap.cc:654] T 45397041e3ac43c8901c5cb03f27c7b9 P e2909cff1da44ad387f96a8ae3aa5afb: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:18.706064 17585 catalog_manager.cc:5697] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb reported cstate change: term changed from 0 to 1, leader changed from <none> to e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66). New cstate: current_term: 1 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:18.709777 17585 catalog_manager.cc:5697] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb reported cstate change: term changed from 0 to 1, leader changed from <none> to e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66). New cstate: current_term: 1 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:18.710587 17992 raft_consensus.cc:493] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:18.711069 17992 raft_consensus.cc:515] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:18.712404 17995 consensus_queue.cc:1048] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:20:18.713201 17992 leader_election.cc:290] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:18.714289 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:20:18.716177 17727 leader_election.cc:343] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:18.718967 17949 consensus_queue.cc:1048] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:20:18.732123 17935 consensus_queue.cc:1048] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:20:18.737079 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:20:18.737900 17684 raft_consensus.cc:2468] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 0.
I20260709 12:20:18.740304 17728 leader_election.cc:304] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:20:18.741832 18036 raft_consensus.cc:2804] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:18.742409 18036 raft_consensus.cc:493] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:18.742937 18036 raft_consensus.cc:3060] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:18.746675 17878 tablet_bootstrap.cc:492] T 19dad6ab92894d708ec8eb7e8ac28185 P 0da169ff82674dc1947055ce43de68a3: No bootstrap required, opened a new log
I20260709 12:20:18.747138 17878 ts_tablet_manager.cc:1403] T 19dad6ab92894d708ec8eb7e8ac28185 P 0da169ff82674dc1947055ce43de68a3: Time spent bootstrapping tablet: real 0.164s	user 0.015s	sys 0.003s
I20260709 12:20:18.748507 17881 tablet_bootstrap.cc:654] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:18.756007 17878 raft_consensus.cc:359] T 19dad6ab92894d708ec8eb7e8ac28185 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:18.757000 17878 raft_consensus.cc:385] T 19dad6ab92894d708ec8eb7e8ac28185 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:18.757400 17878 raft_consensus.cc:740] T 19dad6ab92894d708ec8eb7e8ac28185 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Initialized, Role: FOLLOWER
I20260709 12:20:18.761171 17878 consensus_queue.cc:260] T 19dad6ab92894d708ec8eb7e8ac28185 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:18.773404 18036 raft_consensus.cc:515] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:18.801579 18036 leader_election.cc:290] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 1 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:18.811101 17878 ts_tablet_manager.cc:1434] T 19dad6ab92894d708ec8eb7e8ac28185 P 0da169ff82674dc1947055ce43de68a3: Time spent starting tablet: real 0.064s	user 0.006s	sys 0.003s
I20260709 12:20:18.833500 17880 tablet_bootstrap.cc:492] T 45397041e3ac43c8901c5cb03f27c7b9 P e2909cff1da44ad387f96a8ae3aa5afb: No bootstrap required, opened a new log
I20260709 12:20:18.833976 17880 ts_tablet_manager.cc:1403] T 45397041e3ac43c8901c5cb03f27c7b9 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent bootstrapping tablet: real 0.343s	user 0.010s	sys 0.009s
I20260709 12:20:18.834156 17878 tablet_bootstrap.cc:492] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3: Bootstrap starting.
I20260709 12:20:18.834372 17881 tablet_bootstrap.cc:492] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b: No bootstrap required, opened a new log
I20260709 12:20:18.834946 17881 ts_tablet_manager.cc:1403] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b: Time spent bootstrapping tablet: real 0.168s	user 0.010s	sys 0.014s
I20260709 12:20:18.837291 17880 raft_consensus.cc:359] T 45397041e3ac43c8901c5cb03f27c7b9 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:18.833987 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:20:18.839413 17688 raft_consensus.cc:3060] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:18.844404 17940 raft_consensus.cc:493] T 19dad6ab92894d708ec8eb7e8ac28185 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:18.844905 17940 raft_consensus.cc:515] T 19dad6ab92894d708ec8eb7e8ac28185 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:18.845597 17878 tablet_bootstrap.cc:654] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:18.847343 17940 leader_election.cc:290] T 19dad6ab92894d708ec8eb7e8ac28185 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:18.844075 17880 raft_consensus.cc:385] T 45397041e3ac43c8901c5cb03f27c7b9 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:18.848184 17880 raft_consensus.cc:740] T 45397041e3ac43c8901c5cb03f27c7b9 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Initialized, Role: FOLLOWER
I20260709 12:20:18.848284 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:20:18.849001 17880 consensus_queue.cc:260] T 45397041e3ac43c8901c5cb03f27c7b9 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:20:18.849833 17652 leader_election.cc:343] T 19dad6ab92894d708ec8eb7e8ac28185 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:18.852975 17880 ts_tablet_manager.cc:1434] T 45397041e3ac43c8901c5cb03f27c7b9 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent starting tablet: real 0.019s	user 0.007s	sys 0.000s
I20260709 12:20:18.857437 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:20:18.866232 17652 leader_election.cc:343] T 19dad6ab92894d708ec8eb7e8ac28185 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:18.866722 17652 leader_election.cc:304] T 19dad6ab92894d708ec8eb7e8ac28185 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:20:18.867758 17940 raft_consensus.cc:2749] T 19dad6ab92894d708ec8eb7e8ac28185 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:18.870048 17912 raft_consensus.cc:493] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:18.870522 17912 raft_consensus.cc:515] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:18.871243 17881 raft_consensus.cc:359] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:18.872272 17881 raft_consensus.cc:385] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:18.872551 17881 raft_consensus.cc:740] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Initialized, Role: FOLLOWER
I20260709 12:20:18.873057 17912 leader_election.cc:290] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:20:18.873246 17881 consensus_queue.cc:260] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:18.876505 17881 ts_tablet_manager.cc:1434] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b: Time spent starting tablet: real 0.041s	user 0.008s	sys 0.000s
I20260709 12:20:18.877547 17881 tablet_bootstrap.cc:492] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b: Bootstrap starting.
I20260709 12:20:18.880465 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:20:18.881340 17878 tablet_bootstrap.cc:492] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3: No bootstrap required, opened a new log
I20260709 12:20:18.881769 17878 ts_tablet_manager.cc:1403] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3: Time spent bootstrapping tablet: real 0.048s	user 0.017s	sys 0.021s
I20260709 12:20:18.884690 17881 tablet_bootstrap.cc:654] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:18.884407 17878 raft_consensus.cc:359] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:18.885217 17878 raft_consensus.cc:385] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:18.885519 17878 raft_consensus.cc:740] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Initialized, Role: FOLLOWER
I20260709 12:20:18.881187 17763 raft_consensus.cc:2468] T 45397041e3ac43c8901c5cb03f27c7b9 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 0.
I20260709 12:20:18.886391 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:20:18.886536 17878 consensus_queue.cc:260] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:18.887207 17684 raft_consensus.cc:2468] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 0.
I20260709 12:20:18.889226 17878 ts_tablet_manager.cc:1434] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3: Time spent starting tablet: real 0.007s	user 0.008s	sys 0.000s
I20260709 12:20:18.890122 17878 tablet_bootstrap.cc:492] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3: Bootstrap starting.
I20260709 12:20:18.894510 17881 tablet_bootstrap.cc:492] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b: No bootstrap required, opened a new log
I20260709 12:20:18.895038 17881 ts_tablet_manager.cc:1403] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b: Time spent bootstrapping tablet: real 0.018s	user 0.012s	sys 0.001s
I20260709 12:20:18.896121 17880 tablet_bootstrap.cc:492] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb: Bootstrap starting.
I20260709 12:20:18.898629 17951 raft_consensus.cc:493] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:18.906059 17951 raft_consensus.cc:515] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:18.898797 17802 leader_election.cc:304] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:20:18.908416 17951 leader_election.cc:290] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:20:18.908943 17951 raft_consensus.cc:2804] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:18.909284 17951 raft_consensus.cc:493] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:18.909614 17951 raft_consensus.cc:3060] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:18.911870 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:20:18.914902 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:20:18.915530 17684 raft_consensus.cc:2468] T 11896555f5d7458dbb02a3203b27f1cf P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 0.
I20260709 12:20:18.900724 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:20:18.902861 17878 tablet_bootstrap.cc:654] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3: Neither blocks nor log segments found. Creating new log.
W20260709 12:20:18.917510 17801 leader_election.cc:343] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:18.905174 17881 raft_consensus.cc:359] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:18.920152 17881 raft_consensus.cc:385] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:18.920451 17881 raft_consensus.cc:740] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Initialized, Role: FOLLOWER
W20260709 12:20:18.920560 17727 leader_election.cc:343] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:18.921103 17881 consensus_queue.cc:260] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:18.922295 17951 raft_consensus.cc:515] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:18.923223 17881 ts_tablet_manager.cc:1434] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b: Time spent starting tablet: real 0.028s	user 0.003s	sys 0.000s
I20260709 12:20:18.923041 17802 leader_election.cc:304] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:20:18.923991 17912 raft_consensus.cc:2804] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:18.924160 17881 tablet_bootstrap.cc:492] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b: Bootstrap starting.
I20260709 12:20:18.924379 17912 raft_consensus.cc:493] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:18.924753 17912 raft_consensus.cc:3060] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:18.925580 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:20:18.926151 17762 raft_consensus.cc:3060] T 45397041e3ac43c8901c5cb03f27c7b9 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:18.930138 17878 tablet_bootstrap.cc:492] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3: No bootstrap required, opened a new log
I20260709 12:20:18.930572 17878 ts_tablet_manager.cc:1403] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3: Time spent bootstrapping tablet: real 0.041s	user 0.007s	sys 0.005s
I20260709 12:20:18.931758 17912 raft_consensus.cc:515] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:18.859542 17688 raft_consensus.cc:2468] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 1.
I20260709 12:20:18.933321 17878 raft_consensus.cc:359] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:18.933997 17878 raft_consensus.cc:385] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:18.934279 17878 raft_consensus.cc:740] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Initialized, Role: FOLLOWER
I20260709 12:20:18.934324 17728 leader_election.cc:304] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:20:18.934720 17912 leader_election.cc:290] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 1 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:20:18.932489 17881 tablet_bootstrap.cc:654] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:18.935118 17878 consensus_queue.cc:260] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:18.935247 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:20:18.935491 18036 raft_consensus.cc:2804] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:18.936050 17684 raft_consensus.cc:3060] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:18.936141 18036 raft_consensus.cc:697] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:20:18.937013 18036 consensus_queue.cc:237] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:18.941497 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
W20260709 12:20:18.942979 17801 leader_election.cc:343] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:18.933329 17762 raft_consensus.cc:2468] T 45397041e3ac43c8901c5cb03f27c7b9 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 1.
I20260709 12:20:18.944695 17801 leader_election.cc:304] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:20:18.945564 17912 raft_consensus.cc:2804] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:18.946059 17912 raft_consensus.cc:697] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 1 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:20:18.947196 17912 consensus_queue.cc:237] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:18.951793 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:20:18.952987 17689 raft_consensus.cc:3060] T 11896555f5d7458dbb02a3203b27f1cf P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:18.955566 17684 raft_consensus.cc:2468] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 1.
I20260709 12:20:18.965930 17880 tablet_bootstrap.cc:654] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:18.966122 17878 ts_tablet_manager.cc:1434] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3: Time spent starting tablet: real 0.035s	user 0.004s	sys 0.002s
I20260709 12:20:18.962868 17951 leader_election.cc:290] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 1 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:20:18.968439 17878 tablet_bootstrap.cc:492] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3: Bootstrap starting.
I20260709 12:20:18.962875 17689 raft_consensus.cc:2468] T 11896555f5d7458dbb02a3203b27f1cf P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 1.
I20260709 12:20:18.989959 17881 tablet_bootstrap.cc:492] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b: No bootstrap required, opened a new log
I20260709 12:20:18.990856 17881 ts_tablet_manager.cc:1403] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b: Time spent bootstrapping tablet: real 0.067s	user 0.022s	sys 0.022s
I20260709 12:20:18.988510 17802 leader_election.cc:304] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:20:18.993996 17912 raft_consensus.cc:2804] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:18.994737 17949 raft_consensus.cc:493] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:18.994951 17585 catalog_manager.cc:5697] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb reported cstate change: term changed from 0 to 1, leader changed from <none> to e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66). New cstate: current_term: 1 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:18.995391 17949 raft_consensus.cc:515] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:18.997778 17949 leader_election.cc:290] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:20:18.999363 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:20:18.999593 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:20:19.000015 17689 raft_consensus.cc:2468] T 19dad6ab92894d708ec8eb7e8ac28185 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 0.
W20260709 12:20:19.000892 17801 leader_election.cc:343] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:19.000883 17881 raft_consensus.cc:359] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:19.001670 17881 raft_consensus.cc:385] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:19.001941 17881 raft_consensus.cc:740] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Initialized, Role: FOLLOWER
I20260709 12:20:19.002625 17881 consensus_queue.cc:260] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:19.002310 17802 leader_election.cc:304] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:20:19.005196 17881 ts_tablet_manager.cc:1434] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b: Time spent starting tablet: real 0.014s	user 0.008s	sys 0.000s
I20260709 12:20:19.006074 17881 tablet_bootstrap.cc:492] T 8e417cb980e44a77962d2e3dfd58aad0 P e063541542a94969898868bf6eb5ce1b: Bootstrap starting.
I20260709 12:20:19.006796 17878 tablet_bootstrap.cc:654] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:19.008031 17949 raft_consensus.cc:2804] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:19.008361 17949 raft_consensus.cc:493] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:19.008371 17912 raft_consensus.cc:697] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 1 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:20:19.008850 17949 raft_consensus.cc:3060] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:19.010077 17951 raft_consensus.cc:493] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:19.011051 17951 raft_consensus.cc:515] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:19.014219 17585 catalog_manager.cc:5697] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b reported cstate change: term changed from 0 to 1, leader changed from <none> to e063541542a94969898868bf6eb5ce1b (127.16.29.67). New cstate: current_term: 1 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: HEALTHY } } }
I20260709 12:20:19.015136 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:20:19.019140 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:20:19.015764 17951 leader_election.cc:290] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:20:19.020597 17801 leader_election.cc:343] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:19.024233 17940 consensus_queue.cc:1048] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:19.019784 17689 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 0.
I20260709 12:20:19.027817 17802 leader_election.cc:304] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:20:19.009362 17912 consensus_queue.cc:237] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:19.030280 17951 raft_consensus.cc:2804] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:19.030781 17951 raft_consensus.cc:493] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:19.031198 17951 raft_consensus.cc:3060] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:19.043068 17880 tablet_bootstrap.cc:492] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb: No bootstrap required, opened a new log
I20260709 12:20:19.044095 17880 ts_tablet_manager.cc:1403] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb: Time spent bootstrapping tablet: real 0.148s	user 0.023s	sys 0.008s
I20260709 12:20:19.045215 17878 tablet_bootstrap.cc:492] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3: No bootstrap required, opened a new log
I20260709 12:20:19.045665 17878 ts_tablet_manager.cc:1403] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3: Time spent bootstrapping tablet: real 0.077s	user 0.012s	sys 0.006s
I20260709 12:20:19.046602 17949 raft_consensus.cc:515] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:19.049722 17881 tablet_bootstrap.cc:654] T 8e417cb980e44a77962d2e3dfd58aad0 P e063541542a94969898868bf6eb5ce1b: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:19.049813 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:20:19.050043 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:20:19.050487 17689 raft_consensus.cc:3060] T 19dad6ab92894d708ec8eb7e8ac28185 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:20:19.052106 17801 leader_election.cc:343] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:19.049850 17951 raft_consensus.cc:515] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:19.055526 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:20:19.056368 17684 raft_consensus.cc:3060] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:19.062464 17878 raft_consensus.cc:359] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:19.063191 17878 raft_consensus.cc:385] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:19.061328 17951 leader_election.cc:290] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 1 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:20:19.063439 17878 raft_consensus.cc:740] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Initialized, Role: FOLLOWER
I20260709 12:20:19.063329 17880 raft_consensus.cc:359] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:19.064514 17878 consensus_queue.cc:260] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:19.065255 17880 raft_consensus.cc:385] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:19.067061 18005 consensus_queue.cc:1048] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:20:19.067248 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:20:19.068451 17878 ts_tablet_manager.cc:1434] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3: Time spent starting tablet: real 0.023s	user 0.009s	sys 0.000s
I20260709 12:20:19.065838 17880 raft_consensus.cc:740] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Initialized, Role: FOLLOWER
I20260709 12:20:19.069681 17878 tablet_bootstrap.cc:492] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3: Bootstrap starting.
I20260709 12:20:19.069675 17880 consensus_queue.cc:260] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:19.072392 17880 ts_tablet_manager.cc:1434] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb: Time spent starting tablet: real 0.028s	user 0.009s	sys 0.000s
I20260709 12:20:19.074802 17880 tablet_bootstrap.cc:492] T 3219dcbfd86340ecbde507517060ff81 P e2909cff1da44ad387f96a8ae3aa5afb: Bootstrap starting.
W20260709 12:20:19.075021 17801 leader_election.cc:343] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:19.088377 17689 raft_consensus.cc:2468] T 19dad6ab92894d708ec8eb7e8ac28185 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 1.
I20260709 12:20:19.071996 17949 leader_election.cc:290] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 1 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:20:19.093853 17802 leader_election.cc:304] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:20:19.097153 17880 tablet_bootstrap.cc:654] T 3219dcbfd86340ecbde507517060ff81 P e2909cff1da44ad387f96a8ae3aa5afb: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:19.097317 17951 raft_consensus.cc:2804] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:19.097790 17951 raft_consensus.cc:697] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 1 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:20:19.109823 17951 consensus_queue.cc:237] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:19.104154 18036 raft_consensus.cc:493] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:19.112643 17837 raft_consensus.cc:3060] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:19.112573 18036 raft_consensus.cc:515] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:19.120594 17878 tablet_bootstrap.cc:654] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:19.113860 17684 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 1.
I20260709 12:20:19.133082 17880 tablet_bootstrap.cc:492] T 3219dcbfd86340ecbde507517060ff81 P e2909cff1da44ad387f96a8ae3aa5afb: No bootstrap required, opened a new log
I20260709 12:20:19.133776 17880 ts_tablet_manager.cc:1403] T 3219dcbfd86340ecbde507517060ff81 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent bootstrapping tablet: real 0.061s	user 0.016s	sys 0.003s
I20260709 12:20:19.139760 17880 raft_consensus.cc:359] T 3219dcbfd86340ecbde507517060ff81 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:19.140332 17802 leader_election.cc:304] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:20:19.140395 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:20:19.140470 17880 raft_consensus.cc:385] T 3219dcbfd86340ecbde507517060ff81 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:19.141218 17880 raft_consensus.cc:740] T 3219dcbfd86340ecbde507517060ff81 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Initialized, Role: FOLLOWER
I20260709 12:20:19.132764 18005 raft_consensus.cc:493] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:19.140451 17586 catalog_manager.cc:5697] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b reported cstate change: term changed from 0 to 1, leader changed from <none> to e063541542a94969898868bf6eb5ce1b (127.16.29.67). New cstate: current_term: 1 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: HEALTHY } } }
I20260709 12:20:19.141925 17880 consensus_queue.cc:260] T 3219dcbfd86340ecbde507517060ff81 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:19.141973 18005 raft_consensus.cc:515] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:19.144994 17912 raft_consensus.cc:2804] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:19.145678 17912 raft_consensus.cc:697] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 1 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:20:19.146023 17880 ts_tablet_manager.cc:1434] T 3219dcbfd86340ecbde507517060ff81 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent starting tablet: real 0.012s	user 0.006s	sys 0.001s
I20260709 12:20:19.148332 17880 tablet_bootstrap.cc:492] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb: Bootstrap starting.
I20260709 12:20:19.138578 18036 leader_election.cc:290] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:19.150554 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:20:19.151391 17684 raft_consensus.cc:2393] T 11896555f5d7458dbb02a3203b27f1cf P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 1: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:20:19.152588 17728 leader_election.cc:304] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:20:19.154274 17975 raft_consensus.cc:3060] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:19.201391 17912 consensus_queue.cc:237] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:19.211723 17880 tablet_bootstrap.cc:654] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:19.216284 17881 tablet_bootstrap.cc:492] T 8e417cb980e44a77962d2e3dfd58aad0 P e063541542a94969898868bf6eb5ce1b: No bootstrap required, opened a new log
I20260709 12:20:19.217057 17881 ts_tablet_manager.cc:1403] T 8e417cb980e44a77962d2e3dfd58aad0 P e063541542a94969898868bf6eb5ce1b: Time spent bootstrapping tablet: real 0.211s	user 0.028s	sys 0.039s
I20260709 12:20:19.218721 18005 leader_election.cc:290] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:19.222324 17878 tablet_bootstrap.cc:492] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3: No bootstrap required, opened a new log
I20260709 12:20:19.222816 17878 ts_tablet_manager.cc:1403] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3: Time spent bootstrapping tablet: real 0.153s	user 0.013s	sys 0.003s
I20260709 12:20:19.219959 17935 raft_consensus.cc:493] T 3219dcbfd86340ecbde507517060ff81 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:19.234668 17935 raft_consensus.cc:515] T 3219dcbfd86340ecbde507517060ff81 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:19.238642 17881 raft_consensus.cc:359] T 8e417cb980e44a77962d2e3dfd58aad0 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:19.242751 17881 raft_consensus.cc:385] T 8e417cb980e44a77962d2e3dfd58aad0 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:19.243119 17881 raft_consensus.cc:740] T 8e417cb980e44a77962d2e3dfd58aad0 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Initialized, Role: FOLLOWER
I20260709 12:20:19.243626 17878 raft_consensus.cc:359] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:19.243762 17881 consensus_queue.cc:260] T 8e417cb980e44a77962d2e3dfd58aad0 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:19.244428 17878 raft_consensus.cc:385] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:19.244791 17878 raft_consensus.cc:740] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Initialized, Role: FOLLOWER
I20260709 12:20:19.245929 17878 consensus_queue.cc:260] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:19.244781 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:20:19.229239 18036 consensus_queue.cc:1048] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.016s
I20260709 12:20:19.239566 17935 leader_election.cc:290] T 3219dcbfd86340ecbde507517060ff81 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:20:19.256726 17727 leader_election.cc:343] T 3219dcbfd86340ecbde507517060ff81 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:19.258082 17878 ts_tablet_manager.cc:1434] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3: Time spent starting tablet: real 0.035s	user 0.007s	sys 0.000s
I20260709 12:20:19.242090 17975 raft_consensus.cc:2749] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:19.259117 17878 tablet_bootstrap.cc:492] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3: Bootstrap starting.
I20260709 12:20:19.270758 17881 ts_tablet_manager.cc:1434] T 8e417cb980e44a77962d2e3dfd58aad0 P e063541542a94969898868bf6eb5ce1b: Time spent starting tablet: real 0.053s	user 0.006s	sys 0.001s
I20260709 12:20:19.272938 17881 tablet_bootstrap.cc:492] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b: Bootstrap starting.
I20260709 12:20:19.275758 18057 raft_consensus.cc:493] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:19.281911 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:20:19.259066 17995 consensus_queue.cc:1048] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.027s
I20260709 12:20:19.276386 18057 raft_consensus.cc:515] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:19.289930 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:20:19.291224 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:20:19.300181 17763 raft_consensus.cc:2468] T 3219dcbfd86340ecbde507517060ff81 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 0.
W20260709 12:20:19.295854 17652 leader_election.cc:343] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:19.299391 17684 raft_consensus.cc:2468] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 0.
I20260709 12:20:19.311627 17728 leader_election.cc:304] T 3219dcbfd86340ecbde507517060ff81 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:20:19.313041 17975 raft_consensus.cc:2804] T 3219dcbfd86340ecbde507517060ff81 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:19.313453 17975 raft_consensus.cc:493] T 3219dcbfd86340ecbde507517060ff81 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:19.313831 17975 raft_consensus.cc:3060] T 3219dcbfd86340ecbde507517060ff81 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:19.314982 17652 leader_election.cc:304] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:20:19.315326 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:20:19.317027 18005 raft_consensus.cc:2804] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20260709 12:20:19.317400 17652 leader_election.cc:343] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:19.317420 18005 raft_consensus.cc:493] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:19.318001 18005 raft_consensus.cc:3060] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:19.318034 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:20:19.316339 18057 leader_election.cc:290] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:20:19.319504 17652 leader_election.cc:343] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:19.319936 17652 leader_election.cc:304] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:20:19.320662 18057 raft_consensus.cc:2749] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:19.321190 17880 tablet_bootstrap.cc:492] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb: No bootstrap required, opened a new log
I20260709 12:20:19.321651 17880 ts_tablet_manager.cc:1403] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb: Time spent bootstrapping tablet: real 0.174s	user 0.015s	sys 0.004s
I20260709 12:20:19.324954 17880 raft_consensus.cc:359] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:19.325609 17880 raft_consensus.cc:385] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:19.325876 17880 raft_consensus.cc:740] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Initialized, Role: FOLLOWER
I20260709 12:20:19.327255 18057 raft_consensus.cc:493] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:19.328040 18057 raft_consensus.cc:515] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:19.347092 17585 catalog_manager.cc:5697] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b reported cstate change: term changed from 0 to 1, leader changed from <none> to e063541542a94969898868bf6eb5ce1b (127.16.29.67). New cstate: current_term: 1 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: HEALTHY } } }
I20260709 12:20:19.349637 18057 leader_election.cc:290] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:19.350349 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:20:19.350448 17880 consensus_queue.cc:260] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:19.352768 17880 ts_tablet_manager.cc:1434] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb: Time spent starting tablet: real 0.031s	user 0.004s	sys 0.008s
W20260709 12:20:19.353179 17652 leader_election.cc:343] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:19.353665 17880 tablet_bootstrap.cc:492] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb: Bootstrap starting.
I20260709 12:20:19.354108 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:20:19.355496 17652 leader_election.cc:343] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:19.355962 17652 leader_election.cc:304] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:20:19.351197 17585 catalog_manager.cc:5697] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b reported cstate change: term changed from 0 to 1, leader changed from <none> to e063541542a94969898868bf6eb5ce1b (127.16.29.67). New cstate: current_term: 1 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: HEALTHY } } }
I20260709 12:20:19.356701 18057 raft_consensus.cc:2749] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:19.362677 17951 consensus_queue.cc:1048] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:20:19.365098 17878 tablet_bootstrap.cc:654] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:19.367458 18005 raft_consensus.cc:515] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:19.368283 17995 consensus_queue.cc:1048] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:20:19.368644 17835 raft_consensus.cc:3060] T 8e417cb980e44a77962d2e3dfd58aad0 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:19.376962 17881 tablet_bootstrap.cc:654] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:19.378932 17975 raft_consensus.cc:515] T 3219dcbfd86340ecbde507517060ff81 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:19.385510 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:20:19.386348 17763 raft_consensus.cc:2393] T 3219dcbfd86340ecbde507517060ff81 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 1: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:20:19.390664 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:20:19.390949 18005 leader_election.cc:290] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:20:19.395287 17652 leader_election.cc:343] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:19.395766 17652 leader_election.cc:304] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:20:19.396908 18005 raft_consensus.cc:2749] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 12:20:19.395246 17880 tablet_bootstrap.cc:654] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:19.409722 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:20:19.407702 17975 leader_election.cc:290] T 3219dcbfd86340ecbde507517060ff81 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 1 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:19.410131 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:20:19.410612 17688 raft_consensus.cc:2393] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 1: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:20:19.412339 17881 tablet_bootstrap.cc:492] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b: No bootstrap required, opened a new log
I20260709 12:20:19.412761 17881 ts_tablet_manager.cc:1403] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b: Time spent bootstrapping tablet: real 0.140s	user 0.011s	sys 0.008s
I20260709 12:20:19.413789 17951 raft_consensus.cc:493] T 8e417cb980e44a77962d2e3dfd58aad0 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:19.414454 17951 raft_consensus.cc:515] T 8e417cb980e44a77962d2e3dfd58aad0 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:20:19.413767 17727 leader_election.cc:343] T 3219dcbfd86340ecbde507517060ff81 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:19.405155 17995 consensus_queue.cc:1048] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.008s
I20260709 12:20:19.421916 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:20:19.423540 17728 leader_election.cc:304] T 3219dcbfd86340ecbde507517060ff81 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:20:19.424755 17975 raft_consensus.cc:2749] T 3219dcbfd86340ecbde507517060ff81 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 12:20:19.427959 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:20:19.430657 17881 raft_consensus.cc:359] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:19.431725 17881 raft_consensus.cc:385] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:19.432149 17881 raft_consensus.cc:740] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Initialized, Role: FOLLOWER
I20260709 12:20:19.432246 17802 leader_election.cc:304] T 8e417cb980e44a77962d2e3dfd58aad0 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:20:19.433104 17952 raft_consensus.cc:2749] T 8e417cb980e44a77962d2e3dfd58aad0 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:20:19.433171 17881 consensus_queue.cc:260] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:19.436830 17881 ts_tablet_manager.cc:1434] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b: Time spent starting tablet: real 0.024s	user 0.009s	sys 0.000s
I20260709 12:20:19.438174 17881 tablet_bootstrap.cc:492] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b: Bootstrap starting.
I20260709 12:20:19.426687 17951 leader_election.cc:290] T 8e417cb980e44a77962d2e3dfd58aad0 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:20:19.441954 17880 tablet_bootstrap.cc:492] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb: No bootstrap required, opened a new log
I20260709 12:20:19.443852 17878 tablet_bootstrap.cc:492] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3: No bootstrap required, opened a new log
I20260709 12:20:19.444384 17878 ts_tablet_manager.cc:1403] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3: Time spent bootstrapping tablet: real 0.185s	user 0.019s	sys 0.011s
I20260709 12:20:19.450228 17950 consensus_queue.cc:1048] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:19.447221 17878 raft_consensus.cc:359] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:19.451339 17878 raft_consensus.cc:385] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:19.451658 17878 raft_consensus.cc:740] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Initialized, Role: FOLLOWER
I20260709 12:20:19.452493 17878 consensus_queue.cc:260] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:19.455200 17878 ts_tablet_manager.cc:1434] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3: Time spent starting tablet: real 0.011s	user 0.006s	sys 0.000s
I20260709 12:20:19.455662 17880 ts_tablet_manager.cc:1403] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent bootstrapping tablet: real 0.102s	user 0.022s	sys 0.016s
I20260709 12:20:19.457113 17878 tablet_bootstrap.cc:492] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3: Bootstrap starting.
I20260709 12:20:19.463480 17881 tablet_bootstrap.cc:654] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:19.481433 17881 tablet_bootstrap.cc:492] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b: No bootstrap required, opened a new log
I20260709 12:20:19.481832 17881 ts_tablet_manager.cc:1403] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b: Time spent bootstrapping tablet: real 0.044s	user 0.011s	sys 0.013s
I20260709 12:20:19.483665 17881 raft_consensus.cc:359] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:19.484242 17880 raft_consensus.cc:359] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:19.484236 17881 raft_consensus.cc:385] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:19.485075 17880 raft_consensus.cc:385] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:19.485102 17881 raft_consensus.cc:740] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Initialized, Role: FOLLOWER
I20260709 12:20:19.485481 17880 raft_consensus.cc:740] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Initialized, Role: FOLLOWER
I20260709 12:20:19.485977 17881 consensus_queue.cc:260] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:19.486096 17880 consensus_queue.cc:260] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:19.488005 17880 ts_tablet_manager.cc:1434] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent starting tablet: real 0.032s	user 0.007s	sys 0.000s
I20260709 12:20:19.488355 17881 ts_tablet_manager.cc:1434] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b: Time spent starting tablet: real 0.006s	user 0.006s	sys 0.001s
I20260709 12:20:19.489300 17881 tablet_bootstrap.cc:492] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b: Bootstrap starting.
I20260709 12:20:19.489306 17880 tablet_bootstrap.cc:492] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb: Bootstrap starting.
I20260709 12:20:19.492851 17878 tablet_bootstrap.cc:654] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:19.495944 17880 tablet_bootstrap.cc:654] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:19.499467 17878 tablet_bootstrap.cc:492] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3: No bootstrap required, opened a new log
I20260709 12:20:19.499956 17878 ts_tablet_manager.cc:1403] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3: Time spent bootstrapping tablet: real 0.043s	user 0.013s	sys 0.000s
I20260709 12:20:19.504410 17881 tablet_bootstrap.cc:654] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:19.510842 17880 tablet_bootstrap.cc:492] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb: No bootstrap required, opened a new log
I20260709 12:20:19.512458 17880 ts_tablet_manager.cc:1403] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb: Time spent bootstrapping tablet: real 0.023s	user 0.015s	sys 0.008s
I20260709 12:20:19.512629 17878 raft_consensus.cc:359] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:19.515302 17878 raft_consensus.cc:385] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:19.515092 17880 raft_consensus.cc:359] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:19.515601 17878 raft_consensus.cc:740] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Initialized, Role: FOLLOWER
I20260709 12:20:19.515707 17880 raft_consensus.cc:385] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:19.515997 17880 raft_consensus.cc:740] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Initialized, Role: FOLLOWER
I20260709 12:20:19.516268 17878 consensus_queue.cc:260] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:19.516578 17880 consensus_queue.cc:260] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:19.514513 17881 tablet_bootstrap.cc:492] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b: No bootstrap required, opened a new log
I20260709 12:20:19.517506 17881 ts_tablet_manager.cc:1403] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b: Time spent bootstrapping tablet: real 0.028s	user 0.011s	sys 0.004s
I20260709 12:20:19.518486 17880 ts_tablet_manager.cc:1434] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb: Time spent starting tablet: real 0.006s	user 0.004s	sys 0.000s
I20260709 12:20:19.519801 17880 tablet_bootstrap.cc:492] T b5462347eb74482b8cd11beb1d39b8a9 P e2909cff1da44ad387f96a8ae3aa5afb: Bootstrap starting.
I20260709 12:20:19.519840 17881 raft_consensus.cc:359] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:19.520541 17881 raft_consensus.cc:385] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:19.520784 17881 raft_consensus.cc:740] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Initialized, Role: FOLLOWER
I20260709 12:20:19.521413 17881 consensus_queue.cc:260] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:19.523444 17881 ts_tablet_manager.cc:1434] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b: Time spent starting tablet: real 0.006s	user 0.003s	sys 0.000s
I20260709 12:20:19.524343 17881 tablet_bootstrap.cc:492] T d11ed0a8f1094765abe028dc60aa8bed P e063541542a94969898868bf6eb5ce1b: Bootstrap starting.
I20260709 12:20:19.526543 17880 tablet_bootstrap.cc:654] T b5462347eb74482b8cd11beb1d39b8a9 P e2909cff1da44ad387f96a8ae3aa5afb: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:19.528191 17878 ts_tablet_manager.cc:1434] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3: Time spent starting tablet: real 0.028s	user 0.007s	sys 0.000s
I20260709 12:20:19.529232 17878 tablet_bootstrap.cc:492] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3: Bootstrap starting.
I20260709 12:20:19.531665 17881 tablet_bootstrap.cc:654] T d11ed0a8f1094765abe028dc60aa8bed P e063541542a94969898868bf6eb5ce1b: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:19.539147 17880 tablet_bootstrap.cc:492] T b5462347eb74482b8cd11beb1d39b8a9 P e2909cff1da44ad387f96a8ae3aa5afb: No bootstrap required, opened a new log
I20260709 12:20:19.539759 17880 ts_tablet_manager.cc:1403] T b5462347eb74482b8cd11beb1d39b8a9 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent bootstrapping tablet: real 0.020s	user 0.012s	sys 0.004s
I20260709 12:20:19.543910 17880 raft_consensus.cc:359] T b5462347eb74482b8cd11beb1d39b8a9 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:19.544867 17880 raft_consensus.cc:385] T b5462347eb74482b8cd11beb1d39b8a9 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:19.545116 17880 raft_consensus.cc:740] T b5462347eb74482b8cd11beb1d39b8a9 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Initialized, Role: FOLLOWER
I20260709 12:20:19.545745 17880 consensus_queue.cc:260] T b5462347eb74482b8cd11beb1d39b8a9 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:19.548563 17880 ts_tablet_manager.cc:1434] T b5462347eb74482b8cd11beb1d39b8a9 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent starting tablet: real 0.008s	user 0.007s	sys 0.000s
I20260709 12:20:19.549460 17881 tablet_bootstrap.cc:492] T d11ed0a8f1094765abe028dc60aa8bed P e063541542a94969898868bf6eb5ce1b: No bootstrap required, opened a new log
I20260709 12:20:19.549912 17881 ts_tablet_manager.cc:1403] T d11ed0a8f1094765abe028dc60aa8bed P e063541542a94969898868bf6eb5ce1b: Time spent bootstrapping tablet: real 0.026s	user 0.014s	sys 0.008s
I20260709 12:20:19.552357 17881 raft_consensus.cc:359] T d11ed0a8f1094765abe028dc60aa8bed P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:19.552980 17881 raft_consensus.cc:385] T d11ed0a8f1094765abe028dc60aa8bed P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:19.553303 17878 tablet_bootstrap.cc:654] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:19.553258 17881 raft_consensus.cc:740] T d11ed0a8f1094765abe028dc60aa8bed P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Initialized, Role: FOLLOWER
I20260709 12:20:19.553944 17881 consensus_queue.cc:260] T d11ed0a8f1094765abe028dc60aa8bed P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:19.555150 17763 raft_consensus.cc:3060] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:19.556145 17881 ts_tablet_manager.cc:1434] T d11ed0a8f1094765abe028dc60aa8bed P e063541542a94969898868bf6eb5ce1b: Time spent starting tablet: real 0.006s	user 0.006s	sys 0.000s
I20260709 12:20:19.557343 17881 tablet_bootstrap.cc:492] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b: Bootstrap starting.
I20260709 12:20:19.549475 17880 tablet_bootstrap.cc:492] T bd3425997bcf4fe7abc0399fd06e54d9 P e2909cff1da44ad387f96a8ae3aa5afb: Bootstrap starting.
I20260709 12:20:19.562922 17951 consensus_queue.cc:1048] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:20:19.567862 17880 tablet_bootstrap.cc:654] T bd3425997bcf4fe7abc0399fd06e54d9 P e2909cff1da44ad387f96a8ae3aa5afb: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:19.571580 17950 consensus_queue.cc:1048] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:20:19.604130 17881 tablet_bootstrap.cc:654] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:19.605860 17880 tablet_bootstrap.cc:492] T bd3425997bcf4fe7abc0399fd06e54d9 P e2909cff1da44ad387f96a8ae3aa5afb: No bootstrap required, opened a new log
I20260709 12:20:19.606317 17880 ts_tablet_manager.cc:1403] T bd3425997bcf4fe7abc0399fd06e54d9 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent bootstrapping tablet: real 0.057s	user 0.022s	sys 0.012s
I20260709 12:20:19.607611 17878 tablet_bootstrap.cc:492] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3: No bootstrap required, opened a new log
I20260709 12:20:19.609364 17880 raft_consensus.cc:359] T bd3425997bcf4fe7abc0399fd06e54d9 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:19.611475 17880 raft_consensus.cc:385] T bd3425997bcf4fe7abc0399fd06e54d9 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:19.611640 17878 ts_tablet_manager.cc:1403] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3: Time spent bootstrapping tablet: real 0.083s	user 0.030s	sys 0.014s
I20260709 12:20:19.611864 17880 raft_consensus.cc:740] T bd3425997bcf4fe7abc0399fd06e54d9 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Initialized, Role: FOLLOWER
I20260709 12:20:19.613500 17949 consensus_queue.cc:1048] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:19.614790 17880 consensus_queue.cc:260] T bd3425997bcf4fe7abc0399fd06e54d9 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:19.627144 17878 raft_consensus.cc:359] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:19.627302 17952 consensus_queue.cc:1048] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:20:19.629478 17880 ts_tablet_manager.cc:1434] T bd3425997bcf4fe7abc0399fd06e54d9 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent starting tablet: real 0.023s	user 0.008s	sys 0.000s
I20260709 12:20:19.632750 17880 tablet_bootstrap.cc:492] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb: Bootstrap starting.
I20260709 12:20:19.632541 17878 raft_consensus.cc:385] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:19.648133 17878 raft_consensus.cc:740] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Initialized, Role: FOLLOWER
I20260709 12:20:19.648458 17950 raft_consensus.cc:493] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:19.649996 17950 raft_consensus.cc:515] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:19.651410 17761 raft_consensus.cc:3060] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:19.650278 17878 consensus_queue.cc:260] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:19.654040 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:20:19.654923 17684 raft_consensus.cc:2468] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 0.
I20260709 12:20:19.654510 17878 ts_tablet_manager.cc:1434] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3: Time spent starting tablet: real 0.042s	user 0.005s	sys 0.001s
I20260709 12:20:19.655745 17995 raft_consensus.cc:493] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:19.656419 17802 leader_election.cc:304] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:20:19.656423 17995 raft_consensus.cc:515] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:19.657335 17912 raft_consensus.cc:2804] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:19.657706 17912 raft_consensus.cc:493] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:19.658038 17912 raft_consensus.cc:3060] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:19.660557 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:20:19.661253 17688 raft_consensus.cc:2468] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 0.
I20260709 12:20:19.660701 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:20:19.662696 17878 tablet_bootstrap.cc:492] T 1a4040191930435786e2eeea338bb3d3 P 0da169ff82674dc1947055ce43de68a3: Bootstrap starting.
I20260709 12:20:19.662640 17834 raft_consensus.cc:2468] T d11ed0a8f1094765abe028dc60aa8bed P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 0.
I20260709 12:20:19.663302 17728 leader_election.cc:304] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:20:19.663991 17880 tablet_bootstrap.cc:654] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:19.667223 17935 raft_consensus.cc:2804] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:19.667439 17950 leader_election.cc:290] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:20:19.667652 17935 raft_consensus.cc:493] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:19.668087 17935 raft_consensus.cc:3060] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:19.668999 17949 consensus_queue.cc:1048] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:20:19.669250 17995 leader_election.cc:290] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:19.666339 17912 raft_consensus.cc:515] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:19.677764 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:20:19.678512 17684 raft_consensus.cc:3060] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:19.681641 17935 raft_consensus.cc:515] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:19.688475 17912 leader_election.cc:290] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 1 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:20:19.689392 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:20:19.689742 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:20:19.690277 17688 raft_consensus.cc:3060] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:19.690393 17763 raft_consensus.cc:3060] T b5462347eb74482b8cd11beb1d39b8a9 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:19.693286 17684 raft_consensus.cc:2468] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 1.
I20260709 12:20:19.693879 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:20:19.694605 17802 leader_election.cc:304] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:20:19.695454 17912 raft_consensus.cc:2804] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:19.696012 17912 raft_consensus.cc:697] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 1 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:20:19.696877 17912 consensus_queue.cc:237] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:19.748916 17950 raft_consensus.cc:493] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:19.749614 17950 raft_consensus.cc:515] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:19.783299 17881 tablet_bootstrap.cc:492] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b: No bootstrap required, opened a new log
I20260709 12:20:19.785805 17688 raft_consensus.cc:2468] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 1.
I20260709 12:20:19.786162 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:20:19.786862 17834 raft_consensus.cc:3060] T d11ed0a8f1094765abe028dc60aa8bed P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:19.783938 17881 ts_tablet_manager.cc:1403] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b: Time spent bootstrapping tablet: real 0.227s	user 0.026s	sys 0.024s
I20260709 12:20:19.789302 17587 catalog_manager.cc:5697] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b reported cstate change: term changed from 0 to 1, leader changed from <none> to e063541542a94969898868bf6eb5ce1b (127.16.29.67). New cstate: current_term: 1 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: HEALTHY } } }
I20260709 12:20:19.790786 17881 raft_consensus.cc:359] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:19.791410 17881 raft_consensus.cc:385] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:19.791683 17881 raft_consensus.cc:740] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Initialized, Role: FOLLOWER
I20260709 12:20:19.792042 17728 leader_election.cc:304] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:20:19.792322 17881 consensus_queue.cc:260] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:19.784389 17878 tablet_bootstrap.cc:654] T 1a4040191930435786e2eeea338bb3d3 P 0da169ff82674dc1947055ce43de68a3: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:19.785291 17763 raft_consensus.cc:2468] T b5462347eb74482b8cd11beb1d39b8a9 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 1.
I20260709 12:20:19.783826 17935 leader_election.cc:290] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 1 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:19.800247 18036 raft_consensus.cc:2804] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:19.800760 18036 raft_consensus.cc:697] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 1 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:20:19.801882 18036 consensus_queue.cc:237] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:20:19.804425 17635 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:20:19.807272 17950 leader_election.cc:290] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:20:19.808495 17881 ts_tablet_manager.cc:1434] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b: Time spent starting tablet: real 0.020s	user 0.005s	sys 0.000s
I20260709 12:20:19.808400 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:20:19.809878 17761 raft_consensus.cc:2393] T 3219dcbfd86340ecbde507517060ff81 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 1: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:20:19.809976 17881 tablet_bootstrap.cc:492] T ea7da467817a4ba88aa5797c3f8f493f P e063541542a94969898868bf6eb5ce1b: Bootstrap starting.
I20260709 12:20:19.816370 17949 consensus_queue.cc:1048] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:20:19.803802 17834 raft_consensus.cc:2468] T d11ed0a8f1094765abe028dc60aa8bed P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 1.
I20260709 12:20:19.827562 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:20:19.836987 17689 raft_consensus.cc:2393] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 1: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:20:19.845400 17802 leader_election.cc:304] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:20:19.846983 17949 raft_consensus.cc:3060] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:19.862321 17952 raft_consensus.cc:493] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:19.865571 17952 raft_consensus.cc:515] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:19.869695 17952 leader_election.cc:290] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:20:19.869899 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:20:19.861640 17949 raft_consensus.cc:2749] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:19.874397 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:20:19.875301 17689 raft_consensus.cc:2468] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 0.
I20260709 12:20:19.877342 18005 raft_consensus.cc:493] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:19.877955 18005 raft_consensus.cc:515] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:19.879964 18005 leader_election.cc:290] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:20:19.877425 17801 leader_election.cc:343] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:19.881480 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:20:19.881820 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:20:19.882105 17762 raft_consensus.cc:2468] T bd3425997bcf4fe7abc0399fd06e54d9 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 0.
I20260709 12:20:19.882436 17835 raft_consensus.cc:2468] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 0.
I20260709 12:20:19.884271 17652 leader_election.cc:304] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:20:19.885972 17881 tablet_bootstrap.cc:654] T ea7da467817a4ba88aa5797c3f8f493f P e063541542a94969898868bf6eb5ce1b: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:19.893055 18005 raft_consensus.cc:2804] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:19.893622 18005 raft_consensus.cc:493] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:19.893975 18005 raft_consensus.cc:3060] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:19.895608 17802 leader_election.cc:304] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:20:19.897003 17880 tablet_bootstrap.cc:492] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb: No bootstrap required, opened a new log
I20260709 12:20:19.898363 17880 ts_tablet_manager.cc:1403] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent bootstrapping tablet: real 0.266s	user 0.037s	sys 0.041s
I20260709 12:20:19.901144 17912 raft_consensus.cc:2804] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:19.901029 17880 raft_consensus.cc:359] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:19.901592 17912 raft_consensus.cc:493] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:19.902000 17912 raft_consensus.cc:3060] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:19.901822 17585 catalog_manager.cc:5697] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb reported cstate change: term changed from 0 to 1, leader changed from <none> to e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66). New cstate: current_term: 1 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:19.901757 17880 raft_consensus.cc:385] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:19.904405 17880 raft_consensus.cc:740] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Initialized, Role: FOLLOWER
I20260709 12:20:19.905438 17880 consensus_queue.cc:260] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:19.908452 17880 ts_tablet_manager.cc:1434] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent starting tablet: real 0.010s	user 0.006s	sys 0.001s
I20260709 12:20:19.913661 17995 raft_consensus.cc:493] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:19.914289 17995 raft_consensus.cc:515] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:19.922847 17878 tablet_bootstrap.cc:492] T 1a4040191930435786e2eeea338bb3d3 P 0da169ff82674dc1947055ce43de68a3: No bootstrap required, opened a new log
I20260709 12:20:19.923337 17878 ts_tablet_manager.cc:1403] T 1a4040191930435786e2eeea338bb3d3 P 0da169ff82674dc1947055ce43de68a3: Time spent bootstrapping tablet: real 0.261s	user 0.012s	sys 0.031s
I20260709 12:20:19.924675 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:20:19.925328 17688 raft_consensus.cc:2468] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 0.
I20260709 12:20:19.927085 17880 tablet_bootstrap.cc:492] T 5ad2828ec58f490a9ea564d9805b56b7 P e2909cff1da44ad387f96a8ae3aa5afb: Bootstrap starting.
I20260709 12:20:19.927532 17878 raft_consensus.cc:359] T 1a4040191930435786e2eeea338bb3d3 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:19.928457 17878 raft_consensus.cc:385] T 1a4040191930435786e2eeea338bb3d3 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:19.929056 17878 raft_consensus.cc:740] T 1a4040191930435786e2eeea338bb3d3 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Initialized, Role: FOLLOWER
I20260709 12:20:19.935406 17728 leader_election.cc:304] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:20:19.936030 17878 consensus_queue.cc:260] T 1a4040191930435786e2eeea338bb3d3 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:19.938388 17878 ts_tablet_manager.cc:1434] T 1a4040191930435786e2eeea338bb3d3 P 0da169ff82674dc1947055ce43de68a3: Time spent starting tablet: real 0.015s	user 0.005s	sys 0.003s
I20260709 12:20:19.939260 17878 tablet_bootstrap.cc:492] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3: Bootstrap starting.
I20260709 12:20:19.947696 17995 leader_election.cc:290] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:19.949817 18005 raft_consensus.cc:515] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:19.941469 18036 raft_consensus.cc:2804] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:19.951114 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:20:19.951637 18036 raft_consensus.cc:493] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:19.952256 18005 leader_election.cc:290] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:19.952497 18036 raft_consensus.cc:3060] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:20:19.954756 17727 leader_election.cc:343] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:19.957665 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:20:19.958725 17835 raft_consensus.cc:3060] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:19.957665 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:20:19.959592 17762 raft_consensus.cc:3060] T bd3425997bcf4fe7abc0399fd06e54d9 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:19.964962 17912 raft_consensus.cc:515] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:19.967155 17912 leader_election.cc:290] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 1 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:20:19.971237 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:20:19.971518 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:20:19.971843 17684 raft_consensus.cc:3060] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:19.972293 17835 raft_consensus.cc:2468] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 1.
I20260709 12:20:19.973955 17762 raft_consensus.cc:2468] T bd3425997bcf4fe7abc0399fd06e54d9 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 1.
I20260709 12:20:19.974099 17652 leader_election.cc:304] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
W20260709 12:20:19.974009 17801 leader_election.cc:343] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:19.974992 18055 raft_consensus.cc:493] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:19.975472 18055 raft_consensus.cc:515] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:19.976784 18036 raft_consensus.cc:515] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:19.977301 18055 leader_election.cc:290] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:19.979001 17881 tablet_bootstrap.cc:492] T ea7da467817a4ba88aa5797c3f8f493f P e063541542a94969898868bf6eb5ce1b: No bootstrap required, opened a new log
I20260709 12:20:19.979254 17684 raft_consensus.cc:2468] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 1.
I20260709 12:20:19.980698 17881 ts_tablet_manager.cc:1403] T ea7da467817a4ba88aa5797c3f8f493f P e063541542a94969898868bf6eb5ce1b: Time spent bootstrapping tablet: real 0.171s	user 0.019s	sys 0.020s
I20260709 12:20:19.982158 18005 raft_consensus.cc:2804] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:19.984623 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:20:19.984788 17881 raft_consensus.cc:359] T ea7da467817a4ba88aa5797c3f8f493f P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:19.985884 18036 leader_election.cc:290] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 1 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:19.987232 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:20:19.987510 17881 raft_consensus.cc:385] T ea7da467817a4ba88aa5797c3f8f493f P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:19.987820 17689 raft_consensus.cc:3060] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:19.988152 17881 raft_consensus.cc:740] T ea7da467817a4ba88aa5797c3f8f493f P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Initialized, Role: FOLLOWER
I20260709 12:20:19.989182 17881 consensus_queue.cc:260] T ea7da467817a4ba88aa5797c3f8f493f P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:19.991165 17881 ts_tablet_manager.cc:1434] T ea7da467817a4ba88aa5797c3f8f493f P e063541542a94969898868bf6eb5ce1b: Time spent starting tablet: real 0.010s	user 0.004s	sys 0.000s
I20260709 12:20:19.993242 17881 tablet_bootstrap.cc:492] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b: Bootstrap starting.
I20260709 12:20:19.994226 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
W20260709 12:20:19.997993 17727 leader_election.cc:343] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:19.986706 18005 raft_consensus.cc:697] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 1 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:20:20.009013 17802 leader_election.cc:304] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:20:20.009317 17652 leader_election.cc:343] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:20.009824 17949 raft_consensus.cc:2804] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:20.010368 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:20:20.012703 17949 raft_consensus.cc:697] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 1 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:20:20.012871 17835 raft_consensus.cc:2468] T ea7da467817a4ba88aa5797c3f8f493f P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 0.
I20260709 12:20:20.013582 17949 consensus_queue.cc:237] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:20.014034 17652 leader_election.cc:304] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:20:20.014808 18055 raft_consensus.cc:2804] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:20.015698 18055 raft_consensus.cc:493] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:20.016078 18055 raft_consensus.cc:3060] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:20.016651 17689 raft_consensus.cc:2468] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 1.
I20260709 12:20:20.017689 17728 leader_election.cc:304] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:20:20.018766 17995 raft_consensus.cc:2804] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:20.016177 18005 consensus_queue.cc:237] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:20.019234 17995 raft_consensus.cc:697] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:20:20.020248 17995 consensus_queue.cc:237] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:20.021945 17878 tablet_bootstrap.cc:654] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:20.023603 17880 tablet_bootstrap.cc:654] T 5ad2828ec58f490a9ea564d9805b56b7 P e2909cff1da44ad387f96a8ae3aa5afb: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:20.041400 18055 raft_consensus.cc:515] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:20.044281 17584 catalog_manager.cc:5697] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b reported cstate change: term changed from 0 to 1, leader changed from <none> to e063541542a94969898868bf6eb5ce1b (127.16.29.67). New cstate: current_term: 1 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: HEALTHY } } }
I20260709 12:20:20.044281 17587 catalog_manager.cc:5697] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb reported cstate change: term changed from 0 to 1, leader changed from <none> to e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66). New cstate: current_term: 1 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:20.044143 17585 catalog_manager.cc:5697] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0da169ff82674dc1947055ce43de68a3 (127.16.29.65). New cstate: current_term: 1 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:20.049885 17881 tablet_bootstrap.cc:654] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:20.051065 18055 leader_election.cc:290] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:20.051779 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:20:20.053002 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
W20260709 12:20:20.053591 17652 leader_election.cc:343] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:20.053614 17835 raft_consensus.cc:3060] T ea7da467817a4ba88aa5797c3f8f493f P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:20.064236 17878 tablet_bootstrap.cc:492] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3: No bootstrap required, opened a new log
I20260709 12:20:20.064769 17878 ts_tablet_manager.cc:1403] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3: Time spent bootstrapping tablet: real 0.126s	user 0.005s	sys 0.011s
I20260709 12:20:20.064664 17835 raft_consensus.cc:2468] T ea7da467817a4ba88aa5797c3f8f493f P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 1.
I20260709 12:20:20.066466 17652 leader_election.cc:304] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:20:20.067549 18005 raft_consensus.cc:2804] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:20.068157 18005 raft_consensus.cc:697] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 1 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:20:20.069098 18005 consensus_queue.cc:237] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:20.067559 17878 raft_consensus.cc:359] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:20.070266 17878 raft_consensus.cc:385] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:20.070564 17878 raft_consensus.cc:740] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Initialized, Role: FOLLOWER
I20260709 12:20:20.071233 17878 consensus_queue.cc:260] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:20.073721 17878 ts_tablet_manager.cc:1434] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3: Time spent starting tablet: real 0.009s	user 0.006s	sys 0.000s
I20260709 12:20:20.074946 17878 tablet_bootstrap.cc:492] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3: Bootstrap starting.
I20260709 12:20:20.077575 17880 tablet_bootstrap.cc:492] T 5ad2828ec58f490a9ea564d9805b56b7 P e2909cff1da44ad387f96a8ae3aa5afb: No bootstrap required, opened a new log
I20260709 12:20:20.078262 17880 ts_tablet_manager.cc:1403] T 5ad2828ec58f490a9ea564d9805b56b7 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent bootstrapping tablet: real 0.169s	user 0.014s	sys 0.024s
I20260709 12:20:20.082223 17880 raft_consensus.cc:359] T 5ad2828ec58f490a9ea564d9805b56b7 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:20.083253 17880 raft_consensus.cc:385] T 5ad2828ec58f490a9ea564d9805b56b7 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:20.083590 17880 raft_consensus.cc:740] T 5ad2828ec58f490a9ea564d9805b56b7 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Initialized, Role: FOLLOWER
I20260709 12:20:20.087941 17881 tablet_bootstrap.cc:492] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b: No bootstrap required, opened a new log
I20260709 12:20:20.094182 17880 consensus_queue.cc:260] T 5ad2828ec58f490a9ea564d9805b56b7 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:20.096565 17880 ts_tablet_manager.cc:1434] T 5ad2828ec58f490a9ea564d9805b56b7 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent starting tablet: real 0.018s	user 0.008s	sys 0.000s
I20260709 12:20:20.097499 17880 tablet_bootstrap.cc:492] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb: Bootstrap starting.
I20260709 12:20:20.093616 17878 tablet_bootstrap.cc:654] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:20.104173 17881 ts_tablet_manager.cc:1403] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b: Time spent bootstrapping tablet: real 0.111s	user 0.021s	sys 0.000s
I20260709 12:20:20.105338 17880 tablet_bootstrap.cc:654] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:20.107139 17881 raft_consensus.cc:359] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:20.108071 17881 raft_consensus.cc:385] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:20.108351 17881 raft_consensus.cc:740] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Initialized, Role: FOLLOWER
I20260709 12:20:20.108985 17881 consensus_queue.cc:260] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:20.110982 17881 ts_tablet_manager.cc:1434] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b: Time spent starting tablet: real 0.006s	user 0.004s	sys 0.000s
I20260709 12:20:20.112246 17881 tablet_bootstrap.cc:492] T 154e6b769a8d476ba26f7fe3a521fc8d P e063541542a94969898868bf6eb5ce1b: Bootstrap starting.
I20260709 12:20:20.116598 17584 catalog_manager.cc:5697] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0da169ff82674dc1947055ce43de68a3 (127.16.29.65). New cstate: current_term: 1 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:20.117913 17881 tablet_bootstrap.cc:654] T 154e6b769a8d476ba26f7fe3a521fc8d P e063541542a94969898868bf6eb5ce1b: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:20.120076 18005 raft_consensus.cc:493] T 1a4040191930435786e2eeea338bb3d3 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:20.120553 18005 raft_consensus.cc:515] T 1a4040191930435786e2eeea338bb3d3 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:20.122697 18005 leader_election.cc:290] T 1a4040191930435786e2eeea338bb3d3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:20.123528 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:20:20.125640 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:20:20.127055 17995 raft_consensus.cc:493] T 5ad2828ec58f490a9ea564d9805b56b7 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:20.127628 17995 raft_consensus.cc:515] T 5ad2828ec58f490a9ea564d9805b56b7 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:20.129313 17881 tablet_bootstrap.cc:492] T 154e6b769a8d476ba26f7fe3a521fc8d P e063541542a94969898868bf6eb5ce1b: No bootstrap required, opened a new log
I20260709 12:20:20.129778 17881 ts_tablet_manager.cc:1403] T 154e6b769a8d476ba26f7fe3a521fc8d P e063541542a94969898868bf6eb5ce1b: Time spent bootstrapping tablet: real 0.018s	user 0.014s	sys 0.001s
I20260709 12:20:20.130048 17995 leader_election.cc:290] T 5ad2828ec58f490a9ea564d9805b56b7 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:20.131625 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:20:20.132404 17689 raft_consensus.cc:2393] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 1: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
W20260709 12:20:20.133576 17652 leader_election.cc:343] T 1a4040191930435786e2eeea338bb3d3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:20:20.134537 17652 leader_election.cc:343] T 1a4040191930435786e2eeea338bb3d3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:20.135111 17652 leader_election.cc:304] T 1a4040191930435786e2eeea338bb3d3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:20:20.136994 18005 raft_consensus.cc:2749] T 1a4040191930435786e2eeea338bb3d3 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:20.140525 17880 tablet_bootstrap.cc:492] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb: No bootstrap required, opened a new log
I20260709 12:20:20.140609 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:20:20.141000 17878 tablet_bootstrap.cc:492] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3: No bootstrap required, opened a new log
I20260709 12:20:20.141605 17878 ts_tablet_manager.cc:1403] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3: Time spent bootstrapping tablet: real 0.067s	user 0.009s	sys 0.006s
I20260709 12:20:20.141059 17880 ts_tablet_manager.cc:1403] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb: Time spent bootstrapping tablet: real 0.044s	user 0.026s	sys 0.010s
I20260709 12:20:20.142231 17727 leader_election.cc:304] T 5ad2828ec58f490a9ea564d9805b56b7 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:20:20.143345 17995 raft_consensus.cc:3060] T 5ad2828ec58f490a9ea564d9805b56b7 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:20.144567 17878 raft_consensus.cc:359] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:20.145277 17878 raft_consensus.cc:385] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:20.145709 17878 raft_consensus.cc:740] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Initialized, Role: FOLLOWER
I20260709 12:20:20.145488 17880 raft_consensus.cc:359] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:20.146198 17880 raft_consensus.cc:385] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:20.146490 17880 raft_consensus.cc:740] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Initialized, Role: FOLLOWER
I20260709 12:20:20.146629 17878 consensus_queue.cc:260] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:20.147168 17880 consensus_queue.cc:260] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:20.148713 17880 ts_tablet_manager.cc:1434] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb: Time spent starting tablet: real 0.006s	user 0.005s	sys 0.000s
I20260709 12:20:20.150897 17880 tablet_bootstrap.cc:492] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb: Bootstrap starting.
I20260709 12:20:20.155205 17878 ts_tablet_manager.cc:1434] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3: Time spent starting tablet: real 0.013s	user 0.005s	sys 0.000s
I20260709 12:20:20.156260 17878 tablet_bootstrap.cc:492] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3: Bootstrap starting.
I20260709 12:20:20.157179 17880 tablet_bootstrap.cc:654] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:20.150175 17995 raft_consensus.cc:2749] T 5ad2828ec58f490a9ea564d9805b56b7 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:20.169381 17878 tablet_bootstrap.cc:654] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:20.171708 17881 raft_consensus.cc:359] T 154e6b769a8d476ba26f7fe3a521fc8d P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:20.172513 17881 raft_consensus.cc:385] T 154e6b769a8d476ba26f7fe3a521fc8d P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:20.172856 17881 raft_consensus.cc:740] T 154e6b769a8d476ba26f7fe3a521fc8d P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Initialized, Role: FOLLOWER
I20260709 12:20:20.173705 17881 consensus_queue.cc:260] T 154e6b769a8d476ba26f7fe3a521fc8d P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:20.188638 17881 ts_tablet_manager.cc:1434] T 154e6b769a8d476ba26f7fe3a521fc8d P e063541542a94969898868bf6eb5ce1b: Time spent starting tablet: real 0.029s	user 0.007s	sys 0.002s
I20260709 12:20:20.189296 17878 tablet_bootstrap.cc:492] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3: No bootstrap required, opened a new log
I20260709 12:20:20.189586 17881 tablet_bootstrap.cc:492] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b: Bootstrap starting.
I20260709 12:20:20.189841 17878 ts_tablet_manager.cc:1403] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3: Time spent bootstrapping tablet: real 0.034s	user 0.020s	sys 0.004s
I20260709 12:20:20.195669 17881 tablet_bootstrap.cc:654] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:20.192517 17878 raft_consensus.cc:359] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:20.197379 17878 raft_consensus.cc:385] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:20.197749 17878 raft_consensus.cc:740] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Initialized, Role: FOLLOWER
I20260709 12:20:20.198632 17878 consensus_queue.cc:260] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:20.201879 17878 ts_tablet_manager.cc:1434] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3: Time spent starting tablet: real 0.012s	user 0.005s	sys 0.000s
I20260709 12:20:20.205307 17878 tablet_bootstrap.cc:492] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3: Bootstrap starting.
I20260709 12:20:20.207126 17881 tablet_bootstrap.cc:492] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b: No bootstrap required, opened a new log
I20260709 12:20:20.207592 17881 ts_tablet_manager.cc:1403] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b: Time spent bootstrapping tablet: real 0.018s	user 0.008s	sys 0.008s
I20260709 12:20:20.209945 17949 consensus_queue.cc:1048] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:20:20.210178 17881 raft_consensus.cc:359] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:20.211732 17881 raft_consensus.cc:385] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:20.212041 17881 raft_consensus.cc:740] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Initialized, Role: FOLLOWER
I20260709 12:20:20.212764 17881 consensus_queue.cc:260] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:20.214484 17881 ts_tablet_manager.cc:1434] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b: Time spent starting tablet: real 0.007s	user 0.007s	sys 0.000s
I20260709 12:20:20.215377 17881 tablet_bootstrap.cc:492] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b: Bootstrap starting.
I20260709 12:20:20.221241 17881 tablet_bootstrap.cc:654] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:20.221257 17880 tablet_bootstrap.cc:492] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb: No bootstrap required, opened a new log
I20260709 12:20:20.224771 17880 ts_tablet_manager.cc:1403] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent bootstrapping tablet: real 0.074s	user 0.019s	sys 0.051s
I20260709 12:20:20.227363 17880 raft_consensus.cc:359] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:20.228056 17880 raft_consensus.cc:385] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:20.228315 17880 raft_consensus.cc:740] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Initialized, Role: FOLLOWER
I20260709 12:20:20.228065 17878 tablet_bootstrap.cc:654] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:20.228996 17880 consensus_queue.cc:260] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:20.231413 17880 ts_tablet_manager.cc:1434] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent starting tablet: real 0.006s	user 0.005s	sys 0.000s
I20260709 12:20:20.234372 17881 tablet_bootstrap.cc:492] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b: No bootstrap required, opened a new log
I20260709 12:20:20.234896 17881 ts_tablet_manager.cc:1403] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b: Time spent bootstrapping tablet: real 0.020s	user 0.009s	sys 0.000s
I20260709 12:20:20.236270 17880 tablet_bootstrap.cc:492] T ab4d9071484d4dc0998db462cb01a10f P e2909cff1da44ad387f96a8ae3aa5afb: Bootstrap starting.
I20260709 12:20:20.237879 17881 raft_consensus.cc:359] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:20.238525 17881 raft_consensus.cc:385] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:20.238788 17881 raft_consensus.cc:740] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Initialized, Role: FOLLOWER
I20260709 12:20:20.242357 17880 tablet_bootstrap.cc:654] T ab4d9071484d4dc0998db462cb01a10f P e2909cff1da44ad387f96a8ae3aa5afb: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:20.245077 17878 tablet_bootstrap.cc:492] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3: No bootstrap required, opened a new log
I20260709 12:20:20.245512 17878 ts_tablet_manager.cc:1403] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3: Time spent bootstrapping tablet: real 0.043s	user 0.010s	sys 0.005s
I20260709 12:20:20.245674 17912 raft_consensus.cc:493] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:20.246331 17912 raft_consensus.cc:515] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:20.248170 17878 raft_consensus.cc:359] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:20.248871 17878 raft_consensus.cc:385] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:20.249143 17878 raft_consensus.cc:740] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Initialized, Role: FOLLOWER
I20260709 12:20:20.249766 17878 consensus_queue.cc:260] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:20.251730 17878 ts_tablet_manager.cc:1434] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3: Time spent starting tablet: real 0.006s	user 0.002s	sys 0.003s
I20260709 12:20:20.252627 17878 tablet_bootstrap.cc:492] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3: Bootstrap starting.
I20260709 12:20:20.239377 17881 consensus_queue.cc:260] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:20.254354 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:20:20.254976 17689 raft_consensus.cc:2468] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 0.
I20260709 12:20:20.255275 17880 tablet_bootstrap.cc:492] T ab4d9071484d4dc0998db462cb01a10f P e2909cff1da44ad387f96a8ae3aa5afb: No bootstrap required, opened a new log
I20260709 12:20:20.255825 17880 ts_tablet_manager.cc:1403] T ab4d9071484d4dc0998db462cb01a10f P e2909cff1da44ad387f96a8ae3aa5afb: Time spent bootstrapping tablet: real 0.020s	user 0.008s	sys 0.010s
I20260709 12:20:20.258637 17878 tablet_bootstrap.cc:654] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:20.258438 17880 raft_consensus.cc:359] T ab4d9071484d4dc0998db462cb01a10f P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:20.259331 17880 raft_consensus.cc:385] T ab4d9071484d4dc0998db462cb01a10f P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:20.259673 17880 raft_consensus.cc:740] T ab4d9071484d4dc0998db462cb01a10f P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Initialized, Role: FOLLOWER
I20260709 12:20:20.260350 17880 consensus_queue.cc:260] T ab4d9071484d4dc0998db462cb01a10f P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:20.262166 18055 raft_consensus.cc:493] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:20.262403 17880 ts_tablet_manager.cc:1434] T ab4d9071484d4dc0998db462cb01a10f P e2909cff1da44ad387f96a8ae3aa5afb: Time spent starting tablet: real 0.006s	user 0.005s	sys 0.000s
I20260709 12:20:20.262678 18055 raft_consensus.cc:515] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:20.263505 17880 tablet_bootstrap.cc:492] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb: Bootstrap starting.
I20260709 12:20:20.268980 18055 leader_election.cc:290] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:20.269649 17880 tablet_bootstrap.cc:654] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:20.272926 17912 leader_election.cc:290] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:20:20.274922 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:20:20.277863 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:20:20.275787 17763 raft_consensus.cc:2468] T ab4d9071484d4dc0998db462cb01a10f P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 0.
I20260709 12:20:20.281114 17801 leader_election.cc:304] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:20:20.281983 17880 tablet_bootstrap.cc:492] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb: No bootstrap required, opened a new log
W20260709 12:20:20.282104 17652 leader_election.cc:343] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:20.282564 17880 ts_tablet_manager.cc:1403] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb: Time spent bootstrapping tablet: real 0.019s	user 0.009s	sys 0.008s
I20260709 12:20:20.282919 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:20:20.284337 17950 raft_consensus.cc:2804] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:20.284724 17950 raft_consensus.cc:493] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:20.285059 17950 raft_consensus.cc:3060] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:20.285773 17881 ts_tablet_manager.cc:1434] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b: Time spent starting tablet: real 0.051s	user 0.008s	sys 0.000s
I20260709 12:20:20.285665 17880 raft_consensus.cc:359] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:20.286460 17880 raft_consensus.cc:385] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:20.286808 17880 raft_consensus.cc:740] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Initialized, Role: FOLLOWER
I20260709 12:20:20.287622 17880 consensus_queue.cc:260] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:20.289517 17949 raft_consensus.cc:493] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:20.290000 17949 raft_consensus.cc:515] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:20.293469 17880 ts_tablet_manager.cc:1434] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb: Time spent starting tablet: real 0.011s	user 0.005s	sys 0.000s
I20260709 12:20:20.293759 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:20:20.294348 17763 raft_consensus.cc:2468] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 0.
I20260709 12:20:20.294545 17880 tablet_bootstrap.cc:492] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb: Bootstrap starting.
I20260709 12:20:20.295871 17801 leader_election.cc:304] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:20:20.297394 17951 raft_consensus.cc:2804] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:20.297767 17951 raft_consensus.cc:493] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:20.298110 17951 raft_consensus.cc:3060] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:20.299047 17881 tablet_bootstrap.cc:492] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b: Bootstrap starting.
W20260709 12:20:20.296172 17652 leader_election.cc:343] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:20.304113 17652 leader_election.cc:304] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:20:20.305889 17881 tablet_bootstrap.cc:654] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:20.306389 17880 tablet_bootstrap.cc:654] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:20.306794 17951 raft_consensus.cc:515] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:20.307900 18055 raft_consensus.cc:2749] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:20.309262 17951 leader_election.cc:290] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 1 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:20:20.318002 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:20:20.318331 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:20:20.318683 17689 raft_consensus.cc:2468] T 1a4040191930435786e2eeea338bb3d3 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 0.
I20260709 12:20:20.319252 18005 raft_consensus.cc:493] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:20.319868 18005 raft_consensus.cc:515] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:20.320847 17880 tablet_bootstrap.cc:492] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb: No bootstrap required, opened a new log
I20260709 12:20:20.321338 17880 ts_tablet_manager.cc:1403] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb: Time spent bootstrapping tablet: real 0.027s	user 0.016s	sys 0.003s
I20260709 12:20:20.321117 17952 consensus_queue.cc:1048] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:20:20.322002 17949 leader_election.cc:290] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:20:20.323086 18005 leader_election.cc:290] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:20.324054 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:20:20.324685 17762 raft_consensus.cc:2468] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 0.
I20260709 12:20:20.324375 17880 raft_consensus.cc:359] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:20.325230 17880 raft_consensus.cc:385] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:20.325551 17880 raft_consensus.cc:740] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Initialized, Role: FOLLOWER
I20260709 12:20:20.326836 17950 raft_consensus.cc:515] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:20.326799 17880 consensus_queue.cc:260] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:20.327317 17652 leader_election.cc:304] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:20:20.328250 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:20:20.328868 17835 raft_consensus.cc:2468] T 154e6b769a8d476ba26f7fe3a521fc8d P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 0.
I20260709 12:20:20.331014 18005 raft_consensus.cc:2804] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:20.331393 18005 raft_consensus.cc:493] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:20.331475 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:20:20.331776 18005 raft_consensus.cc:3060] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:20.332213 17763 raft_consensus.cc:3060] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:20.339820 18005 raft_consensus.cc:515] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:20.340328 17880 ts_tablet_manager.cc:1434] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb: Time spent starting tablet: real 0.019s	user 0.007s	sys 0.000s
I20260709 12:20:20.346172 17880 tablet_bootstrap.cc:492] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb: Bootstrap starting.
I20260709 12:20:20.346393 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:20:20.347177 17764 raft_consensus.cc:3060] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:20.349491 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:20:20.350117 17835 raft_consensus.cc:3060] T 154e6b769a8d476ba26f7fe3a521fc8d P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:20.350680 18005 leader_election.cc:290] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:20.354950 17880 tablet_bootstrap.cc:654] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:20.361042 17764 raft_consensus.cc:2468] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 1.
I20260709 12:20:20.362545 17652 leader_election.cc:304] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:20:20.363577 18005 raft_consensus.cc:2804] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:20.364092 18005 raft_consensus.cc:697] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 1 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:20:20.364928 18005 consensus_queue.cc:237] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:20.366142 17881 tablet_bootstrap.cc:492] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b: No bootstrap required, opened a new log
I20260709 12:20:20.366616 17881 ts_tablet_manager.cc:1403] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b: Time spent bootstrapping tablet: real 0.068s	user 0.008s	sys 0.003s
I20260709 12:20:20.367077 17880 tablet_bootstrap.cc:492] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb: No bootstrap required, opened a new log
I20260709 12:20:20.367628 17880 ts_tablet_manager.cc:1403] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb: Time spent bootstrapping tablet: real 0.022s	user 0.013s	sys 0.003s
I20260709 12:20:20.370867 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:20:20.370831 17880 raft_consensus.cc:359] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:20.371829 17880 raft_consensus.cc:385] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:20.371709 17689 raft_consensus.cc:3060] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:20.372303 17880 raft_consensus.cc:740] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Initialized, Role: FOLLOWER
I20260709 12:20:20.373131 17880 consensus_queue.cc:260] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:20.376333 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:20:20.377074 17761 raft_consensus.cc:3060] T ab4d9071484d4dc0998db462cb01a10f P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:20.377888 17763 raft_consensus.cc:2468] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 1.
I20260709 12:20:20.378978 17880 ts_tablet_manager.cc:1434] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb: Time spent starting tablet: real 0.011s	user 0.006s	sys 0.000s
I20260709 12:20:20.379083 17801 leader_election.cc:304] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:20:20.381325 17880 tablet_bootstrap.cc:492] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb: Bootstrap starting.
I20260709 12:20:20.383332 17587 catalog_manager.cc:5697] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0da169ff82674dc1947055ce43de68a3 (127.16.29.65). New cstate: current_term: 1 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:20.380149 17835 raft_consensus.cc:2468] T 154e6b769a8d476ba26f7fe3a521fc8d P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 1.
I20260709 12:20:20.387974 17880 tablet_bootstrap.cc:654] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:20.376374 17950 leader_election.cc:290] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 1 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:20:20.394910 17761 raft_consensus.cc:2468] T ab4d9071484d4dc0998db462cb01a10f P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 1.
I20260709 12:20:20.396941 17881 raft_consensus.cc:359] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:20.397841 17881 raft_consensus.cc:385] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:20.398212 17881 raft_consensus.cc:740] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Initialized, Role: FOLLOWER
I20260709 12:20:20.400776 17880 tablet_bootstrap.cc:492] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb: No bootstrap required, opened a new log
I20260709 12:20:20.401353 17880 ts_tablet_manager.cc:1403] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent bootstrapping tablet: real 0.020s	user 0.014s	sys 0.004s
I20260709 12:20:20.402271 17689 raft_consensus.cc:2468] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 1.
I20260709 12:20:20.404194 17801 leader_election.cc:304] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:20:20.405197 17949 raft_consensus.cc:2804] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:20.405272 17951 raft_consensus.cc:2804] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:20.405820 17949 raft_consensus.cc:697] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 1 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:20:20.405900 17951 raft_consensus.cc:697] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 1 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:20:20.406883 17951 consensus_queue.cc:237] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:20.406958 17949 consensus_queue.cc:237] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:20.409215 18005 consensus_queue.cc:1048] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:20:20.399112 17881 consensus_queue.cc:260] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:20.416916 17881 ts_tablet_manager.cc:1434] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b: Time spent starting tablet: real 0.050s	user 0.009s	sys 0.000s
I20260709 12:20:20.418051 17881 tablet_bootstrap.cc:492] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b: Bootstrap starting.
I20260709 12:20:20.417502 18036 consensus_queue.cc:1048] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:20.427167 17880 raft_consensus.cc:359] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:20.428081 17880 raft_consensus.cc:385] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:20.430701 17880 raft_consensus.cc:740] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Initialized, Role: FOLLOWER
I20260709 12:20:20.431686 17880 consensus_queue.cc:260] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:20.436918 17878 tablet_bootstrap.cc:492] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3: No bootstrap required, opened a new log
I20260709 12:20:20.437399 17878 ts_tablet_manager.cc:1403] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3: Time spent bootstrapping tablet: real 0.185s	user 0.068s	sys 0.031s
I20260709 12:20:20.439515 17995 consensus_queue.cc:1048] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:20:20.440483 17878 raft_consensus.cc:359] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:20.441177 17878 raft_consensus.cc:385] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:20.441440 17878 raft_consensus.cc:740] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Initialized, Role: FOLLOWER
I20260709 12:20:20.442047 17878 consensus_queue.cc:260] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:20.443447 17878 ts_tablet_manager.cc:1434] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3: Time spent starting tablet: real 0.006s	user 0.004s	sys 0.000s
I20260709 12:20:20.443768 17881 tablet_bootstrap.cc:654] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:20.444347 17878 tablet_bootstrap.cc:492] T 0540d79e2ec2406588b295145aa6d54d P 0da169ff82674dc1947055ce43de68a3: Bootstrap starting.
I20260709 12:20:20.451267 17878 tablet_bootstrap.cc:654] T 0540d79e2ec2406588b295145aa6d54d P 0da169ff82674dc1947055ce43de68a3: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:20.454018 18055 consensus_queue.cc:1048] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:20:20.460959 18055 raft_consensus.cc:493] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:20.461432 18055 raft_consensus.cc:515] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:20.465687 18055 leader_election.cc:290] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:20.467780 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:20:20.469795 17652 leader_election.cc:343] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:20.474277 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:20:20.475227 17763 raft_consensus.cc:2468] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 0.
I20260709 12:20:20.476775 17880 ts_tablet_manager.cc:1434] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent starting tablet: real 0.075s	user 0.007s	sys 0.000s
I20260709 12:20:20.483047 17880 tablet_bootstrap.cc:492] T 6fcb5bdaa4be4d619bb945ffd44090ba P e2909cff1da44ad387f96a8ae3aa5afb: Bootstrap starting.
I20260709 12:20:20.485069 17652 leader_election.cc:304] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:20:20.475459 17585 catalog_manager.cc:5697] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b reported cstate change: term changed from 0 to 1, leader changed from <none> to e063541542a94969898868bf6eb5ce1b (127.16.29.67). New cstate: current_term: 1 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: HEALTHY } } }
I20260709 12:20:20.486610 18055 raft_consensus.cc:2804] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:20.487144 18055 raft_consensus.cc:493] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:20.487552 18055 raft_consensus.cc:3060] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:20.486776 17585 catalog_manager.cc:5697] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b reported cstate change: term changed from 0 to 1, leader changed from <none> to e063541542a94969898868bf6eb5ce1b (127.16.29.67). New cstate: current_term: 1 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: HEALTHY } } }
I20260709 12:20:20.493218 17951 consensus_queue.cc:1048] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:20.498575 17880 tablet_bootstrap.cc:654] T 6fcb5bdaa4be4d619bb945ffd44090ba P e2909cff1da44ad387f96a8ae3aa5afb: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:20.514252 18055 raft_consensus.cc:515] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:20.539790 17935 raft_consensus.cc:493] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:20.540756 17935 raft_consensus.cc:515] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:20.544027 17935 leader_election.cc:290] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:20.546653 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:20:20.550570 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:20:20.551540 17834 raft_consensus.cc:2393] T ea7da467817a4ba88aa5797c3f8f493f P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 1: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:20:20.552544 17727 leader_election.cc:304] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:20:20.553709 17995 raft_consensus.cc:3060] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:20.555334 17880 tablet_bootstrap.cc:492] T 6fcb5bdaa4be4d619bb945ffd44090ba P e2909cff1da44ad387f96a8ae3aa5afb: No bootstrap required, opened a new log
I20260709 12:20:20.556053 17880 ts_tablet_manager.cc:1403] T 6fcb5bdaa4be4d619bb945ffd44090ba P e2909cff1da44ad387f96a8ae3aa5afb: Time spent bootstrapping tablet: real 0.073s	user 0.014s	sys 0.006s
I20260709 12:20:20.558662 17880 raft_consensus.cc:359] T 6fcb5bdaa4be4d619bb945ffd44090ba P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:20.559551 17880 raft_consensus.cc:385] T 6fcb5bdaa4be4d619bb945ffd44090ba P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:20.560128 17880 raft_consensus.cc:740] T 6fcb5bdaa4be4d619bb945ffd44090ba P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Initialized, Role: FOLLOWER
I20260709 12:20:20.560959 17880 consensus_queue.cc:260] T 6fcb5bdaa4be4d619bb945ffd44090ba P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:20.563738 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:20:20.564431 17764 raft_consensus.cc:3060] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:20.565876 17880 ts_tablet_manager.cc:1434] T 6fcb5bdaa4be4d619bb945ffd44090ba P e2909cff1da44ad387f96a8ae3aa5afb: Time spent starting tablet: real 0.009s	user 0.005s	sys 0.001s
I20260709 12:20:20.566851 17995 raft_consensus.cc:2749] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:20.563980 18133 raft_consensus.cc:493] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:20.567795 18133 raft_consensus.cc:515] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:20.570262 18005 consensus_queue.cc:1048] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:20.571359 17880 tablet_bootstrap.cc:492] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb: Bootstrap starting.
I20260709 12:20:20.585278 17834 raft_consensus.cc:3060] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:20.589069 17952 consensus_queue.cc:1048] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:20:20.589687 17881 tablet_bootstrap.cc:492] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b: No bootstrap required, opened a new log
I20260709 12:20:20.590301 17881 ts_tablet_manager.cc:1403] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b: Time spent bootstrapping tablet: real 0.172s	user 0.016s	sys 0.000s
I20260709 12:20:20.598819 17878 tablet_bootstrap.cc:492] T 0540d79e2ec2406588b295145aa6d54d P 0da169ff82674dc1947055ce43de68a3: No bootstrap required, opened a new log
I20260709 12:20:20.599329 17878 ts_tablet_manager.cc:1403] T 0540d79e2ec2406588b295145aa6d54d P 0da169ff82674dc1947055ce43de68a3: Time spent bootstrapping tablet: real 0.155s	user 0.034s	sys 0.047s
I20260709 12:20:20.601891 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:20:20.602779 18055 leader_election.cc:290] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:20.602835 17878 raft_consensus.cc:359] T 0540d79e2ec2406588b295145aa6d54d P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:20.603713 17878 raft_consensus.cc:385] T 0540d79e2ec2406588b295145aa6d54d P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:20.604039 17878 raft_consensus.cc:740] T 0540d79e2ec2406588b295145aa6d54d P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Initialized, Role: FOLLOWER
I20260709 12:20:20.604674 17878 consensus_queue.cc:260] T 0540d79e2ec2406588b295145aa6d54d P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:20.605175 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:20:20.606773 17878 ts_tablet_manager.cc:1434] T 0540d79e2ec2406588b295145aa6d54d P 0da169ff82674dc1947055ce43de68a3: Time spent starting tablet: real 0.007s	user 0.006s	sys 0.000s
I20260709 12:20:20.607208 17764 raft_consensus.cc:2468] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 1.
I20260709 12:20:20.607645 17878 tablet_bootstrap.cc:492] T 8cab6f01c6a2485cb5024322a1bd5462 P 0da169ff82674dc1947055ce43de68a3: Bootstrap starting.
W20260709 12:20:20.608088 17652 leader_election.cc:343] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:20.609283 18133 leader_election.cc:290] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:20:20.602921 17687 raft_consensus.cc:2393] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 1: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:20:20.613176 17880 tablet_bootstrap.cc:654] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:20.616671 17652 leader_election.cc:304] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:20:20.618572 18055 raft_consensus.cc:2804] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:20.619017 18055 raft_consensus.cc:697] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 1 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:20:20.619827 18055 consensus_queue.cc:237] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:20.621706 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:20:20.630117 18133 raft_consensus.cc:493] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:20.642213 18133 raft_consensus.cc:515] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:20.715538 17878 tablet_bootstrap.cc:654] T 8cab6f01c6a2485cb5024322a1bd5462 P 0da169ff82674dc1947055ce43de68a3: Neither blocks nor log segments found. Creating new log.
W20260709 12:20:20.716166 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 103447us
I20260709 12:20:20.718231 18133 leader_election.cc:290] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:20:20.726168 17878 tablet_bootstrap.cc:492] T 8cab6f01c6a2485cb5024322a1bd5462 P 0da169ff82674dc1947055ce43de68a3: No bootstrap required, opened a new log
I20260709 12:20:20.726284 17880 tablet_bootstrap.cc:492] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb: No bootstrap required, opened a new log
I20260709 12:20:20.726673 17878 ts_tablet_manager.cc:1403] T 8cab6f01c6a2485cb5024322a1bd5462 P 0da169ff82674dc1947055ce43de68a3: Time spent bootstrapping tablet: real 0.119s	user 0.030s	sys 0.023s
I20260709 12:20:20.724495 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:20:20.726861 17880 ts_tablet_manager.cc:1403] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent bootstrapping tablet: real 0.156s	user 0.022s	sys 0.006s
I20260709 12:20:20.727172 17686 raft_consensus.cc:2468] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 0.
I20260709 12:20:20.727900 18055 consensus_queue.cc:1048] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:20:20.729460 17802 leader_election.cc:304] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:20:20.730990 17912 raft_consensus.cc:2804] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:20.731345 17912 raft_consensus.cc:493] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:20.731704 17912 raft_consensus.cc:3060] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:20.733193 17880 raft_consensus.cc:359] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:20.733951 17880 raft_consensus.cc:385] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:20.734297 17880 raft_consensus.cc:740] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Initialized, Role: FOLLOWER
I20260709 12:20:20.735406 17881 raft_consensus.cc:359] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:20.736021 17881 raft_consensus.cc:385] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:20.736279 17881 raft_consensus.cc:740] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Initialized, Role: FOLLOWER
I20260709 12:20:20.737054 17881 consensus_queue.cc:260] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:20.740458 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:20:20.741178 17763 raft_consensus.cc:2468] T 6fcb5bdaa4be4d619bb945ffd44090ba P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 0.
I20260709 12:20:20.746044 17912 raft_consensus.cc:515] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:20.749577 17880 consensus_queue.cc:260] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:20.748914 17878 raft_consensus.cc:359] T 8cab6f01c6a2485cb5024322a1bd5462 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:20.751773 17878 raft_consensus.cc:385] T 8cab6f01c6a2485cb5024322a1bd5462 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:20.752131 17878 raft_consensus.cc:740] T 8cab6f01c6a2485cb5024322a1bd5462 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Initialized, Role: FOLLOWER
I20260709 12:20:20.754796 17881 ts_tablet_manager.cc:1434] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b: Time spent starting tablet: real 0.164s	user 0.007s	sys 0.016s
I20260709 12:20:20.750593 17995 raft_consensus.cc:493] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:20.756250 17881 tablet_bootstrap.cc:492] T ac984176358d453a8800cc9e0e304061 P e063541542a94969898868bf6eb5ce1b: Bootstrap starting.
I20260709 12:20:20.758529 17878 consensus_queue.cc:260] T 8cab6f01c6a2485cb5024322a1bd5462 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:20.757601 17995 raft_consensus.cc:515] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:20.761950 17995 leader_election.cc:290] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:20.765156 17995 consensus_queue.cc:1048] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:20:20.764724 17878 ts_tablet_manager.cc:1434] T 8cab6f01c6a2485cb5024322a1bd5462 P 0da169ff82674dc1947055ce43de68a3: Time spent starting tablet: real 0.036s	user 0.006s	sys 0.000s
I20260709 12:20:20.770771 17878 tablet_bootstrap.cc:492] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3: Bootstrap starting.
I20260709 12:20:20.769564 18124 consensus_queue.cc:1048] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:20:20.772697 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:20:20.776391 18131 raft_consensus.cc:493] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:20.776877 17881 tablet_bootstrap.cc:654] T ac984176358d453a8800cc9e0e304061 P e063541542a94969898868bf6eb5ce1b: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:20.776937 18131 raft_consensus.cc:515] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:20.758134 17801 leader_election.cc:304] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:20:20.786796 17727 leader_election.cc:343] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:20.794314 18131 leader_election.cc:290] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:20:20.805095 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 49796.2us
I20260709 12:20:20.808410 17912 leader_election.cc:290] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 1 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:20:20.809868 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:20:20.810580 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:20:20.810734 17689 raft_consensus.cc:3060] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:20.811259 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:20:20.817492 17802 leader_election.cc:343] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:20.824136 17880 ts_tablet_manager.cc:1434] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent starting tablet: real 0.097s	user 0.014s	sys 0.003s
I20260709 12:20:20.823753 17584 catalog_manager.cc:5697] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0da169ff82674dc1947055ce43de68a3 (127.16.29.65). New cstate: current_term: 1 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:20.825397 17880 tablet_bootstrap.cc:492] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb: Bootstrap starting.
I20260709 12:20:20.790848 17935 consensus_queue.cc:1048] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.038s
W20260709 12:20:20.827812 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 45013.9us
I20260709 12:20:20.847376 17950 raft_consensus.cc:2749] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:20.859833 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:20:20.867302 17801 leader_election.cc:343] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:20.872598 17688 raft_consensus.cc:2468] T 0540d79e2ec2406588b295145aa6d54d P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 0.
I20260709 12:20:20.876271 17935 raft_consensus.cc:493] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:20.881875 17935 raft_consensus.cc:515] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:20.884049 17935 leader_election.cc:290] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:20.887686 17687 raft_consensus.cc:3060] T 1a4040191930435786e2eeea338bb3d3 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:20.889647 18005 consensus_queue.cc:1048] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:20.893005 17881 tablet_bootstrap.cc:492] T ac984176358d453a8800cc9e0e304061 P e063541542a94969898868bf6eb5ce1b: No bootstrap required, opened a new log
I20260709 12:20:20.893522 17881 ts_tablet_manager.cc:1403] T ac984176358d453a8800cc9e0e304061 P e063541542a94969898868bf6eb5ce1b: Time spent bootstrapping tablet: real 0.138s	user 0.015s	sys 0.010s
I20260709 12:20:20.874512 17801 leader_election.cc:304] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:20:20.895340 18131 raft_consensus.cc:2749] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:20.896193 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:20:20.896942 17760 raft_consensus.cc:3060] T 6fcb5bdaa4be4d619bb945ffd44090ba P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:20.907202 18131 consensus_queue.cc:1048] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:20:20.908115 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:20:20.909746 17689 raft_consensus.cc:2468] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 1.
I20260709 12:20:20.921237 17728 leader_election.cc:304] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:20:20.922187 17881 raft_consensus.cc:359] T ac984176358d453a8800cc9e0e304061 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:20.922921 17881 raft_consensus.cc:385] T ac984176358d453a8800cc9e0e304061 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:20.923122 17935 raft_consensus.cc:2804] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:20.923199 17881 raft_consensus.cc:740] T ac984176358d453a8800cc9e0e304061 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Initialized, Role: FOLLOWER
I20260709 12:20:20.923610 17935 raft_consensus.cc:493] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:20.923943 18124 raft_consensus.cc:493] T 8cab6f01c6a2485cb5024322a1bd5462 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:20.924028 17935 raft_consensus.cc:3060] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:20.924448 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:20:20.924449 18124 raft_consensus.cc:515] T 8cab6f01c6a2485cb5024322a1bd5462 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:20.926132 17802 leader_election.cc:304] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:20:20.924075 18005 raft_consensus.cc:493] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:20.926838 18131 consensus_queue.cc:1048] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:20.927277 18005 raft_consensus.cc:515] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:20.928350 18131 raft_consensus.cc:2804] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:20.928813 18131 raft_consensus.cc:697] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 1 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:20:20.929279 17881 consensus_queue.cc:260] T ac984176358d453a8800cc9e0e304061 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:20.929970 18005 leader_election.cc:290] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:20.929620 18131 consensus_queue.cc:237] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:20:20.930147 17728 leader_election.cc:343] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:20:20.930724 17727 leader_election.cc:343] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:20.931152 17727 leader_election.cc:304] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:20:20.931860 18036 raft_consensus.cc:2749] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:20.941285 17880 tablet_bootstrap.cc:654] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:20.945739 17881 ts_tablet_manager.cc:1434] T ac984176358d453a8800cc9e0e304061 P e063541542a94969898868bf6eb5ce1b: Time spent starting tablet: real 0.052s	user 0.007s	sys 0.002s
I20260709 12:20:20.946697 17881 tablet_bootstrap.cc:492] T 69716f6d5e7549418fdd8d54d7b722ce P e063541542a94969898868bf6eb5ce1b: Bootstrap starting.
I20260709 12:20:20.947346 18124 leader_election.cc:290] T 8cab6f01c6a2485cb5024322a1bd5462 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:20.948230 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:20:20.947261 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:20:20.949064 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:20:20.950793 17652 leader_election.cc:343] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:20:20.951689 17652 leader_election.cc:343] T 8cab6f01c6a2485cb5024322a1bd5462 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:20.952584 17878 tablet_bootstrap.cc:654] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:20.956000 17760 raft_consensus.cc:2468] T 6fcb5bdaa4be4d619bb945ffd44090ba P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 1.
I20260709 12:20:20.957365 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:20:20.959931 17834 raft_consensus.cc:2468] T ac984176358d453a8800cc9e0e304061 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 0.
I20260709 12:20:20.962255 17652 leader_election.cc:304] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:20:20.963498 17652 leader_election.cc:343] T 8cab6f01c6a2485cb5024322a1bd5462 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:20.963963 17652 leader_election.cc:304] T 8cab6f01c6a2485cb5024322a1bd5462 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:20:20.968724 17635 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:20:21.006703 18124 raft_consensus.cc:2804] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:21.008968 18124 raft_consensus.cc:493] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:21.009378 18124 raft_consensus.cc:3060] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:21.001137 17935 raft_consensus.cc:515] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:21.013226 17881 tablet_bootstrap.cc:654] T 69716f6d5e7549418fdd8d54d7b722ce P e063541542a94969898868bf6eb5ce1b: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:21.017045 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:20:21.017632 17688 raft_consensus.cc:3060] T 0540d79e2ec2406588b295145aa6d54d P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:21.020398 18122 raft_consensus.cc:2749] T 8cab6f01c6a2485cb5024322a1bd5462 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260709 12:20:21.023362 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 59893.4us
I20260709 12:20:21.023567 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
W20260709 12:20:21.025110 17727 leader_election.cc:343] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:21.026358 17935 leader_election.cc:290] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 1 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:20:21.028869 17652 consensus_peers.cc:597] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 -> Peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837): Couldn't send request to peer e063541542a94969898868bf6eb5ce1b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:21.030592 17688 raft_consensus.cc:2468] T 0540d79e2ec2406588b295145aa6d54d P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 1.
I20260709 12:20:21.035607 17728 leader_election.cc:304] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:20:21.045794 17935 raft_consensus.cc:2804] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:21.046413 17935 raft_consensus.cc:697] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 1 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:20:21.047230 17935 consensus_queue.cc:237] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:21.057999 18124 raft_consensus.cc:515] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:21.062378 18133 consensus_queue.cc:1048] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:21.065585 18131 consensus_queue.cc:1048] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:20:21.072829 17880 tablet_bootstrap.cc:492] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb: No bootstrap required, opened a new log
I20260709 12:20:21.073328 17880 ts_tablet_manager.cc:1403] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent bootstrapping tablet: real 0.248s	user 0.015s	sys 0.008s
I20260709 12:20:21.076851 17587 catalog_manager.cc:5697] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b reported cstate change: term changed from 0 to 1, leader changed from <none> to e063541542a94969898868bf6eb5ce1b (127.16.29.67). New cstate: current_term: 1 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: HEALTHY } } }
I20260709 12:20:21.076146 17584 catalog_manager.cc:5697] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb reported cstate change: term changed from 0 to 1, leader changed from <none> to e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66). New cstate: current_term: 1 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:21.079737 17880 raft_consensus.cc:359] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:21.080410 17880 raft_consensus.cc:385] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:21.080677 17880 raft_consensus.cc:740] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Initialized, Role: FOLLOWER
I20260709 12:20:21.081322 17880 consensus_queue.cc:260] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:21.083416 17880 ts_tablet_manager.cc:1434] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent starting tablet: real 0.010s	user 0.004s	sys 0.000s
I20260709 12:20:21.085408 17880 tablet_bootstrap.cc:492] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb: Bootstrap starting.
I20260709 12:20:21.088376 18124 leader_election.cc:290] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:21.103708 18036 raft_consensus.cc:493] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:21.111341 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:20:21.155772 18036 raft_consensus.cc:515] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:21.159159 17878 tablet_bootstrap.cc:492] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3: No bootstrap required, opened a new log
I20260709 12:20:21.159629 17878 ts_tablet_manager.cc:1403] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3: Time spent bootstrapping tablet: real 0.389s	user 0.021s	sys 0.006s
I20260709 12:20:21.108094 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:20:21.165122 17878 raft_consensus.cc:359] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:20:21.166208 17652 leader_election.cc:343] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:21.166214 17878 raft_consensus.cc:385] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:21.167061 17878 raft_consensus.cc:740] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Initialized, Role: FOLLOWER
I20260709 12:20:21.167620 17878 consensus_queue.cc:260] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:21.169642 17878 ts_tablet_manager.cc:1434] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3: Time spent starting tablet: real 0.010s	user 0.006s	sys 0.000s
I20260709 12:20:21.169736 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:20:21.170523 17834 raft_consensus.cc:3060] T ac984176358d453a8800cc9e0e304061 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:20:21.171553 17728 leader_election.cc:343] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:21.177757 17881 tablet_bootstrap.cc:492] T 69716f6d5e7549418fdd8d54d7b722ce P e063541542a94969898868bf6eb5ce1b: No bootstrap required, opened a new log
I20260709 12:20:21.178227 17881 ts_tablet_manager.cc:1403] T 69716f6d5e7549418fdd8d54d7b722ce P e063541542a94969898868bf6eb5ce1b: Time spent bootstrapping tablet: real 0.232s	user 0.010s	sys 0.011s
I20260709 12:20:21.180858 18036 leader_election.cc:290] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:21.180766 17881 raft_consensus.cc:359] T 69716f6d5e7549418fdd8d54d7b722ce P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:21.181264 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:20:21.181463 17881 raft_consensus.cc:385] T 69716f6d5e7549418fdd8d54d7b722ce P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:21.181895 17833 raft_consensus.cc:2468] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 0.
I20260709 12:20:21.182261 17881 raft_consensus.cc:740] T 69716f6d5e7549418fdd8d54d7b722ce P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Initialized, Role: FOLLOWER
I20260709 12:20:21.183029 17881 consensus_queue.cc:260] T 69716f6d5e7549418fdd8d54d7b722ce P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:21.183413 17834 raft_consensus.cc:2468] T ac984176358d453a8800cc9e0e304061 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 1.
I20260709 12:20:21.183753 17727 leader_election.cc:304] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:20:21.184800 17652 leader_election.cc:304] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:20:21.185562 18124 raft_consensus.cc:2804] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:21.186028 18124 raft_consensus.cc:697] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 1 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:20:21.186316 18036 raft_consensus.cc:2804] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:21.186690 18036 raft_consensus.cc:493] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:21.187028 18036 raft_consensus.cc:3060] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:21.186786 18124 consensus_queue.cc:237] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:21.187417 17881 ts_tablet_manager.cc:1434] T 69716f6d5e7549418fdd8d54d7b722ce P e063541542a94969898868bf6eb5ce1b: Time spent starting tablet: real 0.009s	user 0.006s	sys 0.000s
I20260709 12:20:21.188274 17881 tablet_bootstrap.cc:492] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b: Bootstrap starting.
I20260709 12:20:21.192714 17880 tablet_bootstrap.cc:654] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:21.193245 17878 tablet_bootstrap.cc:492] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3: Bootstrap starting.
I20260709 12:20:21.203245 18036 raft_consensus.cc:515] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:21.206462 17881 tablet_bootstrap.cc:654] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:21.210799 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
W20260709 12:20:21.212217 17728 leader_election.cc:343] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:21.213694 18036 leader_election.cc:290] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 1 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:21.214401 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:20:21.214927 17834 raft_consensus.cc:3060] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:21.215701 17878 tablet_bootstrap.cc:654] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:21.219244 17881 tablet_bootstrap.cc:492] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b: No bootstrap required, opened a new log
I20260709 12:20:21.219678 17881 ts_tablet_manager.cc:1403] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b: Time spent bootstrapping tablet: real 0.032s	user 0.012s	sys 0.000s
I20260709 12:20:21.220417 17834 raft_consensus.cc:2468] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 1.
I20260709 12:20:21.221606 17727 leader_election.cc:304] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:20:21.222311 17881 raft_consensus.cc:359] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:21.222710 18036 raft_consensus.cc:2804] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:21.222999 17881 raft_consensus.cc:385] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:21.223063 17880 tablet_bootstrap.cc:492] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb: No bootstrap required, opened a new log
I20260709 12:20:21.223265 18036 raft_consensus.cc:697] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:20:21.223388 17881 raft_consensus.cc:740] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Initialized, Role: FOLLOWER
I20260709 12:20:21.223903 17880 ts_tablet_manager.cc:1403] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent bootstrapping tablet: real 0.139s	user 0.008s	sys 0.004s
I20260709 12:20:21.224265 17881 consensus_queue.cc:260] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:21.224246 18036 consensus_queue.cc:237] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:21.226414 17881 ts_tablet_manager.cc:1434] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b: Time spent starting tablet: real 0.006s	user 0.004s	sys 0.000s
I20260709 12:20:21.226387 17880 raft_consensus.cc:359] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:21.227165 17880 raft_consensus.cc:385] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:21.227411 17880 raft_consensus.cc:740] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Initialized, Role: FOLLOWER
I20260709 12:20:21.228080 17880 consensus_queue.cc:260] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:21.229684 17881 tablet_bootstrap.cc:492] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b: Bootstrap starting.
I20260709 12:20:21.230803 17878 tablet_bootstrap.cc:492] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3: No bootstrap required, opened a new log
I20260709 12:20:21.231231 17878 ts_tablet_manager.cc:1403] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3: Time spent bootstrapping tablet: real 0.038s	user 0.010s	sys 0.000s
I20260709 12:20:21.235823 17881 tablet_bootstrap.cc:654] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:21.247923 17880 ts_tablet_manager.cc:1434] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent starting tablet: real 0.024s	user 0.002s	sys 0.003s
I20260709 12:20:21.249297 17880 tablet_bootstrap.cc:492] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb: Bootstrap starting.
I20260709 12:20:21.248941 17584 catalog_manager.cc:5697] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0da169ff82674dc1947055ce43de68a3 (127.16.29.65). New cstate: current_term: 1 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:21.249926 18124 consensus_queue.cc:1048] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:20:21.252202 17878 raft_consensus.cc:359] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:21.253021 17878 raft_consensus.cc:385] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:21.253283 17878 raft_consensus.cc:740] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Initialized, Role: FOLLOWER
I20260709 12:20:21.253954 17878 consensus_queue.cc:260] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:21.256176 17878 ts_tablet_manager.cc:1434] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3: Time spent starting tablet: real 0.025s	user 0.008s	sys 0.000s
I20260709 12:20:21.257012 17878 tablet_bootstrap.cc:492] T f5f0dd0890f44e35bbb838da838faa9c P 0da169ff82674dc1947055ce43de68a3: Bootstrap starting.
I20260709 12:20:21.265949 17935 raft_consensus.cc:493] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:21.266538 17935 raft_consensus.cc:515] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:21.273041 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:20:21.273247 17878 tablet_bootstrap.cc:654] T f5f0dd0890f44e35bbb838da838faa9c P 0da169ff82674dc1947055ce43de68a3: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:21.277025 17880 tablet_bootstrap.cc:654] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb: Neither blocks nor log segments found. Creating new log.
W20260709 12:20:21.279732 17728 leader_election.cc:343] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:21.283321 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:20:21.285601 17727 leader_election.cc:343] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:21.286257 17727 leader_election.cc:304] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:20:21.287230 17995 raft_consensus.cc:2749] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:21.290812 18005 raft_consensus.cc:493] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:21.297003 18005 raft_consensus.cc:515] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:21.298089 17935 leader_election.cc:290] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:21.301050 17995 raft_consensus.cc:493] T 3219dcbfd86340ecbde507517060ff81 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:21.309728 17995 raft_consensus.cc:515] T 3219dcbfd86340ecbde507517060ff81 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:21.311966 17995 leader_election.cc:290] T 3219dcbfd86340ecbde507517060ff81 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:21.312726 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:20:21.313419 17759 raft_consensus.cc:2468] T 3219dcbfd86340ecbde507517060ff81 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 1.
I20260709 12:20:21.314195 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:20:21.314845 17833 raft_consensus.cc:2468] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 1.
I20260709 12:20:21.317900 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:20:21.318724 17833 raft_consensus.cc:2468] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 1.
I20260709 12:20:21.320772 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:20:21.321327 17688 raft_consensus.cc:2468] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 1.
I20260709 12:20:21.322392 17727 leader_election.cc:304] T 3219dcbfd86340ecbde507517060ff81 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:20:21.323266 17995 raft_consensus.cc:2804] T 3219dcbfd86340ecbde507517060ff81 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:20:21.323611 17995 raft_consensus.cc:493] T 3219dcbfd86340ecbde507517060ff81 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:21.323945 17995 raft_consensus.cc:3060] T 3219dcbfd86340ecbde507517060ff81 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:21.324527 17652 leader_election.cc:304] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:20:21.325459 18055 raft_consensus.cc:2804] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:20:21.325929 18055 raft_consensus.cc:493] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:21.326341 18055 raft_consensus.cc:3060] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:21.327353 18005 leader_election.cc:290] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:21.332192 18131 raft_consensus.cc:493] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:21.333709 18131 raft_consensus.cc:515] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:21.335651 18131 leader_election.cc:290] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:20:21.336839 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:20:21.338786 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:20:21.343255 17688 raft_consensus.cc:2393] T 0540d79e2ec2406588b295145aa6d54d P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 1: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:20:21.343976 17881 tablet_bootstrap.cc:492] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b: No bootstrap required, opened a new log
I20260709 12:20:21.344463 17881 ts_tablet_manager.cc:1403] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b: Time spent bootstrapping tablet: real 0.115s	user 0.025s	sys 0.039s
I20260709 12:20:21.344421 17802 leader_election.cc:304] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:20:21.345903 18130 raft_consensus.cc:3060] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:21.342453 17584 catalog_manager.cc:5697] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb reported cstate change: term changed from 0 to 1, leader changed from <none> to e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66). New cstate: current_term: 1 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:21.349608 18055 raft_consensus.cc:515] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:21.352469 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:20:21.353607 18055 leader_election.cc:290] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 2 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:21.353247 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:20:21.354806 17834 raft_consensus.cc:3060] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:21.355366 17880 tablet_bootstrap.cc:492] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb: No bootstrap required, opened a new log
I20260709 12:20:21.355818 17880 ts_tablet_manager.cc:1403] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb: Time spent bootstrapping tablet: real 0.107s	user 0.017s	sys 0.002s
I20260709 12:20:21.359629 17881 raft_consensus.cc:359] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:21.360312 17881 raft_consensus.cc:385] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:21.360584 17881 raft_consensus.cc:740] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Initialized, Role: FOLLOWER
I20260709 12:20:21.360692 18130 raft_consensus.cc:2749] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:21.361965 17881 consensus_queue.cc:260] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:21.365490 17881 ts_tablet_manager.cc:1434] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b: Time spent starting tablet: real 0.021s	user 0.006s	sys 0.000s
I20260709 12:20:21.366402 17881 tablet_bootstrap.cc:492] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b: Bootstrap starting.
I20260709 12:20:21.358009 17995 raft_consensus.cc:515] T 3219dcbfd86340ecbde507517060ff81 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:21.369496 17759 raft_consensus.cc:2393] T 3219dcbfd86340ecbde507517060ff81 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 2: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:20:21.372951 17878 tablet_bootstrap.cc:492] T f5f0dd0890f44e35bbb838da838faa9c P 0da169ff82674dc1947055ce43de68a3: No bootstrap required, opened a new log
I20260709 12:20:21.374272 17878 ts_tablet_manager.cc:1403] T f5f0dd0890f44e35bbb838da838faa9c P 0da169ff82674dc1947055ce43de68a3: Time spent bootstrapping tablet: real 0.117s	user 0.021s	sys 0.000s
I20260709 12:20:21.379047 17878 raft_consensus.cc:359] T f5f0dd0890f44e35bbb838da838faa9c P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:21.379710 17878 raft_consensus.cc:385] T f5f0dd0890f44e35bbb838da838faa9c P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:21.379995 17878 raft_consensus.cc:740] T f5f0dd0890f44e35bbb838da838faa9c P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Initialized, Role: FOLLOWER
I20260709 12:20:21.380858 18122 raft_consensus.cc:493] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:21.381378 18122 raft_consensus.cc:515] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:21.373076 17881 tablet_bootstrap.cc:654] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:21.383756 18122 leader_election.cc:290] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:21.360715 17880 raft_consensus.cc:359] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:21.384191 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:20:21.387339 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:20:21.388044 17995 leader_election.cc:290] T 3219dcbfd86340ecbde507517060ff81 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 2 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:21.390046 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:20:21.394162 17688 raft_consensus.cc:2393] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 2: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:20:21.387392 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:20:21.384488 17880 raft_consensus.cc:385] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:21.395395 17881 tablet_bootstrap.cc:492] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b: No bootstrap required, opened a new log
I20260709 12:20:21.395582 17880 raft_consensus.cc:740] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Initialized, Role: FOLLOWER
I20260709 12:20:21.395766 17833 raft_consensus.cc:2393] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 1: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:20:21.395931 17881 ts_tablet_manager.cc:1403] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b: Time spent bootstrapping tablet: real 0.030s	user 0.011s	sys 0.004s
I20260709 12:20:21.396382 17880 consensus_queue.cc:260] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:21.390532 17834 raft_consensus.cc:2468] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 2.
I20260709 12:20:21.397434 17652 leader_election.cc:304] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:20:21.398821 17652 leader_election.cc:304] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:20:21.399758 18122 raft_consensus.cc:3060] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:21.395107 17728 leader_election.cc:304] T 3219dcbfd86340ecbde507517060ff81 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:20:21.400990 17881 raft_consensus.cc:359] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:21.401651 17881 raft_consensus.cc:385] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:21.401927 17995 raft_consensus.cc:2749] T 3219dcbfd86340ecbde507517060ff81 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:20:21.401930 17881 raft_consensus.cc:740] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Initialized, Role: FOLLOWER
I20260709 12:20:21.392204 17878 consensus_queue.cc:260] T f5f0dd0890f44e35bbb838da838faa9c P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:21.403007 18005 raft_consensus.cc:2804] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:20:21.402881 17881 consensus_queue.cc:260] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:21.403481 18005 raft_consensus.cc:697] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 2 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:20:21.404476 17878 ts_tablet_manager.cc:1434] T f5f0dd0890f44e35bbb838da838faa9c P 0da169ff82674dc1947055ce43de68a3: Time spent starting tablet: real 0.028s	user 0.006s	sys 0.000s
I20260709 12:20:21.404330 18005 consensus_queue.cc:237] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:21.405305 17881 ts_tablet_manager.cc:1434] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b: Time spent starting tablet: real 0.009s	user 0.006s	sys 0.000s
I20260709 12:20:21.405340 17878 tablet_bootstrap.cc:492] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3: Bootstrap starting.
I20260709 12:20:21.412842 18122 raft_consensus.cc:2749] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:21.414541 17881 tablet_bootstrap.cc:492] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b: Bootstrap starting.
I20260709 12:20:21.415472 18130 consensus_queue.cc:1048] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:20:21.417953 17880 ts_tablet_manager.cc:1434] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb: Time spent starting tablet: real 0.062s	user 0.006s	sys 0.000s
I20260709 12:20:21.418910 17880 tablet_bootstrap.cc:492] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb: Bootstrap starting.
I20260709 12:20:21.424211 17878 tablet_bootstrap.cc:654] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:21.424754 17880 tablet_bootstrap.cc:654] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:21.432324 17952 consensus_queue.cc:1048] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:20:21.487746 17584 catalog_manager.cc:5697] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 reported cstate change: term changed from 0 to 2, leader changed from <none> to 0da169ff82674dc1947055ce43de68a3 (127.16.29.65). New cstate: current_term: 2 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:21.526551 17881 tablet_bootstrap.cc:654] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:21.547596 17880 tablet_bootstrap.cc:492] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb: No bootstrap required, opened a new log
I20260709 12:20:21.548133 17880 ts_tablet_manager.cc:1403] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent bootstrapping tablet: real 0.129s	user 0.062s	sys 0.034s
I20260709 12:20:21.550776 17880 raft_consensus.cc:359] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:21.551465 17880 raft_consensus.cc:385] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:21.552182 17880 raft_consensus.cc:740] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Initialized, Role: FOLLOWER
I20260709 12:20:21.552901 17880 consensus_queue.cc:260] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:21.555361 17880 ts_tablet_manager.cc:1434] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent starting tablet: real 0.007s	user 0.004s	sys 0.000s
I20260709 12:20:21.556492 17880 tablet_bootstrap.cc:492] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb: Bootstrap starting.
I20260709 12:20:21.558513 17878 tablet_bootstrap.cc:492] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3: No bootstrap required, opened a new log
I20260709 12:20:21.559006 17878 ts_tablet_manager.cc:1403] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3: Time spent bootstrapping tablet: real 0.154s	user 0.026s	sys 0.078s
I20260709 12:20:21.567359 18005 raft_consensus.cc:493] T f5f0dd0890f44e35bbb838da838faa9c P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:21.568177 17881 tablet_bootstrap.cc:492] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b: No bootstrap required, opened a new log
I20260709 12:20:21.567871 18005 raft_consensus.cc:515] T f5f0dd0890f44e35bbb838da838faa9c P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:21.574554 17881 ts_tablet_manager.cc:1403] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b: Time spent bootstrapping tablet: real 0.160s	user 0.013s	sys 0.000s
I20260709 12:20:21.569141 17878 raft_consensus.cc:359] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:21.574970 17878 raft_consensus.cc:385] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:21.575261 17878 raft_consensus.cc:740] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Initialized, Role: FOLLOWER
I20260709 12:20:21.576056 18005 leader_election.cc:290] T f5f0dd0890f44e35bbb838da838faa9c P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:21.576126 17878 consensus_queue.cc:260] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:21.577455 17881 raft_consensus.cc:359] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:21.578133 17881 raft_consensus.cc:385] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:21.578377 17881 raft_consensus.cc:740] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Initialized, Role: FOLLOWER
I20260709 12:20:21.578948 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:20:21.578998 17881 consensus_queue.cc:260] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:21.581180 17881 ts_tablet_manager.cc:1434] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b: Time spent starting tablet: real 0.006s	user 0.004s	sys 0.002s
I20260709 12:20:21.568060 17880 tablet_bootstrap.cc:654] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:21.583588 17878 ts_tablet_manager.cc:1434] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3: Time spent starting tablet: real 0.024s	user 0.009s	sys 0.000s
I20260709 12:20:21.583735 17995 consensus_queue.cc:1048] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:21.584808 17878 tablet_bootstrap.cc:492] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3: Bootstrap starting.
I20260709 12:20:21.585009 17881 tablet_bootstrap.cc:492] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b: Bootstrap starting.
I20260709 12:20:21.589879 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:20:21.591768 17878 tablet_bootstrap.cc:654] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3: Neither blocks nor log segments found. Creating new log.
W20260709 12:20:21.592242 17652 leader_election.cc:343] T f5f0dd0890f44e35bbb838da838faa9c P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:20:21.593041 17652 leader_election.cc:343] T f5f0dd0890f44e35bbb838da838faa9c P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:21.594007 18130 raft_consensus.cc:493] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:21.595461 18130 raft_consensus.cc:515] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:21.591776 17881 tablet_bootstrap.cc:654] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:21.593380 17652 leader_election.cc:304] T f5f0dd0890f44e35bbb838da838faa9c P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:20:21.599341 18130 leader_election.cc:290] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:20:21.601421 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:20:21.603246 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:20:21.604193 17686 raft_consensus.cc:2468] T 8cab6f01c6a2485cb5024322a1bd5462 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 0.
I20260709 12:20:21.605511 18005 raft_consensus.cc:2749] T f5f0dd0890f44e35bbb838da838faa9c P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:21.612607 17995 raft_consensus.cc:493] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:21.613207 17995 raft_consensus.cc:515] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:21.617744 17995 leader_election.cc:290] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:21.618223 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:20:21.620821 17880 tablet_bootstrap.cc:492] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb: No bootstrap required, opened a new log
W20260709 12:20:21.621050 17728 leader_election.cc:343] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:21.621344 17880 ts_tablet_manager.cc:1403] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb: Time spent bootstrapping tablet: real 0.065s	user 0.007s	sys 0.012s
I20260709 12:20:21.622136 17763 raft_consensus.cc:3060] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:21.624145 17880 raft_consensus.cc:359] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:21.624809 17880 raft_consensus.cc:385] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:21.627624 18122 raft_consensus.cc:493] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:21.607120 17802 leader_election.cc:304] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:20:21.628229 18122 raft_consensus.cc:515] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:20:21.628852 17801 leader_election.cc:343] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:21.631780 18122 leader_election.cc:290] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:21.632547 17935 consensus_queue.cc:1048] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:21.634570 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:20:21.635478 17833 raft_consensus.cc:2468] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 0.
I20260709 12:20:21.638198 17949 raft_consensus.cc:2804] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:21.638751 17949 raft_consensus.cc:493] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:21.639086 17949 raft_consensus.cc:3060] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:21.640010 17880 raft_consensus.cc:740] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Initialized, Role: FOLLOWER
I20260709 12:20:21.640939 17880 consensus_queue.cc:260] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:21.643041 17880 ts_tablet_manager.cc:1434] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb: Time spent starting tablet: real 0.021s	user 0.005s	sys 0.000s
I20260709 12:20:21.643492 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:20:21.644361 17759 raft_consensus.cc:2468] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 0.
I20260709 12:20:21.648909 17952 raft_consensus.cc:493] T 69716f6d5e7549418fdd8d54d7b722ce P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:21.649461 17952 raft_consensus.cc:515] T 69716f6d5e7549418fdd8d54d7b722ce P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:21.651567 17952 leader_election.cc:290] T 69716f6d5e7549418fdd8d54d7b722ce P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:20:21.655275 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:20:21.659201 18122 raft_consensus.cc:493] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:21.660159 17833 raft_consensus.cc:3060] T 69716f6d5e7549418fdd8d54d7b722ce P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:21.660090 18122 raft_consensus.cc:515] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:21.663661 18122 leader_election.cc:290] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:21.664455 17878 tablet_bootstrap.cc:492] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3: No bootstrap required, opened a new log
I20260709 12:20:21.664920 17878 ts_tablet_manager.cc:1403] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3: Time spent bootstrapping tablet: real 0.081s	user 0.014s	sys 0.007s
I20260709 12:20:21.665565 17652 leader_election.cc:304] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:20:21.666908 18124 raft_consensus.cc:2804] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:21.667335 18124 raft_consensus.cc:493] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:21.667699 18124 raft_consensus.cc:3060] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:21.671869 17995 raft_consensus.cc:493] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:21.667511 17878 raft_consensus.cc:359] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:21.672463 17995 raft_consensus.cc:515] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:21.673136 17878 raft_consensus.cc:385] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:21.673764 17878 raft_consensus.cc:740] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Initialized, Role: FOLLOWER
I20260709 12:20:21.675004 17878 consensus_queue.cc:260] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:21.679786 17878 ts_tablet_manager.cc:1434] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3: Time spent starting tablet: real 0.015s	user 0.007s	sys 0.000s
I20260709 12:20:21.680663 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:20:21.681169 17878 tablet_bootstrap.cc:492] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3: Bootstrap starting.
I20260709 12:20:21.673905 17949 raft_consensus.cc:515] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:21.683898 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:20:21.684787 17995 leader_election.cc:290] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:20:21.686802 17727 leader_election.cc:343] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:21.689518 17727 leader_election.cc:304] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:20:21.690109 18124 raft_consensus.cc:515] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:21.687247 17935 consensus_queue.cc:1048] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:20:21.692631 17935 raft_consensus.cc:2749] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:21.692889 18124 leader_election.cc:290] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:21.687453 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:20:21.693132 17878 tablet_bootstrap.cc:654] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:21.696048 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:20:21.696097 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:20:21.696997 17836 raft_consensus.cc:3060] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:21.698627 17880 tablet_bootstrap.cc:492] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb: Bootstrap starting.
I20260709 12:20:21.699270 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:20:21.703265 17837 raft_consensus.cc:2468] T ac984176358d453a8800cc9e0e304061 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 1.
I20260709 12:20:21.703579 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:20:21.699290 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:20:21.705343 18005 consensus_queue.cc:1048] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260709 12:20:21.702663 17652 leader_election.cc:343] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:21.706676 17801 leader_election.cc:304] T 69716f6d5e7549418fdd8d54d7b722ce P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:20:21.706934 17652 leader_election.cc:343] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:21.707374 17652 leader_election.cc:304] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:20:21.704263 17758 raft_consensus.cc:3060] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:21.714344 17727 leader_election.cc:304] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:20:21.714980 17949 leader_election.cc:290] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 1 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:20:21.712821 18122 raft_consensus.cc:2749] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:21.722728 17881 tablet_bootstrap.cc:492] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b: No bootstrap required, opened a new log
I20260709 12:20:21.723408 17881 ts_tablet_manager.cc:1403] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b: Time spent bootstrapping tablet: real 0.139s	user 0.023s	sys 0.035s
I20260709 12:20:21.724046 17935 raft_consensus.cc:2804] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:20:21.724841 17935 raft_consensus.cc:493] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:21.725076 18055 raft_consensus.cc:493] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:21.725189 17935 raft_consensus.cc:3060] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:21.725248 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:20:21.725428 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:20:21.725673 18055 raft_consensus.cc:515] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:21.726142 17687 raft_consensus.cc:3060] T 8cab6f01c6a2485cb5024322a1bd5462 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:20:21.737661 17801 leader_election.cc:343] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:21.746228 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:20:21.747051 17760 raft_consensus.cc:2468] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 0.
I20260709 12:20:21.749431 17652 leader_election.cc:304] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:20:21.754995 18005 raft_consensus.cc:2804] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:21.755414 18005 raft_consensus.cc:493] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:21.755725 18005 raft_consensus.cc:3060] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:21.767093 17758 raft_consensus.cc:2468] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 1.
I20260709 12:20:21.763650 17881 raft_consensus.cc:359] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:21.768497 17881 raft_consensus.cc:385] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:21.768797 17881 raft_consensus.cc:740] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Initialized, Role: FOLLOWER
I20260709 12:20:21.769568 17881 consensus_queue.cc:260] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:21.770268 17687 raft_consensus.cc:2468] T 8cab6f01c6a2485cb5024322a1bd5462 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 1.
I20260709 12:20:21.770766 17652 leader_election.cc:304] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:20:21.771584 17881 ts_tablet_manager.cc:1434] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b: Time spent starting tablet: real 0.048s	user 0.005s	sys 0.001s
I20260709 12:20:21.778698 18036 consensus_queue.cc:1048] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:20:21.779664 17881 tablet_bootstrap.cc:492] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b: Bootstrap starting.
I20260709 12:20:21.781708 18005 raft_consensus.cc:515] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:21.783967 18005 leader_election.cc:290] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:21.781473 17935 raft_consensus.cc:515] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:21.785557 17761 raft_consensus.cc:1240] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 1. Current term is 2. Ops: [1.1-1.1]
I20260709 12:20:21.785940 17880 tablet_bootstrap.cc:654] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:21.789098 17836 raft_consensus.cc:2468] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 1.
I20260709 12:20:21.771492 18055 leader_election.cc:290] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:21.772149 18122 raft_consensus.cc:2804] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:21.792588 18122 raft_consensus.cc:697] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 1 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:20:21.772619 18124 consensus_queue.cc:1048] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:20:21.773699 18133 raft_consensus.cc:2749] T 69716f6d5e7549418fdd8d54d7b722ce P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:21.793437 18122 consensus_queue.cc:237] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:21.774417 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:20:21.796245 17835 raft_consensus.cc:2468] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 0.
I20260709 12:20:21.778272 17802 leader_election.cc:304] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:20:21.778337 17949 raft_consensus.cc:493] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:21.797632 18133 raft_consensus.cc:2804] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:21.798192 18133 raft_consensus.cc:697] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 1 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:20:21.799024 18133 consensus_queue.cc:237] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:21.801012 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:20:21.806998 18036 raft_consensus.cc:493] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:21.807731 18036 raft_consensus.cc:515] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:21.834945 17935 leader_election.cc:290] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 2 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:21.836884 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:20:21.842454 18036 leader_election.cc:290] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:21.842655 17836 raft_consensus.cc:3060] T ac984176358d453a8800cc9e0e304061 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:21.836982 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:20:21.848331 17728 leader_election.cc:343] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:21.848798 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:20:21.855152 17836 raft_consensus.cc:2468] T ac984176358d453a8800cc9e0e304061 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 2.
I20260709 12:20:21.856698 17727 leader_election.cc:304] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:20:21.857749 18140 raft_consensus.cc:2804] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:20:21.858385 18140 raft_consensus.cc:697] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:20:21.859725 18140 consensus_queue.cc:237] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:21.797109 17949 raft_consensus.cc:515] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:21.866099 17835 raft_consensus.cc:2468] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 0.
I20260709 12:20:21.878085 17880 tablet_bootstrap.cc:492] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb: No bootstrap required, opened a new log
I20260709 12:20:21.878638 17880 ts_tablet_manager.cc:1403] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent bootstrapping tablet: real 0.181s	user 0.009s	sys 0.014s
I20260709 12:20:21.881381 17727 leader_election.cc:304] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:20:21.882761 17878 tablet_bootstrap.cc:492] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3: No bootstrap required, opened a new log
I20260709 12:20:21.883249 17878 ts_tablet_manager.cc:1403] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3: Time spent bootstrapping tablet: real 0.202s	user 0.020s	sys 0.015s
I20260709 12:20:21.883100 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:20:21.883778 17761 raft_consensus.cc:2468] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 0.
I20260709 12:20:21.885298 17801 leader_election.cc:304] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:20:21.886250 18130 raft_consensus.cc:2804] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:21.886713 18130 raft_consensus.cc:493] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:21.887272 18130 raft_consensus.cc:3060] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:21.889189 17880 raft_consensus.cc:359] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:21.889865 17880 raft_consensus.cc:385] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:21.890153 17880 raft_consensus.cc:740] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Initialized, Role: FOLLOWER
I20260709 12:20:21.890805 17880 consensus_queue.cc:260] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:21.886018 17878 raft_consensus.cc:359] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:21.891714 17878 raft_consensus.cc:385] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:21.886749 18140 raft_consensus.cc:2804] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:21.895640 18140 raft_consensus.cc:493] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:21.895757 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:20:21.896062 18140 raft_consensus.cc:3060] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:21.892045 17878 raft_consensus.cc:740] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Initialized, Role: FOLLOWER
I20260709 12:20:21.838703 17881 tablet_bootstrap.cc:654] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b: Neither blocks nor log segments found. Creating new log.
W20260709 12:20:21.896382 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 124584us
I20260709 12:20:21.897898 17949 leader_election.cc:290] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:20:21.898866 18130 raft_consensus.cc:515] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:21.897899 17878 consensus_queue.cc:260] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:21.902052 17878 ts_tablet_manager.cc:1434] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3: Time spent starting tablet: real 0.019s	user 0.003s	sys 0.004s
I20260709 12:20:21.896569 17587 catalog_manager.cc:5697] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb reported cstate change: term changed from 1 to 2, leader changed from 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) to e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66). New cstate: current_term: 2 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:21.903283 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:20:21.905483 17878 tablet_bootstrap.cc:492] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3: Bootstrap starting.
I20260709 12:20:21.905704 17880 ts_tablet_manager.cc:1434] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent starting tablet: real 0.027s	user 0.004s	sys 0.004s
I20260709 12:20:21.906667 17880 tablet_bootstrap.cc:492] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb: Bootstrap starting.
I20260709 12:20:21.909150 18124 consensus_queue.cc:1059] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:20:21.910531 18124 raft_consensus.cc:3055] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:20:21.910843 18124 raft_consensus.cc:740] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:20:21.912078 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:20:21.911705 18124 consensus_queue.cc:260] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:21.912725 17835 raft_consensus.cc:3060] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:21.904625 17687 raft_consensus.cc:2468] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 0.
I20260709 12:20:21.913717 18124 raft_consensus.cc:3060] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:21.915259 17836 raft_consensus.cc:1240] T ac984176358d453a8800cc9e0e304061 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 1. Current term is 2. Ops: [1.1-1.1]
I20260709 12:20:21.904036 17761 raft_consensus.cc:3060] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:21.922678 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:20:21.923336 17687 raft_consensus.cc:3060] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:21.924453 17584 catalog_manager.cc:5697] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0da169ff82674dc1947055ce43de68a3 (127.16.29.65). New cstate: current_term: 1 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:21.931999 17687 raft_consensus.cc:2468] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 1.
I20260709 12:20:21.933740 17802 leader_election.cc:304] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:20:21.934670 17949 raft_consensus.cc:2804] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:21.936719 17761 raft_consensus.cc:2468] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 1.
I20260709 12:20:21.938390 17652 leader_election.cc:304] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:20:21.937930 17949 raft_consensus.cc:697] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 1 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:20:21.940759 18179 raft_consensus.cc:2804] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:21.941978 18179 raft_consensus.cc:697] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 1 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:20:21.942905 18179 consensus_queue.cc:237] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:21.945712 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:20:21.950681 17949 consensus_queue.cc:237] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:21.953624 17758 raft_consensus.cc:1275] T 3219dcbfd86340ecbde507517060ff81 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260709 12:20:21.966131 17587 catalog_manager.cc:5697] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b reported cstate change: term changed from 0 to 1, leader changed from <none> to e063541542a94969898868bf6eb5ce1b (127.16.29.67). New cstate: current_term: 1 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: HEALTHY } } }
I20260709 12:20:21.949587 18130 leader_election.cc:290] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 1 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:20:21.967260 18122 consensus_queue.cc:1048] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
I20260709 12:20:21.977597 17881 tablet_bootstrap.cc:492] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b: No bootstrap required, opened a new log
I20260709 12:20:21.978083 17881 ts_tablet_manager.cc:1403] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b: Time spent bootstrapping tablet: real 0.199s	user 0.015s	sys 0.001s
I20260709 12:20:21.980837 17881 raft_consensus.cc:359] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:21.981529 17881 raft_consensus.cc:385] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:21.981786 17881 raft_consensus.cc:740] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Initialized, Role: FOLLOWER
I20260709 12:20:21.982632 17881 consensus_queue.cc:260] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:21.984747 17881 ts_tablet_manager.cc:1434] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b: Time spent starting tablet: real 0.006s	user 0.006s	sys 0.000s
I20260709 12:20:21.985819 17881 tablet_bootstrap.cc:492] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b: Bootstrap starting.
I20260709 12:20:22.021037 17587 catalog_manager.cc:5697] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0da169ff82674dc1947055ce43de68a3 (127.16.29.65). New cstate: current_term: 1 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:22.025821 17835 raft_consensus.cc:2468] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 1.
I20260709 12:20:22.030593 17881 tablet_bootstrap.cc:654] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:22.034782 17834 raft_consensus.cc:1275] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260709 12:20:22.037846 17880 tablet_bootstrap.cc:654] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:22.044675 17935 raft_consensus.cc:493] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:22.045178 17935 raft_consensus.cc:515] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:22.047223 17935 leader_election.cc:290] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:22.046698 17585 catalog_manager.cc:5697] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b reported cstate change: term changed from 0 to 1, leader changed from <none> to e063541542a94969898868bf6eb5ce1b (127.16.29.67). New cstate: current_term: 1 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: HEALTHY } } }
I20260709 12:20:22.049175 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:20:22.049808 17686 raft_consensus.cc:2468] T f5f0dd0890f44e35bbb838da838faa9c P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 0.
I20260709 12:20:22.051038 17728 leader_election.cc:304] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:20:22.052273 18183 raft_consensus.cc:2804] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:22.037840 18140 raft_consensus.cc:515] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:22.053762 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:20:22.038017 18124 consensus_queue.cc:1048] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.014s
I20260709 12:20:22.055707 18183 raft_consensus.cc:493] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:22.074564 18183 raft_consensus.cc:3060] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:22.077859 18140 leader_election.cc:290] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 1 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:22.078943 17761 raft_consensus.cc:2393] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 1: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:20:22.080222 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:20:22.080960 17687 raft_consensus.cc:2393] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 1: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:20:22.089531 18133 raft_consensus.cc:493] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:20:22.095602 17727 leader_election.cc:343] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:22.098831 18133 raft_consensus.cc:515] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:22.102083 18133 leader_election.cc:290] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:20:22.102859 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:20:22.104861 17801 leader_election.cc:343] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:22.105671 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:20:22.108889 17881 tablet_bootstrap.cc:492] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b: No bootstrap required, opened a new log
I20260709 12:20:22.109382 17878 tablet_bootstrap.cc:654] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:22.109459 17881 ts_tablet_manager.cc:1403] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b: Time spent bootstrapping tablet: real 0.124s	user 0.011s	sys 0.008s
I20260709 12:20:22.114426 17727 leader_election.cc:304] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:20:22.116046 18140 raft_consensus.cc:2749] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 12:20:22.116042 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:20:22.117636 17686 raft_consensus.cc:2468] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 0.
I20260709 12:20:22.119424 17802 leader_election.cc:304] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:20:22.121487 17949 raft_consensus.cc:2804] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:22.122493 17949 raft_consensus.cc:493] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:22.122869 17949 raft_consensus.cc:3060] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:22.124425 18183 raft_consensus.cc:515] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:22.129828 17949 raft_consensus.cc:515] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:22.132555 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:20:22.133502 17686 raft_consensus.cc:3060] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:22.141925 17881 raft_consensus.cc:359] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:22.143157 17881 raft_consensus.cc:385] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:22.143522 17881 raft_consensus.cc:740] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Initialized, Role: FOLLOWER
I20260709 12:20:22.148190 17881 consensus_queue.cc:260] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:22.148975 17880 tablet_bootstrap.cc:492] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb: No bootstrap required, opened a new log
I20260709 12:20:22.149405 17880 ts_tablet_manager.cc:1403] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb: Time spent bootstrapping tablet: real 0.243s	user 0.012s	sys 0.005s
W20260709 12:20:22.152309 17635 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:20:22.156323 17880 raft_consensus.cc:359] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:22.156983 17880 raft_consensus.cc:385] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:22.157233 17880 raft_consensus.cc:740] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Initialized, Role: FOLLOWER
I20260709 12:20:22.157858 17880 consensus_queue.cc:260] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:22.158485 17949 leader_election.cc:290] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 1 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:20:22.163832 18140 raft_consensus.cc:493] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:22.164218 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:20:22.164363 18140 raft_consensus.cc:515] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:22.164819 17762 raft_consensus.cc:3060] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:22.165135 17686 raft_consensus.cc:2468] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 1.
I20260709 12:20:22.166870 18140 leader_election.cc:290] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:22.167330 17880 ts_tablet_manager.cc:1434] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb: Time spent starting tablet: real 0.018s	user 0.007s	sys 0.000s
I20260709 12:20:22.168280 17880 tablet_bootstrap.cc:492] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb: Bootstrap starting.
I20260709 12:20:22.168303 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:20:22.169304 17686 raft_consensus.cc:2393] T 8cab6f01c6a2485cb5024322a1bd5462 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 1: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:20:22.172183 17802 leader_election.cc:304] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:20:22.173362 18133 raft_consensus.cc:2804] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:22.173877 18133 raft_consensus.cc:697] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 1 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:20:22.174773 18133 consensus_queue.cc:237] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:22.176081 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:20:22.177937 17727 leader_election.cc:304] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:20:22.180518 18140 raft_consensus.cc:3060] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:22.183300 17881 ts_tablet_manager.cc:1434] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b: Time spent starting tablet: real 0.073s	user 0.007s	sys 0.000s
I20260709 12:20:22.186564 18183 leader_election.cc:290] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 1 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:22.187837 17881 tablet_bootstrap.cc:492] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b: Bootstrap starting.
I20260709 12:20:22.188591 17762 raft_consensus.cc:2468] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 1.
I20260709 12:20:22.189851 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:20:22.190447 17835 raft_consensus.cc:3060] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:22.204013 17880 tablet_bootstrap.cc:654] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:22.205983 18124 consensus_queue.cc:1048] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:20:22.200737 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:20:22.208401 17686 raft_consensus.cc:3060] T f5f0dd0890f44e35bbb838da838faa9c P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:22.223090 17835 raft_consensus.cc:2468] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 1.
I20260709 12:20:22.236164 18140 raft_consensus.cc:2749] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:22.242682 17585 catalog_manager.cc:5697] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b reported cstate change: term changed from 0 to 1, leader changed from <none> to e063541542a94969898868bf6eb5ce1b (127.16.29.67). New cstate: current_term: 1 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: HEALTHY } } }
I20260709 12:20:22.241246 17727 leader_election.cc:304] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:20:22.246497 18183 raft_consensus.cc:2804] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:22.247002 18183 raft_consensus.cc:697] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 1 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:20:22.247829 18183 consensus_queue.cc:237] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:22.254392 17878 tablet_bootstrap.cc:492] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3: No bootstrap required, opened a new log
I20260709 12:20:22.254906 17878 ts_tablet_manager.cc:1403] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3: Time spent bootstrapping tablet: real 0.350s	user 0.018s	sys 0.009s
I20260709 12:20:22.258183 17878 raft_consensus.cc:359] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:22.259006 17878 raft_consensus.cc:385] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:22.259217 17878 raft_consensus.cc:740] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Initialized, Role: FOLLOWER
I20260709 12:20:22.263573 17878 consensus_queue.cc:260] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:22.272285 17878 ts_tablet_manager.cc:1434] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3: Time spent starting tablet: real 0.017s	user 0.008s	sys 0.000s
I20260709 12:20:22.282150 17834 raft_consensus.cc:1275] T ac984176358d453a8800cc9e0e304061 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260709 12:20:22.283691 18179 consensus_queue.cc:1048] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:22.292449 17935 consensus_queue.cc:1048] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:22.294130 17878 tablet_bootstrap.cc:492] T 0e02889970bc40dc8ca2e81483f211e0 P 0da169ff82674dc1947055ce43de68a3: Bootstrap starting.
I20260709 12:20:22.295095 17881 tablet_bootstrap.cc:654] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:22.299191 17880 tablet_bootstrap.cc:492] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb: No bootstrap required, opened a new log
I20260709 12:20:22.299628 17880 ts_tablet_manager.cc:1403] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent bootstrapping tablet: real 0.132s	user 0.018s	sys 0.026s
I20260709 12:20:22.302412 17880 raft_consensus.cc:359] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:22.303052 17880 raft_consensus.cc:385] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:22.303309 17880 raft_consensus.cc:740] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Initialized, Role: FOLLOWER
I20260709 12:20:22.294155 17686 raft_consensus.cc:2468] T f5f0dd0890f44e35bbb838da838faa9c P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 1.
I20260709 12:20:22.314659 17881 tablet_bootstrap.cc:492] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b: No bootstrap required, opened a new log
I20260709 12:20:22.317586 17881 ts_tablet_manager.cc:1403] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b: Time spent bootstrapping tablet: real 0.134s	user 0.003s	sys 0.011s
I20260709 12:20:22.315248 17880 consensus_queue.cc:260] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:22.320724 17880 ts_tablet_manager.cc:1434] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent starting tablet: real 0.021s	user 0.008s	sys 0.000s
I20260709 12:20:22.321866 17880 tablet_bootstrap.cc:492] T ea2f2b5792f84bbba2738f1ce1e90122 P e2909cff1da44ad387f96a8ae3aa5afb: Bootstrap starting.
I20260709 12:20:22.323136 17881 raft_consensus.cc:359] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:22.314889 17878 tablet_bootstrap.cc:654] T 0e02889970bc40dc8ca2e81483f211e0 P 0da169ff82674dc1947055ce43de68a3: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:22.325413 17881 raft_consensus.cc:385] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:22.326124 17881 raft_consensus.cc:740] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Initialized, Role: FOLLOWER
I20260709 12:20:22.327855 17881 consensus_queue.cc:260] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:22.336027 17880 tablet_bootstrap.cc:654] T ea2f2b5792f84bbba2738f1ce1e90122 P e2909cff1da44ad387f96a8ae3aa5afb: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:22.338851 17881 ts_tablet_manager.cc:1434] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b: Time spent starting tablet: real 0.020s	user 0.003s	sys 0.003s
I20260709 12:20:22.340598 17881 tablet_bootstrap.cc:492] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b: Bootstrap starting.
I20260709 12:20:22.341710 18179 consensus_queue.cc:1048] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:20:22.349534 17587 catalog_manager.cc:5697] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb reported cstate change: term changed from 0 to 1, leader changed from <none> to e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66). New cstate: current_term: 1 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:22.350612 17881 tablet_bootstrap.cc:654] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:22.356727 17687 raft_consensus.cc:1275] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 1. (term mismatch)
I20260709 12:20:22.360436 17687 pending_rounds.cc:85] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3: Aborting all ops after (but not including) 0
I20260709 12:20:22.360742 17687 pending_rounds.cc:107] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3: Aborting uncommitted NO_OP operation due to leader change: 1.1
I20260709 12:20:22.361039 17687 raft_consensus.cc:2889] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:20:22.364523 18140 consensus_queue.cc:1048] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:20:22.376753 17881 tablet_bootstrap.cc:492] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b: No bootstrap required, opened a new log
I20260709 12:20:22.378799 17881 ts_tablet_manager.cc:1403] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b: Time spent bootstrapping tablet: real 0.038s	user 0.015s	sys 0.000s
I20260709 12:20:22.379202 17878 tablet_bootstrap.cc:492] T 0e02889970bc40dc8ca2e81483f211e0 P 0da169ff82674dc1947055ce43de68a3: No bootstrap required, opened a new log
I20260709 12:20:22.379954 17878 ts_tablet_manager.cc:1403] T 0e02889970bc40dc8ca2e81483f211e0 P 0da169ff82674dc1947055ce43de68a3: Time spent bootstrapping tablet: real 0.086s	user 0.021s	sys 0.005s
I20260709 12:20:22.384081 17878 raft_consensus.cc:359] T 0e02889970bc40dc8ca2e81483f211e0 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:22.390939 17878 raft_consensus.cc:385] T 0e02889970bc40dc8ca2e81483f211e0 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:22.387317 17881 raft_consensus.cc:359] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:22.391757 17881 raft_consensus.cc:385] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:22.399101 17881 raft_consensus.cc:740] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Initialized, Role: FOLLOWER
I20260709 12:20:22.397182 18055 consensus_queue.cc:1048] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:20:22.391340 17878 raft_consensus.cc:740] T 0e02889970bc40dc8ca2e81483f211e0 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Initialized, Role: FOLLOWER
I20260709 12:20:22.400440 17878 consensus_queue.cc:260] T 0e02889970bc40dc8ca2e81483f211e0 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:22.402475 17878 ts_tablet_manager.cc:1434] T 0e02889970bc40dc8ca2e81483f211e0 P 0da169ff82674dc1947055ce43de68a3: Time spent starting tablet: real 0.022s	user 0.006s	sys 0.000s
I20260709 12:20:22.404564 17878 tablet_bootstrap.cc:492] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3: Bootstrap starting.
I20260709 12:20:22.403239 17881 consensus_queue.cc:260] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:22.410203 17881 ts_tablet_manager.cc:1434] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b: Time spent starting tablet: real 0.025s	user 0.005s	sys 0.000s
I20260709 12:20:22.411059 17881 tablet_bootstrap.cc:492] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b: Bootstrap starting.
I20260709 12:20:22.419142 17878 tablet_bootstrap.cc:654] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:22.425215 18130 consensus_queue.cc:1048] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:20:22.433144 17881 tablet_bootstrap.cc:654] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:22.440212 17880 tablet_bootstrap.cc:492] T ea2f2b5792f84bbba2738f1ce1e90122 P e2909cff1da44ad387f96a8ae3aa5afb: No bootstrap required, opened a new log
I20260709 12:20:22.440769 17880 ts_tablet_manager.cc:1403] T ea2f2b5792f84bbba2738f1ce1e90122 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent bootstrapping tablet: real 0.119s	user 0.026s	sys 0.005s
I20260709 12:20:22.449873 17880 raft_consensus.cc:359] T ea2f2b5792f84bbba2738f1ce1e90122 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:22.450809 17880 raft_consensus.cc:385] T ea2f2b5792f84bbba2738f1ce1e90122 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:22.451152 17880 raft_consensus.cc:740] T ea2f2b5792f84bbba2738f1ce1e90122 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Initialized, Role: FOLLOWER
I20260709 12:20:22.514370 17880 consensus_queue.cc:260] T ea2f2b5792f84bbba2738f1ce1e90122 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:22.522176 17880 ts_tablet_manager.cc:1434] T ea2f2b5792f84bbba2738f1ce1e90122 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent starting tablet: real 0.081s	user 0.010s	sys 0.002s
I20260709 12:20:22.529453 18124 raft_consensus.cc:493] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:22.530442 18124 raft_consensus.cc:515] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:22.533638 18124 leader_election.cc:290] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:22.538357 18131 consensus_queue.cc:1048] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:22.538905 17881 tablet_bootstrap.cc:492] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b: No bootstrap required, opened a new log
I20260709 12:20:22.539161 17878 tablet_bootstrap.cc:492] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3: No bootstrap required, opened a new log
I20260709 12:20:22.540001 17881 ts_tablet_manager.cc:1403] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b: Time spent bootstrapping tablet: real 0.129s	user 0.014s	sys 0.004s
I20260709 12:20:22.540004 17878 ts_tablet_manager.cc:1403] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3: Time spent bootstrapping tablet: real 0.136s	user 0.005s	sys 0.013s
I20260709 12:20:22.543918 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:20:22.545113 17758 raft_consensus.cc:2468] T ea2f2b5792f84bbba2738f1ce1e90122 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 0.
I20260709 12:20:22.546787 17878 raft_consensus.cc:359] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:22.547645 17878 raft_consensus.cc:385] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:22.549136 17878 raft_consensus.cc:740] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Initialized, Role: FOLLOWER
I20260709 12:20:22.548743 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:20:22.556692 17835 raft_consensus.cc:2468] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 0.
I20260709 12:20:22.557811 17935 raft_consensus.cc:493] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:22.558202 17880 tablet_bootstrap.cc:492] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb: Bootstrap starting.
I20260709 12:20:22.558523 17935 raft_consensus.cc:515] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:22.564008 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:20:22.565639 17881 raft_consensus.cc:359] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:22.567063 17881 raft_consensus.cc:385] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260709 12:20:22.566308 17728 leader_election.cc:343] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:22.568171 17878 consensus_queue.cc:260] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:22.568673 17881 raft_consensus.cc:740] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Initialized, Role: FOLLOWER
I20260709 12:20:22.574731 17652 leader_election.cc:304] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:20:22.576682 17949 raft_consensus.cc:493] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:22.574810 17881 consensus_queue.cc:260] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:22.582991 17878 ts_tablet_manager.cc:1434] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3: Time spent starting tablet: real 0.041s	user 0.003s	sys 0.005s
I20260709 12:20:22.588732 18124 raft_consensus.cc:2804] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:22.589355 18124 raft_consensus.cc:493] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:22.589728 18124 raft_consensus.cc:3060] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:22.612977 17912 consensus_queue.cc:1048] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:22.637148 18122 raft_consensus.cc:493] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:22.637673 18122 raft_consensus.cc:515] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:22.651379 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:20:22.652329 18122 leader_election.cc:290] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:20:22.653290 17727 leader_election.cc:343] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:22.653817 17727 leader_election.cc:304] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:20:22.654912 18183 raft_consensus.cc:2749] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:22.657156 17880 tablet_bootstrap.cc:654] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:22.657612 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:20:22.660172 18179 raft_consensus.cc:493] T 0e02889970bc40dc8ca2e81483f211e0 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:22.660833 18179 raft_consensus.cc:515] T 0e02889970bc40dc8ca2e81483f211e0 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:22.663482 18179 leader_election.cc:290] T 0e02889970bc40dc8ca2e81483f211e0 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:22.665225 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:20:22.665347 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:20:22.667188 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:20:22.668541 17760 raft_consensus.cc:2468] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 0.
I20260709 12:20:22.577845 17949 raft_consensus.cc:515] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:22.670269 17834 raft_consensus.cc:2468] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 0.
I20260709 12:20:22.671550 17935 leader_election.cc:290] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:22.671710 17881 ts_tablet_manager.cc:1434] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b: Time spent starting tablet: real 0.130s	user 0.015s	sys 0.008s
W20260709 12:20:22.674633 17652 leader_election.cc:343] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:22.676074 18214 consensus_queue.cc:1048] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:22.665163 17878 tablet_bootstrap.cc:492] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3: Bootstrap starting.
I20260709 12:20:22.679768 17652 leader_election.cc:304] T 0e02889970bc40dc8ca2e81483f211e0 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:20:22.681861 17652 leader_election.cc:343] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:22.682250 17652 leader_election.cc:304] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:20:22.685418 17949 leader_election.cc:290] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:20:22.686079 18179 raft_consensus.cc:2804] T 0e02889970bc40dc8ca2e81483f211e0 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:22.686470 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:20:22.686493 18179 raft_consensus.cc:493] T 0e02889970bc40dc8ca2e81483f211e0 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:22.687306 18179 raft_consensus.cc:3060] T 0e02889970bc40dc8ca2e81483f211e0 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:22.686630 18122 raft_consensus.cc:2749] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:22.695302 18183 consensus_queue.cc:1048] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:22.698228 17952 consensus_queue.cc:1048] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:22.705531 17881 tablet_bootstrap.cc:492] T 10f96c39fba3496696f64385105bfeaa P e063541542a94969898868bf6eb5ce1b: Bootstrap starting.
I20260709 12:20:22.706511 17912 consensus_queue.cc:1048] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:22.706918 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:20:22.707787 17760 raft_consensus.cc:2468] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 0.
I20260709 12:20:22.711716 17801 leader_election.cc:304] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:20:22.716274 18131 raft_consensus.cc:2804] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:22.716696 18131 raft_consensus.cc:493] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:22.717044 18131 raft_consensus.cc:3060] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:22.755949 18131 raft_consensus.cc:515] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:22.774288 18179 raft_consensus.cc:515] T 0e02889970bc40dc8ca2e81483f211e0 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:22.756914 18124 raft_consensus.cc:515] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:22.776439 17878 tablet_bootstrap.cc:654] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:22.783288 17689 raft_consensus.cc:2393] T 0e02889970bc40dc8ca2e81483f211e0 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 1: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:20:22.793534 18131 leader_election.cc:290] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 1 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:20:22.796516 18179 leader_election.cc:290] T 0e02889970bc40dc8ca2e81483f211e0 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:22.797709 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:20:22.801842 18124 leader_election.cc:290] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:22.807914 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:20:22.808746 17762 raft_consensus.cc:3060] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:20:22.847098 18217 log.cc:927] Time spent T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.066s	user 0.000s	sys 0.001s
I20260709 12:20:22.854835 17880 tablet_bootstrap.cc:492] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb: No bootstrap required, opened a new log
I20260709 12:20:22.855554 17880 ts_tablet_manager.cc:1403] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent bootstrapping tablet: real 0.298s	user 0.009s	sys 0.008s
I20260709 12:20:22.856077 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:20:22.856257 17912 raft_consensus.cc:493] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:22.872411 17912 raft_consensus.cc:515] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:22.857554 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:20:22.876655 17835 raft_consensus.cc:2393] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 1: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:20:22.845921 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:20:22.880636 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:20:22.880950 17762 raft_consensus.cc:2468] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 1.
I20260709 12:20:22.881384 17833 raft_consensus.cc:3060] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:22.866284 17881 tablet_bootstrap.cc:654] T 10f96c39fba3496696f64385105bfeaa P e063541542a94969898868bf6eb5ce1b: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:22.875582 17761 raft_consensus.cc:3060] T ea2f2b5792f84bbba2738f1ce1e90122 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:22.887969 17912 leader_election.cc:290] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:20:22.922595 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:20:22.922571 17652 leader_election.cc:304] T 0e02889970bc40dc8ca2e81483f211e0 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:20:22.921423 17801 leader_election.cc:304] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
W20260709 12:20:22.912066 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 59138.1us
I20260709 12:20:22.929134 17878 tablet_bootstrap.cc:492] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3: No bootstrap required, opened a new log
I20260709 12:20:22.929687 17878 ts_tablet_manager.cc:1403] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3: Time spent bootstrapping tablet: real 0.265s	user 0.012s	sys 0.012s
I20260709 12:20:22.933729 18122 raft_consensus.cc:2749] T 0e02889970bc40dc8ca2e81483f211e0 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 12:20:22.936801 17880 raft_consensus.cc:359] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:22.948309 17880 raft_consensus.cc:385] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:22.948666 17880 raft_consensus.cc:740] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Initialized, Role: FOLLOWER
I20260709 12:20:22.952215 17880 consensus_queue.cc:260] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:22.954870 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:20:22.956189 18133 raft_consensus.cc:2804] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:22.967612 17880 ts_tablet_manager.cc:1434] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent starting tablet: real 0.112s	user 0.007s	sys 0.001s
W20260709 12:20:22.969908 17801 leader_election.cc:343] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:20:23.013558 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 74699.9us
I20260709 12:20:23.032694 18226 consensus_queue.cc:1048] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:22.960485 18133 raft_consensus.cc:697] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 1 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:20:23.041540 18133 consensus_queue.cc:237] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:23.049731 17878 raft_consensus.cc:359] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:23.059558 17833 raft_consensus.cc:2468] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 1.
I20260709 12:20:23.060494 17761 raft_consensus.cc:2468] T ea2f2b5792f84bbba2738f1ce1e90122 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 1.
I20260709 12:20:23.064827 17881 tablet_bootstrap.cc:492] T 10f96c39fba3496696f64385105bfeaa P e063541542a94969898868bf6eb5ce1b: No bootstrap required, opened a new log
I20260709 12:20:23.065447 17881 ts_tablet_manager.cc:1403] T 10f96c39fba3496696f64385105bfeaa P e063541542a94969898868bf6eb5ce1b: Time spent bootstrapping tablet: real 0.386s	user 0.023s	sys 0.009s
I20260709 12:20:23.065479 17878 raft_consensus.cc:385] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:23.066066 17878 raft_consensus.cc:740] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Initialized, Role: FOLLOWER
I20260709 12:20:23.069017 18185 raft_consensus.cc:493] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:23.069617 18185 raft_consensus.cc:515] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:23.072638 17881 raft_consensus.cc:359] T 10f96c39fba3496696f64385105bfeaa P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:23.074008 18185 leader_election.cc:290] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:23.066963 17878 consensus_queue.cc:260] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:23.074522 17881 raft_consensus.cc:385] T 10f96c39fba3496696f64385105bfeaa P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:23.075639 17881 raft_consensus.cc:740] T 10f96c39fba3496696f64385105bfeaa P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Initialized, Role: FOLLOWER
I20260709 12:20:23.077010 17881 consensus_queue.cc:260] T 10f96c39fba3496696f64385105bfeaa P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:23.078770 17878 ts_tablet_manager.cc:1434] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3: Time spent starting tablet: real 0.149s	user 0.017s	sys 0.013s
I20260709 12:20:23.079584 17881 ts_tablet_manager.cc:1434] T 10f96c39fba3496696f64385105bfeaa P e063541542a94969898868bf6eb5ce1b: Time spent starting tablet: real 0.014s	user 0.008s	sys 0.002s
I20260709 12:20:23.080097 17878 tablet_bootstrap.cc:492] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3: Bootstrap starting.
I20260709 12:20:23.081027 17881 tablet_bootstrap.cc:492] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b: Bootstrap starting.
I20260709 12:20:23.083182 17652 leader_election.cc:304] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:20:23.087139 17880 tablet_bootstrap.cc:492] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb: Bootstrap starting.
I20260709 12:20:23.089329 17881 tablet_bootstrap.cc:654] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b: Neither blocks nor log segments found. Creating new log.
W20260709 12:20:23.089938 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 48077.3us
I20260709 12:20:23.092167 18223 raft_consensus.cc:2804] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:23.092744 18223 raft_consensus.cc:697] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 1 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:20:23.093704 18223 consensus_queue.cc:237] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:23.103037 17878 tablet_bootstrap.cc:654] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3: Neither blocks nor log segments found. Creating new log.
W20260709 12:20:23.119042 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 163308us
I20260709 12:20:23.164688 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:20:23.168812 17802 leader_election.cc:343] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:23.176903 17802 leader_election.cc:304] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:20:23.183750 17912 raft_consensus.cc:2749] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:23.184695 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:20:23.185401 17686 raft_consensus.cc:2468] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 0.
I20260709 12:20:23.201030 17728 leader_election.cc:304] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:20:23.213459 18233 raft_consensus.cc:2804] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:23.214080 18233 raft_consensus.cc:493] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:23.214445 18233 raft_consensus.cc:3060] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:23.213459 18220 raft_consensus.cc:493] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:20:23.220214 17727 leader_election.cc:343] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:23.225623 18220 raft_consensus.cc:515] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:23.225574 17880 tablet_bootstrap.cc:654] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:23.242038 18220 leader_election.cc:290] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:23.249660 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:20:23.257383 17834 raft_consensus.cc:2468] T 10f96c39fba3496696f64385105bfeaa P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 0.
I20260709 12:20:23.251564 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:20:23.258724 17652 leader_election.cc:304] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:20:23.259503 18223 raft_consensus.cc:2804] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:23.256567 17878 tablet_bootstrap.cc:492] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3: No bootstrap required, opened a new log
I20260709 12:20:23.259893 18223 raft_consensus.cc:493] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:23.260452 17878 ts_tablet_manager.cc:1403] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3: Time spent bootstrapping tablet: real 0.181s	user 0.015s	sys 0.008s
I20260709 12:20:23.260416 18223 raft_consensus.cc:3060] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:23.265385 17881 tablet_bootstrap.cc:492] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b: No bootstrap required, opened a new log
I20260709 12:20:23.247835 17587 catalog_manager.cc:5697] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b reported cstate change: term changed from 0 to 1, leader changed from <none> to e063541542a94969898868bf6eb5ce1b (127.16.29.67). New cstate: current_term: 1 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: HEALTHY } } }
I20260709 12:20:23.265934 17881 ts_tablet_manager.cc:1403] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b: Time spent bootstrapping tablet: real 0.185s	user 0.014s	sys 0.010s
W20260709 12:20:23.269652 17652 leader_election.cc:343] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:23.272429 17878 raft_consensus.cc:359] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:23.273214 17878 raft_consensus.cc:385] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:23.273496 17878 raft_consensus.cc:740] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Initialized, Role: FOLLOWER
I20260709 12:20:23.274535 17881 raft_consensus.cc:359] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:23.274569 18223 raft_consensus.cc:515] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:23.278190 17881 raft_consensus.cc:385] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:23.278710 17881 raft_consensus.cc:740] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Initialized, Role: FOLLOWER
I20260709 12:20:23.279580 17881 consensus_queue.cc:260] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:23.280354 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
W20260709 12:20:23.282128 17652 leader_election.cc:343] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:23.283056 17881 ts_tablet_manager.cc:1434] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b: Time spent starting tablet: real 0.017s	user 0.008s	sys 0.000s
I20260709 12:20:23.285567 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:20:23.286145 17834 raft_consensus.cc:3060] T 10f96c39fba3496696f64385105bfeaa P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:23.301975 17878 consensus_queue.cc:260] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:23.288033 18233 raft_consensus.cc:515] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:23.308780 17834 raft_consensus.cc:2468] T 10f96c39fba3496696f64385105bfeaa P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 1.
I20260709 12:20:23.292744 18223 leader_election.cc:290] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:23.314055 17880 tablet_bootstrap.cc:492] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb: No bootstrap required, opened a new log
I20260709 12:20:23.314534 17880 ts_tablet_manager.cc:1403] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb: Time spent bootstrapping tablet: real 0.228s	user 0.014s	sys 0.002s
I20260709 12:20:23.318753 17878 ts_tablet_manager.cc:1434] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3: Time spent starting tablet: real 0.058s	user 0.007s	sys 0.000s
I20260709 12:20:23.319823 17878 tablet_bootstrap.cc:492] T f8f6bfcb37be41c6917282a17239ad06 P 0da169ff82674dc1947055ce43de68a3: Bootstrap starting.
I20260709 12:20:23.320693 17880 raft_consensus.cc:359] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:23.321764 17880 raft_consensus.cc:385] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:23.322302 17652 leader_election.cc:304] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:20:23.322459 17880 raft_consensus.cc:740] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Initialized, Role: FOLLOWER
I20260709 12:20:23.323767 18219 raft_consensus.cc:2804] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:23.324767 18219 raft_consensus.cc:697] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 1 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:20:23.326072 17880 consensus_queue.cc:260] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:23.328142 17881 tablet_bootstrap.cc:492] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b: Bootstrap starting.
I20260709 12:20:23.329260 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:20:23.336006 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:20:23.336553 17837 raft_consensus.cc:3060] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:23.336632 17684 raft_consensus.cc:3060] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:23.327713 18233 leader_election.cc:290] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 1 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:23.325721 18219 consensus_queue.cc:237] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:23.361397 17837 raft_consensus.cc:2468] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 1.
I20260709 12:20:23.365468 17880 ts_tablet_manager.cc:1434] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb: Time spent starting tablet: real 0.051s	user 0.008s	sys 0.002s
I20260709 12:20:23.365957 17727 leader_election.cc:304] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:20:23.373136 18183 raft_consensus.cc:2804] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:23.373646 18183 raft_consensus.cc:697] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:20:23.374754 17881 tablet_bootstrap.cc:654] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:23.374423 18183 consensus_queue.cc:237] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:23.376811 17880 tablet_bootstrap.cc:492] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb: Bootstrap starting.
W20260709 12:20:23.367942 17635 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:20:23.386219 18219 raft_consensus.cc:493] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:23.386736 18219 raft_consensus.cc:515] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:23.389070 18219 leader_election.cc:290] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:23.389842 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:20:23.390601 17759 raft_consensus.cc:2468] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 0.
I20260709 12:20:23.392602 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:20:23.392989 17652 leader_election.cc:304] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:20:23.393949 17652 leader_election.cc:343] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:23.394774 18219 raft_consensus.cc:2804] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:23.395219 18219 raft_consensus.cc:493] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:23.395606 18219 raft_consensus.cc:3060] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:23.396775 17878 tablet_bootstrap.cc:654] T f8f6bfcb37be41c6917282a17239ad06 P 0da169ff82674dc1947055ce43de68a3: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:23.436126 17684 raft_consensus.cc:2468] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 1.
I20260709 12:20:23.451496 18219 raft_consensus.cc:515] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:23.477856 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:20:23.478505 17758 raft_consensus.cc:3060] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:23.482098 17880 tablet_bootstrap.cc:654] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:23.494848 17584 catalog_manager.cc:5697] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb reported cstate change: term changed from 0 to 1, leader changed from <none> to e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66). New cstate: current_term: 1 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:23.503736 18133 consensus_queue.cc:1048] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:20:23.498070 17586 catalog_manager.cc:5697] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0da169ff82674dc1947055ce43de68a3 (127.16.29.65). New cstate: current_term: 1 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:23.508975 18230 raft_consensus.cc:493] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:23.509459 18230 raft_consensus.cc:515] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:23.510878 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
W20260709 12:20:23.512130 17652 leader_election.cc:343] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:23.512130 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:20:23.513532 17728 leader_election.cc:343] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:23.513036 18230 leader_election.cc:290] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:23.524128 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:20:23.540562 17837 raft_consensus.cc:2468] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 0.
I20260709 12:20:23.509019 17758 raft_consensus.cc:2468] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 1.
I20260709 12:20:23.541934 17727 leader_election.cc:304] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:20:23.542182 18219 leader_election.cc:290] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:23.543150 18230 raft_consensus.cc:2804] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:23.543567 18230 raft_consensus.cc:493] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:23.543972 18230 raft_consensus.cc:3060] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:23.545222 17652 leader_election.cc:304] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:20:23.546813 18223 raft_consensus.cc:2804] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:23.547448 18223 raft_consensus.cc:697] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 1 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:20:23.548852 18223 consensus_queue.cc:237] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:23.550657 18230 raft_consensus.cc:515] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:23.529533 17880 tablet_bootstrap.cc:492] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb: No bootstrap required, opened a new log
I20260709 12:20:23.554872 17880 ts_tablet_manager.cc:1403] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent bootstrapping tablet: real 0.178s	user 0.022s	sys 0.000s
I20260709 12:20:23.555003 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
W20260709 12:20:23.556717 17728 leader_election.cc:343] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:23.559075 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:20:23.559722 17838 raft_consensus.cc:3060] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:23.560263 18122 consensus_queue.cc:1048] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:20:23.564297 17881 tablet_bootstrap.cc:492] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b: No bootstrap required, opened a new log
I20260709 12:20:23.564790 17881 ts_tablet_manager.cc:1403] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b: Time spent bootstrapping tablet: real 0.255s	user 0.061s	sys 0.055s
I20260709 12:20:23.568328 17881 raft_consensus.cc:359] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:23.569000 17881 raft_consensus.cc:385] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:23.569247 17881 raft_consensus.cc:740] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Initialized, Role: FOLLOWER
I20260709 12:20:23.559371 17880 raft_consensus.cc:359] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:23.572237 18131 consensus_queue.cc:1048] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:20:23.569343 17878 tablet_bootstrap.cc:492] T f8f6bfcb37be41c6917282a17239ad06 P 0da169ff82674dc1947055ce43de68a3: No bootstrap required, opened a new log
I20260709 12:20:23.573307 17878 ts_tablet_manager.cc:1403] T f8f6bfcb37be41c6917282a17239ad06 P 0da169ff82674dc1947055ce43de68a3: Time spent bootstrapping tablet: real 0.254s	user 0.019s	sys 0.006s
I20260709 12:20:23.572214 17880 raft_consensus.cc:385] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:23.573853 18230 leader_election.cc:290] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 1 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:23.575285 17838 raft_consensus.cc:2468] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 1.
I20260709 12:20:23.576388 17727 leader_election.cc:304] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:20:23.577448 17935 raft_consensus.cc:2804] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:23.577909 17935 raft_consensus.cc:697] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 1 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:20:23.578717 17935 consensus_queue.cc:237] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:23.589686 17878 raft_consensus.cc:359] T f8f6bfcb37be41c6917282a17239ad06 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:23.590619 17878 raft_consensus.cc:385] T f8f6bfcb37be41c6917282a17239ad06 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:23.590934 17878 raft_consensus.cc:740] T f8f6bfcb37be41c6917282a17239ad06 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Initialized, Role: FOLLOWER
I20260709 12:20:23.570258 17881 consensus_queue.cc:260] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:23.592916 18231 raft_consensus.cc:493] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:23.593353 18231 raft_consensus.cc:515] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:23.596271 17881 ts_tablet_manager.cc:1434] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b: Time spent starting tablet: real 0.031s	user 0.004s	sys 0.004s
I20260709 12:20:23.573846 17880 raft_consensus.cc:740] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Initialized, Role: FOLLOWER
I20260709 12:20:23.600198 18231 leader_election.cc:290] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:20:23.602433 17881 tablet_bootstrap.cc:492] T 12b1fcd35e12402e8d6df7504c578e01 P e063541542a94969898868bf6eb5ce1b: Bootstrap starting.
I20260709 12:20:23.602337 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:20:23.602885 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:20:23.604307 17802 leader_election.cc:343] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:23.597682 17880 consensus_queue.cc:260] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:23.606750 17756 raft_consensus.cc:2468] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 0.
I20260709 12:20:23.608214 17880 ts_tablet_manager.cc:1434] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent starting tablet: real 0.053s	user 0.009s	sys 0.000s
I20260709 12:20:23.609238 17880 tablet_bootstrap.cc:492] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb: Bootstrap starting.
I20260709 12:20:23.591646 17878 consensus_queue.cc:260] T f8f6bfcb37be41c6917282a17239ad06 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:23.613332 17878 ts_tablet_manager.cc:1434] T f8f6bfcb37be41c6917282a17239ad06 P 0da169ff82674dc1947055ce43de68a3: Time spent starting tablet: real 0.040s	user 0.005s	sys 0.004s
I20260709 12:20:23.617797 17801 leader_election.cc:304] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:20:23.618597 17912 raft_consensus.cc:2804] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:23.618976 17912 raft_consensus.cc:493] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:23.619313 17912 raft_consensus.cc:3060] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:23.621291 17881 tablet_bootstrap.cc:654] T 12b1fcd35e12402e8d6df7504c578e01 P e063541542a94969898868bf6eb5ce1b: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:23.624743 17878 tablet_bootstrap.cc:492] T 9160d9f98b8c406db2771d9ea2e0995f P 0da169ff82674dc1947055ce43de68a3: Bootstrap starting.
I20260709 12:20:23.630736 17912 raft_consensus.cc:515] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:23.632625 17912 leader_election.cc:290] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 1 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:20:23.634292 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
W20260709 12:20:23.635711 17802 leader_election.cc:343] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:23.635593 17584 catalog_manager.cc:5697] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0da169ff82674dc1947055ce43de68a3 (127.16.29.65). New cstate: current_term: 1 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:23.637885 17584 catalog_manager.cc:5697] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0da169ff82674dc1947055ce43de68a3 (127.16.29.65). New cstate: current_term: 1 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: UNKNOWN } } }
W20260709 12:20:23.638476 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 57983.1us
I20260709 12:20:23.642197 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:20:23.642874 17759 raft_consensus.cc:3060] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:23.644039 17880 tablet_bootstrap.cc:654] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:23.651695 17878 tablet_bootstrap.cc:654] T 9160d9f98b8c406db2771d9ea2e0995f P 0da169ff82674dc1947055ce43de68a3: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:23.653294 17881 tablet_bootstrap.cc:492] T 12b1fcd35e12402e8d6df7504c578e01 P e063541542a94969898868bf6eb5ce1b: No bootstrap required, opened a new log
I20260709 12:20:23.653918 17881 ts_tablet_manager.cc:1403] T 12b1fcd35e12402e8d6df7504c578e01 P e063541542a94969898868bf6eb5ce1b: Time spent bootstrapping tablet: real 0.052s	user 0.010s	sys 0.003s
I20260709 12:20:23.657086 18055 consensus_queue.cc:1048] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:23.657689 17881 raft_consensus.cc:359] T 12b1fcd35e12402e8d6df7504c578e01 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:23.658588 17881 raft_consensus.cc:385] T 12b1fcd35e12402e8d6df7504c578e01 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:23.658959 17881 raft_consensus.cc:740] T 12b1fcd35e12402e8d6df7504c578e01 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Initialized, Role: FOLLOWER
I20260709 12:20:23.659192 17759 raft_consensus.cc:2468] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 1.
I20260709 12:20:23.663537 17801 leader_election.cc:304] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:20:23.663213 17881 consensus_queue.cc:260] T 12b1fcd35e12402e8d6df7504c578e01 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:23.668466 17881 ts_tablet_manager.cc:1434] T 12b1fcd35e12402e8d6df7504c578e01 P e063541542a94969898868bf6eb5ce1b: Time spent starting tablet: real 0.014s	user 0.009s	sys 0.000s
I20260709 12:20:23.668257 18231 raft_consensus.cc:2804] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:23.674710 18231 raft_consensus.cc:697] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 1 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:20:23.675580 18231 consensus_queue.cc:237] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:23.689113 17881 tablet_bootstrap.cc:492] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b: Bootstrap starting.
I20260709 12:20:23.703537 17586 catalog_manager.cc:5697] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb reported cstate change: term changed from 0 to 1, leader changed from <none> to e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66). New cstate: current_term: 1 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:23.716554 17880 tablet_bootstrap.cc:492] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb: No bootstrap required, opened a new log
I20260709 12:20:23.717053 17880 ts_tablet_manager.cc:1403] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb: Time spent bootstrapping tablet: real 0.108s	user 0.017s	sys 0.008s
I20260709 12:20:23.720222 17880 raft_consensus.cc:359] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:23.720927 17880 raft_consensus.cc:385] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:23.721220 17880 raft_consensus.cc:740] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Initialized, Role: FOLLOWER
I20260709 12:20:23.725108 17880 consensus_queue.cc:260] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:23.726696 17880 ts_tablet_manager.cc:1434] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb: Time spent starting tablet: real 0.009s	user 0.005s	sys 0.000s
I20260709 12:20:23.736506 17881 tablet_bootstrap.cc:654] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:23.738742 17880 tablet_bootstrap.cc:492] T 8a208e793881438b975b83755dc05cfa P e2909cff1da44ad387f96a8ae3aa5afb: Bootstrap starting.
I20260709 12:20:23.742416 17584 catalog_manager.cc:5697] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b reported cstate change: term changed from 0 to 1, leader changed from <none> to e063541542a94969898868bf6eb5ce1b (127.16.29.67). New cstate: current_term: 1 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: HEALTHY } } }
I20260709 12:20:23.755846 17880 tablet_bootstrap.cc:654] T 8a208e793881438b975b83755dc05cfa P e2909cff1da44ad387f96a8ae3aa5afb: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:23.754040 17881 tablet_bootstrap.cc:492] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b: No bootstrap required, opened a new log
I20260709 12:20:23.757028 17881 ts_tablet_manager.cc:1403] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b: Time spent bootstrapping tablet: real 0.072s	user 0.008s	sys 0.005s
I20260709 12:20:23.757620 17878 tablet_bootstrap.cc:492] T 9160d9f98b8c406db2771d9ea2e0995f P 0da169ff82674dc1947055ce43de68a3: No bootstrap required, opened a new log
I20260709 12:20:23.759214 17878 ts_tablet_manager.cc:1403] T 9160d9f98b8c406db2771d9ea2e0995f P 0da169ff82674dc1947055ce43de68a3: Time spent bootstrapping tablet: real 0.135s	user 0.012s	sys 0.031s
I20260709 12:20:23.759749 17881 raft_consensus.cc:359] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:23.760960 17881 raft_consensus.cc:385] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:23.761323 17881 raft_consensus.cc:740] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Initialized, Role: FOLLOWER
I20260709 12:20:23.762025 17878 raft_consensus.cc:359] T 9160d9f98b8c406db2771d9ea2e0995f P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:23.762300 17881 consensus_queue.cc:260] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:23.762770 17878 raft_consensus.cc:385] T 9160d9f98b8c406db2771d9ea2e0995f P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:23.769572 17878 raft_consensus.cc:740] T 9160d9f98b8c406db2771d9ea2e0995f P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Initialized, Role: FOLLOWER
I20260709 12:20:23.770424 17878 consensus_queue.cc:260] T 9160d9f98b8c406db2771d9ea2e0995f P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:23.774950 17881 ts_tablet_manager.cc:1434] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b: Time spent starting tablet: real 0.018s	user 0.006s	sys 0.001s
I20260709 12:20:23.776721 17881 tablet_bootstrap.cc:492] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b: Bootstrap starting.
I20260709 12:20:23.783504 17878 ts_tablet_manager.cc:1434] T 9160d9f98b8c406db2771d9ea2e0995f P 0da169ff82674dc1947055ce43de68a3: Time spent starting tablet: real 0.024s	user 0.001s	sys 0.005s
I20260709 12:20:23.785025 17878 tablet_bootstrap.cc:492] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3: Bootstrap starting.
I20260709 12:20:23.785204 17880 tablet_bootstrap.cc:492] T 8a208e793881438b975b83755dc05cfa P e2909cff1da44ad387f96a8ae3aa5afb: No bootstrap required, opened a new log
I20260709 12:20:23.785684 17880 ts_tablet_manager.cc:1403] T 8a208e793881438b975b83755dc05cfa P e2909cff1da44ad387f96a8ae3aa5afb: Time spent bootstrapping tablet: real 0.047s	user 0.016s	sys 0.014s
I20260709 12:20:23.788479 17880 raft_consensus.cc:359] T 8a208e793881438b975b83755dc05cfa P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:23.788957 18133 raft_consensus.cc:493] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:23.789124 17880 raft_consensus.cc:385] T 8a208e793881438b975b83755dc05cfa P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:23.789472 18133 raft_consensus.cc:515] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:23.791647 18133 leader_election.cc:290] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:20:23.793118 17878 tablet_bootstrap.cc:654] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:23.793255 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:20:23.794672 17801 leader_election.cc:343] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:23.798372 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:20:23.798995 17689 raft_consensus.cc:2468] T f8f6bfcb37be41c6917282a17239ad06 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 0.
I20260709 12:20:23.789438 17880 raft_consensus.cc:740] T 8a208e793881438b975b83755dc05cfa P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Initialized, Role: FOLLOWER
I20260709 12:20:23.800316 17881 tablet_bootstrap.cc:654] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:23.800582 17880 consensus_queue.cc:260] T 8a208e793881438b975b83755dc05cfa P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:23.801136 17878 tablet_bootstrap.cc:492] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3: No bootstrap required, opened a new log
I20260709 12:20:23.802695 17880 ts_tablet_manager.cc:1434] T 8a208e793881438b975b83755dc05cfa P e2909cff1da44ad387f96a8ae3aa5afb: Time spent starting tablet: real 0.017s	user 0.006s	sys 0.000s
I20260709 12:20:23.803566 17880 tablet_bootstrap.cc:492] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb: Bootstrap starting.
I20260709 12:20:23.804471 17802 leader_election.cc:304] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:20:23.806552 18133 raft_consensus.cc:2804] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:23.801551 17878 ts_tablet_manager.cc:1403] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3: Time spent bootstrapping tablet: real 0.017s	user 0.009s	sys 0.004s
I20260709 12:20:23.806931 18133 raft_consensus.cc:493] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:23.807433 18133 raft_consensus.cc:3060] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:23.809772 17878 raft_consensus.cc:359] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:23.810519 17878 raft_consensus.cc:385] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:23.810791 17878 raft_consensus.cc:740] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Initialized, Role: FOLLOWER
I20260709 12:20:23.811450 17878 consensus_queue.cc:260] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:23.817744 17880 tablet_bootstrap.cc:654] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:23.825845 17878 ts_tablet_manager.cc:1434] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3: Time spent starting tablet: real 0.018s	user 0.005s	sys 0.001s
I20260709 12:20:23.826236 18133 raft_consensus.cc:515] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:23.828989 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:20:23.829525 17689 raft_consensus.cc:3060] T f8f6bfcb37be41c6917282a17239ad06 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:23.847131 17878 tablet_bootstrap.cc:492] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3: Bootstrap starting.
I20260709 12:20:23.859422 18220 raft_consensus.cc:493] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:23.862690 18133 leader_election.cc:290] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 1 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:20:23.863682 17689 raft_consensus.cc:2468] T f8f6bfcb37be41c6917282a17239ad06 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 1.
I20260709 12:20:23.864715 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:20:23.860013 18220 raft_consensus.cc:515] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:23.865945 17802 leader_election.cc:304] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
W20260709 12:20:23.867362 17801 leader_election.cc:343] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:23.868480 18231 raft_consensus.cc:2804] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:23.869223 18231 raft_consensus.cc:697] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 1 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:20:23.870179 18231 consensus_queue.cc:237] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:23.873030 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:20:23.868034 18220 leader_election.cc:290] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:23.878396 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:20:23.878938 17878 tablet_bootstrap.cc:654] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:23.879055 17756 raft_consensus.cc:2468] T 8a208e793881438b975b83755dc05cfa P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 0.
I20260709 12:20:23.867623 17881 tablet_bootstrap.cc:492] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b: No bootstrap required, opened a new log
I20260709 12:20:23.882069 17881 ts_tablet_manager.cc:1403] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b: Time spent bootstrapping tablet: real 0.106s	user 0.020s	sys 0.021s
I20260709 12:20:23.884724 17881 raft_consensus.cc:359] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:23.885380 17881 raft_consensus.cc:385] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:23.885649 17881 raft_consensus.cc:740] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Initialized, Role: FOLLOWER
I20260709 12:20:23.886493 17881 consensus_queue.cc:260] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:23.885841 17880 tablet_bootstrap.cc:492] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb: No bootstrap required, opened a new log
I20260709 12:20:23.888737 17880 ts_tablet_manager.cc:1403] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent bootstrapping tablet: real 0.085s	user 0.023s	sys 0.044s
I20260709 12:20:23.888962 17881 ts_tablet_manager.cc:1434] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b: Time spent starting tablet: real 0.007s	user 0.006s	sys 0.000s
I20260709 12:20:23.889920 17881 tablet_bootstrap.cc:492] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b: Bootstrap starting.
I20260709 12:20:23.891239 17880 raft_consensus.cc:359] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:23.891969 17880 raft_consensus.cc:385] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:23.892300 17880 raft_consensus.cc:740] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Initialized, Role: FOLLOWER
I20260709 12:20:23.892992 17880 consensus_queue.cc:260] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:23.895284 17880 ts_tablet_manager.cc:1434] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent starting tablet: real 0.006s	user 0.004s	sys 0.000s
I20260709 12:20:23.896421 17880 tablet_bootstrap.cc:492] T 23c0cde5227f470388b8ed1fc0786119 P e2909cff1da44ad387f96a8ae3aa5afb: Bootstrap starting.
I20260709 12:20:23.902005 17881 tablet_bootstrap.cc:654] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:23.903414 17878 tablet_bootstrap.cc:492] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3: No bootstrap required, opened a new log
I20260709 12:20:23.903930 17878 ts_tablet_manager.cc:1403] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3: Time spent bootstrapping tablet: real 0.057s	user 0.013s	sys 0.007s
I20260709 12:20:23.906090 17880 tablet_bootstrap.cc:654] T 23c0cde5227f470388b8ed1fc0786119 P e2909cff1da44ad387f96a8ae3aa5afb: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:23.906766 17878 raft_consensus.cc:359] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:23.907497 17878 raft_consensus.cc:385] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:23.907985 17878 raft_consensus.cc:740] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Initialized, Role: FOLLOWER
I20260709 12:20:23.908761 17878 consensus_queue.cc:260] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:23.910995 17878 ts_tablet_manager.cc:1434] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3: Time spent starting tablet: real 0.007s	user 0.004s	sys 0.001s
I20260709 12:20:23.916523 17652 leader_election.cc:304] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:20:23.924867 17652 leader_election.cc:343] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:23.918455 17880 tablet_bootstrap.cc:492] T 23c0cde5227f470388b8ed1fc0786119 P e2909cff1da44ad387f96a8ae3aa5afb: No bootstrap required, opened a new log
I20260709 12:20:23.925706 17880 ts_tablet_manager.cc:1403] T 23c0cde5227f470388b8ed1fc0786119 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent bootstrapping tablet: real 0.030s	user 0.015s	sys 0.004s
I20260709 12:20:23.926519 18220 raft_consensus.cc:2804] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:23.921154 17881 tablet_bootstrap.cc:492] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b: No bootstrap required, opened a new log
I20260709 12:20:23.926923 18220 raft_consensus.cc:493] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:23.927392 17881 ts_tablet_manager.cc:1403] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b: Time spent bootstrapping tablet: real 0.038s	user 0.006s	sys 0.012s
I20260709 12:20:23.927287 18122 consensus_queue.cc:1048] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:23.927289 18220 raft_consensus.cc:3060] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:23.928364 17880 raft_consensus.cc:359] T 23c0cde5227f470388b8ed1fc0786119 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:23.929059 17880 raft_consensus.cc:385] T 23c0cde5227f470388b8ed1fc0786119 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:23.929335 17880 raft_consensus.cc:740] T 23c0cde5227f470388b8ed1fc0786119 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Initialized, Role: FOLLOWER
I20260709 12:20:23.930932 17880 consensus_queue.cc:260] T 23c0cde5227f470388b8ed1fc0786119 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:23.932231 17878 tablet_bootstrap.cc:492] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3: Bootstrap starting.
I20260709 12:20:23.933431 17880 ts_tablet_manager.cc:1434] T 23c0cde5227f470388b8ed1fc0786119 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent starting tablet: real 0.007s	user 0.006s	sys 0.000s
I20260709 12:20:23.934579 17880 tablet_bootstrap.cc:492] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb: Bootstrap starting.
I20260709 12:20:23.939567 17878 tablet_bootstrap.cc:654] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:23.941169 17880 tablet_bootstrap.cc:654] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:23.947422 17584 catalog_manager.cc:5697] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b reported cstate change: term changed from 0 to 1, leader changed from <none> to e063541542a94969898868bf6eb5ce1b (127.16.29.67). New cstate: current_term: 1 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: HEALTHY } } }
I20260709 12:20:23.977347 17881 raft_consensus.cc:359] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:23.978319 17881 raft_consensus.cc:385] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:23.978674 17881 raft_consensus.cc:740] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Initialized, Role: FOLLOWER
I20260709 12:20:23.978434 18230 consensus_queue.cc:1048] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:20:23.979614 17881 consensus_queue.cc:260] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:23.984704 18230 consensus_queue.cc:1048] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:20:23.984769 17881 ts_tablet_manager.cc:1434] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b: Time spent starting tablet: real 0.057s	user 0.009s	sys 0.000s
I20260709 12:20:23.993572 18220 raft_consensus.cc:515] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:23.997468 17835 raft_consensus.cc:3060] T 12b1fcd35e12402e8d6df7504c578e01 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:24.004707 17881 tablet_bootstrap.cc:492] T 50f9ff8f24a14a079db34b1ba88fa278 P e063541542a94969898868bf6eb5ce1b: Bootstrap starting.
I20260709 12:20:24.008607 18220 leader_election.cc:290] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:24.009570 17935 raft_consensus.cc:493] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:24.010028 17935 raft_consensus.cc:515] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:24.012696 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:20:24.013315 17758 raft_consensus.cc:3060] T 8a208e793881438b975b83755dc05cfa P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:24.021852 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:20:24.022550 17838 raft_consensus.cc:3060] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:24.023422 17758 raft_consensus.cc:2468] T 8a208e793881438b975b83755dc05cfa P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 1.
I20260709 12:20:24.022584 18220 raft_consensus.cc:493] T 9160d9f98b8c406db2771d9ea2e0995f P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:24.025564 18220 raft_consensus.cc:515] T 9160d9f98b8c406db2771d9ea2e0995f P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:24.025027 17880 tablet_bootstrap.cc:492] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb: No bootstrap required, opened a new log
I20260709 12:20:24.027310 17880 ts_tablet_manager.cc:1403] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent bootstrapping tablet: real 0.093s	user 0.022s	sys 0.029s
I20260709 12:20:24.029237 17935 leader_election.cc:290] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:24.035933 18055 consensus_queue.cc:1048] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:24.037724 17652 leader_election.cc:304] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:20:24.039811 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:20:24.039646 18219 raft_consensus.cc:2804] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:24.037591 18222 consensus_queue.cc:1048] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20260709 12:20:24.039250 17759 raft_consensus.cc:3060] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:24.041055 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:20:24.051167 17912 raft_consensus.cc:493] T 12b1fcd35e12402e8d6df7504c578e01 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:24.051676 17912 raft_consensus.cc:515] T 12b1fcd35e12402e8d6df7504c578e01 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:24.053546 17912 leader_election.cc:290] T 12b1fcd35e12402e8d6df7504c578e01 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:20:24.066238 17881 tablet_bootstrap.cc:654] T 50f9ff8f24a14a079db34b1ba88fa278 P e063541542a94969898868bf6eb5ce1b: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:24.067569 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:20:24.069348 18219 raft_consensus.cc:697] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 1 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:20:24.076273 17878 tablet_bootstrap.cc:492] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3: No bootstrap required, opened a new log
I20260709 12:20:24.076747 17878 ts_tablet_manager.cc:1403] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3: Time spent bootstrapping tablet: real 0.145s	user 0.017s	sys 0.029s
I20260709 12:20:24.083220 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:20:24.084256 17835 raft_consensus.cc:2393] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 1: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:20:24.075652 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:20:24.087512 17728 leader_election.cc:304] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:20:24.077255 18220 leader_election.cc:290] T 9160d9f98b8c406db2771d9ea2e0995f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:24.078992 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:20:24.086408 18223 raft_consensus.cc:493] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:24.100135 17652 leader_election.cc:304] T 9160d9f98b8c406db2771d9ea2e0995f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:20:24.096526 18223 raft_consensus.cc:515] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:24.102632 18055 raft_consensus.cc:3060] T 9160d9f98b8c406db2771d9ea2e0995f P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:24.103929 17878 raft_consensus.cc:359] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:24.109174 17878 raft_consensus.cc:385] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:24.109546 17878 raft_consensus.cc:740] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Initialized, Role: FOLLOWER
I20260709 12:20:24.118106 17801 leader_election.cc:304] T 12b1fcd35e12402e8d6df7504c578e01 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:20:24.127202 17878 consensus_queue.cc:260] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:24.133440 17935 raft_consensus.cc:2749] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:24.136839 17880 raft_consensus.cc:359] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:24.138315 17880 raft_consensus.cc:385] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:24.140115 17952 raft_consensus.cc:2749] T 12b1fcd35e12402e8d6df7504c578e01 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:20:24.139349 17880 raft_consensus.cc:740] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Initialized, Role: FOLLOWER
I20260709 12:20:24.092844 18219 consensus_queue.cc:237] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:24.141811 17880 consensus_queue.cc:260] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:24.145859 17878 ts_tablet_manager.cc:1434] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3: Time spent starting tablet: real 0.069s	user 0.014s	sys 0.003s
I20260709 12:20:24.151221 17880 ts_tablet_manager.cc:1434] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent starting tablet: real 0.124s	user 0.008s	sys 0.000s
I20260709 12:20:24.144918 17838 raft_consensus.cc:2468] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 1.
I20260709 12:20:24.163369 17878 tablet_bootstrap.cc:492] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3: Bootstrap starting.
I20260709 12:20:24.148669 18122 consensus_queue.cc:1048] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.013s
I20260709 12:20:24.172341 17880 tablet_bootstrap.cc:492] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb: Bootstrap starting.
I20260709 12:20:24.165330 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:20:24.168937 18223 leader_election.cc:290] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:24.176710 18214 consensus_queue.cc:1048] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.005s
I20260709 12:20:24.185768 17881 tablet_bootstrap.cc:492] T 50f9ff8f24a14a079db34b1ba88fa278 P e063541542a94969898868bf6eb5ce1b: No bootstrap required, opened a new log
I20260709 12:20:24.188146 17881 ts_tablet_manager.cc:1403] T 50f9ff8f24a14a079db34b1ba88fa278 P e063541542a94969898868bf6eb5ce1b: Time spent bootstrapping tablet: real 0.191s	user 0.013s	sys 0.005s
I20260709 12:20:24.198835 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:20:24.197152 18055 raft_consensus.cc:2749] T 9160d9f98b8c406db2771d9ea2e0995f P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:24.205955 17881 raft_consensus.cc:359] T 50f9ff8f24a14a079db34b1ba88fa278 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:24.207139 17881 raft_consensus.cc:385] T 50f9ff8f24a14a079db34b1ba88fa278 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:24.207479 17881 raft_consensus.cc:740] T 50f9ff8f24a14a079db34b1ba88fa278 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Initialized, Role: FOLLOWER
I20260709 12:20:24.208276 17881 consensus_queue.cc:260] T 50f9ff8f24a14a079db34b1ba88fa278 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:24.211189 17881 ts_tablet_manager.cc:1434] T 50f9ff8f24a14a079db34b1ba88fa278 P e063541542a94969898868bf6eb5ce1b: Time spent starting tablet: real 0.014s	user 0.006s	sys 0.000s
I20260709 12:20:24.212371 17683 raft_consensus.cc:3060] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:20:24.209101 17652 leader_election.cc:343] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:24.225554 18230 consensus_queue.cc:1048] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260709 12:20:24.218393 17652 leader_election.cc:343] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:24.229698 17652 leader_election.cc:304] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:20:24.230237 18230 consensus_queue.cc:1048] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:20:24.231050 18219 raft_consensus.cc:2749] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:24.245180 17881 tablet_bootstrap.cc:492] T 9439565c5dbb4f4ba3375aaef2c9add5 P e063541542a94969898868bf6eb5ce1b: Bootstrap starting.
I20260709 12:20:24.374292 17878 tablet_bootstrap.cc:654] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:24.381054 18183 raft_consensus.cc:493] T 23c0cde5227f470388b8ed1fc0786119 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:24.381574 18183 raft_consensus.cc:515] T 23c0cde5227f470388b8ed1fc0786119 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:24.383982 17880 tablet_bootstrap.cc:654] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb: Neither blocks nor log segments found. Creating new log.
W20260709 12:20:24.388610 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 119248us
I20260709 12:20:24.391921 18183 leader_election.cc:290] T 23c0cde5227f470388b8ed1fc0786119 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:24.382704 18230 raft_consensus.cc:493] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:24.393093 18230 raft_consensus.cc:515] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:24.394995 18230 leader_election.cc:290] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:24.388937 17584 catalog_manager.cc:5697] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0da169ff82674dc1947055ce43de68a3 (127.16.29.65). New cstate: current_term: 1 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:24.422590 17912 raft_consensus.cc:493] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:24.423110 17912 raft_consensus.cc:515] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:24.425802 17762 raft_consensus.cc:3060] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:24.432354 17912 leader_election.cc:290] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:20:24.433629 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:20:24.438210 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:20:24.441756 17635 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:20:24.443547 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:20:24.451045 17881 tablet_bootstrap.cc:654] T 9439565c5dbb4f4ba3375aaef2c9add5 P e063541542a94969898868bf6eb5ce1b: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:24.457566 18233 raft_consensus.cc:493] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:20:24.462028 17727 leader_election.cc:343] T 23c0cde5227f470388b8ed1fc0786119 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:24.466082 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:20:24.466408 18133 consensus_queue.cc:1048] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.007s
I20260709 12:20:24.475647 17878 tablet_bootstrap.cc:492] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3: No bootstrap required, opened a new log
I20260709 12:20:24.476182 17878 ts_tablet_manager.cc:1403] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3: Time spent bootstrapping tablet: real 0.321s	user 0.018s	sys 0.006s
I20260709 12:20:24.476260 18219 raft_consensus.cc:493] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:24.476629 17880 tablet_bootstrap.cc:492] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb: No bootstrap required, opened a new log
I20260709 12:20:24.477106 17880 ts_tablet_manager.cc:1403] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent bootstrapping tablet: real 0.316s	user 0.022s	sys 0.001s
I20260709 12:20:24.476838 18219 raft_consensus.cc:515] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:20:24.479456 17802 leader_election.cc:343] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:24.458099 18233 raft_consensus.cc:515] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:24.487206 17878 raft_consensus.cc:359] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:24.488196 17878 raft_consensus.cc:385] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:24.488765 17878 raft_consensus.cc:740] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Initialized, Role: FOLLOWER
I20260709 12:20:24.489459 17878 consensus_queue.cc:260] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:24.491815 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:20:24.485806 18231 consensus_queue.cc:1048] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.010s
I20260709 12:20:24.499845 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:20:24.501006 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:20:24.502281 18219 leader_election.cc:290] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:24.503234 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:20:24.503757 18214 consensus_queue.cc:1048] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260709 12:20:24.506193 17801 leader_election.cc:343] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:24.500022 18131 raft_consensus.cc:493] T 50f9ff8f24a14a079db34b1ba88fa278 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:24.506707 17801 leader_election.cc:304] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:20:24.515244 18228 raft_consensus.cc:2749] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:24.520431 17878 ts_tablet_manager.cc:1434] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3: Time spent starting tablet: real 0.044s	user 0.006s	sys 0.001s
I20260709 12:20:24.519256 17728 leader_election.cc:304] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:20:24.508949 18131 raft_consensus.cc:515] T 50f9ff8f24a14a079db34b1ba88fa278 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:24.530947 18233 leader_election.cc:290] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:24.535168 17878 tablet_bootstrap.cc:492] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3: Bootstrap starting.
W20260709 12:20:24.538861 18283 log.cc:927] Time spent T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.099s	user 0.000s	sys 0.003s
I20260709 12:20:24.549329 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:20:24.550988 17652 leader_election.cc:304] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:20:24.524085 18183 raft_consensus.cc:2749] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:24.556897 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:20:24.558224 18131 leader_election.cc:290] T 50f9ff8f24a14a079db34b1ba88fa278 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:20:24.564565 17727 leader_election.cc:343] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:20:24.566802 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 43696.1us
I20260709 12:20:24.570225 18275 raft_consensus.cc:2749] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260709 12:20:24.574951 17728 leader_election.cc:343] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:24.575369 17728 leader_election.cc:304] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:20:24.575217 17880 raft_consensus.cc:359] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:24.575912 17880 raft_consensus.cc:385] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:24.576368 17880 raft_consensus.cc:740] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Initialized, Role: FOLLOWER
I20260709 12:20:24.585222 17880 consensus_queue.cc:260] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:24.586455 18183 raft_consensus.cc:493] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:24.587260 17881 tablet_bootstrap.cc:492] T 9439565c5dbb4f4ba3375aaef2c9add5 P e063541542a94969898868bf6eb5ce1b: No bootstrap required, opened a new log
I20260709 12:20:24.586982 18183 raft_consensus.cc:515] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:24.587867 17881 ts_tablet_manager.cc:1403] T 9439565c5dbb4f4ba3375aaef2c9add5 P e063541542a94969898868bf6eb5ce1b: Time spent bootstrapping tablet: real 0.375s	user 0.031s	sys 0.034s
I20260709 12:20:24.591220 18183 leader_election.cc:290] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:24.591676 17880 ts_tablet_manager.cc:1434] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent starting tablet: real 0.114s	user 0.008s	sys 0.000s
W20260709 12:20:24.592444 17728 leader_election.cc:343] T 23c0cde5227f470388b8ed1fc0786119 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:24.594177 17728 leader_election.cc:304] T 23c0cde5227f470388b8ed1fc0786119 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:20:24.596287 18288 raft_consensus.cc:2749] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:24.608490 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:20:24.608884 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:20:24.602488 17881 raft_consensus.cc:359] T 9439565c5dbb4f4ba3375aaef2c9add5 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:24.609689 17881 raft_consensus.cc:385] T 9439565c5dbb4f4ba3375aaef2c9add5 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:24.610069 17881 raft_consensus.cc:740] T 9439565c5dbb4f4ba3375aaef2c9add5 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Initialized, Role: FOLLOWER
I20260709 12:20:24.608532 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:20:24.612917 17727 leader_election.cc:343] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:24.599784 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:20:24.616076 17880 tablet_bootstrap.cc:492] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb: Bootstrap starting.
I20260709 12:20:24.617462 17878 tablet_bootstrap.cc:654] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3: Neither blocks nor log segments found. Creating new log.
W20260709 12:20:24.618117 17801 leader_election.cc:343] T 50f9ff8f24a14a079db34b1ba88fa278 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:20:24.620442 17728 leader_election.cc:343] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:24.620771 17728 leader_election.cc:304] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
W20260709 12:20:24.624331 17802 leader_election.cc:343] T 50f9ff8f24a14a079db34b1ba88fa278 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:24.624750 17802 leader_election.cc:304] T 50f9ff8f24a14a079db34b1ba88fa278 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:20:24.625690 18214 raft_consensus.cc:2749] T 50f9ff8f24a14a079db34b1ba88fa278 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:24.626677 18230 raft_consensus.cc:2749] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:24.646950 18275 consensus_queue.cc:1048] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:20:24.650681 17881 consensus_queue.cc:260] T 9439565c5dbb4f4ba3375aaef2c9add5 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:24.596321 18294 raft_consensus.cc:2749] T 23c0cde5227f470388b8ed1fc0786119 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:24.665939 17880 tablet_bootstrap.cc:654] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:24.673786 17881 ts_tablet_manager.cc:1434] T 9439565c5dbb4f4ba3375aaef2c9add5 P e063541542a94969898868bf6eb5ce1b: Time spent starting tablet: real 0.084s	user 0.008s	sys 0.001s
I20260709 12:20:24.674851 17881 tablet_bootstrap.cc:492] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b: Bootstrap starting.
I20260709 12:20:24.698149 17880 tablet_bootstrap.cc:492] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb: No bootstrap required, opened a new log
I20260709 12:20:24.698794 17880 ts_tablet_manager.cc:1403] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb: Time spent bootstrapping tablet: real 0.083s	user 0.012s	sys 0.014s
I20260709 12:20:24.701205 18222 consensus_queue.cc:1048] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:20:24.702507 17880 raft_consensus.cc:359] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:24.703469 17880 raft_consensus.cc:385] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:24.703836 17880 raft_consensus.cc:740] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Initialized, Role: FOLLOWER
I20260709 12:20:24.704698 17880 consensus_queue.cc:260] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:24.708461 17880 ts_tablet_manager.cc:1434] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb: Time spent starting tablet: real 0.009s	user 0.009s	sys 0.001s
I20260709 12:20:24.709687 17880 tablet_bootstrap.cc:492] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb: Bootstrap starting.
I20260709 12:20:24.715770 17881 tablet_bootstrap.cc:654] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:24.717042 17880 tablet_bootstrap.cc:654] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:24.723668 17878 tablet_bootstrap.cc:492] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3: No bootstrap required, opened a new log
I20260709 12:20:24.725879 17878 ts_tablet_manager.cc:1403] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3: Time spent bootstrapping tablet: real 0.198s	user 0.048s	sys 0.012s
I20260709 12:20:24.735600 17878 raft_consensus.cc:359] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:24.736343 17878 raft_consensus.cc:385] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:24.736744 17878 raft_consensus.cc:740] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Initialized, Role: FOLLOWER
I20260709 12:20:24.737434 17878 consensus_queue.cc:260] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:24.738585 17880 tablet_bootstrap.cc:492] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb: No bootstrap required, opened a new log
I20260709 12:20:24.739096 17880 ts_tablet_manager.cc:1403] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent bootstrapping tablet: real 0.030s	user 0.011s	sys 0.001s
I20260709 12:20:24.739770 17878 ts_tablet_manager.cc:1434] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3: Time spent starting tablet: real 0.013s	user 0.005s	sys 0.000s
I20260709 12:20:24.741768 17880 raft_consensus.cc:359] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:24.742441 17880 raft_consensus.cc:385] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:24.742776 17880 raft_consensus.cc:740] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Initialized, Role: FOLLOWER
I20260709 12:20:24.743188 17878 tablet_bootstrap.cc:492] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3: Bootstrap starting.
I20260709 12:20:24.743435 17880 consensus_queue.cc:260] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:24.745877 17880 ts_tablet_manager.cc:1434] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent starting tablet: real 0.006s	user 0.005s	sys 0.000s
I20260709 12:20:24.746918 17880 tablet_bootstrap.cc:492] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb: Bootstrap starting.
I20260709 12:20:24.751461 17878 tablet_bootstrap.cc:654] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:24.753365 17880 tablet_bootstrap.cc:654] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:24.765602 17878 tablet_bootstrap.cc:492] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3: No bootstrap required, opened a new log
I20260709 12:20:24.766052 17878 ts_tablet_manager.cc:1403] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3: Time spent bootstrapping tablet: real 0.023s	user 0.009s	sys 0.008s
I20260709 12:20:24.768626 17878 raft_consensus.cc:359] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:24.769503 17878 raft_consensus.cc:385] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:24.769764 17878 raft_consensus.cc:740] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Initialized, Role: FOLLOWER
I20260709 12:20:24.770404 17878 consensus_queue.cc:260] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:24.772711 17878 ts_tablet_manager.cc:1434] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3: Time spent starting tablet: real 0.006s	user 0.005s	sys 0.000s
I20260709 12:20:24.773548 17878 tablet_bootstrap.cc:492] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3: Bootstrap starting.
I20260709 12:20:24.779593 17878 tablet_bootstrap.cc:654] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:24.784861 17881 tablet_bootstrap.cc:492] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b: No bootstrap required, opened a new log
I20260709 12:20:24.785460 17881 ts_tablet_manager.cc:1403] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b: Time spent bootstrapping tablet: real 0.111s	user 0.010s	sys 0.000s
I20260709 12:20:24.796576 17878 tablet_bootstrap.cc:492] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3: No bootstrap required, opened a new log
I20260709 12:20:24.789625 17881 raft_consensus.cc:359] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:24.801059 17878 ts_tablet_manager.cc:1403] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3: Time spent bootstrapping tablet: real 0.028s	user 0.017s	sys 0.000s
I20260709 12:20:24.801095 17881 raft_consensus.cc:385] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:24.801458 17881 raft_consensus.cc:740] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Initialized, Role: FOLLOWER
I20260709 12:20:24.802423 17881 consensus_queue.cc:260] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:24.804636 17878 raft_consensus.cc:359] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:24.805334 17878 raft_consensus.cc:385] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:24.805619 17878 raft_consensus.cc:740] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Initialized, Role: FOLLOWER
I20260709 12:20:24.804636 17881 ts_tablet_manager.cc:1434] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b: Time spent starting tablet: real 0.019s	user 0.007s	sys 0.000s
I20260709 12:20:24.806262 17878 consensus_queue.cc:260] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:24.806743 17881 tablet_bootstrap.cc:492] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b: Bootstrap starting.
I20260709 12:20:24.807623 17880 tablet_bootstrap.cc:492] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb: No bootstrap required, opened a new log
I20260709 12:20:24.808122 17880 ts_tablet_manager.cc:1403] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent bootstrapping tablet: real 0.061s	user 0.017s	sys 0.042s
I20260709 12:20:24.813448 17878 ts_tablet_manager.cc:1434] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3: Time spent starting tablet: real 0.012s	user 0.006s	sys 0.000s
I20260709 12:20:24.813179 17880 raft_consensus.cc:359] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:24.813896 17880 raft_consensus.cc:385] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:24.814221 17880 raft_consensus.cc:740] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Initialized, Role: FOLLOWER
I20260709 12:20:24.814496 17878 tablet_bootstrap.cc:492] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3: Bootstrap starting.
I20260709 12:20:24.815078 17880 consensus_queue.cc:260] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:24.818420 17880 ts_tablet_manager.cc:1434] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent starting tablet: real 0.010s	user 0.006s	sys 0.000s
I20260709 12:20:24.818878 18219 raft_consensus.cc:493] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:24.819434 17881 tablet_bootstrap.cc:654] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:24.819391 18219 raft_consensus.cc:515] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:24.821810 18219 leader_election.cc:290] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:24.822971 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:20:24.823518 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:20:24.823783 17763 raft_consensus.cc:2468] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 0.
I20260709 12:20:24.824119 17834 raft_consensus.cc:2468] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 0.
I20260709 12:20:24.825455 17652 leader_election.cc:304] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:20:24.827178 18219 raft_consensus.cc:2804] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:24.827591 18219 raft_consensus.cc:493] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:24.827966 18219 raft_consensus.cc:3060] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:24.828442 17878 tablet_bootstrap.cc:654] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:24.837251 17880 tablet_bootstrap.cc:492] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb: Bootstrap starting.
I20260709 12:20:24.847450 18219 raft_consensus.cc:515] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:24.851272 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:20:24.851898 17763 raft_consensus.cc:3060] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:24.856150 18294 raft_consensus.cc:493] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:24.856678 18294 raft_consensus.cc:515] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:24.858804 18294 leader_election.cc:290] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:24.860316 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:20:24.860934 17834 raft_consensus.cc:2468] T 9439565c5dbb4f4ba3375aaef2c9add5 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 0.
I20260709 12:20:24.861912 17763 raft_consensus.cc:2468] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 1.
I20260709 12:20:24.862066 17727 leader_election.cc:304] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:20:24.862804 18294 raft_consensus.cc:2804] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:24.862933 17652 leader_election.cc:304] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:20:24.863147 18294 raft_consensus.cc:493] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:24.863694 18222 raft_consensus.cc:2804] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:24.863736 18294 raft_consensus.cc:3060] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:24.863838 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:20:24.864306 18222 raft_consensus.cc:697] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 1 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:20:24.864570 17687 raft_consensus.cc:2468] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 0.
I20260709 12:20:24.865484 17878 tablet_bootstrap.cc:492] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3: No bootstrap required, opened a new log
I20260709 12:20:24.865187 18222 consensus_queue.cc:237] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:24.866034 17878 ts_tablet_manager.cc:1403] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3: Time spent bootstrapping tablet: real 0.052s	user 0.017s	sys 0.021s
I20260709 12:20:24.868528 18219 leader_election.cc:290] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:24.868781 17878 raft_consensus.cc:359] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:24.869587 17878 raft_consensus.cc:385] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:24.869837 17880 tablet_bootstrap.cc:654] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:24.869966 17878 raft_consensus.cc:740] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Initialized, Role: FOLLOWER
I20260709 12:20:24.870723 17878 consensus_queue.cc:260] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:24.872984 17878 ts_tablet_manager.cc:1434] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3: Time spent starting tablet: real 0.007s	user 0.005s	sys 0.001s
I20260709 12:20:24.873937 17878 tablet_bootstrap.cc:492] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3: Bootstrap starting.
I20260709 12:20:24.869974 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:20:24.876052 17834 raft_consensus.cc:3060] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:24.878916 18294 raft_consensus.cc:515] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:24.879629 17881 tablet_bootstrap.cc:492] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b: No bootstrap required, opened a new log
I20260709 12:20:24.879591 17878 tablet_bootstrap.cc:654] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:24.880175 17881 ts_tablet_manager.cc:1403] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b: Time spent bootstrapping tablet: real 0.074s	user 0.023s	sys 0.002s
I20260709 12:20:24.882184 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:20:24.882737 17687 raft_consensus.cc:3060] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:24.882993 17881 raft_consensus.cc:359] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:24.883637 17881 raft_consensus.cc:385] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:24.883925 17881 raft_consensus.cc:740] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Initialized, Role: FOLLOWER
I20260709 12:20:24.884552 17834 raft_consensus.cc:2468] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 1.
I20260709 12:20:24.888618 18294 leader_election.cc:290] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 1 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:24.890193 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:20:24.890959 17838 raft_consensus.cc:3060] T 9439565c5dbb4f4ba3375aaef2c9add5 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:24.901194 17880 tablet_bootstrap.cc:492] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb: No bootstrap required, opened a new log
I20260709 12:20:24.901839 17880 ts_tablet_manager.cc:1403] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent bootstrapping tablet: real 0.065s	user 0.010s	sys 0.008s
I20260709 12:20:24.905555 17880 raft_consensus.cc:359] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:24.906481 17880 raft_consensus.cc:385] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:24.908294 17586 catalog_manager.cc:5697] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0da169ff82674dc1947055ce43de68a3 (127.16.29.65). New cstate: current_term: 1 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:24.908385 18222 raft_consensus.cc:493] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:24.916087 18222 raft_consensus.cc:515] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:24.918702 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:20:24.919256 17763 raft_consensus.cc:2468] T 23c0cde5227f470388b8ed1fc0786119 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 0.
I20260709 12:20:24.920732 17652 leader_election.cc:304] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:20:24.920786 17880 raft_consensus.cc:740] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Initialized, Role: FOLLOWER
I20260709 12:20:24.921494 18219 raft_consensus.cc:2804] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:24.921895 18219 raft_consensus.cc:493] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:24.922254 18219 raft_consensus.cc:3060] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:24.923759 17880 consensus_queue.cc:260] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:24.929167 18219 raft_consensus.cc:515] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:24.931162 17880 ts_tablet_manager.cc:1434] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent starting tablet: real 0.029s	user 0.009s	sys 0.000s
I20260709 12:20:24.931949 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:20:24.932217 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:20:24.932528 17756 raft_consensus.cc:3060] T 23c0cde5227f470388b8ed1fc0786119 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:24.932621 17878 tablet_bootstrap.cc:492] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3: No bootstrap required, opened a new log
I20260709 12:20:24.932760 17835 raft_consensus.cc:2468] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 0.
I20260709 12:20:24.887629 17687 raft_consensus.cc:2468] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 1.
I20260709 12:20:24.887900 18183 raft_consensus.cc:493] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:24.933208 17878 ts_tablet_manager.cc:1403] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3: Time spent bootstrapping tablet: real 0.059s	user 0.021s	sys 0.023s
I20260709 12:20:24.936659 18183 raft_consensus.cc:515] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:24.939337 18183 leader_election.cc:290] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:24.940263 17878 raft_consensus.cc:359] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:24.940997 17878 raft_consensus.cc:385] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:24.941133 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:20:24.941318 17878 raft_consensus.cc:740] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Initialized, Role: FOLLOWER
I20260709 12:20:24.942215 17878 consensus_queue.cc:260] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:24.943094 17728 leader_election.cc:304] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:20:24.944027 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:20:24.944474 17878 ts_tablet_manager.cc:1434] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3: Time spent starting tablet: real 0.008s	user 0.006s	sys 0.000s
I20260709 12:20:24.885170 17881 consensus_queue.cc:260] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:24.944623 17687 raft_consensus.cc:2468] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 0.
I20260709 12:20:24.945977 17838 raft_consensus.cc:2468] T 9439565c5dbb4f4ba3375aaef2c9add5 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 1.
I20260709 12:20:24.946252 17834 raft_consensus.cc:2468] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 0.
I20260709 12:20:24.952250 18275 raft_consensus.cc:493] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:24.952726 18275 raft_consensus.cc:515] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:24.955574 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:20:24.956116 17763 raft_consensus.cc:2468] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 0.
I20260709 12:20:24.956660 18275 leader_election.cc:290] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:24.951527 17728 leader_election.cc:304] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:20:24.958173 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:20:24.959239 17652 leader_election.cc:304] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:20:24.946887 18222 leader_election.cc:290] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:20:24.960093 17652 leader_election.cc:343] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:24.948239 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:20:24.961401 17835 raft_consensus.cc:3060] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:24.946219 18294 raft_consensus.cc:2804] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:24.964318 18294 raft_consensus.cc:697] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:20:24.959921 18275 raft_consensus.cc:2804] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:24.966521 18183 raft_consensus.cc:2804] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:24.966944 18183 raft_consensus.cc:493] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:24.967262 18183 raft_consensus.cc:3060] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:24.965864 18275 raft_consensus.cc:493] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:24.974231 18275 raft_consensus.cc:3060] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:24.978363 17881 ts_tablet_manager.cc:1434] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b: Time spent starting tablet: real 0.098s	user 0.007s	sys 0.000s
I20260709 12:20:24.982589 17881 tablet_bootstrap.cc:492] T 56b500bd1b914ebfb3f19afc8a330665 P e063541542a94969898868bf6eb5ce1b: Bootstrap starting.
I20260709 12:20:24.947999 18219 leader_election.cc:290] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:24.966452 18294 consensus_queue.cc:237] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:24.978652 17835 raft_consensus.cc:2468] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 1.
I20260709 12:20:24.985241 17652 leader_election.cc:304] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:20:24.989877 18276 raft_consensus.cc:2804] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:24.990409 18276 raft_consensus.cc:697] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 1 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:20:24.991299 18276 consensus_queue.cc:237] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:25.003589 17881 tablet_bootstrap.cc:654] T 56b500bd1b914ebfb3f19afc8a330665 P e063541542a94969898868bf6eb5ce1b: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:24.944521 17756 raft_consensus.cc:2468] T 23c0cde5227f470388b8ed1fc0786119 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 1.
I20260709 12:20:25.021420 18275 raft_consensus.cc:515] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:25.021847 17586 catalog_manager.cc:5697] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb reported cstate change: term changed from 0 to 1, leader changed from <none> to e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66). New cstate: current_term: 1 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:25.023809 18275 leader_election.cc:290] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:25.025681 18183 raft_consensus.cc:515] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:25.039911 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:20:25.039885 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:20:25.040741 17761 raft_consensus.cc:3060] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:25.048626 17586 catalog_manager.cc:5697] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0da169ff82674dc1947055ce43de68a3 (127.16.29.65). New cstate: current_term: 1 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:25.051391 17761 raft_consensus.cc:2468] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 1.
W20260709 12:20:25.052192 17652 leader_election.cc:343] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:25.054153 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:20:25.054664 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:20:25.055567 17687 raft_consensus.cc:3060] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:25.056010 17652 leader_election.cc:304] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:20:25.056849 18222 raft_consensus.cc:2804] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:25.057484 18222 raft_consensus.cc:697] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 1 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:20:25.058380 18222 consensus_queue.cc:237] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:25.058948 17881 tablet_bootstrap.cc:492] T 56b500bd1b914ebfb3f19afc8a330665 P e063541542a94969898868bf6eb5ce1b: No bootstrap required, opened a new log
I20260709 12:20:25.059419 17881 ts_tablet_manager.cc:1403] T 56b500bd1b914ebfb3f19afc8a330665 P e063541542a94969898868bf6eb5ce1b: Time spent bootstrapping tablet: real 0.077s	user 0.016s	sys 0.022s
I20260709 12:20:25.062325 17687 raft_consensus.cc:2468] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 1.
I20260709 12:20:25.062779 17881 raft_consensus.cc:359] T 56b500bd1b914ebfb3f19afc8a330665 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:25.063354 17728 leader_election.cc:304] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:20:25.063508 17881 raft_consensus.cc:385] T 56b500bd1b914ebfb3f19afc8a330665 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:25.053231 18183 leader_election.cc:290] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 1 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:25.063870 17881 raft_consensus.cc:740] T 56b500bd1b914ebfb3f19afc8a330665 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Initialized, Role: FOLLOWER
I20260709 12:20:25.064086 18185 raft_consensus.cc:2804] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:25.064610 18185 raft_consensus.cc:697] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:20:25.064486 17835 raft_consensus.cc:3060] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:25.065062 17881 consensus_queue.cc:260] T 56b500bd1b914ebfb3f19afc8a330665 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:25.065380 18185 consensus_queue.cc:237] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:25.067386 17881 ts_tablet_manager.cc:1434] T 56b500bd1b914ebfb3f19afc8a330665 P e063541542a94969898868bf6eb5ce1b: Time spent starting tablet: real 0.008s	user 0.007s	sys 0.000s
I20260709 12:20:25.068410 17881 tablet_bootstrap.cc:492] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b: Bootstrap starting.
I20260709 12:20:25.072819 18275 raft_consensus.cc:493] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:25.073722 18275 raft_consensus.cc:515] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:25.076215 18275 leader_election.cc:290] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:25.079262 17881 tablet_bootstrap.cc:654] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:25.080402 17835 raft_consensus.cc:2468] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 1.
I20260709 12:20:25.081202 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:20:25.081995 17761 raft_consensus.cc:2468] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 0.
I20260709 12:20:25.087428 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:20:25.089532 17587 catalog_manager.cc:5697] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0da169ff82674dc1947055ce43de68a3 (127.16.29.65). New cstate: current_term: 1 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:25.092370 17652 leader_election.cc:304] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:20:25.093508 18222 raft_consensus.cc:2804] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:25.093976 18222 raft_consensus.cc:493] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:25.094409 18222 raft_consensus.cc:3060] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:20:25.098131 17652 leader_election.cc:343] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:25.119297 18122 raft_consensus.cc:493] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:25.119797 18122 raft_consensus.cc:515] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:25.120021 18222 raft_consensus.cc:515] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:25.121803 17881 tablet_bootstrap.cc:492] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b: No bootstrap required, opened a new log
I20260709 12:20:25.121932 18122 leader_election.cc:290] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:25.122032 18222 leader_election.cc:290] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:25.122460 17881 ts_tablet_manager.cc:1403] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b: Time spent bootstrapping tablet: real 0.054s	user 0.029s	sys 0.014s
I20260709 12:20:25.123104 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:20:25.123852 17761 raft_consensus.cc:2468] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 0.
I20260709 12:20:25.124361 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:20:25.124670 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:20:25.124943 17756 raft_consensus.cc:3060] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:25.125357 17835 raft_consensus.cc:2468] T 50f9ff8f24a14a079db34b1ba88fa278 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 0.
I20260709 12:20:25.126914 17881 raft_consensus.cc:359] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:25.127544 17881 raft_consensus.cc:385] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:25.127619 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:20:25.127848 17881 raft_consensus.cc:740] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Initialized, Role: FOLLOWER
I20260709 12:20:25.128968 17652 leader_election.cc:304] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:20:25.128621 17881 consensus_queue.cc:260] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:25.129765 17835 raft_consensus.cc:3060] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:25.130230 18122 raft_consensus.cc:2804] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:25.131109 17881 ts_tablet_manager.cc:1434] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b: Time spent starting tablet: real 0.006s	user 0.004s	sys 0.000s
I20260709 12:20:25.131156 18122 raft_consensus.cc:493] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:25.131660 18122 raft_consensus.cc:3060] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:25.132174 17756 raft_consensus.cc:2468] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 1.
I20260709 12:20:25.135710 17652 leader_election.cc:304] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:20:25.137393 18122 raft_consensus.cc:515] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:25.137826 17835 raft_consensus.cc:2468] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 1.
I20260709 12:20:25.138844 18276 raft_consensus.cc:2804] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:25.139298 18276 raft_consensus.cc:697] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 1 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:20:25.140070 18276 consensus_queue.cc:237] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:25.128831 17586 catalog_manager.cc:5697] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb reported cstate change: term changed from 0 to 1, leader changed from <none> to e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66). New cstate: current_term: 1 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:25.145937 18122 leader_election.cc:290] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:25.151382 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:20:25.151750 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:20:25.152019 17756 raft_consensus.cc:3060] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:25.152288 17835 raft_consensus.cc:3060] T 50f9ff8f24a14a079db34b1ba88fa278 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:25.155040 17587 catalog_manager.cc:5697] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0da169ff82674dc1947055ce43de68a3 (127.16.29.65). New cstate: current_term: 1 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:25.158635 17756 raft_consensus.cc:2468] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 1.
I20260709 12:20:25.160588 17835 raft_consensus.cc:2468] T 50f9ff8f24a14a079db34b1ba88fa278 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 1.
I20260709 12:20:25.162557 17652 leader_election.cc:304] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:20:25.163285 18122 raft_consensus.cc:2804] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:25.164049 18122 raft_consensus.cc:697] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 1 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:20:25.164888 18122 consensus_queue.cc:237] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:25.177138 17586 catalog_manager.cc:5697] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0da169ff82674dc1947055ce43de68a3 (127.16.29.65). New cstate: current_term: 1 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:25.310240 17952 raft_consensus.cc:493] T 56b500bd1b914ebfb3f19afc8a330665 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:25.310750 17952 raft_consensus.cc:515] T 56b500bd1b914ebfb3f19afc8a330665 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:25.312709 17952 leader_election.cc:290] T 56b500bd1b914ebfb3f19afc8a330665 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:20:25.313325 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:20:25.314393 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:20:25.314463 18122 consensus_queue.cc:1048] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:25.315490 17761 raft_consensus.cc:2393] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 1: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:20:25.316638 17801 leader_election.cc:304] T 56b500bd1b914ebfb3f19afc8a330665 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:20:25.317585 17952 raft_consensus.cc:3060] T 56b500bd1b914ebfb3f19afc8a330665 P e063541542a94969898868bf6eb5ce1b [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:25.325727 16501 file_cache.cc:504] Constructed file cache file cache with capacity 419430
I20260709 12:20:25.334824 17952 raft_consensus.cc:2749] T 56b500bd1b914ebfb3f19afc8a330665 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260709 12:20:25.337303 18322 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:20:25.340358 18122 consensus_queue.cc:1048] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260709 12:20:25.358275 18323 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:20:25.360656 16501 server_base.cc:1061] running on GCE node
W20260709 12:20:25.374588 18327 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:20:25.376044 16501 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:20:25.376353 16501 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:20:25.376564 16501 hybrid_clock.cc:648] HybridClock initialized: now 1783599625376547 us; error 0 us; skew 500 ppm
I20260709 12:20:25.377367 16501 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:20:25.381062 16501 webserver.cc:533] Webserver started at http://127.16.29.68:42309/ using document root <none> and password file <none>
I20260709 12:20:25.381605 16501 fs_manager.cc:362] Metadata directory not provided
I20260709 12:20:25.381842 16501 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:20:25.382243 16501 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:20:25.383836 16501 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599551912677-16501-0/minicluster-data/ts-3-root/instance:
uuid: "334a4e1f59944544b3b3ff36616b57da"
format_stamp: "Formatted at 2026-07-09 12:20:25 on dist-test-slave-5n9v"
I20260709 12:20:25.389492 16501 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.005s	sys 0.000s
I20260709 12:20:25.393667 18332 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:20:25.394585 16501 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.001s	sys 0.000s
I20260709 12:20:25.394891 16501 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599551912677-16501-0/minicluster-data/ts-3-root
uuid: "334a4e1f59944544b3b3ff36616b57da"
format_stamp: "Formatted at 2026-07-09 12:20:25 on dist-test-slave-5n9v"
I20260709 12:20:25.395258 16501 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599551912677-16501-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599551912677-16501-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599551912677-16501-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:20:25.438657 16501 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:20:25.440605 16501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:20:25.441900 16501 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:20:25.444906 16501 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:20:25.445216 16501 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:20:25.445561 16501 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:20:25.445974 16501 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.001s
I20260709 12:20:25.512079 18183 consensus_queue.cc:1048] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:25.519853 18122 consensus_queue.cc:1048] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:20:25.532245 18294 consensus_queue.cc:1048] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:20:25.560091 18275 consensus_queue.cc:1048] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:20:25.567759 18222 consensus_queue.cc:1048] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:20:25.601027 18185 consensus_queue.cc:1048] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:25.637925 18183 consensus_queue.cc:1048] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:25.639353 18276 consensus_queue.cc:1048] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:20:25.641675 18276 consensus_queue.cc:1048] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
I20260709 12:20:25.717681 18222 consensus_queue.cc:1048] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:20:25.732170 18222 consensus_queue.cc:1048] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:25.823172 18219 consensus_queue.cc:1048] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260709 12:20:25.825285 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 44296.5us
I20260709 12:20:25.952132 16501 rpc_server.cc:307] RPC server started. Bound to: 127.16.29.68:42577
I20260709 12:20:25.952293 18409 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.29.68:42577 every 8 connection(s)
I20260709 12:20:25.962890 16501 auto_leader_rebalancer-test.cc:439] add a tserver: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:20:25.977957 18410 heartbeater.cc:344] Connected to a master server at 127.16.29.126:39149
I20260709 12:20:25.978420 18410 heartbeater.cc:461] Registering TS with master...
I20260709 12:20:25.979174 18410 heartbeater.cc:507] Master 127.16.29.126:39149 requested a full tablet report, sending...
I20260709 12:20:25.981360 17587 ts_manager.cc:194] Registered new tserver with Master: 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:20:25.983059 17587 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:37172
I20260709 12:20:26.598732 17760 consensus_queue.cc:237] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:20:26.606560 17687 raft_consensus.cc:1275] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:20:26.608868 17835 raft_consensus.cc:1275] T 8e417cb980e44a77962d2e3dfd58aad0 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:20:26.611678 18185 consensus_queue.cc:1048] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:20:26.613526 18418 consensus_queue.cc:1048] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260709 12:20:26.665176 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 44727.7us
I20260709 12:20:26.669373 18417 raft_consensus.cc:2955] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } }
I20260709 12:20:26.673048 17687 raft_consensus.cc:2955] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } }
W20260709 12:20:26.683382 17729 consensus_peers.cc:597] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb -> Peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Couldn't send request to peer 334a4e1f59944544b3b3ff36616b57da. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8e417cb980e44a77962d2e3dfd58aad0. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:26.700584 17587 catalog_manager.cc:5697] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb reported cstate change: config changed from index -1 to 2, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New cstate: current_term: 1 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:26.708422 17835 raft_consensus.cc:2955] T 8e417cb980e44a77962d2e3dfd58aad0 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } }
I20260709 12:20:26.721968 17835 consensus_queue.cc:237] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:20:26.731648 17687 raft_consensus.cc:1275] T 11896555f5d7458dbb02a3203b27f1cf P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:20:26.732219 17760 raft_consensus.cc:1275] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:20:26.733803 18231 consensus_queue.cc:1048] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:20:26.734948 18214 consensus_queue.cc:1048] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260709 12:20:26.758044 17803 consensus_peers.cc:597] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b -> Peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Couldn't send request to peer 334a4e1f59944544b3b3ff36616b57da. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 11896555f5d7458dbb02a3203b27f1cf. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:26.763804 18231 raft_consensus.cc:2955] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } }
I20260709 12:20:26.766079 17686 raft_consensus.cc:2955] T 11896555f5d7458dbb02a3203b27f1cf P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } }
I20260709 12:20:26.769923 17761 raft_consensus.cc:2955] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } }
I20260709 12:20:26.780956 17586 catalog_manager.cc:5697] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b reported cstate change: config changed from index -1 to 2, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New cstate: current_term: 1 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:26.804853 17686 consensus_queue.cc:237] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:20:26.812352 17761 raft_consensus.cc:1275] T ea2f2b5792f84bbba2738f1ce1e90122 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:20:26.812748 17838 raft_consensus.cc:1275] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:20:26.813724 18422 consensus_queue.cc:1048] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:20:26.814911 18219 consensus_queue.cc:1048] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260709 12:20:26.828156 17654 consensus_peers.cc:597] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 -> Peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Couldn't send request to peer 334a4e1f59944544b3b3ff36616b57da. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ea2f2b5792f84bbba2738f1ce1e90122. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:26.828133 18423 raft_consensus.cc:2955] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } }
I20260709 12:20:26.837853 17760 raft_consensus.cc:2955] T ea2f2b5792f84bbba2738f1ce1e90122 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } }
I20260709 12:20:26.842707 17686 consensus_queue.cc:237] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:20:26.847944 17835 raft_consensus.cc:2955] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } }
I20260709 12:20:26.850201 17838 raft_consensus.cc:1275] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260709 12:20:26.851972 17654 consensus_peers.cc:597] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 -> Peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Couldn't send request to peer 334a4e1f59944544b3b3ff36616b57da. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e4dcc70202ad4cd6880749fbb3a6d0c6. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:26.853139 18276 consensus_queue.cc:1048] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:20:26.859020 17760 raft_consensus.cc:1275] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:20:26.858749 17584 catalog_manager.cc:5697] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 reported cstate change: config changed from index -1 to 2, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New cstate: current_term: 1 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:26.861534 18276 consensus_queue.cc:1048] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:20:26.877372 18219 raft_consensus.cc:2955] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } }
I20260709 12:20:26.885249 17838 raft_consensus.cc:2955] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } }
I20260709 12:20:26.916574 17761 raft_consensus.cc:2955] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } }
I20260709 12:20:26.927287 17586 catalog_manager.cc:5697] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b reported cstate change: config changed from index -1 to 2, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New cstate: current_term: 1 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } }
I20260709 12:20:26.985718 18410 heartbeater.cc:499] Master 127.16.29.126:39149 was elected leader, sending a full tablet report...
I20260709 12:20:27.185788 18446 ts_tablet_manager.cc:933] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da: Initiating tablet copy from peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:20:27.215829 18446 tablet_copy_client.cc:323] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Beginning tablet copy session from remote peer at address 127.16.29.66:39651
I20260709 12:20:27.218333 18448 ts_tablet_manager.cc:933] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da: Initiating tablet copy from peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071)
I20260709 12:20:27.235818 18448 tablet_copy_client.cc:323] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Beginning tablet copy session from remote peer at address 127.16.29.65:33071
I20260709 12:20:27.246955 17774 tablet_copy_service.cc:140] P e2909cff1da44ad387f96a8ae3aa5afb: Received BeginTabletCopySession request for tablet 8e417cb980e44a77962d2e3dfd58aad0 from peer 334a4e1f59944544b3b3ff36616b57da ({username='slave'} at 127.0.0.1:36572)
I20260709 12:20:27.247660 17774 tablet_copy_service.cc:161] P e2909cff1da44ad387f96a8ae3aa5afb: Beginning new tablet copy session on tablet 8e417cb980e44a77962d2e3dfd58aad0 from peer 334a4e1f59944544b3b3ff36616b57da at {username='slave'} at 127.0.0.1:36572: session id = 334a4e1f59944544b3b3ff36616b57da-8e417cb980e44a77962d2e3dfd58aad0
I20260709 12:20:27.253675 17699 tablet_copy_service.cc:140] P 0da169ff82674dc1947055ce43de68a3: Received BeginTabletCopySession request for tablet ea2f2b5792f84bbba2738f1ce1e90122 from peer 334a4e1f59944544b3b3ff36616b57da ({username='slave'} at 127.0.0.1:44898)
I20260709 12:20:27.254374 17699 tablet_copy_service.cc:161] P 0da169ff82674dc1947055ce43de68a3: Beginning new tablet copy session on tablet ea2f2b5792f84bbba2738f1ce1e90122 from peer 334a4e1f59944544b3b3ff36616b57da at {username='slave'} at 127.0.0.1:44898: session id = 334a4e1f59944544b3b3ff36616b57da-ea2f2b5792f84bbba2738f1ce1e90122
I20260709 12:20:27.260175 17774 tablet_copy_source_session.cc:215] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:20:27.264430 18446 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e417cb980e44a77962d2e3dfd58aad0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:27.272234 17699 tablet_copy_source_session.cc:215] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:20:27.276087 18448 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea2f2b5792f84bbba2738f1ce1e90122. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:27.287771 18446 tablet_copy_client.cc:806] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Starting download of 0 data blocks...
I20260709 12:20:27.288568 18446 tablet_copy_client.cc:670] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Starting download of 1 WAL segments...
I20260709 12:20:27.292582 18448 tablet_copy_client.cc:806] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Starting download of 0 data blocks...
I20260709 12:20:27.293412 18448 tablet_copy_client.cc:670] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Starting download of 1 WAL segments...
I20260709 12:20:27.297425 18446 tablet_copy_client.cc:538] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:20:27.299791 18448 tablet_copy_client.cc:538] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:20:27.308421 18446 tablet_bootstrap.cc:492] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da: Bootstrap starting.
I20260709 12:20:27.310989 18448 tablet_bootstrap.cc:492] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da: Bootstrap starting.
I20260709 12:20:27.344558 18446 tablet_bootstrap.cc:492] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:20:27.344621 18448 tablet_bootstrap.cc:492] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:20:27.346422 18446 tablet_bootstrap.cc:492] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da: Bootstrap complete.
I20260709 12:20:27.347319 18446 ts_tablet_manager.cc:1403] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da: Time spent bootstrapping tablet: real 0.039s	user 0.030s	sys 0.005s
I20260709 12:20:27.351583 18446 raft_consensus.cc:359] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:20:27.352797 18446 raft_consensus.cc:740] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Initialized, Role: LEARNER
I20260709 12:20:27.353672 18446 consensus_queue.cc:260] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:20:27.362145 18455 ts_tablet_manager.cc:933] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da: Initiating tablet copy from peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:27.364428 18448 tablet_bootstrap.cc:492] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da: Bootstrap complete.
I20260709 12:20:27.365235 18448 ts_tablet_manager.cc:1403] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da: Time spent bootstrapping tablet: real 0.054s	user 0.025s	sys 0.004s
I20260709 12:20:27.367522 18446 ts_tablet_manager.cc:1434] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da: Time spent starting tablet: real 0.020s	user 0.012s	sys 0.007s
I20260709 12:20:27.368202 18448 raft_consensus.cc:359] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:20:27.369016 18448 raft_consensus.cc:740] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Initialized, Role: LEARNER
I20260709 12:20:27.369685 18448 consensus_queue.cc:260] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:20:27.371954 18448 ts_tablet_manager.cc:1434] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da: Time spent starting tablet: real 0.006s	user 0.008s	sys 0.000s
I20260709 12:20:27.372478 17774 tablet_copy_service.cc:342] P e2909cff1da44ad387f96a8ae3aa5afb: Request end of tablet copy session 334a4e1f59944544b3b3ff36616b57da-8e417cb980e44a77962d2e3dfd58aad0 received from {username='slave'} at 127.0.0.1:36572
I20260709 12:20:27.372958 17774 tablet_copy_service.cc:434] P e2909cff1da44ad387f96a8ae3aa5afb: ending tablet copy session 334a4e1f59944544b3b3ff36616b57da-8e417cb980e44a77962d2e3dfd58aad0 on tablet 8e417cb980e44a77962d2e3dfd58aad0 with peer 334a4e1f59944544b3b3ff36616b57da
I20260709 12:20:27.374832 17699 tablet_copy_service.cc:342] P 0da169ff82674dc1947055ce43de68a3: Request end of tablet copy session 334a4e1f59944544b3b3ff36616b57da-ea2f2b5792f84bbba2738f1ce1e90122 received from {username='slave'} at 127.0.0.1:44898
I20260709 12:20:27.375237 17699 tablet_copy_service.cc:434] P 0da169ff82674dc1947055ce43de68a3: ending tablet copy session 334a4e1f59944544b3b3ff36616b57da-ea2f2b5792f84bbba2738f1ce1e90122 on tablet ea2f2b5792f84bbba2738f1ce1e90122 with peer 334a4e1f59944544b3b3ff36616b57da
I20260709 12:20:27.375387 18455 tablet_copy_client.cc:323] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Beginning tablet copy session from remote peer at address 127.16.29.67:36837
I20260709 12:20:27.401539 17848 tablet_copy_service.cc:140] P e063541542a94969898868bf6eb5ce1b: Received BeginTabletCopySession request for tablet 11896555f5d7458dbb02a3203b27f1cf from peer 334a4e1f59944544b3b3ff36616b57da ({username='slave'} at 127.0.0.1:37994)
I20260709 12:20:27.402202 17848 tablet_copy_service.cc:161] P e063541542a94969898868bf6eb5ce1b: Beginning new tablet copy session on tablet 11896555f5d7458dbb02a3203b27f1cf from peer 334a4e1f59944544b3b3ff36616b57da at {username='slave'} at 127.0.0.1:37994: session id = 334a4e1f59944544b3b3ff36616b57da-11896555f5d7458dbb02a3203b27f1cf
I20260709 12:20:27.408474 17848 tablet_copy_source_session.cc:215] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:20:27.411201 18455 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11896555f5d7458dbb02a3203b27f1cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:27.422521 18448 ts_tablet_manager.cc:933] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da: Initiating tablet copy from peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071)
I20260709 12:20:27.429240 18455 tablet_copy_client.cc:806] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Starting download of 0 data blocks...
I20260709 12:20:27.429862 18455 tablet_copy_client.cc:670] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Starting download of 1 WAL segments...
I20260709 12:20:27.431216 18448 tablet_copy_client.cc:323] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Beginning tablet copy session from remote peer at address 127.16.29.65:33071
I20260709 12:20:27.432824 17699 tablet_copy_service.cc:140] P 0da169ff82674dc1947055ce43de68a3: Received BeginTabletCopySession request for tablet e4dcc70202ad4cd6880749fbb3a6d0c6 from peer 334a4e1f59944544b3b3ff36616b57da ({username='slave'} at 127.0.0.1:44898)
I20260709 12:20:27.433331 17699 tablet_copy_service.cc:161] P 0da169ff82674dc1947055ce43de68a3: Beginning new tablet copy session on tablet e4dcc70202ad4cd6880749fbb3a6d0c6 from peer 334a4e1f59944544b3b3ff36616b57da at {username='slave'} at 127.0.0.1:44898: session id = 334a4e1f59944544b3b3ff36616b57da-e4dcc70202ad4cd6880749fbb3a6d0c6
I20260709 12:20:27.434360 18455 tablet_copy_client.cc:538] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:20:27.442512 17699 tablet_copy_source_session.cc:215] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:20:27.443032 18455 tablet_bootstrap.cc:492] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da: Bootstrap starting.
I20260709 12:20:27.445755 18448 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e4dcc70202ad4cd6880749fbb3a6d0c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:27.467015 18448 tablet_copy_client.cc:806] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Starting download of 0 data blocks...
I20260709 12:20:27.467602 18448 tablet_copy_client.cc:670] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Starting download of 1 WAL segments...
I20260709 12:20:27.472307 18448 tablet_copy_client.cc:538] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:20:27.479049 18455 tablet_bootstrap.cc:492] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:20:27.479218 18448 tablet_bootstrap.cc:492] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da: Bootstrap starting.
I20260709 12:20:27.480162 18455 tablet_bootstrap.cc:492] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da: Bootstrap complete.
I20260709 12:20:27.480801 18455 ts_tablet_manager.cc:1403] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da: Time spent bootstrapping tablet: real 0.038s	user 0.012s	sys 0.015s
I20260709 12:20:27.485100 18455 raft_consensus.cc:359] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:20:27.485916 18455 raft_consensus.cc:740] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Initialized, Role: LEARNER
I20260709 12:20:27.486564 18455 consensus_queue.cc:260] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:20:27.497745 18455 ts_tablet_manager.cc:1434] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da: Time spent starting tablet: real 0.017s	user 0.003s	sys 0.003s
I20260709 12:20:27.500949 17848 tablet_copy_service.cc:342] P e063541542a94969898868bf6eb5ce1b: Request end of tablet copy session 334a4e1f59944544b3b3ff36616b57da-11896555f5d7458dbb02a3203b27f1cf received from {username='slave'} at 127.0.0.1:37994
I20260709 12:20:27.501395 17848 tablet_copy_service.cc:434] P e063541542a94969898868bf6eb5ce1b: ending tablet copy session 334a4e1f59944544b3b3ff36616b57da-11896555f5d7458dbb02a3203b27f1cf on tablet 11896555f5d7458dbb02a3203b27f1cf with peer 334a4e1f59944544b3b3ff36616b57da
I20260709 12:20:27.509423 18448 tablet_bootstrap.cc:492] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:20:27.510380 18448 tablet_bootstrap.cc:492] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da: Bootstrap complete.
I20260709 12:20:27.511061 18448 ts_tablet_manager.cc:1403] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da: Time spent bootstrapping tablet: real 0.032s	user 0.017s	sys 0.006s
I20260709 12:20:27.513957 18448 raft_consensus.cc:359] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:20:27.514782 18448 raft_consensus.cc:740] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Initialized, Role: LEARNER
I20260709 12:20:27.515384 18448 consensus_queue.cc:260] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:20:27.518080 18448 ts_tablet_manager.cc:1434] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da: Time spent starting tablet: real 0.007s	user 0.008s	sys 0.000s
I20260709 12:20:27.519824 17699 tablet_copy_service.cc:342] P 0da169ff82674dc1947055ce43de68a3: Request end of tablet copy session 334a4e1f59944544b3b3ff36616b57da-e4dcc70202ad4cd6880749fbb3a6d0c6 received from {username='slave'} at 127.0.0.1:44898
I20260709 12:20:27.520279 17699 tablet_copy_service.cc:434] P 0da169ff82674dc1947055ce43de68a3: ending tablet copy session 334a4e1f59944544b3b3ff36616b57da-e4dcc70202ad4cd6880749fbb3a6d0c6 on tablet e4dcc70202ad4cd6880749fbb3a6d0c6 with peer 334a4e1f59944544b3b3ff36616b57da
I20260709 12:20:27.698817 18374 raft_consensus.cc:1217] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:20:27.706153 18374 raft_consensus.cc:1217] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:20:27.846845 18374 raft_consensus.cc:1217] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:20:27.928393 18374 raft_consensus.cc:1217] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:20:28.088668 18444 raft_consensus.cc:1064] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3: attempting to promote NON_VOTER 334a4e1f59944544b3b3ff36616b57da to VOTER
I20260709 12:20:28.091355 18444 consensus_queue.cc:237] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:20:28.098892 17835 raft_consensus.cc:1275] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:20:28.098929 18374 raft_consensus.cc:1275] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 1 LEARNER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:20:28.100553 18445 consensus_queue.cc:1048] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:20:28.101480 18423 consensus_queue.cc:1048] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:20:28.102202 17760 raft_consensus.cc:1275] T ea2f2b5792f84bbba2738f1ce1e90122 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:20:28.105064 18445 consensus_queue.cc:1048] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:20:28.127099 18444 raft_consensus.cc:2955] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:28.128988 18371 raft_consensus.cc:2955] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:28.142992 17760 raft_consensus.cc:2955] T ea2f2b5792f84bbba2738f1ce1e90122 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:28.150873 17587 catalog_manager.cc:5697] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da reported cstate change: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:28.171656 18456 raft_consensus.cc:1064] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb: attempting to promote NON_VOTER 334a4e1f59944544b3b3ff36616b57da to VOTER
I20260709 12:20:28.175755 18456 consensus_queue.cc:237] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:20:28.181643 17835 raft_consensus.cc:2955] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:28.202395 18371 raft_consensus.cc:1275] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 1 LEARNER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:20:28.206341 18385 consensus_queue.cc:1048] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:20:28.231824 17687 raft_consensus.cc:1275] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:20:28.234383 18417 consensus_queue.cc:1048] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:20:28.234146 17686 consensus_queue.cc:237] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:20:28.237049 17835 raft_consensus.cc:1275] T 8e417cb980e44a77962d2e3dfd58aad0 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:20:28.240813 18371 raft_consensus.cc:1275] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:20:28.240911 18417 consensus_queue.cc:1048] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:20:28.243086 18445 consensus_queue.cc:1048] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:20:28.247202 17838 raft_consensus.cc:1275] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:20:28.250224 18423 consensus_queue.cc:1048] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:20:28.252156 18444 raft_consensus.cc:2955] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:28.266002 17572 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet ea2f2b5792f84bbba2738f1ce1e90122 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:20:28.262909 17838 raft_consensus.cc:2955] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:28.290195 17585 catalog_manager.cc:5697] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 reported cstate change: config changed from index 3 to 4, VOTER e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66) evicted. New cstate: current_term: 1 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:28.302462 18385 raft_consensus.cc:2955] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:28.307283 17753 tablet_service.cc:1558] Processing DeleteTablet for tablet ea2f2b5792f84bbba2738f1ce1e90122 with delete_type TABLET_DATA_TOMBSTONED (TS e2909cff1da44ad387f96a8ae3aa5afb not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:38774
I20260709 12:20:28.315786 17838 raft_consensus.cc:2955] T 8e417cb980e44a77962d2e3dfd58aad0 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:28.322676 18463 raft_consensus.cc:1064] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b: attempting to promote NON_VOTER 334a4e1f59944544b3b3ff36616b57da to VOTER
I20260709 12:20:28.325438 18463 consensus_queue.cc:237] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:20:28.328217 18374 raft_consensus.cc:2955] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:28.328644 17686 raft_consensus.cc:2955] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:28.343456 18477 raft_consensus.cc:1064] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3: attempting to promote NON_VOTER 334a4e1f59944544b3b3ff36616b57da to VOTER
I20260709 12:20:28.346706 18477 consensus_queue.cc:237] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:20:28.385406 17584 catalog_manager.cc:5697] T 8e417cb980e44a77962d2e3dfd58aad0 P e063541542a94969898868bf6eb5ce1b reported cstate change: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:28.422024 18374 raft_consensus.cc:1275] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 1 LEARNER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:20:28.438934 18444 consensus_queue.cc:1048] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.012s
I20260709 12:20:28.459407 17761 raft_consensus.cc:1275] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:20:28.470657 18371 raft_consensus.cc:2955] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:28.472079 17761 consensus_queue.cc:237] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:20:28.477315 18479 tablet_replica.cc:333] T ea2f2b5792f84bbba2738f1ce1e90122 P e2909cff1da44ad387f96a8ae3aa5afb: stopping tablet replica
I20260709 12:20:28.478355 18479 raft_consensus.cc:2243] T ea2f2b5792f84bbba2738f1ce1e90122 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:28.510511 17761 raft_consensus.cc:1275] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:20:28.468309 18477 consensus_queue.cc:1048] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.042s
I20260709 12:20:28.513727 18367 raft_consensus.cc:1275] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 1 LEARNER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:20:28.516113 18479 raft_consensus.cc:2272] T ea2f2b5792f84bbba2738f1ce1e90122 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:28.516251 18421 consensus_queue.cc:1048] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20260709 12:20:28.524245 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 56566us
I20260709 12:20:28.526502 17686 raft_consensus.cc:1275] T 11896555f5d7458dbb02a3203b27f1cf P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:20:28.517573 18485 consensus_queue.cc:1048] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.018s
I20260709 12:20:28.538745 18368 raft_consensus.cc:1275] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:20:28.545360 18482 consensus_queue.cc:1048] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:20:28.546429 17684 raft_consensus.cc:1275] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:20:28.547052 18479 ts_tablet_manager.cc:1916] T ea2f2b5792f84bbba2738f1ce1e90122 P e2909cff1da44ad387f96a8ae3aa5afb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:20:28.548192 18482 consensus_queue.cc:1048] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:20:28.552769 18456 raft_consensus.cc:2955] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER e063541542a94969898868bf6eb5ce1b (127.16.29.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:28.560431 17686 raft_consensus.cc:2955] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER e063541542a94969898868bf6eb5ce1b (127.16.29.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:28.561755 18367 raft_consensus.cc:2955] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER e063541542a94969898868bf6eb5ce1b (127.16.29.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
W20260709 12:20:28.585705 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 73925.8us
I20260709 12:20:28.593562 17838 raft_consensus.cc:1275] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:20:28.597136 18490 consensus_queue.cc:1048] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.011s
I20260709 12:20:28.603328 17586 catalog_manager.cc:5697] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 reported cstate change: config changed from index 3 to 4, VOTER e063541542a94969898868bf6eb5ce1b (127.16.29.67) evicted. New cstate: current_term: 1 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:28.603740 17571 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 8e417cb980e44a77962d2e3dfd58aad0 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:20:28.607208 18479 ts_tablet_manager.cc:1929] T ea2f2b5792f84bbba2738f1ce1e90122 P e2909cff1da44ad387f96a8ae3aa5afb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:20:28.607658 18479 log.cc:1199] T ea2f2b5792f84bbba2738f1ce1e90122 P e2909cff1da44ad387f96a8ae3aa5afb: Deleting WAL directory at /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599551912677-16501-0/minicluster-data/ts-1-root/wals/ea2f2b5792f84bbba2738f1ce1e90122
I20260709 12:20:28.613813 17571 catalog_manager.cc:5028] TS e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): tablet ea2f2b5792f84bbba2738f1ce1e90122 (table test-workload [id=ddc5b74250df4c65909b73b3742d615f]) successfully deleted
I20260709 12:20:28.617450 18477 raft_consensus.cc:2955] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:28.620858 17760 raft_consensus.cc:2955] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:28.637967 17821 tablet_service.cc:1558] Processing DeleteTablet for tablet 8e417cb980e44a77962d2e3dfd58aad0 with delete_type TABLET_DATA_TOMBSTONED (TS e063541542a94969898868bf6eb5ce1b not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:35032
I20260709 12:20:28.639912 18478 consensus_queue.cc:1048] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:20:28.673173 18495 tablet_replica.cc:333] T 8e417cb980e44a77962d2e3dfd58aad0 P e063541542a94969898868bf6eb5ce1b: stopping tablet replica
I20260709 12:20:28.673326 17585 catalog_manager.cc:5697] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 reported cstate change: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:20:28.674340 18495 raft_consensus.cc:2243] T 8e417cb980e44a77962d2e3dfd58aad0 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:28.675247 18495 raft_consensus.cc:2272] T 8e417cb980e44a77962d2e3dfd58aad0 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:28.681409 17838 raft_consensus.cc:2955] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:28.690313 18495 ts_tablet_manager.cc:1916] T 8e417cb980e44a77962d2e3dfd58aad0 P e063541542a94969898868bf6eb5ce1b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:20:28.672534 18368 raft_consensus.cc:2955] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:28.714188 18490 raft_consensus.cc:2955] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:28.719437 18495 ts_tablet_manager.cc:1929] T 8e417cb980e44a77962d2e3dfd58aad0 P e063541542a94969898868bf6eb5ce1b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:20:28.719981 18495 log.cc:1199] T 8e417cb980e44a77962d2e3dfd58aad0 P e063541542a94969898868bf6eb5ce1b: Deleting WAL directory at /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599551912677-16501-0/minicluster-data/ts-2-root/wals/8e417cb980e44a77962d2e3dfd58aad0
I20260709 12:20:28.733166 17684 raft_consensus.cc:2955] T 11896555f5d7458dbb02a3203b27f1cf P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:28.742340 17571 catalog_manager.cc:5028] TS e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837): tablet 8e417cb980e44a77962d2e3dfd58aad0 (table test-workload [id=ddc5b74250df4c65909b73b3742d615f]) successfully deleted
I20260709 12:20:28.743500 18368 raft_consensus.cc:2955] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:28.749317 17761 raft_consensus.cc:2955] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:28.765977 17586 catalog_manager.cc:5697] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da reported cstate change: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:28.786001 17838 consensus_queue.cc:237] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:20:28.792096 18368 raft_consensus.cc:1275] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:20:28.792259 17756 raft_consensus.cc:1275] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:20:28.793910 18488 consensus_queue.cc:1048] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:20:28.794842 18463 consensus_queue.cc:1048] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:20:28.801488 18488 raft_consensus.cc:2955] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:28.806754 17756 raft_consensus.cc:2955] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:28.813333 17571 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 11896555f5d7458dbb02a3203b27f1cf with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:20:28.813773 18368 raft_consensus.cc:2955] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:28.824458 17584 catalog_manager.cc:5697] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b reported cstate change: config changed from index 3 to 4, VOTER 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) evicted. New cstate: current_term: 1 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:20:28.836862 17676 tablet_service.cc:1558] Processing DeleteTablet for tablet 11896555f5d7458dbb02a3203b27f1cf with delete_type TABLET_DATA_TOMBSTONED (TS 0da169ff82674dc1947055ce43de68a3 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:37384
I20260709 12:20:28.854655 18498 tablet_replica.cc:333] T 11896555f5d7458dbb02a3203b27f1cf P 0da169ff82674dc1947055ce43de68a3: stopping tablet replica
I20260709 12:20:28.860064 18498 raft_consensus.cc:2243] T 11896555f5d7458dbb02a3203b27f1cf P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:28.862179 18498 raft_consensus.cc:2272] T 11896555f5d7458dbb02a3203b27f1cf P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:28.874728 18498 ts_tablet_manager.cc:1916] T 11896555f5d7458dbb02a3203b27f1cf P 0da169ff82674dc1947055ce43de68a3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:20:28.896405 18498 ts_tablet_manager.cc:1929] T 11896555f5d7458dbb02a3203b27f1cf P 0da169ff82674dc1947055ce43de68a3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:20:28.896871 18498 log.cc:1199] T 11896555f5d7458dbb02a3203b27f1cf P 0da169ff82674dc1947055ce43de68a3: Deleting WAL directory at /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599551912677-16501-0/minicluster-data/ts-0-root/wals/11896555f5d7458dbb02a3203b27f1cf
I20260709 12:20:28.899070 17572 catalog_manager.cc:5028] TS 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): tablet 11896555f5d7458dbb02a3203b27f1cf (table test-workload [id=ddc5b74250df4c65909b73b3742d615f]) successfully deleted
I20260709 12:20:29.202546 17686 consensus_queue.cc:237] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:20:29.208066 18368 raft_consensus.cc:1275] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:20:29.210019 18483 consensus_queue.cc:1048] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:20:29.218037 17836 raft_consensus.cc:1275] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:20:29.222365 18444 raft_consensus.cc:2955] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:29.228542 18368 raft_consensus.cc:2955] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:29.234184 18423 consensus_queue.cc:1048] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:20:29.264573 17836 raft_consensus.cc:2955] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:29.276494 17572 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet e4dcc70202ad4cd6880749fbb3a6d0c6 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:20:29.262843 17585 catalog_manager.cc:5697] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da reported cstate change: config changed from index 3 to 4, VOTER e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66) evicted. New cstate: current_term: 1 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:29.300485 17753 tablet_service.cc:1558] Processing DeleteTablet for tablet e4dcc70202ad4cd6880749fbb3a6d0c6 with delete_type TABLET_DATA_TOMBSTONED (TS e2909cff1da44ad387f96a8ae3aa5afb not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:38774
I20260709 12:20:29.302248 18500 tablet_replica.cc:333] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e2909cff1da44ad387f96a8ae3aa5afb: stopping tablet replica
I20260709 12:20:29.303284 18500 raft_consensus.cc:2243] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:29.304155 18500 raft_consensus.cc:2272] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:29.311266 18500 ts_tablet_manager.cc:1916] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e2909cff1da44ad387f96a8ae3aa5afb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:20:29.325472 18500 ts_tablet_manager.cc:1929] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e2909cff1da44ad387f96a8ae3aa5afb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:20:29.325826 18500 log.cc:1199] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e2909cff1da44ad387f96a8ae3aa5afb: Deleting WAL directory at /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599551912677-16501-0/minicluster-data/ts-1-root/wals/e4dcc70202ad4cd6880749fbb3a6d0c6
I20260709 12:20:29.327788 17571 catalog_manager.cc:5028] TS e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): tablet e4dcc70202ad4cd6880749fbb3a6d0c6 (table test-workload [id=ddc5b74250df4c65909b73b3742d615f]) successfully deleted
I20260709 12:20:30.374752 17761 consensus_queue.cc:237] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:20:30.381546 17838 raft_consensus.cc:1275] T 9439565c5dbb4f4ba3375aaef2c9add5 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260709 12:20:30.382103 17729 consensus_peers.cc:597] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb -> Peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Couldn't send request to peer 334a4e1f59944544b3b3ff36616b57da. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9439565c5dbb4f4ba3375aaef2c9add5. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:30.382261 17686 raft_consensus.cc:1275] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:20:30.382860 18484 consensus_queue.cc:1048] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:20:30.383623 18482 consensus_queue.cc:1048] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:20:30.394927 18482 raft_consensus.cc:2955] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } }
I20260709 12:20:30.396324 17686 raft_consensus.cc:2955] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } }
I20260709 12:20:30.402264 17838 raft_consensus.cc:2955] T 9439565c5dbb4f4ba3375aaef2c9add5 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } }
I20260709 12:20:30.412796 17587 catalog_manager.cc:5697] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb reported cstate change: config changed from index -1 to 2, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New cstate: current_term: 1 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:30.426172 17838 consensus_queue.cc:237] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
W20260709 12:20:30.444795 17803 consensus_peers.cc:597] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b -> Peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Couldn't send request to peer 334a4e1f59944544b3b3ff36616b57da. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0f940f29438f475294357324452abe76. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:30.446455 17686 raft_consensus.cc:1275] T 0f940f29438f475294357324452abe76 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:20:30.455641 17760 raft_consensus.cc:1275] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:20:30.457401 18501 consensus_queue.cc:1048] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:20:30.461079 18488 consensus_queue.cc:1048] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:20:30.481326 18501 raft_consensus.cc:2955] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } }
I20260709 12:20:30.482903 17760 raft_consensus.cc:2955] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } }
I20260709 12:20:30.492349 17686 raft_consensus.cc:2955] T 0f940f29438f475294357324452abe76 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } }
I20260709 12:20:30.508535 17585 catalog_manager.cc:5697] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb reported cstate change: config changed from index -1 to 2, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New cstate: current_term: 1 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } }
I20260709 12:20:30.509861 17838 consensus_queue.cc:237] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:20:30.530467 17760 raft_consensus.cc:1275] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:20:30.531031 17686 raft_consensus.cc:1275] T 217734cc2eb24a7a8d92bfef30ee5c52 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:20:30.531628 18512 consensus_queue.cc:1048] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.002s
W20260709 12:20:30.535866 17803 consensus_peers.cc:597] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b -> Peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Couldn't send request to peer 334a4e1f59944544b3b3ff36616b57da. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 217734cc2eb24a7a8d92bfef30ee5c52. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:30.544986 18488 consensus_queue.cc:1048] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:20:30.547361 18488 raft_consensus.cc:2955] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } }
I20260709 12:20:30.548863 17763 raft_consensus.cc:2955] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } }
I20260709 12:20:30.556502 17686 raft_consensus.cc:2955] T 217734cc2eb24a7a8d92bfef30ee5c52 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } }
I20260709 12:20:30.566798 17585 catalog_manager.cc:5697] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb reported cstate change: config changed from index -1 to 2, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New cstate: current_term: 1 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } }
I20260709 12:20:30.568532 17684 consensus_queue.cc:237] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:20:30.582643 17835 raft_consensus.cc:1275] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260709 12:20:30.582747 17654 consensus_peers.cc:597] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 -> Peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Couldn't send request to peer 334a4e1f59944544b3b3ff36616b57da. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 23c0cde5227f470388b8ed1fc0786119. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:30.584909 18444 consensus_queue.cc:1048] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:20:30.603823 17763 raft_consensus.cc:1275] T 23c0cde5227f470388b8ed1fc0786119 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:20:30.605800 18444 consensus_queue.cc:1048] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:20:30.608174 18423 raft_consensus.cc:2955] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } }
I20260709 12:20:30.609611 17835 raft_consensus.cc:2955] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } }
I20260709 12:20:30.615506 17763 raft_consensus.cc:2955] T 23c0cde5227f470388b8ed1fc0786119 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } }
I20260709 12:20:30.627514 17584 catalog_manager.cc:5697] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b reported cstate change: config changed from index -1 to 2, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New cstate: current_term: 1 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } }
I20260709 12:20:30.886931 18523 ts_tablet_manager.cc:933] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da: Initiating tablet copy from peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:20:30.895236 18523 tablet_copy_client.cc:323] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Beginning tablet copy session from remote peer at address 127.16.29.66:39651
I20260709 12:20:30.896992 17774 tablet_copy_service.cc:140] P e2909cff1da44ad387f96a8ae3aa5afb: Received BeginTabletCopySession request for tablet 9439565c5dbb4f4ba3375aaef2c9add5 from peer 334a4e1f59944544b3b3ff36616b57da ({username='slave'} at 127.0.0.1:36572)
I20260709 12:20:30.897640 17774 tablet_copy_service.cc:161] P e2909cff1da44ad387f96a8ae3aa5afb: Beginning new tablet copy session on tablet 9439565c5dbb4f4ba3375aaef2c9add5 from peer 334a4e1f59944544b3b3ff36616b57da at {username='slave'} at 127.0.0.1:36572: session id = 334a4e1f59944544b3b3ff36616b57da-9439565c5dbb4f4ba3375aaef2c9add5
I20260709 12:20:30.905493 17774 tablet_copy_source_session.cc:215] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:20:30.908947 18523 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9439565c5dbb4f4ba3375aaef2c9add5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:30.925395 18523 tablet_copy_client.cc:806] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Starting download of 0 data blocks...
I20260709 12:20:30.925968 18523 tablet_copy_client.cc:670] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Starting download of 1 WAL segments...
I20260709 12:20:30.929847 18523 tablet_copy_client.cc:538] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:20:30.940917 18523 tablet_bootstrap.cc:492] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da: Bootstrap starting.
I20260709 12:20:30.971758 18523 tablet_bootstrap.cc:492] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:20:30.972723 18523 tablet_bootstrap.cc:492] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da: Bootstrap complete.
I20260709 12:20:30.973398 18523 ts_tablet_manager.cc:1403] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da: Time spent bootstrapping tablet: real 0.033s	user 0.014s	sys 0.008s
I20260709 12:20:30.976441 18523 raft_consensus.cc:359] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:20:30.977286 18523 raft_consensus.cc:740] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Initialized, Role: LEARNER
I20260709 12:20:30.977988 18523 consensus_queue.cc:260] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:20:30.981734 18523 ts_tablet_manager.cc:1434] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da: Time spent starting tablet: real 0.008s	user 0.005s	sys 0.002s
I20260709 12:20:30.983755 17774 tablet_copy_service.cc:342] P e2909cff1da44ad387f96a8ae3aa5afb: Request end of tablet copy session 334a4e1f59944544b3b3ff36616b57da-9439565c5dbb4f4ba3375aaef2c9add5 received from {username='slave'} at 127.0.0.1:36572
I20260709 12:20:30.984225 17774 tablet_copy_service.cc:434] P e2909cff1da44ad387f96a8ae3aa5afb: ending tablet copy session 334a4e1f59944544b3b3ff36616b57da-9439565c5dbb4f4ba3375aaef2c9add5 on tablet 9439565c5dbb4f4ba3375aaef2c9add5 with peer 334a4e1f59944544b3b3ff36616b57da
I20260709 12:20:30.996754 18523 ts_tablet_manager.cc:933] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da: Initiating tablet copy from peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:30.999001 18523 tablet_copy_client.cc:323] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Beginning tablet copy session from remote peer at address 127.16.29.67:36837
I20260709 12:20:31.000700 17848 tablet_copy_service.cc:140] P e063541542a94969898868bf6eb5ce1b: Received BeginTabletCopySession request for tablet 217734cc2eb24a7a8d92bfef30ee5c52 from peer 334a4e1f59944544b3b3ff36616b57da ({username='slave'} at 127.0.0.1:37994)
I20260709 12:20:31.001247 17848 tablet_copy_service.cc:161] P e063541542a94969898868bf6eb5ce1b: Beginning new tablet copy session on tablet 217734cc2eb24a7a8d92bfef30ee5c52 from peer 334a4e1f59944544b3b3ff36616b57da at {username='slave'} at 127.0.0.1:37994: session id = 334a4e1f59944544b3b3ff36616b57da-217734cc2eb24a7a8d92bfef30ee5c52
I20260709 12:20:31.010221 17848 tablet_copy_source_session.cc:215] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:20:31.014442 18523 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 217734cc2eb24a7a8d92bfef30ee5c52. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:31.028882 18528 ts_tablet_manager.cc:933] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da: Initiating tablet copy from peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:31.031474 18523 tablet_copy_client.cc:806] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Starting download of 0 data blocks...
I20260709 12:20:31.032109 18523 tablet_copy_client.cc:670] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Starting download of 1 WAL segments...
I20260709 12:20:31.035507 18523 tablet_copy_client.cc:538] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:20:31.038424 18528 tablet_copy_client.cc:323] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Beginning tablet copy session from remote peer at address 127.16.29.67:36837
I20260709 12:20:31.040747 17848 tablet_copy_service.cc:140] P e063541542a94969898868bf6eb5ce1b: Received BeginTabletCopySession request for tablet 0f940f29438f475294357324452abe76 from peer 334a4e1f59944544b3b3ff36616b57da ({username='slave'} at 127.0.0.1:37994)
I20260709 12:20:31.041277 17848 tablet_copy_service.cc:161] P e063541542a94969898868bf6eb5ce1b: Beginning new tablet copy session on tablet 0f940f29438f475294357324452abe76 from peer 334a4e1f59944544b3b3ff36616b57da at {username='slave'} at 127.0.0.1:37994: session id = 334a4e1f59944544b3b3ff36616b57da-0f940f29438f475294357324452abe76
I20260709 12:20:31.048800 17848 tablet_copy_source_session.cc:215] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:20:31.049448 18523 tablet_bootstrap.cc:492] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da: Bootstrap starting.
I20260709 12:20:31.052196 18528 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0f940f29438f475294357324452abe76. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:31.087692 18528 tablet_copy_client.cc:806] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Starting download of 0 data blocks...
I20260709 12:20:31.088366 18528 tablet_copy_client.cc:670] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Starting download of 1 WAL segments...
I20260709 12:20:31.093318 18528 tablet_copy_client.cc:538] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:20:31.096253 18523 tablet_bootstrap.cc:492] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:20:31.097570 18523 tablet_bootstrap.cc:492] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da: Bootstrap complete.
I20260709 12:20:31.098239 18523 ts_tablet_manager.cc:1403] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da: Time spent bootstrapping tablet: real 0.049s	user 0.017s	sys 0.014s
I20260709 12:20:31.104903 18523 raft_consensus.cc:359] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:20:31.107020 18523 raft_consensus.cc:740] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Initialized, Role: LEARNER
I20260709 12:20:31.107795 18523 consensus_queue.cc:260] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:20:31.111917 18523 ts_tablet_manager.cc:1434] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da: Time spent starting tablet: real 0.013s	user 0.009s	sys 0.000s
I20260709 12:20:31.118158 17848 tablet_copy_service.cc:342] P e063541542a94969898868bf6eb5ce1b: Request end of tablet copy session 334a4e1f59944544b3b3ff36616b57da-217734cc2eb24a7a8d92bfef30ee5c52 received from {username='slave'} at 127.0.0.1:37994
I20260709 12:20:31.118968 17848 tablet_copy_service.cc:434] P e063541542a94969898868bf6eb5ce1b: ending tablet copy session 334a4e1f59944544b3b3ff36616b57da-217734cc2eb24a7a8d92bfef30ee5c52 on tablet 217734cc2eb24a7a8d92bfef30ee5c52 with peer 334a4e1f59944544b3b3ff36616b57da
I20260709 12:20:31.122373 18528 tablet_bootstrap.cc:492] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da: Bootstrap starting.
I20260709 12:20:31.122578 18523 ts_tablet_manager.cc:933] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da: Initiating tablet copy from peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071)
I20260709 12:20:31.138732 18523 tablet_copy_client.cc:323] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Beginning tablet copy session from remote peer at address 127.16.29.65:33071
I20260709 12:20:31.140987 17699 tablet_copy_service.cc:140] P 0da169ff82674dc1947055ce43de68a3: Received BeginTabletCopySession request for tablet 23c0cde5227f470388b8ed1fc0786119 from peer 334a4e1f59944544b3b3ff36616b57da ({username='slave'} at 127.0.0.1:44898)
I20260709 12:20:31.141536 17699 tablet_copy_service.cc:161] P 0da169ff82674dc1947055ce43de68a3: Beginning new tablet copy session on tablet 23c0cde5227f470388b8ed1fc0786119 from peer 334a4e1f59944544b3b3ff36616b57da at {username='slave'} at 127.0.0.1:44898: session id = 334a4e1f59944544b3b3ff36616b57da-23c0cde5227f470388b8ed1fc0786119
I20260709 12:20:31.157430 17699 tablet_copy_source_session.cc:215] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:20:31.161003 18523 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 23c0cde5227f470388b8ed1fc0786119. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:31.192888 18523 tablet_copy_client.cc:806] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Starting download of 0 data blocks...
I20260709 12:20:31.193503 18523 tablet_copy_client.cc:670] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Starting download of 1 WAL segments...
I20260709 12:20:31.198194 18523 tablet_copy_client.cc:538] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:20:31.202307 18528 tablet_bootstrap.cc:492] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:20:31.203706 18528 tablet_bootstrap.cc:492] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da: Bootstrap complete.
I20260709 12:20:31.204438 18528 ts_tablet_manager.cc:1403] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da: Time spent bootstrapping tablet: real 0.082s	user 0.036s	sys 0.039s
I20260709 12:20:31.205909 18523 tablet_bootstrap.cc:492] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da: Bootstrap starting.
I20260709 12:20:31.207309 18528 raft_consensus.cc:359] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:20:31.208174 18528 raft_consensus.cc:740] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Initialized, Role: LEARNER
I20260709 12:20:31.208807 18528 consensus_queue.cc:260] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:20:31.211109 18528 ts_tablet_manager.cc:1434] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da: Time spent starting tablet: real 0.006s	user 0.004s	sys 0.000s
I20260709 12:20:31.212827 17848 tablet_copy_service.cc:342] P e063541542a94969898868bf6eb5ce1b: Request end of tablet copy session 334a4e1f59944544b3b3ff36616b57da-0f940f29438f475294357324452abe76 received from {username='slave'} at 127.0.0.1:37994
I20260709 12:20:31.213259 17848 tablet_copy_service.cc:434] P e063541542a94969898868bf6eb5ce1b: ending tablet copy session 334a4e1f59944544b3b3ff36616b57da-0f940f29438f475294357324452abe76 on tablet 0f940f29438f475294357324452abe76 with peer 334a4e1f59944544b3b3ff36616b57da
I20260709 12:20:31.230754 18523 tablet_bootstrap.cc:492] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:20:31.231691 18523 tablet_bootstrap.cc:492] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da: Bootstrap complete.
I20260709 12:20:31.232404 18523 ts_tablet_manager.cc:1403] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da: Time spent bootstrapping tablet: real 0.027s	user 0.022s	sys 0.000s
I20260709 12:20:31.235119 18523 raft_consensus.cc:359] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:20:31.235951 18523 raft_consensus.cc:740] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Initialized, Role: LEARNER
I20260709 12:20:31.236564 18523 consensus_queue.cc:260] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:20:31.238850 18523 ts_tablet_manager.cc:1434] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da: Time spent starting tablet: real 0.006s	user 0.004s	sys 0.000s
I20260709 12:20:31.241288 17699 tablet_copy_service.cc:342] P 0da169ff82674dc1947055ce43de68a3: Request end of tablet copy session 334a4e1f59944544b3b3ff36616b57da-23c0cde5227f470388b8ed1fc0786119 received from {username='slave'} at 127.0.0.1:44898
I20260709 12:20:31.241750 17699 tablet_copy_service.cc:434] P 0da169ff82674dc1947055ce43de68a3: ending tablet copy session 334a4e1f59944544b3b3ff36616b57da-23c0cde5227f470388b8ed1fc0786119 on tablet 23c0cde5227f470388b8ed1fc0786119 with peer 334a4e1f59944544b3b3ff36616b57da
I20260709 12:20:31.379434 18368 raft_consensus.cc:1217] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:20:31.445936 18368 raft_consensus.cc:1217] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:20:31.557336 18368 raft_consensus.cc:1217] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:20:31.742532 18368 raft_consensus.cc:1217] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:20:31.885463 18519 raft_consensus.cc:1064] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b: attempting to promote NON_VOTER 334a4e1f59944544b3b3ff36616b57da to VOTER
I20260709 12:20:31.888152 18519 consensus_queue.cc:237] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:20:31.897372 18368 raft_consensus.cc:1275] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 1 LEARNER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:20:31.897855 17758 raft_consensus.cc:1275] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:20:31.899307 17686 raft_consensus.cc:1275] T 217734cc2eb24a7a8d92bfef30ee5c52 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:20:31.900056 18488 consensus_queue.cc:1048] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:20:31.906751 18488 consensus_queue.cc:1048] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:20:31.907625 18535 consensus_queue.cc:1048] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:20:31.919715 18482 raft_consensus.cc:1064] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb: attempting to promote NON_VOTER 334a4e1f59944544b3b3ff36616b57da to VOTER
I20260709 12:20:31.924283 18482 consensus_queue.cc:237] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:20:31.949450 18540 raft_consensus.cc:2955] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:31.951229 18367 raft_consensus.cc:1275] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 1 LEARNER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:20:31.963254 17686 raft_consensus.cc:1275] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:20:31.965023 17763 raft_consensus.cc:2955] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:31.967084 17838 raft_consensus.cc:1275] T 9439565c5dbb4f4ba3375aaef2c9add5 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:20:31.960752 18543 consensus_queue.cc:1048] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.007s
I20260709 12:20:31.981191 18537 consensus_queue.cc:1048] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:20:31.996299 18545 consensus_queue.cc:1048] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.002s
W20260709 12:20:31.996537 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 43761.9us
I20260709 12:20:32.021888 17587 catalog_manager.cc:5697] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb reported cstate change: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
W20260709 12:20:32.015425 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 46232.5us
I20260709 12:20:32.022899 17681 raft_consensus.cc:2955] T 217734cc2eb24a7a8d92bfef30ee5c52 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
W20260709 12:20:32.096966 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 59451.5us
I20260709 12:20:32.147706 18538 raft_consensus.cc:1064] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3: attempting to promote NON_VOTER 334a4e1f59944544b3b3ff36616b57da to VOTER
I20260709 12:20:32.151027 18538 consensus_queue.cc:237] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:20:32.189685 18368 raft_consensus.cc:2955] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
W20260709 12:20:32.205096 18552 log.cc:927] Time spent T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.079s	user 0.002s	sys 0.000s
I20260709 12:20:32.205255 18559 raft_consensus.cc:1064] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b: attempting to promote NON_VOTER 334a4e1f59944544b3b3ff36616b57da to VOTER
I20260709 12:20:32.217047 18559 consensus_queue.cc:237] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:20:32.223474 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 53843.6us
I20260709 12:20:32.226284 18367 raft_consensus.cc:1275] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 1 LEARNER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:20:32.231848 18537 raft_consensus.cc:2955] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:32.236940 18367 raft_consensus.cc:1275] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 1 LEARNER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
W20260709 12:20:32.242228 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 134049us
I20260709 12:20:32.246492 17760 raft_consensus.cc:1275] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:20:32.246796 17763 raft_consensus.cc:1275] T 23c0cde5227f470388b8ed1fc0786119 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:20:32.251230 17836 raft_consensus.cc:1275] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:20:32.251232 18538 consensus_queue.cc:1048] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.002s
I20260709 12:20:32.256296 18522 consensus_queue.cc:1048] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:20:32.264488 18488 consensus_queue.cc:1048] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:20:32.266588 18444 consensus_queue.cc:1048] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:20:32.280664 18444 raft_consensus.cc:2955] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:32.283593 17681 raft_consensus.cc:1275] T 0f940f29438f475294357324452abe76 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:20:32.292578 17836 raft_consensus.cc:2955] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:32.290560 17681 raft_consensus.cc:2955] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:32.301319 18488 consensus_queue.cc:1048] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:20:32.300812 18368 raft_consensus.cc:2955] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:32.315838 18488 consensus_queue.cc:1048] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:20:32.315322 18368 raft_consensus.cc:2955] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:32.319422 17835 raft_consensus.cc:2955] T 9439565c5dbb4f4ba3375aaef2c9add5 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:32.350325 17756 raft_consensus.cc:2955] T 23c0cde5227f470388b8ed1fc0786119 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:32.410744 18540 raft_consensus.cc:2955] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:32.418931 18368 raft_consensus.cc:2955] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:32.430344 17584 catalog_manager.cc:5697] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da reported cstate change: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:32.432245 17584 catalog_manager.cc:5697] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da reported cstate change: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:32.442121 17756 raft_consensus.cc:2955] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:32.450455 17684 raft_consensus.cc:2955] T 0f940f29438f475294357324452abe76 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:32.485635 17586 catalog_manager.cc:5697] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b reported cstate change: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:20:32.484136 17684 consensus_queue.cc:237] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:20:32.492422 18368 raft_consensus.cc:1275] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:20:32.494247 18563 consensus_queue.cc:1048] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:20:32.499611 17838 raft_consensus.cc:1275] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:20:32.501166 18564 consensus_queue.cc:1048] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:20:32.492812 17756 consensus_queue.cc:237] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:20:32.509974 17684 raft_consensus.cc:1275] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:20:32.509614 17838 raft_consensus.cc:2955] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:32.511329 18482 consensus_queue.cc:1048] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:20:32.511930 17836 consensus_queue.cc:237] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:20:32.519251 17835 consensus_queue.cc:237] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:20:32.520841 18367 raft_consensus.cc:1275] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:20:32.505153 18564 raft_consensus.cc:2955] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:32.522442 18547 consensus_queue.cc:1048] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:20:32.526816 18543 raft_consensus.cc:2955] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER e063541542a94969898868bf6eb5ce1b (127.16.29.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:32.531337 18367 raft_consensus.cc:1275] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:20:32.533031 17756 raft_consensus.cc:1275] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:20:32.534296 18368 raft_consensus.cc:2955] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:32.538870 17684 raft_consensus.cc:2955] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER e063541542a94969898868bf6eb5ce1b (127.16.29.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:32.562064 18535 consensus_queue.cc:1048] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:20:32.564278 18561 consensus_queue.cc:1048] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:20:32.564987 18371 raft_consensus.cc:2955] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER e063541542a94969898868bf6eb5ce1b (127.16.29.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:32.572259 18367 raft_consensus.cc:1275] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:20:32.573904 17571 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 9439565c5dbb4f4ba3375aaef2c9add5 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:20:32.580236 17572 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 23c0cde5227f470388b8ed1fc0786119 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:20:32.582209 17756 raft_consensus.cc:1275] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:20:32.583449 18535 consensus_queue.cc:1048] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:20:32.585170 18558 raft_consensus.cc:2955] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:32.593029 18560 consensus_queue.cc:1048] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:20:32.595388 18374 raft_consensus.cc:2955] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:32.597352 17758 raft_consensus.cc:2955] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:32.616770 17585 catalog_manager.cc:5697] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b reported cstate change: config changed from index 3 to 4, VOTER e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66) evicted. New cstate: current_term: 1 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:32.615579 17584 catalog_manager.cc:5697] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 reported cstate change: config changed from index 3 to 4, VOTER e063541542a94969898868bf6eb5ce1b (127.16.29.67) evicted. New cstate: current_term: 1 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:32.623144 18540 raft_consensus.cc:2955] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:32.627039 17571 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 0f940f29438f475294357324452abe76 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:20:32.634276 17571 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 217734cc2eb24a7a8d92bfef30ee5c52 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:20:32.635794 17753 tablet_service.cc:1558] Processing DeleteTablet for tablet 23c0cde5227f470388b8ed1fc0786119 with delete_type TABLET_DATA_TOMBSTONED (TS e2909cff1da44ad387f96a8ae3aa5afb not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:38774
I20260709 12:20:32.635385 18367 raft_consensus.cc:2955] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:32.638783 18576 tablet_replica.cc:333] T 23c0cde5227f470388b8ed1fc0786119 P e2909cff1da44ad387f96a8ae3aa5afb: stopping tablet replica
I20260709 12:20:32.641309 18576 raft_consensus.cc:2243] T 23c0cde5227f470388b8ed1fc0786119 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:32.643025 18576 raft_consensus.cc:2272] T 23c0cde5227f470388b8ed1fc0786119 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:32.640182 17758 raft_consensus.cc:2955] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:32.653105 18576 ts_tablet_manager.cc:1916] T 23c0cde5227f470388b8ed1fc0786119 P e2909cff1da44ad387f96a8ae3aa5afb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:20:32.695392 17586 catalog_manager.cc:5697] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b reported cstate change: config changed from index 3 to 4, VOTER 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) evicted. New cstate: current_term: 1 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:20:32.777513 17586 catalog_manager.cc:5697] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b reported cstate change: config changed from index 3 to 4, VOTER 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) evicted. New cstate: current_term: 1 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:20:32.862767 17676 tablet_service.cc:1558] Processing DeleteTablet for tablet 0f940f29438f475294357324452abe76 with delete_type TABLET_DATA_TOMBSTONED (TS 0da169ff82674dc1947055ce43de68a3 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:37384
I20260709 12:20:32.864867 17674 tablet_service.cc:1558] Processing DeleteTablet for tablet 217734cc2eb24a7a8d92bfef30ee5c52 with delete_type TABLET_DATA_TOMBSTONED (TS 0da169ff82674dc1947055ce43de68a3 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:37384
I20260709 12:20:32.937146 18576 ts_tablet_manager.cc:1929] T 23c0cde5227f470388b8ed1fc0786119 P e2909cff1da44ad387f96a8ae3aa5afb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:20:32.937810 18576 log.cc:1199] T 23c0cde5227f470388b8ed1fc0786119 P e2909cff1da44ad387f96a8ae3aa5afb: Deleting WAL directory at /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599551912677-16501-0/minicluster-data/ts-1-root/wals/23c0cde5227f470388b8ed1fc0786119
I20260709 12:20:32.940374 17571 catalog_manager.cc:5028] TS e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): tablet 23c0cde5227f470388b8ed1fc0786119 (table test-workload [id=ddc5b74250df4c65909b73b3742d615f]) successfully deleted
I20260709 12:20:32.946065 17821 tablet_service.cc:1558] Processing DeleteTablet for tablet 9439565c5dbb4f4ba3375aaef2c9add5 with delete_type TABLET_DATA_TOMBSTONED (TS e063541542a94969898868bf6eb5ce1b not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:35032
I20260709 12:20:32.949168 18577 tablet_replica.cc:333] T 0f940f29438f475294357324452abe76 P 0da169ff82674dc1947055ce43de68a3: stopping tablet replica
I20260709 12:20:32.959959 18577 raft_consensus.cc:2243] T 0f940f29438f475294357324452abe76 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:32.964563 18577 raft_consensus.cc:2272] T 0f940f29438f475294357324452abe76 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:32.975512 18577 ts_tablet_manager.cc:1916] T 0f940f29438f475294357324452abe76 P 0da169ff82674dc1947055ce43de68a3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:20:32.986917 18578 tablet_replica.cc:333] T 9439565c5dbb4f4ba3375aaef2c9add5 P e063541542a94969898868bf6eb5ce1b: stopping tablet replica
I20260709 12:20:32.989221 18578 raft_consensus.cc:2243] T 9439565c5dbb4f4ba3375aaef2c9add5 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:32.991536 18578 raft_consensus.cc:2272] T 9439565c5dbb4f4ba3375aaef2c9add5 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:32.999228 18577 ts_tablet_manager.cc:1929] T 0f940f29438f475294357324452abe76 P 0da169ff82674dc1947055ce43de68a3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:20:32.999807 18577 log.cc:1199] T 0f940f29438f475294357324452abe76 P 0da169ff82674dc1947055ce43de68a3: Deleting WAL directory at /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599551912677-16501-0/minicluster-data/ts-0-root/wals/0f940f29438f475294357324452abe76
I20260709 12:20:33.003166 17572 catalog_manager.cc:5028] TS 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): tablet 0f940f29438f475294357324452abe76 (table test-workload [id=ddc5b74250df4c65909b73b3742d615f]) successfully deleted
I20260709 12:20:33.004634 18577 tablet_replica.cc:333] T 217734cc2eb24a7a8d92bfef30ee5c52 P 0da169ff82674dc1947055ce43de68a3: stopping tablet replica
I20260709 12:20:33.005913 18577 raft_consensus.cc:2243] T 217734cc2eb24a7a8d92bfef30ee5c52 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:33.006920 18577 raft_consensus.cc:2272] T 217734cc2eb24a7a8d92bfef30ee5c52 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:33.007682 18578 ts_tablet_manager.cc:1916] T 9439565c5dbb4f4ba3375aaef2c9add5 P e063541542a94969898868bf6eb5ce1b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:20:33.011432 18577 ts_tablet_manager.cc:1916] T 217734cc2eb24a7a8d92bfef30ee5c52 P 0da169ff82674dc1947055ce43de68a3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:20:33.030682 18578 ts_tablet_manager.cc:1929] T 9439565c5dbb4f4ba3375aaef2c9add5 P e063541542a94969898868bf6eb5ce1b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:20:33.031163 18578 log.cc:1199] T 9439565c5dbb4f4ba3375aaef2c9add5 P e063541542a94969898868bf6eb5ce1b: Deleting WAL directory at /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599551912677-16501-0/minicluster-data/ts-2-root/wals/9439565c5dbb4f4ba3375aaef2c9add5
I20260709 12:20:33.033004 18577 ts_tablet_manager.cc:1929] T 217734cc2eb24a7a8d92bfef30ee5c52 P 0da169ff82674dc1947055ce43de68a3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:20:33.033234 17571 catalog_manager.cc:5028] TS e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837): tablet 9439565c5dbb4f4ba3375aaef2c9add5 (table test-workload [id=ddc5b74250df4c65909b73b3742d615f]) successfully deleted
I20260709 12:20:33.033619 18577 log.cc:1199] T 217734cc2eb24a7a8d92bfef30ee5c52 P 0da169ff82674dc1947055ce43de68a3: Deleting WAL directory at /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599551912677-16501-0/minicluster-data/ts-0-root/wals/217734cc2eb24a7a8d92bfef30ee5c52
I20260709 12:20:33.034873 17572 catalog_manager.cc:5028] TS 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): tablet 217734cc2eb24a7a8d92bfef30ee5c52 (table test-workload [id=ddc5b74250df4c65909b73b3742d615f]) successfully deleted
I20260709 12:20:34.054411 17758 consensus_queue.cc:237] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
W20260709 12:20:34.064023 17729 consensus_peers.cc:597] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb -> Peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Couldn't send request to peer 334a4e1f59944544b3b3ff36616b57da. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d11ed0a8f1094765abe028dc60aa8bed. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:34.067915 17834 raft_consensus.cc:1275] T d11ed0a8f1094765abe028dc60aa8bed P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:20:34.067915 17682 raft_consensus.cc:1275] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:20:34.070106 18579 consensus_queue.cc:1048] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:20:34.071202 18537 consensus_queue.cc:1048] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:20:34.141108 18537 raft_consensus.cc:2955] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } }
W20260709 12:20:34.160985 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 63607.7us
I20260709 12:20:34.164675 17682 raft_consensus.cc:2955] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } }
I20260709 12:20:34.188531 17686 consensus_queue.cc:237] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:20:34.199849 17834 raft_consensus.cc:2955] T d11ed0a8f1094765abe028dc60aa8bed P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } }
I20260709 12:20:34.207744 17586 catalog_manager.cc:5697] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb reported cstate change: config changed from index -1 to 2, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New cstate: current_term: 1 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20260709 12:20:34.224471 17654 consensus_peers.cc:597] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 -> Peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Couldn't send request to peer 334a4e1f59944544b3b3ff36616b57da. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 56b500bd1b914ebfb3f19afc8a330665. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:34.226182 17834 raft_consensus.cc:1275] T 56b500bd1b914ebfb3f19afc8a330665 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:20:34.228401 17756 raft_consensus.cc:1275] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:20:34.228209 18593 consensus_queue.cc:1048] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:20:34.232168 18593 consensus_queue.cc:1048] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:20:34.240972 18593 raft_consensus.cc:2955] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } }
I20260709 12:20:34.250721 17835 raft_consensus.cc:2955] T 56b500bd1b914ebfb3f19afc8a330665 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } }
I20260709 12:20:34.251587 17756 raft_consensus.cc:2955] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } }
I20260709 12:20:34.266404 17761 consensus_queue.cc:237] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
W20260709 12:20:34.279567 17729 consensus_peers.cc:597] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb -> Peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Couldn't send request to peer 334a4e1f59944544b3b3ff36616b57da. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f5f0dd0890f44e35bbb838da838faa9c. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:34.279253 17584 catalog_manager.cc:5697] T 56b500bd1b914ebfb3f19afc8a330665 P e063541542a94969898868bf6eb5ce1b reported cstate change: config changed from index -1 to 2, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New cstate: current_term: 1 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } }
I20260709 12:20:34.282424 17835 raft_consensus.cc:1275] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:20:34.284441 18587 consensus_queue.cc:1048] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:20:34.287822 17684 raft_consensus.cc:1275] T f5f0dd0890f44e35bbb838da838faa9c P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:20:34.289525 18587 consensus_queue.cc:1048] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:20:34.326426 18537 raft_consensus.cc:2955] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } }
I20260709 12:20:34.328666 17835 raft_consensus.cc:2955] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } }
I20260709 12:20:34.346006 17684 raft_consensus.cc:2955] T f5f0dd0890f44e35bbb838da838faa9c P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } }
I20260709 12:20:34.349807 17584 catalog_manager.cc:5697] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b reported cstate change: config changed from index -1 to 2, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New cstate: current_term: 1 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } }
I20260709 12:20:34.353457 17686 consensus_queue.cc:237] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
W20260709 12:20:34.368144 17654 consensus_peers.cc:597] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 -> Peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Couldn't send request to peer 334a4e1f59944544b3b3ff36616b57da. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8a208e793881438b975b83755dc05cfa. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:34.369004 17756 raft_consensus.cc:1275] T 8a208e793881438b975b83755dc05cfa P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:20:34.370078 17835 raft_consensus.cc:1275] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:20:34.371778 18592 consensus_queue.cc:1048] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:20:34.372752 18588 consensus_queue.cc:1048] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:20:34.383558 18588 raft_consensus.cc:2955] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } }
I20260709 12:20:34.385663 17835 raft_consensus.cc:2955] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } }
I20260709 12:20:34.388909 17756 raft_consensus.cc:2955] T 8a208e793881438b975b83755dc05cfa P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } }
I20260709 12:20:34.404474 17584 catalog_manager.cc:5697] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 reported cstate change: config changed from index -1 to 2, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New cstate: current_term: 1 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:34.522286 18608 ts_tablet_manager.cc:933] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da: Initiating tablet copy from peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:20:34.524784 18608 tablet_copy_client.cc:323] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Beginning tablet copy session from remote peer at address 127.16.29.66:39651
I20260709 12:20:34.526762 17774 tablet_copy_service.cc:140] P e2909cff1da44ad387f96a8ae3aa5afb: Received BeginTabletCopySession request for tablet d11ed0a8f1094765abe028dc60aa8bed from peer 334a4e1f59944544b3b3ff36616b57da ({username='slave'} at 127.0.0.1:36572)
I20260709 12:20:34.527329 17774 tablet_copy_service.cc:161] P e2909cff1da44ad387f96a8ae3aa5afb: Beginning new tablet copy session on tablet d11ed0a8f1094765abe028dc60aa8bed from peer 334a4e1f59944544b3b3ff36616b57da at {username='slave'} at 127.0.0.1:36572: session id = 334a4e1f59944544b3b3ff36616b57da-d11ed0a8f1094765abe028dc60aa8bed
I20260709 12:20:34.534281 17774 tablet_copy_source_session.cc:215] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:20:34.537928 18608 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d11ed0a8f1094765abe028dc60aa8bed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:34.553766 18608 tablet_copy_client.cc:806] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Starting download of 0 data blocks...
I20260709 12:20:34.554459 18608 tablet_copy_client.cc:670] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Starting download of 1 WAL segments...
I20260709 12:20:34.558830 18608 tablet_copy_client.cc:538] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:20:34.567799 18608 tablet_bootstrap.cc:492] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da: Bootstrap starting.
I20260709 12:20:34.587924 18608 tablet_bootstrap.cc:492] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:20:34.588843 18608 tablet_bootstrap.cc:492] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da: Bootstrap complete.
I20260709 12:20:34.589514 18608 ts_tablet_manager.cc:1403] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da: Time spent bootstrapping tablet: real 0.022s	user 0.017s	sys 0.006s
I20260709 12:20:34.593271 18608 raft_consensus.cc:359] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:20:34.594503 18608 raft_consensus.cc:740] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Initialized, Role: LEARNER
I20260709 12:20:34.595230 18608 consensus_queue.cc:260] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:20:34.604310 18608 ts_tablet_manager.cc:1434] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da: Time spent starting tablet: real 0.014s	user 0.009s	sys 0.002s
I20260709 12:20:34.606315 17774 tablet_copy_service.cc:342] P e2909cff1da44ad387f96a8ae3aa5afb: Request end of tablet copy session 334a4e1f59944544b3b3ff36616b57da-d11ed0a8f1094765abe028dc60aa8bed received from {username='slave'} at 127.0.0.1:36572
I20260709 12:20:34.607153 17774 tablet_copy_service.cc:434] P e2909cff1da44ad387f96a8ae3aa5afb: ending tablet copy session 334a4e1f59944544b3b3ff36616b57da-d11ed0a8f1094765abe028dc60aa8bed on tablet d11ed0a8f1094765abe028dc60aa8bed with peer 334a4e1f59944544b3b3ff36616b57da
I20260709 12:20:34.690605 18608 ts_tablet_manager.cc:933] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da: Initiating tablet copy from peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071)
I20260709 12:20:34.692755 18608 tablet_copy_client.cc:323] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Beginning tablet copy session from remote peer at address 127.16.29.65:33071
I20260709 12:20:34.694582 17699 tablet_copy_service.cc:140] P 0da169ff82674dc1947055ce43de68a3: Received BeginTabletCopySession request for tablet 56b500bd1b914ebfb3f19afc8a330665 from peer 334a4e1f59944544b3b3ff36616b57da ({username='slave'} at 127.0.0.1:44898)
I20260709 12:20:34.695128 17699 tablet_copy_service.cc:161] P 0da169ff82674dc1947055ce43de68a3: Beginning new tablet copy session on tablet 56b500bd1b914ebfb3f19afc8a330665 from peer 334a4e1f59944544b3b3ff36616b57da at {username='slave'} at 127.0.0.1:44898: session id = 334a4e1f59944544b3b3ff36616b57da-56b500bd1b914ebfb3f19afc8a330665
I20260709 12:20:34.702435 17699 tablet_copy_source_session.cc:215] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:20:34.705615 18608 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 56b500bd1b914ebfb3f19afc8a330665. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:34.710841 18613 ts_tablet_manager.cc:933] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da: Initiating tablet copy from peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:20:34.712823 18613 tablet_copy_client.cc:323] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Beginning tablet copy session from remote peer at address 127.16.29.66:39651
I20260709 12:20:34.714156 17774 tablet_copy_service.cc:140] P e2909cff1da44ad387f96a8ae3aa5afb: Received BeginTabletCopySession request for tablet f5f0dd0890f44e35bbb838da838faa9c from peer 334a4e1f59944544b3b3ff36616b57da ({username='slave'} at 127.0.0.1:36572)
I20260709 12:20:34.714589 17774 tablet_copy_service.cc:161] P e2909cff1da44ad387f96a8ae3aa5afb: Beginning new tablet copy session on tablet f5f0dd0890f44e35bbb838da838faa9c from peer 334a4e1f59944544b3b3ff36616b57da at {username='slave'} at 127.0.0.1:36572: session id = 334a4e1f59944544b3b3ff36616b57da-f5f0dd0890f44e35bbb838da838faa9c
I20260709 12:20:34.719493 17774 tablet_copy_source_session.cc:215] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:20:34.722818 18613 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f5f0dd0890f44e35bbb838da838faa9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:34.729344 18608 tablet_copy_client.cc:806] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Starting download of 0 data blocks...
I20260709 12:20:34.730010 18608 tablet_copy_client.cc:670] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Starting download of 1 WAL segments...
I20260709 12:20:34.736379 18608 tablet_copy_client.cc:538] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:20:34.740868 18613 tablet_copy_client.cc:806] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Starting download of 0 data blocks...
I20260709 12:20:34.741463 18613 tablet_copy_client.cc:670] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Starting download of 1 WAL segments...
I20260709 12:20:34.744499 18613 tablet_copy_client.cc:538] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:20:34.745879 18608 tablet_bootstrap.cc:492] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da: Bootstrap starting.
I20260709 12:20:34.754086 18613 tablet_bootstrap.cc:492] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da: Bootstrap starting.
I20260709 12:20:34.779812 18608 tablet_bootstrap.cc:492] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:20:34.780632 18613 tablet_bootstrap.cc:492] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:20:34.780872 18608 tablet_bootstrap.cc:492] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da: Bootstrap complete.
I20260709 12:20:34.781574 18613 tablet_bootstrap.cc:492] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da: Bootstrap complete.
I20260709 12:20:34.781656 18608 ts_tablet_manager.cc:1403] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da: Time spent bootstrapping tablet: real 0.036s	user 0.012s	sys 0.012s
I20260709 12:20:34.782335 18613 ts_tablet_manager.cc:1403] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da: Time spent bootstrapping tablet: real 0.028s	user 0.011s	sys 0.014s
I20260709 12:20:34.785698 18613 raft_consensus.cc:359] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:20:34.786473 18613 raft_consensus.cc:740] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Initialized, Role: LEARNER
I20260709 12:20:34.787137 18613 consensus_queue.cc:260] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:20:34.790679 18608 raft_consensus.cc:359] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:20:34.791566 18608 raft_consensus.cc:740] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Initialized, Role: LEARNER
I20260709 12:20:34.792232 18608 consensus_queue.cc:260] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:20:34.794780 18608 ts_tablet_manager.cc:1434] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da: Time spent starting tablet: real 0.013s	user 0.001s	sys 0.005s
I20260709 12:20:34.795293 18613 ts_tablet_manager.cc:1434] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da: Time spent starting tablet: real 0.012s	user 0.002s	sys 0.003s
I20260709 12:20:34.797257 17774 tablet_copy_service.cc:342] P e2909cff1da44ad387f96a8ae3aa5afb: Request end of tablet copy session 334a4e1f59944544b3b3ff36616b57da-f5f0dd0890f44e35bbb838da838faa9c received from {username='slave'} at 127.0.0.1:36572
I20260709 12:20:34.797672 17774 tablet_copy_service.cc:434] P e2909cff1da44ad387f96a8ae3aa5afb: ending tablet copy session 334a4e1f59944544b3b3ff36616b57da-f5f0dd0890f44e35bbb838da838faa9c on tablet f5f0dd0890f44e35bbb838da838faa9c with peer 334a4e1f59944544b3b3ff36616b57da
I20260709 12:20:34.798395 17699 tablet_copy_service.cc:342] P 0da169ff82674dc1947055ce43de68a3: Request end of tablet copy session 334a4e1f59944544b3b3ff36616b57da-56b500bd1b914ebfb3f19afc8a330665 received from {username='slave'} at 127.0.0.1:44898
I20260709 12:20:34.798846 17699 tablet_copy_service.cc:434] P 0da169ff82674dc1947055ce43de68a3: ending tablet copy session 334a4e1f59944544b3b3ff36616b57da-56b500bd1b914ebfb3f19afc8a330665 on tablet 56b500bd1b914ebfb3f19afc8a330665 with peer 334a4e1f59944544b3b3ff36616b57da
I20260709 12:20:34.850535 18608 ts_tablet_manager.cc:933] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da: Initiating tablet copy from peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071)
I20260709 12:20:34.852383 18608 tablet_copy_client.cc:323] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Beginning tablet copy session from remote peer at address 127.16.29.65:33071
I20260709 12:20:34.854158 17699 tablet_copy_service.cc:140] P 0da169ff82674dc1947055ce43de68a3: Received BeginTabletCopySession request for tablet 8a208e793881438b975b83755dc05cfa from peer 334a4e1f59944544b3b3ff36616b57da ({username='slave'} at 127.0.0.1:44898)
I20260709 12:20:34.854691 17699 tablet_copy_service.cc:161] P 0da169ff82674dc1947055ce43de68a3: Beginning new tablet copy session on tablet 8a208e793881438b975b83755dc05cfa from peer 334a4e1f59944544b3b3ff36616b57da at {username='slave'} at 127.0.0.1:44898: session id = 334a4e1f59944544b3b3ff36616b57da-8a208e793881438b975b83755dc05cfa
I20260709 12:20:34.862272 17699 tablet_copy_source_session.cc:215] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:20:34.867452 18608 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a208e793881438b975b83755dc05cfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:34.885861 18608 tablet_copy_client.cc:806] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Starting download of 0 data blocks...
I20260709 12:20:34.886572 18608 tablet_copy_client.cc:670] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Starting download of 1 WAL segments...
I20260709 12:20:34.890197 18608 tablet_copy_client.cc:538] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:20:34.895776 18608 tablet_bootstrap.cc:492] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da: Bootstrap starting.
I20260709 12:20:34.911424 18608 tablet_bootstrap.cc:492] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:20:34.912407 18608 tablet_bootstrap.cc:492] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da: Bootstrap complete.
I20260709 12:20:34.913121 18608 ts_tablet_manager.cc:1403] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da: Time spent bootstrapping tablet: real 0.017s	user 0.016s	sys 0.004s
I20260709 12:20:34.915855 18608 raft_consensus.cc:359] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:20:34.916738 18608 raft_consensus.cc:740] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Initialized, Role: LEARNER
I20260709 12:20:34.917428 18608 consensus_queue.cc:260] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:20:34.919636 18608 ts_tablet_manager.cc:1434] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da: Time spent starting tablet: real 0.006s	user 0.000s	sys 0.005s
I20260709 12:20:34.921602 17699 tablet_copy_service.cc:342] P 0da169ff82674dc1947055ce43de68a3: Request end of tablet copy session 334a4e1f59944544b3b3ff36616b57da-8a208e793881438b975b83755dc05cfa received from {username='slave'} at 127.0.0.1:44898
I20260709 12:20:34.922036 17699 tablet_copy_service.cc:434] P 0da169ff82674dc1947055ce43de68a3: ending tablet copy session 334a4e1f59944544b3b3ff36616b57da-8a208e793881438b975b83755dc05cfa on tablet 8a208e793881438b975b83755dc05cfa with peer 334a4e1f59944544b3b3ff36616b57da
I20260709 12:20:34.970363 18374 raft_consensus.cc:1217] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:20:35.155001 18374 raft_consensus.cc:1217] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:20:35.241858 18374 raft_consensus.cc:1217] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:20:35.450381 18374 raft_consensus.cc:1217] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:20:35.596441 18619 raft_consensus.cc:1064] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb: attempting to promote NON_VOTER 334a4e1f59944544b3b3ff36616b57da to VOTER
I20260709 12:20:35.598974 18619 consensus_queue.cc:237] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:20:35.605710 18374 raft_consensus.cc:1275] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 1 LEARNER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:20:35.606755 17684 raft_consensus.cc:1275] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:20:35.607654 18537 consensus_queue.cc:1048] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:20:35.609025 18537 consensus_queue.cc:1048] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:20:35.622128 18619 raft_consensus.cc:2955] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:35.624096 18374 raft_consensus.cc:2955] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:35.629477 17837 raft_consensus.cc:1275] T d11ed0a8f1094765abe028dc60aa8bed P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:20:35.630987 18537 consensus_queue.cc:1048] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:20:35.640920 17684 raft_consensus.cc:2955] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:35.644395 17837 raft_consensus.cc:2955] T d11ed0a8f1094765abe028dc60aa8bed P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:35.679157 17587 catalog_manager.cc:5697] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da reported cstate change: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:35.700469 17761 consensus_queue.cc:237] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:20:35.705427 18374 raft_consensus.cc:1275] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:20:35.706974 18537 consensus_queue.cc:1048] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:20:35.710157 17684 raft_consensus.cc:1275] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:20:35.711570 18537 consensus_queue.cc:1048] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:20:35.715698 18619 raft_consensus.cc:2955] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER e063541542a94969898868bf6eb5ce1b (127.16.29.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:35.717962 18374 raft_consensus.cc:2955] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER e063541542a94969898868bf6eb5ce1b (127.16.29.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:35.719660 17684 raft_consensus.cc:2955] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER e063541542a94969898868bf6eb5ce1b (127.16.29.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:35.743690 17571 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet d11ed0a8f1094765abe028dc60aa8bed with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:20:35.743203 17585 catalog_manager.cc:5697] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 reported cstate change: config changed from index 3 to 4, VOTER e063541542a94969898868bf6eb5ce1b (127.16.29.67) evicted. New cstate: current_term: 1 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:35.756620 17821 tablet_service.cc:1558] Processing DeleteTablet for tablet d11ed0a8f1094765abe028dc60aa8bed with delete_type TABLET_DATA_TOMBSTONED (TS e063541542a94969898868bf6eb5ce1b not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:35032
I20260709 12:20:35.767668 18588 raft_consensus.cc:1064] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3: attempting to promote NON_VOTER 334a4e1f59944544b3b3ff36616b57da to VOTER
I20260709 12:20:35.768983 18631 tablet_replica.cc:333] T d11ed0a8f1094765abe028dc60aa8bed P e063541542a94969898868bf6eb5ce1b: stopping tablet replica
I20260709 12:20:35.770058 18631 raft_consensus.cc:2243] T d11ed0a8f1094765abe028dc60aa8bed P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:35.770566 18631 raft_consensus.cc:2272] T d11ed0a8f1094765abe028dc60aa8bed P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:35.770238 18588 consensus_queue.cc:237] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:20:35.775583 18631 ts_tablet_manager.cc:1916] T d11ed0a8f1094765abe028dc60aa8bed P e063541542a94969898868bf6eb5ce1b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:20:35.786638 18374 raft_consensus.cc:1275] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 1 LEARNER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:20:35.787251 17837 raft_consensus.cc:1275] T 56b500bd1b914ebfb3f19afc8a330665 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:20:35.788686 18620 consensus_queue.cc:1048] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:20:35.792891 17756 raft_consensus.cc:1275] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:20:35.794452 18631 ts_tablet_manager.cc:1929] T d11ed0a8f1094765abe028dc60aa8bed P e063541542a94969898868bf6eb5ce1b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:20:35.801183 18631 log.cc:1199] T d11ed0a8f1094765abe028dc60aa8bed P e063541542a94969898868bf6eb5ce1b: Deleting WAL directory at /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599551912677-16501-0/minicluster-data/ts-2-root/wals/d11ed0a8f1094765abe028dc60aa8bed
I20260709 12:20:35.796003 18592 consensus_queue.cc:1048] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.006s
I20260709 12:20:35.802884 17571 catalog_manager.cc:5028] TS e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837): tablet d11ed0a8f1094765abe028dc60aa8bed (table test-workload [id=ddc5b74250df4c65909b73b3742d615f]) successfully deleted
I20260709 12:20:35.802642 18620 consensus_queue.cc:1048] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.002s
I20260709 12:20:35.830466 17837 raft_consensus.cc:2955] T 56b500bd1b914ebfb3f19afc8a330665 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:35.832568 17756 raft_consensus.cc:2955] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:35.829293 18588 raft_consensus.cc:2955] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:35.848321 17584 catalog_manager.cc:5697] T 56b500bd1b914ebfb3f19afc8a330665 P e063541542a94969898868bf6eb5ce1b reported cstate change: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:35.866079 18619 raft_consensus.cc:1064] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb: attempting to promote NON_VOTER 334a4e1f59944544b3b3ff36616b57da to VOTER
I20260709 12:20:35.868757 18619 consensus_queue.cc:237] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:20:35.868294 18367 raft_consensus.cc:2955] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:35.880606 17684 consensus_queue.cc:237] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:20:35.881165 18374 raft_consensus.cc:1275] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 1 LEARNER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:20:35.884266 18537 consensus_queue.cc:1048] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:20:35.890707 17837 raft_consensus.cc:1275] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:20:35.893278 17686 raft_consensus.cc:1275] T f5f0dd0890f44e35bbb838da838faa9c P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:20:35.893024 18619 consensus_queue.cc:1048] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.003s
I20260709 12:20:35.896345 18537 consensus_queue.cc:1048] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:20:35.900581 18374 raft_consensus.cc:1275] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:20:35.902120 18588 consensus_queue.cc:1048] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:20:35.905207 17756 raft_consensus.cc:1275] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:20:35.908438 18620 consensus_queue.cc:1048] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:20:35.920408 18635 raft_consensus.cc:2955] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER e063541542a94969898868bf6eb5ce1b (127.16.29.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:35.995020 18368 raft_consensus.cc:2955] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER e063541542a94969898868bf6eb5ce1b (127.16.29.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
W20260709 12:20:35.997859 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 64336.4us
I20260709 12:20:35.999655 17756 raft_consensus.cc:2955] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER e063541542a94969898868bf6eb5ce1b (127.16.29.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:36.001072 18588 raft_consensus.cc:1064] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3: attempting to promote NON_VOTER 334a4e1f59944544b3b3ff36616b57da to VOTER
I20260709 12:20:36.006150 18626 raft_consensus.cc:2955] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:36.003561 18588 consensus_queue.cc:237] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:20:36.098044 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 77986.6us
I20260709 12:20:36.103919 17572 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 56b500bd1b914ebfb3f19afc8a330665 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:20:36.116964 17584 catalog_manager.cc:5697] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb reported cstate change: config changed from index 3 to 4, VOTER e063541542a94969898868bf6eb5ce1b (127.16.29.67) evicted. New cstate: current_term: 1 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
W20260709 12:20:36.139681 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 49785.9us
I20260709 12:20:36.146996 17837 raft_consensus.cc:1275] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:20:36.148711 18588 consensus_queue.cc:1048] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:20:36.152940 18368 raft_consensus.cc:1275] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 1 LEARNER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:20:36.156584 18620 consensus_queue.cc:1048] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:20:36.168612 18367 raft_consensus.cc:2955] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:36.177440 17761 raft_consensus.cc:1275] T 8a208e793881438b975b83755dc05cfa P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:20:36.179404 18620 consensus_queue.cc:1048] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:20:36.203714 17684 raft_consensus.cc:2955] T f5f0dd0890f44e35bbb838da838faa9c P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:36.222393 18588 raft_consensus.cc:2955] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:36.225369 17821 tablet_service.cc:1558] Processing DeleteTablet for tablet 56b500bd1b914ebfb3f19afc8a330665 with delete_type TABLET_DATA_TOMBSTONED (TS e063541542a94969898868bf6eb5ce1b not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:35032
I20260709 12:20:36.226359 18631 tablet_replica.cc:333] T 56b500bd1b914ebfb3f19afc8a330665 P e063541542a94969898868bf6eb5ce1b: stopping tablet replica
I20260709 12:20:36.227581 18631 raft_consensus.cc:2243] T 56b500bd1b914ebfb3f19afc8a330665 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:36.228228 18631 raft_consensus.cc:2272] T 56b500bd1b914ebfb3f19afc8a330665 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:36.227984 17836 raft_consensus.cc:2955] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:36.233740 18368 raft_consensus.cc:2955] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:36.236994 18631 ts_tablet_manager.cc:1916] T 56b500bd1b914ebfb3f19afc8a330665 P e063541542a94969898868bf6eb5ce1b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20260709 12:20:36.239939 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 67321.7us
I20260709 12:20:36.255017 17586 catalog_manager.cc:5697] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da reported cstate change: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:36.261415 17761 raft_consensus.cc:2955] T 8a208e793881438b975b83755dc05cfa P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:36.272593 17586 catalog_manager.cc:5697] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da reported cstate change: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:36.274403 18631 ts_tablet_manager.cc:1929] T 56b500bd1b914ebfb3f19afc8a330665 P e063541542a94969898868bf6eb5ce1b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:20:36.274923 18631 log.cc:1199] T 56b500bd1b914ebfb3f19afc8a330665 P e063541542a94969898868bf6eb5ce1b: Deleting WAL directory at /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599551912677-16501-0/minicluster-data/ts-2-root/wals/56b500bd1b914ebfb3f19afc8a330665
I20260709 12:20:36.282799 17571 catalog_manager.cc:5028] TS e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837): tablet 56b500bd1b914ebfb3f19afc8a330665 (table test-workload [id=ddc5b74250df4c65909b73b3742d615f]) successfully deleted
I20260709 12:20:36.296461 17837 raft_consensus.cc:2955] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:36.324339 17761 consensus_queue.cc:237] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:20:36.327256 17684 consensus_queue.cc:237] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:20:36.329963 18368 raft_consensus.cc:1275] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:20:36.331334 17837 raft_consensus.cc:1275] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:20:36.333061 17834 raft_consensus.cc:1275] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:20:36.333338 18645 consensus_queue.cc:1048] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:20:36.334465 18537 consensus_queue.cc:1048] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:20:36.335312 18643 consensus_queue.cc:1048] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:20:36.338513 18368 raft_consensus.cc:1275] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:20:36.340348 18620 consensus_queue.cc:1048] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:20:36.344934 18537 raft_consensus.cc:2955] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:36.347970 17834 raft_consensus.cc:2955] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:36.349850 17836 raft_consensus.cc:2955] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:36.351560 18588 raft_consensus.cc:2955] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:36.359954 18367 raft_consensus.cc:2955] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:36.365198 17571 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet f5f0dd0890f44e35bbb838da838faa9c with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:20:36.368439 17584 catalog_manager.cc:5697] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b reported cstate change: config changed from index 3 to 4, VOTER 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) evicted. New cstate: current_term: 1 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:36.372866 17572 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 8a208e793881438b975b83755dc05cfa with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:20:36.385416 18374 raft_consensus.cc:2955] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:36.385685 17586 catalog_manager.cc:5697] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 reported cstate change: config changed from index 3 to 4, VOTER e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66) evicted. New cstate: current_term: 1 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:20:36.398108 17676 tablet_service.cc:1558] Processing DeleteTablet for tablet f5f0dd0890f44e35bbb838da838faa9c with delete_type TABLET_DATA_TOMBSTONED (TS 0da169ff82674dc1947055ce43de68a3 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:37384
I20260709 12:20:36.418293 18659 tablet_replica.cc:333] T f5f0dd0890f44e35bbb838da838faa9c P 0da169ff82674dc1947055ce43de68a3: stopping tablet replica
I20260709 12:20:36.419407 18659 raft_consensus.cc:2243] T f5f0dd0890f44e35bbb838da838faa9c P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:36.420145 18659 raft_consensus.cc:2272] T f5f0dd0890f44e35bbb838da838faa9c P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:36.423604 18659 ts_tablet_manager.cc:1916] T f5f0dd0890f44e35bbb838da838faa9c P 0da169ff82674dc1947055ce43de68a3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:20:36.425300 17753 tablet_service.cc:1558] Processing DeleteTablet for tablet 8a208e793881438b975b83755dc05cfa with delete_type TABLET_DATA_TOMBSTONED (TS e2909cff1da44ad387f96a8ae3aa5afb not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:38774
I20260709 12:20:36.440763 18660 tablet_replica.cc:333] T 8a208e793881438b975b83755dc05cfa P e2909cff1da44ad387f96a8ae3aa5afb: stopping tablet replica
I20260709 12:20:36.442569 18659 ts_tablet_manager.cc:1929] T f5f0dd0890f44e35bbb838da838faa9c P 0da169ff82674dc1947055ce43de68a3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:20:36.443070 18659 log.cc:1199] T f5f0dd0890f44e35bbb838da838faa9c P 0da169ff82674dc1947055ce43de68a3: Deleting WAL directory at /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599551912677-16501-0/minicluster-data/ts-0-root/wals/f5f0dd0890f44e35bbb838da838faa9c
I20260709 12:20:36.443130 18660 raft_consensus.cc:2243] T 8a208e793881438b975b83755dc05cfa P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:36.444024 18660 raft_consensus.cc:2272] T 8a208e793881438b975b83755dc05cfa P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:36.444706 17572 catalog_manager.cc:5028] TS 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): tablet f5f0dd0890f44e35bbb838da838faa9c (table test-workload [id=ddc5b74250df4c65909b73b3742d615f]) successfully deleted
I20260709 12:20:36.447460 18660 ts_tablet_manager.cc:1916] T 8a208e793881438b975b83755dc05cfa P e2909cff1da44ad387f96a8ae3aa5afb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:20:36.461663 18660 ts_tablet_manager.cc:1929] T 8a208e793881438b975b83755dc05cfa P e2909cff1da44ad387f96a8ae3aa5afb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:20:36.462110 18660 log.cc:1199] T 8a208e793881438b975b83755dc05cfa P e2909cff1da44ad387f96a8ae3aa5afb: Deleting WAL directory at /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599551912677-16501-0/minicluster-data/ts-1-root/wals/8a208e793881438b975b83755dc05cfa
I20260709 12:20:36.463974 17571 catalog_manager.cc:5028] TS e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): tablet 8a208e793881438b975b83755dc05cfa (table test-workload [id=ddc5b74250df4c65909b73b3742d615f]) successfully deleted
I20260709 12:20:37.571386 17834 consensus_queue.cc:237] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:20:37.579329 17761 raft_consensus.cc:1275] T b5462347eb74482b8cd11beb1d39b8a9 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260709 12:20:37.580510 17803 consensus_peers.cc:597] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b -> Peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Couldn't send request to peer 334a4e1f59944544b3b3ff36616b57da. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b5462347eb74482b8cd11beb1d39b8a9. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:37.580681 17686 raft_consensus.cc:1275] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:20:37.581130 18662 consensus_queue.cc:1048] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:20:37.582108 18649 consensus_queue.cc:1048] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:20:37.592919 18561 raft_consensus.cc:2955] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } }
I20260709 12:20:37.594161 17686 raft_consensus.cc:2955] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } }
I20260709 12:20:37.594655 17760 raft_consensus.cc:2955] T b5462347eb74482b8cd11beb1d39b8a9 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } }
I20260709 12:20:37.613945 17586 catalog_manager.cc:5697] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 reported cstate change: config changed from index -1 to 2, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New cstate: current_term: 1 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } }
I20260709 12:20:37.617759 17686 consensus_queue.cc:237] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:20:37.624198 17760 raft_consensus.cc:1275] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:20:37.625025 17834 raft_consensus.cc:1275] T 69716f6d5e7549418fdd8d54d7b722ce P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:20:37.625528 18620 consensus_queue.cc:1048] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:20:37.626653 18661 consensus_queue.cc:1048] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260709 12:20:37.629343 17654 consensus_peers.cc:597] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 -> Peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Couldn't send request to peer 334a4e1f59944544b3b3ff36616b57da. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 69716f6d5e7549418fdd8d54d7b722ce. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:37.640822 18620 raft_consensus.cc:2955] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } }
I20260709 12:20:37.642693 17760 raft_consensus.cc:2955] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } }
I20260709 12:20:37.668713 17834 raft_consensus.cc:2955] T 69716f6d5e7549418fdd8d54d7b722ce P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } }
I20260709 12:20:37.671744 17835 consensus_queue.cc:237] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:20:37.683182 17761 raft_consensus.cc:1275] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260709 12:20:37.683226 17803 consensus_peers.cc:597] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b -> Peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Couldn't send request to peer 334a4e1f59944544b3b3ff36616b57da. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 19dad6ab92894d708ec8eb7e8ac28185. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:37.683823 17586 catalog_manager.cc:5697] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb reported cstate change: config changed from index -1 to 2, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New cstate: current_term: 1 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } }
I20260709 12:20:37.684980 18649 consensus_queue.cc:1048] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:20:37.690145 17685 raft_consensus.cc:1275] T 19dad6ab92894d708ec8eb7e8ac28185 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:20:37.693929 18664 consensus_queue.cc:1048] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:20:37.709313 18664 raft_consensus.cc:2955] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } }
I20260709 12:20:37.711023 17685 raft_consensus.cc:2955] T 19dad6ab92894d708ec8eb7e8ac28185 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } }
I20260709 12:20:37.714298 17760 raft_consensus.cc:2955] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } }
I20260709 12:20:37.727969 17685 consensus_queue.cc:237] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:20:37.733608 17586 catalog_manager.cc:5697] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b reported cstate change: config changed from index -1 to 2, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New cstate: current_term: 1 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20260709 12:20:37.753919 17654 consensus_peers.cc:597] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 -> Peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Couldn't send request to peer 334a4e1f59944544b3b3ff36616b57da. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bd3425997bcf4fe7abc0399fd06e54d9. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:37.758113 17760 raft_consensus.cc:1275] T bd3425997bcf4fe7abc0399fd06e54d9 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:20:37.758113 17835 raft_consensus.cc:1275] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:20:37.759554 18665 consensus_queue.cc:1048] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:20:37.762918 18620 consensus_queue.cc:1048] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.004s
I20260709 12:20:37.779609 18665 raft_consensus.cc:2955] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } }
I20260709 12:20:37.788880 17760 raft_consensus.cc:2955] T bd3425997bcf4fe7abc0399fd06e54d9 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } }
I20260709 12:20:37.797974 17837 raft_consensus.cc:2955] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } }
I20260709 12:20:37.806718 17583 catalog_manager.cc:5697] T bd3425997bcf4fe7abc0399fd06e54d9 P e2909cff1da44ad387f96a8ae3aa5afb reported cstate change: config changed from index -1 to 2, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New cstate: current_term: 1 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } }
I20260709 12:20:38.068787 18684 ts_tablet_manager.cc:933] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da: Initiating tablet copy from peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:38.070581 18684 tablet_copy_client.cc:323] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Beginning tablet copy session from remote peer at address 127.16.29.67:36837
I20260709 12:20:38.072429 17848 tablet_copy_service.cc:140] P e063541542a94969898868bf6eb5ce1b: Received BeginTabletCopySession request for tablet b5462347eb74482b8cd11beb1d39b8a9 from peer 334a4e1f59944544b3b3ff36616b57da ({username='slave'} at 127.0.0.1:37994)
I20260709 12:20:38.073019 17848 tablet_copy_service.cc:161] P e063541542a94969898868bf6eb5ce1b: Beginning new tablet copy session on tablet b5462347eb74482b8cd11beb1d39b8a9 from peer 334a4e1f59944544b3b3ff36616b57da at {username='slave'} at 127.0.0.1:37994: session id = 334a4e1f59944544b3b3ff36616b57da-b5462347eb74482b8cd11beb1d39b8a9
I20260709 12:20:38.081205 17848 tablet_copy_source_session.cc:215] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:20:38.085101 18684 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b5462347eb74482b8cd11beb1d39b8a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:38.101516 18684 tablet_copy_client.cc:806] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Starting download of 0 data blocks...
I20260709 12:20:38.102444 18684 tablet_copy_client.cc:670] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Starting download of 1 WAL segments...
I20260709 12:20:38.105597 18686 ts_tablet_manager.cc:933] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da: Initiating tablet copy from peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071)
I20260709 12:20:38.106734 18684 tablet_copy_client.cc:538] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:20:38.111754 18686 tablet_copy_client.cc:323] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Beginning tablet copy session from remote peer at address 127.16.29.65:33071
I20260709 12:20:38.114382 17699 tablet_copy_service.cc:140] P 0da169ff82674dc1947055ce43de68a3: Received BeginTabletCopySession request for tablet 69716f6d5e7549418fdd8d54d7b722ce from peer 334a4e1f59944544b3b3ff36616b57da ({username='slave'} at 127.0.0.1:44898)
I20260709 12:20:38.114916 17699 tablet_copy_service.cc:161] P 0da169ff82674dc1947055ce43de68a3: Beginning new tablet copy session on tablet 69716f6d5e7549418fdd8d54d7b722ce from peer 334a4e1f59944544b3b3ff36616b57da at {username='slave'} at 127.0.0.1:44898: session id = 334a4e1f59944544b3b3ff36616b57da-69716f6d5e7549418fdd8d54d7b722ce
I20260709 12:20:38.117401 18684 tablet_bootstrap.cc:492] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da: Bootstrap starting.
I20260709 12:20:38.123029 17699 tablet_copy_source_session.cc:215] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:20:38.126677 18686 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69716f6d5e7549418fdd8d54d7b722ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:38.147797 18686 tablet_copy_client.cc:806] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Starting download of 0 data blocks...
I20260709 12:20:38.148572 18686 tablet_copy_client.cc:670] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Starting download of 1 WAL segments...
I20260709 12:20:38.155131 18688 ts_tablet_manager.cc:933] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da: Initiating tablet copy from peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:38.164280 18688 tablet_copy_client.cc:323] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Beginning tablet copy session from remote peer at address 127.16.29.67:36837
I20260709 12:20:38.166553 17848 tablet_copy_service.cc:140] P e063541542a94969898868bf6eb5ce1b: Received BeginTabletCopySession request for tablet 19dad6ab92894d708ec8eb7e8ac28185 from peer 334a4e1f59944544b3b3ff36616b57da ({username='slave'} at 127.0.0.1:37994)
I20260709 12:20:38.167189 17848 tablet_copy_service.cc:161] P e063541542a94969898868bf6eb5ce1b: Beginning new tablet copy session on tablet 19dad6ab92894d708ec8eb7e8ac28185 from peer 334a4e1f59944544b3b3ff36616b57da at {username='slave'} at 127.0.0.1:37994: session id = 334a4e1f59944544b3b3ff36616b57da-19dad6ab92894d708ec8eb7e8ac28185
I20260709 12:20:38.176193 18684 tablet_bootstrap.cc:492] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:20:38.178220 18686 tablet_copy_client.cc:538] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:20:38.178854 18684 tablet_bootstrap.cc:492] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da: Bootstrap complete.
I20260709 12:20:38.179571 18684 ts_tablet_manager.cc:1403] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da: Time spent bootstrapping tablet: real 0.062s	user 0.029s	sys 0.007s
I20260709 12:20:38.182564 18684 raft_consensus.cc:359] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:20:38.183418 18684 raft_consensus.cc:740] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Initialized, Role: LEARNER
I20260709 12:20:38.184111 18684 consensus_queue.cc:260] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:20:38.177279 17848 tablet_copy_source_session.cc:215] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:20:38.190857 18686 tablet_bootstrap.cc:492] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da: Bootstrap starting.
I20260709 12:20:38.191825 18688 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 19dad6ab92894d708ec8eb7e8ac28185. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:38.203683 18691 ts_tablet_manager.cc:933] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da: Initiating tablet copy from peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071)
I20260709 12:20:38.203788 18684 ts_tablet_manager.cc:1434] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da: Time spent starting tablet: real 0.024s	user 0.009s	sys 0.004s
I20260709 12:20:38.210741 17848 tablet_copy_service.cc:342] P e063541542a94969898868bf6eb5ce1b: Request end of tablet copy session 334a4e1f59944544b3b3ff36616b57da-b5462347eb74482b8cd11beb1d39b8a9 received from {username='slave'} at 127.0.0.1:37994
I20260709 12:20:38.211252 17848 tablet_copy_service.cc:434] P e063541542a94969898868bf6eb5ce1b: ending tablet copy session 334a4e1f59944544b3b3ff36616b57da-b5462347eb74482b8cd11beb1d39b8a9 on tablet b5462347eb74482b8cd11beb1d39b8a9 with peer 334a4e1f59944544b3b3ff36616b57da
I20260709 12:20:38.216112 18688 tablet_copy_client.cc:806] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Starting download of 0 data blocks...
I20260709 12:20:38.217025 18688 tablet_copy_client.cc:670] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Starting download of 1 WAL segments...
I20260709 12:20:38.222458 18688 tablet_copy_client.cc:538] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:20:38.230243 18691 tablet_copy_client.cc:323] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Beginning tablet copy session from remote peer at address 127.16.29.65:33071
I20260709 12:20:38.235999 18688 tablet_bootstrap.cc:492] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da: Bootstrap starting.
I20260709 12:20:38.238309 17699 tablet_copy_service.cc:140] P 0da169ff82674dc1947055ce43de68a3: Received BeginTabletCopySession request for tablet bd3425997bcf4fe7abc0399fd06e54d9 from peer 334a4e1f59944544b3b3ff36616b57da ({username='slave'} at 127.0.0.1:44898)
I20260709 12:20:38.239091 17699 tablet_copy_service.cc:161] P 0da169ff82674dc1947055ce43de68a3: Beginning new tablet copy session on tablet bd3425997bcf4fe7abc0399fd06e54d9 from peer 334a4e1f59944544b3b3ff36616b57da at {username='slave'} at 127.0.0.1:44898: session id = 334a4e1f59944544b3b3ff36616b57da-bd3425997bcf4fe7abc0399fd06e54d9
I20260709 12:20:38.240005 18686 tablet_bootstrap.cc:492] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:20:38.241410 18686 tablet_bootstrap.cc:492] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da: Bootstrap complete.
I20260709 12:20:38.242092 18686 ts_tablet_manager.cc:1403] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da: Time spent bootstrapping tablet: real 0.051s	user 0.030s	sys 0.008s
I20260709 12:20:38.245257 18686 raft_consensus.cc:359] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:20:38.246160 18686 raft_consensus.cc:740] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Initialized, Role: LEARNER
I20260709 12:20:38.246857 18686 consensus_queue.cc:260] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:20:38.251036 17699 tablet_copy_source_session.cc:215] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:20:38.255985 18691 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd3425997bcf4fe7abc0399fd06e54d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:38.274551 18686 ts_tablet_manager.cc:1434] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da: Time spent starting tablet: real 0.032s	user 0.005s	sys 0.004s
I20260709 12:20:38.291428 17699 tablet_copy_service.cc:342] P 0da169ff82674dc1947055ce43de68a3: Request end of tablet copy session 334a4e1f59944544b3b3ff36616b57da-69716f6d5e7549418fdd8d54d7b722ce received from {username='slave'} at 127.0.0.1:44898
I20260709 12:20:38.292212 17699 tablet_copy_service.cc:434] P 0da169ff82674dc1947055ce43de68a3: ending tablet copy session 334a4e1f59944544b3b3ff36616b57da-69716f6d5e7549418fdd8d54d7b722ce on tablet 69716f6d5e7549418fdd8d54d7b722ce with peer 334a4e1f59944544b3b3ff36616b57da
I20260709 12:20:38.306032 18691 tablet_copy_client.cc:806] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Starting download of 0 data blocks...
I20260709 12:20:38.306742 18691 tablet_copy_client.cc:670] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Starting download of 1 WAL segments...
I20260709 12:20:38.307757 18688 tablet_bootstrap.cc:492] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:20:38.308763 18688 tablet_bootstrap.cc:492] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da: Bootstrap complete.
I20260709 12:20:38.309580 18688 ts_tablet_manager.cc:1403] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da: Time spent bootstrapping tablet: real 0.074s	user 0.036s	sys 0.024s
I20260709 12:20:38.311563 18691 tablet_copy_client.cc:538] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:20:38.314021 18688 raft_consensus.cc:359] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:20:38.314939 18688 raft_consensus.cc:740] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Initialized, Role: LEARNER
I20260709 12:20:38.315645 18688 consensus_queue.cc:260] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:20:38.320709 18688 ts_tablet_manager.cc:1434] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da: Time spent starting tablet: real 0.011s	user 0.007s	sys 0.000s
I20260709 12:20:38.325578 18691 tablet_bootstrap.cc:492] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da: Bootstrap starting.
I20260709 12:20:38.326771 17848 tablet_copy_service.cc:342] P e063541542a94969898868bf6eb5ce1b: Request end of tablet copy session 334a4e1f59944544b3b3ff36616b57da-19dad6ab92894d708ec8eb7e8ac28185 received from {username='slave'} at 127.0.0.1:37994
I20260709 12:20:38.327198 17848 tablet_copy_service.cc:434] P e063541542a94969898868bf6eb5ce1b: ending tablet copy session 334a4e1f59944544b3b3ff36616b57da-19dad6ab92894d708ec8eb7e8ac28185 on tablet 19dad6ab92894d708ec8eb7e8ac28185 with peer 334a4e1f59944544b3b3ff36616b57da
I20260709 12:20:38.362092 18691 tablet_bootstrap.cc:492] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:20:38.363336 18691 tablet_bootstrap.cc:492] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da: Bootstrap complete.
I20260709 12:20:38.364132 18691 ts_tablet_manager.cc:1403] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da: Time spent bootstrapping tablet: real 0.039s	user 0.016s	sys 0.012s
I20260709 12:20:38.367369 18691 raft_consensus.cc:359] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:20:38.368307 18691 raft_consensus.cc:740] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Initialized, Role: LEARNER
I20260709 12:20:38.368973 18691 consensus_queue.cc:260] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:20:38.372087 18691 ts_tablet_manager.cc:1434] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da: Time spent starting tablet: real 0.007s	user 0.007s	sys 0.000s
I20260709 12:20:38.374634 17699 tablet_copy_service.cc:342] P 0da169ff82674dc1947055ce43de68a3: Request end of tablet copy session 334a4e1f59944544b3b3ff36616b57da-bd3425997bcf4fe7abc0399fd06e54d9 received from {username='slave'} at 127.0.0.1:44898
I20260709 12:20:38.375082 17699 tablet_copy_service.cc:434] P 0da169ff82674dc1947055ce43de68a3: ending tablet copy session 334a4e1f59944544b3b3ff36616b57da-bd3425997bcf4fe7abc0399fd06e54d9 on tablet bd3425997bcf4fe7abc0399fd06e54d9 with peer 334a4e1f59944544b3b3ff36616b57da
I20260709 12:20:38.545938 18374 raft_consensus.cc:1217] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:20:38.635963 18374 raft_consensus.cc:1217] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
W20260709 12:20:38.738627 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 131176us
I20260709 12:20:38.773955 18374 raft_consensus.cc:1217] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:20:38.779403 18374 raft_consensus.cc:1217] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:20:39.041150 18620 raft_consensus.cc:1064] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3: attempting to promote NON_VOTER 334a4e1f59944544b3b3ff36616b57da to VOTER
I20260709 12:20:39.043681 18620 consensus_queue.cc:237] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:20:39.057211 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 250257us
I20260709 12:20:39.064602 17760 raft_consensus.cc:1275] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:20:39.069100 18368 raft_consensus.cc:1275] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 1 LEARNER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:20:39.077484 18697 consensus_queue.cc:1048] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:20:39.096508 17836 raft_consensus.cc:1275] T 69716f6d5e7549418fdd8d54d7b722ce P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:20:39.129828 18699 consensus_queue.cc:1048] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:20:39.176518 18664 raft_consensus.cc:1064] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b: attempting to promote NON_VOTER 334a4e1f59944544b3b3ff36616b57da to VOTER
I20260709 12:20:39.179121 18664 consensus_queue.cc:237] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:20:39.180248 18697 consensus_queue.cc:1048] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:20:39.184379 18665 raft_consensus.cc:2955] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:39.184314 18368 raft_consensus.cc:2955] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:39.195503 18699 raft_consensus.cc:1064] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3: attempting to promote NON_VOTER 334a4e1f59944544b3b3ff36616b57da to VOTER
I20260709 12:20:39.199069 18374 raft_consensus.cc:1275] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 1 LEARNER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:20:39.201303 18703 consensus_queue.cc:1048] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:20:39.199038 18699 consensus_queue.cc:237] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:20:39.203601 17763 raft_consensus.cc:1275] T b5462347eb74482b8cd11beb1d39b8a9 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:20:39.204560 17686 raft_consensus.cc:1275] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:20:39.205895 18662 consensus_queue.cc:1048] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:20:39.214776 17583 catalog_manager.cc:5697] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da reported cstate change: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:39.216125 17836 raft_consensus.cc:2955] T 69716f6d5e7549418fdd8d54d7b722ce P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:39.234668 18664 consensus_queue.cc:1048] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:20:39.252079 18703 raft_consensus.cc:1064] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b: attempting to promote NON_VOTER 334a4e1f59944544b3b3ff36616b57da to VOTER
I20260709 12:20:39.255467 18703 consensus_queue.cc:237] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:20:39.277838 17760 raft_consensus.cc:2955] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:39.305716 18368 raft_consensus.cc:1275] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 1 LEARNER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:20:39.310474 17686 raft_consensus.cc:1275] T 19dad6ab92894d708ec8eb7e8ac28185 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:20:39.318153 17763 raft_consensus.cc:1275] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:20:39.323128 18662 consensus_queue.cc:1048] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:20:39.357714 17764 raft_consensus.cc:1275] T bd3425997bcf4fe7abc0399fd06e54d9 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:20:39.360549 18662 raft_consensus.cc:2955] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:39.364951 18664 consensus_queue.cc:1048] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:20:39.366112 18665 consensus_queue.cc:1048] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:20:39.370242 18367 raft_consensus.cc:1275] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 1 LEARNER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:20:39.371989 17838 raft_consensus.cc:1275] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:20:39.377295 18705 consensus_queue.cc:1048] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20260709 12:20:39.386070 18712 log.cc:927] Time spent T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.052s	user 0.001s	sys 0.000s
I20260709 12:20:39.387823 18665 consensus_queue.cc:1048] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.015s
I20260709 12:20:39.415936 18368 raft_consensus.cc:2955] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:39.430989 18699 consensus_queue.cc:1048] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:20:39.433374 17685 raft_consensus.cc:2955] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:39.460927 17587 catalog_manager.cc:5697] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da reported cstate change: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:39.485703 17685 consensus_queue.cc:237] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:20:39.494236 18368 raft_consensus.cc:1275] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:20:39.497897 18665 consensus_queue.cc:1048] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:20:39.504830 17838 consensus_queue.cc:237] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:20:39.506981 18699 raft_consensus.cc:2955] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER e063541542a94969898868bf6eb5ce1b (127.16.29.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:39.511098 18368 raft_consensus.cc:1275] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:20:39.513477 18662 consensus_queue.cc:1048] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:20:39.521359 17688 raft_consensus.cc:1275] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:20:39.523469 18705 consensus_queue.cc:1048] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:20:39.524247 17762 raft_consensus.cc:1275] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:20:39.525374 18371 raft_consensus.cc:2955] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:39.526547 18703 raft_consensus.cc:2955] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:39.530608 18367 raft_consensus.cc:2955] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER e063541542a94969898868bf6eb5ce1b (127.16.29.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:39.533450 18697 consensus_queue.cc:1048] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:20:39.534649 17763 raft_consensus.cc:2955] T b5462347eb74482b8cd11beb1d39b8a9 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:39.536461 18698 raft_consensus.cc:2955] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:39.540675 17571 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet b5462347eb74482b8cd11beb1d39b8a9 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:20:39.550969 17572 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 69716f6d5e7549418fdd8d54d7b722ce with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:20:39.551189 17685 raft_consensus.cc:2955] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:39.556247 18703 raft_consensus.cc:2955] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:39.558569 18371 raft_consensus.cc:2955] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:39.570102 17583 catalog_manager.cc:5697] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da reported cstate change: config changed from index 3 to 4, VOTER e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66) evicted. New cstate: current_term: 1 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:39.571354 17688 raft_consensus.cc:2955] T 19dad6ab92894d708ec8eb7e8ac28185 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:39.578234 17586 catalog_manager.cc:5697] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 reported cstate change: config changed from index 3 to 4, VOTER e063541542a94969898868bf6eb5ce1b (127.16.29.67) evicted. New cstate: current_term: 1 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:20:39.589500 17757 raft_consensus.cc:2955] T bd3425997bcf4fe7abc0399fd06e54d9 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:39.599045 17763 raft_consensus.cc:2955] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER e063541542a94969898868bf6eb5ce1b (127.16.29.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:39.600936 17821 tablet_service.cc:1558] Processing DeleteTablet for tablet 69716f6d5e7549418fdd8d54d7b722ce with delete_type TABLET_DATA_TOMBSTONED (TS e063541542a94969898868bf6eb5ce1b not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:35032
I20260709 12:20:39.609494 17760 raft_consensus.cc:2955] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:39.618542 17753 tablet_service.cc:1558] Processing DeleteTablet for tablet b5462347eb74482b8cd11beb1d39b8a9 with delete_type TABLET_DATA_TOMBSTONED (TS e2909cff1da44ad387f96a8ae3aa5afb not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:38774
I20260709 12:20:39.614409 18371 raft_consensus.cc:2955] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:39.631397 17583 catalog_manager.cc:5697] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da reported cstate change: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:39.657809 17836 raft_consensus.cc:2955] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:39.661660 18730 tablet_replica.cc:333] T b5462347eb74482b8cd11beb1d39b8a9 P e2909cff1da44ad387f96a8ae3aa5afb: stopping tablet replica
I20260709 12:20:39.675858 17586 catalog_manager.cc:5697] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da reported cstate change: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:39.686179 17836 consensus_queue.cc:237] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:20:39.693312 18730 raft_consensus.cc:2243] T b5462347eb74482b8cd11beb1d39b8a9 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:39.694063 18730 raft_consensus.cc:2272] T b5462347eb74482b8cd11beb1d39b8a9 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:39.700570 18730 ts_tablet_manager.cc:1916] T b5462347eb74482b8cd11beb1d39b8a9 P e2909cff1da44ad387f96a8ae3aa5afb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:20:39.703426 17757 raft_consensus.cc:1275] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:20:39.704883 18662 consensus_queue.cc:1048] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:20:39.708175 18371 raft_consensus.cc:1275] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:20:39.709956 18662 consensus_queue.cc:1048] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:20:39.711598 18729 tablet_replica.cc:333] T 69716f6d5e7549418fdd8d54d7b722ce P e063541542a94969898868bf6eb5ce1b: stopping tablet replica
I20260709 12:20:39.712615 18729 raft_consensus.cc:2243] T 69716f6d5e7549418fdd8d54d7b722ce P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:39.713276 18729 raft_consensus.cc:2272] T 69716f6d5e7549418fdd8d54d7b722ce P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:39.716399 17685 consensus_queue.cc:237] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:20:39.717118 18664 raft_consensus.cc:2955] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:39.723767 18371 raft_consensus.cc:1275] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:20:39.725361 18620 consensus_queue.cc:1048] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:20:39.729624 17760 raft_consensus.cc:2955] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:39.736613 18665 raft_consensus.cc:2955] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:39.741704 18367 raft_consensus.cc:2955] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:39.742884 18729 ts_tablet_manager.cc:1916] T 69716f6d5e7549418fdd8d54d7b722ce P e063541542a94969898868bf6eb5ce1b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:20:39.744378 18730 ts_tablet_manager.cc:1929] T b5462347eb74482b8cd11beb1d39b8a9 P e2909cff1da44ad387f96a8ae3aa5afb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:20:39.744853 18730 log.cc:1199] T b5462347eb74482b8cd11beb1d39b8a9 P e2909cff1da44ad387f96a8ae3aa5afb: Deleting WAL directory at /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599551912677-16501-0/minicluster-data/ts-1-root/wals/b5462347eb74482b8cd11beb1d39b8a9
I20260709 12:20:39.749287 17572 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet bd3425997bcf4fe7abc0399fd06e54d9 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:20:39.749933 17571 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 19dad6ab92894d708ec8eb7e8ac28185 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:20:39.758468 17571 catalog_manager.cc:5028] TS e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): tablet b5462347eb74482b8cd11beb1d39b8a9 (table test-workload [id=ddc5b74250df4c65909b73b3742d615f]) successfully deleted
I20260709 12:20:39.758601 18374 raft_consensus.cc:2955] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:39.764503 17583 catalog_manager.cc:5697] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 reported cstate change: config changed from index 3 to 4, VOTER e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66) evicted. New cstate: current_term: 1 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:20:39.766563 17586 catalog_manager.cc:5697] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb reported cstate change: config changed from index 3 to 4, VOTER 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) evicted. New cstate: current_term: 1 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:39.781702 18729 ts_tablet_manager.cc:1929] T 69716f6d5e7549418fdd8d54d7b722ce P e063541542a94969898868bf6eb5ce1b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:20:39.781769 17838 raft_consensus.cc:1275] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:20:39.782518 17753 tablet_service.cc:1558] Processing DeleteTablet for tablet bd3425997bcf4fe7abc0399fd06e54d9 with delete_type TABLET_DATA_TOMBSTONED (TS e2909cff1da44ad387f96a8ae3aa5afb not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:38774
I20260709 12:20:39.783432 18730 tablet_replica.cc:333] T bd3425997bcf4fe7abc0399fd06e54d9 P e2909cff1da44ad387f96a8ae3aa5afb: stopping tablet replica
I20260709 12:20:39.783514 17676 tablet_service.cc:1558] Processing DeleteTablet for tablet 19dad6ab92894d708ec8eb7e8ac28185 with delete_type TABLET_DATA_TOMBSTONED (TS 0da169ff82674dc1947055ce43de68a3 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:37384
I20260709 12:20:39.784659 18730 raft_consensus.cc:2243] T bd3425997bcf4fe7abc0399fd06e54d9 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:39.785346 18730 raft_consensus.cc:2272] T bd3425997bcf4fe7abc0399fd06e54d9 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:39.786279 18729 log.cc:1199] T 69716f6d5e7549418fdd8d54d7b722ce P e063541542a94969898868bf6eb5ce1b: Deleting WAL directory at /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599551912677-16501-0/minicluster-data/ts-2-root/wals/69716f6d5e7549418fdd8d54d7b722ce
I20260709 12:20:39.787830 18699 consensus_queue.cc:1048] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:20:39.788160 17571 catalog_manager.cc:5028] TS e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837): tablet 69716f6d5e7549418fdd8d54d7b722ce (table test-workload [id=ddc5b74250df4c65909b73b3742d615f]) successfully deleted
I20260709 12:20:39.789225 18730 ts_tablet_manager.cc:1916] T bd3425997bcf4fe7abc0399fd06e54d9 P e2909cff1da44ad387f96a8ae3aa5afb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:20:39.793900 17838 raft_consensus.cc:2955] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:39.810586 18730 ts_tablet_manager.cc:1929] T bd3425997bcf4fe7abc0399fd06e54d9 P e2909cff1da44ad387f96a8ae3aa5afb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:20:39.811129 18730 log.cc:1199] T bd3425997bcf4fe7abc0399fd06e54d9 P e2909cff1da44ad387f96a8ae3aa5afb: Deleting WAL directory at /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599551912677-16501-0/minicluster-data/ts-1-root/wals/bd3425997bcf4fe7abc0399fd06e54d9
I20260709 12:20:39.814250 17571 catalog_manager.cc:5028] TS e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): tablet bd3425997bcf4fe7abc0399fd06e54d9 (table test-workload [id=ddc5b74250df4c65909b73b3742d615f]) successfully deleted
I20260709 12:20:39.820259 18732 tablet_replica.cc:333] T 19dad6ab92894d708ec8eb7e8ac28185 P 0da169ff82674dc1947055ce43de68a3: stopping tablet replica
I20260709 12:20:39.821501 18732 raft_consensus.cc:2243] T 19dad6ab92894d708ec8eb7e8ac28185 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:39.822372 18732 raft_consensus.cc:2272] T 19dad6ab92894d708ec8eb7e8ac28185 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:39.825197 18732 ts_tablet_manager.cc:1916] T 19dad6ab92894d708ec8eb7e8ac28185 P 0da169ff82674dc1947055ce43de68a3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:20:39.837101 18732 ts_tablet_manager.cc:1929] T 19dad6ab92894d708ec8eb7e8ac28185 P 0da169ff82674dc1947055ce43de68a3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:20:39.837416 18732 log.cc:1199] T 19dad6ab92894d708ec8eb7e8ac28185 P 0da169ff82674dc1947055ce43de68a3: Deleting WAL directory at /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599551912677-16501-0/minicluster-data/ts-0-root/wals/19dad6ab92894d708ec8eb7e8ac28185
I20260709 12:20:39.838828 17572 catalog_manager.cc:5028] TS 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): tablet 19dad6ab92894d708ec8eb7e8ac28185 (table test-workload [id=ddc5b74250df4c65909b73b3742d615f]) successfully deleted
W20260709 12:20:39.861617 17710 debug-util.cc:398] Leaking SignalData structure 0x7b08000bd060 after lost signal to thread 16504
W20260709 12:20:39.862663 17710 debug-util.cc:398] Leaking SignalData structure 0x7b08000bfa80 after lost signal to thread 17621
W20260709 12:20:39.863780 17710 debug-util.cc:398] Leaking SignalData structure 0x7b0800044200 after lost signal to thread 17713
W20260709 12:20:39.865159 17710 debug-util.cc:398] Leaking SignalData structure 0x7b080032aa60 after lost signal to thread 17788
W20260709 12:20:39.866757 17710 debug-util.cc:398] Leaking SignalData structure 0x7b080025cf40 after lost signal to thread 17862
W20260709 12:20:39.868265 17710 debug-util.cc:398] Leaking SignalData structure 0x7b0800147700 after lost signal to thread 18409
I20260709 12:20:40.893555 17762 consensus_queue.cc:237] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
W20260709 12:20:40.901677 17729 consensus_peers.cc:597] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb -> Peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Couldn't send request to peer 334a4e1f59944544b3b3ff36616b57da. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0540d79e2ec2406588b295145aa6d54d. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:40.903868 17836 raft_consensus.cc:1275] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:20:40.904836 17688 raft_consensus.cc:1275] T 0540d79e2ec2406588b295145aa6d54d P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:20:40.907230 18667 consensus_queue.cc:1048] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:20:40.909600 18700 consensus_queue.cc:1048] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:20:40.917089 18734 raft_consensus.cc:2955] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } }
I20260709 12:20:40.923061 17836 raft_consensus.cc:2955] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } }
I20260709 12:20:40.928505 17685 raft_consensus.cc:2955] T 0540d79e2ec2406588b295145aa6d54d P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } }
I20260709 12:20:40.975845 17762 consensus_queue.cc:237] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:20:40.991569 17836 raft_consensus.cc:1275] T ac984176358d453a8800cc9e0e304061 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20260709 12:20:40.994444 17729 consensus_peers.cc:597] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb -> Peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Couldn't send request to peer 334a4e1f59944544b3b3ff36616b57da. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ac984176358d453a8800cc9e0e304061. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:40.991577 17685 raft_consensus.cc:1275] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:20:40.999398 18734 consensus_queue.cc:1048] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:20:40.997422 17587 catalog_manager.cc:5697] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb reported cstate change: config changed from index -1 to 2, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New cstate: current_term: 1 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:41.001762 18737 consensus_queue.cc:1048] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:20:41.012177 18734 raft_consensus.cc:2955] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 LEADER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } }
I20260709 12:20:41.014093 17838 raft_consensus.cc:2955] T ac984176358d453a8800cc9e0e304061 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } }
I20260709 12:20:41.033931 17688 consensus_queue.cc:237] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:20:41.033456 17685 raft_consensus.cc:2955] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } }
W20260709 12:20:41.092772 17654 consensus_peers.cc:597] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 -> Peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Couldn't send request to peer 334a4e1f59944544b3b3ff36616b57da. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3219dcbfd86340ecbde507517060ff81. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:41.097487 17838 raft_consensus.cc:1275] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:20:41.102905 17762 raft_consensus.cc:1275] T 3219dcbfd86340ecbde507517060ff81 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:20:41.106529 17587 catalog_manager.cc:5697] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb reported cstate change: config changed from index -1 to 2, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New cstate: current_term: 2 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:41.110333 18741 consensus_queue.cc:1048] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:20:41.123451 18699 consensus_queue.cc:1048] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.002s
I20260709 12:20:41.148504 18699 raft_consensus.cc:2955] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 2 LEADER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } }
I20260709 12:20:41.153185 17838 raft_consensus.cc:2955] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } }
I20260709 12:20:41.154408 17760 raft_consensus.cc:2955] T 3219dcbfd86340ecbde507517060ff81 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } }
I20260709 12:20:41.174613 17587 catalog_manager.cc:5697] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b reported cstate change: config changed from index -1 to 2, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New cstate: current_term: 2 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } }
I20260709 12:20:41.187343 17835 consensus_queue.cc:237] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
W20260709 12:20:41.194585 17803 consensus_peers.cc:597] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b -> Peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Couldn't send request to peer 334a4e1f59944544b3b3ff36616b57da. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1a4040191930435786e2eeea338bb3d3. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:41.194433 17760 raft_consensus.cc:1275] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:20:41.197145 18731 consensus_queue.cc:1048] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:20:41.204797 17688 raft_consensus.cc:1275] T 1a4040191930435786e2eeea338bb3d3 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:20:41.207322 18664 consensus_queue.cc:1048] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:20:41.216121 18561 raft_consensus.cc:2955] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } }
I20260709 12:20:41.218606 17760 raft_consensus.cc:2955] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } }
I20260709 12:20:41.247051 17688 raft_consensus.cc:2955] T 1a4040191930435786e2eeea338bb3d3 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } }
I20260709 12:20:41.256507 17585 catalog_manager.cc:5697] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b reported cstate change: config changed from index -1 to 2, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New cstate: current_term: 1 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:41.375478 18755 ts_tablet_manager.cc:933] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da: Initiating tablet copy from peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:20:41.377866 18755 tablet_copy_client.cc:323] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Beginning tablet copy session from remote peer at address 127.16.29.66:39651
I20260709 12:20:41.379724 17774 tablet_copy_service.cc:140] P e2909cff1da44ad387f96a8ae3aa5afb: Received BeginTabletCopySession request for tablet 0540d79e2ec2406588b295145aa6d54d from peer 334a4e1f59944544b3b3ff36616b57da ({username='slave'} at 127.0.0.1:36572)
I20260709 12:20:41.380321 17774 tablet_copy_service.cc:161] P e2909cff1da44ad387f96a8ae3aa5afb: Beginning new tablet copy session on tablet 0540d79e2ec2406588b295145aa6d54d from peer 334a4e1f59944544b3b3ff36616b57da at {username='slave'} at 127.0.0.1:36572: session id = 334a4e1f59944544b3b3ff36616b57da-0540d79e2ec2406588b295145aa6d54d
I20260709 12:20:41.388397 17774 tablet_copy_source_session.cc:215] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:20:41.400236 18755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0540d79e2ec2406588b295145aa6d54d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:41.418041 18755 tablet_copy_client.cc:806] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Starting download of 0 data blocks...
I20260709 12:20:41.418712 18755 tablet_copy_client.cc:670] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Starting download of 1 WAL segments...
I20260709 12:20:41.423144 18755 tablet_copy_client.cc:538] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:20:41.434587 18755 tablet_bootstrap.cc:492] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da: Bootstrap starting.
I20260709 12:20:41.465857 18755 tablet_bootstrap.cc:492] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:20:41.467033 18755 tablet_bootstrap.cc:492] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da: Bootstrap complete.
I20260709 12:20:41.467777 18755 ts_tablet_manager.cc:1403] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da: Time spent bootstrapping tablet: real 0.033s	user 0.015s	sys 0.013s
I20260709 12:20:41.470911 18755 raft_consensus.cc:359] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:20:41.471889 18755 raft_consensus.cc:740] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Initialized, Role: LEARNER
I20260709 12:20:41.472594 18755 consensus_queue.cc:260] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:20:41.486400 18755 ts_tablet_manager.cc:1434] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da: Time spent starting tablet: real 0.018s	user 0.007s	sys 0.007s
I20260709 12:20:41.503343 17774 tablet_copy_service.cc:342] P e2909cff1da44ad387f96a8ae3aa5afb: Request end of tablet copy session 334a4e1f59944544b3b3ff36616b57da-0540d79e2ec2406588b295145aa6d54d received from {username='slave'} at 127.0.0.1:36572
I20260709 12:20:41.503800 17774 tablet_copy_service.cc:434] P e2909cff1da44ad387f96a8ae3aa5afb: ending tablet copy session 334a4e1f59944544b3b3ff36616b57da-0540d79e2ec2406588b295145aa6d54d on tablet 0540d79e2ec2406588b295145aa6d54d with peer 334a4e1f59944544b3b3ff36616b57da
I20260709 12:20:41.513841 18755 ts_tablet_manager.cc:933] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da: Initiating tablet copy from peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:20:41.520169 18760 ts_tablet_manager.cc:933] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da: Initiating tablet copy from peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071)
I20260709 12:20:41.527401 18760 tablet_copy_client.cc:323] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Beginning tablet copy session from remote peer at address 127.16.29.65:33071
I20260709 12:20:41.533519 17699 tablet_copy_service.cc:140] P 0da169ff82674dc1947055ce43de68a3: Received BeginTabletCopySession request for tablet 3219dcbfd86340ecbde507517060ff81 from peer 334a4e1f59944544b3b3ff36616b57da ({username='slave'} at 127.0.0.1:44898)
I20260709 12:20:41.534269 17699 tablet_copy_service.cc:161] P 0da169ff82674dc1947055ce43de68a3: Beginning new tablet copy session on tablet 3219dcbfd86340ecbde507517060ff81 from peer 334a4e1f59944544b3b3ff36616b57da at {username='slave'} at 127.0.0.1:44898: session id = 334a4e1f59944544b3b3ff36616b57da-3219dcbfd86340ecbde507517060ff81
I20260709 12:20:41.556777 17699 tablet_copy_source_session.cc:215] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:20:41.565629 18760 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3219dcbfd86340ecbde507517060ff81. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:41.584335 18759 ts_tablet_manager.cc:933] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da: Initiating tablet copy from peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:41.585469 18755 tablet_copy_client.cc:323] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Beginning tablet copy session from remote peer at address 127.16.29.66:39651
I20260709 12:20:41.616092 17774 tablet_copy_service.cc:140] P e2909cff1da44ad387f96a8ae3aa5afb: Received BeginTabletCopySession request for tablet ac984176358d453a8800cc9e0e304061 from peer 334a4e1f59944544b3b3ff36616b57da ({username='slave'} at 127.0.0.1:36572)
I20260709 12:20:41.616710 17774 tablet_copy_service.cc:161] P e2909cff1da44ad387f96a8ae3aa5afb: Beginning new tablet copy session on tablet ac984176358d453a8800cc9e0e304061 from peer 334a4e1f59944544b3b3ff36616b57da at {username='slave'} at 127.0.0.1:36572: session id = 334a4e1f59944544b3b3ff36616b57da-ac984176358d453a8800cc9e0e304061
I20260709 12:20:41.675587 17774 tablet_copy_source_session.cc:215] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:20:41.679484 18755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac984176358d453a8800cc9e0e304061. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:41.681770 18759 tablet_copy_client.cc:323] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Beginning tablet copy session from remote peer at address 127.16.29.67:36837
I20260709 12:20:41.684304 17848 tablet_copy_service.cc:140] P e063541542a94969898868bf6eb5ce1b: Received BeginTabletCopySession request for tablet 1a4040191930435786e2eeea338bb3d3 from peer 334a4e1f59944544b3b3ff36616b57da ({username='slave'} at 127.0.0.1:37994)
I20260709 12:20:41.684826 17848 tablet_copy_service.cc:161] P e063541542a94969898868bf6eb5ce1b: Beginning new tablet copy session on tablet 1a4040191930435786e2eeea338bb3d3 from peer 334a4e1f59944544b3b3ff36616b57da at {username='slave'} at 127.0.0.1:37994: session id = 334a4e1f59944544b3b3ff36616b57da-1a4040191930435786e2eeea338bb3d3
I20260709 12:20:41.695166 17848 tablet_copy_source_session.cc:215] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:20:41.723265 18759 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a4040191930435786e2eeea338bb3d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:41.740150 18760 tablet_copy_client.cc:806] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Starting download of 0 data blocks...
I20260709 12:20:41.742427 18760 tablet_copy_client.cc:670] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Starting download of 1 WAL segments...
I20260709 12:20:41.750828 18760 tablet_copy_client.cc:538] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:20:41.753950 18755 tablet_copy_client.cc:806] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Starting download of 0 data blocks...
I20260709 12:20:41.754565 18755 tablet_copy_client.cc:670] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Starting download of 1 WAL segments...
I20260709 12:20:41.760640 18755 tablet_copy_client.cc:538] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:20:41.769323 18755 tablet_bootstrap.cc:492] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da: Bootstrap starting.
I20260709 12:20:41.771206 18759 tablet_copy_client.cc:806] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Starting download of 0 data blocks...
I20260709 12:20:41.771953 18759 tablet_copy_client.cc:670] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Starting download of 1 WAL segments...
I20260709 12:20:41.773010 18760 tablet_bootstrap.cc:492] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da: Bootstrap starting.
I20260709 12:20:41.776525 18759 tablet_copy_client.cc:538] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:20:41.799412 18759 tablet_bootstrap.cc:492] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da: Bootstrap starting.
I20260709 12:20:41.829663 18759 tablet_bootstrap.cc:492] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:20:41.830585 18759 tablet_bootstrap.cc:492] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da: Bootstrap complete.
I20260709 12:20:41.831276 18759 ts_tablet_manager.cc:1403] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da: Time spent bootstrapping tablet: real 0.032s	user 0.020s	sys 0.004s
I20260709 12:20:41.834447 18759 raft_consensus.cc:359] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:20:41.835439 18759 raft_consensus.cc:740] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Initialized, Role: LEARNER
I20260709 12:20:41.836161 18759 consensus_queue.cc:260] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:20:41.839354 18759 ts_tablet_manager.cc:1434] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da: Time spent starting tablet: real 0.008s	user 0.009s	sys 0.000s
I20260709 12:20:41.841192 18760 tablet_bootstrap.cc:492] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:20:41.842149 18760 tablet_bootstrap.cc:492] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da: Bootstrap complete.
I20260709 12:20:41.842878 18760 ts_tablet_manager.cc:1403] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da: Time spent bootstrapping tablet: real 0.070s	user 0.018s	sys 0.008s
I20260709 12:20:41.842794 17848 tablet_copy_service.cc:342] P e063541542a94969898868bf6eb5ce1b: Request end of tablet copy session 334a4e1f59944544b3b3ff36616b57da-1a4040191930435786e2eeea338bb3d3 received from {username='slave'} at 127.0.0.1:37994
I20260709 12:20:41.843521 17848 tablet_copy_service.cc:434] P e063541542a94969898868bf6eb5ce1b: ending tablet copy session 334a4e1f59944544b3b3ff36616b57da-1a4040191930435786e2eeea338bb3d3 on tablet 1a4040191930435786e2eeea338bb3d3 with peer 334a4e1f59944544b3b3ff36616b57da
I20260709 12:20:41.847111 18760 raft_consensus.cc:359] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:20:41.847993 18760 raft_consensus.cc:740] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Initialized, Role: LEARNER
I20260709 12:20:41.848748 18760 consensus_queue.cc:260] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:20:41.851075 18760 ts_tablet_manager.cc:1434] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da: Time spent starting tablet: real 0.007s	user 0.003s	sys 0.000s
I20260709 12:20:41.853075 17699 tablet_copy_service.cc:342] P 0da169ff82674dc1947055ce43de68a3: Request end of tablet copy session 334a4e1f59944544b3b3ff36616b57da-3219dcbfd86340ecbde507517060ff81 received from {username='slave'} at 127.0.0.1:44898
I20260709 12:20:41.853572 17699 tablet_copy_service.cc:434] P 0da169ff82674dc1947055ce43de68a3: ending tablet copy session 334a4e1f59944544b3b3ff36616b57da-3219dcbfd86340ecbde507517060ff81 on tablet 3219dcbfd86340ecbde507517060ff81 with peer 334a4e1f59944544b3b3ff36616b57da
I20260709 12:20:41.857860 18755 tablet_bootstrap.cc:492] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:20:41.858954 18755 tablet_bootstrap.cc:492] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da: Bootstrap complete.
I20260709 12:20:41.859699 18755 ts_tablet_manager.cc:1403] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da: Time spent bootstrapping tablet: real 0.091s	user 0.023s	sys 0.006s
I20260709 12:20:41.862792 18755 raft_consensus.cc:359] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:20:41.863694 18755 raft_consensus.cc:740] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Initialized, Role: LEARNER
I20260709 12:20:41.864353 18755 consensus_queue.cc:260] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:20:41.867111 18755 ts_tablet_manager.cc:1434] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da: Time spent starting tablet: real 0.007s	user 0.005s	sys 0.000s
I20260709 12:20:41.869179 17774 tablet_copy_service.cc:342] P e2909cff1da44ad387f96a8ae3aa5afb: Request end of tablet copy session 334a4e1f59944544b3b3ff36616b57da-ac984176358d453a8800cc9e0e304061 received from {username='slave'} at 127.0.0.1:36572
I20260709 12:20:41.869645 17774 tablet_copy_service.cc:434] P e2909cff1da44ad387f96a8ae3aa5afb: ending tablet copy session 334a4e1f59944544b3b3ff36616b57da-ac984176358d453a8800cc9e0e304061 on tablet ac984176358d453a8800cc9e0e304061 with peer 334a4e1f59944544b3b3ff36616b57da
I20260709 12:20:41.897042 18374 raft_consensus.cc:1217] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:20:41.948020 18374 raft_consensus.cc:1217] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 2 LEARNER]: Deduplicated request from leader. Original: 2.1->[2.2-2.2]   Dedup: 2.2->[]
I20260709 12:20:42.076100 18374 raft_consensus.cc:1217] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:20:42.100364 18374 raft_consensus.cc:1217] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 2 LEARNER]: Deduplicated request from leader. Original: 2.1->[2.2-2.2]   Dedup: 2.2->[]
I20260709 12:20:42.292397 18700 raft_consensus.cc:1064] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb: attempting to promote NON_VOTER 334a4e1f59944544b3b3ff36616b57da to VOTER
I20260709 12:20:42.295535 18700 consensus_queue.cc:237] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:20:42.318393 18374 raft_consensus.cc:1275] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 1 LEARNER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:20:42.319295 17835 raft_consensus.cc:1275] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:20:42.320468 17688 raft_consensus.cc:1275] T 0540d79e2ec2406588b295145aa6d54d P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:20:42.323909 18734 consensus_queue.cc:1048] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:20:42.326484 18734 consensus_queue.cc:1048] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:20:42.330332 18737 consensus_queue.cc:1048] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:20:42.392814 18734 raft_consensus.cc:2955] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:42.394481 18367 raft_consensus.cc:2955] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:42.409825 17585 catalog_manager.cc:5697] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da reported cstate change: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:42.430786 17686 raft_consensus.cc:2955] T 0540d79e2ec2406588b295145aa6d54d P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:42.437479 17835 raft_consensus.cc:2955] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:42.482131 18731 raft_consensus.cc:1064] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b: attempting to promote NON_VOTER 334a4e1f59944544b3b3ff36616b57da to VOTER
I20260709 12:20:42.484831 18731 consensus_queue.cc:237] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:20:42.511543 18374 raft_consensus.cc:1275] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 1 LEARNER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:20:42.513445 17686 raft_consensus.cc:1275] T 1a4040191930435786e2eeea338bb3d3 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:20:42.514717 17762 raft_consensus.cc:1275] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:20:42.516692 18753 consensus_queue.cc:1048] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:20:42.517697 18754 consensus_queue.cc:1048] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:20:42.526391 18731 consensus_queue.cc:1048] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:20:42.535166 18737 raft_consensus.cc:1064] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb: attempting to promote NON_VOTER 334a4e1f59944544b3b3ff36616b57da to VOTER
I20260709 12:20:42.549219 17757 consensus_queue.cc:237] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:20:42.568114 18664 raft_consensus.cc:2955] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:42.572523 18737 consensus_queue.cc:237] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 0, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:20:42.573483 17762 raft_consensus.cc:2955] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:42.575809 18665 raft_consensus.cc:1064] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3: attempting to promote NON_VOTER 334a4e1f59944544b3b3ff36616b57da to VOTER
I20260709 12:20:42.587320 18665 consensus_queue.cc:237] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 0, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:20:42.600605 17835 raft_consensus.cc:1275] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:20:42.603159 18768 consensus_queue.cc:1048] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:20:42.608006 18374 raft_consensus.cc:1275] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:20:42.623662 18371 raft_consensus.cc:2955] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:42.626446 18734 consensus_queue.cc:1048] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:20:42.633908 18734 raft_consensus.cc:2955] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:42.628865 18374 raft_consensus.cc:1275] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 2 LEARNER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260709 12:20:42.628504 17686 raft_consensus.cc:2955] T 1a4040191930435786e2eeea338bb3d3 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:42.630420 17836 raft_consensus.cc:1275] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260709 12:20:42.643541 18741 consensus_queue.cc:1048] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:20:42.645596 18665 consensus_queue.cc:1048] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:20:42.650375 17760 raft_consensus.cc:1275] T 3219dcbfd86340ecbde507517060ff81 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260709 12:20:42.652006 18665 consensus_queue.cc:1048] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:20:42.655666 18367 raft_consensus.cc:2955] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:42.667203 17583 catalog_manager.cc:5697] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b reported cstate change: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:20:42.674721 17838 raft_consensus.cc:2955] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:42.688719 17835 consensus_queue.cc:237] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:20:42.702517 17583 catalog_manager.cc:5697] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da reported cstate change: config changed from index 3 to 4, VOTER 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) evicted. New cstate: current_term: 1 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:42.714494 17758 raft_consensus.cc:1275] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:20:42.718719 18731 consensus_queue.cc:1048] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:20:42.720202 17571 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 0540d79e2ec2406588b295145aa6d54d with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:20:42.724294 18367 raft_consensus.cc:1275] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:20:42.728891 18664 consensus_queue.cc:1048] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:20:42.734128 17688 raft_consensus.cc:1275] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260709 12:20:42.735533 18367 raft_consensus.cc:1275] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 2 LEARNER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260709 12:20:42.737303 17836 raft_consensus.cc:1275] T ac984176358d453a8800cc9e0e304061 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260709 12:20:42.749462 17676 tablet_service.cc:1558] Processing DeleteTablet for tablet 0540d79e2ec2406588b295145aa6d54d with delete_type TABLET_DATA_TOMBSTONED (TS 0da169ff82674dc1947055ce43de68a3 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:37384
I20260709 12:20:42.738551 18737 consensus_queue.cc:1048] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:20:42.738365 18753 raft_consensus.cc:2955] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:42.742723 17758 raft_consensus.cc:2955] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:42.744405 18368 raft_consensus.cc:2955] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:42.781416 17587 catalog_manager.cc:5697] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da reported cstate change: config changed from index 3 to 4, VOTER 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) evicted. New cstate: current_term: 1 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:42.788548 18700 consensus_queue.cc:1048] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:20:42.797984 17571 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 1a4040191930435786e2eeea338bb3d3 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:20:42.803334 17674 tablet_service.cc:1558] Processing DeleteTablet for tablet 1a4040191930435786e2eeea338bb3d3 with delete_type TABLET_DATA_TOMBSTONED (TS 0da169ff82674dc1947055ce43de68a3 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:37384
I20260709 12:20:42.813724 18789 tablet_replica.cc:333] T 0540d79e2ec2406588b295145aa6d54d P 0da169ff82674dc1947055ce43de68a3: stopping tablet replica
I20260709 12:20:42.828084 18789 raft_consensus.cc:2243] T 0540d79e2ec2406588b295145aa6d54d P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:42.829898 18789 raft_consensus.cc:2272] T 0540d79e2ec2406588b295145aa6d54d P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:42.815393 18771 consensus_queue.cc:1048] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:20:42.840196 18789 ts_tablet_manager.cc:1916] T 0540d79e2ec2406588b295145aa6d54d P 0da169ff82674dc1947055ce43de68a3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20260709 12:20:42.849771 18788 log.cc:927] Time spent T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.085s	user 0.001s	sys 0.000s
I20260709 12:20:42.853101 18783 raft_consensus.cc:2955] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:42.861721 17762 raft_consensus.cc:2955] T 3219dcbfd86340ecbde507517060ff81 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:42.962306 17684 raft_consensus.cc:2955] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:42.977279 18737 raft_consensus.cc:2955] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:42.980149 17835 raft_consensus.cc:2955] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:42.951320 18371 raft_consensus.cc:2955] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:43.008620 18366 raft_consensus.cc:2955] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:43.041599 18789 ts_tablet_manager.cc:1929] T 0540d79e2ec2406588b295145aa6d54d P 0da169ff82674dc1947055ce43de68a3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:20:43.048022 18789 log.cc:1199] T 0540d79e2ec2406588b295145aa6d54d P 0da169ff82674dc1947055ce43de68a3: Deleting WAL directory at /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599551912677-16501-0/minicluster-data/ts-0-root/wals/0540d79e2ec2406588b295145aa6d54d
I20260709 12:20:43.061520 17583 catalog_manager.cc:5697] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 reported cstate change: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:43.064411 17572 catalog_manager.cc:5028] TS 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): tablet 0540d79e2ec2406588b295145aa6d54d (table test-workload [id=ddc5b74250df4c65909b73b3742d615f]) successfully deleted
I20260709 12:20:43.067028 18789 tablet_replica.cc:333] T 1a4040191930435786e2eeea338bb3d3 P 0da169ff82674dc1947055ce43de68a3: stopping tablet replica
I20260709 12:20:43.091356 17838 raft_consensus.cc:2955] T ac984176358d453a8800cc9e0e304061 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:43.136153 18789 raft_consensus.cc:2243] T 1a4040191930435786e2eeea338bb3d3 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:43.137332 18789 raft_consensus.cc:2272] T 1a4040191930435786e2eeea338bb3d3 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:43.142311 18789 ts_tablet_manager.cc:1916] T 1a4040191930435786e2eeea338bb3d3 P 0da169ff82674dc1947055ce43de68a3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:20:43.144646 17584 catalog_manager.cc:5697] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da reported cstate change: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:43.149226 17688 consensus_queue.cc:237] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:20:43.200537 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 75076.6us
I20260709 12:20:43.247519 17762 consensus_queue.cc:237] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:20:43.256805 17684 raft_consensus.cc:1275] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260709 12:20:43.258435 18789 ts_tablet_manager.cc:1929] T 1a4040191930435786e2eeea338bb3d3 P 0da169ff82674dc1947055ce43de68a3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:20:43.259006 18789 log.cc:1199] T 1a4040191930435786e2eeea338bb3d3 P 0da169ff82674dc1947055ce43de68a3: Deleting WAL directory at /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599551912677-16501-0/minicluster-data/ts-0-root/wals/1a4040191930435786e2eeea338bb3d3
I20260709 12:20:43.258728 18771 consensus_queue.cc:1048] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:20:43.266829 18371 raft_consensus.cc:1275] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260709 12:20:43.267597 18794 raft_consensus.cc:2955] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER e063541542a94969898868bf6eb5ce1b (127.16.29.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:43.271477 17572 catalog_manager.cc:5028] TS 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): tablet 1a4040191930435786e2eeea338bb3d3 (table test-workload [id=ddc5b74250df4c65909b73b3742d615f]) successfully deleted
I20260709 12:20:43.274029 18737 consensus_queue.cc:1048] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:20:43.278541 17688 raft_consensus.cc:2955] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER e063541542a94969898868bf6eb5ce1b (127.16.29.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:43.286103 18371 raft_consensus.cc:1275] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260709 12:20:43.297817 18782 consensus_queue.cc:1048] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:20:43.301275 17571 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet ac984176358d453a8800cc9e0e304061 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:20:43.310838 18366 raft_consensus.cc:2955] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER e063541542a94969898868bf6eb5ce1b (127.16.29.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:43.321800 17583 catalog_manager.cc:5697] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb reported cstate change: config changed from index 3 to 4, VOTER e063541542a94969898868bf6eb5ce1b (127.16.29.67) evicted. New cstate: current_term: 2 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:43.345896 18782 raft_consensus.cc:2955] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:43.349134 17821 tablet_service.cc:1558] Processing DeleteTablet for tablet ac984176358d453a8800cc9e0e304061 with delete_type TABLET_DATA_TOMBSTONED (TS e063541542a94969898868bf6eb5ce1b not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:35032
I20260709 12:20:43.358167 18374 raft_consensus.cc:2955] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:43.370886 17837 raft_consensus.cc:1275] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260709 12:20:43.410519 18800 tablet_replica.cc:333] T ac984176358d453a8800cc9e0e304061 P e063541542a94969898868bf6eb5ce1b: stopping tablet replica
I20260709 12:20:43.411913 18800 raft_consensus.cc:2243] T ac984176358d453a8800cc9e0e304061 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:43.412858 18800 raft_consensus.cc:2272] T ac984176358d453a8800cc9e0e304061 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:43.419503 18800 ts_tablet_manager.cc:1916] T ac984176358d453a8800cc9e0e304061 P e063541542a94969898868bf6eb5ce1b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:20:43.432619 18781 consensus_queue.cc:1048] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.002s
I20260709 12:20:43.442638 17587 catalog_manager.cc:5697] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da reported cstate change: config changed from index 3 to 4, VOTER e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66) evicted. New cstate: current_term: 2 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:43.447188 17572 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 3219dcbfd86340ecbde507517060ff81 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:20:43.460182 17836 raft_consensus.cc:2955] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:43.464145 18800 ts_tablet_manager.cc:1929] T ac984176358d453a8800cc9e0e304061 P e063541542a94969898868bf6eb5ce1b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.3
I20260709 12:20:43.470783 18800 log.cc:1199] T ac984176358d453a8800cc9e0e304061 P e063541542a94969898868bf6eb5ce1b: Deleting WAL directory at /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599551912677-16501-0/minicluster-data/ts-2-root/wals/ac984176358d453a8800cc9e0e304061
I20260709 12:20:43.472697 17571 catalog_manager.cc:5028] TS e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837): tablet ac984176358d453a8800cc9e0e304061 (table test-workload [id=ddc5b74250df4c65909b73b3742d615f]) successfully deleted
I20260709 12:20:43.491194 17753 tablet_service.cc:1558] Processing DeleteTablet for tablet 3219dcbfd86340ecbde507517060ff81 with delete_type TABLET_DATA_TOMBSTONED (TS e2909cff1da44ad387f96a8ae3aa5afb not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:38774
I20260709 12:20:43.506533 18801 tablet_replica.cc:333] T 3219dcbfd86340ecbde507517060ff81 P e2909cff1da44ad387f96a8ae3aa5afb: stopping tablet replica
I20260709 12:20:43.528903 18801 raft_consensus.cc:2243] T 3219dcbfd86340ecbde507517060ff81 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:43.529691 18801 raft_consensus.cc:2272] T 3219dcbfd86340ecbde507517060ff81 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:43.533231 18801 ts_tablet_manager.cc:1916] T 3219dcbfd86340ecbde507517060ff81 P e2909cff1da44ad387f96a8ae3aa5afb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:20:43.546278 18801 ts_tablet_manager.cc:1929] T 3219dcbfd86340ecbde507517060ff81 P e2909cff1da44ad387f96a8ae3aa5afb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.3
I20260709 12:20:43.546790 18801 log.cc:1199] T 3219dcbfd86340ecbde507517060ff81 P e2909cff1da44ad387f96a8ae3aa5afb: Deleting WAL directory at /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599551912677-16501-0/minicluster-data/ts-1-root/wals/3219dcbfd86340ecbde507517060ff81
I20260709 12:20:43.548972 17571 catalog_manager.cc:5028] TS e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): tablet 3219dcbfd86340ecbde507517060ff81 (table test-workload [id=ddc5b74250df4c65909b73b3742d615f]) successfully deleted
I20260709 12:20:44.572234 17684 consensus_queue.cc:237] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:20:44.579355 17836 raft_consensus.cc:1275] T 12b1fcd35e12402e8d6df7504c578e01 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260709 12:20:44.580389 17654 consensus_peers.cc:597] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 -> Peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Couldn't send request to peer 334a4e1f59944544b3b3ff36616b57da. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 12b1fcd35e12402e8d6df7504c578e01. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:44.581194 18781 consensus_queue.cc:1048] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:20:44.585435 17762 raft_consensus.cc:1275] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:20:44.587524 18781 consensus_queue.cc:1048] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:20:44.601925 18803 raft_consensus.cc:2955] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } }
I20260709 12:20:44.614269 17836 raft_consensus.cc:2955] T 12b1fcd35e12402e8d6df7504c578e01 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } }
I20260709 12:20:44.624156 17585 catalog_manager.cc:5697] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 reported cstate change: config changed from index -1 to 2, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New cstate: current_term: 1 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:44.633591 17684 consensus_queue.cc:237] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:20:44.640515 17762 raft_consensus.cc:2955] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } }
W20260709 12:20:44.645803 17654 consensus_peers.cc:597] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 -> Peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Couldn't send request to peer 334a4e1f59944544b3b3ff36616b57da. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 50f9ff8f24a14a079db34b1ba88fa278. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:44.646049 17836 raft_consensus.cc:1275] T 50f9ff8f24a14a079db34b1ba88fa278 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:20:44.647728 17758 raft_consensus.cc:1275] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:20:44.647524 18781 consensus_queue.cc:1048] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:20:44.663057 18803 consensus_queue.cc:1048] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:20:44.706027 18811 raft_consensus.cc:2955] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } }
I20260709 12:20:44.736745 17836 raft_consensus.cc:2955] T 50f9ff8f24a14a079db34b1ba88fa278 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } }
I20260709 12:20:44.737495 17835 consensus_queue.cc:237] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:20:44.741559 17760 raft_consensus.cc:2955] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } }
I20260709 12:20:44.759033 17584 catalog_manager.cc:5697] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 reported cstate change: config changed from index -1 to 2, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New cstate: current_term: 1 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20260709 12:20:44.764524 17803 consensus_peers.cc:597] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b -> Peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Couldn't send request to peer 334a4e1f59944544b3b3ff36616b57da. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8b56346a04a14c5d9f73c5205b74cf2d. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:44.765684 17760 raft_consensus.cc:1275] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:20:44.769268 17684 raft_consensus.cc:1275] T 8b56346a04a14c5d9f73c5205b74cf2d P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:20:44.769462 18809 consensus_queue.cc:1048] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:20:44.770918 18799 consensus_queue.cc:1048] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:20:44.804977 18731 raft_consensus.cc:2955] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } }
I20260709 12:20:44.807003 17681 raft_consensus.cc:2955] T 8b56346a04a14c5d9f73c5205b74cf2d P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } }
I20260709 12:20:44.815409 17760 raft_consensus.cc:2955] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } }
I20260709 12:20:44.823278 17684 consensus_queue.cc:237] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:20:44.836041 17836 raft_consensus.cc:1275] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:20:44.837646 18781 consensus_queue.cc:1048] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260709 12:20:44.841085 17654 consensus_peers.cc:597] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 -> Peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Couldn't send request to peer 334a4e1f59944544b3b3ff36616b57da. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 044ba5aee1d6499a975c6b29d932fd95. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:44.840448 17584 catalog_manager.cc:5697] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b reported cstate change: config changed from index -1 to 2, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New cstate: current_term: 1 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:44.842511 17762 raft_consensus.cc:1275] T 044ba5aee1d6499a975c6b29d932fd95 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:20:44.854663 18810 consensus_queue.cc:1048] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:20:44.878669 18811 raft_consensus.cc:2955] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } }
I20260709 12:20:44.881028 17760 raft_consensus.cc:2955] T 044ba5aee1d6499a975c6b29d932fd95 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } }
I20260709 12:20:44.881047 17836 raft_consensus.cc:2955] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } }
I20260709 12:20:44.900177 17584 catalog_manager.cc:5697] T 044ba5aee1d6499a975c6b29d932fd95 P e2909cff1da44ad387f96a8ae3aa5afb reported cstate change: config changed from index -1 to 2, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New cstate: current_term: 1 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } }
I20260709 12:20:45.015170 18822 ts_tablet_manager.cc:933] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da: Initiating tablet copy from peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071)
I20260709 12:20:45.020826 18822 tablet_copy_client.cc:323] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Beginning tablet copy session from remote peer at address 127.16.29.65:33071
I20260709 12:20:45.022970 17699 tablet_copy_service.cc:140] P 0da169ff82674dc1947055ce43de68a3: Received BeginTabletCopySession request for tablet 12b1fcd35e12402e8d6df7504c578e01 from peer 334a4e1f59944544b3b3ff36616b57da ({username='slave'} at 127.0.0.1:44898)
I20260709 12:20:45.023531 17699 tablet_copy_service.cc:161] P 0da169ff82674dc1947055ce43de68a3: Beginning new tablet copy session on tablet 12b1fcd35e12402e8d6df7504c578e01 from peer 334a4e1f59944544b3b3ff36616b57da at {username='slave'} at 127.0.0.1:44898: session id = 334a4e1f59944544b3b3ff36616b57da-12b1fcd35e12402e8d6df7504c578e01
I20260709 12:20:45.032828 17699 tablet_copy_source_session.cc:215] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:20:45.038152 18822 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 12b1fcd35e12402e8d6df7504c578e01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:45.061287 18822 tablet_copy_client.cc:806] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Starting download of 0 data blocks...
I20260709 12:20:45.062216 18822 tablet_copy_client.cc:670] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Starting download of 1 WAL segments...
I20260709 12:20:45.066635 18822 tablet_copy_client.cc:538] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:20:45.077221 18822 tablet_bootstrap.cc:492] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da: Bootstrap starting.
I20260709 12:20:45.115717 18822 tablet_bootstrap.cc:492] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:20:45.116768 18822 tablet_bootstrap.cc:492] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da: Bootstrap complete.
I20260709 12:20:45.117496 18822 ts_tablet_manager.cc:1403] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da: Time spent bootstrapping tablet: real 0.041s	user 0.027s	sys 0.004s
I20260709 12:20:45.120647 18822 raft_consensus.cc:359] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:20:45.121559 18822 raft_consensus.cc:740] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Initialized, Role: LEARNER
I20260709 12:20:45.122287 18822 consensus_queue.cc:260] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:20:45.359022 18822 ts_tablet_manager.cc:1434] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da: Time spent starting tablet: real 0.241s	user 0.132s	sys 0.107s
I20260709 12:20:45.370893 18828 ts_tablet_manager.cc:933] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da: Initiating tablet copy from peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071)
I20260709 12:20:45.372138 17699 tablet_copy_service.cc:342] P 0da169ff82674dc1947055ce43de68a3: Request end of tablet copy session 334a4e1f59944544b3b3ff36616b57da-12b1fcd35e12402e8d6df7504c578e01 received from {username='slave'} at 127.0.0.1:44898
I20260709 12:20:45.378638 17699 tablet_copy_service.cc:434] P 0da169ff82674dc1947055ce43de68a3: ending tablet copy session 334a4e1f59944544b3b3ff36616b57da-12b1fcd35e12402e8d6df7504c578e01 on tablet 12b1fcd35e12402e8d6df7504c578e01 with peer 334a4e1f59944544b3b3ff36616b57da
I20260709 12:20:45.376206 18374 raft_consensus.cc:1217] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:20:45.387072 18822 ts_tablet_manager.cc:933] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da: Initiating tablet copy from peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:45.430573 18822 tablet_copy_client.cc:323] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Beginning tablet copy session from remote peer at address 127.16.29.67:36837
I20260709 12:20:45.432803 17848 tablet_copy_service.cc:140] P e063541542a94969898868bf6eb5ce1b: Received BeginTabletCopySession request for tablet 8b56346a04a14c5d9f73c5205b74cf2d from peer 334a4e1f59944544b3b3ff36616b57da ({username='slave'} at 127.0.0.1:37994)
I20260709 12:20:45.433360 17848 tablet_copy_service.cc:161] P e063541542a94969898868bf6eb5ce1b: Beginning new tablet copy session on tablet 8b56346a04a14c5d9f73c5205b74cf2d from peer 334a4e1f59944544b3b3ff36616b57da at {username='slave'} at 127.0.0.1:37994: session id = 334a4e1f59944544b3b3ff36616b57da-8b56346a04a14c5d9f73c5205b74cf2d
I20260709 12:20:45.442181 17848 tablet_copy_source_session.cc:215] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:20:45.446514 18822 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b56346a04a14c5d9f73c5205b74cf2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:45.465540 18822 tablet_copy_client.cc:806] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Starting download of 0 data blocks...
I20260709 12:20:45.466343 18822 tablet_copy_client.cc:670] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Starting download of 1 WAL segments...
I20260709 12:20:45.472046 18830 ts_tablet_manager.cc:933] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da: Initiating tablet copy from peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071)
I20260709 12:20:45.474279 18828 tablet_copy_client.cc:323] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Beginning tablet copy session from remote peer at address 127.16.29.65:33071
I20260709 12:20:45.475966 17699 tablet_copy_service.cc:140] P 0da169ff82674dc1947055ce43de68a3: Received BeginTabletCopySession request for tablet 50f9ff8f24a14a079db34b1ba88fa278 from peer 334a4e1f59944544b3b3ff36616b57da ({username='slave'} at 127.0.0.1:44898)
I20260709 12:20:45.476500 17699 tablet_copy_service.cc:161] P 0da169ff82674dc1947055ce43de68a3: Beginning new tablet copy session on tablet 50f9ff8f24a14a079db34b1ba88fa278 from peer 334a4e1f59944544b3b3ff36616b57da at {username='slave'} at 127.0.0.1:44898: session id = 334a4e1f59944544b3b3ff36616b57da-50f9ff8f24a14a079db34b1ba88fa278
I20260709 12:20:45.485028 18822 tablet_copy_client.cc:538] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:20:45.490577 17699 tablet_copy_source_session.cc:215] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:20:45.495703 18822 tablet_bootstrap.cc:492] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da: Bootstrap starting.
I20260709 12:20:45.514762 18828 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 50f9ff8f24a14a079db34b1ba88fa278. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:45.530700 18830 tablet_copy_client.cc:323] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Beginning tablet copy session from remote peer at address 127.16.29.65:33071
I20260709 12:20:45.532716 17699 tablet_copy_service.cc:140] P 0da169ff82674dc1947055ce43de68a3: Received BeginTabletCopySession request for tablet 044ba5aee1d6499a975c6b29d932fd95 from peer 334a4e1f59944544b3b3ff36616b57da ({username='slave'} at 127.0.0.1:44898)
I20260709 12:20:45.533245 17699 tablet_copy_service.cc:161] P 0da169ff82674dc1947055ce43de68a3: Beginning new tablet copy session on tablet 044ba5aee1d6499a975c6b29d932fd95 from peer 334a4e1f59944544b3b3ff36616b57da at {username='slave'} at 127.0.0.1:44898: session id = 334a4e1f59944544b3b3ff36616b57da-044ba5aee1d6499a975c6b29d932fd95
I20260709 12:20:45.539047 18828 tablet_copy_client.cc:806] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Starting download of 0 data blocks...
I20260709 12:20:45.539695 18828 tablet_copy_client.cc:670] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Starting download of 1 WAL segments...
I20260709 12:20:45.546056 17699 tablet_copy_source_session.cc:215] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:20:45.546573 18828 tablet_copy_client.cc:538] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:20:45.556115 18828 tablet_bootstrap.cc:492] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da: Bootstrap starting.
I20260709 12:20:45.558684 18830 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 044ba5aee1d6499a975c6b29d932fd95. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:45.558678 18822 tablet_bootstrap.cc:492] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:20:45.565909 18822 tablet_bootstrap.cc:492] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da: Bootstrap complete.
I20260709 12:20:45.566807 18822 ts_tablet_manager.cc:1403] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da: Time spent bootstrapping tablet: real 0.071s	user 0.033s	sys 0.006s
I20260709 12:20:45.589571 18830 tablet_copy_client.cc:806] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Starting download of 0 data blocks...
I20260709 12:20:45.590286 18830 tablet_copy_client.cc:670] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Starting download of 1 WAL segments...
I20260709 12:20:45.599357 18830 tablet_copy_client.cc:538] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:20:45.600584 18822 raft_consensus.cc:359] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:20:45.601511 18822 raft_consensus.cc:740] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Initialized, Role: LEARNER
I20260709 12:20:45.602210 18822 consensus_queue.cc:260] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:20:45.605624 18822 ts_tablet_manager.cc:1434] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da: Time spent starting tablet: real 0.038s	user 0.002s	sys 0.007s
I20260709 12:20:45.614893 17848 tablet_copy_service.cc:342] P e063541542a94969898868bf6eb5ce1b: Request end of tablet copy session 334a4e1f59944544b3b3ff36616b57da-8b56346a04a14c5d9f73c5205b74cf2d received from {username='slave'} at 127.0.0.1:37994
I20260709 12:20:45.615418 17848 tablet_copy_service.cc:434] P e063541542a94969898868bf6eb5ce1b: ending tablet copy session 334a4e1f59944544b3b3ff36616b57da-8b56346a04a14c5d9f73c5205b74cf2d on tablet 8b56346a04a14c5d9f73c5205b74cf2d with peer 334a4e1f59944544b3b3ff36616b57da
I20260709 12:20:45.639565 18830 tablet_bootstrap.cc:492] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da: Bootstrap starting.
I20260709 12:20:45.679436 18828 tablet_bootstrap.cc:492] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:20:45.680497 18828 tablet_bootstrap.cc:492] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da: Bootstrap complete.
I20260709 12:20:45.681223 18828 ts_tablet_manager.cc:1403] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da: Time spent bootstrapping tablet: real 0.125s	user 0.024s	sys 0.009s
I20260709 12:20:45.684358 18828 raft_consensus.cc:359] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:20:45.685294 18828 raft_consensus.cc:740] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Initialized, Role: LEARNER
I20260709 12:20:45.686058 18828 consensus_queue.cc:260] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:20:45.691918 18828 ts_tablet_manager.cc:1434] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da: Time spent starting tablet: real 0.010s	user 0.005s	sys 0.001s
I20260709 12:20:45.693894 17699 tablet_copy_service.cc:342] P 0da169ff82674dc1947055ce43de68a3: Request end of tablet copy session 334a4e1f59944544b3b3ff36616b57da-50f9ff8f24a14a079db34b1ba88fa278 received from {username='slave'} at 127.0.0.1:44898
I20260709 12:20:45.694304 17699 tablet_copy_service.cc:434] P 0da169ff82674dc1947055ce43de68a3: ending tablet copy session 334a4e1f59944544b3b3ff36616b57da-50f9ff8f24a14a079db34b1ba88fa278 on tablet 50f9ff8f24a14a079db34b1ba88fa278 with peer 334a4e1f59944544b3b3ff36616b57da
I20260709 12:20:45.698616 18830 tablet_bootstrap.cc:492] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:20:45.699710 18830 tablet_bootstrap.cc:492] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da: Bootstrap complete.
I20260709 12:20:45.700464 18830 ts_tablet_manager.cc:1403] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da: Time spent bootstrapping tablet: real 0.061s	user 0.026s	sys 0.006s
I20260709 12:20:45.703713 18830 raft_consensus.cc:359] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:20:45.704604 18830 raft_consensus.cc:740] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Initialized, Role: LEARNER
I20260709 12:20:45.705273 18830 consensus_queue.cc:260] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:20:45.708137 18830 ts_tablet_manager.cc:1434] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da: Time spent starting tablet: real 0.007s	user 0.007s	sys 0.000s
I20260709 12:20:45.711292 17699 tablet_copy_service.cc:342] P 0da169ff82674dc1947055ce43de68a3: Request end of tablet copy session 334a4e1f59944544b3b3ff36616b57da-044ba5aee1d6499a975c6b29d932fd95 received from {username='slave'} at 127.0.0.1:44898
I20260709 12:20:45.711687 17699 tablet_copy_service.cc:434] P 0da169ff82674dc1947055ce43de68a3: ending tablet copy session 334a4e1f59944544b3b3ff36616b57da-044ba5aee1d6499a975c6b29d932fd95 on tablet 044ba5aee1d6499a975c6b29d932fd95 with peer 334a4e1f59944544b3b3ff36616b57da
I20260709 12:20:45.805060 18371 raft_consensus.cc:1217] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
W20260709 12:20:45.830001 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 46727.2us
I20260709 12:20:45.883526 18781 raft_consensus.cc:1064] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3: attempting to promote NON_VOTER 334a4e1f59944544b3b3ff36616b57da to VOTER
I20260709 12:20:45.886817 18781 consensus_queue.cc:237] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:20:45.903473 18371 raft_consensus.cc:1275] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 1 LEARNER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:20:45.930228 18374 raft_consensus.cc:1217] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:20:45.921203 18840 consensus_queue.cc:1048] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:20:45.961669 18371 raft_consensus.cc:1217] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:20:45.984624 17837 raft_consensus.cc:1275] T 12b1fcd35e12402e8d6df7504c578e01 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:20:45.988092 17757 raft_consensus.cc:1275] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:20:45.994946 18841 consensus_queue.cc:1048] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:20:45.998598 18841 consensus_queue.cc:1048] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:20:46.020526 18781 raft_consensus.cc:2955] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:46.022152 17836 raft_consensus.cc:2955] T 12b1fcd35e12402e8d6df7504c578e01 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:46.044603 17762 raft_consensus.cc:2955] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:46.056445 17585 catalog_manager.cc:5697] T 12b1fcd35e12402e8d6df7504c578e01 P e063541542a94969898868bf6eb5ce1b reported cstate change: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:46.071054 18371 raft_consensus.cc:2955] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:46.079313 16501 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
W20260709 12:20:46.091238 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 45407.7us
I20260709 12:20:46.168246 17684 consensus_queue.cc:237] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:20:46.175601 18371 raft_consensus.cc:1275] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:20:46.178707 18803 consensus_queue.cc:1048] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:20:46.188282 18781 raft_consensus.cc:2955] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER e063541542a94969898868bf6eb5ce1b (127.16.29.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:46.201928 18371 raft_consensus.cc:2955] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER e063541542a94969898868bf6eb5ce1b (127.16.29.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:46.206694 17757 raft_consensus.cc:1275] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:20:46.206817 17762 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d"
dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
mode: GRACEFUL
new_leader_uuid: "334a4e1f59944544b3b3ff36616b57da"
 from {username='slave'} at 127.0.0.1:52946
I20260709 12:20:46.209597 17762 raft_consensus.cc:606] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 1 LEADER]: Received request to transfer leadership to TS 334a4e1f59944544b3b3ff36616b57da
I20260709 12:20:46.210117 18803 consensus_queue.cc:1048] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.002s
I20260709 12:20:46.212931 18843 raft_consensus.cc:1064] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3: attempting to promote NON_VOTER 334a4e1f59944544b3b3ff36616b57da to VOTER
I20260709 12:20:46.221293 18843 consensus_queue.cc:237] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:20:46.221693 17762 raft_consensus.cc:2955] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER e063541542a94969898868bf6eb5ce1b (127.16.29.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:46.246743 17584 catalog_manager.cc:5697] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb reported cstate change: config changed from index 3 to 4, VOTER e063541542a94969898868bf6eb5ce1b (127.16.29.67) evicted. New cstate: current_term: 1 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:46.252982 17572 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 12b1fcd35e12402e8d6df7504c578e01 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:20:46.310281 17821 tablet_service.cc:1558] Processing DeleteTablet for tablet 12b1fcd35e12402e8d6df7504c578e01 with delete_type TABLET_DATA_TOMBSTONED (TS e063541542a94969898868bf6eb5ce1b not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:35032
I20260709 12:20:46.345589 18855 tablet_replica.cc:333] T 12b1fcd35e12402e8d6df7504c578e01 P e063541542a94969898868bf6eb5ce1b: stopping tablet replica
I20260709 12:20:46.346700 17835 raft_consensus.cc:1275] T 50f9ff8f24a14a079db34b1ba88fa278 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:20:46.349022 17762 raft_consensus.cc:1275] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:20:46.350786 18781 consensus_queue.cc:1048] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:20:46.364332 18843 consensus_queue.cc:1048] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:20:46.385229 18781 raft_consensus.cc:2955] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:46.391608 18855 raft_consensus.cc:2243] T 12b1fcd35e12402e8d6df7504c578e01 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:46.392596 18855 raft_consensus.cc:2272] T 12b1fcd35e12402e8d6df7504c578e01 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:46.397722 18855 ts_tablet_manager.cc:1916] T 12b1fcd35e12402e8d6df7504c578e01 P e063541542a94969898868bf6eb5ce1b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:20:46.391603 17762 raft_consensus.cc:2955] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:46.400244 17835 raft_consensus.cc:2955] T 50f9ff8f24a14a079db34b1ba88fa278 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:46.417737 18855 ts_tablet_manager.cc:1929] T 12b1fcd35e12402e8d6df7504c578e01 P e063541542a94969898868bf6eb5ce1b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:20:46.418316 18855 log.cc:1199] T 12b1fcd35e12402e8d6df7504c578e01 P e063541542a94969898868bf6eb5ce1b: Deleting WAL directory at /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599551912677-16501-0/minicluster-data/ts-2-root/wals/12b1fcd35e12402e8d6df7504c578e01
I20260709 12:20:46.417784 18371 raft_consensus.cc:1275] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 1 LEARNER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:20:46.420064 17571 catalog_manager.cc:5028] TS e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837): tablet 12b1fcd35e12402e8d6df7504c578e01 (table test-workload [id=ddc5b74250df4c65909b73b3742d615f]) successfully deleted
I20260709 12:20:46.426419 18781 consensus_queue.cc:1048] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:20:46.429929 17583 catalog_manager.cc:5697] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb reported cstate change: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:46.447194 17688 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665"
dest_uuid: "0da169ff82674dc1947055ce43de68a3"
mode: GRACEFUL
new_leader_uuid: "334a4e1f59944544b3b3ff36616b57da"
 from {username='slave'} at 127.0.0.1:54540
I20260709 12:20:46.447809 17688 raft_consensus.cc:606] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 1 LEADER]: Received request to transfer leadership to TS 334a4e1f59944544b3b3ff36616b57da
I20260709 12:20:46.450879 18374 raft_consensus.cc:2955] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:46.459403 17688 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce"
dest_uuid: "0da169ff82674dc1947055ce43de68a3"
mode: GRACEFUL
new_leader_uuid: "334a4e1f59944544b3b3ff36616b57da"
 from {username='slave'} at 127.0.0.1:54540
I20260709 12:20:46.460208 17688 raft_consensus.cc:606] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 1 LEADER]: Received request to transfer leadership to TS 334a4e1f59944544b3b3ff36616b57da
I20260709 12:20:46.477299 18858 raft_consensus.cc:993] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3: : Instructing follower 334a4e1f59944544b3b3ff36616b57da to start an election
I20260709 12:20:46.477887 18781 raft_consensus.cc:1081] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 1 LEADER]: Signalling peer 334a4e1f59944544b3b3ff36616b57da to start an election
I20260709 12:20:46.479689 18374 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce"
dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
 from {username='slave'} at 127.0.0.1:59918
I20260709 12:20:46.482764 18374 raft_consensus.cc:493] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:20:46.483157 18374 raft_consensus.cc:3060] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:46.484915 17835 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d"
dest_uuid: "e063541542a94969898868bf6eb5ce1b"
mode: GRACEFUL
new_leader_uuid: "0da169ff82674dc1947055ce43de68a3"
 from {username='slave'} at 127.0.0.1:59820
I20260709 12:20:46.485504 17835 raft_consensus.cc:606] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 1 LEADER]: Received request to transfer leadership to TS 0da169ff82674dc1947055ce43de68a3
I20260709 12:20:46.488258 17688 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad"
dest_uuid: "0da169ff82674dc1947055ce43de68a3"
mode: GRACEFUL
new_leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
 from {username='slave'} at 127.0.0.1:54540
I20260709 12:20:46.488842 17688 raft_consensus.cc:606] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 1 LEADER]: Received request to transfer leadership to TS e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:20:46.493956 17688 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f"
dest_uuid: "0da169ff82674dc1947055ce43de68a3"
mode: GRACEFUL
new_leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
 from {username='slave'} at 127.0.0.1:54540
I20260709 12:20:46.494513 17688 raft_consensus.cc:606] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 1 LEADER]: Received request to transfer leadership to TS e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:20:46.495276 18842 raft_consensus.cc:1064] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3: attempting to promote NON_VOTER 334a4e1f59944544b3b3ff36616b57da to VOTER
I20260709 12:20:46.495832 16501 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 6
I20260709 12:20:46.496376 18374 raft_consensus.cc:515] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:20:46.497613 16501 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:20:46.498935 16501 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:20:46.498286 18842 consensus_queue.cc:237] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:20:46.500288 18374 leader_election.cc:290] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 2 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:20:46.501400 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:20:46.501994 17760 raft_consensus.cc:3060] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:46.504222 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:20:46.505003 17688 raft_consensus.cc:3055] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:20:46.505409 17688 raft_consensus.cc:740] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:20:46.506255 17688 consensus_queue.cc:260] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:20:46.507515 17688 raft_consensus.cc:3060] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:46.520133 17688 raft_consensus.cc:2468] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 2.
I20260709 12:20:46.521178 17760 raft_consensus.cc:2468] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 2.
I20260709 12:20:46.521484 18334 leader_election.cc:304] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:20:46.522425 18827 raft_consensus.cc:2804] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:20:46.532370 18799 raft_consensus.cc:1064] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b: attempting to promote NON_VOTER 334a4e1f59944544b3b3ff36616b57da to VOTER
W20260709 12:20:46.532855 18799 raft_consensus.cc:1068] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b: Unable to promote non-voter 334a4e1f59944544b3b3ff36616b57da: Service unavailable: leader transfer in progress
I20260709 12:20:46.546149 18827 raft_consensus.cc:697] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 2 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:20:46.547178 18827 consensus_queue.cc:237] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:20:46.551447 18374 raft_consensus.cc:1275] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 1 LEARNER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:20:46.553027 18841 consensus_queue.cc:1048] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:20:46.568080 18838 raft_consensus.cc:993] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb: : Instructing follower 334a4e1f59944544b3b3ff36616b57da to start an election
I20260709 12:20:46.568610 18838 raft_consensus.cc:1081] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 1 LEADER]: Signalling peer 334a4e1f59944544b3b3ff36616b57da to start an election
I20260709 12:20:46.570813 18367 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d"
dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
 from {username='slave'} at 127.0.0.1:59886
I20260709 12:20:46.571445 18367 raft_consensus.cc:493] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:20:46.571871 18367 raft_consensus.cc:3060] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:46.581043 18367 raft_consensus.cc:515] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:20:46.584221 17835 raft_consensus.cc:1275] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:20:46.585291 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:20:46.585887 17758 raft_consensus.cc:3055] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:20:46.586148 17758 raft_consensus.cc:740] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:20:46.586711 17758 consensus_queue.cc:260] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:20:46.587599 17758 raft_consensus.cc:3060] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:46.584221 17757 raft_consensus.cc:1275] T 044ba5aee1d6499a975c6b29d932fd95 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:20:46.588462 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:20:46.589136 17835 raft_consensus.cc:3060] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:46.592768 18811 consensus_queue.cc:1048] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:20:46.596982 18367 leader_election.cc:290] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 2 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:46.597545 17835 raft_consensus.cc:2468] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 2.
I20260709 12:20:46.599785 18333 leader_election.cc:304] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:20:46.600886 18862 raft_consensus.cc:2804] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:20:46.601548 18842 consensus_queue.cc:1048] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:20:46.609668 18842 raft_consensus.cc:993] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3: : Instructing follower 334a4e1f59944544b3b3ff36616b57da to start an election
I20260709 12:20:46.610373 18842 raft_consensus.cc:1081] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 1 LEADER]: Signalling peer 334a4e1f59944544b3b3ff36616b57da to start an election
I20260709 12:20:46.619807 18367 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665"
dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
 from {username='slave'} at 127.0.0.1:59918
I20260709 12:20:46.620440 18367 raft_consensus.cc:493] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:20:46.621073 18367 raft_consensus.cc:3060] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:46.625240 18862 raft_consensus.cc:697] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 2 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:20:46.626322 18862 consensus_queue.cc:237] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:20:46.633841 17758 raft_consensus.cc:2468] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 2.
I20260709 12:20:46.636547 18367 raft_consensus.cc:515] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:20:46.638566 18367 leader_election.cc:290] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 2 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:20:46.639317 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:20:46.639897 17688 raft_consensus.cc:3055] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:20:46.640189 17688 raft_consensus.cc:740] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:20:46.640961 17688 consensus_queue.cc:260] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:20:46.642490 17688 raft_consensus.cc:3060] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:46.640151 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:20:46.656397 17757 raft_consensus.cc:3060] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:46.652910 17583 catalog_manager.cc:5697] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 1 to 2, leader changed from 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) to 334a4e1f59944544b3b3ff36616b57da (127.16.29.68). New cstate: current_term: 2 leader_uuid: "334a4e1f59944544b3b3ff36616b57da" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:20:46.658900 17583 catalog_manager.cc:5697] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 1 to 2, leader changed from e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66) to 334a4e1f59944544b3b3ff36616b57da (127.16.29.68). New cstate: current_term: 2 leader_uuid: "334a4e1f59944544b3b3ff36616b57da" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:20:46.660259 18811 raft_consensus.cc:2955] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:46.663763 17760 raft_consensus.cc:2955] T 044ba5aee1d6499a975c6b29d932fd95 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:46.670161 18371 raft_consensus.cc:2955] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:46.683867 17587 catalog_manager.cc:5697] T 044ba5aee1d6499a975c6b29d932fd95 P e2909cff1da44ad387f96a8ae3aa5afb reported cstate change: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:46.690708 17688 raft_consensus.cc:2468] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 2.
I20260709 12:20:46.693653 17757 raft_consensus.cc:2468] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 2.
I20260709 12:20:46.694828 18333 leader_election.cc:304] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:20:46.695892 18827 raft_consensus.cc:2804] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:20:46.696511 18827 raft_consensus.cc:697] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 2 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:20:46.697695 18827 consensus_queue.cc:237] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:20:46.717046 18811 raft_consensus.cc:993] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3: : Instructing follower e2909cff1da44ad387f96a8ae3aa5afb to start an election
I20260709 12:20:46.718173 18811 raft_consensus.cc:1081] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 1 LEADER]: Signalling peer e2909cff1da44ad387f96a8ae3aa5afb to start an election
I20260709 12:20:46.720070 17760 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad"
dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
 from {username='slave'} at 127.0.0.1:38776
I20260709 12:20:46.719148 17835 raft_consensus.cc:2955] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:46.720719 17760 raft_consensus.cc:493] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:20:46.721071 17760 raft_consensus.cc:3060] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:46.725049 17688 consensus_queue.cc:237] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:20:46.729035 17760 raft_consensus.cc:515] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:46.731006 17760 leader_election.cc:290] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 2 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:46.731135 18371 raft_consensus.cc:1275] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:20:46.733217 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:20:46.733731 17836 raft_consensus.cc:3060] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:46.734999 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:20:46.735634 17688 raft_consensus.cc:3055] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:20:46.735937 17688 raft_consensus.cc:740] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:20:46.737690 17583 catalog_manager.cc:5697] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 1 to 2, leader changed from 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) to 334a4e1f59944544b3b3ff36616b57da (127.16.29.68). New cstate: current_term: 2 leader_uuid: "334a4e1f59944544b3b3ff36616b57da" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:20:46.740563 18811 consensus_queue.cc:1048] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:20:46.743752 17836 raft_consensus.cc:2468] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 2.
I20260709 12:20:46.745067 17727 leader_election.cc:304] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:20:46.746865 18826 raft_consensus.cc:2804] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:20:46.747349 17688 consensus_queue.cc:260] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:46.747342 18826 raft_consensus.cc:697] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 2 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:20:46.748672 17688 raft_consensus.cc:3060] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:46.749485 18826 consensus_queue.cc:237] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:46.755483 18371 raft_consensus.cc:2955] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:46.753978 18811 raft_consensus.cc:2955] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:46.759050 17688 raft_consensus.cc:2468] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 2.
I20260709 12:20:46.770027 17572 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 044ba5aee1d6499a975c6b29d932fd95 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:20:46.775629 17837 raft_consensus.cc:1275] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:20:46.783670 17587 catalog_manager.cc:5697] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 reported cstate change: config changed from index 3 to 4, VOTER e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66) evicted. New cstate: current_term: 1 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:20:46.787742 18811 consensus_queue.cc:1048] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:20:46.794697 17835 raft_consensus.cc:2955] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:46.796475 17585 catalog_manager.cc:5697] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb reported cstate change: term changed from 1 to 2, leader changed from 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) to e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66). New cstate: current_term: 2 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:46.815618 17753 tablet_service.cc:1558] Processing DeleteTablet for tablet 044ba5aee1d6499a975c6b29d932fd95 with delete_type TABLET_DATA_TOMBSTONED (TS e2909cff1da44ad387f96a8ae3aa5afb not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:38774
I20260709 12:20:46.818183 18872 tablet_replica.cc:333] T 044ba5aee1d6499a975c6b29d932fd95 P e2909cff1da44ad387f96a8ae3aa5afb: stopping tablet replica
I20260709 12:20:46.822630 18872 raft_consensus.cc:2243] T 044ba5aee1d6499a975c6b29d932fd95 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:46.823565 18872 raft_consensus.cc:2272] T 044ba5aee1d6499a975c6b29d932fd95 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:46.830183 18872 ts_tablet_manager.cc:1916] T 044ba5aee1d6499a975c6b29d932fd95 P e2909cff1da44ad387f96a8ae3aa5afb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:20:46.840497 18858 raft_consensus.cc:993] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3: : Instructing follower e2909cff1da44ad387f96a8ae3aa5afb to start an election
I20260709 12:20:46.841085 18781 raft_consensus.cc:1081] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 1 LEADER]: Signalling peer e2909cff1da44ad387f96a8ae3aa5afb to start an election
I20260709 12:20:46.842361 17757 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f"
dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
 from {username='slave'} at 127.0.0.1:38776
I20260709 12:20:46.842800 17757 raft_consensus.cc:493] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:20:46.843101 17757 raft_consensus.cc:3060] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:46.846087 18872 ts_tablet_manager.cc:1929] T 044ba5aee1d6499a975c6b29d932fd95 P e2909cff1da44ad387f96a8ae3aa5afb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:20:46.846521 18872 log.cc:1199] T 044ba5aee1d6499a975c6b29d932fd95 P e2909cff1da44ad387f96a8ae3aa5afb: Deleting WAL directory at /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599551912677-16501-0/minicluster-data/ts-1-root/wals/044ba5aee1d6499a975c6b29d932fd95
I20260709 12:20:46.848263 17571 catalog_manager.cc:5028] TS e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): tablet 044ba5aee1d6499a975c6b29d932fd95 (table test-workload [id=ddc5b74250df4c65909b73b3742d615f]) successfully deleted
I20260709 12:20:46.847643 17757 raft_consensus.cc:515] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:46.855369 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:20:46.856041 17688 raft_consensus.cc:3055] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:20:46.856334 17688 raft_consensus.cc:740] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:20:46.857038 17688 consensus_queue.cc:260] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:46.858073 17688 raft_consensus.cc:3060] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:46.861471 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:20:46.862043 17835 raft_consensus.cc:3060] T ea7da467817a4ba88aa5797c3f8f493f P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:46.863683 17757 leader_election.cc:290] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 2 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:46.864122 17688 raft_consensus.cc:2468] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 2.
I20260709 12:20:46.865499 17728 leader_election.cc:304] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:20:46.866256 18771 raft_consensus.cc:2804] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:20:46.866729 18771 raft_consensus.cc:697] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 2 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:20:46.867527 18771 consensus_queue.cc:237] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:46.870678 17681 consensus_queue.cc:237] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:20:46.875180 17835 raft_consensus.cc:2468] T ea7da467817a4ba88aa5797c3f8f493f P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 2.
I20260709 12:20:46.875651 18371 raft_consensus.cc:1275] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:20:46.877208 18842 consensus_queue.cc:1048] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:20:46.885825 18781 raft_consensus.cc:2955] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER e063541542a94969898868bf6eb5ce1b (127.16.29.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:46.887620 18371 raft_consensus.cc:2955] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER e063541542a94969898868bf6eb5ce1b (127.16.29.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:46.898360 17572 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 50f9ff8f24a14a079db34b1ba88fa278 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:20:46.907481 17585 catalog_manager.cc:5697] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 reported cstate change: config changed from index 3 to 4, VOTER e063541542a94969898868bf6eb5ce1b (127.16.29.67) evicted. New cstate: current_term: 1 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:20:46.911674 17757 raft_consensus.cc:1275] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:20:46.920230 18869 raft_consensus.cc:993] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b: : Instructing follower 0da169ff82674dc1947055ce43de68a3 to start an election
I20260709 12:20:46.918728 17587 catalog_manager.cc:5697] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb reported cstate change: term changed from 1 to 2, leader changed from 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) to e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66). New cstate: current_term: 2 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:46.921994 18825 raft_consensus.cc:1081] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 1 LEADER]: Signalling peer 0da169ff82674dc1947055ce43de68a3 to start an election
I20260709 12:20:46.929840 17681 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d"
dest_uuid: "0da169ff82674dc1947055ce43de68a3"
 from {username='slave'} at 127.0.0.1:37402
I20260709 12:20:46.930570 17681 raft_consensus.cc:493] T 8b56346a04a14c5d9f73c5205b74cf2d P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:20:46.930933 17681 raft_consensus.cc:3060] T 8b56346a04a14c5d9f73c5205b74cf2d P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:46.933398 17821 tablet_service.cc:1558] Processing DeleteTablet for tablet 50f9ff8f24a14a079db34b1ba88fa278 with delete_type TABLET_DATA_TOMBSTONED (TS e063541542a94969898868bf6eb5ce1b not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:35032
I20260709 12:20:46.925079 18858 consensus_queue.cc:1048] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:20:46.940692 17757 raft_consensus.cc:2955] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER e063541542a94969898868bf6eb5ce1b (127.16.29.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:46.967120 17681 raft_consensus.cc:515] T 8b56346a04a14c5d9f73c5205b74cf2d P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:20:46.969012 18874 tablet_replica.cc:333] T 50f9ff8f24a14a079db34b1ba88fa278 P e063541542a94969898868bf6eb5ce1b: stopping tablet replica
I20260709 12:20:46.972672 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:20:46.973768 17760 raft_consensus.cc:3060] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:46.976059 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:20:46.976663 17837 raft_consensus.cc:3055] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:20:46.976913 17837 raft_consensus.cc:740] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:20:46.977617 17837 consensus_queue.cc:260] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:20:46.978565 17837 raft_consensus.cc:3060] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:46.981302 17760 raft_consensus.cc:2468] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 2.
I20260709 12:20:46.982533 17652 leader_election.cc:304] T 8b56346a04a14c5d9f73c5205b74cf2d P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:20:46.983736 18781 raft_consensus.cc:2804] T 8b56346a04a14c5d9f73c5205b74cf2d P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:20:46.984241 18781 raft_consensus.cc:697] T 8b56346a04a14c5d9f73c5205b74cf2d P 0da169ff82674dc1947055ce43de68a3 [term 2 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:20:46.985059 18781 consensus_queue.cc:237] T 8b56346a04a14c5d9f73c5205b74cf2d P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:20:46.986195 17837 raft_consensus.cc:2468] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 2.
I20260709 12:20:46.986837 18874 raft_consensus.cc:2243] T 50f9ff8f24a14a079db34b1ba88fa278 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:46.988778 18874 raft_consensus.cc:2272] T 50f9ff8f24a14a079db34b1ba88fa278 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:46.987286 17681 leader_election.cc:290] T 8b56346a04a14c5d9f73c5205b74cf2d P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 2 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:47.013115 18874 ts_tablet_manager.cc:1916] T 50f9ff8f24a14a079db34b1ba88fa278 P e063541542a94969898868bf6eb5ce1b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:20:47.021772 17757 raft_consensus.cc:1275] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:20:47.025429 18827 consensus_queue.cc:1048] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:20:47.032325 18874 ts_tablet_manager.cc:1929] T 50f9ff8f24a14a079db34b1ba88fa278 P e063541542a94969898868bf6eb5ce1b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:20:47.032912 18874 log.cc:1199] T 50f9ff8f24a14a079db34b1ba88fa278 P e063541542a94969898868bf6eb5ce1b: Deleting WAL directory at /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599551912677-16501-0/minicluster-data/ts-2-root/wals/50f9ff8f24a14a079db34b1ba88fa278
I20260709 12:20:47.034471 17571 catalog_manager.cc:5028] TS e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837): tablet 50f9ff8f24a14a079db34b1ba88fa278 (table test-workload [id=ddc5b74250df4c65909b73b3742d615f]) successfully deleted
I20260709 12:20:47.034068 17587 catalog_manager.cc:5697] T 8b56346a04a14c5d9f73c5205b74cf2d P 0da169ff82674dc1947055ce43de68a3 reported cstate change: term changed from 1 to 2, leader changed from e063541542a94969898868bf6eb5ce1b (127.16.29.67) to 0da169ff82674dc1947055ce43de68a3 (127.16.29.65). New cstate: current_term: 2 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:47.052135 17681 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d"
dest_uuid: "0da169ff82674dc1947055ce43de68a3"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:37384
I20260709 12:20:47.052783 17681 raft_consensus.cc:606] T 8b56346a04a14c5d9f73c5205b74cf2d P 0da169ff82674dc1947055ce43de68a3 [term 2 LEADER]: Received request to transfer leadership
I20260709 12:20:47.056012 17837 raft_consensus.cc:1275] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:20:47.056442 17681 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d"
dest_uuid: "0da169ff82674dc1947055ce43de68a3"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:37384
I20260709 12:20:47.057014 17681 raft_consensus.cc:606] T 8b56346a04a14c5d9f73c5205b74cf2d P 0da169ff82674dc1947055ce43de68a3 [term 2 LEADER]: Received request to transfer leadership
W20260709 12:20:47.058373 17572 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: leadership transfer for 8b56346a04a14c5d9f73c5205b74cf2d already in progress [suppressed 2 similar messages]
I20260709 12:20:47.058385 18862 consensus_queue.cc:1048] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:20:47.063963 17681 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d"
dest_uuid: "0da169ff82674dc1947055ce43de68a3"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:37384
I20260709 12:20:47.064618 17681 raft_consensus.cc:606] T 8b56346a04a14c5d9f73c5205b74cf2d P 0da169ff82674dc1947055ce43de68a3 [term 2 LEADER]: Received request to transfer leadership
W20260709 12:20:47.071121 17635 auto_rebalancer.cc:893] Removing replace marker failed: Service unavailable: leader transfer in progress
W20260709 12:20:47.071552 17635 auto_rebalancer.cc:901] Could not move replica: Remote error: Service unavailable: leadership transfer for 8b56346a04a14c5d9f73c5205b74cf2d already in progress
W20260709 12:20:47.071782 17635 auto_rebalancer.cc:330] scheduled replica move failed to complete: Remote error: Service unavailable: leadership transfer for 8b56346a04a14c5d9f73c5205b74cf2d already in progress
I20260709 12:20:47.146198 17681 raft_consensus.cc:1275] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:20:47.147469 18862 consensus_queue.cc:1048] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:20:47.155970 17757 raft_consensus.cc:1275] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:20:47.157729 18827 consensus_queue.cc:1048] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:20:47.168215 17681 raft_consensus.cc:1275] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:20:47.170845 18837 consensus_queue.cc:1048] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:20:47.192852 17835 raft_consensus.cc:1275] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:20:47.194206 18837 consensus_queue.cc:1048] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:20:47.202059 17688 raft_consensus.cc:1275] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:20:47.206023 18827 consensus_queue.cc:1048] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:20:47.219605 17757 raft_consensus.cc:1275] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:20:47.221422 18865 consensus_queue.cc:1048] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:20:47.324149 17688 raft_consensus.cc:1275] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:20:47.325397 18837 consensus_queue.cc:1048] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:20:47.336058 17835 raft_consensus.cc:1275] T ea7da467817a4ba88aa5797c3f8f493f P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:20:47.338483 18837 consensus_queue.cc:1048] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:20:47.527781 17757 raft_consensus.cc:1275] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260709 12:20:47.529422 18841 consensus_queue.cc:1048] T 8b56346a04a14c5d9f73c5205b74cf2d P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:20:47.537246 18841 raft_consensus.cc:993] T 8b56346a04a14c5d9f73c5205b74cf2d P 0da169ff82674dc1947055ce43de68a3: : Instructing follower e2909cff1da44ad387f96a8ae3aa5afb to start an election
I20260709 12:20:47.537786 18811 raft_consensus.cc:1081] T 8b56346a04a14c5d9f73c5205b74cf2d P 0da169ff82674dc1947055ce43de68a3 [term 2 LEADER]: Signalling peer e2909cff1da44ad387f96a8ae3aa5afb to start an election
I20260709 12:20:47.542171 18371 raft_consensus.cc:3060] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 1 LEARNER]: Advancing to term 2
I20260709 12:20:47.542016 17835 raft_consensus.cc:1275] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260709 12:20:47.543903 18781 consensus_queue.cc:1048] T 8b56346a04a14c5d9f73c5205b74cf2d P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:20:47.544397 17757 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d"
dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
 from {username='slave'} at 127.0.0.1:38776
I20260709 12:20:47.544886 17757 raft_consensus.cc:493] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:20:47.545210 17757 raft_consensus.cc:3060] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:20:47.549821 18371 raft_consensus.cc:1275] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 2 LEARNER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260709 12:20:47.551273 18841 consensus_queue.cc:1048] T 8b56346a04a14c5d9f73c5205b74cf2d P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:20:47.551637 17757 raft_consensus.cc:515] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:20:47.554478 17757 leader_election.cc:290] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 3 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:47.555496 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: true dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:20:47.556083 17688 raft_consensus.cc:3055] T 8b56346a04a14c5d9f73c5205b74cf2d P 0da169ff82674dc1947055ce43de68a3 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:20:47.556354 17688 raft_consensus.cc:740] T 8b56346a04a14c5d9f73c5205b74cf2d P 0da169ff82674dc1947055ce43de68a3 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:20:47.557067 17688 consensus_queue.cc:260] T 8b56346a04a14c5d9f73c5205b74cf2d P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:20:47.558188 17688 raft_consensus.cc:3060] T 8b56346a04a14c5d9f73c5205b74cf2d P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:20:47.559593 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: true dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:20:47.568671 17688 raft_consensus.cc:2468] T 8b56346a04a14c5d9f73c5205b74cf2d P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 3.
I20260709 12:20:47.570544 17728 leader_election.cc:304] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:20:47.571319 18771 raft_consensus.cc:2804] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:20:47.571873 18771 raft_consensus.cc:697] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 3 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:20:47.572775 18771 consensus_queue.cc:237] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:20:47.591120 17585 catalog_manager.cc:5697] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb reported cstate change: term changed from 2 to 3, leader changed from 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) to e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66). New cstate: current_term: 3 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:47.619496 16501 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:20:47.662200 17835 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52"
dest_uuid: "e063541542a94969898868bf6eb5ce1b"
mode: GRACEFUL
new_leader_uuid: "334a4e1f59944544b3b3ff36616b57da"
 from {username='slave'} at 127.0.0.1:59820
I20260709 12:20:47.662801 17835 raft_consensus.cc:606] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 1 LEADER]: Received request to transfer leadership to TS 334a4e1f59944544b3b3ff36616b57da
I20260709 12:20:47.665297 17835 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "3de52d85935746e9a1e770fef3529988"
dest_uuid: "e063541542a94969898868bf6eb5ce1b"
mode: GRACEFUL
new_leader_uuid: "0da169ff82674dc1947055ce43de68a3"
 from {username='slave'} at 127.0.0.1:59820
I20260709 12:20:47.665788 17835 raft_consensus.cc:606] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 1 LEADER]: Received request to transfer leadership to TS 0da169ff82674dc1947055ce43de68a3
I20260709 12:20:47.668869 17835 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d"
dest_uuid: "e063541542a94969898868bf6eb5ce1b"
mode: GRACEFUL
new_leader_uuid: "0da169ff82674dc1947055ce43de68a3"
 from {username='slave'} at 127.0.0.1:59820
I20260709 12:20:47.669340 17835 raft_consensus.cc:606] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 1 LEADER]: Received request to transfer leadership to TS 0da169ff82674dc1947055ce43de68a3
I20260709 12:20:47.672224 17835 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462"
dest_uuid: "e063541542a94969898868bf6eb5ce1b"
mode: GRACEFUL
new_leader_uuid: "0da169ff82674dc1947055ce43de68a3"
 from {username='slave'} at 127.0.0.1:59820
I20260709 12:20:47.672947 17835 raft_consensus.cc:606] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 1 LEADER]: Received request to transfer leadership to TS 0da169ff82674dc1947055ce43de68a3
I20260709 12:20:47.676374 17835 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b"
dest_uuid: "e063541542a94969898868bf6eb5ce1b"
mode: GRACEFUL
new_leader_uuid: "0da169ff82674dc1947055ce43de68a3"
 from {username='slave'} at 127.0.0.1:59820
I20260709 12:20:47.676970 17835 raft_consensus.cc:606] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 1 LEADER]: Received request to transfer leadership to TS 0da169ff82674dc1947055ce43de68a3
I20260709 12:20:47.680042 17760 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f"
dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
mode: GRACEFUL
new_leader_uuid: "e063541542a94969898868bf6eb5ce1b"
 from {username='slave'} at 127.0.0.1:52946
I20260709 12:20:47.680702 17760 raft_consensus.cc:606] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 2 LEADER]: Received request to transfer leadership to TS e063541542a94969898868bf6eb5ce1b
I20260709 12:20:47.681967 16501 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 6
I20260709 12:20:47.683298 16501 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:20:47.683956 16501 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:20:47.747675 18897 raft_consensus.cc:993] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b: : Instructing follower 0da169ff82674dc1947055ce43de68a3 to start an election
I20260709 12:20:47.748337 18869 raft_consensus.cc:1081] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 1 LEADER]: Signalling peer 0da169ff82674dc1947055ce43de68a3 to start an election
I20260709 12:20:47.750041 17688 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "3de52d85935746e9a1e770fef3529988"
dest_uuid: "0da169ff82674dc1947055ce43de68a3"
 from {username='slave'} at 127.0.0.1:37402
I20260709 12:20:47.750602 17688 raft_consensus.cc:493] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:20:47.750937 17688 raft_consensus.cc:3060] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:47.757606 18826 raft_consensus.cc:993] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb: : Instructing follower e063541542a94969898868bf6eb5ce1b to start an election
I20260709 12:20:47.758157 18771 raft_consensus.cc:1081] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 2 LEADER]: Signalling peer e063541542a94969898868bf6eb5ce1b to start an election
I20260709 12:20:47.759441 17688 raft_consensus.cc:515] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:47.762060 17835 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f"
dest_uuid: "e063541542a94969898868bf6eb5ce1b"
 from {username='slave'} at 127.0.0.1:35042
I20260709 12:20:47.762640 17835 raft_consensus.cc:493] T ea7da467817a4ba88aa5797c3f8f493f P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:20:47.762962 17835 raft_consensus.cc:3060] T ea7da467817a4ba88aa5797c3f8f493f P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:20:47.765218 17688 leader_election.cc:290] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 2 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:47.767722 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:20:47.768263 17837 raft_consensus.cc:3055] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:20:47.768771 17837 raft_consensus.cc:740] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:20:47.769466 17837 consensus_queue.cc:260] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:47.770586 17837 raft_consensus.cc:3060] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:47.771062 17835 raft_consensus.cc:515] T ea7da467817a4ba88aa5797c3f8f493f P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:47.773710 17835 leader_election.cc:290] T ea7da467817a4ba88aa5797c3f8f493f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 3 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:20:47.776401 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:20:47.777031 17688 raft_consensus.cc:3060] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:20:47.778014 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:20:47.778563 17760 raft_consensus.cc:3060] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:47.784061 17688 raft_consensus.cc:2468] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 3.
I20260709 12:20:47.785229 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:20:47.785801 17757 raft_consensus.cc:3055] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:20:47.786070 17757 raft_consensus.cc:740] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 2 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:20:47.786314 17802 leader_election.cc:304] T ea7da467817a4ba88aa5797c3f8f493f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:20:47.786732 17757 consensus_queue.cc:260] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:47.787230 18869 raft_consensus.cc:2804] T ea7da467817a4ba88aa5797c3f8f493f P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:20:47.787460 17837 raft_consensus.cc:2468] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 2.
I20260709 12:20:47.787820 18869 raft_consensus.cc:697] T ea7da467817a4ba88aa5797c3f8f493f P e063541542a94969898868bf6eb5ce1b [term 3 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:20:47.787868 17757 raft_consensus.cc:3060] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:20:47.788691 17652 leader_election.cc:304] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:20:47.791188 18781 raft_consensus.cc:2804] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:20:47.792305 18869 consensus_queue.cc:237] T ea7da467817a4ba88aa5797c3f8f493f P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:47.793462 18781 raft_consensus.cc:697] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 2 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:20:47.794412 18781 consensus_queue.cc:237] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:47.799844 17760 raft_consensus.cc:2468] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 2.
I20260709 12:20:47.801388 17757 raft_consensus.cc:2468] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 3.
I20260709 12:20:47.808424 18869 raft_consensus.cc:993] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b: : Instructing follower 0da169ff82674dc1947055ce43de68a3 to start an election
I20260709 12:20:47.808974 18897 raft_consensus.cc:1081] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 1 LEADER]: Signalling peer 0da169ff82674dc1947055ce43de68a3 to start an election
I20260709 12:20:47.809546 17585 catalog_manager.cc:5697] T ea7da467817a4ba88aa5797c3f8f493f P e063541542a94969898868bf6eb5ce1b reported cstate change: term changed from 2 to 3, leader changed from e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66) to e063541542a94969898868bf6eb5ce1b (127.16.29.67). New cstate: current_term: 3 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: HEALTHY } } }
I20260709 12:20:47.811329 17688 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462"
dest_uuid: "0da169ff82674dc1947055ce43de68a3"
 from {username='slave'} at 127.0.0.1:37402
I20260709 12:20:47.812191 17688 raft_consensus.cc:493] T 8cab6f01c6a2485cb5024322a1bd5462 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:20:47.812660 17688 raft_consensus.cc:3060] T 8cab6f01c6a2485cb5024322a1bd5462 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:47.812134 17584 catalog_manager.cc:5697] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 reported cstate change: term changed from 1 to 2, leader changed from e063541542a94969898868bf6eb5ce1b (127.16.29.67) to 0da169ff82674dc1947055ce43de68a3 (127.16.29.65). New cstate: current_term: 2 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:47.820755 17688 raft_consensus.cc:515] T 8cab6f01c6a2485cb5024322a1bd5462 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:47.822904 17688 leader_election.cc:290] T 8cab6f01c6a2485cb5024322a1bd5462 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 2 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:47.823597 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:20:47.824088 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:20:47.824182 17757 raft_consensus.cc:3060] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:47.824745 17836 raft_consensus.cc:3055] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:20:47.825088 17836 raft_consensus.cc:740] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:20:47.825896 17836 consensus_queue.cc:260] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:47.826743 17836 raft_consensus.cc:3060] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:47.832170 17757 raft_consensus.cc:2468] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 2.
I20260709 12:20:47.833473 17652 leader_election.cc:304] T 8cab6f01c6a2485cb5024322a1bd5462 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:20:47.834282 18841 raft_consensus.cc:2804] T 8cab6f01c6a2485cb5024322a1bd5462 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:20:47.834760 18841 raft_consensus.cc:697] T 8cab6f01c6a2485cb5024322a1bd5462 P 0da169ff82674dc1947055ce43de68a3 [term 2 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:20:47.835579 18841 consensus_queue.cc:237] T 8cab6f01c6a2485cb5024322a1bd5462 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:47.836905 17836 raft_consensus.cc:2468] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 2.
I20260709 12:20:47.846961 17584 catalog_manager.cc:5697] T 8cab6f01c6a2485cb5024322a1bd5462 P 0da169ff82674dc1947055ce43de68a3 reported cstate change: term changed from 1 to 2, leader changed from e063541542a94969898868bf6eb5ce1b (127.16.29.67) to 0da169ff82674dc1947055ce43de68a3 (127.16.29.65). New cstate: current_term: 2 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:47.954128 17685 raft_consensus.cc:1275] T 8b56346a04a14c5d9f73c5205b74cf2d P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260709 12:20:47.954854 18897 raft_consensus.cc:993] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b: : Instructing follower 334a4e1f59944544b3b3ff36616b57da to start an election
I20260709 12:20:47.955379 18897 raft_consensus.cc:1081] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 1 LEADER]: Signalling peer 334a4e1f59944544b3b3ff36616b57da to start an election
I20260709 12:20:47.955521 18826 consensus_queue.cc:1048] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:20:47.956120 18371 raft_consensus.cc:3060] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 2 LEARNER]: Advancing to term 3
I20260709 12:20:47.957056 18367 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52"
dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
 from {username='slave'} at 127.0.0.1:59902
I20260709 12:20:47.957826 18367 raft_consensus.cc:493] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:20:47.958201 18367 raft_consensus.cc:3060] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:47.968704 18367 raft_consensus.cc:515] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:20:47.971495 18367 leader_election.cc:290] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 2 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:47.976084 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:20:47.977495 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:20:47.976872 17837 raft_consensus.cc:3055] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:20:47.978056 17760 raft_consensus.cc:3060] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:47.978341 17837 raft_consensus.cc:740] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:20:47.976796 17836 raft_consensus.cc:3060] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:20:47.979571 17837 consensus_queue.cc:260] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:20:47.980991 17837 raft_consensus.cc:3060] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:47.985786 17760 raft_consensus.cc:2468] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 2.
I20260709 12:20:47.986722 18371 raft_consensus.cc:1275] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 3 LEARNER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260709 12:20:47.987037 18333 leader_election.cc:304] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:20:47.987738 17836 raft_consensus.cc:1275] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260709 12:20:47.988188 18862 raft_consensus.cc:2804] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:20:47.988961 18862 raft_consensus.cc:697] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 2 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:20:47.989885 18862 consensus_queue.cc:237] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:20:47.993268 17837 raft_consensus.cc:2468] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 2.
I20260709 12:20:47.994560 18826 consensus_queue.cc:1048] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:20:47.995967 18826 consensus_queue.cc:1048] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:20:47.996063 18897 raft_consensus.cc:993] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b: : Instructing follower 0da169ff82674dc1947055ce43de68a3 to start an election
I20260709 12:20:47.996989 18897 raft_consensus.cc:1081] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 1 LEADER]: Signalling peer 0da169ff82674dc1947055ce43de68a3 to start an election
I20260709 12:20:48.000681 17685 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b"
dest_uuid: "0da169ff82674dc1947055ce43de68a3"
 from {username='slave'} at 127.0.0.1:37402
I20260709 12:20:48.001201 17685 raft_consensus.cc:493] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:20:48.001526 17685 raft_consensus.cc:3060] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:48.004446 18771 raft_consensus.cc:1064] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb: attempting to promote NON_VOTER 334a4e1f59944544b3b3ff36616b57da to VOTER
I20260709 12:20:48.006563 18771 consensus_queue.cc:237] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 3, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:20:48.009343 17685 raft_consensus.cc:515] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:48.015853 17585 catalog_manager.cc:5697] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 1 to 2, leader changed from e063541542a94969898868bf6eb5ce1b (127.16.29.67) to 334a4e1f59944544b3b3ff36616b57da (127.16.29.68). New cstate: current_term: 2 leader_uuid: "334a4e1f59944544b3b3ff36616b57da" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:20:48.018128 18367 raft_consensus.cc:1275] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 3 LEARNER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 12:20:48.019757 18826 consensus_queue.cc:1048] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:20:48.022339 17685 leader_election.cc:290] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 2 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:48.024387 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:20:48.024978 17837 raft_consensus.cc:3055] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:20:48.025249 17837 raft_consensus.cc:740] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:20:48.054970 17688 raft_consensus.cc:1275] T 8b56346a04a14c5d9f73c5205b74cf2d P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 12:20:48.055169 17835 raft_consensus.cc:1275] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 12:20:48.055096 17837 consensus_queue.cc:260] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:48.061483 17837 raft_consensus.cc:3060] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:48.056939 18771 consensus_queue.cc:1048] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.005s
I20260709 12:20:48.064256 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:20:48.064904 17760 raft_consensus.cc:3060] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:48.067694 18901 consensus_queue.cc:1048] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:20:48.073606 17760 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 2.
I20260709 12:20:48.073606 17837 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 2.
I20260709 12:20:48.078187 17652 leader_election.cc:304] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:20:48.078958 18781 raft_consensus.cc:2804] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:20:48.082422 18902 raft_consensus.cc:2955] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 3 LEADER]: Committing config change with OpId 3.5: config changed from index 2 to 5, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:48.083487 17688 raft_consensus.cc:2955] T 8b56346a04a14c5d9f73c5205b74cf2d P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 2 to 5, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:48.095485 18781 raft_consensus.cc:697] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 2 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:20:48.096571 18781 consensus_queue.cc:237] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:48.108527 17837 raft_consensus.cc:2955] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 2 to 5, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:48.114959 18367 raft_consensus.cc:2955] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 2 to 5, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:48.131070 17585 catalog_manager.cc:5697] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb reported cstate change: config changed from index 2 to 5, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:20:48.143283 17587 catalog_manager.cc:5697] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 reported cstate change: term changed from 1 to 2, leader changed from e063541542a94969898868bf6eb5ce1b (127.16.29.67) to 0da169ff82674dc1947055ce43de68a3 (127.16.29.65). New cstate: current_term: 2 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:48.188714 18869 raft_consensus.cc:993] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b: : Instructing follower 0da169ff82674dc1947055ce43de68a3 to start an election
I20260709 12:20:48.189271 18897 raft_consensus.cc:1081] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 1 LEADER]: Signalling peer 0da169ff82674dc1947055ce43de68a3 to start an election
I20260709 12:20:48.191349 17688 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d"
dest_uuid: "0da169ff82674dc1947055ce43de68a3"
 from {username='slave'} at 127.0.0.1:37402
I20260709 12:20:48.191831 17688 raft_consensus.cc:493] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:20:48.192222 17688 raft_consensus.cc:3060] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:48.200044 17760 raft_consensus.cc:1275] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:20:48.200623 17688 raft_consensus.cc:515] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:48.202037 18897 consensus_queue.cc:1048] T ea7da467817a4ba88aa5797c3f8f493f P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:20:48.203651 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:20:48.204034 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:20:48.204294 17760 raft_consensus.cc:3060] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:48.204679 17837 raft_consensus.cc:3055] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:20:48.205150 17837 raft_consensus.cc:740] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:20:48.205941 17837 consensus_queue.cc:260] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:48.207001 17837 raft_consensus.cc:3060] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:48.203814 17688 leader_election.cc:290] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 2 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:48.222558 17685 raft_consensus.cc:1275] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:20:48.222800 17837 raft_consensus.cc:2468] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 2.
I20260709 12:20:48.224624 18799 consensus_queue.cc:1048] T ea7da467817a4ba88aa5797c3f8f493f P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:20:48.229957 17760 raft_consensus.cc:2468] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 2.
I20260709 12:20:48.232544 17835 consensus_queue.cc:237] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:20:48.237623 17652 leader_election.cc:304] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:20:48.238801 18781 raft_consensus.cc:2804] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:20:48.239254 18781 raft_consensus.cc:697] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 2 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:20:48.240051 18781 consensus_queue.cc:237] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:20:48.254106 17803 consensus_peers.cc:597] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b -> Peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Couldn't send request to peer 334a4e1f59944544b3b3ff36616b57da. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4092aabb101d411798b0354f88a5dd2b. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:48.258086 17760 raft_consensus.cc:1275] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:20:48.260612 18897 consensus_queue.cc:1048] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:20:48.274574 17688 raft_consensus.cc:1275] T 4092aabb101d411798b0354f88a5dd2b P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:20:48.276186 18799 raft_consensus.cc:2955] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } }
I20260709 12:20:48.277585 18897 consensus_queue.cc:1048] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.002s
I20260709 12:20:48.278061 17757 raft_consensus.cc:2955] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } }
I20260709 12:20:48.288760 17587 catalog_manager.cc:5697] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 reported cstate change: term changed from 1 to 2, leader changed from e063541542a94969898868bf6eb5ce1b (127.16.29.67) to 0da169ff82674dc1947055ce43de68a3 (127.16.29.65). New cstate: current_term: 2 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:48.292758 17835 raft_consensus.cc:1275] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:20:48.294168 17685 consensus_queue.cc:237] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:20:48.296506 18781 consensus_queue.cc:1048] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:20:48.303269 17688 raft_consensus.cc:2955] T 4092aabb101d411798b0354f88a5dd2b P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } }
I20260709 12:20:48.315158 17585 catalog_manager.cc:5697] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b reported cstate change: config changed from index -1 to 2, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New cstate: current_term: 1 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20260709 12:20:48.329399 17654 consensus_peers.cc:597] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 -> Peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Couldn't send request to peer 334a4e1f59944544b3b3ff36616b57da. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 154e6b769a8d476ba26f7fe3a521fc8d. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:48.333045 17835 raft_consensus.cc:1275] T 154e6b769a8d476ba26f7fe3a521fc8d P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:20:48.336550 18781 consensus_queue.cc:1048] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:20:48.339303 17760 raft_consensus.cc:1275] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:20:48.341269 18781 consensus_queue.cc:1048] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:20:48.341930 17757 raft_consensus.cc:1275] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:20:48.343696 18904 consensus_queue.cc:1048] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:20:48.370453 18904 raft_consensus.cc:2955] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } }
I20260709 12:20:48.373835 17757 raft_consensus.cc:2955] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } }
I20260709 12:20:48.384264 17836 raft_consensus.cc:1275] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:20:48.386689 18903 consensus_queue.cc:1048] T 8cab6f01c6a2485cb5024322a1bd5462 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:20:48.392468 17837 consensus_queue.cc:237] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:20:48.405731 17585 catalog_manager.cc:5697] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 reported cstate change: config changed from index -1 to 2, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New cstate: current_term: 1 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:48.414742 17757 raft_consensus.cc:1275] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:20:48.416345 18903 consensus_queue.cc:1048] T 8cab6f01c6a2485cb5024322a1bd5462 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260709 12:20:48.418035 17803 consensus_peers.cc:597] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b -> Peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Couldn't send request to peer 334a4e1f59944544b3b3ff36616b57da. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5ad2828ec58f490a9ea564d9805b56b7. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:48.423005 17685 raft_consensus.cc:1275] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:20:48.430765 18869 consensus_queue.cc:1048] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:20:48.433080 17757 raft_consensus.cc:1275] T 5ad2828ec58f490a9ea564d9805b56b7 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:20:48.438891 18799 consensus_queue.cc:1048] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:20:48.438971 17835 raft_consensus.cc:2955] T 154e6b769a8d476ba26f7fe3a521fc8d P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } }
I20260709 12:20:48.453018 18869 raft_consensus.cc:2955] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } }
I20260709 12:20:48.454802 17757 raft_consensus.cc:2955] T 5ad2828ec58f490a9ea564d9805b56b7 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } }
I20260709 12:20:48.457363 17685 raft_consensus.cc:2955] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } }
I20260709 12:20:48.474774 17583 catalog_manager.cc:5697] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 reported cstate change: config changed from index -1 to 2, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New cstate: current_term: 1 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } }
I20260709 12:20:48.475926 17836 consensus_queue.cc:237] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
W20260709 12:20:48.489822 17803 consensus_peers.cc:597] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b -> Peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Couldn't send request to peer 334a4e1f59944544b3b3ff36616b57da. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0e02889970bc40dc8ca2e81483f211e0. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:48.490073 17757 raft_consensus.cc:1275] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:20:48.491494 18897 consensus_queue.cc:1048] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:20:48.496089 17685 raft_consensus.cc:1275] T 0e02889970bc40dc8ca2e81483f211e0 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:20:48.497659 18897 consensus_queue.cc:1048] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:20:48.502528 18799 raft_consensus.cc:2955] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } }
I20260709 12:20:48.505712 17760 raft_consensus.cc:2955] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } }
I20260709 12:20:48.508155 17685 raft_consensus.cc:2955] T 0e02889970bc40dc8ca2e81483f211e0 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } }
I20260709 12:20:48.511807 17758 consensus_queue.cc:237] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:20:48.515707 17762 raft_consensus.cc:1275] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:20:48.517777 17836 raft_consensus.cc:1275] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260709 12:20:48.518061 18862 consensus_queue.cc:1048] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:20:48.520509 18371 raft_consensus.cc:1275] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260709 12:20:48.522330 18901 consensus_queue.cc:1048] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 12:20:48.526425 18902 consensus_queue.cc:1048] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 12:20:48.540356 17585 catalog_manager.cc:5697] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b reported cstate change: config changed from index -1 to 2, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New cstate: current_term: 1 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:48.571630 18901 raft_consensus.cc:2955] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 3 LEADER]: Committing config change with OpId 3.6: config changed from index 5 to 6, VOTER 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:48.588759 17836 raft_consensus.cc:1275] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:20:48.589700 17835 raft_consensus.cc:2955] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Committing config change with OpId 3.6: config changed from index 5 to 6, VOTER 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:48.592054 17571 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 8b56346a04a14c5d9f73c5205b74cf2d with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:20:48.594730 18841 consensus_queue.cc:1048] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:20:48.602129 17838 raft_consensus.cc:1275] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:20:48.611248 18865 consensus_queue.cc:1048] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:20:48.624831 17760 raft_consensus.cc:1275] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:20:48.626250 18841 consensus_queue.cc:1048] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:20:48.631433 18367 raft_consensus.cc:2955] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Committing config change with OpId 3.6: config changed from index 5 to 6, VOTER 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:48.663493 17585 catalog_manager.cc:5697] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da reported cstate change: config changed from index 5 to 6, VOTER 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) evicted. New cstate: current_term: 3 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:48.679359 17676 tablet_service.cc:1558] Processing DeleteTablet for tablet 8b56346a04a14c5d9f73c5205b74cf2d with delete_type TABLET_DATA_TOMBSTONED (TS 0da169ff82674dc1947055ce43de68a3 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:37384
I20260709 12:20:48.715210 18929 ts_tablet_manager.cc:933] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da: Initiating tablet copy from peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:48.724861 16501 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:20:48.734047 18930 tablet_replica.cc:333] T 8b56346a04a14c5d9f73c5205b74cf2d P 0da169ff82674dc1947055ce43de68a3: stopping tablet replica
I20260709 12:20:48.741977 18930 raft_consensus.cc:2243] T 8b56346a04a14c5d9f73c5205b74cf2d P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:48.751614 18929 tablet_copy_client.cc:323] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Beginning tablet copy session from remote peer at address 127.16.29.67:36837
I20260709 12:20:48.753925 17848 tablet_copy_service.cc:140] P e063541542a94969898868bf6eb5ce1b: Received BeginTabletCopySession request for tablet 4092aabb101d411798b0354f88a5dd2b from peer 334a4e1f59944544b3b3ff36616b57da ({username='slave'} at 127.0.0.1:37994)
I20260709 12:20:48.755021 17848 tablet_copy_service.cc:161] P e063541542a94969898868bf6eb5ce1b: Beginning new tablet copy session on tablet 4092aabb101d411798b0354f88a5dd2b from peer 334a4e1f59944544b3b3ff36616b57da at {username='slave'} at 127.0.0.1:37994: session id = 334a4e1f59944544b3b3ff36616b57da-4092aabb101d411798b0354f88a5dd2b
I20260709 12:20:48.758378 18930 raft_consensus.cc:2272] T 8b56346a04a14c5d9f73c5205b74cf2d P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:48.762547 18930 ts_tablet_manager.cc:1916] T 8b56346a04a14c5d9f73c5205b74cf2d P 0da169ff82674dc1947055ce43de68a3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:20:48.767495 17848 tablet_copy_source_session.cc:215] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:20:48.780097 18929 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4092aabb101d411798b0354f88a5dd2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:48.781522 18930 ts_tablet_manager.cc:1929] T 8b56346a04a14c5d9f73c5205b74cf2d P 0da169ff82674dc1947055ce43de68a3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.5
I20260709 12:20:48.782047 18930 log.cc:1199] T 8b56346a04a14c5d9f73c5205b74cf2d P 0da169ff82674dc1947055ce43de68a3: Deleting WAL directory at /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599551912677-16501-0/minicluster-data/ts-0-root/wals/8b56346a04a14c5d9f73c5205b74cf2d
I20260709 12:20:48.783717 17572 catalog_manager.cc:5028] TS 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): tablet 8b56346a04a14c5d9f73c5205b74cf2d (table test-workload [id=ddc5b74250df4c65909b73b3742d615f]) successfully deleted
I20260709 12:20:48.794765 17760 raft_consensus.cc:1275] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:20:48.797191 18903 consensus_queue.cc:1048] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:20:48.799774 18929 tablet_copy_client.cc:806] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Starting download of 0 data blocks...
I20260709 12:20:48.800554 18929 tablet_copy_client.cc:670] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Starting download of 1 WAL segments...
I20260709 12:20:48.804735 18929 tablet_copy_client.cc:538] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:20:48.814555 17838 raft_consensus.cc:1275] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:20:48.816110 18903 consensus_queue.cc:1048] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:20:48.822983 18929 tablet_bootstrap.cc:492] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da: Bootstrap starting.
I20260709 12:20:48.905743 18934 ts_tablet_manager.cc:933] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da: Initiating tablet copy from peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:48.916975 17685 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95"
dest_uuid: "0da169ff82674dc1947055ce43de68a3"
mode: GRACEFUL
new_leader_uuid: "334a4e1f59944544b3b3ff36616b57da"
 from {username='slave'} at 127.0.0.1:54540
I20260709 12:20:48.917616 17685 raft_consensus.cc:606] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 1 LEADER]: Received request to transfer leadership to TS 334a4e1f59944544b3b3ff36616b57da
I20260709 12:20:48.920569 17760 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3"
dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
mode: GRACEFUL
new_leader_uuid: "0da169ff82674dc1947055ce43de68a3"
 from {username='slave'} at 127.0.0.1:52946
I20260709 12:20:48.921684 17760 raft_consensus.cc:606] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 LEADER]: Received request to transfer leadership to TS 0da169ff82674dc1947055ce43de68a3
I20260709 12:20:48.928628 17685 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d"
dest_uuid: "0da169ff82674dc1947055ce43de68a3"
mode: GRACEFUL
new_leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
 from {username='slave'} at 127.0.0.1:54540
I20260709 12:20:48.928869 18936 ts_tablet_manager.cc:933] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da: Initiating tablet copy from peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071)
I20260709 12:20:48.929219 17685 raft_consensus.cc:606] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 1 LEADER]: Received request to transfer leadership to TS e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:20:48.933208 18934 tablet_copy_client.cc:323] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Beginning tablet copy session from remote peer at address 127.16.29.67:36837
I20260709 12:20:48.934830 17848 tablet_copy_service.cc:140] P e063541542a94969898868bf6eb5ce1b: Received BeginTabletCopySession request for tablet 5ad2828ec58f490a9ea564d9805b56b7 from peer 334a4e1f59944544b3b3ff36616b57da ({username='slave'} at 127.0.0.1:37994)
I20260709 12:20:48.935088 17685 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "3de52d85935746e9a1e770fef3529988"
dest_uuid: "0da169ff82674dc1947055ce43de68a3"
mode: GRACEFUL
new_leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
 from {username='slave'} at 127.0.0.1:54540
I20260709 12:20:48.935433 17848 tablet_copy_service.cc:161] P e063541542a94969898868bf6eb5ce1b: Beginning new tablet copy session on tablet 5ad2828ec58f490a9ea564d9805b56b7 from peer 334a4e1f59944544b3b3ff36616b57da at {username='slave'} at 127.0.0.1:37994: session id = 334a4e1f59944544b3b3ff36616b57da-5ad2828ec58f490a9ea564d9805b56b7
I20260709 12:20:48.935702 17685 raft_consensus.cc:606] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 2 LEADER]: Received request to transfer leadership to TS e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:20:48.945775 18935 ts_tablet_manager.cc:933] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da: Initiating tablet copy from peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:48.946023 17685 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278"
dest_uuid: "0da169ff82674dc1947055ce43de68a3"
mode: GRACEFUL
new_leader_uuid: "334a4e1f59944544b3b3ff36616b57da"
 from {username='slave'} at 127.0.0.1:54540
I20260709 12:20:48.946821 17685 raft_consensus.cc:606] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 1 LEADER]: Received request to transfer leadership to TS 334a4e1f59944544b3b3ff36616b57da
I20260709 12:20:48.950284 17848 tablet_copy_source_session.cc:215] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:20:48.959590 18934 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ad2828ec58f490a9ea564d9805b56b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:48.971540 18936 tablet_copy_client.cc:323] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Beginning tablet copy session from remote peer at address 127.16.29.65:33071
I20260709 12:20:48.975518 18904 raft_consensus.cc:993] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3: : Instructing follower 334a4e1f59944544b3b3ff36616b57da to start an election
I20260709 12:20:48.979225 18904 raft_consensus.cc:1081] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 1 LEADER]: Signalling peer 334a4e1f59944544b3b3ff36616b57da to start an election
I20260709 12:20:48.985364 17699 tablet_copy_service.cc:140] P 0da169ff82674dc1947055ce43de68a3: Received BeginTabletCopySession request for tablet 154e6b769a8d476ba26f7fe3a521fc8d from peer 334a4e1f59944544b3b3ff36616b57da ({username='slave'} at 127.0.0.1:44898)
I20260709 12:20:48.985785 18935 tablet_copy_client.cc:323] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Beginning tablet copy session from remote peer at address 127.16.29.67:36837
I20260709 12:20:48.985981 17699 tablet_copy_service.cc:161] P 0da169ff82674dc1947055ce43de68a3: Beginning new tablet copy session on tablet 154e6b769a8d476ba26f7fe3a521fc8d from peer 334a4e1f59944544b3b3ff36616b57da at {username='slave'} at 127.0.0.1:44898: session id = 334a4e1f59944544b3b3ff36616b57da-154e6b769a8d476ba26f7fe3a521fc8d
I20260709 12:20:48.994460 18929 tablet_bootstrap.cc:492] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:20:48.995651 18929 tablet_bootstrap.cc:492] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da: Bootstrap complete.
I20260709 12:20:48.996361 18929 ts_tablet_manager.cc:1403] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da: Time spent bootstrapping tablet: real 0.174s	user 0.023s	sys 0.003s
I20260709 12:20:48.997175 17848 tablet_copy_service.cc:140] P e063541542a94969898868bf6eb5ce1b: Received BeginTabletCopySession request for tablet 0e02889970bc40dc8ca2e81483f211e0 from peer 334a4e1f59944544b3b3ff36616b57da ({username='slave'} at 127.0.0.1:37994)
I20260709 12:20:48.997733 17848 tablet_copy_service.cc:161] P e063541542a94969898868bf6eb5ce1b: Beginning new tablet copy session on tablet 0e02889970bc40dc8ca2e81483f211e0 from peer 334a4e1f59944544b3b3ff36616b57da at {username='slave'} at 127.0.0.1:37994: session id = 334a4e1f59944544b3b3ff36616b57da-0e02889970bc40dc8ca2e81483f211e0
I20260709 12:20:48.998185 18934 tablet_copy_client.cc:806] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Starting download of 0 data blocks...
I20260709 12:20:48.998764 18934 tablet_copy_client.cc:670] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Starting download of 1 WAL segments...
I20260709 12:20:49.007401 17848 tablet_copy_source_session.cc:215] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:20:49.007258 17838 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343"
dest_uuid: "e063541542a94969898868bf6eb5ce1b"
mode: GRACEFUL
new_leader_uuid: "0da169ff82674dc1947055ce43de68a3"
 from {username='slave'} at 127.0.0.1:59820
I20260709 12:20:49.008198 17838 raft_consensus.cc:606] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 1 LEADER]: Received request to transfer leadership to TS 0da169ff82674dc1947055ce43de68a3
I20260709 12:20:49.009268 18934 tablet_copy_client.cc:538] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:20:49.010669 18935 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0e02889970bc40dc8ca2e81483f211e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:49.021893 18929 raft_consensus.cc:359] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:20:49.026003 18929 raft_consensus.cc:740] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Initialized, Role: LEARNER
I20260709 12:20:49.027323 18799 raft_consensus.cc:993] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b: : Instructing follower 0da169ff82674dc1947055ce43de68a3 to start an election
I20260709 12:20:49.023020 18934 tablet_bootstrap.cc:492] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da: Bootstrap starting.
I20260709 12:20:49.028090 18869 raft_consensus.cc:1081] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 1 LEADER]: Signalling peer 0da169ff82674dc1947055ce43de68a3 to start an election
I20260709 12:20:49.031203 17684 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343"
dest_uuid: "0da169ff82674dc1947055ce43de68a3"
 from {username='slave'} at 127.0.0.1:37402
I20260709 12:20:49.031535 18366 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95"
dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
 from {username='slave'} at 127.0.0.1:59918
I20260709 12:20:49.031853 17684 raft_consensus.cc:493] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:20:49.032228 18366 raft_consensus.cc:493] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:20:49.032393 17684 raft_consensus.cc:3060] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:49.032672 18366 raft_consensus.cc:3060] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:49.026863 18929 consensus_queue.cc:260] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:20:49.041932 18929 ts_tablet_manager.cc:1434] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da: Time spent starting tablet: real 0.045s	user 0.004s	sys 0.001s
I20260709 12:20:49.045006 17848 tablet_copy_service.cc:342] P e063541542a94969898868bf6eb5ce1b: Request end of tablet copy session 334a4e1f59944544b3b3ff36616b57da-4092aabb101d411798b0354f88a5dd2b received from {username='slave'} at 127.0.0.1:37994
I20260709 12:20:49.045437 17848 tablet_copy_service.cc:434] P e063541542a94969898868bf6eb5ce1b: ending tablet copy session 334a4e1f59944544b3b3ff36616b57da-4092aabb101d411798b0354f88a5dd2b on tablet 4092aabb101d411798b0354f88a5dd2b with peer 334a4e1f59944544b3b3ff36616b57da
I20260709 12:20:49.047215 18366 raft_consensus.cc:515] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:20:49.049600 18366 leader_election.cc:290] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 2 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:49.050563 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:20:49.051152 17836 raft_consensus.cc:3060] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:49.051489 18935 tablet_copy_client.cc:806] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Starting download of 0 data blocks...
I20260709 12:20:49.052242 18935 tablet_copy_client.cc:670] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Starting download of 1 WAL segments...
I20260709 12:20:49.054692 17684 raft_consensus.cc:515] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:49.055476 16501 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 6
I20260709 12:20:49.061160 16501 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:20:49.064471 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:20:49.064863 16501 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:20:49.065001 17838 raft_consensus.cc:3055] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:20:49.065323 17838 raft_consensus.cc:740] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:20:49.066030 17838 consensus_queue.cc:260] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:49.067034 17838 raft_consensus.cc:3060] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:49.071672 18934 tablet_bootstrap.cc:492] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:20:49.058188 18935 tablet_copy_client.cc:538] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:20:49.073748 18934 tablet_bootstrap.cc:492] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da: Bootstrap complete.
I20260709 12:20:49.074508 18934 ts_tablet_manager.cc:1403] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da: Time spent bootstrapping tablet: real 0.052s	user 0.025s	sys 0.006s
I20260709 12:20:49.062999 17684 leader_election.cc:290] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 2 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:49.075423 17836 raft_consensus.cc:2468] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 2.
I20260709 12:20:49.063287 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:20:49.058571 17699 tablet_copy_source_session.cc:215] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:20:49.076179 17760 raft_consensus.cc:3060] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:49.076650 18333 leader_election.cc:304] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:20:49.082069 18924 raft_consensus.cc:2804] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:20:49.082584 18924 raft_consensus.cc:697] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 2 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:20:49.083432 18924 consensus_queue.cc:237] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:20:49.085585 18934 raft_consensus.cc:359] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:20:49.087400 18934 raft_consensus.cc:740] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Initialized, Role: LEARNER
I20260709 12:20:49.088053 18934 consensus_queue.cc:260] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:20:49.089380 17760 raft_consensus.cc:2468] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 2.
I20260709 12:20:49.091032 17652 leader_election.cc:304] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:20:49.092128 18841 raft_consensus.cc:2804] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:20:49.092612 18841 raft_consensus.cc:697] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 2 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:20:49.093829 18936 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 154e6b769a8d476ba26f7fe3a521fc8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:49.085827 17838 raft_consensus.cc:2468] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 2.
I20260709 12:20:49.116411 18901 raft_consensus.cc:993] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb: : Instructing follower 0da169ff82674dc1947055ce43de68a3 to start an election
I20260709 12:20:49.117113 18901 raft_consensus.cc:1081] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 LEADER]: Signalling peer 0da169ff82674dc1947055ce43de68a3 to start an election
I20260709 12:20:49.126601 17838 raft_consensus.cc:1240] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 1. Current term is 2. Ops: []
I20260709 12:20:49.131932 18903 consensus_queue.cc:1059] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:20:49.142772 18934 ts_tablet_manager.cc:1434] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da: Time spent starting tablet: real 0.068s	user 0.010s	sys 0.000s
I20260709 12:20:49.148829 18367 raft_consensus.cc:1217] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:20:49.149120 17583 catalog_manager.cc:5697] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 1 to 2, leader changed from 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) to 334a4e1f59944544b3b3ff36616b57da (127.16.29.68). New cstate: current_term: 2 leader_uuid: "334a4e1f59944544b3b3ff36616b57da" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:20:49.153570 17848 tablet_copy_service.cc:342] P e063541542a94969898868bf6eb5ce1b: Request end of tablet copy session 334a4e1f59944544b3b3ff36616b57da-5ad2828ec58f490a9ea564d9805b56b7 received from {username='slave'} at 127.0.0.1:37994
I20260709 12:20:49.154081 17848 tablet_copy_service.cc:434] P e063541542a94969898868bf6eb5ce1b: ending tablet copy session 334a4e1f59944544b3b3ff36616b57da-5ad2828ec58f490a9ea564d9805b56b7 on tablet 5ad2828ec58f490a9ea564d9805b56b7 with peer 334a4e1f59944544b3b3ff36616b57da
I20260709 12:20:49.155383 18935 tablet_bootstrap.cc:492] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da: Bootstrap starting.
I20260709 12:20:49.158696 18841 consensus_queue.cc:237] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:20:49.254773 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 120895us
W20260709 12:20:49.289513 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 155875us
I20260709 12:20:49.311480 18781 raft_consensus.cc:3055] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:20:49.311934 18781 raft_consensus.cc:740] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:20:49.312664 18781 consensus_queue.cc:260] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:20:49.313912 18781 raft_consensus.cc:3060] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Advancing to term 2
W20260709 12:20:49.382457 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 133464us
I20260709 12:20:49.433334 18366 raft_consensus.cc:1217] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:20:49.459681 17584 catalog_manager.cc:5697] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 reported cstate change: term changed from 1 to 2, leader changed from e063541542a94969898868bf6eb5ce1b (127.16.29.67) to 0da169ff82674dc1947055ce43de68a3 (127.16.29.65). New cstate: current_term: 2 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: UNKNOWN } } }
W20260709 12:20:49.482685 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 192074us
I20260709 12:20:49.484519 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:20:49.510903 17684 raft_consensus.cc:2468] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 2.
I20260709 12:20:49.502054 18936 tablet_copy_client.cc:806] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Starting download of 0 data blocks...
I20260709 12:20:49.526311 17684 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3"
dest_uuid: "0da169ff82674dc1947055ce43de68a3"
 from {username='slave'} at 127.0.0.1:37396
I20260709 12:20:49.526989 17684 raft_consensus.cc:493] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:20:49.527376 17684 raft_consensus.cc:3060] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:49.545806 17684 raft_consensus.cc:515] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:49.547369 17684 leader_election.cc:290] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 2 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:49.524518 18936 tablet_copy_client.cc:670] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Starting download of 1 WAL segments...
W20260709 12:20:49.589335 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 85671.4us
W20260709 12:20:49.598675 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 296748us
I20260709 12:20:49.610152 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:20:49.610755 17837 raft_consensus.cc:3060] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:49.619472 17837 raft_consensus.cc:2468] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 2.
I20260709 12:20:49.623098 18935 tablet_bootstrap.cc:492] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:20:49.624125 18935 tablet_bootstrap.cc:492] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da: Bootstrap complete.
I20260709 12:20:49.624799 18935 ts_tablet_manager.cc:1403] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da: Time spent bootstrapping tablet: real 0.470s	user 0.026s	sys 0.005s
I20260709 12:20:49.627321 17837 raft_consensus.cc:1275] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:20:49.628918 18948 consensus_queue.cc:1048] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20260709 12:20:49.632223 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 90219.5us
I20260709 12:20:49.634137 18947 raft_consensus.cc:993] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3: : Instructing follower e2909cff1da44ad387f96a8ae3aa5afb to start an election
I20260709 12:20:49.634691 18947 raft_consensus.cc:1081] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 2 LEADER]: Signalling peer e2909cff1da44ad387f96a8ae3aa5afb to start an election
I20260709 12:20:49.636688 17652 leader_election.cc:304] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:20:49.638033 18947 raft_consensus.cc:2804] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:20:49.638720 18947 raft_consensus.cc:697] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 2 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:20:49.642210 18947 consensus_queue.cc:237] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:49.651662 18935 raft_consensus.cc:359] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:20:49.652611 18935 raft_consensus.cc:740] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Initialized, Role: LEARNER
I20260709 12:20:49.653285 18935 consensus_queue.cc:260] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:20:49.655638 18935 ts_tablet_manager.cc:1434] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da: Time spent starting tablet: real 0.031s	user 0.004s	sys 0.000s
I20260709 12:20:49.659617 18947 raft_consensus.cc:993] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3: : Instructing follower 334a4e1f59944544b3b3ff36616b57da to start an election
I20260709 12:20:49.661855 17587 catalog_manager.cc:5697] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 reported cstate change: term changed from 1 to 2, leader changed from e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66) to 0da169ff82674dc1947055ce43de68a3 (127.16.29.65). New cstate: current_term: 2 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:49.666072 17848 tablet_copy_service.cc:342] P e063541542a94969898868bf6eb5ce1b: Request end of tablet copy session 334a4e1f59944544b3b3ff36616b57da-0e02889970bc40dc8ca2e81483f211e0 received from {username='slave'} at 127.0.0.1:37994
I20260709 12:20:49.666356 18947 raft_consensus.cc:1081] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 1 LEADER]: Signalling peer 334a4e1f59944544b3b3ff36616b57da to start an election
I20260709 12:20:49.666586 17848 tablet_copy_service.cc:434] P e063541542a94969898868bf6eb5ce1b: ending tablet copy session 334a4e1f59944544b3b3ff36616b57da-0e02889970bc40dc8ca2e81483f211e0 on tablet 0e02889970bc40dc8ca2e81483f211e0 with peer 334a4e1f59944544b3b3ff36616b57da
I20260709 12:20:49.668889 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:20:49.669457 17760 raft_consensus.cc:3055] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:20:49.669735 17760 raft_consensus.cc:740] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:20:49.670534 17760 consensus_queue.cc:260] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:49.671805 17760 raft_consensus.cc:3060] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:49.684994 17757 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "3de52d85935746e9a1e770fef3529988"
dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
 from {username='slave'} at 127.0.0.1:38776
I20260709 12:20:49.685761 17757 raft_consensus.cc:493] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:20:49.686157 17757 raft_consensus.cc:3060] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:20:49.691012 18367 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278"
dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
 from {username='slave'} at 127.0.0.1:59918
I20260709 12:20:49.691298 18936 tablet_copy_client.cc:538] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:20:49.691610 18367 raft_consensus.cc:493] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:20:49.691994 18367 raft_consensus.cc:3060] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:49.699025 18367 raft_consensus.cc:515] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:20:49.700745 17757 raft_consensus.cc:515] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:49.701278 17684 raft_consensus.cc:1240] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 2 LEADER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 1. Current term is 2. Ops: []
I20260709 12:20:49.702600 17757 leader_election.cc:290] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 3 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:49.703485 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:20:49.704104 17837 raft_consensus.cc:3060] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:20:49.707315 18367 leader_election.cc:290] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 2 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:20:49.710137 17837 raft_consensus.cc:2468] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 3.
I20260709 12:20:49.711138 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:20:49.717923 17684 raft_consensus.cc:3055] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:20:49.718283 17684 raft_consensus.cc:740] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:20:49.719065 17684 consensus_queue.cc:260] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:49.713274 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:20:49.714237 17685 raft_consensus.cc:1275] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:20:49.715382 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:20:49.715992 18936 tablet_bootstrap.cc:492] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da: Bootstrap starting.
I20260709 12:20:49.720232 17684 raft_consensus.cc:3060] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:20:49.721361 17688 raft_consensus.cc:3055] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:20:49.744376 17688 raft_consensus.cc:740] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:20:49.745107 17688 consensus_queue.cc:260] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:20:49.746302 17688 raft_consensus.cc:3060] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:49.725068 18949 raft_consensus.cc:993] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3: : Instructing follower e2909cff1da44ad387f96a8ae3aa5afb to start an election
I20260709 12:20:49.748399 18949 raft_consensus.cc:1081] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 1 LEADER]: Signalling peer e2909cff1da44ad387f96a8ae3aa5afb to start an election
I20260709 12:20:49.728091 17757 raft_consensus.cc:3060] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:49.728886 17727 leader_election.cc:304] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:20:49.752045 18771 raft_consensus.cc:2804] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:20:49.752499 18771 raft_consensus.cc:697] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:20:49.753203 18771 consensus_queue.cc:237] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:49.757797 17764 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d"
dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
 from {username='slave'} at 127.0.0.1:38776
I20260709 12:20:49.758533 17764 raft_consensus.cc:493] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:20:49.758950 17764 raft_consensus.cc:3060] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:49.760583 17757 raft_consensus.cc:2468] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 2.
I20260709 12:20:49.761518 18333 leader_election.cc:304] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:20:49.762327 18948 raft_consensus.cc:2804] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:20:49.762749 18948 raft_consensus.cc:697] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 2 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:20:49.763574 18948 consensus_queue.cc:237] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:20:49.766959 17688 raft_consensus.cc:2468] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 2.
I20260709 12:20:49.768527 17684 raft_consensus.cc:2468] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 3.
I20260709 12:20:49.719847 17760 raft_consensus.cc:2468] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 2.
I20260709 12:20:49.730039 18924 consensus_queue.cc:1048] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.022s
I20260709 12:20:49.782958 17764 raft_consensus.cc:515] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:20:49.792027 17587 catalog_manager.cc:5697] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 1 to 2, leader changed from 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) to 334a4e1f59944544b3b3ff36616b57da (127.16.29.68). New cstate: current_term: 2 leader_uuid: "334a4e1f59944544b3b3ff36616b57da" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:20:49.794250 18951 raft_consensus.cc:1064] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b: attempting to promote NON_VOTER 334a4e1f59944544b3b3ff36616b57da to VOTER
I20260709 12:20:49.800066 17764 leader_election.cc:290] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 2 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:49.801704 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:20:49.804440 17688 raft_consensus.cc:3055] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:20:49.804893 17688 raft_consensus.cc:740] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:20:49.806002 17688 consensus_queue.cc:260] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:20:49.807842 17688 raft_consensus.cc:3060] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:49.818400 18958 raft_consensus.cc:1064] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b: attempting to promote NON_VOTER 334a4e1f59944544b3b3ff36616b57da to VOTER
I20260709 12:20:49.821645 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:20:49.822290 17831 raft_consensus.cc:3060] T 154e6b769a8d476ba26f7fe3a521fc8d P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:49.826224 18951 consensus_queue.cc:237] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:20:49.829600 17688 raft_consensus.cc:2468] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 2.
I20260709 12:20:49.830816 17728 leader_election.cc:304] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:20:49.831801 18837 raft_consensus.cc:2804] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:20:49.832294 18837 raft_consensus.cc:697] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 2 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:20:49.833142 18837 consensus_queue.cc:237] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:20:49.830245 17583 catalog_manager.cc:5697] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb reported cstate change: term changed from 2 to 3, leader changed from 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) to e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66). New cstate: current_term: 3 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:49.857445 17831 raft_consensus.cc:2468] T 154e6b769a8d476ba26f7fe3a521fc8d P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 2.
I20260709 12:20:49.877924 17837 raft_consensus.cc:1275] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:20:49.879576 18949 consensus_queue.cc:1048] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:20:49.853061 18958 consensus_queue.cc:237] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:20:49.893669 17764 raft_consensus.cc:1275] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:20:49.895839 18841 consensus_queue.cc:1048] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:20:49.948642 17584 catalog_manager.cc:5697] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb reported cstate change: term changed from 1 to 2, leader changed from 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) to e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66). New cstate: current_term: 2 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:49.989598 17684 raft_consensus.cc:1275] T 4092aabb101d411798b0354f88a5dd2b P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:20:49.993027 18869 consensus_queue.cc:1048] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:20:50.001598 18367 raft_consensus.cc:1275] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 1 LEARNER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:20:50.021040 18936 tablet_bootstrap.cc:492] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:20:50.022200 18936 tablet_bootstrap.cc:492] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da: Bootstrap complete.
I20260709 12:20:50.023003 18936 ts_tablet_manager.cc:1403] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da: Time spent bootstrapping tablet: real 0.320s	user 0.035s	sys 0.006s
I20260709 12:20:50.024061 17757 raft_consensus.cc:1275] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:20:50.042857 18366 raft_consensus.cc:1275] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 1 LEARNER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:20:50.043028 17688 raft_consensus.cc:1275] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:20:50.045181 18869 consensus_queue.cc:1048] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:20:50.046837 18958 consensus_queue.cc:1048] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:20:50.057862 17757 raft_consensus.cc:1275] T 5ad2828ec58f490a9ea564d9805b56b7 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:20:50.062752 18897 consensus_queue.cc:1048] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:20:50.047348 18936 raft_consensus.cc:359] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:20:50.003762 18975 consensus_queue.cc:1048] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.046s
I20260709 12:20:50.074600 18936 raft_consensus.cc:740] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Initialized, Role: LEARNER
I20260709 12:20:50.095301 18936 consensus_queue.cc:260] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:20:50.098639 18936 ts_tablet_manager.cc:1434] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da: Time spent starting tablet: real 0.064s	user 0.007s	sys 0.000s
I20260709 12:20:50.156744 18799 consensus_queue.cc:1048] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20260709 12:20:50.166740 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 94624.3us
I20260709 12:20:50.165894 17699 tablet_copy_service.cc:342] P 0da169ff82674dc1947055ce43de68a3: Request end of tablet copy session 334a4e1f59944544b3b3ff36616b57da-154e6b769a8d476ba26f7fe3a521fc8d received from {username='slave'} at 127.0.0.1:44898
I20260709 12:20:50.211850 17699 tablet_copy_service.cc:434] P 0da169ff82674dc1947055ce43de68a3: ending tablet copy session 334a4e1f59944544b3b3ff36616b57da-154e6b769a8d476ba26f7fe3a521fc8d on tablet 154e6b769a8d476ba26f7fe3a521fc8d with peer 334a4e1f59944544b3b3ff36616b57da
W20260709 12:20:50.300298 18978 log.cc:927] Time spent T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.156s	user 0.001s	sys 0.000s
W20260709 12:20:50.303316 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 44750.4us
I20260709 12:20:50.316932 17758 raft_consensus.cc:1275] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:20:50.330021 18975 raft_consensus.cc:2955] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
W20260709 12:20:50.378902 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 96037.1us
I20260709 12:20:50.395530 18945 consensus_queue.cc:1048] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:20:50.406746 17759 raft_consensus.cc:2955] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
W20260709 12:20:50.414275 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 190157us
W20260709 12:20:50.425830 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 172505us
I20260709 12:20:50.439148 17684 raft_consensus.cc:1275] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:20:50.441450 17759 raft_consensus.cc:1275] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:20:50.440610 18996 consensus_queue.cc:1048] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:20:50.453042 16501 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
W20260709 12:20:50.453291 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 315632us
I20260709 12:20:50.460193 18374 raft_consensus.cc:3060] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 1 LEARNER]: Advancing to term 2
I20260709 12:20:50.459319 18990 raft_consensus.cc:2955] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:50.465073 17837 raft_consensus.cc:1275] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:20:50.466404 18948 consensus_queue.cc:1048] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:20:50.468933 18374 raft_consensus.cc:1275] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 2 LEARNER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260709 12:20:50.470496 18960 consensus_queue.cc:1048] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20260709 12:20:50.489593 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 318072us
I20260709 12:20:50.495349 17684 raft_consensus.cc:1275] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:20:50.498167 18944 consensus_queue.cc:1048] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:20:50.499145 18368 raft_consensus.cc:1217] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:20:50.503762 18368 raft_consensus.cc:2955] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:50.506942 17688 raft_consensus.cc:1275] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260709 12:20:50.543560 18957 consensus_queue.cc:1048] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:20:50.553846 18957 consensus_queue.cc:1048] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:20:50.555603 17757 raft_consensus.cc:2955] T 5ad2828ec58f490a9ea564d9805b56b7 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:50.568990 18364 raft_consensus.cc:2955] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:50.578349 17685 raft_consensus.cc:2955] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:50.588604 17831 raft_consensus.cc:1275] T 154e6b769a8d476ba26f7fe3a521fc8d P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260709 12:20:50.596263 17831 raft_consensus.cc:1275] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:20:50.601687 17682 raft_consensus.cc:2955] T 4092aabb101d411798b0354f88a5dd2b P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:50.609335 18957 consensus_queue.cc:1048] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:20:50.615741 18771 consensus_queue.cc:1048] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:20:50.629237 18992 raft_consensus.cc:1064] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb: attempting to promote NON_VOTER 334a4e1f59944544b3b3ff36616b57da to VOTER
I20260709 12:20:50.639814 17685 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81"
dest_uuid: "0da169ff82674dc1947055ce43de68a3"
mode: GRACEFUL
new_leader_uuid: "334a4e1f59944544b3b3ff36616b57da"
 from {username='slave'} at 127.0.0.1:54540
I20260709 12:20:50.640858 17685 raft_consensus.cc:606] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 2 LEADER]: Received request to transfer leadership to TS 334a4e1f59944544b3b3ff36616b57da
W20260709 12:20:50.630065 18992 raft_consensus.cc:1068] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb: Unable to promote non-voter 334a4e1f59944544b3b3ff36616b57da: Illegal state: Leader has not yet committed an operation in its own term
I20260709 12:20:50.670051 17587 catalog_manager.cc:5697] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da reported cstate change: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:50.688989 18983 raft_consensus.cc:1064] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb: attempting to promote NON_VOTER 334a4e1f59944544b3b3ff36616b57da to VOTER
I20260709 12:20:50.672186 17831 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7"
dest_uuid: "e063541542a94969898868bf6eb5ce1b"
mode: GRACEFUL
new_leader_uuid: "0da169ff82674dc1947055ce43de68a3"
 from {username='slave'} at 127.0.0.1:59820
I20260709 12:20:50.709842 17831 raft_consensus.cc:606] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 1 LEADER]: Received request to transfer leadership to TS 0da169ff82674dc1947055ce43de68a3
I20260709 12:20:50.714425 17835 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9"
dest_uuid: "e063541542a94969898868bf6eb5ce1b"
mode: GRACEFUL
new_leader_uuid: "0da169ff82674dc1947055ce43de68a3"
 from {username='slave'} at 127.0.0.1:59820
I20260709 12:20:50.715049 17835 raft_consensus.cc:606] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 1 LEADER]: Received request to transfer leadership to TS 0da169ff82674dc1947055ce43de68a3
I20260709 12:20:50.714830 18983 consensus_queue.cc:237] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 2, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:20:50.718546 17684 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122"
dest_uuid: "0da169ff82674dc1947055ce43de68a3"
mode: GRACEFUL
new_leader_uuid: "334a4e1f59944544b3b3ff36616b57da"
 from {username='slave'} at 127.0.0.1:54540
I20260709 12:20:50.719184 17684 raft_consensus.cc:606] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 1 LEADER]: Received request to transfer leadership to TS 334a4e1f59944544b3b3ff36616b57da
W20260709 12:20:50.721673 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 47023.7us
I20260709 12:20:50.724210 17759 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c"
dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
mode: GRACEFUL
new_leader_uuid: "e063541542a94969898868bf6eb5ce1b"
 from {username='slave'} at 127.0.0.1:52946
I20260709 12:20:50.724843 17759 raft_consensus.cc:606] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 1 LEADER]: Received request to transfer leadership to TS e063541542a94969898868bf6eb5ce1b
I20260709 12:20:50.728338 16501 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260709 12:20:50.730460 16501 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:20:50.734881 16501 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:20:50.756079 18364 raft_consensus.cc:1275] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 2 LEARNER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260709 12:20:50.758030 17685 raft_consensus.cc:1275] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260709 12:20:50.761776 18837 consensus_queue.cc:1048] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:20:50.766222 18983 consensus_queue.cc:1048] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.007s
I20260709 12:20:50.776414 17583 catalog_manager.cc:5697] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b reported cstate change: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:20:50.791522 17831 raft_consensus.cc:1275] T 154e6b769a8d476ba26f7fe3a521fc8d P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260709 12:20:50.800642 18957 consensus_queue.cc:1048] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:20:50.802785 18992 raft_consensus.cc:2955] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 2 to 4, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:50.809337 18364 raft_consensus.cc:2955] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:50.842430 17836 consensus_queue.cc:237] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:20:50.869237 18365 raft_consensus.cc:1275] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:20:50.870309 17837 raft_consensus.cc:2955] T 154e6b769a8d476ba26f7fe3a521fc8d P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:50.876356 18987 consensus_queue.cc:1048] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:20:50.889616 17759 raft_consensus.cc:1275] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20260709 12:20:50.893607 17571 catalog_manager.cc:4755] TS e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837): ChangeConfig:REMOVE_PEER RPC failed for tablet 5ad2828ec58f490a9ea564d9805b56b7: Remote error: Service unavailable: leader transfer in progress
I20260709 12:20:50.901283 18869 raft_consensus.cc:993] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b: : Instructing follower 0da169ff82674dc1947055ce43de68a3 to start an election
I20260709 12:20:50.901829 18869 raft_consensus.cc:1081] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 1 LEADER]: Signalling peer 0da169ff82674dc1947055ce43de68a3 to start an election
I20260709 12:20:50.902945 18781 raft_consensus.cc:993] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3: : Instructing follower 334a4e1f59944544b3b3ff36616b57da to start an election
I20260709 12:20:50.903442 18781 raft_consensus.cc:1081] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 2 LEADER]: Signalling peer 334a4e1f59944544b3b3ff36616b57da to start an election
I20260709 12:20:50.905303 18371 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81"
dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
 from {username='slave'} at 127.0.0.1:59918
I20260709 12:20:50.905969 18371 raft_consensus.cc:493] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:20:50.906342 18371 raft_consensus.cc:3060] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:20:50.907681 17587 catalog_manager.cc:5697] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da reported cstate change: config changed from index 2 to 4, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:50.909119 17682 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9"
dest_uuid: "0da169ff82674dc1947055ce43de68a3"
 from {username='slave'} at 127.0.0.1:37402
I20260709 12:20:50.909739 17682 raft_consensus.cc:493] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:20:50.910151 17682 raft_consensus.cc:3060] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:50.909370 18869 raft_consensus.cc:2955] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:50.913630 18371 raft_consensus.cc:515] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:20:50.920934 18366 raft_consensus.cc:2955] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:50.922458 18987 consensus_queue.cc:1048] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:20:50.936179 18371 leader_election.cc:290] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 3 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:50.937222 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:20:50.938265 17836 raft_consensus.cc:3060] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:20:50.948253 18771 raft_consensus.cc:993] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb: : Instructing follower e063541542a94969898868bf6eb5ce1b to start an election
I20260709 12:20:50.948793 18771 raft_consensus.cc:1081] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 1 LEADER]: Signalling peer e063541542a94969898868bf6eb5ce1b to start an election
I20260709 12:20:50.954634 17837 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c"
dest_uuid: "e063541542a94969898868bf6eb5ce1b"
 from {username='slave'} at 127.0.0.1:35042
I20260709 12:20:50.955595 17837 raft_consensus.cc:493] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:20:50.956225 17837 raft_consensus.cc:3060] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:50.956326 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:20:50.957104 17684 raft_consensus.cc:3055] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:20:50.957670 17684 raft_consensus.cc:740] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:20:50.958468 17684 consensus_queue.cc:260] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:20:50.961109 17684 raft_consensus.cc:3060] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:20:50.963270 17836 raft_consensus.cc:2468] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 3.
I20260709 12:20:50.962605 17759 raft_consensus.cc:2955] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:50.964236 18897 raft_consensus.cc:1064] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b: attempting to promote NON_VOTER 334a4e1f59944544b3b3ff36616b57da to VOTER
I20260709 12:20:50.965689 17571 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 4092aabb101d411798b0354f88a5dd2b with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:20:50.966720 18897 consensus_queue.cc:237] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:20:50.968391 18333 leader_election.cc:304] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:20:50.970072 18924 raft_consensus.cc:2804] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:20:50.970801 18924 raft_consensus.cc:697] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 3 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:20:50.971757 18924 consensus_queue.cc:237] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:20:50.973835 17584 catalog_manager.cc:5697] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da reported cstate change: config changed from index 3 to 4, VOTER 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) evicted. New cstate: current_term: 1 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:50.982901 17682 raft_consensus.cc:515] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:20:50.985771 17684 raft_consensus.cc:2468] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 3.
I20260709 12:20:50.988991 17676 tablet_service.cc:1558] Processing DeleteTablet for tablet 4092aabb101d411798b0354f88a5dd2b with delete_type TABLET_DATA_TOMBSTONED (TS 0da169ff82674dc1947055ce43de68a3 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:37384
I20260709 12:20:50.999195 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:20:50.999955 17830 raft_consensus.cc:3055] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:20:51.000342 17830 raft_consensus.cc:740] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:20:51.000398 17587 catalog_manager.cc:5697] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 2 to 3, leader changed from 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) to 334a4e1f59944544b3b3ff36616b57da (127.16.29.68). New cstate: current_term: 3 leader_uuid: "334a4e1f59944544b3b3ff36616b57da" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:20:51.000566 17685 raft_consensus.cc:2955] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:51.005275 19015 tablet_replica.cc:333] T 4092aabb101d411798b0354f88a5dd2b P 0da169ff82674dc1947055ce43de68a3: stopping tablet replica
I20260709 12:20:51.007066 18869 raft_consensus.cc:993] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b: : Instructing follower 0da169ff82674dc1947055ce43de68a3 to start an election
I20260709 12:20:51.006114 17837 raft_consensus.cc:515] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:20:51.010835 18975 raft_consensus.cc:1081] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 1 LEADER]: Signalling peer 0da169ff82674dc1947055ce43de68a3 to start an election
I20260709 12:20:51.013216 17681 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7"
dest_uuid: "0da169ff82674dc1947055ce43de68a3"
 from {username='slave'} at 127.0.0.1:37402
I20260709 12:20:51.013780 17681 raft_consensus.cc:493] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:20:51.014129 17681 raft_consensus.cc:3060] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:51.021406 17682 leader_election.cc:290] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 2 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:20:51.023118 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:20:51.023763 18366 raft_consensus.cc:3060] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:51.001432 17830 consensus_queue.cc:260] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:20:51.025662 17830 raft_consensus.cc:3060] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:51.038002 17837 leader_election.cc:290] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 2 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:20:51.028573 17681 raft_consensus.cc:515] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:20:51.037752 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:20:51.038946 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:20:51.064157 17762 raft_consensus.cc:3055] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:20:51.064846 17762 raft_consensus.cc:740] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:20:51.065857 18904 raft_consensus.cc:993] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3: : Instructing follower 334a4e1f59944544b3b3ff36616b57da to start an election
I20260709 12:20:51.066418 18781 raft_consensus.cc:1081] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 1 LEADER]: Signalling peer 334a4e1f59944544b3b3ff36616b57da to start an election
I20260709 12:20:51.067837 18371 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122"
dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
 from {username='slave'} at 127.0.0.1:59918
I20260709 12:20:51.068332 18371 raft_consensus.cc:493] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:20:51.068661 18371 raft_consensus.cc:3060] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:51.065866 17762 consensus_queue.cc:260] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:20:51.073449 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:20:51.074186 17757 raft_consensus.cc:3060] T 5ad2828ec58f490a9ea564d9805b56b7 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:51.074653 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:20:51.075255 17837 raft_consensus.cc:3055] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:20:51.075544 17837 raft_consensus.cc:740] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:20:51.076702 17837 consensus_queue.cc:260] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:20:51.077981 17837 raft_consensus.cc:3060] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:51.080816 17681 leader_election.cc:290] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 2 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:20:51.084167 17762 raft_consensus.cc:3060] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:51.086114 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:20:51.086890 18365 raft_consensus.cc:3060] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:51.064734 18364 raft_consensus.cc:3060] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:51.099990 17762 raft_consensus.cc:2468] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 2.
I20260709 12:20:51.103020 17830 raft_consensus.cc:2468] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 2.
I20260709 12:20:51.036481 18366 raft_consensus.cc:2468] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 2.
I20260709 12:20:51.106637 18374 raft_consensus.cc:1275] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 1 LEARNER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:20:51.108258 18869 consensus_queue.cc:1048] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:20:51.118517 18371 raft_consensus.cc:515] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:20:51.119853 17801 leader_election.cc:304] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:20:51.120790 19015 raft_consensus.cc:2243] T 4092aabb101d411798b0354f88a5dd2b P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:51.121557 19015 raft_consensus.cc:2272] T 4092aabb101d411798b0354f88a5dd2b P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:51.121619 18987 raft_consensus.cc:2804] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:20:51.122190 18987 raft_consensus.cc:697] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 2 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:20:51.124393 17757 raft_consensus.cc:2468] T 5ad2828ec58f490a9ea564d9805b56b7 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 2.
I20260709 12:20:51.124940 17652 leader_election.cc:304] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:20:51.126832 17762 raft_consensus.cc:1275] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:20:51.129024 17837 raft_consensus.cc:2468] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 2.
I20260709 12:20:51.130743 18371 leader_election.cc:290] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 2 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:51.123076 18987 consensus_queue.cc:237] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:20:51.134826 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:20:51.134982 18365 raft_consensus.cc:2468] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 2.
I20260709 12:20:51.135397 17684 raft_consensus.cc:3055] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:20:51.135735 17684 raft_consensus.cc:740] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:20:51.136469 17684 consensus_queue.cc:260] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:20:51.137302 17684 raft_consensus.cc:3060] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:51.139269 17652 leader_election.cc:304] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:20:51.140658 18945 raft_consensus.cc:2804] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:20:51.140727 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:20:51.141223 18945 raft_consensus.cc:697] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 2 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:20:51.141333 17838 raft_consensus.cc:3060] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:51.142153 18945 consensus_queue.cc:237] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:20:51.144037 18904 raft_consensus.cc:2804] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:20:51.144640 18904 raft_consensus.cc:697] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 2 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:20:51.144819 17681 raft_consensus.cc:1275] T 0e02889970bc40dc8ca2e81483f211e0 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:20:51.146733 18799 consensus_queue.cc:1048] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:20:51.147771 17838 raft_consensus.cc:2468] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 2.
I20260709 12:20:51.150275 18333 leader_election.cc:304] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:20:51.123263 18364 raft_consensus.cc:2468] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 2.
W20260709 12:20:51.151542 17571 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet 5ad2828ec58f490a9ea564d9805b56b7 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica e063541542a94969898868bf6eb5ce1b is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:51.186013 18799 consensus_queue.cc:1048] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:20:51.151985 18924 raft_consensus.cc:2804] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:20:51.145552 18904 consensus_queue.cc:237] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:20:51.210338 18924 raft_consensus.cc:697] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 2 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:20:51.229753 17684 raft_consensus.cc:2468] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 2.
I20260709 12:20:51.236290 18869 raft_consensus.cc:2955] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:51.224223 18924 consensus_queue.cc:237] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:20:51.245486 19015 ts_tablet_manager.cc:1916] T 4092aabb101d411798b0354f88a5dd2b P 0da169ff82674dc1947055ce43de68a3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:20:51.245051 18367 raft_consensus.cc:2955] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:51.274400 17762 raft_consensus.cc:2955] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:51.277154 17587 catalog_manager.cc:5697] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 1 to 2, leader changed from 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) to 334a4e1f59944544b3b3ff36616b57da (127.16.29.68). New cstate: current_term: 2 leader_uuid: "334a4e1f59944544b3b3ff36616b57da" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:20:51.283241 17583 catalog_manager.cc:5697] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 reported cstate change: term changed from 1 to 2, leader changed from e063541542a94969898868bf6eb5ce1b (127.16.29.67) to 0da169ff82674dc1947055ce43de68a3 (127.16.29.65). New cstate: current_term: 2 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:51.285343 17583 catalog_manager.cc:5697] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 reported cstate change: term changed from 1 to 2, leader changed from e063541542a94969898868bf6eb5ce1b (127.16.29.67) to 0da169ff82674dc1947055ce43de68a3 (127.16.29.65). New cstate: current_term: 2 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:51.308372 17757 consensus_queue.cc:237] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:20:51.323766 17584 catalog_manager.cc:5697] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da reported cstate change: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:51.329783 18371 raft_consensus.cc:1275] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:20:51.329385 17681 raft_consensus.cc:2955] T 0e02889970bc40dc8ca2e81483f211e0 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:51.334517 17684 raft_consensus.cc:1275] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:20:51.334393 18837 consensus_queue.cc:1048] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:20:51.339936 17587 catalog_manager.cc:5697] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b reported cstate change: term changed from 1 to 2, leader changed from e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66) to e063541542a94969898868bf6eb5ce1b (127.16.29.67). New cstate: current_term: 2 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:51.343737 18960 consensus_queue.cc:1048] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:20:51.346697 18957 raft_consensus.cc:2955] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER e063541542a94969898868bf6eb5ce1b (127.16.29.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:51.350808 18371 raft_consensus.cc:2955] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER e063541542a94969898868bf6eb5ce1b (127.16.29.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:51.361497 19015 ts_tablet_manager.cc:1929] T 4092aabb101d411798b0354f88a5dd2b P 0da169ff82674dc1947055ce43de68a3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:20:51.365876 19015 log.cc:1199] T 4092aabb101d411798b0354f88a5dd2b P 0da169ff82674dc1947055ce43de68a3: Deleting WAL directory at /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599551912677-16501-0/minicluster-data/ts-0-root/wals/4092aabb101d411798b0354f88a5dd2b
I20260709 12:20:51.367409 17571 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 154e6b769a8d476ba26f7fe3a521fc8d with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:20:51.368219 17572 catalog_manager.cc:5028] TS 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): tablet 4092aabb101d411798b0354f88a5dd2b (table test-workload [id=ddc5b74250df4c65909b73b3742d615f]) successfully deleted
I20260709 12:20:51.372617 17830 consensus_queue.cc:237] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:20:51.381189 18367 raft_consensus.cc:1275] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:20:51.382380 17681 raft_consensus.cc:2955] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER e063541542a94969898868bf6eb5ce1b (127.16.29.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:51.383827 18951 consensus_queue.cc:1048] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:20:51.382403 17757 raft_consensus.cc:1275] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:20:51.388744 17684 raft_consensus.cc:1275] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 12:20:51.389925 18951 consensus_queue.cc:1048] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:20:51.390200 18924 consensus_queue.cc:1048] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:20:51.401127 18987 raft_consensus.cc:2955] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:51.405895 18367 raft_consensus.cc:2955] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:51.414583 17757 raft_consensus.cc:2955] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:51.424266 17584 catalog_manager.cc:5697] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da reported cstate change: config changed from index 4 to 5, VOTER e063541542a94969898868bf6eb5ce1b (127.16.29.67) evicted. New cstate: current_term: 2 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:51.442782 17571 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 0e02889970bc40dc8ca2e81483f211e0 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:20:51.448144 17830 raft_consensus.cc:1275] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 12:20:51.455063 18996 consensus_queue.cc:1048] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:20:51.490840 17585 catalog_manager.cc:5697] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b reported cstate change: config changed from index 3 to 4, VOTER 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) evicted. New cstate: current_term: 1 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:20:51.509222 17821 tablet_service.cc:1558] Processing DeleteTablet for tablet 154e6b769a8d476ba26f7fe3a521fc8d with delete_type TABLET_DATA_TOMBSTONED (TS e063541542a94969898868bf6eb5ce1b not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:35032
I20260709 12:20:51.510401 17676 tablet_service.cc:1558] Processing DeleteTablet for tablet 0e02889970bc40dc8ca2e81483f211e0 with delete_type TABLET_DATA_TOMBSTONED (TS 0da169ff82674dc1947055ce43de68a3 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:37384
I20260709 12:20:51.512319 19015 tablet_replica.cc:333] T 0e02889970bc40dc8ca2e81483f211e0 P 0da169ff82674dc1947055ce43de68a3: stopping tablet replica
I20260709 12:20:51.513298 19015 raft_consensus.cc:2243] T 0e02889970bc40dc8ca2e81483f211e0 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:51.513944 19015 raft_consensus.cc:2272] T 0e02889970bc40dc8ca2e81483f211e0 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:51.518235 19030 tablet_replica.cc:333] T 154e6b769a8d476ba26f7fe3a521fc8d P e063541542a94969898868bf6eb5ce1b: stopping tablet replica
I20260709 12:20:51.523260 19015 ts_tablet_manager.cc:1916] T 0e02889970bc40dc8ca2e81483f211e0 P 0da169ff82674dc1947055ce43de68a3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:20:51.539109 19030 raft_consensus.cc:2243] T 154e6b769a8d476ba26f7fe3a521fc8d P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:51.544528 17757 raft_consensus.cc:1275] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:20:51.546537 18987 consensus_queue.cc:1048] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:20:51.565187 19030 raft_consensus.cc:2272] T 154e6b769a8d476ba26f7fe3a521fc8d P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:51.568313 18367 raft_consensus.cc:1275] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:20:51.571183 18951 consensus_queue.cc:1048] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:20:51.579430 19030 ts_tablet_manager.cc:1916] T 154e6b769a8d476ba26f7fe3a521fc8d P e063541542a94969898868bf6eb5ce1b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:20:51.594040 19015 ts_tablet_manager.cc:1929] T 0e02889970bc40dc8ca2e81483f211e0 P 0da169ff82674dc1947055ce43de68a3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:20:51.597748 19030 ts_tablet_manager.cc:1929] T 154e6b769a8d476ba26f7fe3a521fc8d P e063541542a94969898868bf6eb5ce1b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20260709 12:20:51.609212 19030 log.cc:1199] T 154e6b769a8d476ba26f7fe3a521fc8d P e063541542a94969898868bf6eb5ce1b: Deleting WAL directory at /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599551912677-16501-0/minicluster-data/ts-2-root/wals/154e6b769a8d476ba26f7fe3a521fc8d
I20260709 12:20:51.594565 19015 log.cc:1199] T 0e02889970bc40dc8ca2e81483f211e0 P 0da169ff82674dc1947055ce43de68a3: Deleting WAL directory at /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599551912677-16501-0/minicluster-data/ts-0-root/wals/0e02889970bc40dc8ca2e81483f211e0
I20260709 12:20:51.611768 17571 catalog_manager.cc:5028] TS e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837): tablet 154e6b769a8d476ba26f7fe3a521fc8d (table test-workload [id=ddc5b74250df4c65909b73b3742d615f]) successfully deleted
I20260709 12:20:51.614272 17572 catalog_manager.cc:5028] TS 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): tablet 0e02889970bc40dc8ca2e81483f211e0 (table test-workload [id=ddc5b74250df4c65909b73b3742d615f]) successfully deleted
I20260709 12:20:51.622205 17757 raft_consensus.cc:1275] T 5ad2828ec58f490a9ea564d9805b56b7 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260709 12:20:51.625653 18945 consensus_queue.cc:1048] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:20:51.643602 18367 raft_consensus.cc:1275] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:20:51.647353 18904 consensus_queue.cc:1048] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:20:51.665994 17838 raft_consensus.cc:1275] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:20:51.667848 19023 consensus_queue.cc:1048] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:20:51.688336 18367 raft_consensus.cc:1275] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260709 12:20:51.691339 18904 consensus_queue.cc:1048] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:20:51.702435 17838 raft_consensus.cc:1275] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:20:51.705325 18924 consensus_queue.cc:1048] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:20:51.709821 17838 raft_consensus.cc:1275] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260709 12:20:51.714802 18904 consensus_queue.cc:1048] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:20:51.751550 17684 raft_consensus.cc:1275] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:20:51.753137 18924 consensus_queue.cc:1048] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:20:51.797226 17684 consensus_queue.cc:237] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:20:51.801927 18367 raft_consensus.cc:1275] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:20:51.802527 17831 raft_consensus.cc:1275] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:20:51.803735 18841 consensus_queue.cc:1048] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:20:51.804435 18904 consensus_queue.cc:1048] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:20:51.812307 19023 raft_consensus.cc:2955] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:51.814482 17831 raft_consensus.cc:2955] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:51.827013 17572 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 5ad2828ec58f490a9ea564d9805b56b7 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20260709 12:20:51.827100 18367 raft_consensus.cc:2955] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:51.836822 17584 catalog_manager.cc:5697] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b reported cstate change: config changed from index 3 to 5, VOTER e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66) evicted. New cstate: current_term: 2 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:51.853521 17753 tablet_service.cc:1558] Processing DeleteTablet for tablet 5ad2828ec58f490a9ea564d9805b56b7 with delete_type TABLET_DATA_TOMBSTONED (TS e2909cff1da44ad387f96a8ae3aa5afb not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:38774
I20260709 12:20:51.855934 19041 tablet_replica.cc:333] T 5ad2828ec58f490a9ea564d9805b56b7 P e2909cff1da44ad387f96a8ae3aa5afb: stopping tablet replica
I20260709 12:20:51.856778 19041 raft_consensus.cc:2243] T 5ad2828ec58f490a9ea564d9805b56b7 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:51.857412 19041 raft_consensus.cc:2272] T 5ad2828ec58f490a9ea564d9805b56b7 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:51.860250 19041 ts_tablet_manager.cc:1916] T 5ad2828ec58f490a9ea564d9805b56b7 P e2909cff1da44ad387f96a8ae3aa5afb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:20:51.870841 19041 ts_tablet_manager.cc:1929] T 5ad2828ec58f490a9ea564d9805b56b7 P e2909cff1da44ad387f96a8ae3aa5afb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20260709 12:20:51.871290 19041 log.cc:1199] T 5ad2828ec58f490a9ea564d9805b56b7 P e2909cff1da44ad387f96a8ae3aa5afb: Deleting WAL directory at /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599551912677-16501-0/minicluster-data/ts-1-root/wals/5ad2828ec58f490a9ea564d9805b56b7
I20260709 12:20:51.873078 17571 catalog_manager.cc:5028] TS e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): tablet 5ad2828ec58f490a9ea564d9805b56b7 (table test-workload [id=ddc5b74250df4c65909b73b3742d615f]) successfully deleted
I20260709 12:20:52.010067 16501 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:20:52.051623 17831 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185"
dest_uuid: "e063541542a94969898868bf6eb5ce1b"
mode: GRACEFUL
new_leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
 from {username='slave'} at 127.0.0.1:59820
I20260709 12:20:52.052289 17831 raft_consensus.cc:606] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 1 LEADER]: Received request to transfer leadership to TS e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:20:52.054734 17836 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3"
dest_uuid: "e063541542a94969898868bf6eb5ce1b"
mode: GRACEFUL
new_leader_uuid: "334a4e1f59944544b3b3ff36616b57da"
 from {username='slave'} at 127.0.0.1:59820
I20260709 12:20:52.055177 17836 raft_consensus.cc:606] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 1 LEADER]: Received request to transfer leadership to TS 334a4e1f59944544b3b3ff36616b57da
I20260709 12:20:52.057933 17681 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119"
dest_uuid: "0da169ff82674dc1947055ce43de68a3"
mode: GRACEFUL
new_leader_uuid: "334a4e1f59944544b3b3ff36616b57da"
 from {username='slave'} at 127.0.0.1:54540
I20260709 12:20:52.058606 17681 raft_consensus.cc:606] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 1 LEADER]: Received request to transfer leadership to TS 334a4e1f59944544b3b3ff36616b57da
I20260709 12:20:52.061975 17681 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343"
dest_uuid: "0da169ff82674dc1947055ce43de68a3"
mode: GRACEFUL
new_leader_uuid: "e063541542a94969898868bf6eb5ce1b"
 from {username='slave'} at 127.0.0.1:54540
I20260709 12:20:52.062633 17681 raft_consensus.cc:606] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 2 LEADER]: Received request to transfer leadership to TS e063541542a94969898868bf6eb5ce1b
I20260709 12:20:52.065366 17762 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "ac984176358d453a8800cc9e0e304061"
dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
mode: GRACEFUL
new_leader_uuid: "0da169ff82674dc1947055ce43de68a3"
 from {username='slave'} at 127.0.0.1:52946
I20260709 12:20:52.066000 17762 raft_consensus.cc:606] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 LEADER]: Received request to transfer leadership to TS 0da169ff82674dc1947055ce43de68a3
I20260709 12:20:52.067803 16501 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260709 12:20:52.069458 16501 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:20:52.070140 16501 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:20:52.253815 18975 raft_consensus.cc:993] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b: : Instructing follower e2909cff1da44ad387f96a8ae3aa5afb to start an election
I20260709 12:20:52.254432 18951 raft_consensus.cc:1081] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 1 LEADER]: Signalling peer e2909cff1da44ad387f96a8ae3aa5afb to start an election
I20260709 12:20:52.255769 17760 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185"
dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
 from {username='slave'} at 127.0.0.1:38786
I20260709 12:20:52.256491 17760 raft_consensus.cc:493] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:20:52.256726 17760 raft_consensus.cc:3060] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:52.261976 17760 raft_consensus.cc:515] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:20:52.264235 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:20:52.264407 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:20:52.264853 17831 raft_consensus.cc:3055] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:20:52.265096 18367 raft_consensus.cc:3060] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:52.265240 17831 raft_consensus.cc:740] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:20:52.266239 17831 consensus_queue.cc:260] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:20:52.267169 17831 raft_consensus.cc:3060] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:52.271842 18367 raft_consensus.cc:2468] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 2.
I20260709 12:20:52.264416 17760 leader_election.cc:290] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 2 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:20:52.273099 17729 leader_election.cc:304] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:20:52.273900 18988 raft_consensus.cc:2804] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:20:52.274339 18988 raft_consensus.cc:697] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:20:52.275095 18988 consensus_queue.cc:237] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:20:52.275869 17831 raft_consensus.cc:2468] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 2.
I20260709 12:20:52.286311 17584 catalog_manager.cc:5697] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb reported cstate change: term changed from 1 to 2, leader changed from e063541542a94969898868bf6eb5ce1b (127.16.29.67) to e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66). New cstate: current_term: 2 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:52.369029 18988 raft_consensus.cc:993] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb: : Instructing follower 0da169ff82674dc1947055ce43de68a3 to start an election
I20260709 12:20:52.369457 18960 raft_consensus.cc:1081] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 LEADER]: Signalling peer 0da169ff82674dc1947055ce43de68a3 to start an election
I20260709 12:20:52.371047 17684 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "ac984176358d453a8800cc9e0e304061"
dest_uuid: "0da169ff82674dc1947055ce43de68a3"
 from {username='slave'} at 127.0.0.1:37396
I20260709 12:20:52.371589 17684 raft_consensus.cc:493] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:20:52.371953 17684 raft_consensus.cc:3060] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:20:52.378995 17684 raft_consensus.cc:515] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:20:52.381709 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:20:52.382292 17762 raft_consensus.cc:3055] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:20:52.382728 17762 raft_consensus.cc:740] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:20:52.383399 17762 consensus_queue.cc:260] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:20:52.384181 17762 raft_consensus.cc:3060] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:20:52.381246 17684 leader_election.cc:290] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 3 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:20:52.384891 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:20:52.385455 18371 raft_consensus.cc:3060] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:20:52.389612 18371 raft_consensus.cc:2468] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 3.
I20260709 12:20:52.389696 17762 raft_consensus.cc:2468] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 3.
I20260709 12:20:52.390611 17654 leader_election.cc:304] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:20:52.391438 18841 raft_consensus.cc:2804] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:20:52.391966 18841 raft_consensus.cc:697] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 3 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:20:52.392716 18841 consensus_queue.cc:237] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:20:52.393950 18975 raft_consensus.cc:993] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b: : Instructing follower 334a4e1f59944544b3b3ff36616b57da to start an election
I20260709 12:20:52.394398 18951 raft_consensus.cc:1081] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 1 LEADER]: Signalling peer 334a4e1f59944544b3b3ff36616b57da to start an election
I20260709 12:20:52.396040 18371 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3"
dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
 from {username='slave'} at 127.0.0.1:59902
I20260709 12:20:52.396541 18371 raft_consensus.cc:493] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:20:52.396852 18371 raft_consensus.cc:3060] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:52.402184 18371 raft_consensus.cc:515] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:20:52.403901 18371 leader_election.cc:290] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 2 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:52.404927 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:20:52.404031 17584 catalog_manager.cc:5697] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 reported cstate change: term changed from 2 to 3, leader changed from e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66) to 0da169ff82674dc1947055ce43de68a3 (127.16.29.65). New cstate: current_term: 3 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:52.405407 17762 raft_consensus.cc:3060] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:52.407131 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:20:52.407716 17831 raft_consensus.cc:3055] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:20:52.408061 17831 raft_consensus.cc:740] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:20:52.408891 17831 consensus_queue.cc:260] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:20:52.409986 17831 raft_consensus.cc:3060] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:52.412559 17762 raft_consensus.cc:2468] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 2.
I20260709 12:20:52.413610 18333 leader_election.cc:304] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:20:52.414371 18924 raft_consensus.cc:2804] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:20:52.414844 18924 raft_consensus.cc:697] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 2 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:20:52.415692 18924 consensus_queue.cc:237] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:20:52.416661 17831 raft_consensus.cc:2468] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 2.
I20260709 12:20:52.425453 17584 catalog_manager.cc:5697] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 1 to 2, leader changed from e063541542a94969898868bf6eb5ce1b (127.16.29.67) to 334a4e1f59944544b3b3ff36616b57da (127.16.29.68). New cstate: current_term: 2 leader_uuid: "334a4e1f59944544b3b3ff36616b57da" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:20:52.457393 19023 raft_consensus.cc:993] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3: : Instructing follower e063541542a94969898868bf6eb5ce1b to start an election
I20260709 12:20:52.457926 18944 raft_consensus.cc:1081] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 2 LEADER]: Signalling peer e063541542a94969898868bf6eb5ce1b to start an election
I20260709 12:20:52.459451 17831 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343"
dest_uuid: "e063541542a94969898868bf6eb5ce1b"
 from {username='slave'} at 127.0.0.1:35040
I20260709 12:20:52.460052 17831 raft_consensus.cc:493] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:20:52.460409 17831 raft_consensus.cc:3060] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:20:52.466128 17831 raft_consensus.cc:515] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:52.468356 17831 leader_election.cc:290] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 3 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:20:52.469376 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:20:52.469890 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:20:52.470000 17762 raft_consensus.cc:3060] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:20:52.470487 17681 raft_consensus.cc:3055] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:20:52.470852 17681 raft_consensus.cc:740] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:20:52.471604 17681 consensus_queue.cc:260] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:52.472668 17681 raft_consensus.cc:3060] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:20:52.475835 17762 raft_consensus.cc:2468] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 3.
I20260709 12:20:52.476912 17801 leader_election.cc:304] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:20:52.477726 18975 raft_consensus.cc:2804] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:20:52.478190 18975 raft_consensus.cc:697] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 3 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:20:52.478637 17681 raft_consensus.cc:2468] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 3.
I20260709 12:20:52.479030 18975 consensus_queue.cc:237] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:52.490142 17584 catalog_manager.cc:5697] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b reported cstate change: term changed from 2 to 3, leader changed from 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) to e063541542a94969898868bf6eb5ce1b (127.16.29.67). New cstate: current_term: 3 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: HEALTHY } } }
I20260709 12:20:52.558202 18944 raft_consensus.cc:993] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3: : Instructing follower 334a4e1f59944544b3b3ff36616b57da to start an election
I20260709 12:20:52.558706 19023 raft_consensus.cc:1081] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 1 LEADER]: Signalling peer 334a4e1f59944544b3b3ff36616b57da to start an election
I20260709 12:20:52.560232 18367 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119"
dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
 from {username='slave'} at 127.0.0.1:59918
I20260709 12:20:52.560765 18367 raft_consensus.cc:493] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:20:52.561163 18367 raft_consensus.cc:3060] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:52.565621 18367 raft_consensus.cc:515] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:20:52.567499 18367 leader_election.cc:290] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 2 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:52.567811 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:20:52.568146 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:20:52.568492 17684 raft_consensus.cc:3055] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:20:52.568853 17684 raft_consensus.cc:740] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:20:52.568830 17831 raft_consensus.cc:3060] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:52.569831 17684 consensus_queue.cc:260] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:20:52.571102 17684 raft_consensus.cc:3060] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:52.576362 17831 raft_consensus.cc:2468] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 2.
I20260709 12:20:52.577643 18333 leader_election.cc:304] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:20:52.577886 17684 raft_consensus.cc:2468] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 2.
I20260709 12:20:52.578421 18948 raft_consensus.cc:2804] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:20:52.578931 18948 raft_consensus.cc:697] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 2 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:20:52.579849 18948 consensus_queue.cc:237] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:20:52.589299 17584 catalog_manager.cc:5697] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 1 to 2, leader changed from 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) to 334a4e1f59944544b3b3ff36616b57da (127.16.29.68). New cstate: current_term: 2 leader_uuid: "334a4e1f59944544b3b3ff36616b57da" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:20:52.742255 18367 raft_consensus.cc:1275] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:20:52.743399 18988 consensus_queue.cc:1048] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:20:52.756059 17831 raft_consensus.cc:1275] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:20:52.757730 18988 consensus_queue.cc:1048] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:20:52.809557 18367 raft_consensus.cc:1275] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 12:20:52.810797 18944 consensus_queue.cc:1048] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:20:52.819754 17760 raft_consensus.cc:1275] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 12:20:52.821830 19023 consensus_queue.cc:1048] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:20:52.924914 17760 consensus_queue.cc:237] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:20:52.930373 17684 raft_consensus.cc:1275] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:20:52.930373 17831 raft_consensus.cc:1275] T 84421f094faa40f39b30b4c3d0737639 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260709 12:20:52.930989 17729 consensus_peers.cc:597] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb -> Peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Couldn't send request to peer 334a4e1f59944544b3b3ff36616b57da. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 84421f094faa40f39b30b4c3d0737639. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:52.931733 18988 consensus_queue.cc:1048] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:20:52.932641 19046 consensus_queue.cc:1048] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:20:52.945726 19046 raft_consensus.cc:2955] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } }
I20260709 12:20:52.948179 17684 raft_consensus.cc:2955] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } }
I20260709 12:20:52.950032 17836 raft_consensus.cc:2955] T 84421f094faa40f39b30b4c3d0737639 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } }
I20260709 12:20:52.967854 17584 catalog_manager.cc:5697] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb reported cstate change: config changed from index -1 to 2, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New cstate: current_term: 1 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:52.970498 17836 consensus_queue.cc:237] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
W20260709 12:20:52.982661 17803 consensus_peers.cc:597] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b -> Peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Couldn't send request to peer 334a4e1f59944544b3b3ff36616b57da. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ab4d9071484d4dc0998db462cb01a10f. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:52.990171 17762 raft_consensus.cc:1275] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:20:52.992434 18975 consensus_queue.cc:1048] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:20:52.993968 17760 raft_consensus.cc:1275] T ab4d9071484d4dc0998db462cb01a10f P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:20:52.995986 17684 raft_consensus.cc:1275] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:20:52.996819 18951 consensus_queue.cc:1048] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:20:52.997673 18975 consensus_queue.cc:1048] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:20:53.002450 17836 raft_consensus.cc:1275] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:20:53.005008 18948 consensus_queue.cc:1048] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:20:53.012606 19062 raft_consensus.cc:2955] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } }
I20260709 12:20:53.014098 17684 raft_consensus.cc:1275] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:20:53.017069 18951 consensus_queue.cc:1048] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:20:53.025005 17760 raft_consensus.cc:1275] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:20:53.026381 18948 consensus_queue.cc:1048] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:20:53.013998 17681 raft_consensus.cc:2955] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } }
I20260709 12:20:53.036876 17831 raft_consensus.cc:1275] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:20:53.048841 17762 raft_consensus.cc:2955] T ab4d9071484d4dc0998db462cb01a10f P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } }
I20260709 12:20:53.052152 17760 consensus_queue.cc:237] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:20:53.062671 18948 consensus_queue.cc:1048] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:20:53.111831 17684 raft_consensus.cc:1275] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:20:53.116173 16501 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:20:53.174165 19070 consensus_queue.cc:1048] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.007s
W20260709 12:20:53.250727 17729 consensus_peers.cc:597] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb -> Peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Couldn't send request to peer 334a4e1f59944544b3b3ff36616b57da. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 28062a50142448aa97994888b4765920. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:20:53.257997 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 63908.7us
I20260709 12:20:53.270646 17585 catalog_manager.cc:5697] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 reported cstate change: config changed from index -1 to 2, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New cstate: current_term: 1 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } }
W20260709 12:20:53.291277 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 44587.9us
I20260709 12:20:53.303894 17830 raft_consensus.cc:1275] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:20:53.335619 19073 consensus_queue.cc:1048] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260709 12:20:53.345541 19076 log.cc:927] Time spent T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.098s	user 0.001s	sys 0.000s
I20260709 12:20:53.357654 17688 raft_consensus.cc:1275] T 28062a50142448aa97994888b4765920 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:20:53.420522 18960 consensus_queue.cc:1048] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:20:53.451753 19086 ts_tablet_manager.cc:933] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da: Initiating tablet copy from peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:20:53.453372 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 116797us
W20260709 12:20:53.461923 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 45087us
I20260709 12:20:53.495011 19086 tablet_copy_client.cc:323] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Beginning tablet copy session from remote peer at address 127.16.29.66:39651
I20260709 12:20:53.497328 17774 tablet_copy_service.cc:140] P e2909cff1da44ad387f96a8ae3aa5afb: Received BeginTabletCopySession request for tablet 84421f094faa40f39b30b4c3d0737639 from peer 334a4e1f59944544b3b3ff36616b57da ({username='slave'} at 127.0.0.1:36572)
I20260709 12:20:53.498188 17774 tablet_copy_service.cc:161] P e2909cff1da44ad387f96a8ae3aa5afb: Beginning new tablet copy session on tablet 84421f094faa40f39b30b4c3d0737639 from peer 334a4e1f59944544b3b3ff36616b57da at {username='slave'} at 127.0.0.1:36572: session id = 334a4e1f59944544b3b3ff36616b57da-84421f094faa40f39b30b4c3d0737639
I20260709 12:20:53.519013 19087 raft_consensus.cc:2955] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } }
I20260709 12:20:53.515377 17774 tablet_copy_source_session.cc:215] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:20:53.534231 19086 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 84421f094faa40f39b30b4c3d0737639. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:53.537755 17681 raft_consensus.cc:2955] T 28062a50142448aa97994888b4765920 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } }
I20260709 12:20:53.544637 19092 ts_tablet_manager.cc:933] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da: Initiating tablet copy from peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:53.561575 19086 tablet_copy_client.cc:806] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Starting download of 0 data blocks...
I20260709 12:20:53.562318 19086 tablet_copy_client.cc:670] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Starting download of 1 WAL segments...
I20260709 12:20:53.569350 19086 tablet_copy_client.cc:538] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:20:53.570863 17831 raft_consensus.cc:2955] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } }
I20260709 12:20:53.576785 17838 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0"
dest_uuid: "e063541542a94969898868bf6eb5ce1b"
mode: GRACEFUL
new_leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
 from {username='slave'} at 127.0.0.1:59820
I20260709 12:20:53.577638 17838 raft_consensus.cc:606] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 1 LEADER]: Received request to transfer leadership to TS e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:20:53.588500 19086 tablet_bootstrap.cc:492] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da: Bootstrap starting.
I20260709 12:20:53.594067 17759 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "28062a50142448aa97994888b4765920"
dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
mode: GRACEFUL
new_leader_uuid: "e063541542a94969898868bf6eb5ce1b"
 from {username='slave'} at 127.0.0.1:52946
I20260709 12:20:53.608266 17759 raft_consensus.cc:606] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 LEADER]: Received request to transfer leadership to TS e063541542a94969898868bf6eb5ce1b
I20260709 12:20:53.609598 19092 tablet_copy_client.cc:323] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Beginning tablet copy session from remote peer at address 127.16.29.67:36837
I20260709 12:20:53.605315 17584 catalog_manager.cc:5697] T 28062a50142448aa97994888b4765920 P 0da169ff82674dc1947055ce43de68a3 reported cstate change: config changed from index -1 to 2, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New cstate: current_term: 1 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } }
I20260709 12:20:53.614542 17848 tablet_copy_service.cc:140] P e063541542a94969898868bf6eb5ce1b: Received BeginTabletCopySession request for tablet ab4d9071484d4dc0998db462cb01a10f from peer 334a4e1f59944544b3b3ff36616b57da ({username='slave'} at 127.0.0.1:37994)
I20260709 12:20:53.615478 17848 tablet_copy_service.cc:161] P e063541542a94969898868bf6eb5ce1b: Beginning new tablet copy session on tablet ab4d9071484d4dc0998db462cb01a10f from peer 334a4e1f59944544b3b3ff36616b57da at {username='slave'} at 127.0.0.1:37994: session id = 334a4e1f59944544b3b3ff36616b57da-ab4d9071484d4dc0998db462cb01a10f
I20260709 12:20:53.622789 19088 raft_consensus.cc:993] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb: : Instructing follower e063541542a94969898868bf6eb5ce1b to start an election
I20260709 12:20:53.623349 19088 raft_consensus.cc:1081] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 LEADER]: Signalling peer e063541542a94969898868bf6eb5ce1b to start an election
I20260709 12:20:53.632346 17848 tablet_copy_source_session.cc:215] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:20:53.632340 17681 consensus_queue.cc:237] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:20:53.635438 17831 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "28062a50142448aa97994888b4765920"
dest_uuid: "e063541542a94969898868bf6eb5ce1b"
 from {username='slave'} at 127.0.0.1:35042
I20260709 12:20:53.636878 19092 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ab4d9071484d4dc0998db462cb01a10f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:53.637215 17759 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0"
dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
mode: GRACEFUL
new_leader_uuid: "0da169ff82674dc1947055ce43de68a3"
 from {username='slave'} at 127.0.0.1:52946
I20260709 12:20:53.639029 17759 raft_consensus.cc:606] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 LEADER]: Received request to transfer leadership to TS 0da169ff82674dc1947055ce43de68a3
W20260709 12:20:53.650128 17654 consensus_peers.cc:597] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 -> Peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Couldn't send request to peer 334a4e1f59944544b3b3ff36616b57da. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 10f96c39fba3496696f64385105bfeaa. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:53.650727 17831 raft_consensus.cc:493] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:20:53.650921 17838 raft_consensus.cc:1275] T 10f96c39fba3496696f64385105bfeaa P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:20:53.651098 17831 raft_consensus.cc:3060] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:53.651527 19086 tablet_bootstrap.cc:492] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:20:53.652966 19086 tablet_bootstrap.cc:492] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da: Bootstrap complete.
I20260709 12:20:53.653684 19086 ts_tablet_manager.cc:1403] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da: Time spent bootstrapping tablet: real 0.065s	user 0.015s	sys 0.014s
I20260709 12:20:53.655265 18944 consensus_queue.cc:1048] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:20:53.661408 19086 raft_consensus.cc:359] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:20:53.662510 19086 raft_consensus.cc:740] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Initialized, Role: LEARNER
I20260709 12:20:53.663290 19086 consensus_queue.cc:260] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:20:53.664001 17684 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9"
dest_uuid: "0da169ff82674dc1947055ce43de68a3"
mode: GRACEFUL
new_leader_uuid: "e063541542a94969898868bf6eb5ce1b"
 from {username='slave'} at 127.0.0.1:54540
I20260709 12:20:53.664602 17684 raft_consensus.cc:606] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 2 LEADER]: Received request to transfer leadership to TS e063541542a94969898868bf6eb5ce1b
I20260709 12:20:53.664335 17831 raft_consensus.cc:515] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:20:53.666661 16501 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 4
I20260709 12:20:53.668506 16501 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:20:53.669533 16501 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:20:53.675022 17759 raft_consensus.cc:1275] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:20:53.676530 19081 consensus_queue.cc:1048] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:20:53.677901 17831 leader_election.cc:290] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 2 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:20:53.677774 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:20:53.678462 17684 raft_consensus.cc:3060] T 28062a50142448aa97994888b4765920 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:53.682646 19092 tablet_copy_client.cc:806] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Starting download of 0 data blocks...
I20260709 12:20:53.666257 19086 ts_tablet_manager.cc:1434] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da: Time spent starting tablet: real 0.012s	user 0.009s	sys 0.001s
I20260709 12:20:53.683575 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:20:53.684162 17759 raft_consensus.cc:3055] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:20:53.684448 17759 raft_consensus.cc:740] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:20:53.685433 18944 raft_consensus.cc:2955] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } }
I20260709 12:20:53.686964 17759 consensus_queue.cc:260] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:20:53.688120 17774 tablet_copy_service.cc:342] P e2909cff1da44ad387f96a8ae3aa5afb: Request end of tablet copy session 334a4e1f59944544b3b3ff36616b57da-84421f094faa40f39b30b4c3d0737639 received from {username='slave'} at 127.0.0.1:36572
I20260709 12:20:53.688504 17759 raft_consensus.cc:3060] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:53.683236 19092 tablet_copy_client.cc:670] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Starting download of 1 WAL segments...
I20260709 12:20:53.691404 17684 raft_consensus.cc:2468] T 28062a50142448aa97994888b4765920 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 2.
I20260709 12:20:53.695405 17774 tablet_copy_service.cc:434] P e2909cff1da44ad387f96a8ae3aa5afb: ending tablet copy session 334a4e1f59944544b3b3ff36616b57da-84421f094faa40f39b30b4c3d0737639 on tablet 84421f094faa40f39b30b4c3d0737639 with peer 334a4e1f59944544b3b3ff36616b57da
I20260709 12:20:53.696584 17802 leader_election.cc:304] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:20:53.699352 19080 raft_consensus.cc:2804] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:20:53.701728 19080 raft_consensus.cc:697] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 2 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:20:53.703118 19092 tablet_copy_client.cc:538] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:20:53.704537 18364 raft_consensus.cc:1217] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:20:53.712365 19080 consensus_queue.cc:237] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:20:53.727658 17759 raft_consensus.cc:2468] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 2.
I20260709 12:20:53.728510 19092 tablet_bootstrap.cc:492] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da: Bootstrap starting.
I20260709 12:20:53.744722 17762 raft_consensus.cc:2955] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } }
I20260709 12:20:53.753577 17831 raft_consensus.cc:2955] T 10f96c39fba3496696f64385105bfeaa P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } }
I20260709 12:20:53.757681 18975 raft_consensus.cc:993] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b: : Instructing follower e2909cff1da44ad387f96a8ae3aa5afb to start an election
I20260709 12:20:53.758154 18975 raft_consensus.cc:1081] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 1 LEADER]: Signalling peer e2909cff1da44ad387f96a8ae3aa5afb to start an election
I20260709 12:20:53.759984 17760 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0"
dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
 from {username='slave'} at 127.0.0.1:38786
I20260709 12:20:53.760478 17760 raft_consensus.cc:493] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:20:53.760787 17760 raft_consensus.cc:3060] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:53.752825 17584 catalog_manager.cc:5697] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 reported cstate change: config changed from index -1 to 2, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New cstate: current_term: 1 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:53.785772 18944 raft_consensus.cc:993] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3: : Instructing follower e063541542a94969898868bf6eb5ce1b to start an election
I20260709 12:20:53.787591 18944 raft_consensus.cc:1081] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 2 LEADER]: Signalling peer e063541542a94969898868bf6eb5ce1b to start an election
I20260709 12:20:53.793269 17838 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9"
dest_uuid: "e063541542a94969898868bf6eb5ce1b"
 from {username='slave'} at 127.0.0.1:35040
I20260709 12:20:53.793826 17838 raft_consensus.cc:493] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:20:53.794188 17838 raft_consensus.cc:3060] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:20:53.797125 17760 raft_consensus.cc:515] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:20:53.801337 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:20:53.801882 17831 raft_consensus.cc:3055] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:20:53.802337 17831 raft_consensus.cc:740] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:20:53.803189 17831 consensus_queue.cc:260] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:20:53.804178 17831 raft_consensus.cc:3060] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:53.806023 19046 raft_consensus.cc:993] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb: : Instructing follower 0da169ff82674dc1947055ce43de68a3 to start an election
I20260709 12:20:53.806598 19046 raft_consensus.cc:1081] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 LEADER]: Signalling peer 0da169ff82674dc1947055ce43de68a3 to start an election
I20260709 12:20:53.815958 17684 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0"
dest_uuid: "0da169ff82674dc1947055ce43de68a3"
 from {username='slave'} at 127.0.0.1:37396
I20260709 12:20:53.816569 17684 raft_consensus.cc:493] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:20:53.817052 17684 raft_consensus.cc:3060] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:53.852157 17760 leader_election.cc:290] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 2 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:20:53.853147 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:20:53.853744 18364 raft_consensus.cc:3060] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:53.857553 17684 raft_consensus.cc:515] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:20:53.859483 17684 leader_election.cc:290] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 2 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:20:53.855170 17838 raft_consensus.cc:515] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:20:53.862094 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:20:53.862457 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:20:53.862684 18371 raft_consensus.cc:3060] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:53.863057 17759 raft_consensus.cc:3055] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:20:53.863371 17759 raft_consensus.cc:740] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:20:53.864275 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:20:53.864912 18367 raft_consensus.cc:3060] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:20:53.871025 17838 leader_election.cc:290] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 3 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:20:53.871706 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:20:53.872334 17682 raft_consensus.cc:3055] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:20:53.872628 17682 raft_consensus.cc:740] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:20:53.873492 17682 consensus_queue.cc:260] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:20:53.874509 17682 raft_consensus.cc:3060] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:20:53.875345 17584 catalog_manager.cc:5697] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b reported cstate change: term changed from 1 to 2, leader changed from e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66) to e063541542a94969898868bf6eb5ce1b (127.16.29.67). New cstate: current_term: 2 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:53.877146 18364 raft_consensus.cc:2468] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 2.
I20260709 12:20:53.877513 18367 raft_consensus.cc:2468] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 3.
I20260709 12:20:53.878552 17729 leader_election.cc:304] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:20:53.878816 17803 leader_election.cc:304] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:20:53.879446 19088 raft_consensus.cc:2804] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:20:53.879973 19088 raft_consensus.cc:697] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:20:53.871683 18371 raft_consensus.cc:2468] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 2.
I20260709 12:20:53.880852 19088 consensus_queue.cc:237] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:20:53.882067 17654 leader_election.cc:304] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:20:53.883069 19072 raft_consensus.cc:2804] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:20:53.883524 19072 raft_consensus.cc:697] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 2 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:20:53.851939 17831 raft_consensus.cc:2468] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 2.
I20260709 12:20:53.885217 19072 consensus_queue.cc:237] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:20:53.879937 19084 raft_consensus.cc:2804] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:20:53.891999 17682 raft_consensus.cc:2468] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 3.
I20260709 12:20:53.893740 19084 raft_consensus.cc:697] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 3 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:20:53.894832 19084 consensus_queue.cc:237] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:20:53.864260 17759 consensus_queue.cc:260] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:20:53.909825 17759 raft_consensus.cc:3060] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:53.909228 17583 catalog_manager.cc:5697] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb reported cstate change: term changed from 1 to 2, leader changed from e063541542a94969898868bf6eb5ce1b (127.16.29.67) to e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66). New cstate: current_term: 2 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:53.918176 17759 raft_consensus.cc:2468] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 2.
I20260709 12:20:53.927983 17584 catalog_manager.cc:5697] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 reported cstate change: term changed from 1 to 2, leader changed from e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66) to 0da169ff82674dc1947055ce43de68a3 (127.16.29.65). New cstate: current_term: 2 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:53.932160 19092 tablet_bootstrap.cc:492] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:20:53.933202 19092 tablet_bootstrap.cc:492] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da: Bootstrap complete.
I20260709 12:20:53.933919 19092 ts_tablet_manager.cc:1403] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da: Time spent bootstrapping tablet: real 0.222s	user 0.023s	sys 0.008s
I20260709 12:20:53.934913 17583 catalog_manager.cc:5697] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b reported cstate change: term changed from 2 to 3, leader changed from 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) to e063541542a94969898868bf6eb5ce1b (127.16.29.67). New cstate: current_term: 3 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:53.936858 19092 raft_consensus.cc:359] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:20:53.937831 19092 raft_consensus.cc:740] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Initialized, Role: LEARNER
I20260709 12:20:53.938580 19092 consensus_queue.cc:260] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:20:53.941063 19092 ts_tablet_manager.cc:1434] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da: Time spent starting tablet: real 0.007s	user 0.005s	sys 0.002s
I20260709 12:20:53.943694 17848 tablet_copy_service.cc:342] P e063541542a94969898868bf6eb5ce1b: Request end of tablet copy session 334a4e1f59944544b3b3ff36616b57da-ab4d9071484d4dc0998db462cb01a10f received from {username='slave'} at 127.0.0.1:37994
I20260709 12:20:53.944178 17848 tablet_copy_service.cc:434] P e063541542a94969898868bf6eb5ce1b: ending tablet copy session 334a4e1f59944544b3b3ff36616b57da-ab4d9071484d4dc0998db462cb01a10f on tablet ab4d9071484d4dc0998db462cb01a10f with peer 334a4e1f59944544b3b3ff36616b57da
I20260709 12:20:53.972594 18367 raft_consensus.cc:1217] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:20:54.090031 18988 raft_consensus.cc:1064] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb: attempting to promote NON_VOTER 334a4e1f59944544b3b3ff36616b57da to VOTER
I20260709 12:20:54.092804 18988 consensus_queue.cc:237] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:20:54.099824 18367 raft_consensus.cc:1275] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 1 LEARNER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:20:54.101406 17831 raft_consensus.cc:1275] T 84421f094faa40f39b30b4c3d0737639 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:20:54.102530 18988 consensus_queue.cc:1048] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:20:54.103695 17684 raft_consensus.cc:1275] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:20:54.103546 19046 consensus_queue.cc:1048] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:20:54.106045 19046 consensus_queue.cc:1048] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:20:54.118885 19092 ts_tablet_manager.cc:933] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da: Initiating tablet copy from peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071)
I20260709 12:20:54.124449 19092 tablet_copy_client.cc:323] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Beginning tablet copy session from remote peer at address 127.16.29.65:33071
I20260709 12:20:54.126855 17699 tablet_copy_service.cc:140] P 0da169ff82674dc1947055ce43de68a3: Received BeginTabletCopySession request for tablet 10f96c39fba3496696f64385105bfeaa from peer 334a4e1f59944544b3b3ff36616b57da ({username='slave'} at 127.0.0.1:44898)
I20260709 12:20:54.127600 17699 tablet_copy_service.cc:161] P 0da169ff82674dc1947055ce43de68a3: Beginning new tablet copy session on tablet 10f96c39fba3496696f64385105bfeaa from peer 334a4e1f59944544b3b3ff36616b57da at {username='slave'} at 127.0.0.1:44898: session id = 334a4e1f59944544b3b3ff36616b57da-10f96c39fba3496696f64385105bfeaa
I20260709 12:20:54.138851 17699 tablet_copy_source_session.cc:215] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:20:54.140791 18988 raft_consensus.cc:2955] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:54.144991 19092 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 10f96c39fba3496696f64385105bfeaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:54.145861 17682 raft_consensus.cc:2955] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:54.144215 17831 raft_consensus.cc:2955] T 84421f094faa40f39b30b4c3d0737639 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:54.162758 18367 raft_consensus.cc:2955] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
W20260709 12:20:54.167415 17803 consensus_peers.cc:597] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b -> Peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Couldn't send request to peer 334a4e1f59944544b3b3ff36616b57da. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 28062a50142448aa97994888b4765920. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:54.172890 17583 catalog_manager.cc:5697] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 reported cstate change: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:54.179136 19092 tablet_copy_client.cc:806] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Starting download of 0 data blocks...
I20260709 12:20:54.179827 19092 tablet_copy_client.cc:670] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Starting download of 1 WAL segments...
I20260709 12:20:54.184170 19092 tablet_copy_client.cc:538] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:20:54.194275 19092 tablet_bootstrap.cc:492] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da: Bootstrap starting.
I20260709 12:20:54.194562 17759 consensus_queue.cc:237] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:20:54.201593 17682 raft_consensus.cc:1275] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:20:54.203469 19087 consensus_queue.cc:1048] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:20:54.216693 18367 raft_consensus.cc:1275] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:20:54.220096 19088 consensus_queue.cc:1048] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:20:54.227144 19087 raft_consensus.cc:2955] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER e063541542a94969898868bf6eb5ce1b (127.16.29.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:54.240515 18364 raft_consensus.cc:2955] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER e063541542a94969898868bf6eb5ce1b (127.16.29.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:54.264222 17682 raft_consensus.cc:2955] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER e063541542a94969898868bf6eb5ce1b (127.16.29.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:54.269899 17571 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 84421f094faa40f39b30b4c3d0737639 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:20:54.278529 17587 catalog_manager.cc:5697] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da reported cstate change: config changed from index 3 to 4, VOTER e063541542a94969898868bf6eb5ce1b (127.16.29.67) evicted. New cstate: current_term: 1 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:54.286580 18364 raft_consensus.cc:1275] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:20:54.301676 19088 consensus_queue.cc:1048] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:20:54.303972 19092 tablet_bootstrap.cc:492] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:20:54.305053 19092 tablet_bootstrap.cc:492] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da: Bootstrap complete.
I20260709 12:20:54.305805 19092 ts_tablet_manager.cc:1403] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da: Time spent bootstrapping tablet: real 0.112s	user 0.037s	sys 0.039s
I20260709 12:20:54.309371 19092 raft_consensus.cc:359] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:20:54.310352 19092 raft_consensus.cc:740] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Initialized, Role: LEARNER
I20260709 12:20:54.311211 19092 consensus_queue.cc:260] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:20:54.313055 17821 tablet_service.cc:1558] Processing DeleteTablet for tablet 84421f094faa40f39b30b4c3d0737639 with delete_type TABLET_DATA_TOMBSTONED (TS e063541542a94969898868bf6eb5ce1b not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:35032
I20260709 12:20:54.316893 19110 tablet_replica.cc:333] T 84421f094faa40f39b30b4c3d0737639 P e063541542a94969898868bf6eb5ce1b: stopping tablet replica
I20260709 12:20:54.319010 19110 raft_consensus.cc:2243] T 84421f094faa40f39b30b4c3d0737639 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:54.322077 19110 raft_consensus.cc:2272] T 84421f094faa40f39b30b4c3d0737639 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:54.322880 19092 ts_tablet_manager.cc:1434] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da: Time spent starting tablet: real 0.017s	user 0.011s	sys 0.000s
I20260709 12:20:54.325774 17699 tablet_copy_service.cc:342] P 0da169ff82674dc1947055ce43de68a3: Request end of tablet copy session 334a4e1f59944544b3b3ff36616b57da-10f96c39fba3496696f64385105bfeaa received from {username='slave'} at 127.0.0.1:44898
I20260709 12:20:54.326094 19110 ts_tablet_manager.cc:1916] T 84421f094faa40f39b30b4c3d0737639 P e063541542a94969898868bf6eb5ce1b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:20:54.326277 17699 tablet_copy_service.cc:434] P 0da169ff82674dc1947055ce43de68a3: ending tablet copy session 334a4e1f59944544b3b3ff36616b57da-10f96c39fba3496696f64385105bfeaa on tablet 10f96c39fba3496696f64385105bfeaa with peer 334a4e1f59944544b3b3ff36616b57da
I20260709 12:20:54.332506 17831 raft_consensus.cc:1275] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:20:54.334029 19046 consensus_queue.cc:1048] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:20:54.349189 18367 raft_consensus.cc:1275] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260709 12:20:54.350559 19084 consensus_queue.cc:1048] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 12:20:54.357785 17684 raft_consensus.cc:1275] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260709 12:20:54.361671 19062 consensus_queue.cc:1048] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 12:20:54.363157 19110 ts_tablet_manager.cc:1929] T 84421f094faa40f39b30b4c3d0737639 P e063541542a94969898868bf6eb5ce1b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:20:54.365144 17684 raft_consensus.cc:1275] T 28062a50142448aa97994888b4765920 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260709 12:20:54.363698 19110 log.cc:1199] T 84421f094faa40f39b30b4c3d0737639 P e063541542a94969898868bf6eb5ce1b: Deleting WAL directory at /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599551912677-16501-0/minicluster-data/ts-2-root/wals/84421f094faa40f39b30b4c3d0737639
I20260709 12:20:54.373616 17571 catalog_manager.cc:5028] TS e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837): tablet 84421f094faa40f39b30b4c3d0737639 (table test-workload [id=ddc5b74250df4c65909b73b3742d615f]) successfully deleted
I20260709 12:20:54.378558 19080 consensus_queue.cc:1048] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:20:54.391856 17760 raft_consensus.cc:1275] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260709 12:20:54.393472 19077 consensus_queue.cc:1048] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:20:54.438410 17760 raft_consensus.cc:1275] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:20:54.443837 18944 consensus_queue.cc:1048] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.019s
I20260709 12:20:54.479846 18371 raft_consensus.cc:1275] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:20:54.482962 19072 consensus_queue.cc:1048] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:20:54.493714 19080 raft_consensus.cc:1064] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b: attempting to promote NON_VOTER 334a4e1f59944544b3b3ff36616b57da to VOTER
I20260709 12:20:54.496672 19080 consensus_queue.cc:237] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:20:54.508232 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 57852.9us
I20260709 12:20:54.507766 18364 raft_consensus.cc:1217] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:20:54.568611 18364 raft_consensus.cc:1275] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 1 LEARNER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:20:54.577809 19077 consensus_queue.cc:1048] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:20:54.597815 17759 raft_consensus.cc:1275] T ab4d9071484d4dc0998db462cb01a10f P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:20:54.602768 19077 consensus_queue.cc:1048] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:20:54.603653 17681 raft_consensus.cc:1275] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:20:54.605741 19084 consensus_queue.cc:1048] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:20:54.623755 19080 raft_consensus.cc:2955] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:54.629217 17760 raft_consensus.cc:2955] T ab4d9071484d4dc0998db462cb01a10f P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:54.634934 17681 raft_consensus.cc:2955] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:54.637910 18367 raft_consensus.cc:2955] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:54.649816 17587 catalog_manager.cc:5697] T ab4d9071484d4dc0998db462cb01a10f P e2909cff1da44ad387f96a8ae3aa5afb reported cstate change: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:54.652596 19092 ts_tablet_manager.cc:933] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da: Initiating tablet copy from peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:54.738603 19092 tablet_copy_client.cc:323] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Beginning tablet copy session from remote peer at address 127.16.29.67:36837
I20260709 12:20:54.742048 17848 tablet_copy_service.cc:140] P e063541542a94969898868bf6eb5ce1b: Received BeginTabletCopySession request for tablet 28062a50142448aa97994888b4765920 from peer 334a4e1f59944544b3b3ff36616b57da ({username='slave'} at 127.0.0.1:37994)
I20260709 12:20:54.742781 17848 tablet_copy_service.cc:161] P e063541542a94969898868bf6eb5ce1b: Beginning new tablet copy session on tablet 28062a50142448aa97994888b4765920 from peer 334a4e1f59944544b3b3ff36616b57da at {username='slave'} at 127.0.0.1:37994: session id = 334a4e1f59944544b3b3ff36616b57da-28062a50142448aa97994888b4765920
I20260709 12:20:54.917018 16501 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:20:54.922626 17848 tablet_copy_source_session.cc:215] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:20:54.996624 17837 consensus_queue.cc:237] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:20:55.019058 17681 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa"
dest_uuid: "0da169ff82674dc1947055ce43de68a3"
mode: GRACEFUL
new_leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
 from {username='slave'} at 127.0.0.1:54540
I20260709 12:20:55.019680 17681 raft_consensus.cc:606] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 1 LEADER]: Received request to transfer leadership to TS e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:20:55.033572 17838 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "28062a50142448aa97994888b4765920"
dest_uuid: "e063541542a94969898868bf6eb5ce1b"
mode: GRACEFUL
new_leader_uuid: "0da169ff82674dc1947055ce43de68a3"
 from {username='slave'} at 127.0.0.1:59820
I20260709 12:20:55.034360 17838 raft_consensus.cc:606] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 2 LEADER]: Received request to transfer leadership to TS 0da169ff82674dc1947055ce43de68a3
I20260709 12:20:55.037820 17760 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f"
dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
mode: GRACEFUL
new_leader_uuid: "e063541542a94969898868bf6eb5ce1b"
 from {username='slave'} at 127.0.0.1:52946
I20260709 12:20:55.038431 17760 raft_consensus.cc:606] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 1 LEADER]: Received request to transfer leadership to TS e063541542a94969898868bf6eb5ce1b
I20260709 12:20:55.040861 19092 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 28062a50142448aa97994888b4765920. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:55.041389 17838 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f"
dest_uuid: "e063541542a94969898868bf6eb5ce1b"
mode: GRACEFUL
new_leader_uuid: "0da169ff82674dc1947055ce43de68a3"
 from {username='slave'} at 127.0.0.1:59820
I20260709 12:20:55.041942 17838 raft_consensus.cc:606] T ea7da467817a4ba88aa5797c3f8f493f P e063541542a94969898868bf6eb5ce1b [term 3 LEADER]: Received request to transfer leadership to TS 0da169ff82674dc1947055ce43de68a3
I20260709 12:20:55.043272 16501 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 4
I20260709 12:20:55.045300 16501 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:20:55.046093 16501 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:20:55.055488 18371 raft_consensus.cc:1275] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:20:55.060913 19077 consensus_queue.cc:1048] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:20:55.064572 17682 raft_consensus.cc:1275] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:20:55.083746 19120 raft_consensus.cc:2955] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:55.096776 19080 raft_consensus.cc:993] T ea7da467817a4ba88aa5797c3f8f493f P e063541542a94969898868bf6eb5ce1b: : Instructing follower 0da169ff82674dc1947055ce43de68a3 to start an election
I20260709 12:20:55.098172 19080 consensus_queue.cc:1048] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:20:55.099375 19084 raft_consensus.cc:1081] T ea7da467817a4ba88aa5797c3f8f493f P e063541542a94969898868bf6eb5ce1b [term 3 LEADER]: Signalling peer 0da169ff82674dc1947055ce43de68a3 to start an election
I20260709 12:20:55.100018 19077 raft_consensus.cc:993] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b: : Instructing follower 0da169ff82674dc1947055ce43de68a3 to start an election
I20260709 12:20:55.100505 19077 raft_consensus.cc:1081] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 2 LEADER]: Signalling peer 0da169ff82674dc1947055ce43de68a3 to start an election
I20260709 12:20:55.111313 18371 raft_consensus.cc:2955] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:55.129567 17585 catalog_manager.cc:5697] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da reported cstate change: config changed from index 3 to 4, VOTER e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66) evicted. New cstate: current_term: 1 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:55.146047 17753 tablet_service.cc:1558] Processing DeleteTablet for tablet ab4d9071484d4dc0998db462cb01a10f with delete_type TABLET_DATA_TOMBSTONED (TS e2909cff1da44ad387f96a8ae3aa5afb not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:38774
I20260709 12:20:55.169622 19137 tablet_replica.cc:333] T ab4d9071484d4dc0998db462cb01a10f P e2909cff1da44ad387f96a8ae3aa5afb: stopping tablet replica
W20260709 12:20:55.171046 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 77897.3us
I20260709 12:20:55.171339 18944 raft_consensus.cc:1064] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3: attempting to promote NON_VOTER 334a4e1f59944544b3b3ff36616b57da to VOTER
W20260709 12:20:55.171825 18944 raft_consensus.cc:1068] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3: Unable to promote non-voter 334a4e1f59944544b3b3ff36616b57da: Service unavailable: leader transfer in progress
I20260709 12:20:55.172465 17682 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "28062a50142448aa97994888b4765920"
dest_uuid: "0da169ff82674dc1947055ce43de68a3"
 from {username='slave'} at 127.0.0.1:37402
I20260709 12:20:55.173020 17682 raft_consensus.cc:493] T 28062a50142448aa97994888b4765920 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:20:55.173183 17681 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f"
dest_uuid: "0da169ff82674dc1947055ce43de68a3"
 from {username='slave'} at 127.0.0.1:37402
I20260709 12:20:55.173385 17682 raft_consensus.cc:3060] T 28062a50142448aa97994888b4765920 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:20:55.173753 17681 raft_consensus.cc:493] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:20:55.174124 17681 raft_consensus.cc:3060] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:20:55.178686 17682 raft_consensus.cc:515] T 28062a50142448aa97994888b4765920 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:20:55.179050 17681 raft_consensus.cc:515] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:55.180456 17571 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet ab4d9071484d4dc0998db462cb01a10f with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:20:55.180490 17682 leader_election.cc:290] T 28062a50142448aa97994888b4765920 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 3 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:55.180846 17681 leader_election.cc:290] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 4 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:55.198434 17682 raft_consensus.cc:2955] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:55.209631 19137 raft_consensus.cc:2243] T ab4d9071484d4dc0998db462cb01a10f P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:55.209631 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: true dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:20:55.210227 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:20:55.210670 17831 raft_consensus.cc:3055] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:20:55.210832 17838 raft_consensus.cc:3055] T ea7da467817a4ba88aa5797c3f8f493f P e063541542a94969898868bf6eb5ce1b [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:20:55.200420 19092 tablet_copy_client.cc:806] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Starting download of 0 data blocks...
I20260709 12:20:55.207154 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: true dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:20:55.211705 17831 raft_consensus.cc:740] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 2 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:20:55.212200 17763 raft_consensus.cc:3060] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:20:55.213498 19092 tablet_copy_client.cc:670] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Starting download of 1 WAL segments...
I20260709 12:20:55.213059 17831 consensus_queue.cc:260] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:20:55.218628 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:20:55.219252 17757 raft_consensus.cc:3060] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:20:55.221602 17831 raft_consensus.cc:3060] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:20:55.223691 19137 raft_consensus.cc:2272] T ab4d9071484d4dc0998db462cb01a10f P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:55.232514 17831 raft_consensus.cc:2468] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 3.
I20260709 12:20:55.211769 17838 raft_consensus.cc:740] T ea7da467817a4ba88aa5797c3f8f493f P e063541542a94969898868bf6eb5ce1b [term 3 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:20:55.234454 19092 tablet_copy_client.cc:538] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:20:55.234730 19072 raft_consensus.cc:993] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3: : Instructing follower e2909cff1da44ad387f96a8ae3aa5afb to start an election
I20260709 12:20:55.234874 17838 consensus_queue.cc:260] T ea7da467817a4ba88aa5797c3f8f493f P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:55.235342 19132 raft_consensus.cc:1081] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 1 LEADER]: Signalling peer e2909cff1da44ad387f96a8ae3aa5afb to start an election
I20260709 12:20:55.236311 17838 raft_consensus.cc:3060] T ea7da467817a4ba88aa5797c3f8f493f P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:20:55.238006 17652 leader_election.cc:304] T 28062a50142448aa97994888b4765920 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:20:55.238741 18944 raft_consensus.cc:2804] T 28062a50142448aa97994888b4765920 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:20:55.239420 17760 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa"
dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
 from {username='slave'} at 127.0.0.1:38776
I20260709 12:20:55.239571 18944 raft_consensus.cc:697] T 28062a50142448aa97994888b4765920 P 0da169ff82674dc1947055ce43de68a3 [term 3 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:20:55.239970 17760 raft_consensus.cc:493] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:20:55.241124 17760 raft_consensus.cc:3060] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:55.241518 18944 consensus_queue.cc:237] T 28062a50142448aa97994888b4765920 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:20:55.242326 17757 raft_consensus.cc:2468] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 4.
I20260709 12:20:55.243345 17763 raft_consensus.cc:2468] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 3.
I20260709 12:20:55.246384 19137 ts_tablet_manager.cc:1916] T ab4d9071484d4dc0998db462cb01a10f P e2909cff1da44ad387f96a8ae3aa5afb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:20:55.248477 17760 raft_consensus.cc:515] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:20:55.251543 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:20:55.252185 17681 raft_consensus.cc:3055] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:20:55.252236 17652 leader_election.cc:304] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:20:55.252563 17681 raft_consensus.cc:740] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:20:55.253755 17760 leader_election.cc:290] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 2 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:55.253358 17681 consensus_queue.cc:260] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:20:55.254709 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:20:55.255105 19132 raft_consensus.cc:2804] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:20:55.255642 17831 raft_consensus.cc:3060] T 10f96c39fba3496696f64385105bfeaa P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:55.255745 19132 raft_consensus.cc:697] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 4 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:20:55.255105 17681 raft_consensus.cc:3060] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:55.257370 19132 consensus_queue.cc:237] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:55.260351 17838 raft_consensus.cc:2468] T ea7da467817a4ba88aa5797c3f8f493f P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 4.
I20260709 12:20:55.242738 19092 tablet_bootstrap.cc:492] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da: Bootstrap starting.
I20260709 12:20:55.274392 19137 ts_tablet_manager.cc:1929] T ab4d9071484d4dc0998db462cb01a10f P e2909cff1da44ad387f96a8ae3aa5afb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:20:55.274983 19137 log.cc:1199] T ab4d9071484d4dc0998db462cb01a10f P e2909cff1da44ad387f96a8ae3aa5afb: Deleting WAL directory at /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599551912677-16501-0/minicluster-data/ts-1-root/wals/ab4d9071484d4dc0998db462cb01a10f
I20260709 12:20:55.276685 17571 catalog_manager.cc:5028] TS e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): tablet ab4d9071484d4dc0998db462cb01a10f (table test-workload [id=ddc5b74250df4c65909b73b3742d615f]) successfully deleted
I20260709 12:20:55.277390 17681 raft_consensus.cc:2468] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 2.
I20260709 12:20:55.285423 17728 leader_election.cc:304] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:20:55.283048 17585 catalog_manager.cc:5697] T 28062a50142448aa97994888b4765920 P 0da169ff82674dc1947055ce43de68a3 reported cstate change: term changed from 2 to 3, leader changed from e063541542a94969898868bf6eb5ce1b (127.16.29.67) to 0da169ff82674dc1947055ce43de68a3 (127.16.29.65). New cstate: current_term: 3 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:55.286327 19129 raft_consensus.cc:2804] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:20:55.286808 19129 raft_consensus.cc:697] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 2 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:20:55.287722 19129 consensus_queue.cc:237] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:20:55.276887 17831 raft_consensus.cc:2468] T 10f96c39fba3496696f64385105bfeaa P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 2.
I20260709 12:20:55.298882 17682 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "28062a50142448aa97994888b4765920"
dest_uuid: "0da169ff82674dc1947055ce43de68a3"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:37384
I20260709 12:20:55.299425 17682 raft_consensus.cc:606] T 28062a50142448aa97994888b4765920 P 0da169ff82674dc1947055ce43de68a3 [term 3 LEADER]: Received request to transfer leadership
I20260709 12:20:55.303483 17682 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "28062a50142448aa97994888b4765920"
dest_uuid: "0da169ff82674dc1947055ce43de68a3"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:37384
I20260709 12:20:55.304207 17682 raft_consensus.cc:606] T 28062a50142448aa97994888b4765920 P 0da169ff82674dc1947055ce43de68a3 [term 3 LEADER]: Received request to transfer leadership
W20260709 12:20:55.305734 17572 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: leadership transfer for 28062a50142448aa97994888b4765920 already in progress [suppressed 3 similar messages]
I20260709 12:20:55.306838 17585 catalog_manager.cc:5697] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 reported cstate change: term changed from 3 to 4, leader changed from e063541542a94969898868bf6eb5ce1b (127.16.29.67) to 0da169ff82674dc1947055ce43de68a3 (127.16.29.65). New cstate: current_term: 4 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:55.314626 19092 tablet_bootstrap.cc:492] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:20:55.315611 19092 tablet_bootstrap.cc:492] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da: Bootstrap complete.
I20260709 12:20:55.316368 19092 ts_tablet_manager.cc:1403] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da: Time spent bootstrapping tablet: real 0.074s	user 0.032s	sys 0.004s
I20260709 12:20:55.319000 19092 raft_consensus.cc:359] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:20:55.319592 19092 raft_consensus.cc:740] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Initialized, Role: LEARNER
I20260709 12:20:55.320119 19092 consensus_queue.cc:260] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:20:55.322153 19092 ts_tablet_manager.cc:1434] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da: Time spent starting tablet: real 0.005s	user 0.003s	sys 0.000s
I20260709 12:20:55.322667 17682 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "28062a50142448aa97994888b4765920"
dest_uuid: "0da169ff82674dc1947055ce43de68a3"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:37384
I20260709 12:20:55.323314 17682 raft_consensus.cc:606] T 28062a50142448aa97994888b4765920 P 0da169ff82674dc1947055ce43de68a3 [term 3 LEADER]: Received request to transfer leadership
I20260709 12:20:55.324839 17848 tablet_copy_service.cc:342] P e063541542a94969898868bf6eb5ce1b: Request end of tablet copy session 334a4e1f59944544b3b3ff36616b57da-28062a50142448aa97994888b4765920 received from {username='slave'} at 127.0.0.1:37994
I20260709 12:20:55.325255 17848 tablet_copy_service.cc:434] P e063541542a94969898868bf6eb5ce1b: ending tablet copy session 334a4e1f59944544b3b3ff36616b57da-28062a50142448aa97994888b4765920 on tablet 28062a50142448aa97994888b4765920 with peer 334a4e1f59944544b3b3ff36616b57da
W20260709 12:20:55.328266 17635 auto_rebalancer.cc:893] Removing replace marker failed: Service unavailable: leader transfer in progress
W20260709 12:20:55.328756 17635 auto_rebalancer.cc:901] Could not move replica: Remote error: Service unavailable: leadership transfer for 28062a50142448aa97994888b4765920 already in progress
W20260709 12:20:55.329030 17635 auto_rebalancer.cc:330] scheduled replica move failed to complete: Remote error: Service unavailable: leadership transfer for 28062a50142448aa97994888b4765920 already in progress
I20260709 12:20:55.335323 17584 catalog_manager.cc:5697] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb reported cstate change: term changed from 1 to 2, leader changed from 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) to e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66). New cstate: current_term: 2 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:55.393164 19129 raft_consensus.cc:993] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb: : Instructing follower e063541542a94969898868bf6eb5ce1b to start an election
I20260709 12:20:55.393859 19087 raft_consensus.cc:1081] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 1 LEADER]: Signalling peer e063541542a94969898868bf6eb5ce1b to start an election
I20260709 12:20:55.395380 17838 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f"
dest_uuid: "e063541542a94969898868bf6eb5ce1b"
 from {username='slave'} at 127.0.0.1:35042
I20260709 12:20:55.395821 17838 raft_consensus.cc:493] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:20:55.396166 17838 raft_consensus.cc:3060] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:55.403139 17838 raft_consensus.cc:515] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:55.404889 17838 leader_election.cc:290] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 2 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:20:55.405432 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:20:55.405988 17681 raft_consensus.cc:3060] T 9160d9f98b8c406db2771d9ea2e0995f P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:55.407675 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:20:55.408262 17760 raft_consensus.cc:3055] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:20:55.408553 17760 raft_consensus.cc:740] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:20:55.409240 17760 consensus_queue.cc:260] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:55.410230 17760 raft_consensus.cc:3060] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:55.418639 17681 raft_consensus.cc:2468] T 9160d9f98b8c406db2771d9ea2e0995f P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 2.
I20260709 12:20:55.420233 17802 leader_election.cc:304] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:20:55.421025 19077 raft_consensus.cc:2804] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:20:55.421514 19077 raft_consensus.cc:697] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 2 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:20:55.422583 17760 raft_consensus.cc:2468] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 2.
I20260709 12:20:55.422375 19077 consensus_queue.cc:237] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:55.445305 17584 catalog_manager.cc:5697] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b reported cstate change: term changed from 1 to 2, leader changed from e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66) to e063541542a94969898868bf6eb5ce1b (127.16.29.67). New cstate: current_term: 2 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: HEALTHY } } }
I20260709 12:20:55.674966 17757 raft_consensus.cc:1275] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:20:55.676296 19072 consensus_queue.cc:1048] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:20:55.706928 17831 raft_consensus.cc:1275] T ea7da467817a4ba88aa5797c3f8f493f P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:20:55.708348 19072 consensus_queue.cc:1048] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:20:55.717273 17831 raft_consensus.cc:1275] T 10f96c39fba3496696f64385105bfeaa P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260709 12:20:55.718606 19088 consensus_queue.cc:1048] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:20:55.746790 17681 raft_consensus.cc:1275] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260709 12:20:55.748967 19129 consensus_queue.cc:1048] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:20:55.749492 18367 raft_consensus.cc:3060] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 1 LEARNER]: Advancing to term 2
I20260709 12:20:55.751730 17757 raft_consensus.cc:1275] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260709 12:20:55.753479 19072 consensus_queue.cc:1048] T 28062a50142448aa97994888b4765920 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:20:55.758041 18367 raft_consensus.cc:1275] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 2 LEARNER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260709 12:20:55.759855 19129 consensus_queue.cc:1048] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:20:55.795197 18944 raft_consensus.cc:993] T 28062a50142448aa97994888b4765920 P 0da169ff82674dc1947055ce43de68a3: : Instructing follower e2909cff1da44ad387f96a8ae3aa5afb to start an election
I20260709 12:20:55.795830 18944 raft_consensus.cc:1081] T 28062a50142448aa97994888b4765920 P 0da169ff82674dc1947055ce43de68a3 [term 3 LEADER]: Signalling peer e2909cff1da44ad387f96a8ae3aa5afb to start an election
I20260709 12:20:55.810567 17760 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "28062a50142448aa97994888b4765920"
dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
 from {username='slave'} at 127.0.0.1:38776
I20260709 12:20:55.811280 17760 raft_consensus.cc:493] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:20:55.811653 17760 raft_consensus.cc:3060] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:20:55.817968 17831 raft_consensus.cc:1275] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260709 12:20:55.822834 17757 raft_consensus.cc:1275] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:20:55.825629 19077 consensus_queue.cc:1048] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:20:55.828266 18944 consensus_queue.cc:1048] T 28062a50142448aa97994888b4765920 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:20:55.833282 18367 raft_consensus.cc:3060] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 2 LEARNER]: Advancing to term 3
I20260709 12:20:55.838093 19088 raft_consensus.cc:1064] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb: attempting to promote NON_VOTER 334a4e1f59944544b3b3ff36616b57da to VOTER
I20260709 12:20:55.841949 19088 consensus_queue.cc:237] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 2, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:20:55.859678 17681 raft_consensus.cc:1275] T 9160d9f98b8c406db2771d9ea2e0995f P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:20:55.861982 17760 raft_consensus.cc:515] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:20:55.864375 19080 consensus_queue.cc:1048] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:20:55.865447 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:20:55.883443 17760 leader_election.cc:290] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 4 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:55.884605 17836 raft_consensus.cc:1275] T 10f96c39fba3496696f64385105bfeaa P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260709 12:20:55.886153 19156 consensus_queue.cc:1048] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20260709 12:20:55.907991 18406 debug-util.cc:398] Leaking SignalData structure 0x7b08002fe060 after lost signal to thread 16504
W20260709 12:20:55.909171 18406 debug-util.cc:398] Leaking SignalData structure 0x7b08002fe740 after lost signal to thread 17621
W20260709 12:20:55.910506 18406 debug-util.cc:398] Leaking SignalData structure 0x7b0800368ca0 after lost signal to thread 17713
W20260709 12:20:55.914687 18406 debug-util.cc:398] Leaking SignalData structure 0x7b0800214fe0 after lost signal to thread 17788
W20260709 12:20:55.916509 18406 debug-util.cc:398] Leaking SignalData structure 0x7b08002904c0 after lost signal to thread 17862
W20260709 12:20:55.919191 18406 debug-util.cc:398] Leaking SignalData structure 0x7b0800257a00 after lost signal to thread 18409
I20260709 12:20:55.966509 18367 raft_consensus.cc:1275] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 3 LEARNER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260709 12:20:55.968307 19132 consensus_queue.cc:1048] T 28062a50142448aa97994888b4765920 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:20:56.000348 19132 raft_consensus.cc:1064] T 28062a50142448aa97994888b4765920 P 0da169ff82674dc1947055ce43de68a3: attempting to promote NON_VOTER 334a4e1f59944544b3b3ff36616b57da to VOTER
W20260709 12:20:56.000801 19132 raft_consensus.cc:1068] T 28062a50142448aa97994888b4765920 P 0da169ff82674dc1947055ce43de68a3: Unable to promote non-voter 334a4e1f59944544b3b3ff36616b57da: Service unavailable: leader transfer in progress
W20260709 12:20:56.011178 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 154592us
I20260709 12:20:56.013348 18371 raft_consensus.cc:1275] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 2 LEARNER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260709 12:20:56.078378 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:20:56.078974 17682 raft_consensus.cc:3055] T 28062a50142448aa97994888b4765920 P 0da169ff82674dc1947055ce43de68a3 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:20:56.079252 17682 raft_consensus.cc:740] T 28062a50142448aa97994888b4765920 P 0da169ff82674dc1947055ce43de68a3 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:20:56.080292 17682 consensus_queue.cc:260] T 28062a50142448aa97994888b4765920 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:20:56.081833 17682 raft_consensus.cc:3060] T 28062a50142448aa97994888b4765920 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:20:56.088660 17682 raft_consensus.cc:2468] T 28062a50142448aa97994888b4765920 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 4.
I20260709 12:20:56.092543 17681 raft_consensus.cc:1275] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260709 12:20:56.118396 19157 consensus_queue.cc:1048] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:20:56.121165 19088 consensus_queue.cc:1048] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:20:56.123689 17728 leader_election.cc:304] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:20:56.124539 19088 raft_consensus.cc:2804] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:20:56.125029 19088 raft_consensus.cc:697] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:20:56.125877 19088 consensus_queue.cc:237] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:20:56.135216 16501 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:20:56.154829 19157 raft_consensus.cc:2955] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 2 to 4, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:56.170697 17682 raft_consensus.cc:2955] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:56.175163 18367 raft_consensus.cc:2955] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:56.205612 17838 raft_consensus.cc:2955] T 10f96c39fba3496696f64385105bfeaa P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
W20260709 12:20:56.934448 17804 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.618s	user 0.005s	sys 0.002s
W20260709 12:20:56.934861 17804 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.618s	user 0.006s	sys 0.002s
I20260709 12:20:57.118295 19179 raft_consensus.cc:493] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:20:57.119314 19179 raft_consensus.cc:515] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:20:57.122445 19179 leader_election.cc:290] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:20:57.193066 19161 raft_consensus.cc:493] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:20:57.193979 19161 raft_consensus.cc:515] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:20:57.196525 19161 leader_election.cc:290] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:20:57.196756 19175 raft_consensus.cc:493] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:20:57.197794 19175 raft_consensus.cc:515] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:57.200965 19175 leader_election.cc:290] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:57.211928 19161 raft_consensus.cc:493] T ea7da467817a4ba88aa5797c3f8f493f P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:20:57.212733 19161 raft_consensus.cc:515] T ea7da467817a4ba88aa5797c3f8f493f P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:57.215965 19161 leader_election.cc:290] T ea7da467817a4ba88aa5797c3f8f493f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:20:57.241526 17836 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "0f940f29438f475294357324452abe76"
dest_uuid: "e063541542a94969898868bf6eb5ce1b"
mode: GRACEFUL
new_leader_uuid: "334a4e1f59944544b3b3ff36616b57da"
 from {username='slave'} at 127.0.0.1:59820
I20260709 12:20:57.242197 17836 raft_consensus.cc:606] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 1 LEADER]: Received request to transfer leadership to TS 334a4e1f59944544b3b3ff36616b57da
I20260709 12:20:57.255580 19172 raft_consensus.cc:493] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:20:57.257805 19172 raft_consensus.cc:515] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:57.260202 19177 raft_consensus.cc:493] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:20:57.267115 19177 raft_consensus.cc:515] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:57.269512 19177 leader_election.cc:290] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:57.270083 19177 raft_consensus.cc:493] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:20:57.270649 19177 raft_consensus.cc:515] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:57.271549 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:20:57.275204 17682 raft_consensus.cc:1275] T 28062a50142448aa97994888b4765920 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20260709 12:20:57.276883 18374 raft_consensus.cc:3060] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 3 LEARNER]: Advancing to term 4
I20260709 12:20:57.282671 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:20:57.285147 19170 raft_consensus.cc:493] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:20:57.298841 19170 raft_consensus.cc:515] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:57.301054 19170 leader_election.cc:290] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:57.290337 17684 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3"
dest_uuid: "0da169ff82674dc1947055ce43de68a3"
mode: GRACEFUL
new_leader_uuid: "e063541542a94969898868bf6eb5ce1b"
 from {username='slave'} at 127.0.0.1:54540
I20260709 12:20:57.301666 17684 raft_consensus.cc:606] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 1 LEADER]: Received request to transfer leadership to TS e063541542a94969898868bf6eb5ce1b
I20260709 12:20:57.294204 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:20:57.303268 17760 raft_consensus.cc:2468] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 4.
I20260709 12:20:57.308580 19170 consensus_queue.cc:1048] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:20:57.310235 19177 leader_election.cc:290] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:57.296078 17836 raft_consensus.cc:3060] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:20:57.294883 19172 leader_election.cc:290] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:57.314582 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:20:57.326200 19172 raft_consensus.cc:493] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:20:57.326810 19172 raft_consensus.cc:515] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:57.315479 17760 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639"
dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
mode: GRACEFUL
new_leader_uuid: "0da169ff82674dc1947055ce43de68a3"
 from {username='slave'} at 127.0.0.1:52946
I20260709 12:20:57.327855 17760 raft_consensus.cc:606] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 LEADER]: Received request to transfer leadership to TS 0da169ff82674dc1947055ce43de68a3
I20260709 12:20:57.334874 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:20:57.335554 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:20:57.364146 19134 raft_consensus.cc:493] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:20:57.369498 19134 raft_consensus.cc:515] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:20:57.370543 17584 catalog_manager.cc:5697] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb reported cstate change: config changed from index 2 to 4, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 12:20:57.330116 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 50944.8us
I20260709 12:20:57.328962 19172 leader_election.cc:290] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:57.374358 17688 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "ac984176358d453a8800cc9e0e304061"
dest_uuid: "0da169ff82674dc1947055ce43de68a3"
mode: GRACEFUL
new_leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
 from {username='slave'} at 127.0.0.1:54540
I20260709 12:20:57.374910 17688 raft_consensus.cc:606] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 3 LEADER]: Received request to transfer leadership to TS e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:20:57.378855 16501 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 4
I20260709 12:20:57.380511 16501 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:20:57.381208 16501 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:20:57.381718 17584 catalog_manager.cc:5697] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb reported cstate change: term changed from 3 to 4, leader changed from 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) to e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66). New cstate: current_term: 4 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:57.371608 19134 leader_election.cc:290] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:57.401479 19080 raft_consensus.cc:993] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b: : Instructing follower 334a4e1f59944544b3b3ff36616b57da to start an election
I20260709 12:20:57.402099 19160 raft_consensus.cc:1081] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 1 LEADER]: Signalling peer 334a4e1f59944544b3b3ff36616b57da to start an election
I20260709 12:20:57.405012 19168 raft_consensus.cc:493] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:20:57.405593 19168 raft_consensus.cc:515] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:57.407706 19185 raft_consensus.cc:493] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:20:57.408303 19185 raft_consensus.cc:515] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:57.410549 19185 leader_election.cc:290] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:57.411080 19169 raft_consensus.cc:493] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:20:57.411576 19169 raft_consensus.cc:515] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:57.413646 19169 leader_election.cc:290] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:57.413883 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:20:57.417922 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:20:57.419291 19178 raft_consensus.cc:993] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb: : Instructing follower 0da169ff82674dc1947055ce43de68a3 to start an election
I20260709 12:20:57.419831 19178 raft_consensus.cc:1081] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 LEADER]: Signalling peer 0da169ff82674dc1947055ce43de68a3 to start an election
I20260709 12:20:57.407810 19168 leader_election.cc:290] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:57.421425 17684 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639"
dest_uuid: "0da169ff82674dc1947055ce43de68a3"
 from {username='slave'} at 127.0.0.1:37396
I20260709 12:20:57.422134 17684 raft_consensus.cc:493] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:20:57.422586 17684 raft_consensus.cc:3060] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:57.424541 17836 raft_consensus.cc:1275] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20260709 12:20:57.425108 19171 raft_consensus.cc:493] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:57.425933 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:20:57.426590 19185 raft_consensus.cc:493] T 9160d9f98b8c406db2771d9ea2e0995f P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:20:57.432286 19185 raft_consensus.cc:515] T 9160d9f98b8c406db2771d9ea2e0995f P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:57.434907 19185 leader_election.cc:290] T 9160d9f98b8c406db2771d9ea2e0995f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:57.425683 19171 raft_consensus.cc:515] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:20:57.437067 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:20:57.431754 17684 raft_consensus.cc:515] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:20:57.450331 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:20:57.451313 18362 raft_consensus.cc:3060] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:57.456135 17684 leader_election.cc:290] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 2 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:20:57.438845 17727 leader_election.cc:304] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:20:57.427765 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:20:57.440948 19171 leader_election.cc:290] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:20:57.442121 19185 raft_consensus.cc:493] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:20:57.465150 19185 raft_consensus.cc:515] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:20:57.468334 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:20:57.470265 19185 leader_election.cc:290] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:20:57.461711 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 48880us
I20260709 12:20:57.470712 19168 raft_consensus.cc:2749] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:20:57.470166 17727 leader_election.cc:304] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:20:57.461237 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:20:57.480558 19175 consensus_queue.cc:1048] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:20:57.429838 18374 raft_consensus.cc:1275] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 4 LEARNER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20260709 12:20:57.480185 19180 raft_consensus.cc:2749] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:20:57.484153 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:20:57.485385 19168 consensus_queue.cc:1048] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:20:57.489228 18333 leader_election.cc:304] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:20:57.496436 17801 leader_election.cc:304] T ea7da467817a4ba88aa5797c3f8f493f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:20:57.497889 19080 raft_consensus.cc:2804] T ea7da467817a4ba88aa5797c3f8f493f P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:20:57.498426 19080 raft_consensus.cc:493] T ea7da467817a4ba88aa5797c3f8f493f P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:20:57.498842 19080 raft_consensus.cc:3060] T ea7da467817a4ba88aa5797c3f8f493f P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:20:57.499289 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:20:57.501415 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:20:57.501843 19078 raft_consensus.cc:2749] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:20:57.504006 17727 leader_election.cc:304] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:20:57.504128 18367 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "0f940f29438f475294357324452abe76"
dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
 from {username='slave'} at 127.0.0.1:59902
I20260709 12:20:57.505260 18367 raft_consensus.cc:493] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:20:57.505467 17727 leader_election.cc:304] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:20:57.505625 18367 raft_consensus.cc:3060] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:57.511778 19180 raft_consensus.cc:2749] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:20:57.512566 19180 raft_consensus.cc:2749] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:20:57.518580 18362 raft_consensus.cc:2468] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 2.
I20260709 12:20:57.519547 19177 raft_consensus.cc:993] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3: : Instructing follower e063541542a94969898868bf6eb5ce1b to start an election
I20260709 12:20:57.520006 17654 leader_election.cc:304] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:20:57.520128 19177 raft_consensus.cc:1081] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 1 LEADER]: Signalling peer e063541542a94969898868bf6eb5ce1b to start an election
I20260709 12:20:57.520887 19172 raft_consensus.cc:2804] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:20:57.521445 19172 raft_consensus.cc:697] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 2 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:20:57.522451 19172 consensus_queue.cc:237] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:20:57.523594 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:20:57.530664 18362 raft_consensus.cc:1240] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 1. Current term is 2. Ops: []
I20260709 12:20:57.532838 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:20:57.533398 19180 consensus_queue.cc:1059] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:20:57.535651 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:20:57.538236 18367 raft_consensus.cc:515] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:20:57.538930 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:20:57.539989 18371 raft_consensus.cc:1240] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 1. Current term is 2. Ops: []
I20260709 12:20:57.543795 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:20:57.551743 19175 raft_consensus.cc:3055] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:20:57.552522 19175 raft_consensus.cc:740] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:20:57.553479 19175 consensus_queue.cc:260] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:20:57.555136 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:20:57.555176 19175 raft_consensus.cc:3060] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:57.556389 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:20:57.558164 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:20:57.564167 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:20:57.567243 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:20:57.567994 17758 raft_consensus.cc:3060] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:57.570247 18367 leader_election.cc:290] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 2 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:57.570596 19184 consensus_queue.cc:1059] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:20:57.571718 19184 raft_consensus.cc:3055] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:20:57.572135 19184 raft_consensus.cc:740] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:20:57.573097 19184 consensus_queue.cc:260] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:20:57.573845 19080 raft_consensus.cc:515] T ea7da467817a4ba88aa5797c3f8f493f P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:57.574271 19184 raft_consensus.cc:3060] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:57.581295 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:20:57.583612 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:20:57.577953 17802 leader_election.cc:304] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:20:57.589457 17758 raft_consensus.cc:2468] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 2.
I20260709 12:20:57.591048 17802 leader_election.cc:304] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:20:57.592988 18333 leader_election.cc:304] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:20:57.596845 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:20:57.599817 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:20:57.601048 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:20:57.602005 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:20:57.603296 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:20:57.617614 17837 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3"
dest_uuid: "e063541542a94969898868bf6eb5ce1b"
 from {username='slave'} at 127.0.0.1:35040
I20260709 12:20:57.618386 17837 raft_consensus.cc:493] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:20:57.618801 17837 raft_consensus.cc:3060] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Advancing to term 2
W20260709 12:20:57.616035 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 48849.5us
I20260709 12:20:57.622366 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:20:57.614905 17652 leader_election.cc:304] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:20:57.618676 19080 leader_election.cc:290] T ea7da467817a4ba88aa5797c3f8f493f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 5 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:20:57.635583 17763 raft_consensus.cc:2468] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 2.
I20260709 12:20:57.624291 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:20:57.631397 19185 raft_consensus.cc:2749] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:20:57.640197 17652 leader_election.cc:304] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:20:57.642797 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 51942.7us
I20260709 12:20:57.643939 17652 leader_election.cc:304] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:20:57.644264 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:20:57.638103 19196 raft_consensus.cc:2749] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:20:57.649832 19167 raft_consensus.cc:2749] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:20:57.646991 19172 raft_consensus.cc:2749] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:20:57.648916 17652 leader_election.cc:304] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:20:57.651947 19181 raft_consensus.cc:993] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3: : Instructing follower e2909cff1da44ad387f96a8ae3aa5afb to start an election
I20260709 12:20:57.646622 19190 raft_consensus.cc:2804] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:20:57.669714 19183 raft_consensus.cc:1081] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 3 LEADER]: Signalling peer e2909cff1da44ad387f96a8ae3aa5afb to start an election
I20260709 12:20:57.669875 19190 raft_consensus.cc:697] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 2 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:20:57.653934 19168 raft_consensus.cc:1064] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb: attempting to promote NON_VOTER 334a4e1f59944544b3b3ff36616b57da to VOTER
I20260709 12:20:57.671571 19190 consensus_queue.cc:237] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:20:57.676227 19168 consensus_queue.cc:237] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 4, Current term: 4, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:20:57.669137 17837 raft_consensus.cc:515] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:57.680342 19183 raft_consensus.cc:2749] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:20:57.681017 17837 leader_election.cc:290] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 2 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:20:57.687397 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:20:57.691251 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:20:57.692003 17758 raft_consensus.cc:3060] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:57.698829 18374 raft_consensus.cc:1275] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 4 LEARNER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:20:57.700498 19178 consensus_queue.cc:1048] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 12:20:57.702159 17760 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "ac984176358d453a8800cc9e0e304061"
dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
 from {username='slave'} at 127.0.0.1:38776
I20260709 12:20:57.702711 17760 raft_consensus.cc:493] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:20:57.703085 17760 raft_consensus.cc:3060] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:20:57.713999 17758 raft_consensus.cc:2468] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 2.
I20260709 12:20:57.715188 17801 leader_election.cc:304] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:20:57.722759 17760 raft_consensus.cc:515] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:20:57.726825 19161 raft_consensus.cc:2804] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:20:57.780225 17837 raft_consensus.cc:2468] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 2.
I20260709 12:20:57.798203 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:20:57.798609 17760 leader_election.cc:290] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 4 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:20:57.798822 17684 raft_consensus.cc:3055] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:20:57.799147 17684 raft_consensus.cc:740] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:20:57.800729 17652 leader_election.cc:304] T 9160d9f98b8c406db2771d9ea2e0995f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:20:57.796701 19161 raft_consensus.cc:697] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 2 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:20:57.841032 17652 leader_election.cc:304] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:20:57.805755 17684 consensus_queue.cc:260] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:20:57.850837 17652 leader_election.cc:304] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:20:57.857227 17684 raft_consensus.cc:3060] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:20:57.845039 19172 raft_consensus.cc:2749] T 9160d9f98b8c406db2771d9ea2e0995f P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:20:57.866789 17652 leader_election.cc:304] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
W20260709 12:20:57.867414 17655 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:57.845363 19080 raft_consensus.cc:493] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:20:57.872840 17760 raft_consensus.cc:1240] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 1. Current term is 2. Ops: []
I20260709 12:20:57.867892 19185 raft_consensus.cc:2749] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:20:57.802253 19161 consensus_queue.cc:237] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:57.875588 19167 raft_consensus.cc:2749] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:20:57.898429 19171 raft_consensus.cc:2749] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:20:57.880729 19080 raft_consensus.cc:515] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:20:57.923393 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 144642us
I20260709 12:20:57.931743 17684 raft_consensus.cc:2468] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 4.
W20260709 12:20:57.934314 17859 debug-util.cc:398] Leaking SignalData structure 0x7b080005a220 after lost signal to thread 16504
I20260709 12:20:57.934464 19164 consensus_queue.cc:1059] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.010s
I20260709 12:20:57.940137 17728 leader_election.cc:304] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:20:57.935369 17859 debug-util.cc:398] Leaking SignalData structure 0x7b0800173d60 after lost signal to thread 17621
I20260709 12:20:57.952248 19166 raft_consensus.cc:2749] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:20:57.970201 17837 raft_consensus.cc:1275] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:20:57.975409 17802 leader_election.cc:304] T ea7da467817a4ba88aa5797c3f8f493f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:20:57.961355 19169 raft_consensus.cc:2804] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Leader election won for term 4
W20260709 12:20:57.994359 19202 log.cc:927] Time spent T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.059s	user 0.001s	sys 0.000s
I20260709 12:20:57.976573 19184 raft_consensus.cc:493] T 10f96c39fba3496696f64385105bfeaa P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:20:58.004690 19080 leader_election.cc:290] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:20:58.013083 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:20:58.014011 17688 raft_consensus.cc:3055] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:20:58.014442 17688 raft_consensus.cc:740] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:20:58.015442 17688 consensus_queue.cc:260] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:57.999727 19169 raft_consensus.cc:697] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:20:58.002334 19184 raft_consensus.cc:515] T 10f96c39fba3496696f64385105bfeaa P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:20:58.018069 19169 consensus_queue.cc:237] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:20:58.020084 19208 raft_consensus.cc:2749] T ea7da467817a4ba88aa5797c3f8f493f P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20260709 12:20:57.999902 17859 debug-util.cc:398] Leaking SignalData structure 0x7b0800308da0 after lost signal to thread 17713
W20260709 12:20:58.026345 17859 debug-util.cc:398] Leaking SignalData structure 0x7b080027d8c0 after lost signal to thread 17788
W20260709 12:20:58.026755 17859 debug-util.cc:398] Leaking SignalData structure 0x7b08000bac60 after lost signal to thread 17789
I20260709 12:20:58.028796 17838 raft_consensus.cc:1240] T ea7da467817a4ba88aa5797c3f8f493f P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 4. Current term is 5. Ops: []
I20260709 12:20:58.041841 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:20:58.043756 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:20:58.068719 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 99978.1us
W20260709 12:20:58.074405 17859 debug-util.cc:398] Leaking SignalData structure 0x7b08003563a0 after lost signal to thread 17862
I20260709 12:20:58.086805 19219 consensus_queue.cc:1048] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.014s
W20260709 12:20:58.089658 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 95786.7us
W20260709 12:20:58.085268 17859 debug-util.cc:398] Leaking SignalData structure 0x7b0800356380 after lost signal to thread 17863
I20260709 12:20:58.096977 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:20:58.101329 18367 raft_consensus.cc:3060] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:20:58.103969 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:20:58.089608 19184 leader_election.cc:290] T 10f96c39fba3496696f64385105bfeaa P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:20:58.107838 17803 leader_election.cc:304] T 10f96c39fba3496696f64385105bfeaa P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:20:58.146492 19200 raft_consensus.cc:2749] T 10f96c39fba3496696f64385105bfeaa P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260709 12:20:58.151541 17859 debug-util.cc:398] Leaking SignalData structure 0x7b08002b6940 after lost signal to thread 18409
W20260709 12:20:58.152036 17859 debug-util.cc:398] Leaking SignalData structure 0x7b08000a5840 after lost signal to thread 18410
W20260709 12:20:58.256275 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 127231us
W20260709 12:20:58.258188 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 126484us
W20260709 12:20:58.258284 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 131527us
I20260709 12:20:58.261960 18367 raft_consensus.cc:2468] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 4.
W20260709 12:20:58.304371 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 195180us
W20260709 12:20:58.305030 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 194230us
I20260709 12:20:58.310359 17836 raft_consensus.cc:1275] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:20:58.346282 19175 raft_consensus.cc:2955] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 LEADER]: Committing config change with OpId 4.6: config changed from index 2 to 6, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:58.367496 19225 consensus_queue.cc:1048] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:20:58.369596 17838 raft_consensus.cc:2955] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Committing config change with OpId 4.6: config changed from index 2 to 6, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:58.380600 17684 raft_consensus.cc:1275] T 28062a50142448aa97994888b4765920 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:20:58.399523 18371 raft_consensus.cc:1275] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:20:58.407688 19168 consensus_queue.cc:1048] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:20:58.413578 17758 raft_consensus.cc:1275] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:20:58.427320 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:20:58.429049 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:20:58.418584 19222 consensus_queue.cc:1048] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.004s
I20260709 12:20:58.422746 19164 consensus_queue.cc:1048] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.013s
I20260709 12:20:58.433666 18371 raft_consensus.cc:2955] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Committing config change with OpId 4.6: config changed from index 2 to 6, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:58.444648 19210 consensus_queue.cc:1059] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.005s
I20260709 12:20:58.446975 17757 raft_consensus.cc:1275] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:20:58.447239 19210 raft_consensus.cc:3055] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:20:58.447610 19210 raft_consensus.cc:740] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:20:58.458803 17831 raft_consensus.cc:1240] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 2 LEADER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 1. Current term is 2. Ops: []
I20260709 12:20:58.501910 17801 leader_election.cc:304] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:20:58.504371 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 46641.3us
I20260709 12:20:58.506223 17688 raft_consensus.cc:3060] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:58.515569 19216 raft_consensus.cc:2749] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:20:58.505597 17583 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35958 (request call id 330) took 1169 ms. Trace:
I20260709 12:20:58.519170 17583 rpcz_store.cc:276] 0709 12:20:57.318628 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:57.318813 (+   185us) service_pool.cc:224] Handling call
0709 12:20:58.488422 (+1169609us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":58751,"reported_tablets":3,"spinlock_wait_cycles":1792}
I20260709 12:20:58.505715 17584 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35962 (request call id 292) took 2279 ms. Trace:
I20260709 12:20:58.508325 17587 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:37172 (request call id 192) took 2289 ms. Trace:
I20260709 12:20:58.540995 17587 rpcz_store.cc:276] 0709 12:20:56.219281 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:56.219441 (+   160us) service_pool.cc:224] Handling call
0709 12:20:58.508264 (+2288823us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5319,"reported_tablets":1}
W20260709 12:20:58.639353 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 130601us
I20260709 12:20:58.642838 19177 consensus_queue.cc:1048] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
W20260709 12:20:58.646005 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 135308us
W20260709 12:20:58.646178 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 137076us
I20260709 12:20:58.647776 17585 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35966 (request call id 272) took 2330 ms. Trace:
I20260709 12:20:58.649840 17585 rpcz_store.cc:276] 0709 12:20:56.316998 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:57.272612 (+955614us) service_pool.cc:224] Handling call
0709 12:20:58.647689 (+1375077us) inbound_call.cc:177] Queueing success response
Metrics: {"reported_tablets":1}
I20260709 12:20:58.528149 19210 consensus_queue.cc:260] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:58.660305 17688 raft_consensus.cc:2468] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 2.
I20260709 12:20:58.660867 19210 raft_consensus.cc:3060] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:20:58.673379 19212 consensus_queue.cc:1175] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" responder_term: 4 status { last_received { term: 4 index: 4 } last_committed_idx: 4 last_received_current_leader { term: 4 index: 4 } }
I20260709 12:20:58.733003 17681 raft_consensus.cc:2955] T 28062a50142448aa97994888b4765920 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Committing config change with OpId 4.6: config changed from index 2 to 6, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:58.524883 17584 rpcz_store.cc:276] 0709 12:20:56.209455 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:56.209838 (+   383us) service_pool.cc:224] Handling call
0709 12:20:58.489010 (+2279172us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:20:57.525838 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:20:57.526428 (+   590us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:20:57.526480 (+    52us) write_op.cc:435] Acquired schema lock
0709 12:20:57.526502 (+    22us) tablet.cc:662] Decoding operations
0709 12:20:57.526931 (+   429us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:20:57.527005 (+    74us) write_op.cc:641] Partition lock acquired for write op
0709 12:20:57.527042 (+    37us) tablet.cc:685] Acquiring locks for 2 operations
0709 12:20:57.527311 (+   269us) tablet.cc:701] Row locks acquired
0709 12:20:57.527338 (+    27us) write_op.cc:260] PREPARE: finished
0709 12:20:57.578679 (+ 51341us) write_op.cc:270] Start()
0709 12:20:57.615222 (+ 36543us) write_op.cc:276] Timestamp: P: 1783599657527456 usec, L: 0
0709 12:20:57.616285 (+  1063us) op_driver.cc:348] REPLICATION: starting
0709 12:20:57.646426 (+ 30141us) log.cc:844] Serialized 1037 byte log entry
0709 12:20:57.942994 (+296568us) op_driver.cc:464] REPLICATION: finished
0709 12:20:57.998281 (+ 55287us) write_op.cc:301] APPLY: starting
0709 12:20:57.998376 (+    95us) tablet.cc:1367] starting BulkCheckPresence
0709 12:20:57.998522 (+   146us) tablet.cc:1370] finished BulkCheckPresence
0709 12:20:58.001951 (+  3429us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:20:58.017465 (+ 15514us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:20:58.017638 (+   173us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0709 12:20:58.017685 (+    47us) write_op.cc:312] APPLY: finished
0709 12:20:58.017896 (+   211us) log.cc:844] Serialized 29 byte log entry
0709 12:20:58.018072 (+   176us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:20:58.018156 (+    84us) write_op.cc:454] Released schema lock
0709 12:20:58.018195 (+    39us) write_op.cc:341] FINISH: Updating metrics
0709 12:20:58.018545 (+   350us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:20:58.018572 (+    27us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":276,"reported_tablets":7,"spinlock_wait_cycles":1294208,"child_traces":[["op",{"apply.queue_time_us":54982,"apply.run_cpu_time_us":2213,"apply.run_wall_time_us":20263,"num_ops":2,"prepare.queue_time_us":140856,"prepare.run_cpu_time_us":14228,"prepare.run_wall_time_us":170215,"raft.queue_time_us":199862,"raft.run_cpu_time_us":16033,"raft.run_wall_time_us":63514,"replication_time_us":308376,"thread_start_us":363764,"threads_started":4,"wal-append.queue_time_us":48890}]]}
I20260709 12:20:58.791251 17685 raft_consensus.cc:1275] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:20:58.792876 19206 consensus_queue.cc:1048] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:20:58.783541 17763 consensus_queue.cc:237] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:20:58.804224 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 120745us
W20260709 12:20:58.830580 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 73680.4us
I20260709 12:20:58.849609 17683 raft_consensus.cc:1275] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:20:58.865748 18367 raft_consensus.cc:1275] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:20:58.870738 19227 consensus_queue.cc:1048] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:20:58.904409 17584 catalog_manager.cc:5697] T 28062a50142448aa97994888b4765920 P 0da169ff82674dc1947055ce43de68a3 reported cstate change: config changed from index 2 to 6, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New cstate: current_term: 4 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:58.907126 17584 catalog_manager.cc:5697] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 reported cstate change: term changed from 1 to 2, leader changed from e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66) to 0da169ff82674dc1947055ce43de68a3 (127.16.29.65). New cstate: current_term: 2 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:58.924480 17688 raft_consensus.cc:1275] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
W20260709 12:20:59.009541 17730 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:20:59.054541 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 125303us
I20260709 12:20:59.054629 19168 consensus_queue.cc:1048] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:20:59.061367 18367 raft_consensus.cc:1275] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:20:59.067130 19234 consensus_queue.cc:1048] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
W20260709 12:20:59.085588 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 159374us
W20260709 12:20:59.085795 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 159927us
I20260709 12:20:59.146522 17760 raft_consensus.cc:1275] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:20:59.131786 19165 consensus_queue.cc:1048] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.042s
I20260709 12:20:59.179498 19208 consensus_queue.cc:1048] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:20:59.243217 19235 raft_consensus.cc:2955] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER e063541542a94969898868bf6eb5ce1b (127.16.29.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:59.246392 18374 raft_consensus.cc:2955] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER e063541542a94969898868bf6eb5ce1b (127.16.29.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:59.267830 17571 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 10f96c39fba3496696f64385105bfeaa with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
W20260709 12:20:59.272081 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 49785.7us
I20260709 12:20:59.332594 17683 raft_consensus.cc:2955] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER e063541542a94969898868bf6eb5ce1b (127.16.29.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:59.361636 17587 catalog_manager.cc:5697] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b reported cstate change: term changed from 1 to 2, leader changed from 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) to e063541542a94969898868bf6eb5ce1b (127.16.29.67). New cstate: current_term: 2 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: HEALTHY } } }
I20260709 12:20:59.363458 17587 catalog_manager.cc:5697] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b reported cstate change: term changed from 1 to 2, leader changed from e063541542a94969898868bf6eb5ce1b (127.16.29.67) to 334a4e1f59944544b3b3ff36616b57da (127.16.29.68). New cstate: current_term: 2 leader_uuid: "334a4e1f59944544b3b3ff36616b57da" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:59.386461 17584 catalog_manager.cc:5697] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 reported cstate change: term changed from 3 to 4, leader changed from 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) to e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66). New cstate: current_term: 4 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:59.387681 17584 catalog_manager.cc:5697] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 reported cstate change: config changed from index 4 to 5, VOTER e063541542a94969898868bf6eb5ce1b (127.16.29.67) evicted. New cstate: current_term: 2 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:59.406852 17821 tablet_service.cc:1558] Processing DeleteTablet for tablet 10f96c39fba3496696f64385105bfeaa with delete_type TABLET_DATA_TOMBSTONED (TS e063541542a94969898868bf6eb5ce1b not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:35032
I20260709 12:20:59.413314 19260 tablet_replica.cc:333] T 10f96c39fba3496696f64385105bfeaa P e063541542a94969898868bf6eb5ce1b: stopping tablet replica
I20260709 12:20:59.414980 17763 consensus_queue.cc:237] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:20:59.421196 19260 raft_consensus.cc:2243] T 10f96c39fba3496696f64385105bfeaa P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:59.422039 19260 raft_consensus.cc:2272] T 10f96c39fba3496696f64385105bfeaa P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:59.424877 19260 ts_tablet_manager.cc:1916] T 10f96c39fba3496696f64385105bfeaa P e063541542a94969898868bf6eb5ce1b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:20:59.454717 16501 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:20:59.467710 18374 raft_consensus.cc:1275] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20260709 12:20:59.469647 19235 consensus_queue.cc:1048] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 12:20:59.480409 19219 raft_consensus.cc:2955] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 LEADER]: Committing config change with OpId 4.7: config changed from index 6 to 7, VOTER 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:59.482133 18367 raft_consensus.cc:2955] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Committing config change with OpId 4.7: config changed from index 6 to 7, VOTER 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:59.496366 17831 raft_consensus.cc:1275] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20260709 12:20:59.503329 19175 consensus_queue.cc:1048] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 12:20:59.528729 17834 raft_consensus.cc:2955] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Committing config change with OpId 4.7: config changed from index 6 to 7, VOTER 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:59.544929 17571 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 28062a50142448aa97994888b4765920 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:20:59.560673 17587 catalog_manager.cc:5697] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b reported cstate change: config changed from index 6 to 7, VOTER 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) evicted. New cstate: current_term: 4 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:20:59.574383 19260 ts_tablet_manager.cc:1929] T 10f96c39fba3496696f64385105bfeaa P e063541542a94969898868bf6eb5ce1b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20260709 12:20:59.588202 19260 log.cc:1199] T 10f96c39fba3496696f64385105bfeaa P e063541542a94969898868bf6eb5ce1b: Deleting WAL directory at /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599551912677-16501-0/minicluster-data/ts-2-root/wals/10f96c39fba3496696f64385105bfeaa
I20260709 12:20:59.590888 17571 catalog_manager.cc:5028] TS e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837): tablet 10f96c39fba3496696f64385105bfeaa (table test-workload [id=ddc5b74250df4c65909b73b3742d615f]) successfully deleted
I20260709 12:20:59.592927 17676 tablet_service.cc:1558] Processing DeleteTablet for tablet 28062a50142448aa97994888b4765920 with delete_type TABLET_DATA_TOMBSTONED (TS 0da169ff82674dc1947055ce43de68a3 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:37384
I20260709 12:20:59.661541 19263 tablet_replica.cc:333] T 28062a50142448aa97994888b4765920 P 0da169ff82674dc1947055ce43de68a3: stopping tablet replica
I20260709 12:20:59.684299 19263 raft_consensus.cc:2243] T 28062a50142448aa97994888b4765920 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:59.685155 19263 raft_consensus.cc:2272] T 28062a50142448aa97994888b4765920 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:59.688020 19181 raft_consensus.cc:493] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:59.688537 19181 raft_consensus.cc:515] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:59.690917 19181 leader_election.cc:290] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:59.694177 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:20:59.694612 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:20:59.694800 17835 raft_consensus.cc:2468] T ea7da467817a4ba88aa5797c3f8f493f P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 5.
I20260709 12:20:59.696201 17652 leader_election.cc:304] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:20:59.696967 19212 raft_consensus.cc:2804] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:20:59.697340 19212 raft_consensus.cc:493] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:59.697645 19212 raft_consensus.cc:3060] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:20:59.707417 19212 raft_consensus.cc:515] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:59.711634 19212 leader_election.cc:290] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 6 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:59.713687 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:20:59.715407 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:20:59.718043 17835 raft_consensus.cc:3060] T ea7da467817a4ba88aa5797c3f8f493f P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:20:59.716672 19263 ts_tablet_manager.cc:1916] T 28062a50142448aa97994888b4765920 P 0da169ff82674dc1947055ce43de68a3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:20:59.730409 17835 raft_consensus.cc:2468] T ea7da467817a4ba88aa5797c3f8f493f P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 6.
I20260709 12:20:59.731685 17652 leader_election.cc:304] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:20:59.732506 19185 raft_consensus.cc:2804] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:20:59.733690 19185 raft_consensus.cc:697] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 6 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:20:59.734598 19185 consensus_queue.cc:237] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:59.743477 19263 ts_tablet_manager.cc:1929] T 28062a50142448aa97994888b4765920 P 0da169ff82674dc1947055ce43de68a3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 4.6
I20260709 12:20:59.744055 19263 log.cc:1199] T 28062a50142448aa97994888b4765920 P 0da169ff82674dc1947055ce43de68a3: Deleting WAL directory at /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599551912677-16501-0/minicluster-data/ts-0-root/wals/28062a50142448aa97994888b4765920
I20260709 12:20:59.746088 17572 catalog_manager.cc:5028] TS 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): tablet 28062a50142448aa97994888b4765920 (table test-workload [id=ddc5b74250df4c65909b73b3742d615f]) successfully deleted
I20260709 12:20:59.791251 17763 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "22fa38faae58422e9cecc99973cab688"
dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
mode: GRACEFUL
new_leader_uuid: "0da169ff82674dc1947055ce43de68a3"
 from {username='slave'} at 127.0.0.1:52946
I20260709 12:20:59.791900 17763 raft_consensus.cc:606] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 LEADER]: Received request to transfer leadership to TS 0da169ff82674dc1947055ce43de68a3
I20260709 12:20:59.795181 17831 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442"
dest_uuid: "e063541542a94969898868bf6eb5ce1b"
mode: GRACEFUL
new_leader_uuid: "0da169ff82674dc1947055ce43de68a3"
 from {username='slave'} at 127.0.0.1:59820
I20260709 12:20:59.795761 17831 raft_consensus.cc:606] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 1 LEADER]: Received request to transfer leadership to TS 0da169ff82674dc1947055ce43de68a3
I20260709 12:20:59.801192 17585 catalog_manager.cc:5697] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:59.804857 17759 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad"
dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
mode: GRACEFUL
new_leader_uuid: "e063541542a94969898868bf6eb5ce1b"
 from {username='slave'} at 127.0.0.1:52946
I20260709 12:20:59.805475 17759 raft_consensus.cc:606] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 2 LEADER]: Received request to transfer leadership to TS e063541542a94969898868bf6eb5ce1b
I20260709 12:20:59.806993 16501 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 3
I20260709 12:20:59.808820 16501 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:20:59.809644 16501 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:20:59.941174 19209 raft_consensus.cc:993] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b: : Instructing follower 0da169ff82674dc1947055ce43de68a3 to start an election
I20260709 12:20:59.941815 19206 raft_consensus.cc:1081] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 1 LEADER]: Signalling peer 0da169ff82674dc1947055ce43de68a3 to start an election
I20260709 12:20:59.945824 17684 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442"
dest_uuid: "0da169ff82674dc1947055ce43de68a3"
 from {username='slave'} at 127.0.0.1:37402
I20260709 12:20:59.946358 17684 raft_consensus.cc:493] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:20:59.946702 17684 raft_consensus.cc:3060] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:59.953622 19219 raft_consensus.cc:493] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:20:59.954185 19219 raft_consensus.cc:515] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:59.956429 19219 leader_election.cc:290] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:20:59.957986 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:20:59.963271 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:20:59.964178 17831 raft_consensus.cc:2368] T ea7da467817a4ba88aa5797c3f8f493f P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for earlier term 5. Current term is 6.
I20260709 12:20:59.968420 17727 leader_election.cc:400] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 5 pre-election: Vote denied by peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837) with higher term. Message: Invalid argument: T ea7da467817a4ba88aa5797c3f8f493f P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for earlier term 5. Current term is 6.
I20260709 12:20:59.968760 17727 leader_election.cc:403] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 5 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:20:59.970183 19175 raft_consensus.cc:3060] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Advancing to term 6
I20260709 12:20:59.982578 19175 raft_consensus.cc:2749] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: Vote denied by peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837) with higher term. Message: Invalid argument: T ea7da467817a4ba88aa5797c3f8f493f P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for earlier term 5. Current term is 6.
I20260709 12:20:59.983665 17684 raft_consensus.cc:515] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:20:59.987262 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:20:59.996532 17759 raft_consensus.cc:3060] T 7cb3768d215e409fb9f5baf18d124442 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:00.006964 17684 leader_election.cc:290] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 2 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:00.011322 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:00.012018 17831 raft_consensus.cc:3055] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:21:00.012332 17831 raft_consensus.cc:740] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:21:00.022847 17759 raft_consensus.cc:2468] T 7cb3768d215e409fb9f5baf18d124442 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 2.
I20260709 12:21:00.024353 17652 leader_election.cc:304] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:21:00.025218 19212 raft_consensus.cc:2804] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:21:00.025668 19212 raft_consensus.cc:697] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 2 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:21:00.026599 19212 consensus_queue.cc:237] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:00.013970 17831 consensus_queue.cc:260] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:00.033000 17831 raft_consensus.cc:3060] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:00.046501 17831 raft_consensus.cc:2468] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 2.
I20260709 12:21:00.125892 17585 catalog_manager.cc:5697] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 reported cstate change: term changed from 1 to 2, leader changed from e063541542a94969898868bf6eb5ce1b (127.16.29.67) to 0da169ff82674dc1947055ce43de68a3 (127.16.29.65). New cstate: current_term: 2 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:00.158244 19235 raft_consensus.cc:993] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb: : Instructing follower e063541542a94969898868bf6eb5ce1b to start an election
I20260709 12:21:00.158730 19168 raft_consensus.cc:1081] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 2 LEADER]: Signalling peer e063541542a94969898868bf6eb5ce1b to start an election
I20260709 12:21:00.160498 17831 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad"
dest_uuid: "e063541542a94969898868bf6eb5ce1b"
 from {username='slave'} at 127.0.0.1:35042
I20260709 12:21:00.161018 17831 raft_consensus.cc:493] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:21:00.161342 17831 raft_consensus.cc:3060] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:00.166888 17831 raft_consensus.cc:515] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:00.170512 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:00.170722 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:00.171259 17683 raft_consensus.cc:3060] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:00.171489 17759 raft_consensus.cc:3055] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:21:00.171928 17759 raft_consensus.cc:740] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 2 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:21:00.173185 17759 consensus_queue.cc:260] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:00.181551 17759 raft_consensus.cc:3060] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:00.184105 17683 raft_consensus.cc:2468] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 3.
I20260709 12:21:00.185436 17802 leader_election.cc:304] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:21:00.187103 19249 raft_consensus.cc:2804] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:21:00.187574 19249 raft_consensus.cc:697] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 3 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:21:00.188776 19249 consensus_queue.cc:237] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:00.200016 17759 raft_consensus.cc:2468] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 3.
I20260709 12:21:00.201745 17831 leader_election.cc:290] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 3 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:00.205703 17585 catalog_manager.cc:5697] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b reported cstate change: term changed from 2 to 3, leader changed from e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66) to e063541542a94969898868bf6eb5ce1b (127.16.29.67). New cstate: current_term: 3 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: HEALTHY } } }
I20260709 12:21:00.237051 17759 raft_consensus.cc:1275] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260709 12:21:00.238592 19212 consensus_queue.cc:1048] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:21:00.257977 19235 raft_consensus.cc:993] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb: : Instructing follower 0da169ff82674dc1947055ce43de68a3 to start an election
I20260709 12:21:00.258473 19235 raft_consensus.cc:1081] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 LEADER]: Signalling peer 0da169ff82674dc1947055ce43de68a3 to start an election
I20260709 12:21:00.261165 17684 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "22fa38faae58422e9cecc99973cab688"
dest_uuid: "0da169ff82674dc1947055ce43de68a3"
 from {username='slave'} at 127.0.0.1:37396
I20260709 12:21:00.261708 17684 raft_consensus.cc:493] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:21:00.262104 17684 raft_consensus.cc:3060] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:00.266271 17831 raft_consensus.cc:1275] T ea7da467817a4ba88aa5797c3f8f493f P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260709 12:21:00.267849 19214 consensus_queue.cc:1048] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:21:00.335116 17684 raft_consensus.cc:515] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:00.337862 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:00.338465 17759 raft_consensus.cc:3055] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:21:00.338798 17759 raft_consensus.cc:740] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:21:00.339644 17759 consensus_queue.cc:260] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:00.341127 17759 raft_consensus.cc:3060] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:00.342444 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:00.343132 17831 raft_consensus.cc:3060] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:00.342280 17684 leader_election.cc:290] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 2 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:00.349390 17831 raft_consensus.cc:2468] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 2.
I20260709 12:21:00.350469 17652 leader_election.cc:304] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:21:00.351243 19212 raft_consensus.cc:2804] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:21:00.351956 19212 raft_consensus.cc:697] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 2 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:21:00.352998 19212 consensus_queue.cc:237] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:00.359848 17759 raft_consensus.cc:2468] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 2.
I20260709 12:21:00.371322 17585 catalog_manager.cc:5697] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 reported cstate change: term changed from 1 to 2, leader changed from e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66) to 0da169ff82674dc1947055ce43de68a3 (127.16.29.65). New cstate: current_term: 2 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:00.432365 17759 raft_consensus.cc:1275] T 7cb3768d215e409fb9f5baf18d124442 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:21:00.439594 19212 consensus_queue.cc:1048] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:21:00.509523 17835 raft_consensus.cc:1275] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:21:00.524061 19212 consensus_queue.cc:1048] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:21:00.565776 17834 consensus_queue.cc:237] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:21:00.581864 17760 raft_consensus.cc:1275] T 45397041e3ac43c8901c5cb03f27c7b9 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:21:00.583559 17683 raft_consensus.cc:1275] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:21:00.583645 19209 consensus_queue.cc:1048] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260709 12:21:00.584944 17803 consensus_peers.cc:597] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b -> Peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Couldn't send request to peer 334a4e1f59944544b3b3ff36616b57da. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 45397041e3ac43c8901c5cb03f27c7b9. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:21:00.586365 19249 consensus_queue.cc:1048] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:21:00.707093 19249 raft_consensus.cc:2955] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } }
I20260709 12:21:00.727797 17760 raft_consensus.cc:2955] T 45397041e3ac43c8901c5cb03f27c7b9 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } }
I20260709 12:21:00.736295 17759 raft_consensus.cc:1275] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:21:00.745599 19281 consensus_queue.cc:1048] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:21:00.752153 17684 raft_consensus.cc:2955] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } }
I20260709 12:21:00.758816 17687 raft_consensus.cc:1275] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:21:00.761025 17835 raft_consensus.cc:1275] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:21:00.772673 17835 consensus_queue.cc:237] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:21:00.769541 19214 consensus_queue.cc:1048] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.008s
I20260709 12:21:00.786110 17585 catalog_manager.cc:5697] T 45397041e3ac43c8901c5cb03f27c7b9 P e2909cff1da44ad387f96a8ae3aa5afb reported cstate change: config changed from index -1 to 2, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New cstate: current_term: 1 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } }
I20260709 12:21:00.814462 19249 consensus_queue.cc:1048] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:21:00.936036 16501 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
W20260709 12:21:00.997026 19282 log.cc:927] Time spent T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.058s	user 0.001s	sys 0.000s
I20260709 12:21:01.114066 17684 raft_consensus.cc:1275] T 9160d9f98b8c406db2771d9ea2e0995f P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260709 12:21:01.117367 19249 consensus_queue.cc:1048] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20260709 12:21:01.121764 17803 consensus_peers.cc:597] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b -> Peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Couldn't send request to peer 334a4e1f59944544b3b3ff36616b57da. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9160d9f98b8c406db2771d9ea2e0995f. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:21:01.120441 17687 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8"
dest_uuid: "0da169ff82674dc1947055ce43de68a3"
mode: GRACEFUL
new_leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
 from {username='slave'} at 127.0.0.1:54540
I20260709 12:21:01.123626 17687 raft_consensus.cc:606] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 1 LEADER]: Received request to transfer leadership to TS e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:21:01.140918 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 325367us
I20260709 12:21:01.141249 17834 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9"
dest_uuid: "e063541542a94969898868bf6eb5ce1b"
mode: GRACEFUL
new_leader_uuid: "0da169ff82674dc1947055ce43de68a3"
 from {username='slave'} at 127.0.0.1:59820
I20260709 12:21:01.141798 17834 raft_consensus.cc:606] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 3 LEADER]: Received request to transfer leadership to TS 0da169ff82674dc1947055ce43de68a3
W20260709 12:21:01.144028 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 56443.4us
I20260709 12:21:01.169085 17682 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b"
dest_uuid: "0da169ff82674dc1947055ce43de68a3"
mode: GRACEFUL
new_leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
 from {username='slave'} at 127.0.0.1:54540
I20260709 12:21:01.169942 17682 raft_consensus.cc:606] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 2 LEADER]: Received request to transfer leadership to TS e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:01.171581 19307 ts_tablet_manager.cc:933] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da: Initiating tablet copy from peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:01.185573 19289 raft_consensus.cc:993] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b: : Instructing follower 0da169ff82674dc1947055ce43de68a3 to start an election
I20260709 12:21:01.186134 19289 raft_consensus.cc:1081] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 3 LEADER]: Signalling peer 0da169ff82674dc1947055ce43de68a3 to start an election
I20260709 12:21:01.196018 17687 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9"
dest_uuid: "0da169ff82674dc1947055ce43de68a3"
 from {username='slave'} at 127.0.0.1:37402
I20260709 12:21:01.209175 16501 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 3
I20260709 12:21:01.211546 16501 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:21:01.212841 16501 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:21:01.197228 17687 raft_consensus.cc:493] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:21:01.217020 17687 raft_consensus.cc:3060] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:01.220716 17759 raft_consensus.cc:1275] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260709 12:21:01.248219 17687 raft_consensus.cc:515] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:01.259233 17687 leader_election.cc:290] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 4 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:01.261965 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: true dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:01.262547 18367 raft_consensus.cc:3060] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:01.272439 19206 consensus_queue.cc:1048] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.016s
I20260709 12:21:01.290845 19249 raft_consensus.cc:2955] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } }
I20260709 12:21:01.299180 17687 raft_consensus.cc:2955] T 9160d9f98b8c406db2771d9ea2e0995f P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } }
I20260709 12:21:01.317397 18367 raft_consensus.cc:2468] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 4.
I20260709 12:21:01.319959 17654 leader_election.cc:304] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:21:01.323561 19285 raft_consensus.cc:2804] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:21:01.327767 19285 raft_consensus.cc:697] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 4 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:21:01.334719 19307 tablet_copy_client.cc:323] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Beginning tablet copy session from remote peer at address 127.16.29.67:36837
I20260709 12:21:01.328595 19285 consensus_queue.cc:237] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:01.337020 17848 tablet_copy_service.cc:140] P e063541542a94969898868bf6eb5ce1b: Received BeginTabletCopySession request for tablet 45397041e3ac43c8901c5cb03f27c7b9 from peer 334a4e1f59944544b3b3ff36616b57da ({username='slave'} at 127.0.0.1:37994)
I20260709 12:21:01.337611 17848 tablet_copy_service.cc:161] P e063541542a94969898868bf6eb5ce1b: Beginning new tablet copy session on tablet 45397041e3ac43c8901c5cb03f27c7b9 from peer 334a4e1f59944544b3b3ff36616b57da at {username='slave'} at 127.0.0.1:37994: session id = 334a4e1f59944544b3b3ff36616b57da-45397041e3ac43c8901c5cb03f27c7b9
I20260709 12:21:01.348726 17848 tablet_copy_source_session.cc:215] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:21:01.382730 17757 raft_consensus.cc:1275] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:21:01.386191 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: true dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:01.386926 17834 raft_consensus.cc:3055] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:21:01.387280 17834 raft_consensus.cc:740] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 3 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:21:01.388209 17834 consensus_queue.cc:260] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:01.397614 17834 raft_consensus.cc:3060] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:01.435141 19307 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 45397041e3ac43c8901c5cb03f27c7b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:01.437451 17834 raft_consensus.cc:2468] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 4.
I20260709 12:21:01.452531 19297 raft_consensus.cc:993] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3: : Instructing follower e2909cff1da44ad387f96a8ae3aa5afb to start an election
I20260709 12:21:01.453100 19297 raft_consensus.cc:1081] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 1 LEADER]: Signalling peer e2909cff1da44ad387f96a8ae3aa5afb to start an election
I20260709 12:21:01.439553 19185 consensus_queue.cc:1048] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.015s
I20260709 12:21:01.456111 19315 ts_tablet_manager.cc:933] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da: Initiating tablet copy from peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:01.459023 19310 raft_consensus.cc:993] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3: : Instructing follower e2909cff1da44ad387f96a8ae3aa5afb to start an election
I20260709 12:21:01.459460 19310 raft_consensus.cc:1081] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 2 LEADER]: Signalling peer e2909cff1da44ad387f96a8ae3aa5afb to start an election
I20260709 12:21:01.474855 17763 raft_consensus.cc:2955] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } }
I20260709 12:21:01.479662 17585 catalog_manager.cc:5697] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 reported cstate change: term changed from 3 to 4, leader changed from e063541542a94969898868bf6eb5ce1b (127.16.29.67) to 0da169ff82674dc1947055ce43de68a3 (127.16.29.65). New cstate: current_term: 4 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20260709 12:21:01.485476 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 55155.9us
I20260709 12:21:01.523815 17757 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b"
dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
 from {username='slave'} at 127.0.0.1:38776
I20260709 12:21:01.524446 17757 raft_consensus.cc:493] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:21:01.525166 17757 raft_consensus.cc:3060] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:01.528146 17763 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8"
dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
 from {username='slave'} at 127.0.0.1:38776
I20260709 12:21:01.529981 17583 catalog_manager.cc:5697] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb reported cstate change: config changed from index -1 to 3, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New cstate: current_term: 2 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } }
W20260709 12:21:01.531687 17635 auto_rebalancer.cc:629] Failed to schedule move for tablet 00a02ab39fcd4078ad49697693cf3be8: Remote error: Service unavailable: leader transfer in progress
I20260709 12:21:01.530369 19307 tablet_copy_client.cc:806] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Starting download of 0 data blocks...
I20260709 12:21:01.533704 19307 tablet_copy_client.cc:670] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Starting download of 1 WAL segments...
I20260709 12:21:01.531128 17763 raft_consensus.cc:493] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:21:01.539770 17763 raft_consensus.cc:3060] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:01.540449 17683 consensus_queue.cc:237] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:21:01.545902 17757 raft_consensus.cc:515] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:01.547019 17762 raft_consensus.cc:1240] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 2. Current term is 3. Ops: []
I20260709 12:21:01.549715 17757 leader_election.cc:290] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 3 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:01.550908 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:01.551469 17834 raft_consensus.cc:3060] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:01.552641 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:01.553210 17686 raft_consensus.cc:3055] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:21:01.553483 17686 raft_consensus.cc:740] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:21:01.554144 17686 consensus_queue.cc:260] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:01.555132 17686 raft_consensus.cc:3060] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:01.557045 17763 raft_consensus.cc:515] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:01.560988 19315 tablet_copy_client.cc:323] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Beginning tablet copy session from remote peer at address 127.16.29.67:36837
I20260709 12:21:01.564476 17848 tablet_copy_service.cc:140] P e063541542a94969898868bf6eb5ce1b: Received BeginTabletCopySession request for tablet 9160d9f98b8c406db2771d9ea2e0995f from peer 334a4e1f59944544b3b3ff36616b57da ({username='slave'} at 127.0.0.1:37994)
I20260709 12:21:01.548869 19307 tablet_copy_client.cc:538] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:21:01.585390 17848 tablet_copy_service.cc:161] P e063541542a94969898868bf6eb5ce1b: Beginning new tablet copy session on tablet 9160d9f98b8c406db2771d9ea2e0995f from peer 334a4e1f59944544b3b3ff36616b57da at {username='slave'} at 127.0.0.1:37994: session id = 334a4e1f59944544b3b3ff36616b57da-9160d9f98b8c406db2771d9ea2e0995f
I20260709 12:21:01.593966 17848 tablet_copy_source_session.cc:215] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:21:01.599483 19315 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9160d9f98b8c406db2771d9ea2e0995f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:01.600090 17834 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 3.
I20260709 12:21:01.608817 17727 leader_election.cc:304] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:21:01.609637 19298 raft_consensus.cc:2804] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:21:01.610220 19298 raft_consensus.cc:697] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 3 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:21:01.611083 19298 consensus_queue.cc:237] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:21:01.617085 17654 consensus_peers.cc:597] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 -> Peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Couldn't send request to peer 334a4e1f59944544b3b3ff36616b57da. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 44ad5363c5954da4a1bff5857780fd5c. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:21:01.611550 19307 tablet_bootstrap.cc:492] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da: Bootstrap starting.
I20260709 12:21:01.620829 17835 raft_consensus.cc:1275] T 44ad5363c5954da4a1bff5857780fd5c P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:21:01.630872 17686 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 3.
I20260709 12:21:01.634027 17757 raft_consensus.cc:1275] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:21:01.634686 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:01.634828 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:01.635252 17834 raft_consensus.cc:3060] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:01.635562 17683 raft_consensus.cc:3055] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:21:01.636006 17683 raft_consensus.cc:740] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:21:01.638731 17683 consensus_queue.cc:260] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:01.640015 17683 raft_consensus.cc:3060] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:01.644003 17763 leader_election.cc:290] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 2 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:01.658252 17834 raft_consensus.cc:2468] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 2.
I20260709 12:21:01.659214 17727 leader_election.cc:304] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:21:01.660048 17683 raft_consensus.cc:2468] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 2.
I20260709 12:21:01.664088 19306 raft_consensus.cc:2804] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:21:01.664594 19306 raft_consensus.cc:697] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:21:01.665434 19306 consensus_queue.cc:237] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:01.668977 19315 tablet_copy_client.cc:806] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Starting download of 0 data blocks...
I20260709 12:21:01.669842 19315 tablet_copy_client.cc:670] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Starting download of 1 WAL segments...
I20260709 12:21:01.684239 19307 tablet_bootstrap.cc:492] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:21:01.685281 19307 tablet_bootstrap.cc:492] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da: Bootstrap complete.
I20260709 12:21:01.686021 19307 ts_tablet_manager.cc:1403] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da: Time spent bootstrapping tablet: real 0.075s	user 0.022s	sys 0.000s
I20260709 12:21:01.689253 19307 raft_consensus.cc:359] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:21:01.689724 19315 tablet_copy_client.cc:538] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:21:01.690146 19307 raft_consensus.cc:740] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Initialized, Role: LEARNER
I20260709 12:21:01.690897 19307 consensus_queue.cc:260] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:21:01.703260 19307 ts_tablet_manager.cc:1434] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da: Time spent starting tablet: real 0.017s	user 0.007s	sys 0.002s
I20260709 12:21:01.707737 17848 tablet_copy_service.cc:342] P e063541542a94969898868bf6eb5ce1b: Request end of tablet copy session 334a4e1f59944544b3b3ff36616b57da-45397041e3ac43c8901c5cb03f27c7b9 received from {username='slave'} at 127.0.0.1:37994
I20260709 12:21:01.708225 17848 tablet_copy_service.cc:434] P e063541542a94969898868bf6eb5ce1b: ending tablet copy session 334a4e1f59944544b3b3ff36616b57da-45397041e3ac43c8901c5cb03f27c7b9 on tablet 45397041e3ac43c8901c5cb03f27c7b9 with peer 334a4e1f59944544b3b3ff36616b57da
I20260709 12:21:01.716406 19318 consensus_queue.cc:1048] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:21:01.721695 19318 consensus_queue.cc:1048] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.013s
I20260709 12:21:01.760718 19315 tablet_bootstrap.cc:492] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da: Bootstrap starting.
I20260709 12:21:01.774896 17835 raft_consensus.cc:1275] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20260709 12:21:01.778155 19318 consensus_queue.cc:1048] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
W20260709 12:21:01.782704 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 61187us
I20260709 12:21:01.793777 17583 catalog_manager.cc:5697] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb reported cstate change: term changed from 2 to 3, leader changed from 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) to e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66). New cstate: current_term: 3 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:01.799012 17583 catalog_manager.cc:5697] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb reported cstate change: term changed from 1 to 2, leader changed from 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) to e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66). New cstate: current_term: 2 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:01.815521 19287 raft_consensus.cc:2955] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } }
I20260709 12:21:01.823951 17837 raft_consensus.cc:2955] T 44ad5363c5954da4a1bff5857780fd5c P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } }
I20260709 12:21:01.844293 18371 raft_consensus.cc:1275] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20260709 12:21:01.845693 17759 raft_consensus.cc:2955] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } }
I20260709 12:21:01.847455 19318 consensus_queue.cc:1048] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 12:21:01.864178 17583 catalog_manager.cc:5697] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb reported cstate change: config changed from index -1 to 2, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New cstate: current_term: 1 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } }
I20260709 12:21:01.939486 19315 tablet_bootstrap.cc:492] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:21:01.940541 19315 tablet_bootstrap.cc:492] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da: Bootstrap complete.
I20260709 12:21:01.956817 19315 ts_tablet_manager.cc:1403] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da: Time spent bootstrapping tablet: real 0.196s	user 0.029s	sys 0.008s
I20260709 12:21:01.961563 19315 raft_consensus.cc:359] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:21:01.962445 19315 raft_consensus.cc:740] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Initialized, Role: LEARNER
I20260709 12:21:01.963145 19315 consensus_queue.cc:260] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:21:01.977612 19315 ts_tablet_manager.cc:1434] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da: Time spent starting tablet: real 0.020s	user 0.002s	sys 0.005s
I20260709 12:21:02.007690 17848 tablet_copy_service.cc:342] P e063541542a94969898868bf6eb5ce1b: Request end of tablet copy session 334a4e1f59944544b3b3ff36616b57da-9160d9f98b8c406db2771d9ea2e0995f received from {username='slave'} at 127.0.0.1:37994
I20260709 12:21:02.008152 17848 tablet_copy_service.cc:434] P e063541542a94969898868bf6eb5ce1b: ending tablet copy session 334a4e1f59944544b3b3ff36616b57da-9160d9f98b8c406db2771d9ea2e0995f on tablet 9160d9f98b8c406db2771d9ea2e0995f with peer 334a4e1f59944544b3b3ff36616b57da
I20260709 12:21:02.063926 17837 raft_consensus.cc:1275] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:21:02.067246 19168 consensus_queue.cc:1048] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:21:02.073777 19307 ts_tablet_manager.cc:933] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da: Initiating tablet copy from peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071)
I20260709 12:21:02.087802 19307 tablet_copy_client.cc:323] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Beginning tablet copy session from remote peer at address 127.16.29.65:33071
I20260709 12:21:02.089542 17699 tablet_copy_service.cc:140] P 0da169ff82674dc1947055ce43de68a3: Received BeginTabletCopySession request for tablet 44ad5363c5954da4a1bff5857780fd5c from peer 334a4e1f59944544b3b3ff36616b57da ({username='slave'} at 127.0.0.1:44898)
I20260709 12:21:02.090129 17699 tablet_copy_service.cc:161] P 0da169ff82674dc1947055ce43de68a3: Beginning new tablet copy session on tablet 44ad5363c5954da4a1bff5857780fd5c from peer 334a4e1f59944544b3b3ff36616b57da at {username='slave'} at 127.0.0.1:44898: session id = 334a4e1f59944544b3b3ff36616b57da-44ad5363c5954da4a1bff5857780fd5c
I20260709 12:21:02.102792 17699 tablet_copy_source_session.cc:215] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:21:02.108690 19307 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 44ad5363c5954da4a1bff5857780fd5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:02.140354 17686 raft_consensus.cc:1275] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:21:02.143144 19168 consensus_queue.cc:1048] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:21:02.148936 17683 raft_consensus.cc:1275] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:21:02.151062 19306 consensus_queue.cc:1048] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:21:02.156137 18367 raft_consensus.cc:1217] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:21:02.161273 17837 raft_consensus.cc:1275] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:21:02.162778 19306 consensus_queue.cc:1048] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:21:02.268913 19307 tablet_copy_client.cc:806] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Starting download of 0 data blocks...
I20260709 12:21:02.269558 19307 tablet_copy_client.cc:670] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Starting download of 1 WAL segments...
I20260709 12:21:02.278198 19307 tablet_copy_client.cc:538] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:21:02.298491 19307 tablet_bootstrap.cc:492] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da: Bootstrap starting.
I20260709 12:21:02.301873 18367 raft_consensus.cc:1217] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 2 LEARNER]: Deduplicated request from leader. Original: 2.2->[2.3-2.3]   Dedup: 2.3->[]
I20260709 12:21:02.305881 16501 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:21:02.359306 19307 tablet_bootstrap.cc:492] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:21:02.360399 19307 tablet_bootstrap.cc:492] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da: Bootstrap complete.
I20260709 12:21:02.361107 19307 ts_tablet_manager.cc:1403] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da: Time spent bootstrapping tablet: real 0.063s	user 0.031s	sys 0.009s
I20260709 12:21:02.364858 19307 raft_consensus.cc:359] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:21:02.366190 19307 raft_consensus.cc:740] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Initialized, Role: LEARNER
I20260709 12:21:02.366979 19307 consensus_queue.cc:260] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:21:02.382041 19307 ts_tablet_manager.cc:1434] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da: Time spent starting tablet: real 0.020s	user 0.007s	sys 0.000s
I20260709 12:21:02.409435 17699 tablet_copy_service.cc:342] P 0da169ff82674dc1947055ce43de68a3: Request end of tablet copy session 334a4e1f59944544b3b3ff36616b57da-44ad5363c5954da4a1bff5857780fd5c received from {username='slave'} at 127.0.0.1:44898
I20260709 12:21:02.409889 17699 tablet_copy_service.cc:434] P 0da169ff82674dc1947055ce43de68a3: ending tablet copy session 334a4e1f59944544b3b3ff36616b57da-44ad5363c5954da4a1bff5857780fd5c on tablet 44ad5363c5954da4a1bff5857780fd5c with peer 334a4e1f59944544b3b3ff36616b57da
I20260709 12:21:02.422693 17683 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462"
dest_uuid: "0da169ff82674dc1947055ce43de68a3"
mode: GRACEFUL
new_leader_uuid: "e063541542a94969898868bf6eb5ce1b"
 from {username='slave'} at 127.0.0.1:54540
I20260709 12:21:02.423449 17683 raft_consensus.cc:606] T 8cab6f01c6a2485cb5024322a1bd5462 P 0da169ff82674dc1947055ce43de68a3 [term 2 LEADER]: Received request to transfer leadership to TS e063541542a94969898868bf6eb5ce1b
I20260709 12:21:02.426472 17757 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "ac984176358d453a8800cc9e0e304061"
dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
mode: GRACEFUL
new_leader_uuid: "0da169ff82674dc1947055ce43de68a3"
 from {username='slave'} at 127.0.0.1:52946
I20260709 12:21:02.427038 17757 raft_consensus.cc:606] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 LEADER]: Received request to transfer leadership to TS 0da169ff82674dc1947055ce43de68a3
I20260709 12:21:02.428517 16501 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260709 12:21:02.430174 16501 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:21:02.431115 16501 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:21:02.540942 19330 raft_consensus.cc:993] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb: : Instructing follower 0da169ff82674dc1947055ce43de68a3 to start an election
I20260709 12:21:02.541503 19330 raft_consensus.cc:1081] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 LEADER]: Signalling peer 0da169ff82674dc1947055ce43de68a3 to start an election
I20260709 12:21:02.544001 17683 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "ac984176358d453a8800cc9e0e304061"
dest_uuid: "0da169ff82674dc1947055ce43de68a3"
 from {username='slave'} at 127.0.0.1:37396
I20260709 12:21:02.544656 17683 raft_consensus.cc:493] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:21:02.545064 17683 raft_consensus.cc:3060] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:02.552347 17683 raft_consensus.cc:515] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:02.571348 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: true dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:02.573807 17759 raft_consensus.cc:3055] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:21:02.574158 17759 raft_consensus.cc:740] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:21:02.574919 17759 consensus_queue.cc:260] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:02.592625 18371 raft_consensus.cc:1217] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:21:02.576057 17759 raft_consensus.cc:3060] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:02.601843 17683 leader_election.cc:290] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 5 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:02.603849 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: true dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:02.604476 18371 raft_consensus.cc:3060] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:02.602169 17759 raft_consensus.cc:2468] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 5.
I20260709 12:21:02.615666 17652 leader_election.cc:304] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:21:02.616482 19287 raft_consensus.cc:2804] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:21:02.616958 19287 raft_consensus.cc:697] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 5 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:21:02.617772 19287 consensus_queue.cc:237] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:02.633649 18371 raft_consensus.cc:2468] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 5.
I20260709 12:21:02.640456 19318 raft_consensus.cc:993] T 8cab6f01c6a2485cb5024322a1bd5462 P 0da169ff82674dc1947055ce43de68a3: : Instructing follower e063541542a94969898868bf6eb5ce1b to start an election
I20260709 12:21:02.647951 19318 raft_consensus.cc:1081] T 8cab6f01c6a2485cb5024322a1bd5462 P 0da169ff82674dc1947055ce43de68a3 [term 2 LEADER]: Signalling peer e063541542a94969898868bf6eb5ce1b to start an election
I20260709 12:21:02.653060 17837 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462"
dest_uuid: "e063541542a94969898868bf6eb5ce1b"
 from {username='slave'} at 127.0.0.1:35040
I20260709 12:21:02.653690 17837 raft_consensus.cc:493] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:21:02.655290 17837 raft_consensus.cc:3060] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:02.657290 19288 raft_consensus.cc:1064] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b: attempting to promote NON_VOTER 334a4e1f59944544b3b3ff36616b57da to VOTER
I20260709 12:21:02.660547 19288 consensus_queue.cc:237] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:02.676863 17583 catalog_manager.cc:5697] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 reported cstate change: term changed from 4 to 5, leader changed from e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66) to 0da169ff82674dc1947055ce43de68a3 (127.16.29.65). New cstate: current_term: 5 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:02.684077 17837 raft_consensus.cc:515] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:02.694530 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:02.695103 17683 raft_consensus.cc:3055] T 8cab6f01c6a2485cb5024322a1bd5462 P 0da169ff82674dc1947055ce43de68a3 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:21:02.695382 17683 raft_consensus.cc:740] T 8cab6f01c6a2485cb5024322a1bd5462 P 0da169ff82674dc1947055ce43de68a3 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:21:02.696115 17683 consensus_queue.cc:260] T 8cab6f01c6a2485cb5024322a1bd5462 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:02.697264 17683 raft_consensus.cc:3060] T 8cab6f01c6a2485cb5024322a1bd5462 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:02.722107 17683 raft_consensus.cc:2468] T 8cab6f01c6a2485cb5024322a1bd5462 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 3.
I20260709 12:21:02.726290 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:02.729843 17759 raft_consensus.cc:3060] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:02.729087 17837 leader_election.cc:290] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 3 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:02.748407 17802 leader_election.cc:304] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:21:02.749341 19206 raft_consensus.cc:2804] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:21:02.750648 19206 raft_consensus.cc:697] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 3 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:21:02.751683 19206 consensus_queue.cc:237] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:02.766260 17587 catalog_manager.cc:5697] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b reported cstate change: term changed from 2 to 3, leader changed from 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) to e063541542a94969898868bf6eb5ce1b (127.16.29.67). New cstate: current_term: 3 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: HEALTHY } } }
I20260709 12:21:02.743065 17759 raft_consensus.cc:2468] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 3.
I20260709 12:21:02.778672 18371 raft_consensus.cc:1275] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 1 LEARNER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:21:02.782327 17685 raft_consensus.cc:1275] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:21:02.792331 19206 consensus_queue.cc:1048] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:21:02.795462 19206 consensus_queue.cc:1048] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:21:02.834028 17759 raft_consensus.cc:1275] T 45397041e3ac43c8901c5cb03f27c7b9 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:21:02.839344 19206 consensus_queue.cc:1048] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:21:02.847451 19288 raft_consensus.cc:2955] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:21:02.880087 17757 raft_consensus.cc:2955] T 45397041e3ac43c8901c5cb03f27c7b9 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:21:02.908131 18367 raft_consensus.cc:2955] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:21:02.913870 17683 raft_consensus.cc:2955] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:21:02.942080 19291 raft_consensus.cc:1064] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b: attempting to promote NON_VOTER 334a4e1f59944544b3b3ff36616b57da to VOTER
I20260709 12:21:02.947908 19291 consensus_queue.cc:237] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 1, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:02.930552 17587 catalog_manager.cc:5697] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b reported cstate change: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 12:21:03.022112 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 59544.5us
I20260709 12:21:03.036059 17763 raft_consensus.cc:1275] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260709 12:21:03.038666 19288 consensus_queue.cc:1048] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.002s
I20260709 12:21:03.046314 17683 raft_consensus.cc:1275] T 9160d9f98b8c406db2771d9ea2e0995f P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260709 12:21:03.050660 19288 consensus_queue.cc:1048] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
W20260709 12:21:03.072964 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 46229.9us
I20260709 12:21:03.077230 18367 raft_consensus.cc:1275] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 2 LEARNER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260709 12:21:03.077385 17837 consensus_queue.cc:237] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:03.085471 19206 raft_consensus.cc:2955] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:21:03.089354 17683 raft_consensus.cc:1275] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:21:03.092132 19355 consensus_queue.cc:1048] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:21:03.098754 19358 consensus_queue.cc:1048] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:21:03.100572 17686 raft_consensus.cc:2955] T 9160d9f98b8c406db2771d9ea2e0995f P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:21:03.116703 18367 raft_consensus.cc:1275] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260709 12:21:03.118345 19318 consensus_queue.cc:1048] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 12:21:03.128775 17763 raft_consensus.cc:1275] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260709 12:21:03.131055 19297 consensus_queue.cc:1048] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 12:21:03.134788 17757 raft_consensus.cc:2955] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:21:03.153594 18371 raft_consensus.cc:1275] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:21:03.168123 18374 raft_consensus.cc:2955] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:21:03.197582 19297 raft_consensus.cc:1064] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3: attempting to promote NON_VOTER 334a4e1f59944544b3b3ff36616b57da to VOTER
I20260709 12:21:03.225468 17584 catalog_manager.cc:5697] T 9160d9f98b8c406db2771d9ea2e0995f P 0da169ff82674dc1947055ce43de68a3 reported cstate change: config changed from index 3 to 4, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:21:03.249172 19291 raft_consensus.cc:2955] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:21:03.202402 19297 consensus_queue.cc:237] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:03.264106 19288 consensus_queue.cc:1048] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:21:03.270102 18374 raft_consensus.cc:2955] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:21:03.272280 17757 raft_consensus.cc:1275] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:21:03.273970 19288 consensus_queue.cc:1048] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:21:03.380285 17571 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 45397041e3ac43c8901c5cb03f27c7b9 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:21:03.417662 17683 raft_consensus.cc:2955] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:21:03.428771 17685 raft_consensus.cc:1275] T 8cab6f01c6a2485cb5024322a1bd5462 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
W20260709 12:21:03.439493 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 72430.2us
I20260709 12:21:03.447811 18367 raft_consensus.cc:1275] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 1 LEARNER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:21:03.471596 17585 catalog_manager.cc:5697] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da reported cstate change: config changed from index 3 to 4, VOTER e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66) evicted. New cstate: current_term: 1 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:21:03.517674 19369 consensus_queue.cc:1048] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:21:03.535909 17837 raft_consensus.cc:1275] T 44ad5363c5954da4a1bff5857780fd5c P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:21:03.533819 16501 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:21:03.548907 17757 raft_consensus.cc:1275] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:21:03.551756 19358 consensus_queue.cc:1048] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20260709 12:21:03.567543 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 76431.8us
W20260709 12:21:03.574497 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 77081.4us
W20260709 12:21:03.585636 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 61479.7us
I20260709 12:21:03.588748 17753 tablet_service.cc:1558] Processing DeleteTablet for tablet 45397041e3ac43c8901c5cb03f27c7b9 with delete_type TABLET_DATA_TOMBSTONED (TS e2909cff1da44ad387f96a8ae3aa5afb not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:38774
I20260709 12:21:03.596441 19287 consensus_queue.cc:1048] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:21:03.683856 19366 consensus_queue.cc:1048] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:21:03.745828 19380 tablet_replica.cc:333] T 45397041e3ac43c8901c5cb03f27c7b9 P e2909cff1da44ad387f96a8ae3aa5afb: stopping tablet replica
I20260709 12:21:03.747372 19380 raft_consensus.cc:2243] T 45397041e3ac43c8901c5cb03f27c7b9 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:21:03.772984 19380 raft_consensus.cc:2272] T 45397041e3ac43c8901c5cb03f27c7b9 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 12:21:03.799525 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 63136us
W20260709 12:21:03.803119 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 62609.3us
I20260709 12:21:03.811375 19369 raft_consensus.cc:2955] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:21:03.819213 17835 raft_consensus.cc:2955] T 44ad5363c5954da4a1bff5857780fd5c P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:21:03.786906 19380 ts_tablet_manager.cc:1916] T 45397041e3ac43c8901c5cb03f27c7b9 P e2909cff1da44ad387f96a8ae3aa5afb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:21:03.879036 19380 ts_tablet_manager.cc:1929] T 45397041e3ac43c8901c5cb03f27c7b9 P e2909cff1da44ad387f96a8ae3aa5afb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:21:03.879637 19380 log.cc:1199] T 45397041e3ac43c8901c5cb03f27c7b9 P e2909cff1da44ad387f96a8ae3aa5afb: Deleting WAL directory at /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599551912677-16501-0/minicluster-data/ts-1-root/wals/45397041e3ac43c8901c5cb03f27c7b9
I20260709 12:21:03.881403 17571 catalog_manager.cc:5028] TS e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): tablet 45397041e3ac43c8901c5cb03f27c7b9 (table test-workload [id=ddc5b74250df4c65909b73b3742d615f]) successfully deleted
I20260709 12:21:03.910298 18371 raft_consensus.cc:2955] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:21:03.915845 17585 catalog_manager.cc:5697] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 reported cstate change: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:21:03.938176 17763 raft_consensus.cc:2955] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:21:03.951282 17835 consensus_queue.cc:237] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:03.958663 17687 consensus_queue.cc:237] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:03.962127 18365 raft_consensus.cc:1275] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:21:03.964535 19355 consensus_queue.cc:1048] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:21:03.986104 19356 raft_consensus.cc:2955] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:21:03.988332 18371 raft_consensus.cc:1275] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:21:03.993361 19378 consensus_queue.cc:1048] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:21:03.989481 17757 raft_consensus.cc:1275] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:21:04.004879 17762 raft_consensus.cc:1275] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:21:04.009117 19291 consensus_queue.cc:1048] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:21:04.012044 19378 consensus_queue.cc:1048] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:21:04.012826 17757 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8"
dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
mode: GRACEFUL
new_leader_uuid: "0da169ff82674dc1947055ce43de68a3"
 from {username='slave'} at 127.0.0.1:52946
I20260709 12:21:04.013617 17757 raft_consensus.cc:606] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 LEADER]: Received request to transfer leadership to TS 0da169ff82674dc1947055ce43de68a3
I20260709 12:21:04.016098 19371 raft_consensus.cc:2955] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER e063541542a94969898868bf6eb5ce1b (127.16.29.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:21:04.025914 18365 raft_consensus.cc:2955] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER e063541542a94969898868bf6eb5ce1b (127.16.29.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:21:04.024839 17835 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b"
dest_uuid: "e063541542a94969898868bf6eb5ce1b"
mode: GRACEFUL
new_leader_uuid: "334a4e1f59944544b3b3ff36616b57da"
 from {username='slave'} at 127.0.0.1:59820
I20260709 12:21:04.038800 17835 raft_consensus.cc:606] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 1 LEADER]: Received request to transfer leadership to TS 334a4e1f59944544b3b3ff36616b57da
I20260709 12:21:04.047938 17762 raft_consensus.cc:2955] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:21:04.052294 19384 raft_consensus.cc:993] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb: : Instructing follower 0da169ff82674dc1947055ce43de68a3 to start an election
I20260709 12:21:04.052933 19168 raft_consensus.cc:1081] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 LEADER]: Signalling peer 0da169ff82674dc1947055ce43de68a3 to start an election
I20260709 12:21:04.057789 17687 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442"
dest_uuid: "0da169ff82674dc1947055ce43de68a3"
mode: GRACEFUL
new_leader_uuid: "e063541542a94969898868bf6eb5ce1b"
 from {username='slave'} at 127.0.0.1:54540
I20260709 12:21:04.058588 17685 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8"
dest_uuid: "0da169ff82674dc1947055ce43de68a3"
 from {username='slave'} at 127.0.0.1:37396
I20260709 12:21:04.059316 17685 raft_consensus.cc:493] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:21:04.059633 17685 raft_consensus.cc:3060] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:04.048753 18371 raft_consensus.cc:2955] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:21:04.068529 17687 raft_consensus.cc:606] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 2 LEADER]: Received request to transfer leadership to TS e063541542a94969898868bf6eb5ce1b
I20260709 12:21:04.073448 16501 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 3
I20260709 12:21:04.075567 16501 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:21:04.076941 16501 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:21:04.093497 17571 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 9160d9f98b8c406db2771d9ea2e0995f with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:21:04.100683 17572 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 44ad5363c5954da4a1bff5857780fd5c with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:21:04.104815 17585 catalog_manager.cc:5697] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b reported cstate change: config changed from index 4 to 5, VOTER 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) evicted. New cstate: current_term: 2 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:21:04.128207 17583 catalog_manager.cc:5697] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 reported cstate change: config changed from index 3 to 4, VOTER e063541542a94969898868bf6eb5ce1b (127.16.29.67) evicted. New cstate: current_term: 1 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:21:04.067363 17685 raft_consensus.cc:515] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:04.152524 19358 raft_consensus.cc:993] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b: : Instructing follower 334a4e1f59944544b3b3ff36616b57da to start an election
I20260709 12:21:04.153436 19355 raft_consensus.cc:1081] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 1 LEADER]: Signalling peer 334a4e1f59944544b3b3ff36616b57da to start an election
I20260709 12:21:04.156481 18371 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b"
dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
 from {username='slave'} at 127.0.0.1:59902
I20260709 12:21:04.157042 18371 raft_consensus.cc:493] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:21:04.157372 18371 raft_consensus.cc:3060] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:04.157963 17685 leader_election.cc:290] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 3 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:04.166558 18371 raft_consensus.cc:515] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:04.178063 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:04.178761 17831 raft_consensus.cc:3060] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:04.179564 18371 leader_election.cc:290] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 2 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:04.187001 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:04.188123 17676 tablet_service.cc:1558] Processing DeleteTablet for tablet 9160d9f98b8c406db2771d9ea2e0995f with delete_type TABLET_DATA_TOMBSTONED (TS 0da169ff82674dc1947055ce43de68a3 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:37384
I20260709 12:21:04.189828 17759 raft_consensus.cc:3060] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:04.196285 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:04.197063 17763 raft_consensus.cc:3055] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:21:04.197667 17763 raft_consensus.cc:740] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:21:04.198791 17763 consensus_queue.cc:260] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:04.200064 17763 raft_consensus.cc:3060] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:04.204726 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:04.205862 17837 raft_consensus.cc:3055] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:21:04.206226 17837 raft_consensus.cc:740] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:21:04.207782 17837 consensus_queue.cc:260] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:04.215777 17837 raft_consensus.cc:3060] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:04.294384 17759 raft_consensus.cc:2468] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 2.
W20260709 12:21:04.295922 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RunLeaderElection -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 70554.6us
I20260709 12:21:04.245690 17762 raft_consensus.cc:2955] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER e063541542a94969898868bf6eb5ce1b (127.16.29.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:21:04.301256 19389 tablet_replica.cc:333] T 9160d9f98b8c406db2771d9ea2e0995f P 0da169ff82674dc1947055ce43de68a3: stopping tablet replica
I20260709 12:21:04.293080 17763 raft_consensus.cc:2468] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 3.
I20260709 12:21:04.304769 19389 raft_consensus.cc:2243] T 9160d9f98b8c406db2771d9ea2e0995f P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:21:04.306270 19389 raft_consensus.cc:2272] T 9160d9f98b8c406db2771d9ea2e0995f P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:04.309965 19389 ts_tablet_manager.cc:1916] T 9160d9f98b8c406db2771d9ea2e0995f P 0da169ff82674dc1947055ce43de68a3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:21:04.329242 17652 leader_election.cc:304] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:21:04.331283 19318 raft_consensus.cc:2804] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:21:04.334364 17821 tablet_service.cc:1558] Processing DeleteTablet for tablet 44ad5363c5954da4a1bff5857780fd5c with delete_type TABLET_DATA_TOMBSTONED (TS e063541542a94969898868bf6eb5ce1b not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:35032
I20260709 12:21:04.338502 17831 raft_consensus.cc:2468] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 3.
W20260709 12:21:04.341017 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 106877us
I20260709 12:21:04.345611 18333 leader_election.cc:304] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:21:04.346035 19389 ts_tablet_manager.cc:1929] T 9160d9f98b8c406db2771d9ea2e0995f P 0da169ff82674dc1947055ce43de68a3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20260709 12:21:04.346665 19299 raft_consensus.cc:2804] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:21:04.347338 19299 raft_consensus.cc:697] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 2 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:21:04.348510 19299 consensus_queue.cc:237] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:04.350083 19318 raft_consensus.cc:697] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 3 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:21:04.346680 19389 log.cc:1199] T 9160d9f98b8c406db2771d9ea2e0995f P 0da169ff82674dc1947055ce43de68a3: Deleting WAL directory at /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599551912677-16501-0/minicluster-data/ts-0-root/wals/9160d9f98b8c406db2771d9ea2e0995f
I20260709 12:21:04.354034 19318 consensus_queue.cc:237] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:04.355990 17837 raft_consensus.cc:2468] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 2.
I20260709 12:21:04.356112 17572 catalog_manager.cc:5028] TS 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): tablet 9160d9f98b8c406db2771d9ea2e0995f (table test-workload [id=ddc5b74250df4c65909b73b3742d615f]) successfully deleted
I20260709 12:21:04.359472 19391 tablet_replica.cc:333] T 44ad5363c5954da4a1bff5857780fd5c P e063541542a94969898868bf6eb5ce1b: stopping tablet replica
I20260709 12:21:04.372539 17587 catalog_manager.cc:5697] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 1 to 2, leader changed from e063541542a94969898868bf6eb5ce1b (127.16.29.67) to 334a4e1f59944544b3b3ff36616b57da (127.16.29.68). New cstate: current_term: 2 leader_uuid: "334a4e1f59944544b3b3ff36616b57da" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:21:04.373309 17583 catalog_manager.cc:5697] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 reported cstate change: term changed from 2 to 3, leader changed from e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66) to 0da169ff82674dc1947055ce43de68a3 (127.16.29.65). New cstate: current_term: 3 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:04.406546 19391 raft_consensus.cc:2243] T 44ad5363c5954da4a1bff5857780fd5c P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:21:04.408775 19391 raft_consensus.cc:2272] T 44ad5363c5954da4a1bff5857780fd5c P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:04.413146 19391 ts_tablet_manager.cc:1916] T 44ad5363c5954da4a1bff5857780fd5c P e063541542a94969898868bf6eb5ce1b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:21:04.430737 19378 raft_consensus.cc:993] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3: : Instructing follower e063541542a94969898868bf6eb5ce1b to start an election
I20260709 12:21:04.431375 19378 raft_consensus.cc:1081] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 2 LEADER]: Signalling peer e063541542a94969898868bf6eb5ce1b to start an election
I20260709 12:21:04.439106 17834 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442"
dest_uuid: "e063541542a94969898868bf6eb5ce1b"
 from {username='slave'} at 127.0.0.1:35040
I20260709 12:21:04.439704 17834 raft_consensus.cc:493] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:21:04.440099 17834 raft_consensus.cc:3060] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:04.488087 19391 ts_tablet_manager.cc:1929] T 44ad5363c5954da4a1bff5857780fd5c P e063541542a94969898868bf6eb5ce1b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:21:04.488746 19391 log.cc:1199] T 44ad5363c5954da4a1bff5857780fd5c P e063541542a94969898868bf6eb5ce1b: Deleting WAL directory at /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599551912677-16501-0/minicluster-data/ts-2-root/wals/44ad5363c5954da4a1bff5857780fd5c
I20260709 12:21:04.490592 17571 catalog_manager.cc:5028] TS e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837): tablet 44ad5363c5954da4a1bff5857780fd5c (table test-workload [id=ddc5b74250df4c65909b73b3742d615f]) successfully deleted
I20260709 12:21:04.495903 17834 raft_consensus.cc:515] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:04.500222 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:04.500872 17762 raft_consensus.cc:3060] T 7cb3768d215e409fb9f5baf18d124442 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:04.499579 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:04.503159 17685 raft_consensus.cc:3055] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:21:04.503594 17685 raft_consensus.cc:740] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:21:04.504388 17685 consensus_queue.cc:260] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:04.505465 17685 raft_consensus.cc:3060] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:04.508122 17762 raft_consensus.cc:2468] T 7cb3768d215e409fb9f5baf18d124442 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 3.
I20260709 12:21:04.509708 17801 leader_election.cc:304] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:21:04.510629 19356 raft_consensus.cc:2804] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:21:04.511142 19356 raft_consensus.cc:697] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 3 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:21:04.514326 19356 consensus_queue.cc:237] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:04.537775 17834 leader_election.cc:290] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 3 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:04.541672 17685 raft_consensus.cc:2468] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 3.
I20260709 12:21:04.542912 17583 catalog_manager.cc:5697] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b reported cstate change: term changed from 2 to 3, leader changed from 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) to e063541542a94969898868bf6eb5ce1b (127.16.29.67). New cstate: current_term: 3 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: HEALTHY } } }
I20260709 12:21:04.774425 17831 raft_consensus.cc:1275] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:21:04.775951 19299 consensus_queue.cc:1048] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:21:04.787412 17837 raft_consensus.cc:1275] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:21:04.790849 19370 consensus_queue.cc:1048] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:21:04.881543 17762 raft_consensus.cc:1275] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:21:04.887328 19265 consensus_queue.cc:1048] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20260709 12:21:04.925170 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 50941.8us
I20260709 12:21:04.974139 17759 raft_consensus.cc:1275] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:21:04.975728 19370 consensus_queue.cc:1048] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:21:04.997481 17763 raft_consensus.cc:1275] T 7cb3768d215e409fb9f5baf18d124442 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:21:04.999891 19291 consensus_queue.cc:1048] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20260709 12:21:05.012719 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 84950.8us
W20260709 12:21:05.143604 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 57056.4us
W20260709 12:21:05.145471 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 47677.7us
W20260709 12:21:05.136543 19399 log.cc:927] Time spent T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.059s	user 0.001s	sys 0.000s
I20260709 12:21:05.173406 17683 raft_consensus.cc:1275] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
W20260709 12:21:05.227224 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 99574.4us
I20260709 12:21:05.237561 19404 consensus_queue.cc:1048] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.041s
W20260709 12:21:05.286674 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 74392.4us
W20260709 12:21:05.366989 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 87944.8us
I20260709 12:21:05.382165 16501 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:21:05.537847 17686 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "ac984176358d453a8800cc9e0e304061"
dest_uuid: "0da169ff82674dc1947055ce43de68a3"
mode: GRACEFUL
new_leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
 from {username='slave'} at 127.0.0.1:54540
I20260709 12:21:05.538480 17686 raft_consensus.cc:606] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 5 LEADER]: Received request to transfer leadership to TS e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:05.547690 17835 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b"
dest_uuid: "e063541542a94969898868bf6eb5ce1b"
mode: GRACEFUL
new_leader_uuid: "0da169ff82674dc1947055ce43de68a3"
 from {username='slave'} at 127.0.0.1:59820
I20260709 12:21:05.548288 17835 raft_consensus.cc:606] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 1 LEADER]: Received request to transfer leadership to TS 0da169ff82674dc1947055ce43de68a3
I20260709 12:21:05.550537 16501 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260709 12:21:05.552243 16501 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:21:05.552960 16501 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:21:05.609320 17837 consensus_queue.cc:237] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:21:05.639802 17686 raft_consensus.cc:1275] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260709 12:21:05.640889 17803 consensus_peers.cc:597] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b -> Peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Couldn't send request to peer 334a4e1f59944544b3b3ff36616b57da. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c64345b0a2614ba4a382b83feb235cad. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:21:05.643102 17759 raft_consensus.cc:1275] T c64345b0a2614ba4a382b83feb235cad P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:21:05.646365 19404 consensus_queue.cc:1048] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:21:05.654922 19404 consensus_queue.cc:1048] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:21:05.679715 19404 raft_consensus.cc:2955] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } }
I20260709 12:21:05.682375 17685 raft_consensus.cc:2955] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } }
I20260709 12:21:05.684401 17763 raft_consensus.cc:2955] T c64345b0a2614ba4a382b83feb235cad P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } }
I20260709 12:21:05.710296 17583 catalog_manager.cc:5697] T c64345b0a2614ba4a382b83feb235cad P e2909cff1da44ad387f96a8ae3aa5afb reported cstate change: config changed from index -1 to 2, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New cstate: current_term: 1 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } }
I20260709 12:21:05.710557 17837 consensus_queue.cc:237] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
W20260709 12:21:05.738948 17803 consensus_peers.cc:597] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b -> Peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Couldn't send request to peer 334a4e1f59944544b3b3ff36616b57da. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8cab6f01c6a2485cb5024322a1bd5462. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:21:05.739388 17763 raft_consensus.cc:1275] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260709 12:21:05.741441 19410 consensus_queue.cc:1048] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:21:05.749096 19404 raft_consensus.cc:993] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b: : Instructing follower 0da169ff82674dc1947055ce43de68a3 to start an election
I20260709 12:21:05.749584 19404 raft_consensus.cc:1081] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 1 LEADER]: Signalling peer 0da169ff82674dc1947055ce43de68a3 to start an election
I20260709 12:21:05.752496 17685 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b"
dest_uuid: "0da169ff82674dc1947055ce43de68a3"
 from {username='slave'} at 127.0.0.1:37402
I20260709 12:21:05.753037 17685 raft_consensus.cc:493] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:21:05.753367 17685 raft_consensus.cc:3060] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:05.766503 17686 raft_consensus.cc:1275] T 8cab6f01c6a2485cb5024322a1bd5462 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260709 12:21:05.768775 19410 consensus_queue.cc:1048] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:21:05.777714 17685 raft_consensus.cc:515] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:05.791607 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:05.792347 17759 raft_consensus.cc:3060] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:05.806306 19414 raft_consensus.cc:2955] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 3 LEADER]: Committing config change with OpId 3.4: config changed from index -1 to 4, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } }
I20260709 12:21:05.813371 17687 raft_consensus.cc:2955] T 8cab6f01c6a2485cb5024322a1bd5462 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Committing config change with OpId 3.4: config changed from index -1 to 4, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } }
I20260709 12:21:05.820993 17759 raft_consensus.cc:2468] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 2.
I20260709 12:21:05.835522 17685 leader_election.cc:290] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 2 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:05.836889 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:05.853233 17837 raft_consensus.cc:3055] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:21:05.853616 17837 raft_consensus.cc:740] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:21:05.854385 17837 consensus_queue.cc:260] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:05.855561 17837 raft_consensus.cc:3060] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:05.874513 17837 raft_consensus.cc:2468] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 2.
I20260709 12:21:05.876358 17652 leader_election.cc:304] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:21:05.875633 17835 consensus_queue.cc:237] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:21:05.878018 19431 raft_consensus.cc:2804] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:21:05.878616 19431 raft_consensus.cc:697] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 2 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:21:05.879741 19431 consensus_queue.cc:237] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:05.894740 17763 raft_consensus.cc:2955] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Committing config change with OpId 3.4: config changed from index -1 to 4, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } }
W20260709 12:21:05.928584 17803 consensus_peers.cc:597] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b -> Peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Couldn't send request to peer 334a4e1f59944544b3b3ff36616b57da. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6fcb5bdaa4be4d619bb945ffd44090ba. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:21:05.955862 17763 raft_consensus.cc:1275] T 6fcb5bdaa4be4d619bb945ffd44090ba P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:21:05.958321 17685 raft_consensus.cc:1275] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:21:05.961078 17587 catalog_manager.cc:5697] T 8cab6f01c6a2485cb5024322a1bd5462 P 0da169ff82674dc1947055ce43de68a3 reported cstate change: config changed from index -1 to 4, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New cstate: current_term: 3 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } }
I20260709 12:21:05.958343 19417 consensus_queue.cc:1048] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.005s
I20260709 12:21:05.962782 17587 catalog_manager.cc:5697] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 reported cstate change: term changed from 1 to 2, leader changed from e063541542a94969898868bf6eb5ce1b (127.16.29.67) to 0da169ff82674dc1947055ce43de68a3 (127.16.29.65). New cstate: current_term: 2 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:05.965719 19410 consensus_queue.cc:1048] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:21:05.971390 19429 raft_consensus.cc:993] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3: : Instructing follower e2909cff1da44ad387f96a8ae3aa5afb to start an election
I20260709 12:21:05.971980 19430 raft_consensus.cc:1081] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 5 LEADER]: Signalling peer e2909cff1da44ad387f96a8ae3aa5afb to start an election
I20260709 12:21:05.993083 17763 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "ac984176358d453a8800cc9e0e304061"
dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
 from {username='slave'} at 127.0.0.1:38776
I20260709 12:21:05.993862 17763 raft_consensus.cc:493] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:21:05.995857 17763 raft_consensus.cc:3060] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:06.017287 17763 raft_consensus.cc:515] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:06.020291 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: true dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:06.021029 17685 raft_consensus.cc:3055] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:21:06.021324 17685 raft_consensus.cc:740] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:21:06.025447 17763 leader_election.cc:290] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 6 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:06.028923 17685 consensus_queue.cc:260] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:06.033591 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: true dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:06.034487 18371 raft_consensus.cc:3060] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:06.033630 17685 raft_consensus.cc:3060] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:06.044691 19405 raft_consensus.cc:2955] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } }
I20260709 12:21:06.047441 17759 raft_consensus.cc:2955] T 6fcb5bdaa4be4d619bb945ffd44090ba P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } }
I20260709 12:21:06.049646 17685 raft_consensus.cc:2468] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 6.
I20260709 12:21:06.051131 17728 leader_election.cc:304] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:21:06.052157 19401 raft_consensus.cc:2804] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:21:06.052613 19401 raft_consensus.cc:697] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:21:06.053437 19401 consensus_queue.cc:237] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:06.064376 17687 consensus_queue.cc:237] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:21:06.088668 17686 raft_consensus.cc:2955] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } }
W20260709 12:21:06.098268 17654 consensus_peers.cc:597] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 -> Peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Couldn't send request to peer 334a4e1f59944544b3b3ff36616b57da. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ea7da467817a4ba88aa5797c3f8f493f. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:21:06.090451 18371 raft_consensus.cc:2468] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 6.
I20260709 12:21:06.100991 17757 raft_consensus.cc:1275] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20260709 12:21:06.100991 17835 raft_consensus.cc:1275] T ea7da467817a4ba88aa5797c3f8f493f P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20260709 12:21:06.103427 19430 consensus_queue.cc:1048] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 12:21:06.103191 17585 catalog_manager.cc:5697] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b reported cstate change: config changed from index -1 to 2, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New cstate: current_term: 1 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:06.105494 19430 consensus_queue.cc:1048] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 12:21:06.096958 17584 catalog_manager.cc:5697] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb reported cstate change: term changed from 5 to 6, leader changed from 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) to e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66). New cstate: current_term: 6 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:06.144210 19427 raft_consensus.cc:2955] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 6 LEADER]: Committing config change with OpId 6.6: config changed from index -1 to 6, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } }
I20260709 12:21:06.146358 17835 raft_consensus.cc:2955] T ea7da467817a4ba88aa5797c3f8f493f P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Committing config change with OpId 6.6: config changed from index -1 to 6, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } }
I20260709 12:21:06.165812 17759 raft_consensus.cc:2955] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Committing config change with OpId 6.6: config changed from index -1 to 6, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } }
I20260709 12:21:06.189190 17585 catalog_manager.cc:5697] T ea7da467817a4ba88aa5797c3f8f493f P e063541542a94969898868bf6eb5ce1b reported cstate change: config changed from index -1 to 6, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New cstate: current_term: 6 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } }
I20260709 12:21:06.214111 19454 ts_tablet_manager.cc:933] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da: Initiating tablet copy from peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:06.247210 19455 ts_tablet_manager.cc:933] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da: Initiating tablet copy from peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:06.250569 19455 tablet_copy_client.cc:323] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Beginning tablet copy session from remote peer at address 127.16.29.67:36837
I20260709 12:21:06.252737 17848 tablet_copy_service.cc:140] P e063541542a94969898868bf6eb5ce1b: Received BeginTabletCopySession request for tablet 8cab6f01c6a2485cb5024322a1bd5462 from peer 334a4e1f59944544b3b3ff36616b57da ({username='slave'} at 127.0.0.1:37994)
I20260709 12:21:06.253283 17848 tablet_copy_service.cc:161] P e063541542a94969898868bf6eb5ce1b: Beginning new tablet copy session on tablet 8cab6f01c6a2485cb5024322a1bd5462 from peer 334a4e1f59944544b3b3ff36616b57da at {username='slave'} at 127.0.0.1:37994: session id = 334a4e1f59944544b3b3ff36616b57da-8cab6f01c6a2485cb5024322a1bd5462
I20260709 12:21:06.261381 17848 tablet_copy_source_session.cc:215] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:21:06.264707 19455 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8cab6f01c6a2485cb5024322a1bd5462. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:06.270291 17759 raft_consensus.cc:1275] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:21:06.271481 19454 tablet_copy_client.cc:323] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Beginning tablet copy session from remote peer at address 127.16.29.67:36837
I20260709 12:21:06.272850 19426 consensus_queue.cc:1048] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:21:06.273321 17848 tablet_copy_service.cc:140] P e063541542a94969898868bf6eb5ce1b: Received BeginTabletCopySession request for tablet c64345b0a2614ba4a382b83feb235cad from peer 334a4e1f59944544b3b3ff36616b57da ({username='slave'} at 127.0.0.1:37994)
I20260709 12:21:06.273828 17848 tablet_copy_service.cc:161] P e063541542a94969898868bf6eb5ce1b: Beginning new tablet copy session on tablet c64345b0a2614ba4a382b83feb235cad from peer 334a4e1f59944544b3b3ff36616b57da at {username='slave'} at 127.0.0.1:37994: session id = 334a4e1f59944544b3b3ff36616b57da-c64345b0a2614ba4a382b83feb235cad
I20260709 12:21:06.299142 17837 raft_consensus.cc:1275] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:21:06.302722 17848 tablet_copy_source_session.cc:215] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:21:06.308727 19454 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c64345b0a2614ba4a382b83feb235cad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:06.308547 19430 consensus_queue.cc:1048] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:21:06.362771 19461 ts_tablet_manager.cc:933] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da: Initiating tablet copy from peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:06.365710 19454 tablet_copy_client.cc:806] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Starting download of 0 data blocks...
I20260709 12:21:06.366482 19454 tablet_copy_client.cc:670] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Starting download of 1 WAL segments...
I20260709 12:21:06.369211 19455 tablet_copy_client.cc:806] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Starting download of 0 data blocks...
I20260709 12:21:06.373948 19461 tablet_copy_client.cc:323] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Beginning tablet copy session from remote peer at address 127.16.29.67:36837
I20260709 12:21:06.375948 17848 tablet_copy_service.cc:140] P e063541542a94969898868bf6eb5ce1b: Received BeginTabletCopySession request for tablet 6fcb5bdaa4be4d619bb945ffd44090ba from peer 334a4e1f59944544b3b3ff36616b57da ({username='slave'} at 127.0.0.1:37994)
I20260709 12:21:06.376515 17848 tablet_copy_service.cc:161] P e063541542a94969898868bf6eb5ce1b: Beginning new tablet copy session on tablet 6fcb5bdaa4be4d619bb945ffd44090ba from peer 334a4e1f59944544b3b3ff36616b57da at {username='slave'} at 127.0.0.1:37994: session id = 334a4e1f59944544b3b3ff36616b57da-6fcb5bdaa4be4d619bb945ffd44090ba
I20260709 12:21:06.377242 19455 tablet_copy_client.cc:670] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Starting download of 1 WAL segments...
I20260709 12:21:06.388199 19455 tablet_copy_client.cc:538] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:21:06.396026 17848 tablet_copy_source_session.cc:215] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:21:06.396615 19454 tablet_copy_client.cc:538] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:21:06.407378 19461 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6fcb5bdaa4be4d619bb945ffd44090ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:06.414731 19455 tablet_bootstrap.cc:492] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da: Bootstrap starting.
I20260709 12:21:06.416621 19454 tablet_bootstrap.cc:492] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da: Bootstrap starting.
I20260709 12:21:06.486804 19461 tablet_copy_client.cc:806] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Starting download of 0 data blocks...
I20260709 12:21:06.487578 19461 tablet_copy_client.cc:670] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Starting download of 1 WAL segments...
I20260709 12:21:06.527359 19461 tablet_copy_client.cc:538] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:21:06.540472 19461 tablet_bootstrap.cc:492] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da: Bootstrap starting.
I20260709 12:21:06.565371 19454 tablet_bootstrap.cc:492] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:21:06.566475 19454 tablet_bootstrap.cc:492] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da: Bootstrap complete.
I20260709 12:21:06.567288 19454 ts_tablet_manager.cc:1403] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da: Time spent bootstrapping tablet: real 0.151s	user 0.018s	sys 0.010s
I20260709 12:21:06.570338 19455 tablet_bootstrap.cc:492] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:21:06.571300 19455 tablet_bootstrap.cc:492] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da: Bootstrap complete.
I20260709 12:21:06.574484 17685 raft_consensus.cc:1275] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260709 12:21:06.576138 19448 consensus_queue.cc:1048] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 12:21:06.583946 19455 ts_tablet_manager.cc:1403] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da: Time spent bootstrapping tablet: real 0.169s	user 0.020s	sys 0.018s
I20260709 12:21:06.582391 19454 raft_consensus.cc:359] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:21:06.589267 18371 raft_consensus.cc:1275] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260709 12:21:06.590687 19401 consensus_queue.cc:1048] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 12:21:06.586930 19454 raft_consensus.cc:740] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Initialized, Role: LEARNER
I20260709 12:21:06.606227 19455 raft_consensus.cc:359] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 3 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:21:06.607137 19455 raft_consensus.cc:740] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Initialized, Role: LEARNER
I20260709 12:21:06.602913 19454 consensus_queue.cc:260] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:21:06.611598 19461 tablet_bootstrap.cc:492] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:21:06.611920 19454 ts_tablet_manager.cc:1434] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da: Time spent starting tablet: real 0.044s	user 0.009s	sys 0.001s
I20260709 12:21:06.607817 19455 consensus_queue.cc:260] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:21:06.620774 19461 tablet_bootstrap.cc:492] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da: Bootstrap complete.
I20260709 12:21:06.621574 19461 ts_tablet_manager.cc:1403] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da: Time spent bootstrapping tablet: real 0.081s	user 0.017s	sys 0.010s
I20260709 12:21:06.622336 19455 ts_tablet_manager.cc:1434] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da: Time spent starting tablet: real 0.038s	user 0.007s	sys 0.001s
I20260709 12:21:06.625484 17848 tablet_copy_service.cc:342] P e063541542a94969898868bf6eb5ce1b: Request end of tablet copy session 334a4e1f59944544b3b3ff36616b57da-c64345b0a2614ba4a382b83feb235cad received from {username='slave'} at 127.0.0.1:37994
I20260709 12:21:06.634748 17848 tablet_copy_service.cc:434] P e063541542a94969898868bf6eb5ce1b: ending tablet copy session 334a4e1f59944544b3b3ff36616b57da-c64345b0a2614ba4a382b83feb235cad on tablet c64345b0a2614ba4a382b83feb235cad with peer 334a4e1f59944544b3b3ff36616b57da
I20260709 12:21:06.637936 19461 raft_consensus.cc:359] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:21:06.641390 19454 ts_tablet_manager.cc:933] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da: Initiating tablet copy from peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071)
I20260709 12:21:06.644106 17848 tablet_copy_service.cc:342] P e063541542a94969898868bf6eb5ce1b: Request end of tablet copy session 334a4e1f59944544b3b3ff36616b57da-8cab6f01c6a2485cb5024322a1bd5462 received from {username='slave'} at 127.0.0.1:37994
I20260709 12:21:06.644641 17848 tablet_copy_service.cc:434] P e063541542a94969898868bf6eb5ce1b: ending tablet copy session 334a4e1f59944544b3b3ff36616b57da-8cab6f01c6a2485cb5024322a1bd5462 on tablet 8cab6f01c6a2485cb5024322a1bd5462 with peer 334a4e1f59944544b3b3ff36616b57da
I20260709 12:21:06.654886 18362 raft_consensus.cc:1217] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 3 LEARNER]: Deduplicated request from leader. Original: 3.3->[3.4-3.4]   Dedup: 3.4->[]
I20260709 12:21:06.639242 19461 raft_consensus.cc:740] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Initialized, Role: LEARNER
I20260709 12:21:06.659334 19454 tablet_copy_client.cc:323] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Beginning tablet copy session from remote peer at address 127.16.29.65:33071
I20260709 12:21:06.658329 19461 consensus_queue.cc:260] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:21:06.672765 19461 ts_tablet_manager.cc:1434] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da: Time spent starting tablet: real 0.039s	user 0.005s	sys 0.001s
I20260709 12:21:06.675035 17699 tablet_copy_service.cc:140] P 0da169ff82674dc1947055ce43de68a3: Received BeginTabletCopySession request for tablet ea7da467817a4ba88aa5797c3f8f493f from peer 334a4e1f59944544b3b3ff36616b57da ({username='slave'} at 127.0.0.1:44898)
I20260709 12:21:06.675662 17848 tablet_copy_service.cc:342] P e063541542a94969898868bf6eb5ce1b: Request end of tablet copy session 334a4e1f59944544b3b3ff36616b57da-6fcb5bdaa4be4d619bb945ffd44090ba received from {username='slave'} at 127.0.0.1:37994
I20260709 12:21:06.676180 17848 tablet_copy_service.cc:434] P e063541542a94969898868bf6eb5ce1b: ending tablet copy session 334a4e1f59944544b3b3ff36616b57da-6fcb5bdaa4be4d619bb945ffd44090ba on tablet 6fcb5bdaa4be4d619bb945ffd44090ba with peer 334a4e1f59944544b3b3ff36616b57da
I20260709 12:21:06.675601 17699 tablet_copy_service.cc:161] P 0da169ff82674dc1947055ce43de68a3: Beginning new tablet copy session on tablet ea7da467817a4ba88aa5797c3f8f493f from peer 334a4e1f59944544b3b3ff36616b57da at {username='slave'} at 127.0.0.1:44898: session id = 334a4e1f59944544b3b3ff36616b57da-ea7da467817a4ba88aa5797c3f8f493f
I20260709 12:21:06.691193 17699 tablet_copy_source_session.cc:215] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:21:06.703598 16501 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:21:06.799226 19454 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea7da467817a4ba88aa5797c3f8f493f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:06.829321 19454 tablet_copy_client.cc:806] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Starting download of 0 data blocks...
I20260709 12:21:06.829962 19454 tablet_copy_client.cc:670] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Starting download of 1 WAL segments...
I20260709 12:21:06.862787 19454 tablet_copy_client.cc:538] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:21:06.876003 19454 tablet_bootstrap.cc:492] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da: Bootstrap starting.
I20260709 12:21:06.881467 18362 raft_consensus.cc:1217] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:21:06.935461 19454 tablet_bootstrap.cc:492] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:21:06.936985 19454 tablet_bootstrap.cc:492] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da: Bootstrap complete.
I20260709 12:21:06.938162 19454 ts_tablet_manager.cc:1403] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da: Time spent bootstrapping tablet: real 0.062s	user 0.035s	sys 0.007s
I20260709 12:21:06.959033 19454 raft_consensus.cc:359] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 6 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:21:06.964607 19454 raft_consensus.cc:740] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 6 LEARNER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Initialized, Role: LEARNER
I20260709 12:21:06.965431 19454 consensus_queue.cc:260] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:21:06.972669 19454 ts_tablet_manager.cc:1434] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da: Time spent starting tablet: real 0.034s	user 0.009s	sys 0.000s
I20260709 12:21:06.989460 17699 tablet_copy_service.cc:342] P 0da169ff82674dc1947055ce43de68a3: Request end of tablet copy session 334a4e1f59944544b3b3ff36616b57da-ea7da467817a4ba88aa5797c3f8f493f received from {username='slave'} at 127.0.0.1:44898
I20260709 12:21:06.989944 17699 tablet_copy_service.cc:434] P 0da169ff82674dc1947055ce43de68a3: ending tablet copy session 334a4e1f59944544b3b3ff36616b57da-ea7da467817a4ba88aa5797c3f8f493f on tablet ea7da467817a4ba88aa5797c3f8f493f with peer 334a4e1f59944544b3b3ff36616b57da
I20260709 12:21:07.022399 17687 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "8a208e793881438b975b83755dc05cfa"
dest_uuid: "0da169ff82674dc1947055ce43de68a3"
mode: GRACEFUL
new_leader_uuid: "e063541542a94969898868bf6eb5ce1b"
 from {username='slave'} at 127.0.0.1:54540
I20260709 12:21:07.023079 17687 raft_consensus.cc:606] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 1 LEADER]: Received request to transfer leadership to TS e063541542a94969898868bf6eb5ce1b
I20260709 12:21:07.033818 17757 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995"
dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
mode: GRACEFUL
new_leader_uuid: "0da169ff82674dc1947055ce43de68a3"
 from {username='slave'} at 127.0.0.1:52946
I20260709 12:21:07.034422 17757 raft_consensus.cc:606] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 LEADER]: Received request to transfer leadership to TS 0da169ff82674dc1947055ce43de68a3
I20260709 12:21:07.039829 16501 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260709 12:21:07.041535 16501 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:21:07.044755 18371 raft_consensus.cc:1217] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 6 LEARNER]: Deduplicated request from leader. Original: 6.5->[6.6-6.6]   Dedup: 6.6->[]
I20260709 12:21:07.053342 16501 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:21:07.080094 19410 raft_consensus.cc:1064] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b: attempting to promote NON_VOTER 334a4e1f59944544b3b3ff36616b57da to VOTER
I20260709 12:21:07.083251 19410 consensus_queue.cc:237] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 2, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:07.163945 18371 raft_consensus.cc:1217] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:21:07.176050 19431 raft_consensus.cc:993] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3: : Instructing follower e063541542a94969898868bf6eb5ce1b to start an election
I20260709 12:21:07.176533 19431 raft_consensus.cc:1081] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 1 LEADER]: Signalling peer e063541542a94969898868bf6eb5ce1b to start an election
I20260709 12:21:07.187839 17837 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "8a208e793881438b975b83755dc05cfa"
dest_uuid: "e063541542a94969898868bf6eb5ce1b"
 from {username='slave'} at 127.0.0.1:35040
I20260709 12:21:07.188434 17837 raft_consensus.cc:493] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:21:07.188750 17837 raft_consensus.cc:3060] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:07.237152 17837 raft_consensus.cc:515] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:07.303105 17687 raft_consensus.cc:1275] T 8cab6f01c6a2485cb5024322a1bd5462 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 12:21:07.311430 19478 consensus_queue.cc:1048] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:21:07.326578 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:07.327435 17686 raft_consensus.cc:3055] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:21:07.327847 17686 raft_consensus.cc:740] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:21:07.328819 17686 consensus_queue.cc:260] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:07.330035 17686 raft_consensus.cc:3060] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:07.348774 17837 leader_election.cc:290] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 2 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:07.371071 17757 raft_consensus.cc:1275] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 12:21:07.418785 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:07.419435 18374 raft_consensus.cc:3060] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:07.419673 18371 raft_consensus.cc:1275] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 3 LEARNER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 12:21:07.432610 18374 raft_consensus.cc:2468] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 2.
I20260709 12:21:07.439136 17686 raft_consensus.cc:2468] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 2.
I20260709 12:21:07.440668 17802 leader_election.cc:304] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:21:07.441725 19479 raft_consensus.cc:2804] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:21:07.442480 19479 raft_consensus.cc:697] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 2 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:21:07.446458 19472 consensus_queue.cc:1048] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:21:07.460578 19479 consensus_queue.cc:237] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:07.463688 19477 raft_consensus.cc:1064] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b: attempting to promote NON_VOTER 334a4e1f59944544b3b3ff36616b57da to VOTER
I20260709 12:21:07.540671 19477 consensus_queue.cc:237] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:07.547609 19426 raft_consensus.cc:1064] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3: attempting to promote NON_VOTER 334a4e1f59944544b3b3ff36616b57da to VOTER
I20260709 12:21:07.597621 19426 consensus_queue.cc:237] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 4, Current term: 6, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:07.555254 19478 raft_consensus.cc:2955] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 3 LEADER]: Committing config change with OpId 3.5: config changed from index 4 to 5, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:21:07.575588 18367 raft_consensus.cc:2955] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 4 to 5, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:21:07.582481 19472 consensus_queue.cc:1048] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:21:07.590026 19375 raft_consensus.cc:993] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb: : Instructing follower 0da169ff82674dc1947055ce43de68a3 to start an election
I20260709 12:21:07.621019 19375 raft_consensus.cc:1081] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 LEADER]: Signalling peer 0da169ff82674dc1947055ce43de68a3 to start an election
W20260709 12:21:07.628250 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 44075.3us
I20260709 12:21:07.633736 17583 catalog_manager.cc:5697] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da reported cstate change: config changed from index 4 to 5, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:21:07.636425 17585 catalog_manager.cc:5697] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b reported cstate change: term changed from 1 to 2, leader changed from 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) to e063541542a94969898868bf6eb5ce1b (127.16.29.67). New cstate: current_term: 2 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:07.652012 18367 raft_consensus.cc:1275] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 6 LEARNER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260709 12:21:07.654716 19473 raft_consensus.cc:1064] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b: attempting to promote NON_VOTER 334a4e1f59944544b3b3ff36616b57da to VOTER
I20260709 12:21:07.655139 17831 raft_consensus.cc:1275] T ea7da467817a4ba88aa5797c3f8f493f P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260709 12:21:07.656523 17757 raft_consensus.cc:1275] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260709 12:21:07.657245 19473 consensus_queue.cc:237] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:07.659271 19430 consensus_queue.cc:1048] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 12:21:07.661580 19430 consensus_queue.cc:1048] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
W20260709 12:21:07.643721 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 54559.6us
I20260709 12:21:07.665409 19427 consensus_queue.cc:1048] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 12:21:07.695324 17687 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995"
dest_uuid: "0da169ff82674dc1947055ce43de68a3"
 from {username='slave'} at 127.0.0.1:37396
I20260709 12:21:07.696408 17687 raft_consensus.cc:493] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:21:07.697140 17687 raft_consensus.cc:3060] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:07.729399 19430 raft_consensus.cc:2955] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 6 LEADER]: Committing config change with OpId 6.7: config changed from index 6 to 7, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:21:07.737287 18374 raft_consensus.cc:2955] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Committing config change with OpId 6.7: config changed from index 6 to 7, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:21:07.738433 17759 raft_consensus.cc:2955] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 4 to 5, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:21:07.758457 17757 raft_consensus.cc:2955] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Committing config change with OpId 6.7: config changed from index 6 to 7, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:21:07.803323 17687 raft_consensus.cc:515] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:07.808473 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:07.809192 17763 raft_consensus.cc:3055] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:21:07.809559 17763 raft_consensus.cc:740] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:21:07.810369 17763 consensus_queue.cc:260] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:07.813356 17763 raft_consensus.cc:3060] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:07.816368 17687 leader_election.cc:290] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 2 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:07.822459 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:07.823191 17831 raft_consensus.cc:3060] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:07.822628 17835 raft_consensus.cc:2955] T ea7da467817a4ba88aa5797c3f8f493f P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Committing config change with OpId 6.7: config changed from index 6 to 7, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:21:07.829909 17687 raft_consensus.cc:2955] T 8cab6f01c6a2485cb5024322a1bd5462 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 4 to 5, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:21:07.831338 17831 raft_consensus.cc:2468] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 2.
I20260709 12:21:07.838264 17652 leader_election.cc:304] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:21:07.844269 19426 raft_consensus.cc:2804] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:21:07.866636 17584 catalog_manager.cc:5697] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 reported cstate change: config changed from index 6 to 7, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New cstate: current_term: 6 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:21:07.869192 18367 raft_consensus.cc:1275] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 1 LEARNER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:21:07.883049 18374 raft_consensus.cc:1275] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 1 LEARNER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:21:07.897742 17763 raft_consensus.cc:2468] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 2.
I20260709 12:21:07.908048 19472 consensus_queue.cc:1048] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:21:07.913659 17685 raft_consensus.cc:1275] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:21:07.939368 17685 raft_consensus.cc:1275] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:21:07.893846 19426 raft_consensus.cc:697] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 2 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:21:07.950613 19410 consensus_queue.cc:1048] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:21:07.954854 17685 raft_consensus.cc:1275] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:21:07.910616 19472 consensus_queue.cc:1048] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.060s
I20260709 12:21:07.971312 19426 consensus_queue.cc:237] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:21:07.974946 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 90994.6us
I20260709 12:21:07.978931 19410 consensus_queue.cc:1048] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
W20260709 12:21:08.005502 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 166364us
I20260709 12:21:08.012547 17757 raft_consensus.cc:1275] T c64345b0a2614ba4a382b83feb235cad P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:21:08.012557 17763 raft_consensus.cc:1275] T 6fcb5bdaa4be4d619bb945ffd44090ba P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:21:07.977484 19478 consensus_queue.cc:1048] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.044s
I20260709 12:21:08.084173 19479 consensus_queue.cc:1048] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.015s
I20260709 12:21:08.100646 17687 consensus_queue.cc:237] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:21:08.111526 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 68079.4us
W20260709 12:21:08.132223 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 92352.9us
W20260709 12:21:08.179226 19483 log.cc:927] Time spent T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.051s	user 0.001s	sys 0.000s
W20260709 12:21:08.211565 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 44311us
I20260709 12:21:08.169157 19479 consensus_queue.cc:1048] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.041s
W20260709 12:21:08.221346 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 54199.2us
W20260709 12:21:08.264341 19502 log.cc:927] Time spent T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.094s	user 0.001s	sys 0.000s
W20260709 12:21:08.307273 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 88736.5us
W20260709 12:21:08.333307 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 124788us
W20260709 12:21:08.340793 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 215574us
W20260709 12:21:08.377282 19518 log.cc:927] Time spent T c64345b0a2614ba4a382b83feb235cad P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.112s	user 0.001s	sys 0.000s
W20260709 12:21:08.395383 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 61296.6us
I20260709 12:21:08.416383 18367 raft_consensus.cc:1275] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
W20260709 12:21:08.448230 17654 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:08.459295 19520 consensus_queue.cc:1048] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:21:08.462997 19410 raft_consensus.cc:2955] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:21:08.471064 18374 raft_consensus.cc:2955] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:21:08.485168 17759 raft_consensus.cc:1275] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260709 12:21:08.487524 17762 raft_consensus.cc:1275] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:21:08.222890 16501 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:21:08.538514 18371 raft_consensus.cc:1275] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260709 12:21:08.553839 19528 raft_consensus.cc:2955] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:21:08.612010 17583 catalog_manager.cc:5697] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 reported cstate change: term changed from 1 to 2, leader changed from e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66) to 0da169ff82674dc1947055ce43de68a3 (127.16.29.65). New cstate: current_term: 2 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: UNKNOWN } } }
W20260709 12:21:08.646652 19531 log.cc:927] Time spent T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.134s	user 0.000s	sys 0.001s
I20260709 12:21:08.592507 19431 consensus_queue.cc:1048] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.039s
W20260709 12:21:08.661700 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 54828.5us
I20260709 12:21:08.662102 17831 raft_consensus.cc:1275] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:21:08.669466 17834 consensus_queue.cc:237] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:08.654558 19427 consensus_queue.cc:1048] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.019s
I20260709 12:21:08.681865 18371 raft_consensus.cc:2955] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:21:08.699383 17763 raft_consensus.cc:2955] T 6fcb5bdaa4be4d619bb945ffd44090ba P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:21:08.696799 17756 raft_consensus.cc:2955] T c64345b0a2614ba4a382b83feb235cad P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:21:08.712615 19504 consensus_queue.cc:1048] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:21:08.726657 17684 raft_consensus.cc:2955] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:21:08.731580 19504 consensus_queue.cc:1048] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:21:08.765515 17834 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9"
dest_uuid: "e063541542a94969898868bf6eb5ce1b"
mode: GRACEFUL
new_leader_uuid: "0da169ff82674dc1947055ce43de68a3"
 from {username='slave'} at 127.0.0.1:59820
I20260709 12:21:08.766167 17834 raft_consensus.cc:606] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 1 LEADER]: Received request to transfer leadership to TS 0da169ff82674dc1947055ce43de68a3
I20260709 12:21:08.769333 18363 raft_consensus.cc:1275] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260709 12:21:08.770785 19503 raft_consensus.cc:2955] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 6 LEADER]: Committing config change with OpId 6.8: config changed from index 7 to 8, VOTER e063541542a94969898868bf6eb5ce1b (127.16.29.67) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:21:08.777128 17756 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d"
dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
mode: GRACEFUL
new_leader_uuid: "e063541542a94969898868bf6eb5ce1b"
 from {username='slave'} at 127.0.0.1:52946
I20260709 12:21:08.777833 17756 raft_consensus.cc:606] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 3 LEADER]: Received request to transfer leadership to TS e063541542a94969898868bf6eb5ce1b
I20260709 12:21:08.779174 16501 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260709 12:21:08.780894 16501 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
W20260709 12:21:08.796069 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 199582us
I20260709 12:21:08.800534 17764 raft_consensus.cc:2955] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Committing config change with OpId 6.8: config changed from index 7 to 8, VOTER e063541542a94969898868bf6eb5ce1b (127.16.29.67) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:21:08.802239 17756 raft_consensus.cc:1275] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260709 12:21:08.812009 19479 consensus_queue.cc:1048] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:21:08.882936 17687 raft_consensus.cc:2955] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:21:08.895704 16501 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:21:08.897652 19520 consensus_queue.cc:1048] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 12:21:08.975071 17572 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet ea7da467817a4ba88aa5797c3f8f493f with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:21:08.967257 18365 raft_consensus.cc:2955] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Committing config change with OpId 6.8: config changed from index 7 to 8, VOTER e063541542a94969898868bf6eb5ce1b (127.16.29.67) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:21:09.007902 19473 raft_consensus.cc:2955] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 3 LEADER]: Committing config change with OpId 3.6: config changed from index 5 to 6, VOTER 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:21:09.023401 17757 raft_consensus.cc:2955] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Committing config change with OpId 3.6: config changed from index 5 to 6, VOTER 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:21:09.033864 19476 raft_consensus.cc:993] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb: : Instructing follower e063541542a94969898868bf6eb5ce1b to start an election
I20260709 12:21:09.037667 19514 raft_consensus.cc:993] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b: : Instructing follower 0da169ff82674dc1947055ce43de68a3 to start an election
I20260709 12:21:09.038429 19514 raft_consensus.cc:1081] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 1 LEADER]: Signalling peer 0da169ff82674dc1947055ce43de68a3 to start an election
I20260709 12:21:09.044864 19476 raft_consensus.cc:1081] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 3 LEADER]: Signalling peer e063541542a94969898868bf6eb5ce1b to start an election
I20260709 12:21:09.047227 17684 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9"
dest_uuid: "0da169ff82674dc1947055ce43de68a3"
 from {username='slave'} at 127.0.0.1:37402
I20260709 12:21:09.047794 17684 raft_consensus.cc:493] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:21:09.048264 17684 raft_consensus.cc:3060] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:09.071943 17838 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d"
dest_uuid: "e063541542a94969898868bf6eb5ce1b"
 from {username='slave'} at 127.0.0.1:35042
I20260709 12:21:09.072569 17838 raft_consensus.cc:493] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:21:09.072917 17838 raft_consensus.cc:3060] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:09.081307 17684 raft_consensus.cc:515] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:09.083350 17684 leader_election.cc:290] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 2 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:09.083755 17838 raft_consensus.cc:515] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:09.085255 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:09.085844 18371 raft_consensus.cc:3060] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:09.087281 17838 leader_election.cc:290] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 4 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:09.088137 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:09.089021 17834 raft_consensus.cc:3055] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:21:09.089399 17834 raft_consensus.cc:740] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:21:09.090253 17834 consensus_queue.cc:260] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:09.091768 17834 raft_consensus.cc:3060] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:09.087981 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: true dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:09.094362 17757 raft_consensus.cc:3055] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:21:09.094574 17757 raft_consensus.cc:740] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 3 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:21:09.108796 17757 consensus_queue.cc:260] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:09.110358 18371 raft_consensus.cc:2468] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 2.
I20260709 12:21:09.111737 17654 leader_election.cc:304] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:21:09.116871 19539 raft_consensus.cc:2804] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:21:09.117486 19539 raft_consensus.cc:697] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 2 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:21:09.117955 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: true dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:09.118654 18363 raft_consensus.cc:3060] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:09.104377 18365 raft_consensus.cc:2955] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Committing config change with OpId 3.6: config changed from index 5 to 6, VOTER 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:21:09.118412 19539 consensus_queue.cc:237] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:09.130295 17757 raft_consensus.cc:3060] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:09.114948 17834 raft_consensus.cc:2468] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 2.
I20260709 12:21:09.141037 18363 raft_consensus.cc:2468] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 4.
I20260709 12:21:09.143257 17803 leader_election.cc:304] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:21:09.144232 19477 raft_consensus.cc:2804] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:21:09.144845 19477 raft_consensus.cc:697] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 4 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:21:09.145921 19477 consensus_queue.cc:237] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:09.148975 17584 catalog_manager.cc:5697] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b reported cstate change: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:21:09.162190 17571 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 8cab6f01c6a2485cb5024322a1bd5462 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:21:09.175719 17757 raft_consensus.cc:2468] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 4.
I20260709 12:21:09.184829 17834 consensus_queue.cc:237] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:09.195102 17584 catalog_manager.cc:5697] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb reported cstate change: config changed from index 7 to 8, VOTER e063541542a94969898868bf6eb5ce1b (127.16.29.67) evicted. New cstate: current_term: 6 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:21:09.202498 17584 catalog_manager.cc:5697] T c64345b0a2614ba4a382b83feb235cad P e2909cff1da44ad387f96a8ae3aa5afb reported cstate change: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:21:09.211625 18371 raft_consensus.cc:1275] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:21:09.214037 19479 consensus_queue.cc:1048] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:21:09.204583 17584 catalog_manager.cc:5697] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb reported cstate change: config changed from index 5 to 6, VOTER 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) evicted. New cstate: current_term: 3 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:21:09.227833 17684 raft_consensus.cc:1275] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:21:09.230607 19477 consensus_queue.cc:1048] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:21:09.239794 19410 raft_consensus.cc:2955] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:21:09.241024 17684 raft_consensus.cc:2955] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:21:09.241024 18365 raft_consensus.cc:2955] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:21:09.263639 17571 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 6fcb5bdaa4be4d619bb945ffd44090ba with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:21:09.275398 17585 catalog_manager.cc:5697] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 reported cstate change: term changed from 1 to 2, leader changed from e063541542a94969898868bf6eb5ce1b (127.16.29.67) to 0da169ff82674dc1947055ce43de68a3 (127.16.29.65). New cstate: current_term: 2 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20260709 12:21:09.411226 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 93516.3us
I20260709 12:21:09.423411 19410 raft_consensus.cc:493] T ea7da467817a4ba88aa5797c3f8f493f P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:21:09.424106 19410 raft_consensus.cc:515] T ea7da467817a4ba88aa5797c3f8f493f P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:09.428231 19410 leader_election.cc:290] T ea7da467817a4ba88aa5797c3f8f493f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:09.431118 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:09.431664 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:09.434540 17802 leader_election.cc:304] T ea7da467817a4ba88aa5797c3f8f493f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:21:09.435693 19477 raft_consensus.cc:2749] T ea7da467817a4ba88aa5797c3f8f493f P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:21:09.439227 17821 tablet_service.cc:1558] Processing DeleteTablet for tablet ea7da467817a4ba88aa5797c3f8f493f with delete_type TABLET_DATA_TOMBSTONED (TS e063541542a94969898868bf6eb5ce1b not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:35032
I20260709 12:21:09.461575 17583 catalog_manager.cc:5697] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b reported cstate change: term changed from 3 to 4, leader changed from e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66) to e063541542a94969898868bf6eb5ce1b (127.16.29.67). New cstate: current_term: 4 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:09.478324 17586 catalog_manager.cc:5697] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 reported cstate change: config changed from index 3 to 4, VOTER e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66) evicted. New cstate: current_term: 1 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:21:09.491242 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:21:09.493081 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 44969us
I20260709 12:21:09.515391 17834 consensus_queue.cc:237] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:09.540977 17676 tablet_service.cc:1558] Processing DeleteTablet for tablet 8cab6f01c6a2485cb5024322a1bd5462 with delete_type TABLET_DATA_TOMBSTONED (TS 0da169ff82674dc1947055ce43de68a3 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:37384
I20260709 12:21:09.554702 17753 tablet_service.cc:1558] Processing DeleteTablet for tablet 6fcb5bdaa4be4d619bb945ffd44090ba with delete_type TABLET_DATA_TOMBSTONED (TS e2909cff1da44ad387f96a8ae3aa5afb not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:38774
I20260709 12:21:09.588340 19557 tablet_replica.cc:333] T ea7da467817a4ba88aa5797c3f8f493f P e063541542a94969898868bf6eb5ce1b: stopping tablet replica
I20260709 12:21:09.589493 19557 raft_consensus.cc:2243] T ea7da467817a4ba88aa5797c3f8f493f P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Raft consensus shutting down.
W20260709 12:21:09.592010 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 64682.5us
I20260709 12:21:09.597934 19557 raft_consensus.cc:2272] T ea7da467817a4ba88aa5797c3f8f493f P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:09.622912 19557 ts_tablet_manager.cc:1916] T ea7da467817a4ba88aa5797c3f8f493f P e063541542a94969898868bf6eb5ce1b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20260709 12:21:09.656659 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 53892.6us
I20260709 12:21:09.672549 18365 raft_consensus.cc:1275] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:21:09.678328 17838 raft_consensus.cc:1275] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:21:09.698349 17682 raft_consensus.cc:1275] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:21:09.685035 19508 consensus_queue.cc:1048] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.015s
I20260709 12:21:09.703118 19473 consensus_queue.cc:1048] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:21:09.704635 17760 raft_consensus.cc:1275] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20260709 12:21:09.717188 19478 consensus_queue.cc:1048] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 12:21:09.719868 19557 ts_tablet_manager.cc:1929] T ea7da467817a4ba88aa5797c3f8f493f P e063541542a94969898868bf6eb5ce1b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 6.7
I20260709 12:21:09.720351 19557 log.cc:1199] T ea7da467817a4ba88aa5797c3f8f493f P e063541542a94969898868bf6eb5ce1b: Deleting WAL directory at /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599551912677-16501-0/minicluster-data/ts-2-root/wals/ea7da467817a4ba88aa5797c3f8f493f
I20260709 12:21:09.736431 18363 raft_consensus.cc:1275] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:21:09.739037 17571 catalog_manager.cc:5028] TS e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837): tablet ea7da467817a4ba88aa5797c3f8f493f (table test-workload [id=ddc5b74250df4c65909b73b3742d615f]) successfully deleted
I20260709 12:21:09.739308 19563 consensus_queue.cc:1048] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:21:09.772280 19503 consensus_queue.cc:1048] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:21:09.781697 19560 tablet_replica.cc:333] T 8cab6f01c6a2485cb5024322a1bd5462 P 0da169ff82674dc1947055ce43de68a3: stopping tablet replica
I20260709 12:21:09.783635 18362 raft_consensus.cc:1275] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20260709 12:21:09.789130 19563 consensus_queue.cc:1048] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 12:21:09.812944 19564 tablet_replica.cc:333] T 6fcb5bdaa4be4d619bb945ffd44090ba P e2909cff1da44ad387f96a8ae3aa5afb: stopping tablet replica
I20260709 12:21:09.814016 19564 raft_consensus.cc:2243] T 6fcb5bdaa4be4d619bb945ffd44090ba P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:21:09.831637 19560 raft_consensus.cc:2243] T 8cab6f01c6a2485cb5024322a1bd5462 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:21:09.832523 19560 raft_consensus.cc:2272] T 8cab6f01c6a2485cb5024322a1bd5462 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:09.823400 19564 raft_consensus.cc:2272] T 6fcb5bdaa4be4d619bb945ffd44090ba P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:09.841049 19560 ts_tablet_manager.cc:1916] T 8cab6f01c6a2485cb5024322a1bd5462 P 0da169ff82674dc1947055ce43de68a3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:21:09.850050 19520 raft_consensus.cc:2955] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:21:09.875169 19564 ts_tablet_manager.cc:1916] T 6fcb5bdaa4be4d619bb945ffd44090ba P e2909cff1da44ad387f96a8ae3aa5afb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:21:09.887734 17687 raft_consensus.cc:2955] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:21:09.929714 18365 raft_consensus.cc:2955] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:21:09.960940 17571 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet c64345b0a2614ba4a382b83feb235cad with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:21:09.995783 17583 catalog_manager.cc:5697] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b reported cstate change: config changed from index 3 to 4, VOTER e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66) evicted. New cstate: current_term: 1 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:21:10.139402 19564 ts_tablet_manager.cc:1929] T 6fcb5bdaa4be4d619bb945ffd44090ba P e2909cff1da44ad387f96a8ae3aa5afb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:21:10.152143 19564 log.cc:1199] T 6fcb5bdaa4be4d619bb945ffd44090ba P e2909cff1da44ad387f96a8ae3aa5afb: Deleting WAL directory at /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599551912677-16501-0/minicluster-data/ts-1-root/wals/6fcb5bdaa4be4d619bb945ffd44090ba
I20260709 12:21:10.159477 17571 catalog_manager.cc:5028] TS e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): tablet 6fcb5bdaa4be4d619bb945ffd44090ba (table test-workload [id=ddc5b74250df4c65909b73b3742d615f]) successfully deleted
I20260709 12:21:10.161353 19560 ts_tablet_manager.cc:1929] T 8cab6f01c6a2485cb5024322a1bd5462 P 0da169ff82674dc1947055ce43de68a3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.5
I20260709 12:21:10.161849 19560 log.cc:1199] T 8cab6f01c6a2485cb5024322a1bd5462 P 0da169ff82674dc1947055ce43de68a3: Deleting WAL directory at /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599551912677-16501-0/minicluster-data/ts-0-root/wals/8cab6f01c6a2485cb5024322a1bd5462
I20260709 12:21:10.163381 17572 catalog_manager.cc:5028] TS 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): tablet 8cab6f01c6a2485cb5024322a1bd5462 (table test-workload [id=ddc5b74250df4c65909b73b3742d615f]) successfully deleted
I20260709 12:21:10.194396 17753 tablet_service.cc:1558] Processing DeleteTablet for tablet c64345b0a2614ba4a382b83feb235cad with delete_type TABLET_DATA_TOMBSTONED (TS e2909cff1da44ad387f96a8ae3aa5afb not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:38774
I20260709 12:21:10.195351 19564 tablet_replica.cc:333] T c64345b0a2614ba4a382b83feb235cad P e2909cff1da44ad387f96a8ae3aa5afb: stopping tablet replica
I20260709 12:21:10.196357 19564 raft_consensus.cc:2243] T c64345b0a2614ba4a382b83feb235cad P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:21:10.197099 19564 raft_consensus.cc:2272] T c64345b0a2614ba4a382b83feb235cad P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:10.202483 19564 ts_tablet_manager.cc:1916] T c64345b0a2614ba4a382b83feb235cad P e2909cff1da44ad387f96a8ae3aa5afb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:21:10.233973 19564 ts_tablet_manager.cc:1929] T c64345b0a2614ba4a382b83feb235cad P e2909cff1da44ad387f96a8ae3aa5afb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:21:10.234407 19564 log.cc:1199] T c64345b0a2614ba4a382b83feb235cad P e2909cff1da44ad387f96a8ae3aa5afb: Deleting WAL directory at /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599551912677-16501-0/minicluster-data/ts-1-root/wals/c64345b0a2614ba4a382b83feb235cad
I20260709 12:21:10.236478 17571 catalog_manager.cc:5028] TS e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): tablet c64345b0a2614ba4a382b83feb235cad (table test-workload [id=ddc5b74250df4c65909b73b3742d615f]) successfully deleted
I20260709 12:21:10.279183 16501 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:21:10.344993 17686 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3"
dest_uuid: "0da169ff82674dc1947055ce43de68a3"
mode: GRACEFUL
new_leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
 from {username='slave'} at 127.0.0.1:54540
I20260709 12:21:10.345536 17686 raft_consensus.cc:606] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 2 LEADER]: Received request to transfer leadership to TS e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:10.348966 17686 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9"
dest_uuid: "0da169ff82674dc1947055ce43de68a3"
mode: GRACEFUL
new_leader_uuid: "334a4e1f59944544b3b3ff36616b57da"
 from {username='slave'} at 127.0.0.1:54540
I20260709 12:21:10.349565 17686 raft_consensus.cc:606] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 1 LEADER]: Received request to transfer leadership to TS 334a4e1f59944544b3b3ff36616b57da
I20260709 12:21:10.353698 17831 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad"
dest_uuid: "e063541542a94969898868bf6eb5ce1b"
mode: GRACEFUL
new_leader_uuid: "0da169ff82674dc1947055ce43de68a3"
 from {username='slave'} at 127.0.0.1:59820
I20260709 12:21:10.354279 17831 raft_consensus.cc:606] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 3 LEADER]: Received request to transfer leadership to TS 0da169ff82674dc1947055ce43de68a3
I20260709 12:21:10.355489 16501 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 3
I20260709 12:21:10.358233 16501 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:21:10.358992 16501 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:21:10.637698 19539 raft_consensus.cc:993] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3: : Instructing follower 334a4e1f59944544b3b3ff36616b57da to start an election
I20260709 12:21:10.638332 19559 raft_consensus.cc:1081] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 1 LEADER]: Signalling peer 334a4e1f59944544b3b3ff36616b57da to start an election
I20260709 12:21:10.640442 18363 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9"
dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
 from {username='slave'} at 127.0.0.1:59918
I20260709 12:21:10.641299 18363 raft_consensus.cc:493] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:21:10.641745 18363 raft_consensus.cc:3060] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:10.669981 18363 raft_consensus.cc:515] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:10.673480 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:10.674067 18363 leader_election.cc:290] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 2 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:10.674201 17686 raft_consensus.cc:3055] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:21:10.674603 17686 raft_consensus.cc:740] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:21:10.675390 17686 consensus_queue.cc:260] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:10.676555 17686 raft_consensus.cc:3060] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:10.673545 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:10.678295 17831 raft_consensus.cc:3060] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:10.685072 17831 raft_consensus.cc:2468] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 2.
I20260709 12:21:10.686626 18333 leader_election.cc:304] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:21:10.687523 19526 raft_consensus.cc:2804] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:21:10.688024 19526 raft_consensus.cc:697] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 2 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:21:10.688702 17686 raft_consensus.cc:2468] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 2.
I20260709 12:21:10.688913 19526 consensus_queue.cc:237] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:10.704447 17586 catalog_manager.cc:5697] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 1 to 2, leader changed from 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) to 334a4e1f59944544b3b3ff36616b57da (127.16.29.68). New cstate: current_term: 2 leader_uuid: "334a4e1f59944544b3b3ff36616b57da" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:21:10.811432 19473 raft_consensus.cc:993] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b: : Instructing follower 0da169ff82674dc1947055ce43de68a3 to start an election
I20260709 12:21:10.811971 19473 raft_consensus.cc:1081] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 3 LEADER]: Signalling peer 0da169ff82674dc1947055ce43de68a3 to start an election
I20260709 12:21:10.813983 17687 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad"
dest_uuid: "0da169ff82674dc1947055ce43de68a3"
 from {username='slave'} at 127.0.0.1:37402
I20260709 12:21:10.814570 17687 raft_consensus.cc:493] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:21:10.814929 17687 raft_consensus.cc:3060] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:10.825204 17687 raft_consensus.cc:515] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:10.827009 17687 leader_election.cc:290] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 4 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:10.827948 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:10.828357 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:10.828622 17757 raft_consensus.cc:3060] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:10.828979 17835 raft_consensus.cc:3055] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:21:10.829361 17835 raft_consensus.cc:740] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 3 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:21:10.830284 17835 consensus_queue.cc:260] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:10.832258 17835 raft_consensus.cc:3060] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:10.839231 17757 raft_consensus.cc:2468] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 4.
I20260709 12:21:10.840794 17652 leader_election.cc:304] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:21:10.841176 17835 raft_consensus.cc:2468] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 4.
I20260709 12:21:10.842617 19539 raft_consensus.cc:2804] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:21:10.843185 19539 raft_consensus.cc:697] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 4 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:21:10.844089 19539 consensus_queue.cc:237] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:10.861757 19539 raft_consensus.cc:993] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3: : Instructing follower e2909cff1da44ad387f96a8ae3aa5afb to start an election
I20260709 12:21:10.862453 19559 raft_consensus.cc:1081] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 2 LEADER]: Signalling peer e2909cff1da44ad387f96a8ae3aa5afb to start an election
I20260709 12:21:10.861725 17586 catalog_manager.cc:5697] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 reported cstate change: term changed from 3 to 4, leader changed from e063541542a94969898868bf6eb5ce1b (127.16.29.67) to 0da169ff82674dc1947055ce43de68a3 (127.16.29.65). New cstate: current_term: 4 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:10.864079 17757 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3"
dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
 from {username='slave'} at 127.0.0.1:38776
I20260709 12:21:10.864727 17757 raft_consensus.cc:493] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:21:10.865159 17757 raft_consensus.cc:3060] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:10.875305 17757 raft_consensus.cc:515] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:10.877604 17757 leader_election.cc:290] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 3 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:10.878952 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:10.879163 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:10.879577 17687 raft_consensus.cc:3055] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:21:10.879762 17835 raft_consensus.cc:3060] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:10.879954 17687 raft_consensus.cc:740] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:21:10.880759 17687 consensus_queue.cc:260] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:10.881913 17687 raft_consensus.cc:3060] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:10.889379 17835 raft_consensus.cc:2468] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 3.
I20260709 12:21:10.890599 17727 leader_election.cc:304] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:21:10.890849 17687 raft_consensus.cc:2468] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 3.
I20260709 12:21:10.891746 19497 raft_consensus.cc:2804] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:21:10.892220 19497 raft_consensus.cc:697] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:21:10.893698 19497 consensus_queue.cc:237] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:10.926127 17586 catalog_manager.cc:5697] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb reported cstate change: term changed from 2 to 3, leader changed from 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) to e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66). New cstate: current_term: 3 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:11.181116 17687 raft_consensus.cc:1275] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:21:11.182497 19526 consensus_queue.cc:1048] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:21:11.220566 17831 raft_consensus.cc:1275] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:21:11.222609 19526 consensus_queue.cc:1048] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:21:11.227803 17760 raft_consensus.cc:1275] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:21:11.229640 19539 consensus_queue.cc:1048] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:21:11.278685 17835 raft_consensus.cc:1275] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:21:11.281872 19559 consensus_queue.cc:1048] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:21:11.312076 17831 consensus_queue.cc:237] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:21:11.338167 17757 raft_consensus.cc:1275] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:21:11.343649 17834 raft_consensus.cc:1275] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:21:11.345865 19514 consensus_queue.cc:1048] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:21:11.347209 17687 raft_consensus.cc:1275] T f8f6bfcb37be41c6917282a17239ad06 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:21:11.345511 19497 consensus_queue.cc:1048] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.004s
I20260709 12:21:11.374076 17687 raft_consensus.cc:1275] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:21:11.376041 19401 consensus_queue.cc:1048] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:21:11.380759 19479 consensus_queue.cc:1048] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:21:11.415657 17686 raft_consensus.cc:2955] T f8f6bfcb37be41c6917282a17239ad06 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } }
W20260709 12:21:11.421043 17803 consensus_peers.cc:597] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b -> Peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Couldn't send request to peer 334a4e1f59944544b3b3ff36616b57da. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f8f6bfcb37be41c6917282a17239ad06. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:21:11.421476 17760 raft_consensus.cc:2955] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } }
I20260709 12:21:11.442224 16501 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:21:11.417106 19479 raft_consensus.cc:2955] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } }
I20260709 12:21:11.469367 17587 catalog_manager.cc:5697] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb reported cstate change: config changed from index -1 to 2, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New cstate: current_term: 1 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } }
I20260709 12:21:11.508605 17831 consensus_queue.cc:237] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:21:11.536350 17760 raft_consensus.cc:1275] T 7cb3768d215e409fb9f5baf18d124442 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260709 12:21:11.536931 17686 raft_consensus.cc:1275] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260709 12:21:11.538048 19514 consensus_queue.cc:1048] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:21:11.539263 19594 consensus_queue.cc:1048] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:21:11.549628 19593 raft_consensus.cc:2955] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 3 LEADER]: Committing config change with OpId 3.4: config changed from index -1 to 4, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } }
I20260709 12:21:11.551538 17687 raft_consensus.cc:2955] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Committing config change with OpId 3.4: config changed from index -1 to 4, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } }
W20260709 12:21:11.555932 17803 consensus_peers.cc:597] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b -> Peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Couldn't send request to peer 334a4e1f59944544b3b3ff36616b57da. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7cb3768d215e409fb9f5baf18d124442. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:21:11.570022 17757 raft_consensus.cc:2955] T 7cb3768d215e409fb9f5baf18d124442 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Committing config change with OpId 3.4: config changed from index -1 to 4, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } }
I20260709 12:21:11.576488 17583 catalog_manager.cc:5697] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 reported cstate change: config changed from index -1 to 4, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New cstate: current_term: 3 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } }
I20260709 12:21:11.581672 17760 consensus_queue.cc:237] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
W20260709 12:21:11.608974 17729 consensus_peers.cc:597] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb -> Peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Couldn't send request to peer 334a4e1f59944544b3b3ff36616b57da. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e5dabc1347544bd8b4e81fa5903d4a8b. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:21:11.611505 17831 raft_consensus.cc:1275] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260709 12:21:11.613927 19497 consensus_queue.cc:1048] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:21:11.622694 19497 raft_consensus.cc:2955] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 3 LEADER]: Committing config change with OpId 3.4: config changed from index -1 to 4, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } }
I20260709 12:21:11.624464 17831 raft_consensus.cc:2955] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Committing config change with OpId 3.4: config changed from index -1 to 4, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } }
I20260709 12:21:11.637954 17687 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9"
dest_uuid: "0da169ff82674dc1947055ce43de68a3"
mode: GRACEFUL
new_leader_uuid: "e063541542a94969898868bf6eb5ce1b"
 from {username='slave'} at 127.0.0.1:54540
I20260709 12:21:11.638677 17687 raft_consensus.cc:606] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 2 LEADER]: Received request to transfer leadership to TS e063541542a94969898868bf6eb5ce1b
I20260709 12:21:11.640429 16501 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 1
I20260709 12:21:11.643090 16501 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:21:11.643850 16501 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:21:11.653669 17687 raft_consensus.cc:1275] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260709 12:21:11.663103 19610 consensus_queue.cc:1048] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:21:11.686218 17583 catalog_manager.cc:5697] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b reported cstate change: config changed from index -1 to 4, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New cstate: current_term: 3 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } }
I20260709 12:21:11.701843 19503 raft_consensus.cc:993] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3: : Instructing follower e063541542a94969898868bf6eb5ce1b to start an election
I20260709 12:21:11.722074 19503 raft_consensus.cc:1081] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 2 LEADER]: Signalling peer e063541542a94969898868bf6eb5ce1b to start an election
W20260709 12:21:11.723131 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 55160.5us
I20260709 12:21:11.724761 17831 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9"
dest_uuid: "e063541542a94969898868bf6eb5ce1b"
 from {username='slave'} at 127.0.0.1:35040
I20260709 12:21:11.725378 17831 raft_consensus.cc:493] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:21:11.725703 17831 raft_consensus.cc:3060] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:11.738682 17831 raft_consensus.cc:515] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:11.722705 17760 consensus_queue.cc:237] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
W20260709 12:21:11.741169 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 90667.6us
I20260709 12:21:11.746052 17831 leader_election.cc:290] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 3 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:11.746805 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:11.747368 17687 raft_consensus.cc:3055] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:21:11.747645 17687 raft_consensus.cc:740] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:21:11.748448 17687 consensus_queue.cc:260] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:11.749604 17687 raft_consensus.cc:3060] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:11.756331 17687 raft_consensus.cc:2468] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 3.
I20260709 12:21:11.757596 17802 leader_election.cc:304] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
W20260709 12:21:11.774082 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 55521.6us
I20260709 12:21:11.781498 17686 raft_consensus.cc:2955] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Committing config change with OpId 3.4: config changed from index -1 to 4, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } }
I20260709 12:21:11.764264 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:11.765915 19603 raft_consensus.cc:2804] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:21:11.788722 18363 raft_consensus.cc:3060] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:11.793659 19603 raft_consensus.cc:697] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 3 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:21:11.794610 19603 consensus_queue.cc:237] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:11.812778 17587 catalog_manager.cc:5697] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b reported cstate change: term changed from 2 to 3, leader changed from 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) to e063541542a94969898868bf6eb5ce1b (127.16.29.67). New cstate: current_term: 3 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:11.824155 17831 raft_consensus.cc:1275] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:21:11.825630 17687 raft_consensus.cc:1275] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260709 12:21:11.825816 17729 consensus_peers.cc:597] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb -> Peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Couldn't send request to peer 334a4e1f59944544b3b3ff36616b57da. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d522f70beb1f417cb98940d90b594823. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:21:11.827945 19613 consensus_queue.cc:1048] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:21:11.829783 19610 consensus_queue.cc:1048] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:21:11.815062 18363 raft_consensus.cc:2468] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 3.
I20260709 12:21:11.845402 19613 raft_consensus.cc:2955] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } }
I20260709 12:21:11.855172 17686 raft_consensus.cc:2955] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } }
I20260709 12:21:11.856393 17831 raft_consensus.cc:2955] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } }
I20260709 12:21:11.878216 17587 catalog_manager.cc:5697] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb reported cstate change: config changed from index -1 to 2, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New cstate: current_term: 1 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:11.922498 19622 ts_tablet_manager.cc:933] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da: Initiating tablet copy from peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:11.925774 19622 tablet_copy_client.cc:323] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Beginning tablet copy session from remote peer at address 127.16.29.67:36837
I20260709 12:21:11.927779 17848 tablet_copy_service.cc:140] P e063541542a94969898868bf6eb5ce1b: Received BeginTabletCopySession request for tablet f8f6bfcb37be41c6917282a17239ad06 from peer 334a4e1f59944544b3b3ff36616b57da ({username='slave'} at 127.0.0.1:37994)
I20260709 12:21:11.928279 17848 tablet_copy_service.cc:161] P e063541542a94969898868bf6eb5ce1b: Beginning new tablet copy session on tablet f8f6bfcb37be41c6917282a17239ad06 from peer 334a4e1f59944544b3b3ff36616b57da at {username='slave'} at 127.0.0.1:37994: session id = 334a4e1f59944544b3b3ff36616b57da-f8f6bfcb37be41c6917282a17239ad06
I20260709 12:21:11.945972 17848 tablet_copy_source_session.cc:215] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:21:11.949505 19622 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f8f6bfcb37be41c6917282a17239ad06. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:11.973402 19622 tablet_copy_client.cc:806] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Starting download of 0 data blocks...
I20260709 12:21:11.974538 19622 tablet_copy_client.cc:670] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Starting download of 1 WAL segments...
I20260709 12:21:11.989789 19624 ts_tablet_manager.cc:933] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da: Initiating tablet copy from peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:11.993906 19622 tablet_copy_client.cc:538] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:21:12.009402 19622 tablet_bootstrap.cc:492] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da: Bootstrap starting.
I20260709 12:21:12.022783 19624 tablet_copy_client.cc:323] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Beginning tablet copy session from remote peer at address 127.16.29.67:36837
I20260709 12:21:12.026479 17848 tablet_copy_service.cc:140] P e063541542a94969898868bf6eb5ce1b: Received BeginTabletCopySession request for tablet 7cb3768d215e409fb9f5baf18d124442 from peer 334a4e1f59944544b3b3ff36616b57da ({username='slave'} at 127.0.0.1:37994)
I20260709 12:21:12.027005 17848 tablet_copy_service.cc:161] P e063541542a94969898868bf6eb5ce1b: Beginning new tablet copy session on tablet 7cb3768d215e409fb9f5baf18d124442 from peer 334a4e1f59944544b3b3ff36616b57da at {username='slave'} at 127.0.0.1:37994: session id = 334a4e1f59944544b3b3ff36616b57da-7cb3768d215e409fb9f5baf18d124442
I20260709 12:21:12.034581 17848 tablet_copy_source_session.cc:215] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:21:12.041836 19624 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7cb3768d215e409fb9f5baf18d124442. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:12.100063 19622 tablet_bootstrap.cc:492] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:21:12.101233 19622 tablet_bootstrap.cc:492] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da: Bootstrap complete.
I20260709 12:21:12.102953 19622 ts_tablet_manager.cc:1403] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da: Time spent bootstrapping tablet: real 0.094s	user 0.036s	sys 0.032s
I20260709 12:21:12.106401 19622 raft_consensus.cc:359] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:21:12.107331 19622 raft_consensus.cc:740] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Initialized, Role: LEARNER
I20260709 12:21:12.108090 19622 consensus_queue.cc:260] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:21:12.115007 19622 ts_tablet_manager.cc:1434] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da: Time spent starting tablet: real 0.012s	user 0.005s	sys 0.000s
I20260709 12:21:12.116813 17848 tablet_copy_service.cc:342] P e063541542a94969898868bf6eb5ce1b: Request end of tablet copy session 334a4e1f59944544b3b3ff36616b57da-f8f6bfcb37be41c6917282a17239ad06 received from {username='slave'} at 127.0.0.1:37994
I20260709 12:21:12.117233 17848 tablet_copy_service.cc:434] P e063541542a94969898868bf6eb5ce1b: ending tablet copy session 334a4e1f59944544b3b3ff36616b57da-f8f6bfcb37be41c6917282a17239ad06 on tablet f8f6bfcb37be41c6917282a17239ad06 with peer 334a4e1f59944544b3b3ff36616b57da
I20260709 12:21:12.123687 19624 tablet_copy_client.cc:806] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Starting download of 0 data blocks...
I20260709 12:21:12.124415 19624 tablet_copy_client.cc:670] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Starting download of 1 WAL segments...
I20260709 12:21:12.135980 19624 tablet_copy_client.cc:538] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:21:12.154414 19624 tablet_bootstrap.cc:492] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da: Bootstrap starting.
I20260709 12:21:12.197906 19622 ts_tablet_manager.cc:933] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da: Initiating tablet copy from peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:12.235806 19629 ts_tablet_manager.cc:933] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da: Initiating tablet copy from peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:12.252703 19629 tablet_copy_client.cc:323] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Beginning tablet copy session from remote peer at address 127.16.29.66:39651
I20260709 12:21:12.278158 17686 raft_consensus.cc:1275] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260709 12:21:12.282670 19603 consensus_queue.cc:1048] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 12:21:12.287395 17774 tablet_copy_service.cc:140] P e2909cff1da44ad387f96a8ae3aa5afb: Received BeginTabletCopySession request for tablet d522f70beb1f417cb98940d90b594823 from peer 334a4e1f59944544b3b3ff36616b57da ({username='slave'} at 127.0.0.1:36572)
I20260709 12:21:12.288273 17774 tablet_copy_service.cc:161] P e2909cff1da44ad387f96a8ae3aa5afb: Beginning new tablet copy session on tablet d522f70beb1f417cb98940d90b594823 from peer 334a4e1f59944544b3b3ff36616b57da at {username='slave'} at 127.0.0.1:36572: session id = 334a4e1f59944544b3b3ff36616b57da-d522f70beb1f417cb98940d90b594823
W20260709 12:21:12.348348 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 90169.8us
W20260709 12:21:12.377389 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 47466.8us
I20260709 12:21:12.402761 18362 raft_consensus.cc:1217] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:21:12.409060 18365 raft_consensus.cc:1275] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260709 12:21:12.417852 17774 tablet_copy_source_session.cc:215] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:21:12.445751 19622 tablet_copy_client.cc:323] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Beginning tablet copy session from remote peer at address 127.16.29.66:39651
I20260709 12:21:12.468041 19514 consensus_queue.cc:1048] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:21:12.476052 17774 tablet_copy_service.cc:140] P e2909cff1da44ad387f96a8ae3aa5afb: Received BeginTabletCopySession request for tablet e5dabc1347544bd8b4e81fa5903d4a8b from peer 334a4e1f59944544b3b3ff36616b57da ({username='slave'} at 127.0.0.1:36572)
I20260709 12:21:12.476655 17774 tablet_copy_service.cc:161] P e2909cff1da44ad387f96a8ae3aa5afb: Beginning new tablet copy session on tablet e5dabc1347544bd8b4e81fa5903d4a8b from peer 334a4e1f59944544b3b3ff36616b57da at {username='slave'} at 127.0.0.1:36572: session id = 334a4e1f59944544b3b3ff36616b57da-e5dabc1347544bd8b4e81fa5903d4a8b
I20260709 12:21:12.489445 19629 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d522f70beb1f417cb98940d90b594823. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:12.491134 17774 tablet_copy_source_session.cc:215] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:21:12.506039 19622 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e5dabc1347544bd8b4e81fa5903d4a8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:12.515108 19624 tablet_bootstrap.cc:492] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:21:12.527719 19624 tablet_bootstrap.cc:492] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da: Bootstrap complete.
I20260709 12:21:12.528565 19624 ts_tablet_manager.cc:1403] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da: Time spent bootstrapping tablet: real 0.374s	user 0.111s	sys 0.115s
I20260709 12:21:12.532157 19624 raft_consensus.cc:359] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 3 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:21:12.533036 19624 raft_consensus.cc:740] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Initialized, Role: LEARNER
I20260709 12:21:12.533797 19624 consensus_queue.cc:260] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:21:12.551757 19624 ts_tablet_manager.cc:1434] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da: Time spent starting tablet: real 0.023s	user 0.005s	sys 0.001s
I20260709 12:21:12.553627 17848 tablet_copy_service.cc:342] P e063541542a94969898868bf6eb5ce1b: Request end of tablet copy session 334a4e1f59944544b3b3ff36616b57da-7cb3768d215e409fb9f5baf18d124442 received from {username='slave'} at 127.0.0.1:37994
I20260709 12:21:12.554023 17848 tablet_copy_service.cc:434] P e063541542a94969898868bf6eb5ce1b: ending tablet copy session 334a4e1f59944544b3b3ff36616b57da-7cb3768d215e409fb9f5baf18d124442 on tablet 7cb3768d215e409fb9f5baf18d124442 with peer 334a4e1f59944544b3b3ff36616b57da
I20260709 12:21:12.587045 19622 tablet_copy_client.cc:806] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Starting download of 0 data blocks...
I20260709 12:21:12.587838 19622 tablet_copy_client.cc:670] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Starting download of 1 WAL segments...
I20260709 12:21:12.588482 19629 tablet_copy_client.cc:806] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Starting download of 0 data blocks...
I20260709 12:21:12.589169 19629 tablet_copy_client.cc:670] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Starting download of 1 WAL segments...
I20260709 12:21:12.593799 19622 tablet_copy_client.cc:538] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:21:12.594300 19629 tablet_copy_client.cc:538] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:21:12.605523 19622 tablet_bootstrap.cc:492] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da: Bootstrap starting.
I20260709 12:21:12.623966 19629 tablet_bootstrap.cc:492] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da: Bootstrap starting.
I20260709 12:21:12.650213 19622 tablet_bootstrap.cc:492] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:21:12.651342 19622 tablet_bootstrap.cc:492] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da: Bootstrap complete.
I20260709 12:21:12.652613 19622 ts_tablet_manager.cc:1403] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da: Time spent bootstrapping tablet: real 0.047s	user 0.032s	sys 0.004s
I20260709 12:21:12.656147 19622 raft_consensus.cc:359] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 3 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:21:12.658980 19622 raft_consensus.cc:740] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Initialized, Role: LEARNER
I20260709 12:21:12.659785 19622 consensus_queue.cc:260] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:21:12.667019 19622 ts_tablet_manager.cc:1434] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da: Time spent starting tablet: real 0.014s	user 0.005s	sys 0.002s
I20260709 12:21:12.673533 17774 tablet_copy_service.cc:342] P e2909cff1da44ad387f96a8ae3aa5afb: Request end of tablet copy session 334a4e1f59944544b3b3ff36616b57da-e5dabc1347544bd8b4e81fa5903d4a8b received from {username='slave'} at 127.0.0.1:36572
I20260709 12:21:12.674116 17774 tablet_copy_service.cc:434] P e2909cff1da44ad387f96a8ae3aa5afb: ending tablet copy session 334a4e1f59944544b3b3ff36616b57da-e5dabc1347544bd8b4e81fa5903d4a8b on tablet e5dabc1347544bd8b4e81fa5903d4a8b with peer 334a4e1f59944544b3b3ff36616b57da
I20260709 12:21:12.692939 18363 raft_consensus.cc:1217] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 3 LEARNER]: Deduplicated request from leader. Original: 3.3->[3.4-3.4]   Dedup: 3.4->[]
I20260709 12:21:12.712622 16501 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:21:12.733762 19629 tablet_bootstrap.cc:492] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:21:12.734787 19629 tablet_bootstrap.cc:492] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da: Bootstrap complete.
I20260709 12:21:12.735551 19629 ts_tablet_manager.cc:1403] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da: Time spent bootstrapping tablet: real 0.112s	user 0.020s	sys 0.004s
I20260709 12:21:12.750723 19629 raft_consensus.cc:359] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:21:12.754999 19629 raft_consensus.cc:740] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Initialized, Role: LEARNER
I20260709 12:21:12.755790 19629 consensus_queue.cc:260] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:21:12.769608 19629 ts_tablet_manager.cc:1434] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da: Time spent starting tablet: real 0.025s	user 0.006s	sys 0.000s
I20260709 12:21:12.779572 17774 tablet_copy_service.cc:342] P e2909cff1da44ad387f96a8ae3aa5afb: Request end of tablet copy session 334a4e1f59944544b3b3ff36616b57da-d522f70beb1f417cb98940d90b594823 received from {username='slave'} at 127.0.0.1:36572
I20260709 12:21:12.780089 17774 tablet_copy_service.cc:434] P e2909cff1da44ad387f96a8ae3aa5afb: ending tablet copy session 334a4e1f59944544b3b3ff36616b57da-d522f70beb1f417cb98940d90b594823 on tablet d522f70beb1f417cb98940d90b594823 with peer 334a4e1f59944544b3b3ff36616b57da
I20260709 12:21:12.826279 18363 raft_consensus.cc:1217] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:21:12.832463 17835 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad"
dest_uuid: "e063541542a94969898868bf6eb5ce1b"
mode: GRACEFUL
new_leader_uuid: "0da169ff82674dc1947055ce43de68a3"
 from {username='slave'} at 127.0.0.1:59820
I20260709 12:21:12.833093 17835 raft_consensus.cc:606] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 1 LEADER]: Received request to transfer leadership to TS 0da169ff82674dc1947055ce43de68a3
I20260709 12:21:12.834347 16501 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 1
I20260709 12:21:12.836575 16501 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:21:12.837363 16501 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:21:12.878248 19603 raft_consensus.cc:1064] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b: attempting to promote NON_VOTER 334a4e1f59944544b3b3ff36616b57da to VOTER
I20260709 12:21:12.883431 19603 consensus_queue.cc:237] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:12.903353 18362 raft_consensus.cc:1275] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 1 LEARNER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:21:12.904856 19594 consensus_queue.cc:1048] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:21:12.911404 17760 raft_consensus.cc:1275] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:21:12.913920 17686 raft_consensus.cc:1275] T f8f6bfcb37be41c6917282a17239ad06 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:21:12.916114 19594 consensus_queue.cc:1048] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:21:12.923861 19593 consensus_queue.cc:1048] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:21:12.925988 18362 raft_consensus.cc:1217] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 3 LEARNER]: Deduplicated request from leader. Original: 3.3->[3.4-3.4]   Dedup: 3.4->[]
I20260709 12:21:13.018321 19603 raft_consensus.cc:2955] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:21:13.026691 17682 raft_consensus.cc:2955] T f8f6bfcb37be41c6917282a17239ad06 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:21:13.030090 17760 raft_consensus.cc:2955] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:21:13.063921 17587 catalog_manager.cc:5697] T f8f6bfcb37be41c6917282a17239ad06 P 0da169ff82674dc1947055ce43de68a3 reported cstate change: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:21:13.084167 18363 raft_consensus.cc:2955] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } attrs { replace: true } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:21:13.116665 19603 raft_consensus.cc:993] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b: : Instructing follower 0da169ff82674dc1947055ce43de68a3 to start an election
I20260709 12:21:13.117197 19603 raft_consensus.cc:1081] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 1 LEADER]: Signalling peer 0da169ff82674dc1947055ce43de68a3 to start an election
I20260709 12:21:13.120641 17686 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad"
dest_uuid: "0da169ff82674dc1947055ce43de68a3"
 from {username='slave'} at 127.0.0.1:37402
I20260709 12:21:13.121284 17686 raft_consensus.cc:493] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:21:13.121625 17686 raft_consensus.cc:3060] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:13.138278 17686 raft_consensus.cc:515] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:13.141135 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:13.141875 17831 raft_consensus.cc:3055] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:21:13.142258 17831 raft_consensus.cc:740] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:21:13.143224 17831 consensus_queue.cc:260] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:13.146692 17831 raft_consensus.cc:3060] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:13.170935 19613 raft_consensus.cc:1064] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb: attempting to promote NON_VOTER 334a4e1f59944544b3b3ff36616b57da to VOTER
I20260709 12:21:13.177263 17686 leader_election.cc:290] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 2 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:13.181931 17831 raft_consensus.cc:2468] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 2.
I20260709 12:21:13.183138 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:13.183898 18371 raft_consensus.cc:3060] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:13.193389 19613 consensus_queue.cc:237] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 2, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:13.197389 17835 consensus_queue.cc:237] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:13.210726 18363 raft_consensus.cc:1275] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:21:13.213214 19603 consensus_queue.cc:1048] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:21:13.224308 17760 raft_consensus.cc:1275] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:21:13.230331 19594 raft_consensus.cc:2955] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:21:13.233961 18362 raft_consensus.cc:2955] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:21:13.238927 19603 consensus_queue.cc:1048] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:21:13.257670 17760 raft_consensus.cc:2955] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:21:13.314144 17571 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet f8f6bfcb37be41c6917282a17239ad06 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:21:13.336247 18371 raft_consensus.cc:2468] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 2.
I20260709 12:21:13.338231 17654 leader_election.cc:304] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:21:13.342584 19503 raft_consensus.cc:2804] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:21:13.343204 19503 raft_consensus.cc:697] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 2 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:21:13.344647 19503 consensus_queue.cc:237] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:13.401235 19593 raft_consensus.cc:1064] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b: attempting to promote NON_VOTER 334a4e1f59944544b3b3ff36616b57da to VOTER
W20260709 12:21:13.403774 17804 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:13.406240 19593 consensus_queue.cc:237] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 2, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:13.423261 17583 catalog_manager.cc:5697] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb reported cstate change: config changed from index 3 to 4, VOTER 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) evicted. New cstate: current_term: 1 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:21:13.434466 19612 raft_consensus.cc:1064] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb: attempting to promote NON_VOTER 334a4e1f59944544b3b3ff36616b57da to VOTER
I20260709 12:21:13.435866 17676 tablet_service.cc:1558] Processing DeleteTablet for tablet f8f6bfcb37be41c6917282a17239ad06 with delete_type TABLET_DATA_TOMBSTONED (TS 0da169ff82674dc1947055ce43de68a3 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:37384
I20260709 12:21:13.437438 19612 consensus_queue.cc:237] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:13.443424 18363 raft_consensus.cc:1275] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 3 LEARNER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 12:21:13.444880 17760 raft_consensus.cc:1275] T 7cb3768d215e409fb9f5baf18d124442 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 12:21:13.447789 19653 tablet_replica.cc:333] T f8f6bfcb37be41c6917282a17239ad06 P 0da169ff82674dc1947055ce43de68a3: stopping tablet replica
I20260709 12:21:13.448181 19594 consensus_queue.cc:1048] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:21:13.452723 19594 consensus_queue.cc:1048] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:21:13.457420 19653 raft_consensus.cc:2243] T f8f6bfcb37be41c6917282a17239ad06 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:21:13.458365 19653 raft_consensus.cc:2272] T f8f6bfcb37be41c6917282a17239ad06 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:13.481119 19653 ts_tablet_manager.cc:1916] T f8f6bfcb37be41c6917282a17239ad06 P 0da169ff82674dc1947055ce43de68a3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:21:13.481606 17583 catalog_manager.cc:5697] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 reported cstate change: term changed from 1 to 2, leader changed from e063541542a94969898868bf6eb5ce1b (127.16.29.67) to 0da169ff82674dc1947055ce43de68a3 (127.16.29.65). New cstate: current_term: 2 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:13.517642 18362 raft_consensus.cc:1275] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 3 LEARNER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 12:21:13.518970 18371 raft_consensus.cc:1275] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 1 LEARNER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:21:13.522984 17686 raft_consensus.cc:1275] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 12:21:13.528252 19652 consensus_queue.cc:1048] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:21:13.528653 19615 consensus_queue.cc:1048] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:21:13.849463 19653 ts_tablet_manager.cc:1929] T f8f6bfcb37be41c6917282a17239ad06 P 0da169ff82674dc1947055ce43de68a3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:21:13.850144 19653 log.cc:1199] T f8f6bfcb37be41c6917282a17239ad06 P 0da169ff82674dc1947055ce43de68a3: Deleting WAL directory at /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599551912677-16501-0/minicluster-data/ts-0-root/wals/f8f6bfcb37be41c6917282a17239ad06
I20260709 12:21:14.134859 16501 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
W20260709 12:21:14.148262 17652 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.794s	user 0.009s	sys 0.004s
W20260709 12:21:14.159446 17652 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.805s	user 0.009s	sys 0.005s
W20260709 12:21:14.169275 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 814964us
I20260709 12:21:14.172703 17572 catalog_manager.cc:5028] TS 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): tablet f8f6bfcb37be41c6917282a17239ad06 (table test-workload [id=ddc5b74250df4c65909b73b3742d615f]) successfully deleted
W20260709 12:21:14.219599 19656 log.cc:927] Time spent T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.318s	user 0.003s	sys 0.005s
I20260709 12:21:14.227242 19667 raft_consensus.cc:493] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:21:14.228449 19667 raft_consensus.cc:515] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:14.231698 19667 leader_election.cc:290] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:14.242956 19667 consensus_queue.cc:1048] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.002s
I20260709 12:21:14.245030 17836 raft_consensus.cc:1275] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 12:21:14.357227 19669 raft_consensus.cc:493] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:21:14.358309 19669 raft_consensus.cc:515] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:14.361193 19669 leader_election.cc:290] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:14.370473 19617 raft_consensus.cc:493] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:21:14.371850 19617 raft_consensus.cc:515] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:21:14.406852 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:14.435458 19503 raft_consensus.cc:493] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:21:14.436496 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:21:14.549317 19612 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.031s	user 0.007s	sys 0.004s
I20260709 12:21:14.548894 19503 raft_consensus.cc:515] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
W20260709 12:21:14.549724 19612 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.034s	user 0.008s	sys 0.004s
I20260709 12:21:14.558538 19593 raft_consensus.cc:493] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:21:14.551615 19503 leader_election.cc:290] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:14.534729 19673 raft_consensus.cc:493] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
W20260709 12:21:14.579698 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 160003us
I20260709 12:21:14.577741 19667 raft_consensus.cc:493] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:21:14.612056 17583 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:37172 (request call id 250) took 1090 ms. Trace:
I20260709 12:21:14.612625 17583 rpcz_store.cc:276] 0709 12:21:13.521113 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:13.521248 (+   135us) service_pool.cc:224] Handling call
0709 12:21:14.611733 (+1090485us) inbound_call.cc:177] Queueing success response
Metrics: {"reported_tablets":1}
I20260709 12:21:14.614974 18363 raft_consensus.cc:1275] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:21:14.566869 19593 raft_consensus.cc:515] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:21:14.628193 19684 raft_consensus.cc:493] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:21:14.629263 19684 raft_consensus.cc:515] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:14.593092 19668 raft_consensus.cc:493] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:21:14.641592 19666 raft_consensus.cc:493] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:21:14.659138 19662 raft_consensus.cc:493] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:21:14.662210 19662 raft_consensus.cc:515] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:14.664144 19670 raft_consensus.cc:493] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:21:14.642544 19666 raft_consensus.cc:515] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:14.665067 19670 raft_consensus.cc:515] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:14.674082 19672 raft_consensus.cc:493] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:21:14.676105 19617 leader_election.cc:290] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:14.650570 19593 leader_election.cc:290] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:14.704161 19658 raft_consensus.cc:493] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:21:14.680097 19657 consensus_queue.cc:1048] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:21:14.694641 19677 raft_consensus.cc:493] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:21:14.695472 19671 raft_consensus.cc:493] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:21:14.696108 19616 raft_consensus.cc:493] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:21:14.696585 19514 raft_consensus.cc:493] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:21:14.699175 19662 leader_election.cc:290] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:14.674127 19684 leader_election.cc:290] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:14.704921 19658 raft_consensus.cc:515] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:14.724802 17834 raft_consensus.cc:1275] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:21:14.671343 19667 raft_consensus.cc:515] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:21:14.710433 19668 raft_consensus.cc:515] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:14.652750 19673 raft_consensus.cc:515] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:14.717386 19617 raft_consensus.cc:493] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:21:14.727241 19593 raft_consensus.cc:493] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:21:14.748124 19617 raft_consensus.cc:515] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:14.748420 19593 raft_consensus.cc:515] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:14.750356 19661 raft_consensus.cc:493] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:21:14.750558 19593 leader_election.cc:290] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:14.744644 19669 raft_consensus.cc:493] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:21:14.741312 19682 raft_consensus.cc:493] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:21:14.726322 19672 raft_consensus.cc:515] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:14.741010 19684 raft_consensus.cc:493] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
W20260709 12:21:14.768857 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 45572.6us
I20260709 12:21:14.759697 19682 raft_consensus.cc:515] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:14.777515 19593 raft_consensus.cc:493] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:21:14.778174 19593 raft_consensus.cc:515] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:14.781265 19658 leader_election.cc:290] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:14.777149 19686 raft_consensus.cc:493] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:21:14.731758 19616 raft_consensus.cc:515] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:14.747694 19514 raft_consensus.cc:515] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:14.721294 19677 raft_consensus.cc:515] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:21:14.883939 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 86759.7us
I20260709 12:21:14.873963 19661 raft_consensus.cc:515] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:14.787778 19593 leader_election.cc:290] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:14.922431 19613 raft_consensus.cc:493] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:21:14.922935 19659 raft_consensus.cc:493] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:21:14.923143 19514 leader_election.cc:290] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:14.818655 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:14.923812 19613 raft_consensus.cc:515] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:14.883575 19594 raft_consensus.cc:493] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
W20260709 12:21:14.968514 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 192369us
I20260709 12:21:14.943578 19659 raft_consensus.cc:515] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:14.972533 19659 leader_election.cc:290] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:14.974906 17681 raft_consensus.cc:1275] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 12:21:14.977613 17681 raft_consensus.cc:1275] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:21:14.978559 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:14.980037 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:14.761305 19671 raft_consensus.cc:515] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:14.925388 19668 leader_election.cc:290] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:14.958165 19514 raft_consensus.cc:493] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:21:14.965479 17757 raft_consensus.cc:2468] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 2.
I20260709 12:21:14.983244 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:14.832732 19669 raft_consensus.cc:515] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:14.794613 19684 raft_consensus.cc:515] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:14.881893 19686 raft_consensus.cc:515] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:14.991437 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:14.992908 19616 leader_election.cc:290] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:14.993079 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:14.985584 19691 raft_consensus.cc:493] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:21:14.999600 19678 raft_consensus.cc:493] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:21:14.993474 19686 leader_election.cc:290] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:15.004812 19686 raft_consensus.cc:493] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:21:15.005519 19671 leader_election.cc:290] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:21:15.015733 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 271017us
I20260709 12:21:15.058671 19648 raft_consensus.cc:493] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:21:15.058864 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:15.059152 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:15.059273 19675 raft_consensus.cc:493] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:21:15.059847 19665 raft_consensus.cc:493] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:21:15.008715 19526 raft_consensus.cc:493] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:21:15.060364 19695 raft_consensus.cc:493] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:21:15.001359 19613 leader_election.cc:290] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:15.016819 19671 raft_consensus.cc:493] T 7cb3768d215e409fb9f5baf18d124442 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:21:15.000212 19678 raft_consensus.cc:515] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:15.002302 19659 raft_consensus.cc:493] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:21:15.056071 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:14.988266 19594 raft_consensus.cc:515] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:14.998551 19514 raft_consensus.cc:515] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:15.117812 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:15.066656 19669 leader_election.cc:290] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:15.069657 19672 leader_election.cc:290] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:15.120337 19682 leader_election.cc:290] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:15.072969 19661 leader_election.cc:290] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:15.073743 19693 raft_consensus.cc:493] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:21:15.120961 19526 raft_consensus.cc:515] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:15.006390 19658 raft_consensus.cc:493] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:21:15.005681 19686 raft_consensus.cc:515] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
W20260709 12:21:15.085175 19685 log.cc:927] Time spent T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.324s	user 0.000s	sys 0.002s
I20260709 12:21:15.085973 19696 raft_consensus.cc:493] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:21:15.090514 18363 raft_consensus.cc:2468] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 1.
I20260709 12:21:15.092600 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:15.083478 19667 leader_election.cc:290] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:15.060791 19688 consensus_queue.cc:1048] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.099s
I20260709 12:21:15.121901 19693 raft_consensus.cc:515] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:15.134451 19526 leader_election.cc:290] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:15.061539 19666 leader_election.cc:290] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:15.076020 19670 leader_election.cc:290] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:21:15.140931 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 171305us
I20260709 12:21:15.134258 19665 raft_consensus.cc:515] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:15.086849 19671 raft_consensus.cc:515] T 7cb3768d215e409fb9f5baf18d124442 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:15.117885 19514 leader_election.cc:290] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:21:15.167443 17785 debug-util.cc:398] Leaking SignalData structure 0x7b08001687c0 after lost signal to thread 16504
I20260709 12:21:15.058893 19692 raft_consensus.cc:493] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:21:15.169672 19649 consensus_queue.cc:1048] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.021s
I20260709 12:21:15.146850 19673 leader_election.cc:290] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:15.150398 19696 raft_consensus.cc:515] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:15.151752 19658 raft_consensus.cc:515] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:15.082171 19691 raft_consensus.cc:515] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:15.160369 19695 raft_consensus.cc:515] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:15.153266 19675 raft_consensus.cc:515] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:15.137856 19648 raft_consensus.cc:515] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:15.374095 19678 leader_election.cc:290] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:15.229000 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:15.380530 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:15.313525 19617 leader_election.cc:290] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:15.460839 17584 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35966 (request call id 351) took 1999 ms. Trace:
I20260709 12:21:15.383937 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:21:15.297878 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 49099.3us
I20260709 12:21:15.460116 19613 raft_consensus.cc:493] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:21:15.483235 19690 raft_consensus.cc:493] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:21:15.377126 18362 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59886 (request call id 616) took 1094 ms. Trace:
I20260709 12:21:15.314432 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:15.381281 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:15.323256 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:15.483783 19613 raft_consensus.cc:515] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:15.480894 17584 rpcz_store.cc:276] 0709 12:21:13.460861 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:13.461276 (+   415us) service_pool.cc:224] Handling call
0709 12:21:15.460635 (+1999359us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":204452,"reported_tablets":2}
I20260709 12:21:15.170145 19667 raft_consensus.cc:493] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:21:15.230000 19692 raft_consensus.cc:515] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:15.545945 19672 raft_consensus.cc:493] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:21:15.546105 17681 raft_consensus.cc:2468] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 3.
I20260709 12:21:15.546882 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:15.546716 19672 raft_consensus.cc:515] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:15.229218 19514 raft_consensus.cc:493] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:15.364812 19686 leader_election.cc:290] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:15.562763 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:15.333973 19594 leader_election.cc:290] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:15.229147 19660 raft_consensus.cc:493] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:21:15.505084 19615 raft_consensus.cc:493] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:21:15.188279 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:15.303773 19684 leader_election.cc:290] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:15.549398 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:15.549620 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:15.547915 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:15.378201 19665 leader_election.cc:290] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:15.242375 19668 raft_consensus.cc:493] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:21:15.255066 19682 raft_consensus.cc:493] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:21:15.250044 19671 leader_election.cc:290] T 7cb3768d215e409fb9f5baf18d124442 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:15.483935 19690 raft_consensus.cc:515] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:15.560734 17802 leader_election.cc:304] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:21:15.562301 19695 leader_election.cc:290] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:15.311976 19693 leader_election.cc:290] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:15.184166 19659 raft_consensus.cc:515] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:15.627036 19691 leader_election.cc:290] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:15.639474 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:15.640492 19659 leader_election.cc:290] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:15.484663 18362 rpcz_store.cc:276] 0709 12:21:14.282201 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:14.282419 (+   218us) service_pool.cc:224] Handling call
0709 12:21:14.283441 (+  1022us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:14.284337 (+   896us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:14.284375 (+    38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:14.286085 (+  1710us) log.cc:844] Serialized 556 byte log entry
0709 12:21:14.340410 (+ 54325us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:21:14.340476 (+    66us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:14.340581 (+   105us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:15.115469 (+774888us) spinlock_profiling.cc:243] Waited 13.2 ms on lock 0x7b54006e2ba0. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a595dbe 00007f7b4a5a0930 00007f7b4a5a08d8 00007f7b4a5a0858 00007f7b4a5a0813 00007f7b4a59fa9c 00007f7b5048b2f6 00007f7b50485aa8 00007f7b4a5dc7cb 00007f7b4a5df2c0
0709 12:21:15.336767 (+221298us) raft_consensus.cc:1685] finished
0709 12:21:15.336789 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:15.376892 (+ 40103us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":30373760,"thread_start_us":36276,"threads_started":1,"wal-append.queue_time_us":38657}
I20260709 12:21:15.640763 17681 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 3.
I20260709 12:21:15.387110 19663 raft_consensus.cc:493] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:21:15.304015 19677 leader_election.cc:290] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:15.571481 19514 raft_consensus.cc:515] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:15.609877 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:15.580513 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:15.575289 19652 raft_consensus.cc:493] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
W20260709 12:21:15.548240 17785 debug-util.cc:398] Leaking SignalData structure 0x7b0800168ea0 after lost signal to thread 17621
I20260709 12:21:15.613761 19692 leader_election.cc:290] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:15.615269 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:15.573108 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:15.622699 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:15.624913 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:15.625062 19672 leader_election.cc:290] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:15.546032 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:21:15.644876 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 84217.2us
I20260709 12:21:15.641680 19663 raft_consensus.cc:515] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:21:15.673272 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 247919us
I20260709 12:21:15.637012 19617 raft_consensus.cc:493] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:21:15.631382 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:21:15.688262 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 69750.8us
I20260709 12:21:15.631692 19668 raft_consensus.cc:515] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:15.617619 19667 raft_consensus.cc:515] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:15.653406 19675 leader_election.cc:290] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:15.841125 18367 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59886 (request call id 617) took 1322 ms. Trace:
I20260709 12:21:15.878559 17831 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35040 (request call id 1751) took 1114 ms. Trace:
I20260709 12:21:15.884143 19693 raft_consensus.cc:493] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:21:15.636439 19660 raft_consensus.cc:515] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:15.897258 17831 rpcz_store.cc:276] 0709 12:21:14.759194 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:14.767199 (+  8005us) service_pool.cc:224] Handling call
0709 12:21:15.609574 (+842375us) spinlock_profiling.cc:243] Waited 946 us on lock 0x7b580013fe98. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b474fbb71 00007f7b474fba22 00007f7b504148d9 00007f7b4f56eee0 00007f7b4f6380ab 00007f7b4f638131 00007f7b4f5e60a0 00007f7b4f5e5984 00007f7b48b606d4 00007f7b48b60662
0709 12:21:15.869020 (+259446us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2176384}
I20260709 12:21:15.884418 19702 raft_consensus.cc:2804] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:21:15.895592 19696 leader_election.cc:290] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:15.686944 19682 raft_consensus.cc:515] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:15.877767 19526 raft_consensus.cc:493] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:21:15.900177 17801 leader_election.cc:304] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:21:15.879854 19677 raft_consensus.cc:493] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:21:15.643226 19615 raft_consensus.cc:515] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:15.911175 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:15.919981 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:15.896539 18367 rpcz_store.cc:276] 0709 12:21:14.282618 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:14.552324 (+269706us) service_pool.cc:224] Handling call
0709 12:21:14.553119 (+   795us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:14.553906 (+   787us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:14.553936 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:14.555265 (+  1329us) log.cc:844] Serialized 556 byte log entry
0709 12:21:14.649294 (+ 94029us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:21:14.649356 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:14.649644 (+   288us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:15.604618 (+954974us) raft_consensus.cc:1685] finished
0709 12:21:15.604646 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:15.604887 (+   241us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":88931,"threads_started":1,"wal-append.queue_time_us":151515}
I20260709 12:21:15.879506 17833 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 1.
I20260709 12:21:15.952337 17832 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35040 (request call id 1754) took 1167 ms. Trace:
I20260709 12:21:15.952163 19526 raft_consensus.cc:515] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:15.952757 17832 rpcz_store.cc:276] 0709 12:21:14.784610 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:14.784766 (+   156us) service_pool.cc:224] Handling call
0709 12:21:15.629409 (+844643us) spinlock_profiling.cc:243] Waited 5.29 ms on lock 0x7b580013fe98. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b474fbe9c 00007f7b474fba5b 00007f7b504148d9 00007f7b4f56eee0 00007f7b4f6380ab 00007f7b4f638131 00007f7b4f5e60a0 00007f7b4f5e5984 00007f7b48b606d4 00007f7b48b60662
0709 12:21:15.952262 (+322853us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":12178304}
I20260709 12:21:15.983616 17728 leader_election.cc:304] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
W20260709 12:21:15.985394 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 85287.4us
I20260709 12:21:15.917582 19660 leader_election.cc:290] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:15.902119 19692 consensus_queue.cc:1048] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.087s
I20260709 12:21:15.629534 19689 raft_consensus.cc:2955] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 3 LEADER]: Committing config change with OpId 3.5: config changed from index 4 to 5, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
W20260709 12:21:15.925792 17785 debug-util.cc:398] Leaking SignalData structure 0x7b080032ef00 after lost signal to thread 17713
I20260709 12:21:15.994282 19526 leader_election.cc:290] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:15.994580 19706 raft_consensus.cc:493] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:21:15.995313 19514 leader_election.cc:290] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:15.997450 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:15.999194 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:16.004264 19707 raft_consensus.cc:493] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:21:16.004777 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:16.000187 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:16.004923 19707 raft_consensus.cc:515] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:16.010759 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:16.013108 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:16.018620 19668 leader_election.cc:290] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:16.018707 19673 raft_consensus.cc:2749] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:21:15.982705 19677 raft_consensus.cc:515] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:16.022138 19677 leader_election.cc:290] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:15.982474 19702 raft_consensus.cc:493] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:21:15.877331 19652 raft_consensus.cc:515] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:16.001423 18364 raft_consensus.cc:2955] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 4 to 5, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:21:16.031121 19663 leader_election.cc:290] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:16.032406 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:15.904151 19617 raft_consensus.cc:515] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:21:15.999680 17785 debug-util.cc:398] Leaking SignalData structure 0x7b0800337ce0 after lost signal to thread 17788
I20260709 12:21:15.983242 19687 raft_consensus.cc:493] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:21:16.044818 19615 leader_election.cc:290] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:15.944578 19693 raft_consensus.cc:515] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:16.006582 19696 raft_consensus.cc:2804] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:21:16.057628 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:16.058547 19667 leader_election.cc:290] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:16.036105 19648 leader_election.cc:290] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:16.064297 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:16.060738 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:15.921665 19658 leader_election.cc:290] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:16.080864 17802 leader_election.cc:304] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:21:15.991847 19675 consensus_queue.cc:1048] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.045s
I20260709 12:21:16.078351 18363 raft_consensus.cc:2468] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 2.
I20260709 12:21:16.036616 19702 raft_consensus.cc:3060] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:16.089067 19703 raft_consensus.cc:493] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:21:16.061414 19690 leader_election.cc:290] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:16.090330 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:16.091220 19652 leader_election.cc:290] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:16.093211 19617 leader_election.cc:290] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:16.104138 19613 leader_election.cc:290] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:16.082523 19696 raft_consensus.cc:493] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:21:16.077313 19687 raft_consensus.cc:515] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:16.114351 19705 raft_consensus.cc:2804] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:21:16.110729 19648 raft_consensus.cc:493] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:21:16.168663 19696 raft_consensus.cc:3060] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:15.995314 19706 raft_consensus.cc:515] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:16.117144 19703 raft_consensus.cc:515] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:16.114285 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:21:16.220621 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 119732us
I20260709 12:21:16.222230 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:16.209134 19705 raft_consensus.cc:493] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:21:16.224021 17802 leader_election.cc:304] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:21:16.221383 17785 debug-util.cc:398] Leaking SignalData structure 0x7b08002e6c60 after lost signal to thread 17862
I20260709 12:21:16.430681 19613 raft_consensus.cc:493] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:21:16.431348 19613 raft_consensus.cc:515] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:16.433470 19613 leader_election.cc:290] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:16.434043 17803 leader_election.cc:304] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
W20260709 12:21:16.436633 17652 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.16.29.67:36837 timed out after 1.513s (SENT)
I20260709 12:21:16.439919 17835 raft_consensus.cc:1275] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:21:16.440555 17835 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35042 (request call id 1491) took 1431 ms. Trace:
I20260709 12:21:16.440657 19652 raft_consensus.cc:493] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:21:16.226861 19705 raft_consensus.cc:3060] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:16.440878 17835 rpcz_store.cc:276] 0709 12:21:15.009491 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:15.009629 (+   138us) service_pool.cc:224] Handling call
0709 12:21:16.189380 (+1179751us) spinlock_profiling.cc:243] Waited 1.53 ms on lock 0x7b580013fe98. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b474fbe9c 00007f7b474fba5b 00007f7b504148d9 00007f7b4f56eee0 00007f7b4f6380ab 00007f7b4f638131 00007f7b4f5f9fc0 00007f7b4f5e4929 00007f7b4f5e43d2 00007f7b48b5bbd4
0709 12:21:16.189635 (+   255us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:16.440287 (+250652us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:16.440478 (+   191us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3510016}
I20260709 12:21:16.235132 19707 leader_election.cc:290] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:16.228729 19692 raft_consensus.cc:493] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:21:16.237370 19668 raft_consensus.cc:2749] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:21:16.232318 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:16.457334 19659 raft_consensus.cc:493] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:21:16.225157 19648 raft_consensus.cc:515] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:16.459394 19615 raft_consensus.cc:493] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:21:16.459836 19663 raft_consensus.cc:493] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:21:16.463807 17830 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37994 (request call id 448) took 1691 ms. Trace:
I20260709 12:21:16.475584 17680 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37396 (request call id 1627) took 1501 ms. Trace:
I20260709 12:21:16.458923 19658 raft_consensus.cc:493] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:21:16.441195 19652 raft_consensus.cc:515] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:16.487534 17801 leader_election.cc:304] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:16.488322 19594 raft_consensus.cc:493] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:21:16.491385 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:16.493548 19682 leader_election.cc:290] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:16.495186 19706 leader_election.cc:290] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:16.497705 19703 leader_election.cc:290] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:16.498093 19660 raft_consensus.cc:2749] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:21:16.499356 19687 leader_election.cc:290] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:16.499998 19693 leader_election.cc:290] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:16.500396 19692 raft_consensus.cc:515] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:16.505196 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:16.508654 19663 raft_consensus.cc:515] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:16.507550 17830 rpcz_store.cc:276] 0709 12:21:14.771918 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:14.785805 (+ 13887us) service_pool.cc:224] Handling call
0709 12:21:15.324037 (+538232us) spinlock_profiling.cc:243] Waited 36.8 ms on lock 0x7b580013fe98. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b474fbb71 00007f7b474fba22 00007f7b504148d9 00007f7b4f56eee0 00007f7b4f6380ab 00007f7b4f638131 00007f7b4f5f9fc0 00007f7b4f5e4929 00007f7b4f5e43d2 00007f7b48b5bbd4
0709 12:21:15.324284 (+   247us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:15.952097 (+627813us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:15.952153 (+    56us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:15.952221 (+    68us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:21:15.952308 (+    87us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:16.463324 (+511016us) spinlock_profiling.cc:243] Waited 604 ms on lock 0x7b5400158bd4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:21:16.463463 (+   139us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:16.463637 (+   174us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1473523968}
I20260709 12:21:16.509675 19615 raft_consensus.cc:515] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:16.447387 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:16.477164 19707 raft_consensus.cc:493] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:21:16.515166 17831 raft_consensus.cc:2468] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 1.
I20260709 12:21:16.509704 17680 rpcz_store.cc:276] 0709 12:21:14.974445 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:14.974641 (+   196us) service_pool.cc:224] Handling call
0709 12:21:14.975008 (+   367us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:15.213218 (+238210us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:15.214547 (+  1329us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:15.228177 (+ 13630us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:21:15.242465 (+ 14288us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:16.475276 (+1232811us) spinlock_profiling.cc:243] Waited 217 ms on lock 0x7b5400207954. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:21:16.475323 (+    47us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:16.475472 (+   149us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":499214720}
I20260709 12:21:16.505681 19659 raft_consensus.cc:515] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:16.525341 17762 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38776 (request call id 1575) took 1390 ms. Trace:
I20260709 12:21:16.518091 19594 raft_consensus.cc:515] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:16.536603 17755 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38786 (request call id 2218) took 1055 ms. Trace:
I20260709 12:21:16.537515 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:16.536296 17762 rpcz_store.cc:276] 0709 12:21:15.134602 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:15.187870 (+ 53268us) service_pool.cc:224] Handling call
0709 12:21:15.382356 (+194486us) spinlock_profiling.cc:243] Waited 31.1 ms on lock 0x7b5800033398. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b474fbb71 00007f7b474fba22 00007f7b504148d9 00007f7b4f56eee0 00007f7b4f6380ab 00007f7b4f638131 00007f7b4f5f9fc0 00007f7b4f5e4929 00007f7b4f5e43d2 00007f7b48b5bbd4
0709 12:21:15.382530 (+   174us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:15.483155 (+100625us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:15.483184 (+    29us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:15.483227 (+    43us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:21:15.483255 (+    28us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:16.524969 (+1041714us) spinlock_profiling.cc:243] Waited 100 ms on lock 0x7b54003e44d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:21:16.525020 (+    51us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:16.525152 (+   132us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":302315136}
I20260709 12:21:16.537254 17755 rpcz_store.cc:276] 0709 12:21:15.480966 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:16.222740 (+741774us) service_pool.cc:224] Handling call
0709 12:21:16.536505 (+313765us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:16.542631 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:16.543828 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:16.544842 19663 leader_election.cc:290] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:16.546413 17759 raft_consensus.cc:2468] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 2.
I20260709 12:21:16.546231 19696 raft_consensus.cc:515] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:21:16.550206 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 113597us
I20260709 12:21:16.552433 17654 leader_election.cc:304] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:16.515682 19668 raft_consensus.cc:2749] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:21:16.547556 19659 leader_election.cc:290] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:16.547827 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:16.558327 17762 raft_consensus.cc:2468] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 2.
W20260709 12:21:16.560001 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 342671us
I20260709 12:21:16.556167 19713 raft_consensus.cc:493] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:21:16.551101 19707 raft_consensus.cc:515] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:16.529445 19652 leader_election.cc:290] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:16.561319 17654 leader_election.cc:304] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:21:16.553810 19687 raft_consensus.cc:493] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:21:16.571316 17801 leader_election.cc:304] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:21:16.555964 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 98851.9us
I20260709 12:21:16.528070 19658 raft_consensus.cc:515] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:16.558941 19693 raft_consensus.cc:493] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:21:16.548359 17755 raft_consensus.cc:2468] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 2.
I20260709 12:21:16.581537 17728 leader_election.cc:304] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:21:16.578392 17801 leader_election.cc:304] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:21:16.568779 19679 raft_consensus.cc:493] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
W20260709 12:21:16.624317 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 45994.8us
I20260709 12:21:16.624830 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:16.585108 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:21:16.625780 17763 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38786 (request call id 2216) took 1505 ms (client timeout 1630 ms). Trace:
I20260709 12:21:16.625973 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:21:16.626185 17763 rpcz_store.cc:269] 0709 12:21:15.120820 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:15.151100 (+ 30280us) service_pool.cc:224] Handling call
0709 12:21:16.574284 (+1423184us) spinlock_profiling.cc:243] Waited 1.03 ms on lock 0x7b5800033398. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b474fbb71 00007f7b474fba22 00007f7b504148d9 00007f7b4f56eee0 00007f7b4f6380ab 00007f7b4f638131 00007f7b4f5e60a0 00007f7b4f5e5984 00007f7b48b606d4 00007f7b48b60662
0709 12:21:16.625698 (+ 51414us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2364672}
I20260709 12:21:16.627161 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:16.627374 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:16.627445 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:16.589573 19615 leader_election.cc:290] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:16.554481 19702 raft_consensus.cc:515] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:16.564659 19503 raft_consensus.cc:493] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:21:16.591562 19648 leader_election.cc:290] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:16.592067 19668 raft_consensus.cc:2804] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:21:16.592500 19649 raft_consensus.cc:2804] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:21:16.583393 19687 raft_consensus.cc:515] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:16.593351 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:16.641881 17759 raft_consensus.cc:2468] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 2.
I20260709 12:21:16.593597 19692 leader_election.cc:290] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:16.568709 19703 raft_consensus.cc:2749] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:21:16.589627 19705 raft_consensus.cc:515] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:16.597011 19707 leader_election.cc:290] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:16.567368 19706 raft_consensus.cc:493] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:21:16.603847 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:16.606281 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:16.658736 17687 raft_consensus.cc:2468] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 2.
W20260709 12:21:16.659322 17687 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37402 (request call id 2028) took 1773 ms (client timeout 2209 ms). Trace:
I20260709 12:21:16.606922 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:16.660744 17684 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 1.
W20260709 12:21:16.661356 17684 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37402 (request call id 2027) took 1775 ms (client timeout 2210 ms). Trace:
I20260709 12:21:16.608304 17681 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37402 (request call id 2037) took 1439 ms. Trace:
I20260709 12:21:16.566605 17803 leader_election.cc:304] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:21:16.627831 19693 raft_consensus.cc:515] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:16.669404 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:16.669816 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:16.670055 17755 raft_consensus.cc:2468] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 2.
I20260709 12:21:16.670301 19693 leader_election.cc:290] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:16.670509 17760 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38776 (request call id 1585) took 1052 ms. Trace:
I20260709 12:21:16.573876 19713 raft_consensus.cc:515] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:16.671756 19702 leader_election.cc:290] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 4 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:16.628355 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:16.589191 19696 leader_election.cc:290] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 3 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:16.676441 19715 raft_consensus.cc:2749] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:21:16.611768 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:16.678464 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:16.679652 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:16.655864 19663 raft_consensus.cc:493] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:21:16.680179 17682 raft_consensus.cc:2468] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 1.
I20260709 12:21:16.680253 19663 raft_consensus.cc:515] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:16.681089 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:16.681205 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:16.682080 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:16.682300 19663 leader_election.cc:290] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:16.629143 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:16.630041 19658 leader_election.cc:290] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:16.583367 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:21:16.624462 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 54859.8us
I20260709 12:21:16.633308 17685 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37402 (request call id 2039) took 1446 ms. Trace:
I20260709 12:21:16.687652 17802 leader_election.cc:304] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
W20260709 12:21:16.641072 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 59620.3us
I20260709 12:21:16.636157 19679 raft_consensus.cc:515] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:16.643873 19503 raft_consensus.cc:515] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:16.644655 19688 raft_consensus.cc:493] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:21:16.641522 19649 raft_consensus.cc:493] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:21:16.695719 19649 raft_consensus.cc:3060] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:16.645102 19615 raft_consensus.cc:2749] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:21:16.646215 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:16.645143 19687 leader_election.cc:290] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:16.647131 17764 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38786 (request call id 2211) took 1683 ms. Trace:
I20260709 12:21:16.647420 17761 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38776 (request call id 1572) took 1524 ms. Trace:
I20260709 12:21:16.648058 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:16.641645 19714 raft_consensus.cc:493] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:21:16.638571 19617 raft_consensus.cc:493] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:21:16.648653 19603 raft_consensus.cc:2804] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:21:16.649228 17683 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37396 (request call id 1624) took 1584 ms. Trace:
I20260709 12:21:16.640767 19668 raft_consensus.cc:493] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:21:16.649784 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:16.692729 19715 raft_consensus.cc:2804] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260709 12:21:16.663527 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 97034.8us
I20260709 12:21:16.664640 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:16.704623 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:16.664835 17756 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38786 (request call id 2212) took 1680 ms. Trace:
I20260709 12:21:16.664732 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:16.665620 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:16.706665 19690 raft_consensus.cc:2955] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 3 LEADER]: Committing config change with OpId 3.5: config changed from index 4 to 5, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:21:16.666946 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:16.667135 19705 leader_election.cc:290] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 4 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:16.651027 19682 raft_consensus.cc:493] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:21:16.667862 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:16.718708 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:21:16.659795 17687 rpcz_store.cc:269] 0709 12:21:14.886096 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:14.887403 (+  1307us) service_pool.cc:224] Handling call
0709 12:21:15.679751 (+792348us) spinlock_profiling.cc:243] Waited 3.78 ms on lock 0x7b5800002a98. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b474fbb71 00007f7b474fba22 00007f7b504148d9 00007f7b4f56eee0 00007f7b4f6380ab 00007f7b4f638131 00007f7b4f5e60a0 00007f7b4f5e5984 00007f7b48b606d4 00007f7b48b60662
0709 12:21:16.659243 (+979492us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":8697088}
I20260709 12:21:16.675666 19707 raft_consensus.cc:493] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:21:16.723968 19682 raft_consensus.cc:515] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:16.725179 19503 leader_election.cc:290] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:16.676295 19713 leader_election.cc:290] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:16.674467 17757 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38786 (request call id 2214) took 1606 ms. Trace:
I20260709 12:21:16.670862 17760 rpcz_store.cc:276] 0709 12:21:15.618583 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:16.235470 (+616887us) service_pool.cc:224] Handling call
0709 12:21:16.669685 (+434215us) spinlock_profiling.cc:243] Waited 3.89 ms on lock 0x7b5800033398. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b474fbb71 00007f7b474fba22 00007f7b504148d9 00007f7b4f56eee0 00007f7b4f6380ab 00007f7b4f638131 00007f7b4f5e60a0 00007f7b4f5e5984 00007f7b48b606d4 00007f7b48b60662
0709 12:21:16.670459 (+   774us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":8943488}
W20260709 12:21:16.661795 17684 rpcz_store.cc:269] 0709 12:21:14.885636 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:14.904652 (+ 19016us) service_pool.cc:224] Handling call
0709 12:21:15.880070 (+975418us) spinlock_profiling.cc:243] Waited 2.32 ms on lock 0x7b5800002a98. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b474fbb71 00007f7b474fba22 00007f7b504148d9 00007f7b4f56eee0 00007f7b4f6380ab 00007f7b4f638131 00007f7b4f5e60a0 00007f7b4f5e5984 00007f7b48b606d4 00007f7b48b60662
0709 12:21:16.661282 (+781212us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5327232}
I20260709 12:21:16.682665 17682 raft_consensus.cc:3060] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:16.671057 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:16.686682 18367 raft_consensus.cc:2468] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 2.
I20260709 12:21:16.690760 17728 leader_election.cc:304] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:21:16.687232 17685 rpcz_store.cc:276] 0709 12:21:15.186642 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:15.675482 (+488840us) service_pool.cc:224] Handling call
0709 12:21:16.632705 (+957223us) spinlock_profiling.cc:243] Waited 26.4 ms on lock 0x7b5800002a98. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b474fbb71 00007f7b474fba22 00007f7b504148d9 00007f7b4f56eee0 00007f7b4f6380ab 00007f7b4f638131 00007f7b4f5f9fc0 00007f7b4f5e4929 00007f7b4f5e43d2 00007f7b48b5bbd4
0709 12:21:16.632884 (+   179us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:16.633028 (+   144us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:16.633050 (+    22us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:16.633085 (+    35us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:21:16.633112 (+    27us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:16.633157 (+    45us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:16.633258 (+   101us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":60686208}
I20260709 12:21:16.671983 19706 raft_consensus.cc:515] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:16.694800 19688 raft_consensus.cc:515] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:16.699535 17764 rpcz_store.cc:276] 0709 12:21:14.947474 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:14.989695 (+ 42221us) service_pool.cc:224] Handling call
0709 12:21:15.057478 (+ 67783us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:15.228157 (+170679us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 12:21:15.228187 (+    30us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:15.242385 (+ 14198us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:21:15.242438 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:16.630288 (+1387850us) spinlock_profiling.cc:243] Waited 146 ms on lock 0x7b5400414bd4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:21:16.630327 (+    39us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:16.630454 (+   127us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":335962496}
W20260709 12:21:16.703114 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 73014.8us
I20260709 12:21:16.652076 17836 raft_consensus.cc:1240] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 3. Current term is 4. Ops: []
I20260709 12:21:16.699649 17761 rpcz_store.cc:276] 0709 12:21:15.122511 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:15.131713 (+  9202us) service_pool.cc:224] Handling call
0709 12:21:15.134105 (+  2392us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:15.476843 (+342738us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:15.476871 (+    28us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:15.476913 (+    42us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:21:15.476945 (+    32us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:16.647116 (+1170171us) spinlock_profiling.cc:243] Waited 342 ms on lock 0x7b540019fad4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:21:16.647158 (+    42us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:16.647260 (+   102us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":787746688}
I20260709 12:21:16.706245 19603 raft_consensus.cc:493] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:21:16.704083 17683 rpcz_store.cc:276] 0709 12:21:14.973475 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:14.973643 (+   168us) service_pool.cc:224] Handling call
0709 12:21:14.973965 (+   322us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:14.998158 (+ 24193us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:14.998205 (+    47us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:14.998264 (+    59us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:21:14.998315 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:16.557727 (+1559412us) spinlock_profiling.cc:243] Waited 19.8 ms on lock 0x7b5400425ed4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:21:16.557764 (+    37us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:16.557885 (+   121us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":45542272}
I20260709 12:21:16.710044 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:16.710327 17763 raft_consensus.cc:3060] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:16.666368 17758 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38776 (request call id 1570) took 1544 ms. Trace:
I20260709 12:21:16.709740 17756 rpcz_store.cc:276] 0709 12:21:14.986667 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:14.989992 (+  3325us) service_pool.cc:224] Handling call
0709 12:21:15.006184 (+ 16192us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:15.641366 (+635182us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:15.641405 (+    39us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:15.641446 (+    41us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:21:15.641481 (+    35us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:16.664514 (+1023033us) spinlock_profiling.cc:243] Waited 635 ms on lock 0x7b54000fe6d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:21:16.664563 (+    49us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:16.664678 (+   115us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1460257536}
I20260709 12:21:16.714855 18362 raft_consensus.cc:2468] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 1.
I20260709 12:21:16.716125 19593 raft_consensus.cc:493] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:21:16.716729 17803 leader_election.cc:304] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:21:16.712229 17680 raft_consensus.cc:2955] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 4 to 5, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:21:16.718251 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:16.693753 19648 raft_consensus.cc:493] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:21:16.662971 17681 rpcz_store.cc:276] 0709 12:21:15.169103 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:15.642463 (+473360us) service_pool.cc:224] Handling call
0709 12:21:16.607591 (+965128us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:16.607829 (+   238us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:16.607866 (+    37us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:16.607956 (+    90us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:21:16.608018 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:16.608096 (+    78us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:16.608232 (+   136us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:16.745021 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:16.699853 19669 raft_consensus.cc:493] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:21:16.727454 17757 rpcz_store.cc:276] 0709 12:21:15.065455 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:15.083727 (+ 18272us) service_pool.cc:224] Handling call
0709 12:21:15.129482 (+ 45755us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:15.504817 (+375335us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:15.545346 (+ 40529us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:15.545396 (+    50us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:21:15.545438 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:16.671679 (+1126241us) spinlock_profiling.cc:243] Waited 373 ms on lock 0x7b540019f5d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:21:16.671716 (+    37us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:16.671825 (+   109us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":857979776}
I20260709 12:21:16.708458 19668 raft_consensus.cc:3060] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:16.733780 17728 leader_election.cc:304] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:21:16.748900 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:16.750905 19593 raft_consensus.cc:515] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:16.739450 17758 rpcz_store.cc:276] 0709 12:21:15.121707 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:15.146377 (+ 24670us) service_pool.cc:224] Handling call
0709 12:21:15.414487 (+268110us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:15.903312 (+488825us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:15.903347 (+    35us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:15.903395 (+    48us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:21:15.903427 (+    32us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:16.666070 (+762643us) spinlock_profiling.cc:243] Waited 486 ms on lock 0x7b54003e5dd4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:21:16.666114 (+    44us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:16.666212 (+    98us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1117801344}
I20260709 12:21:16.734948 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:16.710232 19715 raft_consensus.cc:493] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:21:16.746408 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:16.741209 17760 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38776 (request call id 1576) took 1582 ms. Trace:
I20260709 12:21:16.760583 17760 rpcz_store.cc:276] 0709 12:21:15.160343 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:16.734800 (+1574457us) service_pool.cc:224] Handling call
0709 12:21:16.735073 (+   273us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:16.735235 (+   162us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:16.735261 (+    26us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:16.735299 (+    38us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:21:16.735332 (+    33us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:16.735396 (+    64us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:16.735504 (+   108us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:16.752385 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:16.740660 19603 raft_consensus.cc:3060] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:16.745626 17685 raft_consensus.cc:2468] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 2.
I20260709 12:21:16.704928 19657 consensus_queue.cc:1048] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.032s
I20260709 12:21:16.751199 17683 raft_consensus.cc:2468] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 1.
I20260709 12:21:16.745759 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:16.752928 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:16.754582 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:16.754961 19688 leader_election.cc:290] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:16.725909 19714 raft_consensus.cc:515] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:16.756449 19706 leader_election.cc:290] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:16.745029 19705 raft_consensus.cc:493] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:21:16.723831 19707 raft_consensus.cc:515] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:16.796370 17801 leader_election.cc:304] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:21:16.733992 18333 leader_election.cc:304] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:21:16.720495 19617 raft_consensus.cc:515] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:16.788007 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:16.786834 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:21:16.798236 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 64303.6us
I20260709 12:21:16.788275 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:16.761693 17837 raft_consensus.cc:2468] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 2.
I20260709 12:21:16.764008 17763 raft_consensus.cc:2468] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 3.
I20260709 12:21:16.800114 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:16.764894 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:16.793182 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:16.788669 17759 raft_consensus.cc:2468] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 2.
I20260709 12:21:16.796576 17686 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37402 (request call id 2041) took 1479 ms. Trace:
I20260709 12:21:16.739019 19692 raft_consensus.cc:2749] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:21:16.740540 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:16.758628 19679 leader_election.cc:290] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:16.752975 19702 raft_consensus.cc:2804] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:21:16.803721 17727 leader_election.cc:304] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:21:16.803987 19693 raft_consensus.cc:2804] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:21:16.804906 19696 raft_consensus.cc:2804] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:21:16.804854 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:16.805378 19705 raft_consensus.cc:515] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:16.787797 19648 raft_consensus.cc:515] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:16.764714 19669 raft_consensus.cc:515] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:16.874931 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:16.875268 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:16.800971 17686 rpcz_store.cc:276] 0709 12:21:15.317217 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:15.882628 (+565411us) service_pool.cc:224] Handling call
0709 12:21:16.795914 (+913286us) spinlock_profiling.cc:243] Waited 26.5 ms on lock 0x7b5800002a98. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b474fbb71 00007f7b474fba22 00007f7b504148d9 00007f7b4f56eee0 00007f7b4f6380ab 00007f7b4f638131 00007f7b4f5f9fc0 00007f7b4f5e4929 00007f7b4f5e43d2 00007f7b48b5bbd4
0709 12:21:16.796106 (+   192us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:16.796262 (+   156us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:16.796285 (+    23us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:16.796322 (+    37us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:21:16.796357 (+    35us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:16.796409 (+    52us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:16.796521 (+   112us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":60978432}
I20260709 12:21:16.797677 17801 leader_election.cc:304] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:21:16.765398 19715 raft_consensus.cc:3060] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:16.807081 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:16.863778 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:16.865823 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:16.788230 19615 raft_consensus.cc:2749] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260709 12:21:16.868932 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 69359.1us
W20260709 12:21:16.934954 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 137305us
I20260709 12:21:16.862900 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:16.913342 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:16.937767 19706 raft_consensus.cc:2804] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:21:17.062155 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:17.063581 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:17.064778 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:17.065805 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:17.066831 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:17.067788 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:16.942602 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:16.944845 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:16.945094 17684 raft_consensus.cc:2468] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 1.
I20260709 12:21:16.947201 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:16.868546 19702 raft_consensus.cc:493] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:17.074357 19702 raft_consensus.cc:3060] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:17.075320 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:17.060591 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:17.061487 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:17.076432 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:16.878198 19693 raft_consensus.cc:493] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:21:17.071928 18365 raft_consensus.cc:3060] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:16.911645 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:16.900882 19613 raft_consensus.cc:2804] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:21:17.072650 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:17.085592 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:17.089783 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:17.090946 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:17.067703 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:17.091554 17685 raft_consensus.cc:2468] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 2.
I20260709 12:21:17.098383 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:17.100097 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:17.062121 19706 raft_consensus.cc:493] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:21:17.105545 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:17.114840 17684 raft_consensus.cc:3060] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:17.113826 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:17.115700 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:17.071936 19707 leader_election.cc:290] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:17.117183 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:17.122452 17686 raft_consensus.cc:2468] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 2.
I20260709 12:21:17.120209 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:17.085997 19693 raft_consensus.cc:3060] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:17.121749 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:17.125684 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:17.132242 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:17.088757 19613 raft_consensus.cc:493] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:21:17.117615 19706 raft_consensus.cc:3060] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:17.150666 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:17.152637 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:17.121073 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:17.121718 19667 raft_consensus.cc:493] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:21:17.151443 17686 raft_consensus.cc:2468] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 6.
I20260709 12:21:17.163173 17757 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38776 (request call id 1580) took 1948 ms. Trace:
I20260709 12:21:17.169004 17757 rpcz_store.cc:276] 0709 12:21:15.214629 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:17.136375 (+1921746us) service_pool.cc:224] Handling call
0709 12:21:17.152403 (+ 16028us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:17.162662 (+ 10259us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:17.162724 (+    62us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:17.162785 (+    61us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:21:17.162836 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:17.162917 (+    81us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:17.163099 (+   182us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:17.140974 19613 raft_consensus.cc:3060] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:17.180112 17682 raft_consensus.cc:2468] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 4.
I20260709 12:21:17.183036 19593 leader_election.cc:290] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:17.184432 17685 raft_consensus.cc:1240] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 3. Current term is 4. Ops: []
I20260709 12:21:17.185268 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:17.181891 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:17.190302 17682 raft_consensus.cc:2468] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 2.
I20260709 12:21:17.192678 17757 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38776 (request call id 1582) took 1581 ms. Trace:
I20260709 12:21:17.193099 17757 rpcz_store.cc:276] 0709 12:21:15.611656 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:17.170154 (+1558498us) service_pool.cc:224] Handling call
0709 12:21:17.185490 (+ 15336us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:17.192283 (+  6793us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:17.192327 (+    44us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:17.192373 (+    46us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:21:17.192413 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:17.192499 (+    86us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:17.192613 (+   114us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:17.174335 19667 raft_consensus.cc:515] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:17.215068 17757 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38786 (request call id 2221) took 1187 ms. Trace:
I20260709 12:21:17.219141 17757 rpcz_store.cc:276] 0709 12:21:16.028200 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:17.201043 (+1172843us) service_pool.cc:224] Handling call
0709 12:21:17.214465 (+ 13422us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:17.214656 (+   191us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:17.214689 (+    33us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:17.214744 (+    55us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:21:17.214794 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:17.214857 (+    63us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:17.214989 (+   132us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:17.217159 19617 leader_election.cc:290] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:17.210235 19705 leader_election.cc:290] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:17.234134 17757 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38786 (request call id 2222) took 1187 ms. Trace:
I20260709 12:21:17.236207 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:17.237069 18365 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 4.
I20260709 12:21:17.236685 17757 rpcz_store.cc:276] 0709 12:21:16.046668 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:17.220402 (+1173734us) service_pool.cc:224] Handling call
0709 12:21:17.233503 (+ 13101us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:17.233712 (+   209us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:17.233746 (+    34us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:17.233801 (+    55us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:21:17.233843 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:17.233913 (+    70us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:17.234064 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:21:17.238184 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 117336us
W20260709 12:21:17.244625 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 121362us
I20260709 12:21:17.244743 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:17.245409 17760 raft_consensus.cc:2468] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 1.
I20260709 12:21:17.246743 17760 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38786 (request call id 2224) took 1189 ms. Trace:
W20260709 12:21:17.249358 17785 debug-util.cc:398] Leaking SignalData structure 0x7b0800386e00 after lost signal to thread 18409
I20260709 12:21:17.251358 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:17.253230 17756 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38786 (request call id 2225) took 1195 ms. Trace:
I20260709 12:21:17.253569 17756 rpcz_store.cc:276] 0709 12:21:16.057717 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:17.252347 (+1194630us) service_pool.cc:224] Handling call
0709 12:21:17.252617 (+   270us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:17.252775 (+   158us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:17.252798 (+    23us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:17.252835 (+    37us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:21:17.252864 (+    29us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:17.252912 (+    48us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:17.253008 (+    96us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:17.255244 17756 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38776 (request call id 1591) took 1026 ms. Trace:
I20260709 12:21:17.263034 17688 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37396 (request call id 1628) took 2287 ms. Trace:
I20260709 12:21:17.263499 17688 rpcz_store.cc:276] 0709 12:21:14.975425 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:14.975524 (+    99us) service_pool.cc:224] Handling call
0709 12:21:14.975813 (+   289us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:15.477356 (+501543us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:15.477389 (+    33us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:15.477442 (+    53us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:21:15.477473 (+    31us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:17.262659 (+1785186us) spinlock_profiling.cc:243] Waited 501 ms on lock 0x7b54001aac54. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:21:17.262701 (+    42us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:17.262830 (+   129us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1153008512}
I20260709 12:21:17.264915 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:17.265805 17833 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37994 (request call id 454) took 1044 ms. Trace:
I20260709 12:21:17.266126 17833 rpcz_store.cc:276] 0709 12:21:16.221708 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:16.224985 (+  3277us) service_pool.cc:224] Handling call
0709 12:21:17.264774 (+1039789us) spinlock_profiling.cc:243] Waited 2.04 ms on lock 0x7b580013fe98. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b474fbb71 00007f7b474fba22 00007f7b504148d9 00007f7b4f56eee0 00007f7b4f6380ab 00007f7b4f638131 00007f7b4f5e60a0 00007f7b4f5e5984 00007f7b48b606d4 00007f7b48b60662
0709 12:21:17.265727 (+   953us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4700544}
I20260709 12:21:17.247064 17760 rpcz_store.cc:276] 0709 12:21:16.057385 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:17.246038 (+1188653us) service_pool.cc:224] Handling call
0709 12:21:17.246278 (+   240us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:17.246435 (+   157us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:17.246459 (+    24us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:17.246496 (+    37us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:21:17.246535 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:17.246585 (+    50us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:17.246671 (+    86us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:17.274255 17761 raft_consensus.cc:1240] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 2. Current term is 3. Ops: []
I20260709 12:21:17.278864 19594 leader_election.cc:290] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:17.255568 17756 rpcz_store.cc:276] 0709 12:21:16.229052 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:17.254509 (+1025457us) service_pool.cc:224] Handling call
0709 12:21:17.254779 (+   270us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:17.254967 (+   188us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:17.254991 (+    24us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:17.255028 (+    37us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:21:17.255057 (+    29us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:17.255103 (+    46us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:17.255196 (+    93us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:17.286093 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:17.298249 17757 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38786 (request call id 2223) took 1241 ms. Trace:
I20260709 12:21:17.298727 17757 rpcz_store.cc:276] 0709 12:21:16.057015 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:17.287772 (+1230757us) service_pool.cc:224] Handling call
0709 12:21:17.297785 (+ 10013us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:17.297953 (+   168us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:17.297977 (+    24us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:17.298016 (+    39us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:21:17.298044 (+    28us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:17.298095 (+    51us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:17.298194 (+    99us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:21:17.298808 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 59495.8us
I20260709 12:21:17.308314 17759 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38776 (request call id 1581) took 1700 ms. Trace:
I20260709 12:21:17.308915 17759 rpcz_store.cc:276] 0709 12:21:15.607950 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:17.156179 (+1548229us) service_pool.cc:224] Handling call
0709 12:21:17.307190 (+151011us) spinlock_profiling.cc:243] Waited 2.82 ms on lock 0x7b5800033398. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b474fbb71 00007f7b474fba22 00007f7b504148d9 00007f7b4f56eee0 00007f7b4f6380ab 00007f7b4f638131 00007f7b4f5f9fc0 00007f7b4f5e4929 00007f7b4f5e43d2 00007f7b48b5bbd4
0709 12:21:17.307467 (+   277us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:17.307777 (+   310us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:17.307820 (+    43us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:17.307932 (+   112us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:21:17.307989 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:17.308072 (+    83us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:17.308213 (+   141us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":6489472}
I20260709 12:21:17.312206 19682 leader_election.cc:290] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:17.291631 19648 leader_election.cc:290] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:17.291424 17755 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38776 (request call id 1577) took 2104 ms. Trace:
W20260709 12:21:17.303980 19668 consensus_meta.cc:287] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.507s	user 0.001s	sys 0.009s
I20260709 12:21:17.301172 17829 raft_consensus.cc:1240] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 3. Current term is 4. Ops: []
I20260709 12:21:17.295683 17680 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37402 (request call id 2042) took 1213 ms. Trace:
I20260709 12:21:17.350351 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:21:17.365759 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 118092us
I20260709 12:21:17.347848 19668 raft_consensus.cc:515] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:17.378286 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:17.379582 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:17.381870 17684 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 4.
I20260709 12:21:17.386076 17683 raft_consensus.cc:1240] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 3. Current term is 4. Ops: [3.5-3.5]
I20260709 12:21:17.341292 17755 rpcz_store.cc:276] 0709 12:21:15.186168 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:17.126209 (+1940041us) service_pool.cc:224] Handling call
0709 12:21:17.186576 (+ 60367us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:17.209810 (+ 23234us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:17.212011 (+  2201us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:17.218268 (+  6257us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:21:17.221277 (+  3009us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:17.232140 (+ 10863us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:17.288578 (+ 56438us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:17.369701 19593 raft_consensus.cc:493] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:21:17.394490 19696 raft_consensus.cc:493] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:21:17.395027 19696 raft_consensus.cc:3060] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:17.347601 17680 rpcz_store.cc:276] 0709 12:21:16.082345 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:16.685998 (+603653us) service_pool.cc:224] Handling call
0709 12:21:16.702757 (+ 16759us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:16.712105 (+  9348us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 12:21:17.066860 (+354755us) log.cc:844] Serialized 14 byte log entry
0709 12:21:17.221143 (+154283us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:17.222481 (+  1338us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:21:17.232117 (+  9636us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:17.240788 (+  8671us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:17.293661 (+ 52873us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":7072,"raft.run_cpu_time_us":129,"raft.run_wall_time_us":131,"thread_start_us":144012,"threads_started":1,"wal-append.queue_time_us":176169}
I20260709 12:21:17.366671 19702 raft_consensus.cc:515] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:17.381827 19667 leader_election.cc:290] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:17.406069 19693 raft_consensus.cc:515] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:17.410893 19668 leader_election.cc:290] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 3 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:17.412271 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:17.412436 19715 raft_consensus.cc:515] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:17.413210 17829 raft_consensus.cc:1240] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 2. Current term is 3. Ops: []
I20260709 12:21:17.441684 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:17.413535 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:17.442342 18365 raft_consensus.cc:2468] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 4.
I20260709 12:21:17.442548 17834 raft_consensus.cc:2468] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 4.
I20260709 12:21:17.416779 19714 leader_election.cc:290] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:17.417514 19706 raft_consensus.cc:515] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:17.418181 19669 leader_election.cc:290] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:17.418728 19593 raft_consensus.cc:515] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:17.420025 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:17.420217 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:17.420346 19613 raft_consensus.cc:515] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:17.421036 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:17.421485 17835 raft_consensus.cc:1240] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 2. Current term is 3. Ops: []
I20260709 12:21:17.421677 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:21:17.424278 19690 consensus_meta.cc:287] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.548s	user 0.009s	sys 0.001s
I20260709 12:21:17.424530 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:17.425419 17836 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35042 (request call id 1493) took 2415 ms. Trace:
I20260709 12:21:17.427350 17801 leader_election.cc:304] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:21:17.428031 19682 raft_consensus.cc:493] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:21:17.430529 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:17.438839 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:17.449055 17756 raft_consensus.cc:2468] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 1.
I20260709 12:21:17.451045 17803 leader_election.cc:304] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:21:17.462604 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:17.452261 17652 leader_election.cc:304] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:21:17.459481 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:17.455441 17836 rpcz_store.cc:276] 0709 12:21:15.010091 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:15.010196 (+   105us) service_pool.cc:224] Handling call
0709 12:21:15.880561 (+870365us) spinlock_profiling.cc:243] Waited 1.55 ms on lock 0x7b580013fe98. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b474fbe9c 00007f7b474fba5b 00007f7b504148d9 00007f7b4f56eee0 00007f7b4f6380ab 00007f7b4f638131 00007f7b4f5f9fc0 00007f7b4f5e4929 00007f7b4f5e43d2 00007f7b48b5bbd4
0709 12:21:15.880773 (+   212us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:16.740663 (+859890us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:17.425151 (+684488us) spinlock_profiling.cc:243] Waited 771 ms on lock 0x7b54003345d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:21:17.425324 (+   173us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1776983680}
I20260709 12:21:17.465533 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:17.448521 19667 raft_consensus.cc:493] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:21:17.466224 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:17.466593 19667 raft_consensus.cc:515] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:17.467301 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:17.467159 19682 raft_consensus.cc:515] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:17.467897 17756 raft_consensus.cc:2468] T 7cb3768d215e409fb9f5baf18d124442 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 3.
I20260709 12:21:17.468715 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:17.452075 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:17.483750 17760 raft_consensus.cc:1240] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 2. Current term is 3. Ops: []
I20260709 12:21:17.488590 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:17.491449 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:17.479482 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:17.494520 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:17.485592 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:17.485841 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:17.597065 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:17.597187 17755 raft_consensus.cc:2468] T 7cb3768d215e409fb9f5baf18d124442 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 3.
I20260709 12:21:17.482059 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:17.482091 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:17.480450 19728 raft_consensus.cc:2804] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:21:17.602705 19728 raft_consensus.cc:493] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:21:17.603147 19728 raft_consensus.cc:3060] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:17.603205 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:17.503185 17831 raft_consensus.cc:1240] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 2. Current term is 3. Ops: [2.5-2.5]
I20260709 12:21:17.479487 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:21:17.605854 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 76910.3us
I20260709 12:21:17.598658 17758 raft_consensus.cc:2955] T 7cb3768d215e409fb9f5baf18d124442 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 4 to 5, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:21:17.481868 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:17.609494 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:17.472805 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:17.610275 17838 raft_consensus.cc:2393] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 4: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:21:17.485201 17837 raft_consensus.cc:1240] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 1. Current term is 2. Ops: []
I20260709 12:21:17.596216 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:17.493881 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:17.610992 17834 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 1.
I20260709 12:21:17.472291 17801 leader_election.cc:304] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:17.443173 19679 raft_consensus.cc:493] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:21:17.502949 17757 raft_consensus.cc:2468] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 3.
W20260709 12:21:17.617596 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 145270us
I20260709 12:21:17.609192 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:17.618870 17801 leader_election.cc:304] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:17.623770 18333 leader_election.cc:304] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:17.625474 18333 leader_election.cc:304] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:17.627321 18333 leader_election.cc:304] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:17.469208 19666 raft_consensus.cc:493] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
W20260709 12:21:17.492959 19603 consensus_meta.cc:287] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.686s	user 0.005s	sys 0.004s
W20260709 12:21:17.651239 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 170397us
I20260709 12:21:17.490893 19593 leader_election.cc:290] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:21:17.495343 19649 consensus_meta.cc:287] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.799s	user 0.006s	sys 0.003s
I20260709 12:21:17.489120 19648 raft_consensus.cc:2804] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:21:17.484972 19715 leader_election.cc:290] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 2 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:17.483668 19706 leader_election.cc:290] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 3 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:17.483454 19613 leader_election.cc:290] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 3 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:17.674711 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:17.489732 19660 raft_consensus.cc:2804] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:21:17.703795 19660 raft_consensus.cc:697] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 3 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:21:17.704811 19660 consensus_queue.cc:237] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:17.441982 19657 consensus_queue.cc:1059] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.224s
I20260709 12:21:17.676582 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:17.680428 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:17.642582 19677 raft_consensus.cc:2749] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:21:17.641970 19718 raft_consensus.cc:2749] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:21:17.641333 19722 raft_consensus.cc:2749] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:21:17.642897 19668 raft_consensus.cc:2749] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:21:17.693120 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:17.689956 18361 raft_consensus.cc:1240] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 3. Current term is 4. Ops: []
I20260709 12:21:17.686434 19649 raft_consensus.cc:515] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:17.681787 19648 raft_consensus.cc:493] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:21:17.671126 19694 raft_consensus.cc:2749] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260709 12:21:17.709591 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 211855us
I20260709 12:21:17.671947 19702 leader_election.cc:290] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 3 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:21:17.718628 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 113645us
I20260709 12:21:17.676342 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:17.729622 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:17.733732 17834 raft_consensus.cc:2393] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 4: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:21:17.735288 17654 leader_election.cc:304] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:17.650923 19679 raft_consensus.cc:515] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:17.675660 19603 raft_consensus.cc:515] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:17.743535 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:21:17.723421 19689 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.218s	user 0.028s	sys 0.037s
I20260709 12:21:17.731842 19648 raft_consensus.cc:3060] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:17.748912 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:17.689378 19666 raft_consensus.cc:515] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:17.745784 17689 raft_consensus.cc:2393] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 2: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:21:17.736639 19667 leader_election.cc:290] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:17.763362 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:17.764212 17831 raft_consensus.cc:2468] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 1.
I20260709 12:21:17.755659 17836 raft_consensus.cc:1240] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 2. Current term is 3. Ops: []
W20260709 12:21:17.762362 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 92643.7us
I20260709 12:21:17.743391 19682 leader_election.cc:290] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:17.766412 17802 leader_election.cc:304] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:17.772567 19733 raft_consensus.cc:2804] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:21:17.773065 19733 raft_consensus.cc:493] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:21:17.773429 19733 raft_consensus.cc:3060] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:17.777588 19666 leader_election.cc:290] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:21:17.756597 19689 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.252s	user 0.029s	sys 0.037s
I20260709 12:21:17.780319 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:17.781193 17834 raft_consensus.cc:2468] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 3.
I20260709 12:21:17.789109 17680 raft_consensus.cc:2468] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 2.
I20260709 12:21:17.801880 17833 raft_consensus.cc:1240] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 2. Current term is 3. Ops: []
W20260709 12:21:17.805264 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 70048.3us
I20260709 12:21:17.805970 17654 leader_election.cc:304] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:17.803040 19603 leader_election.cc:290] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 3 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:17.810717 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:17.811949 19679 leader_election.cc:290] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:17.813714 19649 leader_election.cc:290] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 2 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:17.815407 19693 leader_election.cc:290] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 2 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:17.843312 18333 leader_election.cc:304] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
W20260709 12:21:17.847460 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 177074us
W20260709 12:21:17.847380 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 185393us
I20260709 12:21:17.849555 18333 leader_election.cc:304] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:17.817461 17584 catalog_manager.cc:5697] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 reported cstate change: config changed from index 4 to 5, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
W20260709 12:21:17.853508 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 85252.5us
I20260709 12:21:17.854586 17729 leader_election.cc:304] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:21:17.856202 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:17.861363 18334 leader_election.cc:304] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:17.854211 17728 leader_election.cc:304] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:21:17.863022 17728 leader_election.cc:304] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:21:17.863773 19659 raft_consensus.cc:2804] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:21:17.864205 19659 raft_consensus.cc:493] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
W20260709 12:21:17.864605 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 98269.2us
I20260709 12:21:17.864777 19594 raft_consensus.cc:2804] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:21:17.865357 19594 raft_consensus.cc:697] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 4 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:21:17.865631 17802 leader_election.cc:304] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:17.866304 19594 consensus_queue.cc:237] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:17.867245 19722 raft_consensus.cc:2804] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:21:17.867784 19722 raft_consensus.cc:697] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 4 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:21:17.864614 19659 raft_consensus.cc:3060] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:17.868490 19657 raft_consensus.cc:2749] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:21:17.870743 17728 leader_election.cc:304] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:21:17.874539 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:17.878588 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:17.844779 19707 raft_consensus.cc:2804] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:21:17.892606 17728 leader_election.cc:304] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:21:17.895776 19690 raft_consensus.cc:2749] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:21:17.896957 19690 raft_consensus.cc:2749] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:21:17.897059 17728 leader_election.cc:304] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:21:17.897783 19690 raft_consensus.cc:3055] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:21:17.898629 17802 leader_election.cc:304] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
W20260709 12:21:17.899060 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 44561.1us
I20260709 12:21:17.899801 17729 leader_election.cc:304] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:21:17.900166 17802 leader_election.cc:304] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:17.898152 17728 leader_election.cc:304] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:21:17.856281 19682 raft_consensus.cc:2804] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:21:17.903601 17729 leader_election.cc:304] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:21:17.854954 19677 raft_consensus.cc:2749] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:21:17.910655 17802 leader_election.cc:304] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
W20260709 12:21:17.911813 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 150702us
I20260709 12:21:17.908885 17801 leader_election.cc:304] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:17.868691 19722 consensus_queue.cc:237] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:17.919042 17729 leader_election.cc:304] T 7cb3768d215e409fb9f5baf18d124442 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:21:17.903889 19720 raft_consensus.cc:2749] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:21:17.903267 19657 raft_consensus.cc:2749] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:21:17.900002 19707 raft_consensus.cc:493] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:21:17.898129 19690 raft_consensus.cc:740] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 3 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:21:17.921571 17652 leader_election.cc:304] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:21:17.902395 19673 raft_consensus.cc:2804] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:21:17.890873 19694 consensus_queue.cc:1059] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.037s
I20260709 12:21:17.931941 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:17.902566 19660 raft_consensus.cc:2749] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:21:17.901173 19737 raft_consensus.cc:2749] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:21:17.909052 19714 raft_consensus.cc:2804] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:21:17.926465 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:17.910569 19682 raft_consensus.cc:493] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:21:17.925246 17801 leader_election.cc:304] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:17.937680 17729 leader_election.cc:304] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:21:17.929852 19652 raft_consensus.cc:2749] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:21:17.935722 19707 raft_consensus.cc:3060] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:17.929409 19593 raft_consensus.cc:2749] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:21:17.924607 19744 raft_consensus.cc:2749] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:21:17.968677 17689 raft_consensus.cc:1240] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 1. Current term is 2. Ops: []
W20260709 12:21:17.968768 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 43606us
I20260709 12:21:17.976042 17801 leader_election.cc:304] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:21:17.930212 19713 raft_consensus.cc:2749] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:21:17.978350 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:17.938529 19673 raft_consensus.cc:493] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:21:17.955004 17684 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c"
dest_uuid: "0da169ff82674dc1947055ce43de68a3"
mode: GRACEFUL
new_leader_uuid: "334a4e1f59944544b3b3ff36616b57da"
 from {username='slave'} at 127.0.0.1:54540
I20260709 12:21:18.005074 17684 raft_consensus.cc:606] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 1 LEADER]: Received request to transfer leadership to TS 334a4e1f59944544b3b3ff36616b57da
I20260709 12:21:17.965057 19682 raft_consensus.cc:3060] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:17.963307 19714 raft_consensus.cc:493] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
W20260709 12:21:18.011497 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 152387us
I20260709 12:21:18.014079 19689 raft_consensus.cc:2804] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260709 12:21:18.016470 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 78880.9us
I20260709 12:21:17.992084 19673 raft_consensus.cc:3060] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:18.037294 19689 raft_consensus.cc:493] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:21:18.046114 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:18.037989 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:18.053884 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:18.039371 17729 leader_election.cc:304] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:21:18.041007 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:18.055902 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:18.041386 18334 leader_election.cc:304] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:17.991251 19742 raft_consensus.cc:2749] T 7cb3768d215e409fb9f5baf18d124442 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:21:18.061086 17802 leader_election.cc:304] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:21:18.057946 18334 leader_election.cc:304] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:21:18.059211 17729 leader_election.cc:304] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:21:18.010325 19737 raft_consensus.cc:2749] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:21:18.006829 19711 raft_consensus.cc:2749] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:21:18.064422 17729 leader_election.cc:304] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:21:18.015580 19714 raft_consensus.cc:3060] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:18.067895 17729 leader_election.cc:304] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:21:18.069236 19754 raft_consensus.cc:2749] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:21:18.069439 17729 leader_election.cc:304] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
W20260709 12:21:18.070325 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 133154us
I20260709 12:21:18.064484 18334 leader_election.cc:304] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:18.006654 19745 raft_consensus.cc:2804] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:21:18.050302 19689 raft_consensus.cc:3060] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:18.073369 17729 leader_election.cc:304] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:21:18.072069 18334 leader_election.cc:304] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:18.065884 19657 raft_consensus.cc:2749] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:21:18.075042 19754 raft_consensus.cc:2749] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:21:18.065790 19613 raft_consensus.cc:2749] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:21:18.066480 19718 raft_consensus.cc:2804] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:21:18.084424 18334 leader_election.cc:304] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:18.070062 19652 raft_consensus.cc:2749] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260709 12:21:18.086248 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 134512us
W20260709 12:21:18.088447 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 112536us
I20260709 12:21:18.073019 19745 raft_consensus.cc:493] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:21:18.077028 19744 raft_consensus.cc:2749] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20260709 12:21:18.089653 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 120392us
I20260709 12:21:18.075935 19750 raft_consensus.cc:2749] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:21:18.092039 18334 leader_election.cc:304] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:18.091589 18333 leader_election.cc:304] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:18.098610 17729 leader_election.cc:304] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:21:18.019608 19755 consensus_queue.cc:1059] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.071s
I20260709 12:21:18.101666 17729 leader_election.cc:304] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:21:18.087015 19718 raft_consensus.cc:493] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:21:18.093325 19704 raft_consensus.cc:2749] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:21:18.096114 19754 raft_consensus.cc:2749] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:21:18.099913 19750 raft_consensus.cc:2804] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:21:18.097219 19745 raft_consensus.cc:3060] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:18.081929 19748 raft_consensus.cc:2749] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:21:18.085937 19733 raft_consensus.cc:515] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:18.104027 19718 raft_consensus.cc:3060] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:18.112836 19677 raft_consensus.cc:2804] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:21:18.058238 19690 consensus_queue.cc:260] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:18.117378 19690 raft_consensus.cc:3060] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:18.119125 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:18.121536 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:18.098198 19665 consensus_queue.cc:1059] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.012s
I20260709 12:21:18.095368 19652 consensus_queue.cc:1059] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.015s
I20260709 12:21:18.114135 19750 raft_consensus.cc:493] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:21:18.118232 19677 raft_consensus.cc:493] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:21:18.323477 19677 raft_consensus.cc:3060] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:18.367498 19760 raft_consensus.cc:2749] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:21:18.299767 19613 raft_consensus.cc:2749] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:21:18.300834 19750 raft_consensus.cc:3060] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:21:18.376770 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 285298us
I20260709 12:21:18.305105 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
W20260709 12:21:18.389549 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 281271us
I20260709 12:21:18.377206 19744 raft_consensus.cc:493] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
W20260709 12:21:18.384150 19659 consensus_meta.cc:287] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.516s	user 0.013s	sys 0.000s
W20260709 12:21:18.420176 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 318564us
I20260709 12:21:18.421077 17729 leader_election.cc:304] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:21:18.371325 19749 raft_consensus.cc:2804] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:21:18.386587 19748 raft_consensus.cc:493] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:21:18.388164 19704 raft_consensus.cc:493] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:21:18.410759 19613 raft_consensus.cc:3055] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:21:18.388062 19720 raft_consensus.cc:493] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
W20260709 12:21:18.441489 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 364954us
I20260709 12:21:18.438305 17729 leader_election.cc:304] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
W20260709 12:21:18.442997 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 345197us
I20260709 12:21:18.451707 19704 raft_consensus.cc:515] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:21:18.451182 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 63276.4us
I20260709 12:21:18.430042 19613 raft_consensus.cc:740] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:21:18.461655 17729 leader_election.cc:304] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:21:18.463641 18334 leader_election.cc:304] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:18.464524 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:18.465303 17684 raft_consensus.cc:2468] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 3.
I20260709 12:21:18.465255 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:18.432713 19749 raft_consensus.cc:493] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:21:18.467607 19763 raft_consensus.cc:2749] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:21:18.468713 19763 raft_consensus.cc:2749] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:21:18.423005 19659 raft_consensus.cc:515] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:18.432447 19737 consensus_queue.cc:1059] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.036s
I20260709 12:21:18.482208 17729 leader_election.cc:304] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:21:18.424613 19744 raft_consensus.cc:515] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:18.490262 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:18.449018 19720 raft_consensus.cc:515] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:18.471324 19749 raft_consensus.cc:3060] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:18.500123 19613 consensus_queue.cc:260] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:18.492738 19754 raft_consensus.cc:2749] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:21:18.501201 19669 raft_consensus.cc:3055] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:21:18.502058 19669 raft_consensus.cc:740] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:21:18.502285 19613 raft_consensus.cc:3060] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Advancing to term 2
W20260709 12:21:18.492275 19648 consensus_meta.cc:287] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.688s	user 0.009s	sys 0.003s
I20260709 12:21:18.440323 19748 raft_consensus.cc:515] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:18.495123 19733 leader_election.cc:290] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 3 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:18.502903 19669 consensus_queue.cc:260] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:18.522495 19669 raft_consensus.cc:3060] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:21:18.513756 19707 consensus_meta.cc:287] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.517s	user 0.005s	sys 0.005s
I20260709 12:21:18.495381 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:18.538861 19720 leader_election.cc:290] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:18.540040 19768 raft_consensus.cc:2749] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:21:18.523759 17680 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37402 (request call id 2055) took 1059 ms. Trace:
I20260709 12:21:18.565716 19748 leader_election.cc:290] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:18.570583 19677 raft_consensus.cc:515] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:18.523496 19648 raft_consensus.cc:515] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:18.601063 19760 raft_consensus.cc:3055] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:21:18.603722 19760 raft_consensus.cc:740] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:21:18.536145 19707 raft_consensus.cc:515] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:21:18.596685 19696 consensus_meta.cc:287] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 1.201s	user 0.014s	sys 0.000s
I20260709 12:21:18.593827 19748 raft_consensus.cc:2749] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:21:18.609066 19760 consensus_queue.cc:260] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:21:18.640626 19673 consensus_meta.cc:287] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.603s	user 0.012s	sys 0.002s
I20260709 12:21:18.614990 19745 raft_consensus.cc:515] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:21:18.621948 19690 consensus_meta.cc:287] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.504s	user 0.008s	sys 0.003s
W20260709 12:21:18.627372 19728 consensus_meta.cc:287] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 1.024s	user 0.006s	sys 0.008s
I20260709 12:21:18.629289 19707 leader_election.cc:290] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 2 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:18.595750 19704 leader_election.cc:290] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:18.629297 19659 leader_election.cc:290] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 2 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:18.572783 17680 rpcz_store.cc:276] 0709 12:21:17.458396 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:17.461375 (+  2979us) service_pool.cc:224] Handling call
0709 12:21:18.512499 (+1051124us) spinlock_profiling.cc:243] Waited 323 ms on lock 0x7b54000c4754. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:21:18.517315 (+  4816us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":742530304}
W20260709 12:21:18.632613 19767 log.cc:927] Time spent T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.127s	user 0.001s	sys 0.000s
I20260709 12:21:18.655376 18364 raft_consensus.cc:1240] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 4. Current term is 5. Ops: []
W20260709 12:21:18.657274 19682 consensus_meta.cc:287] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.609s	user 0.009s	sys 0.002s
I20260709 12:21:18.657763 19682 raft_consensus.cc:515] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:18.641889 19673 raft_consensus.cc:515] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:18.661935 19673 leader_election.cc:290] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 2 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:18.635388 19666 raft_consensus.cc:493] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:21:18.662827 19666 raft_consensus.cc:515] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:18.664084 19677 leader_election.cc:290] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 5 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:18.636147 17683 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44898 (request call id 414) took 1193 ms. Trace:
I20260709 12:21:18.665201 19666 leader_election.cc:290] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:18.665073 17683 rpcz_store.cc:276] 0709 12:21:17.453497 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:17.453670 (+   173us) service_pool.cc:224] Handling call
0709 12:21:18.512759 (+1059089us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:18.512947 (+   188us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:18.512974 (+    27us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:18.513035 (+    61us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:21:18.513080 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:18.513136 (+    56us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:18.635905 (+122769us) spinlock_profiling.cc:243] Waited 904 ms on lock 0x7b54000c4750. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a61f8ad 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:21:18.636057 (+   152us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2079060352}
I20260709 12:21:18.666538 19749 raft_consensus.cc:515] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:18.637997 19696 raft_consensus.cc:515] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:18.635303 19648 leader_election.cc:290] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 3 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:18.669809 17836 raft_consensus.cc:1240] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 2. Current term is 3. Ops: []
I20260709 12:21:18.643074 19613 raft_consensus.cc:2955] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:21:18.674962 19728 raft_consensus.cc:515] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:18.673295 19760 raft_consensus.cc:3060] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:18.691488 17837 raft_consensus.cc:1240] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 4. Current term is 5. Ops: []
I20260709 12:21:18.720754 19696 leader_election.cc:290] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 3 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:18.723167 19682 leader_election.cc:290] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 2 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:18.734179 19749 leader_election.cc:290] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 7 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:21:18.742831 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 289169us
I20260709 12:21:18.723459 19617 raft_consensus.cc:493] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:21:18.715937 19648 raft_consensus.cc:493] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
W20260709 12:21:18.744694 17652 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.642s	user 0.018s	sys 0.036s
W20260709 12:21:18.745431 17652 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.645s	user 0.019s	sys 0.036s
W20260709 12:21:18.745827 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 645033us
I20260709 12:21:18.745432 18333 leader_election.cc:304] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:21:18.752935 19617 raft_consensus.cc:515] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:18.756408 19750 raft_consensus.cc:515] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:18.758750 19617 leader_election.cc:290] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:18.758929 19750 leader_election.cc:290] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 3 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:18.759637 18365 raft_consensus.cc:1240] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 2. Current term is 3. Ops: []
I20260709 12:21:18.764554 17652 leader_election.cc:304] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:18.748622 19659 raft_consensus.cc:493] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
W20260709 12:21:18.754221 19714 consensus_meta.cc:287] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.679s	user 0.010s	sys 0.002s
W20260709 12:21:18.754372 19689 consensus_meta.cc:287] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.668s	user 0.009s	sys 0.001s
I20260709 12:21:18.750900 19749 raft_consensus.cc:2749] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:21:18.759222 19682 consensus_queue.cc:1059] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.005s
I20260709 12:21:18.741053 19764 consensus_queue.cc:1059] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.260s
W20260709 12:21:18.767266 19718 consensus_meta.cc:287] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.523s	user 0.006s	sys 0.004s
I20260709 12:21:18.749115 19648 raft_consensus.cc:515] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:21:18.772161 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 309185us
I20260709 12:21:18.752561 19744 leader_election.cc:290] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:18.798448 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:18.791491 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:18.803263 18334 leader_election.cc:304] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:21:18.799315 18367 raft_consensus.cc:2468] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 3.
I20260709 12:21:18.791209 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:18.806196 19742 raft_consensus.cc:493] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:21:18.810304 17654 leader_election.cc:304] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:21:18.820186 18368 raft_consensus.cc:1240] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 1. Current term is 2. Ops: []
I20260709 12:21:18.781880 19666 raft_consensus.cc:2749] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:21:18.774755 19714 raft_consensus.cc:515] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:18.783872 19704 raft_consensus.cc:493] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:21:18.772974 19659 raft_consensus.cc:515] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:18.781597 19689 raft_consensus.cc:515] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:18.811861 16501 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 1
W20260709 12:21:18.859762 17729 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:18.821743 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:18.933195 18333 leader_election.cc:304] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:21:18.865223 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 44788.9us
W20260709 12:21:18.945688 17727 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:18.945178 18333 leader_election.cc:304] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:18.961844 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:18.931272 16501 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:21:18.902467 19617 raft_consensus.cc:2804] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:21:18.911060 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:18.900980 19667 raft_consensus.cc:493] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:21:18.968165 19617 raft_consensus.cc:493] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:21:18.968431 18361 raft_consensus.cc:2468] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 4.
I20260709 12:21:18.968436 19667 raft_consensus.cc:515] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:18.968633 19617 raft_consensus.cc:3060] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:18.966785 18334 leader_election.cc:304] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:18.827224 19648 leader_election.cc:290] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:18.810840 19742 raft_consensus.cc:515] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:18.976862 19667 leader_election.cc:290] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:18.980121 19742 leader_election.cc:290] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:18.986259 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:18.987437 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:18.907562 19770 raft_consensus.cc:2749] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:21:18.814146 19718 raft_consensus.cc:515] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:18.989949 18334 leader_election.cc:304] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:18.990767 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:18.817950 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:18.811647 19748 raft_consensus.cc:493] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:21:18.878530 19745 leader_election.cc:290] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 3 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:18.902132 19728 leader_election.cc:290] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 5 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:18.934898 17833 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35042 (request call id 1523) took 1009 ms. Trace:
I20260709 12:21:19.005936 17759 raft_consensus.cc:1240] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 2. Current term is 3. Ops: []
I20260709 12:21:18.902396 19707 raft_consensus.cc:493] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:21:19.007079 19707 raft_consensus.cc:515] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:19.008633 19659 leader_election.cc:290] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:19.009130 19707 leader_election.cc:290] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:18.866097 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:19.009469 17837 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35042 (request call id 1524) took 1086 ms. Trace:
I20260709 12:21:18.994908 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:19.009819 17837 rpcz_store.cc:276] 0709 12:21:17.923161 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:17.923305 (+   144us) service_pool.cc:224] Handling call
0709 12:21:17.923656 (+   351us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:18.692263 (+768607us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:19.009266 (+317003us) spinlock_profiling.cc:243] Waited 767 ms on lock 0x7b5400089dd4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:21:19.009416 (+   150us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1765167872}
I20260709 12:21:19.010411 17764 raft_consensus.cc:3060] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:19.010602 19714 leader_election.cc:290] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 3 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:18.994153 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:18.995818 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:18.959249 19764 raft_consensus.cc:2749] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:21:19.018288 18365 raft_consensus.cc:2393] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 2: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:21:18.997453 18334 leader_election.cc:304] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:21:18.998530 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:19.000712 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:18.826220 19774 raft_consensus.cc:493] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:21:19.001226 19748 raft_consensus.cc:515] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:19.002175 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:19.005705 19781 raft_consensus.cc:3060] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:19.006150 17833 rpcz_store.cc:276] 0709 12:21:17.922725 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:17.924312 (+  1587us) service_pool.cc:224] Handling call
0709 12:21:18.049724 (+125412us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:18.099387 (+ 49663us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:18.100984 (+  1597us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:18.243287 (+142303us) log.cc:844] Serialized 556 byte log entry
0709 12:21:18.443323 (+200036us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:21:18.443365 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:18.833636 (+390271us) spinlock_profiling.cc:243] Waited 20.2 ms on lock 0x7b540022ae54. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:21:18.833702 (+    66us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:18.833774 (+    72us) raft_consensus.cc:1685] finished
0709 12:21:18.833798 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:18.931712 (+ 97914us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":46382976,"thread_start_us":199498,"threads_started":1,"wal-append.queue_time_us":234152}
I20260709 12:21:19.012322 17763 raft_consensus.cc:2468] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 2.
W20260709 12:21:19.012458 17728 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.547s	user 0.047s	sys 0.090s
W20260709 12:21:19.045199 17728 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.580s	user 0.048s	sys 0.091s
I20260709 12:21:18.964443 19593 raft_consensus.cc:493] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:21:19.026732 17756 raft_consensus.cc:2393] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 3: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:21:19.027293 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:18.963053 19704 raft_consensus.cc:515] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:18.999459 19768 raft_consensus.cc:493] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:21:18.999650 19677 raft_consensus.cc:2749] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:21:19.000756 19732 raft_consensus.cc:493] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:21:18.959988 19750 raft_consensus.cc:2749] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:21:19.058983 19689 leader_election.cc:290] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 4 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:19.058439 19593 raft_consensus.cc:515] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:19.062530 19714 raft_consensus.cc:493] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:18.991101 16501 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:21:19.086131 18362 raft_consensus.cc:2468] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 3.
I20260709 12:21:19.080474 17584 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35958 (request call id 407) took 1269 ms. Trace:
I20260709 12:21:19.079738 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:19.110551 17756 raft_consensus.cc:1240] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 2. Current term is 3. Ops: []
I20260709 12:21:19.090530 17764 raft_consensus.cc:2468] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 3.
I20260709 12:21:19.105671 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:21:19.094074 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 96734.3us
I20260709 12:21:19.125314 17728 leader_election.cc:304] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:21:19.122206 17803 leader_election.cc:304] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:21:19.129487 17803 leader_election.cc:304] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:21:19.132360 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:19.090762 19714 raft_consensus.cc:515] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:19.133518 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:19.134097 17759 raft_consensus.cc:3060] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:19.134507 19593 leader_election.cc:290] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:19.135200 19714 leader_election.cc:290] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:19.135437 19751 raft_consensus.cc:2749] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:21:19.137519 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:19.077005 19673 raft_consensus.cc:493] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:21:19.069065 19748 leader_election.cc:290] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:19.136651 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:19.051514 19774 raft_consensus.cc:515] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:19.173352 17801 leader_election.cc:304] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:19.175185 17801 leader_election.cc:304] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:21:19.167395 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:19.176622 17763 raft_consensus.cc:2468] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 3.
I20260709 12:21:19.155975 19781 raft_consensus.cc:2749] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:21:19.175518 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:19.186317 17759 raft_consensus.cc:2468] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 2.
I20260709 12:21:19.188026 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:19.158021 19715 raft_consensus.cc:2804] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260709 12:21:19.190475 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 380230us
I20260709 12:21:19.160609 19689 raft_consensus.cc:2804] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:21:19.161270 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:19.196290 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:19.196975 19753 raft_consensus.cc:493] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:21:19.193559 19593 raft_consensus.cc:493] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:21:19.197711 19707 raft_consensus.cc:493] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:21:19.199564 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:19.199769 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:19.200611 17686 raft_consensus.cc:1240] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 2. Current term is 3. Ops: []
I20260709 12:21:19.178979 19751 raft_consensus.cc:493] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:21:19.205269 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:19.206835 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:19.194617 17756 raft_consensus.cc:2393] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 3: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:21:19.207142 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:19.207913 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:19.066365 19768 raft_consensus.cc:515] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:19.089569 17584 rpcz_store.cc:276] 0709 12:21:17.810995 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:17.811171 (+   176us) service_pool.cc:224] Handling call
0709 12:21:19.080404 (+1269233us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:21:17.896545 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:21:17.897235 (+   690us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:21:17.897281 (+    46us) write_op.cc:435] Acquired schema lock
0709 12:21:17.902333 (+  5052us) tablet.cc:662] Decoding operations
0709 12:21:17.926566 (+ 24233us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:21:17.926677 (+   111us) write_op.cc:641] Partition lock acquired for write op
0709 12:21:17.926707 (+    30us) tablet.cc:685] Acquiring locks for 1 operations
0709 12:21:17.926997 (+   290us) tablet.cc:701] Row locks acquired
0709 12:21:17.927052 (+    55us) write_op.cc:260] PREPARE: finished
0709 12:21:17.927246 (+   194us) write_op.cc:270] Start()
0709 12:21:17.927399 (+   153us) write_op.cc:276] Timestamp: P: 1783599677927180 usec, L: 0
0709 12:21:17.927440 (+    41us) op_driver.cc:348] REPLICATION: starting
0709 12:21:17.928665 (+  1225us) log.cc:844] Serialized 595 byte log entry
0709 12:21:18.483001 (+554336us) op_driver.cc:464] REPLICATION: finished
0709 12:21:18.772458 (+289457us) write_op.cc:301] APPLY: starting
0709 12:21:18.782767 (+ 10309us) tablet.cc:1367] starting BulkCheckPresence
0709 12:21:18.784496 (+  1729us) tablet.cc:1370] finished BulkCheckPresence
0709 12:21:18.867217 (+ 82721us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:21:18.968852 (+101635us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:21:18.969609 (+   757us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0709 12:21:18.969668 (+    59us) write_op.cc:312] APPLY: finished
0709 12:21:18.969947 (+   279us) log.cc:844] Serialized 23 byte log entry
0709 12:21:18.970120 (+   173us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:21:18.970204 (+    84us) write_op.cc:454] Released schema lock
0709 12:21:18.970257 (+    53us) write_op.cc:341] FINISH: Updating metrics
0709 12:21:18.970707 (+   450us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:21:18.970735 (+    28us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":218,"reported_tablets":1,"child_traces":[["op",{"apply.queue_time_us":269189,"apply.run_cpu_time_us":3487,"apply.run_wall_time_us":199283,"num_ops":1,"prepare.queue_time_us":35278,"prepare.run_cpu_time_us":19567,"prepare.run_wall_time_us":172763,"raft.queue_time_us":409483,"raft.run_cpu_time_us":70756,"raft.run_wall_time_us":265924,"replication_time_us":555385,"thread_start_us":794726,"threads_started":4,"wal-append.queue_time_us":122726}]]}
I20260709 12:21:19.213071 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:19.145156 19690 consensus_queue.cc:1059] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.046s
I20260709 12:21:19.179849 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:19.163507 19673 raft_consensus.cc:515] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:19.186571 19713 raft_consensus.cc:2749] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:21:19.213321 19593 raft_consensus.cc:515] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:19.224373 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:19.224647 17685 raft_consensus.cc:1240] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 2. Current term is 3. Ops: []
I20260709 12:21:19.224934 17682 raft_consensus.cc:1240] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 2. Current term is 3. Ops: []
I20260709 12:21:19.225387 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:19.225551 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:19.225901 17680 raft_consensus.cc:3060] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:19.229807 17759 raft_consensus.cc:1240] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 1. Current term is 2. Ops: []
I20260709 12:21:19.192428 19735 raft_consensus.cc:2804] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:21:19.233106 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:19.234277 17801 leader_election.cc:304] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:19.298980 17681 raft_consensus.cc:2393] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 3: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
W20260709 12:21:19.299126 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 64929.7us
I20260709 12:21:19.241315 17760 raft_consensus.cc:2468] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 2.
I20260709 12:21:19.165652 19677 consensus_queue.cc:1059] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.050s
I20260709 12:21:19.200549 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:19.195497 19715 raft_consensus.cc:493] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:21:19.233547 19689 raft_consensus.cc:697] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 3 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:21:19.213059 19753 raft_consensus.cc:515] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:19.307138 17758 raft_consensus.cc:2468] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 4.
I20260709 12:21:19.252884 19771 raft_consensus.cc:2749] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:21:19.252485 19594 raft_consensus.cc:493] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:21:19.232671 19707 raft_consensus.cc:515] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:19.158357 19732 raft_consensus.cc:515] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:19.248400 19751 raft_consensus.cc:515] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:19.306246 17832 raft_consensus.cc:2468] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 2.
I20260709 12:21:19.587751 18334 leader_election.cc:304] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:21:19.594430 19677 raft_consensus.cc:2749] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:21:19.304600 19718 leader_election.cc:290] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 3 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:19.454141 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:19.562296 19750 consensus_queue.cc:1059] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.035s
I20260709 12:21:19.597628 19750 raft_consensus.cc:3055] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:21:19.597968 19750 raft_consensus.cc:740] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:21:19.598778 19593 leader_election.cc:290] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:19.598958 19750 consensus_queue.cc:260] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:19.596611 17680 raft_consensus.cc:2468] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 7.
I20260709 12:21:19.454569 19728 raft_consensus.cc:493] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:21:19.600440 19774 leader_election.cc:290] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:19.376490 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:21:19.430759 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 256375us
W20260709 12:21:19.368417 17652 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.533s	user 0.014s	sys 0.034s
W20260709 12:21:19.620227 17652 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.785s	user 0.017s	sys 0.035s
I20260709 12:21:19.621057 17688 raft_consensus.cc:3055] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:21:19.603698 19707 leader_election.cc:290] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:19.436365 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:19.628111 17834 raft_consensus.cc:1240] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 4 LEADER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 3. Current term is 4. Ops: []
I20260709 12:21:19.618942 17682 raft_consensus.cc:1240] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 6. Current term is 7. Ops: []
I20260709 12:21:19.622421 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:19.628183 18334 leader_election.cc:304] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:21:19.623530 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:19.627311 19652 raft_consensus.cc:493] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:21:19.630029 17683 raft_consensus.cc:2393] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 3: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:21:19.629938 19652 raft_consensus.cc:515] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:19.625614 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:21:19.630818 17652 connection.cc:256] RPC call timeout handler was delayed by -1.6258067119997577s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1714s for a response
I20260709 12:21:19.626212 17759 raft_consensus.cc:1275] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
W20260709 12:21:19.631289 17652 connection.cc:256] RPC call timeout handler was delayed by -1.603306712000176s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17390000000000003s for a response
W20260709 12:21:19.631654 17652 connection.cc:256] RPC call timeout handler was delayed by -1.4917067119999956s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18630000000000002s for a response
I20260709 12:21:19.632426 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:19.632653 19783 raft_consensus.cc:493] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:19.308630 19715 raft_consensus.cc:3060] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:19.633248 19783 raft_consensus.cc:515] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:19.309042 19713 raft_consensus.cc:493] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:21:19.626551 19790 consensus_queue.cc:1059] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 12:21:19.621522 19594 raft_consensus.cc:515] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:19.621635 17688 raft_consensus.cc:740] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:21:19.606915 17760 raft_consensus.cc:1240] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 1. Current term is 2. Ops: []
W20260709 12:21:19.590850 19617 consensus_meta.cc:287] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.620s	user 0.011s	sys 0.002s
I20260709 12:21:19.303146 19668 raft_consensus.cc:2749] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:21:19.625656 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
W20260709 12:21:19.550575 19789 log.cc:927] Time spent T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.057s	user 0.000s	sys 0.000s
I20260709 12:21:19.636979 19786 raft_consensus.cc:493] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:21:19.635843 17684 raft_consensus.cc:2955] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:21:19.637733 19783 leader_election.cc:290] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:19.638182 19652 leader_election.cc:290] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:19.638249 17688 consensus_queue.cc:260] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:19.308108 19735 raft_consensus.cc:697] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 3 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:21:19.640436 17763 raft_consensus.cc:3060] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:19.454403 19725 raft_consensus.cc:493] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:21:19.649320 17688 raft_consensus.cc:3060] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:19.635607 19728 raft_consensus.cc:515] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:19.641134 19718 raft_consensus.cc:2804] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:21:19.643272 19750 raft_consensus.cc:3060] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:19.658382 17763 raft_consensus.cc:2468] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 4.
I20260709 12:21:19.546514 19689 consensus_queue.cc:237] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:19.639051 19713 raft_consensus.cc:515] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:19.640602 19617 raft_consensus.cc:515] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:19.650780 19768 leader_election.cc:290] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:19.666097 19704 leader_election.cc:290] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:21:19.690088 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 77391us
I20260709 12:21:19.666234 19718 raft_consensus.cc:697] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 7 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:21:19.692077 17686 raft_consensus.cc:2393] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 4: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:21:19.653331 19786 raft_consensus.cc:515] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:19.708039 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:19.708482 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:19.708767 17681 raft_consensus.cc:2468] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 3.
I20260709 12:21:19.708977 17680 raft_consensus.cc:1240] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 1. Current term is 2. Ops: []
I20260709 12:21:19.709170 19786 leader_election.cc:290] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:19.711853 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:19.714032 17728 leader_election.cc:304] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:21:19.716805 19652 raft_consensus.cc:2804] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:21:19.682420 19725 raft_consensus.cc:515] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:19.720485 17688 raft_consensus.cc:1275] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:21:19.717329 19652 raft_consensus.cc:493] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
W20260709 12:21:19.722390 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 90133.1us
I20260709 12:21:19.722868 17688 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37402 (request call id 2068) took 1341 ms. Trace:
I20260709 12:21:19.730216 17688 rpcz_store.cc:276] 0709 12:21:18.381769 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:18.381949 (+   180us) service_pool.cc:224] Handling call
0709 12:21:18.382356 (+   407us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:19.620944 (+1238588us) spinlock_profiling.cc:243] Waited 65.6 ms on lock 0x7b540039f6a0. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a597dc0 00007f7b4a6221c0 00007f7b4a623231 00007f7b4a61fd8f 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c
0709 12:21:19.721991 (+101047us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:19.722787 (+   796us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":150764928}
W20260709 12:21:19.728932 19752 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 3 current term: 3 is higher.
I20260709 12:21:19.736799 19718 consensus_queue.cc:237] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:19.729259 19652 raft_consensus.cc:3060] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:21:19.744354 19788 log.cc:927] Time spent T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.081s	user 0.000s	sys 0.000s
I20260709 12:21:19.746186 19751 leader_election.cc:290] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:19.747097 19751 raft_consensus.cc:493] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:19.747682 19751 raft_consensus.cc:515] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:19.750710 19732 leader_election.cc:290] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:19.729943 19768 consensus_queue.cc:1059] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.026s
I20260709 12:21:19.783390 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:19.785317 17688 raft_consensus.cc:2393] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 2: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:21:19.792083 17652 leader_election.cc:304] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:19.799257 19732 raft_consensus.cc:493] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:21:19.799963 19732 raft_consensus.cc:515] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:19.800235 19713 leader_election.cc:290] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:19.804494 19704 raft_consensus.cc:493] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:21:19.805091 18364 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59886 (request call id 651) took 1320 ms. Trace:
I20260709 12:21:19.805213 19704 raft_consensus.cc:515] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:19.805878 18364 rpcz_store.cc:276] 0709 12:21:18.484785 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:18.486446 (+  1661us) service_pool.cc:224] Handling call
0709 12:21:18.496631 (+ 10185us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:18.655909 (+159278us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:19.804698 (+1148789us) spinlock_profiling.cc:243] Waited 147 ms on lock 0x7b54008039d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:21:19.804985 (+   287us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":339236224}
I20260709 12:21:19.719419 19791 consensus_queue.cc:1059] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.109s
I20260709 12:21:19.823606 19705 raft_consensus.cc:493] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:21:19.833606 19690 raft_consensus.cc:3055] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:21:19.834154 19690 raft_consensus.cc:740] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 2 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:21:19.833719 19705 raft_consensus.cc:515] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:19.835072 19690 consensus_queue.cc:260] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:19.836683 19690 raft_consensus.cc:3060] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:19.835665 17683 raft_consensus.cc:1240] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 3. Current term is 4. Ops: []
I20260709 12:21:19.840512 19786 raft_consensus.cc:2749] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260709 12:21:19.856313 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 141381us
I20260709 12:21:19.857126 17652 leader_election.cc:304] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:19.858268 17652 leader_election.cc:304] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:19.860111 17652 leader_election.cc:304] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:19.861451 19751 leader_election.cc:290] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:19.862182 19766 raft_consensus.cc:2804] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:21:19.862646 19766 raft_consensus.cc:493] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
W20260709 12:21:19.862910 19784 log.cc:927] Time spent T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.245s	user 0.004s	sys 0.000s
I20260709 12:21:19.863001 19766 raft_consensus.cc:3060] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:19.864825 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:19.870863 19783 raft_consensus.cc:2749] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:21:19.859965 19786 raft_consensus.cc:2749] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:21:19.874101 19725 leader_election.cc:290] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:19.878412 19766 raft_consensus.cc:515] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:19.878939 19652 raft_consensus.cc:515] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:19.880122 19791 consensus_queue.cc:1059] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 4.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 12:21:19.813843 19715 raft_consensus.cc:515] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:19.881147 19791 raft_consensus.cc:3055] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:21:19.881579 19791 raft_consensus.cc:740] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:21:19.882601 17833 raft_consensus.cc:2393] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 5: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:21:19.882431 19791 consensus_queue.cc:260] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:19.719699 19735 consensus_queue.cc:237] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:21:20.042469 19794 log.cc:927] Time spent T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.170s	user 0.001s	sys 0.000s
I20260709 12:21:20.093361 17652 leader_election.cc:304] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:21:20.139991 19786 raft_consensus.cc:2749] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:21:20.053756 19722 raft_consensus.cc:493] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:20.136152 19690 raft_consensus.cc:493] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:21:20.139931 19728 leader_election.cc:290] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:20.141228 19722 raft_consensus.cc:515] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:20.142158 19660 raft_consensus.cc:493] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:21:20.141762 19690 raft_consensus.cc:515] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:20.142890 19660 raft_consensus.cc:515] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:20.145566 19704 leader_election.cc:290] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:20.146660 19758 raft_consensus.cc:493] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:21:19.861310 19753 leader_election.cc:290] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:21:20.150740 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 57466.2us
I20260709 12:21:20.163163 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:20.164821 17652 leader_election.cc:304] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:20.078547 19652 leader_election.cc:290] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 4 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:20.146639 19728 raft_consensus.cc:493] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:21:20.179394 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:20.171429 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:20.167280 19786 raft_consensus.cc:2749] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:21:20.186228 17652 leader_election.cc:304] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:21:20.197650 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:20.147415 19758 raft_consensus.cc:515] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:20.201251 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:20.205184 17652 leader_election.cc:304] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:21:20.176548 19728 raft_consensus.cc:515] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:20.209023 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:20.181883 19791 raft_consensus.cc:3060] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:20.182329 19753 consensus_queue.cc:1059] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.022s
I20260709 12:21:20.216138 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:20.199081 19722 leader_election.cc:290] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:20.207484 19690 leader_election.cc:290] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:20.197417 19715 leader_election.cc:290] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 5 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:20.221531 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:20.207408 19792 raft_consensus.cc:2749] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:21:20.214864 19786 raft_consensus.cc:2749] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:21:20.232971 17687 raft_consensus.cc:1275] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 12:21:20.216084 19652 raft_consensus.cc:493] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:20.234669 17683 raft_consensus.cc:2468] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 3.
I20260709 12:21:20.222581 19749 raft_consensus.cc:3055] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:21:20.220584 19753 raft_consensus.cc:3055] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 2 LEADER]: Stepping down as leader of term 2
W20260709 12:21:20.250131 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 45028.2us
I20260709 12:21:20.252249 17802 leader_election.cc:304] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:21:20.260486 19603 consensus_queue.cc:1048] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:21:20.264882 19718 raft_consensus.cc:493] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:21:20.265465 19718 raft_consensus.cc:515] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:20.267584 19718 leader_election.cc:290] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:20.257802 19735 raft_consensus.cc:2804] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:21:20.268371 19735 raft_consensus.cc:493] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:21:20.268741 19735 raft_consensus.cc:3060] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:20.257683 19798 raft_consensus.cc:493] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:21:20.272650 19798 raft_consensus.cc:515] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:20.275475 19798 leader_election.cc:290] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:20.251698 19715 raft_consensus.cc:493] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:20.276055 19660 leader_election.cc:290] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:20.276269 19715 raft_consensus.cc:515] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:20.256577 19766 leader_election.cc:290] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 2 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:20.242030 19749 raft_consensus.cc:740] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:21:20.250715 19753 raft_consensus.cc:740] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:21:20.286470 19749 consensus_queue.cc:260] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:20.286716 19753 consensus_queue.cc:260] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:20.287979 19753 raft_consensus.cc:3060] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:20.240342 19722 raft_consensus.cc:493] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:21:20.282362 17652 leader_election.cc:304] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:21:20.284085 19617 leader_election.cc:290] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 4 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:20.284456 19715 leader_election.cc:290] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:20.284652 19728 leader_election.cc:290] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:20.240568 19652 raft_consensus.cc:515] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:20.298552 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:20.299120 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:20.299423 17829 raft_consensus.cc:2393] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 5: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:21:20.299744 19652 leader_election.cc:290] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:20.300113 17829 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37994 (request call id 484) took 1220 ms. Trace:
I20260709 12:21:20.300226 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:20.285530 19758 leader_election.cc:290] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:20.300459 17829 rpcz_store.cc:276] 0709 12:21:19.079377 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:19.088495 (+  9118us) service_pool.cc:224] Handling call
0709 12:21:20.298371 (+1209876us) spinlock_profiling.cc:243] Waited 5.92 ms on lock 0x7b580013fe98. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b474fbb71 00007f7b474fba22 00007f7b504148d9 00007f7b4f56eee0 00007f7b4f6380ab 00007f7b4f638131 00007f7b4f5e60a0 00007f7b4f5e5984 00007f7b48b606d4 00007f7b48b60662
0709 12:21:20.300051 (+  1680us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":13620992}
I20260709 12:21:20.301510 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:20.301707 17652 leader_election.cc:304] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:20.302845 17687 raft_consensus.cc:1240] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 2. Current term is 3. Ops: []
I20260709 12:21:20.300809 17685 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 4.
I20260709 12:21:20.304889 19766 raft_consensus.cc:2749] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:21:20.312542 19727 raft_consensus.cc:493] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:21:20.313134 19727 raft_consensus.cc:515] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:20.315303 19727 leader_election.cc:290] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:20.316380 19718 raft_consensus.cc:493] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:21:20.316864 19718 raft_consensus.cc:515] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:20.319778 19617 raft_consensus.cc:2749] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:21:20.327620 17682 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37396 (request call id 1671) took 1079 ms. Trace:
I20260709 12:21:20.328365 17682 rpcz_store.cc:276] 0709 12:21:19.248997 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:19.250617 (+  1620us) service_pool.cc:224] Handling call
0709 12:21:19.618724 (+368107us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:19.628737 (+ 10013us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:20.327376 (+698639us) spinlock_profiling.cc:243] Waited 368 ms on lock 0x7b5400208350. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a61f8ad 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:21:20.327547 (+   171us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":845941632}
I20260709 12:21:20.331223 19764 raft_consensus.cc:3055] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:21:20.331715 19764 raft_consensus.cc:740] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:21:20.332721 19764 consensus_queue.cc:260] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:20.349895 19722 raft_consensus.cc:515] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:20.361811 19718 leader_election.cc:290] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:20.362717 17652 leader_election.cc:304] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:21:20.363675 19753 raft_consensus.cc:2749] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:21:20.366763 17762 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38776 (request call id 1608) took 1377 ms. Trace:
I20260709 12:21:20.366787 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:20.367504 17762 rpcz_store.cc:276] 0709 12:21:18.989698 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:18.998181 (+  8483us) service_pool.cc:224] Handling call
0709 12:21:19.070001 (+ 71820us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:19.310370 (+240369us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:19.310413 (+    43us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:19.310469 (+    56us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:21:19.310522 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:20.366474 (+1055952us) spinlock_profiling.cc:243] Waited 240 ms on lock 0x7b5400217854. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:21:20.366530 (+    56us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:20.366686 (+   156us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":552218880}
W20260709 12:21:20.371441 18333 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:20.372896 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:20.373394 19617 consensus_queue.cc:1059] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:21:20.373595 17689 raft_consensus.cc:2468] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 3.
I20260709 12:21:20.374470 19792 raft_consensus.cc:3055] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:21:20.374825 19792 raft_consensus.cc:740] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:21:20.375748 17652 leader_election.cc:304] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:20.376515 19752 raft_consensus.cc:2804] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:21:20.376868 19752 raft_consensus.cc:493] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:21:20.376935 17652 leader_election.cc:304] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:21:20.377170 17729 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:20.377300 19752 raft_consensus.cc:3060] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:20.377527 19766 consensus_queue.cc:1059] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20260709 12:21:20.376987 17763 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38786 (request call id 2264) took 1208 ms (client timeout 1510 ms). Trace:
I20260709 12:21:20.378650 19766 raft_consensus.cc:3055] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 1 LEADER]: Stepping down as leader of term 1
W20260709 12:21:20.378477 17763 rpcz_store.cc:269] 0709 12:21:19.168620 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:19.177504 (+  8884us) service_pool.cc:224] Handling call
0709 12:21:20.376823 (+1199319us) spinlock_profiling.cc:243] Waited 417 ms on lock 0x7b54003bd3d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:21:20.376930 (+   107us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":958992640}
I20260709 12:21:20.379202 19766 raft_consensus.cc:740] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:21:20.379259 19753 raft_consensus.cc:2749] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:21:20.380177 19714 consensus_queue.cc:1059] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 12:21:20.381227 19714 raft_consensus.cc:3055] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:21:20.381577 19714 raft_consensus.cc:740] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:21:20.380213 19766 consensus_queue.cc:260] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:20.382498 19714 consensus_queue.cc:260] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:20.383617 19766 raft_consensus.cc:3060] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:20.386065 18334 leader_election.cc:304] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:21:20.362183 19722 leader_election.cc:290] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:20.388543 17652 leader_election.cc:304] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:21:20.389271 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 135963us
I20260709 12:21:20.383670 19714 raft_consensus.cc:3060] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:20.394745 19667 consensus_queue.cc:1059] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 12:21:20.396747 19728 raft_consensus.cc:3055] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:21:20.397126 19728 raft_consensus.cc:740] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 3 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:21:20.398193 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:20.388610 19761 raft_consensus.cc:493] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:21:20.398615 19792 consensus_queue.cc:260] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:20.398891 19761 raft_consensus.cc:515] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:20.399449 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:20.396780 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:20.400220 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:20.397908 19728 consensus_queue.cc:260] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:20.401762 18365 raft_consensus.cc:1240] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 1. Current term is 2. Ops: []
I20260709 12:21:20.400286 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:20.404110 19761 leader_election.cc:290] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:20.404403 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:20.400367 19753 raft_consensus.cc:2804] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:21:20.405017 19753 raft_consensus.cc:493] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:21:20.405457 19753 raft_consensus.cc:3060] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:20.406107 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:20.407262 19749 raft_consensus.cc:3060] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:20.408334 19735 raft_consensus.cc:515] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:20.410593 19735 leader_election.cc:290] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 3 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:20.410699 19752 raft_consensus.cc:515] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:20.403766 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:20.412451 18367 raft_consensus.cc:3060] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:20.413523 17684 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37396 (request call id 1668) took 1205 ms. Trace:
I20260709 12:21:20.413890 19728 raft_consensus.cc:3060] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:20.416031 19718 raft_consensus.cc:493] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:21:20.416621 19718 raft_consensus.cc:515] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:20.417079 19694 raft_consensus.cc:2804] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:21:20.417689 19694 raft_consensus.cc:493] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:21:20.418067 19694 raft_consensus.cc:3060] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:20.387343 19764 raft_consensus.cc:3060] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:20.413923 17684 rpcz_store.cc:276] 0709 12:21:19.208432 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:19.226470 (+ 18038us) service_pool.cc:224] Handling call
0709 12:21:19.226695 (+   225us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:19.635727 (+409032us) raft_consensus.cc:1534] Early marking committed up to index 3
0709 12:21:19.678156 (+ 42429us) log.cc:844] Serialized 14 byte log entry
0709 12:21:19.718674 (+ 40518us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:19.718744 (+    70us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:21:19.718827 (+    83us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:20.412787 (+693960us) spinlock_profiling.cc:243] Waited 1.23 ms on lock 0x7b180012e6e0. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b475012a6 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419
0709 12:21:20.413302 (+   515us) spinlock_profiling.cc:243] Waited 409 ms on lock 0x7b5400345dd4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:21:20.413346 (+    44us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:20.413466 (+   120us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":129,"raft.run_cpu_time_us":144,"raft.run_wall_time_us":1429,"spinlock_wait_cycles":942977792,"thread_start_us":40021,"threads_started":1,"wal-append.queue_time_us":171022}
I20260709 12:21:20.424381 17802 leader_election.cc:304] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:21:20.399667 19792 raft_consensus.cc:3060] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:20.421413 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:20.426508 18363 raft_consensus.cc:2468] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 3.
I20260709 12:21:20.427443 18365 raft_consensus.cc:2468] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 2.
I20260709 12:21:20.428182 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:20.401907 17652 leader_election.cc:304] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:21:20.429062 18363 raft_consensus.cc:2468] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 2.
W20260709 12:21:20.430661 17833 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37994 (request call id 481) took 1352 ms (client timeout 1598 ms). Trace:
W20260709 12:21:20.431155 17833 rpcz_store.cc:269] 0709 12:21:19.078589 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:19.089048 (+ 10459us) service_pool.cc:224] Handling call
0709 12:21:20.430451 (+1341403us) spinlock_profiling.cc:243] Waited 635 ms on lock 0x7b540008a2d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:21:20.430595 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1459409280}
I20260709 12:21:20.432279 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:20.433460 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:20.437487 19594 leader_election.cc:290] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:20.427043 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:20.444510 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:20.417125 19705 leader_election.cc:290] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:20.426911 19732 leader_election.cc:290] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:20.454880 17834 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35042 (request call id 1530) took 2507 ms. Trace:
I20260709 12:21:20.455315 17834 rpcz_store.cc:276] 0709 12:21:17.947708 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:17.957843 (+ 10135us) service_pool.cc:224] Handling call
0709 12:21:17.982434 (+ 24591us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:19.629241 (+1646807us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:20.454658 (+825417us) spinlock_profiling.cc:243] Waited 1.65 s on lock 0x7b5400266454. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:21:20.454816 (+   158us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3784199424}
I20260709 12:21:20.456149 19801 raft_consensus.cc:3060] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:20.456588 19779 raft_consensus.cc:3055] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:21:20.456806 18374 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59902 (request call id 659) took 1639 ms. Trace:
I20260709 12:21:20.456985 19779 raft_consensus.cc:740] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:21:20.457572 19705 consensus_queue.cc:1048] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:21:20.457964 19779 consensus_queue.cc:260] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:20.457263 18374 rpcz_store.cc:276] 0709 12:21:18.817110 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:18.817229 (+   119us) service_pool.cc:224] Handling call
0709 12:21:18.817522 (+   293us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:19.089170 (+271648us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:19.089201 (+    31us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:19.089242 (+    41us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:21:19.089278 (+    36us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:20.456594 (+1367316us) spinlock_profiling.cc:243] Waited 271 ms on lock 0x7b5400305f54. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:21:20.456640 (+    46us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:20.456748 (+   108us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":624231040}
I20260709 12:21:20.459018 19779 raft_consensus.cc:3060] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:20.444579 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:20.445504 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:20.449169 18371 raft_consensus.cc:1275] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:21:20.460165 17802 leader_election.cc:304] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:21:20.449791 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:20.460341 17688 raft_consensus.cc:1240] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 1. Current term is 2. Ops: []
I20260709 12:21:20.449788 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:20.462010 17763 raft_consensus.cc:1240] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 2. Current term is 3. Ops: []
I20260709 12:21:20.418092 19752 leader_election.cc:290] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 2 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:20.466059 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:20.467617 18367 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 2.
I20260709 12:21:20.470031 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:20.470692 17836 raft_consensus.cc:3060] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:20.486442 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:20.446158 19786 raft_consensus.cc:2749] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:21:20.533179 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:20.534210 18362 raft_consensus.cc:2393] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 2: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:21:20.471626 19594 raft_consensus.cc:2804] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:21:20.536348 19594 raft_consensus.cc:493] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:21:20.536844 19594 raft_consensus.cc:3060] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:20.540058 19659 raft_consensus.cc:493] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:20.533265 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:20.543922 17834 raft_consensus.cc:2468] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 2.
I20260709 12:21:20.541302 19659 raft_consensus.cc:515] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:21:20.546900 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 71692.2us
I20260709 12:21:20.562325 19755 consensus_queue.cc:1059] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:21:20.563716 19791 raft_consensus.cc:3055] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:21:20.564155 19791 raft_consensus.cc:740] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:21:20.565145 19791 consensus_queue.cc:260] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:20.471944 19760 raft_consensus.cc:3055] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:21:20.566282 19791 raft_consensus.cc:3060] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:20.566534 19760 raft_consensus.cc:740] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:21:20.567816 19760 consensus_queue.cc:260] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:21:20.570598 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:20.526557 19690 raft_consensus.cc:493] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:21:20.524225 19737 raft_consensus.cc:493] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:21:20.585623 19694 raft_consensus.cc:515] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:20.585963 19690 raft_consensus.cc:515] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:20.549575 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:20.588490 19761 raft_consensus.cc:493] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:21:20.586207 19737 raft_consensus.cc:515] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:20.605093 18368 raft_consensus.cc:2410] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 4 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260709 12:21:20.605389 19742 raft_consensus.cc:493] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:21:20.611001 19761 raft_consensus.cc:515] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:20.611868 19742 raft_consensus.cc:515] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:20.533968 19718 leader_election.cc:290] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:20.626487 19652 raft_consensus.cc:493] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:21:20.627324 19652 raft_consensus.cc:515] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:21:20.632196 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 82439.7us
I20260709 12:21:20.633849 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
W20260709 12:21:20.635644 17728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:20.636082 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:20.643287 19694 leader_election.cc:290] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 3 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:20.763289 19742 leader_election.cc:290] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:20.748759 18364 raft_consensus.cc:2468] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 3.
I20260709 12:21:20.659693 19737 leader_election.cc:290] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:21:20.668129 17730 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:20.757170 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:20.766194 19714 raft_consensus.cc:493] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:21:20.684670 17762 raft_consensus.cc:2468] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 7.
I20260709 12:21:20.766927 19714 raft_consensus.cc:515] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:21:20.738277 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:20.769770 19714 leader_election.cc:290] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:20.770433 19714 raft_consensus.cc:493] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:21:20.770985 19714 raft_consensus.cc:515] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:21:20.771957 17728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:20.755393 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 205683us
I20260709 12:21:20.774338 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:20.775067 17837 raft_consensus.cc:2393] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 3: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:21:20.773556 17761 raft_consensus.cc:1240] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 1. Current term is 2. Ops: []
W20260709 12:21:20.776551 17729 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:20.778538 18363 raft_consensus.cc:2468] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 3.
I20260709 12:21:20.582887 19659 leader_election.cc:290] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:20.776126 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:20.820262 17838 raft_consensus.cc:2468] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 2.
I20260709 12:21:20.796366 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:20.821478 18362 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 4.
I20260709 12:21:20.822810 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:20.798923 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:20.823581 18362 raft_consensus.cc:2468] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 3.
I20260709 12:21:20.802522 17652 leader_election.cc:304] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:20.825891 19799 consensus_queue.cc:1059] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:21:20.804697 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:20.755077 19761 leader_election.cc:290] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:20.827440 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:20.827718 19757 raft_consensus.cc:3055] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:21:20.827953 17652 leader_election.cc:304] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:21:20.805455 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:20.828646 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:20.829097 17687 raft_consensus.cc:2410] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 4 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260709 12:21:20.829391 19738 raft_consensus.cc:2804] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:21:20.829820 19738 raft_consensus.cc:493] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:21:20.829988 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:20.830235 19738 raft_consensus.cc:3060] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:20.830528 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:20.831110 17760 raft_consensus.cc:2468] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 2.
I20260709 12:21:20.806810 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:20.831948 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:20.698369 17830 raft_consensus.cc:2468] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 4.
I20260709 12:21:20.808480 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:20.772943 19760 raft_consensus.cc:3060] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:20.734423 17836 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 2.
I20260709 12:21:20.810751 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:20.840283 19652 leader_election.cc:290] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:20.840931 19652 raft_consensus.cc:493] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:21:20.841423 19652 raft_consensus.cc:515] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:20.842221 19738 raft_consensus.cc:515] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:20.843462 17682 raft_consensus.cc:2468] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 3.
I20260709 12:21:20.800209 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:21:20.793708 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:20.798738 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:20.824322 17729 leader_election.cc:304] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:21:20.775836 17686 raft_consensus.cc:1275] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 7 index: 9. (index mismatch)
I20260709 12:21:20.826016 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:20.828193 19757 raft_consensus.cc:740] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:21:20.808796 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:20.824347 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:20.806746 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:20.828305 19617 raft_consensus.cc:2749] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:21:20.804081 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:20.759328 19753 raft_consensus.cc:515] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:21:20.845278 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:20.846148 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:20.912607 17684 raft_consensus.cc:2393] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 4: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:21:20.829414 17652 leader_election.cc:304] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:20.847682 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:20.849294 17729 leader_election.cc:304] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:21:20.913905 18362 raft_consensus.cc:2468] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 2.
W20260709 12:21:20.914263 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 65045.9us
I20260709 12:21:20.914911 17729 leader_election.cc:304] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
W20260709 12:21:20.913533 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 84194.7us
I20260709 12:21:20.853391 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:20.846328 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:20.854318 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:20.916055 17652 leader_election.cc:304] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:20.916657 18367 raft_consensus.cc:2393] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 3: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
W20260709 12:21:20.916730 18333 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:20.864913 17801 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:20.856094 18371 raft_consensus.cc:2468] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 2.
I20260709 12:21:20.915966 17756 raft_consensus.cc:2468] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 4.
W20260709 12:21:20.913578 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 93185us
I20260709 12:21:20.855043 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:20.919625 17729 leader_election.cc:304] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:21:20.920012 17654 leader_election.cc:304] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:21:20.921008 17729 leader_election.cc:304] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:21:20.921433 18363 raft_consensus.cc:1240] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 1. Current term is 2. Ops: []
I20260709 12:21:20.921967 18333 leader_election.cc:304] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:20.922124 18362 raft_consensus.cc:1240] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 7 LEADER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 6. Current term is 7. Ops: []
I20260709 12:21:20.857247 17727 leader_election.cc:304] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:21:20.945734 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:20.942828 17652 leader_election.cc:304] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:20.937322 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:20.851064 19801 raft_consensus.cc:2749] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:21:20.937322 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:20.949775 17756 raft_consensus.cc:2468] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 2.
I20260709 12:21:20.950265 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:20.950707 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:20.845010 19738 leader_election.cc:290] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 3 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:20.848425 19690 leader_election.cc:290] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:20.949182 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:20.959488 18363 raft_consensus.cc:2468] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 4.
I20260709 12:21:20.855338 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:20.960688 18364 raft_consensus.cc:2468] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 3.
I20260709 12:21:20.954919 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:20.962996 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:20.846640 19659 raft_consensus.cc:2804] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:21:20.917507 19617 raft_consensus.cc:2749] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:21:20.925581 17728 leader_election.cc:304] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:21:20.923537 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:20.843819 19652 leader_election.cc:290] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:20.951989 17756 raft_consensus.cc:2468] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 2.
I20260709 12:21:20.939741 19742 raft_consensus.cc:2804] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:21:20.923321 19792 raft_consensus.cc:2749] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:21:20.923648 19791 raft_consensus.cc:2804] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:21:20.857736 19714 leader_election.cc:290] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:20.940287 19793 raft_consensus.cc:2804] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:21:20.940264 19786 raft_consensus.cc:2804] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Leader election won for term 3
W20260709 12:21:20.967540 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 49019us
I20260709 12:21:21.125723 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:21.125406 19786 raft_consensus.cc:697] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 3 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
W20260709 12:21:20.970739 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 48826.6us
I20260709 12:21:21.126560 17801 leader_election.cc:304] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:21:21.126843 18333 leader_election.cc:304] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:21:21.127322 19603 raft_consensus.cc:2804] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:21:21.127360 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:21.137835 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:21.138447 17763 raft_consensus.cc:2468] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 2.
I20260709 12:21:21.139196 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:20.972898 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:21.140141 19742 raft_consensus.cc:697] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 3 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:21:20.955273 19755 raft_consensus.cc:2804] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:21:21.141300 17584 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35958 (request call id 408) took 1687 ms. Trace:
I20260709 12:21:21.141100 19742 consensus_queue.cc:237] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:21.141577 19755 raft_consensus.cc:493] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:21:21.141669 17584 rpcz_store.cc:276] 0709 12:21:19.454184 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:19.454328 (+   144us) service_pool.cc:224] Handling call
0709 12:21:21.141241 (+1686913us) inbound_call.cc:177] Queueing success response
Metrics: {"reported_tablets":1}
I20260709 12:21:21.142059 19755 raft_consensus.cc:3060] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:21:21.143200 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 286026us
I20260709 12:21:21.143474 18374 raft_consensus.cc:1240] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 2. Current term is 3. Ops: []
I20260709 12:21:20.920953 19749 raft_consensus.cc:493] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:21.144488 19749 raft_consensus.cc:515] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:21.145578 19737 raft_consensus.cc:493] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:21:21.146135 19737 raft_consensus.cc:515] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:21.146591 19749 leader_election.cc:290] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:21:20.976742 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:21.147588 17583 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:37172 (request call id 251) took 1976 ms. Trace:
I20260709 12:21:20.858270 19594 raft_consensus.cc:515] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:20.966934 19659 raft_consensus.cc:493] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:20.967164 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:20.848627 19757 consensus_queue.cc:260] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:20.974992 19791 raft_consensus.cc:493] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:21:21.207224 19791 raft_consensus.cc:3060] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:21.207484 19659 raft_consensus.cc:3060] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:20.979287 19793 raft_consensus.cc:493] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:21:20.974354 19652 raft_consensus.cc:2749] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:21:20.970587 19690 consensus_queue.cc:1059] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.080s
I20260709 12:21:21.120178 19617 raft_consensus.cc:2749] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:21:20.974648 19761 raft_consensus.cc:2749] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:21:20.946715 19694 consensus_queue.cc:1048] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.025s
I20260709 12:21:21.217008 19690 raft_consensus.cc:3055] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:21:21.217430 19690 raft_consensus.cc:740] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:21:21.125841 19728 raft_consensus.cc:493] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:21:20.927260 18334 leader_election.cc:304] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:21:21.218346 19690 consensus_queue.cc:260] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:21:20.971336 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 45850us
I20260709 12:21:21.218564 19728 raft_consensus.cc:515] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:21.209682 19757 raft_consensus.cc:3060] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:21.127955 19603 raft_consensus.cc:697] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 2 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:21:21.221369 19728 leader_election.cc:290] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:21.127781 17801 leader_election.cc:304] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:21.222247 19603 consensus_queue.cc:237] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:21:21.222622 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 94897.6us
I20260709 12:21:21.128131 19779 raft_consensus.cc:2804] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:21:21.129451 19786 consensus_queue.cc:237] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:21.131356 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:21.127959 18333 leader_election.cc:304] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:21:21.224491 19705 raft_consensus.cc:2749] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:21:21.132562 19714 raft_consensus.cc:2749] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:21:21.224865 19795 raft_consensus.cc:493] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:21.225245 19714 raft_consensus.cc:493] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:21:21.225394 19795 raft_consensus.cc:515] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:21.225752 19714 raft_consensus.cc:515] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:21.226076 19705 consensus_queue.cc:1059] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:21:21.209538 19793 raft_consensus.cc:3060] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:21.227065 19705 raft_consensus.cc:3055] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:21:21.227432 19705 raft_consensus.cc:740] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:21:21.227540 19795 leader_election.cc:290] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:21.134869 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:21.232915 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:21.135906 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:21.233379 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:21.233868 17759 raft_consensus.cc:2468] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 4.
I20260709 12:21:21.149170 19807 raft_consensus.cc:493] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:21:21.150112 19760 raft_consensus.cc:493] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:21:21.149612 17586 catalog_manager.cc:5697] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb reported cstate change: config changed from index 4 to 5, term changed from 3 to 4, leader changed from e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66) to <none>, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New cstate: current_term: 4 committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:21:20.988123 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:21.148343 19737 leader_election.cc:290] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:21.150964 18368 raft_consensus.cc:1240] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 1. Current term is 2. Ops: []
I20260709 12:21:21.158265 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:21.126444 17761 raft_consensus.cc:2468] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 1.
I20260709 12:21:21.185881 19755 raft_consensus.cc:515] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:21.189685 17684 raft_consensus.cc:1275] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:21:21.192392 17682 raft_consensus.cc:1240] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 2. Current term is 3. Ops: []
W20260709 12:21:21.192826 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 216434us
I20260709 12:21:21.196120 17803 leader_election.cc:304] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3
W20260709 12:21:21.281064 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 85018.4us
I20260709 12:21:21.281774 17803 leader_election.cc:304] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:21:21.201872 17583 rpcz_store.cc:276] 0709 12:21:19.171266 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:19.171444 (+   178us) service_pool.cc:224] Handling call
0709 12:21:21.147515 (+1976071us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":640148,"reported_tablets":3}
I20260709 12:21:21.139704 17763 raft_consensus.cc:2468] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 2.
I20260709 12:21:21.205942 19594 leader_election.cc:290] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 4 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:21.207496 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
W20260709 12:21:21.213265 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 262521us
I20260709 12:21:21.219792 19690 raft_consensus.cc:3060] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Advancing to term 2
W20260709 12:21:21.218679 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 291504us
I20260709 12:21:21.285754 17652 leader_election.cc:304] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:21.290280 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:21.291451 17836 raft_consensus.cc:2393] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 4: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:21:21.133157 19753 leader_election.cc:290] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 4 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:20.969759 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:21.299335 17756 raft_consensus.cc:2468] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 2.
I20260709 12:21:21.227849 19714 leader_election.cc:290] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:21.223802 19779 raft_consensus.cc:493] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
W20260709 12:21:21.224887 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 97017.6us
I20260709 12:21:21.228240 19761 raft_consensus.cc:2804] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:21:21.228040 19808 raft_consensus.cc:493] T 7cb3768d215e409fb9f5baf18d124442 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:21:21.229187 19694 raft_consensus.cc:2804] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:21:21.229642 19705 consensus_queue.cc:260] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:21.230383 17801 leader_election.cc:304] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:21:21.263628 17758 raft_consensus.cc:2468] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 2.
I20260709 12:21:21.230381 19667 consensus_queue.cc:1048] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.051s
W20260709 12:21:21.272450 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 51116.2us
I20260709 12:21:21.276132 17762 raft_consensus.cc:2468] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 1.
I20260709 12:21:21.296129 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:21.262593 17586 catalog_manager.cc:5697] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb reported cstate change: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 12:21:21.309038 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:21.316612 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:21.231925 19689 consensus_queue.cc:1059] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.050s
I20260709 12:21:21.316833 17689 raft_consensus.cc:1240] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 1. Current term is 2. Ops: []
I20260709 12:21:21.474396 19714 raft_consensus.cc:493] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:21.316740 19812 raft_consensus.cc:2804] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:21:21.476583 19812 raft_consensus.cc:493] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:21:21.476341 19714 raft_consensus.cc:515] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:21.477240 19812 raft_consensus.cc:3060] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:21:21.477815 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 166624us
I20260709 12:21:21.479938 17834 raft_consensus.cc:2468] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 2.
I20260709 12:21:21.324517 17761 raft_consensus.cc:1275] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:21:21.481712 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:21:21.485603 17728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:21.374097 17757 raft_consensus.cc:2393] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 3: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:21:21.281710 19652 raft_consensus.cc:493] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:21.489578 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 159760us
I20260709 12:21:21.495146 17652 leader_election.cc:304] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:21:21.492003 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 207245us
I20260709 12:21:21.496268 17729 leader_election.cc:304] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:21:21.499987 17835 raft_consensus.cc:1240] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 1. Current term is 2. Ops: []
I20260709 12:21:21.493777 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:21.500777 17837 raft_consensus.cc:3060] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:21.318245 19779 raft_consensus.cc:3060] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:21.478174 17687 raft_consensus.cc:2468] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 2.
I20260709 12:21:21.297950 19728 raft_consensus.cc:2804] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:21:21.323479 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:21.507709 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:21.511655 17683 raft_consensus.cc:2468] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 3.
I20260709 12:21:21.311635 19809 raft_consensus.cc:2749] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:21:21.286594 19673 leader_election.cc:290] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:21.317651 19737 raft_consensus.cc:493] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:21.279484 19760 raft_consensus.cc:515] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:21:21.515833 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 233258us
I20260709 12:21:21.530434 17803 leader_election.cc:304] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:21:21.351766 17755 raft_consensus.cc:1240] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 3. Current term is 4. Ops: []
I20260709 12:21:21.517560 19821 raft_consensus.cc:2804] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:21:21.371760 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:21.533416 19705 raft_consensus.cc:3060] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:21.473089 19815 raft_consensus.cc:493] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:21:21.503500 19742 raft_consensus.cc:3060] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:21.535008 17728 leader_election.cc:304] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:21:21.535434 17729 leader_election.cc:304] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:21:21.509091 19749 raft_consensus.cc:493] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:21.510533 19728 raft_consensus.cc:493] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:21:21.276568 19807 raft_consensus.cc:515] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:21.515105 19738 raft_consensus.cc:2749] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260709 12:21:21.548027 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 317690us
I20260709 12:21:21.549515 17801 leader_election.cc:304] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:21:21.556056 17803 leader_election.cc:304] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:21:21.322701 19793 raft_consensus.cc:515] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:21.373039 19659 raft_consensus.cc:515] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:21.536794 19689 raft_consensus.cc:2749] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:21:21.572207 17729 leader_election.cc:304] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:21:21.569636 17755 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38776 (request call id 1620) took 1104 ms. Trace:
W20260709 12:21:21.575141 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 80066.8us
I20260709 12:21:21.509608 19652 raft_consensus.cc:515] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:21.576717 17652 leader_election.cc:304] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:21:21.585049 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:21.549232 19728 raft_consensus.cc:3060] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:21.548623 19796 raft_consensus.cc:2804] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:21:21.590665 17760 raft_consensus.cc:2468] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 3.
I20260709 12:21:21.592650 17652 leader_election.cc:304] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:21:21.598328 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:21.599134 17761 raft_consensus.cc:3060] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:21.599532 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:21.601670 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:21.537712 19673 consensus_queue.cc:1059] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.039s
I20260709 12:21:21.578234 17729 leader_election.cc:304] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:21:21.604920 17762 raft_consensus.cc:1240] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 4. Current term is 5. Ops: []
I20260709 12:21:21.539623 19737 raft_consensus.cc:515] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:21.573045 19795 raft_consensus.cc:2749] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:21:21.613770 17830 raft_consensus.cc:1240] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 1. Current term is 2. Ops: []
I20260709 12:21:21.502058 19808 raft_consensus.cc:515] T 7cb3768d215e409fb9f5baf18d124442 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:21:21.616799 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 81876.4us
I20260709 12:21:21.549403 19749 raft_consensus.cc:515] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:21.586764 19826 raft_consensus.cc:2804] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:21:21.625268 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:21.627140 19761 raft_consensus.cc:697] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 3 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:21:21.574117 17755 rpcz_store.cc:276] 0709 12:21:20.465414 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:20.469321 (+  3907us) service_pool.cc:224] Handling call
0709 12:21:21.237450 (+768129us) spinlock_profiling.cc:243] Waited 12 ms on lock 0x7b5800033398. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b474fbb71 00007f7b474fba22 00007f7b504148d9 00007f7b4f56eee0 00007f7b4f6380ab 00007f7b4f638131 00007f7b4f5f9fc0 00007f7b4f5e4929 00007f7b4f5e43d2 00007f7b48b5bbd4
0709 12:21:21.279037 (+ 41587us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:21.534417 (+255380us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:21.563576 (+ 29159us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":27566464}
W20260709 12:21:21.634063 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 84662.5us
I20260709 12:21:21.646606 17801 leader_election.cc:304] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:21.648597 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:21.649960 17801 leader_election.cc:304] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:21.636662 19817 raft_consensus.cc:2804] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:21:21.631392 17835 raft_consensus.cc:2468] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 2.
I20260709 12:21:21.534049 19815 raft_consensus.cc:515] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:21.612857 19796 raft_consensus.cc:493] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:21.650120 17758 raft_consensus.cc:1275] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
W20260709 12:21:21.657675 17727 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:21.660811 17727 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:21.641486 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:21.624694 19757 raft_consensus.cc:493] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:21:21.654989 17801 leader_election.cc:304] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:21.616725 19667 raft_consensus.cc:2749] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:21:21.611707 19738 raft_consensus.cc:2804] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:21:21.660557 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:21.662475 19796 raft_consensus.cc:3060] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:21.666042 19753 raft_consensus.cc:2764] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Leader pre-election decision vote started in defunct term 6: won
I20260709 12:21:21.639818 19826 raft_consensus.cc:493] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:21:21.671634 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:21.672181 19757 raft_consensus.cc:515] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:21.678570 19760 leader_election.cc:290] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:21:21.684392 19819 log.cc:927] Time spent T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.108s	user 0.001s	sys 0.000s
I20260709 12:21:21.659001 19755 leader_election.cc:290] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 3 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:21.647962 19791 raft_consensus.cc:515] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:21.745736 19791 leader_election.cc:290] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 3 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:21.686399 19749 leader_election.cc:290] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:21.656508 19827 raft_consensus.cc:2749] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:21:21.746281 19809 raft_consensus.cc:493] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:21.746817 19809 raft_consensus.cc:515] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:21.694089 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:21.748337 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:21.696310 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:21.749424 17835 raft_consensus.cc:2468] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 2.
I20260709 12:21:21.749521 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:21.749744 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:21.750814 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:21.751170 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:21.697232 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:21.723078 19737 leader_election.cc:290] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:21.753116 19809 leader_election.cc:290] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:21.698964 19714 leader_election.cc:290] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:21.753677 19761 consensus_queue.cc:237] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:21.700270 19652 leader_election.cc:290] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:21.700703 19835 raft_consensus.cc:3060] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:21.701854 19793 leader_election.cc:290] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 3 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:21:21.702708 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 82976.8us
I20260709 12:21:21.635839 19821 raft_consensus.cc:697] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 4 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:21:21.709530 17727 leader_election.cc:304] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:21:21.655813 19817 raft_consensus.cc:493] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:21:21.711161 17837 raft_consensus.cc:2468] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 2.
W20260709 12:21:21.712857 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 213990us
I20260709 12:21:21.775484 18333 leader_election.cc:304] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:21:21.713681 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 380402us
I20260709 12:21:21.776059 17836 raft_consensus.cc:1240] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 1. Current term is 2. Ops: []
I20260709 12:21:21.660256 19833 raft_consensus.cc:2804] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:21:21.722126 17759 raft_consensus.cc:1240] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 2. Current term is 3. Ops: []
I20260709 12:21:21.666790 17801 leader_election.cc:304] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:21:21.617748 19831 consensus_queue.cc:1048] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.090s
I20260709 12:21:21.779273 18334 leader_election.cc:304] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:21:21.727236 17833 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35040 (request call id 1784) took 1261 ms. Trace:
I20260709 12:21:21.789350 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:21.856191 17688 raft_consensus.cc:2468] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 2.
I20260709 12:21:21.734474 19757 leader_election.cc:290] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:21.735033 17689 raft_consensus.cc:2468] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 1.
I20260709 12:21:21.739198 18368 raft_consensus.cc:1275] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260709 12:21:21.881085 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:21.881700 18368 raft_consensus.cc:3060] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:21.882606 19742 raft_consensus.cc:2749] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:21:21.742384 17682 raft_consensus.cc:1240] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 3. Current term is 4. Ops: []
I20260709 12:21:21.885072 17728 leader_election.cc:304] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:21:21.855836 19690 raft_consensus.cc:2749] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:21:21.886415 19690 raft_consensus.cc:2804] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:21:21.886791 19690 raft_consensus.cc:493] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:21.887135 19690 raft_consensus.cc:3060] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:21.888568 19793 consensus_queue.cc:1059] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:21:21.889019 17760 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38786 (request call id 2274) took 1062 ms. Trace:
I20260709 12:21:21.889631 19793 raft_consensus.cc:3055] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:21:21.890007 19793 raft_consensus.cc:740] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:21:21.742648 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:21.743472 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:21.890760 17759 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38776 (request call id 1624) took 1423 ms. Trace:
I20260709 12:21:21.891218 17684 raft_consensus.cc:2468] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 3.
I20260709 12:21:21.891276 17759 rpcz_store.cc:276] 0709 12:21:20.467704 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:21.234674 (+766970us) service_pool.cc:224] Handling call
0709 12:21:21.721834 (+487160us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:21.781992 (+ 60158us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:21.890478 (+108486us) spinlock_profiling.cc:243] Waited 487 ms on lock 0x7b54000fc3d0. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a61f8ad 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:21:21.890658 (+   180us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1119441280}
I20260709 12:21:21.728121 19826 raft_consensus.cc:3060] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:21.743788 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:21.752697 17831 raft_consensus.cc:2393] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 3: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:21:21.697986 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:21.699683 19603 raft_consensus.cc:3055] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:21:21.759531 17683 raft_consensus.cc:1275] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260709 12:21:21.761900 19659 leader_election.cc:290] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 5 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:21.900225 19603 raft_consensus.cc:740] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:21:21.766491 17761 raft_consensus.cc:2468] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 3.
I20260709 12:21:21.879230 19812 raft_consensus.cc:515] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:21.901489 19603 consensus_queue.cc:260] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:21.902755 19603 raft_consensus.cc:3060] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:21.903471 19812 leader_election.cc:290] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 3 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:21.714425 19694 raft_consensus.cc:697] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 2 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:21:21.687638 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
W20260709 12:21:21.776743 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 67315.2us
I20260709 12:21:21.905869 17727 leader_election.cc:304] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
W20260709 12:21:21.779022 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 112308us
I20260709 12:21:21.777932 19817 raft_consensus.cc:3060] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:21.853662 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:21.739961 19808 leader_election.cc:290] T 7cb3768d215e409fb9f5baf18d124442 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:21.705466 19738 raft_consensus.cc:493] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:21:21.851397 19829 raft_consensus.cc:2749] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:21:21.918550 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:21.782001 19749 raft_consensus.cc:493] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:21:21.794466 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:21.933719 18362 raft_consensus.cc:2468] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 3.
I20260709 12:21:21.862625 19728 raft_consensus.cc:515] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:21.936033 17834 raft_consensus.cc:1240] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 2. Current term is 3. Ops: []
I20260709 12:21:21.937649 19728 leader_election.cc:290] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 3 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:21.862943 19807 leader_election.cc:290] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:21.864177 19795 raft_consensus.cc:3055] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:21:21.940276 19795 raft_consensus.cc:740] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:21:21.865916 19815 leader_election.cc:290] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:21.777053 19689 raft_consensus.cc:2749] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:21:21.856017 19594 raft_consensus.cc:2804] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:21:21.796067 17764 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 2.
I20260709 12:21:21.953445 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:21.911586 17830 raft_consensus.cc:1240] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 2. Current term is 3. Ops: []
I20260709 12:21:21.894474 17728 leader_election.cc:304] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:21:21.809149 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:21.853224 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:21.782181 19755 consensus_queue.cc:1059] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.118s
I20260709 12:21:21.968540 19755 raft_consensus.cc:3055] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:21:21.968909 19755 raft_consensus.cc:740] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 2 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:21:21.970127 19755 consensus_queue.cc:260] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:21.852404 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:21.855665 17833 rpcz_store.cc:276] 0709 12:21:20.466303 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:20.471440 (+  5137us) service_pool.cc:224] Handling call
0709 12:21:21.642235 (+1170795us) spinlock_profiling.cc:243] Waited 3.73 ms on lock 0x7b580013fe98. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b474fbb71 00007f7b474fba22 00007f7b504148d9 00007f7b4f56eee0 00007f7b4f6380ab 00007f7b4f638131 00007f7b4f5e60a0 00007f7b4f5e5984 00007f7b48b606d4 00007f7b48b60662
0709 12:21:21.727169 (+ 84934us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":8581376}
I20260709 12:21:21.971565 18366 raft_consensus.cc:2468] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 1.
I20260709 12:21:21.965821 17832 raft_consensus.cc:1240] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 2. Current term is 3. Ops: []
I20260709 12:21:21.972676 18367 raft_consensus.cc:2468] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 3.
I20260709 12:21:21.907199 18368 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 5.
I20260709 12:21:21.972952 19690 raft_consensus.cc:515] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:21.907752 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:21.975329 17761 raft_consensus.cc:2468] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 4.
I20260709 12:21:21.975168 19779 raft_consensus.cc:515] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:21.907747 17727 leader_election.cc:304] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:21:21.888586 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:21:21.995548 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 87868.2us
I20260709 12:21:21.996353 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:21.997145 17727 leader_election.cc:304] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:21:21.778334 19833 raft_consensus.cc:493] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:21:21.997852 17829 raft_consensus.cc:2393] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 3: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:21:21.998035 19833 raft_consensus.cc:3060] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:21.852828 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:21.908588 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:21:21.909747 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 130547us
I20260709 12:21:21.921440 17838 raft_consensus.cc:2468] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 2.
I20260709 12:21:21.924039 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:21.925465 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:22.010926 17831 raft_consensus.cc:1240] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 2. Current term is 3. Ops: []
I20260709 12:21:22.011606 17837 raft_consensus.cc:2393] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 3: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:21:21.790073 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:21.858170 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:21.791944 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:21.941695 18371 raft_consensus.cc:2468] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 3.
I20260709 12:21:21.889425 17760 rpcz_store.cc:276] 0709 12:21:20.826756 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:20.972401 (+145645us) service_pool.cc:224] Handling call
0709 12:21:21.888871 (+916470us) spinlock_profiling.cc:243] Waited 450 ms on lock 0x7b54004155d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:21:21.888964 (+    93us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1035643264}
I20260709 12:21:22.013408 18363 raft_consensus.cc:2468] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 5.
I20260709 12:21:21.891384 17685 raft_consensus.cc:2393] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 3: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:21:21.783420 18333 leader_election.cc:304] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:21:22.085358 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 302028us
I20260709 12:21:22.086055 18333 leader_election.cc:304] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:21:21.924569 19738 raft_consensus.cc:3060] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:21:21.966806 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 72688.2us
I20260709 12:21:21.967327 19807 raft_consensus.cc:2804] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:21:21.785641 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:21.913342 19821 consensus_queue.cc:237] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:21.972676 19817 raft_consensus.cc:515] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:21.965658 19594 raft_consensus.cc:493] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:21:21.978763 19826 raft_consensus.cc:515] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:21.980672 17729 leader_election.cc:304] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:21:21.981135 17764 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38776 (request call id 1625) took 1512 ms (client timeout 1803 ms). Trace:
W20260709 12:21:22.108191 17764 rpcz_store.cc:269] 0709 12:21:20.468794 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:20.486749 (+ 17955us) service_pool.cc:224] Handling call
0709 12:21:21.131200 (+644451us) spinlock_profiling.cc:243] Waited 12.6 ms on lock 0x7b5800033398. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b474fbb71 00007f7b474fba22 00007f7b504148d9 00007f7b4f56eee0 00007f7b4f6380ab 00007f7b4f638131 00007f7b4f5e60a0 00007f7b4f5e5984 00007f7b48b606d4 00007f7b48b60662
0709 12:21:21.980904 (+849704us) spinlock_profiling.cc:243] Waited 541 ms on lock 0x7b54003bced4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:21:21.981051 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1272419456}
I20260709 12:21:21.890766 19793 consensus_queue.cc:260] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:22.109313 19845 raft_consensus.cc:493] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:21:21.941267 19795 consensus_queue.cc:260] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:21.905745 19806 consensus_queue.cc:1048] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.059s
I20260709 12:21:21.910465 19694 consensus_queue.cc:237] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:22.096398 19829 raft_consensus.cc:2804] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:21:22.110677 19829 raft_consensus.cc:493] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:21:22.111052 19829 raft_consensus.cc:3060] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:21.991981 17801 leader_election.cc:304] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:21:21.993258 17833 raft_consensus.cc:1240] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 2. Current term is 3. Ops: []
I20260709 12:21:21.993940 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:21.999778 17727 leader_election.cc:304] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
W20260709 12:21:22.116689 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 116963us
I20260709 12:21:22.118083 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:22.119109 17727 leader_election.cc:304] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:21:21.910223 19842 raft_consensus.cc:2804] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:21:22.007093 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:21.919384 19659 raft_consensus.cc:2804] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:21:22.015816 18364 raft_consensus.cc:2393] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 4: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:21:22.009835 18334 leader_election.cc:304] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:21:22.109750 19793 raft_consensus.cc:3060] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:22.017540 19742 raft_consensus.cc:2749] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:21:21.998044 19779 leader_election.cc:290] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 4 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:22.017515 19652 raft_consensus.cc:2749] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:21:22.038751 17652 leader_election.cc:304] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:21:22.136217 19817 leader_election.cc:290] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 5 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:21:22.136623 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 97928.4us
I20260709 12:21:22.138311 17652 leader_election.cc:304] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
W20260709 12:21:22.043759 19844 log.cc:927] Time spent T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.110s	user 0.002s	sys 0.000s
I20260709 12:21:22.045645 19801 consensus_queue.cc:1059] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:21:22.048277 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:21.952392 19749 raft_consensus.cc:515] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:22.048969 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:22.049356 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:22.051424 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:22.109895 19845 raft_consensus.cc:515] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:21:22.160105 17803 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.16.29.68:42577 timed out after 1.742s (SENT) [suppressed 11 similar messages]
I20260709 12:21:22.043751 19689 consensus_queue.cc:1048] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:21:22.048643 19755 raft_consensus.cc:3060] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:22.120604 19842 raft_consensus.cc:493] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:21:22.074534 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:22.075254 19835 raft_consensus.cc:2749] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:21:22.094027 18374 raft_consensus.cc:2468] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 3.
I20260709 12:21:22.094678 18333 leader_election.cc:304] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:21:22.096063 19807 raft_consensus.cc:493] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:22.105484 19830 consensus_queue.cc:1048] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.006s
W20260709 12:21:22.108250 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 127684us
I20260709 12:21:22.142902 19825 raft_consensus.cc:2804] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:21:22.119226 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:22.121440 17727 leader_election.cc:304] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:21:22.111979 19594 raft_consensus.cc:3060] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:22.173843 19807 raft_consensus.cc:3060] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:22.122556 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:22.096400 19840 raft_consensus.cc:2804] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260709 12:21:22.186249 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 64851.2us
I20260709 12:21:22.186973 17727 leader_election.cc:304] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:21:22.094542 19808 raft_consensus.cc:493] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:22.128968 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 119249us
I20260709 12:21:22.203214 19845 leader_election.cc:290] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:21.952457 19761 raft_consensus.cc:493] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:21:22.203933 19761 raft_consensus.cc:515] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:22.205476 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:22.123698 19737 raft_consensus.cc:2749] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:21:22.112644 19613 raft_consensus.cc:2804] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:21:22.133240 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:22.139986 17652 leader_election.cc:304] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:21:22.121340 19815 raft_consensus.cc:2804] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:21:22.143440 17831 raft_consensus.cc:2393] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 3: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:21:22.143761 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:22.052129 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:22.124467 19714 raft_consensus.cc:2804] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:21:22.161072 17838 raft_consensus.cc:2468] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 5.
I20260709 12:21:22.187510 19779 raft_consensus.cc:2804] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:21:22.128754 19659 raft_consensus.cc:493] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:21:22.130056 19690 leader_election.cc:290] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 3 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:22.072992 19796 raft_consensus.cc:515] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:22.129262 19826 leader_election.cc:290] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 4 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:22.131914 19841 raft_consensus.cc:2804] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260709 12:21:22.169941 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 75346.8us
I20260709 12:21:22.143663 19799 raft_consensus.cc:2804] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:21:22.222249 19833 raft_consensus.cc:515] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:22.174513 17729 leader_election.cc:304] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:21:22.127360 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:22.188062 17835 raft_consensus.cc:1240] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 3. Current term is 4. Ops: []
W20260709 12:21:22.234545 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 60127.9us
I20260709 12:21:22.234898 17834 raft_consensus.cc:2393] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 3: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:21:22.168349 19842 raft_consensus.cc:3060] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:22.226346 19659 raft_consensus.cc:3060] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:22.215284 17727 leader_election.cc:304] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:21:22.220430 19714 raft_consensus.cc:493] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:21:22.174500 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
W20260709 12:21:22.217803 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 57721.5us
I20260709 12:21:22.164498 16501 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:21:22.213825 19808 raft_consensus.cc:515] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:22.187743 19825 raft_consensus.cc:493] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:21:22.207166 19837 raft_consensus.cc:2804] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:21:22.200356 19840 raft_consensus.cc:493] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:21:22.213250 19742 raft_consensus.cc:2749] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260709 12:21:22.113395 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 121552us
I20260709 12:21:22.232278 19841 raft_consensus.cc:493] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:22.215757 19613 raft_consensus.cc:493] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:21:22.217907 19815 raft_consensus.cc:493] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:21:22.235535 17729 leader_election.cc:304] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:21:22.230917 18333 leader_election.cc:304] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:21:22.237731 17727 leader_election.cc:304] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:21:22.169531 19786 raft_consensus.cc:493] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
W20260709 12:21:22.248557 17730 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:22.223752 19779 raft_consensus.cc:493] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
W20260709 12:21:22.221410 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 81553.6us
I20260709 12:21:22.252529 17652 leader_election.cc:304] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:21:22.225212 19761 leader_election.cc:290] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:22.236533 19799 raft_consensus.cc:697] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 2 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:21:22.240968 19690 raft_consensus.cc:2749] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:21:22.245716 17801 leader_election.cc:304] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:21:22.249336 17729 leader_election.cc:304] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:21:22.239835 19826 raft_consensus.cc:2804] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:21:22.245157 17680 raft_consensus.cc:3060] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Advancing to term 3
I20260709 12:21:22.244987 19825 raft_consensus.cc:3060] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:22.253334 17727 leader_election.cc:304] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:21:22.241981 19714 raft_consensus.cc:3060] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:22.247620 18333 leader_election.cc:304] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:21:22.246394 19841 raft_consensus.cc:3060] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:22.259763 19838 raft_consensus.cc:2804] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:21:22.259724 19742 raft_consensus.cc:2749] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:21:22.254259 19786 raft_consensus.cc:515] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:22.267853 19705 consensus_queue.cc:1048] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:21:22.268770 19848 consensus_queue.cc:1048] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.006s
I20260709 12:21:22.254199 19817 raft_consensus.cc:2749] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:21:22.260228 17729 leader_election.cc:304] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:21:22.246526 19840 raft_consensus.cc:3060] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:22.248471 19815 raft_consensus.cc:3060] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:22.248550 19613 raft_consensus.cc:3060] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:22.263764 18333 leader_election.cc:304] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:21:22.265491 17801 leader_election.cc:304] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:21:22.244753 19837 raft_consensus.cc:493] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:21:22.254153 19779 raft_consensus.cc:3060] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:22.271889 17652 leader_election.cc:304] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:22.258296 19809 raft_consensus.cc:2804] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:21:22.272660 17801 leader_election.cc:304] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:21:22.273808 17729 leader_election.cc:304] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:21:22.257998 19652 consensus_queue.cc:1059] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.008s
I20260709 12:21:22.261077 19835 raft_consensus.cc:2804] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:21:22.269858 19786 leader_election.cc:290] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:22.293222 17759 raft_consensus.cc:1240] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 2. Current term is 3. Ops: []
I20260709 12:21:22.286728 17760 raft_consensus.cc:1240] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 3. Current term is 4. Ops: []
I20260709 12:21:22.295859 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:22.298308 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:22.296782 17838 raft_consensus.cc:1240] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 2. Current term is 3. Ops: []
I20260709 12:21:22.298620 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:22.299353 17757 raft_consensus.cc:2368] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for earlier term 2. Current term is 3.
I20260709 12:21:22.272868 19690 raft_consensus.cc:2749] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20260709 12:21:22.303943 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 97892.9us
I20260709 12:21:22.304332 17761 raft_consensus.cc:1275] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260709 12:21:22.274686 19837 raft_consensus.cc:3060] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:22.305358 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:22.264019 19738 raft_consensus.cc:515] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:22.307996 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:21:22.309582 17730 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:22.310698 19838 raft_consensus.cc:697] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:21:22.270179 19801 consensus_queue.cc:1059] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.032s
I20260709 12:21:22.278461 19848 raft_consensus.cc:2804] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:21:22.280552 19835 raft_consensus.cc:493] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:21:22.299825 19705 raft_consensus.cc:2804] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:21:22.307485 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:22.292976 19809 raft_consensus.cc:697] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 4 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:21:22.323179 17652 leader_election.cc:304] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:21:22.303691 19795 raft_consensus.cc:3060] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:22.303735 19742 raft_consensus.cc:2804] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:21:22.301081 19749 leader_election.cc:290] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:22.303553 19764 raft_consensus.cc:2804] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:21:22.321866 19804 raft_consensus.cc:2749] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:21:22.325851 19845 raft_consensus.cc:493] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:22.349458 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:22.350229 17682 raft_consensus.cc:2468] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 2.
I20260709 12:21:22.328019 19705 raft_consensus.cc:493] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:22.328892 19835 raft_consensus.cc:3060] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:22.313360 19766 consensus_queue.cc:1059] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.025s
I20260709 12:21:22.327991 19848 raft_consensus.cc:493] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:21:22.335034 17830 raft_consensus.cc:1240] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 4. Current term is 5. Ops: []
I20260709 12:21:22.334869 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:22.340636 19764 raft_consensus.cc:493] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:21:22.349216 19749 raft_consensus.cc:2804] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:21:22.348591 19761 raft_consensus.cc:2804] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:21:22.351277 19843 raft_consensus.cc:2804] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:21:22.347774 19858 raft_consensus.cc:2804] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:21:22.347648 19742 raft_consensus.cc:697] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:21:22.329212 19839 consensus_queue.cc:1059] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.037s
I20260709 12:21:22.365286 19757 raft_consensus.cc:2804] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:21:22.362718 19848 raft_consensus.cc:3060] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:22.321048 19799 consensus_queue.cc:237] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:22.364042 19705 raft_consensus.cc:3060] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:22.380728 19764 raft_consensus.cc:3060] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:22.378429 19833 leader_election.cc:290] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 3 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:22.400900 19861 raft_consensus.cc:3055] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:21:22.380260 19749 raft_consensus.cc:493] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:21:22.351840 19845 raft_consensus.cc:515] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:22.383848 19761 raft_consensus.cc:493] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:21:22.384188 19858 raft_consensus.cc:493] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:22.387458 19808 leader_election.cc:290] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:21:22.418390 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 75843.4us
I20260709 12:21:22.424741 19829 raft_consensus.cc:515] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:22.425945 19841 raft_consensus.cc:515] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:22.399634 17833 raft_consensus.cc:2393] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 4: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:21:22.362520 19838 consensus_queue.cc:237] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:22.414292 19861 raft_consensus.cc:740] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 4 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:21:22.429584 19749 raft_consensus.cc:3060] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:22.427628 19757 raft_consensus.cc:697] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 3 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:21:22.450084 19829 leader_election.cc:290] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 2 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:22.451896 17652 leader_election.cc:400] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 2 pre-election: Vote denied by peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651) with higher term. Message: Invalid argument: T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for earlier term 2. Current term is 3.
I20260709 12:21:22.462966 17652 leader_election.cc:403] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:21:22.429729 19843 raft_consensus.cc:697] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 2 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
W20260709 12:21:22.456516 19859 log.cc:927] Time spent T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.088s	user 0.001s	sys 0.000s
I20260709 12:21:22.397874 19809 consensus_queue.cc:237] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:22.457156 19742 consensus_queue.cc:237] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:22.431944 19761 raft_consensus.cc:3060] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:22.433629 19858 raft_consensus.cc:3060] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:22.485962 19843 consensus_queue.cc:237] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:22.465127 19738 leader_election.cc:290] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 5 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:22.429476 19823 consensus_queue.cc:1059] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.172s
I20260709 12:21:22.538952 19757 consensus_queue.cc:237] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:22.593331 19842 raft_consensus.cc:515] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:22.541217 19738 consensus_queue.cc:1048] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.050s
I20260709 12:21:22.605118 19667 raft_consensus.cc:493] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:22.605759 19667 raft_consensus.cc:515] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:22.604252 19841 leader_election.cc:290] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 3 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:22.634212 19667 leader_election.cc:290] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:22.649555 19865 raft_consensus.cc:493] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:21:22.641728 19845 leader_election.cc:290] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:22.628978 19837 raft_consensus.cc:515] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:22.597163 19861 consensus_queue.cc:260] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:22.650663 19613 raft_consensus.cc:515] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:22.658622 19861 raft_consensus.cc:3060] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:22.661278 19594 raft_consensus.cc:515] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:22.661927 19705 raft_consensus.cc:515] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:22.664403 19594 leader_election.cc:290] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 2 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:22.665030 19837 leader_election.cc:290] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 3 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:22.668746 19714 raft_consensus.cc:515] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:22.669421 19839 raft_consensus.cc:3055] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:21:22.672999 19659 raft_consensus.cc:515] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:22.687713 19839 raft_consensus.cc:740] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
W20260709 12:21:22.681463 19857 log.cc:927] Time spent T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.336s	user 0.000s	sys 0.005s
I20260709 12:21:22.673030 19749 raft_consensus.cc:515] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:22.682533 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:22.671341 19840 raft_consensus.cc:515] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:22.689286 17761 raft_consensus.cc:1240] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 2. Current term is 3. Ops: []
I20260709 12:21:22.650267 19865 raft_consensus.cc:515] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:21:22.689378 19807 consensus_meta.cc:287] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.503s	user 0.012s	sys 0.000s
W20260709 12:21:22.694186 19694 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.557s	user 0.015s	sys 0.022s
I20260709 12:21:22.718863 17757 raft_consensus.cc:1240] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 2. Current term is 3. Ops: []
I20260709 12:21:22.698760 19705 leader_election.cc:290] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 4 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:22.688174 19779 raft_consensus.cc:515] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:22.689276 19761 raft_consensus.cc:515] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:22.736420 19714 leader_election.cc:290] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 3 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:22.736588 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:22.744307 19832 raft_consensus.cc:3055] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:21:22.744771 19832 raft_consensus.cc:740] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 2 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:21:22.693854 19839 consensus_queue.cc:260] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:22.702382 19807 raft_consensus.cc:515] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:22.715152 19835 raft_consensus.cc:515] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:21:22.919850 17758 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.518s	user 0.034s	sys 0.052s
W20260709 12:21:23.086485 17758 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.685s	user 0.038s	sys 0.060s
W20260709 12:21:23.086625 19826 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.817s	user 0.003s	sys 0.009s
W20260709 12:21:23.086978 19826 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.817s	user 0.003s	sys 0.009s
I20260709 12:21:23.087218 19826 raft_consensus.cc:697] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
W20260709 12:21:22.942224 19815 consensus_meta.cc:287] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.663s	user 0.010s	sys 0.002s
I20260709 12:21:23.088378 19815 raft_consensus.cc:515] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:22.796523 19796 leader_election.cc:290] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 4 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:22.996307 19799 raft_consensus.cc:493] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:21:23.090031 19799 raft_consensus.cc:515] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:23.090623 19815 leader_election.cc:290] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 3 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:23.088821 17755 raft_consensus.cc:2468] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 4.
W20260709 12:21:23.091436 18333 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.819s	user 0.000s	sys 0.012s
W20260709 12:21:23.091758 18333 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.820s	user 0.000s	sys 0.012s
W20260709 12:21:22.979537 19689 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.618s	user 0.006s	sys 0.037s
I20260709 12:21:23.092020 19779 leader_election.cc:290] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 3 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:21:23.092211 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 820040us
W20260709 12:21:23.093040 18336 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.791s	user 0.013s	sys 0.024s
W20260709 12:21:23.093546 18336 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.866s	user 0.013s	sys 0.024s
W20260709 12:21:23.069540 17654 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.605s	user 0.005s	sys 0.012s
W20260709 12:21:23.096693 17654 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.632s	user 0.005s	sys 0.013s
W20260709 12:21:22.864564 17802 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.542s	user 0.008s	sys 0.089s
W20260709 12:21:23.098569 17802 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.777s	user 0.018s	sys 0.098s
I20260709 12:21:23.099095 17728 leader_election.cc:304] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:21:23.100659 17652 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.637s	user 0.007s	sys 0.015s
W20260709 12:21:23.101020 17652 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.637s	user 0.007s	sys 0.015s
W20260709 12:21:23.101330 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 637580us
W20260709 12:21:23.092456 19689 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.743s	user 0.010s	sys 0.041s
W20260709 12:21:22.983271 18374 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.761s	user 0.015s	sys 0.019s
W20260709 12:21:23.110265 18374 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.888s	user 0.015s	sys 0.029s
W20260709 12:21:23.103804 17730 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:23.091161 17727 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.829s	user 0.013s	sys 0.000s
I20260709 12:21:23.004019 19757 raft_consensus.cc:493] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
W20260709 12:21:23.098920 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 776995us
W20260709 12:21:23.036140 19652 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.730s	user 0.003s	sys 0.007s
I20260709 12:21:23.108052 18333 leader_election.cc:304] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:23.122892 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:23.099989 19826 consensus_queue.cc:237] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:21:23.034787 19821 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.792s	user 0.002s	sys 0.014s
W20260709 12:21:23.085536 19836 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.602s	user 0.009s	sys 0.045s
I20260709 12:21:23.146131 17759 raft_consensus.cc:1240] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 3. Current term is 4. Ops: []
I20260709 12:21:23.131369 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:23.147609 17831 raft_consensus.cc:2468] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 2.
W20260709 12:21:23.138532 17727 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.863s	user 0.016s	sys 0.000s
I20260709 12:21:23.139489 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:23.140482 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:22.750012 19842 leader_election.cc:290] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 3 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:23.142486 17762 raft_consensus.cc:1240] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20260709 12:21:23.150808 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:22.790031 19659 leader_election.cc:290] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 3 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:21:22.708839 19694 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.568s	user 0.015s	sys 0.023s
I20260709 12:21:23.106060 19728 consensus_queue.cc:1059] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.006s
W20260709 12:21:23.149204 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 895878us
I20260709 12:21:23.149204 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:23.158113 17763 raft_consensus.cc:1240] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 1. Current term is 2. Ops: []
W20260709 12:21:22.954113 19848 consensus_meta.cc:287] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.542s	user 0.012s	sys 0.002s
W20260709 12:21:23.152092 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 44136us
I20260709 12:21:23.158774 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:21:23.161568 17729 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.881s	user 0.002s	sys 0.014s
I20260709 12:21:23.157673 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
W20260709 12:21:23.144419 19821 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.903s	user 0.006s	sys 0.014s
I20260709 12:21:23.123067 19808 raft_consensus.cc:2804] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260709 12:21:23.141532 19652 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.834s	user 0.006s	sys 0.009s
I20260709 12:21:22.782769 19865 leader_election.cc:290] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:23.107390 19829 raft_consensus.cc:493] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:23.106231 19867 consensus_queue.cc:1059] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.040s
I20260709 12:21:23.150331 17756 raft_consensus.cc:1275] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:21:22.790851 19749 leader_election.cc:290] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 2 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:23.163489 17727 leader_election.cc:304] T 7cb3768d215e409fb9f5baf18d124442 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:21:23.165688 17759 raft_consensus.cc:2393] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 3: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:21:23.154575 19812 raft_consensus.cc:493] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:23.159175 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:23.246508 17682 raft_consensus.cc:3060] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:23.167194 17760 raft_consensus.cc:2393] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 2: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:21:23.132889 19840 leader_election.cc:290] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 5 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:23.132315 19613 leader_election.cc:290] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 2 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:23.158607 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:21:23.170007 17729 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.890s	user 0.003s	sys 0.014s
W20260709 12:21:23.270298 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 996480us
I20260709 12:21:23.271217 17729 leader_election.cc:304] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
W20260709 12:21:23.161329 17803 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.839s	user 0.007s	sys 0.010s
W20260709 12:21:23.273996 17803 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.953s	user 0.007s	sys 0.010s
I20260709 12:21:22.743386 19825 raft_consensus.cc:515] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:23.275238 17764 raft_consensus.cc:2468] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 2.
I20260709 12:21:22.756423 17680 raft_consensus.cc:2468] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 3.
I20260709 12:21:23.141458 19806 raft_consensus.cc:493] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:23.112015 19594 consensus_queue.cc:1059] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.049s
I20260709 12:21:23.198695 17837 raft_consensus.cc:1240] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 4 LEADER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 3. Current term is 4. Ops: []
I20260709 12:21:23.184190 19848 raft_consensus.cc:515] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:21:23.158347 19836 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.829s	user 0.010s	sys 0.046s
I20260709 12:21:23.157292 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:21:23.233098 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 93145.9us
I20260709 12:21:23.157770 19879 raft_consensus.cc:493] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:23.143218 19757 raft_consensus.cc:515] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:23.189841 19808 raft_consensus.cc:493] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:23.324733 19808 raft_consensus.cc:3060] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:23.325305 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:23.325961 18361 raft_consensus.cc:3060] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:23.329501 19757 leader_election.cc:290] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:23.193831 19659 raft_consensus.cc:493] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:23.190197 19694 raft_consensus.cc:493] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:21:23.280987 19766 raft_consensus.cc:3055] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 2 LEADER]: Stepping down as leader of term 2
W20260709 12:21:23.285647 17655 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.710s	user 0.009s	sys 0.010s
I20260709 12:21:23.292101 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:23.295519 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:23.297852 17687 raft_consensus.cc:3060] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:23.299134 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:23.364257 18365 raft_consensus.cc:3060] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:23.297290 19799 leader_election.cc:290] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:23.307600 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:23.368131 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:23.308488 17683 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "22fa38faae58422e9cecc99973cab688"
dest_uuid: "0da169ff82674dc1947055ce43de68a3"
mode: GRACEFUL
new_leader_uuid: "e063541542a94969898868bf6eb5ce1b"
 from {username='slave'} at 127.0.0.1:54540
I20260709 12:21:23.369127 17683 raft_consensus.cc:606] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 2 LEADER]: Received request to transfer leadership to TS e063541542a94969898868bf6eb5ce1b
W20260709 12:21:23.310472 18334 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.003s	user 0.014s	sys 0.053s
W20260709 12:21:23.369709 18334 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.063s	user 0.014s	sys 0.054s
I20260709 12:21:23.369941 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:23.316268 19860 raft_consensus.cc:2749] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651) with higher term. Message: Invalid argument: T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for earlier term 2. Current term is 3.
I20260709 12:21:23.287739 19848 leader_election.cc:290] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 5 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:23.318773 19613 raft_consensus.cc:493] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:23.319327 19835 leader_election.cc:290] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 3 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:23.320762 17832 raft_consensus.cc:1240] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 4. Current term is 5. Ops: []
I20260709 12:21:23.306306 19617 raft_consensus.cc:3055] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:21:23.387742 19617 raft_consensus.cc:740] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:21:23.322501 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:23.323761 19807 leader_election.cc:290] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 4 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:21:23.280310 19809 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.620s	user 0.040s	sys 0.056s
I20260709 12:21:23.270977 19824 raft_consensus.cc:3060] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:23.302392 17583 catalog_manager.cc:5697] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 reported cstate change: term changed from 2 to 3, leader changed from 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) to 0da169ff82674dc1947055ce43de68a3 (127.16.29.65). New cstate: current_term: 3 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20260709 12:21:23.285434 19861 consensus_meta.cc:287] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.602s	user 0.008s	sys 0.004s
I20260709 12:21:23.323797 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:23.422204 18371 raft_consensus.cc:2393] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 2: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
W20260709 12:21:23.393692 19858 consensus_meta.cc:287] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.878s	user 0.012s	sys 0.000s
I20260709 12:21:23.340970 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:23.347841 17757 raft_consensus.cc:1240] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 3. Current term is 4. Ops: [3.6-3.6]
W20260709 12:21:23.286306 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 122894us
W20260709 12:21:23.412768 19809 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.758s	user 0.040s	sys 0.056s
I20260709 12:21:23.295120 19761 leader_election.cc:290] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 3 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:23.364954 18367 raft_consensus.cc:2393] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 3: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:21:23.306699 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:23.442551 19808 raft_consensus.cc:515] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:23.420482 19848 consensus_queue.cc:1059] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.024s
I20260709 12:21:23.444733 19848 raft_consensus.cc:3055] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:21:23.445240 19848 raft_consensus.cc:740] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:21:23.445698 19817 raft_consensus.cc:493] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:21:23.446413 19817 raft_consensus.cc:515] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:23.294665 19749 raft_consensus.cc:493] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:21:23.307283 17680 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37396 (request call id 1703) took 1181 ms. Trace:
W20260709 12:21:23.374949 17655 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.888s	user 0.016s	sys 0.010s
W20260709 12:21:23.456624 17655 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:23.457602 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:23.376868 17760 raft_consensus.cc:2393] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 3: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:21:23.279022 19878 raft_consensus.cc:2749] T 7cb3768d215e409fb9f5baf18d124442 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:21:23.276285 19812 raft_consensus.cc:515] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:23.269270 19829 raft_consensus.cc:515] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:23.345768 19837 raft_consensus.cc:493] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:21:23.350349 19766 raft_consensus.cc:740] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:21:23.281240 19806 raft_consensus.cc:515] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:23.391578 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:23.395198 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:23.474364 17681 raft_consensus.cc:3060] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:23.395198 17682 raft_consensus.cc:2468] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 4.
I20260709 12:21:23.474653 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:23.475762 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:23.398600 18365 raft_consensus.cc:2468] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 4.
I20260709 12:21:23.372020 19796 raft_consensus.cc:493] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:23.401350 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 232630us
I20260709 12:21:23.349934 19659 raft_consensus.cc:515] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:23.403420 19689 consensus_queue.cc:1059] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:21:23.483737 17682 raft_consensus.cc:1275] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260709 12:21:23.482713 17729 leader_election.cc:304] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:21:23.404624 19845 raft_consensus.cc:493] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:21:23.403457 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:21:23.407419 19886 log.cc:927] Time spent T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.151s	user 0.000s	sys 0.000s
I20260709 12:21:23.409482 17762 raft_consensus.cc:1275] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:21:23.411994 19823 consensus_queue.cc:1059] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:21:23.280783 19737 raft_consensus.cc:3060] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:23.385732 19842 raft_consensus.cc:493] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
W20260709 12:21:23.287734 17755 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38776 (request call id 1628) took 1689 ms (client timeout 2183 ms). Trace:
W20260709 12:21:23.498876 17755 rpcz_store.cc:269] 0709 12:21:21.597951 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:21.687279 (+ 89328us) service_pool.cc:224] Handling call
0709 12:21:23.287529 (+1600250us) spinlock_profiling.cc:243] Waited 621 ms on lock 0x7b5400216e54. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:21:23.287671 (+   142us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1428734848}
I20260709 12:21:23.499650 19823 raft_consensus.cc:3055] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:21:23.500048 19823 raft_consensus.cc:740] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:21:23.500227 19829 leader_election.cc:290] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:23.492242 19796 raft_consensus.cc:515] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:23.500811 19766 consensus_queue.cc:260] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:23.501941 19766 raft_consensus.cc:3060] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:23.455662 17680 rpcz_store.cc:276] 0709 12:21:22.114549 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:22.143236 (+ 28687us) service_pool.cc:224] Handling call
0709 12:21:23.296261 (+1153025us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:23.432374 17830 raft_consensus.cc:1240] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 1. Current term is 2. Ops: []
I20260709 12:21:23.432719 17727 leader_election.cc:304] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:21:23.434244 19690 raft_consensus.cc:3055] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:21:23.435006 19821 consensus_queue.cc:1059] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:21:23.436878 19825 leader_election.cc:290] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 3 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:21:23.447903 19764 consensus_meta.cc:287] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 1.012s	user 0.008s	sys 0.005s
I20260709 12:21:23.400080 19804 raft_consensus.cc:493] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:23.390214 19613 raft_consensus.cc:515] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:23.364463 18363 raft_consensus.cc:2393] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 5: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:21:23.452812 17652 leader_election.cc:304] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:21:23.459318 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:23.460170 17689 raft_consensus.cc:1240] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 2. Current term is 3. Ops: []
W20260709 12:21:23.464613 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 173276us
I20260709 12:21:23.464802 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:23.521387 17688 raft_consensus.cc:3060] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:21:23.521566 17803 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:23.484059 19832 consensus_queue.cc:260] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:23.470556 17802 leader_election.cc:304] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:23.482502 18333 leader_election.cc:304] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
W20260709 12:21:23.528990 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 46562.6us
I20260709 12:21:23.529668 18333 leader_election.cc:304] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:21:23.530704 18333 leader_election.cc:304] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:21:23.483346 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:23.490617 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:23.390081 19617 consensus_queue.cc:260] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:21:23.480881 19875 log.cc:927] Time spent T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.175s	user 0.000s	sys 0.001s
I20260709 12:21:23.495088 19894 consensus_queue.cc:1048] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 12:21:23.429543 19858 raft_consensus.cc:515] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:23.556144 19806 leader_election.cc:290] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:23.496276 16501 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 1
I20260709 12:21:23.493108 19845 raft_consensus.cc:515] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:23.498068 19842 raft_consensus.cc:515] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:23.362933 19694 raft_consensus.cc:515] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:23.330590 19879 raft_consensus.cc:515] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:23.476540 19837 raft_consensus.cc:515] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:23.506327 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:23.563898 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:23.507280 17682 raft_consensus.cc:1240] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 2. Current term is 3. Ops: []
I20260709 12:21:23.566248 19842 leader_election.cc:290] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:23.462314 19749 raft_consensus.cc:515] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:23.509546 19841 raft_consensus.cc:493] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:21:23.510434 18362 raft_consensus.cc:1275] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260709 12:21:23.491613 19807 raft_consensus.cc:493] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:23.492748 19826 raft_consensus.cc:2804] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Leader election won for term 4
W20260709 12:21:23.516261 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 83626.8us
I20260709 12:21:23.516492 19742 raft_consensus.cc:2749] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:21:23.469832 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:23.581460 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:21:23.522904 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 96659.2us
I20260709 12:21:23.527057 17681 raft_consensus.cc:2468] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 3.
I20260709 12:21:23.531697 18333 leader_election.cc:304] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
W20260709 12:21:23.536397 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 65917us
I20260709 12:21:23.500826 19823 consensus_queue.cc:260] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:23.588682 16501 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:21:23.589506 16501 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:21:23.538985 17688 raft_consensus.cc:2468] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 3.
I20260709 12:21:23.539546 19761 raft_consensus.cc:3060] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:23.539961 19812 leader_election.cc:290] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:23.541770 19829 raft_consensus.cc:2804] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:21:23.542902 19868 raft_consensus.cc:2749] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:21:23.512759 19690 raft_consensus.cc:740] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:21:23.547173 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:23.548372 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:23.570530 19753 raft_consensus.cc:493] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:21:23.548967 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:23.549177 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:23.549830 17728 leader_election.cc:304] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:21:23.522751 19764 raft_consensus.cc:515] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:23.551364 19883 raft_consensus.cc:2749] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:21:23.551972 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:23.552542 19689 raft_consensus.cc:493] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:23.548949 19840 raft_consensus.cc:2749] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:21:23.556816 19839 raft_consensus.cc:3060] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:23.698681 19690 consensus_queue.cc:260] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:23.700609 19690 raft_consensus.cc:3060] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:23.700804 19689 raft_consensus.cc:515] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:23.703424 18374 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 811) took 1487 ms. Trace:
I20260709 12:21:23.703967 17759 raft_consensus.cc:1240] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 1. Current term is 2. Ops: []
I20260709 12:21:23.560868 19858 leader_election.cc:290] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 6 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:23.561435 19838 raft_consensus.cc:493] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:23.568493 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:23.568676 18371 raft_consensus.cc:1240] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 2. Current term is 3. Ops: []
I20260709 12:21:23.569937 17762 raft_consensus.cc:1240] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260709 12:21:23.570720 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:21:23.593761 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 62130.7us
I20260709 12:21:23.581044 19613 leader_election.cc:290] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:23.596875 17652 leader_election.cc:304] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:21:23.598112 17802 leader_election.cc:304] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:23.598446 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:23.598155 18334 leader_election.cc:304] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
W20260709 12:21:23.720566 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 122479us
I20260709 12:21:23.721216 18334 leader_election.cc:304] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:21:23.693203 19845 leader_election.cc:290] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:23.589102 17583 catalog_manager.cc:5697] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 reported cstate change: term changed from 3 to 4, leader changed from e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66) to e063541542a94969898868bf6eb5ce1b (127.16.29.67). New cstate: current_term: 4 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } }
I20260709 12:21:23.608783 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:23.725414 18367 raft_consensus.cc:1275] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260709 12:21:23.550565 19808 leader_election.cc:290] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 3 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:23.594461 19617 raft_consensus.cc:3060] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:23.603932 18361 raft_consensus.cc:2468] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 4.
I20260709 12:21:23.618629 19827 raft_consensus.cc:3055] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:21:23.619737 19824 raft_consensus.cc:2749] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260709 12:21:23.620239 17654 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:23.687515 17689 raft_consensus.cc:2393] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 3: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:21:23.734988 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:23.691249 18365 raft_consensus.cc:2393] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 3: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
W20260709 12:21:23.694074 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 144304us
I20260709 12:21:23.604219 19894 consensus_queue.cc:1059] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.095s
I20260709 12:21:23.737154 17728 leader_election.cc:304] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:21:23.698760 19764 leader_election.cc:290] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 3 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:23.696260 19673 raft_consensus.cc:2804] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:21:23.612465 18366 raft_consensus.cc:1275] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260709 12:21:23.703868 18374 rpcz_store.cc:276] 0709 12:21:22.215919 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:22.216311 (+   392us) service_pool.cc:224] Handling call
0709 12:21:22.222157 (+  5846us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:23.125525 (+903368us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:23.125555 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:23.146538 (+ 20983us) log.cc:844] Serialized 26 byte log entry
0709 12:21:23.509331 (+362793us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:21:23.509383 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:23.509496 (+   113us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:23.703218 (+193722us) raft_consensus.cc:1685] finished
0709 12:21:23.703238 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:23.703366 (+   128us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":33912,"raft.run_cpu_time_us":153,"raft.run_wall_time_us":5628,"thread_start_us":1213901,"threads_started":2,"wal-append.queue_time_us":538823}
I20260709 12:21:23.614689 19829 raft_consensus.cc:493] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:21:23.600586 19825 raft_consensus.cc:493] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:21:23.612816 17755 raft_consensus.cc:1240] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 4. Current term is 5. Ops: []
I20260709 12:21:23.716465 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:23.519565 19804 raft_consensus.cc:515] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:23.748392 19880 consensus_queue.cc:1048] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.005s
I20260709 12:21:23.749117 17683 raft_consensus.cc:2393] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 2: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:21:23.608450 19806 consensus_queue.cc:1059] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 4.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.114s
I20260709 12:21:23.687659 19742 raft_consensus.cc:493] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:23.721625 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 124875us
I20260709 12:21:23.731238 19827 raft_consensus.cc:740] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 3 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:21:23.753458 17652 leader_election.cc:304] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:21:23.754806 19659 leader_election.cc:290] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:21:23.654279 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:23.599208 19877 raft_consensus.cc:3060] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:23.723232 18363 raft_consensus.cc:2468] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 2.
I20260709 12:21:23.756248 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:23.594880 19841 raft_consensus.cc:515] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:23.725308 18362 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 2.
I20260709 12:21:23.563743 19837 leader_election.cc:290] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:23.725571 17583 catalog_manager.cc:5697] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 reported cstate change: term changed from 2 to 3, leader changed from 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) to e063541542a94969898868bf6eb5ce1b (127.16.29.67). New cstate: current_term: 3 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } }
I20260709 12:21:23.611016 19872 consensus_queue.cc:1048] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.136s
I20260709 12:21:23.605180 19826 raft_consensus.cc:697] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 4 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:21:23.696345 19652 raft_consensus.cc:493] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:23.729854 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 131794us
I20260709 12:21:23.738281 18364 raft_consensus.cc:2393] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 3: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:21:23.697575 19753 raft_consensus.cc:515] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:23.742301 18368 raft_consensus.cc:1240] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 2. Current term is 3. Ops: []
I20260709 12:21:23.752144 19673 raft_consensus.cc:697] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 4 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
W20260709 12:21:23.744408 17801 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:23.755647 19825 raft_consensus.cc:515] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:23.730343 19749 leader_election.cc:290] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:23.746011 18371 raft_consensus.cc:3060] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:23.731211 19836 raft_consensus.cc:493] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:23.732031 19728 raft_consensus.cc:2749] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:21:23.754858 19893 raft_consensus.cc:3060] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:23.722357 17586 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35962 (request call id 354) took 3343 ms. Trace:
I20260709 12:21:23.756994 18367 raft_consensus.cc:2393] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 3: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:21:23.723338 19838 raft_consensus.cc:515] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:23.743275 19874 raft_consensus.cc:2804] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:21:23.752063 19742 raft_consensus.cc:515] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:23.768776 17652 leader_election.cc:304] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:23.766306 18333 leader_election.cc:304] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:23.749933 19829 raft_consensus.cc:3060] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:23.771512 17802 leader_election.cc:304] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:23.752349 19808 raft_consensus.cc:2804] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:21:23.594694 19807 raft_consensus.cc:515] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:23.729411 19779 raft_consensus.cc:2804] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:21:23.764408 19883 raft_consensus.cc:2804] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:21:23.767710 19694 leader_election.cc:290] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:23.776243 19652 raft_consensus.cc:515] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:23.897073 17652 leader_election.cc:304] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:21:23.764252 17571 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet d522f70beb1f417cb98940d90b594823 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica e2909cff1da44ad387f96a8ae3aa5afb is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
W20260709 12:21:23.898983 17571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 145850us
W20260709 12:21:23.889784 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 123525us
W20260709 12:21:23.890969 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 119527us
I20260709 12:21:23.774595 19806 consensus_queue.cc:1059] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.019s
I20260709 12:21:23.782688 19796 leader_election.cc:290] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:23.901135 17652 leader_election.cc:304] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:21:23.902735 19869 raft_consensus.cc:2749] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:21:23.905527 19862 raft_consensus.cc:3055] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:21:23.905658 19753 leader_election.cc:290] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:23.900085 19757 consensus_queue.cc:1059] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.006s
I20260709 12:21:23.906441 19895 consensus_queue.cc:1059] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.139s
I20260709 12:21:23.907434 19742 leader_election.cc:290] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:23.907621 19855 raft_consensus.cc:493] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:23.890992 19799 raft_consensus.cc:2749] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:21:23.890353 19749 raft_consensus.cc:2749] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:21:23.888244 19837 raft_consensus.cc:493] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:21:23.786620 19836 raft_consensus.cc:515] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:23.954037 19804 leader_election.cc:290] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:23.970589 19890 raft_consensus.cc:493] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:21:23.971380 19890 raft_consensus.cc:515] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:23.789371 17586 rpcz_store.cc:276] 0709 12:21:20.378727 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:20.380993 (+  2266us) service_pool.cc:224] Handling call
0709 12:21:23.722293 (+3341300us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:21:21.704717 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:21:21.733404 (+ 28687us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:21:21.733457 (+    53us) write_op.cc:435] Acquired schema lock
0709 12:21:21.733478 (+    21us) tablet.cc:662] Decoding operations
0709 12:21:21.734048 (+   570us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:21:21.734116 (+    68us) write_op.cc:641] Partition lock acquired for write op
0709 12:21:21.734148 (+    32us) tablet.cc:685] Acquiring locks for 2 operations
0709 12:21:21.734440 (+   292us) tablet.cc:701] Row locks acquired
0709 12:21:21.734457 (+    17us) write_op.cc:260] PREPARE: finished
0709 12:21:21.734621 (+   164us) write_op.cc:270] Start()
0709 12:21:21.734755 (+   134us) write_op.cc:276] Timestamp: P: 1783599681734567 usec, L: 0
0709 12:21:21.734779 (+    24us) op_driver.cc:348] REPLICATION: starting
0709 12:21:21.781328 (+ 46549us) log.cc:844] Serialized 1003 byte log entry
0709 12:21:22.300927 (+519599us) op_driver.cc:464] REPLICATION: finished
0709 12:21:22.933651 (+632724us) spinlock_profiling.cc:243] Waited 2.64 ms on lock 0x7b18003500e0. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b475333f3 00007f7b475357c1 00007f7b47535779 00007f7b47535709 00007f7b475356d1 00007f7b475349cd 0000000000388634 0000000000388479 00007f7b4751683a
0709 12:21:22.937366 (+  3715us) write_op.cc:301] APPLY: starting
0709 12:21:22.943855 (+  6489us) tablet.cc:1367] starting BulkCheckPresence
0709 12:21:22.971038 (+ 27183us) tablet.cc:1370] finished BulkCheckPresence
0709 12:21:23.065787 (+ 94749us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:21:23.085090 (+ 19303us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:21:23.085346 (+   256us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0709 12:21:23.085414 (+    68us) write_op.cc:312] APPLY: finished
0709 12:21:23.085677 (+   263us) log.cc:844] Serialized 29 byte log entry
0709 12:21:23.085907 (+   230us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:21:23.096453 (+ 10546us) write_op.cc:454] Released schema lock
0709 12:21:23.102554 (+  6101us) write_op.cc:341] FINISH: Updating metrics
0709 12:21:23.152332 (+ 49778us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:21:23.155214 (+  2882us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":37738,"reported_tablets":3,"child_traces":[["op",{"apply.queue_time_us":106402,"apply.run_cpu_time_us":2849,"apply.run_wall_time_us":199968,"num_ops":2,"prepare.queue_time_us":141427,"prepare.run_cpu_time_us":25671,"prepare.run_wall_time_us":381917,"raft.queue_time_us":184110,"raft.run_cpu_time_us":19859,"raft.run_wall_time_us":137232,"replication_time_us":542426,"spinlock_wait_cycles":6060928,"thread_start_us":551352,"threads_started":4,"wal-append.queue_time_us":273883}]]}
I20260709 12:21:23.894246 19815 raft_consensus.cc:493] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:23.894073 19874 raft_consensus.cc:697] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 3 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:21:23.897848 19883 raft_consensus.cc:493] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:21:23.898972 19808 raft_consensus.cc:697] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 3 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:21:23.904455 19738 raft_consensus.cc:2749] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:21:23.987639 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:23.989400 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:23.989909 17686 raft_consensus.cc:1240] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 4. Current term is 5. Ops: []
I20260709 12:21:23.918784 17758 raft_consensus.cc:1275] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 7 index: 9. (index mismatch)
I20260709 12:21:23.991211 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:23.992444 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:23.913436 19862 raft_consensus.cc:740] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 3 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:21:23.920342 17652 leader_election.cc:304] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:21:23.947196 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:23.994408 17761 raft_consensus.cc:2393] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 3: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:21:23.903817 19779 raft_consensus.cc:697] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 3 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:21:23.911350 19737 raft_consensus.cc:2749] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:21:23.949390 18371 raft_consensus.cc:2468] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 6.
I20260709 12:21:23.950297 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:23.915786 19694 consensus_queue.cc:1059] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.037s
I20260709 12:21:23.952474 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:23.912524 19835 raft_consensus.cc:493] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:23.952796 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:23.910475 19603 raft_consensus.cc:493] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:21:23.953222 19838 leader_election.cc:290] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:23.912818 19673 consensus_queue.cc:237] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:23.918045 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:23.959846 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:23.917500 19826 consensus_queue.cc:237] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:23.928042 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:23.975608 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:23.984889 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:23.985837 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:23.986877 19652 leader_election.cc:290] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:23.932041 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:23.996781 18362 raft_consensus.cc:1240] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 3. Current term is 4. Ops: []
I20260709 12:21:23.997045 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:23.952410 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:23.997542 18364 raft_consensus.cc:1240] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 2. Current term is 3. Ops: []
I20260709 12:21:23.971639 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:23.781333 19827 consensus_queue.cc:260] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:23.971092 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:23.959331 19855 raft_consensus.cc:515] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:23.971449 19837 raft_consensus.cc:515] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:23.974790 19899 consensus_queue.cc:1059] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.309s
I20260709 12:21:24.003849 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:24.077270 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:24.022866 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:24.197809 17763 raft_consensus.cc:3060] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:24.024098 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:24.207597 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:24.178392 17759 raft_consensus.cc:2393] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 2: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:21:23.973730 19879 leader_election.cc:290] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:24.210822 18363 raft_consensus.cc:1240] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 1. Current term is 2. Ops: []
I20260709 12:21:24.072610 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
W20260709 12:21:24.172886 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 252614us
I20260709 12:21:24.212356 17831 raft_consensus.cc:1240] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 3. Current term is 4. Ops: []
I20260709 12:21:24.087311 17760 raft_consensus.cc:1240] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 3. Current term is 4. Ops: []
I20260709 12:21:24.217113 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:24.211113 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:24.207103 17834 raft_consensus.cc:2393] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 3: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:21:24.221668 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:24.222041 17834 raft_consensus.cc:3060] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:24.223090 17835 raft_consensus.cc:2393] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 2: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:21:24.209692 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:24.205825 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:24.224766 17761 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 2.
I20260709 12:21:24.225045 18363 raft_consensus.cc:1240] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260709 12:21:24.082908 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:24.208334 17756 raft_consensus.cc:2468] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 6.
I20260709 12:21:24.225621 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:24.066991 19885 raft_consensus.cc:2804] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:21:24.209148 17758 raft_consensus.cc:2393] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 5: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:21:24.212193 17838 raft_consensus.cc:2393] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 3: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:21:24.226385 17836 raft_consensus.cc:1275] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:21:24.231487 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:23.976253 19890 leader_election.cc:290] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:21:24.020511 19906 log.cc:927] Time spent T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.153s	user 0.001s	sys 0.000s
I20260709 12:21:23.974707 19882 consensus_queue.cc:1048] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.320s
I20260709 12:21:24.229521 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:24.072234 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:24.178714 19825 leader_election.cc:290] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:24.079802 19761 raft_consensus.cc:2749] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260709 12:21:24.020812 19904 log.cc:927] Time spent T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.101s	user 0.000s	sys 0.001s
I20260709 12:21:24.074472 19893 raft_consensus.cc:2749] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:21:24.228430 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:24.232048 17838 raft_consensus.cc:2468] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 3.
I20260709 12:21:23.985229 19815 raft_consensus.cc:515] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:21:24.214241 19839 consensus_meta.cc:287] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.509s	user 0.007s	sys 0.003s
I20260709 12:21:24.229647 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:24.252830 17755 raft_consensus.cc:2393] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 3: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:21:24.251082 18363 raft_consensus.cc:1240] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 3. Current term is 4. Ops: []
I20260709 12:21:24.179057 19737 raft_consensus.cc:493] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:23.990491 19865 raft_consensus.cc:2804] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:21:24.157732 19705 raft_consensus.cc:3055] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:21:24.163327 19836 leader_election.cc:290] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:24.252528 17762 raft_consensus.cc:1275] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:21:24.233364 19779 consensus_queue.cc:237] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:24.020761 19874 consensus_queue.cc:237] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:24.250129 19829 raft_consensus.cc:515] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:24.210470 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:24.211344 17832 raft_consensus.cc:2468] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 2.
I20260709 12:21:23.989612 19883 raft_consensus.cc:3060] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:24.245788 19804 raft_consensus.cc:493] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:21:24.163587 19808 consensus_queue.cc:237] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:24.070405 19824 raft_consensus.cc:493] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:24.211125 19603 raft_consensus.cc:515] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:24.264700 17831 raft_consensus.cc:2468] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 2.
I20260709 12:21:24.258916 17764 raft_consensus.cc:1240] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 2. Current term is 3. Ops: []
I20260709 12:21:24.281908 17829 raft_consensus.cc:2393] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 3: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:21:24.285717 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:24.760887 17830 raft_consensus.cc:3060] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:24.343000 17763 raft_consensus.cc:2468] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 3.
W20260709 12:21:24.682361 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 344004us
W20260709 12:21:24.294154 17803 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:24.760946 17831 raft_consensus.cc:1275] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
W20260709 12:21:24.764760 17803 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:24.765473 17728 leader_election.cc:304] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:21:24.285528 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:24.325624 17834 raft_consensus.cc:2468] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 3.
I20260709 12:21:24.555111 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:24.425985 18363 raft_consensus.cc:1240] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 4. Current term is 5. Ops: []
I20260709 12:21:24.426782 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:24.775323 17838 raft_consensus.cc:2393] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 2: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:21:24.775238 17836 raft_consensus.cc:2468] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 2.
I20260709 12:21:24.777980 16501 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:21:24.295989 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:24.301195 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:21:24.781003 17757 consensus_meta.cc:287] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.574s	user 0.010s	sys 0.003s
I20260709 12:21:24.782195 17757 raft_consensus.cc:2468] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 2.
I20260709 12:21:24.782682 17687 raft_consensus.cc:1240] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 3 LEADER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 2. Current term is 3. Ops: []
I20260709 12:21:24.273036 17688 raft_consensus.cc:1240] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 3. Current term is 4. Ops: []
I20260709 12:21:24.792580 17829 raft_consensus.cc:1275] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:21:24.303757 17729 leader_election.cc:304] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
W20260709 12:21:24.284330 19617 consensus_meta.cc:287] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.553s	user 0.010s	sys 0.000s
I20260709 12:21:24.386826 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:21:24.804172 17803 leader_election.cc:336] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Timed out: RequestConsensusVote RPC to 127.16.29.68:42577 timed out after 1.742s (SENT)
I20260709 12:21:24.804735 19603 leader_election.cc:290] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:24.804979 17803 leader_election.cc:304] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:24.805593 19603 raft_consensus.cc:493] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:24.806656 17684 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37402 (request call id 2129) took 1347 ms. Trace:
I20260709 12:21:24.794266 17681 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44898 (request call id 472) took 1196 ms. Trace:
I20260709 12:21:24.357555 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:24.806818 19603 raft_consensus.cc:515] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:24.557098 18368 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 817) took 1260 ms. Trace:
I20260709 12:21:24.294353 19893 consensus_queue.cc:1048] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.521s
I20260709 12:21:24.815999 19705 raft_consensus.cc:740] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 2 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:21:24.436059 19897 raft_consensus.cc:3055] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:21:24.817955 19897 raft_consensus.cc:740] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 3 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:21:24.818943 19728 raft_consensus.cc:493] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:24.818338 19705 consensus_queue.cc:260] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:24.814391 18368 rpcz_store.cc:276] 0709 12:21:23.296633 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:23.296763 (+   130us) service_pool.cc:224] Handling call
0709 12:21:23.297208 (+   445us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:23.773476 (+476268us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:24.556797 (+783321us) spinlock_profiling.cc:243] Waited 445 ms on lock 0x7b5400724054. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:21:24.557017 (+   220us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1023060224}
I20260709 12:21:24.807309 17684 rpcz_store.cc:276] 0709 12:21:23.459111 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:23.459306 (+   195us) service_pool.cc:224] Handling call
0709 12:21:23.459571 (+   265us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:24.804591 (+1345020us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:24.804650 (+    59us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:24.804724 (+    74us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:21:24.804810 (+    86us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:24.806059 (+  1249us) spinlock_profiling.cc:243] Waited 1.34 s on lock 0x7b54003016d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:21:24.806114 (+    55us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:24.806428 (+   314us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3091765120}
I20260709 12:21:24.832109 19705 raft_consensus.cc:3060] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:24.832337 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:24.831954 19804 raft_consensus.cc:515] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:24.833415 18368 raft_consensus.cc:2468] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 6.
I20260709 12:21:24.835641 19804 leader_election.cc:290] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:24.835295 19737 raft_consensus.cc:515] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:24.837334 19862 consensus_queue.cc:260] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:24.840770 19865 raft_consensus.cc:493] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:21:24.841387 19862 raft_consensus.cc:3060] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:24.841622 19799 raft_consensus.cc:493] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:24.842459 19799 raft_consensus.cc:515] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:24.844743 19824 raft_consensus.cc:515] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:24.846912 19869 raft_consensus.cc:2749] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:21:24.380820 19817 leader_election.cc:290] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:24.841401 19865 raft_consensus.cc:3060] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:24.848816 19817 raft_consensus.cc:2804] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:21:24.849308 19817 raft_consensus.cc:493] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:24.849725 19817 raft_consensus.cc:3060] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:24.257524 19885 raft_consensus.cc:697] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 4 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:21:24.345896 19911 consensus_queue.cc:1059] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.511s
I20260709 12:21:24.857858 19911 raft_consensus.cc:3055] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:21:24.858385 19911 raft_consensus.cc:740] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:21:24.819679 19728 raft_consensus.cc:515] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:24.837605 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:24.385135 19879 raft_consensus.cc:3055] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:21:24.287164 19841 leader_election.cc:290] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:24.256174 19823 raft_consensus.cc:3060] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:24.938634 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:21:24.376120 19877 consensus_meta.cc:287] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.617s	user 0.007s	sys 0.005s
I20260709 12:21:24.866218 19832 raft_consensus.cc:3060] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:24.859689 19838 raft_consensus.cc:493] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:24.862308 19829 leader_election.cc:290] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 3 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:24.296784 19845 raft_consensus.cc:3055] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:21:24.374205 19753 raft_consensus.cc:3055] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:21:24.807749 17681 rpcz_store.cc:276] 0709 12:21:23.597771 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:23.597901 (+   130us) service_pool.cc:224] Handling call
0709 12:21:23.598219 (+   318us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:23.598423 (+   204us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:23.598450 (+    27us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:23.598904 (+   454us) log.cc:844] Serialized 26 byte log entry
0709 12:21:23.721266 (+122362us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:21:23.721300 (+    34us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:23.721368 (+    68us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:24.793517 (+1072149us) raft_consensus.cc:1685] finished
0709 12:21:24.793594 (+    77us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:24.793962 (+   368us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":109042,"threads_started":1,"wal-append.queue_time_us":150515}
I20260709 12:21:24.855084 19885 consensus_queue.cc:237] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:24.373435 19757 raft_consensus.cc:3055] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:21:24.388422 19827 raft_consensus.cc:3060] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:24.865780 19890 raft_consensus.cc:2749] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:21:24.958667 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:24.959367 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:24.963955 19869 raft_consensus.cc:493] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:24.964582 19867 raft_consensus.cc:493] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:24.965109 19807 leader_election.cc:290] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:24.965237 17829 raft_consensus.cc:1275] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260709 12:21:24.564641 19689 leader_election.cc:290] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:24.966816 19603 leader_election.cc:290] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:24.556069 19764 raft_consensus.cc:3055] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:21:24.372794 19837 leader_election.cc:290] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:24.969084 18371 raft_consensus.cc:2393] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 3: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:21:24.978664 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:24.382084 19848 consensus_queue.cc:260] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:24.226838 19835 raft_consensus.cc:515] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:24.820686 19897 consensus_queue.cc:260] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:25.009850 19897 raft_consensus.cc:3060] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:24.984104 19824 leader_election.cc:290] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:24.984398 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:24.990638 19893 raft_consensus.cc:493] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:21:24.994344 19617 raft_consensus.cc:493] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:21:24.997529 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:24.997872 17838 raft_consensus.cc:1240] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 4. Current term is 5. Ops: []
I20260709 12:21:24.997751 18368 raft_consensus.cc:2468] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 4.
I20260709 12:21:24.966657 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:24.859627 19911 consensus_queue.cc:260] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:24.983590 19873 raft_consensus.cc:493] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:25.019555 19893 raft_consensus.cc:515] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:25.019726 19873 raft_consensus.cc:515] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:25.020995 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:25.008738 19848 raft_consensus.cc:3060] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:25.024153 19652 raft_consensus.cc:493] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
W20260709 12:21:25.025432 19914 log.cc:927] Time spent T 00000000000000000000000000000000 P 2193af31cb894ec9b3d65d939a2a6851: Append to log took a long time: real 0.058s	user 0.002s	sys 0.000s
I20260709 12:21:25.025103 19652 raft_consensus.cc:515] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:24.291098 19855 leader_election.cc:290] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:25.016893 19757 raft_consensus.cc:740] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:21:25.021914 19895 raft_consensus.cc:493] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:21:25.024366 17835 raft_consensus.cc:3060] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:24.998227 19869 raft_consensus.cc:515] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:25.032200 19764 raft_consensus.cc:740] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:21:25.032397 19867 raft_consensus.cc:515] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:25.032989 17837 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 2.
I20260709 12:21:25.024242 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:25.028010 19911 raft_consensus.cc:3060] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:25.041662 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:21:25.066316 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 762538us
I20260709 12:21:25.027729 19652 leader_election.cc:290] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:25.022569 19737 leader_election.cc:290] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:25.052058 19838 raft_consensus.cc:515] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:25.081130 19757 consensus_queue.cc:260] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:25.082365 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:25.082553 19617 raft_consensus.cc:515] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:25.083469 19838 leader_election.cc:290] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:25.085130 19753 raft_consensus.cc:740] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:21:25.086301 19753 consensus_queue.cc:260] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:25.087543 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:25.088912 19835 leader_election.cc:290] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:21:25.090396 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 77030us
I20260709 12:21:25.033370 19603 consensus_queue.cc:1059] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.058s
I20260709 12:21:25.099597 19815 leader_election.cc:290] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:25.092116 18365 raft_consensus.cc:3060] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:25.100692 19869 leader_election.cc:290] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:25.101280 19728 leader_election.cc:290] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:25.101344 19845 raft_consensus.cc:740] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:21:25.033672 19890 raft_consensus.cc:493] T 7cb3768d215e409fb9f5baf18d124442 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:21:25.102456 19890 raft_consensus.cc:515] T 7cb3768d215e409fb9f5baf18d124442 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:25.105368 19867 leader_election.cc:290] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:25.105751 19890 leader_election.cc:290] T 7cb3768d215e409fb9f5baf18d124442 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:25.093966 19893 leader_election.cc:290] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:25.096491 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:25.097231 19799 leader_election.cc:290] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:25.134357 17836 raft_consensus.cc:1275] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260709 12:21:25.097440 18333 leader_election.cc:304] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:21:25.136732 18333 leader_election.cc:304] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:21:25.137928 17836 raft_consensus.cc:1240] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 3. Current term is 4. Ops: []
I20260709 12:21:25.138496 19845 consensus_queue.cc:260] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:25.139194 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
W20260709 12:21:25.097800 17652 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:25.098153 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:25.140225 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:25.140625 18366 raft_consensus.cc:2468] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 3.
I20260709 12:21:25.141631 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:25.141665 17652 leader_election.cc:304] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:21:25.098527 19895 raft_consensus.cc:515] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:25.142627 17838 raft_consensus.cc:2393] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 4: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:21:25.144302 19893 consensus_queue.cc:1048] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:21:25.145056 19766 raft_consensus.cc:2804] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:21:25.145364 19895 leader_election.cc:290] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:25.145493 19766 raft_consensus.cc:493] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:21:25.146027 19766 raft_consensus.cc:3060] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:25.146278 19855 raft_consensus.cc:2749] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:21:25.147495 19874 consensus_queue.cc:1048] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:21:25.146027 19829 raft_consensus.cc:2749] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:21:25.099261 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:25.148293 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:25.148653 18368 raft_consensus.cc:2468] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 3.
I20260709 12:21:25.092485 19877 raft_consensus.cc:2749] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:21:25.156503 19757 raft_consensus.cc:3060] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:25.119266 19673 raft_consensus.cc:493] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:21:25.160701 19673 raft_consensus.cc:515] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:25.163518 19673 leader_election.cc:290] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:25.108047 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:25.170524 18371 raft_consensus.cc:2410] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for term 4 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260709 12:21:25.039636 19879 raft_consensus.cc:740] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
W20260709 12:21:25.098101 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 60033.8us
W20260709 12:21:25.118059 17654 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:25.117679 19817 raft_consensus.cc:515] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:25.091504 17727 leader_election.cc:304] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:21:25.087668 19753 raft_consensus.cc:3060] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:25.164625 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:25.107780 19617 leader_election.cc:290] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:25.143412 17652 leader_election.cc:304] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:21:25.143477 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:25.074986 19873 leader_election.cc:290] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:25.179014 18374 raft_consensus.cc:1275] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20260709 12:21:25.179908 18367 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59902 (request call id 687) took 1014 ms. Trace:
I20260709 12:21:25.180226 18365 raft_consensus.cc:2468] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 4.
I20260709 12:21:25.180428 18367 rpcz_store.cc:276] 0709 12:21:23.982305 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:23.982461 (+   156us) service_pool.cc:224] Handling call
0709 12:21:24.357353 (+374892us) spinlock_profiling.cc:243] Waited 31.9 ms on lock 0x7b5800102398. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b474fbb71 00007f7b474fba22 00007f7b504148d9 00007f7b4f56eee0 00007f7b4f6380ab 00007f7b4f638131 00007f7b4f5e60a0 00007f7b4f5e5984 00007f7b48b606d4 00007f7b48b60662
0709 12:21:24.989907 (+632554us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":73398656}
I20260709 12:21:25.193946 18362 raft_consensus.cc:1240] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20260709 12:21:25.182286 18363 raft_consensus.cc:2468] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 2.
I20260709 12:21:25.196125 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
W20260709 12:21:25.208709 17804 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:25.209357 19879 consensus_queue.cc:260] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:25.209995 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:25.210680 19817 leader_election.cc:290] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 3 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:21:25.211148 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 119740us
W20260709 12:21:25.211412 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 183222us
W20260709 12:21:25.213328 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 70045.3us
I20260709 12:21:25.215099 17652 leader_election.cc:304] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:21:25.215969 19673 raft_consensus.cc:2749] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:21:25.217165 19895 consensus_queue.cc:1048] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:21:25.219231 19895 raft_consensus.cc:2804] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:21:25.220219 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:25.220858 19766 raft_consensus.cc:515] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:25.184310 17830 raft_consensus.cc:2468] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 6.
I20260709 12:21:25.206689 19845 raft_consensus.cc:3060] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:25.221643 17833 raft_consensus.cc:2468] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 4.
W20260709 12:21:25.222872 17801 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:25.223397 17801 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:25.171056 19865 raft_consensus.cc:515] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:25.224356 19766 leader_election.cc:290] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 4 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:25.215526 19883 raft_consensus.cc:515] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:21:25.225622 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:25.226251 19865 leader_election.cc:290] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 3 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:25.227072 19848 consensus_queue.cc:1059] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 4.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:21:25.227589 19883 leader_election.cc:290] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 2 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:25.228132 19705 consensus_queue.cc:1059] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 12:21:25.229528 19705 raft_consensus.cc:3055] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:21:25.229923 19705 raft_consensus.cc:740] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 3 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:21:25.231113 19705 consensus_queue.cc:260] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:21:25.232358 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 56569.7us
I20260709 12:21:25.232769 19911 raft_consensus.cc:493] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:25.234436 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:25.235448 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:25.235843 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:25.236387 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:25.242724 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:25.240439 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:25.241473 17727 leader_election.cc:304] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:21:25.241269 19673 consensus_queue.cc:1059] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:21:25.249203 19673 raft_consensus.cc:3055] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:21:25.249788 19673 raft_consensus.cc:740] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:21:25.238727 17835 raft_consensus.cc:2468] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 3.
I20260709 12:21:25.250731 19835 raft_consensus.cc:493] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:25.250877 19673 consensus_queue.cc:260] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:25.249302 17688 raft_consensus.cc:2468] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 2.
I20260709 12:21:25.251415 19867 consensus_queue.cc:1059] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:21:25.251595 19728 consensus_queue.cc:1059] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 12:21:25.241845 17833 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35040 (request call id 1814) took 1007 ms. Trace:
I20260709 12:21:25.250783 19804 raft_consensus.cc:2804] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:21:25.253031 19804 raft_consensus.cc:493] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:21:25.252529 17833 rpcz_store.cc:276] 0709 12:21:24.234652 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:24.258195 (+ 23543us) service_pool.cc:224] Handling call
0709 12:21:25.241655 (+983460us) spinlock_profiling.cc:243] Waited 454 ms on lock 0x7b54001581d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:21:25.241787 (+   132us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1045342464}
I20260709 12:21:25.253628 19804 raft_consensus.cc:3060] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:25.252703 19862 consensus_queue.cc:1059] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.002s
I20260709 12:21:25.255456 19862 raft_consensus.cc:3055] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:21:25.258882 19862 raft_consensus.cc:740] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 4 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:21:25.251416 19835 raft_consensus.cc:515] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:25.261216 19835 leader_election.cc:290] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:25.210705 19879 raft_consensus.cc:3060] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:25.255555 19815 raft_consensus.cc:3055] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 2 LEADER]: Stepping down as leader of term 2
W20260709 12:21:25.263705 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 48805.9us
I20260709 12:21:25.264052 19815 raft_consensus.cc:740] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 2 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:21:25.265273 19815 consensus_queue.cc:260] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:25.267201 19815 raft_consensus.cc:3060] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:25.256526 19817 raft_consensus.cc:2749] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:21:25.256814 17727 leader_election.cc:304] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:21:25.268880 19817 raft_consensus.cc:2749] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:21:25.270712 19673 raft_consensus.cc:3060] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:25.247275 19911 raft_consensus.cc:515] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:25.257833 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:25.257988 19897 raft_consensus.cc:3055] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:21:25.272343 17832 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 2.
I20260709 12:21:25.272294 19897 raft_consensus.cc:740] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 4 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:21:25.274302 19764 consensus_queue.cc:260] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:25.258193 18333 leader_election.cc:304] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:25.275835 19764 raft_consensus.cc:3060] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:21:25.276105 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 53918.8us
I20260709 12:21:25.277217 19842 raft_consensus.cc:2749] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:21:25.277515 18333 leader_election.cc:304] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:21:25.279364 19761 raft_consensus.cc:2804] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:21:25.280758 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:25.281489 17688 raft_consensus.cc:2468] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 3.
I20260709 12:21:25.281896 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:25.282404 19761 raft_consensus.cc:493] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:21:25.282609 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:25.282866 19761 raft_consensus.cc:3060] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:25.283303 17688 raft_consensus.cc:3060] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:25.284972 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:25.286265 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:25.287410 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:25.288200 17680 raft_consensus.cc:2468] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 2.
I20260709 12:21:25.288609 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:25.282759 17685 raft_consensus.cc:2410] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for term 6 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20260709 12:21:25.290046 17689 raft_consensus.cc:1275] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:21:25.291381 17689 raft_consensus.cc:1240] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 3. Current term is 4. Ops: []
I20260709 12:21:25.287441 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:25.292815 19705 raft_consensus.cc:3060] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:25.260768 19862 consensus_queue.cc:260] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:25.296432 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:25.296610 19862 raft_consensus.cc:3060] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:25.238477 19895 raft_consensus.cc:697] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 3 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:21:25.298903 19842 raft_consensus.cc:493] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:25.298765 19895 consensus_queue.cc:237] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:25.274166 19897 consensus_queue.cc:260] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:25.299600 19842 raft_consensus.cc:515] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:25.300905 19897 raft_consensus.cc:3060] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:25.301890 17829 raft_consensus.cc:1240] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 2. Current term is 3. Ops: []
I20260709 12:21:25.302012 19842 leader_election.cc:290] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:25.275074 19911 leader_election.cc:290] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:25.308259 17683 raft_consensus.cc:2468] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 3.
I20260709 12:21:25.309199 17688 raft_consensus.cc:2468] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 3.
I20260709 12:21:25.311453 17684 raft_consensus.cc:2468] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 2.
I20260709 12:21:25.312949 17832 raft_consensus.cc:1240] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 2. Current term is 3. Ops: []
W20260709 12:21:25.313664 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 48580.6us
I20260709 12:21:25.314554 17803 leader_election.cc:304] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:21:25.314728 19885 consensus_queue.cc:1048] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:21:25.316527 19728 raft_consensus.cc:2749] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:21:25.314566 17652 leader_election.cc:304] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:21:25.317409 17686 raft_consensus.cc:2468] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 5.
I20260709 12:21:25.317740 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:25.318620 17833 raft_consensus.cc:1240] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 2. Current term is 3. Ops: []
I20260709 12:21:25.318828 17829 raft_consensus.cc:2468] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 3.
I20260709 12:21:25.320351 17837 raft_consensus.cc:1240] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 4. Current term is 5. Ops: []
I20260709 12:21:25.320881 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:25.323552 19845 raft_consensus.cc:2804] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:21:25.269518 17727 leader_election.cc:304] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:21:25.324201 19845 raft_consensus.cc:493] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:21:25.324848 19845 raft_consensus.cc:3060] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:21:25.324970 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 55621.2us
I20260709 12:21:25.326025 17727 leader_election.cc:304] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:21:25.326840 19890 raft_consensus.cc:2749] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:21:25.327484 19890 raft_consensus.cc:2749] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:21:25.328442 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:25.329011 17838 raft_consensus.cc:3060] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:25.331346 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:25.332502 17685 raft_consensus.cc:1240] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260709 12:21:25.332406 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:25.333957 17837 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 5.
I20260709 12:21:25.335986 17834 raft_consensus.cc:2468] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 2.
I20260709 12:21:25.336361 19817 raft_consensus.cc:493] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:21:25.339314 19817 raft_consensus.cc:515] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:25.341595 19817 leader_election.cc:290] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:25.346717 17836 raft_consensus.cc:2468] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 2.
I20260709 12:21:25.358525 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:25.364478 17836 raft_consensus.cc:2468] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 3.
I20260709 12:21:25.366791 17803 leader_election.cc:304] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:21:25.371047 17652 leader_election.cc:304] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:21:25.372822 19753 raft_consensus.cc:2804] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:21:25.373649 19911 raft_consensus.cc:493] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:25.374190 19911 raft_consensus.cc:515] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:25.376577 19911 leader_election.cc:290] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:25.377209 19753 raft_consensus.cc:697] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 2 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:21:25.378705 19753 consensus_queue.cc:237] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:25.394841 17686 raft_consensus.cc:1275] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 12:21:25.398073 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:25.392669 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:25.400700 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:25.401482 17836 raft_consensus.cc:2410] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for term 5 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260709 12:21:25.384941 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:25.408535 17838 raft_consensus.cc:2468] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 4.
I20260709 12:21:25.409524 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:25.406366 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:25.411620 17727 leader_election.cc:304] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:21:25.414077 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:25.415719 17834 raft_consensus.cc:1240] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 2. Current term is 3. Ops: []
I20260709 12:21:25.359870 19761 raft_consensus.cc:515] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:25.419735 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:21:25.438390 18334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:25.406637 17680 raft_consensus.cc:1240] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 3 LEADER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 2. Current term is 3. Ops: []
I20260709 12:21:25.456192 19779 consensus_queue.cc:1059] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.051s
W20260709 12:21:25.457580 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 89830.8us
I20260709 12:21:25.458142 19817 raft_consensus.cc:2749] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:21:25.457453 18363 raft_consensus.cc:2955] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:21:25.459929 19925 raft_consensus.cc:493] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:21:25.459267 17837 raft_consensus.cc:2955] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:21:25.455744 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:25.467073 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:25.411367 19897 raft_consensus.cc:2749] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:21:25.463537 17652 leader_election.cc:304] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:21:25.469517 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:25.469754 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:25.470057 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:25.470245 18368 raft_consensus.cc:2468] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 3.
I20260709 12:21:25.470772 19868 raft_consensus.cc:493] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:25.464324 19873 raft_consensus.cc:493] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:25.471266 19868 raft_consensus.cc:515] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:25.471625 19873 raft_consensus.cc:515] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:21:25.472471 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 116124us
W20260709 12:21:25.473239 17803 leader_election.cc:336] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Timed out: RequestConsensusVote RPC to 127.16.29.68:42577 timed out after 1.536s (SENT)
I20260709 12:21:25.473620 17803 leader_election.cc:304] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:21:25.474716 19873 leader_election.cc:290] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:25.466748 17685 raft_consensus.cc:2468] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 5.
I20260709 12:21:25.475102 17803 leader_election.cc:304] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:21:25.476249 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:25.477021 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:25.477517 17831 raft_consensus.cc:1275] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:21:25.469431 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:25.469659 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:25.479147 17652 leader_election.cc:304] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:21:25.479017 18364 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 5.
I20260709 12:21:25.468230 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:25.481484 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:25.482309 17687 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 4.
I20260709 12:21:25.477057 18366 raft_consensus.cc:2468] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 5.
I20260709 12:21:25.487586 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:25.485242 19911 raft_consensus.cc:2804] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:21:25.488651 18368 raft_consensus.cc:2393] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 5: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:21:25.488768 19911 raft_consensus.cc:697] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 4 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:21:25.489985 19911 consensus_queue.cc:237] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:25.490568 17727 leader_election.cc:304] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:21:25.492187 19921 raft_consensus.cc:2804] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:21:25.492650 19921 raft_consensus.cc:493] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:25.493095 17727 leader_election.cc:304] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:21:25.493568 19890 consensus_queue.cc:1048] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:21:25.494820 19761 leader_election.cc:290] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 7 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:21:25.497453 17571 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet d522f70beb1f417cb98940d90b594823 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica e2909cff1da44ad387f96a8ae3aa5afb is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:21:25.498835 19807 raft_consensus.cc:493] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:21:25.499536 19807 raft_consensus.cc:515] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:25.500000 19925 raft_consensus.cc:515] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:25.493098 19921 raft_consensus.cc:3060] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:25.501901 19807 leader_election.cc:290] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:25.502488 19925 leader_election.cc:290] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:25.502775 19826 raft_consensus.cc:493] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:25.503647 19890 raft_consensus.cc:2804] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:21:25.503456 19826 raft_consensus.cc:515] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:25.504244 19838 raft_consensus.cc:493] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:21:25.504877 19838 raft_consensus.cc:515] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:25.506138 19826 leader_election.cc:290] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:25.506697 19838 leader_election.cc:290] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:25.506742 19837 raft_consensus.cc:493] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:25.507336 19838 raft_consensus.cc:493] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:21:25.507413 19837 raft_consensus.cc:515] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:25.509588 19837 leader_election.cc:290] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:25.510048 19890 raft_consensus.cc:697] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:21:25.512581 18367 raft_consensus.cc:3060] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:21:25.516386 17802 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:25.519081 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 44085.8us
I20260709 12:21:25.529203 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:25.528577 18366 raft_consensus.cc:1240] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 2. Current term is 3. Ops: []
I20260709 12:21:25.483799 19895 raft_consensus.cc:2804] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:21:25.482807 19815 raft_consensus.cc:2749] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:21:25.485734 19832 raft_consensus.cc:2804] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260709 12:21:25.533061 17654 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:25.531522 18334 leader_election.cc:304] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:21:25.538534 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:25.511067 19890 consensus_queue.cc:237] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:25.539199 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:25.539345 19779 consensus_queue.cc:1059] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 12:21:25.539924 19895 raft_consensus.cc:493] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:25.540894 18364 raft_consensus.cc:1240] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 4. Current term is 5. Ops: []
I20260709 12:21:25.541072 18333 leader_election.cc:304] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:21:25.544714 17758 raft_consensus.cc:1240] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 2. Current term is 3. Ops: []
W20260709 12:21:25.546810 17802 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:25.543292 19804 raft_consensus.cc:515] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:25.559338 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:21:25.562489 17730 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:25.519028 19807 raft_consensus.cc:493] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:21:25.564297 17803 leader_election.cc:304] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:21:25.507824 19838 raft_consensus.cc:515] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:25.548969 19832 raft_consensus.cc:493] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:21:25.551222 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:25.575846 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:25.559039 19895 raft_consensus.cc:3060] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:25.578540 19873 raft_consensus.cc:2749] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:21:25.563323 19883 raft_consensus.cc:493] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:25.586653 19924 raft_consensus.cc:2804] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:21:25.587220 19924 raft_consensus.cc:697] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 3 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:21:25.587363 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:25.589362 19753 raft_consensus.cc:493] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:25.590055 19753 raft_consensus.cc:515] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:25.590627 19931 raft_consensus.cc:493] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:25.591296 19931 raft_consensus.cc:515] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:25.576113 19925 consensus_queue.cc:1048] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.020s
I20260709 12:21:25.593243 19753 leader_election.cc:290] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:25.593982 19931 leader_election.cc:290] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:25.581689 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:25.595268 19838 leader_election.cc:290] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:25.595784 17761 raft_consensus.cc:2410] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for term 4 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260709 12:21:25.594003 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:25.597151 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:25.557107 19761 consensus_queue.cc:1059] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.025s
I20260709 12:21:25.599397 17729 leader_election.cc:304] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:21:25.579775 19832 raft_consensus.cc:3060] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:25.603190 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:25.619251 17756 raft_consensus.cc:2468] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 3.
I20260709 12:21:25.631675 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:25.632598 17836 raft_consensus.cc:2410] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 12:21:25.633147 18367 raft_consensus.cc:2468] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 3.
I20260709 12:21:25.633224 17835 raft_consensus.cc:1275] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260709 12:21:25.621261 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:25.624143 18333 leader_election.cc:304] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:25.641295 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:25.603695 19603 raft_consensus.cc:493] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:25.643286 17654 leader_election.cc:304] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:21:25.628389 19728 raft_consensus.cc:2749] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:21:25.627840 19804 leader_election.cc:290] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 3 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:25.653625 17758 raft_consensus.cc:1240] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 2. Current term is 3. Ops: []
I20260709 12:21:25.585070 19921 raft_consensus.cc:515] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:21:25.660616 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 74465.9us
I20260709 12:21:25.639056 19838 raft_consensus.cc:2804] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:21:25.664868 17728 leader_election.cc:304] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:21:25.670104 17728 leader_election.cc:304] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:21:25.662464 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:25.670888 17835 raft_consensus.cc:2468] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 2.
I20260709 12:21:25.664338 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:25.664788 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:25.672078 17838 raft_consensus.cc:2410] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for term 6 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20260709 12:21:25.605594 19883 raft_consensus.cc:515] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:25.666716 17756 raft_consensus.cc:1240] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 2. Current term is 3. Ops: []
I20260709 12:21:25.609382 19807 raft_consensus.cc:515] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:25.674149 17729 leader_election.cc:304] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:21:25.668198 17831 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35040 (request call id 1818) took 1093 ms. Trace:
I20260709 12:21:25.676987 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:21:25.680361 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 128366us
I20260709 12:21:25.649168 19761 raft_consensus.cc:2749] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:21:25.681281 17831 rpcz_store.cc:276] 0709 12:21:24.567273 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:24.765100 (+197827us) service_pool.cc:224] Handling call
0709 12:21:24.765758 (+   658us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:25.477907 (+712149us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:25.660777 (+182870us) spinlock_profiling.cc:243] Waited 711 ms on lock 0x7b54002ebb54. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:21:25.660941 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1635980032}
I20260709 12:21:25.683452 18333 leader_election.cc:304] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:21:25.687314 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 44182.9us
I20260709 12:21:25.691015 18333 leader_election.cc:304] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:21:25.693178 17652 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:25.694181 18333 leader_election.cc:304] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:21:25.694854 19779 raft_consensus.cc:2749] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:21:25.694981 19694 raft_consensus.cc:2749] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:21:25.696028 19855 raft_consensus.cc:3055] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:21:25.686625 17763 raft_consensus.cc:2468] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 3.
I20260709 12:21:25.699369 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:25.700459 17838 raft_consensus.cc:2393] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 5: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:21:25.701222 18333 leader_election.cc:304] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:25.704512 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:25.674813 19838 raft_consensus.cc:493] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:25.709542 19826 raft_consensus.cc:2749] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:21:25.710103 19673 consensus_queue.cc:1048] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:21:25.747293 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:25.713302 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:25.748132 17763 raft_consensus.cc:2393] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 4: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:21:25.716491 19808 raft_consensus.cc:2804] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:21:25.748565 19808 raft_consensus.cc:493] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:25.748958 19808 raft_consensus.cc:3060] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:25.749157 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:25.749940 17763 raft_consensus.cc:2393] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 5: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
W20260709 12:21:25.717063 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:25.729880 19868 leader_election.cc:290] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:25.730732 17728 leader_election.cc:304] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:21:25.753276 17728 leader_election.cc:304] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:21:25.754377 17728 leader_election.cc:304] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:21:25.755059 19817 raft_consensus.cc:2804] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:21:25.755610 19817 raft_consensus.cc:697] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:21:25.756733 17583 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35958 (request call id 409) took 4215 ms. Trace:
I20260709 12:21:25.756644 19817 consensus_queue.cc:237] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:25.757473 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:25.732720 19883 leader_election.cc:290] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:25.676530 19753 raft_consensus.cc:2749] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:21:25.736940 19807 leader_election.cc:290] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:25.783624 17837 raft_consensus.cc:1275] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
I20260709 12:21:25.784379 17837 pending_rounds.cc:85] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b: Aborting all ops after (but not including) 1
I20260709 12:21:25.696561 19855 raft_consensus.cc:740] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
W20260709 12:21:25.738689 19941 log.cc:927] Time spent T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.050s	user 0.000s	sys 0.000s
I20260709 12:21:25.738817 19845 raft_consensus.cc:515] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:21:25.798072 17804 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:25.798946 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 103130us
I20260709 12:21:25.739434 17803 leader_election.cc:304] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:21:25.740082 17832 raft_consensus.cc:2393] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 3: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:21:25.743755 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:25.803692 17764 raft_consensus.cc:2468] T 7cb3768d215e409fb9f5baf18d124442 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 3.
I20260709 12:21:25.803910 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:25.804881 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:25.805130 19845 leader_election.cc:290] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 4 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:25.805622 17764 raft_consensus.cc:3060] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:25.744371 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:25.785950 19873 raft_consensus.cc:2749] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:21:25.808872 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:25.809397 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:25.810293 17755 raft_consensus.cc:2468] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 3.
I20260709 12:21:25.744731 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:25.745241 17761 raft_consensus.cc:1240] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260709 12:21:25.755437 19832 raft_consensus.cc:515] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:25.746335 19838 raft_consensus.cc:3060] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:21:25.761745 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 73544.5us
I20260709 12:21:25.687976 19931 raft_consensus.cc:2804] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:21:25.661818 19603 raft_consensus.cc:515] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:25.802798 19893 raft_consensus.cc:493] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:25.805781 17655 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:25.745383 17756 raft_consensus.cc:1275] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:21:25.636123 19924 consensus_queue.cc:237] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:21:25.811726 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 137688us
I20260709 12:21:25.785181 19837 raft_consensus.cc:2804] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:21:25.797933 19826 raft_consensus.cc:2804] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:21:25.785267 17837 pending_rounds.cc:107] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20260709 12:21:25.825917 17837 raft_consensus.cc:2889] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:21:25.827368 17801 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:25.829638 18333 leader_election.cc:304] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:21:25.811481 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:25.866606 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:21:25.866478 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 49967.3us
W20260709 12:21:25.873744 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 51750.1us
I20260709 12:21:25.866669 17755 raft_consensus.cc:2468] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 3.
I20260709 12:21:25.876365 19779 raft_consensus.cc:2804] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:21:25.876821 19779 raft_consensus.cc:493] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:25.876947 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:25.877099 17654 leader_election.cc:304] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:21:25.877254 19779 raft_consensus.cc:3060] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:25.877964 19893 raft_consensus.cc:515] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:25.878757 19911 raft_consensus.cc:493] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:21:25.879369 19911 raft_consensus.cc:515] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:25.870707 19804 raft_consensus.cc:493] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:21:25.880095 19753 consensus_queue.cc:1048] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 12:21:25.880429 19804 raft_consensus.cc:515] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:25.881160 19603 leader_election.cc:290] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:25.881456 19911 leader_election.cc:290] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:25.878233 17654 leader_election.cc:304] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:21:25.881536 19753 raft_consensus.cc:2764] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20260709 12:21:25.886938 17729 leader_election.cc:304] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:21:25.817845 19931 raft_consensus.cc:493] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
W20260709 12:21:25.888273 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 148953us
I20260709 12:21:25.889129 17764 raft_consensus.cc:2468] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 3.
I20260709 12:21:25.893126 18333 leader_election.cc:304] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:25.812191 19895 raft_consensus.cc:515] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:25.827090 19897 raft_consensus.cc:493] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:25.894054 17654 leader_election.cc:304] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:21:25.817921 19877 raft_consensus.cc:3055] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:21:25.896623 17761 raft_consensus.cc:1240] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20260709 12:21:25.901919 17760 raft_consensus.cc:1275] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:21:25.823189 19673 raft_consensus.cc:493] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:25.824992 19705 raft_consensus.cc:2749] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:21:25.827510 19873 consensus_queue.cc:1059] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.071s
I20260709 12:21:25.825744 19837 raft_consensus.cc:493] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:21:25.828593 19826 raft_consensus.cc:493] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:21:25.918229 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:25.932021 17683 raft_consensus.cc:2410] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for term 6 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20260709 12:21:25.923189 19874 raft_consensus.cc:493] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:25.899770 19757 raft_consensus.cc:2804] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:21:25.925057 19694 raft_consensus.cc:2804] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:21:25.892866 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:25.910177 19897 raft_consensus.cc:515] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:25.926383 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:25.892580 19931 raft_consensus.cc:3060] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:25.870934 19842 consensus_queue.cc:1059] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.041s
I20260709 12:21:25.996837 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:25.997587 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:26.006385 17688 raft_consensus.cc:2468] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 4.
I20260709 12:21:25.928404 17687 raft_consensus.cc:2393] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 4: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:21:25.898587 19804 leader_election.cc:290] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:25.933396 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:25.924652 17689 raft_consensus.cc:1240] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 2. Current term is 3. Ops: []
W20260709 12:21:25.935361 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 63294.5us
I20260709 12:21:25.937255 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:25.886837 19890 consensus_queue.cc:1048] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.029s
I20260709 12:21:25.899821 19877 raft_consensus.cc:740] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:21:25.941514 17729 leader_election.cc:304] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:21:25.908291 19807 raft_consensus.cc:2804] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:21:25.946669 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:25.908378 19845 raft_consensus.cc:493] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:21:25.912283 19911 raft_consensus.cc:2749] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:21:25.998979 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:25.948787 17760 raft_consensus.cc:2468] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 3.
I20260709 12:21:25.923790 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:25.997526 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:26.007612 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:26.032163 17728 leader_election.cc:304] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:21:25.934067 19826 raft_consensus.cc:3060] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:26.092998 17834 raft_consensus.cc:2468] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 4.
I20260709 12:21:25.997211 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:21:26.008060 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 58766.4us
I20260709 12:21:25.949635 17681 raft_consensus.cc:1240] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 2. Current term is 3. Ops: []
I20260709 12:21:26.027783 17756 raft_consensus.cc:1240] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 3. Current term is 4. Ops: []
I20260709 12:21:26.027110 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:25.925012 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:21:26.055473 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 159574us
I20260709 12:21:25.962060 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:25.757170 17583 rpcz_store.cc:276] 0709 12:21:21.541323 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:21.541471 (+   148us) service_pool.cc:224] Handling call
0709 12:21:25.756479 (+4215008us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:21:23.950336 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:21:23.950977 (+   641us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:21:23.951010 (+    33us) write_op.cc:435] Acquired schema lock
0709 12:21:23.951027 (+    17us) tablet.cc:662] Decoding operations
0709 12:21:23.951586 (+   559us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:21:23.951660 (+    74us) write_op.cc:641] Partition lock acquired for write op
0709 12:21:23.951688 (+    28us) tablet.cc:685] Acquiring locks for 3 operations
0709 12:21:23.952026 (+   338us) tablet.cc:701] Row locks acquired
0709 12:21:23.952044 (+    18us) write_op.cc:260] PREPARE: finished
0709 12:21:23.983828 (+ 31784us) write_op.cc:270] Start()
0709 12:21:24.020002 (+ 36174us) write_op.cc:276] Timestamp: P: 1783599683979419 usec, L: 0
0709 12:21:24.068899 (+ 48897us) op_driver.cc:348] REPLICATION: starting
0709 12:21:24.256578 (+187679us) log.cc:844] Serialized 1303 byte log entry
0709 12:21:25.212326 (+955748us) op_driver.cc:464] REPLICATION: finished
0709 12:21:25.246537 (+ 34211us) spinlock_profiling.cc:243] Waited 25.8 ms on lock 0x7b4400022b4c. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b47459c85 00007f7b4745a6a2 00007f7b4a1cc1db 00007f7b4a1c7aaf 00007f7b475160ac 00007f7b503709a9 00007f7b4753269b 00007f7b4753093f 00007f7b47532c02 00007f7b4e056ef0
0709 12:21:25.476570 (+230033us) write_op.cc:301] APPLY: starting
0709 12:21:25.476696 (+   126us) tablet.cc:1367] starting BulkCheckPresence
0709 12:21:25.476887 (+   191us) tablet.cc:1370] finished BulkCheckPresence
0709 12:21:25.476942 (+    55us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:21:25.547416 (+ 70474us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:21:25.547730 (+   314us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=3
0709 12:21:25.547794 (+    64us) write_op.cc:312] APPLY: finished
0709 12:21:25.578414 (+ 30620us) log.cc:844] Serialized 35 byte log entry
0709 12:21:25.596462 (+ 18048us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:21:25.596645 (+   183us) write_op.cc:454] Released schema lock
0709 12:21:25.602998 (+  6353us) write_op.cc:341] FINISH: Updating metrics
0709 12:21:25.624330 (+ 21332us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:21:25.624369 (+    39us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":238864,"reported_tablets":4,"child_traces":[["op",{"apply.queue_time_us":263880,"apply.run_cpu_time_us":7038,"apply.run_wall_time_us":134465,"num_ops":3,"prepare.queue_time_us":169596,"prepare.run_cpu_time_us":18178,"prepare.run_wall_time_us":663770,"raft.queue_time_us":171015,"raft.run_cpu_time_us":17519,"raft.run_wall_time_us":214659,"replication_time_us":1143153,"spinlock_wait_cycles":59429248,"thread_start_us":852689,"threads_started":4,"wal-append.queue_time_us":708737}]]}
I20260709 12:21:26.082494 17837 raft_consensus.cc:2468] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 3.
I20260709 12:21:25.931543 19837 raft_consensus.cc:3060] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:26.052598 18368 raft_consensus.cc:2468] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 3.
W20260709 12:21:26.089747 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 62096us
I20260709 12:21:26.089605 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:25.940729 19757 raft_consensus.cc:493] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:26.004818 19832 leader_election.cc:290] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 5 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:25.948949 19885 raft_consensus.cc:493] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:25.999408 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:25.998029 19897 leader_election.cc:290] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:25.909433 19855 consensus_queue.cc:260] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:26.119305 17684 raft_consensus.cc:2468] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 3.
I20260709 12:21:25.941133 19893 leader_election.cc:290] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:26.137997 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:26.085824 19694 raft_consensus.cc:697] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 2 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:21:26.001681 17830 raft_consensus.cc:2468] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 4.
I20260709 12:21:26.080289 19807 raft_consensus.cc:493] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:21:26.062423 19652 raft_consensus.cc:493] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:26.132071 17802 leader_election.cc:304] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:21:26.084707 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:26.136327 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:26.089432 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:26.122771 17688 raft_consensus.cc:2468] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 2.
I20260709 12:21:26.138698 17803 leader_election.cc:304] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:21:26.104820 19921 leader_election.cc:290] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 3 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:26.120538 19804 raft_consensus.cc:493] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:21:26.150373 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:26.150447 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:26.154955 17837 raft_consensus.cc:2410] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for term 6 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20260709 12:21:26.025247 19673 raft_consensus.cc:515] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:26.132673 19757 raft_consensus.cc:3060] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:26.025331 19874 raft_consensus.cc:515] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:26.155822 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:26.159206 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:26.164901 17688 raft_consensus.cc:2468] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 2.
I20260709 12:21:26.161334 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:26.161815 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:26.165292 17837 raft_consensus.cc:1275] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 12:21:26.150132 18366 raft_consensus.cc:1275] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260709 12:21:26.151111 19807 raft_consensus.cc:3060] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:26.152534 19652 raft_consensus.cc:515] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:26.181044 17689 raft_consensus.cc:2393] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 5: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:21:26.181280 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:26.183856 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:26.132042 19845 raft_consensus.cc:515] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:26.159904 19832 raft_consensus.cc:2749] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260709 12:21:26.186470 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 47841.9us
I20260709 12:21:26.147243 17833 raft_consensus.cc:2468] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 5.
I20260709 12:21:26.150669 17680 raft_consensus.cc:2468] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 4.
I20260709 12:21:26.163677 19868 raft_consensus.cc:2804] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:21:26.189544 17802 leader_election.cc:304] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:21:26.146893 19877 consensus_queue.cc:260] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:26.191424 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:26.156452 17755 raft_consensus.cc:2468] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 4.
I20260709 12:21:26.192737 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:26.193849 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:26.194624 17837 raft_consensus.cc:2393] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 3: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:21:26.146991 19955 consensus_queue.cc:1048] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.052s
I20260709 12:21:26.139968 19885 raft_consensus.cc:515] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:26.204735 18367 raft_consensus.cc:2468] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 3.
I20260709 12:21:26.207232 17756 raft_consensus.cc:2393] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 3: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:21:26.209990 19603 raft_consensus.cc:2804] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:21:26.211400 19603 raft_consensus.cc:493] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:26.211745 19603 raft_consensus.cc:3060] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:26.211745 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:26.212496 17756 raft_consensus.cc:2468] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 3.
I20260709 12:21:26.213310 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:26.216951 19877 raft_consensus.cc:3060] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:26.217535 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:26.218361 17760 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 5.
I20260709 12:21:26.186084 19652 leader_election.cc:290] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:26.186559 19921 raft_consensus.cc:493] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:21:26.211771 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:26.222749 17803 leader_election.cc:304] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:21:26.235419 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 114865us
I20260709 12:21:26.195123 19868 raft_consensus.cc:493] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:26.239260 19603 raft_consensus.cc:515] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:21:26.248149 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 320865us
I20260709 12:21:26.221388 19895 leader_election.cc:290] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 4 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:26.266506 18333 leader_election.cc:304] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:21:26.268040 17729 leader_election.cc:304] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
W20260709 12:21:26.239609 19949 log.cc:927] Time spent T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.098s	user 0.000s	sys 0.001s
I20260709 12:21:26.202447 19804 raft_consensus.cc:515] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:26.241721 19652 raft_consensus.cc:2804] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:21:26.242167 19868 raft_consensus.cc:3060] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:26.210668 19808 raft_consensus.cc:515] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:26.271335 19958 raft_consensus.cc:2804] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260709 12:21:26.275942 19946 log.cc:927] Time spent T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.343s	user 0.000s	sys 0.001s
I20260709 12:21:26.273725 17830 raft_consensus.cc:1240] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 3. Current term is 4. Ops: []
I20260709 12:21:26.194471 17684 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d"
dest_uuid: "0da169ff82674dc1947055ce43de68a3"
mode: GRACEFUL
new_leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
 from {username='slave'} at 127.0.0.1:54540
I20260709 12:21:26.309933 17684 raft_consensus.cc:606] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Received request to transfer leadership to TS e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:26.310421 17684 raft_consensus.cc:614] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:21:26.278702 19652 raft_consensus.cc:493] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:21:26.293011 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:26.307552 17654 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:26.308228 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 59296.3us
I20260709 12:21:26.319455 17763 raft_consensus.cc:2368] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for earlier term 2. Current term is 3.
I20260709 12:21:26.266121 19921 raft_consensus.cc:515] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:26.287221 19705 raft_consensus.cc:2749] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20260709 12:21:26.323990 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 292016us
I20260709 12:21:26.289170 19958 raft_consensus.cc:493] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:21:26.324584 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 123409us
I20260709 12:21:26.326478 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:26.328814 18333 leader_election.cc:304] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:21:26.330972 17727 leader_election.cc:304] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:21:26.332647 19603 leader_election.cc:290] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 3 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:26.315948 19652 raft_consensus.cc:3060] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:26.336162 17729 leader_election.cc:304] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:21:26.275395 19694 consensus_queue.cc:237] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:26.303574 19956 consensus_queue.cc:1059] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.029s
I20260709 12:21:26.351415 19964 consensus_queue.cc:1048] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.031s
W20260709 12:21:26.352224 19952 log.cc:927] Time spent T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.174s	user 0.001s	sys 0.000s
I20260709 12:21:26.352402 19807 raft_consensus.cc:515] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:26.353794 19960 raft_consensus.cc:2804] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:21:26.354341 19960 raft_consensus.cc:697] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 2 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:21:26.354923 19807 leader_election.cc:290] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 7 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:26.350265 17803 leader_election.cc:304] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:26.332644 19958 raft_consensus.cc:3060] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:26.336941 19957 raft_consensus.cc:493] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:26.355657 19807 raft_consensus.cc:2749] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:21:26.424350 19966 raft_consensus.cc:2749] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260709 12:21:26.432665 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 82487.8us
W20260709 12:21:26.425801 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 68828.8us
I20260709 12:21:26.433143 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
W20260709 12:21:26.424583 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 80794.7us
I20260709 12:21:26.346431 19673 leader_election.cc:290] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:21:26.433485 17802 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:26.355230 19960 consensus_queue.cc:237] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:21:26.435511 17802 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:26.432765 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 406546us
W20260709 12:21:26.436147 17655 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:26.363337 19924 raft_consensus.cc:2804] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Leader election won for term 3
W20260709 12:21:26.441056 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 104979us
I20260709 12:21:26.441939 18334 leader_election.cc:304] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:21:26.425873 19874 leader_election.cc:290] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:26.444460 17727 leader_election.cc:304] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:21:26.432399 19897 raft_consensus.cc:2804] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:21:26.447675 19804 leader_election.cc:290] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:26.448876 19921 leader_election.cc:290] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:21:26.440677 19968 log.cc:927] Time spent T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.089s	user 0.001s	sys 0.000s
I20260709 12:21:26.437463 19957 raft_consensus.cc:515] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:26.453298 19757 raft_consensus.cc:515] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:21:26.454674 19779 consensus_meta.cc:287] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.577s	user 0.002s	sys 0.007s
I20260709 12:21:26.455561 19757 leader_election.cc:290] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 4 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:26.422612 19925 consensus_queue.cc:1048] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.020s
I20260709 12:21:26.455199 19957 leader_election.cc:290] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:26.464639 17803 leader_election.cc:304] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:21:26.466159 17803 leader_election.cc:304] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:26.455147 19779 raft_consensus.cc:515] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:26.467753 17803 leader_election.cc:304] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:26.438000 19956 raft_consensus.cc:493] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:26.440179 19845 leader_election.cc:290] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:26.477028 17727 leader_election.cc:304] T 7cb3768d215e409fb9f5baf18d124442 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:21:26.441246 19965 raft_consensus.cc:2749] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:21:26.468063 19832 raft_consensus.cc:2749] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:21:26.468305 19842 raft_consensus.cc:493] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:26.470468 19897 raft_consensus.cc:493] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:26.470144 19955 raft_consensus.cc:2749] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:21:26.446779 19885 leader_election.cc:290] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:26.495774 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:26.497251 19897 raft_consensus.cc:3060] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:21:26.498842 19954 log.cc:927] Time spent T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.177s	user 0.001s	sys 0.000s
I20260709 12:21:26.470285 19924 raft_consensus.cc:697] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 3 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:21:26.500653 19924 consensus_queue.cc:237] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:26.441154 19931 raft_consensus.cc:515] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:26.473762 19705 raft_consensus.cc:2804] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:21:26.486827 19879 raft_consensus.cc:2804] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260709 12:21:26.511389 17652 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:26.488456 19807 raft_consensus.cc:2749] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:21:26.491554 19845 raft_consensus.cc:493] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:26.510361 19842 raft_consensus.cc:515] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:26.470721 19961 consensus_queue.cc:1048] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.040s
I20260709 12:21:26.522222 17758 raft_consensus.cc:2393] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 4: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:21:26.523003 17764 raft_consensus.cc:1275] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 12:21:26.505501 19832 raft_consensus.cc:493] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:26.526937 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:26.543072 18363 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 830) took 1173 ms. Trace:
I20260709 12:21:26.543464 17680 raft_consensus.cc:2468] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 3.
I20260709 12:21:26.527520 19965 raft_consensus.cc:2749] T 7cb3768d215e409fb9f5baf18d124442 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:21:26.527592 19842 leader_election.cc:290] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:26.528987 17652 leader_election.cc:304] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:26.529057 19970 consensus_queue.cc:1059] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.015s
I20260709 12:21:26.514434 19705 raft_consensus.cc:697] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 4 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:21:26.529795 19804 raft_consensus.cc:493] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:26.530285 19883 raft_consensus.cc:493] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:21:26.550055 19757 raft_consensus.cc:2749] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:21:26.550467 19705 consensus_queue.cc:237] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:26.551096 19970 raft_consensus.cc:3055] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:21:26.551522 19970 raft_consensus.cc:740] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:21:26.552130 19804 raft_consensus.cc:515] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:26.552475 19970 consensus_queue.cc:260] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:26.553784 19970 raft_consensus.cc:3060] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:26.554111 19804 leader_election.cc:290] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:26.536674 19779 leader_election.cc:290] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 4 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:26.544718 19897 raft_consensus.cc:515] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:26.543722 18363 rpcz_store.cc:276] 0709 12:21:25.369664 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:25.385590 (+ 15926us) service_pool.cc:224] Handling call
0709 12:21:25.406830 (+ 21240us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:25.457237 (+ 50407us) raft_consensus.cc:1534] Early marking committed up to index 3
0709 12:21:25.784863 (+327626us) log.cc:844] Serialized 14 byte log entry
0709 12:21:25.822878 (+ 38015us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:26.479840 (+656962us) spinlock_profiling.cc:243] Waited 2.32 ms on lock 0x7b2c000123a8. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b48f19a11 00007f7b48f29372 00007f7b4a65ba01 00007f7b4a62043a 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2
0709 12:21:26.480396 (+   556us) log.cc:844] Serialized 26 byte log entry
0709 12:21:26.480608 (+   212us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:21:26.480664 (+    56us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:26.480769 (+   105us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:26.542641 (+ 61872us) raft_consensus.cc:1685] finished
0709 12:21:26.542770 (+   129us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:26.542954 (+   184us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":16711,"raft.queue_time_us":13648,"raft.run_cpu_time_us":293,"raft.run_wall_time_us":40207,"spinlock_wait_cycles":5330176,"thread_start_us":19995,"threads_started":1,"wal-append.queue_time_us":122245}
I20260709 12:21:26.556695 17652 leader_election.cc:304] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:21:26.525144 19845 raft_consensus.cc:515] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:26.557885 19897 leader_election.cc:290] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 4 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:26.558950 19845 leader_election.cc:290] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:26.559149 17652 leader_election.cc:304] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:26.556380 19883 raft_consensus.cc:515] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:26.524224 19848 raft_consensus.cc:2749] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:21:26.560557 19895 raft_consensus.cc:3060] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:26.560628 19832 raft_consensus.cc:515] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:26.561600 19883 leader_election.cc:290] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:26.547143 17832 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37994 (request call id 525) took 1232 ms. Trace:
W20260709 12:21:26.520710 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 43785.7us
I20260709 12:21:26.530629 19868 raft_consensus.cc:515] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:26.483160 19956 raft_consensus.cc:515] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:26.559547 19845 raft_consensus.cc:2804] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:21:26.544469 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:26.488976 19652 raft_consensus.cc:515] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:26.595537 17727 leader_election.cc:304] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:21:26.601491 19956 leader_election.cc:290] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:26.610563 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:26.605561 17727 leader_election.cc:304] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:21:26.615919 17727 leader_election.cc:304] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:21:26.620787 19832 leader_election.cc:290] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:26.623073 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:26.624157 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:26.628633 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:26.629247 17760 raft_consensus.cc:3060] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:26.634253 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:26.635061 17838 raft_consensus.cc:2468] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 3.
I20260709 12:21:26.588903 19845 raft_consensus.cc:493] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
W20260709 12:21:26.637696 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 170003us
I20260709 12:21:26.638320 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:26.639853 17803 leader_election.cc:304] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:21:26.637774 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:26.613620 19961 raft_consensus.cc:2804] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:21:26.643709 17836 raft_consensus.cc:2468] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 3.
I20260709 12:21:26.641021 17831 raft_consensus.cc:2468] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 4.
I20260709 12:21:26.640976 17803 leader_election.cc:304] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:21:26.629688 19957 raft_consensus.cc:2804] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:21:26.647312 17652 leader_election.cc:304] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:21:26.654356 19893 raft_consensus.cc:2804] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:21:26.654443 17652 leader_election.cc:304] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:26.655004 19893 raft_consensus.cc:493] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:26.655540 19893 raft_consensus.cc:3060] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:26.656070 19757 consensus_queue.cc:1059] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:21:26.656903 19757 consensus_queue.cc:1048] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:21:26.658216 19757 raft_consensus.cc:2749] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:21:26.648813 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:26.659996 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:26.660694 17834 raft_consensus.cc:3060] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:26.628774 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:26.664899 17833 raft_consensus.cc:2468] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 3.
I20260709 12:21:26.666718 17833 raft_consensus.cc:1240] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20260709 12:21:26.670687 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
W20260709 12:21:26.671736 18334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:26.673556 17727 leader_election.cc:304] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:21:26.674769 17727 leader_election.cc:304] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:21:26.607373 19807 consensus_queue.cc:1048] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.037s
I20260709 12:21:26.639379 19845 raft_consensus.cc:3060] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:26.676687 17652 leader_election.cc:304] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:21:26.574548 17832 rpcz_store.cc:276] 0709 12:21:25.315118 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:25.345248 (+ 30130us) service_pool.cc:224] Handling call
0709 12:21:26.546931 (+1201683us) spinlock_profiling.cc:243] Waited 339 ms on lock 0x7b54000ab4d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:21:26.547060 (+   129us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":779397888}
I20260709 12:21:26.642807 19832 raft_consensus.cc:2749] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:21:26.677752 17727 leader_election.cc:304] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:21:26.691529 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:21:26.691622 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 50728.2us
I20260709 12:21:26.641528 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:26.685112 17834 raft_consensus.cc:2468] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 4.
I20260709 12:21:26.646050 19961 raft_consensus.cc:493] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:26.696319 17652 leader_election.cc:304] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:26.646322 19960 raft_consensus.cc:493] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:21:26.690454 19959 raft_consensus.cc:2804] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:21:26.653224 19957 raft_consensus.cc:493] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:21:26.699234 19804 raft_consensus.cc:2804] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:21:26.700873 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:26.701318 19971 raft_consensus.cc:2804] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260709 12:21:26.702284 19969 log.cc:927] Time spent T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.127s	user 0.000s	sys 0.000s
I20260709 12:21:26.668673 19893 raft_consensus.cc:515] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:26.690745 19673 raft_consensus.cc:2804] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:21:26.704869 19652 leader_election.cc:290] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 4 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:26.700299 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:26.735888 19885 raft_consensus.cc:2749] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:21:26.709298 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:26.711063 19873 raft_consensus.cc:3055] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:21:26.711185 19958 raft_consensus.cc:515] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:26.713745 17654 leader_election.cc:304] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
W20260709 12:21:26.695317 19838 consensus_meta.cc:287] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.864s	user 0.012s	sys 0.001s
I20260709 12:21:26.739416 19971 raft_consensus.cc:493] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:21:26.724320 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:21:26.747785 19837 consensus_meta.cc:287] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.614s	user 0.009s	sys 0.002s
I20260709 12:21:26.738575 17762 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 5.
I20260709 12:21:26.738467 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:26.739401 17803 leader_election.cc:304] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:21:26.720512 19961 raft_consensus.cc:3060] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:26.703373 19976 raft_consensus.cc:2804] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:21:26.722817 19921 raft_consensus.cc:2804] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:21:26.788148 19921 raft_consensus.cc:493] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:26.788527 19921 raft_consensus.cc:3060] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:26.744638 17652 leader_election.cc:400] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 2 election: Vote denied by peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651) with higher term. Message: Invalid argument: T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for earlier term 2. Current term is 3.
I20260709 12:21:26.791867 17833 raft_consensus.cc:1275] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
W20260709 12:21:26.792305 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 54462.1us
I20260709 12:21:26.793089 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:26.723064 19959 raft_consensus.cc:493] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:26.707612 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:26.738271 19957 raft_consensus.cc:3060] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:21:26.755719 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 172554us
I20260709 12:21:26.756263 17680 raft_consensus.cc:3060] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:21:26.757988 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 44305.2us
I20260709 12:21:26.758313 18364 raft_consensus.cc:1240] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 3. Current term is 4. Ops: []
I20260709 12:21:26.756769 17832 raft_consensus.cc:1275] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:21:26.759083 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:26.759294 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:26.698518 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:26.759673 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:26.760983 19984 consensus_queue.cc:1059] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.016s
I20260709 12:21:26.821765 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:26.762154 19893 leader_election.cc:290] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 4 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:21:26.824321 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 103537us
I20260709 12:21:26.763988 19974 consensus_queue.cc:1048] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.042s
W20260709 12:21:26.768852 19826 consensus_meta.cc:287] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.637s	user 0.009s	sys 0.001s
I20260709 12:21:26.833693 19826 raft_consensus.cc:515] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:26.769608 18363 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 5.
I20260709 12:21:26.724663 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:26.787055 19976 raft_consensus.cc:493] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:26.780092 19971 raft_consensus.cc:3060] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:26.746058 19804 raft_consensus.cc:697] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 3 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:21:26.735594 19960 raft_consensus.cc:515] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:26.786296 18362 raft_consensus.cc:2393] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 3: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:21:26.786614 19603 raft_consensus.cc:2749] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:21:26.787626 17764 raft_consensus.cc:2368] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for earlier term 4. Current term is 5.
I20260709 12:21:26.794751 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:26.877946 17684 raft_consensus.cc:3060] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:21:26.791982 17687 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37402 (request call id 2153) took 1206 ms (client timeout 1597 ms). Trace:
I20260709 12:21:26.748229 19673 raft_consensus.cc:493] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:26.770023 19757 raft_consensus.cc:2804] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:21:26.757464 19873 raft_consensus.cc:740] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:21:26.802886 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:26.769349 19931 leader_election.cc:290] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 4 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:26.812067 18333 leader_election.cc:304] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:26.812472 17654 leader_election.cc:304] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:21:26.816265 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:26.818255 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:26.821753 19753 consensus_queue.cc:1048] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:21:26.785549 19837 raft_consensus.cc:515] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:26.822454 17832 raft_consensus.cc:1275] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 6. (term mismatch)
I20260709 12:21:26.831817 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:26.823262 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:26.925235 18371 raft_consensus.cc:2393] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 7: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:21:26.823925 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:26.824304 19895 raft_consensus.cc:2749] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:21:26.873126 19976 raft_consensus.cc:3060] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:26.778771 17760 raft_consensus.cc:2468] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 4.
I20260709 12:21:26.804062 19959 raft_consensus.cc:3060] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:21:26.847920 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:26.925686 19952 log.cc:927] Time spent T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.071s	user 0.000s	sys 0.000s
I20260709 12:21:26.837615 17803 leader_election.cc:304] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:26.872116 18368 raft_consensus.cc:2468] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 3.
I20260709 12:21:26.931658 19877 raft_consensus.cc:2749] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:21:26.934091 18368 raft_consensus.cc:1275] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:21:26.931823 19974 raft_consensus.cc:2749] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:21:26.934520 18361 raft_consensus.cc:3055] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:21:26.818773 19842 consensus_queue.cc:1059] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.055s
I20260709 12:21:26.934888 18361 raft_consensus.cc:740] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
W20260709 12:21:26.875938 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 85158.8us
I20260709 12:21:26.881775 18363 raft_consensus.cc:1275] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260709 12:21:26.888130 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:26.787231 19845 raft_consensus.cc:515] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:26.937098 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:26.890125 19757 raft_consensus.cc:493] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:21:26.890769 19673 raft_consensus.cc:3060] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:26.922433 18365 raft_consensus.cc:2393] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 3: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:21:26.822868 19917 raft_consensus.cc:493] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:26.922497 17762 raft_consensus.cc:2468] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 3.
I20260709 12:21:26.954615 18367 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59918 (request call id 839) took 1035 ms. Trace:
W20260709 12:21:26.899904 18334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:26.927356 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 115017us
I20260709 12:21:26.954984 18367 rpcz_store.cc:276] 0709 12:21:25.919174 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:25.933165 (+ 13991us) service_pool.cc:224] Handling call
0709 12:21:26.954488 (+1021323us) spinlock_profiling.cc:243] Waited 152 ms on lock 0x7b5400724054. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:21:26.954566 (+    78us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":349231360}
I20260709 12:21:26.876248 19806 raft_consensus.cc:3055] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:21:26.843995 19826 leader_election.cc:290] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 5 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:26.849321 19958 leader_election.cc:290] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 5 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:26.824636 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:26.972908 18364 raft_consensus.cc:2468] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 4.
I20260709 12:21:26.923189 17832 pending_rounds.cc:85] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b: Aborting all ops after (but not including) 5
I20260709 12:21:26.973809 17832 pending_rounds.cc:107] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b: Aborting uncommitted NO_OP operation due to leader change: 3.6
I20260709 12:21:26.974123 17832 raft_consensus.cc:2889] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:21:26.933912 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:26.975135 17830 raft_consensus.cc:2468] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 4.
I20260709 12:21:26.932818 19921 raft_consensus.cc:515] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:26.976444 17760 raft_consensus.cc:1240] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 2. Current term is 3. Ops: [2.5-2.5]
I20260709 12:21:26.977736 17760 raft_consensus.cc:1240] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 2. Current term is 3. Ops: []
I20260709 12:21:26.977866 19921 leader_election.cc:290] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 5 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:26.978180 19960 leader_election.cc:290] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:26.978566 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:26.979362 17833 raft_consensus.cc:2393] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 5: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:21:26.978900 19917 raft_consensus.cc:515] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:26.979995 17755 raft_consensus.cc:2393] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 5: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:21:26.980177 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:26.980814 17833 raft_consensus.cc:3060] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:26.981127 19917 leader_election.cc:290] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:26.935765 18362 raft_consensus.cc:3060] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:26.961244 19806 raft_consensus.cc:740] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:21:26.982519 19806 consensus_queue.cc:260] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:26.983484 19806 raft_consensus.cc:3060] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:26.985741 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:26.986402 17837 raft_consensus.cc:2468] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 5.
W20260709 12:21:26.880879 17687 rpcz_store.cc:269] 0709 12:21:25.585780 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:25.585898 (+   118us) service_pool.cc:224] Handling call
0709 12:21:26.791724 (+1205826us) spinlock_profiling.cc:243] Waited 334 ms on lock 0x7b5400348ad4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:21:26.791852 (+   128us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":767189760}
I20260709 12:21:26.935990 17801 leader_election.cc:304] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:21:26.988734 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 52858.4us
I20260709 12:21:26.779400 19838 raft_consensus.cc:515] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:21:26.938236 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 126379us
I20260709 12:21:26.932710 18366 raft_consensus.cc:2468] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 3.
I20260709 12:21:26.997048 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:26.997625 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:26.998624 17728 leader_election.cc:304] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:21:26.998881 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:26.999478 18366 raft_consensus.cc:3060] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:26.939507 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:27.000175 17761 raft_consensus.cc:2468] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 3.
I20260709 12:21:27.000254 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:27.001492 18368 raft_consensus.cc:1240] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 6. Current term is 7. Ops: []
I20260709 12:21:27.002243 17759 raft_consensus.cc:1240] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 2. Current term is 3. Ops: []
I20260709 12:21:27.002307 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:27.002991 18364 raft_consensus.cc:2468] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 2.
I20260709 12:21:27.004050 17762 raft_consensus.cc:1240] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 2. Current term is 3. Ops: []
I20260709 12:21:26.952315 19757 raft_consensus.cc:3060] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:26.945495 19705 raft_consensus.cc:2804] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Leader election won for term 3
W20260709 12:21:26.946111 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 108590us
I20260709 12:21:26.999706 17728 leader_election.cc:304] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:21:26.945996 18361 consensus_queue.cc:260] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:26.946848 19837 leader_election.cc:290] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 2 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:26.881904 19832 consensus_queue.cc:1059] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.060s
I20260709 12:21:26.948467 17680 raft_consensus.cc:2468] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 3.
I20260709 12:21:27.019528 19991 consensus_queue.cc:1048] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.020s
I20260709 12:21:27.020418 17830 raft_consensus.cc:1240] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20260709 12:21:26.949326 19804 consensus_queue.cc:237] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:27.021934 17801 leader_election.cc:304] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:26.951628 19845 leader_election.cc:290] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 3 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:26.928524 19807 raft_consensus.cc:3055] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:21:26.947641 18371 raft_consensus.cc:1275] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:21:26.953833 17684 raft_consensus.cc:2468] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 4.
I20260709 12:21:27.033056 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:26.960644 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:26.951512 19976 raft_consensus.cc:515] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:26.991519 18362 raft_consensus.cc:2468] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 4.
I20260709 12:21:26.994481 18333 leader_election.cc:304] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:26.995230 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:21:27.006019 19993 log.cc:927] Time spent T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.061s	user 0.001s	sys 0.000s
I20260709 12:21:27.006628 19978 raft_consensus.cc:493] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:27.007004 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:27.007150 19963 consensus_queue.cc:1059] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 12:21:27.002526 18334 leader_election.cc:304] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
W20260709 12:21:26.944823 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 150466us
I20260709 12:21:27.016522 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:27.025298 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:21:27.060725 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:27.053938 17689 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37402 (request call id 2155) took 1052 ms. Trace:
W20260709 12:21:27.061574 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:27.044528 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:27.040719 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:26.989565 19603 raft_consensus.cc:2804] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:21:27.142005 19603 raft_consensus.cc:493] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:27.142457 19603 raft_consensus.cc:3060] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:27.042021 17759 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38776 (request call id 1657) took 1299 ms. Trace:
I20260709 12:21:27.051707 17688 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37402 (request call id 2156) took 1049 ms. Trace:
I20260709 12:21:27.142028 17684 raft_consensus.cc:2368] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for earlier term 2. Current term is 3.
I20260709 12:21:27.028640 17654 leader_election.cc:304] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:21:27.140735 17836 raft_consensus.cc:2368] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for earlier term 2. Current term is 3.
I20260709 12:21:27.039819 18365 raft_consensus.cc:2468] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 1.
W20260709 12:21:27.146201 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 117649us
I20260709 12:21:27.149734 17728 leader_election.cc:304] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:21:27.003515 19826 raft_consensus.cc:2749] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:21:27.003165 19921 raft_consensus.cc:493] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:27.171530 17756 raft_consensus.cc:1240] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 3. Current term is 4. Ops: []
I20260709 12:21:27.143517 17688 rpcz_store.cc:276] 0709 12:21:26.001915 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:26.182159 (+180244us) service_pool.cc:224] Handling call
0709 12:21:26.182452 (+   293us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:26.237697 (+ 55245us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:26.237733 (+    36us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:26.242290 (+  4557us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:21:26.242351 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:27.051408 (+809057us) spinlock_profiling.cc:243] Waited 40.9 ms on lock 0x7b54003485d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:21:27.051484 (+    76us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:27.051634 (+   150us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":94156544}
I20260709 12:21:27.140720 18363 raft_consensus.cc:2468] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 4.
I20260709 12:21:27.130630 17689 rpcz_store.cc:276] 0709 12:21:26.001651 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:26.181808 (+180157us) service_pool.cc:224] Handling call
0709 12:21:26.182173 (+   365us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:26.311937 (+129764us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:26.311978 (+    41us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:26.312037 (+    59us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:21:26.312125 (+    88us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:27.053698 (+741573us) spinlock_profiling.cc:243] Waited 108 ms on lock 0x7b54003471d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:21:27.053750 (+    52us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:27.053881 (+   131us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":249442176}
I20260709 12:21:27.155490 17727 leader_election.cc:400] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 2 election: Vote denied by peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837) with higher term. Message: Invalid argument: T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for earlier term 2. Current term is 3.
I20260709 12:21:27.176565 17727 leader_election.cc:403] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20260709 12:21:27.155344 19984 consensus_queue.cc:1048] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 12:21:27.156340 19890 consensus_queue.cc:1048] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 12:21:27.155243 17584 catalog_manager.cc:5697] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b reported cstate change: config changed from index 5 to 4, leader changed from <none> to e063541542a94969898868bf6eb5ce1b (127.16.29.67), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from VOTER to NON_VOTER, now has a pending config: VOTER 0da169ff82674dc1947055ce43de68a3 (127.16.29.65), VOTER e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66), VOTER e063541542a94969898868bf6eb5ce1b (127.16.29.67), VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68). New cstate: current_term: 4 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } health_report { overall_health: UNKNOWN } } } pending_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:21:27.157052 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
W20260709 12:21:27.155243 19949 log.cc:927] Time spent T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.077s	user 0.000s	sys 0.000s
I20260709 12:21:27.158259 19753 raft_consensus.cc:2804] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:21:27.032318 19807 raft_consensus.cc:740] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:21:27.037019 19837 raft_consensus.cc:2804] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:21:27.022428 19808 leader_election.cc:290] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 3 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:21:27.163692 19979 log.cc:927] Time spent T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.474s	user 0.001s	sys 0.001s
I20260709 12:21:27.042423 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:27.220788 17830 raft_consensus.cc:3060] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:27.053231 19848 raft_consensus.cc:2749] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260709 12:21:27.223965 17571 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet d522f70beb1f417cb98940d90b594823 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica e2909cff1da44ad387f96a8ae3aa5afb is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 3 committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:21:27.143110 17759 rpcz_store.cc:276] 0709 12:21:25.742557 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:25.745277 (+  2720us) service_pool.cc:224] Handling call
0709 12:21:25.745622 (+   345us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:27.005453 (+1259831us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:27.041698 (+ 36245us) spinlock_profiling.cc:243] Waited 1.25 s on lock 0x7b54003bced4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:21:27.041923 (+   225us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2886432512}
I20260709 12:21:27.225765 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:27.226078 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:27.164438 19817 raft_consensus.cc:2749] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260709 12:21:27.164680 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 137739us
I20260709 12:21:27.168093 18361 raft_consensus.cc:3060] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:27.168614 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:27.167659 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:27.240811 17681 raft_consensus.cc:3060] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:27.240952 17685 raft_consensus.cc:3060] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:21:27.170217 17758 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38776 (request call id 1661) took 1413 ms (client timeout 1625 ms). Trace:
I20260709 12:21:27.152364 17652 leader_election.cc:304] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
W20260709 12:21:27.243026 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 90780.3us
I20260709 12:21:27.244520 17652 leader_election.cc:304] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:21:27.245419 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:27.197306 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:27.248183 17684 raft_consensus.cc:2393] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 4: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:21:27.197844 17831 raft_consensus.cc:3060] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:21:27.198617 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 51717.4us
W20260709 12:21:27.200809 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 198340us
W20260709 12:21:27.211751 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 217345us
I20260709 12:21:27.249521 18333 leader_election.cc:304] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:27.250590 19761 raft_consensus.cc:2749] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:21:27.251271 19761 consensus_queue.cc:1048] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:21:27.181464 19603 raft_consensus.cc:515] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:21:27.168989 19973 log.cc:927] Time spent T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.526s	user 0.003s	sys 0.000s
W20260709 12:21:27.169642 19975 log.cc:927] Time spent T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.479s	user 0.001s	sys 0.000s
I20260709 12:21:27.150206 19897 raft_consensus.cc:493] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:27.209945 19753 raft_consensus.cc:493] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:21:27.217923 19963 raft_consensus.cc:3055] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:21:27.070053 19838 leader_election.cc:290] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 3 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:27.061247 19879 raft_consensus.cc:493] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:21:27.218230 17584 catalog_manager.cc:5697] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b reported cstate change: term changed from 2 to 3, leader changed from 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) to e063541542a94969898868bf6eb5ce1b (127.16.29.67). New cstate: current_term: 3 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:27.219412 19962 raft_consensus.cc:2804] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260709 12:21:27.210283 19737 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.059s	user 0.029s	sys 0.056s
I20260709 12:21:27.217985 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:27.226625 18362 raft_consensus.cc:3060] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:27.050431 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:27.068130 19705 raft_consensus.cc:697] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 3 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:21:27.230611 17728 leader_election.cc:304] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
W20260709 12:21:27.166833 19986 log.cc:927] Time spent T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.410s	user 0.002s	sys 0.001s
I20260709 12:21:27.200605 19972 raft_consensus.cc:3060] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:27.251868 18333 leader_election.cc:304] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:27.255357 18374 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 842) took 1327 ms. Trace:
I20260709 12:21:27.178474 19978 raft_consensus.cc:515] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:21:27.264523 19737 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.116s	user 0.029s	sys 0.057s
I20260709 12:21:27.264721 19960 raft_consensus.cc:2749] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:21:27.255925 19963 raft_consensus.cc:740] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:21:27.266059 19978 leader_election.cc:290] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:27.219436 19925 raft_consensus.cc:2749] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:21:27.266541 19963 consensus_queue.cc:260] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:27.173336 17833 raft_consensus.cc:2468] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 5.
I20260709 12:21:27.263818 18374 rpcz_store.cc:276] 0709 12:21:25.927519 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:25.927988 (+   469us) service_pool.cc:224] Handling call
0709 12:21:25.950252 (+ 22264us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:26.319160 (+368908us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:26.319210 (+    50us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:26.321497 (+  2287us) log.cc:844] Serialized 26 byte log entry
0709 12:21:26.424913 (+103416us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:21:26.424964 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:26.425052 (+    88us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:27.168311 (+743259us) spinlock_profiling.cc:243] Waited 3.41 ms on lock 0x7b4400022b4c. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b47459c85 00007f7b4745a6a2 00007f7b4a1cc1db 00007f7b4a1c7aaf 00007f7b4a1bb86e 00007f7b4a1babb6 00007f7b4a1ba7d8 00007f7b4a1c4221 00007f7b4a1c41d9 00007f7b4a1c4169
0709 12:21:27.255057 (+ 86746us) raft_consensus.cc:1685] finished
0709 12:21:27.255081 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:27.255292 (+   211us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":56048,"raft.queue_time_us":242113,"raft.run_cpu_time_us":203,"raft.run_wall_time_us":10644,"spinlock_wait_cycles":7832320,"thread_start_us":308191,"threads_started":2,"wal-append.queue_time_us":235196}
I20260709 12:21:27.257298 18366 raft_consensus.cc:2468] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 4.
I20260709 12:21:27.167601 19957 raft_consensus.cc:515] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:27.271358 19957 leader_election.cc:290] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 3 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:27.272028 19764 raft_consensus.cc:3055] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:21:27.272431 19764 raft_consensus.cc:740] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:21:27.273300 19764 consensus_queue.cc:260] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:27.274233 19868 leader_election.cc:290] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 6 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:27.261176 19757 raft_consensus.cc:515] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:27.280403 19976 leader_election.cc:290] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 6 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:27.280627 17836 raft_consensus.cc:1275] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260709 12:21:27.280970 19757 leader_election.cc:290] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 3 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:27.282248 17834 raft_consensus.cc:1275] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20260709 12:21:27.015668 19961 raft_consensus.cc:515] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:27.069295 19873 consensus_queue.cc:260] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:27.284578 19962 raft_consensus.cc:493] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:27.284768 19873 raft_consensus.cc:3060] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:27.284994 19962 raft_consensus.cc:3060] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:27.289065 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:27.289625 17687 raft_consensus.cc:3060] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:27.210505 19837 raft_consensus.cc:493] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:27.292150 19837 raft_consensus.cc:3060] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:27.292765 17584 catalog_manager.cc:5697] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b reported cstate change: term changed from 2 to 3, leader changed from 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) to e063541542a94969898868bf6eb5ce1b (127.16.29.67). New cstate: current_term: 3 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: HEALTHY } } }
I20260709 12:21:27.198669 19971 raft_consensus.cc:515] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:27.296217 17833 raft_consensus.cc:1275] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:21:27.297216 17832 raft_consensus.cc:3060] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:27.297755 19971 leader_election.cc:290] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 5 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:27.231406 17830 raft_consensus.cc:2468] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 3.
I20260709 12:21:27.253367 19603 leader_election.cc:290] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 4 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:27.210107 19921 raft_consensus.cc:515] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:27.301203 19705 consensus_queue.cc:237] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:27.157965 19931 consensus_queue.cc:1059] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.055s
I20260709 12:21:27.301864 19921 leader_election.cc:290] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:27.255612 19897 raft_consensus.cc:515] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:21:27.303967 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:27.304257 19897 leader_election.cc:290] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:21:27.305469 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 74930.1us
I20260709 12:21:27.306057 19817 raft_consensus.cc:2749] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:21:27.306598 17729 leader_election.cc:304] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:21:27.306854 19817 consensus_queue.cc:1059] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:21:27.307915 19817 raft_consensus.cc:2749] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:21:27.308276 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:27.255775 19753 raft_consensus.cc:3060] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:27.261300 18334 leader_election.cc:304] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
W20260709 12:21:27.309361 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 48147.5us
W20260709 12:21:27.310856 18334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:27.311225 18334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:27.311728 19960 raft_consensus.cc:2764] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20260709 12:21:27.312151 19895 raft_consensus.cc:2804] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:21:27.236783 17683 raft_consensus.cc:2468] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 5.
W20260709 12:21:27.313267 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 59775.9us
I20260709 12:21:27.313485 17683 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9"
dest_uuid: "0da169ff82674dc1947055ce43de68a3"
mode: GRACEFUL
new_leader_uuid: "334a4e1f59944544b3b3ff36616b57da"
 from {username='slave'} at 127.0.0.1:54540
I20260709 12:21:27.314442 17654 leader_election.cc:304] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
W20260709 12:21:27.241899 17758 rpcz_store.cc:269] 0709 12:21:25.756433 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:25.803595 (+ 47162us) service_pool.cc:224] Handling call
0709 12:21:27.169923 (+1366328us) spinlock_profiling.cc:243] Waited 717 ms on lock 0x7b540019f0d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:21:27.170120 (+   197us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1650000768}
I20260709 12:21:27.294793 17584 catalog_manager.cc:5697] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b reported cstate change: term changed from 1 to 2, leader changed from e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66) to e063541542a94969898868bf6eb5ce1b (127.16.29.67). New cstate: current_term: 2 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: HEALTHY } } }
W20260709 12:21:27.318645 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:27.319362 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:27.319636 17729 leader_election.cc:304] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:21:27.320811 18362 raft_consensus.cc:2468] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 5.
I20260709 12:21:27.324432 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:27.325071 17834 raft_consensus.cc:2468] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 4.
I20260709 12:21:27.325359 17684 raft_consensus.cc:1275] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 12:21:27.325974 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:27.301959 17727 leader_election.cc:304] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:21:27.332933 17838 raft_consensus.cc:2468] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 3.
I20260709 12:21:27.334468 17654 leader_election.cc:304] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:21:27.264243 19879 raft_consensus.cc:3060] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:27.336071 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:27.341468 17836 raft_consensus.cc:2468] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 4.
W20260709 12:21:27.337538 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 315644us
I20260709 12:21:27.334299 17727 leader_election.cc:304] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:21:27.338292 17684 raft_consensus.cc:1240] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 3. Current term is 4. Ops: []
I20260709 12:21:27.339200 17834 raft_consensus.cc:2468] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 5.
I20260709 12:21:27.346467 17729 leader_election.cc:304] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:21:27.312896 19806 raft_consensus.cc:2749] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:21:27.312562 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:27.352444 17727 leader_election.cc:304] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:21:27.274276 19764 raft_consensus.cc:3060] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:27.267357 19959 raft_consensus.cc:515] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:27.339246 17652 leader_election.cc:304] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:21:27.353276 17727 leader_election.cc:304] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:21:27.327577 19963 raft_consensus.cc:3060] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:27.349741 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:27.318148 19924 raft_consensus.cc:493] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:27.350126 17801 leader_election.cc:304] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:27.351287 17729 leader_election.cc:304] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:21:27.312577 19895 raft_consensus.cc:493] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:21:27.348538 17684 raft_consensus.cc:1240] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 3. Current term is 4. Ops: []
I20260709 12:21:27.333956 19931 raft_consensus.cc:493] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:27.340965 19890 raft_consensus.cc:2804] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:21:27.353276 17654 leader_election.cc:304] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
W20260709 12:21:27.360025 18334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:27.361865 17729 leader_election.cc:304] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:21:27.344203 19883 raft_consensus.cc:2804] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:21:27.342173 19957 raft_consensus.cc:493] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:27.364142 17801 leader_election.cc:304] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:27.340215 19761 raft_consensus.cc:493] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:21:27.334973 19757 consensus_queue.cc:1059] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.022s
I20260709 12:21:27.328366 19673 raft_consensus.cc:515] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:27.353785 19826 raft_consensus.cc:3060] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:27.359099 19992 raft_consensus.cc:2804] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:21:27.354489 19921 raft_consensus.cc:2749] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:21:27.359274 19737 raft_consensus.cc:2804] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:21:27.363972 19817 raft_consensus.cc:2749] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:21:27.369170 17801 leader_election.cc:304] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:21:27.383620 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:27.367069 19838 raft_consensus.cc:2804] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:21:27.397483 17838 raft_consensus.cc:3060] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:27.359925 19961 leader_election.cc:290] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 5 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:27.358867 19917 raft_consensus.cc:2804] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:21:27.358659 19971 raft_consensus.cc:2804] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:21:27.389170 17829 raft_consensus.cc:2393] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 3: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:21:27.362479 19895 raft_consensus.cc:3060] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:27.382368 17687 raft_consensus.cc:2468] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 5.
I20260709 12:21:27.365617 19883 raft_consensus.cc:493] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:21:27.388008 19806 raft_consensus.cc:2804] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:21:27.375924 19991 raft_consensus.cc:2804] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Leader election won for term 4
W20260709 12:21:27.404052 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 50835.3us
I20260709 12:21:27.375105 19832 raft_consensus.cc:3060] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:27.367256 19996 raft_consensus.cc:2749] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:21:27.380937 19890 raft_consensus.cc:697] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:21:27.414381 17834 raft_consensus.cc:1240] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 4. Current term is 5. Ops: []
I20260709 12:21:27.381626 19868 raft_consensus.cc:493] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:21:27.381417 19931 raft_consensus.cc:515] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:27.405042 17763 raft_consensus.cc:2368] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 3. Current term is 4.
W20260709 12:21:27.423985 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:27.426041 17728 leader_election.cc:304] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:21:27.428546 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:27.364291 19924 raft_consensus.cc:515] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:27.365854 19958 consensus_queue.cc:1048] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.039s
I20260709 12:21:27.374821 19984 consensus_queue.cc:1048] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.039s
I20260709 12:21:27.464103 19603 raft_consensus.cc:493] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:27.464829 19603 raft_consensus.cc:515] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:21:27.432425 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 63318.5us
I20260709 12:21:27.366053 19808 consensus_queue.cc:1048] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.040s
I20260709 12:21:27.434616 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:27.376345 19957 raft_consensus.cc:515] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:27.437942 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:27.468331 17835 raft_consensus.cc:2393] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 4: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:21:27.388193 19992 raft_consensus.cc:493] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:27.439632 18368 raft_consensus.cc:2368] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for earlier term 3. Current term is 4.
I20260709 12:21:27.471447 17681 raft_consensus.cc:2468] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 6.
W20260709 12:21:27.445375 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 106202us
I20260709 12:21:27.405973 19737 raft_consensus.cc:697] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 3 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:21:27.449440 18362 raft_consensus.cc:2468] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 3.
I20260709 12:21:27.386586 19761 raft_consensus.cc:515] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:27.424748 19883 raft_consensus.cc:3060] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:27.453301 17728 leader_election.cc:400] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 2 election: Vote denied by peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071) with higher term. Message: Invalid argument: T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for earlier term 2. Current term is 3.
I20260709 12:21:27.472733 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:27.474553 17688 raft_consensus.cc:1275] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 12:21:27.487994 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:27.488914 17684 raft_consensus.cc:2393] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 5: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:21:27.435560 19817 raft_consensus.cc:493] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:27.435135 19970 consensus_queue.cc:1059] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.020s
I20260709 12:21:27.477399 17801 leader_election.cc:304] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:27.472339 19992 raft_consensus.cc:3060] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:27.474828 19838 raft_consensus.cc:697] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 4 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:21:27.482432 17683 raft_consensus.cc:606] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Received request to transfer leadership to TS 334a4e1f59944544b3b3ff36616b57da
I20260709 12:21:27.507952 17683 raft_consensus.cc:614] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:21:27.485860 17652 leader_election.cc:400] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 4 election: Vote denied by peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651) with higher term. Message: Invalid argument: T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for earlier term 4. Current term is 5.
I20260709 12:21:27.508590 17652 leader_election.cc:403] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
I20260709 12:21:27.443817 17838 raft_consensus.cc:2468] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 5.
I20260709 12:21:27.475729 17654 leader_election.cc:400] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 3 election: Vote denied by peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577) with higher term. Message: Invalid argument: T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for earlier term 3. Current term is 4.
I20260709 12:21:27.510397 17654 leader_election.cc:403] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
W20260709 12:21:27.489840 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 44676.1us
I20260709 12:21:27.511531 17728 leader_election.cc:304] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:21:27.513919 17728 leader_election.cc:304] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:21:27.497303 19603 leader_election.cc:290] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:27.444916 19868 raft_consensus.cc:515] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:21:27.518559 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 44319.8us
I20260709 12:21:27.520717 17654 leader_election.cc:304] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:21:27.487552 19877 raft_consensus.cc:493] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:27.496372 19961 raft_consensus.cc:2804] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:21:27.538966 19806 raft_consensus.cc:697] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 4 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:21:27.511289 19996 raft_consensus.cc:2749] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:21:27.540426 19806 consensus_queue.cc:237] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:27.552035 19971 raft_consensus.cc:697] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 3 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:21:27.524729 20008 raft_consensus.cc:2804] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:21:27.564632 20002 raft_consensus.cc:2804] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:21:27.497475 19817 raft_consensus.cc:515] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:27.550336 19961 raft_consensus.cc:697] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 4 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:21:27.563664 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:27.553354 19959 leader_election.cc:290] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 4 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:21:27.591997 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 110088us
W20260709 12:21:27.628218 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 79799.2us
I20260709 12:21:27.549981 19958 consensus_queue.cc:1048] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.040s
W20260709 12:21:27.634994 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 114403us
I20260709 12:21:27.541186 19877 raft_consensus.cc:515] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:27.577101 20002 raft_consensus.cc:697] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 3 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
W20260709 12:21:27.637907 17652 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.16.29.66:39651 timed out after 1.625s (SENT) [suppressed 16 similar messages]
I20260709 12:21:27.575349 19998 raft_consensus.cc:2804] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:21:27.641937 17831 raft_consensus.cc:2410] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20260709 12:21:27.562693 19890 consensus_queue.cc:237] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:27.585438 19737 consensus_queue.cc:237] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:21:27.631179 19995 log.cc:927] Time spent T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.226s	user 0.000s	sys 0.001s
W20260709 12:21:27.691066 19999 log.cc:927] Time spent T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.257s	user 0.000s	sys 0.000s
I20260709 12:21:27.590368 20004 raft_consensus.cc:2749] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651) with higher term. Message: Invalid argument: T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for earlier term 4. Current term is 5.
I20260709 12:21:27.698032 17834 raft_consensus.cc:1240] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 2. Current term is 3. Ops: []
I20260709 12:21:27.649966 17685 raft_consensus.cc:2468] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 4.
W20260709 12:21:27.677371 20000 log.cc:927] Time spent T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.186s	user 0.000s	sys 0.001s
I20260709 12:21:27.678015 18361 raft_consensus.cc:1275] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 5. (term mismatch)
I20260709 12:21:27.685808 19868 leader_election.cc:290] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:27.727867 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:27.635717 19984 raft_consensus.cc:2804] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:21:27.684793 19761 leader_election.cc:290] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:27.637198 20008 raft_consensus.cc:697] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 5 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:21:27.730955 17831 raft_consensus.cc:2393] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 4: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:21:27.747114 20009 consensus_queue.cc:1048] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.027s
I20260709 12:21:27.747355 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:27.748067 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:27.748852 17829 raft_consensus.cc:2393] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 3: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:21:27.751330 19817 leader_election.cc:290] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:21:27.711266 19973 log.cc:927] Time spent T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.075s	user 0.000s	sys 0.000s
I20260709 12:21:27.726943 18361 pending_rounds.cc:85] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da: Aborting all ops after (but not including) 4
I20260709 12:21:27.739944 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:27.711017 19885 raft_consensus.cc:2749] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577) with higher term. Message: Invalid argument: T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for earlier term 3. Current term is 4.
I20260709 12:21:27.600605 19838 consensus_queue.cc:237] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:27.767529 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:27.736984 19957 leader_election.cc:290] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:27.736814 19931 leader_election.cc:290] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:27.769817 17685 raft_consensus.cc:2410] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for term 5 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260709 12:21:27.744204 19924 leader_election.cc:290] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:21:27.770367 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 387874us
I20260709 12:21:27.759449 18361 pending_rounds.cc:107] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da: Aborting uncommitted NO_OP operation due to leader change: 2.5
I20260709 12:21:27.794232 18361 raft_consensus.cc:2889] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:21:27.794412 17729 leader_election.cc:304] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:21:27.790580 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:27.783344 17682 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37396 (request call id 1742) took 1835 ms. Trace:
I20260709 12:21:27.743561 19961 consensus_queue.cc:237] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:27.737478 19971 consensus_queue.cc:237] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:27.796824 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:27.796176 17682 rpcz_store.cc:276] 0709 12:21:25.947915 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:26.155806 (+207891us) service_pool.cc:224] Handling call
0709 12:21:26.199996 (+ 44190us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:26.222181 (+ 22185us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:26.222227 (+    46us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:26.243770 (+ 21543us) log.cc:844] Serialized 26 byte log entry
0709 12:21:26.347287 (+103517us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:21:26.347328 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:26.347393 (+    65us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:26.770237 (+422844us) raft_consensus.cc:1685] finished
0709 12:21:26.770257 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:27.782959 (+1012702us) spinlock_profiling.cc:243] Waited 35.4 ms on lock 0x7b54003480d0. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a61f8ad 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:21:27.783255 (+   296us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":13984,"raft.run_cpu_time_us":191,"raft.run_wall_time_us":11935,"spinlock_wait_cycles":81394560,"thread_start_us":65312,"threads_started":1,"wal-append.queue_time_us":215863,"wal-append.run_cpu_time_us":2860,"wal-append.run_wall_time_us":1278548}
W20260709 12:21:27.787724 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:27.789438 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 149986us
I20260709 12:21:27.799535 17727 leader_election.cc:304] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:21:27.768332 19984 raft_consensus.cc:697] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:21:27.745713 19753 raft_consensus.cc:515] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:21:27.804497 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:27.804616 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:27.784523 19991 raft_consensus.cc:697] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 4 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:21:27.786413 20004 raft_consensus.cc:493] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:21:27.792188 20015 raft_consensus.cc:493] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:21:27.791544 19924 raft_consensus.cc:493] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:21:27.821974 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:27.852552 18362 raft_consensus.cc:2468] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 4.
I20260709 12:21:27.823760 19959 raft_consensus.cc:2749] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:21:27.823864 19958 consensus_queue.cc:1048] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:21:27.824684 19957 raft_consensus.cc:493] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:27.825390 19980 log.cc:927] Time spent T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 1.124s	user 0.000s	sys 0.004s
I20260709 12:21:27.774638 19962 raft_consensus.cc:515] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:21:27.828353 20003 log.cc:927] Time spent T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.234s	user 0.001s	sys 0.000s
I20260709 12:21:27.791738 20002 consensus_queue.cc:237] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:27.830039 19895 raft_consensus.cc:515] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:21:27.830960 20001 log.cc:927] Time spent T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.145s	user 0.000s	sys 0.001s
I20260709 12:21:27.832177 19877 leader_election.cc:290] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:27.832052 20008 consensus_queue.cc:237] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:27.888365 17686 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44898 (request call id 496) took 1371 ms. Trace:
W20260709 12:21:27.891968 17832 consensus_meta.cc:287] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.594s	user 0.006s	sys 0.006s
I20260709 12:21:27.832650 19879 raft_consensus.cc:515] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:27.892529 17832 raft_consensus.cc:2468] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 5.
I20260709 12:21:27.893229 17832 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35042 (request call id 1616) took 1020 ms. Trace:
I20260709 12:21:27.893649 17832 rpcz_store.cc:276] 0709 12:21:26.872426 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:26.975036 (+102610us) service_pool.cc:224] Handling call
0709 12:21:27.050243 (+ 75207us) spinlock_profiling.cc:243] Waited 6.43 ms on lock 0x7b580013fe98. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b474fbb71 00007f7b474fba22 00007f7b504148d9 00007f7b4f56eee0 00007f7b4f6380ab 00007f7b4f638131 00007f7b4f5e60a0 00007f7b4f5e5984 00007f7b48b606d4 00007f7b48b60662
0709 12:21:27.893154 (+842911us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":14790912}
I20260709 12:21:27.894930 19879 leader_election.cc:290] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 3 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:27.834357 19978 raft_consensus.cc:3055] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:21:27.835976 20014 raft_consensus.cc:493] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:27.821257 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:27.854367 17729 leader_election.cc:304] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:21:27.802928 20018 raft_consensus.cc:493] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:27.850698 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 51310.9us
W20260709 12:21:27.812474 19837 consensus_meta.cc:287] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.520s	user 0.008s	sys 0.001s
I20260709 12:21:27.864882 19753 leader_election.cc:290] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 3 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:27.869248 19832 raft_consensus.cc:2749] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:21:27.825866 19991 consensus_queue.cc:237] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:27.835188 19984 consensus_queue.cc:237] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:27.860626 19971 raft_consensus.cc:2804] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:21:27.886260 19895 leader_election.cc:290] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 4 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:27.858349 20015 raft_consensus.cc:515] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:27.858632 19924 raft_consensus.cc:515] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:27.851732 20004 raft_consensus.cc:515] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:27.897202 18371 raft_consensus.cc:2393] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 3: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
W20260709 12:21:27.939072 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:28.005661 16501 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 2
W20260709 12:21:28.006357 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 368440us
I20260709 12:21:28.026808 16501 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:21:28.027567 17652 leader_election.cc:304] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da
W20260709 12:21:27.934959 20011 log.cc:927] Time spent T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.071s	user 0.001s	sys 0.000s
I20260709 12:21:28.010299 19962 leader_election.cc:290] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 5 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:21:27.929342 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:28.032425 18367 raft_consensus.cc:2393] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 3: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:21:27.998662 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:28.034173 19971 raft_consensus.cc:697] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:21:28.076059 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:28.035055 20018 raft_consensus.cc:515] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:27.945300 19976 raft_consensus.cc:493] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:28.036931 17727 leader_election.cc:304] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:21:28.038023 20015 leader_election.cc:290] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:28.037665 19837 raft_consensus.cc:515] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:21:27.937531 19963 consensus_meta.cc:287] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.579s	user 0.009s	sys 0.002s
W20260709 12:21:28.019649 19826 consensus_meta.cc:287] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.631s	user 0.008s	sys 0.000s
I20260709 12:21:27.897805 17687 raft_consensus.cc:1240] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 3. Current term is 4. Ops: []
I20260709 12:21:27.899003 18374 raft_consensus.cc:1240] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 4. Current term is 5. Ops: [4.6-4.6]
W20260709 12:21:28.043200 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 188941us
I20260709 12:21:28.097496 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:28.048333 17802 leader_election.cc:304] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:28.039273 19978 raft_consensus.cc:740] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 2 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:21:28.076751 17682 raft_consensus.cc:3060] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:28.070713 19925 raft_consensus.cc:493] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:27.944499 19883 raft_consensus.cc:515] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:21:28.091351 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 54581.4us
I20260709 12:21:28.104575 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:27.888942 17686 rpcz_store.cc:276] 0709 12:21:26.517119 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:26.517242 (+   123us) service_pool.cc:224] Handling call
0709 12:21:26.531535 (+ 14293us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:26.531907 (+   372us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:26.531934 (+    27us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:26.532423 (+   489us) log.cc:844] Serialized 26 byte log entry
0709 12:21:26.642080 (+109657us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:21:26.642141 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:26.642207 (+    66us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:27.766529 (+1124322us) spinlock_profiling.cc:243] Waited 11.2 ms on lock 0x7b4400022b4c. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b47459c85 00007f7b4745a6a2 00007f7b4a1cc1db 00007f7b4a1c7aaf 00007f7b4a1bb86e 00007f7b4a1babb6 00007f7b4a1ba7d8 00007f7b4a1c4221 00007f7b4a1c41d9 00007f7b4a1c4169
0709 12:21:27.887979 (+121450us) raft_consensus.cc:1685] finished
0709 12:21:27.888011 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:27.888285 (+   274us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":247,"raft.run_cpu_time_us":98,"raft.run_wall_time_us":98,"spinlock_wait_cycles":25724800,"thread_start_us":109251,"threads_started":1,"wal-append.queue_time_us":166771}
I20260709 12:21:28.105248 17685 raft_consensus.cc:2468] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 3.
I20260709 12:21:28.048813 17652 leader_election.cc:304] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:28.098731 18367 raft_consensus.cc:3060] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:21:28.099255 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 51009.7us
I20260709 12:21:27.882891 19957 raft_consensus.cc:515] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:28.075098 19753 raft_consensus.cc:2804] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:21:28.071496 19890 raft_consensus.cc:2804] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:21:28.078529 18361 raft_consensus.cc:3060] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:21:28.070690 19972 consensus_meta.cc:287] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.746s	user 0.007s	sys 0.006s
W20260709 12:21:28.109313 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 60590.5us
I20260709 12:21:28.109658 17802 leader_election.cc:304] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:21:28.116526 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:28.134706 17685 raft_consensus.cc:2468] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 5.
I20260709 12:21:28.101281 19826 raft_consensus.cc:2749] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:21:28.114108 19991 raft_consensus.cc:2804] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:21:28.115818 19971 consensus_queue.cc:237] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:28.111624 19925 raft_consensus.cc:515] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:28.092562 19976 raft_consensus.cc:515] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:28.135615 17802 leader_election.cc:304] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:21:28.144579 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:28.108953 19832 raft_consensus.cc:2804] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:21:28.079684 19804 consensus_queue.cc:1048] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.037s
I20260709 12:21:28.095199 19838 raft_consensus.cc:2749] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:21:28.152531 19957 leader_election.cc:290] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:28.152611 19895 consensus_queue.cc:1048] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:21:28.153357 19924 leader_election.cc:290] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:28.111699 19848 consensus_queue.cc:1059] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.030s
I20260709 12:21:28.118391 20028 raft_consensus.cc:493] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:28.137812 20015 consensus_queue.cc:1048] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:21:28.138877 19807 consensus_queue.cc:260] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:28.139467 20018 leader_election.cc:290] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:28.139997 19897 raft_consensus.cc:493] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:21:28.140412 19868 raft_consensus.cc:2804] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:21:28.140815 19997 raft_consensus.cc:493] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:28.141400 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:28.146575 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:28.175935 17763 raft_consensus.cc:2468] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 4.
I20260709 12:21:28.147897 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:28.118218 19753 raft_consensus.cc:697] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 4 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:21:28.133229 19972 raft_consensus.cc:2749] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837) with higher term. Message: Invalid argument: T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for earlier term 2. Current term is 3.
I20260709 12:21:28.146324 17652 leader_election.cc:304] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:21:28.153479 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:28.115572 19890 raft_consensus.cc:493] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:21:28.189626 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:28.083123 20014 raft_consensus.cc:515] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:28.162323 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:28.141508 19991 raft_consensus.cc:493] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:28.190915 17758 raft_consensus.cc:2393] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 4: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:21:28.167271 17760 raft_consensus.cc:1240] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 2. Current term is 3. Ops: []
I20260709 12:21:28.169075 18367 raft_consensus.cc:2468] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 4.
I20260709 12:21:28.166173 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:28.173738 20028 raft_consensus.cc:515] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:28.185627 17762 raft_consensus.cc:3060] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:28.201129 17759 raft_consensus.cc:2468] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 5.
I20260709 12:21:28.188659 17764 raft_consensus.cc:1240] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 3. Current term is 4. Ops: []
I20260709 12:21:28.145654 20002 consensus_queue.cc:1048] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.029s
I20260709 12:21:28.210835 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:28.211970 17759 raft_consensus.cc:2393] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 4: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:21:28.212976 17756 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38786 (request call id 2320) took 2289 ms. Trace:
I20260709 12:21:28.213419 17756 rpcz_store.cc:276] 0709 12:21:25.923551 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:26.214138 (+290587us) service_pool.cc:224] Handling call
0709 12:21:27.171301 (+957163us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:27.195677 (+ 24376us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:28.212776 (+1017099us) spinlock_profiling.cc:243] Waited 957 ms on lock 0x7b540019f0d0. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a61f8ad 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:21:28.212915 (+   139us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2200580864}
W20260709 12:21:28.214047 20032 log.cc:927] Time spent T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.058s	user 0.000s	sys 0.000s
W20260709 12:21:28.214395 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:28.216266 19753 consensus_queue.cc:237] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:28.191298 19893 raft_consensus.cc:2804] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:21:28.190742 19991 raft_consensus.cc:3060] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:28.204181 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:28.239344 17760 raft_consensus.cc:2468] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 3.
W20260709 12:21:28.204316 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 937359us
I20260709 12:21:28.144850 19978 consensus_queue.cc:260] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:28.184676 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:28.191180 19806 consensus_queue.cc:1059] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 4.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.048s
I20260709 12:21:28.252210 19806 raft_consensus.cc:3055] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:21:28.252260 19925 leader_election.cc:290] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:28.252768 19806 raft_consensus.cc:740] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:21:28.255195 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:28.256304 18362 raft_consensus.cc:2468] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 3.
I20260709 12:21:28.193516 19890 raft_consensus.cc:3060] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:21:28.239184 19992 consensus_meta.cc:287] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.721s	user 0.011s	sys 0.001s
I20260709 12:21:28.264308 17756 raft_consensus.cc:1240] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20260709 12:21:28.176791 19897 raft_consensus.cc:515] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:28.271251 20004 leader_election.cc:290] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:28.238489 19893 raft_consensus.cc:493] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:21:28.255524 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:28.179342 19997 raft_consensus.cc:515] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:28.291242 17682 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 6.
I20260709 12:21:28.268546 19978 raft_consensus.cc:3060] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:28.286057 19893 raft_consensus.cc:3060] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:28.276021 19992 raft_consensus.cc:515] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:28.287958 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:28.305023 19960 raft_consensus.cc:3055] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:21:28.253912 19806 consensus_queue.cc:260] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:28.480343 17755 raft_consensus.cc:2393] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 4: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
W20260709 12:21:28.494185 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:28.390349 19960 raft_consensus.cc:740] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
W20260709 12:21:28.537501 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 280582us
W20260709 12:21:28.538568 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 354008us
I20260709 12:21:28.541375 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:28.538568 18333 leader_election.cc:400] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 3 pre-election: Vote denied by peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651) with higher term. Message: Invalid argument: T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 3. Current term is 4.
I20260709 12:21:28.548772 18333 leader_election.cc:403] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:21:28.561077 19974 raft_consensus.cc:493] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:21:28.562146 17756 raft_consensus.cc:1240] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260709 12:21:28.561954 19974 raft_consensus.cc:515] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:28.565194 19974 leader_election.cc:290] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:28.586010 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:28.587010 18362 raft_consensus.cc:2468] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 3.
W20260709 12:21:28.595981 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:28.530352 19976 leader_election.cc:290] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:28.600279 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:28.601178 18367 raft_consensus.cc:2468] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 5.
I20260709 12:21:28.550532 19925 raft_consensus.cc:2749] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651) with higher term. Message: Invalid argument: T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 3. Current term is 4.
I20260709 12:21:28.604707 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:28.551496 19837 leader_election.cc:290] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 4 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:28.607940 17687 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44898 (request call id 504) took 1174 ms. Trace:
I20260709 12:21:28.608461 17687 rpcz_store.cc:276] 0709 12:21:27.433963 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:27.437050 (+  3087us) service_pool.cc:224] Handling call
0709 12:21:27.473202 (+ 36152us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:28.095549 (+622347us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:28.607705 (+512156us) spinlock_profiling.cc:243] Waited 424 ms on lock 0x7b5400137ed4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:21:28.607862 (+   157us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":976058624}
I20260709 12:21:28.619282 17762 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 6.
I20260709 12:21:28.551254 19883 leader_election.cc:290] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 2 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:28.659694 20035 raft_consensus.cc:493] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:21:28.613444 19974 raft_consensus.cc:493] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:28.698385 19806 raft_consensus.cc:3060] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:28.573364 19804 consensus_queue.cc:1059] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true }, Status: INVALID_TERM, Last received: 2.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.094s
I20260709 12:21:28.656905 19976 raft_consensus.cc:2804] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:21:28.660611 20035 raft_consensus.cc:515] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:28.769855 19673 leader_election.cc:290] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 4 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:21:28.771931 18334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:28.803169 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:28.841503 19998 raft_consensus.cc:493] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:21:28.842235 19998 raft_consensus.cc:3060] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:28.668772 19960 consensus_queue.cc:260] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:21:28.983676 17804 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.822s	user 0.028s	sys 0.036s
W20260709 12:21:29.528743 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 990246us
W20260709 12:21:29.529174 17804 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.375s	user 0.047s	sys 0.046s
I20260709 12:21:29.015733 17688 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37396 (request call id 1766) took 1211 ms. Trace:
W20260709 12:21:29.530198 17804 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:29.530364 17688 rpcz_store.cc:276] 0709 12:21:27.804768 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:27.938917 (+134149us) service_pool.cc:224] Handling call
0709 12:21:27.943732 (+  4815us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:27.950803 (+  7071us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:27.950848 (+    45us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:27.953292 (+  2444us) log.cc:844] Serialized 26 byte log entry
0709 12:21:28.228361 (+275069us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:21:28.228429 (+    68us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:28.228529 (+   100us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:29.015407 (+786878us) raft_consensus.cc:1685] finished
0709 12:21:29.015443 (+    36us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:29.015648 (+   205us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":8058,"raft.queue_time_us":7395,"raft.run_cpu_time_us":196,"raft.run_wall_time_us":22446,"thread_start_us":274555,"threads_started":1,"wal-append.queue_time_us":1059582}
W20260709 12:21:28.961985 17683 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.861s	user 0.021s	sys 0.036s
W20260709 12:21:29.532189 17683 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.431s	user 0.027s	sys 0.050s
W20260709 12:21:29.049820 19832 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.874s	user 0.041s	sys 0.025s
I20260709 12:21:29.532765 18365 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59886 (request call id 732) took 1712 ms. Trace:
I20260709 12:21:29.533401 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:29.534514 18362 raft_consensus.cc:2410] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for term 4 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260709 12:21:29.533293 18365 rpcz_store.cc:276] 0709 12:21:27.819938 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:27.823007 (+  3069us) service_pool.cc:224] Handling call
0709 12:21:27.823308 (+   301us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:27.859591 (+ 36283us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:27.859637 (+    46us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:27.881918 (+ 22281us) log.cc:844] Serialized 26 byte log entry
0709 12:21:27.894416 (+ 12498us) spinlock_profiling.cc:243] Waited 8.69 ms on lock 0x7b4800220a58. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4f549aa1 00007f7b4f540b37 00007f7b4f5409ca 00007f7b4f63826f 00007f7b4f637f37 00007f7b4f64c55c 00007f7b4f64c508 00007f7b4f64c488 00007f7b4f64c443 00007f7b4f64b6ec 00007f7b4e575a96
0709 12:21:28.291162 (+396746us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:21:28.291204 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:28.291270 (+    66us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:29.012729 (+721459us) raft_consensus.cc:1685] finished
0709 12:21:29.012762 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:29.532705 (+519943us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":39337,"raft.queue_time_us":36649,"raft.run_cpu_time_us":1870,"raft.run_wall_time_us":23922,"spinlock_wait_cycles":19997952,"thread_start_us":324018,"threads_started":1,"wal-append.queue_time_us":1067165}
I20260709 12:21:29.534876 20054 raft_consensus.cc:493] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:29.049595 19737 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.070s	user 0.022s	sys 0.038s
W20260709 12:21:29.536003 19737 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.557s	user 0.025s	sys 0.045s
W20260709 12:21:29.012183 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:29.535595 20054 raft_consensus.cc:515] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:28.704800 19974 raft_consensus.cc:515] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:21:29.537119 19868 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.349s	user 0.013s	sys 0.048s
W20260709 12:21:29.537533 19868 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.350s	user 0.013s	sys 0.049s
I20260709 12:21:29.537830 19868 raft_consensus.cc:697] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 6 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:21:29.538549 19974 leader_election.cc:290] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:21:28.769860 20038 log.cc:927] Time spent T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.242s	user 0.000s	sys 0.000s
I20260709 12:21:29.538815 19868 consensus_queue.cc:237] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:21:29.432328 17654 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.818s	user 0.014s	sys 0.010s
W20260709 12:21:29.543237 17654 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.929s	user 0.017s	sys 0.011s
W20260709 12:21:29.545087 19961 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 1.373s	user 0.020s	sys 0.020s
W20260709 12:21:29.545619 19961 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.377s	user 0.020s	sys 0.020s
W20260709 12:21:29.546149 19826 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 1.363s	user 0.010s	sys 0.035s
W20260709 12:21:29.546670 19826 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.367s	user 0.010s	sys 0.036s
I20260709 12:21:29.548522 20058 raft_consensus.cc:493] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:29.549245 20058 raft_consensus.cc:515] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:21:29.557179 17803 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.294s	user 0.001s	sys 0.017s
W20260709 12:21:29.557430 20012 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.354s	user 0.010s	sys 0.030s
W20260709 12:21:29.557902 19848 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.396s	user 0.002s	sys 0.015s
W20260709 12:21:29.557955 20012 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.354s	user 0.010s	sys 0.030s
W20260709 12:21:29.558403 19848 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.397s	user 0.002s	sys 0.015s
W20260709 12:21:29.560141 19917 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 2.121s	user 0.018s	sys 0.018s
W20260709 12:21:29.560832 19917 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 2.125s	user 0.018s	sys 0.018s
I20260709 12:21:29.561228 19917 raft_consensus.cc:697] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:21:29.562573 19917 consensus_queue.cc:237] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:28.946712 20035 leader_election.cc:290] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:21:29.557651 17803 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.295s	user 0.001s	sys 0.017s
W20260709 12:21:29.566212 17801 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.018s	user 0.001s	sys 0.021s
W20260709 12:21:29.567366 17803 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:29.584419 17801 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.037s	user 0.001s	sys 0.021s
I20260709 12:21:28.767570 19976 raft_consensus.cc:493] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:21:29.532958 19832 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.360s	user 0.044s	sys 0.039s
W20260709 12:21:29.566866 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:29.535059 18362 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59886 (request call id 735) took 1714 ms. Trace:
W20260709 12:21:29.526706 19978 consensus_meta.cc:287] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 1.106s	user 0.010s	sys 0.000s
I20260709 12:21:29.592015 19868 raft_consensus.cc:493] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:29.682920 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:29.691695 20054 leader_election.cc:290] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:29.691985 19961 raft_consensus.cc:493] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
W20260709 12:21:29.593422 18335 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.729s	user 0.002s	sys 0.006s
W20260709 12:21:29.693267 18335 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.838s	user 0.003s	sys 0.006s
W20260709 12:21:29.693966 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:29.694715 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:29.593902 19883 raft_consensus.cc:3055] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 2 LEADER]: Stepping down as leader of term 2
W20260709 12:21:29.411657 17802 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.250s	user 0.008s	sys 0.008s
W20260709 12:21:29.600802 18333 connection.cc:256] RPC call timeout handler was delayed by -2.5150510210000903s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1925s for a response
W20260709 12:21:29.698707 17802 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.537s	user 0.008s	sys 0.010s
W20260709 12:21:28.986675 19838 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.804s	user 0.016s	sys 0.046s
I20260709 12:21:29.591987 19848 raft_consensus.cc:493] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
W20260709 12:21:29.666479 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:29.007421 19817 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.873s	user 0.008s	sys 0.057s
I20260709 12:21:29.666188 18362 rpcz_store.cc:276] 0709 12:21:27.820681 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:28.622013 (+801332us) service_pool.cc:224] Handling call
0709 12:21:29.535004 (+912991us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:29.683652 19855 raft_consensus.cc:3060] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:21:29.527733 19893 consensus_meta.cc:287] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 1.139s	user 0.008s	sys 0.001s
W20260709 12:21:29.590067 19753 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.031s	user 0.005s	sys 0.008s
I20260709 12:21:29.634033 19997 leader_election.cc:290] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:21:29.596048 18361 consensus_meta.cc:287] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 1.453s	user 0.012s	sys 0.002s
I20260709 12:21:28.844372 20014 leader_election.cc:290] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:29.634369 19976 raft_consensus.cc:3060] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:21:29.699056 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 1.56339e+06us
I20260709 12:21:29.698715 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:29.715596 17762 raft_consensus.cc:1275] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:21:29.699800 17759 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38776 (request call id 1682) took 1081 ms. Trace:
I20260709 12:21:29.592787 20067 raft_consensus.cc:493] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:21:29.666244 19832 raft_consensus.cc:697] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 3 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
W20260709 12:21:29.708062 19753 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.149s	user 0.013s	sys 0.010s
I20260709 12:21:29.710063 17834 raft_consensus.cc:1240] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 2. Current term is 3. Ops: []
I20260709 12:21:29.697639 20054 raft_consensus.cc:493] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:21:29.723435 17830 raft_consensus.cc:1240] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 3. Current term is 4. Ops: []
I20260709 12:21:29.711484 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:29.724759 18362 raft_consensus.cc:2468] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 3.
I20260709 12:21:29.724970 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:29.725239 18362 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59886 (request call id 738) took 1582 ms. Trace:
I20260709 12:21:29.712704 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:29.725558 18362 rpcz_store.cc:276] 0709 12:21:28.143287 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:29.701689 (+1558402us) service_pool.cc:224] Handling call
0709 12:21:29.725187 (+ 23498us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:21:29.703485 19817 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.588s	user 0.018s	sys 0.065s
I20260709 12:21:29.728430 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:29.717334 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:29.730315 17687 raft_consensus.cc:2393] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 3: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:21:29.698604 19883 raft_consensus.cc:740] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:21:29.666523 20010 raft_consensus.cc:493] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:29.719185 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:29.713210 19974 raft_consensus.cc:493] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:29.718081 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:29.722502 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:29.725564 17837 raft_consensus.cc:2468] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 3.
W20260709 12:21:29.703418 19838 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.521s	user 0.031s	sys 0.060s
I20260709 12:21:29.700139 19868 raft_consensus.cc:515] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:29.714711 17832 raft_consensus.cc:1240] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 2. Current term is 3. Ops: []
I20260709 12:21:29.709479 18361 raft_consensus.cc:2468] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 3.
I20260709 12:21:29.711563 19992 leader_election.cc:290] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 5 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:29.733364 17833 raft_consensus.cc:2468] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 2.
I20260709 12:21:29.692818 19961 raft_consensus.cc:515] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:29.735674 20012 raft_consensus.cc:493] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:29.757608 20012 raft_consensus.cc:515] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:21:29.758069 19806 consensus_meta.cc:287] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 1.059s	user 0.009s	sys 0.002s
I20260709 12:21:29.711203 20050 raft_consensus.cc:493] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:21:29.706406 19978 raft_consensus.cc:493] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:21:29.715080 19897 leader_election.cc:290] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:29.731922 17836 raft_consensus.cc:2468] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 3.
I20260709 12:21:29.714571 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:29.734887 18363 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59886 (request call id 723) took 2510 ms. Trace:
I20260709 12:21:29.718333 19997 raft_consensus.cc:493] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:29.719322 20015 raft_consensus.cc:493] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:21:29.706615 19848 raft_consensus.cc:515] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:29.716899 17759 rpcz_store.cc:276] 0709 12:21:28.618440 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:28.618568 (+   128us) service_pool.cc:224] Handling call
0709 12:21:29.534179 (+915611us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:29.534397 (+   218us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:21:29.534722 (+   325us) log.cc:844] Serialized 14 byte log entry
0709 12:21:29.699460 (+164738us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:29.699510 (+    50us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:21:29.699567 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:29.699636 (+    69us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:29.699749 (+   113us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1154,"prepare.queue_time_us":10040,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":45,"raft.queue_time_us":1346,"raft.run_cpu_time_us":134,"raft.run_wall_time_us":5394,"thread_start_us":163225,"threads_started":1}
I20260709 12:21:29.734858 20075 raft_consensus.cc:493] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:21:29.749801 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:29.791786 17830 raft_consensus.cc:2468] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 5.
I20260709 12:21:29.723348 19804 raft_consensus.cc:493] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:29.742321 18362 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 867) took 1665 ms. Trace:
I20260709 12:21:29.819209 18362 rpcz_store.cc:276] 0709 12:21:28.077207 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:29.726359 (+1649152us) service_pool.cc:224] Handling call
0709 12:21:29.741506 (+ 15147us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:29.741768 (+   262us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:29.741810 (+    42us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:29.741876 (+    66us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:21:29.741986 (+   110us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:29.742066 (+    80us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:29.742242 (+   176us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:29.718415 20073 raft_consensus.cc:493] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:29.751920 20057 raft_consensus.cc:3055] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:21:29.751513 19883 consensus_queue.cc:260] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:29.723067 20067 raft_consensus.cc:515] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:29.762544 19779 raft_consensus.cc:493] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:21:29.779311 20074 raft_consensus.cc:493] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:21:29.736114 20052 raft_consensus.cc:493] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:29.736013 20010 raft_consensus.cc:515] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:29.788568 17755 raft_consensus.cc:2468] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 2.
W20260709 12:21:29.759282 18361 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59918 (request call id 864) took 1881 ms (client timeout 1830 ms). Trace:
I20260709 12:21:29.789532 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
W20260709 12:21:29.789837 19890 consensus_meta.cc:287] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 1.522s	user 0.012s	sys 0.000s
I20260709 12:21:29.790442 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:29.729632 20054 raft_consensus.cc:515] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:29.708102 19893 raft_consensus.cc:515] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:29.813422 20085 raft_consensus.cc:493] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:29.816890 19855 raft_consensus.cc:493] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
W20260709 12:21:29.818298 19998 consensus_meta.cc:287] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.975s	user 0.009s	sys 0.000s
I20260709 12:21:29.820647 18364 raft_consensus.cc:1275] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20260709 12:21:29.735827 19974 raft_consensus.cc:515] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:29.783845 19897 raft_consensus.cc:493] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:29.779420 18363 rpcz_store.cc:276] 0709 12:21:27.224345 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:27.224472 (+   127us) service_pool.cc:224] Handling call
0709 12:21:27.224807 (+   335us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:27.267537 (+ 42730us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:27.267569 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:27.268111 (+   542us) log.cc:844] Serialized 26 byte log entry
0709 12:21:27.438374 (+170263us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:21:27.571136 (+132762us) log.cc:844] Serialized 14 byte log entry
0709 12:21:27.571381 (+   245us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:29.728865 (+2157484us) spinlock_profiling.cc:243] Waited 42.3 ms on lock 0x7b54006946d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:21:29.728904 (+    39us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:29.728946 (+    42us) raft_consensus.cc:1685] finished
0709 12:21:29.734679 (+  5733us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:29.734831 (+   152us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":132267,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":47,"raft.queue_time_us":138515,"raft.run_cpu_time_us":214,"raft.run_wall_time_us":5077,"spinlock_wait_cycles":97317248,"thread_start_us":200756,"threads_started":2,"wal-append.queue_time_us":266735,"wal-append.run_cpu_time_us":3770,"wal-append.run_wall_time_us":1399945}
I20260709 12:21:29.785367 19895 raft_consensus.cc:493] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:29.780364 20050 raft_consensus.cc:515] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:29.788090 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:29.843328 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:29.750708 19832 consensus_queue.cc:237] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:21:29.880002 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 130493us
I20260709 12:21:29.819226 19961 leader_election.cc:290] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:29.840431 20014 raft_consensus.cc:493] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:29.915683 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:29.779196 19978 raft_consensus.cc:515] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:29.890108 17802 leader_election.cc:304] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:21:29.821712 20057 raft_consensus.cc:740] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 3 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:21:29.962534 17652 leader_election.cc:304] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
W20260709 12:21:30.040804 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 150824us
I20260709 12:21:29.962028 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:29.964425 18362 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59886 (request call id 737) took 1849 ms. Trace:
I20260709 12:21:29.973955 19873 raft_consensus.cc:493] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:29.964764 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:29.965817 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:30.043605 17836 raft_consensus.cc:2468] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 3.
I20260709 12:21:30.042331 18362 rpcz_store.cc:276] 0709 12:21:28.114884 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:29.820255 (+1705371us) service_pool.cc:224] Handling call
0709 12:21:29.961376 (+141121us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:29.963114 (+  1738us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:29.963199 (+    85us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:29.963271 (+    72us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:21:29.963331 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:29.964234 (+   903us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:29.964366 (+   132us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:29.788198 19992 raft_consensus.cc:493] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:21:29.860020 20073 raft_consensus.cc:515] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:29.961535 19883 raft_consensus.cc:3060] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:29.980645 17801 leader_election.cc:304] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:21:29.859661 19991 consensus_meta.cc:287] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 1.609s	user 0.006s	sys 0.007s
W20260709 12:21:29.880198 18334 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.107s	user 0.005s	sys 0.014s
I20260709 12:21:29.964617 20002 raft_consensus.cc:493] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:29.788115 20015 raft_consensus.cc:515] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:29.841234 19806 raft_consensus.cc:493] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:29.970029 20078 raft_consensus.cc:493] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:29.971463 20080 raft_consensus.cc:493] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:29.964375 20052 raft_consensus.cc:515] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:30.038547 19895 raft_consensus.cc:515] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:30.066223 19873 raft_consensus.cc:515] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:21:29.980820 20070 log.cc:927] Time spent T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.051s	user 0.000s	sys 0.000s
I20260709 12:21:29.965847 20090 raft_consensus.cc:493] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:29.897380 20081 raft_consensus.cc:493] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:21:29.788796 19997 raft_consensus.cc:515] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:29.964185 20065 raft_consensus.cc:493] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:21:29.908385 19779 raft_consensus.cc:515] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:29.839784 19804 raft_consensus.cc:515] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:30.066485 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:30.113596 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:29.971383 20079 raft_consensus.cc:493] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:30.098940 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:21:30.099624 18334 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.327s	user 0.017s	sys 0.021s
W20260709 12:21:30.099858 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 137375us
I20260709 12:21:30.104542 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:30.122382 17836 raft_consensus.cc:2468] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 3.
I20260709 12:21:30.105696 19838 raft_consensus.cc:493] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:30.087486 20057 consensus_queue.cc:260] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:30.123783 19753 consensus_queue.cc:1059] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:21:30.124670 19779 leader_election.cc:290] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:30.124800 19753 raft_consensus.cc:3055] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:21:30.125286 19753 raft_consensus.cc:740] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:21:30.116755 20052 leader_election.cc:290] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:30.088310 20054 leader_election.cc:290] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:30.036201 19897 raft_consensus.cc:515] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:30.126153 19753 consensus_queue.cc:260] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:30.126714 19868 leader_election.cc:290] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:29.818760 20075 raft_consensus.cc:515] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:30.128053 17835 raft_consensus.cc:1240] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260709 12:21:30.046038 20014 raft_consensus.cc:515] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:30.106699 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:21:30.107729 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 269671us
W20260709 12:21:30.108415 17803 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:30.108729 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:30.108830 18368 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 866) took 2060 ms. Trace:
I20260709 12:21:30.091936 19961 raft_consensus.cc:493] T 7cb3768d215e409fb9f5baf18d124442 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:21:30.108054 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:30.110525 17756 raft_consensus.cc:1275] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:21:30.110832 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:29.964272 19890 raft_consensus.cc:515] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:30.114054 17830 raft_consensus.cc:2468] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 5.
W20260709 12:21:29.961923 18361 rpcz_store.cc:269] 0709 12:21:27.877948 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:27.976418 (+ 98470us) service_pool.cc:224] Handling call
0709 12:21:29.756368 (+1779950us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:29.964183 20074 raft_consensus.cc:515] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:30.114769 20002 raft_consensus.cc:515] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:30.116387 20067 leader_election.cc:290] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:30.070174 20077 raft_consensus.cc:2804] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:21:30.175666 20014 leader_election.cc:290] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:30.066802 19992 raft_consensus.cc:515] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:30.068380 19978 leader_election.cc:290] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:30.124682 20057 raft_consensus.cc:3060] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:30.085085 20059 raft_consensus.cc:2804] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:21:30.119323 20073 leader_election.cc:290] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:30.132196 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:30.133005 17652 leader_election.cc:304] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:21:30.175999 20077 raft_consensus.cc:493] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
W20260709 12:21:30.209088 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 76200.6us
I20260709 12:21:29.974169 20085 raft_consensus.cc:515] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:29.973747 19855 raft_consensus.cc:515] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:30.149290 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:30.149611 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:21:30.152316 18333 connection.cc:256] RPC call timeout handler was delayed by -1.400152755000363s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1766s for a response
I20260709 12:21:30.111747 20090 raft_consensus.cc:515] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:30.217043 20090 leader_election.cc:290] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:30.217811 18363 raft_consensus.cc:1275] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260709 12:21:29.982060 19998 raft_consensus.cc:515] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:30.166687 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:30.121929 20028 leader_election.cc:290] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:30.173480 20010 leader_election.cc:290] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:30.130515 19753 raft_consensus.cc:3060] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:30.224439 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:30.173719 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:30.174199 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:30.175022 20083 raft_consensus.cc:493] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:30.117117 19991 raft_consensus.cc:515] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:30.112919 20080 raft_consensus.cc:515] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:30.195931 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:30.196753 20092 raft_consensus.cc:493] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:30.130810 19779 raft_consensus.cc:493] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:21:30.198252 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:30.198902 19992 leader_election.cc:290] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:30.199331 20088 raft_consensus.cc:493] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:30.126175 20050 leader_election.cc:290] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:30.201371 17757 raft_consensus.cc:1240] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 4. Current term is 5. Ops: []
I20260709 12:21:30.201450 17729 leader_election.cc:304] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:21:30.242661 17729 leader_election.cc:304] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:21:30.203281 17837 raft_consensus.cc:3060] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:30.112672 19806 raft_consensus.cc:515] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:30.210948 20077 raft_consensus.cc:3060] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:30.128329 19893 leader_election.cc:290] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 4 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:30.223703 20076 raft_consensus.cc:2804] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:21:30.135519 19838 raft_consensus.cc:515] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:30.224426 17688 raft_consensus.cc:2468] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 1.
I20260709 12:21:30.227095 17759 raft_consensus.cc:2393] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 3: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:21:30.205199 19978 raft_consensus.cc:493] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:30.202960 20058 leader_election.cc:290] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:30.235442 18362 raft_consensus.cc:3060] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:30.230024 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:30.121548 20078 raft_consensus.cc:515] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:30.221545 20052 raft_consensus.cc:493] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:30.231068 19897 leader_election.cc:290] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:30.236024 20010 consensus_queue.cc:1059] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.012s
I20260709 12:21:30.248277 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:30.248346 20014 raft_consensus.cc:2804] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:21:30.168758 19961 raft_consensus.cc:515] T 7cb3768d215e409fb9f5baf18d124442 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:30.229946 20094 raft_consensus.cc:2749] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:21:30.245433 19779 raft_consensus.cc:515] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:30.266609 19978 raft_consensus.cc:515] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:30.281401 19779 leader_election.cc:290] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:30.256362 17688 raft_consensus.cc:1240] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 2. Current term is 3. Ops: []
I20260709 12:21:30.281710 19978 leader_election.cc:290] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:30.281968 19779 raft_consensus.cc:493] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:21:30.269502 19838 leader_election.cc:290] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:30.260102 17762 raft_consensus.cc:1275] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260709 12:21:30.282618 19779 raft_consensus.cc:515] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:30.283865 19757 raft_consensus.cc:3055] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:21:30.284668 19757 raft_consensus.cc:740] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:21:30.285605 20014 raft_consensus.cc:697] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:21:30.285511 19757 consensus_queue.cc:260] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:30.286947 19757 raft_consensus.cc:3060] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:30.286618 20014 consensus_queue.cc:237] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:21:30.287535 19976 consensus_meta.cc:287] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.560s	user 0.006s	sys 0.000s
I20260709 12:21:30.239373 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:30.288659 17755 raft_consensus.cc:2468] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 3.
I20260709 12:21:30.259841 18367 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59886 (request call id 736) took 2151 ms. Trace:
I20260709 12:21:30.289124 18367 rpcz_store.cc:276] 0709 12:21:28.108385 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:29.815161 (+1706776us) service_pool.cc:224] Handling call
0709 12:21:29.815530 (+   369us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:29.965525 (+149995us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:29.965557 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:29.966012 (+   455us) log.cc:844] Serialized 26 byte log entry
0709 12:21:30.109111 (+143099us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:21:30.109148 (+    37us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:30.109212 (+    64us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:30.259622 (+150410us) raft_consensus.cc:1685] finished
0709 12:21:30.259645 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:30.259772 (+   127us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1772,"raft.queue_time_us":253099,"raft.run_cpu_time_us":218,"raft.run_wall_time_us":28095,"thread_start_us":292344,"threads_started":2,"wal-append.queue_time_us":206030}
I20260709 12:21:30.256659 18333 leader_election.cc:304] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:30.248559 20088 raft_consensus.cc:515] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:30.256281 19855 leader_election.cc:290] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:30.226560 20059 raft_consensus.cc:493] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:30.263240 17756 raft_consensus.cc:2468] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 5.
I20260709 12:21:30.131012 20079 raft_consensus.cc:515] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:30.231026 19890 leader_election.cc:290] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 5 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:30.111590 20081 raft_consensus.cc:515] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:30.225780 20073 consensus_queue.cc:1059] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.028s
I20260709 12:21:30.231175 19991 leader_election.cc:290] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 6 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:30.128867 20065 raft_consensus.cc:515] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:30.291412 19974 leader_election.cc:290] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:30.300736 19893 raft_consensus.cc:3055] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:21:30.281762 19961 leader_election.cc:290] T 7cb3768d215e409fb9f5baf18d124442 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:30.238139 19848 leader_election.cc:290] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:30.252997 18361 raft_consensus.cc:2393] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 4: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:21:30.253333 20028 raft_consensus.cc:2804] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:21:30.270143 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:30.358268 17759 raft_consensus.cc:2468] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 1.
I20260709 12:21:30.298245 20015 leader_election.cc:290] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:30.267364 20076 raft_consensus.cc:493] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:30.365548 17685 raft_consensus.cc:2468] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 2.
I20260709 12:21:30.270141 20075 leader_election.cc:290] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:21:30.306155 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 49564.9us
I20260709 12:21:30.373283 18333 leader_election.cc:304] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:30.254391 20083 raft_consensus.cc:515] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:30.308099 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:21:30.312505 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:30.381803 17688 raft_consensus.cc:2393] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 5: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:21:30.248932 20092 raft_consensus.cc:515] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:30.346022 17837 raft_consensus.cc:2468] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 4.
I20260709 12:21:30.275385 20052 raft_consensus.cc:515] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:30.355075 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:30.382767 18367 raft_consensus.cc:2468] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 3.
W20260709 12:21:30.383129 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 402538us
I20260709 12:21:30.332592 17689 raft_consensus.cc:2468] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 3.
I20260709 12:21:30.331858 19893 raft_consensus.cc:740] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:21:30.317993 19855 raft_consensus.cc:2749] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260709 12:21:30.302469 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 341386us
I20260709 12:21:30.317415 20002 leader_election.cc:290] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:21:30.374702 17804 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:30.325847 19895 leader_election.cc:290] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:30.305542 19976 raft_consensus.cc:515] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:30.380662 18362 raft_consensus.cc:2468] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 2.
I20260709 12:21:30.253755 17729 leader_election.cc:304] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:21:30.394479 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 140814us
I20260709 12:21:30.305516 20059 raft_consensus.cc:3060] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:21:30.384282 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 252779us
I20260709 12:21:30.367379 20080 leader_election.cc:290] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:30.396894 17729 leader_election.cc:304] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:21:30.372818 20076 raft_consensus.cc:3060] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:30.357894 20028 raft_consensus.cc:493] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:30.384399 20073 raft_consensus.cc:2749] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:21:30.387755 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:30.365218 20099 raft_consensus.cc:493] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:30.390280 17654 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:30.367641 19804 leader_election.cc:290] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:21:30.396103 18333 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:30.395982 17803 leader_election.cc:304] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:21:30.396894 17801 leader_election.cc:304] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:30.363364 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:30.396883 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:30.401494 17729 leader_election.cc:304] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:21:30.407722 17654 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:30.413556 17836 raft_consensus.cc:2393] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 5: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:21:30.393296 20079 leader_election.cc:290] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:30.419847 17803 leader_election.cc:304] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:21:30.400472 19961 raft_consensus.cc:2804] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:21:30.408941 20028 raft_consensus.cc:3060] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:30.408260 19890 raft_consensus.cc:2749] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260709 12:21:30.425714 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:30.429701 17801 leader_election.cc:304] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:21:30.430897 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:30.431595 18374 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59886 (request call id 731) took 2611 ms. Trace:
I20260709 12:21:30.410882 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:30.431969 18374 rpcz_store.cc:276] 0709 12:21:27.819673 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:27.819791 (+   118us) service_pool.cc:224] Handling call
0709 12:21:27.820159 (+   368us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:28.098220 (+278061us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:30.431383 (+2333163us) spinlock_profiling.cc:243] Waited 77.7 ms on lock 0x7b5400802d54. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:21:30.431542 (+   159us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":178736512}
I20260709 12:21:30.433637 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:30.434612 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:30.435487 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:30.438342 17688 raft_consensus.cc:2393] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 6: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:21:30.403235 20093 raft_consensus.cc:493] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:21:30.420053 19998 leader_election.cc:290] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 4 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:21:30.444628 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 234674us
I20260709 12:21:30.444738 18333 leader_election.cc:304] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:30.421635 19961 raft_consensus.cc:493] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:30.446985 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:30.448241 17652 leader_election.cc:304] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:30.449427 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:30.429066 19806 leader_election.cc:290] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:30.435268 19848 raft_consensus.cc:2749] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:21:30.451611 18333 leader_election.cc:304] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:30.454224 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:30.450649 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:30.455276 17688 raft_consensus.cc:2393] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 4: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:21:30.434788 19808 raft_consensus.cc:493] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:30.416977 19994 raft_consensus.cc:2804] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:21:30.432283 19832 raft_consensus.cc:2749] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:21:30.450121 17759 raft_consensus.cc:2468] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 3.
I20260709 12:21:30.456074 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:30.438001 19978 raft_consensus.cc:2804] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:21:30.435977 19997 leader_election.cc:290] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:30.454648 17801 leader_election.cc:304] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:21:30.444382 20079 raft_consensus.cc:493] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:30.412885 20099 raft_consensus.cc:515] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:30.424181 19893 consensus_queue.cc:260] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:30.705672 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:21:30.960103 19757 consensus_meta.cc:287] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.666s	user 0.010s	sys 0.003s
I20260709 12:21:30.969977 18374 raft_consensus.cc:2393] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 5: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
W20260709 12:21:30.841158 20048 log.cc:927] Time spent T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 1.250s	user 0.004s	sys 0.007s
W20260709 12:21:30.865847 17654 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:30.971133 17654 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:30.971169 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:30.743093 20104 raft_consensus.cc:2749] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:21:30.461671 19978 raft_consensus.cc:493] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:30.898742 20065 leader_election.cc:290] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:30.768570 20109 raft_consensus.cc:2804] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260709 12:21:30.628094 17803 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:30.973320 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 518706us
I20260709 12:21:30.975469 19883 raft_consensus.cc:493] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
W20260709 12:21:30.994601 17833 consensus_meta.cc:287] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.810s	user 0.008s	sys 0.004s
I20260709 12:21:30.998695 17833 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 3.
W20260709 12:21:30.998929 20071 log.cc:927] Time spent T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 1.406s	user 0.002s	sys 0.000s
I20260709 12:21:30.453202 19895 raft_consensus.cc:2749] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:21:30.974249 20099 leader_election.cc:290] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:30.442759 18361 raft_consensus.cc:2468] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 1.
I20260709 12:21:30.450767 20093 raft_consensus.cc:515] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:21:30.811383 19753 consensus_meta.cc:287] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.579s	user 0.009s	sys 0.000s
I20260709 12:21:30.450492 20073 raft_consensus.cc:2749] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:21:30.788869 20083 leader_election.cc:290] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:30.641108 20108 consensus_queue.cc:1048] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.525s
I20260709 12:21:31.008164 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:30.645088 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:31.009243 18361 raft_consensus.cc:2410] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20260709 12:21:31.010013 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:31.010243 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:31.010761 17688 raft_consensus.cc:2393] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 2: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:21:30.975942 19757 raft_consensus.cc:493] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:31.011454 17686 raft_consensus.cc:2410] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 12:21:31.011768 19757 raft_consensus.cc:515] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:31.013590 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:31.013979 19757 leader_election.cc:290] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:31.014232 17686 raft_consensus.cc:2468] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 3.
I20260709 12:21:31.015236 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:30.642037 20105 consensus_queue.cc:1059] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.545s
I20260709 12:21:30.812438 20052 leader_election.cc:290] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:31.016359 20052 raft_consensus.cc:3055] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:21:31.011487 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:31.016860 20052 raft_consensus.cc:740] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 3 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:21:31.017354 17688 raft_consensus.cc:2468] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 5.
W20260709 12:21:30.789474 20106 log.cc:927] Time spent T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.334s	user 0.000s	sys 0.001s
I20260709 12:21:30.457747 19994 raft_consensus.cc:493] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:21:31.018723 20076 consensus_meta.cc:287] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.606s	user 0.001s	sys 0.007s
I20260709 12:21:31.019044 19994 raft_consensus.cc:3060] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:31.017861 20052 consensus_queue.cc:260] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:31.020535 20052 raft_consensus.cc:3060] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:21:31.021391 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:31.021239 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:21:31.021787 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:31.022380 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:31.022941 18363 raft_consensus.cc:3060] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:30.472164 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:30.788357 19855 raft_consensus.cc:493] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:31.024253 17759 raft_consensus.cc:2468] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 4.
W20260709 12:21:31.024672 20077 consensus_meta.cc:287] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.759s	user 0.011s	sys 0.000s
I20260709 12:21:31.024317 19855 raft_consensus.cc:515] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:31.025309 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:31.025979 17759 raft_consensus.cc:2393] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 4: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:21:31.026616 19855 leader_election.cc:290] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:31.026714 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:30.660326 17687 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 3.
W20260709 12:21:30.968513 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 516938us
W20260709 12:21:30.844025 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 198632us
I20260709 12:21:31.027707 20093 leader_election.cc:290] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:21:31.028187 18334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:31.029439 18333 leader_election.cc:304] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:21:31.030203 18333 leader_election.cc:304] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:21:31.030814 20108 consensus_queue.cc:1048] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 12:21:31.018474 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:31.032114 20108 raft_consensus.cc:2804] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:21:31.032176 17729 leader_election.cc:304] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:21:31.032485 17688 raft_consensus.cc:2393] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 5: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
W20260709 12:21:30.847070 20049 log.cc:927] Time spent T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 1.255s	user 0.001s	sys 0.002s
I20260709 12:21:30.835592 19976 leader_election.cc:290] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 5 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:30.451310 19961 raft_consensus.cc:3060] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:21:30.980000 17652 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:30.980088 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:31.038875 17652 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:30.986085 17761 raft_consensus.cc:1240] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 6. Current term is 7. Ops: []
I20260709 12:21:31.038924 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:31.039610 17761 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38776 (request call id 1684) took 2337 ms. Trace:
I20260709 12:21:31.040056 17761 rpcz_store.cc:276] 0709 12:21:28.702407 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:28.702549 (+   142us) service_pool.cc:224] Handling call
0709 12:21:30.985591 (+2283042us) spinlock_profiling.cc:243] Waited 9.82 ms on lock 0x7b5800033398. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b474fbb71 00007f7b474fba22 00007f7b504148d9 00007f7b4f56eee0 00007f7b4f6380ab 00007f7b4f638131 00007f7b4f5f9fc0 00007f7b4f5e4929 00007f7b4f5e43d2 00007f7b48b5bbd4
0709 12:21:30.985853 (+   262us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:31.039340 (+ 53487us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:31.039533 (+   193us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":22577536}
I20260709 12:21:31.040638 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:31.041222 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:31.041488 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:31.042021 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:31.042316 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:31.043147 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:31.044147 17830 raft_consensus.cc:2468] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 3.
I20260709 12:21:31.044353 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:31.044864 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:31.045377 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:31.045984 17764 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 6.
I20260709 12:21:31.004635 19893 raft_consensus.cc:3060] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:30.980623 20079 raft_consensus.cc:515] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:30.990036 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:30.474681 19808 raft_consensus.cc:515] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:21:30.994132 20057 consensus_meta.cc:287] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.794s	user 0.002s	sys 0.007s
I20260709 12:21:30.998402 19883 raft_consensus.cc:515] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:30.973413 20075 raft_consensus.cc:2804] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:21:31.055266 20075 raft_consensus.cc:697] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 6 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:21:31.056241 20075 consensus_queue.cc:237] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:31.056457 19895 consensus_queue.cc:1048] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
W20260709 12:21:31.058233 20028 consensus_meta.cc:287] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.622s	user 0.008s	sys 0.000s
I20260709 12:21:31.058328 18361 raft_consensus.cc:2468] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 5.
I20260709 12:21:31.058712 20028 raft_consensus.cc:515] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:31.059083 17838 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37994 (request call id 533) took 3524 ms. Trace:
I20260709 12:21:30.462446 20085 leader_election.cc:290] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:31.059541 17838 rpcz_store.cc:276] 0709 12:21:27.527792 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:27.534386 (+  6594us) service_pool.cc:224] Handling call
0709 12:21:28.216072 (+681686us) spinlock_profiling.cc:243] Waited 10.2 ms on lock 0x7b580013fe98. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b474fbb71 00007f7b474fba22 00007f7b504148d9 00007f7b4f56eee0 00007f7b4f6380ab 00007f7b4f638131 00007f7b4f5f9fc0 00007f7b4f5e4929 00007f7b4f5e43d2 00007f7b48b5bbd4
0709 12:21:28.216298 (+   226us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:28.296500 (+ 80202us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:28.296546 (+    46us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:28.296604 (+    58us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:21:28.296658 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:31.051757 (+2755099us) spinlock_profiling.cc:243] Waited 79.9 ms on lock 0x7b540008cfd4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:21:31.051803 (+    46us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:31.051939 (+   136us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":207287424}
I20260709 12:21:31.033497 19994 raft_consensus.cc:515] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:31.061352 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:31.062006 17838 raft_consensus.cc:2468] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 2.
I20260709 12:21:31.062318 19994 leader_election.cc:290] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 4 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:31.063050 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:31.065106 20079 leader_election.cc:290] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:31.068172 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:31.069240 18367 raft_consensus.cc:3060] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:31.069775 19883 leader_election.cc:290] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:31.070730 17686 raft_consensus.cc:2468] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 3.
I20260709 12:21:31.071944 19873 leader_election.cc:290] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:30.996137 19978 raft_consensus.cc:3060] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:31.007201 20074 leader_election.cc:290] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:31.025048 20077 raft_consensus.cc:515] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:21:31.067652 20059 consensus_meta.cc:287] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.660s	user 0.007s	sys 0.003s
I20260709 12:21:31.027375 17759 raft_consensus.cc:2468] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 3.
I20260709 12:21:30.833122 17831 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35040 (request call id 1862) took 1040 ms. Trace:
I20260709 12:21:30.628777 19890 consensus_queue.cc:1059] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.523s
I20260709 12:21:31.032619 20108 raft_consensus.cc:493] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:31.033046 20073 raft_consensus.cc:2804] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:21:31.033321 18333 leader_election.cc:304] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:21:31.032992 19921 raft_consensus.cc:2804] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:21:31.036338 17829 raft_consensus.cc:1275] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:21:31.090703 20077 leader_election.cc:290] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 4 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:31.091365 17829 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37994 (request call id 547) took 1271 ms. Trace:
I20260709 12:21:31.091751 17829 rpcz_store.cc:276] 0709 12:21:29.819851 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:29.822060 (+  2209us) service_pool.cc:224] Handling call
0709 12:21:31.035757 (+1213697us) spinlock_profiling.cc:243] Waited 3.49 ms on lock 0x7b580013fe98. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b474fbb71 00007f7b474fba22 00007f7b504148d9 00007f7b4f56eee0 00007f7b4f6380ab 00007f7b4f638131 00007f7b4f5f9fc0 00007f7b4f5e4929 00007f7b4f5e43d2 00007f7b48b5bbd4
0709 12:21:31.035992 (+   235us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:31.091036 (+ 55044us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:31.091278 (+   242us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":8024448}
I20260709 12:21:31.093286 20028 leader_election.cc:290] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 4 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:30.982776 18374 raft_consensus.cc:2393] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 5: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:21:31.100037 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:31.100833 18374 raft_consensus.cc:2468] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 1.
I20260709 12:21:31.102259 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:31.046705 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:31.103497 20061 raft_consensus.cc:3055] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:21:31.104241 20061 raft_consensus.cc:740] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 4 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:21:31.003093 20109 raft_consensus.cc:493] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:21:31.086885 20108 raft_consensus.cc:3060] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:31.105381 20109 raft_consensus.cc:3060] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:31.047762 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:31.089390 19921 raft_consensus.cc:493] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:21:31.107431 19921 raft_consensus.cc:3060] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:30.977785 19779 leader_election.cc:290] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:31.109057 17764 raft_consensus.cc:2468] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 3.
I20260709 12:21:31.110031 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:31.110167 18364 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 852) took 4114 ms. Trace:
I20260709 12:21:31.107780 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:31.110563 18364 rpcz_store.cc:276] 0709 12:21:26.995954 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:27.006428 (+ 10474us) service_pool.cc:224] Handling call
0709 12:21:27.006687 (+   259us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:29.972857 (+2966170us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:31.109929 (+1137072us) spinlock_profiling.cc:243] Waited 2.81 s on lock 0x7b540047bdd4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:21:31.110106 (+   177us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":6470745472}
I20260709 12:21:31.077847 17831 rpcz_store.cc:276] 0709 12:21:29.793100 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:29.793248 (+   148us) service_pool.cc:224] Handling call
0709 12:21:29.964837 (+171589us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 12:21:30.385875 (+421038us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:30.385895 (+    20us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 12:21:30.391481 (+  5586us) log.cc:844] Serialized 78 byte log entry
0709 12:21:30.449116 (+ 57635us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:21:30.450025 (+   909us) log.cc:844] Serialized 14 byte log entry
0709 12:21:30.478622 (+ 28597us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:30.828547 (+349925us) spinlock_profiling.cc:243] Waited 154 ms on lock 0x7b54003cc3d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:21:30.828611 (+    64us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:30.828684 (+    73us) raft_consensus.cc:1685] finished
0709 12:21:30.832800 (+  4116us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:30.833031 (+   231us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":38918,"prepare.queue_time_us":405264,"prepare.run_cpu_time_us":62,"prepare.run_wall_time_us":65,"raft.queue_time_us":158056,"raft.run_cpu_time_us":273,"raft.run_wall_time_us":27486,"spinlock_wait_cycles":354489088,"thread_start_us":316123,"threads_started":3,"wal-append.queue_time_us":80285}
I20260709 12:21:31.111779 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:31.099852 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:31.113268 18374 raft_consensus.cc:3060] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:31.112892 17759 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 3.
I20260709 12:21:31.109099 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:31.050422 18363 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 6.
W20260709 12:21:31.049830 17652 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:31.054152 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:31.081697 18367 raft_consensus.cc:2468] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 5.
I20260709 12:21:31.086297 17685 raft_consensus.cc:1240] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 3. Current term is 4. Ops: []
W20260709 12:21:31.087517 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 54263.7us
I20260709 12:21:31.088254 19873 raft_consensus.cc:2804] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:21:31.019156 20076 raft_consensus.cc:515] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:31.077209 20059 raft_consensus.cc:515] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:31.090520 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:31.098757 19808 leader_election.cc:290] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:31.114754 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:31.148461 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:31.147531 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:31.146751 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:31.186488 17761 raft_consensus.cc:2393] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 4: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:21:31.131032 17801 leader_election.cc:304] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:21:31.187458 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 56555.7us
I20260709 12:21:31.146322 17764 raft_consensus.cc:2393] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 4: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:21:31.176420 17681 raft_consensus.cc:2468] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 4.
I20260709 12:21:31.182696 18364 raft_consensus.cc:2376] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 3. Re-sending same reply.
I20260709 12:21:31.172199 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:31.183848 17756 raft_consensus.cc:2468] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 2.
I20260709 12:21:31.112771 17654 leader_election.cc:304] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
W20260709 12:21:31.191188 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 78498.8us
I20260709 12:21:31.186379 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:31.185705 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:31.183617 18333 leader_election.cc:304] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:21:31.190671 17830 raft_consensus.cc:2393] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 4: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:21:31.192281 18363 raft_consensus.cc:2468] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 3.
I20260709 12:21:31.194321 17830 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35042 (request call id 1632) took 1313 ms. Trace:
I20260709 12:21:31.193904 19868 raft_consensus.cc:2804] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:21:31.122077 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:31.194051 20077 raft_consensus.cc:493] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:21:31.198618 20077 raft_consensus.cc:515] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:31.199437 19883 raft_consensus.cc:2804] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:21:31.199851 19883 raft_consensus.cc:493] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:21:31.200240 19883 raft_consensus.cc:3060] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:31.200784 20077 leader_election.cc:290] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:31.146867 19961 raft_consensus.cc:515] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:31.202909 19961 leader_election.cc:290] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 6 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:31.197947 19868 raft_consensus.cc:493] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:31.204535 19868 raft_consensus.cc:3060] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:31.204437 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:31.205226 18364 raft_consensus.cc:2468] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 4.
I20260709 12:21:31.205459 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:31.205722 18364 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59918 (request call id 878) took 1032 ms. Trace:
I20260709 12:21:31.206193 18367 raft_consensus.cc:2393] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 4: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:21:31.206981 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:31.206032 18364 rpcz_store.cc:276] 0709 12:21:30.173566 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:31.204091 (+1030525us) service_pool.cc:224] Handling call
0709 12:21:31.205674 (+  1583us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:31.197979 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:31.208652 17838 raft_consensus.cc:2468] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 5.
I20260709 12:21:31.089748 20073 raft_consensus.cc:493] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:31.194810 17830 rpcz_store.cc:276] 0709 12:21:29.880933 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:31.053827 (+1172894us) service_pool.cc:224] Handling call
0709 12:21:31.194266 (+140439us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:31.213440 17803 leader_election.cc:304] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:31.214183 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:21:31.214725 18333 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:31.195766 17652 leader_election.cc:304] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:21:31.207727 18367 raft_consensus.cc:2410] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 5 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260709 12:21:31.218310 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:31.219084 17688 raft_consensus.cc:2393] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 6: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:21:31.221750 17684 raft_consensus.cc:1240] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 3. Current term is 4. Ops: []
I20260709 12:21:31.169912 20088 leader_election.cc:290] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:31.169021 20078 leader_election.cc:290] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:31.224689 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:31.230796 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:31.231936 17760 raft_consensus.cc:1240] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 LEADER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 4. Current term is 5. Ops: [4.4-4.4]
I20260709 12:21:31.233086 17761 raft_consensus.cc:2468] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 4.
I20260709 12:21:31.189640 19873 raft_consensus.cc:493] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:21:31.219182 20073 raft_consensus.cc:3060] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:31.231247 18333 leader_election.cc:304] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:21:31.220827 20090 raft_consensus.cc:2804] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:21:31.217718 20052 raft_consensus.cc:2749] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:21:31.248803 18374 raft_consensus.cc:2468] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 5.
I20260709 12:21:31.236603 19873 raft_consensus.cc:3060] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:31.228339 20077 raft_consensus.cc:2804] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:21:31.174290 20061 consensus_queue.cc:260] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:31.203689 19978 raft_consensus.cc:515] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:31.239655 17829 raft_consensus.cc:2393] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 6: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:21:31.240257 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:31.242710 19921 raft_consensus.cc:515] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:31.238191 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:21:31.246472 20110 log.cc:927] Time spent T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.486s	user 0.008s	sys 0.008s
I20260709 12:21:31.246938 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:31.219506 20093 raft_consensus.cc:493] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:31.235694 17652 leader_election.cc:304] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:21:31.248163 20090 raft_consensus.cc:493] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:31.238214 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:31.273842 17831 raft_consensus.cc:2393] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 5: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:21:31.239305 20078 raft_consensus.cc:2749] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:21:31.250022 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:31.272431 17652 leader_election.cc:304] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:21:31.252213 20077 raft_consensus.cc:493] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:21:31.274782 19895 raft_consensus.cc:2804] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:21:31.275652 20090 raft_consensus.cc:3060] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:31.255447 18333 leader_election.cc:304] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:31.256659 19971 raft_consensus.cc:493] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:31.270360 17838 raft_consensus.cc:2468] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 3.
I20260709 12:21:31.220613 20079 raft_consensus.cc:493] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:31.271014 17688 raft_consensus.cc:2468] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 7.
I20260709 12:21:31.271037 18367 raft_consensus.cc:2468] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 3.
I20260709 12:21:31.271759 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:30.137619 18368 rpcz_store.cc:276] 0709 12:21:28.048274 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:29.704480 (+1656206us) service_pool.cc:224] Handling call
0709 12:21:29.704773 (+   293us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:29.704968 (+   195us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:21:29.791028 (+ 86060us) log.cc:844] Serialized 14 byte log entry
0709 12:21:30.108457 (+317429us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:30.108516 (+    59us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:21:30.108566 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:30.108633 (+    67us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:30.108775 (+   142us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":78638,"prepare.run_cpu_time_us":52,"prepare.run_wall_time_us":54,"raft.queue_time_us":173913,"thread_start_us":387606,"threads_started":3,"wal-append.queue_time_us":174706}
I20260709 12:21:31.280838 17838 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35042 (request call id 1636) took 1398 ms. Trace:
I20260709 12:21:31.278019 20077 raft_consensus.cc:3060] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:31.219276 19893 consensus_queue.cc:1048] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.034s
I20260709 12:21:31.280962 18366 raft_consensus.cc:1240] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 2. Current term is 3. Ops: []
W20260709 12:21:31.280861 18371 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59886 (request call id 727) took 4035 ms (client timeout 2036 ms). Trace:
I20260709 12:21:31.281816 18333 leader_election.cc:304] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:31.281921 17652 leader_election.cc:304] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:31.283489 18367 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59918 (request call id 874) took 1116 ms. Trace:
I20260709 12:21:31.287164 17729 leader_election.cc:304] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:21:31.283668 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:31.283668 17838 rpcz_store.cc:276] 0709 12:21:29.888739 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:31.233582 (+1344843us) service_pool.cc:224] Handling call
0709 12:21:31.280782 (+ 47200us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:31.285698 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:31.318626 18333 leader_election.cc:304] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:21:31.318815 18363 raft_consensus.cc:2468] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 3.
I20260709 12:21:31.278060 19895 raft_consensus.cc:493] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:31.320034 19757 raft_consensus.cc:2749] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:21:31.323302 20088 raft_consensus.cc:2804] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:21:31.323738 20088 raft_consensus.cc:493] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:21:31.323935 20076 leader_election.cc:290] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 4 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:31.243724 19883 raft_consensus.cc:515] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:31.319193 17652 leader_election.cc:304] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:21:31.321339 18368 raft_consensus.cc:2393] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 3: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:21:31.317585 18367 rpcz_store.cc:276] 0709 12:21:30.167238 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:31.226434 (+1059196us) service_pool.cc:224] Handling call
0709 12:21:31.283435 (+ 57001us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:31.319555 19917 raft_consensus.cc:2804] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:21:31.324456 19855 raft_consensus.cc:2804] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260709 12:21:31.322266 18371 rpcz_store.cc:269] 0709 12:21:27.245052 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:27.245194 (+   142us) service_pool.cc:224] Handling call
0709 12:21:31.280657 (+4035463us) spinlock_profiling.cc:243] Waited 650 ms on lock 0x7b54006e23d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:21:31.280793 (+   136us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1496092928}
I20260709 12:21:31.320721 18363 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59902 (request call id 750) took 1063 ms. Trace:
I20260709 12:21:31.324218 20088 raft_consensus.cc:3060] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:31.325819 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:31.287348 20061 raft_consensus.cc:3060] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:31.330402 17688 raft_consensus.cc:3060] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:31.328931 19917 raft_consensus.cc:697] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:21:31.331691 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:31.332257 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:31.330926 18363 rpcz_store.cc:276] 0709 12:21:30.255463 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:31.285429 (+1029966us) service_pool.cc:224] Handling call
0709 12:21:31.319247 (+ 33818us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:31.334182 19921 leader_election.cc:290] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 2 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:31.284138 20112 raft_consensus.cc:2804] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:21:31.328835 20076 raft_consensus.cc:2749] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:21:31.335925 19883 leader_election.cc:290] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 2 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:31.323308 19895 raft_consensus.cc:3060] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:31.336143 18362 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59918 (request call id 869) took 1226 ms. Trace:
I20260709 12:21:31.352725 18362 rpcz_store.cc:276] 0709 12:21:30.109973 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:30.110577 (+   604us) service_pool.cc:224] Handling call
0709 12:21:31.335987 (+1225410us) spinlock_profiling.cc:243] Waited 93 ms on lock 0x7b540069e1d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:21:31.336088 (+   101us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":213836672}
I20260709 12:21:31.354235 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:21:31.354534 17728 connection.cc:256] RPC call timeout handler was delayed by -1.7360864649999712s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1589s for a response
I20260709 12:21:31.355072 18362 raft_consensus.cc:2468] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 6.
I20260709 12:21:31.336541 17835 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35040 (request call id 1859) took 1545 ms. Trace:
I20260709 12:21:31.355633 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:31.356424 18367 raft_consensus.cc:3060] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:31.338408 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:31.357739 18365 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59886 (request call id 742) took 1181 ms. Trace:
I20260709 12:21:31.358235 18365 rpcz_store.cc:276] 0709 12:21:30.176704 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:31.338157 (+1161453us) service_pool.cc:224] Handling call
0709 12:21:31.357658 (+ 19501us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:31.340142 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:31.304518 20079 raft_consensus.cc:515] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:31.283921 19779 raft_consensus.cc:2804] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:21:31.274783 20093 raft_consensus.cc:515] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:31.304447 20078 raft_consensus.cc:493] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:31.328064 17652 leader_election.cc:304] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:31.342862 18333 leader_election.cc:304] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:21:31.342948 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:31.365598 18363 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 3.
I20260709 12:21:31.366143 17803 leader_election.cc:304] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:21:31.366490 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:31.367424 20075 raft_consensus.cc:2804] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:21:31.349152 17762 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36572 (request call id 463) took 1238 ms. Trace:
I20260709 12:21:31.328982 19855 raft_consensus.cc:493] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:21:31.368160 20075 raft_consensus.cc:493] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:31.368528 19855 raft_consensus.cc:3060] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:31.368706 20075 raft_consensus.cc:3060] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:31.348049 20112 raft_consensus.cc:493] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:31.369628 20112 raft_consensus.cc:3060] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:31.339622 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:31.372244 17803 leader_election.cc:304] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:31.368251 19883 raft_consensus.cc:2804] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:21:31.372833 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:31.373108 20052 raft_consensus.cc:2804] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:21:31.373533 20052 raft_consensus.cc:493] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:21:31.373929 20052 raft_consensus.cc:3060] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:31.374135 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:31.375037 17755 raft_consensus.cc:2393] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 4: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:21:31.373539 17803 leader_election.cc:304] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:21:31.356076 17835 rpcz_store.cc:276] 0709 12:21:29.791389 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:29.791504 (+   115us) service_pool.cc:224] Handling call
0709 12:21:29.813764 (+ 22260us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:30.128488 (+314724us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:31.336278 (+1207790us) spinlock_profiling.cc:243] Waited 314 ms on lock 0x7b54002eb654. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:21:31.336471 (+   193us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":722147584}
I20260709 12:21:31.378341 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:31.378938 17831 raft_consensus.cc:2468] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 4.
I20260709 12:21:31.362531 17728 leader_election.cc:304] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:21:31.284781 19971 raft_consensus.cc:515] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:31.367532 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:31.385738 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:31.386428 17835 raft_consensus.cc:2468] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 4.
I20260709 12:21:31.386862 20079 leader_election.cc:290] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:31.387159 19971 leader_election.cc:290] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:31.387501 18374 raft_consensus.cc:2468] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 3.
I20260709 12:21:31.360739 17761 raft_consensus.cc:3060] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:31.388253 20099 raft_consensus.cc:2804] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:21:31.343918 20027 raft_consensus.cc:493] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:21:31.383419 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:31.386008 18363 raft_consensus.cc:2410] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20260709 12:21:31.398782 18362 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 6.
I20260709 12:21:31.399672 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:31.400435 18363 raft_consensus.cc:2376] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 5. Re-sending same reply.
I20260709 12:21:31.401523 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:31.334435 20019 consensus_queue.cc:1048] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:21:31.368206 17762 rpcz_store.cc:276] 0709 12:21:30.110458 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:30.283206 (+172748us) service_pool.cc:224] Handling call
0709 12:21:30.283545 (+   339us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:30.283971 (+   426us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:30.284002 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:30.284467 (+   465us) log.cc:844] Serialized 26 byte log entry
0709 12:21:30.407414 (+122947us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:21:30.407455 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:30.407523 (+    68us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:31.245916 (+838393us) spinlock_profiling.cc:243] Waited 7.04 ms on lock 0x7b4400022b4c. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b47459c85 00007f7b4745a6a2 00007f7b4a1cc1db 00007f7b4a1c7aaf 00007f7b4a1bb86e 00007f7b4a1babb6 00007f7b4a1ba7d8 00007f7b4a1c4221 00007f7b4a1c41d9 00007f7b4a1c4169
0709 12:21:31.344049 (+ 98133us) raft_consensus.cc:1685] finished
0709 12:21:31.344069 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:31.349079 (+  5010us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":171,"raft.run_cpu_time_us":94,"raft.run_wall_time_us":94,"spinlock_wait_cycles":16182528,"thread_start_us":122585,"threads_started":1,"wal-append.queue_time_us":342562}
I20260709 12:21:31.387387 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:31.403920 17803 leader_election.cc:304] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:31.404696 17763 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38786 (request call id 2346) took 3227 ms. Trace:
I20260709 12:21:31.405002 20057 raft_consensus.cc:2749] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:21:31.366030 19779 raft_consensus.cc:493] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:31.407675 18367 raft_consensus.cc:2468] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 5.
I20260709 12:21:31.377604 20093 leader_election.cc:290] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:31.408389 18367 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59918 (request call id 875) took 1235 ms. Trace:
I20260709 12:21:31.408502 17682 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37402 (request call id 2175) took 1691 ms. Trace:
I20260709 12:21:31.389124 20085 raft_consensus.cc:2804] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:21:31.409056 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:31.388715 20099 raft_consensus.cc:493] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:31.372876 19883 raft_consensus.cc:493] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:21:31.381615 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:31.442117 18366 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 860) took 3999 ms. Trace:
I20260709 12:21:31.442605 17755 raft_consensus.cc:2468] T 7cb3768d215e409fb9f5baf18d124442 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 3.
I20260709 12:21:31.442530 18366 rpcz_store.cc:276] 0709 12:21:27.442443 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:27.444613 (+  2170us) service_pool.cc:224] Handling call
0709 12:21:31.280779 (+3836166us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:31.317626 (+ 36847us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:31.441906 (+124280us) spinlock_profiling.cc:243] Waited 3.79 s on lock 0x7b54006e23d0. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a61f8ad 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:21:31.442051 (+   145us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":8716927872}
I20260709 12:21:31.443815 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:31.444414 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:31.444962 17684 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44898 (request call id 510) took 1029 ms. Trace:
I20260709 12:21:31.445199 18366 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59902 (request call id 739) took 1228 ms. Trace:
I20260709 12:21:31.409963 17803 leader_election.cc:304] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:21:31.445650 18366 rpcz_store.cc:276] 0709 12:21:30.217269 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:31.443670 (+1226401us) service_pool.cc:224] Handling call
0709 12:21:31.445145 (+  1475us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:31.410548 18367 rpcz_store.cc:276] 0709 12:21:30.172664 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:31.342402 (+1169738us) service_pool.cc:224] Handling call
0709 12:21:31.408325 (+ 65923us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:31.446678 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:31.447321 18367 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59886 (request call id 741) took 1271 ms. Trace:
I20260709 12:21:31.447649 18367 rpcz_store.cc:276] 0709 12:21:30.176409 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:31.446434 (+1270025us) service_pool.cc:224] Handling call
0709 12:21:31.447269 (+   835us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:31.448449 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:31.448945 18367 raft_consensus.cc:2468] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 3.
I20260709 12:21:31.449344 18367 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59902 (request call id 748) took 1194 ms. Trace:
I20260709 12:21:31.449611 18367 rpcz_store.cc:276] 0709 12:21:30.254890 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:31.448226 (+1193336us) service_pool.cc:224] Handling call
0709 12:21:31.449304 (+  1078us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:31.450359 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:31.450676 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:31.451056 18367 raft_consensus.cc:2410] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20260709 12:21:31.452056 19848 raft_consensus.cc:2804] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:21:31.452571 19848 raft_consensus.cc:697] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 5 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:21:31.445381 17684 rpcz_store.cc:276] 0709 12:21:30.415572 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:30.415720 (+   148us) service_pool.cc:224] Handling call
0709 12:21:30.425430 (+  9710us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:31.222799 (+797369us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:31.444767 (+221968us) spinlock_profiling.cc:243] Waited 796 ms on lock 0x7b54003462d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:21:31.444910 (+   143us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1830905600}
I20260709 12:21:31.455987 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:31.410442 17682 rpcz_store.cc:276] 0709 12:21:29.716584 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:29.717493 (+   909us) service_pool.cc:224] Handling call
0709 12:21:29.717737 (+   244us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:30.131177 (+413440us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:30.131210 (+    33us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:30.131254 (+    44us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:21:30.131290 (+    36us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:31.408291 (+1277001us) spinlock_profiling.cc:243] Waited 413 ms on lock 0x7b5400207e54. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:21:31.408339 (+    48us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:31.408452 (+   113us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":950355328}
I20260709 12:21:31.409881 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:31.405045 17763 rpcz_store.cc:276] 0709 12:21:28.177357 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:28.177489 (+   132us) service_pool.cc:224] Handling call
0709 12:21:31.343856 (+3166367us) spinlock_profiling.cc:243] Waited 2.7 ms on lock 0x7b5800033398. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b474fbb71 00007f7b474fba22 00007f7b504148d9 00007f7b4f56eee0 00007f7b4f6380ab 00007f7b4f638131 00007f7b4f5f9fc0 00007f7b4f5e4929 00007f7b4f5e43d2 00007f7b48b5bbd4
0709 12:21:31.344071 (+   215us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:31.344294 (+   223us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 12:21:31.366658 (+ 22364us) log.cc:844] Serialized 14 byte log entry
0709 12:21:31.404399 (+ 37741us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:31.404454 (+    55us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:21:31.404496 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:31.404555 (+    59us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:31.404647 (+    92us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":22491,"prepare.run_cpu_time_us":29,"prepare.run_wall_time_us":30,"raft.queue_time_us":198,"raft.run_cpu_time_us":80,"raft.run_wall_time_us":80,"spinlock_wait_cycles":6229760,"thread_start_us":54190,"threads_started":2}
I20260709 12:21:31.400439 19808 raft_consensus.cc:2749] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:21:31.408536 19779 raft_consensus.cc:3060] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:31.413694 20085 raft_consensus.cc:493] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:31.420037 19807 raft_consensus.cc:3060] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:31.504323 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:31.421262 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:31.505473 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:31.506115 17686 raft_consensus.cc:2468] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 4.
I20260709 12:21:31.506919 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:31.507624 17686 raft_consensus.cc:2393] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 6: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:21:31.423666 19883 raft_consensus.cc:3060] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:31.426733 17758 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38776 (request call id 1681) took 2846 ms. Trace:
I20260709 12:21:31.508190 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:31.508203 17758 rpcz_store.cc:276] 0709 12:21:28.580518 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:28.582408 (+  1890us) service_pool.cc:224] Handling call
0709 12:21:28.606816 (+ 24408us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:28.866214 (+259398us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:28.866257 (+    43us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:28.866313 (+    56us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:21:28.866366 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:31.426498 (+2560132us) spinlock_profiling.cc:243] Waited 259 ms on lock 0x7b54003e5dd4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:21:31.426541 (+    43us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:31.426674 (+   133us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":595943168}
I20260709 12:21:31.509505 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:31.510156 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:31.510612 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:31.403576 19917 consensus_queue.cc:237] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:31.381037 19873 raft_consensus.cc:515] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:31.430840 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:31.431794 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:31.432055 20081 leader_election.cc:290] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:31.526576 20084 raft_consensus.cc:3055] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:21:31.526981 20084 raft_consensus.cc:740] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:21:31.503427 20073 raft_consensus.cc:515] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:31.433298 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:31.389228 20078 raft_consensus.cc:515] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:31.433869 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:31.436321 20099 raft_consensus.cc:3060] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:31.402854 20027 raft_consensus.cc:515] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:31.440824 17654 leader_election.cc:304] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:21:31.401762 20059 leader_election.cc:290] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 6 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:31.441084 19753 consensus_queue.cc:1048] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 12:21:31.411603 17831 raft_consensus.cc:2468] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 5.
I20260709 12:21:31.458019 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:31.458835 17764 raft_consensus.cc:3060] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:31.451819 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
W20260709 12:21:31.489734 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 79853.7us
I20260709 12:21:31.532312 18363 raft_consensus.cc:2393] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 4: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
W20260709 12:21:31.534655 17836 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35042 (request call id 1634) took 1653 ms (client timeout 2070 ms). Trace:
I20260709 12:21:31.511646 17829 raft_consensus.cc:2468] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 4.
I20260709 12:21:31.579386 18367 raft_consensus.cc:3060] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:31.579083 17835 raft_consensus.cc:2468] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 3.
I20260709 12:21:31.453685 19848 consensus_queue.cc:237] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.7, Last appended by leader: 7, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:31.498270 19978 leader_election.cc:290] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 6 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:31.498579 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:31.499130 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:31.444957 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:31.595494 17755 raft_consensus.cc:2393] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 6: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:21:31.500349 20085 raft_consensus.cc:3060] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:31.596457 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:31.585589 20019 raft_consensus.cc:493] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:21:31.420885 20052 raft_consensus.cc:515] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:31.597760 20088 raft_consensus.cc:515] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:31.505475 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:31.527835 20084 consensus_queue.cc:260] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:31.497064 20108 raft_consensus.cc:515] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:31.519032 17688 raft_consensus.cc:2468] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 5.
I20260709 12:21:31.529372 17689 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37402 (request call id 2177) took 1473 ms. Trace:
W20260709 12:21:31.580776 17803 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:31.528174 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:21:31.492620 17655 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:31.563105 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:21:31.545557 18334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:31.573271 17758 raft_consensus.cc:2468] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 5.
I20260709 12:21:31.588768 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:31.592873 17761 raft_consensus.cc:2468] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 5.
I20260709 12:21:31.598062 20019 raft_consensus.cc:515] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:31.753360 18364 raft_consensus.cc:2393] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 6: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:21:31.754346 20108 leader_election.cc:290] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 6 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:31.755317 17761 raft_consensus.cc:1275] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 12:21:31.755940 17762 raft_consensus.cc:1275] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
W20260709 12:21:31.579031 17836 rpcz_store.cc:269] 0709 12:21:29.881526 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:30.131823 (+250297us) service_pool.cc:224] Handling call
0709 12:21:31.534463 (+1402640us) spinlock_profiling.cc:243] Waited 205 ms on lock 0x7b5400267354. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:21:31.534593 (+   130us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":471346048}
I20260709 12:21:31.716672 20077 raft_consensus.cc:515] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:31.762732 17761 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38776 (request call id 1690) took 1303 ms. Trace:
I20260709 12:21:31.787290 17686 raft_consensus.cc:2468] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 5.
I20260709 12:21:31.787472 17761 rpcz_store.cc:276] 0709 12:21:30.459279 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:31.752874 (+1293595us) service_pool.cc:224] Handling call
0709 12:21:31.754905 (+  2031us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:31.755805 (+   900us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:31.762659 (+  6854us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:31.680184 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:31.788748 17762 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38776 (request call id 1691) took 1329 ms. Trace:
I20260709 12:21:31.789149 17831 raft_consensus.cc:2393] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 6: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:21:31.785295 18367 raft_consensus.cc:2468] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 4.
I20260709 12:21:31.673372 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:21:31.763751 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 322984us
I20260709 12:21:31.789304 17762 rpcz_store.cc:276] 0709 12:21:30.459529 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:31.755389 (+1295860us) service_pool.cc:224] Handling call
0709 12:21:31.755653 (+   264us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:31.756364 (+   711us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:31.788674 (+ 32310us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:31.792264 17654 leader_election.cc:304] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:31.791113 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:31.794251 17803 leader_election.cc:304] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:21:31.763603 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:31.795222 17803 leader_election.cc:304] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:31.795620 17763 raft_consensus.cc:2468] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 7.
W20260709 12:21:31.740255 20109 consensus_meta.cc:287] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.634s	user 0.009s	sys 0.004s
I20260709 12:21:31.536172 19961 raft_consensus.cc:493] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:31.796726 17764 raft_consensus.cc:2468] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 4.
I20260709 12:21:31.810451 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:31.800309 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:31.789479 17689 rpcz_store.cc:276] 0709 12:21:30.055561 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:30.055743 (+   182us) service_pool.cc:224] Handling call
0709 12:21:31.529227 (+1473484us) spinlock_profiling.cc:243] Waited 190 ms on lock 0x7b54003507d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:21:31.529319 (+    92us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":437490944}
I20260709 12:21:31.818084 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
W20260709 12:21:31.627241 17652 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:31.855105 17652 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:31.853799 17836 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37994 (request call id 548) took 1924 ms. Trace:
I20260709 12:21:31.737414 20078 leader_election.cc:290] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:31.855959 17836 rpcz_store.cc:276] 0709 12:21:29.929055 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:31.800037 (+1870982us) service_pool.cc:224] Handling call
0709 12:21:31.853745 (+ 53708us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:31.865608 17686 raft_consensus.cc:1240] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 3. Current term is 4. Ops: []
I20260709 12:21:31.866166 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:31.537550 19883 raft_consensus.cc:515] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:31.870791 17681 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37402 (request call id 2187) took 1494 ms. Trace:
I20260709 12:21:31.871196 17681 rpcz_store.cc:276] 0709 12:21:30.376880 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:30.377067 (+   187us) service_pool.cc:224] Handling call
0709 12:21:31.870636 (+1493569us) spinlock_profiling.cc:243] Waited 707 ms on lock 0x7b54001388d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:21:31.870741 (+   105us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1625571584}
I20260709 12:21:31.631794 19757 raft_consensus.cc:2804] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:21:31.528848 18361 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59918 (request call id 882) took 1120 ms. Trace:
I20260709 12:21:31.616391 19893 raft_consensus.cc:493] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:21:31.808163 20059 raft_consensus.cc:2804] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:21:31.874302 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:31.875424 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:31.599769 20052 leader_election.cc:290] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 7 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:31.540205 20092 leader_election.cc:290] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:31.876484 18363 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 872) took 1755 ms. Trace:
I20260709 12:21:31.877298 18366 raft_consensus.cc:1275] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260709 12:21:31.894887 18363 rpcz_store.cc:276] 0709 12:21:30.120943 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:31.854907 (+1733964us) service_pool.cc:224] Handling call
0709 12:21:31.875958 (+ 21051us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:31.876151 (+   193us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:31.876174 (+    23us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:31.876214 (+    40us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:21:31.876257 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:31.876314 (+    57us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:31.876419 (+   105us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:31.877724 17759 raft_consensus.cc:1240] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20260709 12:21:31.799890 20084 raft_consensus.cc:3060] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:31.616132 20081 raft_consensus.cc:493] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:21:31.864866 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:31.615585 20112 raft_consensus.cc:515] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:31.869235 17831 raft_consensus.cc:3060] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:31.808308 20074 raft_consensus.cc:2749] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:21:31.882712 18362 raft_consensus.cc:3060] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:31.788177 19868 raft_consensus.cc:515] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:31.889144 17760 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38776 (request call id 1679) took 3590 ms. Trace:
I20260709 12:21:31.889876 19873 leader_election.cc:290] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 2 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:31.891264 19883 leader_election.cc:290] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 3 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:31.807307 19779 raft_consensus.cc:515] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:31.891402 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:31.891863 19757 raft_consensus.cc:697] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 3 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:21:31.895350 18366 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 873) took 1743 ms. Trace:
I20260709 12:21:31.915818 17834 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35040 (request call id 1860) took 2124 ms. Trace:
I20260709 12:21:31.916253 18366 rpcz_store.cc:276] 0709 12:21:30.152103 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:31.864936 (+1712833us) service_pool.cc:224] Handling call
0709 12:21:31.876902 (+ 11966us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:31.895105 (+ 18203us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:31.895302 (+   197us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:31.807988 19753 raft_consensus.cc:2804] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:21:31.536759 19855 raft_consensus.cc:515] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:31.906958 17683 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37396 (request call id 1767) took 4061 ms. Trace:
I20260709 12:21:31.916383 17834 rpcz_store.cc:276] 0709 12:21:29.791826 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:29.792001 (+   175us) service_pool.cc:224] Handling call
0709 12:21:29.792320 (+   319us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:30.075858 (+283538us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:30.075924 (+    66us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:30.075985 (+    61us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:21:30.076040 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:31.915504 (+1839464us) spinlock_profiling.cc:243] Waited 283 ms on lock 0x7b54000898d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:21:31.915557 (+    53us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:31.915742 (+   185us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":651405056}
W20260709 12:21:31.907347 17837 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35040 (request call id 1857) took 2110 ms (client timeout 2153 ms). Trace:
W20260709 12:21:31.907795 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:31.883499 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:31.933260 18365 raft_consensus.cc:3060] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:31.911180 17760 rpcz_store.cc:276] 0709 12:21:28.299075 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:28.299233 (+   158us) service_pool.cc:224] Handling call
0709 12:21:28.299620 (+   387us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:31.233421 (+2933801us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:31.888796 (+655375us) spinlock_profiling.cc:243] Waited 2.93 s on lock 0x7b5400216e54. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:21:31.889069 (+   273us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":6743109504}
I20260709 12:21:31.886785 20059 raft_consensus.cc:493] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:31.942147 17685 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44898 (request call id 516) took 1484 ms. Trace:
I20260709 12:21:31.856240 20109 raft_consensus.cc:515] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:31.918403 18374 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59902 (request call id 744) took 1699 ms. Trace:
I20260709 12:21:31.939538 20027 leader_election.cc:290] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:31.929658 17836 raft_consensus.cc:2393] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 5: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:21:31.932744 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:31.935869 17684 raft_consensus.cc:1240] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 3. Current term is 4. Ops: []
W20260709 12:21:31.914490 20075 consensus_meta.cc:287] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.545s	user 0.010s	sys 0.001s
I20260709 12:21:31.938319 20019 leader_election.cc:290] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:31.930724 17683 rpcz_store.cc:276] 0709 12:21:27.837433 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:27.994231 (+156798us) service_pool.cc:224] Handling call
0709 12:21:27.994651 (+   420us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:28.081035 (+ 86384us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:21:28.100680 (+ 19645us) log.cc:844] Serialized 14 byte log entry
0709 12:21:29.712057 (+1611377us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:29.712116 (+    59us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:21:29.712164 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:30.257479 (+545315us) spinlock_profiling.cc:243] Waited 2.56 ms on lock 0x7b1800004900. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b475012a6 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b47459c85 00007f7b4745a6a2 00007f7b4a1cc1db
0709 12:21:30.788038 (+530559us) spinlock_profiling.cc:243] Waited 41.7 ms on lock 0x7b4400022b4c. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b47459c85 00007f7b4745a6a2 00007f7b4a1cc1db 00007f7b4a1c7aaf 00007f7b4a1bb86e 00007f7b4a1babb6 00007f7b4a1ba7d8 00007f7b4a1c4221 00007f7b4a1c41d9 00007f7b4a1c4169
0709 12:21:31.898667 (+1110629us) spinlock_profiling.cc:243] Waited 86.1 ms on lock 0x7b54003480d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:21:31.898716 (+    49us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:31.898860 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":78363,"prepare.queue_time_us":14561,"prepare.run_cpu_time_us":70,"prepare.run_wall_time_us":2595,"raft.queue_time_us":62280,"raft.run_cpu_time_us":148,"raft.run_wall_time_us":34149,"spinlock_wait_cycles":299838848,"thread_start_us":1549035,"threads_started":2,"wal-append.queue_time_us":1490276}
W20260709 12:21:31.932209 17837 rpcz_store.cc:269] 0709 12:21:29.790078 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:29.790202 (+   124us) service_pool.cc:224] Handling call
0709 12:21:31.899473 (+2109271us) spinlock_profiling.cc:243] Waited 244 ms on lock 0x7b54002eac54. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:21:31.899567 (+    94us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":560471936}
I20260709 12:21:31.978514 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:21:31.909365 17833 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37994 (request call id 546) took 2139 ms (client timeout 1865 ms). Trace:
I20260709 12:21:31.878665 18361 rpcz_store.cc:276] 0709 12:21:30.408081 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:31.009955 (+601874us) service_pool.cc:224] Handling call
0709 12:21:31.528650 (+518695us) spinlock_profiling.cc:243] Waited 42.6 ms on lock 0x7b54006144d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:21:31.528786 (+   136us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":97875328}
W20260709 12:21:31.910488 20090 consensus_meta.cc:287] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.632s	user 0.009s	sys 0.004s
I20260709 12:21:31.939975 18366 raft_consensus.cc:1240] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 6. Current term is 7. Ops: []
I20260709 12:21:31.983206 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:31.983417 18366 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 877) took 1810 ms. Trace:
I20260709 12:21:31.983817 17837 raft_consensus.cc:2468] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 5.
I20260709 12:21:31.940738 19757 consensus_queue.cc:237] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:31.947242 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:31.867549 19961 raft_consensus.cc:515] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:31.930222 19753 raft_consensus.cc:493] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:31.977114 18374 rpcz_store.cc:276] 0709 12:21:30.218915 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:31.249789 (+1030874us) service_pool.cc:224] Handling call
0709 12:21:31.918189 (+668400us) spinlock_profiling.cc:243] Waited 113 ms on lock 0x7b5400672d54. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:21:31.918327 (+   138us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":258799872}
I20260709 12:21:31.942731 17685 rpcz_store.cc:276] 0709 12:21:30.457939 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:31.042367 (+584428us) service_pool.cc:224] Handling call
0709 12:21:31.042647 (+   280us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:31.184846 (+142199us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:31.941791 (+756945us) spinlock_profiling.cc:243] Waited 36.5 ms on lock 0x7b5400425ed4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:21:31.942061 (+   270us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":84006144}
I20260709 12:21:31.979312 17836 raft_consensus.cc:2393] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 3: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:21:31.907820 19806 consensus_queue.cc:1048] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.032s
I20260709 12:21:31.981067 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:31.941352 19779 leader_election.cc:290] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 4 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:31.989056 17838 raft_consensus.cc:2468] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 3.
I20260709 12:21:31.989401 17830 raft_consensus.cc:3060] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:31.990130 18374 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59902 (request call id 727) took 1775 ms. Trace:
I20260709 12:21:31.990758 17757 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38786 (request call id 2342) took 3844 ms. Trace:
I20260709 12:21:31.990661 18374 rpcz_store.cc:276] 0709 12:21:30.214293 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:31.988883 (+1774590us) service_pool.cc:224] Handling call
0709 12:21:31.989304 (+   421us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:31.989494 (+   190us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:31.989518 (+    24us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:31.989594 (+    76us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:21:31.989631 (+    37us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:31.989691 (+    60us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:31.990068 (+   377us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:31.958810 18362 raft_consensus.cc:2468] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 7.
W20260709 12:21:31.984489 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 191340us
I20260709 12:21:31.984030 18366 rpcz_store.cc:276] 0709 12:21:30.173282 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:31.917775 (+1744493us) service_pool.cc:224] Handling call
0709 12:21:31.939782 (+ 22007us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:31.983066 (+ 43284us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:31.983349 (+   283us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:31.993446 17654 leader_election.cc:304] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:21:31.993798 18366 raft_consensus.cc:1240] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 1. Current term is 2. Ops: []
I20260709 12:21:31.994490 18366 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59902 (request call id 731) took 1779 ms. Trace:
I20260709 12:21:31.994143 18362 raft_consensus.cc:1240] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 4. Current term is 5. Ops: []
I20260709 12:21:31.995924 18362 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59902 (request call id 736) took 1779 ms. Trace:
I20260709 12:21:31.986338 18365 raft_consensus.cc:2468] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 4.
I20260709 12:21:31.950812 20059 raft_consensus.cc:3060] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:31.884331 19893 raft_consensus.cc:515] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:31.991149 17757 rpcz_store.cc:276] 0709 12:21:28.146078 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:28.146728 (+   650us) service_pool.cc:224] Handling call
0709 12:21:28.163327 (+ 16599us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:30.249229 (+2085902us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:31.990536 (+1741307us) spinlock_profiling.cc:243] Waited 2.04 s on lock 0x7b54004155d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:21:31.990703 (+   167us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4686658944}
I20260709 12:21:31.995801 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:31.995059 18366 rpcz_store.cc:276] 0709 12:21:30.215355 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:31.993378 (+1778023us) service_pool.cc:224] Handling call
0709 12:21:31.993637 (+   259us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:31.994186 (+   549us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:31.994424 (+   238us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:31.985692 17834 raft_consensus.cc:3060] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:31.906526 20081 raft_consensus.cc:515] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:31.997596 18334 leader_election.cc:304] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:32.013247 18334 leader_election.cc:304] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:21:32.014396 18334 leader_election.cc:304] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:21:32.015237 18334 leader_election.cc:304] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:21:31.996943 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:31.998214 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:32.016387 18365 raft_consensus.cc:2376] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 2. Re-sending same reply.
I20260709 12:21:32.002010 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:31.992846 18374 raft_consensus.cc:1240] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 2. Current term is 3. Ops: []
I20260709 12:21:32.017504 17683 raft_consensus.cc:2393] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 4: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:21:32.018181 18374 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59902 (request call id 728) took 1803 ms. Trace:
I20260709 12:21:32.018569 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:32.018625 18374 rpcz_store.cc:276] 0709 12:21:30.214565 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:31.992143 (+1777578us) service_pool.cc:224] Handling call
0709 12:21:31.992483 (+   340us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:32.017885 (+ 25402us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:32.018114 (+   229us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:32.019680 17836 raft_consensus.cc:2393] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 4: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:21:32.020484 18374 raft_consensus.cc:1240] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 1. Current term is 2. Ops: []
I20260709 12:21:32.017494 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:32.004521 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:32.021427 17834 raft_consensus.cc:2468] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 4.
I20260709 12:21:32.005715 18362 rpcz_store.cc:276] 0709 12:21:30.216567 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:31.993751 (+1777184us) service_pool.cc:224] Handling call
0709 12:21:31.993984 (+   233us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:31.995614 (+  1630us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:31.995844 (+   230us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:32.006897 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:32.022758 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:32.023494 17688 raft_consensus.cc:1275] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
I20260709 12:21:32.024353 17688 pending_rounds.cc:85] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3: Aborting all ops after (but not including) 1
I20260709 12:21:32.024634 17688 pending_rounds.cc:107] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20260709 12:21:32.024668 17830 raft_consensus.cc:2468] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 6.
I20260709 12:21:32.025015 17688 raft_consensus.cc:2889] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:21:32.025413 17830 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35040 (request call id 1872) took 1258 ms. Trace:
W20260709 12:21:32.011679 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 154284us
I20260709 12:21:31.991641 19753 raft_consensus.cc:3060] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:32.021706 18365 raft_consensus.cc:2468] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 3.
I20260709 12:21:32.030431 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:31.999401 19883 raft_consensus.cc:2804] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:21:32.031244 18365 raft_consensus.cc:2468] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 3.
I20260709 12:21:32.028198 18334 leader_election.cc:304] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:21:32.032438 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:32.033234 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:32.033878 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:32.005242 17831 raft_consensus.cc:2468] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 5.
I20260709 12:21:32.035118 17831 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35040 (request call id 1867) took 1366 ms. Trace:
I20260709 12:21:32.025859 17830 rpcz_store.cc:276] 0709 12:21:30.767321 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:31.980818 (+1213497us) service_pool.cc:224] Handling call
0709 12:21:32.025341 (+ 44523us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:32.035753 18334 leader_election.cc:304] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:32.036463 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:32.027196 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:32.027760 17834 raft_consensus.cc:2410] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 12:21:31.981675 20075 raft_consensus.cc:515] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:32.020985 20019 raft_consensus.cc:2749] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:21:32.027160 18374 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59902 (request call id 738) took 1810 ms. Trace:
I20260709 12:21:31.986500 20090 raft_consensus.cc:515] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:32.039255 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:32.020821 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:32.040058 17830 raft_consensus.cc:2393] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 6: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:21:32.041251 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:32.058904 17688 raft_consensus.cc:1240] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 1. Current term is 2. Ops: []
I20260709 12:21:32.068823 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:32.074069 17688 raft_consensus.cc:1275] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260709 12:21:32.032568 19873 raft_consensus.cc:2804] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:21:32.046839 18368 raft_consensus.cc:1240] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 2. Current term is 4. Ops: []
I20260709 12:21:32.077621 19873 raft_consensus.cc:493] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:32.077936 18368 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59902 (request call id 740) took 1860 ms. Trace:
I20260709 12:21:32.036214 20088 leader_election.cc:290] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 4 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:32.061468 17802 leader_election.cc:304] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:21:32.068997 18374 rpcz_store.cc:276] 0709 12:21:30.217022 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:32.019735 (+1802713us) service_pool.cc:224] Handling call
0709 12:21:32.020258 (+   523us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:32.026871 (+  6613us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:32.027097 (+   226us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:32.025414 20092 raft_consensus.cc:2749] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:21:32.034855 19883 raft_consensus.cc:493] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:32.029838 19779 raft_consensus.cc:2764] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20260709 12:21:32.075176 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:21:31.980295 17833 rpcz_store.cc:269] 0709 12:21:29.768031 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:29.775952 (+  7921us) service_pool.cc:224] Handling call
0709 12:21:31.893795 (+2117843us) spinlock_profiling.cc:243] Waited 109 ms on lock 0x7b540022ae54. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:21:31.906853 (+ 13058us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":250647936}
I20260709 12:21:32.029474 20078 raft_consensus.cc:493] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:32.075757 18365 raft_consensus.cc:1240] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 3. Current term is 4. Ops: []
I20260709 12:21:32.077203 17836 raft_consensus.cc:2393] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 4: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:21:32.077103 17830 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35042 (request call id 1635) took 2195 ms. Trace:
I20260709 12:21:32.095621 17830 rpcz_store.cc:276] 0709 12:21:29.881814 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:32.076351 (+2194537us) service_pool.cc:224] Handling call
0709 12:21:32.076618 (+   267us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:32.076792 (+   174us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:32.076819 (+    27us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:32.076855 (+    36us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:21:32.076886 (+    31us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:32.076935 (+    49us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:32.077051 (+   116us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:32.078032 19873 raft_consensus.cc:3060] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:32.098183 17830 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35042 (request call id 1639) took 2208 ms. Trace:
I20260709 12:21:32.098603 17830 rpcz_store.cc:276] 0709 12:21:29.889761 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:32.097253 (+2207492us) service_pool.cc:224] Handling call
0709 12:21:32.097643 (+   390us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:32.097845 (+   202us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:32.097882 (+    37us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:32.097931 (+    49us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:21:32.097971 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:32.098021 (+    50us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:32.098131 (+   110us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:32.035557 17831 rpcz_store.cc:276] 0709 12:21:30.669260 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:31.789798 (+1120538us) service_pool.cc:224] Handling call
0709 12:21:32.035044 (+245246us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:32.100037 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:32.078740 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:32.078356 18368 rpcz_store.cc:276] 0709 12:21:30.217499 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:32.046214 (+1828715us) service_pool.cc:224] Handling call
0709 12:21:32.046613 (+   399us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:32.077705 (+ 31092us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:32.077883 (+   178us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:32.102562 18368 raft_consensus.cc:1240] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 6. Current term is 7. Ops: []
I20260709 12:21:32.103332 18368 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59886 (request call id 750) took 1079 ms. Trace:
I20260709 12:21:32.103739 18368 rpcz_store.cc:276] 0709 12:21:31.023404 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:32.101969 (+1078565us) service_pool.cc:224] Handling call
0709 12:21:32.102328 (+   359us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:32.103028 (+   700us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:32.103264 (+   236us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:32.045778 20093 raft_consensus.cc:2804] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:21:32.105588 18365 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59902 (request call id 741) took 1887 ms. Trace:
I20260709 12:21:32.090332 17802 leader_election.cc:304] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
W20260709 12:21:32.105782 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 81839us
I20260709 12:21:32.110540 17830 raft_consensus.cc:1240] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 4. Current term is 5. Ops: [4.4-4.4]
I20260709 12:21:32.111431 17830 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35040 (request call id 1864) took 1654 ms. Trace:
I20260709 12:21:32.098477 20019 raft_consensus.cc:2804] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:21:32.112021 17830 rpcz_store.cc:276] 0709 12:21:30.457396 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:32.109545 (+1652149us) service_pool.cc:224] Handling call
0709 12:21:32.110311 (+   766us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:32.111134 (+   823us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:32.111368 (+   234us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:32.106384 18365 rpcz_store.cc:276] 0709 12:21:30.217745 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:32.052892 (+1835147us) service_pool.cc:224] Handling call
0709 12:21:32.075523 (+ 22631us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:32.105347 (+ 29824us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:32.105533 (+   186us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:32.123549 17688 raft_consensus.cc:1275] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:21:32.119700 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:32.122372 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:21:32.124451 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 96957.3us
I20260709 12:21:32.127188 17686 raft_consensus.cc:2468] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 3.
I20260709 12:21:32.127604 17652 leader_election.cc:304] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:21:32.126273 17685 raft_consensus.cc:2468] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 7.
I20260709 12:21:32.097635 19883 raft_consensus.cc:3060] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:32.084322 20127 consensus_queue.cc:1048] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.040s
W20260709 12:21:32.122680 20085 consensus_meta.cc:287] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.526s	user 0.002s	sys 0.007s
I20260709 12:21:32.131505 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:32.129706 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:32.099339 20074 raft_consensus.cc:2749] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:21:32.131076 17652 leader_election.cc:304] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
W20260709 12:21:32.122859 19895 consensus_meta.cc:287] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.742s	user 0.011s	sys 0.018s
I20260709 12:21:32.105147 20093 raft_consensus.cc:493] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:32.135552 17688 raft_consensus.cc:1240] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 5. Current term is 6. Ops: []
I20260709 12:21:32.131594 17654 leader_election.cc:304] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:21:32.111110 20052 raft_consensus.cc:2804] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:21:32.130160 18365 raft_consensus.cc:1240] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 2. Current term is 3. Ops: []
I20260709 12:21:32.158780 18365 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59886 (request call id 751) took 1134 ms. Trace:
I20260709 12:21:32.159173 18365 rpcz_store.cc:276] 0709 12:21:31.034182 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:32.129091 (+1094909us) service_pool.cc:224] Handling call
0709 12:21:32.129379 (+   288us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:32.158527 (+ 29148us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:32.158721 (+   194us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:32.159554 17685 raft_consensus.cc:1240] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 2. Current term is 3. Ops: []
I20260709 12:21:32.160341 18365 raft_consensus.cc:3060] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:32.112811 20019 raft_consensus.cc:493] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:21:32.124094 19960 raft_consensus.cc:3060] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:32.109894 20077 leader_election.cc:290] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 5 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:21:32.154726 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 118919us
I20260709 12:21:32.190022 18334 leader_election.cc:304] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:21:32.131935 20128 raft_consensus.cc:2804] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:21:32.190995 18334 leader_election.cc:304] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:32.154767 18368 raft_consensus.cc:1275] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260709 12:21:32.113056 18361 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59902 (request call id 726) took 1895 ms. Trace:
I20260709 12:21:32.192512 18368 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59886 (request call id 753) took 1124 ms. Trace:
I20260709 12:21:32.123070 19855 leader_election.cc:290] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 3 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:21:32.196568 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 65559.7us
I20260709 12:21:32.113714 20112 leader_election.cc:290] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 4 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:32.217345 18365 raft_consensus.cc:1275] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
W20260709 12:21:32.171000 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:32.173004 17688 raft_consensus.cc:3055] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 4 LEADER]: Stepping down as leader of term 4
W20260709 12:21:32.184423 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 72486.3us
I20260709 12:21:32.129271 19868 leader_election.cc:290] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 3 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:32.192994 18368 rpcz_store.cc:276] 0709 12:21:31.068475 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:32.134404 (+1065929us) service_pool.cc:224] Handling call
0709 12:21:32.138862 (+  4458us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:32.192186 (+ 53324us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:32.192438 (+   252us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:21:32.158044 20126 log.cc:927] Time spent T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.084s	user 0.001s	sys 0.000s
I20260709 12:21:32.106506 20078 raft_consensus.cc:515] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:32.157155 20093 raft_consensus.cc:3060] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:21:32.131160 19848 consensus_queue.cc:1059] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.055s
I20260709 12:21:32.248108 19757 raft_consensus.cc:2804] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:21:32.248711 19757 raft_consensus.cc:493] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:21:32.249217 19757 raft_consensus.cc:3060] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:32.158432 20052 raft_consensus.cc:493] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:21:32.184206 20019 raft_consensus.cc:3060] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:32.272921 18362 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59886 (request call id 746) took 1838 ms. Trace:
I20260709 12:21:32.280556 18362 rpcz_store.cc:276] 0709 12:21:30.434684 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:32.074114 (+1639430us) service_pool.cc:224] Handling call
0709 12:21:32.074365 (+   251us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:32.074567 (+   202us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:21:32.164596 (+ 90029us) log.cc:844] Serialized 14 byte log entry
0709 12:21:32.272554 (+107958us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:32.272606 (+    52us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:21:32.272672 (+    66us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:32.272737 (+    65us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:32.272862 (+   125us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":3607,"thread_start_us":180938,"threads_started":2}
I20260709 12:21:32.225446 17802 leader_election.cc:304] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
W20260709 12:21:32.282442 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 57119.4us
I20260709 12:21:32.283929 17802 leader_election.cc:304] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:32.223857 18365 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59902 (request call id 762) took 1002 ms. Trace:
I20260709 12:21:32.285440 18365 rpcz_store.cc:276] 0709 12:21:31.221482 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:32.159988 (+938506us) service_pool.cc:224] Handling call
0709 12:21:32.160230 (+   242us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:32.223539 (+ 63309us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:32.223788 (+   249us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:32.287204 18365 raft_consensus.cc:1240] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20260709 12:21:32.224804 17688 raft_consensus.cc:740] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:21:32.289513 17802 leader_election.cc:304] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:21:32.295780 17802 leader_election.cc:304] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:32.288767 17688 consensus_queue.cc:260] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:32.157229 19895 raft_consensus.cc:515] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:32.239473 19868 raft_consensus.cc:3055] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:21:32.246651 20088 raft_consensus.cc:2804] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260709 12:21:32.314761 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 183280us
I20260709 12:21:32.217900 20012 leader_election.cc:290] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:32.219156 19779 raft_consensus.cc:2749] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:21:32.136085 20085 raft_consensus.cc:515] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:32.319500 20112 consensus_queue.cc:1059] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
W20260709 12:21:32.321746 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 123856us
W20260709 12:21:32.353780 17652 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:32.324627 18374 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59886 (request call id 749) took 1301 ms. Trace:
I20260709 12:21:32.354761 18374 rpcz_store.cc:276] 0709 12:21:31.023113 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:32.082212 (+1059099us) service_pool.cc:224] Handling call
0709 12:21:32.082635 (+   423us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:32.082876 (+   241us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 12:21:32.219584 (+136708us) log.cc:844] Serialized 14 byte log entry
0709 12:21:32.324152 (+104568us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:32.324223 (+    71us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:21:32.324293 (+    70us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:32.324382 (+    89us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:32.324539 (+   157us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":9740,"thread_start_us":221597,"threads_started":2}
I20260709 12:21:32.279560 20052 raft_consensus.cc:3060] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:32.232184 20078 leader_election.cc:290] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:32.270985 20128 raft_consensus.cc:697] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 4 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:21:32.326692 17654 leader_election.cc:304] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
W20260709 12:21:32.330399 17571 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet d522f70beb1f417cb98940d90b594823 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica e2909cff1da44ad387f96a8ae3aa5afb is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 3 committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:21:32.332404 18333 leader_election.cc:304] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:32.336865 17838 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37994 (request call id 550) took 1334 ms. Trace:
I20260709 12:21:32.362907 17838 rpcz_store.cc:276] 0709 12:21:31.002026 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:31.324872 (+322846us) service_pool.cc:224] Handling call
0709 12:21:32.336631 (+1011759us) spinlock_profiling.cc:243] Waited 656 ms on lock 0x7b540008cfd4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:21:32.336782 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1508983680}
W20260709 12:21:32.338016 18364 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59902 (request call id 734) took 2121 ms (client timeout 2183 ms). Trace:
I20260709 12:21:32.325927 20088 raft_consensus.cc:493] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:32.305670 19848 raft_consensus.cc:2749] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:21:32.197578 20092 consensus_queue.cc:1059] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.117s
I20260709 12:21:32.306486 19978 raft_consensus.cc:2749] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:21:32.408872 19978 raft_consensus.cc:2804] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:21:32.409454 19978 raft_consensus.cc:493] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:21:32.409917 19978 raft_consensus.cc:3060] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:32.413589 20085 leader_election.cc:290] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 4 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:32.305668 20074 raft_consensus.cc:2749] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:21:32.246038 20108 consensus_queue.cc:1059] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.076s
I20260709 12:21:32.349805 17729 leader_election.cc:304] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:21:32.416790 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:21:32.417076 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 67363.3us
I20260709 12:21:32.417584 18366 raft_consensus.cc:2468] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 5.
I20260709 12:21:32.417821 17729 leader_election.cc:304] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:21:32.418694 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:32.350649 17683 raft_consensus.cc:2393] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 4: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:21:32.311334 20109 leader_election.cc:290] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 3 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:32.311872 20136 raft_consensus.cc:2804] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:21:32.356622 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:21:32.364080 18364 rpcz_store.cc:269] 0709 12:21:30.216096 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:31.224426 (+1008330us) service_pool.cc:224] Handling call
0709 12:21:32.337782 (+1113356us) spinlock_profiling.cc:243] Waited 520 ms on lock 0x7b5400305f54. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:21:32.337942 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1197010176}
I20260709 12:21:32.339498 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:32.364944 17835 raft_consensus.cc:1275] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 12:21:32.324303 19893 leader_election.cc:290] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:32.501585 17835 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35040 (request call id 1881) took 1508 ms. Trace:
I20260709 12:21:32.501978 17835 rpcz_store.cc:276] 0709 12:21:30.993352 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:32.340907 (+1347555us) service_pool.cc:224] Handling call
0709 12:21:32.364406 (+ 23499us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:32.501346 (+136940us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:32.501535 (+   189us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:32.324999 19868 raft_consensus.cc:740] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 2 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:21:32.368592 17837 raft_consensus.cc:2393] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 4: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:21:32.361320 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:21:32.398465 20134 log.cc:927] Time spent T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.092s	user 0.000s	sys 0.000s
W20260709 12:21:32.404886 20141 log.cc:927] Time spent T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.059s	user 0.001s	sys 0.000s
W20260709 12:21:32.405612 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 78996.9us
I20260709 12:21:32.406296 20012 raft_consensus.cc:2804] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:21:32.423556 18366 raft_consensus.cc:1240] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 3. Current term is 4. Ops: []
I20260709 12:21:32.198329 18361 rpcz_store.cc:276] 0709 12:21:30.213976 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:31.989865 (+1775889us) service_pool.cc:224] Handling call
0709 12:21:32.024574 (+ 34709us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:32.071149 (+ 46575us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:32.075810 (+  4661us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:32.078450 (+  2640us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:21:32.084946 (+  6496us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:32.098785 (+ 13839us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:32.109779 (+ 10994us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:21:32.439003 17655 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:32.456656 17652 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:32.362522 20075 leader_election.cc:290] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 4 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:32.460551 17834 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35042 (request call id 1637) took 2571 ms. Trace:
W20260709 12:21:32.757020 17652 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:32.756141 17834 rpcz_store.cc:276] 0709 12:21:29.889151 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:32.083741 (+2194590us) service_pool.cc:224] Handling call
0709 12:21:32.084865 (+  1124us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:32.085080 (+   215us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:21:32.246365 (+161285us) log.cc:844] Serialized 14 byte log entry
0709 12:21:32.457187 (+210822us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:32.459734 (+  2547us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:21:32.459920 (+   186us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:32.460090 (+   170us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:32.460428 (+   338us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":417640,"raft.queue_time_us":96367,"raft.run_cpu_time_us":138,"raft.run_wall_time_us":138,"thread_start_us":347544,"threads_started":3,"wal-append.queue_time_us":216177}
I20260709 12:21:32.372223 20088 raft_consensus.cc:3060] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:21:32.398466 20129 log.cc:927] Time spent T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.238s	user 0.000s	sys 0.001s
W20260709 12:21:32.468375 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:32.472370 20128 consensus_queue.cc:237] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:32.462371 20018 consensus_queue.cc:1059] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.014s
I20260709 12:21:32.477161 19895 leader_election.cc:290] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 4 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:32.457715 19838 raft_consensus.cc:2804] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:21:32.435552 20078 raft_consensus.cc:2749] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:21:32.369863 20112 consensus_queue.cc:1243] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Peer e2909cff1da44ad387f96a8ae3aa5afb log is divergent from this leader: its last log entry 5.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20260709 12:21:32.467159 20091 consensus_queue.cc:1059] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.013s
I20260709 12:21:32.480865 20135 raft_consensus.cc:3055] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 3 LEADER]: Stepping down as leader of term 3
W20260709 12:21:32.484700 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 176936us
I20260709 12:21:32.783291 17802 leader_election.cc:304] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:21:32.784396 17802 leader_election.cc:304] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
W20260709 12:21:32.497332 17687 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44898 (request call id 508) took 2641 ms (client timeout 1865 ms). Trace:
I20260709 12:21:32.455535 20090 leader_election.cc:290] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 4 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:32.420492 17838 raft_consensus.cc:1240] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 3. Current term is 4. Ops: []
I20260709 12:21:32.485570 20143 raft_consensus.cc:493] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:32.751053 17801 leader_election.cc:304] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
W20260709 12:21:32.752928 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 333604us
W20260709 12:21:32.757547 17652 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.16.29.67:36837 timed out after 1.814s (SENT) [suppressed 22 similar messages]
W20260709 12:21:32.758689 17804 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:32.761798 17835 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35042 (request call id 1640) took 1489 ms. Trace:
I20260709 12:21:32.765666 17654 leader_election.cc:304] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b
W20260709 12:21:32.835613 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 70033us
W20260709 12:21:32.836917 20099 consensus_meta.cc:287] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 1.205s	user 0.010s	sys 0.002s
I20260709 12:21:32.837113 17654 leader_election.cc:304] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:21:32.754063 20108 raft_consensus.cc:493] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:32.837510 20099 raft_consensus.cc:515] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:32.838348 20108 raft_consensus.cc:515] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:32.841053 20108 leader_election.cc:290] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:32.433660 20136 raft_consensus.cc:493] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:32.848883 20136 raft_consensus.cc:3060] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:32.766686 20075 raft_consensus.cc:493] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:32.785426 17802 leader_election.cc:304] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:32.455224 20027 raft_consensus.cc:2804] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Leader election won for term 4
W20260709 12:21:32.785198 17687 rpcz_store.cc:269] 0709 12:21:29.842760 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:29.843013 (+   253us) service_pool.cc:224] Handling call
0709 12:21:32.483765 (+2640752us) spinlock_profiling.cc:243] Waited 387 ms on lock 0x7b5400345dd4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:21:32.483873 (+   108us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":890989440}
W20260709 12:21:32.774056 20093 consensus_meta.cc:287] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.541s	user 0.011s	sys 0.001s
I20260709 12:21:32.794353 17831 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35040 (request call id 1865) took 2294 ms. Trace:
I20260709 12:21:32.768523 20076 raft_consensus.cc:493] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:32.851466 18364 raft_consensus.cc:1275] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
W20260709 12:21:32.853695 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 68350.5us
I20260709 12:21:32.856004 17802 leader_election.cc:304] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:21:32.784747 20150 log.cc:927] Time spent T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.385s	user 0.000s	sys 0.001s
I20260709 12:21:32.830662 20147 raft_consensus.cc:493] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:32.767853 17832 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35040 (request call id 1858) took 2976 ms (client timeout 2169 ms). Trace:
I20260709 12:21:32.834949 17835 rpcz_store.cc:276] 0709 12:21:31.272415 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:32.502947 (+1230532us) service_pool.cc:224] Handling call
0709 12:21:32.761321 (+258374us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:32.761494 (+   173us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:32.761516 (+    22us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:32.761555 (+    39us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:21:32.761589 (+    34us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:32.761639 (+    50us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:32.761743 (+   104us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:32.851141 17831 rpcz_store.cc:276] 0709 12:21:30.499971 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:32.107494 (+1607523us) service_pool.cc:224] Handling call
0709 12:21:32.107830 (+   336us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:32.108385 (+   555us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:32.108417 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:32.109005 (+   588us) log.cc:844] Serialized 26 byte log entry
0709 12:21:32.190976 (+ 81971us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:21:32.323578 (+132602us) log.cc:844] Serialized 14 byte log entry
0709 12:21:32.496261 (+172683us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:32.496455 (+   194us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:32.784627 (+288172us) raft_consensus.cc:1685] finished
0709 12:21:32.784647 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:32.794280 (+  9633us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":583672,"prepare.run_cpu_time_us":63,"prepare.run_wall_time_us":3056,"raft.queue_time_us":91638,"raft.run_cpu_time_us":226,"raft.run_wall_time_us":16388,"thread_start_us":319328,"threads_started":3,"wal-append.queue_time_us":644827}
I20260709 12:21:32.841522 18368 raft_consensus.cc:1240] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 4. Current term is 5. Ops: []
I20260709 12:21:32.778021 20012 raft_consensus.cc:697] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 5 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:21:32.844828 19848 raft_consensus.cc:2804] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:21:32.768654 20127 consensus_queue.cc:1048] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.083s
I20260709 12:21:32.781890 20135 raft_consensus.cc:740] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
W20260709 12:21:32.860299 20059 consensus_meta.cc:287] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.833s	user 0.012s	sys 0.000s
I20260709 12:21:32.892354 20133 raft_consensus.cc:493] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:32.876137 17654 leader_election.cc:304] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:32.916914 17654 leader_election.cc:304] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:32.795796 20155 raft_consensus.cc:493] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:32.866077 20109 raft_consensus.cc:2804] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Leader election won for term 5
W20260709 12:21:32.897993 20129 log.cc:927] Time spent T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.111s	user 0.000s	sys 0.000s
I20260709 12:21:32.926983 20012 consensus_queue.cc:237] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:32.927969 17835 raft_consensus.cc:1240] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 6. Current term is 7. Ops: []
I20260709 12:21:32.964197 17835 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35042 (request call id 1641) took 1690 ms. Trace:
I20260709 12:21:32.895437 19848 raft_consensus.cc:493] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:21:32.929219 20019 consensus_meta.cc:287] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.613s	user 0.006s	sys 0.003s
W20260709 12:21:32.926222 19960 consensus_meta.cc:287] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.738s	user 0.008s	sys 0.000s
I20260709 12:21:32.895190 19895 raft_consensus.cc:2804] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:21:32.894954 20112 raft_consensus.cc:2804] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:21:32.964697 17835 rpcz_store.cc:276] 0709 12:21:31.273778 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:32.892786 (+1619008us) service_pool.cc:224] Handling call
0709 12:21:32.915598 (+ 22812us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:32.963775 (+ 48177us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:32.964123 (+   348us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:21:32.895117 17832 rpcz_store.cc:269] 0709 12:21:29.791092 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:29.793242 (+  2150us) service_pool.cc:224] Handling call
0709 12:21:32.347004 (+2553762us) spinlock_profiling.cc:243] Waited 3.87 ms on lock 0x7b580013fe98. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b474fbb71 00007f7b474fba22 00007f7b504148d9 00007f7b4f56eee0 00007f7b4f6380ab 00007f7b4f638131 00007f7b4f5e60a0 00007f7b4f5e5984 00007f7b48b606d4 00007f7b48b60662
0709 12:21:32.767749 (+420745us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":8910080}
I20260709 12:21:32.892079 20076 raft_consensus.cc:515] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:32.863422 20093 raft_consensus.cc:515] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:32.865857 20075 raft_consensus.cc:515] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:32.932361 20155 raft_consensus.cc:515] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:21:33.003005 20145 log.cc:927] Time spent T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.225s	user 0.000s	sys 0.000s
I20260709 12:21:33.002827 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:33.003855 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:32.829749 20102 raft_consensus.cc:3055] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 3 LEADER]: Stepping down as leader of term 3
W20260709 12:21:33.006175 20052 consensus_meta.cc:287] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.636s	user 0.009s	sys 0.000s
I20260709 12:21:32.897794 20158 consensus_queue.cc:1048] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.104s
W20260709 12:21:33.010504 19873 consensus_meta.cc:287] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.901s	user 0.007s	sys 0.003s
I20260709 12:21:33.011133 20127 raft_consensus.cc:2804] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:21:32.947721 19893 raft_consensus.cc:2804] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260709 12:21:33.055728 19753 consensus_meta.cc:287] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 1.015s	user 0.006s	sys 0.001s
W20260709 12:21:33.018209 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 160795us
W20260709 12:21:33.015503 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 257971us
I20260709 12:21:32.851424 19868 consensus_queue.cc:260] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:32.985423 19848 raft_consensus.cc:3060] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:32.893339 20143 raft_consensus.cc:515] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:32.914610 20059 raft_consensus.cc:515] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:32.896940 20147 raft_consensus.cc:515] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:33.016397 20112 raft_consensus.cc:697] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 2 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:21:32.993810 19895 raft_consensus.cc:493] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:21:33.050930 19757 consensus_meta.cc:287] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.801s	user 0.008s	sys 0.002s
W20260709 12:21:33.055941 19883 consensus_meta.cc:287] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.920s	user 0.011s	sys 0.000s
I20260709 12:21:33.056439 20099 leader_election.cc:290] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 5 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:33.054016 20088 raft_consensus.cc:515] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:33.061693 20093 leader_election.cc:290] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 8 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:33.088940 20139 raft_consensus.cc:2804] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Leader election won for term 4
W20260709 12:21:33.062709 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 43816.5us
I20260709 12:21:33.063423 18371 raft_consensus.cc:2393] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 8: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
W20260709 12:21:32.780776 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 406253us
W20260709 12:21:33.079268 17652 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:32.933074 20133 raft_consensus.cc:515] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:32.996357 20019 raft_consensus.cc:515] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:33.104991 17831 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35042 (request call id 1642) took 1821 ms. Trace:
I20260709 12:21:33.064646 20018 raft_consensus.cc:493] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:21:33.083546 19873 raft_consensus.cc:515] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:33.076050 20127 raft_consensus.cc:493] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:21:33.077288 19893 raft_consensus.cc:493] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:21:33.111239 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
W20260709 12:21:33.130209 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 379212us
I20260709 12:21:33.156970 17835 raft_consensus.cc:3060] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:33.079867 20027 raft_consensus.cc:697] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
W20260709 12:21:33.106159 20084 consensus_meta.cc:287] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 1.201s	user 0.010s	sys 0.002s
I20260709 12:21:33.134737 17830 raft_consensus.cc:1240] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 3. Current term is 4. Ops: []
I20260709 12:21:32.996275 20067 raft_consensus.cc:493] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:33.168983 20067 raft_consensus.cc:515] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:33.169947 17680 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44898 (request call id 523) took 1849 ms. Trace:
I20260709 12:21:33.170439 17680 rpcz_store.cc:276] 0709 12:21:31.320596 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:31.756995 (+436399us) service_pool.cc:224] Handling call
0709 12:21:31.807732 (+ 50737us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:32.045427 (+237695us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:32.056369 (+ 10942us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:32.071767 (+ 15398us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:21:32.074363 (+  2596us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:33.169620 (+1095257us) spinlock_profiling.cc:243] Waited 224 ms on lock 0x7b5400190354. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:21:33.169723 (+   103us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:33.169883 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":515516800}
I20260709 12:21:33.109345 20157 raft_consensus.cc:2804] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260709 12:21:33.138684 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 305396us
I20260709 12:21:33.172098 20157 raft_consensus.cc:493] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:21:33.172740 20157 raft_consensus.cc:3060] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:33.173535 17683 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37396 (request call id 1778) took 1167 ms. Trace:
I20260709 12:21:33.174124 20165 consensus_queue.cc:1059] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 12:21:33.143380 17652 leader_election.cc:304] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:21:33.067203 20052 raft_consensus.cc:515] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:33.175930 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:33.176765 19807 raft_consensus.cc:3055] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:21:33.177148 19807 raft_consensus.cc:740] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 3 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:21:33.177362 18367 raft_consensus.cc:2410] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 5 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260709 12:21:33.177436 19806 raft_consensus.cc:2804] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:21:33.177624 17652 leader_election.cc:304] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:33.177985 19806 raft_consensus.cc:493] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:33.178512 19806 raft_consensus.cc:3060] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:33.177961 19807 consensus_queue.cc:260] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:33.178755 17652 leader_election.cc:304] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:33.179041 18367 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59902 (request call id 755) took 1962 ms. Trace:
I20260709 12:21:33.143455 18333 leader_election.cc:304] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:33.065189 20135 consensus_queue.cc:260] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:33.173969 17683 rpcz_store.cc:276] 0709 12:21:32.005815 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:32.026920 (+ 21105us) service_pool.cc:224] Handling call
0709 12:21:33.173318 (+1146398us) spinlock_profiling.cc:243] Waited 313 ms on lock 0x7b54000c5b54. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:21:33.173466 (+   148us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":719757056}
I20260709 12:21:33.184422 18363 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 876) took 3011 ms. Trace:
I20260709 12:21:33.184755 20135 raft_consensus.cc:3060] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:33.184844 18363 rpcz_store.cc:276] 0709 12:21:30.173007 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:31.896037 (+1723030us) service_pool.cc:224] Handling call
0709 12:21:33.182805 (+1286768us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:33.183102 (+   297us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 12:21:33.183129 (+    27us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:33.183203 (+    74us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:21:33.183240 (+    37us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:33.183294 (+    54us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:33.184271 (+   977us) spinlock_profiling.cc:243] Waited 1.25 s on lock 0x7b54001efd50. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a61f8ad 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:21:33.184369 (+    98us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2883965312}
I20260709 12:21:33.146365 18362 raft_consensus.cc:1240] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 4. Current term is 5. Ops: []
I20260709 12:21:33.186483 20099 raft_consensus.cc:2749] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:21:33.110301 19895 raft_consensus.cc:3060] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:33.189471 20147 leader_election.cc:290] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:33.190549 20147 raft_consensus.cc:2804] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:21:33.179448 18367 rpcz_store.cc:276] 0709 12:21:31.216499 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:31.790766 (+574267us) service_pool.cc:224] Handling call
0709 12:21:33.178863 (+1388097us) spinlock_profiling.cc:243] Waited 1.38 s on lock 0x7b54001efd54. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:21:33.178989 (+   126us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3182045568}
I20260709 12:21:33.191036 20147 raft_consensus.cc:493] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:21:33.191447 20147 raft_consensus.cc:3060] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:33.192451 20027 consensus_queue.cc:237] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:33.193915 20067 leader_election.cc:290] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:33.194486 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:33.065804 20102 raft_consensus.cc:740] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:21:33.110013 20074 raft_consensus.cc:2804] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:21:33.196833 18368 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 886) took 1200 ms. Trace:
W20260709 12:21:33.198282 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:33.198073 18368 rpcz_store.cc:276] 0709 12:21:31.996140 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:32.227826 (+231686us) service_pool.cc:224] Handling call
0709 12:21:32.228209 (+   383us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:32.894230 (+666021us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:33.196377 (+302147us) spinlock_profiling.cc:243] Waited 611 ms on lock 0x7b5400724054. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:21:33.196772 (+   395us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1406123904}
I20260709 12:21:33.179345 19807 raft_consensus.cc:3060] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:33.147924 17831 rpcz_store.cc:276] 0709 12:21:31.283459 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:32.892961 (+1609502us) service_pool.cc:224] Handling call
0709 12:21:32.905675 (+ 12714us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:32.905925 (+   250us) raft_consensus.cc:1534] Early marking committed up to index 2
0709 12:21:32.945280 (+ 39355us) log.cc:844] Serialized 14 byte log entry
0709 12:21:33.104310 (+159030us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:33.104385 (+    75us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:21:33.104453 (+    68us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:33.104550 (+    97us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:33.104901 (+   351us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":39163,"prepare.queue_time_us":56755,"prepare.run_cpu_time_us":54,"prepare.run_wall_time_us":56,"thread_start_us":119320,"threads_started":1}
I20260709 12:21:33.091475 20170 raft_consensus.cc:3055] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:21:33.088836 19753 raft_consensus.cc:515] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:33.136837 19868 raft_consensus.cc:3060] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:33.140225 20153 raft_consensus.cc:493] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:33.142127 20109 raft_consensus.cc:697] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 5 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:21:33.161532 17728 leader_election.cc:304] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:21:33.155555 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:33.145481 19838 raft_consensus.cc:697] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 4 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:21:33.219671 20153 raft_consensus.cc:515] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:33.220227 19838 consensus_queue.cc:237] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:33.221757 20153 leader_election.cc:290] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:33.222278 19806 raft_consensus.cc:515] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:30.966097 16501 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:21:33.208431 20170 raft_consensus.cc:740] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:21:33.223918 20170 consensus_queue.cc:260] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:33.224349 20109 consensus_queue.cc:237] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:33.225163 20170 raft_consensus.cc:3060] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:33.228066 17689 raft_consensus.cc:1240] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 4. Current term is 5. Ops: []
W20260709 12:21:33.195528 17829 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35042 (request call id 1649) took 1762 ms (client timeout 1810 ms). Trace:
I20260709 12:21:33.197944 17801 leader_election.cc:304] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:33.155169 20174 raft_consensus.cc:493] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:33.203655 17835 raft_consensus.cc:2468] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 4.
W20260709 12:21:33.210625 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 132487us
I20260709 12:21:33.211330 20139 raft_consensus.cc:697] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 4 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:21:33.109126 20169 consensus_queue.cc:1059] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.066s
W20260709 12:21:33.216073 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 54673.4us
I20260709 12:21:33.253660 17728 leader_election.cc:304] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:21:33.117561 19757 raft_consensus.cc:515] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:33.146283 19883 raft_consensus.cc:515] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:33.219318 17584 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35966 (request call id 352) took 11123 ms. Trace:
I20260709 12:21:33.219167 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:33.254245 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:33.159431 20127 raft_consensus.cc:3060] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:33.257687 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:33.218662 20175 raft_consensus.cc:2804] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:21:33.259569 17728 leader_election.cc:304] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:21:33.158098 19893 raft_consensus.cc:3060] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:21:33.271274 17727 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:33.283239 17728 leader_election.cc:304] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:21:33.252015 20166 raft_consensus.cc:2749] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:21:33.223821 19753 leader_election.cc:290] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 5 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:21:33.276486 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 94654.7us
I20260709 12:21:33.251787 20153 raft_consensus.cc:2749] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:21:33.156076 20018 raft_consensus.cc:515] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:33.196826 20102 consensus_queue.cc:260] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:33.291581 18333 leader_election.cc:304] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
W20260709 12:21:33.287374 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 370518us
I20260709 12:21:33.266654 17584 rpcz_store.cc:276] 0709 12:21:22.096235 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:22.096369 (+   134us) service_pool.cc:224] Handling call
0709 12:21:33.219237 (+11122868us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:21:27.836338 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:21:27.867198 (+ 30860us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:21:27.867263 (+    65us) write_op.cc:435] Acquired schema lock
0709 12:21:27.867288 (+    25us) tablet.cc:662] Decoding operations
0709 12:21:27.868076 (+   788us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:21:27.868143 (+    67us) write_op.cc:641] Partition lock acquired for write op
0709 12:21:27.868179 (+    36us) tablet.cc:685] Acquiring locks for 4 operations
0709 12:21:27.868629 (+   450us) tablet.cc:701] Row locks acquired
0709 12:21:27.868672 (+    43us) write_op.cc:260] PREPARE: finished
0709 12:21:27.868846 (+   174us) write_op.cc:270] Start()
0709 12:21:27.869009 (+   163us) write_op.cc:276] Timestamp: P: 1783599687868766 usec, L: 0
0709 12:21:27.869026 (+    17us) op_driver.cc:348] REPLICATION: starting
0709 12:21:27.869745 (+   719us) log.cc:844] Serialized 1993 byte log entry
0709 12:21:29.539601 (+1669856us) op_driver.cc:464] REPLICATION: finished
0709 12:21:29.574021 (+ 34420us) write_op.cc:301] APPLY: starting
0709 12:21:29.574104 (+    83us) tablet.cc:1367] starting BulkCheckPresence
0709 12:21:29.574250 (+   146us) tablet.cc:1370] finished BulkCheckPresence
0709 12:21:29.574380 (+   130us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:21:29.574930 (+   550us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:21:29.575144 (+   214us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=4
0709 12:21:29.575186 (+    42us) write_op.cc:312] APPLY: finished
0709 12:21:29.575406 (+   220us) log.cc:844] Serialized 41 byte log entry
0709 12:21:29.576249 (+   843us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:21:29.576354 (+   105us) write_op.cc:454] Released schema lock
0709 12:21:29.576394 (+    40us) write_op.cc:341] FINISH: Updating metrics
0709 12:21:29.696185 (+119791us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:21:29.698907 (+  2722us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":12228,"reported_tablets":9,"child_traces":[["op",{"apply.queue_time_us":34131,"apply.run_cpu_time_us":2151,"apply.run_wall_time_us":22144,"num_ops":4,"prepare.queue_time_us":259365,"prepare.run_cpu_time_us":74647,"prepare.run_wall_time_us":862273,"raft.queue_time_us":283665,"raft.run_cpu_time_us":1616,"raft.run_wall_time_us":43794,"replication_time_us":1670378,"spinlock_wait_cycles":1280,"thread_start_us":959359,"threads_started":4,"wal-append.queue_time_us":1139787,"wal-append.run_cpu_time_us":7528,"wal-append.run_wall_time_us":1635209}]]}
I20260709 12:21:33.306157 18333 leader_election.cc:304] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:21:33.304908 17654 leader_election.cc:304] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:21:33.228816 17829 rpcz_store.cc:269] 0709 12:21:31.432707 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:33.194140 (+1761433us) service_pool.cc:224] Handling call
0709 12:21:33.195456 (+  1316us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:33.309561 18333 leader_election.cc:304] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:21:33.312009 18333 leader_election.cc:304] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:21:33.312480 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:33.313114 17831 raft_consensus.cc:2393] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 4: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:21:33.270244 20173 raft_consensus.cc:2749] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:21:33.289242 20165 raft_consensus.cc:2749] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:21:33.284561 20175 raft_consensus.cc:697] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 6 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:21:33.229281 20167 consensus_queue.cc:1048] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.095s
I20260709 12:21:33.291487 20075 leader_election.cc:290] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:33.323197 17836 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35042 (request call id 1653) took 1514 ms. Trace:
I20260709 12:21:33.323577 17836 rpcz_store.cc:276] 0709 12:21:31.808745 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:32.020543 (+211798us) service_pool.cc:224] Handling call
0709 12:21:33.323004 (+1302461us) spinlock_profiling.cc:243] Waited 5.09 ms on lock 0x7b540028d054. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:21:33.323127 (+   123us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":11704192}
I20260709 12:21:33.324189 18333 leader_election.cc:304] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:21:33.309541 19753 consensus_queue.cc:1048] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.023s
I20260709 12:21:33.333197 20019 leader_election.cc:290] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 4 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:33.334407 19873 leader_election.cc:290] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 4 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:33.308137 20067 raft_consensus.cc:2749] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:21:33.317385 20099 raft_consensus.cc:2749] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:21:33.340626 20059 leader_election.cc:290] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 4 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:33.342896 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:33.320267 20084 raft_consensus.cc:2749] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:21:33.344859 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:33.338739 20157 raft_consensus.cc:515] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:33.345845 20018 leader_election.cc:290] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:21:33.311451 19978 consensus_meta.cc:287] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.901s	user 0.011s	sys 0.000s
I20260709 12:21:33.279628 19895 raft_consensus.cc:515] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:33.309111 19826 raft_consensus.cc:2749] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:21:33.363070 19757 leader_election.cc:290] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 5 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:33.311044 19960 raft_consensus.cc:2764] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
W20260709 12:21:33.369509 18333 leader_election.cc:336] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837): Timed out: RequestConsensusVote RPC to 127.16.29.67:36837 timed out after 1.865s (SENT)
I20260709 12:21:33.316370 20102 raft_consensus.cc:3060] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:33.277778 20162 consensus_queue.cc:1048] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.089s
I20260709 12:21:33.252801 20112 consensus_queue.cc:237] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:33.384414 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:33.309980 20088 leader_election.cc:290] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 5 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:21:33.385452 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 156893us
I20260709 12:21:33.387758 17652 leader_election.cc:304] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:21:33.392079 19806 leader_election.cc:290] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 6 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:33.394850 20133 leader_election.cc:290] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:33.365428 20139 consensus_queue.cc:237] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:33.382133 20019 raft_consensus.cc:2749] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:21:33.396090 17835 raft_consensus.cc:1275] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 12:21:33.399399 17835 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35042 (request call id 1659) took 1418 ms. Trace:
I20260709 12:21:33.399722 17835 rpcz_store.cc:276] 0709 12:21:31.980928 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:33.395379 (+1414451us) service_pool.cc:224] Handling call
0709 12:21:33.395734 (+   355us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:33.399153 (+  3419us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:33.399344 (+   191us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:33.267867 20174 raft_consensus.cc:515] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:21:33.384644 20171 log.cc:927] Time spent T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.104s	user 0.000s	sys 0.001s
I20260709 12:21:33.380822 19978 raft_consensus.cc:515] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:33.401461 17835 raft_consensus.cc:1240] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 5. Current term is 6. Ops: []
I20260709 12:21:33.401091 20012 raft_consensus.cc:493] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:33.392153 17829 raft_consensus.cc:1275] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260709 12:21:33.406975 17838 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37994 (request call id 555) took 2403 ms. Trace:
I20260709 12:21:33.409775 17838 rpcz_store.cc:276] 0709 12:21:31.003407 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:32.363868 (+1360461us) service_pool.cc:224] Handling call
0709 12:21:32.367357 (+  3489us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:32.797419 (+430062us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:33.406754 (+609335us) spinlock_profiling.cc:243] Waited 52.9 ms on lock 0x7b54002eac54. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:21:33.406912 (+   158us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":121681792}
I20260709 12:21:33.407366 17835 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35042 (request call id 1660) took 1398 ms. Trace:
I20260709 12:21:33.408839 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:33.411346 17683 raft_consensus.cc:2368] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 3. Current term is 4.
I20260709 12:21:33.412246 17835 rpcz_store.cc:276] 0709 12:21:32.009224 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:33.400497 (+1391273us) service_pool.cc:224] Handling call
0709 12:21:33.401244 (+   747us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:33.407141 (+  5897us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:33.407309 (+   168us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:33.415027 17835 raft_consensus.cc:1275] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:21:33.405022 19806 raft_consensus.cc:2749] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:21:33.415689 17835 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35042 (request call id 1661) took 1359 ms. Trace:
I20260709 12:21:33.416085 17835 rpcz_store.cc:276] 0709 12:21:32.056473 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:33.413217 (+1356744us) service_pool.cc:224] Handling call
0709 12:21:33.414531 (+  1314us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:33.415398 (+   867us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:33.415637 (+   239us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:21:33.416596 17804 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:33.417671 17681 raft_consensus.cc:1240] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 4. Current term is 5. Ops: []
I20260709 12:21:33.418169 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
W20260709 12:21:33.418512 17655 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:33.418784 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:33.406769 20128 raft_consensus.cc:2749] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:21:33.420421 17802 leader_election.cc:400] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 3 election: Vote denied by peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071) with higher term. Message: Invalid argument: T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 3. Current term is 4.
I20260709 12:21:33.422405 17802 leader_election.cc:403] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
I20260709 12:21:33.426978 17838 raft_consensus.cc:1275] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:21:33.429788 17652 leader_election.cc:304] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:21:33.430850 17652 leader_election.cc:304] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:33.432024 17652 leader_election.cc:304] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:21:33.419716 17832 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35042 (request call id 1652) took 1611 ms. Trace:
I20260709 12:21:33.429833 20169 raft_consensus.cc:3060] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:21:33.436270 17730 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:33.434762 17727 leader_election.cc:304] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:21:33.451541 18362 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59886 (request call id 757) took 1118 ms. Trace:
I20260709 12:21:33.435180 17652 leader_election.cc:304] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:33.454674 17727 leader_election.cc:304] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:21:33.423959 20175 consensus_queue.cc:237] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:33.437436 20064 raft_consensus.cc:493] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:21:33.433503 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:33.456965 20064 raft_consensus.cc:515] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:33.457715 17680 raft_consensus.cc:3060] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:33.435842 19826 raft_consensus.cc:493] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:33.440330 17832 rpcz_store.cc:276] 0709 12:21:31.808399 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:33.345749 (+1537350us) service_pool.cc:224] Handling call
0709 12:21:33.346087 (+   338us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:33.387004 (+ 40917us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:33.401792 (+ 14788us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:33.402320 (+   528us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:21:33.406911 (+  4591us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:33.411044 (+  4133us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:33.416628 (+  5584us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":35161,"raft.queue_time_us":38606,"raft.run_cpu_time_us":118,"raft.run_wall_time_us":6306}
I20260709 12:21:33.459131 20064 leader_election.cc:290] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:33.459208 19826 raft_consensus.cc:515] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:33.421962 17829 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35042 (request call id 1654) took 1612 ms. Trace:
I20260709 12:21:33.460659 17829 rpcz_store.cc:276] 0709 12:21:31.809077 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:33.384335 (+1575258us) service_pool.cc:224] Handling call
0709 12:21:33.391800 (+  7465us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:33.410400 (+ 18600us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:33.420800 (+ 10400us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:33.461115 20174 leader_election.cc:290] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:33.461449 19826 leader_election.cc:290] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:33.433236 19917 raft_consensus.cc:493] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:21:33.450047 17836 raft_consensus.cc:1240] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 2. Current term is 3. Ops: [2.5-2.5]
I20260709 12:21:33.462241 17829 raft_consensus.cc:1275] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 6. (term mismatch)
I20260709 12:21:33.462596 17829 pending_rounds.cc:85] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b: Aborting all ops after (but not including) 5
I20260709 12:21:33.462836 17829 pending_rounds.cc:107] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b: Aborting uncommitted NO_OP operation due to leader change: 3.6
I20260709 12:21:33.463066 17829 raft_consensus.cc:2889] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:21:33.463313 17836 raft_consensus.cc:1275] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260709 12:21:33.464387 17836 raft_consensus.cc:1240] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 5. Current term is 6. Ops: []
I20260709 12:21:33.453012 18362 rpcz_store.cc:276] 0709 12:21:32.333441 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:32.334681 (+  1240us) service_pool.cc:224] Handling call
0709 12:21:32.335080 (+   399us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:33.186220 (+851140us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:33.450943 (+264723us) spinlock_profiling.cc:243] Waited 811 ms on lock 0x7b54007602d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:21:33.451357 (+   414us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1865108864}
I20260709 12:21:33.412557 20012 raft_consensus.cc:515] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:33.465087 17832 consensus_queue.cc:237] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:33.467135 20012 leader_election.cc:290] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:33.447427 20081 leader_election.cc:290] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:33.468036 19917 raft_consensus.cc:515] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:33.468813 20088 consensus_queue.cc:1059] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:21:33.469590 20088 raft_consensus.cc:3055] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:21:33.469929 20162 raft_consensus.cc:493] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:21:33.470065 19917 leader_election.cc:290] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:33.470360 19838 raft_consensus.cc:493] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:33.470665 19917 raft_consensus.cc:493] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:21:33.470556 20162 raft_consensus.cc:515] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:33.470898 19838 raft_consensus.cc:515] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:33.471184 19917 raft_consensus.cc:515] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:33.471561 20158 raft_consensus.cc:493] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:33.472026 20173 raft_consensus.cc:2804] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:21:33.472517 20173 raft_consensus.cc:493] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:33.472143 20158 raft_consensus.cc:515] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:33.473029 19838 leader_election.cc:290] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:33.435738 20186 raft_consensus.cc:2749] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:21:33.435245 20185 raft_consensus.cc:2749] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:21:33.470016 20088 raft_consensus.cc:740] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:21:33.474733 20193 raft_consensus.cc:493] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:21:33.475824 19978 leader_election.cc:290] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 3 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:33.476145 17727 leader_election.cc:304] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:21:33.475658 20088 consensus_queue.cc:260] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:33.441776 19848 raft_consensus.cc:515] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:33.478206 20091 raft_consensus.cc:3060] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:33.475694 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:33.481710 17764 raft_consensus.cc:2393] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 6: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:21:33.472880 20173 raft_consensus.cc:3060] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:33.482930 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:33.477959 19826 raft_consensus.cc:2749] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:21:33.494508 17836 raft_consensus.cc:1240] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 5. Current term is 6. Ops: []
I20260709 12:21:33.495534 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:33.496989 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:33.498227 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
W20260709 12:21:33.489996 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 54909.2us
I20260709 12:21:33.500022 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:33.477509 17727 leader_election.cc:304] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:21:33.434527 19806 raft_consensus.cc:2804] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:21:33.504863 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:33.506218 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:21:33.506750 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 223598us
I20260709 12:21:33.507570 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
W20260709 12:21:33.508608 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 199340us
I20260709 12:21:33.505699 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:33.474980 20158 leader_election.cc:290] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:33.479776 19848 leader_election.cc:290] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 6 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:33.512506 17727 leader_election.cc:304] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
W20260709 12:21:33.513625 17652 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:33.493543 20088 raft_consensus.cc:3060] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:33.494215 20081 raft_consensus.cc:2749] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:21:33.440268 20176 consensus_queue.cc:1059] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.070s
I20260709 12:21:33.491359 20019 raft_consensus.cc:493] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:21:33.518715 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:33.711014 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:33.513087 19806 raft_consensus.cc:697] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 4 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:21:33.527807 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:21:33.733525 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 66617.9us
I20260709 12:21:33.528224 20185 consensus_queue.cc:1059] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 12:21:33.480058 20193 raft_consensus.cc:515] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:33.516769 20064 raft_consensus.cc:2749] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:21:33.571569 20018 consensus_queue.cc:1048] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.043s
I20260709 12:21:33.572086 20162 leader_election.cc:290] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:33.576159 20165 raft_consensus.cc:493] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:33.529619 19826 raft_consensus.cc:2749] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 12:21:33.577704 20175 raft_consensus.cc:493] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
W20260709 12:21:33.607647 17801 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:33.467763 19893 raft_consensus.cc:515] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:33.608922 19895 leader_election.cc:290] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 7 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:33.616078 17680 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 4.
I20260709 12:21:33.621155 20127 raft_consensus.cc:515] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:33.753708 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:21:33.622502 20136 consensus_meta.cc:287] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.773s	user 0.009s	sys 0.005s
I20260709 12:21:33.754426 19893 leader_election.cc:290] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 4 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:33.755035 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:33.754933 20136 raft_consensus.cc:515] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:21:33.755736 20135 consensus_meta.cc:287] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.570s	user 0.010s	sys 0.000s
I20260709 12:21:33.629951 20139 raft_consensus.cc:493] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:21:33.750123 20165 raft_consensus.cc:515] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:33.756981 20139 raft_consensus.cc:515] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:33.757670 20136 leader_election.cc:290] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 7 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:33.646265 19917 leader_election.cc:290] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:33.522650 18334 leader_election.cc:304] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:21:33.765651 20193 leader_election.cc:290] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:21:33.766031 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 243489us
I20260709 12:21:33.767783 19806 consensus_queue.cc:237] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:33.759660 20175 raft_consensus.cc:515] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:33.769635 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:33.770664 20175 leader_election.cc:290] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:33.771505 20168 raft_consensus.cc:3055] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 LEADER]: Stepping down as leader of term 2
W20260709 12:21:33.656896 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:33.777311 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:33.671677 17830 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35040 (request call id 1880) took 2678 ms. Trace:
I20260709 12:21:33.711810 17764 raft_consensus.cc:3060] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:33.770720 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:33.738646 17762 raft_consensus.cc:2468] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 7.
I20260709 12:21:33.736858 20195 consensus_queue.cc:1059] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.003s
I20260709 12:21:33.579298 20169 raft_consensus.cc:2749] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071) with higher term. Message: Invalid argument: T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 3. Current term is 4.
I20260709 12:21:33.738152 17728 leader_election.cc:304] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:21:33.739770 19994 consensus_queue.cc:1059] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.002s
I20260709 12:21:33.712644 20019 raft_consensus.cc:515] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:33.742295 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:33.744472 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:33.739086 20176 consensus_queue.cc:1243] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer e2909cff1da44ad387f96a8ae3aa5afb log is divergent from this leader: its last log entry 3.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
W20260709 12:21:33.625232 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 46233.9us
I20260709 12:21:33.758297 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:33.754506 20127 leader_election.cc:290] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 4 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:33.798589 17756 raft_consensus.cc:2410] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 4 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260709 12:21:33.789546 17801 leader_election.cc:304] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:33.758481 20165 leader_election.cc:290] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:33.521425 20158 raft_consensus.cc:493] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:21:33.759093 17683 raft_consensus.cc:1240] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 6. Current term is 7. Ops: []
I20260709 12:21:33.806126 20158 raft_consensus.cc:515] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:33.807068 17829 raft_consensus.cc:2393] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 6: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:21:33.806943 20165 raft_consensus.cc:3055] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:21:33.807682 20165 raft_consensus.cc:740] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:21:33.808264 20158 leader_election.cc:290] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:33.771919 20164 raft_consensus.cc:3055] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 1 LEADER]: Stepping down as leader of term 1
W20260709 12:21:33.759227 19868 consensus_meta.cc:287] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.531s	user 0.010s	sys 0.000s
I20260709 12:21:33.773227 20135 raft_consensus.cc:2804] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260709 12:21:33.654968 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 142480us
I20260709 12:21:33.778373 17761 raft_consensus.cc:3060] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:33.777813 17830 rpcz_store.cc:276] 0709 12:21:30.993006 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:32.113711 (+1120705us) service_pool.cc:224] Handling call
0709 12:21:32.114001 (+   290us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:33.168567 (+1054566us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:33.671308 (+502741us) spinlock_profiling.cc:243] Waited 1.02 s on lock 0x7b54003d4ad4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:21:33.671588 (+   280us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2346863872}
I20260709 12:21:33.821751 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:33.822512 17830 raft_consensus.cc:3060] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:33.755848 17680 raft_consensus.cc:2393] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 4: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:21:33.789407 17684 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37402 (request call id 2194) took 2535 ms. Trace:
W20260709 12:21:33.794425 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 56398.3us
I20260709 12:21:33.824819 17684 rpcz_store.cc:276] 0709 12:21:31.253683 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:31.599191 (+345508us) service_pool.cc:224] Handling call
0709 12:21:31.599432 (+   241us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:31.977760 (+378328us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:33.789188 (+1811428us) spinlock_profiling.cc:243] Waited 336 ms on lock 0x7b5400426dd4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:21:33.789349 (+   161us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":772408064}
I20260709 12:21:33.826936 17728 leader_election.cc:304] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:21:33.795835 20173 raft_consensus.cc:515] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:33.808094 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:33.796339 19855 raft_consensus.cc:493] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:21:33.839730 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:33.783943 20168 raft_consensus.cc:740] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:21:33.808504 18364 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59886 (request call id 762) took 1462 ms. Trace:
I20260709 12:21:33.840946 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:21:33.794569 20147 consensus_meta.cc:287] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.603s	user 0.008s	sys 0.004s
I20260709 12:21:33.811820 17764 raft_consensus.cc:2468] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 4.
I20260709 12:21:33.821539 20164 raft_consensus.cc:740] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:21:33.784153 20196 raft_consensus.cc:3055] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:21:33.846791 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:33.819515 20091 raft_consensus.cc:2749] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:21:33.832857 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:33.793052 20102 raft_consensus.cc:493] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:21:33.821161 20127 consensus_queue.cc:1048] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.024s
I20260709 12:21:33.846756 20147 raft_consensus.cc:515] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:33.850610 20102 raft_consensus.cc:515] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:33.850967 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:33.762594 20073 leader_election.cc:290] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 4 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:33.851603 17680 raft_consensus.cc:2468] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 7.
I20260709 12:21:33.836124 19994 raft_consensus.cc:2749] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:21:33.852609 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:33.852875 20102 leader_election.cc:290] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:33.853011 17685 raft_consensus.cc:1240] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20260709 12:21:33.853755 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:33.854576 17680 raft_consensus.cc:2393] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 5: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:21:33.820122 20152 raft_consensus.cc:493] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:33.855367 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:33.855659 20152 raft_consensus.cc:515] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:33.856237 17829 raft_consensus.cc:2393] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 2: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:21:33.857129 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:33.857717 17829 raft_consensus.cc:2468] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 4.
I20260709 12:21:33.857806 20152 leader_election.cc:290] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:33.858644 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:33.859784 20173 leader_election.cc:290] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 5 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:33.860911 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:33.861747 17838 raft_consensus.cc:2468] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 4.
I20260709 12:21:33.861579 17728 leader_election.cc:304] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:21:33.803484 19826 raft_consensus.cc:3055] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:21:33.822930 20019 leader_election.cc:290] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:33.841075 18364 rpcz_store.cc:276] 0709 12:21:32.346274 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:32.500806 (+154532us) service_pool.cc:224] Handling call
0709 12:21:32.753224 (+252418us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:32.852464 (+ 99240us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:33.808289 (+955825us) spinlock_profiling.cc:243] Waited 97.9 ms on lock 0x7b54005de8d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:21:33.808440 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":225143040}
I20260709 12:21:33.809002 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:33.843356 17761 raft_consensus.cc:2468] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 4.
I20260709 12:21:33.870059 17681 raft_consensus.cc:2393] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 4: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:21:33.870398 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:33.836840 17727 leader_election.cc:304] T 7cb3768d215e409fb9f5baf18d124442 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:21:33.844141 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:33.870988 17761 raft_consensus.cc:2468] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 4.
I20260709 12:21:33.820259 20135 raft_consensus.cc:493] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:21:33.822042 20153 raft_consensus.cc:2749] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 12:21:33.792120 20139 leader_election.cc:290] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:33.837628 17830 raft_consensus.cc:2468] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 4.
I20260709 12:21:33.850054 17686 raft_consensus.cc:2393] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 4: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:21:33.866317 17802 leader_election.cc:304] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:33.868559 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:33.874495 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:33.847190 20196 raft_consensus.cc:740] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:21:33.808683 20165 consensus_queue.cc:260] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:33.879220 20164 consensus_queue.cc:260] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:33.879557 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:33.874981 17756 raft_consensus.cc:2468] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 7.
W20260709 12:21:33.879559 20195 consensus_queue.cc:1175] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "e063541542a94969898868bf6eb5ce1b" responder_term: 2 status { last_received { term: 2 index: 2 } last_committed_idx: 2 last_received_current_leader { term: 2 index: 2 } }
I20260709 12:21:33.880384 20164 raft_consensus.cc:3060] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:33.880854 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:33.881649 17756 raft_consensus.cc:3060] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:33.882531 19994 consensus_queue.cc:1059] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:21:33.883278 20162 consensus_queue.cc:1048] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:21:33.872017 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:33.884024 19807 consensus_queue.cc:1048] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:21:33.871044 19976 consensus_queue.cc:1059] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.005s
I20260709 12:21:33.876111 19868 raft_consensus.cc:2749] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260709 12:21:33.884634 17837 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35042 (request call id 1656) took 2014 ms (client timeout 2401 ms). Trace:
I20260709 12:21:33.870551 20064 raft_consensus.cc:2804] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:21:33.884763 20162 consensus_queue.cc:1059] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:21:33.885205 20064 raft_consensus.cc:493] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:33.885653 20064 raft_consensus.cc:3060] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:21:33.886183 17833 raft_consensus.cc:1240] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 2. Current term is 4. Ops: []
I20260709 12:21:33.876726 20168 consensus_queue.cc:260] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:33.854099 20147 leader_election.cc:290] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 6 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:33.886514 20196 consensus_queue.cc:260] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:33.870677 20191 raft_consensus.cc:2749] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:21:33.877916 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:33.888302 20153 consensus_queue.cc:1059] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:21:33.888772 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:33.889114 20153 raft_consensus.cc:3055] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:21:33.889381 17681 raft_consensus.cc:2468] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 4.
I20260709 12:21:33.889479 20153 raft_consensus.cc:740] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 3 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:21:33.890247 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:33.890278 20153 consensus_queue.cc:260] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:21:33.885066 17837 rpcz_store.cc:269] 0709 12:21:31.870547 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:32.016954 (+146407us) service_pool.cc:224] Handling call
0709 12:21:33.884466 (+1867512us) spinlock_profiling.cc:243] Waited 297 ms on lock 0x7b540022ae54. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:21:33.884583 (+   117us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":682013056}
I20260709 12:21:33.891968 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:33.892343 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:33.892534 17681 raft_consensus.cc:2468] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 7.
I20260709 12:21:33.893045 17837 raft_consensus.cc:2468] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 4.
I20260709 12:21:33.893329 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:33.879655 20165 raft_consensus.cc:3060] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:33.893886 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:33.894069 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:33.881167 17835 raft_consensus.cc:3055] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:21:33.894976 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:33.883330 19994 consensus_queue.cc:1048] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 12:21:33.869270 19826 raft_consensus.cc:740] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:21:33.843570 19855 raft_consensus.cc:515] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:33.896765 19826 consensus_queue.cc:260] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:33.897432 19868 raft_consensus.cc:3055] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:21:33.877210 20173 raft_consensus.cc:2749] T 7cb3768d215e409fb9f5baf18d124442 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:21:33.895102 17835 raft_consensus.cc:740] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 4 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:21:33.897589 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:33.897981 19868 raft_consensus.cc:740] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 4 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:21:33.887498 17833 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35040 (request call id 1868) took 3180 ms. Trace:
I20260709 12:21:33.899085 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:33.874404 20135 raft_consensus.cc:3060] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:33.898715 17835 consensus_queue.cc:260] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:33.900797 17835 raft_consensus.cc:3060] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:33.898900 19868 consensus_queue.cc:260] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:33.901583 20167 raft_consensus.cc:493] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:33.901787 20155 leader_election.cc:290] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:33.898713 17833 rpcz_store.cc:276] 0709 12:21:30.707385 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:32.110056 (+1402671us) service_pool.cc:224] Handling call
0709 12:21:33.886014 (+1775958us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:33.886558 (+   544us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:33.887321 (+   763us) spinlock_profiling.cc:243] Waited 1.76 s on lock 0x7b540022ae50. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a61f8ad 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:21:33.887443 (+   122us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4053656320}
I20260709 12:21:33.902130 20167 raft_consensus.cc:515] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:33.902838 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:33.903213 17763 raft_consensus.cc:1275] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260709 12:21:33.903517 17833 raft_consensus.cc:2468] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 5.
I20260709 12:21:33.884656 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:33.904208 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:33.869534 20152 raft_consensus.cc:493] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:33.904562 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:33.904799 17763 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 3.
I20260709 12:21:33.905241 20052 leader_election.cc:290] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 2 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:33.905854 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:33.906522 17689 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37396 (request call id 1781) took 1876 ms. Trace:
I20260709 12:21:33.906765 17833 raft_consensus.cc:2393] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 6: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:21:33.906879 17689 rpcz_store.cc:276] 0709 12:21:32.030362 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:32.030480 (+   118us) service_pool.cc:224] Handling call
0709 12:21:32.068873 (+ 38393us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:33.228451 (+1159578us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:33.906308 (+677857us) spinlock_profiling.cc:243] Waited 1.16 s on lock 0x7b54002530d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:21:33.906467 (+   159us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2665449216}
I20260709 12:21:33.908255 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:33.909503 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:33.904389 17761 raft_consensus.cc:2393] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 6: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
W20260709 12:21:33.906301 17728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:33.859793 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:33.911115 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:33.912269 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:33.905786 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:33.880793 20018 consensus_queue.cc:1048] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:21:33.913590 20074 raft_consensus.cc:493] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:21:33.913743 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:33.914093 20074 raft_consensus.cc:3060] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:33.914660 17763 raft_consensus.cc:2410] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 3 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
W20260709 12:21:33.916519 17802 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:33.887413 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:33.875155 17838 raft_consensus.cc:2468] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 4.
I20260709 12:21:33.895670 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:33.921589 17837 raft_consensus.cc:2468] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 3.
I20260709 12:21:33.904860 20152 raft_consensus.cc:515] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:33.933225 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:33.894086 17681 raft_consensus.cc:2393] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 4: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:21:33.926381 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:33.917418 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:33.937742 17758 raft_consensus.cc:2393] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 6: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:21:33.938571 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:33.939038 17758 raft_consensus.cc:3060] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:33.926815 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:33.907764 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:33.887672 20196 raft_consensus.cc:3060] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:33.941288 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:33.946010 17835 raft_consensus.cc:1275] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 3 index: 6. (term mismatch)
I20260709 12:21:33.946403 17835 pending_rounds.cc:85] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b: Aborting all ops after (but not including) 5
I20260709 12:21:33.946619 17835 pending_rounds.cc:107] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b: Aborting uncommitted NO_OP operation due to leader change: 4.6
I20260709 12:21:33.946847 17835 raft_consensus.cc:2889] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:21:33.927258 17688 raft_consensus.cc:3060] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:33.928807 20173 consensus_queue.cc:1048] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 12:21:33.931172 17802 leader_election.cc:304] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:33.931599 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:33.932613 20153 raft_consensus.cc:3060] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:33.933929 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:33.948504 17834 raft_consensus.cc:2468] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 2.
I20260709 12:21:33.948896 17802 leader_election.cc:304] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:33.949090 17836 raft_consensus.cc:2468] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 4.
I20260709 12:21:33.949393 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:33.949980 17834 raft_consensus.cc:2468] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 4.
I20260709 12:21:33.950800 17834 raft_consensus.cc:1240] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 2. Current term is 3. Ops: []
I20260709 12:21:33.951454 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:33.907764 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:33.952142 17834 raft_consensus.cc:2393] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 3: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:21:33.952296 17685 raft_consensus.cc:2468] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 5.
I20260709 12:21:33.939916 17830 raft_consensus.cc:2393] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 4: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:21:33.954967 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:33.955904 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:33.956308 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:33.956445 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:33.956918 17834 raft_consensus.cc:3060] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:33.957021 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
W20260709 12:21:33.872761 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 72187.3us
I20260709 12:21:33.957821 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:33.957830 17835 raft_consensus.cc:3060] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:33.932819 19826 raft_consensus.cc:3060] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:33.958388 17652 leader_election.cc:304] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:21:33.955469 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:33.961463 17830 raft_consensus.cc:1240] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 3. Current term is 4. Ops: []
I20260709 12:21:33.962771 17652 leader_election.cc:304] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:21:33.961512 17727 leader_election.cc:304] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:21:33.963702 17652 leader_election.cc:304] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:21:33.939045 19868 raft_consensus.cc:3060] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:33.967566 17688 raft_consensus.cc:1275] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 4. (term mismatch)
I20260709 12:21:33.969698 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:33.971474 17682 raft_consensus.cc:3060] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:21:33.967792 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:33.979992 17688 pending_rounds.cc:85] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3: Aborting all ops after (but not including) 3
I20260709 12:21:33.981416 17688 pending_rounds.cc:107] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3: Aborting uncommitted NO_OP operation due to leader change: 4.4
I20260709 12:21:33.983053 17688 raft_consensus.cc:2889] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:21:33.997787 17688 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37396 (request call id 1789) took 1876 ms. Trace:
I20260709 12:21:33.998716 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:33.998821 17688 rpcz_store.cc:276] 0709 12:21:32.121195 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:32.155369 (+ 34174us) service_pool.cc:224] Handling call
0709 12:21:32.155772 (+   403us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:33.885527 (+1729755us) spinlock_profiling.cc:243] Waited 11.9 ms on lock 0x7b58006101d0. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a57c0aa 00007f7b4a5fe447 00007f7b4a616fe7 00007f7b4a6187de 00007f7b4a622b49 00007f7b4a623416 00007f7b4a61fd8f 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4
0709 12:21:33.997530 (+112003us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:33.997731 (+   201us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":27423488}
I20260709 12:21:33.954475 20139 raft_consensus.cc:2749] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 12:21:33.962100 20095 raft_consensus.cc:2804] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:21:34.000734 20095 raft_consensus.cc:493] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:34.001125 20095 raft_consensus.cc:3060] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:33.964151 17680 raft_consensus.cc:2393] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 3: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:21:33.992123 20076 leader_election.cc:290] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:33.992748 17830 raft_consensus.cc:1275] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:21:33.993634 20147 raft_consensus.cc:2804] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:21:33.964197 20127 raft_consensus.cc:2804] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:21:33.969165 20144 raft_consensus.cc:2804] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:21:33.969151 19994 raft_consensus.cc:3060] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:33.996227 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:33.996338 20167 leader_election.cc:290] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:33.996702 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:33.960021 20191 consensus_queue.cc:1059] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.043s
I20260709 12:21:33.963084 20143 leader_election.cc:290] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:34.022334 17687 raft_consensus.cc:1240] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 3. Current term is 4. Ops: []
I20260709 12:21:34.024538 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:34.025202 17759 raft_consensus.cc:2468] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 3.
I20260709 12:21:33.999040 20186 raft_consensus.cc:493] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:33.964680 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:33.988111 17833 raft_consensus.cc:1275] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:21:34.030455 17834 raft_consensus.cc:2468] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 5.
I20260709 12:21:33.989147 17727 leader_election.cc:304] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:21:34.015677 17835 raft_consensus.cc:2468] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 4.
I20260709 12:21:34.018683 18333 leader_election.cc:304] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:21:34.001780 19855 leader_election.cc:290] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:34.020757 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:21:34.029403 17803 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:34.036381 17803 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:34.036747 17803 connection.cc:256] RPC call timeout handler was delayed by -1.0422861960000773s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2183s for a response
I20260709 12:21:34.030532 17685 raft_consensus.cc:2468] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 4.
I20260709 12:21:33.969810 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:34.032502 17836 raft_consensus.cc:1240] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260709 12:21:34.033325 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:34.034078 18362 raft_consensus.cc:1240] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 3. Current term is 4. Ops: []
W20260709 12:21:34.039496 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 67324.6us
I20260709 12:21:33.964421 20158 consensus_queue.cc:1048] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.061s
I20260709 12:21:34.011998 20147 raft_consensus.cc:493] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:21:34.013573 20144 raft_consensus.cc:493] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:34.041631 17681 raft_consensus.cc:1240] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 2 LEADER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 1. Current term is 2. Ops: []
I20260709 12:21:34.042989 17682 raft_consensus.cc:2468] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 8.
I20260709 12:21:34.049011 17801 leader_election.cc:304] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:21:34.049587 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:34.057667 18362 raft_consensus.cc:3060] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:34.058645 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:34.050148 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:34.059271 17830 raft_consensus.cc:2468] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 4.
I20260709 12:21:34.060130 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
W20260709 12:21:34.061455 20202 log.cc:927] Time spent T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.051s	user 0.000s	sys 0.000s
I20260709 12:21:34.062201 19960 raft_consensus.cc:2804] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:21:34.062613 19960 raft_consensus.cc:493] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:21:34.043534 20191 raft_consensus.cc:2749] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:21:34.063423 20144 raft_consensus.cc:3060] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:34.036078 20018 raft_consensus.cc:3055] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:21:34.064064 20018 raft_consensus.cc:740] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:21:34.065707 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:34.065075 20018 consensus_queue.cc:260] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:34.066562 18363 raft_consensus.cc:2393] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 4: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:21:34.067001 20018 raft_consensus.cc:3060] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:34.052299 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:34.068245 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:34.068559 18365 raft_consensus.cc:2368] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 3. Current term is 4.
I20260709 12:21:34.052726 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:34.069090 18367 raft_consensus.cc:2393] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 5: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:21:34.069603 18364 raft_consensus.cc:2393] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 5: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:21:34.069978 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:34.070612 18367 raft_consensus.cc:2468] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 5.
I20260709 12:21:34.053679 17803 leader_election.cc:304] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:21:34.050076 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:34.062995 19960 raft_consensus.cc:3060] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:34.017426 19961 leader_election.cc:290] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:34.017129 20127 raft_consensus.cc:697] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 4 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:21:34.052309 18333 leader_election.cc:304] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:21:34.071451 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:34.082495 18367 raft_consensus.cc:2393] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 4: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:21:34.071532 18374 raft_consensus.cc:3060] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:34.070613 17685 raft_consensus.cc:1240] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 6. Current term is 7. Ops: [6.8-6.8]
I20260709 12:21:34.074966 18362 raft_consensus.cc:2468] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 6.
I20260709 12:21:34.076644 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:34.086170 18365 raft_consensus.cc:2468] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 4.
I20260709 12:21:34.088420 17803 leader_election.cc:304] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:34.078791 17758 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 4.
I20260709 12:21:34.089532 17756 raft_consensus.cc:2468] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 5.
I20260709 12:21:34.090224 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:34.090349 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:34.091022 17758 raft_consensus.cc:2410] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for term 4 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260709 12:21:34.067553 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:34.091801 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:34.038659 20019 raft_consensus.cc:493] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:34.077591 17763 raft_consensus.cc:2468] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 5.
I20260709 12:21:34.101583 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:34.102602 17763 raft_consensus.cc:3060] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:34.086855 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:34.093938 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:34.079977 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:34.094696 17801 leader_election.cc:304] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:21:34.081317 17682 raft_consensus.cc:1240] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 3. Current term is 4. Ops: []
I20260709 12:21:34.087996 18333 leader_election.cc:304] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:34.078325 17681 raft_consensus.cc:1240] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 5. Current term is 6. Ops: []
I20260709 12:21:34.044034 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:34.090957 17756 raft_consensus.cc:2468] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 5.
I20260709 12:21:34.038983 20186 raft_consensus.cc:515] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:34.098450 19855 raft_consensus.cc:2804] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:21:34.152730 17685 raft_consensus.cc:1240] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 3. Current term is 4. Ops: []
I20260709 12:21:34.171975 19855 raft_consensus.cc:697] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 4 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:21:34.172689 17756 raft_consensus.cc:1240] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 4. Current term is 5. Ops: []
I20260709 12:21:34.157401 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:34.173710 18365 raft_consensus.cc:3060] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:34.143862 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
W20260709 12:21:34.176843 20088 consensus_meta.cc:287] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.503s	user 0.005s	sys 0.005s
I20260709 12:21:34.177273 17757 raft_consensus.cc:2410] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 3 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
W20260709 12:21:34.149474 18371 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59886 (request call id 759) took 1815 ms (client timeout 1536 ms). Trace:
I20260709 12:21:34.159404 17684 raft_consensus.cc:2393] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 2: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
W20260709 12:21:34.159554 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:34.160218 17831 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b"
dest_uuid: "e063541542a94969898868bf6eb5ce1b"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:35032
I20260709 12:21:34.108352 18363 raft_consensus.cc:2393] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 4: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:21:34.180891 17831 raft_consensus.cc:606] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 6 LEADER]: Received request to transfer leadership
I20260709 12:21:34.099301 17803 leader_election.cc:304] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:21:34.178831 18371 rpcz_store.cc:269] 0709 12:21:32.334125 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:32.339165 (+  5040us) service_pool.cc:224] Handling call
0709 12:21:34.149262 (+1810097us) spinlock_profiling.cc:243] Waited 562 ms on lock 0x7b54006fe354. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:21:34.149402 (+   140us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1292610432}
I20260709 12:21:34.143780 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:34.040745 20143 consensus_queue.cc:1059] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.061s
I20260709 12:21:34.038394 20139 consensus_queue.cc:1048] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.053s
I20260709 12:21:34.150403 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:34.310094 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:34.311331 18363 raft_consensus.cc:3060] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:34.055796 20147 raft_consensus.cc:3060] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:21:34.311482 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:34.312654 18367 raft_consensus.cc:2393] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 5: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
W20260709 12:21:34.313282 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 225398us
I20260709 12:21:34.182750 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:34.273253 17757 raft_consensus.cc:1240] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 1. Current term is 2. Ops: []
W20260709 12:21:34.254025 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 159360us
I20260709 12:21:34.324811 18371 raft_consensus.cc:2393] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 2: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:21:34.325819 17763 raft_consensus.cc:2468] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 4.
I20260709 12:21:34.146579 19883 leader_election.cc:290] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 4 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:34.317777 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:34.318223 18333 leader_election.cc:304] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:34.105213 17758 raft_consensus.cc:1240] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 3. Current term is 4. Ops: []
I20260709 12:21:34.339711 17759 raft_consensus.cc:1240] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260709 12:21:34.095911 20176 raft_consensus.cc:2749] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 12:21:34.340246 18333 leader_election.cc:304] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:21:34.234773 17755 raft_consensus.cc:3060] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:34.172905 19855 consensus_queue.cc:237] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:34.155341 20135 raft_consensus.cc:515] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:34.346908 20135 leader_election.cc:290] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 4 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:34.347456 20135 raft_consensus.cc:2804] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:21:34.144282 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:34.348407 18364 raft_consensus.cc:1240] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20260709 12:21:34.266026 18365 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 4.
I20260709 12:21:34.351672 18365 raft_consensus.cc:1240] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 3. Current term is 4. Ops: []
I20260709 12:21:34.352159 18366 raft_consensus.cc:2393] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 4: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:21:34.332472 20169 raft_consensus.cc:2804] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:21:34.352849 18365 raft_consensus.cc:1275] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:21:34.211382 18374 raft_consensus.cc:2468] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 7.
I20260709 12:21:34.352895 20169 raft_consensus.cc:697] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 7 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:21:34.347823 20135 raft_consensus.cc:493] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:34.354838 20169 consensus_queue.cc:237] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.8, Last appended by leader: 8, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:34.356712 20135 raft_consensus.cc:3060] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:34.092538 19976 raft_consensus.cc:2749] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:21:34.326103 18371 raft_consensus.cc:1240] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 5. Current term is 6. Ops: []
I20260709 12:21:34.080862 20062 raft_consensus.cc:2764] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20260709 12:21:34.327147 17757 raft_consensus.cc:1275] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
W20260709 12:21:34.254388 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:34.095757 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:34.169504 20152 leader_election.cc:290] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:34.348536 17680 raft_consensus.cc:2410] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for term 6 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260709 12:21:34.299170 20052 raft_consensus.cc:2804] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Leader election won for term 4
W20260709 12:21:34.377265 17801 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:34.378284 17654 leader_election.cc:304] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:21:34.382149 20164 consensus_meta.cc:287] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.501s	user 0.009s	sys 0.006s
I20260709 12:21:34.381839 20133 consensus_queue.cc:1059] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:21:34.382890 20178 raft_consensus.cc:2749] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:21:34.383598 20178 raft_consensus.cc:3055] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:21:34.383936 20178 raft_consensus.cc:740] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:21:34.381132 17654 leader_election.cc:304] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:34.362177 18363 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 7.
I20260709 12:21:34.363404 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:34.365002 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:34.386117 18374 raft_consensus.cc:2468] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 5.
I20260709 12:21:34.386719 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:34.387133 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:34.387485 18365 raft_consensus.cc:2468] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 4.
I20260709 12:21:34.387936 18374 raft_consensus.cc:2468] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 4.
I20260709 12:21:34.388522 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:34.389412 18365 raft_consensus.cc:2393] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 4: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:21:34.272367 20088 raft_consensus.cc:2804] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:21:34.371378 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
W20260709 12:21:34.318789 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 219557us
I20260709 12:21:34.390796 17654 leader_election.cc:304] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:21:34.396536 17654 leader_election.cc:304] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:34.398028 17654 leader_election.cc:304] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
W20260709 12:21:34.393826 17655 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:34.391719 18367 raft_consensus.cc:1240] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 6. Current term is 7. Ops: []
I20260709 12:21:34.393673 18365 raft_consensus.cc:1275] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:21:34.377982 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:34.401211 18371 raft_consensus.cc:2393] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 4: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:21:34.404388 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:34.405684 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:34.406853 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:34.411244 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:34.414109 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:34.417505 18363 raft_consensus.cc:1240] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 3. Current term is 4. Ops: []
W20260709 12:21:34.402398 20064 consensus_meta.cc:287] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.507s	user 0.010s	sys 0.005s
I20260709 12:21:34.422706 17759 raft_consensus.cc:2393] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 2: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:21:34.424922 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:34.178767 20019 raft_consensus.cc:515] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:34.394124 20133 raft_consensus.cc:2749] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:21:34.427978 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:34.410051 20143 raft_consensus.cc:2804] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:21:34.431353 17803 leader_election.cc:304] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:34.409835 20175 raft_consensus.cc:2749] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:21:34.403589 20052 raft_consensus.cc:697] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 4 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:21:34.412330 19753 raft_consensus.cc:2804] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:21:34.394407 20088 raft_consensus.cc:493] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
W20260709 12:21:34.423203 20074 consensus_meta.cc:287] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.508s	user 0.012s	sys 0.000s
I20260709 12:21:34.437166 17803 leader_election.cc:304] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:21:34.433419 17764 raft_consensus.cc:2410] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for term 5 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260709 12:21:34.442740 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:34.433956 20153 consensus_queue.cc:1059] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.011s
I20260709 12:21:34.444519 17829 raft_consensus.cc:2410] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 12:21:34.408385 20062 consensus_queue.cc:1048] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.026s
I20260709 12:21:34.384662 20178 consensus_queue.cc:260] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:21:34.496891 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 59820.8us
I20260709 12:21:34.337900 20127 consensus_queue.cc:237] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:21:34.500092 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 204419us
W20260709 12:21:34.486996 17804 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:34.448220 20088 raft_consensus.cc:3060] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:34.505679 17764 raft_consensus.cc:1240] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 2. Current term is 3. Ops: []
I20260709 12:21:34.444905 20164 raft_consensus.cc:2749] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:21:34.433210 20064 raft_consensus.cc:515] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:34.563987 20064 leader_election.cc:290] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 8 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:34.497750 20139 raft_consensus.cc:2749] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:21:34.565927 17759 raft_consensus.cc:2393] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 8: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:21:34.495586 20176 raft_consensus.cc:493] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:21:34.567097 17762 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38786 (request call id 2366) took 1097 ms. Trace:
I20260709 12:21:34.567028 20176 raft_consensus.cc:515] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:34.445099 19976 consensus_queue.cc:1048] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.079s
I20260709 12:21:34.567627 17762 rpcz_store.cc:276] 0709 12:21:33.469210 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:33.475998 (+  6788us) service_pool.cc:224] Handling call
0709 12:21:34.566888 (+1090890us) spinlock_profiling.cc:243] Waited 37.2 ms on lock 0x7b54002b5554. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:21:34.567024 (+   136us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":85624192}
I20260709 12:21:34.569495 20176 leader_election.cc:290] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:34.549077 20019 leader_election.cc:290] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:34.549533 18361 raft_consensus.cc:2393] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 4: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:21:34.552495 19960 raft_consensus.cc:515] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:34.553622 20144 raft_consensus.cc:515] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:21:34.556104 19994 consensus_meta.cc:287] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.526s	user 0.007s	sys 0.003s
I20260709 12:21:34.496352 19753 raft_consensus.cc:697] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 7 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:21:34.557090 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:34.560184 17836 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35040 (request call id 1897) took 1056 ms. Trace:
W20260709 12:21:34.573503 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 233344us
I20260709 12:21:34.584455 17836 rpcz_store.cc:276] 0709 12:21:33.503626 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:33.949908 (+446282us) service_pool.cc:224] Handling call
0709 12:21:34.031028 (+ 81120us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:34.039471 (+  8443us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:34.559950 (+520479us) spinlock_profiling.cc:243] Waited 65.4 ms on lock 0x7b54004416d0. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a61f8ad 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:21:34.560108 (+   158us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":150309632}
I20260709 12:21:34.585026 18333 leader_election.cc:304] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:21:34.585868 18333 leader_election.cc:304] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:21:34.574126 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:34.502914 20074 raft_consensus.cc:515] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:34.588442 17681 raft_consensus.cc:2393] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 8: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
W20260709 12:21:34.563776 20165 consensus_meta.cc:287] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.669s	user 0.012s	sys 0.004s
I20260709 12:21:34.572347 17829 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35040 (request call id 1894) took 1070 ms. Trace:
I20260709 12:21:34.575606 20178 raft_consensus.cc:3060] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:34.591157 19855 raft_consensus.cc:2804] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:21:34.593981 17680 raft_consensus.cc:2393] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 4: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:21:34.594266 19855 raft_consensus.cc:493] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:34.594805 19855 raft_consensus.cc:3060] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:34.588152 19994 raft_consensus.cc:2749] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:21:34.596408 17758 raft_consensus.cc:1240] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 2. Current term is 3. Ops: []
W20260709 12:21:34.599050 20196 consensus_meta.cc:287] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.639s	user 0.011s	sys 0.005s
I20260709 12:21:34.601271 20074 leader_election.cc:290] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 4 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:34.601781 20019 raft_consensus.cc:2804] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:21:34.602171 20019 raft_consensus.cc:493] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:34.602501 20019 raft_consensus.cc:3060] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:34.603691 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:34.604445 18362 raft_consensus.cc:2393] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 4: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
W20260709 12:21:34.604852 20095 consensus_meta.cc:287] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.603s	user 0.007s	sys 0.003s
I20260709 12:21:34.605274 20095 raft_consensus.cc:515] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:34.606392 17837 raft_consensus.cc:1240] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 4. Current term is 5. Ops: [4.4-4.4]
I20260709 12:21:34.607610 20167 raft_consensus.cc:3055] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:21:34.608026 20167 raft_consensus.cc:740] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:21:34.608193 20095 leader_election.cc:290] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 5 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:34.608481 17728 leader_election.cc:304] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:21:34.608836 20167 consensus_queue.cc:260] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:34.609550 18334 leader_election.cc:304] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:21:34.610271 17689 raft_consensus.cc:2410] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for term 4 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260709 12:21:34.610532 20064 raft_consensus.cc:2804] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260709 12:21:34.610708 20018 consensus_meta.cc:287] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.543s	user 0.007s	sys 0.001s
I20260709 12:21:34.611008 20064 raft_consensus.cc:493] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:21:34.611152 20152 raft_consensus.cc:2749] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:21:34.611510 20064 raft_consensus.cc:3060] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:21:34.608242 17834 raft_consensus.cc:1240] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 3. Current term is 4. Ops: []
I20260709 12:21:34.612802 17829 rpcz_store.cc:276] 0709 12:21:33.502049 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:33.859529 (+357480us) service_pool.cc:224] Handling call
0709 12:21:34.572090 (+712561us) spinlock_profiling.cc:243] Waited 516 ms on lock 0x7b54000abed4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:21:34.572268 (+   178us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1186393472}
W20260709 12:21:34.614923 20205 log.cc:927] Time spent T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.170s	user 0.001s	sys 0.001s
I20260709 12:21:34.615203 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:21:34.590420 20059 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 5 current term: 5 is higher.
I20260709 12:21:34.621778 17762 raft_consensus.cc:1275] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:21:34.616528 18333 leader_election.cc:304] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:21:34.624804 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:34.626087 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:34.627009 18371 raft_consensus.cc:2393] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 4: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:21:34.617040 17728 leader_election.cc:304] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:21:34.627861 18362 raft_consensus.cc:2393] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 4: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
W20260709 12:21:34.629789 17652 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:34.636083 18371 raft_consensus.cc:1275] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260709 12:21:34.572348 20052 consensus_queue.cc:237] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 4, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:34.618854 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
W20260709 12:21:34.636986 17652 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:34.637208 18365 raft_consensus.cc:1275] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:21:34.628485 18333 leader_election.cc:304] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
W20260709 12:21:34.640578 17652 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:34.650888 17652 leader_election.cc:304] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:34.652204 18362 raft_consensus.cc:1240] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260709 12:21:34.446751 20143 raft_consensus.cc:697] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 6 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:21:34.655619 20186 leader_election.cc:290] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:34.646860 20102 raft_consensus.cc:2804] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:21:34.657290 20102 raft_consensus.cc:493] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:34.657814 20102 raft_consensus.cc:3060] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:34.657970 20158 consensus_queue.cc:1048] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:21:34.618499 17755 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 7.
I20260709 12:21:34.637022 20152 raft_consensus.cc:2804] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:21:34.660823 19826 consensus_queue.cc:1059] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 12:21:34.661213 18363 raft_consensus.cc:1275] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 6. (term mismatch)
I20260709 12:21:34.661295 20152 raft_consensus.cc:697] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 4 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:21:34.661645 17831 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:35032 (request call id 94) took 3404 ms. Trace:
I20260709 12:21:34.661911 19826 raft_consensus.cc:3055] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:21:34.662369 19826 raft_consensus.cc:740] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:21:34.662882 20144 leader_election.cc:290] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 5 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:34.663556 19960 leader_election.cc:290] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 5 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:34.662760 17764 raft_consensus.cc:1240] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 6. Current term is 7. Ops: [4.6-6.7]
I20260709 12:21:34.664949 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:34.661603 18363 pending_rounds.cc:85] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da: Aborting all ops after (but not including) 5
I20260709 12:21:34.666879 17759 raft_consensus.cc:2410] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for term 6 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260709 12:21:34.667816 18363 pending_rounds.cc:107] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da: Aborting uncommitted NO_OP operation due to leader change: 3.6
I20260709 12:21:34.668433 18363 raft_consensus.cc:2889] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:21:34.664601 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:34.662096 17831 rpcz_store.cc:276] 0709 12:21:31.270750 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:34.160031 (+2889281us) service_pool.cc:224] Handling call
0709 12:21:34.661356 (+501325us) spinlock_profiling.cc:243] Waited 133 ms on lock 0x7b54000ecca0. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a59d520 00007f7b4a61a652 00007f7b4f5e860a 00007f7b48b6e7d4 00007f7b48b6e762 00007f7b48b6e691 00007f7b48b6e60c 00007f7b48b6d8c2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:21:34.661561 (+   205us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":305769344}
I20260709 12:21:34.657441 19855 raft_consensus.cc:515] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:34.593253 20135 raft_consensus.cc:515] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:34.642045 19994 raft_consensus.cc:2804] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:21:34.676210 18333 leader_election.cc:304] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:21:34.680037 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 295574us
I20260709 12:21:34.689121 17831 raft_consensus.cc:1275] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260709 12:21:34.671525 19779 raft_consensus.cc:493] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:34.611532 20018 consensus_queue.cc:1059] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.067s
I20260709 12:21:34.636643 20088 raft_consensus.cc:515] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:34.663374 19826 consensus_queue.cc:260] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:34.681663 20073 raft_consensus.cc:2804] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:21:34.697902 18367 raft_consensus.cc:1240] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 3. Current term is 4. Ops: []
I20260709 12:21:34.705914 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:34.707980 20196 consensus_queue.cc:1048] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20260709 12:21:34.708321 20207 log.cc:927] Time spent T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.196s	user 0.000s	sys 0.002s
I20260709 12:21:34.714533 19855 leader_election.cc:290] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 5 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:34.674218 20158 consensus_queue.cc:1059] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.022s
I20260709 12:21:34.685081 19994 raft_consensus.cc:493] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:21:34.722344 17802 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:34.722995 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:34.728555 17689 raft_consensus.cc:3060] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:34.721465 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:34.732023 20095 raft_consensus.cc:493] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:34.732157 18366 raft_consensus.cc:1240] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 4. Current term is 5. Ops: []
I20260709 12:21:34.736287 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:34.737087 17835 raft_consensus.cc:2468] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 4.
I20260709 12:21:34.730919 19994 raft_consensus.cc:3060] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:34.680616 18362 raft_consensus.cc:1240] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260709 12:21:34.740586 17802 leader_election.cc:304] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:34.742015 17757 raft_consensus.cc:2468] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 3.
I20260709 12:21:34.732721 20095 raft_consensus.cc:515] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:34.748117 20088 leader_election.cc:290] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 5 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:34.748364 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:34.766748 17760 raft_consensus.cc:2376] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 4. Re-sending same reply.
I20260709 12:21:34.767481 17760 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38786 (request call id 2370) took 1297 ms. Trace:
I20260709 12:21:34.768033 17760 rpcz_store.cc:276] 0709 12:21:33.470301 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:33.477060 (+  6759us) service_pool.cc:224] Handling call
0709 12:21:34.748215 (+1271155us) spinlock_profiling.cc:243] Waited 6.38 ms on lock 0x7b5800033398. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b474fbb71 00007f7b474fba22 00007f7b504148d9 00007f7b4f56eee0 00007f7b4f6380ab 00007f7b4f638131 00007f7b4f5e60a0 00007f7b4f5e5984 00007f7b48b606d4 00007f7b48b60662
0709 12:21:34.767404 (+ 19189us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":14677248}
W20260709 12:21:34.751572 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 122130us
I20260709 12:21:34.752940 19883 raft_consensus.cc:3055] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:21:34.744655 19826 raft_consensus.cc:3060] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:34.773908 19883 raft_consensus.cc:740] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:21:34.774899 19883 consensus_queue.cc:260] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:34.775942 19883 raft_consensus.cc:3060] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:34.736476 20143 consensus_queue.cc:237] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:34.733660 19960 raft_consensus.cc:2749] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 12:21:34.744915 19976 raft_consensus.cc:2749] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:21:34.758060 18374 raft_consensus.cc:1275] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:21:34.784309 20073 raft_consensus.cc:493] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:21:34.784912 20073 raft_consensus.cc:3060] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:21:34.759263 19868 consensus_meta.cc:287] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.756s	user 0.008s	sys 0.000s
I20260709 12:21:34.785637 17838 raft_consensus.cc:2468] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 5.
I20260709 12:21:34.770849 18334 leader_election.cc:304] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:21:34.786972 19960 raft_consensus.cc:2804] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:21:34.787496 19960 raft_consensus.cc:697] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 8 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:21:34.788393 19960 consensus_queue.cc:237] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:34.674723 19753 consensus_queue.cc:237] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:34.662375 20152 consensus_queue.cc:237] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:34.691726 20144 consensus_queue.cc:1048] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.044s
I20260709 12:21:34.763257 20064 raft_consensus.cc:515] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:21:34.760277 20209 log.cc:927] Time spent T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.063s	user 0.001s	sys 0.000s
I20260709 12:21:34.697007 19779 raft_consensus.cc:515] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:34.795229 17583 catalog_manager.cc:5697] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 6 to 7, leader changed from e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66) to 334a4e1f59944544b3b3ff36616b57da (127.16.29.68). New cstate: current_term: 7 leader_uuid: "334a4e1f59944544b3b3ff36616b57da" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:21:34.806052 19976 consensus_queue.cc:1059] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:21:34.807196 19976 raft_consensus.cc:3055] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:21:34.807554 19976 raft_consensus.cc:740] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 5 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:21:34.808439 19976 consensus_queue.cc:260] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.8, Last appended by leader: 8, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:34.810781 19855 raft_consensus.cc:493] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:34.811466 17684 raft_consensus.cc:1275] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260709 12:21:34.811336 19855 raft_consensus.cc:515] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:34.812247 20135 leader_election.cc:290] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 4 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:34.812582 17832 raft_consensus.cc:1275] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260709 12:21:34.813613 19855 leader_election.cc:290] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:34.813323 20147 raft_consensus.cc:515] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:34.818106 20102 raft_consensus.cc:515] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:34.820168 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:34.806102 17583 catalog_manager.cc:5697] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 1 to 3, leader changed from e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66) to 334a4e1f59944544b3b3ff36616b57da (127.16.29.68). New cstate: current_term: 3 leader_uuid: "334a4e1f59944544b3b3ff36616b57da" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:21:34.821302 17833 raft_consensus.cc:2393] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 6: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:21:34.822151 17682 raft_consensus.cc:1240] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 7. Current term is 8. Ops: []
I20260709 12:21:34.814029 20157 leader_election.cc:290] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 3 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:34.834668 17727 leader_election.cc:304] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:21:34.821841 17583 catalog_manager.cc:5697] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 1 to 2, leader changed from e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66) to 0da169ff82674dc1947055ce43de68a3 (127.16.29.65). New cstate: current_term: 2 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:21:34.850081 17583 catalog_manager.cc:5697] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 1 to 3, leader changed from 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) to 334a4e1f59944544b3b3ff36616b57da (127.16.29.68). New cstate: current_term: 3 leader_uuid: "334a4e1f59944544b3b3ff36616b57da" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 12:21:34.861024 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:34.764072 20176 consensus_queue.cc:1059] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.104s
I20260709 12:21:34.820039 20019 raft_consensus.cc:515] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:34.882912 19779 leader_election.cc:290] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:34.883733 17756 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38776 (request call id 1715) took 1368 ms. Trace:
I20260709 12:21:34.889098 17756 rpcz_store.cc:276] 0709 12:21:33.515308 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:34.175415 (+660107us) service_pool.cc:224] Handling call
0709 12:21:34.175723 (+   308us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:21:34.176125 (+   402us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:34.176151 (+    26us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:21:34.195118 (+ 18967us) log.cc:844] Serialized 52 byte log entry
0709 12:21:34.639101 (+443983us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:21:34.639140 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:34.639203 (+    63us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:34.883473 (+244270us) raft_consensus.cc:1685] finished
0709 12:21:34.883499 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:34.883674 (+   175us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":106,"raft.run_cpu_time_us":118,"raft.run_wall_time_us":118,"thread_start_us":308020,"threads_started":1,"wal-append.queue_time_us":580677}
I20260709 12:21:34.809546 19868 consensus_queue.cc:1059] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 4.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.071s
I20260709 12:21:34.884821 20095 leader_election.cc:290] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:21:34.885962 17763 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.508s	user 0.031s	sys 0.046s
W20260709 12:21:34.902153 17763 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.524s	user 0.032s	sys 0.046s
I20260709 12:21:34.853863 17583 catalog_manager.cc:5697] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 2 to 3, leader changed from 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) to 334a4e1f59944544b3b3ff36616b57da (127.16.29.68). New cstate: current_term: 3 leader_uuid: "334a4e1f59944544b3b3ff36616b57da" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:21:34.926206 17687 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37396 (request call id 1802) took 1100 ms. Trace:
I20260709 12:21:34.877511 19976 raft_consensus.cc:3060] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:34.926841 17687 rpcz_store.cc:276] 0709 12:21:33.825550 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:34.023074 (+197524us) service_pool.cc:224] Handling call
0709 12:21:34.023365 (+   291us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:34.031507 (+  8142us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:34.031535 (+    28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:34.038485 (+  6950us) log.cc:844] Serialized 26 byte log entry
0709 12:21:34.148581 (+110096us) spinlock_profiling.cc:243] Waited 15.1 ms on lock 0x7b1800287f20. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b47515e07 00007f7b503709a9 00007f7b4753269b 00007f7b4753093f 00007f7b47532c02 00007f7b4a1ba20b 00007f7b4a1bfd23 00007f7b4a1c00d1 00007f7b4a5dba0f
0709 12:21:34.431290 (+282709us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:21:34.431327 (+    37us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:34.431396 (+    69us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:34.925545 (+494149us) raft_consensus.cc:1685] finished
0709 12:21:34.925732 (+   187us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:34.926025 (+   293us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":7784,"raft.queue_time_us":8437,"raft.run_cpu_time_us":130,"raft.run_wall_time_us":7207,"spinlock_wait_cycles":34685696,"thread_start_us":274041,"threads_started":1,"wal-append.queue_time_us":607004}
W20260709 12:21:34.934324 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 283517us
I20260709 12:21:34.886765 20064 leader_election.cc:290] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 8 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:21:35.017840 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 57406.3us
I20260709 12:21:35.018803 17652 leader_election.cc:304] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:35.024505 17652 leader_election.cc:304] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:21:34.959376 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 457333us
I20260709 12:21:35.027650 20147 leader_election.cc:290] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 8 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:35.034694 20076 raft_consensus.cc:2804] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:21:35.032582 20064 raft_consensus.cc:493] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:35.027079 19806 raft_consensus.cc:3060] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Advancing to term 2
W20260709 12:21:35.126284 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 450127us
I20260709 12:21:35.025791 20144 consensus_queue.cc:1059] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.036s
W20260709 12:21:35.106093 17801 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:35.129267 18364 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59886 (request call id 763) took 1077 ms. Trace:
W20260709 12:21:35.130092 17801 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:35.127839 18333 leader_election.cc:304] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:35.129989 18364 rpcz_store.cc:276] 0709 12:21:34.051988 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:34.349315 (+297327us) service_pool.cc:224] Handling call
0709 12:21:34.349584 (+   269us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:34.349909 (+   325us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:34.349935 (+    26us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:34.350349 (+   414us) log.cc:844] Serialized 26 byte log entry
0709 12:21:34.677205 (+326856us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:21:34.677243 (+    38us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:34.677308 (+    65us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:35.129067 (+451759us) raft_consensus.cc:1685] finished
0709 12:21:35.129090 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:35.129209 (+   119us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":2494,"raft.run_cpu_time_us":101,"raft.run_wall_time_us":100,"thread_start_us":326484,"threads_started":1,"wal-append.queue_time_us":541582}
I20260709 12:21:35.132391 18333 leader_election.cc:304] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:35.141023 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:35.123843 20076 raft_consensus.cc:493] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:21:35.135870 19779 raft_consensus.cc:2749] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:21:35.140805 20135 raft_consensus.cc:2749] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:21:35.129673 20191 consensus_queue.cc:1048] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.028s
I20260709 12:21:35.142951 20127 raft_consensus.cc:3055] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:21:35.156629 20076 raft_consensus.cc:3060] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:35.131186 20064 raft_consensus.cc:515] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:35.140648 20169 consensus_queue.cc:1059] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.023s
I20260709 12:21:35.139405 20018 consensus_queue.cc:1048] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.024s
I20260709 12:21:35.162068 20152 raft_consensus.cc:493] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:35.161339 20088 raft_consensus.cc:493] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:35.159845 20102 leader_election.cc:290] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 5 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:35.162905 20158 raft_consensus.cc:2804] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:21:35.168413 20127 raft_consensus.cc:740] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:21:35.206270 20157 raft_consensus.cc:3055] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:21:35.205914 20127 consensus_queue.cc:260] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:35.206729 20157 raft_consensus.cc:740] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 6 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:21:35.207320 20127 raft_consensus.cc:3060] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:35.209403 20168 raft_consensus.cc:3060] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:35.212478 20176 raft_consensus.cc:3055] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:21:35.212970 20176 raft_consensus.cc:740] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 6 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:21:35.183331 20152 raft_consensus.cc:515] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:35.213792 20176 consensus_queue.cc:260] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:35.214926 20176 raft_consensus.cc:3060] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:35.215584 20152 leader_election.cc:290] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:35.216127 20073 raft_consensus.cc:515] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:35.217134 20064 leader_election.cc:290] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:35.219128 20073 leader_election.cc:290] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 6 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:35.221948 20019 leader_election.cc:290] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 5 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:35.219347 20167 raft_consensus.cc:3060] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:35.207656 20157 consensus_queue.cc:260] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:35.227336 20170 raft_consensus.cc:3055] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:21:35.227764 20170 raft_consensus.cc:740] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 4 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:21:35.184620 20088 raft_consensus.cc:515] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:35.228613 20170 consensus_queue.cc:260] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:35.229626 20170 raft_consensus.cc:3060] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:35.230938 20088 leader_election.cc:290] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:35.231649 19994 raft_consensus.cc:515] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:35.238910 19994 leader_election.cc:290] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 6 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:21:35.240073 20218 log.cc:927] Time spent T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.080s	user 0.000s	sys 0.004s
I20260709 12:21:35.200989 20158 raft_consensus.cc:697] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:21:35.244146 20158 consensus_queue.cc:237] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:35.254419 20076 raft_consensus.cc:515] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:35.031698 17652 leader_election.cc:304] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
W20260709 12:21:35.288857 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 257058us
W20260709 12:21:35.292904 17728 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.576s	user 0.001s	sys 0.020s
W20260709 12:21:35.294616 17688 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.609s	user 0.020s	sys 0.042s
W20260709 12:21:35.300418 17728 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.583s	user 0.004s	sys 0.023s
W20260709 12:21:35.300709 17688 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.615s	user 0.024s	sys 0.042s
W20260709 12:21:35.294025 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:35.304611 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 467807us
I20260709 12:21:35.305989 17727 leader_election.cc:304] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:21:35.306562 17728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:35.291426 17652 leader_election.cc:304] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da
W20260709 12:21:35.317631 17803 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.793s	user 0.009s	sys 0.021s
W20260709 12:21:35.318218 17803 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.794s	user 0.009s	sys 0.021s
W20260709 12:21:35.319105 17804 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:35.320564 17763 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38776 (request call id 1725) took 1513 ms. Trace:
W20260709 12:21:35.322407 17803 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:35.323027 17803 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:35.321245 17763 rpcz_store.cc:276] 0709 12:21:33.806509 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:34.331069 (+524560us) service_pool.cc:224] Handling call
0709 12:21:34.331350 (+   281us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:34.331546 (+   196us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:21:34.903690 (+572144us) log.cc:844] Serialized 14 byte log entry
0709 12:21:35.319453 (+415763us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:35.319592 (+   139us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:21:35.319751 (+   159us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:35.319943 (+   192us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:35.320222 (+   279us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":306236,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":46,"thread_start_us":940954,"threads_started":2}
I20260709 12:21:35.325337 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:35.326068 17831 raft_consensus.cc:1275] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20260709 12:21:35.326440 17829 raft_consensus.cc:3060] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:35.328583 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:35.330981 17830 raft_consensus.cc:1240] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 1. Current term is 2. Ops: []
I20260709 12:21:35.332546 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:35.333461 17833 raft_consensus.cc:3060] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:21:35.291579 17654 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.597s	user 0.008s	sys 0.014s
W20260709 12:21:35.341284 17654 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.647s	user 0.008s	sys 0.015s
I20260709 12:21:35.323505 17652 leader_election.cc:304] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:21:35.361254 20178 raft_consensus.cc:2804] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:21:35.361956 20178 raft_consensus.cc:493] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:21:35.362470 20178 raft_consensus.cc:3060] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:35.362300 20018 consensus_queue.cc:1059] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:21:35.361042 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:35.368985 20170 raft_consensus.cc:493] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:35.369393 20165 raft_consensus.cc:3055] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:21:35.369978 20165 raft_consensus.cc:740] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 5 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:21:35.370158 17764 raft_consensus.cc:1275] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:21:35.370023 20170 raft_consensus.cc:515] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:35.372215 19994 raft_consensus.cc:2804] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:21:35.372772 19994 raft_consensus.cc:493] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:35.373200 19994 raft_consensus.cc:3060] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:35.373581 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:35.374529 17837 raft_consensus.cc:3060] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:21:35.380216 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:35.380388 19883 consensus_meta.cc:287] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.604s	user 0.010s	sys 0.001s
I20260709 12:21:35.381335 20233 raft_consensus.cc:3055] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:21:35.381945 20233 raft_consensus.cc:740] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 4 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:21:35.383247 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:35.384603 17832 raft_consensus.cc:2468] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 4.
I20260709 12:21:35.385703 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:35.388875 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:35.390157 20127 raft_consensus.cc:2749] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:21:35.390617 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:35.390408 18364 raft_consensus.cc:1240] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 6. Current term is 7. Ops: [6.8-6.8]
I20260709 12:21:35.383764 20233 consensus_queue.cc:260] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:35.396034 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:35.396816 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:35.397126 20233 raft_consensus.cc:3060] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:35.397887 18366 raft_consensus.cc:2393] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 3: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:21:35.402172 20170 leader_election.cc:290] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:35.402400 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:35.403676 17764 raft_consensus.cc:2410] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 7 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 4.
I20260709 12:21:35.405372 19826 raft_consensus.cc:3055] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:21:35.405894 19826 raft_consensus.cc:740] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 3 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:21:35.410590 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:35.411619 17834 raft_consensus.cc:2468] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 2.
I20260709 12:21:35.413455 20164 consensus_queue.cc:1048] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:21:35.416221 20227 consensus_queue.cc:1059] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:21:35.418200 20227 consensus_queue.cc:1048] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:21:35.418664 17764 raft_consensus.cc:1275] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20260709 12:21:35.420439 20146 raft_consensus.cc:493] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:35.420743 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:35.420917 17803 leader_election.cc:304] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:35.421275 20146 raft_consensus.cc:515] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:35.422020 20059 raft_consensus.cc:493] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:35.422199 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:35.422809 20059 raft_consensus.cc:515] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:35.424168 20052 raft_consensus.cc:493] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:35.424997 17760 raft_consensus.cc:1275] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 7 index: 9. (index mismatch)
I20260709 12:21:35.426230 20223 raft_consensus.cc:3060] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:35.407295 19826 consensus_queue.cc:260] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:35.427866 19826 raft_consensus.cc:3060] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:35.428151 17838 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b"
dest_uuid: "e063541542a94969898868bf6eb5ce1b"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:35032
I20260709 12:21:35.257925 20076 leader_election.cc:290] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 5 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:21:35.354138 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 668206us
I20260709 12:21:35.432653 20059 leader_election.cc:290] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:35.423476 19806 raft_consensus.cc:2749] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:21:35.379966 20168 consensus_queue.cc:1048] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.016s
I20260709 12:21:35.397516 20179 consensus_queue.cc:1059] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:21:35.438565 17833 raft_consensus.cc:2468] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 5.
I20260709 12:21:35.435613 20176 raft_consensus.cc:493] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:21:35.442899 17756 raft_consensus.cc:2468] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 3.
W20260709 12:21:35.456781 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 99515.9us
I20260709 12:21:35.425076 20052 raft_consensus.cc:515] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:35.477164 20176 raft_consensus.cc:515] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:21:35.478200 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 161190us
I20260709 12:21:35.478989 20052 leader_election.cc:290] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:35.481724 17729 leader_election.cc:304] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
W20260709 12:21:35.488930 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 164899us
I20260709 12:21:35.489880 17729 leader_election.cc:304] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:21:35.481812 20179 raft_consensus.cc:2749] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:21:35.482173 20231 consensus_queue.cc:1048] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.010s
I20260709 12:21:35.482481 17684 raft_consensus.cc:1240] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 4. Current term is 5. Ops: []
I20260709 12:21:35.486717 20144 raft_consensus.cc:3055] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:21:35.495359 20144 raft_consensus.cc:740] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:21:35.495434 17687 raft_consensus.cc:1240] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 1. Current term is 2. Ops: []
I20260709 12:21:35.496676 20144 consensus_queue.cc:260] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:35.498132 17682 raft_consensus.cc:1275] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260709 12:21:35.498440 20144 raft_consensus.cc:3060] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:35.487216 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:35.500588 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:35.487529 20196 consensus_queue.cc:1059] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.015s
I20260709 12:21:35.502676 20196 raft_consensus.cc:3055] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:21:35.503123 20196 raft_consensus.cc:740] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:21:35.504055 20169 consensus_queue.cc:1048] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.064s
I20260709 12:21:35.504338 20196 consensus_queue.cc:260] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:35.505761 20196 raft_consensus.cc:3060] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:35.506220 20076 consensus_queue.cc:1059] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.044s
I20260709 12:21:35.507380 20076 consensus_queue.cc:1243] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer 334a4e1f59944544b3b3ff36616b57da log is divergent from this leader: its last log entry 3.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260709 12:21:35.508193 20191 raft_consensus.cc:3060] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:35.480794 17652 leader_election.cc:304] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:21:35.508211 20095 consensus_queue.cc:1048] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.053s
I20260709 12:21:35.512585 20179 raft_consensus.cc:2804] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:21:35.513250 20179 raft_consensus.cc:697] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 5 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:21:35.513360 17683 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37396 (request call id 1806) took 1650 ms. Trace:
I20260709 12:21:35.514623 17652 leader_election.cc:304] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:35.515712 20147 raft_consensus.cc:2749] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:21:35.441813 20146 leader_election.cc:290] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:21:35.516553 17654 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:35.516772 17761 raft_consensus.cc:2468] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 3.
I20260709 12:21:35.440054 20165 consensus_queue.cc:260] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:21:35.517294 17654 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:35.517380 20018 raft_consensus.cc:493] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:35.518025 20165 raft_consensus.cc:3060] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:35.518046 20018 raft_consensus.cc:515] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:35.519083 20234 raft_consensus.cc:3060] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:35.526878 17729 leader_election.cc:304] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
W20260709 12:21:35.527637 17727 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:35.528302 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 75576.2us
I20260709 12:21:35.529032 20222 consensus_queue.cc:1048] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:21:35.528357 20019 raft_consensus.cc:493] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:35.539623 20019 raft_consensus.cc:515] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:35.540096 17801 leader_election.cc:304] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:21:35.540470 18374 raft_consensus.cc:1275] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260709 12:21:35.542765 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:35.543565 17756 raft_consensus.cc:2468] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 4.
I20260709 12:21:35.530689 20169 raft_consensus.cc:493] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:35.544672 20169 raft_consensus.cc:515] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:35.532456 19961 consensus_queue.cc:1048] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:21:35.533370 20232 raft_consensus.cc:493] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:35.548333 17757 raft_consensus.cc:2410] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 5 because replica has last-logged OpId of term: 4 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260709 12:21:35.548483 20232 raft_consensus.cc:515] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:35.550866 17727 leader_election.cc:304] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:21:35.553177 20168 raft_consensus.cc:2764] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20260709 12:21:35.555040 17801 leader_election.cc:304] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:21:35.555508 20073 consensus_queue.cc:1048] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:21:35.542235 20019 leader_election.cc:290] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:35.551028 20232 leader_election.cc:290] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:35.551611 19961 consensus_queue.cc:1059] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:21:35.536826 20179 consensus_queue.cc:237] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:35.557757 19753 raft_consensus.cc:3055] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:21:35.557696 17652 leader_election.cc:304] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:21:35.558491 20191 raft_consensus.cc:2749] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:21:35.558882 20076 raft_consensus.cc:2804] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:21:35.559064 20186 raft_consensus.cc:3060] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:35.559485 20076 raft_consensus.cc:697] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 5 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:21:35.551612 20222 raft_consensus.cc:2804] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:21:35.560861 20222 raft_consensus.cc:697] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 3 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:21:35.560740 20076 consensus_queue.cc:237] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:35.562173 20222 consensus_queue.cc:237] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:35.556083 20153 raft_consensus.cc:2804] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:21:35.564446 20153 raft_consensus.cc:493] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:35.564872 20153 raft_consensus.cc:3060] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:35.565092 17652 leader_election.cc:304] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:21:35.566133 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:35.567814 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:35.569622 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:35.569941 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
W20260709 12:21:35.540174 17583 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.523s	user 0.005s	sys 0.021s
W20260709 12:21:35.570909 17583 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.554s	user 0.005s	sys 0.021s
I20260709 12:21:35.571089 17688 raft_consensus.cc:2410] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for term 4 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260709 12:21:35.513990 17683 rpcz_store.cc:276] 0709 12:21:33.862657 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:34.035066 (+172409us) service_pool.cc:224] Handling call
0709 12:21:34.605635 (+570569us) spinlock_profiling.cc:243] Waited 4.9 ms on lock 0x7b5800002a98. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b474fbb71 00007f7b474fba22 00007f7b504148d9 00007f7b4f56eee0 00007f7b4f6380ab 00007f7b4f638131 00007f7b4f5f9fc0 00007f7b4f5e4929 00007f7b4f5e43d2 00007f7b48b5bbd4
0709 12:21:34.605818 (+   183us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:34.606015 (+   197us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 12:21:35.026654 (+420639us) log.cc:844] Serialized 14 byte log entry
0709 12:21:35.512523 (+485869us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:35.512661 (+   138us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:21:35.512767 (+   106us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:35.512898 (+   131us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:35.513118 (+   220us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":564396,"prepare.run_cpu_time_us":52,"prepare.run_wall_time_us":47,"raft.queue_time_us":1408,"raft.run_cpu_time_us":216,"raft.run_wall_time_us":217,"spinlock_wait_cycles":11258624,"thread_start_us":905580,"threads_started":2}
I20260709 12:21:35.555091 18333 leader_election.cc:304] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:21:35.531715 20234 raft_consensus.cc:2749] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:21:35.594703 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:35.594477 17837 raft_consensus.cc:2468] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 4.
I20260709 12:21:35.595346 17835 raft_consensus.cc:2468] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 5.
I20260709 12:21:35.558346 19753 raft_consensus.cc:740] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:21:35.585069 20146 raft_consensus.cc:2804] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:21:35.590508 20152 raft_consensus.cc:493] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:35.617647 20088 raft_consensus.cc:493] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:21:35.618762 20064 consensus_queue.cc:1059] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:21:35.606621 20127 raft_consensus.cc:493] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:35.620662 20157 raft_consensus.cc:3060] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:35.620945 20232 raft_consensus.cc:2804] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:21:35.663988 20223 raft_consensus.cc:2749] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:21:35.665635 18334 leader_election.cc:304] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:21:35.621496 20191 raft_consensus.cc:493] T 7cb3768d215e409fb9f5baf18d124442 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:21:35.668224 20102 raft_consensus.cc:3060] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:35.668021 20127 raft_consensus.cc:515] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:35.668367 20191 raft_consensus.cc:515] T 7cb3768d215e409fb9f5baf18d124442 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:35.670468 20127 leader_election.cc:290] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:35.671010 20088 raft_consensus.cc:515] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:35.671531 20191 leader_election.cc:290] T 7cb3768d215e409fb9f5baf18d124442 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:21:35.628196 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 73392.6us
I20260709 12:21:35.677189 17684 raft_consensus.cc:1275] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20260709 12:21:35.677646 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:35.678154 18333 leader_election.cc:304] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:21:35.678474 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:35.678795 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:35.679267 17684 raft_consensus.cc:2393] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 8: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:21:35.625376 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:35.679942 18333 leader_election.cc:304] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:21:35.681308 18333 leader_election.cc:304] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:21:35.681213 20152 raft_consensus.cc:515] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:35.681897 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:35.632803 16501 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:21:35.629583 20169 leader_election.cc:290] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:35.633369 18374 raft_consensus.cc:1275] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 7 index: 9. (index mismatch)
I20260709 12:21:35.633935 20231 raft_consensus.cc:3055] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:21:35.635421 20234 raft_consensus.cc:493] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:35.583053 19883 consensus_queue.cc:1048] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.053s
I20260709 12:21:35.636152 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:35.636977 19961 raft_consensus.cc:493] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:35.637534 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 216793us
I20260709 12:21:35.638684 20144 raft_consensus.cc:493] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
W20260709 12:21:35.620656 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 93894.7us
I20260709 12:21:35.705042 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:35.629405 20229 raft_consensus.cc:2804] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260709 12:21:35.640412 17689 consensus_meta.cc:287] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.905s	user 0.016s	sys 0.003s
I20260709 12:21:35.608819 20146 raft_consensus.cc:493] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:21:35.648880 17801 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:35.604472 20073 raft_consensus.cc:493] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:21:35.651460 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:35.654102 17652 leader_election.cc:304] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:35.654600 20175 consensus_queue.cc:1059] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:21:35.655403 17680 raft_consensus.cc:1275] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20260709 12:21:35.656837 17829 raft_consensus.cc:2468] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 5.
I20260709 12:21:35.659229 19994 raft_consensus.cc:515] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:35.660022 17686 raft_consensus.cc:2410] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for term 4 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260709 12:21:35.662945 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:35.664067 20232 raft_consensus.cc:493] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:21:35.672261 19855 consensus_queue.cc:1059] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 7.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:21:35.673202 20214 raft_consensus.cc:493] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:21:35.674059 20088 leader_election.cc:290] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:35.647428 20176 leader_election.cc:290] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:35.676031 20165 raft_consensus.cc:493] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:21:35.676744 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:35.624827 18368 raft_consensus.cc:1275] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
W20260709 12:21:35.632656 17761 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36572 (request call id 493) took 1841 ms (client timeout 2044 ms). Trace:
I20260709 12:21:35.709347 20146 raft_consensus.cc:3060] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:21:35.724967 17761 rpcz_store.cc:269] 0709 12:21:33.791093 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:33.912046 (+120953us) service_pool.cc:224] Handling call
0709 12:21:35.632432 (+1720386us) spinlock_profiling.cc:243] Waited 1.4 s on lock 0x7b54001b8854. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:21:35.632568 (+   136us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3217742720}
I20260709 12:21:35.667450 20178 raft_consensus.cc:515] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:35.663935 20064 raft_consensus.cc:3055] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:21:35.686640 20231 raft_consensus.cc:740] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:21:35.712901 17689 raft_consensus.cc:2468] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 4.
I20260709 12:21:35.706753 20229 raft_consensus.cc:493] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:35.723548 18361 raft_consensus.cc:2468] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 4.
I20260709 12:21:35.634089 19753 consensus_queue.cc:260] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:35.727926 17756 raft_consensus.cc:2468] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 4.
I20260709 12:21:35.722697 18334 leader_election.cc:304] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:21:35.716630 20232 raft_consensus.cc:3060] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:35.748761 17684 raft_consensus.cc:1240] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 3. Current term is 4. Ops: []
I20260709 12:21:35.749995 17687 raft_consensus.cc:3060] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:35.730121 17680 raft_consensus.cc:1240] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 3. Current term is 4. Ops: []
I20260709 12:21:35.753326 19960 raft_consensus.cc:2804] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:21:35.753976 19960 raft_consensus.cc:697] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 5 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
W20260709 12:21:35.755574 19976 consensus_meta.cc:287] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.821s	user 0.010s	sys 0.007s
I20260709 12:21:35.746881 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:21:35.757462 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 103488us
I20260709 12:21:35.757786 20231 consensus_queue.cc:260] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:35.759306 20231 raft_consensus.cc:3060] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:35.760389 20247 raft_consensus.cc:3055] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:21:35.760870 20247 raft_consensus.cc:740] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 3 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:21:35.761577 20178 leader_election.cc:290] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 3 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:35.761838 20247 consensus_queue.cc:260] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:35.740983 20064 raft_consensus.cc:740] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:21:35.763275 20247 raft_consensus.cc:3060] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:35.763722 20064 consensus_queue.cc:260] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:35.765270 20064 raft_consensus.cc:3060] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:35.768558 17801 leader_election.cc:304] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:21:35.770470 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:35.771401 20143 consensus_queue.cc:1059] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.025s
I20260709 12:21:35.746665 20227 raft_consensus.cc:2749] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:21:35.775081 20018 leader_election.cc:290] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:35.716344 20073 raft_consensus.cc:515] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:21:35.815640 17728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:35.780126 20088 raft_consensus.cc:2804] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:21:35.751083 20214 raft_consensus.cc:515] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:35.783172 20095 raft_consensus.cc:2749] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:21:35.784250 17682 raft_consensus.cc:1240] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 1. Current term is 2. Ops: []
I20260709 12:21:35.745389 20019 raft_consensus.cc:2804] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:21:35.784631 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:35.747833 17803 leader_election.cc:304] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:35.786306 17729 leader_election.cc:304] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:21:35.784278 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:21:35.787504 17730 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:35.787756 20152 leader_election.cc:290] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:35.779978 20144 raft_consensus.cc:515] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:35.791198 20222 raft_consensus.cc:493] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:35.707106 19961 raft_consensus.cc:515] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:35.707288 20234 raft_consensus.cc:515] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:35.797384 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:35.798560 17801 leader_election.cc:304] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:21:35.799561 17838 raft_consensus.cc:606] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Received request to transfer leadership
I20260709 12:21:35.829794 17838 raft_consensus.cc:614] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:21:35.801431 19994 leader_election.cc:290] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 5 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:35.802066 17688 raft_consensus.cc:1240] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 2. Current term is 3. Ops: []
I20260709 12:21:35.802541 17829 raft_consensus.cc:1240] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 5. Current term is 6. Ops: [5.5-5.5]
W20260709 12:21:35.809358 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 51022.9us
I20260709 12:21:35.828074 17654 leader_election.cc:304] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:21:35.777937 20179 raft_consensus.cc:3055] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:21:35.747354 20165 raft_consensus.cc:515] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:35.744741 20229 raft_consensus.cc:3060] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:35.823148 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:35.823346 20153 raft_consensus.cc:515] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:35.755121 19960 consensus_queue.cc:237] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:35.823359 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:35.822257 17689 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37402 (request call id 2213) took 1098 ms. Trace:
I20260709 12:21:35.832897 20175 raft_consensus.cc:2804] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:21:35.829402 17683 raft_consensus.cc:2468] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 6.
I20260709 12:21:35.833855 20186 raft_consensus.cc:2749] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:21:35.834101 19976 raft_consensus.cc:2804] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:21:35.838397 17652 leader_election.cc:304] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:21:35.831537 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:35.840379 17685 raft_consensus.cc:2468] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 3.
I20260709 12:21:35.844523 17689 rpcz_store.cc:276] 0709 12:21:34.719511 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:34.722688 (+  3177us) service_pool.cc:224] Handling call
0709 12:21:35.808521 (+1085833us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:35.824241 20076 raft_consensus.cc:493] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:35.848997 17801 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:35.831562 20179 raft_consensus.cc:740] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:21:35.858227 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:21:35.848706 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 167554us
W20260709 12:21:35.868976 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 154053us
I20260709 12:21:35.852706 17835 raft_consensus.cc:2468] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 4.
I20260709 12:21:35.870525 17687 raft_consensus.cc:2468] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 5.
I20260709 12:21:35.818042 20178 consensus_queue.cc:1048] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.023s
W20260709 12:21:35.881826 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 171999us
I20260709 12:21:35.853497 19976 raft_consensus.cc:493] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:35.852694 20175 raft_consensus.cc:493] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:35.860446 20158 raft_consensus.cc:2749] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:21:35.872905 20227 raft_consensus.cc:2804] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:21:35.862066 19855 raft_consensus.cc:2749] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:21:35.836004 19883 consensus_queue.cc:1048] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.045s
I20260709 12:21:35.880873 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
W20260709 12:21:35.896914 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 58645.7us
I20260709 12:21:35.897028 20214 leader_election.cc:290] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:35.888762 18333 leader_election.cc:304] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:21:35.882795 20157 raft_consensus.cc:2804] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:21:35.896273 20076 raft_consensus.cc:515] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:35.905259 17728 leader_election.cc:304] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:21:35.892916 19976 raft_consensus.cc:3060] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:35.898442 20175 raft_consensus.cc:3060] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:35.860760 20222 raft_consensus.cc:515] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:21:35.920426 17655 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:35.902419 20147 raft_consensus.cc:493] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:35.922245 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 46297us
I20260709 12:21:35.929046 17803 leader_election.cc:400] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 3 election: Vote denied by peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577) with higher term. Message: Invalid argument: T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 3. Current term is 4.
I20260709 12:21:35.932374 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:35.934113 17652 leader_election.cc:304] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:35.916021 20157 raft_consensus.cc:493] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:21:35.939834 17684 raft_consensus.cc:1275] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260709 12:21:35.931031 20018 raft_consensus.cc:2804] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260709 12:21:35.951788 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 235896us
I20260709 12:21:35.952618 18333 leader_election.cc:304] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:21:35.942246 20157 raft_consensus.cc:3060] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:35.943446 19855 raft_consensus.cc:2804] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:21:35.952946 17835 raft_consensus.cc:2468] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 6.
I20260709 12:21:35.949427 20018 raft_consensus.cc:493] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:35.948052 20088 raft_consensus.cc:697] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 4 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:21:35.976512 17729 leader_election.cc:304] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:21:35.938302 19994 consensus_queue.cc:1048] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.031s
W20260709 12:21:35.981145 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 72003.9us
I20260709 12:21:35.963801 20102 raft_consensus.cc:2749] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:21:35.965137 19855 raft_consensus.cc:493] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:21:35.988327 20019 raft_consensus.cc:493] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:21:35.989104 20088 consensus_queue.cc:237] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:21:35.991231 17804 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:35.991334 20253 log.cc:927] Time spent T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.108s	user 0.000s	sys 0.001s
I20260709 12:21:35.943655 20147 raft_consensus.cc:515] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:35.974192 20018 raft_consensus.cc:3060] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:35.996232 20234 leader_election.cc:290] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:36.011251 18374 raft_consensus.cc:2468] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 2.
I20260709 12:21:36.000602 19855 raft_consensus.cc:3060] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:36.017041 17803 leader_election.cc:304] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:36.020843 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:36.024755 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:36.009914 20019 raft_consensus.cc:3060] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:36.026091 20264 raft_consensus.cc:2804] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:21:36.036341 17763 raft_consensus.cc:3060] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:35.982513 20073 leader_election.cc:290] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:36.037256 20266 consensus_queue.cc:1048] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.051s
I20260709 12:21:36.039052 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:36.047811 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:36.000317 20153 leader_election.cc:290] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 5 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:21:36.050931 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 87526.4us
I20260709 12:21:35.957926 20179 consensus_queue.cc:260] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:35.988348 20191 raft_consensus.cc:3060] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:36.014369 20165 leader_election.cc:290] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:36.022269 20052 raft_consensus.cc:493] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:36.077207 17830 raft_consensus.cc:2468] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 4.
I20260709 12:21:36.095446 20227 raft_consensus.cc:697] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 7 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
W20260709 12:21:36.123021 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 122939us
W20260709 12:21:36.088063 20254 log.cc:927] Time spent T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.056s	user 0.001s	sys 0.000s
I20260709 12:21:36.082383 20095 raft_consensus.cc:2804] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260709 12:21:36.127136 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 114859us
I20260709 12:21:36.124394 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:36.054178 20232 raft_consensus.cc:515] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:36.082787 20264 raft_consensus.cc:697] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 5 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:21:36.097589 20144 leader_election.cc:290] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:21:36.134660 17654 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:36.122958 20179 raft_consensus.cc:3060] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:36.132947 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:36.128237 20095 raft_consensus.cc:493] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:21:36.146137 17838 raft_consensus.cc:2468] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 3.
I20260709 12:21:36.131255 20176 raft_consensus.cc:493] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:21:36.137249 20196 raft_consensus.cc:493] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:36.153267 17829 raft_consensus.cc:2393] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 4: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:21:36.148597 19883 raft_consensus.cc:2749] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 12:21:36.174918 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:36.138300 20052 raft_consensus.cc:515] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:36.154925 20247 consensus_queue.cc:1059] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.023s
W20260709 12:21:36.222766 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 81315.1us
W20260709 12:21:36.176271 20270 log.cc:927] Time spent T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.147s	user 0.001s	sys 0.000s
I20260709 12:21:36.154171 20095 raft_consensus.cc:3060] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:21:36.178596 20218 log.cc:927] Time spent T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.050s	user 0.000s	sys 0.001s
W20260709 12:21:36.179476 20272 log.cc:927] Time spent T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.141s	user 0.001s	sys 0.000s
I20260709 12:21:36.179375 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:36.144387 20165 consensus_queue.cc:1048] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.036s
I20260709 12:21:36.194921 20264 consensus_queue.cc:237] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:36.196259 20284 raft_consensus.cc:493] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:36.198056 19806 raft_consensus.cc:493] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:36.236250 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 114031us
I20260709 12:21:36.198707 20102 consensus_queue.cc:1048] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 12:21:36.218843 20153 consensus_queue.cc:1059] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.014s
I20260709 12:21:36.155617 20222 leader_election.cc:290] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:36.205503 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:36.160826 20076 leader_election.cc:290] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:21:36.181459 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 247425us
I20260709 12:21:36.160456 20167 raft_consensus.cc:493] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:36.169267 20223 raft_consensus.cc:493] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:36.213891 20228 log.cc:927] Time spent T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.197s	user 0.001s	sys 0.000s
I20260709 12:21:36.216204 17727 leader_election.cc:304] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:21:36.217514 20229 raft_consensus.cc:515] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:36.218364 19868 raft_consensus.cc:493] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:36.150959 20146 raft_consensus.cc:515] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:36.228857 17835 raft_consensus.cc:2468] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 6.
I20260709 12:21:36.234947 17834 raft_consensus.cc:2468] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 2.
I20260709 12:21:36.207655 20227 consensus_queue.cc:237] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.6, Last appended by leader: 6, Current term: 7, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:36.164345 20196 raft_consensus.cc:515] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:36.199388 20273 raft_consensus.cc:493] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:36.241864 17803 leader_election.cc:304] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:21:36.242391 17763 raft_consensus.cc:2468] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 5.
I20260709 12:21:36.180881 20176 raft_consensus.cc:515] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:36.297644 20176 leader_election.cc:290] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:36.226190 20164 consensus_queue.cc:1059] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.025s
W20260709 12:21:36.260403 18333 leader_election.cc:336] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Timed out: RequestConsensusVote RPC to 127.16.29.66:39651 timed out after 1.687s (SENT)
W20260709 12:21:36.284354 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 68241.9us
I20260709 12:21:36.298604 18333 leader_election.cc:304] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:36.299356 20164 raft_consensus.cc:3055] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:21:36.299610 17727 leader_election.cc:304] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:21:36.243023 19806 raft_consensus.cc:515] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:36.251515 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:36.288749 17837 raft_consensus.cc:1275] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
W20260709 12:21:36.301654 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:36.290333 20147 leader_election.cc:290] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:36.316812 20223 raft_consensus.cc:515] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:21:36.332566 20285 log.cc:927] Time spent T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.081s	user 0.000s	sys 0.000s
W20260709 12:21:36.332957 20283 log.cc:927] Time spent T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.065s	user 0.000s	sys 0.000s
W20260709 12:21:36.335338 20064 consensus_meta.cc:287] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.570s	user 0.007s	sys 0.005s
I20260709 12:21:36.336090 17829 raft_consensus.cc:2393] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 5: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:21:36.336179 17802 leader_election.cc:304] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:36.337181 17831 raft_consensus.cc:1275] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260709 12:21:36.260016 20284 raft_consensus.cc:515] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:36.338620 20223 leader_election.cc:290] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:36.299856 20164 raft_consensus.cc:740] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:21:36.340549 17652 leader_election.cc:304] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:21:36.340811 20164 consensus_queue.cc:260] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:36.342079 20164 raft_consensus.cc:3060] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:36.342914 20146 leader_election.cc:290] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 6 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:36.343937 20146 raft_consensus.cc:2804] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Leader election won for term 3
W20260709 12:21:36.348161 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 106390us
I20260709 12:21:36.348641 18333 leader_election.cc:304] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:36.349009 17803 leader_election.cc:304] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:21:36.349658 20102 raft_consensus.cc:2749] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20260709 12:21:36.350343 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 87604.7us
I20260709 12:21:36.349673 20153 raft_consensus.cc:2749] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:21:36.351271 20153 raft_consensus.cc:2749] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:21:36.352033 20153 consensus_queue.cc:1048] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:21:36.352432 17802 leader_election.cc:304] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:36.353005 20018 raft_consensus.cc:515] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:36.353628 20153 raft_consensus.cc:2804] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:21:36.354228 20153 raft_consensus.cc:697] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 4 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:21:36.355106 20018 leader_election.cc:290] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 7 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:36.355173 20153 consensus_queue.cc:237] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:36.321434 20232 leader_election.cc:290] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 4 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:36.357249 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:36.358611 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:36.359275 20146 raft_consensus.cc:697] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 3 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:21:36.306783 19976 raft_consensus.cc:515] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:36.306375 19868 raft_consensus.cc:515] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:36.308301 20273 raft_consensus.cc:515] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:36.365016 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:36.267664 20167 raft_consensus.cc:515] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:36.323076 20266 raft_consensus.cc:493] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:36.339725 20284 leader_election.cc:290] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:36.391155 19855 raft_consensus.cc:515] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:21:36.405218 20260 log.cc:927] Time spent T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.586s	user 0.006s	sys 0.000s
W20260709 12:21:36.406193 20258 log.cc:927] Time spent T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.566s	user 0.003s	sys 0.003s
I20260709 12:21:36.390882 20152 raft_consensus.cc:3055] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:21:36.360800 20146 consensus_queue.cc:237] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:36.308672 20247 raft_consensus.cc:2804] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:21:36.254354 20222 consensus_queue.cc:1048] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.117s
I20260709 12:21:36.300717 20234 raft_consensus.cc:2804] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:21:36.300083 20073 raft_consensus.cc:2749] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:21:36.451218 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
W20260709 12:21:36.465994 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 53149.2us
I20260709 12:21:36.462785 20234 raft_consensus.cc:493] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:36.466781 17834 raft_consensus.cc:1240] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 4. Current term is 5. Ops: []
I20260709 12:21:36.466835 20234 raft_consensus.cc:3060] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:36.467312 19976 leader_election.cc:290] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 5 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:36.472273 19753 raft_consensus.cc:3060] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:36.476389 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:36.477767 20157 raft_consensus.cc:515] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:36.478873 17836 raft_consensus.cc:1240] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 5. Current term is 6. Ops: []
I20260709 12:21:36.467724 17803 leader_election.cc:304] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:36.486145 17833 raft_consensus.cc:1275] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260709 12:21:36.487496 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
W20260709 12:21:36.461792 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 160188us
I20260709 12:21:36.461149 20247 raft_consensus.cc:493] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:36.496253 17834 raft_consensus.cc:2393] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 5: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:21:36.487778 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:36.489054 17837 raft_consensus.cc:1240] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 4. Current term is 5. Ops: []
I20260709 12:21:36.482323 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
W20260709 12:21:36.501029 17655 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:36.518186 17727 leader_election.cc:304] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:21:36.486222 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:36.457049 20152 raft_consensus.cc:740] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:21:36.484786 17758 raft_consensus.cc:1240] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 4. Current term is 5. Ops: []
I20260709 12:21:36.521149 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:36.461421 20052 leader_election.cc:290] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:36.482626 17829 raft_consensus.cc:1275] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:21:36.485646 19806 leader_election.cc:290] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:36.547075 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:36.523000 17757 raft_consensus.cc:2468] T 7cb3768d215e409fb9f5baf18d124442 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 3.
I20260709 12:21:36.550159 17683 raft_consensus.cc:1240] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 7 LEADER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 6. Current term is 7. Ops: [6.7-6.7]
W20260709 12:21:36.548198 17652 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:36.486912 20167 leader_election.cc:290] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:36.513278 20247 raft_consensus.cc:3060] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:36.485950 20019 raft_consensus.cc:515] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:36.486097 20095 raft_consensus.cc:515] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:36.557003 20296 raft_consensus.cc:2749] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260709 12:21:36.557557 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 89974.2us
I20260709 12:21:36.484520 19868 leader_election.cc:290] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:36.560801 17763 raft_consensus.cc:2393] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 8: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:21:36.522549 20191 raft_consensus.cc:2749] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:21:36.461035 20266 raft_consensus.cc:515] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:36.553977 20232 raft_consensus.cc:2804] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:21:36.480738 20157 leader_election.cc:290] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 6 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:36.563776 17833 raft_consensus.cc:1240] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20260709 12:21:36.565342 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:36.522243 19976 raft_consensus.cc:493] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:36.582257 20095 leader_election.cc:290] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 5 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:36.562225 20196 leader_election.cc:290] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:36.562103 20229 leader_election.cc:290] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 5 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:36.565372 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:36.583612 18361 raft_consensus.cc:1240] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 2. Current term is 3. Ops: []
I20260709 12:21:36.583740 17757 raft_consensus.cc:2468] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 8.
I20260709 12:21:36.574513 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:36.588227 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:36.589068 17755 raft_consensus.cc:2468] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 4.
I20260709 12:21:36.589538 20273 leader_election.cc:290] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:36.590215 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:36.590481 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:36.590937 17680 raft_consensus.cc:2468] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 5.
I20260709 12:21:36.591358 17762 raft_consensus.cc:2468] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 5.
I20260709 12:21:36.574818 17803 leader_election.cc:304] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:36.592429 20232 raft_consensus.cc:493] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:36.593001 20232 raft_consensus.cc:3060] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:36.593787 17803 leader_election.cc:304] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:36.594735 20169 raft_consensus.cc:2749] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:21:36.594823 20296 raft_consensus.cc:3060] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:36.595172 17761 raft_consensus.cc:2468] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 4.
I20260709 12:21:36.578293 17652 leader_election.cc:304] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:36.580399 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:36.580688 19806 consensus_queue.cc:1059] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:21:36.581986 19976 raft_consensus.cc:515] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:36.562304 17760 raft_consensus.cc:2468] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 5.
I20260709 12:21:36.582470 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:36.607342 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:36.608274 20196 raft_consensus.cc:2749] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:21:36.608495 17652 leader_election.cc:304] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:36.609074 20196 consensus_queue.cc:1048] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:21:36.609709 20284 raft_consensus.cc:2749] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:21:36.608242 17686 raft_consensus.cc:2468] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 6.
I20260709 12:21:36.610699 20284 consensus_queue.cc:1059] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:21:36.611271 20196 consensus_queue.cc:1048] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:21:36.611515 20284 consensus_queue.cc:1243] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer e063541542a94969898868bf6eb5ce1b log is divergent from this leader: its last log entry 3.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20260709 12:21:36.612198 20284 raft_consensus.cc:3055] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:21:36.612578 20284 raft_consensus.cc:740] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:21:36.613044 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:21:36.613569 17571 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet d522f70beb1f417cb98940d90b594823 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica e2909cff1da44ad387f96a8ae3aa5afb is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 4 committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:21:36.613088 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:36.609124 19806 raft_consensus.cc:3055] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:21:36.613569 20284 consensus_queue.cc:260] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:36.614653 19806 raft_consensus.cc:740] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:21:36.613052 19826 raft_consensus.cc:493] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:36.615657 20284 raft_consensus.cc:3060] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:36.616149 19826 raft_consensus.cc:515] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:36.619074 19826 leader_election.cc:290] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:36.620289 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:36.621066 17681 raft_consensus.cc:2468] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 2.
I20260709 12:21:36.621446 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:36.622084 17757 raft_consensus.cc:2468] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 4.
I20260709 12:21:36.623172 17802 leader_election.cc:304] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:21:36.623889 17686 raft_consensus.cc:2410] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 5 because replica has last-logged OpId of term: 4 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260709 12:21:36.625110 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:36.634027 19976 leader_election.cc:290] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:36.640513 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:36.642412 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:36.647541 17680 raft_consensus.cc:2410] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 7 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 4.
I20260709 12:21:36.660565 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:36.662075 17833 raft_consensus.cc:2468] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 2.
I20260709 12:21:36.684983 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
W20260709 12:21:36.686678 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 81702.7us
W20260709 12:21:36.686818 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 63742.3us
I20260709 12:21:36.688251 17802 leader_election.cc:304] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:21:36.696744 17652 leader_election.cc:304] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:36.698484 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:36.699282 18368 raft_consensus.cc:2468] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 4.
I20260709 12:21:36.699365 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:36.700325 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:36.701190 18368 raft_consensus.cc:2393] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 5: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:21:36.701761 18363 raft_consensus.cc:1275] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20260709 12:21:36.701947 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:36.700680 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:36.703359 18365 raft_consensus.cc:2468] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 2.
I20260709 12:21:36.704371 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:36.705078 18371 raft_consensus.cc:2393] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 6: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:21:36.705258 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:36.706130 18363 raft_consensus.cc:2393] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 5: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:21:36.707022 17687 raft_consensus.cc:1275] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 3. (term mismatch)
I20260709 12:21:36.707453 17687 pending_rounds.cc:85] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3: Aborting all ops after (but not including) 2
I20260709 12:21:36.707741 17687 pending_rounds.cc:107] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20260709 12:21:36.708081 17687 raft_consensus.cc:2889] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:21:36.708429 20266 leader_election.cc:290] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:36.709676 17831 raft_consensus.cc:1275] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 7 index: 8. (term mismatch)
I20260709 12:21:36.710068 17831 pending_rounds.cc:85] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b: Aborting all ops after (but not including) 7
I20260709 12:21:36.710453 17831 pending_rounds.cc:107] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b: Aborting uncommitted NO_OP operation due to leader change: 6.8
I20260709 12:21:36.692667 20153 raft_consensus.cc:2804] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:21:36.714823 20127 raft_consensus.cc:2749] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 12:21:36.720553 20153 raft_consensus.cc:493] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:21:36.721040 20153 raft_consensus.cc:3060] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:36.580425 20152 consensus_queue.cc:260] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 7.9, Last appended by leader: 9, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:36.719734 17831 raft_consensus.cc:2889] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:21:36.710129 19826 raft_consensus.cc:493] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:36.712172 17652 leader_election.cc:304] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:21:36.731719 17652 leader_election.cc:304] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:21:36.732815 17652 leader_election.cc:304] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:21:36.732265 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:36.710398 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:21:36.735965 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 47797us
I20260709 12:21:36.739684 17802 leader_election.cc:304] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:21:36.735697 17803 leader_election.cc:304] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:21:36.738929 17652 leader_election.cc:304] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:21:36.748016 18363 raft_consensus.cc:3060] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:21:36.759934 17802 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:36.657122 20234 raft_consensus.cc:515] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:36.756968 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:36.736404 19976 raft_consensus.cc:493] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:36.735984 20223 raft_consensus.cc:2749] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:21:36.736898 20127 raft_consensus.cc:2804] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:21:36.714083 20019 leader_election.cc:290] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 3 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:36.736229 20186 raft_consensus.cc:2804] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260709 12:21:36.744529 20179 consensus_meta.cc:287] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.600s	user 0.007s	sys 0.006s
I20260709 12:21:36.765515 18364 raft_consensus.cc:2468] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 3.
I20260709 12:21:36.694455 20095 consensus_queue.cc:1048] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.063s
I20260709 12:21:36.769805 18365 raft_consensus.cc:1240] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260709 12:21:36.746497 20147 raft_consensus.cc:2804] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:21:36.771523 17652 leader_election.cc:304] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:21:36.781514 20186 raft_consensus.cc:493] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:36.736172 20157 consensus_queue.cc:1048] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.035s
I20260709 12:21:36.745015 19826 raft_consensus.cc:515] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:36.768601 20052 raft_consensus.cc:2749] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:21:36.770753 20284 raft_consensus.cc:2804] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:21:36.745849 19994 raft_consensus.cc:493] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:36.725314 20299 consensus_queue.cc:1059] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.078s
I20260709 12:21:36.744854 20167 raft_consensus.cc:2804] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:21:36.783027 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:36.785915 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:36.789956 17764 raft_consensus.cc:1240] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 6. Current term is 7. Ops: []
I20260709 12:21:36.782250 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:36.795644 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:36.797660 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:36.802004 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:36.771320 20127 raft_consensus.cc:493] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:36.808285 20152 raft_consensus.cc:3060] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:21:36.786248 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:36.794212 20186 raft_consensus.cc:3060] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:36.790498 20147 raft_consensus.cc:493] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:36.864962 18365 raft_consensus.cc:2468] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 3.
I20260709 12:21:36.783705 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:36.867430 20127 raft_consensus.cc:3060] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:21:36.867811 20284 raft_consensus.cc:697] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 5 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:21:36.868693 17763 raft_consensus.cc:2393] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 6: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:21:36.788092 19976 raft_consensus.cc:515] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:36.868842 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:36.869172 20266 raft_consensus.cc:493] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:36.813660 20102 raft_consensus.cc:493] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:36.869827 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:36.857981 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:36.860014 20167 raft_consensus.cc:493] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:36.797804 20179 raft_consensus.cc:2804] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:21:36.798923 20176 raft_consensus.cc:3055] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 4 LEADER]: Stepping down as leader of term 4
W20260709 12:21:36.859639 20175 consensus_meta.cc:287] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.918s	user 0.004s	sys 0.009s
I20260709 12:21:36.760744 20296 raft_consensus.cc:2749] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260709 12:21:36.922634 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 187012us
I20260709 12:21:36.801677 19826 leader_election.cc:290] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:36.876302 17758 raft_consensus.cc:2393] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 5: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:21:36.793438 20297 raft_consensus.cc:493] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:36.803840 20234 leader_election.cc:290] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 7 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:36.877171 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:36.875815 18368 raft_consensus.cc:2468] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 3.
I20260709 12:21:36.911012 17757 raft_consensus.cc:2468] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 4.
I20260709 12:21:36.781646 20146 raft_consensus.cc:493] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:36.874729 20052 raft_consensus.cc:493] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:21:36.932965 19868 consensus_queue.cc:1048] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.006s
I20260709 12:21:36.793287 20295 raft_consensus.cc:2804] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:21:36.916980 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:36.872583 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:36.877130 20147 raft_consensus.cc:3060] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:36.920679 17763 raft_consensus.cc:1275] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:21:36.870082 20227 raft_consensus.cc:493] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:36.801976 20232 raft_consensus.cc:515] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:36.925108 17803 leader_election.cc:304] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:21:36.906744 20233 raft_consensus.cc:493] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:36.875140 20018 raft_consensus.cc:493] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:21:36.932919 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:36.934267 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:21:36.923625 20300 log.cc:927] Time spent T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.061s	user 0.000s	sys 0.001s
I20260709 12:21:36.918552 20167 raft_consensus.cc:3060] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:36.921147 20179 raft_consensus.cc:493] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:36.872977 19994 raft_consensus.cc:515] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:36.919155 20284 consensus_queue.cc:237] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:36.921041 20176 raft_consensus.cc:740] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 4 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:21:36.949368 17756 raft_consensus.cc:2468] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 3.
I20260709 12:21:36.802644 20247 raft_consensus.cc:515] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:36.936009 20052 raft_consensus.cc:515] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:36.957530 20247 leader_election.cc:290] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 5 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:36.957846 20052 leader_election.cc:290] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:36.940863 20295 raft_consensus.cc:493] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:36.953016 17803 leader_election.cc:304] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:21:36.916275 20102 raft_consensus.cc:515] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:36.955305 17838 raft_consensus.cc:1275] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 5 index: 8. Preceding OpId from leader: term: 6 index: 8. (term mismatch)
I20260709 12:21:36.930172 20175 raft_consensus.cc:515] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:36.960412 20311 raft_consensus.cc:2804] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260709 12:21:36.962321 20261 log.cc:927] Time spent T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 1.066s	user 0.006s	sys 0.000s
I20260709 12:21:36.967708 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:36.940413 20146 raft_consensus.cc:515] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:36.959793 17652 leader_election.cc:304] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:21:36.955698 20296 raft_consensus.cc:2749] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:21:36.929827 20266 raft_consensus.cc:515] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:36.951472 20179 raft_consensus.cc:3060] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:36.966459 17763 raft_consensus.cc:1275] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260709 12:21:36.958427 20295 raft_consensus.cc:3060] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:36.945575 20297 raft_consensus.cc:515] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:36.953522 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:36.951702 20018 raft_consensus.cc:515] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:36.963748 17838 pending_rounds.cc:85] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b: Aborting all ops after (but not including) 7
I20260709 12:21:37.011344 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:37.022480 18362 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 924) took 1425 ms. Trace:
I20260709 12:21:36.990775 18333 leader_election.cc:304] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:21:36.992954 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:36.976851 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:36.980652 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:37.011211 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:21:37.000654 17801 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:36.992385 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:37.030368 17760 raft_consensus.cc:2410] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260709 12:21:36.987087 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:37.027663 17838 pending_rounds.cc:107] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b: Aborting uncommitted NO_OP operation due to leader change: 5.8
I20260709 12:21:37.029378 18333 leader_election.cc:304] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:21:37.035116 17838 raft_consensus.cc:2889] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:21:36.994786 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:36.968283 20311 raft_consensus.cc:493] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:36.982656 17756 raft_consensus.cc:2468] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 4.
I20260709 12:21:37.036738 17758 raft_consensus.cc:2368] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 3. Current term is 4.
I20260709 12:21:37.029634 17757 raft_consensus.cc:2393] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 2: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:21:37.028167 17831 raft_consensus.cc:3060] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:37.105041 17761 raft_consensus.cc:2468] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 2.
I20260709 12:21:37.076184 18363 raft_consensus.cc:2468] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 6.
W20260709 12:21:37.076800 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 117164us
I20260709 12:21:37.074877 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:37.034310 17764 raft_consensus.cc:2376] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 4. Re-sending same reply.
I20260709 12:21:37.110260 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:37.111936 17652 leader_election.cc:304] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:37.112514 17763 raft_consensus.cc:2468] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 4.
I20260709 12:21:37.114210 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:37.115337 17758 raft_consensus.cc:2410] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for term 4 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260709 12:21:36.990118 19806 consensus_queue.cc:260] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:37.116938 19806 raft_consensus.cc:3060] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:37.120023 20273 raft_consensus.cc:493] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:37.120623 20273 raft_consensus.cc:515] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:36.976842 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:37.130091 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:37.111959 17760 raft_consensus.cc:3060] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:37.131753 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:37.125589 17652 leader_election.cc:304] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:21:37.017616 17829 raft_consensus.cc:1275] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260709 12:21:36.964653 20233 raft_consensus.cc:515] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:37.132534 17758 raft_consensus.cc:2468] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 3.
I20260709 12:21:37.017686 20175 leader_election.cc:290] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 5 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:36.995074 20232 leader_election.cc:290] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 4 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:37.016315 20266 leader_election.cc:290] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:37.036528 20247 raft_consensus.cc:493] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:37.035745 20019 raft_consensus.cc:2804] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:21:37.025229 19855 leader_election.cc:290] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 6 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:37.143028 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:37.152899 18363 raft_consensus.cc:2468] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 5.
I20260709 12:21:37.146184 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:37.153640 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:36.959784 20227 raft_consensus.cc:515] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:37.074946 20052 raft_consensus.cc:493] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:37.014694 20102 leader_election.cc:290] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:37.109429 20076 raft_consensus.cc:2749] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:21:37.157733 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:37.075763 17755 raft_consensus.cc:2468] T 7cb3768d215e409fb9f5baf18d124442 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 3.
I20260709 12:21:37.111691 20311 raft_consensus.cc:3060] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:37.161988 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:37.162389 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:37.162474 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:37.162762 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:37.021648 20186 raft_consensus.cc:515] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:37.028756 18362 rpcz_store.cc:276] 0709 12:21:35.596645 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:35.605577 (+  8932us) service_pool.cc:224] Handling call
0709 12:21:35.641182 (+ 35605us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:35.651642 (+ 10460us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:35.651677 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:35.652306 (+   629us) log.cc:844] Serialized 26 byte log entry
0709 12:21:35.741266 (+ 88960us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:21:35.741305 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:35.741412 (+   107us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:36.961632 (+1220220us) spinlock_profiling.cc:243] Waited 6.3 ms on lock 0x7b4400022b4c. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b47459c85 00007f7b4745a6a2 00007f7b4a1cc1db 00007f7b4a1c7aaf 00007f7b4a1bb86e 00007f7b4a1babb6 00007f7b4a1ba7d8 00007f7b4a1c4221 00007f7b4a1c41d9 00007f7b4a1c4169
0709 12:21:37.022022 (+ 60390us) raft_consensus.cc:1685] finished
0709 12:21:37.022043 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:37.022271 (+   228us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":7518,"raft.run_cpu_time_us":105,"raft.run_wall_time_us":105,"spinlock_wait_cycles":14493952,"thread_start_us":88539,"threads_started":1,"wal-append.queue_time_us":193389}
I20260709 12:21:37.166678 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:37.123562 20273 leader_election.cc:290] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:37.163605 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:37.023845 20179 raft_consensus.cc:515] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:36.967916 20176 consensus_queue.cc:260] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:37.170758 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:37.353766 17688 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44898 (request call id 554) took 1759 ms. Trace:
I20260709 12:21:37.405453 17762 raft_consensus.cc:2393] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 7: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
W20260709 12:21:37.353679 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 324356us
I20260709 12:21:37.406754 18333 leader_election.cc:304] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:21:37.231292 17760 raft_consensus.cc:2468] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 5.
W20260709 12:21:37.407317 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 273398us
W20260709 12:21:37.408347 17652 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:37.276379 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:37.342429 17835 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35042 (request call id 1693) took 1345 ms. Trace:
W20260709 12:21:37.410673 17729 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:37.329648 20178 raft_consensus.cc:493] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:21:37.138516 20196 raft_consensus.cc:2749] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 12:21:37.144285 20299 raft_consensus.cc:2804] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:21:37.407985 17838 raft_consensus.cc:3060] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:37.405375 17688 rpcz_store.cc:276] 0709 12:21:35.594836 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:35.595033 (+   197us) service_pool.cc:224] Handling call
0709 12:21:35.616202 (+ 21169us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:35.832164 (+215962us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:37.353375 (+1521211us) spinlock_profiling.cc:243] Waited 186 ms on lock 0x7b5400347bd4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:21:37.353626 (+   251us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":426891008}
I20260709 12:21:37.410499 17835 rpcz_store.cc:276] 0709 12:21:35.997316 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:36.008684 (+ 11368us) service_pool.cc:224] Handling call
0709 12:21:37.342200 (+1333516us) spinlock_profiling.cc:243] Waited 168 ms on lock 0x7b54001581d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:21:37.342340 (+   140us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":385783168}
I20260709 12:21:37.153702 20019 raft_consensus.cc:493] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:21:37.148975 20293 raft_consensus.cc:493] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:37.152493 20095 raft_consensus.cc:493] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:37.342104 20307 raft_consensus.cc:493] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:37.297339 20076 raft_consensus.cc:493] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:37.164800 20296 raft_consensus.cc:493] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:37.435320 18363 raft_consensus.cc:2468] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 4.
I20260709 12:21:37.285851 20232 raft_consensus.cc:493] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:37.364928 19976 leader_election.cc:290] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:37.438388 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:37.223464 19868 raft_consensus.cc:493] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:37.341574 20284 raft_consensus.cc:493] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:37.439112 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:37.451294 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:37.403455 20227 leader_election.cc:290] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:37.334880 19826 raft_consensus.cc:493] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:21:37.224784 20146 leader_election.cc:290] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:37.411249 20158 raft_consensus.cc:493] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:37.418362 20299 raft_consensus.cc:493] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:37.424587 18361 raft_consensus.cc:3060] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:37.418111 19855 raft_consensus.cc:2804] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:21:37.447333 17729 leader_election.cc:304] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:21:37.477864 19826 raft_consensus.cc:515] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:37.510015 19826 leader_election.cc:290] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:37.510886 19826 raft_consensus.cc:2749] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 12:21:37.479779 20158 raft_consensus.cc:515] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:37.514602 20158 leader_election.cc:290] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:37.514914 20233 leader_election.cc:290] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:37.480490 20076 raft_consensus.cc:515] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:37.520875 20076 leader_election.cc:290] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:37.521059 18374 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59918 (request call id 926) took 1800 ms. Trace:
I20260709 12:21:37.521919 20297 leader_election.cc:290] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:37.434568 20019 raft_consensus.cc:3060] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:21:37.526166 20152 consensus_meta.cc:287] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.653s	user 0.006s	sys 0.003s
I20260709 12:21:37.464546 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:37.434906 20196 raft_consensus.cc:493] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:37.210821 20052 raft_consensus.cc:515] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:37.528853 18363 raft_consensus.cc:3060] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:37.528939 20196 raft_consensus.cc:515] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:37.531129 20196 leader_election.cc:290] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:37.210009 20153 raft_consensus.cc:515] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:37.418154 20102 consensus_queue.cc:1048] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.039s
I20260709 12:21:37.474617 20284 raft_consensus.cc:515] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:37.223016 20127 raft_consensus.cc:515] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:37.533075 20153 leader_election.cc:290] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 5 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:37.533737 17837 raft_consensus.cc:1240] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 4. Current term is 5. Ops: []
I20260709 12:21:37.533923 20284 leader_election.cc:290] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:37.170246 20314 consensus_queue.cc:1048] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.307s
I20260709 12:21:37.534475 20127 leader_election.cc:290] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 9 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:37.535506 20052 leader_election.cc:290] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:37.535660 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:37.536626 17832 raft_consensus.cc:3060] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:37.437474 20178 raft_consensus.cc:515] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:37.539618 20178 leader_election.cc:290] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:37.540951 17830 raft_consensus.cc:1240] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 4. Current term is 5. Ops: []
I20260709 12:21:37.476651 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:37.545243 17835 raft_consensus.cc:1275] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:21:37.547786 17687 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37396 (request call id 1822) took 1620 ms. Trace:
I20260709 12:21:37.548162 20018 leader_election.cc:290] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:37.548249 17687 rpcz_store.cc:276] 0709 12:21:35.927032 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:35.934091 (+  7059us) service_pool.cc:224] Handling call
0709 12:21:35.934396 (+   305us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:36.708865 (+774469us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:37.547552 (+838687us) spinlock_profiling.cc:243] Waited 772 ms on lock 0x7b5400348ad4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:21:37.547727 (+   175us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1775965440}
I20260709 12:21:37.477296 17683 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37402 (request call id 2225) took 1684 ms. Trace:
I20260709 12:21:37.549451 17683 rpcz_store.cc:276] 0709 12:21:35.792991 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:35.858490 (+ 65499us) service_pool.cc:224] Handling call
0709 12:21:35.881450 (+ 22960us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:36.554025 (+672575us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:37.477025 (+923000us) spinlock_profiling.cc:243] Waited 644 ms on lock 0x7b54004259d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:21:37.477232 (+   207us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1482047872}
W20260709 12:21:37.551424 20147 consensus_meta.cc:287] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.603s	user 0.006s	sys 0.001s
I20260709 12:21:37.440882 20293 raft_consensus.cc:515] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:37.467909 20232 raft_consensus.cc:515] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:37.553999 20232 leader_election.cc:290] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:37.554056 20293 leader_election.cc:290] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:37.555848 18366 raft_consensus.cc:1240] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 1. Current term is 2. Ops: []
I20260709 12:21:37.528115 20186 leader_election.cc:290] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 5 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:37.481782 20299 raft_consensus.cc:3060] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:37.551872 20147 raft_consensus.cc:515] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:37.579677 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:37.580416 17834 raft_consensus.cc:2468] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 4.
I20260709 12:21:37.581290 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:37.582461 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:37.582964 17834 raft_consensus.cc:2468] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 4.
I20260709 12:21:37.584106 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:37.478524 20266 raft_consensus.cc:2749] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:21:37.543624 17830 raft_consensus.cc:1275] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260709 12:21:37.570214 18362 raft_consensus.cc:2468] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 4.
I20260709 12:21:37.482518 19994 leader_election.cc:290] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:37.521660 18374 rpcz_store.cc:276] 0709 12:21:35.720858 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:35.726946 (+  6088us) service_pool.cc:224] Handling call
0709 12:21:37.520742 (+1793796us) spinlock_profiling.cc:243] Waited 254 ms on lock 0x7b54008020d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:21:37.520901 (+   159us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":584690432}
I20260709 12:21:37.492926 19976 raft_consensus.cc:493] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:21:37.481204 19868 raft_consensus.cc:515] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:37.585696 17831 raft_consensus.cc:2468] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 5.
I20260709 12:21:37.478919 20298 raft_consensus.cc:493] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:21:37.458000 20095 raft_consensus.cc:515] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:37.640832 20298 raft_consensus.cc:515] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:37.641747 19868 leader_election.cc:290] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:37.642560 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:37.573073 18363 raft_consensus.cc:2468] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 4.
I20260709 12:21:37.643245 17830 raft_consensus.cc:2468] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 3.
I20260709 12:21:37.585844 18371 raft_consensus.cc:2468] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 5.
I20260709 12:21:37.644744 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:37.645390 18371 raft_consensus.cc:3060] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:37.600224 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:37.454118 20307 raft_consensus.cc:515] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:37.584318 20323 raft_consensus.cc:493] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:21:37.635228 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:37.486656 19855 raft_consensus.cc:493] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:37.157030 20247 raft_consensus.cc:515] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:37.471069 20296 raft_consensus.cc:515] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:37.584889 20019 raft_consensus.cc:515] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:37.671617 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:37.672240 20264 raft_consensus.cc:493] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:37.658960 19976 raft_consensus.cc:515] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:21:37.677691 20167 consensus_meta.cc:287] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.710s	user 0.011s	sys 0.000s
W20260709 12:21:37.714322 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 300332us
I20260709 12:21:37.716922 17833 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35042 (request call id 1701) took 1145 ms. Trace:
I20260709 12:21:37.720436 17757 raft_consensus.cc:2410] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 7 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 4.
I20260709 12:21:37.721318 20323 raft_consensus.cc:515] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:37.730294 20311 raft_consensus.cc:515] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:37.731043 18367 raft_consensus.cc:2468] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 8.
I20260709 12:21:37.731340 20323 leader_election.cc:290] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:37.732298 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:37.733740 20019 leader_election.cc:290] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 5 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:21:37.736006 20295 consensus_meta.cc:287] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.616s	user 0.008s	sys 0.003s
I20260709 12:21:37.736908 19976 leader_election.cc:290] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:37.736625 20295 raft_consensus.cc:515] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:37.739313 18333 leader_election.cc:304] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:37.740499 20019 raft_consensus.cc:2749] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:21:37.749225 20168 raft_consensus.cc:3055] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:21:37.749747 20168 raft_consensus.cc:740] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 4 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:21:37.751700 18368 raft_consensus.cc:1240] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 4. Current term is 5. Ops: []
I20260709 12:21:37.724393 20264 raft_consensus.cc:515] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:37.754557 17836 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35042 (request call id 1691) took 1808 ms. Trace:
I20260709 12:21:37.754940 17836 rpcz_store.cc:276] 0709 12:21:35.945835 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:35.945996 (+   161us) service_pool.cc:224] Handling call
0709 12:21:35.954073 (+  8077us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:36.479237 (+525164us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:37.754299 (+1275062us) spinlock_profiling.cc:243] Waited 525 ms on lock 0x7b54003359d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:21:37.754471 (+   172us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1206482176}
I20260709 12:21:37.755803 18362 raft_consensus.cc:2468] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 4.
I20260709 12:21:37.756625 17835 raft_consensus.cc:2468] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 6.
I20260709 12:21:37.758554 20324 raft_consensus.cc:493] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:37.759114 20324 raft_consensus.cc:515] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:37.706758 19855 raft_consensus.cc:3060] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:37.765290 20311 leader_election.cc:290] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 3 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:37.765537 20179 leader_election.cc:290] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 5 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:37.765851 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:37.766366 20179 raft_consensus.cc:493] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:37.767629 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:37.767026 20179 raft_consensus.cc:515] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:37.768514 17680 raft_consensus.cc:3060] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:37.770044 20179 leader_election.cc:290] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:37.772338 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:37.773415 17686 raft_consensus.cc:2393] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 5: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:21:37.774508 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:37.774982 17756 raft_consensus.cc:2468] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 2.
I20260709 12:21:37.775158 17682 raft_consensus.cc:2468] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 5.
I20260709 12:21:37.776300 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:37.780337 20176 raft_consensus.cc:3060] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:37.728165 20167 raft_consensus.cc:515] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:21:37.838922 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 405029us
I20260709 12:21:37.840821 18333 leader_election.cc:304] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:37.766945 17688 raft_consensus.cc:2393] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 4: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
W20260709 12:21:37.844300 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:37.765399 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:37.799386 17583 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:37172 (request call id 252) took 6581 ms. Trace:
I20260709 12:21:37.857393 17838 raft_consensus.cc:2468] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 4.
I20260709 12:21:37.863605 18371 raft_consensus.cc:2468] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 5.
W20260709 12:21:37.804397 18334 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.16.29.65:33071 timed out after 1.605s (SENT) [suppressed 26 similar messages]
I20260709 12:21:37.878916 18367 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59886 (request call id 780) took 1305 ms. Trace:
I20260709 12:21:37.879294 18367 rpcz_store.cc:276] 0709 12:21:36.573928 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:36.574067 (+   139us) service_pool.cc:224] Handling call
0709 12:21:37.878723 (+1304656us) spinlock_profiling.cc:243] Waited 955 ms on lock 0x7b54006739d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:21:37.878859 (+   136us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2196677888}
I20260709 12:21:37.882269 18367 raft_consensus.cc:1240] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 5. Current term is 6. Ops: []
I20260709 12:21:37.880632 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:37.787958 20167 leader_election.cc:290] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 7 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:37.884053 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:37.885720 20324 leader_election.cc:290] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:37.886134 17832 raft_consensus.cc:2468] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 3.
I20260709 12:21:37.885980 20299 raft_consensus.cc:515] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:37.886541 17829 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35042 (request call id 1702) took 1028 ms. Trace:
I20260709 12:21:37.886847 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:37.884068 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 8 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:37.905087 17835 raft_consensus.cc:2468] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 7.
I20260709 12:21:37.888489 20264 leader_election.cc:290] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:21:37.889160 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 84787.8us
I20260709 12:21:37.890231 18366 raft_consensus.cc:1275] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260709 12:21:37.890328 17654 leader_election.cc:304] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:21:37.891492 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:37.890286 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:21:37.803408 20310 log.cc:927] Time spent T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.403s	user 0.001s	sys 0.003s
I20260709 12:21:37.896977 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:37.913030 17683 raft_consensus.cc:2468] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 4.
I20260709 12:21:37.894551 17685 raft_consensus.cc:2468] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 2.
I20260709 12:21:37.898144 18364 raft_consensus.cc:1240] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260709 12:21:37.898617 20295 leader_election.cc:290] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 3 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:37.857055 17583 rpcz_store.cc:276] 0709 12:21:31.224401 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:31.237410 (+ 13009us) service_pool.cc:224] Handling call
0709 12:21:37.799255 (+6561845us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:21:35.661989 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:21:35.781561 (+119572us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:21:35.781697 (+   136us) write_op.cc:435] Acquired schema lock
0709 12:21:35.781746 (+    49us) tablet.cc:662] Decoding operations
0709 12:21:35.782708 (+   962us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:21:35.782792 (+    84us) write_op.cc:641] Partition lock acquired for write op
0709 12:21:35.782822 (+    30us) tablet.cc:685] Acquiring locks for 5 operations
0709 12:21:35.783556 (+   734us) tablet.cc:701] Row locks acquired
0709 12:21:35.783603 (+    47us) write_op.cc:260] PREPARE: finished
0709 12:21:35.783807 (+   204us) write_op.cc:270] Start()
0709 12:21:35.824383 (+ 40576us) write_op.cc:276] Timestamp: P: 1783599695783738 usec, L: 0
0709 12:21:35.826971 (+  2588us) op_driver.cc:348] REPLICATION: starting
0709 12:21:35.943787 (+116816us) log.cc:844] Serialized 2116 byte log entry
0709 12:21:36.450932 (+507145us) op_driver.cc:464] REPLICATION: finished
0709 12:21:36.724602 (+273670us) write_op.cc:301] APPLY: starting
0709 12:21:36.724752 (+   150us) tablet.cc:1367] starting BulkCheckPresence
0709 12:21:36.725087 (+   335us) tablet.cc:1370] finished BulkCheckPresence
0709 12:21:36.760285 (+ 35198us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:21:36.914431 (+154146us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:21:36.919300 (+  4869us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=5
0709 12:21:36.919356 (+    56us) write_op.cc:312] APPLY: finished
0709 12:21:36.923990 (+  4634us) log.cc:844] Serialized 47 byte log entry
0709 12:21:36.938256 (+ 14266us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:21:36.946146 (+  7890us) write_op.cc:454] Released schema lock
0709 12:21:36.949697 (+  3551us) write_op.cc:341] FINISH: Updating metrics
0709 12:21:36.960412 (+ 10715us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:21:36.960450 (+    38us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":1888806,"reported_tablets":8,"child_traces":[["op",{"apply.queue_time_us":266646,"apply.run_cpu_time_us":5340,"apply.run_wall_time_us":235810,"num_ops":5,"prepare.queue_time_us":644749,"prepare.run_cpu_time_us":82175,"prepare.run_wall_time_us":475186,"raft.queue_time_us":115057,"raft.run_cpu_time_us":17872,"raft.run_wall_time_us":258396,"replication_time_us":501290,"thread_start_us":821492,"threads_started":4,"wal-append.queue_time_us":190368}]]}
I20260709 12:21:37.900806 20232 consensus_queue.cc:1059] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.016s
I20260709 12:21:37.793630 20157 raft_consensus.cc:2955] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 3 LEADER]: Committing config change with OpId 3.6: config changed from index 5 to 6, VOTER e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:21:37.917049 20232 raft_consensus.cc:3055] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:21:37.917152 17684 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37402 (request call id 2230) took 1350 ms. Trace:
I20260709 12:21:37.917477 20232 raft_consensus.cc:740] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:21:37.918320 20232 consensus_queue.cc:260] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:37.917698 17684 rpcz_store.cc:276] 0709 12:21:36.566676 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:36.566840 (+   164us) service_pool.cc:224] Handling call
0709 12:21:36.595391 (+ 28551us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:36.600914 (+  5523us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:36.600956 (+    42us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:36.709749 (+108793us) log.cc:844] Serialized 26 byte log entry
0709 12:21:36.947507 (+237758us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:21:36.947551 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:36.947621 (+    70us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:37.802613 (+854992us) spinlock_profiling.cc:243] Waited 5.26 ms on lock 0x7b4400022b4c. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b47459c85 00007f7b4745a6a2 00007f7b4a1cc1db 00007f7b4a1c7aaf 00007f7b4a1bb86e 00007f7b4a1babb6 00007f7b4a1ba7d8 00007f7b4a1c4221 00007f7b4a1c41d9 00007f7b4a1c4169
0709 12:21:37.916800 (+114187us) raft_consensus.cc:1685] finished
0709 12:21:37.916829 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:37.917081 (+   252us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1372,"raft.queue_time_us":119,"raft.run_cpu_time_us":195,"raft.run_wall_time_us":57948,"spinlock_wait_cycles":12098816,"thread_start_us":237383,"threads_started":1,"wal-append.queue_time_us":515670}
I20260709 12:21:37.902870 20296 leader_election.cc:290] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:37.900071 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:37.901996 18333 leader_election.cc:304] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:37.925428 17830 raft_consensus.cc:2468] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 6.
I20260709 12:21:37.877434 20266 raft_consensus.cc:2749] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:21:37.906984 19961 leader_election.cc:290] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:37.910306 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:37.899564 18371 raft_consensus.cc:1240] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 4. Current term is 5. Ops: []
I20260709 12:21:37.927589 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:37.927587 17688 raft_consensus.cc:2468] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 6.
I20260709 12:21:37.920517 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:37.911269 17686 raft_consensus.cc:2468] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 2.
W20260709 12:21:37.929419 17652 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.520s	user 0.003s	sys 0.026s
I20260709 12:21:37.899928 20147 leader_election.cc:290] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 6 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:37.930433 17689 raft_consensus.cc:2393] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 5: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:21:37.930848 17837 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35042 (request call id 1704) took 1067 ms. Trace:
W20260709 12:21:37.931479 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 52797us
W20260709 12:21:37.929806 17652 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.520s	user 0.003s	sys 0.026s
I20260709 12:21:37.919695 20232 raft_consensus.cc:3060] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:37.932883 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:37.933215 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:37.907994 17829 rpcz_store.cc:276] 0709 12:21:36.857850 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:37.296321 (+438471us) service_pool.cc:224] Handling call
0709 12:21:37.401593 (+105272us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:37.401852 (+   259us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:21:37.402212 (+   360us) log.cc:844] Serialized 14 byte log entry
0709 12:21:37.886134 (+483922us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:37.886198 (+    64us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:21:37.886256 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:37.886334 (+    78us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:37.886464 (+   130us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":137300,"prepare.queue_time_us":2975,"prepare.run_cpu_time_us":59,"prepare.run_wall_time_us":6450,"raft.queue_time_us":146028,"raft.run_cpu_time_us":81,"raft.run_wall_time_us":83,"thread_start_us":338355,"threads_started":1}
I20260709 12:21:37.936597 17683 raft_consensus.cc:2468] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 2.
I20260709 12:21:37.944401 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:37.921234 20299 leader_election.cc:290] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 3 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:21:37.936604 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 44403.7us
I20260709 12:21:37.936053 17685 raft_consensus.cc:2468] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 4.
I20260709 12:21:37.939239 17830 raft_consensus.cc:1240] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 2. Current term is 3. Ops: [2.5-2.5]
I20260709 12:21:37.958526 17729 leader_election.cc:304] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:21:37.954946 17760 raft_consensus.cc:3060] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:37.959893 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:37.960106 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:37.960968 17835 raft_consensus.cc:1240] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260709 12:21:37.961014 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:37.961272 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:37.961845 17762 raft_consensus.cc:2468] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 3.
I20260709 12:21:37.962082 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:37.962491 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:37.962725 17835 raft_consensus.cc:2468] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 5.
I20260709 12:21:37.963040 17763 raft_consensus.cc:2468] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 4.
I20260709 12:21:37.963083 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:37.963752 17830 raft_consensus.cc:2468] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 4.
I20260709 12:21:37.964344 17835 raft_consensus.cc:1275] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:21:37.964699 17757 raft_consensus.cc:1240] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 5. Current term is 6. Ops: []
I20260709 12:21:37.968351 17729 leader_election.cc:304] T 7cb3768d215e409fb9f5baf18d124442 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:21:37.949331 18333 leader_election.cc:304] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:37.914927 17680 raft_consensus.cc:2468] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 7.
I20260709 12:21:37.927204 20324 raft_consensus.cc:2804] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:21:37.960235 17756 raft_consensus.cc:1275] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260709 12:21:37.930913 20264 raft_consensus.cc:2749] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:21:37.986194 17652 leader_election.cc:304] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:21:37.988196 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:37.989008 17763 raft_consensus.cc:3060] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:37.989624 17689 raft_consensus.cc:1240] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 8. Current term is 9. Ops: [8.9-8.9]
I20260709 12:21:37.990720 17759 raft_consensus.cc:1275] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260709 12:21:37.991024 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:37.991904 17762 raft_consensus.cc:3060] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:37.940953 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:37.931442 17837 rpcz_store.cc:276] 0709 12:21:36.863545 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:37.537537 (+673992us) service_pool.cc:224] Handling call
0709 12:21:37.555414 (+ 17877us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:37.555650 (+   236us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:21:37.579042 (+ 23392us) log.cc:844] Serialized 14 byte log entry
0709 12:21:37.930495 (+351453us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:37.930550 (+    55us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:21:37.930602 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:37.930673 (+    71us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:37.930793 (+   120us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":28989,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":36,"thread_start_us":345955,"threads_started":1}
I20260709 12:21:37.993471 17758 raft_consensus.cc:3060] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:37.977922 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:37.979717 17729 leader_election.cc:304] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:21:37.956761 20127 raft_consensus.cc:493] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:37.961884 17764 raft_consensus.cc:3060] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:21:37.985487 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:37.986332 17683 raft_consensus.cc:1240] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 7. Current term is 8. Ops: []
I20260709 12:21:37.977698 17761 raft_consensus.cc:1240] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 4. Current term is 5. Ops: [2.2-4.3]
I20260709 12:21:37.939442 20178 consensus_queue.cc:1059] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.037s
I20260709 12:21:37.999747 17652 leader_election.cc:304] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:21:38.009570 17836 raft_consensus.cc:2468] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 4.
I20260709 12:21:38.010797 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:38.016161 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:38.014772 17685 raft_consensus.cc:1275] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 6. (term mismatch)
I20260709 12:21:38.018611 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:38.023272 17685 pending_rounds.cc:85] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3: Aborting all ops after (but not including) 5
I20260709 12:21:38.023617 17685 pending_rounds.cc:107] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3: Aborting uncommitted NO_OP operation due to leader change: 3.6
I20260709 12:21:37.981881 19961 raft_consensus.cc:2749] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:21:38.024066 17685 raft_consensus.cc:2889] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:21:38.013572 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
W20260709 12:21:38.023481 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 43893.9us
I20260709 12:21:38.024757 17834 raft_consensus.cc:1275] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260709 12:21:38.016331 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:38.022647 17652 leader_election.cc:304] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:38.020759 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:38.025441 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:38.025918 17757 raft_consensus.cc:2468] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 2.
I20260709 12:21:38.026228 17759 raft_consensus.cc:2468] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 4.
I20260709 12:21:38.027341 17760 raft_consensus.cc:2468] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 5.
I20260709 12:21:38.029533 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:38.030282 17837 raft_consensus.cc:3060] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:21:38.031111 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 95497.8us
I20260709 12:21:38.032558 20178 consensus_queue.cc:1243] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer e2909cff1da44ad387f96a8ae3aa5afb log is divergent from this leader: its last log entry 5.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260709 12:21:38.020328 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:38.033388 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:37.985332 19994 raft_consensus.cc:2804] T 7cb3768d215e409fb9f5baf18d124442 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:21:37.984467 20324 raft_consensus.cc:493] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:21:38.036660 17685 raft_consensus.cc:2393] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 3: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:21:38.038509 17652 leader_election.cc:304] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:21:38.033238 20178 consensus_queue.cc:1059] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.003s
I20260709 12:21:38.015163 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:37.999364 20321 raft_consensus.cc:493] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:37.995095 20266 raft_consensus.cc:2749] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 12:21:38.040714 17729 leader_election.cc:304] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:21:38.016281 19753 raft_consensus.cc:2804] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:21:38.057474 18333 leader_election.cc:304] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:38.065215 17728 leader_election.cc:304] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:21:37.985730 20175 consensus_queue.cc:1048] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.056s
I20260709 12:21:38.014962 20299 raft_consensus.cc:2804] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:21:38.040704 20324 raft_consensus.cc:3060] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:38.048018 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:37.988063 20284 consensus_queue.cc:1048] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.053s
I20260709 12:21:38.067512 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 8 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:38.010496 18361 raft_consensus.cc:2468] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 5.
I20260709 12:21:38.033663 19826 raft_consensus.cc:2804] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:21:38.037634 19994 raft_consensus.cc:493] T 7cb3768d215e409fb9f5baf18d124442 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:21:38.079684 20196 raft_consensus.cc:2804] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:21:38.042999 20186 raft_consensus.cc:2804] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260709 12:21:38.089843 17803 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.946s	user 0.028s	sys 0.073s
W20260709 12:21:38.126209 17803 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.982s	user 0.029s	sys 0.073s
I20260709 12:21:38.086989 17728 leader_election.cc:304] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
W20260709 12:21:38.132902 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 46024.2us
I20260709 12:21:38.080729 19753 raft_consensus.cc:493] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:21:38.147094 17803 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:38.147554 17803 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:38.082597 20299 raft_consensus.cc:493] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:38.059175 20234 consensus_queue.cc:1048] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.070s
I20260709 12:21:38.070982 20144 consensus_queue.cc:1048] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.066s
I20260709 12:21:38.070315 20102 consensus_queue.cc:1048] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.067s
I20260709 12:21:38.120154 20196 raft_consensus.cc:493] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:38.131404 20266 raft_consensus.cc:2749] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:21:38.125948 20186 raft_consensus.cc:493] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:38.123750 19994 raft_consensus.cc:3060] T 7cb3768d215e409fb9f5baf18d124442 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:38.154031 19753 raft_consensus.cc:3060] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:38.088063 20264 consensus_queue.cc:1059] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.082s
I20260709 12:21:38.088796 20127 raft_consensus.cc:515] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:21:38.247614 17730 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:38.212150 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 319965us
I20260709 12:21:38.164878 19826 raft_consensus.cc:697] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 6 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:21:38.172580 20299 raft_consensus.cc:3060] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:38.165055 20018 raft_consensus.cc:2804] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:21:38.165725 20332 raft_consensus.cc:2804] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:21:38.209986 20186 raft_consensus.cc:3060] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:38.227465 17763 raft_consensus.cc:2468] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 6.
I20260709 12:21:38.201123 20196 raft_consensus.cc:3060] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:38.100873 20321 raft_consensus.cc:515] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:38.288255 17837 raft_consensus.cc:2468] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 3.
W20260709 12:21:38.294966 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 161288us
I20260709 12:21:38.258229 20018 raft_consensus.cc:493] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:38.298313 17728 leader_election.cc:304] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:21:38.270900 17680 raft_consensus.cc:2468] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 4.
I20260709 12:21:38.264276 20266 consensus_queue.cc:1048] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.032s
I20260709 12:21:38.288754 20234 raft_consensus.cc:2749] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260709 12:21:38.305274 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 272644us
I20260709 12:21:38.317338 17682 raft_consensus.cc:1240] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 5. Current term is 6. Ops: [5.4-5.4]
W20260709 12:21:38.328649 20328 log.cc:927] Time spent T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.055s	user 0.000s	sys 0.000s
I20260709 12:21:38.327752 19753 raft_consensus.cc:515] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:38.329241 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:38.330056 20321 leader_election.cc:290] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:38.330654 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:38.332377 17762 raft_consensus.cc:2468] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 5.
I20260709 12:21:38.332527 18365 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59902 (request call id 791) took 1372 ms. Trace:
I20260709 12:21:38.330086 17688 raft_consensus.cc:2393] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 3: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:21:38.336509 17728 leader_election.cc:304] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:21:38.337689 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 297064us
I20260709 12:21:38.339509 17729 leader_election.cc:304] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:21:38.342495 17728 leader_election.cc:304] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:21:38.370594 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:38.316987 20178 raft_consensus.cc:493] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:38.369737 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:38.374477 17728 leader_election.cc:304] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:21:38.318840 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 280157us
I20260709 12:21:38.377913 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:38.333065 18365 rpcz_store.cc:276] 0709 12:21:36.960574 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:37.018958 (+ 58384us) service_pool.cc:224] Handling call
0709 12:21:37.019323 (+   365us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:37.020566 (+  1243us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 12:21:37.285664 (+265098us) log.cc:844] Serialized 14 byte log entry
0709 12:21:37.587701 (+302037us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:37.699073 (+111372us) log.cc:844] Serialized 26 byte log entry
0709 12:21:37.699295 (+   222us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:21:37.699667 (+   372us) log.cc:844] Serialized 14 byte log entry
0709 12:21:37.719095 (+ 19428us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:37.719192 (+    97us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:38.332241 (+613049us) raft_consensus.cc:1685] finished
0709 12:21:38.332269 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:38.332443 (+   174us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":613239,"prepare.run_cpu_time_us":116,"prepare.run_wall_time_us":6523,"raft.queue_time_us":31027,"raft.run_cpu_time_us":424,"raft.run_wall_time_us":12677,"thread_start_us":396726,"threads_started":2,"wal-append.queue_time_us":844264}
I20260709 12:21:38.380173 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:38.342360 20334 raft_consensus.cc:493] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:38.312109 20018 raft_consensus.cc:3060] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:38.385135 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:38.385802 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:38.381673 17728 leader_election.cc:304] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:21:38.390021 19855 consensus_meta.cc:287] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.628s	user 0.004s	sys 0.008s
I20260709 12:21:38.390713 20336 raft_consensus.cc:493] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:38.368011 20314 raft_consensus.cc:493] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
W20260709 12:21:38.406771 17802 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:38.409261 17728 leader_election.cc:304] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:21:38.384176 20234 raft_consensus.cc:2804] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:21:38.349655 18334 leader_election.cc:304] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:21:38.379956 19753 leader_election.cc:290] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 4 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:38.399794 17758 raft_consensus.cc:2468] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 5.
I20260709 12:21:38.376567 19961 raft_consensus.cc:2749] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:21:38.397115 17652 leader_election.cc:304] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
W20260709 12:21:38.433837 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 94443.5us
I20260709 12:21:38.414108 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:38.415220 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:38.416194 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:38.366601 20158 consensus_queue.cc:1048] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.030s
I20260709 12:21:38.444564 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:21:38.393822 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 312216us
W20260709 12:21:38.431619 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 82103.7us
I20260709 12:21:38.451295 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:38.452178 18366 raft_consensus.cc:3060] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:38.454747 17764 raft_consensus.cc:2468] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 9.
I20260709 12:21:38.456801 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:38.457641 18371 raft_consensus.cc:3060] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:38.457721 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:38.458801 18363 raft_consensus.cc:2468] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 1.
I20260709 12:21:38.461337 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:38.462109 18363 raft_consensus.cc:3060] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:38.464383 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:38.465219 18367 raft_consensus.cc:3060] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:38.466540 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:38.348219 19826 consensus_queue.cc:237] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:38.445449 18365 raft_consensus.cc:2468] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 4.
I20260709 12:21:38.468712 18333 leader_election.cc:304] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:38.419656 17654 leader_election.cc:304] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:21:38.381968 20178 raft_consensus.cc:515] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:38.487186 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:38.489439 17687 raft_consensus.cc:3060] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:38.418236 20234 raft_consensus.cc:493] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:38.426736 20346 raft_consensus.cc:493] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:21:38.508661 18367 raft_consensus.cc:2468] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 5.
I20260709 12:21:38.411022 20350 raft_consensus.cc:2804] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260709 12:21:38.494115 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 97070us
W20260709 12:21:38.496277 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 76729.2us
I20260709 12:21:38.427157 20296 raft_consensus.cc:493] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:38.496889 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 244323us
I20260709 12:21:38.506902 18374 raft_consensus.cc:1275] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260709 12:21:38.533047 17727 leader_election.cc:304] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:21:38.491994 18371 raft_consensus.cc:2468] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 5.
I20260709 12:21:38.511471 18333 leader_election.cc:304] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:21:38.445011 20321 raft_consensus.cc:2804] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:21:38.520790 18363 raft_consensus.cc:2468] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 3.
I20260709 12:21:38.522137 18365 raft_consensus.cc:1240] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 5. Current term is 6. Ops: []
W20260709 12:21:38.492961 20330 log.cc:927] Time spent T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.238s	user 0.000s	sys 0.001s
I20260709 12:21:38.534600 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:38.542279 18371 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 4.
I20260709 12:21:38.542598 17687 raft_consensus.cc:2468] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 3.
I20260709 12:21:38.507560 19961 raft_consensus.cc:2804] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260709 12:21:38.548080 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 138862us
I20260709 12:21:38.491539 20347 raft_consensus.cc:2804] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:21:38.539685 17802 leader_election.cc:304] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:21:38.548846 17728 leader_election.cc:304] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:21:38.551304 17802 leader_election.cc:304] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:21:38.507905 20234 raft_consensus.cc:3060] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:38.483052 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:38.407836 19855 raft_consensus.cc:515] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:38.554250 19855 leader_election.cc:290] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 4 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:38.545686 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:38.550875 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:38.543633 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:38.556931 18363 raft_consensus.cc:2468] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 1.
I20260709 12:21:38.563089 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:21:38.536898 20329 log.cc:927] Time spent T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.261s	user 0.000s	sys 0.001s
I20260709 12:21:38.564013 18374 raft_consensus.cc:2468] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 4.
I20260709 12:21:38.412195 20336 raft_consensus.cc:515] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:38.400311 20334 raft_consensus.cc:515] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:38.508800 20332 raft_consensus.cc:697] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 4 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:21:38.506958 20344 raft_consensus.cc:2804] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:21:38.611332 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:38.482657 20076 consensus_queue.cc:1059] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.075s
I20260709 12:21:38.612742 17764 raft_consensus.cc:1275] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:21:38.638887 17685 raft_consensus.cc:2393] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 7: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:21:38.533748 20350 raft_consensus.cc:493] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:38.614239 18363 raft_consensus.cc:1240] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 2. Current term is 3. Ops: []
I20260709 12:21:38.613996 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:21:38.548410 20337 log.cc:927] Time spent T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.261s	user 0.000s	sys 0.001s
I20260709 12:21:38.615439 17763 raft_consensus.cc:1240] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 6. Current term is 7. Ops: [6.5-6.5]
W20260709 12:21:38.626307 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 75065.3us
W20260709 12:21:38.628142 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 99629.4us
I20260709 12:21:38.560885 20347 raft_consensus.cc:493] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:38.417853 20314 raft_consensus.cc:515] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:38.541401 20321 raft_consensus.cc:493] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:21:38.639407 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:38.665210 17762 raft_consensus.cc:3060] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:38.648548 18365 raft_consensus.cc:2468] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 6.
I20260709 12:21:38.650075 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:38.559393 19961 raft_consensus.cc:493] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:21:38.609853 20102 raft_consensus.cc:2749] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:21:38.638916 20344 raft_consensus.cc:493] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:38.667500 18367 raft_consensus.cc:2393] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 4: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:21:38.568154 20349 raft_consensus.cc:2804] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:21:38.669477 17681 raft_consensus.cc:1240] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 4. Current term is 5. Ops: [4.4-4.4]
I20260709 12:21:38.674113 19961 raft_consensus.cc:3060] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:38.546989 20176 consensus_queue.cc:1059] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.109s
I20260709 12:21:38.544191 20296 raft_consensus.cc:515] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:38.651542 17654 leader_election.cc:304] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:21:38.568763 20327 consensus_queue.cc:1048] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.101s
I20260709 12:21:38.525009 20346 raft_consensus.cc:515] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:21:38.668335 17635 auto_rebalancer.cc:893] Removing replace marker failed: Replica e063541542a94969898868bf6eb5ce1b is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 7 committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } } pending_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
W20260709 12:21:38.674129 20345 log.cc:927] Time spent T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.142s	user 0.000s	sys 0.001s
W20260709 12:21:38.683081 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 134327us
W20260709 12:21:38.643043 20330 log.cc:927] Time spent T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.081s	user 0.000s	sys 0.000s
I20260709 12:21:38.687731 17654 leader_election.cc:304] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:21:38.676602 20307 leader_election.cc:290] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:21:38.688681 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 177283us
I20260709 12:21:38.674538 20344 raft_consensus.cc:3060] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:38.705969 18333 leader_election.cc:304] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
W20260709 12:21:38.689159 17635 auto_rebalancer.cc:901] Could not move replica: Illegal state: not currently leader
I20260709 12:21:38.649461 19855 raft_consensus.cc:2764] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Leader election decision vote started in defunct term 4: won
I20260709 12:21:38.694406 17762 raft_consensus.cc:2468] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 4.
I20260709 12:21:38.694870 18366 raft_consensus.cc:2468] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 6.
I20260709 12:21:38.650255 20350 raft_consensus.cc:3060] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:38.670621 20321 raft_consensus.cc:3060] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:38.613631 20284 consensus_queue.cc:1048] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.069s
I20260709 12:21:38.665738 20347 raft_consensus.cc:3060] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:21:38.707293 17635 auto_rebalancer.cc:330] scheduled replica move failed to complete: Illegal state: not currently leader
I20260709 12:21:38.647898 20352 consensus_queue.cc:1059] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.073s
I20260709 12:21:38.732599 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:38.734465 17762 raft_consensus.cc:2468] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 6.
I20260709 12:21:38.737155 17729 leader_election.cc:304] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
W20260709 12:21:38.748329 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 60652.8us
W20260709 12:21:38.749670 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 47237.1us
I20260709 12:21:38.752364 17762 raft_consensus.cc:1275] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 6. (term mismatch)
I20260709 12:21:38.754830 17654 leader_election.cc:304] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:21:38.731062 20176 raft_consensus.cc:3060] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:38.760728 20368 raft_consensus.cc:2804] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:21:38.753250 17762 pending_rounds.cc:85] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb: Aborting all ops after (but not including) 5
I20260709 12:21:38.768412 17762 pending_rounds.cc:107] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb: Aborting uncommitted NO_OP operation due to leader change: 3.6
W20260709 12:21:38.764106 20358 log.cc:927] Time spent T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.058s	user 0.000s	sys 0.000s
W20260709 12:21:38.761533 20343 log.cc:927] Time spent T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.331s	user 0.002s	sys 0.000s
I20260709 12:21:38.768780 17762 raft_consensus.cc:2889] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:21:38.835225 17803 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:38.903223 17803 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:38.924693 20362 raft_consensus.cc:2749] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:21:38.926530 20334 leader_election.cc:290] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:38.927353 20334 raft_consensus.cc:2804] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:21:38.927821 20334 raft_consensus.cc:493] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:38.928318 20334 raft_consensus.cc:3060] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:38.929230 20346 leader_election.cc:290] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:38.931735 20375 raft_consensus.cc:2804] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:21:38.874053 20363 raft_consensus.cc:493] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:38.928020 20342 log.cc:927] Time spent T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.346s	user 0.001s	sys 0.001s
I20260709 12:21:38.732527 20332 consensus_queue.cc:237] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:21:39.053068 20356 log.cc:927] Time spent T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.094s	user 0.000s	sys 0.001s
I20260709 12:21:39.053709 20378 raft_consensus.cc:2764] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
W20260709 12:21:38.902817 19994 consensus_meta.cc:287] T 7cb3768d215e409fb9f5baf18d124442 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.647s	user 0.009s	sys 0.002s
I20260709 12:21:38.950321 20362 raft_consensus.cc:493] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:21:38.933849 20178 leader_election.cc:290] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:39.028745 17830 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35040 (request call id 1942) took 1048 ms. Trace:
I20260709 12:21:38.950290 20325 raft_consensus.cc:2804] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:21:38.933215 20368 raft_consensus.cc:697] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
W20260709 12:21:39.062780 17688 consensus_meta.cc:287] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.684s	user 0.010s	sys 0.001s
I20260709 12:21:39.108433 17688 raft_consensus.cc:2468] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 5.
I20260709 12:21:38.902400 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:21:39.055400 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 318279us
I20260709 12:21:37.717671 17833 rpcz_store.cc:276] 0709 12:21:36.571736 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:37.018327 (+446591us) service_pool.cc:224] Handling call
0709 12:21:37.018738 (+   411us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:37.018979 (+   241us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 12:21:37.288209 (+269230us) log.cc:844] Serialized 14 byte log entry
0709 12:21:37.716115 (+427906us) log.cc:844] Serialized 14 byte log entry
0709 12:21:37.716319 (+   204us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:37.716377 (+    58us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:21:37.716438 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:37.716523 (+    85us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:37.716852 (+   329us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":5416,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":33,"raft.queue_time_us":189,"raft.run_cpu_time_us":92,"raft.run_wall_time_us":93,"spinlock_wait_cycles":17190272,"thread_start_us":696032,"threads_started":2}
I20260709 12:21:38.873862 20364 raft_consensus.cc:3055] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:21:39.055117 20296 leader_election.cc:290] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:39.115864 17729 leader_election.cc:304] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:21:39.088356 17830 rpcz_store.cc:276] 0709 12:21:37.980164 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:37.982365 (+  2201us) service_pool.cc:224] Handling call
0709 12:21:37.982724 (+   359us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:37.983151 (+   427us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:37.983192 (+    41us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:37.983762 (+   570us) log.cc:844] Serialized 26 byte log entry
0709 12:21:38.271068 (+287306us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:21:38.271128 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:38.271238 (+   110us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:39.028423 (+757185us) raft_consensus.cc:1685] finished
0709 12:21:39.028460 (+    37us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:39.028658 (+   198us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":4042,"raft.run_cpu_time_us":150,"raft.run_wall_time_us":2069,"thread_start_us":231119,"threads_started":1,"wal-append.queue_time_us":404827}
I20260709 12:21:39.122908 20364 raft_consensus.cc:740] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:21:39.133016 17755 raft_consensus.cc:2410] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for term 5 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20260709 12:21:39.108022 20362 raft_consensus.cc:515] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:39.133975 20364 consensus_queue.cc:260] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:21:39.135589 20372 log.cc:927] Time spent T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.060s	user 0.000s	sys 0.000s
I20260709 12:21:39.136368 20362 leader_election.cc:290] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:39.137567 20368 consensus_queue.cc:237] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:39.140163 20357 raft_consensus.cc:493] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:39.140954 20357 raft_consensus.cc:515] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:21:39.144891 20324 consensus_meta.cc:287] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 1.021s	user 0.012s	sys 0.000s
I20260709 12:21:39.145680 20324 raft_consensus.cc:515] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:39.147336 17686 raft_consensus.cc:1240] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 3. Current term is 4. Ops: []
I20260709 12:21:39.088439 19994 raft_consensus.cc:515] T 7cb3768d215e409fb9f5baf18d124442 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:39.150867 19994 leader_election.cc:290] T 7cb3768d215e409fb9f5baf18d124442 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 4 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:39.116389 20178 raft_consensus.cc:2804] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Leader election won for term 5
W20260709 12:21:39.151397 20234 consensus_meta.cc:287] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.599s	user 0.009s	sys 0.001s
I20260709 12:21:39.152002 20178 raft_consensus.cc:697] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 5 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
W20260709 12:21:39.154526 20232 consensus_meta.cc:287] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 1.199s	user 0.005s	sys 0.005s
I20260709 12:21:39.156219 20284 raft_consensus.cc:493] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:39.158320 17729 leader_election.cc:304] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:21:39.158490 19961 raft_consensus.cc:515] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:39.159683 20361 raft_consensus.cc:2804] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:21:39.161305 20365 raft_consensus.cc:3055] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:21:39.159051 20324 leader_election.cc:290] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 4 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:39.161779 20095 leader_election.cc:290] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:39.133998 20373 raft_consensus.cc:493] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:39.163517 20227 raft_consensus.cc:3055] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:21:39.165454 20321 raft_consensus.cc:515] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:39.056106 20363 raft_consensus.cc:515] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:39.172494 20336 leader_election.cc:290] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:39.173883 20314 leader_election.cc:290] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:39.188023 20157 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:35032 (request call id 95) took 5848 ms. Trace:
I20260709 12:21:39.190383 20346 raft_consensus.cc:2764] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Leader election decision vote started in defunct term 4: won
I20260709 12:21:39.190584 19855 raft_consensus.cc:3055] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:21:39.074786 20354 raft_consensus.cc:2804] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:21:39.191642 18368 raft_consensus.cc:1240] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 5. Current term is 6. Ops: []
W20260709 12:21:39.190263 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 484331us
I20260709 12:21:39.206460 20365 raft_consensus.cc:740] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 5 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:21:39.229435 20365 consensus_queue.cc:260] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:39.230048 20363 leader_election.cc:290] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:39.205554 20284 raft_consensus.cc:515] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:21:39.192596 20186 consensus_meta.cc:287] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.919s	user 0.011s	sys 0.000s
I20260709 12:21:39.230804 20186 raft_consensus.cc:515] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:39.124792 20325 raft_consensus.cc:697] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 5 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:21:39.232113 20284 leader_election.cc:290] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:39.232990 20186 leader_election.cc:290] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 6 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:39.233104 20325 consensus_queue.cc:237] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:21:39.234676 20344 consensus_meta.cc:287] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.503s	user 0.003s	sys 0.006s
I20260709 12:21:39.235172 20344 raft_consensus.cc:515] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:21:39.207562 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 49338.9us
I20260709 12:21:39.237368 20344 leader_election.cc:290] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 5 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:21:39.231706 20018 consensus_meta.cc:287] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.811s	user 0.005s	sys 0.006s
I20260709 12:21:39.239151 18361 raft_consensus.cc:2468] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 4.
I20260709 12:21:39.239162 20321 leader_election.cc:290] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 4 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:39.215287 20373 raft_consensus.cc:515] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:39.212116 20227 raft_consensus.cc:740] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:21:39.241554 20227 consensus_queue.cc:260] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:39.242221 20373 leader_election.cc:290] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:39.244263 20232 consensus_queue.cc:1059] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
W20260709 12:21:39.197202 20196 consensus_meta.cc:287] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.907s	user 0.005s	sys 0.006s
W20260709 12:21:39.201136 20299 consensus_meta.cc:287] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.927s	user 0.013s	sys 0.001s
I20260709 12:21:39.200364 20247 leader_election.cc:290] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:21:39.191085 17727 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.580s	user 0.024s	sys 0.081s
W20260709 12:21:39.249061 17727 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.638s	user 0.024s	sys 0.081s
W20260709 12:21:39.249440 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 689151us
I20260709 12:21:39.225445 18333 leader_election.cc:304] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:21:39.251096 18333 leader_election.cc:400] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 3 election: Vote denied by peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651) with higher term. Message: Invalid argument: T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 3. Current term is 4.
W20260709 12:21:39.251163 18334 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.759s	user 0.019s	sys 0.048s
I20260709 12:21:39.226467 17680 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44898 (request call id 574) took 1211 ms. Trace:
I20260709 12:21:39.228103 19961 leader_election.cc:290] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 3 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:39.210922 20361 raft_consensus.cc:697] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 5 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:21:39.239545 20178 consensus_queue.cc:237] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:39.179415 20357 leader_election.cc:290] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:21:39.251566 18334 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.759s	user 0.019s	sys 0.048s
W20260709 12:21:39.331985 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 858342us
I20260709 12:21:39.218716 20298 leader_election.cc:290] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:39.251489 18333 leader_election.cc:403] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
I20260709 12:21:39.197396 20234 raft_consensus.cc:515] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:39.224013 19855 raft_consensus.cc:740] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:21:39.250136 17727 leader_election.cc:304] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:21:39.238193 20018 raft_consensus.cc:515] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:21:39.345263 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 95213.7us
I20260709 12:21:39.346071 17727 leader_election.cc:304] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:21:39.277925 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:39.347254 17837 raft_consensus.cc:2468] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 4.
W20260709 12:21:39.270089 18374 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.519s	user 0.018s	sys 0.029s
W20260709 12:21:39.349078 18374 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.598s	user 0.019s	sys 0.030s
I20260709 12:21:39.251377 17729 leader_election.cc:304] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:21:39.242739 20227 raft_consensus.cc:3060] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:39.223785 20354 raft_consensus.cc:493] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:39.227504 20334 raft_consensus.cc:515] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:21:39.276944 20347 consensus_meta.cc:287] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.544s	user 0.010s	sys 0.000s
I20260709 12:21:39.219679 20336 raft_consensus.cc:493] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
W20260709 12:21:39.251121 20349 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.523s	user 0.016s	sys 0.051s
I20260709 12:21:39.274906 20321 raft_consensus.cc:2804] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:21:39.355155 20370 raft_consensus.cc:3055] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:21:39.242763 20373 consensus_queue.cc:1059] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.109s
I20260709 12:21:39.355562 20376 raft_consensus.cc:2804] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:21:39.275540 20299 raft_consensus.cc:515] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:39.356007 20383 consensus_queue.cc:1059] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.112s
I20260709 12:21:39.356410 20234 leader_election.cc:290] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 5 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:39.356791 20374 raft_consensus.cc:493] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:39.357456 20127 leader_election.cc:290] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:39.356950 19855 consensus_queue.cc:260] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.9, Last appended by leader: 9, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:39.357874 20019 raft_consensus.cc:2749] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651) with higher term. Message: Invalid argument: T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 3. Current term is 4.
I20260709 12:21:39.352859 20354 raft_consensus.cc:3060] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:39.361936 20334 leader_election.cc:290] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 3 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:21:39.365190 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 113879us
W20260709 12:21:39.365782 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 114811us
I20260709 12:21:39.472877 18333 leader_election.cc:304] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:39.372813 17727 leader_election.cc:304] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
W20260709 12:21:39.474340 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 101635us
I20260709 12:21:39.475349 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:39.476003 18371 raft_consensus.cc:1813] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Handling vote request from an unknown peer e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:39.476387 18371 raft_consensus.cc:3060] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:21:39.348773 20176 consensus_meta.cc:287] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.580s	user 0.008s	sys 0.004s
I20260709 12:21:39.478384 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:39.479161 18365 raft_consensus.cc:3060] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:39.387321 17686 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37402 (request call id 2239) took 1074 ms. Trace:
W20260709 12:21:39.375559 20349 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.649s	user 0.022s	sys 0.073s
I20260709 12:21:39.253657 20196 raft_consensus.cc:515] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:39.252770 17680 rpcz_store.cc:276] 0709 12:21:38.014615 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:38.014907 (+   292us) service_pool.cc:224] Handling call
0709 12:21:39.226260 (+1211353us) spinlock_profiling.cc:243] Waited 206 ms on lock 0x7b54004277d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:21:39.226385 (+   125us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":473794304}
I20260709 12:21:39.373764 20357 raft_consensus.cc:2804] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:21:39.388047 20298 raft_consensus.cc:2804] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:21:39.374482 20347 raft_consensus.cc:515] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:39.391083 20321 raft_consensus.cc:493] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:39.390117 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:39.462267 17683 raft_consensus.cc:2393] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 5: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:21:39.393661 20018 leader_election.cc:290] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 6 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:39.352666 20346 consensus_queue.cc:1048] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.074s
I20260709 12:21:39.391363 20370 raft_consensus.cc:740] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:21:39.479770 17686 rpcz_store.cc:276] 0709 12:21:38.313218 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:38.313352 (+   134us) service_pool.cc:224] Handling call
0709 12:21:38.328300 (+ 14948us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:39.147994 (+819694us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:39.386944 (+238950us) spinlock_profiling.cc:243] Waited 819 ms on lock 0x7b54004254d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:21:39.387234 (+   290us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1883012864}
I20260709 12:21:39.442830 20179 raft_consensus.cc:2749] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:21:39.388350 20336 raft_consensus.cc:515] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:39.463645 20373 raft_consensus.cc:3055] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:21:39.499300 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:39.459780 20387 raft_consensus.cc:2749] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:21:39.496737 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 8 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:39.462906 20376 raft_consensus.cc:697] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:21:39.500181 17801 leader_election.cc:304] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:21:39.509924 18371 raft_consensus.cc:2410] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for term 4 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
W20260709 12:21:39.495101 20350 consensus_meta.cc:287] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.764s	user 0.011s	sys 0.000s
W20260709 12:21:39.514562 17652 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.955s	user 0.008s	sys 0.016s
I20260709 12:21:39.517148 18333 leader_election.cc:304] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:39.489709 20383 raft_consensus.cc:2804] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:21:39.494199 20298 raft_consensus.cc:493] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:39.495811 20321 raft_consensus.cc:3060] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:39.488615 20357 raft_consensus.cc:493] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:39.486200 20349 raft_consensus.cc:697] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:21:39.494846 20176 raft_consensus.cc:2749] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:21:39.496243 19855 raft_consensus.cc:3060] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:21:39.529412 18374 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 7.
W20260709 12:21:39.532728 17803 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:39.560597 17803 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:39.364317 20361 consensus_queue.cc:237] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:39.508383 20019 raft_consensus.cc:2749] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20260709 12:21:39.535332 17652 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.975s	user 0.008s	sys 0.016s
I20260709 12:21:39.464355 20374 raft_consensus.cc:515] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:21:39.537129 17654 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.771s	user 0.000s	sys 0.028s
I20260709 12:21:39.491555 19994 consensus_queue.cc:1059] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.039s
I20260709 12:21:39.518625 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:21:39.562543 17652 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:39.536878 20383 raft_consensus.cc:493] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:39.539273 20298 raft_consensus.cc:3060] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:21:39.529474 19806 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.844s	user 0.005s	sys 0.011s
W20260709 12:21:39.565801 17654 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.800s	user 0.000s	sys 0.029s
W20260709 12:21:39.574255 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 819447us
I20260709 12:21:39.575124 17654 leader_election.cc:304] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:21:39.576151 17654 leader_election.cc:304] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:21:39.541553 20357 raft_consensus.cc:3060] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:39.218405 20157 rpcz_store.cc:276] 0709 12:21:33.276958 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:33.459513 (+182555us) service_pool.cc:224] Handling call
0709 12:21:34.345935 (+886422us) spinlock_profiling.cc:243] Waited 32.7 ms on lock 0x7b4c0041cbc0. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b488eca20 00007f7b488fda29 00007f7b488c3644 00007f7b488eaf2d 00007f7b488eab93 00007f7b4a5791af 00007f7b4a578f3e 00007f7b4a5fe888 00007f7b4a61b75c 00007f7b4a61c76a 00007f7b4a61b91b
0709 12:21:34.347353 (+  1418us) log.cc:844] Serialized 494 byte log entry
0709 12:21:34.715812 (+368459us) spinlock_profiling.cc:243] Waited 2.05 ms on lock 0x7b180035dd00. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b475333f3 00007f7b475357c1 00007f7b47535779 00007f7b47535709 00007f7b475356d1 00007f7b475349cd 0000000000388634 0000000000388479 00007f7b4751683a
0709 12:21:39.108590 (+4392778us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":13880,"raft.run_cpu_time_us":1818,"raft.run_wall_time_us":91991,"spinlock_wait_cycles":79891712,"thread_start_us":280651,"threads_started":1,"wal-append.queue_time_us":536100,"wal-append.run_cpu_time_us":2386,"wal-append.run_wall_time_us":1028489}
W20260709 12:21:39.578157 17653 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.768s	user 0.003s	sys 0.016s
W20260709 12:21:39.578506 17653 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.811s	user 0.003s	sys 0.016s
W20260709 12:21:39.563037 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 45979.4us
I20260709 12:21:39.579982 18333 leader_election.cc:304] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:21:39.570019 17652 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:39.581525 20186 raft_consensus.cc:2804] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:21:39.582065 20186 raft_consensus.cc:697] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 3 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:21:39.582787 17652 leader_election.cc:304] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:21:39.582986 20186 consensus_queue.cc:237] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:39.584033 20284 consensus_queue.cc:1059] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 12:21:39.584872 20284 consensus_queue.cc:1048] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:21:39.586128 20284 consensus_queue.cc:1048] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:21:39.586395 20344 raft_consensus.cc:2804] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:21:39.586796 20344 raft_consensus.cc:493] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:39.587121 20344 raft_consensus.cc:3060] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:39.587432 20284 raft_consensus.cc:3055] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:21:39.525599 20373 raft_consensus.cc:740] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:21:39.564564 20387 raft_consensus.cc:493] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:39.595394 18333 leader_election.cc:304] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:39.598484 17654 leader_election.cc:304] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
W20260709 12:21:39.578305 19806 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.898s	user 0.006s	sys 0.011s
I20260709 12:21:39.568315 20379 raft_consensus.cc:2749] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:21:39.539629 20350 raft_consensus.cc:515] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:21:39.578213 20375 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.503s	user 0.016s	sys 0.026s
I20260709 12:21:39.574012 20383 raft_consensus.cc:3060] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:39.599995 17652 leader_election.cc:304] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:39.575977 20381 raft_consensus.cc:493] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:39.591084 20127 raft_consensus.cc:2804] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:21:39.594823 20346 raft_consensus.cc:3060] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:39.626348 17652 leader_election.cc:304] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:21:39.630311 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 152774us
W20260709 12:21:39.591707 17832 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37994 (request call id 613) took 1666 ms (client timeout 1680 ms). Trace:
I20260709 12:21:39.587786 20284 raft_consensus.cc:740] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
W20260709 12:21:39.608974 20375 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.581s	user 0.016s	sys 0.026s
I20260709 12:21:39.599567 20384 raft_consensus.cc:493] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:21:39.601804 20314 consensus_queue.cc:1048] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.060s
I20260709 12:21:39.601651 20334 consensus_queue.cc:1059] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.064s
I20260709 12:21:39.676064 17571 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 7cb3768d215e409fb9f5baf18d124442 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
W20260709 12:21:39.684010 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 85561.7us
W20260709 12:21:39.669523 17802 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:39.690199 17802 leader_election.cc:304] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:21:39.665262 17652 leader_election.cc:304] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:21:39.663268 17688 raft_consensus.cc:1240] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20260709 12:21:39.606781 20019 raft_consensus.cc:2749] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260709 12:21:39.676270 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 80947.1us
I20260709 12:21:39.694343 18362 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59886 (request call id 807) took 1059 ms. Trace:
W20260709 12:21:39.680474 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 180383us
I20260709 12:21:39.696457 17801 leader_election.cc:304] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:39.696964 18333 leader_election.cc:304] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:21:39.668567 20386 log.cc:927] Time spent T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.133s	user 0.001s	sys 0.000s
I20260709 12:21:39.639102 19806 raft_consensus.cc:2749] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260709 12:21:39.702805 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 361510us
I20260709 12:21:39.679953 20196 leader_election.cc:290] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 5 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:39.590451 20370 consensus_queue.cc:260] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:39.669080 20127 raft_consensus.cc:697] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 5 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:21:39.668954 20299 leader_election.cc:290] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 5 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:39.665330 20398 raft_consensus.cc:493] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:21:39.668398 20394 raft_consensus.cc:2804] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:21:39.601877 20387 raft_consensus.cc:515] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:39.721138 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:39.679356 20375 raft_consensus.cc:697] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 3 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:21:39.723464 20390 raft_consensus.cc:493] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:39.724042 20388 log.cc:927] Time spent T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.178s	user 0.001s	sys 0.000s
I20260709 12:21:39.594805 20349 consensus_queue.cc:237] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:21:39.716598 18334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:39.699272 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:39.725207 17837 raft_consensus.cc:2468] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 6.
I20260709 12:21:39.703614 20327 raft_consensus.cc:2804] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:21:39.709265 20264 raft_consensus.cc:2804] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260709 12:21:39.651023 17832 rpcz_store.cc:269] 0709 12:21:37.928117 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:39.491741 (+1563624us) service_pool.cc:224] Handling call
0709 12:21:39.578648 (+ 86907us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:39.758908 20394 raft_consensus.cc:697] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 6 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:21:39.723325 20295 raft_consensus.cc:2804] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:21:39.723829 17829 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f"
dest_uuid: "e063541542a94969898868bf6eb5ce1b"
mode: GRACEFUL
new_leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
 from {username='slave'} at 127.0.0.1:59820
I20260709 12:21:39.724341 20377 raft_consensus.cc:2804] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:21:39.639336 20381 raft_consensus.cc:515] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:39.762821 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:39.764320 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:39.601780 20376 consensus_queue.cc:237] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 7.9, Last appended by leader: 9, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:39.712237 20347 leader_election.cc:290] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 7 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:39.723686 20374 leader_election.cc:290] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:39.773105 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:39.874737 18365 raft_consensus.cc:2468] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 6.
I20260709 12:21:39.888888 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
W20260709 12:21:39.823716 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 123449us
I20260709 12:21:39.733572 20370 raft_consensus.cc:3060] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:39.890172 17831 raft_consensus.cc:2468] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 5.
I20260709 12:21:39.719216 20019 consensus_queue.cc:1048] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.140s
I20260709 12:21:39.732750 20385 raft_consensus.cc:493] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:21:39.721390 20325 consensus_queue.cc:1048] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.086s
I20260709 12:21:39.688551 20384 raft_consensus.cc:515] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:39.739385 20390 raft_consensus.cc:515] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:39.734992 20398 raft_consensus.cc:515] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:39.891337 18333 leader_election.cc:304] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:39.695067 18362 rpcz_store.cc:276] 0709 12:21:38.635074 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:38.668776 (+ 33702us) service_pool.cc:224] Handling call
0709 12:21:38.684641 (+ 15865us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:38.685013 (+   372us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 12:21:38.715628 (+ 30615us) log.cc:844] Serialized 14 byte log entry
0709 12:21:39.165981 (+450353us) spinlock_profiling.cc:243] Waited 11.3 ms on lock 0x7b18002bde60. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b47515e07 00007f7b503709a9 00007f7b4753269b 00007f7b4753093f 00007f7b47532c02 00007f7b4a1ba20b 00007f7b4a1bfd23 00007f7b4a1c023a 00007f7b4a62aa2a
0709 12:21:39.682470 (+516489us) log.cc:844] Serialized 14 byte log entry
0709 12:21:39.694012 (+ 11542us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:39.694066 (+    54us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:21:39.694115 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:39.694175 (+    60us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:39.694290 (+   115us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":22993,"prepare.queue_time_us":41043,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":44,"raft.queue_time_us":12662,"raft.run_cpu_time_us":352,"raft.run_wall_time_us":17020,"spinlock_wait_cycles":26018688,"thread_start_us":514464,"threads_started":2,"wal-append.queue_time_us":955948}
W20260709 12:21:40.298957 17802 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:39.940851 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 240829us
I20260709 12:21:40.223594 17729 leader_election.cc:304] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:21:40.185209 17683 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44898 (request call id 573) took 2170 ms (client timeout 1923 ms). Trace:
I20260709 12:21:40.228178 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:21:40.059926 17572 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet d522f70beb1f417cb98940d90b594823 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 0da169ff82674dc1947055ce43de68a3 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 4 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:21:40.303174 17689 raft_consensus.cc:2468] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 5.
I20260709 12:21:40.064091 20374 raft_consensus.cc:493] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:21:40.303779 20352 raft_consensus.cc:2749] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:21:40.303804 20234 raft_consensus.cc:2804] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:21:40.171864 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:40.084064 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:21:40.303440 17572 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 581959us
I20260709 12:21:39.727149 20018 raft_consensus.cc:493] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:21:39.775202 20368 raft_consensus.cc:493] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:40.061487 20393 log.cc:927] Time spent T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.168s	user 0.001s	sys 0.000s
I20260709 12:21:40.332815 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:39.961107 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:40.178478 18368 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 939) took 3054 ms. Trace:
W20260709 12:21:40.124878 20344 consensus_meta.cc:287] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.536s	user 0.007s	sys 0.005s
I20260709 12:21:40.360857 18368 rpcz_store.cc:276] 0709 12:21:37.123648 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:37.877837 (+754189us) service_pool.cc:224] Handling call
0709 12:21:37.899491 (+ 21654us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:39.222923 (+1323432us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:40.178052 (+955129us) spinlock_profiling.cc:243] Waited 1.29 s on lock 0x7b5400802d54. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:21:40.178358 (+   306us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2971192704}
W20260709 12:21:40.306443 17801 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.593s	user 0.003s	sys 0.008s
I20260709 12:21:40.103323 20365 raft_consensus.cc:3060] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:40.365569 17802 leader_election.cc:304] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:21:39.888157 17829 raft_consensus.cc:606] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Received request to transfer leadership to TS e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:40.336442 17838 raft_consensus.cc:2468] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 5.
I20260709 12:21:40.273919 20422 raft_consensus.cc:493] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:40.095106 20410 raft_consensus.cc:2749] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 12:21:40.227762 17688 raft_consensus.cc:1240] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 3. Current term is 4. Ops: []
W20260709 12:21:40.006151 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:40.302518 17683 rpcz_store.cc:269] 0709 12:21:38.014315 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:38.018370 (+  4055us) service_pool.cc:224] Handling call
0709 12:21:40.184968 (+2166598us) spinlock_profiling.cc:243] Waited 1.44 s on lock 0x7b5400425ed4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:21:40.185105 (+   137us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3303356288}
W20260709 12:21:40.134994 18361 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59918 (request call id 943) took 1723 ms (client timeout 1862 ms). Trace:
W20260709 12:21:40.359028 20298 consensus_meta.cc:287] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.769s	user 0.007s	sys 0.005s
I20260709 12:21:40.096517 20381 leader_election.cc:290] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:40.304157 20374 raft_consensus.cc:515] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:40.302486 17652 leader_election.cc:304] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:21:40.302476 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 78982.8us
I20260709 12:21:40.101438 20324 raft_consensus.cc:3055] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:21:40.164196 20273 raft_consensus.cc:3055] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 2 LEADER]: Stepping down as leader of term 2
W20260709 12:21:40.184523 20321 consensus_meta.cc:287] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.615s	user 0.011s	sys 0.004s
W20260709 12:21:40.298964 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 407681us
I20260709 12:21:40.225982 20144 raft_consensus.cc:493] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:40.081261 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:40.062438 20394 consensus_queue.cc:237] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:40.235050 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:21:40.098366 20227 consensus_meta.cc:287] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.739s	user 0.010s	sys 0.000s
I20260709 12:21:39.770223 20264 raft_consensus.cc:493] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:40.262446 20373 consensus_queue.cc:260] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:21:40.202343 20383 consensus_meta.cc:287] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.552s	user 0.009s	sys 0.001s
W20260709 12:21:40.214566 20413 log.cc:927] Time spent T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.050s	user 0.001s	sys 0.000s
W20260709 12:21:40.271126 20354 consensus_meta.cc:287] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.774s	user 0.007s	sys 0.003s
W20260709 12:21:40.094806 20389 log.cc:927] Time spent T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.477s	user 0.001s	sys 0.000s
W20260709 12:21:40.172180 20346 consensus_meta.cc:287] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.521s	user 0.005s	sys 0.005s
W20260709 12:21:40.234288 19855 consensus_meta.cc:287] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.669s	user 0.002s	sys 0.007s
I20260709 12:21:39.765187 20102 raft_consensus.cc:493] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:21:40.460156 20102 raft_consensus.cc:515] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:39.874529 20379 raft_consensus.cc:493] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:40.462401 20379 raft_consensus.cc:515] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:40.464536 20379 leader_election.cc:290] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:21:39.977561 20392 log.cc:927] Time spent T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.252s	user 0.000s	sys 0.001s
I20260709 12:21:40.360731 17837 raft_consensus.cc:3060] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:21:40.362852 17801 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.649s	user 0.005s	sys 0.008s
I20260709 12:21:39.976722 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:40.226820 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:40.048918 20157 raft_consensus.cc:493] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:40.273518 20357 consensus_meta.cc:287] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.677s	user 0.009s	sys 0.004s
I20260709 12:21:40.360720 17836 raft_consensus.cc:2468] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 3.
I20260709 12:21:40.228461 20364 raft_consensus.cc:3060] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:40.004616 19826 raft_consensus.cc:493] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:39.888576 20299 raft_consensus.cc:493] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:40.222502 20336 leader_election.cc:290] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:21:40.469835 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 773399us
I20260709 12:21:40.049321 20418 raft_consensus.cc:493] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:40.472380 20357 raft_consensus.cc:515] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:40.472518 20418 raft_consensus.cc:515] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:40.473795 20157 raft_consensus.cc:515] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:40.035194 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:40.474841 18367 raft_consensus.cc:1240] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 5. Current term is 6. Ops: []
I20260709 12:21:40.475415 17832 raft_consensus.cc:2393] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 7: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:21:40.475252 20299 raft_consensus.cc:515] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:39.874923 20377 raft_consensus.cc:493] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:40.476377 20377 raft_consensus.cc:3060] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:40.446858 20354 raft_consensus.cc:515] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:40.478458 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:40.478942 20354 leader_election.cc:290] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 3 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:40.180938 17835 raft_consensus.cc:1275] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260709 12:21:40.059379 20401 raft_consensus.cc:2804] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:21:40.184274 20332 raft_consensus.cc:3055] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:21:39.777541 20327 raft_consensus.cc:697] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 9 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:21:39.777729 17680 raft_consensus.cc:2468] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 5.
I20260709 12:21:40.186774 20378 raft_consensus.cc:493] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:21:40.334604 18364 raft_consensus.cc:2468] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 5.
I20260709 12:21:40.244166 20387 leader_election.cc:290] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:40.081935 20350 leader_election.cc:290] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 3 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:40.304847 20234 raft_consensus.cc:697] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 6 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:21:39.977523 20295 raft_consensus.cc:697] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 5 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:21:40.524000 20357 leader_election.cc:290] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 5 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:40.238873 17584 catalog_manager.cc:5697] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:40.344763 20430 raft_consensus.cc:493] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:39.865105 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 165438us
I20260709 12:21:40.059552 20405 raft_consensus.cc:493] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:21:40.372741 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
W20260709 12:21:40.374822 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 72441.9us
I20260709 12:21:40.378408 18371 raft_consensus.cc:1240] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 4. Current term is 5. Ops: [4.6-4.6]
W20260709 12:21:40.387359 17830 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.657s	user 0.014s	sys 0.013s
I20260709 12:21:40.367506 17829 raft_consensus.cc:614] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:21:40.403301 17727 leader_election.cc:304] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:21:40.405277 18362 raft_consensus.cc:1240] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 4. Current term is 5. Ops: []
I20260709 12:21:40.405771 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 8 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:40.059096 20407 consensus_queue.cc:1059] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.407s
I20260709 12:21:40.372213 20296 raft_consensus.cc:2804] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:21:40.358613 20018 raft_consensus.cc:515] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:40.410660 20410 consensus_queue.cc:1059] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.006s
I20260709 12:21:39.860836 20284 consensus_queue.cc:260] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:21:40.213614 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 481022us
I20260709 12:21:40.388032 19806 raft_consensus.cc:2804] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:21:40.363452 20368 raft_consensus.cc:515] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:40.424293 20264 raft_consensus.cc:3060] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:21:40.424633 20431 log.cc:927] Time spent T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.076s	user 0.001s	sys 0.000s
W20260709 12:21:40.059653 20396 consensus_queue.cc:1175] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "334a4e1f59944544b3b3ff36616b57da" responder_term: 3 status { last_received { term: 2 index: 5 } last_committed_idx: 4 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 2. Current term is 3. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260709 12:21:40.433475 20390 leader_election.cc:290] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:40.367396 20344 raft_consensus.cc:515] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:40.397861 20352 raft_consensus.cc:493] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:40.399657 20324 raft_consensus.cc:740] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:21:40.246642 20168 consensus_queue.cc:260] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:40.441010 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:40.416333 20321 raft_consensus.cc:515] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:40.434157 20383 raft_consensus.cc:515] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:40.349143 20385 raft_consensus.cc:515] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:40.044045 20127 consensus_queue.cc:237] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:40.371232 20422 raft_consensus.cc:515] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:40.446341 17833 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35040 (request call id 1949) took 1183 ms. Trace:
I20260709 12:21:40.399734 20273 raft_consensus.cc:740] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
W20260709 12:21:40.447374 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 68593.9us
I20260709 12:21:40.411652 20406 raft_consensus.cc:493] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:40.452261 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:40.435009 20144 raft_consensus.cc:515] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:21:40.371297 18361 rpcz_store.cc:269] 0709 12:21:38.411707 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:38.417558 (+  5851us) service_pool.cc:224] Handling call
0709 12:21:40.134788 (+1717230us) spinlock_profiling.cc:243] Waited 687 ms on lock 0x7b54006b3854. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:21:40.134925 (+   137us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1579147520}
I20260709 12:21:40.459278 20398 leader_election.cc:290] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:40.479228 18367 raft_consensus.cc:2393] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 4: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:21:40.016600 17681 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37402 (request call id 2237) took 1704 ms. Trace:
I20260709 12:21:40.480131 20360 raft_consensus.cc:493] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:40.481783 18374 raft_consensus.cc:1275] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:21:40.469717 20346 raft_consensus.cc:2749] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:21:40.035873 20375 consensus_queue.cc:237] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:40.483716 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:40.472002 19826 raft_consensus.cc:515] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:40.379566 20298 raft_consensus.cc:515] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:40.487936 20374 leader_election.cc:290] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:40.491659 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:40.504664 17837 raft_consensus.cc:2468] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 7.
I20260709 12:21:40.481163 20384 leader_election.cc:290] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:40.513314 20019 raft_consensus.cc:493] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:21:40.520110 20378 raft_consensus.cc:515] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:40.522733 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:40.522881 20299 leader_election.cc:290] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:40.881484 17835 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35042 (request call id 1728) took 1146 ms. Trace:
W20260709 12:21:40.864290 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 392295us
I20260709 12:21:40.891345 17838 raft_consensus.cc:2410] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for term 8 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20260709 12:21:40.891500 17831 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35042 (request call id 1724) took 1157 ms. Trace:
W20260709 12:21:40.885471 17830 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.160s	user 0.020s	sys 0.022s
W20260709 12:21:40.893011 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 489798us
I20260709 12:21:40.893916 17727 leader_election.cc:304] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:21:40.885589 18364 raft_consensus.cc:3060] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:40.525422 17802 leader_election.cc:304] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:21:40.889017 17833 rpcz_store.cc:276] 0709 12:21:39.262626 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:39.277290 (+ 14664us) service_pool.cc:224] Handling call
0709 12:21:39.337883 (+ 60593us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:40.254240 (+916357us) spinlock_profiling.cc:243] Waited 15.2 ms on lock 0x7b1800210c40. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b4a1c84bc 00007f7b4751648b 00007f7b503709a9 00007f7b4753269b 00007f7b4753093f 00007f7b47530455 00007f7b4a618060 00007f7b4a618bfb 00007f7b4a62364b
0709 12:21:40.254452 (+   212us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:40.254491 (+    39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:40.255118 (+   627us) log.cc:844] Serialized 26 byte log entry
0709 12:21:40.365582 (+110464us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:21:40.365619 (+    37us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:40.365684 (+    65us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:40.446057 (+ 80373us) raft_consensus.cc:1685] finished
0709 12:21:40.446075 (+    18us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:40.446283 (+   208us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":15314,"raft.queue_time_us":365740,"raft.run_cpu_time_us":197,"raft.run_wall_time_us":25122,"spinlock_wait_cycles":34910464,"thread_start_us":507886,"threads_started":2,"wal-append.queue_time_us":189218}
I20260709 12:21:40.885573 17837 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35042 (request call id 1730) took 1145 ms. Trace:
I20260709 12:21:40.897370 17837 rpcz_store.cc:276] 0709 12:21:39.739782 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:39.891274 (+151492us) service_pool.cc:224] Handling call
0709 12:21:40.885509 (+994235us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:40.899335 17801 leader_election.cc:304] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
W20260709 12:21:40.899750 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:40.513124 20332 raft_consensus.cc:740] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:21:40.888363 17836 raft_consensus.cc:2468] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 5.
I20260709 12:21:40.911851 17833 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35042 (request call id 1729) took 1173 ms. Trace:
I20260709 12:21:40.905402 17727 leader_election.cc:304] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:21:40.912283 17833 rpcz_store.cc:276] 0709 12:21:39.738705 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:40.911093 (+1172388us) service_pool.cc:224] Handling call
0709 12:21:40.911382 (+   289us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:40.911543 (+   161us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:40.911567 (+    24us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:40.911609 (+    42us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:21:40.911640 (+    31us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:40.911691 (+    51us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:40.911798 (+   107us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:40.885972 17681 rpcz_store.cc:276] 0709 12:21:38.312457 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:38.318205 (+  5748us) service_pool.cc:224] Handling call
0709 12:21:38.368251 (+ 50046us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:38.678703 (+310452us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:40.016230 (+1337527us) spinlock_profiling.cc:243] Waited 301 ms on lock 0x7b54001388d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:21:40.016507 (+   277us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":692167552}
I20260709 12:21:40.915661 17838 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35042 (request call id 1723) took 1181 ms. Trace:
I20260709 12:21:40.903231 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:40.916579 18367 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 7.
W20260709 12:21:40.861022 20370 consensus_meta.cc:287] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.529s	user 0.008s	sys 0.000s
W20260709 12:21:40.908648 17654 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:40.891955 17831 rpcz_store.cc:276] 0709 12:21:39.734258 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:40.367207 (+632949us) service_pool.cc:224] Handling call
0709 12:21:40.891440 (+524233us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:40.910257 17758 raft_consensus.cc:1275] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260709 12:21:40.887420 17835 rpcz_store.cc:276] 0709 12:21:39.735255 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:40.496707 (+761452us) service_pool.cc:224] Handling call
0709 12:21:40.520602 (+ 23895us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:40.520803 (+   201us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:40.520829 (+    26us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:40.520871 (+    42us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:21:40.520914 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:40.520968 (+    54us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:40.881386 (+360418us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:40.914414 17584 catalog_manager.cc:5697] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 reported cstate change: term changed from 1 to 4, leader changed from e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66) to 0da169ff82674dc1947055ce43de68a3 (127.16.29.65). New cstate: current_term: 4 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:40.933126 18365 raft_consensus.cc:3060] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:40.933652 17584 catalog_manager.cc:5697] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 reported cstate change: term changed from 4 to 6, leader changed from e063541542a94969898868bf6eb5ce1b (127.16.29.67) to 0da169ff82674dc1947055ce43de68a3 (127.16.29.65). New cstate: current_term: 6 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:40.916114 17838 rpcz_store.cc:276] 0709 12:21:39.734019 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:40.372654 (+638635us) service_pool.cc:224] Handling call
0709 12:21:40.915587 (+542933us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:40.919443 17836 raft_consensus.cc:1240] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 5. Current term is 6. Ops: []
I20260709 12:21:40.919930 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:40.935819 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:40.920248 17684 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44898 (request call id 566) took 2964 ms. Trace:
W20260709 12:21:40.903589 17803 connection.cc:256] RPC call timeout handler was delayed by -1.1435829869997178s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1797s for a response
I20260709 12:21:40.946851 17685 raft_consensus.cc:2410] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 4 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260709 12:21:40.936178 17760 raft_consensus.cc:2393] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 3: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
W20260709 12:21:40.954289 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 65473.9us
I20260709 12:21:40.861738 20373 raft_consensus.cc:3060] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:40.970955 17729 leader_election.cc:304] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:21:40.954900 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 49593.5us
I20260709 12:21:40.944082 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:40.972669 17837 raft_consensus.cc:1275] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
W20260709 12:21:40.946058 18334 leader_election.cc:336] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Timed out: RequestConsensusVote RPC to 127.16.29.65:33071 timed out after 1.605s (SENT)
I20260709 12:21:40.860198 20354 raft_consensus.cc:493] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:40.861673 20397 raft_consensus.cc:493] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:40.973479 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:40.974004 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:40.974355 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:40.973999 17833 raft_consensus.cc:1240] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 2. Current term is 3. Ops: [2.5-2.5]
I20260709 12:21:40.974921 18368 raft_consensus.cc:1240] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 4. Current term is 5. Ops: []
I20260709 12:21:40.889958 20364 raft_consensus.cc:493] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:40.981897 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:40.973394 17584 catalog_manager.cc:5697] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 reported cstate change: term changed from 1 to 3, leader changed from e063541542a94969898868bf6eb5ce1b (127.16.29.67) to e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66). New cstate: current_term: 3 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } }
I20260709 12:21:40.890789 20296 raft_consensus.cc:493] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:40.970419 17684 rpcz_store.cc:276] 0709 12:21:37.937700 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:37.937812 (+   112us) service_pool.cc:224] Handling call
0709 12:21:37.978475 (+ 40663us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:38.000022 (+ 21547us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:38.000069 (+    47us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:39.071769 (+1071700us) spinlock_profiling.cc:243] Waited 7.44 ms on lock 0x7b2c00000c88. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b48f19a11 00007f7b48f29372 00007f7b4a65ba01 00007f7b4a62043a 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2
0709 12:21:39.090062 (+ 18293us) log.cc:844] Serialized 26 byte log entry
0709 12:21:39.424816 (+334754us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:21:39.424874 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:39.424983 (+   109us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:40.862511 (+1437528us) raft_consensus.cc:1685] finished
0709 12:21:40.862537 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:40.901724 (+ 39187us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":41030,"raft.queue_time_us":23796,"raft.run_cpu_time_us":119,"raft.run_wall_time_us":120,"spinlock_wait_cycles":17113856,"thread_start_us":298015,"threads_started":1,"wal-append.queue_time_us":439187}
W20260709 12:21:40.987789 18333 connection.cc:256] RPC call timeout handler was delayed by -2.3362755540001672s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1797s for a response
W20260709 12:21:40.987720 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 462357us
W20260709 12:21:41.008927 17802 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:40.998430 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:41.003304 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:41.010593 17836 raft_consensus.cc:2468] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 5.
W20260709 12:21:40.988031 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 88783.4us
I20260709 12:21:40.521466 20401 raft_consensus.cc:697] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 5 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:21:40.998430 18363 raft_consensus.cc:2468] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 7.
I20260709 12:21:40.609483 20336 raft_consensus.cc:493] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:40.921685 20350 raft_consensus.cc:2804] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:21:40.997407 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:40.943356 20168 raft_consensus.cc:3060] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:41.019415 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:40.930006 20376 raft_consensus.cc:2804] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:21:41.006369 17584 catalog_manager.cc:5697] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 reported cstate change: term changed from 3 to 5, leader changed from 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) to 0da169ff82674dc1947055ce43de68a3 (127.16.29.65). New cstate: current_term: 5 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:40.995045 20381 raft_consensus.cc:3055] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:21:41.014528 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:41.013365 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
W20260709 12:21:41.006705 18333 connection.cc:256] RPC call timeout handler was delayed by -2.1182492040002217s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19910000000000003s for a response
I20260709 12:21:40.943780 20102 leader_election.cc:290] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:40.943799 17829 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:59820 (request call id 22) took 1223 ms. Trace:
I20260709 12:21:40.919135 20019 raft_consensus.cc:515] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:41.003767 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:40.986105 20370 raft_consensus.cc:493] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:41.020931 20410 raft_consensus.cc:2804] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:21:41.049465 17584 catalog_manager.cc:5697] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:21:41.012384 17801 leader_election.cc:304] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:40.902376 20352 raft_consensus.cc:515] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:40.990477 20232 raft_consensus.cc:2804] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:21:41.004436 20427 raft_consensus.cc:493] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:21:41.058578 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:41.148782 18363 raft_consensus.cc:2393] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 5: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:21:40.524518 20295 consensus_queue.cc:237] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:40.981472 20349 consensus_queue.cc:1059] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.079s
I20260709 12:21:41.004837 20346 raft_consensus.cc:493] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:40.523774 20234 consensus_queue.cc:237] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:40.904003 20360 raft_consensus.cc:515] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:41.009310 20147 raft_consensus.cc:493] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:41.087610 20018 leader_election.cc:290] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:41.089293 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:40.988945 20398 raft_consensus.cc:3055] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:21:41.102833 18365 raft_consensus.cc:2468] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 3.
I20260709 12:21:41.017702 20296 raft_consensus.cc:3060] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:41.106149 17838 raft_consensus.cc:1275] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260709 12:21:40.524564 20327 consensus_queue.cc:237] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:40.942691 19806 raft_consensus.cc:697] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 4 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:21:41.140228 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:41.140353 17584 catalog_manager.cc:5697] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:41.211030 17758 raft_consensus.cc:2410] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for term 7 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260709 12:21:41.089296 17829 rpcz_store.cc:276] 0709 12:21:39.689716 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:39.692828 (+  3112us) service_pool.cc:224] Handling call
0709 12:21:40.913374 (+1220546us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:41.148824 20370 raft_consensus.cc:515] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:41.216702 18374 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59886 (request call id 812) took 1616 ms. Trace:
I20260709 12:21:41.217077 18374 rpcz_store.cc:276] 0709 12:21:39.600446 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:39.600626 (+   180us) service_pool.cc:224] Handling call
0709 12:21:40.178359 (+577733us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:40.888812 (+710453us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:40.911612 (+ 22800us) spinlock_profiling.cc:243] Waited 292 ms on lock 0x7b5400802d54. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:21:41.216479 (+304867us) spinlock_profiling.cc:243] Waited 552 ms on lock 0x7b5400802d50. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a61f8ad 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:21:41.216643 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1941036032}
I20260709 12:21:41.218232 20344 leader_election.cc:290] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 5 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:41.218340 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
W20260709 12:21:41.144594 18333 connection.cc:256] RPC call timeout handler was delayed by -1.9168659410001965s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21760000000000002s for a response
W20260709 12:21:41.220278 18333 connection.cc:256] RPC call timeout handler was delayed by -1.5506248149999919s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.168s for a response
I20260709 12:21:41.220504 20321 leader_election.cc:290] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 5 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:21:41.220759 18333 connection.cc:256] RPC call timeout handler was delayed by -1.520880431000478s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2469s for a response
I20260709 12:21:41.221204 20321 raft_consensus.cc:493] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:41.221688 20321 raft_consensus.cc:515] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:41.106443 17835 raft_consensus.cc:3060] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:21:41.151584 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 139272us
I20260709 12:21:41.223399 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:41.223553 17801 leader_election.cc:304] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:21:41.192737 20346 raft_consensus.cc:515] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:41.224479 20422 leader_election.cc:290] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:41.225011 20407 raft_consensus.cc:2804] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:21:41.225589 20407 raft_consensus.cc:697] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 5 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:21:41.226099 20346 leader_election.cc:290] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:41.226725 20346 raft_consensus.cc:493] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:41.226641 20407 consensus_queue.cc:237] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:41.006403 20402 raft_consensus.cc:2804] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:21:40.913200 20406 raft_consensus.cc:515] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:41.038450 20350 raft_consensus.cc:493] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:21:41.160540 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:21:41.161137 18334 leader_election.cc:336] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Timed out: RequestConsensusVote RPC to 127.16.29.65:33071 timed out after 1.797s (SENT)
W20260709 12:21:41.237871 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 76766.2us
I20260709 12:21:41.047967 20357 raft_consensus.cc:493] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:41.051369 20299 raft_consensus.cc:493] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:41.166832 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:41.087867 20384 raft_consensus.cc:493] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:41.008160 20354 raft_consensus.cc:515] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:41.194124 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:41.208654 20398 raft_consensus.cc:740] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 3 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:21:40.916442 20405 raft_consensus.cc:515] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:41.086535 20144 leader_election.cc:290] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:41.086226 17760 raft_consensus.cc:1240] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20260709 12:21:41.030640 20364 raft_consensus.cc:515] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:21:41.137228 20429 log.cc:927] Time spent T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.105s	user 0.001s	sys 0.000s
I20260709 12:21:41.200255 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:41.086587 20418 leader_election.cc:290] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:41.037549 20324 consensus_queue.cc:260] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:21:41.244820 20394 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.755s	user 0.077s	sys 0.098s
I20260709 12:21:40.912899 20377 raft_consensus.cc:515] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:40.904213 20430 raft_consensus.cc:515] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:41.150530 20410 raft_consensus.cc:493] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:21:41.154471 20390 raft_consensus.cc:2749] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:21:41.102361 20381 raft_consensus.cc:740] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 3 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:21:41.209834 17684 raft_consensus.cc:2393] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 5: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:21:41.159710 20427 raft_consensus.cc:515] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:41.213344 17584 catalog_manager.cc:5697] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 reported cstate change: term changed from 4 to 5, leader changed from 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) to e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66). New cstate: current_term: 5 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } }
W20260709 12:21:41.289877 20394 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.797s	user 0.077s	sys 0.098s
I20260709 12:21:41.195212 20349 raft_consensus.cc:493] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:41.061308 20157 leader_election.cc:290] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:41.001071 20347 raft_consensus.cc:2804] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:21:41.245707 20357 raft_consensus.cc:515] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:41.105260 20273 consensus_queue.cc:260] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:41.251919 20384 raft_consensus.cc:515] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:41.046940 20336 raft_consensus.cc:515] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:41.215610 20019 leader_election.cc:290] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:41.246496 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:41.391268 17836 raft_consensus.cc:3060] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:21:41.153133 20365 consensus_meta.cc:287] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.779s	user 0.003s	sys 0.007s
I20260709 12:21:41.247360 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:41.247817 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:41.397545 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:41.398893 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:41.399403 17831 raft_consensus.cc:3060] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:41.246892 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:41.400302 18371 raft_consensus.cc:2393] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 3: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:21:41.235252 20402 raft_consensus.cc:493] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:41.241442 20350 raft_consensus.cc:3060] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:41.148617 20232 raft_consensus.cc:493] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:41.203516 20147 raft_consensus.cc:515] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:41.286130 17764 raft_consensus.cc:1275] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260709 12:21:41.285979 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:41.419464 18374 raft_consensus.cc:3060] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:41.246744 20321 leader_election.cc:290] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:41.147940 20401 consensus_queue.cc:237] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:41.294551 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
W20260709 12:21:41.319553 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:41.423893 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:41.424223 18363 raft_consensus.cc:2393] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 5: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:21:41.356029 20324 raft_consensus.cc:3060] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:41.282750 19826 leader_election.cc:290] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:41.016337 20397 raft_consensus.cc:515] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:41.366941 17584 catalog_manager.cc:5697] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:41.154304 17755 raft_consensus.cc:2468] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 5.
I20260709 12:21:41.249524 20299 raft_consensus.cc:515] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:41.227347 20346 raft_consensus.cc:515] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:41.359738 20410 raft_consensus.cc:3060] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:41.358278 20298 leader_election.cc:290] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 7 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:41.366214 19806 consensus_queue.cc:237] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:41.384346 18365 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59886 (request call id 813) took 1304 ms. Trace:
I20260709 12:21:41.386719 17835 raft_consensus.cc:2468] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 5.
I20260709 12:21:41.440171 18365 rpcz_store.cc:276] 0709 12:21:40.080431 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:40.080980 (+   549us) service_pool.cc:224] Handling call
0709 12:21:41.384114 (+1303134us) spinlock_profiling.cc:243] Waited 523 ms on lock 0x7b5400305554. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:21:41.384261 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1203901696}
I20260709 12:21:41.441061 17835 raft_consensus.cc:1275] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 12:21:41.387620 18362 raft_consensus.cc:2410] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 7 because replica has last-logged OpId of term: 6 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 7.
I20260709 12:21:41.401757 17834 raft_consensus.cc:2468] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 6.
I20260709 12:21:41.371708 20347 raft_consensus.cc:493] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:21:41.405725 17728 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.543s	user 0.009s	sys 0.032s
W20260709 12:21:41.444013 17728 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.582s	user 0.009s	sys 0.032s
W20260709 12:21:41.406420 18364 consensus_meta.cc:287] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.510s	user 0.008s	sys 0.004s
I20260709 12:21:41.407754 17838 raft_consensus.cc:1275] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 7. (term mismatch)
W20260709 12:21:41.452064 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 645100us
I20260709 12:21:41.452385 17838 pending_rounds.cc:85] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b: Aborting all ops after (but not including) 6
I20260709 12:21:41.452641 17838 pending_rounds.cc:107] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b: Aborting uncommitted NO_OP operation due to leader change: 6.7
I20260709 12:21:41.452859 17838 raft_consensus.cc:2889] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:21:41.414610 17652 connection.cc:256] RPC call timeout handler was delayed by -1.2109970159999648s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1938s for a response
W20260709 12:21:41.400851 17654 connection.cc:256] RPC call timeout handler was delayed by -1.1973004820001734s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18620000000000003s for a response
I20260709 12:21:41.396126 20383 leader_election.cc:290] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 4 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:41.472100 17836 raft_consensus.cc:2468] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 5.
I20260709 12:21:41.425345 17830 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35040 (request call id 1948) took 2160 ms. Trace:
W20260709 12:21:41.426354 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 456341us
I20260709 12:21:41.421198 20232 raft_consensus.cc:3060] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:41.429275 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:41.477178 18371 raft_consensus.cc:2468] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 8.
I20260709 12:21:41.430013 17584 catalog_manager.cc:5697] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 reported cstate change: term changed from 2 to 3, leader changed from e063541542a94969898868bf6eb5ce1b (127.16.29.67) to e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66). New cstate: current_term: 3 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } }
I20260709 12:21:41.408310 20402 raft_consensus.cc:3060] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:41.435840 20370 leader_election.cc:290] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:41.379503 20349 raft_consensus.cc:515] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:41.419082 20394 raft_consensus.cc:493] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:41.419122 20144 raft_consensus.cc:493] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:41.457690 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:41.389973 20398 consensus_queue.cc:260] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:41.424921 20273 raft_consensus.cc:3060] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:41.459539 17686 raft_consensus.cc:2468] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 5.
I20260709 12:21:41.429134 20378 leader_election.cc:290] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:21:41.476022 17803 connection.cc:256] RPC call timeout handler was delayed by -1.0850829869996232s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18620000000000003s for a response
W20260709 12:21:41.492262 17803 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:41.492755 17803 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:41.478034 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:41.478439 17584 catalog_manager.cc:5697] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 reported cstate change: term changed from 2 to 3, leader changed from e063541542a94969898868bf6eb5ce1b (127.16.29.67) to 0da169ff82674dc1947055ce43de68a3 (127.16.29.65). New cstate: current_term: 3 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:41.496337 17584 catalog_manager.cc:5697] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 reported cstate change: term changed from 3 to 6, leader changed from e063541542a94969898868bf6eb5ce1b (127.16.29.67) to 0da169ff82674dc1947055ce43de68a3 (127.16.29.65). New cstate: current_term: 6 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:41.479351 18364 raft_consensus.cc:2468] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 3.
W20260709 12:21:41.479621 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 235295us
I20260709 12:21:41.480643 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:41.483137 17687 raft_consensus.cc:2368] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 4. Current term is 5.
I20260709 12:21:41.497792 18364 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59902 (request call id 798) took 1027 ms. Trace:
I20260709 12:21:41.498936 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:41.451210 20364 leader_election.cc:290] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:21:41.422359 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 198875us
I20260709 12:21:41.484169 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:41.499010 17584 catalog_manager.cc:5697] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 reported cstate change: term changed from 3 to 4, leader changed from 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) to 0da169ff82674dc1947055ce43de68a3 (127.16.29.65). New cstate: current_term: 4 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:41.498164 18364 rpcz_store.cc:276] 0709 12:21:40.470561 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:40.522472 (+ 51911us) service_pool.cc:224] Handling call
0709 12:21:41.497729 (+975257us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:41.501279 17801 leader_election.cc:304] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:41.484969 17831 raft_consensus.cc:2468] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 5.
I20260709 12:21:41.485119 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:41.451465 20347 raft_consensus.cc:3060] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:41.486948 17688 raft_consensus.cc:1240] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 4. Current term is 5. Ops: [4.6-4.6]
W20260709 12:21:41.519107 17759 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.505s	user 0.000s	sys 0.021s
I20260709 12:21:41.520197 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:41.521143 17688 raft_consensus.cc:2393] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 5: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:21:41.522080 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:41.522708 17688 raft_consensus.cc:3060] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:41.487167 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:41.524991 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:41.525708 17681 raft_consensus.cc:2468] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 8.
I20260709 12:21:41.526685 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:41.527273 17681 raft_consensus.cc:3060] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:41.490746 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:41.529049 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:41.530328 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:41.503513 17801 leader_election.cc:304] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:41.533031 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:41.419559 20381 consensus_queue.cc:260] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:41.515974 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:41.539335 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:41.473819 17830 rpcz_store.cc:276] 0709 12:21:39.245973 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:39.274730 (+ 28757us) service_pool.cc:224] Handling call
0709 12:21:39.275126 (+   396us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:39.508404 (+233278us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 12:21:39.695090 (+186686us) log.cc:844] Serialized 14 byte log entry
0709 12:21:40.986210 (+1291120us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:40.987168 (+   958us) log.cc:844] Serialized 26 byte log entry
0709 12:21:40.987339 (+   171us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:21:41.404130 (+416791us) log.cc:844] Serialized 14 byte log entry
0709 12:21:41.405960 (+  1830us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:41.406087 (+   127us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:41.406125 (+    38us) raft_consensus.cc:1685] finished
0709 12:21:41.406137 (+    12us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:41.406250 (+   113us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":153884,"prepare.queue_time_us":381489,"prepare.run_cpu_time_us":49,"prepare.run_wall_time_us":50,"raft.queue_time_us":504893,"raft.run_cpu_time_us":439,"raft.run_wall_time_us":44236,"thread_start_us":1903709,"threads_started":4,"wal-append.queue_time_us":752602}
I20260709 12:21:41.480715 20298 raft_consensus.cc:493] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:41.480804 20360 leader_election.cc:290] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:41.533681 17584 catalog_manager.cc:5697] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 reported cstate change: term changed from 1 to 2, leader changed from e063541542a94969898868bf6eb5ce1b (127.16.29.67) to 0da169ff82674dc1947055ce43de68a3 (127.16.29.65). New cstate: current_term: 2 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:41.534695 17760 raft_consensus.cc:1240] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 4. Current term is 5. Ops: [4.6-4.6]
W20260709 12:21:41.537197 18333 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:41.537066 17652 leader_election.cc:304] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:21:41.538443 17757 raft_consensus.cc:1240] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 2. Current term is 3. Ops: []
I20260709 12:21:41.538650 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:41.553535 17761 raft_consensus.cc:2468] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 7.
I20260709 12:21:41.539577 17689 raft_consensus.cc:2393] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 5: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:21:41.516813 17683 raft_consensus.cc:2393] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 5: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:21:41.517406 17684 raft_consensus.cc:2410] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for term 6 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20260709 12:21:41.531613 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:41.483268 20168 raft_consensus.cc:493] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:21:41.558220 17584 catalog_manager.cc:5697] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } }
I20260709 12:21:41.560271 17584 catalog_manager.cc:5697] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 reported cstate change: term changed from 2 to 3, leader changed from 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) to 0da169ff82674dc1947055ce43de68a3 (127.16.29.65). New cstate: current_term: 3 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:41.561810 17584 catalog_manager.cc:5697] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:41.565470 17688 raft_consensus.cc:2468] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 6.
I20260709 12:21:41.570755 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:41.514997 20354 leader_election.cc:290] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:41.516023 20365 raft_consensus.cc:2749] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:21:41.515347 20383 raft_consensus.cc:493] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:21:41.542269 20390 raft_consensus.cc:2749] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:21:41.583323 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:41.542340 20376 raft_consensus.cc:493] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:21:41.574534 18333 leader_election.cc:304] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:21:41.482302 20296 raft_consensus.cc:515] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:41.572611 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:41.498529 20144 raft_consensus.cc:515] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:41.587143 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:41.591612 17688 raft_consensus.cc:2393] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 6: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:21:41.596617 17683 raft_consensus.cc:3060] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:41.595378 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:41.598325 17684 raft_consensus.cc:2393] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 5: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:21:41.601184 17831 raft_consensus.cc:1240] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260709 12:21:41.491520 20394 raft_consensus.cc:515] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:41.599682 17654 leader_election.cc:304] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:21:41.607422 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:41.576525 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:41.584666 20360 raft_consensus.cc:2804] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:21:41.615058 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:41.580211 20443 raft_consensus.cc:493] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:41.623978 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:41.616686 17837 raft_consensus.cc:2393] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 6: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:21:41.557209 20298 raft_consensus.cc:515] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:41.628979 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:41.629276 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:41.633730 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:41.596975 20376 raft_consensus.cc:3060] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:41.634915 17836 raft_consensus.cc:2393] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 3: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:21:41.632011 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:41.641466 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:41.584129 20321 consensus_queue.cc:1048] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.037s
I20260709 12:21:41.643813 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:41.649909 17836 raft_consensus.cc:3060] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:41.603500 20377 leader_election.cc:290] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 5 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:41.644990 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:41.591375 20452 raft_consensus.cc:493] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:41.650933 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:41.656291 17688 raft_consensus.cc:2393] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 3: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:21:41.651629 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:41.611143 20352 leader_election.cc:290] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:41.660915 17684 raft_consensus.cc:2393] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 3: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
W20260709 12:21:41.654922 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 55325.7us
I20260709 12:21:41.657876 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:41.625979 20360 raft_consensus.cc:493] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:41.659047 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:41.588063 20445 consensus_queue.cc:1048] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.068s
I20260709 12:21:41.664103 17835 raft_consensus.cc:2468] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 5.
I20260709 12:21:41.673360 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:41.576885 20168 raft_consensus.cc:515] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:41.665992 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:41.593182 20383 raft_consensus.cc:515] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:41.666757 17727 leader_election.cc:304] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:21:41.673990 17835 raft_consensus.cc:2468] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 5.
I20260709 12:21:41.673899 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:41.675292 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:41.676247 17831 raft_consensus.cc:3060] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:41.685806 17689 raft_consensus.cc:1813] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Handling vote request from an unknown peer e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:21:41.652019 20435 log.cc:927] Time spent T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.246s	user 0.000s	sys 0.001s
I20260709 12:21:41.674002 17836 raft_consensus.cc:2468] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 4.
I20260709 12:21:41.686429 17689 raft_consensus.cc:2393] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 4: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:21:41.644550 20157 raft_consensus.cc:493] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:21:41.691645 17728 leader_election.cc:304] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:21:41.709045 20466 raft_consensus.cc:493] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:41.712406 17838 raft_consensus.cc:2393] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 3: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:21:41.720712 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:41.721355 17838 raft_consensus.cc:2468] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 5.
I20260709 12:21:41.648814 20406 leader_election.cc:290] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:41.658421 20441 raft_consensus.cc:493] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:41.680181 20377 raft_consensus.cc:2804] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:21:41.740042 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:41.722455 17684 raft_consensus.cc:1240] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20260709 12:21:41.654371 20357 leader_election.cc:290] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:41.662734 20354 raft_consensus.cc:493] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:41.658252 20384 leader_election.cc:290] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:41.727566 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 8 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:41.726554 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:41.685096 19855 raft_consensus.cc:2804] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:21:41.734388 17831 raft_consensus.cc:2468] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 7.
I20260709 12:21:41.672254 20360 raft_consensus.cc:3060] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:41.618611 20418 consensus_queue.cc:1048] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.056s
I20260709 12:21:41.684037 20352 raft_consensus.cc:493] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:41.746450 18374 raft_consensus.cc:2468] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 5.
I20260709 12:21:41.768951 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:41.664403 20442 consensus_queue.cc:1048] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.077s
I20260709 12:21:41.638854 20443 raft_consensus.cc:515] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:41.718165 20147 leader_election.cc:290] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:21:41.765607 20437 log.cc:927] Time spent T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.338s	user 0.001s	sys 0.000s
W20260709 12:21:41.772475 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 179395us
I20260709 12:21:41.777895 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:41.787909 18374 raft_consensus.cc:2376] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 3. Re-sending same reply.
I20260709 12:21:41.788204 17652 leader_election.cc:304] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:21:41.780203 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:41.722651 20453 raft_consensus.cc:2804] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:21:41.794556 20453 raft_consensus.cc:697] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 7 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:21:41.795307 20168 leader_election.cc:290] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:41.795491 20453 consensus_queue.cc:237] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:21:41.764514 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 100827us
I20260709 12:21:41.743324 20377 raft_consensus.cc:493] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:41.784404 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:41.801117 17835 raft_consensus.cc:2368] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 4. Current term is 5.
I20260709 12:21:41.655834 20452 raft_consensus.cc:515] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:41.776319 20357 raft_consensus.cc:2749] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:21:41.768805 20466 raft_consensus.cc:515] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:41.809013 20296 leader_election.cc:290] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 6 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:41.799273 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:41.810585 18371 raft_consensus.cc:2393] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 5: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:21:41.811385 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:41.805647 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:21:41.812477 20451 log.cc:927] Time spent T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.162s	user 0.000s	sys 0.000s
I20260709 12:21:41.812911 18374 raft_consensus.cc:2410] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 4 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260709 12:21:41.813766 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:41.814339 18374 raft_consensus.cc:3060] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:41.807539 18366 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59902 (request call id 793) took 1398 ms. Trace:
I20260709 12:21:41.732041 20157 raft_consensus.cc:515] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:41.792796 17652 leader_election.cc:304] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:21:41.816051 18366 rpcz_store.cc:276] 0709 12:21:40.408958 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:40.409117 (+   159us) service_pool.cc:224] Handling call
0709 12:21:40.409405 (+   288us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:40.414097 (+  4692us) raft_consensus.cc:1534] Early marking committed up to index 3
0709 12:21:40.414133 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:40.420862 (+  6729us) log.cc:844] Serialized 26 byte log entry
0709 12:21:40.883711 (+462849us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:21:40.883753 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:40.883832 (+    79us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:41.807333 (+923501us) raft_consensus.cc:1685] finished
0709 12:21:41.807365 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:41.807485 (+   120us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":79,"raft.run_cpu_time_us":98,"raft.run_wall_time_us":99,"thread_start_us":459054,"threads_started":1,"wal-append.queue_time_us":933859}
I20260709 12:21:41.834784 20445 consensus_queue.cc:1059] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:21:41.835733 20383 leader_election.cc:290] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:41.835903 20321 raft_consensus.cc:493] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:41.835742 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:41.836406 20321 raft_consensus.cc:515] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:41.836871 18366 raft_consensus.cc:2393] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 6: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
W20260709 12:21:41.838037 20373 consensus_meta.cc:287] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.849s	user 0.010s	sys 0.000s
I20260709 12:21:41.770892 20354 raft_consensus.cc:515] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:41.840117 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:41.840744 17681 raft_consensus.cc:2468] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 3.
I20260709 12:21:41.842010 17681 raft_consensus.cc:1275] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
W20260709 12:21:41.800206 20446 log.cc:927] Time spent T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.284s	user 0.001s	sys 0.000s
I20260709 12:21:41.777195 20352 raft_consensus.cc:515] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:41.840902 17684 raft_consensus.cc:2410] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for term 6 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20260709 12:21:41.838097 18366 raft_consensus.cc:1275] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260709 12:21:41.743803 20441 raft_consensus.cc:515] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:41.800505 20147 consensus_queue.cc:1048] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.094s
I20260709 12:21:41.839497 20360 raft_consensus.cc:515] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:42.349469 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:42.042830 17837 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35040 (request call id 1954) took 1056 ms. Trace:
I20260709 12:21:41.804934 20377 raft_consensus.cc:3060] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:21:42.349395 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:41.951519 18374 raft_consensus.cc:2468] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 5.
W20260709 12:21:41.932390 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 246209us
I20260709 12:21:42.358237 17837 rpcz_store.cc:276] 0709 12:21:40.986862 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:40.998506 (+ 11644us) service_pool.cc:224] Handling call
0709 12:21:42.042581 (+1044075us) spinlock_profiling.cc:243] Waited 568 ms on lock 0x7b54002ebb54. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:21:42.042754 (+   173us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1306935680}
W20260709 12:21:42.229118 20234 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.605s	user 0.010s	sys 0.009s
I20260709 12:21:42.358206 18364 raft_consensus.cc:2468] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 3.
W20260709 12:21:41.932694 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 219105us
I20260709 12:21:41.852587 17683 raft_consensus.cc:2468] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 7.
I20260709 12:21:42.358105 18366 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 966) took 1341 ms. Trace:
I20260709 12:21:42.386935 18366 rpcz_store.cc:276] 0709 12:21:41.016217 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:41.837509 (+821292us) service_pool.cc:224] Handling call
0709 12:21:41.837792 (+   283us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:42.357809 (+520017us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:42.358046 (+   237us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:21:41.902915 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 211315us
I20260709 12:21:42.387676 18333 leader_election.cc:304] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
W20260709 12:21:41.909710 17801 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:42.387138 17727 leader_election.cc:304] T 7cb3768d215e409fb9f5baf18d124442 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:21:42.377202 20352 leader_election.cc:290] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:41.845842 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:41.946838 17755 raft_consensus.cc:2468] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 5.
I20260709 12:21:41.946965 20232 raft_consensus.cc:515] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:21:41.865674 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 73390.1us
I20260709 12:21:41.801788 20384 raft_consensus.cc:2804] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Leader election won for term 5
W20260709 12:21:42.349345 20350 consensus_meta.cc:287] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.925s	user 0.006s	sys 0.005s
I20260709 12:21:42.446560 17684 raft_consensus.cc:1240] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 4. Current term is 5. Ops: []
W20260709 12:21:41.940412 20455 log.cc:927] Time spent T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.151s	user 0.000s	sys 0.000s
I20260709 12:21:41.944293 20443 leader_election.cc:290] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:42.454335 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:21:42.230819 20264 consensus_meta.cc:287] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 1.304s	user 0.007s	sys 0.006s
W20260709 12:21:42.090685 20324 consensus_meta.cc:287] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.662s	user 0.008s	sys 0.000s
I20260709 12:21:41.943933 20102 raft_consensus.cc:3055] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:21:41.892493 20327 raft_consensus.cc:2804] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:21:42.455084 17835 raft_consensus.cc:2468] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 6.
I20260709 12:21:42.455581 18366 raft_consensus.cc:1240] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 5. Current term is 6. Ops: []
W20260709 12:21:42.455502 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 58805.1us
I20260709 12:21:41.842257 20387 raft_consensus.cc:493] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:41.942196 18361 consensus_meta.cc:287] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.510s	user 0.006s	sys 0.002s
I20260709 12:21:42.513551 18361 raft_consensus.cc:2468] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 4.
I20260709 12:21:42.087626 20349 leader_election.cc:290] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:42.459944 20443 raft_consensus.cc:2749] T 7cb3768d215e409fb9f5baf18d124442 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:21:41.875829 20465 raft_consensus.cc:493] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
W20260709 12:21:42.117956 20402 consensus_meta.cc:287] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.626s	user 0.006s	sys 0.003s
W20260709 12:21:42.396085 20234 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.772s	user 0.013s	sys 0.011s
I20260709 12:21:42.530949 20476 consensus_queue.cc:1048] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 12:21:42.531625 20327 raft_consensus.cc:697] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 5 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:21:42.532639 20327 consensus_queue.cc:237] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:42.460317 20232 leader_election.cc:290] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 4 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:42.460700 20441 leader_election.cc:290] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:42.461171 17762 raft_consensus.cc:2393] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 4: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:21:42.536464 20442 consensus_queue.cc:1048] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:21:42.538621 17652 leader_election.cc:304] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:42.462431 20332 consensus_queue.cc:260] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:41.933300 17680 raft_consensus.cc:1240] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 2. Current term is 3. Ops: []
I20260709 12:21:42.539619 20442 raft_consensus.cc:2749] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20260709 12:21:42.464629 17686 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37402 (request call id 2258) took 1973 ms (client timeout 1988 ms). Trace:
W20260709 12:21:42.540407 17686 rpcz_store.cc:269] 0709 12:21:40.491115 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:40.491313 (+   198us) service_pool.cc:224] Handling call
0709 12:21:42.464460 (+1973147us) spinlock_profiling.cc:243] Waited 566 ms on lock 0x7b54003458d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:21:42.464568 (+   108us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1300671872}
I20260709 12:21:42.540869 20360 leader_election.cc:290] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 6 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:42.465221 18374 raft_consensus.cc:1275] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 12:21:42.081657 20452 leader_election.cc:290] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:42.514531 17833 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35040 (request call id 1955) took 1512 ms. Trace:
I20260709 12:21:42.511525 18366 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 970) took 1297 ms. Trace:
I20260709 12:21:42.541693 20472 raft_consensus.cc:2749] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:21:42.541841 18366 rpcz_store.cc:276] 0709 12:21:41.213943 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:42.443874 (+1229931us) service_pool.cc:224] Handling call
0709 12:21:42.455378 (+ 11504us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:42.511177 (+ 55799us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:42.511452 (+   275us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:42.541721 18374 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 969) took 1332 ms. Trace:
I20260709 12:21:42.543386 18374 rpcz_store.cc:276] 0709 12:21:41.209370 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:42.408235 (+1198865us) service_pool.cc:224] Handling call
0709 12:21:42.464892 (+ 56657us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:42.541484 (+ 76592us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:42.541667 (+   183us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:42.543817 18366 raft_consensus.cc:1275] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
W20260709 12:21:42.453749 20410 consensus_meta.cc:287] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 1.017s	user 0.010s	sys 0.003s
W20260709 12:21:42.457742 20273 consensus_meta.cc:287] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.958s	user 0.004s	sys 0.008s
I20260709 12:21:42.521404 20465 raft_consensus.cc:515] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:42.507493 20102 raft_consensus.cc:740] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:21:42.507833 20384 raft_consensus.cc:697] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 5 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:21:42.514235 20144 leader_election.cc:290] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:42.518018 20321 leader_election.cc:290] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:42.518244 20387 raft_consensus.cc:515] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:42.514424 20264 raft_consensus.cc:515] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:42.529441 20349 consensus_queue.cc:1059] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:21:42.530094 20402 raft_consensus.cc:515] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:42.530432 18363 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 965) took 1553 ms. Trace:
I20260709 12:21:42.605472 18363 rpcz_store.cc:276] 0709 12:21:40.977345 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:41.543169 (+565824us) service_pool.cc:224] Handling call
0709 12:21:41.576579 (+ 33410us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:21:41.585781 (+  9202us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 12:21:41.585826 (+    45us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:21:41.594498 (+  8672us) log.cc:844] Serialized 52 byte log entry
0709 12:21:41.820334 (+225836us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:21:41.820389 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:41.820480 (+    91us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:42.530207 (+709727us) raft_consensus.cc:1685] finished
0709 12:21:42.530227 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:42.530377 (+   150us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":9623,"raft.run_cpu_time_us":187,"raft.run_wall_time_us":7765,"thread_start_us":140143,"threads_started":1,"wal-append.queue_time_us":922194}
I20260709 12:21:42.606622 20402 leader_election.cc:290] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 6 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:21:42.607654 20347 consensus_meta.cc:287] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 1.064s	user 0.012s	sys 0.000s
I20260709 12:21:42.607942 18363 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59886 (request call id 823) took 1375 ms. Trace:
I20260709 12:21:42.608086 20347 raft_consensus.cc:515] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:42.608307 18363 rpcz_store.cc:276] 0709 12:21:41.232485 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:42.607067 (+1374582us) service_pool.cc:224] Handling call
0709 12:21:42.607412 (+   345us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:42.607595 (+   183us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:42.607621 (+    26us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:42.607663 (+    42us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:21:42.607705 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:42.607757 (+    52us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:42.607857 (+   100us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:42.609866 18363 raft_consensus.cc:1275] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:21:42.610200 20347 leader_election.cc:290] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 6 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:42.541695 17833 rpcz_store.cc:276] 0709 12:21:41.002240 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:41.002431 (+   191us) service_pool.cc:224] Handling call
0709 12:21:41.002754 (+   323us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:21:41.031820 (+ 29066us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:41.031853 (+    33us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:21:41.032462 (+   609us) log.cc:844] Serialized 52 byte log entry
0709 12:21:41.421877 (+389415us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:21:41.421930 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:41.422024 (+    94us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:42.514319 (+1092295us) raft_consensus.cc:1685] finished
0709 12:21:42.514348 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:42.514475 (+   127us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":9435,"raft.queue_time_us":9600,"raft.run_cpu_time_us":751,"raft.run_wall_time_us":20862,"thread_start_us":274077,"threads_started":1,"wal-append.queue_time_us":368400}
I20260709 12:21:42.610982 18367 raft_consensus.cc:2393] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 7: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
W20260709 12:21:42.611918 18367 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59902 (request call id 799) took 2141 ms (client timeout 1862 ms). Trace:
W20260709 12:21:42.612332 18367 rpcz_store.cc:269] 0709 12:21:40.470839 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:40.935514 (+464675us) service_pool.cc:224] Handling call
0709 12:21:42.611732 (+1676218us) spinlock_profiling.cc:243] Waited 1.67 s on lock 0x7b5400305f54. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:21:42.611837 (+   105us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3850709376}
I20260709 12:21:42.541354 17652 leader_election.cc:304] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:42.517922 20354 leader_election.cc:290] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:42.580171 18366 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59886 (request call id 820) took 1330 ms. Trace:
W20260709 12:21:42.614161 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 72913us
I20260709 12:21:42.614014 18366 rpcz_store.cc:276] 0709 12:21:41.215094 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:42.542986 (+1327892us) service_pool.cc:224] Handling call
0709 12:21:42.543403 (+   417us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:42.545066 (+  1663us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:42.545295 (+   229us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:42.581424 18374 raft_consensus.cc:1275] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:21:42.615103 17652 leader_election.cc:304] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:42.615480 18374 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59886 (request call id 822) took 1399 ms. Trace:
I20260709 12:21:42.616169 20144 raft_consensus.cc:2749] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:21:42.616869 20144 raft_consensus.cc:2749] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 12:21:42.539867 20332 raft_consensus.cc:3060] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:42.618536 20427 leader_election.cc:290] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:21:42.619714 18362 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59902 (request call id 796) took 2149 ms (client timeout 1960 ms). Trace:
W20260709 12:21:42.620127 18362 rpcz_store.cc:269] 0709 12:21:40.469954 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:40.943388 (+473434us) service_pool.cc:224] Handling call
0709 12:21:42.619504 (+1676116us) spinlock_profiling.cc:243] Waited 348 ms on lock 0x7b54007e7cd4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:21:42.619646 (+   142us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":800886656}
I20260709 12:21:42.622011 20384 consensus_queue.cc:237] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:42.615967 18374 rpcz_store.cc:276] 0709 12:21:41.215721 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:42.544161 (+1328440us) service_pool.cc:224] Handling call
0709 12:21:42.562328 (+ 18167us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:42.615180 (+ 52852us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:42.615419 (+   239us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:42.624155 17685 raft_consensus.cc:2468] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 6.
W20260709 12:21:42.583487 17830 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.846s	user 0.020s	sys 0.030s
W20260709 12:21:42.625128 17830 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.902s	user 0.020s	sys 0.034s
W20260709 12:21:42.545022 20443 consensus_queue.cc:1175] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "e063541542a94969898868bf6eb5ce1b" responder_term: 6 status { last_received { term: 5 index: 7 } last_committed_idx: 6 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 5. Current term is 6. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260709 12:21:42.601403 17759 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38776 (request call id 1777) took 1674 ms. Trace:
I20260709 12:21:42.600940 20102 consensus_queue.cc:260] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:42.602196 20264 leader_election.cc:290] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 7 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:42.583794 20418 raft_consensus.cc:493] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:42.603893 20422 raft_consensus.cc:3055] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:21:42.546303 17835 raft_consensus.cc:1275] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:21:42.617179 20466 leader_election.cc:290] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:42.627017 17652 leader_election.cc:304] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:42.466393 20350 raft_consensus.cc:515] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:42.628705 17830 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37994 (request call id 615) took 1029 ms. Trace:
I20260709 12:21:42.603391 20465 leader_election.cc:290] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:42.628099 17759 rpcz_store.cc:276] 0709 12:21:40.927090 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:40.927221 (+   131us) service_pool.cc:224] Handling call
0709 12:21:40.994137 (+ 66916us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:41.519837 (+525700us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:41.519916 (+    79us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:41.520631 (+   715us) log.cc:844] Serialized 26 byte log entry
0709 12:21:41.583073 (+ 62442us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:21:41.583135 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:41.583263 (+   128us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:42.601140 (+1017877us) raft_consensus.cc:1685] finished
0709 12:21:42.601164 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:42.601321 (+   157us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":10535,"raft.queue_time_us":205672,"raft.run_cpu_time_us":612,"raft.run_wall_time_us":22816,"thread_start_us":567992,"threads_started":2,"wal-append.queue_time_us":153676}
I20260709 12:21:42.670760 17835 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35042 (request call id 1744) took 1004 ms. Trace:
I20260709 12:21:42.687698 17835 rpcz_store.cc:276] 0709 12:21:41.666012 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:42.519017 (+853005us) service_pool.cc:224] Handling call
0709 12:21:42.545989 (+ 26972us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:42.670290 (+124301us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:42.670644 (+   354us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:21:42.673091 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 275731us
W20260709 12:21:42.689983 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 172819us
W20260709 12:21:42.675348 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 287727us
I20260709 12:21:42.624888 20364 raft_consensus.cc:493] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:42.675348 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 48394.3us
W20260709 12:21:42.674901 20376 consensus_meta.cc:287] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 1.020s	user 0.011s	sys 0.000s
I20260709 12:21:42.678675 17830 rpcz_store.cc:276] 0709 12:21:41.599448 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:41.599603 (+   155us) service_pool.cc:224] Handling call
0709 12:21:41.599906 (+   303us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:41.600116 (+   210us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 12:21:41.722601 (+122485us) log.cc:844] Serialized 14 byte log entry
0709 12:21:42.628384 (+905783us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:42.628439 (+    55us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:21:42.628487 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:42.628550 (+    63us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:42.628654 (+   104us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":31790,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":44,"raft.queue_time_us":1274,"raft.run_cpu_time_us":190,"raft.run_wall_time_us":32931,"thread_start_us":1003910,"threads_started":2}
I20260709 12:21:42.692443 18333 leader_election.cc:304] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:21:42.694204 17652 leader_election.cc:304] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:42.690776 17727 leader_election.cc:304] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:21:42.580551 20410 raft_consensus.cc:515] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:42.670125 20422 raft_consensus.cc:740] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:21:42.704699 17756 raft_consensus.cc:2410] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for term 6 because replica has last-logged OpId of term: 4 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260709 12:21:42.680112 20440 raft_consensus.cc:2749] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:21:42.696532 20102 raft_consensus.cc:3060] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:21:42.717092 17730 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:42.719059 17831 raft_consensus.cc:2393] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 4: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
W20260709 12:21:42.721151 17727 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:42.706298 18333 leader_election.cc:304] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:21:42.707118 20144 raft_consensus.cc:2749] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:21:42.702908 20370 raft_consensus.cc:493] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:42.708457 20465 raft_consensus.cc:2749] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:21:42.727065 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:21:42.704155 20127 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.801s	user 0.000s	sys 0.015s
W20260709 12:21:42.727965 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:42.725785 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:42.731331 18334 leader_election.cc:304] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:21:42.674659 20418 raft_consensus.cc:515] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:42.733755 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:42.738569 17830 raft_consensus.cc:3060] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:42.740206 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:42.739709 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:42.740808 17835 raft_consensus.cc:2468] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 4.
I20260709 12:21:42.741724 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:42.742458 17833 raft_consensus.cc:2393] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 4: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:21:42.739960 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:42.736194 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:42.743451 17761 raft_consensus.cc:2468] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 7.
I20260709 12:21:42.698626 20364 raft_consensus.cc:515] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:42.723841 20397 leader_election.cc:290] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:42.727741 20347 raft_consensus.cc:2749] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:21:42.763304 18364 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 967) took 1626 ms. Trace:
I20260709 12:21:42.747814 17760 raft_consensus.cc:2393] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 5: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:21:42.763749 18364 rpcz_store.cc:276] 0709 12:21:41.137211 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:42.407776 (+1270565us) service_pool.cc:224] Handling call
0709 12:21:42.497277 (+ 89501us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:42.613281 (+116004us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:42.613310 (+    29us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:42.613346 (+    36us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:21:42.613373 (+    27us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:42.763045 (+149672us) spinlock_profiling.cc:243] Waited 116 ms on lock 0x7b5400499654. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:21:42.763110 (+    65us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:42.763250 (+   140us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":266339072}
I20260709 12:21:42.764716 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:42.766571 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:42.766736 20299 leader_election.cc:290] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:42.767058 17760 raft_consensus.cc:3060] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:21:42.768190 20436 log.cc:927] Time spent T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 1.889s	user 0.010s	sys 0.014s
W20260709 12:21:42.749135 17803 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:42.749027 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:42.749246 20350 leader_election.cc:290] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 5 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:42.750110 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:42.776149 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:42.744335 20466 raft_consensus.cc:2804] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:21:42.786279 17727 leader_election.cc:304] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:21:42.753125 20478 log.cc:927] Time spent T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.056s	user 0.000s	sys 0.000s
I20260709 12:21:42.755698 17764 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38776 (request call id 1780) took 1741 ms. Trace:
I20260709 12:21:42.734601 20480 raft_consensus.cc:2804] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:21:42.758333 17758 raft_consensus.cc:2468] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 6.
I20260709 12:21:42.759224 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:42.703791 20376 raft_consensus.cc:515] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:42.725878 20385 leader_election.cc:290] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:42.740003 20370 raft_consensus.cc:515] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:42.783483 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:42.799943 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:42.805945 17761 raft_consensus.cc:2468] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 3.
I20260709 12:21:42.750998 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:42.770412 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:42.785483 17757 raft_consensus.cc:2468] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 5.
W20260709 12:21:42.806878 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 105105us
I20260709 12:21:42.785773 17689 raft_consensus.cc:2468] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 7.
I20260709 12:21:42.779659 17686 raft_consensus.cc:1240] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 4. Current term is 5. Ops: []
I20260709 12:21:42.807857 17652 leader_election.cc:304] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:42.735055 20264 consensus_queue.cc:1059] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.018s
I20260709 12:21:42.787722 17727 leader_election.cc:304] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:21:42.742529 20127 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.839s	user 0.001s	sys 0.015s
I20260709 12:21:42.788570 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 8 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:21:42.786123 17803 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:42.813225 17803 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:42.787142 17832 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35042 (request call id 1721) took 3049 ms. Trace:
I20260709 12:21:42.793012 17830 raft_consensus.cc:2468] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 6.
I20260709 12:21:42.793552 17764 rpcz_store.cc:276] 0709 12:21:41.014027 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:41.014314 (+   287us) service_pool.cc:224] Handling call
0709 12:21:41.014603 (+   289us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:41.419090 (+404487us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:42.755455 (+1336365us) spinlock_profiling.cc:243] Waited 271 ms on lock 0x7b54003e44d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:21:42.755628 (+   173us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":623652096}
I20260709 12:21:42.804543 17687 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44898 (request call id 590) took 1322 ms. Trace:
I20260709 12:21:42.814568 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:42.804785 17760 raft_consensus.cc:2468] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 6.
I20260709 12:21:42.752343 20410 leader_election.cc:290] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 4 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:42.808606 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:42.807621 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:42.819494 17689 raft_consensus.cc:3060] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:42.819577 17757 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 7.
I20260709 12:21:42.809509 17838 raft_consensus.cc:2368] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for earlier term 5. Current term is 6.
I20260709 12:21:42.820147 17802 leader_election.cc:304] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:21:42.820756 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:42.732677 20452 consensus_queue.cc:1048] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.063s
I20260709 12:21:42.763522 20377 raft_consensus.cc:515] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:42.811159 17727 leader_election.cc:304] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:21:42.791772 20466 raft_consensus.cc:493] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:42.795928 20406 raft_consensus.cc:2804] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:21:42.815129 17830 raft_consensus.cc:3060] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:42.814956 17687 rpcz_store.cc:276] 0709 12:21:41.482584 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:41.533894 (+ 51310us) service_pool.cc:224] Handling call
0709 12:21:42.759065 (+1225171us) spinlock_profiling.cc:243] Waited 7.18 ms on lock 0x7b5800002a98. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b474fbb71 00007f7b474fba22 00007f7b504148d9 00007f7b4f56eee0 00007f7b4f6380ab 00007f7b4f638131 00007f7b4f5e60a0 00007f7b4f5e5984 00007f7b48b606d4 00007f7b48b60662
0709 12:21:42.804479 (+ 45414us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":16510720}
I20260709 12:21:42.816924 20350 raft_consensus.cc:2804] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:21:42.815282 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:42.795585 20401 raft_consensus.cc:493] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:42.817011 17803 leader_election.cc:304] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:21:42.816323 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:42.816462 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:42.812204 20397 consensus_queue.cc:1059] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.006s
I20260709 12:21:42.797097 20480 raft_consensus.cc:493] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:21:42.796710 20471 raft_consensus.cc:493] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:42.827328 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:42.861634 17684 raft_consensus.cc:2468] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 5.
I20260709 12:21:42.862628 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:42.863020 20377 leader_election.cc:290] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 9 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:42.850695 20466 raft_consensus.cc:3060] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:21:42.828059 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:42.865429 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:42.866171 18374 raft_consensus.cc:2393] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 6: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:21:42.860843 20401 raft_consensus.cc:515] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:42.867313 17802 leader_election.cc:304] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:21:42.792996 20422 consensus_queue.cc:260] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:42.868417 20422 raft_consensus.cc:3060] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:42.869103 20296 raft_consensus.cc:2804] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:21:42.869498 20296 raft_consensus.cc:493] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:21:42.869719 17762 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36572 (request call id 536) took 1380 ms. Trace:
I20260709 12:21:42.869890 20296 raft_consensus.cc:3060] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:42.870127 17762 rpcz_store.cc:276] 0709 12:21:41.489303 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:41.489860 (+   557us) service_pool.cc:224] Handling call
0709 12:21:42.869547 (+1379687us) spinlock_profiling.cc:243] Waited 921 ms on lock 0x7b54000fe6d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:21:42.869665 (+   118us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2118058880}
I20260709 12:21:42.871282 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:42.828603 20370 leader_election.cc:290] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:42.831480 20387 leader_election.cc:290] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:42.832120 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:42.875032 18364 raft_consensus.cc:2393] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 5: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:21:42.833461 20485 raft_consensus.cc:3055] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:21:42.833413 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:42.875424 20485 raft_consensus.cc:740] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:21:42.833803 20402 raft_consensus.cc:2804] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:21:42.876077 20402 raft_consensus.cc:493] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:21:42.876492 20402 raft_consensus.cc:3060] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:42.878582 20394 leader_election.cc:290] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:42.835031 20284 raft_consensus.cc:3060] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:42.880000 20471 raft_consensus.cc:515] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:42.881210 20430 leader_election.cc:290] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:42.882551 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:42.883157 17688 raft_consensus.cc:2468] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 3.
I20260709 12:21:42.810379 20385 raft_consensus.cc:493] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:42.835578 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:42.861479 20410 raft_consensus.cc:2749] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:21:42.837587 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:42.872510 20480 raft_consensus.cc:3060] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:42.871798 17762 raft_consensus.cc:2376] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 6. Re-sending same reply.
W20260709 12:21:42.837857 17756 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38776 (request call id 1771) took 1930 ms (client timeout 2460 ms). Trace:
I20260709 12:21:42.838513 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:42.810026 20299 raft_consensus.cc:2749] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:21:42.839550 18366 raft_consensus.cc:1275] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 6. (term mismatch)
I20260709 12:21:42.840142 17683 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44898 (request call id 592) took 1146 ms. Trace:
I20260709 12:21:42.850589 20347 consensus_queue.cc:1048] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.039s
I20260709 12:21:42.842077 17727 leader_election.cc:304] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:21:42.843734 20368 leader_election.cc:290] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:42.845162 17680 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44898 (request call id 584) took 1387 ms. Trace:
I20260709 12:21:42.850955 17689 raft_consensus.cc:2468] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 6.
I20260709 12:21:42.851258 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:42.859731 20406 raft_consensus.cc:697] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 5 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:21:42.816607 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:42.813884 17832 rpcz_store.cc:276] 0709 12:21:39.733507 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:40.480834 (+747327us) service_pool.cc:224] Handling call
0709 12:21:40.481314 (+   480us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:40.481859 (+   545us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:40.481891 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:40.482529 (+   638us) log.cc:844] Serialized 26 byte log entry
0709 12:21:40.878487 (+395958us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:21:40.878559 (+    72us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:40.878683 (+   124us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:42.645506 (+1766823us) spinlock_profiling.cc:243] Waited 25.9 ms on lock 0x7b4400022b4c. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b47459c85 00007f7b4745a6a2 00007f7b4a1cc1db 00007f7b4a1c7aaf 00007f7b4a1bb86e 00007f7b4a1babb6 00007f7b4a1ba7d8 00007f7b4a1c4221 00007f7b4a1c41d9 00007f7b4a1c4169
0709 12:21:42.769087 (+123581us) raft_consensus.cc:1685] finished
0709 12:21:42.769119 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:42.783335 (+ 14216us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":95,"raft.run_cpu_time_us":97,"raft.run_wall_time_us":98,"spinlock_wait_cycles":59655552,"thread_start_us":395397,"threads_started":1,"wal-append.queue_time_us":395925}
I20260709 12:21:42.854358 20350 raft_consensus.cc:493] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:21:42.851040 20453 raft_consensus.cc:2804] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:21:42.867635 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:42.851752 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:42.847056 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:42.850152 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:42.856462 17761 raft_consensus.cc:3060] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:42.857558 17803 leader_election.cc:304] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:21:42.883878 20471 leader_election.cc:290] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:42.858074 17830 raft_consensus.cc:2410] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260709 12:21:42.884168 20385 raft_consensus.cc:515] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:42.916291 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:42.868366 20401 leader_election.cc:290] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:42.828739 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
W20260709 12:21:42.917019 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:42.917356 17836 raft_consensus.cc:3060] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:42.918084 20385 leader_election.cc:290] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:21:42.918524 17685 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44898 (request call id 582) took 1461 ms (client timeout 1797 ms). Trace:
W20260709 12:21:42.918941 17685 rpcz_store.cc:269] 0709 12:21:41.456683 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:41.456845 (+   162us) service_pool.cc:224] Handling call
0709 12:21:42.918368 (+1461523us) spinlock_profiling.cc:243] Waited 1.02 s on lock 0x7b54001aa254. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:21:42.918468 (+   100us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2336772352}
I20260709 12:21:42.919466 17831 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35042 (request call id 1742) took 1261 ms. Trace:
I20260709 12:21:42.874518 20466 raft_consensus.cc:515] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:42.919912 17831 rpcz_store.cc:276] 0709 12:21:41.658458 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:41.779968 (+121510us) service_pool.cc:224] Handling call
0709 12:21:42.919273 (+1139305us) spinlock_profiling.cc:243] Waited 929 ms on lock 0x7b540008c5d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:21:42.919409 (+   136us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2137008512}
I20260709 12:21:42.921046 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:42.921377 20364 leader_election.cc:290] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:42.921509 20466 leader_election.cc:290] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 9 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:42.921679 17831 raft_consensus.cc:2468] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 3.
I20260709 12:21:42.835633 20491 consensus_queue.cc:1059] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:21:42.922565 20491 raft_consensus.cc:3055] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:21:42.922741 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:42.922992 20491 raft_consensus.cc:740] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 4 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:21:42.923437 17831 raft_consensus.cc:2468] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 4.
I20260709 12:21:42.924396 20491 consensus_queue.cc:260] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:42.925077 19855 raft_consensus.cc:493] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:21:42.925529 19855 raft_consensus.cc:3060] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:21:42.884738 17758 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36572 (request call id 542) took 1391 ms. Trace:
I20260709 12:21:42.926230 17758 rpcz_store.cc:276] 0709 12:21:41.493353 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:41.531326 (+ 37973us) service_pool.cc:224] Handling call
0709 12:21:42.884609 (+1353283us) spinlock_profiling.cc:243] Waited 1.2 s on lock 0x7b54003e58d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:21:42.884683 (+    74us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2767570176}
I20260709 12:21:42.927232 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:42.885046 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:42.884285 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:21:42.885986 17834 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35042 (request call id 1722) took 3152 ms (client timeout 1943 ms). Trace:
I20260709 12:21:42.885793 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:42.888695 18365 raft_consensus.cc:2393] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 7: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:21:42.889355 18366 pending_rounds.cc:85] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da: Aborting all ops after (but not including) 5
I20260709 12:21:42.931967 18366 pending_rounds.cc:107] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da: Aborting uncommitted NO_OP operation due to leader change: 3.6
I20260709 12:21:42.932188 18366 raft_consensus.cc:2889] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:21:42.889557 17683 rpcz_store.cc:276] 0709 12:21:41.693570 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:42.446129 (+752559us) service_pool.cc:224] Handling call
0709 12:21:42.464799 (+ 18670us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:42.465043 (+   244us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:42.465069 (+    26us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:42.468871 (+  3802us) log.cc:844] Serialized 26 byte log entry
0709 12:21:42.727771 (+258900us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:21:42.727812 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:42.727874 (+    62us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:42.839926 (+112052us) raft_consensus.cc:1685] finished
0709 12:21:42.839946 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:42.840081 (+   135us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":258540,"threads_started":1,"wal-append.queue_time_us":339100}
W20260709 12:21:42.898065 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 56067.3us
I20260709 12:21:42.934482 17683 raft_consensus.cc:1240] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 5. Current term is 6. Ops: []
I20260709 12:21:42.898123 18362 raft_consensus.cc:2468] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 3.
I20260709 12:21:42.935547 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:42.936909 17686 raft_consensus.cc:2393] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 5: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:21:42.901331 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:42.906948 17652 leader_election.cc:304] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:21:42.910513 17757 raft_consensus.cc:2393] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 4: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:21:42.900002 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:42.885535 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:42.935096 17838 raft_consensus.cc:2376] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 3. Re-sending same reply.
W20260709 12:21:42.990406 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 51472.3us
I20260709 12:21:43.000128 18368 raft_consensus.cc:2468] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 5.
W20260709 12:21:42.888093 17756 rpcz_store.cc:269] 0709 12:21:40.907146 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:40.907341 (+   195us) service_pool.cc:224] Handling call
0709 12:21:42.837599 (+1930258us) spinlock_profiling.cc:243] Waited 1.71 s on lock 0x7b54000ff0d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:21:42.837768 (+   169us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3922799360}
I20260709 12:21:43.000569 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:42.965217 17836 raft_consensus.cc:2468] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 6.
W20260709 12:21:43.072507 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 137060us
I20260709 12:21:43.119269 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:21:42.911252 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 53803us
I20260709 12:21:42.985307 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
W20260709 12:21:43.065973 17728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:43.118912 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 46556.4us
I20260709 12:21:42.948958 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:43.121366 17652 leader_election.cc:304] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:21:43.091616 17730 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:43.121874 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:43.122447 18368 raft_consensus.cc:1240] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 5. Current term is 6. Ops: []
I20260709 12:21:43.123208 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 8 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:43.122174 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:43.124310 17831 raft_consensus.cc:2468] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 7.
W20260709 12:21:43.117527 18334 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.16.29.65:33071 timed out after 1.875s (SENT) [suppressed 27 similar messages]
I20260709 12:21:42.903065 18333 leader_election.cc:304] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:42.938897 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:43.076498 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:42.941731 17761 raft_consensus.cc:2468] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 3.
I20260709 12:21:43.126911 18374 raft_consensus.cc:2468] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 6.
I20260709 12:21:43.127216 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:21:42.970471 17729 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:43.127964 17836 raft_consensus.cc:2393] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 7: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:21:42.931068 18363 raft_consensus.cc:2468] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 5.
I20260709 12:21:43.127777 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:42.887069 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:42.981904 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:43.129966 18374 raft_consensus.cc:2468] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 5.
W20260709 12:21:42.929862 17834 rpcz_store.cc:269] 0709 12:21:39.733757 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:39.888131 (+154374us) service_pool.cc:224] Handling call
0709 12:21:42.885808 (+2997677us) spinlock_profiling.cc:243] Waited 932 ms on lock 0x7b5400267354. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:21:42.885895 (+    87us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2142463744}
I20260709 12:21:43.130924 17762 raft_consensus.cc:2468] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 5.
I20260709 12:21:42.898753 17829 raft_consensus.cc:2468] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 6.
I20260709 12:21:43.133750 17757 raft_consensus.cc:2393] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 3: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:21:43.124208 17652 leader_election.cc:304] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:21:43.076380 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:43.135489 17652 leader_election.cc:304] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:21:43.139698 17803 leader_election.cc:304] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:21:42.894534 20350 raft_consensus.cc:3060] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:43.144484 17652 leader_election.cc:304] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:21:43.146339 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 243345us
I20260709 12:21:43.147722 17761 raft_consensus.cc:1275] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260709 12:21:43.132920 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:43.148988 17759 raft_consensus.cc:1240] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 3. Current term is 4. Ops: []
I20260709 12:21:42.895000 20418 leader_election.cc:290] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:42.894131 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:42.978314 20347 raft_consensus.cc:2804] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:21:43.205020 20347 raft_consensus.cc:493] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:43.205447 20347 raft_consensus.cc:3060] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:21:43.206910 20448 log.cc:927] Time spent T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 1.561s	user 0.012s	sys 0.005s
I20260709 12:21:43.208333 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:43.208992 17834 raft_consensus.cc:2468] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 6.
I20260709 12:21:43.209839 18362 raft_consensus.cc:1240] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 8. Current term is 9. Ops: []
I20260709 12:21:43.210012 17832 raft_consensus.cc:2468] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 4.
I20260709 12:21:43.210809 17832 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343"
dest_uuid: "e063541542a94969898868bf6eb5ce1b"
mode: GRACEFUL
new_leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
 from {username='slave'} at 127.0.0.1:59820
I20260709 12:21:43.211386 17832 raft_consensus.cc:606] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Received request to transfer leadership to TS e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:43.211730 17832 raft_consensus.cc:614] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:21:43.213052 17832 raft_consensus.cc:1240] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260709 12:21:43.213536 18371 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 968) took 2021 ms. Trace:
I20260709 12:21:43.213909 18371 rpcz_store.cc:276] 0709 12:21:41.192050 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:42.411569 (+1219519us) service_pool.cc:224] Handling call
0709 12:21:42.623751 (+212182us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:42.623943 (+   192us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:42.623967 (+    24us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:42.624014 (+    47us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:21:42.624044 (+    30us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:42.624097 (+    53us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:43.213381 (+589284us) spinlock_profiling.cc:243] Waited 146 ms on lock 0x7b54007e7cd0. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a61f8ad 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:21:43.213482 (+   101us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":335318784}
I20260709 12:21:43.209862 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:43.216861 17834 raft_consensus.cc:3060] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:43.166034 17727 leader_election.cc:304] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:21:43.217928 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 51972.6us
I20260709 12:21:42.909386 20336 leader_election.cc:290] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:43.178633 18333 leader_election.cc:304] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:42.913160 20376 leader_election.cc:290] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 6 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:42.901226 20298 leader_election.cc:290] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:42.898590 20157 leader_election.cc:290] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:42.914793 20410 raft_consensus.cc:2804] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:21:43.241370 20410 raft_consensus.cc:697] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 3 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:21:43.242136 17763 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38776 (request call id 1778) took 2233 ms. Trace:
I20260709 12:21:43.180728 17803 leader_election.cc:304] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:21:42.909271 20346 leader_election.cc:290] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:43.243280 20346 raft_consensus.cc:2749] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:21:43.182054 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:43.248950 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:43.249281 18374 raft_consensus.cc:2468] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 5.
I20260709 12:21:43.249702 18364 raft_consensus.cc:3060] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:42.886759 20405 leader_election.cc:290] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:43.183444 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:42.978075 20440 raft_consensus.cc:2804] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:21:43.151896 18367 raft_consensus.cc:2393] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 3: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:21:43.182293 20365 raft_consensus.cc:2749] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:21:43.182541 20384 raft_consensus.cc:2749] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:21:43.172345 20327 raft_consensus.cc:2804] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:21:43.269300 17834 raft_consensus.cc:2468] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 5.
I20260709 12:21:42.910825 20402 raft_consensus.cc:515] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:43.190912 17761 raft_consensus.cc:1275] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260709 12:21:43.193832 19855 raft_consensus.cc:515] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:43.194998 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:43.284844 18368 raft_consensus.cc:2393] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 4: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:21:42.888163 20102 raft_consensus.cc:493] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:21:43.195794 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:21:42.910528 18361 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59918 (request call id 958) took 2019 ms (client timeout 2078 ms). Trace:
I20260709 12:21:42.876492 20485 consensus_queue.cc:260] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:43.142499 20397 raft_consensus.cc:2804] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:21:42.891709 17680 rpcz_store.cc:276] 0709 12:21:41.457683 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:41.462692 (+  5009us) service_pool.cc:224] Handling call
0709 12:21:41.860670 (+397978us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:42.540072 (+679402us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:42.844953 (+304881us) spinlock_profiling.cc:243] Waited 293 ms on lock 0x7b5400190350. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a61f8ad 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:21:42.845091 (+   138us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":673245568}
I20260709 12:21:42.907051 20480 raft_consensus.cc:515] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:43.148756 20394 raft_consensus.cc:2804] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:21:43.172484 20466 raft_consensus.cc:2749] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:21:42.914462 17755 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36572 (request call id 534) took 1426 ms. Trace:
I20260709 12:21:43.197925 17756 raft_consensus.cc:1275] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
W20260709 12:21:43.198484 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:43.304945 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:43.148809 20370 consensus_queue.cc:1048] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.051s
W20260709 12:21:43.199383 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 1.39332e+06us
I20260709 12:21:43.200923 20377 consensus_queue.cc:1048] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:21:42.925812 20296 raft_consensus.cc:515] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:43.320291 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:43.320859 18371 raft_consensus.cc:2468] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 5.
I20260709 12:21:43.321689 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:43.321848 20296 leader_election.cc:290] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 5 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:43.204440 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:43.322383 18371 raft_consensus.cc:2393] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 9: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:21:43.322778 19855 leader_election.cc:290] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 8 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:43.323047 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:43.166266 17729 leader_election.cc:304] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:21:43.235834 17762 raft_consensus.cc:1240] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 4. Current term is 5. Ops: []
I20260709 12:21:43.324076 20485 raft_consensus.cc:3060] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:43.324225 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:43.324525 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:43.324839 18362 raft_consensus.cc:2468] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 5.
I20260709 12:21:43.325068 17762 raft_consensus.cc:2468] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 5.
I20260709 12:21:43.325855 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:43.325994 18362 raft_consensus.cc:1275] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260709 12:21:43.326462 17762 raft_consensus.cc:2393] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 7: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:21:43.327203 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:43.258183 20350 raft_consensus.cc:515] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:43.239179 17727 leader_election.cc:400] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 5 election: Vote denied by peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837) with higher term. Message: Invalid argument: T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for earlier term 5. Current term is 6.
I20260709 12:21:43.328117 17727 leader_election.cc:403] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 5 election: Cancelling election due to peer responding with higher term
I20260709 12:21:43.329329 17762 raft_consensus.cc:1275] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260709 12:21:43.329452 20350 leader_election.cc:290] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 6 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:21:43.240417 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 61890.5us
I20260709 12:21:43.243747 20447 raft_consensus.cc:2749] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:21:43.242436 20410 consensus_queue.cc:237] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:43.330483 17762 raft_consensus.cc:1240] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20260709 12:21:43.331740 17762 raft_consensus.cc:1240] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 4. Current term is 5. Ops: []
I20260709 12:21:43.332517 20480 leader_election.cc:290] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 6 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:43.333117 18333 leader_election.cc:304] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:21:43.243489 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 62978.7us
I20260709 12:21:43.334036 17652 leader_election.cc:304] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:21:43.334609 17803 leader_election.cc:304] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:21:43.334787 20398 raft_consensus.cc:3060] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:43.242645 17763 rpcz_store.cc:276] 0709 12:21:40.995741 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:41.005342 (+  9601us) service_pool.cc:224] Handling call
0709 12:21:41.045937 (+ 40595us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:41.143755 (+ 97818us) raft_consensus.cc:1534] Early marking committed up to index 3
0709 12:21:41.143806 (+    51us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:41.156606 (+ 12800us) log.cc:844] Serialized 26 byte log entry
0709 12:21:41.541704 (+385098us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:21:41.541822 (+   118us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:41.941985 (+400163us) spinlock_profiling.cc:243] Waited 9.51 ms on lock 0x7b48002214d8. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4f549aa1 00007f7b4f540b37 00007f7b4f5409ca 00007f7b4f63826f 00007f7b4f637f37 00007f7b4f64c55c 00007f7b4f64c508 00007f7b4f64c488 00007f7b4f64c443 00007f7b4f64b6ec 00007f7b4e575a96
0709 12:21:42.978865 (+1036880us) spinlock_profiling.cc:243] Waited 97.2 ms on lock 0x7b540019f0d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:21:42.978914 (+    49us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:43.206314 (+227400us) spinlock_profiling.cc:243] Waited 4.87 ms on lock 0x7b4400022b4c. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b47459c85 00007f7b4745a6a2 00007f7b4a1cc1db 00007f7b4a1c7aaf 00007f7b4a1bb86e 00007f7b4a1babb6 00007f7b4a1ba7d8 00007f7b4a1c4221 00007f7b4a1c41d9 00007f7b4a1c4169
0709 12:21:43.216008 (+  9694us) raft_consensus.cc:1685] finished
0709 12:21:43.216029 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:43.216133 (+   104us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":81362,"raft.run_cpu_time_us":5613,"raft.run_wall_time_us":717121,"spinlock_wait_cycles":257777408,"thread_start_us":305305,"threads_started":1,"wal-append.queue_time_us":375050}
I20260709 12:21:42.925836 20491 raft_consensus.cc:3060] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:43.247960 20368 consensus_queue.cc:1059] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 12:21:43.254814 17801 leader_election.cc:304] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:43.263252 17836 raft_consensus.cc:2393] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 7: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:21:43.271171 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:43.288348 18363 raft_consensus.cc:2468] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 5.
I20260709 12:21:43.261018 20479 raft_consensus.cc:2804] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:21:43.265025 20440 raft_consensus.cc:493] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:42.892509 20406 consensus_queue.cc:237] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:43.297534 17830 raft_consensus.cc:2468] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 7.
I20260709 12:21:43.236902 20354 consensus_queue.cc:1048] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.052s
I20260709 12:21:43.315369 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:43.315986 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:43.281651 18364 raft_consensus.cc:2468] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 6.
I20260709 12:21:43.319336 18367 raft_consensus.cc:1275] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260709 12:21:43.303771 20394 raft_consensus.cc:493] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:43.317905 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:42.894940 20453 raft_consensus.cc:697] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 7 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:21:43.296815 20397 raft_consensus.cc:493] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:43.333298 20381 raft_consensus.cc:3060] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:43.410223 20347 raft_consensus.cc:515] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:43.412117 20479 raft_consensus.cc:697] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:21:43.413089 20479 consensus_queue.cc:237] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:43.335579 20418 raft_consensus.cc:2804] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:21:43.323093 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:43.363282 17756 raft_consensus.cc:2393] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 4: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:21:43.416615 17831 raft_consensus.cc:2468] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 5.
I20260709 12:21:43.417099 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:43.417665 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:43.418540 17756 raft_consensus.cc:1275] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260709 12:21:43.418730 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:43.336154 17803 leader_election.cc:304] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
W20260709 12:21:43.420538 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 84455.8us
I20260709 12:21:43.344754 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:43.421269 17803 leader_election.cc:304] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:21:43.363391 18371 raft_consensus.cc:2468] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 3.
I20260709 12:21:43.294294 20327 raft_consensus.cc:697] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 6 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:21:43.422567 17803 leader_election.cc:304] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
W20260709 12:21:43.394428 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 139677us
I20260709 12:21:43.423640 17801 leader_election.cc:304] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:21:43.290251 18361 rpcz_store.cc:269] 0709 12:21:40.891103 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:41.222954 (+331851us) service_pool.cc:224] Handling call
0709 12:21:42.910322 (+1687368us) spinlock_profiling.cc:243] Waited 197 ms on lock 0x7b5400727254. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:21:42.910451 (+   129us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":452419968}
I20260709 12:21:43.306344 17755 rpcz_store.cc:276] 0709 12:21:41.488576 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:41.491676 (+  3100us) service_pool.cc:224] Handling call
0709 12:21:42.914265 (+1422589us) spinlock_profiling.cc:243] Waited 257 ms on lock 0x7b54004150d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:21:42.914397 (+   132us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":591389568}
W20260709 12:21:43.405232 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 239058us
I20260709 12:21:43.426239 20407 raft_consensus.cc:2749] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:21:43.426898 20402 leader_election.cc:290] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 7 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:43.426836 17729 leader_election.cc:304] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:21:43.428031 20476 consensus_queue.cc:1048] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 12:21:43.428376 19826 consensus_queue.cc:1048] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 12:21:43.429418 20476 raft_consensus.cc:2749] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:21:43.427350 17803 leader_election.cc:304] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:21:43.432704 17803 leader_election.cc:304] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:21:43.419811 17756 raft_consensus.cc:1275] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 12:21:43.338881 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:43.364028 17761 raft_consensus.cc:3060] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:43.434355 17729 leader_election.cc:304] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:21:43.364319 18368 raft_consensus.cc:2468] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 4.
I20260709 12:21:43.435012 17830 raft_consensus.cc:1240] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260709 12:21:43.397238 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:43.400404 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:43.401242 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:43.417707 17837 raft_consensus.cc:2468] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 6.
W20260709 12:21:43.404184 18333 leader_election.cc:336] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Timed out: RequestConsensusVote RPC to 127.16.29.66:39651 timed out after 2.027s (SENT)
I20260709 12:21:43.394232 20440 raft_consensus.cc:3060] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:21:43.405213 20296 raft_consensus.cc:2749] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:21:43.393815 20394 raft_consensus.cc:3060] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:43.405408 20234 raft_consensus.cc:2804] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:21:43.444386 17801 leader_election.cc:304] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:43.312279 20344 raft_consensus.cc:3055] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:21:43.445488 17801 leader_election.cc:304] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:43.337248 20447 raft_consensus.cc:2749] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260709 12:21:43.405982 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 187101us
I20260709 12:21:43.296288 20102 raft_consensus.cc:515] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:43.408437 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:43.448279 17727 leader_election.cc:304] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:21:43.451982 17688 raft_consensus.cc:2468] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 3.
I20260709 12:21:43.412292 18363 raft_consensus.cc:1275] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260709 12:21:43.416060 20418 raft_consensus.cc:493] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:21:43.422392 20457 raft_consensus.cc:2749] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:21:43.458429 17729 leader_election.cc:304] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:21:43.423422 20327 consensus_queue.cc:237] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:21:43.394266 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 83373.6us
I20260709 12:21:43.432937 17802 leader_election.cc:304] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:21:43.402017 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:21:43.460476 17803 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:43.437734 18333 leader_election.cc:304] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:43.437829 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:21:43.461155 17803 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:43.366780 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:43.438282 18365 raft_consensus.cc:2393] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 7: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:21:43.439211 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:43.463282 17686 raft_consensus.cc:2393] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 5: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:21:43.439211 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:43.439844 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:43.435194 17831 raft_consensus.cc:1240] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 5. Current term is 6. Ops: [5.6-5.6]
I20260709 12:21:43.465003 17687 raft_consensus.cc:2376] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 6. Re-sending same reply.
I20260709 12:21:43.335791 17652 leader_election.cc:304] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:43.363930 17834 raft_consensus.cc:2468] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 3.
I20260709 12:21:43.449252 17801 leader_election.cc:304] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:43.461781 18362 raft_consensus.cc:2468] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 6.
I20260709 12:21:43.333164 20480 consensus_queue.cc:1059] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.119s
I20260709 12:21:43.483690 17801 leader_election.cc:304] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:43.333154 17762 raft_consensus.cc:1240] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 5. Current term is 6. Ops: []
I20260709 12:21:43.345498 20324 raft_consensus.cc:2804] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:21:43.432014 20397 raft_consensus.cc:3060] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:21:43.499727 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:43.500094 18366 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 4.
I20260709 12:21:43.439461 20405 raft_consensus.cc:2749] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:21:43.483698 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:43.484653 17727 leader_election.cc:304] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:21:43.503993 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 99834.7us
I20260709 12:21:43.467751 17802 leader_election.cc:304] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:21:43.488045 20157 raft_consensus.cc:2804] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:21:43.488116 19855 raft_consensus.cc:2749] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:21:43.488592 20350 raft_consensus.cc:2804] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:21:43.456017 20336 raft_consensus.cc:2749] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:21:43.488066 19826 consensus_queue.cc:1048] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.001s
W20260709 12:21:43.489114 20422 consensus_meta.cc:287] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.620s	user 0.009s	sys 0.000s
I20260709 12:21:43.458127 20418 raft_consensus.cc:3060] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:21:43.490018 20480 raft_consensus.cc:3055] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:21:43.446836 20344 raft_consensus.cc:740] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:21:43.467967 20402 raft_consensus.cc:2749] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 12:21:43.492182 20296 consensus_queue.cc:1048] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:21:43.488487 20453 consensus_queue.cc:237] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:43.494026 20459 raft_consensus.cc:2804] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:21:43.483963 20443 raft_consensus.cc:3055] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:21:43.495314 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:43.456106 20385 raft_consensus.cc:2804] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:21:43.422466 17833 raft_consensus.cc:1240] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 3. Current term is 4. Ops: []
W20260709 12:21:43.452195 20284 consensus_meta.cc:287] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.572s	user 0.005s	sys 0.005s
I20260709 12:21:43.501053 17729 leader_election.cc:304] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
W20260709 12:21:43.504112 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 386613us
I20260709 12:21:43.504433 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
W20260709 12:21:43.505295 18333 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:43.535492 17729 leader_election.cc:304] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:21:43.472153 17761 raft_consensus.cc:2468] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 6.
I20260709 12:21:43.537297 17729 leader_election.cc:304] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:21:43.487802 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 152065us
I20260709 12:21:43.509038 17801 leader_election.cc:304] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:43.348577 18374 raft_consensus.cc:1240] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 7. Current term is 8. Ops: []
I20260709 12:21:43.450284 20365 raft_consensus.cc:2749] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:21:43.456571 20407 raft_consensus.cc:2749] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:21:43.511106 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:43.499425 20324 raft_consensus.cc:493] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:43.549885 20324 raft_consensus.cc:3060] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:43.550367 20387 consensus_queue.cc:1059] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 12:21:43.517483 20332 raft_consensus.cc:2804] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:21:43.550374 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:43.552134 20332 raft_consensus.cc:493] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:21:43.499202 20295 raft_consensus.cc:2749] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:21:43.503669 20368 raft_consensus.cc:2749] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:21:43.503475 20349 raft_consensus.cc:2749] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:21:43.519671 17654 leader_election.cc:304] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:43.564797 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
W20260709 12:21:43.564981 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 45427.9us
I20260709 12:21:43.565961 17654 leader_election.cc:304] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:21:43.527139 20459 raft_consensus.cc:493] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:43.568531 20459 raft_consensus.cc:3060] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:21:43.569741 20227 raft_consensus.cc:2749] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:21:43.570626 20227 raft_consensus.cc:2804] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:21:43.571210 20227 raft_consensus.cc:697] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 5 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:21:43.572235 20377 consensus_queue.cc:1048] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:21:43.574554 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:43.513767 20447 raft_consensus.cc:2749] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:21:43.516566 20347 leader_election.cc:290] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 6 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:43.513964 20350 raft_consensus.cc:493] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:43.535701 17685 raft_consensus.cc:2393] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 9: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:21:43.541023 17801 leader_election.cc:304] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:21:43.583101 17801 leader_election.cc:304] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:21:43.511998 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 44331.3us
I20260709 12:21:43.510934 20452 consensus_queue.cc:1059] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.021s
I20260709 12:21:43.544301 20471 raft_consensus.cc:2749] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Leader election lost for term 5. Reason: Vote denied by peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837) with higher term. Message: Invalid argument: T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for earlier term 5. Current term is 6.
I20260709 12:21:43.536676 18333 leader_election.cc:304] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
W20260709 12:21:43.588194 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 51622.4us
I20260709 12:21:43.588320 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:43.527236 20443 raft_consensus.cc:740] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 4 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:21:43.529968 20385 raft_consensus.cc:493] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:43.484244 20234 raft_consensus.cc:697] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:21:43.601016 20385 raft_consensus.cc:3060] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:43.601706 20234 consensus_queue.cc:237] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:43.551909 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:43.553455 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:43.553683 17682 raft_consensus.cc:2410] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for term 7 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260709 12:21:43.603363 17689 raft_consensus.cc:2376] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 3. Re-sending same reply.
I20260709 12:21:43.519121 20480 raft_consensus.cc:740] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:21:43.559616 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:43.603929 17688 raft_consensus.cc:2368] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for earlier term 5. Current term is 6.
I20260709 12:21:43.604398 17687 raft_consensus.cc:2393] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 6: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:21:43.525604 20457 raft_consensus.cc:2804] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:21:43.555825 20298 raft_consensus.cc:2804] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:21:43.627472 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:43.629184 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:43.590206 18333 leader_election.cc:304] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:21:43.534798 18334 leader_election.cc:304] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:21:43.545176 20376 raft_consensus.cc:2804] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:21:43.598058 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:43.556272 19855 raft_consensus.cc:493] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:43.579676 20350 raft_consensus.cc:3060] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:43.560912 20332 raft_consensus.cc:3060] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:43.641490 17838 raft_consensus.cc:2410] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260709 12:21:43.588313 20447 raft_consensus.cc:2764] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20260709 12:21:43.598531 17654 leader_election.cc:304] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:21:43.553052 17652 leader_election.cc:304] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:21:43.645196 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 92371.5us
I20260709 12:21:43.557093 20336 raft_consensus.cc:2749] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:21:43.607224 20485 raft_consensus.cc:2804] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:21:43.632491 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:43.632570 18333 leader_election.cc:304] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:43.599428 20344 consensus_queue.cc:260] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:43.636842 17682 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44898 (request call id 588) took 2155 ms. Trace:
I20260709 12:21:43.627286 20364 raft_consensus.cc:493] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:43.638468 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:43.639376 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 8 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:43.594161 20296 raft_consensus.cc:2749] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:21:43.606006 20347 raft_consensus.cc:3055] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:21:43.605342 20405 consensus_queue.cc:1059] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.058s
I20260709 12:21:43.652769 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:43.649693 17652 leader_election.cc:304] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:21:43.629542 20457 raft_consensus.cc:493] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:21:43.654750 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 56344.7us
I20260709 12:21:43.685072 17654 leader_election.cc:304] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:43.635403 20298 raft_consensus.cc:493] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:43.634882 20376 raft_consensus.cc:493] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:43.637977 20466 raft_consensus.cc:2804] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:21:43.654930 20485 raft_consensus.cc:493] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:43.652433 20179 raft_consensus.cc:2749] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:21:43.699362 20384 raft_consensus.cc:2804] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:21:43.585106 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:43.605494 20284 consensus_queue.cc:1048] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.058s
I20260709 12:21:43.640815 19855 raft_consensus.cc:515] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:43.605166 20299 consensus_queue.cc:1048] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.059s
I20260709 12:21:43.656296 17689 raft_consensus.cc:2468] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 4.
I20260709 12:21:43.685972 17835 raft_consensus.cc:2410] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for term 5 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
W20260709 12:21:43.696987 17652 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:43.724521 20298 raft_consensus.cc:3060] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:43.675139 20347 raft_consensus.cc:740] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:21:43.652881 20491 consensus_queue.cc:1048] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.046s
I20260709 12:21:43.682610 20344 raft_consensus.cc:3060] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:43.684862 20457 raft_consensus.cc:3060] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:43.658128 17682 rpcz_store.cc:276] 0709 12:21:41.481684 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:41.481809 (+   125us) service_pool.cc:224] Handling call
0709 12:21:43.636582 (+2154773us) spinlock_profiling.cc:243] Waited 2.05 s on lock 0x7b54003480d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:21:43.636776 (+   194us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4723144064}
I20260709 12:21:43.701355 20458 raft_consensus.cc:2749] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:21:43.734570 17683 raft_consensus.cc:1240] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 8. Current term is 9. Ops: []
I20260709 12:21:43.704635 20405 consensus_queue.cc:1243] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer 0da169ff82674dc1947055ce43de68a3 log is divergent from this leader: its last log entry 4.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20260709 12:21:43.642153 20394 raft_consensus.cc:515] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:21:43.787425 20498 log.cc:927] Time spent T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.070s	user 0.000s	sys 0.000s
W20260709 12:21:43.761648 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 247836us
I20260709 12:21:43.726125 20466 raft_consensus.cc:493] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:43.788738 20466 raft_consensus.cc:3060] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:43.789240 20394 leader_election.cc:290] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 6 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:43.724701 20376 raft_consensus.cc:3060] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:43.727658 20485 raft_consensus.cc:3060] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:43.768963 17684 raft_consensus.cc:1275] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260709 12:21:43.680629 20364 raft_consensus.cc:515] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:43.728086 20144 raft_consensus.cc:493] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:21:43.674685 20227 consensus_queue.cc:237] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:43.684564 20480 consensus_queue.cc:260] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:21:43.744539 17832 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.529s	user 0.005s	sys 0.012s
I20260709 12:21:43.734331 20384 raft_consensus.cc:493] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:43.785531 17688 raft_consensus.cc:1275] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260709 12:21:43.786818 20458 raft_consensus.cc:2749] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:21:43.804296 17764 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38786 (request call id 2419) took 1037 ms. Trace:
I20260709 12:21:43.823454 17764 rpcz_store.cc:276] 0709 12:21:42.766586 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:42.815053 (+ 48467us) service_pool.cc:224] Handling call
0709 12:21:43.804058 (+989005us) spinlock_profiling.cc:243] Waited 910 ms on lock 0x7b54002b5a54. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:21:43.804220 (+   162us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2091886336}
I20260709 12:21:43.808743 17681 raft_consensus.cc:1275] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 7. (term mismatch)
I20260709 12:21:43.825461 17681 pending_rounds.cc:85] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3: Aborting all ops after (but not including) 6
I20260709 12:21:43.825735 17681 pending_rounds.cc:107] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3: Aborting uncommitted NO_OP operation due to leader change: 5.7
I20260709 12:21:43.826014 17681 raft_consensus.cc:2889] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:21:43.802206 17835 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35042 (request call id 1750) took 1080 ms. Trace:
I20260709 12:21:43.827669 17835 rpcz_store.cc:276] 0709 12:21:42.721678 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:42.741386 (+ 19708us) service_pool.cc:224] Handling call
0709 12:21:43.801911 (+1060525us) spinlock_profiling.cc:243] Waited 879 ms on lock 0x7b54000898d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:21:43.802132 (+   221us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2021132672}
I20260709 12:21:43.787007 20102 leader_election.cc:290] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:43.814524 17837 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442"
dest_uuid: "e063541542a94969898868bf6eb5ce1b"
mode: GRACEFUL
new_leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
 from {username='slave'} at 127.0.0.1:59820
I20260709 12:21:43.820428 17685 raft_consensus.cc:2393] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 6: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:21:43.821002 17758 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38776 (request call id 1795) took 1055 ms. Trace:
W20260709 12:21:43.863449 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:43.902606 17758 rpcz_store.cc:276] 0709 12:21:42.766048 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:43.145580 (+379532us) service_pool.cc:224] Handling call
0709 12:21:43.201805 (+ 56225us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:21:43.235454 (+ 33649us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:43.235495 (+    41us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:21:43.260506 (+ 25011us) log.cc:844] Serialized 52 byte log entry
0709 12:21:43.646171 (+385665us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:21:43.646207 (+    36us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:43.646268 (+    61us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:43.820717 (+174449us) raft_consensus.cc:1685] finished
0709 12:21:43.820747 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:43.820930 (+   183us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":45209,"raft.queue_time_us":46885,"raft.run_cpu_time_us":103,"raft.run_wall_time_us":104,"thread_start_us":385262,"threads_started":1,"wal-append.queue_time_us":548800}
I20260709 12:21:43.796020 20396 raft_consensus.cc:2749] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:21:43.824146 20384 raft_consensus.cc:3060] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:43.926393 17835 raft_consensus.cc:1275] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
W20260709 12:21:43.812852 20497 log.cc:927] Time spent T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.132s	user 0.000s	sys 0.000s
W20260709 12:21:43.822026 17832 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.606s	user 0.005s	sys 0.012s
I20260709 12:21:43.811935 20480 raft_consensus.cc:3060] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:43.796172 20179 consensus_queue.cc:1048] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.118s
I20260709 12:21:43.796103 20336 consensus_queue.cc:1059] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.115s
I20260709 12:21:43.798211 19855 leader_election.cc:290] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:43.895430 20157 raft_consensus.cc:697] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 6 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:21:43.897469 17680 raft_consensus.cc:2468] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 5.
W20260709 12:21:43.916380 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 381646us
I20260709 12:21:43.808346 20144 raft_consensus.cc:515] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:43.885408 20440 raft_consensus.cc:515] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:21:43.975142 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 230353us
I20260709 12:21:43.791870 20443 consensus_queue.cc:260] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:43.813776 20418 raft_consensus.cc:515] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:21:43.981127 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 441825us
I20260709 12:21:43.884682 20459 raft_consensus.cc:515] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:43.989167 17729 leader_election.cc:304] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:21:43.975394 20370 raft_consensus.cc:2804] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260709 12:21:44.004431 18334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:43.810048 20347 consensus_queue.cc:260] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.10, Last appended by leader: 10, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:44.006867 17686 raft_consensus.cc:2393] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 8: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
W20260709 12:21:44.034505 17728 leader_election.cc:336] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Timed out: RequestConsensusVote RPC to 127.16.29.65:33071 timed out after 1.626s (SENT)
I20260709 12:21:44.146961 17728 leader_election.cc:304] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
W20260709 12:21:44.039261 17730 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:44.153451 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 118974us
W20260709 12:21:44.154261 17728 leader_election.cc:336] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Timed out: RequestConsensusVote RPC to 127.16.29.65:33071 timed out after 1.629s (SENT)
W20260709 12:21:44.046379 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 361364us
I20260709 12:21:43.986442 20394 consensus_queue.cc:1048] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.052s
I20260709 12:21:44.005301 20364 leader_election.cc:290] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:44.014173 20443 raft_consensus.cc:3060] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:21:44.019992 20398 consensus_meta.cc:287] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.685s	user 0.010s	sys 0.001s
I20260709 12:21:43.997165 20284 consensus_queue.cc:1048] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.063s
I20260709 12:21:44.165550 17836 raft_consensus.cc:2468] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 4.
I20260709 12:21:44.166045 17837 raft_consensus.cc:606] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Received request to transfer leadership to TS e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:44.166436 17837 raft_consensus.cc:614] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:21:44.166857 20374 raft_consensus.cc:493] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:44.167364 20374 raft_consensus.cc:515] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:44.173993 17681 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37396 (request call id 1853) took 2679 ms. Trace:
I20260709 12:21:44.174424 17681 rpcz_store.cc:276] 0709 12:21:41.495083 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:42.360233 (+865150us) service_pool.cc:224] Handling call
0709 12:21:43.636739 (+1276506us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:43.826545 (+189806us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:44.173157 (+346612us) spinlock_profiling.cc:243] Waited 172 ms on lock 0x7b54003480d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:21:44.173780 (+   623us) spinlock_profiling.cc:243] Waited 1.19 s on lock 0x7b54003480d0. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a61f8ad 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:21:44.173931 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3129508864}
W20260709 12:21:44.072377 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 439866us
I20260709 12:21:44.049605 20347 raft_consensus.cc:3060] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:21:44.051326 19826 raft_consensus.cc:493] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:44.069589 20499 log.cc:927] Time spent T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.052s	user 0.001s	sys 0.000s
I20260709 12:21:44.055090 20512 raft_consensus.cc:2749] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260709 12:21:44.091189 20509 log.cc:927] Time spent T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.111s	user 0.001s	sys 0.001s
W20260709 12:21:44.102973 20508 log.cc:927] Time spent T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.066s	user 0.000s	sys 0.000s
I20260709 12:21:44.104249 20459 leader_election.cc:290] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 8 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:21:44.110522 20385 consensus_meta.cc:287] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.509s	user 0.010s	sys 0.001s
I20260709 12:21:44.160074 20516 raft_consensus.cc:2749] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:21:44.182096 20385 raft_consensus.cc:515] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:44.182705 19826 raft_consensus.cc:515] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:44.184350 20385 leader_election.cc:290] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 4 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:21:44.112432 20506 log.cc:927] Time spent T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.099s	user 0.000s	sys 0.000s
I20260709 12:21:44.184847 19826 leader_election.cc:290] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:44.160221 20485 raft_consensus.cc:515] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:44.185864 17830 raft_consensus.cc:1240] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 3. Current term is 4. Ops: [3.5-3.5]
W20260709 12:21:44.113487 20523 log.cc:927] Time spent T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.059s	user 0.000s	sys 0.000s
W20260709 12:21:44.114214 20507 log.cc:927] Time spent T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.144s	user 0.000s	sys 0.001s
I20260709 12:21:44.187234 20485 leader_election.cc:290] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 6 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:44.101321 20422 raft_consensus.cc:2804] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:21:44.121157 20440 leader_election.cc:290] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 8 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:44.115196 20350 raft_consensus.cc:515] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:44.103255 20332 raft_consensus.cc:515] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:44.101670 20346 raft_consensus.cc:493] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
W20260709 12:21:44.123498 20324 consensus_meta.cc:287] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.573s	user 0.007s	sys 0.000s
I20260709 12:21:44.092084 20527 consensus_queue.cc:1048] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.041s
W20260709 12:21:44.128688 20517 log.cc:927] Time spent T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.058s	user 0.000s	sys 0.000s
I20260709 12:21:44.123533 20418 leader_election.cc:290] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 8 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:21:44.127693 20397 consensus_meta.cc:287] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.616s	user 0.004s	sys 0.004s
I20260709 12:21:44.132804 20457 raft_consensus.cc:515] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:44.195617 20346 raft_consensus.cc:515] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:44.195925 20324 raft_consensus.cc:515] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:44.133795 20376 raft_consensus.cc:515] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:44.123548 20298 raft_consensus.cc:515] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:44.058071 20157 consensus_queue.cc:237] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:21:44.145814 17801 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.509s	user 0.003s	sys 0.016s
I20260709 12:21:44.005903 17729 leader_election.cc:304] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
W20260709 12:21:44.156520 17728 leader_election.cc:336] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Timed out: RequestConsensusVote RPC to 127.16.29.65:33071 timed out after 1.731s (SENT)
I20260709 12:21:44.156585 17654 leader_election.cc:304] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b
W20260709 12:21:44.161303 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 154061us
I20260709 12:21:44.176184 17689 raft_consensus.cc:1240] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 5. Current term is 6. Ops: []
I20260709 12:21:44.176525 18333 leader_election.cc:304] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:21:44.161427 17757 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38786 (request call id 2433) took 1306 ms. Trace:
I20260709 12:21:44.177286 20532 raft_consensus.cc:493] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:44.187779 17832 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37994 (request call id 629) took 1382 ms. Trace:
I20260709 12:21:44.179613 20384 raft_consensus.cc:515] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:21:44.259380 17801 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.623s	user 0.003s	sys 0.016s
W20260709 12:21:44.259981 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 254178us
W20260709 12:21:44.269745 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 633290us
I20260709 12:21:44.269927 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:44.270229 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:44.270807 17801 leader_election.cc:304] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:44.260185 17728 leader_election.cc:304] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
W20260709 12:21:44.266058 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 109615us
W20260709 12:21:44.268378 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 91926.4us
I20260709 12:21:44.273185 18333 leader_election.cc:304] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:21:44.270171 17729 leader_election.cc:304] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:21:44.270891 17759 raft_consensus.cc:3060] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:21:44.274276 18333 leader_election.cc:304] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:21:44.275863 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:44.276371 17764 raft_consensus.cc:1240] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 5. Current term is 6. Ops: [5.6-5.6]
I20260709 12:21:44.276633 17836 raft_consensus.cc:2468] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 5.
I20260709 12:21:44.278461 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:44.260803 17757 rpcz_store.cc:276] 0709 12:21:42.854686 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:43.173264 (+318578us) service_pool.cc:224] Handling call
0709 12:21:43.173696 (+   432us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:43.173963 (+   267us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 12:21:43.630051 (+456088us) log.cc:844] Serialized 14 byte log entry
0709 12:21:43.984123 (+354072us) log.cc:844] Serialized 14 byte log entry
0709 12:21:44.161006 (+176883us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:44.161063 (+    57us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:21:44.161123 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:44.161196 (+    73us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:44.161350 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":347,"prepare.queue_time_us":533029,"prepare.run_cpu_time_us":75,"prepare.run_wall_time_us":76,"raft.queue_time_us":47781,"raft.run_cpu_time_us":3745,"raft.run_wall_time_us":23460,"spinlock_wait_cycles":5816576,"thread_start_us":910789,"threads_started":3,"wal-append.queue_time_us":477644}
I20260709 12:21:44.198652 20324 leader_election.cc:290] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 4 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:21:44.282871 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 126966us
I20260709 12:21:44.284961 17762 raft_consensus.cc:1240] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 5. Current term is 6. Ops: [5.7-5.7]
I20260709 12:21:44.163625 20466 raft_consensus.cc:515] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:44.200470 20332 leader_election.cc:290] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 6 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:44.286254 17756 raft_consensus.cc:1240] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 5. Current term is 6. Ops: []
I20260709 12:21:44.286418 17654 leader_election.cc:304] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:21:44.277985 18333 leader_election.cc:304] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:21:44.262687 20376 leader_election.cc:290] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 6 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:44.282588 19855 raft_consensus.cc:2804] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:21:44.289104 19855 raft_consensus.cc:697] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 6 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:21:44.289788 20512 raft_consensus.cc:2749] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:21:44.199010 20346 leader_election.cc:290] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:44.290067 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 8 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:44.266242 20284 raft_consensus.cc:2804] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:21:44.293503 17654 leader_election.cc:304] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:21:44.304059 17654 leader_election.cc:304] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:21:44.305696 18365 raft_consensus.cc:1240] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 5. Current term is 6. Ops: [3.6-5.7]
I20260709 12:21:44.288168 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:44.307116 17758 raft_consensus.cc:3060] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:44.307256 18367 raft_consensus.cc:1275] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260709 12:21:44.298149 18333 leader_election.cc:304] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:21:44.299126 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:44.299396 17652 leader_election.cc:304] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:44.300891 20521 raft_consensus.cc:2804] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:21:44.267516 20516 raft_consensus.cc:2749] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:21:44.259040 20385 raft_consensus.cc:493] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:21:44.274583 20418 raft_consensus.cc:2764] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20260709 12:21:44.265552 20384 leader_election.cc:290] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 7 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:44.284473 20485 raft_consensus.cc:2804] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:21:44.261351 20532 raft_consensus.cc:515] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:44.297829 20394 raft_consensus.cc:493] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:44.316449 20385 raft_consensus.cc:515] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:44.312718 20476 raft_consensus.cc:2804] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:21:44.315441 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:44.346061 18368 raft_consensus.cc:3060] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:44.314792 18367 raft_consensus.cc:1275] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:21:44.317417 19855 consensus_queue.cc:237] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:44.318924 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
W20260709 12:21:44.350962 17728 leader_election.cc:336] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Timed out: RequestConsensusVote RPC to 127.16.29.65:33071 timed out after 1.943s (SENT)
W20260709 12:21:44.320456 17729 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:44.198087 20397 raft_consensus.cc:515] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:44.313190 18365 raft_consensus.cc:1240] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 5. Current term is 6. Ops: []
I20260709 12:21:44.326748 18374 raft_consensus.cc:2393] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 7: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:21:44.284220 20398 raft_consensus.cc:2749] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:21:44.316026 20457 leader_election.cc:290] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 4 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:44.332132 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
W20260709 12:21:44.334290 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 51805.9us
I20260709 12:21:44.334565 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:44.305112 20377 raft_consensus.cc:2804] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:21:44.341131 18333 leader_election.cc:304] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:21:44.311268 20521 raft_consensus.cc:493] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:21:44.312129 20284 raft_consensus.cc:493] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
W20260709 12:21:44.363912 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 59811.8us
I20260709 12:21:44.330899 20324 raft_consensus.cc:2804] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:21:44.301688 20459 consensus_queue.cc:1048] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.044s
W20260709 12:21:44.365986 17729 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:44.373919 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:44.388245 17836 raft_consensus.cc:2468] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 6.
I20260709 12:21:44.389237 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:44.389863 17836 raft_consensus.cc:3060] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:44.393939 18366 raft_consensus.cc:1275] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 12:21:44.393965 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:44.398159 18365 raft_consensus.cc:3060] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:44.349076 20418 consensus_queue.cc:1059] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.037s
I20260709 12:21:44.372917 20377 raft_consensus.cc:493] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:44.288193 20539 consensus_queue.cc:1048] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.067s
W20260709 12:21:44.378961 18333 connection.cc:256] RPC call timeout handler was delayed by -1.0022747640005036s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.23900000000000002s for a response
I20260709 12:21:44.351260 20476 raft_consensus.cc:493] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:44.385239 17687 raft_consensus.cc:1240] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 3. Current term is 4. Ops: []
I20260709 12:21:44.384124 20284 raft_consensus.cc:3060] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:21:44.399320 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 84940.7us
I20260709 12:21:44.386660 20324 raft_consensus.cc:493] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:44.379724 20521 raft_consensus.cc:3060] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:44.395466 20514 raft_consensus.cc:493] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:21:44.380427 20350 leader_election.cc:290] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 6 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:44.348692 20394 raft_consensus.cc:515] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:44.380702 20364 raft_consensus.cc:2749] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:21:44.351955 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:44.437762 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:44.376121 17758 raft_consensus.cc:2468] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 6.
I20260709 12:21:44.385624 20458 raft_consensus.cc:2804] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:21:44.408829 17761 raft_consensus.cc:1275] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20260709 12:21:44.452268 18365 raft_consensus.cc:2468] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 6.
I20260709 12:21:44.453567 17729 leader_election.cc:304] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:21:44.453716 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:44.420498 17801 leader_election.cc:304] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:44.422127 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:44.426641 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:44.456272 18371 raft_consensus.cc:2393] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 5: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:21:44.428208 20532 leader_election.cc:290] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:44.428560 20234 consensus_queue.cc:1048] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:21:44.428766 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:44.425065 20324 raft_consensus.cc:3060] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:44.429394 20512 consensus_queue.cc:1059] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 12:21:44.425227 20443 consensus_queue.cc:1048] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.005s
I20260709 12:21:44.430126 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:44.465296 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:44.405634 17755 raft_consensus.cc:1275] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 12:21:44.410507 20538 consensus_queue.cc:1059] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.033s
I20260709 12:21:44.410331 20377 raft_consensus.cc:3060] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:44.439543 17756 raft_consensus.cc:1240] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 5. Current term is 6. Ops: [2.2-5.3]
I20260709 12:21:44.412114 20476 raft_consensus.cc:3060] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:44.481200 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:44.481778 18367 raft_consensus.cc:3060] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:44.440209 17836 raft_consensus.cc:2468] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 7.
I20260709 12:21:44.411335 20398 raft_consensus.cc:493] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:21:44.406122 19826 consensus_queue.cc:1048] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.025s
I20260709 12:21:44.460121 18364 raft_consensus.cc:2393] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 5: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:21:44.429006 20385 leader_election.cc:290] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:44.500154 18367 raft_consensus.cc:2468] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 6.
I20260709 12:21:44.460951 18368 raft_consensus.cc:2468] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 6.
I20260709 12:21:44.502444 17729 leader_election.cc:304] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:21:44.461656 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 81009.6us
I20260709 12:21:44.454464 17835 raft_consensus.cc:2468] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 5.
I20260709 12:21:44.433344 20346 raft_consensus.cc:2804] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:21:44.468693 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:44.504781 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:44.504995 20485 raft_consensus.cc:697] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 3 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:21:44.505493 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:44.505988 20485 consensus_queue.cc:237] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:44.506573 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:44.261497 17832 rpcz_store.cc:276] 0709 12:21:42.805592 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:43.213875 (+408283us) service_pool.cc:224] Handling call
0709 12:21:43.214182 (+   307us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:43.214600 (+   418us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:43.214648 (+    48us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:43.215309 (+   661us) log.cc:844] Serialized 26 byte log entry
0709 12:21:43.967573 (+752264us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:21:43.970078 (+  2505us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:43.980045 (+  9967us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:44.187548 (+207503us) raft_consensus.cc:1685] finished
0709 12:21:44.187583 (+    35us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:44.187722 (+   139us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":20648,"raft.run_cpu_time_us":128,"raft.run_wall_time_us":128,"thread_start_us":722755,"threads_started":1,"wal-append.queue_time_us":748879}
I20260709 12:21:44.476904 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:44.479346 20298 leader_election.cc:290] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 6 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:44.452365 20458 raft_consensus.cc:493] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:44.507663 20458 raft_consensus.cc:3060] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:44.507859 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:44.508008 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:44.508733 17761 raft_consensus.cc:2393] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 6: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:21:44.509436 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:44.509756 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:44.510401 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:44.482686 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:44.458750 18361 raft_consensus.cc:3060] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:44.467715 20381 raft_consensus.cc:3055] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:21:44.491228 18363 raft_consensus.cc:1275] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 12:21:44.492053 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:44.525301 17802 leader_election.cc:304] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:21:44.526273 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:44.354749 20384 raft_consensus.cc:2955] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 7 LEADER]: Committing config change with OpId 3.5: config changed from index 4 to 5, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
W20260709 12:21:44.464993 20540 log.cc:927] Time spent T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.072s	user 0.000s	sys 0.001s
I20260709 12:21:44.492791 20296 raft_consensus.cc:493] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:21:44.501423 20295 raft_consensus.cc:493] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:44.515268 17831 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35040 (request call id 1979) took 1254 ms. Trace:
I20260709 12:21:44.542932 18334 leader_election.cc:304] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:21:44.515064 20458 raft_consensus.cc:515] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:44.462421 20550 consensus_queue.cc:1059] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.079s
I20260709 12:21:44.518353 17729 leader_election.cc:304] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:21:44.522420 20344 consensus_meta.cc:287] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.710s	user 0.009s	sys 0.003s
I20260709 12:21:44.519907 17801 leader_election.cc:304] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:44.520453 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:44.468825 20350 raft_consensus.cc:2804] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:21:44.524324 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:44.437362 20542 consensus_queue.cc:1048] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.068s
I20260709 12:21:44.434000 20514 raft_consensus.cc:515] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:44.527724 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:44.551177 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:44.533603 18367 raft_consensus.cc:1275] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 6. (term mismatch)
I20260709 12:21:44.552346 18367 pending_rounds.cc:85] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da: Aborting all ops after (but not including) 5
I20260709 12:21:44.542795 17831 rpcz_store.cc:276] 0709 12:21:43.260451 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:43.465390 (+204939us) service_pool.cc:224] Handling call
0709 12:21:43.465686 (+   296us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:43.507523 (+ 41837us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 12:21:43.507554 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:43.514025 (+  6471us) log.cc:844] Serialized 26 byte log entry
0709 12:21:43.907529 (+393504us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:21:43.907587 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:44.514892 (+607305us) spinlock_profiling.cc:243] Waited 19.3 ms on lock 0x7b5400266454. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:21:44.514935 (+    43us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:44.514983 (+    48us) raft_consensus.cc:1685] finished
0709 12:21:44.514996 (+    13us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:44.515215 (+   219us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":15356,"raft.queue_time_us":22729,"raft.run_cpu_time_us":113,"raft.run_wall_time_us":4291,"spinlock_wait_cycles":44305280,"thread_start_us":366862,"threads_started":1,"wal-append.queue_time_us":492074}
I20260709 12:21:44.533571 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:44.524802 20512 raft_consensus.cc:2804] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:21:44.540307 17728 leader_election.cc:304] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:21:44.540966 20539 consensus_queue.cc:1059] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 12:21:44.541812 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:44.465848 20364 consensus_queue.cc:1048] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.056s
I20260709 12:21:44.544001 17758 raft_consensus.cc:2410] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for term 9 because replica has last-logged OpId of term: 8 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 9.
I20260709 12:21:44.553146 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:44.572014 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:44.572773 17831 raft_consensus.cc:3060] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:44.552575 18367 pending_rounds.cc:107] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da: Aborting uncommitted NO_OP operation due to leader change: 4.6
I20260709 12:21:44.573541 18367 raft_consensus.cc:2889] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:21:44.574116 17835 raft_consensus.cc:2468] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 6.
I20260709 12:21:44.552901 18333 leader_election.cc:304] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:44.554086 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:44.574585 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:44.468408 20534 raft_consensus.cc:2749] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:21:44.537271 20385 raft_consensus.cc:2804] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:21:44.518033 20398 raft_consensus.cc:515] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:44.584003 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:44.551393 17764 raft_consensus.cc:2468] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 6.
I20260709 12:21:44.585079 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:44.551867 20532 raft_consensus.cc:2804] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:21:44.551419 20295 raft_consensus.cc:515] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:44.557065 20299 raft_consensus.cc:3055] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:21:44.558033 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:44.600059 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:44.555675 20447 raft_consensus.cc:2749] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:21:44.559101 18363 raft_consensus.cc:1275] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 9 index: 9. (term mismatch)
I20260709 12:21:44.601413 18363 pending_rounds.cc:85] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da: Aborting all ops after (but not including) 8
I20260709 12:21:44.601758 18363 pending_rounds.cc:107] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da: Aborting uncommitted NO_OP operation due to leader change: 8.9
I20260709 12:21:44.602077 18363 raft_consensus.cc:2889] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:21:44.525344 20466 leader_election.cc:290] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 7 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:44.559890 20397 leader_election.cc:290] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 4 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:44.560779 20458 leader_election.cc:290] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 4 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:44.561519 18366 raft_consensus.cc:2393] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 4: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:21:44.525555 20381 raft_consensus.cc:740] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 5 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:21:44.576287 20350 raft_consensus.cc:697] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 6 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:21:44.553328 18334 leader_election.cc:304] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:44.554878 18368 raft_consensus.cc:3060] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:44.571848 17728 leader_election.cc:304] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:21:44.580986 20550 raft_consensus.cc:493] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:44.581435 17836 raft_consensus.cc:3060] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:44.551100 20365 raft_consensus.cc:493] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:44.595719 20532 raft_consensus.cc:493] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:44.594980 19855 raft_consensus.cc:2749] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:21:44.585757 17761 raft_consensus.cc:2393] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 6: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:21:44.551978 20347 raft_consensus.cc:2804] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:21:44.586266 17831 raft_consensus.cc:2468] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 4.
I20260709 12:21:44.598822 20299 raft_consensus.cc:740] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
W20260709 12:21:44.594430 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 216211us
I20260709 12:21:44.600816 17838 raft_consensus.cc:2376] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 4. Re-sending same reply.
I20260709 12:21:44.650529 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:44.651262 17838 raft_consensus.cc:2468] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 5.
I20260709 12:21:44.593369 18333 leader_election.cc:304] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:44.652343 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:44.605666 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:44.574301 17759 raft_consensus.cc:2468] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 8.
I20260709 12:21:44.545468 20296 raft_consensus.cc:515] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:44.627252 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:21:44.630287 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 58524.6us
I20260709 12:21:44.577308 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:44.635571 20443 raft_consensus.cc:2749] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:21:44.584658 20512 raft_consensus.cc:697] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 6 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:21:44.583365 20234 raft_consensus.cc:2749] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260709 12:21:44.639824 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 86551.9us
I20260709 12:21:44.578536 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:44.710812 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:44.647167 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:44.681830 20559 raft_consensus.cc:2749] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 12:21:44.648224 17688 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37396 (request call id 1875) took 1199 ms. Trace:
I20260709 12:21:44.654556 18368 raft_consensus.cc:2468] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 4.
I20260709 12:21:44.714202 17688 rpcz_store.cc:276] 0709 12:21:43.448741 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:43.680641 (+231900us) service_pool.cc:224] Handling call
0709 12:21:43.706953 (+ 26312us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:43.822500 (+115547us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:44.648004 (+825504us) spinlock_profiling.cc:243] Waited 69.8 ms on lock 0x7b54000c6054. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:21:44.648170 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":160552320}
I20260709 12:21:44.715224 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:44.715535 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:21:44.657734 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 64486us
I20260709 12:21:44.717048 18333 leader_election.cc:304] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:44.717765 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:44.718386 18333 leader_election.cc:304] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:44.606168 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:44.720249 17684 raft_consensus.cc:2468] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 3.
I20260709 12:21:44.673599 17728 leader_election.cc:304] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:21:44.623229 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:44.673830 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:44.673911 20397 consensus_queue.cc:1059] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 5.5, Next index: 6, Last known committed idx: 3, Time since last communication: 0.038s
I20260709 12:21:44.676391 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:44.666823 17838 raft_consensus.cc:2368] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 4. Current term is 5.
I20260709 12:21:44.622641 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:44.734925 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:44.607611 20385 raft_consensus.cc:697] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 5 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:21:44.687256 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:44.736125 17835 raft_consensus.cc:3060] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Advancing to term 6
I20260709 12:21:44.687422 17652 leader_election.cc:304] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:21:44.689623 17836 raft_consensus.cc:2468] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 6.
I20260709 12:21:44.656111 20532 raft_consensus.cc:3060] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:44.682682 17832 raft_consensus.cc:2410] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for term 8 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 5.
I20260709 12:21:44.660573 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:44.657637 20485 raft_consensus.cc:2749] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:21:44.613481 20284 raft_consensus.cc:515] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:44.713557 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 8 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:44.743901 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
W20260709 12:21:44.744298 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 56962.6us
I20260709 12:21:44.744575 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:44.647063 20365 raft_consensus.cc:515] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:44.718407 17688 raft_consensus.cc:2468] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 6.
I20260709 12:21:44.746472 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:44.747107 17838 raft_consensus.cc:2468] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 4.
I20260709 12:21:44.721648 18333 leader_election.cc:304] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:44.748500 17682 raft_consensus.cc:2393] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 6: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:21:44.751205 18333 leader_election.cc:400] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 4 election: Vote denied by peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837) with higher term. Message: Invalid argument: T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 4. Current term is 5.
I20260709 12:21:44.722637 17685 raft_consensus.cc:1240] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 7. Current term is 8. Ops: [5.4-7.5]
I20260709 12:21:44.645651 20550 raft_consensus.cc:515] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:44.674984 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:44.757577 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:44.758147 17761 raft_consensus.cc:3060] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:44.751765 18333 leader_election.cc:403] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
W20260709 12:21:44.731263 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 57783.1us
I20260709 12:21:44.760835 18333 leader_election.cc:304] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:44.762362 17728 leader_election.cc:304] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:21:44.747956 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:44.755964 17831 raft_consensus.cc:2393] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 4: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:21:44.725378 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:44.765750 17838 raft_consensus.cc:3060] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:44.706360 20443 raft_consensus.cc:493] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:21:44.734149 17680 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 4.
I20260709 12:21:44.773571 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:44.740556 20480 raft_consensus.cc:2804] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:21:44.778446 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:44.791415 17761 raft_consensus.cc:2468] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 7.
I20260709 12:21:44.783461 17835 raft_consensus.cc:2468] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 6.
I20260709 12:21:44.734367 20447 raft_consensus.cc:493] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:44.788790 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:44.792981 18333 leader_election.cc:304] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:21:44.774060 19855 raft_consensus.cc:2804] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:21:44.734895 20559 raft_consensus.cc:493] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:44.794095 19855 raft_consensus.cc:493] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:44.690207 20381 consensus_queue.cc:260] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:44.794464 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:44.757261 20539 raft_consensus.cc:2804] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:21:44.793650 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:44.757015 20234 raft_consensus.cc:2749] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:21:44.799722 17683 raft_consensus.cc:1240] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260709 12:21:44.802234 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:44.802963 18363 raft_consensus.cc:2468] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 3.
I20260709 12:21:44.797727 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:44.773773 20534 raft_consensus.cc:493] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:21:44.790028 20485 raft_consensus.cc:2749] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837) with higher term. Message: Invalid argument: T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 4. Current term is 5.
I20260709 12:21:44.812422 17835 raft_consensus.cc:3060] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:44.807744 17838 raft_consensus.cc:2468] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 7.
I20260709 12:21:44.722576 20350 consensus_queue.cc:237] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:21:44.800812 20556 log.cc:927] Time spent T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.090s	user 0.001s	sys 0.001s
I20260709 12:21:44.796325 19855 raft_consensus.cc:3060] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:44.825665 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:44.817972 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:44.819501 18363 raft_consensus.cc:1240] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 3. Current term is 4. Ops: [3.5-3.5]
I20260709 12:21:44.778293 20538 consensus_queue.cc:1048] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.034s
I20260709 12:21:44.779369 20443 raft_consensus.cc:515] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:44.822639 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:44.828189 17763 raft_consensus.cc:1240] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 5. Current term is 6. Ops: []
I20260709 12:21:44.828125 20512 consensus_queue.cc:237] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:44.789460 20555 consensus_queue.cc:1048] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.042s
I20260709 12:21:44.723066 20299 consensus_queue.cc:260] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:44.812566 20381 raft_consensus.cc:3060] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:44.835940 20485 raft_consensus.cc:2804] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:21:44.827019 20569 raft_consensus.cc:2749] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:21:44.813633 20567 raft_consensus.cc:2749] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260709 12:21:44.837004 20561 log.cc:927] Time spent T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.188s	user 0.001s	sys 0.000s
I20260709 12:21:44.835780 20385 consensus_queue.cc:237] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:44.837193 20476 raft_consensus.cc:515] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:21:44.837783 20558 log.cc:927] Time spent T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.126s	user 0.000s	sys 0.001s
I20260709 12:21:44.838085 20534 raft_consensus.cc:515] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:44.840065 20550 leader_election.cc:290] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:44.842259 19855 raft_consensus.cc:515] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:44.800621 20559 raft_consensus.cc:515] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:44.845923 17835 raft_consensus.cc:2410] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for term 6 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260709 12:21:44.845738 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:44.879418 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:44.846858 18334 leader_election.cc:304] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:21:44.879331 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 134239us
I20260709 12:21:44.848851 17685 raft_consensus.cc:1275] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 12:21:44.814574 20539 raft_consensus.cc:697] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 6 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:21:44.827064 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:44.833326 20295 leader_election.cc:290] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:44.818181 20453 consensus_queue.cc:1048] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.039s
I20260709 12:21:44.862403 20299 raft_consensus.cc:3060] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:44.863006 20365 leader_election.cc:290] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:21:44.866202 17760 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38786 (request call id 2426) took 2097 ms (client timeout 1862 ms). Trace:
I20260709 12:21:44.797150 20447 raft_consensus.cc:515] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:21:44.872416 20563 log.cc:927] Time spent T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.129s	user 0.000s	sys 0.001s
I20260709 12:21:44.884652 17835 raft_consensus.cc:1275] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260709 12:21:44.908499 20539 consensus_queue.cc:237] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:44.909358 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:44.910212 19855 leader_election.cc:290] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 6 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:44.887080 18334 leader_election.cc:304] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:21:44.877872 20296 leader_election.cc:290] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:21:44.901435 17760 rpcz_store.cc:269] 0709 12:21:42.768498 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:42.816403 (+ 47905us) service_pool.cc:224] Handling call
0709 12:21:44.865972 (+2049569us) spinlock_profiling.cc:243] Waited 1.01 s on lock 0x7b54001b8354. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:21:44.866136 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2315857280}
W20260709 12:21:44.894209 17729 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:44.876561 20458 raft_consensus.cc:2749] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:21:44.910849 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:44.911914 18334 leader_election.cc:304] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:44.921008 18334 leader_election.cc:304] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:44.861652 20514 leader_election.cc:290] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:44.914073 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:44.915727 17652 leader_election.cc:304] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:21:44.917905 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:44.885856 20284 leader_election.cc:290] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 6 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:44.925468 18334 leader_election.cc:304] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:44.933003 20559 leader_election.cc:290] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:44.935048 20534 leader_election.cc:290] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:44.935153 17832 raft_consensus.cc:1240] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 5. Current term is 6. Ops: []
I20260709 12:21:44.938868 18334 leader_election.cc:304] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:44.916918 19855 raft_consensus.cc:2749] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 12:21:44.890497 20542 consensus_queue.cc:1048] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.051s
I20260709 12:21:44.954296 20443 leader_election.cc:290] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:44.956117 17835 raft_consensus.cc:1275] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260709 12:21:44.944264 20559 raft_consensus.cc:2749] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:21:44.924345 20397 consensus_queue.cc:1048] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.034s
I20260709 12:21:44.985149 17830 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35040 (request call id 1976) took 1810 ms. Trace:
I20260709 12:21:44.985610 17830 rpcz_store.cc:276] 0709 12:21:43.174429 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:43.436935 (+262506us) service_pool.cc:224] Handling call
0709 12:21:43.456204 (+ 19269us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:44.186226 (+730022us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:44.984888 (+798662us) spinlock_profiling.cc:243] Waited 729 ms on lock 0x7b5400266e54. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:21:44.985088 (+   200us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1677517056}
I20260709 12:21:44.962973 19855 raft_consensus.cc:2749] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:21:45.010298 17829 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35040 (request call id 1974) took 2008 ms. Trace:
W20260709 12:21:44.988379 20324 consensus_meta.cc:287] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.512s	user 0.012s	sys 0.000s
I20260709 12:21:44.976603 20347 raft_consensus.cc:697] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 6 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:21:45.020885 20347 consensus_queue.cc:237] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:45.021601 20447 leader_election.cc:290] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:45.022390 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:45.017421 20572 raft_consensus.cc:2804] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:21:45.010813 17829 rpcz_store.cc:276] 0709 12:21:43.002035 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:43.428994 (+426959us) service_pool.cc:224] Handling call
0709 12:21:43.429296 (+   302us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:43.505845 (+ 76549us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 12:21:43.914834 (+408989us) log.cc:844] Serialized 14 byte log entry
0709 12:21:44.159026 (+244192us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:44.159105 (+    79us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:21:44.159177 (+    72us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:44.985066 (+825889us) spinlock_profiling.cc:243] Waited 29.4 ms on lock 0x7b54003cc8d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:21:45.001529 (+ 16463us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:45.010225 (+  8696us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":34251,"prepare.queue_time_us":501145,"prepare.run_cpu_time_us":66,"prepare.run_wall_time_us":3807,"raft.queue_time_us":92,"raft.run_cpu_time_us":110,"raft.run_wall_time_us":111,"spinlock_wait_cycles":67651840,"thread_start_us":510405,"threads_started":2,"wal-append.queue_time_us":233993}
I20260709 12:21:45.019263 20324 raft_consensus.cc:515] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:45.027184 20574 raft_consensus.cc:2749] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260709 12:21:45.037350 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 121741us
I20260709 12:21:45.039992 20485 raft_consensus.cc:697] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 7 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:21:45.041357 20485 consensus_queue.cc:237] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:45.054631 17833 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35040 (request call id 1985) took 1733 ms. Trace:
I20260709 12:21:45.057653 17833 rpcz_store.cc:276] 0709 12:21:43.320834 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:43.530786 (+209952us) service_pool.cc:224] Handling call
0709 12:21:43.531123 (+   337us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:43.549115 (+ 17992us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 12:21:43.766514 (+217399us) log.cc:844] Serialized 14 byte log entry
0709 12:21:44.054844 (+288330us) log.cc:844] Serialized 14 byte log entry
0709 12:21:44.055028 (+   184us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:44.055076 (+    48us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:21:44.055117 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:45.053741 (+998624us) spinlock_profiling.cc:243] Waited 3.09 ms on lock 0x7b1800133de0. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b475012a6 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419
0709 12:21:45.054337 (+   596us) spinlock_profiling.cc:243] Waited 17.7 ms on lock 0x7b54003340d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:21:45.054391 (+    54us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:45.054557 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":43366,"prepare.queue_time_us":499411,"raft.queue_time_us":1994,"raft.run_cpu_time_us":306,"raft.run_wall_time_us":42113,"spinlock_wait_cycles":62025728,"thread_start_us":460762,"threads_started":2,"wal-append.queue_time_us":369670}
I20260709 12:21:45.061709 17652 leader_election.cc:304] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:21:45.063226 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:45.063853 17682 raft_consensus.cc:2410] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for term 6 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20260709 12:21:45.064090 17830 raft_consensus.cc:3060] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:45.052827 20394 leader_election.cc:290] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:45.024816 20440 raft_consensus.cc:2764] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Leader election decision vote started in defunct term 4: won
I20260709 12:21:45.011112 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:45.060230 20476 leader_election.cc:290] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 6 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:45.101686 17833 pending_rounds.cc:85] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b: Aborting all ops after (but not including) 2
I20260709 12:21:45.060081 20572 raft_consensus.cc:697] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 7 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:21:45.163460 17833 pending_rounds.cc:107] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b: Aborting uncommitted NO_OP operation due to leader change: 3.3
I20260709 12:21:45.164798 17833 raft_consensus.cc:2889] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:21:45.168648 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 107023us
I20260709 12:21:45.174206 17652 leader_election.cc:304] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:45.190033 20179 raft_consensus.cc:2749] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 12:21:45.213511 17830 raft_consensus.cc:2468] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 6.
I20260709 12:21:45.167824 20324 leader_election.cc:290] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 5 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:45.163444 17832 raft_consensus.cc:2468] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 5.
I20260709 12:21:45.118494 20480 raft_consensus.cc:697] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 4 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:21:45.183990 20583 raft_consensus.cc:2804] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Leader election won for term 4
W20260709 12:21:45.228488 18361 consensus_meta.cc:287] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.700s	user 0.012s	sys 0.000s
W20260709 12:21:45.284581 20377 consensus_meta.cc:287] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.783s	user 0.012s	sys 0.000s
I20260709 12:21:45.300495 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:21:45.302369 18367 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.713s	user 0.026s	sys 0.044s
I20260709 12:21:45.263561 20583 raft_consensus.cc:697] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 4 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:21:45.263713 17728 leader_election.cc:304] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:21:45.312307 18367 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.723s	user 0.026s	sys 0.044s
I20260709 12:21:45.275974 20571 raft_consensus.cc:493] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:21:45.243093 20476 raft_consensus.cc:493] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:45.347919 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 84277.8us
I20260709 12:21:45.273419 20538 consensus_queue.cc:1048] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.043s
I20260709 12:21:45.391814 20452 consensus_queue.cc:1059] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:21:45.278750 20572 consensus_queue.cc:237] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:45.393146 20512 raft_consensus.cc:3055] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:21:45.393539 20512 raft_consensus.cc:740] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:21:45.303580 20377 raft_consensus.cc:515] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:45.394724 17681 raft_consensus.cc:2393] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 4: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:21:45.392158 17831 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35042 (request call id 1761) took 1135 ms. Trace:
I20260709 12:21:45.397966 17831 rpcz_store.cc:276] 0709 12:21:44.256819 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:44.794977 (+538158us) service_pool.cc:224] Handling call
0709 12:21:44.812372 (+ 17395us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:44.812749 (+   377us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:44.812780 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:44.813271 (+   491us) log.cc:844] Serialized 26 byte log entry
0709 12:21:45.285379 (+472108us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:21:45.285444 (+    65us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:45.285522 (+    78us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:45.391950 (+106428us) raft_consensus.cc:1685] finished
0709 12:21:45.391971 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:45.392099 (+   128us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":721,"raft.run_cpu_time_us":168,"raft.run_wall_time_us":11283,"thread_start_us":432581,"threads_started":1,"wal-append.queue_time_us":561934}
I20260709 12:21:45.391601 17728 leader_election.cc:304] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:21:45.304811 20574 raft_consensus.cc:493] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:21:45.292220 18361 raft_consensus.cc:2468] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 6.
I20260709 12:21:45.318856 20578 raft_consensus.cc:493] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:45.432880 18365 raft_consensus.cc:1240] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 5. Current term is 6. Ops: [5.6-5.6]
W20260709 12:21:45.438453 18371 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.552s	user 0.025s	sys 0.010s
W20260709 12:21:45.443076 18371 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.557s	user 0.026s	sys 0.011s
I20260709 12:21:45.437602 20567 raft_consensus.cc:2749] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:21:45.426196 20550 raft_consensus.cc:493] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:45.396529 20573 raft_consensus.cc:2804] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:21:45.428658 20350 raft_consensus.cc:2749] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:21:45.431362 20453 consensus_queue.cc:1048] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.013s
I20260709 12:21:45.449987 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:45.443456 20584 raft_consensus.cc:493] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:45.455180 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:45.424329 20476 raft_consensus.cc:515] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:45.447593 18361 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59886 (request call id 852) took 1122 ms. Trace:
I20260709 12:21:45.394348 20512 consensus_queue.cc:260] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:45.451030 20573 raft_consensus.cc:493] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:21:45.469168 20521 consensus_meta.cc:287] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 1.033s	user 0.013s	sys 0.000s
I20260709 12:21:45.322824 20571 raft_consensus.cc:515] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:45.433533 20574 raft_consensus.cc:515] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:21:45.519980 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:45.443332 20480 consensus_queue.cc:237] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:45.442759 20578 raft_consensus.cc:515] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:21:45.529115 17730 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:45.506737 17803 leader_election.cc:304] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:45.453356 20550 raft_consensus.cc:515] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:45.496206 20577 raft_consensus.cc:493] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:45.461702 20584 raft_consensus.cc:515] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:45.493364 18361 rpcz_store.cc:276] 0709 12:21:44.321206 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:44.326374 (+  5168us) service_pool.cc:224] Handling call
0709 12:21:45.443174 (+1116800us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:45.508100 20336 raft_consensus.cc:493] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:45.562911 20336 raft_consensus.cc:515] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:45.564991 18361 raft_consensus.cc:1240] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 5. Current term is 6. Ops: [3.6-5.7]
I20260709 12:21:45.500088 20573 raft_consensus.cc:3060] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:45.565353 20336 leader_election.cc:290] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:45.566427 20584 leader_election.cc:290] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:45.508764 20385 raft_consensus.cc:493] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:21:45.567155 20385 raft_consensus.cc:515] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:45.532999 20512 raft_consensus.cc:3060] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:21:45.569227 18368 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59886 (request call id 858) took 1017 ms. Trace:
I20260709 12:21:45.569602 18368 rpcz_store.cc:276] 0709 12:21:44.551825 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:44.792133 (+240308us) service_pool.cc:224] Handling call
0709 12:21:44.808674 (+ 16541us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:21:45.041851 (+233177us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:45.041885 (+    34us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:21:45.044335 (+  2450us) log.cc:844] Serialized 52 byte log entry
0709 12:21:45.303517 (+259182us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:21:45.303559 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:45.303634 (+    75us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:45.569026 (+265392us) raft_consensus.cc:1685] finished
0709 12:21:45.569044 (+    18us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:45.569171 (+   127us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":55062,"raft.run_cpu_time_us":72,"raft.run_wall_time_us":73,"thread_start_us":491427,"threads_started":2,"wal-append.queue_time_us":410188}
I20260709 12:21:45.462634 20583 consensus_queue.cc:237] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:45.572352 20377 leader_election.cc:290] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 4 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:45.572912 18368 raft_consensus.cc:1240] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 5. Current term is 6. Ops: [5.7-5.7]
W20260709 12:21:45.573740 20582 log.cc:927] Time spent T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.127s	user 0.000s	sys 0.000s
I20260709 12:21:45.553081 20453 raft_consensus.cc:493] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:45.552788 20324 raft_consensus.cc:493] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:45.557639 20585 raft_consensus.cc:493] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:45.596755 20550 leader_election.cc:290] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:45.561124 20577 raft_consensus.cc:515] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:45.598202 20476 leader_election.cc:290] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:45.600226 20571 leader_election.cc:290] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:45.605427 20578 leader_election.cc:290] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:45.606305 20385 leader_election.cc:290] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:45.607019 20385 raft_consensus.cc:2749] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:21:45.519134 20521 raft_consensus.cc:515] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:45.594020 20453 raft_consensus.cc:515] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:45.609179 18363 raft_consensus.cc:1240] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 6. Current term is 7. Ops: []
I20260709 12:21:45.609988 20585 raft_consensus.cc:515] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:45.610697 20521 leader_election.cc:290] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 7 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:45.599684 20577 leader_election.cc:290] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:45.622822 20453 leader_election.cc:290] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:45.611078 20573 raft_consensus.cc:515] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:45.654220 20573 leader_election.cc:290] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 7 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:21:45.655803 20532 consensus_meta.cc:287] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.882s	user 0.014s	sys 0.000s
I20260709 12:21:45.656523 20532 raft_consensus.cc:515] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:45.659108 20585 leader_election.cc:290] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:45.525660 17652 leader_election.cc:304] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
W20260709 12:21:45.552573 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 218582us
W20260709 12:21:45.695333 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 169640us
W20260709 12:21:45.696542 17727 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:45.699293 17727 leader_election.cc:304] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
W20260709 12:21:45.625790 20347 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.601s	user 0.067s	sys 0.044s
W20260709 12:21:45.708343 20347 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.684s	user 0.067s	sys 0.045s
I20260709 12:21:45.712229 20347 raft_consensus.cc:2749] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:21:45.713824 20179 raft_consensus.cc:2804] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:21:45.714623 20179 raft_consensus.cc:697] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 8 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:21:45.715058 18371 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 990) took 1112 ms. Trace:
I20260709 12:21:45.608889 20324 raft_consensus.cc:515] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:45.720187 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:45.720427 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:45.721704 17758 raft_consensus.cc:2468] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 3.
I20260709 12:21:45.776257 20179 consensus_queue.cc:237] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:45.787489 20577 raft_consensus.cc:493] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:45.788667 20577 raft_consensus.cc:515] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:45.794667 20573 raft_consensus.cc:493] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:45.795572 20573 raft_consensus.cc:515] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:45.798275 20573 leader_election.cc:290] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:45.809293 20324 leader_election.cc:290] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:45.812196 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:45.813448 17758 raft_consensus.cc:2410] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for term 6 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20260709 12:21:45.814504 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:45.815196 17758 raft_consensus.cc:2468] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 4.
I20260709 12:21:45.838438 17652 leader_election.cc:304] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:21:45.839979 17652 leader_election.cc:304] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:21:45.840963 17686 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37396 (request call id 1877) took 2391 ms (client timeout 1777 ms). Trace:
W20260709 12:21:45.841549 17686 rpcz_store.cc:269] 0709 12:21:43.449210 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:43.510834 (+ 61624us) service_pool.cc:224] Handling call
0709 12:21:45.840386 (+2329552us) spinlock_profiling.cc:243] Waited 458 ms on lock 0x7b5400206f54. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:21:45.840707 (+   321us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1052835584}
I20260709 12:21:45.843421 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:45.844925 20532 leader_election.cc:290] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 5 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:45.848904 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:45.849480 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:45.849924 17833 raft_consensus.cc:3060] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:45.850327 17834 raft_consensus.cc:3060] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Advancing to term 7
I20260709 12:21:45.848945 17830 raft_consensus.cc:1275] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260709 12:21:45.859863 17830 raft_consensus.cc:1275] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260709 12:21:45.715655 18371 rpcz_store.cc:276] 0709 12:21:44.602119 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:44.874411 (+272292us) service_pool.cc:224] Handling call
0709 12:21:44.874731 (+   320us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:21:44.875056 (+   325us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:44.875074 (+    18us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:21:44.878054 (+  2980us) log.cc:844] Serialized 52 byte log entry
0709 12:21:45.445996 (+567942us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:21:45.713756 (+267760us) log.cc:844] Serialized 14 byte log entry
0709 12:21:45.714323 (+   567us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:45.714460 (+   137us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:45.714540 (+    80us) raft_consensus.cc:1685] finished
0709 12:21:45.714555 (+    15us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:45.714818 (+   263us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":799,"raft.queue_time_us":39347,"raft.run_cpu_time_us":140,"raft.run_wall_time_us":7536,"thread_start_us":806197,"threads_started":2,"wal-append.queue_time_us":684635}
I20260709 12:21:45.891155 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:45.892270 17687 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37402 (request call id 2275) took 2458 ms. Trace:
I20260709 12:21:45.892740 17689 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37402 (request call id 2277) took 2299 ms. Trace:
I20260709 12:21:45.893157 17689 rpcz_store.cc:276] 0709 12:21:43.593412 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:43.743642 (+150230us) service_pool.cc:224] Handling call
0709 12:21:44.176009 (+432367us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:44.260643 (+ 84634us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:45.892349 (+1631706us) spinlock_profiling.cc:243] Waited 432 ms on lock 0x7b54000c5b50. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a61f8ad 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:21:45.892583 (+   234us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":993561344}
I20260709 12:21:45.894694 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:45.894912 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:45.896409 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:45.897953 17686 raft_consensus.cc:2410] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for term 5 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260709 12:21:45.898980 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:45.901482 17757 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36572 (request call id 549) took 1512 ms. Trace:
I20260709 12:21:45.902045 17757 rpcz_store.cc:276] 0709 12:21:44.388983 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:44.526058 (+137075us) service_pool.cc:224] Handling call
0709 12:21:45.900943 (+1374885us) spinlock_profiling.cc:243] Waited 869 ms on lock 0x7b54003e53d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:21:45.901237 (+   294us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1997723904}
I20260709 12:21:45.904927 20324 raft_consensus.cc:2804] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:21:45.905625 20324 raft_consensus.cc:493] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:45.906073 20324 raft_consensus.cc:3060] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:45.909014 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:45.910442 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:45.915759 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:21:45.917250 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 410506us
I20260709 12:21:45.917593 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:45.918520 17761 raft_consensus.cc:2468] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 5.
I20260709 12:21:45.920023 20585 raft_consensus.cc:2749] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:21:45.921134 20577 leader_election.cc:290] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:21:45.921360 17836 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37994 (request call id 643) took 1537 ms (client timeout 1503 ms). Trace:
W20260709 12:21:45.921739 17836 rpcz_store.cc:269] 0709 12:21:44.383584 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:44.483444 (+ 99860us) service_pool.cc:224] Handling call
0709 12:21:45.920983 (+1437539us) spinlock_profiling.cc:243] Waited 32.8 ms on lock 0x7b5400158bd4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:21:45.921144 (+   161us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":75422976}
W20260709 12:21:45.922896 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 162582us
I20260709 12:21:45.924077 17727 leader_election.cc:304] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:21:45.925180 18363 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 989) took 1342 ms. Trace:
I20260709 12:21:45.925700 18363 rpcz_store.cc:276] 0709 12:21:44.582757 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:44.852637 (+269880us) service_pool.cc:224] Handling call
0709 12:21:44.853002 (+   365us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:45.609615 (+756613us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:45.924582 (+314967us) spinlock_profiling.cc:243] Waited 755 ms on lock 0x7b54007e7cd4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:21:45.924954 (+   372us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1736908544}
I20260709 12:21:45.926297 17727 leader_election.cc:304] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:21:45.927700 17838 raft_consensus.cc:1240] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 5. Current term is 6. Ops: []
I20260709 12:21:45.929494 17838 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35042 (request call id 1764) took 1671 ms. Trace:
I20260709 12:21:45.929863 17838 rpcz_store.cc:276] 0709 12:21:44.257646 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:44.817269 (+559623us) service_pool.cc:224] Handling call
0709 12:21:45.927118 (+1109849us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:45.928209 (+  1091us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:45.929252 (+  1043us) spinlock_profiling.cc:243] Waited 1.11 s on lock 0x7b5400158bd0. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a61f8ad 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:21:45.929420 (+   168us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2551296896}
I20260709 12:21:45.934664 17837 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:59820 (request call id 24) took 2341 ms. Trace:
I20260709 12:21:45.935096 17837 rpcz_store.cc:276] 0709 12:21:43.592733 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:43.814371 (+221638us) service_pool.cc:224] Handling call
0709 12:21:45.934327 (+2119956us) spinlock_profiling.cc:243] Waited 264 ms on lock 0x7b5400265a54. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a61a853 00007f7b4f5e860a 00007f7b48b6e7d4 00007f7b48b6e762 00007f7b48b6e691 00007f7b48b6e60c 00007f7b48b6d8c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:21:45.934452 (+   125us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":606312832}
W20260709 12:21:45.936646 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 49104.1us
I20260709 12:21:45.937407 17652 leader_election.cc:304] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:21:45.886813 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:45.940442 17835 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37994 (request call id 654) took 1184 ms. Trace:
I20260709 12:21:45.941857 17758 raft_consensus.cc:1240] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 6 LEADER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 5. Current term is 6. Ops: [5.7-5.7]
I20260709 12:21:45.940841 17835 rpcz_store.cc:276] 0709 12:21:44.755758 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:44.957277 (+201519us) service_pool.cc:224] Handling call
0709 12:21:44.983031 (+ 25754us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:44.983329 (+   298us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:21:45.288981 (+305652us) log.cc:844] Serialized 14 byte log entry
0709 12:21:45.879902 (+590921us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:45.939841 (+ 59939us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:21:45.939985 (+   144us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:45.940092 (+   107us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:45.940274 (+   182us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2623,"prepare.queue_time_us":557255,"prepare.run_cpu_time_us":615,"prepare.run_wall_time_us":4295,"raft.queue_time_us":62482,"raft.run_cpu_time_us":146,"raft.run_wall_time_us":4681,"thread_start_us":704525,"threads_started":3,"wal-append.queue_time_us":360093}
W20260709 12:21:45.945163 18374 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59886 (request call id 842) took 1646 ms (client timeout 1520 ms). Trace:
W20260709 12:21:45.945806 18374 rpcz_store.cc:269] 0709 12:21:44.298698 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:44.298856 (+   158us) service_pool.cc:224] Handling call
0709 12:21:45.944542 (+1645686us) spinlock_profiling.cc:243] Waited 17.9 ms on lock 0x7b540051a254. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:21:45.944821 (+   279us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":41139072}
I20260709 12:21:45.948683 20583 raft_consensus.cc:2804] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:21:45.949121 20583 raft_consensus.cc:493] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:45.949486 20583 raft_consensus.cc:3060] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:45.949566 20127 consensus_queue.cc:1059] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:21:45.903605 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:45.952360 17757 raft_consensus.cc:2410] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for term 7 because replica has last-logged OpId of term: 4 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260709 12:21:45.952260 20572 raft_consensus.cc:3055] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:21:45.952724 20572 raft_consensus.cc:740] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:21:45.953987 20572 consensus_queue.cc:260] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:45.955809 17654 leader_election.cc:304] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:45.957759 20585 raft_consensus.cc:2749] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:21:45.961370 20394 consensus_queue.cc:1059] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:21:45.892692 17687 rpcz_store.cc:276] 0709 12:21:43.433783 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:43.675161 (+241378us) service_pool.cc:224] Handling call
0709 12:21:43.675546 (+   385us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:44.409939 (+734393us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:45.872102 (+1462163us) spinlock_profiling.cc:243] Waited 709 ms on lock 0x7b5400207954. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:21:45.892065 (+ 19963us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1631730048}
I20260709 12:21:45.962751 18367 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59886 (request call id 857) took 1412 ms. Trace:
I20260709 12:21:45.963406 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:45.965325 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:45.966324 17687 raft_consensus.cc:2393] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 6: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:21:45.963141 18367 rpcz_store.cc:276] 0709 12:21:44.550358 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:44.575389 (+ 25031us) service_pool.cc:224] Handling call
0709 12:21:44.575653 (+   264us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:21:45.316331 (+740678us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:45.316389 (+    58us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:21:45.320982 (+  4593us) log.cc:844] Serialized 52 byte log entry
0709 12:21:45.579661 (+258679us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:21:45.961807 (+382146us) log.cc:844] Serialized 14 byte log entry
0709 12:21:45.962213 (+   406us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:45.962295 (+    82us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:45.962345 (+    50us) raft_consensus.cc:1685] finished
0709 12:21:45.962355 (+    10us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:45.962586 (+   231us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":12947,"prepare.queue_time_us":198548,"prepare.run_cpu_time_us":87,"prepare.run_wall_time_us":86,"raft.queue_time_us":46826,"spinlock_wait_cycles":25193856,"thread_start_us":1168922,"threads_started":3,"wal-append.queue_time_us":224617}
I20260709 12:21:45.967146 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:45.967578 17834 raft_consensus.cc:2468] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 7.
I20260709 12:21:45.969352 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:45.970691 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:45.978477 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:45.919549 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:45.980361 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:45.981860 20346 raft_consensus.cc:493] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:21:45.981303 17686 raft_consensus.cc:2393] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 5: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:21:45.967685 17687 raft_consensus.cc:3060] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:45.982962 17833 raft_consensus.cc:2468] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 6.
W20260709 12:21:45.983872 17801 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:45.984812 20336 raft_consensus.cc:493] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:45.911772 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 10 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:45.987370 17830 raft_consensus.cc:1275] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260709 12:21:45.895684 17689 raft_consensus.cc:2468] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 5.
I20260709 12:21:45.987432 20370 raft_consensus.cc:493] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:45.989588 20368 raft_consensus.cc:3055] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:21:45.990196 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:45.919474 17803 leader_election.cc:304] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:21:45.990000 17829 raft_consensus.cc:2955] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 4 to 5, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:21:45.995615 17764 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36572 (request call id 548) took 1607 ms. Trace:
I20260709 12:21:45.999207 17757 raft_consensus.cc:2410] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for term 6 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20260709 12:21:46.000890 17764 rpcz_store.cc:276] 0709 12:21:44.388613 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:44.603831 (+215218us) service_pool.cc:224] Handling call
0709 12:21:45.977975 (+1374144us) spinlock_profiling.cc:243] Waited 1.3 ms on lock 0x7b5800033398. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b474fbb71 00007f7b474fba22 00007f7b504148d9 00007f7b4f56eee0 00007f7b4f6380ab 00007f7b4f638131 00007f7b4f5e60a0 00007f7b4f5e5984 00007f7b48b606d4 00007f7b48b60662
0709 12:21:45.995548 (+ 17573us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3000064}
I20260709 12:21:45.998732 20346 raft_consensus.cc:3060] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:46.001680 20370 raft_consensus.cc:3060] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:46.010591 20227 raft_consensus.cc:3055] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:21:46.011150 20227 raft_consensus.cc:740] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
W20260709 12:21:46.014235 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 49581.2us
I20260709 12:21:46.014376 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:46.015586 17764 raft_consensus.cc:2393] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 6: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:21:45.997512 20368 raft_consensus.cc:740] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:21:46.015453 17729 leader_election.cc:304] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:21:46.016353 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:46.017584 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:46.018201 17764 raft_consensus.cc:2468] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 4.
I20260709 12:21:46.019976 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:46.020933 20515 raft_consensus.cc:3055] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:21:46.022243 20479 raft_consensus.cc:2804] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:21:46.023116 17764 raft_consensus.cc:2468] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 3.
I20260709 12:21:46.025854 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:46.002223 20573 raft_consensus.cc:2749] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 12:21:46.029004 17680 raft_consensus.cc:2468] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 9.
I20260709 12:21:46.037202 20453 raft_consensus.cc:493] T 7cb3768d215e409fb9f5baf18d124442 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:46.038010 20453 raft_consensus.cc:515] T 7cb3768d215e409fb9f5baf18d124442 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:46.041900 17764 raft_consensus.cc:1240] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 5. Current term is 6. Ops: []
I20260709 12:21:46.024987 20394 consensus_queue.cc:1243] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer e2909cff1da44ad387f96a8ae3aa5afb log is divergent from this leader: its last log entry 6.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260709 12:21:46.046080 17764 raft_consensus.cc:1275] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260709 12:21:46.045832 20583 raft_consensus.cc:515] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:46.046281 20368 consensus_queue.cc:260] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:46.047472 20368 raft_consensus.cc:3060] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:46.047919 17764 raft_consensus.cc:1275] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260709 12:21:46.033571 20370 raft_consensus.cc:515] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:46.043787 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:45.995586 20336 raft_consensus.cc:515] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:46.050925 20324 raft_consensus.cc:515] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:46.051327 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:46.052457 20336 leader_election.cc:290] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:46.024857 20385 raft_consensus.cc:2749] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:21:46.053191 19855 raft_consensus.cc:493] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:21:46.026625 20571 raft_consensus.cc:2804] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:21:46.053433 20336 consensus_queue.cc:1048] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:21:46.054023 20571 raft_consensus.cc:697] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 6 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:21:46.053876 19855 raft_consensus.cc:515] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:46.052456 17686 raft_consensus.cc:2393] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 6: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:21:46.041952 17801 leader_election.cc:304] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:21:46.058601 20336 raft_consensus.cc:2764] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
I20260709 12:21:46.052584 20479 raft_consensus.cc:697] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:21:46.061235 17835 raft_consensus.cc:3060] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:46.064540 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:46.023379 20515 raft_consensus.cc:740] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 5 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:21:46.070785 17685 raft_consensus.cc:2393] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 5: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:21:46.070158 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:46.055248 20571 consensus_queue.cc:237] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:46.078675 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:46.080457 17833 raft_consensus.cc:2393] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 4: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:21:46.081161 17764 raft_consensus.cc:3060] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:46.082217 17801 leader_election.cc:304] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:21:46.078135 20605 raft_consensus.cc:493] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:21:46.096150 20583 leader_election.cc:290] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 5 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:46.096334 20605 raft_consensus.cc:515] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:46.099416 20605 leader_election.cc:290] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:21:46.099776 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:46.100823 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:46.101102 20453 leader_election.cc:290] T 7cb3768d215e409fb9f5baf18d124442 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:46.102833 20324 leader_election.cc:290] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 4 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:46.108168 20604 consensus_queue.cc:1059] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:21:46.109443 20604 raft_consensus.cc:3055] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 5 LEADER]: Stepping down as leader of term 5
W20260709 12:21:46.109954 20510 log.cc:927] Time spent T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 2.073s	user 0.005s	sys 0.006s
I20260709 12:21:46.109897 20604 raft_consensus.cc:740] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 5 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:21:46.117941 20479 consensus_queue.cc:237] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:21:46.120790 18334 leader_election.cc:336] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Timed out: RequestConsensusVote RPC to 127.16.29.65:33071 timed out after 1.567s (SENT)
I20260709 12:21:46.120890 20604 consensus_queue.cc:260] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:46.122288 19855 leader_election.cc:290] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:46.016342 20227 consensus_queue.cc:260] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:46.126123 20525 consensus_queue.cc:1059] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 5.5, Next index: 6, Last known committed idx: 3, Time since last communication: 0.001s
W20260709 12:21:46.129206 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 57937.2us
W20260709 12:21:46.132076 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 57317.7us
W20260709 12:21:46.133023 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:46.131736 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 58375us
W20260709 12:21:46.132407 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 213087us
I20260709 12:21:46.143206 17761 raft_consensus.cc:2393] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 6: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
W20260709 12:21:46.144794 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 218726us
I20260709 12:21:46.137110 20577 raft_consensus.cc:493] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:21:46.130867 20324 consensus_queue.cc:1048] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 1, Time since last communication: 0.036s
I20260709 12:21:46.160313 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:21:46.268466 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 82678.6us
I20260709 12:21:46.165848 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:46.270843 17830 raft_consensus.cc:1275] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 4 index: 2. (term mismatch)
I20260709 12:21:46.142666 20604 raft_consensus.cc:3060] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:46.195276 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:46.331514 17833 raft_consensus.cc:2468] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 4.
W20260709 12:21:46.220166 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 80804.5us
W20260709 12:21:46.249838 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 112494us
W20260709 12:21:46.221606 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 60954.4us
I20260709 12:21:46.420255 17760 raft_consensus.cc:3060] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:46.413599 17830 pending_rounds.cc:85] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b: Aborting all ops after (but not including) 1
I20260709 12:21:46.213348 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:46.329289 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:46.330607 18362 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59886 (request call id 844) took 2027 ms. Trace:
I20260709 12:21:46.328732 18333 leader_election.cc:304] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
W20260709 12:21:46.186708 17728 leader_election.cc:336] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Timed out: RequestConsensusVote RPC to 127.16.29.65:33071 timed out after 1.515s (SENT)
I20260709 12:21:46.421836 17830 pending_rounds.cc:107] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b: Aborting uncommitted NO_OP operation due to leader change: 3.2
I20260709 12:21:46.326103 20102 raft_consensus.cc:493] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
W20260709 12:21:46.443627 20613 log.cc:927] Time spent T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.203s	user 0.001s	sys 0.000s
I20260709 12:21:46.444142 20102 raft_consensus.cc:515] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:46.445668 20324 raft_consensus.cc:493] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:21:46.446416 20324 raft_consensus.cc:515] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:46.447357 20102 leader_election.cc:290] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:46.330246 20532 raft_consensus.cc:3055] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:21:46.449016 18363 raft_consensus.cc:3060] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:46.333796 19855 raft_consensus.cc:493] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:46.457557 18368 raft_consensus.cc:1240] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 7. Current term is 8. Ops: []
I20260709 12:21:46.433367 18362 rpcz_store.cc:276] 0709 12:21:44.303255 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:44.320010 (+ 16755us) service_pool.cc:224] Handling call
0709 12:21:44.343091 (+ 23081us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:21:44.343487 (+   396us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:44.343525 (+    38us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:21:44.364028 (+ 20503us) log.cc:844] Serialized 52 byte log entry
0709 12:21:44.463755 (+ 99727us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:21:44.483827 (+ 20072us) log.cc:844] Serialized 14 byte log entry
0709 12:21:44.499796 (+ 15969us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:44.499923 (+   127us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:46.119158 (+1619235us) spinlock_profiling.cc:243] Waited 1.81 ms on lock 0x7b180028c120. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b475333f3 00007f7b475357c1 00007f7b47535779 00007f7b47535709 00007f7b475356d1 00007f7b475349cd 0000000000388634 0000000000388479 00007f7b4751683a
0709 12:21:46.277349 (+158191us) raft_consensus.cc:1685] finished
0709 12:21:46.307895 (+ 30546us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:46.330451 (+ 22556us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2167,"prepare.queue_time_us":16099,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":40,"raft.queue_time_us":16197,"raft.run_cpu_time_us":238,"raft.run_wall_time_us":20689,"spinlock_wait_cycles":4164864,"thread_start_us":66025,"threads_started":1,"wal-append.queue_time_us":136596}
I20260709 12:21:46.441289 17830 raft_consensus.cc:2889] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:21:46.249150 20385 raft_consensus.cc:2804] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:21:46.451800 20324 leader_election.cc:290] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:46.463495 20584 raft_consensus.cc:493] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:21:46.464577 18365 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 992) took 1838 ms. Trace:
I20260709 12:21:46.464295 20584 raft_consensus.cc:515] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:46.464933 18365 rpcz_store.cc:276] 0709 12:21:44.626391 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:44.929174 (+302783us) service_pool.cc:224] Handling call
0709 12:21:45.432593 (+503419us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:45.433429 (+   836us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:46.464155 (+1030726us) spinlock_profiling.cc:243] Waited 503 ms on lock 0x7b5400466ed0. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a61f8ad 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:21:46.464380 (+   225us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1156796672}
I20260709 12:21:46.466337 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:46.466666 20584 leader_election.cc:290] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:46.467162 18365 raft_consensus.cc:2393] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 5: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:21:46.467641 17762 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38776 (request call id 1807) took 2772 ms. Trace:
I20260709 12:21:46.459256 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:46.327906 17832 raft_consensus.cc:1275] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
W20260709 12:21:46.454849 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 290147us
I20260709 12:21:46.456080 17801 leader_election.cc:304] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:21:46.468150 17762 rpcz_store.cc:276] 0709 12:21:43.695285 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:43.695469 (+   184us) service_pool.cc:224] Handling call
0709 12:21:43.695893 (+   424us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:44.285445 (+589552us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:46.467000 (+2181555us) spinlock_profiling.cc:243] Waited 589 ms on lock 0x7b54003e44d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:21:46.467422 (+   422us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1353755392}
I20260709 12:21:46.459086 20394 raft_consensus.cc:493] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:21:46.475116 20394 raft_consensus.cc:515] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:46.475986 18366 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59902 (request call id 819) took 2062 ms. Trace:
I20260709 12:21:46.476408 18366 rpcz_store.cc:276] 0709 12:21:44.413036 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:44.421285 (+  8249us) service_pool.cc:224] Handling call
0709 12:21:46.475555 (+2054270us) spinlock_profiling.cc:243] Waited 123 ms on lock 0x7b540034b554. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:21:46.475780 (+   225us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":282306816}
I20260709 12:21:46.477474 20394 leader_election.cc:290] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:46.165968 20299 consensus_queue.cc:1059] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.329s
I20260709 12:21:46.263386 20577 raft_consensus.cc:515] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:46.436990 20336 raft_consensus.cc:493] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:46.487419 20144 leader_election.cc:290] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:46.487836 20336 raft_consensus.cc:515] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:46.490317 20336 leader_election.cc:290] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:46.482082 17687 raft_consensus.cc:2468] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 7.
W20260709 12:21:46.485312 20619 log.cc:927] Time spent T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.157s	user 0.000s	sys 0.001s
I20260709 12:21:46.486677 17684 raft_consensus.cc:2368] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for earlier term 5. Current term is 6.
I20260709 12:21:46.487129 20577 leader_election.cc:290] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:46.268857 20525 consensus_queue.cc:1059] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.194s
I20260709 12:21:46.453110 20532 raft_consensus.cc:740] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:21:46.326635 20621 consensus_queue.cc:1048] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.154s
I20260709 12:21:46.484445 17830 pending_rounds.cc:107] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b: Aborting uncommitted NO_OP operation due to leader change: 5.3
I20260709 12:21:46.494653 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:46.497753 17830 raft_consensus.cc:2889] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:21:46.494846 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
W20260709 12:21:46.496767 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 75988us
I20260709 12:21:46.498080 17836 raft_consensus.cc:3060] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:46.469918 20385 raft_consensus.cc:493] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:46.496014 18371 raft_consensus.cc:2468] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 5.
W20260709 12:21:46.502525 17684 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37396 (request call id 1891) took 1698 ms (client timeout 1515 ms). Trace:
W20260709 12:21:46.502935 17684 rpcz_store.cc:269] 0709 12:21:44.800473 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:44.812723 (+ 12250us) service_pool.cc:224] Handling call
0709 12:21:46.136869 (+1324146us) spinlock_profiling.cc:243] Waited 4.43 ms on lock 0x7b5800002a98. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b474fbb71 00007f7b474fba22 00007f7b504148d9 00007f7b4f56eee0 00007f7b4f6380ab 00007f7b4f638131 00007f7b4f5e60a0 00007f7b4f5e5984 00007f7b48b606d4 00007f7b48b60662
0709 12:21:46.498961 (+362092us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":10182912}
I20260709 12:21:46.458431 18367 raft_consensus.cc:2955] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 4 to 5, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:21:46.504549 17684 raft_consensus.cc:1240] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 6 LEADER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 5. Current term is 6. Ops: [3.2-5.3]
I20260709 12:21:46.480554 20102 raft_consensus.cc:2749] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:21:46.331297 20515 consensus_queue.cc:260] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:46.514963 20374 leader_election.cc:290] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:46.524363 20422 raft_consensus.cc:493] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:21:46.528934 17685 raft_consensus.cc:3055] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 5 LEADER]: Stepping down as leader of term 5
W20260709 12:21:46.531237 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:46.539037 17764 raft_consensus.cc:2468] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 5.
I20260709 12:21:46.500974 20385 raft_consensus.cc:3060] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:46.540328 17685 raft_consensus.cc:740] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
W20260709 12:21:46.562655 17730 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:46.565045 17730 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:46.572147 17803 leader_election.cc:304] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:21:46.562762 20422 raft_consensus.cc:3060] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:21:46.581512 17730 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:46.584977 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:46.581815 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 80034.5us
I20260709 12:21:46.574949 20515 raft_consensus.cc:3060] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:46.493403 19855 raft_consensus.cc:515] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:21:46.601787 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 457189us
I20260709 12:21:46.538265 20394 consensus_queue.cc:1059] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.062s
W20260709 12:21:46.617296 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 481892us
I20260709 12:21:46.585381 20584 raft_consensus.cc:493] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:46.581105 17572 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet d522f70beb1f417cb98940d90b594823 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 0da169ff82674dc1947055ce43de68a3 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 4 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:21:46.593890 20179 raft_consensus.cc:493] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:46.637343 17572 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 67570us
I20260709 12:21:46.637989 17685 consensus_queue.cc:260] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:46.651746 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:46.656289 20179 raft_consensus.cc:515] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:21:46.657574 17681 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44898 (request call id 613) took 2076 ms (client timeout 1567 ms). Trace:
W20260709 12:21:46.657991 17681 rpcz_store.cc:269] 0709 12:21:44.580859 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:44.594322 (+ 13463us) service_pool.cc:224] Handling call
0709 12:21:46.657206 (+2062884us) spinlock_profiling.cc:243] Waited 736 ms on lock 0x7b5400347bd4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:21:46.657400 (+   194us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1693120384}
I20260709 12:21:46.658630 20179 leader_election.cc:290] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:46.659619 20584 raft_consensus.cc:515] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:46.661674 17685 raft_consensus.cc:3060] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:46.637730 20374 raft_consensus.cc:2804] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:21:46.668783 17683 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37396 (request call id 1883) took 2031 ms. Trace:
I20260709 12:21:46.672580 17834 raft_consensus.cc:2376] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 6. Re-sending same reply.
W20260709 12:21:46.676038 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 162357us
I20260709 12:21:46.669611 17683 rpcz_store.cc:276] 0709 12:21:44.637726 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:44.723263 (+ 85537us) service_pool.cc:224] Handling call
0709 12:21:44.723532 (+   269us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:44.801874 (+ 78342us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:46.668262 (+1866388us) spinlock_profiling.cc:243] Waited 75.9 ms on lock 0x7b5400425ed4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:21:46.668590 (+   328us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":174669184}
W20260709 12:21:46.684746 17835 consensus_meta.cc:287] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.591s	user 0.009s	sys 0.002s
I20260709 12:21:46.685513 17835 raft_consensus.cc:2468] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 5.
I20260709 12:21:46.635946 20324 consensus_queue.cc:1059] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.040s
I20260709 12:21:46.691512 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:46.677805 20374 raft_consensus.cc:697] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 4 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:21:46.714468 20559 raft_consensus.cc:2804] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:21:46.715132 20605 raft_consensus.cc:493] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:46.715700 20605 raft_consensus.cc:515] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:46.716017 18361 raft_consensus.cc:1275] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260709 12:21:46.717345 20370 leader_election.cc:290] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 6 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:46.717926 20605 leader_election.cc:290] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:46.715952 20559 raft_consensus.cc:493] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:21:46.718880 20559 raft_consensus.cc:3060] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:46.721010 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:46.721537 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:46.722159 17835 raft_consensus.cc:3060] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:46.726060 17654 leader_election.cc:304] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:46.731813 17685 raft_consensus.cc:1275] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 7. (term mismatch)
I20260709 12:21:46.732205 17685 pending_rounds.cc:85] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3: Aborting all ops after (but not including) 6
I20260709 12:21:46.735379 17685 pending_rounds.cc:107] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3: Aborting uncommitted NO_OP operation due to leader change: 5.7
W20260709 12:21:46.736840 20564 log.cc:927] Time spent T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 2.079s	user 0.013s	sys 0.000s
I20260709 12:21:46.737449 17687 raft_consensus.cc:3055] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 5 LEADER]: Stepping down as leader of term 5
W20260709 12:21:46.737790 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 100439us
W20260709 12:21:46.740875 17682 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37396 (request call id 1888) took 1993 ms (client timeout 1952 ms). Trace:
W20260709 12:21:46.741262 17682 rpcz_store.cc:269] 0709 12:21:44.747554 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:44.773305 (+ 25751us) service_pool.cc:224] Handling call
0709 12:21:46.740489 (+1967184us) spinlock_profiling.cc:243] Waited 289 ms on lock 0x7b54001aac54. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:21:46.740701 (+   212us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":664049536}
I20260709 12:21:46.741521 20422 raft_consensus.cc:515] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:46.744297 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:46.745153 18371 raft_consensus.cc:2393] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 5: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:21:46.745069 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:46.737818 17687 raft_consensus.cc:740] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:21:46.753250 17685 raft_consensus.cc:2889] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:21:46.755322 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:46.676800 20532 consensus_queue.cc:260] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:21:46.759059 20327 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 6 current term: 6 is higher.
I20260709 12:21:46.760969 17680 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37396 (request call id 1900) took 1235 ms. Trace:
I20260709 12:21:46.757153 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:46.731397 20584 leader_election.cc:290] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:21:46.762253 20628 log.cc:927] Time spent T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.057s	user 0.000s	sys 0.000s
I20260709 12:21:46.764225 18362 raft_consensus.cc:2410] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for term 7 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260709 12:21:46.764364 17687 consensus_queue.cc:260] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:46.760528 20374 consensus_queue.cc:237] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:46.774870 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:46.773813 20422 leader_election.cc:290] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 7 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:46.771714 17680 rpcz_store.cc:276] 0709 12:21:45.525177 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:46.333190 (+808013us) service_pool.cc:224] Handling call
0709 12:21:46.333705 (+   515us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:46.334238 (+   533us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:46.334263 (+    25us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:46.437412 (+103149us) log.cc:844] Serialized 26 byte log entry
0709 12:21:46.620790 (+183378us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:21:46.620845 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:46.620938 (+    93us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:46.760741 (+139803us) raft_consensus.cc:1685] finished
0709 12:21:46.760765 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:46.760913 (+   148us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5311,"raft.queue_time_us":70042,"raft.run_cpu_time_us":280,"raft.run_wall_time_us":38042,"thread_start_us":141250,"threads_started":1,"wal-append.queue_time_us":227300}
I20260709 12:21:46.795212 17687 raft_consensus.cc:3060] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:46.795324 20532 raft_consensus.cc:3060] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:46.797669 20559 raft_consensus.cc:515] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:46.800191 20559 leader_election.cc:290] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 4 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:46.800627 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:46.801776 18362 raft_consensus.cc:2368] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 5. Current term is 6.
I20260709 12:21:46.802775 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:46.803535 18362 raft_consensus.cc:3060] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:46.814584 18363 raft_consensus.cc:2468] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 6.
I20260709 12:21:46.818473 18366 raft_consensus.cc:2393] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 6: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:21:46.819629 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:46.820868 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:46.821492 17688 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37396 (request call id 1899) took 1332 ms. Trace:
I20260709 12:21:46.821970 17688 rpcz_store.cc:276] 0709 12:21:45.489387 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:46.125390 (+636003us) service_pool.cc:224] Handling call
0709 12:21:46.125975 (+   585us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:46.142463 (+ 16488us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:46.142482 (+    19us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:46.252936 (+110454us) log.cc:844] Serialized 26 byte log entry
0709 12:21:46.501884 (+248948us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:21:46.501923 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:46.501983 (+    60us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:46.821139 (+319156us) raft_consensus.cc:1685] finished
0709 12:21:46.821158 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:46.821296 (+   138us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":82477,"raft.queue_time_us":3032,"raft.run_cpu_time_us":249,"raft.run_wall_time_us":35334,"thread_start_us":173446,"threads_started":1,"wal-append.queue_time_us":340565}
I20260709 12:21:46.821713 18366 raft_consensus.cc:2393] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 7: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:21:46.824188 20394 raft_consensus.cc:2804] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260709 12:21:46.837091 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:46.837812 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:46.840338 20394 raft_consensus.cc:493] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:46.845929 20394 raft_consensus.cc:3060] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:21:46.833586 20624 log.cc:927] Time spent T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.199s	user 0.000s	sys 0.001s
I20260709 12:21:46.853250 17834 raft_consensus.cc:1275] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:21:46.854184 17687 raft_consensus.cc:1275] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 7. (term mismatch)
I20260709 12:21:46.855705 17687 pending_rounds.cc:85] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3: Aborting all ops after (but not including) 6
I20260709 12:21:46.859185 18374 raft_consensus.cc:1240] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 7. Current term is 8. Ops: []
I20260709 12:21:46.859100 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:46.858858 18366 raft_consensus.cc:2468] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 6.
I20260709 12:21:46.860819 17687 pending_rounds.cc:107] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3: Aborting uncommitted NO_OP operation due to leader change: 5.7
I20260709 12:21:46.861034 18362 raft_consensus.cc:2468] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 6.
I20260709 12:21:46.861153 17687 raft_consensus.cc:2889] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:21:46.857671 20346 consensus_meta.cc:287] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.764s	user 0.007s	sys 0.006s
I20260709 12:21:46.867264 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:46.869992 20466 raft_consensus.cc:493] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:21:46.870352 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:46.871160 18366 raft_consensus.cc:3060] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:46.870692 20466 raft_consensus.cc:515] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:21:46.878028 20614 log.cc:927] Time spent T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.221s	user 0.001s	sys 0.001s
I20260709 12:21:46.878232 18364 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59886 (request call id 856) took 2332 ms. Trace:
I20260709 12:21:46.878652 18364 rpcz_store.cc:276] 0709 12:21:44.545931 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:44.546067 (+   136us) service_pool.cc:224] Handling call
0709 12:21:44.575324 (+ 29257us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:44.588109 (+ 12785us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:44.588146 (+    37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:44.588686 (+   540us) log.cc:844] Serialized 26 byte log entry
0709 12:21:46.509162 (+1920476us) spinlock_profiling.cc:243] Waited 1.42 ms on lock 0x7b18002984e0. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b47515e07 00007f7b503709a9 00007f7b4753269b 00007f7b4753093f 00007f7b47532c02 00007f7b4a1ba20b 00007f7b4a1bfd23 00007f7b4a1c00d1 00007f7b4a5dba0f
0709 12:21:46.707763 (+198601us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:21:46.713874 (+  6111us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:46.713977 (+   103us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:46.877773 (+163796us) raft_consensus.cc:1685] finished
0709 12:21:46.877795 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:46.878034 (+   239us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":27235,"raft.queue_time_us":20463,"raft.run_cpu_time_us":200,"raft.run_wall_time_us":29039,"spinlock_wait_cycles":3271552,"thread_start_us":177667,"threads_started":1,"wal-append.queue_time_us":2235113}
I20260709 12:21:46.880743 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:46.881399 17834 raft_consensus.cc:2393] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 5: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:21:46.886433 20398 leader_election.cc:290] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:46.887835 20621 raft_consensus.cc:493] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:46.887754 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:46.888526 20621 raft_consensus.cc:515] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:46.889410 17832 raft_consensus.cc:2393] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 6: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:21:46.891315 20621 leader_election.cc:290] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:46.892853 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:46.893877 18362 raft_consensus.cc:2393] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 6: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:21:46.892977 18364 raft_consensus.cc:3055] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:21:46.896358 17832 raft_consensus.cc:1240] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 6. Current term is 7. Ops: []
I20260709 12:21:46.896643 18364 raft_consensus.cc:740] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:21:46.897897 17832 raft_consensus.cc:1240] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 5. Current term is 6. Ops: [5.6-5.6]
I20260709 12:21:46.897598 18364 consensus_queue.cc:260] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:46.898662 18364 raft_consensus.cc:3060] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:46.899453 20572 raft_consensus.cc:3060] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:46.900888 18366 raft_consensus.cc:2468] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 4.
I20260709 12:21:46.902369 17835 raft_consensus.cc:2468] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 6.
I20260709 12:21:46.902627 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:46.904531 20398 raft_consensus.cc:493] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:46.905189 20398 raft_consensus.cc:515] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:46.910591 17760 raft_consensus.cc:2468] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 4.
I20260709 12:21:46.913709 20398 leader_election.cc:290] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:46.913686 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:46.914304 17836 raft_consensus.cc:2468] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 5.
I20260709 12:21:46.914477 17834 raft_consensus.cc:1275] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260709 12:21:46.914875 20336 raft_consensus.cc:2764] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20260709 12:21:46.918835 17831 raft_consensus.cc:1275] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260709 12:21:46.867364 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:46.867991 19855 leader_election.cc:290] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:46.869346 20346 raft_consensus.cc:515] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:46.899618 17832 raft_consensus.cc:1240] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 3. Current term is 4. Ops: []
I20260709 12:21:46.934639 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:46.936625 18362 raft_consensus.cc:3055] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:21:46.939361 18362 raft_consensus.cc:740] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
W20260709 12:21:46.943322 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 83621.2us
I20260709 12:21:46.944092 18362 consensus_queue.cc:260] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:46.948915 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:46.951103 18362 raft_consensus.cc:3060] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:21:46.969555 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:46.974493 17686 raft_consensus.cc:1240] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 4 LEADER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 3. Current term is 4. Ops: []
I20260709 12:21:46.974681 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:46.975608 17682 raft_consensus.cc:2393] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 4: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:21:46.977655 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:21:46.979242 18333 leader_election.cc:336] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837): Timed out: RequestConsensusVote RPC to 127.16.29.67:36837 timed out after 1.503s (SENT)
I20260709 12:21:46.979872 18333 leader_election.cc:304] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:46.980911 20532 raft_consensus.cc:2749] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20260709 12:21:46.981966 18333 leader_election.cc:336] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Timed out: RequestConsensusVote RPC to 127.16.29.66:39651 timed out after 1.578s (SENT)
I20260709 12:21:46.989385 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:46.990345 17680 raft_consensus.cc:2393] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 6: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:21:46.989794 20394 raft_consensus.cc:515] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:46.994721 20394 leader_election.cc:290] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 6 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:47.001547 17685 raft_consensus.cc:1240] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 7 LEADER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 6. Current term is 7. Ops: []
I20260709 12:21:47.002739 17803 leader_election.cc:400] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 5 pre-election: Vote denied by peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577) with higher term. Message: Invalid argument: T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 5. Current term is 6.
I20260709 12:21:47.003062 17803 leader_election.cc:403] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 5 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:21:47.004169 17803 leader_election.cc:304] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:47.006934 17654 leader_election.cc:304] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:47.009783 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:47.022876 18371 raft_consensus.cc:1240] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 5. Current term is 6. Ops: []
W20260709 12:21:47.028641 18334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:47.028503 20604 raft_consensus.cc:2749] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:21:47.029762 20559 raft_consensus.cc:493] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:47.030030 20604 consensus_queue.cc:1059] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 12:21:47.030350 20559 raft_consensus.cc:515] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:47.031178 20604 raft_consensus.cc:3055] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:21:47.031540 20604 raft_consensus.cc:740] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 7 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:21:47.007016 20374 raft_consensus.cc:2749] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: Vote denied by peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577) with higher term. Message: Invalid argument: T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 5. Current term is 6.
I20260709 12:21:47.032083 20532 raft_consensus.cc:493] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:47.032545 17801 leader_election.cc:304] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:47.032750 20559 leader_election.cc:290] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:47.033023 20346 leader_election.cc:290] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 5 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:47.033584 18367 raft_consensus.cc:2393] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 6: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:21:47.033775 20635 raft_consensus.cc:3060] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:21:47.017658 20636 log.cc:927] Time spent T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.161s	user 0.003s	sys 0.000s
I20260709 12:21:47.034660 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:47.035220 17688 raft_consensus.cc:1813] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Handling vote request from an unknown peer e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:47.035655 17688 raft_consensus.cc:2393] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 5: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:21:47.035758 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:47.035831 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:47.036456 17834 raft_consensus.cc:2376] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 6. Re-sending same reply.
I20260709 12:21:47.036583 17831 raft_consensus.cc:2468] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 5.
I20260709 12:21:47.007611 20525 raft_consensus.cc:3055] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:21:47.037249 20525 raft_consensus.cc:740] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:21:47.007979 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:47.038568 17829 raft_consensus.cc:2468] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 4.
I20260709 12:21:47.014045 18362 raft_consensus.cc:1275] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 7. (term mismatch)
I20260709 12:21:47.038484 20525 consensus_queue.cc:260] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:47.039214 18362 pending_rounds.cc:85] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da: Aborting all ops after (but not including) 6
I20260709 12:21:47.039523 18362 pending_rounds.cc:107] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da: Aborting uncommitted NO_OP operation due to leader change: 6.7
I20260709 12:21:47.016355 20553 raft_consensus.cc:3055] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:21:47.039784 18362 raft_consensus.cc:2889] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:21:47.039656 20525 raft_consensus.cc:3060] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:47.040026 20553 raft_consensus.cc:740] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:21:47.032447 20604 consensus_queue.cc:260] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 7.9, Last appended by leader: 9, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:47.046132 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:47.048767 18363 raft_consensus.cc:2410] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 7 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20260709 12:21:47.050132 20553 consensus_queue.cc:260] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:47.051563 20553 raft_consensus.cc:3060] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:21:47.052240 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:47.054239 20466 leader_election.cc:290] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:47.032620 20532 raft_consensus.cc:515] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:47.057224 20532 leader_election.cc:290] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:47.061398 18333 leader_election.cc:304] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:21:47.044580 20635 raft_consensus.cc:2749] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:21:47.026309 20385 raft_consensus.cc:515] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:47.044323 20572 raft_consensus.cc:2749] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260709 12:21:47.068957 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 94888us
I20260709 12:21:47.072490 20466 raft_consensus.cc:493] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:21:47.073272 20466 raft_consensus.cc:515] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:47.075351 17654 leader_election.cc:304] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:21:47.076232 20418 raft_consensus.cc:2749] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:21:47.076447 20604 raft_consensus.cc:3060] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:21:47.081969 17654 leader_election.cc:304] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:21:47.085104 20572 raft_consensus.cc:2804] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:21:47.085577 20572 raft_consensus.cc:493] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:47.085969 20572 raft_consensus.cc:3060] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:21:47.086640 20616 log.cc:927] Time spent T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.752s	user 0.000s	sys 0.002s
W20260709 12:21:47.089895 20631 log.cc:927] Time spent T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.256s	user 0.000s	sys 0.001s
I20260709 12:21:47.092128 20532 raft_consensus.cc:493] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:47.092788 19855 raft_consensus.cc:493] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:47.093354 19855 raft_consensus.cc:515] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:47.093809 17756 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38776 (request call id 1817) took 1376 ms. Trace:
I20260709 12:21:47.095474 19855 leader_election.cc:290] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:21:47.096359 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 629743us
W20260709 12:21:47.097457 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 615646us
I20260709 12:21:47.094269 17756 rpcz_store.cc:276] 0709 12:21:45.717536 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:45.756456 (+ 38920us) service_pool.cc:224] Handling call
0709 12:21:45.810428 (+ 53972us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:45.811022 (+   594us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 12:21:45.992632 (+181610us) log.cc:844] Serialized 14 byte log entry
0709 12:21:46.337711 (+345079us) log.cc:844] Serialized 14 byte log entry
0709 12:21:47.050757 (+713046us) spinlock_profiling.cc:243] Waited 6.58 ms on lock 0x7b180014e1e0. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b475333f3 00007f7b475357c1 00007f7b47535779 00007f7b47535709 00007f7b475356d1 00007f7b475349cd 0000000000388634 0000000000388479 00007f7b4751683a
0709 12:21:47.093111 (+ 42354us) spinlock_profiling.cc:243] Waited 6.54 ms on lock 0x7b180014e1e0. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b474adc0a 00007f7b50bed219 00007f7b47535a40 00007f7b475304e4 00007f7b47530455 00007f7b4a618060 00007f7b4a62a716 00007f7b4a63b287 00007f7b4a63b208
0709 12:21:47.093265 (+   154us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:47.093327 (+    62us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:21:47.093389 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:47.093473 (+    84us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:47.093612 (+   139us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":65578,"prepare.queue_time_us":200716,"prepare.run_cpu_time_us":86,"prepare.run_wall_time_us":87,"raft.queue_time_us":84837,"raft.run_cpu_time_us":125,"raft.run_wall_time_us":126,"spinlock_wait_cycles":30333952,"thread_start_us":371716,"threads_started":2,"wal-append.queue_time_us":395300}
I20260709 12:21:47.100715 17654 leader_election.cc:304] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:21:47.102349 18333 leader_election.cc:304] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:21:47.092662 20418 raft_consensus.cc:493] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:47.105453 19855 raft_consensus.cc:2749] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:21:47.106617 18333 leader_election.cc:304] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:21:47.110404 20349 raft_consensus.cc:493] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
W20260709 12:21:47.110760 17730 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:47.111218 17730 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:47.111073 20349 raft_consensus.cc:515] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:47.113993 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:47.114523 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:47.114751 17755 raft_consensus.cc:2468] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 8.
I20260709 12:21:47.109171 20346 consensus_queue.cc:1048] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.006s
I20260709 12:21:47.105125 19826 consensus_queue.cc:1048] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.006s
I20260709 12:21:47.108076 20394 raft_consensus.cc:2804] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:21:47.117177 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:47.117573 20394 raft_consensus.cc:697] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 4 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:21:47.113279 20349 leader_election.cc:290] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:47.120651 17801 leader_election.cc:304] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:21:47.124068 19855 raft_consensus.cc:2749] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:21:47.122601 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:47.126943 17801 leader_election.cc:304] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:47.128077 17801 leader_election.cc:304] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:47.131978 20385 leader_election.cc:290] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 6 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:47.132848 20385 raft_consensus.cc:2749] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:21:47.109494 20418 raft_consensus.cc:515] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:21:47.135082 17802 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:47.136613 20418 leader_election.cc:290] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:47.139104 20385 raft_consensus.cc:493] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:47.139645 20385 raft_consensus.cc:515] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:47.140842 17654 leader_election.cc:304] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:21:47.141661 20635 raft_consensus.cc:493] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:21:47.142428 20635 raft_consensus.cc:515] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:47.143584 20553 raft_consensus.cc:2804] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:21:47.144218 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:47.144944 20553 raft_consensus.cc:697] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 6 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:21:47.145366 17756 raft_consensus.cc:2410] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 5 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260709 12:21:47.145416 20397 consensus_queue.cc:1048] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:21:47.146087 20553 consensus_queue.cc:237] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:47.150202 20385 leader_election.cc:290] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:21:47.153786 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 47237.5us
I20260709 12:21:47.155687 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
W20260709 12:21:47.157192 17655 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:47.157341 18365 raft_consensus.cc:3060] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:21:47.157922 17655 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:47.156148 20157 raft_consensus.cc:2804] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:21:47.160223 20157 raft_consensus.cc:493] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:21:47.160751 20157 raft_consensus.cc:3060] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:21:47.156790 20626 raft_consensus.cc:2749] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:21:47.166327 20466 leader_election.cc:290] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:47.118633 20394 consensus_queue.cc:237] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:47.092656 20532 raft_consensus.cc:515] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:47.163378 17728 leader_election.cc:304] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:21:47.194157 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:47.221909 18333 leader_election.cc:304] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:21:47.191824 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:47.181566 17759 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38776 (request call id 1823) took 1402 ms. Trace:
I20260709 12:21:47.194578 18365 raft_consensus.cc:2468] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 6.
W20260709 12:21:47.216122 17730 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:47.223227 17759 rpcz_store.cc:276] 0709 12:21:45.779506 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:45.779910 (+   404us) service_pool.cc:224] Handling call
0709 12:21:47.180670 (+1400760us) spinlock_profiling.cc:243] Waited 12.1 ms on lock 0x7b5800033398. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b474fbb71 00007f7b474fba22 00007f7b504148d9 00007f7b4f56eee0 00007f7b4f6380ab 00007f7b4f638131 00007f7b4f5f9fc0 00007f7b4f5e4929 00007f7b4f5e43d2 00007f7b48b5bbd4
0709 12:21:47.180899 (+   229us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:47.181131 (+   232us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:47.181157 (+    26us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:47.181209 (+    52us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:21:47.181239 (+    30us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:47.181287 (+    48us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:47.181401 (+   114us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":27846400}
I20260709 12:21:47.240314 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:47.242569 17837 raft_consensus.cc:3060] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:21:47.186972 20562 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 6 current term: 6 is higher.
I20260709 12:21:47.245015 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:47.245250 18366 raft_consensus.cc:1240] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 6. Current term is 7. Ops: []
I20260709 12:21:47.246810 19855 raft_consensus.cc:2749] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:21:47.243383 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:47.246541 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:47.250068 17838 raft_consensus.cc:2468] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 3.
I20260709 12:21:47.246336 18333 leader_election.cc:304] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:21:47.169373 20227 raft_consensus.cc:3060] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:47.174077 20574 leader_election.cc:290] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:47.252866 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 10 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:47.255986 17689 raft_consensus.cc:2468] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 9.
I20260709 12:21:47.250977 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
W20260709 12:21:47.256781 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 150418us
I20260709 12:21:47.249152 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:47.256793 17687 raft_consensus.cc:1240] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 6 LEADER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 5. Current term is 6. Ops: []
I20260709 12:21:47.172226 20459 raft_consensus.cc:2804] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:21:47.249523 20370 raft_consensus.cc:493] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:47.259128 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:47.255267 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:47.255815 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:47.261873 20370 raft_consensus.cc:515] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:47.264796 20370 leader_election.cc:290] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:47.268308 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:47.268784 17757 raft_consensus.cc:3060] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:47.267802 18333 leader_election.cc:304] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:21:47.270582 19855 raft_consensus.cc:2804] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:21:47.270936 19855 raft_consensus.cc:493] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:47.271256 19855 raft_consensus.cc:3060] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:47.261340 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:47.272428 18333 leader_election.cc:304] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:21:47.274163 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 10 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:47.274477 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:47.252302 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:47.274262 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:47.277449 18374 raft_consensus.cc:3060] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:47.277572 18333 leader_election.cc:304] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:21:47.280155 17654 leader_election.cc:304] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
W20260709 12:21:47.281945 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 153938us
I20260709 12:21:47.278975 17727 leader_election.cc:304] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:21:47.284188 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:47.371526 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 10 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:47.244141 20157 raft_consensus.cc:515] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:47.372455 18365 raft_consensus.cc:2468] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 9.
I20260709 12:21:47.373158 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:47.377653 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:47.378177 17762 raft_consensus.cc:3060] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:47.275530 20574 raft_consensus.cc:2804] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:21:47.269956 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:47.374004 17801 leader_election.cc:304] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:47.372973 17689 raft_consensus.cc:2468] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 8.
I20260709 12:21:47.278496 20418 raft_consensus.cc:2804] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:21:47.387581 17683 raft_consensus.cc:2393] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 8: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:21:47.285056 20346 raft_consensus.cc:2749] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260709 12:21:47.386483 17801 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:47.380733 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 103218us
W20260709 12:21:47.381979 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 101890us
I20260709 12:21:47.383720 17688 raft_consensus.cc:2468] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 6.
I20260709 12:21:47.396474 18334 leader_election.cc:304] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:21:47.382263 17728 leader_election.cc:304] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:21:47.282624 20459 raft_consensus.cc:697] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 6 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:21:47.405277 17652 leader_election.cc:304] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:21:47.406908 18364 consensus_meta.cc:287] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.508s	user 0.008s	sys 0.003s
I20260709 12:21:47.374943 19826 raft_consensus.cc:2804] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:21:47.378733 20385 raft_consensus.cc:493] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:47.412529 18364 raft_consensus.cc:1275] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 6. (term mismatch)
I20260709 12:21:47.410023 17803 leader_election.cc:304] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:47.419618 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:47.381289 20398 raft_consensus.cc:493] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:47.420895 18364 pending_rounds.cc:85] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da: Aborting all ops after (but not including) 5
I20260709 12:21:47.428570 18364 pending_rounds.cc:107] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da: Aborting uncommitted NO_OP operation due to leader change: 3.6
I20260709 12:21:47.428844 18364 raft_consensus.cc:2889] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:21:47.422152 18334 leader_election.cc:304] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:21:47.423671 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:47.426426 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:47.429869 17833 raft_consensus.cc:2468] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 8.
I20260709 12:21:47.389382 20418 raft_consensus.cc:493] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:21:47.393241 20299 raft_consensus.cc:2804] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:21:47.431502 18334 leader_election.cc:304] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:21:47.388422 17832 raft_consensus.cc:2468] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 5.
I20260709 12:21:47.404431 20336 raft_consensus.cc:2749] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:21:47.433596 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:21:47.433755 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 60359.6us
I20260709 12:21:47.434501 18365 raft_consensus.cc:2393] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 6: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:21:47.393586 20574 raft_consensus.cc:697] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 6 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
W20260709 12:21:47.437743 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 181331us
I20260709 12:21:47.433267 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:47.429400 20466 raft_consensus.cc:2804] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:21:47.441576 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:47.438011 20398 raft_consensus.cc:515] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:47.417497 19826 raft_consensus.cc:493] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:47.446954 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:47.450431 17838 raft_consensus.cc:3060] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:47.434206 20418 raft_consensus.cc:3060] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:47.434736 20559 raft_consensus.cc:2804] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:21:47.434746 20635 leader_election.cc:290] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:21:47.463587 20652 log.cc:927] Time spent T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.186s	user 0.001s	sys 0.000s
I20260709 12:21:47.438261 20127 raft_consensus.cc:2749] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:21:47.443015 20299 raft_consensus.cc:697] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 6 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:21:47.449481 20346 raft_consensus.cc:2804] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260709 12:21:47.468601 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 63404.3us
I20260709 12:21:47.448616 20466 raft_consensus.cc:493] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:47.452419 19826 raft_consensus.cc:3060] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:21:47.431216 20385 raft_consensus.cc:515] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:47.444669 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:21:47.498672 18333 leader_election.cc:336] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Timed out: RequestConsensusVote RPC to 127.16.29.66:39651 timed out after 2.140s (SENT)
I20260709 12:21:47.468760 20559 raft_consensus.cc:493] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:47.472501 20368 raft_consensus.cc:2804] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:21:47.497310 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:47.503366 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:47.505620 17836 raft_consensus.cc:1275] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:21:47.507565 18333 leader_election.cc:304] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:47.510944 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:47.511664 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:47.512565 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:47.513451 20377 consensus_queue.cc:1048] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.062s
I20260709 12:21:47.516449 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:47.516898 20512 raft_consensus.cc:2749] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:21:47.517017 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:21:47.555022 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 50645.2us
I20260709 12:21:47.555332 17759 raft_consensus.cc:2393] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 7: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:21:47.545684 20422 consensus_queue.cc:1048] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.010s
I20260709 12:21:47.556214 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:47.557422 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
W20260709 12:21:47.557626 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 59011.4us
I20260709 12:21:47.558589 18333 leader_election.cc:304] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:47.560194 20480 raft_consensus.cc:2749] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260709 12:21:47.562417 20658 log.cc:927] Time spent T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.101s	user 0.000s	sys 0.000s
I20260709 12:21:47.569279 20385 leader_election.cc:290] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:47.570624 17755 raft_consensus.cc:2393] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 4: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:21:47.509811 20559 raft_consensus.cc:3060] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:21:47.572299 20662 consensus_queue.cc:1059] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:21:47.573436 20662 raft_consensus.cc:3055] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:21:47.573783 20662 raft_consensus.cc:740] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:21:47.574605 20662 consensus_queue.cc:260] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:47.518172 20656 raft_consensus.cc:2749] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:21:47.516482 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:47.545547 20572 raft_consensus.cc:515] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:47.446254 20459 consensus_queue.cc:237] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:47.577345 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:47.578130 18368 raft_consensus.cc:2393] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 4: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:21:47.578483 18368 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59918 (request call id 1006) took 1114 ms. Trace:
I20260709 12:21:47.579039 20572 leader_election.cc:290] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 7 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:47.518567 20344 consensus_queue.cc:1048] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.131s
I20260709 12:21:47.496623 20346 raft_consensus.cc:493] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:21:47.522083 20532 leader_election.cc:290] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:47.578894 18368 rpcz_store.cc:276] 0709 12:21:46.464307 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:47.576884 (+1112577us) service_pool.cc:224] Handling call
0709 12:21:47.578430 (+  1546us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:47.522552 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:47.583758 17680 raft_consensus.cc:2368] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for earlier term 6. Current term is 7.
I20260709 12:21:47.523849 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:47.528406 20299 consensus_queue.cc:237] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:47.498781 20655 raft_consensus.cc:493] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:47.508358 20397 consensus_queue.cc:1048] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.027s
I20260709 12:21:47.532395 20398 leader_election.cc:290] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:47.532977 20550 consensus_queue.cc:1048] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
W20260709 12:21:47.534399 20653 log.cc:927] Time spent T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.074s	user 0.000s	sys 0.000s
I20260709 12:21:47.535315 20157 leader_election.cc:290] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 8 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:47.536728 17652 leader_election.cc:304] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:21:47.537199 18361 raft_consensus.cc:1240] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 5. Current term is 6. Ops: [3.6-5.7]
I20260709 12:21:47.613109 20480 raft_consensus.cc:2749] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 12:21:47.613289 18361 raft_consensus.cc:1275] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260709 12:21:47.501087 20466 raft_consensus.cc:3060] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:21:47.615091 18361 raft_consensus.cc:1275] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260709 12:21:47.544440 20629 raft_consensus.cc:493] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:21:47.547377 17762 raft_consensus.cc:2468] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 6.
I20260709 12:21:47.616164 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:47.616779 18365 raft_consensus.cc:3060] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:47.589331 20368 raft_consensus.cc:697] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:21:47.620587 18333 leader_election.cc:304] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:47.620812 20368 consensus_queue.cc:237] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:47.622076 18333 leader_election.cc:304] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:47.622982 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 7 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:47.550154 18374 raft_consensus.cc:2468] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 6.
I20260709 12:21:47.624004 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:47.624764 18374 raft_consensus.cc:1240] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 5. Current term is 6. Ops: []
I20260709 12:21:47.623332 18333 leader_election.cc:304] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:21:47.552300 17688 raft_consensus.cc:2410] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for term 7 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260709 12:21:47.497002 20635 consensus_queue.cc:1059] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 5, Time since last communication: 0.026s
I20260709 12:21:47.584838 18371 raft_consensus.cc:1813] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Handling vote request from an unknown peer e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:47.626307 18371 raft_consensus.cc:2393] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 5: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:21:47.584870 19855 raft_consensus.cc:515] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:47.590991 17837 raft_consensus.cc:1275] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260709 12:21:47.627089 17762 raft_consensus.cc:2410] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 7 because replica has last-logged OpId of term: 6 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 12:21:47.592631 17838 raft_consensus.cc:2468] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 5.
I20260709 12:21:47.597359 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:47.628093 17758 raft_consensus.cc:2468] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 7.
I20260709 12:21:47.600021 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:47.575831 20662 raft_consensus.cc:3060] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:47.588217 20370 consensus_queue.cc:1048] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.015s
I20260709 12:21:47.596580 20655 raft_consensus.cc:515] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:47.603201 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 10 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:47.604370 20572 raft_consensus.cc:493] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:47.605068 20550 raft_consensus.cc:2804] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Leader election won for term 5
W20260709 12:21:47.605136 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 68512.6us
I20260709 12:21:47.605885 20512 consensus_queue.cc:1048] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 12:21:47.588024 20346 raft_consensus.cc:3060] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:21:47.546128 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 57719.9us
I20260709 12:21:47.622148 20422 raft_consensus.cc:493] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:47.517580 20574 consensus_queue.cc:237] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:47.652361 18361 raft_consensus.cc:1240] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 4. Current term is 5. Ops: []
I20260709 12:21:47.655196 19855 leader_election.cc:290] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 6 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:21:47.656427 17730 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:47.624154 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:47.637475 20532 raft_consensus.cc:2804] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:21:47.657315 20532 raft_consensus.cc:493] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:21:47.657785 20532 raft_consensus.cc:3060] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:47.630872 17764 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 7.
I20260709 12:21:47.660141 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:47.660470 18365 raft_consensus.cc:2468] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 7.
I20260709 12:21:47.661209 20572 raft_consensus.cc:515] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:47.663314 20572 leader_election.cc:290] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:47.633330 18366 raft_consensus.cc:1275] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260709 12:21:47.664072 18374 raft_consensus.cc:1275] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260709 12:21:47.666926 20655 leader_election.cc:290] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:47.627422 20605 raft_consensus.cc:493] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:47.667588 20605 raft_consensus.cc:515] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:47.675195 17652 leader_election.cc:304] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:21:47.634590 20480 raft_consensus.cc:2749] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:21:47.657505 18368 raft_consensus.cc:2393] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 8: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:21:47.684731 17652 leader_election.cc:304] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
W20260709 12:21:47.687071 17730 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:47.688423 17760 raft_consensus.cc:2468] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 8.
I20260709 12:21:47.582958 17757 raft_consensus.cc:2410] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for term 6 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20260709 12:21:47.646112 20458 raft_consensus.cc:2749] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:21:47.646657 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:47.646806 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 10 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:47.635293 20377 raft_consensus.cc:493] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:21:47.623867 19826 raft_consensus.cc:515] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:47.648801 20559 raft_consensus.cc:515] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:47.640091 17801 leader_election.cc:304] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:21:47.707043 17729 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:47.702946 19855 consensus_queue.cc:1048] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.006s
I20260709 12:21:47.712699 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:21:47.714993 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 75025.5us
I20260709 12:21:47.716404 17801 leader_election.cc:304] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:21:47.719580 17838 raft_consensus.cc:1240] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 5. Current term is 6. Ops: []
I20260709 12:21:47.721220 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:47.616221 20629 raft_consensus.cc:515] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:47.724021 17652 leader_election.cc:304] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:21:47.677093 20550 raft_consensus.cc:697] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 5 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:21:47.748772 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:47.752085 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:47.704270 20144 raft_consensus.cc:2804] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:21:47.771350 20144 raft_consensus.cc:697] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 5 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:21:47.725451 20655 raft_consensus.cc:2804] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:21:47.772372 20655 raft_consensus.cc:697] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 8 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:21:47.772432 20144 consensus_queue.cc:237] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:47.773706 20655 consensus_queue.cc:237] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:47.676898 20227 consensus_queue.cc:1048] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.051s
I20260709 12:21:47.776250 20234 raft_consensus.cc:493] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:47.776862 20234 raft_consensus.cc:515] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:47.779105 20234 leader_election.cc:290] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:47.754113 17761 raft_consensus.cc:2393] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 10: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:21:47.725386 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:47.784157 17837 raft_consensus.cc:3060] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:47.762529 20539 raft_consensus.cc:2804] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:21:47.762744 20398 raft_consensus.cc:493] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:47.725297 20374 raft_consensus.cc:2804] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:21:47.765604 19826 leader_election.cc:290] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 10 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:47.673875 20422 raft_consensus.cc:515] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:47.786026 17758 raft_consensus.cc:1275] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260709 12:21:47.676184 20418 raft_consensus.cc:515] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:21:47.789515 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 65612.7us
I20260709 12:21:47.793964 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:47.812086 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:47.817981 18363 raft_consensus.cc:2468] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 7.
I20260709 12:21:47.814523 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:21:47.821815 17655 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:47.812266 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:47.823545 17834 raft_consensus.cc:2468] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 5.
I20260709 12:21:47.751296 20458 raft_consensus.cc:493] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:47.815766 17757 raft_consensus.cc:1240] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 6. Current term is 7. Ops: []
I20260709 12:21:47.813261 17833 raft_consensus.cc:1275] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260709 12:21:47.796576 17654 leader_election.cc:304] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:21:47.832054 18367 raft_consensus.cc:1240] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 6 LEADER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 5. Current term is 6. Ops: []
I20260709 12:21:47.833240 17728 leader_election.cc:400] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 6 election: Vote denied by peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071) with higher term. Message: Invalid argument: T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for earlier term 6. Current term is 7.
I20260709 12:21:47.790632 17727 leader_election.cc:304] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:21:47.839107 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:47.840180 18364 raft_consensus.cc:2468] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 5.
W20260709 12:21:47.842919 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 47783.9us
I20260709 12:21:47.844164 18368 raft_consensus.cc:1240] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 5. Current term is 6. Ops: []
I20260709 12:21:47.763608 20440 consensus_queue.cc:1048] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.112s
I20260709 12:21:47.847298 20440 consensus_queue.cc:1243] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer e063541542a94969898868bf6eb5ce1b log is divergent from this leader: its last log entry 5.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20260709 12:21:47.847644 20458 raft_consensus.cc:515] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:47.848323 20559 leader_election.cc:290] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 9 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:47.848425 20418 leader_election.cc:290] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 5 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:47.849807 20458 leader_election.cc:290] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:47.850723 20374 raft_consensus.cc:697] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 6 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:21:47.851670 20374 consensus_queue.cc:237] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:47.853813 20629 leader_election.cc:290] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:47.854287 18368 raft_consensus.cc:1240] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 3. Current term is 4. Ops: []
I20260709 12:21:47.794178 20398 raft_consensus.cc:515] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:47.853950 17837 raft_consensus.cc:2468] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 6.
I20260709 12:21:47.857791 17829 raft_consensus.cc:1240] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 5. Current term is 6. Ops: [5.6-5.6]
I20260709 12:21:47.790115 20479 consensus_queue.cc:1048] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.069s
I20260709 12:21:47.858429 20422 leader_election.cc:290] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:47.750553 20377 raft_consensus.cc:515] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:47.860602 20377 leader_election.cc:290] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:47.854553 20234 raft_consensus.cc:493] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:47.864655 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 73007.9us
I20260709 12:21:47.858573 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:21:47.867657 17655 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:47.802527 20370 raft_consensus.cc:2749] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:21:47.872495 18333 leader_election.cc:304] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
W20260709 12:21:47.870359 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 119108us
I20260709 12:21:47.787215 20539 raft_consensus.cc:493] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:21:47.869553 17830 raft_consensus.cc:2410] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for term 7 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260709 12:21:47.876541 17685 raft_consensus.cc:1240] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 6 LEADER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 5. Current term is 6. Ops: []
I20260709 12:21:47.873307 17729 leader_election.cc:304] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
W20260709 12:21:47.875355 20653 log.cc:927] Time spent T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.056s	user 0.000s	sys 0.000s
I20260709 12:21:47.878306 17762 raft_consensus.cc:1275] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260709 12:21:47.879153 17802 leader_election.cc:304] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:21:47.875547 20440 raft_consensus.cc:493] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:47.876050 20512 raft_consensus.cc:2804] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Leader election won for term 6
W20260709 12:21:47.887965 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 91509.9us
I20260709 12:21:47.896353 17729 leader_election.cc:304] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:21:47.877789 18333 leader_election.cc:304] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
W20260709 12:21:47.898231 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 243733us
I20260709 12:21:47.904486 17727 leader_election.cc:304] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:21:47.880246 20418 raft_consensus.cc:2804] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:21:47.908586 17654 leader_election.cc:304] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:21:47.909162 20422 raft_consensus.cc:2749] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:21:47.901293 20440 raft_consensus.cc:515] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:47.909647 20559 raft_consensus.cc:2749] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:21:47.909600 20397 raft_consensus.cc:2749] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:21:47.911676 20368 raft_consensus.cc:2804] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:21:47.879750 20626 raft_consensus.cc:493] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:47.884430 20479 raft_consensus.cc:2749] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:21:47.873265 20234 raft_consensus.cc:515] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:21:47.919473 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 45586.5us
I20260709 12:21:47.920307 17802 leader_election.cc:304] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:21:47.920827 17729 leader_election.cc:304] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:21:47.917083 20396 raft_consensus.cc:2804] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:21:47.914949 20418 raft_consensus.cc:493] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:47.878711 20539 raft_consensus.cc:3060] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:47.884132 20525 raft_consensus.cc:493] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:47.913378 20656 raft_consensus.cc:2749] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:21:47.917289 20299 raft_consensus.cc:493] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:47.918092 20368 raft_consensus.cc:493] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:21:47.913635 18333 leader_election.cc:304] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
W20260709 12:21:47.936255 17801 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:47.924278 20385 raft_consensus.cc:3060] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:21:47.984097 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 63899.4us
I20260709 12:21:47.934082 20418 raft_consensus.cc:3060] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:21:47.995816 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 82255.6us
I20260709 12:21:47.996819 18333 leader_election.cc:304] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
W20260709 12:21:47.997896 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 70374.3us
W20260709 12:21:47.998457 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 87041.5us
I20260709 12:21:47.922951 20349 consensus_queue.cc:1048] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.061s
I20260709 12:21:47.940140 20479 raft_consensus.cc:2749] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:21:47.913221 20550 consensus_queue.cc:237] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:47.971149 20368 raft_consensus.cc:3060] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:47.920210 20626 raft_consensus.cc:515] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:48.004272 17729 leader_election.cc:304] T 7cb3768d215e409fb9f5baf18d124442 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
W20260709 12:21:48.007360 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 94332.8us
I20260709 12:21:48.013214 17803 leader_election.cc:304] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:48.014981 18333 leader_election.cc:304] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:48.016692 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:47.935021 20396 raft_consensus.cc:697] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 7 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:21:47.930892 20422 consensus_queue.cc:1048] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.070s
I20260709 12:21:47.941025 20525 raft_consensus.cc:515] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:48.031572 17727 leader_election.cc:304] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:21:48.026202 18333 leader_election.cc:304] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:48.001332 20662 raft_consensus.cc:2749] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:21:48.032789 19855 raft_consensus.cc:2749] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:21:48.033222 18333 leader_election.cc:304] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:48.038434 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:48.003679 20572 raft_consensus.cc:493] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:48.040170 17683 raft_consensus.cc:3060] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:21:48.041330 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:48.041268 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:48.041992 17830 raft_consensus.cc:3060] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:48.006057 20629 raft_consensus.cc:493] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:48.041997 17836 raft_consensus.cc:2468] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 6.
I20260709 12:21:47.985455 20299 raft_consensus.cc:515] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:47.994441 20466 raft_consensus.cc:515] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:48.077862 17802 leader_election.cc:304] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:21:48.081524 17687 raft_consensus.cc:1275] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260709 12:21:48.022433 20621 raft_consensus.cc:2804] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:21:48.010989 20674 raft_consensus.cc:2804] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:21:48.030503 20584 raft_consensus.cc:3060] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:48.087073 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
W20260709 12:21:48.093426 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 60266.7us
I20260709 12:21:48.009927 20656 consensus_queue.cc:1059] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.076s
I20260709 12:21:48.077659 20512 raft_consensus.cc:697] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:21:48.117620 18333 leader_election.cc:304] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:48.028295 20336 consensus_queue.cc:1059] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.059s
W20260709 12:21:48.107949 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 87962.6us
I20260709 12:21:48.083683 20480 raft_consensus.cc:2749] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:21:48.085129 20621 raft_consensus.cc:493] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:48.030028 20574 consensus_queue.cc:1048] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.057s
I20260709 12:21:48.122398 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:48.131619 17689 raft_consensus.cc:2468] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 6.
W20260709 12:21:48.121836 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 98463.3us
I20260709 12:21:48.144327 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:48.145212 18365 raft_consensus.cc:2468] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 6.
I20260709 12:21:48.149228 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 10 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:48.149861 17687 raft_consensus.cc:3060] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:21:48.152086 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:48.077651 20572 raft_consensus.cc:515] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:48.079625 20629 raft_consensus.cc:515] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:48.164882 17762 raft_consensus.cc:1275] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:21:48.166949 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:48.166101 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:48.109890 20683 raft_consensus.cc:3060] T 7cb3768d215e409fb9f5baf18d124442 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:48.180071 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 10 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:48.123404 20349 raft_consensus.cc:2804] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:21:48.178189 18368 raft_consensus.cc:1240] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 6. Current term is 7. Ops: []
W20260709 12:21:48.186578 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 142564us
I20260709 12:21:48.142041 20621 raft_consensus.cc:3060] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:21:48.123608 20674 raft_consensus.cc:697] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 5 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:21:48.188199 17802 leader_election.cc:304] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:21:48.193987 17687 raft_consensus.cc:2468] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 10.
I20260709 12:21:48.197695 18362 raft_consensus.cc:2393] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 10: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:21:48.105283 20479 consensus_queue.cc:1048] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.077s
I20260709 12:21:48.092705 20679 consensus_queue.cc:1048] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.095s
W20260709 12:21:48.230109 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 135549us
I20260709 12:21:48.178508 20635 raft_consensus.cc:493] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:48.178876 20480 raft_consensus.cc:2749] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:21:48.154078 20515 consensus_queue.cc:1059] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.066s
I20260709 12:21:48.198956 20440 leader_election.cc:290] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:48.235775 18333 leader_election.cc:304] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:48.221618 20234 leader_election.cc:290] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:48.229461 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:48.234843 20688 raft_consensus.cc:2749] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:21:48.265394 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:48.268620 18362 raft_consensus.cc:2468] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 6.
W20260709 12:21:48.249146 20670 log.cc:927] Time spent T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.327s	user 0.001s	sys 0.001s
I20260709 12:21:48.245472 20655 raft_consensus.cc:3055] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:21:48.220628 20396 consensus_queue.cc:237] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:21:48.277956 20678 log.cc:927] Time spent T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.096s	user 0.000s	sys 0.000s
I20260709 12:21:48.275429 20689 consensus_queue.cc:1048] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.022s
W20260709 12:21:48.255474 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 96692.8us
I20260709 12:21:48.244889 20635 raft_consensus.cc:515] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:21:48.304378 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 67447.5us
I20260709 12:21:48.303746 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 10 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:48.238184 20686 consensus_queue.cc:1048] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.052s
W20260709 12:21:48.307422 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 275887us
W20260709 12:21:48.307658 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 119545us
I20260709 12:21:48.312386 17680 raft_consensus.cc:2393] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 10: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:21:48.312800 17802 leader_election.cc:304] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
W20260709 12:21:48.297360 20675 log.cc:927] Time spent T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.290s	user 0.001s	sys 0.000s
I20260709 12:21:48.274585 20574 raft_consensus.cc:2749] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:21:48.321183 18333 leader_election.cc:304] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:48.324167 18362 raft_consensus.cc:1275] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20260709 12:21:48.326858 18333 leader_election.cc:304] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:48.278160 20702 raft_consensus.cc:3055] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 LEADER]: Stepping down as leader of term 4
W20260709 12:21:48.331313 20669 log.cc:927] Time spent T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.282s	user 0.000s	sys 0.001s
W20260709 12:21:48.331631 20681 log.cc:927] Time spent T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.222s	user 0.001s	sys 0.000s
I20260709 12:21:48.279726 20655 raft_consensus.cc:740] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 3 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:21:48.333504 20635 leader_election.cc:290] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:48.291424 20397 raft_consensus.cc:493] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
W20260709 12:21:48.338703 20685 log.cc:927] Time spent T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.104s	user 0.000s	sys 0.000s
I20260709 12:21:48.338485 20701 consensus_queue.cc:1059] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:21:48.338835 20397 raft_consensus.cc:515] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:21:48.339743 20676 log.cc:927] Time spent T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.294s	user 0.001s	sys 0.001s
I20260709 12:21:48.340112 20440 raft_consensus.cc:3055] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:21:48.323400 20234 raft_consensus.cc:493] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:48.340539 20440 raft_consensus.cc:740] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:21:48.341197 20397 leader_election.cc:290] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:48.341629 20440 consensus_queue.cc:260] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:48.343281 20440 raft_consensus.cc:3060] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:48.344918 20299 leader_election.cc:290] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:48.339655 20701 consensus_queue.cc:1048] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.006s
W20260709 12:21:48.345708 20532 consensus_meta.cc:287] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.687s	user 0.003s	sys 0.007s
I20260709 12:21:48.345980 20701 consensus_queue.cc:1243] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer e2909cff1da44ad387f96a8ae3aa5afb log is divergent from this leader: its last log entry 5.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260709 12:21:48.346129 20532 raft_consensus.cc:515] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:48.347275 18374 raft_consensus.cc:1240] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 3. Current term is 4. Ops: []
I20260709 12:21:48.347347 17830 raft_consensus.cc:2468] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 5.
I20260709 12:21:48.273187 20512 consensus_queue.cc:237] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:48.340883 20234 raft_consensus.cc:515] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:48.354174 18368 raft_consensus.cc:1275] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:21:48.357358 18333 leader_election.cc:304] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:48.334463 20702 raft_consensus.cc:740] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
W20260709 12:21:48.376907 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 64198.6us
I20260709 12:21:48.334228 20662 raft_consensus.cc:2749] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:21:48.391245 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:48.405690 18368 raft_consensus.cc:2468] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 6.
I20260709 12:21:48.350869 20466 leader_election.cc:290] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 10 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:48.388688 17830 pending_rounds.cc:85] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b: Aborting all ops after (but not including) 2
I20260709 12:21:48.438184 17830 pending_rounds.cc:107] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b: Aborting uncommitted NO_OP operation due to leader change: 5.3
I20260709 12:21:48.439114 17830 raft_consensus.cc:2889] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:21:48.441887 20693 raft_consensus.cc:493] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:21:48.442688 20693 raft_consensus.cc:515] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:48.354529 20574 raft_consensus.cc:2749] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:21:48.376893 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 9 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:48.449529 17763 raft_consensus.cc:2410] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for term 9 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260709 12:21:48.367789 20635 raft_consensus.cc:2749] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:21:48.450349 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:48.449743 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 9 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:48.452965 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:48.457921 18368 raft_consensus.cc:2468] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 6.
I20260709 12:21:48.456926 17831 raft_consensus.cc:2468] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 7.
I20260709 12:21:48.453173 17802 leader_election.cc:304] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:48.384495 20525 leader_election.cc:290] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:48.376148 20690 raft_consensus.cc:2749] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:21:48.467236 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:21:48.449546 20346 consensus_meta.cc:287] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.768s	user 0.012s	sys 0.003s
I20260709 12:21:48.471729 17802 leader_election.cc:304] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:21:48.370859 20480 consensus_queue.cc:1048] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.083s
I20260709 12:21:48.447283 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:48.467133 20349 raft_consensus.cc:697] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 6 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:21:48.485862 17689 raft_consensus.cc:1240] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 7 LEADER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 6. Current term is 7. Ops: []
I20260709 12:21:48.456002 20532 leader_election.cc:290] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 4 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:48.485447 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:48.459342 20479 raft_consensus.cc:493] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:48.484499 20707 raft_consensus.cc:493] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:21:48.483634 20572 leader_election.cc:290] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:48.486999 20629 leader_election.cc:290] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:48.507958 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:48.515208 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:48.494673 20336 raft_consensus.cc:2749] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:21:48.334571 20674 consensus_queue.cc:237] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:48.518102 17684 raft_consensus.cc:1275] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 5. (term mismatch)
I20260709 12:21:48.519227 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:48.522698 17680 raft_consensus.cc:2468] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 6.
I20260709 12:21:48.495806 20700 raft_consensus.cc:2749] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:21:48.522169 17684 pending_rounds.cc:85] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3: Aborting all ops after (but not including) 4
I20260709 12:21:48.536526 17684 pending_rounds.cc:107] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3: Aborting uncommitted NO_OP operation due to leader change: 3.5
I20260709 12:21:48.536823 17683 raft_consensus.cc:2468] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 9.
I20260709 12:21:48.536844 17684 raft_consensus.cc:2889] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:21:48.538295 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:48.539376 17683 raft_consensus.cc:2393] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 8: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:21:48.539229 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 7 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:48.540272 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 10 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:48.487221 20346 raft_consensus.cc:515] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:48.545044 17763 raft_consensus.cc:2393] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 10: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:21:48.546378 18366 raft_consensus.cc:1240] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 5. Current term is 6. Ops: []
I20260709 12:21:48.545877 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:48.550532 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:48.551434 17688 raft_consensus.cc:3060] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:48.553646 17686 raft_consensus.cc:1275] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260709 12:21:48.482303 20655 consensus_queue.cc:260] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:48.558077 18371 raft_consensus.cc:1240] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 6. Current term is 7. Ops: []
I20260709 12:21:48.507414 20479 raft_consensus.cc:515] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:48.562588 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:48.566299 17833 raft_consensus.cc:2468] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 6.
I20260709 12:21:48.546897 20157 raft_consensus.cc:3055] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:21:48.570029 20706 raft_consensus.cc:493] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:48.569880 20704 consensus_queue.cc:1059] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.061s
I20260709 12:21:48.600364 20706 raft_consensus.cc:515] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:48.570995 17755 raft_consensus.cc:3055] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:21:48.576216 20349 consensus_queue.cc:237] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:48.517218 20702 consensus_queue.cc:260] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:48.578245 20655 raft_consensus.cc:3060] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:48.544308 20696 raft_consensus.cc:2749] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:21:48.584659 20693 leader_election.cc:290] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:48.584954 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:48.585049 20683 raft_consensus.cc:2749] T 7cb3768d215e409fb9f5baf18d124442 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:21:48.586577 20346 leader_election.cc:290] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 7 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:48.520514 20707 raft_consensus.cc:515] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:21:48.590684 20539 consensus_meta.cc:287] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.598s	user 0.021s	sys 0.000s
I20260709 12:21:48.614562 20706 leader_election.cc:290] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:48.590911 20234 leader_election.cc:290] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:21:48.624869 20667 log.cc:927] Time spent T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.763s	user 0.005s	sys 0.000s
I20260709 12:21:48.629568 17688 raft_consensus.cc:2468] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 6.
W20260709 12:21:48.632572 20368 consensus_meta.cc:287] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.603s	user 0.017s	sys 0.000s
W20260709 12:21:48.633177 20385 consensus_meta.cc:287] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.640s	user 0.013s	sys 0.003s
W20260709 12:21:48.634370 20418 consensus_meta.cc:287] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.631s	user 0.015s	sys 0.001s
I20260709 12:21:48.634493 17837 raft_consensus.cc:1275] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260709 12:21:48.666133 20418 raft_consensus.cc:515] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:48.563961 20700 raft_consensus.cc:493] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:21:48.638389 18333 leader_election.cc:304] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:21:48.679740 20385 raft_consensus.cc:2749] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:21:48.680655 18333 leader_election.cc:304] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:48.681401 20346 raft_consensus.cc:2804] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:21:48.672137 20368 raft_consensus.cc:515] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:48.682118 18333 leader_election.cc:304] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:48.682390 20480 raft_consensus.cc:2804] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:21:48.683013 20532 raft_consensus.cc:2804] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:21:48.683588 20532 raft_consensus.cc:697] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 5 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:21:48.684031 20368 leader_election.cc:290] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 6 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:48.684526 20532 consensus_queue.cc:237] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:48.686640 20707 leader_election.cc:290] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:21:48.687454 20671 log.cc:927] Time spent T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.792s	user 0.003s	sys 0.000s
I20260709 12:21:48.608592 20704 consensus_queue.cc:1243] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer 0da169ff82674dc1947055ce43de68a3 log is divergent from this leader: its last log entry 6.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
I20260709 12:21:48.683015 20480 raft_consensus.cc:493] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:48.689371 20480 raft_consensus.cc:3060] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:48.610343 17755 raft_consensus.cc:740] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 5 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:21:48.690893 20621 raft_consensus.cc:515] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:48.682164 20346 raft_consensus.cc:697] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 6 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:21:48.647347 20539 raft_consensus.cc:515] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:48.691431 17755 consensus_queue.cc:260] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:48.692408 20346 consensus_queue.cc:237] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:48.692899 17755 raft_consensus.cc:3060] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:48.693842 20539 leader_election.cc:290] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 4 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:48.594498 20157 raft_consensus.cc:740] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 5 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:21:48.697150 20157 consensus_queue.cc:260] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:48.698313 20157 raft_consensus.cc:3060] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:48.699532 17682 raft_consensus.cc:2393] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 4: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:21:48.679766 20700 raft_consensus.cc:515] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:48.700517 18333 leader_election.cc:304] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:21:48.704238 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:48.704847 18333 leader_election.cc:304] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:48.649170 20702 raft_consensus.cc:3060] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:48.710593 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:48.711341 17760 raft_consensus.cc:3060] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Advancing to term 9
I20260709 12:21:48.713654 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:48.714380 17837 raft_consensus.cc:2393] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 5: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:21:48.729063 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:48.699597 20479 leader_election.cc:290] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:48.713469 20574 raft_consensus.cc:2749] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:21:48.782796 18366 raft_consensus.cc:1240] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 5. Current term is 6. Ops: [5.8-5.8]
W20260709 12:21:48.774360 17728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:48.720309 20466 consensus_queue.cc:1048] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.041s
I20260709 12:21:48.739583 17760 raft_consensus.cc:2468] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 9.
I20260709 12:21:48.793864 17728 leader_election.cc:304] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:21:48.763574 20673 raft_consensus.cc:2804] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:21:48.796038 17728 leader_election.cc:304] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:21:48.796591 20673 raft_consensus.cc:493] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:48.796824 20704 raft_consensus.cc:493] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:21:48.796975 17803 leader_election.cc:304] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:21:48.797044 20673 raft_consensus.cc:3060] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:21:48.763370 20672 log.cc:927] Time spent T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.778s	user 0.000s	sys 0.001s
I20260709 12:21:48.797562 20704 raft_consensus.cc:515] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:48.799808 20704 leader_election.cc:290] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:48.800634 20512 raft_consensus.cc:2804] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:21:48.801157 20385 raft_consensus.cc:2804] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:21:48.801187 20512 raft_consensus.cc:697] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:21:48.801597 20385 raft_consensus.cc:493] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:48.802000 20385 raft_consensus.cc:3060] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:48.802398 20512 consensus_queue.cc:237] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.10, Last appended by leader: 10, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:48.800524 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:48.809540 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:48.809233 17837 raft_consensus.cc:2468] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 8.
I20260709 12:21:48.811666 18366 raft_consensus.cc:2368] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 8. Current term is 9.
I20260709 12:21:48.814142 20336 consensus_queue.cc:1059] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.002s
I20260709 12:21:48.768769 20626 leader_election.cc:290] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:48.762938 20700 leader_election.cc:290] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:48.812233 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:48.817833 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:48.818353 17837 raft_consensus.cc:2468] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 6.
I20260709 12:21:48.759514 20674 raft_consensus.cc:493] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:21:48.738574 20418 leader_election.cc:290] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 9 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:48.762131 20479 raft_consensus.cc:493] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:21:48.824108 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:48.810424 20711 raft_consensus.cc:2804] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:21:48.825536 17837 raft_consensus.cc:3060] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:21:48.820426 20707 raft_consensus.cc:493] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:48.831290 17803 leader_election.cc:400] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 8 pre-election: Vote denied by peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577) with higher term. Message: Invalid argument: T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 8. Current term is 9.
W20260709 12:21:48.825093 20584 consensus_meta.cc:287] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.704s	user 0.012s	sys 0.002s
I20260709 12:21:48.837736 17803 leader_election.cc:403] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 8 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:21:48.806958 20683 consensus_queue.cc:1048] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.019s
I20260709 12:21:48.771837 18361 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59886 (request call id 878) took 1233 ms. Trace:
I20260709 12:21:48.825258 17755 raft_consensus.cc:1275] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260709 12:21:48.826201 20479 raft_consensus.cc:515] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:48.826954 20711 raft_consensus.cc:493] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:48.844014 18368 raft_consensus.cc:1275] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20260709 12:21:48.845630 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:48.841037 20584 raft_consensus.cc:2749] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:21:48.831656 20707 raft_consensus.cc:515] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:48.843720 20715 raft_consensus.cc:3060] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Advancing to term 9
I20260709 12:21:48.824975 20674 raft_consensus.cc:515] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:48.835687 20385 raft_consensus.cc:515] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:48.843791 20621 leader_election.cc:290] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 8 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:48.845783 20711 raft_consensus.cc:3060] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:48.861595 20704 consensus_queue.cc:1048] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.008s
I20260709 12:21:48.865904 20479 leader_election.cc:290] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:48.875028 17802 leader_election.cc:304] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:21:48.876744 17802 leader_election.cc:304] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:21:48.879144 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:48.881517 17802 leader_election.cc:304] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:48.883944 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:48.884559 17688 raft_consensus.cc:2468] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 6.
I20260709 12:21:48.884931 20707 leader_election.cc:290] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:48.885392 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:48.885497 20731 raft_consensus.cc:2749] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:21:48.886082 17687 raft_consensus.cc:2368] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 8. Current term is 9.
I20260709 12:21:48.885898 20480 raft_consensus.cc:515] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:48.886482 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:48.887140 17681 raft_consensus.cc:2468] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 7.
I20260709 12:21:48.882501 20621 raft_consensus.cc:2749] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:21:48.889219 20674 leader_election.cc:290] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:48.891423 20704 consensus_queue.cc:1048] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:21:48.892344 20655 raft_consensus.cc:2804] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:21:48.892763 20655 raft_consensus.cc:493] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:48.894034 17802 leader_election.cc:304] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:21:48.896968 17802 leader_election.cc:304] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:21:48.898746 17837 raft_consensus.cc:2468] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 9.
I20260709 12:21:48.899712 18333 leader_election.cc:304] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:21:48.901798 17802 leader_election.cc:304] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:21:48.902666 20374 raft_consensus.cc:2804] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:21:48.902938 17802 leader_election.cc:400] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 8 pre-election: Vote denied by peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071) with higher term. Message: Invalid argument: T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 8. Current term is 9.
I20260709 12:21:48.905135 20704 consensus_queue.cc:1048] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 2, Time since last communication: 0.008s
I20260709 12:21:48.908985 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:48.909654 17833 raft_consensus.cc:2468] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 6.
I20260709 12:21:48.910446 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:48.910957 17833 raft_consensus.cc:2468] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 6.
I20260709 12:21:48.904016 20621 raft_consensus.cc:2749] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:21:48.842453 18361 rpcz_store.cc:276] 0709 12:21:47.538486 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:47.653298 (+114812us) service_pool.cc:224] Handling call
0709 12:21:47.653651 (+   353us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:47.869294 (+215643us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 12:21:47.903238 (+ 33944us) log.cc:844] Serialized 14 byte log entry
0709 12:21:48.040875 (+137637us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:48.047436 (+  6561us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:21:48.078522 (+ 31086us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:48.771571 (+693049us) spinlock_profiling.cc:243] Waited 150 ms on lock 0x7b5400802d54. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:21:48.771634 (+    63us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:48.771773 (+   139us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":8390,"prepare.queue_time_us":21960,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":42,"raft.queue_time_us":9100,"raft.run_cpu_time_us":118,"raft.run_wall_time_us":7374,"spinlock_wait_cycles":343973120,"thread_start_us":96159,"threads_started":1,"wal-append.queue_time_us":260172}
I20260709 12:21:48.914315 20731 raft_consensus.cc:2749] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:21:48.893128 20655 raft_consensus.cc:3060] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:48.920684 20480 leader_election.cc:290] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 7 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:48.926348 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:48.903119 20374 raft_consensus.cc:493] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:48.924737 17802 leader_election.cc:304] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:21:48.928022 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:48.911706 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:48.927475 20374 raft_consensus.cc:3060] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:48.932328 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:48.933208 17836 raft_consensus.cc:3060] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:48.935678 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:48.936853 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:48.937096 20385 leader_election.cc:290] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 7 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:48.937932 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:48.938696 17837 raft_consensus.cc:1813] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Handling vote request from an unknown peer e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:48.938838 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:48.939535 18361 raft_consensus.cc:3060] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:48.940081 20349 raft_consensus.cc:493] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:21:48.940678 20349 raft_consensus.cc:515] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:48.941841 17833 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d"
dest_uuid: "e063541542a94969898868bf6eb5ce1b"
mode: GRACEFUL
new_leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
 from {username='slave'} at 127.0.0.1:59820
I20260709 12:21:48.942802 20674 raft_consensus.cc:2804] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:21:48.939368 17837 raft_consensus.cc:2393] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 5: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:21:48.946780 17833 raft_consensus.cc:606] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Received request to transfer leadership to TS e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:48.947207 17833 raft_consensus.cc:614] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:21:48.943509 20674 raft_consensus.cc:697] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 9 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:21:48.927346 20673 raft_consensus.cc:515] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:48.950134 17836 raft_consensus.cc:2468] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 7.
I20260709 12:21:48.954165 18333 leader_election.cc:304] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:21:48.930968 20715 raft_consensus.cc:2749] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Leader pre-election lost for term 8. Reason: Vote denied by peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577) with higher term. Message: Invalid argument: T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 8. Current term is 9.
I20260709 12:21:48.950806 20385 raft_consensus.cc:2804] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:21:48.964326 20605 leader_election.cc:290] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:48.964886 20673 leader_election.cc:290] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 7 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:48.965941 20674 consensus_queue.cc:237] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 7.9, Last appended by leader: 9, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:48.977124 20711 raft_consensus.cc:515] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:48.972747 20385 raft_consensus.cc:493] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:48.980682 20385 raft_consensus.cc:3060] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:21:48.958575 20559 raft_consensus.cc:2804] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:21:48.984400 20559 raft_consensus.cc:697] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 7 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:21:48.985450 20559 consensus_queue.cc:237] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:48.968336 20398 leader_election.cc:290] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:48.969576 20655 raft_consensus.cc:515] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:48.973280 17833 raft_consensus.cc:1240] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 4. Current term is 5. Ops: []
I20260709 12:21:48.943634 20349 leader_election.cc:290] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:48.990841 17837 raft_consensus.cc:1240] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 7. Current term is 8. Ops: [7.8-7.8]
I20260709 12:21:48.992295 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:48.999996 17833 raft_consensus.cc:3060] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:49.002347 20711 leader_election.cc:290] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 7 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:49.031601 17837 raft_consensus.cc:1240] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 4. Current term is 5. Ops: []
I20260709 12:21:49.003409 20374 raft_consensus.cc:515] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:49.048174 20629 raft_consensus.cc:493] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:49.074954 17832 raft_consensus.cc:1240] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 5. Current term is 6. Ops: [5.6-5.6]
I20260709 12:21:49.048233 20655 leader_election.cc:290] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 7 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:49.051102 20629 raft_consensus.cc:515] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:49.214505 17833 raft_consensus.cc:2468] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 7.
I20260709 12:21:49.155318 20687 raft_consensus.cc:493] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:49.217306 20374 leader_election.cc:290] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 7 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:49.223976 17835 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35040 (request call id 2026) took 1413 ms. Trace:
I20260709 12:21:49.224411 17835 rpcz_store.cc:276] 0709 12:21:47.810523 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:47.810638 (+   115us) service_pool.cc:224] Handling call
0709 12:21:47.810980 (+   342us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:47.842645 (+ 31665us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:47.842682 (+    37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:47.863625 (+ 20943us) log.cc:844] Serialized 26 byte log entry
0709 12:21:48.003767 (+140142us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:21:48.139918 (+136151us) log.cc:844] Serialized 14 byte log entry
0709 12:21:48.363936 (+224018us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:48.364042 (+   106us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:49.219623 (+855581us) spinlock_profiling.cc:243] Waited 11.5 ms on lock 0x7b1800134140. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b475012a6 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b475333f3 00007f7b475357c1
0709 12:21:49.220436 (+   813us) spinlock_profiling.cc:243] Waited 9.9 ms on lock 0x7b1800134140. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b475333f3 00007f7b475357c1 00007f7b47535779 00007f7b47535709 00007f7b475356d1 00007f7b475349cd 0000000000388634 0000000000388479 00007f7b4751683a
0709 12:21:49.223736 (+  3300us) raft_consensus.cc:1685] finished
0709 12:21:49.223759 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:49.223874 (+   115us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":65329,"prepare.queue_time_us":112188,"prepare.run_cpu_time_us":130,"prepare.run_wall_time_us":130,"raft.queue_time_us":195234,"raft.run_cpu_time_us":315,"raft.run_wall_time_us":62158,"spinlock_wait_cycles":49151232,"thread_start_us":438747,"threads_started":3,"wal-append.queue_time_us":264156}
I20260709 12:21:49.229166 20629 leader_election.cc:290] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:49.223323 20466 raft_consensus.cc:493] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:21:49.230486 20466 raft_consensus.cc:515] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:49.232688 20466 leader_election.cc:290] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:49.234292 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:49.236131 17832 raft_consensus.cc:1275] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260709 12:21:49.236680 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:49.237284 18368 raft_consensus.cc:2468] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 6.
I20260709 12:21:49.238222 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:49.239653 18368 raft_consensus.cc:1240] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 7 LEADER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 6. Current term is 7. Ops: [6.7-6.7]
W20260709 12:21:49.242128 20677 log.cc:927] Time spent T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 1.132s	user 0.003s	sys 0.000s
I20260709 12:21:49.259677 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:49.259969 20332 consensus_queue.cc:1059] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:21:49.260489 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:49.260874 20332 raft_consensus.cc:3055] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:21:49.261051 17686 raft_consensus.cc:3060] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:49.261286 20332 raft_consensus.cc:740] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:21:49.262207 20332 consensus_queue.cc:260] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:49.263356 20332 raft_consensus.cc:3060] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:49.242924 20656 raft_consensus.cc:3055] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:21:49.264130 20656 raft_consensus.cc:740] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 5 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:21:49.265030 20656 consensus_queue.cc:260] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.8, Last appended by leader: 8, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:49.266024 20656 raft_consensus.cc:3060] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:49.268503 17686 raft_consensus.cc:2468] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 7.
I20260709 12:21:49.279320 18363 raft_consensus.cc:1240] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 5 LEADER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 4. Current term is 5. Ops: []
I20260709 12:21:49.243819 20385 raft_consensus.cc:515] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:21:49.246716 17652 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.16.29.67:36837 timed out after 1.504s (SENT) [suppressed 29 similar messages]
I20260709 12:21:49.247526 18361 raft_consensus.cc:2468] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 7.
I20260709 12:21:49.237243 20686 raft_consensus.cc:493] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:21:49.251250 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:21:49.256737 17728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:49.257442 17802 leader_election.cc:304] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:49.257877 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:49.283363 17689 raft_consensus.cc:3060] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:49.284216 20374 raft_consensus.cc:3060] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:49.258277 17681 raft_consensus.cc:1275] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260709 12:21:49.258734 17729 leader_election.cc:304] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:21:49.259214 17682 raft_consensus.cc:1275] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260709 12:21:49.288933 20686 raft_consensus.cc:515] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:49.292603 17802 leader_election.cc:304] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:21:49.294811 18367 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59886 (request call id 862) took 2028 ms. Trace:
I20260709 12:21:49.295261 17829 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35040 (request call id 2025) took 1485 ms. Trace:
I20260709 12:21:49.295202 18367 rpcz_store.cc:276] 0709 12:21:47.266279 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:47.266942 (+   663us) service_pool.cc:224] Handling call
0709 12:21:47.278473 (+ 11531us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:47.833616 (+555143us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:49.294592 (+1460976us) spinlock_profiling.cc:243] Waited 553 ms on lock 0x7b540047ca54. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:21:49.294757 (+   165us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1272617856}
I20260709 12:21:49.295810 17829 rpcz_store.cc:276] 0709 12:21:47.810202 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:47.814688 (+  4486us) service_pool.cc:224] Handling call
0709 12:21:47.857550 (+ 42862us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:47.858247 (+   697us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:49.295036 (+1436789us) spinlock_profiling.cc:243] Waited 42.5 ms on lock 0x7b540008cfd0. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a61f8ad 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:21:49.295192 (+   156us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":97643776}
I20260709 12:21:49.298344 17689 raft_consensus.cc:2468] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 7.
I20260709 12:21:49.281275 20559 raft_consensus.cc:493] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:21:49.301707 17729 leader_election.cc:304] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:21:49.305028 20706 consensus_queue.cc:1048] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:21:49.305191 20559 raft_consensus.cc:515] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:49.307389 18362 raft_consensus.cc:2468] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 4.
I20260709 12:21:49.304597 17802 leader_election.cc:304] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:21:49.324077 20571 raft_consensus.cc:2804] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:21:49.314457 20697 consensus_queue.cc:1048] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.010s
I20260709 12:21:49.324573 20571 raft_consensus.cc:493] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:49.324962 20571 raft_consensus.cc:3060] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:49.325170 18374 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59902 (request call id 843) took 1647 ms. Trace:
I20260709 12:21:49.307075 20635 raft_consensus.cc:2804] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:21:49.307982 19826 raft_consensus.cc:2804] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:21:49.325615 18374 rpcz_store.cc:276] 0709 12:21:47.677584 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:47.677826 (+   242us) service_pool.cc:224] Handling call
0709 12:21:47.709399 (+ 31573us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:48.347818 (+638419us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:49.324918 (+977100us) spinlock_profiling.cc:243] Waited 638 ms on lock 0x7b5400570154. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:21:49.325105 (+   187us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1466430336}
I20260709 12:21:49.314117 17829 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f"
dest_uuid: "e063541542a94969898868bf6eb5ce1b"
mode: GRACEFUL
new_leader_uuid: "0da169ff82674dc1947055ce43de68a3"
 from {username='slave'} at 127.0.0.1:59820
I20260709 12:21:49.326968 17829 raft_consensus.cc:606] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 4 LEADER]: Received request to transfer leadership to TS 0da169ff82674dc1947055ce43de68a3
I20260709 12:21:49.326246 19826 raft_consensus.cc:493] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:49.327569 19826 raft_consensus.cc:3060] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:49.227288 20687 raft_consensus.cc:515] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:49.338781 17729 leader_election.cc:304] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:21:49.314401 20422 consensus_queue.cc:1059] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.025s
I20260709 12:21:49.333704 20336 raft_consensus.cc:2804] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:21:49.348268 20336 raft_consensus.cc:697] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 7 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:21:49.349198 20336 consensus_queue.cc:237] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:49.352240 20368 raft_consensus.cc:3055] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:21:49.352689 20368 raft_consensus.cc:740] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:21:49.353979 20559 leader_election.cc:290] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:49.341154 20635 raft_consensus.cc:697] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 7 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:21:49.355969 20635 consensus_queue.cc:237] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:49.359566 17685 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37402 (request call id 2292) took 2200 ms. Trace:
I20260709 12:21:49.353516 20368 consensus_queue.cc:260] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:49.359972 17685 rpcz_store.cc:276] 0709 12:21:47.159450 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:47.162324 (+  2874us) service_pool.cc:224] Handling call
0709 12:21:47.162973 (+   649us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:47.877073 (+714100us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:49.359350 (+1482277us) spinlock_profiling.cc:243] Waited 713 ms on lock 0x7b54003458d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:21:49.359508 (+   158us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1640626944}
I20260709 12:21:49.363854 17834 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35040 (request call id 2027) took 1518 ms. Trace:
I20260709 12:21:49.364293 17834 rpcz_store.cc:276] 0709 12:21:47.845448 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:47.845672 (+   224us) service_pool.cc:224] Handling call
0709 12:21:47.846124 (+   452us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:21:47.873952 (+ 27828us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:47.873997 (+    45us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:21:47.885804 (+ 11807us) log.cc:844] Serialized 52 byte log entry
0709 12:21:48.001364 (+115560us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:21:48.023133 (+ 21769us) log.cc:844] Serialized 14 byte log entry
0709 12:21:48.089034 (+ 65901us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:48.090934 (+  1900us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:49.295035 (+1204101us) spinlock_profiling.cc:243] Waited 12 ms on lock 0x7b18002db380. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b475333f3 00007f7b475357c1 00007f7b47535779 00007f7b47535709 00007f7b475356d1 00007f7b475349cd 0000000000388634 0000000000388479 00007f7b4751683a
0709 12:21:49.359939 (+ 64904us) raft_consensus.cc:1685] finished
0709 12:21:49.363659 (+  3720us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:49.363797 (+   138us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":6925,"prepare.queue_time_us":44764,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":43,"raft.queue_time_us":97814,"raft.run_cpu_time_us":301,"raft.run_wall_time_us":29703,"spinlock_wait_cycles":27660160,"thread_start_us":158997,"threads_started":2,"wal-append.queue_time_us":248267}
I20260709 12:21:49.373477 20702 raft_consensus.cc:2804] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:21:49.419766 20368 raft_consensus.cc:3060] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:21:49.459913 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 213185us
I20260709 12:21:49.420042 20604 raft_consensus.cc:493] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:21:49.391536 20479 consensus_queue.cc:1059] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.063s
I20260709 12:21:49.393061 20679 consensus_queue.cc:1059] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.063s
I20260709 12:21:49.453125 20702 raft_consensus.cc:493] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:49.450886 20655 raft_consensus.cc:493] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:21:49.473469 20686 leader_election.cc:290] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:49.548707 20705 consensus_queue.cc:1048] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.006s
I20260709 12:21:49.547386 20571 raft_consensus.cc:515] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:21:49.555860 17804 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:49.538144 20702 raft_consensus.cc:3060] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:49.565578 17803 leader_election.cc:304] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:21:49.557288 20571 leader_election.cc:290] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 7 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:49.576359 17833 raft_consensus.cc:1275] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20260709 12:21:49.526525 20604 raft_consensus.cc:515] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:49.586906 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:49.549439 20480 raft_consensus.cc:493] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:21:49.557655 20479 consensus_queue.cc:1243] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer 334a4e1f59944544b3b3ff36616b57da log is divergent from this leader: its last log entry 7.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260709 12:21:49.559048 20679 consensus_queue.cc:1243] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer 334a4e1f59944544b3b3ff36616b57da log is divergent from this leader: its last log entry 5.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260709 12:21:49.603466 18333 leader_election.cc:304] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:21:49.590219 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 116392us
I20260709 12:21:49.558369 20418 raft_consensus.cc:493] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:21:49.593954 17835 raft_consensus.cc:2468] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 5.
I20260709 12:21:49.595038 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:21:49.609565 17804 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:49.598873 18365 raft_consensus.cc:1275] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260709 12:21:49.599413 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:49.612129 18368 raft_consensus.cc:2468] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 6.
I20260709 12:21:49.599545 18364 raft_consensus.cc:1240] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 6 LEADER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 5. Current term is 6. Ops: [5.6-5.6]
I20260709 12:21:49.583614 20398 consensus_queue.cc:1059] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.017s
I20260709 12:21:49.583835 20697 raft_consensus.cc:2804] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:21:49.575687 20629 raft_consensus.cc:493] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:49.600643 18361 raft_consensus.cc:1275] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260709 12:21:49.551962 20655 raft_consensus.cc:515] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:49.620046 18366 raft_consensus.cc:1275] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260709 12:21:49.621595 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:49.600641 20459 raft_consensus.cc:493] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:49.624306 20459 raft_consensus.cc:515] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:49.626547 20459 leader_election.cc:290] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:49.628170 20629 raft_consensus.cc:515] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:49.620935 20687 leader_election.cc:290] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:49.630790 20629 leader_election.cc:290] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:49.606762 20385 leader_election.cc:290] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 8 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:49.631983 20374 raft_consensus.cc:2749] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:21:49.639374 20655 leader_election.cc:290] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:49.618461 20466 raft_consensus.cc:493] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:21:49.618937 20418 raft_consensus.cc:515] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:49.642519 20702 raft_consensus.cc:515] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:49.616428 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:49.631990 20559 raft_consensus.cc:2749] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:21:49.606415 20480 raft_consensus.cc:515] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:21:49.676307 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 307080us
W20260709 12:21:49.684536 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 61289.5us
I20260709 12:21:49.681644 18334 leader_election.cc:304] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b
W20260709 12:21:49.696000 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 90195us
I20260709 12:21:49.696724 17652 leader_election.cc:304] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:21:49.699535 18333 leader_election.cc:304] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:21:49.704995 17652 leader_election.cc:304] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:49.662246 20466 raft_consensus.cc:515] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:49.682302 17832 raft_consensus.cc:2468] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 4.
I20260709 12:21:49.697640 20559 raft_consensus.cc:2804] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:21:49.716382 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:49.855233 17834 raft_consensus.cc:2468] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 7.
I20260709 12:21:49.722877 17833 raft_consensus.cc:1275] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 7 index: 9. Preceding OpId from leader: term: 9 index: 10. (index mismatch)
W20260709 12:21:49.880010 17729 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:49.880815 17729 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:49.706290 20574 raft_consensus.cc:2804] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:21:49.920380 17836 raft_consensus.cc:1275] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260709 12:21:49.918597 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:49.923218 17757 raft_consensus.cc:2468] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 4.
I20260709 12:21:49.706202 20346 consensus_queue.cc:1048] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.013s
I20260709 12:21:49.690097 19826 raft_consensus.cc:515] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:49.933576 18333 leader_election.cc:304] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:21:49.934851 17837 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37994 (request call id 692) took 1114 ms. Trace:
I20260709 12:21:49.937192 17764 raft_consensus.cc:1275] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260709 12:21:49.935531 17837 rpcz_store.cc:276] 0709 12:21:48.819990 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:49.035412 (+215422us) service_pool.cc:224] Handling call
0709 12:21:49.121216 (+ 85804us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:21:49.208536 (+ 87320us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:49.208580 (+    44us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:21:49.209327 (+   747us) log.cc:844] Serialized 52 byte log entry
0709 12:21:49.336272 (+126945us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:21:49.336328 (+    56us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:49.336411 (+    83us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:49.337890 (+  1479us) raft_consensus.cc:1685] finished
0709 12:21:49.337910 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:49.934479 (+596569us) spinlock_profiling.cc:243] Waited 53.1 ms on lock 0x7b540022b850. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a61f8ad 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:21:49.934751 (+   272us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":86337,"raft.queue_time_us":86520,"raft.run_cpu_time_us":127,"raft.run_wall_time_us":130,"spinlock_wait_cycles":122047232,"thread_start_us":126451,"threads_started":1,"wal-append.queue_time_us":126834}
I20260709 12:21:50.008271 17762 raft_consensus.cc:3055] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:21:50.008797 17762 raft_consensus.cc:740] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 6 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:21:49.712191 20629 consensus_queue.cc:1048] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.217s
W20260709 12:21:50.014153 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 309219us
I20260709 12:21:49.922468 20693 raft_consensus.cc:493] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:50.015270 17761 raft_consensus.cc:3055] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:21:50.015827 17761 raft_consensus.cc:740] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 4 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:21:49.931217 20346 consensus_queue.cc:1243] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer e2909cff1da44ad387f96a8ae3aa5afb log is divergent from this leader: its last log entry 5.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260709 12:21:50.016649 17652 leader_election.cc:304] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:21:50.005802 20604 leader_election.cc:290] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:21:50.065487 17804 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:50.066205 17802 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:50.059079 17762 consensus_queue.cc:260] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:49.995399 20697 raft_consensus.cc:697] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 7 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:21:50.068879 20157 raft_consensus.cc:493] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:50.078658 17762 raft_consensus.cc:3060] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:50.067608 18371 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59886 (request call id 873) took 2605 ms. Trace:
I20260709 12:21:50.016010 20299 raft_consensus.cc:2749] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:21:50.080340 18371 rpcz_store.cc:276] 0709 12:21:47.462054 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:47.631862 (+169808us) service_pool.cc:224] Handling call
0709 12:21:47.633643 (+  1781us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:48.560700 (+927057us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:50.067044 (+1506344us) spinlock_profiling.cc:243] Waited 924 ms on lock 0x7b54007602d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:21:50.067550 (+   506us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2125561728}
I20260709 12:21:50.074577 17761 consensus_queue.cc:260] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:50.082805 17761 raft_consensus.cc:3060] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:50.064791 20766 raft_consensus.cc:493] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:50.094074 17755 raft_consensus.cc:1275] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 7 index: 9. Preceding OpId from leader: term: 9 index: 10. (index mismatch)
I20260709 12:21:50.049726 20693 raft_consensus.cc:515] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:50.109277 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:21:50.114130 17727 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:50.020433 20764 consensus_queue.cc:1048] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.086s
I20260709 12:21:50.066560 20674 consensus_queue.cc:1048] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.035s
I20260709 12:21:50.118614 18371 raft_consensus.cc:2468] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 9.
W20260709 12:21:50.123379 17804 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:50.130414 17762 raft_consensus.cc:1275] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 7. (term mismatch)
I20260709 12:21:50.131074 17762 pending_rounds.cc:85] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb: Aborting all ops after (but not including) 6
I20260709 12:21:50.131453 17762 pending_rounds.cc:107] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb: Aborting uncommitted NO_OP operation due to leader change: 6.7
I20260709 12:21:50.131843 17762 raft_consensus.cc:2889] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:21:50.141019 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 207543us
I20260709 12:21:50.104727 20702 leader_election.cc:290] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 7 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:50.092007 20157 raft_consensus.cc:515] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:50.125129 20332 raft_consensus.cc:493] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:21:50.148797 20629 raft_consensus.cc:2804] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:21:50.125285 20418 leader_election.cc:290] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:50.168792 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:50.146826 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
W20260709 12:21:50.165045 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 148490us
I20260709 12:21:50.134004 20772 raft_consensus.cc:493] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:21:50.144436 20512 raft_consensus.cc:493] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:50.110530 20766 raft_consensus.cc:515] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:50.172858 20332 raft_consensus.cc:515] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:50.184105 18362 raft_consensus.cc:1240] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 5. Current term is 6. Ops: [5.8-5.8]
I20260709 12:21:50.159750 20349 raft_consensus.cc:493] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:21:50.185290 20771 raft_consensus.cc:2804] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:21:50.185755 20771 raft_consensus.cc:493] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:21:50.185669 20349 raft_consensus.cc:515] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:50.186049 18362 raft_consensus.cc:1240] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 6. Current term is 7. Ops: []
I20260709 12:21:50.186256 20771 raft_consensus.cc:3060] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:21:50.178134 20772 raft_consensus.cc:515] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:21:50.190132 20752 log.cc:927] Time spent T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.180s	user 0.001s	sys 0.000s
I20260709 12:21:50.150498 20559 raft_consensus.cc:697] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 9 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:21:50.191296 20559 consensus_queue.cc:237] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:50.192394 17755 pending_rounds.cc:85] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb: Aborting all ops after (but not including) 5
I20260709 12:21:50.192729 17755 pending_rounds.cc:107] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb: Aborting uncommitted NO_OP operation due to leader change: 5.6
I20260709 12:21:50.192999 17755 raft_consensus.cc:2889] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:21:50.162467 20773 raft_consensus.cc:493] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:50.158740 20759 log.cc:927] Time spent T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.452s	user 0.001s	sys 0.002s
I20260709 12:21:50.197201 20512 raft_consensus.cc:515] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:50.173506 20574 raft_consensus.cc:697] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 7 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:21:50.174521 20629 raft_consensus.cc:493] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:21:50.185340 20332 leader_election.cc:290] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:50.169845 17836 raft_consensus.cc:2468] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 5.
I20260709 12:21:50.245199 20385 raft_consensus.cc:493] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:50.259245 20385 raft_consensus.cc:515] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:21:50.261144 20757 log.cc:927] Time spent T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.331s	user 0.000s	sys 0.001s
I20260709 12:21:50.262542 17830 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35040 (request call id 2028) took 1876 ms. Trace:
I20260709 12:21:50.205250 20418 consensus_queue.cc:1048] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.058s
I20260709 12:21:50.262878 17830 rpcz_store.cc:276] 0709 12:21:48.386631 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:48.387974 (+  1343us) service_pool.cc:224] Handling call
0709 12:21:48.388400 (+   426us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:21:48.485486 (+ 97086us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:48.485522 (+    36us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:21:48.486153 (+   631us) log.cc:844] Serialized 52 byte log entry
0709 12:21:49.359659 (+873506us) spinlock_profiling.cc:243] Waited 4.86 ms on lock 0x7b1800134020. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b47515e07 00007f7b503709a9 00007f7b4753269b 00007f7b4753093f 00007f7b47532c02 00007f7b4a1ba20b 00007f7b4a1bfd23 00007f7b4a1c00d1 00007f7b4a5dba0f
0709 12:21:49.558232 (+198573us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:21:49.558302 (+    70us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:49.558397 (+    95us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:50.262221 (+703824us) raft_consensus.cc:1685] finished
0709 12:21:50.262247 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:50.262477 (+   230us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":20103,"raft.queue_time_us":28798,"raft.run_cpu_time_us":179,"raft.run_wall_time_us":14510,"spinlock_wait_cycles":11182848,"thread_start_us":187073,"threads_started":1,"wal-append.queue_time_us":1192379}
I20260709 12:21:50.201825 20349 leader_election.cc:290] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:50.245645 20776 consensus_queue.cc:1048] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.018s
I20260709 12:21:50.246639 19826 leader_election.cc:290] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 7 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:50.240319 20374 raft_consensus.cc:493] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:21:50.309067 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:50.309894 17686 raft_consensus.cc:3060] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:21:50.309569 20374 raft_consensus.cc:515] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:50.177309 20702 consensus_queue.cc:1059] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 5, Time since last communication: 0.089s
I20260709 12:21:50.248306 20629 raft_consensus.cc:3060] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:50.316084 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:50.322021 17684 raft_consensus.cc:2468] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 5.
I20260709 12:21:50.162364 20697 consensus_queue.cc:237] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:50.345479 17686 raft_consensus.cc:2468] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 8.
I20260709 12:21:50.345927 20771 raft_consensus.cc:515] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:21:50.347096 20368 consensus_meta.cc:287] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.816s	user 0.007s	sys 0.004s
I20260709 12:21:50.241714 20773 raft_consensus.cc:515] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:50.393735 18371 raft_consensus.cc:3060] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:50.426550 17680 raft_consensus.cc:1275] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260709 12:21:50.427569 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:21:50.428211 17730 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:50.429776 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:50.431396 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:50.433881 17833 raft_consensus.cc:1240] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 6. Current term is 7. Ops: [6.7-6.7]
W20260709 12:21:50.435559 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 225909us
I20260709 12:21:50.437201 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:50.437994 17834 raft_consensus.cc:3060] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:50.436156 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:50.445013 17830 raft_consensus.cc:1240] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 7. Current term is 8. Ops: []
I20260709 12:21:50.445335 17833 raft_consensus.cc:3060] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:21:50.448766 17728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:50.449592 17728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:50.449996 17728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:50.451421 17688 raft_consensus.cc:1275] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260709 12:21:50.318842 20574 consensus_queue.cc:237] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:50.453367 17758 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36572 (request call id 600) took 1745 ms. Trace:
I20260709 12:21:50.456969 18371 raft_consensus.cc:2468] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 7.
I20260709 12:21:50.459582 20787 raft_consensus.cc:493] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:21:50.472025 17761 raft_consensus.cc:1275] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260709 12:21:50.473065 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:50.476676 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:50.477519 17833 raft_consensus.cc:2468] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 7.
I20260709 12:21:50.478933 17757 raft_consensus.cc:1275] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260709 12:21:50.484110 17802 leader_election.cc:304] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:21:50.486980 20157 leader_election.cc:290] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:50.487919 17832 raft_consensus.cc:1240] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 7 LEADER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 6. Current term is 7. Ops: []
I20260709 12:21:50.435269 20374 leader_election.cc:290] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:50.467069 17758 rpcz_store.cc:276] 0709 12:21:48.707866 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:48.708066 (+   200us) service_pool.cc:224] Handling call
0709 12:21:48.725543 (+ 17477us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:48.736316 (+ 10773us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 12:21:48.764095 (+ 27779us) log.cc:844] Serialized 14 byte log entry
0709 12:21:49.244501 (+480406us) spinlock_profiling.cc:243] Waited 9.22 ms on lock 0x7b18003371e0. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b47515e07 00007f7b503709a9 00007f7b4753269b 00007f7b4753093f 00007f7b47532c02 00007f7b4a1ba20b 00007f7b4a1bfd23 00007f7b4a1c023a 00007f7b4a62aa2a
0709 12:21:49.358393 (+113892us) spinlock_profiling.cc:243] Waited 9.43 ms on lock 0x7b18003371e0. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b475333f3 00007f7b475357c1 00007f7b47535779 00007f7b47535709 00007f7b475356d1 00007f7b475349cd 0000000000388634 0000000000388479 00007f7b4751683a
0709 12:21:49.444992 (+ 86599us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:50.380187 (+935195us) spinlock_profiling.cc:243] Waited 7.67 ms on lock 0x7b2c0000e5c8. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b48f19a11 00007f7b48f29372 00007f7b4a65ba01 00007f7b4a62043a 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2
0709 12:21:50.392823 (+ 12636us) log.cc:844] Serialized 26 byte log entry
0709 12:21:50.393027 (+   204us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:21:50.393092 (+    65us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:50.393183 (+    91us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:50.453112 (+ 59929us) raft_consensus.cc:1685] finished
0709 12:21:50.453139 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:50.453293 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":27649,"prepare.run_cpu_time_us":31,"prepare.run_wall_time_us":31,"raft.queue_time_us":8189,"raft.run_cpu_time_us":249,"raft.run_wall_time_us":24159,"spinlock_wait_cycles":60521856,"thread_start_us":200194,"threads_started":1,"wal-append.queue_time_us":854483}
I20260709 12:21:50.496606 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:50.489863 17802 leader_election.cc:304] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:21:50.453861 20679 raft_consensus.cc:493] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:50.488598 20771 leader_election.cc:290] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 8 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:50.509018 20771 raft_consensus.cc:493] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:50.509486 20771 raft_consensus.cc:515] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:50.510075 20157 raft_consensus.cc:2804] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:21:50.503760 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:50.507259 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:50.508008 20679 raft_consensus.cc:515] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:21:50.517495 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 345020us
I20260709 12:21:50.483548 20332 raft_consensus.cc:493] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:50.523818 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:50.524037 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:50.524781 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:50.527444 17760 raft_consensus.cc:2468] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 8.
I20260709 12:21:50.492357 20479 raft_consensus.cc:493] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:50.533531 17834 raft_consensus.cc:2468] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 7.
I20260709 12:21:50.533797 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:21:50.522682 20683 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 7 current term: 7 is higher.
I20260709 12:21:50.535820 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:50.535924 20683 raft_consensus.cc:493] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:50.464959 20706 consensus_queue.cc:1048] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.052s
I20260709 12:21:50.502473 20768 raft_consensus.cc:2804] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:21:50.537248 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:50.548933 17757 raft_consensus.cc:1275] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260709 12:21:50.543116 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
W20260709 12:21:50.544055 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 386818us
W20260709 12:21:50.541079 18335 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.523s	user 0.035s	sys 0.023s
I20260709 12:21:50.540709 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:50.542765 17801 leader_election.cc:304] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:21:50.551674 18335 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.534s	user 0.035s	sys 0.023s
I20260709 12:21:50.552644 19855 raft_consensus.cc:493] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:50.553112 19855 raft_consensus.cc:515] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:50.553498 17729 leader_election.cc:304] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:21:50.550360 20683 raft_consensus.cc:515] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:50.557436 20773 leader_election.cc:290] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:50.557981 20683 leader_election.cc:290] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:50.556239 19855 leader_election.cc:290] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:50.517254 20157 raft_consensus.cc:697] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 8 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:21:50.533382 20466 leader_election.cc:290] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:50.542122 20368 raft_consensus.cc:493] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:21:50.491859 20787 raft_consensus.cc:515] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:50.517424 20655 consensus_queue.cc:1048] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.046s
I20260709 12:21:50.525203 20398 consensus_queue.cc:1048] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 3, Time since last communication: 0.045s
I20260709 12:21:50.536767 20332 raft_consensus.cc:515] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:50.540879 20479 raft_consensus.cc:515] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:21:50.901422 18362 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.589s	user 0.011s	sys 0.010s
W20260709 12:21:51.052815 18362 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.740s	user 0.017s	sys 0.015s
W20260709 12:21:51.049604 17729 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:50.963446 17653 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.503s	user 0.002s	sys 0.006s
W20260709 12:21:51.054448 17653 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.594s	user 0.004s	sys 0.006s
W20260709 12:21:51.053586 17727 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.558s	user 0.000s	sys 0.009s
W20260709 12:21:51.054893 17727 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.560s	user 0.001s	sys 0.009s
I20260709 12:21:51.057329 18365 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59886 (request call id 900) took 1354 ms. Trace:
I20260709 12:21:51.057858 18365 rpcz_store.cc:276] 0709 12:21:49.702468 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:50.566852 (+864384us) service_pool.cc:224] Handling call
0709 12:21:51.056663 (+489811us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:51.056889 (+   226us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:51.056928 (+    39us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:51.056986 (+    58us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:21:51.057034 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:51.057118 (+    84us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:51.057259 (+   141us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:21:51.039170 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 74654.3us
W20260709 12:21:51.062880 18363 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.506s	user 0.005s	sys 0.029s
W20260709 12:21:51.063444 18363 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.507s	user 0.005s	sys 0.029s
I20260709 12:21:51.064499 17727 leader_election.cc:304] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
W20260709 12:21:51.053579 17652 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.520s	user 0.001s	sys 0.022s
W20260709 12:21:51.066071 17652 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.533s	user 0.001s	sys 0.023s
W20260709 12:21:51.066483 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 547856us
I20260709 12:21:51.072093 18362 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59886 (request call id 896) took 1370 ms. Trace:
I20260709 12:21:51.072750 18362 rpcz_store.cc:276] 0709 12:21:49.701193 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:50.187161 (+485968us) service_pool.cc:224] Handling call
0709 12:21:50.187488 (+   327us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:50.279685 (+ 92197us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 12:21:50.312033 (+ 32348us) log.cc:844] Serialized 14 byte log entry
0709 12:21:51.071541 (+759508us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:51.071634 (+    93us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:21:51.071710 (+    76us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:51.071810 (+   100us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:51.072005 (+   195us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":18084,"prepare.queue_time_us":28244,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":38,"thread_start_us":740908,"threads_started":1}
I20260709 12:21:51.075851 18361 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59886 (request call id 897) took 1374 ms. Trace:
I20260709 12:21:51.076877 18361 rpcz_store.cc:276] 0709 12:21:49.701503 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:50.346204 (+644701us) service_pool.cc:224] Handling call
0709 12:21:50.346517 (+   313us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:50.346735 (+   218us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 12:21:50.347051 (+   316us) log.cc:844] Serialized 14 byte log entry
0709 12:21:51.049117 (+702066us) log.cc:844] Serialized 14 byte log entry
0709 12:21:51.075365 (+ 26248us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:51.075442 (+    77us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:21:51.075518 (+    76us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:51.075613 (+    95us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:51.075773 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":88261,"prepare.queue_time_us":2960,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":38,"raft.queue_time_us":212377,"spinlock_wait_cycles":25802368,"thread_start_us":491861,"threads_started":1}
I20260709 12:21:51.079635 18363 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59886 (request call id 898) took 1377 ms. Trace:
I20260709 12:21:51.085808 18363 rpcz_store.cc:276] 0709 12:21:49.701798 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:50.543058 (+841260us) service_pool.cc:224] Handling call
0709 12:21:50.543449 (+   391us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:50.543742 (+   293us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 12:21:50.555797 (+ 12055us) log.cc:844] Serialized 14 byte log entry
0709 12:21:51.079147 (+523350us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:51.079226 (+    79us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:21:51.079300 (+    74us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:51.079396 (+    96us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:51.079559 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":11120,"prepare.queue_time_us":14665,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":34,"raft.queue_time_us":12214,"thread_start_us":507629,"threads_started":1}
I20260709 12:21:50.548884 20768 raft_consensus.cc:493] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
W20260709 12:21:51.049494 20791 log.cc:927] Time spent T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.245s	user 0.001s	sys 0.000s
I20260709 12:21:50.562310 20762 raft_consensus.cc:493] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:50.562458 20349 raft_consensus.cc:493] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:21:50.565246 20683 raft_consensus.cc:2804] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:21:51.049654 20771 leader_election.cc:290] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:21:51.109834 17755 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.683s	user 0.006s	sys 0.047s
W20260709 12:21:51.121865 17755 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.695s	user 0.006s	sys 0.048s
I20260709 12:21:51.126096 18365 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59902 (request call id 861) took 1201 ms. Trace:
I20260709 12:21:51.126710 18365 rpcz_store.cc:276] 0709 12:21:49.924730 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:51.059087 (+1134357us) service_pool.cc:224] Handling call
0709 12:21:51.059482 (+   395us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:51.059689 (+   207us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:51.059726 (+    37us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:51.059779 (+    53us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:21:51.059822 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:51.125809 (+ 65987us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:51.126005 (+   196us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:51.084497 20689 raft_consensus.cc:493] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
W20260709 12:21:51.036183 20574 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.547s	user 0.002s	sys 0.007s
I20260709 12:21:50.641597 20374 raft_consensus.cc:2749] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 12:21:51.103991 20768 raft_consensus.cc:3060] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:50.568893 20385 leader_election.cc:290] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:51.135504 20783 raft_consensus.cc:493] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:51.147959 17801 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.586s	user 0.015s	sys 0.037s
W20260709 12:21:51.136055 20789 log.cc:927] Time spent T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.208s	user 0.000s	sys 0.001s
I20260709 12:21:51.138247 17829 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:59820 (request call id 26) took 1833 ms. Trace:
I20260709 12:21:51.142907 18371 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59886 (request call id 901) took 1440 ms. Trace:
I20260709 12:21:51.151369 18371 rpcz_store.cc:276] 0709 12:21:49.702759 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:50.145529 (+442770us) service_pool.cc:224] Handling call
0709 12:21:51.142680 (+997151us) spinlock_profiling.cc:243] Waited 222 ms on lock 0x7b5400031bd4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:21:51.142829 (+   149us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":510802816}
I20260709 12:21:51.080797 20368 raft_consensus.cc:515] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:51.135349 20771 consensus_queue.cc:1059] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.002s
W20260709 12:21:51.148650 17801 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.587s	user 0.015s	sys 0.037s
W20260709 12:21:51.156129 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 613413us
I20260709 12:21:50.616829 20782 consensus_queue.cc:1048] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.516s
W20260709 12:21:51.116734 17802 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.592s	user 0.001s	sys 0.046s
W20260709 12:21:51.157416 17802 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.633s	user 0.001s	sys 0.046s
W20260709 12:21:51.158754 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 652127us
W20260709 12:21:51.160041 17802 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:51.121186 20683 raft_consensus.cc:493] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
W20260709 12:21:51.160558 17831 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.662s	user 0.003s	sys 0.017s
W20260709 12:21:51.160317 17685 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.695s	user 0.011s	sys 0.010s
W20260709 12:21:51.164023 17831 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.665s	user 0.003s	sys 0.018s
W20260709 12:21:51.164044 17685 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.698s	user 0.011s	sys 0.010s
I20260709 12:21:51.153026 17829 rpcz_store.cc:276] 0709 12:21:49.304333 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:49.306952 (+  2619us) service_pool.cc:224] Handling call
0709 12:21:51.137363 (+1830411us) spinlock_profiling.cc:243] Waited 2.39 ms on lock 0x7b54003affa0. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a59d520 00007f7b4a61a652 00007f7b4f5e860a 00007f7b48b6e7d4 00007f7b48b6e762 00007f7b48b6e691 00007f7b48b6e60c 00007f7b48b6d8c2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:21:51.138109 (+   746us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5490944}
I20260709 12:21:51.153409 20783 raft_consensus.cc:515] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:51.170902 20798 raft_consensus.cc:2749] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:21:51.172804 17682 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37402 (request call id 2318) took 1839 ms. Trace:
I20260709 12:21:51.173321 17682 rpcz_store.cc:276] 0709 12:21:49.333088 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:49.333217 (+   129us) service_pool.cc:224] Handling call
0709 12:21:49.339415 (+  6198us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:21:49.347335 (+  7920us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:49.347375 (+    40us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:21:49.348004 (+   629us) log.cc:844] Serialized 52 byte log entry
0709 12:21:50.261066 (+913062us) spinlock_profiling.cc:243] Waited 8.41 ms on lock 0x7b1800166ba0. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b47515e07 00007f7b503709a9 00007f7b4753269b 00007f7b4753093f 00007f7b47532c02 00007f7b4a1ba20b 00007f7b4a1bfd23 00007f7b4a1c00d1 00007f7b4a5dba0f
0709 12:21:50.458508 (+197442us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:21:50.458548 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:50.458624 (+    76us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:51.172491 (+713867us) raft_consensus.cc:1685] finished
0709 12:21:51.172512 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:51.172744 (+   232us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1437,"raft.queue_time_us":2462,"raft.run_cpu_time_us":58,"raft.run_wall_time_us":57,"spinlock_wait_cycles":19339520,"thread_start_us":197251,"threads_started":1,"wal-append.queue_time_us":1210641}
W20260709 12:21:51.182420 20755 log.cc:927] Time spent T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 1.514s	user 0.002s	sys 0.018s
I20260709 12:21:51.194375 20766 leader_election.cc:290] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:21:51.144707 20574 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.656s	user 0.003s	sys 0.008s
W20260709 12:21:51.206979 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 151010us
I20260709 12:21:51.207912 17727 leader_election.cc:304] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:21:51.149073 20799 raft_consensus.cc:493] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:21:51.128517 20349 raft_consensus.cc:515] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:51.161468 20683 raft_consensus.cc:3060] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:21:51.224646 20788 raft_consensus.cc:3055] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:21:51.225219 20788 raft_consensus.cc:740] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:21:51.229396 17755 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36572 (request call id 609) took 1072 ms. Trace:
I20260709 12:21:51.229971 20787 leader_election.cc:290] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:51.144608 20689 raft_consensus.cc:515] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:51.229813 17755 rpcz_store.cc:276] 0709 12:21:50.156955 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:50.160054 (+  3099us) service_pool.cc:224] Handling call
0709 12:21:50.192192 (+ 32138us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:21:50.194138 (+  1946us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:50.194220 (+    82us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:21:50.426359 (+232139us) log.cc:844] Serialized 52 byte log entry
0709 12:21:51.122362 (+696003us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:21:51.122424 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:51.122515 (+    91us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:51.229187 (+106672us) raft_consensus.cc:1685] finished
0709 12:21:51.229207 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:51.229338 (+   131us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":37509,"raft.run_cpu_time_us":63,"raft.run_wall_time_us":64,"thread_start_us":695592,"threads_started":1,"wal-append.queue_time_us":801152}
I20260709 12:21:51.232412 20689 leader_election.cc:290] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:51.130021 20762 raft_consensus.cc:515] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:51.211210 20798 raft_consensus.cc:2749] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260709 12:21:51.238234 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 170489us
W20260709 12:21:51.241025 17804 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.678s	user 0.028s	sys 0.022s
W20260709 12:21:51.241904 17804 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.679s	user 0.029s	sys 0.022s
I20260709 12:21:51.217671 20679 leader_election.cc:290] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:51.247116 17757 raft_consensus.cc:1275] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 6. (term mismatch)
I20260709 12:21:51.247534 17757 pending_rounds.cc:85] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb: Aborting all ops after (but not including) 5
I20260709 12:21:51.247767 17757 pending_rounds.cc:107] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb: Aborting uncommitted NO_OP operation due to leader change: 4.6
I20260709 12:21:51.248040 17757 raft_consensus.cc:2889] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:21:51.216164 20157 consensus_queue.cc:237] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:51.250543 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:51.278612 20368 leader_election.cc:290] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:51.279922 17832 raft_consensus.cc:1275] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
W20260709 12:21:51.281963 20802 log.cc:927] Time spent T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.066s	user 0.000s	sys 0.000s
I20260709 12:21:51.284422 20762 leader_election.cc:290] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:21:51.284991 20629 consensus_meta.cc:287] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.938s	user 0.008s	sys 0.002s
I20260709 12:21:51.285444 20629 raft_consensus.cc:515] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:51.288122 18368 raft_consensus.cc:1240] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 4. Current term is 5. Ops: []
I20260709 12:21:51.233433 20799 raft_consensus.cc:515] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:51.291790 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:51.292637 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:51.292915 20349 leader_election.cc:290] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:51.292994 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 8 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:51.252435 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:51.261686 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:51.263069 17759 pending_rounds.cc:85] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb: Aborting all ops after (but not including) 4
I20260709 12:21:51.318310 17759 pending_rounds.cc:107] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb: Aborting uncommitted NO_OP operation due to leader change: 5.5
I20260709 12:21:51.318385 17833 raft_consensus.cc:1275] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260709 12:21:51.318593 17759 raft_consensus.cc:2889] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:21:51.257439 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 195777us
I20260709 12:21:51.323617 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:51.209007 17727 leader_election.cc:304] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
W20260709 12:21:51.330017 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 121079us
I20260709 12:21:51.332261 17727 leader_election.cc:304] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
W20260709 12:21:51.332783 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 132299us
I20260709 12:21:51.332639 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 8 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:51.316869 20629 leader_election.cc:290] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 5 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:51.333806 17763 raft_consensus.cc:1275] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:21:51.337522 20810 raft_consensus.cc:2749] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:21:51.338349 17757 raft_consensus.cc:1240] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 5. Current term is 6. Ops: []
W20260709 12:21:51.358410 17714 heartbeater.cc:646] Failed to heartbeat to 127.16.29.126:39149 (0 consecutive failures): Timed out: Failed to send heartbeat to master: TSHeartbeat RPC to 127.16.29.126:39149 timed out after 15.000s (SENT)
I20260709 12:21:51.360169 20349 raft_consensus.cc:493] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:51.360257 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:51.360845 20349 raft_consensus.cc:515] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:51.361608 20799 leader_election.cc:290] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:51.333494 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:51.333408 17829 raft_consensus.cc:2368] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for earlier term 6. Current term is 7.
I20260709 12:21:51.363392 20349 leader_election.cc:290] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:21:51.365350 20711 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 5 current term: 5 is higher.
I20260709 12:21:51.369824 20768 raft_consensus.cc:515] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:51.371869 17727 leader_election.cc:304] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:21:51.365015 20349 consensus_queue.cc:1059] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 4.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:21:51.345162 20808 raft_consensus.cc:2764] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Leader pre-election decision vote started in defunct term 7: won
I20260709 12:21:51.377935 17761 raft_consensus.cc:1275] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260709 12:21:51.375774 20811 raft_consensus.cc:493] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:21:51.379136 20811 raft_consensus.cc:515] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:51.378348 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
W20260709 12:21:51.345844 17652 connection.cc:256] RPC call timeout handler was delayed by -1.5295581539999148s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1526s for a response
W20260709 12:21:51.387944 17652 connection.cc:256] RPC call timeout handler was delayed by -1.229858153999885s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1859s for a response
I20260709 12:21:51.391455 17832 raft_consensus.cc:2468] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 4.
I20260709 12:21:51.393790 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:51.389804 17652 leader_election.cc:304] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:51.365082 20635 raft_consensus.cc:3055] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:21:51.399281 20635 raft_consensus.cc:740] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 3 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:21:51.376400 20799 raft_consensus.cc:2749] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:21:51.395172 17829 raft_consensus.cc:1275] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 5 index: 8. Preceding OpId from leader: term: 6 index: 8. (term mismatch)
I20260709 12:21:51.350958 20787 raft_consensus.cc:493] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:21:51.350368 20785 raft_consensus.cc:493] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
W20260709 12:21:51.395964 20803 log.cc:927] Time spent T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.092s	user 0.000s	sys 0.000s
I20260709 12:21:51.396147 20683 raft_consensus.cc:515] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:51.400496 20635 consensus_queue.cc:260] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:51.404455 17727 leader_election.cc:304] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:21:51.470208 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 65864.2us
I20260709 12:21:51.430318 17832 raft_consensus.cc:2468] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 7.
I20260709 12:21:51.471256 17727 leader_election.cc:304] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:21:51.468053 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:51.473892 17833 raft_consensus.cc:2410] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
W20260709 12:21:51.430194 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 174442us
I20260709 12:21:51.475998 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:51.469929 17829 pending_rounds.cc:85] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b: Aborting all ops after (but not including) 7
I20260709 12:21:51.477759 17833 raft_consensus.cc:2393] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 6: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:21:51.476619 20711 consensus_queue.cc:1059] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.071s
I20260709 12:21:51.546805 20711 consensus_queue.cc:1243] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer e063541542a94969898868bf6eb5ce1b log is divergent from this leader: its last log entry 7.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260709 12:21:51.547346 20711 raft_consensus.cc:3055] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:21:51.547667 20711 raft_consensus.cc:740] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 6 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:21:51.547827 20683 leader_election.cc:290] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 10 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:51.482790 17829 pending_rounds.cc:107] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b: Aborting uncommitted NO_OP operation due to leader change: 5.8
I20260709 12:21:51.549119 17829 raft_consensus.cc:2889] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:21:51.548642 20711 consensus_queue.cc:260] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:51.549968 20711 raft_consensus.cc:3060] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:51.550094 20811 leader_election.cc:290] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:51.551118 20811 consensus_queue.cc:1048] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:21:51.476687 20810 raft_consensus.cc:2804] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:21:51.552173 20810 raft_consensus.cc:697] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 7 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:21:51.552995 20768 leader_election.cc:290] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 6 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:51.553089 20810 consensus_queue.cc:237] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:51.476250 20785 raft_consensus.cc:515] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:51.431725 20799 consensus_queue.cc:1059] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.124s
I20260709 12:21:51.475775 20349 raft_consensus.cc:2804] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:21:51.555799 20785 leader_election.cc:290] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:51.464478 20781 raft_consensus.cc:493] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:21:51.561861 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:51.556165 20799 raft_consensus.cc:3055] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:21:51.562543 17834 raft_consensus.cc:3060] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:21:51.559085 17832 raft_consensus.cc:1275] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260709 12:21:51.612953 20762 raft_consensus.cc:493] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:51.613418 17764 raft_consensus.cc:1275] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260709 12:21:51.613597 20762 raft_consensus.cc:515] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:51.476123 20787 raft_consensus.cc:515] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:51.556254 20349 raft_consensus.cc:697] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:21:51.548558 20635 raft_consensus.cc:3060] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:21:51.731503 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 256382us
I20260709 12:21:51.573110 20799 raft_consensus.cc:740] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:21:51.791290 20349 consensus_queue.cc:237] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:21:51.796402 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 53139.1us
I20260709 12:21:51.616403 20762 leader_election.cc:290] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:51.620337 20781 raft_consensus.cc:515] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:51.710750 20766 raft_consensus.cc:493] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:51.807150 17804 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:51.808344 20683 raft_consensus.cc:493] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:51.810811 20683 raft_consensus.cc:515] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:51.811159 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:51.808305 20768 raft_consensus.cc:493] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:51.812143 20787 leader_election.cc:290] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:51.812000 20768 raft_consensus.cc:515] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:51.813066 20683 leader_election.cc:290] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:51.813294 20799 consensus_queue.cc:260] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:51.814042 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:51.814510 20799 raft_consensus.cc:3060] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:21:51.814872 17764 raft_consensus.cc:2468] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 4.
I20260709 12:21:51.813992 20768 leader_election.cc:290] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:51.796854 20809 consensus_queue.cc:1048] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.050s
I20260709 12:21:51.824733 17834 raft_consensus.cc:2468] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 8.
I20260709 12:21:51.826172 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:51.820415 20766 raft_consensus.cc:515] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:51.829690 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:51.933523 20766 leader_election.cc:290] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:51.831732 20781 leader_election.cc:290] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:51.847961 20418 raft_consensus.cc:493] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:21:51.935285 20418 raft_consensus.cc:515] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:51.935858 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:51.937358 17755 raft_consensus.cc:2393] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 6: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:21:51.937842 20418 leader_election.cc:290] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:21:51.880935 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 63066.9us
I20260709 12:21:51.938880 17652 leader_election.cc:400] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 6 pre-election: Vote denied by peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837) with higher term. Message: Invalid argument: T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for earlier term 6. Current term is 7.
I20260709 12:21:51.939188 17652 leader_election.cc:403] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 6 pre-election: Cancelling election due to peer responding with higher term
W20260709 12:21:51.918056 20525 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.829s	user 0.013s	sys 0.029s
W20260709 12:21:51.940116 20525 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.856s	user 0.013s	sys 0.030s
I20260709 12:21:51.940778 20811 raft_consensus.cc:3060] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Advancing to term 7
I20260709 12:21:51.944031 20768 raft_consensus.cc:493] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:51.944563 20768 raft_consensus.cc:515] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:51.944792 17652 leader_election.cc:304] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:21:51.945947 17652 leader_election.cc:304] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:21:51.946758 20157 raft_consensus.cc:493] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:51.947053 20768 leader_election.cc:290] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:51.947254 20459 raft_consensus.cc:2804] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:21:51.947103 17652 leader_election.cc:304] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:51.947292 20157 raft_consensus.cc:515] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:51.948952 20825 raft_consensus.cc:2804] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:21:51.947646 20768 raft_consensus.cc:493] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:21:51.949352 20825 raft_consensus.cc:493] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:51.949832 20825 raft_consensus.cc:3060] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:21:51.950132 20818 consensus_queue.cc:1048] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:21:51.949839 20768 raft_consensus.cc:515] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:51.947829 20459 raft_consensus.cc:697] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 4 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:21:51.949749 20760 raft_consensus.cc:3060] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:51.952260 20768 leader_election.cc:290] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:51.952072 20459 consensus_queue.cc:237] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:51.952780 20635 raft_consensus.cc:493] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:51.953266 20635 raft_consensus.cc:515] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:51.924242 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:51.926388 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:51.956272 17652 leader_election.cc:304] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
W20260709 12:21:51.948458 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 2.25007e+06us
I20260709 12:21:51.957324 20763 consensus_queue.cc:1048] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 12:21:51.958089 20818 raft_consensus.cc:2804] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:21:51.958747 20818 raft_consensus.cc:697] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 8 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:21:51.950444 20157 leader_election.cc:290] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:51.959676 20144 consensus_queue.cc:1048] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:21:51.959755 20818 consensus_queue.cc:237] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:51.959707 20525 consensus_queue.cc:1048] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.003s
I20260709 12:21:51.964666 17764 raft_consensus.cc:1275] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 5. (term mismatch)
I20260709 12:21:51.965040 17764 pending_rounds.cc:85] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb: Aborting all ops after (but not including) 4
I20260709 12:21:51.965286 17764 pending_rounds.cc:107] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb: Aborting uncommitted NO_OP operation due to leader change: 6.5
I20260709 12:21:51.965566 17764 raft_consensus.cc:2889] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:21:51.966029 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:51.966816 17762 raft_consensus.cc:2468] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 5.
I20260709 12:21:51.959795 17654 leader_election.cc:304] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:21:51.970182 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:51.970500 17654 leader_election.cc:304] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:21:51.970990 17761 raft_consensus.cc:2393] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 7: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:21:51.977761 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:51.978617 20525 consensus_queue.cc:1048] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 12:21:51.955694 20635 leader_election.cc:290] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:51.971426 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:51.971771 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:51.972337 17689 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37402 (request call id 2317) took 2640 ms. Trace:
I20260709 12:21:51.972780 20157 raft_consensus.cc:493] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:21:51.965586 20825 raft_consensus.cc:515] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:51.974737 20771 raft_consensus.cc:2749] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:21:51.975762 17757 raft_consensus.cc:3055] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:21:51.976572 20781 raft_consensus.cc:2749] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:21:51.984226 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:51.985452 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:51.981992 17757 raft_consensus.cc:740] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 6 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:21:51.985508 20825 leader_election.cc:290] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 8 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:51.981381 20157 raft_consensus.cc:515] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:51.986496 20480 leader_election.cc:290] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:51.986974 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:51.982995 17689 rpcz_store.cc:276] 0709 12:21:49.331840 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:49.335108 (+  3268us) service_pool.cc:224] Handling call
0709 12:21:49.335477 (+   369us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 12:21:49.339992 (+  4515us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:49.340046 (+    54us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 12:21:51.914068 (+2574022us) spinlock_profiling.cc:243] Waited 917 us on lock 0x7b2c00000c88. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b48f19a11 00007f7b48f29372 00007f7b4a65ba01 00007f7b4a62043a 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2
0709 12:21:51.914774 (+   706us) log.cc:844] Serialized 78 byte log entry
0709 12:21:51.971852 (+ 57078us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:21:51.971921 (+    69us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:51.971993 (+    72us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:51.972031 (+    38us) raft_consensus.cc:1685] finished
0709 12:21:51.972040 (+     9us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:51.972269 (+   229us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":9050,"raft.queue_time_us":94,"raft.run_cpu_time_us":89,"raft.run_wall_time_us":1657,"spinlock_wait_cycles":2108800,"thread_start_us":56687,"threads_started":1,"wal-append.queue_time_us":51002}
I20260709 12:21:51.976661 20811 raft_consensus.cc:2749] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: Vote denied by peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837) with higher term. Message: Invalid argument: T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for earlier term 6. Current term is 7.
I20260709 12:21:52.035645 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:51.994836 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:51.999182 17834 raft_consensus.cc:2468] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 4.
I20260709 12:21:51.923979 20808 raft_consensus.cc:3055] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:21:51.918496 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:51.981472 20763 raft_consensus.cc:2804] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:21:51.992378 17757 consensus_queue.cc:260] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:52.048151 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:52.051061 17689 raft_consensus.cc:1240] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 7. Current term is 8. Ops: []
I20260709 12:21:52.052127 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:52.000507 20766 raft_consensus.cc:493] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:52.076040 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:52.076691 17759 raft_consensus.cc:2468] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 5.
I20260709 12:21:52.076869 17686 raft_consensus.cc:2393] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 4: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:21:52.052865 17761 raft_consensus.cc:2468] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 7.
I20260709 12:21:52.054947 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:52.055507 17652 leader_election.cc:304] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:21:52.000475 20768 raft_consensus.cc:493] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:52.000957 20629 raft_consensus.cc:493] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:52.089421 17689 raft_consensus.cc:1240] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 6. Current term is 7. Ops: [5.4-6.5]
I20260709 12:21:51.994999 20418 raft_consensus.cc:493] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:52.081287 17757 raft_consensus.cc:3060] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:52.059152 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:52.036224 20674 raft_consensus.cc:493] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:21:52.051607 20763 raft_consensus.cc:493] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:21:52.095677 17762 raft_consensus.cc:2468] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 7.
I20260709 12:21:52.045028 20480 raft_consensus.cc:493] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:52.100723 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:52.093622 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:51.991815 20525 consensus_queue.cc:1048] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.085s
I20260709 12:21:52.045462 20808 raft_consensus.cc:740] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 7 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:21:52.090937 17681 raft_consensus.cc:1240] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 4 LEADER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 3. Current term is 4. Ops: []
I20260709 12:21:52.036062 20762 consensus_queue.cc:1059] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.087s
I20260709 12:21:52.101962 17838 raft_consensus.cc:3060] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:21:52.098889 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:52.106586 17680 raft_consensus.cc:3060] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:52.104158 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:52.108031 18364 raft_consensus.cc:3060] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:52.104437 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:52.036080 20377 consensus_queue.cc:1048] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.088s
I20260709 12:21:52.106088 18371 raft_consensus.cc:1275] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 7 index: 9. Preceding OpId from leader: term: 10 index: 11. (index mismatch)
I20260709 12:21:52.099598 20418 raft_consensus.cc:515] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:52.089872 20635 raft_consensus.cc:493] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:52.120934 17680 raft_consensus.cc:2410] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for term 7 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 4.
I20260709 12:21:52.120968 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:52.121743 18367 raft_consensus.cc:3060] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:21:52.115809 20796 log.cc:927] Time spent T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.818s	user 0.008s	sys 0.010s
I20260709 12:21:52.112598 20479 leader_election.cc:290] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:52.095081 20629 raft_consensus.cc:515] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:52.103189 20584 raft_consensus.cc:493] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:52.158424 20584 raft_consensus.cc:515] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:52.160782 20584 leader_election.cc:290] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:52.100931 20674 raft_consensus.cc:515] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:52.161831 17838 raft_consensus.cc:2468] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 8.
I20260709 12:21:52.145061 18367 raft_consensus.cc:2468] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 7.
I20260709 12:21:52.090432 20768 raft_consensus.cc:515] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:52.123420 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:52.123962 20635 raft_consensus.cc:515] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:52.164155 17729 leader_election.cc:304] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:21:52.167198 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:52.167817 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:52.168286 20635 leader_election.cc:290] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:52.168467 20479 raft_consensus.cc:2804] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:21:52.124887 20349 raft_consensus.cc:493] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:21:52.051680 20459 consensus_queue.cc:1059] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.050s
I20260709 12:21:52.105818 20480 raft_consensus.cc:515] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:52.171962 20459 consensus_queue.cc:1243] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer 334a4e1f59944544b3b3ff36616b57da log is divergent from this leader: its last log entry 6.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260709 12:21:52.129761 17689 raft_consensus.cc:1240] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 6. Current term is 7. Ops: []
I20260709 12:21:52.099510 20763 raft_consensus.cc:3060] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:52.136462 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:52.172767 18371 raft_consensus.cc:2468] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 6.
I20260709 12:21:52.173020 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:52.096598 20829 raft_consensus.cc:2804] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260709 12:21:52.173959 17728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:52.174074 17729 leader_election.cc:304] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:21:52.147578 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:52.150280 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:52.151158 17687 raft_consensus.cc:1240] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 8 LEADER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 7. Current term is 8. Ops: []
I20260709 12:21:52.152153 17757 raft_consensus.cc:1275] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 7 index: 6. (term mismatch)
I20260709 12:21:52.177691 18364 raft_consensus.cc:2468] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 7.
I20260709 12:21:52.157770 20629 leader_election.cc:290] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:52.109714 20828 raft_consensus.cc:493] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:21:52.163136 17652 leader_election.cc:304] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:21:52.124147 20798 consensus_queue.cc:1048] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 12:21:52.168716 17683 raft_consensus.cc:2410] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for term 9 because replica has last-logged OpId of term: 8 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 5.
I20260709 12:21:52.168289 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:52.169382 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:52.090142 20766 raft_consensus.cc:515] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:52.188580 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:52.200405 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:52.189587 17757 pending_rounds.cc:85] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb: Aborting all ops after (but not including) 5
I20260709 12:21:52.190766 17755 raft_consensus.cc:1240] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 6. Current term is 7. Ops: [6.5-6.5]
I20260709 12:21:52.202823 18366 raft_consensus.cc:2393] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 4: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:21:52.193162 17835 raft_consensus.cc:2468] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 4.
I20260709 12:21:52.207803 17757 pending_rounds.cc:107] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb: Aborting uncommitted NO_OP operation due to leader change: 6.6
I20260709 12:21:52.215960 17757 raft_consensus.cc:2889] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:21:52.209638 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 9 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:52.210382 20768 leader_election.cc:290] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:52.166343 20760 raft_consensus.cc:2749] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:21:52.165664 20776 raft_consensus.cc:493] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:52.228963 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:52.214403 17681 raft_consensus.cc:1275] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 7 index: 9. Preceding OpId from leader: term: 10 index: 11. (index mismatch)
I20260709 12:21:52.188724 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:52.208868 20418 leader_election.cc:290] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:52.200459 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:52.230923 17680 raft_consensus.cc:2468] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 7.
I20260709 12:21:52.222635 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:52.173987 20479 raft_consensus.cc:697] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:21:52.255838 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:52.189399 20584 raft_consensus.cc:493] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:21:52.257136 17832 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 7.
I20260709 12:21:52.184346 20829 raft_consensus.cc:493] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:21:52.163172 20674 leader_election.cc:290] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:52.191339 20349 raft_consensus.cc:515] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:52.211944 20377 raft_consensus.cc:2804] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:21:52.200305 20799 raft_consensus.cc:2804] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260709 12:21:52.460170 17804 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:52.339373 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:52.409418 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:52.494668 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:52.341005 17682 raft_consensus.cc:1240] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 5. Current term is 6. Ops: []
I20260709 12:21:52.282416 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:52.201972 20828 raft_consensus.cc:515] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:52.960975 20799 raft_consensus.cc:493] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:52.615619 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 11 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:52.341337 20795 raft_consensus.cc:493] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:52.420792 20480 leader_election.cc:290] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:52.787137 18368 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 1044) took 3188 ms. Trace:
I20260709 12:21:52.962149 20795 raft_consensus.cc:515] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:52.962555 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:52.418171 20479 consensus_queue.cc:237] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:52.962937 18368 rpcz_store.cc:276] 0709 12:21:49.598843 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:49.636304 (+ 37461us) service_pool.cc:224] Handling call
0709 12:21:50.157126 (+520822us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:51.288633 (+1131507us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:52.785638 (+1497005us) spinlock_profiling.cc:243] Waited 1.13 s on lock 0x7b5400499654. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:21:52.786860 (+  1222us) spinlock_profiling.cc:243] Waited 497 ms on lock 0x7b5400499650. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a61f8ad 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:21:52.787061 (+   201us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3743992448}
I20260709 12:21:52.621554 20772 leader_election.cc:290] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:21:53.001837 17755 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.522s	user 0.012s	sys 0.000s
W20260709 12:21:53.002291 17755 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.522s	user 0.012s	sys 0.000s
I20260709 12:21:53.002486 17759 raft_consensus.cc:2468] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 3.
I20260709 12:21:52.964756 17764 raft_consensus.cc:2468] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 3.
I20260709 12:21:52.612929 17763 raft_consensus.cc:3055] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:21:52.960073 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:53.004401 18364 raft_consensus.cc:2468] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 7.
I20260709 12:21:52.630158 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:52.620292 20711 raft_consensus.cc:3055] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:21:52.283023 20776 raft_consensus.cc:515] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:52.960412 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:21:53.000977 17804 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:52.570367 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 61816.5us
W20260709 12:21:52.690686 17728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:52.961706 20799 raft_consensus.cc:3060] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:21:52.973213 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 465132us
I20260709 12:21:53.040361 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:21:52.733613 20763 consensus_meta.cc:287] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.526s	user 0.007s	sys 0.002s
I20260709 12:21:53.006927 17756 raft_consensus.cc:3060] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:52.964109 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:53.005474 17763 raft_consensus.cc:740] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 3 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:21:53.051975 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:52.622181 17837 raft_consensus.cc:1240] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 5. Current term is 6. Ops: []
W20260709 12:21:52.776413 20767 log.cc:927] Time spent T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 2.761s	user 0.006s	sys 0.031s
I20260709 12:21:52.411123 20766 leader_election.cc:290] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:52.667560 20349 leader_election.cc:290] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:53.063738 18364 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 7.
W20260709 12:21:52.962646 20811 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.500s	user 0.015s	sys 0.009s
I20260709 12:21:53.064730 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:53.067054 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 8 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:53.069228 17837 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35042 (request call id 1802) took 2633 ms. Trace:
I20260709 12:21:52.624238 20674 raft_consensus.cc:493] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:21:52.733275 20512 leader_election.cc:290] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:52.627568 20693 leader_election.cc:290] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:52.511270 20829 raft_consensus.cc:3060] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:52.508869 20418 raft_consensus.cc:493] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:52.648816 20604 raft_consensus.cc:493] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:52.912958 20377 raft_consensus.cc:697] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 8 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:21:53.069199 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:52.767686 20559 raft_consensus.cc:493] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:21:53.069199 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:21:53.069808 17729 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:53.071264 17837 rpcz_store.cc:276] 0709 12:21:50.432369 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:50.432528 (+   159us) service_pool.cc:224] Handling call
0709 12:21:52.621797 (+2189269us) spinlock_profiling.cc:243] Waited 20.3 ms on lock 0x7b580013fe98. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b474fbb71 00007f7b474fba22 00007f7b504148d9 00007f7b4f56eee0 00007f7b4f6380ab 00007f7b4f638131 00007f7b4f5f9fc0 00007f7b4f5e4929 00007f7b4f5e43d2 00007f7b48b5bbd4
0709 12:21:52.622006 (+   209us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:53.065497 (+443491us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:53.065692 (+   195us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":46650240}
I20260709 12:21:53.072427 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:52.962599 20480 raft_consensus.cc:493] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:21:53.072790 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:52.719987 20635 raft_consensus.cc:493] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:52.626386 20157 leader_election.cc:290] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:53.021872 20711 raft_consensus.cc:740] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:21:52.303146 20584 raft_consensus.cc:515] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:52.732355 20783 leader_election.cc:290] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:52.304495 20808 consensus_queue.cc:260] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:21:53.073388 20811 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.611s	user 0.016s	sys 0.011s
I20260709 12:21:53.073319 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:53.070693 18374 raft_consensus.cc:2468] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 7.
I20260709 12:21:53.091997 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:53.257937 17837 raft_consensus.cc:1275] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260709 12:21:53.258201 18365 raft_consensus.cc:2468] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 4.
I20260709 12:21:53.258586 17837 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37994 (request call id 709) took 1046 ms. Trace:
W20260709 12:21:53.245795 18333 connection.cc:256] RPC call timeout handler was delayed by -1.1143813939997926s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1703s for a response
W20260709 12:21:53.395288 17729 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:53.398105 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:21:53.128072 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 49224.8us
I20260709 12:21:53.259163 17837 rpcz_store.cc:276] 0709 12:21:52.212238 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:53.257332 (+1045094us) service_pool.cc:224] Handling call
0709 12:21:53.257657 (+   325us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:53.258348 (+   691us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:53.258530 (+   182us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:53.073635 17764 raft_consensus.cc:2410] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for term 8 because replica has last-logged OpId of term: 6 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20260709 12:21:53.401906 17757 raft_consensus.cc:2468] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 4.
W20260709 12:21:53.221143 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 70435.6us
I20260709 12:21:53.074082 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:53.403226 18368 raft_consensus.cc:2468] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 3.
I20260709 12:21:53.151240 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:53.256851 20418 raft_consensus.cc:515] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:53.404820 18367 raft_consensus.cc:2393] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 5: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:21:53.152768 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:53.262734 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:53.179072 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:53.450831 18361 raft_consensus.cc:2376] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 7. Re-sending same reply.
I20260709 12:21:53.451815 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:53.452499 18361 raft_consensus.cc:2468] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 6.
I20260709 12:21:53.453526 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:53.454905 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:53.408221 17834 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37994 (request call id 711) took 1195 ms. Trace:
I20260709 12:21:53.455376 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:21:53.431178 17728 leader_election.cc:336] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Timed out: RequestConsensusVote RPC to 127.16.29.65:33071 timed out after 1.854s (SENT)
I20260709 12:21:53.169656 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:53.062552 20768 raft_consensus.cc:493] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:21:53.039737 20702 raft_consensus.cc:493] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:21:53.455579 17834 rpcz_store.cc:276] 0709 12:21:52.212937 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:53.051185 (+838248us) service_pool.cc:224] Handling call
0709 12:21:53.408136 (+356951us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:53.392184 20697 raft_consensus.cc:493] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:53.468329 18333 leader_election.cc:304] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:21:53.474898 18333 leader_election.cc:304] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:21:53.455411 17764 raft_consensus.cc:1240] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 9. Current term is 10. Ops: []
I20260709 12:21:53.246939 20766 raft_consensus.cc:493] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:53.105003 17763 consensus_queue.cc:260] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:53.247141 20349 raft_consensus.cc:493] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:53.449939 17759 raft_consensus.cc:2468] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 7.
I20260709 12:21:53.479774 17763 raft_consensus.cc:3060] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:53.482726 17762 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36572 (request call id 612) took 1259 ms. Trace:
I20260709 12:21:53.483722 17762 rpcz_store.cc:276] 0709 12:21:52.223816 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:52.224715 (+   899us) service_pool.cc:224] Handling call
0709 12:21:52.263950 (+ 39235us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:52.264296 (+   346us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:52.264322 (+    26us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:52.568849 (+304527us) log.cc:844] Serialized 26 byte log entry
0709 12:21:53.061064 (+492215us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:21:53.061114 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:53.061399 (+   285us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:53.482495 (+421096us) raft_consensus.cc:1685] finished
0709 12:21:53.482514 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:53.482664 (+   150us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":167,"raft.run_cpu_time_us":163,"raft.run_wall_time_us":66674,"thread_start_us":447260,"threads_started":1,"wal-append.queue_time_us":802111}
I20260709 12:21:53.484654 20349 raft_consensus.cc:515] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:53.485693 20418 leader_election.cc:290] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:53.486465 20418 raft_consensus.cc:2804] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:21:53.486883 20418 raft_consensus.cc:493] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
W20260709 12:21:53.487502 17728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:53.487468 20349 leader_election.cc:290] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:53.488969 20828 leader_election.cc:290] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:53.489583 20828 raft_consensus.cc:493] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:53.490101 20828 raft_consensus.cc:515] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:53.462635 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 11 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:53.466810 17758 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38776 (request call id 1858) took 2217 ms. Trace:
I20260709 12:21:53.469553 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:53.512297 18362 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 7.
I20260709 12:21:53.510766 17758 rpcz_store.cc:276] 0709 12:21:51.249292 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:51.249472 (+   180us) service_pool.cc:224] Handling call
0709 12:21:51.269316 (+ 19844us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:51.298852 (+ 29536us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:51.298896 (+    44us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:51.298953 (+    57us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:21:51.298997 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:53.466503 (+2167506us) spinlock_profiling.cc:243] Waited 29.3 ms on lock 0x7b54000fc8d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:21:53.466563 (+    60us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:53.466738 (+   175us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":67344256}
I20260709 12:21:53.150111 20227 raft_consensus.cc:493] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:21:53.160653 20795 leader_election.cc:290] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:53.477715 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:53.449333 20635 raft_consensus.cc:515] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:53.461776 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:53.495313 17763 raft_consensus.cc:1275] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 12:21:53.556232 17763 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38776 (request call id 1877) took 1094 ms. Trace:
I20260709 12:21:53.496652 20772 raft_consensus.cc:2804] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:21:53.444299 20604 raft_consensus.cc:515] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:53.498198 20377 consensus_queue.cc:237] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:53.500361 20584 leader_election.cc:290] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:53.563987 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:53.500780 17830 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35040 (request call id 2061) took 1502 ms. Trace:
I20260709 12:21:53.501689 20766 raft_consensus.cc:515] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:53.452989 20559 raft_consensus.cc:515] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:53.571149 20559 leader_election.cc:290] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:53.487386 20418 raft_consensus.cc:3060] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:53.572566 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:53.573366 17686 raft_consensus.cc:2393] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 4: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:21:53.573784 17686 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37396 (request call id 1950) took 1343 ms. Trace:
I20260709 12:21:53.574111 17686 rpcz_store.cc:276] 0709 12:21:52.229935 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:53.572288 (+1342353us) service_pool.cc:224] Handling call
0709 12:21:53.573723 (+  1435us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:53.576807 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:53.577478 17836 raft_consensus.cc:2468] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 9.
I20260709 12:21:53.577685 20604 leader_election.cc:290] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:21:53.578016 17836 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35040 (request call id 2050) took 2016 ms (client timeout 1683 ms). Trace:
W20260709 12:21:53.578378 17836 rpcz_store.cc:269] 0709 12:21:51.561295 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:51.616128 (+ 54833us) service_pool.cc:224] Handling call
0709 12:21:53.576652 (+1960524us) spinlock_profiling.cc:243] Waited 2.32 ms on lock 0x7b580013fe98. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b474fbb71 00007f7b474fba22 00007f7b504148d9 00007f7b4f56eee0 00007f7b4f6380ab 00007f7b4f638131 00007f7b4f5e60a0 00007f7b4f5e5984 00007f7b48b606d4 00007f7b48b60662
0709 12:21:53.577946 (+  1294us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5332224}
I20260709 12:21:53.581914 18371 raft_consensus.cc:2468] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 5.
I20260709 12:21:53.582242 20766 leader_election.cc:290] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:53.583112 20635 leader_election.cc:290] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:53.557682 20772 raft_consensus.cc:493] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:53.583786 17836 raft_consensus.cc:1275] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260709 12:21:53.584095 20772 raft_consensus.cc:3060] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:21:53.584384 17836 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37994 (request call id 713) took 1082 ms. Trace:
I20260709 12:21:53.584725 17836 rpcz_store.cc:276] 0709 12:21:52.501771 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:53.583147 (+1081376us) service_pool.cc:224] Handling call
0709 12:21:53.583473 (+   326us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:53.584153 (+   680us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:53.584333 (+   180us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:53.587232 18371 raft_consensus.cc:1240] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 6. Current term is 7. Ops: [4.6-6.7]
I20260709 12:21:53.578408 17832 raft_consensus.cc:2393] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 7: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:21:53.566169 17830 rpcz_store.cc:276] 0709 12:21:51.998726 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:52.211073 (+212347us) service_pool.cc:224] Handling call
0709 12:21:52.211366 (+   293us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:52.211715 (+   349us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:52.211743 (+    28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:52.461040 (+249297us) log.cc:844] Serialized 26 byte log entry
0709 12:21:52.724283 (+263243us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:21:52.724349 (+    66us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:52.724460 (+   111us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:53.500560 (+776100us) raft_consensus.cc:1685] finished
0709 12:21:53.500583 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:53.500721 (+   138us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":18418,"raft.run_cpu_time_us":117,"raft.run_wall_time_us":9975,"thread_start_us":215560,"threads_started":1,"wal-append.queue_time_us":787689}
W20260709 12:21:53.604127 17804 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:53.611114 20776 leader_election.cc:290] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:53.613878 20847 raft_consensus.cc:493] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:21:53.614537 20847 raft_consensus.cc:515] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:53.616478 20851 raft_consensus.cc:493] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:21:53.617074 20851 raft_consensus.cc:515] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:53.619360 20851 leader_election.cc:290] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:53.509087 17834 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35040 (request call id 2063) took 1049 ms. Trace:
W20260709 12:21:53.621773 20800 log.cc:927] Time spent T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 2.328s	user 0.013s	sys 0.028s
I20260709 12:21:53.623579 17831 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37994 (request call id 706) took 3463 ms. Trace:
I20260709 12:21:53.624044 17831 rpcz_store.cc:276] 0709 12:21:50.160086 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:50.160408 (+   322us) service_pool.cc:224] Handling call
0709 12:21:50.179178 (+ 18770us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:50.207277 (+ 28099us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:21:50.448140 (+240863us) log.cc:844] Serialized 14 byte log entry
0709 12:21:51.164993 (+716853us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:51.171666 (+  6673us) log.cc:844] Serialized 26 byte log entry
0709 12:21:51.171848 (+   182us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:21:51.171922 (+    74us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:51.172012 (+    90us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:53.607634 (+2435622us) spinlock_profiling.cc:243] Waited 4.97 ms on lock 0x7b4400022b4c. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b47459c85 00007f7b4745a6a2 00007f7b4a1cc1db 00007f7b4a1c7aaf 00007f7b4a1bb86e 00007f7b4a1babb6 00007f7b4a1ba7d8 00007f7b4a1c4221 00007f7b4a1c41d9 00007f7b4a1c4169
0709 12:21:53.623217 (+ 15583us) raft_consensus.cc:1685] finished
0709 12:21:53.623350 (+   133us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:53.623507 (+   157us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":41091,"prepare.queue_time_us":863354,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":42,"raft.queue_time_us":27838,"raft.run_cpu_time_us":304,"raft.run_wall_time_us":35957,"spinlock_wait_cycles":11433856,"thread_start_us":876873,"threads_started":2,"wal-append.queue_time_us":789700}
I20260709 12:21:53.626945 17831 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35040 (request call id 2064) took 1057 ms. Trace:
I20260709 12:21:53.630095 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:53.633292 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:53.633812 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:53.635959 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:53.636441 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:53.636631 17684 raft_consensus.cc:2468] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 7.
I20260709 12:21:53.637094 17683 raft_consensus.cc:2468] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 8.
I20260709 12:21:53.637566 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:53.637995 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:53.639415 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:53.640249 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:53.640931 19826 raft_consensus.cc:3055] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:21:53.641291 19826 raft_consensus.cc:740] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 6 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:21:53.641320 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:53.642419 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:53.642170 19826 consensus_queue.cc:260] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:53.644352 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:53.645434 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:53.646337 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:53.066888 20763 raft_consensus.cc:515] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:53.472503 20808 raft_consensus.cc:3060] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:21:53.107343 20674 raft_consensus.cc:515] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:53.458395 18363 raft_consensus.cc:2376] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 7. Re-sending same reply.
I20260709 12:21:53.447801 20480 raft_consensus.cc:515] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:53.536448 18374 raft_consensus.cc:1275] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
W20260709 12:21:53.539244 17728 leader_election.cc:336] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Timed out: RequestConsensusVote RPC to 127.16.29.65:33071 timed out after 1.981s (SENT)
W20260709 12:21:53.663915 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 124698us
W20260709 12:21:53.665413 17728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:53.665712 17838 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35040 (request call id 2060) took 1667 ms. Trace:
W20260709 12:21:53.666107 17728 leader_election.cc:336] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Timed out: RequestConsensusVote RPC to 127.16.29.65:33071 timed out after 1.984s (SENT)
W20260709 12:21:53.666805 17728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:53.666064 17838 rpcz_store.cc:276] 0709 12:21:51.998465 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:52.162530 (+164065us) service_pool.cc:224] Handling call
0709 12:21:52.162866 (+   336us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:52.169756 (+  6890us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:52.169775 (+    19us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:52.183339 (+ 13564us) log.cc:844] Serialized 26 byte log entry
0709 12:21:52.564807 (+381468us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:21:52.564864 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:53.627649 (+1062785us) spinlock_profiling.cc:243] Waited 6.55 ms on lock 0x7b54003cc3d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:21:53.627698 (+    49us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:53.627749 (+    51us) raft_consensus.cc:1685] finished
0709 12:21:53.627764 (+    15us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:53.665643 (+ 37879us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":99,"raft.run_cpu_time_us":221,"raft.run_wall_time_us":16664,"spinlock_wait_cycles":15069440,"thread_start_us":334366,"threads_started":1,"wal-append.queue_time_us":1197964}
I20260709 12:21:53.514128 20157 raft_consensus.cc:493] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:21:53.668287 20157 raft_consensus.cc:515] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:53.671418 20157 leader_election.cc:290] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:53.673671 20480 leader_election.cc:290] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:21:53.674183 17728 leader_election.cc:336] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Timed out: RequestConsensusVote RPC to 127.16.29.65:33071 timed out after 2.041s (SENT)
I20260709 12:21:53.674815 17728 leader_election.cc:304] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
W20260709 12:21:53.675364 17835 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35042 (request call id 1799) took 3246 ms (client timeout 1981 ms). Trace:
I20260709 12:21:53.675464 20847 leader_election.cc:290] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:53.675765 20810 raft_consensus.cc:2749] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 12:21:53.676692 18364 raft_consensus.cc:3060] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:21:53.678050 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:21:53.664885 17801 leader_election.cc:336] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Timed out: RequestConsensusVote RPC to 127.16.29.66:39651 timed out after 1.573s (SENT)
I20260709 12:21:53.472977 20768 raft_consensus.cc:515] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:53.546291 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:53.548373 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:53.698258 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:53.698992 17758 raft_consensus.cc:2410] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 9 because replica has last-logged OpId of term: 8 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 5.
I20260709 12:21:53.476893 20702 raft_consensus.cc:515] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:21:53.554968 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 152777us
I20260709 12:21:53.560324 17729 leader_election.cc:304] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:21:53.699584 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:21:53.700198 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 139956us
I20260709 12:21:53.701268 17729 leader_election.cc:304] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:21:53.702029 17729 leader_election.cc:304] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:21:53.603601 20799 consensus_meta.cc:287] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.536s	user 0.005s	sys 0.001s
I20260709 12:21:53.702968 17729 leader_election.cc:304] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:21:53.556484 17763 rpcz_store.cc:276] 0709 12:21:52.461452 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:52.461657 (+   205us) service_pool.cc:224] Handling call
0709 12:21:52.462183 (+   526us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:53.555974 (+1093791us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:53.556186 (+   212us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:53.504932 20697 raft_consensus.cc:515] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:53.567416 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:53.638164 17684 raft_consensus.cc:3060] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:53.647308 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:53.735639 18367 raft_consensus.cc:2468] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 4.
I20260709 12:21:53.736405 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:53.620071 17834 rpcz_store.cc:276] 0709 12:21:52.459845 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:53.464602 (+1004757us) service_pool.cc:224] Handling call
0709 12:21:53.508438 (+ 43836us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:53.508689 (+   251us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:53.508731 (+    42us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:53.508789 (+    58us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:21:53.508827 (+    38us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:53.508883 (+    56us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:53.509016 (+   133us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:53.627297 17831 rpcz_store.cc:276] 0709 12:21:52.569647 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:53.626077 (+1056430us) service_pool.cc:224] Handling call
0709 12:21:53.626428 (+   351us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:53.626621 (+   193us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:53.626650 (+    29us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:53.626692 (+    42us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:21:53.626725 (+    33us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:53.626780 (+    55us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:53.626894 (+   114us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:53.516208 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:53.737574 18361 raft_consensus.cc:2468] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 4.
I20260709 12:21:53.649341 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:53.651247 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:53.643647 19826 raft_consensus.cc:3060] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:53.521963 20711 consensus_queue.cc:260] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:53.543766 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:21:53.675793 17835 rpcz_store.cc:269] 0709 12:21:50.428996 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:50.429117 (+   121us) service_pool.cc:224] Handling call
0709 12:21:51.918254 (+1489137us) spinlock_profiling.cc:243] Waited 17.8 ms on lock 0x7b580013fe98. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b474fbb71 00007f7b474fba22 00007f7b504148d9 00007f7b4f56eee0 00007f7b4f6380ab 00007f7b4f638131 00007f7b4f5e60a0 00007f7b4f5e5984 00007f7b48b606d4 00007f7b48b60662
0709 12:21:53.675158 (+1756904us) spinlock_profiling.cc:243] Waited 147 ms on lock 0x7b540008a7d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:21:53.675268 (+   110us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":379705856}
W20260709 12:21:53.683892 17801 leader_election.cc:336] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Timed out: RequestConsensusVote RPC to 127.16.29.66:39651 timed out after 1.638s (SENT)
I20260709 12:21:53.771771 17681 raft_consensus.cc:2410] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for term 7 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 4.
W20260709 12:21:53.772491 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 88662.2us
I20260709 12:21:53.687104 18364 raft_consensus.cc:2468] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 10.
I20260709 12:21:53.693030 17838 raft_consensus.cc:1275] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 12:21:53.678941 17689 raft_consensus.cc:2468] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 3.
I20260709 12:21:53.699215 17757 raft_consensus.cc:2410] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 8 because replica has last-logged OpId of term: 7 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20260709 12:21:53.704370 20683 raft_consensus.cc:2804] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:21:53.704782 20810 raft_consensus.cc:2804] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:21:53.780974 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:53.781570 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:53.704882 20674 leader_election.cc:290] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:53.705255 20762 raft_consensus.cc:2804] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:21:53.705672 20784 raft_consensus.cc:2804] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:21:53.706326 17729 leader_election.cc:304] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
W20260709 12:21:53.793452 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 89232.3us
I20260709 12:21:53.713791 20763 leader_election.cc:290] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 7 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:53.724732 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:53.725538 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:53.726168 17688 raft_consensus.cc:1240] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 6. Current term is 7. Ops: []
I20260709 12:21:53.796985 17763 raft_consensus.cc:2468] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 6.
W20260709 12:21:53.759826 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 60098.5us
I20260709 12:21:53.562101 20227 raft_consensus.cc:515] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:21:53.766209 17756 consensus_meta.cc:287] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.703s	user 0.010s	sys 0.000s
I20260709 12:21:53.784900 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:21:53.785179 17801 leader_election.cc:336] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Timed out: RequestConsensusVote RPC to 127.16.29.66:39651 timed out after 1.759s (SENT)
I20260709 12:21:53.796366 17686 raft_consensus.cc:2468] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 4.
I20260709 12:21:53.758338 20635 raft_consensus.cc:493] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:53.800814 17729 leader_election.cc:304] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:21:53.806460 17801 leader_election.cc:336] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Timed out: RequestConsensusVote RPC to 127.16.29.66:39651 timed out after 1.791s (SENT)
I20260709 12:21:53.807438 17756 raft_consensus.cc:2468] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 5.
I20260709 12:21:53.808149 17756 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36572 (request call id 616) took 1299 ms. Trace:
I20260709 12:21:53.808514 17756 rpcz_store.cc:276] 0709 12:21:52.508620 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:52.629819 (+121199us) service_pool.cc:224] Handling call
0709 12:21:53.808095 (+1178276us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:53.808715 17801 leader_election.cc:304] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:53.782457 20683 raft_consensus.cc:493] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:21:53.811456 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 336643us
I20260709 12:21:53.785926 20810 raft_consensus.cc:493] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:53.785612 20847 raft_consensus.cc:493] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:53.816159 18333 leader_election.cc:304] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:21:53.818619 17685 raft_consensus.cc:2468] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 7.
W20260709 12:21:53.817771 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 228976us
W20260709 12:21:53.820647 18334 connection.cc:256] RPC call timeout handler was delayed by -1.4078367479996814s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2778s for a response
I20260709 12:21:53.823259 17755 raft_consensus.cc:2468] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 7.
I20260709 12:21:53.785849 20385 raft_consensus.cc:493] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:21:53.785351 20851 raft_consensus.cc:493] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
W20260709 12:21:53.821049 18334 connection.cc:256] RPC call timeout handler was delayed by -1.0040798959998938s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15480000000000002s for a response
I20260709 12:21:53.789933 20762 raft_consensus.cc:493] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:53.790174 20784 raft_consensus.cc:493] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:21:53.792608 20711 raft_consensus.cc:3060] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:53.832823 17652 leader_election.cc:304] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:53.837510 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:53.837556 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:53.837342 18333 leader_election.cc:304] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
W20260709 12:21:53.855146 17728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:53.811256 20783 raft_consensus.cc:2749] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:21:53.824707 20810 raft_consensus.cc:3060] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:21:53.858552 18334 leader_election.cc:336] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Timed out: RequestConsensusVote RPC to 127.16.29.65:33071 timed out after 1.643s (SENT)
I20260709 12:21:53.866791 18334 leader_election.cc:304] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:53.815383 20683 raft_consensus.cc:3060] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:21:53.860044 17652 leader_election.cc:304] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:53.769034 20799 raft_consensus.cc:515] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:53.830014 20539 raft_consensus.cc:493] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:53.868455 20144 raft_consensus.cc:2749] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:21:53.870486 20825 consensus_queue.cc:1048] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:21:53.870841 17652 leader_election.cc:304] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:53.872185 20394 consensus_queue.cc:1048] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:21:53.842146 20762 raft_consensus.cc:3060] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:53.904158 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:53.850426 20480 raft_consensus.cc:2804] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:21:53.886324 17688 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37396 (request call id 1925) took 1848 ms. Trace:
W20260709 12:21:53.891708 18366 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59886 (request call id 903) took 1787 ms (client timeout 1704 ms). Trace:
I20260709 12:21:53.887635 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:53.831544 20674 consensus_queue.cc:1048] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.063s
I20260709 12:21:53.909963 17688 rpcz_store.cc:276] 0709 12:21:52.038234 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:52.038387 (+   153us) service_pool.cc:224] Handling call
0709 12:21:52.038753 (+   366us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:53.797039 (+1758286us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:53.869298 (+ 72259us) spinlock_profiling.cc:243] Waited 1.69 s on lock 0x7b54003480d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:21:53.886253 (+ 16955us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3879974784}
I20260709 12:21:53.811131 20635 raft_consensus.cc:515] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:21:53.907668 17728 leader_election.cc:336] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Timed out: RequestConsensusVote RPC to 127.16.29.65:33071 timed out after 1.780s (SENT)
W20260709 12:21:53.906900 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 98226.6us
I20260709 12:21:53.917268 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:21:53.917639 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 70135.8us
I20260709 12:21:53.863583 20776 raft_consensus.cc:2804] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:21:53.850534 20784 raft_consensus.cc:3060] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:21:53.910231 18366 rpcz_store.cc:269] 0709 12:21:52.104095 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:52.104210 (+   115us) service_pool.cc:224] Handling call
0709 12:21:53.891446 (+1787236us) spinlock_profiling.cc:243] Waited 93.7 ms on lock 0x7b5400570154. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:21:53.891620 (+   174us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":215571712}
I20260709 12:21:53.837258 20847 raft_consensus.cc:515] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:53.815451 20349 consensus_queue.cc:1048] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.051s
I20260709 12:21:53.859937 20705 raft_consensus.cc:2749] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:21:53.864213 20157 raft_consensus.cc:2749] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:21:53.919708 20764 consensus_queue.cc:1059] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 9.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:21:53.921272 20144 consensus_queue.cc:1048] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:21:53.856515 20849 consensus_queue.cc:1048] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.088s
I20260709 12:21:53.910634 17685 raft_consensus.cc:2376] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 7. Re-sending same reply.
I20260709 12:21:53.897276 20837 raft_consensus.cc:2804] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260709 12:21:53.930768 17728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:53.930486 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:53.911507 20397 raft_consensus.cc:2749] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:21:53.861068 20346 consensus_queue.cc:1048] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.062s
I20260709 12:21:53.904043 20332 leader_election.cc:290] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:53.912840 20480 raft_consensus.cc:493] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:53.906629 20394 consensus_queue.cc:1243] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer e2909cff1da44ad387f96a8ae3aa5afb log is divergent from this leader: its last log entry 6.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260709 12:21:53.849872 20851 raft_consensus.cc:515] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:53.855778 20385 raft_consensus.cc:515] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:53.865801 20459 consensus_queue.cc:1048] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.063s
I20260709 12:21:53.905390 20828 leader_election.cc:290] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:53.866503 20584 consensus_queue.cc:1048] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.061s
I20260709 12:21:53.852170 20852 consensus_queue.cc:1048] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.075s
I20260709 12:21:53.913129 20772 raft_consensus.cc:515] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:53.829432 20829 raft_consensus.cc:515] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:53.843786 20773 consensus_queue.cc:1048] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.063s
I20260709 12:21:53.936972 17801 leader_election.cc:304] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:21:53.915309 20604 raft_consensus.cc:493] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:21:53.865919 20801 consensus_queue.cc:1059] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 5.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.067s
W20260709 12:21:53.974443 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 115909us
W20260709 12:21:54.041254 18334 leader_election.cc:336] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Timed out: RequestConsensusVote RPC to 127.16.29.65:33071 timed out after 1.703s (SENT)
W20260709 12:21:54.042037 18334 leader_election.cc:336] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Timed out: RequestConsensusVote RPC to 127.16.29.65:33071 timed out after 1.729s (SENT)
I20260709 12:21:53.965865 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:54.042495 20852 raft_consensus.cc:2804] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:21:54.043028 20852 raft_consensus.cc:493] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:54.043439 20852 raft_consensus.cc:3060] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:54.043612 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:54.045066 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:54.046363 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:53.900812 20539 raft_consensus.cc:515] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:53.983347 17687 raft_consensus.cc:1275] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
W20260709 12:21:53.982254 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 55021.1us
I20260709 12:21:53.922459 20776 raft_consensus.cc:493] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:21:54.005362 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 11 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:21:54.004748 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 203995us
I20260709 12:21:54.006577 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:53.973732 20837 raft_consensus.cc:493] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:53.996886 20480 raft_consensus.cc:3060] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:54.020375 20799 leader_election.cc:290] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 7 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:54.022905 20227 leader_election.cc:290] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:53.935643 20418 raft_consensus.cc:515] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:21:54.028921 20839 log.cc:927] Time spent T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.795s	user 0.004s	sys 0.001s
I20260709 12:21:54.023639 20697 leader_election.cc:290] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:54.033366 17802 leader_election.cc:304] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:21:54.035892 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 99043.1us
I20260709 12:21:54.053038 17688 raft_consensus.cc:2468] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 10.
W20260709 12:21:54.055707 17729 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:54.056313 17833 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35040 (request call id 2047) took 2496 ms. Trace:
W20260709 12:21:54.106515 17729 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:54.066987 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:54.067325 17685 raft_consensus.cc:2468] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 4.
I20260709 12:21:54.065552 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:54.113389 18368 raft_consensus.cc:3060] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:54.068871 18366 raft_consensus.cc:1275] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260709 12:21:54.042520 18334 leader_election.cc:304] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:21:54.047052 17683 raft_consensus.cc:2468] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 3.
I20260709 12:21:54.106451 17833 rpcz_store.cc:276] 0709 12:21:51.560305 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:51.615763 (+ 55458us) service_pool.cc:224] Handling call
0709 12:21:53.645867 (+2030104us) spinlock_profiling.cc:243] Waited 42.9 ms on lock 0x7b580013fe98. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b474fbb71 00007f7b474fba22 00007f7b504148d9 00007f7b4f56eee0 00007f7b4f6380ab 00007f7b4f638131 00007f7b4f5f9fc0 00007f7b4f5e4929 00007f7b4f5e43d2 00007f7b48b5bbd4
0709 12:21:53.646037 (+   170us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:21:53.646449 (+   412us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:53.646477 (+    28us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:21:53.647161 (+   684us) log.cc:844] Serialized 52 byte log entry
0709 12:21:53.868630 (+221469us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:21:54.052846 (+184216us) log.cc:844] Serialized 14 byte log entry
0709 12:21:54.054328 (+  1482us) log.cc:844] Serialized 14 byte log entry
0709 12:21:54.055544 (+  1216us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:54.055706 (+   162us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:54.055937 (+   231us) raft_consensus.cc:1685] finished
0709 12:21:54.055955 (+    18us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:54.056206 (+   251us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":101566,"raft.queue_time_us":11463,"raft.run_cpu_time_us":307,"raft.run_wall_time_us":7779,"spinlock_wait_cycles":98559232,"thread_start_us":395399,"threads_started":2,"wal-append.queue_time_us":284557}
I20260709 12:21:54.122606 17684 raft_consensus.cc:2468] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 7.
I20260709 12:21:54.008184 20862 raft_consensus.cc:493] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:54.137547 17689 raft_consensus.cc:1240] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 4. Current term is 5. Ops: []
W20260709 12:21:54.130329 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 57939.9us
W20260709 12:21:54.135286 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 93257.9us
W20260709 12:21:54.139076 18334 leader_election.cc:336] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Timed out: RequestConsensusVote RPC to 127.16.29.65:33071 timed out after 1.850s (SENT)
I20260709 12:21:54.139149 17801 leader_election.cc:304] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:21:54.140388 18334 leader_election.cc:336] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Timed out: RequestConsensusVote RPC to 127.16.29.65:33071 timed out after 1.886s (SENT)
I20260709 12:21:54.003046 20559 consensus_queue.cc:1048] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.044s
I20260709 12:21:54.037514 20702 leader_election.cc:290] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:54.038226 20525 raft_consensus.cc:3055] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:21:54.063819 20697 raft_consensus.cc:3060] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:21:54.144511 18334 leader_election.cc:336] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Timed out: RequestConsensusVote RPC to 127.16.29.65:33071 timed out after 2.172s (SENT)
I20260709 12:21:54.051623 20776 raft_consensus.cc:3060] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:54.147759 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 8 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:53.981788 20842 consensus_queue.cc:1048] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.144s
I20260709 12:21:54.062026 20418 leader_election.cc:290] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 6 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:53.982903 20863 consensus_queue.cc:1048] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.158s
I20260709 12:21:54.056124 20837 raft_consensus.cc:3060] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:54.122066 20799 raft_consensus.cc:493] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:21:54.171041 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:54.171859 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:54.128997 20866 raft_consensus.cc:2749] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:21:54.170337 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:54.121568 20604 raft_consensus.cc:515] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:54.152647 20525 raft_consensus.cc:740] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
W20260709 12:21:54.167508 20848 log.cc:927] Time spent T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.349s	user 0.001s	sys 0.000s
I20260709 12:21:54.181666 17831 raft_consensus.cc:2468] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 9.
I20260709 12:21:54.190625 17652 leader_election.cc:304] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:21:54.052134 20852 raft_consensus.cc:515] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:54.195618 20863 raft_consensus.cc:2804] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:21:54.230927 20863 raft_consensus.cc:493] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:54.231355 20863 raft_consensus.cc:3060] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:21:54.232372 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:54.233116 18367 raft_consensus.cc:2393] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 8: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
W20260709 12:21:54.200397 20860 log.cc:927] Time spent T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.191s	user 0.000s	sys 0.000s
W20260709 12:21:54.200675 20859 log.cc:927] Time spent T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.159s	user 0.000s	sys 0.000s
I20260709 12:21:54.236984 20157 raft_consensus.cc:493] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:54.201085 20857 log.cc:927] Time spent T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.170s	user 0.000s	sys 0.001s
I20260709 12:21:54.204630 20385 leader_election.cc:290] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:54.206650 20852 leader_election.cc:290] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 6 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:54.144845 20855 consensus_queue.cc:1048] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.141s
I20260709 12:21:54.241685 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:21:54.231496 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 571662us
I20260709 12:21:54.243191 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:54.244261 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:54.245023 18374 raft_consensus.cc:2393] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 4: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:21:54.246229 18374 raft_consensus.cc:1240] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 9. Current term is 10. Ops: []
I20260709 12:21:54.246522 17834 raft_consensus.cc:1275] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
W20260709 12:21:54.247601 17584 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35958 (request call id 410) took 18046 ms (client timeout 15000 ms). Trace:
I20260709 12:21:54.209874 20604 leader_election.cc:290] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:54.210551 20525 consensus_queue.cc:260] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:54.248368 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:54.248685 20525 raft_consensus.cc:3060] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:21:54.211473 20635 leader_election.cc:290] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:54.185381 20825 raft_consensus.cc:493] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:54.252481 20825 raft_consensus.cc:515] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:54.252945 17837 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37994 (request call id 710) took 2040 ms. Trace:
I20260709 12:21:54.253301 17837 rpcz_store.cc:276] 0709 12:21:52.212574 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:53.444379 (+1231805us) service_pool.cc:224] Handling call
0709 12:21:53.461964 (+ 17585us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:53.679346 (+217382us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:53.679376 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:53.727058 (+ 47682us) log.cc:844] Serialized 26 byte log entry
0709 12:21:53.928186 (+201128us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:21:53.928221 (+    35us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:54.252504 (+324283us) spinlock_profiling.cc:243] Waited 122 ms on lock 0x7b540008cfd4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:21:54.252554 (+    50us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:54.252608 (+    54us) raft_consensus.cc:1685] finished
0709 12:21:54.252626 (+    18us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:54.252880 (+   254us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":160866,"raft.queue_time_us":79863,"raft.run_cpu_time_us":105,"raft.run_wall_time_us":105,"spinlock_wait_cycles":279598976,"thread_start_us":116073,"threads_started":1,"wal-append.queue_time_us":232080}
I20260709 12:21:54.254637 20825 leader_election.cc:290] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:21:54.212077 20856 log.cc:927] Time spent T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.283s	user 0.001s	sys 0.000s
I20260709 12:21:54.212756 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:21:54.194772 20853 log.cc:927] Time spent T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.281s	user 0.000s	sys 0.001s
I20260709 12:21:54.255172 17583 catalog_manager.cc:5697] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b reported cstate change: term changed from 3 to 4, leader changed from e063541542a94969898868bf6eb5ce1b (127.16.29.67) to e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66). New cstate: current_term: 4 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:21:54.257050 18365 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59886 (request call id 920) took 1064 ms. Trace:
I20260709 12:21:54.256917 17583 catalog_manager.cc:5697] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: HEALTHY } } }
I20260709 12:21:54.257409 18365 rpcz_store.cc:276] 0709 12:21:53.192734 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:53.259149 (+ 66415us) service_pool.cc:224] Handling call
0709 12:21:53.259525 (+   376us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:21:53.260636 (+  1111us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:53.260684 (+    48us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:21:53.261684 (+  1000us) log.cc:844] Serialized 52 byte log entry
0709 12:21:53.555576 (+293892us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:21:53.555617 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:53.555682 (+    65us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:54.256835 (+701153us) raft_consensus.cc:1685] finished
0709 12:21:54.256855 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:54.256995 (+   140us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":129,"raft.run_cpu_time_us":268,"raft.run_wall_time_us":264,"thread_start_us":293290,"threads_started":1,"wal-append.queue_time_us":581261}
I20260709 12:21:54.258402 17583 catalog_manager.cc:5697] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b reported cstate change: term changed from 4 to 7, leader changed from e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66) to e063541542a94969898868bf6eb5ce1b (127.16.29.67). New cstate: current_term: 7 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:54.216698 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:54.260188 17683 raft_consensus.cc:2468] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 4.
I20260709 12:21:54.261090 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:54.217409 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:54.261682 20863 raft_consensus.cc:515] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:54.267408 20863 leader_election.cc:290] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 8 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:21:54.269537 17727 leader_election.cc:336] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837): Timed out: RequestConsensusVote RPC to 127.16.29.67:36837 timed out after 1.981s (SENT)
I20260709 12:21:54.269974 17727 leader_election.cc:304] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:21:54.269356 20771 raft_consensus.cc:2955] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 5 LEADER]: Committing config change with OpId 3.6: config changed from index 5 to 6, VOTER e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:21:54.272656 20368 raft_consensus.cc:2749] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:21:54.273547 18374 raft_consensus.cc:1275] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260709 12:21:54.218528 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:54.221077 18361 raft_consensus.cc:1240] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 5. Current term is 6. Ops: []
I20260709 12:21:54.228324 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:54.228965 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:54.275584 17684 raft_consensus.cc:2468] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 7.
I20260709 12:21:54.275722 17688 raft_consensus.cc:2393] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 7: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:21:54.276464 17686 raft_consensus.cc:2468] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 10.
I20260709 12:21:54.276511 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:54.278653 20227 raft_consensus.cc:3055] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:21:54.279053 18364 raft_consensus.cc:2393] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 8: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:21:54.279160 20227 raft_consensus.cc:740] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:21:54.198609 20768 leader_election.cc:290] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:54.237550 20157 raft_consensus.cc:515] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:54.281441 20157 leader_election.cc:290] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:54.247699 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:54.282308 18363 raft_consensus.cc:2393] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 7: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
W20260709 12:21:54.248174 17584 rpcz_store.cc:269] 0709 12:21:36.254285 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:36.294891 (+ 40606us) service_pool.cc:224] Handling call
0709 12:21:36.978419 (+683528us) spinlock_profiling.cc:243] Waited 40.6 ms on lock 0x7b540010d2a8. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b50812331 00007f7b5080d3fc 00007f7b503dee84 00007f7b504dd04e 00007f7b4e99e2a4 00007f7b4e99e232 00007f7b4e99e161 00007f7b4e99e0dc 00007f7b4e99d392 00007f7b4895b48c 00007f7b4895a8c6
0709 12:21:54.247356 (+17268937us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:21:42.534684 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:21:42.535257 (+   573us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:21:42.535297 (+    40us) write_op.cc:435] Acquired schema lock
0709 12:21:42.535313 (+    16us) tablet.cc:662] Decoding operations
0709 12:21:42.536868 (+  1555us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:21:42.536940 (+    72us) write_op.cc:641] Partition lock acquired for write op
0709 12:21:42.536975 (+    35us) tablet.cc:685] Acquiring locks for 17 operations
0709 12:21:42.538198 (+  1223us) tablet.cc:701] Row locks acquired
0709 12:21:42.538226 (+    28us) write_op.cc:260] PREPARE: finished
0709 12:21:42.538364 (+   138us) write_op.cc:270] Start()
0709 12:21:42.538471 (+   107us) write_op.cc:276] Timestamp: P: 1783599702538319 usec, L: 0
0709 12:21:42.538490 (+    19us) op_driver.cc:348] REPLICATION: starting
0709 12:21:42.539144 (+   654us) log.cc:844] Serialized 6654 byte log entry
0709 12:21:43.542411 (+1003267us) op_driver.cc:464] REPLICATION: finished
0709 12:21:44.115190 (+572779us) write_op.cc:301] APPLY: starting
0709 12:21:44.115280 (+    90us) tablet.cc:1367] starting BulkCheckPresence
0709 12:21:44.115621 (+   341us) tablet.cc:1370] finished BulkCheckPresence
0709 12:21:44.115774 (+   153us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:21:44.119012 (+  3238us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:21:44.119406 (+   394us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=17
0709 12:21:44.119451 (+    45us) write_op.cc:312] APPLY: finished
0709 12:21:44.119844 (+   393us) log.cc:844] Serialized 119 byte log entry
0709 12:21:44.128386 (+  8542us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:21:44.128598 (+   212us) write_op.cc:454] Released schema lock
0709 12:21:44.128699 (+   101us) write_op.cc:341] FINISH: Updating metrics
0709 12:21:44.134628 (+  5929us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:21:44.134681 (+    53us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":636291,"reported_tablets":33,"spinlock_wait_cycles":93294464,"child_traces":[["op",{"apply.queue_time_us":572468,"apply.run_cpu_time_us":5803,"apply.run_wall_time_us":19372,"num_ops":17,"prepare.queue_time_us":781849,"prepare.run_cpu_time_us":35316,"prepare.run_wall_time_us":193854,"raft.queue_time_us":691444,"raft.run_cpu_time_us":29167,"raft.run_wall_time_us":384780,"replication_time_us":1003726,"thread_start_us":1135094,"threads_started":4,"wal-append.queue_time_us":257607,"wal-append.run_cpu_time_us":96161,"wal-append.run_wall_time_us":2364270}]]}
I20260709 12:21:54.218086 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:54.208436 20851 leader_election.cc:290] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:54.290120 20784 raft_consensus.cc:515] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:54.293545 17682 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37396 (request call id 1924) took 2255 ms. Trace:
I20260709 12:21:54.189217 20766 raft_consensus.cc:2749] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260709 12:21:54.294569 17728 leader_election.cc:336] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Timed out: RequestConsensusVote RPC to 127.16.29.65:33071 timed out after 1.843s (SENT)
W20260709 12:21:54.300526 17729 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.16.29.68:42577 timed out after 1.560s (SENT) [suppressed 67 similar messages]
W20260709 12:21:54.301431 17832 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37994 (request call id 712) took 2088 ms (client timeout 2180 ms). Trace:
I20260709 12:21:54.145794 20862 raft_consensus.cc:515] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:54.203636 20762 raft_consensus.cc:515] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:21:54.304107 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:54.306761 20829 leader_election.cc:290] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 5 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:54.306208 17583 catalog_manager.cc:5697] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b reported cstate change: term changed from 3 to 4, leader changed from e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66) to e063541542a94969898868bf6eb5ce1b (127.16.29.67). New cstate: current_term: 4 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: HEALTHY } } }
I20260709 12:21:54.308251 17761 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36572 (request call id 613) took 2083 ms. Trace:
I20260709 12:21:54.309515 17830 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37994 (request call id 715) took 1805 ms. Trace:
I20260709 12:21:54.310590 18368 raft_consensus.cc:2468] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 7.
I20260709 12:21:54.316325 20776 raft_consensus.cc:515] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:54.292765 20781 raft_consensus.cc:2804] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:21:54.317499 20539 leader_election.cc:290] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:54.298305 20847 leader_election.cc:290] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:54.318553 20772 leader_election.cc:290] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 8 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:54.320945 20480 raft_consensus.cc:515] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:54.324563 17836 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35040 (request call id 2065) took 1274 ms. Trace:
W20260709 12:21:54.325829 17730 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:54.421789 17730 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:54.421201 17836 rpcz_store.cc:276] 0709 12:21:53.050434 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:53.661193 (+610759us) service_pool.cc:224] Handling call
0709 12:21:53.661474 (+   281us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:53.672540 (+ 11066us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:53.672586 (+    46us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:53.672648 (+    62us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:21:53.672701 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:54.324348 (+651647us) spinlock_profiling.cc:243] Waited 10.8 ms on lock 0x7b5400267854. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:21:54.324392 (+    44us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:54.324506 (+   114us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":24774912}
W20260709 12:21:54.420892 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 489034us
I20260709 12:21:54.423235 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:54.325443 20227 consensus_queue.cc:260] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:54.328383 17829 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35040 (request call id 2046) took 2771 ms. Trace:
I20260709 12:21:54.332891 20683 raft_consensus.cc:515] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:54.334069 20837 raft_consensus.cc:515] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:54.336673 20697 raft_consensus.cc:2749] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:21:54.336905 20788 consensus_queue.cc:260] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:54.447400 20788 raft_consensus.cc:3060] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:54.338153 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:54.449244 17838 raft_consensus.cc:3060] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:21:54.293941 20808 consensus_meta.cc:287] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.616s	user 0.005s	sys 0.006s
I20260709 12:21:54.338886 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:54.450263 17831 raft_consensus.cc:2468] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 6.
I20260709 12:21:54.183223 20799 raft_consensus.cc:515] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:54.349229 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 8 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:54.380726 17682 rpcz_store.cc:276] 0709 12:21:52.037895 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:52.041781 (+  3886us) service_pool.cc:224] Handling call
0709 12:21:52.058672 (+ 16891us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:53.016490 (+957818us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:54.291415 (+1274925us) spinlock_profiling.cc:243] Waited 282 ms on lock 0x7b54001aa754. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:21:54.293480 (+  2065us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":648712448}
W20260709 12:21:54.392432 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 91939.3us
W20260709 12:21:54.393467 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 98927.3us
W20260709 12:21:54.395087 18334 leader_election.cc:336] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Timed out: RequestConsensusVote RPC to 127.16.29.65:33071 timed out after 2.368s (SENT)
W20260709 12:21:54.456430 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 61383.3us
W20260709 12:21:54.396055 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 45256.1us
I20260709 12:21:54.458954 17652 leader_election.cc:304] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:21:54.396855 18361 raft_consensus.cc:1240] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 6 LEADER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 5. Current term is 6. Ops: [5.6-5.6]
W20260709 12:21:54.392226 17832 rpcz_store.cc:269] 0709 12:21:52.213383 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:52.282779 (+ 69396us) service_pool.cc:224] Handling call
0709 12:21:54.301235 (+2018456us) spinlock_profiling.cc:243] Waited 605 ms on lock 0x7b5400334fd4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:21:54.301372 (+   137us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1392238464}
I20260709 12:21:54.459733 17838 raft_consensus.cc:2468] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 5.
I20260709 12:21:54.409119 18371 raft_consensus.cc:1240] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 6. Current term is 7. Ops: [6.6-6.6]
I20260709 12:21:54.408586 17830 rpcz_store.cc:276] 0709 12:21:52.504607 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:53.597348 (+1092741us) service_pool.cc:224] Handling call
0709 12:21:53.597643 (+   295us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:53.792601 (+194958us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:53.792649 (+    48us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:53.792713 (+    64us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:21:53.792771 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:54.309281 (+516510us) spinlock_profiling.cc:243] Waited 195 ms on lock 0x7b540008cad4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:21:54.309327 (+    46us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:54.309455 (+   128us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":447767424}
W20260709 12:21:54.411084 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:54.408461 17761 rpcz_store.cc:276] 0709 12:21:52.225056 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:52.225182 (+   126us) service_pool.cc:224] Handling call
0709 12:21:52.463184 (+238002us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:52.568551 (+105367us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:52.568584 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:53.024950 (+456366us) log.cc:844] Serialized 26 byte log entry
0709 12:21:53.547950 (+523000us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:21:53.547986 (+    36us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:53.548053 (+    67us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:54.307952 (+759899us) raft_consensus.cc:1685] finished
0709 12:21:54.307985 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:54.308173 (+   188us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":23739,"raft.queue_time_us":23892,"raft.run_cpu_time_us":132,"raft.run_wall_time_us":81909,"thread_start_us":479249,"threads_started":1,"wal-append.queue_time_us":723137}
W20260709 12:21:54.472029 17728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:54.472391 17652 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:54.474058 17757 raft_consensus.cc:1240] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 7. Current term is 8. Ops: [5.6-7.7]
W20260709 12:21:54.472556 17728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:54.412506 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 343865us
W20260709 12:21:54.385296 20711 consensus_meta.cc:287] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.527s	user 0.007s	sys 0.003s
I20260709 12:21:54.426115 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:54.427686 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:54.494529 18334 leader_election.cc:304] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:21:54.429155 18363 raft_consensus.cc:1240] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 5. Current term is 6. Ops: []
I20260709 12:21:54.429888 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 11 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:54.432224 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:54.432986 18333 leader_election.cc:304] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:21:54.434192 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 295170us
I20260709 12:21:54.439015 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:54.439507 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:54.384893 20687 consensus_queue.cc:1048] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.043s
I20260709 12:21:54.410645 20781 raft_consensus.cc:493] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:21:54.444258 17759 raft_consensus.cc:1240] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 6. Current term is 7. Ops: [6.6-6.6]
I20260709 12:21:54.506422 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:54.500058 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:54.507251 17761 raft_consensus.cc:2368] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for earlier term 4. Current term is 5.
I20260709 12:21:54.501453 17760 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 4.
W20260709 12:21:54.502466 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 46948.3us
W20260709 12:21:54.502753 17728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:54.504791 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:54.499555 17762 raft_consensus.cc:1240] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260709 12:21:54.513137 17583 catalog_manager.cc:5697] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b reported cstate change: term changed from 2 to 5, leader changed from 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) to e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66). New cstate: current_term: 5 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:21:54.505458 17801 leader_election.cc:304] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:21:54.517751 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:54.518427 17688 raft_consensus.cc:2468] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 4.
I20260709 12:21:54.521034 17756 raft_consensus.cc:2468] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 4.
W20260709 12:21:54.495448 20810 consensus_meta.cc:287] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.596s	user 0.010s	sys 0.000s
I20260709 12:21:54.469961 20512 raft_consensus.cc:493] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:54.527251 17801 leader_election.cc:304] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:21:54.528132 17802 connection.cc:256] RPC call timeout handler was delayed by -2.2961623880000843s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 1.5s for a response
I20260709 12:21:54.529263 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:54.529806 17838 raft_consensus.cc:3060] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:54.531090 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:54.532696 17729 leader_election.cc:304] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:21:54.538149 19826 raft_consensus.cc:2804] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:21:54.538405 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:54.539052 17832 raft_consensus.cc:2468] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 4.
I20260709 12:21:54.540671 17832 raft_consensus.cc:1240] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 7 LEADER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 6. Current term is 7. Ops: []
W20260709 12:21:54.586210 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 50656us
W20260709 12:21:54.588670 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 50634.6us
I20260709 12:21:54.591246 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:54.676808 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:54.676992 18368 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 4.
I20260709 12:21:54.499735 20760 raft_consensus.cc:2804] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:21:54.678829 20760 raft_consensus.cc:493] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:54.679251 20760 raft_consensus.cc:3060] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:54.679867 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:54.591737 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:54.592494 17758 raft_consensus.cc:2393] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 7: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:21:54.593238 17652 leader_election.cc:304] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:21:54.681921 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 8 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:54.514357 20781 raft_consensus.cc:3060] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:21:54.682514 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:54.682768 17764 raft_consensus.cc:2410] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for term 8 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260709 12:21:54.682998 17758 raft_consensus.cc:3055] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:21:54.683094 17684 raft_consensus.cc:2468] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 4.
I20260709 12:21:54.683461 17758 raft_consensus.cc:740] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 7 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:21:54.505874 20693 raft_consensus.cc:493] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:21:54.668834 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:54.592622 17801 leader_election.cc:304] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:21:54.540729 20769 raft_consensus.cc:493] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:54.700234 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:21:54.700341 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 107769us
I20260709 12:21:54.701639 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 9 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:54.690994 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:54.692045 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:54.692509 17654 leader_election.cc:304] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:54.693586 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:54.712774 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:54.591012 17583 catalog_manager.cc:5697] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b reported cstate change: term changed from 2 to 4, leader changed from 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) to e063541542a94969898868bf6eb5ce1b (127.16.29.67). New cstate: current_term: 4 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:54.515151 20808 consensus_queue.cc:1048] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.159s
I20260709 12:21:54.521899 20711 consensus_queue.cc:1059] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.165s
I20260709 12:21:54.661613 20851 raft_consensus.cc:2749] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260709 12:21:54.718708 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 224254us
I20260709 12:21:54.731040 18334 leader_election.cc:304] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:21:54.718364 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 285439us
I20260709 12:21:54.719035 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:54.722831 17688 raft_consensus.cc:1275] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 8 index: 4. (index mismatch)
I20260709 12:21:54.723191 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:54.735019 17835 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 7.
I20260709 12:21:54.737016 17832 raft_consensus.cc:2393] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 6: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:21:54.740496 17832 raft_consensus.cc:1240] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 7. Current term is 8. Ops: [7.8-7.8]
I20260709 12:21:54.661315 20157 raft_consensus.cc:2804] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260709 12:21:54.724763 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 131599us
I20260709 12:21:54.725929 17833 raft_consensus.cc:1240] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 7. Current term is 8. Ops: []
I20260709 12:21:54.727912 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 9 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:54.533703 20697 consensus_queue.cc:1048] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 3, Time since last communication: 0.168s
I20260709 12:21:54.540939 20810 raft_consensus.cc:515] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:54.592415 20144 consensus_queue.cc:1059] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.126s
W20260709 12:21:54.744601 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 52184.9us
I20260709 12:21:54.757347 17838 raft_consensus.cc:2468] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 7.
I20260709 12:21:54.699534 20635 raft_consensus.cc:493] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:21:54.590714 20512 raft_consensus.cc:515] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:54.743484 18333 leader_election.cc:304] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:21:54.716946 20604 raft_consensus.cc:2804] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:21:54.773156 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:54.774993 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:54.780223 17652 leader_election.cc:400] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 4 pre-election: Vote denied by peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651) with higher term. Message: Invalid argument: T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for earlier term 4. Current term is 5.
I20260709 12:21:54.828922 17652 leader_election.cc:403] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:21:54.780318 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:54.829764 18361 raft_consensus.cc:2376] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 10. Re-sending same reply.
I20260709 12:21:54.783583 18371 raft_consensus.cc:1275] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260709 12:21:54.830533 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:54.689754 20368 consensus_queue.cc:1059] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 7.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.065s
I20260709 12:21:54.792215 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:54.834978 17756 raft_consensus.cc:3060] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:21:54.824960 17730 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:54.825081 17830 raft_consensus.cc:1240] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 7 LEADER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 6. Current term is 7. Ops: []
I20260709 12:21:54.732326 20829 raft_consensus.cc:2804] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:21:54.732486 20674 raft_consensus.cc:2804] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:21:54.590353 20887 consensus_queue.cc:1059] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.191s
W20260709 12:21:54.826009 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 48454us
I20260709 12:21:54.826431 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:54.865373 17833 raft_consensus.cc:2393] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 8: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:21:54.866217 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:54.867583 17833 raft_consensus.cc:1275] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260709 12:21:54.826750 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:54.827492 16501 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260709 12:21:54.833019 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:54.833246 17838 raft_consensus.cc:1275] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260709 12:21:54.838187 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:54.869187 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:54.869529 17759 raft_consensus.cc:3060] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:54.869874 17760 raft_consensus.cc:2468] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 4.
I20260709 12:21:54.870601 16501 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:21:54.870759 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:54.871833 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:54.768177 20846 raft_consensus.cc:2749] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:21:54.767459 20809 raft_consensus.cc:2749] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:21:54.834236 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:54.766449 20157 raft_consensus.cc:493] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:54.873652 20157 raft_consensus.cc:3060] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:54.832057 20635 raft_consensus.cc:515] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:54.725097 20769 raft_consensus.cc:515] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:54.879146 20635 leader_election.cc:290] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:54.839455 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:54.879930 20769 leader_election.cc:290] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:54.880175 20683 leader_election.cc:290] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 8 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:54.880641 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:54.880838 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:54.881472 17755 raft_consensus.cc:2468] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 6.
I20260709 12:21:54.833935 17729 leader_election.cc:304] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:21:54.882283 20888 log.cc:927] Time spent T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.286s	user 0.000s	sys 0.001s
I20260709 12:21:54.880226 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:54.883340 20368 consensus_queue.cc:1243] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer 0da169ff82674dc1947055ce43de68a3 log is divergent from this leader: its last log entry 8.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260709 12:21:54.884289 20762 leader_election.cc:290] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 5 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:54.851994 17583 catalog_manager.cc:5697] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b reported cstate change: term changed from 2 to 4, leader changed from 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) to e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66). New cstate: current_term: 4 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:21:54.857239 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:54.858486 17757 raft_consensus.cc:2410] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 9 because replica has last-logged OpId of term: 7 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 7.
W20260709 12:21:54.885663 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 105561us
I20260709 12:21:54.718326 20693 raft_consensus.cc:515] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:54.871793 16501 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:21:54.882335 17831 raft_consensus.cc:2410] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for term 6 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260709 12:21:54.888557 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:21:54.882521 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 48674.5us
I20260709 12:21:54.892760 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:54.893477 18361 raft_consensus.cc:3060] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:54.887634 17652 leader_election.cc:304] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:54.859226 17756 raft_consensus.cc:2468] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 8.
I20260709 12:21:54.911228 17759 raft_consensus.cc:2468] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 6.
I20260709 12:21:54.923561 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:54.928519 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:54.929162 17838 raft_consensus.cc:2468] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 6.
I20260709 12:21:54.880884 20683 raft_consensus.cc:493] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:54.931479 17684 raft_consensus.cc:1240] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 6. Current term is 7. Ops: []
I20260709 12:21:54.933302 17583 catalog_manager.cc:5697] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "334a4e1f59944544b3b3ff36616b57da" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:21:54.828277 20604 raft_consensus.cc:493] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
W20260709 12:21:54.883419 20893 log.cc:927] Time spent T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.194s	user 0.000s	sys 0.000s
I20260709 12:21:54.931097 20683 raft_consensus.cc:515] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:54.930038 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:54.951133 17833 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 7.
I20260709 12:21:54.944625 17760 raft_consensus.cc:1275] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 7. (term mismatch)
I20260709 12:21:54.945076 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
W20260709 12:21:54.948578 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 83384.1us
I20260709 12:21:54.955212 17654 leader_election.cc:304] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:54.854990 20674 raft_consensus.cc:493] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:21:54.955384 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:54.956069 20674 raft_consensus.cc:3060] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:54.977957 17760 pending_rounds.cc:85] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb: Aborting all ops after (but not including) 6
I20260709 12:21:54.983472 17760 pending_rounds.cc:107] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb: Aborting uncommitted NO_OP operation due to leader change: 6.7
I20260709 12:21:54.984737 17760 raft_consensus.cc:2889] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:21:54.992985 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 105424us
I20260709 12:21:54.940145 20808 raft_consensus.cc:493] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:55.015595 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 60446.7us
I20260709 12:21:54.936825 20776 leader_election.cc:290] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 6 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:54.947180 20762 raft_consensus.cc:493] T 7cb3768d215e409fb9f5baf18d124442 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:54.937427 20697 raft_consensus.cc:3055] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:21:54.937162 20480 leader_election.cc:290] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 5 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:55.018548 17652 leader_election.cc:304] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:55.022672 17583 catalog_manager.cc:5697] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b reported cstate change: term changed from 2 to 4, leader changed from 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) to e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66). New cstate: current_term: 4 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:21:55.024127 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:54.947394 20604 raft_consensus.cc:3060] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:54.944008 20687 raft_consensus.cc:3060] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Advancing to term 5
I20260709 12:21:55.041318 17583 catalog_manager.cc:5697] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b reported cstate change: term changed from 2 to 3, leader changed from 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) to e063541542a94969898868bf6eb5ce1b (127.16.29.67). New cstate: current_term: 3 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: HEALTHY } } }
I20260709 12:21:54.951974 20829 raft_consensus.cc:697] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 5 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:21:54.903323 20879 consensus_queue.cc:1048] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.187s
I20260709 12:21:54.980087 20837 leader_election.cc:290] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 5 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:55.062305 17654 leader_election.cc:304] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:54.980634 20702 raft_consensus.cc:2804] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:21:55.016939 20795 raft_consensus.cc:2749] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260709 12:21:55.065110 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 174249us
I20260709 12:21:55.069893 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:55.070559 17833 raft_consensus.cc:2468] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 4.
I20260709 12:21:55.071520 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:55.074677 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:54.431298 17829 rpcz_store.cc:276] 0709 12:21:51.556929 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:51.557938 (+  1009us) service_pool.cc:224] Handling call
0709 12:21:53.676395 (+2118457us) spinlock_profiling.cc:243] Waited 6.49 ms on lock 0x7b580013fe98. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b474fbb71 00007f7b474fbeee 000000000036eb39 00007f7b4f56f1e0 00007f7b4f6380c4 00007f7b4f638131 00007f7b4f5f9fc0 00007f7b4f5e4929 00007f7b4f5e43d2 00007f7b48b5bbd4
0709 12:21:53.676554 (+   159us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:53.811935 (+135381us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:53.825423 (+ 13488us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:53.906047 (+ 80624us) log.cc:844] Serialized 26 byte log entry
0709 12:21:54.168071 (+262024us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:21:54.174953 (+  6882us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:54.180949 (+  5996us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:54.293455 (+112506us) raft_consensus.cc:1685] finished
0709 12:21:54.293472 (+    17us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:54.328309 (+ 34837us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":113576,"raft.queue_time_us":121552,"raft.run_cpu_time_us":186,"raft.run_wall_time_us":22011,"spinlock_wait_cycles":14923520,"thread_start_us":178932,"threads_started":1,"wal-append.queue_time_us":256180}
W20260709 12:21:55.219607 20891 log.cc:927] Time spent T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.275s	user 0.000s	sys 0.001s
W20260709 12:21:55.125301 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 394325us
I20260709 12:21:55.064780 20810 leader_election.cc:290] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 8 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:55.025864 20809 consensus_queue.cc:1059] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.083s
I20260709 12:21:55.125957 20855 raft_consensus.cc:2749] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:21:55.014201 20864 consensus_queue.cc:1048] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.076s
I20260709 12:21:54.994546 20299 consensus_queue.cc:1059] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.077s
W20260709 12:21:55.114477 20896 log.cc:927] Time spent T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.094s	user 0.000s	sys 0.000s
I20260709 12:21:55.116482 20702 raft_consensus.cc:697] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 10 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:21:55.277503 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:55.409924 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:55.411372 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:21:55.115489 20897 log.cc:927] Time spent T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.256s	user 0.000s	sys 0.000s
W20260709 12:21:55.210901 20898 log.cc:927] Time spent T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.185s	user 0.001s	sys 0.000s
I20260709 12:21:55.131318 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:55.029011 20762 raft_consensus.cc:515] T 7cb3768d215e409fb9f5baf18d124442 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:55.025470 20763 raft_consensus.cc:2749] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:21:54.978760 20377 consensus_queue.cc:1059] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.083s
W20260709 12:21:55.234954 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 491492us
I20260709 12:21:55.419919 18333 leader_election.cc:304] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:21:55.025458 20862 leader_election.cc:290] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:55.193473 20851 raft_consensus.cc:493] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:21:55.022686 20697 raft_consensus.cc:740] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 6 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:21:55.135113 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:55.423141 18368 raft_consensus.cc:3060] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:21:55.423219 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:21:55.211138 20781 consensus_meta.cc:287] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.528s	user 0.008s	sys 0.003s
I20260709 12:21:55.235363 20795 raft_consensus.cc:2749] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:21:55.321198 18361 raft_consensus.cc:2468] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 5.
W20260709 12:21:55.319231 17763 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.811s	user 0.000s	sys 0.011s
W20260709 12:21:55.244406 19826 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.705s	user 0.012s	sys 0.031s
I20260709 12:21:55.126941 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
W20260709 12:21:55.244009 17762 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.519s	user 0.007s	sys 0.034s
W20260709 12:21:55.339787 20890 log.cc:927] Time spent T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.488s	user 0.004s	sys 0.001s
W20260709 12:21:55.351199 17834 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.001s	user 0.005s	sys 0.010s
I20260709 12:21:55.354455 18334 leader_election.cc:304] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:55.309897 20842 raft_consensus.cc:2749] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20260709 12:21:55.210628 20889 log.cc:927] Time spent T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.501s	user 0.001s	sys 0.000s
W20260709 12:21:55.425987 20900 log.cc:927] Time spent T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.207s	user 0.001s	sys 0.000s
I20260709 12:21:55.330250 20799 leader_election.cc:290] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:55.412999 17838 raft_consensus.cc:2468] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 6.
W20260709 12:21:55.499768 17837 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.812s	user 0.000s	sys 0.041s
W20260709 12:21:55.508538 17837 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.821s	user 0.001s	sys 0.041s
I20260709 12:21:55.508656 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:55.509469 17838 raft_consensus.cc:2468] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 4.
I20260709 12:21:55.510536 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:55.511463 17838 raft_consensus.cc:2393] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 6: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
W20260709 12:21:55.417876 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 399362us
W20260709 12:21:55.381181 20771 consensus_meta.cc:287] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 1.110s	user 0.005s	sys 0.008s
W20260709 12:21:55.500861 17730 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.644s	user 0.038s	sys 0.036s
W20260709 12:21:55.512202 17730 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.656s	user 0.038s	sys 0.036s
W20260709 12:21:55.451695 17728 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.778s	user 0.055s	sys 0.053s
W20260709 12:21:55.479902 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 417626us
W20260709 12:21:55.512719 17728 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.839s	user 0.056s	sys 0.055s
W20260709 12:21:55.513031 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 993259us
I20260709 12:21:55.513234 17652 leader_election.cc:304] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:55.513736 17801 leader_election.cc:304] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:21:55.380084 20886 consensus_queue.cc:1048] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.118s
W20260709 12:21:55.488955 17763 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.980s	user 0.000s	sys 0.011s
W20260709 12:21:55.489519 19826 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.951s	user 0.013s	sys 0.033s
I20260709 12:21:55.524158 19826 raft_consensus.cc:697] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 7 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
W20260709 12:21:55.490029 17762 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.772s	user 0.010s	sys 0.036s
W20260709 12:21:55.495111 17834 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.145s	user 0.005s	sys 0.011s
W20260709 12:21:55.495868 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 76095.5us
W20260709 12:21:55.496538 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 142176us
I20260709 12:21:55.527071 18334 leader_election.cc:304] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:55.412146 20809 consensus_queue.cc:1243] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer e2909cff1da44ad387f96a8ae3aa5afb log is divergent from this leader: its last log entry 6.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260709 12:21:55.494298 18368 raft_consensus.cc:2468] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 8.
I20260709 12:21:55.391605 17758 consensus_queue.cc:260] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:55.504551 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:55.503993 17802 leader_election.cc:304] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
W20260709 12:21:55.426605 17572 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet d522f70beb1f417cb98940d90b594823 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 0da169ff82674dc1947055ce43de68a3 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 4 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
W20260709 12:21:55.480057 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:55.529243 17572 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 102684us
I20260709 12:21:55.024166 20808 raft_consensus.cc:515] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:55.516928 17801 leader_election.cc:304] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:55.517467 17728 leader_election.cc:304] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:21:55.532902 17728 leader_election.cc:304] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:21:55.521924 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:21:55.495258 20891 log.cc:927] Time spent T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.087s	user 0.000s	sys 0.000s
I20260709 12:21:55.534413 17838 raft_consensus.cc:2393] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 8: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:21:55.535239 17654 leader_election.cc:304] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:55.537746 19826 consensus_queue.cc:237] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:55.540768 20903 raft_consensus.cc:2804] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:21:55.488813 20880 raft_consensus.cc:2764] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20260709 12:21:55.526229 18333 leader_election.cc:304] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:21:55.528412 17758 raft_consensus.cc:3060] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:21:55.502964 20683 leader_election.cc:290] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:55.532480 18362 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 1058) took 1167 ms. Trace:
I20260709 12:21:55.569001 18362 rpcz_store.cc:276] 0709 12:21:54.365257 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:54.382302 (+ 17045us) service_pool.cc:224] Handling call
0709 12:21:54.382662 (+   360us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:54.383030 (+   368us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:54.383068 (+    38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:54.407615 (+ 24547us) log.cc:844] Serialized 26 byte log entry
0709 12:21:54.502145 (+ 94530us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:21:54.502192 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:54.502256 (+    64us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:55.532276 (+1030020us) raft_consensus.cc:1685] finished
0709 12:21:55.532294 (+    18us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:55.532425 (+   131us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":732,"raft.run_cpu_time_us":124,"raft.run_wall_time_us":7442,"thread_start_us":64758,"threads_started":1,"wal-append.queue_time_us":314749}
I20260709 12:21:55.528486 20847 raft_consensus.cc:2804] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:21:55.546692 20808 leader_election.cc:290] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:55.542305 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:55.547274 20799 raft_consensus.cc:2749] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:21:55.548115 20905 consensus_queue.cc:1059] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:21:55.550199 17764 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38776 (request call id 1894) took 1119 ms. Trace:
I20260709 12:21:55.586287 17764 rpcz_store.cc:276] 0709 12:21:54.431277 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:54.683417 (+252140us) service_pool.cc:224] Handling call
0709 12:21:54.683677 (+   260us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:54.687918 (+  4241us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:54.687946 (+    28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:54.701929 (+ 13983us) log.cc:844] Serialized 26 byte log entry
0709 12:21:54.989104 (+287175us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:21:54.989161 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:54.989254 (+    93us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:55.549995 (+560741us) raft_consensus.cc:1685] finished
0709 12:21:55.550019 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:55.550139 (+   120us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":7088,"raft.queue_time_us":6525,"raft.run_cpu_time_us":134,"raft.run_wall_time_us":17422,"thread_start_us":219582,"threads_started":1,"wal-append.queue_time_us":771101}
I20260709 12:21:55.556052 20702 consensus_queue.cc:237] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:21:55.560927 20788 consensus_meta.cc:287] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 1.113s	user 0.012s	sys 0.000s
I20260709 12:21:55.561463 17682 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37402 (request call id 2330) took 1078 ms. Trace:
I20260709 12:21:55.561698 18365 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 1055) took 1321 ms. Trace:
I20260709 12:21:55.513935 20773 raft_consensus.cc:2749] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:21:55.515775 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:55.583823 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:55.618618 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:55.671126 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:55.621675 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:55.524367 20299 raft_consensus.cc:3055] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:21:55.497835 20851 raft_consensus.cc:515] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:55.529306 20846 raft_consensus.cc:2764] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Leader pre-election decision vote started in defunct term 7: won
I20260709 12:21:55.642094 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:55.628554 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:21:55.530522 17730 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:55.704735 17681 raft_consensus.cc:1275] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 6. (term mismatch)
I20260709 12:21:55.705200 17681 pending_rounds.cc:85] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3: Aborting all ops after (but not including) 5
I20260709 12:21:55.705492 17681 pending_rounds.cc:107] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3: Aborting uncommitted NO_OP operation due to leader change: 5.6
I20260709 12:21:55.544013 20480 raft_consensus.cc:2749] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:21:55.498111 20781 raft_consensus.cc:515] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:55.657171 18365 rpcz_store.cc:276] 0709 12:21:54.240379 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:54.258923 (+ 18544us) service_pool.cc:224] Handling call
0709 12:21:54.259284 (+   361us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:54.259633 (+   349us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:54.259660 (+    27us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:54.262739 (+  3079us) log.cc:844] Serialized 26 byte log entry
0709 12:21:54.345058 (+ 82319us) spinlock_profiling.cc:243] Waited 38.1 ms on lock 0x7b4800220a58. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4f549aa1 00007f7b4f540b37 00007f7b4f5409ca 00007f7b4f63826f 00007f7b4f637f37 00007f7b4f64c55c 00007f7b4f64c508 00007f7b4f64c488 00007f7b4f64c443 00007f7b4f64b6ec 00007f7b4e575a96
0709 12:21:54.422527 (+ 77469us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:21:54.422585 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:54.422684 (+    99us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:55.291051 (+868367us) spinlock_profiling.cc:243] Waited 71.9 ms on lock 0x7b4400022b4c. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b47459c85 00007f7b4745a6a2 00007f7b4a1cc1db 00007f7b4a1c7aaf 00007f7b4a1bb86e 00007f7b4a1babb6 00007f7b4a1ba7d8 00007f7b4a1c4221 00007f7b4a1c41d9 00007f7b4a1c4169
0709 12:21:55.561510 (+270459us) raft_consensus.cc:1685] finished
0709 12:21:55.561531 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:55.561645 (+   114us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":118,"raft.run_cpu_time_us":2196,"raft.run_wall_time_us":85471,"spinlock_wait_cycles":253049472,"thread_start_us":159389,"threads_started":1,"wal-append.queue_time_us":505327}
I20260709 12:21:55.563935 20762 leader_election.cc:290] T 7cb3768d215e409fb9f5baf18d124442 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:55.666810 18368 raft_consensus.cc:1275] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260709 12:21:55.703337 18367 raft_consensus.cc:2955] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Committing config change with OpId 3.6: config changed from index 5 to 6, VOTER e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:21:55.665771 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:55.541215 20903 raft_consensus.cc:493] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:55.657006 17682 rpcz_store.cc:276] 0709 12:21:54.482636 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:54.482808 (+   172us) service_pool.cc:224] Handling call
0709 12:21:54.505476 (+ 22668us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:21:54.523669 (+ 18193us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:54.523714 (+    45us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:21:54.530405 (+  6691us) log.cc:844] Serialized 52 byte log entry
0709 12:21:54.757257 (+226852us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:21:54.827838 (+ 70581us) log.cc:844] Serialized 14 byte log entry
0709 12:21:54.881711 (+ 53873us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:54.881797 (+    86us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:55.561173 (+679376us) raft_consensus.cc:1685] finished
0709 12:21:55.561203 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:55.561382 (+   179us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":93043,"prepare.queue_time_us":70624,"prepare.run_cpu_time_us":50,"prepare.run_wall_time_us":4288,"raft.queue_time_us":75308,"raft.run_cpu_time_us":239,"raft.run_wall_time_us":58646,"thread_start_us":47125,"threads_started":1,"wal-append.queue_time_us":387164}
W20260709 12:21:55.572955 20674 consensus_meta.cc:287] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.616s	user 0.011s	sys 0.000s
W20260709 12:21:55.698962 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 185797us
I20260709 12:21:55.564930 20787 raft_consensus.cc:2749] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:21:55.564772 20810 raft_consensus.cc:2804] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:21:55.588282 20346 raft_consensus.cc:2804] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:21:55.701645 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:55.529543 20837 consensus_queue.cc:1048] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.127s
I20260709 12:21:55.576329 20847 raft_consensus.cc:493] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:21:55.771507 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:55.737932 20903 raft_consensus.cc:3060] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:55.774132 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:55.772229 17688 raft_consensus.cc:2468] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 7.
I20260709 12:21:55.775866 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:55.705771 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:55.705773 17681 raft_consensus.cc:2889] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:21:55.707690 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:55.506108 20829 consensus_queue.cc:237] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:55.566897 20377 consensus_queue.cc:1059] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.135s
W20260709 12:21:55.702049 20891 log.cc:927] Time spent T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.124s	user 0.000s	sys 0.000s
I20260709 12:21:55.730710 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:55.762324 20811 raft_consensus.cc:493] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:55.718799 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:55.745527 20892 consensus_queue.cc:1059] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.182s
I20260709 12:21:55.746450 20872 raft_consensus.cc:2749] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260709 12:21:55.816045 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 312130us
I20260709 12:21:55.719739 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:55.700711 20299 raft_consensus.cc:740] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:21:55.759974 17680 raft_consensus.cc:2468] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 10.
W20260709 12:21:55.762738 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 245852us
I20260709 12:21:55.729355 20864 raft_consensus.cc:3055] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:21:55.773087 20847 raft_consensus.cc:3060] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:55.820474 17801 leader_election.cc:304] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:21:55.769887 17652 leader_election.cc:304] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:21:55.777719 17760 raft_consensus.cc:2468] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 5.
W20260709 12:21:55.769098 20910 log.cc:927] Time spent T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.208s	user 0.001s	sys 0.000s
I20260709 12:21:55.784468 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:21:55.838644 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 68856.5us
I20260709 12:21:55.787591 17762 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38776 (request call id 1895) took 1356 ms. Trace:
I20260709 12:21:55.782091 20843 raft_consensus.cc:493] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:55.729733 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:55.779485 20346 raft_consensus.cc:697] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 5 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:21:55.812398 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:55.768319 20810 raft_consensus.cc:493] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:55.818089 17802 leader_election.cc:304] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:55.564261 20697 consensus_queue.cc:260] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:21:55.823050 17728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:55.874601 17728 leader_election.cc:336] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Timed out: RequestConsensusVote RPC to 127.16.29.65:33071 timed out after 1.532s (SENT)
I20260709 12:21:55.874289 20299 consensus_queue.cc:260] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:21:55.875218 17728 leader_election.cc:336] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Timed out: RequestConsensusVote RPC to 127.16.29.65:33071 timed out after 1.533s (SENT)
I20260709 12:21:55.824301 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:55.876072 17680 raft_consensus.cc:3060] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:21:55.781900 20842 consensus_queue.cc:1059] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.073s
I20260709 12:21:55.825991 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:55.756892 20674 raft_consensus.cc:515] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:21:55.828349 20604 consensus_meta.cc:287] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.764s	user 0.012s	sys 0.001s
I20260709 12:21:55.828953 20872 raft_consensus.cc:2804] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:21:55.829787 17801 leader_election.cc:304] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:21:55.829834 20902 log.cc:927] Time spent T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.156s	user 0.000s	sys 0.000s
I20260709 12:21:55.798417 17763 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38776 (request call id 1883) took 1375 ms. Trace:
I20260709 12:21:55.851534 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:55.793056 20762 raft_consensus.cc:493] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:55.864914 20763 raft_consensus.cc:2804] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:21:55.868808 20915 raft_consensus.cc:2749] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:21:55.817183 20811 raft_consensus.cc:515] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:55.873308 20810 raft_consensus.cc:3060] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:21:55.824892 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:21:55.872452 20908 log.cc:927] Time spent T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.073s	user 0.000s	sys 0.000s
I20260709 12:21:55.838898 20864 raft_consensus.cc:740] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 6 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
W20260709 12:21:55.894137 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 76118.5us
W20260709 12:21:55.894240 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 64522.1us
I20260709 12:21:55.875723 20299 raft_consensus.cc:3060] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:21:55.875738 17728 leader_election.cc:336] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 10 election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Timed out: RequestConsensusVote RPC to 127.16.29.65:33071 timed out after 1.560s (SENT)
W20260709 12:21:55.923152 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 47433.6us
W20260709 12:21:55.923779 17728 leader_election.cc:336] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Timed out: RequestConsensusVote RPC to 127.16.29.65:33071 timed out after 1.654s (SENT)
W20260709 12:21:55.924578 17728 leader_election.cc:336] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Timed out: RequestConsensusVote RPC to 127.16.29.65:33071 timed out after 1.712s (SENT)
W20260709 12:21:55.925076 17728 leader_election.cc:336] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Timed out: RequestConsensusVote RPC to 127.16.29.65:33071 timed out after 1.815s (SENT)
W20260709 12:21:55.925570 17728 leader_election.cc:336] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Timed out: RequestConsensusVote RPC to 127.16.29.65:33071 timed out after 1.842s (SENT)
I20260709 12:21:55.884936 20916 raft_consensus.cc:3055] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:21:55.909725 17680 raft_consensus.cc:2468] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 8.
I20260709 12:21:55.888855 20872 raft_consensus.cc:493] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
W20260709 12:21:55.893828 20918 log.cc:927] Time spent T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.055s	user 0.000s	sys 0.001s
I20260709 12:21:55.939234 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:55.895795 20763 raft_consensus.cc:493] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
W20260709 12:21:55.892907 20760 consensus_meta.cc:287] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 1.213s	user 0.008s	sys 0.004s
I20260709 12:21:55.912412 17801 leader_election.cc:304] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:21:55.916049 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:55.919328 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:55.920920 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:55.927441 19826 consensus_queue.cc:1059] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
W20260709 12:21:55.893505 20912 log.cc:927] Time spent T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.125s	user 0.000s	sys 0.001s
I20260709 12:21:55.960584 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:55.931579 17802 leader_election.cc:304] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:55.932996 17728 leader_election.cc:304] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:21:55.909516 18374 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 1056) took 1668 ms. Trace:
I20260709 12:21:55.910181 20862 raft_consensus.cc:2804] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:21:55.940847 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 11 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:55.945560 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:55.963660 17684 raft_consensus.cc:2468] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 10.
W20260709 12:21:55.955868 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 43549.8us
I20260709 12:21:55.957602 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:55.961886 18374 rpcz_store.cc:276] 0709 12:21:54.240617 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:54.274412 (+ 33795us) service_pool.cc:224] Handling call
0709 12:21:54.274755 (+   343us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:54.275097 (+   342us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 12:21:54.278175 (+  3078us) log.cc:844] Serialized 14 byte log entry
0709 12:21:54.341439 (+ 63264us) spinlock_profiling.cc:243] Waited 21 ms on lock 0x7b4800220a58. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4f549aa1 00007f7b4f540b37 00007f7b4f5409ca 00007f7b4f63826f 00007f7b4f637f37 00007f7b4f64c55c 00007f7b4f64c508 00007f7b4f64c488 00007f7b4f64c443 00007f7b4f64b6ec 00007f7b4e575a96
0709 12:21:54.469768 (+128329us) log.cc:844] Serialized 14 byte log entry
0709 12:21:54.710208 (+240440us) spinlock_profiling.cc:243] Waited 19.8 ms on lock 0x7b180034d1a0. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b474adc0a 00007f7b50bed219 00007f7b47535a40 00007f7b475304e4 00007f7b47530455 00007f7b4a618060 00007f7b4a62a716 00007f7b4a63b287 00007f7b4a63b208
0709 12:21:54.710384 (+   176us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:54.710964 (+   580us) log.cc:844] Serialized 26 byte log entry
0709 12:21:54.711127 (+   163us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:21:54.711173 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:54.711252 (+    79us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:55.909295 (+1198043us) raft_consensus.cc:1685] finished
0709 12:21:55.909312 (+    17us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:55.909455 (+   143us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1025546,"prepare.queue_time_us":6067,"prepare.run_cpu_time_us":76,"prepare.run_wall_time_us":76,"raft.queue_time_us":263042,"raft.run_cpu_time_us":1497,"raft.run_wall_time_us":77626,"spinlock_wait_cycles":93797120,"thread_start_us":173524,"threads_started":1,"wal-append.queue_time_us":508933}
I20260709 12:21:55.964745 17685 raft_consensus.cc:2468] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 6.
I20260709 12:21:55.959450 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:55.861054 20843 raft_consensus.cc:515] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:55.893266 20604 raft_consensus.cc:515] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:21:55.953356 20914 log.cc:927] Time spent T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.067s	user 0.000s	sys 0.001s
I20260709 12:21:55.972107 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
W20260709 12:21:55.974859 17728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:55.975910 17830 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35042 (request call id 1811) took 1286 ms. Trace:
I20260709 12:21:55.984613 17689 raft_consensus.cc:3060] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:55.984961 17830 rpcz_store.cc:276] 0709 12:21:54.689754 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:54.689904 (+   150us) service_pool.cc:224] Handling call
0709 12:21:54.690164 (+   260us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:54.793482 (+103318us) spinlock_profiling.cc:243] Waited 82.5 ms on lock 0x7b54000553a0. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a597dc0 00007f7b4a6221c0 00007f7b4a623231 00007f7b4a61fd8f 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c
0709 12:21:54.836421 (+ 42939us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:55.975640 (+1139219us) spinlock_profiling.cc:243] Waited 8.13 ms on lock 0x7b54000553a0. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a597dc0 00007f7b4a623a02 00007f7b4a61fdd6 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2
0709 12:21:55.975839 (+   199us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":208531968}
I20260709 12:21:55.976753 17758 raft_consensus.cc:1275] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
I20260709 12:21:55.940186 20872 raft_consensus.cc:3060] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:55.929839 20916 raft_consensus.cc:740] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
W20260709 12:21:55.987962 17728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:55.987583 20916 consensus_queue.cc:260] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:55.988790 17728 leader_election.cc:304] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:21:55.989632 20693 leader_election.cc:290] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:55.986413 20837 raft_consensus.cc:2749] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:21:55.949407 20811 leader_election.cc:290] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:55.990253 17728 leader_election.cc:304] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:21:55.959654 20760 raft_consensus.cc:515] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:55.992631 20760 leader_election.cc:290] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 7 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:55.993410 17687 raft_consensus.cc:2368] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 6. Current term is 7.
I20260709 12:21:55.993989 17728 leader_election.cc:304] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:21:55.993883 20872 raft_consensus.cc:515] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:55.994875 20911 raft_consensus.cc:2749] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 12:21:55.988822 20916 raft_consensus.cc:3060] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:55.900379 20762 raft_consensus.cc:515] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:55.995491 17687 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37402 (request call id 2332) took 1491 ms. Trace:
I20260709 12:21:55.998147 17758 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38776 (request call id 1892) took 1570 ms. Trace:
I20260709 12:21:56.044989 17833 raft_consensus.cc:2393] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 7: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:21:55.998798 20227 raft_consensus.cc:3060] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:21:56.021348 20711 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 8 current term: 8 is higher.
I20260709 12:21:55.962458 20346 consensus_queue.cc:237] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:55.958038 20763 raft_consensus.cc:3060] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:55.990160 20693 raft_consensus.cc:2804] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:21:55.962335 20768 raft_consensus.cc:2804] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:21:56.021759 20781 leader_election.cc:290] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 10 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:56.183068 17687 rpcz_store.cc:276] 0709 12:21:54.504198 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:54.504561 (+   363us) service_pool.cc:224] Handling call
0709 12:21:55.995136 (+1490575us) spinlock_profiling.cc:243] Waited 1.48 s on lock 0x7b54001aa754. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:21:55.995425 (+   289us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3405264768}
I20260709 12:21:56.266041 17757 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38786 (request call id 2517) took 1562 ms. Trace:
I20260709 12:21:56.266589 17757 rpcz_store.cc:276] 0709 12:21:54.702957 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:54.704298 (+  1341us) service_pool.cc:224] Handling call
0709 12:21:56.265486 (+1561188us) spinlock_profiling.cc:243] Waited 90.9 ms on lock 0x7b54001b8354. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:21:56.265801 (+   315us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":209034496}
W20260709 12:21:56.273051 20157 consensus_meta.cc:287] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 1.399s	user 0.010s	sys 0.006s
I20260709 12:21:56.273746 20157 raft_consensus.cc:515] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:56.275338 17835 raft_consensus.cc:2368] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for earlier term 4. Current term is 5.
W20260709 12:21:56.276194 17831 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35042 (request call id 1813) took 1584 ms (client timeout 1567 ms). Trace:
I20260709 12:21:56.276710 20157 leader_election.cc:290] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 5 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:21:56.277137 17835 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35042 (request call id 1821) took 1556 ms (client timeout 1981 ms). Trace:
I20260709 12:21:56.238458 17834 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35040 (request call id 2080) took 1889 ms. Trace:
W20260709 12:21:56.277932 17835 rpcz_store.cc:269] 0709 12:21:54.720881 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:54.735956 (+ 15075us) service_pool.cc:224] Handling call
0709 12:21:56.276918 (+1540962us) spinlock_profiling.cc:243] Waited 1.45 s on lock 0x7b540008a7d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:21:56.277086 (+   168us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3326296960}
I20260709 12:21:56.278687 17834 rpcz_store.cc:276] 0709 12:21:54.348615 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:54.348723 (+   108us) service_pool.cc:224] Handling call
0709 12:21:54.349060 (+   337us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:54.349648 (+   588us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:54.349677 (+    29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:54.350117 (+   440us) log.cc:844] Serialized 26 byte log entry
0709 12:21:55.525471 (+1175354us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:21:55.525509 (+    38us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:55.525573 (+    64us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:56.237817 (+712244us) raft_consensus.cc:1685] finished
0709 12:21:56.237861 (+    44us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:56.238235 (+   374us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":117,"raft.run_cpu_time_us":72,"raft.run_wall_time_us":72,"thread_start_us":1175004,"threads_started":1,"wal-append.queue_time_us":1321843}
I20260709 12:21:56.247319 20760 raft_consensus.cc:493] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:56.277935 20157 consensus_queue.cc:1048] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.006s
I20260709 12:21:56.284278 20760 raft_consensus.cc:515] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:56.255770 20693 raft_consensus.cc:493] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:56.289275 20693 raft_consensus.cc:3060] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:21:56.257298 20604 leader_election.cc:290] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 7 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:56.260201 20674 leader_election.cc:290] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 4 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:56.220690 17758 rpcz_store.cc:276] 0709 12:21:54.428872 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:54.682628 (+253756us) service_pool.cc:224] Handling call
0709 12:21:54.682891 (+   263us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:55.373254 (+690363us) spinlock_profiling.cc:243] Waited 199 ms on lock 0x7b54003aaaa0. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a59e06f 00007f7b4a616f60 00007f7b4a6187de 00007f7b4a622b49 00007f7b4a623416 00007f7b4a61fd8f 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62
0709 12:21:55.986729 (+613475us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:55.998078 (+ 11349us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":456635776}
I20260709 12:21:56.238366 17836 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35042 (request call id 1818) took 1543 ms. Trace:
I20260709 12:21:56.295920 17836 rpcz_store.cc:276] 0709 12:21:54.694694 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:54.717099 (+ 22405us) service_pool.cc:224] Handling call
0709 12:21:54.717436 (+   337us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:54.717651 (+   215us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:54.717680 (+    29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:54.718133 (+   453us) log.cc:844] Serialized 26 byte log entry
0709 12:21:55.059254 (+341121us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:21:55.320085 (+260831us) log.cc:844] Serialized 14 byte log entry
0709 12:21:55.320382 (+   297us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:55.320478 (+    96us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:56.237608 (+917130us) raft_consensus.cc:1685] finished
0709 12:21:56.237648 (+    40us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:56.238049 (+   401us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":13166,"prepare.run_cpu_time_us":52,"prepare.run_wall_time_us":54,"raft.queue_time_us":116,"raft.run_cpu_time_us":135,"raft.run_wall_time_us":28426,"thread_start_us":282203,"threads_started":1,"wal-append.queue_time_us":763146}
I20260709 12:21:56.258033 20776 raft_consensus.cc:493] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:21:56.299207 20776 raft_consensus.cc:515] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:21:56.276789 17831 rpcz_store.cc:269] 0709 12:21:54.691638 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:54.691766 (+   128us) service_pool.cc:224] Handling call
0709 12:21:56.275604 (+1583838us) spinlock_profiling.cc:243] Waited 179 ms on lock 0x7b54003d4fd4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:21:56.275944 (+   340us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":411888384}
I20260709 12:21:56.300948 20760 leader_election.cc:290] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:56.302273 20776 leader_election.cc:290] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:21:56.208838 17730 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:56.177829 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 338098us
I20260709 12:21:56.250968 20762 leader_election.cc:290] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:56.115943 20847 raft_consensus.cc:515] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:56.313483 17687 raft_consensus.cc:1240] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 9. Current term is 10. Ops: [9.9-9.9]
I20260709 12:21:56.001073 20864 consensus_queue.cc:260] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:56.315376 20762 raft_consensus.cc:493] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:55.945801 20921 raft_consensus.cc:2749] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:21:56.136200 20862 raft_consensus.cc:697] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 8 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:21:56.318019 20862 consensus_queue.cc:237] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:56.315554 20847 leader_election.cc:290] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 5 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:56.320045 20512 leader_election.cc:290] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:56.126773 20943 raft_consensus.cc:2749] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260709 12:21:55.997727 20919 log.cc:927] Time spent T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.109s	user 0.000s	sys 0.000s
I20260709 12:21:56.324081 17832 raft_consensus.cc:1240] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 3. Current term is 4. Ops: []
I20260709 12:21:56.105886 20903 raft_consensus.cc:515] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:56.329869 20903 leader_election.cc:290] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 5 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:21:55.996429 20924 log.cc:927] Time spent T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.084s	user 0.001s	sys 0.000s
I20260709 12:21:56.332310 20783 raft_consensus.cc:3055] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:21:56.332890 20783 raft_consensus.cc:740] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:21:55.959022 20923 consensus_queue.cc:1048] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.384s
I20260709 12:21:56.334548 20783 consensus_queue.cc:260] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:56.337925 20784 leader_election.cc:290] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 4 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:56.340289 17834 raft_consensus.cc:1240] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 6. Current term is 7. Ops: []
I20260709 12:21:55.995522 20911 raft_consensus.cc:2804] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:21:56.342896 20911 raft_consensus.cc:697] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:21:56.344769 20911 consensus_queue.cc:237] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:56.348014 20783 raft_consensus.cc:3060] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:56.354857 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:55.889590 17763 rpcz_store.cc:276] 0709 12:21:54.423404 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:54.423525 (+   121us) service_pool.cc:224] Handling call
0709 12:21:54.440860 (+ 17335us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:21:54.441374 (+   514us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:54.441416 (+    42us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:21:54.442220 (+   804us) log.cc:844] Serialized 52 byte log entry
0709 12:21:55.523751 (+1081531us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:21:55.776675 (+252924us) log.cc:844] Serialized 14 byte log entry
0709 12:21:55.796783 (+ 20108us) log.cc:844] Serialized 14 byte log entry
0709 12:21:55.798062 (+  1279us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:55.798141 (+    79us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:55.798176 (+    35us) raft_consensus.cc:1685] finished
0709 12:21:55.798187 (+    11us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:55.798343 (+   156us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":74418,"prepare.queue_time_us":254614,"prepare.run_cpu_time_us":86,"prepare.run_wall_time_us":90,"raft.queue_time_us":22642,"raft.run_cpu_time_us":78,"raft.run_wall_time_us":79,"spinlock_wait_cycles":19934848,"thread_start_us":1170350,"threads_started":2,"wal-append.queue_time_us":1072414}
I20260709 12:21:56.382782 20768 raft_consensus.cc:697] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 6 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:21:56.384521 20768 consensus_queue.cc:237] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:56.198459 17689 raft_consensus.cc:2468] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 5.
I20260709 12:21:56.369879 17755 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38786 (request call id 2520) took 1666 ms. Trace:
I20260709 12:21:56.398059 17755 rpcz_store.cc:276] 0709 12:21:54.703810 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:54.882062 (+178252us) service_pool.cc:224] Handling call
0709 12:21:56.354347 (+1472285us) spinlock_profiling.cc:243] Waited 14.4 ms on lock 0x7b5800033398. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b474fbb71 00007f7b474fba22 00007f7b504148d9 00007f7b4f56eee0 00007f7b4f6380ab 00007f7b4f638131 00007f7b4f5e60a0 00007f7b4f5e5984 00007f7b48b606d4 00007f7b48b60662
0709 12:21:56.369805 (+ 15458us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":33056640}
I20260709 12:21:56.351455 20693 raft_consensus.cc:515] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:21:56.411305 17728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:56.133117 20810 raft_consensus.cc:515] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:21:56.426712 18333 connection.cc:256] RPC call timeout handler was delayed by -1.3970548090001103s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21800000000000003s for a response
I20260709 12:21:56.426810 20808 raft_consensus.cc:493] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
W20260709 12:21:56.427246 20687 consensus_meta.cc:287] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 1.370s	user 0.006s	sys 0.016s
I20260709 12:21:56.427919 20687 raft_consensus.cc:2749] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651) with higher term. Message: Invalid argument: T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for earlier term 4. Current term is 5.
I20260709 12:21:56.428097 20808 raft_consensus.cc:515] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:56.430965 20808 leader_election.cc:290] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:21:56.433167 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 402610us
W20260709 12:21:56.433900 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 502160us
W20260709 12:21:56.436914 17654 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:56.440169 17801 leader_election.cc:304] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:21:56.414606 20762 raft_consensus.cc:515] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:56.423046 20512 raft_consensus.cc:493] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:56.446861 17727 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:56.448558 18367 raft_consensus.cc:1275] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
W20260709 12:21:56.449127 17730 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:56.450269 20693 leader_election.cc:290] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 8 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:56.451081 18361 raft_consensus.cc:1240] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 5 LEADER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 4. Current term is 5. Ops: []
I20260709 12:21:56.452121 20945 raft_consensus.cc:2804] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:21:56.452448 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:56.445677 20907 raft_consensus.cc:493] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:56.453303 17758 raft_consensus.cc:2468] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 7.
I20260709 12:21:56.453722 17763 raft_consensus.cc:1240] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 3. Current term is 4. Ops: []
I20260709 12:21:56.444619 20763 raft_consensus.cc:515] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:56.455061 20512 raft_consensus.cc:515] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:56.456943 20763 leader_election.cc:290] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 5 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:56.453498 20907 raft_consensus.cc:515] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:56.458058 17760 raft_consensus.cc:1240] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 8. Current term is 9. Ops: []
I20260709 12:21:56.452589 20945 raft_consensus.cc:493] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:21:56.420523 20776 consensus_queue.cc:1059] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 4.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.022s
W20260709 12:21:56.447706 17727 leader_election.cc:336] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837): Timed out: RequestConsensusVote RPC to 127.16.29.67:36837 timed out after 1.532s (SENT)
W20260709 12:21:56.474457 17728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:56.469949 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:56.478170 17834 raft_consensus.cc:1275] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260709 12:21:56.478919 20525 raft_consensus.cc:3055] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:21:56.479439 20525 raft_consensus.cc:740] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:21:56.480792 20525 consensus_queue.cc:260] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:56.485670 20525 raft_consensus.cc:3060] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:21:56.479854 20346 raft_consensus.cc:3055] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:21:56.487032 20346 raft_consensus.cc:740] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:21:56.488132 17802 leader_election.cc:304] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:56.489744 20828 raft_consensus.cc:2749] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:21:56.491493 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:56.462246 20945 raft_consensus.cc:3060] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:56.495427 17755 raft_consensus.cc:1240] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 4. Current term is 5. Ops: []
I20260709 12:21:56.497053 17757 raft_consensus.cc:2393] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 5: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
W20260709 12:21:56.500167 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 64313.5us
W20260709 12:21:56.502663 17803 leader_election.cc:336] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Timed out: RequestConsensusVote RPC to 127.16.29.68:42577 timed out after 2.088s (SENT)
I20260709 12:21:56.503214 17803 leader_election.cc:304] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:56.504196 20879 raft_consensus.cc:3055] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:21:56.504627 20879 raft_consensus.cc:740] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:21:56.505314 17682 raft_consensus.cc:2468] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 5.
I20260709 12:21:56.507673 20847 consensus_queue.cc:1059] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 12:21:56.510493 20851 leader_election.cc:290] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:56.511315 20346 consensus_queue.cc:260] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:56.492847 17727 leader_election.cc:304] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:21:56.504554 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:56.494550 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:56.516820 20512 leader_election.cc:290] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:56.517311 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:21:56.518075 17803 leader_election.cc:336] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Timed out: RequestConsensusVote RPC to 127.16.29.68:42577 timed out after 2.141s (SENT)
I20260709 12:21:56.518338 17760 raft_consensus.cc:2468] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 3.
I20260709 12:21:56.519315 17763 raft_consensus.cc:3060] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:56.519621 20851 raft_consensus.cc:2749] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:21:56.520375 20851 raft_consensus.cc:493] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:56.520545 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:56.521063 20851 raft_consensus.cc:515] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:56.521850 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:56.522826 17758 raft_consensus.cc:2468] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 8.
I20260709 12:21:56.523772 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:56.524113 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 8 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:56.524451 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 9 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:56.525120 17761 raft_consensus.cc:2410] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for term 8 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260709 12:21:56.525490 18365 raft_consensus.cc:2410] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 9 because replica has last-logged OpId of term: 7 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 7.
I20260709 12:21:56.525408 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:56.526297 18362 raft_consensus.cc:2468] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 4.
I20260709 12:21:56.527354 20851 leader_election.cc:290] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:56.528154 17758 raft_consensus.cc:1240] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 5. Current term is 6. Ops: [5.7-5.7]
I20260709 12:21:56.534582 20394 raft_consensus.cc:3055] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:21:56.534752 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:56.535213 20394 raft_consensus.cc:740] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:21:56.527626 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:56.536034 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:56.536329 17758 raft_consensus.cc:1240] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260709 12:21:56.537002 18363 raft_consensus.cc:2393] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 6: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:21:56.537871 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:56.537955 17757 raft_consensus.cc:1240] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 7. Current term is 8. Ops: [7.7-7.7]
I20260709 12:21:56.538034 20843 leader_election.cc:290] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:56.539441 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:56.540486 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:56.541489 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:56.542156 18362 raft_consensus.cc:2468] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 7.
I20260709 12:21:56.544168 20783 raft_consensus.cc:2749] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:21:56.545157 20762 leader_election.cc:290] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:21:56.539573 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 103440us
W20260709 12:21:56.543787 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 96207.5us
I20260709 12:21:56.548053 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:56.549049 18365 raft_consensus.cc:2393] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 7: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:21:56.514199 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:56.537035 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:56.557263 17681 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44898 (request call id 651) took 2916 ms. Trace:
I20260709 12:21:56.557623 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:56.557829 17681 rpcz_store.cc:276] 0709 12:21:53.638086 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:53.783072 (+144986us) service_pool.cc:224] Handling call
0709 12:21:53.783429 (+   357us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:55.777237 (+1993808us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:56.549940 (+772703us) spinlock_profiling.cc:243] Waited 1.92 s on lock 0x7b54004277d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:21:56.554089 (+  4149us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4417769728}
I20260709 12:21:56.558518 17760 raft_consensus.cc:3060] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:56.506032 20879 consensus_queue.cc:260] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:56.536487 20394 consensus_queue.cc:260] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:56.562914 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:56.563309 20394 raft_consensus.cc:3060] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:56.572316 18334 leader_election.cc:304] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:21:56.570988 17760 raft_consensus.cc:2468] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 7.
I20260709 12:21:56.563835 17831 raft_consensus.cc:2468] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 4.
I20260709 12:21:56.586741 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:56.578804 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:56.584120 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:56.590135 18368 raft_consensus.cc:2410] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 7 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260709 12:21:56.568094 20787 raft_consensus.cc:493] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:56.563350 20346 raft_consensus.cc:3060] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:56.600073 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:56.565770 20936 raft_consensus.cc:493] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:56.611583 17681 raft_consensus.cc:1275] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 7 index: 6. (term mismatch)
I20260709 12:21:56.601441 20810 leader_election.cc:290] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 9 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:56.614009 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:56.612185 17681 pending_rounds.cc:85] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3: Aborting all ops after (but not including) 5
I20260709 12:21:56.613116 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:56.619625 17763 raft_consensus.cc:2468] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 7.
I20260709 12:21:56.621064 17757 raft_consensus.cc:3060] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:21:56.621552 20895 log.cc:927] Time spent T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.905s	user 0.005s	sys 0.000s
W20260709 12:21:56.624164 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:56.626505 16501 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:21:56.620041 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:56.627920 18366 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59886 (request call id 935) took 2469 ms. Trace:
I20260709 12:21:55.839316 17762 rpcz_store.cc:276] 0709 12:21:54.431534 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:54.685704 (+254170us) service_pool.cc:224] Handling call
0709 12:21:54.717280 (+ 31576us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:54.717499 (+   219us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:21:55.524982 (+807483us) log.cc:844] Serialized 14 byte log entry
0709 12:21:55.787121 (+262139us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:55.787195 (+    74us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:21:55.787260 (+    65us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:55.787351 (+    91us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:55.787512 (+   161us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":25888,"prepare.queue_time_us":824990,"prepare.run_cpu_time_us":3065,"prepare.run_wall_time_us":33781,"raft.queue_time_us":32257,"raft.run_cpu_time_us":130,"raft.run_wall_time_us":23768,"thread_start_us":1042993,"threads_started":2,"wal-append.queue_time_us":301430}
I20260709 12:21:56.631177 17802 leader_election.cc:400] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 6 election: Vote denied by peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071) with higher term. Message: Invalid argument: T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 6. Current term is 7.
I20260709 12:21:56.631484 17802 leader_election.cc:403] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 6 election: Cancelling election due to peer responding with higher term
I20260709 12:21:56.632637 20867 raft_consensus.cc:2749] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:21:56.633687 20887 raft_consensus.cc:3060] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:21:56.635344 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 73845.6us
I20260709 12:21:56.633711 17764 raft_consensus.cc:1240] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 6. Current term is 7. Ops: []
I20260709 12:21:56.628386 18366 rpcz_store.cc:276] 0709 12:21:54.158407 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:54.158650 (+   243us) service_pool.cc:224] Handling call
0709 12:21:54.159029 (+   379us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 12:21:54.379084 (+220055us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:54.379129 (+    45us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 12:21:54.379977 (+   848us) log.cc:844] Serialized 78 byte log entry
0709 12:21:54.665276 (+285299us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:21:54.685293 (+ 20017us) log.cc:844] Serialized 14 byte log entry
0709 12:21:55.122576 (+437283us) spinlock_profiling.cc:243] Waited 22.2 ms on lock 0x7b18002e6cc0. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b474adc0a 00007f7b50bed219 00007f7b47535a40 00007f7b475304e4 00007f7b47530455 00007f7b4a618060 00007f7b4a62a716 00007f7b4a63b287 00007f7b4a63b208
0709 12:21:55.122926 (+   350us) log.cc:844] Serialized 14 byte log entry
0709 12:21:55.123082 (+   156us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:55.123154 (+    72us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:56.555698 (+1432544us) spinlock_profiling.cc:243] Waited 3.55 ms on lock 0x7b2400308408. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b473b99c4 00007f7b473b7acb 00007f7b47572dc9 00007f7b47572a55 00007f7b4757209d 00007f7b4a1a410e 00007f7b4a1bb88c 00007f7b4a1babb6 00007f7b4a1ba7d8 00007f7b4a1c4221
0709 12:21:56.627381 (+ 71683us) raft_consensus.cc:1685] finished
0709 12:21:56.627405 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:56.627638 (+   233us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":136480,"prepare.queue_time_us":20557,"prepare.run_cpu_time_us":79,"prepare.run_wall_time_us":80,"raft.queue_time_us":498807,"raft.run_cpu_time_us":1665,"raft.run_wall_time_us":75054,"spinlock_wait_cycles":59130880,"thread_start_us":391810,"threads_started":2,"wal-append.queue_time_us":1112352}
I20260709 12:21:56.642926 20892 raft_consensus.cc:3055] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:21:56.614719 20911 raft_consensus.cc:493] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:21:56.643417 20892 raft_consensus.cc:740] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:21:56.617308 17681 pending_rounds.cc:107] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3: Aborting uncommitted NO_OP operation due to leader change: 6.6
I20260709 12:21:56.647214 17681 raft_consensus.cc:2889] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:21:56.647763 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:56.647958 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:56.648634 17835 raft_consensus.cc:3060] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:21:56.642241 17756 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38786 (request call id 2525) took 1923 ms. Trace:
I20260709 12:21:56.643440 17758 raft_consensus.cc:3060] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Advancing to term 5
I20260709 12:21:56.640570 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:21:56.650231 17728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:56.650940 17728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:56.652802 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:56.655265 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:56.653545 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:56.648634 17686 raft_consensus.cc:3060] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:21:56.643486 20930 log.cc:927] Time spent T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.572s	user 0.000s	sys 0.004s
I20260709 12:21:56.664271 20936 raft_consensus.cc:515] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:56.602176 20787 raft_consensus.cc:515] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:56.657790 20911 raft_consensus.cc:515] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:56.692948 17682 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37396 (request call id 1970) took 1047 ms. Trace:
I20260709 12:21:56.693418 17682 rpcz_store.cc:276] 0709 12:21:55.645947 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:55.764188 (+118241us) service_pool.cc:224] Handling call
0709 12:21:56.692709 (+928521us) spinlock_profiling.cc:243] Waited 658 ms on lock 0x7b54003002d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:21:56.692863 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1512559872}
I20260709 12:21:56.693718 20787 leader_election.cc:290] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:56.694497 20911 leader_election.cc:290] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:56.696642 17829 raft_consensus.cc:2410] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for term 6 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20260709 12:21:56.697710 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:56.697827 17801 leader_election.cc:304] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:21:56.698473 17829 raft_consensus.cc:3060] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:56.700177 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:56.700778 20892 consensus_queue.cc:260] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:56.701916 20892 raft_consensus.cc:3060] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:56.702104 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:56.703382 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
W20260709 12:21:56.703977 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:56.704635 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:56.705391 17834 raft_consensus.cc:2393] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 8: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:21:56.706168 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:56.706794 17834 raft_consensus.cc:2393] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 8: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:21:56.706966 20847 raft_consensus.cc:493] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:56.707726 20847 raft_consensus.cc:515] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:56.669986 20913 consensus_queue.cc:1059] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
W20260709 12:21:56.671118 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:56.671430 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:56.649158 17756 rpcz_store.cc:276] 0709 12:21:54.718661 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:54.940509 (+221848us) service_pool.cc:224] Handling call
0709 12:21:54.940819 (+   310us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:55.634259 (+693440us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:55.634303 (+    44us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:55.634843 (+   540us) log.cc:844] Serialized 26 byte log entry
0709 12:21:55.887852 (+253009us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:21:55.887929 (+    77us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:56.641456 (+753527us) spinlock_profiling.cc:243] Waited 602 ms on lock 0x7b54002977d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:21:56.641564 (+   108us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:56.641692 (+   128us) raft_consensus.cc:1685] finished
0709 12:21:56.641707 (+    15us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:56.642003 (+   296us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":91225,"raft.queue_time_us":108472,"raft.run_cpu_time_us":170,"raft.run_wall_time_us":38620,"spinlock_wait_cycles":1383838464,"thread_start_us":252629,"threads_started":1,"wal-append.queue_time_us":359279}
W20260709 12:21:56.675195 17728 leader_election.cc:336] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Timed out: RequestConsensusVote RPC to 127.16.29.65:33071 timed out after 2.146s (SENT)
I20260709 12:21:56.676323 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:56.671453 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:56.677479 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:56.677816 20936 leader_election.cc:290] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:56.678246 20693 consensus_queue.cc:1048] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:21:56.716338 20787 consensus_queue.cc:1048] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:21:56.677517 20368 consensus_queue.cc:1048] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.004s
I20260709 12:21:56.682265 20697 raft_consensus.cc:3060] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:56.719717 20913 raft_consensus.cc:3055] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:21:56.720117 20913 raft_consensus.cc:740] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:21:56.683277 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:56.721061 20913 consensus_queue.cc:260] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:56.721668 17688 raft_consensus.cc:3060] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:21:56.722816 17803 leader_election.cc:336] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Timed out: RequestConsensusVote RPC to 127.16.29.68:42577 timed out after 2.212s (SENT)
I20260709 12:21:56.723389 17803 leader_election.cc:304] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
W20260709 12:21:56.724323 20894 log.cc:927] Time spent T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 1.227s	user 0.003s	sys 0.002s
I20260709 12:21:56.724655 20923 raft_consensus.cc:2749] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:21:56.725510 20923 raft_consensus.cc:493] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:21:56.726313 18364 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 1060) took 2360 ms. Trace:
I20260709 12:21:56.726128 20923 raft_consensus.cc:515] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:56.726667 18364 rpcz_store.cc:276] 0709 12:21:54.365983 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:54.368610 (+  2627us) service_pool.cc:224] Handling call
0709 12:21:54.368887 (+   277us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:21:54.383671 (+ 14784us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:54.383707 (+    36us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:21:54.384996 (+  1289us) log.cc:844] Serialized 52 byte log entry
0709 12:21:54.527744 (+142748us) spinlock_profiling.cc:243] Waited 5.23 ms on lock 0x7b4800220a58. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4f549aa1 00007f7b4f540b37 00007f7b4f5409ca 00007f7b4f63826f 00007f7b4f637f37 00007f7b4f64c55c 00007f7b4f64c508 00007f7b4f64c488 00007f7b4f64c443 00007f7b4f64b6ec 00007f7b4e575a96
0709 12:21:54.720206 (+192462us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:21:54.720255 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:54.720321 (+    66us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:56.514646 (+1794325us) spinlock_profiling.cc:243] Waited 5.09 ms on lock 0x7b4400022b4c. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b47459c85 00007f7b4745a6a2 00007f7b4a1cc1db 00007f7b4a1c7aaf 00007f7b4a1bb86e 00007f7b4a1babb6 00007f7b4a1ba7d8 00007f7b4a1c4221 00007f7b4a1c41d9 00007f7b4a1c4169
0709 12:21:56.725899 (+211253us) raft_consensus.cc:1685] finished
0709 12:21:56.725925 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:56.726148 (+   223us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":130000,"raft.queue_time_us":20074,"raft.run_cpu_time_us":1514,"raft.run_wall_time_us":128663,"spinlock_wait_cycles":23730304,"thread_start_us":302450,"threads_started":1,"wal-append.queue_time_us":971208}
I20260709 12:21:56.728371 20923 leader_election.cc:290] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:56.729009 20923 raft_consensus.cc:493] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:56.729594 20923 raft_consensus.cc:515] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:56.730295 17757 raft_consensus.cc:2468] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 7.
I20260709 12:21:56.683459 18334 leader_election.cc:304] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:56.684613 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:56.685606 20864 raft_consensus.cc:3060] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:56.685708 18361 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 1072) took 1019 ms. Trace:
I20260709 12:21:56.689428 20907 leader_election.cc:290] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:56.690832 20872 leader_election.cc:290] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 7 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:56.738603 17801 leader_election.cc:304] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:21:56.739038 17684 raft_consensus.cc:2393] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 5: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:21:56.708056 20862 consensus_queue.cc:1059] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:21:56.668526 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:56.679560 17727 leader_election.cc:304] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:21:56.715970 17838 raft_consensus.cc:2468] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 7.
I20260709 12:21:56.735646 18361 rpcz_store.cc:276] 0709 12:21:55.666668 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:55.667765 (+  1097us) service_pool.cc:224] Handling call
0709 12:21:55.668124 (+   359us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:56.451584 (+783460us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:56.685453 (+233869us) spinlock_profiling.cc:243] Waited 740 ms on lock 0x7b5400499654. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:21:56.685651 (+   198us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1703019776}
I20260709 12:21:56.707618 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:56.740892 17834 raft_consensus.cc:2468] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 7.
I20260709 12:21:56.744834 17835 raft_consensus.cc:2468] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 10.
I20260709 12:21:56.722388 20913 raft_consensus.cc:3060] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:56.749241 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:56.752350 20862 consensus_queue.cc:1243] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer e2909cff1da44ad387f96a8ae3aa5afb log is divergent from this leader: its last log entry 7.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20260709 12:21:56.752936 20862 raft_consensus.cc:3055] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:21:56.753346 20862 raft_consensus.cc:740] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 8 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:21:56.754614 20862 consensus_queue.cc:260] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 8.4, Last appended by leader: 4, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:56.756175 20862 raft_consensus.cc:3060] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:21:56.756647 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:56.757858 20903 raft_consensus.cc:2804] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:21:56.761535 17833 raft_consensus.cc:2393] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 5: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:21:56.762192 17758 raft_consensus.cc:2468] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 5.
I20260709 12:21:56.739630 20851 raft_consensus.cc:2804] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:21:56.784150 17801 leader_election.cc:304] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:21:56.770673 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
W20260709 12:21:56.795418 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 120367us
I20260709 12:21:56.769979 20810 raft_consensus.cc:493] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:56.802635 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:56.799450 20903 raft_consensus.cc:697] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 7 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:21:56.811317 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:56.812521 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:56.810705 18368 raft_consensus.cc:2468] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 8.
I20260709 12:21:56.786417 20936 raft_consensus.cc:493] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:21:56.798077 20867 raft_consensus.cc:2804] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:21:56.817396 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:56.836083 17758 raft_consensus.cc:2468] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 6.
I20260709 12:21:56.803984 20943 raft_consensus.cc:2804] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:21:56.839092 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:56.838894 20904 consensus_queue.cc:1048] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.007s
I20260709 12:21:56.848263 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:56.839833 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:21:56.841091 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 161664us
I20260709 12:21:56.841531 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:56.842226 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:56.842885 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:21:56.849217 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 183820us
I20260709 12:21:56.843570 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:56.844228 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:56.844316 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:56.846805 17838 raft_consensus.cc:2468] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 7.
I20260709 12:21:56.827852 17689 raft_consensus.cc:2468] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 6.
I20260709 12:21:56.797628 20368 raft_consensus.cc:2749] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:21:56.850168 18365 raft_consensus.cc:2468] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 4.
I20260709 12:21:56.847779 18368 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 7.
I20260709 12:21:56.847443 20945 raft_consensus.cc:515] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:56.863973 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:56.868469 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:56.869715 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:56.870913 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:56.871493 18367 raft_consensus.cc:2468] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 7.
I20260709 12:21:56.872144 18334 leader_election.cc:304] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:56.872759 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:56.873092 18366 raft_consensus.cc:2393] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 5: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:21:56.871037 20903 consensus_queue.cc:237] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:56.873940 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:56.872403 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:56.873945 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:56.877691 18363 raft_consensus.cc:2393] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 6: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:21:56.872419 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:56.879069 18367 raft_consensus.cc:2468] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 5.
I20260709 12:21:56.815212 20810 raft_consensus.cc:515] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:56.886441 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:21:56.888545 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:56.893394 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:56.893640 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:56.894281 17764 raft_consensus.cc:2468] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 6.
I20260709 12:21:56.894443 17683 raft_consensus.cc:1275] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
W20260709 12:21:56.895251 17801 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:56.904512 18364 raft_consensus.cc:2410] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 8 because replica has last-logged OpId of term: 7 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 6.
I20260709 12:21:56.905911 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:56.906457 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:56.906996 18362 raft_consensus.cc:2393] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 6: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:21:56.908362 20397 consensus_queue.cc:1048] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.141s
I20260709 12:21:56.913538 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:56.915382 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:56.927465 20923 leader_election.cc:290] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:56.928315 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:56.954633 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:56.828795 17652 leader_election.cc:304] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:21:56.928315 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:56.819533 17834 raft_consensus.cc:2468] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 8.
I20260709 12:21:56.935521 18334 leader_election.cc:304] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:21:56.837647 20867 raft_consensus.cc:493] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:21:56.965131 17829 raft_consensus.cc:2468] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 6.
W20260709 12:21:56.923071 20964 log.cc:927] Time spent T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.124s	user 0.004s	sys 0.000s
I20260709 12:21:56.928684 20846 raft_consensus.cc:2804] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:21:56.951078 17686 raft_consensus.cc:2468] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 8.
I20260709 12:21:56.930680 17836 raft_consensus.cc:2468] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 7.
I20260709 12:21:56.953383 17835 raft_consensus.cc:2410] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for term 7 because replica has last-logged OpId of term: 6 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260709 12:21:56.953692 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:56.954332 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:56.919498 20943 raft_consensus.cc:697] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 5 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:21:56.967449 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:56.959705 17688 raft_consensus.cc:2468] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 5.
I20260709 12:21:56.837689 20936 raft_consensus.cc:515] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:56.822669 20851 raft_consensus.cc:697] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 7 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:21:56.955153 17683 raft_consensus.cc:2468] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 6.
W20260709 12:21:57.035224 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:57.038940 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:57.025250 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:56.978893 17727 leader_election.cc:304] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:21:56.992231 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:21:57.036376 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 234042us
I20260709 12:21:57.046335 17685 raft_consensus.cc:2468] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 5.
W20260709 12:21:56.992405 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 68320.9us
I20260709 12:21:56.971277 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:57.024350 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 9 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:57.048403 17760 raft_consensus.cc:2393] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 9: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
W20260709 12:21:56.993471 17803 leader_election.cc:336] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Timed out: RequestConsensusVote RPC to 127.16.29.68:42577 timed out after 1.879s (SENT)
I20260709 12:21:57.053303 17761 raft_consensus.cc:1240] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 6. Current term is 7. Ops: []
I20260709 12:21:57.056661 20783 raft_consensus.cc:2804] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:21:57.060995 20787 raft_consensus.cc:2804] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:21:57.061102 20783 raft_consensus.cc:493] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:21:57.061661 20787 raft_consensus.cc:493] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:57.061874 20783 raft_consensus.cc:3060] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:57.062134 20787 raft_consensus.cc:3060] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:21:57.065894 20936 leader_election.cc:290] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:57.025844 20945 leader_election.cc:290] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 6 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:57.049031 20846 raft_consensus.cc:493] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:21:57.006513 20776 raft_consensus.cc:2804] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:21:56.939126 20768 raft_consensus.cc:493] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:21:56.964649 20904 consensus_queue.cc:1048] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.092s
I20260709 12:21:56.971028 20867 raft_consensus.cc:3060] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:21:57.037869 20397 raft_consensus.cc:493] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:21:57.074971 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 9 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:56.953385 20346 consensus_queue.cc:1059] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.115s
I20260709 12:21:57.077718 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:57.084499 17834 raft_consensus.cc:2468] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 6.
I20260709 12:21:57.076172 20846 raft_consensus.cc:3060] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:21:57.117861 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 188303us
I20260709 12:21:57.082856 20687 consensus_queue.cc:1059] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.036s
I20260709 12:21:57.134552 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 9 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:57.090888 20768 raft_consensus.cc:515] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:57.124370 20851 consensus_queue.cc:237] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:57.124228 20397 raft_consensus.cc:515] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:57.144511 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:57.149142 17727 leader_election.cc:304] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
W20260709 12:21:57.152473 17652 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:57.155032 17652 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:57.155772 17652 leader_election.cc:304] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
W20260709 12:21:57.157100 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 185392us
W20260709 12:21:57.158114 20887 consensus_meta.cc:287] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.524s	user 0.011s	sys 0.000s
I20260709 12:21:57.158883 20887 raft_consensus.cc:2749] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Leader election lost for term 6. Reason: Vote denied by peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071) with higher term. Message: Invalid argument: T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 6. Current term is 7.
W20260709 12:21:57.159545 17803 leader_election.cc:336] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Timed out: RequestConsensusVote RPC to 127.16.29.68:42577 timed out after 1.573s (SENT)
I20260709 12:21:57.160291 17803 leader_election.cc:304] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:21:57.161175 17728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:57.161864 17728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:57.158963 20879 raft_consensus.cc:3060] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:57.167691 17834 raft_consensus.cc:2410] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for term 9 because replica has last-logged OpId of term: 8 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 5.
I20260709 12:21:57.165494 17728 leader_election.cc:304] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:21:57.174846 17652 leader_election.cc:304] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
W20260709 12:21:57.176528 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 199819us
I20260709 12:21:57.179555 17728 leader_election.cc:304] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:21:57.166785 20904 raft_consensus.cc:2749] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:21:57.177593 20693 raft_consensus.cc:2804] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:21:57.190353 17652 leader_election.cc:304] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:21:57.189759 17728 leader_election.cc:304] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:21:57.178431 20887 raft_consensus.cc:2749] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260709 12:21:57.193367 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 133137us
I20260709 12:21:57.204952 18333 leader_election.cc:304] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:21:57.243952 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:57.246407 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:57.187400 20687 raft_consensus.cc:2804] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:21:57.189481 20693 raft_consensus.cc:493] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:57.194334 20776 raft_consensus.cc:697] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 5 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:21:57.144170 20943 consensus_queue.cc:237] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:57.243064 20911 raft_consensus.cc:2804] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260709 12:21:57.256771 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:57.257275 17727 leader_election.cc:304] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:21:57.254869 20693 raft_consensus.cc:3060] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:57.265690 20911 raft_consensus.cc:493] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:57.296121 20911 raft_consensus.cc:3060] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:57.297302 20847 leader_election.cc:290] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:57.298194 20346 raft_consensus.cc:2749] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:21:57.285598 17838 raft_consensus.cc:1240] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 7 LEADER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 6. Current term is 7. Ops: []
I20260709 12:21:57.287236 20227 raft_consensus.cc:2804] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:21:57.300879 20227 raft_consensus.cc:493] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:21:57.301331 20227 raft_consensus.cc:3060] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:21:57.302166 17832 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35042 (request call id 1823) took 2559 ms. Trace:
I20260709 12:21:57.302565 17832 rpcz_store.cc:276] 0709 12:21:54.742992 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:54.743193 (+   201us) service_pool.cc:224] Handling call
0709 12:21:56.239251 (+1496058us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:56.324597 (+ 85346us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:56.888276 (+563679us) spinlock_profiling.cc:243] Waited 84.2 ms on lock 0x7b54002eb654. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:21:57.301914 (+413638us) spinlock_profiling.cc:243] Waited 1.5 s on lock 0x7b54002eb650. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a61f8ad 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:21:57.302087 (+   173us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3633128064}
I20260709 12:21:57.275954 20377 raft_consensus.cc:2804] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:21:57.304080 20377 raft_consensus.cc:493] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:57.304482 20377 raft_consensus.cc:3060] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:57.304917 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:21:57.307180 17655 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:57.272161 20904 raft_consensus.cc:2749] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260709 12:21:57.308009 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 148510us
I20260709 12:21:57.294461 20702 raft_consensus.cc:2804] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:21:57.295418 20977 raft_consensus.cc:3055] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:21:57.308933 20977 raft_consensus.cc:740] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:21:57.309984 20975 raft_consensus.cc:493] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:21:57.287642 20916 raft_consensus.cc:493] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:21:57.310055 20977 consensus_queue.cc:260] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 7, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:57.310655 20975 raft_consensus.cc:515] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:57.311062 20916 raft_consensus.cc:515] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:57.308964 20702 raft_consensus.cc:697] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 8 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:21:57.313043 20975 leader_election.cc:290] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:57.311653 20977 raft_consensus.cc:3060] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:21:57.313464 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:21:57.315973 20970 log.cc:927] Time spent T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.150s	user 0.000s	sys 0.001s
I20260709 12:21:57.317613 20397 leader_election.cc:290] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:57.318261 20397 raft_consensus.cc:493] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:57.294525 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 8 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:57.318954 17830 raft_consensus.cc:2468] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 7.
I20260709 12:21:57.312636 20702 consensus_queue.cc:237] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:57.318768 20397 raft_consensus.cc:515] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:57.322721 20397 leader_election.cc:290] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:57.325170 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:57.325866 20787 raft_consensus.cc:515] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:21:57.326879 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 104350us
I20260709 12:21:57.327159 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:21:57.330858 20913 consensus_meta.cc:287] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.569s	user 0.006s	sys 0.001s
I20260709 12:21:57.337466 20776 consensus_queue.cc:237] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:21:57.344422 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 95747.5us
I20260709 12:21:57.346117 17801 leader_election.cc:304] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:57.339632 20810 leader_election.cc:290] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:57.352351 20941 raft_consensus.cc:2749] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:21:57.352713 17801 leader_election.cc:304] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:21:57.354344 20971 log.cc:927] Time spent T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.070s	user 0.000s	sys 0.000s
I20260709 12:21:57.343113 17652 leader_election.cc:304] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:57.360669 20847 raft_consensus.cc:2804] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260709 12:21:57.365319 17728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:57.378994 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:21:57.364830 20862 consensus_meta.cc:287] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.596s	user 0.012s	sys 0.000s
I20260709 12:21:57.389533 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:57.390266 20916 leader_election.cc:290] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:57.392458 17832 raft_consensus.cc:1240] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 8 LEADER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 7. Current term is 8. Ops: [7.7-7.7]
I20260709 12:21:57.393745 17727 leader_election.cc:304] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:21:57.357583 20783 raft_consensus.cc:515] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:57.358919 20693 raft_consensus.cc:515] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:57.401984 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:57.404501 17729 leader_election.cc:304] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:21:57.381783 20913 raft_consensus.cc:2749] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:21:57.363965 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:57.475907 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:57.476483 17761 raft_consensus.cc:1275] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260709 12:21:57.476698 17764 raft_consensus.cc:2468] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 8.
I20260709 12:21:57.477320 20783 leader_election.cc:290] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 4 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:57.417471 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:57.477870 17761 raft_consensus.cc:1275] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
W20260709 12:21:57.478356 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 117359us
I20260709 12:21:57.391099 20974 consensus_queue.cc:1048] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.075s
I20260709 12:21:57.479167 17727 leader_election.cc:304] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:21:57.400807 20768 leader_election.cc:290] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:57.407313 20377 raft_consensus.cc:515] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:57.473332 20693 leader_election.cc:290] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 7 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:57.478473 17763 raft_consensus.cc:2393] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 4: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:21:57.480118 20904 raft_consensus.cc:2749] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:21:57.480576 17727 leader_election.cc:304] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:21:57.480682 20810 raft_consensus.cc:2764] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Leader pre-election decision vote started in defunct term 6: won
I20260709 12:21:57.490614 17652 leader_election.cc:304] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:21:57.465013 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 260271us
I20260709 12:21:57.482261 17829 raft_consensus.cc:2410] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for term 9 because replica has last-logged OpId of term: 8 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260709 12:21:57.487131 17801 leader_election.cc:304] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:21:57.492674 17684 raft_consensus.cc:2393] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 7: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:21:57.466907 20847 raft_consensus.cc:493] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:57.499230 20810 raft_consensus.cc:2804] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:21:57.499444 17652 leader_election.cc:304] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:57.501621 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:57.505111 17652 leader_election.cc:304] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:57.499811 20904 raft_consensus.cc:2804] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:21:57.500257 20921 consensus_queue.cc:1048] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.005s
I20260709 12:21:57.500872 17764 raft_consensus.cc:1275] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
W20260709 12:21:57.504907 20864 consensus_meta.cc:287] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.759s	user 0.006s	sys 0.006s
I20260709 12:21:57.499244 20394 raft_consensus.cc:2804] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:21:57.496001 20768 raft_consensus.cc:2804] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:21:57.523015 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:57.525578 17688 raft_consensus.cc:2393] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 7: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:21:57.508266 20904 raft_consensus.cc:493] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:21:57.517553 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:57.527153 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:57.507947 20847 raft_consensus.cc:3060] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:57.529273 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:57.492936 20862 consensus_queue.cc:1048] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.024s
I20260709 12:21:57.511116 20913 raft_consensus.cc:2749] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:21:57.529942 17685 raft_consensus.cc:2468] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 4.
I20260709 12:21:57.529503 18333 leader_election.cc:304] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:21:57.533450 20525 raft_consensus.cc:493] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:21:57.541078 20973 consensus_queue.cc:1048] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.012s
I20260709 12:21:57.541457 20525 raft_consensus.cc:515] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:57.543569 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:57.533133 17759 raft_consensus.cc:2468] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 7.
I20260709 12:21:57.549793 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:57.550571 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:57.550663 17682 raft_consensus.cc:2468] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 5.
I20260709 12:21:57.551290 17764 raft_consensus.cc:3060] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:21:57.553598 20982 consensus_queue.cc:1048] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:21:57.556841 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:57.556454 20525 leader_election.cc:290] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:57.560185 20980 raft_consensus.cc:2804] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:21:57.560811 20980 raft_consensus.cc:493] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:57.561053 17728 leader_election.cc:304] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:21:57.562368 20864 raft_consensus.cc:2804] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:21:57.562877 20864 raft_consensus.cc:493] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:57.563333 20864 raft_consensus.cc:3060] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:21:57.570039 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 89584.1us
I20260709 12:21:57.571059 17727 leader_election.cc:304] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:21:57.572038 20693 raft_consensus.cc:2749] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:21:57.573571 17727 leader_election.cc:304] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:21:57.577657 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:57.524572 20787 leader_election.cc:290] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 8 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:57.518749 20810 raft_consensus.cc:697] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:21:57.580302 20810 consensus_queue.cc:237] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:57.561304 20980 raft_consensus.cc:3060] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:57.585335 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:57.509140 20867 raft_consensus.cc:515] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:57.588815 20377 leader_election.cc:290] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 7 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:57.560527 18334 leader_election.cc:304] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:21:57.587556 17680 raft_consensus.cc:1275] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260709 12:21:57.527578 20768 raft_consensus.cc:493] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:21:57.611982 20693 raft_consensus.cc:2749] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:21:57.590150 20867 leader_election.cc:290] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 11 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:57.613570 20227 raft_consensus.cc:515] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:21:57.615446 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 55030.5us
W20260709 12:21:57.618984 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 214715us
I20260709 12:21:57.609913 18333 leader_election.cc:304] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:21:57.623088 17688 raft_consensus.cc:1240] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 7. Current term is 8. Ops: []
W20260709 12:21:57.622496 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:57.527719 20904 raft_consensus.cc:3060] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:57.607924 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:57.625396 17760 raft_consensus.cc:2393] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 7: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:21:57.544965 20394 raft_consensus.cc:697] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 7 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:21:57.529933 20916 raft_consensus.cc:2749] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260709 12:21:57.641392 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 48069.9us
I20260709 12:21:57.622442 20768 raft_consensus.cc:3060] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:21:57.651759 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 11 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
W20260709 12:21:57.668108 17728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:57.676888 17830 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35040 (request call id 2095) took 1011 ms. Trace:
I20260709 12:21:57.627805 20867 raft_consensus.cc:2804] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:21:57.677467 17830 rpcz_store.cc:276] 0709 12:21:56.665586 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:56.665711 (+   125us) service_pool.cc:224] Handling call
0709 12:21:57.676654 (+1010943us) spinlock_profiling.cc:243] Waited 459 ms on lock 0x7b54003354d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:21:57.676809 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1056079232}
W20260709 12:21:57.684239 17728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:57.687628 18333 leader_election.cc:304] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:21:57.625849 20687 raft_consensus.cc:697] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 10 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
W20260709 12:21:57.692308 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:57.695760 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:57.622236 20846 raft_consensus.cc:515] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:57.708598 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:57.715590 20978 raft_consensus.cc:2749] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:21:57.716174 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:57.717244 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 11 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:57.717744 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:57.718544 17830 raft_consensus.cc:3060] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:21:57.702956 17729 leader_election.cc:304] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
W20260709 12:21:57.720089 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 146598us
I20260709 12:21:57.722008 17727 leader_election.cc:304] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:21:57.630899 20702 consensus_queue.cc:1048] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.067s
I20260709 12:21:57.723250 20394 consensus_queue.cc:237] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:57.724996 20702 raft_consensus.cc:2749] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:21:57.726660 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:21:57.734047 20961 log.cc:927] Time spent T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.910s	user 0.004s	sys 0.010s
I20260709 12:21:57.737514 20693 raft_consensus.cc:2749] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260709 12:21:57.739840 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:57.741075 17835 raft_consensus.cc:1240] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 5. Current term is 7. Ops: []
I20260709 12:21:57.741767 17835 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35042 (request call id 1849) took 1014 ms. Trace:
I20260709 12:21:57.742079 17835 rpcz_store.cc:276] 0709 12:21:56.727473 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:57.740524 (+1013051us) service_pool.cc:224] Handling call
0709 12:21:57.740865 (+   341us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:57.741523 (+   658us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:57.741713 (+   190us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:57.744179 20972 raft_consensus.cc:493] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:21:57.744801 20972 raft_consensus.cc:515] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:57.747145 20972 leader_election.cc:290] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:57.749845 17834 raft_consensus.cc:1240] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 6 LEADER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 5. Current term is 6. Ops: [5.7-5.7]
I20260709 12:21:57.750651 17834 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37994 (request call id 731) took 1077 ms. Trace:
I20260709 12:21:57.750995 17834 rpcz_store.cc:276] 0709 12:21:56.673722 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:57.495203 (+821481us) service_pool.cc:224] Handling call
0709 12:21:57.509834 (+ 14631us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:57.749575 (+239741us) spinlock_profiling.cc:243] Waited 41.8 ms on lock 0x7b54003b1da0. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a597dc0 00007f7b4a6221c0 00007f7b4a623231 00007f7b4a61fd8f 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c
0709 12:21:57.750362 (+   787us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:57.750589 (+   227us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":96116992}
I20260709 12:21:57.759802 17831 raft_consensus.cc:1240] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 4. Current term is 5. Ops: []
I20260709 12:21:57.760629 17831 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37994 (request call id 728) took 1088 ms. Trace:
I20260709 12:21:57.760947 17831 rpcz_store.cc:276] 0709 12:21:56.672533 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:57.320184 (+647651us) service_pool.cc:224] Handling call
0709 12:21:57.759317 (+439133us) spinlock_profiling.cc:243] Waited 5.38 ms on lock 0x7b580013fe98. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b474fbb71 00007f7b474fba22 00007f7b504148d9 00007f7b4f56eee0 00007f7b4f6380ab 00007f7b4f638131 00007f7b4f5f9fc0 00007f7b4f5e4929 00007f7b4f5e43d2 00007f7b48b5bbd4
0709 12:21:57.759580 (+   263us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:57.760352 (+   772us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:57.760569 (+   217us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":12365056}
I20260709 12:21:57.766165 17829 raft_consensus.cc:1240] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 5. Current term is 6. Ops: []
I20260709 12:21:57.766844 17829 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35040 (request call id 2099) took 1030 ms. Trace:
I20260709 12:21:57.767243 17829 rpcz_store.cc:276] 0709 12:21:56.736347 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:57.765595 (+1029248us) service_pool.cc:224] Handling call
0709 12:21:57.765954 (+   359us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:57.766604 (+   650us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:57.766784 (+   180us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:57.768422 17689 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37402 (request call id 2353) took 1093 ms. Trace:
I20260709 12:21:57.768877 17689 rpcz_store.cc:276] 0709 12:21:56.675197 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:56.677185 (+  1988us) service_pool.cc:224] Handling call
0709 12:21:57.768239 (+1091054us) spinlock_profiling.cc:243] Waited 105 ms on lock 0x7b54000c4754. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:21:57.768343 (+   104us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":241929600}
I20260709 12:21:57.770699 17836 raft_consensus.cc:1240] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 5. Current term is 6. Ops: []
I20260709 12:21:57.771471 17836 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37994 (request call id 727) took 1099 ms. Trace:
I20260709 12:21:57.771798 17836 rpcz_store.cc:276] 0709 12:21:56.672147 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:57.315490 (+643343us) service_pool.cc:224] Handling call
0709 12:21:57.770252 (+454762us) spinlock_profiling.cc:243] Waited 10 ms on lock 0x7b580013fe98. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b474fbb71 00007f7b474fba22 00007f7b504148d9 00007f7b4f56eee0 00007f7b4f6380ab 00007f7b4f638131 00007f7b4f5f9fc0 00007f7b4f5e4929 00007f7b4f5e43d2 00007f7b48b5bbd4
0709 12:21:57.770478 (+   226us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:57.771217 (+   739us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:57.771414 (+   197us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":23043456}
I20260709 12:21:57.780944 20843 raft_consensus.cc:3055] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:21:57.781453 20843 raft_consensus.cc:740] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:21:57.782336 20843 consensus_queue.cc:260] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:57.784533 17830 raft_consensus.cc:2468] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 8.
I20260709 12:21:57.794122 17830 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35040 (request call id 2101) took 1031 ms. Trace:
I20260709 12:21:57.794368 17838 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37994 (request call id 726) took 1122 ms. Trace:
I20260709 12:21:57.794529 17830 rpcz_store.cc:276] 0709 12:21:56.763225 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:57.785277 (+1022052us) service_pool.cc:224] Handling call
0709 12:21:57.793426 (+  8149us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:57.793730 (+   304us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:57.793757 (+    27us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:57.793833 (+    76us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:21:57.793868 (+    35us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:57.793931 (+    63us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:57.794038 (+   107us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:57.794741 17838 rpcz_store.cc:276] 0709 12:21:56.671738 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:57.153230 (+481492us) service_pool.cc:224] Handling call
0709 12:21:57.160308 (+  7078us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:57.300196 (+139888us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:57.794130 (+493934us) spinlock_profiling.cc:243] Waited 124 ms on lock 0x7b540022b854. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:21:57.794290 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":285771136}
W20260709 12:21:57.808653 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 113354us
I20260709 12:21:57.679823 20977 raft_consensus.cc:493] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:21:57.640368 20847 raft_consensus.cc:515] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:57.687489 20867 raft_consensus.cc:493] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
W20260709 12:21:57.828274 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 154525us
I20260709 12:21:57.833719 17830 raft_consensus.cc:1275] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 5 index: 8. (index mismatch)
I20260709 12:21:57.846275 17830 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35040 (request call id 2103) took 1056 ms. Trace:
I20260709 12:21:57.846748 17830 rpcz_store.cc:276] 0709 12:21:56.790249 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:57.795686 (+1005437us) service_pool.cc:224] Handling call
0709 12:21:57.805663 (+  9977us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:57.845780 (+ 40117us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:57.846042 (+   262us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:21:57.847831 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:57.743723 17835 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35040 (request call id 2098) took 1007 ms. Trace:
I20260709 12:21:57.849042 17835 rpcz_store.cc:276] 0709 12:21:56.736051 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:57.742841 (+1006790us) service_pool.cc:224] Handling call
0709 12:21:57.743157 (+   316us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:57.743362 (+   205us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:57.743388 (+    26us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:57.743427 (+    39us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:21:57.743461 (+    34us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:57.743517 (+    56us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:57.743635 (+   118us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:57.850605 18333 leader_election.cc:304] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
W20260709 12:21:57.853811 17728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:57.702553 20776 raft_consensus.cc:2749] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:21:57.858472 17802 leader_election.cc:304] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:21:57.862497 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:21:57.817214 20892 consensus_meta.cc:287] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 1.108s	user 0.011s	sys 0.002s
I20260709 12:21:57.885418 17689 raft_consensus.cc:1240] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 4. Current term is 5. Ops: []
W20260709 12:21:57.855919 17803 leader_election.cc:336] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Timed out: RequestConsensusVote RPC to 127.16.29.68:42577 timed out after 1.555s (SENT)
I20260709 12:21:57.873009 18365 raft_consensus.cc:1240] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 6. Current term is 7. Ops: []
I20260709 12:21:57.783671 20843 raft_consensus.cc:3060] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:57.889834 18371 raft_consensus.cc:2468] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 7.
I20260709 12:21:57.853935 20867 raft_consensus.cc:3060] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:57.882915 17832 raft_consensus.cc:2393] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 7: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:21:57.889650 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:57.874883 20227 leader_election.cc:290] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 8 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:57.927412 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:57.933372 17682 raft_consensus.cc:1275] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260709 12:21:57.928203 18371 raft_consensus.cc:2468] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 7.
I20260709 12:21:57.867038 20977 raft_consensus.cc:515] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:57.822476 17652 leader_election.cc:304] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:57.896420 17829 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35040 (request call id 2104) took 1073 ms. Trace:
I20260709 12:21:57.908623 20828 raft_consensus.cc:2804] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260709 12:21:57.928673 20911 consensus_meta.cc:287] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.632s	user 0.008s	sys 0.005s
W20260709 12:21:57.957597 17759 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36572 (request call id 628) took 1438 ms (client timeout 1797 ms). Trace:
I20260709 12:21:57.968411 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:21:57.964811 17759 rpcz_store.cc:269] 0709 12:21:56.519023 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:56.520042 (+  1019us) service_pool.cc:224] Handling call
0709 12:21:57.957419 (+1437377us) spinlock_profiling.cc:243] Waited 1.01 s on lock 0x7b54003eb554. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:21:57.957528 (+   109us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2326594816}
W20260709 12:21:57.970790 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 137822us
W20260709 12:21:57.973769 17655 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:57.975745 17685 raft_consensus.cc:1275] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260709 12:21:57.978201 18368 raft_consensus.cc:1813] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Handling vote request from an unknown peer e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:57.979122 18368 raft_consensus.cc:2410] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for term 6 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20260709 12:21:57.942052 20892 raft_consensus.cc:493] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:57.982648 18364 raft_consensus.cc:2468] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 7.
W20260709 12:21:57.999908 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 177521us
I20260709 12:21:58.001075 17652 leader_election.cc:304] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:21:58.008111 20984 raft_consensus.cc:2749] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 12:21:58.059687 20787 raft_consensus.cc:2804] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:21:58.060889 20787 raft_consensus.cc:697] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 8 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:21:58.062024 20787 consensus_queue.cc:237] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:21:58.066200 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 65262.7us
I20260709 12:21:58.016808 20846 leader_election.cc:290] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 7 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:57.919600 20687 consensus_queue.cc:237] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:58.068776 20525 consensus_queue.cc:1048] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20260709 12:21:58.036123 20989 log.cc:927] Time spent T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.197s	user 0.001s	sys 0.000s
I20260709 12:21:57.992753 20851 raft_consensus.cc:493] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:21:58.071823 20525 raft_consensus.cc:493] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:58.072247 20851 raft_consensus.cc:515] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:58.072623 20525 raft_consensus.cc:515] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:58.074548 20851 leader_election.cc:290] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:57.926960 20974 consensus_queue.cc:1059] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.055s
I20260709 12:21:58.074900 20525 leader_election.cc:290] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:57.944556 20921 raft_consensus.cc:493] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:21:58.076000 20921 raft_consensus.cc:515] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:57.968410 20776 consensus_queue.cc:1048] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.112s
I20260709 12:21:57.953327 20828 raft_consensus.cc:493] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:21:58.086628 20828 raft_consensus.cc:3060] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:57.975899 20911 raft_consensus.cc:515] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:57.950067 17829 rpcz_store.cc:276] 0709 12:21:56.823041 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:57.836922 (+1013881us) service_pool.cc:224] Handling call
0709 12:21:57.837354 (+   432us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:57.895955 (+ 58601us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:57.895995 (+    40us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:57.896058 (+    63us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:21:57.896110 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:57.896204 (+    94us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:57.896352 (+   148us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:58.056876 20990 raft_consensus.cc:3060] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:21:58.157509 17829 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35040 (request call id 2111) took 1003 ms. Trace:
I20260709 12:21:58.158174 17829 rpcz_store.cc:276] 0709 12:21:57.153815 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:58.156416 (+1002601us) service_pool.cc:224] Handling call
0709 12:21:58.156850 (+   434us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:58.157073 (+   223us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:58.157111 (+    38us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:58.157167 (+    56us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:21:58.157209 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:58.157286 (+    77us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:58.157445 (+   159us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:21:58.171895 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:58.173570 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 323051us
W20260709 12:21:58.097569 20941 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.557s	user 0.036s	sys 0.096s
W20260709 12:21:58.168867 20980 consensus_meta.cc:287] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.505s	user 0.009s	sys 0.002s
I20260709 12:21:58.193212 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:58.195997 18364 raft_consensus.cc:1240] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 3. Current term is 4. Ops: []
W20260709 12:21:58.184975 20941 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.645s	user 0.037s	sys 0.096s
W20260709 12:21:58.199776 17764 consensus_meta.cc:287] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.589s	user 0.012s	sys 0.000s
I20260709 12:21:58.202951 17764 raft_consensus.cc:2468] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 8.
I20260709 12:21:58.041766 20892 raft_consensus.cc:515] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:58.220597 18333 leader_election.cc:304] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:21:58.222702 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:58.223711 18364 raft_consensus.cc:2410] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for term 10 because replica has last-logged OpId of term: 9 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 8.
I20260709 12:21:58.186481 20921 leader_election.cc:290] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:58.189533 20980 raft_consensus.cc:515] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:21:58.237761 20864 consensus_meta.cc:287] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.672s	user 0.008s	sys 0.003s
W20260709 12:21:58.279048 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 58520.6us
W20260709 12:21:58.285881 17729 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.564s	user 0.006s	sys 0.009s
W20260709 12:21:58.286334 17729 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.564s	user 0.006s	sys 0.009s
W20260709 12:21:58.286706 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 564660us
I20260709 12:21:58.244863 20846 raft_consensus.cc:2749] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
W20260709 12:21:58.298596 17727 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.576s	user 0.000s	sys 0.017s
I20260709 12:21:58.299721 18333 leader_election.cc:304] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
W20260709 12:21:58.307416 17727 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.585s	user 0.000s	sys 0.018s
W20260709 12:21:58.305601 18374 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.569s	user 0.011s	sys 0.004s
W20260709 12:21:58.314780 18374 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.578s	user 0.011s	sys 0.004s
I20260709 12:21:58.274073 20847 leader_election.cc:290] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 7 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:58.321539 18333 leader_election.cc:304] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:21:58.298918 20983 raft_consensus.cc:493] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
W20260709 12:21:58.307951 20693 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.569s	user 0.006s	sys 0.009s
W20260709 12:21:58.326031 20783 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.586s	user 0.009s	sys 0.007s
I20260709 12:21:58.274901 20864 raft_consensus.cc:515] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:58.333389 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:58.322407 20977 leader_election.cc:290] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:21:58.353304 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:58.357568 17655 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:58.330478 20776 raft_consensus.cc:2804] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260709 12:21:58.329793 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 607962us
W20260709 12:21:58.333209 20783 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.594s	user 0.009s	sys 0.008s
I20260709 12:21:58.367380 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:58.344153 20974 raft_consensus.cc:2804] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Leader election won for term 6
W20260709 12:21:58.371846 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 463818us
I20260709 12:21:58.279907 17829 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "d522f70beb1f417cb98940d90b594823"
dest_uuid: "e063541542a94969898868bf6eb5ce1b"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:35032
W20260709 12:21:58.343032 20693 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.594s	user 0.006s	sys 0.009s
I20260709 12:21:58.480312 18362 raft_consensus.cc:3060] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:58.341940 20983 raft_consensus.cc:515] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:21:58.457341 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 83161us
I20260709 12:21:58.488513 17829 raft_consensus.cc:606] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 4 LEADER]: Received request to transfer leadership
I20260709 12:21:58.359947 20397 raft_consensus.cc:493] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
W20260709 12:21:58.449419 20992 log.cc:927] Time spent T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.398s	user 0.000s	sys 0.001s
W20260709 12:21:58.493733 17803 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.512s	user 0.021s	sys 0.069s
I20260709 12:21:58.374506 20394 raft_consensus.cc:493] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:58.380463 20998 log.cc:927] Time spent T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.065s	user 0.000s	sys 0.001s
W20260709 12:21:58.490058 17760 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.615s	user 0.009s	sys 0.012s
W20260709 12:21:58.501482 17803 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.520s	user 0.022s	sys 0.069s
I20260709 12:21:58.373144 20911 leader_election.cc:290] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 6 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:58.499775 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:21:58.509690 17760 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.640s	user 0.009s	sys 0.012s
W20260709 12:21:58.491017 20368 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.501s	user 0.005s	sys 0.017s
W20260709 12:21:58.515812 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 339211us
W20260709 12:21:58.517762 17803 leader_election.cc:336] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Timed out: RequestConsensusVote RPC to 127.16.29.68:42577 timed out after 1.596s (SENT)
I20260709 12:21:58.514356 17829 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:35032 (request call id 97) took 1622 ms. Trace:
I20260709 12:21:58.522681 17829 rpcz_store.cc:276] 0709 12:21:56.875392 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:58.279690 (+1404298us) service_pool.cc:224] Handling call
0709 12:21:58.497412 (+217722us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:58.526384 18367 raft_consensus.cc:1275] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 5 index: 8. (index mismatch)
I20260709 12:21:58.528170 17727 leader_election.cc:304] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:21:58.529430 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:58.530030 17760 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38786 (request call id 2556) took 1055 ms. Trace:
I20260709 12:21:58.541402 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:58.542212 18367 raft_consensus.cc:2468] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 6.
I20260709 12:21:58.544308 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:58.545800 17727 leader_election.cc:304] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:21:58.522322 20368 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.563s	user 0.006s	sys 0.017s
W20260709 12:21:58.562937 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:58.494493 20984 consensus_queue.cc:1059] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.036s
I20260709 12:21:58.572166 20977 raft_consensus.cc:493] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:21:58.574715 17762 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38786 (request call id 2558) took 1030 ms. Trace:
I20260709 12:21:58.597419 17762 rpcz_store.cc:276] 0709 12:21:57.543827 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:57.773079 (+229252us) service_pool.cc:224] Handling call
0709 12:21:57.773436 (+   357us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:57.773788 (+   352us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:57.773813 (+    25us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:57.774334 (+   521us) log.cc:844] Serialized 26 byte log entry
0709 12:21:58.247515 (+473181us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:21:58.247556 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:58.247626 (+    70us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:58.574509 (+326883us) raft_consensus.cc:1685] finished
0709 12:21:58.574528 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:58.574656 (+   128us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":531758,"raft.run_cpu_time_us":107,"raft.run_wall_time_us":2890,"thread_start_us":472772,"threads_started":1,"wal-append.queue_time_us":719866}
I20260709 12:21:58.578459 17832 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35040 (request call id 2119) took 1024 ms. Trace:
I20260709 12:21:58.529676 20525 raft_consensus.cc:493] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:58.580816 20864 leader_election.cc:290] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 6 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:58.581435 20980 leader_election.cc:290] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 5 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:58.514577 20394 raft_consensus.cc:515] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:58.530894 17760 rpcz_store.cc:276] 0709 12:21:57.474010 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:57.719549 (+245539us) service_pool.cc:224] Handling call
0709 12:21:57.719836 (+   287us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:57.824348 (+104512us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 12:21:57.869189 (+ 44841us) log.cc:844] Serialized 14 byte log entry
0709 12:21:58.529009 (+659820us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:58.529066 (+    57us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:21:58.529120 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:58.529192 (+    72us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:58.529297 (+   105us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":16816,"prepare.queue_time_us":47459,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":35,"raft.queue_time_us":651450,"raft.run_cpu_time_us":181,"raft.run_wall_time_us":25288,"thread_start_us":644306,"threads_started":1}
I20260709 12:21:58.594332 21008 raft_consensus.cc:2749] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:21:58.595062 17758 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38776 (request call id 1912) took 1251 ms. Trace:
W20260709 12:21:58.634040 17755 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.848s	user 0.002s	sys 0.021s
W20260709 12:21:58.595748 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 86179us
I20260709 12:21:58.634647 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:58.634907 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:58.599009 17832 rpcz_store.cc:276] 0709 12:21:57.553701 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:57.577252 (+ 23551us) service_pool.cc:224] Handling call
0709 12:21:58.578282 (+1001030us) spinlock_profiling.cc:243] Waited 304 ms on lock 0x7b54003cc8d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:21:58.578405 (+   123us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":699190016}
W20260709 12:21:58.584403 21001 log.cc:927] Time spent T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.341s	user 0.000s	sys 0.001s
W20260709 12:21:58.583580 20996 log.cc:927] Time spent T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.242s	user 0.001s	sys 0.000s
I20260709 12:21:58.633725 17758 rpcz_store.cc:276] 0709 12:21:57.343387 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:57.343767 (+   380us) service_pool.cc:224] Handling call
0709 12:21:57.355328 (+ 11561us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:57.355755 (+   427us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:57.355790 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:57.356302 (+   512us) log.cc:844] Serialized 26 byte log entry
0709 12:21:57.631554 (+275252us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:21:57.686476 (+ 54922us) log.cc:844] Serialized 14 byte log entry
0709 12:21:57.686704 (+   228us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:57.686807 (+   103us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:58.513944 (+827137us) spinlock_profiling.cc:243] Waited 2.69 ms on lock 0x7b18003d9420. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b475333f3 00007f7b475357c1 00007f7b47535779 00007f7b47535709 00007f7b475356d1 00007f7b475349cd 0000000000388634 0000000000388479 00007f7b4751683a
0709 12:21:58.584449 (+ 70505us) raft_consensus.cc:1685] finished
0709 12:21:58.594812 (+ 10363us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:58.594987 (+   175us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":14436,"prepare.queue_time_us":54723,"prepare.run_cpu_time_us":67,"prepare.run_wall_time_us":6746,"raft.queue_time_us":558,"raft.run_cpu_time_us":265,"raft.run_wall_time_us":10748,"spinlock_wait_cycles":6189440,"thread_start_us":274863,"threads_started":1,"wal-append.queue_time_us":569678}
W20260709 12:21:58.583066 20768 consensus_meta.cc:287] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.887s	user 0.011s	sys 0.000s
W20260709 12:21:58.634646 17755 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.848s	user 0.002s	sys 0.021s
W20260709 12:21:58.599273 20843 consensus_meta.cc:287] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.702s	user 0.011s	sys 0.001s
I20260709 12:21:58.642438 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 11 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:58.622151 18362 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 5.
I20260709 12:21:58.660543 18367 raft_consensus.cc:2393] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 11: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:21:58.547240 20397 raft_consensus.cc:515] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:58.677855 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:58.640640 20984 consensus_queue.cc:1243] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer e063541542a94969898868bf6eb5ce1b log is divergent from this leader: its last log entry 8.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260709 12:21:58.603410 20977 raft_consensus.cc:515] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:58.667162 20843 raft_consensus.cc:493] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:58.657378 20980 raft_consensus.cc:493] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:58.659643 20768 raft_consensus.cc:515] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:58.633713 20525 raft_consensus.cc:515] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:21:58.795267 17655 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:58.810539 17687 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.591s	user 0.064s	sys 0.122s
W20260709 12:21:58.811268 17687 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.592s	user 0.065s	sys 0.122s
I20260709 12:21:58.821894 17683 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37396 (request call id 1993) took 1257 ms. Trace:
W20260709 12:21:58.824466 17831 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.681s	user 0.009s	sys 0.003s
I20260709 12:21:58.825345 17830 raft_consensus.cc:1240] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 8. Current term is 9. Ops: []
W20260709 12:21:58.787458 20996 log.cc:927] Time spent T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.056s	user 0.001s	sys 0.000s
W20260709 12:21:58.833653 20904 consensus_meta.cc:287] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 1.202s	user 0.014s	sys 0.000s
I20260709 12:21:58.834414 20904 raft_consensus.cc:515] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:58.833724 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:58.836465 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:58.837173 17759 raft_consensus.cc:2468] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 8.
W20260709 12:21:58.837787 21020 log.cc:927] Time spent T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.085s	user 0.000s	sys 0.000s
I20260709 12:21:58.837865 20525 leader_election.cc:290] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:58.839389 21017 raft_consensus.cc:2804] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:21:58.839857 21017 raft_consensus.cc:493] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:21:58.840224 21017 raft_consensus.cc:3060] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:21:58.837195 18364 raft_consensus.cc:2368] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for earlier term 4. Current term is 5.
I20260709 12:21:58.843012 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:21:58.834470 21006 log.cc:927] Time spent T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.167s	user 0.000s	sys 0.000s
I20260709 12:21:58.822387 17683 rpcz_store.cc:276] 0709 12:21:57.536333 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:57.536470 (+   137us) service_pool.cc:224] Handling call
0709 12:21:57.540702 (+  4232us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:57.553973 (+ 13271us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:57.554002 (+    29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:57.554540 (+   538us) log.cc:844] Serialized 26 byte log entry
0709 12:21:57.874651 (+320111us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:21:57.874710 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:57.874800 (+    90us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:58.793235 (+918435us) raft_consensus.cc:1685] finished
0709 12:21:58.793254 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:58.793378 (+   124us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":12886,"raft.queue_time_us":14532,"raft.run_cpu_time_us":49,"raft.run_wall_time_us":50,"thread_start_us":319635,"threads_started":1,"wal-append.queue_time_us":636349}
I20260709 12:21:58.846169 18374 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59886 (request call id 950) took 1134 ms. Trace:
I20260709 12:21:58.846542 18374 rpcz_store.cc:276] 0709 12:21:57.711988 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:57.712124 (+   136us) service_pool.cc:224] Handling call
0709 12:21:57.734994 (+ 22870us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:57.735427 (+   433us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:57.735463 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:57.736038 (+   575us) log.cc:844] Serialized 26 byte log entry
0709 12:21:58.326279 (+590241us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:21:58.326345 (+    66us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:58.326437 (+    92us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:58.845939 (+519502us) raft_consensus.cc:1685] finished
0709 12:21:58.845960 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:58.846091 (+   131us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":8418,"raft.run_cpu_time_us":121,"raft.run_wall_time_us":121,"thread_start_us":578779,"threads_started":1,"wal-append.queue_time_us":800777}
I20260709 12:21:58.848589 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:58.849242 17833 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35040 (request call id 2109) took 1953 ms. Trace:
W20260709 12:21:58.850080 21009 log.cc:927] Time spent T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.251s	user 0.000s	sys 0.001s
I20260709 12:21:58.849633 17833 rpcz_store.cc:276] 0709 12:21:56.895503 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:57.917210 (+1021707us) service_pool.cc:224] Handling call
0709 12:21:57.937964 (+ 20754us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:57.953530 (+ 15566us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:57.953575 (+    45us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:58.041511 (+ 87936us) log.cc:844] Serialized 26 byte log entry
0709 12:21:58.520451 (+478940us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:21:58.520498 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:58.520563 (+    65us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:58.849052 (+328489us) raft_consensus.cc:1685] finished
0709 12:21:58.849070 (+    18us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:58.849190 (+   120us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":12518,"raft.queue_time_us":21358,"raft.run_cpu_time_us":144,"raft.run_wall_time_us":7838,"thread_start_us":329073,"threads_started":1,"wal-append.queue_time_us":507319}
I20260709 12:21:58.849617 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:58.852793 20768 leader_election.cc:290] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 9 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:58.853098 18374 raft_consensus.cc:3060] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:21:58.857000 20892 leader_election.cc:290] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:58.861302 20397 leader_election.cc:290] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:58.866278 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:21:58.823676 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 321970us
W20260709 12:21:58.827441 17831 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.774s	user 0.009s	sys 0.003s
W20260709 12:21:58.825618 17728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:58.753257 20984 raft_consensus.cc:3055] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 7 LEADER]: Stepping down as leader of term 7
W20260709 12:21:58.873425 17728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:58.873835 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 328093us
I20260709 12:21:58.875574 17727 leader_election.cc:304] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:21:58.910497 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:58.866853 18374 raft_consensus.cc:2468] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 8.
I20260709 12:21:58.919994 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:58.767306 20980 raft_consensus.cc:515] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:58.752815 20843 raft_consensus.cc:515] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:21:58.932046 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 56586.8us
W20260709 12:21:58.913120 21003 log.cc:927] Time spent T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.666s	user 0.008s	sys 0.012s
I20260709 12:21:58.939760 17727 leader_election.cc:304] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:21:58.944545 17829 raft_consensus.cc:2468] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 7.
I20260709 12:21:58.877305 20984 raft_consensus.cc:740] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
W20260709 12:21:58.932224 20828 consensus_meta.cc:287] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.845s	user 0.004s	sys 0.008s
I20260709 12:21:58.953541 17836 raft_consensus.cc:1240] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 4. Current term is 5. Ops: []
I20260709 12:21:58.957628 17727 leader_election.cc:304] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:21:58.957963 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
W20260709 12:21:58.965982 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:58.931849 20397 raft_consensus.cc:493] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:58.963963 17729 leader_election.cc:304] T 7cb3768d215e409fb9f5baf18d124442 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:21:58.944077 20783 raft_consensus.cc:2749] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:21:58.954449 21015 raft_consensus.cc:2804] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260709 12:21:58.973388 20990 consensus_meta.cc:287] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.795s	user 0.013s	sys 0.000s
I20260709 12:21:58.983752 18364 raft_consensus.cc:1240] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 10. Current term is 11. Ops: []
I20260709 12:21:58.972957 21013 raft_consensus.cc:493] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:21:58.976559 21015 raft_consensus.cc:493] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:58.964831 20828 raft_consensus.cc:515] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:58.985425 20990 raft_consensus.cc:2749] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:21:58.993305 21015 raft_consensus.cc:3060] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:21:58.973202 20397 raft_consensus.cc:515] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:58.995177 20846 raft_consensus.cc:493] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:21:58.996014 20776 raft_consensus.cc:493] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:21:58.996263 21045 raft_consensus.cc:493] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:59.041991 20783 raft_consensus.cc:493] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:21:59.072471 20783 raft_consensus.cc:515] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:59.042414 20368 consensus_queue.cc:1059] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.032s
I20260709 12:21:59.074615 20783 leader_election.cc:290] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:59.074774 20368 raft_consensus.cc:3055] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 6 LEADER]: Stepping down as leader of term 6
W20260709 12:21:58.996945 18333 connection.cc:256] RPC call timeout handler was delayed by -1.4703191300000071s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16290000000000002s for a response
I20260709 12:21:58.981616 20394 leader_election.cc:290] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:59.006958 17729 leader_election.cc:400] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 4 election: Vote denied by peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577) with higher term. Message: Invalid argument: T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for earlier term 4. Current term is 5.
W20260709 12:21:59.008472 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 56494.3us
W20260709 12:21:59.043300 21022 log.cc:927] Time spent T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.211s	user 0.000s	sys 0.001s
I20260709 12:21:59.038139 20984 consensus_queue.cc:260] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:59.007534 21013 raft_consensus.cc:515] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:58.987591 21033 raft_consensus.cc:2749] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260709 12:21:59.058820 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:59.143115 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 66852.7us
I20260709 12:21:59.075297 20368 raft_consensus.cc:740] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 6 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
W20260709 12:21:59.001499 20867 consensus_meta.cc:287] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 1.062s	user 0.014s	sys 0.000s
I20260709 12:21:59.075686 17729 leader_election.cc:403] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
W20260709 12:21:59.076892 18333 connection.cc:256] RPC call timeout handler was delayed by -1.4307191299999431s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1673s for a response
I20260709 12:21:59.053406 20843 leader_election.cc:290] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:21:59.057310 21026 log.cc:927] Time spent T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.077s	user 0.000s	sys 0.000s
I20260709 12:21:59.059262 20990 raft_consensus.cc:493] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:21:59.042513 20810 raft_consensus.cc:2749] T 7cb3768d215e409fb9f5baf18d124442 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:21:59.010593 21042 raft_consensus.cc:3055] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 7 LEADER]: Stepping down as leader of term 7
W20260709 12:21:59.197213 18333 connection.cc:256] RPC call timeout handler was delayed by -1.3191191300002174s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1797s for a response
W20260709 12:21:59.205704 18333 connection.cc:256] RPC call timeout handler was delayed by -1.2606191300001228s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18620000000000003s for a response
W20260709 12:21:59.199692 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 192924us
I20260709 12:21:59.087641 20776 raft_consensus.cc:3060] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:21:59.174181 17655 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:59.037851 21047 raft_consensus.cc:493] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:59.037113 20999 log.cc:927] Time spent T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.523s	user 0.008s	sys 0.001s
I20260709 12:21:59.038914 20904 leader_election.cc:290] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 5 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:59.059599 20980 leader_election.cc:290] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:59.118700 20846 raft_consensus.cc:515] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:59.055291 20397 leader_election.cc:290] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:59.143091 20394 raft_consensus.cc:493] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:21:59.186902 17727 leader_election.cc:304] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:21:59.186192 20787 raft_consensus.cc:493] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:21:59.210496 18333 leader_election.cc:304] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:59.201953 21028 raft_consensus.cc:3060] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:59.226202 17683 raft_consensus.cc:1275] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260709 12:21:59.228322 17729 leader_election.cc:304] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:21:59.057260 20828 leader_election.cc:290] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 6 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:59.255908 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:59.262516 18374 raft_consensus.cc:2468] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 6.
I20260709 12:21:59.257714 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:59.258167 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:59.205464 21042 raft_consensus.cc:740] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
W20260709 12:21:59.260536 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 50096.8us
I20260709 12:21:59.264719 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:59.201704 20867 raft_consensus.cc:515] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:59.282236 17682 raft_consensus.cc:3060] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:59.294654 17681 raft_consensus.cc:1275] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260709 12:21:59.294809 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:59.283453 17689 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9"
dest_uuid: "0da169ff82674dc1947055ce43de68a3"
mode: GRACEFUL
new_leader_uuid: "334a4e1f59944544b3b3ff36616b57da"
 from {username='slave'} at 127.0.0.1:54540
I20260709 12:21:59.297621 17689 raft_consensus.cc:606] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 8 LEADER]: Received request to transfer leadership to TS 334a4e1f59944544b3b3ff36616b57da
I20260709 12:21:59.118011 21045 raft_consensus.cc:515] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:59.297401 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:59.303376 17683 raft_consensus.cc:1240] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 7. Current term is 8. Ops: []
I20260709 12:21:59.299553 17834 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35040 (request call id 2105) took 2438 ms. Trace:
I20260709 12:21:59.303700 18371 raft_consensus.cc:2468] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 7.
I20260709 12:21:59.299777 17684 raft_consensus.cc:1275] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260709 12:21:59.306845 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:21:59.308257 17831 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35040 (request call id 2110) took 2154 ms. Trace:
I20260709 12:21:59.206053 20990 raft_consensus.cc:515] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:59.312986 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:59.282339 20980 raft_consensus.cc:3060] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:59.262043 21016 raft_consensus.cc:493] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:59.303956 17834 rpcz_store.cc:276] 0709 12:21:56.861059 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:57.869952 (+1008893us) service_pool.cc:224] Handling call
0709 12:21:57.893432 (+ 23480us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:57.902220 (+  8788us) raft_consensus.cc:1534] Early marking committed up to index 3
0709 12:21:57.949323 (+ 47103us) log.cc:844] Serialized 14 byte log entry
0709 12:21:58.581567 (+632244us) spinlock_profiling.cc:243] Waited 54.9 ms on lock 0x7b18003d15c0. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b47515e07 00007f7b503709a9 00007f7b4753269b 00007f7b4753093f 00007f7b47532c02 00007f7b4a1ba20b 00007f7b4a1bfd23 00007f7b4a1c023a 00007f7b4a62aa2a
0709 12:21:59.299121 (+717554us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:59.299188 (+    67us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:21:59.299251 (+    63us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:59.299346 (+    95us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:59.299483 (+   137us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":455949,"prepare.queue_time_us":41802,"prepare.run_cpu_time_us":73,"prepare.run_wall_time_us":8151,"spinlock_wait_cycles":208498176,"thread_start_us":246089,"threads_started":1,"wal-append.queue_time_us":873531}
I20260709 12:21:59.265475 20810 raft_consensus.cc:2749] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 12:21:59.308707 17831 rpcz_store.cc:276] 0709 12:21:57.153493 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:57.989424 (+835931us) service_pool.cc:224] Handling call
0709 12:21:58.052926 (+ 63502us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:58.053169 (+   243us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 12:21:58.053528 (+   359us) log.cc:844] Serialized 14 byte log entry
0709 12:21:59.307919 (+1254391us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:59.307970 (+    51us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:21:59.308024 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:59.308091 (+    67us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:59.308195 (+   104us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":428560,"prepare.queue_time_us":7074,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":41,"thread_start_us":819205,"threads_started":1,"wal-append.queue_time_us":949848}
I20260709 12:21:59.313650 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:59.323110 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:59.324455 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 11 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:21:59.326798 17687 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37396 (request call id 2001) took 1395 ms. Trace:
I20260709 12:21:59.315235 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:59.316712 18365 raft_consensus.cc:2393] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 5: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:21:59.316439 18333 leader_election.cc:304] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:21:59.328399 18371 raft_consensus.cc:2468] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 4.
I20260709 12:21:59.330834 17681 raft_consensus.cc:2393] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 7: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:21:59.330722 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:59.331861 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:21:59.333484 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 146656us
I20260709 12:21:59.319325 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:59.334678 17836 raft_consensus.cc:2468] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 6.
I20260709 12:21:59.299137 20904 raft_consensus.cc:2749] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:21:59.344837 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:59.338675 20702 raft_consensus.cc:493] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:21:59.346581 18333 leader_election.cc:304] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:21:59.349522 17728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:59.327178 17687 rpcz_store.cc:276] 0709 12:21:57.931868 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:57.932061 (+   193us) service_pool.cc:224] Handling call
0709 12:21:57.954376 (+ 22315us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:58.169981 (+215605us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:58.170020 (+    39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:58.180061 (+ 10041us) log.cc:844] Serialized 26 byte log entry
0709 12:21:58.811911 (+631850us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:21:58.811970 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:58.812075 (+   105us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:59.326592 (+514517us) raft_consensus.cc:1685] finished
0709 12:21:59.326614 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:59.326746 (+   132us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":36039,"raft.queue_time_us":85698,"raft.run_cpu_time_us":168,"raft.run_wall_time_us":169,"thread_start_us":592608,"threads_started":1,"wal-append.queue_time_us":728371}
I20260709 12:21:59.225770 21047 raft_consensus.cc:515] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:59.346669 20702 raft_consensus.cc:515] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:59.264694 20394 raft_consensus.cc:515] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:21:59.356987 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:59.358232 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:59.313719 20921 raft_consensus.cc:493] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:21:59.355722 20846 leader_election.cc:290] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:59.362075 20921 raft_consensus.cc:515] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:59.362442 20846 consensus_queue.cc:1059] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 12:21:59.362804 18333 leader_election.cc:304] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:59.358850 21049 raft_consensus.cc:2749] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:21:59.364356 20921 leader_election.cc:290] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:59.364462 21049 consensus_queue.cc:1059] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 12:21:59.365473 21013 leader_election.cc:290] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:21:59.367280 21044 log.cc:927] Time spent T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.105s	user 0.000s	sys 0.000s
I20260709 12:21:59.367973 17837 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37994 (request call id 724) took 2534 ms. Trace:
I20260709 12:21:59.368479 17837 rpcz_store.cc:276] 0709 12:21:56.670962 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:57.151316 (+480354us) service_pool.cc:224] Handling call
0709 12:21:57.151748 (+   432us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:21:57.178947 (+ 27199us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 12:21:57.517739 (+338792us) log.cc:844] Serialized 14 byte log entry
0709 12:21:57.959821 (+442082us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:21:57.984799 (+ 24978us) log.cc:844] Serialized 52 byte log entry
0709 12:21:57.985007 (+   208us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:21:57.985668 (+   661us) log.cc:844] Serialized 14 byte log entry
0709 12:21:58.591846 (+606178us) spinlock_profiling.cc:243] Waited 3.6 ms on lock 0x7b1800281800. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b47533522 00007f7b475357c1 00007f7b47535779 00007f7b47535709 00007f7b475356d1 00007f7b475349cd 0000000000388634 0000000000388479 00007f7b4751683a
0709 12:21:59.204486 (+612640us) spinlock_profiling.cc:243] Waited 43.4 ms on lock 0x7b1800281800. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b474adc0a 00007f7b50bed219 00007f7b47535a40 00007f7b475304e4 00007f7b47530455 00007f7b4a618060 00007f7b4a62a716 00007f7b4a63b287 00007f7b4a63b208
0709 12:21:59.204699 (+   213us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:59.204775 (+    76us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:59.204814 (+    39us) raft_consensus.cc:1685] finished
0709 12:21:59.204826 (+    12us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:59.204971 (+   145us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":797158,"prepare.queue_time_us":445960,"prepare.run_cpu_time_us":87,"prepare.run_wall_time_us":90,"raft.queue_time_us":1263441,"raft.run_cpu_time_us":458,"raft.run_wall_time_us":23697,"spinlock_wait_cycles":108073600,"thread_start_us":711662,"threads_started":2,"wal-append.queue_time_us":687139,"wal-append.run_cpu_time_us":3152,"wal-append.run_wall_time_us":1113568}
I20260709 12:21:59.371457 21041 raft_consensus.cc:2804] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260709 12:21:59.346606 21043 log.cc:927] Time spent T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.091s	user 0.001s	sys 0.000s
I20260709 12:21:59.372052 21041 raft_consensus.cc:493] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:21:59.372602 21041 raft_consensus.cc:3060] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:21:59.373106 17755 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38786 (request call id 2561) took 1718 ms. Trace:
I20260709 12:21:59.373732 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:59.374487 18368 raft_consensus.cc:2393] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 5: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:21:59.338529 21017 raft_consensus.cc:515] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:59.375270 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:59.373489 17755 rpcz_store.cc:276] 0709 12:21:57.655191 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:57.785816 (+130625us) service_pool.cc:224] Handling call
0709 12:21:57.786084 (+   268us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:58.651155 (+865071us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:58.651201 (+    46us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:58.665254 (+ 14053us) log.cc:844] Serialized 26 byte log entry
0709 12:21:58.963732 (+298478us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:21:58.963779 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:58.963848 (+    69us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:59.372883 (+409035us) raft_consensus.cc:1685] finished
0709 12:21:59.372909 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:59.373051 (+   142us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":554110,"raft.run_cpu_time_us":132,"raft.run_wall_time_us":8329,"thread_start_us":1075324,"threads_started":2,"wal-append.queue_time_us":408468}
I20260709 12:21:59.376541 21029 consensus_queue.cc:1059] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 10.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.068s
I20260709 12:21:59.377745 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:21:59.378861 21015 raft_consensus.cc:515] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:59.382957 20851 consensus_queue.cc:1059] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 4.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 12:21:59.383345 21015 leader_election.cc:290] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 9 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:59.384194 20851 consensus_queue.cc:1048] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:21:59.352425 20913 raft_consensus.cc:493] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:21:59.385311 20913 raft_consensus.cc:515] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:59.387326 20913 leader_election.cc:290] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:59.389190 20394 leader_election.cc:290] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:59.389878 20943 raft_consensus.cc:3055] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:21:59.390374 20943 raft_consensus.cc:740] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 4 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:21:59.337296 21016 raft_consensus.cc:515] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:59.394465 21016 leader_election.cc:290] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:59.397500 20702 leader_election.cc:290] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:59.397928 21047 leader_election.cc:290] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:59.398478 21042 consensus_queue.cc:260] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:59.399809 21042 raft_consensus.cc:3060] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:21:59.401298 21019 log.cc:927] Time spent T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.550s	user 0.003s	sys 0.013s
I20260709 12:21:59.410475 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:59.264595 20787 raft_consensus.cc:515] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:59.291186 20368 consensus_queue.cc:260] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:59.360155 17758 raft_consensus.cc:1240] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 5. Current term is 6. Ops: []
I20260709 12:21:59.417229 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:59.411715 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:59.421224 18367 raft_consensus.cc:2410] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for term 10 because replica has last-logged OpId of term: 9 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 8.
I20260709 12:21:59.418011 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:59.422475 18362 raft_consensus.cc:2468] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 7.
W20260709 12:21:59.423519 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 548904us
I20260709 12:21:59.398655 21047 raft_consensus.cc:2749] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:21:59.417202 21029 raft_consensus.cc:3055] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:21:59.412676 20843 raft_consensus.cc:493] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:21:59.376456 21017 leader_election.cc:290] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 8 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:59.431680 17682 raft_consensus.cc:2468] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 6.
I20260709 12:21:59.453963 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:21:59.460773 18371 raft_consensus.cc:2376] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 8. Re-sending same reply.
I20260709 12:21:59.461088 21052 raft_consensus.cc:493] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:21:59.376096 21018 raft_consensus.cc:493] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
W20260709 12:21:59.461951 17803 connection.cc:256] RPC call timeout handler was delayed by -1.4677648000001682s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.30600000000000005s for a response
I20260709 12:21:59.461814 21052 raft_consensus.cc:515] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:59.462428 21018 raft_consensus.cc:515] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:21:59.462447 17803 connection.cc:256] RPC call timeout handler was delayed by -1.4392731800003276s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.4752s for a response
W20260709 12:21:59.464504 17803 connection.cc:256] RPC call timeout handler was delayed by -1.3652091330000076s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2017s for a response
W20260709 12:21:59.464823 17803 connection.cc:256] RPC call timeout handler was delayed by -1.3564932510003018s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1557s for a response
I20260709 12:21:59.364539 20990 leader_election.cc:290] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:59.465025 21045 leader_election.cc:290] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:59.432848 20702 raft_consensus.cc:493] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:59.413393 20397 raft_consensus.cc:493] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:59.466053 20702 raft_consensus.cc:515] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:59.466711 20397 raft_consensus.cc:515] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:59.468914 20397 leader_election.cc:290] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:59.469933 18371 raft_consensus.cc:1275] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20260709 12:21:59.411628 20867 leader_election.cc:290] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 5 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:59.464968 21018 leader_election.cc:290] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:21:59.471079 20702 leader_election.cc:290] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:21:59.465181 17803 connection.cc:256] RPC call timeout handler was delayed by -1.3525699990004796s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17310000000000003s for a response
W20260709 12:21:59.481405 17803 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.16.29.68:42577 timed out after 1.620s (SENT) [suppressed 36 similar messages]
I20260709 12:21:59.494623 21028 raft_consensus.cc:2749] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577) with higher term. Message: Invalid argument: T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for earlier term 4. Current term is 5.
I20260709 12:21:59.496013 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:21:59.498198 18366 raft_consensus.cc:1275] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260709 12:21:59.502236 20787 leader_election.cc:290] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:21:59.391534 20943 consensus_queue.cc:260] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:59.453922 21029 raft_consensus.cc:740] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:21:59.520840 17684 raft_consensus.cc:2468] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 6.
I20260709 12:21:59.499388 21041 raft_consensus.cc:515] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:21:59.566706 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:59.764019 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 401243us
W20260709 12:21:59.786876 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 464508us
I20260709 12:21:59.520320 21028 raft_consensus.cc:493] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
W20260709 12:21:59.816484 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 49509.1us
I20260709 12:21:59.824180 21029 consensus_queue.cc:260] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 10.11, Last appended by leader: 11, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:59.503558 20843 raft_consensus.cc:515] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:21:59.825409 17804 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:59.525677 20980 raft_consensus.cc:2749] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260709 12:21:59.833182 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:59.837183 18368 raft_consensus.cc:2468] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 8.
W20260709 12:21:59.851697 17655 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:59.881332 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 464333us
I20260709 12:21:59.882282 17654 leader_election.cc:304] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:21:59.883080 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:59.882320 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:21:59.884045 17682 raft_consensus.cc:2468] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 9.
I20260709 12:21:59.884119 17654 leader_election.cc:304] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:21:59.884907 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 465912us
I20260709 12:21:59.886229 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:21:59.893543 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 412155us
W20260709 12:21:59.894131 17803 connection.cc:256] RPC call timeout handler was delayed by -1.3057699990004039s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17830000000000001s for a response
W20260709 12:21:59.894663 17803 connection.cc:256] RPC call timeout handler was delayed by -1.2824091330003284s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2109s for a response
W20260709 12:21:59.851537 20776 consensus_meta.cc:287] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.593s	user 0.010s	sys 0.000s
W20260709 12:21:59.900540 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 62096.8us
I20260709 12:21:59.884490 20397 raft_consensus.cc:493] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:59.823628 21028 raft_consensus.cc:515] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:21:59.915096 18333 leader_election.cc:304] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:59.898099 20843 leader_election.cc:290] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:59.914350 20921 raft_consensus.cc:493] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:59.931212 17803 leader_election.cc:336] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Timed out: RequestConsensusVote RPC to 127.16.29.68:42577 timed out after 2.179s (SENT)
I20260709 12:21:59.936296 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:59.943488 17835 raft_consensus.cc:2468] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 9.
I20260709 12:21:59.936364 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:59.944937 17836 raft_consensus.cc:2410] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for term 9 because replica has last-logged OpId of term: 8 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 4.
I20260709 12:21:59.893045 21054 consensus_queue.cc:1048] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.041s
I20260709 12:21:59.938004 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:59.939088 18333 leader_election.cc:304] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:21:59.940903 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:59.950044 17837 raft_consensus.cc:2410] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for term 9 because replica has last-logged OpId of term: 8 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 6.
I20260709 12:21:59.941673 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:59.953590 17833 raft_consensus.cc:2393] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 7: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:21:59.947206 17831 raft_consensus.cc:2468] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 5.
I20260709 12:21:59.943279 17803 leader_election.cc:304] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:59.925374 20787 raft_consensus.cc:493] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:21:59.928364 20980 raft_consensus.cc:2749] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20260709 12:21:59.955165 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 69641.4us
I20260709 12:21:59.904366 20776 raft_consensus.cc:515] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:59.960270 18366 raft_consensus.cc:1275] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260709 12:21:59.933383 20843 raft_consensus.cc:2749] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:21:59.961524 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:21:59.968535 21041 leader_election.cc:290] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 8 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:21:59.919729 20397 raft_consensus.cc:515] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:21:59.972683 17652 leader_election.cc:304] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:59.974030 18333 leader_election.cc:304] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:21:59.978327 20994 log.cc:927] Time spent T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 2.036s	user 0.008s	sys 0.024s
I20260709 12:21:59.981554 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:59.984122 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:59.986167 18333 leader_election.cc:304] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:21:59.929322 21057 consensus_queue.cc:1059] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 5, Time since last communication: 0.045s
I20260709 12:21:59.964301 21040 raft_consensus.cc:2804] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:22:00.008417 17837 raft_consensus.cc:2393] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 7: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:21:59.970541 20913 raft_consensus.cc:2804] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:22:00.019791 20913 raft_consensus.cc:493] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:22:00.028504 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 123108us
W20260709 12:22:00.029856 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 145825us
I20260709 12:22:00.031543 17654 leader_election.cc:304] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:22:00.000674 21042 raft_consensus.cc:2749] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:21:59.999396 20980 raft_consensus.cc:2749] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 12:22:00.041776 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
W20260709 12:22:00.042366 17803 connection.cc:256] RPC call timeout handler was delayed by -1.2122091330002149s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2187s for a response
I20260709 12:22:00.020394 20913 raft_consensus.cc:3060] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:21:59.948387 20921 raft_consensus.cc:515] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:00.014060 21054 raft_consensus.cc:2749] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:22:00.015115 21040 raft_consensus.cc:493] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
W20260709 12:22:00.047592 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 74972.1us
I20260709 12:22:00.050030 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:21:59.973093 20787 raft_consensus.cc:515] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:00.052832 17837 raft_consensus.cc:2393] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 5: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:21:59.972163 21059 consensus_queue.cc:1048] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.079s
W20260709 12:22:00.053606 17803 leader_election.cc:336] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Timed out: RequestConsensusVote RPC to 127.16.29.68:42577 timed out after 2.920s (SENT)
I20260709 12:22:00.056171 17803 leader_election.cc:304] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:00.056871 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:00.060084 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:00.062788 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:00.063395 17833 raft_consensus.cc:2468] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 10.
I20260709 12:22:00.064405 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:00.065258 17756 raft_consensus.cc:2393] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 8: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:22:00.065706 17833 raft_consensus.cc:1275] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260709 12:22:00.063421 20397 leader_election.cc:290] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:00.066833 20397 raft_consensus.cc:2749] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 12:22:00.068620 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:00.073642 20977 leader_election.cc:290] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:00.076463 18371 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59886 (request call id 964) took 1098 ms. Trace:
I20260709 12:22:00.076911 20787 leader_election.cc:290] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:00.076881 18371 rpcz_store.cc:276] 0709 12:21:58.978207 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:59.461479 (+483272us) service_pool.cc:224] Handling call
0709 12:21:59.461805 (+   326us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:59.470308 (+  8503us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:00.076234 (+605926us) spinlock_profiling.cc:243] Waited 7.74 ms on lock 0x7b54006fe354. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:22:00.076412 (+   178us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":17806208}
I20260709 12:22:00.078194 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:00.078658 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:00.079150 17689 raft_consensus.cc:1240] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 5. Current term is 6. Ops: [3.2-5.3]
I20260709 12:22:00.079840 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:00.080868 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:00.081347 17833 raft_consensus.cc:3060] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:00.083606 17729 leader_election.cc:304] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:22:00.084873 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:00.085635 20911 raft_consensus.cc:2804] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:22:00.085906 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:00.086081 20911 raft_consensus.cc:493] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:22:00.086519 20911 raft_consensus.cc:3060] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:00.086678 18367 raft_consensus.cc:2393] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 6: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:22:00.078747 18371 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 7.
I20260709 12:22:00.094072 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:00.097812 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:00.053709 21040 raft_consensus.cc:3060] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:00.101428 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:00.108031 17835 raft_consensus.cc:2468] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 4.
I20260709 12:22:00.109732 17762 raft_consensus.cc:2468] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 5.
I20260709 12:22:00.111802 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:00.112084 18333 leader_election.cc:304] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:22:00.112640 18371 raft_consensus.cc:2468] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 4.
I20260709 12:22:00.113742 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:00.115406 17837 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "d522f70beb1f417cb98940d90b594823"
dest_uuid: "e063541542a94969898868bf6eb5ce1b"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:35032
I20260709 12:22:00.119364 18333 leader_election.cc:304] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:00.120715 18333 leader_election.cc:304] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:22:00.121902 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:00.091182 20843 raft_consensus.cc:2749] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:22:00.118273 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:00.098803 21057 raft_consensus.cc:493] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:00.170506 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:00.171283 18371 raft_consensus.cc:2468] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 6.
W20260709 12:22:00.104074 21058 log.cc:927] Time spent T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.146s	user 0.000s	sys 0.001s
I20260709 12:22:00.099336 21031 raft_consensus.cc:493] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:00.168624 17836 raft_consensus.cc:2393] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 6: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:22:00.099586 20977 raft_consensus.cc:493] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
W20260709 12:22:00.179275 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 125692us
I20260709 12:22:00.117071 21054 raft_consensus.cc:2749] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:22:00.097702 21036 consensus_queue.cc:1048] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.028s
I20260709 12:22:00.125784 20846 raft_consensus.cc:2749] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260709 12:22:00.185209 17803 connection.cc:256] RPC call timeout handler was delayed by -1.1870091330001742s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2215s for a response
I20260709 12:22:00.113158 21056 consensus_queue.cc:1059] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 5, Time since last communication: 0.071s
W20260709 12:22:00.187659 21066 log.cc:927] Time spent T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.074s	user 0.000s	sys 0.000s
I20260709 12:22:00.192369 20776 leader_election.cc:290] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 8 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:00.123060 21042 consensus_queue.cc:1059] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.070s
I20260709 12:22:00.175159 20702 consensus_queue.cc:1048] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.018s
I20260709 12:22:00.182637 21028 leader_election.cc:290] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:00.230685 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:00.231922 18374 raft_consensus.cc:2410] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 8 because replica has last-logged OpId of term: 7 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20260709 12:22:00.233018 18363 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 1088) took 1639 ms. Trace:
I20260709 12:22:00.195298 20977 raft_consensus.cc:515] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:00.201270 20921 leader_election.cc:290] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:00.199018 21040 raft_consensus.cc:515] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:00.173305 21067 raft_consensus.cc:493] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:00.103502 20980 consensus_queue.cc:1059] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.078s
W20260709 12:22:00.218423 17803 connection.cc:256] RPC call timeout handler was delayed by -1.0960699990000649s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2016s for a response
W20260709 12:22:00.245690 17803 connection.cc:256] RPC call timeout handler was delayed by -1.0933699990000605s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20190000000000002s for a response
W20260709 12:22:00.246296 17803 connection.cc:256] RPC call timeout handler was delayed by -1.0049558160003471s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21640000000000004s for a response
W20260709 12:22:00.238286 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 206935us
I20260709 12:22:00.250305 17654 leader_election.cc:304] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:00.235267 21056 consensus_queue.cc:1243] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer e063541542a94969898868bf6eb5ce1b log is divergent from this leader: its last log entry 7.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260709 12:22:00.183188 21031 raft_consensus.cc:515] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:22:00.267680 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 215070us
I20260709 12:22:00.183385 21057 raft_consensus.cc:515] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:00.255146 20980 consensus_queue.cc:1243] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer e063541542a94969898868bf6eb5ce1b log is divergent from this leader: its last log entry 6.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260709 12:22:00.233572 18363 rpcz_store.cc:276] 0709 12:21:58.593430 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:58.593543 (+   113us) service_pool.cc:224] Handling call
0709 12:21:58.593864 (+   321us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:58.790347 (+196483us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 12:21:58.835749 (+ 45402us) log.cc:844] Serialized 14 byte log entry
0709 12:21:58.952232 (+116483us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:58.952292 (+    60us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:21:58.953023 (+   731us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:00.232685 (+1279662us) spinlock_profiling.cc:243] Waited 196 ms on lock 0x7b54001efd54. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:22:00.232752 (+    67us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:00.232947 (+   195us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":13315,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":37,"raft.queue_time_us":42675,"raft.run_cpu_time_us":78,"raft.run_wall_time_us":78,"spinlock_wait_cycles":451255680,"thread_start_us":110450,"threads_started":1,"wal-append.queue_time_us":151715,"wal-append.run_cpu_time_us":1687,"wal-append.run_wall_time_us":1049262}
I20260709 12:22:00.252655 20921 raft_consensus.cc:493] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
W20260709 12:22:00.277628 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 156964us
I20260709 12:22:00.257540 20776 raft_consensus.cc:2804] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:22:00.256668 20846 raft_consensus.cc:2749] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:22:00.279333 18366 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 1092) took 1051 ms. Trace:
W20260709 12:22:00.278237 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 192327us
W20260709 12:22:00.286249 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:00.292557 17729 leader_election.cc:304] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:00.268931 21059 raft_consensus.cc:3060] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:00.288368 17803 leader_election.cc:304] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:00.290033 18366 rpcz_store.cc:276] 0709 12:21:59.227471 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:59.974773 (+747302us) service_pool.cc:224] Handling call
0709 12:22:00.001396 (+ 26623us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:00.043540 (+ 42144us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 12:22:00.054080 (+ 10540us) log.cc:844] Serialized 14 byte log entry
0709 12:22:00.071687 (+ 17607us) spinlock_profiling.cc:243] Waited 2.32 ms on lock 0x7b180034d1a0. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b475333f3 00007f7b475357c1 00007f7b47535779 00007f7b47535709 00007f7b475356d1 00007f7b475349cd 0000000000388634 0000000000388479 00007f7b4751683a
0709 12:22:00.278815 (+207128us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:00.278898 (+    83us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:22:00.278975 (+    77us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:00.279067 (+    92us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:00.279256 (+   189us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":10379,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":36,"raft.queue_time_us":3259,"raft.run_cpu_time_us":121,"raft.run_wall_time_us":2466,"spinlock_wait_cycles":5346560,"thread_start_us":194427,"threads_started":2,"wal-append.queue_time_us":195197}
I20260709 12:22:00.298810 17652 leader_election.cc:304] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:22:00.267406 21056 raft_consensus.cc:493] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:00.311337 21031 leader_election.cc:290] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:00.311473 21056 raft_consensus.cc:515] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:00.313606 21057 leader_election.cc:290] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:00.299934 18363 raft_consensus.cc:1240] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 7. Current term is 8. Ops: []
I20260709 12:22:00.314664 21061 raft_consensus.cc:2804] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:22:00.315274 21061 raft_consensus.cc:493] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:00.315706 21061 raft_consensus.cc:3060] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:00.316009 18363 raft_consensus.cc:3055] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:22:00.316443 18363 raft_consensus.cc:740] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:22:00.252225 21067 raft_consensus.cc:515] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:00.317373 18363 consensus_queue.cc:260] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:00.318420 18363 raft_consensus.cc:3060] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:00.301069 17654 leader_election.cc:304] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:22:00.301803 18374 raft_consensus.cc:1275] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260709 12:22:00.302335 21079 raft_consensus.cc:2804] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:22:00.319324 21067 leader_election.cc:290] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:00.319595 21079 raft_consensus.cc:493] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:00.320011 21079 raft_consensus.cc:3060] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:22:00.327534 21070 log.cc:927] Time spent T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.053s	user 0.000s	sys 0.000s
I20260709 12:22:00.288092 20921 raft_consensus.cc:515] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:00.330319 20921 leader_election.cc:290] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:00.266458 21063 consensus_queue.cc:1048] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.051s
I20260709 12:22:00.330734 21061 raft_consensus.cc:515] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:00.301239 20977 leader_election.cc:290] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:00.313689 21056 leader_election.cc:290] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:00.334131 17654 leader_election.cc:304] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:22:00.335060 17654 leader_election.cc:304] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:22:00.284183 20776 raft_consensus.cc:493] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:00.343467 17833 raft_consensus.cc:2468] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 9.
I20260709 12:22:00.335986 20787 raft_consensus.cc:2804] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:22:00.345398 21069 raft_consensus.cc:2804] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:22:00.280098 21028 raft_consensus.cc:493] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
W20260709 12:22:00.350278 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 55841.9us
I20260709 12:22:00.358858 21028 raft_consensus.cc:515] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:00.360325 18333 leader_election.cc:304] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
W20260709 12:22:00.294955 18334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:00.350688 21057 raft_consensus.cc:493] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:00.347229 20776 raft_consensus.cc:3060] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:00.346083 21079 raft_consensus.cc:515] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:00.353070 20943 raft_consensus.cc:3060] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:00.368108 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:00.368711 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:00.372781 17833 raft_consensus.cc:2468] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 6.
I20260709 12:22:00.373464 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:00.354641 20787 raft_consensus.cc:493] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:00.365199 20977 consensus_queue.cc:1059] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.009s
I20260709 12:22:00.371040 18333 leader_election.cc:304] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:22:00.336849 21036 consensus_queue.cc:1048] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.074s
I20260709 12:22:00.374656 21079 leader_election.cc:290] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 5 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:00.376237 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:22:00.377243 21027 log.cc:927] Time spent T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 1.541s	user 0.015s	sys 0.014s
I20260709 12:22:00.381100 18362 raft_consensus.cc:3055] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:22:00.381778 17830 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35040 (request call id 2106) took 3487 ms. Trace:
I20260709 12:22:00.396291 18362 raft_consensus.cc:740] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:22:00.370214 21056 raft_consensus.cc:2804] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:22:00.384620 21040 leader_election.cc:290] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 7 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:00.384845 18368 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59886 (request call id 972) took 1029 ms. Trace:
I20260709 12:22:00.386897 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:00.388228 21073 raft_consensus.cc:2804] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:22:00.355320 21069 raft_consensus.cc:493] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:00.372017 20911 raft_consensus.cc:515] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:00.364879 21028 leader_election.cc:290] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:00.389328 17686 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37396 (request call id 2007) took 1124 ms. Trace:
I20260709 12:22:00.371624 21057 raft_consensus.cc:515] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:00.391858 20907 raft_consensus.cc:493] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:22:00.354803 20913 raft_consensus.cc:515] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:00.394831 17832 raft_consensus.cc:2468] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 4.
I20260709 12:22:00.396503 17830 rpcz_store.cc:276] 0709 12:21:56.894426 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:57.888632 (+994206us) service_pool.cc:224] Handling call
0709 12:21:57.958457 (+ 69825us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:58.826166 (+867709us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:00.381562 (+1555396us) spinlock_profiling.cc:243] Waited 867 ms on lock 0x7b54003345d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:22:00.381724 (+   162us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1993221504}
I20260709 12:22:00.400233 18368 rpcz_store.cc:276] 0709 12:21:59.355307 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:59.375048 (+ 19741us) service_pool.cc:224] Handling call
0709 12:22:00.384684 (+1009636us) spinlock_profiling.cc:243] Waited 461 ms on lock 0x7b5400305f54. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:22:00.384788 (+   104us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1060833536}
I20260709 12:22:00.402959 17829 raft_consensus.cc:2468] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 5.
W20260709 12:22:00.405789 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 70794.8us
I20260709 12:22:00.408627 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 8 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:00.400579 18362 consensus_queue.cc:260] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:00.396013 20787 raft_consensus.cc:3060] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:00.429774 20776 raft_consensus.cc:515] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:00.398152 21082 consensus_queue.cc:1048] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.062s
I20260709 12:22:00.433141 20776 leader_election.cc:290] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 8 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:00.423439 18363 raft_consensus.cc:1275] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 7. (term mismatch)
I20260709 12:22:00.433745 18363 pending_rounds.cc:85] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da: Aborting all ops after (but not including) 6
I20260709 12:22:00.434123 18363 pending_rounds.cc:107] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da: Aborting uncommitted NO_OP operation due to leader change: 5.7
I20260709 12:22:00.434376 18363 raft_consensus.cc:2889] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:22:00.435245 21049 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 6 current term: 6 is higher.
I20260709 12:22:00.424902 20907 raft_consensus.cc:515] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:00.438526 20907 leader_election.cc:290] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:00.401481 21056 raft_consensus.cc:493] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:00.440250 21056 raft_consensus.cc:3060] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:00.425583 17760 raft_consensus.cc:1240] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 5. Current term is 6. Ops: [5.7-5.7]
I20260709 12:22:00.442058 17837 raft_consensus.cc:606] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Received request to transfer leadership
I20260709 12:22:00.442629 17837 raft_consensus.cc:614] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:22:00.422466 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:00.411826 17686 rpcz_store.cc:276] 0709 12:21:59.265700 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:59.294564 (+ 28864us) service_pool.cc:224] Handling call
0709 12:21:59.308159 (+ 13595us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:59.435990 (+127831us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:59.436038 (+    48us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:59.436095 (+    57us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:21:59.436141 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:00.389105 (+952964us) spinlock_profiling.cc:243] Waited 128 ms on lock 0x7b54004268d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:22:00.389151 (+    46us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:00.389269 (+   118us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":293357824}
I20260709 12:22:00.398479 21061 leader_election.cc:290] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 7 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:00.426929 18362 raft_consensus.cc:3060] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:22:00.430179 21025 log.cc:927] Time spent T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 1.591s	user 0.009s	sys 0.026s
I20260709 12:22:00.404376 21073 raft_consensus.cc:493] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:00.410305 21069 raft_consensus.cc:3060] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:22:00.456034 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 85069.3us
W20260709 12:22:00.461846 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 73997.4us
I20260709 12:22:00.462201 17762 raft_consensus.cc:1240] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 5. Current term is 6. Ops: []
W20260709 12:22:00.463943 18333 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:00.465054 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 183964us
I20260709 12:22:00.463999 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:00.448508 21059 raft_consensus.cc:2749] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:22:00.466784 17762 raft_consensus.cc:2468] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 4.
I20260709 12:22:00.472999 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:00.473412 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:00.474254 17762 raft_consensus.cc:2468] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 9.
I20260709 12:22:00.475314 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:00.467352 17764 raft_consensus.cc:1240] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 7. Current term is 8. Ops: []
I20260709 12:22:00.468183 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:00.477844 17760 raft_consensus.cc:2368] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 8. Current term is 9.
I20260709 12:22:00.423979 21028 raft_consensus.cc:493] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:00.488168 17755 raft_consensus.cc:2393] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 7: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:22:00.493947 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:00.478740 17762 raft_consensus.cc:2468] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 7.
I20260709 12:22:00.464767 20977 raft_consensus.cc:2804] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:22:00.460229 21067 raft_consensus.cc:2749] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:22:00.479616 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:00.459969 21073 raft_consensus.cc:3060] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:00.481272 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:00.482332 20907 raft_consensus.cc:493] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:22:00.482652 17685 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44898 (request call id 706) took 2396 ms. Trace:
I20260709 12:22:00.482910 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:00.484510 17803 leader_election.cc:304] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
W20260709 12:22:00.485226 17802 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:00.494074 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 112989us
I20260709 12:22:00.494637 17757 raft_consensus.cc:3060] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:00.479496 21087 raft_consensus.cc:493] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:00.496914 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:00.498394 17760 raft_consensus.cc:1275] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260709 12:22:00.470376 18333 leader_election.cc:304] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:22:00.480661 20847 raft_consensus.cc:493] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:00.504612 17759 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 7.
I20260709 12:22:00.505436 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:00.515215 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:22:00.516870 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 46604.4us
W20260709 12:22:00.519865 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 95998.5us
I20260709 12:22:00.521636 21063 raft_consensus.cc:2749] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:22:00.521019 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:00.564668 18366 raft_consensus.cc:3060] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:00.520694 20847 raft_consensus.cc:515] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:00.508050 17685 rpcz_store.cc:276] 0709 12:21:58.086174 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:58.086386 (+   212us) service_pool.cc:224] Handling call
0709 12:21:58.086847 (+   461us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:58.087342 (+   495us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:58.087383 (+    41us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:58.175477 (+ 88094us) log.cc:844] Serialized 26 byte log entry
0709 12:21:58.503262 (+327785us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:21:58.503336 (+    74us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:58.503414 (+    78us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:00.429487 (+1926073us) spinlock_profiling.cc:243] Waited 34.1 ms on lock 0x7b4400022b4c. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b47459c85 00007f7b4745a6a2 00007f7b4a1cc1db 00007f7b4a1c7aaf 00007f7b4a1bb86e 00007f7b4a1babb6 00007f7b4a1ba7d8 00007f7b4a1c4221 00007f7b4a1c41d9 00007f7b4a1c4169
0709 12:22:00.482384 (+ 52897us) raft_consensus.cc:1685] finished
0709 12:22:00.482404 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:00.482596 (+   192us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":81713,"raft.run_cpu_time_us":137,"raft.run_wall_time_us":6257,"spinlock_wait_cycles":78404480,"thread_start_us":300741,"threads_started":1,"wal-append.queue_time_us":638506}
I20260709 12:22:00.519687 17755 raft_consensus.cc:2468] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 5.
I20260709 12:22:00.493618 21028 raft_consensus.cc:515] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:00.547384 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:00.548802 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:00.566601 17762 raft_consensus.cc:2393] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 5: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:22:00.548923 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:00.552093 17803 leader_election.cc:304] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:22:00.511601 20977 raft_consensus.cc:697] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 5 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:22:00.568539 20911 leader_election.cc:290] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 8 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:00.569131 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:00.569656 20847 leader_election.cc:290] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:00.570726 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:00.571915 21028 leader_election.cc:290] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:00.560374 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:00.572775 17759 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 7.
W20260709 12:22:00.573256 21084 log.cc:927] Time spent T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.093s	user 0.000s	sys 0.001s
I20260709 12:22:00.573181 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:00.573988 20977 consensus_queue.cc:237] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:00.548619 21036 consensus_queue.cc:1059] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.013s
I20260709 12:22:00.561058 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:00.557634 20776 raft_consensus.cc:493] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:00.524950 21082 raft_consensus.cc:2804] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:22:00.562558 17801 leader_election.cc:304] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
W20260709 12:22:00.521831 21086 log.cc:927] Time spent T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.079s	user 0.001s	sys 0.000s
I20260709 12:22:00.520753 18333 leader_election.cc:304] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:22:00.521968 20907 raft_consensus.cc:515] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:00.525506 21087 raft_consensus.cc:515] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:00.544921 21067 raft_consensus.cc:493] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:00.559165 18362 raft_consensus.cc:1275] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260709 12:22:00.585016 17757 raft_consensus.cc:2468] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 9.
I20260709 12:22:00.589880 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:00.587646 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:00.609347 18367 raft_consensus.cc:2468] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 7.
I20260709 12:22:00.581555 17756 raft_consensus.cc:2468] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 8.
I20260709 12:22:00.593619 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:22:00.597499 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 76807.9us
W20260709 12:22:00.603965 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 60225.6us
I20260709 12:22:00.601173 17801 leader_election.cc:304] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:00.604625 17728 leader_election.cc:304] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:00.611965 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:00.611228 18333 leader_election.cc:304] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:22:00.610993 17760 raft_consensus.cc:2468] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 6.
I20260709 12:22:00.614882 18368 raft_consensus.cc:2393] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 7: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:22:00.614935 17801 leader_election.cc:304] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:22:00.605458 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 59617.5us
I20260709 12:22:00.616230 17801 leader_election.cc:304] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:00.618050 18333 leader_election.cc:400] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 8 election: Vote denied by peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651) with higher term. Message: Invalid argument: T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 8. Current term is 9.
W20260709 12:22:00.618484 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 94505.7us
I20260709 12:22:00.618638 18333 leader_election.cc:403] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 8 election: Cancelling election due to peer responding with higher term
I20260709 12:22:00.596576 20847 raft_consensus.cc:2804] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:22:00.626876 18333 leader_election.cc:304] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:00.595196 21082 raft_consensus.cc:493] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:00.620299 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:00.625320 20862 raft_consensus.cc:2804] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:22:00.600085 20776 raft_consensus.cc:515] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:00.579988 20851 consensus_queue.cc:1059] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.034s
I20260709 12:22:00.561764 21089 consensus_queue.cc:1059] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 8.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.053s
I20260709 12:22:00.628791 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:00.636544 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:00.629148 18367 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 5.
I20260709 12:22:00.652475 17803 leader_election.cc:304] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:00.662209 18333 leader_election.cc:304] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:22:00.672293 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 52018.1us
I20260709 12:22:00.595207 21028 consensus_queue.cc:1048] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.058s
I20260709 12:22:00.676012 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:00.632767 21081 raft_consensus.cc:2804] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:22:00.627291 20847 raft_consensus.cc:493] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:00.640878 21082 raft_consensus.cc:3060] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:00.651187 21042 raft_consensus.cc:2749] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 12:22:00.607091 21036 consensus_queue.cc:1048] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.055s
I20260709 12:22:00.632917 20862 raft_consensus.cc:493] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:00.635248 20776 leader_election.cc:290] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:00.632952 21064 raft_consensus.cc:2749] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:22:00.654464 21063 raft_consensus.cc:2804] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260709 12:22:00.753528 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 68956.5us
I20260709 12:22:00.650985 21049 consensus_queue.cc:1059] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.049s
I20260709 12:22:00.673580 21057 leader_election.cc:290] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:00.617094 21067 raft_consensus.cc:515] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:00.753392 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:00.791225 18362 raft_consensus.cc:3060] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:00.688941 20847 raft_consensus.cc:3060] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:00.755934 20975 raft_consensus.cc:2749] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 12:22:00.673440 20913 leader_election.cc:290] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 8 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:22:00.785555 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 171851us
I20260709 12:22:00.754926 17801 leader_election.cc:304] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:00.684775 21081 raft_consensus.cc:493] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:00.755709 21098 raft_consensus.cc:2804] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:22:00.753981 20862 raft_consensus.cc:3060] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:00.820510 17764 raft_consensus.cc:2468] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 7.
W20260709 12:22:00.806748 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 202192us
I20260709 12:22:00.756345 21063 raft_consensus.cc:493] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
W20260709 12:22:00.820202 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 65400.7us
I20260709 12:22:00.825611 17801 leader_election.cc:304] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:00.827797 21108 raft_consensus.cc:2749] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:22:00.830853 21108 raft_consensus.cc:2804] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:22:00.803589 20787 raft_consensus.cc:515] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:00.832463 21053 raft_consensus.cc:2804] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:22:00.833127 21053 raft_consensus.cc:493] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:00.833657 21053 raft_consensus.cc:3060] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:00.834470 20787 leader_election.cc:290] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 7 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:00.834712 20851 raft_consensus.cc:2804] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:22:00.835114 20851 raft_consensus.cc:493] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:00.835472 20851 raft_consensus.cc:3060] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:00.836921 21083 raft_consensus.cc:3060] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:00.840660 21085 raft_consensus.cc:2804] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:22:00.841344 21085 raft_consensus.cc:493] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:00.841825 21085 raft_consensus.cc:3060] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:00.844136 21096 raft_consensus.cc:3055] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:22:00.844583 21096 raft_consensus.cc:740] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 5 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:22:00.845484 21096 consensus_queue.cc:260] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:00.846738 21096 raft_consensus.cc:3060] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:00.829031 17801 leader_election.cc:304] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:00.831559 21108 raft_consensus.cc:697] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 9 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:22:00.815567 21081 raft_consensus.cc:3060] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:00.825407 21063 raft_consensus.cc:3060] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:00.862008 18362 raft_consensus.cc:2468] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 8.
I20260709 12:22:00.864997 21104 raft_consensus.cc:2804] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:22:00.865473 21104 raft_consensus.cc:493] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:00.865882 21104 raft_consensus.cc:3060] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:22:00.866641 21030 log.cc:927] Time spent T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 2.013s	user 0.013s	sys 0.037s
I20260709 12:22:00.872272 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:00.872403 21108 consensus_queue.cc:237] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.6, Last appended by leader: 6, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:00.829293 21098 raft_consensus.cc:697] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:22:00.880179 18366 raft_consensus.cc:2468] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 5.
W20260709 12:22:00.885771 21037 log.cc:927] Time spent T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 2.044s	user 0.025s	sys 0.010s
I20260709 12:22:00.823745 21064 consensus_queue.cc:1048] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.036s
I20260709 12:22:00.871457 21056 raft_consensus.cc:515] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:22:00.890801 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 54659.7us
W20260709 12:22:00.892467 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 285837us
W20260709 12:22:00.894697 21065 log.cc:927] Time spent T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.841s	user 0.000s	sys 0.002s
I20260709 12:22:00.895685 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:22:00.898486 18365 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59918 (request call id 1085) took 2295 ms (client timeout 1756 ms). Trace:
I20260709 12:22:00.899080 18366 raft_consensus.cc:1240] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 7. Current term is 8. Ops: []
W20260709 12:22:00.898919 18365 rpcz_store.cc:269] 0709 12:21:58.592173 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:58.596463 (+  4290us) service_pool.cc:224] Handling call
0709 12:22:00.887495 (+2291032us) spinlock_profiling.cc:243] Waited 681 ms on lock 0x7b54008020d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:22:00.887670 (+   175us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1566408960}
W20260709 12:22:00.907497 21095 log.cc:927] Time spent T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.195s	user 0.001s	sys 0.000s
I20260709 12:22:00.909438 21083 raft_consensus.cc:2749] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Leader election lost for term 8. Reason: Vote denied by peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651) with higher term. Message: Invalid argument: T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 8. Current term is 9.
I20260709 12:22:00.913451 21104 raft_consensus.cc:515] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:00.923076 17757 raft_consensus.cc:1240] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 4. Current term is 5. Ops: []
I20260709 12:22:00.941887 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:00.951581 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:00.957695 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:00.941582 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:00.958532 18371 raft_consensus.cc:2468] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 7.
I20260709 12:22:00.922847 17838 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35040 (request call id 2122) took 1616 ms. Trace:
I20260709 12:22:00.958668 17689 raft_consensus.cc:2468] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 7.
I20260709 12:22:00.915695 21104 leader_election.cc:290] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 9 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:22:01.478078 17686 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.586s	user 0.013s	sys 0.021s
W20260709 12:22:00.997112 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 168134us
W20260709 12:22:01.615163 17686 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.723s	user 0.020s	sys 0.022s
I20260709 12:22:01.615824 17801 leader_election.cc:304] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:01.019814 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:22:01.027480 17802 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:00.990598 18333 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:01.585104 17684 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.690s	user 0.006s	sys 0.013s
W20260709 12:22:01.600868 18362 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.592s	user 0.029s	sys 0.017s
I20260709 12:22:01.020249 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:22:01.619591 18362 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.611s	user 0.030s	sys 0.017s
I20260709 12:22:00.959407 17838 rpcz_store.cc:276] 0709 12:21:59.306033 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:59.306174 (+   141us) service_pool.cc:224] Handling call
0709 12:21:59.331483 (+ 25309us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:59.416966 (+ 85483us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:59.416997 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:00.434795 (+1017798us) spinlock_profiling.cc:243] Waited 4.19 ms on lock 0x7b2c000472a8. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b48f19a11 00007f7b48f29372 00007f7b4a65ba01 00007f7b4a62043a 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2
0709 12:22:00.435709 (+   914us) log.cc:844] Serialized 26 byte log entry
0709 12:22:00.546561 (+110852us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:22:00.546599 (+    38us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:00.546660 (+    61us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:00.912093 (+365433us) raft_consensus.cc:1685] finished
0709 12:22:00.912118 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:00.922787 (+ 10669us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":67949,"raft.queue_time_us":68113,"raft.run_cpu_time_us":82,"raft.run_wall_time_us":82,"spinlock_wait_cycles":9648384,"thread_start_us":86636,"threads_started":1,"wal-append.queue_time_us":183747}
I20260709 12:22:01.008955 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:01.022862 17727 leader_election.cc:304] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
W20260709 12:22:01.612677 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 960065us
W20260709 12:22:01.622924 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 600117us
I20260709 12:22:01.622995 17803 leader_election.cc:304] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:22:01.624047 17727 leader_election.cc:304] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:22:01.614588 18365 raft_consensus.cc:1240] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 7. Current term is 8. Ops: []
I20260709 12:22:01.477547 18363 raft_consensus.cc:1275] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260709 12:22:01.021406 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:01.072758 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:01.629019 17832 raft_consensus.cc:2468] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 6.
I20260709 12:22:01.021904 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:00.965267 17756 raft_consensus.cc:1240] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 7. Current term is 8. Ops: []
W20260709 12:22:01.618825 17684 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.723s	user 0.007s	sys 0.013s
I20260709 12:22:01.620167 17830 raft_consensus.cc:2468] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 4.
I20260709 12:22:01.643836 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:00.920269 21042 raft_consensus.cc:493] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:01.680986 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
W20260709 12:22:01.605293 21073 consensus_meta.cc:287] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 1.088s	user 0.005s	sys 0.005s
W20260709 12:22:01.603804 21063 consensus_meta.cc:287] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.722s	user 0.010s	sys 0.000s
I20260709 12:22:00.991910 20907 leader_election.cc:290] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:01.670202 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:01.720780 21063 raft_consensus.cc:515] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:00.916224 21067 leader_election.cc:290] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:01.722299 21067 raft_consensus.cc:2804] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:22:01.722823 21067 raft_consensus.cc:493] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:01.723268 21067 raft_consensus.cc:3060] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:01.724064 21116 raft_consensus.cc:2804] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:22:01.724629 21116 raft_consensus.cc:493] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:01.725033 21116 raft_consensus.cc:3060] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:00.924228 21087 leader_election.cc:290] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:01.058481 21028 raft_consensus.cc:493] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:01.728540 21110 raft_consensus.cc:2804] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:22:01.729055 21110 raft_consensus.cc:493] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:22:01.728677 21028 raft_consensus.cc:515] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:01.729499 21110 raft_consensus.cc:3060] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:01.020705 21052 leader_election.cc:290] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:01.020951 21056 leader_election.cc:290] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 11 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:00.985525 21107 raft_consensus.cc:3055] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:22:01.072150 21115 raft_consensus.cc:493] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:01.728081 21105 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.604s	user 0.007s	sys 0.007s
I20260709 12:22:01.091953 21090 raft_consensus.cc:493] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:22:00.917696 20851 raft_consensus.cc:515] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:00.923408 20862 raft_consensus.cc:515] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:00.915760 21069 raft_consensus.cc:515] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:00.924369 20847 raft_consensus.cc:515] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:01.009228 21082 raft_consensus.cc:515] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:00.965356 20776 consensus_queue.cc:1059] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.888s
I20260709 12:22:00.921241 21081 raft_consensus.cc:515] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:00.920495 21098 consensus_queue.cc:237] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:01.732187 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:02.252736 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:01.926563 17689 raft_consensus.cc:1275] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
W20260709 12:22:02.261808 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 638869us
I20260709 12:22:01.860512 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:22:02.259056 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 635092us
W20260709 12:22:01.834020 18366 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.890s	user 0.008s	sys 0.004s
W20260709 12:22:02.248004 21118 log.cc:927] Time spent T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.250s	user 0.000s	sys 0.001s
I20260709 12:22:02.284843 17803 leader_election.cc:304] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:02.285508 21089 raft_consensus.cc:3055] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 8 LEADER]: Stepping down as leader of term 8
W20260709 12:22:02.285874 17835 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35042 (request call id 1862) took 2319 ms (client timeout 1753 ms). Trace:
I20260709 12:22:02.273591 17727 leader_election.cc:304] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
W20260709 12:22:02.286363 17835 rpcz_store.cc:269] 0709 12:21:59.969835 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:59.975701 (+  5866us) service_pool.cc:224] Handling call
0709 12:22:02.285645 (+2309944us) spinlock_profiling.cc:243] Waited 78.2 ms on lock 0x7b5400334ad4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:22:02.285773 (+   128us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":179841408}
I20260709 12:22:02.279109 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:02.288311 17682 raft_consensus.cc:3060] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:22:02.273804 18366 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.330s	user 0.014s	sys 0.006s
I20260709 12:22:02.272343 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:02.262704 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:22:01.913771 20943 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.947s	user 0.030s	sys 0.078s
W20260709 12:22:02.291033 17684 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.641s	user 0.000s	sys 0.023s
W20260709 12:22:01.898509 21053 consensus_meta.cc:287] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 1.064s	user 0.005s	sys 0.005s
I20260709 12:22:02.290112 17689 raft_consensus.cc:2468] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 5.
I20260709 12:22:02.297016 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
W20260709 12:22:02.253247 21085 consensus_meta.cc:287] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 1.411s	user 0.013s	sys 0.000s
W20260709 12:22:02.293439 17684 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.661s	user 0.000s	sys 0.024s
W20260709 12:22:01.916759 21124 log.cc:927] Time spent T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.092s	user 0.000s	sys 0.001s
W20260709 12:22:01.864230 21096 consensus_meta.cc:287] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 1.012s	user 0.005s	sys 0.007s
I20260709 12:22:02.293773 17727 leader_election.cc:304] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:22:01.682548 20783 raft_consensus.cc:2804] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260709 12:22:02.297921 20943 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.331s	user 0.044s	sys 0.090s
I20260709 12:22:02.299518 21122 raft_consensus.cc:2749] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:22:02.323465 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:02.325778 17689 raft_consensus.cc:2368] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 8. Current term is 9.
I20260709 12:22:02.294260 21056 raft_consensus.cc:493] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:02.312486 21125 log.cc:927] Time spent T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.278s	user 0.001s	sys 0.000s
W20260709 12:22:02.260104 21105 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.386s	user 0.023s	sys 0.028s
I20260709 12:22:02.262279 21028 leader_election.cc:290] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:01.702193 21104 raft_consensus.cc:493] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:02.259960 21107 raft_consensus.cc:740] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:22:02.344221 21107 consensus_queue.cc:260] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:02.345412 21107 raft_consensus.cc:3060] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:02.310333 17682 raft_consensus.cc:2468] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 5.
I20260709 12:22:02.279523 21063 leader_election.cc:290] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 8 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:02.285956 21089 raft_consensus.cc:740] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:22:02.290622 21090 raft_consensus.cc:515] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:02.350538 21090 leader_election.cc:290] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:02.267031 17831 raft_consensus.cc:2468] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 7.
I20260709 12:22:02.315076 18363 raft_consensus.cc:1240] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 10. Current term is 11. Ops: [10.9-10.9]
I20260709 12:22:02.353135 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:02.353942 18363 raft_consensus.cc:2393] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 5: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:22:02.314960 17803 leader_election.cc:304] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:02.299006 20368 raft_consensus.cc:3060] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:02.327981 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:02.328284 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:01.899068 21067 raft_consensus.cc:515] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:01.804239 21073 raft_consensus.cc:515] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:22:02.335928 18333 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.682s	user 0.005s	sys 0.040s
I20260709 12:22:02.290158 17833 raft_consensus.cc:2468] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 8.
I20260709 12:22:02.295485 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:22:02.372524 18333 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.719s	user 0.006s	sys 0.040s
I20260709 12:22:02.334311 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:02.373152 18365 raft_consensus.cc:3060] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:02.299518 21052 raft_consensus.cc:2749] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:22:02.310663 20983 leader_election.cc:290] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:01.739590 21042 raft_consensus.cc:515] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:02.354086 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:02.312038 20776 raft_consensus.cc:493] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:02.359483 17689 raft_consensus.cc:2368] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 7. Current term is 8.
I20260709 12:22:02.313043 20783 raft_consensus.cc:493] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:02.361234 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:02.354528 21089 consensus_queue.cc:260] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:22:02.365618 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 50740.9us
I20260709 12:22:02.279001 21115 raft_consensus.cc:515] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:02.375735 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:02.353305 21122 raft_consensus.cc:2749] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:22:02.386364 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 8 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:02.388056 17685 raft_consensus.cc:2393] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 5: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:22:02.311406 21085 raft_consensus.cc:515] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:02.390203 18362 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 1106) took 1774 ms. Trace:
I20260709 12:22:02.305974 21053 raft_consensus.cc:515] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:02.393414 18362 rpcz_store.cc:276] 0709 12:22:00.616248 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:00.877475 (+261227us) service_pool.cc:224] Handling call
0709 12:22:00.893051 (+ 15576us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:00.896576 (+  3525us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:00.896608 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:00.903438 (+  6830us) log.cc:844] Serialized 26 byte log entry
0709 12:22:01.620085 (+716647us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:22:01.620125 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:01.620197 (+    72us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:02.389994 (+769797us) raft_consensus.cc:1685] finished
0709 12:22:02.390014 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:02.390132 (+   118us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":82,"raft.run_cpu_time_us":139,"raft.run_wall_time_us":10544,"thread_start_us":611258,"threads_started":1,"wal-append.queue_time_us":1026819}
I20260709 12:22:02.391872 18363 raft_consensus.cc:3060] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:02.378283 21063 raft_consensus.cc:2749] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:22:02.394942 17680 raft_consensus.cc:2410] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for term 9 because replica has last-logged OpId of term: 8 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 7.
I20260709 12:22:02.385303 20783 raft_consensus.cc:3060] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:22:02.448972 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 58135.1us
I20260709 12:22:02.362164 21104 raft_consensus.cc:515] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:02.396061 21089 raft_consensus.cc:3060] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:02.450081 17803 leader_election.cc:304] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:02.343078 21056 raft_consensus.cc:515] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:02.551367 18368 raft_consensus.cc:1240] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 4. Current term is 5. Ops: []
I20260709 12:22:02.385560 20776 raft_consensus.cc:515] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:22:02.554247 21116 consensus_meta.cc:287] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.825s	user 0.014s	sys 0.000s
I20260709 12:22:02.552971 21063 raft_consensus.cc:2749] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260709 12:22:02.589727 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 139737us
I20260709 12:22:02.591259 17803 leader_election.cc:304] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:02.615869 18365 raft_consensus.cc:2468] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 8.
W20260709 12:22:02.489221 17686 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.519s	user 0.024s	sys 0.059s
W20260709 12:22:02.626452 17686 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.680s	user 0.025s	sys 0.059s
I20260709 12:22:02.627470 17686 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37402 (request call id 2383) took 2084 ms. Trace:
W20260709 12:22:02.654183 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 63015.6us
I20260709 12:22:02.583287 21130 raft_consensus.cc:493] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:02.588629 20847 leader_election.cc:290] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 5 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:22:02.698098 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 389403us
W20260709 12:22:02.714241 17652 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:02.716786 18366 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59886 (request call id 988) took 1815 ms. Trace:
I20260709 12:22:02.596820 20851 leader_election.cc:290] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 6 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:02.653877 21063 raft_consensus.cc:2749] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:22:02.723543 17684 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37396 (request call id 2012) took 1829 ms. Trace:
I20260709 12:22:02.928243 17684 rpcz_store.cc:276] 0709 12:22:00.894417 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:00.894544 (+   127us) service_pool.cc:224] Handling call
0709 12:22:00.894879 (+   335us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:00.895064 (+   185us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 12:22:01.632137 (+737073us) log.cc:844] Serialized 14 byte log entry
0709 12:22:02.722954 (+1090817us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:02.723025 (+    71us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:22:02.723091 (+    66us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:02.723195 (+   104us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:02.723377 (+   182us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":841975,"prepare.run_cpu_time_us":50,"prepare.run_wall_time_us":51,"raft.queue_time_us":90541,"raft.run_cpu_time_us":107,"raft.run_wall_time_us":2650,"thread_start_us":1827021,"threads_started":3,"wal-append.queue_time_us":753582}
I20260709 12:22:02.589394 21116 raft_consensus.cc:515] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:02.933218 21116 leader_election.cc:290] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 7 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:02.925946 18366 rpcz_store.cc:276] 0709 12:22:00.901709 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:00.901896 (+   187us) service_pool.cc:224] Handling call
0709 12:22:00.942711 (+ 40815us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:00.943124 (+   413us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:22:02.291834 (+1348710us) log.cc:844] Serialized 14 byte log entry
0709 12:22:02.716422 (+424588us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:02.716476 (+    54us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:22:02.716532 (+    56us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:02.716600 (+    68us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:02.716722 (+   122us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":1453831,"prepare.run_cpu_time_us":102,"prepare.run_wall_time_us":59850,"thread_start_us":1770202,"threads_started":2,"wal-append.queue_time_us":536212}
I20260709 12:22:02.721844 21069 leader_election.cc:290] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 9 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:02.796914 17833 raft_consensus.cc:1240] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 5. Current term is 6. Ops: []
I20260709 12:22:02.806815 17831 raft_consensus.cc:1275] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260709 12:22:02.937325 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:02.938992 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:02.940560 17831 raft_consensus.cc:2468] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 6.
I20260709 12:22:02.942746 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:22:02.807449 21128 log.cc:927] Time spent T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.317s	user 0.001s	sys 0.000s
I20260709 12:22:02.945636 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:02.815583 21067 leader_election.cc:290] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 8 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:02.804138 21052 raft_consensus.cc:493] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:22:02.946914 17831 raft_consensus.cc:3060] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:02.947436 21052 raft_consensus.cc:515] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:02.825062 21053 leader_election.cc:290] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 5 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:02.826288 21115 leader_election.cc:290] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:02.827384 21042 leader_election.cc:290] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:02.950086 21042 raft_consensus.cc:493] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:02.950764 21042 raft_consensus.cc:515] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:02.750635 21081 leader_election.cc:290] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 6 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:02.848326 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:22:02.857434 21110 consensus_meta.cc:287] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 1.127s	user 0.007s	sys 0.008s
I20260709 12:22:02.841338 21073 leader_election.cc:290] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 5 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:02.878604 20776 leader_election.cc:290] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:02.892180 18363 raft_consensus.cc:2468] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 7.
I20260709 12:22:02.906270 17803 leader_election.cc:304] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:22:02.921546 17761 raft_consensus.cc:2468] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 7.
W20260709 12:22:02.927280 20974 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 4.413s	user 0.034s	sys 0.034s
I20260709 12:22:02.814307 21104 leader_election.cc:290] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:02.943655 17833 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 5.
I20260709 12:22:02.986341 17758 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36572 (request call id 638) took 3621 ms. Trace:
W20260709 12:22:03.016403 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 110279us
I20260709 12:22:03.016760 17758 rpcz_store.cc:276] 0709 12:21:59.302416 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:59.302813 (+   397us) service_pool.cc:224] Handling call
0709 12:21:59.316045 (+ 13232us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:59.419733 (+103688us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:02.923252 (+3503519us) spinlock_profiling.cc:243] Waited 43.9 ms on lock 0x7b5400194e54. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:22:02.923494 (+   242us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":100889984}
W20260709 12:22:02.997668 21021 log.cc:927] Time spent T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 4.159s	user 0.018s	sys 0.038s
I20260709 12:22:03.017756 20847 raft_consensus.cc:2804] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:22:03.025808 20847 raft_consensus.cc:493] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:03.026255 21141 raft_consensus.cc:493] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:03.026829 21141 raft_consensus.cc:515] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:03.027468 21145 raft_consensus.cc:493] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:03.027989 17803 leader_election.cc:304] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:03.000365 21131 raft_consensus.cc:493] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:03.028931 21141 leader_election.cc:290] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:03.028815 21131 raft_consensus.cc:515] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:03.029358 17803 leader_election.cc:304] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:22:03.030845 21131 leader_election.cc:290] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:02.987183 21130 raft_consensus.cc:515] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:22:03.016515 20974 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 4.507s	user 0.035s	sys 0.034s
I20260709 12:22:03.033985 20974 raft_consensus.cc:697] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 6 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:22:03.034649 21130 leader_election.cc:290] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:03.026270 20847 raft_consensus.cc:3060] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:03.034962 20974 consensus_queue.cc:237] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:03.036023 21145 raft_consensus.cc:515] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:03.036787 17836 raft_consensus.cc:1240] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 5. Current term is 6. Ops: []
I20260709 12:22:03.037370 17836 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37994 (request call id 755) took 1374 ms. Trace:
I20260709 12:22:03.037704 17836 rpcz_store.cc:276] 0709 12:22:01.662912 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:03.036309 (+1373397us) service_pool.cc:224] Handling call
0709 12:22:03.036629 (+   320us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:03.037151 (+   522us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:03.037320 (+   169us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:03.038223 21145 leader_election.cc:290] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:03.038918 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:03.039263 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:03.039397 21053 raft_consensus.cc:2749] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 12:22:03.040107 21053 raft_consensus.cc:2804] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:22:03.040295 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:03.040529 21053 raft_consensus.cc:493] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:03.040997 21053 raft_consensus.cc:3060] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:22:03.041148 21121 log.cc:927] Time spent T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.651s	user 0.013s	sys 0.000s
I20260709 12:22:03.041427 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:03.042855 18374 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59886 (request call id 969) took 3727 ms. Trace:
I20260709 12:22:03.043234 18374 rpcz_store.cc:276] 0709 12:21:59.315671 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:00.235408 (+919737us) service_pool.cc:224] Handling call
0709 12:22:00.235782 (+   374us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:00.319173 (+ 83391us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:03.042623 (+2723450us) spinlock_profiling.cc:243] Waited 61.3 ms on lock 0x7b5400724054. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:22:03.042792 (+   169us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":140908416}
W20260709 12:22:03.031500 17764 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36572 (request call id 653) took 2521 ms (client timeout 1585 ms). Trace:
W20260709 12:22:03.044265 17764 rpcz_store.cc:269] 0709 12:22:00.509680 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:00.524020 (+ 14340us) service_pool.cc:224] Handling call
0709 12:22:03.031231 (+2507211us) spinlock_profiling.cc:243] Waited 71.1 ms on lock 0x7b54003eb554. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:22:03.031384 (+   153us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":163457280}
I20260709 12:22:03.039760 17838 raft_consensus.cc:3060] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:02.998739 21052 leader_election.cc:290] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:03.045866 17803 leader_election.cc:304] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:03.052438 17654 leader_election.cc:304] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
W20260709 12:22:02.885418 20943 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.523s	user 0.004s	sys 0.004s
I20260709 12:22:02.975637 21085 leader_election.cc:290] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 5 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:03.058822 17684 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba"
dest_uuid: "0da169ff82674dc1947055ce43de68a3"
mode: GRACEFUL
new_leader_uuid: "334a4e1f59944544b3b3ff36616b57da"
 from {username='slave'} at 127.0.0.1:54540
I20260709 12:22:03.061435 17684 raft_consensus.cc:606] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 4 LEADER]: Received request to transfer leadership to TS 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:03.066695 17803 leader_election.cc:304] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:03.068436 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:03.070412 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:03.071254 17683 raft_consensus.cc:2468] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 8.
W20260709 12:22:03.055395 18335 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.531s	user 0.007s	sys 0.003s
W20260709 12:22:03.074869 18335 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.550s	user 0.007s	sys 0.003s
W20260709 12:22:03.056375 20943 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.697s	user 0.004s	sys 0.005s
I20260709 12:22:03.057339 21073 raft_consensus.cc:2804] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:22:03.059563 21052 raft_consensus.cc:2749] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:22:03.083460 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:03.086509 21042 leader_election.cc:290] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:03.081760 21104 raft_consensus.cc:2749] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:22:03.088322 17803 leader_election.cc:304] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:03.084293 21073 raft_consensus.cc:493] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:03.084730 21142 raft_consensus.cc:493] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:03.102182 21142 raft_consensus.cc:515] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:03.092751 21073 raft_consensus.cc:3060] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:03.105993 21142 leader_election.cc:290] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:03.067255 21056 leader_election.cc:290] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:02.904191 17686 rpcz_store.cc:276] 0709 12:22:00.543179 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:00.543639 (+   460us) service_pool.cc:224] Handling call
0709 12:22:00.553883 (+ 10244us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:00.890990 (+337107us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:00.891022 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:00.891559 (+   537us) log.cc:844] Serialized 26 byte log entry
0709 12:22:01.615963 (+724404us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:22:01.945987 (+330024us) log.cc:844] Serialized 14 byte log entry
0709 12:22:02.627020 (+681033us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:02.627130 (+   110us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:02.627192 (+    62us) raft_consensus.cc:1685] finished
0709 12:22:02.627226 (+    34us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:02.627399 (+   173us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":22985,"prepare.queue_time_us":121290,"prepare.run_cpu_time_us":27,"prepare.run_wall_time_us":27,"raft.queue_time_us":514790,"raft.run_cpu_time_us":225,"raft.run_wall_time_us":5389,"thread_start_us":1996088,"threads_started":4,"wal-append.queue_time_us":909784}
I20260709 12:22:03.116724 17803 leader_election.cc:304] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:03.120455 21053 raft_consensus.cc:515] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:22:03.121445 21089 consensus_meta.cc:287] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.568s	user 0.008s	sys 0.003s
W20260709 12:22:03.122529 21107 consensus_meta.cc:287] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.777s	user 0.007s	sys 0.007s
I20260709 12:22:03.129122 17832 raft_consensus.cc:1240] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 7. Current term is 8. Ops: [7.5-7.5]
I20260709 12:22:03.130362 21073 raft_consensus.cc:515] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:03.132521 21029 raft_consensus.cc:3060] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:03.122565 20847 raft_consensus.cc:515] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:03.135859 17834 raft_consensus.cc:3060] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:03.139643 17838 raft_consensus.cc:2468] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 7.
I20260709 12:22:03.001350 21110 raft_consensus.cc:515] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:03.142527 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:03.145546 17803 leader_election.cc:304] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
W20260709 12:22:03.146757 18334 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.866s	user 0.050s	sys 0.116s
I20260709 12:22:03.166370 18374 raft_consensus.cc:3060] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Advancing to term 9
W20260709 12:22:03.166477 18334 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.885s	user 0.051s	sys 0.116s
W20260709 12:22:03.147359 17727 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.835s	user 0.012s	sys 0.024s
I20260709 12:22:03.166901 17838 raft_consensus.cc:1240] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 6. Current term is 7. Ops: []
W20260709 12:22:03.167101 17727 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.854s	user 0.012s	sys 0.025s
I20260709 12:22:03.168026 20157 raft_consensus.cc:3060] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:22:03.147931 18333 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.774s	user 0.015s	sys 0.023s
W20260709 12:22:03.149369 21041 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 7 current term: 7 is higher.
I20260709 12:22:03.149189 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:03.174978 18367 raft_consensus.cc:1240] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 5. Current term is 6. Ops: []
I20260709 12:22:03.149668 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:03.176497 18365 raft_consensus.cc:2393] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 5: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:22:03.149688 21054 raft_consensus.cc:2804] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:22:03.158952 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:03.152084 20984 raft_consensus.cc:3060] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:22:03.169762 18333 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.796s	user 0.015s	sys 0.023s
I20260709 12:22:03.199154 21140 raft_consensus.cc:493] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:03.202066 17831 raft_consensus.cc:2468] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 7.
W20260709 12:22:03.167795 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 874029us
I20260709 12:22:03.203207 17831 raft_consensus.cc:1240] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 7. Current term is 8. Ops: []
I20260709 12:22:03.204349 17831 raft_consensus.cc:1240] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 7. Current term is 8. Ops: []
I20260709 12:22:03.204502 17727 leader_election.cc:304] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:22:03.205811 17834 raft_consensus.cc:2468] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 11.
I20260709 12:22:03.177048 18362 raft_consensus.cc:2468] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 8.
I20260709 12:22:03.206372 18374 raft_consensus.cc:2468] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 9.
I20260709 12:22:03.207023 17727 leader_election.cc:304] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:22:03.200925 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:03.179250 21110 leader_election.cc:290] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 10 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:03.179729 20973 consensus_queue.cc:1048] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 12:22:03.180246 21081 raft_consensus.cc:493] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:03.180835 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 12 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:03.181655 21133 raft_consensus.cc:493] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:03.183238 20157 raft_consensus.cc:2749] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:22:03.185009 18334 leader_election.cc:304] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:03.186048 17837 raft_consensus.cc:1240] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 4. Current term is 5. Ops: []
I20260709 12:22:03.185766 20776 consensus_queue.cc:1048] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
W20260709 12:22:03.190024 17685 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37396 (request call id 2011) took 2296 ms (client timeout 2026 ms). Trace:
W20260709 12:22:03.241734 17685 rpcz_store.cc:269] 0709 12:22:00.894060 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:00.895457 (+  1397us) service_pool.cc:224] Handling call
0709 12:22:03.189803 (+2294346us) spinlock_profiling.cc:243] Waited 1.49 s on lock 0x7b54000cf154. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:22:03.189933 (+   130us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3430290816}
I20260709 12:22:03.192116 20862 leader_election.cc:290] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:03.192788 20980 raft_consensus.cc:3055] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:22:03.193298 20847 leader_election.cc:290] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 6 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:03.196028 20943 consensus_queue.cc:1048] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 12:22:03.209376 17838 raft_consensus.cc:1240] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 4. Current term is 5. Ops: []
W20260709 12:22:03.213053 17681 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37396 (request call id 2014) took 2317 ms (client timeout 2111 ms). Trace:
W20260709 12:22:03.270249 17681 rpcz_store.cc:269] 0709 12:22:00.895237 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:00.895453 (+   216us) service_pool.cc:224] Handling call
0709 12:22:03.200777 (+2305324us) spinlock_profiling.cc:243] Waited 10.6 ms on lock 0x7b5800002a98. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b474fbb71 00007f7b474fba22 00007f7b504148d9 00007f7b4f56eee0 00007f7b4f6380ab 00007f7b4f638131 00007f7b4f5e60a0 00007f7b4f5e5984 00007f7b48b606d4 00007f7b48b60662
0709 12:22:03.212944 (+ 12167us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":24273664}
I20260709 12:22:03.213941 17727 leader_election.cc:304] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:22:03.215838 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:03.214864 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:03.272541 17831 raft_consensus.cc:3060] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:03.217069 21135 raft_consensus.cc:493] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:03.208976 21085 raft_consensus.cc:493] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:03.245187 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 60266.4us
I20260709 12:22:03.246234 17686 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9"
dest_uuid: "0da169ff82674dc1947055ce43de68a3"
mode: GRACEFUL
new_leader_uuid: "334a4e1f59944544b3b3ff36616b57da"
 from {username='slave'} at 127.0.0.1:54540
I20260709 12:22:03.218479 17654 leader_election.cc:304] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:22:03.220556 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:03.280823 17832 raft_consensus.cc:3060] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:03.201443 21054 raft_consensus.cc:493] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:03.199671 21140 raft_consensus.cc:515] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:03.221953 21107 raft_consensus.cc:2749] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:22:03.273953 17687 raft_consensus.cc:2393] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 5: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
W20260709 12:22:03.288893 17687 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37396 (request call id 2015) took 2393 ms (client timeout 1668 ms). Trace:
I20260709 12:22:03.246392 21145 raft_consensus.cc:493] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:22:03.243553 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:22:03.275650 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 61804.3us
I20260709 12:22:03.353726 17727 leader_election.cc:304] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
W20260709 12:22:03.277216 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 58308.5us
I20260709 12:22:03.353367 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:03.357903 17727 leader_election.cc:304] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:22:03.281780 17686 raft_consensus.cc:606] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 10 LEADER]: Received request to transfer leadership to TS 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:03.284389 18334 leader_election.cc:400] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 8 pre-election: Vote denied by peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071) with higher term. Message: Invalid argument: T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 8. Current term is 9.
I20260709 12:22:03.360095 18334 leader_election.cc:403] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 8 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:22:03.244760 21056 raft_consensus.cc:493] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:03.270686 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:03.245719 21053 leader_election.cc:290] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 8 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:03.267313 21130 raft_consensus.cc:2804] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260709 12:22:03.286667 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 68312.7us
I20260709 12:22:03.217881 21013 raft_consensus.cc:2749] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:22:03.283233 21110 raft_consensus.cc:2749] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 12:22:03.301769 17831 raft_consensus.cc:2468] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 7.
W20260709 12:22:03.289270 17687 rpcz_store.cc:269] 0709 12:22:00.895525 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:00.895621 (+    96us) service_pool.cc:224] Handling call
0709 12:22:03.215719 (+2320098us) spinlock_profiling.cc:243] Waited 10.5 ms on lock 0x7b5800002a98. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b474fbb71 00007f7b474fba22 00007f7b504148d9 00007f7b4f56eee0 00007f7b4f6380ab 00007f7b4f638131 00007f7b4f5e60a0 00007f7b4f5e5984 00007f7b48b606d4 00007f7b48b60662
0709 12:22:03.288822 (+ 73103us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":24234240}
I20260709 12:22:03.385306 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 9 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:03.386055 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:03.364037 17688 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37396 (request call id 2008) took 4081 ms. Trace:
I20260709 12:22:03.239172 21081 raft_consensus.cc:515] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:03.387403 17688 rpcz_store.cc:276] 0709 12:21:59.283188 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:59.283349 (+   161us) service_pool.cc:224] Handling call
0709 12:21:59.298747 (+ 15398us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:59.352548 (+ 53801us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:59.352589 (+    41us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:59.352634 (+    45us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:21:59.352668 (+    34us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:03.363717 (+4011049us) spinlock_profiling.cc:243] Waited 53.6 ms on lock 0x7b5400425ed4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:22:03.363779 (+    62us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:03.363966 (+   187us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":123223296}
I20260709 12:22:03.362659 17727 leader_election.cc:304] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:03.239212 21139 consensus_queue.cc:1059] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.069s
I20260709 12:22:03.363526 21137 raft_consensus.cc:2749] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:22:03.394802 21137 raft_consensus.cc:2804] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:22:03.395319 21137 raft_consensus.cc:697] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 9 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
W20260709 12:22:03.395748 18361 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59918 (request call id 1091) took 4801 ms (client timeout 1753 ms). Trace:
W20260709 12:22:03.396200 18361 rpcz_store.cc:269] 0709 12:21:58.594411 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:58.677434 (+ 83023us) service_pool.cc:224] Handling call
0709 12:22:03.395523 (+4718089us) spinlock_profiling.cc:243] Waited 4.6 s on lock 0x7b54006af9d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:22:03.395667 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":10591440768}
I20260709 12:22:03.396376 21137 consensus_queue.cc:237] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:03.397485 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:03.398023 18361 raft_consensus.cc:3060] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:03.398181 18371 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 1108) took 1091 ms. Trace:
I20260709 12:22:03.398583 18371 rpcz_store.cc:276] 0709 12:22:02.306513 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:02.306691 (+   178us) service_pool.cc:224] Handling call
0709 12:22:02.307129 (+   438us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:03.246243 (+939114us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:03.246279 (+    36us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:03.246319 (+    40us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:22:03.246353 (+    34us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:03.397976 (+151623us) spinlock_profiling.cc:243] Waited 939 ms on lock 0x7b540047bdd4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:22:03.398016 (+    40us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:03.398130 (+   114us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2159296384}
I20260709 12:22:03.402715 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:03.287128 21085 raft_consensus.cc:515] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:03.405249 21085 leader_election.cc:290] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:03.405323 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:03.378793 21042 consensus_queue.cc:1059] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.027s
I20260709 12:22:03.406213 18361 raft_consensus.cc:2468] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 8.
I20260709 12:22:03.406805 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:03.407351 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:03.407985 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:03.408787 18371 raft_consensus.cc:2393] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 5: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:22:03.408979 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:03.409511 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:03.410519 18361 raft_consensus.cc:1240] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 6. Current term is 7. Ops: []
I20260709 12:22:03.410807 18371 raft_consensus.cc:1240] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 7. Current term is 8. Ops: []
I20260709 12:22:03.411062 17830 raft_consensus.cc:1240] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 6. Current term is 7. Ops: []
I20260709 12:22:03.411942 18361 raft_consensus.cc:1240] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 7. Current term is 8. Ops: [4.6-7.7]
I20260709 12:22:03.412299 18366 raft_consensus.cc:2468] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 11.
I20260709 12:22:03.413189 21081 leader_election.cc:290] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:03.413738 18371 raft_consensus.cc:1240] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 4. Current term is 5. Ops: []
I20260709 12:22:03.417330 21140 leader_election.cc:290] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:03.417474 18364 raft_consensus.cc:1275] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 8 index: 7. (term mismatch)
I20260709 12:22:03.417817 18364 pending_rounds.cc:85] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da: Aborting all ops after (but not including) 6
I20260709 12:22:03.418030 18364 pending_rounds.cc:107] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da: Aborting uncommitted NO_OP operation due to leader change: 7.7
I20260709 12:22:03.418275 18364 raft_consensus.cc:2889] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:03.418824 21132 raft_consensus.cc:493] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:03.419452 18364 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59902 (request call id 919) took 3389 ms. Trace:
I20260709 12:22:03.419425 21132 raft_consensus.cc:515] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:03.419996 18364 rpcz_store.cc:276] 0709 12:22:00.030225 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:00.377129 (+346904us) service_pool.cc:224] Handling call
0709 12:22:00.377432 (+   303us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:03.418682 (+3041250us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:03.419240 (+   558us) spinlock_profiling.cc:243] Waited 3.04 s on lock 0x7b54002a30d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:22:03.419398 (+   158us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":6990726528}
I20260709 12:22:03.420478 21115 raft_consensus.cc:493] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:03.421168 21115 raft_consensus.cc:515] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:03.423372 21132 leader_election.cc:290] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:03.352348 21147 raft_consensus.cc:2749] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:22:03.382134 21130 raft_consensus.cc:493] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:03.271708 21088 consensus_queue.cc:1048] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.094s
I20260709 12:22:03.284674 21054 raft_consensus.cc:3060] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:03.389783 17836 raft_consensus.cc:2393] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 7: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:22:03.390092 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:03.430956 17837 raft_consensus.cc:2468] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 9.
I20260709 12:22:03.432458 21073 leader_election.cc:290] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 8 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:03.432821 17760 raft_consensus.cc:1240] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 7. Current term is 8. Ops: []
I20260709 12:22:03.432950 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:03.433133 21073 raft_consensus.cc:493] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:03.433845 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:03.369093 21052 raft_consensus.cc:2749] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:22:03.430972 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:03.239390 21133 raft_consensus.cc:515] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:03.434427 17760 raft_consensus.cc:2468] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 7.
I20260709 12:22:03.374147 21056 raft_consensus.cc:515] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:03.435356 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:03.436156 21133 leader_election.cc:290] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:03.366755 20974 raft_consensus.cc:493] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:03.436823 20974 raft_consensus.cc:515] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:22:03.391621 20783 consensus_meta.cc:287] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.901s	user 0.009s	sys 0.005s
I20260709 12:22:03.418871 21082 leader_election.cc:290] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 7 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:03.407812 18368 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 1102) took 2807 ms. Trace:
I20260709 12:22:03.443914 18368 rpcz_store.cc:276] 0709 12:22:00.600603 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:00.680916 (+ 80313us) service_pool.cc:224] Handling call
0709 12:22:00.681341 (+   425us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:02.557048 (+1875707us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:03.407612 (+850564us) spinlock_profiling.cc:243] Waited 1.87 s on lock 0x7b540069e1d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:22:03.407763 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4300095360}
I20260709 12:22:03.386757 21142 raft_consensus.cc:2749] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Leader pre-election lost for term 8. Reason: Vote denied by peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071) with higher term. Message: Invalid argument: T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 8. Current term is 9.
I20260709 12:22:03.424367 21110 raft_consensus.cc:493] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:22:03.425093 18363 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59918 (request call id 1111) took 1092 ms. Trace:
I20260709 12:22:03.378836 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:03.451929 18363 rpcz_store.cc:276] 0709 12:22:02.332778 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:02.360898 (+ 28120us) service_pool.cc:224] Handling call
0709 12:22:03.424903 (+1064005us) spinlock_profiling.cc:243] Waited 9.56 ms on lock 0x7b54006b3854. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:22:03.425012 (+   109us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":21993856}
I20260709 12:22:03.368374 21029 raft_consensus.cc:3060] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:03.455188 21115 leader_election.cc:290] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:03.285362 21135 raft_consensus.cc:515] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:03.455971 21031 raft_consensus.cc:3055] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:22:03.456370 21031 raft_consensus.cc:740] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
W20260709 12:22:03.458099 17761 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36572 (request call id 641) took 4151 ms (client timeout 2190 ms). Trace:
W20260709 12:22:03.458519 17761 rpcz_store.cc:269] 0709 12:21:59.306625 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:59.307050 (+   425us) service_pool.cc:224] Handling call
0709 12:22:03.457933 (+4150883us) spinlock_profiling.cc:243] Waited 3.39 s on lock 0x7b54003e44d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:22:03.458032 (+    99us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":7800131968}
I20260709 12:22:03.459312 21057 raft_consensus.cc:3055] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:22:03.459559 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:03.460364 17761 raft_consensus.cc:2393] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 5: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:22:03.428224 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:03.428460 17831 raft_consensus.cc:1240] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 4. Current term is 5. Ops: []
I20260709 12:22:03.429987 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:03.461354 17764 raft_consensus.cc:2468] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 8.
I20260709 12:22:03.461470 17763 raft_consensus.cc:3060] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:03.431805 21130 raft_consensus.cc:3060] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:03.437088 21056 leader_election.cc:290] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:03.282964 21146 consensus_queue.cc:1059] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.100s
I20260709 12:22:03.437677 18365 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 5.
W20260709 12:22:03.438232 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 157972us
I20260709 12:22:03.469957 18334 leader_election.cc:304] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:22:03.438728 17760 raft_consensus.cc:2468] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 7.
I20260709 12:22:03.390931 18333 leader_election.cc:304] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:22:03.441179 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:03.471283 17762 raft_consensus.cc:3060] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:03.471551 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:03.472695 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:03.473184 17760 raft_consensus.cc:3060] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:03.363291 21145 raft_consensus.cc:515] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:03.452310 17838 raft_consensus.cc:2393] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 5: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:22:03.381476 17654 leader_election.cc:304] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:22:03.264614 20980 raft_consensus.cc:740] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:22:03.459710 21057 raft_consensus.cc:740] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:22:03.433635 21073 raft_consensus.cc:515] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:22:03.465749 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 69919.4us
W20260709 12:22:03.476543 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 95125us
I20260709 12:22:03.480894 17762 raft_consensus.cc:2468] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 9.
I20260709 12:22:03.457302 21135 leader_election.cc:290] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:03.483525 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:03.486920 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:03.495153 17761 raft_consensus.cc:3060] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:03.489550 17654 leader_election.cc:304] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:22:03.491235 17763 raft_consensus.cc:2468] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 8.
I20260709 12:22:03.495460 17762 raft_consensus.cc:3060] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:03.481720 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:03.492053 17760 raft_consensus.cc:2468] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 10.
I20260709 12:22:03.452970 21085 raft_consensus.cc:493] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:03.500514 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:03.501156 17763 raft_consensus.cc:3060] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:03.476140 21089 raft_consensus.cc:493] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:03.504820 17762 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 8.
I20260709 12:22:03.443380 20783 raft_consensus.cc:515] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:03.501353 21139 consensus_queue.cc:1243] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer 334a4e1f59944544b3b3ff36616b57da log is divergent from this leader: its last log entry 9.9 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 8
I20260709 12:22:03.485205 21147 raft_consensus.cc:2804] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:22:03.493166 21098 raft_consensus.cc:493] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:03.518230 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 48379.9us
I20260709 12:22:03.532902 17761 raft_consensus.cc:2468] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 9.
I20260709 12:22:03.533341 18334 leader_election.cc:304] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
W20260709 12:22:03.519026 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 128166us
I20260709 12:22:03.534714 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:03.535513 17761 raft_consensus.cc:2468] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 5.
I20260709 12:22:03.536727 17803 leader_election.cc:304] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:03.537194 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:03.538354 17803 leader_election.cc:304] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:22:03.539474 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:03.503604 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:03.539868 20973 raft_consensus.cc:2749] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:22:03.540109 20941 raft_consensus.cc:2804] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:22:03.542505 20941 raft_consensus.cc:697] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 8 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:22:03.543869 20941 consensus_queue.cc:237] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:03.521849 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:03.524948 17763 raft_consensus.cc:2468] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 7.
I20260709 12:22:03.548456 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:03.548590 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:03.549860 17762 raft_consensus.cc:2410] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for term 8 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20260709 12:22:03.550060 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:03.550832 17763 raft_consensus.cc:3060] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:03.550838 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:03.517870 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:03.551862 17762 raft_consensus.cc:2468] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 8.
I20260709 12:22:03.553164 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:03.553208 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:03.554080 17764 raft_consensus.cc:3060] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:03.554615 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 12 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:03.540791 17760 raft_consensus.cc:2393] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 5: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:22:03.561139 17760 raft_consensus.cc:1275] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
W20260709 12:22:03.562028 17803 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:03.562994 17760 raft_consensus.cc:1240] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 7. Current term is 8. Ops: []
I20260709 12:22:03.563993 17803 leader_election.cc:304] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:03.565213 20862 raft_consensus.cc:3060] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:03.567281 20828 consensus_queue.cc:1059] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 12:22:03.568078 20847 consensus_queue.cc:1059] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:22:03.464954 21110 raft_consensus.cc:515] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:03.574536 17801 leader_election.cc:304] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:22:03.581087 17652 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:03.581919 17760 raft_consensus.cc:1240] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 6. Current term is 7. Ops: []
I20260709 12:22:03.582679 17764 raft_consensus.cc:2468] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 8.
I20260709 12:22:03.540498 17761 raft_consensus.cc:2393] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 7: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:22:03.513943 21085 raft_consensus.cc:515] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:03.586426 17652 leader_election.cc:304] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:22:03.592965 17755 raft_consensus.cc:2393] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 6: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:22:03.530941 21139 raft_consensus.cc:2804] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:22:03.608098 21139 raft_consensus.cc:493] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:03.573078 21147 raft_consensus.cc:697] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 8 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:22:03.608695 21141 raft_consensus.cc:493] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:03.609568 21141 raft_consensus.cc:515] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:03.609905 21147 consensus_queue.cc:237] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:03.573438 21042 raft_consensus.cc:2804] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:22:03.612501 21042 raft_consensus.cc:697] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 7 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:22:03.612818 21073 leader_election.cc:290] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:22:03.600525 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 67268.7us
I20260709 12:22:03.608700 21139 raft_consensus.cc:3060] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:03.531309 21056 raft_consensus.cc:2804] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:22:03.572536 21098 raft_consensus.cc:515] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:03.614995 21056 raft_consensus.cc:697] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 5 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:22:03.568238 20828 consensus_queue.cc:1048] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.008s
I20260709 12:22:03.602610 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:03.617828 17757 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 4.
I20260709 12:22:03.575191 20921 raft_consensus.cc:3055] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:22:03.618898 20921 raft_consensus.cc:740] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 7 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:22:03.601127 17832 raft_consensus.cc:2468] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 6.
I20260709 12:22:03.604063 21154 raft_consensus.cc:2749] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:22:03.604822 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:03.603066 17838 raft_consensus.cc:1240] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 5. Current term is 6. Ops: []
I20260709 12:22:03.603490 18333 leader_election.cc:304] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:22:03.613687 21042 consensus_queue.cc:237] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:03.633919 18334 leader_election.cc:400] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 7 election: Vote denied by peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071) with higher term. Message: Invalid argument: T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 7. Current term is 8.
I20260709 12:22:03.634388 18334 leader_election.cc:403] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 7 election: Cancelling election due to peer responding with higher term
I20260709 12:22:03.634761 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:03.619861 17757 raft_consensus.cc:1240] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 7. Current term is 8. Ops: [7.5-7.5]
I20260709 12:22:03.635186 18362 raft_consensus.cc:3060] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:03.639326 17763 raft_consensus.cc:2468] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 5.
I20260709 12:22:03.634213 17762 raft_consensus.cc:2468] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 11.
I20260709 12:22:03.643863 17763 raft_consensus.cc:1240] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 8. Current term is 9. Ops: []
I20260709 12:22:03.694880 17763 raft_consensus.cc:1275] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20260709 12:22:03.528302 21089 raft_consensus.cc:515] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:22:03.676378 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 90052.7us
I20260709 12:22:03.603788 21061 raft_consensus.cc:993] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3: : Instructing follower 334a4e1f59944544b3b3ff36616b57da to start an election
I20260709 12:22:03.638214 21098 leader_election.cc:290] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:22:03.716822 17729 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:03.717576 17729 connection.cc:256] RPC call timeout handler was delayed by -1.7002886559998842s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1668s for a response
I20260709 12:22:03.717970 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:03.676041 21155 raft_consensus.cc:2804] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:22:03.597976 21053 consensus_queue.cc:1059] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.102s
W20260709 12:22:03.678876 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 45064.7us
W20260709 12:22:03.717990 17729 connection.cc:256] RPC call timeout handler was delayed by -1.424823218000256s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18100000000000002s for a response
I20260709 12:22:03.719619 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:03.718703 17652 leader_election.cc:304] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:22:03.729476 17729 connection.cc:256] RPC call timeout handler was delayed by -1.1285503440003595s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1817s for a response
W20260709 12:22:03.729892 17729 connection.cc:256] RPC call timeout handler was delayed by -1.0226733250001416s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1894s for a response
I20260709 12:22:03.729800 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:03.730546 17729 leader_election.cc:304] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:03.731019 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:03.731382 17729 leader_election.cc:304] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:03.732225 18366 raft_consensus.cc:2410] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for term 6 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260709 12:22:03.730996 17652 leader_election.cc:304] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:22:03.734880 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 12 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:03.735388 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:03.638579 20394 raft_consensus.cc:493] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:03.733064 17729 leader_election.cc:304] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:03.735582 18366 raft_consensus.cc:2468] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 11.
W20260709 12:22:03.752038 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 262553us
I20260709 12:22:03.753337 17654 leader_election.cc:304] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:22:03.676983 21029 raft_consensus.cc:2749] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:22:03.746742 18334 leader_election.cc:304] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:03.705293 21142 raft_consensus.cc:2749] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Leader election lost for term 7. Reason: Vote denied by peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071) with higher term. Message: Invalid argument: T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 7. Current term is 8.
I20260709 12:22:03.714408 21073 raft_consensus.cc:2804] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:22:03.737062 21156 raft_consensus.cc:1081] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 8 LEADER]: Signalling peer 334a4e1f59944544b3b3ff36616b57da to start an election
I20260709 12:22:03.741076 21098 raft_consensus.cc:2804] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:22:03.740944 21107 raft_consensus.cc:2804] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:22:03.740245 21155 raft_consensus.cc:493] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:03.593333 20980 consensus_queue.cc:260] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:03.593490 21057 consensus_queue.cc:260] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:03.748867 21146 raft_consensus.cc:2804] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:22:03.766484 18334 leader_election.cc:304] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:03.797474 18334 leader_election.cc:304] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:22:03.744182 17652 leader_election.cc:304] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:03.742987 21157 raft_consensus.cc:2749] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:22:03.784546 18362 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 8.
I20260709 12:22:03.753996 20911 raft_consensus.cc:2764] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Leader pre-election decision vote started in defunct term 7: won
W20260709 12:22:03.787827 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 213372us
I20260709 12:22:03.816321 17801 leader_election.cc:304] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:03.818388 17801 leader_election.cc:304] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:03.752358 21110 leader_election.cc:290] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:03.821444 17801 leader_election.cc:304] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:03.786837 21155 raft_consensus.cc:3060] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:03.822985 17801 leader_election.cc:304] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:03.714037 21056 consensus_queue.cc:237] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:03.748623 21081 consensus_queue.cc:1059] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.068s
I20260709 12:22:03.759740 21141 leader_election.cc:290] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:22:03.827986 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 83904.3us
I20260709 12:22:03.801290 21088 raft_consensus.cc:2804] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:22:03.825407 17801 leader_election.cc:304] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:03.808600 20974 leader_election.cc:290] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:03.801646 20975 raft_consensus.cc:2749] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:22:03.762514 21061 consensus_queue.cc:1059] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.057s
I20260709 12:22:03.796521 21146 raft_consensus.cc:493] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:22:03.807163 21132 raft_consensus.cc:2804] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:22:03.828917 17652 leader_election.cc:304] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:03.787181 21013 consensus_queue.cc:1048] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.035s
I20260709 12:22:03.797646 21098 raft_consensus.cc:697] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:22:03.761332 21079 consensus_queue.cc:1059] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.058s
I20260709 12:22:03.789445 20984 consensus_queue.cc:1059] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.035s
I20260709 12:22:03.820041 20397 raft_consensus.cc:2749] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:22:03.759205 20394 raft_consensus.cc:515] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:22:03.880978 21164 log.cc:927] Time spent T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.058s	user 0.001s	sys 0.000s
I20260709 12:22:03.880509 21098 consensus_queue.cc:237] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:03.881440 20394 leader_election.cc:290] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:22:03.882539 21168 log.cc:927] Time spent T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.067s	user 0.000s	sys 0.000s
I20260709 12:22:03.883960 21139 raft_consensus.cc:515] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:03.886651 21139 leader_election.cc:290] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 9 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:03.794494 21115 consensus_queue.cc:1048] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.032s
I20260709 12:22:03.844892 21132 raft_consensus.cc:493] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:03.795342 20846 consensus_queue.cc:1059] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.033s
I20260709 12:22:03.837993 17728 leader_election.cc:304] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
W20260709 12:22:03.888962 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 51069.3us
I20260709 12:22:03.760425 21073 raft_consensus.cc:493] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:22:03.892092 21073 raft_consensus.cc:3060] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:03.894085 21110 raft_consensus.cc:2804] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:22:03.894512 21110 raft_consensus.cc:493] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:22:03.894876 21110 raft_consensus.cc:3060] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:22:03.841166 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 82761.4us
I20260709 12:22:03.824642 21053 raft_consensus.cc:2804] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:22:03.825512 21016 raft_consensus.cc:2804] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:22:03.825629 21104 raft_consensus.cc:2804] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:22:03.841949 20783 leader_election.cc:290] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 6 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:03.841282 21141 raft_consensus.cc:3055] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:22:03.841090 21088 raft_consensus.cc:697] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 9 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:22:03.832463 21171 raft_consensus.cc:2804] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:22:03.830010 21170 raft_consensus.cc:2804] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:22:03.876808 20847 raft_consensus.cc:2804] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:22:03.878620 21137 raft_consensus.cc:493] T 7cb3768d215e409fb9f5baf18d124442 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:03.879240 21079 raft_consensus.cc:3055] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:22:03.880213 21054 raft_consensus.cc:515] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:03.888164 21132 raft_consensus.cc:3060] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:22:03.838325 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 85086us
I20260709 12:22:03.915724 17654 leader_election.cc:304] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:22:03.917024 17654 leader_election.cc:304] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:22:03.898221 17729 leader_election.cc:304] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:03.920372 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:03.921294 18366 raft_consensus.cc:2468] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 8.
I20260709 12:22:03.923316 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:03.839807 21145 leader_election.cc:290] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:22:03.898975 17728 connection.cc:256] RPC call timeout handler was delayed by -2.5985815909994017s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2026s for a response
W20260709 12:22:03.925818 17728 connection.cc:256] RPC call timeout handler was delayed by -1.9961943489997793s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1668s for a response
I20260709 12:22:03.841719 21107 raft_consensus.cc:697] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:22:03.862120 21156 raft_consensus.cc:2804] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:22:03.932183 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:03.932736 21156 raft_consensus.cc:493] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:03.912513 21079 raft_consensus.cc:740] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 7 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:22:03.867434 17652 leader_election.cc:304] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:03.912541 21171 raft_consensus.cc:697] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 8 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:22:03.922200 18361 raft_consensus.cc:1240] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 7. Current term is 8. Ops: []
I20260709 12:22:03.937400 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:03.937862 18361 raft_consensus.cc:3060] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:03.899196 21141 raft_consensus.cc:740] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
W20260709 12:22:03.926179 17728 connection.cc:256] RPC call timeout handler was delayed by -1.6268643180001163s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21110000000000004s for a response
I20260709 12:22:03.921757 21089 leader_election.cc:290] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:03.914839 20847 raft_consensus.cc:697] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 7 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:22:03.899955 21053 raft_consensus.cc:697] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 9 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
W20260709 12:22:03.927248 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 101935us
I20260709 12:22:03.930886 17758 raft_consensus.cc:1240] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 5. Current term is 6. Ops: []
I20260709 12:22:03.932025 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:03.933141 17763 raft_consensus.cc:2376] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 7. Re-sending same reply.
I20260709 12:22:03.920112 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:03.908021 21085 leader_election.cc:290] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:03.844990 21146 raft_consensus.cc:3060] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:03.899930 21104 raft_consensus.cc:697] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 8 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:22:03.927640 20394 consensus_queue.cc:1059] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.067s
I20260709 12:22:03.929399 21061 consensus_queue.cc:1059] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.086s
I20260709 12:22:03.938851 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:04.037915 17685 raft_consensus.cc:1240] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 8. Current term is 9. Ops: []
I20260709 12:22:03.929291 20397 raft_consensus.cc:2804] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:22:04.014550 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:04.044148 21141 consensus_queue.cc:260] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:03.939050 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 9 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:04.068710 18333 leader_election.cc:304] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
W20260709 12:22:04.049118 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 181840us
I20260709 12:22:04.054049 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:04.077095 17757 raft_consensus.cc:2368] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 7. Current term is 8.
I20260709 12:22:04.076870 17652 leader_election.cc:304] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:03.930742 21145 raft_consensus.cc:2804] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:22:04.048007 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:04.062552 18361 raft_consensus.cc:2468] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 9.
W20260709 12:22:04.058007 17803 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:04.046084 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:22:04.070892 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 131607us
I20260709 12:22:04.085325 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:22:04.043645 21163 log.cc:927] Time spent T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.211s	user 0.001s	sys 0.000s
I20260709 12:22:04.072849 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:04.078161 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:04.088690 17831 raft_consensus.cc:2468] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 7.
I20260709 12:22:04.089888 18362 raft_consensus.cc:1275] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260709 12:22:04.094835 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:04.096457 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 9 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:04.096601 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:03.932248 21175 consensus_queue.cc:1059] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.203s
I20260709 12:22:04.035477 21079 consensus_queue.cc:260] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:03.952374 18366 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9"
dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
 from {username='slave'} at 127.0.0.1:59918
I20260709 12:22:04.015110 21088 consensus_queue.cc:237] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.7, Last appended by leader: 7, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:03.930145 21137 raft_consensus.cc:515] T 7cb3768d215e409fb9f5baf18d124442 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:04.041363 21107 consensus_queue.cc:237] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:03.936087 21156 raft_consensus.cc:3060] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:04.179828 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 12 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:04.248888 21016 raft_consensus.cc:493] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:04.131898 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:04.283056 21016 raft_consensus.cc:3060] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:04.284922 21155 raft_consensus.cc:515] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:04.155566 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:04.286595 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:04.098438 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:04.159034 17654 leader_election.cc:304] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:22:04.223097 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:04.289374 17759 raft_consensus.cc:2468] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 5.
I20260709 12:22:04.100708 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:04.290836 17757 raft_consensus.cc:2468] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 8.
I20260709 12:22:04.278676 21031 consensus_queue.cc:260] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:22:04.290688 17802 connection.cc:256] RPC call timeout handler was delayed by -2.4735564969996631s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2137s for a response
I20260709 12:22:04.070889 20397 raft_consensus.cc:493] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:03.937048 21139 raft_consensus.cc:493] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:04.204260 21171 consensus_queue.cc:237] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 8, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:22:04.137535 20862 consensus_meta.cc:287] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.572s	user 0.009s	sys 0.001s
I20260709 12:22:04.097204 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:04.192612 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:04.136646 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:04.182065 21054 leader_election.cc:290] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 7 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:04.280849 18365 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59918 (request call id 1122) took 1097 ms. Trace:
I20260709 12:22:04.182535 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:04.136246 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:04.097540 17837 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 5.
I20260709 12:22:04.351209 18365 rpcz_store.cc:276] 0709 12:22:03.192074 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:03.216405 (+ 24331us) service_pool.cc:224] Handling call
0709 12:22:04.280681 (+1064276us) spinlock_profiling.cc:243] Waited 78.1 ms on lock 0x7b54006e23d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:22:04.280790 (+   109us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":179608320}
I20260709 12:22:03.978842 21147 raft_consensus.cc:2749] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:22:04.287015 21137 leader_election.cc:290] T 7cb3768d215e409fb9f5baf18d124442 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:04.192941 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 9 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:04.137352 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 12 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:04.156844 21092 raft_consensus.cc:3055] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:22:04.243867 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:03.946923 21176 raft_consensus.cc:2804] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:22:04.275923 18366 raft_consensus.cc:493] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:22:04.330268 20397 raft_consensus.cc:3060] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:04.133131 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:04.326438 20157 raft_consensus.cc:493] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:22:04.329844 17757 raft_consensus.cc:1240] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 9. Current term is 10. Ops: []
I20260709 12:22:04.365756 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 12 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:22:04.256814 21179 log.cc:927] Time spent T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.052s	user 0.000s	sys 0.000s
I20260709 12:22:04.352172 20862 raft_consensus.cc:2749] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:22:04.350827 17763 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 4.
I20260709 12:22:04.276054 21089 raft_consensus.cc:2749] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:22:04.154860 21141 raft_consensus.cc:3060] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:04.353521 17833 raft_consensus.cc:2468] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 8.
W20260709 12:22:04.353782 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 194864us
I20260709 12:22:04.374084 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:04.353426 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:04.356272 17684 raft_consensus.cc:2393] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 5: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:22:04.111706 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:04.377288 17838 raft_consensus.cc:2393] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 7: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:22:04.363113 21176 raft_consensus.cc:493] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:22:04.261884 21130 consensus_meta.cc:287] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.776s	user 0.011s	sys 0.000s
I20260709 12:22:04.358539 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:04.362329 18366 raft_consensus.cc:3060] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:03.960000 21170 raft_consensus.cc:697] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 10 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:22:04.329635 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:22:04.364205 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 287462us
I20260709 12:22:04.391760 17756 raft_consensus.cc:1240] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 4. Current term is 5. Ops: []
I20260709 12:22:04.392331 17652 leader_election.cc:304] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:04.394068 17652 leader_election.cc:304] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:22:04.340617 21183 raft_consensus.cc:493] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:04.366962 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:04.351552 20973 raft_consensus.cc:493] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:04.395557 17652 leader_election.cc:304] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:04.363353 17680 raft_consensus.cc:2468] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 6.
I20260709 12:22:04.330183 20828 raft_consensus.cc:493] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
W20260709 12:22:04.251286 21180 log.cc:927] Time spent T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.069s	user 0.000s	sys 0.000s
I20260709 12:22:04.356314 21079 raft_consensus.cc:3060] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:04.265468 21073 raft_consensus.cc:515] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:04.359725 21031 raft_consensus.cc:3060] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:04.360354 21162 raft_consensus.cc:2804] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:22:04.373169 17763 raft_consensus.cc:1275] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260709 12:22:04.360786 21092 raft_consensus.cc:740] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
W20260709 12:22:04.374169 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 76394.2us
I20260709 12:22:04.361295 21145 raft_consensus.cc:697] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 7 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:22:04.351096 21139 raft_consensus.cc:515] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:04.246147 21104 consensus_queue.cc:237] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:04.132728 21053 consensus_queue.cc:237] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.7, Last appended by leader: 7, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:04.388284 17728 leader_election.cc:304] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:04.131587 20847 consensus_queue.cc:237] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:04.371100 20983 raft_consensus.cc:493] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:04.404407 20828 raft_consensus.cc:515] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:04.420351 20983 raft_consensus.cc:515] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:04.421718 20828 leader_election.cc:290] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:04.182205 21115 consensus_queue.cc:1059] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.179s
I20260709 12:22:04.423521 20983 leader_election.cc:290] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:04.424328 18366 raft_consensus.cc:515] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:04.427292 21157 raft_consensus.cc:3055] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:22:04.427775 18366 leader_election.cc:290] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 9 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:04.404871 20973 raft_consensus.cc:515] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:04.421084 21139 leader_election.cc:290] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:04.429939 17685 raft_consensus.cc:3060] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:04.437130 20973 leader_election.cc:290] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:04.401381 21061 raft_consensus.cc:2804] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:22:04.440600 21061 raft_consensus.cc:493] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:22:04.441056 21061 raft_consensus.cc:3060] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:04.447592 21191 consensus_queue.cc:1048] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:22:04.380223 21175 consensus_queue.cc:1059] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.032s
I20260709 12:22:04.454195 21185 raft_consensus.cc:3055] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:22:04.454666 21185 raft_consensus.cc:740] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:22:04.403760 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:22:04.410511 17802 connection.cc:256] RPC call timeout handler was delayed by -2.2943918870000743s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3025s for a response
W20260709 12:22:04.456274 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 76383us
W20260709 12:22:04.456889 17728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:04.411293 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:04.399302 21089 raft_consensus.cc:2749] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:22:04.377939 20157 raft_consensus.cc:515] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:04.457667 21098 raft_consensus.cc:2749] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260709 12:22:04.456679 17802 connection.cc:256] RPC call timeout handler was delayed by -2.1282645890000822s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17520000000000002s for a response
W20260709 12:22:04.458236 17802 connection.cc:256] RPC call timeout handler was delayed by -2.1165645890000633s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1765s for a response
W20260709 12:22:04.458900 21182 log.cc:927] Time spent T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.095s	user 0.000s	sys 0.001s
I20260709 12:22:04.458549 20921 consensus_queue.cc:260] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:04.459438 20157 leader_election.cc:290] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:04.380251 21176 raft_consensus.cc:3060] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:04.461853 17755 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38786 (request call id 2589) took 1031 ms. Trace:
I20260709 12:22:04.462235 17755 rpcz_store.cc:276] 0709 12:22:03.430321 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:03.430448 (+   127us) service_pool.cc:224] Handling call
0709 12:22:04.461631 (+1031183us) spinlock_profiling.cc:243] Waited 142 ms on lock 0x7b5400415fd4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:22:04.461786 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":327147264}
I20260709 12:22:04.455477 21175 raft_consensus.cc:3055] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:22:04.463182 21175 raft_consensus.cc:740] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:22:04.465443 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:04.466630 17761 raft_consensus.cc:1240] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 5. Current term is 6. Ops: []
I20260709 12:22:04.451514 21016 raft_consensus.cc:515] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:22:04.456596 21166 log.cc:927] Time spent T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.623s	user 0.002s	sys 0.001s
W20260709 12:22:04.458571 17802 connection.cc:256] RPC call timeout handler was delayed by -2.0640583320000587s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17700000000000002s for a response
I20260709 12:22:04.472872 21135 raft_consensus.cc:3055] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 5 LEADER]: Stepping down as leader of term 5
W20260709 12:22:04.473035 17802 connection.cc:256] RPC call timeout handler was delayed by -1.9310616479997407s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1592s for a response
I20260709 12:22:04.468048 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:04.473318 21135 raft_consensus.cc:740] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:22:04.468454 20394 raft_consensus.cc:3055] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:22:04.474661 20394 raft_consensus.cc:740] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:22:04.474718 21135 consensus_queue.cc:260] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:04.475950 21135 raft_consensus.cc:3060] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:04.475610 20394 consensus_queue.cc:260] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:04.476356 21073 leader_election.cc:290] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 8 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:04.476918 20394 raft_consensus.cc:3060] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:04.479171 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
W20260709 12:22:04.473409 17802 connection.cc:256] RPC call timeout handler was delayed by -1.7707404519996999s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16970000000000002s for a response
I20260709 12:22:04.459867 20921 raft_consensus.cc:3060] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:22:04.480005 17834 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37994 (request call id 756) took 2815 ms (client timeout 1628 ms). Trace:
W20260709 12:22:04.480427 17834 rpcz_store.cc:269] 0709 12:22:01.663304 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:02.263085 (+599781us) service_pool.cc:224] Handling call
0709 12:22:04.478773 (+2215688us) spinlock_profiling.cc:243] Waited 838 ms on lock 0x7b540008a2d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:22:04.478903 (+   130us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1927401856}
W20260709 12:22:04.482443 18334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:04.483366 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 414718us
I20260709 12:22:04.484356 21154 raft_consensus.cc:2804] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:22:04.484977 21154 raft_consensus.cc:697] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 6 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:22:04.486662 21131 raft_consensus.cc:3055] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:22:04.487135 21131 raft_consensus.cc:740] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:22:04.486579 17584 catalog_manager.cc:5697] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:22:04.488471 17584 catalog_manager.cc:5697] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 7 to 10, leader changed from 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) to e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66). New cstate: current_term: 10 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:22:04.479720 17835 raft_consensus.cc:1240] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 10. Current term is 11. Ops: []
I20260709 12:22:04.489974 17584 catalog_manager.cc:5697] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 3 to 7. New cstate: current_term: 7 leader_uuid: "334a4e1f59944544b3b3ff36616b57da" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:22:04.491292 17835 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35040 (request call id 2128) took 1694 ms. Trace:
I20260709 12:22:04.491639 17835 rpcz_store.cc:276] 0709 12:22:02.796999 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:02.798281 (+  1282us) service_pool.cc:224] Handling call
0709 12:22:04.479534 (+1681253us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:04.489728 (+ 10194us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:04.491132 (+  1404us) spinlock_profiling.cc:243] Waited 1.68 s on lock 0x7b540008a2d0. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a61f8ad 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:22:04.491246 (+   114us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3864502016}
I20260709 12:22:04.493657 17686 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:54540 (request call id 31) took 1265 ms. Trace:
I20260709 12:22:04.494095 17686 rpcz_store.cc:276] 0709 12:22:03.210023 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:03.218550 (+  8527us) service_pool.cc:224] Handling call
0709 12:22:04.282450 (+1063900us) spinlock_profiling.cc:243] Waited 168 ms on lock 0x7b5400309420. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a59d520 00007f7b4a61a652 00007f7b4f5e860a 00007f7b48b6e7d4 00007f7b48b6e762 00007f7b48b6e691 00007f7b48b6e60c 00007f7b48b6d8c2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:22:04.475571 (+193121us) spinlock_profiling.cc:243] Waited 45.4 ms on lock 0x7b5400309420. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a59b5e7 00007f7b4a61ac68 00007f7b4a61a84a 00007f7b4f5e860a 00007f7b48b6e7d4 00007f7b48b6e762 00007f7b48b6e691 00007f7b48b6e60c 00007f7b48b6d8c2 00007f7b4895b48c
0709 12:22:04.475773 (+   202us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":489980672}
I20260709 12:22:04.501722 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:22:04.504087 17727 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.16.29.67:36837 timed out after 2.111s (SENT) [suppressed 15 similar messages]
I20260709 12:22:04.505445 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:04.506361 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:04.506685 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:22:04.507575 18333 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:04.507993 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:04.508771 17833 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 5.
I20260709 12:22:04.509570 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:04.509981 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:04.510231 17837 raft_consensus.cc:2368] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for earlier term 4. Current term is 5.
I20260709 12:22:04.510672 17833 raft_consensus.cc:2393] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 6: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:22:04.511198 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:04.511799 17833 raft_consensus.cc:1275] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
W20260709 12:22:04.513202 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:04.516196 17837 raft_consensus.cc:1813] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Handling vote request from an unknown peer e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:04.516759 17837 raft_consensus.cc:2393] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 6: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:22:04.455514 21185 consensus_queue.cc:260] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:04.521420 21185 raft_consensus.cc:3060] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:04.526073 17685 raft_consensus.cc:2468] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 10.
W20260709 12:22:04.527529 18334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:04.529088 18333 leader_election.cc:304] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:22:04.533656 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:04.535745 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:04.536746 18366 raft_consensus.cc:2393] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 5: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:22:04.536777 18367 raft_consensus.cc:1240] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 8. Current term is 9. Ops: [8.8-8.8]
I20260709 12:22:04.531186 17652 leader_election.cc:304] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:04.392930 21130 raft_consensus.cc:515] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:04.545220 18333 leader_election.cc:304] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:04.547816 17838 raft_consensus.cc:1240] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 6. Current term is 7. Ops: []
I20260709 12:22:04.548565 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:04.549386 17686 raft_consensus.cc:2468] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 10.
I20260709 12:22:04.397454 21132 raft_consensus.cc:515] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:04.427724 21157 raft_consensus.cc:740] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:22:04.412146 21162 raft_consensus.cc:493] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:04.554524 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:04.556366 18364 raft_consensus.cc:1240] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 8. Current term is 9. Ops: []
I20260709 12:22:04.556294 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:22:04.719642 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 179051us
I20260709 12:22:04.579056 21170 consensus_queue.cc:237] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 8.4, Last appended by leader: 4, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:04.485917 21154 consensus_queue.cc:237] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:04.685598 21085 raft_consensus.cc:493] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:04.735987 18366 raft_consensus.cc:1275] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260709 12:22:04.735817 17654 leader_election.cc:304] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:04.543685 21082 raft_consensus.cc:493] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:04.749603 18371 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59886 (request call id 993) took 1032 ms. Trace:
I20260709 12:22:04.598055 21092 consensus_queue.cc:260] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:04.587229 17801 leader_election.cc:304] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:22:04.756124 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 168974us
I20260709 12:22:04.675662 21176 raft_consensus.cc:515] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:04.638540 21098 raft_consensus.cc:493] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:04.525215 21131 consensus_queue.cc:260] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 10.9, Last appended by leader: 9, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:04.584371 20973 consensus_queue.cc:1048] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.062s
I20260709 12:22:04.594516 20862 raft_consensus.cc:493] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:04.595036 21029 consensus_queue.cc:1059] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.076s
I20260709 12:22:04.759074 21052 raft_consensus.cc:2804] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Leader election won for term 5
W20260709 12:22:04.647696 21110 consensus_meta.cc:287] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.752s	user 0.015s	sys 0.003s
I20260709 12:22:04.490741 21196 raft_consensus.cc:2749] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:22:04.574932 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:04.490525 21080 raft_consensus.cc:2749] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:22:04.571759 21060 consensus_queue.cc:1059] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 6.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.070s
I20260709 12:22:04.571467 20397 raft_consensus.cc:515] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:04.788496 21132 leader_election.cc:290] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 9 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:04.401408 21183 raft_consensus.cc:515] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:04.788519 17682 raft_consensus.cc:2393] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 12: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
W20260709 12:22:04.790215 21146 consensus_meta.cc:287] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.696s	user 0.009s	sys 0.001s
I20260709 12:22:04.790520 21208 raft_consensus.cc:493] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:04.790673 21146 raft_consensus.cc:515] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:04.557310 21056 raft_consensus.cc:2804] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:22:04.791299 21208 raft_consensus.cc:515] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:04.791828 21056 raft_consensus.cc:697] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 7 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:22:04.792572 17834 raft_consensus.cc:1275] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260709 12:22:04.792740 21056 consensus_queue.cc:237] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:04.787300 21110 raft_consensus.cc:515] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:04.793387 21208 leader_election.cc:290] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:04.794842 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:04.795292 21110 leader_election.cc:290] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 8 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:04.795444 17687 raft_consensus.cc:2468] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 5.
I20260709 12:22:04.557008 21191 raft_consensus.cc:3060] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:04.795930 21110 raft_consensus.cc:493] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:04.796381 21110 raft_consensus.cc:515] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:04.797667 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:04.741910 21085 raft_consensus.cc:515] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:04.702522 20943 raft_consensus.cc:3055] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:22:04.799942 21085 leader_election.cc:290] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:22:04.702714 21193 log.cc:927] Time spent T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.053s	user 0.000s	sys 0.000s
I20260709 12:22:04.801919 21196 raft_consensus.cc:3055] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:22:04.704838 21016 leader_election.cc:290] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 9 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:04.705158 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:04.802372 21196 raft_consensus.cc:740] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:22:04.703229 17762 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38776 (request call id 1953) took 1197 ms. Trace:
I20260709 12:22:04.803067 17762 rpcz_store.cc:276] 0709 12:22:03.505965 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:03.554313 (+ 48348us) service_pool.cc:224] Handling call
0709 12:22:04.703055 (+1148742us) spinlock_profiling.cc:243] Waited 78.8 ms on lock 0x7b54003f2854. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:22:04.703167 (+   112us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":181310208}
I20260709 12:22:04.803984 17829 raft_consensus.cc:1240] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 6. Current term is 7. Ops: []
W20260709 12:22:04.716032 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 133604us
I20260709 12:22:04.716365 21089 consensus_queue.cc:1059] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:22:04.814354 21110 leader_election.cc:290] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:04.814623 21089 raft_consensus.cc:3055] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:22:04.815109 21089 raft_consensus.cc:740] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
W20260709 12:22:04.717509 21169 log.cc:927] Time spent T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.782s	user 0.003s	sys 0.000s
I20260709 12:22:04.584758 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
W20260709 12:22:04.821405 17655 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:04.458573 21115 raft_consensus.cc:3055] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:22:04.556954 21184 consensus_queue.cc:1059] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.085s
I20260709 12:22:04.582641 17802 leader_election.cc:304] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
W20260709 12:22:04.825919 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 243339us
I20260709 12:22:04.788913 20397 leader_election.cc:290] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 12 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:04.757107 21082 raft_consensus.cc:515] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:04.588826 18333 leader_election.cc:304] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:22:04.831372 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 242648us
I20260709 12:22:04.831640 21146 leader_election.cc:290] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 8 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:04.832211 18333 leader_election.cc:400] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 7 election: Vote denied by peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651) with higher term. Message: Invalid argument: T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 7. Current term is 8.
I20260709 12:22:04.832446 21130 leader_election.cc:290] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 6 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:04.833154 21130 raft_consensus.cc:2749] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:22:04.833469 18333 leader_election.cc:304] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:04.835052 18333 leader_election.cc:304] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:04.804970 21092 raft_consensus.cc:3060] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:04.837867 18333 leader_election.cc:304] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:04.722463 20921 raft_consensus.cc:493] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:04.728570 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:04.728384 18334 leader_election.cc:304] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
W20260709 12:22:04.738425 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 155019us
W20260709 12:22:04.842808 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 114584us
I20260709 12:22:04.843088 18368 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59902 (request call id 939) took 1240 ms. Trace:
I20260709 12:22:04.740365 17654 leader_election.cc:304] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:22:04.742306 18364 raft_consensus.cc:1240] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 10. Current term is 11. Ops: [10.9-10.9]
I20260709 12:22:04.584749 21147 consensus_queue.cc:1059] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 8.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.064s
I20260709 12:22:04.671583 21145 consensus_queue.cc:237] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:04.750029 18371 rpcz_store.cc:276] 0709 12:22:03.716886 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:03.731184 (+ 14298us) service_pool.cc:224] Handling call
0709 12:22:03.731482 (+   298us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:03.745943 (+ 14461us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:03.745993 (+    50us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:04.284132 (+538139us) spinlock_profiling.cc:243] Waited 1.36 ms on lock 0x7b2c000123a8. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b48f19a11 00007f7b48f29372 00007f7b4a65ba01 00007f7b4a62043a 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2
0709 12:22:04.284562 (+   430us) log.cc:844] Serialized 26 byte log entry
0709 12:22:04.372496 (+ 87934us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:22:04.388981 (+ 16485us) log.cc:844] Serialized 14 byte log entry
0709 12:22:04.486786 (+ 97805us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:04.486866 (+    80us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:04.749282 (+262416us) raft_consensus.cc:1685] finished
0709 12:22:04.749392 (+   110us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:04.749539 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":115138,"prepare.queue_time_us":6663,"prepare.run_cpu_time_us":71,"prepare.run_wall_time_us":409,"raft.queue_time_us":112409,"raft.run_cpu_time_us":163,"raft.run_wall_time_us":163,"spinlock_wait_cycles":3132288,"thread_start_us":84421,"threads_started":1,"wal-append.queue_time_us":257496}
W20260709 12:22:04.758561 21197 log.cc:927] Time spent T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.050s	user 0.000s	sys 0.000s
I20260709 12:22:04.759011 21061 raft_consensus.cc:515] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:04.849720 21132 raft_consensus.cc:2749] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:22:04.851188 21061 leader_election.cc:290] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 9 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:22:04.760185 21189 log.cc:927] Time spent T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.164s	user 0.000s	sys 0.001s
I20260709 12:22:04.743584 21093 raft_consensus.cc:2749] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 12:22:04.852461 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 12 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:04.853085 18371 raft_consensus.cc:3060] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:04.583348 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:04.769822 20862 raft_consensus.cc:515] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:04.770789 21131 raft_consensus.cc:3060] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:04.770900 21176 leader_election.cc:290] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 12 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:04.842058 21141 raft_consensus.cc:2749] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:22:04.766150 21098 raft_consensus.cc:515] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:04.772792 20157 consensus_queue.cc:1048] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260709 12:22:04.773399 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:04.774084 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:04.669545 21157 consensus_queue.cc:260] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:04.785590 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:04.786005 21052 raft_consensus.cc:697] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 5 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:22:04.865193 21098 leader_election.cc:290] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:04.864981 21052 consensus_queue.cc:237] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:04.790655 20980 raft_consensus.cc:3060] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:04.559284 21155 leader_election.cc:290] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 10 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:04.571926 21162 raft_consensus.cc:3060] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:04.800046 20943 raft_consensus.cc:740] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 4 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:22:04.800241 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:04.706357 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:04.711099 17832 raft_consensus.cc:2468] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 8.
I20260709 12:22:04.797699 17756 raft_consensus.cc:1240] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 7. Current term is 8. Ops: []
I20260709 12:22:04.804713 17652 leader_election.cc:304] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da
W20260709 12:22:04.811496 21198 log.cc:927] Time spent T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.111s	user 0.001s	sys 0.000s
I20260709 12:22:04.790627 21183 leader_election.cc:290] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:22:04.813654 21156 consensus_meta.cc:287] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.532s	user 0.009s	sys 0.000s
I20260709 12:22:04.820681 17829 raft_consensus.cc:1240] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 5. Current term is 6. Ops: []
I20260709 12:22:04.821412 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: true dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:04.881995 18371 raft_consensus.cc:2468] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 12.
I20260709 12:22:04.828684 17802 leader_election.cc:304] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
W20260709 12:22:04.885030 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 56381.9us
I20260709 12:22:04.558568 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:04.826164 21196 consensus_queue.cc:260] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:22:04.844228 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 103998us
I20260709 12:22:04.889484 17654 leader_election.cc:304] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:04.844256 18334 leader_election.cc:304] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:22:04.890995 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 46823.4us
I20260709 12:22:04.891551 18334 leader_election.cc:304] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:22:04.845700 17685 raft_consensus.cc:2468] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 5.
I20260709 12:22:04.845795 18374 raft_consensus.cc:1240] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 5. Current term is 6. Ops: []
I20260709 12:22:04.845634 18333 leader_election.cc:304] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:04.848362 18362 raft_consensus.cc:1240] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 9. Current term is 10. Ops: []
I20260709 12:22:04.816097 21089 consensus_queue.cc:260] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:04.823712 21115 raft_consensus.cc:740] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:22:04.853611 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:04.897570 17688 raft_consensus.cc:2468] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 6.
I20260709 12:22:04.829355 21054 raft_consensus.cc:2749] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:22:04.843437 18368 rpcz_store.cc:276] 0709 12:22:03.602225 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:03.606974 (+  4749us) service_pool.cc:224] Handling call
0709 12:22:03.607296 (+   322us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:03.607632 (+   336us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:03.607663 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:03.608170 (+   507us) log.cc:844] Serialized 26 byte log entry
0709 12:22:03.768968 (+160798us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:22:03.769029 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:03.769127 (+    98us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:04.716939 (+947812us) spinlock_profiling.cc:243] Waited 8.17 ms on lock 0x7b4400022b4c. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b47459c85 00007f7b4745a6a2 00007f7b4a1cc1db 00007f7b4a1c7aaf 00007f7b4a1bb86e 00007f7b4a1babb6 00007f7b4a1ba7d8 00007f7b4a1c4221 00007f7b4a1c41d9 00007f7b4a1c4169
0709 12:22:04.840182 (+123243us) raft_consensus.cc:1685] finished
0709 12:22:04.840197 (+    15us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:04.843030 (+  2833us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":5640,"raft.run_cpu_time_us":49,"raft.run_wall_time_us":49,"spinlock_wait_cycles":18797440,"thread_start_us":160362,"threads_started":1,"wal-append.queue_time_us":254363}
I20260709 12:22:04.854148 17686 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 5.
I20260709 12:22:04.849167 20921 raft_consensus.cc:515] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:04.863970 21157 raft_consensus.cc:3060] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:04.857657 17729 leader_election.cc:304] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:04.862095 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:04.846812 21130 raft_consensus.cc:2804] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:22:04.924082 21130 raft_consensus.cc:493] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:04.924357 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: true dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:04.924564 21130 raft_consensus.cc:3060] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:04.927459 21191 raft_consensus.cc:2749] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:22:04.928143 21107 raft_consensus.cc:3060] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:04.929440 21184 raft_consensus.cc:3055] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:22:04.929948 21184 raft_consensus.cc:740] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:22:04.932186 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:04.932896 17755 raft_consensus.cc:2468] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 5.
W20260709 12:22:04.874709 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 70077us
I20260709 12:22:04.853896 21085 raft_consensus.cc:2804] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:22:04.857172 20397 raft_consensus.cc:2804] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:22:04.937177 17584 catalog_manager.cc:5697] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:22:04.878616 21175 consensus_queue.cc:260] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:04.882182 17831 raft_consensus.cc:3060] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:04.583978 20776 raft_consensus.cc:2749] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260709 12:22:04.883915 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:04.855701 20862 leader_election.cc:290] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:04.891666 17802 leader_election.cc:304] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:04.894014 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:22:04.894624 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 49068.5us
I20260709 12:22:04.898648 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 12 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:04.901331 21089 raft_consensus.cc:3060] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:04.901149 21088 consensus_queue.cc:1059] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 8.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
W20260709 12:22:04.901834 21190 consensus_queue.cc:1175] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "e063541542a94969898868bf6eb5ce1b" responder_term: 8 status { last_received { term: 5 index: 4 } last_committed_idx: 4 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 7. Current term is 8. Ops: [7.5-7.5]" } } last_received_current_leader { term: 0 index: 0 } }
I20260709 12:22:04.902882 21060 consensus_queue.cc:1048] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:22:04.909055 18334 leader_election.cc:304] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:04.909734 21056 raft_consensus.cc:2749] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:22:04.936573 21147 raft_consensus.cc:2804] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:22:04.914088 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
W20260709 12:22:04.914192 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 56650.4us
I20260709 12:22:04.918653 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:04.920411 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:04.893587 21183 raft_consensus.cc:2749] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 12:22:04.922199 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:05.009060 17757 raft_consensus.cc:2393] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 8: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:22:04.922796 20921 leader_election.cc:290] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:04.880327 21093 raft_consensus.cc:2804] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:22:04.923025 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:04.936115 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:22:04.938479 17728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:04.939251 17801 leader_election.cc:304] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:05.015355 17761 raft_consensus.cc:3060] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:22:05.015900 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 76713.3us
I20260709 12:22:04.940747 17683 raft_consensus.cc:2468] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 6.
I20260709 12:22:05.018409 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 12 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
W20260709 12:22:04.948705 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 57082.4us
I20260709 12:22:05.022553 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 7 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:05.024102 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:05.025053 17759 raft_consensus.cc:2393] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 6: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:22:05.025345 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:05.026096 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:05.026515 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:04.950232 20397 raft_consensus.cc:697] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 9 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:22:04.951565 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:05.029330 17727 leader_election.cc:304] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
W20260709 12:22:04.952289 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:04.896081 21155 raft_consensus.cc:2804] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:22:04.920473 21141 raft_consensus.cc:2749] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:22:05.034085 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
W20260709 12:22:04.964669 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 55706.9us
I20260709 12:22:05.034953 18334 leader_election.cc:304] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:05.035169 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:05.036283 20921 raft_consensus.cc:2804] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:22:04.981053 21088 consensus_queue.cc:1059] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:22:04.984730 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:05.036984 20921 raft_consensus.cc:493] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:05.037351 17829 raft_consensus.cc:2468] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 7.
I20260709 12:22:05.037525 20921 raft_consensus.cc:3060] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:05.038461 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:04.986447 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:04.987154 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:04.914439 18365 raft_consensus.cc:2410] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for term 7 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260709 12:22:04.983800 17831 raft_consensus.cc:2468] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 9.
I20260709 12:22:04.986114 21080 consensus_queue.cc:1059] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 9.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.058s
I20260709 12:22:05.044642 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:04.987779 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:05.045224 21080 raft_consensus.cc:2804] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:22:04.989260 18364 raft_consensus.cc:1240] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 7. Current term is 8. Ops: [4.6-7.7]
I20260709 12:22:05.045809 21080 raft_consensus.cc:493] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:05.046312 21080 raft_consensus.cc:3060] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:05.046890 17688 raft_consensus.cc:3055] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:22:05.047260 17688 raft_consensus.cc:740] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:22:04.990717 18366 raft_consensus.cc:1240] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 5 LEADER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 4. Current term is 5. Ops: []
W20260709 12:22:05.051014 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:05.054293 21073 raft_consensus.cc:493] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:05.054858 21073 raft_consensus.cc:515] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:05.057519 21073 leader_election.cc:290] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:05.058341 21093 raft_consensus.cc:697] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 7 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:22:04.990381 18333 leader_election.cc:304] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:22:04.992903 17836 raft_consensus.cc:1240] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 5. Current term is 6. Ops: []
I20260709 12:22:04.993393 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:22:05.061915 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 71601.9us
I20260709 12:22:04.952828 21196 raft_consensus.cc:3060] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:04.993400 20943 consensus_queue.cc:260] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:04.939980 21085 raft_consensus.cc:493] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:04.997856 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:04.998736 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 7 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:05.079716 17838 raft_consensus.cc:2410] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for term 7 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 12:22:04.999293 20984 raft_consensus.cc:2804] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:22:05.000320 17835 raft_consensus.cc:1240] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 5. Current term is 6. Ops: []
I20260709 12:22:05.000941 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:04.988185 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:04.983716 21135 consensus_queue.cc:1059] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 5.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.019s
I20260709 12:22:04.889673 21156 raft_consensus.cc:515] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:05.005647 21132 consensus_queue.cc:1048] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:22:05.008442 17686 raft_consensus.cc:2393] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 8: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
W20260709 12:22:05.033290 17655 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:05.033793 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:05.036475 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 12 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:05.090930 17683 raft_consensus.cc:2468] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 9.
I20260709 12:22:05.092152 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:05.036644 18362 raft_consensus.cc:2393] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 12: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
W20260709 12:22:04.982265 17803 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:05.063282 18333 leader_election.cc:304] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:05.028540 17755 raft_consensus.cc:2468] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 7.
I20260709 12:22:04.949553 20941 raft_consensus.cc:2749] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:22:04.982681 21145 consensus_queue.cc:1059] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:22:05.036340 18334 leader_election.cc:304] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:22:05.111537 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 75269.8us
W20260709 12:22:05.066165 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 48687.1us
I20260709 12:22:05.039865 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:05.069036 17729 leader_election.cc:304] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:22:05.031590 17727 leader_election.cc:304] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:22:05.072697 18364 raft_consensus.cc:1240] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 7. Current term is 8. Ops: []
I20260709 12:22:05.067097 17688 consensus_queue.cc:260] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:22:05.117900 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 49065.9us
I20260709 12:22:04.933900 21184 consensus_queue.cc:260] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.7, Last appended by leader: 7, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:04.998096 21115 consensus_queue.cc:260] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:05.088032 20911 raft_consensus.cc:2749] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:22:05.100116 17761 raft_consensus.cc:2468] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 9.
I20260709 12:22:05.100868 17762 raft_consensus.cc:2393] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 12: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:22:05.108345 17728 leader_election.cc:304] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:05.060786 21155 raft_consensus.cc:493] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:05.132670 17830 raft_consensus.cc:1240] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 9 LEADER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 8. Current term is 9. Ops: []
W20260709 12:22:05.125154 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 93648.3us
I20260709 12:22:05.071066 21056 raft_consensus.cc:2749] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:22:05.083827 20984 raft_consensus.cc:493] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
W20260709 12:22:05.134768 17652 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:05.135547 17652 leader_election.cc:304] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:22:05.138511 17757 raft_consensus.cc:1240] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 8. Current term is 9. Ops: [6.5-8.6]
I20260709 12:22:05.120345 20943 raft_consensus.cc:3060] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:22:05.109390 17803 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:05.120007 21031 raft_consensus.cc:2749] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:22:05.127193 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:05.127763 17688 raft_consensus.cc:3060] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:05.153030 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:05.080868 21085 raft_consensus.cc:3060] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:05.082552 21141 raft_consensus.cc:3055] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:22:05.129194 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:05.082332 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:05.130436 17681 raft_consensus.cc:2393] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 6: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:22:05.110138 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:05.162029 17686 raft_consensus.cc:2376] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 10. Re-sending same reply.
I20260709 12:22:05.163252 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:05.139788 17757 raft_consensus.cc:1275] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260709 12:22:05.141355 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:05.107676 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:22:05.126471 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 63296.1us
I20260709 12:22:05.165130 17836 raft_consensus.cc:2468] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 4.
I20260709 12:22:05.142746 18334 leader_election.cc:304] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:05.144060 17727 leader_election.cc:400] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 4 election: Vote denied by peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837) with higher term. Message: Invalid argument: T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for earlier term 4. Current term is 5.
I20260709 12:22:05.166519 17727 leader_election.cc:304] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:22:05.146857 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:22:05.167549 18334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:05.059641 21093 consensus_queue.cc:237] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:05.150257 18368 raft_consensus.cc:1275] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260709 12:22:05.150848 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:05.154927 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:05.155509 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:05.127530 20776 raft_consensus.cc:2749] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:22:05.077806 17802 leader_election.cc:304] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:05.130023 21079 raft_consensus.cc:2749] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:22:05.158025 17762 raft_consensus.cc:1240] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 4. Current term is 5. Ops: []
I20260709 12:22:05.161677 17801 leader_election.cc:304] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:05.173067 17760 raft_consensus.cc:2393] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 10: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
W20260709 12:22:05.246649 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 85079.6us
I20260709 12:22:05.247511 17801 leader_election.cc:304] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:05.172771 20941 raft_consensus.cc:493] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:05.248373 21016 raft_consensus.cc:2804] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:22:05.248255 20941 raft_consensus.cc:515] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:05.249083 21016 raft_consensus.cc:697] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 9 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:22:05.249401 21053 raft_consensus.cc:2804] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:22:05.249855 21053 raft_consensus.cc:493] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:05.250203 21053 raft_consensus.cc:3060] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:05.250574 20941 leader_election.cc:290] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:05.251112 20941 raft_consensus.cc:493] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:05.136397 20984 raft_consensus.cc:3060] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:05.245239 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:05.255635 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:05.256733 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:05.257287 17829 raft_consensus.cc:2468] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 5.
I20260709 12:22:05.258529 17829 raft_consensus.cc:1275] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 3. (term mismatch)
I20260709 12:22:05.258831 17829 pending_rounds.cc:85] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b: Aborting all ops after (but not including) 2
I20260709 12:22:05.259047 17829 pending_rounds.cc:107] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b: Aborting uncommitted NO_OP operation due to leader change: 5.3
I20260709 12:22:05.259272 17829 raft_consensus.cc:2889] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:05.246541 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:05.134277 21155 raft_consensus.cc:3060] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:05.250262 21016 consensus_queue.cc:237] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.6, Last appended by leader: 6, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:05.261179 20941 raft_consensus.cc:515] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:05.160854 21141 raft_consensus.cc:740] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:22:05.173561 17652 leader_election.cc:304] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:22:05.276808 17680 raft_consensus.cc:2468] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 6.
I20260709 12:22:05.156215 21184 raft_consensus.cc:3060] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:05.131570 21211 consensus_queue.cc:1059] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.044s
I20260709 12:22:05.278328 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:05.280405 17761 raft_consensus.cc:2468] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 7.
I20260709 12:22:05.136622 21029 raft_consensus.cc:2804] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Leader election won for term 12
W20260709 12:22:05.281466 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 203746us
I20260709 12:22:05.298149 17755 raft_consensus.cc:1275] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
W20260709 12:22:05.323591 17655 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:05.353482 17802 leader_election.cc:304] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:05.355820 17688 raft_consensus.cc:2468] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 9.
I20260709 12:22:05.163570 21115 raft_consensus.cc:3060] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:22:05.440876 17654 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:05.441519 17654 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:05.442471 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 268957us
I20260709 12:22:05.443248 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:05.444025 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:05.444103 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:05.445338 17683 raft_consensus.cc:3060] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:05.244280 21132 raft_consensus.cc:2804] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:22:05.274309 21190 raft_consensus.cc:2804] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:22:05.271330 20941 leader_election.cc:290] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:05.153767 21177 consensus_queue.cc:1059] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.207s
I20260709 12:22:05.296867 20157 raft_consensus.cc:2804] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:22:05.449630 17680 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad"
dest_uuid: "0da169ff82674dc1947055ce43de68a3"
mode: GRACEFUL
new_leader_uuid: "334a4e1f59944544b3b3ff36616b57da"
 from {username='slave'} at 127.0.0.1:54540
W20260709 12:22:05.441601 21146 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 9 is higher.
I20260709 12:22:05.463053 17689 raft_consensus.cc:2410] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 8 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 4.
I20260709 12:22:05.465643 17680 raft_consensus.cc:606] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 6 LEADER]: Received request to transfer leadership to TS 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:05.357965 20847 raft_consensus.cc:493] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:05.497684 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:05.544139 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 79803.9us
I20260709 12:22:05.555940 17652 leader_election.cc:304] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:05.447152 21088 raft_consensus.cc:2749] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:22:05.565168 17652 leader_election.cc:304] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:05.450860 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:05.452983 21029 raft_consensus.cc:697] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
W20260709 12:22:05.563282 21092 consensus_meta.cc:287] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.721s	user 0.015s	sys 0.001s
I20260709 12:22:05.448165 21207 raft_consensus.cc:2804] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:22:05.457226 21190 raft_consensus.cc:493] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:05.155985 20397 consensus_queue.cc:237] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:05.469566 21079 raft_consensus.cc:493] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:05.571000 17652 leader_election.cc:304] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:05.464538 20157 raft_consensus.cc:493] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
W20260709 12:22:05.577454 20980 consensus_meta.cc:287] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.709s	user 0.011s	sys 0.000s
I20260709 12:22:05.589712 17755 raft_consensus.cc:1275] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260709 12:22:05.452515 21132 raft_consensus.cc:493] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:05.586601 17652 leader_election.cc:304] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:05.552675 20941 raft_consensus.cc:2804] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:22:05.584719 21092 raft_consensus.cc:2749] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:22:05.593004 17652 leader_election.cc:304] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:05.597707 18371 raft_consensus.cc:2468] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 5.
I20260709 12:22:05.598047 17652 leader_election.cc:304] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:22:05.594972 17684 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.501s	user 0.100s	sys 0.133s
W20260709 12:22:05.599555 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 246135us
I20260709 12:22:05.595582 17683 raft_consensus.cc:2468] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 9.
I20260709 12:22:05.602149 17802 leader_election.cc:304] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:05.604004 17802 leader_election.cc:304] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:05.606936 17802 leader_election.cc:304] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:05.582372 21190 raft_consensus.cc:3060] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:05.570741 20984 raft_consensus.cc:515] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:05.621577 17761 raft_consensus.cc:1275] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260709 12:22:05.591749 20157 raft_consensus.cc:3060] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:05.628062 21178 raft_consensus.cc:2749] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260709 12:22:05.621790 21154 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.835s	user 0.045s	sys 0.064s
I20260709 12:22:05.565583 20847 raft_consensus.cc:515] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:05.593982 21132 raft_consensus.cc:3060] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:05.594318 21110 raft_consensus.cc:2749] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 12:22:05.632828 21205 raft_consensus.cc:2804] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260709 12:22:05.625453 21107 consensus_meta.cc:287] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.697s	user 0.007s	sys 0.004s
W20260709 12:22:05.632254 21157 consensus_meta.cc:287] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.716s	user 0.011s	sys 0.000s
I20260709 12:22:05.652232 21234 raft_consensus.cc:493] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:22:05.642467 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:05.652632 21110 raft_consensus.cc:3055] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:22:05.652951 17755 raft_consensus.cc:2468] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 6.
I20260709 12:22:05.653344 21110 raft_consensus.cc:740] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 9 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:22:05.626655 20941 raft_consensus.cc:697] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 6 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:22:05.656703 20941 consensus_queue.cc:237] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:05.639461 21029 consensus_queue.cc:237] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 10.11, Last appended by leader: 11, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:05.664079 21107 raft_consensus.cc:2749] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:22:05.664829 21082 leader_election.cc:290] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:05.665572 17687 raft_consensus.cc:1813] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Handling vote request from an unknown peer e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:05.666204 17687 raft_consensus.cc:2410] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for term 6 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20260709 12:22:05.667169 21155 raft_consensus.cc:515] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:05.668084 21230 raft_consensus.cc:493] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:05.668588 21230 raft_consensus.cc:515] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:22:05.669250 21222 log.cc:927] Time spent T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.086s	user 0.000s	sys 0.000s
I20260709 12:22:05.670567 17756 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38786 (request call id 2598) took 1600 ms. Trace:
I20260709 12:22:05.670747 21230 leader_election.cc:290] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:05.670980 17756 rpcz_store.cc:276] 0709 12:22:04.069821 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:04.392837 (+323016us) service_pool.cc:224] Handling call
0709 12:22:04.405778 (+ 12941us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:04.874803 (+469025us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:05.670371 (+795568us) spinlock_profiling.cc:243] Waited 392 ms on lock 0x7b540019f0d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:22:05.670505 (+   134us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":900933376}
I20260709 12:22:05.672796 20847 leader_election.cc:290] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:05.644166 21104 raft_consensus.cc:2749] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 12:22:05.639528 21228 raft_consensus.cc:3055] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:22:05.589731 21079 raft_consensus.cc:515] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:05.635980 21092 raft_consensus.cc:2749] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:22:05.654403 21110 consensus_queue.cc:260] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:05.689287 17687 raft_consensus.cc:1275] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20260709 12:22:05.689292 21110 raft_consensus.cc:3060] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:05.690939 17687 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37402 (request call id 2402) took 1284 ms. Trace:
W20260709 12:22:05.692459 21218 log.cc:927] Time spent T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.135s	user 0.000s	sys 0.001s
I20260709 12:22:05.693514 21227 raft_consensus.cc:493] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:05.694077 21227 raft_consensus.cc:515] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:22:05.695465 20943 consensus_meta.cc:287] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.539s	user 0.007s	sys 0.003s
I20260709 12:22:05.696244 17834 raft_consensus.cc:2468] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 5.
I20260709 12:22:05.651751 21205 raft_consensus.cc:493] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:05.697474 21205 raft_consensus.cc:3060] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:05.691687 17687 rpcz_store.cc:276] 0709 12:22:04.406457 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:05.687434 (+1280977us) service_pool.cc:224] Handling call
0709 12:22:05.687776 (+   342us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:05.689836 (+  2060us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:05.690645 (+   809us) spinlock_profiling.cc:243] Waited 1.1 ms on lock 0x7b5400206f54. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:22:05.690870 (+   225us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2519936}
I20260709 12:22:05.700487 17687 raft_consensus.cc:1275] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260709 12:22:05.652760 21234 raft_consensus.cc:515] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:05.638024 21061 raft_consensus.cc:2749] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 12:22:05.676496 17832 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35042 (request call id 1885) took 1170 ms. Trace:
I20260709 12:22:05.636746 21145 raft_consensus.cc:2749] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:22:05.678148 17837 raft_consensus.cc:1275] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260709 12:22:05.683315 17759 raft_consensus.cc:2468] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 9.
W20260709 12:22:05.647295 21154 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.861s	user 0.045s	sys 0.064s
W20260709 12:22:05.619462 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 452995us
I20260709 12:22:05.757591 17727 leader_election.cc:304] T 7cb3768d215e409fb9f5baf18d124442 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:22:05.647468 21053 raft_consensus.cc:515] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:22:05.645696 17730 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:05.684271 21228 raft_consensus.cc:740] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:22:05.687220 17584 catalog_manager.cc:5697] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 5 to 7. New cstate: current_term: 7 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:22:05.705592 20157 raft_consensus.cc:515] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:05.643082 21156 leader_election.cc:290] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 6 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:22:05.707369 17803 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:05.647533 21089 raft_consensus.cc:2749] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 12:22:05.709513 17687 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37402 (request call id 2403) took 1293 ms. Trace:
I20260709 12:22:05.770613 17687 rpcz_store.cc:276] 0709 12:22:04.416354 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:05.699577 (+1283223us) service_pool.cc:224] Handling call
0709 12:22:05.700066 (+   489us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:05.702838 (+  2772us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:05.709428 (+  6590us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:05.711692 17832 rpcz_store.cc:276] 0709 12:22:04.506027 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:04.506139 (+   112us) service_pool.cc:224] Handling call
0709 12:22:05.676323 (+1170184us) spinlock_profiling.cc:243] Waited 197 ms on lock 0x7b540028d054. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:22:05.676442 (+   119us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":453943168}
W20260709 12:22:05.710429 21223 log.cc:927] Time spent T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.061s	user 0.000s	sys 0.000s
W20260709 12:22:05.717185 21162 consensus_meta.cc:287] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.846s	user 0.006s	sys 0.004s
I20260709 12:22:05.719738 18374 raft_consensus.cc:1240] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 6. Current term is 7. Ops: []
I20260709 12:22:05.651989 21178 raft_consensus.cc:493] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:05.707149 21080 consensus_meta.cc:287] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.660s	user 0.012s	sys 0.000s
I20260709 12:22:05.786890 21178 raft_consensus.cc:515] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:05.787420 21080 raft_consensus.cc:515] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:05.790318 21080 leader_election.cc:290] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 6 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:05.791165 18368 raft_consensus.cc:1240] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 5. Current term is 6. Ops: []
I20260709 12:22:05.791340 21178 leader_election.cc:290] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:22:05.671079 21221 log.cc:927] Time spent T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.209s	user 0.000s	sys 0.001s
W20260709 12:22:05.711091 20921 consensus_meta.cc:287] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.673s	user 0.011s	sys 0.000s
I20260709 12:22:05.769500 18363 raft_consensus.cc:1240] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 6 LEADER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 5. Current term is 6. Ops: []
I20260709 12:22:05.770329 21230 consensus_queue.cc:1059] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.005s
I20260709 12:22:05.776294 17687 raft_consensus.cc:1275] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20260709 12:22:05.809311 17687 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37396 (request call id 2028) took 1083 ms. Trace:
I20260709 12:22:05.809757 17687 rpcz_store.cc:276] 0709 12:22:04.726164 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:05.775468 (+1049304us) service_pool.cc:224] Handling call
0709 12:22:05.775863 (+   395us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:05.808903 (+ 33040us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:05.809230 (+   327us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:22:05.779009 18334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:05.771593 21227 leader_election.cc:290] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:05.785591 21162 raft_consensus.cc:515] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:05.768370 21115 raft_consensus.cc:3060] T 7cb3768d215e409fb9f5baf18d124442 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:05.807739 17727 leader_election.cc:304] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:22:05.816107 20157 leader_election.cc:290] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 7 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:05.773761 17584 catalog_manager.cc:5697] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 4 to 5, leader changed from e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66) to 334a4e1f59944544b3b3ff36616b57da (127.16.29.68). New cstate: current_term: 5 leader_uuid: "334a4e1f59944544b3b3ff36616b57da" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:22:05.841131 17687 raft_consensus.cc:1275] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 7 index: 9. (index mismatch)
I20260709 12:22:05.839666 17684 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44898 (request call id 731) took 1705 ms. Trace:
I20260709 12:22:05.842022 17684 rpcz_store.cc:276] 0709 12:22:04.134366 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:04.881112 (+746746us) service_pool.cc:224] Handling call
0709 12:22:04.899045 (+ 17933us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:04.899315 (+   270us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 12:22:05.093130 (+193815us) log.cc:844] Serialized 14 byte log entry
0709 12:22:05.839246 (+746116us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:05.839306 (+    60us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:22:05.839353 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:05.839414 (+    61us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:05.839605 (+   191us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":667322,"prepare.run_cpu_time_us":63,"prepare.run_wall_time_us":2215,"raft.queue_time_us":53330,"raft.run_cpu_time_us":84,"raft.run_wall_time_us":83,"thread_start_us":914706,"threads_started":3,"wal-append.queue_time_us":716982}
I20260709 12:22:05.852026 17584 catalog_manager.cc:5697] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 3 to 7, leader changed from e063541542a94969898868bf6eb5ce1b (127.16.29.67) to 334a4e1f59944544b3b3ff36616b57da (127.16.29.68). New cstate: current_term: 7 leader_uuid: "334a4e1f59944544b3b3ff36616b57da" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:22:05.854135 17687 raft_consensus.cc:1240] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 8. Current term is 9. Ops: []
I20260709 12:22:05.855562 17832 raft_consensus.cc:1240] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 9 LEADER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 8. Current term is 9. Ops: []
I20260709 12:22:05.856488 20157 raft_consensus.cc:493] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
W20260709 12:22:05.857918 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 259930us
W20260709 12:22:05.859872 17682 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37396 (request call id 2024) took 1728 ms (client timeout 1991 ms). Trace:
W20260709 12:22:05.861510 17685 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37402 (request call id 2392) took 1618 ms (client timeout 2066 ms). Trace:
W20260709 12:22:05.862056 17685 rpcz_store.cc:269] 0709 12:22:04.243269 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:04.704895 (+461626us) service_pool.cc:224] Handling call
0709 12:22:05.861118 (+1156223us) spinlock_profiling.cc:243] Waited 13.2 ms on lock 0x7b54003471d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:22:05.861418 (+   300us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":30265728}
I20260709 12:22:05.838234 21080 raft_consensus.cc:493] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:05.860500 17682 rpcz_store.cc:269] 0709 12:22:04.131689 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:04.155077 (+ 23388us) service_pool.cc:224] Handling call
0709 12:22:05.859488 (+1704411us) spinlock_profiling.cc:243] Waited 506 ms on lock 0x7b5400208354. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:22:05.859786 (+   298us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1163877632}
I20260709 12:22:05.869167 20157 raft_consensus.cc:515] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:05.860321 17652 leader_election.cc:304] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:05.815069 20921 raft_consensus.cc:515] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:05.846517 21205 raft_consensus.cc:515] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:05.859414 17584 catalog_manager.cc:5697] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
W20260709 12:22:05.852573 21085 consensus_meta.cc:287] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.578s	user 0.005s	sys 0.005s
I20260709 12:22:05.848250 21088 raft_consensus.cc:3060] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:22:05.875444 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 107720us
I20260709 12:22:05.876739 17685 raft_consensus.cc:1275] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260709 12:22:05.894183 17727 leader_election.cc:304] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:22:05.894644 17685 raft_consensus.cc:1240] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 7. Current term is 8. Ops: []
I20260709 12:22:05.839267 20846 raft_consensus.cc:493] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:05.839051 21178 raft_consensus.cc:493] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
W20260709 12:22:05.855437 17802 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:05.887137 17652 leader_election.cc:304] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:22:05.888226 20157 leader_election.cc:290] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:05.888718 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:05.899538 17763 raft_consensus.cc:3060] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:22:05.874871 21130 consensus_meta.cc:287] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.950s	user 0.011s	sys 0.000s
W20260709 12:22:05.906812 21216 log.cc:927] Time spent T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 1.095s	user 0.003s	sys 0.000s
I20260709 12:22:05.906715 21130 raft_consensus.cc:515] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:05.907401 21089 consensus_queue.cc:1048] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 12:22:05.908977 21130 leader_election.cc:290] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 6 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:05.889130 21239 raft_consensus.cc:2749] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:22:05.909947 21239 raft_consensus.cc:2804] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:22:05.910295 21239 raft_consensus.cc:493] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:05.910599 21239 raft_consensus.cc:3060] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:05.890493 17584 catalog_manager.cc:5697] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 2 to 6, leader changed from 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) to e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66). New cstate: current_term: 6 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:22:05.912154 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:05.901865 17682 raft_consensus.cc:1240] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 7. Current term is 8. Ops: []
I20260709 12:22:05.901528 21061 consensus_queue.cc:1048] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.000s
W20260709 12:22:05.902284 17583 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35966 (request call id 353) took 27039 ms (client timeout 15000 ms). Trace:
I20260709 12:22:05.902683 21162 leader_election.cc:290] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 9 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:05.902350 20846 raft_consensus.cc:515] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:05.903514 17830 raft_consensus.cc:1240] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 5. Current term is 6. Ops: []
I20260709 12:22:05.903934 21178 raft_consensus.cc:515] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:05.905835 21147 raft_consensus.cc:493] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:05.898643 21073 consensus_queue.cc:1059] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.011s
W20260709 12:22:05.901997 21215 log.cc:927] Time spent T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 1.095s	user 0.000s	sys 0.003s
I20260709 12:22:05.877564 21080 raft_consensus.cc:515] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:05.917825 17763 raft_consensus.cc:2468] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 6.
I20260709 12:22:05.917037 17584 catalog_manager.cc:5697] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 2 to 6, leader changed from e063541542a94969898868bf6eb5ce1b (127.16.29.67) to 334a4e1f59944544b3b3ff36616b57da (127.16.29.68). New cstate: current_term: 6 leader_uuid: "334a4e1f59944544b3b3ff36616b57da" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:22:05.920321 17756 raft_consensus.cc:1275] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260709 12:22:05.922013 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:05.938285 17764 raft_consensus.cc:2468] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 8.
I20260709 12:22:05.847616 21228 consensus_queue.cc:260] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:05.885615 21240 raft_consensus.cc:2749] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:22:05.926975 17755 raft_consensus.cc:1275] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 4. (term mismatch)
I20260709 12:22:05.931754 20157 consensus_queue.cc:1048] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:22:05.917040 17727 leader_election.cc:304] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:22:05.895653 20984 leader_election.cc:290] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 11 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:05.935156 17761 raft_consensus.cc:1275] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20260709 12:22:05.918413 21061 consensus_queue.cc:1243] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer e2909cff1da44ad387f96a8ae3aa5afb log is divergent from this leader: its last log entry 6.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260709 12:22:05.942231 17683 raft_consensus.cc:1240] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 7 LEADER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 6. Current term is 7. Ops: []
W20260709 12:22:05.927493 21217 log.cc:927] Time spent T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.800s	user 0.000s	sys 0.003s
I20260709 12:22:05.939236 17755 pending_rounds.cc:85] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb: Aborting all ops after (but not including) 3
I20260709 12:22:05.920620 21177 raft_consensus.cc:2749] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:22:05.942790 18361 raft_consensus.cc:1240] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 10. Current term is 11. Ops: []
I20260709 12:22:05.943835 17727 leader_election.cc:304] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:22:05.945118 17755 pending_rounds.cc:107] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb: Aborting uncommitted NO_OP operation due to leader change: 7.4
I20260709 12:22:05.946413 17755 raft_consensus.cc:2889] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:05.898877 21085 raft_consensus.cc:515] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:05.880860 21237 consensus_queue.cc:1059] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.063s
I20260709 12:22:05.931066 21147 raft_consensus.cc:3060] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:05.946934 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:05.948244 17727 leader_election.cc:304] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:22:05.950258 17584 catalog_manager.cc:5697] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 7 to 9, leader changed from e063541542a94969898868bf6eb5ce1b (127.16.29.67) to 334a4e1f59944544b3b3ff36616b57da (127.16.29.68). New cstate: current_term: 9 leader_uuid: "334a4e1f59944544b3b3ff36616b57da" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:22:05.932085 21155 leader_election.cc:290] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 7 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:05.953559 18361 raft_consensus.cc:2468] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 6.
I20260709 12:22:05.958575 20846 leader_election.cc:290] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:05.959805 20921 leader_election.cc:290] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 6 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:05.960072 21092 consensus_queue.cc:1059] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 8.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 12:22:05.956656 17652 leader_election.cc:304] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:05.961174 21234 leader_election.cc:290] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:05.961709 21241 raft_consensus.cc:2804] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:22:05.962285 21241 raft_consensus.cc:697] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 6 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:22:05.926672 21230 consensus_queue.cc:1048] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.026s
I20260709 12:22:05.949537 21227 raft_consensus.cc:2749] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:22:05.947003 21137 raft_consensus.cc:2804] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:22:05.960994 21073 consensus_queue.cc:1059] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 5.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
W20260709 12:22:05.915182 17583 rpcz_store.cc:269] 0709 12:21:38.760829 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:38.810395 (+ 49566us) service_pool.cc:224] Handling call
0709 12:22:05.781231 (+26970836us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:21:56.089820 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:21:56.090685 (+   865us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:21:56.090760 (+    75us) write_op.cc:435] Acquired schema lock
0709 12:21:56.090787 (+    27us) tablet.cc:662] Decoding operations
0709 12:21:56.110181 (+ 19394us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:21:56.110311 (+   130us) write_op.cc:641] Partition lock acquired for write op
0709 12:21:56.110350 (+    39us) tablet.cc:685] Acquiring locks for 10 operations
0709 12:21:56.111468 (+  1118us) tablet.cc:701] Row locks acquired
0709 12:21:56.111525 (+    57us) write_op.cc:260] PREPARE: finished
0709 12:21:56.111772 (+   247us) write_op.cc:270] Start()
0709 12:21:56.111978 (+   206us) write_op.cc:276] Timestamp: P: 1783599716111678 usec, L: 0
0709 12:21:56.112006 (+    28us) op_driver.cc:348] REPLICATION: starting
0709 12:21:56.337130 (+225124us) log.cc:844] Serialized 4027 byte log entry
0709 12:21:56.480343 (+143213us) op_driver.cc:464] REPLICATION: finished
0709 12:21:56.657723 (+177380us) write_op.cc:301] APPLY: starting
0709 12:21:56.664351 (+  6628us) tablet.cc:1367] starting BulkCheckPresence
0709 12:21:56.664653 (+   302us) tablet.cc:1370] finished BulkCheckPresence
0709 12:21:56.666603 (+  1950us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:21:56.670309 (+  3706us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:21:56.670694 (+   385us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=10
0709 12:21:56.670740 (+    46us) write_op.cc:312] APPLY: finished
0709 12:21:56.671147 (+   407us) log.cc:844] Serialized 77 byte log entry
0709 12:21:56.671359 (+   212us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:21:56.671611 (+   252us) write_op.cc:454] Released schema lock
0709 12:21:56.671745 (+   134us) write_op.cc:341] FINISH: Updating metrics
0709 12:21:56.885615 (+213870us) spinlock_profiling.cc:243] Waited 194 ms on lock 0x7b540010d454. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a61be28 00007f7b4e056581 00007f7b4e055d58 00007f7b4e05a9f1 00007f7b4e05a9a9 00007f7b4e05a939 00007f7b4e05a901 00007f7b4e059b2d 0000000000388634 0000000000388479
0709 12:21:56.885884 (+   269us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:21:56.885926 (+    42us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":6523315,"reported_tablets":28,"child_traces":[["op",{"apply.queue_time_us":122677,"apply.run_cpu_time_us":4809,"apply.run_wall_time_us":78870,"num_ops":10,"prepare.queue_time_us":209292,"prepare.run_cpu_time_us":17321,"prepare.run_wall_time_us":799838,"raft.queue_time_us":79241,"raft.run_cpu_time_us":12102,"raft.run_wall_time_us":69831,"replication_time_us":368108,"spinlock_wait_cycles":446720384,"thread_start_us":198658,"threads_started":4,"wal-append.queue_time_us":45891,"wal-append.run_cpu_time_us":17783,"wal-append.run_wall_time_us":1297417}]]}
I20260709 12:22:05.955655 17584 catalog_manager.cc:5697] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 6 to 9. New cstate: current_term: 9 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:22:05.965936 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:05.945485 21053 leader_election.cc:290] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 8 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:05.967641 17727 leader_election.cc:304] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:22:05.948329 21079 leader_election.cc:290] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:05.970166 20911 raft_consensus.cc:2749] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 12:22:05.970386 20846 raft_consensus.cc:493] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:05.972991 21225 log.cc:927] Time spent T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.105s	user 0.000s	sys 0.001s
I20260709 12:22:05.974361 21178 leader_election.cc:290] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:22:05.975608 21184 consensus_meta.cc:287] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.625s	user 0.009s	sys 0.001s
I20260709 12:22:05.980386 21205 leader_election.cc:290] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 7 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:05.985798 21190 raft_consensus.cc:515] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:05.998632 17584 catalog_manager.cc:5697] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 4 to 7, leader changed from e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66) to 334a4e1f59944544b3b3ff36616b57da (127.16.29.68). New cstate: current_term: 7 leader_uuid: "334a4e1f59944544b3b3ff36616b57da" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:22:06.012758 17727 leader_election.cc:304] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:06.008256 17761 pending_rounds.cc:85] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb: Aborting all ops after (but not including) 5
I20260709 12:22:05.966560 21137 raft_consensus.cc:493] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
W20260709 12:22:05.998448 21196 consensus_meta.cc:287] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.920s	user 0.008s	sys 0.002s
I20260709 12:22:06.013909 17761 pending_rounds.cc:107] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb: Aborting uncommitted NO_OP operation due to leader change: 6.6
I20260709 12:22:06.017993 17761 raft_consensus.cc:2889] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:06.001796 21092 consensus_queue.cc:1243] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer e063541542a94969898868bf6eb5ce1b log is divergent from this leader: its last log entry 9.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260709 12:22:06.020435 17685 raft_consensus.cc:1275] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260709 12:22:06.020718 17727 leader_election.cc:304] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
W20260709 12:22:06.021867 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 48499.9us
I20260709 12:22:06.011464 20846 raft_consensus.cc:515] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:06.027197 20846 leader_election.cc:290] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:06.011679 20911 raft_consensus.cc:2749] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:22:06.011716 21237 raft_consensus.cc:493] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:06.030294 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:06.030633 17686 raft_consensus.cc:2393] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 8: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
W20260709 12:22:06.032622 21206 log.cc:927] Time spent T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 1.226s	user 0.000s	sys 0.007s
I20260709 12:22:06.041097 17833 raft_consensus.cc:2468] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 6.
I20260709 12:22:06.038018 21190 leader_election.cc:290] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 8 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:06.052188 18334 leader_election.cc:304] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:22:06.028173 21230 raft_consensus.cc:2749] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:22:06.054878 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:06.055737 17830 raft_consensus.cc:2393] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 8: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
W20260709 12:22:06.056293 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 126712us
I20260709 12:22:06.047189 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:06.022712 21079 raft_consensus.cc:2804] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:22:06.017057 21137 raft_consensus.cc:3060] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:06.059527 18374 raft_consensus.cc:1275] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20260709 12:22:06.060325 21079 raft_consensus.cc:493] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:06.060757 21079 raft_consensus.cc:3060] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:06.044219 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:05.963174 21241 consensus_queue.cc:237] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:06.047129 21244 consensus_queue.cc:1048] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.019s
I20260709 12:22:06.062600 17764 raft_consensus.cc:3060] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:22:06.062814 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:06.023010 17584 catalog_manager.cc:5697] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da reported cstate change: config changed from index 4 to 5, term changed from 4 to 7, leader changed from e063541542a94969898868bf6eb5ce1b (127.16.29.67) to 0da169ff82674dc1947055ce43de68a3 (127.16.29.65), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER, no longer has a pending config: VOTER 0da169ff82674dc1947055ce43de68a3 (127.16.29.65), VOTER e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66), VOTER e063541542a94969898868bf6eb5ce1b (127.16.29.67), VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68). New cstate: current_term: 7 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:22:06.049217 21175 raft_consensus.cc:3060] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:06.049722 21115 raft_consensus.cc:2749] T 7cb3768d215e409fb9f5baf18d124442 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:22:06.057272 17801 leader_election.cc:304] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:06.057237 17838 raft_consensus.cc:2468] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 7.
I20260709 12:22:06.064932 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:06.065860 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:06.067277 18334 leader_election.cc:304] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:22:06.069684 18374 raft_consensus.cc:2393] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 6: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:22:06.072691 20911 raft_consensus.cc:493] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:06.073060 21232 raft_consensus.cc:493] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:06.073194 20941 raft_consensus.cc:2804] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:22:06.074282 17801 leader_election.cc:304] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:06.074649 17654 leader_election.cc:304] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:06.077527 17727 leader_election.cc:304] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:06.081097 17764 raft_consensus.cc:2468] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 8.
I20260709 12:22:06.040154 21080 leader_election.cc:290] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:06.083369 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:06.083999 18362 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59918 (request call id 1136) took 1185 ms. Trace:
I20260709 12:22:06.089557 18362 rpcz_store.cc:276] 0709 12:22:04.898146 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:04.898285 (+   139us) service_pool.cc:224] Handling call
0709 12:22:06.083808 (+1185523us) spinlock_profiling.cc:243] Waited 87.5 ms on lock 0x7b54006739d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:22:06.083949 (+   141us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":201325824}
I20260709 12:22:06.086258 17584 catalog_manager.cc:5697] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 1 to 4, leader changed from e063541542a94969898868bf6eb5ce1b (127.16.29.67) to 334a4e1f59944544b3b3ff36616b57da (127.16.29.68). New cstate: current_term: 4 leader_uuid: "334a4e1f59944544b3b3ff36616b57da" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:22:06.075985 20941 raft_consensus.cc:493] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:06.076287 20984 raft_consensus.cc:2804] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:22:06.073271 20911 raft_consensus.cc:515] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:06.090965 17654 leader_election.cc:304] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:06.093339 20941 raft_consensus.cc:3060] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:06.097978 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:06.073734 21155 raft_consensus.cc:2804] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:22:06.068485 21196 raft_consensus.cc:493] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:06.093848 17584 catalog_manager.cc:5697] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 2 to 5, leader changed from e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66) to 334a4e1f59944544b3b3ff36616b57da (127.16.29.68). New cstate: current_term: 5 leader_uuid: "334a4e1f59944544b3b3ff36616b57da" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:22:06.094513 17727 leader_election.cc:304] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:06.088457 20973 raft_consensus.cc:2804] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:22:06.057376 21237 raft_consensus.cc:515] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:06.109054 17830 raft_consensus.cc:1240] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 6. Current term is 7. Ops: [6.3-6.3]
I20260709 12:22:06.071065 21016 consensus_queue.cc:1048] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.032s
I20260709 12:22:06.110244 17584 catalog_manager.cc:5697] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 2 to 6. New cstate: current_term: 6 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:22:06.074422 21073 consensus_queue.cc:1048] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.029s
I20260709 12:22:06.113754 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:06.099000 20984 raft_consensus.cc:697] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 9 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:22:06.076323 21205 consensus_queue.cc:1059] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 7.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.032s
I20260709 12:22:06.104024 21208 raft_consensus.cc:2804] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:22:06.117586 17756 raft_consensus.cc:3060] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:06.093096 21183 consensus_queue.cc:1048] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.025s
I20260709 12:22:06.108518 21110 raft_consensus.cc:2804] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:22:06.116531 21249 raft_consensus.cc:2749] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:22:06.124200 21060 raft_consensus.cc:3055] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:22:06.124619 21060 raft_consensus.cc:740] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:22:06.096658 21236 consensus_queue.cc:1048] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.043s
I20260709 12:22:06.107008 21061 raft_consensus.cc:2749] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 12:22:06.123557 21110 raft_consensus.cc:493] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:06.109025 21184 raft_consensus.cc:493] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:06.126595 17584 catalog_manager.cc:5697] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:22:06.081988 21170 consensus_queue.cc:1048] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.029s
I20260709 12:22:06.111624 20973 raft_consensus.cc:493] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:06.111351 21155 raft_consensus.cc:697] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 9 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
W20260709 12:22:06.132376 21210 log.cc:927] Time spent T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 1.315s	user 0.004s	sys 0.000s
I20260709 12:22:06.108670 21132 raft_consensus.cc:515] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:06.129364 21085 leader_election.cc:290] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 9 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:06.134840 21057 raft_consensus.cc:3060] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:06.121634 21208 raft_consensus.cc:493] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:06.120499 21115 consensus_queue.cc:1059] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.017s
I20260709 12:22:06.096411 21239 raft_consensus.cc:515] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:06.109891 21196 raft_consensus.cc:515] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:06.140129 17584 catalog_manager.cc:5697] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 2 to 6. New cstate: current_term: 6 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:22:06.131003 21110 raft_consensus.cc:3060] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:06.144167 17584 catalog_manager.cc:5697] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:22:06.088796 21232 raft_consensus.cc:515] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:06.136384 20973 raft_consensus.cc:3060] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:06.152635 17682 raft_consensus.cc:1240] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 5. Current term is 6. Ops: []
I20260709 12:22:06.141748 21208 raft_consensus.cc:3060] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:06.135944 21184 raft_consensus.cc:515] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:06.163390 17756 raft_consensus.cc:2468] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 6.
I20260709 12:22:06.164737 21184 leader_election.cc:290] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:06.154603 21060 consensus_queue.cc:260] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 5.8, Last appended by leader: 8, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:06.166580 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:06.169222 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:06.171495 17756 raft_consensus.cc:2468] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 7.
I20260709 12:22:06.170795 21183 raft_consensus.cc:493] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:06.153939 21132 leader_election.cc:290] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 6 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:06.179639 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:06.180483 17756 raft_consensus.cc:2393] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 8: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
W20260709 12:22:06.183642 17803 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:06.196470 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 9 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:06.156694 20984 consensus_queue.cc:237] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:06.199862 17764 raft_consensus.cc:2393] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 10: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:22:06.200879 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:22:06.193574 21257 log.cc:927] Time spent T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.117s	user 0.001s	sys 0.000s
I20260709 12:22:06.185362 21183 raft_consensus.cc:515] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:06.175783 21137 raft_consensus.cc:515] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:06.205260 21137 leader_election.cc:290] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 5 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:06.205371 21183 leader_election.cc:290] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:06.206923 21237 leader_election.cc:290] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:06.183023 21155 consensus_queue.cc:237] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:06.209731 21230 raft_consensus.cc:493] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:06.210103 17838 raft_consensus.cc:1275] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260709 12:22:06.210268 21230 raft_consensus.cc:515] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:06.211189 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:06.211045 21115 consensus_queue.cc:1059] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 10.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 12:22:06.212185 21240 consensus_queue.cc:1059] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 12:22:06.212949 21227 raft_consensus.cc:3055] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:22:06.213459 21227 raft_consensus.cc:740] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 10 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:22:06.214300 21227 consensus_queue.cc:260] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:06.215101 18367 raft_consensus.cc:2468] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 8.
I20260709 12:22:06.215559 21227 raft_consensus.cc:3060] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:06.196239 21060 raft_consensus.cc:3060] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:06.216168 18367 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59902 (request call id 946) took 1229 ms. Trace:
I20260709 12:22:06.216342 21205 raft_consensus.cc:493] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:06.218382 20973 raft_consensus.cc:515] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:06.220568 21230 leader_election.cc:290] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:06.228381 18367 rpcz_store.cc:276] 0709 12:22:04.987125 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:04.987567 (+   442us) service_pool.cc:224] Handling call
0709 12:22:06.216012 (+1228445us) spinlock_profiling.cc:243] Waited 1.17 s on lock 0x7b540051a254. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:22:06.216115 (+   103us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2689512704}
I20260709 12:22:06.195793 20911 leader_election.cc:290] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:06.228757 20394 raft_consensus.cc:493] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:22:06.242311 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:06.243113 17830 raft_consensus.cc:3060] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:06.243175 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:06.244098 18365 raft_consensus.cc:2468] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 8.
I20260709 12:22:06.193291 21079 raft_consensus.cc:515] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:06.232537 21183 raft_consensus.cc:493] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:06.241659 20394 raft_consensus.cc:515] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:06.212916 21115 consensus_queue.cc:1059] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.028s
I20260709 12:22:06.289884 20973 leader_election.cc:290] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 8 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:06.290218 20394 leader_election.cc:290] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:22:06.247243 21254 log.cc:927] Time spent T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.129s	user 0.000s	sys 0.001s
W20260709 12:22:06.301600 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 71031.4us
I20260709 12:22:06.233471 21237 consensus_queue.cc:1048] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.050s
I20260709 12:22:06.310323 17830 raft_consensus.cc:2468] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 6.
I20260709 12:22:06.312242 17652 leader_election.cc:304] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:22:06.236529 21205 raft_consensus.cc:515] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:06.322957 20157 raft_consensus.cc:493] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:06.324708 20157 raft_consensus.cc:515] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:06.301275 20394 raft_consensus.cc:493] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:06.308015 21088 raft_consensus.cc:2749] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:22:06.300427 21183 raft_consensus.cc:515] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:06.322723 21241 raft_consensus.cc:2804] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:22:06.338726 17683 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44898 (request call id 738) took 2181 ms. Trace:
I20260709 12:22:06.343958 17683 rpcz_store.cc:276] 0709 12:22:04.157588 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:05.623063 (+1465475us) service_pool.cc:224] Handling call
0709 12:22:05.623455 (+   392us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:05.942667 (+319212us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:06.338468 (+395801us) spinlock_profiling.cc:243] Waited 319 ms on lock 0x7b54003485d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:22:06.338646 (+   178us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":732675456}
I20260709 12:22:06.331234 20394 raft_consensus.cc:515] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:06.342499 21110 raft_consensus.cc:515] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:22:06.343382 21220 log.cc:927] Time spent T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.881s	user 0.000s	sys 0.002s
I20260709 12:22:06.327947 21239 leader_election.cc:290] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 6 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:06.357586 20157 leader_election.cc:290] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:06.348892 21241 raft_consensus.cc:697] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 6 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:22:06.365986 18371 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59902 (request call id 944) took 1379 ms. Trace:
I20260709 12:22:06.366428 18371 rpcz_store.cc:276] 0709 12:22:04.986615 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:04.986942 (+   327us) service_pool.cc:224] Handling call
0709 12:22:06.365812 (+1378870us) spinlock_profiling.cc:243] Waited 554 ms on lock 0x7b54006c5cd4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:22:06.365922 (+   110us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1273809792}
I20260709 12:22:06.376835 17683 raft_consensus.cc:1275] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260709 12:22:06.378758 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:06.380211 17683 raft_consensus.cc:3060] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:06.364913 21241 consensus_queue.cc:237] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:06.383091 21240 raft_consensus.cc:493] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:06.409536 17728 leader_election.cc:304] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:06.416631 17686 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37402 (request call id 2410) took 1305 ms. Trace:
I20260709 12:22:06.417130 17686 rpcz_store.cc:276] 0709 12:22:05.111245 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:05.246310 (+135065us) service_pool.cc:224] Handling call
0709 12:22:06.416400 (+1170090us) spinlock_profiling.cc:243] Waited 770 ms on lock 0x7b54003016d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:22:06.416561 (+   161us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1770692992}
I20260709 12:22:06.350840 21147 raft_consensus.cc:515] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:22:06.415781 17759 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38776 (request call id 1962) took 1424 ms (client timeout 1895 ms). Trace:
W20260709 12:22:06.422425 17759 rpcz_store.cc:269] 0709 12:22:04.991777 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:05.025796 (+ 34019us) service_pool.cc:224] Handling call
0709 12:22:06.415562 (+1389766us) spinlock_profiling.cc:243] Waited 643 ms on lock 0x7b54000fc3d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:22:06.415704 (+   142us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1479775104}
I20260709 12:22:06.418563 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:06.423619 17686 raft_consensus.cc:2468] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 6.
I20260709 12:22:06.424526 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:06.425294 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:06.425720 17686 raft_consensus.cc:2468] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 7.
I20260709 12:22:06.426292 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:22:06.427080 17654 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:06.431592 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:06.432636 17686 raft_consensus.cc:2468] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 9.
I20260709 12:22:06.433298 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:06.458541 17686 raft_consensus.cc:2393] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 6: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
W20260709 12:22:06.458482 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 49044.2us
I20260709 12:22:06.461589 17683 raft_consensus.cc:2468] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 6.
I20260709 12:22:06.462440 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:06.464772 17683 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 8.
I20260709 12:22:06.466063 21205 leader_election.cc:290] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:06.467478 21147 leader_election.cc:290] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 6 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:06.468024 20394 leader_election.cc:290] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:06.467614 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:06.461086 20941 raft_consensus.cc:515] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:06.413970 21240 raft_consensus.cc:515] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:06.470265 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:22:06.465034 17802 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:06.464849 21261 log.cc:927] Time spent T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.181s	user 0.000s	sys 0.001s
W20260709 12:22:06.464602 21260 log.cc:927] Time spent T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.225s	user 0.002s	sys 0.005s
I20260709 12:22:06.463517 21079 leader_election.cc:290] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 6 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:22:06.494480 21264 log.cc:927] Time spent T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.138s	user 0.001s	sys 0.000s
I20260709 12:22:06.498015 20941 leader_election.cc:290] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 7 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:06.503428 21272 raft_consensus.cc:2749] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:22:06.506968 21275 consensus_queue.cc:1048] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.003s
I20260709 12:22:06.414589 21141 consensus_queue.cc:260] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 9.10, Last appended by leader: 10, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:06.471509 21267 raft_consensus.cc:493] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:06.505347 21265 log.cc:927] Time spent T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.112s	user 0.000s	sys 0.001s
I20260709 12:22:06.549671 21240 leader_election.cc:290] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:06.542508 21267 raft_consensus.cc:515] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:06.554296 21267 leader_election.cc:290] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:06.561625 21208 raft_consensus.cc:515] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:06.566470 21208 leader_election.cc:290] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 7 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:22:06.612171 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 151374us
W20260709 12:22:06.618909 17689 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37402 (request call id 2391) took 2386 ms (client timeout 2207 ms). Trace:
I20260709 12:22:06.612170 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 9 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:06.624599 17686 raft_consensus.cc:2410] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for term 9 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 4.
I20260709 12:22:06.626317 17680 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:54540 (request call id 32) took 1719 ms. Trace:
I20260709 12:22:06.626840 17680 rpcz_store.cc:276] 0709 12:22:04.906674 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:05.449195 (+542521us) service_pool.cc:224] Handling call
0709 12:22:06.625356 (+1176161us) spinlock_profiling.cc:243] Waited 6.99 ms on lock 0x7b540039d3a0. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a59d520 00007f7b4a61a652 00007f7b4f5e860a 00007f7b48b6e7d4 00007f7b48b6e762 00007f7b48b6e691 00007f7b48b6e60c 00007f7b48b6d8c2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:22:06.626030 (+   674us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":16084352}
W20260709 12:22:06.619807 17689 rpcz_store.cc:269] 0709 12:22:04.232207 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:04.706073 (+473866us) service_pool.cc:224] Handling call
0709 12:22:06.618108 (+1912035us) spinlock_profiling.cc:243] Waited 589 ms on lock 0x7b54001aa754. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:22:06.618576 (+   468us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1353919360}
W20260709 12:22:06.632310 17803 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:06.644780 17684 raft_consensus.cc:1240] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 9 LEADER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 8. Current term is 9. Ops: []
I20260709 12:22:06.647199 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:06.649542 17683 raft_consensus.cc:2468] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 6.
W20260709 12:22:06.650595 17681 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37396 (request call id 2022) took 2566 ms (client timeout 1922 ms). Trace:
I20260709 12:22:06.656965 21031 raft_consensus.cc:493] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:06.650487 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
W20260709 12:22:06.658286 17681 rpcz_store.cc:269] 0709 12:22:04.083940 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:04.084085 (+   145us) service_pool.cc:224] Handling call
0709 12:22:06.649984 (+2565899us) spinlock_profiling.cc:243] Waited 847 ms on lock 0x7b5400190354. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:22:06.650313 (+   329us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1947309440}
I20260709 12:22:06.665410 21227 consensus_queue.cc:1048] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.003s
I20260709 12:22:06.668778 17652 leader_election.cc:304] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:22:06.673688 21085 raft_consensus.cc:493] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:22:06.674607 21085 raft_consensus.cc:515] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:06.676501 17831 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.BulkChangeConfig from 127.0.0.1:37988 (request call id 19) took 1655 ms. Trace:
I20260709 12:22:06.677165 17831 rpcz_store.cc:276] 0709 12:22:05.020431 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:05.045641 (+ 25210us) service_pool.cc:224] Handling call
0709 12:22:06.675906 (+1630265us) spinlock_profiling.cc:243] Waited 89.2 ms on lock 0x7b540022ae54. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a626735 00007f7b4f5e6d29 00007f7b48b651d4 00007f7b48b65162 00007f7b48b65091 00007f7b48b6500c 00007f7b48b642c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:22:06.676255 (+   349us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":205167488}
I20260709 12:22:06.678009 21241 raft_consensus.cc:2804] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:22:06.678802 21241 raft_consensus.cc:697] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 8 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:22:06.676626 17684 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37396 (request call id 2033) took 1770 ms. Trace:
I20260709 12:22:06.692276 20941 consensus_queue.cc:1048] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:22:06.693557 21267 consensus_queue.cc:1059] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 7.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:22:06.692577 21054 consensus_queue.cc:1048] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:22:06.695477 21267 raft_consensus.cc:3055] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:22:06.660073 21031 raft_consensus.cc:515] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:06.693969 21085 leader_election.cc:290] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:06.681160 17684 rpcz_store.cc:276] 0709 12:22:04.906083 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:05.844520 (+938437us) service_pool.cc:224] Handling call
0709 12:22:05.844975 (+   455us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:06.645751 (+800776us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:06.676266 (+ 30515us) spinlock_profiling.cc:243] Waited 799 ms on lock 0x7b5400380254. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:22:06.676520 (+   254us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1837885312}
I20260709 12:22:06.706166 21031 leader_election.cc:290] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:06.712745 17837 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35042 (request call id 1891) took 1638 ms. Trace:
I20260709 12:22:06.713313 17837 rpcz_store.cc:276] 0709 12:22:05.074082 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:05.074248 (+   166us) service_pool.cc:224] Handling call
0709 12:22:05.677833 (+603585us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:05.713872 (+ 36039us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:06.712270 (+998398us) spinlock_profiling.cc:243] Waited 598 ms on lock 0x7b540028d050. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a61f8ad 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:22:06.712508 (+   238us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1374765824}
I20260709 12:22:06.712778 17652 leader_election.cc:304] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:22:06.700453 21267 raft_consensus.cc:740] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 7 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:22:06.739357 21110 leader_election.cc:290] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 9 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:06.739730 21052 raft_consensus.cc:493] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:06.739930 21237 raft_consensus.cc:2804] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:22:06.740783 21052 raft_consensus.cc:515] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:06.745447 20921 consensus_queue.cc:1048] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.013s
W20260709 12:22:06.746147 18334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:06.747393 17801 leader_election.cc:304] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:22:06.745904 17635 auto_rebalancer.cc:893] Removing replace marker failed: Replica e063541542a94969898868bf6eb5ce1b is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 5 committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:22:06.698016 21241 consensus_queue.cc:237] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:22:06.755627 17635 auto_rebalancer.cc:901] Could not move replica: Illegal state: not currently leader
W20260709 12:22:06.756232 17635 auto_rebalancer.cc:330] scheduled replica move failed to complete: Illegal state: not currently leader
I20260709 12:22:06.744275 21237 raft_consensus.cc:493] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:06.747687 21052 leader_election.cc:290] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:06.767195 17688 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44898 (request call id 732) took 2632 ms. Trace:
I20260709 12:22:06.772151 21177 raft_consensus.cc:3055] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:22:06.772750 21177 raft_consensus.cc:740] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 6 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:22:06.772907 21230 raft_consensus.cc:3055] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:22:06.773502 21230 raft_consensus.cc:740] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:22:06.774134 21177 consensus_queue.cc:260] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:06.776343 21231 raft_consensus.cc:3055] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:22:06.776623 17684 raft_consensus.cc:1275] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 5. (term mismatch)
I20260709 12:22:06.776865 21231 raft_consensus.cc:740] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:22:06.777127 17684 pending_rounds.cc:85] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3: Aborting all ops after (but not including) 4
I20260709 12:22:06.777551 17684 pending_rounds.cc:107] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3: Aborting uncommitted NO_OP operation due to leader change: 7.5
I20260709 12:22:06.777926 17684 raft_consensus.cc:2889] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:22:06.780196 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 67736.9us
I20260709 12:22:06.781569 17652 leader_election.cc:304] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:06.732587 20941 consensus_queue.cc:1059] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 7.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.036s
I20260709 12:22:06.782634 21191 raft_consensus.cc:2749] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:22:06.785380 20973 raft_consensus.cc:493] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:06.786285 20973 raft_consensus.cc:515] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:06.788072 21244 raft_consensus.cc:3055] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:22:06.788596 21244 raft_consensus.cc:740] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:22:06.790122 20973 leader_election.cc:290] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:06.790059 21244 consensus_queue.cc:260] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 6.3, Last appended by leader: 3, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:06.792548 21267 consensus_queue.cc:260] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:06.795185 21267 raft_consensus.cc:3060] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:22:06.796154 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 46440.6us
I20260709 12:22:06.797168 21255 raft_consensus.cc:3055] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:22:06.797672 21255 raft_consensus.cc:740] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 7 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:22:06.758831 21237 raft_consensus.cc:3060] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:06.798813 21255 consensus_queue.cc:260] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:06.778371 21231 consensus_queue.cc:260] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:06.812083 20921 raft_consensus.cc:2804] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:22:06.774940 21230 consensus_queue.cc:260] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:06.812845 21185 raft_consensus.cc:3055] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:22:06.813416 21185 raft_consensus.cc:740] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:22:06.814734 21230 raft_consensus.cc:3060] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:06.809648 17685 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37402 (request call id 2411) took 1678 ms. Trace:
I20260709 12:22:06.816781 17685 rpcz_store.cc:276] 0709 12:22:05.131496 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:05.897096 (+765600us) service_pool.cc:224] Handling call
0709 12:22:05.897434 (+   338us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:06.020905 (+123471us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:06.809191 (+788286us) spinlock_profiling.cc:243] Waited 122 ms on lock 0x7b5400426dd4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:22:06.809444 (+   253us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":281731968}
I20260709 12:22:06.817247 21185 consensus_queue.cc:260] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:06.819432 21207 raft_consensus.cc:493] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:22:06.812842 20921 raft_consensus.cc:697] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 6 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:22:06.821781 21232 leader_election.cc:290] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:06.824173 21196 leader_election.cc:290] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:06.819832 21185 raft_consensus.cc:3060] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:06.820173 21207 raft_consensus.cc:3060] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:06.827702 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:06.828235 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:06.829422 17687 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37396 (request call id 2029) took 1924 ms. Trace:
I20260709 12:22:06.829727 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 12 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:06.829867 17687 rpcz_store.cc:276] 0709 12:22:04.904921 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:05.841887 (+936966us) service_pool.cc:224] Handling call
0709 12:22:05.842156 (+   269us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:05.854743 (+ 12587us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:06.829005 (+974262us) spinlock_profiling.cc:243] Waited 9.08 ms on lock 0x7b54002530d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:22:06.829248 (+   243us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":20894592}
I20260709 12:22:06.830687 17685 raft_consensus.cc:2410] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 12 because replica has last-logged OpId of term: 10 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 8.
I20260709 12:22:06.831056 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:06.832050 17681 raft_consensus.cc:1275] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 8 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20260709 12:22:06.822206 20921 consensus_queue.cc:237] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:06.833405 17681 raft_consensus.cc:1240] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 7. Current term is 8. Ops: []
I20260709 12:22:06.832490 17684 raft_consensus.cc:1240] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 8. Current term is 9. Ops: []
I20260709 12:22:06.834896 21255 raft_consensus.cc:3060] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:06.791652 21244 raft_consensus.cc:3060] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:22:06.836382 17730 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:06.839262 18374 raft_consensus.cc:3055] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:22:06.839658 18374 raft_consensus.cc:740] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:22:06.840157 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:06.841024 17757 raft_consensus.cc:2468] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 8.
I20260709 12:22:06.840710 18374 consensus_queue.cc:260] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:06.841967 18374 raft_consensus.cc:3060] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:06.844306 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:06.844449 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:06.845145 17837 raft_consensus.cc:3060] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:06.846256 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:06.847054 17831 raft_consensus.cc:3060] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:06.829476 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:22:06.852918 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 65644.9us
I20260709 12:22:06.854915 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:06.843804 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:06.886634 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:06.888823 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:06.892751 17763 raft_consensus.cc:2393] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 7: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:22:06.894591 17801 leader_election.cc:304] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:06.899839 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:06.902701 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:06.903154 17682 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44898 (request call id 744) took 1657 ms. Trace:
I20260709 12:22:06.903565 17759 raft_consensus.cc:1240] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 7. Current term is 8. Ops: []
I20260709 12:22:06.904171 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:06.905426 17761 raft_consensus.cc:1275] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 8 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
W20260709 12:22:06.908398 17804 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:06.911685 21137 consensus_queue.cc:1048] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.040s
I20260709 12:22:06.913149 21085 raft_consensus.cc:493] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:06.903662 17682 rpcz_store.cc:276] 0709 12:22:05.245931 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:05.925837 (+679906us) service_pool.cc:224] Handling call
0709 12:22:05.936208 (+ 10371us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:06.153248 (+217040us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:06.902725 (+749477us) spinlock_profiling.cc:243] Waited 216 ms on lock 0x7b5400207e54. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:22:06.902983 (+   258us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":497079040}
I20260709 12:22:06.767902 17688 rpcz_store.cc:276] 0709 12:22:04.134652 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:05.577903 (+1443251us) service_pool.cc:224] Handling call
0709 12:22:05.583801 (+  5898us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:05.584121 (+   320us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:05.584164 (+    43us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:06.677461 (+1093297us) spinlock_profiling.cc:243] Waited 4.13 ms on lock 0x7b2c00000c88. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b48f19a11 00007f7b48f29372 00007f7b4a65ba01 00007f7b4a62043a 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2
0709 12:22:06.678404 (+   943us) log.cc:844] Serialized 26 byte log entry
0709 12:22:06.729011 (+ 50607us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:22:06.731826 (+  2815us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:06.734198 (+  2372us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:06.757848 (+ 23650us) raft_consensus.cc:1685] finished
0709 12:22:06.757868 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:06.766964 (+  9096us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":12230,"raft.run_cpu_time_us":152,"raft.run_wall_time_us":10284,"spinlock_wait_cycles":9508096,"thread_start_us":50093,"threads_started":1,"wal-append.queue_time_us":69183}
I20260709 12:22:06.917892 17833 raft_consensus.cc:2393] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 7: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:22:06.902741 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:06.921006 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:06.925096 21085 raft_consensus.cc:515] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:06.927690 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
W20260709 12:22:06.929136 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 1.73155e+06us
I20260709 12:22:06.930408 18333 leader_election.cc:304] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:22:06.931785 21155 raft_consensus.cc:2749] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 12:22:06.936095 17835 raft_consensus.cc:3060] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Advancing to term 9
I20260709 12:22:06.940065 18363 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 1130) took 2362 ms. Trace:
I20260709 12:22:06.940444 17687 raft_consensus.cc:3060] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:06.942338 17838 raft_consensus.cc:2468] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 7.
I20260709 12:22:06.944727 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:06.945590 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:06.945748 17832 raft_consensus.cc:2468] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 11.
I20260709 12:22:06.946556 18365 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 8.
I20260709 12:22:06.947468 17686 raft_consensus.cc:1275] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 10 index: 11. Preceding OpId from leader: term: 12 index: 12. (index mismatch)
I20260709 12:22:06.947827 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:06.949198 18368 raft_consensus.cc:3060] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:06.949597 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:06.950378 18367 raft_consensus.cc:3060] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:06.925540 17755 raft_consensus.cc:1275] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260709 12:22:06.956029 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:06.957369 17758 raft_consensus.cc:2410] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 10 because replica has last-logged OpId of term: 9 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 7.
I20260709 12:22:06.959693 18366 raft_consensus.cc:1275] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 10 index: 11. Preceding OpId from leader: term: 12 index: 12. (index mismatch)
I20260709 12:22:06.966223 21145 raft_consensus.cc:3055] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:22:06.966094 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:06.966921 21145 raft_consensus.cc:740] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:22:06.967311 17764 raft_consensus.cc:3060] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:06.967501 21231 raft_consensus.cc:3060] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:06.969170 20394 raft_consensus.cc:493] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:06.970027 21056 raft_consensus.cc:493] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
W20260709 12:22:06.975773 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 45515.4us
I20260709 12:22:06.976675 17837 raft_consensus.cc:2468] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 5.
W20260709 12:22:06.984452 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 96664.4us
I20260709 12:22:06.963647 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:06.966087 17761 raft_consensus.cc:1240] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 4. Current term is 5. Ops: []
I20260709 12:22:06.988649 17652 leader_election.cc:304] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:22:06.940588 18363 rpcz_store.cc:276] 0709 12:22:04.534780 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:04.534907 (+   127us) service_pool.cc:224] Handling call
0709 12:22:04.535204 (+   297us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:05.807066 (+1271862us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:06.864226 (+1057160us) spinlock_profiling.cc:243] Waited 1.23 s on lock 0x7b5400727254. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:22:06.896931 (+ 32705us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2838224128}
I20260709 12:22:06.992954 17831 raft_consensus.cc:2468] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 6.
I20260709 12:22:06.996462 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:06.996914 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:06.984917 21085 leader_election.cc:290] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:22:07.001345 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:07.002912 17728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:06.985235 21228 raft_consensus.cc:3060] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:07.008267 18374 raft_consensus.cc:1275] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 7. (term mismatch)
I20260709 12:22:07.009867 18374 pending_rounds.cc:85] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da: Aborting all ops after (but not including) 6
I20260709 12:22:07.015487 18374 pending_rounds.cc:107] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da: Aborting uncommitted NO_OP operation due to leader change: 6.7
I20260709 12:22:07.015971 18374 raft_consensus.cc:2889] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:07.015090 18371 raft_consensus.cc:2468] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 8.
W20260709 12:22:07.024739 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 50686.5us
I20260709 12:22:07.015605 21207 raft_consensus.cc:515] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:07.003835 21056 raft_consensus.cc:515] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:07.027706 21056 leader_election.cc:290] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:07.028055 20157 raft_consensus.cc:2804] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:22:07.028575 20157 raft_consensus.cc:493] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:07.028975 20157 raft_consensus.cc:3060] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:07.034082 17728 leader_election.cc:304] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:07.036011 21147 raft_consensus.cc:493] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:07.036427 17761 raft_consensus.cc:1275] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260709 12:22:07.036837 21147 raft_consensus.cc:515] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:07.041690 17758 raft_consensus.cc:1217] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Deduplicated request from leader. Original: 4.4->[6.5-8.6]   Dedup: 8.6->[]
I20260709 12:22:07.042411 17758 raft_consensus.cc:1240] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 8. Current term is 9. Ops: [6.5-8.6]
I20260709 12:22:07.002558 21145 consensus_queue.cc:260] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:07.044173 21145 raft_consensus.cc:3060] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:22:07.048092 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 47367.9us
I20260709 12:22:07.049356 18334 leader_election.cc:304] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:22:07.049862 21085 consensus_queue.cc:1048] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 12:22:07.056226 21183 leader_election.cc:290] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:07.057032 21156 raft_consensus.cc:2804] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:22:07.057575 21156 raft_consensus.cc:493] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:06.970032 20394 raft_consensus.cc:515] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:07.039647 21147 leader_election.cc:290] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:22:07.071028 17802 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:07.069650 18368 raft_consensus.cc:2468] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 9.
I20260709 12:22:07.073230 17687 raft_consensus.cc:2468] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 7.
I20260709 12:22:07.050347 18334 leader_election.cc:304] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:22:07.078364 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:07.027654 21207 leader_election.cc:290] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 10 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:07.084585 18367 raft_consensus.cc:2468] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 7.
I20260709 12:22:07.088617 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:07.061807 21156 raft_consensus.cc:3060] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Advancing to term 12
W20260709 12:22:07.135010 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 128971us
I20260709 12:22:07.146063 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:22:07.157145 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 106887us
W20260709 12:22:07.268887 21289 log.cc:927] Time spent T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.264s	user 0.000s	sys 0.004s
I20260709 12:22:07.158954 21133 raft_consensus.cc:493] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:22:07.159958 21155 raft_consensus.cc:2804] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:22:07.210585 21240 consensus_queue.cc:1048] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:22:07.376607 17835 raft_consensus.cc:2468] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 9.
W20260709 12:22:07.379899 17730 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:07.380558 18365 raft_consensus.cc:1240] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 6. Current term is 7. Ops: []
W20260709 12:22:07.395555 17757 consensus_meta.cc:287] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.504s	user 0.020s	sys 0.026s
I20260709 12:22:07.398031 17757 raft_consensus.cc:2468] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 8.
I20260709 12:22:07.398546 18374 raft_consensus.cc:2410] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for term 9 because replica has last-logged OpId of term: 8 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 7.
I20260709 12:22:07.421393 18363 raft_consensus.cc:1275] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260709 12:22:07.422556 21053 raft_consensus.cc:493] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:07.422858 21267 consensus_meta.cc:287] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.627s	user 0.015s	sys 0.004s
I20260709 12:22:07.408313 21241 raft_consensus.cc:493] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:07.423362 21053 raft_consensus.cc:515] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:07.424157 20980 raft_consensus.cc:2804] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:22:07.424019 21241 raft_consensus.cc:515] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:07.426354 21241 leader_election.cc:290] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:07.427647 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:07.428369 17760 raft_consensus.cc:2468] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 6.
I20260709 12:22:07.428479 21141 raft_consensus.cc:3060] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:22:07.430787 21185 consensus_meta.cc:287] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.605s	user 0.016s	sys 0.000s
I20260709 12:22:07.431653 21056 raft_consensus.cc:493] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:22:07.432093 21133 raft_consensus.cc:515] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:07.432636 21056 raft_consensus.cc:515] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:07.435467 21079 raft_consensus.cc:2804] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:22:07.436064 21079 raft_consensus.cc:493] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:07.436471 21079 raft_consensus.cc:3060] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:07.436363 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:07.437951 17837 raft_consensus.cc:3060] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:22:07.417519 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 93465.6us
W20260709 12:22:07.409154 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 337593us
W20260709 12:22:07.513379 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 64980.5us
W20260709 12:22:07.542876 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 508937us
I20260709 12:22:07.461364 17764 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 6.
W20260709 12:22:07.541168 17802 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:07.509671 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 420040us
I20260709 12:22:07.510932 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:07.518299 17831 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06"
dest_uuid: "e063541542a94969898868bf6eb5ce1b"
mode: GRACEFUL
new_leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
 from {username='slave'} at 127.0.0.1:59820
I20260709 12:22:09.031957 17831 raft_consensus.cc:606] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 6 LEADER]: Received request to transfer leadership to TS e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:09.032861 17831 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:59820 (request call id 27) took 1515 ms. Trace:
I20260709 12:22:09.033252 17831 rpcz_store.cc:276] 0709 12:22:07.517879 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:07.518026 (+   147us) service_pool.cc:224] Handling call
0709 12:22:09.032752 (+1514726us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:07.498273 17837 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 6.
I20260709 12:22:07.460139 18333 leader_election.cc:304] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:22:07.556167 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 80587.1us
I20260709 12:22:07.542557 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:09.070314 18368 raft_consensus.cc:2468] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 11.
W20260709 12:22:07.565055 20157 consensus_meta.cc:287] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.536s	user 0.012s	sys 0.000s
I20260709 12:22:09.136169 20157 raft_consensus.cc:515] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:07.491977 20394 leader_election.cc:290] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:07.524425 21031 raft_consensus.cc:493] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:07.432550 20980 raft_consensus.cc:697] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 6 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:22:07.484164 21060 raft_consensus.cc:493] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:22:09.147722 21060 raft_consensus.cc:515] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:07.514454 21088 raft_consensus.cc:493] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:09.134007 21311 raft_consensus.cc:3060] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:09.151649 21088 raft_consensus.cc:515] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:09.154234 21088 leader_election.cc:290] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:07.483099 21177 raft_consensus.cc:3060] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:07.463804 21053 leader_election.cc:290] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:07.519269 21079 raft_consensus.cc:515] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:07.435678 21133 leader_election.cc:290] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:07.401827 21155 raft_consensus.cc:493] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:07.374286 21147 raft_consensus.cc:493] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:22:09.172544 21147 raft_consensus.cc:515] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:07.542202 21185 consensus_queue.cc:1048] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.030s
W20260709 12:22:07.550674 21145 consensus_meta.cc:287] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.506s	user 0.015s	sys 0.001s
I20260709 12:22:07.435698 21056 leader_election.cc:290] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:07.447443 21244 raft_consensus.cc:493] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:09.185197 21244 raft_consensus.cc:515] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:07.489270 21241 raft_consensus.cc:493] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:22:09.188766 21241 raft_consensus.cc:515] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:09.191059 21241 leader_election.cc:290] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:22:07.414785 21230 consensus_meta.cc:287] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.599s	user 0.012s	sys 0.011s
W20260709 12:22:07.535516 21231 consensus_meta.cc:287] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.567s	user 0.010s	sys 0.001s
W20260709 12:22:07.455106 21237 consensus_meta.cc:287] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.656s	user 0.020s	sys 0.001s
I20260709 12:22:09.194056 21237 raft_consensus.cc:515] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:07.747771 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:09.200573 18362 raft_consensus.cc:2393] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 6: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:22:09.201236 18362 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59918 (request call id 1151) took 2117 ms. Trace:
I20260709 12:22:09.201642 18362 rpcz_store.cc:276] 0709 12:22:07.083521 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:07.083649 (+   128us) service_pool.cc:224] Handling call
0709 12:22:07.747504 (+663855us) spinlock_profiling.cc:243] Waited 1.15 ms on lock 0x7b5800102398. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b474fbb71 00007f7b474fbeee 000000000036eb39 00007f7b4f56f1e0 00007f7b4f6380c4 00007f7b4f638131 00007f7b4f5e60a0 00007f7b4f5e5984 00007f7b48b606d4 00007f7b48b60662
0709 12:22:09.201127 (+1453623us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2635008}
I20260709 12:22:07.821373 17727 leader_election.cc:304] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:22:07.883697 21290 log.cc:927] Time spent T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.462s	user 0.002s	sys 0.000s
W20260709 12:22:07.910390 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 369395us
W20260709 12:22:07.915923 21295 log.cc:927] Time spent T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.425s	user 0.005s	sys 0.001s
W20260709 12:22:08.638254 17618 debug-util.cc:398] Leaking SignalData structure 0x7b08000358c0 after lost signal to thread 16501
W20260709 12:22:09.203302 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 1.38201e+06us
W20260709 12:22:09.203651 17618 debug-util.cc:398] Leaking SignalData structure 0x7b08003679c0 after lost signal to thread 16504
W20260709 12:22:09.204743 17618 debug-util.cc:398] Leaking SignalData structure 0x7b0800035d20 after lost signal to thread 17621
I20260709 12:22:09.204778 17763 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38786 (request call id 2648) took 2318 ms. Trace:
W20260709 12:22:09.206147 17618 debug-util.cc:398] Leaking SignalData structure 0x7b080034a680 after lost signal to thread 17713
I20260709 12:22:09.205292 17763 rpcz_store.cc:276] 0709 12:22:06.886202 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:06.900637 (+ 14435us) service_pool.cc:224] Handling call
0709 12:22:06.901014 (+   377us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:22:06.958104 (+ 57090us) raft_consensus.cc:1534] Early marking committed up to index 2
0709 12:22:06.958134 (+    30us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:22:06.958851 (+   717us) log.cc:844] Serialized 52 byte log entry
0709 12:22:07.131907 (+173056us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:22:07.131962 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:07.132061 (+    99us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:07.915213 (+783152us) spinlock_profiling.cc:243] Waited 28.1 ms on lock 0x7b4400022b4c. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b47459c85 00007f7b4745a6a2 00007f7b4a1cc1db 00007f7b4a1c7aaf 00007f7b4a1bb86e 00007f7b4a1babb6 00007f7b4a1ba7d8 00007f7b4a1c4221 00007f7b4a1c41d9 00007f7b4a1c4169
0709 12:22:09.204463 (+1289250us) raft_consensus.cc:1685] finished
0709 12:22:09.204483 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:09.204697 (+   214us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":7304,"raft.run_cpu_time_us":87,"raft.run_wall_time_us":87,"spinlock_wait_cycles":64689024,"thread_start_us":172569,"threads_started":1,"wal-append.queue_time_us":531655}
W20260709 12:22:09.206459 17618 debug-util.cc:398] Leaking SignalData structure 0x7b08002ec5e0 after lost signal to thread 17714
W20260709 12:22:09.206501 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 355584us
W20260709 12:22:08.844962 17803 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 1.419s	user 0.037s	sys 0.041s
W20260709 12:22:09.207453 17803 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 1.781s	user 0.038s	sys 0.042s
W20260709 12:22:09.207804 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 1.80487e+06us
I20260709 12:22:09.208436 21231 raft_consensus.cc:2804] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:22:09.208961 21231 raft_consensus.cc:493] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:09.209369 21231 raft_consensus.cc:3060] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:22:09.210516 17803 connection.cc:256] RPC call timeout handler was delayed by -2.0501134420001108s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3024s for a response
W20260709 12:22:09.210950 17803 connection.cc:256] RPC call timeout handler was delayed by -2.0415181970001868s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1502s for a response
I20260709 12:22:09.210858 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:22:09.211395 17803 connection.cc:256] RPC call timeout handler was delayed by -2.0381760889999896s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.166s for a response
I20260709 12:22:08.848820 21304 raft_consensus.cc:493] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:09.211993 18362 raft_consensus.cc:2393] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 6: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:22:09.212090 17727 leader_election.cc:304] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:22:09.212142 21304 raft_consensus.cc:515] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:09.213539 17727 leader_election.cc:304] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:09.213994 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:09.214419 21228 raft_consensus.cc:2804] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:22:09.214704 18374 raft_consensus.cc:2468] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 5.
I20260709 12:22:08.853232 21305 raft_consensus.cc:493] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:09.215445 21305 raft_consensus.cc:515] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:09.216202 17727 leader_election.cc:304] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:09.217167 21230 raft_consensus.cc:2804] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:22:09.217749 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:08.854637 21303 raft_consensus.cc:493] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:09.218487 18365 raft_consensus.cc:1275] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260709 12:22:09.218878 21303 raft_consensus.cc:515] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:09.219238 21145 consensus_queue.cc:1048] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20260709 12:22:08.858815 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:08.869731 21306 raft_consensus.cc:493] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:09.235119 21306 raft_consensus.cc:515] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:09.236778 21145 raft_consensus.cc:493] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:09.237370 21306 leader_election.cc:290] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:09.237287 21145 raft_consensus.cc:515] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:09.238942 21318 raft_consensus.cc:2804] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:22:08.905026 17834 raft_consensus.cc:1240] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 5 LEADER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 4. Current term is 5. Ops: []
I20260709 12:22:09.248637 21318 raft_consensus.cc:697] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:22:08.905027 17833 raft_consensus.cc:1275] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260709 12:22:08.908360 17830 raft_consensus.cc:1240] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 4. Current term is 5. Ops: []
W20260709 12:22:08.913285 17832 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 1.369s	user 0.008s	sys 0.008s
W20260709 12:22:09.250670 17832 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.775s	user 0.009s	sys 0.009s
W20260709 12:22:08.913879 18366 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 1.477s	user 0.004s	sys 0.020s
W20260709 12:22:09.252156 18366 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.837s	user 0.005s	sys 0.021s
I20260709 12:22:08.921682 21178 raft_consensus.cc:493] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:08.933245 17804 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:08.940557 17838 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.558s	user 0.058s	sys 0.072s
I20260709 12:22:08.946336 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:22:08.952600 21291 log.cc:927] Time spent T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 1.508s	user 0.022s	sys 0.000s
I20260709 12:22:08.966084 17728 leader_election.cc:304] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:22:08.980867 17764 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38776 (request call id 1983) took 2015 ms. Trace:
I20260709 12:22:07.471676 17685 raft_consensus.cc:1275] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
W20260709 12:22:09.012357 17652 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.559s	user 0.011s	sys 0.029s
I20260709 12:22:09.035774 17837 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35040 (request call id 2174) took 1948 ms. Trace:
I20260709 12:22:09.042522 21085 raft_consensus.cc:493] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:09.045243 18334 leader_election.cc:304] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:22:09.049685 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:22:09.045396 17835 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35040 (request call id 2175) took 1533 ms (client timeout 1661 ms). Trace:
W20260709 12:22:09.060516 17730 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:09.060843 17802 connection.cc:256] RPC call timeout handler was delayed by -1.2650192440000865s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19810000000000003s for a response
I20260709 12:22:09.064687 21240 raft_consensus.cc:493] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:09.074541 17801 leader_election.cc:304] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:09.070012 21307 consensus_queue.cc:1048] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.007s
W20260709 12:22:09.075519 18368 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59918 (request call id 1155) took 1529 ms (client timeout 1556 ms). Trace:
I20260709 12:22:09.075842 21308 consensus_queue.cc:1048] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.004s
W20260709 12:22:09.097580 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 1.72292e+06us
I20260709 12:22:09.120112 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:09.140295 20157 leader_election.cc:290] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 8 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:09.143131 20984 raft_consensus.cc:493] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:09.143994 21313 raft_consensus.cc:493] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:09.145226 21031 raft_consensus.cc:515] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:22:09.146517 21156 consensus_meta.cc:287] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 2.001s	user 0.025s	sys 0.014s
I20260709 12:22:09.155349 17687 raft_consensus.cc:1275] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260709 12:22:09.150508 21060 leader_election.cc:290] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:09.146615 20980 consensus_queue.cc:237] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:09.157467 21315 raft_consensus.cc:493] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:09.170624 21079 leader_election.cc:290] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 8 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:09.171716 21155 raft_consensus.cc:3060] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:09.174840 21311 raft_consensus.cc:2749] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:22:09.180315 17688 raft_consensus.cc:1275] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20260709 12:22:08.838752 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:09.219733 21088 raft_consensus.cc:993] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3: : Instructing follower 334a4e1f59944544b3b3ff36616b57da to start an election
W20260709 12:22:09.221228 17655 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:08.855183 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:09.222193 21237 leader_election.cc:290] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 8 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:09.222435 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:09.218537 18363 raft_consensus.cc:2393] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 7: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
W20260709 12:22:09.224007 17618 debug-util.cc:398] Leaking SignalData structure 0x7b080020f9a0 after lost signal to thread 17788
I20260709 12:22:09.223989 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:22:09.226161 17803 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:09.230059 21303 leader_election.cc:290] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:09.230924 20394 raft_consensus.cc:493] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:09.232589 21305 leader_election.cc:290] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:09.174861 21147 leader_election.cc:290] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:09.239398 21145 leader_election.cc:290] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:08.886757 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:09.239730 21306 raft_consensus.cc:493] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:22:09.241781 21185 raft_consensus.cc:493] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:09.187575 21244 leader_election.cc:290] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:09.244419 21241 raft_consensus.cc:493] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:09.207578 17729 leader_election.cc:304] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:22:09.460167 17686 raft_consensus.cc:2376] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 5. Re-sending same reply.
W20260709 12:22:09.460848 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 253369us
W20260709 12:22:09.461529 17652 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 2.010s	user 0.016s	sys 0.043s
I20260709 12:22:09.461454 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:09.462502 17686 raft_consensus.cc:2410] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 8 because replica has last-logged OpId of term: 7 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 8.
W20260709 12:22:09.462770 17835 rpcz_store.cc:269] 0709 12:22:07.510067 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:07.510255 (+   188us) service_pool.cc:224] Handling call
0709 12:22:09.043241 (+1532986us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:22:09.463258 17803 connection.cc:256] RPC call timeout handler was delayed by -1.6190959520004071s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1807s for a response
W20260709 12:22:09.464068 17655 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:09.464524 17618 debug-util.cc:398] Leaking SignalData structure 0x7b0800358f20 after lost signal to thread 17862
W20260709 12:22:09.297129 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 80988.3us
W20260709 12:22:09.465752 17618 debug-util.cc:398] Leaking SignalData structure 0x7b08003d6320 after lost signal to thread 18409
W20260709 12:22:09.466277 17618 debug-util.cc:398] Leaking SignalData structure 0x7b08001fa400 after lost signal to thread 18410
I20260709 12:22:09.460980 17837 rpcz_store.cc:276] 0709 12:22:07.089401 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:07.411976 (+322575us) service_pool.cc:224] Handling call
0709 12:22:09.035678 (+1623702us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:09.460914 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:09.469038 17689 raft_consensus.cc:3060] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:09.476643 20984 raft_consensus.cc:515] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:09.477048 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:09.477735 21088 consensus_queue.cc:1059] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:22:09.478075 21240 raft_consensus.cc:515] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:09.478432 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:09.478811 21088 consensus_queue.cc:1048] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 12:22:09.479503 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:09.480237 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:09.480743 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:09.480867 17684 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 8.
I20260709 12:22:09.481062 20984 leader_election.cc:290] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:09.295815 21304 leader_election.cc:290] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:09.481510 21313 raft_consensus.cc:515] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:09.482163 21304 raft_consensus.cc:1081] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 6 LEADER]: Signalling peer 334a4e1f59944544b3b3ff36616b57da to start an election
W20260709 12:22:09.465567 18368 rpcz_store.cc:269] 0709 12:22:07.541856 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:07.542087 (+   231us) service_pool.cc:224] Handling call
0709 12:22:09.070814 (+1528727us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:09.464370 17764 rpcz_store.cc:276] 0709 12:22:06.965290 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:06.965508 (+   218us) service_pool.cc:224] Handling call
0709 12:22:08.980662 (+2015154us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":268672}
I20260709 12:22:09.483243 21304 raft_consensus.cc:3055] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:22:09.481498 17686 raft_consensus.cc:2393] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 6: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:22:09.483429 21315 raft_consensus.cc:515] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:09.484089 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:09.484330 21241 raft_consensus.cc:515] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:09.481784 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 10 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:09.482777 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:22:09.487846 17838 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 2.106s	user 0.068s	sys 0.073s
W20260709 12:22:09.489014 17618 debug-util.cc:398] Leaking SignalData structure 0x7b08003e0480 after lost signal to thread 21110
W20260709 12:22:09.490247 17618 debug-util.cc:398] Leaking SignalData structure 0x7b080012daa0 after lost signal to thread 21196
W20260709 12:22:09.490577 17618 debug-util.cc:398] Leaking SignalData structure 0x7b08003a2fc0 after lost signal to thread 21205
I20260709 12:22:09.490898 20157 raft_consensus.cc:493] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:09.491529 20157 raft_consensus.cc:515] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:09.483703 21304 raft_consensus.cc:740] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:22:09.502789 17838 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35040 (request call id 2173) took 2470 ms. Trace:
W20260709 12:22:09.490506 17803 connection.cc:256] RPC call timeout handler was delayed by -1.4742651740002657s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17920000000000003s for a response
W20260709 12:22:09.504261 17803 connection.cc:256] RPC call timeout handler was delayed by -1.4037076999998135s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19810000000000003s for a response
W20260709 12:22:09.504776 17803 connection.cc:256] RPC call timeout handler was delayed by -1.1110763570004565s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21030000000000004s for a response
W20260709 12:22:09.504979 17618 debug-util.cc:398] Leaking SignalData structure 0x7b08003bc680 after lost signal to thread 21208
W20260709 12:22:09.505509 17618 debug-util.cc:398] Leaking SignalData structure 0x7b08003da6c0 after lost signal to thread 21210
W20260709 12:22:09.505999 17618 debug-util.cc:398] Leaking SignalData structure 0x7b0800409960 after lost signal to thread 21232
W20260709 12:22:09.506315 17618 debug-util.cc:398] Leaking SignalData structure 0x7b080033ade0 after lost signal to thread 21234
W20260709 12:22:09.506825 17618 debug-util.cc:398] Leaking SignalData structure 0x7b08003f39c0 after lost signal to thread 21238
W20260709 12:22:09.507162 17618 debug-util.cc:398] Leaking SignalData structure 0x7b08002702a0 after lost signal to thread 21242
W20260709 12:22:09.507489 17618 debug-util.cc:398] Leaking SignalData structure 0x7b08003a3860 after lost signal to thread 21253
W20260709 12:22:09.507800 17618 debug-util.cc:398] Leaking SignalData structure 0x7b08003b4180 after lost signal to thread 21256
W20260709 12:22:09.508096 17618 debug-util.cc:398] Leaking SignalData structure 0x7b08003ba1a0 after lost signal to thread 21257
W20260709 12:22:09.508380 17618 debug-util.cc:398] Leaking SignalData structure 0x7b0800409a20 after lost signal to thread 21258
W20260709 12:22:09.508684 17618 debug-util.cc:398] Leaking SignalData structure 0x7b08001428c0 after lost signal to thread 21261
W20260709 12:22:09.509003 17618 debug-util.cc:398] Leaking SignalData structure 0x7b08003e2700 after lost signal to thread 21263
I20260709 12:22:09.506114 18366 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59886 (request call id 1033) took 2502 ms. Trace:
I20260709 12:22:09.509446 18366 rpcz_store.cc:276] 0709 12:22:07.006809 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:07.014665 (+  7856us) service_pool.cc:224] Handling call
0709 12:22:07.072479 (+ 57814us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:07.402446 (+329967us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 12:22:09.252507 (+1850061us) log.cc:844] Serialized 14 byte log entry
0709 12:22:09.500410 (+247903us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:09.500466 (+    56us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:22:09.500524 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:09.505894 (+  5370us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:09.506045 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":292564,"prepare.run_cpu_time_us":57,"prepare.run_wall_time_us":284,"raft.queue_time_us":114,"raft.run_cpu_time_us":84,"raft.run_wall_time_us":85,"spinlock_wait_cycles":774784,"thread_start_us":2084484,"threads_started":2,"wal-append.queue_time_us":247850}
W20260709 12:22:09.524731 17618 debug-util.cc:398] Leaking SignalData structure 0x7b080027fde0 after lost signal to thread 21268
I20260709 12:22:09.526322 17832 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35042 (request call id 1914) took 2449 ms. Trace:
W20260709 12:22:09.549412 17618 debug-util.cc:398] Leaking SignalData structure 0x7b08003e2760 after lost signal to thread 21269
I20260709 12:22:09.549945 17687 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37402 (request call id 2428) took 2072 ms. Trace:
W20260709 12:22:09.549734 17618 debug-util.cc:398] Leaking SignalData structure 0x7b080038f8c0 after lost signal to thread 21270
W20260709 12:22:09.551175 17618 debug-util.cc:398] Leaking SignalData structure 0x7b08003ba5c0 after lost signal to thread 21275
W20260709 12:22:09.552131 17618 debug-util.cc:398] Leaking SignalData structure 0x7b08003f2d40 after lost signal to thread 21276
W20260709 12:22:09.552356 17618 debug-util.cc:398] Leaking SignalData structure 0x7b080035e260 after lost signal to thread 21277
W20260709 12:22:09.552582 17618 debug-util.cc:398] Leaking SignalData structure 0x7b08003f99e0 after lost signal to thread 21280
W20260709 12:22:09.552865 17618 debug-util.cc:398] Leaking SignalData structure 0x7b0800325fe0 after lost signal to thread 21286
W20260709 12:22:09.553174 17618 debug-util.cc:398] Leaking SignalData structure 0x7b08003ad9a0 after lost signal to thread 21293
I20260709 12:22:09.550664 17687 rpcz_store.cc:276] 0709 12:22:07.478007 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:07.479293 (+  1286us) service_pool.cc:224] Handling call
0709 12:22:07.479648 (+   355us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:09.459832 (+1980184us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:09.549621 (+ 89789us) spinlock_profiling.cc:243] Waited 1.68 s on lock 0x7b5400338954. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:22:09.549865 (+   244us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3852962688}
I20260709 12:22:09.556499 17687 raft_consensus.cc:1275] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 9 index: 10. (index mismatch)
I20260709 12:22:09.558962 21031 leader_election.cc:290] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:22:09.562492 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 100015us
I20260709 12:22:09.545029 21177 raft_consensus.cc:2804] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:22:09.497046 21230 raft_consensus.cc:697] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 6 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:22:09.482102 21156 raft_consensus.cc:515] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:09.570791 20157 leader_election.cc:290] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:09.581197 17682 raft_consensus.cc:2393] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 9: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:22:09.583206 17688 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37402 (request call id 2427) took 2646 ms. Trace:
W20260709 12:22:09.581055 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 535900us
I20260709 12:22:09.582068 17685 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37402 (request call id 2423) took 2735 ms. Trace:
I20260709 12:22:09.580719 17684 raft_consensus.cc:1275] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260709 12:22:09.583813 17688 rpcz_store.cc:276] 0709 12:22:06.919791 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:07.415681 (+495890us) service_pool.cc:224] Handling call
0709 12:22:07.425529 (+  9848us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:09.463601 (+2038072us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:09.565831 (+102230us) spinlock_profiling.cc:243] Waited 1.75 s on lock 0x7b54001388d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:22:09.566049 (+   218us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4034857856}
I20260709 12:22:09.584125 17685 rpcz_store.cc:276] 0709 12:22:06.827456 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:06.831215 (+  3759us) service_pool.cc:224] Handling call
0709 12:22:06.831557 (+   342us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:09.467997 (+2636440us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:09.563104 (+ 95107us) spinlock_profiling.cc:243] Waited 625 ms on lock 0x7b5400206a54. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:22:09.563254 (+   150us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1436981888}
I20260709 12:22:09.503506 17838 rpcz_store.cc:276] 0709 12:22:07.059341 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:07.063746 (+  4405us) service_pool.cc:224] Handling call
0709 12:22:07.081783 (+ 18037us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:07.381196 (+299413us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:07.381223 (+    27us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:07.381746 (+   523us) log.cc:844] Serialized 26 byte log entry
0709 12:22:09.488329 (+2106583us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:22:09.488365 (+    36us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:09.488427 (+    62us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:09.488463 (+    36us) raft_consensus.cc:1685] finished
0709 12:22:09.488474 (+    11us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:09.488576 (+   102us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":2158,"raft.run_cpu_time_us":157,"raft.run_wall_time_us":158,"thread_start_us":2106138,"threads_started":1,"wal-append.queue_time_us":287938,"wal-append.run_cpu_time_us":3611,"wal-append.run_wall_time_us":1030677}
I20260709 12:22:09.489696 21313 leader_election.cc:290] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:09.562274 18371 raft_consensus.cc:2468] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 10.
I20260709 12:22:09.588963 17687 raft_consensus.cc:1275] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20260709 12:22:09.514214 21309 raft_consensus.cc:493] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:09.484022 21185 raft_consensus.cc:515] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:09.489650 21315 leader_election.cc:290] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:09.538998 21053 raft_consensus.cc:493] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:09.563628 21244 raft_consensus.cc:493] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:09.525441 21060 raft_consensus.cc:493] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:09.549708 17832 rpcz_store.cc:276] 0709 12:22:07.077355 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:07.077574 (+   219us) service_pool.cc:224] Handling call
0709 12:22:07.382663 (+305089us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:07.383243 (+   580us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:07.383269 (+    26us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:07.383899 (+   630us) log.cc:844] Serialized 26 byte log entry
0709 12:22:07.407838 (+ 23939us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:22:09.251071 (+1843233us) log.cc:844] Serialized 14 byte log entry
0709 12:22:09.526006 (+274935us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:09.526084 (+    78us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:09.526123 (+    39us) raft_consensus.cc:1685] finished
0709 12:22:09.526145 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:09.526267 (+   122us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":29700,"prepare.queue_time_us":291633,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":40,"raft.queue_time_us":444,"raft.run_cpu_time_us":126,"raft.run_wall_time_us":126,"thread_start_us":2043408,"threads_started":3,"wal-append.queue_time_us":381127,"wal-append.run_cpu_time_us":2710,"wal-append.run_wall_time_us":1002737}
W20260709 12:22:09.592458 17803 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.16.29.68:42577 timed out after 1.502s (SENT) [suppressed 14 similar messages]
I20260709 12:22:09.605881 17688 raft_consensus.cc:1240] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 5. Current term is 6. Ops: [5.7-5.7]
W20260709 12:22:09.610461 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 644583us
I20260709 12:22:09.611510 20157 raft_consensus.cc:2804] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:22:09.563371 21079 raft_consensus.cc:493] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:09.569958 21308 raft_consensus.cc:493] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:09.621840 20157 raft_consensus.cc:697] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 8 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:22:09.502769 21085 raft_consensus.cc:515] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:09.625239 21319 raft_consensus.cc:493] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:09.625973 21331 raft_consensus.cc:493] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:09.626593 21332 raft_consensus.cc:2749] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:22:09.253258 21178 raft_consensus.cc:515] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:09.504817 21231 raft_consensus.cc:515] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:22:09.630996 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 556519us
I20260709 12:22:09.627753 17683 raft_consensus.cc:2468] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 6.
I20260709 12:22:09.586021 20394 raft_consensus.cc:515] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:09.596393 21228 raft_consensus.cc:697] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 5 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:22:09.526306 21306 raft_consensus.cc:515] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:09.634650 21079 raft_consensus.cc:515] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:09.634500 21308 raft_consensus.cc:515] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:09.640547 17728 leader_election.cc:304] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
W20260709 12:22:09.639735 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 47323.6us
W20260709 12:22:09.636976 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 73874.5us
I20260709 12:22:09.638139 21312 consensus_queue.cc:1048] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.011s
I20260709 12:22:09.622442 21031 raft_consensus.cc:2804] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:22:09.618990 21147 raft_consensus.cc:493] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
W20260709 12:22:09.670050 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 64885.6us
I20260709 12:22:09.663133 17759 raft_consensus.cc:1240] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 4. Current term is 5. Ops: []
I20260709 12:22:09.627948 21237 raft_consensus.cc:493] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:09.662593 20157 consensus_queue.cc:237] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:09.668303 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:09.674007 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:09.674729 18365 raft_consensus.cc:2468] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 8.
I20260709 12:22:09.253182 21318 consensus_queue.cc:237] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:09.628468 20984 consensus_queue.cc:1059] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.039s
I20260709 12:22:09.623485 21244 raft_consensus.cc:515] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:09.627803 21060 raft_consensus.cc:515] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:09.612531 21304 consensus_queue.cc:260] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:09.628221 21053 raft_consensus.cc:515] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:09.662276 21332 raft_consensus.cc:493] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:22:09.690646 18333 leader_election.cc:304] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:09.701035 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:09.700866 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:22:09.693648 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 53203.1us
I20260709 12:22:09.707444 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:09.611864 21309 raft_consensus.cc:515] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:09.706308 18361 raft_consensus.cc:1275] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260709 12:22:09.664336 21327 raft_consensus.cc:493] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
W20260709 12:22:09.711951 18334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:09.728204 18334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:09.728710 18334 connection.cc:256] RPC call timeout handler was delayed by -1.9720684750000146s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 1.5s for a response
W20260709 12:22:09.729189 18334 connection.cc:256] RPC call timeout handler was delayed by -1.5033288039999206s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.164s for a response
I20260709 12:22:09.686390 18362 raft_consensus.cc:2468] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 7.
I20260709 12:22:09.715555 17762 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38786 (request call id 2635) took 2831 ms. Trace:
I20260709 12:22:09.715668 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 13 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:09.715979 21334 raft_consensus.cc:493] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
W20260709 12:22:09.717532 17584 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:37172 (request call id 253) took 15364 ms (client timeout 15000 ms). Trace:
I20260709 12:22:09.717869 21053 leader_election.cc:290] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:09.688685 21241 leader_election.cc:290] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:09.718194 21228 consensus_queue.cc:237] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:09.719357 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:09.671620 21031 raft_consensus.cc:493] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:09.710048 17832 raft_consensus.cc:2468] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 5.
I20260709 12:22:09.721678 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:09.644446 21230 consensus_queue.cc:237] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:09.726639 17837 raft_consensus.cc:2468] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 5.
I20260709 12:22:09.799633 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:09.801403 17685 raft_consensus.cc:1240] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 6. Current term is 7. Ops: [3.2-6.3]
W20260709 12:22:09.728263 17803 leader_election.cc:336] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Timed out: RequestConsensusVote RPC to 127.16.29.68:42577 timed out after 1.792s (SENT)
I20260709 12:22:09.782624 17759 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38776 (request call id 1982) took 2817 ms. Trace:
I20260709 12:22:09.799454 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:09.637404 21331 raft_consensus.cc:515] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:09.806100 17728 leader_election.cc:304] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:22:09.720602 21177 raft_consensus.cc:697] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:22:09.682826 21056 consensus_queue.cc:1048] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.099s
I20260709 12:22:09.719977 21317 raft_consensus.cc:493] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:22:09.683836 21237 raft_consensus.cc:515] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:09.802507 17759 rpcz_store.cc:276] 0709 12:22:06.964797 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:06.964940 (+   143us) service_pool.cc:224] Handling call
0709 12:22:09.662731 (+2697791us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:09.671023 (+  8292us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:09.777533 (+106510us) spinlock_profiling.cc:243] Waited 2.7 s on lock 0x7b54000fbed0. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a61f8ad 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:22:09.782543 (+  5010us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":6203340672}
I20260709 12:22:09.720049 18365 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad"
dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
 from {username='slave'} at 127.0.0.1:59918
I20260709 12:22:09.730716 17762 rpcz_store.cc:276] 0709 12:22:06.838160 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:06.838374 (+   214us) service_pool.cc:224] Handling call
0709 12:22:06.838807 (+   433us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:06.846263 (+  7456us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:06.846301 (+    38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:06.846963 (+   662us) log.cc:844] Serialized 26 byte log entry
0709 12:22:07.016763 (+169800us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:22:07.016823 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:07.016920 (+    97us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:08.951846 (+1934926us) spinlock_profiling.cc:243] Waited 31.5 ms on lock 0x7b4400022b4c. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b47459c85 00007f7b4745a6a2 00007f7b4a1cc1db 00007f7b4a1c7aaf 00007f7b4a1bb86e 00007f7b4a1babb6 00007f7b4a1ba7d8 00007f7b4a1c4221 00007f7b4a1c41d9 00007f7b4a1c4169
0709 12:22:09.662489 (+710643us) raft_consensus.cc:1685] finished
0709 12:22:09.662514 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:09.669580 (+  7066us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":6872,"raft.queue_time_us":8239,"raft.run_cpu_time_us":174,"raft.run_wall_time_us":10224,"spinlock_wait_cycles":72546688,"thread_start_us":157288,"threads_started":1,"wal-append.queue_time_us":342337}
I20260709 12:22:09.808408 17838 raft_consensus.cc:2393] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 7: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:22:09.688349 21147 raft_consensus.cc:515] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:09.802286 17803 leader_election.cc:304] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:09.810528 18368 raft_consensus.cc:3055] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:22:09.816144 18368 raft_consensus.cc:740] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:22:09.694875 21307 consensus_queue.cc:1048] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.109s
I20260709 12:22:09.796865 21053 raft_consensus.cc:493] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:09.792867 20984 raft_consensus.cc:2749] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260709 12:22:09.788793 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 98698.2us
I20260709 12:22:09.815145 18365 raft_consensus.cc:493] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:22:09.800184 21031 raft_consensus.cc:3060] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:09.642028 21319 raft_consensus.cc:515] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:09.834398 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:09.834895 18365 raft_consensus.cc:3060] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:09.718231 21332 raft_consensus.cc:515] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:22:09.843008 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 133063us
W20260709 12:22:09.849689 17652 connection.cc:256] RPC call timeout handler was delayed by -1.3179958820001048s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16610000000000003s for a response
W20260709 12:22:09.850229 17652 connection.cc:256] RPC call timeout handler was delayed by -1.1142681920000541s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1652s for a response
W20260709 12:22:09.862180 17655 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:09.816658 21304 raft_consensus.cc:3060] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:09.830238 21305 raft_consensus.cc:493] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:09.839596 18368 consensus_queue.cc:260] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:09.829986 21336 raft_consensus.cc:493] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:09.909302 17834 raft_consensus.cc:1275] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260709 12:22:09.810883 21322 raft_consensus.cc:493] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:09.840437 17833 raft_consensus.cc:2468] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 9.
I20260709 12:22:09.811282 21241 raft_consensus.cc:493] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:09.858112 21240 leader_election.cc:290] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:22:09.922591 17655 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:09.792023 21334 raft_consensus.cc:515] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:09.903000 21156 leader_election.cc:290] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 12 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:09.937637 17762 raft_consensus.cc:1275] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260709 12:22:09.806842 21327 raft_consensus.cc:515] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:09.953972 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 13 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:09.938414 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:09.938510 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:09.954999 17758 raft_consensus.cc:3060] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:09.939230 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:22:09.941885 17803 leader_election.cc:336] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Timed out: RequestConsensusVote RPC to 127.16.29.68:42577 timed out after 2.137s (SENT)
I20260709 12:22:09.955329 17757 raft_consensus.cc:2468] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 8.
I20260709 12:22:09.944212 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:09.944033 21177 consensus_queue.cc:237] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:09.945016 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:09.938211 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:22:09.960462 17586 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35962 (request call id 355) took 26456 ms (client timeout 15000 ms). Trace:
W20260709 12:22:09.770368 17584 rpcz_store.cc:269] 0709 12:21:54.312543 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:54.312835 (+   292us) service_pool.cc:224] Handling call
0709 12:22:09.669049 (+15356214us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:22:06.327815 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:22:06.471451 (+143636us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:22:06.499962 (+ 28511us) write_op.cc:435] Acquired schema lock
0709 12:22:06.499994 (+    32us) tablet.cc:662] Decoding operations
0709 12:22:06.501810 (+  1816us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:22:06.501913 (+   103us) write_op.cc:641] Partition lock acquired for write op
0709 12:22:06.501946 (+    33us) tablet.cc:685] Acquiring locks for 20 operations
0709 12:22:06.568424 (+ 66478us) tablet.cc:701] Row locks acquired
0709 12:22:06.568491 (+    67us) write_op.cc:260] PREPARE: finished
0709 12:22:06.568796 (+   305us) write_op.cc:270] Start()
0709 12:22:06.569012 (+   216us) write_op.cc:276] Timestamp: P: 1783599726568669 usec, L: 0
0709 12:22:06.569036 (+    24us) op_driver.cc:348] REPLICATION: starting
0709 12:22:06.570062 (+  1026us) log.cc:844] Serialized 7816 byte log entry
0709 12:22:06.949740 (+379678us) op_driver.cc:464] REPLICATION: finished
0709 12:22:07.435033 (+485293us) spinlock_profiling.cc:243] Waited 37.7 ms on lock 0x7b540010d454. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a61d32a 00007f7b4a5a2f14 00007f7b4a5a2ec3 00007f7b4a5a2e38 00007f7b4a5a2df3 00007f7b4a5a20dc 00007f7b4a5cbe16 00007f7b4a5b3cbb 00007f7b4a59e39d 00007f7b4a5a1d75
0709 12:22:07.582049 (+147016us) write_op.cc:301] APPLY: starting
0709 12:22:07.582208 (+   159us) tablet.cc:1367] starting BulkCheckPresence
0709 12:22:07.582967 (+   759us) tablet.cc:1370] finished BulkCheckPresence
0709 12:22:07.583027 (+    60us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:22:07.877506 (+294479us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:22:07.877928 (+   422us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=20
0709 12:22:07.877981 (+    53us) write_op.cc:312] APPLY: finished
0709 12:22:07.878659 (+   678us) log.cc:844] Serialized 139 byte log entry
0709 12:22:07.878913 (+   254us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:22:07.879319 (+   406us) write_op.cc:454] Released schema lock
0709 12:22:07.879406 (+    87us) write_op.cc:341] FINISH: Updating metrics
0709 12:22:07.884400 (+  4994us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:22:07.884436 (+    36us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":2682720,"reported_tablets":33,"child_traces":[["op",{"apply.queue_time_us":520546,"apply.run_cpu_time_us":10561,"apply.run_wall_time_us":297879,"num_ops":20,"prepare.queue_time_us":132778,"prepare.run_cpu_time_us":12551,"prepare.run_wall_time_us":259872,"raft.queue_time_us":240606,"raft.run_cpu_time_us":61707,"raft.run_wall_time_us":620368,"replication_time_us":318299,"spinlock_wait_cycles":87090432,"thread_start_us":325863,"threads_started":4,"wal-append.queue_time_us":1125}]]}
I20260709 12:22:09.944916 17583 catalog_manager.cc:5697] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 reported cstate change: term changed from 4 to 7, leader changed from 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) to e063541542a94969898868bf6eb5ce1b (127.16.29.67). New cstate: current_term: 7 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } }
I20260709 12:22:09.947141 21308 leader_election.cc:290] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:09.947822 21349 raft_consensus.cc:2804] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Leader pre-election won for term 10
W20260709 12:22:09.948462 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 275472us
I20260709 12:22:09.948935 18368 raft_consensus.cc:3060] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:09.949378 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:10.011376 17756 raft_consensus.cc:2393] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 10: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:22:09.949541 21060 leader_election.cc:290] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:09.921545 17652 leader_election.cc:304] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:09.950048 21178 leader_election.cc:290] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:09.934782 21336 raft_consensus.cc:515] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:09.936041 21305 raft_consensus.cc:515] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:09.955624 17803 leader_election.cc:304] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:09.924816 21312 raft_consensus.cc:2749] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:22:09.955638 17760 raft_consensus.cc:2468] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 6.
I20260709 12:22:09.956988 17761 raft_consensus.cc:2468] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 8.
W20260709 12:22:09.984189 21338 log.cc:927] Time spent T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.059s	user 0.001s	sys 0.000s
I20260709 12:22:10.124459 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:09.961336 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:09.848634 21053 raft_consensus.cc:515] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:22:09.961035 17586 rpcz_store.cc:269] 0709 12:21:43.255543 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:43.255829 (+   286us) service_pool.cc:224] Handling call
0709 12:22:09.694400 (+26438571us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":129210,"reported_tablets":28}
I20260709 12:22:09.961058 17763 raft_consensus.cc:2468] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 6.
I20260709 12:22:09.999158 17686 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37402 (request call id 2436) took 1158 ms. Trace:
W20260709 12:22:09.968598 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:09.962478 17583 catalog_manager.cc:5697] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 reported cstate change: term changed from 2 to 6, leader changed from 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) to 0da169ff82674dc1947055ce43de68a3 (127.16.29.65). New cstate: current_term: 6 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:22:10.116622 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:22:10.117444 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 431267us
W20260709 12:22:10.081005 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 62846.2us
I20260709 12:22:09.986603 21147 leader_election.cc:290] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:09.909229 20980 consensus_queue.cc:1048] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.041s
I20260709 12:22:09.940659 21241 raft_consensus.cc:515] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:09.949674 21331 leader_election.cc:290] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:09.948464 21306 leader_election.cc:290] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:09.826711 21317 raft_consensus.cc:515] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:10.146509 18371 raft_consensus.cc:2468] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 6.
W20260709 12:22:09.982893 21346 log.cc:927] Time spent T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.053s	user 0.000s	sys 0.000s
I20260709 12:22:09.954347 21240 raft_consensus.cc:493] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:09.960806 21085 leader_election.cc:290] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:10.122587 17802 leader_election.cc:304] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:09.987901 21332 leader_election.cc:290] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:09.993423 21327 leader_election.cc:290] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:09.983037 21244 leader_election.cc:290] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:10.514392 17654 leader_election.cc:304] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
W20260709 12:22:10.501377 18365 consensus_meta.cc:287] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.652s	user 0.024s	sys 0.000s
I20260709 12:22:10.157051 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:10.146514 17759 raft_consensus.cc:2468] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 7.
I20260709 12:22:10.189889 17760 raft_consensus.cc:1275] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260709 12:22:10.493419 18368 raft_consensus.cc:1275] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 6. (term mismatch)
W20260709 12:22:10.504611 17689 consensus_meta.cc:287] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 1.002s	user 0.007s	sys 0.018s
I20260709 12:22:10.519011 17689 raft_consensus.cc:2468] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 7.
I20260709 12:22:10.518748 17583 catalog_manager.cc:5697] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 reported cstate change: term changed from 3 to 4, leader changed from e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66) to 0da169ff82674dc1947055ce43de68a3 (127.16.29.65). New cstate: current_term: 4 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: HEALTHY } } }
I20260709 12:22:10.520165 17681 raft_consensus.cc:1240] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 6. Current term is 7. Ops: []
W20260709 12:22:10.156548 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 235130us
W20260709 12:22:10.342893 17680 consensus_meta.cc:287] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.839s	user 0.007s	sys 0.004s
I20260709 12:22:09.977623 21343 raft_consensus.cc:2749] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:22:09.984876 21321 raft_consensus.cc:493] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:09.984146 21324 raft_consensus.cc:493] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:10.013600 21349 raft_consensus.cc:493] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:09.984532 21329 raft_consensus.cc:493] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:10.520253 17729 leader_election.cc:304] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:10.524816 17680 raft_consensus.cc:2468] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 8.
I20260709 12:22:10.125063 21079 leader_election.cc:290] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:22:10.504894 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 713915us
W20260709 12:22:10.368013 17655 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:10.247710 21347 raft_consensus.cc:493] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:10.515828 18365 raft_consensus.cc:515] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:09.961310 21322 raft_consensus.cc:515] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:10.561089 21053 leader_election.cc:290] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:10.560973 21088 consensus_queue.cc:1048] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 12:22:10.561729 21343 raft_consensus.cc:2804] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:22:10.562144 21343 raft_consensus.cc:493] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:09.962493 21231 leader_election.cc:290] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 9 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:10.562467 21322 leader_election.cc:290] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:10.562971 18374 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 1159) took 1346 ms. Trace:
I20260709 12:22:10.564049 18365 leader_election.cc:290] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 7 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:10.564214 17729 leader_election.cc:304] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:09.996266 21185 leader_election.cc:290] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:22:10.565372 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 485534us
I20260709 12:22:10.565534 21241 leader_election.cc:290] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:22:10.520768 17689 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37402 (request call id 2441) took 1472 ms (client timeout 1502 ms). Trace:
I20260709 12:22:10.558599 21329 raft_consensus.cc:515] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:22:10.566601 17689 rpcz_store.cc:269] 0709 12:22:09.048447 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:09.460531 (+412084us) service_pool.cc:224] Handling call
0709 12:22:10.520678 (+1060147us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:10.567549 21244 raft_consensus.cc:2804] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:22:09.986125 21145 raft_consensus.cc:493] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:22:10.518152 18368 pending_rounds.cc:85] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da: Aborting all ops after (but not including) 5
I20260709 12:22:10.532388 17654 leader_election.cc:304] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:22:10.568214 21329 leader_election.cc:290] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:10.568451 21331 raft_consensus.cc:493] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:10.569259 21342 consensus_queue.cc:1048] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:22:10.570068 18368 pending_rounds.cc:107] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da: Aborting uncommitted NO_OP operation due to leader change: 5.6
I20260709 12:22:10.570411 18368 raft_consensus.cc:2889] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:10.570212 21331 raft_consensus.cc:515] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:10.571079 21342 consensus_queue.cc:1059] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 12:22:10.568254 21145 raft_consensus.cc:515] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:10.571792 21079 consensus_queue.cc:1059] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:22:10.527020 21304 raft_consensus.cc:2804] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:22:10.117671 20394 leader_election.cc:290] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:10.582717 17802 leader_election.cc:304] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:10.515666 17686 rpcz_store.cc:276] 0709 12:22:08.840999 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:09.485868 (+644869us) service_pool.cc:224] Handling call
0709 12:22:09.526082 (+ 40214us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:09.627607 (+101525us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:09.627638 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:09.650088 (+ 22450us) log.cc:844] Serialized 26 byte log entry
0709 12:22:09.818154 (+168066us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:22:09.818192 (+    38us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:09.818256 (+    64us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:09.998963 (+180707us) raft_consensus.cc:1685] finished
0709 12:22:09.998986 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:09.999102 (+   116us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":120635,"raft.queue_time_us":107977,"raft.run_cpu_time_us":94,"raft.run_wall_time_us":95,"thread_start_us":124199,"threads_started":1,"wal-append.queue_time_us":311812}
I20260709 12:22:10.545364 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:10.545599 17681 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44898 (request call id 760) took 1338 ms. Trace:
I20260709 12:22:10.584851 17681 rpcz_store.cc:276] 0709 12:22:09.206778 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:09.588521 (+381743us) service_pool.cc:224] Handling call
0709 12:22:10.519912 (+931391us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:10.523350 (+  3438us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:10.532572 (+  9222us) spinlock_profiling.cc:243] Waited 905 ms on lock 0x7b54004277d0. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a61f8ad 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:22:10.545513 (+ 12941us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2082266496}
I20260709 12:22:10.121874 21060 raft_consensus.cc:2749] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:22:10.014133 21156 raft_consensus.cc:493] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:10.555634 17680 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37402 (request call id 2432) took 1717 ms (client timeout 2207 ms). Trace:
W20260709 12:22:10.605793 17680 rpcz_store.cc:269] 0709 12:22:08.838808 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:08.886359 (+ 47551us) service_pool.cc:224] Handling call
0709 12:22:10.555572 (+1669213us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:22:10.513716 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 571794us
W20260709 12:22:10.522226 21155 consensus_meta.cc:287] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 1.031s	user 0.013s	sys 0.000s
I20260709 12:22:10.156504 21178 raft_consensus.cc:2749] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:22:10.372711 21337 raft_consensus.cc:493] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:10.155077 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 384755us
I20260709 12:22:10.544749 21357 raft_consensus.cc:493] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:10.527794 17583 catalog_manager.cc:5697] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 reported cstate change: term changed from 2 to 6. New cstate: current_term: 6 leader_uuid: "334a4e1f59944544b3b3ff36616b57da" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:22:10.564960 21333 raft_consensus.cc:2749] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 12:22:10.562491 21343 raft_consensus.cc:3060] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:10.567981 21244 raft_consensus.cc:493] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:22:10.186522 17762 raft_consensus.cc:1275] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260709 12:22:10.568858 21329 raft_consensus.cc:2804] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:22:10.569398 17654 leader_election.cc:304] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:22:10.572115 21335 consensus_queue.cc:1059] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 8.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260709 12:22:10.515569 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 367394us
I20260709 12:22:10.517180 18367 raft_consensus.cc:3060] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:22:10.538019 17758 consensus_meta.cc:287] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.576s	user 0.010s	sys 0.001s
I20260709 12:22:09.997157 21309 leader_election.cc:290] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:10.521944 21305 leader_election.cc:290] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:10.515472 21336 leader_election.cc:290] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:10.563344 18374 rpcz_store.cc:276] 0709 12:22:09.216437 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:09.216590 (+   153us) service_pool.cc:224] Handling call
0709 12:22:09.216899 (+   309us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:09.479313 (+262414us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 12:22:09.590078 (+110765us) log.cc:844] Serialized 14 byte log entry
0709 12:22:10.418322 (+828244us) log.cc:844] Serialized 14 byte log entry
0709 12:22:10.502111 (+ 83789us) log.cc:844] Serialized 14 byte log entry
0709 12:22:10.545824 (+ 43713us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:10.545878 (+    54us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:22:10.545923 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:10.562661 (+ 16738us) spinlock_profiling.cc:243] Waited 262 ms on lock 0x7b5400466ed4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:22:10.562704 (+    43us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:10.562897 (+   193us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":73608,"prepare.queue_time_us":158639,"prepare.run_cpu_time_us":132,"prepare.run_wall_time_us":5869,"raft.queue_time_us":253873,"raft.run_cpu_time_us":405,"raft.run_wall_time_us":25042,"spinlock_wait_cycles":602938240,"thread_start_us":839894,"threads_started":4,"wal-append.queue_time_us":206652}
W20260709 12:22:10.572724 21311 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 6 current term: 6 is higher.
I20260709 12:22:10.582178 21304 raft_consensus.cc:493] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:10.604851 17756 raft_consensus.cc:2393] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 6: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:22:10.773247 17758 raft_consensus.cc:2410] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for term 8 because replica has last-logged OpId of term: 7 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 3.
I20260709 12:22:10.607698 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:10.609120 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:22:10.774878 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 205537us
I20260709 12:22:10.775594 18365 raft_consensus.cc:2468] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 9.
I20260709 12:22:10.529343 21240 raft_consensus.cc:515] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:10.555091 21327 raft_consensus.cc:493] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:22:10.397250 21031 raft_consensus.cc:515] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:10.555327 21306 consensus_queue.cc:1048] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.062s
I20260709 12:22:10.776329 18368 raft_consensus.cc:2468] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 10.
I20260709 12:22:10.777595 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:10.626492 20921 raft_consensus.cc:2804] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Leader election won for term 5
W20260709 12:22:10.778637 17801 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.852s	user 0.073s	sys 0.118s
W20260709 12:22:10.795356 17801 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.870s	user 0.074s	sys 0.118s
I20260709 12:22:10.557406 21349 raft_consensus.cc:3060] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:10.779554 17729 leader_election.cc:304] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:10.554921 21321 raft_consensus.cc:515] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:10.782835 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:22:10.786667 17727 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:10.787413 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:10.661175 21319 leader_election.cc:290] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:10.784304 17583 catalog_manager.cc:5697] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 12:22:10.616135 21230 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.618s	user 0.027s	sys 0.017s
W20260709 12:22:10.776913 21318 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.532s	user 0.001s	sys 0.011s
W20260709 12:22:10.645318 21346 log.cc:927] Time spent T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.064s	user 0.000s	sys 0.000s
I20260709 12:22:10.813051 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:10.807168 17583 catalog_manager.cc:5697] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 reported cstate change: term changed from 6 to 8. New cstate: current_term: 8 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 12:22:10.778085 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 199019us
I20260709 12:22:10.814759 17652 leader_election.cc:304] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:22:10.610198 21334 leader_election.cc:290] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:10.796438 17763 raft_consensus.cc:2368] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 8. Current term is 9.
I20260709 12:22:10.669617 21361 raft_consensus.cc:3055] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:22:10.797075 21369 raft_consensus.cc:2804] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:22:10.796484 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:10.779052 21231 raft_consensus.cc:493] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:10.779634 21304 raft_consensus.cc:3060] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:10.782953 21329 raft_consensus.cc:493] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:22:10.799113 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 871400us
I20260709 12:22:10.779402 21244 raft_consensus.cc:3060] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:10.804840 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:10.806059 17687 raft_consensus.cc:1275] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260709 12:22:10.560163 21324 raft_consensus.cc:515] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:10.783160 21060 raft_consensus.cc:2804] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:22:10.813019 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 13 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:22:10.789109 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 215918us
I20260709 12:22:10.814528 17583 catalog_manager.cc:5697] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 reported cstate change: term changed from 2 to 8, leader changed from e063541542a94969898868bf6eb5ce1b (127.16.29.67) to 0da169ff82674dc1947055ce43de68a3 (127.16.29.65). New cstate: current_term: 8 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: UNKNOWN } } }
I20260709 12:22:10.812850 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:10.820843 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:10.921970 17683 raft_consensus.cc:2468] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 12.
I20260709 12:22:10.922328 18364 raft_consensus.cc:3060] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:10.928326 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:10.929126 18374 raft_consensus.cc:2468] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 6.
I20260709 12:22:10.929577 21237 leader_election.cc:290] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:10.930696 17583 catalog_manager.cc:5697] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 reported cstate change: term changed from 1 to 5, leader changed from e063541542a94969898868bf6eb5ce1b (127.16.29.67) to 0da169ff82674dc1947055ce43de68a3 (127.16.29.65). New cstate: current_term: 5 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: UNKNOWN } } }
I20260709 12:22:10.931670 21321 leader_election.cc:290] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:10.932173 21307 raft_consensus.cc:3055] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:22:10.932598 21307 raft_consensus.cc:740] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 8 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:22:10.932433 17583 catalog_manager.cc:5697] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 reported cstate change: term changed from 1 to 6, leader changed from 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) to e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66). New cstate: current_term: 6 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:22:10.933480 21307 consensus_queue.cc:260] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:10.933982 17583 catalog_manager.cc:5697] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 reported cstate change: term changed from 1 to 6. New cstate: current_term: 6 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: HEALTHY } } }
I20260709 12:22:10.934841 21307 raft_consensus.cc:3060] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:10.908416 21318 raft_consensus.cc:493] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:10.787607 21327 raft_consensus.cc:515] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:10.863449 20921 raft_consensus.cc:697] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 5 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:22:10.936072 21318 raft_consensus.cc:515] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:10.936833 20921 consensus_queue.cc:237] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:10.938282 21318 leader_election.cc:290] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:10.938803 21318 raft_consensus.cc:493] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:22:10.941215 17761 raft_consensus.cc:1275] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260709 12:22:10.941620 17763 raft_consensus.cc:1240] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 9. Current term is 10. Ops: []
I20260709 12:22:10.942355 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:10.783085 21354 consensus_queue.cc:1048] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.067s
I20260709 12:22:10.824067 21361 raft_consensus.cc:740] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 8 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
W20260709 12:22:10.945990 21346 log.cc:927] Time spent T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.057s	user 0.001s	sys 0.000s
I20260709 12:22:10.945535 21361 consensus_queue.cc:260] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:10.946830 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:10.947606 17762 raft_consensus.cc:2468] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 8.
I20260709 12:22:10.949071 21031 leader_election.cc:290] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 6 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:10.950547 21331 leader_election.cc:290] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:10.951097 21060 raft_consensus.cc:493] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:10.951469 21060 raft_consensus.cc:3060] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:10.952647 21088 raft_consensus.cc:2749] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:22:10.953135 21308 raft_consensus.cc:3055] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:22:10.953529 21308 raft_consensus.cc:740] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 7 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:22:10.954347 21308 consensus_queue.cc:260] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:10.955276 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:10.955507 21308 raft_consensus.cc:3060] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:10.918776 21231 raft_consensus.cc:515] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:10.956485 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:10.957753 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:10.958700 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:22:10.960134 17804 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:10.962522 17763 raft_consensus.cc:1240] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 7. Current term is 8. Ops: []
I20260709 12:22:10.819764 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:22:10.806006 21230 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.808s	user 0.042s	sys 0.060s
I20260709 12:22:10.939272 21318 raft_consensus.cc:515] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:10.850059 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: true dest_uuid: "0da169ff82674dc1947055ce43de68a3"
W20260709 12:22:10.890357 18334 outbound_call.cc:321] RPC callback for RPC call kudu.master.MasterService.TSHeartbeat -> {remote=127.16.29.126:39149, user_credentials={real_user=slave}} blocked reactor thread for 67678.6us
I20260709 12:22:10.773034 21156 raft_consensus.cc:515] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:10.779932 21155 raft_consensus.cc:515] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:10.808429 21145 leader_election.cc:290] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:10.606808 21347 raft_consensus.cc:515] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:10.913033 18367 raft_consensus.cc:2468] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 9.
I20260709 12:22:10.789700 20394 consensus_queue.cc:1048] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.037s
I20260709 12:22:10.791747 21322 consensus_queue.cc:1048] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.058s
I20260709 12:22:10.823405 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:10.921362 21329 raft_consensus.cc:3060] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:10.937546 21327 leader_election.cc:290] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:10.970397 17680 raft_consensus.cc:3055] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:22:10.996183 17680 raft_consensus.cc:740] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:22:10.967128 17583 catalog_manager.cc:5697] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 reported cstate change: term changed from 5 to 8, leader changed from e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66) to 0da169ff82674dc1947055ce43de68a3 (127.16.29.65). New cstate: current_term: 8 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: HEALTHY } } }
I20260709 12:22:10.929143 21240 leader_election.cc:290] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:10.968966 17654 leader_election.cc:304] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:22:10.972034 18364 raft_consensus.cc:2468] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 12.
I20260709 12:22:10.976861 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:11.034557 18362 raft_consensus.cc:2468] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 6.
I20260709 12:22:10.977290 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:10.975039 17729 leader_election.cc:304] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:22:11.036474 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 61510.2us
I20260709 12:22:11.037622 17729 leader_election.cc:304] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:10.978350 18363 raft_consensus.cc:2393] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 7: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:22:10.976765 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:10.910579 21177 raft_consensus.cc:493] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:10.980942 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:10.983068 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:10.785864 21337 raft_consensus.cc:515] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:10.985236 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:10.963160 21231 leader_election.cc:290] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:10.848539 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:10.784507 21357 raft_consensus.cc:515] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:11.103411 17837 raft_consensus.cc:3060] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:10.988441 21360 consensus_queue.cc:1048] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.009s
I20260709 12:22:10.989090 21156 leader_election.cc:290] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:11.038720 17729 leader_election.cc:304] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:22:11.049343 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 435774us
W20260709 12:22:11.040987 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 72236.8us
I20260709 12:22:11.082856 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:11.096302 21230 raft_consensus.cc:2804] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:22:11.087082 18365 raft_consensus.cc:2393] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 11: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:22:11.044487 17583 catalog_manager.cc:5697] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 reported cstate change: term changed from 9 to 10, leader changed from 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) to e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66). New cstate: current_term: 10 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:22:11.097745 18368 raft_consensus.cc:2468] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 5.
I20260709 12:22:11.103291 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:11.041625 17680 consensus_queue.cc:260] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:11.127612 17832 raft_consensus.cc:3060] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:11.126677 17583 catalog_manager.cc:5697] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 reported cstate change: term changed from 3 to 8. New cstate: current_term: 8 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } }
W20260709 12:22:11.129760 17728 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.564s	user 0.003s	sys 0.016s
W20260709 12:22:11.133517 17728 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.568s	user 0.003s	sys 0.016s
W20260709 12:22:11.135113 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 569248us
I20260709 12:22:10.917783 21369 raft_consensus.cc:493] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:11.142102 17654 leader_election.cc:304] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:22:11.048699 21322 raft_consensus.cc:2804] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260709 12:22:11.139070 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 100434us
I20260709 12:22:10.996552 21311 consensus_queue.cc:1059] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 6.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.134s
I20260709 12:22:11.097976 21327 raft_consensus.cc:2804] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:22:11.139113 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:11.140129 17728 leader_election.cc:304] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:11.145515 17583 catalog_manager.cc:5697] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 reported cstate change: term changed from 1 to 4. New cstate: current_term: 4 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:22:11.154876 17831 raft_consensus.cc:2393] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 7: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:22:11.144140 18334 leader_election.cc:304] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:22:11.032629 21309 consensus_queue.cc:1048] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.094s
I20260709 12:22:11.155263 17583 catalog_manager.cc:5697] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 reported cstate change: term changed from 4 to 5, leader changed from 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) to 334a4e1f59944544b3b3ff36616b57da (127.16.29.68). New cstate: current_term: 5 leader_uuid: "334a4e1f59944544b3b3ff36616b57da" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:22:11.157665 17583 catalog_manager.cc:5697] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } }
I20260709 12:22:11.125710 21230 raft_consensus.cc:697] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 9 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:22:11.161898 17583 catalog_manager.cc:5697] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:22:11.167906 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
W20260709 12:22:11.142608 21332 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.527s	user 0.001s	sys 0.005s
I20260709 12:22:11.099543 21177 raft_consensus.cc:515] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:11.148422 21228 raft_consensus.cc:2804] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:22:11.165326 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:11.173355 17583 catalog_manager.cc:5697] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 reported cstate change: term changed from 4 to 10. New cstate: current_term: 10 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:22:11.147147 21322 raft_consensus.cc:493] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:11.170591 17837 raft_consensus.cc:2468] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 10.
I20260709 12:22:11.196309 18363 raft_consensus.cc:1275] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260709 12:22:11.145238 21369 raft_consensus.cc:3060] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:11.148267 21156 raft_consensus.cc:2804] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:22:11.175699 17831 raft_consensus.cc:1275] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 7 index: 9. (index mismatch)
I20260709 12:22:11.175848 17728 leader_election.cc:304] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
W20260709 12:22:11.176723 21363 log.cc:927] Time spent T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.053s	user 0.000s	sys 0.000s
W20260709 12:22:11.179440 21365 log.cc:927] Time spent T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.082s	user 0.000s	sys 0.000s
I20260709 12:22:11.181316 21357 leader_election.cc:290] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:11.148234 21240 consensus_queue.cc:1048] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.033s
W20260709 12:22:11.182876 21368 log.cc:927] Time spent T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.261s	user 0.000s	sys 0.001s
W20260709 12:22:11.184973 21364 log.cc:927] Time spent T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.266s	user 0.000s	sys 0.001s
I20260709 12:22:11.186168 21347 leader_election.cc:290] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:11.187583 21324 leader_election.cc:290] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:11.188651 21318 leader_election.cc:290] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:11.192708 21155 leader_election.cc:290] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 7 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:11.192874 21361 raft_consensus.cc:3060] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:11.194578 17680 raft_consensus.cc:3060] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:11.157555 21327 raft_consensus.cc:493] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:22:11.173957 18334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:11.206251 17729 leader_election.cc:304] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:11.206688 17832 raft_consensus.cc:2468] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 9.
W20260709 12:22:11.216569 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 234252us
I20260709 12:22:11.221706 21053 consensus_queue.cc:1048] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.013s
I20260709 12:22:11.160188 21317 leader_election.cc:290] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:22:11.416749 17727 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:11.420444 17727 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:11.385215 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
W20260709 12:22:11.420493 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 214281us
I20260709 12:22:11.334986 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:11.423189 18362 raft_consensus.cc:2468] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 7.
W20260709 12:22:11.355559 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 188079us
W20260709 12:22:11.289039 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 66796.9us
I20260709 12:22:11.286723 18363 raft_consensus.cc:1240] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 6. Current term is 7. Ops: []
I20260709 12:22:11.172399 21331 raft_consensus.cc:2749] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260709 12:22:11.214327 17764 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.883s	user 0.004s	sys 0.010s
W20260709 12:22:11.430465 17764 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.099s	user 0.010s	sys 0.011s
I20260709 12:22:11.235762 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:22:11.431222 17755 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.815s	user 0.003s	sys 0.015s
W20260709 12:22:11.431578 17755 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.822s	user 0.003s	sys 0.016s
W20260709 12:22:11.425880 17757 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.799s	user 0.007s	sys 0.005s
W20260709 12:22:11.173940 21332 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.559s	user 0.001s	sys 0.005s
I20260709 12:22:11.165993 21085 raft_consensus.cc:2804] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Leader pre-election won for term 13
W20260709 12:22:11.432124 17757 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.805s	user 0.007s	sys 0.005s
W20260709 12:22:11.436026 17684 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.612s	user 0.000s	sys 0.014s
W20260709 12:22:11.436408 17684 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.613s	user 0.000s	sys 0.014s
W20260709 12:22:11.414651 21343 consensus_meta.cc:287] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.626s	user 0.007s	sys 0.002s
W20260709 12:22:11.435341 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 259591us
I20260709 12:22:11.442871 17728 leader_election.cc:304] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
W20260709 12:22:11.439366 17652 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.617s	user 0.004s	sys 0.031s
W20260709 12:22:11.425684 21349 consensus_meta.cc:287] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.591s	user 0.009s	sys 0.001s
I20260709 12:22:11.437861 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
W20260709 12:22:11.443838 17652 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.621s	user 0.004s	sys 0.031s
I20260709 12:22:11.195626 21228 raft_consensus.cc:493] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:11.425503 21327 raft_consensus.cc:3060] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:11.429664 21311 raft_consensus.cc:493] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:11.444037 17758 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.564s	user 0.010s	sys 0.005s
I20260709 12:22:11.225294 21322 raft_consensus.cc:3060] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:11.447608 18374 raft_consensus.cc:2393] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 9: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:22:11.230911 17583 catalog_manager.cc:5697] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 reported cstate change: config changed from index 5 to 6, term changed from 3 to 5, leader changed from e063541542a94969898868bf6eb5ce1b (127.16.29.67) to 0da169ff82674dc1947055ce43de68a3 (127.16.29.65), VOTER e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66) evicted. New cstate: current_term: 5 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:22:11.390666 21354 raft_consensus.cc:3055] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:22:11.445859 17681 raft_consensus.cc:3060] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:11.191674 21304 raft_consensus.cc:515] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:11.416599 21177 leader_election.cc:290] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:11.449352 17802 leader_election.cc:304] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:22:11.429651 21352 raft_consensus.cc:2804] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260709 12:22:11.446306 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 631561us
I20260709 12:22:11.441643 21357 raft_consensus.cc:2749] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
W20260709 12:22:11.495527 17653 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.576s	user 0.010s	sys 0.010s
W20260709 12:22:11.495608 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 43675.5us
W20260709 12:22:11.496220 17653 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.577s	user 0.011s	sys 0.010s
I20260709 12:22:11.194789 21329 raft_consensus.cc:515] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:22:11.498948 17758 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.619s	user 0.010s	sys 0.011s
W20260709 12:22:11.505156 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 62389us
W20260709 12:22:11.506546 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 44234.6us
I20260709 12:22:11.440869 21085 raft_consensus.cc:493] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
W20260709 12:22:11.498312 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 49149.2us
I20260709 12:22:11.507577 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:22:11.472956 21305 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.555s	user 0.008s	sys 0.008s
I20260709 12:22:11.453006 21228 raft_consensus.cc:3060] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:11.508010 17802 leader_election.cc:304] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:22:11.512619 17801 leader_election.cc:304] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:11.439673 21156 raft_consensus.cc:697] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 12 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:22:11.435714 21060 raft_consensus.cc:515] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:11.516871 17652 leader_election.cc:304] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:11.557555 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 9 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:11.495180 21352 raft_consensus.cc:493] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:11.563386 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:22:11.563601 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 46829.5us
I20260709 12:22:11.495293 21317 raft_consensus.cc:2804] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260709 12:22:11.563449 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 55555.4us
W20260709 12:22:11.563625 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 51122.9us
I20260709 12:22:11.568080 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:11.568914 17688 raft_consensus.cc:2468] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 7.
W20260709 12:22:11.572526 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:11.385589 21230 consensus_queue.cc:237] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:11.451166 21312 consensus_queue.cc:1048] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.066s
I20260709 12:22:11.446373 21343 raft_consensus.cc:515] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:11.504153 21354 raft_consensus.cc:740] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:22:11.576617 17652 leader_election.cc:304] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:22:11.461273 21308 consensus_queue.cc:1048] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.069s
I20260709 12:22:11.511502 21334 raft_consensus.cc:2804] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:22:11.648998 17680 raft_consensus.cc:2468] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 7.
I20260709 12:22:11.578373 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:11.513621 21085 raft_consensus.cc:3060] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:11.598294 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:11.514815 21306 raft_consensus.cc:493] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:11.592549 18371 raft_consensus.cc:2468] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 11.
I20260709 12:22:11.495728 21311 raft_consensus.cc:515] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:22:11.521901 21305 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.597s	user 0.008s	sys 0.012s
I20260709 12:22:11.516230 17583 catalog_manager.cc:5697] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 reported cstate change: term changed from 3 to 4, leader changed from e063541542a94969898868bf6eb5ce1b (127.16.29.67) to 0da169ff82674dc1947055ce43de68a3 (127.16.29.65). New cstate: current_term: 4 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: HEALTHY } } }
I20260709 12:22:11.632869 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
W20260709 12:22:11.634087 17686 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.662s	user 0.016s	sys 0.033s
W20260709 12:22:11.630829 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 209145us
W20260709 12:22:11.673872 17686 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.702s	user 0.016s	sys 0.037s
I20260709 12:22:11.633708 17801 leader_election.cc:400] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 8 election: Vote denied by peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651) with higher term. Message: Invalid argument: T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 8. Current term is 9.
I20260709 12:22:11.674537 17801 leader_election.cc:403] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 8 election: Cancelling election due to peer responding with higher term
I20260709 12:22:11.632014 21307 consensus_queue.cc:1048] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.045s
W20260709 12:22:11.680913 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 115721us
I20260709 12:22:11.682081 17652 leader_election.cc:304] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:11.451018 21349 raft_consensus.cc:515] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:11.656652 17683 raft_consensus.cc:2468] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 6.
I20260709 12:22:11.671959 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:22:11.684480 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 115530us
W20260709 12:22:11.688267 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 121583us
I20260709 12:22:11.655738 17688 raft_consensus.cc:2468] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 6.
I20260709 12:22:11.591140 20921 raft_consensus.cc:2804] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:22:11.506399 21319 consensus_queue.cc:1048] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.096s
I20260709 12:22:11.678632 17583 catalog_manager.cc:5697] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 reported cstate change: term changed from 6 to 8, leader changed from 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) to e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66). New cstate: current_term: 8 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:22:11.598452 21342 raft_consensus.cc:2749] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:22:11.658524 21334 raft_consensus.cc:493] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:11.523837 21177 raft_consensus.cc:493] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:22:11.659812 21053 raft_consensus.cc:493] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:11.713973 17759 raft_consensus.cc:1240] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 9. Current term is 10. Ops: [9.8-9.8]
W20260709 12:22:11.710219 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 157381us
I20260709 12:22:11.708854 18374 raft_consensus.cc:2468] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 7.
I20260709 12:22:11.577313 21317 raft_consensus.cc:493] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:11.711728 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:11.703293 17583 catalog_manager.cc:5697] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 reported cstate change: term changed from 4 to 5, leader changed from 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) to 0da169ff82674dc1947055ce43de68a3 (127.16.29.65). New cstate: current_term: 5 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:22:11.712692 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:11.714408 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:11.716017 17680 raft_consensus.cc:2468] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 5.
I20260709 12:22:11.577636 21352 raft_consensus.cc:3060] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:11.720029 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:11.693874 20394 raft_consensus.cc:2804] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:22:11.720652 17683 raft_consensus.cc:2468] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 7.
W20260709 12:22:11.733609 17803 connection.cc:256] RPC call timeout handler was delayed by -1.8855732709998847s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2913s for a response
W20260709 12:22:11.737182 17803 connection.cc:256] RPC call timeout handler was delayed by -1.744273271000111s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.307s for a response
I20260709 12:22:11.726531 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:11.687155 21379 raft_consensus.cc:2749] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:22:11.711149 21241 raft_consensus.cc:2804] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Leader pre-election won for term 10
W20260709 12:22:11.720973 21244 consensus_meta.cc:287] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.781s	user 0.010s	sys 0.000s
I20260709 12:22:11.711854 21334 raft_consensus.cc:3060] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:11.745059 17688 raft_consensus.cc:2468] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 8.
I20260709 12:22:11.679437 21306 raft_consensus.cc:515] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:22:11.754248 17803 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:11.764943 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:11.767256 17689 raft_consensus.cc:2393] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 8: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:22:11.721896 21317 raft_consensus.cc:3060] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:11.747489 21379 consensus_queue.cc:1048] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.025s
W20260709 12:22:11.772948 21396 log.cc:927] Time spent T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.050s	user 0.000s	sys 0.000s
I20260709 12:22:11.773824 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:11.774430 17683 raft_consensus.cc:2468] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 10.
I20260709 12:22:11.774567 21343 leader_election.cc:290] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 11 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:11.747718 21342 raft_consensus.cc:3060] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:11.776219 18334 leader_election.cc:304] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
W20260709 12:22:11.777138 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 87395.5us
I20260709 12:22:11.779517 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:11.745950 21241 raft_consensus.cc:493] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:22:11.816452 17688 raft_consensus.cc:2393] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 9: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:22:11.816586 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:11.819856 17689 raft_consensus.cc:2468] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 10.
I20260709 12:22:11.724738 21053 raft_consensus.cc:515] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:22:11.827601 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 139519us
I20260709 12:22:11.829819 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:11.830461 18371 raft_consensus.cc:3060] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:11.733610 21177 raft_consensus.cc:515] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:11.874001 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:11.874938 17680 raft_consensus.cc:2376] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 7. Re-sending same reply.
I20260709 12:22:11.884305 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:11.753331 21244 raft_consensus.cc:515] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:11.885882 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:11.790354 20921 raft_consensus.cc:697] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 7 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:22:11.891657 17688 raft_consensus.cc:2468] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 5.
W20260709 12:22:11.892984 17802 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:11.718755 21156 consensus_queue.cc:237] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 10.9, Last appended by leader: 9, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:11.900753 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:11.828380 21241 raft_consensus.cc:3060] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:11.745147 21354 consensus_queue.cc:260] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:11.909233 17763 raft_consensus.cc:1240] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 6 LEADER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 5. Current term is 6. Ops: [5.6-5.6]
I20260709 12:22:11.910495 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:11.914502 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:11.838382 21404 raft_consensus.cc:2804] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:22:11.920321 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:11.924401 17756 raft_consensus.cc:1240] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 6. Current term is 7. Ops: []
I20260709 12:22:11.920979 17831 raft_consensus.cc:2468] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 10.
W20260709 12:22:11.925344 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 159157us
I20260709 12:22:11.930338 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:11.945073 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:11.951982 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 9 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:11.955605 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:22:11.956663 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 126499us
I20260709 12:22:11.946054 17830 raft_consensus.cc:2468] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 9.
I20260709 12:22:11.959221 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:11.952685 17836 raft_consensus.cc:2468] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 8.
I20260709 12:22:11.961174 17680 raft_consensus.cc:2468] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 8.
I20260709 12:22:11.920171 21329 leader_election.cc:290] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 10 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:11.973955 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:11.975332 16501 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260709 12:22:11.946616 21404 raft_consensus.cc:697] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 7 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:22:12.004390 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:22:11.968613 21394 log.cc:927] Time spent T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.255s	user 0.001s	sys 0.000s
I20260709 12:22:11.913590 21401 consensus_queue.cc:1048] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.066s
I20260709 12:22:12.015383 17689 raft_consensus.cc:1240] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 7. Current term is 8. Ops: [7.4-7.4]
W20260709 12:22:12.364146 18364 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.802s	user 0.000s	sys 0.019s
I20260709 12:22:12.043852 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:22:12.021340 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:12.013752 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:22:12.381116 18364 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.819s	user 0.000s	sys 0.020s
I20260709 12:22:12.012657 17801 leader_election.cc:304] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:22:12.049007 17681 consensus_meta.cc:287] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.532s	user 0.007s	sys 0.003s
I20260709 12:22:12.398618 17681 raft_consensus.cc:2468] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 7.
I20260709 12:22:11.968041 21311 leader_election.cc:290] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:12.311394 16501 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
W20260709 12:22:12.377643 21395 log.cc:927] Time spent T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.426s	user 0.000s	sys 0.000s
I20260709 12:22:12.226101 21415 consensus_queue.cc:1059] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 9.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.274s
I20260709 12:22:12.074393 20921 consensus_queue.cc:237] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:22:12.225644 21399 log.cc:927] Time spent T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.453s	user 0.002s	sys 0.000s
W20260709 12:22:12.387897 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 375436us
W20260709 12:22:12.527658 17652 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.826s	user 0.005s	sys 0.009s
W20260709 12:22:12.582257 17652 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.881s	user 0.005s	sys 0.009s
W20260709 12:22:12.582777 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 881504us
I20260709 12:22:12.493098 17681 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44898 (request call id 775) took 1071 ms. Trace:
W20260709 12:22:12.555548 21402 log.cc:927] Time spent T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.399s	user 0.001s	sys 0.001s
W20260709 12:22:12.223835 21400 log.cc:927] Time spent T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.264s	user 0.001s	sys 0.000s
W20260709 12:22:12.561398 18367 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.646s	user 0.005s	sys 0.015s
I20260709 12:22:12.010965 21060 leader_election.cc:290] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 9 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:22:12.610297 18367 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.694s	user 0.005s	sys 0.015s
I20260709 12:22:12.583974 17681 rpcz_store.cc:276] 0709 12:22:11.421981 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:11.422112 (+   131us) service_pool.cc:224] Handling call
0709 12:22:12.493023 (+1070911us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:12.485070 21053 leader_election.cc:290] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:22:12.616645 21385 log.cc:927] Time spent T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.730s	user 0.001s	sys 0.001s
W20260709 12:22:12.568396 21403 log.cc:927] Time spent T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.553s	user 0.000s	sys 0.001s
W20260709 12:22:12.647352 21414 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.584s	user 0.005s	sys 0.007s
W20260709 12:22:12.647704 21397 log.cc:927] Time spent T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.090s	user 0.000s	sys 0.000s
W20260709 12:22:12.648504 21389 log.cc:927] Time spent T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.609s	user 0.001s	sys 0.000s
I20260709 12:22:12.535007 21177 leader_election.cc:290] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:12.820909 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:12.708902 21401 raft_consensus.cc:2804] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:22:12.719798 18366 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 1185) took 1504 ms. Trace:
I20260709 12:22:12.825973 18366 rpcz_store.cc:276] 0709 12:22:11.215330 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:11.235240 (+ 19910us) service_pool.cc:224] Handling call
0709 12:22:11.272871 (+ 37631us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:22:11.273395 (+   524us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:11.273440 (+    45us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:22:11.312859 (+ 39419us) log.cc:844] Serialized 52 byte log entry
0709 12:22:11.499380 (+186521us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:22:11.499424 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:11.499482 (+    58us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:12.719427 (+1219945us) raft_consensus.cc:1685] finished
0709 12:22:12.719465 (+    38us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:12.719695 (+   230us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":108,"raft.run_cpu_time_us":250,"raft.run_wall_time_us":80224,"thread_start_us":134147,"threads_started":1,"wal-append.queue_time_us":1354251}
I20260709 12:22:12.724572 18361 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 1184) took 1526 ms. Trace:
W20260709 12:22:12.803249 18371 consensus_meta.cc:287] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.955s	user 0.010s	sys 0.001s
I20260709 12:22:12.820098 17837 raft_consensus.cc:1275] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20260709 12:22:12.820377 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:12.831748 17831 raft_consensus.cc:2468] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 7.
I20260709 12:22:12.727063 21406 consensus_queue.cc:1059] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.020s
I20260709 12:22:12.822216 17763 raft_consensus.cc:2410] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for term 6 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
W20260709 12:22:12.860845 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:12.883930 17686 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44898 (request call id 761) took 2081 ms. Trace:
I20260709 12:22:12.884510 17686 rpcz_store.cc:276] 0709 12:22:10.802844 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:10.802975 (+   131us) service_pool.cc:224] Handling call
0709 12:22:10.819798 (+ 16823us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:10.820009 (+   211us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:10.820035 (+    26us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:10.820394 (+   359us) log.cc:844] Serialized 26 byte log entry
0709 12:22:11.674311 (+853917us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:22:11.674354 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:11.674424 (+    70us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:12.883539 (+1209115us) raft_consensus.cc:1685] finished
0709 12:22:12.883569 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:12.883820 (+   251us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":38016,"thread_start_us":702581,"threads_started":1,"wal-append.queue_time_us":904462}
I20260709 12:22:12.842976 21401 raft_consensus.cc:493] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:12.904384 21401 raft_consensus.cc:3060] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:12.908829 16501 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:22:12.908911 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 8 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:12.911553 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:12.911796 18368 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 1186) took 1696 ms. Trace:
I20260709 12:22:12.912127 17833 raft_consensus.cc:2410] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for term 8 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 12:22:12.832878 18371 raft_consensus.cc:2468] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 11.
I20260709 12:22:12.828450 18361 rpcz_store.cc:276] 0709 12:22:11.197839 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:11.197965 (+   126us) service_pool.cc:224] Handling call
0709 12:22:11.443007 (+245042us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:11.455081 (+ 12074us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:11.455111 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:11.497813 (+ 42702us) log.cc:844] Serialized 26 byte log entry
0709 12:22:11.704163 (+206350us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:22:11.704222 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:11.704312 (+    90us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:12.724248 (+1019936us) raft_consensus.cc:1685] finished
0709 12:22:12.724283 (+    35us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:12.724490 (+   207us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":151684,"raft.run_cpu_time_us":131,"raft.run_wall_time_us":25612,"thread_start_us":112551,"threads_started":1,"wal-append.queue_time_us":248362}
I20260709 12:22:12.823911 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:12.379384 21404 consensus_queue.cc:237] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:22:13.326889 18334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:13.078039 21327 consensus_meta.cc:287] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 1.563s	user 0.011s	sys 0.002s
I20260709 12:22:12.823192 21053 raft_consensus.cc:493] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:13.225049 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:22:13.080470 21322 consensus_meta.cc:287] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 1.574s	user 0.008s	sys 0.005s
W20260709 12:22:13.351832 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:12.946787 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 323320us
W20260709 12:22:13.072471 21361 consensus_meta.cc:287] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 1.642s	user 0.008s	sys 0.003s
W20260709 12:22:13.326037 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 1.37089e+06us
I20260709 12:22:12.912793 17758 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38786 (request call id 2661) took 2183 ms. Trace:
I20260709 12:22:12.961707 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:13.331665 17687 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44898 (request call id 776) took 1908 ms. Trace:
I20260709 12:22:13.389114 17835 raft_consensus.cc:1275] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
W20260709 12:22:13.221104 21085 consensus_meta.cc:287] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 1.541s	user 0.013s	sys 0.000s
W20260709 12:22:13.075057 21228 consensus_meta.cc:287] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 1.497s	user 0.007s	sys 0.005s
W20260709 12:22:13.071250 21369 consensus_meta.cc:287] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 1.715s	user 0.011s	sys 0.002s
I20260709 12:22:12.910398 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:12.912261 18368 rpcz_store.cc:276] 0709 12:22:11.215634 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:11.217846 (+  2212us) service_pool.cc:224] Handling call
0709 12:22:11.281812 (+ 63966us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:11.577751 (+295939us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:11.577795 (+    44us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:11.578453 (+   658us) log.cc:844] Serialized 26 byte log entry
0709 12:22:11.816872 (+238419us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:22:11.816908 (+    36us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:11.816976 (+    68us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:12.722144 (+905168us) raft_consensus.cc:1685] finished
0709 12:22:12.722173 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:12.911738 (+189565us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":79564,"raft.run_cpu_time_us":88,"raft.run_wall_time_us":89,"thread_start_us":461453,"threads_started":2,"wal-append.queue_time_us":721540}
I20260709 12:22:13.333731 18364 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 1189) took 1881 ms. Trace:
I20260709 12:22:13.341105 18371 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59886 (request call id 1058) took 1554 ms. Trace:
W20260709 12:22:13.182698 21230 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.649s	user 0.020s	sys 0.011s
I20260709 12:22:13.337392 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:22:12.878082 21399 log.cc:927] Time spent T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.077s	user 0.000s	sys 0.000s
W20260709 12:22:13.354133 17830 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.548s	user 0.005s	sys 0.005s
I20260709 12:22:13.384877 18334 leader_election.cc:304] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
W20260709 12:22:13.426558 17830 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.621s	user 0.005s	sys 0.008s
I20260709 12:22:13.404841 18364 rpcz_store.cc:276] 0709 12:22:11.451840 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:11.452012 (+   172us) service_pool.cc:224] Handling call
0709 12:22:11.496464 (+ 44452us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:11.496671 (+   207us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 12:22:12.384044 (+887373us) log.cc:844] Serialized 14 byte log entry
0709 12:22:13.333305 (+949261us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:13.333374 (+    69us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:22:13.333435 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:13.333521 (+    86us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:13.333654 (+   133us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":517499,"prepare.run_cpu_time_us":57,"prepare.run_wall_time_us":5026,"raft.queue_time_us":752190,"raft.run_cpu_time_us":123,"raft.run_wall_time_us":165,"thread_start_us":1022702,"threads_started":2,"wal-append.queue_time_us":1014624}
W20260709 12:22:13.325222 21317 consensus_meta.cc:287] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 1.508s	user 0.012s	sys 0.000s
W20260709 12:22:13.020831 21390 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.713s	user 0.004s	sys 0.015s
I20260709 12:22:13.392725 17834 raft_consensus.cc:3060] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:13.389730 17687 rpcz_store.cc:276] 0709 12:22:11.423685 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:11.423800 (+   115us) service_pool.cc:224] Handling call
0709 12:22:11.438687 (+ 14887us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:11.440074 (+  1387us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:11.440115 (+    41us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:11.461163 (+ 21048us) log.cc:844] Serialized 26 byte log entry
0709 12:22:11.817702 (+356539us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:22:11.817745 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:11.817806 (+    61us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:13.331360 (+1513554us) raft_consensus.cc:1685] finished
0709 12:22:13.331401 (+    41us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:13.331584 (+   183us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":967,"raft.queue_time_us":1317,"spinlock_wait_cycles":7143680,"thread_start_us":302700,"threads_started":1,"wal-append.queue_time_us":893257}
I20260709 12:22:13.056011 21060 raft_consensus.cc:493] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
W20260709 12:22:13.052630 21241 consensus_meta.cc:287] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 1.128s	user 0.008s	sys 0.003s
I20260709 12:22:13.450953 17833 raft_consensus.cc:1240] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 7. Current term is 9. Ops: []
I20260709 12:22:13.388581 17758 rpcz_store.cc:276] 0709 12:22:10.729943 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:10.784175 (+ 54232us) service_pool.cc:224] Handling call
0709 12:22:10.784505 (+   330us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:10.787092 (+  2587us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:10.787124 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:10.792514 (+  5390us) log.cc:844] Serialized 26 byte log entry
0709 12:22:11.499718 (+707204us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:22:11.499753 (+    35us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:11.499825 (+    72us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:12.912586 (+1412761us) raft_consensus.cc:1685] finished
0709 12:22:12.912606 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:12.912728 (+   122us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":12682,"raft.queue_time_us":6300,"raft.run_cpu_time_us":121,"raft.run_wall_time_us":8184,"thread_start_us":620035,"threads_started":1,"wal-append.queue_time_us":1546243}
W20260709 12:22:12.941694 21156 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.964s	user 0.008s	sys 0.035s
W20260709 12:22:13.085572 21334 consensus_meta.cc:287] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 1.319s	user 0.003s	sys 0.006s
W20260709 12:22:13.055317 21342 consensus_meta.cc:287] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 1.240s	user 0.008s	sys 0.000s
W20260709 12:22:13.323810 21352 consensus_meta.cc:287] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 1.602s	user 0.011s	sys 0.002s
I20260709 12:22:13.396318 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:13.397226 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:13.398223 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:13.401782 21405 raft_consensus.cc:493] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:13.402354 21435 raft_consensus.cc:3055] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:22:13.405661 17837 raft_consensus.cc:2468] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 6.
I20260709 12:22:13.404336 17680 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37402 (request call id 2447) took 1528 ms. Trace:
I20260709 12:22:13.397221 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:13.470958 18365 raft_consensus.cc:3060] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:13.471449 17830 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35042 (request call id 1925) took 1502 ms. Trace:
I20260709 12:22:13.338599 21244 leader_election.cc:290] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 7 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:22:13.427183 17803 leader_election.cc:336] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Timed out: RequestConsensusVote RPC to 127.16.29.68:42577 timed out after 1.748s (SENT)
I20260709 12:22:13.427320 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:13.404822 18371 rpcz_store.cc:276] 0709 12:22:11.786549 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:11.786711 (+   162us) service_pool.cc:224] Handling call
0709 12:22:13.341039 (+1554328us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:13.476120 17803 leader_election.cc:304] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
W20260709 12:22:13.455737 18367 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.513s	user 0.006s	sys 0.011s
I20260709 12:22:13.453375 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
W20260709 12:22:13.407142 21230 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.905s	user 0.022s	sys 0.017s
I20260709 12:22:13.058470 21401 raft_consensus.cc:515] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:13.470772 17680 rpcz_store.cc:276] 0709 12:22:11.875646 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:11.961894 (+ 86248us) service_pool.cc:224] Handling call
0709 12:22:11.984967 (+ 23073us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:12.008388 (+ 23421us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:12.008436 (+    48us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:12.018761 (+ 10325us) log.cc:844] Serialized 26 byte log entry
0709 12:22:12.405754 (+386993us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:22:12.557246 (+151492us) log.cc:844] Serialized 14 byte log entry
0709 12:22:12.557482 (+   236us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:12.557554 (+    72us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:13.404129 (+846575us) raft_consensus.cc:1685] finished
0709 12:22:13.404149 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:13.404279 (+   130us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":109198,"prepare.queue_time_us":163599,"prepare.run_wall_time_us":7100,"raft.queue_time_us":14031,"raft.run_cpu_time_us":402,"raft.run_wall_time_us":20897,"spinlock_wait_cycles":190349056,"thread_start_us":307818,"threads_started":1,"wal-append.queue_time_us":1353725}
I20260709 12:22:13.472918 18361 raft_consensus.cc:3060] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Advancing to term 10
W20260709 12:22:13.458642 21156 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.481s	user 0.016s	sys 0.044s
I20260709 12:22:13.405175 21085 raft_consensus.cc:515] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:22:13.454607 21390 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.154s	user 0.015s	sys 0.035s
I20260709 12:22:13.479146 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:13.521317 18364 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 8.
I20260709 12:22:13.521376 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:13.394182 21053 raft_consensus.cc:515] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:13.389402 21327 raft_consensus.cc:515] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:13.490842 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:13.404316 21228 raft_consensus.cc:515] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:13.532408 21228 leader_election.cc:290] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 7 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:22:13.500026 21432 log.cc:927] Time spent T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.096s	user 0.001s	sys 0.000s
I20260709 12:22:13.534387 17760 raft_consensus.cc:2368] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 6. Current term is 7.
I20260709 12:22:13.481040 21405 raft_consensus.cc:515] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:13.475574 21435 raft_consensus.cc:740] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 7 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:22:13.502969 17686 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37402 (request call id 2454) took 1573 ms. Trace:
I20260709 12:22:13.504002 17685 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44898 (request call id 773) took 2222 ms. Trace:
I20260709 12:22:13.458995 21317 raft_consensus.cc:515] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:13.452427 21306 leader_election.cc:290] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:13.393898 21322 raft_consensus.cc:515] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:13.472077 17830 rpcz_store.cc:276] 0709 12:22:11.968597 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:11.968747 (+   150us) service_pool.cc:224] Handling call
0709 12:22:12.005658 (+ 36911us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:12.034819 (+ 29161us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 12:22:12.488645 (+453826us) log.cc:844] Serialized 14 byte log entry
0709 12:22:13.471178 (+982533us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:13.471230 (+    52us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:22:13.471276 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:13.471323 (+    47us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:13.471406 (+    83us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":44125,"prepare.queue_time_us":681726,"prepare.run_cpu_time_us":31,"prepare.run_wall_time_us":32,"raft.queue_time_us":608691,"raft.run_cpu_time_us":105,"raft.run_wall_time_us":106,"thread_start_us":943316,"threads_started":2,"wal-append.queue_time_us":841266}
I20260709 12:22:13.452731 21425 raft_consensus.cc:2804] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Leader election won for term 7
W20260709 12:22:13.509752 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 82602.4us
I20260709 12:22:13.510562 17834 raft_consensus.cc:2468] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 12.
I20260709 12:22:13.550515 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:13.551313 17838 raft_consensus.cc:2468] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 8.
W20260709 12:22:13.477869 18367 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.536s	user 0.006s	sys 0.011s
I20260709 12:22:13.425904 21369 raft_consensus.cc:515] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:13.520195 18365 raft_consensus.cc:2468] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 6.
I20260709 12:22:13.522414 17833 raft_consensus.cc:2393] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 6: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:22:13.456571 21241 raft_consensus.cc:515] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:13.455478 21060 raft_consensus.cc:515] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:22:13.537371 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 192042us
I20260709 12:22:13.540823 17686 rpcz_store.cc:276] 0709 12:22:11.929735 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:12.886434 (+956699us) service_pool.cc:224] Handling call
0709 12:22:12.886918 (+   484us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:12.893467 (+  6549us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:12.893513 (+    46us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:12.912478 (+ 18965us) log.cc:844] Serialized 26 byte log entry
0709 12:22:13.428076 (+515598us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:22:13.428106 (+    30us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:13.428156 (+    50us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:13.502726 (+ 74570us) raft_consensus.cc:1685] finished
0709 12:22:13.502747 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:13.502908 (+   161us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":5904,"raft.run_cpu_time_us":101,"raft.run_wall_time_us":103,"thread_start_us":354308,"threads_started":1,"wal-append.queue_time_us":482058}
I20260709 12:22:13.462906 21334 raft_consensus.cc:515] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:13.475389 21352 raft_consensus.cc:515] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:13.517043 21426 raft_consensus.cc:493] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:13.523756 21329 raft_consensus.cc:493] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:13.526171 20921 raft_consensus.cc:2749] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:22:13.473304 21342 raft_consensus.cc:2749] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Leader election lost for term 8. Reason: Vote denied by peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651) with higher term. Message: Invalid argument: T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 8. Current term is 9.
I20260709 12:22:13.563128 18368 raft_consensus.cc:3060] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Advancing to term 9
I20260709 12:22:13.563024 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:13.563858 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:22:13.564747 17652 connection.cc:256] RPC call timeout handler was delayed by -1.5605123720001757s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1527s for a response
I20260709 12:22:13.540663 17685 rpcz_store.cc:276] 0709 12:22:11.281787 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:11.281934 (+   147us) service_pool.cc:224] Handling call
0709 12:22:11.361863 (+ 79929us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:11.364231 (+  2368us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 12:22:11.520327 (+156096us) log.cc:844] Serialized 14 byte log entry
0709 12:22:11.700618 (+180291us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:11.722649 (+ 22031us) log.cc:844] Serialized 26 byte log entry
0709 12:22:11.722810 (+   161us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:22:11.722858 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:11.722929 (+    71us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:12.224889 (+501960us) spinlock_profiling.cc:243] Waited 3.9 ms on lock 0x7b4400022b4c. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b47459c85 00007f7b4745a6a2 00007f7b4a1cc1db 00007f7b4a1c7aaf 00007f7b4a1bb86e 00007f7b4a1babb6 00007f7b4a1ba7d8 00007f7b4a1c4221 00007f7b4a1c41d9 00007f7b4a1c4169
0709 12:22:13.503585 (+1278696us) raft_consensus.cc:1685] finished
0709 12:22:13.503719 (+   134us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:13.503914 (+   195us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":15891,"prepare.queue_time_us":533454,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":40,"raft.queue_time_us":35399,"raft.run_cpu_time_us":222,"raft.run_wall_time_us":5944,"spinlock_wait_cycles":8959360,"thread_start_us":335299,"threads_started":2,"wal-append.queue_time_us":251925}
I20260709 12:22:13.574234 17682 raft_consensus.cc:2368] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 6. Current term is 7.
I20260709 12:22:13.575417 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:13.557732 21401 leader_election.cc:290] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 9 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:13.644352 17837 raft_consensus.cc:3060] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:13.616283 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:22:13.645859 17652 connection.cc:256] RPC call timeout handler was delayed by -1.412912371999937s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16910000000000003s for a response
I20260709 12:22:13.619040 17686 raft_consensus.cc:1240] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 9. Current term is 10. Ops: []
W20260709 12:22:13.646678 17652 connection.cc:256] RPC call timeout handler was delayed by -1.3274123720002535s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1786s for a response
I20260709 12:22:13.647085 17834 raft_consensus.cc:2410] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for term 8 because replica has last-logged OpId of term: 7 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
W20260709 12:22:13.647320 17652 connection.cc:256] RPC call timeout handler was delayed by -1.1510123719999683s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19820000000000002s for a response
W20260709 12:22:13.649634 17652 connection.cc:256] RPC call timeout handler was delayed by -1.0664123720002863s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2076s for a response
I20260709 12:22:13.643029 18368 raft_consensus.cc:2468] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 9.
I20260709 12:22:13.612309 17831 raft_consensus.cc:2393] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 7: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:22:13.655527 17832 raft_consensus.cc:2468] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 9.
I20260709 12:22:13.566325 21306 raft_consensus.cc:493] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:13.657029 21306 raft_consensus.cc:515] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:13.647837 17830 raft_consensus.cc:2468] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 7.
I20260709 12:22:13.659665 17830 raft_consensus.cc:1275] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
W20260709 12:22:13.661782 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 111156us
I20260709 12:22:13.671429 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:13.672407 21306 leader_election.cc:290] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:22:13.673216 21439 log.cc:927] Time spent T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.105s	user 0.000s	sys 0.000s
I20260709 12:22:13.676239 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:22:13.676002 17803 connection.cc:256] RPC call timeout handler was delayed by -1.5849023259997921s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.24980000000000002s for a response
I20260709 12:22:13.672449 17685 raft_consensus.cc:3060] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:13.678031 17652 leader_election.cc:304] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:22:13.676628 21347 raft_consensus.cc:493] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:13.679535 17803 connection.cc:256] RPC call timeout handler was delayed by -1.3448960339997029s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3165s for a response
I20260709 12:22:13.683784 17652 leader_election.cc:304] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:13.684960 21342 raft_consensus.cc:493] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:13.685217 21347 raft_consensus.cc:515] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:13.685595 21342 raft_consensus.cc:515] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:13.687449 21347 leader_election.cc:290] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:13.601899 21426 raft_consensus.cc:515] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:13.688737 21444 raft_consensus.cc:2804] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:22:13.689205 21444 raft_consensus.cc:493] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:13.689584 21444 raft_consensus.cc:3060] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:13.690582 21426 leader_election.cc:290] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:13.692595 17680 raft_consensus.cc:1240] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 6. Current term is 7. Ops: []
I20260709 12:22:13.692523 17802 leader_election.cc:304] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:22:13.680904 17803 leader_election.cc:336] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Timed out: RequestConsensusVote RPC to 127.16.29.68:42577 timed out after 1.502s (SENT)
W20260709 12:22:13.694754 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 149107us
I20260709 12:22:13.643385 21085 leader_election.cc:290] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 13 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:13.675570 21060 leader_election.cc:290] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:13.701097 17652 leader_election.cc:304] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:22:13.692121 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:13.715641 18363 raft_consensus.cc:2468] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 8.
I20260709 12:22:13.705077 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:22:13.694481 17803 leader_election.cc:336] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Timed out: RequestConsensusVote RPC to 127.16.29.68:42577 timed out after 1.660s (SENT)
I20260709 12:22:13.716578 17652 leader_election.cc:304] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:22:13.717067 17803 leader_election.cc:336] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Timed out: RequestConsensusVote RPC to 127.16.29.68:42577 timed out after 1.807s (SENT)
I20260709 12:22:13.717433 17803 leader_election.cc:304] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:13.718108 17652 leader_election.cc:304] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:22:13.719115 17652 leader_election.cc:304] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:13.719590 17837 raft_consensus.cc:2468] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 11.
I20260709 12:22:13.716519 17761 raft_consensus.cc:2468] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 7.
I20260709 12:22:13.729087 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:13.729148 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:13.611128 21329 raft_consensus.cc:515] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:13.731632 17830 raft_consensus.cc:1275] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20260709 12:22:13.729840 17762 raft_consensus.cc:1275] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260709 12:22:13.691659 21349 leader_election.cc:290] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 10 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:13.697156 20921 raft_consensus.cc:493] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:13.743662 17761 raft_consensus.cc:2393] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 9: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
W20260709 12:22:13.744151 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 51692.1us
I20260709 12:22:13.744925 17802 leader_election.cc:304] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:13.707791 21426 raft_consensus.cc:2749] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:22:13.747498 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:22:13.748319 17803 leader_election.cc:336] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Timed out: RequestConsensusVote RPC to 127.16.29.68:42577 timed out after 1.906s (SENT)
I20260709 12:22:13.722685 21177 raft_consensus.cc:493] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:22:13.728168 21185 raft_consensus.cc:2804] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:22:13.754434 17802 leader_election.cc:304] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:13.752784 17803 leader_election.cc:304] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:13.729714 21361 raft_consensus.cc:493] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
W20260709 12:22:13.760219 18334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:13.732547 21445 raft_consensus.cc:3060] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:13.741102 21156 raft_consensus.cc:2804] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:22:13.697273 21306 consensus_queue.cc:1048] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.050s
W20260709 12:22:13.763733 18333 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:13.738478 21424 raft_consensus.cc:2804] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:22:13.782686 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:13.753310 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:22:13.767841 21438 log.cc:927] Time spent T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.134s	user 0.001s	sys 0.000s
I20260709 12:22:13.677311 21435 consensus_queue.cc:260] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:13.772715 21354 raft_consensus.cc:3060] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:22:13.776703 17760 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36572 (request call id 681) took 2630 ms (client timeout 1622 ms). Trace:
I20260709 12:22:13.777452 21329 leader_election.cc:290] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:13.751751 21085 raft_consensus.cc:493] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:13.777765 21053 leader_election.cc:290] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:13.764942 21332 raft_consensus.cc:493] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:13.778761 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:13.778836 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:13.779541 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:22:13.779759 21441 log.cc:927] Time spent T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.168s	user 0.001s	sys 0.000s
I20260709 12:22:13.762960 21185 raft_consensus.cc:493] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:22:13.765393 21334 leader_election.cc:290] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 7 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:13.784421 21156 raft_consensus.cc:493] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:22:13.796346 21426 raft_consensus.cc:2804] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:22:13.827502 17764 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38776 (request call id 1990) took 3901 ms. Trace:
I20260709 12:22:13.792059 17762 raft_consensus.cc:2410] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for term 6 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260709 12:22:13.794373 17728 leader_election.cc:304] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:22:13.792224 17760 rpcz_store.cc:269] 0709 12:22:11.145980 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:11.155672 (+  9692us) service_pool.cc:224] Handling call
0709 12:22:13.776469 (+2620797us) spinlock_profiling.cc:243] Waited 2.37 s on lock 0x7b54003e5dd4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:22:13.776603 (+   134us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5440538368}
I20260709 12:22:13.796255 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: true dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:13.829380 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:13.755869 21431 raft_consensus.cc:3060] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:13.753948 21342 leader_election.cc:290] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:13.753770 21231 raft_consensus.cc:493] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:22:13.801429 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:13.802521 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:13.802649 21322 leader_election.cc:290] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 7 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:13.800235 17838 raft_consensus.cc:2468] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 7.
I20260709 12:22:13.802904 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:13.812845 21185 raft_consensus.cc:3060] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:13.826151 21053 raft_consensus.cc:2804] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:22:13.803094 21435 raft_consensus.cc:3060] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:13.803614 21241 leader_election.cc:290] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 10 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:13.830178 17764 rpcz_store.cc:276] 0709 12:22:09.926475 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:09.926582 (+   107us) service_pool.cc:224] Handling call
0709 12:22:09.926885 (+   303us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:09.937955 (+ 11070us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 12:22:10.184485 (+246530us) log.cc:844] Serialized 14 byte log entry
0709 12:22:11.517446 (+1332961us) log.cc:844] Serialized 14 byte log entry
0709 12:22:13.644888 (+2127442us) spinlock_profiling.cc:243] Waited 13.2 ms on lock 0x7b18003d9f00. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b474adc0a 00007f7b50bed219 00007f7b47535a40 00007f7b475304e4 00007f7b47530455 00007f7b4a618060 00007f7b4a62a716 00007f7b4a63b287 00007f7b4a63b208
0709 12:22:13.724195 (+ 79307us) log.cc:844] Serialized 14 byte log entry
0709 12:22:13.813004 (+ 88809us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:13.813050 (+    46us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:22:13.813104 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:13.813843 (+   739us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:13.827414 (+ 13571us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2285238,"prepare.queue_time_us":967086,"prepare.run_cpu_time_us":112,"prepare.run_wall_time_us":2080,"raft.queue_time_us":2213249,"raft.run_cpu_time_us":441,"raft.run_wall_time_us":53771,"spinlock_wait_cycles":30329728,"thread_start_us":1471030,"threads_started":4,"wal-append.queue_time_us":934302,"wal-append.run_cpu_time_us":1963,"wal-append.run_wall_time_us":1358615}
I20260709 12:22:13.864312 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:13.805061 21405 leader_election.cc:290] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:13.864956 17762 raft_consensus.cc:2468] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 8.
I20260709 12:22:13.865111 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:13.865463 21228 raft_consensus.cc:2804] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:22:13.865697 17764 raft_consensus.cc:3060] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:13.865911 21228 raft_consensus.cc:493] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:13.866253 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:13.866859 17760 raft_consensus.cc:2468] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 6.
I20260709 12:22:13.867342 18366 raft_consensus.cc:1275] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 12 index: 10. (index mismatch)
I20260709 12:22:13.867682 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:13.868216 17760 raft_consensus.cc:3060] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:13.868942 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:13.800362 17829 raft_consensus.cc:2468] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 8.
I20260709 12:22:13.798913 21343 raft_consensus.cc:493] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:13.797736 21369 leader_election.cc:290] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 10 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:13.803634 21327 leader_election.cc:290] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 11 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:13.800559 21155 raft_consensus.cc:493] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:13.752732 21450 consensus_queue.cc:1059] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 9.10, Next index: 11, Last known committed idx: 9, Time since last communication: 0.083s
W20260709 12:22:13.841799 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 127023us
I20260709 12:22:13.842347 17757 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38786 (request call id 2658) took 3232 ms. Trace:
I20260709 12:22:13.782245 21361 raft_consensus.cc:515] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:22:13.844624 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 96320us
I20260709 12:22:13.886808 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:13.887603 18366 raft_consensus.cc:3060] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:13.765147 21177 raft_consensus.cc:515] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:13.829361 21156 raft_consensus.cc:3060] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:13.863325 21353 raft_consensus.cc:493] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:13.863565 21433 raft_consensus.cc:2749] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:22:13.863946 17838 raft_consensus.cc:1240] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 5. Current term is 6. Ops: [5.6-5.6]
I20260709 12:22:13.866420 21228 raft_consensus.cc:3060] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:13.752249 20921 raft_consensus.cc:515] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:13.872031 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 12 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:13.899672 18365 raft_consensus.cc:2393] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 12: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:22:13.865806 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:13.900281 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:13.785650 17685 raft_consensus.cc:2468] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 9.
I20260709 12:22:13.900946 17758 raft_consensus.cc:2468] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 11.
I20260709 12:22:13.811444 21332 raft_consensus.cc:515] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:13.821058 21455 raft_consensus.cc:2749] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:22:13.829291 21426 raft_consensus.cc:493] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:13.881469 17727 leader_election.cc:304] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:22:13.812644 21085 raft_consensus.cc:515] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:13.882032 17757 rpcz_store.cc:276] 0709 12:22:10.610096 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:10.610284 (+   188us) service_pool.cc:224] Handling call
0709 12:22:10.626336 (+ 16052us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:10.626567 (+   231us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 12:22:11.436301 (+809734us) log.cc:844] Serialized 14 byte log entry
0709 12:22:13.722085 (+2285784us) log.cc:844] Serialized 14 byte log entry
0709 12:22:13.822296 (+100211us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:13.822343 (+    47us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:22:13.822406 (+    63us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:13.842215 (+ 19809us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:13.842306 (+    91us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2189241,"prepare.queue_time_us":575144,"prepare.run_cpu_time_us":76,"prepare.run_wall_time_us":78,"raft.queue_time_us":2142812,"raft.run_cpu_time_us":218,"raft.run_wall_time_us":26231,"thread_start_us":1082441,"threads_started":4,"wal-append.queue_time_us":230901,"wal-append.run_cpu_time_us":10068,"wal-append.run_wall_time_us":1434116}
I20260709 12:22:13.830251 18364 raft_consensus.cc:2393] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 7: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
W20260709 12:22:13.888415 17803 leader_election.cc:336] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Timed out: RequestConsensusVote RPC to 127.16.29.68:42577 timed out after 1.981s (SENT)
W20260709 12:22:13.894707 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 140343us
I20260709 12:22:13.907559 17803 leader_election.cc:304] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:13.861765 21053 raft_consensus.cc:493] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:13.896934 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:13.898361 17764 raft_consensus.cc:2468] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 10.
W20260709 12:22:13.938069 18334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:13.902936 17801 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:13.862759 21231 raft_consensus.cc:515] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:22:13.971136 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 61690.2us
I20260709 12:22:13.972417 17837 raft_consensus.cc:3060] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:13.967006 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:13.977463 18365 raft_consensus.cc:2368] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 8. Current term is 9.
I20260709 12:22:13.979458 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:13.980121 17685 raft_consensus.cc:3060] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:13.879455 21343 raft_consensus.cc:515] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:13.990217 18361 raft_consensus.cc:2468] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 10.
I20260709 12:22:13.991112 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:13.899219 21353 raft_consensus.cc:515] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:13.889886 21155 raft_consensus.cc:515] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:22:13.994663 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 106343us
I20260709 12:22:13.994722 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:13.906873 21317 leader_election.cc:290] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 9 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:13.915856 21426 raft_consensus.cc:3060] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:14.020743 21177 leader_election.cc:290] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:14.000133 21370 raft_consensus.cc:493] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:22:13.995133 21455 raft_consensus.cc:2749] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:22:14.044562 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:14.011445 21332 leader_election.cc:290] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:14.044725 21370 raft_consensus.cc:515] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:13.967819 21053 raft_consensus.cc:3060] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:14.045840 17680 raft_consensus.cc:2368] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 8. Current term is 9.
I20260709 12:22:14.046680 17762 raft_consensus.cc:2410] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for term 10 because replica has last-logged OpId of term: 9 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 6.
I20260709 12:22:14.046885 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:14.013922 21447 raft_consensus.cc:493] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
W20260709 12:22:14.000797 21452 log.cc:927] Time spent T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.132s	user 0.001s	sys 0.000s
I20260709 12:22:14.047941 21447 raft_consensus.cc:515] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:14.049067 18364 raft_consensus.cc:3060] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:13.996033 21424 raft_consensus.cc:697] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 7 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
W20260709 12:22:14.051847 17803 leader_election.cc:336] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Timed out: RequestConsensusVote RPC to 127.16.29.68:42577 timed out after 2.103s (SENT)
I20260709 12:22:14.052205 21424 consensus_queue.cc:237] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:22:14.052891 17803 leader_election.cc:336] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Timed out: RequestConsensusVote RPC to 127.16.29.68:42577 timed out after 2.498s (SENT)
I20260709 12:22:14.056257 17803 leader_election.cc:304] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:14.017786 21231 leader_election.cc:290] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:14.057255 21455 raft_consensus.cc:2749] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260709 12:22:14.057973 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 60058us
I20260709 12:22:14.009608 21085 leader_election.cc:290] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:14.014616 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:14.058811 17802 leader_election.cc:304] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:22:14.059684 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:14.060463 17758 raft_consensus.cc:2393] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 8: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:22:14.060670 21433 raft_consensus.cc:2804] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:22:14.061231 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:14.068574 17758 raft_consensus.cc:2468] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 7.
I20260709 12:22:14.010083 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:14.069628 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:14.069819 18371 raft_consensus.cc:2468] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 10.
I20260709 12:22:14.063045 21433 raft_consensus.cc:697] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 9 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:22:14.053599 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
W20260709 12:22:14.062923 17571 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet d522f70beb1f417cb98940d90b594823 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica e063541542a94969898868bf6eb5ce1b is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 6 committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:22:14.012945 21361 leader_election.cc:290] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:14.073760 21433 consensus_queue.cc:237] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:14.051149 21447 leader_election.cc:290] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:14.055714 21343 leader_election.cc:290] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:14.070086 17680 raft_consensus.cc:2393] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 7: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:22:14.029996 20921 leader_election.cc:290] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:14.074072 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:14.076853 18365 raft_consensus.cc:2368] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 8. Current term is 9.
I20260709 12:22:14.077742 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:14.078517 21444 raft_consensus.cc:515] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:14.079633 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:14.051168 21155 leader_election.cc:290] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:14.079032 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:14.096678 18366 raft_consensus.cc:2468] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 7.
I20260709 12:22:14.079870 21053 raft_consensus.cc:515] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:14.100737 17757 raft_consensus.cc:1240] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 6. Current term is 7. Ops: [6.7-6.7]
I20260709 12:22:14.104866 20394 raft_consensus.cc:493] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:22:14.105571 20394 raft_consensus.cc:3060] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:14.110651 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:14.111702 18362 raft_consensus.cc:2368] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 10. Current term is 11.
I20260709 12:22:14.112107 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:14.113093 17684 raft_consensus.cc:2393] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 10: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:22:14.114238 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
W20260709 12:22:14.118317 17682 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44898 (request call id 774) took 2836 ms (client timeout 2178 ms). Trace:
W20260709 12:22:14.118848 17682 rpcz_store.cc:269] 0709 12:22:11.282132 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:11.334021 (+ 51889us) service_pool.cc:224] Handling call
0709 12:22:14.118080 (+2784059us) spinlock_profiling.cc:243] Waited 2.15 s on lock 0x7b54003467d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:22:14.118225 (+   145us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4948604288}
I20260709 12:22:14.120325 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:14.121059 17682 raft_consensus.cc:2468] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 5.
I20260709 12:22:14.122145 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:14.115566 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:14.123678 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:14.261934 18367 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59886 (request call id 1059) took 2383 ms. Trace:
W20260709 12:22:14.246417 17832 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35042 (request call id 1917) took 2341 ms (client timeout 1751 ms). Trace:
I20260709 12:22:14.092346 17837 raft_consensus.cc:2468] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 7.
I20260709 12:22:14.258152 17755 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38786 (request call id 2650) took 4245 ms. Trace:
I20260709 12:22:14.088219 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:22:14.272220 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 390831us
W20260709 12:22:14.273308 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 554213us
I20260709 12:22:14.131981 18363 raft_consensus.cc:2468] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 7.
I20260709 12:22:14.180930 18364 raft_consensus.cc:2468] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 6.
I20260709 12:22:14.128286 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:14.284286 18371 raft_consensus.cc:2368] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 8. Current term is 9.
I20260709 12:22:14.284607 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:14.285553 18366 raft_consensus.cc:2410] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 8 because replica has last-logged OpId of term: 7 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 6.
I20260709 12:22:14.080956 18365 raft_consensus.cc:2393] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 7: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:22:14.090418 17685 raft_consensus.cc:2468] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 13.
I20260709 12:22:14.286590 17652 leader_election.cc:304] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:14.123771 18334 leader_election.cc:400] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 8 pre-election: Vote denied by peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071) with higher term. Message: Invalid argument: T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 8. Current term is 9.
I20260709 12:22:14.124912 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:14.287428 18334 leader_election.cc:403] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 8 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:22:14.087862 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 8 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:14.086316 17760 raft_consensus.cc:2468] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 9.
W20260709 12:22:14.278164 17832 rpcz_store.cc:269] 0709 12:22:11.900339 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:11.900453 (+   114us) service_pool.cc:224] Handling call
0709 12:22:14.241668 (+2341215us) spinlock_profiling.cc:243] Waited 1.72 s on lock 0x7b540008bbd4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:22:14.241814 (+   146us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3950458112}
I20260709 12:22:14.080267 17680 raft_consensus.cc:2468] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 8.
W20260709 12:22:14.265113 17729 connection.cc:256] RPC call timeout handler was delayed by -1.0149053930003902s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2093s for a response
I20260709 12:22:14.084928 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:14.290109 17683 raft_consensus.cc:3060] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Advancing to term 7
I20260709 12:22:14.290154 17802 leader_election.cc:304] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:22:14.290928 17684 raft_consensus.cc:2368] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 6. Current term is 7.
I20260709 12:22:14.135722 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:14.070448 17758 raft_consensus.cc:2410] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for term 7 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20260709 12:22:14.277880 18367 rpcz_store.cc:276] 0709 12:22:11.878576 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:11.878871 (+   295us) service_pool.cc:224] Handling call
0709 12:22:11.902991 (+ 24120us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:12.611072 (+708081us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:12.611125 (+    53us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:12.801736 (+190611us) log.cc:844] Serialized 26 byte log entry
0709 12:22:14.233139 (+1431403us) spinlock_profiling.cc:243] Waited 9.83 ms on lock 0x7b18003c5f20. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b4a1c84bc 00007f7b4751648b 00007f7b503709a9 00007f7b4753269b 00007f7b4753093f 00007f7b47532c02 00007f7b4a1ba20b 00007f7b4a1bfd23 00007f7b4a1c00d1
0709 12:22:14.233434 (+   295us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:22:14.233486 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:14.233584 (+    98us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:14.233635 (+    51us) raft_consensus.cc:1685] finished
0709 12:22:14.233649 (+    14us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:14.261873 (+ 28224us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":11136,"raft.queue_time_us":101967,"raft.run_cpu_time_us":153,"raft.run_wall_time_us":16994,"spinlock_wait_cycles":22609792,"thread_start_us":1305052,"threads_started":2,"wal-append.queue_time_us":725578}
I20260709 12:22:14.292037 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:14.085867 17756 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38786 (request call id 2665) took 3146 ms. Trace:
I20260709 12:22:14.298475 18362 raft_consensus.cc:1275] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260709 12:22:14.298738 17654 leader_election.cc:304] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
W20260709 12:22:14.299958 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 176343us
W20260709 12:22:14.299785 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 505455us
W20260709 12:22:14.256891 21446 log.cc:927] Time spent T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.587s	user 0.013s	sys 0.007s
I20260709 12:22:14.301968 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:22:14.144505 21418 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.546s	user 0.003s	sys 0.008s
I20260709 12:22:14.354743 21465 raft_consensus.cc:2804] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:22:14.354673 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:22:14.293259 21425 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.653s	user 0.004s	sys 0.007s
I20260709 12:22:14.281823 17755 rpcz_store.cc:276] 0709 12:22:10.012612 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:10.081823 (+ 69211us) service_pool.cc:224] Handling call
0709 12:22:10.161088 (+ 79265us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:10.556821 (+395733us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 12:22:11.431933 (+875112us) log.cc:844] Serialized 14 byte log entry
0709 12:22:13.666317 (+2234384us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:13.666366 (+    49us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:22:13.666426 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:14.257801 (+591375us) spinlock_profiling.cc:243] Waited 395 ms on lock 0x7b5400216e54. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:22:14.257866 (+    65us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:14.258042 (+   176us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2069915,"prepare.queue_time_us":262017,"prepare.run_cpu_time_us":3968,"prepare.run_wall_time_us":39916,"raft.queue_time_us":2094935,"raft.run_cpu_time_us":106,"raft.run_wall_time_us":105,"spinlock_wait_cycles":909542272,"thread_start_us":965128,"threads_started":2,"wal-append.queue_time_us":854246,"wal-append.run_cpu_time_us":2159,"wal-append.run_wall_time_us":1516999}
I20260709 12:22:14.286468 21053 leader_election.cc:290] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 6 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:14.139341 21369 raft_consensus.cc:2804] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:22:14.378053 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:14.302340 21401 raft_consensus.cc:2804] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260709 12:22:14.235477 21467 log.cc:927] Time spent T 00000000000000000000000000000000 P 2193af31cb894ec9b3d65d939a2a6851: Append to log took a long time: real 0.216s	user 0.004s	sys 0.000s
I20260709 12:22:14.345727 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:14.301919 21420 raft_consensus.cc:2749] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Leader pre-election lost for term 8. Reason: Vote denied by peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071) with higher term. Message: Invalid argument: T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 8. Current term is 9.
I20260709 12:22:14.378814 18362 raft_consensus.cc:2468] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 7.
W20260709 12:22:14.343487 18334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:14.389331 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:22:14.360970 21418 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.038s	user 0.005s	sys 0.014s
I20260709 12:22:14.083662 21185 raft_consensus.cc:515] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:22:14.359593 17803 leader_election.cc:336] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Timed out: RequestConsensusVote RPC to 127.16.29.68:42577 timed out after 2.450s (SENT)
I20260709 12:22:14.136694 21337 leader_election.cc:290] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:14.259096 21309 raft_consensus.cc:3055] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:22:14.171042 21445 raft_consensus.cc:2749] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:22:14.260963 21431 raft_consensus.cc:2749] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:22:14.126788 21352 leader_election.cc:290] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 7 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:14.360836 17728 leader_election.cc:304] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
W20260709 12:22:14.418185 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 116913us
I20260709 12:22:14.418745 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:14.419468 17728 leader_election.cc:304] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:22:14.362540 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:14.367178 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:14.366766 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:14.422751 18365 raft_consensus.cc:2468] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 10.
I20260709 12:22:14.419632 17755 raft_consensus.cc:2468] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 6.
I20260709 12:22:14.378196 21369 raft_consensus.cc:493] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:14.429250 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:14.430348 18362 raft_consensus.cc:2368] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 8. Current term is 9.
I20260709 12:22:14.374782 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:14.297142 17756 rpcz_store.cc:276] 0709 12:22:10.939930 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:10.958134 (+ 18204us) service_pool.cc:224] Handling call
0709 12:22:10.958434 (+   300us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:11.924833 (+966399us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:14.085600 (+2160767us) spinlock_profiling.cc:243] Waited 962 ms on lock 0x7b54000fdcd4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:22:14.085802 (+   202us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2211536768}
I20260709 12:22:14.366889 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:14.356943 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:14.381423 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:14.383404 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:14.432905 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:14.433188 17830 raft_consensus.cc:2468] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 7.
I20260709 12:22:14.433624 17764 raft_consensus.cc:3060] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:22:14.360208 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 75768.5us
I20260709 12:22:14.390838 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:14.435097 17758 raft_consensus.cc:2393] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 9: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
W20260709 12:22:14.391522 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 105011us
W20260709 12:22:14.367118 21425 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.755s	user 0.005s	sys 0.007s
I20260709 12:22:14.088436 21228 raft_consensus.cc:515] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:14.312631 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:14.432909 17832 raft_consensus.cc:2468] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 9.
I20260709 12:22:14.436854 17682 raft_consensus.cc:3060] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:22:14.388676 18334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:14.438288 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:14.383849 21414 raft_consensus.cc:493] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:14.411646 17759 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38786 (request call id 2663) took 3631 ms. Trace:
I20260709 12:22:14.441996 17834 raft_consensus.cc:2468] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 10.
I20260709 12:22:14.442298 17759 rpcz_store.cc:276] 0709 12:22:10.780390 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:10.793920 (+ 13530us) service_pool.cc:224] Handling call
0709 12:22:10.796685 (+  2765us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:11.714481 (+917796us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:14.411401 (+2696920us) spinlock_profiling.cc:243] Waited 917 ms on lock 0x7b54003e62d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:22:14.411584 (+   183us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2109041024}
W20260709 12:22:14.411675 18374 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59886 (request call id 1057) took 2917 ms (client timeout 1931 ms). Trace:
W20260709 12:22:14.443813 18374 rpcz_store.cc:269] 0709 12:22:11.494694 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:11.507280 (+ 12586us) service_pool.cc:224] Handling call
0709 12:22:14.411418 (+2904138us) spinlock_profiling.cc:243] Waited 200 ms on lock 0x7b54007701d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:22:14.411582 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":460871424}
I20260709 12:22:14.303858 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
W20260709 12:22:14.416181 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 56645.5us
W20260709 12:22:14.415603 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 116005us
I20260709 12:22:14.452571 17654 leader_election.cc:304] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:22:14.426008 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:14.453649 18371 raft_consensus.cc:3060] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:14.400193 21329 raft_consensus.cc:2749] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 12:22:14.377866 21304 leader_election.cc:290] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 6 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:14.473519 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:14.474119 18366 raft_consensus.cc:3060] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:14.387327 21401 raft_consensus.cc:493] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:14.374022 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:14.476753 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:14.477384 17758 raft_consensus.cc:2468] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 8.
I20260709 12:22:14.396344 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:14.478922 17763 raft_consensus.cc:2393] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 8: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:22:14.426831 17728 leader_election.cc:304] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:22:14.440256 21337 raft_consensus.cc:2804] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:22:14.460925 21413 raft_consensus.cc:3055] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:22:14.481617 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:14.482146 18361 raft_consensus.cc:3060] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:14.425074 21370 leader_election.cc:290] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:14.461735 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:14.441468 21425 raft_consensus.cc:697] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 7 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:22:14.468184 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:14.466509 17683 raft_consensus.cc:2468] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 7.
I20260709 12:22:14.472301 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:14.474999 17760 raft_consensus.cc:2468] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 11.
I20260709 12:22:14.497229 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:14.475174 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:14.497987 18374 raft_consensus.cc:3060] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:14.498610 21480 consensus_queue.cc:1048] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 12:22:14.502588 17755 raft_consensus.cc:2393] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 6: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:22:14.477448 18364 raft_consensus.cc:2410] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 7 because replica has last-logged OpId of term: 6 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 5.
I20260709 12:22:14.439572 17652 leader_election.cc:304] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:14.481093 17576 maintenance_manager.cc:419] P 2193af31cb894ec9b3d65d939a2a6851: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033339
I20260709 12:22:14.428731 21240 raft_consensus.cc:2749] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 12:22:14.417384 21309 raft_consensus.cc:740] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:22:14.440380 21369 raft_consensus.cc:3060] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:14.488703 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:14.489382 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 10 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:14.493052 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:14.461230 21228 leader_election.cc:290] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 8 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:14.494228 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:14.497257 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:14.524982 17760 raft_consensus.cc:2468] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 9.
I20260709 12:22:14.478783 21361 consensus_queue.cc:1048] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.012s
I20260709 12:22:14.481454 21401 raft_consensus.cc:3060] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:22:14.505065 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 65570.7us
I20260709 12:22:14.517311 17756 raft_consensus.cc:2393] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 10: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
W20260709 12:22:14.487517 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 60856.7us
I20260709 12:22:14.497977 17758 raft_consensus.cc:2468] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 7.
I20260709 12:22:14.507695 18368 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59918 (request call id 1195) took 1109 ms. Trace:
I20260709 12:22:14.525739 18374 raft_consensus.cc:2410] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for term 10 because replica has last-logged OpId of term: 9 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 8.
I20260709 12:22:14.529549 17764 raft_consensus.cc:2468] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 9.
I20260709 12:22:14.481451 21413 raft_consensus.cc:740] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 9 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:22:14.536139 17681 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37402 (request call id 2453) took 2622 ms. Trace:
I20260709 12:22:14.482336 21337 raft_consensus.cc:493] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:22:14.541734 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:14.538141 17685 raft_consensus.cc:1240] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 6. Current term is 7. Ops: []
I20260709 12:22:14.502966 21177 raft_consensus.cc:2804] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:22:14.484627 21444 leader_election.cc:290] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 10 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:14.568533 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:14.579799 17762 raft_consensus.cc:2393] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 11: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:22:14.559791 18361 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 9.
I20260709 12:22:14.571072 17652 leader_election.cc:304] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:14.541062 18365 raft_consensus.cc:2468] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 7.
I20260709 12:22:14.525405 17801 leader_election.cc:304] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:14.473846 21414 raft_consensus.cc:515] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:14.573637 18364 raft_consensus.cc:2468] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 7.
I20260709 12:22:14.575302 17687 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37402 (request call id 2455) took 2632 ms. Trace:
I20260709 12:22:14.569636 17681 rpcz_store.cc:276] 0709 12:22:11.913891 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:12.614450 (+700559us) service_pool.cc:224] Handling call
0709 12:22:12.614769 (+   319us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:12.615102 (+   333us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 12:22:13.156891 (+541789us) log.cc:844] Serialized 14 byte log entry
0709 12:22:13.459216 (+302325us) log.cc:844] Serialized 14 byte log entry
0709 12:22:13.462517 (+  3301us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:13.463008 (+   491us) log.cc:844] Serialized 26 byte log entry
0709 12:22:13.463191 (+   183us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:22:13.463452 (+   261us) log.cc:844] Serialized 14 byte log entry
0709 12:22:14.129634 (+666182us) spinlock_profiling.cc:243] Waited 10.1 ms on lock 0x7b180047f740. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b475333f3 00007f7b475357c1 00007f7b47535779 00007f7b47535709 00007f7b475356d1 00007f7b475349cd 0000000000388634 0000000000388479 00007f7b4751683a
0709 12:22:14.228351 (+ 98717us) spinlock_profiling.cc:243] Waited 9.66 ms on lock 0x7b180047f740. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b475333f3 00007f7b475357c1 00007f7b47535779 00007f7b47535709 00007f7b475356d1 00007f7b475349cd 0000000000388634 0000000000388479 00007f7b4751683a
0709 12:22:14.535672 (+307321us) spinlock_profiling.cc:243] Waited 2.35 ms on lock 0x7b180047f740. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b4a1c84bc 00007f7b4751648b 00007f7b503709a9 00007f7b4753269b 00007f7b4753093f 00007f7b47530455 00007f7b4a618060 00007f7b4a62a716 00007f7b4a63b287
0709 12:22:14.535800 (+   128us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:14.535898 (+    98us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:14.535944 (+    46us) raft_consensus.cc:1685] finished
0709 12:22:14.535956 (+    12us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:14.536083 (+   127us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":18736,"prepare.queue_time_us":1097998,"prepare.run_cpu_time_us":124,"prepare.run_wall_time_us":29903,"raft.queue_time_us":43209,"raft.run_cpu_time_us":412,"raft.run_wall_time_us":38422,"spinlock_wait_cycles":52371072,"thread_start_us":176277,"threads_started":2,"wal-append.queue_time_us":315405}
I20260709 12:22:14.587826 17652 leader_election.cc:304] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:14.576241 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:22:14.580444 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 99356.4us
W20260709 12:22:14.581763 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 56464.4us
I20260709 12:22:14.589123 17728 leader_election.cc:304] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:14.590252 17652 leader_election.cc:304] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:14.593482 17728 leader_election.cc:304] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:22:14.576849 18368 rpcz_store.cc:276] 0709 12:22:13.397856 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:13.425649 (+ 27793us) service_pool.cc:224] Handling call
0709 12:22:14.507493 (+1081844us) spinlock_profiling.cc:243] Waited 85.5 ms on lock 0x7b54006ae0d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:22:14.507617 (+   124us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":196690304}
I20260709 12:22:14.594085 17727 leader_election.cc:304] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:14.585064 17687 rpcz_store.cc:276] 0709 12:22:11.943589 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:13.455909 (+1512320us) service_pool.cc:224] Handling call
0709 12:22:13.470614 (+ 14705us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:13.485035 (+ 14421us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:13.485067 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:13.490831 (+  5764us) log.cc:844] Serialized 26 byte log entry
0709 12:22:13.593087 (+102256us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:22:13.667491 (+ 74404us) log.cc:844] Serialized 14 byte log entry
0709 12:22:13.667738 (+   247us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:13.667838 (+   100us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:14.256143 (+588305us) spinlock_profiling.cc:243] Waited 10.3 ms on lock 0x7b4400022b4c. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b47459c85 00007f7b4745a6a2 00007f7b4a1cc1db 00007f7b4a1c7aaf 00007f7b4a1bb86e 00007f7b4a1babb6 00007f7b4a1ba7d8 00007f7b4a1c4221 00007f7b4a1c41d9 00007f7b4a1c4169
0709 12:22:14.569365 (+313222us) raft_consensus.cc:1685] finished
0709 12:22:14.575103 (+  5738us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:14.575246 (+   143us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":25,"prepare.queue_time_us":82712,"prepare.run_cpu_time_us":29,"prepare.run_wall_time_us":30,"raft.queue_time_us":23212,"raft.run_cpu_time_us":268,"raft.run_wall_time_us":19924,"spinlock_wait_cycles":23789312,"thread_start_us":106821,"threads_started":2,"wal-append.queue_time_us":170340}
W20260709 12:22:14.535727 21156 consensus_meta.cc:287] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.622s	user 0.011s	sys 0.001s
I20260709 12:22:14.608222 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:14.608843 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:14.599975 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:14.599923 18366 raft_consensus.cc:2468] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 7.
I20260709 12:22:14.609963 18361 raft_consensus.cc:2368] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 8. Current term is 9.
I20260709 12:22:14.535002 21185 leader_election.cc:290] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 7 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:14.518853 18371 raft_consensus.cc:2468] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 11.
I20260709 12:22:14.610821 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:14.611977 18368 raft_consensus.cc:2368] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 8. Current term is 9.
I20260709 12:22:14.609318 17764 raft_consensus.cc:2393] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 7: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:22:14.614157 17685 raft_consensus.cc:1275] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260709 12:22:14.609799 18362 raft_consensus.cc:2468] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 8.
I20260709 12:22:14.618196 17756 raft_consensus.cc:1240] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 9. Current term is 11. Ops: []
I20260709 12:22:14.597429 17761 raft_consensus.cc:2468] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 7.
I20260709 12:22:14.623484 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:14.575646 21337 raft_consensus.cc:3060] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:14.629112 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:14.624502 18371 raft_consensus.cc:2393] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 6: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:22:14.631747 17728 leader_election.cc:304] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:14.633716 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:14.635727 18366 raft_consensus.cc:2393] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 7: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:22:14.642205 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 13 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:14.582217 21060 raft_consensus.cc:2804] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:22:14.645550 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:14.646324 18364 raft_consensus.cc:3060] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:14.579272 21177 raft_consensus.cc:493] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
W20260709 12:22:14.651655 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 57751.8us
I20260709 12:22:14.595366 21407 raft_consensus.cc:2804] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:22:14.652830 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:14.595536 21370 raft_consensus.cc:2749] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:22:14.587121 21476 raft_consensus.cc:2764] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Leader pre-election decision vote started in defunct term 8: won
I20260709 12:22:14.677838 17728 leader_election.cc:304] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:22:14.682355 18365 raft_consensus.cc:1240] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 12. Current term is 13. Ops: []
I20260709 12:22:14.574405 20394 raft_consensus.cc:515] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:14.687510 18371 raft_consensus.cc:3055] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:22:14.688042 18371 raft_consensus.cc:740] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:22:14.693657 18362 raft_consensus.cc:1240] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 8. Current term is 9. Ops: []
I20260709 12:22:14.648597 21240 raft_consensus.cc:2804] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:22:14.696102 18366 raft_consensus.cc:1275] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260709 12:22:14.696900 17680 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44898 (request call id 779) took 1349 ms. Trace:
I20260709 12:22:14.697983 17680 rpcz_store.cc:276] 0709 12:22:13.347779 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:14.294093 (+946314us) service_pool.cc:224] Handling call
0709 12:22:14.294367 (+   274us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:14.294530 (+   163us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 12:22:14.371487 (+ 76957us) log.cc:844] Serialized 14 byte log entry
0709 12:22:14.696319 (+324832us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:14.696434 (+   115us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:22:14.696499 (+    65us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:14.696613 (+   114us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:14.696804 (+   191us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":25542,"prepare.queue_time_us":68258,"prepare.run_cpu_time_us":31,"prepare.run_wall_time_us":33,"thread_start_us":298402,"threads_started":1}
I20260709 12:22:14.700848 17762 raft_consensus.cc:1275] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 7. (term mismatch)
I20260709 12:22:14.682495 17728 leader_election.cc:304] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:14.669661 21332 raft_consensus.cc:2804] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:22:14.674389 21407 raft_consensus.cc:493] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:14.653040 21060 raft_consensus.cc:493] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:22:14.701366 17762 pending_rounds.cc:85] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb: Aborting all ops after (but not including) 6
I20260709 12:22:14.797405 17762 pending_rounds.cc:107] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb: Aborting uncommitted NO_OP operation due to leader change: 5.7
I20260709 12:22:14.797725 17762 raft_consensus.cc:2889] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:14.623770 21406 consensus_queue.cc:1048] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.067s
I20260709 12:22:14.653745 21185 raft_consensus.cc:493] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
W20260709 12:22:14.678871 21399 log.cc:927] Time spent T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.068s	user 0.000s	sys 0.000s
I20260709 12:22:14.631331 21156 raft_consensus.cc:515] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:14.792447 17682 raft_consensus.cc:2468] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 6.
W20260709 12:22:14.794153 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 111736us
I20260709 12:22:14.689045 18371 consensus_queue.cc:260] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:14.746973 18364 raft_consensus.cc:2468] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 7.
I20260709 12:22:14.669890 21318 raft_consensus.cc:2749] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:22:14.672072 21177 raft_consensus.cc:3060] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:14.640915 21415 consensus_queue.cc:1059] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.060s
W20260709 12:22:14.800052 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 143456us
W20260709 12:22:14.834160 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 243660us
I20260709 12:22:14.817410 18368 raft_consensus.cc:1240] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 6. Current term is 7. Ops: []
W20260709 12:22:14.796386 21426 consensus_meta.cc:287] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.753s	user 0.012s	sys 0.000s
I20260709 12:22:14.705600 21480 raft_consensus.cc:2804] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:22:14.707273 21352 raft_consensus.cc:2804] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:22:14.819497 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
W20260709 12:22:14.826169 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 325060us
I20260709 12:22:14.826735 18371 raft_consensus.cc:3060] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:14.852829 18366 raft_consensus.cc:2368] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 8. Current term is 9.
I20260709 12:22:14.705109 21240 raft_consensus.cc:493] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
W20260709 12:22:14.853106 17729 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.16.29.68:42577 timed out after 2.073s (SENT) [suppressed 27 similar messages]
I20260709 12:22:14.789099 21332 raft_consensus.cc:493] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:14.834882 17727 leader_election.cc:304] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:14.816430 17728 leader_election.cc:304] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:22:14.859786 17727 leader_election.cc:304] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
W20260709 12:22:14.859993 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 43692.2us
I20260709 12:22:14.861975 17728 leader_election.cc:304] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:14.834052 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:14.837759 17652 leader_election.cc:304] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:14.799624 21407 raft_consensus.cc:3060] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:14.801339 21060 raft_consensus.cc:3060] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:14.807061 21370 raft_consensus.cc:2804] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:22:14.851732 17684 raft_consensus.cc:1240] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 6. Current term is 7. Ops: []
W20260709 12:22:14.876482 18334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:14.877995 17652 leader_election.cc:304] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:14.897047 17652 leader_election.cc:304] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:14.852495 21352 raft_consensus.cc:493] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:22:14.941696 18334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:14.867280 21312 raft_consensus.cc:3055] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:22:14.943269 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:22:14.900105 21474 log.cc:927] Time spent T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.420s	user 0.001s	sys 0.001s
W20260709 12:22:14.915671 21459 log.cc:927] Time spent T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 1.004s	user 0.003s	sys 0.000s
I20260709 12:22:14.955448 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:14.899236 21361 raft_consensus.cc:2804] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:22:14.671833 21425 consensus_queue.cc:237] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:14.670359 21309 consensus_queue.cc:260] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 9.10, Last appended by leader: 10, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:14.906985 17652 leader_election.cc:304] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:22:14.966537 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 59684.4us
I20260709 12:22:14.934602 21379 raft_consensus.cc:2804] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:22:14.945994 21352 raft_consensus.cc:3060] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:22:14.912343 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:14.912624 21398 raft_consensus.cc:493] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:14.913476 21304 raft_consensus.cc:2804] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Leader election won for term 9
W20260709 12:22:14.916654 21481 log.cc:927] Time spent T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.090s	user 0.000s	sys 0.000s
I20260709 12:22:14.916496 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:15.003229 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:15.003670 18366 raft_consensus.cc:2468] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 9.
I20260709 12:22:15.004058 17837 raft_consensus.cc:2393] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 7: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:22:14.873870 21228 raft_consensus.cc:2804] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:22:14.876708 21476 raft_consensus.cc:3060] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:14.819468 21185 raft_consensus.cc:515] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:14.874766 21319 raft_consensus.cc:2804] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:22:14.858656 21240 raft_consensus.cc:3060] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:14.866770 21480 raft_consensus.cc:697] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 10 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:22:14.866588 21332 raft_consensus.cc:3060] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:14.874728 21471 raft_consensus.cc:2804] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:22:14.874689 21370 raft_consensus.cc:493] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:14.852679 21426 raft_consensus.cc:515] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:14.933020 21156 leader_election.cc:290] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 7 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:14.701640 21413 consensus_queue.cc:260] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.7, Last appended by leader: 7, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:14.937918 18371 raft_consensus.cc:1275] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 6. (term mismatch)
I20260709 12:22:15.111184 21413 raft_consensus.cc:3060] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:22:14.951403 17831 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35040 (request call id 2191) took 2049 ms (client timeout 2155 ms). Trace:
I20260709 12:22:14.810218 21244 consensus_queue.cc:1048] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.059s
W20260709 12:22:15.112044 17831 rpcz_store.cc:269] 0709 12:22:12.902526 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:12.902698 (+   172us) service_pool.cc:224] Handling call
0709 12:22:14.951132 (+2048434us) spinlock_profiling.cc:243] Waited 700 ms on lock 0x7b54000aafd4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:22:14.951329 (+   197us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1609557888}
I20260709 12:22:14.947693 18362 raft_consensus.cc:2368] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 8. Current term is 9.
I20260709 12:22:15.113314 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:14.865626 18364 raft_consensus.cc:2368] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 6. Current term is 7.
I20260709 12:22:15.113986 17831 raft_consensus.cc:3060] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:14.972221 17652 leader_election.cc:304] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da
W20260709 12:22:15.115020 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 142888us
I20260709 12:22:15.116096 21380 raft_consensus.cc:2749] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:22:15.116914 17652 leader_election.cc:304] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:15.118238 17652 leader_election.cc:304] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:15.119273 17652 leader_election.cc:304] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:14.691391 21478 raft_consensus.cc:2764] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Leader pre-election decision vote started in defunct term 6: won
I20260709 12:22:14.949919 21312 raft_consensus.cc:740] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 6 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:22:14.984939 21490 raft_consensus.cc:493] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:22:14.988092 21304 raft_consensus.cc:697] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 9 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
W20260709 12:22:14.988639 21473 log.cc:927] Time spent T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.390s	user 0.000s	sys 0.001s
I20260709 12:22:14.963450 21361 raft_consensus.cc:493] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:22:14.991793 17729 leader_election.cc:304] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:22:15.151531 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 159837us
I20260709 12:22:15.153733 17729 leader_election.cc:304] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:15.154556 17729 leader_election.cc:304] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:14.994278 21369 raft_consensus.cc:515] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:14.996019 21309 raft_consensus.cc:3060] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:14.996145 17688 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44898 (request call id 780) took 1648 ms. Trace:
W20260709 12:22:15.160372 17801 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:14.997394 21468 log.cc:927] Time spent T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.984s	user 0.000s	sys 0.003s
I20260709 12:22:14.996980 21352 raft_consensus.cc:515] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:15.000809 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 12 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:15.015414 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:22:14.958473 21465 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.534s	user 0.028s	sys 0.066s
W20260709 12:22:15.195335 21465 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.779s	user 0.028s	sys 0.066s
I20260709 12:22:15.195633 21465 raft_consensus.cc:697] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 6 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:22:15.196715 21465 consensus_queue.cc:237] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:22:15.060424 18334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:15.016671 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:14.974924 21379 raft_consensus.cc:493] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:15.200515 17837 raft_consensus.cc:2468] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 7.
I20260709 12:22:15.091144 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:15.092783 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:14.987366 21060 raft_consensus.cc:515] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:15.102557 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:15.100019 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:15.104977 17686 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44898 (request call id 781) took 1408 ms. Trace:
I20260709 12:22:15.091216 21319 raft_consensus.cc:493] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:22:15.107029 21445 consensus_queue.cc:1048] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.004s
I20260709 12:22:15.111454 18371 pending_rounds.cc:85] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da: Aborting all ops after (but not including) 5
I20260709 12:22:15.236608 18371 pending_rounds.cc:107] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da: Aborting uncommitted NO_OP operation due to leader change: 5.6
I20260709 12:22:15.236894 18371 raft_consensus.cc:2889] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:14.680178 21353 leader_election.cc:290] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:15.238009 21353 raft_consensus.cc:2804] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:22:15.238332 21353 raft_consensus.cc:493] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:22:15.238765 17831 raft_consensus.cc:2468] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 10.
I20260709 12:22:15.238711 21353 raft_consensus.cc:3060] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:15.127321 17652 leader_election.cc:304] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:15.218227 21379 raft_consensus.cc:3060] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:15.135152 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:15.240123 17764 raft_consensus.cc:3060] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Advancing to term 10
I20260709 12:22:15.136425 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:15.140709 17761 raft_consensus.cc:1240] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 9. Current term is 10. Ops: [9.9-9.9]
W20260709 12:22:15.143533 17803 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.575s	user 0.037s	sys 0.058s
W20260709 12:22:15.240974 17803 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.672s	user 0.037s	sys 0.059s
W20260709 12:22:15.241415 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 682196us
I20260709 12:22:15.150691 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:15.094494 21228 raft_consensus.cc:697] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 11 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:22:14.995146 21407 raft_consensus.cc:515] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:15.160316 17688 rpcz_store.cc:276] 0709 12:22:13.348128 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:14.995302 (+1647174us) service_pool.cc:224] Handling call
0709 12:22:14.995605 (+   303us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:14.995791 (+   186us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:14.995815 (+    24us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:14.995863 (+    48us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:22:14.995926 (+    63us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:14.995989 (+    63us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:14.996089 (+   100us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:15.108606 21370 raft_consensus.cc:3060] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:15.173388 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:15.174149 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:15.255323 17762 raft_consensus.cc:2368] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for earlier term 6. Current term is 7.
I20260709 12:22:15.176733 18363 raft_consensus.cc:2393] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 12: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:22:15.105909 21471 raft_consensus.cc:493] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:22:15.233278 21244 raft_consensus.cc:2804] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:22:15.195657 21352 leader_election.cc:290] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 9 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:15.100666 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:15.218425 21415 consensus_queue.cc:1059] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 8.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.031s
I20260709 12:22:15.223601 17834 raft_consensus.cc:3060] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:15.224088 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:15.224287 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:15.264017 17759 raft_consensus.cc:3060] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:15.225468 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:15.160859 21361 raft_consensus.cc:3060] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:15.105041 21337 raft_consensus.cc:515] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:15.173127 21053 consensus_queue.cc:1059] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.046s
I20260709 12:22:15.148000 21414 leader_election.cc:290] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:15.014952 21398 raft_consensus.cc:515] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:22:15.214095 17681 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.566s	user 0.002s	sys 0.011s
I20260709 12:22:15.175957 20394 leader_election.cc:290] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 6 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:15.232997 18366 raft_consensus.cc:2393] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 7: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:22:15.233248 17686 rpcz_store.cc:276] 0709 12:22:13.696675 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:14.372245 (+675570us) service_pool.cc:224] Handling call
0709 12:22:14.372770 (+   525us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:14.373168 (+   398us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 12:22:14.640453 (+267285us) log.cc:844] Serialized 14 byte log entry
0709 12:22:15.104623 (+464170us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:15.104692 (+    69us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:22:15.104741 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:15.104813 (+    72us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:15.104922 (+   109us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":15944,"prepare.queue_time_us":39673,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":44,"raft.queue_time_us":11256,"raft.run_cpu_time_us":64,"raft.run_wall_time_us":64,"thread_start_us":646231,"threads_started":2}
I20260709 12:22:14.709007 21444 raft_consensus.cc:2804] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:22:15.240123 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:15.241575 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:15.218307 21478 raft_consensus.cc:2804] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:22:15.194705 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:15.259497 17830 raft_consensus.cc:2393] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 6: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:22:15.160457 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:15.261044 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:15.240391 21319 raft_consensus.cc:3060] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:15.151546 21490 raft_consensus.cc:515] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:15.251152 21433 raft_consensus.cc:493] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:15.274381 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:15.280759 17763 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38776 (request call id 2002) took 1586 ms. Trace:
I20260709 12:22:15.314850 17763 rpcz_store.cc:276] 0709 12:22:13.694519 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:14.087521 (+393002us) service_pool.cc:224] Handling call
0709 12:22:15.280379 (+1192858us) spinlock_profiling.cc:243] Waited 182 ms on lock 0x7b5400410854. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:22:15.280585 (+   206us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":417735936}
I20260709 12:22:15.320150 17760 raft_consensus.cc:2376] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 6. Re-sending same reply.
I20260709 12:22:15.288479 21352 raft_consensus.cc:2749] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:22:15.287571 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:15.328084 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:15.328630 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:15.329396 18374 raft_consensus.cc:2393] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 9: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
W20260709 12:22:15.290400 17681 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.643s	user 0.002s	sys 0.011s
I20260709 12:22:15.258924 21244 raft_consensus.cc:493] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:15.300228 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:15.302186 18363 raft_consensus.cc:2468] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 7.
I20260709 12:22:15.360883 18366 raft_consensus.cc:2368] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 8. Current term is 9.
I20260709 12:22:15.361115 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:15.358847 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:15.303995 18362 raft_consensus.cc:2410] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 8 because replica has last-logged OpId of term: 7 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 6.
I20260709 12:22:15.258975 21471 raft_consensus.cc:3060] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:15.306749 17756 raft_consensus.cc:2393] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 7: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:22:15.361882 18363 raft_consensus.cc:2376] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 11. Re-sending same reply.
I20260709 12:22:15.240775 21480 consensus_queue.cc:237] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.8, Last appended by leader: 8, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:15.297039 21360 raft_consensus.cc:493] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:15.303259 21308 raft_consensus.cc:3055] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:22:15.273741 17764 raft_consensus.cc:2468] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 10.
I20260709 12:22:15.363368 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:22:15.260687 21372 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.875s	user 0.034s	sys 0.030s
I20260709 12:22:15.365305 17681 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44898 (request call id 786) took 1244 ms. Trace:
I20260709 12:22:15.380743 17681 rpcz_store.cc:276] 0709 12:22:14.120603 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:14.588054 (+467451us) service_pool.cc:224] Handling call
0709 12:22:14.629788 (+ 41734us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:14.630182 (+   394us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 12:22:14.647273 (+ 17091us) log.cc:844] Serialized 14 byte log entry
0709 12:22:15.364854 (+717581us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:15.364928 (+    74us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:22:15.364993 (+    65us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:15.365087 (+    94us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:15.365233 (+   146us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4750,"prepare.queue_time_us":16551,"prepare.run_cpu_time_us":75,"prepare.run_wall_time_us":16624,"thread_start_us":713983,"threads_started":1}
I20260709 12:22:15.366333 18365 raft_consensus.cc:2393] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 9: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:22:15.366887 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:15.303325 21478 raft_consensus.cc:493] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:22:15.296751 21444 raft_consensus.cc:493] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:15.312445 21414 raft_consensus.cc:493] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:15.372225 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:15.487073 18364 raft_consensus.cc:2393] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 7: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:22:15.371658 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:15.324488 20394 raft_consensus.cc:2804] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:22:15.423353 17834 raft_consensus.cc:2468] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 10.
I20260709 12:22:15.365021 21244 raft_consensus.cc:3060] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:22:15.383909 21372 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.998s	user 0.034s	sys 0.030s
I20260709 12:22:15.452572 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:15.514086 18371 raft_consensus.cc:2393] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 7: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
W20260709 12:22:15.456709 21484 log.cc:927] Time spent T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.097s	user 0.000s	sys 0.000s
W20260709 12:22:15.457584 21489 log.cc:927] Time spent T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.228s	user 0.001s	sys 0.000s
I20260709 12:22:15.520182 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:22:15.461946 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:15.520850 18364 raft_consensus.cc:2468] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 6.
I20260709 12:22:15.483249 21312 consensus_queue.cc:260] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:22:15.466328 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:15.521693 21312 raft_consensus.cc:3060] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:15.474416 21490 leader_election.cc:290] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:15.297175 21304 consensus_queue.cc:237] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:15.487982 18368 raft_consensus.cc:2468] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 11.
I20260709 12:22:15.380806 21360 raft_consensus.cc:515] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:15.526648 21360 leader_election.cc:290] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:15.492396 21414 raft_consensus.cc:515] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:15.527143 17759 raft_consensus.cc:2468] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 9.
I20260709 12:22:15.434926 21426 leader_election.cc:290] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 7 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:15.528208 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:15.528877 18365 raft_consensus.cc:2468] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 7.
I20260709 12:22:15.494426 21415 raft_consensus.cc:3055] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:22:15.372984 21156 raft_consensus.cc:2804] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:22:15.501758 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:15.529597 21415 raft_consensus.cc:740] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 6 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:22:15.530158 18374 raft_consensus.cc:2393] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 7: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:22:15.529839 21471 raft_consensus.cc:515] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:15.530633 21415 consensus_queue.cc:260] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:15.484710 21306 raft_consensus.cc:2749] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:22:15.487898 21444 raft_consensus.cc:3060] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:15.501781 21369 leader_election.cc:290] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 9 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:15.330255 21433 raft_consensus.cc:515] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:15.529524 21414 leader_election.cc:290] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:15.376192 21308 raft_consensus.cc:740] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 9 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:22:15.424134 21493 raft_consensus.cc:2749] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:22:15.521071 17689 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37402 (request call id 2452) took 3630 ms. Trace:
I20260709 12:22:15.493916 21060 leader_election.cc:290] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 9 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:15.384895 21185 leader_election.cc:290] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:15.386778 17801 leader_election.cc:304] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:15.546981 17759 raft_consensus.cc:1240] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 9. Current term is 10. Ops: []
I20260709 12:22:15.491176 20394 raft_consensus.cc:493] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:22:15.549052 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:15.559816 17689 rpcz_store.cc:276] 0709 12:22:11.890940 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:12.500691 (+609751us) service_pool.cc:224] Handling call
0709 12:22:12.532757 (+ 32066us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:12.533210 (+   453us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:12.533254 (+    44us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:12.571393 (+ 38139us) log.cc:844] Serialized 26 byte log entry
0709 12:22:14.523266 (+1951873us) spinlock_profiling.cc:243] Waited 40.9 ms on lock 0x7b180048fb20. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b47515e07 00007f7b503709a9 00007f7b4753269b 00007f7b4753093f 00007f7b47532c02 00007f7b4a1ba20b 00007f7b4a1bfd23 00007f7b4a1c00d1 00007f7b4a5dba0f
0709 12:22:14.870176 (+346910us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:22:14.870237 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:14.870323 (+    86us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:15.520705 (+650382us) raft_consensus.cc:1685] finished
0709 12:22:15.520735 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:15.521000 (+   265us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":23973,"raft.run_cpu_time_us":136,"raft.run_wall_time_us":40635,"spinlock_wait_cycles":94153984,"thread_start_us":346566,"threads_started":1,"wal-append.queue_time_us":2454148}
I20260709 12:22:15.393952 21478 raft_consensus.cc:3060] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:15.569139 18374 raft_consensus.cc:2410] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 7 because replica has last-logged OpId of term: 6 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 5.
W20260709 12:22:15.562294 17728 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.654s	user 0.018s	sys 0.014s
W20260709 12:22:15.569875 17728 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.662s	user 0.019s	sys 0.014s
W20260709 12:22:15.570587 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 183919us
W20260709 12:22:15.573555 17801 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:15.545987 21415 raft_consensus.cc:3060] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:15.575233 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:22:15.577783 17801 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:15.578871 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:15.546151 21471 leader_election.cc:290] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 11 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:15.368440 21228 consensus_queue.cc:237] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.9, Last appended by leader: 9, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:22:15.602365 17727 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.686s	user 0.007s	sys 0.032s
W20260709 12:22:15.603065 17727 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.687s	user 0.007s	sys 0.032s
I20260709 12:22:15.519665 21379 raft_consensus.cc:515] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:22:15.606454 17757 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.224s	user 0.011s	sys 0.025s
I20260709 12:22:15.575577 20394 raft_consensus.cc:3060] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:22:15.613667 17803 leader_election.cc:336] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Timed out: RequestConsensusVote RPC to 127.16.29.68:42577 timed out after 1.548s (SENT)
I20260709 12:22:15.614110 17803 leader_election.cc:304] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
W20260709 12:22:15.614799 17654 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.146s	user 0.005s	sys 0.028s
W20260709 12:22:15.615281 17654 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.146s	user 0.005s	sys 0.028s
W20260709 12:22:15.615659 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 1.16312e+06us
I20260709 12:22:15.617949 17654 leader_election.cc:304] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
W20260709 12:22:15.595065 21177 consensus_meta.cc:287] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.739s	user 0.010s	sys 0.000s
I20260709 12:22:15.608793 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:15.637370 18365 raft_consensus.cc:2393] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 6: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:22:15.593034 21414 raft_consensus.cc:2749] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260709 12:22:15.619035 17835 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.096s	user 0.005s	sys 0.031s
W20260709 12:22:15.638118 17835 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.129s	user 0.005s	sys 0.031s
I20260709 12:22:15.639147 17835 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35040 (request call id 2202) took 1250 ms. Trace:
I20260709 12:22:15.639652 17835 rpcz_store.cc:276] 0709 12:22:14.388598 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:14.473537 (+ 84939us) service_pool.cc:224] Handling call
0709 12:22:14.492276 (+ 18739us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:14.492549 (+   273us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 12:22:14.508856 (+ 16307us) log.cc:844] Serialized 14 byte log entry
0709 12:22:15.638711 (+1129855us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:15.638783 (+    72us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:22:15.638853 (+    70us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:15.638941 (+    88us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:15.639093 (+   152us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":16561,"prepare.run_cpu_time_us":56,"prepare.run_wall_time_us":27466,"thread_start_us":1129330,"threads_started":1}
I20260709 12:22:15.599507 21407 leader_election.cc:290] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 8 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:22:15.606935 17757 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.225s	user 0.012s	sys 0.025s
I20260709 12:22:15.568347 21332 raft_consensus.cc:515] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:15.567667 21308 consensus_queue.cc:260] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.8, Last appended by leader: 8, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:15.634549 17683 raft_consensus.cc:1240] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 6 LEADER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 5. Current term is 6. Ops: []
W20260709 12:22:15.605950 21404 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.849s	user 0.013s	sys 0.021s
W20260709 12:22:15.646441 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 519185us
I20260709 12:22:15.656659 20394 raft_consensus.cc:515] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:22:15.661501 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 43629.2us
I20260709 12:22:15.665558 17654 leader_election.cc:304] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
W20260709 12:22:15.661229 21404 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.914s	user 0.014s	sys 0.021s
I20260709 12:22:15.670158 17728 leader_election.cc:304] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:15.674381 20394 leader_election.cc:290] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 8 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:15.676229 18368 raft_consensus.cc:1275] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260709 12:22:15.664582 17652 leader_election.cc:304] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:15.677735 17759 raft_consensus.cc:2393] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 8: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:22:15.642009 21334 raft_consensus.cc:2749] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:22:15.622074 21319 raft_consensus.cc:515] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:15.679986 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:15.681771 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:22:15.682521 21240 consensus_meta.cc:287] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.561s	user 0.008s	sys 0.000s
I20260709 12:22:15.625226 21372 consensus_queue.cc:1048] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.042s
I20260709 12:22:15.682575 18364 raft_consensus.cc:1240] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 9. Current term is 10. Ops: []
I20260709 12:22:15.675490 21369 raft_consensus.cc:2804] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:22:15.685613 18366 raft_consensus.cc:1240] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 5. Current term is 6. Ops: []
I20260709 12:22:15.685837 21370 raft_consensus.cc:515] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:15.675472 21185 raft_consensus.cc:2804] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:22:15.675136 21465 raft_consensus.cc:2804] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:22:15.677652 17652 leader_election.cc:304] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:15.729846 17685 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37402 (request call id 2468) took 1602 ms. Trace:
I20260709 12:22:15.734375 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:15.735616 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:15.736809 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:15.747534 17684 raft_consensus.cc:2468] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 10.
I20260709 12:22:15.746349 17685 rpcz_store.cc:276] 0709 12:22:14.127455 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:14.615030 (+487575us) service_pool.cc:224] Handling call
0709 12:22:14.653555 (+ 38525us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:14.697118 (+ 43563us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:14.697156 (+    38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:14.785482 (+ 88326us) log.cc:844] Serialized 26 byte log entry
0709 12:22:15.326809 (+541327us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:22:15.327432 (+   623us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:15.327504 (+    72us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:15.729473 (+401969us) raft_consensus.cc:1685] finished
0709 12:22:15.729624 (+   151us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:15.729778 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":23377,"raft.queue_time_us":126502,"raft.run_cpu_time_us":139,"raft.run_wall_time_us":13876,"thread_start_us":485709,"threads_started":1,"wal-append.queue_time_us":872735}
I20260709 12:22:15.747205 17688 raft_consensus.cc:2468] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 8.
I20260709 12:22:15.749104 17684 raft_consensus.cc:1240] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 5. Current term is 6. Ops: []
I20260709 12:22:15.738442 17682 raft_consensus.cc:1240] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 12. Current term is 13. Ops: [12.12-12.12]
I20260709 12:22:15.739518 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:15.750890 17680 raft_consensus.cc:2468] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 9.
W20260709 12:22:15.746737 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 69162.7us
I20260709 12:22:15.754484 17652 leader_election.cc:304] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:22:15.757597 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:15.758558 17837 raft_consensus.cc:1275] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260709 12:22:15.760489 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
W20260709 12:22:15.745939 21425 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.736s	user 0.023s	sys 0.020s
W20260709 12:22:15.761801 21425 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.763s	user 0.023s	sys 0.021s
I20260709 12:22:15.762511 21465 raft_consensus.cc:697] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 7 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:22:15.688680 21306 raft_consensus.cc:3060] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:15.646533 21177 raft_consensus.cc:515] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:15.764299 17686 raft_consensus.cc:2368] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for earlier term 9. Current term is 10.
I20260709 12:22:15.770812 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:15.764402 17761 raft_consensus.cc:2468] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 8.
W20260709 12:22:15.765666 17801 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:15.727926 21349 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.727s	user 0.006s	sys 0.033s
I20260709 12:22:15.767627 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:15.773722 17833 raft_consensus.cc:1240] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 7 LEADER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 6. Current term is 7. Ops: []
I20260709 12:22:15.774053 20394 raft_consensus.cc:2804] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:22:15.774158 17838 raft_consensus.cc:2468] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 6.
I20260709 12:22:15.774636 20394 raft_consensus.cc:697] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 10 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:22:15.768707 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:15.768826 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:15.767055 17652 leader_election.cc:304] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:15.773154 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
W20260709 12:22:15.773396 17801 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:15.765908 21060 raft_consensus.cc:2804] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:22:15.775548 17803 leader_election.cc:400] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 pre-election: Vote denied by peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577) with higher term. Message: Invalid argument: T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 10. Current term is 11.
I20260709 12:22:15.773953 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:15.775667 17834 raft_consensus.cc:1240] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 9. Current term is 10. Ops: []
I20260709 12:22:15.745630 21185 raft_consensus.cc:493] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:15.746709 21372 consensus_queue.cc:1243] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer 334a4e1f59944544b3b3ff36616b57da log is divergent from this leader: its last log entry 9.9 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 8
W20260709 12:22:15.773829 21349 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.807s	user 0.006s	sys 0.034s
I20260709 12:22:15.847733 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:15.848587 17688 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 6.
I20260709 12:22:15.763470 21465 consensus_queue.cc:237] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 5.8, Last appended by leader: 8, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:22:15.823028 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 65763.4us
I20260709 12:22:15.823293 17689 raft_consensus.cc:3060] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:15.777115 17830 raft_consensus.cc:2393] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 8: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:22:15.788218 17755 raft_consensus.cc:2393] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 7: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
W20260709 12:22:15.789527 17804 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:15.804580 17835 raft_consensus.cc:2368] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 6. Current term is 7.
I20260709 12:22:15.811117 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:15.859858 18362 raft_consensus.cc:3060] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:15.775046 21398 leader_election.cc:290] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:15.747180 21240 raft_consensus.cc:515] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:15.840502 20394 consensus_queue.cc:237] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:15.866730 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:15.869812 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:15.874881 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:15.877039 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:15.844623 21241 raft_consensus.cc:2804] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Leader election won for term 10
W20260709 12:22:15.845387 18367 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59886 (request call id 1062) took 1556 ms (client timeout 1862 ms). Trace:
W20260709 12:22:15.882862 18367 rpcz_store.cc:269] 0709 12:22:14.288530 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:14.292768 (+  4238us) service_pool.cc:224] Handling call
0709 12:22:15.807203 (+1514435us) spinlock_profiling.cc:243] Waited 1.33 s on lock 0x7b5400499654. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:22:15.845305 (+ 38102us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3061437824}
I20260709 12:22:15.784268 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:15.884719 18371 raft_consensus.cc:2393] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 7: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:22:15.770742 21361 raft_consensus.cc:515] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:15.886277 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:15.789882 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 9 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:15.887185 17838 raft_consensus.cc:3060] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:22:15.775823 18333 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:15.796202 17681 raft_consensus.cc:2468] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 7.
I20260709 12:22:15.769814 21369 raft_consensus.cc:697] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:22:15.771590 21352 raft_consensus.cc:493] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:15.803985 17803 leader_election.cc:403] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:22:15.805274 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:15.812495 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:15.898093 17686 raft_consensus.cc:3060] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:15.898339 17758 raft_consensus.cc:1240] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 7. Current term is 8. Ops: []
W20260709 12:22:15.793243 21413 consensus_meta.cc:287] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.681s	user 0.009s	sys 0.002s
I20260709 12:22:15.878851 17762 raft_consensus.cc:2393] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 9: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
W20260709 12:22:15.862291 21401 consensus_meta.cc:287] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 1.261s	user 0.013s	sys 0.000s
I20260709 12:22:15.903517 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
W20260709 12:22:15.880553 17802 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.587s	user 0.019s	sys 0.023s
W20260709 12:22:15.908869 17802 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.615s	user 0.020s	sys 0.023s
I20260709 12:22:15.790179 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:15.910198 17682 raft_consensus.cc:3060] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:15.862923 21360 raft_consensus.cc:493] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
W20260709 12:22:15.891819 17655 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.628s	user 0.005s	sys 0.040s
I20260709 12:22:15.804271 21060 raft_consensus.cc:493] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:15.784076 21308 raft_consensus.cc:3060] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:15.785676 21177 leader_election.cc:290] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 7 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:15.785338 21379 leader_election.cc:290] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 9 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:15.840071 21185 raft_consensus.cc:3060] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:22:15.878003 17729 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.656s	user 0.044s	sys 0.067s
W20260709 12:22:15.952392 17729 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.731s	user 0.044s	sys 0.067s
W20260709 12:22:15.953083 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 798535us
W20260709 12:22:15.954520 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 185608us
I20260709 12:22:15.954735 17729 leader_election.cc:304] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:15.896813 21425 raft_consensus.cc:493] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:15.930073 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
W20260709 12:22:15.932308 17655 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.669s	user 0.006s	sys 0.040s
I20260709 12:22:15.932933 17831 raft_consensus.cc:2368] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 6. Current term is 7.
I20260709 12:22:15.921286 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:15.958653 17835 raft_consensus.cc:3060] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:15.920233 17830 raft_consensus.cc:1275] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 12 index: 10. (index mismatch)
I20260709 12:22:15.928179 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
W20260709 12:22:15.891044 18333 connection.cc:256] RPC call timeout handler was delayed by -3.4870273179999458s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15510000000000002s for a response
I20260709 12:22:15.943094 17760 raft_consensus.cc:1240] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 5. Current term is 6. Ops: []
I20260709 12:22:15.948064 17680 raft_consensus.cc:1240] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 5. Current term is 6. Ops: []
W20260709 12:22:15.960405 18333 connection.cc:256] RPC call timeout handler was delayed by -3.4870273179999458s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15510000000000002s for a response
I20260709 12:22:15.790771 21244 raft_consensus.cc:515] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:15.867321 21444 raft_consensus.cc:515] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:22:15.943495 21353 consensus_meta.cc:287] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.692s	user 0.010s	sys 0.001s
I20260709 12:22:15.968813 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:16.000854 18367 raft_consensus.cc:2468] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 9.
I20260709 12:22:16.001855 17835 raft_consensus.cc:2468] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 6.
I20260709 12:22:15.972214 17728 leader_election.cc:304] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:16.007370 17728 leader_election.cc:304] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:22:15.973019 17801 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:15.960765 18333 connection.cc:256] RPC call timeout handler was delayed by -3.4726273179999225s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1567s for a response
W20260709 12:22:16.008738 18333 connection.cc:256] RPC call timeout handler was delayed by -3.4231273179998425s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1622s for a response
W20260709 12:22:16.009069 18333 connection.cc:256] RPC call timeout handler was delayed by -3.4141273179998279s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1632s for a response
W20260709 12:22:16.009366 18333 connection.cc:256] RPC call timeout handler was delayed by -3.4141273179998279s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1632s for a response
W20260709 12:22:16.009650 18333 connection.cc:256] RPC call timeout handler was delayed by -3.4069273179998163s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.164s for a response
W20260709 12:22:16.009917 18333 connection.cc:256] RPC call timeout handler was delayed by -3.4015273179998076s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1646s for a response
W20260709 12:22:16.010185 18333 connection.cc:256] RPC call timeout handler was delayed by -3.3997273179998047s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1648s for a response
W20260709 12:22:16.010456 18333 connection.cc:256] RPC call timeout handler was delayed by -3.2818273180000688s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1779s for a response
W20260709 12:22:16.010710 18333 connection.cc:256] RPC call timeout handler was delayed by -3.2053273179999451s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1864s for a response
W20260709 12:22:16.010964 18333 connection.cc:256] RPC call timeout handler was delayed by -3.2053273179999451s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1864s for a response
W20260709 12:22:16.011224 18333 connection.cc:256] RPC call timeout handler was delayed by -3.1522273179998592s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19230000000000003s for a response
W20260709 12:22:16.011495 18333 connection.cc:256] RPC call timeout handler was delayed by -3.1405273179998403s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1936s for a response
W20260709 12:22:16.011767 18333 connection.cc:256] RPC call timeout handler was delayed by -3.1315273179998258s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1946s for a response
I20260709 12:22:15.901211 21337 leader_election.cc:290] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 7 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:22:16.012193 18333 connection.cc:256] RPC call timeout handler was delayed by -3.1315273179998258s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1946s for a response
I20260709 12:22:15.975580 21370 leader_election.cc:290] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 12 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:15.964437 21053 raft_consensus.cc:2749] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Leader pre-election lost for term 10. Reason: Vote denied by peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577) with higher term. Message: Invalid argument: T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 10. Current term is 11.
I20260709 12:22:15.976122 21332 leader_election.cc:290] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 8 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:15.976305 21499 consensus_queue.cc:1059] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 12.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:22:15.970569 21398 raft_consensus.cc:2804] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:22:15.980540 21471 consensus_queue.cc:1048] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:22:15.984473 21319 leader_election.cc:290] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 8 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:15.985020 17685 raft_consensus.cc:1275] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
W20260709 12:22:16.015926 21498 log.cc:927] Time spent T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.149s	user 0.001s	sys 0.000s
I20260709 12:22:15.914068 21352 raft_consensus.cc:515] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:15.986311 21244 leader_election.cc:290] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 8 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:22:15.986727 21309 consensus_meta.cc:287] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.785s	user 0.005s	sys 0.005s
I20260709 12:22:15.987962 21361 leader_election.cc:290] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 6 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:15.987166 17689 raft_consensus.cc:2468] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 8.
W20260709 12:22:15.928661 21491 thread.cc:641] apply [worker] (thread pool) Time spent creating pthread: real 0.678s	user 0.030s	sys 0.032s
I20260709 12:22:15.929188 21401 raft_consensus.cc:515] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:15.948022 21060 raft_consensus.cc:3060] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:16.061035 17684 raft_consensus.cc:1240] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 7. Current term is 8. Ops: []
I20260709 12:22:15.996280 21425 raft_consensus.cc:515] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:15.996860 21306 raft_consensus.cc:2749] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:22:15.966785 17729 leader_election.cc:304] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:15.997839 17682 raft_consensus.cc:2468] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 11.
I20260709 12:22:15.906277 21241 raft_consensus.cc:697] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 10 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:22:15.975693 21240 leader_election.cc:290] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 8 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:16.005198 17803 leader_election.cc:400] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 8 pre-election: Vote denied by peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577) with higher term. Message: Invalid argument: T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 8. Current term is 9.
W20260709 12:22:16.066895 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 61838.8us
I20260709 12:22:16.067711 17803 leader_election.cc:400] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 6 election: Vote denied by peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577) with higher term. Message: Invalid argument: T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 6. Current term is 7.
I20260709 12:22:15.974963 21369 consensus_queue.cc:237] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:22:16.007397 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 68964.6us
I20260709 12:22:16.030413 21177 raft_consensus.cc:2804] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:22:16.024463 17728 leader_election.cc:304] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:15.986106 17680 raft_consensus.cc:1240] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 9. Current term is 10. Ops: []
W20260709 12:22:16.004804 21480 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.507s	user 0.015s	sys 0.009s
I20260709 12:22:15.946791 21360 raft_consensus.cc:515] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:22:16.029296 18334 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.599s	user 0.012s	sys 0.053s
W20260709 12:22:16.100713 18334 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.675s	user 0.012s	sys 0.055s
W20260709 12:22:16.101135 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 711880us
W20260709 12:22:16.036583 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:16.039389 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:16.117358 17685 raft_consensus.cc:2393] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 7: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
W20260709 12:22:16.041671 18363 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.518s	user 0.008s	sys 0.017s
I20260709 12:22:16.043911 18361 raft_consensus.cc:2410] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 10 because replica has last-logged OpId of term: 9 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 9.
I20260709 12:22:16.029583 21505 raft_consensus.cc:2804] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:22:16.064584 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:16.143358 17755 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38776 (request call id 2019) took 1007 ms. Trace:
I20260709 12:22:16.143987 17689 raft_consensus.cc:2393] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 7: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:22:16.143831 17755 rpcz_store.cc:276] 0709 12:22:15.136070 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:15.136201 (+   131us) service_pool.cc:224] Handling call
0709 12:22:16.134473 (+998272us) spinlock_profiling.cc:243] Waited 547 ms on lock 0x7b54003eb054. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:22:16.143288 (+  8815us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1258673920}
W20260709 12:22:16.012761 18333 connection.cc:256] RPC call timeout handler was delayed by -3.0199273180001s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.207s for a response
I20260709 12:22:16.145208 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
W20260709 12:22:16.145275 18333 connection.cc:256] RPC call timeout handler was delayed by -3.0199273180001s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.207s for a response
I20260709 12:22:16.145944 17755 raft_consensus.cc:2368] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for earlier term 7. Current term is 8.
I20260709 12:22:16.080087 17803 leader_election.cc:304] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:16.078531 17686 raft_consensus.cc:2468] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 9.
I20260709 12:22:16.018963 21406 consensus_queue.cc:1059] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.043s
I20260709 12:22:16.146972 21470 raft_consensus.cc:2749] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:22:16.143412 21505 raft_consensus.cc:493] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:22:16.147615 21406 raft_consensus.cc:3055] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:22:16.147771 21505 raft_consensus.cc:3060] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:16.148175 21406 raft_consensus.cc:740] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 8 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:22:16.149237 21406 consensus_queue.cc:260] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.9, Last appended by leader: 9, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:16.150719 21406 raft_consensus.cc:3060] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:16.019130 21312 consensus_queue.cc:1059] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.043s
I20260709 12:22:16.152637 21312 raft_consensus.cc:3055] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:22:16.153069 21312 raft_consensus.cc:740] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 6 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:22:16.154098 21312 consensus_queue.cc:260] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:16.007956 21353 raft_consensus.cc:515] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:16.156517 17761 raft_consensus.cc:1240] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 7. Current term is 8. Ops: []
I20260709 12:22:16.157744 21353 leader_election.cc:290] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 8 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:16.094952 17836 raft_consensus.cc:2393] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 9: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:22:16.039868 21398 raft_consensus.cc:697] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:22:16.146795 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:22:16.065088 21491 thread.cc:608] apply [worker] (thread pool) Time spent starting thread: real 0.821s	user 0.030s	sys 0.033s
I20260709 12:22:16.007138 21431 consensus_queue.cc:1048] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.058s
W20260709 12:22:16.098248 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 131558us
I20260709 12:22:16.212422 17729 leader_election.cc:304] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:22:16.099011 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 74663us
I20260709 12:22:16.017086 21478 raft_consensus.cc:515] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:22:16.118149 18363 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.594s	user 0.008s	sys 0.017s
W20260709 12:22:16.117894 18334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:16.220534 18334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:16.084931 21480 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.582s	user 0.016s	sys 0.009s
I20260709 12:22:16.115623 17802 leader_election.cc:304] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:22:16.225270 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 109732us
I20260709 12:22:16.142871 17764 raft_consensus.cc:1240] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 6. Current term is 7. Ops: []
W20260709 12:22:16.145838 18333 connection.cc:256] RPC call timeout handler was delayed by -3.0091273180000826s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2082s for a response
W20260709 12:22:16.227097 18333 connection.cc:256] RPC call timeout handler was delayed by -3.0091273180000826s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2082s for a response
W20260709 12:22:16.227471 18333 connection.cc:256] RPC call timeout handler was delayed by -2.9911273180000535s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2102s for a response
W20260709 12:22:16.081722 17801 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:16.146803 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 66809.8us
I20260709 12:22:16.228401 17803 leader_election.cc:304] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:16.159588 21398 consensus_queue.cc:237] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 9.10, Last appended by leader: 10, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:16.074299 21177 raft_consensus.cc:493] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:16.040205 21349 consensus_queue.cc:1059] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 10.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.061s
I20260709 12:22:16.029098 21501 consensus_queue.cc:1059] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 9.9, Next index: 10, Last known committed idx: 7, Time since last communication: 0.109s
I20260709 12:22:16.254972 17729 leader_election.cc:304] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:22:16.204860 21379 raft_consensus.cc:493] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:22:16.090588 21433 leader_election.cc:290] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:22:16.215085 21498 log.cc:927] Time spent T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.113s	user 0.000s	sys 0.000s
I20260709 12:22:16.240486 17728 leader_election.cc:400] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 9 pre-election: Vote denied by peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071) with higher term. Message: Invalid argument: T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for earlier term 9. Current term is 10.
I20260709 12:22:16.265199 17728 leader_election.cc:403] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 9 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:22:16.204716 21318 raft_consensus.cc:2804] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:22:16.246475 17756 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38776 (request call id 2021) took 1105 ms. Trace:
I20260709 12:22:16.155774 21312 raft_consensus.cc:3060] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:16.251216 17760 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38786 (request call id 2680) took 1013 ms. Trace:
I20260709 12:22:16.284816 17760 rpcz_store.cc:276] 0709 12:22:15.237390 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:15.541169 (+303779us) service_pool.cc:224] Handling call
0709 12:22:15.541524 (+   355us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:15.960580 (+419056us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:16.250919 (+290339us) spinlock_profiling.cc:243] Waited 401 ms on lock 0x7b54000fbed4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:22:16.251142 (+   223us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":923106560}
I20260709 12:22:16.249199 21177 raft_consensus.cc:3060] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:22:16.265843 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:16.266713 18361 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59902 (request call id 977) took 2008 ms. Trace:
I20260709 12:22:16.267207 17758 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38786 (request call id 2674) took 1125 ms. Trace:
I20260709 12:22:16.267815 17761 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38786 (request call id 2683) took 1010 ms. Trace:
I20260709 12:22:16.268760 17683 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37402 (request call id 2459) took 2206 ms. Trace:
I20260709 12:22:16.215646 21510 raft_consensus.cc:2804] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:22:16.299774 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:16.300649 17759 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 6.
I20260709 12:22:16.273419 17756 rpcz_store.cc:276] 0709 12:22:15.141390 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:15.141540 (+   150us) service_pool.cc:224] Handling call
0709 12:22:16.246215 (+1104675us) spinlock_profiling.cc:243] Waited 4.43 ms on lock 0x7b5400415fd4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:22:16.246401 (+   186us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":10190720}
I20260709 12:22:16.279655 17687 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37402 (request call id 2467) took 2152 ms. Trace:
I20260709 12:22:16.301956 17687 rpcz_store.cc:276] 0709 12:22:14.127049 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:14.602831 (+475782us) service_pool.cc:224] Handling call
0709 12:22:14.645934 (+ 43103us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:14.848158 (+202224us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:14.848199 (+    41us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:14.884254 (+ 36055us) log.cc:844] Serialized 26 byte log entry
0709 12:22:15.358845 (+474591us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:22:15.375539 (+ 16694us) log.cc:844] Serialized 14 byte log entry
0709 12:22:15.384106 (+  8567us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:15.384185 (+    79us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:16.279411 (+895226us) raft_consensus.cc:1685] finished
0709 12:22:16.279433 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:16.279591 (+   158us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":931,"prepare.queue_time_us":21098,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":37,"raft.queue_time_us":140115,"raft.run_cpu_time_us":213,"raft.run_wall_time_us":87611,"thread_start_us":675701,"threads_started":2,"wal-append.queue_time_us":887098}
I20260709 12:22:16.141921 21053 consensus_queue.cc:1048] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.114s
I20260709 12:22:16.281005 17729 leader_election.cc:304] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:16.273710 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:16.313854 17762 raft_consensus.cc:2393] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 11: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:22:16.234717 21470 raft_consensus.cc:2804] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:22:16.253458 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:16.314909 17763 raft_consensus.cc:3060] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:22:16.227756 18333 connection.cc:256] RPC call timeout handler was delayed by -2.9893273180000506s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21040000000000003s for a response
I20260709 12:22:16.265038 21337 raft_consensus.cc:2804] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:22:16.254285 21407 raft_consensus.cc:2749] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:22:16.337216 17762 raft_consensus.cc:1275] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20260709 12:22:16.339042 17762 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38786 (request call id 2686) took 1065 ms. Trace:
I20260709 12:22:16.339629 17762 rpcz_store.cc:276] 0709 12:22:15.273590 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:16.336300 (+1062710us) service_pool.cc:224] Handling call
0709 12:22:16.336766 (+   466us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:16.338090 (+  1324us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:16.338949 (+   859us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:22:16.301523 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 418518us
I20260709 12:22:16.294564 17758 rpcz_store.cc:276] 0709 12:22:15.141878 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:15.430527 (+288649us) service_pool.cc:224] Handling call
0709 12:22:15.489467 (+ 58940us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:15.898686 (+409219us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:16.266930 (+368244us) spinlock_profiling.cc:243] Waited 409 ms on lock 0x7b5400310e54. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:22:16.267137 (+   207us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":939887488}
I20260709 12:22:16.292759 18361 rpcz_store.cc:276] 0709 12:22:14.258832 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:14.610589 (+351757us) service_pool.cc:224] Handling call
0709 12:22:16.266436 (+1655847us) spinlock_profiling.cc:243] Waited 1.37 s on lock 0x7b54007e7cd4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:22:16.266627 (+   191us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3155935104}
W20260709 12:22:16.315804 18333 connection.cc:256] RPC call timeout handler was delayed by -2.9524273179999909s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21450000000000002s for a response
W20260709 12:22:16.345314 18333 connection.cc:256] RPC call timeout handler was delayed by -2.9398273179999705s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21589999999999998s for a response
W20260709 12:22:16.345881 18333 connection.cc:256] RPC call timeout handler was delayed by -2.9398273179999705s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21589999999999998s for a response
W20260709 12:22:16.346429 18333 connection.cc:256] RPC call timeout handler was delayed by -2.9047273179999138s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2198s for a response
W20260709 12:22:16.346789 18333 connection.cc:256] RPC call timeout handler was delayed by -2.8993273179999051s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22040000000000004s for a response
W20260709 12:22:16.347157 18333 connection.cc:256] RPC call timeout handler was delayed by -2.8885273179998876s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22160000000000002s for a response
W20260709 12:22:16.347543 18333 connection.cc:256] RPC call timeout handler was delayed by -2.8885273179998876s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22160000000000002s for a response
W20260709 12:22:16.285473 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 45177.7us
I20260709 12:22:16.350397 17728 leader_election.cc:304] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:16.294839 17683 rpcz_store.cc:276] 0709 12:22:14.062054 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:14.534954 (+472900us) service_pool.cc:224] Handling call
0709 12:22:14.575568 (+ 40614us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:15.658635 (+1083067us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:16.268470 (+609835us) spinlock_profiling.cc:243] Waited 1.06 s on lock 0x7b54003471d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:22:16.268692 (+   222us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2434880384}
I20260709 12:22:16.297022 17803 leader_election.cc:400] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 8 election: Vote denied by peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577) with higher term. Message: Invalid argument: T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 8. Current term is 9.
I20260709 12:22:16.294699 17761 rpcz_store.cc:276] 0709 12:22:15.257741 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:15.877054 (+619313us) service_pool.cc:224] Handling call
0709 12:22:15.877415 (+   361us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:16.156893 (+279478us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:16.267538 (+110645us) spinlock_profiling.cc:243] Waited 279 ms on lock 0x7b54002b5a54. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:22:16.267736 (+   198us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":641508608}
W20260709 12:22:16.352149 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 55343.4us
I20260709 12:22:16.353214 17803 leader_election.cc:304] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
W20260709 12:22:16.354035 18333 connection.cc:256] RPC call timeout handler was delayed by -2.8705273179998585s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22360000000000002s for a response
W20260709 12:22:16.354569 18333 connection.cc:256] RPC call timeout handler was delayed by -2.7301273180000862s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2392s for a response
W20260709 12:22:16.354930 18333 connection.cc:256] RPC call timeout handler was delayed by -2.7301273180000862s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2392s for a response
W20260709 12:22:16.355288 18333 connection.cc:256] RPC call timeout handler was delayed by -2.5501273179997952s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.25920000000000004s for a response
W20260709 12:22:16.356206 18333 connection.cc:256] RPC call timeout handler was delayed by -2.5501273179997952s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.25920000000000004s for a response
I20260709 12:22:16.357538 17759 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38786 (request call id 2684) took 1099 ms. Trace:
I20260709 12:22:16.358038 17759 rpcz_store.cc:276] 0709 12:22:15.258129 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:16.356493 (+1098364us) service_pool.cc:224] Handling call
0709 12:22:16.356889 (+   396us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:16.357106 (+   217us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:16.357144 (+    38us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:16.357195 (+    51us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:22:16.357238 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:16.357305 (+    67us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:16.357448 (+   143us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:16.234951 21413 consensus_queue.cc:1059] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.055s
I20260709 12:22:16.369037 17803 leader_election.cc:304] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:22:16.372630 18362 consensus_meta.cc:287] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.507s	user 0.009s	sys 0.000s
I20260709 12:22:16.373076 18362 raft_consensus.cc:2468] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 8.
W20260709 12:22:16.375845 18333 connection.cc:256] RPC call timeout handler was delayed by -2.4241273180000462s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.27320000000000005s for a response
I20260709 12:22:16.276589 21156 raft_consensus.cc:493] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:22:16.376824 18333 leader_election.cc:304] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:16.377767 17728 leader_election.cc:304] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:16.279573 21318 raft_consensus.cc:493] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:22:16.282605 21501 raft_consensus.cc:493] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:16.260807 21240 consensus_queue.cc:1059] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.066s
I20260709 12:22:16.288681 21319 raft_consensus.cc:2749] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260709 12:22:16.382174 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 101248us
I20260709 12:22:16.410368 17729 leader_election.cc:304] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:16.384482 17652 leader_election.cc:304] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:16.290395 21504 raft_consensus.cc:2749] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Leader pre-election lost for term 9. Reason: Vote denied by peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071) with higher term. Message: Invalid argument: T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for earlier term 9. Current term is 10.
W20260709 12:22:16.361876 21415 consensus_meta.cc:287] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.767s	user 0.009s	sys 0.003s
I20260709 12:22:16.416541 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:16.363531 21510 raft_consensus.cc:697] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
W20260709 12:22:16.404618 18334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:16.361747 21334 raft_consensus.cc:493] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:16.278323 21379 raft_consensus.cc:515] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:16.396024 21318 raft_consensus.cc:3060] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:16.410521 17764 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38786 (request call id 2677) took 1267 ms. Trace:
I20260709 12:22:16.465494 17764 rpcz_store.cc:276] 0709 12:22:15.142754 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:15.471689 (+328935us) service_pool.cc:224] Handling call
0709 12:22:16.142688 (+670999us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:16.226599 (+ 83911us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:16.410271 (+183672us) spinlock_profiling.cc:243] Waited 670 ms on lock 0x7b54003eb050. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a61f8ad 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:22:16.410450 (+   179us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1540835072}
I20260709 12:22:16.265267 21244 consensus_queue.cc:1048] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.111s
I20260709 12:22:16.470007 18366 raft_consensus.cc:1275] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260709 12:22:16.324163 21337 raft_consensus.cc:493] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:16.416028 17652 leader_election.cc:400] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 6 pre-election: Vote denied by peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651) with higher term. Message: Invalid argument: T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for earlier term 6. Current term is 7.
I20260709 12:22:16.411026 21407 raft_consensus.cc:2749] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:22:16.411314 21433 raft_consensus.cc:2749] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
W20260709 12:22:16.419307 17836 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35040 (request call id 2194) took 2147 ms (client timeout 2835 ms). Trace:
I20260709 12:22:16.276265 21514 consensus_queue.cc:1048] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.115s
W20260709 12:22:16.437142 17801 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:16.408288 21501 raft_consensus.cc:515] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:16.440757 18361 pending_rounds.cc:85] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da: Aborting all ops after (but not including) 8
I20260709 12:22:16.297986 21414 raft_consensus.cc:493] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:16.389560 21156 raft_consensus.cc:3060] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:22:16.402630 17838 consensus_meta.cc:287] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.515s	user 0.009s	sys 0.000s
I20260709 12:22:16.498502 17838 raft_consensus.cc:2468] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 8.
I20260709 12:22:16.388736 21332 raft_consensus.cc:2749] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:22:16.389745 21499 raft_consensus.cc:2804] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:22:16.390870 17756 raft_consensus.cc:1240] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 8. Current term is 9. Ops: [8.8-8.8]
I20260709 12:22:16.395315 21471 raft_consensus.cc:493] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:16.426455 21228 raft_consensus.cc:2804] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:22:16.475291 17652 leader_election.cc:403] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 6 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:22:16.418222 18362 raft_consensus.cc:1275] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260709 12:22:16.413856 21477 raft_consensus.cc:493] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:16.476024 17762 raft_consensus.cc:1240] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 8. Current term is 9. Ops: [8.9-8.9]
I20260709 12:22:16.285421 21241 consensus_queue.cc:237] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.10, Last appended by leader: 10, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:16.484100 17760 raft_consensus.cc:1240] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 6. Current term is 7. Ops: []
I20260709 12:22:16.484058 18361 pending_rounds.cc:107] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da: Aborting uncommitted NO_OP operation due to leader change: 9.9
I20260709 12:22:16.455819 21240 raft_consensus.cc:3055] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:22:16.440920 21493 raft_consensus.cc:2749] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:22:16.490399 18368 raft_consensus.cc:1275] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
W20260709 12:22:16.511986 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 98797.6us
I20260709 12:22:16.522429 17652 leader_election.cc:304] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:16.512837 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:16.505230 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:16.415305 21322 consensus_queue.cc:1059] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.076s
I20260709 12:22:16.506769 17756 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38786 (request call id 2685) took 1248 ms. Trace:
I20260709 12:22:16.507603 18334 leader_election.cc:400] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 6 election: Vote denied by peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071) with higher term. Message: Invalid argument: T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 6. Current term is 7.
I20260709 12:22:16.484954 21337 raft_consensus.cc:3060] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:16.511188 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
W20260709 12:22:16.481317 17836 rpcz_store.cc:269] 0709 12:22:14.271716 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:14.271849 (+   133us) service_pool.cc:224] Handling call
0709 12:22:16.419125 (+2147276us) spinlock_profiling.cc:243] Waited 1.79 s on lock 0x7b5400267354. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:22:16.419253 (+   128us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4120848256}
I20260709 12:22:16.523662 18361 raft_consensus.cc:2889] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:16.539902 18334 leader_election.cc:403] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 6 election: Cancelling election due to peer responding with higher term
I20260709 12:22:16.536257 18365 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 6.
I20260709 12:22:16.547631 18362 raft_consensus.cc:3060] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:16.527511 17756 rpcz_store.cc:276] 0709 12:22:15.263105 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:16.325748 (+1062643us) service_pool.cc:224] Handling call
0709 12:22:16.390652 (+ 64904us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:16.506502 (+115850us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:16.506702 (+   200us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:16.549196 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:16.524675 18371 raft_consensus.cc:2393] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 8: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
W20260709 12:22:16.494764 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 125782us
I20260709 12:22:16.501282 21309 raft_consensus.cc:493] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:16.541152 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:16.481557 21504 raft_consensus.cc:3060] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:16.549777 18368 raft_consensus.cc:3060] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:16.551095 17803 leader_election.cc:304] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:16.470685 21334 raft_consensus.cc:515] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:16.553544 17803 leader_election.cc:304] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:16.557348 17803 leader_election.cc:304] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:16.558868 17803 leader_election.cc:304] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:22:16.560266 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 183499us
I20260709 12:22:16.521600 21240 raft_consensus.cc:740] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 5 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
W20260709 12:22:16.538070 21503 log.cc:927] Time spent T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.404s	user 0.000s	sys 0.001s
I20260709 12:22:16.522547 21499 raft_consensus.cc:697] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:22:16.509331 21401 leader_election.cc:290] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 9 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:16.504467 21444 leader_election.cc:290] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 11 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:16.505890 21053 consensus_queue.cc:1059] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.046s
I20260709 12:22:16.528383 21372 raft_consensus.cc:2749] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:22:16.524200 21465 raft_consensus.cc:2749] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: Vote denied by peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651) with higher term. Message: Invalid argument: T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for earlier term 6. Current term is 7.
I20260709 12:22:16.509804 21433 consensus_queue.cc:1059] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.048s
I20260709 12:22:16.501282 21414 raft_consensus.cc:515] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:16.574306 17729 leader_election.cc:304] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:16.525426 21228 raft_consensus.cc:697] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:22:16.521896 21407 consensus_queue.cc:1048] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.052s
I20260709 12:22:16.587644 18368 raft_consensus.cc:2468] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 8.
I20260709 12:22:16.556115 21332 raft_consensus.cc:493] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:16.593524 21509 log.cc:927] Time spent T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.363s	user 0.001s	sys 0.000s
I20260709 12:22:16.520888 21477 raft_consensus.cc:515] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:16.529114 21534 raft_consensus.cc:2764] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Leader election decision vote started in defunct term 10: won
I20260709 12:22:16.556453 21360 leader_election.cc:290] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:16.590464 18362 raft_consensus.cc:2468] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 12.
I20260709 12:22:16.511590 21471 raft_consensus.cc:515] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:16.566766 21413 consensus_queue.cc:1059] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.038s
I20260709 12:22:16.560416 21309 raft_consensus.cc:515] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:22:16.644979 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:16.666244 17763 raft_consensus.cc:2468] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 9.
W20260709 12:22:16.695446 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 121247us
W20260709 12:22:16.672612 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 150262us
W20260709 12:22:16.697465 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 213863us
W20260709 12:22:16.662604 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 412374us
I20260709 12:22:16.600733 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:22:16.640769 21516 log.cc:927] Time spent T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.196s	user 0.000s	sys 0.001s
I20260709 12:22:16.591836 21478 leader_election.cc:290] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 10 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:22:16.664685 21476 consensus_meta.cc:287] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 1.541s	user 0.014s	sys 0.000s
W20260709 12:22:16.652671 21530 log.cc:927] Time spent T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.058s	user 0.000s	sys 0.000s
W20260709 12:22:16.614661 21526 log.cc:927] Time spent T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.093s	user 0.001s	sys 0.000s
I20260709 12:22:16.629566 21433 raft_consensus.cc:2749] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:22:16.628388 21543 raft_consensus.cc:2749] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Leader election lost for term 6. Reason: Vote denied by peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071) with higher term. Message: Invalid argument: T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 6. Current term is 7.
I20260709 12:22:16.591796 21510 consensus_queue.cc:237] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:16.593530 21053 raft_consensus.cc:2804] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260709 12:22:16.594139 21520 log.cc:927] Time spent T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.313s	user 0.000s	sys 0.001s
W20260709 12:22:16.648860 21529 log.cc:927] Time spent T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.057s	user 0.000s	sys 0.000s
I20260709 12:22:16.714167 21414 leader_election.cc:290] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:22:16.584052 21502 log.cc:927] Time spent T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.359s	user 0.000s	sys 0.001s
W20260709 12:22:16.598624 21518 log.cc:927] Time spent T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.329s	user 0.000s	sys 0.001s
I20260709 12:22:16.581386 21470 raft_consensus.cc:697] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 6 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:22:16.759944 21476 raft_consensus.cc:2749] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260709 12:22:16.651775 21528 log.cc:927] Time spent T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.103s	user 0.000s	sys 0.000s
I20260709 12:22:16.595925 21425 leader_election.cc:290] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:16.761332 17652 leader_election.cc:304] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
W20260709 12:22:16.652297 21524 log.cc:927] Time spent T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.164s	user 0.001s	sys 0.000s
W20260709 12:22:16.770253 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 191936us
W20260709 12:22:16.771590 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 190663us
I20260709 12:22:16.769642 17729 leader_election.cc:304] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:22:16.652953 21525 log.cc:927] Time spent T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.058s	user 0.000s	sys 0.000s
I20260709 12:22:16.583897 21244 consensus_queue.cc:1048] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.174s
W20260709 12:22:16.820019 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:16.658437 21501 leader_election.cc:290] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:16.622403 21537 raft_consensus.cc:2749] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
W20260709 12:22:16.815834 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 229823us
I20260709 12:22:16.823041 18363 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59902 (request call id 999) took 1323 ms. Trace:
I20260709 12:22:16.842996 21540 raft_consensus.cc:3055] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:22:16.752539 21353 raft_consensus.cc:2804] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:22:16.827037 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:22:16.650749 21185 consensus_meta.cc:287] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.679s	user 0.008s	sys 0.000s
I20260709 12:22:16.773401 21053 raft_consensus.cc:493] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:16.772413 21413 raft_consensus.cc:3055] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:22:16.757679 21533 raft_consensus.cc:2749] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:22:16.752306 21332 raft_consensus.cc:515] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:16.759469 21360 consensus_queue.cc:1048] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.078s
I20260709 12:22:16.754885 21379 leader_election.cc:290] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:16.863687 18363 rpcz_store.cc:276] 0709 12:22:15.500150 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:15.502744 (+  2594us) service_pool.cc:224] Handling call
0709 12:22:15.503035 (+   291us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:15.503228 (+   193us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 12:22:16.311448 (+808220us) spinlock_profiling.cc:243] Waited 2.36 ms on lock 0x7b18002fad60. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b4a1c84bc 00007f7b4751648b 00007f7b503709a9 00007f7b4753269b 00007f7b4753093f 00007f7b47530455 00007f7b4e0183c7 00007f7b4a62a9a7 00007f7b4a63a0bf
0709 12:22:16.311598 (+   150us) log.cc:844] Serialized 14 byte log entry
0709 12:22:16.822634 (+511036us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:16.822705 (+    71us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:22:16.822764 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:16.822844 (+    80us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:16.822984 (+   140us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1940,"prepare.queue_time_us":714670,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":36,"spinlock_wait_cycles":5435136,"thread_start_us":1124469,"threads_started":2}
I20260709 12:22:16.851941 21433 raft_consensus.cc:3055] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:22:16.631721 21499 consensus_queue.cc:237] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 6.3, Last appended by leader: 3, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:22:16.876119 21536 log.cc:927] Time spent T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.282s	user 0.001s	sys 0.000s
I20260709 12:22:16.861054 21538 raft_consensus.cc:2804] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:22:16.828764 21543 consensus_queue.cc:1048] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.044s
I20260709 12:22:16.879021 21053 raft_consensus.cc:3060] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:16.878784 21185 raft_consensus.cc:515] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:16.899500 21332 leader_election.cc:290] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:16.902880 21470 consensus_queue.cc:237] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:16.889411 21334 leader_election.cc:290] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:16.868491 21353 raft_consensus.cc:493] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:16.882941 21228 consensus_queue.cc:237] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.5, Last appended by leader: 5, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:16.878369 21413 raft_consensus.cc:740] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 5 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:22:16.890177 21433 raft_consensus.cc:740] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 7 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:22:16.936589 21433 consensus_queue.cc:260] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:16.937278 21480 raft_consensus.cc:3055] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:22:16.938297 21480 raft_consensus.cc:740] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:22:16.939379 21413 consensus_queue.cc:260] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:16.940315 21370 raft_consensus.cc:3055] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:22:16.941016 21370 raft_consensus.cc:740] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:22:16.942406 21370 consensus_queue.cc:260] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 12.12, Last appended by leader: 12, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:16.874387 21414 consensus_queue.cc:1059] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.046s
I20260709 12:22:16.925787 21353 raft_consensus.cc:3060] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:16.924072 21538 raft_consensus.cc:697] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 8 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:22:16.949879 21538 consensus_queue.cc:237] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:16.951504 21352 leader_election.cc:290] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:16.866891 21540 raft_consensus.cc:740] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 7 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:22:16.938468 21433 raft_consensus.cc:3060] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:16.916152 21541 raft_consensus.cc:493] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:16.975394 21505 consensus_meta.cc:287] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.825s	user 0.010s	sys 0.008s
I20260709 12:22:16.971422 21504 raft_consensus.cc:2749] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:22:16.984985 21471 leader_election.cc:290] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:16.972287 21541 raft_consensus.cc:515] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:22:16.998659 21531 log.cc:927] Time spent T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.477s	user 0.004s	sys 0.002s
I20260709 12:22:16.992830 21540 consensus_queue.cc:260] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:17.005035 21540 raft_consensus.cc:3060] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:17.009168 21541 leader_election.cc:290] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:22:16.851986 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 82452.6us
I20260709 12:22:17.052546 17729 leader_election.cc:304] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:22:16.855409 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 477684us
I20260709 12:22:17.060772 17728 leader_election.cc:304] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:22:17.060845 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 291524us
I20260709 12:22:17.062714 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
W20260709 12:22:17.064589 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 308795us
I20260709 12:22:17.064965 17761 raft_consensus.cc:1240] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 LEADER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 6. Current term is 7. Ops: [6.7-6.7]
I20260709 12:22:17.066728 17760 raft_consensus.cc:1240] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 LEADER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 7. Current term is 8. Ops: []
W20260709 12:22:17.070070 17801 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:17.070771 17801 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:16.883630 18333 leader_election.cc:304] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
W20260709 12:22:17.084851 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 201190us
I20260709 12:22:17.087038 18333 leader_election.cc:304] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:22:17.089955 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:22:16.882593 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 334800us
W20260709 12:22:17.072882 17804 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:17.073132 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:17.092622 17831 raft_consensus.cc:2410] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for term 10 because replica has last-logged OpId of term: 9 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 7.
W20260709 12:22:17.092464 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 331059us
I20260709 12:22:17.074398 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:17.075435 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:17.095181 17829 raft_consensus.cc:2410] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for term 7 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260709 12:22:17.094856 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:17.076638 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:17.078471 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:17.079593 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:17.080586 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:17.081950 17834 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35040 (request call id 2217) took 1212 ms. Trace:
I20260709 12:22:17.097316 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:17.096603 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:17.076464 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:17.099323 17835 raft_consensus.cc:3060] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:17.100018 17837 raft_consensus.cc:2376] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 11. Re-sending same reply.
I20260709 12:22:17.098652 17832 raft_consensus.cc:2368] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for earlier term 9. Current term is 10.
I20260709 12:22:17.102317 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:17.103094 17830 raft_consensus.cc:2468] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 10.
I20260709 12:22:17.111222 17761 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38786 (request call id 2690) took 1754 ms. Trace:
I20260709 12:22:17.112687 17761 rpcz_store.cc:276] 0709 12:22:15.351250 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:16.352110 (+1000860us) service_pool.cc:224] Handling call
0709 12:22:16.392664 (+ 40554us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:17.065733 (+673069us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:17.104896 (+ 39163us) spinlock_profiling.cc:243] Waited 671 ms on lock 0x7b54000fe6d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:22:17.105221 (+   325us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1543922048}
I20260709 12:22:17.115437 21543 raft_consensus.cc:2749] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 12:22:17.116884 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:17.118090 17761 raft_consensus.cc:2468] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 8.
I20260709 12:22:17.120906 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:17.120527 21407 consensus_queue.cc:1059] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:22:17.121522 21309 leader_election.cc:290] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:17.122042 21407 consensus_queue.cc:1243] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer 0da169ff82674dc1947055ce43de68a3 log is divergent from this leader: its last log entry 6.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260709 12:22:17.122820 21309 consensus_queue.cc:1059] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:22:17.123131 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:17.123452 18362 raft_consensus.cc:1240] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 12. Current term is 13. Ops: []
I20260709 12:22:17.124409 21537 consensus_queue.cc:1059] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:22:17.125976 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:17.128345 21401 raft_consensus.cc:3055] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:22:17.128929 21401 raft_consensus.cc:740] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 5 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:22:16.996940 21477 leader_election.cc:290] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:17.132380 21560 raft_consensus.cc:3055] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:22:17.133090 21560 raft_consensus.cc:740] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 8 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:22:16.988996 21505 raft_consensus.cc:515] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:17.135150 21185 leader_election.cc:290] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 8 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:17.105381 17652 leader_election.cc:304] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:17.001360 21480 consensus_queue.cc:260] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:17.139671 21480 raft_consensus.cc:3060] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:17.122961 21407 consensus_queue.cc:1059] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:22:17.111337 17760 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38786 (request call id 2697) took 1500 ms. Trace:
I20260709 12:22:17.119798 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:17.151180 17756 raft_consensus.cc:2368] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 8. Current term is 9.
I20260709 12:22:17.151816 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:17.152541 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:17.152767 17829 raft_consensus.cc:3060] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:17.164654 21352 raft_consensus.cc:2804] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:22:17.153863 17755 raft_consensus.cc:2393] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 9: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:22:17.166613 18367 raft_consensus.cc:1275] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20260709 12:22:17.160501 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:17.162106 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:17.167601 21514 raft_consensus.cc:493] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:17.210959 21514 raft_consensus.cc:515] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:17.214466 21514 leader_election.cc:290] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:17.169397 21561 raft_consensus.cc:493] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:17.215580 21561 raft_consensus.cc:515] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:17.217985 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:17.218652 21561 leader_election.cc:290] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:17.219303 17684 raft_consensus.cc:2393] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 9: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:22:17.222957 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:17.226001 18374 raft_consensus.cc:2368] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for earlier term 11. Current term is 12.
I20260709 12:22:17.225766 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:17.227146 17683 raft_consensus.cc:2393] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 9: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:22:17.105851 17834 rpcz_store.cc:276] 0709 12:22:15.868943 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:15.869435 (+   492us) service_pool.cc:224] Handling call
0709 12:22:15.890690 (+ 21255us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:15.891183 (+   493us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:15.891221 (+    38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:15.911656 (+ 20435us) log.cc:844] Serialized 26 byte log entry
0709 12:22:16.285790 (+374134us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:22:16.461792 (+176002us) log.cc:844] Serialized 14 byte log entry
0709 12:22:16.472346 (+ 10554us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:16.472418 (+    72us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:16.946048 (+473630us) spinlock_profiling.cc:243] Waited 8.07 ms on lock 0x7b1800359320. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b47533522 00007f7b475357c1 00007f7b47535779 00007f7b47535709 00007f7b475356d1 00007f7b475349cd 0000000000388634 0000000000388479 00007f7b4751683a
0709 12:22:17.081198 (+135150us) raft_consensus.cc:1685] finished
0709 12:22:17.081375 (+   177us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:17.081710 (+   335us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":42774,"prepare.queue_time_us":203096,"prepare.run_cpu_time_us":54,"prepare.run_wall_time_us":4015,"raft.queue_time_us":20011,"raft.run_cpu_time_us":296,"raft.run_wall_time_us":38463,"spinlock_wait_cycles":18555648,"thread_start_us":422062,"threads_started":2,"wal-append.queue_time_us":506412}
I20260709 12:22:17.148860 17760 rpcz_store.cc:276] 0709 12:22:15.607668 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:16.560301 (+952633us) service_pool.cc:224] Handling call
0709 12:22:16.580412 (+ 20111us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:17.067380 (+486968us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:17.087637 (+ 20257us) spinlock_profiling.cc:243] Waited 486 ms on lock 0x7b5400410854. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:22:17.107747 (+ 20110us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1116765056}
I20260709 12:22:17.169574 21240 consensus_queue.cc:260] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:22:17.158834 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 72156us
I20260709 12:22:17.155695 20394 raft_consensus.cc:2804] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:22:17.176926 21379 raft_consensus.cc:2749] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
W20260709 12:22:17.177173 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 83385.9us
I20260709 12:22:17.178694 21505 leader_election.cc:290] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 9 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:17.135192 21560 consensus_queue.cc:260] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:17.182651 21370 raft_consensus.cc:3060] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:17.186431 18363 raft_consensus.cc:1240] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 9. Current term is 10. Ops: []
I20260709 12:22:17.159662 21532 raft_consensus.cc:2749] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:22:17.187989 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:17.188429 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 14 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:17.244102 18368 raft_consensus.cc:2410] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for term 14 because replica has last-logged OpId of term: 12 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 11.
W20260709 12:22:17.190166 17654 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:17.191385 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
W20260709 12:22:17.192167 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:17.192469 21308 raft_consensus.cc:3055] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:22:17.189021 21413 raft_consensus.cc:3060] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:22:17.192118 21312 consensus_meta.cc:287] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.903s	user 0.016s	sys 0.001s
I20260709 12:22:17.248689 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
W20260709 12:22:17.249257 21177 consensus_meta.cc:287] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.926s	user 0.014s	sys 0.005s
I20260709 12:22:17.250667 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:17.243937 17835 raft_consensus.cc:2468] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 11.
I20260709 12:22:17.160626 21504 raft_consensus.cc:2804] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:22:17.243402 20394 raft_consensus.cc:697] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 6 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:22:17.248243 18371 raft_consensus.cc:3060] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:17.130949 21401 consensus_queue.cc:260] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:17.250406 21177 raft_consensus.cc:515] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:17.227412 21352 raft_consensus.cc:697] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 12 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:22:17.255168 17831 raft_consensus.cc:2368] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for earlier term 8. Current term is 9.
I20260709 12:22:17.260094 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:17.265726 17761 raft_consensus.cc:2468] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 7.
I20260709 12:22:17.266925 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:17.303632 17682 raft_consensus.cc:2368] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for earlier term 8. Current term is 9.
I20260709 12:22:17.304108 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:17.304883 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:17.306469 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:17.307072 18371 raft_consensus.cc:2468] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 8.
I20260709 12:22:17.287988 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:17.278152 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:17.288830 17759 raft_consensus.cc:1240] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 8. Current term is 9. Ops: []
I20260709 12:22:17.294957 17756 raft_consensus.cc:2393] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 11: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:22:17.261677 21560 raft_consensus.cc:3060] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:17.294253 21177 leader_election.cc:290] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 11 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:17.267550 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:17.259631 17652 leader_election.cc:400] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 7 election: Vote denied by peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651) with higher term. Message: Invalid argument: T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for earlier term 7. Current term is 8.
I20260709 12:22:17.311342 17652 leader_election.cc:403] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 7 election: Cancelling election due to peer responding with higher term
I20260709 12:22:17.266285 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:17.268776 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:17.299208 17680 raft_consensus.cc:1240] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 7. Current term is 8. Ops: [7.7-7.7]
I20260709 12:22:17.299101 17683 raft_consensus.cc:3060] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:17.247105 21308 raft_consensus.cc:740] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 6 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:22:17.255435 21504 raft_consensus.cc:697] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
W20260709 12:22:17.261922 21318 consensus_meta.cc:287] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.774s	user 0.013s	sys 0.001s
I20260709 12:22:17.300930 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:17.265666 21053 raft_consensus.cc:515] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:22:17.301141 21337 consensus_meta.cc:287] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.739s	user 0.015s	sys 0.000s
W20260709 12:22:17.288206 17755 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38776 (request call id 2022) took 1420 ms (client timeout 1647 ms). Trace:
I20260709 12:22:17.288689 21401 raft_consensus.cc:3060] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:17.305130 17761 raft_consensus.cc:2393] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 9: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:22:17.296734 21352 consensus_queue.cc:237] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:17.278569 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:17.306931 21353 raft_consensus.cc:515] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:17.313517 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:17.321605 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:17.262032 21499 raft_consensus.cc:493] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:17.315749 17836 raft_consensus.cc:2468] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 7.
I20260709 12:22:17.296761 21407 raft_consensus.cc:493] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:17.323980 17755 rpcz_store.cc:269] 0709 12:22:15.868212 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:16.146564 (+278352us) service_pool.cc:224] Handling call
0709 12:22:17.270973 (+1124409us) spinlock_profiling.cc:243] Waited 940 ms on lock 0x7b5400216e54. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:22:17.288110 (+ 17137us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2163093376}
W20260709 12:22:17.329653 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 58106.5us
I20260709 12:22:17.331823 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:17.336414 17686 raft_consensus.cc:3060] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:17.258931 21561 consensus_queue.cc:1059] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.065s
I20260709 12:22:17.317034 21312 raft_consensus.cc:493] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:17.279479 20394 consensus_queue.cc:237] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 6, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:17.348006 21541 raft_consensus.cc:493] T 7cb3768d215e409fb9f5baf18d124442 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:17.348984 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:17.352329 17760 raft_consensus.cc:2393] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 9: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:22:17.353443 17687 raft_consensus.cc:3060] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:17.353108 21308 consensus_queue.cc:260] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:17.354893 17830 raft_consensus.cc:1240] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 6. Current term is 7. Ops: [6.6-6.6]
I20260709 12:22:17.355718 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:17.355928 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:17.363072 17829 raft_consensus.cc:2468] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 9.
I20260709 12:22:17.365595 17835 raft_consensus.cc:2393] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 8: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:22:17.366542 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:17.329660 21240 raft_consensus.cc:3060] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:17.368561 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:17.358206 17802 leader_election.cc:304] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:17.375372 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:17.379606 21471 raft_consensus.cc:493] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:17.385802 21504 consensus_queue.cc:237] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:17.386169 21471 raft_consensus.cc:515] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:17.388463 21537 raft_consensus.cc:2749] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:22:17.390008 21053 leader_election.cc:290] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 7 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:17.389665 21499 raft_consensus.cc:515] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:22:17.391949 17727 leader_election.cc:336] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837): Timed out: RequestConsensusVote RPC to 127.16.29.67:36837 timed out after 1.880s (SENT)
I20260709 12:22:17.391845 17762 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38786 (request call id 2688) took 2078 ms. Trace:
I20260709 12:22:17.396343 17762 rpcz_store.cc:276] 0709 12:22:15.313212 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:16.350638 (+1037426us) service_pool.cc:224] Handling call
0709 12:22:16.351091 (+   453us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:16.512207 (+161116us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:17.375348 (+863141us) spinlock_profiling.cc:243] Waited 125 ms on lock 0x7b54004155d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:22:17.391586 (+ 16238us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":286547712}
I20260709 12:22:17.397279 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:17.398041 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:17.399261 17758 raft_consensus.cc:1240] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 LEADER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 10. Current term is 11. Ops: []
I20260709 12:22:17.400383 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:17.400660 17762 raft_consensus.cc:1240] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 8. Current term is 9. Ops: [8.6-8.6]
I20260709 12:22:17.401414 17762 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38776 (request call id 2024) took 1526 ms. Trace:
I20260709 12:22:17.401762 17762 rpcz_store.cc:276] 0709 12:22:15.875215 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:17.399641 (+1524426us) service_pool.cc:224] Handling call
0709 12:22:17.400409 (+   768us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:17.401095 (+   686us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:17.401335 (+   240us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:17.403631 17762 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38776 (request call id 2026) took 1521 ms. Trace:
W20260709 12:22:17.403913 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 74851.8us
I20260709 12:22:17.404053 17762 rpcz_store.cc:276] 0709 12:22:15.882500 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:17.402583 (+1520083us) service_pool.cc:224] Handling call
0709 12:22:17.402977 (+   394us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:17.403225 (+   248us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:17.403260 (+    35us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:17.403317 (+    57us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:22:17.403360 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:17.403437 (+    77us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:17.403564 (+   127us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:17.404896 17802 leader_election.cc:304] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:17.405930 17762 raft_consensus.cc:1240] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 7. Current term is 8. Ops: []
W20260709 12:22:17.406302 17804 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:17.406742 17804 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:17.408952 21156 consensus_meta.cc:287] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.886s	user 0.011s	sys 0.007s
I20260709 12:22:17.409479 21156 raft_consensus.cc:515] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:17.410915 17757 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38776 (request call id 2001) took 3716 ms. Trace:
I20260709 12:22:17.335427 21337 raft_consensus.cc:515] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:17.411293 17757 rpcz_store.cc:276] 0709 12:22:13.694287 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:14.101729 (+407442us) service_pool.cc:224] Handling call
0709 12:22:14.102123 (+   394us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:22:14.264265 (+162142us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:14.264309 (+    44us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:22:14.298131 (+ 33822us) log.cc:844] Serialized 52 byte log entry
0709 12:22:15.642919 (+1344788us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:22:15.642964 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:15.643033 (+    69us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:17.409214 (+1766181us) spinlock_profiling.cc:243] Waited 3.91 ms on lock 0x7b24000f7268. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b473b99c4 00007f7b473b7acb 00007f7b47572dc9 00007f7b47572a55 00007f7b47571fb0 00007f7b4a1e8488 00007f7b4a1e8be8 00007f7b4a1c0398 00007f7b4a1bb76a 00007f7b4a1babb6
0709 12:22:17.410361 (+  1147us) raft_consensus.cc:1685] finished
0709 12:22:17.410381 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:17.410706 (+   325us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":159837,"raft.queue_time_us":162456,"raft.run_cpu_time_us":165,"raft.run_wall_time_us":32650,"spinlock_wait_cycles":8998272,"thread_start_us":1260701,"threads_started":1,"wal-append.queue_time_us":1654333}
I20260709 12:22:17.373081 21561 consensus_queue.cc:1243] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer e2909cff1da44ad387f96a8ae3aa5afb log is divergent from this leader: its last log entry 8.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20260709 12:22:17.414820 17757 raft_consensus.cc:1240] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 7. Current term is 8. Ops: []
I20260709 12:22:17.415561 21561 raft_consensus.cc:2749] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:22:17.416390 21561 raft_consensus.cc:493] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:22:17.417261 17759 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38786 (request call id 2691) took 1840 ms. Trace:
I20260709 12:22:17.417045 21561 raft_consensus.cc:515] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:17.417605 17759 rpcz_store.cc:276] 0709 12:22:15.576504 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:16.393106 (+816602us) service_pool.cc:224] Handling call
0709 12:22:17.288445 (+895339us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:17.321209 (+ 32764us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:17.417070 (+ 95861us) spinlock_profiling.cc:243] Waited 883 ms on lock 0x7b5400216e50. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a61f8ad 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:22:17.417190 (+   120us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2029908224}
I20260709 12:22:17.418185 21510 consensus_queue.cc:1048] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.001s
W20260709 12:22:17.419405 21406 consensus_meta.cc:287] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 1.232s	user 0.009s	sys 0.014s
I20260709 12:22:17.420414 21541 raft_consensus.cc:515] T 7cb3768d215e409fb9f5baf18d124442 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } attrs { replace: true } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:17.421056 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:17.421826 17834 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 6.
I20260709 12:22:17.422366 21551 raft_consensus.cc:3055] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:22:17.422796 21551 raft_consensus.cc:740] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:22:17.423235 17834 raft_consensus.cc:1275] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260709 12:22:17.423905 21551 consensus_queue.cc:260] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:17.425141 21551 raft_consensus.cc:3060] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:22:17.432878 21060 consensus_meta.cc:287] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 1.352s	user 0.008s	sys 0.010s
I20260709 12:22:17.427482 17685 raft_consensus.cc:2393] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 7: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:22:17.433632 21060 raft_consensus.cc:515] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:17.434967 18371 raft_consensus.cc:1275] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 9 index: 10. Preceding OpId from leader: term: 11 index: 11. (index mismatch)
I20260709 12:22:17.427985 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:17.328539 21318 raft_consensus.cc:515] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:22:17.436857 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 177512us
I20260709 12:22:17.437145 17764 raft_consensus.cc:1240] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 9. Current term is 10. Ops: []
I20260709 12:22:17.437309 21398 consensus_queue.cc:1048] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:22:17.437718 17652 leader_election.cc:304] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:17.438601 17764 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38786 (request call id 2694) took 1843 ms. Trace:
I20260709 12:22:17.438979 17764 rpcz_store.cc:276] 0709 12:22:15.595517 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:16.466451 (+870934us) service_pool.cc:224] Handling call
0709 12:22:16.466735 (+   284us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:17.437627 (+970892us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:17.438276 (+   649us) spinlock_profiling.cc:243] Waited 970 ms on lock 0x7b54000ff0d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:22:17.438536 (+   260us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2230995840}
I20260709 12:22:17.440307 21476 raft_consensus.cc:493] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:22:17.354628 21407 raft_consensus.cc:515] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:17.441422 21476 raft_consensus.cc:515] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:17.443164 21407 leader_election.cc:290] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:17.443334 18361 raft_consensus.cc:1275] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260709 12:22:17.428758 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 9 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:17.443770 21476 leader_election.cc:290] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:17.446439 17758 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38786 (request call id 2687) took 2155 ms. Trace:
I20260709 12:22:17.446882 17758 rpcz_store.cc:276] 0709 12:22:15.291529 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:16.376666 (+1085137us) service_pool.cc:224] Handling call
0709 12:22:16.408787 (+ 32121us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:17.399852 (+991065us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:17.446114 (+ 46262us) spinlock_profiling.cc:243] Waited 931 ms on lock 0x7b5400217854. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:22:17.446379 (+   265us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2141455232}
I20260709 12:22:17.447237 21177 consensus_queue.cc:1048] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:22:17.371027 21312 raft_consensus.cc:515] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:17.449863 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:17.451562 21312 leader_election.cc:290] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:17.398454 17760 raft_consensus.cc:2393] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 8: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:22:17.453953 21308 raft_consensus.cc:3060] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:22:17.431166 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 52890us
W20260709 12:22:17.457540 17801 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:17.444000 17836 raft_consensus.cc:2368] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for earlier term 9. Current term is 10.
I20260709 12:22:17.439615 17652 leader_election.cc:304] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:17.458612 17835 raft_consensus.cc:3060] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:17.428329 17727 leader_election.cc:304] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:22:17.435621 17681 raft_consensus.cc:1240] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 7. Current term is 8. Ops: []
W20260709 12:22:17.436179 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 76859.7us
I20260709 12:22:17.440436 21496 raft_consensus.cc:2749] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:22:17.432638 21540 raft_consensus.cc:3055] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:22:17.444533 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:17.472800 17802 leader_election.cc:304] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:17.475466 17683 raft_consensus.cc:2468] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 12.
I20260709 12:22:17.466410 21540 raft_consensus.cc:740] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 7 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:22:17.492139 21540 consensus_queue.cc:260] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 7.3, Last appended by leader: 3, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:17.419252 21561 leader_election.cc:290] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:17.491349 21053 consensus_queue.cc:1059] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 6.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:22:17.578118 17757 raft_consensus.cc:1240] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 6. Current term is 7. Ops: []
W20260709 12:22:17.566182 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 61983.8us
I20260709 12:22:17.501919 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:17.501421 18333 leader_election.cc:304] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:22:17.562166 17687 raft_consensus.cc:2468] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 8.
I20260709 12:22:17.542709 17686 raft_consensus.cc:2468] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 8.
I20260709 12:22:17.608999 17835 raft_consensus.cc:2410] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for term 9 because replica has last-logged OpId of term: 8 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 5.
I20260709 12:22:17.606573 17688 raft_consensus.cc:1240] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 10 LEADER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 9. Current term is 10. Ops: []
I20260709 12:22:17.642292 17680 raft_consensus.cc:2410] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for term 7 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20260709 12:22:17.935385 21312 raft_consensus.cc:2749] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:22:17.503031 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:17.640707 17830 raft_consensus.cc:1240] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 6 LEADER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20260709 12:22:17.937310 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:17.586567 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:17.456110 17682 raft_consensus.cc:2468] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 10.
I20260709 12:22:17.939272 21540 raft_consensus.cc:3060] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:17.939762 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:22:17.606769 18334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:17.940797 17682 raft_consensus.cc:1275] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
W20260709 12:22:17.941543 18334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:17.942287 18334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:17.828569 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:17.666294 18371 raft_consensus.cc:1275] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260709 12:22:17.944785 21060 leader_election.cc:290] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 8 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:22:17.910481 17804 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:17.567339 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 100619us
I20260709 12:22:17.614853 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:17.945698 21060 raft_consensus.cc:493] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:17.946331 18367 raft_consensus.cc:3060] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:17.946511 21060 raft_consensus.cc:515] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:17.948191 21337 leader_election.cc:290] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 10 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:17.627573 17838 raft_consensus.cc:2468] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 6.
W20260709 12:22:17.668104 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:17.526129 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 53448.9us
W20260709 12:22:17.496085 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 56581.4us
W20260709 12:22:17.879400 17829 consensus_meta.cc:287] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.512s	user 0.012s	sys 0.000s
I20260709 12:22:17.951411 17829 raft_consensus.cc:2468] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 7.
W20260709 12:22:17.503042 21433 consensus_meta.cc:287] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.533s	user 0.007s	sys 0.006s
I20260709 12:22:17.938500 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:17.937563 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:17.961539 21501 consensus_queue.cc:1048] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:22:17.962778 21053 raft_consensus.cc:3055] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:22:17.963284 21053 raft_consensus.cc:740] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 7 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
W20260709 12:22:17.963418 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 535189us
I20260709 12:22:17.964257 21319 raft_consensus.cc:2749] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:22:17.640316 21379 raft_consensus.cc:493] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:22:17.961583 21546 consensus_queue.cc:1048] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.004s
I20260709 12:22:17.934595 21353 leader_election.cc:290] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 7 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:17.966231 21476 consensus_queue.cc:1059] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 5.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
W20260709 12:22:17.639966 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 67545.8us
I20260709 12:22:17.969489 21241 raft_consensus.cc:3055] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:22:17.969947 21241 raft_consensus.cc:740] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:22:17.971063 21241 consensus_queue.cc:260] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:22:17.971671 17803 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:17.972216 17803 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:17.972610 17689 raft_consensus.cc:1240] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 7. Current term is 8. Ops: []
I20260709 12:22:17.960369 17836 raft_consensus.cc:3060] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:17.457942 21156 leader_election.cc:290] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 7 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:22:17.954166 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:17.933993 17801 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:17.970029 21575 log.cc:927] Time spent T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.469s	user 0.002s	sys 0.000s
I20260709 12:22:17.943620 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:22:17.982416 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:17.978339 18361 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.525s	user 0.016s	sys 0.017s
W20260709 12:22:17.986930 17728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:17.987600 18361 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.534s	user 0.016s	sys 0.018s
W20260709 12:22:17.987434 17831 consensus_meta.cc:287] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.586s	user 0.009s	sys 0.003s
I20260709 12:22:17.988593 17831 raft_consensus.cc:2468] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 8.
I20260709 12:22:17.542505 21496 raft_consensus.cc:2804] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Leader election won for term 9
W20260709 12:22:17.972144 21370 consensus_meta.cc:287] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.701s	user 0.007s	sys 0.004s
I20260709 12:22:17.991825 17833 raft_consensus.cc:2468] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 8.
I20260709 12:22:17.949661 21185 raft_consensus.cc:2749] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Leader election lost for term 7. Reason: Vote denied by peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651) with higher term. Message: Invalid argument: T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for earlier term 7. Current term is 8.
I20260709 12:22:17.992801 17654 leader_election.cc:400] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 11 election: Vote denied by peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577) with higher term. Message: Invalid argument: T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for earlier term 11. Current term is 12.
I20260709 12:22:18.001399 17654 leader_election.cc:403] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 11 election: Cancelling election due to peer responding with higher term
W20260709 12:22:17.997605 17801 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:18.000505 17687 raft_consensus.cc:1240] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20260709 12:22:17.495186 21499 leader_election.cc:290] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:17.966125 21379 raft_consensus.cc:515] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:18.011323 17829 raft_consensus.cc:1275] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20260709 12:22:18.013191 21379 leader_election.cc:290] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:18.013875 17685 raft_consensus.cc:2468] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 9.
I20260709 12:22:18.017303 18367 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 7.
W20260709 12:22:18.020829 17655 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:18.021549 21562 log.cc:927] Time spent T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.052s	user 0.001s	sys 0.001s
I20260709 12:22:18.021984 21060 leader_election.cc:290] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:17.577562 21407 consensus_queue.cc:1059] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.428s
I20260709 12:22:17.996637 21318 leader_election.cc:290] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 10 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:18.004258 21561 raft_consensus.cc:493] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:18.033442 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:18.034729 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:18.036763 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:18.042146 18365 raft_consensus.cc:2410] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for term 11 because replica has last-logged OpId of term: 9 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 8.
I20260709 12:22:18.038854 17683 raft_consensus.cc:1240] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 8. Current term is 9. Ops: []
W20260709 12:22:18.050637 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 70700.7us
I20260709 12:22:17.543076 21532 consensus_queue.cc:1048] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.477s
I20260709 12:22:18.015981 21496 raft_consensus.cc:697] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 9 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:22:18.052030 17760 raft_consensus.cc:2468] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 10.
I20260709 12:22:18.054865 17763 raft_consensus.cc:2468] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 13.
I20260709 12:22:18.057960 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:18.066493 17758 raft_consensus.cc:2468] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 10.
I20260709 12:22:18.080051 21534 raft_consensus.cc:2749] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:22:18.057911 21480 raft_consensus.cc:2749] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Leader election lost for term 11. Reason: Vote denied by peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577) with higher term. Message: Invalid argument: T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for earlier term 11. Current term is 12.
I20260709 12:22:18.090286 17654 leader_election.cc:304] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
W20260709 12:22:18.104161 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 63221.9us
W20260709 12:22:18.100220 21401 consensus_meta.cc:287] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.765s	user 0.005s	sys 0.008s
I20260709 12:22:18.059561 21561 raft_consensus.cc:515] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:22:18.105091 21570 log.cc:927] Time spent T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.553s	user 0.005s	sys 0.000s
W20260709 12:22:18.112946 21240 consensus_meta.cc:287] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.727s	user 0.013s	sys 0.000s
I20260709 12:22:18.108073 21501 raft_consensus.cc:2804] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Leader election won for term 8
W20260709 12:22:18.155717 17763 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36572 (request call id 705) took 1601 ms (client timeout 1693 ms). Trace:
W20260709 12:22:18.165299 17763 rpcz_store.cc:269] 0709 12:22:16.554292 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:17.061777 (+507485us) service_pool.cc:224] Handling call
0709 12:22:18.155514 (+1093737us) spinlock_profiling.cc:243] Waited 944 ms on lock 0x7b54003f2854. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:22:18.155658 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2171547008}
I20260709 12:22:18.066625 21053 consensus_queue.cc:260] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:22:18.238862 21563 log.cc:927] Time spent T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.069s	user 0.002s	sys 0.000s
I20260709 12:22:18.239097 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
W20260709 12:22:18.239251 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 86068.5us
I20260709 12:22:18.240779 17727 leader_election.cc:400] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 8 election: Vote denied by peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837) with higher term. Message: Invalid argument: T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for earlier term 8. Current term is 9.
I20260709 12:22:18.240636 17683 raft_consensus.cc:2376] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 13. Re-sending same reply.
I20260709 12:22:18.241219 17727 leader_election.cc:403] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 8 election: Cancelling election due to peer responding with higher term
I20260709 12:22:18.245695 17728 leader_election.cc:400] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 8 election: Vote denied by peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071) with higher term. Message: Invalid argument: T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for earlier term 8. Current term is 9.
I20260709 12:22:18.247355 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:18.252341 17838 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35042 (request call id 1938) took 1180 ms. Trace:
I20260709 12:22:18.253955 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 13 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:22:18.241151 21582 log.cc:927] Time spent T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.134s	user 0.000s	sys 0.000s
I20260709 12:22:18.253005 17838 rpcz_store.cc:276] 0709 12:22:17.065110 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:17.073851 (+  8741us) service_pool.cc:224] Handling call
0709 12:22:18.245167 (+1171316us) spinlock_profiling.cc:243] Waited 533 ms on lock 0x7b54000898d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:22:18.245343 (+   176us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1225766016}
W20260709 12:22:18.258208 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 296418us
I20260709 12:22:18.269025 17836 raft_consensus.cc:2468] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 11.
I20260709 12:22:18.270887 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:18.272310 17683 raft_consensus.cc:2410] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for term 11 because replica has last-logged OpId of term: 9 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 8.
I20260709 12:22:18.273303 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:18.278820 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:18.245265 21501 raft_consensus.cc:697] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 8 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:22:18.305835 17687 raft_consensus.cc:1275] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260709 12:22:18.294598 17728 leader_election.cc:304] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:18.282544 21240 raft_consensus.cc:2749] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Leader election lost for term 8. Reason: Vote denied by peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837) with higher term. Message: Invalid argument: T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for earlier term 8. Current term is 9.
W20260709 12:22:18.317857 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 231587us
I20260709 12:22:18.319031 17802 leader_election.cc:304] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:22:18.321218 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 80651.5us
I20260709 12:22:18.324214 17652 leader_election.cc:304] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:18.329624 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
W20260709 12:22:18.327636 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 826307us
I20260709 12:22:18.333738 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:18.248346 21496 consensus_queue.cc:237] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:18.342063 21470 consensus_queue.cc:1059] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 10.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.002s
I20260709 12:22:18.321143 21407 consensus_queue.cc:1059] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 9.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.024s
I20260709 12:22:18.345204 21407 raft_consensus.cc:2764] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Leader pre-election decision vote started in defunct term 10: won
I20260709 12:22:18.345333 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:18.346027 17687 raft_consensus.cc:3060] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:18.340227 17727 leader_election.cc:304] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:22:18.345932 21407 raft_consensus.cc:493] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:18.348547 21407 raft_consensus.cc:515] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:18.350504 21561 leader_election.cc:290] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:18.350955 21407 leader_election.cc:290] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:18.344445 21470 consensus_queue.cc:1243] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer e2909cff1da44ad387f96a8ae3aa5afb log is divergent from this leader: its last log entry 11.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260709 12:22:18.362957 17687 raft_consensus.cc:2468] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 10.
W20260709 12:22:18.364413 18334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:18.364511 17834 raft_consensus.cc:2410] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for term 8 because replica has last-logged OpId of term: 7 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 3.
I20260709 12:22:18.365501 17833 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35042 (request call id 1945) took 1296 ms. Trace:
I20260709 12:22:18.363545 17689 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37402 (request call id 2479) took 1064 ms. Trace:
I20260709 12:22:18.369127 17689 rpcz_store.cc:276] 0709 12:22:17.299646 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:17.451611 (+151965us) service_pool.cc:224] Handling call
0709 12:22:17.953901 (+502290us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:17.973267 (+ 19366us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:18.363249 (+389982us) spinlock_profiling.cc:243] Waited 483 ms on lock 0x7b54000c5b50. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a61f8ad 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:22:18.363441 (+   192us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1110158720}
I20260709 12:22:18.366221 17683 raft_consensus.cc:1240] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 9. Current term is 10. Ops: []
I20260709 12:22:18.353312 17757 raft_consensus.cc:1275] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 4. (term mismatch)
I20260709 12:22:18.364557 17830 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35042 (request call id 1954) took 1219 ms. Trace:
I20260709 12:22:18.370513 17757 pending_rounds.cc:85] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb: Aborting all ops after (but not including) 3
I20260709 12:22:18.370885 17757 pending_rounds.cc:107] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb: Aborting uncommitted NO_OP operation due to leader change: 5.4
I20260709 12:22:18.371414 17757 raft_consensus.cc:2889] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:18.371903 18367 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 9.
I20260709 12:22:18.368954 17833 rpcz_store.cc:276] 0709 12:22:17.069076 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:17.080126 (+ 11050us) service_pool.cc:224] Handling call
0709 12:22:18.365131 (+1285005us) spinlock_profiling.cc:243] Waited 891 ms on lock 0x7b540022b854. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:22:18.365436 (+   305us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2049416448}
I20260709 12:22:18.375239 21480 raft_consensus.cc:3060] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:18.370618 17830 rpcz_store.cc:276] 0709 12:22:17.144835 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:17.421197 (+276362us) service_pool.cc:224] Handling call
0709 12:22:17.421543 (+   346us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:17.937459 (+515916us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:18.364319 (+426860us) spinlock_profiling.cc:243] Waited 165 ms on lock 0x7b54002eb654. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:22:18.364497 (+   178us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":379227520}
W20260709 12:22:18.382905 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 64130.6us
I20260709 12:22:18.385812 21318 raft_consensus.cc:2749] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:22:18.389606 21241 raft_consensus.cc:3060] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:18.406854 17685 raft_consensus.cc:1240] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 6. Current term is 7. Ops: []
W20260709 12:22:18.407547 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 67664.9us
I20260709 12:22:18.408540 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
W20260709 12:22:18.411587 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:18.413849 21541 leader_election.cc:290] T 7cb3768d215e409fb9f5baf18d124442 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:18.416411 21501 consensus_queue.cc:237] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:22:18.420008 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:18.382268 21561 raft_consensus.cc:493] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:18.420629 21561 raft_consensus.cc:515] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:18.423254 21561 leader_election.cc:290] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:22:18.424834 17655 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:18.428517 17762 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38786 (request call id 2702) took 1338 ms. Trace:
W20260709 12:22:18.428802 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 140205us
I20260709 12:22:18.429641 17728 leader_election.cc:304] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:18.431010 18333 leader_election.cc:304] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3
W20260709 12:22:18.431468 17804 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:18.438197 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 13 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:18.440936 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:18.396968 21398 raft_consensus.cc:2749] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:22:18.441325 17727 leader_election.cc:304] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
W20260709 12:22:18.445209 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 121164us
I20260709 12:22:18.439846 17687 raft_consensus.cc:1240] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 5. Current term is 6. Ops: []
I20260709 12:22:18.446926 17688 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44898 (request call id 789) took 2582 ms. Trace:
I20260709 12:22:18.447306 17757 raft_consensus.cc:2468] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 9.
I20260709 12:22:18.447391 17688 rpcz_store.cc:276] 0709 12:22:15.864515 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:15.864652 (+   137us) service_pool.cc:224] Handling call
0709 12:22:15.864931 (+   279us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:17.935654 (+2070723us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:18.409861 (+474207us) spinlock_profiling.cc:243] Waited 1.71 s on lock 0x7b54000c4c54. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:22:18.446834 (+ 36973us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3924924672}
I20260709 12:22:18.449453 17681 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44898 (request call id 791) took 2478 ms. Trace:
I20260709 12:22:18.449568 17688 raft_consensus.cc:1240] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 7. Current term is 8. Ops: []
I20260709 12:22:18.449996 17801 leader_election.cc:304] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:18.449884 17681 rpcz_store.cc:276] 0709 12:22:15.970584 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:15.977115 (+  6531us) service_pool.cc:224] Handling call
0709 12:22:15.977509 (+   394us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:17.497698 (+1520189us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:18.449082 (+951384us) spinlock_profiling.cc:243] Waited 1.46 s on lock 0x7b54003458d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:22:18.449392 (+   310us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3352196352}
I20260709 12:22:18.451258 17688 raft_consensus.cc:1240] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 10. Current term is 11. Ops: []
I20260709 12:22:18.448158 21433 raft_consensus.cc:3055] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:22:18.452044 21433 raft_consensus.cc:740] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 8 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:22:18.452207 21600 consensus_queue.cc:1059] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.016s
I20260709 12:22:18.454427 21471 leader_election.cc:290] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:18.455156 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:18.457192 17685 raft_consensus.cc:1275] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260709 12:22:18.457623 17689 raft_consensus.cc:1275] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 8 index: 4. (index mismatch)
I20260709 12:22:18.458132 17684 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37402 (request call id 2474) took 1236 ms. Trace:
I20260709 12:22:18.458510 17684 rpcz_store.cc:276] 0709 12:22:17.222132 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:17.222358 (+   226us) service_pool.cc:224] Handling call
0709 12:22:18.457912 (+1235554us) spinlock_profiling.cc:243] Waited 835 ms on lock 0x7b5400345dd4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:22:18.458055 (+   143us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1920933632}
I20260709 12:22:18.460182 18364 raft_consensus.cc:1240] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 6. Current term is 7. Ops: []
W20260709 12:22:18.462152 21308 consensus_meta.cc:287] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 1.008s	user 0.007s	sys 0.008s
I20260709 12:22:18.455148 21590 raft_consensus.cc:2804] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:22:18.463006 21590 raft_consensus.cc:493] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:18.463418 21590 raft_consensus.cc:3060] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:18.458168 17680 raft_consensus.cc:1275] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 10 index: 5. Preceding OpId from leader: term: 11 index: 6. (index mismatch)
I20260709 12:22:18.454542 17688 raft_consensus.cc:1240] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 9. Current term is 10. Ops: []
W20260709 12:22:18.424475 21551 consensus_meta.cc:287] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.989s	user 0.013s	sys 0.002s
I20260709 12:22:18.451571 17681 raft_consensus.cc:1275] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
W20260709 12:22:18.451560 21573 log.cc:927] Time spent T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.141s	user 0.004s	sys 0.000s
I20260709 12:22:18.478865 17686 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37402 (request call id 2480) took 1162 ms. Trace:
I20260709 12:22:18.479432 17728 leader_election.cc:304] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:22:18.472920 18334 leader_election.cc:304] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:22:18.429020 17762 rpcz_store.cc:276] 0709 12:22:17.090596 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:17.406650 (+316054us) service_pool.cc:224] Handling call
0709 12:22:17.407142 (+   492us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:17.953693 (+546551us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:17.953735 (+    42us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:18.018162 (+ 64427us) log.cc:844] Serialized 26 byte log entry
0709 12:22:18.322017 (+303855us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:22:18.322077 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:18.322176 (+    99us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:18.427407 (+105231us) raft_consensus.cc:1685] finished
0709 12:22:18.427433 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:18.428455 (+  1022us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":58748,"raft.queue_time_us":553484,"raft.run_cpu_time_us":126,"raft.run_wall_time_us":126,"thread_start_us":260523,"threads_started":1,"wal-append.queue_time_us":341871}
I20260709 12:22:18.485877 17652 leader_election.cc:304] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:18.479487 17686 rpcz_store.cc:276] 0709 12:22:17.308092 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:17.951751 (+643659us) service_pool.cc:224] Handling call
0709 12:22:18.002064 (+ 50313us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:18.055480 (+ 53416us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:18.055519 (+    39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:18.073585 (+ 18066us) log.cc:844] Serialized 26 byte log entry
0709 12:22:18.398701 (+325116us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:22:18.398744 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:18.398813 (+    69us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:18.470500 (+ 71687us) raft_consensus.cc:1685] finished
0709 12:22:18.470530 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:18.470683 (+   153us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2913,"raft.queue_time_us":5340,"raft.run_cpu_time_us":184,"raft.run_wall_time_us":9347,"thread_start_us":244985,"threads_started":1,"wal-append.queue_time_us":340629}
I20260709 12:22:18.488701 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:18.490659 17801 leader_election.cc:304] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
W20260709 12:22:18.494397 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 57430.7us
I20260709 12:22:18.500607 17687 raft_consensus.cc:1240] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 7. Current term is 8. Ops: []
I20260709 12:22:18.478015 21471 raft_consensus.cc:2749] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 12:22:18.513800 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:22:18.514626 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 90537.2us
W20260709 12:22:18.515543 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:18.481276 21541 raft_consensus.cc:2749] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:22:18.525159 18361 raft_consensus.cc:1275] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260709 12:22:18.526052 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:18.504493 21398 raft_consensus.cc:2749] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 12:22:18.503545 21551 raft_consensus.cc:2804] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:22:18.511965 21308 raft_consensus.cc:2804] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:22:18.536084 21308 raft_consensus.cc:493] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:22:18.539516 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 68948.6us
I20260709 12:22:18.540864 17728 leader_election.cc:304] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
W20260709 12:22:18.541538 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 130753us
I20260709 12:22:18.514129 21499 raft_consensus.cc:493] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:18.522308 21312 raft_consensus.cc:493] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:18.538055 17801 leader_election.cc:304] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:18.536366 21401 raft_consensus.cc:3055] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:22:18.527859 21413 raft_consensus.cc:2804] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:22:18.550197 17652 leader_election.cc:304] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:22:18.530520 21588 raft_consensus.cc:493] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:18.542977 21308 raft_consensus.cc:3060] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:18.522163 21318 consensus_queue.cc:1048] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.030s
I20260709 12:22:18.598028 17728 leader_election.cc:304] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:22:18.573220 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:18.536741 21534 consensus_queue.cc:1059] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 9.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.016s
I20260709 12:22:18.602351 21534 raft_consensus.cc:3055] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:22:18.602788 21534 raft_consensus.cc:740] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:22:18.603781 21534 consensus_queue.cc:260] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:18.605237 21534 raft_consensus.cc:3060] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:18.550647 21540 consensus_queue.cc:1048] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.056s
I20260709 12:22:18.609541 21433 consensus_queue.cc:260] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:18.611100 21433 raft_consensus.cc:3060] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:18.612088 21600 consensus_queue.cc:1048] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:22:18.530627 21532 consensus_queue.cc:1048] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.041s
I20260709 12:22:18.612983 21592 consensus_queue.cc:1059] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 7.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:22:18.614315 21532 consensus_queue.cc:1059] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 12:22:18.614670 21470 raft_consensus.cc:3055] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:22:18.615177 21538 raft_consensus.cc:2804] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:22:18.584316 21379 raft_consensus.cc:2749] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:22:18.568473 21337 raft_consensus.cc:2749] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:22:18.584724 21595 raft_consensus.cc:493] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:18.527678 21476 consensus_queue.cc:1059] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.033s
I20260709 12:22:18.615862 21538 raft_consensus.cc:697] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:22:18.568820 21560 raft_consensus.cc:2804] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:22:18.617156 21476 raft_consensus.cc:3055] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 6 LEADER]: Stepping down as leader of term 6
W20260709 12:22:18.587154 17803 leader_election.cc:336] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Timed out: RequestConsensusVote RPC to 127.16.29.68:42577 timed out after 3.048s (SENT)
I20260709 12:22:18.617547 21560 raft_consensus.cc:493] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:18.617887 17803 leader_election.cc:304] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:18.617642 21538 consensus_queue.cc:237] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:18.578624 21312 raft_consensus.cc:515] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:18.549759 21309 consensus_queue.cc:1059] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.042s
I20260709 12:22:18.616032 21370 consensus_queue.cc:1048] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.004s
I20260709 12:22:18.620747 21312 leader_election.cc:290] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:18.549412 21401 raft_consensus.cc:740] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 9 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:22:18.617647 21476 raft_consensus.cc:740] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:22:18.584002 21561 consensus_queue.cc:1059] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 9.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 12:22:18.621390 21312 raft_consensus.cc:2749] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:22:18.558132 21413 raft_consensus.cc:493] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:18.622365 21413 raft_consensus.cc:3060] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:18.592389 21053 raft_consensus.cc:3060] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:18.618014 21560 raft_consensus.cc:3060] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:22:18.595036 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 60350.9us
I20260709 12:22:18.612300 18333 leader_election.cc:304] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:22:18.552951 21477 raft_consensus.cc:2749] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:22:18.629093 21337 consensus_queue.cc:1048] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.002s
I20260709 12:22:18.559638 21593 consensus_queue.cc:1059] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 10.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.071s
I20260709 12:22:18.632618 18362 raft_consensus.cc:1275] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 9 index: 10. Preceding OpId from leader: term: 10 index: 11. (index mismatch)
I20260709 12:22:18.632736 17652 leader_election.cc:304] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:18.614603 17728 leader_election.cc:304] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:18.633487 21595 raft_consensus.cc:515] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:22:18.634620 17804 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:18.634521 21493 consensus_queue.cc:1048] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:22:18.623003 21588 raft_consensus.cc:515] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:18.636461 21493 raft_consensus.cc:2804] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:22:18.637037 21493 raft_consensus.cc:697] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 6 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:22:18.638254 17728 leader_election.cc:304] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:18.638396 21588 leader_election.cc:290] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:18.615219 21470 raft_consensus.cc:740] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 8 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:22:18.570577 21599 raft_consensus.cc:3055] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:22:18.637951 21493 consensus_queue.cc:237] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:18.639688 17728 leader_election.cc:304] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:18.642695 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:18.555972 21499 raft_consensus.cc:515] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:22:18.700340 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 60737.1us
I20260709 12:22:18.701292 17728 leader_election.cc:304] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:18.639344 21480 raft_consensus.cc:2749] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:22:18.622201 21476 consensus_queue.cc:260] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:18.682183 17652 leader_election.cc:304] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:18.615219 21532 consensus_queue.cc:1059] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.077s
I20260709 12:22:18.661877 21599 raft_consensus.cc:740] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 10 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
W20260709 12:22:18.731153 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:18.734284 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 172454us
I20260709 12:22:18.678390 21588 raft_consensus.cc:493] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:18.700217 21398 raft_consensus.cc:2749] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260709 12:22:18.742591 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 148674us
W20260709 12:22:18.752656 17801 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:18.754467 17727 leader_election.cc:304] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
W20260709 12:22:18.754329 17801 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:18.758291 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:18.722133 21595 leader_election.cc:290] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:18.720697 21504 raft_consensus.cc:2804] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:22:18.763867 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:22:18.764416 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 158955us
I20260709 12:22:18.764669 17687 raft_consensus.cc:1813] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Handling vote request from an unknown peer e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:18.765460 17687 raft_consensus.cc:2393] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 7: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
W20260709 12:22:18.766639 18334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:18.767072 18334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:18.768800 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 184120us
I20260709 12:22:18.769313 17762 raft_consensus.cc:1240] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 8. Current term is 9. Ops: []
I20260709 12:22:18.699950 21477 consensus_queue.cc:1048] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.061s
I20260709 12:22:18.737975 21564 raft_consensus.cc:2749] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:22:18.770680 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:18.745117 21551 raft_consensus.cc:697] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 13 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:22:18.745395 21379 raft_consensus.cc:493] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:18.774045 20394 raft_consensus.cc:493] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:18.772657 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:22:18.785116 18334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:18.774560 20394 raft_consensus.cc:515] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:18.752300 21600 raft_consensus.cc:493] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:18.844904 21600 raft_consensus.cc:515] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:18.847499 21600 leader_election.cc:290] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:18.768364 21480 raft_consensus.cc:2804] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:22:18.771683 21476 raft_consensus.cc:3060] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:22:18.840210 17804 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:18.826108 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 163932us
I20260709 12:22:18.753067 21541 consensus_queue.cc:1048] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.069s
I20260709 12:22:18.759223 21370 consensus_queue.cc:1048] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 1, Time since last communication: 0.066s
I20260709 12:22:18.760458 21352 consensus_queue.cc:1048] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.074s
I20260709 12:22:18.773211 21588 raft_consensus.cc:515] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:18.899770 17728 leader_election.cc:304] T 7cb3768d215e409fb9f5baf18d124442 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:18.897547 20394 leader_election.cc:290] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:18.908950 20394 raft_consensus.cc:493] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:18.909885 20394 raft_consensus.cc:515] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:18.912946 20394 leader_election.cc:290] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:18.834846 21379 raft_consensus.cc:515] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:22:18.921309 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 166925us
I20260709 12:22:18.856186 21600 raft_consensus.cc:493] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:18.842989 21398 raft_consensus.cc:2749] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 12:22:18.848876 21337 raft_consensus.cc:493] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:18.895291 21593 raft_consensus.cc:493] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:18.898670 21228 raft_consensus.cc:2749] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20260709 12:22:18.933637 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 321476us
I20260709 12:22:18.895027 21480 raft_consensus.cc:697] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 8 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
W20260709 12:22:18.955171 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 98624.8us
I20260709 12:22:18.959595 21398 raft_consensus.cc:3060] T 7cb3768d215e409fb9f5baf18d124442 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:18.972669 17686 raft_consensus.cc:2410] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for term 9 because replica has last-logged OpId of term: 7 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 7.
I20260709 12:22:18.958877 21593 raft_consensus.cc:515] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:18.980504 21611 raft_consensus.cc:3055] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:22:18.981014 21611 raft_consensus.cc:740] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:22:18.982028 21611 consensus_queue.cc:260] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.9, Last appended by leader: 9, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:18.982822 21593 leader_election.cc:290] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:18.983467 21611 raft_consensus.cc:3060] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:18.984683 21593 raft_consensus.cc:493] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:18.984753 21614 raft_consensus.cc:493] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:18.985597 21608 log.cc:927] Time spent T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.222s	user 0.000s	sys 0.001s
I20260709 12:22:18.985723 21614 raft_consensus.cc:515] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:18.987720 18333 leader_election.cc:304] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:22:18.988133 21499 leader_election.cc:290] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:18.989055 21614 leader_election.cc:290] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:22:18.989322 21613 log.cc:927] Time spent T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.067s	user 0.001s	sys 0.000s
I20260709 12:22:18.989650 21614 raft_consensus.cc:493] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:18.990155 21614 raft_consensus.cc:515] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:22:18.991243 21609 log.cc:927] Time spent T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.154s	user 0.000s	sys 0.001s
I20260709 12:22:18.992436 21614 leader_election.cc:290] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:18.856031 21590 raft_consensus.cc:515] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:18.995188 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:18.996038 18365 raft_consensus.cc:2468] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 10.
W20260709 12:22:19.011643 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:18.994119 21582 log.cc:927] Time spent T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.062s	user 0.000s	sys 0.000s
W20260709 12:22:19.036638 17804 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:18.976500 21477 raft_consensus.cc:493] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:18.984951 21617 raft_consensus.cc:3055] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 6 LEADER]: Stepping down as leader of term 6
W20260709 12:22:19.079988 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 92435.4us
I20260709 12:22:18.977480 21551 consensus_queue.cc:237] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 12.12, Last appended by leader: 12, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:18.988201 21593 raft_consensus.cc:515] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:19.088156 21546 raft_consensus.cc:493] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:18.935379 21600 raft_consensus.cc:515] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:22:19.115525 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 389300us
I20260709 12:22:18.971261 21337 raft_consensus.cc:515] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:22:19.125633 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 166839us
W20260709 12:22:19.161851 17804 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:19.088937 21617 raft_consensus.cc:740] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
W20260709 12:22:19.180711 18334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:19.120640 21504 raft_consensus.cc:697] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:22:19.191051 17834 raft_consensus.cc:1275] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 7. (term mismatch)
I20260709 12:22:18.961670 21599 consensus_queue.cc:260] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.5, Last appended by leader: 5, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:19.195689 17834 pending_rounds.cc:85] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b: Aborting all ops after (but not including) 6
I20260709 12:22:19.214855 17834 pending_rounds.cc:107] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b: Aborting uncommitted NO_OP operation due to leader change: 5.7
I20260709 12:22:19.215231 17834 raft_consensus.cc:2889] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:19.214666 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:19.218168 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:19.218291 21480 consensus_queue.cc:237] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 7.9, Last appended by leader: 9, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:22:19.222342 21308 consensus_meta.cc:287] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.635s	user 0.010s	sys 0.000s
I20260709 12:22:19.222797 21308 raft_consensus.cc:515] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:19.225126 18333 leader_election.cc:304] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:22:19.231660 21616 consensus_queue.cc:1048] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:22:19.232795 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:19.237867 17687 raft_consensus.cc:1240] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 8. Current term is 9. Ops: []
I20260709 12:22:19.118791 21477 raft_consensus.cc:515] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:19.289458 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:19.286871 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:19.204252 21561 raft_consensus.cc:493] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:19.298062 17685 raft_consensus.cc:1240] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 9. Current term is 10. Ops: []
I20260709 12:22:19.303108 21650 raft_consensus.cc:493] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:19.216838 21636 raft_consensus.cc:493] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:19.312083 18333 leader_election.cc:304] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:22:19.126914 21546 raft_consensus.cc:515] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:19.329481 21644 raft_consensus.cc:2804] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:22:19.330098 21644 raft_consensus.cc:493] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:19.330538 21644 raft_consensus.cc:3060] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:19.233805 21370 consensus_queue.cc:1059] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.079s
I20260709 12:22:19.332419 21628 raft_consensus.cc:3055] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:22:19.332687 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:19.333106 21628 raft_consensus.cc:740] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:22:19.333405 17757 raft_consensus.cc:2468] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 7.
I20260709 12:22:19.334111 21628 consensus_queue.cc:260] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:22:19.335012 21053 consensus_meta.cc:287] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.706s	user 0.007s	sys 0.003s
I20260709 12:22:19.335295 21628 raft_consensus.cc:3060] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:19.332340 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:19.283002 21588 leader_election.cc:290] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:19.343338 17685 raft_consensus.cc:1275] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
W20260709 12:22:19.308859 21433 consensus_meta.cc:287] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.693s	user 0.010s	sys 0.004s
I20260709 12:22:19.348726 17801 leader_election.cc:304] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:19.373039 17688 raft_consensus.cc:2468] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 7.
I20260709 12:22:19.326745 21534 raft_consensus.cc:2764] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Leader pre-election decision vote started in defunct term 10: won
I20260709 12:22:19.351648 17760 raft_consensus.cc:1275] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20260709 12:22:19.344038 21379 leader_election.cc:290] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:19.332401 21370 raft_consensus.cc:3055] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 6 LEADER]: Stepping down as leader of term 6
W20260709 12:22:19.352160 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 65537.9us
W20260709 12:22:19.389732 17803 leader_election.cc:336] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Timed out: RequestConsensusVote RPC to 127.16.29.68:42577 timed out after 2.227s (SENT)
I20260709 12:22:19.392230 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:19.351316 21635 raft_consensus.cc:493] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:19.387562 21370 raft_consensus.cc:740] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 6 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:22:19.320757 21401 consensus_queue.cc:260] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.7, Last appended by leader: 7, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:19.409309 18366 raft_consensus.cc:1240] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 7. Current term is 8. Ops: []
W20260709 12:22:19.410604 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 105420us
I20260709 12:22:19.427517 21645 raft_consensus.cc:493] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:19.428309 21654 raft_consensus.cc:2804] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:22:19.429014 21654 raft_consensus.cc:493] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:19.428404 21645 raft_consensus.cc:515] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:19.429476 21654 raft_consensus.cc:3060] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:19.431411 21645 leader_election.cc:290] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:19.415273 21370 consensus_queue.cc:260] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:19.433157 21595 raft_consensus.cc:493] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:19.433720 21595 raft_consensus.cc:515] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:19.372141 21561 raft_consensus.cc:515] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:19.435689 21595 leader_election.cc:290] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:19.436173 21561 leader_election.cc:290] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:19.412889 21477 leader_election.cc:290] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:19.433910 21370 raft_consensus.cc:3060] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:22:19.439503 21632 log.cc:927] Time spent T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.052s	user 0.000s	sys 0.000s
I20260709 12:22:19.441164 21309 raft_consensus.cc:3055] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:22:19.400115 21649 raft_consensus.cc:2804] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:22:19.411914 17727 leader_election.cc:400] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 9 pre-election: Vote denied by peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837) with higher term. Message: Invalid argument: T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for earlier term 9. Current term is 10.
W20260709 12:22:19.409237 21619 log.cc:927] Time spent T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.371s	user 0.000s	sys 0.001s
I20260709 12:22:19.447899 21309 raft_consensus.cc:740] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:22:19.453704 21401 raft_consensus.cc:3060] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:19.411801 21337 leader_election.cc:290] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:22:19.414384 21638 log.cc:927] Time spent T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.102s	user 0.000s	sys 0.000s
W20260709 12:22:19.411227 21625 log.cc:927] Time spent T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.230s	user 0.000s	sys 0.001s
W20260709 12:22:19.466945 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 83999.8us
I20260709 12:22:19.342761 21650 raft_consensus.cc:515] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:19.350309 21636 raft_consensus.cc:515] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:19.349128 21504 consensus_queue.cc:237] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.7, Last appended by leader: 7, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:19.411304 21593 leader_election.cc:290] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:19.490872 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:19.491852 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:19.510521 17689 raft_consensus.cc:3060] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Advancing to term 11
I20260709 12:22:19.500319 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:19.460525 21649 raft_consensus.cc:493] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:19.506956 21309 consensus_queue.cc:260] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 7.9, Last appended by leader: 9, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:19.511386 17685 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 9.
I20260709 12:22:19.509783 17684 raft_consensus.cc:2468] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 5.
I20260709 12:22:19.380045 21617 consensus_queue.cc:260] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:19.413482 21538 consensus_queue.cc:1048] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.088s
I20260709 12:22:19.497080 21600 leader_election.cc:290] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:19.554425 18333 leader_election.cc:400] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 6 election: Vote denied by peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837) with higher term. Message: Invalid argument: T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 6. Current term is 7.
I20260709 12:22:19.511984 21308 leader_election.cc:290] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 11 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:19.408561 21635 raft_consensus.cc:515] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:19.483230 21312 consensus_queue.cc:1059] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.051s
I20260709 12:22:19.517134 21649 raft_consensus.cc:3060] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:19.561455 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:19.558983 21645 raft_consensus.cc:493] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:19.565358 21645 raft_consensus.cc:515] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:19.567559 21645 leader_election.cc:290] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:19.568218 21590 leader_election.cc:290] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 8 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:19.553740 21309 raft_consensus.cc:3060] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:19.574218 17654 leader_election.cc:304] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
W20260709 12:22:19.577126 17804 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:19.577847 17804 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:19.585568 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 173801us
W20260709 12:22:19.580432 21560 consensus_meta.cc:287] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.891s	user 0.013s	sys 0.000s
I20260709 12:22:19.596297 21308 raft_consensus.cc:493] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:19.628111 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:19.631408 21308 raft_consensus.cc:515] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:19.632248 21635 leader_election.cc:290] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:19.633796 17764 raft_consensus.cc:1240] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 LEADER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 10. Current term is 11. Ops: []
I20260709 12:22:19.633944 21308 leader_election.cc:290] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:19.634128 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:19.634981 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:19.627594 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 10 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:19.635663 17757 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 9.
I20260709 12:22:19.599025 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:19.600250 17833 raft_consensus.cc:1275] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 5 index: 8. Preceding OpId from leader: term: 7 index: 9. (index mismatch)
I20260709 12:22:19.600312 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 10 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:19.601040 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:19.638257 17836 raft_consensus.cc:2393] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 8: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:22:19.601250 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:19.601960 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:19.639110 17831 raft_consensus.cc:2468] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 6.
I20260709 12:22:19.639338 17832 raft_consensus.cc:2410] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for term 8 because replica has last-logged OpId of term: 7 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 6.
I20260709 12:22:19.602191 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:19.639962 17829 raft_consensus.cc:3060] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:19.630786 17727 leader_election.cc:304] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
W20260709 12:22:19.640278 21610 log.cc:927] Time spent T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.725s	user 0.000s	sys 0.004s
I20260709 12:22:19.634946 21560 raft_consensus.cc:515] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:22:19.641636 21611 consensus_meta.cc:287] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.643s	user 0.009s	sys 0.000s
I20260709 12:22:19.602831 21546 leader_election.cc:290] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:19.603520 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
W20260709 12:22:19.642427 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 90039.8us
I20260709 12:22:19.642933 17837 raft_consensus.cc:3060] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:19.643821 18333 leader_election.cc:400] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 6 pre-election: Vote denied by peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837) with higher term. Message: Invalid argument: T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 6. Current term is 7.
I20260709 12:22:19.644245 18333 leader_election.cc:403] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 6 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:22:19.645126 18333 leader_election.cc:304] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:19.646489 18363 raft_consensus.cc:1275] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
I20260709 12:22:19.650350 21599 raft_consensus.cc:3060] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:19.651501 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:19.652207 17756 raft_consensus.cc:3060] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:19.656818 17829 raft_consensus.cc:2468] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 7.
I20260709 12:22:19.628960 18374 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59886 (request call id 1117) took 1027 ms. Trace:
I20260709 12:22:19.605962 17689 raft_consensus.cc:2468] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 11.
I20260709 12:22:19.593952 21501 raft_consensus.cc:2804] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Leader pre-election won for term 11
W20260709 12:22:19.660902 17801 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:19.613283 21636 leader_election.cc:290] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:19.615077 17802 leader_election.cc:304] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:22:19.582420 21600 raft_consensus.cc:493] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:19.617506 21413 consensus_meta.cc:287] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.990s	user 0.008s	sys 0.004s
W20260709 12:22:19.625100 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:19.670349 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:19.627303 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:19.671030 17831 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 9.
I20260709 12:22:19.642204 21532 raft_consensus.cc:2749] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:22:19.660693 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:19.676239 17832 raft_consensus.cc:2368] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for earlier term 10. Current term is 11.
I20260709 12:22:19.657598 18374 rpcz_store.cc:276] 0709 12:22:18.601741 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:18.622095 (+ 20354us) service_pool.cc:224] Handling call
0709 12:22:18.622453 (+   358us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:18.622796 (+   343us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:18.622828 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:18.678249 (+ 55421us) log.cc:844] Serialized 26 byte log entry
0709 12:22:18.855659 (+177410us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:22:18.855696 (+    37us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:18.855747 (+    51us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:19.628755 (+773008us) raft_consensus.cc:1685] finished
0709 12:22:19.628775 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:19.628902 (+   127us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":262058,"raft.run_cpu_time_us":219,"raft.run_wall_time_us":8827,"thread_start_us":108879,"threads_started":1,"wal-append.queue_time_us":355010}
I20260709 12:22:19.666270 21541 raft_consensus.cc:2749] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
W20260709 12:22:19.665114 17803 leader_election.cc:336] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Timed out: RequestConsensusVote RPC to 127.16.29.68:42577 timed out after 1.618s (SENT)
W20260709 12:22:19.667086 18334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:19.701961 18334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:19.671336 17760 raft_consensus.cc:2468] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 9.
I20260709 12:22:19.610828 21650 leader_election.cc:290] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:19.678983 17758 raft_consensus.cc:1240] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 9. Current term is 10. Ops: []
W20260709 12:22:19.679847 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 64855.3us
I20260709 12:22:19.660408 21611 raft_consensus.cc:2749] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260709 12:22:19.681016 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:19.661966 17837 raft_consensus.cc:2468] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 8.
I20260709 12:22:19.689275 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:19.710121 17757 raft_consensus.cc:2468] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 6.
I20260709 12:22:19.710615 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 10 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:19.712206 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:22:19.712565 21656 log.cc:927] Time spent T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.052s	user 0.000s	sys 0.000s
I20260709 12:22:19.712687 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 10 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:19.669960 21600 raft_consensus.cc:515] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:19.714727 21433 raft_consensus.cc:2804] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:22:19.666911 21185 consensus_queue.cc:1059] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.048s
I20260709 12:22:19.715190 21433 raft_consensus.cc:493] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:19.707111 21564 consensus_queue.cc:1048] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.009s
I20260709 12:22:19.715775 21433 raft_consensus.cc:3060] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:19.689882 21617 raft_consensus.cc:3060] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:19.717063 17834 raft_consensus.cc:3060] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:19.699399 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:19.690555 17762 raft_consensus.cc:1275] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20260709 12:22:19.701435 17803 leader_election.cc:304] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:19.722405 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:19.722923 21312 raft_consensus.cc:2749] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:22:19.691009 21156 raft_consensus.cc:493] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:19.662750 21501 raft_consensus.cc:493] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:19.723554 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:19.723800 21501 raft_consensus.cc:3060] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:19.676009 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 12 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:19.725750 21156 raft_consensus.cc:515] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:19.693512 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 10 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:19.643599 21560 leader_election.cc:290] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:19.660986 21616 raft_consensus.cc:2749] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: Vote denied by peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837) with higher term. Message: Invalid argument: T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 6. Current term is 7.
I20260709 12:22:19.707676 17836 raft_consensus.cc:1275] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260709 12:22:19.736934 17687 raft_consensus.cc:1240] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 7. Current term is 8. Ops: []
I20260709 12:22:19.722486 21308 raft_consensus.cc:493] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:19.723205 21660 raft_consensus.cc:493] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:19.706327 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 77880.5us
I20260709 12:22:19.723199 17762 raft_consensus.cc:2393] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 6: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:22:19.728967 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 8 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:19.729673 17685 raft_consensus.cc:1240] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 8 LEADER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 7. Current term is 8. Ops: []
W20260709 12:22:19.735424 21398 consensus_meta.cc:287] T 7cb3768d215e409fb9f5baf18d124442 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.757s	user 0.009s	sys 0.003s
I20260709 12:22:19.730234 17681 raft_consensus.cc:1240] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 6. Current term is 7. Ops: []
W20260709 12:22:19.735605 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 90574.5us
W20260709 12:22:19.735566 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 70479us
I20260709 12:22:19.765803 17756 raft_consensus.cc:2468] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 11.
I20260709 12:22:19.737534 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 9 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:22:19.746007 17801 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:19.766765 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 14 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:19.748274 17838 raft_consensus.cc:2410] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for term 10 because replica has last-logged OpId of term: 7 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 3.
W20260709 12:22:19.753618 17804 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:19.754384 17802 leader_election.cc:304] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:22:19.729564 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:19.767568 17759 raft_consensus.cc:2410] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for term 14 because replica has last-logged OpId of term: 12 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 11.
I20260709 12:22:19.760037 17684 raft_consensus.cc:1275] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
I20260709 12:22:19.679566 21471 raft_consensus.cc:493] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:22:19.764137 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:19.774307 17835 raft_consensus.cc:2468] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 7.
I20260709 12:22:19.715200 21600 leader_election.cc:290] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:19.778546 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:19.779662 17756 raft_consensus.cc:2368] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for earlier term 10. Current term is 11.
I20260709 12:22:19.780710 17762 raft_consensus.cc:1240] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 6. Current term is 7. Ops: []
I20260709 12:22:19.783110 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:19.786584 17687 raft_consensus.cc:2468] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 9.
I20260709 12:22:19.792332 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:19.793174 17761 raft_consensus.cc:2468] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 10.
W20260709 12:22:19.794329 21476 consensus_meta.cc:287] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.936s	user 0.007s	sys 0.014s
I20260709 12:22:19.794760 17760 raft_consensus.cc:1275] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260709 12:22:19.797256 21156 leader_election.cc:290] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:19.756513 21312 raft_consensus.cc:493] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:19.797823 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 43499.9us
I20260709 12:22:19.798382 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 10 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:19.798261 21312 raft_consensus.cc:515] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:19.799726 21600 raft_consensus.cc:2804] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:22:19.800114 17758 raft_consensus.cc:1275] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20260709 12:22:19.800464 21600 raft_consensus.cc:697] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 11 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:22:19.800729 21312 leader_election.cc:290] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:19.801847 21600 consensus_queue.cc:237] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.8, Last appended by leader: 8, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:19.678099 21413 raft_consensus.cc:515] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:19.797698 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:19.812606 17802 leader_election.cc:304] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:22:19.814083 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:19.808257 17764 raft_consensus.cc:2468] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 7.
I20260709 12:22:19.817245 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:19.818133 17762 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 9.
W20260709 12:22:19.823436 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 60718.8us
I20260709 12:22:19.824548 17727 leader_election.cc:304] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:22:19.826176 17727 leader_election.cc:304] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:22:19.827229 21337 raft_consensus.cc:2749] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:22:19.828059 21564 raft_consensus.cc:2804] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Leader election won for term 7
W20260709 12:22:19.828562 21655 log.cc:927] Time spent T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.072s	user 0.000s	sys 0.000s
I20260709 12:22:19.830024 17682 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37396 (request call id 2109) took 1094 ms. Trace:
I20260709 12:22:19.830431 17682 rpcz_store.cc:276] 0709 12:22:18.735653 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:18.736090 (+   437us) service_pool.cc:224] Handling call
0709 12:22:18.757034 (+ 20944us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:18.953475 (+196441us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:18.972117 (+ 18642us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:19.330960 (+358843us) spinlock_profiling.cc:243] Waited 9.19 ms on lock 0x7b2c00000c88. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b48f19a11 00007f7b48f29372 00007f7b4a65ba01 00007f7b4a62043a 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2
0709 12:22:19.331467 (+   507us) log.cc:844] Serialized 26 byte log entry
0709 12:22:19.405267 (+ 73800us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:22:19.405317 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:19.405399 (+    82us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:19.829690 (+424291us) raft_consensus.cc:1685] finished
0709 12:22:19.829711 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:19.829941 (+   230us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":56097,"raft.queue_time_us":118966,"raft.run_cpu_time_us":80,"raft.run_wall_time_us":79,"spinlock_wait_cycles":21138816,"thread_start_us":154529,"threads_started":2,"wal-append.queue_time_us":259690}
I20260709 12:22:19.832350 21060 consensus_queue.cc:1048] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 12:22:19.833926 21060 consensus_queue.cc:1048] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 12:22:19.836457 21546 raft_consensus.cc:493] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:19.837100 21401 raft_consensus.cc:2804] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:22:19.836967 21546 raft_consensus.cc:515] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:19.837481 21401 raft_consensus.cc:493] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:19.837834 21401 raft_consensus.cc:3060] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:19.839035 21546 leader_election.cc:290] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:19.773751 21398 raft_consensus.cc:2749] T 7cb3768d215e409fb9f5baf18d124442 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:22:19.828606 21564 raft_consensus.cc:697] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 7 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:22:19.845721 21564 consensus_queue.cc:237] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:22:19.846487 21644 consensus_meta.cc:287] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.515s	user 0.010s	sys 0.000s
I20260709 12:22:19.848929 18374 raft_consensus.cc:1275] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 7 index: 9. Preceding OpId from leader: term: 8 index: 10. (index mismatch)
I20260709 12:22:19.852588 21476 consensus_queue.cc:1048] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:22:19.763821 21660 raft_consensus.cc:515] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:19.857168 17834 raft_consensus.cc:2468] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 10.
I20260709 12:22:19.819293 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:19.862700 17758 raft_consensus.cc:2468] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 8.
I20260709 12:22:19.847083 21644 raft_consensus.cc:515] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:19.835490 21060 consensus_queue.cc:1048] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 12:22:19.893081 17830 raft_consensus.cc:1240] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 10. Current term is 11. Ops: []
I20260709 12:22:19.761066 21308 raft_consensus.cc:515] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:19.898759 18333 leader_election.cc:304] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:19.908344 17833 raft_consensus.cc:1275] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 8 index: 7. (term mismatch)
I20260709 12:22:19.908579 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:19.909197 17833 pending_rounds.cc:85] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b: Aborting all ops after (but not including) 6
I20260709 12:22:19.910086 17833 pending_rounds.cc:107] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b: Aborting uncommitted NO_OP operation due to leader change: 7.7
I20260709 12:22:19.910513 17833 raft_consensus.cc:2889] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:19.913004 18362 raft_consensus.cc:1240] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 8. Current term is 9. Ops: []
W20260709 12:22:19.918365 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 57761.4us
I20260709 12:22:19.811499 21471 raft_consensus.cc:515] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:19.903105 21662 raft_consensus.cc:493] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:19.944903 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:19.946174 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:19.910995 21636 raft_consensus.cc:2749] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:22:19.948616 21241 raft_consensus.cc:3055] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:22:19.949343 21413 leader_election.cc:290] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 11 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:19.949827 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:22:19.928231 21628 consensus_meta.cc:287] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.590s	user 0.011s	sys 0.003s
I20260709 12:22:19.892976 21433 raft_consensus.cc:515] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:19.955659 17755 raft_consensus.cc:2410] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for term 10 because replica has last-logged OpId of term: 9 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 6.
I20260709 12:22:19.924367 21616 raft_consensus.cc:493] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:19.960615 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 148603us
I20260709 12:22:19.943511 21644 leader_election.cc:290] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 9 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:19.961261 18371 raft_consensus.cc:2468] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 8.
I20260709 12:22:19.972543 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:19.973533 17762 raft_consensus.cc:2468] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 7.
W20260709 12:22:19.976078 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 310023us
W20260709 12:22:19.978737 17804 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:19.969543 17801 leader_election.cc:304] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:19.985404 17801 leader_election.cc:304] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:22:19.953905 21241 raft_consensus.cc:740] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:22:19.985436 17727 leader_election.cc:304] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
W20260709 12:22:20.001070 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 333728us
I20260709 12:22:20.004745 17728 leader_election.cc:304] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:19.993069 21413 raft_consensus.cc:2804] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:22:20.006224 21413 raft_consensus.cc:493] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:20.006629 21413 raft_consensus.cc:3060] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:20.006896 21241 consensus_queue.cc:260] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:20.019443 17728 leader_election.cc:304] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:19.952605 21662 raft_consensus.cc:515] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:20.032416 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:20.035854 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 9 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:22:20.017724 21654 consensus_meta.cc:287] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.588s	user 0.013s	sys 0.000s
I20260709 12:22:20.043334 17801 leader_election.cc:304] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:20.045248 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:20.033123 17688 raft_consensus.cc:2468] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 9.
I20260709 12:22:19.991931 21407 raft_consensus.cc:2804] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:22:20.048616 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:20.049388 17687 raft_consensus.cc:1240] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 8. Current term is 9. Ops: []
I20260709 12:22:20.049815 17762 raft_consensus.cc:3060] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:20.052246 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:20.053552 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:20.054236 17684 raft_consensus.cc:2393] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 10: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:22:20.054347 17764 raft_consensus.cc:3060] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:20.055449 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:19.971752 21616 raft_consensus.cc:515] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:20.030761 21624 raft_consensus.cc:493] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:20.030646 21541 raft_consensus.cc:2749] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:22:20.032269 21640 raft_consensus.cc:2804] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:22:19.992338 21505 consensus_queue.cc:1048] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 7, Time since last communication: 0.058s
I20260709 12:22:20.066740 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:20.071727 17760 raft_consensus.cc:2468] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 9.
I20260709 12:22:20.060710 21654 raft_consensus.cc:515] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:20.080477 21654 leader_election.cc:290] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 8 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:20.081820 21401 raft_consensus.cc:515] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:20.052053 21407 raft_consensus.cc:493] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:20.084733 21407 raft_consensus.cc:3060] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:20.084889 17802 leader_election.cc:304] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:22:20.087001 17831 raft_consensus.cc:2393] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 8: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:22:20.088466 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:20.089147 21667 raft_consensus.cc:2749] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:22:20.089632 21615 raft_consensus.cc:2804] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:22:20.090056 21615 raft_consensus.cc:493] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:20.090396 21615 raft_consensus.cc:3060] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:20.090550 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:20.091136 17688 raft_consensus.cc:3060] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 5 FOLLOWER]: Advancing to term 7
I20260709 12:22:20.093146 21654 consensus_queue.cc:1059] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 12:22:20.093779 21501 raft_consensus.cc:515] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:20.059149 21594 raft_consensus.cc:2749] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260709 12:22:20.098944 17803 leader_election.cc:336] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Timed out: RequestConsensusVote RPC to 127.16.29.68:42577 timed out after 1.819s (SENT)
I20260709 12:22:20.101470 17758 raft_consensus.cc:2468] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 7.
I20260709 12:22:20.107649 17803 leader_election.cc:304] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:20.104143 17762 raft_consensus.cc:2468] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 11.
I20260709 12:22:20.095845 21501 leader_election.cc:290] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 11 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:20.063630 21628 raft_consensus.cc:493] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:20.061892 21645 raft_consensus.cc:493] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:20.108374 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 275675us
I20260709 12:22:20.132712 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:20.050920 21337 consensus_queue.cc:1048] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.052s
W20260709 12:22:20.115083 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 52874.3us
I20260709 12:22:20.097499 21053 raft_consensus.cc:493] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:20.148903 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:20.134053 17652 leader_election.cc:304] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:22:20.059495 21649 raft_consensus.cc:515] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:20.154552 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:20.155365 17760 raft_consensus.cc:3060] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:20.101914 21617 raft_consensus.cc:493] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:22:20.161163 17763 raft_consensus.cc:1240] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 LEADER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 9. Current term is 10. Ops: []
I20260709 12:22:20.103286 21308 leader_election.cc:290] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:20.163798 17652 leader_election.cc:304] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da
W20260709 12:22:20.179992 17801 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:20.179894 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 80183.6us
I20260709 12:22:20.190759 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:20.191814 17835 raft_consensus.cc:2468] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 8.
I20260709 12:22:20.197140 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:19.822377 16501 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
W20260709 12:22:20.177817 21632 log.cc:927] Time spent T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.050s	user 0.000s	sys 0.000s
I20260709 12:22:20.074043 21624 raft_consensus.cc:515] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:20.156558 21654 raft_consensus.cc:2749] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 12:22:20.227142 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:20.201939 18363 raft_consensus.cc:1240] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 6. Current term is 7. Ops: []
I20260709 12:22:20.228691 17835 raft_consensus.cc:2410] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for term 7 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260709 12:22:20.198223 17838 raft_consensus.cc:2376] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 7. Re-sending same reply.
W20260709 12:22:20.210253 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 111329us
I20260709 12:22:20.239732 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:20.244820 18362 raft_consensus.cc:3060] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:22:20.235288 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 71597.5us
I20260709 12:22:20.246450 17652 leader_election.cc:400] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 10 pre-election: Vote denied by peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837) with higher term. Message: Invalid argument: T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for earlier term 10. Current term is 11.
I20260709 12:22:20.246838 17652 leader_election.cc:403] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 10 pre-election: Cancelling election due to peer responding with higher term
W20260709 12:22:20.246559 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 227286us
I20260709 12:22:20.248605 17652 leader_election.cc:304] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da
W20260709 12:22:20.241747 17803 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.16.29.68:42577 timed out after 2.123s (SENT) [suppressed 23 similar messages]
W20260709 12:22:20.258527 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:20.149865 21628 raft_consensus.cc:515] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:20.213605 21433 leader_election.cc:290] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 7 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:20.163054 21053 raft_consensus.cc:515] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:20.157095 21645 raft_consensus.cc:515] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:20.277580 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:22:20.250746 21651 log.cc:927] Time spent T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.054s	user 0.000s	sys 0.000s
I20260709 12:22:20.278328 17835 raft_consensus.cc:2368] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for earlier term 6. Current term is 7.
I20260709 12:22:20.271481 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:20.275622 18364 raft_consensus.cc:1240] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 7. Current term is 8. Ops: []
I20260709 12:22:20.279415 18366 raft_consensus.cc:1275] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260709 12:22:20.276201 18333 leader_election.cc:400] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 6 election: Vote denied by peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651) with higher term. Message: Invalid argument: T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 6. Current term is 7.
I20260709 12:22:20.280447 18333 leader_election.cc:403] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 6 election: Cancelling election due to peer responding with higher term
I20260709 12:22:20.273326 21640 raft_consensus.cc:697] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:22:20.255553 21588 raft_consensus.cc:2749] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Leader pre-election lost for term 10. Reason: Vote denied by peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837) with higher term. Message: Invalid argument: T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for earlier term 10. Current term is 11.
I20260709 12:22:20.190935 21617 raft_consensus.cc:515] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:20.300659 17764 raft_consensus.cc:2468] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 10.
I20260709 12:22:20.301047 17688 raft_consensus.cc:2468] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 7.
I20260709 12:22:20.269413 21401 leader_election.cc:290] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:22:20.298859 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 50315.2us
I20260709 12:22:20.276820 21662 leader_election.cc:290] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:20.277489 21589 raft_consensus.cc:493] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:20.250499 21676 raft_consensus.cc:2804] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:22:20.307123 17652 leader_election.cc:304] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:22:20.308602 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:20.278513 21471 leader_election.cc:290] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:20.257877 21564 consensus_queue.cc:1059] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.047s
I20260709 12:22:20.342658 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:20.343317 17835 raft_consensus.cc:3060] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:20.343498 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:20.344116 17830 raft_consensus.cc:1813] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Handling vote request from an unknown peer e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:20.302138 21501 raft_consensus.cc:2764] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
I20260709 12:22:20.340293 17802 leader_election.cc:304] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:22:20.357009 17830 raft_consensus.cc:2393] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 7: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:22:20.329907 21493 raft_consensus.cc:2804] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:22:20.329229 21471 consensus_queue.cc:1048] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.018s
I20260709 12:22:20.369822 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:22:20.380417 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 73622.6us
W20260709 12:22:20.390458 17804 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:20.392872 17830 raft_consensus.cc:1275] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 10 index: 5. Preceding OpId from leader: term: 11 index: 6. (index mismatch)
I20260709 12:22:20.394749 17830 raft_consensus.cc:1275] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 7 index: 3. Preceding OpId from leader: term: 8 index: 4. (index mismatch)
W20260709 12:22:20.413254 17804 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:20.342774 21616 leader_election.cc:290] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:20.425750 17652 leader_election.cc:304] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:22:20.417601 21546 raft_consensus.cc:493] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:20.430425 21546 raft_consensus.cc:515] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:20.433285 21546 leader_election.cc:290] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:20.436648 17836 raft_consensus.cc:1275] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 7. (term mismatch)
I20260709 12:22:20.437131 17836 pending_rounds.cc:85] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b: Aborting all ops after (but not including) 6
I20260709 12:22:20.437417 17836 pending_rounds.cc:107] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b: Aborting uncommitted NO_OP operation due to leader change: 6.7
I20260709 12:22:20.437698 17836 raft_consensus.cc:2889] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:20.376165 21308 raft_consensus.cc:2804] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:22:20.441548 17830 raft_consensus.cc:1240] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 9. Current term is 10. Ops: []
I20260709 12:22:20.448788 17760 raft_consensus.cc:2468] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 8.
W20260709 12:22:20.451504 21675 log.cc:927] Time spent T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.147s	user 0.001s	sys 0.000s
I20260709 12:22:20.371368 17586 catalog_manager.cc:5697] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b reported cstate change: term changed from 6 to 9, leader changed from 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) to e063541542a94969898868bf6eb5ce1b (127.16.29.67). New cstate: current_term: 9 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: HEALTHY } } }
I20260709 12:22:20.417735 21401 raft_consensus.cc:493] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:20.439811 21673 log.cc:927] Time spent T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.137s	user 0.001s	sys 0.001s
I20260709 12:22:20.440487 21674 raft_consensus.cc:2804] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Leader pre-election won for term 10
W20260709 12:22:20.483740 17801 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:20.500118 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 207512us
I20260709 12:22:20.435479 21493 raft_consensus.cc:697] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 8 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:22:20.462199 21504 raft_consensus.cc:3055] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:22:20.461479 21546 raft_consensus.cc:2804] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:22:20.461793 21308 raft_consensus.cc:697] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 7 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:22:20.512989 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:20.517745 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:20.512055 21649 leader_election.cc:290] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 7 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:20.459810 21379 consensus_queue.cc:1048] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.057s
I20260709 12:22:20.534893 17835 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 7.
I20260709 12:22:20.541888 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 14 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:20.503840 21674 raft_consensus.cc:493] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:20.538866 21628 leader_election.cc:290] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:22:20.540124 21680 log.cc:927] Time spent T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.096s	user 0.001s	sys 0.000s
W20260709 12:22:20.540527 21669 log.cc:927] Time spent T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.318s	user 0.000s	sys 0.001s
I20260709 12:22:20.541035 21624 leader_election.cc:290] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:20.511544 21662 raft_consensus.cc:493] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:20.543932 17833 raft_consensus.cc:2468] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 10.
I20260709 12:22:20.546412 17686 raft_consensus.cc:2468] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 13.
I20260709 12:22:20.515560 21156 raft_consensus.cc:493] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:20.520051 21504 raft_consensus.cc:740] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:22:20.519233 21546 raft_consensus.cc:493] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:20.550228 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 14 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:22:20.551924 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:20.500536 21401 raft_consensus.cc:515] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:20.329711 21589 raft_consensus.cc:515] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:20.549669 21674 raft_consensus.cc:3060] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:22:20.580493 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 141958us
I20260709 12:22:20.436164 21640 consensus_queue.cc:237] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.7, Last appended by leader: 7, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:20.501762 21470 consensus_queue.cc:260] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:20.575090 21617 leader_election.cc:290] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:20.578274 21546 raft_consensus.cc:3060] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:20.596115 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:20.596969 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:20.598320 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:20.574312 21156 raft_consensus.cc:515] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:20.601521 17761 raft_consensus.cc:1240] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 10. Current term is 11. Ops: [10.7-10.7]
W20260709 12:22:20.587571 21671 log.cc:927] Time spent T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.285s	user 0.001s	sys 0.000s
I20260709 12:22:20.606251 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:20.606916 17756 raft_consensus.cc:3060] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:20.607797 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:20.608460 17762 raft_consensus.cc:2468] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 9.
I20260709 12:22:20.561887 21662 raft_consensus.cc:515] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:20.591382 21645 leader_election.cc:290] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:20.616954 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:20.617619 17764 raft_consensus.cc:3060] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:20.619212 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:20.605363 21379 raft_consensus.cc:493] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:20.622954 18362 raft_consensus.cc:2468] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 11.
I20260709 12:22:20.600232 21493 consensus_queue.cc:237] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:20.624838 21685 raft_consensus.cc:493] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:20.637562 21156 leader_election.cc:290] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:20.638715 21401 leader_election.cc:290] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:20.627478 21185 consensus_queue.cc:1059] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.012s
I20260709 12:22:20.583020 17686 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95"
dest_uuid: "0da169ff82674dc1947055ce43de68a3"
mode: GRACEFUL
new_leader_uuid: "e063541542a94969898868bf6eb5ce1b"
 from {username='slave'} at 127.0.0.1:54540
W20260709 12:22:20.643342 21688 log.cc:927] Time spent T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.127s	user 0.000s	sys 0.000s
I20260709 12:22:20.675125 17686 raft_consensus.cc:606] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 7 LEADER]: Received request to transfer leadership to TS e063541542a94969898868bf6eb5ce1b
W20260709 12:22:20.647073 21407 consensus_meta.cc:287] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.562s	user 0.010s	sys 0.000s
I20260709 12:22:20.648916 21241 raft_consensus.cc:3060] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:20.613615 21700 raft_consensus.cc:493] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:20.658174 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:20.634346 17652 leader_election.cc:304] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:22:20.629200 21679 log.cc:927] Time spent T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.185s	user 0.000s	sys 0.001s
I20260709 12:22:20.668913 17756 raft_consensus.cc:2468] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 7.
W20260709 12:22:20.633100 21681 log.cc:927] Time spent T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.086s	user 0.000s	sys 0.000s
I20260709 12:22:20.629901 21633 raft_consensus.cc:493] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:20.677917 21630 consensus_queue.cc:1059] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 9.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.005s
I20260709 12:22:20.683286 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:20.712170 17829 raft_consensus.cc:2393] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 11: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:22:20.683631 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:20.607223 21308 consensus_queue.cc:237] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:20.690158 21156 raft_consensus.cc:3055] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:22:20.689172 21407 raft_consensus.cc:515] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:22:20.697894 21615 consensus_meta.cc:287] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.607s	user 0.011s	sys 0.000s
I20260709 12:22:20.651714 21379 raft_consensus.cc:515] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:22:20.706811 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 81727.5us
I20260709 12:22:20.707512 21697 raft_consensus.cc:2749] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:22:20.612532 21504 consensus_queue.cc:260] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:22:20.720723 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:20.629721 21699 consensus_queue.cc:1048] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.088s
I20260709 12:22:20.727414 17829 raft_consensus.cc:1275] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260709 12:22:20.716723 21156 raft_consensus.cc:740] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:22:20.723467 17652 leader_election.cc:304] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:20.732728 17764 raft_consensus.cc:2468] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 11.
I20260709 12:22:20.732491 17728 leader_election.cc:304] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:20.740968 17652 leader_election.cc:304] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:20.745208 17835 raft_consensus.cc:1275] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 8 index: 10. (index mismatch)
I20260709 12:22:20.746237 17829 raft_consensus.cc:1275] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 10 index: 11. (index mismatch)
I20260709 12:22:20.752099 21156 consensus_queue.cc:260] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.7, Last appended by leader: 7, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:20.757160 21156 raft_consensus.cc:3060] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:20.761904 21379 leader_election.cc:290] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:20.765050 17652 leader_election.cc:400] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 10 pre-election: Vote denied by peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651) with higher term. Message: Invalid argument: T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for earlier term 10. Current term is 11.
W20260709 12:22:20.767097 17804 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:20.680343 21685 raft_consensus.cc:515] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:20.733994 21676 raft_consensus.cc:697] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 7 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:22:20.745008 21185 raft_consensus.cc:2749] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:22:20.726779 21633 raft_consensus.cc:515] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:20.729568 21615 raft_consensus.cc:515] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:20.744607 21240 raft_consensus.cc:2804] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Leader pre-election won for term 14
W20260709 12:22:20.775738 17804 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:20.778189 17836 raft_consensus.cc:1275] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260709 12:22:20.783322 21407 leader_election.cc:290] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:20.787761 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:20.720876 21700 raft_consensus.cc:515] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:20.764564 21630 raft_consensus.cc:2749] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:22:20.789744 17764 raft_consensus.cc:3060] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:20.793054 21702 consensus_queue.cc:1048] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:22:20.794346 17834 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35040 (request call id 2225) took 1195 ms. Trace:
W20260709 12:22:20.795260 21698 log.cc:927] Time spent T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.130s	user 0.000s	sys 0.001s
I20260709 12:22:20.794953 17834 rpcz_store.cc:276] 0709 12:22:19.598563 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:19.598706 (+   143us) service_pool.cc:224] Handling call
0709 12:22:20.794176 (+1195470us) spinlock_profiling.cc:243] Waited 80.6 ms on lock 0x7b54002eac54. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:22:20.794284 (+   108us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":185328512}
I20260709 12:22:20.797183 21685 leader_election.cc:290] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:20.797799 21660 leader_election.cc:290] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:20.794421 21633 leader_election.cc:290] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:20.791611 17652 leader_election.cc:304] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:20.800932 21700 leader_election.cc:290] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:20.798544 21662 leader_election.cc:290] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:20.802884 17727 leader_election.cc:304] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:20.804258 17727 leader_election.cc:400] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 6 pre-election: Vote denied by peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837) with higher term. Message: Invalid argument: T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for earlier term 6. Current term is 7.
I20260709 12:22:20.805002 21538 raft_consensus.cc:2804] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:22:20.805086 17727 leader_election.cc:304] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:22:20.805739 21538 raft_consensus.cc:697] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 7 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:22:20.807017 17727 leader_election.cc:304] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:22:20.809099 17652 leader_election.cc:304] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:20.809721 20394 raft_consensus.cc:2749] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:22:20.764098 21588 consensus_queue.cc:1048] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.023s
I20260709 12:22:20.815569 17832 raft_consensus.cc:2393] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 10: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:22:20.787211 21240 raft_consensus.cc:493] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:20.819736 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:20.821323 21379 raft_consensus.cc:2804] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:22:20.816826 21685 consensus_queue.cc:1048] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.005s
I20260709 12:22:20.835402 17727 leader_election.cc:304] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
W20260709 12:22:20.823693 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 548301us
W20260709 12:22:20.837815 18334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:20.825279 21644 raft_consensus.cc:2749] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Leader election lost for term 6. Reason: Vote denied by peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651) with higher term. Message: Invalid argument: T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 6. Current term is 7.
W20260709 12:22:20.826804 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:20.826696 21616 raft_consensus.cc:493] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
W20260709 12:22:20.820261 21413 consensus_meta.cc:287] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.813s	user 0.010s	sys 0.005s
I20260709 12:22:20.829854 17764 raft_consensus.cc:2468] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 10.
I20260709 12:22:20.829666 17652 leader_election.cc:304] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:20.846597 17761 raft_consensus.cc:1275] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 8 index: 7. Preceding OpId from leader: term: 11 index: 9. (index mismatch)
I20260709 12:22:20.848878 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 12 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:20.861647 21413 raft_consensus.cc:515] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:22:20.866528 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:20.832748 17834 raft_consensus.cc:2468] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 8.
I20260709 12:22:20.828964 17801 leader_election.cc:304] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:22:20.873503 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 44620.9us
I20260709 12:22:20.834654 21240 raft_consensus.cc:3060] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:20.815691 21666 raft_consensus.cc:2749] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:22:20.815450 21692 raft_consensus.cc:2749] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260709 12:22:20.874064 17801 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:20.818780 21541 consensus_queue.cc:1048] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.048s
W20260709 12:22:20.897542 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 652835us
I20260709 12:22:20.900117 17837 raft_consensus.cc:2368] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for earlier term 9. Current term is 10.
I20260709 12:22:20.900651 17801 leader_election.cc:304] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
W20260709 12:22:20.875458 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 45886.5us
W20260709 12:22:20.904251 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:20.834651 21379 raft_consensus.cc:493] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:22:20.850396 21649 raft_consensus.cc:493] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:20.908115 17652 leader_election.cc:304] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:20.910071 17801 leader_election.cc:304] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:20.837949 21700 consensus_queue.cc:1059] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.061s
W20260709 12:22:20.910712 17804 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:20.913167 17801 leader_election.cc:304] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:20.872223 21708 raft_consensus.cc:2804] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:22:20.917665 17652 leader_election.cc:304] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:20.848596 21505 consensus_queue.cc:1048] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.061s
I20260709 12:22:20.899467 21630 raft_consensus.cc:2804] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:22:20.898936 21662 raft_consensus.cc:2804] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:22:20.868290 21616 raft_consensus.cc:515] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:20.958415 17801 leader_election.cc:304] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:20.919396 21599 raft_consensus.cc:2804] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:22:20.873350 21676 consensus_queue.cc:237] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:20.906105 21644 raft_consensus.cc:493] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:20.972080 18333 leader_election.cc:304] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:20.925828 17652 leader_election.cc:304] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:20.913048 21379 raft_consensus.cc:3060] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:20.908432 21541 consensus_queue.cc:1243] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer e063541542a94969898868bf6eb5ce1b log is divergent from this leader: its last log entry 7.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
W20260709 12:22:20.968443 17804 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:20.955956 21660 raft_consensus.cc:2804] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:22:20.959914 21630 raft_consensus.cc:493] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:20.919620 21308 raft_consensus.cc:2804] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:22:20.964102 21707 consensus_queue.cc:1048] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.160s
I20260709 12:22:20.920189 21493 raft_consensus.cc:2749] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:22:20.964792 21624 raft_consensus.cc:3055] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 LEADER]: Stepping down as leader of term 9
W20260709 12:22:21.024145 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 98451.8us
I20260709 12:22:20.926692 21538 consensus_queue.cc:237] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:20.900372 21674 raft_consensus.cc:515] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:20.964383 21708 raft_consensus.cc:697] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:22:20.959498 21588 raft_consensus.cc:2804] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260709 12:22:21.015590 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 140316us
I20260709 12:22:20.999387 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:21.009848 21628 raft_consensus.cc:2749] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 12:22:21.027032 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:20.975472 21477 raft_consensus.cc:2749] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260709 12:22:21.019340 21709 log.cc:927] Time spent T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.253s	user 0.001s	sys 0.000s
I20260709 12:22:21.027129 17652 leader_election.cc:304] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:20.964672 21662 raft_consensus.cc:493] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:21.002774 21413 leader_election.cc:290] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:21.023175 21660 raft_consensus.cc:697] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 8 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
W20260709 12:22:21.050297 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 84127.4us
I20260709 12:22:21.016574 21630 raft_consensus.cc:3060] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:20.975708 21644 raft_consensus.cc:515] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:20.925729 21666 consensus_queue.cc:1048] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.109s
I20260709 12:22:20.920368 21649 raft_consensus.cc:515] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:21.023222 21407 raft_consensus.cc:2804] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Leader election won for term 10
W20260709 12:22:21.035687 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 77367.3us
I20260709 12:22:21.024434 21624 raft_consensus.cc:740] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:22:21.042711 17652 leader_election.cc:304] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:21.053395 18368 raft_consensus.cc:2468] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 7.
I20260709 12:22:21.029136 21308 raft_consensus.cc:697] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 11 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:22:21.032783 21588 raft_consensus.cc:493] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:22:21.065394 17801 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:21.048167 21615 leader_election.cc:290] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:21.049333 21662 raft_consensus.cc:3060] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:21.055801 21599 raft_consensus.cc:697] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 10 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:22:21.069242 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:21.071000 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:21.072103 21407 raft_consensus.cc:697] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 10 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:22:21.092890 21599 consensus_queue.cc:237] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.9, Last appended by leader: 9, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:21.093562 21407 consensus_queue.cc:237] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:22:21.072335 21712 log.cc:927] Time spent T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.162s	user 0.001s	sys 0.000s
I20260709 12:22:21.073509 17757 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38786 (request call id 2731) took 1039 ms. Trace:
W20260709 12:22:21.073830 21711 log.cc:927] Time spent T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.101s	user 0.000s	sys 0.000s
I20260709 12:22:21.096269 17757 rpcz_store.cc:276] 0709 12:22:20.034472 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:20.046228 (+ 11756us) service_pool.cc:224] Handling call
0709 12:22:20.417872 (+371644us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:20.418111 (+   239us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 12:22:20.439309 (+ 21198us) log.cc:844] Serialized 14 byte log entry
0709 12:22:20.521183 (+ 81874us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:20.521241 (+    58us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:22:20.521303 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:20.521377 (+    74us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:21.073301 (+551924us) spinlock_profiling.cc:243] Waited 355 ms on lock 0x7b540019f0d0. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a61f8ad 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:22:21.073445 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":49356,"prepare.queue_time_us":32747,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":37,"raft.queue_time_us":22314,"raft.run_cpu_time_us":192,"raft.run_wall_time_us":9152,"spinlock_wait_cycles":817253632,"thread_start_us":69914,"threads_started":1,"wal-append.queue_time_us":140707}
W20260709 12:22:21.076073 21713 log.cc:927] Time spent T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.056s	user 0.000s	sys 0.001s
I20260709 12:22:21.076434 21477 raft_consensus.cc:2804] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:22:21.092547 17836 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 7.
I20260709 12:22:21.065900 17729 leader_election.cc:304] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:21.048843 21493 raft_consensus.cc:2804] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:22:21.080139 17652 leader_election.cc:304] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:21.082130 21616 leader_election.cc:290] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:22:21.091046 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 54396.5us
I20260709 12:22:21.074450 21624 consensus_queue.cc:260] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 9.5, Last appended by leader: 5, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:21.099179 21477 raft_consensus.cc:697] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 11 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:22:21.103317 21660 consensus_queue.cc:237] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:21.079993 21588 raft_consensus.cc:3060] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:21.111147 17652 leader_election.cc:304] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:21.083755 21674 leader_election.cc:290] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 10 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:21.122346 21708 consensus_queue.cc:237] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 7, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:21.108531 21493 raft_consensus.cc:493] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:21.124301 21493 raft_consensus.cc:3060] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:21.125259 21630 raft_consensus.cc:515] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:21.116181 21624 raft_consensus.cc:3060] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:21.127804 21630 leader_election.cc:290] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 9 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:21.117069 21644 leader_election.cc:290] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:21.117898 18333 leader_election.cc:304] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:21.121312 21637 raft_consensus.cc:3060] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:21.113162 21156 consensus_queue.cc:1048] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.010s
I20260709 12:22:21.137928 18333 leader_election.cc:304] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:22:21.139241 21628 raft_consensus.cc:2749] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:22:21.140110 21628 raft_consensus.cc:2804] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:22:21.139815 21477 consensus_queue.cc:237] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:21.140509 21628 raft_consensus.cc:493] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:22:21.140899 21628 raft_consensus.cc:3060] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:21.086488 21308 consensus_queue.cc:237] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:21.111644 21640 raft_consensus.cc:2804] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260709 12:22:21.154788 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:21.153029 21640 raft_consensus.cc:493] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:21.114681 21633 consensus_queue.cc:1048] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.042s
I20260709 12:22:21.134650 21707 raft_consensus.cc:2804] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Leader election won for term 10
W20260709 12:22:21.162714 17804 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:21.165526 21640 raft_consensus.cc:3060] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:21.173671 18333 leader_election.cc:304] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
W20260709 12:22:21.181083 18371 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59918 (request call id 1253) took 1798 ms (client timeout 1673 ms). Trace:
W20260709 12:22:21.181586 18371 rpcz_store.cc:269] 0709 12:22:19.377816 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:19.377934 (+   118us) service_pool.cc:224] Handling call
0709 12:22:21.175665 (+1797731us) spinlock_profiling.cc:243] Waited 568 ms on lock 0x7b54007e8954. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:22:21.175771 (+   106us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1306672896}
I20260709 12:22:21.151093 21685 consensus_queue.cc:1048] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.025s
I20260709 12:22:21.161571 21649 leader_election.cc:290] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:21.179670 21644 raft_consensus.cc:2764] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Leader election decision vote started in defunct term 11: won
I20260709 12:22:21.203986 21589 leader_election.cc:290] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:21.211066 21379 raft_consensus.cc:515] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:21.212898 18333 leader_election.cc:304] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:22:21.213712 21379 leader_election.cc:290] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 8 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:21.214388 18333 leader_election.cc:304] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
W20260709 12:22:21.217947 21546 consensus_meta.cc:287] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.609s	user 0.005s	sys 0.005s
I20260709 12:22:21.220086 21644 raft_consensus.cc:2804] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:22:21.211766 21588 raft_consensus.cc:515] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:21.188705 21707 raft_consensus.cc:697] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 10 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:22:21.221903 18333 leader_election.cc:304] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:22:21.222609 21649 raft_consensus.cc:2804] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:22:21.238755 17763 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38786 (request call id 2717) took 1910 ms. Trace:
I20260709 12:22:21.240432 17763 rpcz_store.cc:276] 0709 12:22:19.328184 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:19.328351 (+   167us) service_pool.cc:224] Handling call
0709 12:22:19.351015 (+ 22664us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:20.178418 (+827403us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:21.238462 (+1060044us) spinlock_profiling.cc:243] Waited 810 ms on lock 0x7b54000ff0d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:22:21.238652 (+   190us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1862704000}
I20260709 12:22:21.247910 17833 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35042 (request call id 1984) took 1052 ms. Trace:
I20260709 12:22:21.248436 17833 rpcz_store.cc:276] 0709 12:22:20.195621 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:20.245186 (+ 49565us) service_pool.cc:224] Handling call
0709 12:22:21.247679 (+1002493us) spinlock_profiling.cc:243] Waited 35.5 ms on lock 0x7b54002ebb54. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:22:21.247823 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":81753344}
I20260709 12:22:21.250209 17835 raft_consensus.cc:1275] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 9 index: 7. Preceding OpId from leader: term: 10 index: 8. (index mismatch)
I20260709 12:22:21.238481 21644 raft_consensus.cc:493] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:21.252887 18333 leader_election.cc:304] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:21.246232 21616 raft_consensus.cc:2804] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:22:21.256688 21240 raft_consensus.cc:515] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:21.260285 21240 leader_election.cc:290] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 14 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:21.247326 21640 raft_consensus.cc:515] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:22:21.257325 21678 log.cc:927] Time spent T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.686s	user 0.001s	sys 0.012s
I20260709 12:22:21.262619 21640 leader_election.cc:290] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 9 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:21.262897 18374 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 1260) took 1407 ms. Trace:
I20260709 12:22:21.263298 18374 rpcz_store.cc:276] 0709 12:22:19.855417 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:19.855703 (+   286us) service_pool.cc:224] Handling call
0709 12:22:19.856115 (+   412us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:19.867616 (+ 11501us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:19.867663 (+    47us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:19.879065 (+ 11402us) log.cc:844] Serialized 26 byte log entry
0709 12:22:20.141191 (+262126us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:22:20.141251 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:20.141356 (+   105us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:21.171278 (+1029922us) spinlock_profiling.cc:243] Waited 6.5 ms on lock 0x7b4400022b4c. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b47459c85 00007f7b4745a6a2 00007f7b4a1cc1db 00007f7b4a1c7aaf 00007f7b4a1bb86e 00007f7b4a1babb6 00007f7b4a1ba7d8 00007f7b4a1c4221 00007f7b4a1c41d9 00007f7b4a1c4169
0709 12:22:21.262583 (+ 91305us) raft_consensus.cc:1685] finished
0709 12:22:21.262605 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:21.262810 (+   205us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":12547,"raft.run_cpu_time_us":193,"raft.run_wall_time_us":19843,"spinlock_wait_cycles":14947712,"thread_start_us":225484,"threads_started":1,"wal-append.queue_time_us":488009}
I20260709 12:22:21.264632 21600 raft_consensus.cc:493] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:21.257983 21616 raft_consensus.cc:493] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:22:21.265206 21600 raft_consensus.cc:515] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:21.265895 21616 raft_consensus.cc:3060] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:21.267997 21640 raft_consensus.cc:493] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:21.273628 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:21.269227 21662 raft_consensus.cc:515] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:21.248198 21649 raft_consensus.cc:697] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 9 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:22:21.240335 21546 raft_consensus.cc:515] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:21.275111 21676 raft_consensus.cc:493] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:21.280673 21640 raft_consensus.cc:515] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:21.284556 21676 raft_consensus.cc:515] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:21.254340 21644 raft_consensus.cc:3060] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:21.278203 17763 consensus_queue.cc:237] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 8.4, Last appended by leader: 3, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:22:21.257783 21707 consensus_queue.cc:237] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:21.288583 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:21.311185 17833 raft_consensus.cc:3060] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:21.275182 21617 raft_consensus.cc:2749] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260709 12:22:21.275586 21725 log.cc:927] Time spent T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.116s	user 0.000s	sys 0.001s
W20260709 12:22:21.316402 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 63568.1us
I20260709 12:22:21.276744 21588 leader_election.cc:290] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 8 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:22:21.299772 21728 log.cc:927] Time spent T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.057s	user 0.000s	sys 0.000s
I20260709 12:22:21.276021 21600 leader_election.cc:290] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:21.281610 21667 raft_consensus.cc:493] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:21.297092 21501 raft_consensus.cc:3055] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:22:21.297714 21546 leader_election.cc:290] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 8 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:22:21.329385 21689 log.cc:927] Time spent T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.824s	user 0.004s	sys 0.000s
I20260709 12:22:21.332974 17689 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37396 (request call id 2113) took 1651 ms. Trace:
I20260709 12:22:21.289247 21640 leader_election.cc:290] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:21.333469 17689 rpcz_store.cc:276] 0709 12:22:19.681421 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:19.681583 (+   162us) service_pool.cc:224] Handling call
0709 12:22:19.729942 (+ 48359us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:19.802093 (+ 72151us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:19.802123 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:20.089416 (+287293us) spinlock_profiling.cc:243] Waited 11 ms on lock 0x7b2c00000c88. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b48f19a11 00007f7b48f29372 00007f7b4a65ba01 00007f7b4a62043a 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2
0709 12:22:20.089828 (+   412us) log.cc:844] Serialized 26 byte log entry
0709 12:22:20.275670 (+185842us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:22:20.275725 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:20.275821 (+    96us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:21.322034 (+1046213us) spinlock_profiling.cc:243] Waited 6.87 ms on lock 0x7b4400022b4c. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b47459c85 00007f7b4745a6a2 00007f7b4a1cc1db 00007f7b4a1c7aaf 00007f7b4a1bb86e 00007f7b4a1babb6 00007f7b4a1ba7d8 00007f7b4a1c4221 00007f7b4a1c41d9 00007f7b4a1c4169
0709 12:22:21.332746 (+ 10712us) raft_consensus.cc:1685] finished
0709 12:22:21.332769 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:21.332909 (+   140us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":71825,"raft.queue_time_us":74478,"raft.run_cpu_time_us":120,"raft.run_wall_time_us":122,"spinlock_wait_cycles":41062784,"thread_start_us":185463,"threads_started":1,"wal-append.queue_time_us":327417}
W20260709 12:22:21.341964 21681 log.cc:927] Time spent T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.582s	user 0.001s	sys 0.001s
W20260709 12:22:21.343014 18334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:21.333015 21637 raft_consensus.cc:2749] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:22:21.331063 21501 raft_consensus.cc:740] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:22:21.347100 17837 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35042 (request call id 1979) took 1193 ms. Trace:
I20260709 12:22:21.349102 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:21.298789 21649 consensus_queue.cc:237] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:21.349746 18368 raft_consensus.cc:2468] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 8.
I20260709 12:22:21.349370 17833 raft_consensus.cc:2468] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 9.
I20260709 12:22:21.347537 17837 rpcz_store.cc:276] 0709 12:22:20.154065 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:20.154203 (+   138us) service_pool.cc:224] Handling call
0709 12:22:21.346908 (+1192705us) spinlock_profiling.cc:243] Waited 741 ms on lock 0x7b5400266454. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:22:21.347036 (+   128us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1705123072}
I20260709 12:22:21.330401 21667 raft_consensus.cc:515] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:21.352475 18333 leader_election.cc:304] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:22:21.357347 21667 leader_election.cc:290] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:21.357928 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:21.358336 18374 raft_consensus.cc:1275] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
I20260709 12:22:21.358532 18371 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 7.
I20260709 12:22:21.360343 21470 raft_consensus.cc:3060] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:21.363029 21628 raft_consensus.cc:515] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:21.367093 21493 raft_consensus.cc:515] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:21.368155 18333 leader_election.cc:304] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:21.368197 21433 raft_consensus.cc:493] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:21.368840 21662 leader_election.cc:290] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 8 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:21.369338 21493 leader_election.cc:290] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 10 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:21.369139 21433 raft_consensus.cc:515] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:21.370309 17829 raft_consensus.cc:2468] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 9.
I20260709 12:22:21.372884 21053 leader_election.cc:290] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:21.378590 21717 raft_consensus.cc:3060] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:21.378981 18374 raft_consensus.cc:1275] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 5 index: 8. Preceding OpId from leader: term: 7 index: 9. (index mismatch)
I20260709 12:22:21.380088 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:21.381687 17837 pending_rounds.cc:85] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b: Aborting all ops after (but not including) 2
I20260709 12:22:21.381999 17837 pending_rounds.cc:107] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b: Aborting uncommitted NO_OP operation due to leader change: 7.3
I20260709 12:22:21.382319 17837 raft_consensus.cc:2889] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:21.383239 20394 raft_consensus.cc:493] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:21.384002 20394 raft_consensus.cc:515] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:21.384698 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:21.386971 20394 leader_election.cc:290] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:21.388170 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:21.388346 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:21.389518 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:21.390192 17835 raft_consensus.cc:2468] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 7.
I20260709 12:22:21.390478 21628 leader_election.cc:290] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 9 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:21.391268 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:21.391199 21628 consensus_queue.cc:1048] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 12:22:21.392150 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:21.393369 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:21.393992 18366 raft_consensus.cc:3060] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:21.395784 17835 raft_consensus.cc:2393] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 10: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:22:21.399534 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 14 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:21.406098 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:21.407752 18371 raft_consensus.cc:3060] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:21.409258 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:21.414382 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:21.415419 17689 raft_consensus.cc:2393] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 9: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:22:21.416934 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:21.417770 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:21.418288 17681 raft_consensus.cc:1240] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 7. Current term is 8. Ops: []
I20260709 12:22:21.419518 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 8 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:21.420639 17682 raft_consensus.cc:2410] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for term 8 because replica has last-logged OpId of term: 6 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 5.
I20260709 12:22:21.423112 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:21.422719 21676 leader_election.cc:290] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:21.404624 21505 raft_consensus.cc:493] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
W20260709 12:22:21.433252 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:21.368901 21617 raft_consensus.cc:2764] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Leader election decision vote started in defunct term 7: won
I20260709 12:22:21.434478 17689 raft_consensus.cc:1240] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 7. Current term is 8. Ops: []
W20260709 12:22:21.435647 17801 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:21.435599 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:21.442389 18367 raft_consensus.cc:2468] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 6.
W20260709 12:22:21.443483 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 63944.1us
I20260709 12:22:21.400558 21616 raft_consensus.cc:515] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:21.445256 17684 raft_consensus.cc:1275] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260709 12:22:21.446813 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:21.447129 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:22:21.441320 21732 log.cc:927] Time spent T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.113s	user 0.001s	sys 0.000s
I20260709 12:22:21.452100 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 14 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:21.452754 17681 raft_consensus.cc:3060] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:21.453997 18371 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 10.
I20260709 12:22:21.454711 18368 raft_consensus.cc:3060] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:21.464272 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:21.466364 18366 raft_consensus.cc:2468] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 9.
I20260709 12:22:21.466436 17654 leader_election.cc:304] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:22:21.476368 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:21.477263 17728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:21.477209 17654 leader_election.cc:304] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:22:21.479532 17729 leader_election.cc:304] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:21.477882 18363 raft_consensus.cc:2468] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 8.
I20260709 12:22:21.440797 21505 raft_consensus.cc:515] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:21.482138 17654 leader_election.cc:304] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:22:21.469839 21185 consensus_queue.cc:1059] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.020s
I20260709 12:22:21.490527 21630 consensus_queue.cc:1048] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.002s
I20260709 12:22:21.485515 21240 raft_consensus.cc:2804] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:22:21.492717 21240 raft_consensus.cc:493] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:21.493062 21240 raft_consensus.cc:3060] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:21.482041 21633 consensus_queue.cc:1048] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.014s
I20260709 12:22:21.496419 21616 leader_election.cc:290] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 8 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:21.482247 21476 raft_consensus.cc:2804] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:22:21.497591 21476 raft_consensus.cc:493] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:21.463023 21676 consensus_queue.cc:1048] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.020s
I20260709 12:22:21.498034 21476 raft_consensus.cc:3060] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:21.500870 21717 raft_consensus.cc:2749] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:22:21.488384 21674 raft_consensus.cc:2804] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:22:21.481137 21617 raft_consensus.cc:493] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:21.512354 21674 raft_consensus.cc:493] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:21.512790 21674 raft_consensus.cc:3060] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:21.512640 21617 raft_consensus.cc:515] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:21.514814 21617 leader_election.cc:290] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:21.504438 21505 leader_election.cc:290] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:21.520763 17681 raft_consensus.cc:2468] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 14.
I20260709 12:22:21.515350 21644 raft_consensus.cc:515] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:21.532784 18368 raft_consensus.cc:2468] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 9.
W20260709 12:22:21.536145 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 49060.4us
W20260709 12:22:21.538026 17804 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:21.537410 17654 leader_election.cc:304] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:22:21.549525 18333 leader_election.cc:304] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:22:21.553292 20394 raft_consensus.cc:2749] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:22:21.554834 17728 leader_election.cc:304] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:21.555044 21337 consensus_queue.cc:1059] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:22:21.556315 21471 raft_consensus.cc:2749] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:22:21.556636 21337 consensus_queue.cc:1048] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:22:21.556845 17728 leader_election.cc:304] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:21.557226 21471 raft_consensus.cc:3055] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:22:21.557660 21471 raft_consensus.cc:740] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 7 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:22:21.557873 21667 raft_consensus.cc:2804] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:22:21.558604 21667 raft_consensus.cc:697] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:22:21.558564 21471 consensus_queue.cc:260] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:21.560678 21471 raft_consensus.cc:3060] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:21.560154 21667 consensus_queue.cc:237] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 12.12, Last appended by leader: 12, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:21.562350 17682 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "22fa38faae58422e9cecc99973cab688"
dest_uuid: "0da169ff82674dc1947055ce43de68a3"
mode: GRACEFUL
new_leader_uuid: "e063541542a94969898868bf6eb5ce1b"
 from {username='slave'} at 127.0.0.1:54540
W20260709 12:22:21.568765 17804 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:21.601365 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:21.551276 21630 raft_consensus.cc:3055] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:22:21.611707 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:21.614130 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:21.563045 21676 raft_consensus.cc:2804] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:22:21.600368 21617 raft_consensus.cc:2804] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:22:21.548460 21476 raft_consensus.cc:515] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:21.612010 21630 raft_consensus.cc:740] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:22:21.609009 21717 consensus_queue.cc:1059] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.022s
I20260709 12:22:21.622964 21644 leader_election.cc:290] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 10 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:21.623940 21617 raft_consensus.cc:493] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
W20260709 12:22:21.638540 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 149517us
W20260709 12:22:21.661994 17803 leader_election.cc:336] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Timed out: RequestConsensusVote RPC to 127.16.29.68:42577 timed out after 1.785s (SENT)
I20260709 12:22:21.664271 17803 leader_election.cc:304] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:21.629817 21676 raft_consensus.cc:697] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 9 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
W20260709 12:22:21.669859 17801 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:21.672585 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:21.673535 21401 raft_consensus.cc:2749] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:22:21.668924 21617 raft_consensus.cc:3060] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:21.685058 18333 leader_election.cc:400] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 8 pre-election: Vote denied by peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651) with higher term. Message: Invalid argument: T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 8. Current term is 9.
W20260709 12:22:21.693267 17728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:21.668681 21240 raft_consensus.cc:515] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:22:21.695479 17803 leader_election.cc:336] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Timed out: RequestConsensusVote RPC to 127.16.29.68:42577 timed out after 2.122s (SENT)
I20260709 12:22:21.699410 17803 leader_election.cc:304] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:21.709497 21674 raft_consensus.cc:515] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:21.712477 21674 leader_election.cc:290] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 10 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:21.712726 21398 consensus_queue.cc:1048] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 12:22:21.712975 21476 leader_election.cc:290] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 8 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:22:21.716588 17804 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:21.710026 17727 leader_election.cc:400] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 9 election: Vote denied by peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837) with higher term. Message: Invalid argument: T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for earlier term 9. Current term is 10.
I20260709 12:22:21.717247 17727 leader_election.cc:403] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 9 election: Cancelling election due to peer responding with higher term
I20260709 12:22:21.717095 21401 raft_consensus.cc:2749] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:22:21.718513 17727 leader_election.cc:304] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:21.719343 17586 catalog_manager.cc:5697] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b reported cstate change: term changed from 7 to 8, leader changed from 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) to e063541542a94969898868bf6eb5ce1b (127.16.29.67). New cstate: current_term: 8 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 12:22:21.728528 21745 log.cc:927] Time spent T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.063s	user 0.001s	sys 0.000s
I20260709 12:22:21.728922 21398 raft_consensus.cc:3060] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:21.731401 21240 leader_election.cc:290] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 8 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:22:21.733410 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 55100.4us
I20260709 12:22:21.735251 18333 leader_election.cc:304] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:21.736529 21504 raft_consensus.cc:3060] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:21.736935 21501 consensus_queue.cc:260] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:21.745579 21717 raft_consensus.cc:2749] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:22:21.749357 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:21.749754 21060 raft_consensus.cc:493] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:21.750192 17683 raft_consensus.cc:2393] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 8: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:22:21.750296 21060 raft_consensus.cc:515] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:22:21.751169 21748 log.cc:927] Time spent T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.067s	user 0.001s	sys 0.000s
I20260709 12:22:21.751292 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:21.753746 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:21.754900 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:21.755972 21060 leader_election.cc:290] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:21.757099 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:21.757874 17680 raft_consensus.cc:2393] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 10: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:22:21.758592 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:21.759160 17680 raft_consensus.cc:2468] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 10.
W20260709 12:22:21.759493 21741 log.cc:927] Time spent T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.154s	user 0.001s	sys 0.000s
I20260709 12:22:21.762140 21616 raft_consensus.cc:3055] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:22:21.762596 21616 raft_consensus.cc:740] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:22:21.727243 21640 raft_consensus.cc:2804] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:22:21.764946 21640 raft_consensus.cc:697] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 9 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:22:21.765875 21640 consensus_queue.cc:237] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:22:21.767608 21739 log.cc:927] Time spent T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.147s	user 0.000s	sys 0.001s
I20260709 12:22:21.769614 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:21.769701 21433 leader_election.cc:290] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:21.769961 17681 raft_consensus.cc:2468] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 8.
I20260709 12:22:21.770244 17837 raft_consensus.cc:2468] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 7.
I20260709 12:22:21.723839 21676 consensus_queue.cc:237] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:21.771167 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:21.771727 17837 raft_consensus.cc:2468] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 8.
I20260709 12:22:21.777736 17829 raft_consensus.cc:1275] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 9 index: 7. Preceding OpId from leader: term: 11 index: 8. (index mismatch)
I20260709 12:22:21.777498 21630 consensus_queue.cc:260] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:21.778537 21630 raft_consensus.cc:3060] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:21.779222 17829 raft_consensus.cc:1240] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 7. Current term is 8. Ops: []
I20260709 12:22:21.780372 21501 raft_consensus.cc:3060] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:22:21.780769 21742 log.cc:927] Time spent T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.106s	user 0.001s	sys 0.000s
I20260709 12:22:21.782681 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:21.783308 17685 raft_consensus.cc:2468] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 9.
I20260709 12:22:21.784581 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 12 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:21.785240 17680 raft_consensus.cc:2468] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 11.
I20260709 12:22:21.792637 17830 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35042 (request call id 1995) took 1453 ms. Trace:
I20260709 12:22:21.795360 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:21.797314 18361 raft_consensus.cc:2468] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 8.
I20260709 12:22:21.798522 18371 raft_consensus.cc:1240] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 6. Current term is 7. Ops: []
I20260709 12:22:21.798952 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:21.799507 18362 raft_consensus.cc:3060] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:22:21.800663 17803 leader_election.cc:336] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Timed out: RequestConsensusVote RPC to 127.16.29.68:42577 timed out after 2.228s (SENT)
I20260709 12:22:21.801294 17803 leader_election.cc:304] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:21.797117 17830 rpcz_store.cc:276] 0709 12:22:20.339215 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:20.684006 (+344791us) service_pool.cc:224] Handling call
0709 12:22:20.684356 (+   350us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 12:22:20.685020 (+   664us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:20.685048 (+    28us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 12:22:20.685675 (+   627us) log.cc:844] Serialized 78 byte log entry
0709 12:22:20.785081 (+ 99406us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:22:20.814640 (+ 29559us) log.cc:844] Serialized 14 byte log entry
0709 12:22:21.469585 (+654945us) spinlock_profiling.cc:243] Waited 3.64 ms on lock 0x7b4800190158. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4f549aa1 00007f7b4f540b37 00007f7b4f5409ca 00007f7b4f63826f 00007f7b4f637f37 00007f7b4f64c55c 00007f7b4f64c508 00007f7b4f64c488 00007f7b4f64c443 00007f7b4f64b6ec 00007f7b4e575a96
0709 12:22:21.743356 (+273771us) spinlock_profiling.cc:243] Waited 12.4 ms on lock 0x7b180027c5e0. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b474adc0a 00007f7b50bed219 00007f7b47535a40 00007f7b475304e4 00007f7b47530455 00007f7b4e0183c7 00007f7b4a62a9a7 00007f7b4a63a0bf 00007f7b4a63a038
0709 12:22:21.743614 (+   258us) log.cc:844] Serialized 14 byte log entry
0709 12:22:21.791866 (+ 48252us) log.cc:844] Serialized 14 byte log entry
0709 12:22:21.792212 (+   346us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:21.792317 (+   105us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:21.792377 (+    60us) raft_consensus.cc:1685] finished
0709 12:22:21.792395 (+    18us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:21.792563 (+   168us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":882854,"prepare.queue_time_us":958512,"prepare.run_cpu_time_us":63,"prepare.run_wall_time_us":63,"raft.queue_time_us":1197,"raft.run_cpu_time_us":5262,"raft.run_wall_time_us":791059,"spinlock_wait_cycles":36957696,"thread_start_us":98685,"threads_started":1,"wal-append.queue_time_us":221670}
I20260709 12:22:21.802933 21401 raft_consensus.cc:2749] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:22:21.806470 18362 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 8.
I20260709 12:22:21.807962 17834 raft_consensus.cc:1240] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 10 LEADER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 9. Current term is 10. Ops: [9.6-9.6]
I20260709 12:22:21.808561 17729 leader_election.cc:304] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:21.810644 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:21.812553 17685 raft_consensus.cc:1275] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 11 index: 9. (index mismatch)
I20260709 12:22:21.823117 17680 raft_consensus.cc:1275] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260709 12:22:21.828647 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:21.829315 18367 raft_consensus.cc:3060] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Advancing to term 10
W20260709 12:22:21.832491 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 97366.2us
I20260709 12:22:21.834448 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:22:21.839351 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:21.807893 21686 raft_consensus.cc:493] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:21.809603 21687 raft_consensus.cc:493] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:21.838222 21240 raft_consensus.cc:2804] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:22:21.865613 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:21.874584 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:21.880460 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:21.898365 17837 raft_consensus.cc:3060] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:21.884208 17586 catalog_manager.cc:5697] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b reported cstate change: term changed from 4 to 10, leader changed from 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) to e063541542a94969898868bf6eb5ce1b (127.16.29.67). New cstate: current_term: 10 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: HEALTHY } } }
I20260709 12:22:21.903855 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:21.904600 17756 raft_consensus.cc:2368] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 7. Current term is 8.
I20260709 12:22:21.905046 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:21.907501 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:21.763618 21616 consensus_queue.cc:260] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:21.796180 21433 consensus_queue.cc:1048] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.069s
I20260709 12:22:21.910753 17686 raft_consensus.cc:2393] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 10: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:22:21.911083 17689 raft_consensus.cc:3055] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:22:21.904623 17830 raft_consensus.cc:2468] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 9.
I20260709 12:22:21.922261 17689 raft_consensus.cc:740] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:22:21.871326 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:21.923230 17838 raft_consensus.cc:3060] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:21.915216 17652 leader_election.cc:304] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:22:21.923770 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:21.924464 20394 raft_consensus.cc:2804] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:22:21.924880 20394 raft_consensus.cc:493] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:21.925298 20394 raft_consensus.cc:3060] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:22:21.883981 21751 log.cc:927] Time spent T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.143s	user 0.000s	sys 0.001s
I20260709 12:22:21.921690 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
W20260709 12:22:21.884424 21711 log.cc:927] Time spent T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.054s	user 0.000s	sys 0.000s
I20260709 12:22:21.915817 18367 raft_consensus.cc:2468] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 10.
I20260709 12:22:21.928802 17682 raft_consensus.cc:606] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Received request to transfer leadership to TS e063541542a94969898868bf6eb5ce1b
W20260709 12:22:21.872359 21741 log.cc:927] Time spent T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.051s	user 0.000s	sys 0.000s
I20260709 12:22:21.930181 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 8 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:21.938773 17682 raft_consensus.cc:614] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:22:21.933020 17758 raft_consensus.cc:1275] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260709 12:22:21.947496 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:21.888200 21686 raft_consensus.cc:515] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:21.874873 21240 raft_consensus.cc:493] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:21.942648 17652 leader_election.cc:304] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:22:21.950685 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:21.958138 18365 raft_consensus.cc:3060] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Advancing to term 8
I20260709 12:22:21.959080 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:21.952626 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:21.960000 18362 raft_consensus.cc:2368] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 7. Current term is 8.
I20260709 12:22:21.953872 17762 raft_consensus.cc:2393] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 10: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:22:21.960222 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 8 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:21.954259 17686 raft_consensus.cc:1275] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 9 index: 7. Preceding OpId from leader: term: 11 index: 8. (index mismatch)
I20260709 12:22:21.961491 18362 raft_consensus.cc:1275] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 7 index: 9. (index mismatch)
I20260709 12:22:21.959978 17757 raft_consensus.cc:2468] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 9.
I20260709 12:22:21.952275 17689 consensus_queue.cc:260] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:21.965085 17689 raft_consensus.cc:3060] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:21.966487 17837 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 8.
I20260709 12:22:21.971784 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:21.975143 17652 leader_election.cc:304] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:22:21.975800 18371 raft_consensus.cc:1275] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260709 12:22:21.969775 18364 raft_consensus.cc:2376] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 8. Re-sending same reply.
I20260709 12:22:21.982746 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:21.983974 18362 raft_consensus.cc:2410] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 7 because replica has last-logged OpId of term: 6 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
I20260709 12:22:21.983194 20394 raft_consensus.cc:515] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:21.988143 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:21.988998 17830 raft_consensus.cc:2468] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 10.
I20260709 12:22:21.990070 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:21.962577 21240 raft_consensus.cc:3060] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:21.996937 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:22.008121 17837 raft_consensus.cc:3060] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:22.009634 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:21.967473 21719 raft_consensus.cc:493] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:21.888376 21687 raft_consensus.cc:515] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:22.011368 17830 raft_consensus.cc:3060] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:22:22.012621 17730 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:21.972708 21692 raft_consensus.cc:493] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:22.016139 17804 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:22.026576 17802 leader_election.cc:304] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:22.004014 17686 raft_consensus.cc:1275] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 9 index: 7. Preceding OpId from leader: term: 10 index: 8. (index mismatch)
I20260709 12:22:21.985654 21674 raft_consensus.cc:2804] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:22:22.038870 18371 raft_consensus.cc:1240] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 8. Current term is 10. Ops: [8.8-8.8]
I20260709 12:22:22.005694 21633 raft_consensus.cc:2804] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:22:22.044124 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:22.044340 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:22.045217 17686 raft_consensus.cc:2393] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 10: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:22:22.048251 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:22.056293 17837 raft_consensus.cc:2468] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 8.
I20260709 12:22:22.057901 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 8 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:22.058956 17837 raft_consensus.cc:2393] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 8: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
W20260709 12:22:22.060724 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:22.061151 17755 raft_consensus.cc:1240] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 9. Current term is 10. Ops: [9.6-9.6]
I20260709 12:22:22.060489 17686 raft_consensus.cc:1275] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 10 index: 7. Preceding OpId from leader: term: 11 index: 7. (term mismatch)
I20260709 12:22:22.061975 17686 pending_rounds.cc:85] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3: Aborting all ops after (but not including) 6
I20260709 12:22:22.062318 17686 pending_rounds.cc:107] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3: Aborting uncommitted NO_OP operation due to leader change: 10.7
I20260709 12:22:22.062647 17686 raft_consensus.cc:2889] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:22.064777 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:22.063576 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:22.068275 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:22.068894 17759 raft_consensus.cc:3060] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:22.043128 21674 raft_consensus.cc:493] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:22.070266 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:22.070376 17761 raft_consensus.cc:1240] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 LEADER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 13. Current term is 14. Ops: []
I20260709 12:22:22.098027 17758 raft_consensus.cc:2393] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 8: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:22:22.098089 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:22.101861 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:22.102501 17755 raft_consensus.cc:3060] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:22.102764 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:22.103690 17837 raft_consensus.cc:2393] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 7: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:22:22.104063 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:22.104781 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:22.105211 18364 raft_consensus.cc:2410] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 8 because replica has last-logged OpId of term: 7 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 8.
I20260709 12:22:22.029094 21719 raft_consensus.cc:515] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:22.115114 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:22.068867 21633 raft_consensus.cc:697] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 8 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:22:22.150447 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:22.033795 21692 raft_consensus.cc:515] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:22.155026 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 13 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:22.176653 17835 raft_consensus.cc:2410] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for term 13 because replica has last-logged OpId of term: 12 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 9.
W20260709 12:22:22.160529 21753 log.cc:927] Time spent T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.212s	user 0.001s	sys 0.000s
I20260709 12:22:22.165110 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:22.182907 17831 raft_consensus.cc:2468] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 7.
I20260709 12:22:22.173255 21599 raft_consensus.cc:2749] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:22:22.178750 17729 leader_election.cc:304] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
W20260709 12:22:22.179458 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:22.098049 21674 raft_consensus.cc:3060] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:22.181043 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:22.194008 17838 raft_consensus.cc:2468] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 10.
I20260709 12:22:22.195230 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:22.197921 17831 raft_consensus.cc:2368] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 8. Current term is 9.
I20260709 12:22:22.197873 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:22.198753 17835 raft_consensus.cc:2468] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 8.
W20260709 12:22:22.196913 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 221878us
I20260709 12:22:22.200183 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:22.200680 17755 raft_consensus.cc:2468] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 10.
I20260709 12:22:22.201023 18364 raft_consensus.cc:2376] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 8. Re-sending same reply.
I20260709 12:22:22.196833 18365 raft_consensus.cc:2468] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 8.
I20260709 12:22:22.203061 17759 raft_consensus.cc:2468] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 8.
W20260709 12:22:22.204054 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 167169us
I20260709 12:22:22.180677 21564 raft_consensus.cc:493] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
W20260709 12:22:22.207520 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 221619us
I20260709 12:22:22.209649 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:22.208565 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:22.208390 17833 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35040 (request call id 2260) took 1052 ms. Trace:
I20260709 12:22:22.212615 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:22.210178 17654 leader_election.cc:304] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:22:22.215058 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:22.222330 17835 raft_consensus.cc:1217] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Deduplicated request from leader. Original: 6.8->[7.9-7.9]   Dedup: 7.9->[]
I20260709 12:22:22.223147 17835 raft_consensus.cc:1240] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 7. Current term is 8. Ops: [7.9-7.9]
I20260709 12:22:22.193290 21433 raft_consensus.cc:493] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
W20260709 12:22:21.966387 21588 consensus_queue.cc:1175] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "e063541542a94969898868bf6eb5ce1b" responder_term: 10 status { last_received { term: 10 index: 6 } last_committed_idx: 5 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 9. Current term is 10. Ops: [9.6-9.6]" } } last_received_current_leader { term: 0 index: 0 } }
I20260709 12:22:22.240692 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
W20260709 12:22:22.335381 17655 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:22.735260 17758 raft_consensus.cc:3060] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:22:22.242074 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 215571us
W20260709 12:22:22.481789 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:22.270884 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:22.737828 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 559125us
W20260709 12:22:22.651473 17834 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.757s	user 0.005s	sys 0.023s
W20260709 12:22:22.738534 17834 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.845s	user 0.009s	sys 0.023s
I20260709 12:22:22.297482 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:22.740270 17835 raft_consensus.cc:3060] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:22:22.394105 17829 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.612s	user 0.012s	sys 0.003s
W20260709 12:22:22.741088 17829 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.959s	user 0.016s	sys 0.016s
I20260709 12:22:22.299590 17830 raft_consensus.cc:2468] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 9.
I20260709 12:22:22.733605 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:22.734347 17831 raft_consensus.cc:1240] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 9. Current term is 10. Ops: [9.8-9.8]
I20260709 12:22:22.241250 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:22:22.502488 17804 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:22.305372 17803 leader_election.cc:336] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Timed out: RequestConsensusVote RPC to 127.16.29.68:42577 timed out after 1.742s (SENT)
W20260709 12:22:22.367100 18334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:22.748287 17803 leader_election.cc:304] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
W20260709 12:22:22.749661 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 444295us
I20260709 12:22:22.387476 18367 raft_consensus.cc:1240] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 8. Current term is 9. Ops: []
W20260709 12:22:22.750922 17803 leader_election.cc:336] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Timed out: RequestConsensusVote RPC to 127.16.29.68:42577 timed out after 2.123s (SENT)
I20260709 12:22:22.436064 17836 raft_consensus.cc:1240] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 6. Current term is 7. Ops: [6.8-6.8]
I20260709 12:22:22.213023 17833 rpcz_store.cc:276] 0709 12:22:21.156481 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:21.465394 (+308913us) service_pool.cc:224] Handling call
0709 12:22:21.465824 (+   430us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:21.466040 (+   216us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 12:22:21.976487 (+510447us) log.cc:844] Serialized 14 byte log entry
0709 12:22:22.207576 (+231089us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:22.207630 (+    54us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:22:22.207676 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:22.207754 (+    78us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:22.208335 (+   581us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":281471,"prepare.queue_time_us":287048,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":37,"raft.queue_time_us":46693,"raft.run_cpu_time_us":126,"raft.run_wall_time_us":6365,"thread_start_us":459323,"threads_started":2}
I20260709 12:22:22.738107 17802 leader_election.cc:304] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:22.303501 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 13 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:22:22.748648 18334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:22.770692 17801 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:22.771469 18361 raft_consensus.cc:1275] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20260709 12:22:22.751471 17803 leader_election.cc:304] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:22.772094 17802 leader_election.cc:304] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:22.773018 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:22.773594 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:22.774854 17836 raft_consensus.cc:1240] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 10 LEADER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 9. Current term is 10. Ops: []
I20260709 12:22:22.776662 17830 raft_consensus.cc:1275] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 9. (index mismatch)
I20260709 12:22:22.777721 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:22.777915 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:22.778474 18367 raft_consensus.cc:2393] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 8: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:22:22.778326 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:22.778811 17763 raft_consensus.cc:2393] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 10: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
W20260709 12:22:22.740279 21240 consensus_meta.cc:287] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.717s	user 0.011s	sys 0.000s
W20260709 12:22:22.658223 21617 consensus_meta.cc:287] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.971s	user 0.011s	sys 0.000s
W20260709 12:22:22.780778 17654 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:22.780881 18367 raft_consensus.cc:1275] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 7 index: 9. Preceding OpId from leader: term: 8 index: 9. (term mismatch)
I20260709 12:22:22.784013 18367 pending_rounds.cc:85] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da: Aborting all ops after (but not including) 8
I20260709 12:22:22.784417 18367 pending_rounds.cc:107] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da: Aborting uncommitted NO_OP operation due to leader change: 7.9
I20260709 12:22:22.784801 18367 raft_consensus.cc:2889] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:22.265163 21505 raft_consensus.cc:2749] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260709 12:22:22.764706 21398 consensus_meta.cc:287] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 1.035s	user 0.010s	sys 0.001s
I20260709 12:22:22.275161 21676 raft_consensus.cc:493] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:22.282927 21686 leader_election.cc:290] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:22.787222 18365 raft_consensus.cc:1275] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20260709 12:22:22.260212 21705 raft_consensus.cc:2804] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Leader election won for term 10
W20260709 12:22:22.776261 21501 consensus_meta.cc:287] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.993s	user 0.008s	sys 0.003s
I20260709 12:22:22.260421 21379 consensus_queue.cc:1048] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.531s
I20260709 12:22:22.739691 21687 leader_election.cc:290] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:22.790704 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:22.240095 21564 raft_consensus.cc:515] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:22:22.817862 17756 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.560s	user 0.007s	sys 0.030s
I20260709 12:22:22.276309 21708 consensus_queue.cc:1048] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.514s
I20260709 12:22:22.822824 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:22.771755 21767 raft_consensus.cc:2749] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:22:22.789395 17831 raft_consensus.cc:1275] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
W20260709 12:22:22.793737 21762 log.cc:927] Time spent T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.504s	user 0.001s	sys 0.000s
I20260709 12:22:22.826582 17835 raft_consensus.cc:2468] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 9.
I20260709 12:22:22.827279 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:22.740042 21640 consensus_queue.cc:1059] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.059s
I20260709 12:22:22.827870 17833 raft_consensus.cc:1275] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20260709 12:22:22.828894 17759 raft_consensus.cc:2468] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 9.
I20260709 12:22:22.820729 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:22:22.831293 21759 log.cc:927] Time spent T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.311s	user 0.000s	sys 0.000s
W20260709 12:22:22.831965 21764 log.cc:927] Time spent T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.434s	user 0.001s	sys 0.000s
I20260709 12:22:22.832999 21719 leader_election.cc:290] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:22:22.829978 17756 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.578s	user 0.008s	sys 0.031s
I20260709 12:22:22.796352 21398 raft_consensus.cc:2749] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Leader election lost for term 9. Reason: Vote denied by peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837) with higher term. Message: Invalid argument: T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for earlier term 9. Current term is 10.
I20260709 12:22:22.836503 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:22.836594 17682 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f"
dest_uuid: "0da169ff82674dc1947055ce43de68a3"
mode: GRACEFUL
new_leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
 from {username='slave'} at 127.0.0.1:54540
I20260709 12:22:22.844535 17682 raft_consensus.cc:606] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Received request to transfer leadership to TS e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:22.844995 17682 raft_consensus.cc:614] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:22:22.789515 21240 raft_consensus.cc:515] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:22:22.839905 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 625685us
I20260709 12:22:22.790896 21617 raft_consensus.cc:515] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:22.840327 17684 raft_consensus.cc:2468] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 10.
I20260709 12:22:22.772179 21433 raft_consensus.cc:515] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:22.836892 21407 raft_consensus.cc:2804] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:22:22.840380 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:22.864784 17759 raft_consensus.cc:2410] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for term 8 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20260709 12:22:22.850170 17652 leader_election.cc:304] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:22.859823 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:22.865422 18361 raft_consensus.cc:1275] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260709 12:22:22.813671 21676 raft_consensus.cc:515] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:22.866951 17652 leader_election.cc:304] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:22.868166 17652 leader_election.cc:304] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:22.869130 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:22.869844 17756 raft_consensus.cc:2468] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 9.
I20260709 12:22:22.871308 21768 raft_consensus.cc:2749] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:22:22.872035 21768 raft_consensus.cc:2749] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 12:22:22.873252 21768 raft_consensus.cc:2749] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:22:22.874552 17652 leader_election.cc:304] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:22:22.875320 21768 raft_consensus.cc:2804] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:22:22.876545 18368 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 1288) took 1081 ms. Trace:
I20260709 12:22:22.877104 18368 rpcz_store.cc:276] 0709 12:22:21.795583 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:21.796462 (+   879us) service_pool.cc:224] Handling call
0709 12:22:21.796681 (+   219us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:21.830998 (+ 34317us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:21.831032 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:21.831549 (+   517us) log.cc:844] Serialized 26 byte log entry
0709 12:22:22.102423 (+270874us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:22:22.102467 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:22.102533 (+    66us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:22.876254 (+773721us) raft_consensus.cc:1685] finished
0709 12:22:22.876278 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:22.876467 (+   189us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":8099,"raft.queue_time_us":13170,"raft.run_cpu_time_us":153,"raft.run_wall_time_us":13505,"thread_start_us":212970,"threads_started":1,"wal-append.queue_time_us":389092}
I20260709 12:22:22.879657 18362 raft_consensus.cc:1240] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 7. Current term is 8. Ops: [7.7-7.7]
I20260709 12:22:22.881060 21770 raft_consensus.cc:2749] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:22:22.869151 21719 raft_consensus.cc:3055] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:22:22.885273 21719 raft_consensus.cc:740] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 7 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:22:22.880951 17652 leader_election.cc:304] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:22.886379 21719 consensus_queue.cc:260] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:22.891000 17682 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "8a208e793881438b975b83755dc05cfa"
dest_uuid: "0da169ff82674dc1947055ce43de68a3"
mode: GRACEFUL
new_leader_uuid: "e063541542a94969898868bf6eb5ce1b"
 from {username='slave'} at 127.0.0.1:54540
I20260709 12:22:22.891616 17682 raft_consensus.cc:606] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Received request to transfer leadership to TS e063541542a94969898868bf6eb5ce1b
I20260709 12:22:22.891922 17682 raft_consensus.cc:614] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:22:22.891000 21719 raft_consensus.cc:3060] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:22.894132 17755 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 10.
I20260709 12:22:22.899055 18366 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 1289) took 1103 ms. Trace:
I20260709 12:22:22.899482 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:22.900678 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:22.899583 18366 rpcz_store.cc:276] 0709 12:22:21.795842 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:21.797539 (+  1697us) service_pool.cc:224] Handling call
0709 12:22:21.820537 (+ 22998us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:21.820885 (+   348us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:21.820910 (+    25us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:21.843028 (+ 22118us) log.cc:844] Serialized 26 byte log entry
0709 12:22:22.072724 (+229696us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:22:22.110992 (+ 38268us) log.cc:844] Serialized 14 byte log entry
0709 12:22:22.111285 (+   293us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:22.111383 (+    98us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:22.158497 (+ 47114us) spinlock_profiling.cc:243] Waited 10.3 ms on lock 0x7b4800220a58. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4f549aa1 00007f7b4f540b37 00007f7b4f5409ca 00007f7b4f63826f 00007f7b4f637f37 00007f7b4f64c55c 00007f7b4f64c508 00007f7b4f64c488 00007f7b4f64c443 00007f7b4f64b6ec 00007f7b4e575a96
0709 12:22:22.898719 (+740222us) raft_consensus.cc:1685] finished
0709 12:22:22.898835 (+   116us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:22.898985 (+   150us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":37980,"prepare.run_cpu_time_us":80,"prepare.run_wall_time_us":20527,"raft.queue_time_us":987,"raft.run_cpu_time_us":3233,"raft.run_wall_time_us":327847,"spinlock_wait_cycles":36233856,"thread_start_us":187993,"threads_started":1,"wal-append.queue_time_us":336017}
I20260709 12:22:22.901567 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:22.901621 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 12 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:22.902204 18366 raft_consensus.cc:2468] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 7.
I20260709 12:22:22.902906 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:22.898995 17682 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b"
dest_uuid: "0da169ff82674dc1947055ce43de68a3"
mode: GRACEFUL
new_leader_uuid: "e063541542a94969898868bf6eb5ce1b"
 from {username='slave'} at 127.0.0.1:54540
I20260709 12:22:22.903709 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:22.290796 21633 consensus_queue.cc:237] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.6, Last appended by leader: 6, Current term: 8, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:22.904068 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:22.904575 18364 raft_consensus.cc:2410] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 7 because replica has last-logged OpId of term: 6 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
I20260709 12:22:22.905105 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:22.905418 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:22.906167 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:22.906350 18364 raft_consensus.cc:2368] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 7. Current term is 8.
I20260709 12:22:22.907187 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:22.907860 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:22.908321 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:22.908566 18361 raft_consensus.cc:2393] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 7: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:22:22.903750 17682 raft_consensus.cc:606] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Received request to transfer leadership to TS e063541542a94969898868bf6eb5ce1b
I20260709 12:22:22.909166 21617 leader_election.cc:290] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 9 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:22.910871 18371 raft_consensus.cc:1240] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 8. Current term is 9. Ops: []
I20260709 12:22:22.911482 21564 leader_election.cc:290] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:22.913403 21379 raft_consensus.cc:493] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:22.914001 21379 raft_consensus.cc:515] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:22.917476 21240 leader_election.cc:290] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 9 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:22.920397 17834 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35040 (request call id 2272) took 1067 ms. Trace:
I20260709 12:22:22.920892 17834 rpcz_store.cc:276] 0709 12:22:21.853442 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:21.887342 (+ 33900us) service_pool.cc:224] Handling call
0709 12:22:21.887696 (+   354us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:21.893067 (+  5371us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:21.893105 (+    38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:21.893627 (+   522us) log.cc:844] Serialized 26 byte log entry
0709 12:22:22.739167 (+845540us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:22:22.739212 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:22.739285 (+    73us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:22.920122 (+180837us) raft_consensus.cc:1685] finished
0709 12:22:22.920150 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:22.920314 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":4733,"raft.run_cpu_time_us":90,"raft.run_wall_time_us":91,"thread_start_us":845106,"threads_started":1,"wal-append.queue_time_us":973119}
I20260709 12:22:22.902973 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:22.924692 18366 raft_consensus.cc:2468] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 8.
I20260709 12:22:22.924984 21433 leader_election.cc:290] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:22:22.862623 21630 consensus_meta.cc:287] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 1.084s	user 0.007s	sys 0.006s
I20260709 12:22:22.933436 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:22.938761 18368 raft_consensus.cc:2393] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 10: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:22:22.909368 17682 raft_consensus.cc:614] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:22:22.940423 17758 raft_consensus.cc:2468] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 9.
I20260709 12:22:22.940542 16501 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260709 12:22:22.942836 17758 raft_consensus.cc:1275] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260709 12:22:22.909035 18365 raft_consensus.cc:2410] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 10 because replica has last-logged OpId of term: 9 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 7.
I20260709 12:22:22.945258 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:22.945634 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:22.946051 18367 raft_consensus.cc:2393] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 9: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:22:22.867976 21767 raft_consensus.cc:2804] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260709 12:22:22.896139 21674 consensus_meta.cc:287] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.701s	user 0.004s	sys 0.006s
I20260709 12:22:22.868326 21407 raft_consensus.cc:493] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:22.953238 17758 raft_consensus.cc:1275] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20260709 12:22:22.956557 17758 raft_consensus.cc:1275] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260709 12:22:22.961869 17758 raft_consensus.cc:1275] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20260709 12:22:22.963011 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:22.964090 18364 raft_consensus.cc:2393] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 10: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:22:22.964119 17758 raft_consensus.cc:1275] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260709 12:22:22.953763 21767 raft_consensus.cc:493] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:22.955672 21407 raft_consensus.cc:3060] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:22.955833 21630 raft_consensus.cc:493] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:23.044592 16501 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
W20260709 12:22:23.048468 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 167595us
I20260709 12:22:23.049435 17586 catalog_manager.cc:5697] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b reported cstate change: term changed from 8 to 9, leader changed from 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) to e063541542a94969898868bf6eb5ce1b (127.16.29.67). New cstate: current_term: 9 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: HEALTHY } } }
I20260709 12:22:23.050408 17652 leader_election.cc:304] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:22.958833 21379 leader_election.cc:290] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:23.056495 17652 leader_election.cc:304] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:22.954363 21674 raft_consensus.cc:515] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:23.043367 21767 raft_consensus.cc:3060] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:22:23.062160 17727 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.849s	user 0.027s	sys 0.085s
W20260709 12:22:23.062791 17727 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.850s	user 0.027s	sys 0.085s
W20260709 12:22:23.063135 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 291100us
W20260709 12:22:23.063755 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 851897us
I20260709 12:22:23.059602 21379 raft_consensus.cc:2804] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Leader election won for term 8
W20260709 12:22:23.065889 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 315000us
I20260709 12:22:23.055825 21630 raft_consensus.cc:515] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:22:23.059052 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 320347us
I20260709 12:22:23.061007 21633 raft_consensus.cc:2749] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:22:23.081880 17829 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35042 (request call id 2016) took 1359 ms. Trace:
W20260709 12:22:23.067960 17760 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.862s	user 0.000s	sys 0.011s
I20260709 12:22:23.087108 17801 leader_election.cc:304] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:23.122906 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:23.092643 17758 raft_consensus.cc:1275] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260709 12:22:23.100185 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 12 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:23.130796 18365 raft_consensus.cc:2393] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 7: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:22:23.134241 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:23.130716 17829 rpcz_store.cc:276] 0709 12:22:21.722505 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:21.780144 (+ 57639us) service_pool.cc:224] Handling call
0709 12:22:21.780502 (+   358us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:21.781408 (+   906us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:21.781450 (+    42us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:21.782025 (+   575us) log.cc:844] Serialized 26 byte log entry
0709 12:22:22.741470 (+959445us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:22:22.741507 (+    37us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:22.741569 (+    62us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:23.081574 (+340005us) raft_consensus.cc:1685] finished
0709 12:22:23.081603 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:23.081798 (+   195us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":248,"raft.run_cpu_time_us":95,"raft.run_wall_time_us":97,"thread_start_us":959046,"threads_started":1,"wal-append.queue_time_us":1115815}
W20260709 12:22:23.067783 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 295261us
I20260709 12:22:23.132452 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:22:23.130659 17760 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.924s	user 0.000s	sys 0.011s
I20260709 12:22:23.126142 21504 raft_consensus.cc:493] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:23.121215 21791 raft_consensus.cc:2749] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:22:23.142005 21630 leader_election.cc:290] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:23.143226 21785 raft_consensus.cc:2804] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:22:23.143375 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
W20260709 12:22:23.144660 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 57228.1us
W20260709 12:22:23.146873 17803 leader_election.cc:336] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Timed out: RequestConsensusVote RPC to 127.16.29.68:42577 timed out after 3.180s (SENT)
I20260709 12:22:23.150274 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:23.155700 17803 leader_election.cc:304] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:23.152562 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:23.156461 18365 raft_consensus.cc:2468] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 7.
I20260709 12:22:23.152551 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
W20260709 12:22:23.160126 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:23.161388 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:23.162254 18365 raft_consensus.cc:2393] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 8: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:22:23.162026 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:23.163179 18367 raft_consensus.cc:3060] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:23.164162 21781 raft_consensus.cc:493] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
W20260709 12:22:23.164413 21793 log.cc:927] Time spent T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.109s	user 0.001s	sys 0.000s
I20260709 12:22:23.164805 21433 raft_consensus.cc:2749] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:22:23.164768 21781 raft_consensus.cc:515] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:23.165688 21433 consensus_queue.cc:1048] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:22:23.167120 21781 leader_election.cc:290] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:23.170749 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:23.170791 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:23.171739 18364 raft_consensus.cc:2393] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 9: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:22:23.172585 18365 raft_consensus.cc:3055] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:22:23.167147 21433 consensus_queue.cc:1048] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.006s
I20260709 12:22:23.177767 17760 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38776 (request call id 2059) took 1148 ms. Trace:
I20260709 12:22:23.179442 21780 raft_consensus.cc:493] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:23.180199 21780 raft_consensus.cc:515] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:23.181041 21779 raft_consensus.cc:493] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:22:23.181813 21616 raft_consensus.cc:3060] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:23.181598 21779 raft_consensus.cc:515] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:23.154393 21790 raft_consensus.cc:493] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:23.154309 21785 raft_consensus.cc:493] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:23.157250 21676 leader_election.cc:290] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:23.182894 21780 leader_election.cc:290] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:23.172907 18365 raft_consensus.cc:740] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:22:23.178674 17760 rpcz_store.cc:276] 0709 12:22:22.029109 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:22.037700 (+  8591us) service_pool.cc:224] Handling call
0709 12:22:22.065443 (+ 27743us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:22.106893 (+ 41450us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:22.106929 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:22.147352 (+ 40423us) log.cc:844] Serialized 26 byte log entry
0709 12:22:23.138836 (+991484us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:22:23.142655 (+  3819us) log.cc:844] Serialized 14 byte log entry
0709 12:22:23.156438 (+ 13783us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:23.156529 (+    91us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:23.177229 (+ 20700us) raft_consensus.cc:1685] finished
0709 12:22:23.177253 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:23.177625 (+   372us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":76709,"prepare.queue_time_us":2694,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":46,"raft.queue_time_us":47885,"raft.run_cpu_time_us":301,"raft.run_wall_time_us":12827,"thread_start_us":932518,"threads_started":1,"wal-append.queue_time_us":986522}
I20260709 12:22:23.157889 21630 raft_consensus.cc:493] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:22:23.188333 17759 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38776 (request call id 2061) took 1154 ms. Trace:
I20260709 12:22:23.189417 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:23.159792 21791 raft_consensus.cc:493] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:23.198372 21791 raft_consensus.cc:515] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:23.153481 21504 raft_consensus.cc:515] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:23.190745 21779 leader_election.cc:290] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:23.186064 21785 raft_consensus.cc:3060] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:23.182895 21790 raft_consensus.cc:515] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:23.197679 21782 raft_consensus.cc:493] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:23.185902 18365 consensus_queue.cc:260] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:23.200593 21791 leader_election.cc:290] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:23.209394 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:23.210166 18366 raft_consensus.cc:2393] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 10: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:22:23.210742 21790 leader_election.cc:290] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:23.204413 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:23.211408 21782 raft_consensus.cc:515] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:23.212227 18361 raft_consensus.cc:2468] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 10.
I20260709 12:22:23.213570 21782 leader_election.cc:290] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:23.202920 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:23.215209 18364 raft_consensus.cc:2393] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 10: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:22:23.205899 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:23.194599 17759 rpcz_store.cc:276] 0709 12:22:22.029699 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:22.869147 (+839448us) service_pool.cc:224] Handling call
0709 12:22:22.869490 (+   343us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:22.869827 (+   337us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:22.869863 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:22.870362 (+   499us) log.cc:844] Serialized 26 byte log entry
0709 12:22:23.043159 (+172797us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:22:23.065531 (+ 22372us) log.cc:844] Serialized 14 byte log entry
0709 12:22:23.127586 (+ 62055us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:23.127722 (+   136us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:23.184105 (+ 56383us) raft_consensus.cc:1685] finished
0709 12:22:23.184125 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:23.184270 (+   145us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2408,"prepare.queue_time_us":12204,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":37,"raft.queue_time_us":19522,"raft.run_cpu_time_us":356,"raft.run_wall_time_us":37517,"thread_start_us":172334,"threads_started":1,"wal-append.queue_time_us":285110}
I20260709 12:22:23.207561 18367 raft_consensus.cc:2468] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 8.
I20260709 12:22:23.208824 18365 raft_consensus.cc:3060] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:23.216938 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:23.219195 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:23.230049 18366 raft_consensus.cc:2393] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 9: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:22:23.230664 21616 raft_consensus.cc:493] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:23.231226 21616 raft_consensus.cc:515] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:23.233474 21616 leader_election.cc:290] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:23.234532 18361 raft_consensus.cc:2368] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 7. Current term is 8.
I20260709 12:22:23.238097 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:23.202565 21630 raft_consensus.cc:515] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:23.222800 21767 raft_consensus.cc:515] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:23.239207 18365 raft_consensus.cc:1275] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20260709 12:22:23.240831 21767 leader_election.cc:290] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 9 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:23.225193 21407 raft_consensus.cc:515] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:23.243119 21407 leader_election.cc:290] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 11 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:22:23.228945 17803 leader_election.cc:336] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Timed out: RequestConsensusVote RPC to 127.16.29.68:42577 timed out after 1.618s (SENT)
W20260709 12:22:23.244614 17803 leader_election.cc:336] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Timed out: RequestConsensusVote RPC to 127.16.29.68:42577 timed out after 1.622s (SENT)
I20260709 12:22:23.229305 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:23.245007 21504 leader_election.cc:290] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:23.245103 17803 leader_election.cc:304] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:23.220022 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:22:23.228354 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 67072.3us
I20260709 12:22:23.246338 21692 leader_election.cc:290] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:23.247838 17760 raft_consensus.cc:1275] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
I20260709 12:22:23.249624 21407 raft_consensus.cc:2749] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20260709 12:22:23.249804 17762 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36572 (request call id 717) took 1328 ms (client timeout 1652 ms). Trace:
I20260709 12:22:23.249912 17756 raft_consensus.cc:1240] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 9. Current term is 10. Ops: []
I20260709 12:22:23.250305 21407 consensus_queue.cc:1048] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:22:23.250767 21782 consensus_queue.cc:1048] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 12:22:23.252904 21630 leader_election.cc:290] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:23.201431 17758 raft_consensus.cc:2468] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 10.
I20260709 12:22:23.253078 21782 consensus_queue.cc:1048] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 12:22:23.255787 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:23.256408 18367 raft_consensus.cc:3060] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:23.258144 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:23.258752 18362 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59902 (request call id 1017) took 1248 ms. Trace:
I20260709 12:22:23.259052 17681 raft_consensus.cc:2410] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 12 because replica has last-logged OpId of term: 11 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 5.
I20260709 12:22:23.259600 17681 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37402 (request call id 2521) took 1506 ms. Trace:
W20260709 12:22:23.249264 17803 leader_election.cc:336] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 election: RPC error from VoteRequest() call to peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Timed out: RequestConsensusVote RPC to 127.16.29.68:42577 timed out after 1.625s (SENT)
I20260709 12:22:23.259958 17681 rpcz_store.cc:276] 0709 12:22:21.753435 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:21.770658 (+ 17223us) service_pool.cc:224] Handling call
0709 12:22:23.257990 (+1487332us) spinlock_profiling.cc:243] Waited 18 ms on lock 0x7b5800002a98. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b474fbb71 00007f7b474fba22 00007f7b504148d9 00007f7b4f56eee0 00007f7b4f6380ab 00007f7b4f638131 00007f7b4f5e60a0 00007f7b4f5e5984 00007f7b48b606d4 00007f7b48b60662
0709 12:22:23.259542 (+  1552us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":41492864}
I20260709 12:22:23.256426 20394 leader_election.cc:290] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 9 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:22:23.263235 17803 leader_election.cc:336] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Timed out: RequestConsensusVote RPC to 127.16.29.68:42577 timed out after 1.825s (SENT)
I20260709 12:22:23.263809 17803 leader_election.cc:304] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
W20260709 12:22:23.250329 17762 rpcz_store.cc:269] 0709 12:22:21.920961 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:21.921427 (+   466us) service_pool.cc:224] Handling call
0709 12:22:23.249618 (+1328191us) spinlock_profiling.cc:243] Waited 16.6 ms on lock 0x7b54002b6454. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:22:23.249754 (+   136us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":38286080}
W20260709 12:22:23.266366 17803 leader_election.cc:336] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Timed out: RequestConsensusVote RPC to 127.16.29.68:42577 timed out after 2.240s (SENT)
I20260709 12:22:23.266569 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
W20260709 12:22:23.267433 17803 leader_election.cc:336] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Timed out: RequestConsensusVote RPC to 127.16.29.68:42577 timed out after 2.914s (SENT)
I20260709 12:22:23.268265 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:23.269793 17801 leader_election.cc:304] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:23.270153 17803 leader_election.cc:304] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:22:23.271629 18333 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:23.272959 18333 leader_election.cc:304] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:22:23.274501 18333 leader_election.cc:304] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:22:23.227998 21379 raft_consensus.cc:697] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 8 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:22:23.259151 18362 rpcz_store.cc:276] 0709 12:22:22.010289 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:22.010515 (+   226us) service_pool.cc:224] Handling call
0709 12:22:22.879302 (+868787us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:22.880232 (+   930us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:23.258581 (+378349us) spinlock_profiling.cc:243] Waited 868 ms on lock 0x7b54004687d0. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a61f8ad 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:22:23.258699 (+   118us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1997248896}
I20260709 12:22:23.276855 18333 leader_election.cc:304] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:22:23.278805 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:23.279693 17762 raft_consensus.cc:2468] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 8.
I20260709 12:22:23.284366 17801 leader_election.cc:304] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:23.285943 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:23.286258 18367 raft_consensus.cc:2468] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 9.
I20260709 12:22:23.286818 17837 raft_consensus.cc:2468] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 9.
I20260709 12:22:23.290524 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:23.292964 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:23.289480 21781 raft_consensus.cc:2804] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:22:23.294363 21781 raft_consensus.cc:493] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:23.294548 21616 raft_consensus.cc:2749] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:22:23.294895 21781 raft_consensus.cc:3060] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:23.296346 17803 leader_election.cc:304] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:23.290601 18367 raft_consensus.cc:1240] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 7. Current term is 8. Ops: []
I20260709 12:22:23.302891 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:23.303079 18367 raft_consensus.cc:3055] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:22:23.303699 18367 raft_consensus.cc:740] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:22:23.284010 21782 raft_consensus.cc:2749] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:22:23.300982 21780 consensus_queue.cc:1059] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 8.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.008s
I20260709 12:22:23.310093 18365 raft_consensus.cc:1275] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260709 12:22:23.310631 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:23.311980 21616 raft_consensus.cc:493] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:23.278342 21797 raft_consensus.cc:2749] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:22:23.304566 18367 consensus_queue.cc:260] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 13.13, Last appended by leader: 13, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:23.325982 18367 raft_consensus.cc:3060] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:23.333894 18367 raft_consensus.cc:1275] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 13 index: 13. Preceding OpId from leader: term: 14 index: 13. (term mismatch)
W20260709 12:22:23.333968 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:23.290604 21767 raft_consensus.cc:2749] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:22:23.334267 18367 pending_rounds.cc:85] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da: Aborting all ops after (but not including) 12
I20260709 12:22:23.300141 21407 raft_consensus.cc:2804] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:22:23.278447 21794 raft_consensus.cc:2749] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:22:23.338743 18367 pending_rounds.cc:107] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da: Aborting uncommitted NO_OP operation due to leader change: 13.13
I20260709 12:22:23.343976 18367 raft_consensus.cc:2889] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:23.338951 17803 leader_election.cc:304] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:23.345618 21767 raft_consensus.cc:2749] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:22:23.345712 21616 raft_consensus.cc:515] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:23.350044 21616 leader_election.cc:290] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:23.343703 21407 raft_consensus.cc:493] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:23.350872 21407 raft_consensus.cc:3060] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:22:23.358795 17655 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:23.298759 21705 raft_consensus.cc:697] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 10 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
W20260709 12:22:23.360697 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 83946.6us
I20260709 12:22:23.365975 18333 leader_election.cc:304] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:22:23.347820 21783 raft_consensus.cc:493] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:23.369473 18333 leader_election.cc:304] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
W20260709 12:22:23.360388 21754 log.cc:927] Time spent T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.063s	user 0.000s	sys 0.001s
I20260709 12:22:23.358446 21782 raft_consensus.cc:2749] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:22:23.373220 18333 leader_election.cc:304] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:22:23.384477 17728 leader_election.cc:304] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:23.386777 18333 leader_election.cc:304] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:22:23.389240 17757 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36572 (request call id 710) took 1494 ms. Trace:
I20260709 12:22:23.389786 17757 rpcz_store.cc:276] 0709 12:22:21.895086 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:21.895448 (+   362us) service_pool.cc:224] Handling call
0709 12:22:23.389036 (+1493588us) spinlock_profiling.cc:243] Waited 21.7 ms on lock 0x7b540042c054. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:22:23.389168 (+   132us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":49848960}
I20260709 12:22:23.421626 21781 raft_consensus.cc:515] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:23.422564 21504 raft_consensus.cc:2804] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:22:23.424113 21504 raft_consensus.cc:493] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:23.424571 21504 raft_consensus.cc:3060] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:22:23.427213 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:23.423274 17832 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35040 (request call id 2261) took 2185 ms. Trace:
I20260709 12:22:23.376065 21616 raft_consensus.cc:2749] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260709 12:22:23.433055 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 94190us
I20260709 12:22:23.428399 17832 rpcz_store.cc:276] 0709 12:22:21.238134 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:21.689031 (+450897us) service_pool.cc:224] Handling call
0709 12:22:21.693090 (+  4059us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:22:21.693503 (+   413us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:21.693532 (+    29us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:22:21.701207 (+  7675us) log.cc:844] Serialized 52 byte log entry
0709 12:22:21.790087 (+ 88880us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:22:21.820262 (+ 30175us) log.cc:844] Serialized 14 byte log entry
0709 12:22:21.834283 (+ 14021us) log.cc:844] Serialized 14 byte log entry
0709 12:22:21.834545 (+   262us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:21.834642 (+    97us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:23.296657 (+1462015us) spinlock_profiling.cc:243] Waited 4.59 ms on lock 0x7b1800393880. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b475333f3 00007f7b475357c1 00007f7b47535779 00007f7b47535709 00007f7b475356d1 00007f7b475349cd 0000000000388634 0000000000388479 00007f7b4751683a
0709 12:22:23.423021 (+126364us) raft_consensus.cc:1685] finished
0709 12:22:23.423044 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:23.423214 (+   170us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":9987,"prepare.queue_time_us":79074,"prepare.run_cpu_time_us":70,"prepare.run_wall_time_us":74,"raft.queue_time_us":9225,"raft.run_cpu_time_us":225,"raft.run_wall_time_us":16528,"spinlock_wait_cycles":39571584,"thread_start_us":84492,"threads_started":1,"wal-append.queue_time_us":168720}
I20260709 12:22:23.438007 17803 leader_election.cc:304] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:23.440369 17803 leader_election.cc:304] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:23.442052 17803 leader_election.cc:304] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:23.375663 21805 raft_consensus.cc:2804] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:22:23.387312 21794 raft_consensus.cc:2749] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:22:23.463384 17803 leader_election.cc:304] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:23.464871 17803 leader_election.cc:304] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:23.466114 17803 leader_election.cc:304] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:23.390666 21767 raft_consensus.cc:493] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:23.385794 21398 consensus_queue.cc:1048] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.047s
I20260709 12:22:23.372973 21379 consensus_queue.cc:237] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:23.470387 21780 raft_consensus.cc:2749] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:22:23.479974 21705 consensus_queue.cc:237] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:23.480535 21780 raft_consensus.cc:2749] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:22:23.481261 21780 raft_consensus.cc:2749] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:22:23.482188 21493 raft_consensus.cc:993] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3: : Instructing follower e063541542a94969898868bf6eb5ce1b to start an election
I20260709 12:22:23.482651 21493 raft_consensus.cc:1081] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 7 LEADER]: Signalling peer e063541542a94969898868bf6eb5ce1b to start an election
W20260709 12:22:23.486634 21776 log.cc:927] Time spent T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.625s	user 0.001s	sys 0.000s
I20260709 12:22:23.488067 21504 raft_consensus.cc:515] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:23.492815 21785 raft_consensus.cc:515] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:23.470207 21782 raft_consensus.cc:3060] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:22:23.496752 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 367226us
I20260709 12:22:23.497498 17802 leader_election.cc:304] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:23.498778 17802 leader_election.cc:304] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:23.499641 21780 raft_consensus.cc:2804] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:22:23.500041 21780 raft_consensus.cc:493] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:23.500370 21780 raft_consensus.cc:3060] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:23.500994 21781 leader_election.cc:290] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 11 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:23.501631 21781 raft_consensus.cc:493] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:23.502076 21781 raft_consensus.cc:515] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:23.504128 21807 raft_consensus.cc:2749] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:22:23.506489 17802 leader_election.cc:304] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:23.374692 21783 raft_consensus.cc:515] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:23.435866 21772 raft_consensus.cc:493] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:23.387466 21779 consensus_queue.cc:1048] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.084s
I20260709 12:22:23.505671 21616 raft_consensus.cc:2749] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:22:23.470019 21804 raft_consensus.cc:2749] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:22:23.509167 21807 raft_consensus.cc:2749] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:22:23.508172 21785 leader_election.cc:290] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:23.509156 21053 consensus_queue.cc:1048] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.012s
I20260709 12:22:23.472312 21767 raft_consensus.cc:515] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:23.511235 21772 raft_consensus.cc:515] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:23.516439 21781 leader_election.cc:290] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:22:23.539228 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:23.539554 21780 raft_consensus.cc:515] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:23.539999 21804 consensus_queue.cc:1048] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:22:23.542510 21630 raft_consensus.cc:493] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:23.542860 21780 leader_election.cc:290] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 12 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:23.543107 21630 raft_consensus.cc:515] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:23.545861 21630 leader_election.cc:290] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:23.546214 21767 leader_election.cc:290] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:23.549881 21772 leader_election.cc:290] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:23.551689 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 11 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:22:23.533864 21799 log.cc:927] Time spent T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.061s	user 0.001s	sys 0.000s
I20260709 12:22:23.552598 18371 raft_consensus.cc:2410] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for term 11 because replica has last-logged OpId of term: 9 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 8.
I20260709 12:22:23.553119 21407 raft_consensus.cc:515] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:23.553954 21783 leader_election.cc:290] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:23.534183 21807 raft_consensus.cc:2749] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:22:23.556766 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:23.557448 17759 raft_consensus.cc:3060] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:23.581485 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:23.533106 21785 consensus_queue.cc:1048] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.034s
I20260709 12:22:23.582178 18371 raft_consensus.cc:2468] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 7.
W20260709 12:22:23.582090 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 195402us
I20260709 12:22:23.584632 17682 raft_consensus.cc:1275] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 8 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260709 12:22:23.591382 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 12 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:23.585122 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:23.592379 17680 raft_consensus.cc:2468] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 11.
I20260709 12:22:23.592468 17681 raft_consensus.cc:2393] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 10: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
W20260709 12:22:23.585325 17755 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38786 (request call id 2746) took 1393 ms (client timeout 1765 ms). Trace:
W20260709 12:22:23.593101 17755 rpcz_store.cc:269] 0709 12:22:22.192159 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:22.209402 (+ 17243us) service_pool.cc:224] Handling call
0709 12:22:23.585163 (+1375761us) spinlock_profiling.cc:243] Waited 661 ms on lock 0x7b54000fc8d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:22:23.585271 (+   108us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1520625024}
I20260709 12:22:23.589192 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:23.589962 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:23.595512 17684 raft_consensus.cc:2468] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 9.
I20260709 12:22:23.590303 17687 raft_consensus.cc:1275] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 10 index: 11. Preceding OpId from leader: term: 14 index: 13. (index mismatch)
I20260709 12:22:23.597436 18333 leader_election.cc:304] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:22:23.595624 17686 raft_consensus.cc:2468] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 9.
I20260709 12:22:23.599444 18333 leader_election.cc:304] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:22:23.600255 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:23.601173 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 11 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:23.601189 18333 leader_election.cc:304] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:22:23.605662 17759 raft_consensus.cc:2468] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 10.
I20260709 12:22:23.607757 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:23.582389 21504 leader_election.cc:290] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 11 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:23.607954 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 12 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:23.608422 17682 raft_consensus.cc:3060] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:23.609448 17755 raft_consensus.cc:1240] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 9. Current term is 10. Ops: []
I20260709 12:22:23.573400 21477 raft_consensus.cc:493] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:22:23.626271 18333 leader_election.cc:400] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 8 election: Vote denied by peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837) with higher term. Message: Invalid argument: T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 8. Current term is 9.
I20260709 12:22:23.625949 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:23.628101 18333 leader_election.cc:403] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 8 election: Cancelling election due to peer responding with higher term
I20260709 12:22:23.626936 17685 raft_consensus.cc:1240] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 8 LEADER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 7. Current term is 8. Ops: []
I20260709 12:22:23.630282 18333 leader_election.cc:304] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:22:23.600901 21781 raft_consensus.cc:2749] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260709 12:22:23.633476 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 167432us
I20260709 12:22:23.634006 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:23.635211 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:23.635486 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:23.636008 18365 raft_consensus.cc:3060] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:23.636587 17803 leader_election.cc:304] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:23.610033 21649 raft_consensus.cc:2804] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:22:23.602684 21667 raft_consensus.cc:493] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:22:23.626231 21616 raft_consensus.cc:2749] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
W20260709 12:22:23.645735 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 136414us
I20260709 12:22:23.646531 17802 leader_election.cc:304] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:23.627913 21407 leader_election.cc:290] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 8 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:23.632787 21794 raft_consensus.cc:2749] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Leader election lost for term 8. Reason: Vote denied by peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837) with higher term. Message: Invalid argument: T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 8. Current term is 9.
I20260709 12:22:23.652532 18333 leader_election.cc:304] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:22:23.653719 18365 raft_consensus.cc:2468] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 11.
W20260709 12:22:23.655238 18334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:23.655750 18334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:23.658707 18334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:23.661763 18333 leader_election.cc:304] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:22:23.663098 18333 leader_election.cc:400] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 7 pre-election: Vote denied by peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651) with higher term. Message: Invalid argument: T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 7. Current term is 8.
I20260709 12:22:23.663520 18333 leader_election.cc:403] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 7 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:22:23.643326 21649 raft_consensus.cc:493] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
W20260709 12:22:23.654632 21684 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.881s	user 0.007s	sys 0.017s
I20260709 12:22:23.670044 17681 raft_consensus.cc:1240] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 13. Current term is 14. Ops: []
I20260709 12:22:23.673274 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 12 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:23.673592 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:23.673930 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:23.673300 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:23.674598 17829 raft_consensus.cc:3060] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:23.648754 21667 raft_consensus.cc:515] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:23.675088 17836 raft_consensus.cc:2393] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 10: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:22:23.667558 21649 raft_consensus.cc:3060] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:23.647917 21785 raft_consensus.cc:2749] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:22:23.688689 17829 raft_consensus.cc:2468] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 9.
I20260709 12:22:23.689522 17830 raft_consensus.cc:1240] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 8. Current term is 9. Ops: []
I20260709 12:22:23.611488 21779 raft_consensus.cc:493] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:23.637305 21477 raft_consensus.cc:515] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:23.697801 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:23.698081 17833 raft_consensus.cc:1240] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 7. Current term is 8. Ops: []
I20260709 12:22:23.699005 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:23.699137 21772 consensus_queue.cc:1048] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:22:23.699007 21477 leader_election.cc:290] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:23.693327 17830 raft_consensus.cc:1275] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260709 12:22:23.693656 18365 raft_consensus.cc:1275] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
I20260709 12:22:23.694345 17834 raft_consensus.cc:1240] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 11. Current term is 12. Ops: [11.8-11.8]
I20260709 12:22:23.704003 21814 raft_consensus.cc:2749] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Leader pre-election lost for term 7. Reason: Vote denied by peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651) with higher term. Message: Invalid argument: T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 7. Current term is 8.
I20260709 12:22:23.659759 21781 raft_consensus.cc:2804] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:22:23.705222 21781 raft_consensus.cc:697] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 9 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:22:23.705813 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:23.706104 21781 consensus_queue.cc:237] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:23.706693 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:23.706712 17834 raft_consensus.cc:2468] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 10.
I20260709 12:22:23.704300 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:23.709995 21779 raft_consensus.cc:515] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:23.717428 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 11 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:23.719475 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 11 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:23.723166 17685 raft_consensus.cc:1275] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260709 12:22:23.725924 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:23.726943 17688 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37402 (request call id 2527) took 1971 ms. Trace:
I20260709 12:22:23.656700 21808 raft_consensus.cc:2804] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:22:23.696398 17728 leader_election.cc:304] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:23.727386 17688 rpcz_store.cc:276] 0709 12:22:21.755018 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:21.782244 (+ 27226us) service_pool.cc:224] Handling call
0709 12:22:23.725715 (+1943471us) spinlock_profiling.cc:243] Waited 2.24 ms on lock 0x7b5800002a98. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b474fbb71 00007f7b474fba22 00007f7b504148d9 00007f7b4f56eee0 00007f7b4f6380ab 00007f7b4f638131 00007f7b4f5e60a0 00007f7b4f5e5984 00007f7b48b606d4 00007f7b48b60662
0709 12:22:23.726867 (+  1152us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5146496}
I20260709 12:22:23.707355 17832 raft_consensus.cc:3060] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:22:23.690923 21684 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.917s	user 0.008s	sys 0.018s
I20260709 12:22:23.730343 17728 leader_election.cc:304] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:23.669796 21616 raft_consensus.cc:493] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:23.692377 21794 raft_consensus.cc:2804] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Leader pre-election won for term 10
W20260709 12:22:23.768494 17655 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:23.851079 17727 leader_election.cc:304] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:23.762807 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:22:23.856591 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 126360us
I20260709 12:22:23.773826 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:22:23.734580 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 118112us
I20260709 12:22:23.864899 17802 leader_election.cc:304] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:23.864861 17688 raft_consensus.cc:2393] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 8: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:22:23.866293 17728 leader_election.cc:304] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:23.689292 21407 raft_consensus.cc:2749] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20260709 12:22:23.878158 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:23.734131 21808 raft_consensus.cc:493] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:23.735211 21708 raft_consensus.cc:2804] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:22:23.864336 21687 raft_consensus.cc:2804] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:22:23.866434 21624 raft_consensus.cc:2804] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:22:23.865417 21794 raft_consensus.cc:493] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:23.866806 21398 consensus_queue.cc:1059] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 8.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.025s
I20260709 12:22:23.885056 21687 raft_consensus.cc:493] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:23.881819 21808 raft_consensus.cc:3060] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:23.870301 21780 raft_consensus.cc:2749] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:22:23.891928 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:23.908424 17685 raft_consensus.cc:2468] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 8.
I20260709 12:22:23.870432 21685 consensus_queue.cc:1048] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.026s
I20260709 12:22:23.870139 21767 consensus_queue.cc:1048] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.027s
I20260709 12:22:23.867677 21616 raft_consensus.cc:515] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:23.898048 21564 consensus_queue.cc:1048] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.007s
I20260709 12:22:23.900285 21624 raft_consensus.cc:697] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:22:23.906013 21786 consensus_queue.cc:1059] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 9.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 12:22:23.871665 21778 consensus_queue.cc:1059] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.030s
I20260709 12:22:23.882957 21708 raft_consensus.cc:493] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:22:23.886081 17728 leader_election.cc:304] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:22:23.900462 21788 raft_consensus.cc:2804] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Leader pre-election won for term 10
W20260709 12:22:23.914366 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 251424us
I20260709 12:22:23.897563 21794 raft_consensus.cc:3060] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:23.910179 21687 raft_consensus.cc:3060] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:23.924556 18333 leader_election.cc:304] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:22:23.934245 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 48256.2us
I20260709 12:22:23.935093 18333 leader_election.cc:304] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:23.917578 21708 raft_consensus.cc:3060] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:23.943338 18333 leader_election.cc:304] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:23.944912 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:23.953727 18365 raft_consensus.cc:2376] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 8. Re-sending same reply.
I20260709 12:22:23.921772 21788 raft_consensus.cc:493] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:22:23.955222 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:22:23.955403 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 54400us
I20260709 12:22:23.965749 21546 consensus_queue.cc:1048] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:22:23.937454 21505 raft_consensus.cc:3055] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:22:23.968106 21505 raft_consensus.cc:740] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:22:23.968020 21370 consensus_queue.cc:1059] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:22:23.968948 21370 consensus_queue.cc:1243] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer 0da169ff82674dc1947055ce43de68a3 log is divergent from this leader: its last log entry 8.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260709 12:22:23.969220 21505 consensus_queue.cc:260] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:23.970357 21505 raft_consensus.cc:3060] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:22:23.972412 17803 leader_election.cc:336] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Timed out: RequestConsensusVote RPC to 127.16.29.68:42577 timed out after 2.228s (SENT)
I20260709 12:22:23.932410 21667 leader_election.cc:290] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:23.941362 21778 raft_consensus.cc:2749] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:22:23.933277 21781 raft_consensus.cc:2749] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:22:23.938916 21398 raft_consensus.cc:3055] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:22:23.975020 17682 raft_consensus.cc:2468] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 11.
I20260709 12:22:23.939512 21782 raft_consensus.cc:2749] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:22:23.985195 17803 leader_election.cc:400] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 7 pre-election: Vote denied by peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577) with higher term. Message: Invalid argument: T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 7. Current term is 8.
I20260709 12:22:23.985759 17803 leader_election.cc:403] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 7 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:22:23.955796 21814 raft_consensus.cc:2749] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:22:23.959527 21788 raft_consensus.cc:3060] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:23.931245 21477 consensus_queue.cc:1048] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 7, Time since last communication: 0.047s
I20260709 12:22:23.935351 21493 consensus_queue.cc:1048] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.042s
W20260709 12:22:23.992810 17729 consensus_peers.cc:597] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb -> Peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Couldn't send request to peer 334a4e1f59944544b3b3ff36616b57da. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d48b7de191684318aa08f60a7a395387. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:22:23.996476 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 56674.4us
W20260709 12:22:23.997517 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 71931.6us
I20260709 12:22:23.998061 17729 leader_election.cc:304] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:24.002871 18366 raft_consensus.cc:1240] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 8. Current term is 9. Ops: []
I20260709 12:22:23.978220 21398 raft_consensus.cc:740] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:22:24.005892 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:24.011293 17729 leader_election.cc:304] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:22:24.011631 18365 raft_consensus.cc:2393] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 10: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:22:24.006583 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:24.012609 18367 raft_consensus.cc:2468] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 9.
I20260709 12:22:24.008946 17803 leader_election.cc:304] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:24.014335 17803 leader_election.cc:304] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:24.021607 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:24.022946 18374 raft_consensus.cc:2376] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 8. Re-sending same reply.
W20260709 12:22:24.023463 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 63263.1us
I20260709 12:22:24.023265 21780 raft_consensus.cc:2749] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:22:24.032472 21824 raft_consensus.cc:3060] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:23.992522 21820 raft_consensus.cc:2749] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:22:24.005532 21781 raft_consensus.cc:493] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:24.036672 21781 raft_consensus.cc:515] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:24.038733 21781 leader_election.cc:290] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:24.021961 21800 consensus_queue.cc:1048] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.018s
W20260709 12:22:24.040078 21813 log.cc:927] Time spent T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.156s	user 0.001s	sys 0.000s
I20260709 12:22:24.040900 21616 leader_election.cc:290] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:24.041874 21772 consensus_queue.cc:1059] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 12:22:24.024854 21806 consensus_queue.cc:1048] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 7, Time since last communication: 0.017s
I20260709 12:22:24.042699 21806 consensus_queue.cc:1243] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer 334a4e1f59944544b3b3ff36616b57da log is divergent from this leader: its last log entry 9.9 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
I20260709 12:22:24.042097 21398 consensus_queue.cc:260] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.8, Last appended by leader: 8, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:24.045317 21616 raft_consensus.cc:493] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:24.045866 21616 raft_consensus.cc:515] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:24.004664 21767 raft_consensus.cc:2749] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Leader pre-election lost for term 7. Reason: Vote denied by peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577) with higher term. Message: Invalid argument: T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 7. Current term is 8.
I20260709 12:22:24.003917 21783 raft_consensus.cc:2749] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:22:24.023679 21624 consensus_queue.cc:237] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:24.043653 21398 raft_consensus.cc:3060] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:24.046183 21779 leader_election.cc:290] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:24.053279 17832 raft_consensus.cc:2468] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 8.
I20260709 12:22:24.058859 21674 leader_election.cc:290] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 10 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:24.060966 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:24.061978 17802 leader_election.cc:304] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:24.063079 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:24.064133 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:24.064749 17685 raft_consensus.cc:2393] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 8: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
W20260709 12:22:24.066484 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 131446us
I20260709 12:22:24.066726 17686 raft_consensus.cc:2393] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 10: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:22:24.067361 17802 leader_election.cc:304] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:22:24.069058 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 69929.5us
I20260709 12:22:24.070801 17803 leader_election.cc:304] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:22:24.071662 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:24.072912 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 12 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:24.022814 21782 raft_consensus.cc:2749] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:22:24.074426 17803 leader_election.cc:304] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:24.071626 21806 raft_consensus.cc:2749] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 12:22:24.076655 21564 consensus_queue.cc:1048] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 12:22:24.078209 21808 raft_consensus.cc:515] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:24.078691 21407 consensus_queue.cc:1059] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 8.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.006s
I20260709 12:22:24.075989 21785 raft_consensus.cc:2804] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:22:24.080673 21779 consensus_queue.cc:1048] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 12:22:24.082142 21687 raft_consensus.cc:515] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:24.084031 17729 leader_election.cc:304] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:22:24.126530 21801 log.cc:927] Time spent T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.637s	user 0.001s	sys 0.000s
I20260709 12:22:24.111236 17727 leader_election.cc:304] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:22:24.129453 21808 leader_election.cc:290] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 9 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:24.129781 21779 raft_consensus.cc:2804] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:22:24.129956 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:24.130559 21687 leader_election.cc:290] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 11 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:24.130652 21778 raft_consensus.cc:2749] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:22:24.083344 21794 raft_consensus.cc:515] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:24.131598 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:24.118012 17689 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37402 (request call id 2530) took 2362 ms. Trace:
I20260709 12:22:24.133270 17755 raft_consensus.cc:2393] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 10: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:22:24.133415 17727 leader_election.cc:304] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
W20260709 12:22:24.129106 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 45233.3us
I20260709 12:22:24.133461 17689 rpcz_store.cc:276] 0709 12:22:21.755818 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:21.868159 (+112341us) service_pool.cc:224] Handling call
0709 12:22:21.910947 (+ 42788us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:24.117596 (+2206649us) spinlock_profiling.cc:243] Waited 691 ms on lock 0x7b540030da20. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a59b858 00007f7b4a61fe9d 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c
0709 12:22:24.117653 (+    57us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 12:22:24.117679 (+    26us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:24.117720 (+    41us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:22:24.117770 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:24.117846 (+    76us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:24.117954 (+   108us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":921,"raft.queue_time_us":20963,"raft.run_cpu_time_us":114,"raft.run_wall_time_us":2768,"spinlock_wait_cycles":1589051776}
I20260709 12:22:24.135139 17729 leader_election.cc:304] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:24.135116 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:24.132148 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 12 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:24.137252 21777 raft_consensus.cc:2804] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:22:24.137777 21777 raft_consensus.cc:697] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 8 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
W20260709 12:22:24.125471 21803 log.cc:927] Time spent T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.652s	user 0.001s	sys 0.000s
I20260709 12:22:24.132385 21630 raft_consensus.cc:493] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:22:24.138633 21777 consensus_queue.cc:237] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:24.138878 21630 raft_consensus.cc:515] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:24.137880 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:24.140405 18374 raft_consensus.cc:2468] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 10.
W20260709 12:22:24.140750 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 66369.4us
I20260709 12:22:24.142997 18374 raft_consensus.cc:1217] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Deduplicated request from leader. Original: 5.7->[7.8-10.10]   Dedup: 7.8->[8.9-10.10]
I20260709 12:22:24.143519 18374 pending_rounds.cc:85] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da: Aborting all ops after (but not including) 8
I20260709 12:22:24.144164 18374 pending_rounds.cc:107] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da: Aborting uncommitted NO_OP operation due to leader change: 9.9
I20260709 12:22:24.144604 18374 raft_consensus.cc:2889] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:24.007642 21649 raft_consensus.cc:515] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:24.089236 17802 leader_election.cc:304] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:24.144589 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:24.131422 21779 raft_consensus.cc:697] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
W20260709 12:22:24.141342 17652 connection.cc:256] RPC call timeout handler was delayed by -1.0086820090000401s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21989999999999998s for a response
I20260709 12:22:24.150683 21778 raft_consensus.cc:2749] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:22:24.162962 17654 leader_election.cc:304] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:22:24.150887 21770 raft_consensus.cc:2749] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
W20260709 12:22:24.151816 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 62684.6us
I20260709 12:22:24.168959 17729 leader_election.cc:304] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:22:24.155049 17727 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:24.157349 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:24.132586 21785 raft_consensus.cc:493] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:24.172488 17762 raft_consensus.cc:2468] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 8.
I20260709 12:22:24.158066 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:24.158744 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:24.159453 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:24.174525 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:24.166910 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:24.178010 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:24.071863 21780 raft_consensus.cc:2804] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:22:24.072306 21616 leader_election.cc:290] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:24.178575 17837 raft_consensus.cc:3060] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:24.177302 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:24.182166 17830 raft_consensus.cc:1240] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 7. Current term is 8. Ops: [7.6-7.6]
I20260709 12:22:24.188166 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:24.189361 18333 leader_election.cc:304] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:22:24.190407 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:24.191839 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:24.192600 17686 raft_consensus.cc:3060] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Advancing to term 11
I20260709 12:22:24.188827 17762 raft_consensus.cc:2468] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 9.
I20260709 12:22:24.150321 21806 raft_consensus.cc:2804] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:22:24.222980 18333 leader_election.cc:304] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:24.223992 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:24.180966 21624 raft_consensus.cc:2804] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:22:24.185611 21477 raft_consensus.cc:2804] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:22:24.179538 21785 raft_consensus.cc:3060] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:24.118664 21788 raft_consensus.cc:515] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:24.235823 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 9 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:24.233327 18333 leader_election.cc:304] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:24.237475 17686 raft_consensus.cc:2468] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 11.
I20260709 12:22:24.179492 21780 raft_consensus.cc:493] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:24.239156 17837 raft_consensus.cc:2468] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 11.
I20260709 12:22:24.216741 21772 raft_consensus.cc:493] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:24.244015 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:24.172724 21589 consensus_queue.cc:1059] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.061s
I20260709 12:22:24.244068 18333 leader_election.cc:304] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:24.232501 21616 raft_consensus.cc:2804] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:22:24.187459 21501 consensus_queue.cc:1048] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.055s
I20260709 12:22:24.233539 21808 raft_consensus.cc:2749] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:22:24.254853 17755 raft_consensus.cc:2393] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 10: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:22:24.235313 21806 raft_consensus.cc:697] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 8 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:22:24.236752 21624 raft_consensus.cc:493] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:24.239435 21477 raft_consensus.cc:493] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:24.246310 21780 raft_consensus.cc:3060] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:24.244865 21820 raft_consensus.cc:2749] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260709 12:22:24.239557 21805 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.744s	user 0.014s	sys 0.016s
I20260709 12:22:24.268476 21794 leader_election.cc:290] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 10 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:24.251021 21616 raft_consensus.cc:493] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:24.255182 21644 raft_consensus.cc:2804] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:22:24.260659 21781 raft_consensus.cc:493] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:24.273768 21624 raft_consensus.cc:3060] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:24.280330 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
W20260709 12:22:24.278301 21805 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.783s	user 0.014s	sys 0.017s
I20260709 12:22:24.275565 21477 raft_consensus.cc:3060] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:24.251196 21772 raft_consensus.cc:515] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:24.297709 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:24.281068 21616 raft_consensus.cc:3060] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:24.289605 21805 raft_consensus.cc:697] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 8 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:22:24.304646 21808 raft_consensus.cc:493] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:24.283741 21644 raft_consensus.cc:493] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:24.306063 21824 raft_consensus.cc:2749] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:22:24.310211 21788 leader_election.cc:290] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 10 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:22:24.310959 21825 log.cc:927] Time spent T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.143s	user 0.000s	sys 0.000s
I20260709 12:22:24.311142 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:24.338558 17652 leader_election.cc:304] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:24.325978 21805 consensus_queue.cc:237] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:24.255915 21779 consensus_queue.cc:237] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:24.356220 17801 leader_election.cc:304] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:22:24.357390 17801 leader_election.cc:304] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:24.327235 21808 raft_consensus.cc:515] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:24.359476 21708 raft_consensus.cc:515] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:24.359972 21808 leader_election.cc:290] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:24.360551 21808 raft_consensus.cc:493] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:24.289031 21781 raft_consensus.cc:515] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:23.087621 16501 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:22:24.316753 21692 raft_consensus.cc:3055] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:22:24.332414 21644 raft_consensus.cc:3060] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:24.343930 17586 catalog_manager.cc:5697] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b reported cstate change: term changed from 2 to 5. New cstate: current_term: 5 leader_uuid: "334a4e1f59944544b3b3ff36616b57da" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
W20260709 12:22:24.344583 21832 log.cc:927] Time spent T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.114s	user 0.000s	sys 0.001s
I20260709 12:22:24.344988 21800 raft_consensus.cc:2749] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 12:22:24.346451 21772 leader_election.cc:290] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:24.350823 17756 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38776 (request call id 2071) took 2287 ms. Trace:
I20260709 12:22:24.354514 17689 pending_rounds.cc:85] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3: Aborting all ops after (but not including) 5
I20260709 12:22:24.395924 17689 pending_rounds.cc:107] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3: Aborting uncommitted NO_OP operation due to leader change: 9.6
I20260709 12:22:24.396232 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:24.396607 17689 raft_consensus.cc:2889] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:24.397154 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 9 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:24.397969 17837 raft_consensus.cc:2468] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 8.
I20260709 12:22:24.398109 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:24.361054 21782 raft_consensus.cc:2804] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:22:24.393627 17756 rpcz_store.cc:276] 0709 12:22:22.063166 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:22.889766 (+826600us) service_pool.cc:224] Handling call
0709 12:22:23.249745 (+359979us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:23.250419 (+   674us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:24.350620 (+1100201us) spinlock_profiling.cc:243] Waited 359 ms on lock 0x7b54002b6450. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a61f8ad 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:22:24.350759 (+   139us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":825625984}
I20260709 12:22:24.395445 17586 catalog_manager.cc:5697] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b reported cstate change: term changed from 5 to 6, leader changed from 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) to e063541542a94969898868bf6eb5ce1b (127.16.29.67). New cstate: current_term: 6 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: HEALTHY } } }
I20260709 12:22:24.401587 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:24.406440 17837 raft_consensus.cc:3060] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:24.407160 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:24.407644 17681 raft_consensus.cc:3060] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:24.402441 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:24.409276 17835 raft_consensus.cc:3060] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:24.361635 21649 leader_election.cc:290] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 8 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:24.404239 17836 raft_consensus.cc:1275] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260709 12:22:24.404167 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:24.363902 21770 raft_consensus.cc:2749] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 12:22:24.368448 21692 raft_consensus.cc:740] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:22:24.403059 18333 leader_election.cc:304] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:22:24.416946 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:24.423045 21842 consensus_queue.cc:1048] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 12:22:24.423913 21674 consensus_queue.cc:1048] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 12:22:24.424026 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:24.419584 17652 leader_election.cc:304] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:24.451011 17763 raft_consensus.cc:2468] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 8.
W20260709 12:22:24.431871 17655 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:24.432300 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:24.453347 17681 raft_consensus.cc:2468] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 12.
I20260709 12:22:24.431133 17832 raft_consensus.cc:3060] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:24.434211 17837 raft_consensus.cc:2468] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 9.
I20260709 12:22:24.454370 17835 raft_consensus.cc:2468] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 8.
W20260709 12:22:24.457398 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 54444.4us
W20260709 12:22:24.460994 21761 log.cc:927] Time spent T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 1.979s	user 0.003s	sys 0.004s
I20260709 12:22:24.470659 18361 raft_consensus.cc:1240] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 7. Current term is 8. Ops: []
I20260709 12:22:24.361114 21808 raft_consensus.cc:515] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:22:24.475297 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 55833.2us
I20260709 12:22:24.476730 18361 raft_consensus.cc:1240] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 10. Current term is 11. Ops: [10.9-10.9]
I20260709 12:22:24.489250 18363 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 1287) took 2682 ms. Trace:
I20260709 12:22:24.481779 17586 catalog_manager.cc:5697] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b reported cstate change: term changed from 5 to 8, leader changed from e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66) to e063541542a94969898868bf6eb5ce1b (127.16.29.67). New cstate: current_term: 8 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:22:24.489642 18363 rpcz_store.cc:276] 0709 12:22:21.795292 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:21.809506 (+ 14214us) service_pool.cc:224] Handling call
0709 12:22:21.845432 (+ 35926us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:21.871535 (+ 26103us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:21.871571 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:21.880212 (+  8641us) log.cc:844] Serialized 26 byte log entry
0709 12:22:22.073177 (+192965us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:22:22.073229 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:22.073522 (+   293us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:24.460431 (+2386909us) spinlock_profiling.cc:243] Waited 38.2 ms on lock 0x7b4400022b4c. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b47459c85 00007f7b4745a6a2 00007f7b4a1cc1db 00007f7b4a1c7aaf 00007f7b4a1bb86e 00007f7b4a1babb6 00007f7b4a1ba7d8 00007f7b4a1c4221 00007f7b4a1c41d9 00007f7b4a1c4169
0709 12:22:24.466778 (+  6347us) raft_consensus.cc:1685] finished
0709 12:22:24.466799 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:24.477160 (+ 10361us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":23237,"raft.run_cpu_time_us":174,"raft.run_wall_time_us":6723,"spinlock_wait_cycles":87918976,"thread_start_us":189217,"threads_started":1,"wal-append.queue_time_us":397818}
I20260709 12:22:24.508018 18364 raft_consensus.cc:1240] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 8. Current term is 9. Ops: [8.9-8.9]
W20260709 12:22:24.474076 21799 log.cc:927] Time spent T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.057s	user 0.000s	sys 0.000s
I20260709 12:22:24.511277 17832 raft_consensus.cc:2468] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 11.
W20260709 12:22:24.512037 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 109243us
I20260709 12:22:24.512359 17586 catalog_manager.cc:5697] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b reported cstate change: term changed from 7 to 8, leader changed from 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) to e063541542a94969898868bf6eb5ce1b (127.16.29.67). New cstate: current_term: 8 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:22:24.473889 21649 raft_consensus.cc:2804] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:22:24.339552 21806 consensus_queue.cc:237] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:24.474464 21708 leader_election.cc:290] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 12 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:24.516685 18333 leader_election.cc:304] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:24.523655 17586 catalog_manager.cc:5697] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b reported cstate change: term changed from 6 to 7, leader changed from 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) to e063541542a94969898868bf6eb5ce1b (127.16.29.67). New cstate: current_term: 7 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:22:24.526142 18333 leader_election.cc:304] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:22:24.508070 21843 ts_tablet_manager.cc:933] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da: Initiating tablet copy from peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:24.481690 21845 raft_consensus.cc:2749] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:22:24.527545 18333 leader_election.cc:304] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:22:24.473685 21644 raft_consensus.cc:515] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:24.474370 21624 raft_consensus.cc:515] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:24.541627 18333 leader_election.cc:304] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:22:24.532467 17586 catalog_manager.cc:5697] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b reported cstate change: term changed from 8 to 9, leader changed from 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) to e063541542a94969898868bf6eb5ce1b (127.16.29.67). New cstate: current_term: 9 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: HEALTHY } } }
I20260709 12:22:24.521869 21649 raft_consensus.cc:493] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
W20260709 12:22:24.540417 17575 thread.cc:641] dir /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599551912677-16501-0/minicluster-data/master-0-root/data [worker] (thread pool) Time spent creating pthread: real 1.624s	user 0.031s	sys 0.030s
I20260709 12:22:24.533815 21820 raft_consensus.cc:2749] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:22:24.541405 21805 raft_consensus.cc:2804] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:22:24.574434 18362 raft_consensus.cc:1240] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 9. Current term is 10. Ops: [9.9-9.9]
I20260709 12:22:24.548703 21814 raft_consensus.cc:2804] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:22:24.519263 21692 consensus_queue.cc:260] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.6, Last appended by leader: 6, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:22:24.572906 17575 thread.cc:608] dir /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599551912677-16501-0/minicluster-data/master-0-root/data [worker] (thread pool) Time spent starting thread: real 1.717s	user 0.032s	sys 0.030s
I20260709 12:22:24.548043 21782 raft_consensus.cc:697] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 10 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:22:24.568172 21649 raft_consensus.cc:3060] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:22:24.600193 17583 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35958 (request call id 411) took 16967 ms (client timeout 15000 ms). Trace:
I20260709 12:22:24.588653 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:24.540804 21480 consensus_queue.cc:1059] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.036s
W20260709 12:22:24.576419 21838 log.cc:927] Time spent T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.163s	user 0.000s	sys 0.001s
W20260709 12:22:24.610435 21758 log.cc:927] Time spent T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 2.416s	user 0.002s	sys 0.011s
I20260709 12:22:24.546626 21835 consensus_queue.cc:1048] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.058s
I20260709 12:22:24.569128 21674 consensus_queue.cc:1059] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 9.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.031s
I20260709 12:22:24.616026 17586 catalog_manager.cc:5697] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b reported cstate change: term changed from 5 to 6, leader changed from 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) to e063541542a94969898868bf6eb5ce1b (127.16.29.67). New cstate: current_term: 6 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:22:24.623207 17764 raft_consensus.cc:1240] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 9. Current term is 10. Ops: []
I20260709 12:22:24.622718 17586 catalog_manager.cc:5697] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b reported cstate change: term changed from 7 to 8, leader changed from 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) to e063541542a94969898868bf6eb5ce1b (127.16.29.67). New cstate: current_term: 8 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:22:24.619331 21692 raft_consensus.cc:3060] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:22:24.628446 21840 log.cc:927] Time spent T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.219s	user 0.000s	sys 0.001s
I20260709 12:22:24.629987 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:22:24.630563 21847 log.cc:927] Time spent T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.118s	user 0.001s	sys 0.000s
I20260709 12:22:24.630681 17835 raft_consensus.cc:2368] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for earlier term 10. Current term is 11.
I20260709 12:22:24.600687 21820 raft_consensus.cc:2804] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:22:24.631482 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 12 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:24.631687 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:24.633339 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:24.633605 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:22:24.633927 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 92399.8us
I20260709 12:22:24.634416 17837 raft_consensus.cc:2468] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 7.
I20260709 12:22:24.613684 21480 raft_consensus.cc:3055] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:22:24.639775 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:24.647253 17586 catalog_manager.cc:5697] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b reported cstate change: term changed from 4 to 8. New cstate: current_term: 8 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
W20260709 12:22:24.667668 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 186784us
I20260709 12:22:24.675284 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:24.615391 21781 leader_election.cc:290] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:24.685590 21782 consensus_queue.cc:237] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:24.639819 21480 raft_consensus.cc:740] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:22:24.723026 17586 catalog_manager.cc:5697] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b reported cstate change: term changed from 2 to 7, leader changed from 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) to e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66). New cstate: current_term: 7 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
W20260709 12:22:24.784008 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 52894.9us
I20260709 12:22:24.669324 21814 raft_consensus.cc:697] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 8 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:22:24.835445 17586 catalog_manager.cc:5697] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b reported cstate change: term changed from 4 to 5, leader changed from 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) to e063541542a94969898868bf6eb5ce1b (127.16.29.67). New cstate: current_term: 5 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: HEALTHY } } }
I20260709 12:22:24.674751 21805 raft_consensus.cc:697] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 9 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:22:24.729912 21857 raft_consensus.cc:3055] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:22:24.868327 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:24.869005 17835 raft_consensus.cc:2468] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 8.
I20260709 12:22:24.780467 17837 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95"
dest_uuid: "e063541542a94969898868bf6eb5ce1b"
 from {username='slave'} at 127.0.0.1:35040
I20260709 12:22:24.858561 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:24.871304 17837 raft_consensus.cc:493] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:22:24.871486 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:24.871497 17832 raft_consensus.cc:1275] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
I20260709 12:22:24.871749 17837 raft_consensus.cc:3060] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:24.785081 21808 leader_election.cc:290] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:24.920958 17586 catalog_manager.cc:5697] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b reported cstate change: term changed from 4 to 6, leader changed from 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) to e063541542a94969898868bf6eb5ce1b (127.16.29.67). New cstate: current_term: 6 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:22:24.950245 18333 leader_election.cc:304] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:22:24.957203 17586 catalog_manager.cc:5697] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b reported cstate change: term changed from 6 to 7, leader changed from 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) to e063541542a94969898868bf6eb5ce1b (127.16.29.67). New cstate: current_term: 7 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: HEALTHY } } }
I20260709 12:22:24.850703 21820 raft_consensus.cc:697] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 11 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:22:24.960477 17586 catalog_manager.cc:5697] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b reported cstate change: term changed from 5 to 8, leader changed from e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66) to e063541542a94969898868bf6eb5ce1b (127.16.29.67). New cstate: current_term: 8 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: HEALTHY } } }
I20260709 12:22:24.880229 21857 raft_consensus.cc:740] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:22:24.840839 21674 consensus_queue.cc:1048] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.122s
W20260709 12:22:24.600788 17583 rpcz_store.cc:269] 0709 12:22:07.479069 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:07.540815 (+ 61746us) service_pool.cc:224] Handling call
0709 12:22:09.107649 (+1566834us) spinlock_profiling.cc:243] Waited 1.42 s on lock 0x7b540010d454. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a58a090 00007f7b4a628f3a 00007f7b503dee9c 00007f7b504dd04e 00007f7b4e99e2a4 00007f7b4e99e232 00007f7b4e99e161 00007f7b4e99e0dc 00007f7b4e99d392 00007f7b4895b48c 00007f7b4895a8c6
0709 12:22:24.416841 (+15309192us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:22:12.894324 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:22:12.895243 (+   919us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:22:12.895307 (+    64us) write_op.cc:435] Acquired schema lock
0709 12:22:12.895334 (+    27us) tablet.cc:662] Decoding operations
0709 12:22:12.897645 (+  2311us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:22:12.897754 (+   109us) write_op.cc:641] Partition lock acquired for write op
0709 12:22:12.897796 (+    42us) tablet.cc:685] Acquiring locks for 22 operations
0709 12:22:13.267257 (+369461us) tablet.cc:701] Row locks acquired
0709 12:22:13.267587 (+   330us) write_op.cc:260] PREPARE: finished
0709 12:22:13.267825 (+   238us) write_op.cc:270] Start()
0709 12:22:13.268038 (+   213us) write_op.cc:276] Timestamp: P: 1783599733267741 usec, L: 0
0709 12:22:13.268074 (+    36us) op_driver.cc:348] REPLICATION: starting
0709 12:22:13.322718 (+ 54644us) log.cc:844] Serialized 8555 byte log entry
0709 12:22:13.534562 (+211844us) spinlock_profiling.cc:243] Waited 3.96 ms on lock 0x7b4400022b4c. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b47459c85 00007f7b4745a6a2 00007f7b4a1cc1db 00007f7b4a1c7aaf 00007f7b475160ac 00007f7b503709a9 00007f7b4753269b 00007f7b4753093f 00007f7b47532c02 00007f7b4a1ba20b
0709 12:22:14.152789 (+618227us) spinlock_profiling.cc:243] Waited 11 ms on lock 0x7b4400022b4c. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b47459c85 00007f7b4745a6a2 00007f7b4a1cc1db 00007f7b4a1c7aaf 00007f7b4a1bb86e 00007f7b4a1babb6 00007f7b4a1ba7d8 00007f7b4a1c4221 00007f7b4a1c41d9 00007f7b4a1c4169
0709 12:22:15.218707 (+1065918us) op_driver.cc:464] REPLICATION: finished
0709 12:22:16.363769 (+1145062us) write_op.cc:301] APPLY: starting
0709 12:22:16.376750 (+ 12981us) tablet.cc:1367] starting BulkCheckPresence
0709 12:22:16.859405 (+482655us) tablet.cc:1370] finished BulkCheckPresence
0709 12:22:16.861557 (+  2152us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:22:16.934555 (+ 72998us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:22:16.935069 (+   514us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=44
0709 12:22:16.935142 (+    73us) write_op.cc:312] APPLY: finished
0709 12:22:16.988633 (+ 53491us) log.cc:844] Serialized 152 byte log entry
0709 12:22:17.125029 (+136396us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:22:17.126108 (+  1079us) write_op.cc:454] Released schema lock
0709 12:22:17.126272 (+   164us) write_op.cc:341] FINISH: Updating metrics
0709 12:22:17.127490 (+  1218us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:22:17.127541 (+    51us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":105107,"reported_tablets":43,"spinlock_wait_cycles":3276469888,"child_traces":[["op",{"apply.queue_time_us":1061606,"apply.run_cpu_time_us":14221,"apply.run_wall_time_us":828773,"num_ops":22,"prepare.queue_time_us":838674,"prepare.run_cpu_time_us":27700,"prepare.run_wall_time_us":1539480,"raft.queue_time_us":507919,"raft.run_cpu_time_us":73056,"raft.run_wall_time_us":1149882,"replication_time_us":1875303,"spinlock_wait_cycles":34491008,"thread_start_us":2571035,"threads_started":5,"wal-append.queue_time_us":695736,"wal-append.run_cpu_time_us":70748,"wal-append.run_wall_time_us":2913167}]]}
W20260709 12:22:25.000130 21853 log.cc:927] Time spent T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.190s	user 0.001s	sys 0.000s
W20260709 12:22:25.001494 21849 log.cc:927] Time spent T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.373s	user 0.000s	sys 0.001s
I20260709 12:22:25.000032 21644 leader_election.cc:290] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 10 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:24.960594 21835 raft_consensus.cc:493] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:24.869138 21845 consensus_queue.cc:1048] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.101s
I20260709 12:22:25.009311 17837 raft_consensus.cc:515] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:24.962415 21842 consensus_queue.cc:1048] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 6, Time since last communication: 0.052s
I20260709 12:22:25.030994 17837 leader_election.cc:290] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 8 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:25.018394 21867 raft_consensus.cc:493] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
W20260709 12:22:25.033792 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 248698us
I20260709 12:22:25.013058 21624 leader_election.cc:290] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 11 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:24.949470 21814 consensus_queue.cc:237] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.9, Last appended by leader: 9, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:22:25.032065 21477 consensus_meta.cc:287] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.713s	user 0.007s	sys 0.004s
I20260709 12:22:25.021479 21797 raft_consensus.cc:3055] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 8 LEADER]: Stepping down as leader of term 8
W20260709 12:22:25.067490 21616 consensus_meta.cc:287] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.737s	user 0.010s	sys 0.000s
I20260709 12:22:25.057053 21797 raft_consensus.cc:740] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 8 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
W20260709 12:22:25.075662 21780 consensus_meta.cc:287] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.794s	user 0.005s	sys 0.013s
I20260709 12:22:25.092703 17801 leader_election.cc:304] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
W20260709 12:22:25.077373 21785 consensus_meta.cc:287] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.828s	user 0.013s	sys 0.005s
I20260709 12:22:25.024633 21805 consensus_queue.cc:237] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 9, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:25.024506 21835 raft_consensus.cc:515] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:22:25.109310 17727 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:25.077028 21616 raft_consensus.cc:515] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:25.106244 21780 raft_consensus.cc:515] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:25.109310 21785 raft_consensus.cc:515] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:25.116983 21780 leader_election.cc:290] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 9 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:25.117868 21785 leader_election.cc:290] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:25.117769 21797 consensus_queue.cc:260] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.7, Last appended by leader: 7, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:25.119021 21797 raft_consensus.cc:3060] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:25.120561 17833 raft_consensus.cc:1240] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 8. Current term is 9. Ops: [8.5-8.5]
W20260709 12:22:25.123262 21649 consensus_meta.cc:287] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.510s	user 0.006s	sys 0.000s
I20260709 12:22:25.126444 17834 raft_consensus.cc:1240] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 9. Current term is 10. Ops: [9.9-9.9]
I20260709 12:22:25.126729 21864 raft_consensus.cc:2804] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:22:25.063835 21477 raft_consensus.cc:515] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:22:25.132714 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:25.038846 21867 raft_consensus.cc:515] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:22:25.137351 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 74798.8us
I20260709 12:22:25.070461 21843 tablet_copy_client.cc:323] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Beginning tablet copy session from remote peer at address 127.16.29.66:39651
W20260709 12:22:25.113768 21763 log.cc:927] Time spent T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 2.156s	user 0.007s	sys 0.027s
I20260709 12:22:25.112761 21856 raft_consensus.cc:2804] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Leader election won for term 8
W20260709 12:22:25.150441 21859 log.cc:927] Time spent T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.384s	user 0.001s	sys 0.000s
W20260709 12:22:25.150317 21863 log.cc:927] Time spent T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.146s	user 0.001s	sys 0.000s
I20260709 12:22:25.037475 21820 consensus_queue.cc:237] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:25.152530 17652 leader_election.cc:400] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 10 pre-election: Vote denied by peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837) with higher term. Message: Invalid argument: T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for earlier term 10. Current term is 11.
I20260709 12:22:25.152952 17652 leader_election.cc:403] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 10 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:22:25.137127 21864 raft_consensus.cc:493] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:25.154600 17652 leader_election.cc:304] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:25.158608 17652 leader_election.cc:304] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:25.136432 21649 raft_consensus.cc:515] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:25.155321 21856 raft_consensus.cc:697] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 8 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:22:25.171528 17652 leader_election.cc:304] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:25.157475 21864 raft_consensus.cc:3060] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:25.170905 21674 raft_consensus.cc:2749] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:22:25.180249 18368 raft_consensus.cc:1240] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 8. Current term is 9. Ops: []
I20260709 12:22:25.178627 21616 leader_election.cc:290] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 9 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:25.179452 17687 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37396 (request call id 2155) took 1002 ms. Trace:
I20260709 12:22:25.181915 17687 rpcz_store.cc:276] 0709 12:22:24.177103 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:24.232257 (+ 55154us) service_pool.cc:224] Handling call
0709 12:22:24.245421 (+ 13164us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:22:24.393296 (+147875us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:24.399216 (+  5920us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:22:24.409991 (+ 10775us) log.cc:844] Serialized 52 byte log entry
0709 12:22:24.569907 (+159916us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:22:24.598598 (+ 28691us) log.cc:844] Serialized 14 byte log entry
0709 12:22:24.606321 (+  7723us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:24.606424 (+   103us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:25.179210 (+572786us) raft_consensus.cc:1685] finished
0709 12:22:25.179230 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:25.179396 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":7408,"prepare.queue_time_us":27093,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":48,"raft.queue_time_us":37296,"raft.run_cpu_time_us":298,"raft.run_wall_time_us":5874,"thread_start_us":192339,"threads_started":2,"wal-append.queue_time_us":127178}
W20260709 12:22:25.200866 17758 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38776 (request call id 2076) took 2104 ms (client timeout 1792 ms). Trace:
I20260709 12:22:25.204419 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:22:25.241225 17758 rpcz_store.cc:269] 0709 12:22:23.096183 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:23.151978 (+ 55795us) service_pool.cc:224] Handling call
0709 12:22:25.200669 (+2048691us) spinlock_profiling.cc:243] Waited 44.1 ms on lock 0x7b540019fad4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:22:25.200789 (+   120us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":101441920}
I20260709 12:22:25.205286 17833 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35042 (request call id 2041) took 1028 ms. Trace:
I20260709 12:22:25.242053 17832 raft_consensus.cc:2468] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 6.
I20260709 12:22:25.204790 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:25.242503 17833 rpcz_store.cc:276] 0709 12:22:24.176862 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:24.177349 (+   487us) service_pool.cc:224] Handling call
0709 12:22:24.216242 (+ 38893us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:25.121198 (+904956us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:25.204969 (+ 83771us) spinlock_profiling.cc:243] Waited 904 ms on lock 0x7b540008a7d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:22:25.205221 (+   252us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2078869760}
I20260709 12:22:25.244053 17838 raft_consensus.cc:2468] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 7.
I20260709 12:22:25.240710 17652 leader_election.cc:304] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:22:25.244326 17804 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:25.205765 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 9 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:25.169265 21878 raft_consensus.cc:2749] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Leader pre-election lost for term 10. Reason: Vote denied by peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837) with higher term. Message: Invalid argument: T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for earlier term 10. Current term is 11.
I20260709 12:22:25.252902 21433 raft_consensus.cc:493] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
W20260709 12:22:25.191973 21860 log.cc:927] Time spent T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.302s	user 0.002s	sys 0.000s
I20260709 12:22:25.174908 21379 raft_consensus.cc:2749] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:22:25.267261 18333 leader_election.cc:304] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:22:25.193519 21827 raft_consensus.cc:3055] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:22:25.192485 21835 leader_election.cc:290] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:25.186601 21649 leader_election.cc:290] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 9 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:25.183178 21877 raft_consensus.cc:2749] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:22:25.184599 21630 leader_election.cc:290] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:25.187265 21867 leader_election.cc:290] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:25.246101 21785 raft_consensus.cc:493] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
W20260709 12:22:25.281358 21768 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 2.405s	user 0.048s	sys 0.037s
I20260709 12:22:25.295329 18333 leader_election.cc:304] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:25.296972 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:25.282292 21617 raft_consensus.cc:2804] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:22:25.283612 21827 raft_consensus.cc:740] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:22:25.195200 21856 consensus_queue.cc:237] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:25.329047 21649 raft_consensus.cc:493] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:25.312598 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:22:25.298429 21768 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 2.423s	user 0.048s	sys 0.038s
I20260709 12:22:25.282898 21433 raft_consensus.cc:515] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:25.302223 21820 raft_consensus.cc:2749] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:22:25.301442 21617 raft_consensus.cc:493] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
W20260709 12:22:25.341572 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 157415us
I20260709 12:22:25.344132 17652 leader_election.cc:304] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:25.297919 21630 raft_consensus.cc:2749] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:22:25.337337 21768 raft_consensus.cc:697] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 10 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:22:25.329633 21649 raft_consensus.cc:515] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:25.345969 21617 raft_consensus.cc:3060] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:25.354830 17836 raft_consensus.cc:1240] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 8. Current term is 9. Ops: [8.9-8.9]
I20260709 12:22:25.356040 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:25.356191 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:25.356837 17835 raft_consensus.cc:3060] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:25.299118 21785 raft_consensus.cc:515] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:25.350116 21845 consensus_queue.cc:1048] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.012s
I20260709 12:22:25.360189 21785 leader_election.cc:290] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:25.361111 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:25.361675 21649 leader_election.cc:290] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:25.361806 17832 raft_consensus.cc:2376] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 9. Re-sending same reply.
I20260709 12:22:25.362776 21649 raft_consensus.cc:493] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:25.363152 21617 raft_consensus.cc:515] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:25.363274 21649 raft_consensus.cc:515] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:25.365584 21649 leader_election.cc:290] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:25.365603 21617 leader_election.cc:290] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 7 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:25.371856 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:25.373214 17836 raft_consensus.cc:1240] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 8. Current term is 9. Ops: []
I20260709 12:22:25.373466 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:25.372591 17837 raft_consensus.cc:2468] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 10.
I20260709 12:22:25.347673 21674 raft_consensus.cc:2749] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:22:25.375316 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:25.417100 17835 raft_consensus.cc:2468] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 8.
W20260709 12:22:25.424072 21881 log.cc:927] Time spent T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.050s	user 0.001s	sys 0.000s
I20260709 12:22:25.426352 21827 consensus_queue.cc:260] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:25.431762 21877 raft_consensus.cc:493] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:25.419349 21588 raft_consensus.cc:493] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:22:25.418725 17652 leader_election.cc:304] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:22:25.437073 18371 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59918 (request call id 1309) took 1837 ms (client timeout 1886 ms). Trace:
W20260709 12:22:25.441299 18371 rpcz_store.cc:269] 0709 12:22:23.599369 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:23.600021 (+   652us) service_pool.cc:224] Handling call
0709 12:22:25.435625 (+1835604us) spinlock_profiling.cc:243] Waited 904 ms on lock 0x7b54006af9d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:22:25.436982 (+  1357us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2079841664}
I20260709 12:22:25.448736 21886 raft_consensus.cc:3060] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:25.445032 21588 raft_consensus.cc:515] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:25.451342 21588 leader_election.cc:290] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:25.453943 17829 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35042 (request call id 2033) took 1744 ms. Trace:
I20260709 12:22:25.443320 21877 raft_consensus.cc:515] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:25.454459 17829 rpcz_store.cc:276] 0709 12:22:23.710047 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:23.710213 (+   166us) service_pool.cc:224] Handling call
0709 12:22:23.710487 (+   274us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:23.883578 (+173091us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:23.883617 (+    39us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:23.883659 (+    42us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:22:23.883694 (+    35us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:25.453705 (+1570011us) spinlock_profiling.cc:243] Waited 173 ms on lock 0x7b54004416d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:22:25.453754 (+    49us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:25.453877 (+   123us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":397514624}
I20260709 12:22:25.456367 21877 leader_election.cc:290] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:22:25.457432 21836 log.cc:927] Time spent T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 1.154s	user 0.004s	sys 0.004s
I20260709 12:22:25.459015 21433 leader_election.cc:290] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:25.463024 21617 raft_consensus.cc:493] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:25.463651 21617 raft_consensus.cc:515] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:25.464077 21864 raft_consensus.cc:515] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:25.466161 21617 leader_election.cc:290] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:25.466899 21864 leader_election.cc:290] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 9 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:25.466956 21797 raft_consensus.cc:493] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:25.467475 21806 raft_consensus.cc:493] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:25.468659 21806 raft_consensus.cc:515] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:25.470436 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: true dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:25.471020 18364 raft_consensus.cc:3060] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:22:25.445252 21882 log.cc:927] Time spent T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.090s	user 0.001s	sys 0.000s
I20260709 12:22:25.472695 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:25.473043 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:25.472736 18371 raft_consensus.cc:1240] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 8. Current term is 9. Ops: []
I20260709 12:22:25.474283 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:25.476341 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:25.483367 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:25.484396 17652 leader_election.cc:304] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:22:25.485524 17652 leader_election.cc:304] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:22:25.486636 21864 raft_consensus.cc:493] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:25.480724 21379 consensus_queue.cc:1059] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.012s
I20260709 12:22:25.492702 21864 raft_consensus.cc:515] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:25.495617 21864 leader_election.cc:290] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:25.485325 21630 consensus_queue.cc:1059] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.014s
I20260709 12:22:25.501020 21845 raft_consensus.cc:3055] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:22:25.501157 21630 raft_consensus.cc:2804] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:22:25.501693 21845 raft_consensus.cc:740] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:22:25.501947 21630 raft_consensus.cc:697] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 8 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
W20260709 12:22:25.503150 17730 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:25.502739 21845 consensus_queue.cc:260] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:25.502991 21630 consensus_queue.cc:237] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:22:25.505671 17727 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:25.506735 21379 raft_consensus.cc:493] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:25.508105 21705 raft_consensus.cc:2804] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:22:25.507324 21379 raft_consensus.cc:515] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:25.508577 21877 raft_consensus.cc:3055] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:22:25.508999 21877 raft_consensus.cc:740] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:22:25.510025 21379 leader_election.cc:290] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:25.509828 21877 consensus_queue.cc:260] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.9, Last appended by leader: 9, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:25.511148 21877 raft_consensus.cc:3060] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:22:25.512066 21695 consensus_queue.cc:1175] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "334a4e1f59944544b3b3ff36616b57da" responder_term: 9 status { last_received { term: 6 index: 8 } last_committed_idx: 8 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 8. Current term is 9. Ops: [8.9-8.9]" } } last_received_current_leader { term: 0 index: 0 } }
I20260709 12:22:25.513855 17837 raft_consensus.cc:1240] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 10. Current term is 11. Ops: [9.8-10.9]
I20260709 12:22:25.514091 21705 raft_consensus.cc:697] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 9 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:22:25.514606 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:25.515050 21705 consensus_queue.cc:237] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:25.511178 17836 raft_consensus.cc:1240] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 7. Current term is 8. Ops: []
I20260709 12:22:25.467741 21797 raft_consensus.cc:515] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:25.510406 17760 raft_consensus.cc:1240] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 8. Current term is 9. Ops: []
I20260709 12:22:25.516502 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:25.517712 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:25.519052 17832 raft_consensus.cc:2393] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 12: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:22:25.521371 17761 raft_consensus.cc:1240] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 11. Current term is 12. Ops: []
I20260709 12:22:25.521481 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:25.522320 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 11 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:25.523455 17829 raft_consensus.cc:2410] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for term 11 because replica has last-logged OpId of term: 8 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 7.
I20260709 12:22:25.531574 17836 raft_consensus.cc:2393] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 9: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:22:25.538887 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:22:25.542302 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 59594.6us
I20260709 12:22:25.545082 17801 leader_election.cc:304] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:25.545118 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:25.549749 18366 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 1312) took 1081 ms. Trace:
I20260709 12:22:25.553340 18364 raft_consensus.cc:2468] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 8.
I20260709 12:22:25.546942 17762 raft_consensus.cc:2468] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 12.
I20260709 12:22:25.554849 17803 leader_election.cc:304] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:22:25.555987 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:25.556782 17762 raft_consensus.cc:2468] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 9.
I20260709 12:22:25.558286 17762 raft_consensus.cc:1275] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260709 12:22:25.559665 17762 raft_consensus.cc:1240] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 10. Current term is 11. Ops: []
I20260709 12:22:25.576300 17756 raft_consensus.cc:1275] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
I20260709 12:22:25.544921 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:25.550243 18366 rpcz_store.cc:276] 0709 12:22:24.468052 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:24.471198 (+  3146us) service_pool.cc:224] Handling call
0709 12:22:24.471417 (+   219us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:24.513161 (+ 41744us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 12:22:24.569100 (+ 55939us) log.cc:844] Serialized 14 byte log entry
0709 12:22:24.951493 (+382393us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:24.951543 (+    50us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:22:24.951590 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:25.549413 (+597823us) spinlock_profiling.cc:243] Waited 41.5 ms on lock 0x7b54004687d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:22:25.549476 (+    63us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:25.549667 (+   191us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":9231,"prepare.queue_time_us":358956,"prepare.run_cpu_time_us":25,"prepare.run_wall_time_us":25,"raft.queue_time_us":234385,"raft.run_cpu_time_us":200,"raft.run_wall_time_us":62811,"spinlock_wait_cycles":95523584,"thread_start_us":421935,"threads_started":3,"wal-append.queue_time_us":380601}
W20260709 12:22:25.579830 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 60048.3us
W20260709 12:22:25.580618 17652 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.16.29.66:39651 timed out after 1.792s (SENT) [suppressed 20 similar messages]
I20260709 12:22:25.531685 21845 raft_consensus.cc:3060] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:25.545835 21886 raft_consensus.cc:2749] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:22:25.582319 21856 consensus_queue.cc:1059] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 11.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.006s
I20260709 12:22:25.586354 21856 raft_consensus.cc:3055] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:22:25.586730 21856 raft_consensus.cc:740] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 11 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:22:25.587332 21797 leader_election.cc:290] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:25.587726 21856 consensus_queue.cc:260] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:25.589061 21856 raft_consensus.cc:3060] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:25.600929 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:25.602793 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:25.604442 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 9 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
W20260709 12:22:25.604797 21894 log.cc:927] Time spent T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.060s	user 0.000s	sys 0.000s
I20260709 12:22:25.444306 21768 consensus_queue.cc:237] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.6, Last appended by leader: 6, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:25.607506 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:25.609025 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:22:25.610060 21871 log.cc:927] Time spent T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.538s	user 0.003s	sys 0.000s
I20260709 12:22:25.610561 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:25.612427 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:25.582096 21858 raft_consensus.cc:2804] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:22:25.579634 21782 raft_consensus.cc:2749] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:22:25.634652 21617 raft_consensus.cc:493] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:25.638516 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:25.642683 18366 raft_consensus.cc:1240] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 8 LEADER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 7. Current term is 8. Ops: []
I20260709 12:22:25.644268 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:25.647773 18366 raft_consensus.cc:1240] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 7. Current term is 8. Ops: []
I20260709 12:22:25.649668 18374 raft_consensus.cc:1240] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 6. Current term is 7. Ops: []
I20260709 12:22:25.650038 18371 raft_consensus.cc:1240] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 8. Current term is 9. Ops: []
I20260709 12:22:25.655912 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 11 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:25.657210 17689 raft_consensus.cc:2410] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for term 11 because replica has last-logged OpId of term: 8 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 7.
I20260709 12:22:25.658339 18367 raft_consensus.cc:1240] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 11 LEADER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 10. Current term is 11. Ops: [10.9-10.9]
I20260709 12:22:25.640645 21649 raft_consensus.cc:493] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:25.676816 17655 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:25.641085 21858 raft_consensus.cc:697] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 8 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:22:25.664531 21879 consensus_queue.cc:1059] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.017s
I20260709 12:22:25.683475 17831 raft_consensus.cc:1275] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 11 index: 9. (index mismatch)
I20260709 12:22:25.685107 17687 raft_consensus.cc:2410] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for term 10 because replica has last-logged OpId of term: 9 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 8.
I20260709 12:22:25.666524 21820 raft_consensus.cc:493] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:22:25.687935 21820 raft_consensus.cc:515] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:25.690057 21820 leader_election.cc:290] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:25.668630 21800 consensus_queue.cc:1059] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.022s
I20260709 12:22:25.664875 21707 raft_consensus.cc:493] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:25.691298 21707 raft_consensus.cc:515] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:25.693291 21707 leader_election.cc:290] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:25.693866 21879 consensus_queue.cc:1243] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer 334a4e1f59944544b3b3ff36616b57da log is divergent from this leader: its last log entry 8.10 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 8
I20260709 12:22:25.694025 21649 raft_consensus.cc:515] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:25.663936 21782 raft_consensus.cc:493] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:25.668990 21867 raft_consensus.cc:493] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:25.671278 21379 consensus_queue.cc:1059] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 9.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.025s
I20260709 12:22:25.700265 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:25.705407 21797 raft_consensus.cc:493] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:25.681263 21781 raft_consensus.cc:493] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:25.710937 21827 raft_consensus.cc:3060] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:25.647931 21617 raft_consensus.cc:515] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:25.708983 21800 raft_consensus.cc:493] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:25.677858 21888 consensus_queue.cc:1059] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.035s
W20260709 12:22:25.719338 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 138739us
I20260709 12:22:25.711480 21407 raft_consensus.cc:493] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:25.714571 21379 consensus_queue.cc:1243] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer 334a4e1f59944544b3b3ff36616b57da log is divergent from this leader: its last log entry 11.9 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 8
I20260709 12:22:25.714056 21858 consensus_queue.cc:237] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:25.707247 21782 raft_consensus.cc:515] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:25.707898 21867 raft_consensus.cc:515] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:25.713635 21797 raft_consensus.cc:515] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:25.751667 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: true dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:25.753486 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:25.754890 17687 raft_consensus.cc:1240] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 11. Current term is 12. Ops: [11.7-11.7]
I20260709 12:22:25.755138 17684 raft_consensus.cc:3060] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:22:25.756217 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:25.715092 21781 raft_consensus.cc:515] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:25.758967 17688 raft_consensus.cc:1240] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 9 LEADER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 8. Current term is 9. Ops: []
I20260709 12:22:25.741382 21630 raft_consensus.cc:493] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:25.770565 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:25.722383 21800 raft_consensus.cc:515] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:25.741215 21407 raft_consensus.cc:515] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:25.747133 21588 consensus_queue.cc:1059] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.023s
I20260709 12:22:25.753497 21805 raft_consensus.cc:493] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:25.782507 17757 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38786 (request call id 2765) took 2523 ms. Trace:
I20260709 12:22:25.756983 21830 raft_consensus.cc:493] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:25.788493 17757 rpcz_store.cc:276] 0709 12:22:23.259405 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:23.390876 (+131471us) service_pool.cc:224] Handling call
0709 12:22:23.420105 (+ 29229us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:23.520685 (+100580us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:23.520733 (+    48us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:23.524148 (+  3415us) log.cc:844] Serialized 26 byte log entry
0709 12:22:23.655867 (+131719us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:22:23.655956 (+    89us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:25.775750 (+2119794us) spinlock_profiling.cc:243] Waited 94.5 ms on lock 0x7b54002977d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:22:25.775820 (+    70us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:25.775912 (+    92us) raft_consensus.cc:1685] finished
0709 12:22:25.775931 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:25.782447 (+  6516us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2241,"raft.queue_time_us":4360,"raft.run_cpu_time_us":122,"raft.run_wall_time_us":3105,"spinlock_wait_cycles":217429760,"thread_start_us":122662,"threads_started":1,"wal-append.queue_time_us":202943,"wal-append.run_cpu_time_us":2571,"wal-append.run_wall_time_us":1223804}
I20260709 12:22:25.769755 21705 raft_consensus.cc:493] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:22:25.758632 21870 consensus_queue.cc:1059] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.029s
I20260709 12:22:25.809847 17764 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38786 (request call id 2774) took 1446 ms. Trace:
I20260709 12:22:25.810340 17764 rpcz_store.cc:276] 0709 12:22:24.363087 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:24.364879 (+  1792us) service_pool.cc:224] Handling call
0709 12:22:24.623050 (+258171us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:24.623620 (+   570us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:25.809633 (+1186013us) spinlock_profiling.cc:243] Waited 258 ms on lock 0x7b540019f0d0. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a61f8ad 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:22:25.809784 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":592879872}
W20260709 12:22:25.813338 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 1.3411e+06us
I20260709 12:22:25.820420 21797 leader_election.cc:290] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:22:25.822512 21898 log.cc:927] Time spent T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.055s	user 0.000s	sys 0.000s
I20260709 12:22:25.824874 21708 raft_consensus.cc:493] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:25.825059 21893 log.cc:927] Time spent T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.083s	user 0.001s	sys 0.000s
I20260709 12:22:25.825409 21708 raft_consensus.cc:515] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:25.779333 21630 raft_consensus.cc:515] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:25.831529 17728 leader_election.cc:304] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:25.827809 18368 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 1321) took 1357 ms. Trace:
I20260709 12:22:25.828863 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:25.833261 17764 raft_consensus.cc:2468] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 9.
I20260709 12:22:25.833055 18368 rpcz_store.cc:276] 0709 12:22:24.470330 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:24.472837 (+  2507us) service_pool.cc:224] Handling call
0709 12:22:24.473099 (+   262us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:25.180852 (+707753us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:25.827575 (+646723us) spinlock_profiling.cc:243] Waited 707 ms on lock 0x7b54001efd54. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:22:25.827733 (+   158us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1625667840}
I20260709 12:22:25.795949 21768 raft_consensus.cc:493] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:25.835337 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:25.838308 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:25.838759 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:25.838915 17835 raft_consensus.cc:2468] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 8.
W20260709 12:22:25.840256 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 48715.6us
I20260709 12:22:25.793823 21805 raft_consensus.cc:515] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:25.844981 17652 leader_election.cc:304] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:25.846282 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:25.846486 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:25.850607 17759 raft_consensus.cc:2368] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for earlier term 10. Current term is 11.
I20260709 12:22:25.850991 17652 leader_election.cc:304] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:22:25.796350 21705 raft_consensus.cc:515] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:25.827004 21649 leader_election.cc:290] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:25.828032 21810 raft_consensus.cc:493] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:25.854212 17652 leader_election.cc:304] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:25.795089 21830 raft_consensus.cc:515] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:25.853128 17681 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 10.
I20260709 12:22:25.858677 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:25.861496 17837 raft_consensus.cc:2376] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 8. Re-sending same reply.
I20260709 12:22:25.863173 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:25.864056 17686 raft_consensus.cc:2468] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 11.
I20260709 12:22:25.859700 17832 raft_consensus.cc:1275] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20260709 12:22:25.859647 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:25.859210 21370 raft_consensus.cc:2749] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:22:25.872956 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:25.873039 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:25.876034 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:25.876367 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:25.873571 17829 raft_consensus.cc:2468] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 12.
I20260709 12:22:25.876399 17837 raft_consensus.cc:2468] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 9.
I20260709 12:22:25.878639 17652 leader_election.cc:304] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:25.842355 21692 raft_consensus.cc:493] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:22:25.885504 17652 leader_election.cc:400] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 10 pre-election: Vote denied by peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651) with higher term. Message: Invalid argument: T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for earlier term 10. Current term is 11.
I20260709 12:22:25.852458 21624 raft_consensus.cc:493] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:25.855015 21870 raft_consensus.cc:2749] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:22:25.889081 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:25.889846 17763 raft_consensus.cc:3060] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:25.860227 21877 raft_consensus.cc:2804] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:22:25.860590 21617 leader_election.cc:290] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:25.892676 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:25.868777 21886 raft_consensus.cc:2804] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:22:25.900681 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:25.900839 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:25.872551 21916 raft_consensus.cc:493] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:25.838903 21768 raft_consensus.cc:515] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:25.903128 21781 leader_election.cc:290] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:25.874636 21810 raft_consensus.cc:515] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:25.897400 21624 raft_consensus.cc:515] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:25.906180 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 11 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:25.907159 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:25.907721 17764 raft_consensus.cc:2468] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 9.
I20260709 12:22:25.908793 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:25.909042 17833 raft_consensus.cc:1240] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 7. Current term is 8. Ops: [6.8-7.9]
I20260709 12:22:25.909540 21480 consensus_queue.cc:260] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 7.9, Last appended by leader: 9, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:25.910727 21480 raft_consensus.cc:3060] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:25.911111 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:25.911569 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:25.912521 21630 leader_election.cc:290] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:25.868396 21797 consensus_queue.cc:1059] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 9.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.029s
I20260709 12:22:25.902067 17753 tablet_service.cc:1558] Processing DeleteTablet for tablet 7cb3768d215e409fb9f5baf18d124442 with delete_type TABLET_DATA_TOMBSTONED (TS e2909cff1da44ad387f96a8ae3aa5afb not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:38774
I20260709 12:22:25.914201 21797 raft_consensus.cc:3055] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:22:25.902273 17774 tablet_copy_service.cc:140] P e2909cff1da44ad387f96a8ae3aa5afb: Received BeginTabletCopySession request for tablet d48b7de191684318aa08f60a7a395387 from peer 334a4e1f59944544b3b3ff36616b57da ({username='slave'} at 127.0.0.1:36572)
I20260709 12:22:25.914978 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:25.906795 17759 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 10.
I20260709 12:22:25.922835 17684 raft_consensus.cc:2410] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 9 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 4.
I20260709 12:22:25.937309 18333 leader_election.cc:304] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:22:25.938779 18333 leader_election.cc:304] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:25.939308 17763 raft_consensus.cc:2468] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 9.
W20260709 12:22:25.940653 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 108117us
I20260709 12:22:25.942590 17727 leader_election.cc:304] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:25.892755 21589 raft_consensus.cc:2749] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:22:25.944969 18333 leader_election.cc:304] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:25.874372 21888 consensus_queue.cc:1048] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 6, Time since last communication: 0.042s
I20260709 12:22:25.946245 18333 leader_election.cc:304] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:25.914623 21797 raft_consensus.cc:740] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 9 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:22:25.896212 21877 raft_consensus.cc:493] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:25.888358 21879 consensus_queue.cc:1059] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 10.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.050s
I20260709 12:22:25.954471 18333 leader_election.cc:304] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:22:25.891481 21766 consensus_queue.cc:1048] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.057s
W20260709 12:22:25.952757 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 119717us
I20260709 12:22:25.957867 21864 raft_consensus.cc:2804] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:22:25.958118 21800 leader_election.cc:290] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:25.958055 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:25.888900 21692 raft_consensus.cc:515] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:25.901113 21886 raft_consensus.cc:493] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:25.928243 21782 leader_election.cc:290] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:25.946623 21781 raft_consensus.cc:2804] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:22:25.967931 21766 consensus_queue.cc:1243] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer e2909cff1da44ad387f96a8ae3aa5afb log is divergent from this leader: its last log entry 7.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260709 12:22:25.948845 21867 leader_election.cc:290] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:26.000846 21806 leader_election.cc:290] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:25.948177 17774 tablet_copy_service.cc:161] P e2909cff1da44ad387f96a8ae3aa5afb: Beginning new tablet copy session on tablet d48b7de191684318aa08f60a7a395387 from peer 334a4e1f59944544b3b3ff36616b57da at {username='slave'} at 127.0.0.1:36572: session id = 334a4e1f59944544b3b3ff36616b57da-d48b7de191684318aa08f60a7a395387
W20260709 12:22:26.004577 21911 log.cc:927] Time spent T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.104s	user 0.000s	sys 0.000s
I20260709 12:22:26.005141 17802 leader_election.cc:304] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
W20260709 12:22:26.006047 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 140405us
I20260709 12:22:25.954581 21877 raft_consensus.cc:3060] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:26.008982 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:22:26.009879 17654 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:25.985565 21830 leader_election.cc:290] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:25.953248 21617 raft_consensus.cc:2749] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:22:25.955361 21820 raft_consensus.cc:2749] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:22:25.996055 21407 leader_election.cc:290] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:26.020404 21806 raft_consensus.cc:2804] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:22:26.020954 21806 raft_consensus.cc:493] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:26.021430 21806 raft_consensus.cc:3060] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:26.023720 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:26.024559 17756 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 10.
I20260709 12:22:26.025480 18365 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 1314) took 1556 ms. Trace:
I20260709 12:22:25.953699 21379 raft_consensus.cc:493] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:25.989712 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:25.928092 21916 raft_consensus.cc:515] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:25.998958 21708 leader_election.cc:290] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:25.996232 21781 raft_consensus.cc:493] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:26.010743 17802 leader_election.cc:304] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:26.014595 17759 pending_rounds.cc:85] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb: Aborting all ops after (but not including) 5
I20260709 12:22:25.966398 21864 raft_consensus.cc:493] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:25.974942 21886 raft_consensus.cc:3060] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:26.025924 18365 rpcz_store.cc:276] 0709 12:22:24.468592 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:24.471603 (+  3011us) service_pool.cc:224] Handling call
0709 12:22:24.505528 (+ 33925us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:24.505961 (+   433us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 12:22:25.617502 (+1111541us) spinlock_profiling.cc:243] Waited 4.6 ms on lock 0x7b1800498d60. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b47515e07 00007f7b503709a9 00007f7b4753269b 00007f7b4753093f 00007f7b47530455 00007f7b4e0183c7 00007f7b4a62a9a7 00007f7b4a63a0bf 00007f7b4a63a038
0709 12:22:25.745469 (+127967us) log.cc:844] Serialized 14 byte log entry
0709 12:22:25.791396 (+ 45927us) log.cc:844] Serialized 14 byte log entry
0709 12:22:25.793772 (+  2376us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:25.794320 (+   548us) log.cc:844] Serialized 26 byte log entry
0709 12:22:25.794841 (+   521us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:22:25.794886 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:25.794952 (+    66us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:26.025226 (+230274us) raft_consensus.cc:1685] finished
0709 12:22:26.025251 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:26.025399 (+   148us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5853,"prepare.queue_time_us":366259,"prepare.run_cpu_time_us":83,"prepare.run_wall_time_us":82,"raft.queue_time_us":6139,"raft.run_cpu_time_us":379,"raft.run_wall_time_us":8868,"spinlock_wait_cycles":24659968,"thread_start_us":125260,"threads_started":2,"wal-append.queue_time_us":73128}
I20260709 12:22:26.033942 17759 pending_rounds.cc:107] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb: Aborting uncommitted NO_OP operation due to leader change: 7.6
I20260709 12:22:25.997583 21835 raft_consensus.cc:2804] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:22:26.049006 17759 raft_consensus.cc:2889] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:25.996356 21786 raft_consensus.cc:2804] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Leader election won for term 11
W20260709 12:22:26.030123 21906 log.cc:927] Time spent T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.215s	user 0.001s	sys 0.001s
W20260709 12:22:26.033910 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 79493.1us
W20260709 12:22:26.050321 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 89834.9us
I20260709 12:22:26.055614 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:26.034737 21781 raft_consensus.cc:3060] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:26.036257 21864 raft_consensus.cc:3060] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:22:26.061992 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 85516.3us
I20260709 12:22:26.049693 21407 raft_consensus.cc:2749] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:22:26.072578 21705 leader_election.cc:290] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:26.056701 21835 raft_consensus.cc:697] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 9 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:22:26.073985 21835 consensus_queue.cc:237] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:26.033272 21379 raft_consensus.cc:515] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:22:26.076413 21805 leader_election.cc:290] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:26.079993 21684 consensus_queue.cc:1059] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true }, Status: INVALID_TERM, Last received: 8.4, Next index: 5, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:22:26.081019 21684 raft_consensus.cc:3055] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:22:26.081380 21684 raft_consensus.cc:740] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:22:26.082213 21684 consensus_queue.cc:260] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:22:26.082695 21379 leader_election.cc:290] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:26.057776 21617 raft_consensus.cc:493] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:26.089214 21617 raft_consensus.cc:515] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:26.090831 21781 raft_consensus.cc:515] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:26.091955 21686 raft_consensus.cc:493] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:26.092614 21686 raft_consensus.cc:515] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:26.093900 21781 leader_election.cc:290] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 10 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:26.080152 21916 leader_election.cc:290] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:26.099416 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:26.099459 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:26.104142 17763 raft_consensus.cc:1275] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260709 12:22:26.084906 21477 leader_election.cc:290] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 8 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:26.104763 21786 raft_consensus.cc:697] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:22:26.108976 21504 consensus_queue.cc:1048] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20260709 12:22:26.109208 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 55808us
I20260709 12:22:26.083722 21684 raft_consensus.cc:3060] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:26.110427 17834 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35042 (request call id 2040) took 1933 ms. Trace:
I20260709 12:22:26.128554 17728 leader_election.cc:304] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:22:26.128922 17834 rpcz_store.cc:276] 0709 12:22:24.176614 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:24.186355 (+  9741us) service_pool.cc:224] Handling call
0709 12:22:24.186661 (+   306us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:25.134298 (+947637us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:26.110132 (+975834us) spinlock_profiling.cc:243] Waited 939 ms on lock 0x7b540022b854. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:22:26.110354 (+   222us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2160737152}
I20260709 12:22:26.107656 17727 leader_election.cc:304] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:26.130839 17727 leader_election.cc:304] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:22:26.113706 17756 raft_consensus.cc:2410] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for term 9 because replica has last-logged OpId of term: 7 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 7.
I20260709 12:22:26.127300 17689 raft_consensus.cc:1275] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 7 index: 9. Preceding OpId from leader: term: 8 index: 10. (index mismatch)
I20260709 12:22:26.029824 21797 consensus_queue.cc:260] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.7, Last appended by leader: 7, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:26.135396 21407 raft_consensus.cc:493] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:26.136672 21407 raft_consensus.cc:515] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:26.136138 21471 raft_consensus.cc:2804] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:22:26.138971 21407 leader_election.cc:290] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:26.091818 21617 leader_election.cc:290] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:26.139145 21786 consensus_queue.cc:237] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 9.5, Last appended by leader: 5, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:26.140841 21797 raft_consensus.cc:3060] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:26.141244 21927 tablet_replica.cc:333] T 7cb3768d215e409fb9f5baf18d124442 P e2909cff1da44ad387f96a8ae3aa5afb: stopping tablet replica
I20260709 12:22:26.132436 21773 raft_consensus.cc:2749] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:22:26.146544 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 11 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:26.147909 17838 raft_consensus.cc:2410] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for term 11 because replica has last-logged OpId of term: 8 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 5.
I20260709 12:22:26.149034 21916 raft_consensus.cc:2749] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:22:26.149326 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:26.150010 17833 raft_consensus.cc:2468] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 9.
I20260709 12:22:26.127718 21781 consensus_queue.cc:1048] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.023s
I20260709 12:22:26.150743 21806 raft_consensus.cc:515] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:26.158243 17830 raft_consensus.cc:1240] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 10. Current term is 11. Ops: []
I20260709 12:22:26.160118 21857 consensus_queue.cc:260] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 10.9, Last appended by leader: 9, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:26.158281 21471 raft_consensus.cc:697] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:22:26.168794 21471 consensus_queue.cc:237] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.7, Last appended by leader: 7, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:26.175969 21864 raft_consensus.cc:515] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:26.181546 21806 leader_election.cc:290] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 12 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:26.179667 21864 leader_election.cc:290] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 10 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:26.189916 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:26.190678 17763 raft_consensus.cc:3060] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:26.176236 21916 consensus_queue.cc:1059] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.016s
I20260709 12:22:26.192785 21916 raft_consensus.cc:3055] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:22:26.193153 21916 raft_consensus.cc:740] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 9 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:22:26.193993 21916 consensus_queue.cc:260] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:26.195204 21916 raft_consensus.cc:3060] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:26.199728 17763 raft_consensus.cc:2468] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 10.
I20260709 12:22:26.201643 18333 leader_election.cc:304] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:26.197108 17727 leader_election.cc:304] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:26.206653 21864 raft_consensus.cc:2804] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:22:26.174614 21857 raft_consensus.cc:3060] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:26.208063 21684 raft_consensus.cc:2804] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:22:26.171710 21886 raft_consensus.cc:515] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:26.213438 21864 raft_consensus.cc:697] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 10 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:22:26.229323 21806 raft_consensus.cc:493] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:26.238322 21806 raft_consensus.cc:515] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:26.238324 21864 consensus_queue.cc:237] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.7, Last appended by leader: 7, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:26.240789 21806 leader_election.cc:290] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:26.208945 21920 consensus_queue.cc:1059] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 10.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.021s
I20260709 12:22:26.243230 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:26.244186 18368 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 10.
I20260709 12:22:26.246143 18366 raft_consensus.cc:1275] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260709 12:22:26.228400 21684 raft_consensus.cc:493] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:26.248775 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:26.250941 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:26.251667 18365 raft_consensus.cc:2468] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 9.
I20260709 12:22:26.253270 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:26.238445 21686 leader_election.cc:290] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:26.255465 21785 raft_consensus.cc:493] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:26.256075 21785 raft_consensus.cc:515] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:26.258404 21785 leader_election.cc:290] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:26.261327 17803 leader_election.cc:304] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:22:26.262418 17803 leader_election.cc:304] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:26.249783 21877 raft_consensus.cc:515] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:26.266040 21877 leader_election.cc:290] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 12 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:26.262802 21806 consensus_queue.cc:1048] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 12:22:26.263015 21785 raft_consensus.cc:2804] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:22:26.266215 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 9 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:26.266577 21797 raft_consensus.cc:2749] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:22:26.274883 21886 leader_election.cc:290] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 9 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:26.284662 18361 raft_consensus.cc:2393] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 9: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:22:26.282981 21785 raft_consensus.cc:493] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:26.290846 17774 tablet_copy_source_session.cc:215] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:22:26.272365 21684 raft_consensus.cc:3060] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:22:26.307960 17730 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:26.293489 21785 raft_consensus.cc:3060] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:26.309507 21845 raft_consensus.cc:3055] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:22:26.309916 21845 raft_consensus.cc:740] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:22:26.310982 21843 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d48b7de191684318aa08f60a7a395387. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:22:26.310765 21845 consensus_queue.cc:260] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:26.326545 17682 raft_consensus.cc:3055] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:22:26.327090 17682 raft_consensus.cc:740] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:22:26.328323 21845 raft_consensus.cc:3060] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:26.328272 17682 consensus_queue.cc:260] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:26.329630 17682 raft_consensus.cc:3060] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:26.337972 21810 leader_election.cc:290] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:26.340106 17682 raft_consensus.cc:2468] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 8.
I20260709 12:22:27.140089 17689 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44898 (request call id 846) took 1461 ms. Trace:
I20260709 12:22:27.140704 17689 rpcz_store.cc:276] 0709 12:22:25.678826 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:25.679168 (+   342us) service_pool.cc:224] Handling call
0709 12:22:25.679597 (+   429us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:26.131937 (+452340us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:27.139689 (+1007752us) spinlock_profiling.cc:243] Waited 417 ms on lock 0x7b54004254d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:22:27.140006 (+   317us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":958200704}
W20260709 12:22:27.251320 17730 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:27.253325 21843 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.StartTabletCopy from 127.0.0.1:59886 (request call id 1184) took 3097 ms. Trace:
I20260709 12:22:27.253832 21843 rpcz_store.cc:276] 0709 12:22:24.156059 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:24.158888 (+  2829us) service_pool.cc:224] Handling call
0709 12:22:24.542008 (+383120us) ts_tablet_manager.cc:934] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da: Initiating tablet copy from peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
0709 12:22:26.309603 (+1767595us) tablet_copy_client.cc:368] Tablet copy session begun
0709 12:22:27.243865 (+934262us) tablet_metadata.cc:665] Metadata flushed
0709 12:22:27.244354 (+   489us) tablet_copy_client.cc:497] Wrote new tablet metadata
0709 12:22:27.252481 (+  8127us) tablet_copy_client.cc:418] Wrote new consensus metadata
0709 12:22:27.252810 (+   329us) tablet_replica.cc:206] Creating consensus instance
0709 12:22:27.253135 (+   325us) ts_tablet_manager.cc:989] Replica registered.
0709 12:22:27.253215 (+    80us) inbound_call.cc:177] Queueing success response
Metrics: {"dns_us":27408,"tablet-copy.queue_time_us":226041,"thread_start_us":189258,"threads_started":2}
I20260709 12:22:27.255436 21843 tablet_copy_client.cc:806] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Starting download of 0 data blocks...
I20260709 12:22:27.256604 21843 tablet_copy_client.cc:670] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Starting download of 1 WAL segments...
W20260709 12:22:27.257241 17654 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:27.258001 17654 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:27.262605 21918 raft_consensus.cc:493] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:27.263648 21918 raft_consensus.cc:515] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:27.268251 21918 leader_election.cc:290] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:22:27.278328 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 1.14861e+06us
I20260709 12:22:27.286796 21674 raft_consensus.cc:493] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:27.291729 17728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:27.294098 21778 raft_consensus.cc:493] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:27.294037 21918 raft_consensus.cc:493] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:22:27.287546 21674 raft_consensus.cc:515] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:27.294827 21778 raft_consensus.cc:515] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:27.295209 21918 raft_consensus.cc:515] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:27.298609 21778 leader_election.cc:290] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:27.298732 21674 leader_election.cc:290] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:27.299409 21674 raft_consensus.cc:493] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:27.295145 21624 leader_election.cc:290] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:27.299839 21778 raft_consensus.cc:493] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:27.300647 21778 raft_consensus.cc:515] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:27.301151 21624 raft_consensus.cc:493] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:27.302270 21624 raft_consensus.cc:515] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:27.303119 21778 leader_election.cc:290] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:27.301581 21918 leader_election.cc:290] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:27.304803 21624 leader_election.cc:290] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:27.305497 21674 raft_consensus.cc:515] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:27.308674 21674 leader_election.cc:290] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:27.360602 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
W20260709 12:22:27.365060 17680 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 1.420s	user 0.026s	sys 0.030s
W20260709 12:22:27.365777 17680 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.421s	user 0.026s	sys 0.030s
I20260709 12:22:27.374446 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:27.378929 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 11 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:27.376140 17689 raft_consensus.cc:1275] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 11 index: 9. (index mismatch)
I20260709 12:22:27.382436 21842 raft_consensus.cc:493] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:27.385181 21624 raft_consensus.cc:493] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:22:27.385689 21842 raft_consensus.cc:515] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:27.386399 21624 raft_consensus.cc:515] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:27.389954 21918 raft_consensus.cc:493] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:22:27.390983 21918 raft_consensus.cc:515] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:27.399688 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:27.402104 17682 raft_consensus.cc:2468] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 9.
I20260709 12:22:27.402189 21918 leader_election.cc:290] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:27.403803 21624 leader_election.cc:290] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:22:27.425119 17652 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:27.400182 21915 raft_consensus.cc:493] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:27.482089 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:27.482194 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:27.449606 21842 leader_election.cc:290] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:27.483280 17759 raft_consensus.cc:2468] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 8.
I20260709 12:22:27.483614 21842 raft_consensus.cc:493] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:22:27.454826 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:27.484902 21842 raft_consensus.cc:515] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:27.487058 17688 raft_consensus.cc:2468] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 11.
I20260709 12:22:27.459971 17684 raft_consensus.cc:1275] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 12 index: 8. (index mismatch)
I20260709 12:22:27.489184 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:27.490702 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:27.479943 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:27.434442 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:27.456571 17728 leader_election.cc:304] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:27.494038 21624 raft_consensus.cc:2749] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 12:22:27.495268 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:27.457563 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:27.496264 17689 raft_consensus.cc:2468] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 6.
I20260709 12:22:27.482961 21915 raft_consensus.cc:515] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:27.480597 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:27.500008 21842 leader_election.cc:290] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:27.496603 17683 raft_consensus.cc:2468] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 9.
I20260709 12:22:27.507117 17681 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37402 (request call id 2556) took 1737 ms. Trace:
I20260709 12:22:27.514436 17681 rpcz_store.cc:276] 0709 12:22:25.769604 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:25.770124 (+   520us) service_pool.cc:224] Handling call
0709 12:22:27.506649 (+1736525us) spinlock_profiling.cc:243] Waited 81.5 ms on lock 0x7b54004259d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:22:27.506916 (+   267us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":187564416}
I20260709 12:22:27.534720 17728 leader_election.cc:304] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:22:27.536262 21686 raft_consensus.cc:2749] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:22:27.536650 21480 raft_consensus.cc:493] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:27.537613 21480 raft_consensus.cc:515] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:27.491542 17684 raft_consensus.cc:2468] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 8.
I20260709 12:22:27.539913 21915 leader_election.cc:290] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:27.547072 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:27.545320 21379 raft_consensus.cc:493] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:22:27.562143 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:27.508718 17758 raft_consensus.cc:2468] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 9.
I20260709 12:22:27.568671 21477 raft_consensus.cc:493] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:27.571655 17682 raft_consensus.cc:2468] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 10.
I20260709 12:22:27.581290 21477 raft_consensus.cc:515] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:27.509290 17652 leader_election.cc:304] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:27.574932 21370 consensus_queue.cc:1048] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.023s
I20260709 12:22:27.575676 21666 raft_consensus.cc:3055] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:22:27.600523 21666 raft_consensus.cc:740] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:22:27.600590 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 9 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:27.573632 21630 raft_consensus.cc:3055] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:22:27.602906 21630 raft_consensus.cc:740] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:22:27.604383 21630 consensus_queue.cc:260] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:27.606731 21630 raft_consensus.cc:3060] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:27.602286 21666 consensus_queue.cc:260] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 10.8, Last appended by leader: 8, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:27.511493 17683 raft_consensus.cc:1275] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 11 index: 6. (index mismatch)
I20260709 12:22:27.509215 21842 raft_consensus.cc:493] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:27.611740 21842 raft_consensus.cc:515] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:27.569164 21379 raft_consensus.cc:515] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:27.620565 21783 raft_consensus.cc:493] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:27.519819 21692 leader_election.cc:290] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:27.621585 21783 raft_consensus.cc:515] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:22:27.640983 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 71867.2us
I20260709 12:22:27.643191 21783 leader_election.cc:290] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:27.644526 17688 raft_consensus.cc:2468] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 10.
I20260709 12:22:27.644130 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:27.649049 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:27.647706 17684 raft_consensus.cc:2468] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 10.
I20260709 12:22:27.621608 21379 leader_election.cc:290] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:22:27.654603 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 148281us
I20260709 12:22:27.662094 21842 leader_election.cc:290] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:27.657140 17682 raft_consensus.cc:1275] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 11 index: 9. (term mismatch)
I20260709 12:22:27.663069 17759 raft_consensus.cc:1240] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 10. Current term is 11. Ops: []
W20260709 12:22:27.663736 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 89963.3us
I20260709 12:22:27.665509 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:27.614945 21480 leader_election.cc:290] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:27.667117 17758 raft_consensus.cc:2468] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 10.
I20260709 12:22:27.663074 17682 pending_rounds.cc:85] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3: Aborting all ops after (but not including) 8
I20260709 12:22:27.669869 17682 pending_rounds.cc:107] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3: Aborting uncommitted NO_OP operation due to leader change: 10.9
I20260709 12:22:27.670250 17682 raft_consensus.cc:2889] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:27.670868 21848 consensus_queue.cc:1048] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:22:27.671392 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:27.667076 17728 leader_election.cc:304] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:27.653630 21666 raft_consensus.cc:3060] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:27.679906 21928 raft_consensus.cc:493] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:27.680824 21928 raft_consensus.cc:515] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:27.680151 21824 raft_consensus.cc:2804] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:22:27.682199 21824 raft_consensus.cc:493] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:22:27.682678 21824 raft_consensus.cc:3060] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:27.702826 21616 raft_consensus.cc:493] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:27.726166 21707 raft_consensus.cc:493] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:27.730199 21707 raft_consensus.cc:515] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:22:27.706710 21892 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 8 current term: 8 is higher.
W20260709 12:22:27.743690 17710 debug-util.cc:398] Leaking SignalData structure 0x7b08002f1980 after lost signal to thread 16504
I20260709 12:22:27.744025 21768 leader_election.cc:290] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:27.702152 17652 leader_election.cc:304] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:27.705765 21649 raft_consensus.cc:493] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:27.747368 16501 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:22:27.720975 17728 leader_election.cc:304] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:27.728204 21616 raft_consensus.cc:515] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:27.574127 21546 raft_consensus.cc:493] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
W20260709 12:22:27.780879 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 81045.4us
W20260709 12:22:27.793293 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 91538.7us
I20260709 12:22:27.821966 21674 consensus_queue.cc:1059] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 8.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:22:27.823196 18334 leader_election.cc:304] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:27.902420 17652 leader_election.cc:304] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:22:27.823168 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 148940us
W20260709 12:22:27.919713 17727 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:27.905263 17687 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 2.021s	user 0.012s	sys 0.037s
W20260709 12:22:27.975605 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 73307.1us
I20260709 12:22:27.976212 21857 raft_consensus.cc:2804] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:22:27.976820 21857 raft_consensus.cc:493] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:27.977274 21857 raft_consensus.cc:3060] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:22:27.977447 17652 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:27.911674 21617 raft_consensus.cc:493] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:27.980183 21617 raft_consensus.cc:515] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:27.800127 21768 raft_consensus.cc:2804] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:22:27.823477 21674 raft_consensus.cc:3055] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 8 LEADER]: Stepping down as leader of term 8
W20260709 12:22:28.012482 17687 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 2.129s	user 0.013s	sys 0.038s
I20260709 12:22:27.935843 21805 raft_consensus.cc:493] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:27.824478 21692 raft_consensus.cc:2804] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:22:27.944718 21835 raft_consensus.cc:493] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:27.894415 21856 raft_consensus.cc:493] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:27.933856 21782 raft_consensus.cc:493] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:22:27.823807 21848 raft_consensus.cc:493] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:27.937338 21810 raft_consensus.cc:493] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:27.997668 21768 raft_consensus.cc:493] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:28.028252 21477 leader_election.cc:290] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:28.001721 21674 raft_consensus.cc:740] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:22:28.040828 17685 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44898 (request call id 847) took 2348 ms. Trace:
I20260709 12:22:27.947113 21858 raft_consensus.cc:493] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
W20260709 12:22:28.040947 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 267692us
I20260709 12:22:28.041522 21805 raft_consensus.cc:515] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:28.042608 21835 raft_consensus.cc:515] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:27.936888 21814 consensus_queue.cc:1059] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.096s
W20260709 12:22:28.040359 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 63866.4us
I20260709 12:22:28.069072 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:27.938153 21820 consensus_queue.cc:1048] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.086s
I20260709 12:22:28.024438 21692 raft_consensus.cc:493] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:22:27.851775 21649 raft_consensus.cc:515] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:28.020437 21928 leader_election.cc:290] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:22:28.076112 17654 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:28.074504 17728 leader_election.cc:304] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:27.861675 21546 raft_consensus.cc:515] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:22:28.043824 21785 consensus_meta.cc:287] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 1.735s	user 0.011s	sys 0.021s
I20260709 12:22:28.055195 21768 raft_consensus.cc:3060] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:28.028621 21857 raft_consensus.cc:515] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:28.111248 21814 consensus_queue.cc:1243] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer 0da169ff82674dc1947055ce43de68a3 log is divergent from this leader: its last log entry 9.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
I20260709 12:22:28.050419 17685 rpcz_store.cc:276] 0709 12:22:25.692253 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:25.695475 (+  3222us) service_pool.cc:224] Handling call
0709 12:22:25.695771 (+   296us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:25.819132 (+123361us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 12:22:25.819178 (+    46us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:25.819238 (+    60us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:22:25.819292 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:27.969535 (+2150243us) spinlock_profiling.cc:243] Waited 123 ms on lock 0x7b54003458d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:22:27.996131 (+ 26596us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:28.040614 (+ 44483us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":8509,"raft.run_cpu_time_us":123,"raft.run_wall_time_us":6981,"spinlock_wait_cycles":282531584}
I20260709 12:22:28.087270 21692 raft_consensus.cc:3060] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Advancing to term 12
W20260709 12:22:28.156221 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 160107us
I20260709 12:22:28.044521 21856 raft_consensus.cc:515] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:22:28.175992 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 54502.4us
I20260709 12:22:28.146553 21928 raft_consensus.cc:493] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:28.202736 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:28.204016 17728 leader_election.cc:304] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:28.205339 21815 raft_consensus.cc:493] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:22:28.205595 17728 leader_election.cc:304] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:28.206099 21815 raft_consensus.cc:515] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:28.206748 17761 raft_consensus.cc:1240] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 8. Current term is 9. Ops: [5.7-8.9]
I20260709 12:22:28.208133 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:28.209148 17761 raft_consensus.cc:2410] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for term 9 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
W20260709 12:22:28.209263 21666 consensus_meta.cc:287] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.534s	user 0.007s	sys 0.008s
I20260709 12:22:28.209960 21666 raft_consensus.cc:2804] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:22:28.210335 21666 raft_consensus.cc:493] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:28.184928 21858 raft_consensus.cc:515] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:28.210783 21666 raft_consensus.cc:3060] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:28.083772 21782 raft_consensus.cc:515] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:28.146739 21824 raft_consensus.cc:515] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:28.244068 21504 raft_consensus.cc:2804] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:22:28.244169 21782 leader_election.cc:290] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:28.244856 21504 raft_consensus.cc:493] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:28.245222 21782 raft_consensus.cc:493] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:28.245434 21504 raft_consensus.cc:3060] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:28.245951 21782 raft_consensus.cc:515] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:28.246227 21786 raft_consensus.cc:2804] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260709 12:22:28.246717 21684 consensus_meta.cc:287] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 1.939s	user 0.029s	sys 0.000s
I20260709 12:22:28.246862 21786 raft_consensus.cc:493] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:28.162294 21810 raft_consensus.cc:515] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:28.143697 21785 raft_consensus.cc:515] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:28.210376 21923 consensus_queue.cc:1048] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:22:28.248306 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:28.178931 21814 raft_consensus.cc:493] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
W20260709 12:22:28.269763 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:28.249191 17762 raft_consensus.cc:1275] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
W20260709 12:22:28.297583 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 74980.8us
I20260709 12:22:28.211431 21616 leader_election.cc:290] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:28.281108 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:28.298794 17763 raft_consensus.cc:2468] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 12.
I20260709 12:22:28.340935 21805 leader_election.cc:290] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:22:28.341395 21845 consensus_meta.cc:287] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 2.012s	user 0.007s	sys 0.021s
I20260709 12:22:28.248750 21922 raft_consensus.cc:493] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:22:28.342146 21820 raft_consensus.cc:2749] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:22:28.249228 21920 raft_consensus.cc:493] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:28.342523 21922 raft_consensus.cc:515] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:28.283231 21856 leader_election.cc:290] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:28.343585 21920 raft_consensus.cc:515] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:22:28.344440 21856 raft_consensus.cc:493] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:28.272450 21674 consensus_queue.cc:260] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 8.10, Last appended by leader: 10, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:28.345225 21856 raft_consensus.cc:515] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:28.347337 21785 leader_election.cc:290] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:28.348412 17759 raft_consensus.cc:2468] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 10.
I20260709 12:22:28.349174 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:28.350654 21814 raft_consensus.cc:515] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:28.351581 17757 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38776 (request call id 2095) took 2505 ms. Trace:
I20260709 12:22:28.353294 21814 leader_election.cc:290] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:28.260432 21918 raft_consensus.cc:493] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:28.250202 21857 leader_election.cc:290] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 11 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:28.355253 21856 leader_election.cc:290] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:28.265404 21649 leader_election.cc:290] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:28.251497 21707 leader_election.cc:290] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:28.249063 21835 leader_election.cc:290] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:28.255620 21858 leader_election.cc:290] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:28.298585 21786 raft_consensus.cc:3060] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:28.361853 17757 rpcz_store.cc:276] 0709 12:22:25.845732 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:25.845870 (+   138us) service_pool.cc:224] Handling call
0709 12:22:25.859854 (+ 13984us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:25.913186 (+ 53332us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:25.913219 (+    33us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:25.913260 (+    41us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:22:25.913298 (+    38us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:28.351107 (+2437809us) spinlock_profiling.cc:243] Waited 53.1 ms on lock 0x7b54000fcdd4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:22:28.351168 (+    61us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:28.351380 (+   212us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":122120576}
I20260709 12:22:28.251014 21546 leader_election.cc:290] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:22:28.371742 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:28.385708 21805 raft_consensus.cc:3055] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:22:28.211071 21848 raft_consensus.cc:515] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:22:28.431785 17652 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:28.432963 21805 raft_consensus.cc:740] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:22:28.432508 21918 raft_consensus.cc:515] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:28.434016 21810 leader_election.cc:290] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:22:28.436753 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 613859us
I20260709 12:22:28.438012 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:28.440759 21649 raft_consensus.cc:493] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:28.379525 21616 raft_consensus.cc:493] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:28.285071 21928 raft_consensus.cc:515] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:28.247228 21684 raft_consensus.cc:515] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:28.454236 21785 raft_consensus.cc:493] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:28.409868 21814 raft_consensus.cc:493] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:28.476061 21820 raft_consensus.cc:493] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:28.477617 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:28.510515 18365 raft_consensus.cc:2468] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 11.
I20260709 12:22:28.484769 21616 raft_consensus.cc:515] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:28.519011 17760 raft_consensus.cc:2368] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for earlier term 9. Current term is 10.
I20260709 12:22:28.485898 21815 leader_election.cc:290] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:28.445323 21856 raft_consensus.cc:493] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:28.487866 17763 raft_consensus.cc:2393] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 11: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:22:28.486299 21649 raft_consensus.cc:515] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:28.496987 21835 raft_consensus.cc:493] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:28.497685 21928 leader_election.cc:290] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:28.497988 21848 leader_election.cc:290] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:28.498209 21785 raft_consensus.cc:515] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:28.498917 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:28.535471 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:28.555366 21805 consensus_queue.cc:260] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:28.556890 21805 raft_consensus.cc:3060] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:28.556953 21858 raft_consensus.cc:493] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:28.557713 21858 raft_consensus.cc:515] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:28.560699 21856 raft_consensus.cc:515] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:28.562278 21918 leader_election.cc:290] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:28.564998 21810 raft_consensus.cc:493] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:28.565526 21844 raft_consensus.cc:2804] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:22:28.566123 21810 raft_consensus.cc:515] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:28.565400 21768 raft_consensus.cc:515] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:22:28.542807 21630 consensus_meta.cc:287] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.932s	user 0.017s	sys 0.001s
I20260709 12:22:28.579715 21928 raft_consensus.cc:493] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:28.579548 21546 raft_consensus.cc:493] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:28.585824 21928 raft_consensus.cc:515] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:28.586769 21546 raft_consensus.cc:515] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:22:28.588222 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:28.589282 21616 leader_election.cc:290] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:28.568032 21814 raft_consensus.cc:515] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:28.592471 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:28.594126 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:28.609297 17759 raft_consensus.cc:3060] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:28.592517 18367 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59902 (request call id 1086) took 2274 ms. Trace:
I20260709 12:22:28.592687 18362 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59902 (request call id 1078) took 2328 ms. Trace:
W20260709 12:22:28.619995 17654 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:28.649829 17685 raft_consensus.cc:2468] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 11.
I20260709 12:22:28.659037 21477 raft_consensus.cc:493] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:28.680475 21477 raft_consensus.cc:515] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:28.680739 21856 leader_election.cc:290] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:28.680843 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:28.681891 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:22:28.662241 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 116929us
I20260709 12:22:28.629522 18367 rpcz_store.cc:276] 0709 12:22:26.318024 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:26.318175 (+   151us) service_pool.cc:224] Handling call
0709 12:22:26.318487 (+   312us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:26.318676 (+   189us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 12:22:28.028302 (+1709626us) log.cc:844] Serialized 14 byte log entry
0709 12:22:28.539972 (+511670us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:28.550318 (+ 10346us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:22:28.592054 (+ 41736us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:28.592153 (+    99us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:28.592323 (+   170us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1743857,"prepare.queue_time_us":1724537,"prepare.run_cpu_time_us":86,"prepare.run_wall_time_us":88,"raft.queue_time_us":109012,"raft.run_cpu_time_us":105,"raft.run_wall_time_us":105,"thread_start_us":360344,"threads_started":1}
I20260709 12:22:28.685781 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:28.686414 18367 raft_consensus.cc:2468] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 10.
I20260709 12:22:28.687554 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:28.565058 21692 raft_consensus.cc:515] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:28.692516 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:28.629812 18362 rpcz_store.cc:276] 0709 12:22:26.240919 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:26.242298 (+  1379us) service_pool.cc:224] Handling call
0709 12:22:26.248346 (+  6048us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:26.248515 (+   169us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 12:22:27.954411 (+1705896us) log.cc:844] Serialized 14 byte log entry
0709 12:22:28.327144 (+372733us) log.cc:844] Serialized 14 byte log entry
0709 12:22:28.392125 (+ 64981us) log.cc:844] Serialized 14 byte log entry
0709 12:22:28.469424 (+ 77299us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:28.487051 (+ 17627us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:22:28.510482 (+ 23431us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:28.524796 (+ 14314us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:28.568875 (+ 44079us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1828206,"prepare.queue_time_us":1809328,"prepare.run_cpu_time_us":188,"prepare.run_wall_time_us":5322,"raft.queue_time_us":519049,"raft.run_cpu_time_us":425,"raft.run_wall_time_us":36971,"thread_start_us":235775,"threads_started":2}
I20260709 12:22:28.702256 21504 raft_consensus.cc:515] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:28.701937 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:28.692994 21835 raft_consensus.cc:515] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:28.703402 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:28.703930 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:28.591773 21616 raft_consensus.cc:493] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:28.705163 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:28.705426 21616 raft_consensus.cc:515] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:28.705718 21835 leader_election.cc:290] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:28.705835 17762 raft_consensus.cc:2468] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 7.
I20260709 12:22:28.612537 21666 raft_consensus.cc:515] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:28.574958 21820 raft_consensus.cc:515] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:28.706903 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:28.707525 21616 leader_election.cc:290] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:28.708250 21666 leader_election.cc:290] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 9 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:28.708439 21820 leader_election.cc:290] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:28.708998 21820 raft_consensus.cc:493] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:28.709545 21820 raft_consensus.cc:515] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:28.593734 21782 leader_election.cc:290] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:28.702705 18365 raft_consensus.cc:2393] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 9: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:22:28.706451 21835 raft_consensus.cc:493] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:28.703332 21692 leader_election.cc:290] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 12 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:28.718427 18362 raft_consensus.cc:2468] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 8.
I20260709 12:22:28.572348 21858 leader_election.cc:290] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:22:28.719018 17727 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:28.591622 21649 leader_election.cc:290] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:28.593966 21785 leader_election.cc:290] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:28.752341 17764 raft_consensus.cc:2393] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 12: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
W20260709 12:22:28.724887 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 81228.7us
W20260709 12:22:28.765010 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 52156us
I20260709 12:22:28.771121 17729 leader_election.cc:304] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:28.769737 18367 raft_consensus.cc:2468] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 10.
I20260709 12:22:28.770183 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:28.796299 21835 raft_consensus.cc:515] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:28.799021 21835 leader_election.cc:290] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:28.800478 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:28.801002 21814 leader_election.cc:290] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:28.801776 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:28.796730 18368 raft_consensus.cc:1240] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 9. Current term is 10. Ops: []
I20260709 12:22:28.711781 21820 leader_election.cc:290] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:28.830974 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:28.831557 18365 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 10.
W20260709 12:22:28.834095 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 45858.7us
I20260709 12:22:28.817662 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:28.820546 21768 leader_election.cc:290] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 11 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:28.834517 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:28.822278 21504 leader_election.cc:290] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 10 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:28.822839 21684 leader_election.cc:290] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 10 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:28.824002 21705 raft_consensus.cc:493] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:28.823946 21630 consensus_queue.cc:1059] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.041s
I20260709 12:22:28.825412 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:28.802495 21814 raft_consensus.cc:493] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:28.769901 21616 raft_consensus.cc:493] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:28.828269 21928 leader_election.cc:290] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:22:28.840811 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 69772.7us
I20260709 12:22:28.842939 17759 raft_consensus.cc:2468] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 11.
I20260709 12:22:28.844062 17652 leader_election.cc:304] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:28.840096 18367 raft_consensus.cc:2468] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 11.
I20260709 12:22:28.817705 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:28.882710 18368 raft_consensus.cc:3060] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:28.826787 21786 raft_consensus.cc:515] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:28.857786 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:28.859632 21810 leader_election.cc:290] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:28.860399 21630 raft_consensus.cc:3055] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:22:28.862567 21589 raft_consensus.cc:493] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:22:28.852553 21707 consensus_queue.cc:1048] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.011s
W20260709 12:22:28.863690 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 97909.9us
I20260709 12:22:28.865109 21649 consensus_queue.cc:1048] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:22:28.865136 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:28.868297 21705 raft_consensus.cc:515] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:28.894980 18361 raft_consensus.cc:2393] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 10: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:22:28.917734 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:28.918709 21589 raft_consensus.cc:515] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:28.920634 18366 raft_consensus.cc:1275] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
I20260709 12:22:28.903631 21630 raft_consensus.cc:740] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:22:28.924222 21616 raft_consensus.cc:515] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:28.925262 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:28.851583 21692 raft_consensus.cc:2804] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Leader pre-election won for term 11
W20260709 12:22:28.932287 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 97441.3us
I20260709 12:22:28.949677 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:28.953567 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:28.953975 21705 leader_election.cc:290] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:28.953687 21630 consensus_queue.cc:260] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.9, Last appended by leader: 9, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:22:28.955495 17710 debug-util.cc:398] Leaking SignalData structure 0x7b08002ae2c0 after lost signal to thread 17621
I20260709 12:22:28.946131 21707 raft_consensus.cc:2804] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Leader pre-election won for term 11
W20260709 12:22:28.988466 17710 debug-util.cc:398] Leaking SignalData structure 0x7b0800133700 after lost signal to thread 17635
I20260709 12:22:28.988969 21630 raft_consensus.cc:3060] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:22:28.989890 17710 debug-util.cc:398] Leaking SignalData structure 0x7b08000bfbc0 after lost signal to thread 17713
I20260709 12:22:28.990022 21707 raft_consensus.cc:493] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:28.990576 21707 raft_consensus.cc:3060] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:28.968122 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:29.007407 21546 leader_election.cc:290] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:28.970515 21589 leader_election.cc:290] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:22:28.971295 21916 consensus_meta.cc:287] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 2.776s	user 0.007s	sys 0.026s
I20260709 12:22:29.049103 17763 raft_consensus.cc:2468] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 10.
I20260709 12:22:29.049387 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:29.050313 17755 raft_consensus.cc:2468] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 9.
I20260709 12:22:29.050932 17755 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38776 (request call id 2104) took 1579 ms. Trace:
I20260709 12:22:29.051317 17755 rpcz_store.cc:276] 0709 12:22:27.471341 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:29.048927 (+1577586us) service_pool.cc:224] Handling call
0709 12:22:29.050844 (+  1917us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:28.971626 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:29.055542 18365 raft_consensus.cc:2468] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 7.
I20260709 12:22:28.973362 21616 leader_election.cc:290] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:28.974049 21692 raft_consensus.cc:493] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:22:28.974566 21786 leader_election.cc:290] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 9 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:28.980515 17652 leader_election.cc:304] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:28.981873 18361 raft_consensus.cc:2393] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 10: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:22:28.969086 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:29.013749 17774 rpcz_store.cc:275] Call kudu.tserver.TabletCopyService.FetchData from 127.0.0.1:36572 (request call id 748) took 1315 ms. Trace:
I20260709 12:22:29.064005 17774 rpcz_store.cc:276] 0709 12:22:27.698357 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:29.002558 (+1304201us) service_pool.cc:224] Handling call
0709 12:22:29.003615 (+  1057us) tablet_copy_source_session.cc:299] Tablet Copy: log segment 1: 739 total bytes read. Total time elapsed: real 0.000s	user 0.000s	sys 0.000s
0709 12:22:29.013490 (+  9875us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:29.072988 21471 raft_consensus.cc:493] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:29.139477 21471 raft_consensus.cc:515] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:29.140448 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:29.140770 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:29.141677 18362 raft_consensus.cc:2410] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for term 9 because replica has last-logged OpId of term: 8 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 8.
I20260709 12:22:29.142059 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:29.043215 18333 leader_election.cc:304] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:22:29.143613 18371 raft_consensus.cc:2393] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 10: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
W20260709 12:22:29.143795 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 100717us
I20260709 12:22:29.145143 21477 leader_election.cc:290] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:29.045075 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:29.145437 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:29.147040 17763 raft_consensus.cc:1240] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 11. Current term is 12. Ops: []
I20260709 12:22:29.047005 17687 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37402 (request call id 2557) took 3265 ms. Trace:
I20260709 12:22:29.076139 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:22:29.074404 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 111505us
I20260709 12:22:29.148979 18365 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 8.
I20260709 12:22:29.148658 21616 consensus_queue.cc:1048] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:22:29.149883 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:29.150038 17652 leader_election.cc:304] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:29.150223 18333 leader_election.cc:304] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:29.151062 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:29.151892 21546 raft_consensus.cc:2804] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:22:29.146597 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:29.152324 21546 raft_consensus.cc:493] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
W20260709 12:22:29.152729 17727 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:29.153311 17652 leader_election.cc:304] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:29.153469 21835 raft_consensus.cc:2804] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:22:29.143940 17764 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38776 (request call id 2113) took 1243 ms. Trace:
I20260709 12:22:29.075067 21589 raft_consensus.cc:2804] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:22:29.077371 21692 raft_consensus.cc:3060] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:29.154176 17764 rpcz_store.cc:276] 0709 12:22:27.900897 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:28.348686 (+447789us) service_pool.cc:224] Handling call
0709 12:22:29.143714 (+795028us) spinlock_profiling.cc:243] Waited 362 ms on lock 0x7b54003e62d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:22:29.143862 (+   148us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":833219840}
I20260709 12:22:29.154570 21589 raft_consensus.cc:697] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 12 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
W20260709 12:22:29.155001 17760 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38776 (request call id 2106) took 1682 ms (client timeout 1731 ms). Trace:
I20260709 12:22:29.155510 21830 raft_consensus.cc:2804] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:22:29.152724 21546 raft_consensus.cc:3060] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:29.156085 21830 raft_consensus.cc:697] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 11 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
W20260709 12:22:29.155527 17760 rpcz_store.cc:269] 0709 12:22:27.472299 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:27.479118 (+  6819us) service_pool.cc:224] Handling call
0709 12:22:29.154793 (+1675675us) spinlock_profiling.cc:243] Waited 1.01 s on lock 0x7b5400310e54. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:22:29.154922 (+   129us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2313900416}
I20260709 12:22:29.148262 17687 rpcz_store.cc:276] 0709 12:22:25.781886 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:25.782007 (+   121us) service_pool.cc:224] Handling call
0709 12:22:25.793752 (+ 11745us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:25.883554 (+ 89802us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:22:28.032872 (+2149318us) log.cc:844] Serialized 14 byte log entry
0709 12:22:28.404594 (+371722us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:28.454555 (+ 49961us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:22:28.475430 (+ 20875us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:29.046728 (+571298us) spinlock_profiling.cc:243] Waited 89.5 ms on lock 0x7b5400338954. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:22:29.046778 (+    50us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:29.046944 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":73289,"prepare.queue_time_us":304338,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":43,"raft.queue_time_us":29595,"raft.run_cpu_time_us":168,"raft.run_wall_time_us":36935,"spinlock_wait_cycles":205932800,"thread_start_us":2445762,"threads_started":2}
I20260709 12:22:29.159477 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:29.155901 21589 consensus_queue.cc:237] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.8, Last appended by leader: 8, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:29.160863 21830 consensus_queue.cc:237] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.7, Last appended by leader: 7, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:29.142640 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:29.052275 18374 raft_consensus.cc:1275] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
W20260709 12:22:29.080823 17655 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:29.102727 21785 raft_consensus.cc:493] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:29.103410 21810 raft_consensus.cc:493] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:29.172134 21785 raft_consensus.cc:515] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:29.172480 21810 raft_consensus.cc:515] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:29.173305 17680 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37402 (request call id 2558) took 3329 ms. Trace:
I20260709 12:22:29.174887 21810 leader_election.cc:290] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:29.173729 17680 rpcz_store.cc:276] 0709 12:22:25.843985 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:25.844135 (+   150us) service_pool.cc:224] Handling call
0709 12:22:25.868710 (+ 24575us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:25.901356 (+ 32646us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:22:27.366561 (+1465205us) log.cc:844] Serialized 14 byte log entry
0709 12:22:27.593442 (+226881us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:27.593637 (+   195us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:22:27.593843 (+   206us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:29.172970 (+1579127us) spinlock_profiling.cc:243] Waited 30.6 ms on lock 0x7b54000c5654. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:22:29.173021 (+    51us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:29.173205 (+   184us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":286799,"prepare.run_cpu_time_us":25,"prepare.run_wall_time_us":26,"raft.queue_time_us":9357,"raft.run_cpu_time_us":268,"raft.run_wall_time_us":272,"spinlock_wait_cycles":70345600,"thread_start_us":1615270,"threads_started":2}
I20260709 12:22:29.175693 21785 leader_election.cc:290] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:29.176558 17688 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37396 (request call id 2175) took 1722 ms. Trace:
I20260709 12:22:29.177047 17688 rpcz_store.cc:276] 0709 12:22:27.454407 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:27.506840 (+ 52433us) service_pool.cc:224] Handling call
0709 12:22:29.176362 (+1669522us) spinlock_profiling.cc:243] Waited 79.3 ms on lock 0x7b54001aa754. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:22:29.176492 (+   130us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":182276480}
I20260709 12:22:29.177632 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 10 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:29.156024 21649 raft_consensus.cc:2804] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:22:29.179400 21649 raft_consensus.cc:493] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:29.179709 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:29.178675 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:22:29.102105 17802 connection.cc:256] RPC call timeout handler was delayed by -2.0232954109997081s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1592s for a response
I20260709 12:22:29.123795 21844 raft_consensus.cc:493] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
W20260709 12:22:29.127352 17756 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 2.931s	user 0.027s	sys 0.060s
I20260709 12:22:29.130628 17758 raft_consensus.cc:1240] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 6. Current term is 7. Ops: []
I20260709 12:22:29.162823 18362 raft_consensus.cc:2468] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 14.
I20260709 12:22:29.179848 21649 raft_consensus.cc:3060] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:29.184671 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:29.184811 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 11 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:29.185388 18374 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 8.
W20260709 12:22:29.186110 17802 connection.cc:256] RPC call timeout handler was delayed by -1.5692613539999911s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15100000000000002s for a response
I20260709 12:22:29.186203 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:29.186419 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:22:29.186618 17802 connection.cc:256] RPC call timeout handler was delayed by -1.4937806459997773s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1549s for a response
I20260709 12:22:29.186892 18362 raft_consensus.cc:3060] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:22:29.186997 17802 connection.cc:256] RPC call timeout handler was delayed by -1.4533383190000677s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1507s for a response
W20260709 12:22:29.187474 17802 connection.cc:256] RPC call timeout handler was delayed by -1.2906954109998878s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.24060000000000004s for a response
W20260709 12:22:29.187851 17802 connection.cc:256] RPC call timeout handler was delayed by -1.0221994929997891s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2006s for a response
I20260709 12:22:29.188235 21952 raft_consensus.cc:493] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:29.188884 21952 raft_consensus.cc:515] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:22:29.182637 17756 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 2.987s	user 0.028s	sys 0.061s
I20260709 12:22:29.054687 21920 leader_election.cc:290] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:29.133919 17761 raft_consensus.cc:1275] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260709 12:22:29.135584 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:29.136528 17837 raft_consensus.cc:1240] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 9. Current term is 10. Ops: []
I20260709 12:22:29.153973 21835 raft_consensus.cc:493] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:29.078195 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:29.151825 18365 raft_consensus.cc:2393] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 9: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:22:28.962821 18367 raft_consensus.cc:2468] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 6.
I20260709 12:22:29.070276 21858 raft_consensus.cc:493] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:29.132421 17762 raft_consensus.cc:1275] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 8 index: 8. (term mismatch)
I20260709 12:22:29.214783 18366 raft_consensus.cc:2468] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 7.
I20260709 12:22:29.230542 17762 pending_rounds.cc:85] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb: Aborting all ops after (but not including) 7
I20260709 12:22:29.233613 17762 pending_rounds.cc:107] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb: Aborting uncommitted NO_OP operation due to leader change: 7.8
I20260709 12:22:29.233892 17762 raft_consensus.cc:2889] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:29.235659 18334 leader_election.cc:304] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:22:29.187729 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 11 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:29.181476 18361 raft_consensus.cc:2468] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 8.
I20260709 12:22:29.154184 17652 leader_election.cc:304] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:29.212658 18368 raft_consensus.cc:2468] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 8.
I20260709 12:22:29.216574 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 12 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:29.182379 21844 raft_consensus.cc:3060] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:29.159597 21674 raft_consensus.cc:3060] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:22:29.257000 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:29.257474 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:29.257814 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:29.227241 17755 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 10.
I20260709 12:22:29.227732 21952 leader_election.cc:290] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:29.214565 21786 consensus_queue.cc:1059] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 9.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.014s
I20260709 12:22:29.229352 21785 raft_consensus.cc:493] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:29.251140 17729 leader_election.cc:304] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:29.251929 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:29.277259 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:29.181753 21888 raft_consensus.cc:493] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:22:29.254629 21858 raft_consensus.cc:515] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:22:29.260579 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 110483us
I20260709 12:22:29.261535 18362 raft_consensus.cc:2468] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 11.
I20260709 12:22:29.263815 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:22:29.276839 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 122709us
I20260709 12:22:29.227766 21835 raft_consensus.cc:3060] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:29.287504 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:29.287628 21926 raft_consensus.cc:493] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:22:29.310518 21926 raft_consensus.cc:515] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:29.312022 21785 raft_consensus.cc:515] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:29.313155 21858 leader_election.cc:290] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:29.286595 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:29.294631 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:22:29.299836 17710 debug-util.cc:398] Leaking SignalData structure 0x7b08001f0480 after lost signal to thread 17788
I20260709 12:22:29.307822 21888 raft_consensus.cc:515] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:29.308256 21800 raft_consensus.cc:493] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
W20260709 12:22:29.322821 17710 debug-util.cc:398] Leaking SignalData structure 0x7b08001f06e0 after lost signal to thread 17789
I20260709 12:22:29.309372 17684 raft_consensus.cc:1275] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260709 12:22:29.309682 21786 raft_consensus.cc:3055] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:22:28.929507 21814 raft_consensus.cc:515] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:29.308655 21952 raft_consensus.cc:493] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:29.329954 17761 raft_consensus.cc:2468] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 10.
I20260709 12:22:29.330487 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:29.341048 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:22:29.346495 17710 debug-util.cc:398] Leaking SignalData structure 0x7b08003c2500 after lost signal to thread 17862
W20260709 12:22:29.347766 17710 debug-util.cc:398] Leaking SignalData structure 0x7b08003917e0 after lost signal to thread 18409
I20260709 12:22:29.348474 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:29.349301 18367 raft_consensus.cc:1240] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 9. Current term is 10. Ops: []
I20260709 12:22:29.337860 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:22:29.345338 17652 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:29.358836 17710 debug-util.cc:398] Leaking SignalData structure 0x7b08003af080 after lost signal to thread 18410
I20260709 12:22:29.363304 18374 raft_consensus.cc:1275] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 11 index: 9. (index mismatch)
I20260709 12:22:29.370014 21827 raft_consensus.cc:493] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:29.321157 21768 raft_consensus.cc:2804] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Leader pre-election won for term 10
W20260709 12:22:29.376564 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 135177us
I20260709 12:22:29.370729 17680 raft_consensus.cc:2468] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 9.
I20260709 12:22:29.314958 21920 raft_consensus.cc:493] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:22:29.332449 21786 raft_consensus.cc:740] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 9 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:22:29.373271 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:22:29.390209 17710 debug-util.cc:398] Leaking SignalData structure 0x7b08002aa300 after lost signal to thread 21860
I20260709 12:22:29.413022 21919 raft_consensus.cc:3060] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:29.329999 21965 raft_consensus.cc:2804] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:22:29.414587 18367 raft_consensus.cc:1240] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 9 LEADER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 8. Current term is 9. Ops: []
I20260709 12:22:29.415360 17755 raft_consensus.cc:2393] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 10: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:22:29.419306 21920 raft_consensus.cc:515] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:29.422001 18362 raft_consensus.cc:1240] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 11 LEADER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 10. Current term is 11. Ops: [10.9-10.9]
I20260709 12:22:29.431664 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 11 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:22:29.437279 17682 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.567s	user 0.011s	sys 0.004s
W20260709 12:22:29.437770 17682 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.568s	user 0.011s	sys 0.004s
I20260709 12:22:29.445359 17761 raft_consensus.cc:2410] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 11 because replica has last-logged OpId of term: 9 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 8.
I20260709 12:22:29.413437 21786 consensus_queue.cc:260] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:29.452997 18362 raft_consensus.cc:1275] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 9 index: 8. (term mismatch)
I20260709 12:22:29.443847 17652 leader_election.cc:400] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 9 election: Vote denied by peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651) with higher term. Message: Invalid argument: T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for earlier term 9. Current term is 10.
I20260709 12:22:29.466624 17652 leader_election.cc:403] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 9 election: Cancelling election due to peer responding with higher term
I20260709 12:22:29.455583 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:29.456272 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:29.391839 21768 raft_consensus.cc:493] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:29.380827 18374 raft_consensus.cc:1240] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 8. Current term is 9. Ops: []
I20260709 12:22:29.475425 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:29.466289 18362 pending_rounds.cc:85] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da: Aborting all ops after (but not including) 7
I20260709 12:22:29.481190 18362 pending_rounds.cc:107] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da: Aborting uncommitted NO_OP operation due to leader change: 8.8
I20260709 12:22:29.481587 18362 raft_consensus.cc:2889] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:29.482532 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:29.483729 17761 raft_consensus.cc:2393] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 11: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
W20260709 12:22:29.488942 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 117090us
I20260709 12:22:29.341266 21800 raft_consensus.cc:515] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:29.480682 21916 consensus_queue.cc:1048] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.016s
I20260709 12:22:29.480792 21768 raft_consensus.cc:3060] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:29.498984 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:29.433209 21965 raft_consensus.cc:493] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
W20260709 12:22:29.497452 17804 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.899s	user 0.053s	sys 0.076s
W20260709 12:22:29.526204 17804 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.039s	user 0.054s	sys 0.076s
I20260709 12:22:29.510910 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:29.405191 21707 raft_consensus.cc:515] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:29.406280 21952 raft_consensus.cc:515] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:29.461053 21810 raft_consensus.cc:493] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:29.528695 17761 raft_consensus.cc:2468] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 10.
I20260709 12:22:29.526875 21858 raft_consensus.cc:493] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:29.559561 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:29.560376 17761 raft_consensus.cc:3060] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:29.514530 21705 raft_consensus.cc:3060] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:29.564406 17652 leader_election.cc:304] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:22:29.576530 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 181758us
I20260709 12:22:29.585606 21471 leader_election.cc:290] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:29.587316 21920 leader_election.cc:290] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:29.593148 21922 leader_election.cc:290] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:29.594671 21546 raft_consensus.cc:515] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:29.596093 21858 raft_consensus.cc:515] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:29.598280 21858 leader_election.cc:290] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:29.598793 21858 raft_consensus.cc:493] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:29.538641 21888 leader_election.cc:290] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:29.600977 21888 raft_consensus.cc:2804] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:22:29.601452 21888 raft_consensus.cc:493] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:22:29.601924 21888 raft_consensus.cc:3060] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:29.613695 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 12 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:29.528192 21965 raft_consensus.cc:3060] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:29.601696 18333 leader_election.cc:304] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:29.625439 17684 raft_consensus.cc:2410] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 12 because replica has last-logged OpId of term: 9 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 7.
I20260709 12:22:29.599380 21858 raft_consensus.cc:515] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:29.637645 18333 leader_election.cc:304] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:29.461452 21827 raft_consensus.cc:515] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:29.559270 21786 raft_consensus.cc:3060] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:29.669737 17761 raft_consensus.cc:2468] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 11.
W20260709 12:22:29.676453 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:29.685918 21844 raft_consensus.cc:515] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:29.699213 21955 raft_consensus.cc:493] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:29.700418 21955 raft_consensus.cc:515] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:29.700886 21546 leader_election.cc:290] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 8 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:29.702230 21814 leader_election.cc:290] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:29.702610 21955 leader_election.cc:290] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:29.704242 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:29.705956 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 13 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:29.707500 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:29.708403 17761 raft_consensus.cc:2410] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 10 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 4.
I20260709 12:22:29.689476 17652 leader_election.cc:304] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:22:29.709933 17727 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:29.716146 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 12 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:29.700928 17756 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36572 (request call id 746) took 3544 ms. Trace:
W20260709 12:22:29.728281 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:29.728199 18362 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 1355) took 1018 ms. Trace:
I20260709 12:22:29.728997 18362 rpcz_store.cc:276] 0709 12:22:28.710133 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:29.613109 (+902976us) service_pool.cc:224] Handling call
0709 12:22:29.657727 (+ 44618us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:29.676142 (+ 18415us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:29.676169 (+    27us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:29.709095 (+ 32926us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:22:29.727912 (+ 18817us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:29.727984 (+    72us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:29.728130 (+   146us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:29.738232 21707 leader_election.cc:290] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 11 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:29.742591 17684 raft_consensus.cc:1275] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260709 12:22:29.708041 21958 raft_consensus.cc:2749] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
W20260709 12:22:29.713392 21649 consensus_meta.cc:287] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.523s	user 0.008s	sys 0.000s
I20260709 12:22:29.785869 21649 raft_consensus.cc:515] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:29.788591 17833 raft_consensus.cc:1275] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 11 index: 6. (index mismatch)
I20260709 12:22:29.603156 21810 raft_consensus.cc:515] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:29.793794 17834 raft_consensus.cc:1240] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 10. Current term is 11. Ops: [9.8-10.9]
I20260709 12:22:29.794541 21810 leader_election.cc:290] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:22:29.795322 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:29.760793 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:29.796379 17831 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 10.
I20260709 12:22:29.739742 21546 raft_consensus.cc:2804] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:22:29.766845 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:29.768528 17836 raft_consensus.cc:1275] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20260709 12:22:29.768994 21952 leader_election.cc:290] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:22:29.773074 17727 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:29.781590 21946 log.cc:927] Time spent T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.350s	user 0.001s	sys 0.001s
I20260709 12:22:29.782998 21858 leader_election.cc:290] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:29.785054 17835 raft_consensus.cc:1275] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20260709 12:22:29.676424 21835 raft_consensus.cc:515] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:29.804864 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:29.805627 21546 raft_consensus.cc:697] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 11 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:22:29.806583 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:29.806697 21546 consensus_queue.cc:237] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.9, Last appended by leader: 9, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:29.807147 21835 leader_election.cc:290] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 9 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:29.808238 17838 raft_consensus.cc:2393] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 11: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:22:29.808303 17836 raft_consensus.cc:1275] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
I20260709 12:22:29.813585 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:29.804862 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:29.826828 18367 raft_consensus.cc:1240] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 10. Current term is 11. Ops: []
I20260709 12:22:29.833190 18362 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 1356) took 1122 ms. Trace:
I20260709 12:22:29.833621 18362 rpcz_store.cc:276] 0709 12:22:28.710384 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:29.754229 (+1043845us) service_pool.cc:224] Handling call
0709 12:22:29.815687 (+ 61458us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:29.832733 (+ 17046us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:29.832776 (+    43us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:29.832835 (+    59us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:22:29.832895 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:29.832977 (+    82us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:29.833119 (+   142us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:29.726485 17756 rpcz_store.cc:276] 0709 12:22:26.156479 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:26.156620 (+   141us) service_pool.cc:224] Handling call
0709 12:22:26.156927 (+   307us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:26.194887 (+ 37960us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:26.194920 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:26.195436 (+   516us) log.cc:844] Serialized 26 byte log entry
0709 12:22:29.252523 (+3057087us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:22:29.252586 (+    63us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:29.252714 (+   128us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:29.700719 (+448005us) raft_consensus.cc:1685] finished
0709 12:22:29.700741 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:29.700867 (+   126us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":37574,"raft.queue_time_us":38771,"raft.run_cpu_time_us":196,"raft.run_wall_time_us":5289,"thread_start_us":2995655,"threads_started":1,"wal-append.queue_time_us":3363032}
I20260709 12:22:29.839389 18364 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 1357) took 1125 ms. Trace:
I20260709 12:22:29.839869 18364 rpcz_store.cc:276] 0709 12:22:28.710650 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:29.823721 (+1113071us) service_pool.cc:224] Handling call
0709 12:22:29.824117 (+   396us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:29.835439 (+ 11322us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:29.835476 (+    37us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:29.835532 (+    56us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:22:29.835580 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:29.835644 (+    64us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:29.835763 (+   119us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:29.840667 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:29.841499 18364 raft_consensus.cc:1275] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
W20260709 12:22:29.824671 21692 consensus_meta.cc:287] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.670s	user 0.013s	sys 0.000s
I20260709 12:22:29.843713 17684 raft_consensus.cc:1240] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 10. Current term is 11. Ops: []
W20260709 12:22:29.865557 17652 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:29.925875 18362 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 1358) took 1215 ms. Trace:
I20260709 12:22:29.926362 18362 rpcz_store.cc:276] 0709 12:22:28.710932 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:29.834559 (+1123627us) service_pool.cc:224] Handling call
0709 12:22:29.849160 (+ 14601us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:29.925513 (+ 76353us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:29.925543 (+    30us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:29.925590 (+    47us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:22:29.925635 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:29.925694 (+    59us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:29.925819 (+   125us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:29.917323 17833 raft_consensus.cc:1240] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 11. Current term is 12. Ops: []
W20260709 12:22:29.940172 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 61618.4us
W20260709 12:22:29.944235 18365 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.714s	user 0.068s	sys 0.086s
W20260709 12:22:29.944742 18365 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.715s	user 0.068s	sys 0.086s
I20260709 12:22:29.946128 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:29.924612 21919 raft_consensus.cc:2749] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:22:29.947064 17829 raft_consensus.cc:2468] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 9.
I20260709 12:22:29.847662 21692 raft_consensus.cc:515] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:29.940623 21961 raft_consensus.cc:493] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:29.963501 17756 raft_consensus.cc:1240] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 9. Current term is 10. Ops: []
I20260709 12:22:30.022490 17757 raft_consensus.cc:1240] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 10. Current term is 11. Ops: []
W20260709 12:22:30.062945 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 60661.3us
W20260709 12:22:29.958567 21674 consensus_meta.cc:287] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.703s	user 0.004s	sys 0.009s
I20260709 12:22:30.072873 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:30.079377 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
W20260709 12:22:30.064827 21940 log.cc:927] Time spent T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.217s	user 0.000s	sys 0.001s
I20260709 12:22:30.085747 17831 raft_consensus.cc:2393] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 12: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:22:30.065389 21827 leader_election.cc:290] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:30.071172 21707 raft_consensus.cc:493] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:30.075241 21705 raft_consensus.cc:2749] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Leader election lost for term 9. Reason: Vote denied by peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651) with higher term. Message: Invalid argument: T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for earlier term 9. Current term is 10.
I20260709 12:22:30.087809 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:30.074994 21649 leader_election.cc:290] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 11 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:30.084723 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:30.092121 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:30.100014 17829 raft_consensus.cc:2393] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 9: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:22:30.021847 21961 raft_consensus.cc:515] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:30.089630 21707 raft_consensus.cc:515] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:30.126159 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:30.129356 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:22:30.156297 17801 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.655s	user 0.016s	sys 0.036s
W20260709 12:22:30.156793 17801 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.843s	user 0.017s	sys 0.036s
W20260709 12:22:30.158025 17801 connection.cc:256] RPC call timeout handler was delayed by -1.8774060829996415s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1549s for a response
W20260709 12:22:30.158367 17801 connection.cc:256] RPC call timeout handler was delayed by -1.8621060829996168s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15660000000000002s for a response
W20260709 12:22:30.158648 17801 connection.cc:256] RPC call timeout handler was delayed by -1.8252060829995571s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1607s for a response
W20260709 12:22:30.158988 17801 connection.cc:256] RPC call timeout handler was delayed by -1.6011060829996495s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18560000000000001s for a response
I20260709 12:22:30.159129 21858 raft_consensus.cc:493] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:30.159327 17801 connection.cc:256] RPC call timeout handler was delayed by -1.5948060829996393s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18630000000000002s for a response
W20260709 12:22:30.159691 17801 connection.cc:256] RPC call timeout handler was delayed by -1.4661060829994312s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2006s for a response
I20260709 12:22:30.174237 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 11 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:22:30.163972 18363 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.937s	user 0.053s	sys 0.148s
W20260709 12:22:30.190485 18363 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.028s	user 0.053s	sys 0.152s
W20260709 12:22:30.189765 17801 connection.cc:256] RPC call timeout handler was delayed by -1.4562060829994152s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2017s for a response
I20260709 12:22:30.166676 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:22:30.191819 17801 connection.cc:256] RPC call timeout handler was delayed by -1.4256060829998205s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20510000000000003s for a response
W20260709 12:22:30.192477 17801 connection.cc:256] RPC call timeout handler was delayed by -1.3473060829996939s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2138s for a response
I20260709 12:22:30.176947 17829 raft_consensus.cc:1275] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 12 index: 8. (index mismatch)
I20260709 12:22:30.177528 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:30.143564 17831 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 8.
W20260709 12:22:30.166144 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:30.196357 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:30.190407 17836 raft_consensus.cc:2468] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 9.
I20260709 12:22:30.159720 21858 raft_consensus.cc:515] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:30.198334 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:30.142679 17834 raft_consensus.cc:2468] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 8.
I20260709 12:22:30.174284 17682 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37396 (request call id 2182) took 1777 ms. Trace:
W20260709 12:22:30.192900 17801 connection.cc:256] RPC call timeout handler was delayed by -1.3410060829996837s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21450000000000002s for a response
W20260709 12:22:30.200515 17801 connection.cc:256] RPC call timeout handler was delayed by -1.296006082999611s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2195s for a response
W20260709 12:22:30.200968 17801 connection.cc:256] RPC call timeout handler was delayed by -1.1061060829997587s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.24060000000000004s for a response
I20260709 12:22:30.198999 17836 raft_consensus.cc:2468] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 10.
I20260709 12:22:30.178187 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
W20260709 12:22:30.204679 17801 connection.cc:256] RPC call timeout handler was delayed by -1.0170060829996146s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2505s for a response
I20260709 12:22:30.192617 17838 raft_consensus.cc:2468] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 11.
W20260709 12:22:30.160885 21888 consensus_meta.cc:287] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.558s	user 0.014s	sys 0.000s
W20260709 12:22:30.208217 17729 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.755s	user 0.038s	sys 0.070s
W20260709 12:22:30.168720 21956 log.cc:927] Time spent T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.096s	user 0.000s	sys 0.000s
W20260709 12:22:30.164964 21954 log.cc:927] Time spent T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.084s	user 0.000s	sys 0.000s
I20260709 12:22:30.199810 17682 rpcz_store.cc:276] 0709 12:22:28.396945 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:28.397102 (+   157us) service_pool.cc:224] Handling call
0709 12:22:28.868538 (+471436us) spinlock_profiling.cc:243] Waited 3.94 ms on lock 0x7b5800002a98. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b474fbb71 00007f7b474fba22 00007f7b504148d9 00007f7b4f56eee0 00007f7b4f6380ab 00007f7b4f638131 00007f7b4f5f9fc0 00007f7b4f5e4929 00007f7b4f5e43d2 00007f7b48b5bbd4
0709 12:22:28.868745 (+   207us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:22:28.869170 (+   425us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:28.869198 (+    28us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:22:28.869935 (+   737us) log.cc:844] Serialized 52 byte log entry
0709 12:22:29.438250 (+568315us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:22:29.438310 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:29.438398 (+    88us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:30.173938 (+735540us) raft_consensus.cc:1685] finished
0709 12:22:30.173963 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:30.174197 (+   234us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":140,"raft.run_cpu_time_us":110,"raft.run_wall_time_us":110,"spinlock_wait_cycles":9067904,"thread_start_us":567856,"threads_started":1,"wal-append.queue_time_us":1221442}
I20260709 12:22:30.211628 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:30.160012 21952 raft_consensus.cc:493] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:30.211212 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:30.215569 17829 raft_consensus.cc:2468] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 9.
I20260709 12:22:30.215952 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:30.215601 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:22:30.213336 17681 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.381s	user 0.021s	sys 0.046s
W20260709 12:22:30.219066 17681 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.387s	user 0.022s	sys 0.047s
I20260709 12:22:30.225742 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:30.226554 17831 raft_consensus.cc:3060] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:30.235736 21707 leader_election.cc:290] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:30.243662 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:22:30.247766 17727 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:30.244634 17833 raft_consensus.cc:2468] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 10.
I20260709 12:22:30.212422 21888 raft_consensus.cc:515] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:22:30.268640 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 54756.8us
I20260709 12:22:30.269438 21692 leader_election.cc:290] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 11 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:30.269847 17802 leader_election.cc:304] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:30.270355 21888 leader_election.cc:290] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 11 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:30.271303 17802 leader_election.cc:304] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:22:30.268347 17832 raft_consensus.cc:2468] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 9.
W20260709 12:22:30.279557 21786 consensus_meta.cc:287] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.566s	user 0.010s	sys 0.001s
I20260709 12:22:30.283545 17801 leader_election.cc:304] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:30.284797 17831 raft_consensus.cc:2468] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 10.
W20260709 12:22:30.287308 17729 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.760s	user 0.039s	sys 0.070s
W20260709 12:22:30.288585 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 835258us
I20260709 12:22:30.292689 17755 raft_consensus.cc:2468] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 10.
W20260709 12:22:30.303169 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:30.310266 17837 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35042 (request call id 2060) took 3970 ms. Trace:
I20260709 12:22:30.310267 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:22:30.318924 17683 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37396 (request call id 2181) took 1950 ms (client timeout 2058 ms). Trace:
W20260709 12:22:30.319563 17683 rpcz_store.cc:269] 0709 12:22:28.368316 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:28.368785 (+   469us) service_pool.cc:224] Handling call
0709 12:22:30.310108 (+1941323us) spinlock_profiling.cc:243] Waited 3.61 ms on lock 0x7b5800002a98. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b474fbb71 00007f7b474fbeee 000000000036eb39 00007f7b4f56f1e0 00007f7b4f6380c4 00007f7b4f638131 00007f7b4f5e60a0 00007f7b4f5e5984 00007f7b48b606d4 00007f7b48b60662
0709 12:22:30.318853 (+  8745us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":8301952}
I20260709 12:22:30.317341 17837 rpcz_store.cc:276] 0709 12:22:26.324125 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:26.324257 (+   132us) service_pool.cc:224] Handling call
0709 12:22:26.324582 (+   325us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:29.228625 (+2904043us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:30.266807 (+1038182us) spinlock_profiling.cc:243] Waited 2.81 s on lock 0x7b54003359d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:22:30.294027 (+ 27220us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":6466035072}
W20260709 12:22:30.332548 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 120793us
I20260709 12:22:30.333024 21858 leader_election.cc:290] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:30.288516 21967 raft_consensus.cc:2749] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:22:30.235741 21952 raft_consensus.cc:515] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:30.309689 21810 raft_consensus.cc:2804] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Leader pre-election won for term 11
W20260709 12:22:30.339784 17710 debug-util.cc:398] Leaking SignalData structure 0x7b080037fb20 after lost signal to thread 21905
W20260709 12:22:30.341274 17710 debug-util.cc:398] Leaking SignalData structure 0x7b0800403aa0 after lost signal to thread 21917
W20260709 12:22:30.341626 17710 debug-util.cc:398] Leaking SignalData structure 0x7b08002907a0 after lost signal to thread 21921
I20260709 12:22:30.344408 17801 leader_election.cc:304] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:30.358604 17729 leader_election.cc:304] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:30.344338 21810 raft_consensus.cc:493] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:22:30.368875 17728 connection.cc:256] RPC call timeout handler was delayed by -1.2880143259999386s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 1.5s for a response
W20260709 12:22:30.358750 21589 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.813s	user 0.012s	sys 0.072s
I20260709 12:22:30.368076 21810 raft_consensus.cc:3060] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:30.374356 17837 raft_consensus.cc:1275] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20260709 12:22:30.374233 17831 raft_consensus.cc:1275] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
I20260709 12:22:30.373121 21705 raft_consensus.cc:493] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:30.377108 21973 raft_consensus.cc:2749] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:22:30.377604 21705 raft_consensus.cc:515] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:30.380532 21705 leader_election.cc:290] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:22:30.381006 21768 consensus_meta.cc:287] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.842s	user 0.006s	sys 0.006s
I20260709 12:22:30.384650 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:30.381374 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:30.385501 17687 raft_consensus.cc:2468] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 14.
I20260709 12:22:30.385790 17689 raft_consensus.cc:2468] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 9.
W20260709 12:22:30.386191 17687 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44898 (request call id 857) took 1222 ms (client timeout 1547 ms). Trace:
I20260709 12:22:30.386305 18371 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 1339) took 1789 ms. Trace:
I20260709 12:22:30.386988 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:30.381587 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:30.377307 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:30.382361 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:30.389161 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:30.389672 21800 leader_election.cc:290] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:30.389945 17685 raft_consensus.cc:2468] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 7.
I20260709 12:22:30.391203 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:30.392127 21824 leader_election.cc:290] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 7 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:30.392728 21824 raft_consensus.cc:493] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:30.393229 21824 raft_consensus.cc:515] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:30.383158 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:30.386765 18371 rpcz_store.cc:276] 0709 12:22:28.596957 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:29.347563 (+750606us) service_pool.cc:224] Handling call
0709 12:22:30.385026 (+1037463us) spinlock_profiling.cc:243] Waited 12 ms on lock 0x7b5800102398. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b474fbb71 00007f7b474fba22 00007f7b504148d9 00007f7b4f56eee0 00007f7b4f6380ab 00007f7b4f638131 00007f7b4f5f9fc0 00007f7b4f5e4929 00007f7b4f5e43d2 00007f7b48b5bbd4
0709 12:22:30.385379 (+   353us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:30.385865 (+   486us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 12:22:30.385914 (+    49us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:30.385967 (+    53us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:22:30.386051 (+    84us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:30.386121 (+    70us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:30.386252 (+   131us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":27569536}
I20260709 12:22:30.387754 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:22:30.371902 21589 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.826s	user 0.013s	sys 0.073s
I20260709 12:22:30.396800 17684 raft_consensus.cc:2468] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 8.
I20260709 12:22:30.370471 21692 raft_consensus.cc:2804] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:22:30.366730 21967 raft_consensus.cc:2749] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260709 12:22:30.386695 17687 rpcz_store.cc:269] 0709 12:22:29.163780 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:30.384319 (+1220539us) service_pool.cc:224] Handling call
0709 12:22:30.386122 (+  1803us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:22:30.398509 21830 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.208s	user 0.014s	sys 0.036s
W20260709 12:22:30.400179 21965 consensus_meta.cc:287] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.712s	user 0.007s	sys 0.007s
I20260709 12:22:30.418493 17689 pending_rounds.cc:85] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3: Aborting all ops after (but not including) 8
I20260709 12:22:30.418848 17654 leader_election.cc:304] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:22:30.408799 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 69517.4us
W20260709 12:22:30.414247 18374 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.811s	user 0.003s	sys 0.044s
I20260709 12:22:30.416906 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 11 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:30.422564 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:30.423861 17689 pending_rounds.cc:107] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3: Aborting uncommitted NO_OP operation due to leader change: 8.9
W20260709 12:22:30.424226 18374 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.821s	user 0.003s	sys 0.044s
I20260709 12:22:30.424332 17689 raft_consensus.cc:2889] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:30.373224 21786 consensus_queue.cc:1048] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.047s
I20260709 12:22:30.426501 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:30.430816 17756 raft_consensus.cc:2393] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 7: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:22:30.430693 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:30.427570 17682 raft_consensus.cc:1275] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 9 index: 9. (term mismatch)
I20260709 12:22:30.431741 17682 pending_rounds.cc:85] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3: Aborting all ops after (but not including) 8
I20260709 12:22:30.432080 17682 pending_rounds.cc:107] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3: Aborting uncommitted NO_OP operation due to leader change: 8.9
I20260709 12:22:30.432260 17654 leader_election.cc:304] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:30.432353 17682 raft_consensus.cc:2889] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:22:30.405961 21830 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.216s	user 0.015s	sys 0.037s
I20260709 12:22:30.434695 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:30.400435 21692 raft_consensus.cc:493] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:30.420670 21919 raft_consensus.cc:493] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:30.384608 21768 raft_consensus.cc:515] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:22:30.442781 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:30.443651 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:30.444482 17763 raft_consensus.cc:3060] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:30.444736 17654 leader_election.cc:304] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
W20260709 12:22:30.449999 17761 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.711s	user 0.015s	sys 0.034s
W20260709 12:22:30.450428 17761 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.711s	user 0.015s	sys 0.034s
I20260709 12:22:30.422132 21617 leader_election.cc:290] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:22:30.439252 21867 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 1.317s	user 0.018s	sys 0.035s
I20260709 12:22:30.458372 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:30.436583 21589 raft_consensus.cc:2804] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:22:30.468688 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:30.504997 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:30.442565 21674 raft_consensus.cc:2804] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:22:30.423516 21965 raft_consensus.cc:515] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:30.508486 17680 raft_consensus.cc:2393] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 10: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:22:30.441160 21707 consensus_queue.cc:1059] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 9.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.067s
W20260709 12:22:30.463330 21867 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.440s	user 0.019s	sys 0.036s
W20260709 12:22:30.509922 17655 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:30.511121 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:30.518947 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:30.535137 17832 raft_consensus.cc:3060] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:30.514175 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:30.449823 21692 raft_consensus.cc:3060] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:30.449389 21888 raft_consensus.cc:493] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:30.506981 21589 raft_consensus.cc:493] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:30.459290 21857 raft_consensus.cc:2804] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:22:30.449532 21919 raft_consensus.cc:515] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:30.506316 17756 raft_consensus.cc:2468] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 9.
I20260709 12:22:30.536301 18371 raft_consensus.cc:2393] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 10: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:22:30.505110 21785 leader_election.cc:290] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:30.538085 18366 raft_consensus.cc:1240] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 10. Current term is 11. Ops: []
I20260709 12:22:30.539584 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:30.512806 21674 raft_consensus.cc:493] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:30.546356 21957 raft_consensus.cc:493] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:30.547940 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:30.558561 17652 leader_election.cc:304] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:22:30.578892 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 149646us
W20260709 12:22:30.582441 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:30.586509 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 126428us
I20260709 12:22:30.538946 21952 leader_election.cc:290] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:30.544790 21589 raft_consensus.cc:3060] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:30.592658 17763 raft_consensus.cc:2468] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 8.
I20260709 12:22:30.542912 21857 raft_consensus.cc:493] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:30.542713 21830 raft_consensus.cc:493] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:30.580744 21674 raft_consensus.cc:3060] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:22:30.599820 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:30.556504 21919 leader_election.cc:290] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:30.592936 21824 leader_election.cc:290] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:22:30.609295 18334 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.16.29.65:33071 timed out after 1.624s (SENT) [suppressed 35 similar messages]
I20260709 12:22:30.545431 21888 raft_consensus.cc:515] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:30.605278 21830 raft_consensus.cc:515] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:30.605396 21857 raft_consensus.cc:3060] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:30.629792 21768 leader_election.cc:290] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 10 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:30.630501 21768 raft_consensus.cc:2804] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:22:30.630865 21768 raft_consensus.cc:493] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:30.631188 21768 raft_consensus.cc:3060] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:30.636493 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:30.638025 17801 leader_election.cc:304] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:22:30.612777 21967 consensus_queue.cc:1048] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.020s
I20260709 12:22:30.591650 21957 raft_consensus.cc:515] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:30.641261 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
W20260709 12:22:30.641609 18365 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.621s	user 0.002s	sys 0.062s
W20260709 12:22:30.647035 18365 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.627s	user 0.003s	sys 0.062s
I20260709 12:22:30.647966 17763 raft_consensus.cc:3060] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:30.619772 21785 consensus_queue.cc:1059] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.027s
I20260709 12:22:30.650707 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 12 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:30.657490 17834 raft_consensus.cc:2468] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 11.
I20260709 12:22:30.654488 17832 raft_consensus.cc:2468] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 11.
I20260709 12:22:30.648622 21973 raft_consensus.cc:2804] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:22:30.665665 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:30.659335 18365 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59886 (request call id 1202) took 2206 ms. Trace:
I20260709 12:22:30.638077 21827 consensus_queue.cc:1059] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 9.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.021s
I20260709 12:22:30.668206 21785 raft_consensus.cc:3055] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:22:30.666999 18365 rpcz_store.cc:276] 0709 12:22:28.443698 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:29.229053 (+785355us) service_pool.cc:224] Handling call
0709 12:22:29.229450 (+   397us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:29.229690 (+   240us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 12:22:29.947392 (+717702us) log.cc:844] Serialized 14 byte log entry
0709 12:22:30.647809 (+700417us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:30.647905 (+    96us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:22:30.649480 (+  1575us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:30.649584 (+   104us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:30.649749 (+   165us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":1166183,"prepare.run_cpu_time_us":71,"prepare.run_wall_time_us":92,"raft.queue_time_us":107,"raft.run_cpu_time_us":180,"raft.run_wall_time_us":10870,"thread_start_us":1342302,"threads_started":2}
I20260709 12:22:30.646288 21952 consensus_queue.cc:1048] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.018s
I20260709 12:22:30.677680 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:30.678052 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:30.678519 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:30.678943 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:30.679864 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:30.680783 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:30.693562 17835 raft_consensus.cc:2468] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 8.
I20260709 12:22:30.681675 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:30.682243 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 13 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:30.665454 21973 raft_consensus.cc:493] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:30.700449 17833 raft_consensus.cc:2368] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for earlier term 9. Current term is 10.
I20260709 12:22:30.693101 17834 raft_consensus.cc:2393] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 12: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:22:30.700762 17838 raft_consensus.cc:2393] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 11: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:22:30.673799 21785 raft_consensus.cc:740] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 6 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:22:30.702720 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:30.705273 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:30.699584 17832 raft_consensus.cc:2410] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for term 13 because replica has last-logged OpId of term: 11 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 6.
I20260709 12:22:30.709481 21830 leader_election.cc:290] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:30.709808 17759 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38786 (request call id 2802) took 1388 ms. Trace:
I20260709 12:22:30.710301 17759 rpcz_store.cc:276] 0709 12:22:29.321101 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:29.350898 (+ 29797us) service_pool.cc:224] Handling call
0709 12:22:29.392359 (+ 41461us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:29.518328 (+125969us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:29.518357 (+    29us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:29.541808 (+ 23451us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:22:29.541871 (+    63us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:30.709495 (+1167624us) spinlock_profiling.cc:243] Waited 98.4 ms on lock 0x7b5400415fd4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:22:30.709546 (+    51us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:30.709709 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":226396416}
I20260709 12:22:30.713052 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:30.713761 17836 raft_consensus.cc:2468] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 9.
I20260709 12:22:30.714800 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:30.705919 17829 raft_consensus.cc:2376] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 11. Re-sending same reply.
I20260709 12:22:30.716140 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:30.712909 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:30.725430 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:30.727537 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:30.729174 17832 raft_consensus.cc:2468] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 9.
I20260709 12:22:30.735157 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:30.738914 17761 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38776 (request call id 2123) took 1274 ms. Trace:
I20260709 12:22:30.739308 17761 rpcz_store.cc:276] 0709 12:22:29.464445 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:29.737553 (+273108us) service_pool.cc:224] Handling call
0709 12:22:29.737940 (+   387us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:22:29.738310 (+   370us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:29.738341 (+    31us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:22:29.738948 (+   607us) log.cc:844] Serialized 52 byte log entry
0709 12:22:30.451143 (+712195us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:22:30.451203 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:30.451309 (+   106us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:30.738706 (+287397us) raft_consensus.cc:1685] finished
0709 12:22:30.738727 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:30.738860 (+   133us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1830,"raft.run_cpu_time_us":140,"raft.run_wall_time_us":5997,"thread_start_us":711691,"threads_started":1,"wal-append.queue_time_us":950956}
I20260709 12:22:30.738646 21952 consensus_queue.cc:1059] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 11.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
W20260709 12:22:30.742746 21969 log.cc:927] Time spent T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.139s	user 0.000s	sys 0.000s
I20260709 12:22:30.744453 21674 raft_consensus.cc:515] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:22:30.744984 21975 log.cc:927] Time spent T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.107s	user 0.000s	sys 0.000s
I20260709 12:22:30.742484 21952 consensus_queue.cc:1059] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 10.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 12:22:30.746838 21957 leader_election.cc:290] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:30.749104 17681 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44898 (request call id 854) took 2279 ms. Trace:
W20260709 12:22:30.756181 18363 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.519s	user 0.012s	sys 0.026s
I20260709 12:22:30.713393 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:30.767269 17838 raft_consensus.cc:2468] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 11.
I20260709 12:22:30.702435 21973 raft_consensus.cc:3060] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:30.704020 21888 leader_election.cc:290] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:30.712531 21830 raft_consensus.cc:493] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
W20260709 12:22:30.769512 18363 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.538s	user 0.012s	sys 0.027s
W20260709 12:22:30.772791 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:30.792775 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:30.774092 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 485688us
I20260709 12:22:30.749859 17681 rpcz_store.cc:276] 0709 12:22:28.469252 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:28.682745 (+213493us) service_pool.cc:224] Handling call
0709 12:22:28.683096 (+   351us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:28.783462 (+100366us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:28.783502 (+    40us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:28.831580 (+ 48078us) log.cc:844] Serialized 26 byte log entry
0709 12:22:30.220571 (+1388991us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:22:30.220635 (+    64us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:30.221784 (+  1149us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:30.748735 (+526951us) raft_consensus.cc:1685] finished
0709 12:22:30.748766 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:30.749024 (+   258us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":80734,"raft.queue_time_us":149799,"raft.run_cpu_time_us":136,"raft.run_wall_time_us":19418,"thread_start_us":1388470,"threads_started":1,"wal-append.queue_time_us":1699560}
I20260709 12:22:30.792680 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:30.785151 17763 raft_consensus.cc:2468] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 11.
I20260709 12:22:30.638310 17575 maintenance_manager.cc:643] P 2193af31cb894ec9b3d65d939a2a6851: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 15.989s	user 0.225s	sys 0.141s Metrics: {"bytes_written":30280,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":1671690,"dirs.run_cpu_time_us":532,"dirs.run_wall_time_us":52310,"drs_written":1,"lbm_read_time_us":20789,"lbm_reads_1-10_ms":1,"lbm_reads_10-100_ms":1,"lbm_reads_lt_1ms":2,"lbm_write_time_us":266041,"lbm_writes_1-10_ms":1,"lbm_writes_10-100_ms":9,"lbm_writes_lt_1ms":24,"peak_mem_usage":0,"reinsert_count":0,"rows_written":63,"spinlock_wait_cycles":141643136,"thread_start_us":1745762,"threads_started":1,"update_count":22}
I20260709 12:22:30.795192 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
W20260709 12:22:30.797995 17760 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38786 (request call id 2796) took 1535 ms (client timeout 1549 ms). Trace:
W20260709 12:22:30.798673 17760 rpcz_store.cc:269] 0709 12:22:29.262626 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:29.262742 (+   116us) service_pool.cc:224] Handling call
0709 12:22:30.795084 (+1532342us) spinlock_profiling.cc:243] Waited 5.43 ms on lock 0x7b5800033398. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b474fbb71 00007f7b474fbeee 000000000036eb39 00007f7b4f56f1e0 00007f7b4f6380c4 00007f7b4f638131 00007f7b4f5e60a0 00007f7b4f5e5984 00007f7b48b606d4 00007f7b48b60662
0709 12:22:30.797935 (+  2851us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":12507904}
I20260709 12:22:30.801434 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:30.802205 17764 raft_consensus.cc:2468] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 11.
I20260709 12:22:30.797744 18363 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59886 (request call id 1200) took 2354 ms. Trace:
I20260709 12:22:30.800387 21830 raft_consensus.cc:515] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:22:30.804208 17764 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38786 (request call id 2797) took 1541 ms (client timeout 2006 ms). Trace:
W20260709 12:22:30.807080 17764 rpcz_store.cc:269] 0709 12:22:29.263016 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:29.263129 (+   113us) service_pool.cc:224] Handling call
0709 12:22:30.795611 (+1532482us) spinlock_profiling.cc:243] Waited 6.6 ms on lock 0x7b5800033398. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b474fbb71 00007f7b474fba22 00007f7b504148d9 00007f7b4f56eee0 00007f7b4f6380ab 00007f7b4f638131 00007f7b4f5e60a0 00007f7b4f5e5984 00007f7b48b606d4 00007f7b48b60662
0709 12:22:30.804140 (+  8529us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":15181568}
I20260709 12:22:30.787599 21965 leader_election.cc:290] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 11 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:30.791795 21967 raft_consensus.cc:3055] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:22:30.811921 21967 raft_consensus.cc:740] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 11 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:22:30.805163 18363 rpcz_store.cc:276] 0709 12:22:28.443064 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:28.477919 (+ 34855us) service_pool.cc:224] Handling call
0709 12:22:29.161330 (+683411us) spinlock_profiling.cc:243] Waited 9.11 ms on lock 0x7b5800102398. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b474fbb71 00007f7b474fba22 00007f7b504148d9 00007f7b4f56eee0 00007f7b4f6380ab 00007f7b4f638131 00007f7b4f5f9fc0 00007f7b4f5e4929 00007f7b4f5e43d2 00007f7b48b5bbd4
0709 12:22:29.161562 (+   232us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:29.161849 (+   287us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 12:22:30.200327 (+1038478us) log.cc:844] Serialized 14 byte log entry
0709 12:22:30.795922 (+595595us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:30.795985 (+    63us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:22:30.796043 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:30.796109 (+    66us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:30.797685 (+  1576us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":1201409,"prepare.run_cpu_time_us":51,"prepare.run_wall_time_us":1571,"raft.queue_time_us":117,"raft.run_cpu_time_us":103,"raft.run_wall_time_us":3295,"spinlock_wait_cycles":20956416,"thread_start_us":1593479,"threads_started":2}
I20260709 12:22:30.828136 21810 raft_consensus.cc:515] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:22:30.828896 21976 log.cc:927] Time spent T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.067s	user 0.000s	sys 0.001s
I20260709 12:22:30.832038 17837 raft_consensus.cc:2393] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 11: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:22:30.803860 21785 consensus_queue.cc:260] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:30.819447 21967 consensus_queue.cc:260] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.9, Last appended by leader: 9, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:30.852859 18374 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 1354) took 2143 ms. Trace:
I20260709 12:22:30.853459 18374 rpcz_store.cc:276] 0709 12:22:28.709875 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:29.602075 (+892200us) service_pool.cc:224] Handling call
0709 12:22:29.602377 (+   302us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:29.602635 (+   258us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 12:22:30.424947 (+822312us) log.cc:844] Serialized 14 byte log entry
0709 12:22:30.852415 (+427468us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:30.852485 (+    70us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:22:30.852553 (+    68us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:30.852646 (+    93us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:30.852789 (+   143us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":801444,"prepare.run_cpu_time_us":51,"prepare.run_wall_time_us":1678,"raft.queue_time_us":118,"thread_start_us":1169086,"threads_started":2}
I20260709 12:22:30.863096 17576 maintenance_manager.cc:419] P 2193af31cb894ec9b3d65d939a2a6851: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 10291 bytes on disk
I20260709 12:22:30.849175 21674 leader_election.cc:290] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 15 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:30.872756 18361 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59886 (request call id 1207) took 2375 ms. Trace:
I20260709 12:22:30.876618 18361 rpcz_store.cc:276] 0709 12:22:28.497616 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:29.300245 (+802629us) service_pool.cc:224] Handling call
0709 12:22:30.850496 (+1550251us) spinlock_profiling.cc:243] Waited 7.29 ms on lock 0x7b5800102398. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b474fbb71 00007f7b474fba22 00007f7b504148d9 00007f7b4f56eee0 00007f7b4f6380ab 00007f7b4f638131 00007f7b4f5f9fc0 00007f7b4f5e4929 00007f7b4f5e43d2 00007f7b48b5bbd4
0709 12:22:30.850773 (+   277us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:30.872225 (+ 21452us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:30.872276 (+    51us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:30.872346 (+    70us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:22:30.872407 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:30.872497 (+    90us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:30.872678 (+   181us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":16772352}
I20260709 12:22:30.854271 21959 raft_consensus.cc:2749] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20260709 12:22:30.879073 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 285857us
I20260709 12:22:30.869252 21857 raft_consensus.cc:515] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:30.870342 21589 raft_consensus.cc:515] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:30.890250 21785 raft_consensus.cc:3060] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:30.928292 21830 leader_election.cc:290] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:30.916340 21982 raft_consensus.cc:3055] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:22:30.930336 21982 raft_consensus.cc:740] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 10 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:22:30.931000 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:30.932003 21982 consensus_queue.cc:260] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:30.933045 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 11 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:30.934405 21857 leader_election.cc:290] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 9 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:30.930389 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:30.935752 18374 raft_consensus.cc:2468] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 9.
I20260709 12:22:30.936803 21810 leader_election.cc:290] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 11 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:30.937618 21810 raft_consensus.cc:493] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:30.938266 21810 raft_consensus.cc:515] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:30.938875 21961 leader_election.cc:290] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:30.951285 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:30.959769 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:30.960601 18374 raft_consensus.cc:2468] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 11.
W20260709 12:22:30.961974 17729 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.591s	user 0.029s	sys 0.026s
W20260709 12:22:30.962950 17729 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.593s	user 0.029s	sys 0.026s
I20260709 12:22:30.935405 21768 raft_consensus.cc:515] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:22:30.966560 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 598856us
I20260709 12:22:30.970825 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:30.978914 21844 leader_election.cc:290] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 9 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:30.979198 18361 raft_consensus.cc:2393] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 7: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:22:30.979823 18365 raft_consensus.cc:1275] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260709 12:22:30.979234 17830 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35042 (request call id 2062) took 1420 ms. Trace:
I20260709 12:22:30.989147 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:30.988572 17830 rpcz_store.cc:276] 0709 12:22:29.546076 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:29.558896 (+ 12820us) service_pool.cc:224] Handling call
0709 12:22:29.600303 (+ 41407us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:29.805128 (+204825us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:29.805163 (+    35us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:29.805210 (+    47us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:22:29.805249 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:30.966028 (+1160779us) spinlock_profiling.cc:243] Waited 197 ms on lock 0x7b5400089dd4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:22:30.966084 (+    56us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:30.966217 (+   133us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":453824256}
W20260709 12:22:30.994656 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:31.015364 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 308116us
I20260709 12:22:31.017505 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
W20260709 12:22:31.019325 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 50117.3us
I20260709 12:22:31.021054 21589 leader_election.cc:290] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 8 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:31.021803 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:31.022326 21961 raft_consensus.cc:493] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:31.031476 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:31.025278 17575 maintenance_manager.cc:643] P 2193af31cb894ec9b3d65d939a2a6851: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.161s	user 0.004s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":726,"lbm_reads_lt_1ms":4}
I20260709 12:22:31.025586 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:31.024211 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 10 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:31.035276 17830 raft_consensus.cc:3060] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:22:31.024107 17803 connection.cc:256] RPC call timeout handler was delayed by -1.1392988170000535s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2138s for a response
I20260709 12:22:31.032330 21810 leader_election.cc:290] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:22:31.035956 17803 connection.cc:256] RPC call timeout handler was delayed by -1.0317251110000143s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1607s for a response
I20260709 12:22:31.031738 21961 raft_consensus.cc:515] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:22:31.039511 17652 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:31.036459 21984 consensus_queue.cc:1048] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.013s
I20260709 12:22:31.024191 21926 leader_election.cc:290] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:31.042516 17758 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38786 (request call id 2795) took 1780 ms. Trace:
W20260709 12:22:31.049284 17727 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:31.049669 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 188370us
I20260709 12:22:31.036374 21589 raft_consensus.cc:493] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:31.052106 17686 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37396 (request call id 2180) took 2677 ms. Trace:
I20260709 12:22:31.052819 21961 leader_election.cc:290] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:31.041450 17803 leader_election.cc:304] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:31.054821 18366 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 1336) took 2458 ms. Trace:
I20260709 12:22:31.057763 21973 raft_consensus.cc:515] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:31.059680 21991 consensus_queue.cc:1048] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:22:31.068446 17686 rpcz_store.cc:276] 0709 12:22:28.367871 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:30.129232 (+1761361us) service_pool.cc:224] Handling call
0709 12:22:30.212696 (+ 83464us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:30.264348 (+ 51652us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:30.264387 (+    39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:30.316069 (+ 51682us) log.cc:844] Serialized 26 byte log entry
0709 12:22:30.827988 (+511919us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:22:30.828030 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:30.828095 (+    65us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:31.045300 (+217205us) raft_consensus.cc:1685] finished
0709 12:22:31.045320 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:31.045471 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":23126,"raft.run_cpu_time_us":157,"raft.run_wall_time_us":35606,"thread_start_us":477124,"threads_started":1,"wal-append.queue_time_us":631916}
I20260709 12:22:31.070195 21996 raft_consensus.cc:493] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:31.028148 21692 raft_consensus.cc:515] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:31.074647 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:31.076732 21768 leader_election.cc:290] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 10 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:31.077631 18367 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 1350) took 2368 ms. Trace:
I20260709 12:22:31.080096 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:31.085093 17834 raft_consensus.cc:3060] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:31.084748 17836 raft_consensus.cc:2393] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 12: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:22:31.068403 17830 raft_consensus.cc:2468] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 10.
I20260709 12:22:31.079180 18366 rpcz_store.cc:276] 0709 12:22:28.596079 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:29.313976 (+717897us) service_pool.cc:224] Handling call
0709 12:22:29.333118 (+ 19142us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:30.547949 (+1214831us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:31.054546 (+506597us) spinlock_profiling.cc:243] Waited 1.2 s on lock 0x7b54007602d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:22:31.054748 (+   202us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2770664704}
I20260709 12:22:31.089215 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:31.090068 18366 raft_consensus.cc:2393] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 9: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:22:31.043644 21800 raft_consensus.cc:3055] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:22:31.082105 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:22:31.082625 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 44280.2us
I20260709 12:22:31.094183 17803 leader_election.cc:304] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:31.084687 18367 rpcz_store.cc:276] 0709 12:22:28.708636 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:29.435257 (+726621us) service_pool.cc:224] Handling call
0709 12:22:29.435635 (+   378us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:29.827367 (+391732us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:31.067856 (+1240489us) spinlock_profiling.cc:243] Waited 366 ms on lock 0x7b540047de54. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:22:31.077388 (+  9532us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":842495872}
I20260709 12:22:31.092325 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:31.093176 18366 raft_consensus.cc:1240] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 8. Current term is 9. Ops: [8.7-8.7]
I20260709 12:22:31.095376 18362 raft_consensus.cc:2468] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 9.
I20260709 12:22:31.080622 21589 raft_consensus.cc:515] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:31.110896 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:22:31.113368 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 309560us
I20260709 12:22:31.116962 17803 leader_election.cc:304] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:31.083395 21982 raft_consensus.cc:3060] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:31.083407 21858 raft_consensus.cc:493] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:31.045993 17758 rpcz_store.cc:276] 0709 12:22:29.262320 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:29.262482 (+   162us) service_pool.cc:224] Handling call
0709 12:22:30.395899 (+1133417us) spinlock_profiling.cc:243] Waited 6.53 ms on lock 0x7b5800033398. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b474fbb71 00007f7b474fba22 00007f7b504148d9 00007f7b4f56eee0 00007f7b4f6380ab 00007f7b4f638131 00007f7b4f5f9fc0 00007f7b4f5e4929 00007f7b4f5e43d2 00007f7b48b5bbd4
0709 12:22:30.396086 (+   187us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:30.401702 (+  5616us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:22:30.839063 (+437361us) log.cc:844] Serialized 14 byte log entry
0709 12:22:31.041830 (+202767us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:31.041885 (+    55us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:22:31.042175 (+   290us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:31.042321 (+   146us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:31.042457 (+   136us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2515,"prepare.queue_time_us":576925,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":46,"raft.queue_time_us":6221,"spinlock_wait_cycles":15017344,"thread_start_us":606099,"threads_started":2}
I20260709 12:22:31.126273 17837 raft_consensus.cc:1275] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260709 12:22:31.128294 17836 raft_consensus.cc:1275] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 8 index: 10. (index mismatch)
I20260709 12:22:31.093966 21800 raft_consensus.cc:740] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:22:31.129415 17802 leader_election.cc:304] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:22:31.133427 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:31.108517 21844 consensus_queue.cc:1059] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 8.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.036s
I20260709 12:22:31.092406 21996 raft_consensus.cc:515] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:31.128340 21858 raft_consensus.cc:515] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:31.142277 21843 tablet_copy_client.cc:538] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:22:31.144058 21858 leader_election.cc:290] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:31.144651 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:31.148998 21974 raft_consensus.cc:493] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:31.149812 21589 leader_election.cc:290] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:31.149591 21974 raft_consensus.cc:515] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:22:31.150431 17804 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:31.151866 17834 raft_consensus.cc:2468] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 10.
I20260709 12:22:31.152705 21995 raft_consensus.cc:2749] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:22:31.153398 17764 raft_consensus.cc:1275] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 8 index: 7. Preceding OpId from leader: term: 10 index: 8. (index mismatch)
I20260709 12:22:31.163336 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:31.180948 17837 raft_consensus.cc:2393] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 8: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:22:31.181568 17762 raft_consensus.cc:1275] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260709 12:22:31.184216 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:31.184367 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:31.185523 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 12 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:31.185608 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:31.186374 17757 raft_consensus.cc:2410] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for term 12 because replica has last-logged OpId of term: 9 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 8.
I20260709 12:22:31.186414 17760 raft_consensus.cc:2393] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 11: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:22:31.187484 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:31.200937 17680 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37402 (request call id 2576) took 1888 ms. Trace:
I20260709 12:22:31.201368 17680 rpcz_store.cc:276] 0709 12:22:29.312209 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:29.409486 (+ 97277us) service_pool.cc:224] Handling call
0709 12:22:31.200743 (+1791257us) spinlock_profiling.cc:243] Waited 1.04 s on lock 0x7b5400207954. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:22:31.200879 (+   136us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2387384704}
I20260709 12:22:31.201896 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:31.203183 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:31.123330 21768 consensus_queue.cc:1048] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.035s
I20260709 12:22:31.204711 21965 consensus_queue.cc:1048] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 12:22:31.207140 17729 leader_election.cc:304] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:31.207336 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:31.208791 21973 leader_election.cc:290] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 11 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:31.208724 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:31.187795 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:31.209949 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:31.210322 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 11 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:31.183259 21974 leader_election.cc:290] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:31.211040 17830 raft_consensus.cc:2393] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 11: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:22:31.211534 21985 raft_consensus.cc:2804] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:22:31.211957 21985 raft_consensus.cc:493] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:31.211915 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:31.212473 21985 raft_consensus.cc:3060] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:31.212749 17830 raft_consensus.cc:3060] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:31.189489 21692 leader_election.cc:290] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 7 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:31.191008 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:22:31.213485 21948 log.cc:927] Time spent T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 1.112s	user 0.001s	sys 0.004s
I20260709 12:22:31.213609 21996 leader_election.cc:290] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:31.215086 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:31.192243 21997 raft_consensus.cc:2749] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:22:31.192879 21995 raft_consensus.cc:2749] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:22:31.193068 21858 raft_consensus.cc:2804] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:22:31.193293 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 11 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:31.195070 17835 raft_consensus.cc:1275] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 8 index: 7. Preceding OpId from leader: term: 10 index: 8. (index mismatch)
I20260709 12:22:31.197767 21800 consensus_queue.cc:260] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.7, Last appended by leader: 7, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:31.191685 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 10 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:31.224912 17728 leader_election.cc:304] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:31.233625 17838 raft_consensus.cc:2410] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for term 10 because replica has last-logged OpId of term: 9 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 7.
I20260709 12:22:31.211025 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:31.221084 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:31.234306 18367 raft_consensus.cc:2468] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 8.
I20260709 12:22:31.224598 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:31.232686 18334 leader_election.cc:304] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:31.235816 17837 raft_consensus.cc:3060] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:31.236747 17833 raft_consensus.cc:2468] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 7.
I20260709 12:22:31.237933 17836 raft_consensus.cc:2393] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 10: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
W20260709 12:22:31.233430 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 98978.3us
I20260709 12:22:31.238085 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:31.240362 21961 consensus_queue.cc:1048] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 6, Time since last communication: 0.004s
I20260709 12:22:31.240641 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:31.241961 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:31.232240 21858 raft_consensus.cc:493] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:31.243382 21858 raft_consensus.cc:3060] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:31.248353 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:31.249147 18367 raft_consensus.cc:2468] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 9.
I20260709 12:22:31.255604 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:31.257561 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:31.258858 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:31.259685 18368 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 10.
W20260709 12:22:31.260398 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 143524us
I20260709 12:22:31.261837 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:31.262531 17803 leader_election.cc:304] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:31.263290 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:31.266139 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:31.266095 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:31.267019 17689 raft_consensus.cc:3060] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:31.267652 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:31.268038 17834 raft_consensus.cc:2393] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 10: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
W20260709 12:22:31.268648 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 43808.3us
I20260709 12:22:31.266063 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:31.270363 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:31.271669 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:31.272143 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 9 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:31.272601 17680 raft_consensus.cc:2393] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 11: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:22:31.272773 17838 raft_consensus.cc:3060] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:31.267071 18371 raft_consensus.cc:2468] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 7.
I20260709 12:22:31.275962 17688 raft_consensus.cc:2393] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 10: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:22:31.278064 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:31.280536 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:31.281379 17685 raft_consensus.cc:3060] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:31.278862 17683 raft_consensus.cc:2468] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 10.
I20260709 12:22:31.285567 17681 raft_consensus.cc:2393] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 15: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:22:31.255131 21965 raft_consensus.cc:2804] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:22:31.285655 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:31.224653 21692 consensus_queue.cc:1059] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.045s
I20260709 12:22:31.224875 21994 consensus_queue.cc:1059] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.056s
I20260709 12:22:31.291518 17687 raft_consensus.cc:3060] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:22:31.294083 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 170057us
I20260709 12:22:31.295195 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:31.300729 18367 raft_consensus.cc:3060] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:22:31.301548 21843 tablet_bootstrap.cc:492] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da: Bootstrap starting.
I20260709 12:22:31.301923 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:31.302524 18363 raft_consensus.cc:3060] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:31.304430 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:31.305323 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 10 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:31.306027 18371 raft_consensus.cc:2468] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 9.
I20260709 12:22:31.306313 21991 consensus_queue.cc:1048] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 12:22:31.307003 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:31.307543 18371 raft_consensus.cc:3060] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:31.294334 21965 raft_consensus.cc:493] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:31.308274 21965 raft_consensus.cc:3060] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:31.309938 18374 raft_consensus.cc:1275] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20260709 12:22:31.310930 18363 raft_consensus.cc:2468] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 8.
I20260709 12:22:31.311029 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:31.311764 18374 raft_consensus.cc:2468] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 11.
I20260709 12:22:31.312119 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:31.269682 21974 raft_consensus.cc:2764] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Leader pre-election decision vote started in defunct term 10: won
I20260709 12:22:31.292917 17803 leader_election.cc:304] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:22:31.315825 17838 raft_consensus.cc:2468] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 9.
I20260709 12:22:31.316711 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:31.317425 17681 raft_consensus.cc:2393] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 9: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:22:31.315431 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:31.285921 21785 raft_consensus.cc:2749] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:22:31.258373 21961 consensus_queue.cc:1048] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.037s
I20260709 12:22:31.297508 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:31.296494 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 12 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:31.323784 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:31.324597 18368 raft_consensus.cc:3060] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:31.291986 21800 raft_consensus.cc:3060] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:31.320952 17652 leader_election.cc:304] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:31.338919 17683 raft_consensus.cc:2368] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for earlier term 9. Current term is 10.
I20260709 12:22:31.340337 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:31.341848 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:31.347270 17652 leader_election.cc:304] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:31.350047 17803 leader_election.cc:304] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:22:31.339697 17830 raft_consensus.cc:2468] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 9.
I20260709 12:22:31.352672 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:31.353511 17838 raft_consensus.cc:2376] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 10. Re-sending same reply.
I20260709 12:22:31.357426 17681 raft_consensus.cc:1240] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 14. Current term is 15. Ops: []
I20260709 12:22:31.360549 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:22:31.369179 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 110440us
I20260709 12:22:31.333595 22001 raft_consensus.cc:2804] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:22:31.367026 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:31.374601 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:31.318486 21830 consensus_queue.cc:1048] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.041s
I20260709 12:22:31.379159 18367 raft_consensus.cc:2468] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 15.
I20260709 12:22:31.374173 17803 leader_election.cc:304] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:22:31.380560 17652 leader_election.cc:304] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:31.383471 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:31.340466 21768 consensus_queue.cc:1048] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.032s
I20260709 12:22:31.384459 17838 raft_consensus.cc:2468] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 11.
I20260709 12:22:31.387854 17802 leader_election.cc:304] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
W20260709 12:22:31.390151 17804 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:31.391621 17652 leader_election.cc:304] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:31.348379 21674 consensus_queue.cc:1048] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.039s
I20260709 12:22:31.396842 17728 leader_election.cc:304] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:31.402527 17728 leader_election.cc:304] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:31.403597 17728 leader_election.cc:304] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:31.373940 21785 raft_consensus.cc:2804] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:22:31.406059 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:31.373947 22001 raft_consensus.cc:493] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:31.414716 18368 raft_consensus.cc:2468] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 9.
I20260709 12:22:31.388226 21994 raft_consensus.cc:2804] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Leader pre-election won for term 12
W20260709 12:22:31.424059 17727 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:31.388079 21995 raft_consensus.cc:2804] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:22:31.427821 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:31.422529 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 12 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:31.423123 17652 leader_election.cc:304] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:31.420316 17654 leader_election.cc:304] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:31.423561 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:31.390669 21844 raft_consensus.cc:2804] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:22:31.430487 21844 raft_consensus.cc:493] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:22:31.431080 21844 raft_consensus.cc:3060] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:31.439692 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:31.440869 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:31.419404 21848 raft_consensus.cc:2749] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:22:31.409832 21785 raft_consensus.cc:493] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:31.419170 21996 raft_consensus.cc:2749] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260709 12:22:31.450589 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 47037.1us
I20260709 12:22:31.448851 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:31.450984 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 9 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:31.451762 18363 raft_consensus.cc:2393] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 9: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:22:31.421149 22001 raft_consensus.cc:3060] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:31.450243 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:31.428254 21995 raft_consensus.cc:493] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:31.451715 17728 leader_election.cc:400] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 9 election: Vote denied by peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071) with higher term. Message: Invalid argument: T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for earlier term 9. Current term is 10.
I20260709 12:22:31.474924 17728 leader_election.cc:403] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 9 election: Cancelling election due to peer responding with higher term
I20260709 12:22:31.475862 18368 raft_consensus.cc:1240] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 7. Current term is 8. Ops: []
I20260709 12:22:31.447572 21786 raft_consensus.cc:2804] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:22:31.479003 18361 raft_consensus.cc:3060] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:31.480376 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:31.396811 21997 consensus_queue.cc:1048] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.060s
I20260709 12:22:31.449816 22003 raft_consensus.cc:2749] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:22:31.486219 17687 raft_consensus.cc:2468] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 9.
I20260709 12:22:31.453187 21785 raft_consensus.cc:3060] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:31.471493 21768 raft_consensus.cc:3060] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:22:31.487388 17755 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38786 (request call id 2806) took 2062 ms (client timeout 2170 ms). Trace:
I20260709 12:22:31.452913 22009 raft_consensus.cc:2749] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
W20260709 12:22:31.499580 22007 log.cc:927] Time spent T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.054s	user 0.000s	sys 0.001s
I20260709 12:22:31.492849 21857 raft_consensus.cc:2749] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:22:31.500371 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:22:31.504196 17755 rpcz_store.cc:269] 0709 12:22:29.424805 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:29.438952 (+ 14147us) service_pool.cc:224] Handling call
0709 12:22:31.487198 (+2048246us) spinlock_profiling.cc:243] Waited 809 ms on lock 0x7b54002b5f54. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:22:31.487325 (+   127us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1861332352}
I20260709 12:22:31.509647 17688 raft_consensus.cc:2468] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 11.
I20260709 12:22:31.513734 17802 leader_election.cc:304] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:31.520349 17802 leader_election.cc:304] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:31.476999 21995 raft_consensus.cc:3060] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:31.478716 21786 raft_consensus.cc:493] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:31.526839 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:31.540908 17830 raft_consensus.cc:2393] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 8: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:22:31.540993 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:31.541914 17832 raft_consensus.cc:2468] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 9.
I20260709 12:22:31.542533 17685 raft_consensus.cc:2468] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 12.
I20260709 12:22:31.526270 22002 raft_consensus.cc:3060] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:31.531975 21973 consensus_queue.cc:1048] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.021s
I20260709 12:22:31.551642 17831 raft_consensus.cc:2468] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 11.
I20260709 12:22:31.562714 17682 raft_consensus.cc:2468] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 10.
I20260709 12:22:31.554970 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:31.564601 18366 raft_consensus.cc:2468] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 11.
W20260709 12:22:31.574556 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 56362.7us
I20260709 12:22:31.554435 21705 raft_consensus.cc:2749] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:22:31.578724 21858 raft_consensus.cc:515] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:31.551489 21844 raft_consensus.cc:515] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:31.582692 21844 leader_election.cc:290] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 9 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:31.539867 21786 raft_consensus.cc:3060] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:31.554275 21973 raft_consensus.cc:2749] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:22:31.591912 17837 raft_consensus.cc:2468] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 11.
I20260709 12:22:31.571772 22009 raft_consensus.cc:3060] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:31.597154 21858 leader_election.cc:290] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 12 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:31.531853 21965 raft_consensus.cc:515] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:31.602808 21967 raft_consensus.cc:3060] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:31.604884 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:31.606492 22002 raft_consensus.cc:2749] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260709 12:22:31.637594 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 214529us
I20260709 12:22:31.638885 17652 leader_election.cc:400] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 9 pre-election: Vote denied by peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837) with higher term. Message: Invalid argument: T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for earlier term 9. Current term is 10.
I20260709 12:22:31.610459 18371 raft_consensus.cc:2468] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 10.
I20260709 12:22:31.611932 17689 raft_consensus.cc:2468] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 7.
W20260709 12:22:31.615864 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 195662us
I20260709 12:22:31.642596 17654 leader_election.cc:304] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:31.624612 18333 leader_election.cc:304] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:22:31.650770 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 80886.1us
W20260709 12:22:31.650566 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 160776us
I20260709 12:22:31.662199 22016 consensus_queue.cc:1048] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.048s
I20260709 12:22:31.663857 22017 consensus_queue.cc:1059] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 14.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.070s
I20260709 12:22:31.665184 22017 raft_consensus.cc:3055] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:22:31.665602 22017 raft_consensus.cc:740] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:22:31.678848 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:31.614544 21867 consensus_queue.cc:1048] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 7, Time since last communication: 0.046s
W20260709 12:22:31.681445 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 71755.7us
I20260709 12:22:31.679868 17764 raft_consensus.cc:2468] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 10.
I20260709 12:22:31.683612 18333 leader_election.cc:304] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
W20260709 12:22:31.684710 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 164564us
I20260709 12:22:31.685566 17802 leader_election.cc:304] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:31.685817 17652 leader_election.cc:304] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:31.689447 17652 leader_election.cc:304] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:31.694473 17802 leader_election.cc:304] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:31.669852 21961 raft_consensus.cc:2749] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:22:31.667518 22022 raft_consensus.cc:2804] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260709 12:22:31.722555 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 80071.9us
I20260709 12:22:31.723052 17652 leader_election.cc:304] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:31.690737 22003 raft_consensus.cc:2749] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
W20260709 12:22:31.716771 22010 log.cc:927] Time spent T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.071s	user 0.000s	sys 0.000s
I20260709 12:22:31.725806 17654 leader_election.cc:304] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:31.717875 21957 raft_consensus.cc:2749] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:22:31.734654 18333 leader_election.cc:304] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:31.722401 22022 raft_consensus.cc:493] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:31.716193 21858 raft_consensus.cc:2749] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:22:31.735790 17802 leader_election.cc:304] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
W20260709 12:22:31.754597 21985 consensus_meta.cc:287] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.518s	user 0.011s	sys 0.000s
I20260709 12:22:31.755185 21985 raft_consensus.cc:515] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:31.757387 17654 leader_election.cc:304] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:31.759004 17654 leader_election.cc:304] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:22:31.759778 21705 raft_consensus.cc:2804] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:22:31.760186 21705 raft_consensus.cc:493] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:31.760521 21705 raft_consensus.cc:3060] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:31.760701 22003 raft_consensus.cc:2804] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:22:31.761152 22003 raft_consensus.cc:493] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:31.762017 22003 raft_consensus.cc:3060] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:31.744880 18361 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 11.
I20260709 12:22:31.725283 21827 raft_consensus.cc:2749] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 12:22:31.764043 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:31.760537 17654 leader_election.cc:304] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:22:31.750147 21844 raft_consensus.cc:2749] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:22:31.770782 17802 leader_election.cc:304] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:31.746201 17652 leader_election.cc:304] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:31.757992 21985 leader_election.cc:290] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 11 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:31.772522 17654 leader_election.cc:304] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:22:31.743647 21768 raft_consensus.cc:2749] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:22:31.775525 21810 raft_consensus.cc:2749] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:22:31.777745 17802 leader_election.cc:304] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:31.778028 22006 raft_consensus.cc:2804] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:22:31.772919 21827 raft_consensus.cc:2804] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:22:31.749361 22002 raft_consensus.cc:2764] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Leader pre-election decision vote started in defunct term 9: won
I20260709 12:22:31.782410 21830 raft_consensus.cc:2804] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:22:31.796211 21830 raft_consensus.cc:493] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:22:31.796670 21830 raft_consensus.cc:3060] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:31.800149 21973 raft_consensus.cc:2749] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20260709 12:22:31.801396 22020 log.cc:927] Time spent T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.077s	user 0.001s	sys 0.000s
I20260709 12:22:31.780984 21996 raft_consensus.cc:2749] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:22:31.751032 22021 raft_consensus.cc:493] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:31.804162 22021 raft_consensus.cc:515] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:31.804420 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:31.806449 22021 leader_election.cc:290] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:31.782558 17654 leader_election.cc:304] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:22:31.790290 22023 raft_consensus.cc:2804] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:22:31.790347 21997 consensus_queue.cc:1048] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:22:31.791317 21858 consensus_queue.cc:1059] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 12:22:31.812876 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:31.791518 17802 leader_election.cc:304] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:31.775808 21707 consensus_queue.cc:1048] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.020s
I20260709 12:22:31.813448 18371 raft_consensus.cc:3060] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Advancing to term 11
W20260709 12:22:31.796245 22019 log.cc:927] Time spent T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.205s	user 0.000s	sys 0.001s
I20260709 12:22:31.688586 22017 consensus_queue.cc:260] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 14.13, Last appended by leader: 13, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:31.809006 17654 leader_election.cc:304] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:22:31.807677 17729 leader_election.cc:304] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:22:31.767172 22022 raft_consensus.cc:3060] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:31.773723 21957 raft_consensus.cc:2749] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:22:31.827951 22002 raft_consensus.cc:3055] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:22:31.831911 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:31.832450 17756 raft_consensus.cc:1240] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 7. Current term is 8. Ops: []
I20260709 12:22:31.832422 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:31.787124 21827 raft_consensus.cc:493] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:31.810544 22023 raft_consensus.cc:493] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:22:31.833774 21965 leader_election.cc:290] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 9 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:31.833829 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:31.812335 21844 raft_consensus.cc:2804] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Leader pre-election won for term 12
W20260709 12:22:31.837643 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 46209.6us
I20260709 12:22:31.891081 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:31.838305 21997 raft_consensus.cc:2749] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:22:31.825963 21926 consensus_queue.cc:1059] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.028s
I20260709 12:22:31.827992 18367 raft_consensus.cc:1240] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 8. Current term is 9. Ops: []
I20260709 12:22:31.843572 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:31.813544 17652 leader_election.cc:304] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:22:31.898506 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 85049.1us
I20260709 12:22:31.858471 18371 raft_consensus.cc:2468] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 11.
W20260709 12:22:31.860070 21980 log.cc:927] Time spent T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 1.092s	user 0.006s	sys 0.000s
W20260709 12:22:31.865846 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 131319us
I20260709 12:22:31.866531 21785 raft_consensus.cc:515] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:22:31.869395 21800 consensus_meta.cc:287] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.531s	user 0.010s	sys 0.000s
I20260709 12:22:31.874251 21995 raft_consensus.cc:515] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:31.876760 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:31.907318 17759 raft_consensus.cc:2393] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 7: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:22:31.807548 21786 raft_consensus.cc:515] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:31.907738 17681 raft_consensus.cc:2468] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 10.
I20260709 12:22:31.839340 22001 raft_consensus.cc:515] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:31.831000 21705 raft_consensus.cc:515] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:22:31.887212 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 78269.7us
W20260709 12:22:31.889070 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 81464.2us
I20260709 12:22:31.899852 17652 leader_election.cc:304] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:22:31.952895 17654 leader_election.cc:304] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
W20260709 12:22:31.953099 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 53319.1us
I20260709 12:22:31.892493 21919 raft_consensus.cc:2804] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:22:31.890589 21546 raft_consensus.cc:2804] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:22:31.994445 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:31.994829 21546 raft_consensus.cc:697] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 15 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:22:31.879045 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:31.996145 17760 raft_consensus.cc:2368] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 10. Current term is 11.
I20260709 12:22:31.995864 21546 consensus_queue.cc:237] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 14.13, Last appended by leader: 13, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:31.996443 21888 raft_consensus.cc:3055] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:22:31.996842 21888 raft_consensus.cc:740] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:22:31.997054 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:31.997704 21888 consensus_queue.cc:260] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.7, Last appended by leader: 7, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:31.891336 22002 raft_consensus.cc:740] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 10 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:22:31.908439 17801 leader_election.cc:304] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:31.998354 21995 leader_election.cc:290] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 12 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:31.892433 21827 raft_consensus.cc:3060] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:31.906355 18333 leader_election.cc:304] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:22:31.999075 21995 raft_consensus.cc:2749] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:22:31.999138 21888 raft_consensus.cc:3060] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:31.933055 22001 leader_election.cc:290] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 8 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:31.898012 21844 raft_consensus.cc:493] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:31.895066 22023 raft_consensus.cc:3060] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:31.903178 21768 raft_consensus.cc:2749] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:22:31.998265 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:31.884649 21830 raft_consensus.cc:515] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:31.883105 22003 raft_consensus.cc:515] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:31.955706 18366 raft_consensus.cc:1240] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 8. Current term is 9. Ops: [8.7-8.7]
I20260709 12:22:31.956152 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:31.949662 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 12 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:31.951068 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:31.988078 21800 raft_consensus.cc:2804] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:22:31.988943 21990 raft_consensus.cc:2804] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:22:31.989143 17652 leader_election.cc:304] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:31.989490 21786 leader_election.cc:290] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 12 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:32.009215 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:31.960804 22009 raft_consensus.cc:2749] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Leader election lost for term 9. Reason: Vote denied by peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071) with higher term. Message: Invalid argument: T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for earlier term 9. Current term is 10.
I20260709 12:22:31.993135 17654 leader_election.cc:304] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:22:31.993577 21692 consensus_queue.cc:1048] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:22:32.012960 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:31.994117 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:31.994616 21919 raft_consensus.cc:493] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:31.904294 22006 raft_consensus.cc:697] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 8 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:22:31.951037 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 11 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:22:31.999174 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 90887.2us
I20260709 12:22:32.017686 17801 leader_election.cc:304] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:32.021396 21997 consensus_queue.cc:1059] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:22:32.022436 21996 raft_consensus.cc:2749] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:22:32.022614 21997 raft_consensus.cc:3055] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:22:32.023186 21997 raft_consensus.cc:740] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 7 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:22:31.991143 21974 raft_consensus.cc:493] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:32.009956 17834 raft_consensus.cc:2468] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 11.
I20260709 12:22:32.010185 17652 leader_election.cc:304] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:22:32.024731 21974 raft_consensus.cc:515] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:32.027393 21974 leader_election.cc:290] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:32.000103 21844 raft_consensus.cc:3060] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:32.036885 22001 raft_consensus.cc:493] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:32.037788 22001 raft_consensus.cc:515] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:32.039952 21997 consensus_queue.cc:260] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:32.042400 21997 raft_consensus.cc:3060] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:32.042428 22001 leader_election.cc:290] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:32.000260 21984 raft_consensus.cc:3055] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:22:32.080202 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:32.082484 17831 raft_consensus.cc:2468] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 12.
I20260709 12:22:32.083107 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:32.084206 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 12 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:32.085016 17832 raft_consensus.cc:2468] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 11.
W20260709 12:22:32.085970 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 92912.4us
I20260709 12:22:32.087268 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:32.089221 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 13 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:32.091145 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:32.012554 21707 raft_consensus.cc:2764] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Leader pre-election decision vote started in defunct term 6: won
I20260709 12:22:32.073439 17682 raft_consensus.cc:2368] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 7. Current term is 8.
W20260709 12:22:32.073927 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 59257.3us
I20260709 12:22:32.071200 17829 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d"
dest_uuid: "e063541542a94969898868bf6eb5ce1b"
mode: GRACEFUL
new_leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
 from {username='slave'} at 127.0.0.1:59820
I20260709 12:22:32.075925 21843 tablet_bootstrap.cc:492] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=4 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20260709 12:22:32.077721 22028 consensus_queue.cc:1059] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 11.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
W20260709 12:22:32.080139 22035 log.cc:927] Time spent T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.068s	user 0.000s	sys 0.000s
I20260709 12:22:32.071942 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:32.014598 21957 raft_consensus.cc:2749] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:22:32.129335 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:32.130456 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:32.131039 17762 raft_consensus.cc:2468] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 10.
I20260709 12:22:32.132179 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:32.132732 17830 raft_consensus.cc:2376] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 10. Re-sending same reply.
I20260709 12:22:32.133756 21705 leader_election.cc:290] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 11 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:32.134367 21830 leader_election.cc:290] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 12 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:31.999168 22002 consensus_queue.cc:260] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.6, Last appended by leader: 6, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:32.135246 22028 raft_consensus.cc:3055] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:22:32.135352 21848 raft_consensus.cc:3055] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:22:32.135722 22028 raft_consensus.cc:740] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:22:32.131851 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:32.137331 17762 raft_consensus.cc:3060] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:32.118312 17727 leader_election.cc:304] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:32.141513 17837 raft_consensus.cc:2468] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 11.
W20260709 12:22:32.095130 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 193804us
I20260709 12:22:32.112468 17829 raft_consensus.cc:606] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Received request to transfer leadership to TS e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:32.115139 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:32.146014 17829 raft_consensus.cc:614] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:22:32.146522 17685 raft_consensus.cc:2393] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 11: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:22:32.113649 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:32.116783 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:32.116986 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:32.087605 21919 raft_consensus.cc:3060] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:32.081318 21984 raft_consensus.cc:740] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 8 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:22:32.047793 21824 consensus_queue.cc:1048] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.088s
I20260709 12:22:32.097198 21546 raft_consensus.cc:2804] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:22:32.150550 17760 raft_consensus.cc:2393] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 12: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:22:32.151065 17835 raft_consensus.cc:2393] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 12: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:22:32.151552 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:32.136047 22002 raft_consensus.cc:3060] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:32.087064 21990 raft_consensus.cc:697] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 9 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:22:32.135847 21848 raft_consensus.cc:740] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:22:32.114677 21768 raft_consensus.cc:2804] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:22:32.147666 21985 raft_consensus.cc:2804] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:22:32.148912 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:32.166915 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:32.154853 18333 leader_election.cc:304] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:22:32.039105 21844 raft_consensus.cc:515] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:32.161991 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:32.163491 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:22:32.145051 22040 log.cc:927] Time spent T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.064s	user 0.000s	sys 0.001s
I20260709 12:22:32.169569 17685 raft_consensus.cc:2410] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 13 because replica has last-logged OpId of term: 12 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 7.
W20260709 12:22:32.176136 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 58115.7us
I20260709 12:22:32.177156 17837 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 8.
I20260709 12:22:32.174656 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:32.193020 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:32.193899 17832 raft_consensus.cc:2468] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 10.
I20260709 12:22:32.195469 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:32.172655 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:32.200618 17760 raft_consensus.cc:2393] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 12: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:22:32.201762 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:32.202555 17760 raft_consensus.cc:3060] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:22:32.172688 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 85564.8us
I20260709 12:22:32.178048 21927 raft_consensus.cc:2243] T 7cb3768d215e409fb9f5baf18d124442 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Raft consensus shutting down.
I20260709 12:22:32.178421 21990 consensus_queue.cc:237] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:32.209766 18334 leader_election.cc:304] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:32.184134 17833 raft_consensus.cc:2468] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 7.
I20260709 12:22:32.184397 21843 tablet_bootstrap.cc:492] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da: Bootstrap complete.
I20260709 12:22:32.186486 21844 leader_election.cc:290] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 12 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:32.188175 21768 raft_consensus.cc:697] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 8 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:22:32.187624 21848 consensus_queue.cc:260] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:32.189851 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:32.190868 17683 raft_consensus.cc:1240] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 8. Current term is 9. Ops: [7.5-8.6]
I20260709 12:22:32.206959 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:32.177767 21800 raft_consensus.cc:697] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 10 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:22:32.211812 17728 leader_election.cc:304] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:32.213873 18334 leader_election.cc:304] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:32.267277 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 9 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:32.169687 21546 raft_consensus.cc:697] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 10 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:22:32.224125 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:32.224251 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:32.227102 17687 raft_consensus.cc:3060] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:32.231016 22003 leader_election.cc:290] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 8 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:32.285931 17688 raft_consensus.cc:1240] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 9 LEADER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 8. Current term is 9. Ops: [8.9-8.9]
I20260709 12:22:32.232380 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:32.235523 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 13 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:32.236752 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 9 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:32.282658 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:32.288393 17759 raft_consensus.cc:2410] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 13 because replica has last-logged OpId of term: 11 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 7.
I20260709 12:22:32.237318 17729 leader_election.cc:304] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:22:32.237870 17727 leader_election.cc:336] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837): Timed out: RequestConsensusVote RPC to 127.16.29.67:36837 timed out after 1.535s (SENT)
W20260709 12:22:32.289086 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 51866us
W20260709 12:22:32.289364 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 51553.7us
W20260709 12:22:32.290055 17727 leader_election.cc:336] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837): Timed out: RequestConsensusVote RPC to 127.16.29.67:36837 timed out after 1.658s (SENT)
I20260709 12:22:32.290809 21926 raft_consensus.cc:2804] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:22:32.292153 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:32.292289 21848 raft_consensus.cc:3060] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:32.292990 17686 raft_consensus.cc:2410] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for term 12 because replica has last-logged OpId of term: 11 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 8.
W20260709 12:22:32.293431 17803 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:32.293263 22025 consensus_queue.cc:1059] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 12:22:32.293889 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:32.294514 17686 raft_consensus.cc:2468] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 6.
I20260709 12:22:32.295396 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:32.167070 22022 raft_consensus.cc:515] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:32.295836 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:32.254187 22021 consensus_queue.cc:1059] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 9.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.070s
I20260709 12:22:32.297076 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:32.297835 22022 leader_election.cc:290] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 9 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:32.298023 22021 raft_consensus.cc:3055] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:22:32.298290 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:32.239643 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:32.301071 17757 raft_consensus.cc:3060] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:32.240984 17763 raft_consensus.cc:1240] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 8. Current term is 9. Ops: []
I20260709 12:22:32.245270 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:32.302580 18364 raft_consensus.cc:2468] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 11.
I20260709 12:22:32.303232 18364 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59886 (request call id 1224) took 1171 ms. Trace:
I20260709 12:22:32.248088 17801 leader_election.cc:304] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:32.249595 17680 raft_consensus.cc:1240] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 8. Current term is 9. Ops: []
I20260709 12:22:32.251163 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:32.303637 18364 rpcz_store.cc:276] 0709 12:22:31.131352 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:32.244925 (+1113573us) service_pool.cc:224] Handling call
0709 12:22:32.303161 (+ 58236us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:32.305018 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 11 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:32.255806 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:32.215556 21959 raft_consensus.cc:2749] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:22:32.258925 17835 raft_consensus.cc:2468] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 12.
I20260709 12:22:32.263074 17762 raft_consensus.cc:2468] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 10.
W20260709 12:22:32.267396 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 53600.1us
I20260709 12:22:32.168951 21985 raft_consensus.cc:493] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
W20260709 12:22:32.268950 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 57201.1us
I20260709 12:22:32.270205 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:32.286198 21800 consensus_queue.cc:237] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
W20260709 12:22:32.290865 17727 leader_election.cc:336] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837): Timed out: RequestConsensusVote RPC to 127.16.29.67:36837 timed out after 1.699s (SENT)
I20260709 12:22:32.298516 22021 raft_consensus.cc:740] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:22:32.239176 21927 raft_consensus.cc:2272] T 7cb3768d215e409fb9f5baf18d124442 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Raft consensus is shut down!
I20260709 12:22:32.205601 21785 leader_election.cc:290] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:32.215425 21957 raft_consensus.cc:2749] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:22:32.237983 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 12 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:22:32.300344 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 290193us
I20260709 12:22:32.322705 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:32.287231 21965 raft_consensus.cc:2804] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:22:32.255715 17830 raft_consensus.cc:2393] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 10: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:22:32.287213 21843 ts_tablet_manager.cc:1403] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da: Time spent bootstrapping tablet: real 0.986s	user 0.039s	sys 0.017s
W20260709 12:22:32.328022 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 79999.1us
W20260709 12:22:32.328318 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 178042us
I20260709 12:22:32.329843 17801 leader_election.cc:304] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:32.330435 18333 leader_election.cc:304] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:32.332280 18333 leader_election.cc:304] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:32.333328 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:32.315413 17727 leader_election.cc:304] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:22:32.312346 17728 leader_election.cc:304] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:32.171852 22043 consensus_queue.cc:1048] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.115s
I20260709 12:22:32.306156 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:32.323380 17652 leader_election.cc:304] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:32.312846 21985 raft_consensus.cc:3060] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:32.215358 21692 consensus_queue.cc:1059] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.091s
I20260709 12:22:32.344830 17687 raft_consensus.cc:2468] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 12.
I20260709 12:22:32.325800 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:32.350250 17728 leader_election.cc:304] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:32.354569 17684 raft_consensus.cc:1240] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 11. Current term is 12. Ops: []
W20260709 12:22:32.353950 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 63094.4us
W20260709 12:22:32.356511 17727 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:32.305804 22044 raft_consensus.cc:2804] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:22:32.360244 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:32.366195 17837 raft_consensus.cc:3060] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:32.363817 18368 raft_consensus.cc:2393] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 11: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:22:32.348953 17680 raft_consensus.cc:2376] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 10. Re-sending same reply.
I20260709 12:22:32.369447 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:32.172503 22028 consensus_queue.cc:260] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.6, Last appended by leader: 6, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:22:32.371125 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 47818.9us
I20260709 12:22:32.365906 18333 leader_election.cc:304] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:22:32.372622 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:32.324749 21967 raft_consensus.cc:2749] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 12:22:32.298964 21546 consensus_queue.cc:237] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:32.375273 17652 leader_election.cc:304] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:22:32.375540 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:32.375180 17686 raft_consensus.cc:2393] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 12: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:22:32.399127 17684 raft_consensus.cc:1240] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 8. Current term is 9. Ops: []
I20260709 12:22:32.408881 17681 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44898 (request call id 875) took 1049 ms. Trace:
I20260709 12:22:32.409289 17681 rpcz_store.cc:276] 0709 12:22:31.359259 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:31.372917 (+ 13658us) service_pool.cc:224] Handling call
0709 12:22:32.408702 (+1035785us) spinlock_profiling.cc:243] Waited 532 ms on lock 0x7b5400425ed4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:22:32.408814 (+   112us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1224387456}
I20260709 12:22:32.410455 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:32.411182 17830 raft_consensus.cc:2393] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 11: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:22:32.362978 22045 raft_consensus.cc:2749] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:22:32.368799 21786 raft_consensus.cc:2804] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:22:32.364939 21589 raft_consensus.cc:3055] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:22:32.416008 17652 leader_election.cc:304] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:32.363129 22027 raft_consensus.cc:2804] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:22:32.360565 21994 raft_consensus.cc:493] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:32.299265 21768 consensus_queue.cc:237] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:32.211062 22006 consensus_queue.cc:237] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:32.372172 22002 raft_consensus.cc:2804] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:22:32.369454 21965 raft_consensus.cc:697] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 11 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:22:32.382931 22016 raft_consensus.cc:3060] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:32.425886 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:32.376406 21785 consensus_queue.cc:1059] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.068s
I20260709 12:22:32.428514 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:32.466940 17830 raft_consensus.cc:2393] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 12: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:22:32.463178 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:32.432566 17652 leader_election.cc:304] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:32.424451 21589 raft_consensus.cc:740] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:22:32.440687 22015 raft_consensus.cc:2749] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:22:32.490798 17728 leader_election.cc:304] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:32.492260 18334 leader_election.cc:304] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:32.381246 22047 raft_consensus.cc:2749] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:22:32.341399 21827 raft_consensus.cc:515] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:22:32.444860 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 154132us
I20260709 12:22:32.451242 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:32.497108 17838 raft_consensus.cc:2468] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 12.
I20260709 12:22:32.498145 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:32.498764 17838 raft_consensus.cc:3060] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Advancing to term 8
I20260709 12:22:32.361891 21919 raft_consensus.cc:515] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:32.361397 21926 raft_consensus.cc:697] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 11 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
W20260709 12:22:32.462824 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 97010.5us
I20260709 12:22:32.422631 21957 raft_consensus.cc:2804] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:22:32.421125 22022 raft_consensus.cc:2804] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:22:32.463528 17680 raft_consensus.cc:1240] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 10. Current term is 11. Ops: []
I20260709 12:22:32.375437 22039 consensus_queue.cc:1048] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.111s
I20260709 12:22:32.469642 17837 raft_consensus.cc:2468] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 11.
I20260709 12:22:32.434877 17802 leader_election.cc:304] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:32.474874 22044 raft_consensus.cc:697] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 9 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:22:32.477455 17727 leader_election.cc:304] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:22:32.477895 17829 raft_consensus.cc:1240] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 10. Current term is 11. Ops: [10.7-10.7]
I20260709 12:22:32.478757 21786 raft_consensus.cc:697] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 7 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:22:32.480121 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 9 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:32.480661 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:32.485541 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:32.445392 21888 raft_consensus.cc:2749] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:22:32.443099 22028 raft_consensus.cc:3060] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:32.378738 21927 ts_tablet_manager.cc:1916] T 7cb3768d215e409fb9f5baf18d124442 P e2909cff1da44ad387f96a8ae3aa5afb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:22:32.489615 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:32.447278 21994 raft_consensus.cc:3060] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:32.495100 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:32.451027 22002 raft_consensus.cc:493] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:32.454762 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:32.457407 22027 raft_consensus.cc:697] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 12 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:22:32.506240 17760 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 11.
I20260709 12:22:32.512391 22008 raft_consensus.cc:2749] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:22:32.510430 17763 raft_consensus.cc:1240] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 10. Current term is 11. Ops: [10.6-10.6]
I20260709 12:22:32.510483 18333 leader_election.cc:304] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:22:32.490705 21844 raft_consensus.cc:3060] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:32.517729 17837 raft_consensus.cc:1275] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 9 index: 7. Preceding OpId from leader: term: 11 index: 8. (index mismatch)
W20260709 12:22:32.518493 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 83702.9us
I20260709 12:22:32.519215 17832 raft_consensus.cc:3060] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:32.519614 21786 consensus_queue.cc:237] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:32.524470 18368 raft_consensus.cc:3060] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:32.525336 21785 raft_consensus.cc:493] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:32.519544 17836 raft_consensus.cc:2393] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 11: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:22:32.518601 22044 consensus_queue.cc:237] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:32.519798 17829 raft_consensus.cc:1240] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 10. Current term is 11. Ops: [10.9-10.9]
I20260709 12:22:32.521420 17831 raft_consensus.cc:1275] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 9 index: 7. Preceding OpId from leader: term: 12 index: 9. (index mismatch)
I20260709 12:22:32.533252 18367 raft_consensus.cc:2393] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 9: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
W20260709 12:22:32.555871 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 59000.6us
I20260709 12:22:32.583071 17838 raft_consensus.cc:2468] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 8.
I20260709 12:22:32.613368 17802 leader_election.cc:304] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:32.613569 17729 leader_election.cc:304] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:22:32.537880 22002 raft_consensus.cc:3060] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:32.536458 22022 raft_consensus.cc:697] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 11 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:22:32.632529 17836 raft_consensus.cc:1240] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 7. Current term is 8. Ops: []
I20260709 12:22:32.638403 17763 raft_consensus.cc:1275] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20260709 12:22:32.627010 18368 raft_consensus.cc:2468] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 8.
I20260709 12:22:32.641775 17802 leader_election.cc:304] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:32.646696 17760 raft_consensus.cc:1240] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 6. Current term is 7. Ops: []
W20260709 12:22:32.642383 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 151649us
I20260709 12:22:32.642620 17832 raft_consensus.cc:2468] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 9.
I20260709 12:22:32.567638 21996 raft_consensus.cc:2749] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:22:32.649840 17802 leader_election.cc:304] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:32.515290 21843 raft_consensus.cc:359] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [term 9 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:32.649158 17728 leader_election.cc:304] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:32.542073 21707 consensus_queue.cc:1059] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 8.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.098s
I20260709 12:22:32.568171 21965 consensus_queue.cc:237] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:32.626912 21957 raft_consensus.cc:697] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 9 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:22:32.637626 21824 raft_consensus.cc:2804] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260709 12:22:32.722566 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 212182us
W20260709 12:22:32.711167 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 211069us
I20260709 12:22:32.701436 17757 raft_consensus.cc:2410] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 11 because replica has last-logged OpId of term: 10 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 6.
W20260709 12:22:32.706517 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 57443.5us
I20260709 12:22:32.646729 22030 raft_consensus.cc:2749] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:22:32.679260 22001 raft_consensus.cc:2749] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:22:32.681120 21974 raft_consensus.cc:2804] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Leader pre-election won for term 12
W20260709 12:22:32.755028 17803 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:32.619962 21785 raft_consensus.cc:515] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:32.761492 18334 leader_election.cc:304] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:22:32.638840 22008 consensus_queue.cc:1059] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 8.10, Next index: 11, Last known committed idx: 8, Time since last communication: 0.090s
I20260709 12:22:32.790758 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:32.706234 22054 raft_consensus.cc:2804] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Leader pre-election won for term 12
W20260709 12:22:32.768807 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 45316.2us
I20260709 12:22:32.776631 21984 consensus_queue.cc:260] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.7, Last appended by leader: 7, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:32.777340 21957 consensus_queue.cc:237] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 8.10, Last appended by leader: 10, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:32.781136 21996 raft_consensus.cc:2749] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:22:32.784368 21827 leader_election.cc:290] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 10 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:32.785292 21919 leader_election.cc:290] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 12 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:32.683861 21926 consensus_queue.cc:237] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:32.724907 21824 raft_consensus.cc:493] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
W20260709 12:22:32.774745 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 124987us
I20260709 12:22:32.797209 18333 leader_election.cc:304] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:32.770536 21974 raft_consensus.cc:493] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:32.768204 22039 raft_consensus.cc:2804] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:22:32.850768 17802 leader_election.cc:304] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:32.638321 21589 consensus_queue.cc:260] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 10.9, Last appended by leader: 9, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:32.802464 22054 raft_consensus.cc:493] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:32.796031 22030 raft_consensus.cc:493] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:22:32.818425 22015 raft_consensus.cc:2804] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:22:32.855965 17802 leader_election.cc:304] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:32.810175 22057 raft_consensus.cc:2804] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:22:32.865988 21843 raft_consensus.cc:740] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [term 9 LEARNER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Initialized, Role: LEARNER
W20260709 12:22:32.876430 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 79345.1us
I20260709 12:22:32.866700 22054 raft_consensus.cc:3060] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Advancing to term 12
W20260709 12:22:32.880139 22052 log.cc:927] Time spent T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.355s	user 0.001s	sys 0.000s
I20260709 12:22:32.857828 21974 raft_consensus.cc:3060] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Advancing to term 12
W20260709 12:22:32.899147 21997 consensus_meta.cc:287] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.855s	user 0.009s	sys 0.000s
I20260709 12:22:32.809600 22068 consensus_queue.cc:1059] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 11.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.071s
I20260709 12:22:32.851066 22022 consensus_queue.cc:237] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:22:32.900913 22053 log.cc:927] Time spent T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.395s	user 0.000s	sys 0.001s
I20260709 12:22:32.880745 21785 leader_election.cc:290] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:32.850611 21824 raft_consensus.cc:3060] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:32.865823 21984 raft_consensus.cc:3060] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:32.868031 22039 raft_consensus.cc:697] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 9 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:22:32.899910 17834 raft_consensus.cc:2468] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 8.
I20260709 12:22:32.882814 18333 leader_election.cc:304] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:32.859347 22001 raft_consensus.cc:2749] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:22:32.882561 17802 leader_election.cc:400] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 7 pre-election: Vote denied by peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071) with higher term. Message: Invalid argument: T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 7. Current term is 8.
I20260709 12:22:32.908042 17802 leader_election.cc:403] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 7 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:22:32.866263 22070 raft_consensus.cc:3055] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:22:32.869277 22015 raft_consensus.cc:493] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:22:32.934617 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 52179.5us
I20260709 12:22:32.871970 22008 raft_consensus.cc:2804] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:22:32.941670 17802 leader_election.cc:304] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:32.927420 21844 raft_consensus.cc:2749] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:22:32.907779 21589 raft_consensus.cc:3060] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:32.906126 22075 raft_consensus.cc:2749] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:22:32.869134 22069 consensus_queue.cc:1059] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.085s
I20260709 12:22:32.935204 21785 raft_consensus.cc:2749] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Leader pre-election lost for term 7. Reason: Vote denied by peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071) with higher term. Message: Invalid argument: T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 7. Current term is 8.
I20260709 12:22:32.761874 22027 consensus_queue.cc:237] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.9, Last appended by leader: 9, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:32.940788 22015 raft_consensus.cc:3060] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:32.939620 22070 raft_consensus.cc:740] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:22:33.225626 22070 consensus_queue.cc:260] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 8.10, Last appended by leader: 10, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:33.227053 22070 raft_consensus.cc:3060] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:22:33.227519 22002 consensus_meta.cc:287] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.572s	user 0.005s	sys 0.003s
I20260709 12:22:32.940496 22008 raft_consensus.cc:493] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:32.932983 22057 raft_consensus.cc:697] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:22:32.872216 22030 raft_consensus.cc:515] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:32.945962 21997 consensus_queue.cc:1059] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.290s
W20260709 12:22:33.275292 22052 log.cc:927] Time spent T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.320s	user 0.001s	sys 0.000s
W20260709 12:22:33.305112 22016 consensus_meta.cc:287] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.836s	user 0.011s	sys 0.000s
I20260709 12:22:32.969972 22071 raft_consensus.cc:2749] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:22:33.358310 22008 raft_consensus.cc:3060] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Advancing to term 12
W20260709 12:22:33.405279 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 522556us
W20260709 12:22:33.408573 22073 log.cc:927] Time spent T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.438s	user 0.000s	sys 0.001s
I20260709 12:22:32.943238 21843 consensus_queue.cc:260] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 8.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:22:33.415158 18333 leader_election.cc:304] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:33.419651 18362 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59886 (request call id 1227) took 2287 ms. Trace:
I20260709 12:22:33.420163 18362 rpcz_store.cc:276] 0709 12:22:31.132724 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:32.241394 (+1108670us) service_pool.cc:224] Handling call
0709 12:22:32.241652 (+   258us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:32.241963 (+   311us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:32.241989 (+    26us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:32.242367 (+   378us) log.cc:844] Serialized 26 byte log entry
0709 12:22:32.394648 (+152281us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:22:32.431273 (+ 36625us) log.cc:844] Serialized 14 byte log entry
0709 12:22:32.431510 (+   237us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:32.431578 (+    68us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:32.633067 (+201489us) spinlock_profiling.cc:243] Waited 11.1 ms on lock 0x7b18003e0a40. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b47533522 00007f7b475357c1 00007f7b47535779 00007f7b47535709 00007f7b475356d1 00007f7b475349cd 0000000000388634 0000000000388479 00007f7b4751683a
0709 12:22:33.419186 (+786119us) raft_consensus.cc:1685] finished
0709 12:22:33.419364 (+   178us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:33.419573 (+   209us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":46174,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":44,"raft.queue_time_us":12726,"raft.run_cpu_time_us":81,"raft.run_wall_time_us":81,"spinlock_wait_cycles":35022592,"thread_start_us":151815,"threads_started":1,"wal-append.queue_time_us":209656}
I20260709 12:22:33.424058 17684 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37396 (request call id 2214) took 1015 ms. Trace:
I20260709 12:22:33.432394 17684 rpcz_store.cc:276] 0709 12:22:32.416156 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:32.416671 (+   515us) service_pool.cc:224] Handling call
0709 12:22:32.417244 (+   573us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:32.538957 (+121713us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:32.539002 (+    45us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:32.539047 (+    45us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:22:32.539088 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:33.423797 (+884709us) spinlock_profiling.cc:243] Waited 121 ms on lock 0x7b5400252bd4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:22:33.423858 (+    61us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:33.424003 (+   145us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":279405952}
I20260709 12:22:33.427440 22057 consensus_queue.cc:237] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:33.437292 22030 leader_election.cc:290] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:33.437606 22089 raft_consensus.cc:493] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:33.438272 22089 raft_consensus.cc:515] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:33.400066 22016 raft_consensus.cc:2749] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:22:33.371313 22002 raft_consensus.cc:515] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:33.450240 18364 raft_consensus.cc:1240] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 9. Current term is 10. Ops: []
W20260709 12:22:33.447465 21974 consensus_meta.cc:287] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.551s	user 0.003s	sys 0.007s
I20260709 12:22:33.455401 22016 raft_consensus.cc:2749] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:22:33.383404 22039 consensus_queue.cc:237] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:22:33.501489 21848 consensus_meta.cc:287] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 1.209s	user 0.010s	sys 0.009s
W20260709 12:22:33.491298 21692 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.599s	user 0.003s	sys 0.014s
W20260709 12:22:33.503413 21985 consensus_meta.cc:287] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 1.131s	user 0.013s	sys 0.004s
I20260709 12:22:33.486697 22089 leader_election.cc:290] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:22:33.508905 21692 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.655s	user 0.003s	sys 0.015s
I20260709 12:22:33.474552 21974 raft_consensus.cc:515] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:22:33.566028 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 159349us
I20260709 12:22:33.567735 18333 leader_election.cc:304] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:22:33.518643 21985 raft_consensus.cc:515] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:22:33.565649 22023 consensus_meta.cc:287] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 1.553s	user 0.009s	sys 0.005s
I20260709 12:22:33.572750 21843 ts_tablet_manager.cc:1434] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da: Time spent starting tablet: real 1.235s	user 0.008s	sys 0.024s
W20260709 12:22:33.573227 21843 ts_tablet_manager.cc:1467] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da: Tablet startup took 1245ms
I20260709 12:22:33.575618 18333 leader_election.cc:304] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:22:33.580551 21589 consensus_meta.cc:287] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.616s	user 0.005s	sys 0.005s
I20260709 12:22:33.762794 22089 raft_consensus.cc:2749] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:22:33.570050 22002 leader_election.cc:290] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:22:33.572789 22086 log.cc:927] Time spent T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.067s	user 0.000s	sys 0.000s
W20260709 12:22:33.781006 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 205443us
I20260709 12:22:33.788759 18333 leader_election.cc:304] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:22:33.789728 22097 raft_consensus.cc:2749] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260709 12:22:33.790593 21994 consensus_meta.cc:287] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 1.246s	user 0.010s	sys 0.002s
I20260709 12:22:33.791060 21994 raft_consensus.cc:515] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:33.576822 22023 raft_consensus.cc:515] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:33.793176 21994 leader_election.cc:290] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 12 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:33.793783 22023 leader_election.cc:290] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 10 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:22:33.573592 21843 ts_tablet_manager.cc:1469] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da: Trace:
0709 12:22:24.156059 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:24.158888 (+  2829us) service_pool.cc:224] Handling call
0709 12:22:24.542008 (+383120us) ts_tablet_manager.cc:934] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da: Initiating tablet copy from peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
0709 12:22:26.309603 (+1767595us) tablet_copy_client.cc:368] Tablet copy session begun
0709 12:22:27.243865 (+934262us) tablet_metadata.cc:665] Metadata flushed
0709 12:22:27.244354 (+   489us) tablet_copy_client.cc:497] Wrote new tablet metadata
0709 12:22:27.252481 (+  8127us) tablet_copy_client.cc:418] Wrote new consensus metadata
0709 12:22:27.252810 (+   329us) tablet_replica.cc:206] Creating consensus instance
0709 12:22:27.253135 (+   325us) ts_tablet_manager.cc:989] Replica registered.
0709 12:22:27.253215 (+    80us) inbound_call.cc:177] Queueing success response
0709 12:22:31.301051 (+4047836us) ts_tablet_manager.cc:1379] Bootstrapping tablet
0709 12:22:32.361665 (+1060614us) ts_tablet_manager.cc:1435] Starting tablet replica
0709 12:22:32.448357 (+ 86692us) tablet_replica.cc:261] Starting instrumentation
0709 12:22:32.448901 (+   544us) tablet_replica.cc:280] Starting consensus
Metrics: {"dns_us":27408,"raft.queue_time_us":36048,"raft.run_cpu_time_us":178,"raft.run_wall_time_us":27853,"tablet-copy.queue_time_us":226041,"tablet-download-pool.queue_time_us":425067,"tablet-download-pool.run_cpu_time_us":1815,"tablet-download-pool.run_wall_time_us":3429009,"thread_start_us":189258,"threads_started":2}
W20260709 12:22:33.795594 22070 consensus_meta.cc:287] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.568s	user 0.008s	sys 0.001s
W20260709 12:22:33.796240 22054 consensus_meta.cc:287] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.918s	user 0.006s	sys 0.005s
I20260709 12:22:33.805212 21974 leader_election.cc:290] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 12 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:22:33.796658 22090 log.cc:927] Time spent T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.286s	user 0.001s	sys 0.000s
I20260709 12:22:33.798188 21985 leader_election.cc:290] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 11 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:22:33.802542 21984 consensus_meta.cc:287] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.875s	user 0.007s	sys 0.001s
I20260709 12:22:33.804653 18333 leader_election.cc:304] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:33.812839 22097 raft_consensus.cc:2749] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20260709 12:22:33.813781 21824 consensus_meta.cc:287] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.880s	user 0.007s	sys 0.003s
I20260709 12:22:33.814299 21824 raft_consensus.cc:515] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:22:33.814935 17654 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.019s	user 0.019s	sys 0.007s
I20260709 12:22:33.805913 22089 raft_consensus.cc:2749] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260709 12:22:33.815302 17654 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.020s	user 0.019s	sys 0.007s
I20260709 12:22:33.805261 22054 raft_consensus.cc:515] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:22:33.810487 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:33.815990 22015 consensus_meta.cc:287] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.592s	user 0.004s	sys 0.006s
I20260709 12:22:33.816471 22015 raft_consensus.cc:515] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:33.817080 21824 leader_election.cc:290] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 7 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:33.810696 21927 ts_tablet_manager.cc:1929] T 7cb3768d215e409fb9f5baf18d124442 P e2909cff1da44ad387f96a8ae3aa5afb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.5
I20260709 12:22:33.817358 21995 raft_consensus.cc:3055] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:22:33.817898 21927 log.cc:1199] T 7cb3768d215e409fb9f5baf18d124442 P e2909cff1da44ad387f96a8ae3aa5afb: Deleting WAL directory at /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599551912677-16501-0/minicluster-data/ts-1-root/wals/7cb3768d215e409fb9f5baf18d124442
I20260709 12:22:33.818002 21995 raft_consensus.cc:740] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 8 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:22:33.814786 18333 leader_election.cc:304] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3
W20260709 12:22:33.796446 22091 log.cc:927] Time spent T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.226s	user 0.000s	sys 0.000s
W20260709 12:22:33.815596 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 1.27654e+06us
I20260709 12:22:33.817446 22054 leader_election.cc:290] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 12 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:33.819765 22015 leader_election.cc:290] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 11 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:33.822355 17654 leader_election.cc:304] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:22:33.832681 21927 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38774 (request call id 91) took 8011 ms. Trace:
I20260709 12:22:33.833204 21927 rpcz_store.cc:276] 0709 12:22:25.821708 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:25.852398 (+ 30690us) service_pool.cc:224] Handling call
0709 12:22:26.140832 (+288434us) ts_tablet_manager.cc:1151] Deleting tablet 7cb3768d215e409fb9f5baf18d124442
0709 12:22:26.140951 (+   119us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0709 12:22:33.499361 (+7358410us) tablet_metadata.cc:665] Metadata flushed
0709 12:22:33.810542 (+311181us) tablet_metadata.cc:665] Metadata flushed
0709 12:22:33.832578 (+ 22036us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":6015224,"tablet-delete.queue_time_us":226444,"thread_start_us":135127,"threads_started":1}
I20260709 12:22:33.822615 18333 leader_election.cc:304] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:22:33.822675 21995 consensus_queue.cc:260] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:33.836212 21995 raft_consensus.cc:3060] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:33.836508 22015 raft_consensus.cc:2749] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:22:33.840265 18333 leader_election.cc:304] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3
W20260709 12:22:33.826941 17763 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.020s	user 0.008s	sys 0.008s
W20260709 12:22:33.841132 17763 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.034s	user 0.009s	sys 0.009s
I20260709 12:22:33.827802 22070 raft_consensus.cc:2804] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:22:33.841642 18333 leader_election.cc:304] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:33.842048 22070 raft_consensus.cc:493] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:33.842535 22070 raft_consensus.cc:3060] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:33.846500 17763 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38776 (request call id 2146) took 1998 ms. Trace:
I20260709 12:22:33.847013 17763 rpcz_store.cc:276] 0709 12:22:31.847807 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:32.678930 (+831123us) service_pool.cc:224] Handling call
0709 12:22:32.723349 (+ 44419us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:32.723697 (+   348us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 12:22:32.723735 (+    38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:32.760153 (+ 36418us) log.cc:844] Serialized 26 byte log entry
0709 12:22:33.841514 (+1081361us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:22:33.841554 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:33.841622 (+    68us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:33.841664 (+    42us) raft_consensus.cc:1685] finished
0709 12:22:33.841675 (+    11us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:33.846431 (+  4756us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":1034155,"threads_started":1,"wal-append.queue_time_us":635214}
I20260709 12:22:33.830236 22008 raft_consensus.cc:515] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:33.848357 22097 raft_consensus.cc:2804] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:22:33.849062 22097 raft_consensus.cc:697] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 11 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:22:33.850013 22089 raft_consensus.cc:2804] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:22:33.850474 22089 raft_consensus.cc:493] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:22:33.850850 22089 raft_consensus.cc:3060] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:33.855088 22008 leader_election.cc:290] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 12 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:33.856621 22097 consensus_queue.cc:237] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:33.858143 22021 consensus_queue.cc:260] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:33.859247 22021 raft_consensus.cc:3060] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:33.860188 22089 raft_consensus.cc:515] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:33.862726 22089 leader_election.cc:290] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 11 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:22:33.867246 22076 log.cc:927] Time spent T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.925s	user 0.000s	sys 0.002s
I20260709 12:22:33.871953 22097 raft_consensus.cc:493] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:33.872483 22070 raft_consensus.cc:515] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:33.872776 22097 raft_consensus.cc:515] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:33.874953 22070 leader_election.cc:290] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 10 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:33.875275 22097 leader_election.cc:290] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:22:33.876075 18333 connection.cc:256] RPC call timeout handler was delayed by -1.7123650610001278s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1547s for a response
W20260709 12:22:33.876482 18333 connection.cc:256] RPC call timeout handler was delayed by -1.6430650610000157s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16240000000000002s for a response
W20260709 12:22:33.876748 18333 connection.cc:256] RPC call timeout handler was delayed by -1.2322040440003548s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2042s for a response
W20260709 12:22:33.876904 17727 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.344s	user 0.012s	sys 0.007s
W20260709 12:22:33.877252 17727 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.344s	user 0.012s	sys 0.007s
W20260709 12:22:33.877673 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 1.40026e+06us
I20260709 12:22:33.878218 17727 leader_election.cc:304] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:22:33.878518 18333 leader_election.cc:304] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:33.878872 22027 raft_consensus.cc:2749] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:22:33.879408 22097 raft_consensus.cc:3060] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:33.879904 18333 leader_election.cc:400] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 10 pre-election: Vote denied by peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651) with higher term. Message: Invalid argument: T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 10. Current term is 11.
I20260709 12:22:33.879662 17727 leader_election.cc:304] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:33.881515 17830 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35040 (request call id 2365) took 1409 ms. Trace:
I20260709 12:22:33.881908 17830 rpcz_store.cc:276] 0709 12:22:32.471707 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:32.473613 (+  1906us) service_pool.cc:224] Handling call
0709 12:22:32.473998 (+   385us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:32.496184 (+ 22186us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:32.496216 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:32.626575 (+130359us) log.cc:844] Serialized 26 byte log entry
0709 12:22:32.774105 (+147530us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:22:32.774160 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:32.774247 (+    87us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:33.853553 (+1079306us) spinlock_profiling.cc:243] Waited 3.28 ms on lock 0x7b4800190158. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4f549aa1 00007f7b4f540b37 00007f7b4f5409ca 00007f7b4f63826f 00007f7b4f637f37 00007f7b4f64c55c 00007f7b4f64c508 00007f7b4f64c488 00007f7b4f64c443 00007f7b4f64b6ec 00007f7b4e575a96
0709 12:22:33.866658 (+ 13105us) spinlock_profiling.cc:243] Waited 6.99 ms on lock 0x7b4400022b4c. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b47459c85 00007f7b4745a6a2 00007f7b4a1cc1db 00007f7b4a1c7aaf 00007f7b4a1bb86e 00007f7b4a1babb6 00007f7b4a1ba7d8 00007f7b4a1c4221 00007f7b4a1c41d9 00007f7b4a1c4169
0709 12:22:33.881305 (+ 14647us) raft_consensus.cc:1685] finished
0709 12:22:33.881325 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:33.881456 (+   131us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":21683,"raft.queue_time_us":24047,"raft.run_cpu_time_us":9723,"raft.run_wall_time_us":1352876,"spinlock_wait_cycles":23608960,"thread_start_us":81877,"threads_started":1,"wal-append.queue_time_us":223305}
I20260709 12:22:33.884181 22027 raft_consensus.cc:2804] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:22:33.880313 18333 leader_election.cc:403] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 10 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:22:33.885020 22027 raft_consensus.cc:697] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:22:33.885278 22070 raft_consensus.cc:2749] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Leader pre-election lost for term 10. Reason: Vote denied by peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651) with higher term. Message: Invalid argument: T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 10. Current term is 11.
I20260709 12:22:33.886104 22027 consensus_queue.cc:237] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:22:33.890246 22051 log.cc:927] Time spent T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 1.210s	user 0.000s	sys 0.011s
I20260709 12:22:33.895308 17727 leader_election.cc:304] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:22:33.896534 22025 raft_consensus.cc:2804] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:22:33.896909 22025 raft_consensus.cc:493] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:33.897245 22025 raft_consensus.cc:3060] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:33.908670 17727 leader_election.cc:304] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:22:33.909870 22025 raft_consensus.cc:515] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:33.913108 22025 leader_election.cc:290] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 9 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:33.914310 17727 leader_election.cc:304] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:22:33.915666 18368 raft_consensus.cc:1240] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 8. Current term is 9. Ops: [8.9-8.9]
I20260709 12:22:33.917052 22097 raft_consensus.cc:2749] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:22:33.918032 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:33.918644 18367 raft_consensus.cc:3060] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:33.919799 22025 raft_consensus.cc:3060] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:33.919191 17727 leader_election.cc:304] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:22:33.929044 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:33.929708 18362 raft_consensus.cc:3060] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:33.927532 18367 raft_consensus.cc:2468] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 8.
I20260709 12:22:33.931908 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:33.932631 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:33.936365 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:22:33.943063 17834 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35042 (request call id 2090) took 1896 ms (client timeout 1699 ms). Trace:
W20260709 12:22:33.943642 17834 rpcz_store.cc:269] 0709 12:22:32.046875 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:32.047383 (+   508us) service_pool.cc:224] Handling call
0709 12:22:33.942798 (+1895415us) spinlock_profiling.cc:243] Waited 772 ms on lock 0x7b54000898d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:22:33.942949 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1774550400}
I20260709 12:22:33.953410 17683 raft_consensus.cc:1240] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 8 LEADER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 7. Current term is 8. Ops: []
I20260709 12:22:33.955312 18362 raft_consensus.cc:2468] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 12.
I20260709 12:22:33.957614 22027 raft_consensus.cc:2749] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:22:33.958966 17727 leader_election.cc:304] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:22:33.959738 22027 raft_consensus.cc:3060] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:33.959800 17688 raft_consensus.cc:1240] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 11. Current term is 12. Ops: []
I20260709 12:22:33.961194 17688 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37396 (request call id 2216) took 1473 ms. Trace:
I20260709 12:22:33.961556 17688 rpcz_store.cc:276] 0709 12:22:32.487409 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:32.487590 (+   181us) service_pool.cc:224] Handling call
0709 12:22:32.502147 (+ 14557us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:33.960354 (+1458207us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:33.960930 (+   576us) spinlock_profiling.cc:243] Waited 1.46 s on lock 0x7b5400348ad4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:22:33.961127 (+   197us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3351820544}
I20260709 12:22:33.965831 22025 raft_consensus.cc:2749] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:22:33.966075 17727 leader_election.cc:304] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:22:33.967312 17683 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37396 (request call id 2213) took 1591 ms. Trace:
I20260709 12:22:33.967795 17683 rpcz_store.cc:276] 0709 12:22:32.364582 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:32.364715 (+   133us) service_pool.cc:224] Handling call
0709 12:22:32.365035 (+   320us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:33.953989 (+1588954us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:33.955644 (+  1655us) spinlock_profiling.cc:243] Waited 1.59 s on lock 0x7b54003002d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:22:33.955850 (+   206us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3652387712}
I20260709 12:22:33.967796 22025 raft_consensus.cc:2749] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:22:33.972227 22017 raft_consensus.cc:3060] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:22:33.973704 17729 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:33.974313 17729 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:33.974797 21824 raft_consensus.cc:493] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:22:33.976032 21984 raft_consensus.cc:493] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:33.976559 21984 raft_consensus.cc:515] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:33.978649 21984 leader_election.cc:290] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:33.979017 17759 raft_consensus.cc:1240] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 10. Current term is 11. Ops: []
I20260709 12:22:33.980206 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:33.980705 18362 raft_consensus.cc:1240] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 11. Current term is 12. Ops: []
I20260709 12:22:33.980958 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:33.981647 17757 raft_consensus.cc:1240] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 6. Current term is 7. Ops: []
I20260709 12:22:33.982542 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:33.983135 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:33.983981 17760 raft_consensus.cc:2468] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 8.
I20260709 12:22:33.984089 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:33.984897 18367 raft_consensus.cc:3060] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:33.986888 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:33.988234 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:33.989116 17755 raft_consensus.cc:1240] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 11. Current term is 12. Ops: []
I20260709 12:22:33.990515 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:33.975312 22054 raft_consensus.cc:493] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:33.992163 22025 consensus_queue.cc:1059] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 11.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 12:22:33.975323 21824 raft_consensus.cc:515] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:33.993371 22025 raft_consensus.cc:3055] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 LEADER]: Stepping down as leader of term 11
W20260709 12:22:33.993652 17756 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.459s	user 0.024s	sys 0.083s
I20260709 12:22:33.993732 22025 raft_consensus.cc:740] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
W20260709 12:22:33.994230 17756 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.462s	user 0.024s	sys 0.083s
I20260709 12:22:33.994500 21824 leader_election.cc:290] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:33.994694 22025 consensus_queue.cc:260] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 11.11, Last appended by leader: 11, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:33.992067 22054 raft_consensus.cc:515] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:33.996157 22025 raft_consensus.cc:3060] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:33.996308 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:33.997629 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:33.998977 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:33.998865 22027 raft_consensus.cc:2749] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:22:34.000295 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:33.999624 17760 raft_consensus.cc:3060] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:34.004030 22054 leader_election.cc:290] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:34.006767 17681 raft_consensus.cc:1240] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 8 LEADER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 7. Current term is 8. Ops: []
I20260709 12:22:34.011205 18367 raft_consensus.cc:2468] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 12.
I20260709 12:22:34.014225 22055 raft_consensus.cc:3055] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:22:34.015033 17760 raft_consensus.cc:2468] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 11.
W20260709 12:22:34.019455 22063 log.cc:927] Time spent T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 1.173s	user 0.001s	sys 0.000s
I20260709 12:22:34.022466 22055 raft_consensus.cc:740] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 8 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:22:34.024755 17764 raft_consensus.cc:2410] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 9 because replica has last-logged OpId of term: 8 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
W20260709 12:22:34.019954 18333 leader_election.cc:336] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 15 pre-election: RPC error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Timed out: RequestConsensusVote RPC to 127.16.29.66:39651 timed out after 1.547s (SENT)
I20260709 12:22:34.025368 18333 leader_election.cc:304] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:34.024576 22021 raft_consensus.cc:2749] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260709 12:22:34.026952 17764 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38786 (request call id 2819) took 2183 ms (client timeout 2903 ms). Trace:
I20260709 12:22:34.023636 22055 consensus_queue.cc:260] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:34.031265 17756 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38786 (request call id 2816) took 2188 ms. Trace:
I20260709 12:22:34.031656 17756 rpcz_store.cc:276] 0709 12:22:31.842810 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:32.486859 (+644049us) service_pool.cc:224] Handling call
0709 12:22:32.503086 (+ 16227us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:33.994644 (+1491558us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:33.994675 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:33.995201 (+   526us) log.cc:844] Serialized 26 byte log entry
0709 12:22:34.023489 (+ 28288us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:22:34.023796 (+   307us) log.cc:844] Serialized 14 byte log entry
0709 12:22:34.023990 (+   194us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:34.024150 (+   160us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:34.031036 (+  6886us) raft_consensus.cc:1685] finished
0709 12:22:34.031058 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:34.031202 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1999,"prepare.queue_time_us":278,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":37,"raft.queue_time_us":121050,"raft.run_cpu_time_us":247,"raft.run_wall_time_us":40770,"thread_start_us":1489723,"threads_started":2,"wal-append.queue_time_us":28194}
I20260709 12:22:34.046264 22039 raft_consensus.cc:2749] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
W20260709 12:22:34.021692 22060 log.cc:927] Time spent T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 1.175s	user 0.000s	sys 0.001s
W20260709 12:22:34.027416 17764 rpcz_store.cc:269] 0709 12:22:31.843817 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:32.236506 (+392689us) service_pool.cc:224] Handling call
0709 12:22:34.026726 (+1790220us) spinlock_profiling.cc:243] Waited 1.74 s on lock 0x7b540019f5d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:22:34.026890 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3992713984}
I20260709 12:22:34.102416 22067 raft_consensus.cc:3055] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:22:34.103006 22067 raft_consensus.cc:740] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:22:34.104137 22067 consensus_queue.cc:260] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.9, Last appended by leader: 9, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:34.106294 22067 raft_consensus.cc:3060] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Advancing to term 12
W20260709 12:22:34.131528 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 1.49269e+06us
I20260709 12:22:34.131724 22092 consensus_queue.cc:1048] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:22:34.141162 21674 raft_consensus.cc:2749] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 12:22:34.142057 22092 raft_consensus.cc:493] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:34.142627 22092 raft_consensus.cc:515] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:34.145895 17758 raft_consensus.cc:2468] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 10.
I20260709 12:22:34.146060 22092 leader_election.cc:290] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:22:34.135205 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 312919us
I20260709 12:22:34.152244 17654 leader_election.cc:304] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
W20260709 12:22:34.137691 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 1.19603e+06us
I20260709 12:22:34.158099 21952 raft_consensus.cc:493] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:34.158913 21952 raft_consensus.cc:515] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:34.161808 21952 leader_election.cc:290] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:34.163985 17727 leader_election.cc:304] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:34.165041 22044 raft_consensus.cc:2804] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:22:34.165489 22044 raft_consensus.cc:493] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:34.165622 17689 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37396 (request call id 2206) took 1913 ms. Trace:
I20260709 12:22:34.165969 22044 raft_consensus.cc:3060] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:34.166044 17689 rpcz_store.cc:276] 0709 12:22:32.252481 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:32.320226 (+ 67745us) service_pool.cc:224] Handling call
0709 12:22:32.320650 (+   424us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:32.320911 (+   261us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 12:22:32.473009 (+152098us) log.cc:844] Serialized 14 byte log entry
0709 12:22:33.887016 (+1414007us) spinlock_profiling.cc:243] Waited 5.83 ms on lock 0x7b48000030d8. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4f549aa1 00007f7b4f540b37 00007f7b4f5409ca 00007f7b4f63826f 00007f7b4f637f37 00007f7b4f64c55c 00007f7b4f64c508 00007f7b4f64c488 00007f7b4f64c443 00007f7b4f64b6ec 00007f7b4e575a96
0709 12:22:34.018834 (+131818us) spinlock_profiling.cc:243] Waited 53.3 ms on lock 0x7b4400022b4c. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b47459c85 00007f7b4745a6a2 00007f7b4a1cc1db 00007f7b4a1c7aaf 00007f7b4a1bb86e 00007f7b4a1babb6 00007f7b4a1ba7d8 00007f7b4a1c4221 00007f7b4a1c41d9 00007f7b4a1c4169
0709 12:22:34.165126 (+146292us) spinlock_profiling.cc:243] Waited 2.69 ms on lock 0x7b1800259240. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b4a1c84bc 00007f7b4751648b 00007f7b503709a9 00007f7b4753269b 00007f7b4753093f 00007f7b47530455 00007f7b4a618060 00007f7b4a62a716 00007f7b4a63b287
0709 12:22:34.165234 (+   108us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:34.165293 (+    59us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:22:34.165350 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:34.165448 (+    98us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:34.165567 (+   119us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":268193,"prepare.queue_time_us":16119,"prepare.run_cpu_time_us":55,"prepare.run_wall_time_us":56,"raft.queue_time_us":276104,"raft.run_cpu_time_us":13139,"raft.run_wall_time_us":1077702,"spinlock_wait_cycles":142081280,"thread_start_us":249602,"threads_started":3,"wal-append.queue_time_us":289562}
W20260709 12:22:34.196480 17802 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:34.202518 22044 raft_consensus.cc:515] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:34.203783 21952 raft_consensus.cc:493] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:34.212793 21952 raft_consensus.cc:515] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:34.213615 18366 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 1386) took 1990 ms. Trace:
I20260709 12:22:34.214035 18366 rpcz_store.cc:276] 0709 12:22:32.222860 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:32.222992 (+   132us) service_pool.cc:224] Handling call
0709 12:22:32.223307 (+   315us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:22:32.363564 (+140257us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:32.363602 (+    38us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:22:32.381458 (+ 17856us) log.cc:844] Serialized 52 byte log entry
0709 12:22:32.763139 (+381681us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:22:32.763178 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:34.213158 (+1449980us) spinlock_profiling.cc:243] Waited 113 ms on lock 0x7b54007e8954. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:22:34.213214 (+    56us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:34.213261 (+    47us) raft_consensus.cc:1685] finished
0709 12:22:34.213273 (+    12us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:34.213550 (+   277us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":26601,"raft.queue_time_us":38840,"raft.run_cpu_time_us":93,"raft.run_wall_time_us":95,"spinlock_wait_cycles":260192000,"thread_start_us":338796,"threads_started":1,"wal-append.queue_time_us":489441}
I20260709 12:22:34.204948 17681 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37402 (request call id 2595) took 1776 ms. Trace:
I20260709 12:22:34.215834 17681 rpcz_store.cc:276] 0709 12:22:32.428216 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:32.428630 (+   414us) service_pool.cc:224] Handling call
0709 12:22:32.430682 (+  2052us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:34.008590 (+1577908us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:34.204652 (+196062us) spinlock_profiling.cc:243] Waited 1.58 s on lock 0x7b5400206a54. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:22:34.204867 (+   215us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3624274304}
I20260709 12:22:34.217083 18364 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59886 (request call id 1250) took 1310 ms. Trace:
I20260709 12:22:34.217495 18364 rpcz_store.cc:276] 0709 12:22:32.907108 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:32.907300 (+   192us) service_pool.cc:224] Handling call
0709 12:22:32.940877 (+ 33577us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:33.450848 (+509971us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:34.216841 (+765993us) spinlock_profiling.cc:243] Waited 509 ms on lock 0x7b540051a254. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:22:34.217023 (+   182us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1170854784}
I20260709 12:22:34.215126 21952 leader_election.cc:290] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:34.219080 21952 raft_consensus.cc:2804] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:22:34.219688 21952 raft_consensus.cc:697] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 12 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:22:34.221148 21952 consensus_queue.cc:237] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.8, Last appended by leader: 8, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:34.211496 22112 raft_consensus.cc:493] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:34.234220 22112 raft_consensus.cc:515] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:34.243079 22044 leader_election.cc:290] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 13 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:22:34.254886 17730 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:34.260053 17730 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:34.262228 21991 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.449s	user 0.010s	sys 0.001s
W20260709 12:22:34.262917 21991 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.463s	user 0.011s	sys 0.002s
I20260709 12:22:34.263551 22045 raft_consensus.cc:493] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:34.265126 22045 raft_consensus.cc:515] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:34.268265 22045 leader_election.cc:290] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:34.271749 17764 raft_consensus.cc:1240] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 9 LEADER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 8. Current term is 9. Ops: [8.6-8.6]
I20260709 12:22:34.273154 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:34.271709 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:34.282707 17760 raft_consensus.cc:2468] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 8.
I20260709 12:22:34.283732 22045 raft_consensus.cc:493] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:34.284452 22045 raft_consensus.cc:515] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:34.287415 22045 leader_election.cc:290] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:34.289443 17756 raft_consensus.cc:1240] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 9. Current term is 11. Ops: []
W20260709 12:22:34.291808 17758 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36572 (request call id 762) took 3139 ms (client timeout 1971 ms). Trace:
W20260709 12:22:34.292361 17758 rpcz_store.cc:269] 0709 12:22:31.152109 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:31.166223 (+ 14114us) service_pool.cc:224] Handling call
0709 12:22:31.763862 (+597639us) spinlock_profiling.cc:243] Waited 22.6 ms on lock 0x7b5800033398. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b474fbb71 00007f7b474fba22 00007f7b504148d9 00007f7b4f56eee0 00007f7b4f6380ab 00007f7b4f638131 00007f7b4f5e60a0 00007f7b4f5e5984 00007f7b48b606d4 00007f7b48b60662
0709 12:22:34.291572 (+2527710us) spinlock_profiling.cc:243] Waited 2.38 s on lock 0x7b54003eb554. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:22:34.291700 (+   128us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5527877760}
I20260709 12:22:34.295831 22067 raft_consensus.cc:493] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:34.296947 17729 leader_election.cc:304] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:34.298870 21967 consensus_queue.cc:1059] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 12:22:34.299270 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:34.241869 22112 leader_election.cc:290] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:34.300876 22112 raft_consensus.cc:493] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:34.296567 22067 raft_consensus.cc:515] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:34.301653 22112 raft_consensus.cc:515] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:34.303135 22067 leader_election.cc:290] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:34.304677 22112 leader_election.cc:290] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:34.305258 22112 raft_consensus.cc:493] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:34.305737 22112 raft_consensus.cc:515] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:34.306097 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:34.182577 17571 catalog_manager.cc:5028] TS e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): tablet 7cb3768d215e409fb9f5baf18d124442 (table test-workload [id=ddc5b74250df4c65909b73b3742d615f]) successfully deleted
I20260709 12:22:34.306792 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:22:34.307150 17571 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 124772us
I20260709 12:22:34.307472 21974 raft_consensus.cc:2804] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:22:34.308109 21974 raft_consensus.cc:697] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:22:34.308224 17587 catalog_manager.cc:5697] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 5 to 7. New cstate: current_term: 7 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:22:34.309243 21974 consensus_queue.cc:237] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:34.309929 17682 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44898 (request call id 878) took 2069 ms. Trace:
I20260709 12:22:34.310420 17682 rpcz_store.cc:276] 0709 12:22:32.240980 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:32.241104 (+   124us) service_pool.cc:224] Handling call
0709 12:22:32.241402 (+   298us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:32.241699 (+   297us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:32.241724 (+    25us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:32.242130 (+   406us) log.cc:844] Serialized 26 byte log entry
0709 12:22:32.431161 (+189031us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:22:32.431205 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:32.431270 (+    65us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:33.889523 (+1458253us) spinlock_profiling.cc:243] Waited 12.2 ms on lock 0x7b4400022b4c. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b47459c85 00007f7b4745a6a2 00007f7b4a1cc1db 00007f7b4a1c7aaf 00007f7b4a1bb86e 00007f7b4a1babb6 00007f7b4a1ba7d8 00007f7b4a1c4221 00007f7b4a1c41d9 00007f7b4a1c4169
0709 12:22:34.309650 (+420127us) raft_consensus.cc:1685] finished
0709 12:22:34.309670 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:34.309873 (+   203us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1170,"raft.run_cpu_time_us":91,"raft.run_wall_time_us":92,"spinlock_wait_cycles":27979648,"thread_start_us":188657,"threads_started":1,"wal-append.queue_time_us":290170}
I20260709 12:22:34.314667 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:34.315439 17762 raft_consensus.cc:2393] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 9: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
W20260709 12:22:34.320250 17801 connection.cc:256] RPC call timeout handler was delayed by -1.319995795000068s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16180000000000003s for a response
I20260709 12:22:34.309899 17587 catalog_manager.cc:5697] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 6 to 7, leader changed from 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) to 334a4e1f59944544b3b3ff36616b57da (127.16.29.68). New cstate: current_term: 7 leader_uuid: "334a4e1f59944544b3b3ff36616b57da" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:22:34.327343 17774 tablet_copy_service.cc:342] P e2909cff1da44ad387f96a8ae3aa5afb: Request end of tablet copy session 334a4e1f59944544b3b3ff36616b57da-d48b7de191684318aa08f60a7a395387 received from {username='slave'} at 127.0.0.1:36572
I20260709 12:22:34.332388 17774 tablet_copy_service.cc:434] P e2909cff1da44ad387f96a8ae3aa5afb: ending tablet copy session 334a4e1f59944544b3b3ff36616b57da-d48b7de191684318aa08f60a7a395387 on tablet d48b7de191684318aa08f60a7a395387 with peer 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:34.332823 17760 raft_consensus.cc:1240] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 8. Current term is 9. Ops: []
I20260709 12:22:34.333266 22112 leader_election.cc:290] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:34.333842 22112 raft_consensus.cc:493] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
W20260709 12:22:34.333966 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:34.334899 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:34.335969 17762 raft_consensus.cc:2393] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 11: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:22:34.337373 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:34.338195 17762 raft_consensus.cc:2393] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 12: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:22:34.334367 22112 raft_consensus.cc:515] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:34.340126 17762 raft_consensus.cc:1240] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 11. Current term is 12. Ops: []
I20260709 12:22:34.341320 22015 raft_consensus.cc:493] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:34.341766 18371 raft_consensus.cc:1240] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 11. Current term is 12. Ops: []
I20260709 12:22:34.341917 22015 raft_consensus.cc:515] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:34.343485 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:34.344161 18368 raft_consensus.cc:2468] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 7.
I20260709 12:22:34.345511 21952 raft_consensus.cc:493] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
W20260709 12:22:34.349817 18334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:34.350270 22054 consensus_queue.cc:1059] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 11.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.032s
I20260709 12:22:34.382014 22116 raft_consensus.cc:493] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:34.382570 22116 raft_consensus.cc:515] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:34.383689 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 12 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:34.384327 17838 raft_consensus.cc:3060] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:34.384717 22116 leader_election.cc:290] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:34.382830 22054 raft_consensus.cc:3055] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:22:34.387217 22054 raft_consensus.cc:740] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 11 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:22:34.388700 22054 consensus_queue.cc:260] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:34.390463 22054 raft_consensus.cc:3060] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:34.391548 17838 raft_consensus.cc:2468] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 12.
I20260709 12:22:34.392824 22112 leader_election.cc:290] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:34.392974 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:34.393057 17727 leader_election.cc:304] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:34.393869 17838 raft_consensus.cc:2468] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 12.
I20260709 12:22:34.394306 22025 raft_consensus.cc:2804] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:22:34.394982 22025 raft_consensus.cc:697] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 12 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:22:34.395269 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:34.395869 21984 raft_consensus.cc:493] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:34.396242 17838 raft_consensus.cc:2376] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 11. Re-sending same reply.
I20260709 12:22:34.396369 21984 raft_consensus.cc:515] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:34.396518 17727 leader_election.cc:304] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:34.397177 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:34.397804 22025 consensus_queue.cc:237] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.7, Last appended by leader: 7, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:34.398422 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:34.399295 17838 raft_consensus.cc:3060] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:34.400326 22044 raft_consensus.cc:2804] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:22:34.400933 22044 raft_consensus.cc:493] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:34.401435 22044 raft_consensus.cc:3060] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:34.404330 17727 leader_election.cc:304] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:22:34.405826 21984 leader_election.cc:290] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:34.406418 21984 raft_consensus.cc:2749] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:22:34.357574 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:34.366739 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:34.409658 17838 raft_consensus.cc:2468] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 13.
I20260709 12:22:34.368328 17761 raft_consensus.cc:2468] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 15.
I20260709 12:22:34.410943 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:34.411054 17727 leader_election.cc:304] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:22:34.378492 21965 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.603s	user 0.001s	sys 0.009s
I20260709 12:22:34.379433 21967 consensus_queue.cc:1243] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer 0da169ff82674dc1947055ce43de68a3 log is divergent from this leader: its last log entry 9.9 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 8
W20260709 12:22:34.412021 21965 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.637s	user 0.001s	sys 0.009s
I20260709 12:22:34.348553 17835 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35042 (request call id 2111) took 1922 ms. Trace:
I20260709 12:22:34.413504 21984 raft_consensus.cc:2804] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:22:34.414338 21984 raft_consensus.cc:697] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:22:34.412864 17835 rpcz_store.cc:276] 0709 12:22:32.426630 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:32.426790 (+   160us) service_pool.cc:224] Handling call
0709 12:22:32.427102 (+   312us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:22:32.427471 (+   369us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:32.427500 (+    29us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:22:32.441367 (+ 13867us) log.cc:844] Serialized 52 byte log entry
0709 12:22:32.627941 (+186574us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:22:32.627996 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:32.628089 (+    93us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:34.348233 (+1720144us) raft_consensus.cc:1685] finished
0709 12:22:34.348267 (+    34us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:34.348475 (+   208us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":112,"raft.run_cpu_time_us":127,"raft.run_wall_time_us":15522,"thread_start_us":129947,"threads_started":1,"wal-append.queue_time_us":956405}
I20260709 12:22:34.379923 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:34.415934 17835 raft_consensus.cc:1240] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 9. Current term is 10. Ops: []
I20260709 12:22:34.353554 17837 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b"
dest_uuid: "e063541542a94969898868bf6eb5ce1b"
mode: GRACEFUL
new_leader_uuid: "334a4e1f59944544b3b3ff36616b57da"
 from {username='slave'} at 127.0.0.1:59820
I20260709 12:22:34.355551 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:34.418521 17760 raft_consensus.cc:2393] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 12: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:22:34.417783 17835 raft_consensus.cc:1275] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20260709 12:22:34.376431 22117 raft_consensus.cc:493] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:34.420534 22117 raft_consensus.cc:515] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:34.421200 17835 raft_consensus.cc:1275] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 9 index: 6. (term mismatch)
I20260709 12:22:34.354786 17834 raft_consensus.cc:1240] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 8. Current term is 9. Ops: [6.6-8.7]
W20260709 12:22:34.378451 17801 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:34.421794 17835 pending_rounds.cc:85] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b: Aborting all ops after (but not including) 5
I20260709 12:22:34.424244 17835 pending_rounds.cc:107] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b: Aborting uncommitted NO_OP operation due to leader change: 8.6
I20260709 12:22:34.424573 17835 raft_consensus.cc:2889] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:34.425541 22116 raft_consensus.cc:493] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:34.426050 22116 raft_consensus.cc:515] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:34.426518 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:34.426764 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:34.427112 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:34.427498 17759 raft_consensus.cc:2393] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 9: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:22:34.428153 21967 raft_consensus.cc:493] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:34.428452 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 11 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:34.428663 21967 raft_consensus.cc:515] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:34.430032 21984 consensus_queue.cc:237] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.9, Last appended by leader: 9, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:34.430895 21967 leader_election.cc:290] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:34.431450 21967 raft_consensus.cc:493] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:34.429136 17757 raft_consensus.cc:2393] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 11: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:22:34.432605 22116 leader_election.cc:290] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:34.433041 21967 raft_consensus.cc:515] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:34.433486 17838 raft_consensus.cc:2468] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 10.
I20260709 12:22:34.415753 17832 raft_consensus.cc:3060] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:34.435500 21967 leader_election.cc:290] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:34.409983 18363 raft_consensus.cc:2410] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for term 12 because replica has last-logged OpId of term: 11 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 9.
I20260709 12:22:34.436510 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:34.437129 17801 leader_election.cc:304] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:34.441584 17760 raft_consensus.cc:2468] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 9.
I20260709 12:22:34.375415 21995 raft_consensus.cc:493] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:34.423174 22117 leader_election.cc:290] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:34.446007 21995 raft_consensus.cc:515] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:34.446797 17801 leader_election.cc:304] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:34.454183 21995 leader_election.cc:290] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:34.454840 22117 raft_consensus.cc:2749] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 12:22:34.444196 22112 raft_consensus.cc:2749] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:22:34.456316 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 12 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:34.457671 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:34.458333 17838 raft_consensus.cc:2368] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 10. Current term is 11.
I20260709 12:22:34.458588 17834 raft_consensus.cc:1240] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 8. Current term is 9. Ops: [6.8-8.10]
I20260709 12:22:34.458493 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:34.459453 17757 raft_consensus.cc:2468] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 11.
I20260709 12:22:34.460057 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:34.461305 17834 raft_consensus.cc:2468] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 12.
I20260709 12:22:34.462899 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 11 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:34.463729 17834 raft_consensus.cc:2468] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 10.
I20260709 12:22:34.465145 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:34.465246 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:34.466312 17834 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 8.
I20260709 12:22:34.467677 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:34.468627 17834 raft_consensus.cc:2393] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 8: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:22:34.470206 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:34.471052 17832 raft_consensus.cc:2468] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 11.
I20260709 12:22:34.471513 17838 raft_consensus.cc:1275] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 8 index: 10. Preceding OpId from leader: term: 9 index: 11. (index mismatch)
I20260709 12:22:34.479835 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:34.481133 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:34.482949 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:34.483294 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:34.484378 17838 raft_consensus.cc:2393] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 11: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:22:34.484648 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:22:34.350992 17803 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:34.486805 21967 raft_consensus.cc:493] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:34.487206 22117 raft_consensus.cc:493] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:34.487361 21967 raft_consensus.cc:515] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:34.488117 22117 raft_consensus.cc:515] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:34.490339 17801 leader_election.cc:304] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:34.490711 21967 leader_election.cc:290] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:34.490875 22117 leader_election.cc:290] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:34.492990 17756 raft_consensus.cc:2468] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 10.
I20260709 12:22:34.352748 22114 raft_consensus.cc:493] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:34.496711 22114 raft_consensus.cc:515] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:34.503079 22045 consensus_queue.cc:1059] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 9.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:22:34.504323 22045 consensus_queue.cc:1048] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 12:22:34.505120 22112 raft_consensus.cc:2804] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:22:34.505484 22117 raft_consensus.cc:493] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:34.506542 21995 raft_consensus.cc:493] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:34.506444 17801 leader_election.cc:304] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:34.380791 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:34.510918 17758 raft_consensus.cc:2468] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 10.
I20260709 12:22:34.349440 17729 leader_election.cc:304] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:34.511368 17837 raft_consensus.cc:606] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Received request to transfer leadership to TS 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:34.517499 17837 raft_consensus.cc:614] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:22:34.512594 21974 consensus_queue.cc:1048] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:22:34.518193 17834 raft_consensus.cc:1240] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 10. Current term is 11. Ops: [10.8-10.8]
W20260709 12:22:34.513140 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 87333.8us
I20260709 12:22:34.513788 22122 raft_consensus.cc:3055] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:22:34.517130 17829 raft_consensus.cc:3060] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:34.516842 17801 leader_election.cc:304] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:22:34.517357 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 167997us
I20260709 12:22:34.519331 17836 raft_consensus.cc:1240] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 9. Current term is 10. Ops: []
I20260709 12:22:34.522388 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:34.523041 18363 raft_consensus.cc:2468] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 10.
I20260709 12:22:34.523054 22116 raft_consensus.cc:2804] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:22:34.523852 22116 raft_consensus.cc:697] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 10 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:22:34.525231 17758 raft_consensus.cc:1240] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 10. Current term is 11. Ops: []
I20260709 12:22:34.524885 22116 consensus_queue.cc:237] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:34.526288 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:34.526945 17760 raft_consensus.cc:2468] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 10.
I20260709 12:22:34.526836 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:34.527500 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 11 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:34.527539 17758 raft_consensus.cc:2468] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 8.
I20260709 12:22:34.349061 22015 leader_election.cc:290] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:34.528465 17757 raft_consensus.cc:2410] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 11 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 4.
I20260709 12:22:34.520449 17727 leader_election.cc:304] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:34.528831 17756 raft_consensus.cc:1240] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 8. Current term is 9. Ops: []
I20260709 12:22:34.530593 17801 leader_election.cc:304] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:34.514565 22114 leader_election.cc:290] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:34.514680 22112 raft_consensus.cc:493] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:22:34.520076 21967 raft_consensus.cc:2804] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:22:34.346314 21952 raft_consensus.cc:515] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:34.520522 22122 raft_consensus.cc:740] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:22:34.544956 17801 leader_election.cc:304] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:34.520174 22117 raft_consensus.cc:515] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:34.520325 21995 raft_consensus.cc:515] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:34.540511 22112 raft_consensus.cc:3060] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:34.544049 21967 raft_consensus.cc:493] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:34.554873 17803 leader_election.cc:304] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:34.547780 22045 raft_consensus.cc:2804] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:22:34.558753 22129 consensus_queue.cc:1059] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:22:34.559211 22045 raft_consensus.cc:493] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:34.559587 22045 raft_consensus.cc:3060] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:34.559716 21995 leader_election.cc:290] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:34.560736 21995 raft_consensus.cc:2749] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260709 12:22:34.563345 22059 log.cc:927] Time spent T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 1.718s	user 0.003s	sys 0.000s
I20260709 12:22:34.556006 17835 raft_consensus.cc:3060] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:34.565534 22117 leader_election.cc:290] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:34.566176 22054 raft_consensus.cc:2804] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:22:34.566561 22054 raft_consensus.cc:493] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:34.566870 22054 raft_consensus.cc:3060] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:34.556802 22128 raft_consensus.cc:493] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:34.567655 22128 raft_consensus.cc:515] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:34.571424 17835 raft_consensus.cc:2468] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 12.
I20260709 12:22:34.558705 21967 raft_consensus.cc:3060] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:34.548671 22114 consensus_queue.cc:1059] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 10.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.029s
I20260709 12:22:34.563516 22044 raft_consensus.cc:515] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:34.599560 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:34.607095 22112 raft_consensus.cc:515] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:34.588274 22128 leader_election.cc:290] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:34.610550 22044 leader_election.cc:290] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 13 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:34.611042 17829 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 9.
I20260709 12:22:34.587215 22117 consensus_queue.cc:1059] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 8.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.025s
I20260709 12:22:34.612761 22114 raft_consensus.cc:3055] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:22:34.613135 22114 raft_consensus.cc:740] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 10 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
W20260709 12:22:34.613346 18365 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59918 (request call id 1389) took 2389 ms (client timeout 2174 ms). Trace:
W20260709 12:22:34.613713 18365 rpcz_store.cc:269] 0709 12:22:32.223722 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:32.223864 (+   142us) service_pool.cc:224] Handling call
0709 12:22:34.613104 (+2389240us) spinlock_profiling.cc:243] Waited 2.13 s on lock 0x7b54007602d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:22:34.613253 (+   149us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4897684224}
I20260709 12:22:34.614334 21995 raft_consensus.cc:3055] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:22:34.614689 21995 raft_consensus.cc:740] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 8 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:22:34.618024 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 12 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:34.621776 18365 raft_consensus.cc:1275] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 11 index: 11. (index mismatch)
W20260709 12:22:34.625213 17729 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:34.555975 22119 raft_consensus.cc:2804] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:22:34.559744 22129 raft_consensus.cc:2804] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:22:34.648172 22119 raft_consensus.cc:493] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:34.651472 22129 raft_consensus.cc:493] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:22:34.655865 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 101060us
W20260709 12:22:34.655962 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 135598us
I20260709 12:22:34.659673 17727 leader_election.cc:304] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:34.650810 22114 consensus_queue.cc:260] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:34.652084 22119 raft_consensus.cc:3060] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:34.651154 22044 consensus_queue.cc:1048] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.006s
I20260709 12:22:34.657696 22129 raft_consensus.cc:3060] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:34.652455 22054 raft_consensus.cc:515] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:34.658206 17803 leader_election.cc:304] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:34.648732 22122 consensus_queue.cc:260] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:34.674126 22122 raft_consensus.cc:3060] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:34.674172 21952 leader_election.cc:290] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:34.680562 22112 leader_election.cc:290] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:34.667711 22027 raft_consensus.cc:2804] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:22:34.681658 22027 raft_consensus.cc:697] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 11 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:22:34.682278 22045 raft_consensus.cc:515] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:34.682739 22027 consensus_queue.cc:237] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.8, Last appended by leader: 8, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:34.656582 21995 consensus_queue.cc:260] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:22:34.684515 21974 consensus_queue.cc:1175] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "334a4e1f59944544b3b3ff36616b57da" responder_term: 9 status { last_received { term: 9 index: 8 } last_committed_idx: 8 last_received_current_leader { term: 9 index: 8 } }
I20260709 12:22:34.685647 22117 raft_consensus.cc:493] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:34.686390 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:34.686199 22117 raft_consensus.cc:515] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:34.686962 17757 raft_consensus.cc:3060] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:34.684836 21995 raft_consensus.cc:3060] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:34.689448 22054 leader_election.cc:290] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 11 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:34.692358 22045 leader_election.cc:290] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 8 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:34.697211 17685 raft_consensus.cc:1240] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 9. Current term is 10. Ops: []
I20260709 12:22:34.698591 17685 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44898 (request call id 879) took 2452 ms. Trace:
I20260709 12:22:34.698954 17685 rpcz_store.cc:276] 0709 12:22:32.246384 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:32.246869 (+   485us) service_pool.cc:224] Handling call
0709 12:22:32.247205 (+   336us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:34.697728 (+2450523us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:34.698348 (+   620us) spinlock_profiling.cc:243] Waited 2.45 s on lock 0x7b54003458d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:22:34.698532 (+   184us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5634049792}
I20260709 12:22:34.695502 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:34.701545 18368 raft_consensus.cc:3060] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:34.702862 22117 leader_election.cc:290] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:34.703609 22113 raft_consensus.cc:3055] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:22:34.692257 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:34.704392 17756 raft_consensus.cc:2468] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 12.
I20260709 12:22:34.692996 21967 raft_consensus.cc:515] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:34.693858 22114 raft_consensus.cc:3060] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:34.694777 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:34.708609 18363 raft_consensus.cc:3060] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:34.705652 17801 leader_election.cc:304] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:22:34.710489 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 52395.2us
I20260709 12:22:34.718031 17757 raft_consensus.cc:2468] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 10.
I20260709 12:22:34.723506 17801 leader_election.cc:304] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:22:34.750298 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:34.773321 18334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:34.704075 22113 raft_consensus.cc:740] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 8 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
W20260709 12:22:34.793251 17728 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 2.062s	user 0.012s	sys 0.016s
W20260709 12:22:34.793841 17728 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 2.063s	user 0.012s	sys 0.016s
W20260709 12:22:34.794230 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 2.06384e+06us
I20260709 12:22:34.795059 17728 leader_election.cc:304] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:22:34.798113 22113 consensus_queue.cc:260] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.9, Last appended by leader: 9, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:34.784642 22116 raft_consensus.cc:2804] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:22:34.800673 22116 raft_consensus.cc:697] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 10 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
W20260709 12:22:34.803192 17730 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:34.805158 17757 raft_consensus.cc:1275] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 10 index: 8. Preceding OpId from leader: term: 12 index: 9. (index mismatch)
W20260709 12:22:34.816360 17802 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:34.833130 17801 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:34.801682 22116 consensus_queue.cc:237] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:34.839694 18368 raft_consensus.cc:2468] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 8.
W20260709 12:22:34.840215 17728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:34.841102 17729 leader_election.cc:304] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:34.775002 21952 raft_consensus.cc:2764] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Leader pre-election decision vote started in defunct term 11: won
W20260709 12:22:34.844870 17728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:34.859663 18363 raft_consensus.cc:2468] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 11.
I20260709 12:22:34.871811 22129 raft_consensus.cc:515] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:22:34.849113 17728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:34.861030 17730 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:34.848331 22113 raft_consensus.cc:3060] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:34.876492 22119 raft_consensus.cc:515] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:34.832109 21984 raft_consensus.cc:2749] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 12:22:34.848167 22131 consensus_queue.cc:1048] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.039s
I20260709 12:22:34.881649 22054 raft_consensus.cc:2804] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:22:34.882272 22054 raft_consensus.cc:697] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
W20260709 12:22:34.892277 18333 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:34.895398 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:34.895709 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:34.896524 18368 raft_consensus.cc:2368] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 10. Current term is 11.
I20260709 12:22:34.896355 17830 raft_consensus.cc:2468] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 9.
I20260709 12:22:34.897289 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:34.775724 22112 raft_consensus.cc:2749] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260709 12:22:34.899051 17804 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:34.898579 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:34.891269 17729 leader_election.cc:304] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:34.899792 18368 raft_consensus.cc:2468] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 6.
I20260709 12:22:34.883258 22054 consensus_queue.cc:237] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:34.901278 17756 raft_consensus.cc:1240] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 9. Current term is 10. Ops: []
I20260709 12:22:34.901386 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:22:35.071409 17761 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36572 (request call id 767) took 3917 ms (client timeout 1547 ms). Trace:
I20260709 12:22:35.091958 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:22:35.151335 17761 rpcz_store.cc:269] 0709 12:22:31.153694 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:31.181772 (+ 28078us) service_pool.cc:224] Handling call
0709 12:22:31.843376 (+661604us) spinlock_profiling.cc:243] Waited 26.2 ms on lock 0x7b5800033398. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b474fbb71 00007f7b474fba22 00007f7b504148d9 00007f7b4f56eee0 00007f7b4f6380ab 00007f7b4f638131 00007f7b4f5e60a0 00007f7b4f5e5984 00007f7b48b606d4 00007f7b48b60662
0709 12:22:35.071169 (+3227793us) spinlock_profiling.cc:243] Waited 2.47 s on lock 0x7b54003f2854. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:22:35.071331 (+   162us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5741330176}
W20260709 12:22:35.069743 17859 debug-util.cc:398] Leaking SignalData structure 0x7b0800345780 after lost signal to thread 16501
I20260709 12:22:34.916365 17760 raft_consensus.cc:1275] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 8 index: 7. Preceding OpId from leader: term: 9 index: 7. (term mismatch)
W20260709 12:22:35.152267 17859 debug-util.cc:398] Leaking SignalData structure 0x7b08003702a0 after lost signal to thread 16504
I20260709 12:22:35.150180 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:35.152426 17760 pending_rounds.cc:85] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb: Aborting all ops after (but not including) 6
I20260709 12:22:35.153067 17760 pending_rounds.cc:107] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb: Aborting uncommitted NO_OP operation due to leader change: 8.7
I20260709 12:22:35.153416 17832 raft_consensus.cc:1240] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 10 LEADER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 9. Current term is 10. Ops: []
W20260709 12:22:35.152869 17727 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:34.906975 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:35.154579 18368 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 8.
I20260709 12:22:35.152845 17838 raft_consensus.cc:3060] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:35.155414 22015 raft_consensus.cc:493] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:35.153874 17831 raft_consensus.cc:2393] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 9: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:22:35.153678 17760 raft_consensus.cc:2889] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:34.996196 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:35.157545 18364 raft_consensus.cc:2468] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 12.
I20260709 12:22:35.158159 22015 raft_consensus.cc:515] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:35.162444 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 11 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:35.166309 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:34.916682 22131 raft_consensus.cc:493] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:35.144855 22114 raft_consensus.cc:493] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:34.920807 21984 raft_consensus.cc:2804] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:22:35.172789 17831 raft_consensus.cc:1275] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260709 12:22:35.174115 17831 raft_consensus.cc:1275] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 11 index: 11. (index mismatch)
I20260709 12:22:35.178642 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:35.188271 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:35.193137 17838 raft_consensus.cc:2468] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 13.
W20260709 12:22:35.193614 18334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:35.198208 17831 raft_consensus.cc:1275] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 9 index: 7. Preceding OpId from leader: term: 13 index: 10. (index mismatch)
W20260709 12:22:35.207588 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:35.226611 17802 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:35.256726 21984 raft_consensus.cc:697] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 11 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:22:35.258184 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:35.259037 18371 raft_consensus.cc:2468] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 9.
I20260709 12:22:35.176762 22114 raft_consensus.cc:515] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:35.188417 22015 leader_election.cc:290] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:35.171698 22131 raft_consensus.cc:515] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:22:35.274082 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 112802us
I20260709 12:22:35.282150 17727 leader_election.cc:304] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:35.282832 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:35.285413 21984 consensus_queue.cc:237] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:35.293619 22131 leader_election.cc:290] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:22:35.296447 22138 log.cc:927] Time spent T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.056s	user 0.000s	sys 0.000s
W20260709 12:22:35.309562 22140 log.cc:927] Time spent T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.154s	user 0.000s	sys 0.001s
W20260709 12:22:35.310382 22122 consensus_meta.cc:287] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.636s	user 0.011s	sys 0.000s
I20260709 12:22:35.314786 22015 raft_consensus.cc:493] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:35.315331 22015 raft_consensus.cc:515] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:35.316179 22134 raft_consensus.cc:493] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:35.316771 22134 raft_consensus.cc:515] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:35.317303 22015 leader_election.cc:290] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:35.324019 22119 leader_election.cc:290] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 11 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:35.326980 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:35.327605 18371 raft_consensus.cc:2468] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 10.
I20260709 12:22:35.281033 22133 consensus_queue.cc:1048] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.045s
I20260709 12:22:35.331923 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 12 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:35.332512 18371 raft_consensus.cc:2410] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 12 because replica has last-logged OpId of term: 9 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 8.
I20260709 12:22:35.308387 22131 raft_consensus.cc:493] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:35.325302 22148 raft_consensus.cc:2804] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:22:35.337591 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:35.337942 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:35.338548 18364 raft_consensus.cc:2376] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 8. Re-sending same reply.
I20260709 12:22:35.339828 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:35.340337 18364 raft_consensus.cc:2376] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 11. Re-sending same reply.
I20260709 12:22:35.341907 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:35.328442 22027 consensus_queue.cc:1048] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 7, Time since last communication: 0.014s
I20260709 12:22:35.342628 18365 raft_consensus.cc:2393] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 12: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:22:35.344874 22131 raft_consensus.cc:515] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:35.350296 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:35.352057 22055 raft_consensus.cc:3060] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:35.352370 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 9 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:35.356506 22131 leader_election.cc:290] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:35.357308 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:35.357987 18371 raft_consensus.cc:2376] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 12. Re-sending same reply.
W20260709 12:22:35.364500 17859 debug-util.cc:398] Leaking SignalData structure 0x7b08002acee0 after lost signal to thread 17621
W20260709 12:22:35.365000 17859 debug-util.cc:398] Leaking SignalData structure 0x7b080034ea80 after lost signal to thread 17635
W20260709 12:22:35.366238 17859 debug-util.cc:398] Leaking SignalData structure 0x7b0800173c00 after lost signal to thread 17713
W20260709 12:22:35.367476 17859 debug-util.cc:398] Leaking SignalData structure 0x7b08001d2e20 after lost signal to thread 17788
W20260709 12:22:35.368921 17859 debug-util.cc:398] Leaking SignalData structure 0x7b08000d1c00 after lost signal to thread 17862
I20260709 12:22:35.369524 22134 leader_election.cc:290] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:22:35.370327 17859 debug-util.cc:398] Leaking SignalData structure 0x7b08000a91a0 after lost signal to thread 18409
I20260709 12:22:35.371131 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:35.372249 18368 raft_consensus.cc:2410] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 10 because replica has last-logged OpId of term: 8 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
I20260709 12:22:35.326526 22149 consensus_queue.cc:1048] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.054s
I20260709 12:22:35.386162 17833 raft_consensus.cc:1240] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 11. Current term is 12. Ops: []
I20260709 12:22:35.387365 17833 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35042 (request call id 2114) took 1025 ms. Trace:
I20260709 12:22:35.387478 17835 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343"
dest_uuid: "e063541542a94969898868bf6eb5ce1b"
mode: GRACEFUL
new_leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
 from {username='slave'} at 127.0.0.1:59820
I20260709 12:22:35.387766 17833 rpcz_store.cc:276] 0709 12:22:34.362274 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:34.510902 (+148628us) service_pool.cc:224] Handling call
0709 12:22:35.385963 (+875061us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:35.386613 (+   650us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:35.387183 (+   570us) spinlock_profiling.cc:243] Waited 866 ms on lock 0x7b54000893d0. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a61f8ad 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:22:35.387310 (+   127us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1991305088}
I20260709 12:22:35.388504 17835 raft_consensus.cc:606] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Received request to transfer leadership to TS e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:35.389003 17835 raft_consensus.cc:614] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:22:35.392426 17587 catalog_manager.cc:5697] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 7 to 9. New cstate: current_term: 9 leader_uuid: "334a4e1f59944544b3b3ff36616b57da" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:22:35.393227 21967 leader_election.cc:290] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 12 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:35.394061 17587 catalog_manager.cc:5697] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 1 to 9, leader changed from e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66) to 0da169ff82674dc1947055ce43de68a3 (127.16.29.65). New cstate: current_term: 9 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:22:35.396220 18368 raft_consensus.cc:1240] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 10. Current term is 11. Ops: []
I20260709 12:22:35.397346 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:35.395378 17587 catalog_manager.cc:5697] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 5 to 7. New cstate: current_term: 7 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:22:35.398326 18364 raft_consensus.cc:3060] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:22:35.400503 17804 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:35.401651 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 12 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:35.403174 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:35.403826 18365 raft_consensus.cc:2468] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 8.
I20260709 12:22:35.405642 17835 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d"
dest_uuid: "e063541542a94969898868bf6eb5ce1b"
mode: GRACEFUL
new_leader_uuid: "334a4e1f59944544b3b3ff36616b57da"
 from {username='slave'} at 127.0.0.1:59820
I20260709 12:22:35.406272 17835 raft_consensus.cc:606] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Received request to transfer leadership to TS 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:35.406693 17835 raft_consensus.cc:614] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:22:35.408542 18365 raft_consensus.cc:3055] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:22:35.408885 18365 raft_consensus.cc:740] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:22:35.409406 22114 leader_election.cc:290] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:35.409695 18365 consensus_queue.cc:260] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:35.410305 18366 raft_consensus.cc:1275] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
I20260709 12:22:35.410629 18365 raft_consensus.cc:3060] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:22:35.412029 17728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:35.412616 18367 raft_consensus.cc:1275] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
W20260709 12:22:35.412990 17728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:35.414595 17730 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:35.417661 17759 raft_consensus.cc:1275] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20260709 12:22:35.417698 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:35.418828 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:35.419421 17760 raft_consensus.cc:3060] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:35.418452 17757 raft_consensus.cc:3060] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:22:35.422262 17859 debug-util.cc:398] Leaking SignalData structure 0x7b08003a46c0 after lost signal to thread 21843
I20260709 12:22:35.428439 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:35.429148 17755 raft_consensus.cc:3060] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:35.432451 22129 leader_election.cc:290] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 9 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:35.433851 22129 consensus_queue.cc:1048] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 12:22:35.434289 17830 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c"
dest_uuid: "e063541542a94969898868bf6eb5ce1b"
mode: GRACEFUL
new_leader_uuid: "334a4e1f59944544b3b3ff36616b57da"
 from {username='slave'} at 127.0.0.1:59820
I20260709 12:22:35.434886 17830 raft_consensus.cc:606] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Received request to transfer leadership to TS 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:35.435271 17830 raft_consensus.cc:614] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:22:35.418365 22148 raft_consensus.cc:697] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
W20260709 12:22:35.439481 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:35.439673 17587 catalog_manager.cc:5697] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 5 to 7. New cstate: current_term: 7 leader_uuid: "334a4e1f59944544b3b3ff36616b57da" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:22:35.441352 17587 catalog_manager.cc:5697] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 4 to 6, leader changed from e063541542a94969898868bf6eb5ce1b (127.16.29.67) to 0da169ff82674dc1947055ce43de68a3 (127.16.29.65). New cstate: current_term: 6 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:22:35.442873 17587 catalog_manager.cc:5697] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 7 to 9, leader changed from 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) to e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66). New cstate: current_term: 9 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:22:35.444430 22148 consensus_queue.cc:237] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.6, Last appended by leader: 6, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:35.448050 16501 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260709 12:22:35.450006 16501 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:22:35.451129 16501 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
W20260709 12:22:35.451929 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 1.30698e+06us
I20260709 12:22:35.453243 17762 raft_consensus.cc:2468] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 9.
W20260709 12:22:35.455111 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 1.30285e+06us
I20260709 12:22:35.454958 22045 consensus_queue.cc:1048] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:22:35.455246 22111 consensus_queue.cc:1059] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 7.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 12:22:35.454973 22092 consensus_queue.cc:1048] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:22:35.456131 21991 raft_consensus.cc:2749] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 12:22:35.456938 22049 raft_consensus.cc:493] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:35.457388 22049 raft_consensus.cc:515] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:35.457798 17762 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38776 (request call id 2155) took 1102 ms. Trace:
I20260709 12:22:35.458120 22003 raft_consensus.cc:3055] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:22:35.458295 17762 rpcz_store.cc:276] 0709 12:22:34.355024 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:34.357252 (+  2228us) service_pool.cc:224] Handling call
0709 12:22:35.457643 (+1100391us) spinlock_profiling.cc:243] Waited 1.04 s on lock 0x7b54003e53d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:22:35.457742 (+    99us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2385841024}
I20260709 12:22:35.458652 22003 raft_consensus.cc:740] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:22:35.459313 22049 leader_election.cc:290] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:35.458133 22128 consensus_queue.cc:1048] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:22:35.459602 22111 raft_consensus.cc:3055] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:22:35.460044 22111 raft_consensus.cc:740] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:22:35.460224 21994 raft_consensus.cc:493] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:35.460768 21994 raft_consensus.cc:515] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:35.461251 22003 consensus_queue.cc:260] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.7, Last appended by leader: 7, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:35.462458 22003 raft_consensus.cc:3060] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:35.463205 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:35.463641 17685 raft_consensus.cc:1240] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 8. Current term is 9. Ops: []
I20260709 12:22:35.460790 22111 consensus_queue.cc:260] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 7.9, Last appended by leader: 9, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:35.464594 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:35.464815 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:35.465448 17681 raft_consensus.cc:2468] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 12.
I20260709 12:22:35.465655 22047 raft_consensus.cc:493] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:35.466243 22047 raft_consensus.cc:515] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:35.466502 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:35.467662 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:35.468007 22111 raft_consensus.cc:3060] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:35.468827 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 12 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:35.469607 17760 raft_consensus.cc:2468] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 11.
I20260709 12:22:35.469848 22047 leader_election.cc:290] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:35.470059 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:35.468039 21994 leader_election.cc:290] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:35.470700 17684 raft_consensus.cc:2468] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 9.
I20260709 12:22:35.473600 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:35.474185 17684 raft_consensus.cc:2468] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 7.
I20260709 12:22:35.453329 17652 leader_election.cc:304] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:22:35.475307 18368 raft_consensus.cc:3055] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:22:35.475514 17652 leader_election.cc:304] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:35.475826 18368 raft_consensus.cc:740] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:22:35.476696 18368 consensus_queue.cc:260] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 10.8, Last appended by leader: 8, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:35.477034 22078 raft_consensus.cc:2804] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:22:35.477612 22078 raft_consensus.cc:697] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 8 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:22:35.478416 22078 consensus_queue.cc:237] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:35.478884 22022 raft_consensus.cc:2804] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:22:35.479408 22022 raft_consensus.cc:697] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 9 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:22:35.480332 22022 consensus_queue.cc:237] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.10, Last appended by leader: 10, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:35.488504 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:35.488806 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:35.489003 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 10 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:35.489383 18362 raft_consensus.cc:2468] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 8.
I20260709 12:22:35.489406 18366 raft_consensus.cc:2393] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 10: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:22:35.490520 18333 leader_election.cc:400] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 10 pre-election: Vote denied by peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837) with higher term. Message: Invalid argument: T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 10. Current term is 11.
I20260709 12:22:35.491012 17757 raft_consensus.cc:2468] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 9.
I20260709 12:22:35.492012 18333 leader_election.cc:304] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:22:35.492034 18368 raft_consensus.cc:3060] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:35.495247 17755 raft_consensus.cc:2468] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 12.
I20260709 12:22:35.498838 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:35.531898 17688 raft_consensus.cc:2393] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 10: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:22:35.499543 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 11 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:35.500507 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:35.501549 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:35.533506 17689 raft_consensus.cc:2468] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 10.
I20260709 12:22:35.503074 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:35.503952 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:35.507066 18365 raft_consensus.cc:1275] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
W20260709 12:22:35.452059 17730 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:35.509903 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:35.511159 17834 raft_consensus.cc:1240] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 11. Current term is 12. Ops: [11.8-11.8]
I20260709 12:22:35.511940 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:35.535656 17837 raft_consensus.cc:2468] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 9.
I20260709 12:22:35.489585 17830 raft_consensus.cc:3060] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:35.509236 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:35.512746 17836 raft_consensus.cc:1240] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 12. Current term is 13. Ops: [11.8-12.9]
I20260709 12:22:35.513641 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 13 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:35.514257 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:35.519508 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:35.540344 17829 raft_consensus.cc:2468] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 9.
I20260709 12:22:35.540946 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:35.542716 18362 raft_consensus.cc:1275] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260709 12:22:35.520503 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 10 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:35.546193 17762 raft_consensus.cc:1275] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260709 12:22:35.547770 17801 leader_election.cc:304] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:35.548888 17801 leader_election.cc:304] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:35.549909 17801 leader_election.cc:304] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:35.551793 17757 raft_consensus.cc:1275] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20260709 12:22:35.553215 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:35.502681 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:35.529280 18333 leader_election.cc:304] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:22:35.543557 22015 raft_consensus.cc:2804] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:22:35.554973 17688 raft_consensus.cc:1275] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 10 index: 8. Preceding OpId from leader: term: 12 index: 9. (index mismatch)
I20260709 12:22:35.675997 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:35.676757 17688 raft_consensus.cc:2468] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 12.
I20260709 12:22:35.677755 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:35.678946 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:35.556236 17761 raft_consensus.cc:1240] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 11 LEADER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 10. Current term is 11. Ops: []
I20260709 12:22:35.556233 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 12 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:35.563586 18374 raft_consensus.cc:1240] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 9. Current term is 10. Ops: []
I20260709 12:22:35.569526 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:35.680650 17689 raft_consensus.cc:2468] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 11.
I20260709 12:22:35.565500 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:22:35.681762 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 156268us
I20260709 12:22:35.681991 17762 raft_consensus.cc:2468] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 9.
I20260709 12:22:35.587317 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:35.683152 18333 leader_election.cc:304] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:22:35.683385 18361 raft_consensus.cc:2468] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 9.
W20260709 12:22:35.626870 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:35.665562 17652 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:35.570233 17683 raft_consensus.cc:1240] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 11. Current term is 12. Ops: []
I20260709 12:22:35.565279 17685 raft_consensus.cc:1240] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 11. Current term is 12. Ops: []
I20260709 12:22:35.673619 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:35.687975 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:35.697489 18368 raft_consensus.cc:1275] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 10 index: 8. Preceding OpId from leader: term: 11 index: 9. (index mismatch)
I20260709 12:22:35.699153 17684 raft_consensus.cc:2410] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 12 because replica has last-logged OpId of term: 10 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 8.
I20260709 12:22:35.699956 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:35.700951 17683 raft_consensus.cc:2468] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 9.
I20260709 12:22:35.748656 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:35.745290 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:35.747627 17760 raft_consensus.cc:1240] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 10. Current term is 11. Ops: []
I20260709 12:22:35.741709 17587 catalog_manager.cc:5697] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 6 to 7, leader changed from 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) to 334a4e1f59944544b3b3ff36616b57da (127.16.29.68). New cstate: current_term: 7 leader_uuid: "334a4e1f59944544b3b3ff36616b57da" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:22:35.750759 18364 raft_consensus.cc:2468] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 11.
I20260709 12:22:35.747387 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 10 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:35.748013 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:35.752651 17687 raft_consensus.cc:3060] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:35.718797 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 12 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:35.703085 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:22:35.702873 17652 connection.cc:256] RPC call timeout handler was delayed by -1.5847450080000272s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1582s for a response
W20260709 12:22:35.755661 17652 connection.cc:256] RPC call timeout handler was delayed by -1.3489450080001006s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1844s for a response
W20260709 12:22:35.756176 17652 connection.cc:256] RPC call timeout handler was delayed by -1.2868450080000002s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19130000000000003s for a response
W20260709 12:22:35.756618 17652 connection.cc:256] RPC call timeout handler was delayed by -1.0519450080000752s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2174s for a response
I20260709 12:22:35.757139 17830 raft_consensus.cc:2468] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 10.
W20260709 12:22:35.757102 17652 connection.cc:256] RPC call timeout handler was delayed by -1.0006450079999922s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2231s for a response
I20260709 12:22:35.758986 17680 raft_consensus.cc:2393] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 12: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:22:35.759071 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:35.543063 22049 raft_consensus.cc:493] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:35.770488 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 13 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:35.771333 17688 raft_consensus.cc:1240] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 9. Current term is 10. Ops: []
I20260709 12:22:35.772606 17684 raft_consensus.cc:1240] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 11. Current term is 12. Ops: []
I20260709 12:22:35.753403 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:35.564656 22114 raft_consensus.cc:2804] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:22:35.780675 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:35.781502 17682 raft_consensus.cc:2468] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 12.
I20260709 12:22:35.587845 22129 raft_consensus.cc:2804] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:22:35.587538 21967 raft_consensus.cc:2804] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:22:35.746995 21888 raft_consensus.cc:493] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:35.805452 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:35.806268 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:35.702056 22015 raft_consensus.cc:493] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:35.806953 17680 raft_consensus.cc:2468] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 9.
I20260709 12:22:35.812212 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:35.807191 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:35.810181 17802 leader_election.cc:304] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:35.815976 21888 raft_consensus.cc:515] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:35.817533 17689 raft_consensus.cc:1275] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 11 index: 9. (term mismatch)
I20260709 12:22:35.817855 17689 pending_rounds.cc:85] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3: Aborting all ops after (but not including) 8
I20260709 12:22:35.818073 17689 pending_rounds.cc:107] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3: Aborting uncommitted NO_OP operation due to leader change: 10.9
I20260709 12:22:35.818293 17689 raft_consensus.cc:2889] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:35.818518 21888 leader_election.cc:290] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:35.824699 22003 consensus_queue.cc:1048] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 12:22:35.770129 22109 raft_consensus.cc:493] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:35.835338 17802 leader_election.cc:304] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:35.835500 17687 raft_consensus.cc:2468] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 11.
I20260709 12:22:35.837525 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:22:35.848652 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:35.779244 22128 raft_consensus.cc:493] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:35.866376 17687 raft_consensus.cc:1275] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 11 index: 9. Preceding OpId from leader: term: 12 index: 10. (index mismatch)
I20260709 12:22:35.868368 17802 leader_election.cc:304] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:35.881814 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:22:35.884052 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 376899us
W20260709 12:22:35.886967 18334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:35.888499 17802 leader_election.cc:304] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:22:35.807687 22114 raft_consensus.cc:697] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 11 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
W20260709 12:22:35.894678 18334 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.16.29.65:33071 timed out after 1.709s (SENT) [suppressed 46 similar messages]
I20260709 12:22:35.894773 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:35.835858 22015 raft_consensus.cc:3060] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:22:35.891283 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:35.834375 21967 raft_consensus.cc:697] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 12 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:22:35.859912 22002 raft_consensus.cc:2804] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:22:35.779376 22049 raft_consensus.cc:515] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:35.883807 22119 raft_consensus.cc:2749] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260709 12:22:35.897887 17727 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.590s	user 0.015s	sys 0.013s
I20260709 12:22:35.901196 17802 leader_election.cc:304] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:35.901476 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:35.883106 22129 raft_consensus.cc:697] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 9 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:22:35.943934 17682 raft_consensus.cc:3055] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:22:35.930729 17685 raft_consensus.cc:3055] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:22:35.946743 17682 raft_consensus.cc:740] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:22:35.947088 17685 raft_consensus.cc:740] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
W20260709 12:22:35.947225 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 46150us
I20260709 12:22:35.948274 17802 leader_election.cc:304] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:22:35.948693 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:35.949654 17586 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35966 (request call id 354) took 23836 ms (client timeout 15000 ms). Trace:
I20260709 12:22:35.949749 17802 leader_election.cc:304] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:22:35.946326 17727 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.638s	user 0.016s	sys 0.013s
I20260709 12:22:35.946431 17687 raft_consensus.cc:3060] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:22:35.953202 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 645349us
I20260709 12:22:35.948060 17685 consensus_queue.cc:260] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 7, Last appended: 12.9, Last appended by leader: 9, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:35.955681 17685 raft_consensus.cc:3060] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:22:35.959462 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 149413us
I20260709 12:22:35.942054 17683 raft_consensus.cc:1275] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260709 12:22:35.965493 17587 catalog_manager.cc:5697] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 6 to 7, leader changed from e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66) to 334a4e1f59944544b3b3ff36616b57da (127.16.29.68). New cstate: current_term: 7 leader_uuid: "334a4e1f59944544b3b3ff36616b57da" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:22:35.969785 17684 raft_consensus.cc:1275] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20260709 12:22:35.970722 17587 catalog_manager.cc:5697] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 3 to 6, leader changed from 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) to 334a4e1f59944544b3b3ff36616b57da (127.16.29.68). New cstate: current_term: 6 leader_uuid: "334a4e1f59944544b3b3ff36616b57da" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:22:35.973724 17587 catalog_manager.cc:5697] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 6 to 8, leader changed from e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66) to 0da169ff82674dc1947055ce43de68a3 (127.16.29.65). New cstate: current_term: 8 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
W20260709 12:22:35.916775 22141 log.cc:927] Time spent T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.067s	user 0.001s	sys 0.000s
I20260709 12:22:35.978569 18374 raft_consensus.cc:1275] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20260709 12:22:35.991832 17728 leader_election.cc:304] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
W20260709 12:22:35.992224 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 46390.8us
I20260709 12:22:35.889259 22128 raft_consensus.cc:515] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:35.994642 18366 raft_consensus.cc:1240] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 6. Current term is 8. Ops: []
I20260709 12:22:35.909602 21995 raft_consensus.cc:2749] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:22:35.998139 17587 catalog_manager.cc:5697] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 5 to 6, leader changed from 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) to 334a4e1f59944544b3b3ff36616b57da (127.16.29.68). New cstate: current_term: 6 leader_uuid: "334a4e1f59944544b3b3ff36616b57da" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:22:36.001192 17680 raft_consensus.cc:1275] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260709 12:22:35.918135 22002 raft_consensus.cc:493] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:36.005630 17728 leader_election.cc:304] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:36.009930 17652 leader_election.cc:304] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:36.011415 17687 raft_consensus.cc:2468] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 13.
I20260709 12:22:36.002192 17682 consensus_queue.cc:260] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.8, Last appended by leader: 8, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:36.015090 17728 leader_election.cc:304] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:36.015333 17682 raft_consensus.cc:3060] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:35.966306 22130 raft_consensus.cc:2804] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:22:35.917907 22149 consensus_queue.cc:1048] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.090s
I20260709 12:22:35.992116 22150 raft_consensus.cc:493] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:36.027616 17728 leader_election.cc:304] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:36.000712 22133 raft_consensus.cc:493] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:36.030555 17652 leader_election.cc:304] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:36.004820 22054 raft_consensus.cc:493] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:36.036793 17654 leader_election.cc:304] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:22:36.015964 22002 raft_consensus.cc:3060] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:36.038614 17728 leader_election.cc:304] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:35.979594 22003 consensus_queue.cc:1059] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.048s
I20260709 12:22:36.043062 17652 leader_election.cc:304] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:36.023869 22130 raft_consensus.cc:493] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:22:36.050763 18367 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.503s	user 0.029s	sys 0.024s
I20260709 12:22:35.873842 22109 raft_consensus.cc:515] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:36.033073 21995 raft_consensus.cc:2749] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:22:36.025708 22122 raft_consensus.cc:2749] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:22:36.029937 22134 raft_consensus.cc:2804] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:22:36.145358 22134 raft_consensus.cc:493] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:36.145814 22134 raft_consensus.cc:3060] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:36.026659 22055 raft_consensus.cc:2749] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:22:36.126649 22130 raft_consensus.cc:3060] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:36.056556 22129 consensus_queue.cc:237] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:36.086756 22069 raft_consensus.cc:2749] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 12:22:36.151855 21995 raft_consensus.cc:2804] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:22:36.152279 21995 raft_consensus.cc:493] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:36.152659 21995 raft_consensus.cc:3060] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:22:36.153434 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 114897us
I20260709 12:22:36.154467 17728 leader_election.cc:304] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:22:36.155242 22109 leader_election.cc:290] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:22:35.950076 17586 rpcz_store.cc:269] 0709 12:22:11.973437 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:11.987504 (+ 14067us) service_pool.cc:224] Handling call
0709 12:22:35.798589 (+23811085us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:22:25.426621 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:22:25.427443 (+   822us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:22:25.427493 (+    50us) write_op.cc:435] Acquired schema lock
0709 12:22:25.427511 (+    18us) tablet.cc:662] Decoding operations
0709 12:22:25.429184 (+  1673us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:22:25.429298 (+   114us) write_op.cc:641] Partition lock acquired for write op
0709 12:22:25.429324 (+    26us) tablet.cc:685] Acquiring locks for 18 operations
0709 12:22:25.496893 (+ 67569us) tablet.cc:701] Row locks acquired
0709 12:22:25.496955 (+    62us) write_op.cc:260] PREPARE: finished
0709 12:22:25.497166 (+   211us) write_op.cc:270] Start()
0709 12:22:25.497298 (+   132us) write_op.cc:276] Timestamp: P: 1783599745497093 usec, L: 0
0709 12:22:25.497319 (+    21us) op_driver.cc:348] REPLICATION: starting
0709 12:22:25.498145 (+   826us) log.cc:844] Serialized 7088 byte log entry
0709 12:22:26.006783 (+508638us) op_driver.cc:464] REPLICATION: finished
0709 12:22:26.206794 (+200011us) write_op.cc:301] APPLY: starting
0709 12:22:26.207357 (+   563us) tablet.cc:1367] starting BulkCheckPresence
0709 12:22:26.224301 (+ 16944us) tablet.cc:1370] finished BulkCheckPresence
0709 12:22:26.224447 (+   146us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:22:26.332633 (+108186us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:22:26.333092 (+   459us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=18,key_file_lookups=18,delta_file_lookups=18,mrs_lookups=36
0709 12:22:26.333147 (+    55us) write_op.cc:312] APPLY: finished
0709 12:22:26.333734 (+   587us) log.cc:844] Serialized 236 byte log entry
0709 12:22:26.334652 (+   918us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:22:26.334831 (+   179us) write_op.cc:454] Released schema lock
0709 12:22:26.334884 (+    53us) write_op.cc:341] FINISH: Updating metrics
0709 12:22:26.335326 (+   442us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:22:26.335348 (+    22us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":4256300,"reported_tablets":45,"child_traces":[["op",{"apply.queue_time_us":169495,"apply.run_cpu_time_us":24116,"apply.run_wall_time_us":128512,"cfile_cache_hit":34,"cfile_cache_hit_bytes":35564,"cfile_cache_miss":4,"cfile_cache_miss_bytes":6219,"cfile_init":2,"lbm_read_time_us":32366,"lbm_reads_10-100_ms":1,"lbm_reads_lt_1ms":11,"num_ops":18,"prepare.queue_time_us":133194,"prepare.run_cpu_time_us":49498,"prepare.run_wall_time_us":249750,"raft.queue_time_us":135711,"raft.run_cpu_time_us":20614,"raft.run_wall_time_us":174427,"replication_time_us":474947,"thread_start_us":357548,"threads_started":4,"wal-append.queue_time_us":245220,"wal-append.run_cpu_time_us":26785,"wal-append.run_wall_time_us":3509460}]]}
I20260709 12:22:36.155773 22109 raft_consensus.cc:493] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:36.068681 17685 raft_consensus.cc:1275] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 12 index: 9. Preceding OpId from leader: term: 13 index: 10. (index mismatch)
W20260709 12:22:36.156620 22152 log.cc:927] Time spent T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.111s	user 0.000s	sys 0.000s
I20260709 12:22:36.156417 22109 raft_consensus.cc:515] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:22:36.157491 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 114510us
I20260709 12:22:36.158169 17652 leader_election.cc:304] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:36.158459 22109 leader_election.cc:290] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:22:36.089402 17681 consensus_meta.cc:287] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.556s	user 0.011s	sys 0.002s
W20260709 12:22:36.125891 18367 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.578s	user 0.030s	sys 0.024s
I20260709 12:22:36.045476 22111 raft_consensus.cc:2804] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:22:36.159718 22111 raft_consensus.cc:697] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 9 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:22:36.043298 22054 raft_consensus.cc:515] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:36.160128 22079 raft_consensus.cc:2804] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:22:36.160560 22079 raft_consensus.cc:493] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:36.160903 22079 raft_consensus.cc:3060] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:36.160789 22111 consensus_queue.cc:237] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:36.161688 22054 leader_election.cc:290] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:36.162232 22054 raft_consensus.cc:2749] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:22:36.037601 22150 raft_consensus.cc:515] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:36.041142 22133 raft_consensus.cc:515] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:36.042685 22006 consensus_queue.cc:1048] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.106s
I20260709 12:22:36.152691 22143 raft_consensus.cc:2804] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:22:36.022032 21991 consensus_queue.cc:1059] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 8.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.020s
I20260709 12:22:36.159118 17681 raft_consensus.cc:2468] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 8.
I20260709 12:22:36.040202 21768 raft_consensus.cc:2749] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 12:22:36.124428 17682 raft_consensus.cc:1275] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 11 index: 8. Preceding OpId from leader: term: 12 index: 8. (term mismatch)
I20260709 12:22:36.056519 22022 consensus_queue.cc:1048] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.070s
I20260709 12:22:36.067090 21844 raft_consensus.cc:2804] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:22:36.173710 21844 raft_consensus.cc:493] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:36.174265 21844 raft_consensus.cc:3060] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:36.166307 17682 pending_rounds.cc:85] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3: Aborting all ops after (but not including) 7
I20260709 12:22:36.175441 17682 pending_rounds.cc:107] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3: Aborting uncommitted NO_OP operation due to leader change: 11.8
I20260709 12:22:36.175701 17682 raft_consensus.cc:2889] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:22:36.169147 17728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:36.169701 22147 consensus_queue.cc:1048] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260709 12:22:36.170018 22161 log.cc:927] Time spent T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.102s	user 0.000s	sys 0.000s
I20260709 12:22:36.170337 22069 consensus_queue.cc:1048] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
W20260709 12:22:36.195382 18333 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.510s	user 0.011s	sys 0.111s
I20260709 12:22:36.022696 21967 consensus_queue.cc:237] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.6, Last appended by leader: 6, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:36.029022 22114 consensus_queue.cc:237] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:36.172205 18366 raft_consensus.cc:1275] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
W20260709 12:22:36.179730 22162 log.cc:927] Time spent T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.057s	user 0.000s	sys 0.000s
W20260709 12:22:36.223563 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:36.230360 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:36.168088 22055 raft_consensus.cc:2749] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260709 12:22:36.196275 18333 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.511s	user 0.011s	sys 0.111s
I20260709 12:22:36.194394 21768 raft_consensus.cc:2749] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:22:36.151695 22015 raft_consensus.cc:515] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:36.165354 22143 raft_consensus.cc:493] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
W20260709 12:22:36.240087 22158 log.cc:927] Time spent T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.096s	user 0.000s	sys 0.001s
I20260709 12:22:36.203843 22054 raft_consensus.cc:493] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:36.182410 22165 raft_consensus.cc:2804] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:22:36.277348 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:36.168036 22149 consensus_queue.cc:1048] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.096s
I20260709 12:22:36.182193 22078 consensus_queue.cc:1048] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.081s
I20260709 12:22:36.164731 22155 consensus_queue.cc:1059] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 11.6, Next index: 7, Last known committed idx: 4, Time since last communication: 0.125s
I20260709 12:22:36.289928 22054 raft_consensus.cc:515] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:36.296120 18364 raft_consensus.cc:1240] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 12. Current term is 13. Ops: []
W20260709 12:22:36.251897 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 568754us
I20260709 12:22:36.279297 22143 raft_consensus.cc:3060] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:36.286590 22165 raft_consensus.cc:493] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:36.299459 22049 leader_election.cc:290] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:36.411732 22165 raft_consensus.cc:3060] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:22:36.410565 22158 log.cc:927] Time spent T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.114s	user 0.000s	sys 0.001s
W20260709 12:22:36.421092 17801 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.524s	user 0.000s	sys 0.036s
W20260709 12:22:36.422488 17801 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.525s	user 0.000s	sys 0.037s
W20260709 12:22:36.426443 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 476764us
W20260709 12:22:36.424250 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 547132us
W20260709 12:22:36.444573 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:36.398245 22017 consensus_queue.cc:1048] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.023s
I20260709 12:22:36.415493 22128 leader_election.cc:290] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:22:36.469815 22163 log.cc:927] Time spent T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.206s	user 0.000s	sys 0.001s
W20260709 12:22:36.474768 22159 log.cc:927] Time spent T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.202s	user 0.000s	sys 0.001s
W20260709 12:22:36.476861 22168 log.cc:927] Time spent T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.064s	user 0.000s	sys 0.000s
I20260709 12:22:36.476873 17756 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38786 (request call id 2862) took 1040 ms. Trace:
I20260709 12:22:36.477500 17756 rpcz_store.cc:276] 0709 12:22:35.436183 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:35.436317 (+   134us) service_pool.cc:224] Handling call
0709 12:22:35.436655 (+   338us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:35.437057 (+   402us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:35.437090 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:35.437611 (+   521us) log.cc:844] Serialized 26 byte log entry
0709 12:22:35.863844 (+426233us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:22:35.863933 (+    89us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:35.864031 (+    98us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:36.476636 (+612605us) raft_consensus.cc:1685] finished
0709 12:22:36.476658 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:36.476805 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":100,"raft.run_cpu_time_us":86,"raft.run_wall_time_us":117,"thread_start_us":425855,"threads_started":1,"wal-append.queue_time_us":743182}
I20260709 12:22:36.480223 18362 raft_consensus.cc:1240] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 11. Current term is 12. Ops: []
W20260709 12:22:36.459862 22067 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.517s	user 0.012s	sys 0.033s
I20260709 12:22:36.459359 22149 raft_consensus.cc:3055] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 LEADER]: Stepping down as leader of term 11
W20260709 12:22:36.503144 18363 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.592s	user 0.019s	sys 0.023s
W20260709 12:22:36.505254 18363 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.595s	user 0.019s	sys 0.023s
I20260709 12:22:36.488547 22128 raft_consensus.cc:493] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:36.506070 18363 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59886 (request call id 1277) took 1097 ms. Trace:
I20260709 12:22:36.512115 18363 rpcz_store.cc:276] 0709 12:22:35.408879 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:35.414103 (+  5224us) service_pool.cc:224] Handling call
0709 12:22:35.414474 (+   371us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:35.414736 (+   262us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 12:22:35.909599 (+494863us) log.cc:844] Serialized 14 byte log entry
0709 12:22:36.505610 (+596011us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:36.505682 (+    72us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:22:36.505752 (+    70us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:36.505852 (+   100us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:36.506002 (+   150us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":825787,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":42,"thread_start_us":1089620,"threads_started":2}
W20260709 12:22:36.527630 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 490894us
I20260709 12:22:36.504422 22150 leader_election.cc:290] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:36.505544 22149 raft_consensus.cc:740] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:22:36.502369 22167 raft_consensus.cc:2749] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260709 12:22:36.548202 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:36.502856 22023 raft_consensus.cc:3055] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 8 LEADER]: Stepping down as leader of term 8
W20260709 12:22:36.503383 22067 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.565s	user 0.012s	sys 0.034s
W20260709 12:22:36.552889 18334 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.654s	user 0.006s	sys 0.009s
I20260709 12:22:36.553340 18363 raft_consensus.cc:1275] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 8 index: 10. Preceding OpId from leader: term: 9 index: 11. (index mismatch)
W20260709 12:22:36.555285 17727 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.563s	user 0.011s	sys 0.003s
W20260709 12:22:36.600775 17727 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.608s	user 0.012s	sys 0.004s
W20260709 12:22:36.601243 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 647072us
W20260709 12:22:36.558445 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 400326us
I20260709 12:22:36.569154 18365 raft_consensus.cc:1275] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260709 12:22:36.529978 22133 leader_election.cc:290] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:36.520401 22128 raft_consensus.cc:515] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:36.611856 18365 raft_consensus.cc:3060] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [term 9 LEARNER]: Advancing to term 10
I20260709 12:22:36.557874 22023 raft_consensus.cc:740] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:22:36.568995 18367 raft_consensus.cc:1275] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
W20260709 12:22:36.563427 18334 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.665s	user 0.006s	sys 0.010s
W20260709 12:22:36.615187 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 720481us
I20260709 12:22:36.600318 22122 consensus_queue.cc:1048] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.021s
W20260709 12:22:36.619060 17654 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:36.620018 17654 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:36.622931 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:36.623767 17681 raft_consensus.cc:2468] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 11.
W20260709 12:22:36.621346 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 414421us
I20260709 12:22:36.626317 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 12 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:36.627540 17681 raft_consensus.cc:2468] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 11.
I20260709 12:22:36.644161 17682 raft_consensus.cc:1275] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 11 index: 6. Preceding OpId from leader: term: 13 index: 7. (index mismatch)
I20260709 12:22:36.550256 22049 consensus_queue.cc:1059] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 9.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.051s
W20260709 12:22:36.647228 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 44376.2us
I20260709 12:22:36.626749 17728 leader_election.cc:304] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:36.648794 18367 raft_consensus.cc:1275] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 14 index: 13. Preceding OpId from leader: term: 15 index: 14. (index mismatch)
I20260709 12:22:36.650558 17757 raft_consensus.cc:1275] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 14 index: 13. Preceding OpId from leader: term: 15 index: 14. (index mismatch)
I20260709 12:22:36.651247 17689 raft_consensus.cc:3055] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:22:36.654929 17689 raft_consensus.cc:740] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:22:36.656059 17728 leader_election.cc:304] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:22:36.656544 18368 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.656s	user 0.042s	sys 0.077s
I20260709 12:22:36.652551 18365 raft_consensus.cc:1275] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [term 10 LEARNER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260709 12:22:36.660377 17761 raft_consensus.cc:1275] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
W20260709 12:22:36.605572 22119 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.612s	user 0.000s	sys 0.016s
I20260709 12:22:36.600433 22129 raft_consensus.cc:493] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
W20260709 12:22:36.656872 18368 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.656s	user 0.042s	sys 0.077s
I20260709 12:22:36.655771 17689 consensus_queue.cc:260] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:36.667115 17689 raft_consensus.cc:3060] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Advancing to term 12
W20260709 12:22:36.673588 22172 log.cc:927] Time spent T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.075s	user 0.000s	sys 0.000s
I20260709 12:22:36.681581 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 10 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:36.684638 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:36.688931 17764 raft_consensus.cc:1240] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 11. Current term is 12. Ops: []
I20260709 12:22:36.690465 22054 leader_election.cc:290] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:36.695079 22131 raft_consensus.cc:493] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:36.695802 22131 raft_consensus.cc:515] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:36.700017 22149 consensus_queue.cc:260] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.6, Last appended by leader: 6, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:36.658246 22079 raft_consensus.cc:515] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:36.714413 22176 consensus_queue.cc:1048] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.050s
I20260709 12:22:36.716248 22176 raft_consensus.cc:493] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:36.716770 22176 raft_consensus.cc:515] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:36.719321 22176 leader_election.cc:290] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:22:36.728636 22174 log.cc:927] Time spent T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.117s	user 0.000s	sys 0.000s
I20260709 12:22:36.745280 22147 raft_consensus.cc:2804] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:22:36.745833 22147 raft_consensus.cc:493] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:36.746259 22147 raft_consensus.cc:3060] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:36.756057 22133 raft_consensus.cc:2804] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:22:36.756646 22133 raft_consensus.cc:493] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:36.757084 22133 raft_consensus.cc:3060] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:36.761035 22045 consensus_queue.cc:1048] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:22:36.764130 22129 raft_consensus.cc:515] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:36.767081 22129 leader_election.cc:290] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:36.575614 22150 consensus_queue.cc:1048] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 7, Time since last communication: 0.080s
I20260709 12:22:36.599193 22170 raft_consensus.cc:493] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:22:36.777563 18362 raft_consensus.cc:1275] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
W20260709 12:22:36.795598 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:36.778321 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 473063us
I20260709 12:22:36.763257 22045 consensus_queue.cc:1048] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:22:36.573424 21984 consensus_queue.cc:1048] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.081s
I20260709 12:22:36.651829 21888 raft_consensus.cc:493] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:36.810446 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:36.667999 22119 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.690s	user 0.000s	sys 0.017s
I20260709 12:22:36.815788 17652 leader_election.cc:304] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:36.815984 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:22:36.785480 22114 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.534s	user 0.011s	sys 0.030s
I20260709 12:22:36.659309 22015 leader_election.cc:290] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 13 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:36.802268 22149 raft_consensus.cc:3060] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:36.743230 22143 raft_consensus.cc:515] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:36.859187 17652 leader_election.cc:304] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:22:36.859316 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 66996.6us
I20260709 12:22:36.862185 17689 raft_consensus.cc:1275] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
W20260709 12:22:36.862215 21995 consensus_meta.cc:287] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.709s	user 0.009s	sys 0.002s
W20260709 12:22:36.897543 17730 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:36.815121 22042 raft_consensus.cc:3055] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:22:36.659454 21768 consensus_queue.cc:1059] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.176s
W20260709 12:22:36.869616 22130 consensus_meta.cc:287] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.721s	user 0.013s	sys 0.000s
W20260709 12:22:36.908876 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 49755.2us
I20260709 12:22:36.914748 17652 leader_election.cc:304] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
W20260709 12:22:36.854656 22114 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.603s	user 0.011s	sys 0.031s
I20260709 12:22:36.915261 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:22:36.911446 18374 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.731s	user 0.003s	sys 0.063s
W20260709 12:22:36.972035 18374 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.792s	user 0.004s	sys 0.063s
W20260709 12:22:36.973632 17587 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.811s	user 0.018s	sys 0.051s
I20260709 12:22:36.884728 22119 consensus_queue.cc:1059] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.061s
I20260709 12:22:36.873796 22049 raft_consensus.cc:2749] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 12:22:36.896008 21995 raft_consensus.cc:515] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:36.812139 22067 consensus_queue.cc:1048] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.096s
I20260709 12:22:36.830242 21888 raft_consensus.cc:515] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:36.915553 22109 raft_consensus.cc:3055] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 6 LEADER]: Stepping down as leader of term 6
W20260709 12:22:36.919181 22002 consensus_meta.cc:287] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.858s	user 0.007s	sys 0.008s
I20260709 12:22:36.927353 22181 raft_consensus.cc:493] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:22:36.930467 21984 raft_consensus.cc:493] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:36.933007 21768 consensus_queue.cc:1059] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 12.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 12:22:36.933838 22171 raft_consensus.cc:493] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:36.905100 22042 raft_consensus.cc:740] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:22:36.933689 22180 consensus_queue.cc:1048] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
W20260709 12:22:36.968329 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 350835us
W20260709 12:22:36.912855 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 436811us
I20260709 12:22:36.922269 22130 raft_consensus.cc:515] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:22:37.013996 21844 consensus_meta.cc:287] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.839s	user 0.010s	sys 0.003s
I20260709 12:22:36.977262 22048 raft_consensus.cc:2749] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:22:36.857687 22173 consensus_queue.cc:1059] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 9.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.134s
W20260709 12:22:36.974259 17587 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.811s	user 0.018s	sys 0.051s
I20260709 12:22:37.037176 22173 raft_consensus.cc:3055] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:22:37.037582 22173 raft_consensus.cc:740] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:22:37.037633 21844 raft_consensus.cc:515] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:36.804167 22170 raft_consensus.cc:515] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:37.038605 22173 consensus_queue.cc:260] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:36.935307 22188 raft_consensus.cc:2749] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:22:36.968535 22114 raft_consensus.cc:493] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:37.040398 22111 raft_consensus.cc:3055] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:22:37.040959 22111 raft_consensus.cc:740] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:22:37.041922 22111 consensus_queue.cc:260] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.10, Last appended by leader: 10, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:37.043401 22111 raft_consensus.cc:3060] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:36.977643 22128 leader_election.cc:290] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:36.945531 22131 leader_election.cc:290] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:36.968781 22003 consensus_queue.cc:1048] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.023s
I20260709 12:22:37.005246 18334 leader_election.cc:304] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:22:37.015971 22130 leader_election.cc:290] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:36.976456 22092 consensus_queue.cc:1048] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:22:36.986380 22109 raft_consensus.cc:740] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:22:36.968706 22047 consensus_queue.cc:1048] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.019s
W20260709 12:22:37.021903 17655 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:37.023630 21888 leader_election.cc:290] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:37.025079 21995 leader_election.cc:290] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 11 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:37.026640 22079 leader_election.cc:290] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 10 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:37.027235 22194 raft_consensus.cc:493] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:37.028641 22067 raft_consensus.cc:493] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
W20260709 12:22:37.013998 22165 consensus_meta.cc:287] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.595s	user 0.006s	sys 0.003s
I20260709 12:22:37.131688 22165 raft_consensus.cc:515] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:37.133015 22147 raft_consensus.cc:515] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:37.134109 22165 leader_election.cc:290] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 11 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:37.040264 22114 raft_consensus.cc:515] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:37.135313 22147 leader_election.cc:290] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 12 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:37.136121 22114 leader_election.cc:290] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:37.136929 18361 raft_consensus.cc:1240] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 10. Current term is 11. Ops: []
W20260709 12:22:36.980883 22134 consensus_meta.cc:287] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.835s	user 0.010s	sys 0.002s
I20260709 12:22:37.137674 22134 raft_consensus.cc:515] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:36.864203 22023 consensus_queue.cc:260] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.7, Last appended by leader: 7, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:37.043397 22170 leader_election.cc:290] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:36.978804 21705 consensus_queue.cc:1048] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.023s
W20260709 12:22:37.053293 22182 log.cc:927] Time spent T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.140s	user 0.000s	sys 0.001s
I20260709 12:22:36.972313 22078 consensus_queue.cc:1048] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.027s
I20260709 12:22:36.986290 22119 consensus_queue.cc:1243] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer e2909cff1da44ad387f96a8ae3aa5afb log is divergent from this leader: its last log entry 11.10 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 9
W20260709 12:22:37.065354 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 407865us
W20260709 12:22:37.068485 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 63323.3us
W20260709 12:22:37.069052 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:37.090255 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:37.101277 18368 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 1400) took 1120 ms. Trace:
W20260709 12:22:37.108539 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:37.116801 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 318380us
I20260709 12:22:37.120178 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:37.001106 22002 raft_consensus.cc:515] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:37.122977 17759 raft_consensus.cc:1275] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20260709 12:22:37.058342 22196 raft_consensus.cc:493] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:37.005465 22171 raft_consensus.cc:515] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:37.188964 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:37.002403 22181 raft_consensus.cc:515] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:37.128324 22194 raft_consensus.cc:515] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:37.093132 22175 raft_consensus.cc:2804] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:22:37.006270 21984 raft_consensus.cc:515] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:37.190240 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 12 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:37.208006 18371 raft_consensus.cc:2468] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 11.
I20260709 12:22:37.191273 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:37.209029 18367 raft_consensus.cc:2468] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 12.
I20260709 12:22:37.193950 18362 raft_consensus.cc:2468] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 9.
I20260709 12:22:37.209667 22190 raft_consensus.cc:493] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:37.210202 22190 raft_consensus.cc:515] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:22:37.210954 22185 raft_consensus.cc:493] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:37.211485 22185 raft_consensus.cc:515] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:37.143030 22067 raft_consensus.cc:515] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:37.203941 22175 raft_consensus.cc:493] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:37.214075 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:37.214495 22175 raft_consensus.cc:3060] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:37.214730 18371 raft_consensus.cc:2468] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 15.
I20260709 12:22:37.215901 22134 leader_election.cc:290] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 8 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:37.217200 22002 leader_election.cc:290] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 13 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:37.186057 18333 leader_election.cc:304] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:22:37.220008 22192 consensus_queue.cc:1059] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:22:37.146957 21844 leader_election.cc:290] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 10 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:22:37.201540 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 53254.1us
I20260709 12:22:37.146698 22114 raft_consensus.cc:493] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:37.226949 22114 raft_consensus.cc:515] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:37.237910 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:22:37.241385 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 55542.5us
I20260709 12:22:37.185422 22122 raft_consensus.cc:493] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:37.252357 18333 leader_election.cc:304] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:22:37.256176 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
W20260709 12:22:37.209153 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 275149us
W20260709 12:22:37.258003 17803 leader_election.cc:336] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 12 election: RPC error from VoteRequest() call to peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Timed out: RequestConsensusVote RPC to 127.16.29.68:42577 timed out after 2.190s (SENT)
I20260709 12:22:37.258544 17803 leader_election.cc:304] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:22:37.259516 17803 leader_election.cc:336] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 election: RPC error from VoteRequest() call to peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Timed out: RequestConsensusVote RPC to 127.16.29.68:42577 timed out after 2.213s (SENT)
I20260709 12:22:37.259931 17803 leader_election.cc:304] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:37.261152 17832 raft_consensus.cc:1275] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 9 index: 9. (term mismatch)
I20260709 12:22:37.261667 17832 pending_rounds.cc:85] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b: Aborting all ops after (but not including) 8
I20260709 12:22:37.261963 17832 pending_rounds.cc:107] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b: Aborting uncommitted NO_OP operation due to leader change: 8.9
I20260709 12:22:37.262284 17832 raft_consensus.cc:2889] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:37.265439 22192 raft_consensus.cc:2804] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:22:37.266942 18333 leader_election.cc:304] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:22:37.270108 17803 leader_election.cc:336] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Timed out: RequestConsensusVote RPC to 127.16.29.68:42577 timed out after 2.217s (SENT)
W20260709 12:22:37.271674 17803 leader_election.cc:336] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 11 election: RPC error from VoteRequest() call to peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Timed out: RequestConsensusVote RPC to 127.16.29.68:42577 timed out after 2.391s (SENT)
I20260709 12:22:37.272100 17803 leader_election.cc:304] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:37.272562 17830 raft_consensus.cc:1275] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 8 index: 4. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260709 12:22:37.275849 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:37.276093 18333 leader_election.cc:304] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:22:37.207872 22177 raft_consensus.cc:493] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:37.289741 17835 raft_consensus.cc:1275] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 9 index: 11. (index mismatch)
I20260709 12:22:37.280920 17830 raft_consensus.cc:1240] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 12 LEADER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 11. Current term is 12. Ops: []
I20260709 12:22:37.282516 17832 raft_consensus.cc:1240] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 9 LEADER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 8. Current term is 9. Ops: []
I20260709 12:22:37.285239 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 10 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:37.285807 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:37.292754 17831 raft_consensus.cc:2468] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 12.
I20260709 12:22:37.292963 17837 raft_consensus.cc:2410] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for term 10 because replica has last-logged OpId of term: 9 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20260709 12:22:37.287038 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:37.293917 17833 raft_consensus.cc:2468] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 11.
I20260709 12:22:37.197685 22196 raft_consensus.cc:515] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:37.251576 22143 leader_election.cc:290] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 10 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:37.296581 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:37.303143 17838 raft_consensus.cc:1275] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260709 12:22:37.305338 17834 raft_consensus.cc:2468] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 11.
I20260709 12:22:37.327358 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 10 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:22:37.329141 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 414471us
I20260709 12:22:37.274755 21995 raft_consensus.cc:2749] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 12:22:37.279672 22023 raft_consensus.cc:3060] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:37.287451 22119 raft_consensus.cc:2749] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 12:22:37.334585 22190 leader_election.cc:290] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:37.261076 22002 consensus_queue.cc:1048] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.071s
W20260709 12:22:37.286571 22133 consensus_meta.cc:287] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.522s	user 0.012s	sys 0.000s
I20260709 12:22:37.140023 22042 consensus_queue.cc:260] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:22:37.381840 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 105836us
I20260709 12:22:37.390787 18333 leader_election.cc:304] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:37.278496 22130 raft_consensus.cc:2749] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:22:37.385196 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:37.278373 22134 raft_consensus.cc:493] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:37.267248 22122 raft_consensus.cc:515] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:37.330174 17652 leader_election.cc:304] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:37.392921 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 12 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:37.397025 18367 raft_consensus.cc:3060] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:37.299350 22207 raft_consensus.cc:2749] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:22:37.391863 17757 raft_consensus.cc:2468] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 9.
W20260709 12:22:37.381273 22214 log.cc:927] Time spent T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.116s	user 0.001s	sys 0.000s
I20260709 12:22:37.297753 22212 raft_consensus.cc:2804] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Leader pre-election won for term 10
W20260709 12:22:37.395087 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:37.363502 17655 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:37.414275 17730 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:37.399194 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 69118.5us
I20260709 12:22:37.408710 17834 raft_consensus.cc:1240] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 8. Current term is 9. Ops: []
I20260709 12:22:37.379334 21705 raft_consensus.cc:493] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:37.433147 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:37.433705 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:37.434409 17837 raft_consensus.cc:3060] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:37.386289 22114 leader_election.cc:290] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:37.192198 18368 rpcz_store.cc:276] 0709 12:22:35.981110 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:35.982506 (+  1396us) service_pool.cc:224] Handling call
0709 12:22:35.982773 (+   267us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:35.983007 (+   234us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 12:22:36.666437 (+683430us) log.cc:844] Serialized 14 byte log entry
0709 12:22:37.070308 (+403871us) log.cc:844] Serialized 14 byte log entry
0709 12:22:37.100578 (+ 30270us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:37.100654 (+    76us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:22:37.100916 (+   262us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:37.101028 (+   112us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:37.101199 (+   171us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":93939,"prepare.queue_time_us":308829,"prepare.run_cpu_time_us":86,"prepare.run_wall_time_us":88,"raft.queue_time_us":58335,"raft.run_cpu_time_us":159,"raft.run_wall_time_us":12117,"spinlock_wait_cycles":38183040,"thread_start_us":1059232,"threads_started":2,"wal-append.queue_time_us":459678}
W20260709 12:22:37.440687 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:37.433792 17838 raft_consensus.cc:2468] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 11.
I20260709 12:22:37.386467 22067 leader_election.cc:290] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:37.397145 21995 raft_consensus.cc:493] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:37.447432 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:37.448051 17756 raft_consensus.cc:3060] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:22:37.450405 17652 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:37.405771 22048 raft_consensus.cc:2804] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:22:37.455749 17763 raft_consensus.cc:1240] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 LEADER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 6. Current term is 8. Ops: []
I20260709 12:22:37.461236 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:37.458894 17756 raft_consensus.cc:2468] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 10.
I20260709 12:22:37.461951 17762 raft_consensus.cc:1240] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 8. Current term is 9. Ops: []
I20260709 12:22:37.460052 18367 raft_consensus.cc:2468] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 12.
I20260709 12:22:37.462010 17833 raft_consensus.cc:2393] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 10: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:22:37.462863 17758 raft_consensus.cc:1275] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260709 12:22:37.335952 22199 consensus_queue.cc:1059] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.158s
I20260709 12:22:37.390172 22133 raft_consensus.cc:515] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:37.440478 22212 raft_consensus.cc:493] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:37.522936 17837 raft_consensus.cc:2468] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 10.
I20260709 12:22:37.445088 22207 raft_consensus.cc:2764] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Leader pre-election decision vote started in defunct term 7: won
W20260709 12:22:37.521363 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 326343us
I20260709 12:22:37.436463 22194 leader_election.cc:290] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:22:37.521701 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 52144.3us
I20260709 12:22:37.445251 22042 raft_consensus.cc:3060] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:37.469952 22192 raft_consensus.cc:697] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 12 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:22:37.301955 22177 raft_consensus.cc:515] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:22:37.539599 17655 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:37.407239 22134 raft_consensus.cc:515] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:37.463923 21995 raft_consensus.cc:515] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:37.471071 22171 leader_election.cc:290] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:37.563927 18334 leader_election.cc:304] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:22:37.447793 21705 raft_consensus.cc:515] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:37.515806 22048 raft_consensus.cc:697] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 10 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:22:37.535691 22212 raft_consensus.cc:3060] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:37.565546 18334 leader_election.cc:304] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:22:37.532210 22047 raft_consensus.cc:493] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:37.650043 22047 raft_consensus.cc:515] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:37.651830 22228 raft_consensus.cc:493] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:37.652338 22047 leader_election.cc:290] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:37.534066 21984 leader_election.cc:290] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:37.559826 17761 raft_consensus.cc:2393] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 12: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:22:37.562234 21994 consensus_queue.cc:1059] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.017s
I20260709 12:22:37.574074 22229 raft_consensus.cc:2749] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:22:37.576373 18374 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59886 (request call id 1292) took 1397 ms. Trace:
I20260709 12:22:37.665156 18374 rpcz_store.cc:276] 0709 12:22:36.179264 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:36.179418 (+   154us) service_pool.cc:224] Handling call
0709 12:22:36.179731 (+   313us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:36.972641 (+792910us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:36.972671 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:36.973179 (+   508us) log.cc:844] Serialized 26 byte log entry
0709 12:22:37.274838 (+301659us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:22:37.274894 (+    56us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:37.274980 (+    86us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:37.576082 (+301102us) raft_consensus.cc:1685] finished
0709 12:22:37.576101 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:37.576314 (+   213us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":611802,"raft.run_cpu_time_us":232,"raft.run_wall_time_us":28269,"thread_start_us":1093735,"threads_started":2,"wal-append.queue_time_us":453342}
I20260709 12:22:37.577584 22196 leader_election.cc:290] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:37.567459 22185 leader_election.cc:290] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:37.554951 22201 raft_consensus.cc:493] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:37.519148 22188 consensus_queue.cc:1048] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.046s
W20260709 12:22:37.572999 22111 consensus_meta.cc:287] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.511s	user 0.003s	sys 0.025s
I20260709 12:22:37.517504 22216 consensus_queue.cc:1048] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.152s
I20260709 12:22:37.568202 22207 raft_consensus.cc:493] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:37.679339 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 113976us
W20260709 12:22:37.639304 22191 log.cc:927] Time spent T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.431s	user 0.001s	sys 0.001s
I20260709 12:22:37.697165 22218 raft_consensus.cc:3055] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 11 LEADER]: Stepping down as leader of term 11
W20260709 12:22:37.696468 22205 log.cc:927] Time spent T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.281s	user 0.001s	sys 0.000s
I20260709 12:22:37.698508 22218 raft_consensus.cc:740] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:22:37.701367 22177 leader_election.cc:290] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:37.702580 22122 leader_election.cc:290] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:37.567111 22200 consensus_queue.cc:1048] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.431s
I20260709 12:22:37.705531 18361 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 1401) took 1724 ms. Trace:
I20260709 12:22:37.706017 18361 rpcz_store.cc:276] 0709 12:22:35.981443 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:35.981900 (+   457us) service_pool.cc:224] Handling call
0709 12:22:36.275967 (+294067us) spinlock_profiling.cc:243] Waited 4.98 ms on lock 0x7b5800102398. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b474fbb71 00007f7b474fba22 00007f7b504148d9 00007f7b4f56eee0 00007f7b4f6380ab 00007f7b4f638131 00007f7b4f5f9fc0 00007f7b4f5e4929 00007f7b4f5e43d2 00007f7b48b5bbd4
0709 12:22:36.276246 (+   279us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:37.137323 (+861077us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:37.705215 (+567892us) spinlock_profiling.cc:243] Waited 860 ms on lock 0x7b54007e7cd4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:22:37.705452 (+   237us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1990302464}
I20260709 12:22:37.687052 22015 raft_consensus.cc:2804] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:22:37.712991 22015 raft_consensus.cc:493] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:37.713452 22015 raft_consensus.cc:3060] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:22:37.728250 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 337526us
I20260709 12:22:37.652621 22228 raft_consensus.cc:515] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:37.670406 22067 raft_consensus.cc:3055] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:22:37.685592 22201 raft_consensus.cc:515] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:37.683902 22196 raft_consensus.cc:2764] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Leader election decision vote started in defunct term 11: won
I20260709 12:22:37.696943 22207 raft_consensus.cc:515] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:37.750705 22228 leader_election.cc:290] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:37.703672 22192 consensus_queue.cc:237] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:37.752147 21995 leader_election.cc:290] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:37.753158 22133 leader_election.cc:290] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 12 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:37.701180 22218 consensus_queue.cc:260] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 11.6, Last appended by leader: 6, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:37.754495 22218 raft_consensus.cc:3060] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:37.754766 22134 leader_election.cc:290] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:37.756604 22207 leader_election.cc:290] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:37.747046 22167 raft_consensus.cc:493] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:37.757452 22167 raft_consensus.cc:515] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:37.681392 22199 consensus_queue.cc:1059] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.064s
W20260709 12:22:37.747848 22175 consensus_meta.cc:287] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.533s	user 0.007s	sys 0.007s
I20260709 12:22:37.740453 22067 raft_consensus.cc:740] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:22:37.759328 22227 consensus_queue.cc:1048] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.024s
I20260709 12:22:37.767907 22175 raft_consensus.cc:515] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:37.786474 22201 leader_election.cc:290] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:22:37.643266 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 72958.7us
I20260709 12:22:37.829478 17685 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37396 (request call id 2241) took 1207 ms. Trace:
I20260709 12:22:37.827546 17652 leader_election.cc:304] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:37.830147 17685 rpcz_store.cc:276] 0709 12:22:36.622292 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:36.623981 (+  1689us) service_pool.cc:224] Handling call
0709 12:22:36.624345 (+   364us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:36.624756 (+   411us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:36.624797 (+    41us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:36.625363 (+   566us) log.cc:844] Serialized 26 byte log entry
0709 12:22:37.140431 (+515068us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:22:37.386760 (+246329us) log.cc:844] Serialized 14 byte log entry
0709 12:22:37.391322 (+  4562us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:37.391396 (+    74us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:37.828667 (+437271us) raft_consensus.cc:1685] finished
0709 12:22:37.828754 (+    87us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:37.829176 (+   422us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4142,"prepare.queue_time_us":413524,"prepare.run_cpu_time_us":130,"prepare.run_wall_time_us":3314,"raft.queue_time_us":11532,"raft.run_cpu_time_us":300,"raft.run_wall_time_us":40328,"thread_start_us":623453,"threads_started":2,"wal-append.queue_time_us":589294}
I20260709 12:22:37.832841 17652 leader_election.cc:304] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:37.835625 17652 leader_election.cc:304] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:37.842998 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:22:37.843021 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 113377us
I20260709 12:22:37.845372 18333 leader_election.cc:304] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:37.848574 18333 leader_election.cc:304] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:37.851308 18371 raft_consensus.cc:1240] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 11. Current term is 12. Ops: []
I20260709 12:22:37.851413 17762 raft_consensus.cc:1275] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 12 index: 7. (index mismatch)
I20260709 12:22:37.862094 22003 consensus_queue.cc:1059] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:22:37.866812 22173 raft_consensus.cc:3060] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:37.868268 21994 consensus_queue.cc:1048] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:22:37.870533 22111 raft_consensus.cc:2804] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:22:37.871326 22111 raft_consensus.cc:493] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
W20260709 12:22:37.860388 22149 consensus_meta.cc:287] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.883s	user 0.013s	sys 0.000s
I20260709 12:22:37.874122 22188 raft_consensus.cc:2749] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:22:37.874548 21888 consensus_queue.cc:1048] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:22:37.875373 17763 raft_consensus.cc:1240] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 11 LEADER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 10. Current term is 11. Ops: [10.10-10.10]
I20260709 12:22:37.876506 17760 raft_consensus.cc:2468] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 12.
I20260709 12:22:37.877527 22184 consensus_queue.cc:1048] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:22:37.883536 22047 consensus_queue.cc:1048] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:22:37.884093 22199 consensus_queue.cc:1059] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.006s
I20260709 12:22:37.867198 22079 raft_consensus.cc:493] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:37.886861 22079 raft_consensus.cc:515] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:37.871812 21994 raft_consensus.cc:2804] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:22:37.887377 22067 consensus_queue.cc:260] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:37.888494 21994 raft_consensus.cc:493] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:37.785646 22167 leader_election.cc:290] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:37.872135 22207 raft_consensus.cc:2749] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:22:37.880515 22111 raft_consensus.cc:3060] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:37.793006 22212 raft_consensus.cc:515] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:37.890079 22067 raft_consensus.cc:3060] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:37.890959 22079 leader_election.cc:290] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:37.891845 21994 raft_consensus.cc:3060] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:37.892486 22212 leader_election.cc:290] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 10 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:37.893967 21705 leader_election.cc:290] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:37.759564 22048 consensus_queue.cc:237] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 8.10, Last appended by leader: 10, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:22:37.897985 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 60054.5us
I20260709 12:22:37.787351 22175 leader_election.cc:290] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 12 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:37.913707 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:37.914634 18374 raft_consensus.cc:2468] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 12.
I20260709 12:22:37.921947 18365 raft_consensus.cc:1275] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260709 12:22:37.922937 17757 raft_consensus.cc:1240] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 LEADER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 11. Current term is 12. Ops: []
I20260709 12:22:37.924201 17763 raft_consensus.cc:1275] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260709 12:22:37.928859 18363 raft_consensus.cc:1240] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 6. Current term is 8. Ops: []
I20260709 12:22:37.931551 17761 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38776 (request call id 2166) took 1254 ms. Trace:
I20260709 12:22:37.932168 17761 rpcz_store.cc:276] 0709 12:22:36.676613 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:36.773958 (+ 97345us) service_pool.cc:224] Handling call
0709 12:22:37.930988 (+1157030us) spinlock_profiling.cc:243] Waited 731 ms on lock 0x7b54002977d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:22:37.931300 (+   312us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1680669952}
W20260709 12:22:37.935643 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 87418.7us
I20260709 12:22:37.942616 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:37.943898 18365 raft_consensus.cc:2393] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 10: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
W20260709 12:22:37.947427 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:37.949682 17801 leader_election.cc:304] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:37.952587 22134 raft_consensus.cc:2764] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Leader pre-election decision vote started in defunct term 9: won
I20260709 12:22:37.941027 17652 leader_election.cc:304] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:22:37.941079 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 12 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:37.939527 22003 consensus_queue.cc:1243] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer e063541542a94969898868bf6eb5ce1b log is divergent from this leader: its last log entry 9.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260709 12:22:37.958863 17763 raft_consensus.cc:2410] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 12 because replica has last-logged OpId of term: 10 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 6.
W20260709 12:22:37.960106 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:37.964658 22045 raft_consensus.cc:493] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:37.965775 22045 raft_consensus.cc:515] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:37.970182 22131 raft_consensus.cc:3055] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:22:37.970789 22131 raft_consensus.cc:740] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 10 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:22:37.970880 17729 leader_election.cc:304] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:22:37.971998 17655 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:37.976505 22119 raft_consensus.cc:493] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:37.977352 22119 raft_consensus.cc:515] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:37.978518 22109 consensus_queue.cc:260] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:37.982336 22109 raft_consensus.cc:3060] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:37.983376 22181 leader_election.cc:290] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:37.988270 21995 consensus_queue.cc:1048] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:22:37.989126 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:22:37.999274 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:38.000694 22239 raft_consensus.cc:2804] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:22:38.001251 22239 raft_consensus.cc:493] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:38.001641 22239 raft_consensus.cc:3060] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:38.002602 21995 consensus_queue.cc:1048] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:22:38.005080 22131 consensus_queue.cc:260] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.10, Last appended by leader: 10, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:22:38.005846 22211 log.cc:927] Time spent T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.793s	user 0.000s	sys 0.005s
I20260709 12:22:38.006637 22131 raft_consensus.cc:3060] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:38.008435 17729 leader_election.cc:304] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:22:38.007979 22134 consensus_queue.cc:1175] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" responder_term: 11 status { last_received { term: 11 index: 10 } last_committed_idx: 9 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 10. Current term is 11. Ops: [10.10-10.10]" } } last_received_current_leader { term: 0 index: 0 } }
I20260709 12:22:38.009804 17729 leader_election.cc:304] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:38.012403 22044 raft_consensus.cc:2804] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:22:38.012872 22044 raft_consensus.cc:493] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:38.013218 22044 raft_consensus.cc:3060] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:22:38.018715 22238 raft_consensus.cc:493] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:38.019606 22238 raft_consensus.cc:515] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:38.009711 22069 raft_consensus.cc:2804] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:22:38.021391 22069 raft_consensus.cc:493] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:38.021876 22069 raft_consensus.cc:3060] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:38.028509 22119 leader_election.cc:290] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:38.030421 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:22:37.999745 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 91124us
I20260709 12:22:37.979750 22201 raft_consensus.cc:2804] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:22:38.046475 17682 raft_consensus.cc:2393] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 12: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
W20260709 12:22:38.046924 22203 log.cc:927] Time spent T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.813s	user 0.004s	sys 0.001s
I20260709 12:22:38.048898 17729 leader_election.cc:304] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:38.050621 22148 raft_consensus.cc:2804] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:22:38.051255 22148 raft_consensus.cc:697] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 12 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:22:38.052194 17687 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37396 (request call id 2243) took 1425 ms. Trace:
I20260709 12:22:38.052578 22148 consensus_queue.cc:237] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.7, Last appended by leader: 7, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:38.054908 22017 consensus_queue.cc:1059] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 11.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
W20260709 12:22:38.056370 17655 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:38.056946 17655 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:37.998297 22227 raft_consensus.cc:2749] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 12:22:38.035127 22238 leader_election.cc:290] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:38.043572 22201 raft_consensus.cc:493] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:38.085410 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:38.086826 17764 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38786 (request call id 2865) took 1022 ms. Trace:
I20260709 12:22:38.092831 17801 leader_election.cc:304] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:38.052618 17687 rpcz_store.cc:276] 0709 12:22:36.626227 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:36.626338 (+   111us) service_pool.cc:224] Handling call
0709 12:22:36.626621 (+   283us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:36.626947 (+   326us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:36.626975 (+    28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:36.627553 (+   578us) log.cc:844] Serialized 26 byte log entry
0709 12:22:37.121250 (+493697us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:22:37.121286 (+    36us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:37.121349 (+    63us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:38.003428 (+882079us) spinlock_profiling.cc:243] Waited 3.78 ms on lock 0x7b4400022b4c. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b47459c85 00007f7b4745a6a2 00007f7b4a1cc1db 00007f7b4a1c7aaf 00007f7b4a1bb86e 00007f7b4a1babb6 00007f7b4a1ba7d8 00007f7b4a1c4221 00007f7b4a1c41d9 00007f7b4a1c4169
0709 12:22:38.051582 (+ 48154us) raft_consensus.cc:1685] finished
0709 12:22:38.051611 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:38.051998 (+   387us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":108058,"raft.queue_time_us":20165,"raft.run_cpu_time_us":307,"raft.run_wall_time_us":7775,"spinlock_wait_cycles":8687488,"thread_start_us":451555,"threads_started":1,"wal-append.queue_time_us":574895}
W20260709 12:22:38.094437 22242 log.cc:927] Time spent T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.059s	user 0.000s	sys 0.000s
I20260709 12:22:38.094679 22047 raft_consensus.cc:493] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:38.096089 22079 raft_consensus.cc:493] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:38.099001 22023 consensus_meta.cc:287] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.714s	user 0.010s	sys 0.005s
I20260709 12:22:38.099197 22015 raft_consensus.cc:515] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:38.087416 22114 raft_consensus.cc:493] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:38.140529 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:38.141463 17764 rpcz_store.cc:276] 0709 12:22:37.064146 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:37.064303 (+   157us) service_pool.cc:224] Handling call
0709 12:22:37.093317 (+ 29014us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:37.239719 (+146402us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:37.239764 (+    45us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:37.239817 (+    53us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:22:37.239860 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:38.086272 (+846412us) spinlock_profiling.cc:243] Waited 146 ms on lock 0x7b54000ff0d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:22:38.086347 (+    75us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:38.086593 (+   246us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":336134912}
I20260709 12:22:38.169207 22175 raft_consensus.cc:493] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:38.169605 22239 raft_consensus.cc:515] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:38.144853 22201 raft_consensus.cc:3060] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:38.204782 22134 raft_consensus.cc:2804] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:22:38.204140 22079 raft_consensus.cc:515] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:22:38.204227 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 111560us
I20260709 12:22:38.205494 22134 raft_consensus.cc:493] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:38.217295 22134 raft_consensus.cc:3060] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:38.220181 22216 raft_consensus.cc:493] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:38.221170 22216 raft_consensus.cc:515] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:38.222018 22196 raft_consensus.cc:3055] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:22:38.222610 22196 raft_consensus.cc:740] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:22:38.224001 22196 consensus_queue.cc:260] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 9.11, Last appended by leader: 11, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:22:38.224838 22235 log.cc:927] Time spent T 00000000000000000000000000000000 P 2193af31cb894ec9b3d65d939a2a6851: Append to log took a long time: real 0.214s	user 0.002s	sys 0.000s
I20260709 12:22:38.220227 22023 raft_consensus.cc:493] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:38.224874 22218 consensus_queue.cc:1059] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 10.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.002s
I20260709 12:22:38.166777 22017 raft_consensus.cc:493] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:38.234650 17689 raft_consensus.cc:2393] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 12: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
W20260709 12:22:38.236372 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 165841us
I20260709 12:22:38.178545 22238 raft_consensus.cc:493] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:22:38.168001 21994 raft_consensus.cc:515] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:38.216274 22045 leader_election.cc:290] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:38.235769 18364 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 1412) took 1631 ms. Trace:
I20260709 12:22:38.249230 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 12 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:38.236078 22218 raft_consensus.cc:3055] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:22:38.226452 22023 raft_consensus.cc:515] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:38.248659 22196 raft_consensus.cc:3060] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:38.264901 17687 raft_consensus.cc:3060] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:38.269274 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:38.250666 21705 raft_consensus.cc:493] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:38.274389 17682 raft_consensus.cc:2393] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 12: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:22:38.274935 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:38.277920 17654 leader_election.cc:304] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:22:38.278925 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:38.246735 22111 raft_consensus.cc:515] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:38.280405 22239 leader_election.cc:290] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 12 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:38.280925 22015 leader_election.cc:290] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 13 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:38.282510 22149 raft_consensus.cc:3055] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:22:38.283979 22165 raft_consensus.cc:3055] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:22:38.284317 22045 raft_consensus.cc:493] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:38.288892 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:38.289949 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:38.292276 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:22:38.293148 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 295519us
I20260709 12:22:38.293454 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:38.292332 22238 raft_consensus.cc:515] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:38.262200 22216 leader_election.cc:290] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:38.253340 18364 rpcz_store.cc:276] 0709 12:22:36.603947 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:36.604074 (+   127us) service_pool.cc:224] Handling call
0709 12:22:36.649050 (+ 44976us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:36.649376 (+   326us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:36.649406 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:36.649896 (+   490us) log.cc:844] Serialized 26 byte log entry
0709 12:22:37.117099 (+467203us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:22:37.117232 (+   133us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:37.117470 (+   238us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:38.004522 (+887052us) spinlock_profiling.cc:243] Waited 24.3 ms on lock 0x7b4400022b4c. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b47459c85 00007f7b4745a6a2 00007f7b4a1cc1db 00007f7b4a1c7aaf 00007f7b4a1bb86e 00007f7b4a1babb6 00007f7b4a1ba7d8 00007f7b4a1c4221 00007f7b4a1c41d9 00007f7b4a1c4169
0709 12:22:38.227401 (+222879us) raft_consensus.cc:1685] finished
0709 12:22:38.227429 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:38.235563 (+  8134us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":253063,"raft.run_cpu_time_us":134,"raft.run_wall_time_us":5948,"spinlock_wait_cycles":55945984,"thread_start_us":301173,"threads_started":1,"wal-append.queue_time_us":438828}
I20260709 12:22:38.304605 21888 raft_consensus.cc:3055] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:22:38.214916 22114 raft_consensus.cc:515] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:38.258868 22218 raft_consensus.cc:740] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:22:38.227756 22175 raft_consensus.cc:515] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:38.336033 21705 raft_consensus.cc:515] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:38.338191 22175 leader_election.cc:290] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:38.339320 21705 leader_election.cc:290] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:38.313954 22134 raft_consensus.cc:515] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:38.322045 18368 raft_consensus.cc:2393] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 12: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:22:38.391479 17683 raft_consensus.cc:3060] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:38.302271 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:22:38.321617 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 69448us
I20260709 12:22:38.316941 22149 raft_consensus.cc:740] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:22:38.319195 22165 raft_consensus.cc:740] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:22:38.414644 18365 raft_consensus.cc:2410] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for term 13 because replica has last-logged OpId of term: 12 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 10.
W20260709 12:22:38.416987 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:38.327396 21888 raft_consensus.cc:740] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:22:38.328917 22188 raft_consensus.cc:2804] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:22:38.266188 22047 raft_consensus.cc:515] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:38.426096 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:22:38.429785 22173 consensus_meta.cc:287] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.562s	user 0.011s	sys 0.007s
I20260709 12:22:38.297627 22017 raft_consensus.cc:515] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:38.345367 22079 leader_election.cc:290] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:38.446799 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:38.453811 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 12 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:38.445747 17759 raft_consensus.cc:2468] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 12.
I20260709 12:22:38.469218 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:38.469381 22190 raft_consensus.cc:493] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:38.470129 17688 raft_consensus.cc:3060] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:38.470351 22190 raft_consensus.cc:515] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:38.476723 22165 consensus_queue.cc:260] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.10, Last appended by leader: 10, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:38.478652 22165 raft_consensus.cc:3060] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Advancing to term 12
W20260709 12:22:38.478895 17804 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:38.413121 22044 raft_consensus.cc:515] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:38.469260 21888 consensus_queue.cc:260] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:38.481209 21888 raft_consensus.cc:3060] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:22:38.491250 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:38.492591 22245 log.cc:927] Time spent T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.185s	user 0.001s	sys 0.000s
I20260709 12:22:38.494488 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:38.495182 18374 raft_consensus.cc:2468] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 9.
I20260709 12:22:38.496120 22149 consensus_queue.cc:260] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.9, Last appended by leader: 9, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:38.497690 22149 raft_consensus.cc:3060] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:38.499990 22194 raft_consensus.cc:493] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:38.500761 22194 raft_consensus.cc:515] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:22:38.507992 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 159607us
W20260709 12:22:38.509155 17803 leader_election.cc:336] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Timed out: RequestConsensusVote RPC to 127.16.29.68:42577 timed out after 2.903s (SENT)
I20260709 12:22:38.509670 17803 leader_election.cc:304] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:38.510710 21995 raft_consensus.cc:2749] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:22:38.511611 22193 raft_consensus.cc:493] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:38.512631 21994 leader_election.cc:290] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 12 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:38.512331 22193 raft_consensus.cc:515] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:22:38.513427 17730 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:38.515183 22218 consensus_queue.cc:260] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 10.11, Last appended by leader: 11, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:38.517449 22218 raft_consensus.cc:3060] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:22:38.520784 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 63967.5us
I20260709 12:22:38.425549 22188 raft_consensus.cc:493] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:38.443114 22227 raft_consensus.cc:493] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:38.534963 18364 raft_consensus.cc:2468] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 11.
I20260709 12:22:38.536881 17801 leader_election.cc:304] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:38.443207 22181 raft_consensus.cc:493] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
W20260709 12:22:38.586189 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 53932.8us
W20260709 12:22:38.587714 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:38.587714 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:38.590953 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:38.543529 22188 raft_consensus.cc:3060] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:22:38.593616 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 68613.9us
I20260709 12:22:38.589437 18362 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 11.
I20260709 12:22:38.614428 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:38.524139 22045 raft_consensus.cc:515] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:22:38.626983 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 356324us
I20260709 12:22:38.632521 22216 raft_consensus.cc:493] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:38.632876 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:38.633219 22216 raft_consensus.cc:515] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:38.633762 17682 raft_consensus.cc:2393] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 13: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:22:38.634701 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:38.635601 17682 raft_consensus.cc:2393] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 10: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:22:38.636291 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 14 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:38.635844 17729 leader_election.cc:304] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:38.602648 21705 consensus_queue.cc:1048] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.053s
I20260709 12:22:38.635579 22216 leader_election.cc:290] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:38.580812 22227 raft_consensus.cc:515] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:38.656482 17652 leader_election.cc:304] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:22:38.664780 17680 raft_consensus.cc:2393] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 10: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:22:38.657034 22148 raft_consensus.cc:2804] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:22:38.669939 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 13 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:38.670168 22148 raft_consensus.cc:493] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:38.672163 18362 raft_consensus.cc:1240] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 12. Current term is 13. Ops: []
W20260709 12:22:38.671499 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 254484us
I20260709 12:22:38.669006 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:38.678054 17729 leader_election.cc:304] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:22:38.694653 17655 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:38.697523 17655 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:38.700578 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 53042.4us
I20260709 12:22:38.693403 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:38.679608 22148 raft_consensus.cc:3060] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:38.676076 22111 leader_election.cc:290] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 13 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:38.718845 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:22:38.695804 22067 consensus_meta.cc:287] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.805s	user 0.013s	sys 0.004s
I20260709 12:22:38.657282 22181 raft_consensus.cc:515] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:38.675194 22264 raft_consensus.cc:493] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:38.733346 17763 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 9.
I20260709 12:22:38.736215 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:38.695976 22114 leader_election.cc:290] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:38.731491 16501 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:22:38.743005 21995 raft_consensus.cc:493] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:38.729527 22134 leader_election.cc:290] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 12 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:38.743716 21995 raft_consensus.cc:515] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:38.746172 21995 leader_election.cc:290] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:22:38.746330 22201 consensus_meta.cc:287] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.510s	user 0.009s	sys 0.002s
W20260709 12:22:38.734408 22069 consensus_meta.cc:287] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.712s	user 0.007s	sys 0.007s
I20260709 12:22:38.731294 22129 raft_consensus.cc:2804] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:22:38.748219 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:22:38.754685 18334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:38.755348 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 380446us
I20260709 12:22:38.756135 17837 raft_consensus.cc:1275] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 9 index: 7. Preceding OpId from leader: term: 12 index: 8. (index mismatch)
I20260709 12:22:38.754590 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:38.731915 22238 leader_election.cc:290] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:22:38.777449 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:38.760813 22042 consensus_meta.cc:287] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 1.201s	user 0.013s	sys 0.015s
I20260709 12:22:38.761731 17833 raft_consensus.cc:1275] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 8 index: 7. Preceding OpId from leader: term: 11 index: 9. (index mismatch)
I20260709 12:22:38.764088 18362 raft_consensus.cc:2468] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 11.
I20260709 12:22:38.764671 17835 raft_consensus.cc:1275] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 11 index: 10. (term mismatch)
I20260709 12:22:38.764896 17834 raft_consensus.cc:1240] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 12. Current term is 13. Ops: []
I20260709 12:22:38.764971 17838 raft_consensus.cc:1275] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20260709 12:22:38.765409 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:38.767012 17829 raft_consensus.cc:1240] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 7. Current term is 8. Ops: [7.5-7.5]
I20260709 12:22:38.768569 22045 leader_election.cc:290] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:38.753800 22201 raft_consensus.cc:515] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:38.770210 22181 leader_election.cc:290] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:22:38.770282 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 121148us
I20260709 12:22:38.772887 17830 raft_consensus.cc:1240] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 11. Current term is 12. Ops: []
I20260709 12:22:38.773110 17683 raft_consensus.cc:2468] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 10.
W20260709 12:22:38.775197 22250 log.cc:927] Time spent T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.159s	user 0.000s	sys 0.001s
I20260709 12:22:38.781634 17688 raft_consensus.cc:2468] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 11.
I20260709 12:22:38.785354 17835 pending_rounds.cc:85] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b: Aborting all ops after (but not including) 9
I20260709 12:22:38.809131 17759 raft_consensus.cc:2468] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 12.
I20260709 12:22:38.811962 22185 raft_consensus.cc:2804] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:22:38.812455 22185 raft_consensus.cc:493] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:22:38.812811 22185 raft_consensus.cc:3060] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:38.759519 22129 raft_consensus.cc:493] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:38.804222 17835 pending_rounds.cc:107] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b: Aborting uncommitted NO_OP operation due to leader change: 10.10
I20260709 12:22:38.828995 17835 raft_consensus.cc:2889] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:38.827934 22133 consensus_queue.cc:1059] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 12.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.039s
I20260709 12:22:38.844691 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:38.844859 17652 leader_election.cc:304] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:22:38.845458 22133 raft_consensus.cc:3055] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:22:38.845829 17830 raft_consensus.cc:2468] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 11.
I20260709 12:22:38.846060 22133 raft_consensus.cc:740] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 12 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
W20260709 12:22:38.831863 17803 leader_election.cc:336] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Timed out: RequestConsensusVote RPC to 127.16.29.68:42577 timed out after 1.515s (SENT)
I20260709 12:22:38.813421 21844 raft_consensus.cc:2749] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:22:38.858302 17801 leader_election.cc:304] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:38.757010 22069 raft_consensus.cc:515] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:38.825651 22129 raft_consensus.cc:3060] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:22:38.874835 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 59522.2us
I20260709 12:22:38.820886 22201 leader_election.cc:290] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 13 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:38.757548 22264 raft_consensus.cc:515] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:38.886207 17832 raft_consensus.cc:1275] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 11 index: 6. Preceding OpId from leader: term: 13 index: 7. (index mismatch)
I20260709 12:22:38.888011 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:22:38.892458 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 195189us
I20260709 12:22:38.893529 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:22:38.900281 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 55610.4us
I20260709 12:22:38.840617 22173 raft_consensus.cc:493] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
W20260709 12:22:38.904189 17654 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:38.908565 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:38.914749 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:38.915711 17832 raft_consensus.cc:2468] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 10.
W20260709 12:22:38.916312 22223 log.cc:927] Time spent T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.068s	user 0.000s	sys 0.001s
I20260709 12:22:38.918157 22129 raft_consensus.cc:515] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:38.921981 22129 leader_election.cc:290] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 13 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:38.923245 21995 raft_consensus.cc:493] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:38.924279 21995 raft_consensus.cc:515] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:38.929080 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:38.930732 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
W20260709 12:22:38.945273 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 131075us
W20260709 12:22:38.946691 17803 leader_election.cc:336] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 election: RPC error from VoteRequest() call to peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Timed out: RequestConsensusVote RPC to 127.16.29.68:42577 timed out after 1.519s (SENT)
W20260709 12:22:38.948617 17730 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:38.952854 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 11 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:38.956343 22173 raft_consensus.cc:515] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:38.957697 22069 leader_election.cc:290] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 11 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:38.958845 21994 raft_consensus.cc:2804] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Leader election won for term 10
W20260709 12:22:38.985846 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:38.895917 22114 raft_consensus.cc:2804] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Leader election won for term 12
W20260709 12:22:38.989601 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 77510.1us
I20260709 12:22:38.959149 22173 leader_election.cc:290] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:38.996378 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:39.004730 17764 raft_consensus.cc:3060] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:22:39.007642 17687 consensus_meta.cc:287] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.696s	user 0.013s	sys 0.001s
I20260709 12:22:39.008673 17687 raft_consensus.cc:2468] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 12.
W20260709 12:22:39.014199 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 67561.8us
I20260709 12:22:39.013762 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:39.020277 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
W20260709 12:22:39.033845 17804 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:39.017897 22042 raft_consensus.cc:493] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:39.010113 22114 raft_consensus.cc:697] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 12 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:22:39.046980 22042 raft_consensus.cc:515] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:39.017246 22047 leader_election.cc:290] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:38.976001 22133 consensus_queue.cc:260] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:39.050259 21995 leader_election.cc:290] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:39.078682 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:39.053864 17830 raft_consensus.cc:1240] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 10. Current term is 11. Ops: []
W20260709 12:22:39.054415 22149 consensus_meta.cc:287] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.556s	user 0.006s	sys 0.009s
I20260709 12:22:39.055029 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 10 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:39.056576 17802 leader_election.cc:304] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:39.057341 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:39.080832 17838 raft_consensus.cc:3060] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:39.057610 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:39.057616 22193 leader_election.cc:290] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:22:39.062597 17803 leader_election.cc:336] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Timed out: RequestConsensusVote RPC to 127.16.29.68:42577 timed out after 1.540s (SENT)
I20260709 12:22:39.015074 21994 raft_consensus.cc:697] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 10 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
W20260709 12:22:39.064668 17730 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:39.067947 18333 leader_election.cc:304] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:39.072511 22188 raft_consensus.cc:515] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:39.073216 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:22:39.075932 22165 consensus_meta.cc:287] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.597s	user 0.009s	sys 0.006s
W20260709 12:22:39.076009 21888 consensus_meta.cc:287] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.594s	user 0.018s	sys 0.000s
I20260709 12:22:39.051827 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:39.081444 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:39.085037 17833 raft_consensus.cc:2393] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 13: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:22:39.071420 18365 raft_consensus.cc:2468] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 10.
W20260709 12:22:39.080065 22276 log.cc:927] Time spent T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.050s	user 0.001s	sys 0.000s
I20260709 12:22:39.088120 17803 leader_election.cc:304] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:39.090824 17837 raft_consensus.cc:2468] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 9.
I20260709 12:22:39.101238 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 14 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:39.109149 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:39.115101 17835 raft_consensus.cc:2468] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 12.
I20260709 12:22:39.115692 22023 leader_election.cc:290] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:39.116926 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:22:39.117278 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 75047.6us
I20260709 12:22:39.117713 17834 raft_consensus.cc:2468] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 11.
I20260709 12:22:39.118631 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:39.118911 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 14 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:39.119153 17834 raft_consensus.cc:3060] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:39.119505 17837 raft_consensus.cc:2468] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 13.
I20260709 12:22:39.120065 21995 raft_consensus.cc:2749] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:22:39.120630 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 12 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:39.120697 22165 raft_consensus.cc:3060] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:39.121649 17837 raft_consensus.cc:2410] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for term 12 because replica has last-logged OpId of term: 8 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 7.
I20260709 12:22:39.116144 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:22:39.125202 22267 log.cc:927] Time spent T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.095s	user 0.000s	sys 0.000s
I20260709 12:22:39.126844 17838 raft_consensus.cc:2468] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 11.
W20260709 12:22:39.127705 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:39.129227 22193 raft_consensus.cc:2804] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:22:39.130074 22193 raft_consensus.cc:697] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 8 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:22:39.130776 17834 raft_consensus.cc:2468] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 12.
I20260709 12:22:39.132607 21994 consensus_queue.cc:237] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:39.137202 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:39.138131 18365 raft_consensus.cc:3060] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:39.112108 22003 raft_consensus.cc:493] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:39.140828 22003 raft_consensus.cc:515] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:39.152191 17682 raft_consensus.cc:2393] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 13: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:22:39.154328 22264 leader_election.cc:290] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:22:39.162308 22253 log.cc:927] Time spent T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.114s	user 0.001s	sys 0.001s
I20260709 12:22:39.113502 17829 raft_consensus.cc:2468] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 10.
W20260709 12:22:39.156522 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 138453us
I20260709 12:22:39.157380 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:39.159103 22188 leader_election.cc:290] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 13 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:22:39.165298 22284 log.cc:927] Time spent T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.116s	user 0.000s	sys 0.001s
I20260709 12:22:39.168892 22190 leader_election.cc:290] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:39.172082 22227 leader_election.cc:290] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:39.174907 17761 raft_consensus.cc:1240] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 15. Current term is 16. Ops: [15.14-15.14]
I20260709 12:22:39.174680 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
W20260709 12:22:39.190600 17803 leader_election.cc:336] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Timed out: RequestConsensusVote RPC to 127.16.29.68:42577 timed out after 1.852s (SENT)
I20260709 12:22:39.191345 17803 leader_election.cc:304] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:39.192639 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 13 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:22:39.197220 18334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:39.194015 17837 raft_consensus.cc:2468] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 12.
I20260709 12:22:39.198733 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 13 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:22:39.200320 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 144003us
W20260709 12:22:39.183306 22218 consensus_meta.cc:287] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.665s	user 0.016s	sys 0.000s
I20260709 12:22:39.131462 22193 consensus_queue.cc:237] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 7.9, Last appended by leader: 9, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:39.201710 17684 raft_consensus.cc:2468] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 10.
I20260709 12:22:39.216423 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:39.539919 17830 raft_consensus.cc:2393] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 12: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:22:39.216548 18374 raft_consensus.cc:3060] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Advancing to term 11
I20260709 12:22:39.217645 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:39.207818 18365 raft_consensus.cc:2468] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 13.
I20260709 12:22:39.221536 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
W20260709 12:22:39.223551 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:39.259616 17729 leader_election.cc:304] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:39.261982 17764 raft_consensus.cc:2468] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 13.
W20260709 12:22:39.309319 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:39.216019 17837 raft_consensus.cc:1240] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 9 LEADER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 8. Current term is 9. Ops: [8.6-8.6]
W20260709 12:22:39.428797 17804 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:39.459491 17685 raft_consensus.cc:2468] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 11.
W20260709 12:22:39.572077 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:39.227226 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:39.228000 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:39.598042 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:39.265779 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:39.247715 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:39.605204 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:22:39.223837 22281 log.cc:927] Time spent T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.174s	user 0.000s	sys 0.001s
I20260709 12:22:39.628468 22023 raft_consensus.cc:493] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:39.629513 22023 raft_consensus.cc:515] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:39.633496 21888 raft_consensus.cc:493] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:22:39.634377 21888 raft_consensus.cc:515] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:39.637121 21888 leader_election.cc:290] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:39.638908 21888 raft_consensus.cc:493] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:39.641129 21888 raft_consensus.cc:515] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:39.652038 22264 raft_consensus.cc:493] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:39.652841 22264 raft_consensus.cc:515] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:39.655364 22264 leader_election.cc:290] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:39.656777 22264 raft_consensus.cc:493] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:39.657452 22264 raft_consensus.cc:515] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:39.664007 22264 leader_election.cc:290] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:39.673278 22218 consensus_queue.cc:1048] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.002s
W20260709 12:22:39.682967 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:39.696278 17687 raft_consensus.cc:2393] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 10: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
W20260709 12:22:39.220705 22279 log.cc:927] Time spent T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.172s	user 0.002s	sys 0.001s
I20260709 12:22:39.698102 17831 raft_consensus.cc:2393] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 11: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:22:39.598057 17757 raft_consensus.cc:2468] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 12.
W20260709 12:22:39.709723 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 524980us
I20260709 12:22:39.694327 17802 leader_election.cc:304] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:22:39.699896 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:39.711171 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 461730us
W20260709 12:22:39.703187 17655 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:39.699025 17830 raft_consensus.cc:2393] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 8: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:22:39.709024 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 12 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:39.722457 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:22:39.724542 17655 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:39.729417 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:39.730084 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:39.266597 22044 leader_election.cc:290] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 16 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:39.733136 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:39.264950 22003 leader_election.cc:290] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:22:39.735574 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 514606us
I20260709 12:22:39.266274 22194 leader_election.cc:290] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:22:39.732704 17832 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.516s	user 0.009s	sys 0.013s
W20260709 12:22:39.746243 17832 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.530s	user 0.010s	sys 0.013s
I20260709 12:22:39.744127 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:39.744661 17837 raft_consensus.cc:3060] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:39.740832 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:39.748039 17761 raft_consensus.cc:2468] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 12.
I20260709 12:22:39.749701 17801 leader_election.cc:304] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:39.749962 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:39.751376 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:39.749693 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 12 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:39.755927 17764 raft_consensus.cc:2393] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 12: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:22:39.222929 22148 raft_consensus.cc:515] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:39.748198 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:39.758834 18365 raft_consensus.cc:2468] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 8.
I20260709 12:22:39.759583 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
W20260709 12:22:39.760509 17730 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:39.762563 17730 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:39.735606 22201 raft_consensus.cc:2804] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:22:39.225557 22185 raft_consensus.cc:515] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:39.765312 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:39.773593 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:39.775849 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 12 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:39.779527 18365 raft_consensus.cc:2468] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 7.
I20260709 12:22:39.220280 22114 consensus_queue.cc:237] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.6, Last appended by leader: 6, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:22:39.767562 22165 consensus_meta.cc:287] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.643s	user 0.007s	sys 0.005s
I20260709 12:22:39.785583 18365 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59886 (request call id 1319) took 1136 ms. Trace:
I20260709 12:22:39.789446 17756 raft_consensus.cc:2393] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 10: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:22:39.791533 18365 rpcz_store.cc:276] 0709 12:22:38.643565 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:39.773294 (+1129729us) service_pool.cc:224] Handling call
0709 12:22:39.780220 (+  6926us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:39.793475 17759 raft_consensus.cc:1240] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 11. Current term is 12. Ops: []
I20260709 12:22:39.781059 22023 leader_election.cc:290] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:39.798898 17652 leader_election.cc:304] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:39.765534 22194 raft_consensus.cc:493] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:39.800812 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 12 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:39.771396 22201 raft_consensus.cc:493] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:39.770139 21888 leader_election.cc:290] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:39.823212 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:39.836609 17686 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37396 (request call id 2249) took 1626 ms. Trace:
I20260709 12:22:39.851017 22023 consensus_queue.cc:1059] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 14.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:22:39.879635 18361 raft_consensus.cc:2468] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 7.
I20260709 12:22:39.851779 17829 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 11.
I20260709 12:22:39.856642 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:39.804216 22165 raft_consensus.cc:2749] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:22:39.861963 22185 leader_election.cc:290] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 12 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:39.865397 22148 leader_election.cc:290] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 10 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:39.872534 22192 consensus_queue.cc:1048] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:22:39.875068 22129 raft_consensus.cc:2749] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:22:39.774654 22131 raft_consensus.cc:2749] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:22:39.896239 17837 raft_consensus.cc:2468] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 12.
I20260709 12:22:39.876226 22201 raft_consensus.cc:3060] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:39.902611 18374 raft_consensus.cc:2468] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 11.
W20260709 12:22:39.906078 17838 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.708s	user 0.030s	sys 0.080s
W20260709 12:22:39.979490 17838 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.781s	user 0.030s	sys 0.081s
I20260709 12:22:39.990134 17684 raft_consensus.cc:2468] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 13.
I20260709 12:22:39.991084 17684 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44898 (request call id 902) took 1289 ms. Trace:
I20260709 12:22:39.835235 17762 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 11.
I20260709 12:22:39.965780 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:39.963855 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:39.835279 22184 raft_consensus.cc:2749] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:22:39.974864 17837 raft_consensus.cc:1240] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 11. Current term is 12. Ops: []
I20260709 12:22:39.978989 18365 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 9.
W20260709 12:22:39.997599 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 44645.4us
I20260709 12:22:39.907789 17652 leader_election.cc:304] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
W20260709 12:22:40.006155 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 205385us
I20260709 12:22:40.008140 17652 leader_election.cc:304] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:39.991534 17684 rpcz_store.cc:276] 0709 12:22:38.714784 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:39.716879 (+1002095us) service_pool.cc:224] Handling call
0709 12:22:39.991003 (+274124us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:22:39.996383 18334 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.574s	user 0.018s	sys 0.022s
W20260709 12:22:40.283553 18334 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.862s	user 0.018s	sys 0.022s
I20260709 12:22:40.011345 18363 raft_consensus.cc:2393] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 8: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:22:40.012318 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 10 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:39.879565 17686 rpcz_store.cc:276] 0709 12:22:38.097408 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:38.097636 (+   228us) service_pool.cc:224] Handling call
0709 12:22:38.275401 (+177765us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:38.651105 (+375704us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:38.651147 (+    42us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:38.713749 (+ 62602us) log.cc:844] Serialized 26 byte log entry
0709 12:22:38.893613 (+179864us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:22:38.893668 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:38.893751 (+    83us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:39.164484 (+270733us) spinlock_profiling.cc:243] Waited 10.8 ms on lock 0x7b4400022b4c. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b47459c85 00007f7b4745a6a2 00007f7b4a1cc1db 00007f7b4a1c7aaf 00007f7b4a1bb86e 00007f7b4a1babb6 00007f7b4a1ba7d8 00007f7b4a1c4221 00007f7b4a1c41d9 00007f7b4a1c4169
0709 12:22:39.606594 (+442110us) raft_consensus.cc:1685] finished
0709 12:22:39.606626 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:39.724138 (+117512us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":17041,"raft.run_cpu_time_us":130,"raft.run_wall_time_us":130,"spinlock_wait_cycles":25327360,"thread_start_us":502534,"threads_started":2,"wal-append.queue_time_us":311948}
I20260709 12:22:40.018640 22218 raft_consensus.cc:2804] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:22:40.019392 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:40.020124 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:40.027839 22290 raft_consensus.cc:2804] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:22:40.028505 17834 raft_consensus.cc:1275] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20260709 12:22:40.028625 18366 raft_consensus.cc:2468] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 13.
I20260709 12:22:40.029875 22042 leader_election.cc:290] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:40.030858 17685 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44898 (request call id 905) took 1271 ms. Trace:
I20260709 12:22:40.294071 17685 rpcz_store.cc:276] 0709 12:22:38.759797 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:38.952513 (+192716us) service_pool.cc:224] Handling call
0709 12:22:40.030679 (+1078166us) spinlock_profiling.cc:243] Waited 312 ms on lock 0x7b54000c4c54. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:22:40.030777 (+    98us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":716481408}
W20260709 12:22:40.034165 22260 log.cc:927] Time spent T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.841s	user 0.019s	sys 0.019s
I20260709 12:22:40.035997 21888 raft_consensus.cc:2804] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Leader pre-election won for term 12
W20260709 12:22:40.036818 17683 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37396 (request call id 2253) took 1937 ms (client timeout 2070 ms). Trace:
I20260709 12:22:39.916601 22194 raft_consensus.cc:515] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:40.043586 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:40.044704 22045 raft_consensus.cc:493] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:22:40.046084 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:40.047062 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:40.048892 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:40.303397 18364 raft_consensus.cc:3060] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:40.303311 17764 raft_consensus.cc:2468] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 11.
W20260709 12:22:40.051695 22275 log.cc:927] Time spent T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.999s	user 0.012s	sys 0.015s
W20260709 12:22:40.053424 22256 log.cc:927] Time spent T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 1.012s	user 0.002s	sys 0.028s
W20260709 12:22:40.054184 17682 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44898 (request call id 909) took 1232 ms (client timeout 1504 ms). Trace:
I20260709 12:22:40.055684 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 12 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:40.018680 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:40.065395 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:40.066915 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:40.043622 17652 leader_election.cc:304] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:40.309034 18374 raft_consensus.cc:2468] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 10.
W20260709 12:22:40.143527 22216 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.519s	user 0.014s	sys 0.014s
I20260709 12:22:40.311981 17761 raft_consensus.cc:2393] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 7: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:22:40.313416 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:22:40.307816 17682 rpcz_store.cc:269] 0709 12:22:38.821818 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:38.848819 (+ 27001us) service_pool.cc:224] Handling call
0709 12:22:40.053912 (+1205093us) spinlock_profiling.cc:243] Waited 215 ms on lock 0x7b5400137ed4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:22:40.054106 (+   194us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":494468480}
I20260709 12:22:40.011847 22165 raft_consensus.cc:493] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:39.963855 22264 raft_consensus.cc:493] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
W20260709 12:22:40.235914 17655 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:40.327284 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 283825us
I20260709 12:22:40.328333 17652 leader_election.cc:304] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:40.322629 18361 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59886 (request call id 1324) took 1544 ms. Trace:
I20260709 12:22:40.333209 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:22:40.312041 22216 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.687s	user 0.019s	sys 0.014s
I20260709 12:22:40.255230 18362 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59886 (request call id 1322) took 1607 ms. Trace:
I20260709 12:22:40.332137 18361 rpcz_store.cc:276] 0709 12:22:38.778374 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:40.003956 (+1225582us) service_pool.cc:224] Handling call
0709 12:22:40.322546 (+318590us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:22:40.264907 17804 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.539s	user 0.000s	sys 0.008s
W20260709 12:22:40.268910 17729 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.509s	user 0.008s	sys 0.000s
W20260709 12:22:40.266026 17835 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.504s	user 0.004s	sys 0.004s
W20260709 12:22:40.365967 22149 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.548s	user 0.026s	sys 0.080s
I20260709 12:22:40.245425 22017 leader_election.cc:290] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:40.326534 22045 raft_consensus.cc:515] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:40.353869 22264 raft_consensus.cc:515] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:22:40.288811 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 930945us
W20260709 12:22:40.275488 22134 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 1.226s	user 0.015s	sys 0.019s
I20260709 12:22:40.288944 17756 raft_consensus.cc:2468] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 12.
I20260709 12:22:40.374171 22078 raft_consensus.cc:493] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:22:40.331281 22165 raft_consensus.cc:515] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:40.289474 17758 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 9.
W20260709 12:22:40.300058 17683 rpcz_store.cc:269] 0709 12:22:38.099028 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:38.274544 (+175516us) service_pool.cc:224] Handling call
0709 12:22:40.036658 (+1762114us) spinlock_profiling.cc:243] Waited 92.9 ms on lock 0x7b54003462d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:22:40.036762 (+   104us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":213689088}
W20260709 12:22:40.305852 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:40.311347 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:40.308291 18368 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 11.
I20260709 12:22:40.292454 22218 raft_consensus.cc:493] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:40.319636 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:40.322170 17684 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37396 (request call id 2266) took 1154 ms. Trace:
W20260709 12:22:40.237116 17730 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:40.402474 17684 rpcz_store.cc:276] 0709 12:22:39.167342 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:40.313060 (+1145718us) service_pool.cc:224] Handling call
0709 12:22:40.322035 (+  8975us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:22:40.395670 22134 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.345s	user 0.015s	sys 0.020s
I20260709 12:22:40.303747 22290 raft_consensus.cc:697] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 13 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:22:40.303078 21888 raft_consensus.cc:493] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:40.316195 22042 raft_consensus.cc:493] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:40.355739 17654 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.584s	user 0.000s	sys 0.021s
W20260709 12:22:40.410171 17654 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.638s	user 0.000s	sys 0.022s
I20260709 12:22:40.317657 22079 raft_consensus.cc:2804] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:22:40.312996 22192 raft_consensus.cc:493] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:40.357712 17762 raft_consensus.cc:1240] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 9. Current term is 10. Ops: [9.9-9.9]
I20260709 12:22:40.358942 18363 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59886 (request call id 1323) took 1616 ms. Trace:
I20260709 12:22:40.354899 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:40.334684 22184 raft_consensus.cc:2804] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:22:40.365756 17652 leader_election.cc:304] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:22:40.372485 17804 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.647s	user 0.000s	sys 0.008s
I20260709 12:22:40.414573 18367 raft_consensus.cc:2410] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for term 13 because replica has last-logged OpId of term: 11 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 9.
W20260709 12:22:40.416280 17804 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:40.416942 17755 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.582s	user 0.011s	sys 0.015s
I20260709 12:22:40.413851 22042 raft_consensus.cc:515] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:22:40.417271 17755 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.652s	user 0.011s	sys 0.016s
I20260709 12:22:40.418031 17755 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38776 (request call id 2183) took 1228 ms. Trace:
I20260709 12:22:40.418784 22042 leader_election.cc:290] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:40.374286 18361 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 1432) took 1354 ms. Trace:
I20260709 12:22:40.419175 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:22:40.377600 17729 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.620s	user 0.009s	sys 0.000s
I20260709 12:22:40.411378 22079 raft_consensus.cc:493] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:40.413472 21888 raft_consensus.cc:3060] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:40.413913 22184 raft_consensus.cc:493] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
W20260709 12:22:40.273473 17832 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.509s	user 0.007s	sys 0.002s
W20260709 12:22:40.377208 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 764746us
W20260709 12:22:40.390851 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:40.279690 17803 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.554s	user 0.007s	sys 0.003s
W20260709 12:22:40.391983 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 437747us
W20260709 12:22:40.383412 22149 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.573s	user 0.026s	sys 0.080s
I20260709 12:22:40.419844 17831 raft_consensus.cc:2468] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 9.
I20260709 12:22:40.434278 22317 raft_consensus.cc:493] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:40.435256 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:40.435040 22317 raft_consensus.cc:515] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:40.435937 17831 raft_consensus.cc:2468] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 11.
I20260709 12:22:40.436842 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:40.437882 22317 leader_election.cc:290] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:40.438577 22317 raft_consensus.cc:493] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:40.399535 17760 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38786 (request call id 2884) took 2119 ms. Trace:
I20260709 12:22:40.425271 17682 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37396 (request call id 2269) took 1030 ms. Trace:
I20260709 12:22:40.442030 17682 rpcz_store.cc:276] 0709 12:22:39.395389 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:40.325722 (+930333us) service_pool.cc:224] Handling call
0709 12:22:40.425216 (+ 99494us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:40.441795 17760 rpcz_store.cc:276] 0709 12:22:38.276661 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:38.276846 (+   185us) service_pool.cc:224] Handling call
0709 12:22:38.277208 (+   362us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:38.284890 (+  7682us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:38.284942 (+    52us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:38.285899 (+   957us) log.cc:844] Serialized 26 byte log entry
0709 12:22:38.602358 (+316459us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:22:38.602465 (+   107us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:38.602628 (+   163us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:40.033368 (+1430740us) spinlock_profiling.cc:243] Waited 119 ms on lock 0x7b4400022b4c. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b47459c85 00007f7b4745a6a2 00007f7b4a1cc1db 00007f7b4a1c7aaf 00007f7b4a1bb86e 00007f7b4a1babb6 00007f7b4a1ba7d8 00007f7b4a1c4221 00007f7b4a1c41d9 00007f7b4a1c4169
0709 12:22:40.392179 (+358811us) raft_consensus.cc:1685] finished
0709 12:22:40.392200 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:40.396111 (+  3911us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":7108,"raft.queue_time_us":14449,"raft.run_cpu_time_us":157,"raft.run_wall_time_us":160,"spinlock_wait_cycles":274680704,"thread_start_us":307801,"threads_started":1,"wal-append.queue_time_us":781380}
I20260709 12:22:40.443233 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:40.429008 22045 leader_election.cc:290] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:40.419451 18361 rpcz_store.cc:276] 0709 12:22:39.019735 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:40.346887 (+1327152us) service_pool.cc:224] Handling call
0709 12:22:40.373757 (+ 26870us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:40.373946 (+   189us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:40.373970 (+    24us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:40.374011 (+    41us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:22:40.374051 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:40.374112 (+    61us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:40.374231 (+   119us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:40.429611 17802 leader_election.cc:304] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:22:40.445747 22277 log.cc:927] Time spent T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 1.196s	user 0.009s	sys 0.018s
I20260709 12:22:40.446873 17802 leader_election.cc:304] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:40.406054 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:40.448398 17681 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37396 (request call id 2244) took 2353 ms. Trace:
I20260709 12:22:40.431937 21888 raft_consensus.cc:515] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:40.448879 17683 raft_consensus.cc:2468] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 11.
I20260709 12:22:40.351289 18362 rpcz_store.cc:276] 0709 12:22:38.647937 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:40.234561 (+1586624us) service_pool.cc:224] Handling call
0709 12:22:40.245238 (+ 10677us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:40.254730 (+  9492us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:40.254779 (+    49us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:40.254846 (+    67us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:22:40.254893 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:40.254993 (+   100us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:40.255167 (+   174us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:40.444084 17682 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37396 (request call id 2263) took 1366 ms. Trace:
I20260709 12:22:40.418792 17829 raft_consensus.cc:1275] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 8 index: 10. Preceding OpId from leader: term: 10 index: 11. (index mismatch)
I20260709 12:22:40.451717 17683 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37396 (request call id 2258) took 1441 ms. Trace:
I20260709 12:22:40.450822 17682 rpcz_store.cc:276] 0709 12:22:39.077252 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:40.442960 (+1365708us) service_pool.cc:224] Handling call
0709 12:22:40.444017 (+  1057us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:22:40.410912 17686 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37402 (request call id 2632) took 1718 ms (client timeout 2167 ms). Trace:
W20260709 12:22:40.419317 22293 log.cc:927] Time spent T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.063s	user 0.000s	sys 0.000s
W20260709 12:22:40.377305 17835 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.619s	user 0.005s	sys 0.004s
I20260709 12:22:40.412806 18363 rpcz_store.cc:276] 0709 12:22:38.742288 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:40.315695 (+1573407us) service_pool.cc:224] Handling call
0709 12:22:40.336818 (+ 21123us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:40.358479 (+ 21661us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:40.358523 (+    44us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:40.358588 (+    65us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:22:40.358637 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:40.358718 (+    81us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:40.358874 (+   156us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:40.460825 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:40.437520 17831 raft_consensus.cc:2393] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 13: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
W20260709 12:22:40.420300 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 54808.4us
I20260709 12:22:40.461565 17684 raft_consensus.cc:2468] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 12.
I20260709 12:22:40.462239 17684 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44898 (request call id 906) took 1641 ms. Trace:
I20260709 12:22:40.462539 17684 rpcz_store.cc:276] 0709 12:22:38.820660 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:40.403236 (+1582576us) service_pool.cc:224] Handling call
0709 12:22:40.462182 (+ 58946us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:22:40.420578 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 682751us
I20260709 12:22:40.403967 22218 raft_consensus.cc:3060] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:40.420926 22265 raft_consensus.cc:2749] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 12:22:40.424433 22079 raft_consensus.cc:3060] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:40.441185 17832 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35042 (request call id 2158) took 1389 ms. Trace:
I20260709 12:22:40.469201 17832 rpcz_store.cc:276] 0709 12:22:39.051787 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:39.071199 (+ 19412us) service_pool.cc:224] Handling call
0709 12:22:39.076268 (+  5069us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:39.076646 (+   378us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:22:39.192919 (+116273us) log.cc:844] Serialized 14 byte log entry
0709 12:22:40.439716 (+1246797us) log.cc:844] Serialized 14 byte log entry
0709 12:22:40.440804 (+  1088us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:40.440861 (+    57us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:22:40.440912 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:40.440988 (+    76us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:40.441109 (+   121us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":9954,"prepare.queue_time_us":190220,"prepare.run_cpu_time_us":69,"prepare.run_wall_time_us":71,"raft.queue_time_us":227858,"raft.run_cpu_time_us":303,"raft.run_wall_time_us":52965,"thread_start_us":1321193,"threads_started":3,"wal-append.queue_time_us":810100}
W20260709 12:22:40.430442 17803 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.704s	user 0.007s	sys 0.003s
I20260709 12:22:40.447445 18371 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 1426) took 2180 ms. Trace:
I20260709 12:22:40.472098 18371 rpcz_store.cc:276] 0709 12:22:38.266955 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:38.270181 (+  3226us) service_pool.cc:224] Handling call
0709 12:22:38.302808 (+ 32627us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:38.303427 (+   619us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 12:22:38.303481 (+    54us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:38.303987 (+   506us) log.cc:844] Serialized 26 byte log entry
0709 12:22:38.448233 (+144246us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:22:38.448301 (+    68us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:38.448416 (+   115us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:40.052816 (+1604400us) spinlock_profiling.cc:243] Waited 1.19 ms on lock 0x7b4400022b4c. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b47459c85 00007f7b4745a6a2 00007f7b4a1cc1db 00007f7b4a1c7aaf 00007f7b4a1bb86e 00007f7b4a1babb6 00007f7b4a1ba7d8 00007f7b4a1c4221 00007f7b4a1c41d9 00007f7b4a1c4169
0709 12:22:40.447121 (+394305us) raft_consensus.cc:1685] finished
0709 12:22:40.447148 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:40.447370 (+   222us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5534,"raft.queue_time_us":7518,"raft.run_cpu_time_us":256,"raft.run_wall_time_us":27997,"spinlock_wait_cycles":2727808,"thread_start_us":113183,"threads_started":1,"wal-append.queue_time_us":736735}
I20260709 12:22:40.474748 18371 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 1439) took 1347 ms. Trace:
I20260709 12:22:40.428311 22184 raft_consensus.cc:3060] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:40.447961 22306 raft_consensus.cc:3060] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:40.452193 17683 rpcz_store.cc:276] 0709 12:22:39.010588 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:40.405607 (+1395019us) service_pool.cc:224] Handling call
0709 12:22:40.451652 (+ 46045us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:40.476769 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
W20260709 12:22:40.477530 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:40.484081 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:40.475121 18371 rpcz_store.cc:276] 0709 12:22:39.127234 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:40.473820 (+1346586us) service_pool.cc:224] Handling call
0709 12:22:40.474125 (+   305us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:40.474334 (+   209us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:40.474361 (+    27us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:40.474408 (+    47us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:22:40.474453 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:40.474528 (+    75us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:40.474657 (+   129us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:40.452733 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:22:40.281433 22193 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.659s	user 0.009s	sys 0.010s
I20260709 12:22:40.419981 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:40.504853 17834 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 9.
W20260709 12:22:40.452554 17686 rpcz_store.cc:269] 0709 12:22:38.692628 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:40.297692 (+1605064us) service_pool.cc:224] Handling call
0709 12:22:40.410852 (+113160us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:22:40.404459 17730 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:40.420262 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:40.431769 17728 leader_election.cc:304] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:40.428536 17833 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35040 (request call id 2416) took 1214 ms. Trace:
I20260709 12:22:40.438809 22192 raft_consensus.cc:515] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:40.418370 17755 rpcz_store.cc:276] 0709 12:22:39.189426 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:39.189595 (+   169us) service_pool.cc:224] Handling call
0709 12:22:39.211607 (+ 22012us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:39.211932 (+   325us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 12:22:39.696303 (+484371us) log.cc:844] Serialized 14 byte log entry
0709 12:22:40.417690 (+721387us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:40.417741 (+    51us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:22:40.417789 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:40.417852 (+    63us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:40.417954 (+   102us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":484096,"prepare.run_cpu_time_us":52,"prepare.run_wall_time_us":54,"raft.queue_time_us":5235,"raft.run_cpu_time_us":76,"raft.run_wall_time_us":77,"thread_start_us":651910,"threads_started":1,"wal-append.queue_time_us":705274}
I20260709 12:22:40.460232 21888 leader_election.cc:290] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 12 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:40.418054 22165 leader_election.cc:290] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:40.466506 22317 raft_consensus.cc:515] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:40.467060 21995 raft_consensus.cc:2749] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 12:22:40.467934 17729 leader_election.cc:304] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:40.464056 17652 leader_election.cc:304] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:40.524683 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:40.410187 22003 raft_consensus.cc:3055] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 15 LEADER]: Stepping down as leader of term 15
W20260709 12:22:40.496917 18334 leader_election.cc:336] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 10 election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Timed out: RequestConsensusVote RPC to 127.16.29.65:33071 timed out after 1.709s (SENT)
I20260709 12:22:40.533787 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:40.497942 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:40.448827 17681 rpcz_store.cc:276] 0709 12:22:38.094995 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:38.096069 (+  1074us) service_pool.cc:224] Handling call
0709 12:22:38.261006 (+164937us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:22:38.445999 (+184993us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:38.446044 (+    45us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:22:38.621166 (+175122us) log.cc:844] Serialized 52 byte log entry
0709 12:22:38.729613 (+108447us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:22:38.729647 (+    34us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:38.729734 (+    87us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:40.445139 (+1715405us) spinlock_profiling.cc:243] Waited 90.9 ms on lock 0x7b4400022b4c. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b47459c85 00007f7b4745a6a2 00007f7b4a1cc1db 00007f7b4a1c7aaf 00007f7b4a1bb86e 00007f7b4a1babb6 00007f7b4a1ba7d8 00007f7b4a1c4221 00007f7b4a1c41d9 00007f7b4a1c4169
0709 12:22:40.447942 (+  2803us) raft_consensus.cc:1685] finished
0709 12:22:40.447962 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:40.448182 (+   220us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":12743,"raft.run_cpu_time_us":79,"raft.run_wall_time_us":79,"spinlock_wait_cycles":209270784,"thread_start_us":200347,"threads_started":2,"wal-append.queue_time_us":413275}
I20260709 12:22:40.499156 17802 leader_election.cc:304] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:22:40.519410 22193 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.882s	user 0.011s	sys 0.011s
I20260709 12:22:40.503480 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:40.538923 17684 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37396 (request call id 2264) took 1461 ms. Trace:
I20260709 12:22:40.539275 17684 rpcz_store.cc:276] 0709 12:22:39.077648 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:40.463365 (+1385717us) service_pool.cc:224] Handling call
0709 12:22:40.538866 (+ 75501us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:40.398581 22078 raft_consensus.cc:515] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:40.524832 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:22:40.525344 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 61362.1us
I20260709 12:22:40.541551 17652 leader_election.cc:304] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:22:40.535673 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 67859.3us
I20260709 12:22:40.544596 17729 leader_election.cc:304] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:40.502749 17654 leader_election.cc:304] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:22:40.506686 17833 rpcz_store.cc:276] 0709 12:22:39.214353 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:39.787160 (+572807us) service_pool.cc:224] Handling call
0709 12:22:39.787562 (+   402us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:39.796895 (+  9333us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:39.796930 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:39.797571 (+   641us) log.cc:844] Serialized 26 byte log entry
0709 12:22:40.265676 (+468105us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:22:40.265743 (+    67us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:40.265854 (+   111us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:40.428326 (+162472us) raft_consensus.cc:1685] finished
0709 12:22:40.428345 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:40.428479 (+   134us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":259454,"raft.run_cpu_time_us":98,"raft.run_wall_time_us":99,"thread_start_us":467538,"threads_started":1,"wal-append.queue_time_us":591665}
W20260709 12:22:40.537269 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 105569us
I20260709 12:22:40.537688 17830 raft_consensus.cc:3060] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Advancing to term 13
I20260709 12:22:40.545914 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:40.547109 17682 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37396 (request call id 2256) took 1558 ms. Trace:
I20260709 12:22:40.547535 17682 rpcz_store.cc:276] 0709 12:22:38.988505 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:40.538210 (+1549705us) service_pool.cc:224] Handling call
0709 12:22:40.547038 (+  8828us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:40.548635 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 12 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:40.549625 17682 raft_consensus.cc:2410] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for term 12 because replica has last-logged OpId of term: 8 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 7.
I20260709 12:22:40.496474 22042 raft_consensus.cc:493] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:40.551784 17829 raft_consensus.cc:2468] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 12.
I20260709 12:22:40.534294 18334 leader_election.cc:304] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:22:40.561702 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:40.564710 17682 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37396 (request call id 2255) took 1601 ms. Trace:
I20260709 12:22:40.565165 17682 rpcz_store.cc:276] 0709 12:22:38.963159 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:40.548328 (+1585169us) service_pool.cc:224] Handling call
0709 12:22:40.564634 (+ 16306us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:40.566311 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:40.573791 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:40.574698 17681 raft_consensus.cc:2468] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 6.
I20260709 12:22:40.575338 17681 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37396 (request call id 2265) took 1497 ms. Trace:
I20260709 12:22:40.577289 17684 raft_consensus.cc:2393] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 13: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:22:40.575779 17681 rpcz_store.cc:276] 0709 12:22:39.081454 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:40.551653 (+1470199us) service_pool.cc:224] Handling call
0709 12:22:40.575259 (+ 23606us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:40.551369 22017 raft_consensus.cc:2804] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:22:40.580296 22301 raft_consensus.cc:2749] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:22:40.580135 22042 raft_consensus.cc:515] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:40.580366 22017 raft_consensus.cc:493] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:40.581401 22017 raft_consensus.cc:3060] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:22:40.582561 22272 log.cc:927] Time spent T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 1.359s	user 0.012s	sys 0.010s
I20260709 12:22:40.582871 22042 leader_election.cc:290] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:40.584021 17689 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37396 (request call id 2250) took 2486 ms. Trace:
I20260709 12:22:40.584189 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:40.584839 17686 raft_consensus.cc:2468] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 10.
I20260709 12:22:40.584967 22078 leader_election.cc:290] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:40.585613 22078 raft_consensus.cc:2804] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:22:40.585986 22078 raft_consensus.cc:493] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:40.586378 22078 raft_consensus.cc:3060] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:40.589783 17729 leader_election.cc:304] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:40.533885 22003 raft_consensus.cc:740] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:22:40.605063 17763 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38776 (request call id 2178) took 1431 ms. Trace:
I20260709 12:22:40.537936 22307 raft_consensus.cc:2804] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:22:40.605577 17763 rpcz_store.cc:276] 0709 12:22:39.173746 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:39.175873 (+  2127us) service_pool.cc:224] Handling call
0709 12:22:39.176231 (+   358us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:39.176699 (+   468us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 12:22:39.176725 (+    26us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:39.177251 (+   526us) log.cc:844] Serialized 26 byte log entry
0709 12:22:39.495617 (+318366us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:22:39.495678 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:39.495781 (+   103us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:40.604787 (+1109006us) raft_consensus.cc:1685] finished
0709 12:22:40.604818 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:40.604986 (+   168us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1286,"raft.run_cpu_time_us":86,"raft.run_wall_time_us":86,"thread_start_us":317811,"threads_started":1,"wal-append.queue_time_us":1118766}
I20260709 12:22:40.562263 22216 raft_consensus.cc:2804] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:22:40.583451 22042 consensus_queue.cc:1059] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20260709 12:22:40.620497 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 147608us
I20260709 12:22:40.588573 22194 leader_election.cc:290] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:40.607964 18371 raft_consensus.cc:1275] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 9 index: 7. Preceding OpId from leader: term: 12 index: 8. (index mismatch)
I20260709 12:22:40.584421 17689 rpcz_store.cc:276] 0709 12:22:38.097815 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:38.294200 (+196385us) service_pool.cc:224] Handling call
0709 12:22:38.294598 (+   398us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:38.294985 (+   387us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:38.295033 (+    48us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:38.295585 (+   552us) log.cc:844] Serialized 26 byte log entry
0709 12:22:38.641629 (+346044us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:22:38.641675 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:38.641746 (+    71us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:40.566187 (+1924441us) spinlock_profiling.cc:243] Waited 95.4 ms on lock 0x7b4400022b4c. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b47459c85 00007f7b4745a6a2 00007f7b4a1cc1db 00007f7b4a1c7aaf 00007f7b4a1bb86e 00007f7b4a1babb6 00007f7b4a1ba7d8 00007f7b4a1c4221 00007f7b4a1c41d9 00007f7b4a1c4169
0709 12:22:40.583703 (+ 17516us) raft_consensus.cc:1685] finished
0709 12:22:40.583726 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:40.583960 (+   234us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1054,"raft.run_cpu_time_us":77,"raft.run_wall_time_us":77,"spinlock_wait_cycles":219427584,"thread_start_us":345629,"threads_started":1,"wal-append.queue_time_us":927279}
I20260709 12:22:40.589135 22045 raft_consensus.cc:2804] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:22:40.606825 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:40.624897 22311 raft_consensus.cc:2804] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260709 12:22:40.637744 17803 leader_election.cc:336] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Timed out: RequestConsensusVote RPC to 127.16.29.68:42577 timed out after 1.653s (SENT)
I20260709 12:22:40.640869 17803 leader_election.cc:304] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
W20260709 12:22:40.638386 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 94901.4us
W20260709 12:22:40.638933 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 178080us
I20260709 12:22:40.604166 22323 raft_consensus.cc:493] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:40.609551 22307 raft_consensus.cc:493] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:40.620450 22216 raft_consensus.cc:493] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:22:40.654415 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 151746us
I20260709 12:22:40.632486 22165 raft_consensus.cc:493] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:40.655632 17654 leader_election.cc:304] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:40.658360 17802 leader_election.cc:304] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:40.661723 22017 raft_consensus.cc:515] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:40.664062 22017 leader_election.cc:290] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 11 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:40.680562 17689 raft_consensus.cc:1275] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 9 index: 6. (term mismatch)
I20260709 12:22:40.681025 17689 pending_rounds.cc:85] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3: Aborting all ops after (but not including) 5
I20260709 12:22:40.681262 17689 pending_rounds.cc:107] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3: Aborting uncommitted NO_OP operation due to leader change: 8.6
I20260709 12:22:40.681505 17689 raft_consensus.cc:2889] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:40.644459 22311 raft_consensus.cc:493] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:22:40.702175 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 64472.7us
I20260709 12:22:40.646847 22045 raft_consensus.cc:697] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:22:40.660270 22216 raft_consensus.cc:3060] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:40.655964 22307 raft_consensus.cc:3060] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:40.712615 17689 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37402 (request call id 2633) took 2014 ms. Trace:
I20260709 12:22:40.735052 17689 rpcz_store.cc:276] 0709 12:22:38.693059 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:40.637826 (+1944767us) service_pool.cc:224] Handling call
0709 12:22:40.659790 (+ 21964us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:40.706891 (+ 47101us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:40.707158 (+   267us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:40.643622 22290 consensus_queue.cc:237] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:40.708652 22134 raft_consensus.cc:2804] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Leader election won for term 11
W20260709 12:22:40.738070 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 323158us
I20260709 12:22:40.704522 22311 raft_consensus.cc:3060] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:22:40.745810 17803 leader_election.cc:336] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Timed out: RequestConsensusVote RPC to 127.16.29.68:42577 timed out after 1.738s (SENT)
I20260709 12:22:40.747587 17803 leader_election.cc:304] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:40.704692 22301 raft_consensus.cc:2749] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:22:40.693590 22165 raft_consensus.cc:515] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:40.656034 22323 raft_consensus.cc:515] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:22:40.800732 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 46630.9us
I20260709 12:22:40.757910 17689 raft_consensus.cc:1240] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 9. Current term is 10. Ops: []
W20260709 12:22:40.761255 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 118725us
W20260709 12:22:40.841197 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 95438.2us
W20260709 12:22:40.758057 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 115039us
I20260709 12:22:40.814175 18364 raft_consensus.cc:2468] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 13.
I20260709 12:22:40.843588 18364 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59918 (request call id 1443) took 1112 ms. Trace:
W20260709 12:22:40.797315 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 207579us
I20260709 12:22:40.844112 17689 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44898 (request call id 903) took 2113 ms. Trace:
I20260709 12:22:40.844332 18364 rpcz_store.cc:276] 0709 12:22:39.731298 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:40.048587 (+317289us) service_pool.cc:224] Handling call
0709 12:22:40.843507 (+794920us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:40.844784 17729 leader_election.cc:304] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:40.844578 17689 rpcz_store.cc:276] 0709 12:22:38.730413 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:40.743356 (+2012943us) service_pool.cc:224] Handling call
0709 12:22:40.755705 (+ 12349us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:40.843753 (+ 88048us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:40.844055 (+   302us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:40.748457 22264 leader_election.cc:290] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:40.849261 18368 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 1435) took 1779 ms. Trace:
I20260709 12:22:40.850303 17689 raft_consensus.cc:1240] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 11. Current term is 12. Ops: [11.9-11.9]
I20260709 12:22:40.749485 22192 leader_election.cc:290] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:22:40.866160 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:40.756899 22282 log.cc:927] Time spent T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 1.706s	user 0.034s	sys 0.011s
I20260709 12:22:40.736449 22003 consensus_queue.cc:260] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 15.14, Last appended by leader: 14, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:40.896943 17652 leader_election.cc:304] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:40.910172 17688 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44898 (request call id 907) took 2074 ms. Trace:
I20260709 12:22:40.910516 17652 leader_election.cc:304] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:22:40.901814 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 57117.7us
I20260709 12:22:40.911785 17830 raft_consensus.cc:2468] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 13.
I20260709 12:22:40.913731 17762 raft_consensus.cc:2468] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 12.
W20260709 12:22:40.902316 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 54987.2us
I20260709 12:22:40.879388 18368 rpcz_store.cc:276] 0709 12:22:39.070266 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:40.410633 (+1340367us) service_pool.cc:224] Handling call
0709 12:22:40.464044 (+ 53411us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:40.464313 (+   269us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 12:22:40.620797 (+156484us) log.cc:844] Serialized 14 byte log entry
0709 12:22:40.848978 (+228181us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:40.849043 (+    65us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:22:40.849082 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:40.849130 (+    48us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:40.849217 (+    87us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":167656,"prepare.run_cpu_time_us":104,"prepare.run_wall_time_us":5477,"thread_start_us":276111,"threads_started":3,"wal-append.queue_time_us":205966}
I20260709 12:22:40.756640 22334 raft_consensus.cc:3055] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:22:40.915709 17689 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44898 (request call id 910) took 2093 ms. Trace:
I20260709 12:22:40.916101 17689 rpcz_store.cc:276] 0709 12:22:38.822122 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:40.845894 (+2023772us) service_pool.cc:224] Handling call
0709 12:22:40.846346 (+   452us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:40.898102 (+ 51756us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:40.915652 (+ 17550us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:22:40.915079 18334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:40.910694 17688 rpcz_store.cc:276] 0709 12:22:38.821064 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:40.839518 (+2018454us) service_pool.cc:224] Handling call
0709 12:22:40.840197 (+   679us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:40.846971 (+  6774us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:40.847024 (+    53us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:40.895224 (+ 48200us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:22:40.895290 (+    66us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:40.895373 (+    83us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:40.895526 (+   153us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:40.921787 17652 leader_election.cc:304] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:40.915386 22334 raft_consensus.cc:740] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:22:40.842887 22045 consensus_queue.cc:237] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 11.11, Last appended by leader: 11, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:40.925566 22334 consensus_queue.cc:260] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:40.926909 22334 raft_consensus.cc:3060] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:40.927922 22339 raft_consensus.cc:493] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:40.928520 22339 raft_consensus.cc:515] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:22:40.929239 22337 log.cc:927] Time spent T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.112s	user 0.000s	sys 0.000s
I20260709 12:22:40.929399 22335 raft_consensus.cc:2749] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:22:40.930295 22341 raft_consensus.cc:493] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:40.930922 22339 leader_election.cc:290] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:40.930845 22341 raft_consensus.cc:515] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:40.931555 22339 raft_consensus.cc:493] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
W20260709 12:22:40.915091 17802 connection.cc:256] RPC call timeout handler was delayed by -1.6215940459997s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2167s for a response
I20260709 12:22:40.939836 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:40.940217 22003 raft_consensus.cc:3060] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:22:40.940490 17830 raft_consensus.cc:2376] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 11. Re-sending same reply.
I20260709 12:22:40.940855 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:40.941471 17831 raft_consensus.cc:2376] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 12. Re-sending same reply.
I20260709 12:22:40.941588 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:40.942411 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:40.942657 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:40.943141 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:40.943344 17829 raft_consensus.cc:2468] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 12.
I20260709 12:22:40.943601 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 10 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:40.932830 18368 raft_consensus.cc:1275] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 9 index: 11. Preceding OpId from leader: term: 10 index: 11. (term mismatch)
I20260709 12:22:40.933586 22341 leader_election.cc:290] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:22:40.931938 22326 log.cc:927] Time spent T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.103s	user 0.001s	sys 0.001s
I20260709 12:22:40.944240 18368 pending_rounds.cc:85] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da: Aborting all ops after (but not including) 10
I20260709 12:22:40.944689 18368 pending_rounds.cc:107] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da: Aborting uncommitted NO_OP operation due to leader change: 9.11
I20260709 12:22:40.760957 22194 raft_consensus.cc:2749] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:22:40.933866 17835 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35040 (request call id 2409) took 1757 ms. Trace:
I20260709 12:22:40.936870 22184 raft_consensus.cc:515] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:22:40.931941 22294 log.cc:927] Time spent T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.091s	user 0.000s	sys 0.000s
I20260709 12:22:40.933904 22340 raft_consensus.cc:493] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:22:40.762647 22317 leader_election.cc:290] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:40.944940 18368 raft_consensus.cc:2889] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:40.817041 22134 raft_consensus.cc:697] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 11 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:22:40.940260 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:40.944203 17834 raft_consensus.cc:2468] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 9.
W20260709 12:22:40.951622 17803 leader_election.cc:336] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Timed out: RequestConsensusVote RPC to 127.16.29.68:42577 timed out after 1.789s (SENT)
I20260709 12:22:40.974594 17803 leader_election.cc:304] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
W20260709 12:22:40.976018 17729 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:40.972102 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 50454.4us
I20260709 12:22:40.843009 22330 raft_consensus.cc:2749] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:22:40.978068 17652 leader_election.cc:304] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:40.909230 22149 raft_consensus.cc:2804] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:22:40.980755 17688 raft_consensus.cc:1240] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 10 LEADER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 9. Current term is 10. Ops: [9.11-9.11]
I20260709 12:22:40.939152 22339 raft_consensus.cc:515] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:22:41.023564 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:41.149533 17688 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44898 (request call id 914) took 2326 ms. Trace:
I20260709 12:22:41.150024 17688 rpcz_store.cc:276] 0709 12:22:38.823387 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:40.952962 (+2129575us) service_pool.cc:224] Handling call
0709 12:22:40.980450 (+ 27488us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:41.149088 (+168638us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:41.149459 (+   371us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:40.951545 17835 rpcz_store.cc:276] 0709 12:22:39.176939 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:39.184453 (+  7514us) service_pool.cc:224] Handling call
0709 12:22:39.605730 (+421277us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:22:40.460895 (+855165us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:40.460937 (+    42us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:22:40.468124 (+  7187us) log.cc:844] Serialized 525 byte log entry
0709 12:22:40.642391 (+174267us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:22:40.642458 (+    67us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:40.642542 (+    84us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:40.933619 (+291077us) raft_consensus.cc:1685] finished
0709 12:22:40.933638 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:40.933794 (+   156us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":82644,"raft.run_cpu_time_us":117,"raft.run_wall_time_us":119,"thread_start_us":876256,"threads_started":2,"wal-append.queue_time_us":245636}
I20260709 12:22:40.945210 21888 raft_consensus.cc:2749] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:22:40.948745 22264 raft_consensus.cc:3060] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:22:41.189502 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:41.121320 17802 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.16.29.65:33071 timed out after 2.167s (SENT) [suppressed 37 similar messages]
W20260709 12:22:41.259104 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:40.975374 22341 raft_consensus.cc:2749] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:22:41.243697 22149 raft_consensus.cc:697] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
W20260709 12:22:41.267788 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 146527us
I20260709 12:22:41.267812 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:41.198252 22193 raft_consensus.cc:2749] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:22:41.149955 22192 raft_consensus.cc:493] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
W20260709 12:22:41.293244 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:41.264083 22201 consensus_meta.cc:287] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 1.216s	user 0.014s	sys 0.001s
I20260709 12:22:40.981173 22185 consensus_queue.cc:1048] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.288s
W20260709 12:22:41.325311 17654 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.669s	user 0.036s	sys 0.036s
W20260709 12:22:41.326228 17654 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.670s	user 0.036s	sys 0.036s
W20260709 12:22:41.327502 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 671916us
W20260709 12:22:41.329151 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 377546us
I20260709 12:22:41.329169 17654 leader_election.cc:304] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:41.330585 17654 leader_election.cc:304] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:22:41.331684 17728 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.699s	user 0.020s	sys 0.036s
W20260709 12:22:41.332224 17728 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.699s	user 0.020s	sys 0.036s
W20260709 12:22:41.332616 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 730678us
I20260709 12:22:41.339107 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
W20260709 12:22:41.324061 22338 log.cc:927] Time spent T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.376s	user 0.001s	sys 0.000s
I20260709 12:22:41.326237 21888 raft_consensus.cc:2804] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Leader pre-election won for term 12
W20260709 12:22:41.346854 18363 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.513s	user 0.047s	sys 0.050s
W20260709 12:22:41.349536 18363 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.522s	user 0.047s	sys 0.051s
I20260709 12:22:41.122604 22340 raft_consensus.cc:515] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:41.352372 17680 raft_consensus.cc:3060] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:41.351761 18363 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 1438) took 2258 ms. Trace:
W20260709 12:22:41.354101 18362 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.752s	user 0.018s	sys 0.045s
W20260709 12:22:41.354877 18362 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.754s	user 0.019s	sys 0.045s
W20260709 12:22:41.355912 17803 leader_election.cc:336] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 11 election: RPC error from VoteRequest() call to peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Timed out: RequestConsensusVote RPC to 127.16.29.68:42577 timed out after 1.804s (SENT)
I20260709 12:22:41.356977 18334 leader_election.cc:304] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:22:41.358433 22340 leader_election.cc:290] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:41.335754 22201 raft_consensus.cc:515] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:41.361809 22201 leader_election.cc:290] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 12 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:41.362571 22313 raft_consensus.cc:493] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
W20260709 12:22:41.363214 22332 log.cc:927] Time spent T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.642s	user 0.001s	sys 0.000s
W20260709 12:22:41.389050 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 411028us
I20260709 12:22:41.363183 22313 raft_consensus.cc:515] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:41.390010 17652 leader_election.cc:304] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:41.391424 17652 leader_election.cc:304] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:22:41.391598 22313 leader_election.cc:290] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:22:41.396610 18371 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.653s	user 0.014s	sys 0.047s
W20260709 12:22:41.397279 18371 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.660s	user 0.015s	sys 0.047s
I20260709 12:22:41.351142 21888 raft_consensus.cc:493] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:41.335462 22192 raft_consensus.cc:515] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:22:41.410570 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 64799.3us
I20260709 12:22:41.407387 18362 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 1437) took 2318 ms. Trace:
I20260709 12:22:41.354187 18363 rpcz_store.cc:276] 0709 12:22:39.093151 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:40.467659 (+1374508us) service_pool.cc:224] Handling call
0709 12:22:40.468012 (+   353us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:40.468247 (+   235us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 12:22:40.822467 (+354220us) log.cc:844] Serialized 14 byte log entry
0709 12:22:41.350289 (+527822us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:41.350433 (+   144us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:22:41.350500 (+    67us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:41.351526 (+  1026us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:41.351690 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":52034,"prepare.queue_time_us":191896,"prepare.run_cpu_time_us":55,"prepare.run_wall_time_us":3523,"raft.queue_time_us":89,"raft.run_cpu_time_us":135,"raft.run_wall_time_us":14155,"thread_start_us":804325,"threads_started":2}
I20260709 12:22:41.422397 22345 raft_consensus.cc:2804] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:22:41.422561 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:41.423269 22352 raft_consensus.cc:2804] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Leader pre-election won for term 11
W20260709 12:22:41.423395 22333 log.cc:927] Time spent T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.571s	user 0.001s	sys 0.000s
I20260709 12:22:41.423915 22344 raft_consensus.cc:2804] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260709 12:22:41.428753 22287 log.cc:927] Time spent T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 2.377s	user 0.004s	sys 0.050s
I20260709 12:22:41.432035 22149 consensus_queue.cc:237] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:22:41.444159 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 94010.7us
W20260709 12:22:41.404574 22322 log.cc:927] Time spent T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.643s	user 0.001s	sys 0.001s
I20260709 12:22:41.452190 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 11 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:41.465067 18364 raft_consensus.cc:2410] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 11 because replica has last-logged OpId of term: 11 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 7.
I20260709 12:22:41.453681 18363 raft_consensus.cc:3060] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:41.453891 18371 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 1447) took 1037 ms. Trace:
W20260709 12:22:41.458403 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 127875us
I20260709 12:22:41.410635 21888 raft_consensus.cc:3060] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:41.405125 22003 raft_consensus.cc:2804] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:22:41.467790 17654 leader_election.cc:304] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:22:41.450208 18362 rpcz_store.cc:276] 0709 12:22:39.088907 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:40.449614 (+1360707us) service_pool.cc:224] Handling call
0709 12:22:40.449931 (+   317us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:40.450178 (+   247us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 12:22:40.551138 (+100960us) log.cc:844] Serialized 14 byte log entry
0709 12:22:41.406861 (+855723us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:41.406945 (+    84us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:22:41.407023 (+    78us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:41.407126 (+   103us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:41.407304 (+   178us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":41109,"prepare.queue_time_us":138761,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":42,"raft.queue_time_us":57862,"raft.run_cpu_time_us":91,"raft.run_wall_time_us":92,"thread_start_us":841963,"threads_started":2,"wal-append.queue_time_us":856693}
I20260709 12:22:41.467444 18371 rpcz_store.cc:276] 0709 12:22:40.416089 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:40.659387 (+243298us) service_pool.cc:224] Handling call
0709 12:22:40.704512 (+ 45125us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:40.704735 (+   223us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 12:22:40.737367 (+ 32632us) log.cc:844] Serialized 14 byte log entry
0709 12:22:41.449783 (+712416us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:41.449850 (+    67us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:22:41.453610 (+  3760us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:41.453690 (+    80us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:41.453826 (+   136us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":55104,"prepare.queue_time_us":35098,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":49,"raft.queue_time_us":52119,"raft.run_cpu_time_us":131,"raft.run_wall_time_us":8994,"thread_start_us":660159,"threads_started":1}
W20260709 12:22:41.475483 17757 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.840s	user 0.008s	sys 0.016s
W20260709 12:22:41.477094 17757 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.844s	user 0.009s	sys 0.016s
I20260709 12:22:41.354123 22134 consensus_queue.cc:237] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.7, Last appended by leader: 7, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:41.477933 17757 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38776 (request call id 2201) took 1432 ms. Trace:
I20260709 12:22:41.462463 22352 raft_consensus.cc:493] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:41.460005 22344 raft_consensus.cc:493] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:22:41.492913 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:22:41.481503 22334 consensus_meta.cc:287] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.554s	user 0.008s	sys 0.004s
I20260709 12:22:41.496723 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:41.468050 22345 raft_consensus.cc:697] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 13 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:22:41.500299 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:22:41.497901 17803 leader_election.cc:336] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Timed out: RequestConsensusVote RPC to 127.16.29.68:42577 timed out after 1.909s (SENT)
I20260709 12:22:41.501116 18364 raft_consensus.cc:2468] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 6.
I20260709 12:22:41.509497 22334 consensus_queue.cc:1048] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 12:22:41.509754 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:41.509974 22345 consensus_queue.cc:237] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.10, Last appended by leader: 10, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:41.510952 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:41.512032 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:41.512645 18371 raft_consensus.cc:2368] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 12. Current term is 13.
I20260709 12:22:41.513317 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:41.514259 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:41.515257 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:22:41.515621 17803 leader_election.cc:336] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Timed out: RequestConsensusVote RPC to 127.16.29.68:42577 timed out after 1.941s (SENT)
I20260709 12:22:41.516083 17803 leader_election.cc:304] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:41.517055 22301 raft_consensus.cc:2749] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20260709 12:22:41.517563 17803 leader_election.cc:336] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Timed out: RequestConsensusVote RPC to 127.16.29.68:42577 timed out after 2.043s (SENT)
W20260709 12:22:41.518832 17803 leader_election.cc:336] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Timed out: RequestConsensusVote RPC to 127.16.29.68:42577 timed out after 2.107s (SENT)
I20260709 12:22:41.519176 17803 leader_election.cc:304] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:41.491564 22352 raft_consensus.cc:3060] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:41.520372 17836 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35042 (request call id 2145) took 2765 ms. Trace:
W20260709 12:22:41.520720 22311 consensus_meta.cc:287] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.766s	user 0.009s	sys 0.001s
I20260709 12:22:41.520951 22193 raft_consensus.cc:2749] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:22:41.522625 17838 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35040 (request call id 2408) took 2346 ms. Trace:
I20260709 12:22:41.520764 17836 rpcz_store.cc:276] 0709 12:22:38.755254 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:38.765356 (+ 10102us) service_pool.cc:224] Handling call
0709 12:22:38.765790 (+   434us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 12:22:38.812303 (+ 46513us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:38.812360 (+    57us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 12:22:38.813349 (+   989us) log.cc:844] Serialized 78 byte log entry
0709 12:22:39.004978 (+191629us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:22:39.005024 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:39.005096 (+    72us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:41.400001 (+2394905us) spinlock_profiling.cc:243] Waited 10.2 ms on lock 0x7b4400022b4c. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b47459c85 00007f7b4745a6a2 00007f7b4a1cc1db 00007f7b4a1c7aaf 00007f7b4a1bb86e 00007f7b4a1babb6 00007f7b4a1ba7d8 00007f7b4a1c4221 00007f7b4a1c41d9 00007f7b4a1c4169
0709 12:22:41.519936 (+119935us) raft_consensus.cc:1685] finished
0709 12:22:41.519956 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:41.520183 (+   227us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1513,"raft.run_cpu_time_us":160,"raft.run_wall_time_us":23668,"spinlock_wait_cycles":23525120,"thread_start_us":92566,"threads_started":1,"wal-append.queue_time_us":194546}
I20260709 12:22:41.523123 17838 rpcz_store.cc:276] 0709 12:22:39.176568 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:39.176704 (+   136us) service_pool.cc:224] Handling call
0709 12:22:39.177124 (+   420us) raft_consensus.cc:1471] Updating replica for 4 ops
0709 12:22:39.177712 (+   588us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:39.177741 (+    29us) raft_consensus.cc:1539] Triggering prepare for 4 ops
0709 12:22:39.182664 (+  4923us) log.cc:844] Serialized 104 byte log entry
0709 12:22:39.980410 (+797746us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:22:40.062725 (+ 82315us) log.cc:844] Serialized 14 byte log entry
0709 12:22:40.194298 (+131573us) log.cc:844] Serialized 14 byte log entry
0709 12:22:40.194470 (+   172us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:40.194575 (+   105us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:40.447795 (+253220us) spinlock_profiling.cc:243] Waited 4.35 ms on lock 0x7b18004146a0. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b47533522 00007f7b475357c1 00007f7b47535779 00007f7b47535709 00007f7b475356d1 00007f7b475349cd 0000000000388634 0000000000388479 00007f7b4751683a
0709 12:22:40.742458 (+294663us) spinlock_profiling.cc:243] Waited 11.5 ms on lock 0x7b18004146a0. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b475333f3 00007f7b475357c1 00007f7b47535779 00007f7b47535709 00007f7b475356d1 00007f7b475349cd 0000000000388634 0000000000388479 00007f7b4751683a
0709 12:22:41.522361 (+779903us) raft_consensus.cc:1685] finished
0709 12:22:41.522388 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:41.522559 (+   171us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":11527,"prepare.queue_time_us":229882,"prepare.run_cpu_time_us":2832,"prepare.run_wall_time_us":31389,"raft.queue_time_us":45803,"raft.run_cpu_time_us":290,"raft.run_wall_time_us":19355,"spinlock_wait_cycles":64846848,"thread_start_us":781806,"threads_started":1,"wal-append.queue_time_us":1051032}
I20260709 12:22:41.498615 22323 leader_election.cc:290] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:41.528822 22305 raft_consensus.cc:2749] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:22:41.526510 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:41.530578 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:22:41.534811 22307 consensus_meta.cc:287] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.784s	user 0.004s	sys 0.005s
I20260709 12:22:41.535247 22307 raft_consensus.cc:515] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:41.544430 22307 leader_election.cc:290] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 12 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:41.539413 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:41.540261 18362 raft_consensus.cc:2393] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 10: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:22:41.521281 22311 raft_consensus.cc:515] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:22:41.500349 22218 consensus_meta.cc:287] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.972s	user 0.010s	sys 0.003s
I20260709 12:22:41.531654 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:41.498812 22344 raft_consensus.cc:3060] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:41.555423 18374 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 1430) took 2582 ms. Trace:
I20260709 12:22:41.561734 18371 raft_consensus.cc:2468] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 10.
I20260709 12:22:41.508373 22339 leader_election.cc:290] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:41.562510 18374 rpcz_store.cc:276] 0709 12:22:38.973069 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:40.346179 (+1373110us) service_pool.cc:224] Handling call
0709 12:22:40.361427 (+ 15248us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:40.367442 (+  6015us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:40.367487 (+    45us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:40.391901 (+ 24414us) log.cc:844] Serialized 26 byte log entry
0709 12:22:40.608902 (+217001us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:22:40.608962 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:40.609054 (+    92us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:41.555146 (+946092us) raft_consensus.cc:1685] finished
0709 12:22:41.555173 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:41.555348 (+   175us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":57285,"raft.run_cpu_time_us":85,"raft.run_wall_time_us":86,"thread_start_us":193779,"threads_started":1,"wal-append.queue_time_us":299795}
I20260709 12:22:41.595448 18368 raft_consensus.cc:2468] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 10.
W20260709 12:22:41.596478 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 77693.4us
I20260709 12:22:41.480918 17757 rpcz_store.cc:276] 0709 12:22:40.045133 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:40.369112 (+323979us) service_pool.cc:224] Handling call
0709 12:22:40.382396 (+ 13284us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:40.384677 (+  2281us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 12:22:40.474081 (+ 89404us) log.cc:844] Serialized 14 byte log entry
0709 12:22:41.477479 (+1003398us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:41.477549 (+    70us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:22:41.477613 (+    64us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:41.477706 (+    93us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:41.477857 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":16067,"prepare.queue_time_us":150408,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":42,"raft.queue_time_us":83607,"raft.run_cpu_time_us":164,"raft.run_wall_time_us":21473,"spinlock_wait_cycles":9077632,"thread_start_us":1073548,"threads_started":3,"wal-append.queue_time_us":261503}
I20260709 12:22:41.600581 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 11 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:41.547947 22184 leader_election.cc:290] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 13 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:41.604257 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:41.605955 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:41.607048 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:41.607559 18362 raft_consensus.cc:2468] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 10.
I20260709 12:22:41.608337 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:41.609319 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:41.609809 18362 raft_consensus.cc:2468] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 7.
I20260709 12:22:41.610555 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:41.611219 18362 raft_consensus.cc:2368] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 10. Current term is 11.
W20260709 12:22:41.613420 17803 connection.cc:256] RPC call timeout handler was delayed by -1.0619591169997875s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22340000000000002s for a response
I20260709 12:22:41.560235 22355 raft_consensus.cc:2804] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:22:41.616387 22355 raft_consensus.cc:493] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:41.616730 22355 raft_consensus.cc:3060] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:41.619833 22311 leader_election.cc:290] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 8 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:41.628185 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:41.596208 22218 raft_consensus.cc:515] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:41.629453 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 12 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:41.630498 22218 leader_election.cc:290] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 13 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:41.631536 17682 raft_consensus.cc:2393] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 13: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:22:41.630514 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:41.633234 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:41.636049 18364 raft_consensus.cc:2393] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 11: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:22:41.637614 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 12 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:22:41.646188 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 253063us
W20260709 12:22:41.646499 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:41.647001 17652 leader_election.cc:304] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:41.647907 18362 raft_consensus.cc:2410] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 12 because replica has last-logged OpId of term: 11 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 8.
I20260709 12:22:41.648828 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:41.651073 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:41.654066 18368 raft_consensus.cc:2393] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 10: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:22:41.634150 22264 raft_consensus.cc:2749] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:22:41.657997 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:41.658828 17652 leader_election.cc:304] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:41.662050 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:22:41.771018 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:41.777848 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:41.788404 18334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:41.797992 18363 raft_consensus.cc:2468] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 12.
I20260709 12:22:41.801409 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:41.802523 18364 raft_consensus.cc:2393] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 10: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:22:41.804610 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:41.805335 18368 raft_consensus.cc:2468] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 7.
W20260709 12:22:41.807200 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 148398us
I20260709 12:22:41.811172 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:22:41.814009 17652 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:41.814675 17652 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:41.815263 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 345084us
I20260709 12:22:41.817267 18362 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 1450) took 1315 ms. Trace:
I20260709 12:22:41.817790 18362 rpcz_store.cc:276] 0709 12:22:40.501544 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:41.657972 (+1156428us) service_pool.cc:224] Handling call
0709 12:22:41.816599 (+158627us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:41.816859 (+   260us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:41.816896 (+    37us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:41.816952 (+    56us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:22:41.816998 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:41.817087 (+    89us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:41.817200 (+   113us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:22:41.823092 17803 connection.cc:256] RPC call timeout handler was delayed by -1.1285854140000993s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17350000000000002s for a response
I20260709 12:22:41.657629 22184 raft_consensus.cc:3060] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:41.827066 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:41.833695 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:41.840328 18362 raft_consensus.cc:1240] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 11. Current term is 12. Ops: []
I20260709 12:22:41.840819 22339 raft_consensus.cc:493] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:41.844293 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:41.879797 17834 raft_consensus.cc:3060] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:41.848906 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:41.850021 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:41.880854 17829 raft_consensus.cc:3060] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:41.851312 17831 raft_consensus.cc:1240] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 9. Current term is 10. Ops: [9.11-9.11]
W20260709 12:22:41.881366 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:41.851361 17682 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37396 (request call id 2268) took 2542 ms. Trace:
I20260709 12:22:41.852056 17688 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37396 (request call id 2260) took 2812 ms. Trace:
I20260709 12:22:41.882925 17682 rpcz_store.cc:276] 0709 12:22:39.309474 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:40.606168 (+1296694us) service_pool.cc:224] Handling call
0709 12:22:41.851167 (+1244999us) spinlock_profiling.cc:243] Waited 992 ms on lock 0x7b54002530d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:22:41.851303 (+   136us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2281669888}
I20260709 12:22:41.884693 17682 raft_consensus.cc:1275] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 9 index: 7. Preceding OpId from leader: term: 12 index: 8. (index mismatch)
I20260709 12:22:41.860198 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 11 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:22:41.859035 17727 connection.cc:256] RPC call timeout handler was delayed by -1.5265352510000412s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1521s for a response
I20260709 12:22:41.862712 17689 raft_consensus.cc:1240] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 11. Current term is 13. Ops: [11.10-11.10]
I20260709 12:22:41.869491 17759 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38786 (request call id 2894) took 2782 ms. Trace:
I20260709 12:22:41.810947 17838 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d"
dest_uuid: "e063541542a94969898868bf6eb5ce1b"
mode: GRACEFUL
new_leader_uuid: "334a4e1f59944544b3b3ff36616b57da"
 from {username='slave'} at 127.0.0.1:59820
I20260709 12:22:41.818061 22218 raft_consensus.cc:2804] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:22:41.883116 17688 rpcz_store.cc:276] 0709 12:22:39.039567 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:41.151049 (+2111482us) service_pool.cc:224] Handling call
0709 12:22:41.260466 (+109417us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:41.275684 (+ 15218us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:41.275734 (+    50us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:41.276233 (+   499us) log.cc:844] Serialized 26 byte log entry
0709 12:22:41.547588 (+271355us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:22:41.547650 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:41.547748 (+    98us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:41.851850 (+304102us) raft_consensus.cc:1685] finished
0709 12:22:41.851870 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:41.852003 (+   133us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":63553,"raft.run_cpu_time_us":120,"raft.run_wall_time_us":8313,"thread_start_us":201030,"threads_started":1,"wal-append.queue_time_us":480613}
I20260709 12:22:41.889510 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:41.890106 17830 raft_consensus.cc:2468] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 11.
W20260709 12:22:41.890743 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 50157.3us
I20260709 12:22:41.890924 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:41.891487 17830 raft_consensus.cc:2468] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 7.
I20260709 12:22:41.885525 17759 rpcz_store.cc:276] 0709 12:22:39.086745 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:39.086923 (+   178us) service_pool.cc:224] Handling call
0709 12:22:39.087298 (+   375us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:39.794006 (+706708us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:41.868885 (+2074879us) spinlock_profiling.cc:243] Waited 706 ms on lock 0x7b54003e58d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:22:41.869335 (+   450us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1623486208}
I20260709 12:22:41.886636 17838 raft_consensus.cc:606] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 8 LEADER]: Received request to transfer leadership to TS 334a4e1f59944544b3b3ff36616b57da
W20260709 12:22:41.864748 22336 log.cc:927] Time spent T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 1.038s	user 0.011s	sys 0.000s
I20260709 12:22:41.888446 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:41.893565 17831 raft_consensus.cc:2468] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 6.
I20260709 12:22:41.888949 17835 raft_consensus.cc:2468] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 10.
I20260709 12:22:41.894488 17654 leader_election.cc:304] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:22:41.896206 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:41.897284 17762 raft_consensus.cc:2410] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for term 11 because replica has last-logged OpId of term: 10 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 9.
I20260709 12:22:41.900846 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:41.901759 17830 raft_consensus.cc:2468] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 7.
W20260709 12:22:41.888000 17803 leader_election.cc:336] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 election: RPC error from VoteRequest() call to peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Timed out: RequestConsensusVote RPC to 127.16.29.68:42577 timed out after 2.234s (SENT)
I20260709 12:22:41.818744 22003 raft_consensus.cc:697] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 12 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:22:41.817446 22358 consensus_queue.cc:1048] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.088s
W20260709 12:22:41.957871 17727 connection.cc:256] RPC call timeout handler was delayed by -1.0959243980000792s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1512s for a response
W20260709 12:22:41.960232 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 55326.1us
I20260709 12:22:41.888499 22339 raft_consensus.cc:515] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:41.989369 22003 consensus_queue.cc:237] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 9, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:41.907061 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:22:41.992370 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 97986.3us
W20260709 12:22:41.897934 22078 consensus_meta.cc:287] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 1.307s	user 0.014s	sys 0.000s
I20260709 12:22:41.993052 17834 raft_consensus.cc:2468] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 12.
I20260709 12:22:41.993268 22078 raft_consensus.cc:515] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:41.994266 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:41.995364 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:41.886888 22323 raft_consensus.cc:493] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:41.996656 22359 raft_consensus.cc:2804] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:22:41.996479 22323 raft_consensus.cc:515] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:41.997390 22359 raft_consensus.cc:697] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 13 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:22:41.918800 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:41.998586 22359 consensus_queue.cc:237] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 11.9, Last appended by leader: 9, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:42.000516 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:42.001495 17831 raft_consensus.cc:2468] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 11.
I20260709 12:22:42.002490 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:42.003410 17831 raft_consensus.cc:2393] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 10: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:22:41.892827 17682 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37396 (request call id 2270) took 2127 ms. Trace:
I20260709 12:22:41.995952 17834 raft_consensus.cc:2468] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 7.
I20260709 12:22:42.004043 17682 rpcz_store.cc:276] 0709 12:22:39.758955 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:41.884095 (+2125140us) service_pool.cc:224] Handling call
0709 12:22:41.884400 (+   305us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:41.885699 (+  1299us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:41.885865 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:42.004668 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:42.005470 22078 leader_election.cc:290] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 9 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:42.008486 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:41.888578 22218 raft_consensus.cc:493] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:42.012602 22192 leader_election.cc:290] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:42.012997 22218 raft_consensus.cc:3060] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:42.015151 22023 raft_consensus.cc:1027] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 10 LEADER]: attempt to promote peer 334a4e1f59944544b3b3ff36616b57da: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20260709 12:22:42.010514 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:42.016522 17838 raft_consensus.cc:2393] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 12: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:22:42.017546 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:42.018893 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:22:42.011950 22363 log.cc:927] Time spent T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.050s	user 0.001s	sys 0.000s
I20260709 12:22:42.020980 17831 raft_consensus.cc:2468] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 11.
W20260709 12:22:42.027352 17727 connection.cc:256] RPC call timeout handler was delayed by -1.0194243979999555s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1597s for a response
I20260709 12:22:42.031510 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 11 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:42.029114 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 11 candidate_status { last_received { term: 9 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:42.035171 17835 raft_consensus.cc:2468] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 10.
I20260709 12:22:42.036399 17830 raft_consensus.cc:3060] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:42.038018 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:42.038849 17831 raft_consensus.cc:2468] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 11.
I20260709 12:22:41.992504 17836 raft_consensus.cc:2393] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 13: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:22:42.044878 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:42.061921 17762 raft_consensus.cc:2393] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 12: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
W20260709 12:22:41.956580 22079 consensus_meta.cc:287] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 1.434s	user 0.008s	sys 0.004s
I20260709 12:22:42.059350 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:42.065863 17838 raft_consensus.cc:2468] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 9.
I20260709 12:22:42.066709 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 12 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:42.068691 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:42.071347 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:42.063695 17757 raft_consensus.cc:1240] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 12. Current term is 13. Ops: []
W20260709 12:22:42.048992 22216 consensus_meta.cc:287] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 1.305s	user 0.014s	sys 0.000s
I20260709 12:22:42.072656 17835 raft_consensus.cc:3060] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:42.102212 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
W20260709 12:22:42.102396 17728 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.549s	user 0.073s	sys 0.138s
I20260709 12:22:42.103032 17685 raft_consensus.cc:2393] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 12: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
W20260709 12:22:42.103091 17728 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.549s	user 0.074s	sys 0.138s
W20260709 12:22:42.099479 17680 consensus_meta.cc:287] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.746s	user 0.006s	sys 0.001s
W20260709 12:22:42.103641 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 550191us
I20260709 12:22:42.096426 22079 raft_consensus.cc:515] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:42.095108 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:42.109001 22079 leader_election.cc:290] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 14 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:42.108747 22216 raft_consensus.cc:515] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:42.111382 22216 leader_election.cc:290] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 13 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:42.113426 17680 raft_consensus.cc:1275] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260709 12:22:41.906294 22355 raft_consensus.cc:515] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:42.058650 22365 raft_consensus.cc:493] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:42.095048 22290 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.598s	user 0.090s	sys 0.105s
W20260709 12:22:42.116626 22290 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.626s	user 0.091s	sys 0.105s
I20260709 12:22:42.117420 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:42.118155 17836 raft_consensus.cc:2376] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 12. Re-sending same reply.
I20260709 12:22:42.119238 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:42.120178 22341 raft_consensus.cc:2749] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:22:42.120572 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:42.120322 22045 consensus_queue.cc:1059] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:22:42.121420 17836 raft_consensus.cc:2410] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for term 11 because replica has last-logged OpId of term: 10 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 5.
I20260709 12:22:42.121191 22311 consensus_queue.cc:1059] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 12:22:42.122648 17836 raft_consensus.cc:1240] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 13 LEADER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 12. Current term is 13. Ops: []
I20260709 12:22:42.124306 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:42.125912 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:42.126099 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:42.126755 17763 raft_consensus.cc:2468] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 12.
I20260709 12:22:42.127272 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:42.127667 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:42.128058 17761 raft_consensus.cc:2468] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 8.
I20260709 12:22:42.128916 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:42.129521 17758 raft_consensus.cc:1240] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 9. Current term is 10. Ops: []
I20260709 12:22:42.131042 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:22:42.135098 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 247140us
I20260709 12:22:42.137403 17680 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37396 (request call id 2262) took 3060 ms. Trace:
I20260709 12:22:42.138377 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 11 candidate_status { last_received { term: 9 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:42.140501 17680 rpcz_store.cc:276] 0709 12:22:39.076864 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:41.337517 (+2260653us) service_pool.cc:224] Handling call
0709 12:22:41.352205 (+ 14688us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:42.115970 (+763765us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:42.116179 (+   209us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:22:42.144423 17803 leader_election.cc:336] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Timed out: RequestConsensusVote RPC to 127.16.29.68:42577 timed out after 2.402s (SENT)
I20260709 12:22:42.146781 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 11 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:42.147526 17685 raft_consensus.cc:2376] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 11. Re-sending same reply.
I20260709 12:22:42.149681 17829 raft_consensus.cc:2468] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 13.
W20260709 12:22:42.134130 22306 consensus_meta.cc:287] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 1.658s	user 0.006s	sys 0.007s
W20260709 12:22:42.142728 22344 consensus_meta.cc:287] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.539s	user 0.007s	sys 0.002s
I20260709 12:22:42.155589 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:42.156206 17685 raft_consensus.cc:2376] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 10. Re-sending same reply.
I20260709 12:22:42.154634 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:42.154831 17830 raft_consensus.cc:2468] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 11.
I20260709 12:22:42.161217 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:42.162288 17680 raft_consensus.cc:2393] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 13: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:22:42.149283 22365 raft_consensus.cc:515] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:42.115587 22360 consensus_queue.cc:1048] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.056s
I20260709 12:22:42.155421 22306 raft_consensus.cc:2749] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
W20260709 12:22:42.174327 17803 connection.cc:256] RPC call timeout handler was delayed by -1.0412854139999581s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18320000000000003s for a response
I20260709 12:22:42.186375 17835 raft_consensus.cc:2468] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 13.
I20260709 12:22:42.194067 17688 raft_consensus.cc:2376] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 12. Re-sending same reply.
I20260709 12:22:42.161190 22357 consensus_queue.cc:1048] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.028s
I20260709 12:22:42.160812 22344 raft_consensus.cc:515] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:42.167783 22345 raft_consensus.cc:493] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:42.232551 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:42.308295 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:42.235498 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:42.241256 22355 leader_election.cc:290] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 14 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:42.242055 22365 leader_election.cc:290] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:22:42.245896 22367 log.cc:927] Time spent T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.071s	user 0.000s	sys 0.000s
I20260709 12:22:42.326031 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:42.327446 18364 raft_consensus.cc:2468] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 8.
W20260709 12:22:42.325553 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 204756us
W20260709 12:22:42.254355 22351 log.cc:927] Time spent T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.756s	user 0.001s	sys 0.000s
W20260709 12:22:42.256457 18365 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59918 (request call id 1441) took 2544 ms (client timeout 1830 ms). Trace:
W20260709 12:22:42.332515 18365 rpcz_store.cc:269] 0709 12:22:39.711709 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:39.856286 (+144577us) service_pool.cc:224] Handling call
0709 12:22:42.256188 (+2399902us) spinlock_profiling.cc:243] Waited 72.1 ms on lock 0x7b54006e23d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:22:42.256330 (+   142us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":165864704}
W20260709 12:22:42.258729 17730 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:42.334218 17730 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:42.334120 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:42.335026 18364 raft_consensus.cc:3060] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:42.264451 22339 leader_election.cc:290] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:42.270426 22165 leader_election.cc:290] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:42.273121 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:42.274145 17683 raft_consensus.cc:2393] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 12: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:22:42.276185 22290 consensus_queue.cc:1059] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 11.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:22:42.284015 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
W20260709 12:22:42.226727 21888 consensus_meta.cc:287] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.708s	user 0.012s	sys 0.000s
I20260709 12:22:42.285396 17681 raft_consensus.cc:1275] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 11 index: 6. Preceding OpId from leader: term: 10 index: 6. (term mismatch)
I20260709 12:22:42.290096 22344 leader_election.cc:290] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 7 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:42.291926 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:42.355216 17682 raft_consensus.cc:2410] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for term 10 because replica has last-logged OpId of term: 9 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 8.
W20260709 12:22:42.292366 18367 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59918 (request call id 1433) took 3265 ms (client timeout 2856 ms). Trace:
W20260709 12:22:42.355736 18367 rpcz_store.cc:269] 0709 12:22:39.028051 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:39.950681 (+922630us) service_pool.cc:224] Handling call
0709 12:22:42.292113 (+2341432us) spinlock_profiling.cc:243] Waited 106 ms on lock 0x7b54007602d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:22:42.292274 (+   161us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":243656960}
I20260709 12:22:42.357324 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:42.358103 18367 raft_consensus.cc:3060] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:42.299717 22078 consensus_queue.cc:1048] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.038s
W20260709 12:22:42.298710 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 62459.6us
I20260709 12:22:42.368880 17755 raft_consensus.cc:1275] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 11 index: 6. Preceding OpId from leader: term: 12 index: 7. (index mismatch)
W20260709 12:22:42.310717 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:42.311406 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 71544.6us
I20260709 12:22:42.312987 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:42.247491 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 12 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:42.329880 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:42.331149 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
W20260709 12:22:42.261023 22366 log.cc:927] Time spent T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.075s	user 0.000s	sys 0.000s
I20260709 12:22:42.390579 18363 raft_consensus.cc:3060] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:42.371467 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:42.366127 18333 leader_election.cc:304] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
W20260709 12:22:42.229133 22352 consensus_meta.cc:287] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.709s	user 0.011s	sys 0.000s
I20260709 12:22:42.353915 17681 pending_rounds.cc:85] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3: Aborting all ops after (but not including) 5
I20260709 12:22:42.396366 17681 pending_rounds.cc:107] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3: Aborting uncommitted NO_OP operation due to leader change: 11.6
I20260709 12:22:42.396687 17681 raft_consensus.cc:2889] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:42.367810 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:42.367581 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 11 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:42.397514 17686 raft_consensus.cc:2393] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 12: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:22:42.397588 18365 raft_consensus.cc:2468] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 10.
I20260709 12:22:42.389652 18374 raft_consensus.cc:3060] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:42.399349 18364 raft_consensus.cc:2410] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for term 13 because replica has last-logged OpId of term: 11 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 9.
I20260709 12:22:42.389946 18368 raft_consensus.cc:2393] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 13: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:22:42.393653 18333 leader_election.cc:304] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:22:42.346380 22017 raft_consensus.cc:493] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:22:42.345546 22365 raft_consensus.cc:493] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:42.395330 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 11 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:42.408034 17680 raft_consensus.cc:2393] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 11: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:22:42.395550 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:42.395339 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:22:42.396733 17803 leader_election.cc:336] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Timed out: RequestConsensusVote RPC to 127.16.29.68:42577 timed out after 2.518s (SENT)
I20260709 12:22:42.347380 22335 raft_consensus.cc:493] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:42.401473 22165 raft_consensus.cc:2804] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:22:42.426955 22365 raft_consensus.cc:515] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:42.409003 22017 raft_consensus.cc:515] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:42.428493 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:42.429194 18365 raft_consensus.cc:2393] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 12: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:22:42.368150 21888 raft_consensus.cc:515] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:42.402182 17686 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37402 (request call id 2634) took 3703 ms. Trace:
I20260709 12:22:42.405893 17764 raft_consensus.cc:1240] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 9. Current term is 10. Ops: [9.9-9.9]
I20260709 12:22:42.326224 22345 raft_consensus.cc:515] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:42.402315 17681 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37402 (request call id 2623) took 4138 ms. Trace:
I20260709 12:22:42.440569 17681 rpcz_store.cc:276] 0709 12:22:38.263663 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:42.284783 (+4021120us) service_pool.cc:224] Handling call
0709 12:22:42.285084 (+   301us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:42.402096 (+117012us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:42.402266 (+   170us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:42.410305 17803 leader_election.cc:304] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:42.412424 22335 raft_consensus.cc:515] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:42.369464 22339 consensus_queue.cc:1048] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.044s
I20260709 12:22:42.413285 17763 raft_consensus.cc:3055] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:22:42.442936 17763 raft_consensus.cc:740] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:22:42.413841 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:42.414160 22133 raft_consensus.cc:3060] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:42.414664 17654 leader_election.cc:304] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:22:42.415745 22149 raft_consensus.cc:3055] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:22:42.420786 22360 raft_consensus.cc:3055] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:22:42.422132 22131 raft_consensus.cc:3055] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:22:42.424005 22218 raft_consensus.cc:515] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:42.394484 22346 raft_consensus.cc:2804] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:22:42.425278 22306 consensus_queue.cc:1048] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 12:22:42.403841 22352 raft_consensus.cc:515] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:42.390625 22045 consensus_queue.cc:1048] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.037s
I20260709 12:22:42.428046 22181 consensus_queue.cc:1048] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 12:22:42.432337 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:42.353888 22311 raft_consensus.cc:493] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:42.400529 18333 leader_election.cc:304] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:22:42.435122 17686 rpcz_store.cc:276] 0709 12:22:38.698407 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:42.247092 (+3548685us) service_pool.cc:224] Handling call
0709 12:22:42.402127 (+155035us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:22:42.457986 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 57570.9us
I20260709 12:22:42.458880 18333 leader_election.cc:304] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:22:42.436497 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:42.460268 17761 raft_consensus.cc:3060] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:42.436578 22190 consensus_queue.cc:1048] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:22:42.437829 22365 leader_election.cc:290] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:42.427652 22165 raft_consensus.cc:493] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:42.480499 17682 raft_consensus.cc:2393] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 10: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:22:42.350354 22192 consensus_queue.cc:1059] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.081s
I20260709 12:22:42.443868 17763 consensus_queue.cc:260] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.7, Last appended by leader: 7, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:42.452369 17758 raft_consensus.cc:1240] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 10. Current term is 11. Ops: []
I20260709 12:22:42.453538 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:42.483606 17680 raft_consensus.cc:2368] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 12. Current term is 13.
I20260709 12:22:42.473048 18374 raft_consensus.cc:2468] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 12.
I20260709 12:22:42.449875 22149 raft_consensus.cc:740] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 7 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
W20260709 12:22:42.476302 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:42.450107 22360 raft_consensus.cc:740] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 12 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:22:42.502161 22360 consensus_queue.cc:260] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.10, Last appended by leader: 10, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:42.450843 22131 raft_consensus.cc:740] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 11 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:22:42.504844 22131 consensus_queue.cc:260] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:42.482517 17685 raft_consensus.cc:2393] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 11: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:22:42.485081 18363 raft_consensus.cc:2468] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 8.
I20260709 12:22:42.451781 22042 raft_consensus.cc:2804] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:22:42.507730 18363 raft_consensus.cc:1240] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 11. Current term is 12. Ops: []
I20260709 12:22:42.508075 22042 raft_consensus.cc:493] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:42.508530 22042 raft_consensus.cc:3060] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:42.509127 22335 leader_election.cc:290] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:42.509686 22311 raft_consensus.cc:515] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:42.487479 17763 raft_consensus.cc:3060] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:22:42.492198 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 95499.4us
W20260709 12:22:42.516497 17803 leader_election.cc:336] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Timed out: RequestConsensusVote RPC to 127.16.29.68:42577 timed out after 3.038s (SENT)
I20260709 12:22:42.516930 17803 leader_election.cc:304] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:42.517521 21888 leader_election.cc:290] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 12 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:22:42.518110 17803 leader_election.cc:336] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Timed out: RequestConsensusVote RPC to 127.16.29.68:42577 timed out after 3.248s (SENT)
I20260709 12:22:42.518539 17803 leader_election.cc:304] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:42.519323 22290 raft_consensus.cc:2749] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
W20260709 12:22:42.519526 17803 leader_election.cc:336] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Timed out: RequestConsensusVote RPC to 127.16.29.68:42577 timed out after 3.271s (SENT)
I20260709 12:22:42.492842 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:42.520059 22290 raft_consensus.cc:2749] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:22:42.520282 17686 raft_consensus.cc:2468] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 9.
I20260709 12:22:42.521296 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:42.521219 22149 consensus_queue.cc:260] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:42.500289 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 13 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:42.522553 22149 raft_consensus.cc:3060] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:42.501092 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:42.524117 18366 raft_consensus.cc:2368] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for earlier term 7. Current term is 8.
I20260709 12:22:42.525323 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 13 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:22:42.525892 22184 consensus_meta.cc:287] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.689s	user 0.010s	sys 0.000s
I20260709 12:22:42.526080 17685 raft_consensus.cc:2393] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 13: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:22:42.526292 22184 raft_consensus.cc:2749] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:22:42.526998 17687 raft_consensus.cc:1240] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 12. Current term is 13. Ops: []
I20260709 12:22:42.527501 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:42.528045 22352 leader_election.cc:290] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 11 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:42.529577 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:42.530220 18366 raft_consensus.cc:2468] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 10.
I20260709 12:22:42.530516 17687 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37396 (request call id 2267) took 3302 ms. Trace:
I20260709 12:22:42.531013 17687 rpcz_store.cc:276] 0709 12:22:39.227920 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:41.855560 (+2627640us) service_pool.cc:224] Handling call
0709 12:22:41.855850 (+   290us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:42.527492 (+671642us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:42.530200 (+  2708us) spinlock_profiling.cc:243] Waited 671 ms on lock 0x7b54000c6054. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:22:42.530442 (+   242us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1542946560}
I20260709 12:22:42.532297 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:42.532882 17756 raft_consensus.cc:3060] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:42.532929 22131 raft_consensus.cc:3060] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:42.527020 17685 raft_consensus.cc:1240] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 10. Current term is 11. Ops: []
I20260709 12:22:42.534852 17685 raft_consensus.cc:1240] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 12. Current term is 13. Ops: []
I20260709 12:22:42.535869 17685 raft_consensus.cc:1240] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 8. Current term is 9. Ops: []
I20260709 12:22:42.537166 17685 raft_consensus.cc:1240] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 12. Current term is 13. Ops: []
I20260709 12:22:42.452147 22346 raft_consensus.cc:493] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:42.541239 18367 raft_consensus.cc:2468] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 9.
W20260709 12:22:42.546936 18334 leader_election.cc:336] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Timed out: RequestConsensusVote RPC to 127.16.29.65:33071 timed out after 1.504s (SENT)
I20260709 12:22:42.521948 17686 raft_consensus.cc:2468] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 8.
I20260709 12:22:42.545667 17763 raft_consensus.cc:1275] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 10 index: 7. Preceding OpId from leader: term: 11 index: 8. (index mismatch)
I20260709 12:22:42.547931 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 12 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:42.550034 18367 raft_consensus.cc:3060] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:42.551604 22017 leader_election.cc:290] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:42.546609 22346 raft_consensus.cc:3060] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:42.552306 22045 raft_consensus.cc:493] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:42.552840 22045 raft_consensus.cc:515] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:42.554827 22045 leader_election.cc:290] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:42.555110 22339 raft_consensus.cc:2804] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:22:42.555527 22339 raft_consensus.cc:493] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:42.483647 22165 raft_consensus.cc:3060] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:42.555948 22339 raft_consensus.cc:3060] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:42.555538 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:42.481868 22134 raft_consensus.cc:2749] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:22:42.503737 22360 raft_consensus.cc:3060] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:42.566453 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:42.567145 18371 raft_consensus.cc:3060] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:42.559940 17761 raft_consensus.cc:2468] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 13.
I20260709 12:22:42.564615 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 14 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:42.569006 17680 raft_consensus.cc:2393] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 14: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:22:42.511754 22311 leader_election.cc:290] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:42.576349 17761 raft_consensus.cc:1240] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 11. Current term is 12. Ops: []
I20260709 12:22:42.585835 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 13 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:42.586478 17684 raft_consensus.cc:1240] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 12. Current term is 13. Ops: []
I20260709 12:22:42.586715 17686 raft_consensus.cc:2468] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 12.
I20260709 12:22:42.571683 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:42.588797 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:42.589421 17688 raft_consensus.cc:2468] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 7.
I20260709 12:22:42.591197 17680 raft_consensus.cc:1240] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 13. Current term is 14. Ops: []
I20260709 12:22:42.593087 17684 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37396 (request call id 2252) took 4488 ms. Trace:
I20260709 12:22:42.593570 17684 rpcz_store.cc:276] 0709 12:22:38.098666 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:42.583022 (+4484356us) service_pool.cc:224] Handling call
0709 12:22:42.586224 (+  3202us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:42.587075 (+   851us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:42.587311 (+   236us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:22:42.595075 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 75545.4us
I20260709 12:22:42.604790 22042 raft_consensus.cc:515] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:42.605382 17803 leader_election.cc:304] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:42.608731 22358 raft_consensus.cc:2804] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:22:42.609125 22358 raft_consensus.cc:493] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:42.609508 22358 raft_consensus.cc:3060] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:42.609658 22290 raft_consensus.cc:493] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:42.610157 22290 raft_consensus.cc:515] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:42.612036 22306 consensus_queue.cc:1048] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 12:22:42.614035 22290 leader_election.cc:290] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:42.614437 17688 raft_consensus.cc:1275] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 10 index: 7. Preceding OpId from leader: term: 11 index: 8. (index mismatch)
I20260709 12:22:42.626128 17756 raft_consensus.cc:2468] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 12.
I20260709 12:22:42.628566 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:42.697093 18367 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 12.
W20260709 12:22:42.742377 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:42.768383 17728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:42.774386 21994 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 2.914s	user 0.036s	sys 0.041s
W20260709 12:22:42.774988 21994 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 2.915s	user 0.036s	sys 0.041s
W20260709 12:22:42.776564 17652 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.644s	user 0.033s	sys 0.060s
W20260709 12:22:42.776883 17652 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.645s	user 0.033s	sys 0.060s
W20260709 12:22:42.777196 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 671960us
W20260709 12:22:42.782150 17728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:42.784165 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:42.769734 17729 leader_election.cc:304] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:42.785389 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:22:42.785527 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:42.785164 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:42.798573 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 13 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:42.759028 22374 raft_consensus.cc:2749] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:22:42.803493 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:42.804324 17684 raft_consensus.cc:3060] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:42.801362 17682 raft_consensus.cc:1240] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 13. Current term is 14. Ops: []
I20260709 12:22:42.802660 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:42.806557 17833 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b"
dest_uuid: "e063541542a94969898868bf6eb5ce1b"
mode: GRACEFUL
new_leader_uuid: "334a4e1f59944544b3b3ff36616b57da"
 from {username='slave'} at 127.0.0.1:59820
I20260709 12:22:42.807430 17833 raft_consensus.cc:606] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Received request to transfer leadership to TS 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:42.810185 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:42.810985 17834 raft_consensus.cc:3060] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:42.813392 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:42.814267 17831 raft_consensus.cc:3060] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:42.817636 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:42.818460 17835 raft_consensus.cc:2468] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 9.
I20260709 12:22:42.822885 17833 raft_consensus.cc:614] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:22:42.822907 17652 leader_election.cc:304] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:42.840293 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:42.841182 17829 raft_consensus.cc:3060] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:42.763036 22345 leader_election.cc:290] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:42.763044 22341 raft_consensus.cc:493] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:42.861033 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:42.862187 17682 raft_consensus.cc:2410] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for term 10 because replica has last-logged OpId of term: 9 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 8.
I20260709 12:22:42.855916 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:42.856618 17685 raft_consensus.cc:2393] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 12: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
W20260709 12:22:42.857613 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 87948.2us
I20260709 12:22:42.872131 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:42.872498 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 12 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:42.873068 17837 raft_consensus.cc:2468] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 9.
I20260709 12:22:42.873658 17835 raft_consensus.cc:3060] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:42.872208 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:42.874505 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:42.875030 17830 raft_consensus.cc:3060] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Advancing to term 11
I20260709 12:22:42.882493 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:42.885051 18361 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 1436) took 3812 ms. Trace:
W20260709 12:22:42.886385 17683 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37396 (request call id 2261) took 3819 ms (client timeout 2272 ms). Trace:
W20260709 12:22:42.886936 22369 log.cc:927] Time spent T 00000000000000000000000000000000 P 2193af31cb894ec9b3d65d939a2a6851: Append to log took a long time: real 0.120s	user 0.000s	sys 0.000s
I20260709 12:22:42.892803 22042 leader_election.cc:290] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 11 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:42.848850 21994 raft_consensus.cc:493] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:42.893898 21994 raft_consensus.cc:515] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:42.897125 17802 leader_election.cc:304] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:42.898627 17802 leader_election.cc:400] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 12 election: Vote denied by peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071) with higher term. Message: Invalid argument: T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 12. Current term is 13.
I20260709 12:22:42.899426 17802 leader_election.cc:304] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:42.900079 22306 consensus_queue.cc:1048] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 12:22:42.902032 22306 raft_consensus.cc:2749] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:22:42.902740 22317 raft_consensus.cc:493] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
W20260709 12:22:42.886978 17683 rpcz_store.cc:269] 0709 12:22:39.066633 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:40.476537 (+1409904us) service_pool.cc:224] Handling call
0709 12:22:42.886125 (+2409588us) spinlock_profiling.cc:243] Waited 1.8 s on lock 0x7b54000c5654. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:22:42.886268 (+   143us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4131661056}
I20260709 12:22:42.903738 22317 raft_consensus.cc:515] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:42.902834 18361 rpcz_store.cc:276] 0709 12:22:39.072331 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:40.445234 (+1372903us) service_pool.cc:224] Handling call
0709 12:22:40.445618 (+   384us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:40.446075 (+   457us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:40.446101 (+    26us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:40.446645 (+   544us) log.cc:844] Serialized 26 byte log entry
0709 12:22:42.884416 (+2437771us) spinlock_profiling.cc:243] Waited 13.4 ms on lock 0x7b1800411b20. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b4a1c84bc 00007f7b4751648b 00007f7b503709a9 00007f7b4753269b 00007f7b4753093f 00007f7b47532c02 00007f7b4a1ba20b 00007f7b4a1bfd23 00007f7b4a1c00d1
0709 12:22:42.884593 (+   177us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:22:42.884633 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:42.884713 (+    80us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:42.884756 (+    43us) raft_consensus.cc:1685] finished
0709 12:22:42.884768 (+    12us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:42.884991 (+   223us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":34136,"raft.queue_time_us":86485,"raft.run_cpu_time_us":138,"raft.run_wall_time_us":17959,"spinlock_wait_cycles":30902016,"thread_start_us":106274,"threads_started":1,"wal-append.queue_time_us":188877,"wal-append.run_cpu_time_us":3640,"wal-append.run_wall_time_us":1616284}
I20260709 12:22:42.908085 22129 raft_consensus.cc:2804] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:22:42.908546 22129 raft_consensus.cc:493] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:42.908942 22129 raft_consensus.cc:3060] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:42.910846 17802 leader_election.cc:304] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:42.912037 22306 raft_consensus.cc:2749] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:22:42.913780 22306 consensus_queue.cc:1059] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 12:22:42.914897 22114 raft_consensus.cc:3055] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:22:42.915334 22114 raft_consensus.cc:740] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 8 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:22:42.915642 22306 consensus_queue.cc:1059] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:22:42.917184 17802 leader_election.cc:304] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:22:42.918475 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 49288.5us
I20260709 12:22:42.919050 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:42.920715 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:42.921619 17756 raft_consensus.cc:2368] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for earlier term 7. Current term is 8.
I20260709 12:22:42.926833 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:42.939851 17761 raft_consensus.cc:1240] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 11. Current term is 12. Ops: []
I20260709 12:22:42.945878 22335 raft_consensus.cc:2955] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 10 LEADER]: Committing config change with OpId 8.5: config changed from index -1 to 5, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } }
I20260709 12:22:42.877676 22069 raft_consensus.cc:2804] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:22:42.953548 22069 raft_consensus.cc:697] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:22:42.954507 22069 consensus_queue.cc:237] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:42.956336 17832 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343"
dest_uuid: "e063541542a94969898868bf6eb5ce1b"
mode: GRACEFUL
new_leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
 from {username='slave'} at 127.0.0.1:59820
I20260709 12:22:42.956961 17832 raft_consensus.cc:606] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Received request to transfer leadership to TS e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:42.957254 17832 raft_consensus.cc:614] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260709 12:22:42.957314 17728 leader_election.cc:336] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 16 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Timed out: RequestConsensusVote RPC to 127.16.29.65:33071 timed out after 1.560s (SENT)
I20260709 12:22:42.960024 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:42.961738 17682 raft_consensus.cc:2468] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 12.
I20260709 12:22:42.962393 17759 raft_consensus.cc:2468] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 13.
I20260709 12:22:42.969587 17689 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44898 (request call id 912) took 4146 ms. Trace:
I20260709 12:22:42.970108 17689 rpcz_store.cc:276] 0709 12:22:38.822737 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:40.917320 (+2094583us) service_pool.cc:224] Handling call
0709 12:22:41.862375 (+945055us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:41.885424 (+ 23049us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:42.969325 (+1083901us) spinlock_profiling.cc:243] Waited 929 ms on lock 0x7b54002530d0. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a61f8ad 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:22:42.969509 (+   184us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2136277248}
I20260709 12:22:42.969014 17729 leader_election.cc:304] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:22:42.981001 17762 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.561s	user 0.048s	sys 0.065s
W20260709 12:22:42.981771 17762 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.562s	user 0.048s	sys 0.065s
I20260709 12:22:42.947326 22165 raft_consensus.cc:515] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:42.911643 22346 raft_consensus.cc:515] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:42.970803 22356 raft_consensus.cc:493] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:42.987735 17727 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.517s	user 0.009s	sys 0.009s
I20260709 12:22:42.988257 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:22:43.015581 17727 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.545s	user 0.009s	sys 0.009s
I20260709 12:22:42.916226 22114 consensus_queue.cc:260] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.10, Last appended by leader: 10, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:22:42.994232 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:42.995503 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 124539us
I20260709 12:22:43.000793 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 12 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
W20260709 12:22:43.058049 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:43.058395 17761 raft_consensus.cc:3060] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:43.007416 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:43.057884 17652 leader_election.cc:304] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:22:43.059283 17681 raft_consensus.cc:2393] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 13: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:22:43.008960 17835 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 12.
I20260709 12:22:43.060922 17652 leader_election.cc:304] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
W20260709 12:22:43.009596 17760 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.611s	user 0.007s	sys 0.036s
W20260709 12:22:43.061946 17760 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.667s	user 0.007s	sys 0.036s
I20260709 12:22:42.986433 17834 raft_consensus.cc:2468] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 14.
I20260709 12:22:42.989143 17829 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 10.
I20260709 12:22:42.978159 22131 raft_consensus.cc:493] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:43.040004 22301 raft_consensus.cc:493] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:43.024602 18371 raft_consensus.cc:2468] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 11.
I20260709 12:22:42.879233 22341 raft_consensus.cc:515] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:43.026546 17684 raft_consensus.cc:2468] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 8.
W20260709 12:22:43.059263 22359 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.564s	user 0.001s	sys 0.014s
I20260709 12:22:42.914855 22339 raft_consensus.cc:515] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:42.925532 22374 consensus_queue.cc:1059] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 9.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.093s
I20260709 12:22:43.004683 22311 raft_consensus.cc:2804] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:22:42.979218 22134 consensus_queue.cc:1048] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.055s
W20260709 12:22:43.040819 17687 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.507s	user 0.005s	sys 0.016s
I20260709 12:22:43.055177 17831 raft_consensus.cc:2468] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 14.
W20260709 12:22:43.055814 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 604234us
W20260709 12:22:43.056203 18363 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.546s	user 0.006s	sys 0.009s
W20260709 12:22:43.056430 18362 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.647s	user 0.010s	sys 0.033s
I20260709 12:22:42.985610 17830 raft_consensus.cc:2468] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 11.
W20260709 12:22:43.114964 17687 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.581s	user 0.006s	sys 0.016s
W20260709 12:22:43.066679 18334 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.501s	user 0.051s	sys 0.128s
W20260709 12:22:43.116822 18334 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.551s	user 0.051s	sys 0.128s
W20260709 12:22:43.117167 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 551609us
W20260709 12:22:43.058024 22192 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.507s	user 0.011s	sys 0.014s
W20260709 12:22:43.071827 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:43.090459 22359 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.596s	user 0.001s	sys 0.014s
I20260709 12:22:43.086426 22114 raft_consensus.cc:3060] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:43.027318 22356 raft_consensus.cc:515] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:43.086827 17729 leader_election.cc:304] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:43.061556 22334 raft_consensus.cc:493] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:43.060318 18333 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.574s	user 0.001s	sys 0.024s
W20260709 12:22:43.127602 18333 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.642s	user 0.001s	sys 0.024s
W20260709 12:22:43.128196 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 669369us
I20260709 12:22:43.129164 18333 leader_election.cc:304] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
W20260709 12:22:43.115640 18363 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.606s	user 0.006s	sys 0.009s
W20260709 12:22:43.115759 18362 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.706s	user 0.010s	sys 0.033s
I20260709 12:22:43.032301 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:43.132458 17756 raft_consensus.cc:3060] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:22:43.108696 22385 log.cc:927] Time spent T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.334s	user 0.000s	sys 0.005s
I20260709 12:22:43.087175 22290 raft_consensus.cc:2749] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
W20260709 12:22:43.119032 22192 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.568s	user 0.011s	sys 0.014s
I20260709 12:22:43.133926 18361 raft_consensus.cc:1275] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 11 index: 9. (index mismatch)
I20260709 12:22:43.136883 17727 leader_election.cc:304] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:22:43.142262 17762 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36572 (request call id 802) took 1019 ms. Trace:
I20260709 12:22:43.130263 18333 leader_election.cc:304] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:22:43.147327 18333 leader_election.cc:304] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:22:43.142764 17762 rpcz_store.cc:276] 0709 12:22:42.122571 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:42.125565 (+  2994us) service_pool.cc:224] Handling call
0709 12:22:42.418068 (+292503us) spinlock_profiling.cc:243] Waited 6.3 ms on lock 0x7b5800033398. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b474fbb71 00007f7b474fba22 00007f7b504148d9 00007f7b4f56eee0 00007f7b4f6380ab 00007f7b4f638131 00007f7b4f5f9fc0 00007f7b4f5e4929 00007f7b4f5e43d2 00007f7b48b5bbd4
0709 12:22:42.418268 (+   200us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:42.418708 (+   440us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:42.418740 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:42.419261 (+   521us) log.cc:844] Serialized 26 byte log entry
0709 12:22:42.982263 (+563002us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:22:42.982300 (+    37us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:42.982364 (+    64us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:43.141912 (+159548us) raft_consensus.cc:1685] finished
0709 12:22:43.141934 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:43.142191 (+   257us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":2818,"raft.run_cpu_time_us":91,"raft.run_wall_time_us":92,"spinlock_wait_cycles":14481280,"thread_start_us":562623,"threads_started":1,"wal-append.queue_time_us":720257}
W20260709 12:22:43.121376 22378 log.cc:927] Time spent T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.083s	user 0.001s	sys 0.001s
W20260709 12:22:43.149931 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 63225.9us
I20260709 12:22:43.153365 17729 leader_election.cc:304] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:43.112815 22311 raft_consensus.cc:493] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:22:43.129173 17761 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 12.
I20260709 12:22:43.086858 22131 raft_consensus.cc:515] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:22:43.162056 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:43.167436 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 45864.9us
W20260709 12:22:43.192814 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 56018.2us
W20260709 12:22:43.200641 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 223867us
W20260709 12:22:43.204020 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 47326us
I20260709 12:22:43.214476 17756 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 10.
W20260709 12:22:43.216066 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 47249.9us
I20260709 12:22:43.158090 22307 raft_consensus.cc:2804] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:22:43.114892 22301 raft_consensus.cc:515] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:43.160015 22311 raft_consensus.cc:3060] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:43.159647 21888 raft_consensus.cc:2804] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:22:43.221619 18334 leader_election.cc:304] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:22:43.159242 22376 raft_consensus.cc:2804] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:22:43.152413 22389 raft_consensus.cc:2804] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:22:43.133989 22192 consensus_queue.cc:1059] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.072s
I20260709 12:22:43.160724 22306 raft_consensus.cc:493] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
W20260709 12:22:43.227046 17758 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.668s	user 0.008s	sys 0.021s
I20260709 12:22:43.228760 17727 leader_election.cc:304] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:43.144976 22334 raft_consensus.cc:515] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:43.203549 22384 raft_consensus.cc:2749] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
W20260709 12:22:43.234099 17758 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.675s	user 0.009s	sys 0.021s
I20260709 12:22:43.232720 22134 raft_consensus.cc:493] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:43.244565 22134 raft_consensus.cc:515] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:43.247229 22134 leader_election.cc:290] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:43.220397 22045 consensus_queue.cc:1048] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.021s
I20260709 12:22:43.268729 18334 leader_election.cc:304] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
W20260709 12:22:43.257093 22379 log.cc:927] Time spent T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.480s	user 0.001s	sys 0.002s
I20260709 12:22:43.229898 22307 raft_consensus.cc:697] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 12 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:22:43.242355 22306 raft_consensus.cc:515] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:43.260653 22339 leader_election.cc:290] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 8 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:43.238512 22376 raft_consensus.cc:697] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:22:43.205574 22129 raft_consensus.cc:515] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:43.231820 22397 raft_consensus.cc:2804] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:22:43.252918 22341 leader_election.cc:290] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:22:43.256367 22360 consensus_meta.cc:287] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.675s	user 0.011s	sys 0.000s
I20260709 12:22:43.280416 17728 leader_election.cc:304] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:43.241004 22374 raft_consensus.cc:2804] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:22:43.312532 22374 raft_consensus.cc:493] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:43.313000 22374 raft_consensus.cc:3060] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:43.313339 22306 leader_election.cc:290] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:43.236694 21888 raft_consensus.cc:697] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 13 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
W20260709 12:22:43.296625 17801 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:43.242861 22165 leader_election.cc:290] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 12 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:43.265329 22301 leader_election.cc:290] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:43.266824 22133 raft_consensus.cc:2804] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:22:43.308017 22323 leader_election.cc:290] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:43.310391 22311 raft_consensus.cc:515] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:43.311700 22383 raft_consensus.cc:2749] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
W20260709 12:22:43.316807 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 255975us
I20260709 12:22:43.296098 22192 raft_consensus.cc:2804] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:22:43.293279 22390 raft_consensus.cc:2804] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:22:43.299549 22377 raft_consensus.cc:493] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:43.319585 22317 leader_election.cc:290] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:43.297902 22134 raft_consensus.cc:493] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:43.292140 22358 consensus_meta.cc:287] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.682s	user 0.009s	sys 0.003s
I20260709 12:22:43.298291 22397 raft_consensus.cc:493] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
W20260709 12:22:43.322330 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 255279us
I20260709 12:22:43.304481 22334 leader_election.cc:290] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:43.324627 17652 leader_election.cc:304] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:22:43.320428 22133 raft_consensus.cc:493] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
W20260709 12:22:43.380959 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 56399.8us
W20260709 12:22:43.381263 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 234017us
I20260709 12:22:43.382427 18333 leader_election.cc:304] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:22:43.321388 17728 leader_election.cc:304] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:43.371217 17763 raft_consensus.cc:1240] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 8. Current term is 9. Ops: []
I20260709 12:22:43.326537 22184 raft_consensus.cc:2749] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:22:43.322103 22398 raft_consensus.cc:2804] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Leader pre-election won for term 11
W20260709 12:22:43.387106 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 158425us
I20260709 12:22:43.388963 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:43.326751 22192 raft_consensus.cc:493] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:43.390753 17684 raft_consensus.cc:2468] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 10.
I20260709 12:22:43.327051 22346 leader_election.cc:290] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 8 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:43.390780 17727 leader_election.cc:304] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:22:43.322137 22148 consensus_queue.cc:1048] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.062s
I20260709 12:22:43.372236 22390 raft_consensus.cc:697] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 11 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:22:43.381088 22042 raft_consensus.cc:493] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:43.405629 17654 leader_election.cc:400] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 7 election: Vote denied by peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577) with higher term. Message: Invalid argument: T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for earlier term 7. Current term is 8.
I20260709 12:22:43.407893 17654 leader_election.cc:403] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 7 election: Cancelling election due to peer responding with higher term
I20260709 12:22:43.388199 22396 raft_consensus.cc:3055] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:22:43.373981 22301 consensus_queue.cc:1048] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.025s
I20260709 12:22:43.417918 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:43.381840 22397 raft_consensus.cc:3060] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:43.384668 22133 raft_consensus.cc:3060] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:22:43.410075 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 88801.3us
W20260709 12:22:43.435716 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:43.397612 22192 raft_consensus.cc:3060] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:43.389771 22352 raft_consensus.cc:493] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:43.326395 22305 consensus_queue.cc:1048] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.080s
I20260709 12:22:43.405871 22410 raft_consensus.cc:493] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:43.398262 22389 raft_consensus.cc:697] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 12 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:22:43.378949 22134 raft_consensus.cc:515] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:43.370716 22377 raft_consensus.cc:515] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:43.380508 22358 raft_consensus.cc:515] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:43.377842 22307 consensus_queue.cc:237] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.7, Last appended by leader: 7, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:43.388358 22376 consensus_queue.cc:237] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:43.447252 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:43.415072 22042 raft_consensus.cc:515] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:43.465725 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:43.415547 22356 leader_election.cc:290] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:43.479391 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:22:43.496973 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 91504.4us
I20260709 12:22:43.554064 22129 leader_election.cc:290] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:43.607185 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:22:43.576257 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:43.470646 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:43.401917 22317 raft_consensus.cc:2764] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Leader pre-election decision vote started in defunct term 10: won
I20260709 12:22:43.600315 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:22:43.495040 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 113443us
I20260709 12:22:43.601405 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:43.614449 18366 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 9.
I20260709 12:22:43.602372 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:43.406236 22406 raft_consensus.cc:493] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:43.477197 22403 raft_consensus.cc:493] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:43.477597 18361 raft_consensus.cc:1240] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 8. Current term is 9. Ops: []
I20260709 12:22:43.603607 17728 leader_election.cc:304] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:43.603703 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:43.604625 18364 raft_consensus.cc:2368] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 12. Current term is 13.
I20260709 12:22:43.445199 22404 raft_consensus.cc:493] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:43.402302 21888 consensus_queue.cc:237] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.8, Last appended by leader: 8, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:43.606447 17654 leader_election.cc:304] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:22:43.603084 22396 raft_consensus.cc:740] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
W20260709 12:22:43.652103 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 47558.8us
I20260709 12:22:43.611629 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 11 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:43.611567 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:43.653455 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:43.404918 22069 raft_consensus.cc:2804] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Leader election won for term 10
W20260709 12:22:43.484632 22393 log.cc:927] Time spent T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.247s	user 0.000s	sys 0.001s
W20260709 12:22:43.515255 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 58114.9us
I20260709 12:22:43.406350 22045 raft_consensus.cc:2749] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
W20260709 12:22:43.546550 22409 log.cc:927] Time spent T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.170s	user 0.000s	sys 0.001s
I20260709 12:22:43.616403 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:22:43.604542 22388 log.cc:927] Time spent T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.308s	user 0.005s	sys 0.004s
I20260709 12:22:43.407003 22131 leader_election.cc:290] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:43.444353 22402 raft_consensus.cc:493] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:43.664510 22402 raft_consensus.cc:515] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:43.665421 22069 raft_consensus.cc:697] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:22:43.666818 22402 leader_election.cc:290] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:43.607657 22356 raft_consensus.cc:493] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:43.667454 18362 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59886 (request call id 1337) took 1358 ms. Trace:
I20260709 12:22:43.649709 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:43.650453 18367 raft_consensus.cc:2368] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 10. Current term is 11.
W20260709 12:22:43.611531 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:43.406301 22323 consensus_queue.cc:1059] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.214s
I20260709 12:22:43.406514 22341 raft_consensus.cc:2804] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:22:43.655035 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:43.621764 22003 raft_consensus.cc:2749] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Leader election lost for term 7. Reason: Vote denied by peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577) with higher term. Message: Invalid argument: T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for earlier term 7. Current term is 8.
I20260709 12:22:43.658537 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:43.537992 22334 consensus_queue.cc:1059] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 11.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.111s
W20260709 12:22:43.653144 22149 consensus_meta.cc:287] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 1.130s	user 0.014s	sys 0.002s
I20260709 12:22:43.656919 22408 raft_consensus.cc:493] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:43.656447 22404 raft_consensus.cc:515] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:43.666451 22069 consensus_queue.cc:237] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:43.620783 22311 leader_election.cc:290] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 9 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:43.668594 18364 raft_consensus.cc:2468] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 8.
I20260709 12:22:43.658711 17727 leader_election.cc:304] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:22:43.697260 17727 leader_election.cc:304] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:43.660778 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:43.698256 18361 raft_consensus.cc:2393] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 8: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:22:43.650918 22416 raft_consensus.cc:2749] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 12:22:43.648144 17728 leader_election.cc:304] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:43.667505 22356 raft_consensus.cc:515] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:43.702165 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 10 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:43.703214 17836 raft_consensus.cc:2410] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for term 10 because replica has last-logged OpId of term: 9 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 10.
I20260709 12:22:43.703481 22356 leader_election.cc:290] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:43.652621 22403 raft_consensus.cc:515] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:22:43.706274 22403 leader_election.cc:290] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:43.706875 22311 raft_consensus.cc:2804] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:22:43.708771 22399 raft_consensus.cc:493] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:43.709465 22399 raft_consensus.cc:515] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:43.711664 22399 leader_election.cc:290] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:43.668138 22218 leader_election.cc:290] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 10 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:43.716729 17760 raft_consensus.cc:2410] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 9 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20260709 12:22:43.672022 18368 raft_consensus.cc:2393] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 10: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:22:43.606626 22352 raft_consensus.cc:515] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:43.673432 18371 raft_consensus.cc:2468] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 11.
W20260709 12:22:43.682471 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 76085.4us
I20260709 12:22:43.689018 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:43.673223 22341 raft_consensus.cc:493] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:43.690567 17803 leader_election.cc:304] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:43.690727 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:43.729079 17654 leader_election.cc:304] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:22:43.729976 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 14 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:43.730609 17829 raft_consensus.cc:2468] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 12.
I20260709 12:22:43.691394 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:43.691519 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:43.692725 22423 raft_consensus.cc:2804] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:22:43.693547 22404 leader_election.cc:290] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:22:43.694536 22380 log.cc:927] Time spent T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.917s	user 0.003s	sys 0.005s
I20260709 12:22:43.693961 22133 raft_consensus.cc:515] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:43.694104 22396 consensus_queue.cc:260] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:43.695787 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:43.698853 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:43.699098 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 11 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:43.668218 18362 rpcz_store.cc:276] 0709 12:22:42.309565 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:42.309737 (+   172us) service_pool.cc:224] Handling call
0709 12:22:42.333266 (+ 23529us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:42.337490 (+  4224us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:42.337521 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:42.365462 (+ 27941us) log.cc:844] Serialized 26 byte log entry
0709 12:22:43.131700 (+766238us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:22:43.131755 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:43.132005 (+   250us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:43.667214 (+535209us) raft_consensus.cc:1685] finished
0709 12:22:43.667235 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:43.667366 (+   131us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3837,"raft.queue_time_us":4490,"raft.run_cpu_time_us":131,"raft.run_wall_time_us":8382,"thread_start_us":721874,"threads_started":1,"wal-append.queue_time_us":842078}
I20260709 12:22:43.611168 22410 raft_consensus.cc:515] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:43.708738 18374 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 1463) took 1208 ms. Trace:
W20260709 12:22:43.714425 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 66390.6us
I20260709 12:22:43.718454 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:43.721228 17830 raft_consensus.cc:2393] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 13: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:22:43.689474 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:43.726110 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:43.726828 22358 leader_election.cc:290] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 8 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:43.735028 17834 raft_consensus.cc:2393] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 10: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:22:43.734567 22423 raft_consensus.cc:493] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:43.735664 22423 raft_consensus.cc:3060] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:43.735984 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:43.736552 22133 leader_election.cc:290] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 10 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:43.681347 22323 consensus_queue.cc:1243] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer e063541542a94969898868bf6eb5ce1b log is divergent from this leader: its last log entry 13.11 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 10
I20260709 12:22:43.736742 17834 raft_consensus.cc:2393] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 8: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:22:43.479096 22390 consensus_queue.cc:237] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 10.11, Last appended by leader: 11, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:43.736852 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:43.737476 17830 raft_consensus.cc:3060] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:43.737823 17834 raft_consensus.cc:1240] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 13. Current term is 14. Ops: []
I20260709 12:22:43.739355 17838 raft_consensus.cc:2376] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 11. Re-sending same reply.
I20260709 12:22:43.740361 22396 raft_consensus.cc:3060] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:43.740154 18374 rpcz_store.cc:276] 0709 12:22:42.528505 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:42.549169 (+ 20664us) service_pool.cc:224] Handling call
0709 12:22:42.576312 (+ 27143us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:43.053566 (+477254us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:43.053616 (+    50us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:43.059111 (+  5495us) log.cc:844] Serialized 26 byte log entry
0709 12:22:43.319629 (+260518us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:22:43.319680 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:43.321146 (+  1466us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:43.708537 (+387391us) raft_consensus.cc:1685] finished
0709 12:22:43.708569 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:43.708686 (+   117us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":583187,"raft.run_cpu_time_us":205,"raft.run_wall_time_us":49762,"thread_start_us":728013,"threads_started":2,"wal-append.queue_time_us":491418}
I20260709 12:22:43.742267 17831 raft_consensus.cc:2468] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 11.
I20260709 12:22:43.731474 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:43.744588 22384 consensus_queue.cc:1059] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 12.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 12:22:43.745374 22384 raft_consensus.cc:3055] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:22:43.745703 22384 raft_consensus.cc:740] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 12 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:22:43.746541 22384 consensus_queue.cc:260] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.8, Last appended by leader: 8, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:43.748111 17832 raft_consensus.cc:2468] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 8.
I20260709 12:22:43.748572 22384 raft_consensus.cc:3060] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:43.748904 17835 raft_consensus.cc:2468] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 7.
I20260709 12:22:43.744228 17829 raft_consensus.cc:3060] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:43.751748 22410 leader_election.cc:290] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:43.734943 17762 raft_consensus.cc:2468] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 10.
I20260709 12:22:43.732582 22421 raft_consensus.cc:2804] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:22:43.753608 22421 raft_consensus.cc:697] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 9 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:22:43.754572 22421 consensus_queue.cc:237] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 7.9, Last appended by leader: 9, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:43.731465 17654 leader_election.cc:304] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:22:43.757758 22003 raft_consensus.cc:2804] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:22:43.758249 22003 raft_consensus.cc:697] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 11 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:22:43.759159 22003 consensus_queue.cc:237] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.9, Last appended by leader: 9, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:43.761310 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:43.768195 18362 raft_consensus.cc:3060] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:43.762245 18334 leader_election.cc:304] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
W20260709 12:22:43.771064 18334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:43.772167 18334 leader_election.cc:304] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:22:43.773054 22399 raft_consensus.cc:2804] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:22:43.773433 22399 raft_consensus.cc:493] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:43.773509 22323 consensus_queue.cc:1059] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:22:43.773594 22417 raft_consensus.cc:2764] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Leader pre-election decision vote started in defunct term 7: won
I20260709 12:22:43.774546 22323 raft_consensus.cc:3055] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:22:43.766453 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 11 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:43.775036 22323 raft_consensus.cc:740] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:22:43.775467 18371 raft_consensus.cc:3060] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:43.775873 22323 consensus_queue.cc:260] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.11, Last appended by leader: 11, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:43.777695 22323 raft_consensus.cc:3060] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:43.773854 22399 raft_consensus.cc:3060] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:43.766144 17829 raft_consensus.cc:2468] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 8.
I20260709 12:22:43.792205 18371 raft_consensus.cc:2468] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 11.
I20260709 12:22:43.802227 18371 raft_consensus.cc:3055] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:22:43.802620 18371 raft_consensus.cc:740] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:22:43.803386 18371 consensus_queue.cc:260] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 11.9, Last appended by leader: 9, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:43.804488 18371 raft_consensus.cc:3060] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:43.805510 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:43.806134 18374 raft_consensus.cc:2468] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 10.
I20260709 12:22:43.807986 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:43.809005 17830 raft_consensus.cc:2468] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 12.
I20260709 12:22:43.809123 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:43.809867 18364 raft_consensus.cc:2468] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 12.
I20260709 12:22:43.810595 18334 leader_election.cc:304] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:22:43.811291 18371 raft_consensus.cc:1275] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 11 index: 9. Preceding OpId from leader: term: 12 index: 9. (term mismatch)
I20260709 12:22:43.811453 22396 raft_consensus.cc:2804] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:22:43.811689 18371 pending_rounds.cc:85] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da: Aborting all ops after (but not including) 8
I20260709 12:22:43.812003 18371 pending_rounds.cc:107] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da: Aborting uncommitted NO_OP operation due to leader change: 11.9
I20260709 12:22:43.811903 22396 raft_consensus.cc:493] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:43.812299 18371 raft_consensus.cc:2889] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:43.812438 22396 raft_consensus.cc:3060] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:43.820700 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 14 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:43.821707 17756 raft_consensus.cc:2410] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 14 because replica has last-logged OpId of term: 12 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 9.
I20260709 12:22:43.713639 22181 raft_consensus.cc:2749] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:22:43.819346 22396 raft_consensus.cc:515] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:43.831688 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:22:43.830055 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 139559us
I20260709 12:22:43.833223 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
W20260709 12:22:43.833935 17803 leader_election.cc:336] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Timed out: RequestConsensusVote RPC to 127.16.29.68:42577 timed out after 1.955s (SENT)
I20260709 12:22:43.834403 17683 raft_consensus.cc:2410] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for term 16 because replica has last-logged OpId of term: 15 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 13.
I20260709 12:22:43.648957 22406 raft_consensus.cc:515] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:43.834399 17803 leader_election.cc:304] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:43.832561 17689 raft_consensus.cc:2376] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 8. Re-sending same reply.
I20260709 12:22:43.720150 22357 raft_consensus.cc:2749] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:22:43.823096 22341 raft_consensus.cc:3060] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:43.837939 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:43.842151 17681 raft_consensus.cc:2468] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 12.
I20260709 12:22:43.842730 22358 raft_consensus.cc:2749] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:22:43.842993 22134 leader_election.cc:290] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:43.844223 22377 leader_election.cc:290] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:43.728230 22311 raft_consensus.cc:697] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 12 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:22:43.846884 22396 leader_election.cc:290] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 10 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:43.847293 22311 consensus_queue.cc:237] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:43.822938 22424 raft_consensus.cc:2749] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:22:43.840745 22416 consensus_queue.cc:1059] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 12.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.002s
I20260709 12:22:43.835295 22042 leader_election.cc:290] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:43.857622 17803 leader_election.cc:304] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:43.697657 22389 consensus_queue.cc:237] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:43.852480 21994 leader_election.cc:290] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:43.839499 17801 leader_election.cc:304] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:22:43.864715 22413 log.cc:927] Time spent T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.145s	user 0.000s	sys 0.000s
I20260709 12:22:43.887732 17686 raft_consensus.cc:1240] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 9. Current term is 10. Ops: []
I20260709 12:22:43.888935 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:43.877307 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 12 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:43.713425 22408 raft_consensus.cc:515] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:43.881866 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:43.883960 22383 consensus_queue.cc:1059] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 13.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
W20260709 12:22:43.893420 18334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:43.879927 22114 raft_consensus.cc:2749] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:22:43.889894 17681 raft_consensus.cc:2393] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 12: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:22:43.891111 17728 leader_election.cc:304] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
W20260709 12:22:43.894313 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 53256.6us
I20260709 12:22:43.895358 17685 raft_consensus.cc:1240] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 12 LEADER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 11. Current term is 12. Ops: [11.9-11.9]
I20260709 12:22:43.900074 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 17 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:43.900707 18371 raft_consensus.cc:2468] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 15.
I20260709 12:22:43.901467 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:43.904569 18371 raft_consensus.cc:2955] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [term 10 LEARNER]: Committing config change with OpId 8.5: config changed from index -1 to 5, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } }
I20260709 12:22:43.897814 17687 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37396 (request call id 2275) took 1846 ms. Trace:
I20260709 12:22:43.910722 17728 leader_election.cc:304] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:22:43.888836 22406 leader_election.cc:290] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:22:43.913020 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 200634us
I20260709 12:22:43.918933 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 12 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:43.923491 18333 leader_election.cc:304] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:43.925256 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:43.925423 17728 leader_election.cc:304] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:43.928066 17728 leader_election.cc:304] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:43.903254 22384 raft_consensus.cc:3055] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:22:43.901093 22424 raft_consensus.cc:2804] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:22:43.889287 22397 raft_consensus.cc:515] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:43.905298 22181 raft_consensus.cc:2749] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20260709 12:22:43.911581 22335 consensus_meta.cc:287] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.893s	user 0.007s	sys 0.007s
I20260709 12:22:43.939661 18333 leader_election.cc:304] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:22:43.960620 22114 raft_consensus.cc:493] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:43.946272 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:43.925812 22383 raft_consensus.cc:2749] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:22:43.961701 17829 raft_consensus.cc:3060] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:43.968366 17688 raft_consensus.cc:2393] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 10: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:22:43.911842 17687 rpcz_store.cc:276] 0709 12:22:42.051690 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:42.532547 (+480857us) service_pool.cc:224] Handling call
0709 12:22:42.532915 (+   368us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:42.533163 (+   248us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 12:22:43.116427 (+583264us) log.cc:844] Serialized 14 byte log entry
0709 12:22:43.677069 (+560642us) spinlock_profiling.cc:243] Waited 3.62 ms on lock 0x7b180034c840. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b474adc0a 00007f7b50bed219 00007f7b47535a40 00007f7b475304e4 00007f7b47530455 00007f7b4e0183c7 00007f7b4a62a9a7 00007f7b4a6335e5 00007f7b4a633568
0709 12:22:43.721749 (+ 44680us) log.cc:844] Serialized 14 byte log entry
0709 12:22:43.722273 (+   524us) log.cc:844] Serialized 14 byte log entry
0709 12:22:43.891909 (+169636us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:43.891965 (+    56us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:22:43.892016 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:43.892086 (+    70us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:43.897737 (+  5651us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":108750,"prepare.queue_time_us":944815,"prepare.run_cpu_time_us":119,"prepare.run_wall_time_us":121,"raft.queue_time_us":148400,"raft.run_cpu_time_us":264,"raft.run_wall_time_us":19985,"spinlock_wait_cycles":8332160,"thread_start_us":943400,"threads_started":4,"wal-append.queue_time_us":539663}
I20260709 12:22:43.964440 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 12 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:43.971021 18362 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 10.
I20260709 12:22:43.982208 17832 raft_consensus.cc:2393] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 12: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:22:43.965309 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:43.965834 18333 leader_election.cc:304] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:22:43.973301 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:43.944036 17654 leader_election.cc:304] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:22:43.984923 17833 raft_consensus.cc:2468] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 10.
I20260709 12:22:43.974260 17831 raft_consensus.cc:1240] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 12. Current term is 13. Ops: [12.8-12.8]
I20260709 12:22:43.966148 17830 raft_consensus.cc:1240] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 11. Current term is 12. Ops: [10.8-11.9]
I20260709 12:22:43.988313 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:43.975611 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:43.986136 17728 leader_election.cc:304] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:43.993791 17686 raft_consensus.cc:2468] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 10.
I20260709 12:22:43.928519 22396 raft_consensus.cc:2749] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:22:43.996604 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:43.933992 22416 raft_consensus.cc:2804] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:22:43.999197 17833 raft_consensus.cc:1240] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 13. Current term is 14. Ops: [13.7-13.7]
I20260709 12:22:43.999540 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:43.934938 22384 raft_consensus.cc:740] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
W20260709 12:22:43.962410 22414 log.cc:927] Time spent T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.302s	user 0.003s	sys 0.000s
I20260709 12:22:43.938868 22424 raft_consensus.cc:493] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:44.001106 17654 leader_election.cc:304] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:22:44.012557 17833 raft_consensus.cc:1275] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 11 index: 9. (index mismatch)
I20260709 12:22:43.966079 22346 raft_consensus.cc:2764] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Leader pre-election decision vote started in defunct term 12: won
I20260709 12:22:44.001999 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:44.009969 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
W20260709 12:22:44.018536 17802 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:44.010057 17727 leader_election.cc:304] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:22:43.962144 22390 consensus_queue.cc:1059] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.053s
I20260709 12:22:44.021431 22397 leader_election.cc:290] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 12 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:44.022083 22069 raft_consensus.cc:2804] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:22:44.022492 22069 raft_consensus.cc:493] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:44.022831 22069 raft_consensus.cc:3060] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:44.013453 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:44.026569 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:44.028108 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:44.030115 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 14 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:22:43.998708 22427 log.cc:927] Time spent T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.130s	user 0.001s	sys 0.000s
I20260709 12:22:43.964172 22377 raft_consensus.cc:2749] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:22:44.022720 22422 raft_consensus.cc:3055] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:22:43.965067 22352 leader_election.cc:290] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:44.013661 17654 leader_election.cc:304] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:22:44.031292 17687 raft_consensus.cc:2410] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 14 because replica has last-logged OpId of term: 12 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 9.
W20260709 12:22:44.032570 17759 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36572 (request call id 799) took 1985 ms (client timeout 1970 ms). Trace:
W20260709 12:22:44.047256 17759 rpcz_store.cc:269] 0709 12:22:42.047074 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:42.058875 (+ 11801us) service_pool.cc:224] Handling call
0709 12:22:44.032366 (+1973491us) spinlock_profiling.cc:243] Waited 799 ms on lock 0x7b54001b9254. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:22:44.032488 (+   122us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1837755136}
I20260709 12:22:44.035334 17829 raft_consensus.cc:1275] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 11 index: 11. Preceding OpId from leader: term: 13 index: 12. (index mismatch)
I20260709 12:22:44.035234 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:44.008302 17835 raft_consensus.cc:1240] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 7. Current term is 8. Ops: []
I20260709 12:22:44.049916 17683 raft_consensus.cc:2468] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 12.
I20260709 12:22:44.004238 22417 raft_consensus.cc:2804] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:22:44.004170 22416 raft_consensus.cc:493] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:22:44.012362 22181 raft_consensus.cc:2804] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Leader pre-election won for term 14
W20260709 12:22:44.054618 17803 leader_election.cc:336] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Timed out: RequestConsensusVote RPC to 127.16.29.68:42577 timed out after 2.210s (SENT)
I20260709 12:22:44.082381 17803 leader_election.cc:304] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:44.022092 17727 leader_election.cc:304] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:22:43.967734 22114 raft_consensus.cc:515] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:44.001976 22437 raft_consensus.cc:2804] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:22:43.979198 22423 raft_consensus.cc:515] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:44.046192 17654 leader_election.cc:304] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
W20260709 12:22:44.063141 22433 log.cc:927] Time spent T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.065s	user 0.000s	sys 0.000s
I20260709 12:22:44.019098 22424 raft_consensus.cc:3060] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:22:44.063144 22374 consensus_meta.cc:287] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.750s	user 0.012s	sys 0.000s
I20260709 12:22:44.063836 22384 consensus_queue.cc:260] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 13.7, Last appended by leader: 7, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:44.056726 22417 raft_consensus.cc:493] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:22:44.064888 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:44.120836 17683 raft_consensus.cc:2410] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 9 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20260709 12:22:44.067176 22352 raft_consensus.cc:1064] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3: attempting to promote NON_VOTER 334a4e1f59944544b3b3ff36616b57da to VOTER
I20260709 12:22:44.027482 22399 raft_consensus.cc:515] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:44.039915 22422 raft_consensus.cc:740] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:22:44.072002 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:44.067356 22341 raft_consensus.cc:515] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:22:44.084093 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 62099.5us
I20260709 12:22:44.031410 21994 raft_consensus.cc:2804] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Leader pre-election won for term 11
W20260709 12:22:44.122679 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:44.125197 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:44.080489 22357 raft_consensus.cc:3055] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:22:44.059808 22420 raft_consensus.cc:2804] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Leader pre-election won for term 13
W20260709 12:22:44.130369 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 84295.1us
I20260709 12:22:44.079435 22069 raft_consensus.cc:515] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:44.037249 22383 consensus_queue.cc:1059] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 8.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.082s
I20260709 12:22:44.146288 22437 raft_consensus.cc:697] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 11 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
W20260709 12:22:44.187392 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 132831us
W20260709 12:22:44.202250 17803 leader_election.cc:336] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Timed out: RequestConsensusVote RPC to 127.16.29.68:42577 timed out after 1.809s (SENT)
I20260709 12:22:44.202152 22437 consensus_queue.cc:237] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.7, Last appended by leader: 7, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:44.202683 17803 leader_election.cc:304] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:44.081498 22416 raft_consensus.cc:3060] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:44.203406 22114 leader_election.cc:290] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:44.184816 21994 raft_consensus.cc:493] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:44.204553 21994 raft_consensus.cc:3060] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:44.115615 22377 raft_consensus.cc:3060] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:44.154100 21888 raft_consensus.cc:2804] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:22:44.208246 22384 raft_consensus.cc:3060] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:44.210492 22352 consensus_queue.cc:237] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 5, Current term: 10, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:44.212484 18334 leader_election.cc:304] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:22:44.212805 22398 raft_consensus.cc:493] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:44.213204 22399 leader_election.cc:290] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 13 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:44.213825 22399 raft_consensus.cc:2804] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:22:44.214268 22399 raft_consensus.cc:697] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 10 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:22:44.214685 17728 leader_election.cc:304] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:22:44.215044 22399 consensus_queue.cc:237] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.9, Last appended by leader: 9, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:44.145896 22417 raft_consensus.cc:3060] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:44.081095 22181 raft_consensus.cc:493] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:44.218565 17763 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38786 (request call id 2911) took 1196 ms. Trace:
I20260709 12:22:44.219015 17763 rpcz_store.cc:276] 0709 12:22:43.022189 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:43.036354 (+ 14165us) service_pool.cc:224] Handling call
0709 12:22:43.036695 (+   341us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:43.384236 (+347541us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:44.218276 (+834040us) spinlock_profiling.cc:243] Waited 334 ms on lock 0x7b54000fdcd4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:22:44.218495 (+   219us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":768753920}
I20260709 12:22:44.080724 22435 consensus_queue.cc:1048] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.108s
I20260709 12:22:44.231456 18365 raft_consensus.cc:3055] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:22:44.232347 18365 raft_consensus.cc:740] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:22:44.148777 22408 leader_election.cc:290] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:44.223780 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:22:44.224483 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 219348us
I20260709 12:22:44.218605 17684 raft_consensus.cc:1240] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 9. Current term is 10. Ops: []
I20260709 12:22:44.238408 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:44.244297 17755 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38776 (request call id 2207) took 1874 ms. Trace:
I20260709 12:22:44.244808 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:44.245813 18364 raft_consensus.cc:2468] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 8.
I20260709 12:22:44.244976 17755 rpcz_store.cc:276] 0709 12:22:42.369558 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:42.399035 (+ 29477us) service_pool.cc:224] Handling call
0709 12:22:42.414512 (+ 15477us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:42.510801 (+ 96289us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:42.510846 (+    45us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:42.511407 (+   561us) log.cc:844] Serialized 26 byte log entry
0709 12:22:43.004929 (+493522us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:22:43.004989 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:44.236497 (+1231508us) spinlock_profiling.cc:243] Waited 96 ms on lock 0x7b5400415fd4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:22:44.236540 (+    43us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:44.236582 (+    42us) raft_consensus.cc:1685] finished
0709 12:22:44.236599 (+    17us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:44.244218 (+  7619us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":220822912,"thread_start_us":492117,"threads_started":1,"wal-append.queue_time_us":642075}
I20260709 12:22:44.213238 22398 raft_consensus.cc:3060] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:44.248538 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:44.254851 17686 raft_consensus.cc:2468] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 12.
I20260709 12:22:44.248343 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:44.256313 18364 raft_consensus.cc:2393] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 8: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:22:44.247689 18333 leader_election.cc:304] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:22:44.224910 22181 raft_consensus.cc:3060] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:44.194502 22357 raft_consensus.cc:740] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:22:44.208261 21888 raft_consensus.cc:493] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:44.251046 18365 consensus_queue.cc:260] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.11, Last appended by leader: 11, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:44.200990 22420 raft_consensus.cc:493] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:44.129547 22374 raft_consensus.cc:515] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:44.366115 18365 raft_consensus.cc:3060] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:22:44.395927 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 181275us
I20260709 12:22:44.368201 21888 raft_consensus.cc:3060] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:22:44.368588 22420 raft_consensus.cc:3060] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:44.406796 22193 raft_consensus.cc:3055] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 12 LEADER]: Stepping down as leader of term 12
W20260709 12:22:44.407243 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 159655us
I20260709 12:22:44.408212 18333 leader_election.cc:304] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
W20260709 12:22:44.409840 22192 consensus_meta.cc:287] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.792s	user 0.011s	sys 0.000s
I20260709 12:22:44.410320 22192 raft_consensus.cc:515] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:44.386322 22323 raft_consensus.cc:2749] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:22:44.375765 22418 raft_consensus.cc:2749] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20260709 12:22:44.413709 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 799090us
I20260709 12:22:44.414896 17652 leader_election.cc:304] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
W20260709 12:22:44.416530 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 214298us
I20260709 12:22:44.416692 22406 raft_consensus.cc:2804] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:22:44.417230 22406 raft_consensus.cc:697] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 12 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
W20260709 12:22:44.417420 17803 leader_election.cc:336] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Timed out: RequestConsensusVote RPC to 127.16.29.68:42577 timed out after 1.938s (SENT)
I20260709 12:22:44.407322 22193 raft_consensus.cc:740] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 12 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:22:44.410363 22134 raft_consensus.cc:2749] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:22:44.418223 22406 consensus_queue.cc:237] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.8, Last appended by leader: 8, Current term: 12, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:44.453302 18365 raft_consensus.cc:1275] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 13 index: 11. (term mismatch)
I20260709 12:22:44.415699 17652 leader_election.cc:304] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:22:44.475142 18365 pending_rounds.cc:85] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da: Aborting all ops after (but not including) 10
I20260709 12:22:44.471819 17834 raft_consensus.cc:1240] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 11. Current term is 12. Ops: [11.9-11.9]
W20260709 12:22:44.470556 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 62420.6us
I20260709 12:22:44.476097 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:44.476660 18365 pending_rounds.cc:107] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da: Aborting uncommitted NO_OP operation due to leader change: 12.11
I20260709 12:22:44.476569 18333 leader_election.cc:304] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:22:44.477162 18365 raft_consensus.cc:2889] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:44.479600 17835 raft_consensus.cc:1240] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 13 LEADER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 12. Current term is 13. Ops: [12.11-12.11]
I20260709 12:22:44.478017 18333 leader_election.cc:304] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:22:44.437744 22323 raft_consensus.cc:2749] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260709 12:22:44.490792 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 75139.6us
I20260709 12:22:44.485399 18333 leader_election.cc:304] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:22:44.437922 22114 raft_consensus.cc:2749] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:22:44.478583 22358 raft_consensus.cc:493] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:44.490118 22317 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 13 current term: 13 is higher.
I20260709 12:22:44.478528 22404 raft_consensus.cc:493] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:44.482576 22390 raft_consensus.cc:2804] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:22:44.502668 17652 leader_election.cc:304] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:22:44.492266 22435 raft_consensus.cc:2804] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:22:44.491657 22346 raft_consensus.cc:2749] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:22:44.507685 18333 leader_election.cc:304] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:22:44.514452 17652 leader_election.cc:304] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da
W20260709 12:22:44.552109 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 133418us
I20260709 12:22:44.508939 22390 raft_consensus.cc:493] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:22:44.562747 22400 log.cc:927] Time spent T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 1.085s	user 0.002s	sys 0.000s
I20260709 12:22:44.552417 22192 leader_election.cc:290] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 9 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:44.564630 17680 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37402 (request call id 2645) took 1947 ms. Trace:
I20260709 12:22:44.565008 17680 rpcz_store.cc:276] 0709 12:22:42.616951 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:42.617120 (+   169us) service_pool.cc:224] Handling call
0709 12:22:42.638241 (+ 21121us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:42.761029 (+122788us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:42.761081 (+    52us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:42.771893 (+ 10812us) log.cc:844] Serialized 26 byte log entry
0709 12:22:43.298611 (+526718us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:22:43.324980 (+ 26369us) log.cc:844] Serialized 14 byte log entry
0709 12:22:43.325292 (+   312us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:43.325392 (+   100us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:44.562047 (+1236655us) spinlock_profiling.cc:243] Waited 12.7 ms on lock 0x7b4400022b4c. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b47459c85 00007f7b4745a6a2 00007f7b4a1cc1db 00007f7b4a1c7aaf 00007f7b4a1bb86e 00007f7b4a1babb6 00007f7b4a1ba7d8 00007f7b4a1c4221 00007f7b4a1c41d9 00007f7b4a1c4169
0709 12:22:44.564353 (+  2306us) raft_consensus.cc:1685] finished
0709 12:22:44.564441 (+    88us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:44.564574 (+   133us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":298584,"prepare.queue_time_us":24430,"prepare.run_cpu_time_us":58,"prepare.run_wall_time_us":60,"raft.queue_time_us":120493,"raft.run_cpu_time_us":224,"raft.run_wall_time_us":6012,"spinlock_wait_cycles":29150464,"thread_start_us":397848,"threads_started":1,"wal-append.queue_time_us":535631}
I20260709 12:22:44.568502 21994 raft_consensus.cc:515] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:44.571225 21994 leader_election.cc:290] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 11 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:44.501999 22357 consensus_queue.cc:260] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.8, Last appended by leader: 8, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:44.540518 22397 raft_consensus.cc:2749] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:22:44.545186 22435 raft_consensus.cc:697] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 14 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:22:44.553930 18333 leader_election.cc:304] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3
W20260709 12:22:44.553268 17803 leader_election.cc:336] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Timed out: RequestConsensusVote RPC to 127.16.29.68:42577 timed out after 1.735s (SENT)
I20260709 12:22:44.576592 17803 leader_election.cc:304] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:44.578440 17802 leader_election.cc:304] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:44.580332 17833 raft_consensus.cc:1240] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 7. Current term is 8. Ops: []
W20260709 12:22:44.575978 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 61656us
I20260709 12:22:44.580797 17835 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f"
dest_uuid: "e063541542a94969898868bf6eb5ce1b"
mode: GRACEFUL
new_leader_uuid: "334a4e1f59944544b3b3ff36616b57da"
 from {username='slave'} at 127.0.0.1:59820
I20260709 12:22:44.581439 17835 raft_consensus.cc:606] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Received request to transfer leadership to TS 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:44.581765 17835 raft_consensus.cc:614] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:22:44.522142 22374 leader_election.cc:290] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 12 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:44.574888 22346 raft_consensus.cc:2804] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:22:44.591425 22346 raft_consensus.cc:493] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:44.591482 17652 leader_election.cc:304] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:44.592054 22346 raft_consensus.cc:3060] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:44.593384 22383 consensus_queue.cc:1048] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 12:22:44.593355 17652 leader_election.cc:304] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:44.594588 22418 consensus_queue.cc:1048] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 12:22:44.588977 22421 raft_consensus.cc:2804] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:22:44.596468 17838 raft_consensus.cc:1275] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
I20260709 12:22:44.596552 22335 raft_consensus.cc:2804] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:22:44.597095 22335 raft_consensus.cc:697] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 12 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:22:44.597414 22334 raft_consensus.cc:2749] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
W20260709 12:22:44.597678 22449 log.cc:927] Time spent T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.223s	user 0.000s	sys 0.001s
I20260709 12:22:44.509008 22193 consensus_queue.cc:260] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 12.7, Last appended by leader: 7, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:44.599496 17652 leader_election.cc:304] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:22:44.599643 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 113506us
I20260709 12:22:44.545509 22437 raft_consensus.cc:2804] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:22:44.505089 22358 raft_consensus.cc:515] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:44.603271 17652 leader_election.cc:304] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:44.508787 22404 raft_consensus.cc:515] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:44.522716 22134 consensus_queue.cc:1059] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.066s
I20260709 12:22:44.575663 22390 raft_consensus.cc:3060] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:44.629671 17652 leader_election.cc:304] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
W20260709 12:22:44.630038 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:44.633211 17652 leader_election.cc:304] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:22:44.633968 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 80369.7us
W20260709 12:22:44.634976 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:44.636619 17652 leader_election.cc:304] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:44.637952 17836 raft_consensus.cc:1240] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 12. Current term is 13. Ops: []
I20260709 12:22:44.596603 22421 raft_consensus.cc:697] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 14 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:22:44.619587 21994 raft_consensus.cc:3060] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:44.598008 22335 consensus_queue.cc:237] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.9, Last appended by leader: 9, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:44.635622 18333 leader_election.cc:304] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:22:44.622987 22069 leader_election.cc:290] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 11 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:44.628472 22425 raft_consensus.cc:2749] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:22:44.628381 22193 raft_consensus.cc:3060] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:44.629498 22437 raft_consensus.cc:697] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 10 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:22:44.653268 17652 leader_election.cc:400] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 7 election: Vote denied by peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651) with higher term. Message: Invalid argument: T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for earlier term 7. Current term is 8.
I20260709 12:22:44.634121 22365 raft_consensus.cc:2804] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:22:44.610726 22305 consensus_queue.cc:1059] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.034s
W20260709 12:22:44.650116 22345 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 12 current term: 12 is higher.
I20260709 12:22:44.646061 22374 raft_consensus.cc:2804] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:22:44.677372 17834 raft_consensus.cc:1275] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 12 index: 9. (index mismatch)
I20260709 12:22:44.637117 22003 raft_consensus.cc:2749] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:22:44.655931 22436 consensus_queue.cc:1059] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 11.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.004s
I20260709 12:22:44.680282 18367 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59902 (request call id 1180) took 1256 ms. Trace:
I20260709 12:22:44.680259 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 12 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:44.573292 22435 consensus_queue.cc:237] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 13.7, Last appended by leader: 7, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:44.610145 22420 raft_consensus.cc:515] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:44.673345 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 11 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:44.685974 17835 raft_consensus.cc:3060] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:44.646951 22410 raft_consensus.cc:2749] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:22:44.680842 18367 rpcz_store.cc:276] 0709 12:22:43.416744 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:43.446853 (+ 30109us) service_pool.cc:224] Handling call
0709 12:22:44.672523 (+1225670us) spinlock_profiling.cc:243] Waited 47.2 ms on lock 0x7b540047bdd4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:22:44.672688 (+   165us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":108609920}
I20260709 12:22:44.681304 17834 raft_consensus.cc:2468] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 11.
W20260709 12:22:44.664469 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 111239us
I20260709 12:22:44.636643 22334 consensus_queue.cc:1059] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 10.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.022s
I20260709 12:22:44.658776 18333 leader_election.cc:304] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
W20260709 12:22:44.703727 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 45017.9us
I20260709 12:22:44.704667 18333 leader_election.cc:304] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:22:44.710705 17803 leader_election.cc:304] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:44.720093 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:44.653743 22403 raft_consensus.cc:493] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:44.677829 22365 raft_consensus.cc:697] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 13 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:22:44.680200 22374 raft_consensus.cc:493] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
W20260709 12:22:44.721398 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 138584us
I20260709 12:22:44.679718 22397 raft_consensus.cc:2804] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:22:44.726087 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:44.729317 17764 raft_consensus.cc:2468] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 10.
I20260709 12:22:44.727859 17835 raft_consensus.cc:2468] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 11.
W20260709 12:22:44.686148 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 107784us
W20260709 12:22:44.731346 17727 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:44.731833 17802 leader_election.cc:304] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:44.733335 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:44.734092 17837 raft_consensus.cc:2376] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 14. Re-sending same reply.
I20260709 12:22:44.735464 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:44.737423 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:44.738584 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 17 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:44.739341 17760 raft_consensus.cc:2468] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 16.
I20260709 12:22:44.740952 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 11 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:44.710148 22467 raft_consensus.cc:493] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:44.742056 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:44.791196 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:44.712611 22323 raft_consensus.cc:2804] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:22:44.797624 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:44.744930 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:44.805022 17835 raft_consensus.cc:2468] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 12.
I20260709 12:22:44.743309 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:44.673851 22416 raft_consensus.cc:515] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:44.712359 22477 raft_consensus.cc:2749] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:22:44.735090 22192 raft_consensus.cc:2804] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:22:44.729126 22397 raft_consensus.cc:493] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:44.831310 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:44.727407 22374 raft_consensus.cc:3060] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:44.839655 22476 raft_consensus.cc:2749] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:22:44.839970 22478 raft_consensus.cc:3055] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:22:44.840298 22464 raft_consensus.cc:493] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
W20260709 12:22:44.841342 22466 log.cc:927] Time spent T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.189s	user 0.001s	sys 0.000s
W20260709 12:22:44.841670 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 109976us
I20260709 12:22:45.216090 17802 leader_election.cc:304] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:22:44.842350 22452 log.cc:927] Time spent T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.124s	user 0.001s	sys 0.000s
I20260709 12:22:44.842473 22473 raft_consensus.cc:2749] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:22:44.851481 22420 leader_election.cc:290] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 13 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:44.853037 22437 consensus_queue.cc:237] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.7, Last appended by leader: 7, Current term: 10, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:22:44.858629 22384 consensus_meta.cc:287] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.650s	user 0.007s	sys 0.001s
I20260709 12:22:44.860957 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:44.864799 17760 raft_consensus.cc:1275] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20260709 12:22:45.033593 22045 raft_consensus.cc:493] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:45.056383 22448 raft_consensus.cc:493] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:45.057353 22444 log.cc:927] Time spent T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.462s	user 0.001s	sys 0.000s
W20260709 12:22:45.033638 22398 consensus_meta.cc:287] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.705s	user 0.011s	sys 0.000s
W20260709 12:22:45.070453 17804 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:45.072111 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
W20260709 12:22:45.036230 22377 consensus_meta.cc:287] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.830s	user 0.008s	sys 0.003s
I20260709 12:22:45.078338 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:44.744083 22403 raft_consensus.cc:515] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:45.091282 22475 raft_consensus.cc:2749] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:22:45.071149 22042 raft_consensus.cc:493] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:45.075043 22457 raft_consensus.cc:493] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:44.832242 22467 raft_consensus.cc:515] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:44.790835 22421 consensus_queue.cc:237] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.10, Last appended by leader: 10, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:45.116689 22482 raft_consensus.cc:493] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:45.115335 22193 raft_consensus.cc:493] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:45.133555 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 426534us
I20260709 12:22:45.269500 17803 leader_election.cc:304] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:22:45.271171 17803 leader_election.cc:304] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:45.109647 22404 leader_election.cc:290] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:45.186666 17755 raft_consensus.cc:1240] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 12. Current term is 13. Ops: []
I20260709 12:22:45.126529 22358 leader_election.cc:290] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:45.265941 22345 raft_consensus.cc:493] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:45.277665 22480 log.cc:927] Time spent T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.152s	user 0.000s	sys 0.000s
I20260709 12:22:45.277592 22345 raft_consensus.cc:515] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:45.218526 17802 leader_election.cc:304] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:45.279314 22389 raft_consensus.cc:3055] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:22:45.279541 22493 raft_consensus.cc:493] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:45.225850 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:45.075668 22365 consensus_queue.cc:237] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:45.204756 22397 raft_consensus.cc:3060] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:22:45.239619 17655 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:45.210906 22485 raft_consensus.cc:493] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:45.292268 22485 raft_consensus.cc:515] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:45.294224 17838 raft_consensus.cc:3060] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Advancing to term 11
I20260709 12:22:45.211158 22478 raft_consensus.cc:740] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 8 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:22:45.247144 17829 raft_consensus.cc:1275] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 10 index: 11. Preceding OpId from leader: term: 11 index: 12. (index mismatch)
I20260709 12:22:45.220129 22477 raft_consensus.cc:2749] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
W20260709 12:22:45.261986 17727 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:45.218176 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:45.297115 17837 raft_consensus.cc:2468] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 8.
I20260709 12:22:45.298395 17829 raft_consensus.cc:1275] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 12 index: 8. (index mismatch)
I20260709 12:22:45.262804 22457 raft_consensus.cc:515] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:45.301294 22457 leader_election.cc:290] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:45.302372 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:22:45.277776 21994 consensus_meta.cc:287] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.631s	user 0.009s	sys 0.000s
I20260709 12:22:45.303359 21994 raft_consensus.cc:2749] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:22:45.232013 17652 leader_election.cc:304] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:45.280128 22345 leader_election.cc:290] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:45.304808 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:45.305174 21994 raft_consensus.cc:2804] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:22:45.305691 21994 raft_consensus.cc:493] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:45.281345 18363 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 1486) took 1051 ms. Trace:
I20260709 12:22:45.307215 18363 rpcz_store.cc:276] 0709 12:22:44.230470 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:44.230664 (+   194us) service_pool.cc:224] Handling call
0709 12:22:44.231110 (+   446us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:44.231419 (+   309us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:44.231460 (+    41us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:44.237523 (+  6063us) log.cc:844] Serialized 26 byte log entry
0709 12:22:44.500059 (+262536us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:22:44.500114 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:44.500206 (+    92us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:45.281126 (+780920us) raft_consensus.cc:1685] finished
0709 12:22:45.281146 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:45.281273 (+   127us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":101419,"threads_started":1,"wal-append.queue_time_us":284100}
I20260709 12:22:45.307956 17838 raft_consensus.cc:2468] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 11.
W20260709 12:22:45.279315 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 60911.9us
I20260709 12:22:45.310184 17802 leader_election.cc:304] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:45.311169 22468 raft_consensus.cc:2804] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:22:45.311357 17802 leader_election.cc:304] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:45.279817 22389 raft_consensus.cc:740] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 11 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:22:45.313405 22306 raft_consensus.cc:2749] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:22:45.280206 22476 raft_consensus.cc:2804] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:22:45.315219 22476 raft_consensus.cc:493] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:45.313798 22389 consensus_queue.cc:260] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.9, Last appended by leader: 9, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:45.283577 17803 leader_election.cc:304] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
W20260709 12:22:45.290026 22401 log.cc:927] Time spent T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 1.823s	user 0.006s	sys 0.009s
I20260709 12:22:45.317634 22472 raft_consensus.cc:2749] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:22:45.289044 17687 raft_consensus.cc:1240] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 12. Current term is 13. Ops: [12.7-12.7]
I20260709 12:22:45.317940 17758 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38786 (request call id 2906) took 2781 ms. Trace:
W20260709 12:22:45.304729 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 72874.8us
I20260709 12:22:45.289523 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 11 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:45.262450 22192 raft_consensus.cc:697] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 12 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:22:45.312501 17764 raft_consensus.cc:1240] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 10. Current term is 11. Ops: []
I20260709 12:22:45.315454 21994 raft_consensus.cc:3060] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:45.316180 22476 raft_consensus.cc:3060] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:45.316078 22496 raft_consensus.cc:493] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:45.336288 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:45.237964 22045 raft_consensus.cc:515] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:45.320277 17652 leader_election.cc:304] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:45.306361 17761 raft_consensus.cc:1240] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 10. Current term is 11. Ops: []
I20260709 12:22:45.252948 22377 raft_consensus.cc:2749] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:22:45.341776 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:45.326493 17803 leader_election.cc:304] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:22:45.260519 22384 raft_consensus.cc:493] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:45.281718 22181 consensus_meta.cc:287] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.910s	user 0.009s	sys 0.003s
I20260709 12:22:45.221367 22464 raft_consensus.cc:515] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:45.337157 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:45.320173 22404 raft_consensus.cc:2749] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:22:45.253082 22398 raft_consensus.cc:515] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:45.245606 22448 raft_consensus.cc:515] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:45.318352 17758 rpcz_store.cc:276] 0709 12:22:42.536315 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:42.558047 (+ 21732us) service_pool.cc:224] Handling call
0709 12:22:42.558346 (+   299us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:42.558643 (+   297us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:42.558668 (+    25us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:42.559131 (+   463us) log.cc:844] Serialized 26 byte log entry
0709 12:22:43.235780 (+676649us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:22:43.235813 (+    33us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:43.235901 (+    88us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:45.288752 (+2052851us) spinlock_profiling.cc:243] Waited 8.21 ms on lock 0x7b4400022b4c. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b47459c85 00007f7b4745a6a2 00007f7b4a1cc1db 00007f7b4a1c7aaf 00007f7b4a1bb86e 00007f7b4a1babb6 00007f7b4a1ba7d8 00007f7b4a1c4221 00007f7b4a1c41d9 00007f7b4a1c4169
0709 12:22:45.317674 (+ 28922us) raft_consensus.cc:1685] finished
0709 12:22:45.317694 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:45.317875 (+   181us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":230,"raft.run_cpu_time_us":86,"raft.run_wall_time_us":87,"spinlock_wait_cycles":18887424,"thread_start_us":676295,"threads_started":1,"wal-append.queue_time_us":819880}
I20260709 12:22:45.311865 22468 raft_consensus.cc:493] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:45.295197 22485 leader_election.cc:290] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:45.319300 22358 raft_consensus.cc:2749] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:22:45.260505 22042 raft_consensus.cc:515] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:45.343674 17652 leader_election.cc:304] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:22:45.347237 17689 raft_consensus.cc:2368] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 10. Current term is 11.
I20260709 12:22:45.339402 22477 raft_consensus.cc:493] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:45.348068 17803 leader_election.cc:304] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:22:45.335079 22424 consensus_meta.cc:287] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 1.185s	user 0.013s	sys 0.000s
I20260709 12:22:45.338028 17763 raft_consensus.cc:1240] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 12 LEADER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 11. Current term is 12. Ops: []
I20260709 12:22:45.338693 22389 raft_consensus.cc:3060] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:45.390552 17756 raft_consensus.cc:2468] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 8.
W20260709 12:22:45.391919 22452 log.cc:927] Time spent T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.055s	user 0.000s	sys 0.000s
I20260709 12:22:45.280521 22193 raft_consensus.cc:515] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:45.351511 22416 leader_election.cc:290] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 13 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:45.284536 22406 consensus_queue.cc:1048] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.094s
I20260709 12:22:45.346081 22078 raft_consensus.cc:2749] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:22:45.333135 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:22:45.399629 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 56022.7us
I20260709 12:22:45.400210 22489 raft_consensus.cc:2804] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Leader pre-election won for term 13
W20260709 12:22:45.402220 22417 consensus_meta.cc:287] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 1.162s	user 0.008s	sys 0.002s
I20260709 12:22:45.342329 22472 raft_consensus.cc:493] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:22:45.341938 22496 raft_consensus.cc:515] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:45.396275 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 11 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:45.424252 17830 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 10.
I20260709 12:22:45.393401 22468 raft_consensus.cc:3060] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:45.280095 22493 raft_consensus.cc:515] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:45.399078 22306 raft_consensus.cc:2804] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:22:45.414732 17680 raft_consensus.cc:2468] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 13.
I20260709 12:22:45.274552 22482 raft_consensus.cc:515] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:45.418267 17652 leader_election.cc:304] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
W20260709 12:22:45.406675 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 58676.6us
I20260709 12:22:45.433389 22497 raft_consensus.cc:493] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:45.425973 22472 raft_consensus.cc:515] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:45.433845 17803 leader_election.cc:304] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:45.434018 22497 raft_consensus.cc:515] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:45.408643 22477 raft_consensus.cc:515] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:45.436261 22479 consensus_queue.cc:1048] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:22:45.355681 22181 raft_consensus.cc:515] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:45.413085 22473 raft_consensus.cc:2749] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 12:22:45.412350 22404 raft_consensus.cc:493] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:45.355332 22384 raft_consensus.cc:515] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:45.420377 22489 raft_consensus.cc:493] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:45.443013 22406 raft_consensus.cc:2804] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:22:45.408367 22424 raft_consensus.cc:515] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:22:45.449086 22335 consensus_queue.cc:1048] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:22:45.438803 17652 leader_election.cc:304] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
W20260709 12:22:45.537449 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 98733.1us
I20260709 12:22:45.453035 17831 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35042 (request call id 2195) took 1476 ms. Trace:
I20260709 12:22:45.423846 22416 consensus_queue.cc:1059] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.028s
I20260709 12:22:45.485383 22384 leader_election.cc:290] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:45.487501 22477 leader_election.cc:290] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:45.489843 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 13 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:45.490082 22403 leader_election.cc:290] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:22:45.490813 22346 consensus_meta.cc:287] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.898s	user 0.011s	sys 0.000s
I20260709 12:22:45.492161 22467 leader_election.cc:290] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:45.493304 22493 leader_election.cc:290] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:45.494511 22464 leader_election.cc:290] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:45.496196 22045 leader_election.cc:290] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:45.497658 22496 leader_election.cc:290] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:45.498898 22472 leader_election.cc:290] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:45.431710 22306 raft_consensus.cc:493] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:45.538882 17831 rpcz_store.cc:276] 0709 12:22:43.976194 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:43.989711 (+ 13517us) service_pool.cc:224] Handling call
0709 12:22:43.990168 (+   457us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:22:43.991029 (+   861us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:43.991079 (+    50us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:22:43.991921 (+   842us) log.cc:844] Serialized 52 byte log entry
0709 12:22:44.218648 (+226727us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:22:44.474588 (+255940us) log.cc:844] Serialized 14 byte log entry
0709 12:22:44.601143 (+126555us) log.cc:844] Serialized 14 byte log entry
0709 12:22:44.601388 (+   245us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:44.601502 (+   114us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:45.452820 (+851318us) raft_consensus.cc:1685] finished
0709 12:22:45.452844 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:45.452975 (+   131us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":45227,"prepare.queue_time_us":295560,"prepare.run_cpu_time_us":88,"prepare.run_wall_time_us":1184,"raft.queue_time_us":170003,"raft.run_cpu_time_us":351,"raft.run_wall_time_us":89449,"spinlock_wait_cycles":10684672,"thread_start_us":495327,"threads_started":3,"wal-append.queue_time_us":635808}
W20260709 12:22:45.570240 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 224768us
I20260709 12:22:45.654347 17802 leader_election.cc:400] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 pre-election: Vote denied by peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071) with higher term. Message: Invalid argument: T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 10. Current term is 11.
I20260709 12:22:45.654603 17802 leader_election.cc:403] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:22:45.416837 22374 raft_consensus.cc:515] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:45.607187 18363 raft_consensus.cc:1240] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 11. Current term is 12. Ops: []
I20260709 12:22:45.610874 22500 raft_consensus.cc:493] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:45.608436 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:45.614432 22358 raft_consensus.cc:493] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:45.505731 22489 raft_consensus.cc:3060] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:45.503939 22406 raft_consensus.cc:493] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:45.454149 22473 raft_consensus.cc:2749] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:22:45.611490 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:45.612931 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:45.613796 17763 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 12.
I20260709 12:22:45.615031 18367 raft_consensus.cc:1275] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20260709 12:22:45.616717 17833 raft_consensus.cc:1240] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 10. Current term is 11. Ops: []
I20260709 12:22:45.620664 17830 raft_consensus.cc:1275] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 8 index: 10. Preceding OpId from leader: term: 9 index: 10. (term mismatch)
I20260709 12:22:45.503432 22193 leader_election.cc:290] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:45.544301 22502 raft_consensus.cc:493] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:45.502210 22497 leader_election.cc:290] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:45.628058 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:45.500916 22448 leader_election.cc:290] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:45.425140 22417 raft_consensus.cc:515] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:45.536036 22479 raft_consensus.cc:493] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:45.625360 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:45.680394 17761 raft_consensus.cc:2393] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 10: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:22:45.575264 22307 raft_consensus.cc:2804] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:22:45.648736 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:45.682097 18361 raft_consensus.cc:3060] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:45.635351 18362 raft_consensus.cc:1275] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [term 10 LEARNER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20260709 12:22:45.651135 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:45.685139 22398 leader_election.cc:290] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 11 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:45.675158 22406 raft_consensus.cc:3060] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:45.685958 22417 leader_election.cc:290] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 9 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:45.413865 22192 consensus_queue.cc:237] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.9, Last appended by leader: 9, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:45.454320 22404 raft_consensus.cc:515] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:45.610970 22345 raft_consensus.cc:493] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:45.664563 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 13 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:45.659233 22403 raft_consensus.cc:493] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:45.667131 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:22:45.673954 17655 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:45.671801 18368 raft_consensus.cc:3060] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:45.673026 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 13 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:45.670720 17764 raft_consensus.cc:1275] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 10 index: 7. Preceding OpId from leader: term: 12 index: 8. (index mismatch)
I20260709 12:22:45.672924 17830 pending_rounds.cc:85] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b: Aborting all ops after (but not including) 9
I20260709 12:22:45.697512 17830 pending_rounds.cc:107] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b: Aborting uncommitted NO_OP operation due to leader change: 8.10
I20260709 12:22:45.697947 17830 raft_consensus.cc:2889] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:45.652185 22306 raft_consensus.cc:3060] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:45.630466 18365 raft_consensus.cc:1240] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 10. Current term is 11. Ops: []
I20260709 12:22:45.688282 18374 raft_consensus.cc:1240] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 8. Current term is 9. Ops: []
I20260709 12:22:45.689035 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:45.658444 18364 raft_consensus.cc:3060] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:22:45.668970 21888 consensus_meta.cc:287] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 1.269s	user 0.008s	sys 0.004s
W20260709 12:22:45.699167 17804 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:45.624971 22346 raft_consensus.cc:515] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:45.667734 22472 raft_consensus.cc:2749] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Leader pre-election lost for term 10. Reason: Vote denied by peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071) with higher term. Message: Invalid argument: T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 10. Current term is 11.
I20260709 12:22:45.686779 22479 raft_consensus.cc:515] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:45.717864 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:45.718561 18367 raft_consensus.cc:2468] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 13.
I20260709 12:22:45.720640 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:45.721573 18363 raft_consensus.cc:2393] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 9: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:22:45.719540 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:45.737711 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:45.703977 22307 raft_consensus.cc:697] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 11 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:22:45.741487 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:45.743974 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:45.671191 22358 raft_consensus.cc:515] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:45.708017 22345 raft_consensus.cc:515] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:45.720530 22417 raft_consensus.cc:493] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:45.735176 22390 consensus_meta.cc:287] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 1.092s	user 0.011s	sys 0.001s
I20260709 12:22:45.747468 17830 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90"
dest_uuid: "e063541542a94969898868bf6eb5ce1b"
mode: GRACEFUL
new_leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
 from {username='slave'} at 127.0.0.1:59820
W20260709 12:22:45.745651 22506 log.cc:927] Time spent T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.084s	user 0.000s	sys 0.001s
I20260709 12:22:45.758119 18364 raft_consensus.cc:2468] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 11.
I20260709 12:22:45.764886 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:45.771128 18367 raft_consensus.cc:2468] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 12.
I20260709 12:22:45.733682 22042 leader_election.cc:290] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:45.755775 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 14 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:45.773288 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:45.774250 17761 raft_consensus.cc:2393] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 10: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:22:45.705502 22403 raft_consensus.cc:515] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:45.767681 17830 raft_consensus.cc:606] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Received request to transfer leadership to TS e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:45.778342 17830 raft_consensus.cc:614] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:22:45.768651 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 13 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:22:45.746495 22402 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.534s	user 0.000s	sys 0.010s
I20260709 12:22:45.768482 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:45.785094 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:45.786110 17764 raft_consensus.cc:2393] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 10: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:22:45.670670 22500 raft_consensus.cc:515] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:45.788079 18366 raft_consensus.cc:2468] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 8.
I20260709 12:22:45.790659 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:45.770164 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:45.794219 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:45.779779 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:45.796833 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:45.708189 21888 raft_consensus.cc:515] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:45.690894 22502 raft_consensus.cc:515] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:45.801807 22502 leader_election.cc:290] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:45.745277 22477 raft_consensus.cc:493] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:45.795603 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:45.791756 22500 leader_election.cc:290] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:45.803813 22477 raft_consensus.cc:515] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:45.807435 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:45.816399 17758 raft_consensus.cc:1240] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 8. Current term is 9. Ops: []
I20260709 12:22:45.819375 17764 raft_consensus.cc:2393] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 10: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
W20260709 12:22:45.827553 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 496363us
W20260709 12:22:45.784116 22402 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.613s	user 0.000s	sys 0.010s
W20260709 12:22:45.824659 17653 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.593s	user 0.015s	sys 0.020s
I20260709 12:22:45.771896 22181 leader_election.cc:290] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 14 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:45.832160 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
W20260709 12:22:45.839442 17653 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.612s	user 0.015s	sys 0.020s
I20260709 12:22:45.840116 17763 raft_consensus.cc:2393] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 12: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
W20260709 12:22:45.834560 17727 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:45.840655 17727 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:45.831179 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:45.837683 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:22:45.848306 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:45.859468 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:45.879623 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:45.826378 22498 raft_consensus.cc:493] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:45.886929 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:45.897267 17764 raft_consensus.cc:2368] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 13. Current term is 14.
I20260709 12:22:45.770527 22417 raft_consensus.cc:515] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:45.824105 22495 raft_consensus.cc:493] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:45.893050 17758 raft_consensus.cc:1217] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Deduplicated request from leader. Original: 11.10->[12.11-12.11]   Dedup: 12.11->[]
I20260709 12:22:45.774178 22390 raft_consensus.cc:515] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:22:45.885083 22503 log.cc:927] Time spent T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.262s	user 0.000s	sys 0.001s
I20260709 12:22:45.901566 17758 raft_consensus.cc:1240] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 12. Current term is 13. Ops: [12.11-12.11]
W20260709 12:22:45.904583 17838 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.511s	user 0.001s	sys 0.013s
W20260709 12:22:45.905117 17838 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.511s	user 0.002s	sys 0.013s
I20260709 12:22:45.902367 18368 raft_consensus.cc:2468] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 13.
I20260709 12:22:45.905889 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:45.906953 17830 raft_consensus.cc:2468] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 13.
W20260709 12:22:45.915752 17762 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.523s	user 0.020s	sys 0.027s
I20260709 12:22:45.915542 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
W20260709 12:22:45.916163 17762 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.526s	user 0.020s	sys 0.027s
I20260709 12:22:45.917575 18363 raft_consensus.cc:1240] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 13. Current term is 14. Ops: []
I20260709 12:22:45.923941 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:45.926436 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:45.927374 17830 raft_consensus.cc:2468] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 8.
I20260709 12:22:45.930675 22501 raft_consensus.cc:493] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:22:45.931593 22501 raft_consensus.cc:515] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:45.946456 17758 raft_consensus.cc:1240] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 9. Current term is 10. Ops: []
I20260709 12:22:45.946892 17830 raft_consensus.cc:1240] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 9. Current term is 10. Ops: []
I20260709 12:22:45.909950 22482 leader_election.cc:290] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:45.904846 22181 consensus_queue.cc:1048] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.031s
I20260709 12:22:45.919553 22374 leader_election.cc:290] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 9 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:45.973325 17729 leader_election.cc:304] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:45.977564 17729 leader_election.cc:304] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:22:45.911945 22307 consensus_queue.cc:237] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:45.999076 22509 raft_consensus.cc:2749] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:22:45.971474 22345 leader_election.cc:290] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:46.006178 18361 raft_consensus.cc:2468] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 11.
I20260709 12:22:46.008069 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:46.008942 18361 raft_consensus.cc:3060] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:45.954151 22346 leader_election.cc:290] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 12 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:45.976413 22377 raft_consensus.cc:493] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:45.912721 22498 raft_consensus.cc:515] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:46.019102 17729 leader_election.cc:304] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:22:46.020030 22509 raft_consensus.cc:2749] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:22:46.020458 17729 leader_election.cc:304] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:46.020870 22509 raft_consensus.cc:2749] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:22:46.022377 17829 raft_consensus.cc:1240] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 10. Current term is 11. Ops: []
I20260709 12:22:45.917851 22495 raft_consensus.cc:515] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:46.025594 17833 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35040 (request call id 2451) took 1353 ms. Trace:
I20260709 12:22:46.026448 17833 rpcz_store.cc:276] 0709 12:22:44.672350 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:44.672534 (+   184us) service_pool.cc:224] Handling call
0709 12:22:44.672885 (+   351us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:45.674299 (+1001414us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:46.022015 (+347716us) spinlock_profiling.cc:243] Waited 944 ms on lock 0x7b540008a7d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:22:46.025529 (+  3514us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2170126592}
W20260709 12:22:46.037209 22511 log.cc:927] Time spent T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.112s	user 0.000s	sys 0.001s
W20260709 12:22:46.037855 22508 log.cc:927] Time spent T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.172s	user 0.001s	sys 0.000s
I20260709 12:22:46.024389 22377 raft_consensus.cc:515] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:22:46.024937 22404 leader_election.cc:290] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:46.026367 22509 raft_consensus.cc:2804] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Leader pre-election won for term 13
W20260709 12:22:46.042703 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 470977us
W20260709 12:22:46.044113 22451 log.cc:927] Time spent T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 1.571s	user 0.007s	sys 0.022s
I20260709 12:22:46.063328 18361 raft_consensus.cc:2468] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 14.
I20260709 12:22:46.072854 22495 leader_election.cc:290] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:46.073516 18361 raft_consensus.cc:1240] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 11. Current term is 12. Ops: []
I20260709 12:22:46.033584 22403 leader_election.cc:290] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:46.069358 17729 leader_election.cc:304] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:46.046185 22509 raft_consensus.cc:493] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:46.031065 22510 raft_consensus.cc:493] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
W20260709 12:22:46.076357 22389 consensus_meta.cc:287] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.670s	user 0.012s	sys 0.000s
I20260709 12:22:46.046387 22479 leader_election.cc:290] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:22:46.078648 22397 consensus_meta.cc:287] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.778s	user 0.009s	sys 0.002s
W20260709 12:22:46.084916 22512 log.cc:927] Time spent T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.076s	user 0.000s	sys 0.000s
I20260709 12:22:46.091729 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:22:46.095712 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 200229us
I20260709 12:22:46.083328 22417 leader_election.cc:290] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:46.083709 22509 raft_consensus.cc:3060] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:46.101385 17833 raft_consensus.cc:2468] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 13.
W20260709 12:22:46.088824 22520 log.cc:927] Time spent T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.121s	user 0.001s	sys 0.000s
I20260709 12:22:46.044893 22422 consensus_queue.cc:260] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:46.109747 22422 raft_consensus.cc:3060] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:46.084936 22498 leader_election.cc:290] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:46.109102 18362 raft_consensus.cc:1275] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
W20260709 12:22:46.090270 22514 log.cc:927] Time spent T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.073s	user 0.001s	sys 0.001s
I20260709 12:22:46.089406 22377 leader_election.cc:290] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:46.113802 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:46.118042 17830 raft_consensus.cc:2393] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 11: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:22:46.112959 17654 leader_election.cc:304] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:22:46.097075 22390 leader_election.cc:290] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 14 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:46.118882 18362 raft_consensus.cc:1240] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 11. Current term is 12. Ops: []
W20260709 12:22:46.122936 22525 log.cc:927] Time spent T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.072s	user 0.001s	sys 0.000s
I20260709 12:22:46.119052 17654 leader_election.cc:304] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:22:46.132922 18362 raft_consensus.cc:1240] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 12 LEADER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 11. Current term is 12. Ops: []
W20260709 12:22:46.116724 22476 consensus_meta.cc:287] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.761s	user 0.007s	sys 0.004s
I20260709 12:22:46.135710 17683 raft_consensus.cc:1240] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 10. Current term is 11. Ops: []
W20260709 12:22:46.117560 21994 consensus_meta.cc:287] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.777s	user 0.012s	sys 0.000s
I20260709 12:22:46.096876 22389 consensus_queue.cc:1059] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 9.6, Next index: 7, Last known committed idx: 4, Time since last communication: 0.021s
I20260709 12:22:46.107642 22437 consensus_queue.cc:1048] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.022s
I20260709 12:22:46.090966 22397 raft_consensus.cc:515] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:46.138655 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:46.140681 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
W20260709 12:22:46.110517 22307 consensus_queue.cc:1175] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "334a4e1f59944544b3b3ff36616b57da" responder_term: 12 status { last_received { term: 11 index: 10 } last_committed_idx: 10 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 11. Current term is 12. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260709 12:22:46.166205 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:46.100258 22510 raft_consensus.cc:515] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:46.117429 22181 raft_consensus.cc:2804] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:22:46.119755 22516 consensus_queue.cc:1059] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 10.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.029s
W20260709 12:22:46.167088 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 112712us
I20260709 12:22:46.166841 17688 raft_consensus.cc:2468] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 13.
I20260709 12:22:46.167575 18362 raft_consensus.cc:1240] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 12. Current term is 13. Ops: []
I20260709 12:22:46.166767 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:46.116465 22365 consensus_queue.cc:1048] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.051s
I20260709 12:22:46.173506 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:46.176735 17681 raft_consensus.cc:3060] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:46.176803 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 11 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:46.173869 18367 raft_consensus.cc:1240] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 7. Current term is 8. Ops: []
I20260709 12:22:46.177862 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:46.178134 18362 raft_consensus.cc:1240] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 11. Current term is 12. Ops: []
I20260709 12:22:46.174702 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:46.179275 17688 raft_consensus.cc:3060] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:46.179711 18362 raft_consensus.cc:1240] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 12. Current term is 13. Ops: []
I20260709 12:22:46.182078 18362 raft_consensus.cc:1240] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 7. Current term is 8. Ops: []
I20260709 12:22:46.183224 22523 raft_consensus.cc:2804] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:22:46.184177 22341 leader_election.cc:290] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 10 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:46.186546 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:46.188848 22424 leader_election.cc:290] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 11 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:46.189121 17829 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35040 (request call id 2468) took 1120 ms. Trace:
I20260709 12:22:46.192508 16501 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 5
W20260709 12:22:46.193567 22530 log.cc:927] Time spent T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.055s	user 0.000s	sys 0.000s
I20260709 12:22:46.193935 22306 raft_consensus.cc:515] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:46.164547 21994 raft_consensus.cc:515] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:46.170655 22384 raft_consensus.cc:3055] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 LEADER]: Stepping down as leader of term 7
W20260709 12:22:46.197338 22447 log.cc:927] Time spent T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 1.745s	user 0.021s	sys 0.035s
I20260709 12:22:46.201157 22423 leader_election.cc:290] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 11 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:46.229600 22384 raft_consensus.cc:740] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:22:46.203068 17681 raft_consensus.cc:2468] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 14.
I20260709 12:22:46.204716 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:46.230700 22384 consensus_queue.cc:260] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:46.231189 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:46.231453 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:46.232543 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:46.232887 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:46.233134 22306 leader_election.cc:290] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 9 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:46.233634 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:46.213510 17755 raft_consensus.cc:2468] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 12.
I20260709 12:22:46.204613 17829 rpcz_store.cc:276] 0709 12:22:45.068810 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:45.299074 (+230264us) service_pool.cc:224] Handling call
0709 12:22:46.022177 (+723103us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:46.022809 (+   632us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:46.188938 (+166129us) spinlock_profiling.cc:243] Waited 723 ms on lock 0x7b540008a7d0. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a61f8ad 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:22:46.189064 (+   126us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1662406400}
I20260709 12:22:46.216326 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:46.215301 17727 leader_election.cc:304] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:22:46.258697 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 43483.1us
I20260709 12:22:46.169461 22495 raft_consensus.cc:2804] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:22:46.240754 22358 leader_election.cc:290] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:22:46.241271 22489 consensus_meta.cc:287] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.563s	user 0.005s	sys 0.004s
I20260709 12:22:46.243028 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 11 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:22:46.268477 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 199264us
I20260709 12:22:46.181597 22181 raft_consensus.cc:697] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 14 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:22:46.245668 21994 leader_election.cc:290] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 12 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:22:46.246567 22484 log.cc:927] Time spent T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 1.130s	user 0.005s	sys 0.007s
I20260709 12:22:46.222215 22523 raft_consensus.cc:697] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 13 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:22:46.259538 17683 raft_consensus.cc:1240] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 11. Current term is 12. Ops: []
I20260709 12:22:46.258589 17680 raft_consensus.cc:2468] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 13.
I20260709 12:22:46.282541 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
W20260709 12:22:46.216373 22441 log.cc:927] Time spent T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 1.726s	user 0.006s	sys 0.038s
I20260709 12:22:46.325922 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:46.139906 22476 raft_consensus.cc:515] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:46.259577 17727 leader_election.cc:304] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:46.268848 18368 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 10.
I20260709 12:22:46.245103 17835 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35040 (request call id 2459) took 1523 ms. Trace:
W20260709 12:22:46.269333 17729 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:46.274986 17688 raft_consensus.cc:2468] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 9.
I20260709 12:22:46.328940 17835 rpcz_store.cc:276] 0709 12:22:44.722048 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:45.186602 (+464554us) service_pool.cc:224] Handling call
0709 12:22:45.204169 (+ 17567us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:45.264051 (+ 59882us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:45.264087 (+    36us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:45.264128 (+    41us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:22:45.264164 (+    36us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:46.244859 (+980695us) spinlock_profiling.cc:243] Waited 59.7 ms on lock 0x7b54000ab9d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:22:46.244907 (+    48us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:46.245031 (+   124us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":137217920}
I20260709 12:22:46.329764 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:22:46.334926 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 75487.2us
I20260709 12:22:46.267066 22418 raft_consensus.cc:2804] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Leader pre-election won for term 14
W20260709 12:22:46.330135 22468 consensus_meta.cc:287] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.882s	user 0.004s	sys 0.008s
W20260709 12:22:46.347394 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:46.355786 17728 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.16.29.65:33071 timed out after 1.828s (SENT) [suppressed 42 similar messages]
I20260709 12:22:46.343122 22418 raft_consensus.cc:493] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:46.285528 22495 raft_consensus.cc:697] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 11 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:22:46.362391 22477 leader_election.cc:290] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:46.277061 22489 raft_consensus.cc:515] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:22:46.381307 22406 consensus_meta.cc:287] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.695s	user 0.010s	sys 0.002s
I20260709 12:22:46.379559 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:46.425386 22406 raft_consensus.cc:515] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:46.369269 22523 consensus_queue.cc:237] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:46.426641 17684 raft_consensus.cc:2393] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 9: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:22:46.411162 17834 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35040 (request call id 2455) took 1702 ms. Trace:
I20260709 12:22:46.422479 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:46.360402 22494 raft_consensus.cc:2804] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:22:46.429273 17834 rpcz_store.cc:276] 0709 12:22:44.708926 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:44.709112 (+   186us) service_pool.cc:224] Handling call
0709 12:22:44.709534 (+   422us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:44.709815 (+   281us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:44.709854 (+    39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:44.716765 (+  6911us) log.cc:844] Serialized 26 byte log entry
0709 12:22:45.077956 (+361191us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:22:45.077995 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:45.078069 (+    74us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:46.245938 (+1167869us) spinlock_profiling.cc:243] Waited 14.8 ms on lock 0x7b4400022b4c. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b47459c85 00007f7b4745a6a2 00007f7b4a1cc1db 00007f7b4a1c7aaf 00007f7b4a1bb86e 00007f7b4a1babb6 00007f7b4a1ba7d8 00007f7b4a1c4221 00007f7b4a1c41d9 00007f7b4a1c4169
0709 12:22:46.407164 (+161226us) raft_consensus.cc:1685] finished
0709 12:22:46.407199 (+    35us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:46.411061 (+  3862us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":34071808,"thread_start_us":282752,"threads_started":1,"wal-append.queue_time_us":381934}
I20260709 12:22:46.362746 22418 raft_consensus.cc:3060] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:46.440339 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:46.441439 18371 raft_consensus.cc:2393] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 9: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:22:46.440634 18368 raft_consensus.cc:1275] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 7 index: 9. Preceding OpId from leader: term: 9 index: 10. (index mismatch)
I20260709 12:22:46.441591 18363 raft_consensus.cc:1275] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 11 index: 9. Preceding OpId from leader: term: 12 index: 10. (index mismatch)
I20260709 12:22:46.444845 17728 leader_election.cc:304] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:22:46.445705 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:22:46.386094 22531 log.cc:927] Time spent T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.215s	user 0.001s	sys 0.000s
I20260709 12:22:46.450073 18366 raft_consensus.cc:1240] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 10. Current term is 11. Ops: []
I20260709 12:22:46.447911 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 12 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:46.452756 18334 leader_election.cc:304] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:22:46.371842 22501 leader_election.cc:290] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:46.459394 17654 leader_election.cc:304] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:22:46.409512 22192 raft_consensus.cc:493] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:46.386437 22476 leader_election.cc:290] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 9 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:46.463985 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:46.464630 22501 consensus_queue.cc:1059] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 11.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:22:46.437831 22494 raft_consensus.cc:493] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:46.468091 17764 raft_consensus.cc:3060] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:46.369966 22377 consensus_queue.cc:1048] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.092s
I20260709 12:22:46.365008 22468 raft_consensus.cc:515] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:46.369148 22532 consensus_queue.cc:1059] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 9.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.095s
I20260709 12:22:46.378731 22533 consensus_queue.cc:1059] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 11.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.102s
I20260709 12:22:46.470986 22192 raft_consensus.cc:515] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:46.477483 22406 leader_election.cc:290] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 9 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:46.478103 22406 raft_consensus.cc:2804] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:22:46.478447 22406 raft_consensus.cc:493] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:46.478806 22406 raft_consensus.cc:3060] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:46.468546 22495 consensus_queue.cc:237] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.9, Last appended by leader: 9, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:46.463532 22417 raft_consensus.cc:2804] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:22:46.368085 22535 consensus_queue.cc:1059] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 11.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.101s
I20260709 12:22:46.469904 22416 raft_consensus.cc:2749] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:22:46.445554 22356 consensus_queue.cc:1059] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 12.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.027s
I20260709 12:22:46.418390 22181 consensus_queue.cc:237] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:46.471479 22494 raft_consensus.cc:3060] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:46.471273 22397 leader_election.cc:290] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 8 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:46.465996 22133 consensus_queue.cc:1059] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.020s
I20260709 12:22:46.491407 22532 raft_consensus.cc:3055] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:22:46.492084 22417 raft_consensus.cc:697] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 9 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:22:46.498654 22383 raft_consensus.cc:3055] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:22:46.487983 22377 consensus_queue.cc:1059] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 12.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.024s
I20260709 12:22:46.501694 22510 leader_election.cc:290] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:46.493235 22501 consensus_queue.cc:1059] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.025s
I20260709 12:22:46.512624 22532 raft_consensus.cc:740] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
W20260709 12:22:46.527827 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 159057us
I20260709 12:22:46.533881 17764 raft_consensus.cc:2468] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 11.
I20260709 12:22:46.521169 22383 raft_consensus.cc:740] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 11 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:22:46.536351 17727 leader_election.cc:304] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:22:46.539021 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 95949us
W20260709 12:22:46.540867 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 65804.2us
W20260709 12:22:46.544337 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:46.533806 22425 raft_consensus.cc:3055] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:22:46.555493 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:22:46.559254 17727 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:46.561496 17834 raft_consensus.cc:1240] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 7. Current term is 8. Ops: []
I20260709 12:22:46.571307 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:46.567642 22397 raft_consensus.cc:493] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
W20260709 12:22:46.573571 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 73288.3us
I20260709 12:22:46.573405 22397 raft_consensus.cc:515] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:46.575337 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:46.553539 22542 raft_consensus.cc:2804] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:22:46.551367 22425 raft_consensus.cc:740] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
W20260709 12:22:46.571026 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 126277us
I20260709 12:22:46.596592 22425 consensus_queue.cc:260] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:46.598167 22425 raft_consensus.cc:3060] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:46.599152 22042 raft_consensus.cc:493] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:46.599817 22042 raft_consensus.cc:515] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:46.601423 17688 raft_consensus.cc:2468] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 12.
W20260709 12:22:46.602648 22539 log.cc:927] Time spent T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.102s	user 0.000s	sys 0.000s
I20260709 12:22:46.609982 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 11 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:46.611307 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:46.614528 22389 consensus_queue.cc:1059] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:22:46.572505 22489 leader_election.cc:290] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 13 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:46.617786 22192 leader_election.cc:290] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:46.570245 22532 consensus_queue.cc:260] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 11.11, Last appended by leader: 11, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:46.619617 22532 raft_consensus.cc:3060] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:46.620976 17756 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38776 (request call id 2227) took 1887 ms. Trace:
I20260709 12:22:46.622054 17756 rpcz_store.cc:276] 0709 12:22:44.733784 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:44.744626 (+ 10842us) service_pool.cc:224] Handling call
0709 12:22:46.620822 (+1876196us) spinlock_profiling.cc:243] Waited 239 ms on lock 0x7b5400415ad4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:22:46.620923 (+   101us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":548908288}
I20260709 12:22:46.580636 22383 consensus_queue.cc:260] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:22:46.625728 22509 consensus_meta.cc:287] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.511s	user 0.005s	sys 0.005s
I20260709 12:22:46.625965 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:46.626226 22509 raft_consensus.cc:515] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:46.627075 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:46.548141 22417 consensus_queue.cc:237] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:46.633744 21888 leader_election.cc:290] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 17 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:46.639997 17802 leader_election.cc:304] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:22:46.642791 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:46.643949 17686 raft_consensus.cc:2393] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 13: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:22:46.579077 22542 raft_consensus.cc:493] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:46.645965 17802 leader_election.cc:304] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:22:46.649300 17682 raft_consensus.cc:1240] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 13 LEADER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 12. Current term is 13. Ops: []
I20260709 12:22:46.652089 22114 raft_consensus.cc:2804] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:22:46.652603 22114 raft_consensus.cc:493] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:46.653030 22114 raft_consensus.cc:3060] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:46.660739 22468 leader_election.cc:290] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 13 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:46.664057 22542 raft_consensus.cc:3060] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:46.664325 22518 raft_consensus.cc:3055] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:22:46.665563 22518 raft_consensus.cc:740] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 9 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:22:46.666509 22518 consensus_queue.cc:260] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:46.647099 22475 raft_consensus.cc:2804] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:22:46.668227 22475 raft_consensus.cc:493] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:46.668582 22475 raft_consensus.cc:3060] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:46.667531 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:46.672956 22069 raft_consensus.cc:3055] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:22:46.673581 22069 raft_consensus.cc:740] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:22:46.674737 22069 consensus_queue.cc:260] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:22:46.690475 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 227146us
I20260709 12:22:46.691519 17756 raft_consensus.cc:1275] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 12 index: 9. (index mismatch)
I20260709 12:22:46.695716 17764 raft_consensus.cc:1240] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 12. Current term is 13. Ops: []
I20260709 12:22:46.721303 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:46.722469 17685 raft_consensus.cc:1240] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 12. Current term is 13. Ops: []
I20260709 12:22:46.726167 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 17 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:46.727208 17756 raft_consensus.cc:3060] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:22:46.725258 18367 raft_consensus.cc:1275] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 12 index: 9. (index mismatch)
W20260709 12:22:46.729328 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 150522us
I20260709 12:22:46.729979 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:46.734345 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 17 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:46.737382 18367 raft_consensus.cc:3060] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Advancing to term 17
I20260709 12:22:46.737789 18365 raft_consensus.cc:1275] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 13 index: 11. Preceding OpId from leader: term: 14 index: 11. (term mismatch)
I20260709 12:22:46.699683 22509 leader_election.cc:290] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 13 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:46.738142 18365 pending_rounds.cc:85] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da: Aborting all ops after (but not including) 10
I20260709 12:22:46.736519 17728 leader_election.cc:304] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
W20260709 12:22:46.732456 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 189718us
I20260709 12:22:46.756433 18365 pending_rounds.cc:107] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da: Aborting uncommitted NO_OP operation due to leader change: 13.11
I20260709 12:22:46.756901 18365 raft_consensus.cc:2889] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:46.756990 17729 leader_election.cc:304] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:22:46.757248 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:46.739101 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:46.734856 18362 raft_consensus.cc:1275] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 8 index: 7. Preceding OpId from leader: term: 10 index: 8. (index mismatch)
I20260709 12:22:46.725442 22069 raft_consensus.cc:3060] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:46.735280 18366 raft_consensus.cc:1275] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 13 index: 10. (term mismatch)
I20260709 12:22:46.764400 18366 pending_rounds.cc:85] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da: Aborting all ops after (but not including) 9
I20260709 12:22:46.764889 18366 pending_rounds.cc:107] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da: Aborting uncommitted NO_OP operation due to leader change: 11.10
I20260709 12:22:46.765242 18366 raft_consensus.cc:2889] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:46.703810 22335 consensus_queue.cc:1048] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.052s
I20260709 12:22:46.769660 22547 raft_consensus.cc:2749] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:22:46.735510 22518 raft_consensus.cc:3060] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:46.770587 22509 raft_consensus.cc:493] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:22:46.779834 17762 raft_consensus.cc:1275] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 13 index: 7. Preceding OpId from leader: term: 14 index: 8. (index mismatch)
I20260709 12:22:46.782020 18365 raft_consensus.cc:1240] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 9 LEADER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 8. Current term is 9. Ops: []
I20260709 12:22:46.782045 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 10 candidate_status { last_received { term: 9 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:46.786090 18367 raft_consensus.cc:2468] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 17.
I20260709 12:22:46.787622 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:46.788753 18367 raft_consensus.cc:3060] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:46.767750 17756 raft_consensus.cc:2468] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 17.
I20260709 12:22:46.731961 22418 raft_consensus.cc:515] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:46.719208 22435 consensus_queue.cc:1059] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 10.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.042s
I20260709 12:22:46.792433 17654 leader_election.cc:304] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:22:46.769956 17728 leader_election.cc:304] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:22:46.720042 22549 consensus_queue.cc:1048] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.069s
I20260709 12:22:46.736773 22399 consensus_queue.cc:1059] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.039s
I20260709 12:22:46.772614 18366 raft_consensus.cc:1240] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 10. Current term is 11. Ops: []
I20260709 12:22:46.767094 22533 raft_consensus.cc:2804] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:22:46.755697 22517 consensus_queue.cc:1048] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.034s
I20260709 12:22:46.793884 22307 consensus_queue.cc:1048] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.008s
I20260709 12:22:46.800249 22477 raft_consensus.cc:493] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:46.802366 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:46.803320 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
W20260709 12:22:46.808840 17728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:46.791088 22548 consensus_queue.cc:1048] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.033s
I20260709 12:22:46.782274 22003 consensus_queue.cc:1048] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.030s
I20260709 12:22:46.816110 22548 raft_consensus.cc:2804] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:22:46.816642 22548 raft_consensus.cc:697] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 17 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:22:46.822790 18367 raft_consensus.cc:2468] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 13.
I20260709 12:22:46.780077 22406 raft_consensus.cc:515] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:46.804376 22533 raft_consensus.cc:493] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:46.836452 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:22:46.840430 22553 log.cc:927] Time spent T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.073s	user 0.000s	sys 0.000s
I20260709 12:22:46.841058 22547 raft_consensus.cc:493] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:46.841588 22547 raft_consensus.cc:515] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:46.840806 22475 raft_consensus.cc:515] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:46.846349 22114 raft_consensus.cc:515] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:46.847223 22418 leader_election.cc:290] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 14 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:46.813236 22557 raft_consensus.cc:2749] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
W20260709 12:22:46.848166 22546 log.cc:927] Time spent T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.165s	user 0.001s	sys 0.000s
I20260709 12:22:46.791699 22509 raft_consensus.cc:515] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:46.794236 22494 raft_consensus.cc:515] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:46.850062 17729 leader_election.cc:304] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:22:46.893126 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:46.964841 17687 raft_consensus.cc:2393] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 9: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:22:46.965045 17762 raft_consensus.cc:2468] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 9.
I20260709 12:22:46.851017 22533 raft_consensus.cc:3060] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:46.971088 17727 leader_election.cc:304] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:22:46.995925 17756 raft_consensus.cc:2393] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 9: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:22:46.817471 22548 consensus_queue.cc:237] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 15.14, Last appended by leader: 14, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:22:47.012097 17655 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:47.018720 17655 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:47.019663 17655 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:46.884055 22475 leader_election.cc:290] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 14 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:46.887917 22181 raft_consensus.cc:493] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:46.821630 22477 raft_consensus.cc:515] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:47.032752 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:47.033149 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 17 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:46.887590 22418 raft_consensus.cc:493] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:47.038509 17761 raft_consensus.cc:2376] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 17. Re-sending same reply.
I20260709 12:22:47.039621 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:47.039842 22148 raft_consensus.cc:3055] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:22:47.040321 22148 raft_consensus.cc:740] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 11 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:22:47.040849 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:47.041585 22148 consensus_queue.cc:260] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:47.043165 22357 raft_consensus.cc:3060] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:47.043571 22148 raft_consensus.cc:3060] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:47.046118 18368 raft_consensus.cc:1240] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 12. Current term is 13. Ops: [12.10-12.10]
I20260709 12:22:47.046382 22133 raft_consensus.cc:3055] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:22:47.046746 22133 raft_consensus.cc:740] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 12 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:22:47.049645 17755 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36572 (request call id 820) took 1298 ms. Trace:
I20260709 12:22:47.050017 17755 rpcz_store.cc:276] 0709 12:22:45.750914 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:45.795288 (+ 44374us) service_pool.cc:224] Handling call
0709 12:22:47.049462 (+1254174us) spinlock_profiling.cc:243] Waited 283 ms on lock 0x7b54004150d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:22:47.049590 (+   128us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":651499776}
W20260709 12:22:47.052842 22481 log.cc:927] Time spent T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 1.936s	user 0.011s	sys 0.007s
I20260709 12:22:47.059608 18374 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 1498) took 1440 ms. Trace:
I20260709 12:22:47.060272 18374 rpcz_store.cc:276] 0709 12:22:45.618849 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:45.618972 (+   123us) service_pool.cc:224] Handling call
0709 12:22:45.619270 (+   298us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:45.704046 (+ 84776us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:47.059250 (+1355204us) spinlock_profiling.cc:243] Waited 68.6 ms on lock 0x7b54008020d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:22:47.059519 (+   269us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":157698432}
I20260709 12:22:47.064126 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:47.067749 22529 raft_consensus.cc:3055] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:22:47.068372 22529 raft_consensus.cc:740] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:22:47.069451 22529 consensus_queue.cc:260] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 10.11, Last appended by leader: 11, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:47.070788 22529 raft_consensus.cc:3060] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:47.072602 22535 raft_consensus.cc:3055] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:22:47.073134 22535 raft_consensus.cc:740] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:22:47.041601 17761 raft_consensus.cc:2468] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 10.
I20260709 12:22:47.080782 17682 raft_consensus.cc:1240] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 8. Current term is 9. Ops: []
I20260709 12:22:47.081686 22133 consensus_queue.cc:260] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.8, Last appended by leader: 8, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:47.083353 22133 raft_consensus.cc:3060] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:47.087671 22384 raft_consensus.cc:3060] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:47.094537 17689 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37402 (request call id 2669) took 1083 ms. Trace:
I20260709 12:22:47.101590 17836 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35042 (request call id 2205) took 2525 ms. Trace:
I20260709 12:22:47.102173 17836 rpcz_store.cc:276] 0709 12:22:44.576034 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:44.576834 (+   800us) service_pool.cc:224] Handling call
0709 12:22:44.595586 (+ 18752us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:44.638493 (+ 42907us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:47.101326 (+2462833us) spinlock_profiling.cc:243] Waited 42.1 ms on lock 0x7b54001581d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:22:47.101525 (+   199us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":96850304}
I20260709 12:22:47.095086 17689 rpcz_store.cc:276] 0709 12:22:46.011305 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:46.011469 (+   164us) service_pool.cc:224] Handling call
0709 12:22:46.022197 (+ 10728us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:46.211189 (+188992us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:46.211235 (+    46us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:46.211290 (+    55us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:22:46.211340 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:47.094245 (+882905us) spinlock_profiling.cc:243] Waited 189 ms on lock 0x7b5400206a54. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:22:47.094309 (+    64us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:47.094462 (+   153us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":434112768}
I20260709 12:22:47.051064 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:47.110734 22042 leader_election.cc:290] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:47.111126 17760 raft_consensus.cc:2468] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 13.
I20260709 12:22:47.119467 18361 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59886 (request call id 1357) took 1368 ms. Trace:
I20260709 12:22:47.119940 18361 rpcz_store.cc:276] 0709 12:22:45.750653 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:46.075314 (+324661us) service_pool.cc:224] Handling call
0709 12:22:46.209838 (+134524us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:46.210244 (+   406us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:46.210274 (+    30us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:46.210320 (+    46us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:22:46.210356 (+    36us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:46.210431 (+    75us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:47.119284 (+908853us) spinlock_profiling.cc:243] Waited 134 ms on lock 0x7b54006aed50. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a61f8ad 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:22:47.119400 (+   116us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":7305,"raft.run_cpu_time_us":140,"raft.run_wall_time_us":26211,"spinlock_wait_cycles":308383616}
I20260709 12:22:47.131613 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:47.135473 17831 raft_consensus.cc:1240] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 8. Current term is 9. Ops: []
I20260709 12:22:47.117497 16501 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
W20260709 12:22:47.144261 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:47.009701 22311 raft_consensus.cc:493] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:47.137046 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:47.153080 17831 raft_consensus.cc:2468] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 13.
I20260709 12:22:47.154600 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
W20260709 12:22:47.147462 18333 connection.cc:256] RPC call timeout handler was delayed by -1.9367677140003252s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17500000000000002s for a response
I20260709 12:22:47.166738 17830 raft_consensus.cc:1275] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 12 index: 9. (index mismatch)
I20260709 12:22:46.888445 22557 consensus_queue.cc:1059] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.259s
I20260709 12:22:47.021552 22558 raft_consensus.cc:2804] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:22:47.115073 22498 raft_consensus.cc:493] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:47.125780 22418 raft_consensus.cc:515] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:47.172890 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:47.182548 17832 raft_consensus.cc:2393] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 14: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:22:47.114562 22475 raft_consensus.cc:493] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:47.125077 22485 raft_consensus.cc:493] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:47.162540 22508 log.cc:927] Time spent T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.127s	user 0.000s	sys 0.001s
I20260709 12:22:47.153553 22416 raft_consensus.cc:2749] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20260709 12:22:47.170432 22530 log.cc:927] Time spent T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.133s	user 0.000s	sys 0.001s
W20260709 12:22:47.166267 22425 consensus_meta.cc:287] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.568s	user 0.011s	sys 0.000s
I20260709 12:22:47.154101 22547 leader_election.cc:290] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:22:47.195612 17727 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:47.206699 17727 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:47.115348 22181 raft_consensus.cc:515] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:47.165745 22114 leader_election.cc:290] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 14 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:47.202668 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:47.208180 17830 raft_consensus.cc:3060] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:22:47.212050 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 159630us
I20260709 12:22:47.194258 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:47.214290 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 12 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:47.217832 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:47.218755 18361 raft_consensus.cc:2468] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 12.
I20260709 12:22:47.167759 22311 raft_consensus.cc:515] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:22:47.214174 22555 log.cc:927] Time spent T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.413s	user 0.001s	sys 0.000s
I20260709 12:22:47.200220 22558 raft_consensus.cc:697] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:22:47.239943 18374 raft_consensus.cc:1275] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 11 index: 9. (index mismatch)
I20260709 12:22:47.245224 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:47.184434 22498 raft_consensus.cc:515] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:47.207252 22532 raft_consensus.cc:493] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:47.236687 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 92173.6us
I20260709 12:22:47.197882 22475 raft_consensus.cc:515] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:22:47.267836 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 227563us
I20260709 12:22:47.285913 17761 raft_consensus.cc:3055] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:22:47.286782 17761 raft_consensus.cc:740] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
W20260709 12:22:47.287002 17654 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:47.286917 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:47.286113 17763 raft_consensus.cc:1240] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 13. Current term is 14. Ops: []
I20260709 12:22:47.235898 22468 raft_consensus.cc:493] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:47.287470 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:47.200060 22485 raft_consensus.cc:515] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:47.290711 17837 raft_consensus.cc:2393] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 13: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:22:47.236553 22501 raft_consensus.cc:3055] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:22:47.300559 17831 raft_consensus.cc:2393] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 10: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:22:47.228554 22425 consensus_queue.cc:1059] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 11.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.060s
I20260709 12:22:47.301424 17654 leader_election.cc:304] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
W20260709 12:22:47.303526 22542 consensus_meta.cc:287] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.638s	user 0.005s	sys 0.006s
W20260709 12:22:47.314687 17803 leader_election.cc:336] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Timed out: RequestConsensusVote RPC to 127.16.29.68:42577 timed out after 1.617s (SENT)
I20260709 12:22:47.329402 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:47.330158 17803 leader_election.cc:304] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:47.330909 17837 raft_consensus.cc:3060] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:47.304451 17830 raft_consensus.cc:2468] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 9.
W20260709 12:22:47.309252 22575 log.cc:927] Time spent T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.075s	user 0.000s	sys 0.000s
I20260709 12:22:47.310771 22435 consensus_queue.cc:1059] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 8.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 12:22:47.334964 17832 raft_consensus.cc:3060] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:47.312991 22418 leader_election.cc:290] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:47.213028 22478 consensus_queue.cc:260] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:47.297600 22501 raft_consensus.cc:740] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:22:47.333307 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 13 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:47.302778 17761 consensus_queue.cc:260] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:47.366874 17761 raft_consensus.cc:3060] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:47.367823 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
W20260709 12:22:47.302811 22561 log.cc:927] Time spent T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.187s	user 0.000s	sys 0.001s
I20260709 12:22:47.382755 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:47.384101 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:47.384955 17682 raft_consensus.cc:2393] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 9: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:22:47.385293 17683 raft_consensus.cc:1240] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 12. Current term is 13. Ops: []
I20260709 12:22:47.390475 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:47.391388 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:47.391528 17684 raft_consensus.cc:2410] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for term 11 because replica has last-logged OpId of term: 9 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 8.
I20260709 12:22:47.391842 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 13 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:22:47.392431 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:47.313850 22406 leader_election.cc:290] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 9 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:47.339989 22489 raft_consensus.cc:2804] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:22:47.404685 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:47.408252 17830 raft_consensus.cc:2393] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 9: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
W20260709 12:22:47.444671 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 55161.1us
I20260709 12:22:47.453725 17837 raft_consensus.cc:2468] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 14.
I20260709 12:22:47.338450 22494 leader_election.cc:290] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 9 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:47.296516 22468 raft_consensus.cc:515] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:47.288416 22532 raft_consensus.cc:515] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:47.402597 22475 leader_election.cc:290] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:47.491256 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:22:47.468895 22575 log.cc:927] Time spent T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.089s	user 0.000s	sys 0.000s
I20260709 12:22:47.470419 17837 raft_consensus.cc:1240] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 8. Current term is 9. Ops: []
I20260709 12:22:47.475971 22572 raft_consensus.cc:493] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:47.480237 22501 consensus_queue.cc:260] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 13.12, Last appended by leader: 12, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:47.482053 22584 raft_consensus.cc:493] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:47.515229 18366 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.551s	user 0.010s	sys 0.037s
W20260709 12:22:47.515753 18366 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.564s	user 0.010s	sys 0.037s
I20260709 12:22:47.483454 22377 raft_consensus.cc:3055] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:22:47.483559 22181 leader_election.cc:290] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:22:47.485387 17804 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:47.486749 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 371536us
I20260709 12:22:47.413170 22478 raft_consensus.cc:3060] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:22:47.522501 22518 consensus_meta.cc:287] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.739s	user 0.008s	sys 0.002s
I20260709 12:22:47.495920 18368 raft_consensus.cc:1275] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 13 index: 12. (index mismatch)
I20260709 12:22:47.402868 22576 raft_consensus.cc:493] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:47.419437 22489 raft_consensus.cc:493] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:47.507632 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:22:47.509430 17829 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.537s	user 0.013s	sys 0.034s
I20260709 12:22:47.457835 22523 raft_consensus.cc:3055] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:22:47.334367 22542 raft_consensus.cc:515] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:47.468499 22406 raft_consensus.cc:3055] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:22:47.472776 22536 raft_consensus.cc:493] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
W20260709 12:22:47.511442 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 268279us
I20260709 12:22:47.406955 22360 consensus_queue.cc:1059] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.097s
I20260709 12:22:47.394711 22436 consensus_queue.cc:1048] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.102s
I20260709 12:22:47.493605 22578 raft_consensus.cc:493] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:47.532847 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:47.536573 17837 raft_consensus.cc:2468] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 14.
I20260709 12:22:47.564044 22468 leader_election.cc:290] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:47.565285 17837 raft_consensus.cc:1275] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20260709 12:22:47.560201 22576 raft_consensus.cc:515] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:47.567009 17837 raft_consensus.cc:1275] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 11 index: 6. Preceding OpId from leader: term: 14 index: 8. (index mismatch)
I20260709 12:22:47.569979 22542 leader_election.cc:290] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 14 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:22:47.570768 22533 consensus_meta.cc:287] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.533s	user 0.008s	sys 0.000s
I20260709 12:22:47.571154 17764 raft_consensus.cc:2393] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 14: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:22:47.514143 22572 raft_consensus.cc:515] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:47.572005 22475 raft_consensus.cc:493] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:47.572503 22475 raft_consensus.cc:515] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:47.573791 22572 leader_election.cc:290] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:47.534148 22501 raft_consensus.cc:3060] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:47.574061 22360 raft_consensus.cc:3055] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 LEADER]: Stepping down as leader of term 8
W20260709 12:22:47.561549 22069 consensus_meta.cc:287] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.790s	user 0.007s	sys 0.003s
I20260709 12:22:47.574720 22360 raft_consensus.cc:740] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:22:47.575326 22311 leader_election.cc:290] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:47.575904 22360 consensus_queue.cc:260] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:47.576820 22311 raft_consensus.cc:493] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:22:47.577364 22311 raft_consensus.cc:515] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:47.577590 22360 raft_consensus.cc:3060] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:47.579425 22311 leader_election.cc:290] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:47.578931 22536 raft_consensus.cc:515] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:47.589825 17755 raft_consensus.cc:2468] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 10.
W20260709 12:22:47.536569 17829 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.564s	user 0.013s	sys 0.034s
I20260709 12:22:47.466212 17761 raft_consensus.cc:1275] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 13 index: 12. (index mismatch)
W20260709 12:22:47.549350 17835 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.555s	user 0.014s	sys 0.015s
I20260709 12:22:47.508600 22582 raft_consensus.cc:493] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:47.496655 22557 consensus_queue.cc:1059] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.037s
I20260709 12:22:47.555869 17832 raft_consensus.cc:2468] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 12.
W20260709 12:22:47.560547 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 404451us
W20260709 12:22:47.599426 18333 leader_election.cc:336] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Timed out: RequestConsensusVote RPC to 127.16.29.66:39651 timed out after 1.528s (SENT)
I20260709 12:22:47.526135 22377 raft_consensus.cc:740] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:22:47.406136 22558 consensus_queue.cc:237] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.9, Last appended by leader: 9, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:47.529304 22477 leader_election.cc:290] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:47.557281 22406 raft_consensus.cc:740] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:22:47.563496 22578 raft_consensus.cc:515] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:47.580371 22485 leader_election.cc:290] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:47.564090 17682 raft_consensus.cc:2468] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 13.
I20260709 12:22:47.557189 22529 consensus_queue.cc:1048] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.026s
I20260709 12:22:47.583685 22437 consensus_queue.cc:1048] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
W20260709 12:22:47.593819 17835 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.640s	user 0.015s	sys 0.015s
I20260709 12:22:47.594724 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:47.596323 17837 raft_consensus.cc:1275] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 11 index: 9. (index mismatch)
I20260709 12:22:47.639575 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:22:47.539615 17729 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:47.600106 17830 raft_consensus.cc:1275] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 12 index: 10. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
I20260709 12:22:47.547474 22489 raft_consensus.cc:3060] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:22:47.600703 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 286083us
I20260709 12:22:47.552348 22523 raft_consensus.cc:740] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
W20260709 12:22:47.604431 18333 connection.cc:256] RPC call timeout handler was delayed by -1.9313677140003165s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1756s for a response
W20260709 12:22:47.592463 22323 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 2.375s	user 0.008s	sys 0.030s
W20260709 12:22:47.672734 22323 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 2.458s	user 0.008s	sys 0.030s
I20260709 12:22:47.673082 22323 raft_consensus.cc:697] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 8 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:22:47.674151 22323 consensus_queue.cc:237] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:47.679272 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:22:47.680213 17571 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet d522f70beb1f417cb98940d90b594823 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica e063541542a94969898868bf6eb5ce1b is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 10 committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:22:47.680933 18371 raft_consensus.cc:1240] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 8. Current term is 9. Ops: []
I20260709 12:22:47.683179 18371 raft_consensus.cc:1275] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
I20260709 12:22:47.571259 22533 raft_consensus.cc:515] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:47.684340 17681 raft_consensus.cc:1240] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 8. Current term is 9. Ops: []
I20260709 12:22:47.685796 17829 raft_consensus.cc:1240] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 13. Current term is 14. Ops: []
I20260709 12:22:47.687071 22533 leader_election.cc:290] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 11 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:47.574442 22475 leader_election.cc:290] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:47.567809 22576 leader_election.cc:290] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:47.688673 17758 raft_consensus.cc:1240] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 10. Current term is 11. Ops: []
W20260709 12:22:47.632210 17727 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:47.689682 17687 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44898 (request call id 945) took 1348 ms. Trace:
I20260709 12:22:47.690076 17687 rpcz_store.cc:276] 0709 12:22:46.315634 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:46.560588 (+244954us) service_pool.cc:224] Handling call
0709 12:22:47.663678 (+1103090us) spinlock_profiling.cc:243] Waited 378 ms on lock 0x7b54000cf154. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:22:47.663830 (+   152us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":868652416}
W20260709 12:22:47.691021 17727 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:47.691447 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:47.692993 18366 raft_consensus.cc:1240] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 7. Current term is 8. Ops: []
I20260709 12:22:47.693547 17687 raft_consensus.cc:2468] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 11.
I20260709 12:22:47.694836 17687 raft_consensus.cc:1275] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 14 index: 12. (index mismatch)
W20260709 12:22:47.641564 17655 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:47.702898 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:47.635993 17652 leader_election.cc:304] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:22:47.640651 22581 raft_consensus.cc:2749] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:22:47.658113 22493 consensus_queue.cc:1059] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 8.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.018s
I20260709 12:22:47.659122 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:47.660939 22377 consensus_queue.cc:260] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:47.663816 22536 leader_election.cc:290] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:47.665520 17834 raft_consensus.cc:1240] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 12. Current term is 13. Ops: []
I20260709 12:22:47.665628 22532 leader_election.cc:290] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:47.666302 22406 consensus_queue.cc:260] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 12.10, Last appended by leader: 10, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:47.604121 22582 raft_consensus.cc:515] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:47.530108 22584 raft_consensus.cc:515] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:47.640445 17837 raft_consensus.cc:2393] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 14: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:22:47.746168 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 11 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:47.747620 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 13 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:22:47.750046 18361 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.509s	user 0.066s	sys 0.071s
W20260709 12:22:47.751097 18361 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.510s	user 0.067s	sys 0.072s
I20260709 12:22:47.750844 17837 raft_consensus.cc:2410] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for term 13 because replica has last-logged OpId of term: 8 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 7.
I20260709 12:22:47.735038 22549 raft_consensus.cc:2804] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:22:47.694898 17727 leader_election.cc:304] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:22:47.696164 18371 raft_consensus.cc:1240] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 13. Current term is 14. Ops: [13.12-13.12]
W20260709 12:22:47.697548 18333 connection.cc:256] RPC call timeout handler was delayed by -1.920567714000299s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1768s for a response
W20260709 12:22:47.754338 18333 connection.cc:256] RPC call timeout handler was delayed by -1.9043677140002728s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1786s for a response
W20260709 12:22:47.754891 18333 leader_election.cc:336] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Timed out: RequestConsensusVote RPC to 127.16.29.66:39651 timed out after 1.606s (SENT)
W20260709 12:22:47.756361 18333 leader_election.cc:336] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 12 election: RPC error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Timed out: RequestConsensusVote RPC to 127.16.29.66:39651 timed out after 1.654s (SENT)
I20260709 12:22:47.756897 18333 leader_election.cc:304] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:47.702987 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:47.757692 17829 raft_consensus.cc:1275] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 8 index: 7. Preceding OpId from leader: term: 11 index: 8. (index mismatch)
I20260709 12:22:47.658066 17830 raft_consensus.cc:1240] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 7. Current term is 8. Ops: []
I20260709 12:22:47.758689 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:47.759337 17687 raft_consensus.cc:3060] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:47.640475 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:47.705214 18366 raft_consensus.cc:1275] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 14 index: 12. (index mismatch)
W20260709 12:22:47.758932 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 214342us
I20260709 12:22:47.638674 22518 raft_consensus.cc:3055] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 12 LEADER]: Stepping down as leader of term 12
W20260709 12:22:47.709936 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:47.709964 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:47.714008 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:47.600812 22468 consensus_queue.cc:1059] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 12.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.102s
I20260709 12:22:47.617964 22435 consensus_queue.cc:1059] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.093s
I20260709 12:22:47.705791 22576 raft_consensus.cc:493] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
W20260709 12:22:47.724656 17728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:47.730284 17729 leader_election.cc:304] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
W20260709 12:22:47.731084 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 95212.8us
I20260709 12:22:47.757722 22406 raft_consensus.cc:3060] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:47.731549 17835 raft_consensus.cc:3060] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:47.780557 17652 leader_election.cc:304] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:22:47.784042 22517 consensus_queue.cc:1048] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:22:47.786229 22517 raft_consensus.cc:2804] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:22:47.667519 17832 raft_consensus.cc:1275] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 8 index: 7. Preceding OpId from leader: term: 10 index: 8. (index mismatch)
I20260709 12:22:47.786937 22517 raft_consensus.cc:697] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 9 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
W20260709 12:22:47.788455 17728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:47.790125 22436 consensus_queue.cc:1059] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:22:47.791683 22436 raft_consensus.cc:3055] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:22:47.792225 22436 raft_consensus.cc:740] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:22:47.740898 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:47.760193 17684 raft_consensus.cc:2468] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 12.
W20260709 12:22:47.796602 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 66500.8us
I20260709 12:22:47.797669 22360 raft_consensus.cc:2749] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20260709 12:22:47.799157 22587 log.cc:927] Time spent T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.067s	user 0.000s	sys 0.000s
I20260709 12:22:47.758217 22578 leader_election.cc:290] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:22:47.767373 17688 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44898 (request call id 942) took 1516 ms (client timeout 1878 ms). Trace:
W20260709 12:22:47.801975 17688 rpcz_store.cc:269] 0709 12:22:46.251128 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:46.329441 (+ 78313us) service_pool.cc:224] Handling call
0709 12:22:47.767105 (+1437664us) spinlock_profiling.cc:243] Waited 271 ms on lock 0x7b5400207454. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:22:47.767289 (+   184us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":622494464}
I20260709 12:22:47.802986 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:47.804045 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 11 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:47.804582 17688 raft_consensus.cc:2468] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 11.
I20260709 12:22:47.805294 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 9 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:47.806170 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 12 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:47.806789 17688 raft_consensus.cc:2393] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 12: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:22:47.754132 22549 raft_consensus.cc:493] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:22:47.772420 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 77631.5us
I20260709 12:22:47.807209 17835 raft_consensus.cc:2468] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 14.
I20260709 12:22:47.761583 17801 leader_election.cc:304] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:22:47.807921 17727 leader_election.cc:304] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:22:47.764117 22518 raft_consensus.cc:740] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 12 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:22:47.770502 22192 raft_consensus.cc:2749] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
W20260709 12:22:47.808895 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 47413.1us
I20260709 12:22:47.797708 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:47.821327 17684 raft_consensus.cc:3060] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:47.719785 22492 raft_consensus.cc:493] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:47.809660 17727 leader_election.cc:304] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
W20260709 12:22:47.810806 18333 connection.cc:256] RPC call timeout handler was delayed by -1.6568677140003274s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2061s for a response
W20260709 12:22:47.841475 18333 connection.cc:256] RPC call timeout handler was delayed by -1.6451677140003085s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2074s for a response
I20260709 12:22:47.812042 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:47.801469 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:47.842839 17688 raft_consensus.cc:2468] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 10.
I20260709 12:22:47.843107 17761 raft_consensus.cc:2410] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for term 14 because replica has last-logged OpId of term: 13 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 10.
I20260709 12:22:47.776564 22542 raft_consensus.cc:2749] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:22:47.819213 17687 raft_consensus.cc:2468] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 14.
I20260709 12:22:47.828035 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:47.850100 18366 raft_consensus.cc:2468] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 13.
I20260709 12:22:47.828935 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 13 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:47.829603 22532 consensus_queue.cc:1048] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:22:47.830334 22425 consensus_queue.cc:1048] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.010s
W20260709 12:22:47.861351 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 51816.3us
I20260709 12:22:47.862414 17727 leader_election.cc:304] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:22:47.831383 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:47.834824 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 11 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:47.834759 22489 raft_consensus.cc:515] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:47.835847 22517 consensus_queue.cc:237] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:47.821324 22529 consensus_queue.cc:1048] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.044s
I20260709 12:22:47.866983 22192 raft_consensus.cc:493] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:47.836387 18368 raft_consensus.cc:1240] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 10. Current term is 11. Ops: [10.10-10.10]
I20260709 12:22:47.867581 22192 raft_consensus.cc:515] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:47.837911 22582 leader_election.cc:290] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:47.844134 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 15 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:47.869953 22192 leader_election.cc:290] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:47.870632 22533 raft_consensus.cc:493] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:22:47.871194 22533 raft_consensus.cc:515] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:47.872685 22311 raft_consensus.cc:2804] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:22:47.812958 22586 raft_consensus.cc:493] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:47.808481 22549 raft_consensus.cc:3060] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:47.860014 17729 leader_election.cc:304] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:47.821591 22501 raft_consensus.cc:2749] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
W20260709 12:22:47.875815 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 221301us
I20260709 12:22:47.820540 22578 raft_consensus.cc:2804] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:22:47.852324 17801 leader_election.cc:304] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:47.798273 22436 consensus_queue.cc:260] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:47.800184 22576 raft_consensus.cc:515] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:47.895116 17801 leader_election.cc:304] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:47.814348 22437 consensus_queue.cc:1048] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.065s
W20260709 12:22:47.894181 17803 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:47.814311 22365 consensus_queue.cc:1048] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.065s
I20260709 12:22:47.896438 17686 raft_consensus.cc:1240] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 12. Current term is 13. Ops: []
I20260709 12:22:47.890884 22069 raft_consensus.cc:2804] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:22:47.896152 22311 raft_consensus.cc:697] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 14 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:22:47.917866 22069 raft_consensus.cc:493] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:47.918278 22069 raft_consensus.cc:3060] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:22:47.918478 22311 consensus_queue.cc:237] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.8, Last appended by leader: 8, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:47.920060 22590 raft_consensus.cc:2749] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:22:47.921449 22436 raft_consensus.cc:3060] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:22:47.923969 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:47.928848 22578 raft_consensus.cc:697] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 11 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:22:47.909575 17727 leader_election.cc:304] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:22:47.910486 22582 raft_consensus.cc:2749] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:22:47.940843 22590 raft_consensus.cc:493] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:47.941468 22590 raft_consensus.cc:515] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:47.942090 22417 raft_consensus.cc:3055] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:22:47.942550 22417 raft_consensus.cc:740] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:22:47.943812 22590 leader_election.cc:290] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:47.943588 22417 consensus_queue.cc:260] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 11.12, Last appended by leader: 12, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:47.849637 22536 consensus_queue.cc:1048] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.048s
W20260709 12:22:47.949036 18333 connection.cc:256] RPC call timeout handler was delayed by -1.474177999999938s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15680000000000002s for a response
W20260709 12:22:47.949736 18333 connection.cc:256] RPC call timeout handler was delayed by -1.2878780000000916s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1775s for a response
W20260709 12:22:47.950290 18333 connection.cc:256] RPC call timeout handler was delayed by -1.2320677140000953s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2533s for a response
W20260709 12:22:47.950697 18333 connection.cc:256] RPC call timeout handler was delayed by -1.2014779999999519s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18710000000000002s for a response
W20260709 12:22:47.951097 18333 connection.cc:256] RPC call timeout handler was delayed by -1.1123779999998078s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.197s for a response
I20260709 12:22:47.956192 17727 leader_election.cc:304] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:22:47.958504 18333 leader_election.cc:304] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:22:47.989035 17728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:47.852231 22492 raft_consensus.cc:515] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:47.944871 22417 raft_consensus.cc:3060] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Advancing to term 12
W20260709 12:22:48.047847 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 91759.8us
I20260709 12:22:47.913823 22149 raft_consensus.cc:2804] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:22:48.049594 17727 leader_election.cc:304] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:48.067770 22376 raft_consensus.cc:2804] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:22:48.059311 22418 raft_consensus.cc:2749] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:22:47.930330 22578 consensus_queue.cc:237] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 10.6, Last appended by leader: 6, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:22:47.945369 22489 leader_election.cc:290] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 13 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:48.043303 22360 raft_consensus.cc:2749] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 12:22:47.934631 22572 consensus_queue.cc:1059] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 9.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.150s
W20260709 12:22:48.063632 22478 consensus_meta.cc:287] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.523s	user 0.007s	sys 0.003s
I20260709 12:22:48.057588 22149 raft_consensus.cc:493] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
W20260709 12:22:48.162619 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 351877us
I20260709 12:22:48.114014 22583 raft_consensus.cc:493] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:48.164501 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:48.131179 22574 raft_consensus.cc:493] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:48.093351 22376 raft_consensus.cc:697] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 14 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:22:48.132416 22192 raft_consensus.cc:2749] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:22:47.896327 22518 consensus_queue.cc:260] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 12.7, Last appended by leader: 7, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:48.136116 22476 raft_consensus.cc:493] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:48.156803 22501 raft_consensus.cc:493] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:48.165098 22584 leader_election.cc:290] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:48.141453 22149 raft_consensus.cc:3060] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:48.165688 22360 raft_consensus.cc:493] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:47.897367 22586 raft_consensus.cc:515] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:22:48.191896 17728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:48.197701 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:48.198448 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:48.199337 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 11 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:48.200315 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:48.201086 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:48.202039 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:48.211846 17837 raft_consensus.cc:2393] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 9: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:22:48.213126 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:48.214507 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 14 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:48.216485 17835 raft_consensus.cc:2468] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 11.
I20260709 12:22:48.215713 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:48.164391 17652 leader_election.cc:304] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da
W20260709 12:22:48.232970 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 69224.1us
W20260709 12:22:48.234870 17728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:47.852854 17684 raft_consensus.cc:2468] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 14.
I20260709 12:22:48.197765 22609 raft_consensus.cc:493] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:48.280588 17833 raft_consensus.cc:2393] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 11: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:22:48.179929 22468 raft_consensus.cc:493] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:48.280844 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 46793.8us
I20260709 12:22:48.285563 17835 raft_consensus.cc:1275] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 8 index: 7. Preceding OpId from leader: term: 12 index: 10. (index mismatch)
I20260709 12:22:48.194800 22418 raft_consensus.cc:493] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:48.294507 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:48.194995 22572 raft_consensus.cc:493] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:48.321688 18363 raft_consensus.cc:2468] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 10.
I20260709 12:22:48.200904 22590 raft_consensus.cc:493] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:48.288668 22517 raft_consensus.cc:2804] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:22:48.302484 17835 raft_consensus.cc:3060] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:48.305603 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 13 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:48.344941 17830 raft_consensus.cc:2410] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for term 13 because replica has last-logged OpId of term: 12 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 9.
I20260709 12:22:48.345537 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 10 candidate_status { last_received { term: 9 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:48.200663 22518 raft_consensus.cc:3060] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:48.195052 22476 raft_consensus.cc:515] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:22:48.364980 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 406544us
I20260709 12:22:48.366844 18333 leader_election.cc:304] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:22:48.369324 17803 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:48.198982 22501 raft_consensus.cc:515] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:22:48.320889 22495 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.825s	user 0.009s	sys 0.021s
W20260709 12:22:48.198832 22557 consensus_queue.cc:1175] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "e063541542a94969898868bf6eb5ce1b" responder_term: 13 status { last_received { term: 10 index: 6 } last_committed_idx: 6 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 12. Current term is 13. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260709 12:22:48.372304 18333 leader_election.cc:304] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:22:48.373306 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:48.278041 22558 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.604s	user 0.035s	sys 0.076s
I20260709 12:22:48.296468 22406 consensus_queue.cc:1059] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.062s
I20260709 12:22:48.283041 22360 raft_consensus.cc:515] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:48.390049 18333 leader_election.cc:304] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:48.394229 17835 raft_consensus.cc:2468] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 8.
I20260709 12:22:48.352713 22533 leader_election.cc:290] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:48.305032 22468 raft_consensus.cc:515] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:48.316798 22418 raft_consensus.cc:515] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:22:48.377907 22495 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.882s	user 0.009s	sys 0.021s
I20260709 12:22:48.324838 22572 raft_consensus.cc:515] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:48.382257 22192 raft_consensus.cc:2749] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260709 12:22:48.400277 22558 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.726s	user 0.035s	sys 0.076s
I20260709 12:22:48.413703 18333 leader_election.cc:304] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:48.343170 22590 raft_consensus.cc:515] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:48.351837 22376 consensus_queue.cc:237] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.10, Last appended by leader: 10, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:48.180338 22574 raft_consensus.cc:515] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:48.310977 22609 raft_consensus.cc:515] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:22:48.432942 22604 log.cc:927] Time spent T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.152s	user 0.001s	sys 0.000s
I20260709 12:22:48.175370 22583 raft_consensus.cc:515] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:48.424590 22492 leader_election.cc:290] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:48.414355 22418 leader_election.cc:290] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:48.420248 22576 leader_election.cc:290] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:48.483522 22616 raft_consensus.cc:2749] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:22:48.484809 22498 leader_election.cc:290] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:48.484905 22535 consensus_queue.cc:260] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.10, Last appended by leader: 10, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:48.486153 22535 raft_consensus.cc:3060] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:22:48.488996 22605 log.cc:927] Time spent T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.080s	user 0.001s	sys 0.000s
W20260709 12:22:48.491180 22549 consensus_meta.cc:287] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.576s	user 0.007s	sys 0.002s
I20260709 12:22:48.483572 22383 raft_consensus.cc:3060] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:48.492377 22572 leader_election.cc:290] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:48.492519 22468 leader_election.cc:290] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:48.476390 22615 raft_consensus.cc:2749] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:22:48.494215 22501 leader_election.cc:290] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:22:48.495183 22436 consensus_meta.cc:287] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.573s	user 0.013s	sys 0.000s
I20260709 12:22:48.495651 22397 leader_election.cc:290] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:48.496217 22574 leader_election.cc:290] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:48.497329 22586 leader_election.cc:290] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:48.491641 22549 raft_consensus.cc:515] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:48.507277 22509 leader_election.cc:290] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:48.498365 22609 leader_election.cc:290] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:48.518178 22548 raft_consensus.cc:3055] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:22:48.518896 22548 raft_consensus.cc:740] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:22:48.518723 22597 raft_consensus.cc:3055] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:22:48.521008 22597 raft_consensus.cc:740] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 12 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:22:48.522182 22597 consensus_queue.cc:260] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.7, Last appended by leader: 7, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:48.524896 22523 consensus_queue.cc:260] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.7, Last appended by leader: 7, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:48.526363 22549 leader_election.cc:290] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 14 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:48.530205 22548 consensus_queue.cc:260] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.7, Last appended by leader: 7, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:48.526424 22523 raft_consensus.cc:3060] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:48.539294 22583 leader_election.cc:290] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:48.542249 22590 leader_election.cc:290] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:47.211283 16501 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
W20260709 12:22:48.580461 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:48.581645 17583 catalog_manager.cc:5697] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 7 to 9. New cstate: current_term: 9 leader_uuid: "334a4e1f59944544b3b3ff36616b57da" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:22:48.596244 17682 raft_consensus.cc:2393] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 14: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:22:48.596961 17683 raft_consensus.cc:1240] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 11 LEADER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 10. Current term is 11. Ops: [10.10-10.10]
I20260709 12:22:48.597820 17682 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37396 (request call id 2332) took 1012 ms. Trace:
W20260709 12:22:48.598338 17728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:48.602669 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
W20260709 12:22:48.604398 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 323367us
I20260709 12:22:48.605909 17686 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37396 (request call id 2314) took 1883 ms. Trace:
I20260709 12:22:48.606918 17764 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38776 (request call id 2241) took 1909 ms. Trace:
I20260709 12:22:48.606374 17686 rpcz_store.cc:276] 0709 12:22:46.721922 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:46.798222 (+ 76300us) service_pool.cc:224] Handling call
0709 12:22:47.767290 (+969068us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:47.917502 (+150212us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:48.604501 (+686999us) spinlock_profiling.cc:243] Waited 129 ms on lock 0x7b5400207454. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:22:48.605454 (+   953us) spinlock_profiling.cc:243] Waited 955 ms on lock 0x7b5400207450. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a61f8ad 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:22:48.605708 (+   254us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2491932800}
I20260709 12:22:48.609376 17759 raft_consensus.cc:2393] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 9: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
W20260709 12:22:48.609206 17756 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38776 (request call id 2245) took 1806 ms (client timeout 1781 ms). Trace:
W20260709 12:22:48.610725 17756 rpcz_store.cc:269] 0709 12:22:46.802952 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:46.803089 (+   137us) service_pool.cc:224] Handling call
0709 12:22:48.608666 (+1805577us) spinlock_profiling.cc:243] Waited 182 ms on lock 0x7b5400415ad4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:22:48.608943 (+   277us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":418526464}
I20260709 12:22:48.613010 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 14 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:48.614429 17756 raft_consensus.cc:2410] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for term 14 because replica has last-logged OpId of term: 13 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 11.
I20260709 12:22:48.617316 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:48.617985 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:48.618453 17833 raft_consensus.cc:2393] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 14: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
W20260709 12:22:48.618371 17831 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35040 (request call id 2480) took 1482 ms (client timeout 1914 ms). Trace:
W20260709 12:22:48.595397 17680 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44898 (request call id 943) took 2327 ms (client timeout 1726 ms). Trace:
I20260709 12:22:48.609978 17683 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37396 (request call id 2326) took 1222 ms. Trace:
I20260709 12:22:48.607396 17802 leader_election.cc:304] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:22:48.598366 17682 rpcz_store.cc:276] 0709 12:22:47.585153 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:47.658862 (+ 73709us) service_pool.cc:224] Handling call
0709 12:22:48.597459 (+938597us) spinlock_profiling.cc:243] Waited 860 ms on lock 0x7b54000c6054. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:22:48.597710 (+   251us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1978396928}
I20260709 12:22:48.629338 18365 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59886 (request call id 1367) took 1864 ms. Trace:
I20260709 12:22:48.630375 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 14 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:22:48.652637 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:48.631785 18368 raft_consensus.cc:1240] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 10. Current term is 11. Ops: []
I20260709 12:22:48.627265 17837 raft_consensus.cc:3060] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:48.654891 18365 rpcz_store.cc:276] 0709 12:22:46.764896 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:46.766961 (+  2065us) service_pool.cc:224] Handling call
0709 12:22:46.777728 (+ 10767us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:46.782581 (+  4853us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:48.628634 (+1846053us) spinlock_profiling.cc:243] Waited 4.09 ms on lock 0x7b54005730d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:22:48.629082 (+   448us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":9416448}
I20260709 12:22:48.626507 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:48.664247 18366 raft_consensus.cc:2468] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 13.
I20260709 12:22:48.607556 17764 rpcz_store.cc:276] 0709 12:22:46.697568 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:46.697936 (+   368us) service_pool.cc:224] Handling call
0709 12:22:48.606372 (+1908436us) spinlock_profiling.cc:243] Waited 849 ms on lock 0x7b54003e62d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:22:48.606671 (+   299us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1952427392}
I20260709 12:22:48.639371 22114 raft_consensus.cc:493] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:48.666939 22114 raft_consensus.cc:515] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:48.670550 22114 leader_election.cc:290] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:48.623509 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:48.672206 17834 raft_consensus.cc:2468] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 8.
I20260709 12:22:48.676474 22517 raft_consensus.cc:697] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 12 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:22:48.678267 22517 consensus_queue.cc:237] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 10.11, Last appended by leader: 11, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:48.625583 18367 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 1517) took 1800 ms. Trace:
I20260709 12:22:48.630465 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:22:48.649487 17680 rpcz_store.cc:269] 0709 12:22:46.267641 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:46.325618 (+ 57977us) service_pool.cc:224] Handling call
0709 12:22:48.594721 (+2269103us) spinlock_profiling.cc:243] Waited 2.27 s on lock 0x7b54003462d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:22:48.595059 (+   338us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5209520000}
I20260709 12:22:48.649685 17683 rpcz_store.cc:276] 0709 12:22:47.387980 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:47.388123 (+   143us) service_pool.cc:224] Handling call
0709 12:22:48.596202 (+1208079us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:48.597596 (+  1394us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:48.599726 (+  2130us) spinlock_profiling.cc:243] Waited 1.21 s on lock 0x7b54003462d0. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a61f8ad 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:22:48.609895 (+ 10169us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2777330432}
W20260709 12:22:48.619344 17831 rpcz_store.cc:269] 0709 12:22:47.136946 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:47.154225 (+ 17279us) service_pool.cc:224] Handling call
0709 12:22:48.617839 (+1463614us) spinlock_profiling.cc:243] Waited 143 ms on lock 0x7b54000aa5d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:22:48.618132 (+   293us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":328984448}
I20260709 12:22:48.631624 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:48.620782 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:48.632691 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:48.455103 22192 raft_consensus.cc:2749] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 12:22:48.639371 21994 raft_consensus.cc:493] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:48.702668 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 95672.9us
I20260709 12:22:48.425028 22307 consensus_queue.cc:1048] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.059s
I20260709 12:22:48.399827 17832 raft_consensus.cc:2468] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 9.
I20260709 12:22:48.641069 22468 raft_consensus.cc:493] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:48.710079 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:48.710552 21994 raft_consensus.cc:515] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:48.711714 17838 raft_consensus.cc:2468] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 11.
I20260709 12:22:48.713908 21994 leader_election.cc:290] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:48.717360 17728 leader_election.cc:304] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:22:48.717885 18361 raft_consensus.cc:1275] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 15 index: 14. Preceding OpId from leader: term: 17 index: 15. (index mismatch)
I20260709 12:22:48.718508 22476 leader_election.cc:290] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:22:48.719949 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 409988us
I20260709 12:22:48.720249 22588 raft_consensus.cc:2804] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:22:48.720991 22588 raft_consensus.cc:493] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:48.721499 22588 raft_consensus.cc:3060] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:48.721565 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:48.724367 22557 raft_consensus.cc:493] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:48.725538 17802 leader_election.cc:304] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:22:48.725317 22557 raft_consensus.cc:515] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:48.727049 17802 leader_election.cc:304] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:48.728907 22114 raft_consensus.cc:2804] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:22:48.729492 22114 raft_consensus.cc:493] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:48.730010 22114 raft_consensus.cc:3060] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:48.733875 22495 raft_consensus.cc:3055] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:22:48.734403 22495 raft_consensus.cc:740] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:22:48.735980 17829 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35040 (request call id 2492) took 1266 ms. Trace:
I20260709 12:22:48.736394 17829 rpcz_store.cc:276] 0709 12:22:47.469737 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:47.687599 (+217862us) service_pool.cc:224] Handling call
0709 12:22:47.687847 (+   248us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:47.758166 (+ 70319us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:48.735468 (+977302us) spinlock_profiling.cc:243] Waited 68 ms on lock 0x7b540008cfd4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:22:48.735759 (+   291us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":156374272}
I20260709 12:22:48.742054 22476 raft_consensus.cc:493] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:48.742591 18334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:48.745046 18374 raft_consensus.cc:2468] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 10.
I20260709 12:22:48.737571 17728 leader_election.cc:304] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:48.750043 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:48.754886 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:48.766963 22184 consensus_queue.cc:1048] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:22:48.771832 22476 raft_consensus.cc:515] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:48.774622 22583 raft_consensus.cc:2804] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:22:48.775115 22583 raft_consensus.cc:493] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:48.775462 22583 raft_consensus.cc:3060] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Advancing to term 14
W20260709 12:22:48.776885 17762 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38776 (request call id 2246) took 1950 ms (client timeout 1776 ms). Trace:
W20260709 12:22:48.777310 17762 rpcz_store.cc:269] 0709 12:22:46.826232 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:46.826443 (+   211us) service_pool.cc:224] Handling call
0709 12:22:48.776533 (+1950090us) spinlock_profiling.cc:243] Waited 128 ms on lock 0x7b54000fbed4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:22:48.776688 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":293268352}
I20260709 12:22:48.778649 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
W20260709 12:22:48.780246 17760 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38786 (request call id 2929) took 3486 ms (client timeout 2909 ms). Trace:
W20260709 12:22:48.780807 17760 rpcz_store.cc:269] 0709 12:22:45.300685 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:45.302135 (+  1450us) service_pool.cc:224] Handling call
0709 12:22:48.779701 (+3477566us) spinlock_profiling.cc:243] Waited 1.81 s on lock 0x7b54003e3fd4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:22:48.779978 (+   277us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4158202624}
I20260709 12:22:48.782550 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:48.763340 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:48.785709 17837 raft_consensus.cc:2468] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 12.
W20260709 12:22:48.783808 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 56990.1us
W20260709 12:22:48.789139 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 69409.3us
I20260709 12:22:48.681259 18367 rpcz_store.cc:276] 0709 12:22:46.824638 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:46.831349 (+  6711us) service_pool.cc:224] Handling call
0709 12:22:46.831698 (+   349us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:46.832076 (+   378us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:46.832110 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:46.832563 (+   453us) log.cc:844] Serialized 26 byte log entry
0709 12:22:47.662281 (+829718us) spinlock_profiling.cc:243] Waited 35.7 ms on lock 0x7b180040a3e0. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b47515e07 00007f7b503709a9 00007f7b4753269b 00007f7b4753093f 00007f7b47532c02 00007f7b4a1ba20b 00007f7b4a1bfd23 00007f7b4a1c00d1 00007f7b4a5dba0f
0709 12:22:47.775622 (+113341us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:22:47.775683 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:47.775787 (+   104us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:48.624752 (+848965us) raft_consensus.cc:1685] finished
0709 12:22:48.624855 (+   103us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:48.625290 (+   435us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":90,"raft.run_cpu_time_us":91,"raft.run_wall_time_us":90,"spinlock_wait_cycles":82008448,"thread_start_us":113123,"threads_started":1,"wal-append.queue_time_us":1344581}
I20260709 12:22:48.797237 22509 raft_consensus.cc:493] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:48.797590 22578 raft_consensus.cc:2804] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:22:48.798364 22578 raft_consensus.cc:697] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 14 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:22:48.798146 22509 raft_consensus.cc:515] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:48.800215 22521 raft_consensus.cc:2804] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:22:48.800653 22521 raft_consensus.cc:493] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:48.801010 22521 raft_consensus.cc:3060] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:48.790532 22468 raft_consensus.cc:515] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:22:48.788401 22305 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 13 current term: 13 is higher.
I20260709 12:22:48.807130 22468 leader_election.cc:290] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:48.803565 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:48.808604 22305 consensus_queue.cc:1048] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:22:48.809053 18374 raft_consensus.cc:2468] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 12.
I20260709 12:22:48.808832 17728 leader_election.cc:304] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:22:48.784217 22476 leader_election.cc:290] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:48.790518 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:48.833876 17758 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38776 (request call id 2247) took 1884 ms. Trace:
I20260709 12:22:48.835851 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:48.837109 17760 raft_consensus.cc:2393] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 11: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:22:48.838269 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:48.844432 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:48.845291 22416 raft_consensus.cc:2804] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:22:48.845489 18367 raft_consensus.cc:2468] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 8.
I20260709 12:22:48.845821 22416 raft_consensus.cc:493] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:22:48.846247 22416 raft_consensus.cc:3060] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:22:48.847491 17728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:48.834522 17758 rpcz_store.cc:276] 0709 12:22:46.949756 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:46.951004 (+  1248us) service_pool.cc:224] Handling call
0709 12:22:46.951321 (+   317us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:47.689186 (+737865us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:48.833259 (+1144073us) spinlock_profiling.cc:243] Waited 737 ms on lock 0x7b54003bced4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:22:48.833666 (+   407us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1695323904}
W20260709 12:22:48.867682 17728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:48.868759 22376 raft_consensus.cc:493] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:48.870822 22376 raft_consensus.cc:515] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:22:48.871245 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 108966us
I20260709 12:22:48.873430 22376 leader_election.cc:290] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:48.874807 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:48.875511 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 13 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:48.876392 17758 raft_consensus.cc:2410] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 13 because replica has last-logged OpId of term: 11 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 7.
I20260709 12:22:48.877485 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:48.878096 17758 raft_consensus.cc:2468] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 14.
I20260709 12:22:48.879040 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:22:48.880602 22417 consensus_meta.cc:287] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.822s	user 0.011s	sys 0.007s
I20260709 12:22:48.799633 22578 consensus_queue.cc:237] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.8, Last appended by leader: 8, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:48.882356 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:48.883124 17761 raft_consensus.cc:2468] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 9.
I20260709 12:22:48.885255 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:48.886454 17761 raft_consensus.cc:2468] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 8.
I20260709 12:22:48.887990 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:22:48.888736 17654 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:48.889221 17761 raft_consensus.cc:2468] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 13.
I20260709 12:22:48.891387 17759 raft_consensus.cc:2468] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 13.
I20260709 12:22:48.891752 17683 raft_consensus.cc:1240] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 12. Current term is 13. Ops: [12.7-12.7]
I20260709 12:22:48.892042 17759 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36572 (request call id 840) took 1179 ms. Trace:
I20260709 12:22:48.892407 17759 rpcz_store.cc:276] 0709 12:22:47.712577 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:48.692014 (+979437us) service_pool.cc:224] Handling call
0709 12:22:48.891956 (+199942us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:48.895113 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 14 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:48.895222 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:48.896965 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:48.899819 17688 raft_consensus.cc:1275] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 12 index: 10. Preceding OpId from leader: term: 13 index: 10. (term mismatch)
I20260709 12:22:48.900452 17688 pending_rounds.cc:85] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3: Aborting all ops after (but not including) 9
I20260709 12:22:48.900828 22509 leader_election.cc:290] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:48.900954 17688 pending_rounds.cc:107] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3: Aborting uncommitted NO_OP operation due to leader change: 12.10
I20260709 12:22:48.910805 17688 raft_consensus.cc:2889] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:48.911448 17727 leader_election.cc:304] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:22:48.893720 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:48.912210 22540 consensus_queue.cc:1059] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 10.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:22:48.913228 17759 raft_consensus.cc:2468] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 8.
I20260709 12:22:48.913420 22540 raft_consensus.cc:3055] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:22:48.913987 22540 raft_consensus.cc:740] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:22:48.914377 22148 raft_consensus.cc:2749] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:22:48.915012 22360 leader_election.cc:290] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:48.915460 22540 consensus_queue.cc:260] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.7, Last appended by leader: 7, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:48.916150 17727 leader_election.cc:304] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:48.917079 22540 raft_consensus.cc:3060] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:48.903146 22377 raft_consensus.cc:3060] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:48.917526 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:48.918313 18367 raft_consensus.cc:2468] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 9.
W20260709 12:22:48.920563 17728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:48.893702 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 11 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:48.923136 17763 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38786 (request call id 2944) took 1006 ms. Trace:
I20260709 12:22:48.923936 17801 leader_election.cc:304] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:48.924523 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:48.924296 22475 raft_consensus.cc:493] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:48.925290 17832 raft_consensus.cc:2468] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 8.
I20260709 12:22:48.925740 22476 raft_consensus.cc:493] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:48.926195 22475 raft_consensus.cc:515] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:48.926558 22476 raft_consensus.cc:515] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:48.927134 22619 consensus_queue.cc:1048] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:22:48.929705 22590 raft_consensus.cc:2749] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
W20260709 12:22:48.909034 22069 consensus_meta.cc:287] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.934s	user 0.015s	sys 0.011s
I20260709 12:22:48.930799 22069 raft_consensus.cc:515] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:48.932286 17757 raft_consensus.cc:2410] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for term 11 because replica has last-logged OpId of term: 10 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 9.
I20260709 12:22:48.932866 22551 raft_consensus.cc:493] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:48.933307 22551 raft_consensus.cc:515] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:22:48.935173 22149 consensus_meta.cc:287] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.655s	user 0.006s	sys 0.011s
I20260709 12:22:48.934937 22495 consensus_queue.cc:260] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 13.9, Last appended by leader: 9, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:48.933301 22069 leader_election.cc:290] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 15 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:22:48.909937 17655 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:48.937160 17655 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:48.937461 22495 raft_consensus.cc:3060] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:48.938879 22607 raft_consensus.cc:493] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:22:48.937177 17652 leader_election.cc:304] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:48.910228 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:48.902252 17686 raft_consensus.cc:2393] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 14: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
W20260709 12:22:48.917374 17803 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:48.864180 22557 leader_election.cc:290] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:48.941987 17801 leader_election.cc:304] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:48.948393 22590 raft_consensus.cc:2804] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:22:48.948915 22590 raft_consensus.cc:493] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:48.949410 22590 raft_consensus.cc:3060] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:48.949810 22548 raft_consensus.cc:3060] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:48.952354 17801 leader_election.cc:304] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:22:48.947085 22518 consensus_meta.cc:287] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.581s	user 0.006s	sys 0.014s
I20260709 12:22:48.953619 22408 consensus_queue.cc:1048] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:22:48.943136 22476 leader_election.cc:290] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:48.955083 17764 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36572 (request call id 843) took 1241 ms. Trace:
I20260709 12:22:48.951916 17583 catalog_manager.cc:5697] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 9 to 11, leader changed from 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) to e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66). New cstate: current_term: 11 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:22:48.953136 22218 consensus_queue.cc:1048] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:22:48.953420 22468 raft_consensus.cc:2804] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:22:48.956770 17681 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37396 (request call id 2317) took 2234 ms. Trace:
I20260709 12:22:48.956985 22468 raft_consensus.cc:493] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:48.957417 22468 raft_consensus.cc:3060] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:48.957214 17681 rpcz_store.cc:276] 0709 12:22:46.722680 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:47.065512 (+342832us) service_pool.cc:224] Handling call
0709 12:22:47.684154 (+618642us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:47.684756 (+   602us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:48.956339 (+1271583us) spinlock_profiling.cc:243] Waited 618 ms on lock 0x7b54000cf150. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a61f8ad 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:22:48.956571 (+   232us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1421886208}
I20260709 12:22:48.955549 17764 rpcz_store.cc:276] 0709 12:22:47.713423 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:48.909804 (+1196381us) service_pool.cc:224] Handling call
0709 12:22:48.955015 (+ 45211us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:48.960685 17764 raft_consensus.cc:1275] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260709 12:22:48.961421 17764 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36572 (request call id 844) took 1241 ms. Trace:
I20260709 12:22:48.961822 17764 rpcz_store.cc:276] 0709 12:22:47.720210 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:48.959562 (+1239352us) service_pool.cc:224] Handling call
0709 12:22:48.960163 (+   601us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:48.961135 (+   972us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:48.961334 (+   199us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:48.940876 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:48.966815 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 13 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:22:48.968216 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:48.964584 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 12 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:22:48.969581 17755 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38776 (request call id 2252) took 1935 ms (client timeout 1914 ms). Trace:
I20260709 12:22:48.970019 22408 consensus_queue.cc:1048] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:22:48.971954 22515 raft_consensus.cc:2749] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:22:48.958511 17652 leader_election.cc:304] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:22:48.974610 22633 log.cc:927] Time spent T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.100s	user 0.002s	sys 0.002s
I20260709 12:22:48.890457 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 15 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:22:48.977228 17729 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:48.978431 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 62462.5us
I20260709 12:22:48.975558 22425 raft_consensus.cc:2749] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:22:48.942726 17757 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36572 (request call id 835) took 1233 ms. Trace:
I20260709 12:22:48.914428 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 13 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:48.981887 17761 raft_consensus.cc:2393] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 15: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:22:48.923681 17763 rpcz_store.cc:276] 0709 12:22:47.916234 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:48.874231 (+957997us) service_pool.cc:224] Handling call
0709 12:22:48.923048 (+ 48817us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:48.980568 17757 rpcz_store.cc:276] 0709 12:22:47.709385 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:47.709647 (+   262us) service_pool.cc:224] Handling call
0709 12:22:48.942488 (+1232841us) spinlock_profiling.cc:243] Waited 1.17 s on lock 0x7b5400194e54. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:22:48.942664 (+   176us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2682043136}
W20260709 12:22:48.970077 17755 rpcz_store.cc:269] 0709 12:22:47.033894 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:47.050785 (+ 16891us) service_pool.cc:224] Handling call
0709 12:22:48.969137 (+1918352us) spinlock_profiling.cc:243] Waited 448 ms on lock 0x7b54003eb054. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:22:48.969295 (+   158us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1030162816}
I20260709 12:22:48.990394 17761 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38786 (request call id 2945) took 1073 ms. Trace:
I20260709 12:22:48.996445 17761 rpcz_store.cc:276] 0709 12:22:47.916541 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:48.890049 (+973508us) service_pool.cc:224] Handling call
0709 12:22:48.990305 (+100256us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:49.002357 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:49.003361 17837 raft_consensus.cc:2468] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 12.
I20260709 12:22:49.008951 17759 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36572 (request call id 838) took 1297 ms. Trace:
I20260709 12:22:49.009642 17759 rpcz_store.cc:276] 0709 12:22:47.711976 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:48.914004 (+1202028us) service_pool.cc:224] Handling call
0709 12:22:49.008863 (+ 94859us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:49.027595 17801 leader_election.cc:304] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:49.034209 22491 raft_consensus.cc:2804] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:22:49.034761 22491 raft_consensus.cc:493] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:49.035115 22491 raft_consensus.cc:3060] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:49.035780 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:49.035913 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:49.036577 18368 raft_consensus.cc:2468] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 13.
W20260709 12:22:49.036835 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 623075us
I20260709 12:22:49.036959 18361 raft_consensus.cc:2468] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 12.
I20260709 12:22:49.038065 18333 leader_election.cc:400] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 13 election: Vote denied by peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651) with higher term. Message: Invalid argument: T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 13. Current term is 14.
I20260709 12:22:49.038414 18333 leader_election.cc:403] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 13 election: Cancelling election due to peer responding with higher term
I20260709 12:22:49.035152 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:49.039565 22417 raft_consensus.cc:2749] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Leader election lost for term 13. Reason: Vote denied by peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651) with higher term. Message: Invalid argument: T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 13. Current term is 14.
I20260709 12:22:49.040572 22192 raft_consensus.cc:493] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:49.041013 22535 raft_consensus.cc:493] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:49.041639 22535 raft_consensus.cc:515] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:49.042426 17727 leader_election.cc:304] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:49.041661 22192 raft_consensus.cc:515] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:49.044799 18364 raft_consensus.cc:1240] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 8 LEADER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 7. Current term is 8. Ops: []
I20260709 12:22:49.047555 22192 leader_election.cc:290] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:49.049768 17729 leader_election.cc:304] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:48.935734 22149 raft_consensus.cc:515] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:49.055231 22475 leader_election.cc:290] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:49.054234 17758 raft_consensus.cc:1240] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 12. Current term is 13. Ops: []
I20260709 12:22:48.939787 22607 raft_consensus.cc:515] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:49.043596 17727 leader_election.cc:304] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:22:49.067829 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 57784.1us
I20260709 12:22:49.065822 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:49.056903 22360 raft_consensus.cc:2804] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:22:49.076365 17837 raft_consensus.cc:1275] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 9 index: 7. Preceding OpId from leader: term: 14 index: 9. (index mismatch)
I20260709 12:22:49.091627 17727 leader_election.cc:304] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:49.117141 17760 raft_consensus.cc:2468] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 8.
W20260709 12:22:49.119959 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 161571us
W20260709 12:22:49.121999 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 45741.1us
I20260709 12:22:49.077229 22360 raft_consensus.cc:493] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:49.130138 17764 raft_consensus.cc:1240] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 13. Current term is 14. Ops: []
I20260709 12:22:49.107261 22509 raft_consensus.cc:2804] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:22:49.066630 22114 raft_consensus.cc:515] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:49.139053 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:49.130354 22360 raft_consensus.cc:3060] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:49.135448 22509 raft_consensus.cc:493] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:49.163916 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:49.143118 22518 raft_consensus.cc:2804] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Leader pre-election won for term 13
W20260709 12:22:49.169821 22383 consensus_meta.cc:287] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.647s	user 0.012s	sys 0.006s
I20260709 12:22:49.170040 17583 catalog_manager.cc:5697] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 8 to 10. New cstate: current_term: 10 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:22:49.172710 22588 raft_consensus.cc:515] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:49.175318 22588 leader_election.cc:290] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 11 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:49.162547 22069 raft_consensus.cc:493] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:49.171651 17583 catalog_manager.cc:5697] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 7 to 9. New cstate: current_term: 9 leader_uuid: "334a4e1f59944544b3b3ff36616b57da" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:22:49.157406 22509 raft_consensus.cc:3060] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:49.185262 17729 leader_election.cc:304] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:22:49.182441 17756 raft_consensus.cc:2468] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 12.
I20260709 12:22:49.168908 17686 raft_consensus.cc:2468] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 11.
I20260709 12:22:49.137496 22377 consensus_queue.cc:1059] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.028s
I20260709 12:22:49.184151 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:49.167500 22518 raft_consensus.cc:493] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:49.188235 22069 raft_consensus.cc:515] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:49.196355 18361 raft_consensus.cc:1275] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 8 index: 7. Preceding OpId from leader: term: 11 index: 8. (index mismatch)
I20260709 12:22:49.196527 22542 raft_consensus.cc:493] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:49.188089 22536 raft_consensus.cc:2804] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:22:49.197057 22542 raft_consensus.cc:515] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:49.197522 22536 raft_consensus.cc:493] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:49.197928 22536 raft_consensus.cc:3060] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:49.195227 22607 leader_election.cc:290] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:49.158838 22474 raft_consensus.cc:493] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:49.200647 22474 raft_consensus.cc:515] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:49.201812 22551 leader_election.cc:290] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:49.203207 22474 leader_election.cc:290] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:49.204574 18371 raft_consensus.cc:2468] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 8.
I20260709 12:22:49.205317 22518 raft_consensus.cc:3060] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:22:49.192533 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 121591us
W20260709 12:22:49.213480 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 46778.3us
I20260709 12:22:49.226948 17801 leader_election.cc:304] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:49.232129 17688 raft_consensus.cc:1275] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 10 index: 7. Preceding OpId from leader: term: 11 index: 7. (term mismatch)
I20260709 12:22:49.203775 22590 raft_consensus.cc:515] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:49.196379 22069 leader_election.cc:290] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:22:49.239128 17804 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:49.196408 22579 raft_consensus.cc:2804] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:22:49.232662 17688 pending_rounds.cc:85] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3: Aborting all ops after (but not including) 6
W20260709 12:22:49.248317 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 94613.1us
I20260709 12:22:49.246264 22590 leader_election.cc:290] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 14 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:49.206080 22468 raft_consensus.cc:515] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:22:49.253113 22634 log.cc:927] Time spent T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.160s	user 0.000s	sys 0.002s
I20260709 12:22:49.239307 22474 raft_consensus.cc:2749] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20260709 12:22:49.258540 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 92709.3us
W20260709 12:22:49.261336 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 100363us
I20260709 12:22:49.247131 22579 raft_consensus.cc:493] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:49.247646 17688 pending_rounds.cc:107] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3: Aborting uncommitted CHANGE_CONFIG_OP operation due to leader change: 10.7
I20260709 12:22:49.233040 22416 raft_consensus.cc:515] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:49.268435 17729 leader_election.cc:304] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:22:49.253366 22475 raft_consensus.cc:2804] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:22:49.279783 22377 consensus_queue.cc:1048] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:22:49.280062 22521 raft_consensus.cc:515] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:49.280791 22474 consensus_queue.cc:1059] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:22:49.281908 22557 raft_consensus.cc:3055] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:22:49.282294 22557 raft_consensus.cc:740] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:22:49.282673 22521 leader_election.cc:290] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 12 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:49.280211 22114 leader_election.cc:290] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 11 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:49.286187 17652 leader_election.cc:304] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
W20260709 12:22:49.286948 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 55766us
I20260709 12:22:49.285117 17688 raft_consensus.cc:2917] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Aborting config change with OpId 10.7: Aborted: Op aborted by new leader
I20260709 12:22:49.250397 22583 raft_consensus.cc:515] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:49.293948 22583 leader_election.cc:290] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 14 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:22:49.294792 18334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:49.268580 22408 raft_consensus.cc:493] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:49.295097 17762 raft_consensus.cc:2393] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 14: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:22:49.323208 17688 raft_consensus.cc:2889] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:49.297961 17729 leader_election.cc:304] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:22:49.284147 22475 raft_consensus.cc:493] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:49.323334 17757 raft_consensus.cc:1240] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 12. Current term is 13. Ops: []
I20260709 12:22:49.272915 22579 raft_consensus.cc:3060] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:49.329757 22468 leader_election.cc:290] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:22:49.332293 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 59017.5us
I20260709 12:22:49.347270 17652 leader_election.cc:304] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:49.352327 22408 raft_consensus.cc:515] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:49.353937 18333 leader_election.cc:304] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:22:49.355510 22408 leader_election.cc:290] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:49.355966 22192 raft_consensus.cc:2804] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:22:49.356168 22615 consensus_queue.cc:1048] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:22:49.356426 22192 raft_consensus.cc:493] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:49.356894 22192 raft_consensus.cc:3060] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:49.347311 22625 consensus_queue.cc:1059] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 9.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.010s
I20260709 12:22:49.357856 22625 raft_consensus.cc:3055] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:22:49.358198 22625 raft_consensus.cc:740] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:22:49.359151 22625 consensus_queue.cc:260] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.7, Last appended by leader: 7, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:49.367444 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:49.360896 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:49.283407 22557 consensus_queue.cc:260] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:49.368466 22557 raft_consensus.cc:3060] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:49.368597 18367 raft_consensus.cc:2410] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 12 because replica has last-logged OpId of term: 11 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 8.
I20260709 12:22:49.369608 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:49.362113 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:49.370748 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:49.370823 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 13 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:49.371186 22518 raft_consensus.cc:515] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:22:49.339962 22636 log.cc:927] Time spent T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.068s	user 0.001s	sys 0.000s
W20260709 12:22:49.364905 22638 log.cc:927] Time spent T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.080s	user 0.000s	sys 0.000s
I20260709 12:22:49.365835 22515 raft_consensus.cc:2749] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:22:49.367268 17652 leader_election.cc:304] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:49.371794 18368 raft_consensus.cc:2410] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 13 because replica has last-logged OpId of term: 12 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 8.
I20260709 12:22:49.336263 22305 raft_consensus.cc:2804] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:22:49.372759 22625 raft_consensus.cc:3060] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:22:49.374159 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 83202.5us
I20260709 12:22:49.374588 22607 raft_consensus.cc:2804] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Leader pre-election won for term 11
W20260709 12:22:49.376467 22635 log.cc:927] Time spent T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.089s	user 0.001s	sys 0.000s
I20260709 12:22:49.336345 22475 raft_consensus.cc:3060] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:22:49.406643 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 108771us
I20260709 12:22:49.412649 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:49.404052 22376 raft_consensus.cc:2804] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:22:49.424125 22365 raft_consensus.cc:493] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:22:49.424329 22376 raft_consensus.cc:493] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:49.421780 17728 leader_election.cc:304] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:22:49.427584 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 13 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:49.410153 22607 raft_consensus.cc:493] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:49.404486 22305 raft_consensus.cc:493] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:22:49.425010 22376 raft_consensus.cc:3060] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:49.418143 22468 raft_consensus.cc:493] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:49.444026 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:49.451843 17729 leader_election.cc:304] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:22:49.452770 18368 raft_consensus.cc:2410] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 13 because replica has last-logged OpId of term: 12 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 8.
W20260709 12:22:49.458954 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 91809.4us
W20260709 12:22:49.459406 17756 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36572 (request call id 839) took 1747 ms (client timeout 2027 ms). Trace:
W20260709 12:22:49.459795 17756 rpcz_store.cc:269] 0709 12:22:47.712291 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:48.692014 (+979723us) service_pool.cc:224] Handling call
0709 12:22:49.459228 (+767214us) spinlock_profiling.cc:243] Waited 407 ms on lock 0x7b54002b5a54. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:22:49.459321 (+    93us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":935526272}
I20260709 12:22:49.460268 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:49.461088 18366 raft_consensus.cc:2393] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 12: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:22:49.462515 18366 raft_consensus.cc:1240] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 8. Current term is 9. Ops: []
I20260709 12:22:49.466272 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:49.468220 17760 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38786 (request call id 2952) took 1120 ms. Trace:
I20260709 12:22:49.424842 22365 raft_consensus.cc:515] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:49.468591 17760 rpcz_store.cc:276] 0709 12:22:48.347922 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:48.837841 (+489919us) service_pool.cc:224] Handling call
0709 12:22:49.468080 (+630239us) spinlock_profiling.cc:243] Waited 271 ms on lock 0x7b54000fe6d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:22:49.468168 (+    88us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":622593280}
W20260709 12:22:49.469269 17762 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38786 (request call id 2943) took 1553 ms (client timeout 1943 ms). Trace:
W20260709 12:22:49.469651 17762 rpcz_store.cc:269] 0709 12:22:47.915927 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:48.778214 (+862287us) service_pool.cc:224] Handling call
0709 12:22:49.469099 (+690885us) spinlock_profiling.cc:243] Waited 516 ms on lock 0x7b540019fad4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:22:49.469222 (+   123us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1185735424}
I20260709 12:22:49.471053 22365 leader_election.cc:290] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:49.460884 18364 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59886 (request call id 1353) took 3717 ms. Trace:
I20260709 12:22:49.473428 22383 raft_consensus.cc:3055] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:22:49.473946 22383 raft_consensus.cc:740] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 12 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:22:49.473681 18364 rpcz_store.cc:276] 0709 12:22:45.730243 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:45.771273 (+ 41030us) service_pool.cc:224] Handling call
0709 12:22:45.771542 (+   269us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:49.045341 (+3273799us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:49.429406 (+384065us) spinlock_profiling.cc:243] Waited 3.27 s on lock 0x7b540051aed4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:22:49.447371 (+ 17965us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":7526316416}
I20260709 12:22:49.475647 22069 consensus_queue.cc:1048] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:22:49.476020 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:49.478567 17680 raft_consensus.cc:3060] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:49.480270 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 11 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:49.475081 22383 consensus_queue.cc:260] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:49.467343 17802 leader_election.cc:304] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:22:49.434890 22607 raft_consensus.cc:3060] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:49.438620 22305 raft_consensus.cc:3060] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:22:49.484514 17803 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:49.494107 22488 raft_consensus.cc:493] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:49.494976 22488 raft_consensus.cc:515] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:49.497742 22488 leader_election.cc:290] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:49.498828 22406 consensus_queue.cc:1048] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:22:49.505707 17764 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38786 (request call id 2947) took 1588 ms. Trace:
I20260709 12:22:49.506250 17764 rpcz_store.cc:276] 0709 12:22:47.917144 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:48.962706 (+1045562us) service_pool.cc:224] Handling call
0709 12:22:48.963285 (+   579us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:49.130691 (+167406us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:49.505372 (+374681us) spinlock_profiling.cc:243] Waited 162 ms on lock 0x7b54001b8354. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:22:49.505596 (+   224us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":373592832}
I20260709 12:22:49.434698 22495 raft_consensus.cc:2749] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:22:49.455468 22583 raft_consensus.cc:3060] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Advancing to term 14
W20260709 12:22:49.513684 17655 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:49.511687 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 95431us
W20260709 12:22:49.514297 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 62584.8us
I20260709 12:22:49.521142 17686 raft_consensus.cc:2468] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 11.
W20260709 12:22:49.522848 17728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:49.524078 17655 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:49.525435 17655 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:49.512858 22597 raft_consensus.cc:3060] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:49.443027 22536 raft_consensus.cc:515] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:49.515964 22069 raft_consensus.cc:2804] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:22:49.524698 17729 leader_election.cc:304] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:49.530834 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:49.537828 18362 raft_consensus.cc:2410] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 12 because replica has last-logged OpId of term: 11 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 9.
I20260709 12:22:49.523907 22521 raft_consensus.cc:493] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:49.541965 22551 raft_consensus.cc:2804] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:22:49.523847 22383 raft_consensus.cc:3060] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:49.520244 22542 leader_election.cc:290] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:49.533723 22069 raft_consensus.cc:493] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:49.529004 22114 raft_consensus.cc:2804] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:22:49.510835 22468 raft_consensus.cc:515] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:49.556664 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:49.559077 18361 raft_consensus.cc:2393] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 12: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:22:49.564641 22114 raft_consensus.cc:697] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 14 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:22:49.565572 22536 leader_election.cc:290] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 14 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:49.547319 22551 raft_consensus.cc:493] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:49.566262 22114 consensus_queue.cc:237] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 12.7, Last appended by leader: 7, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:49.567112 22551 raft_consensus.cc:3060] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:49.569185 22149 leader_election.cc:290] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 14 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:22:49.572660 22644 log.cc:927] Time spent T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.124s	user 0.000s	sys 0.000s
I20260709 12:22:49.553920 22069 raft_consensus.cc:3060] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:49.575713 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:49.556484 22509 raft_consensus.cc:515] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:49.576702 17684 raft_consensus.cc:1275] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260709 12:22:49.578382 22509 leader_election.cc:290] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 9 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:49.578426 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:49.579443 17683 raft_consensus.cc:2393] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 13: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:22:49.580340 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:49.580677 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:49.580952 17802 leader_election.cc:304] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:49.581717 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:49.582597 17802 leader_election.cc:304] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:22:49.582890 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:49.583660 22588 raft_consensus.cc:2804] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:22:49.584065 22588 raft_consensus.cc:493] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:49.584297 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:49.551927 22521 raft_consensus.cc:515] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:49.585091 17686 raft_consensus.cc:2468] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 12.
I20260709 12:22:49.584474 22588 raft_consensus.cc:3060] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:49.586824 22521 leader_election.cc:290] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:49.587260 22476 raft_consensus.cc:2804] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:22:49.587790 22476 raft_consensus.cc:697] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 11 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:22:49.588141 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:49.588688 22476 consensus_queue.cc:237] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 10.11, Last appended by leader: 11, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:49.589267 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:49.589380 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:49.589983 17686 raft_consensus.cc:3060] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:49.590178 17682 raft_consensus.cc:2393] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 9: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:22:49.592160 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:49.595932 22468 leader_election.cc:290] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:49.598948 22579 raft_consensus.cc:515] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:49.600234 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:49.601915 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:49.602308 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:49.603132 17681 raft_consensus.cc:2393] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 14: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:22:49.603467 17683 raft_consensus.cc:1275] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 13 index: 9. Preceding OpId from leader: term: 14 index: 9. (term mismatch)
I20260709 12:22:49.603864 17683 pending_rounds.cc:85] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3: Aborting all ops after (but not including) 8
I20260709 12:22:49.603947 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 12 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:49.604641 22518 leader_election.cc:290] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 13 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:49.545558 22416 leader_election.cc:290] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 13 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:49.606722 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 14 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:49.607379 18366 raft_consensus.cc:2468] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 13.
I20260709 12:22:49.610993 22535 leader_election.cc:290] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:49.612074 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:49.612802 18366 raft_consensus.cc:2468] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 8.
I20260709 12:22:49.613799 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:49.615144 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:49.584367 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 18 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:22:49.616026 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 106773us
I20260709 12:22:49.617645 17652 leader_election.cc:304] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:22:49.618062 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:49.604233 17683 pending_rounds.cc:107] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3: Aborting uncommitted NO_OP operation due to leader change: 13.9
I20260709 12:22:49.619104 22495 raft_consensus.cc:2804] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:22:49.619449 17683 raft_consensus.cc:2889] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:49.619670 22495 raft_consensus.cc:493] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:22:49.620175 22495 raft_consensus.cc:3060] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:49.621522 22360 raft_consensus.cc:515] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:49.624598 22360 leader_election.cc:290] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 13 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:49.625361 22615 consensus_queue.cc:1048] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 12:22:49.627964 18334 leader_election.cc:304] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:49.628118 22406 consensus_queue.cc:1059] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 12:22:49.629195 22406 raft_consensus.cc:3055] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:22:49.629190 22615 raft_consensus.cc:3060] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:49.630771 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:49.630872 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:49.632107 17829 raft_consensus.cc:3060] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:22:49.633850 17832 raft_consensus.cc:1275] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 11 index: 9. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
I20260709 12:22:49.634958 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:49.637926 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:49.639564 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:49.640304 18363 raft_consensus.cc:2468] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 10.
I20260709 12:22:49.642263 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 13 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:49.639801 17829 raft_consensus.cc:2468] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 15.
I20260709 12:22:49.643030 18363 raft_consensus.cc:2410] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 13 because replica has last-logged OpId of term: 12 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 8.
I20260709 12:22:49.619036 17680 raft_consensus.cc:2468] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 11.
I20260709 12:22:49.631650 17837 raft_consensus.cc:2468] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 11.
I20260709 12:22:49.619208 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:49.652535 17686 raft_consensus.cc:2468] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 12.
I20260709 12:22:49.647464 18334 leader_election.cc:304] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:49.629933 22406 raft_consensus.cc:740] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:22:49.623354 22376 raft_consensus.cc:515] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:49.695494 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:49.696696 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:49.697482 17829 raft_consensus.cc:2468] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 12.
I20260709 12:22:49.698578 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:49.702414 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:22:49.710208 17804 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:49.712415 22491 consensus_meta.cc:287] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.673s	user 0.010s	sys 0.004s
W20260709 12:22:49.715427 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 68041.9us
I20260709 12:22:49.661171 22625 raft_consensus.cc:3060] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:49.712900 22491 raft_consensus.cc:515] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:49.714380 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:49.722432 18364 raft_consensus.cc:3060] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:49.725056 18362 raft_consensus.cc:2393] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 9: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:22:49.723815 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:49.724910 22619 consensus_queue.cc:1048] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.003s
W20260709 12:22:49.730204 17803 leader_election.cc:336] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Timed out: RequestConsensusVote RPC to 127.16.29.68:42577 timed out after 1.627s (SENT)
I20260709 12:22:49.730741 17803 leader_election.cc:304] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:49.732604 22590 raft_consensus.cc:3060] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:49.732983 22376 leader_election.cc:290] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 9 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:49.646433 22588 raft_consensus.cc:515] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:49.733405 22069 raft_consensus.cc:515] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:22:49.734923 22640 log.cc:927] Time spent T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.450s	user 0.006s	sys 0.010s
I20260709 12:22:49.735060 22588 leader_election.cc:290] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 11 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:49.735384 22069 leader_election.cc:290] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 10 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:49.737599 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:49.738387 18365 raft_consensus.cc:2393] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 12: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:22:49.739200 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 9 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:49.740087 22475 raft_consensus.cc:515] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:49.740581 17832 raft_consensus.cc:1240] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 10. Current term is 11. Ops: []
I20260709 12:22:49.742000 17832 raft_consensus.cc:1240] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 14. Current term is 15. Ops: [13.7-14.8]
I20260709 12:22:49.742363 22475 leader_election.cc:290] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 9 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:49.744027 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 11 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:49.744480 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:49.745324 17837 raft_consensus.cc:3060] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:49.750306 18363 raft_consensus.cc:2393] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 8: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:22:49.712955 22557 raft_consensus.cc:493] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:49.754678 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 12 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:49.766666 18362 raft_consensus.cc:2410] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 12 because replica has last-logged OpId of term: 11 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 11.
I20260709 12:22:49.694341 22551 raft_consensus.cc:515] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:49.771917 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:22:49.780725 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 256072us
I20260709 12:22:49.781802 17729 leader_election.cc:304] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:49.709893 22192 raft_consensus.cc:515] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:49.785269 18366 raft_consensus.cc:2393] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 11: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:22:49.785740 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 12 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:49.800443 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:49.807771 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:49.808712 18366 raft_consensus.cc:3060] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:49.790088 18365 raft_consensus.cc:2468] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 11.
I20260709 12:22:49.819378 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:49.820472 18363 raft_consensus.cc:2393] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 12: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:22:49.821317 17729 leader_election.cc:304] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:49.821380 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:49.812119 22417 raft_consensus.cc:493] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:49.832899 17757 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38786 (request call id 2956) took 1129 ms. Trace:
I20260709 12:22:49.833364 17757 rpcz_store.cc:276] 0709 12:22:48.708810 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:49.034519 (+325709us) service_pool.cc:224] Handling call
0709 12:22:49.034915 (+   396us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:49.325024 (+290109us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:49.829455 (+504431us) spinlock_profiling.cc:243] Waited 288 ms on lock 0x7b54000ff0d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:22:49.832838 (+  3383us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":662737280}
I20260709 12:22:49.813426 22069 raft_consensus.cc:2804] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:22:49.842633 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:22:49.820772 17652 connection.cc:256] RPC call timeout handler was delayed by -1.2256278539998675s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19140000000000001s for a response
W20260709 12:22:49.843608 17652 connection.cc:256] RPC call timeout handler was delayed by -1.2256278539998675s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19140000000000001s for a response
W20260709 12:22:49.850033 18334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:49.860244 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:49.862001 18368 raft_consensus.cc:2393] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 9: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:22:49.863024 18364 raft_consensus.cc:2468] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 9.
I20260709 12:22:49.863150 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 13 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:49.864166 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:49.866524 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:49.867424 17759 raft_consensus.cc:2468] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 12.
I20260709 12:22:49.868095 18374 raft_consensus.cc:2468] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 12.
I20260709 12:22:49.870314 17801 leader_election.cc:304] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:49.873644 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 11 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:49.874195 18366 raft_consensus.cc:2468] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 9.
I20260709 12:22:49.875388 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:49.875643 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:49.876317 17727 leader_election.cc:304] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:49.876502 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:49.877379 18364 raft_consensus.cc:2393] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 14: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:22:49.877326 17727 leader_election.cc:304] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:49.878368 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:49.879019 18364 raft_consensus.cc:3060] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:49.880978 22360 consensus_queue.cc:1048] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:22:49.883124 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 11 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
W20260709 12:22:49.860956 22647 log.cc:927] Time spent T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.088s	user 0.000s	sys 0.000s
I20260709 12:22:49.884696 22383 raft_consensus.cc:2804] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:22:49.883312 22360 raft_consensus.cc:2804] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:22:49.886037 18362 raft_consensus.cc:2393] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 11: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:22:49.886559 17686 raft_consensus.cc:1275] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 12 index: 8. Preceding OpId from leader: term: 14 index: 9. (index mismatch)
I20260709 12:22:49.885572 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:49.897567 18371 raft_consensus.cc:3060] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:49.893695 17687 raft_consensus.cc:2468] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 11.
I20260709 12:22:49.783644 22557 raft_consensus.cc:515] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:22:49.898861 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 43995.2us
I20260709 12:22:49.901556 17802 leader_election.cc:304] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:22:49.899649 17837 raft_consensus.cc:2468] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 13.
I20260709 12:22:49.905083 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
W20260709 12:22:49.908315 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 167171us
I20260709 12:22:49.885141 22383 raft_consensus.cc:493] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:49.908134 17583 catalog_manager.cc:5697] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 6 to 7, leader changed from e063541542a94969898868bf6eb5ce1b (127.16.29.67) to e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66). New cstate: current_term: 7 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:22:49.913033 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 18 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:49.915026 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:49.910274 17654 leader_election.cc:304] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:22:49.841575 22069 raft_consensus.cc:493] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:49.915648 18362 raft_consensus.cc:2376] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 9. Re-sending same reply.
I20260709 12:22:49.885324 17729 leader_election.cc:304] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:22:49.917550 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 47333.4us
I20260709 12:22:49.926523 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:49.925020 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:49.927743 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:49.928645 18362 raft_consensus.cc:2410] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 12 because replica has last-logged OpId of term: 11 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 9.
I20260709 12:22:49.930727 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:49.931831 18368 raft_consensus.cc:2393] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 14: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:22:49.932547 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:49.937855 18366 raft_consensus.cc:2393] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 12: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:22:49.923748 22360 raft_consensus.cc:697] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:22:49.916885 22383 raft_consensus.cc:3060] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:49.914502 22583 raft_consensus.cc:2749] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:22:49.916261 22518 raft_consensus.cc:493] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:49.946820 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:49.948503 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:49.951447 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:49.934566 22365 raft_consensus.cc:2804] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:22:49.958451 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:49.849741 22417 raft_consensus.cc:515] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:22:49.980356 17730 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:49.981432 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:49.982287 17837 raft_consensus.cc:2393] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 9: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:22:49.982324 17757 raft_consensus.cc:1240] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 13. Current term is 14. Ops: []
I20260709 12:22:49.983171 17762 raft_consensus.cc:1275] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 14 index: 13. Preceding OpId from leader: term: 17 index: 15. (index mismatch)
I20260709 12:22:49.924166 22069 raft_consensus.cc:3060] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:49.983693 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:49.984407 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:49.985080 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:49.926367 22588 raft_consensus.cc:2804] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Leader election won for term 11
W20260709 12:22:49.987773 17803 leader_election.cc:336] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Timed out: RequestConsensusVote RPC to 127.16.29.68:42577 timed out after 1.694s (SENT)
I20260709 12:22:49.924410 22491 leader_election.cc:290] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 9 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:49.988476 17803 leader_election.cc:304] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:49.910823 22475 consensus_queue.cc:1048] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.034s
I20260709 12:22:49.926281 22476 raft_consensus.cc:2749] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:22:49.958110 18364 raft_consensus.cc:2468] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 13.
I20260709 12:22:49.999361 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:49.960091 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:22:49.954409 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 69198.3us
I20260709 12:22:49.969426 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:49.976238 22518 raft_consensus.cc:515] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:49.829519 22406 consensus_queue.cc:260] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.8, Last appended by leader: 8, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:49.958105 22579 leader_election.cc:290] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 9 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:49.983186 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:50.019440 17837 raft_consensus.cc:2393] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 9: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:22:50.019165 18365 raft_consensus.cc:3060] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Advancing to term 12
I20260709 12:22:50.019150 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 14 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:49.990494 17654 leader_election.cc:304] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:22:50.004427 17729 leader_election.cc:304] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:22:50.010514 18368 raft_consensus.cc:2393] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 9: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:22:50.045190 18366 raft_consensus.cc:3060] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:22:50.042829 17652 leader_election.cc:336] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 10 election: RPC error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Timed out: RequestConsensusVote RPC to 127.16.29.66:39651 timed out after 1.914s (SENT)
I20260709 12:22:50.051756 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:50.052134 17652 leader_election.cc:304] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:50.053691 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:50.056624 17760 raft_consensus.cc:2410] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for term 13 because replica has last-logged OpId of term: 12 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 10.
I20260709 12:22:50.057116 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 11 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:22:50.058719 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 54399.9us
I20260709 12:22:50.062129 17729 leader_election.cc:304] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:49.934371 22488 consensus_queue.cc:1048] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.103s
I20260709 12:22:50.068599 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:50.076393 18371 raft_consensus.cc:2468] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 10.
I20260709 12:22:50.077348 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:50.078397 17764 raft_consensus.cc:2393] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 13: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:22:50.081936 17755 raft_consensus.cc:1240] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 8. Current term is 9. Ops: []
I20260709 12:22:50.144999 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
W20260709 12:22:50.148922 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 219578us
I20260709 12:22:50.151166 18363 raft_consensus.cc:2393] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 9: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:22:50.152573 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:50.069087 22491 raft_consensus.cc:2804] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:22:50.064957 22583 raft_consensus.cc:2749] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 12:22:49.999413 22305 raft_consensus.cc:515] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:22:50.180012 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 243208us
I20260709 12:22:50.182525 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
W20260709 12:22:50.226466 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 238756us
I20260709 12:22:50.227629 17727 leader_election.cc:304] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:50.143702 22579 raft_consensus.cc:2749] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:22:50.149979 22406 raft_consensus.cc:3060] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:50.236383 17727 leader_election.cc:304] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:22:50.230391 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:50.239117 17837 raft_consensus.cc:3060] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Advancing to term 13
I20260709 12:22:50.240720 18363 raft_consensus.cc:3060] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Advancing to term 14
W20260709 12:22:50.234525 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 50485.2us
I20260709 12:22:50.235363 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 14 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:50.241556 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:50.155804 22653 raft_consensus.cc:2749] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:22:50.158093 22551 leader_election.cc:290] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 14 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:50.247074 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:50.185247 22192 leader_election.cc:290] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 14 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:50.280776 17760 raft_consensus.cc:1275] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260709 12:22:50.226171 22588 raft_consensus.cc:697] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 11 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:22:50.280732 17835 raft_consensus.cc:2393] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 10: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
W20260709 12:22:50.288363 17803 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:50.244875 22583 raft_consensus.cc:2749] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
W20260709 12:22:50.321463 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 331031us
I20260709 12:22:50.306094 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:50.355939 18366 raft_consensus.cc:1275] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 10 index: 7. Preceding OpId from leader: term: 11 index: 7. (term mismatch)
I20260709 12:22:50.261703 22548 raft_consensus.cc:2804] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Leader pre-election won for term 10
W20260709 12:22:50.375134 22495 consensus_meta.cc:287] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.755s	user 0.008s	sys 0.001s
I20260709 12:22:50.375192 22548 raft_consensus.cc:493] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:50.375718 22548 raft_consensus.cc:3060] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:50.303669 22515 raft_consensus.cc:2749] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:22:50.381522 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:50.356454 18366 pending_rounds.cc:85] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da: Aborting all ops after (but not including) 6
I20260709 12:22:50.385284 18366 pending_rounds.cc:107] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da: Aborting uncommitted CHANGE_CONFIG_OP operation due to leader change: 10.7
I20260709 12:22:50.385584 18366 raft_consensus.cc:2917] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Aborting config change with OpId 10.7: Aborted: Op aborted by new leader
I20260709 12:22:50.386139 18366 raft_consensus.cc:2889] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [term 11 LEARNER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:50.356750 17762 raft_consensus.cc:2393] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 13: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:22:50.362552 22305 leader_election.cc:290] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 11 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:50.362264 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:50.363023 22652 raft_consensus.cc:2804] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:22:50.390455 17763 raft_consensus.cc:3060] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:50.390566 22652 raft_consensus.cc:697] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 13 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:22:50.391827 22652 consensus_queue.cc:237] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.9, Last appended by leader: 9, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:22:50.402897 17587 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:37172 (request call id 254) took 32976 ms (client timeout 15000 ms). Trace:
I20260709 12:22:50.405012 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
W20260709 12:22:50.364152 22651 log.cc:927] Time spent T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.322s	user 0.001s	sys 0.000s
I20260709 12:22:50.179735 22360 consensus_queue.cc:237] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 13.7, Last appended by leader: 7, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:50.408098 17583 catalog_manager.cc:5697] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 6 to 8. New cstate: current_term: 8 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
W20260709 12:22:50.369824 22597 consensus_meta.cc:287] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.826s	user 0.007s	sys 0.003s
I20260709 12:22:50.226466 22660 consensus_queue.cc:1059] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 11.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.110s
W20260709 12:22:50.373287 22658 log.cc:927] Time spent T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.087s	user 0.000s	sys 0.000s
W20260709 12:22:50.374600 22664 log.cc:927] Time spent T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.090s	user 0.000s	sys 0.001s
I20260709 12:22:50.246064 22491 raft_consensus.cc:697] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
W20260709 12:22:50.374724 18334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:50.405951 17758 raft_consensus.cc:2393] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 10: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
W20260709 12:22:50.281179 22607 consensus_meta.cc:287] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.753s	user 0.011s	sys 0.000s
W20260709 12:22:50.411257 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 49971.3us
I20260709 12:22:50.389465 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 11 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:50.407224 17728 leader_election.cc:304] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
W20260709 12:22:50.367374 22615 consensus_meta.cc:287] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.737s	user 0.006s	sys 0.002s
W20260709 12:22:50.431941 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 250009us
I20260709 12:22:50.440932 18374 raft_consensus.cc:1275] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 11 index: 12. Preceding OpId from leader: term: 12 index: 12. (term mismatch)
I20260709 12:22:50.454321 18374 pending_rounds.cc:85] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da: Aborting all ops after (but not including) 11
I20260709 12:22:50.455132 18374 pending_rounds.cc:107] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da: Aborting uncommitted NO_OP operation due to leader change: 11.12
I20260709 12:22:50.455471 18374 raft_consensus.cc:2889] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:22:50.444224 17803 leader_election.cc:336] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Timed out: RequestConsensusVote RPC to 127.16.29.68:42577 timed out after 1.556s (SENT)
I20260709 12:22:50.431667 17831 raft_consensus.cc:2468] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 13.
I20260709 12:22:50.458045 18363 raft_consensus.cc:2468] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 14.
I20260709 12:22:50.419685 18333 leader_election.cc:304] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:50.450896 17763 raft_consensus.cc:2468] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 11.
I20260709 12:22:50.281956 22475 raft_consensus.cc:493] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:50.457187 17803 leader_election.cc:304] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:22:50.452535 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 45426.7us
I20260709 12:22:50.297410 22649 raft_consensus.cc:2804] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:22:50.475466 17728 leader_election.cc:304] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:22:50.465361 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:50.465945 18365 raft_consensus.cc:2468] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 12.
I20260709 12:22:50.464576 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:50.476357 17688 raft_consensus.cc:3060] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Advancing to term 14
I20260709 12:22:50.454393 17729 leader_election.cc:304] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:50.422363 22417 leader_election.cc:290] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:50.486445 17683 raft_consensus.cc:2468] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 9.
I20260709 12:22:50.421551 22557 leader_election.cc:290] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:50.481456 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:50.487639 17801 leader_election.cc:304] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:50.493742 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:50.493898 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:50.495244 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 13 candidate_status { last_received { term: 11 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:50.375666 22495 raft_consensus.cc:515] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:50.500898 17762 raft_consensus.cc:2393] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 11: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:22:50.500895 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 14 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:50.497611 17761 raft_consensus.cc:2468] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 14.
I20260709 12:22:50.492671 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:50.504922 17683 raft_consensus.cc:2393] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 11: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:22:50.506548 17686 raft_consensus.cc:1275] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 12 index: 10. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
I20260709 12:22:50.494369 17681 raft_consensus.cc:2468] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 11.
I20260709 12:22:50.510110 22518 leader_election.cc:290] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:50.510707 22583 raft_consensus.cc:493] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:50.510874 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 82129.6us
W20260709 12:22:50.511651 17803 leader_election.cc:336] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Timed out: RequestConsensusVote RPC to 127.16.29.68:42577 timed out after 1.576s (SENT)
I20260709 12:22:50.511209 22583 raft_consensus.cc:515] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:50.512207 17803 leader_election.cc:304] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:50.512573 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:50.513439 17762 raft_consensus.cc:2393] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 9: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
W20260709 12:22:50.513628 17803 leader_election.cc:336] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 9 election: RPC error from VoteRequest() call to peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Timed out: RequestConsensusVote RPC to 127.16.29.68:42577 timed out after 1.631s (SENT)
I20260709 12:22:50.514042 22583 leader_election.cc:290] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:50.514179 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:50.514565 22583 raft_consensus.cc:493] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:50.515023 17764 raft_consensus.cc:1240] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 10. Current term is 11. Ops: [10.8-10.8]
I20260709 12:22:50.518347 22548 raft_consensus.cc:515] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:50.518769 22649 raft_consensus.cc:697] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:22:50.514052 17803 leader_election.cc:304] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:50.520462 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:50.453852 22551 raft_consensus.cc:2749] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:22:50.472500 22468 raft_consensus.cc:493] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:50.525457 17758 raft_consensus.cc:2393] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 11: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
W20260709 12:22:50.508512 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 54248.7us
I20260709 12:22:50.523185 17835 raft_consensus.cc:2468] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 9.
W20260709 12:22:50.508713 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 465880us
I20260709 12:22:50.528424 17729 leader_election.cc:304] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:50.529258 17729 leader_election.cc:304] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:50.520097 22588 consensus_queue.cc:237] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:50.530839 22491 consensus_queue.cc:237] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 11.11, Last appended by leader: 11, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:50.483278 22615 raft_consensus.cc:2749] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:22:50.505712 22535 raft_consensus.cc:493] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:50.572826 17653 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.543s	user 0.003s	sys 0.011s
I20260709 12:22:50.566171 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:50.508003 22557 raft_consensus.cc:493] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:50.453454 22607 raft_consensus.cc:515] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:22:50.526034 22590 consensus_meta.cc:287] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.793s	user 0.011s	sys 0.000s
W20260709 12:22:50.606277 17653 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.577s	user 0.003s	sys 0.017s
I20260709 12:22:50.610172 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:50.606140 17583 catalog_manager.cc:5697] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 8 to 11, leader changed from e063541542a94969898868bf6eb5ce1b (127.16.29.67) to 334a4e1f59944544b3b3ff36616b57da (127.16.29.68). New cstate: current_term: 11 leader_uuid: "334a4e1f59944544b3b3ff36616b57da" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:22:50.612238 18364 raft_consensus.cc:1240] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 12. Current term is 13. Ops: []
I20260709 12:22:50.525465 22518 raft_consensus.cc:2804] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:22:50.597944 17837 raft_consensus.cc:2468] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 13.
I20260709 12:22:50.573874 22661 consensus_queue.cc:1048] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.073s
W20260709 12:22:50.624284 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 145896us
W20260709 12:22:50.629494 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 154123us
I20260709 12:22:50.501123 22475 raft_consensus.cc:515] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:50.630596 17728 leader_election.cc:304] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:50.631561 17583 catalog_manager.cc:5697] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 6 to 8. New cstate: current_term: 8 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:22:50.633960 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:50.634850 17583 catalog_manager.cc:5697] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:22:50.635543 18366 raft_consensus.cc:2410] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for term 14 because replica has last-logged OpId of term: 13 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 11.
I20260709 12:22:50.515055 22583 raft_consensus.cc:515] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:50.636597 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
W20260709 12:22:50.637915 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:50.580017 22192 raft_consensus.cc:2749] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 12:22:50.605907 22551 raft_consensus.cc:2749] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 12:22:50.640328 17758 raft_consensus.cc:1275] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 12 index: 8. Preceding OpId from leader: term: 14 index: 9. (index mismatch)
I20260709 12:22:50.620541 22590 raft_consensus.cc:2749] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:22:50.545202 22468 raft_consensus.cc:515] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:50.645694 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:22:50.632467 22651 log.cc:927] Time spent T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.057s	user 0.001s	sys 0.000s
I20260709 12:22:50.631630 17652 leader_election.cc:304] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:50.649559 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
W20260709 12:22:50.637310 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 279533us
I20260709 12:22:50.623345 22597 raft_consensus.cc:2764] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Leader pre-election decision vote started in defunct term 7: won
I20260709 12:22:50.580804 22660 consensus_queue.cc:1059] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.061s
I20260709 12:22:50.667101 17837 raft_consensus.cc:2393] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 14: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:22:50.664278 17830 raft_consensus.cc:2468] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 11.
I20260709 12:22:50.648396 17583 catalog_manager.cc:5697] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 8 to 9, leader changed from e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66) to 334a4e1f59944544b3b3ff36616b57da (127.16.29.68). New cstate: current_term: 9 leader_uuid: "334a4e1f59944544b3b3ff36616b57da" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:22:50.633903 22518 raft_consensus.cc:697] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 10 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:22:50.520088 22649 consensus_queue.cc:237] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:50.618942 22535 raft_consensus.cc:515] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:50.690738 17680 raft_consensus.cc:2410] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 11 because replica has last-logged OpId of term: 10 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 7.
I20260709 12:22:50.643579 22619 raft_consensus.cc:493] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:50.675541 17801 leader_election.cc:304] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
W20260709 12:22:50.677065 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 163473us
I20260709 12:22:50.676867 22529 consensus_queue.cc:1048] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.071s
I20260709 12:22:50.677400 22677 raft_consensus.cc:2749] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:22:50.678342 22590 consensus_queue.cc:1059] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 13.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:22:50.671187 22305 raft_consensus.cc:2804] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:22:50.722518 17688 raft_consensus.cc:2468] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 14.
W20260709 12:22:50.679744 22666 log.cc:927] Time spent T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.323s	user 0.001s	sys 0.000s
W20260709 12:22:50.674033 22625 consensus_meta.cc:287] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.946s	user 0.004s	sys 0.008s
I20260709 12:22:50.676702 22676 raft_consensus.cc:2749] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:22:50.680852 18366 raft_consensus.cc:1240] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 13. Current term is 14. Ops: []
I20260709 12:22:50.682302 22376 raft_consensus.cc:2749] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:22:50.685633 22583 leader_election.cc:290] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:50.682680 17583 catalog_manager.cc:5697] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 10 to 12. New cstate: current_term: 12 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:22:50.687351 22607 leader_election.cc:290] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 11 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:50.686703 22475 leader_election.cc:290] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:50.623145 22557 raft_consensus.cc:515] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:22:50.747668 22684 raft_consensus.cc:2749] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:22:50.757791 22557 leader_election.cc:290] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:50.759157 22365 raft_consensus.cc:493] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:50.759387 17758 raft_consensus.cc:1275] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
W20260709 12:22:50.702859 17803 leader_election.cc:336] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Timed out: RequestConsensusVote RPC to 127.16.29.68:42577 timed out after 1.792s (SENT)
I20260709 12:22:50.760290 17803 leader_election.cc:304] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:50.703516 18367 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59902 (request call id 1206) took 1339 ms. Trace:
W20260709 12:22:50.403349 17587 rpcz_store.cc:269] 0709 12:22:17.426588 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:17.426908 (+   320us) service_pool.cc:224] Handling call
0709 12:22:50.402624 (+32975716us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:22:37.233429 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:22:37.234057 (+   628us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:22:37.234110 (+    53us) write_op.cc:435] Acquired schema lock
0709 12:22:37.234130 (+    20us) tablet.cc:662] Decoding operations
0709 12:22:37.235319 (+  1189us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:22:37.235436 (+   117us) write_op.cc:641] Partition lock acquired for write op
0709 12:22:37.235464 (+    28us) tablet.cc:685] Acquiring locks for 13 operations
0709 12:22:37.333627 (+ 98163us) tablet.cc:701] Row locks acquired
0709 12:22:37.333664 (+    37us) write_op.cc:260] PREPARE: finished
0709 12:22:37.333876 (+   212us) write_op.cc:270] Start()
0709 12:22:37.334042 (+   166us) write_op.cc:276] Timestamp: P: 1783599757333785 usec, L: 0
0709 12:22:37.334064 (+    22us) op_driver.cc:348] REPLICATION: starting
0709 12:22:37.334870 (+   806us) log.cc:844] Serialized 5094 byte log entry
0709 12:22:38.010560 (+675690us) spinlock_profiling.cc:243] Waited 1.14 ms on lock 0x7b180027f4c0. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b475333f3 00007f7b475357c1 00007f7b47535779 00007f7b47535709 00007f7b475356d1 00007f7b475349cd 0000000000388634 0000000000388479 00007f7b4751683a
0709 12:22:38.776566 (+766006us) op_driver.cc:464] REPLICATION: finished
0709 12:22:39.265804 (+489238us) write_op.cc:301] APPLY: starting
0709 12:22:39.265941 (+   137us) tablet.cc:1367] starting BulkCheckPresence
0709 12:22:41.358165 (+2092224us) spinlock_profiling.cc:243] Waited 3.12 ms on lock 0x7b180027f4c0. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b4a35f0ec 00007f7b4a3abc0a 00007f7b4a3ab4f7 00007f7b4a3ab490 00007f7b4a356754 00007f7b4e076f13 00007f7b4e077532 00007f7b4e0beb93 00007f7b4df6bde3
0709 12:22:41.538280 (+180115us) tablet.cc:1370] finished BulkCheckPresence
0709 12:22:41.538329 (+    49us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:22:41.864202 (+325873us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:22:41.864700 (+   498us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=26,key_file_lookups=26,delta_file_lookups=26,mrs_lookups=0
0709 12:22:41.864772 (+    72us) write_op.cc:312] APPLY: finished
0709 12:22:41.865630 (+   858us) log.cc:844] Serialized 121 byte log entry
0709 12:22:42.332117 (+466487us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:22:42.403663 (+ 71546us) write_op.cc:454] Released schema lock
0709 12:22:42.412264 (+  8601us) write_op.cc:341] FINISH: Updating metrics
0709 12:22:42.412734 (+   470us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:22:42.412763 (+    29us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":1531493,"reported_tablets":42,"child_traces":[["op",{"apply.queue_time_us":399697,"apply.run_cpu_time_us":156933,"apply.run_wall_time_us":3146964,"cfile_cache_hit":54,"cfile_cache_hit_bytes":58519,"num_ops":13,"prepare.queue_time_us":1070597,"prepare.run_cpu_time_us":31723,"prepare.run_wall_time_us":530903,"raft.queue_time_us":439496,"raft.run_cpu_time_us":34060,"raft.run_wall_time_us":352572,"replication_time_us":1379402,"spinlock_wait_cycles":9814144,"thread_start_us":1850835,"threads_started":5,"wal-append.queue_time_us":1184618,"wal-append.run_cpu_time_us":12018,"wal-append.run_wall_time_us":2518845}]]}
I20260709 12:22:50.762063 18367 rpcz_store.cc:276] 0709 12:22:49.363619 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:49.376919 (+ 13300us) service_pool.cc:224] Handling call
0709 12:22:49.401624 (+ 24705us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:49.434388 (+ 32764us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:49.434450 (+    62us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:49.511295 (+ 76845us) log.cc:844] Serialized 26 byte log entry
0709 12:22:49.815219 (+303924us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:22:49.834822 (+ 19603us) log.cc:844] Serialized 14 byte log entry
0709 12:22:49.955709 (+120887us) spinlock_profiling.cc:243] Waited 1.02 ms on lock 0x7b1800478240. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b47515e07 00007f7b503709a9 00007f7b4753269b 00007f7b4753093f 00007f7b47530455 00007f7b4a618060 00007f7b4a62a716 00007f7b4a63b287 00007f7b4a63b208
0709 12:22:50.155064 (+199355us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:50.155135 (+    71us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:50.689853 (+534718us) raft_consensus.cc:1685] finished
0709 12:22:50.703267 (+ 13414us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:50.703434 (+   167us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":18623,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":42,"raft.queue_time_us":24026,"raft.run_cpu_time_us":271,"raft.run_wall_time_us":7644,"spinlock_wait_cycles":2334976,"thread_start_us":481342,"threads_started":2,"wal-append.queue_time_us":397767}
I20260709 12:22:50.716683 22691 consensus_queue.cc:1059] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 12.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:22:50.671703 22551 raft_consensus.cc:2804] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Leader election won for term 12
W20260709 12:22:50.734735 17654 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:50.774909 17583 catalog_manager.cc:5697] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 8 to 10, leader changed from e063541542a94969898868bf6eb5ce1b (127.16.29.67) to 334a4e1f59944544b3b3ff36616b57da (127.16.29.68). New cstate: current_term: 10 leader_uuid: "334a4e1f59944544b3b3ff36616b57da" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:22:50.674216 22548 leader_election.cc:290] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 10 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:22:50.703428 22069 consensus_meta.cc:287] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.719s	user 0.010s	sys 0.002s
W20260709 12:22:50.740427 17730 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:50.740043 22305 raft_consensus.cc:697] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 14 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:22:50.743319 22625 raft_consensus.cc:2749] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:22:50.748201 22518 consensus_queue.cc:237] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:50.750264 18333 leader_election.cc:304] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
W20260709 12:22:50.761471 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 67985.8us
I20260709 12:22:50.774747 17830 raft_consensus.cc:1240] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 8. Current term is 9. Ops: []
W20260709 12:22:50.749750 22663 log.cc:927] Time spent T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.345s	user 0.000s	sys 0.001s
I20260709 12:22:50.755668 22673 raft_consensus.cc:2749] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:22:50.796188 17583 catalog_manager.cc:5697] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 7 to 9. New cstate: current_term: 9 leader_uuid: "334a4e1f59944544b3b3ff36616b57da" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 12:22:50.800981 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 271763us
I20260709 12:22:50.759799 22365 raft_consensus.cc:3060] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:50.783623 22590 raft_consensus.cc:2749] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:22:50.701040 22619 raft_consensus.cc:515] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:22:50.814610 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 219354us
I20260709 12:22:50.830695 17829 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35040 (request call id 2493) took 1124 ms. Trace:
I20260709 12:22:50.831190 17829 rpcz_store.cc:276] 0709 12:22:49.709941 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:49.721328 (+ 11387us) service_pool.cc:224] Handling call
0709 12:22:49.721743 (+   415us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:49.765264 (+ 43521us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:49.765309 (+    45us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:49.789600 (+ 24291us) log.cc:844] Serialized 26 byte log entry
0709 12:22:50.003007 (+213407us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:22:50.003059 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:50.003148 (+    89us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:50.825813 (+822665us) raft_consensus.cc:1685] finished
0709 12:22:50.825837 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:50.825995 (+   158us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":31583,"raft.queue_time_us":13049,"raft.run_cpu_time_us":239,"raft.run_wall_time_us":8640,"thread_start_us":154226,"threads_started":1,"wal-append.queue_time_us":332819}
W20260709 12:22:50.840886 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 165413us
I20260709 12:22:50.841702 17652 leader_election.cc:304] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:50.817134 22495 leader_election.cc:290] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 10 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:50.839046 17836 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35040 (request call id 2507) took 1057 ms. Trace:
I20260709 12:22:50.846737 17583 catalog_manager.cc:5697] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 8 to 11, leader changed from e063541542a94969898868bf6eb5ce1b (127.16.29.67) to e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66). New cstate: current_term: 11 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:22:50.819669 22551 raft_consensus.cc:697] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 12 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:22:50.826190 22192 raft_consensus.cc:2749] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:22:50.854019 17652 leader_election.cc:304] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:50.814785 22069 raft_consensus.cc:515] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:22:50.861539 17803 leader_election.cc:336] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Timed out: RequestConsensusVote RPC to 127.16.29.68:42577 timed out after 1.816s (SENT)
I20260709 12:22:50.863973 17803 leader_election.cc:304] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:50.860443 17583 catalog_manager.cc:5697] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 10 to 12. New cstate: current_term: 12 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:22:50.838798 22695 raft_consensus.cc:2804] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:22:50.849516 22548 raft_consensus.cc:2749] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:22:50.875377 18361 raft_consensus.cc:1275] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 11 index: 9. Preceding OpId from leader: term: 13 index: 10. (index mismatch)
I20260709 12:22:50.866972 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:50.879776 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 14 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:50.880482 17758 raft_consensus.cc:2393] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 14: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:22:50.885042 17583 catalog_manager.cc:5697] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 5 to 7, leader changed from 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) to e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66). New cstate: current_term: 7 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:22:50.886163 17764 raft_consensus.cc:2410] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 14 because replica has last-logged OpId of term: 12 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 6.
I20260709 12:22:50.872494 22495 raft_consensus.cc:2749] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:22:50.873701 22677 raft_consensus.cc:2749] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:22:50.874369 22682 raft_consensus.cc:3055] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:22:50.887444 22192 raft_consensus.cc:493] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
W20260709 12:22:50.895942 22675 log.cc:927] Time spent T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.056s	user 0.000s	sys 0.001s
I20260709 12:22:50.896572 17829 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35040 (request call id 2508) took 1079 ms. Trace:
W20260709 12:22:50.897104 22697 log.cc:927] Time spent T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.051s	user 0.000s	sys 0.000s
I20260709 12:22:50.897485 22548 raft_consensus.cc:493] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:50.909876 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 174804us
I20260709 12:22:50.904342 17583 catalog_manager.cc:5697] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 6 to 9. New cstate: current_term: 9 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:22:50.914356 17829 rpcz_store.cc:276] 0709 12:22:49.816817 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:50.846317 (+1029500us) service_pool.cc:224] Handling call
0709 12:22:50.864312 (+ 17995us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:50.895872 (+ 31560us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:50.895937 (+    65us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:50.896016 (+    79us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:22:50.896070 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:50.896134 (+    64us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:50.896419 (+   285us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:50.915838 17758 raft_consensus.cc:1240] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 10. Current term is 11. Ops: []
I20260709 12:22:50.921551 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:50.928474 17764 raft_consensus.cc:3060] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:50.908994 22192 raft_consensus.cc:515] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:50.931202 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:50.880105 22305 consensus_queue.cc:237] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.8, Last appended by leader: 8, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:50.885131 22660 consensus_queue.cc:1059] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.033s
I20260709 12:22:50.898072 22682 raft_consensus.cc:740] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 13 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:22:50.887703 22529 consensus_queue.cc:1048] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.034s
W20260709 12:22:50.936802 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 306294us
I20260709 12:22:50.934711 18333 leader_election.cc:304] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:22:50.937342 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 80225.7us
I20260709 12:22:50.938728 17728 leader_election.cc:304] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:50.940490 17728 leader_election.cc:304] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:50.940639 22700 raft_consensus.cc:493] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:50.944177 22376 raft_consensus.cc:2804] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:22:50.920917 22694 raft_consensus.cc:493] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:50.943413 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:50.946259 17829 raft_consensus.cc:3060] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:50.930980 22535 leader_election.cc:290] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:22:50.939234 22671 log.cc:927] Time spent T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.233s	user 0.000s	sys 0.001s
I20260709 12:22:50.951000 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:50.948917 22376 raft_consensus.cc:493] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:50.956744 22376 raft_consensus.cc:3060] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:22:50.958283 22650 log.cc:927] Time spent T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.905s	user 0.002s	sys 0.002s
I20260709 12:22:50.941506 22652 consensus_queue.cc:1048] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.019s
W20260709 12:22:50.961275 22383 consensus_meta.cc:287] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 1.003s	user 0.012s	sys 0.000s
I20260709 12:22:50.961735 22383 raft_consensus.cc:515] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:50.951043 22700 raft_consensus.cc:515] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:22:50.963124 22670 log.cc:927] Time spent T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.569s	user 0.002s	sys 0.006s
I20260709 12:22:50.964475 17832 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35040 (request call id 2501) took 1225 ms. Trace:
I20260709 12:22:50.964586 22700 leader_election.cc:290] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:50.950312 22693 raft_consensus.cc:493] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:50.964984 17832 rpcz_store.cc:276] 0709 12:22:49.739011 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:49.742663 (+  3652us) service_pool.cc:224] Handling call
0709 12:22:49.743005 (+   342us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:49.743421 (+   416us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:49.743457 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:49.767548 (+ 24091us) log.cc:844] Serialized 26 byte log entry
0709 12:22:50.036898 (+269350us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:22:50.036952 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:50.037043 (+    91us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:50.962838 (+925795us) spinlock_profiling.cc:243] Waited 31.6 ms on lock 0x7b24001cf4a8. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b473b99c4 00007f7b473b7acb 00007f7b47572dc9 00007f7b47572a55 00007f7b4757209d 00007f7b4a1a410e 00007f7b4a1bb88c 00007f7b4a1babb6 00007f7b4a1ba7d8 00007f7b4a1c4221
0709 12:22:50.964160 (+  1322us) raft_consensus.cc:1685] finished
0709 12:22:50.964181 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:50.964405 (+   224us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4659,"raft.queue_time_us":708,"raft.run_cpu_time_us":104,"raft.run_wall_time_us":3587,"spinlock_wait_cycles":72722816,"thread_start_us":206928,"threads_started":1,"wal-append.queue_time_us":538997}
I20260709 12:22:50.966764 22693 raft_consensus.cc:515] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:50.925307 22548 raft_consensus.cc:515] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:50.971400 22693 leader_election.cc:290] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:50.954120 17758 raft_consensus.cc:3060] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:50.954519 17756 raft_consensus.cc:1240] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 LEADER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 14. Current term is 15. Ops: [14.8-14.8]
I20260709 12:22:50.944551 22615 raft_consensus.cc:2749] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:22:50.923554 22551 consensus_queue.cc:237] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:50.944744 22491 raft_consensus.cc:2749] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:22:50.978184 17834 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35040 (request call id 2498) took 1240 ms. Trace:
I20260709 12:22:50.978607 17834 rpcz_store.cc:276] 0709 12:22:49.738028 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:49.743043 (+  5015us) service_pool.cc:224] Handling call
0709 12:22:49.743406 (+   363us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:49.751252 (+  7846us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:49.751311 (+    59us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:49.771455 (+ 20144us) log.cc:844] Serialized 26 byte log entry
0709 12:22:49.919840 (+148385us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:22:49.919906 (+    66us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:49.919971 (+    65us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:50.975409 (+1055438us) spinlock_profiling.cc:243] Waited 71.6 ms on lock 0x7b2400238f88. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b473b99c4 00007f7b473b7acb 00007f7b47572dc9 00007f7b47572a55 00007f7b47571fb0 00007f7b4a1e8488 00007f7b4a1e8be8 00007f7b4a1c0398 00007f7b4a1bb76a 00007f7b4a1babb6
0709 12:22:50.977871 (+  2462us) raft_consensus.cc:1685] finished
0709 12:22:50.977898 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:50.978119 (+   221us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2037,"raft.queue_time_us":8396,"raft.run_cpu_time_us":164,"raft.run_wall_time_us":11409,"spinlock_wait_cycles":164709760,"thread_start_us":147937,"threads_started":1,"wal-append.queue_time_us":303882}
W20260709 12:22:50.980726 22699 log.cc:927] Time spent T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.057s	user 0.000s	sys 0.001s
I20260709 12:22:50.981060 17834 raft_consensus.cc:1275] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 10 index: 11. Preceding OpId from leader: term: 12 index: 12. (index mismatch)
I20260709 12:22:50.981724 17834 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35040 (request call id 2510) took 1072 ms. Trace:
W20260709 12:22:50.955114 22674 log.cc:927] Time spent T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.064s	user 0.000s	sys 0.001s
W20260709 12:22:50.955189 17803 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:50.955709 22619 leader_election.cc:290] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:50.972951 22548 leader_election.cc:290] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:50.982834 17758 raft_consensus.cc:2468] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 10.
I20260709 12:22:50.969293 22383 leader_election.cc:290] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 13 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:50.985162 17834 rpcz_store.cc:276] 0709 12:22:49.909563 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:50.980291 (+1070728us) service_pool.cc:224] Handling call
0709 12:22:50.980637 (+   346us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:50.981481 (+   844us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:50.981678 (+   197us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:50.994069 17758 raft_consensus.cc:1240] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 13. Current term is 14. Ops: [13.10-13.10]
I20260709 12:22:50.941169 22682 consensus_queue.cc:260] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.11, Last appended by leader: 11, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:51.008085 22682 raft_consensus.cc:3060] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Advancing to term 14
W20260709 12:22:51.004112 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 69499.3us
I20260709 12:22:50.953709 17728 leader_election.cc:304] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:50.992763 17832 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35042 (request call id 2268) took 1133 ms. Trace:
W20260709 12:22:50.995941 22701 log.cc:927] Time spent T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.108s	user 0.000s	sys 0.000s
I20260709 12:22:50.997030 22069 leader_election.cc:290] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 13 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:50.997437 17834 raft_consensus.cc:1275] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260709 12:22:51.001309 22365 raft_consensus.cc:515] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:51.003985 22192 leader_election.cc:290] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:50.950362 22694 raft_consensus.cc:515] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:22:51.026378 17655 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:51.028777 22694 leader_election.cc:290] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:51.004236 17583 catalog_manager.cc:5697] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 2 to 11, leader changed from e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66) to 334a4e1f59944544b3b3ff36616b57da (127.16.29.68). New cstate: current_term: 11 leader_uuid: "334a4e1f59944544b3b3ff36616b57da" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:22:50.847139 17836 rpcz_store.cc:276] 0709 12:22:49.773045 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:50.689226 (+916181us) service_pool.cc:224] Handling call
0709 12:22:50.689897 (+   671us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:50.719741 (+ 29844us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 12:22:50.748000 (+ 28259us) log.cc:844] Serialized 14 byte log entry
0709 12:22:50.829143 (+ 81143us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:50.829199 (+    56us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:22:50.829251 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:50.830163 (+   912us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:50.830301 (+   138us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":28675,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":34,"raft.queue_time_us":4143,"raft.run_cpu_time_us":139,"raft.run_wall_time_us":6012,"thread_start_us":79913,"threads_started":1}
W20260709 12:22:51.003856 22657 log.cc:927] Time spent T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.486s	user 0.001s	sys 0.001s
I20260709 12:22:51.011961 17829 raft_consensus.cc:2468] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 10.
W20260709 12:22:51.008133 22685 log.cc:927] Time spent T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.130s	user 0.002s	sys 0.000s
W20260709 12:22:51.039582 17803 leader_election.cc:336] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Timed out: RequestConsensusVote RPC to 127.16.29.68:42577 timed out after 2.077s (SENT)
I20260709 12:22:51.047986 17803 leader_election.cc:304] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:51.049129 22578 raft_consensus.cc:2749] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260709 12:22:51.050027 17803 leader_election.cc:336] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 9 election: RPC error from VoteRequest() call to peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Timed out: RequestConsensusVote RPC to 127.16.29.68:42577 timed out after 2.194s (SENT)
I20260709 12:22:51.049335 17583 catalog_manager.cc:5697] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 3 to 9, leader changed from e063541542a94969898868bf6eb5ce1b (127.16.29.67) to 334a4e1f59944544b3b3ff36616b57da (127.16.29.68). New cstate: current_term: 9 leader_uuid: "334a4e1f59944544b3b3ff36616b57da" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:22:51.050539 17803 leader_election.cc:304] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:51.031762 17831 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35042 (request call id 2256) took 1398 ms. Trace:
I20260709 12:22:51.052060 17831 rpcz_store.cc:276] 0709 12:22:49.633071 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:49.634697 (+  1626us) service_pool.cc:224] Handling call
0709 12:22:51.031550 (+1396853us) spinlock_profiling.cc:243] Waited 796 ms on lock 0x7b54003354d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:22:51.031687 (+   137us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1830814976}
I20260709 12:22:51.032737 22619 raft_consensus.cc:493] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:51.053304 22619 raft_consensus.cc:515] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:22:51.055469 22619 leader_election.cc:290] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:51.060341 22476 raft_consensus.cc:493] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:51.060964 22476 raft_consensus.cc:515] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:51.063030 22476 leader_election.cc:290] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:51.040657 17759 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38776 (request call id 2257) took 1059 ms. Trace:
I20260709 12:22:51.064205 17759 rpcz_store.cc:276] 0709 12:22:49.981313 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:49.983338 (+  2025us) service_pool.cc:224] Handling call
0709 12:22:49.983686 (+   348us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:50.304512 (+320826us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 12:22:50.304559 (+    47us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:50.304621 (+    62us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:22:50.304676 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:51.040372 (+735696us) spinlock_profiling.cc:243] Waited 315 ms on lock 0x7b5400194e54. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:22:51.040430 (+    58us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:51.040585 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":9834,"raft.queue_time_us":58519,"raft.run_cpu_time_us":153,"raft.run_wall_time_us":21927,"spinlock_wait_cycles":723434496}
I20260709 12:22:51.051293 17583 catalog_manager.cc:5697] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 7 to 11, leader changed from 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) to 0da169ff82674dc1947055ce43de68a3 (127.16.29.65). New cstate: current_term: 11 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:22:51.067015 17583 catalog_manager.cc:5697] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 4 to 8, leader changed from e063541542a94969898868bf6eb5ce1b (127.16.29.67) to 0da169ff82674dc1947055ce43de68a3 (127.16.29.65). New cstate: current_term: 8 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:22:51.068262 22468 leader_election.cc:290] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:51.069463 22468 consensus_queue.cc:1048] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:22:51.015730 22491 raft_consensus.cc:2749] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 12:22:51.072429 17830 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35040 (request call id 2502) took 1333 ms. Trace:
I20260709 12:22:51.046839 22376 raft_consensus.cc:515] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:22:51.045138 17652 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:51.072875 17830 rpcz_store.cc:276] 0709 12:22:49.739345 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:50.692026 (+952681us) service_pool.cc:224] Handling call
0709 12:22:50.725672 (+ 33646us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:50.819260 (+ 93588us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:51.072203 (+252943us) spinlock_profiling.cc:243] Waited 18.9 ms on lock 0x7b5400266e54. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:22:51.072372 (+   169us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":43471232}
I20260709 12:22:51.079059 17688 raft_consensus.cc:1275] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 10 index: 11. Preceding OpId from leader: term: 11 index: 12. (index mismatch)
I20260709 12:22:51.077051 17764 raft_consensus.cc:2468] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 11.
I20260709 12:22:51.075103 17727 leader_election.cc:304] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:51.083704 17689 raft_consensus.cc:1240] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 9. Current term is 10. Ops: []
I20260709 12:22:51.084532 17652 leader_election.cc:304] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:51.087255 17802 leader_election.cc:304] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:22:51.012553 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 58918.4us
I20260709 12:22:51.012183 17832 rpcz_store.cc:276] 0709 12:22:49.851156 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:50.967191 (+1116035us) service_pool.cc:224] Handling call
0709 12:22:50.967523 (+   332us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:50.967709 (+   186us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:50.967732 (+    23us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:50.967768 (+    36us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:22:50.967807 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:50.967861 (+    54us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:50.984874 (+ 17013us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:50.971694 22661 raft_consensus.cc:3055] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:22:51.016711 18333 leader_election.cc:304] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:51.024006 22597 raft_consensus.cc:3055] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:22:51.025105 17837 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35040 (request call id 2505) took 1276 ms. Trace:
I20260709 12:22:51.102357 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:51.037022 17757 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38776 (request call id 2265) took 1028 ms. Trace:
I20260709 12:22:51.103045 17764 raft_consensus.cc:2468] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 13.
I20260709 12:22:51.101766 17837 rpcz_store.cc:276] 0709 12:22:49.749191 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:50.679469 (+930278us) service_pool.cc:224] Handling call
0709 12:22:50.680656 (+  1187us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:50.681929 (+  1273us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:50.681991 (+    62us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:50.683867 (+  1876us) log.cc:844] Serialized 26 byte log entry
0709 12:22:50.741334 (+ 57467us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:22:50.741373 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:50.741458 (+    85us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:51.024849 (+283391us) raft_consensus.cc:1685] finished
0709 12:22:51.024878 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:51.025037 (+   159us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":61825,"raft.run_cpu_time_us":136,"raft.run_wall_time_us":20461,"thread_start_us":55735,"threads_started":1,"wal-append.queue_time_us":144342}
I20260709 12:22:51.095830 17727 leader_election.cc:304] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:22:51.105371 22218 raft_consensus.cc:3060] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:51.108305 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:51.109176 17758 raft_consensus.cc:2410] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 10 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20260709 12:22:51.099531 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:51.090072 17652 leader_election.cc:304] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:51.092723 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:51.110942 17830 raft_consensus.cc:2468] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 9.
I20260709 12:22:51.110137 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 14 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:51.112651 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:51.112910 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:51.113585 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:51.116870 17686 raft_consensus.cc:1275] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 11 index: 6. Preceding OpId from leader: term: 14 index: 8. (index mismatch)
I20260709 12:22:51.088943 22529 raft_consensus.cc:493] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:51.128892 17652 leader_election.cc:304] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:22:51.127369 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 117752us
I20260709 12:22:51.088680 22619 raft_consensus.cc:493] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:51.103976 22578 raft_consensus.cc:2749] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:22:51.103969 22491 raft_consensus.cc:2804] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:22:51.101681 22597 raft_consensus.cc:740] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 12 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:22:51.113216 22476 raft_consensus.cc:2804] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:22:51.101477 22661 raft_consensus.cc:740] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:22:51.103034 17757 rpcz_store.cc:276] 0709 12:22:50.009667 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:50.035163 (+ 25496us) service_pool.cc:224] Handling call
0709 12:22:50.086863 (+ 51700us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:22:50.638493 (+551630us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:50.638538 (+    45us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:22:50.669872 (+ 31334us) log.cc:844] Serialized 52 byte log entry
0709 12:22:50.877367 (+207495us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:22:50.877393 (+    26us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:50.877445 (+    52us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:51.036741 (+159296us) raft_consensus.cc:1685] finished
0709 12:22:51.036762 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:51.036948 (+   186us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":52028,"spinlock_wait_cycles":13583744,"thread_start_us":486965,"threads_started":2,"wal-append.queue_time_us":283501}
I20260709 12:22:51.122323 22515 raft_consensus.cc:3060] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:51.143572 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:51.122735 22653 raft_consensus.cc:2804] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:22:51.144234 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:51.144518 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:51.145429 17727 leader_election.cc:304] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:51.122555 22192 raft_consensus.cc:2749] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 12:22:51.181479 17832 raft_consensus.cc:2468] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 10.
I20260709 12:22:51.146615 17680 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37402 (request call id 2703) took 1266 ms. Trace:
I20260709 12:22:51.181768 17831 raft_consensus.cc:2468] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 13.
I20260709 12:22:51.181449 17837 raft_consensus.cc:3060] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:51.124670 22383 raft_consensus.cc:493] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
W20260709 12:22:51.146315 17803 leader_election.cc:336] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Timed out: RequestConsensusVote RPC to 127.16.29.68:42577 timed out after 1.555s (SENT)
I20260709 12:22:51.182243 17680 rpcz_store.cc:276] 0709 12:22:49.873242 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:49.873362 (+   120us) service_pool.cc:224] Handling call
0709 12:22:51.139839 (+1266477us) spinlock_profiling.cc:243] Waited 816 ms on lock 0x7b5400345dd4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:22:51.140060 (+   221us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1876884736}
I20260709 12:22:51.147753 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:51.120105 22649 consensus_queue.cc:1048] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.026s
I20260709 12:22:51.187069 17836 raft_consensus.cc:3060] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Advancing to term 13
I20260709 12:22:51.185521 17727 leader_election.cc:304] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:22:51.186544 17803 leader_election.cc:304] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:51.142731 22491 raft_consensus.cc:493] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:51.194010 17727 leader_election.cc:304] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:51.195644 17727 leader_election.cc:304] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:51.126108 22579 consensus_queue.cc:1048] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.060s
W20260709 12:22:51.204185 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 75406.5us
I20260709 12:22:51.149160 22517 raft_consensus.cc:2749] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 12:22:51.210163 18333 leader_election.cc:304] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:22:51.181062 22653 raft_consensus.cc:493] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:22:51.186200 22656 log.cc:927] Time spent T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.255s	user 0.000s	sys 0.003s
I20260709 12:22:51.191124 22476 raft_consensus.cc:697] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 11 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:22:51.193638 22305 raft_consensus.cc:2804] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:22:51.234067 17837 raft_consensus.cc:2468] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 13.
I20260709 12:22:51.195245 22693 raft_consensus.cc:2749] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:22:51.196731 22491 raft_consensus.cc:3060] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:51.141794 22619 raft_consensus.cc:515] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:51.209594 22700 raft_consensus.cc:2804] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:22:51.149503 22691 consensus_queue.cc:1048] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 4, Time since last communication: 0.067s
I20260709 12:22:51.184037 22696 consensus_queue.cc:1059] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 9.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.086s
I20260709 12:22:51.217971 22360 raft_consensus.cc:493] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:51.232955 22653 raft_consensus.cc:3060] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Advancing to term 14
W20260709 12:22:51.247112 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 65948.2us
I20260709 12:22:51.209681 22588 raft_consensus.cc:2749] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:22:51.235678 17727 leader_election.cc:304] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:51.194655 22383 raft_consensus.cc:515] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:51.243258 17838 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35042 (request call id 2260) took 1604 ms. Trace:
I20260709 12:22:51.211174 22652 raft_consensus.cc:2804] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:22:51.255113 22652 raft_consensus.cc:493] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:51.255304 22673 raft_consensus.cc:3055] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:22:51.255596 22652 raft_consensus.cc:3060] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:51.256932 17837 raft_consensus.cc:1275] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 11 index: 11. Preceding OpId from leader: term: 13 index: 12. (index mismatch)
I20260709 12:22:51.233603 22305 raft_consensus.cc:493] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:51.240005 22700 raft_consensus.cc:493] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:51.236243 22615 raft_consensus.cc:2804] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:22:51.151696 22529 raft_consensus.cc:515] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:22:51.249276 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 103033us
I20260709 12:22:51.263525 22615 raft_consensus.cc:493] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:51.261929 22305 raft_consensus.cc:3060] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:22:51.256341 22711 log.cc:927] Time spent T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.063s	user 0.001s	sys 0.000s
I20260709 12:22:51.251363 17838 rpcz_store.cc:276] 0709 12:22:49.639196 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:49.640573 (+  1377us) service_pool.cc:224] Handling call
0709 12:22:49.640907 (+   334us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:22:49.641335 (+   428us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:49.641364 (+    29us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:22:49.642057 (+   693us) log.cc:844] Serialized 52 byte log entry
0709 12:22:49.919494 (+277437us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:22:50.149806 (+230312us) log.cc:844] Serialized 14 byte log entry
0709 12:22:50.150032 (+   226us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:50.150114 (+    82us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:51.195213 (+1045099us) spinlock_profiling.cc:243] Waited 42.9 ms on lock 0x7b2400238f88. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b473b99c4 00007f7b473b7acb 00007f7b47572dc9 00007f7b47572a55 00007f7b47571fb0 00007f7b4a1e8488 00007f7b4a1e8be8 00007f7b4a1c0398 00007f7b4a1bb76a 00007f7b4a1babb6
0709 12:22:51.238682 (+ 43469us) raft_consensus.cc:1685] finished
0709 12:22:51.243052 (+  4370us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:51.243201 (+   149us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5695,"prepare.queue_time_us":456439,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":42,"raft.queue_time_us":42069,"raft.run_cpu_time_us":351,"raft.run_wall_time_us":44311,"spinlock_wait_cycles":98590464,"thread_start_us":426798,"threads_started":2,"wal-append.queue_time_us":423379}
I20260709 12:22:51.255699 22673 raft_consensus.cc:740] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 9 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:22:51.236308 22661 consensus_queue.cc:260] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.10, Last appended by leader: 10, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:51.269076 17834 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35042 (request call id 2270) took 1039 ms. Trace:
I20260709 12:22:51.375676 17682 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37396 (request call id 2350) took 1792 ms. Trace:
I20260709 12:22:51.323868 18364 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59886 (request call id 1405) took 1524 ms. Trace:
W20260709 12:22:51.261454 22713 log.cc:927] Time spent T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.097s	user 0.000s	sys 0.015s
I20260709 12:22:51.454602 18364 rpcz_store.cc:276] 0709 12:22:49.799723 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:50.020107 (+220384us) service_pool.cc:224] Handling call
0709 12:22:50.020413 (+   306us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:50.612720 (+592307us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:51.323529 (+710809us) spinlock_profiling.cc:243] Waited 592 ms on lock 0x7b54007e8954. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:22:51.323783 (+   254us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1360601472}
I20260709 12:22:51.262250 22365 leader_election.cc:290] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 14 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:51.394775 22607 raft_consensus.cc:493] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:51.423357 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:22:51.358281 17803 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.16.29.68:42577 timed out after 2.027s (SENT) [suppressed 37 similar messages]
W20260709 12:22:51.375450 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 117865us
I20260709 12:22:51.308028 22542 raft_consensus.cc:493] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:51.458186 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 112011us
W20260709 12:22:51.372643 22658 log.cc:927] Time spent T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.076s	user 0.000s	sys 0.000s
I20260709 12:22:51.310891 18333 leader_election.cc:304] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:22:51.444416 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:51.452068 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:51.261210 22684 consensus_queue.cc:1048] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.195s
I20260709 12:22:51.407426 17834 rpcz_store.cc:276] 0709 12:22:50.229677 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:51.019716 (+790039us) service_pool.cc:224] Handling call
0709 12:22:51.020610 (+   894us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:22:51.044965 (+ 24355us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:51.044991 (+    26us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:22:51.045617 (+   626us) log.cc:844] Serialized 52 byte log entry
0709 12:22:51.131091 (+ 85474us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:22:51.183475 (+ 52384us) log.cc:844] Serialized 14 byte log entry
0709 12:22:51.238186 (+ 54711us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:51.238259 (+    73us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:51.268381 (+ 30122us) raft_consensus.cc:1685] finished
0709 12:22:51.268401 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:51.269016 (+   615us) inbound_call.cc:177] Queueing success response
0709 12:22:51.316981 (+ 47965us) spinlock_profiling.cc:243] Waited 2.12 ms on lock 0x7b18003e3f80. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b47533522 00007f7b475357c1 00007f7b47535779 00007f7b47535709 00007f7b475356d1 00007f7b475349cd 0000000000388634 0000000000388479 00007f7b4751683a
Metrics: {"mutex_wait_us":12872,"prepare.queue_time_us":52933,"prepare.run_cpu_time_us":70,"prepare.run_wall_time_us":6106,"raft.queue_time_us":34221,"raft.run_cpu_time_us":90,"raft.run_wall_time_us":12038,"spinlock_wait_cycles":9682432,"thread_start_us":126542,"threads_started":2,"wal-append.queue_time_us":181913}
I20260709 12:22:51.332372 17684 raft_consensus.cc:2468] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 13.
I20260709 12:22:51.313371 22619 leader_election.cc:290] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:22:51.458707 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 100491us
W20260709 12:22:51.327499 22662 log.cc:927] Time spent T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 1.087s	user 0.000s	sys 0.007s
I20260709 12:22:51.312179 22695 raft_consensus.cc:493] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:51.454463 17682 rpcz_store.cc:276] 0709 12:22:49.583341 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:49.590716 (+  7375us) service_pool.cc:224] Handling call
0709 12:22:49.591074 (+   358us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:49.591417 (+   343us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:49.591453 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:49.615997 (+ 24544us) log.cc:844] Serialized 26 byte log entry
0709 12:22:49.929281 (+313284us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:22:49.929319 (+    38us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:49.929391 (+    72us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:51.051733 (+1122342us) spinlock_profiling.cc:243] Waited 3.14 ms on lock 0x7b4400022b4c. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b47459c85 00007f7b4745a6a2 00007f7b4a1cc1db 00007f7b4a1c7aaf 00007f7b4a1bb86e 00007f7b4a1babb6 00007f7b4a1ba7d8 00007f7b4a1c4221 00007f7b4a1c41d9 00007f7b4a1c4169
0709 12:22:51.281087 (+229354us) raft_consensus.cc:1685] finished
0709 12:22:51.281121 (+    34us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:51.375583 (+ 94462us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":7964,"raft.run_cpu_time_us":85,"raft.run_wall_time_us":85,"spinlock_wait_cycles":7212288,"thread_start_us":312903,"threads_started":1,"wal-append.queue_time_us":1272434}
I20260709 12:22:51.267976 22700 raft_consensus.cc:3060] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:22:51.467399 17803 leader_election.cc:336] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Timed out: RequestConsensusVote RPC to 127.16.29.68:42577 timed out after 2.517s (SENT)
I20260709 12:22:51.462782 17685 raft_consensus.cc:2393] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 11: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:22:51.314780 22218 raft_consensus.cc:2749] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:22:51.270116 22615 raft_consensus.cc:3060] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:51.317811 22653 raft_consensus.cc:515] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:51.322041 22376 leader_election.cc:290] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 10 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:51.330394 22383 leader_election.cc:290] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:22:51.458491 22715 log.cc:927] Time spent T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.199s	user 0.007s	sys 0.001s
I20260709 12:22:51.260891 22360 raft_consensus.cc:515] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:51.298892 22488 consensus_queue.cc:1048] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.181s
I20260709 12:22:51.310667 22652 raft_consensus.cc:515] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:51.313541 22192 consensus_queue.cc:1048] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 7, Time since last communication: 0.149s
I20260709 12:22:51.476871 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:51.467559 17652 leader_election.cc:304] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:22:51.321197 22476 consensus_queue.cc:237] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.7, Last appended by leader: 7, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:22:51.590267 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 122764us
I20260709 12:22:51.267591 22557 consensus_queue.cc:1048] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.200s
I20260709 12:22:51.595111 17729 leader_election.cc:304] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
W20260709 12:22:51.617910 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:51.466984 22709 raft_consensus.cc:2804] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:22:51.477445 22695 raft_consensus.cc:3060] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:51.467281 22661 raft_consensus.cc:3060] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:51.654568 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:51.655162 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:51.655373 16501 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:22:51.658739 17687 raft_consensus.cc:2468] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 9.
W20260709 12:22:51.656599 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 61626.2us
I20260709 12:22:51.660868 17729 leader_election.cc:304] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
W20260709 12:22:51.705086 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 47280.9us
W20260709 12:22:51.716233 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 50062.5us
I20260709 12:22:51.718286 17833 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35040 (request call id 2497) took 1980 ms. Trace:
I20260709 12:22:51.718868 17833 rpcz_store.cc:276] 0709 12:22:49.737685 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:49.741197 (+  3512us) service_pool.cc:224] Handling call
0709 12:22:49.741512 (+   315us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:49.742022 (+   510us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 12:22:49.742051 (+    29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:49.742562 (+   511us) log.cc:844] Serialized 26 byte log entry
0709 12:22:49.903113 (+160551us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:22:49.903172 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:49.903267 (+    95us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:51.326919 (+1423652us) spinlock_profiling.cc:243] Waited 69.2 ms on lock 0x7b24001cf4a8. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b473b99c4 00007f7b473b7acb 00007f7b47572dc9 00007f7b47572a55 00007f7b4757209d 00007f7b4a1a410e 00007f7b4a1bb88c 00007f7b4a1babb6 00007f7b4a1ba7d8 00007f7b4a1c4221
0709 12:22:51.717872 (+390953us) raft_consensus.cc:1685] finished
0709 12:22:51.717895 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:51.718178 (+   283us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1055,"raft.run_cpu_time_us":78,"raft.run_wall_time_us":77,"spinlock_wait_cycles":159048960,"thread_start_us":157765,"threads_started":1,"wal-append.queue_time_us":308780}
I20260709 12:22:51.465234 22542 raft_consensus.cc:515] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:51.618772 22517 raft_consensus.cc:2804] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:22:51.462574 22607 raft_consensus.cc:515] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:22:51.731405 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 267341us
I20260709 12:22:51.761401 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:51.702991 22474 raft_consensus.cc:2749] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 12:22:51.766400 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:51.769838 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:51.770361 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:51.770706 17682 raft_consensus.cc:2468] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 13.
W20260709 12:22:51.770026 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 493456us
W20260709 12:22:51.782552 17803 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:51.715219 22709 raft_consensus.cc:697] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 13 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:22:51.795342 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 15 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:51.800287 17757 raft_consensus.cc:2410] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for term 15 because replica has last-logged OpId of term: 12 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 6.
I20260709 12:22:51.724350 22551 consensus_queue.cc:1048] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.075s
I20260709 12:22:51.779425 22517 raft_consensus.cc:697] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 10 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:22:51.819193 22729 raft_consensus.cc:2749] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20260709 12:22:51.812950 22714 log.cc:927] Time spent T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.529s	user 0.001s	sys 0.000s
W20260709 12:22:51.825557 17803 leader_election.cc:336] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Timed out: RequestConsensusVote RPC to 127.16.29.68:42577 timed out after 2.548s (SENT)
I20260709 12:22:51.828949 17803 leader_election.cc:304] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:51.829447 22192 raft_consensus.cc:493] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:51.830252 18333 leader_election.cc:304] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:22:51.834975 22192 raft_consensus.cc:515] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:51.836355 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:51.838858 17761 raft_consensus.cc:2468] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 11.
I20260709 12:22:51.839846 22728 raft_consensus.cc:493] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:51.840431 22728 raft_consensus.cc:515] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:51.841737 22192 leader_election.cc:290] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:51.842451 22728 leader_election.cc:290] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:51.833103 22474 raft_consensus.cc:493] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:51.843187 22474 raft_consensus.cc:515] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:51.833173 22488 raft_consensus.cc:493] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:51.849502 22488 raft_consensus.cc:515] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:51.852685 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:51.812204 22529 leader_election.cc:290] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:22:51.855548 17836 consensus_meta.cc:287] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.666s	user 0.010s	sys 0.000s
I20260709 12:22:51.856825 17836 raft_consensus.cc:2468] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 13.
I20260709 12:22:51.851653 22488 leader_election.cc:290] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:51.861347 22542 leader_election.cc:290] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:51.861824 22517 consensus_queue.cc:237] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:51.480588 22673 consensus_queue.cc:260] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:51.864209 22728 raft_consensus.cc:2749] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:22:51.872900 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:51.873821 18362 raft_consensus.cc:2468] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 9.
I20260709 12:22:51.874802 17652 leader_election.cc:304] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:22:51.875049 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 14 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:51.833772 22722 raft_consensus.cc:2749] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20260709 12:22:51.868191 17804 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:51.877367 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:51.878856 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:51.879419 18364 raft_consensus.cc:3060] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:22:51.879491 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 53985.1us
I20260709 12:22:51.828032 22729 raft_consensus.cc:493] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
W20260709 12:22:51.838689 22682 consensus_meta.cc:287] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.830s	user 0.017s	sys 0.001s
I20260709 12:22:51.876240 17761 raft_consensus.cc:2393] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 14: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
W20260709 12:22:51.876364 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 58549.6us
I20260709 12:22:51.884305 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:51.885125 18371 raft_consensus.cc:2468] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 13.
I20260709 12:22:51.880899 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:51.882589 17652 leader_election.cc:304] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:51.883168 17758 raft_consensus.cc:1240] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 8. Current term is 9. Ops: []
I20260709 12:22:51.883659 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
W20260709 12:22:51.867118 22515 consensus_meta.cc:287] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.682s	user 0.007s	sys 0.000s
I20260709 12:22:51.886759 18362 raft_consensus.cc:3060] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:51.886505 18374 raft_consensus.cc:2468] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 10.
I20260709 12:22:51.892015 17652 leader_election.cc:304] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:51.894013 18374 raft_consensus.cc:1275] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 11 index: 11. Preceding OpId from leader: term: 13 index: 12. (index mismatch)
I20260709 12:22:51.884527 22488 raft_consensus.cc:2804] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:22:51.905915 18364 raft_consensus.cc:2468] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 13.
I20260709 12:22:51.912940 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:51.891427 22515 raft_consensus.cc:2749] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
W20260709 12:22:51.903190 22720 log.cc:927] Time spent T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.387s	user 0.000s	sys 0.001s
I20260709 12:22:51.867692 22695 raft_consensus.cc:515] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:51.900427 22660 raft_consensus.cc:2804] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:22:51.922035 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:51.870395 22709 consensus_queue.cc:237] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:51.936120 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:51.922046 18364 raft_consensus.cc:3060] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:51.924437 18333 leader_election.cc:304] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:22:51.936390 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:51.936774 18361 raft_consensus.cc:2468] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 9.
I20260709 12:22:51.937072 18367 raft_consensus.cc:3060] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:51.938194 18333 leader_election.cc:304] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:22:51.939383 17729 leader_election.cc:304] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:51.897387 22652 leader_election.cc:290] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 14 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:51.940236 22728 raft_consensus.cc:2749] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 12:22:51.940524 22652 raft_consensus.cc:2804] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:22:51.940910 22652 raft_consensus.cc:493] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:51.941287 22652 raft_consensus.cc:3060] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:51.940924 22728 raft_consensus.cc:2749] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:22:51.947903 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:51.948541 17761 raft_consensus.cc:2376] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 14. Re-sending same reply.
I20260709 12:22:51.949724 22722 consensus_queue.cc:1048] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 12:22:51.949877 17801 leader_election.cc:304] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:51.951081 22722 consensus_queue.cc:1048] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:22:51.951265 17801 leader_election.cc:304] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:51.952033 22682 consensus_queue.cc:1059] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 10.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 12:22:51.953037 22682 raft_consensus.cc:2804] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:22:51.953383 22682 raft_consensus.cc:493] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:51.953939 18333 leader_election.cc:304] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:51.960817 18362 raft_consensus.cc:2468] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 13.
I20260709 12:22:51.970038 17729 leader_election.cc:304] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:51.972791 17729 leader_election.cc:304] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:51.913892 22408 consensus_queue.cc:1059] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.042s
I20260709 12:22:51.978471 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:51.979410 17764 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 11.
I20260709 12:22:51.981178 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:51.953685 22682 raft_consensus.cc:3060] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:51.889138 22729 raft_consensus.cc:515] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:51.985431 17830 raft_consensus.cc:1240] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 14. Current term is 15. Ops: [13.7-14.8]
I20260709 12:22:51.986157 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:51.986953 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:51.992053 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 15 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:51.988200 18364 raft_consensus.cc:2468] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 10.
W20260709 12:22:52.000188 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 56333.8us
I20260709 12:22:51.955464 22660 raft_consensus.cc:697] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 11 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:22:51.978617 22536 raft_consensus.cc:2804] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:22:51.988997 22694 raft_consensus.cc:2804] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:22:51.983533 22672 raft_consensus.cc:3055] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:22:52.025547 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:52.033566 17836 raft_consensus.cc:2468] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 9.
I20260709 12:22:52.039952 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 14 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:52.040596 17833 raft_consensus.cc:2468] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 13.
W20260709 12:22:52.040927 22305 consensus_meta.cc:287] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.737s	user 0.011s	sys 0.000s
W20260709 12:22:52.041360 22700 consensus_meta.cc:287] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.556s	user 0.009s	sys 0.001s
I20260709 12:22:52.041718 22660 consensus_queue.cc:237] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.9, Last appended by leader: 9, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:52.025002 22536 raft_consensus.cc:493] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:52.044415 22536 raft_consensus.cc:3060] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:52.048561 17755 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38776 (request call id 2270) took 1817 ms. Trace:
I20260709 12:22:52.050263 17652 leader_election.cc:304] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:52.052223 17652 leader_election.cc:304] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:52.054376 17652 leader_election.cc:304] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:22:51.996125 22360 leader_election.cc:290] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:52.008569 22734 raft_consensus.cc:2804] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:22:52.060453 17802 leader_election.cc:304] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:52.049091 17755 rpcz_store.cc:276] 0709 12:22:50.231075 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:50.231237 (+   162us) service_pool.cc:224] Handling call
0709 12:22:50.260790 (+ 29553us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:50.261297 (+   507us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 12:22:50.303327 (+ 42030us) log.cc:844] Serialized 14 byte log entry
0709 12:22:50.969191 (+665864us) spinlock_profiling.cc:243] Waited 37.2 ms on lock 0x7b1800274f60. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b47515e07 00007f7b503709a9 00007f7b4753269b 00007f7b4753093f 00007f7b47532c02 00007f7b4a1ba20b 00007f7b4a1bfd23 00007f7b4a1c023a 00007f7b4a62aa2a
0709 12:22:51.065676 (+ 96485us) spinlock_profiling.cc:243] Waited 51.3 ms on lock 0x7b1800274f60. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b475333f3 00007f7b475357c1 00007f7b47535779 00007f7b47535709 00007f7b475356d1 00007f7b475349cd 0000000000388634 0000000000388479 00007f7b4751683a
0709 12:22:51.131857 (+ 66181us) log.cc:844] Serialized 14 byte log entry
0709 12:22:51.132091 (+   234us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:51.144459 (+ 12368us) log.cc:844] Serialized 26 byte log entry
0709 12:22:51.146180 (+  1721us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:22:51.180950 (+ 34770us) log.cc:844] Serialized 14 byte log entry
0709 12:22:51.184750 (+  3800us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:51.184834 (+    84us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:51.315067 (+130233us) spinlock_profiling.cc:243] Waited 33.2 ms on lock 0x7b1800274f60. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b47533522 00007f7b475357c1 00007f7b47535779 00007f7b47535709 00007f7b475356d1 00007f7b475349cd 0000000000388634 0000000000388479 00007f7b4751683a
0709 12:22:52.048190 (+733123us) raft_consensus.cc:1685] finished
0709 12:22:52.048211 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:52.048471 (+   260us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":20685,"prepare.queue_time_us":856945,"prepare.run_cpu_time_us":123,"prepare.run_wall_time_us":3222,"raft.queue_time_us":58658,"raft.run_cpu_time_us":471,"raft.run_wall_time_us":21121,"spinlock_wait_cycles":280121600,"thread_start_us":152871,"threads_started":1,"wal-append.queue_time_us":871883}
I20260709 12:22:52.071962 17689 raft_consensus.cc:1240] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 12. Current term is 13. Ops: []
I20260709 12:22:52.033175 22672 raft_consensus.cc:740] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 10 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
W20260709 12:22:52.074899 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 49179.2us
I20260709 12:22:52.075085 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:52.062263 22488 raft_consensus.cc:697] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 10 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:22:52.036984 22722 raft_consensus.cc:2804] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:22:52.037799 22626 raft_consensus.cc:3055] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:22:52.088467 22722 raft_consensus.cc:697] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 14 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:22:52.088795 22626 raft_consensus.cc:740] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:22:52.088915 22488 consensus_queue.cc:237] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:52.089676 22722 consensus_queue.cc:237] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 13.12, Last appended by leader: 12, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:52.089955 22626 consensus_queue.cc:260] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 14.8, Last appended by leader: 8, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:52.091420 22626 raft_consensus.cc:3060] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:22:52.056335 22694 raft_consensus.cc:697] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 13 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:22:52.097582 22694 consensus_queue.cc:237] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:52.033036 22408 raft_consensus.cc:493] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:22:52.100239 22408 raft_consensus.cc:515] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:52.041774 22700 raft_consensus.cc:515] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:52.102679 22408 leader_election.cc:290] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:22:52.105885 22491 consensus_meta.cc:287] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.862s	user 0.010s	sys 0.000s
I20260709 12:22:52.106434 22491 raft_consensus.cc:515] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:51.993510 22731 consensus_queue.cc:1048] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.101s
I20260709 12:22:52.122532 18367 raft_consensus.cc:2468] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 14.
W20260709 12:22:52.123034 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:52.127476 18333 leader_election.cc:304] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:52.111508 22734 raft_consensus.cc:697] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 8 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:22:52.102725 22700 leader_election.cc:290] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 11 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:52.041391 22305 raft_consensus.cc:515] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:52.121354 22607 leader_election.cc:290] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:52.127267 22515 raft_consensus.cc:2804] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:22:52.184849 17757 raft_consensus.cc:1240] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 9. Current term is 10. Ops: [9.6-9.6]
I20260709 12:22:52.185972 22305 leader_election.cc:290] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 10 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:52.192298 22529 raft_consensus.cc:493] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:52.192878 22529 raft_consensus.cc:515] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:52.172452 22737 raft_consensus.cc:2764] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Leader election decision vote started in defunct term 9: won
I20260709 12:22:52.195113 22529 leader_election.cc:290] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:52.066218 17832 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "0f940f29438f475294357324452abe76"
dest_uuid: "e063541542a94969898868bf6eb5ce1b"
mode: GRACEFUL
new_leader_uuid: "334a4e1f59944544b3b3ff36616b57da"
 from {username='slave'} at 127.0.0.1:59820
I20260709 12:22:52.200836 17832 raft_consensus.cc:606] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Received request to transfer leadership to TS 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:52.201179 17832 raft_consensus.cc:614] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260709 12:22:52.232597 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 111934us
I20260709 12:22:52.175514 22515 raft_consensus.cc:493] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:52.232069 22360 raft_consensus.cc:493] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:52.235991 22305 raft_consensus.cc:493] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:52.237967 22376 raft_consensus.cc:2749] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:22:52.239173 22515 raft_consensus.cc:3060] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:22:52.292526 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 232184us
I20260709 12:22:52.293612 17802 leader_election.cc:304] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:52.241672 22684 raft_consensus.cc:2749] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:22:52.309827 18333 leader_election.cc:304] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
W20260709 12:22:52.311105 17655 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:52.272318 22725 raft_consensus.cc:2804] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Leader pre-election won for term 12
W20260709 12:22:52.335018 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 93869.8us
W20260709 12:22:52.329246 17803 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:52.338465 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:52.342446 17837 raft_consensus.cc:1275] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 11 index: 6. Preceding OpId from leader: term: 15 index: 8. (index mismatch)
I20260709 12:22:52.344467 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:52.345221 17837 raft_consensus.cc:2468] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 8.
W20260709 12:22:52.347415 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 293085us
I20260709 12:22:52.348807 17652 leader_election.cc:304] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:52.355852 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:52.271735 22360 raft_consensus.cc:515] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:52.337604 22737 raft_consensus.cc:2764] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Leader election decision vote started in defunct term 12: won
I20260709 12:22:52.366803 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:52.274930 22305 raft_consensus.cc:515] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:52.335318 22749 raft_consensus.cc:3060] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:52.356956 22728 raft_consensus.cc:493] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:52.363131 22729 leader_election.cc:290] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:52.337466 22722 consensus_queue.cc:1059] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 12.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.052s
I20260709 12:22:52.401575 17764 raft_consensus.cc:1275] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 14 index: 8. (index mismatch)
I20260709 12:22:52.366353 22626 raft_consensus.cc:2804] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:22:52.482455 17764 raft_consensus.cc:1240] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 12. Current term is 13. Ops: []
I20260709 12:22:52.402706 22738 raft_consensus.cc:2749] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:22:52.400184 22764 raft_consensus.cc:493] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:52.544521 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 14 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:52.370711 22652 raft_consensus.cc:515] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:22:52.556669 22536 consensus_meta.cc:287] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.512s	user 0.011s	sys 0.000s
I20260709 12:22:52.601567 17837 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3"
dest_uuid: "e063541542a94969898868bf6eb5ce1b"
mode: GRACEFUL
new_leader_uuid: "0da169ff82674dc1947055ce43de68a3"
 from {username='slave'} at 127.0.0.1:59820
I20260709 12:22:52.565701 22360 leader_election.cc:290] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:22:52.567364 22751 log.cc:927] Time spent T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.177s	user 0.000s	sys 0.000s
I20260709 12:22:52.634213 22114 raft_consensus.cc:493] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:52.638381 17689 raft_consensus.cc:1275] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 10 index: 7. Preceding OpId from leader: term: 11 index: 8. (index mismatch)
W20260709 12:22:52.570263 22761 log.cc:927] Time spent T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.175s	user 0.000s	sys 0.000s
W20260709 12:22:52.571244 22744 log.cc:927] Time spent T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.326s	user 0.000s	sys 0.001s
W20260709 12:22:52.576581 22747 log.cc:927] Time spent T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.239s	user 0.001s	sys 0.000s
I20260709 12:22:52.406981 22626 raft_consensus.cc:493] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:52.577841 17764 raft_consensus.cc:2393] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 14: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:22:52.337484 22734 consensus_queue.cc:237] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:52.600505 22536 raft_consensus.cc:515] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:52.410554 22408 raft_consensus.cc:2749] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:22:52.617605 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:52.641463 22737 raft_consensus.cc:493] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:22:52.406009 22728 raft_consensus.cc:515] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:52.666778 17756 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38776 (request call id 2264) took 2671 ms. Trace:
W20260709 12:22:52.575706 22661 consensus_meta.cc:287] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.872s	user 0.010s	sys 0.000s
I20260709 12:22:52.670967 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:52.672266 17756 rpcz_store.cc:276] 0709 12:22:49.983729 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:49.985589 (+  1860us) service_pool.cc:224] Handling call
0709 12:22:49.986041 (+   452us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:50.974378 (+988337us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:52.654695 (+1680317us) spinlock_profiling.cc:243] Waited 968 ms on lock 0x7b5400217854. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:22:52.654931 (+   236us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2226552064}
I20260709 12:22:52.673590 17755 raft_consensus.cc:2368] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 8. Current term is 9.
I20260709 12:22:52.681202 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:52.682507 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:52.683086 17764 raft_consensus.cc:2393] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 11: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:22:52.684602 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:52.677791 22737 raft_consensus.cc:515] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:52.686203 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:52.687206 22737 leader_election.cc:290] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:52.687408 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:52.696838 17755 raft_consensus.cc:2393] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 14: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:22:52.669782 22626 raft_consensus.cc:3060] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:52.700246 17764 raft_consensus.cc:1240] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 8. Current term is 9. Ops: []
W20260709 12:22:52.690052 22759 log.cc:927] Time spent T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.285s	user 0.001s	sys 0.000s
I20260709 12:22:52.713264 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:52.618325 22764 raft_consensus.cc:515] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:52.655675 22114 raft_consensus.cc:515] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:52.715569 17756 raft_consensus.cc:2393] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 11: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:22:52.717623 17764 raft_consensus.cc:1240] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 9. Current term is 10. Ops: [9.9-9.9]
I20260709 12:22:52.693130 22536 leader_election.cc:290] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 14 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:52.666487 22676 consensus_queue.cc:1048] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.039s
W20260709 12:22:52.718943 18333 connection.cc:256] RPC call timeout handler was delayed by -3.6843531270001222s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1726s for a response
W20260709 12:22:52.721524 18333 connection.cc:256] RPC call timeout handler was delayed by -3.4134531269996842s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20270000000000002s for a response
I20260709 12:22:52.723140 22727 raft_consensus.cc:493] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:52.723737 22727 raft_consensus.cc:515] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:22:52.725703 22756 log.cc:927] Time spent T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.059s	user 0.000s	sys 0.000s
I20260709 12:22:52.725984 22727 leader_election.cc:290] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:52.726828 17687 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37396 (request call id 2368) took 1283 ms. Trace:
I20260709 12:22:52.727205 17687 rpcz_store.cc:276] 0709 12:22:51.443352 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:51.775286 (+331934us) service_pool.cc:224] Handling call
0709 12:22:51.788624 (+ 13338us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:51.789189 (+   565us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 12:22:52.009295 (+220106us) log.cc:844] Serialized 14 byte log entry
0709 12:22:52.240252 (+230957us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:52.240700 (+   448us) log.cc:844] Serialized 26 byte log entry
0709 12:22:52.240853 (+   153us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:22:52.240889 (+    36us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:52.240967 (+    78us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:52.726624 (+485657us) raft_consensus.cc:1685] finished
0709 12:22:52.726644 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:52.726775 (+   131us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":74264,"prepare.run_cpu_time_us":26,"prepare.run_wall_time_us":27,"raft.queue_time_us":226,"raft.run_cpu_time_us":222,"raft.run_wall_time_us":2697,"thread_start_us":413540,"threads_started":2,"wal-append.queue_time_us":507350}
I20260709 12:22:52.729168 17682 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37396 (request call id 2372) took 1284 ms. Trace:
W20260709 12:22:52.729976 17803 leader_election.cc:336] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 14 pre-election: RPC error from VoteRequest() call to peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Timed out: RequestConsensusVote RPC to 127.16.29.68:42577 timed out after 2.062s (SENT)
I20260709 12:22:52.730602 17803 leader_election.cc:304] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:52.734855 17760 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38776 (request call id 2272) took 2431 ms. Trace:
W20260709 12:22:52.736390 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:52.729537 17682 rpcz_store.cc:276] 0709 12:22:51.445156 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:51.771460 (+326304us) service_pool.cc:224] Handling call
0709 12:22:51.771757 (+   297us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:51.781962 (+ 10205us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 12:22:51.900743 (+118781us) log.cc:844] Serialized 14 byte log entry
0709 12:22:52.340445 (+439702us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:52.342163 (+  1718us) log.cc:844] Serialized 26 byte log entry
0709 12:22:52.347256 (+  5093us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:22:52.347330 (+    74us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:52.347428 (+    98us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:52.728961 (+381533us) raft_consensus.cc:1685] finished
0709 12:22:52.728980 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:52.729110 (+   130us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":19392,"prepare.queue_time_us":81255,"prepare.run_cpu_time_us":62,"prepare.run_wall_time_us":62,"raft.queue_time_us":105967,"raft.run_cpu_time_us":368,"raft.run_wall_time_us":29643,"thread_start_us":510355,"threads_started":3,"wal-append.queue_time_us":252558}
I20260709 12:22:52.743031 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:52.744258 17755 raft_consensus.cc:2393] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 9: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:22:52.747120 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:52.746965 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 12 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:52.753652 22764 leader_election.cc:290] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:52.746304 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:52.719762 22652 leader_election.cc:290] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 10 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:22:52.725823 22682 consensus_meta.cc:287] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.719s	user 0.009s	sys 0.000s
I20260709 12:22:52.756659 17833 raft_consensus.cc:1275] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 8 index: 7. Preceding OpId from leader: term: 13 index: 10. (index mismatch)
W20260709 12:22:52.767411 22615 consensus_meta.cc:287] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 1.178s	user 0.010s	sys 0.000s
I20260709 12:22:52.767647 22728 leader_election.cc:290] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:52.755851 22749 raft_consensus.cc:2749] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:22:52.735431 17760 rpcz_store.cc:276] 0709 12:22:50.303427 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:50.407580 (+104153us) service_pool.cc:224] Handling call
0709 12:22:50.436322 (+ 28742us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:50.621916 (+185594us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:50.621961 (+    45us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:50.622565 (+   604us) log.cc:844] Serialized 26 byte log entry
0709 12:22:51.953574 (+1331009us) spinlock_profiling.cc:243] Waited 16.1 ms on lock 0x7b1800321820. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b47515e07 00007f7b503709a9 00007f7b4753269b 00007f7b4753093f 00007f7b47532c02 00007f7b4a1ba20b 00007f7b4a1bfd23 00007f7b4a1c00d1 00007f7b4a5dba0f
0709 12:22:52.237010 (+283436us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:22:52.237052 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:52.237125 (+    73us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:52.718834 (+481709us) raft_consensus.cc:1685] finished
0709 12:22:52.718864 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:52.734762 (+ 15898us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":57403,"raft.queue_time_us":182220,"raft.run_cpu_time_us":120,"raft.run_wall_time_us":6063,"spinlock_wait_cycles":37048576,"thread_start_us":393372,"threads_started":2,"wal-append.queue_time_us":1704572}
I20260709 12:22:52.770238 17683 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37402 (request call id 2707) took 1740 ms. Trace:
I20260709 12:22:52.770650 17683 rpcz_store.cc:276] 0709 12:22:51.029407 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:51.072004 (+ 42597us) service_pool.cc:224] Handling call
0709 12:22:51.072351 (+   347us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:51.072716 (+   365us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:51.072743 (+    27us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:52.035778 (+963035us) spinlock_profiling.cc:243] Waited 873 us on lock 0x7b2c00000c88. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b48f19a11 00007f7b48f29372 00007f7b4a65ba01 00007f7b4a62043a 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2
0709 12:22:52.036282 (+   504us) log.cc:844] Serialized 26 byte log entry
0709 12:22:52.267460 (+231178us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:22:52.267522 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:52.267632 (+   110us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:52.769943 (+502311us) raft_consensus.cc:1685] finished
0709 12:22:52.769962 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:52.770180 (+   218us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":345,"raft.run_cpu_time_us":121,"raft.run_wall_time_us":121,"spinlock_wait_cycles":2008960,"thread_start_us":158263,"threads_started":1,"wal-append.queue_time_us":701591}
I20260709 12:22:52.773205 22775 raft_consensus.cc:2749] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:22:52.774549 22114 leader_election.cc:290] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:22:52.774807 22721 log.cc:927] Time spent T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 1.322s	user 0.006s	sys 0.009s
I20260709 12:22:52.776134 17763 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38776 (request call id 2276) took 1889 ms. Trace:
I20260709 12:22:52.760528 17830 raft_consensus.cc:3060] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Advancing to term 14
W20260709 12:22:52.722209 18333 connection.cc:256] RPC call timeout handler was delayed by -1.1535531270001229s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.45380000000000004s for a response
I20260709 12:22:52.777654 17832 raft_consensus.cc:2410] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for term 12 because replica has last-logged OpId of term: 10 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 7.
I20260709 12:22:52.786294 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
W20260709 12:22:52.787569 17803 leader_election.cc:336] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 15 pre-election: RPC error from VoteRequest() call to peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Timed out: RequestConsensusVote RPC to 127.16.29.68:42577 timed out after 3.290s (SENT)
I20260709 12:22:52.788039 17803 leader_election.cc:304] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:52.793378 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:52.794059 17833 raft_consensus.cc:3060] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:52.790376 17727 leader_election.cc:304] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:52.776497 17763 rpcz_store.cc:276] 0709 12:22:50.886837 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:50.992923 (+106086us) service_pool.cc:224] Handling call
0709 12:22:50.993299 (+   376us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:22:51.015786 (+ 22487us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:51.015816 (+    30us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:22:51.072824 (+ 57008us) log.cc:844] Serialized 52 byte log entry
0709 12:22:51.217484 (+144660us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:22:51.392968 (+175484us) log.cc:844] Serialized 14 byte log entry
0709 12:22:51.398636 (+  5668us) log.cc:844] Serialized 14 byte log entry
0709 12:22:51.442324 (+ 43688us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:51.442398 (+    74us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:52.770566 (+1328168us) spinlock_profiling.cc:243] Waited 17.4 ms on lock 0x7b18003d1140. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b475333f3 00007f7b475357c1 00007f7b47535779 00007f7b47535709 00007f7b475356d1 00007f7b475349cd 0000000000388634 0000000000388479 00007f7b4751683a
0709 12:22:52.774099 (+  3533us) spinlock_profiling.cc:243] Waited 2.4 ms on lock 0x7b4400022b4c. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b47459c85 00007f7b4745a6a2 00007f7b4a1cc1db 00007f7b4a1c7aaf 00007f7b4a1bb86e 00007f7b4a1babb6 00007f7b4a1ba7d8 00007f7b4a1c4221 00007f7b4a1c41d9 00007f7b4a1c4169
0709 12:22:52.775829 (+  1730us) raft_consensus.cc:1685] finished
0709 12:22:52.775850 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:52.776050 (+   200us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":20028,"prepare.queue_time_us":1470080,"prepare.run_cpu_time_us":53,"prepare.run_wall_time_us":55,"raft.queue_time_us":64254,"raft.run_cpu_time_us":396,"raft.run_wall_time_us":33289,"spinlock_wait_cycles":45747072,"thread_start_us":295154,"threads_started":2,"wal-append.queue_time_us":226951}
I20260709 12:22:52.795418 22757 raft_consensus.cc:493] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:52.796165 22757 raft_consensus.cc:515] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:52.797044 22745 raft_consensus.cc:2804] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:22:52.797602 22745 raft_consensus.cc:493] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:52.797961 22745 raft_consensus.cc:3060] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:52.798002 22677 raft_consensus.cc:493] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:52.798524 22757 leader_election.cc:290] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:52.798609 22677 raft_consensus.cc:515] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:52.793551 22676 raft_consensus.cc:2749] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:22:52.801204 22677 leader_election.cc:290] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:52.801702 22691 raft_consensus.cc:493] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:52.802240 22691 raft_consensus.cc:515] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:52.804080 22695 leader_election.cc:290] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 13 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:52.804591 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 16 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:52.805187 22691 leader_election.cc:290] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:52.805447 17682 raft_consensus.cc:2410] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 16 because replica has last-logged OpId of term: 14 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 6.
W20260709 12:22:52.805743 17803 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:52.809845 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:52.810285 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:52.810614 17829 raft_consensus.cc:2393] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 13: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:22:52.814093 17833 raft_consensus.cc:2468] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 11.
I20260709 12:22:52.815414 17682 raft_consensus.cc:1240] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 12. Current term is 13. Ops: []
I20260709 12:22:52.817371 22360 consensus_queue.cc:1048] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:22:52.784729 22764 raft_consensus.cc:493] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:52.811172 22536 consensus_queue.cc:1048] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.008s
I20260709 12:22:52.818269 22764 raft_consensus.cc:515] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:52.820072 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:52.820596 22764 leader_election.cc:290] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:52.821188 22114 raft_consensus.cc:493] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:52.821641 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:52.821762 22114 raft_consensus.cc:515] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:52.822413 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 13 candidate_status { last_received { term: 11 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:52.822695 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:52.764858 22515 raft_consensus.cc:515] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:52.823601 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:52.825884 18333 leader_election.cc:304] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:52.828763 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:52.830209 18362 raft_consensus.cc:1275] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20260709 12:22:52.832068 18363 raft_consensus.cc:1240] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 13. Current term is 14. Ops: [13.10-13.10]
I20260709 12:22:52.835700 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:52.836968 17829 raft_consensus.cc:2393] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 11: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:22:52.845855 18371 raft_consensus.cc:1275] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 12 index: 8. Preceding OpId from leader: term: 14 index: 9. (index mismatch)
I20260709 12:22:52.846937 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:52.837919 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 12 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:52.849969 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:52.850762 17829 raft_consensus.cc:2468] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 13.
I20260709 12:22:52.847555 17727 leader_election.cc:304] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:52.854729 22691 consensus_queue.cc:1059] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 12.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:22:52.845412 22757 raft_consensus.cc:493] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:52.854781 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:52.857609 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:52.858294 17831 raft_consensus.cc:2468] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 9.
I20260709 12:22:52.860649 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:52.861209 17833 raft_consensus.cc:1240] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 12. Current term is 13. Ops: [10.8-12.10]
I20260709 12:22:52.862007 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 12 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:52.862836 18365 raft_consensus.cc:2410] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for term 12 because replica has last-logged OpId of term: 10 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 7.
I20260709 12:22:52.859082 22114 leader_election.cc:290] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:52.859185 22360 raft_consensus.cc:2804] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:22:52.864391 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:52.864549 22360 raft_consensus.cc:697] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 11 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:22:52.865816 18361 raft_consensus.cc:1275] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20260709 12:22:52.864969 22626 raft_consensus.cc:515] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:52.865916 22360 consensus_queue.cc:237] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.7, Last appended by leader: 7, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:52.852650 22625 raft_consensus.cc:2749] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:22:52.867868 22626 leader_election.cc:290] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 14 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:52.785597 22615 raft_consensus.cc:515] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:52.868952 22515 leader_election.cc:290] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 10 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:52.870182 22615 leader_election.cc:290] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 12 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:52.870875 18368 raft_consensus.cc:1240] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 11. Current term is 12. Ops: []
I20260709 12:22:52.765187 22682 raft_consensus.cc:515] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:52.873682 22682 leader_election.cc:290] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 14 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:52.855618 22757 raft_consensus.cc:515] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:52.860095 17829 raft_consensus.cc:1275] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20260709 12:22:52.877127 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:52.875793 17837 raft_consensus.cc:606] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Received request to transfer leadership to TS 0da169ff82674dc1947055ce43de68a3
I20260709 12:22:52.880246 17837 raft_consensus.cc:614] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:22:52.863612 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:52.880908 17838 raft_consensus.cc:2393] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 14: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:22:52.880909 18365 raft_consensus.cc:3060] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:52.881752 18333 leader_election.cc:304] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:22:52.882571 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 93080.3us
I20260709 12:22:52.882766 18367 raft_consensus.cc:1275] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 12 index: 10. (index mismatch)
I20260709 12:22:52.889698 17727 leader_election.cc:304] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:52.890730 22615 raft_consensus.cc:2749] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
W20260709 12:22:52.894987 22719 log.cc:927] Time spent T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 1.567s	user 0.005s	sys 0.018s
W20260709 12:22:52.897836 17762 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38786 (request call id 2970) took 2534 ms (client timeout 3153 ms). Trace:
W20260709 12:22:52.898382 17762 rpcz_store.cc:269] 0709 12:22:50.363319 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:50.513939 (+150620us) service_pool.cc:224] Handling call
0709 12:22:52.897588 (+2383649us) spinlock_profiling.cc:243] Waited 590 ms on lock 0x7b54003e4ed4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:22:52.897743 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1356114176}
I20260709 12:22:52.903821 17803 leader_election.cc:304] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:52.906432 17830 raft_consensus.cc:2468] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 14.
I20260709 12:22:52.907749 17727 leader_election.cc:304] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:52.904728 17835 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35040 (request call id 2514) took 2019 ms. Trace:
W20260709 12:22:52.919188 17655 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:52.911507 17835 rpcz_store.cc:276] 0709 12:22:50.885319 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:51.047077 (+161758us) service_pool.cc:224] Handling call
0709 12:22:51.047850 (+   773us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:51.048381 (+   531us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:51.048417 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:51.049045 (+   628us) log.cc:844] Serialized 26 byte log entry
0709 12:22:51.185055 (+136010us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:22:51.185084 (+    29us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:51.185167 (+    83us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:52.894407 (+1709240us) spinlock_profiling.cc:243] Waited 44.6 ms on lock 0x7b4400022b4c. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b47459c85 00007f7b4745a6a2 00007f7b4a1cc1db 00007f7b4a1c7aaf 00007f7b4a1bb86e 00007f7b4a1babb6 00007f7b4a1ba7d8 00007f7b4a1c4221 00007f7b4a1c41d9 00007f7b4a1c4169
0709 12:22:52.897281 (+  2874us) raft_consensus.cc:1685] finished
0709 12:22:52.897315 (+    34us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:52.904646 (+  7331us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":637,"raft.run_cpu_time_us":83,"raft.run_wall_time_us":84,"spinlock_wait_cycles":102585088,"thread_start_us":135584,"threads_started":1,"wal-append.queue_time_us":195230}
I20260709 12:22:52.902936 22652 raft_consensus.cc:2804] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:22:52.902527 22615 raft_consensus.cc:493] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:52.902644 22536 consensus_queue.cc:1048] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.032s
I20260709 12:22:52.933393 18365 raft_consensus.cc:2468] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 11.
I20260709 12:22:52.911063 22590 raft_consensus.cc:2955] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 11 LEADER]: Committing config change with OpId 8.5: config changed from index -1 to 5, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } }
I20260709 12:22:52.951741 22615 raft_consensus.cc:515] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:52.954419 22615 leader_election.cc:290] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:52.954613 18333 leader_election.cc:304] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:52.935156 22383 raft_consensus.cc:493] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:52.950460 22757 leader_election.cc:290] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:52.957710 17680 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37402 (request call id 2716) took 1699 ms. Trace:
I20260709 12:22:52.963443 17680 rpcz_store.cc:276] 0709 12:22:51.258180 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:51.258400 (+   220us) service_pool.cc:224] Handling call
0709 12:22:51.270548 (+ 12148us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:51.393385 (+122837us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:51.393468 (+    83us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:52.736470 (+1343002us) spinlock_profiling.cc:243] Waited 934 us on lock 0x7b1800328660. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b475012a6 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b48f19a11 00007f7b48f29372 00007f7b4a65ba01 00007f7b4a62043a
0709 12:22:52.736861 (+   391us) log.cc:844] Serialized 26 byte log entry
0709 12:22:52.858391 (+121530us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:22:52.858497 (+   106us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:52.858603 (+   106us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:52.957429 (+ 98826us) raft_consensus.cc:1685] finished
0709 12:22:52.957448 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:52.957653 (+   205us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":63872,"raft.queue_time_us":81811,"raft.run_cpu_time_us":139,"raft.run_wall_time_us":27029,"spinlock_wait_cycles":2844288,"thread_start_us":121125,"threads_started":1,"wal-append.queue_time_us":167810}
I20260709 12:22:52.919260 22693 raft_consensus.cc:2804] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Leader election won for term 14
W20260709 12:22:52.971756 17803 leader_election.cc:336] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Timed out: RequestConsensusVote RPC to 127.16.29.68:42577 timed out after 1.726s (SENT)
I20260709 12:22:52.972277 17803 leader_election.cc:304] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:22:52.973712 17803 leader_election.cc:336] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 14 pre-election: RPC error from VoteRequest() call to peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Timed out: RequestConsensusVote RPC to 127.16.29.68:42577 timed out after 1.851s (SENT)
I20260709 12:22:52.932657 22652 raft_consensus.cc:493] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:52.930471 22114 raft_consensus.cc:2749] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:22:52.981752 22652 raft_consensus.cc:3060] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:52.905189 22597 consensus_queue.cc:260] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 12.9, Last appended by leader: 9, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:52.932085 22764 consensus_queue.cc:1048] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.046s
I20260709 12:22:52.979022 22734 raft_consensus.cc:2749] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:22:52.979748 22536 raft_consensus.cc:493] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:52.992808 22536 raft_consensus.cc:515] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:52.988688 17757 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38776 (request call id 2280) took 1178 ms. Trace:
I20260709 12:22:52.999984 17757 rpcz_store.cc:276] 0709 12:22:51.810554 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:51.810763 (+   209us) service_pool.cc:224] Handling call
0709 12:22:51.821093 (+ 10330us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:52.185360 (+364267us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:52.988395 (+803035us) spinlock_profiling.cc:243] Waited 363 ms on lock 0x7b5400415ad4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:22:52.988620 (+   225us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":835870464}
W20260709 12:22:53.003310 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 48787.2us
I20260709 12:22:52.966264 22383 raft_consensus.cc:515] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:52.980610 22693 raft_consensus.cc:697] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 14 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:22:52.985198 22495 raft_consensus.cc:3055] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:22:53.009506 22495 raft_consensus.cc:740] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:22:53.010494 22495 consensus_queue.cc:260] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:53.011552 22495 raft_consensus.cc:3060] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:53.016207 22383 leader_election.cc:290] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:53.009538 18333 leader_election.cc:400] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 8 election: Vote denied by peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651) with higher term. Message: Invalid argument: T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 8. Current term is 9.
I20260709 12:22:53.023577 22597 raft_consensus.cc:3060] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:53.030821 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:53.032460 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:53.038774 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:53.039470 17830 raft_consensus.cc:3060] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:53.007187 22775 raft_consensus.cc:493] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:53.044267 22775 raft_consensus.cc:515] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:53.044991 22745 raft_consensus.cc:515] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:53.046911 22775 leader_election.cc:290] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:53.062311 22745 leader_election.cc:290] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 9 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:53.065665 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:53.066494 17833 raft_consensus.cc:3060] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:53.033707 22738 raft_consensus.cc:2749] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:22:53.068159 22305 leader_election.cc:290] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:53.069315 17684 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37402 (request call id 2710) took 1988 ms. Trace:
I20260709 12:22:53.075042 17684 rpcz_store.cc:276] 0709 12:22:51.080947 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:51.081240 (+   293us) service_pool.cc:224] Handling call
0709 12:22:53.069160 (+1987920us) spinlock_profiling.cc:243] Waited 220 ms on lock 0x7b54000c5654. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:22:53.069257 (+    97us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":506839936}
I20260709 12:22:53.069459 22536 leader_election.cc:290] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:53.077769 17833 raft_consensus.cc:2468] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 9.
I20260709 12:22:53.078948 18333 leader_election.cc:304] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:53.079243 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:53.086690 18374 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 1573) took 1168 ms. Trace:
I20260709 12:22:53.098042 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 14 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:53.099448 17833 raft_consensus.cc:2393] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 14: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:22:53.087258 18374 rpcz_store.cc:276] 0709 12:22:51.917911 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:51.918122 (+   211us) service_pool.cc:224] Handling call
0709 12:22:51.918555 (+   433us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:51.919006 (+   451us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:51.919047 (+    41us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:51.919664 (+   617us) log.cc:844] Serialized 26 byte log entry
0709 12:22:52.904169 (+984505us) spinlock_profiling.cc:243] Waited 28.9 ms on lock 0x7b18003f7f60. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b47515e07 00007f7b503709a9 00007f7b4753269b 00007f7b4753093f 00007f7b47532c02 00007f7b4a1ba20b 00007f7b4a1bfd23 00007f7b4a1c00d1 00007f7b4a5dba0f
0709 12:22:52.975347 (+ 71178us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:22:52.975389 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:52.975458 (+    69us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:53.086343 (+110885us) raft_consensus.cc:1685] finished
0709 12:22:53.086366 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:53.086612 (+   246us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1779,"raft.run_cpu_time_us":314,"raft.run_wall_time_us":10927,"spinlock_wait_cycles":66465024,"thread_start_us":71003,"threads_started":1,"wal-append.queue_time_us":1105996}
I20260709 12:22:53.126013 17830 raft_consensus.cc:2468] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 10.
I20260709 12:22:53.128089 17652 leader_election.cc:304] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:22:53.106629 22734 raft_consensus.cc:2749] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
W20260709 12:22:53.132213 22726 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.514s	user 0.000s	sys 0.010s
I20260709 12:22:53.136315 22684 raft_consensus.cc:2804] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:22:53.184403 17801 leader_election.cc:304] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:22:53.166369 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 122269us
I20260709 12:22:53.107017 22693 consensus_queue.cc:237] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 12.7, Last appended by leader: 7, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:53.176430 22626 consensus_queue.cc:1059] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
W20260709 12:22:53.177232 17803 leader_election.cc:336] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Timed out: RequestConsensusVote RPC to 127.16.29.68:42577 timed out after 2.040s (SENT)
W20260709 12:22:53.178368 22689 log.cc:927] Time spent T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 1.177s	user 0.005s	sys 0.011s
W20260709 12:22:53.180337 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:53.185835 18333 leader_election.cc:304] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:53.186676 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:53.200161 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:53.203094 17756 raft_consensus.cc:2468] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 14.
I20260709 12:22:53.203539 17762 raft_consensus.cc:2468] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 11.
I20260709 12:22:53.185667 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 14 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:53.200974 17763 raft_consensus.cc:1240] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 8. Current term is 9. Ops: []
I20260709 12:22:53.199765 17758 pending_rounds.cc:85] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb: Aborting all ops after (but not including) 9
I20260709 12:22:53.208199 17758 pending_rounds.cc:107] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb: Aborting uncommitted NO_OP operation due to leader change: 10.10
I20260709 12:22:53.208542 17758 raft_consensus.cc:2889] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:53.210294 17759 raft_consensus.cc:1275] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20260709 12:22:53.211865 22626 raft_consensus.cc:3055] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:22:53.214064 22626 raft_consensus.cc:740] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
W20260709 12:22:53.216384 17803 leader_election.cc:336] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Timed out: RequestConsensusVote RPC to 127.16.29.68:42577 timed out after 2.086s (SENT)
I20260709 12:22:53.199038 22722 consensus_queue.cc:1048] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.018s
W20260709 12:22:53.217517 17803 leader_election.cc:336] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Timed out: RequestConsensusVote RPC to 127.16.29.68:42577 timed out after 2.108s (SENT)
I20260709 12:22:53.217931 17803 leader_election.cc:304] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:53.220093 22734 raft_consensus.cc:2749] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [term 11 LEARNER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20260709 12:22:53.221518 22796 log.cc:927] Time spent T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.087s	user 0.000s	sys 0.001s
I20260709 12:22:53.222481 18333 leader_election.cc:304] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:53.223836 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:53.225220 22114 raft_consensus.cc:2749] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:22:53.225718 17760 raft_consensus.cc:1275] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20260709 12:22:53.226801 22727 raft_consensus.cc:2749] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 12:22:53.231290 17834 raft_consensus.cc:2468] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 13.
I20260709 12:22:53.175741 22673 raft_consensus.cc:3060] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:53.193038 22652 raft_consensus.cc:515] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:53.236191 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:53.237059 17757 raft_consensus.cc:2393] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 10: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:22:53.237797 22652 leader_election.cc:290] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 10 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:53.238015 22755 raft_consensus.cc:493] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:53.238870 22755 raft_consensus.cc:515] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:53.215143 22626 consensus_queue.cc:260] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:22:53.241766 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 44121.6us
I20260709 12:22:53.242748 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:53.243870 17830 raft_consensus.cc:3060] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:53.247977 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:53.249792 17763 raft_consensus.cc:1240] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 10. Current term is 11. Ops: []
I20260709 12:22:53.271759 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:53.272867 18371 raft_consensus.cc:2393] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 14: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:22:53.252121 22727 consensus_queue.cc:1059] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:22:53.273836 22727 raft_consensus.cc:3055] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:22:53.274302 22727 raft_consensus.cc:740] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:22:53.275493 22727 consensus_queue.cc:260] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.9, Last appended by leader: 9, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:53.252993 22734 consensus_queue.cc:1059] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 12:22:53.183231 22536 raft_consensus.cc:493] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:53.277160 22625 raft_consensus.cc:3055] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:22:53.277585 22625 raft_consensus.cc:740] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:22:53.277678 22536 raft_consensus.cc:515] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:53.278518 22625 consensus_queue.cc:260] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:53.279731 22625 raft_consensus.cc:3060] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:53.281068 22536 leader_election.cc:290] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:53.193922 22695 raft_consensus.cc:2749] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:22:53.188128 22468 raft_consensus.cc:493] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:53.282382 22468 raft_consensus.cc:515] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:53.284929 17830 raft_consensus.cc:2468] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 10.
I20260709 12:22:53.285629 22468 leader_election.cc:290] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:53.264518 17762 raft_consensus.cc:1240] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 11. Current term is 12. Ops: []
I20260709 12:22:53.266449 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 11 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:53.266651 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:53.287367 18374 raft_consensus.cc:2393] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 11: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:22:53.288419 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:53.289757 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:53.290624 18374 raft_consensus.cc:2393] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 10: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:22:53.269199 17757 raft_consensus.cc:1275] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
I20260709 12:22:53.287267 17764 raft_consensus.cc:2468] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 14.
I20260709 12:22:53.268265 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 9 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:53.293999 18365 raft_consensus.cc:2468] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 9.
I20260709 12:22:53.295337 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 12 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:53.291494 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:53.204692 22684 raft_consensus.cc:697] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 10 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:22:53.314685 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:53.315467 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 11 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:53.318586 17830 raft_consensus.cc:2468] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 14.
I20260709 12:22:53.317126 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:53.316234 18374 raft_consensus.cc:2368] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 12. Current term is 13.
I20260709 12:22:53.320606 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 13 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:53.321107 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 16 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:53.319974 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 11 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:53.322324 18371 raft_consensus.cc:2410] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 11 because replica has last-logged OpId of term: 10 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 7.
I20260709 12:22:53.322350 18361 raft_consensus.cc:1275] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 10 index: 11. Preceding OpId from leader: term: 11 index: 12. (index mismatch)
I20260709 12:22:53.321204 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:53.323519 18374 raft_consensus.cc:2393] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 10: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:22:53.323645 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:53.325556 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:53.326129 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:53.326902 17830 raft_consensus.cc:2468] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 14.
I20260709 12:22:53.329373 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 14 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:22:53.330480 17803 leader_election.cc:336] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Timed out: RequestConsensusVote RPC to 127.16.29.68:42577 timed out after 2.341s (SENT)
I20260709 12:22:53.330273 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:53.332312 17764 raft_consensus.cc:2393] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 14: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:22:53.333544 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 15 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:53.333798 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:53.335772 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:53.336603 18367 raft_consensus.cc:2393] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 14: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:22:53.297032 22626 raft_consensus.cc:3060] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:53.339367 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 14 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:53.336668 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 14 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:53.385015 22727 raft_consensus.cc:3060] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:53.376019 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:53.376809 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:22:53.378007 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 47558.8us
I20260709 12:22:53.392119 22615 raft_consensus.cc:493] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:53.392719 22615 raft_consensus.cc:515] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:22:53.393180 17803 leader_election.cc:336] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Timed out: RequestConsensusVote RPC to 127.16.29.68:42577 timed out after 2.374s (SENT)
I20260709 12:22:53.393772 17803 leader_election.cc:304] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:53.395080 22615 leader_election.cc:290] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:53.378544 22777 raft_consensus.cc:493] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:53.379244 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 15 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:53.340312 17830 raft_consensus.cc:2468] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 13.
I20260709 12:22:53.385056 22625 raft_consensus.cc:493] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:53.395709 17803 leader_election.cc:304] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:53.380195 17832 raft_consensus.cc:1240] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 8. Current term is 9. Ops: []
I20260709 12:22:53.386734 18333 leader_election.cc:304] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:53.406489 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:53.407215 18361 raft_consensus.cc:2468] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 13.
I20260709 12:22:53.411186 17762 raft_consensus.cc:2410] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for term 15 because replica has last-logged OpId of term: 13 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 11.
I20260709 12:22:53.412319 18333 leader_election.cc:304] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:22:53.422327 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:53.414287 17764 raft_consensus.cc:1275] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 13 index: 12. Preceding OpId from leader: term: 14 index: 13. (index mismatch)
I20260709 12:22:53.425029 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:53.425961 17756 raft_consensus.cc:2393] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 14: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:22:53.426177 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
W20260709 12:22:53.404652 22791 log.cc:927] Time spent T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.081s	user 0.000s	sys 0.000s
I20260709 12:22:53.395654 22615 raft_consensus.cc:493] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:53.430373 17830 raft_consensus.cc:1275] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 9 index: 7. Preceding OpId from leader: term: 11 index: 8. (index mismatch)
I20260709 12:22:53.431669 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:53.434660 22693 consensus_queue.cc:1059] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 11.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:22:53.435974 22693 consensus_queue.cc:1048] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 12:22:53.437899 22693 consensus_queue.cc:1048] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 12:22:53.440145 22684 consensus_queue.cc:237] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.7, Last appended by leader: 7, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:53.441776 22578 consensus_queue.cc:1059] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 10.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 12:22:53.442528 22468 consensus_queue.cc:1059] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 12:22:53.442714 22764 consensus_queue.cc:1048] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260709 12:22:53.443423 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:53.443347 22578 raft_consensus.cc:3055] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:22:53.444171 22578 raft_consensus.cc:740] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 11 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:22:53.445101 22578 consensus_queue.cc:260] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.8, Last appended by leader: 8, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:53.446637 22578 raft_consensus.cc:3060] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:53.448693 17652 leader_election.cc:304] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:53.449483 17652 leader_election.cc:304] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:22:53.450495 18361 raft_consensus.cc:2410] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for term 12 because replica has last-logged OpId of term: 11 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 10.
I20260709 12:22:53.396253 22590 raft_consensus.cc:2749] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:22:53.452858 17652 leader_election.cc:304] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:53.423327 22734 raft_consensus.cc:2749] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:22:53.455232 17652 leader_election.cc:304] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:53.422842 22673 raft_consensus.cc:2749] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:22:53.442531 17801 leader_election.cc:304] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:53.459546 17801 leader_election.cc:304] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:53.447774 22192 raft_consensus.cc:3060] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:53.455745 22668 raft_consensus.cc:2804] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:22:53.463477 22668 raft_consensus.cc:493] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:53.457921 22408 raft_consensus.cc:2804] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:22:53.463935 22668 raft_consensus.cc:3060] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:22:53.464216 22408 raft_consensus.cc:493] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:53.464694 22408 raft_consensus.cc:3060] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:53.423710 22625 raft_consensus.cc:515] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:53.467922 22383 consensus_queue.cc:1048] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.011s
I20260709 12:22:53.469064 22625 leader_election.cc:290] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:53.472602 18365 raft_consensus.cc:2410] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 9 because replica has last-logged OpId of term: 8 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 8.
I20260709 12:22:53.413610 22777 raft_consensus.cc:515] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:53.474282 22722 raft_consensus.cc:2804] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:22:53.474912 22722 raft_consensus.cc:493] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:53.475350 22722 raft_consensus.cc:3060] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:22:53.476882 17801 leader_election.cc:304] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:53.478497 17801 leader_election.cc:304] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:53.478592 22468 raft_consensus.cc:2749] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:22:53.479218 22695 raft_consensus.cc:2749] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:22:53.479269 22468 raft_consensus.cc:2749] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:22:53.480795 17801 leader_election.cc:304] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:53.480569 22740 consensus_queue.cc:1048] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:22:53.482681 22694 consensus_queue.cc:1048] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:22:53.483301 22416 raft_consensus.cc:493] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:53.484748 22548 raft_consensus.cc:2749] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:22:53.458094 22760 consensus_queue.cc:1048] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.028s
I20260709 12:22:53.486301 22416 raft_consensus.cc:515] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:53.489591 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:53.492760 17801 leader_election.cc:304] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:53.495814 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:53.496645 18361 raft_consensus.cc:3060] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:22:53.450960 17789 heartbeater.cc:646] Failed to heartbeat to 127.16.29.126:39149 (2 consecutive failures): Timed out: Failed to send heartbeat to master: TSHeartbeat RPC to 127.16.29.126:39149 timed out after 15.000s (SENT)
I20260709 12:22:53.487963 17762 raft_consensus.cc:2955] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Committing config change with OpId 8.5: config changed from index -1 to 5, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } }
W20260709 12:22:53.510689 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:53.516036 22729 raft_consensus.cc:493] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:53.516788 22729 raft_consensus.cc:515] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:53.520493 22729 leader_election.cc:290] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:53.456029 22495 raft_consensus.cc:2749] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
W20260709 12:22:53.528437 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 116218us
I20260709 12:22:53.529318 18333 leader_election.cc:304] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:22:53.501600 22468 raft_consensus.cc:2749] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 12:22:53.498159 22700 raft_consensus.cc:3055] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:22:53.523257 22738 raft_consensus.cc:493] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:53.508605 22590 raft_consensus.cc:2749] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:22:53.451372 22615 raft_consensus.cc:515] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:53.536943 18333 leader_election.cc:304] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:53.548600 22619 raft_consensus.cc:493] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:53.549293 22619 raft_consensus.cc:515] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:53.549746 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:53.550328 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:53.551805 22725 raft_consensus.cc:493] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:22:53.551918 17802 leader_election.cc:304] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:53.552371 22725 raft_consensus.cc:3060] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:53.554347 18368 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 1577) took 1619 ms. Trace:
I20260709 12:22:53.535596 22416 leader_election.cc:290] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:53.535244 22729 raft_consensus.cc:2749] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:22:53.556279 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:53.557935 22474 leader_election.cc:290] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:53.558537 22695 raft_consensus.cc:2764] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Leader pre-election decision vote started in defunct term 13: won
I20260709 12:22:53.558390 22408 raft_consensus.cc:515] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:53.561880 18364 raft_consensus.cc:3055] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:22:53.562078 18333 leader_election.cc:304] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:22:53.562402 18364 raft_consensus.cc:740] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:22:53.554742 18368 rpcz_store.cc:276] 0709 12:22:51.934467 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:51.936832 (+  2365us) service_pool.cc:224] Handling call
0709 12:22:51.937093 (+   261us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:52.871292 (+934199us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:53.554111 (+682819us) spinlock_profiling.cc:243] Waited 934 ms on lock 0x7b540047bdd4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:22:53.554285 (+   174us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2147065600}
I20260709 12:22:53.567035 18333 leader_election.cc:304] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:22:53.573619 17830 raft_consensus.cc:1275] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 13 index: 11. (index mismatch)
I20260709 12:22:53.574429 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:53.575240 18368 raft_consensus.cc:3060] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:53.568724 22488 raft_consensus.cc:493] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:22:53.534162 22700 raft_consensus.cc:740] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:22:53.580037 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:53.601508 17728 leader_election.cc:304] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:53.555181 22668 raft_consensus.cc:515] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:53.578184 18364 consensus_queue.cc:260] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 12.10, Last appended by leader: 10, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:53.607131 17728 leader_election.cc:304] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:22:53.565394 22729 raft_consensus.cc:2749] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:22:53.577684 22488 raft_consensus.cc:515] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:53.611479 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:53.612555 22491 leader_election.cc:290] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 12 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:53.612946 22408 leader_election.cc:290] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 12 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:53.614382 22668 leader_election.cc:290] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 15 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:53.616230 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:53.619661 18364 raft_consensus.cc:3060] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:53.630286 18362 raft_consensus.cc:2468] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 12.
I20260709 12:22:53.566231 22777 leader_election.cc:290] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:53.622064 18361 raft_consensus.cc:2468] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 10.
I20260709 12:22:53.624112 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:53.633934 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:53.570304 22808 raft_consensus.cc:2749] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:22:53.631531 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:53.582547 22673 raft_consensus.cc:493] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:53.559937 22738 raft_consensus.cc:515] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:53.634727 17761 raft_consensus.cc:3060] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:22:53.636601 17729 leader_election.cc:304] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:53.644107 18362 raft_consensus.cc:2468] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 10.
I20260709 12:22:53.645597 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:53.646399 18362 raft_consensus.cc:2468] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 14.
I20260709 12:22:53.647645 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:53.649115 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:53.649914 18362 raft_consensus.cc:2468] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 12.
I20260709 12:22:53.651114 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:53.651918 18362 raft_consensus.cc:2468] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 14.
I20260709 12:22:53.624337 22192 raft_consensus.cc:2749] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:22:53.637924 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:53.636973 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:53.662922 18361 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 10.
I20260709 12:22:53.663990 18364 raft_consensus.cc:1275] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 12 index: 10. Preceding OpId from leader: term: 13 index: 10. (term mismatch)
I20260709 12:22:53.664614 18364 pending_rounds.cc:85] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da: Aborting all ops after (but not including) 9
I20260709 12:22:53.664983 18364 pending_rounds.cc:107] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da: Aborting uncommitted NO_OP operation due to leader change: 12.10
I20260709 12:22:53.665449 18364 raft_consensus.cc:2889] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:22:53.667017 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 100041us
I20260709 12:22:53.668128 18333 leader_election.cc:304] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:22:53.671237 22488 leader_election.cc:290] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:53.678627 22653 leader_election.cc:290] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 14 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:53.643288 22807 raft_consensus.cc:2804] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:22:53.680161 22807 raft_consensus.cc:493] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:53.680538 22807 raft_consensus.cc:3060] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:53.653121 22668 raft_consensus.cc:493] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:53.663806 22383 raft_consensus.cc:2804] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:22:53.684227 22668 raft_consensus.cc:515] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:53.685050 17727 leader_election.cc:304] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:53.686396 22668 leader_election.cc:290] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:53.684643 22383 raft_consensus.cc:697] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
W20260709 12:22:53.689896 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:53.688406 18363 raft_consensus.cc:2468] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 11.
I20260709 12:22:53.698580 18368 raft_consensus.cc:2468] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 14.
I20260709 12:22:53.693933 18333 leader_election.cc:304] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
W20260709 12:22:53.701730 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 79495.4us
I20260709 12:22:53.706382 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:53.695472 17729 leader_election.cc:304] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:53.710587 17757 raft_consensus.cc:2393] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 14: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:22:53.711342 18333 leader_election.cc:304] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:22:53.660655 22619 leader_election.cc:290] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:53.722678 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:53.726030 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:53.726015 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:53.727715 18368 raft_consensus.cc:3060] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:22:53.731357 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 46398.7us
I20260709 12:22:53.712678 17761 raft_consensus.cc:2468] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 15.
W20260709 12:22:53.713732 17803 leader_election.cc:336] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Timed out: RequestConsensusVote RPC to 127.16.29.68:42577 timed out after 1.831s (SENT)
I20260709 12:22:53.715713 17832 raft_consensus.cc:1240] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 12. Current term is 13. Ops: []
I20260709 12:22:53.719923 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:53.720878 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 12 candidate_status { last_received { term: 11 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:53.721822 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:53.736037 22727 raft_consensus.cc:493] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:53.736128 22625 raft_consensus.cc:2804] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:22:53.736793 22625 raft_consensus.cc:493] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:53.736966 22734 raft_consensus.cc:493] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:53.737259 22625 raft_consensus.cc:3060] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:53.737527 22734 raft_consensus.cc:515] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:53.739959 22734 leader_election.cc:290] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:53.740476 22734 raft_consensus.cc:493] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:53.740947 22734 raft_consensus.cc:515] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:53.743381 22734 leader_election.cc:290] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:53.745739 17803 leader_election.cc:304] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:22:53.746798 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 51429us
I20260709 12:22:53.749708 17761 raft_consensus.cc:1240] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 13 LEADER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 12. Current term is 13. Ops: []
I20260709 12:22:53.671241 22738 leader_election.cc:290] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:53.696795 22416 raft_consensus.cc:2749] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:22:53.754768 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:53.777920 17833 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 11.
I20260709 12:22:53.693126 22192 raft_consensus.cc:2749] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:22:53.756637 22619 raft_consensus.cc:493] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:53.756649 22734 raft_consensus.cc:493] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:53.753643 22491 raft_consensus.cc:2804] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:22:53.758590 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:53.760138 17687 raft_consensus.cc:1240] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 11 LEADER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 10. Current term is 11. Ops: []
I20260709 12:22:53.759913 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:53.761129 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:53.761678 17688 raft_consensus.cc:2468] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 13.
I20260709 12:22:53.761902 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:53.809156 17682 raft_consensus.cc:2468] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 9.
I20260709 12:22:53.776001 22652 raft_consensus.cc:2804] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:22:53.772323 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:53.774032 18366 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 1575) took 1840 ms. Trace:
I20260709 12:22:53.661104 22673 raft_consensus.cc:515] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:53.817230 17760 raft_consensus.cc:2468] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 13.
I20260709 12:22:53.788374 17729 leader_election.cc:304] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:53.792841 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:53.832300 18362 raft_consensus.cc:2393] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 9: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:22:53.729053 22729 consensus_queue.cc:1059] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.081s
I20260709 12:22:53.720337 22808 raft_consensus.cc:2804] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:22:53.801151 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:53.679733 22722 raft_consensus.cc:515] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:53.758092 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 14 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:22:53.812002 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 79075.9us
I20260709 12:22:53.928262 17727 leader_election.cc:304] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:22:53.811257 17689 raft_consensus.cc:2468] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 14.
I20260709 12:22:53.814914 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:53.856302 22808 raft_consensus.cc:493] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:22:53.820227 18368 raft_consensus.cc:2468] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 10.
I20260709 12:22:53.910655 22723 raft_consensus.cc:493] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:53.788105 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:53.841572 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:53.784091 22615 leader_election.cc:290] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:53.786783 17803 leader_election.cc:304] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:22:53.938593 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 151908us
I20260709 12:22:53.770136 22755 leader_election.cc:290] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:53.905100 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:53.942437 17687 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 11.
I20260709 12:22:53.943248 22695 raft_consensus.cc:2804] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:22:53.943327 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:22:53.905956 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 126966us
I20260709 12:22:53.943825 22695 raft_consensus.cc:493] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:53.944262 22695 raft_consensus.cc:3060] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:53.776616 22476 consensus_queue.cc:1048] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.038s
I20260709 12:22:53.800678 22738 raft_consensus.cc:2749] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
W20260709 12:22:53.948628 22814 log.cc:927] Time spent T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.136s	user 0.000s	sys 0.000s
I20260709 12:22:53.736816 22727 raft_consensus.cc:515] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:22:53.915474 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 96501.9us
I20260709 12:22:53.831266 22619 raft_consensus.cc:515] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:53.811857 22734 raft_consensus.cc:515] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:53.779485 22383 consensus_queue.cc:237] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:53.922897 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 12 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:53.957340 18362 raft_consensus.cc:2410] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 12 because replica has last-logged OpId of term: 11 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 7.
I20260709 12:22:53.835245 22491 raft_consensus.cc:697] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:22:53.842097 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:22:53.917034 22776 log.cc:927] Time spent T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.103s	user 0.000s	sys 0.001s
I20260709 12:22:53.962656 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:53.816787 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:53.929482 17830 raft_consensus.cc:2468] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 9.
I20260709 12:22:53.817123 18366 rpcz_store.cc:276] 0709 12:22:51.933806 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:51.945583 (+ 11777us) service_pool.cc:224] Handling call
0709 12:22:51.945929 (+   346us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:52.026908 (+ 80979us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:52.026953 (+    45us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:52.027010 (+    57us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:22:52.027054 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:53.773665 (+1746611us) spinlock_profiling.cc:243] Waited 80.7 ms on lock 0x7b5400305554. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:22:53.773716 (+    51us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:53.773967 (+   251us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":185626368}
I20260709 12:22:53.915226 22518 raft_consensus.cc:2804] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:22:53.935643 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:53.937252 17761 raft_consensus.cc:2393] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 11: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:22:53.939241 18368 raft_consensus.cc:1275] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 11 index: 11. Preceding OpId from leader: term: 14 index: 13. (index mismatch)
I20260709 12:22:53.968730 17681 raft_consensus.cc:2393] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 14: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:22:53.934718 17727 leader_election.cc:304] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:53.941269 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:53.816761 22652 raft_consensus.cc:493] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:53.962215 17758 raft_consensus.cc:1240] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 10. Current term is 11. Ops: []
I20260709 12:22:53.972028 18366 raft_consensus.cc:1275] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 13 index: 11. (index mismatch)
W20260709 12:22:53.971489 18333 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:53.972456 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:53.969798 17727 leader_election.cc:304] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:53.973063 17683 raft_consensus.cc:3060] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:53.971259 22615 raft_consensus.cc:493] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:53.971498 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:53.982128 22652 raft_consensus.cc:3060] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:22:53.936389 22808 raft_consensus.cc:3060] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:53.952220 22723 raft_consensus.cc:515] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:22:53.996419 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:53.997404 18368 raft_consensus.cc:1275] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 9 index: 7. Preceding OpId from leader: term: 11 index: 8. (index mismatch)
I20260709 12:22:54.038357 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:54.046976 17681 raft_consensus.cc:3060] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:54.047623 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:54.048524 18368 raft_consensus.cc:2468] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 9.
I20260709 12:22:54.049664 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:54.050354 18368 raft_consensus.cc:2468] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 9.
I20260709 12:22:53.998037 17687 raft_consensus.cc:2468] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 14.
I20260709 12:22:53.998418 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:54.052721 17682 raft_consensus.cc:2468] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 13.
I20260709 12:22:54.053825 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:53.999578 22615 raft_consensus.cc:515] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:54.001134 22652 raft_consensus.cc:515] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:54.056484 22668 raft_consensus.cc:493] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:54.057148 22668 raft_consensus.cc:515] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:54.003973 22745 consensus_queue.cc:1048] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 12:22:54.004232 22518 raft_consensus.cc:697] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 14 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:22:54.062077 22518 consensus_queue.cc:237] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.11, Last appended by leader: 11, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:54.005048 22653 consensus_queue.cc:1048] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 12:22:54.005741 22360 raft_consensus.cc:2804] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:22:54.066812 22360 raft_consensus.cc:493] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:54.067539 22360 raft_consensus.cc:3060] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:22:54.005865 17833 raft_consensus.cc:1275] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20260709 12:22:54.006955 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:53.989627 22474 raft_consensus.cc:2804] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:22:54.077791 22474 raft_consensus.cc:697] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:22:54.008452 22619 leader_election.cc:290] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:54.008811 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:54.010242 18333 leader_election.cc:304] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:54.081461 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:54.083076 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:54.012048 22734 leader_election.cc:290] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:54.014235 22725 raft_consensus.cc:515] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:22:53.975272 22693 raft_consensus.cc:2749] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:22:54.016361 22305 raft_consensus.cc:493] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:54.017311 22722 leader_election.cc:290] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 15 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:54.020907 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:53.984491 22491 consensus_queue.cc:237] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:54.034523 17683 raft_consensus.cc:2468] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 10.
I20260709 12:22:54.044339 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 13 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:54.097561 18367 raft_consensus.cc:2410] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 13 because replica has last-logged OpId of term: 12 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 8.
I20260709 12:22:54.050160 17654 leader_election.cc:304] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:22:54.098685 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 48610.7us
I20260709 12:22:54.099021 17834 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35042 (request call id 2295) took 1313 ms. Trace:
I20260709 12:22:54.085634 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:54.099563 17834 rpcz_store.cc:276] 0709 12:22:52.785573 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:52.822485 (+ 36912us) service_pool.cc:224] Handling call
0709 12:22:54.098847 (+1276362us) spinlock_profiling.cc:243] Waited 408 ms on lock 0x7b54002ebb54. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:22:54.098967 (+   120us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":938068224}
I20260709 12:22:54.100478 17682 raft_consensus.cc:2393] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 14: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:22:54.019713 22729 raft_consensus.cc:493] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:54.088107 17689 raft_consensus.cc:2410] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for term 10 because replica has last-logged OpId of term: 9 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 5.
W20260709 12:22:54.093060 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 82896.4us
I20260709 12:22:54.105106 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:54.106595 17803 leader_election.cc:304] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:54.107404 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:54.059855 22668 leader_election.cc:290] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:54.108934 17803 leader_election.cc:304] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:54.112236 18333 leader_election.cc:304] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:22:54.105101 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:54.113010 17684 raft_consensus.cc:1240] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 11. Current term is 12. Ops: []
W20260709 12:22:54.116802 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 176225us
I20260709 12:22:54.093055 22727 leader_election.cc:290] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:54.121338 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:54.094692 22734 raft_consensus.cc:2749] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:22:54.122371 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:54.111285 17654 leader_election.cc:304] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:54.085345 22807 raft_consensus.cc:515] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:54.127851 22668 raft_consensus.cc:2804] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:22:54.128297 22668 raft_consensus.cc:493] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:54.128679 22668 raft_consensus.cc:3060] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:54.128460 17654 leader_election.cc:304] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:54.109632 22817 raft_consensus.cc:2804] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:22:54.113109 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:54.130779 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:54.137979 17762 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.BulkChangeConfig from 127.0.0.1:37500 (request call id 0) took 32845 ms. Trace:
I20260709 12:22:54.144037 17689 raft_consensus.cc:2468] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 13.
I20260709 12:22:54.155579 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:54.079304 22474 consensus_queue.cc:237] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:54.142735 17762 rpcz_store.cc:276] 0709 12:22:21.275562 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:21.275820 (+   258us) service_pool.cc:224] Handling call
0709 12:22:21.707265 (+431445us) spinlock_profiling.cc:243] Waited 2.45 ms on lock 0x7b4c00242600. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b488eca20 00007f7b488fda29 00007f7b488c3644 00007f7b488eaf2d 00007f7b488eab93 00007f7b4a5791af 00007f7b4a578f3e 00007f7b4a5fe888 00007f7b4a61b75c 00007f7b4a61c76a 00007f7b4a61b91b
0709 12:22:21.708805 (+  1540us) log.cc:844] Serialized 499 byte log entry
0709 12:22:22.166034 (+457229us) spinlock_profiling.cc:243] Waited 10.2 ms on lock 0x7b540022bda0. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a595dbe 00007f7b4a5a0930 00007f7b4a5a08d8 00007f7b4a5a0858 00007f7b4a5a0813 00007f7b4a59fa9c 00007f7b5048b2f6 00007f7b50485aa8 00007f7b4a5dc7cb 00007f7b4a5df2c0
0709 12:22:22.275748 (+109714us) spinlock_profiling.cc:243] Waited 60.3 ms on lock 0x7b1800452a40. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b4a1c84bc 00007f7b4751648b 00007f7b503709a9 00007f7b4753269b 00007f7b4753093f 00007f7b47532c02 00007f7b4a1ba20b 00007f7b4a1bfd23 00007f7b4a1c00d1
0709 12:22:54.115656 (+31839908us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3185,"raft.queue_time_us":557499,"raft.run_cpu_time_us":569,"raft.run_wall_time_us":568,"spinlock_wait_cycles":167806720,"thread_start_us":33811,"threads_started":1,"wal-append.queue_time_us":32950,"wal-append.run_cpu_time_us":7461,"wal-append.run_wall_time_us":2052890}
I20260709 12:22:54.114342 22729 raft_consensus.cc:515] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:54.115599 22763 consensus_queue.cc:1059] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 11.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.033s
I20260709 12:22:54.148908 22817 raft_consensus.cc:697] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 14 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:22:54.150498 22668 raft_consensus.cc:515] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:54.114562 22722 raft_consensus.cc:2749] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:22:54.151790 22734 raft_consensus.cc:2804] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:22:54.129148 22807 leader_election.cc:290] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 14 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:54.131191 22727 raft_consensus.cc:493] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:54.153333 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:54.153541 17686 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37402 (request call id 2717) took 2119 ms. Trace:
I20260709 12:22:54.178402 17833 raft_consensus.cc:2468] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 10.
I20260709 12:22:54.154078 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 12 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:54.180299 17682 raft_consensus.cc:2410] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for term 12 because replica has last-logged OpId of term: 10 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 7.
I20260709 12:22:54.178305 17686 rpcz_store.cc:276] 0709 12:22:52.033833 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:52.034030 (+   197us) service_pool.cc:224] Handling call
0709 12:22:52.066446 (+ 32416us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:22:52.230965 (+164519us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:22:52.237819 (+  6854us) log.cc:844] Serialized 14 byte log entry
0709 12:22:52.400677 (+162858us) log.cc:844] Serialized 14 byte log entry
0709 12:22:52.400930 (+   253us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:22:52.409772 (+  8842us) log.cc:844] Serialized 52 byte log entry
0709 12:22:52.409918 (+   146us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:22:52.409953 (+    35us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:52.410037 (+    84us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:52.733800 (+323763us) spinlock_profiling.cc:243] Waited 20.3 ms on lock 0x7b1800474a60. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b474adc0a 00007f7b50bed219 00007f7b4e4f458f 00007f7b47533571 00007f7b475357c1 00007f7b47535779 00007f7b47535709 00007f7b475356d1 00007f7b475349cd
0709 12:22:54.150718 (+1416918us) spinlock_profiling.cc:243] Waited 9.25 ms on lock 0x7b1800474a60. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b475333f3 00007f7b475357c1 00007f7b47535779 00007f7b47535709 00007f7b475356d1 00007f7b475349cd 0000000000388634 0000000000388479 00007f7b4751683a
0709 12:22:54.153240 (+  2522us) raft_consensus.cc:1685] finished
0709 12:22:54.153263 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:54.153447 (+   184us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":123025,"prepare.queue_time_us":8680,"prepare.run_cpu_time_us":114,"prepare.run_wall_time_us":1550,"raft.queue_time_us":113087,"raft.run_cpu_time_us":400,"raft.run_wall_time_us":23617,"spinlock_wait_cycles":68018816,"thread_start_us":280430,"threads_started":2,"wal-append.queue_time_us":305579}
I20260709 12:22:54.114286 22625 raft_consensus.cc:515] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:54.158332 17801 leader_election.cc:304] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:54.113902 22673 leader_election.cc:290] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:54.162835 17654 leader_election.cc:304] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:22:54.184577 18362 raft_consensus.cc:2468] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 10.
I20260709 12:22:54.159591 22755 consensus_queue.cc:1048] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.014s
I20260709 12:22:54.096873 22305 raft_consensus.cc:515] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:54.155946 17803 leader_election.cc:304] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:54.170645 17689 raft_consensus.cc:1275] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 14 index: 8. Preceding OpId from leader: term: 15 index: 8. (term mismatch)
I20260709 12:22:54.188544 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:54.189845 17689 pending_rounds.cc:85] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3: Aborting all ops after (but not including) 7
I20260709 12:22:54.190197 17689 pending_rounds.cc:107] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3: Aborting uncommitted NO_OP operation due to leader change: 14.8
I20260709 12:22:54.190470 17689 raft_consensus.cc:2889] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:54.193048 17682 raft_consensus.cc:1240] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 12. Current term is 13. Ops: []
I20260709 12:22:54.121165 22738 raft_consensus.cc:2749] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:22:54.200014 18367 raft_consensus.cc:1240] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 13. Current term is 14. Ops: []
I20260709 12:22:54.176873 22734 raft_consensus.cc:493] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:54.195542 22807 raft_consensus.cc:493] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:54.206897 17683 raft_consensus.cc:1275] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 12 index: 8. Preceding OpId from leader: term: 14 index: 9. (index mismatch)
W20260709 12:22:54.200977 22816 log.cc:927] Time spent T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.112s	user 0.000s	sys 0.001s
W20260709 12:22:54.210438 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 98280.7us
I20260709 12:22:54.193699 22749 raft_consensus.cc:2804] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:22:54.146047 22672 consensus_queue.cc:260] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:54.199484 22619 raft_consensus.cc:2804] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:22:54.206658 22734 raft_consensus.cc:3060] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:54.187692 22727 raft_consensus.cc:515] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:54.225641 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:54.226879 17833 raft_consensus.cc:3060] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:54.216619 22749 raft_consensus.cc:493] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:54.218258 22673 raft_consensus.cc:2749] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:22:54.220839 22619 raft_consensus.cc:493] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:54.207863 22807 raft_consensus.cc:515] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:54.226558 22823 raft_consensus.cc:493] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:54.260265 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 116476us
I20260709 12:22:54.263693 17803 leader_election.cc:304] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:22:54.261299 22619 raft_consensus.cc:3060] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:54.219722 22722 consensus_queue.cc:1059] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 10.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.037s
I20260709 12:22:54.251346 22749 raft_consensus.cc:3060] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:54.267226 17803 leader_election.cc:304] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
W20260709 12:22:54.269991 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 112886us
I20260709 12:22:54.201573 22817 consensus_queue.cc:237] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 13.10, Last appended by leader: 10, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:54.246510 22652 leader_election.cc:290] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 15 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:54.273720 17680 raft_consensus.cc:1240] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 11. Current term is 12. Ops: []
I20260709 12:22:54.282927 17803 leader_election.cc:304] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:54.263119 22725 leader_election.cc:290] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 12 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:54.265664 22626 raft_consensus.cc:2804] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:22:54.285302 17757 raft_consensus.cc:1240] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 10. Current term is 11. Ops: []
I20260709 12:22:54.286149 18334 leader_election.cc:304] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:22:54.285151 17833 raft_consensus.cc:2468] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 14.
I20260709 12:22:54.250562 22723 leader_election.cc:290] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:54.272348 22673 raft_consensus.cc:2804] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:22:54.319351 22673 raft_consensus.cc:493] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:54.319757 22673 raft_consensus.cc:3060] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:22:54.323066 22797 log.cc:927] Time spent T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 1.192s	user 0.000s	sys 0.018s
I20260709 12:22:54.324326 22729 leader_election.cc:290] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:54.318929 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:54.324805 22218 raft_consensus.cc:3055] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:22:54.325214 22218 raft_consensus.cc:740] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:22:54.326032 22218 consensus_queue.cc:260] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:54.325847 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:54.327471 22218 raft_consensus.cc:3060] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:54.287578 22476 raft_consensus.cc:2804] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:22:54.290115 22834 consensus_queue.cc:1059] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.002s
I20260709 12:22:54.292389 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:54.297227 22807 leader_election.cc:290] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:54.287731 22615 leader_election.cc:290] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:54.298772 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:54.268214 22755 consensus_queue.cc:1048] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.076s
I20260709 12:22:54.302903 22695 raft_consensus.cc:515] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:54.267649 22823 raft_consensus.cc:515] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:54.305146 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:54.307114 22360 raft_consensus.cc:515] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:54.309023 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:54.310953 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:54.311619 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:54.312301 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:54.312408 17681 raft_consensus.cc:2468] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 14.
I20260709 12:22:54.313112 17831 raft_consensus.cc:1240] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 8. Current term is 9. Ops: [8.8-8.8]
I20260709 12:22:54.328313 22668 leader_election.cc:290] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 13 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:54.328990 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:54.456442 18363 raft_consensus.cc:2468] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 6.
I20260709 12:22:54.333359 17728 leader_election.cc:304] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
W20260709 12:22:54.333890 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 47819.4us
I20260709 12:22:54.289585 22727 leader_election.cc:290] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:54.458250 18334 leader_election.cc:304] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:54.336205 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:54.345875 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:54.460048 17832 raft_consensus.cc:2468] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 13.
I20260709 12:22:54.372051 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:54.288288 22738 raft_consensus.cc:2749] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:22:54.454308 17755 raft_consensus.cc:2468] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 13.
I20260709 12:22:54.367659 22729 raft_consensus.cc:2749] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
W20260709 12:22:54.436312 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 153419us
W20260709 12:22:54.470698 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 137433us
I20260709 12:22:54.471484 17728 leader_election.cc:304] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
W20260709 12:22:54.471499 17803 leader_election.cc:336] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Timed out: RequestConsensusVote RPC to 127.16.29.68:42577 timed out after 3.102s (SENT)
I20260709 12:22:54.371951 22476 raft_consensus.cc:493] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:54.480101 17803 leader_election.cc:304] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:22:54.480926 18334 leader_election.cc:304] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:22:54.482645 18334 leader_election.cc:304] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:54.474260 17728 leader_election.cc:304] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
W20260709 12:22:54.489369 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:54.489439 17758 raft_consensus.cc:2393] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 15: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:22:54.488087 18334 leader_election.cc:304] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:22:54.493270 17728 leader_election.cc:304] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:22:54.478674 22069 raft_consensus.cc:2749] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:22:54.493979 22832 raft_consensus.cc:2804] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:22:54.497030 22808 raft_consensus.cc:515] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:54.495410 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:54.492245 22615 raft_consensus.cc:2749] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:22:54.492755 22536 raft_consensus.cc:2749] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:22:54.483973 22476 raft_consensus.cc:3060] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:54.475440 22834 consensus_queue.cc:1243] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer 0da169ff82674dc1947055ce43de68a3 log is divergent from this leader: its last log entry 11.10 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 8
I20260709 12:22:54.480756 22474 raft_consensus.cc:2749] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:22:54.492399 22729 raft_consensus.cc:2804] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:22:54.507423 17728 leader_election.cc:304] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:22:54.488374 22696 raft_consensus.cc:2804] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:22:54.509475 18362 raft_consensus.cc:2468] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 11.
I20260709 12:22:54.508706 22832 raft_consensus.cc:493] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:54.494247 22305 leader_election.cc:290] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:54.520747 18371 raft_consensus.cc:2468] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 14.
W20260709 12:22:54.521879 17803 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:54.508196 22653 raft_consensus.cc:2749] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:22:54.523721 22305 raft_consensus.cc:2749] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 12:22:54.523033 17728 leader_election.cc:304] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:54.508422 22833 raft_consensus.cc:2804] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:22:54.536134 17803 leader_election.cc:304] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:22:54.529940 22700 consensus_queue.cc:260] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.10, Last appended by leader: 10, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:54.523132 17652 leader_election.cc:304] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:54.546614 17728 leader_election.cc:304] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:22:54.520488 22832 raft_consensus.cc:3060] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:54.516434 22696 raft_consensus.cc:493] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:54.548316 17803 leader_election.cc:304] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:22:54.553911 22844 raft_consensus.cc:2749] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 12:22:54.557355 17728 leader_election.cc:304] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:22:54.506583 22749 raft_consensus.cc:515] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:54.559604 22653 raft_consensus.cc:2749] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 12:22:54.560026 22808 leader_election.cc:290] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 14 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:54.557969 22590 raft_consensus.cc:2804] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:22:54.560731 22590 raft_consensus.cc:493] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:54.561108 22590 raft_consensus.cc:3060] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:54.561452 18366 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59886 (request call id 1453) took 1049 ms. Trace:
I20260709 12:22:54.561813 18366 rpcz_store.cc:276] 0709 12:22:53.511743 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:53.972897 (+461154us) service_pool.cc:224] Handling call
0709 12:22:53.973222 (+   325us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:53.982273 (+  9051us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 12:22:54.330939 (+348666us) spinlock_profiling.cc:243] Waited 2.65 ms on lock 0x7b18003f4600. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b4a1c84bc 00007f7b4751648b 00007f7b503709a9 00007f7b4753269b 00007f7b4753093f 00007f7b47530455 00007f7b4e0183c7 00007f7b4a62a9a7 00007f7b4a63a0bf
0709 12:22:54.331362 (+   423us) log.cc:844] Serialized 14 byte log entry
0709 12:22:54.469227 (+137865us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:54.484138 (+ 14911us) log.cc:844] Serialized 26 byte log entry
0709 12:22:54.488664 (+  4526us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:22:54.488702 (+    38us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:54.488773 (+    71us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:54.561169 (+ 72396us) raft_consensus.cc:1685] finished
0709 12:22:54.561268 (+    99us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:54.561387 (+   119us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":11362,"prepare.queue_time_us":128324,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":46,"raft.queue_time_us":124479,"raft.run_cpu_time_us":264,"raft.run_wall_time_us":11145,"spinlock_wait_cycles":6085376,"thread_start_us":227760,"threads_started":2,"wal-append.queue_time_us":131337}
I20260709 12:22:54.564038 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:54.564741 18366 raft_consensus.cc:2393] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 14: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:22:54.531425 22673 raft_consensus.cc:515] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:54.566881 17654 leader_election.cc:304] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:54.567042 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:54.568554 17829 raft_consensus.cc:2368] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 14. Current term is 15.
I20260709 12:22:54.551585 22069 raft_consensus.cc:2804] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:22:54.552724 22833 raft_consensus.cc:493] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:54.553609 17652 leader_election.cc:304] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:54.567692 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 12 candidate_status { last_received { term: 11 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:54.554508 22840 raft_consensus.cc:2804] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:22:54.574268 22840 raft_consensus.cc:493] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:54.574837 22840 raft_consensus.cc:3060] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:54.584183 18366 raft_consensus.cc:2468] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 11.
I20260709 12:22:54.586164 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:54.586992 18366 raft_consensus.cc:3060] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:22:54.571640 22536 consensus_queue.cc:1048] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.021s
I20260709 12:22:54.592680 22360 leader_election.cc:290] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 15 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:54.556715 22305 raft_consensus.cc:2749] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:22:54.557130 22696 raft_consensus.cc:3060] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:54.598035 17654 leader_election.cc:304] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:22:54.598229 17684 raft_consensus.cc:1275] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 14 index: 8. (index mismatch)
I20260709 12:22:54.598855 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:54.599380 17681 raft_consensus.cc:2468] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 9.
I20260709 12:22:54.569811 22700 raft_consensus.cc:3060] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:54.610584 17759 raft_consensus.cc:1240] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 LEADER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 8. Current term is 9. Ops: []
I20260709 12:22:54.610695 18366 raft_consensus.cc:2468] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 15.
I20260709 12:22:54.573787 22069 raft_consensus.cc:493] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:54.608652 17728 leader_election.cc:304] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:54.613131 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:54.594197 22833 raft_consensus.cc:3060] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:22:54.571868 22474 consensus_queue.cc:1048] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.043s
I20260709 12:22:54.619292 22536 raft_consensus.cc:2804] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:22:54.635121 22823 leader_election.cc:290] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:54.611218 22515 raft_consensus.cc:2749] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:22:54.614462 22069 raft_consensus.cc:3060] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:54.601003 22734 raft_consensus.cc:515] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:22:54.647249 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 93698.5us
I20260709 12:22:54.647274 22851 raft_consensus.cc:2804] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:22:54.652561 18364 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59886 (request call id 1433) took 1823 ms. Trace:
I20260709 12:22:54.653120 18364 rpcz_store.cc:276] 0709 12:22:52.829258 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:52.831443 (+  2185us) service_pool.cc:224] Handling call
0709 12:22:52.853119 (+ 21676us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:53.692591 (+839472us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:54.652243 (+959652us) spinlock_profiling.cc:243] Waited 709 ms on lock 0x7b54002a30d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:22:54.652484 (+   241us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1629636352}
I20260709 12:22:54.654706 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:54.660202 17837 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:59820 (request call id 39) took 2110 ms. Trace:
I20260709 12:22:54.660653 17837 rpcz_store.cc:276] 0709 12:22:52.550191 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:52.601354 (+ 51163us) service_pool.cc:224] Handling call
0709 12:22:54.660006 (+2058652us) spinlock_profiling.cc:243] Waited 273 ms on lock 0x7b54003354d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a61a853 00007f7b4f5e860a 00007f7b48b6e7d4 00007f7b48b6e762 00007f7b48b6e691 00007f7b48b6e60c 00007f7b48b6d8c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:22:54.660120 (+   114us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":627564288}
I20260709 12:22:54.660154 22851 raft_consensus.cc:697] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 15 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:22:54.662799 22851 consensus_queue.cc:237] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.10, Last appended by leader: 10, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:54.666067 22672 raft_consensus.cc:3060] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:54.636446 22536 raft_consensus.cc:493] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:54.637105 22192 raft_consensus.cc:3055] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:22:54.671228 17689 raft_consensus.cc:1240] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 14 LEADER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 13. Current term is 14. Ops: []
I20260709 12:22:54.672561 17652 leader_election.cc:304] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:54.636117 22695 leader_election.cc:290] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 13 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:54.674468 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 12 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:54.671893 22376 raft_consensus.cc:2749] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:22:54.691685 22734 leader_election.cc:290] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 10 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:54.691900 22673 leader_election.cc:290] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:54.677913 22536 raft_consensus.cc:3060] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:54.692843 22619 raft_consensus.cc:515] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:54.678093 17762 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38786 (request call id 2996) took 1250 ms. Trace:
I20260709 12:22:54.666216 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:54.628341 22542 consensus_queue.cc:1048] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.016s
I20260709 12:22:54.697345 17762 rpcz_store.cc:276] 0709 12:22:53.427932 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:53.428102 (+   170us) service_pool.cc:224] Handling call
0709 12:22:53.454270 (+ 26168us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:53.454525 (+   255us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 12:22:53.699721 (+245196us) log.cc:844] Serialized 14 byte log entry
0709 12:22:54.188222 (+488501us) log.cc:844] Serialized 14 byte log entry
0709 12:22:54.668122 (+479900us) spinlock_profiling.cc:243] Waited 11 ms on lock 0x7b18002d17e0. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b474adc0a 00007f7b50bed219 00007f7b47535a40 00007f7b475304e4 00007f7b47530455 00007f7b4e0183c7 00007f7b4a62a9a7 00007f7b4a63a0bf 00007f7b4a63a038
0709 12:22:54.668373 (+   251us) log.cc:844] Serialized 14 byte log entry
0709 12:22:54.668604 (+   231us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:54.668666 (+    62us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:22:54.668736 (+    70us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:54.668841 (+   105us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:54.678039 (+  9198us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":463101,"prepare.queue_time_us":499919,"prepare.run_cpu_time_us":81,"prepare.run_wall_time_us":84,"raft.queue_time_us":8513,"raft.run_cpu_time_us":108,"raft.run_wall_time_us":5438,"spinlock_wait_cycles":52103680,"thread_start_us":386841,"threads_started":1,"wal-append.queue_time_us":433244}
I20260709 12:22:54.718086 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:54.671386 22777 raft_consensus.cc:493] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:54.718981 17762 raft_consensus.cc:3060] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:54.698024 22192 raft_consensus.cc:740] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:22:54.717326 22695 raft_consensus.cc:493] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:54.678139 22823 raft_consensus.cc:493] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:54.696413 22844 raft_consensus.cc:2804] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:22:54.725446 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:54.725880 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:54.726128 18364 raft_consensus.cc:3060] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:54.727041 22844 raft_consensus.cc:493] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:22:54.743709 17729 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:54.764153 17729 leader_election.cc:304] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:22:54.732825 22845 raft_consensus.cc:2804] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:22:54.706918 22840 raft_consensus.cc:515] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:54.736649 22619 leader_election.cc:290] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 12 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:54.769330 22652 consensus_queue.cc:1048] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
W20260709 12:22:54.769961 22847 log.cc:927] Time spent T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.158s	user 0.000s	sys 0.000s
I20260709 12:22:54.766057 22745 consensus_queue.cc:1048] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.020s
I20260709 12:22:54.768224 22360 raft_consensus.cc:2804] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:22:54.768097 22653 consensus_queue.cc:1048] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.021s
I20260709 12:22:54.851231 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:54.852084 17836 raft_consensus.cc:2393] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 15: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:22:54.732587 22823 raft_consensus.cc:515] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:54.854341 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:54.855062 17758 raft_consensus.cc:2368] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 14. Current term is 15.
W20260709 12:22:54.855249 22841 log.cc:927] Time spent T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.568s	user 0.001s	sys 0.000s
I20260709 12:22:54.856124 22729 raft_consensus.cc:493] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:54.856694 22729 raft_consensus.cc:3060] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:22:54.857103 18374 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59886 (request call id 1455) took 1344 ms. Trace:
I20260709 12:22:54.857539 18374 rpcz_store.cc:276] 0709 12:22:53.512225 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:54.088605 (+576380us) service_pool.cc:224] Handling call
0709 12:22:54.088919 (+   314us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:54.209547 (+120628us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:54.209593 (+    46us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:54.210094 (+   501us) log.cc:844] Serialized 26 byte log entry
0709 12:22:54.781631 (+571537us) spinlock_profiling.cc:243] Waited 3.17 ms on lock 0x7b18003d4500. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b4a1c84bc 00007f7b4751648b 00007f7b503709a9 00007f7b4753269b 00007f7b4753093f 00007f7b47532c02 00007f7b4a1ba20b 00007f7b4a1bfd23 00007f7b4a1c00d1
0709 12:22:54.781807 (+   176us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:22:54.781850 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:54.781940 (+    90us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:54.854626 (+ 72686us) spinlock_profiling.cc:243] Waited 31.4 ms on lock 0x7b4400022b4c. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b47459c85 00007f7b4745a6a2 00007f7b4a1cc1db 00007f7b4a1c7aaf 00007f7b4a1bb86e 00007f7b4a1babb6 00007f7b4a1ba7d8 00007f7b4a1c4221 00007f7b4a1c41d9 00007f7b4a1c4169
0709 12:22:54.856871 (+  2245us) raft_consensus.cc:1685] finished
0709 12:22:54.856895 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:54.857036 (+   141us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4042,"raft.queue_time_us":12023,"raft.run_cpu_time_us":189,"raft.run_wall_time_us":9393,"spinlock_wait_cycles":79417472,"thread_start_us":125072,"threads_started":2,"wal-append.queue_time_us":26529}
I20260709 12:22:54.731422 22695 raft_consensus.cc:515] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:22:54.771852 22831 log.cc:927] Time spent T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.531s	user 0.004s	sys 0.000s
W20260709 12:22:54.773454 17688 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37402 (request call id 2715) took 3660 ms (client timeout 1792 ms). Trace:
I20260709 12:22:54.777410 18367 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59886 (request call id 1456) took 1265 ms. Trace:
I20260709 12:22:54.779764 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:54.880270 18361 raft_consensus.cc:3060] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:22:54.881883 17838 raft_consensus.cc:1240] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 6. Current term is 7. Ops: []
W20260709 12:22:54.786475 17730 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:54.795367 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:54.798568 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:22:54.879266 17688 rpcz_store.cc:269] 0709 12:22:51.112582 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:51.112707 (+   125us) service_pool.cc:224] Handling call
0709 12:22:54.773267 (+3660560us) spinlock_profiling.cc:243] Waited 2.65 s on lock 0x7b54001aa754. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:22:54.773379 (+   112us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":6089027968}
I20260709 12:22:54.800319 17762 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 12.
I20260709 12:22:54.800876 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:54.749652 22844 raft_consensus.cc:3060] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:54.801616 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:54.879935 18367 rpcz_store.cc:276] 0709 12:22:53.512430 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:54.103627 (+591197us) service_pool.cc:224] Handling call
0709 12:22:54.119456 (+ 15829us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:54.200482 (+ 81026us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:54.777163 (+576681us) spinlock_profiling.cc:243] Waited 80.3 ms on lock 0x7b54005de8d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:22:54.777344 (+   181us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":184587008}
I20260709 12:22:54.802198 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:54.804838 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:54.790673 22625 leader_election.cc:290] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 14 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:54.805085 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:54.817453 22590 raft_consensus.cc:515] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:54.818275 18333 leader_election.cc:304] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:22:54.821430 22755 raft_consensus.cc:493] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:54.822459 22723 raft_consensus.cc:493] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:54.823211 22738 raft_consensus.cc:493] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:54.825807 18371 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59886 (request call id 1451) took 1400 ms. Trace:
I20260709 12:22:54.826159 18364 raft_consensus.cc:2468] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 10.
I20260709 12:22:54.829149 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:54.829509 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:54.901882 18365 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59902 (request call id 1259) took 1604 ms. Trace:
I20260709 12:22:54.902261 18365 rpcz_store.cc:276] 0709 12:22:53.297658 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:53.305942 (+  8284us) service_pool.cc:224] Handling call
0709 12:22:54.901688 (+1595746us) spinlock_profiling.cc:243] Waited 148 ms on lock 0x7b540051aed4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:22:54.901827 (+   139us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":340272896}
I20260709 12:22:54.829027 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:54.846899 17832 raft_consensus.cc:1240] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 12. Current term is 13. Ops: []
I20260709 12:22:54.765894 22749 leader_election.cc:290] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 14 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:54.833000 22626 raft_consensus.cc:493] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:22:54.849656 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:54.846421 22360 raft_consensus.cc:493] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:54.725369 22777 raft_consensus.cc:515] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:22:54.862372 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 51515.5us
I20260709 12:22:54.859722 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:54.827638 17652 leader_election.cc:304] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:54.975955 18374 raft_consensus.cc:2468] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 8.
W20260709 12:22:54.976406 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 148827us
I20260709 12:22:54.893101 17801 leader_election.cc:304] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:22:54.977850 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 84863.1us
I20260709 12:22:54.979704 22818 raft_consensus.cc:2749] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:22:54.979822 22840 leader_election.cc:290] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 7 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:54.980607 22840 raft_consensus.cc:2749] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:22:54.980556 22818 consensus_queue.cc:1059] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 12.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:22:54.981926 22607 raft_consensus.cc:3055] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:22:54.982344 22607 raft_consensus.cc:740] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:22:54.983157 22607 consensus_queue.cc:260] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.8, Last appended by leader: 8, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:54.988135 22695 leader_election.cc:290] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:54.992345 18365 raft_consensus.cc:3055] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:22:54.992941 18365 raft_consensus.cc:740] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:22:54.994077 18365 consensus_queue.cc:260] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 10.10, Last appended by leader: 10, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:22:54.897203 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 79011.7us
I20260709 12:22:54.996531 22625 raft_consensus.cc:493] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:54.996708 18333 leader_election.cc:304] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:22:54.997274 22625 raft_consensus.cc:515] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:54.899618 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:54.900705 18371 rpcz_store.cc:276] 0709 12:22:53.425081 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:53.759509 (+334428us) service_pool.cc:224] Handling call
0709 12:22:54.825223 (+1065714us) spinlock_profiling.cc:243] Waited 709 ms on lock 0x7b5400305f54. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:22:54.825376 (+   153us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1631834368}
I20260709 12:22:54.998164 22727 raft_consensus.cc:2749] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:22:54.999372 18365 raft_consensus.cc:3060] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:54.902987 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:54.904932 17762 raft_consensus.cc:1240] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 14. Current term is 15. Ops: [14.13-14.13]
I20260709 12:22:55.002065 18364 raft_consensus.cc:3060] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:22:54.905400 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:54.854367 22823 leader_election.cc:290] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:22:54.947463 17803 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:54.788205 22192 consensus_queue.cc:260] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:54.896857 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:54.997648 18333 leader_election.cc:304] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:22:54.838200 17834 raft_consensus.cc:1240] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 13. Current term is 14. Ops: []
I20260709 12:22:54.899513 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:55.007536 17836 raft_consensus.cc:1240] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 11. Current term is 12. Ops: []
I20260709 12:22:55.029914 18333 leader_election.cc:304] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:22:55.023293 22755 raft_consensus.cc:515] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:55.031330 17689 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37396 (request call id 2393) took 1262 ms. Trace:
I20260709 12:22:55.032418 18367 raft_consensus.cc:1240] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 13. Current term is 14. Ops: [13.10-13.10]
I20260709 12:22:55.032868 17834 raft_consensus.cc:1275] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20260709 12:22:55.033039 18367 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59886 (request call id 1459) took 1471 ms. Trace:
I20260709 12:22:55.031801 17689 rpcz_store.cc:276] 0709 12:22:53.768930 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:54.191431 (+422501us) service_pool.cc:224] Handling call
0709 12:22:54.191760 (+   329us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:54.674466 (+482706us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:55.031040 (+356574us) spinlock_profiling.cc:243] Waited 478 ms on lock 0x7b54001aac54. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:22:55.031263 (+   223us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1098446208}
I20260709 12:22:55.035565 18374 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 1588) took 1403 ms. Trace:
I20260709 12:22:55.036087 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:55.038986 17759 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36572 (request call id 864) took 1287 ms. Trace:
I20260709 12:22:55.039343 17759 rpcz_store.cc:276] 0709 12:22:53.751474 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:53.754665 (+  3191us) service_pool.cc:224] Handling call
0709 12:22:53.793936 (+ 39271us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:54.612436 (+818500us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:55.038778 (+426342us) spinlock_profiling.cc:243] Waited 816 ms on lock 0x7b54000fe6d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:22:55.038904 (+   126us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1877652736}
I20260709 12:22:55.042397 17682 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37396 (request call id 2388) took 1275 ms. Trace:
I20260709 12:22:55.042712 17682 rpcz_store.cc:276] 0709 12:22:53.767375 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:54.186025 (+418650us) service_pool.cc:224] Handling call
0709 12:22:54.186322 (+   297us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:54.193697 (+  7375us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:55.042226 (+848529us) spinlock_profiling.cc:243] Waited 2.79 ms on lock 0x7b54003480d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:22:55.042349 (+   123us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":6425344}
I20260709 12:22:55.009094 18361 raft_consensus.cc:2468] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 15.
I20260709 12:22:55.025846 17832 raft_consensus.cc:1275] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 14 index: 8. (index mismatch)
I20260709 12:22:54.896400 22808 raft_consensus.cc:493] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:54.905486 22734 raft_consensus.cc:2749] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
W20260709 12:22:55.075431 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:55.074633 17835 raft_consensus.cc:2393] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 14: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:22:55.033372 18367 rpcz_store.cc:276] 0709 12:22:53.561880 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:55.000498 (+1438618us) service_pool.cc:224] Handling call
0709 12:22:55.032204 (+ 31706us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:55.032808 (+   604us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:55.032989 (+   181us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:55.080354 18363 raft_consensus.cc:1240] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 10. Current term is 11. Ops: []
I20260709 12:22:55.081234 18363 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59902 (request call id 1274) took 1107 ms. Trace:
W20260709 12:22:55.093569 17654 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:55.000422 22360 raft_consensus.cc:3060] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:55.019527 22626 raft_consensus.cc:3060] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:55.081779 18363 rpcz_store.cc:276] 0709 12:22:53.973762 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:55.079646 (+1105884us) service_pool.cc:224] Handling call
0709 12:22:55.080088 (+   442us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:55.080927 (+   839us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:55.081159 (+   232us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:22:55.109364 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 79540.3us
I20260709 12:22:55.110458 18333 leader_election.cc:304] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:22:55.113529 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:55.114411 17832 raft_consensus.cc:3060] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:55.035938 18374 rpcz_store.cc:276] 0709 12:22:53.632558 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:54.976867 (+1344309us) service_pool.cc:224] Handling call
0709 12:22:54.977175 (+   308us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:55.024244 (+ 47069us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:55.024275 (+    31us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:55.035297 (+ 11022us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:22:55.035339 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:55.035397 (+    58us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:55.035514 (+   117us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":46715,"raft.queue_time_us":51339,"raft.run_cpu_time_us":123,"raft.run_wall_time_us":10435}
I20260709 12:22:55.100419 18364 raft_consensus.cc:2468] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 15.
W20260709 12:22:55.074838 22476 consensus_meta.cc:287] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.556s	user 0.010s	sys 0.001s
I20260709 12:22:55.123521 18333 leader_election.cc:304] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:22:55.131423 18333 leader_election.cc:400] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 14 election: Vote denied by peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837) with higher term. Message: Invalid argument: T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 14. Current term is 15.
I20260709 12:22:55.131814 18333 leader_election.cc:403] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 14 election: Cancelling election due to peer responding with higher term
W20260709 12:22:55.094837 22832 consensus_meta.cc:287] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.530s	user 0.009s	sys 0.001s
I20260709 12:22:55.135921 18365 raft_consensus.cc:1275] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 11 index: 10. (term mismatch)
I20260709 12:22:55.078821 22823 raft_consensus.cc:2804] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:22:55.028597 22625 leader_election.cc:290] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:55.171577 18365 pending_rounds.cc:85] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da: Aborting all ops after (but not including) 9
I20260709 12:22:55.175140 18365 pending_rounds.cc:107] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da: Aborting uncommitted NO_OP operation due to leader change: 10.10
I20260709 12:22:55.095384 22192 raft_consensus.cc:3060] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:55.175559 18365 raft_consensus.cc:2889] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:55.176126 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:55.012586 22536 raft_consensus.cc:515] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:54.935658 22738 raft_consensus.cc:515] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:54.938743 22723 raft_consensus.cc:515] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:55.114603 22834 raft_consensus.cc:2749] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:22:55.000595 22542 consensus_queue.cc:1059] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 13.12, Next index: 13, Last known committed idx: 11, Time since last communication: 0.169s
I20260709 12:22:55.109810 22383 raft_consensus.cc:493] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:55.044577 22491 consensus_queue.cc:1048] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.135s
I20260709 12:22:55.188473 18365 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 1587) took 1559 ms. Trace:
I20260709 12:22:55.175572 22727 raft_consensus.cc:2804] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:22:55.201444 18365 rpcz_store.cc:276] 0709 12:22:53.629323 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:54.969670 (+1340347us) service_pool.cc:224] Handling call
0709 12:22:54.992158 (+ 22488us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:55.187672 (+195514us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:55.188195 (+   523us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:55.169526 22734 raft_consensus.cc:2804] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:22:55.172883 22823 raft_consensus.cc:493] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:55.109123 22808 raft_consensus.cc:515] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:55.207115 17831 raft_consensus.cc:1275] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 13 index: 11. (index mismatch)
W20260709 12:22:55.193099 22700 consensus_meta.cc:287] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.576s	user 0.011s	sys 0.000s
I20260709 12:22:55.189955 22615 raft_consensus.cc:2749] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Leader election lost for term 14. Reason: Vote denied by peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837) with higher term. Message: Invalid argument: T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 14. Current term is 15.
I20260709 12:22:55.209419 18365 raft_consensus.cc:1240] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 14. Current term is 15. Ops: [13.12-14.13]
I20260709 12:22:55.197062 22625 raft_consensus.cc:493] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:55.172972 22476 raft_consensus.cc:515] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:55.205433 22727 raft_consensus.cc:493] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:55.208935 22305 raft_consensus.cc:3055] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:22:55.179807 22832 raft_consensus.cc:515] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:55.211848 22823 raft_consensus.cc:3060] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:55.212559 22695 raft_consensus.cc:493] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:55.250980 17833 raft_consensus.cc:2393] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 10: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:22:55.213472 22734 raft_consensus.cc:697] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 14 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:22:55.203756 22383 raft_consensus.cc:515] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:55.236145 22745 raft_consensus.cc:493] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:55.240633 22305 raft_consensus.cc:740] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:22:55.240087 22727 raft_consensus.cc:3060] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:55.261664 18367 raft_consensus.cc:3060] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Advancing to term 12
W20260709 12:22:55.277458 17687 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.511s	user 0.010s	sys 0.006s
I20260709 12:22:55.237689 22625 raft_consensus.cc:515] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:22:55.304898 22696 consensus_meta.cc:287] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.693s	user 0.005s	sys 0.007s
I20260709 12:22:55.261130 22695 raft_consensus.cc:515] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:55.263563 22745 raft_consensus.cc:515] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:55.314687 22676 raft_consensus.cc:493] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:55.328722 22755 leader_election.cc:290] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:55.361768 17832 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 12.
I20260709 12:22:55.364168 17832 raft_consensus.cc:1275] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20260709 12:22:55.369063 22723 leader_election.cc:290] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:55.371588 17757 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36572 (request call id 865) took 1616 ms. Trace:
I20260709 12:22:55.372130 17757 rpcz_store.cc:276] 0709 12:22:53.755057 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:53.778546 (+ 23489us) service_pool.cc:224] Handling call
0709 12:22:54.285142 (+506596us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:54.290886 (+  5744us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:55.371340 (+1080454us) spinlock_profiling.cc:243] Waited 506 ms on lock 0x7b54003e53d0. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a61f8ad 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:22:55.371510 (+   170us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1164090240}
I20260709 12:22:55.373124 22625 leader_election.cc:290] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:55.375003 22695 leader_election.cc:290] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:22:55.379628 22729 consensus_meta.cc:287] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.522s	user 0.008s	sys 0.000s
I20260709 12:22:55.344182 22676 raft_consensus.cc:515] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:22:55.420048 22833 consensus_meta.cc:287] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.781s	user 0.005s	sys 0.005s
W20260709 12:22:55.420261 22807 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 11 current term: 11 is higher.
I20260709 12:22:55.420524 22676 leader_election.cc:290] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:55.390192 22832 leader_election.cc:290] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 11 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:55.420656 22833 raft_consensus.cc:515] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:55.390676 22738 leader_election.cc:290] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:55.391633 22383 leader_election.cc:290] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:55.421945 18366 raft_consensus.cc:2393] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 15: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:22:55.422298 22729 raft_consensus.cc:515] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:55.396307 22808 leader_election.cc:290] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:55.336050 22734 consensus_queue.cc:237] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.10, Last appended by leader: 10, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:55.405624 22360 raft_consensus.cc:515] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:55.406347 22845 raft_consensus.cc:493] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:22:55.327767 22305 consensus_queue.cc:260] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 13.12, Last appended by leader: 12, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:22:55.408491 22672 consensus_meta.cc:287] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.730s	user 0.013s	sys 0.000s
I20260709 12:22:55.354745 22536 leader_election.cc:290] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 10 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:55.413529 22476 leader_election.cc:290] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:55.334043 22696 raft_consensus.cc:515] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:22:55.423800 17730 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:55.481985 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:55.484078 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:55.456522 18367 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 12.
I20260709 12:22:55.501152 17835 raft_consensus.cc:1240] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 9. Current term is 10. Ops: []
W20260709 12:22:55.441171 17802 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.551s	user 0.004s	sys 0.013s
W20260709 12:22:55.516868 17802 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.627s	user 0.006s	sys 0.015s
I20260709 12:22:55.517236 18366 raft_consensus.cc:2468] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 14.
W20260709 12:22:55.521066 18371 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.521s	user 0.010s	sys 0.009s
W20260709 12:22:55.521728 18371 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.522s	user 0.010s	sys 0.010s
W20260709 12:22:55.517396 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 627805us
I20260709 12:22:55.522223 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:22:55.522518 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 391331us
I20260709 12:22:55.524284 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:55.526669 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:55.527611 17757 raft_consensus.cc:2393] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 10: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:22:55.528635 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:55.535502 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:55.536466 18366 raft_consensus.cc:3060] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Advancing to term 10
W20260709 12:22:55.537072 17729 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:55.452867 22865 log.cc:927] Time spent T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.105s	user 0.002s	sys 0.002s
I20260709 12:22:55.543234 22722 raft_consensus.cc:493] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:55.541260 22738 raft_consensus.cc:493] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
W20260709 12:22:55.456449 22830 log.cc:927] Time spent T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 1.230s	user 0.005s	sys 0.005s
I20260709 12:22:55.544205 22738 raft_consensus.cc:515] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:55.468832 22777 leader_election.cc:290] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:55.545184 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:55.538061 22845 raft_consensus.cc:3060] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:55.546300 22738 leader_election.cc:290] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:55.537771 22673 raft_consensus.cc:493] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:55.546929 22673 raft_consensus.cc:515] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:55.542580 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:55.548187 17761 raft_consensus.cc:2468] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 12.
W20260709 12:22:55.388821 22069 consensus_meta.cc:287] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.715s	user 0.009s	sys 0.004s
I20260709 12:22:55.537447 22807 raft_consensus.cc:493] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:55.481236 22844 consensus_meta.cc:287] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.582s	user 0.009s	sys 0.001s
I20260709 12:22:55.502280 22518 raft_consensus.cc:493] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
W20260709 12:22:55.564914 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:55.565049 18374 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 1601) took 1016 ms. Trace:
I20260709 12:22:55.569679 18374 rpcz_store.cc:276] 0709 12:22:54.548282 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:55.538866 (+990584us) service_pool.cc:224] Handling call
0709 12:22:55.564421 (+ 25555us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:55.564652 (+   231us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:55.564688 (+    36us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:55.564745 (+    57us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:22:55.564788 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:55.564856 (+    68us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:55.564984 (+   128us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:22:55.575253 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 52349.3us
I20260709 12:22:55.577102 17802 leader_election.cc:304] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:55.426720 22607 raft_consensus.cc:3060] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:55.454501 22590 leader_election.cc:290] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 12 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:55.578786 17834 raft_consensus.cc:1240] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 10. Current term is 11. Ops: []
W20260709 12:22:55.563472 22842 log.cc:927] Time spent T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 1.003s	user 0.005s	sys 0.005s
I20260709 12:22:55.424443 22729 leader_election.cc:290] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 15 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:55.456224 22833 leader_election.cc:290] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 15 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:55.572458 17803 leader_election.cc:304] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:22:55.447770 22745 leader_election.cc:290] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:22:55.596112 18362 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.551s	user 0.007s	sys 0.019s
W20260709 12:22:55.597172 18362 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.552s	user 0.007s	sys 0.019s
I20260709 12:22:55.583168 18374 raft_consensus.cc:1240] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 14. Current term is 15. Ops: []
I20260709 12:22:55.479374 22727 raft_consensus.cc:515] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:55.595619 17686 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37396 (request call id 2394) took 1826 ms. Trace:
I20260709 12:22:55.602957 18374 raft_consensus.cc:1275] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20260709 12:22:55.596829 17801 leader_election.cc:304] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:55.429065 22626 raft_consensus.cc:515] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:55.395713 22823 raft_consensus.cc:515] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:55.592236 17729 leader_election.cc:304] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:22:55.607888 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 66926.2us
W20260709 12:22:55.527652 18333 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:55.602370 17686 rpcz_store.cc:276] 0709 12:22:53.769211 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:54.193414 (+424203us) service_pool.cc:224] Handling call
0709 12:22:54.203359 (+  9945us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:54.203594 (+   235us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 12:22:54.207020 (+  3426us) log.cc:844] Serialized 14 byte log entry
0709 12:22:54.852257 (+645237us) spinlock_profiling.cc:243] Waited 1.09 ms on lock 0x7b180013afe0. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b47515e07 00007f7b503709a9 00007f7b4753269b 00007f7b4753093f 00007f7b47532c02 00007f7b4a1ba20b 00007f7b4a1bfd23 00007f7b4a1c023a 00007f7b4a62aa2a
0709 12:22:55.595133 (+742876us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:55.595200 (+    67us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:22:55.595268 (+    68us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:55.595350 (+    82us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:55.595510 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":1611,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":35,"raft.queue_time_us":58689,"raft.run_cpu_time_us":201,"raft.run_wall_time_us":8525,"spinlock_wait_cycles":2510592,"thread_start_us":742529,"threads_started":2,"wal-append.queue_time_us":1122201}
I20260709 12:22:55.579180 22672 raft_consensus.cc:493] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:55.578817 22840 raft_consensus.cc:493] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:55.547828 22722 raft_consensus.cc:515] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:55.568742 22069 raft_consensus.cc:515] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:55.600975 22695 raft_consensus.cc:2804] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Leader election won for term 10
W20260709 12:22:55.616626 18333 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:55.610432 22557 raft_consensus.cc:2804] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:22:55.579519 22807 raft_consensus.cc:515] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:55.610769 22755 raft_consensus.cc:493] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:55.600544 22738 consensus_queue.cc:1059] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 13.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.024s
I20260709 12:22:55.617388 22777 raft_consensus.cc:2749] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:22:55.579490 22844 raft_consensus.cc:515] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:55.620752 17801 leader_election.cc:304] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:55.625646 18366 raft_consensus.cc:2468] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 10.
I20260709 12:22:55.695856 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:55.697273 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:55.698020 18366 raft_consensus.cc:2468] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 14.
I20260709 12:22:55.699007 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 9 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:22:55.565837 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 452965us
I20260709 12:22:55.700469 17652 leader_election.cc:304] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:55.616385 17729 leader_election.cc:304] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:55.605074 18374 raft_consensus.cc:1240] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 14. Current term is 15. Ops: []
I20260709 12:22:55.635567 18368 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59886 (request call id 1454) took 2123 ms. Trace:
I20260709 12:22:55.636094 18333 leader_election.cc:400] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 14 election: Vote denied by peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651) with higher term. Message: Invalid argument: T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 14. Current term is 15.
I20260709 12:22:55.594388 22518 raft_consensus.cc:515] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:55.622891 18367 raft_consensus.cc:1240] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 13. Current term is 14. Ops: []
W20260709 12:22:55.704332 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 68445.8us
I20260709 12:22:55.634544 22833 raft_consensus.cc:493] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:55.705075 18333 leader_election.cc:304] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:55.705472 17756 raft_consensus.cc:2393] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 14: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
W20260709 12:22:55.705837 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 89512.1us
I20260709 12:22:55.703554 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:55.609843 22866 raft_consensus.cc:2749] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:22:55.704018 18368 rpcz_store.cc:276] 0709 12:22:53.512016 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:54.051157 (+539141us) service_pool.cc:224] Handling call
0709 12:22:54.051550 (+   393us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:54.228803 (+177253us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:54.228845 (+    42us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:54.241127 (+ 12282us) log.cc:844] Serialized 26 byte log entry
0709 12:22:54.363472 (+122345us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:22:54.363556 (+    84us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:54.425767 (+ 62211us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:55.387449 (+961682us) spinlock_profiling.cc:243] Waited 2.58 ms on lock 0x7b4400022b4c. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b47459c85 00007f7b4745a6a2 00007f7b4a1cc1db 00007f7b4a1c7aaf 00007f7b4a1bb86e 00007f7b4a1babb6 00007f7b4a1ba7d8 00007f7b4a1c4221 00007f7b4a1c41d9 00007f7b4a1c4169
0709 12:22:55.628777 (+241328us) raft_consensus.cc:1685] finished
0709 12:22:55.628807 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:55.635500 (+  6693us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":59218,"raft.queue_time_us":65674,"raft.run_cpu_time_us":203,"raft.run_wall_time_us":9503,"spinlock_wait_cycles":5942144,"thread_start_us":221642,"threads_started":2,"wal-append.queue_time_us":234369}
I20260709 12:22:55.711023 22860 consensus_queue.cc:1059] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 13.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.006s
I20260709 12:22:55.710316 17652 leader_election.cc:304] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:55.711731 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 11 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:22:55.714686 17654 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.596s	user 0.004s	sys 0.016s
W20260709 12:22:55.759932 17654 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.642s	user 0.005s	sys 0.016s
I20260709 12:22:55.715240 18333 leader_election.cc:304] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:55.762674 18367 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59886 (request call id 1462) took 1012 ms. Trace:
I20260709 12:22:55.763141 18367 rpcz_store.cc:276] 0709 12:22:54.750240 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:55.565030 (+814790us) service_pool.cc:224] Handling call
0709 12:22:55.583819 (+ 18789us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:55.704681 (+120862us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:55.762370 (+ 57689us) spinlock_profiling.cc:243] Waited 34 ms on lock 0x7b540047ca54. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:22:55.762600 (+   230us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":78139776}
I20260709 12:22:55.716352 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:55.766000 18374 raft_consensus.cc:2393] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 10: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:22:55.717021 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:55.767972 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:55.769049 18368 raft_consensus.cc:2410] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 10 because replica has last-logged OpId of term: 8 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 7.
I20260709 12:22:55.770004 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:55.770718 18368 raft_consensus.cc:2468] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 13.
I20260709 12:22:55.718415 18366 raft_consensus.cc:2468] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 10.
I20260709 12:22:55.715205 17729 leader_election.cc:304] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:22:55.775506 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 60372.6us
W20260709 12:22:55.778985 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 63797.8us
I20260709 12:22:55.625414 22723 consensus_queue.cc:1059] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.089s
I20260709 12:22:55.694012 22305 raft_consensus.cc:3060] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:55.785856 17654 leader_election.cc:304] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:55.778091 17729 leader_election.cc:304] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:55.629267 22840 raft_consensus.cc:515] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:55.712517 22383 consensus_queue.cc:1059] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.008s
I20260709 12:22:55.632345 22590 consensus_queue.cc:1059] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 11.7, Next index: 8, Last known committed idx: 4, Time since last communication: 0.092s
I20260709 12:22:55.715432 22729 raft_consensus.cc:2749] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:22:55.626214 22672 raft_consensus.cc:515] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:55.792241 17764 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 12.
I20260709 12:22:55.694252 22745 raft_consensus.cc:493] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:55.800617 18368 raft_consensus.cc:1275] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 14 index: 12. (index mismatch)
I20260709 12:22:55.805015 17684 raft_consensus.cc:2393] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 14: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:22:55.719177 22817 consensus_queue.cc:1059] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.124s
I20260709 12:22:55.718353 22860 raft_consensus.cc:3055] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:22:55.713572 22878 raft_consensus.cc:493] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:55.713047 22218 raft_consensus.cc:2749] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 12:22:55.713413 22874 raft_consensus.cc:493] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:55.714138 22536 raft_consensus.cc:2804] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:22:55.714047 22557 raft_consensus.cc:697] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:22:55.655809 22755 raft_consensus.cc:515] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:55.718664 22853 consensus_queue.cc:1048] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.130s
I20260709 12:22:55.758837 22833 raft_consensus.cc:515] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:55.792245 22653 consensus_queue.cc:1048] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.069s
W20260709 12:22:55.813984 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 194653us
I20260709 12:22:55.791385 22416 raft_consensus.cc:2804] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Leader pre-election won for term 15
W20260709 12:22:55.895095 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 93068.5us
I20260709 12:22:55.718523 22495 raft_consensus.cc:2804] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:22:55.852005 18367 raft_consensus.cc:1240] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 14 LEADER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 13. Current term is 14. Ops: []
I20260709 12:22:55.842352 17838 raft_consensus.cc:1275] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 8 index: 7. Preceding OpId from leader: term: 10 index: 8. (index mismatch)
W20260709 12:22:55.840479 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 54729.8us
W20260709 12:22:55.818284 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 197609us
I20260709 12:22:55.966495 18368 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59886 (request call id 1468) took 1160 ms. Trace:
I20260709 12:22:55.965274 18364 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 1593) took 1423 ms. Trace:
I20260709 12:22:55.969004 22900 raft_consensus.cc:493] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:55.968755 22745 raft_consensus.cc:515] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:56.014024 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:56.016702 17835 raft_consensus.cc:2393] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 14: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:22:55.857353 22590 raft_consensus.cc:3055] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:22:55.792448 22757 raft_consensus.cc:493] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:55.800015 22734 raft_consensus.cc:2749] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 12:22:55.837577 22866 raft_consensus.cc:2804] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:22:55.972395 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:55.972514 22755 leader_election.cc:290] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:55.973356 22069 leader_election.cc:290] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 14 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:55.973846 22727 leader_election.cc:290] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 14 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:55.974382 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:55.974637 22807 leader_election.cc:290] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:55.975924 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:55.977284 22672 leader_election.cc:290] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:55.979998 22854 raft_consensus.cc:493] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:22:55.980484 22885 raft_consensus.cc:493] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:55.980706 18371 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59902 (request call id 1272) took 2007 ms. Trace:
I20260709 12:22:55.984015 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:55.984361 22673 leader_election.cc:290] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:55.996522 18364 rpcz_store.cc:276] 0709 12:22:54.541634 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:55.375570 (+833936us) service_pool.cc:224] Handling call
0709 12:22:55.376012 (+   442us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:55.376276 (+   264us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 12:22:55.636509 (+260233us) log.cc:844] Serialized 14 byte log entry
0709 12:22:55.964729 (+328220us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:55.964830 (+   101us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:22:55.964906 (+    76us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:55.964997 (+    91us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:55.965174 (+   177us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":47215,"prepare.queue_time_us":486903,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":39,"raft.queue_time_us":20392,"raft.run_cpu_time_us":130,"raft.run_wall_time_us":4373,"thread_start_us":587600,"threads_started":3}
I20260709 12:22:55.996152 18368 rpcz_store.cc:276] 0709 12:22:54.806300 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:55.789468 (+983168us) service_pool.cc:224] Handling call
0709 12:22:55.790761 (+  1293us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:55.966241 (+175480us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:55.966443 (+   202us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:56.011300 17838 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690"
dest_uuid: "e063541542a94969898868bf6eb5ce1b"
mode: GRACEFUL
new_leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
 from {username='slave'} at 127.0.0.1:59820
I20260709 12:22:55.801110 22668 raft_consensus.cc:493] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:55.978663 22518 leader_election.cc:290] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:56.041352 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:55.862850 22709 raft_consensus.cc:2804] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:22:56.077126 22866 raft_consensus.cc:493] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:55.890583 22700 consensus_queue.cc:1059] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 14.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.224s
W20260709 12:22:56.075831 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 81329.5us
I20260709 12:22:56.050107 17684 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37402 (request call id 2743) took 1221 ms. Trace:
I20260709 12:22:56.078431 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:55.801926 22725 consensus_queue.cc:1059] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 9.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.225s
I20260709 12:22:55.842720 22619 raft_consensus.cc:493] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:22:55.858423 22696 leader_election.cc:290] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 11 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:55.811167 22852 raft_consensus.cc:493] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
W20260709 12:22:55.993929 22693 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.544s	user 0.016s	sys 0.033s
I20260709 12:22:56.081970 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:55.791795 22376 consensus_queue.cc:1059] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 14.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.221s
I20260709 12:22:56.157610 22722 leader_election.cc:290] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:55.992231 22476 raft_consensus.cc:3055] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:22:56.195839 22476 raft_consensus.cc:740] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 11 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:22:56.047641 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
W20260709 12:22:56.196725 22876 log.cc:927] Time spent T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.339s	user 0.001s	sys 0.000s
I20260709 12:22:56.197696 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:56.198509 18363 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59902 (request call id 1277) took 2224 ms. Trace:
I20260709 12:22:56.075169 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:56.084275 17833 raft_consensus.cc:1275] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20260709 12:22:56.198882 18363 rpcz_store.cc:276] 0709 12:22:53.974627 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:55.109422 (+1134795us) service_pool.cc:224] Handling call
0709 12:22:55.109779 (+   357us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:55.128425 (+ 18646us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:55.128457 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:55.171121 (+ 42664us) log.cc:844] Serialized 26 byte log entry
0709 12:22:55.597811 (+426690us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:22:55.814612 (+216801us) log.cc:844] Serialized 14 byte log entry
0709 12:22:55.815708 (+  1096us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:55.815784 (+    76us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:56.198310 (+382526us) raft_consensus.cc:1685] finished
0709 12:22:56.198330 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:56.198454 (+   124us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":18031,"prepare.queue_time_us":322993,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":36,"raft.queue_time_us":18718,"raft.run_cpu_time_us":332,"raft.run_wall_time_us":55288,"thread_start_us":591842,"threads_started":2,"wal-append.queue_time_us":599900}
I20260709 12:22:56.183830 22557 consensus_queue.cc:237] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 13.12, Last appended by leader: 12, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:56.200977 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:56.201781 17833 raft_consensus.cc:2368] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 11. Current term is 12.
W20260709 12:22:56.202767 22884 log.cc:927] Time spent T 00000000000000000000000000000000 P 2193af31cb894ec9b3d65d939a2a6851: Append to log took a long time: real 0.295s	user 0.000s	sys 0.001s
I20260709 12:22:55.863061 22360 leader_election.cc:290] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 12 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:55.966715 22860 raft_consensus.cc:740] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:22:56.204330 22725 raft_consensus.cc:3055] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:22:56.204748 22725 raft_consensus.cc:740] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:22:56.205107 22860 consensus_queue.cc:260] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.10, Last appended by leader: 10, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:56.205525 22725 consensus_queue.cc:260] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:56.206456 22860 raft_consensus.cc:3060] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:56.206782 22725 raft_consensus.cc:3060] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:56.089166 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:22:56.208566 22872 log.cc:927] Time spent T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.214s	user 0.000s	sys 0.001s
W20260709 12:22:56.208997 22693 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.759s	user 0.018s	sys 0.036s
I20260709 12:22:55.993811 22416 raft_consensus.cc:493] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:55.844208 22517 consensus_queue.cc:1059] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 10.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.243s
I20260709 12:22:56.209532 22416 raft_consensus.cc:3060] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:22:56.026671 22808 raft_consensus.cc:493] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:56.209741 18362 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59886 (request call id 1457) took 2697 ms. Trace:
I20260709 12:22:56.074162 22590 raft_consensus.cc:740] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 11 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:22:56.018147 22536 raft_consensus.cc:697] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 15 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:22:56.210407 22852 raft_consensus.cc:515] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:56.210148 18362 rpcz_store.cc:276] 0709 12:22:53.512639 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:54.519197 (+1006558us) service_pool.cc:224] Handling call
0709 12:22:54.519544 (+   347us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:54.519867 (+   323us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:54.520030 (+   163us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:54.524595 (+  4565us) log.cc:844] Serialized 26 byte log entry
0709 12:22:55.044743 (+520148us) spinlock_profiling.cc:243] Waited 3.47 ms on lock 0x7b18003f7b40. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b47515e07 00007f7b503709a9 00007f7b4753269b 00007f7b4753093f 00007f7b47532c02 00007f7b4a1ba20b 00007f7b4a1bfd23 00007f7b4a1c00d1 00007f7b4a5dba0f
0709 12:22:55.607332 (+562589us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:22:55.607372 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:55.607446 (+    74us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:56.209449 (+602003us) raft_consensus.cc:1685] finished
0709 12:22:56.209468 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:56.209685 (+   217us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":2109,"raft.run_cpu_time_us":96,"raft.run_wall_time_us":97,"spinlock_wait_cycles":7987072,"thread_start_us":553490,"threads_started":1,"wal-append.queue_time_us":1395985}
I20260709 12:22:56.211125 22536 consensus_queue.cc:237] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.11, Last appended by leader: 11, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:56.212109 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:56.212639 22852 leader_election.cc:290] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:56.002978 22874 raft_consensus.cc:515] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:56.144630 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:56.144842 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:56.145275 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:56.105376 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:56.106935 18371 rpcz_store.cc:276] 0709 12:22:53.973231 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:54.998587 (+1025356us) service_pool.cc:224] Handling call
0709 12:22:54.998902 (+   315us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:54.999248 (+   346us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:54.999277 (+    29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:54.999705 (+   428us) log.cc:844] Serialized 26 byte log entry
0709 12:22:55.522233 (+522528us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:22:55.976635 (+454402us) log.cc:844] Serialized 14 byte log entry
0709 12:22:55.979287 (+  2652us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:55.979685 (+   398us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:55.979978 (+   293us) raft_consensus.cc:1685] finished
0709 12:22:55.979992 (+    14us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:55.980570 (+   578us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":9933,"prepare.queue_time_us":398315,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":44,"raft.queue_time_us":24876,"raft.run_cpu_time_us":209,"raft.run_wall_time_us":6235,"thread_start_us":930511,"threads_started":2,"wal-append.queue_time_us":503467}
I20260709 12:22:56.108379 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:56.108794 17838 raft_consensus.cc:606] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 14 LEADER]: Received request to transfer leadership to TS e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:56.118783 22832 raft_consensus.cc:493] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:56.118047 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 14 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:56.252049 22832 raft_consensus.cc:515] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:56.118780 18361 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 1598) took 1576 ms. Trace:
I20260709 12:22:56.253654 22840 leader_election.cc:290] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:56.253129 18361 rpcz_store.cc:276] 0709 12:22:54.542757 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:55.428539 (+885782us) service_pool.cc:224] Handling call
0709 12:22:55.428851 (+   312us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:55.429091 (+   240us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 12:22:55.695552 (+266461us) log.cc:844] Serialized 14 byte log entry
0709 12:22:56.118350 (+422798us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:56.118419 (+    69us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:22:56.118477 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:56.118573 (+    96us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:56.118715 (+   142us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":437467,"prepare.run_cpu_time_us":28,"prepare.run_wall_time_us":29,"raft.queue_time_us":3997,"raft.run_cpu_time_us":118,"raft.run_wall_time_us":16876,"thread_start_us":688253,"threads_started":2}
I20260709 12:22:56.256906 18361 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 1605) took 1127 ms. Trace:
I20260709 12:22:56.121340 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 11 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:56.097054 22890 raft_consensus.cc:493] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
W20260709 12:22:56.137495 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 152328us
I20260709 12:22:56.106002 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
W20260709 12:22:56.152946 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 157298us
I20260709 12:22:56.154378 17684 rpcz_store.cc:276] 0709 12:22:54.828672 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:54.828779 (+   107us) service_pool.cc:224] Handling call
0709 12:22:56.049811 (+1221032us) spinlock_profiling.cc:243] Waited 882 ms on lock 0x7b54001388d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:22:56.050021 (+   210us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2028393216}
I20260709 12:22:56.154122 17654 leader_election.cc:304] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:56.150693 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 14 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:56.130554 17803 leader_election.cc:304] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
W20260709 12:22:56.264694 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 134188us
I20260709 12:22:56.266644 17803 leader_election.cc:304] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:56.156610 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:56.157500 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:56.167686 22491 raft_consensus.cc:3055] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:22:56.268384 17762 raft_consensus.cc:2393] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 12: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:22:56.268672 17829 raft_consensus.cc:2393] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 10: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:22:55.996804 22878 raft_consensus.cc:515] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:56.181370 18365 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 1592) took 1639 ms. Trace:
I20260709 12:22:56.187918 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:56.130733 22069 consensus_queue.cc:1059] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.060s
I20260709 12:22:56.022717 22864 consensus_queue.cc:1059] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 11.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.190s
I20260709 12:22:56.194788 17837 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35042 (request call id 2320) took 1352 ms. Trace:
I20260709 12:22:56.194898 22383 raft_consensus.cc:3055] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:22:56.292757 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:56.287300 17837 rpcz_store.cc:276] 0709 12:22:54.841927 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:54.842125 (+   198us) service_pool.cc:224] Handling call
0709 12:22:54.842450 (+   325us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:55.263586 (+421136us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:55.263641 (+    55us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:55.264295 (+   654us) log.cc:844] Serialized 26 byte log entry
0709 12:22:55.628728 (+364433us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:22:55.628780 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:55.628873 (+    93us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:56.194530 (+565657us) raft_consensus.cc:1685] finished
0709 12:22:56.194560 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:56.194722 (+   162us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4785,"raft.queue_time_us":54973,"raft.run_cpu_time_us":94,"raft.run_wall_time_us":11458,"thread_start_us":734432,"threads_started":2,"wal-append.queue_time_us":898420}
I20260709 12:22:56.195962 22376 raft_consensus.cc:3055] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:22:56.200906 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:56.208917 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:56.296911 18363 raft_consensus.cc:3060] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:56.000901 22729 raft_consensus.cc:493] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:56.097709 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 9 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:55.852146 22763 raft_consensus.cc:2749] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:22:56.086534 22757 raft_consensus.cc:515] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:56.023816 22900 raft_consensus.cc:515] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:22:56.216883 18333 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:56.556589 18333 leader_election.cc:400] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 11 election: Vote denied by peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837) with higher term. Message: Invalid argument: T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 11. Current term is 12.
I20260709 12:22:56.557058 18333 leader_election.cc:403] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 11 election: Cancelling election due to peer responding with higher term
I20260709 12:22:55.921531 22626 leader_election.cc:290] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 10 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:56.146986 22866 raft_consensus.cc:3060] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:56.151161 22709 raft_consensus.cc:697] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:22:56.104077 22695 raft_consensus.cc:697] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 10 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:22:56.115302 17763 raft_consensus.cc:2393] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 10: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:22:56.216074 17685 raft_consensus.cc:2468] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 11.
I20260709 12:22:56.227497 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:56.228592 18364 raft_consensus.cc:1240] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 9. Current term is 10. Ops: []
I20260709 12:22:56.228890 22874 leader_election.cc:290] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:56.229171 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:56.230103 22833 leader_election.cc:290] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:56.156286 22495 raft_consensus.cc:697] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 12 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:22:56.231660 17838 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b"
dest_uuid: "e063541542a94969898868bf6eb5ce1b"
mode: GRACEFUL
new_leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
 from {username='slave'} at 127.0.0.1:59820
I20260709 12:22:56.121958 22885 raft_consensus.cc:515] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:56.184646 22855 consensus_queue.cc:1048] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.056s
I20260709 12:22:56.116127 22854 raft_consensus.cc:515] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:56.242344 22590 consensus_queue.cc:260] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:56.222604 22844 leader_election.cc:290] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 14 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:56.215703 22808 raft_consensus.cc:515] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:22:56.186282 22845 consensus_meta.cc:287] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.640s	user 0.008s	sys 0.003s
I20260709 12:22:56.246122 22745 leader_election.cc:290] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:56.251014 22619 raft_consensus.cc:515] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:56.121105 17835 raft_consensus.cc:3060] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:56.257670 18361 rpcz_store.cc:276] 0709 12:22:55.129491 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:56.255838 (+1126347us) service_pool.cc:224] Handling call
0709 12:22:56.256279 (+   441us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:56.256536 (+   257us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:56.256568 (+    32us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:56.256614 (+    46us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:22:56.256668 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:56.256739 (+    71us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:56.256857 (+   118us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:56.594599 22700 raft_consensus.cc:493] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:56.259524 17802 leader_election.cc:304] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
W20260709 12:22:56.260401 17655 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:56.610368 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 350896us
I20260709 12:22:56.273694 17803 leader_election.cc:304] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:56.275077 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
W20260709 12:22:56.612567 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 338981us
I20260709 12:22:56.274714 18365 rpcz_store.cc:276] 0709 12:22:54.541415 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:55.214135 (+672720us) service_pool.cc:224] Handling call
0709 12:22:55.241969 (+ 27834us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:55.242185 (+   216us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 12:22:55.613009 (+370824us) log.cc:844] Serialized 14 byte log entry
0709 12:22:56.170976 (+557967us) spinlock_profiling.cc:243] Waited 10.6 ms on lock 0x7b18003f46c0. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b4a1c84bc 00007f7b4751648b 00007f7b503709a9 00007f7b4753269b 00007f7b4753093f 00007f7b47532c02 00007f7b4a1ba20b 00007f7b4a1bfd23 00007f7b4a1c023a
0709 12:22:56.180188 (+  9212us) log.cc:844] Serialized 14 byte log entry
0709 12:22:56.180555 (+   367us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:56.180626 (+    71us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:22:56.180705 (+    79us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:56.180820 (+   115us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:56.181263 (+   443us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":589,"prepare.queue_time_us":570142,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":42,"raft.queue_time_us":6055,"raft.run_cpu_time_us":158,"raft.run_wall_time_us":33471,"spinlock_wait_cycles":55673600,"thread_start_us":671032,"threads_started":2,"wal-append.queue_time_us":251543}
I20260709 12:22:56.613310 17803 leader_election.cc:304] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:56.614650 18365 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59886 (request call id 1474) took 1147 ms. Trace:
I20260709 12:22:56.274469 17830 raft_consensus.cc:2393] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 14: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
W20260709 12:22:56.287746 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 133709us
I20260709 12:22:56.129401 22668 raft_consensus.cc:515] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:56.298488 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:56.299082 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:56.299394 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 15 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:56.275169 22491 raft_consensus.cc:740] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 12 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:22:56.286592 22890 raft_consensus.cc:515] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:56.307570 22729 raft_consensus.cc:515] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:56.293218 22383 raft_consensus.cc:740] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 14 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:22:56.633074 17683 raft_consensus.cc:1240] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 11. Current term is 12. Ops: []
I20260709 12:22:56.633359 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:56.281488 22840 raft_consensus.cc:2749] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 12:22:56.354185 22518 consensus_queue.cc:1059] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 12.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.121s
I20260709 12:22:56.295873 22376 raft_consensus.cc:740] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 14 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:22:56.281869 22676 raft_consensus.cc:2804] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:22:56.261551 22873 raft_consensus.cc:493] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:56.477839 18363 raft_consensus.cc:1275] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 10 index: 9. (term mismatch)
I20260709 12:22:56.479619 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:56.644874 18363 pending_rounds.cc:85] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da: Aborting all ops after (but not including) 8
I20260709 12:22:56.482164 22653 raft_consensus.cc:493] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:22:56.476925 22755 raft_consensus.cc:493] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:56.477527 22305 consensus_queue.cc:1048] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.028s
I20260709 12:22:56.500010 22069 raft_consensus.cc:493] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:22:56.501947 22763 raft_consensus.cc:493] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:56.512869 22607 consensus_meta.cc:287] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.917s	user 0.008s	sys 0.004s
I20260709 12:22:56.507961 22218 raft_consensus.cc:493] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:56.549270 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 252110us
I20260709 12:22:56.295729 22852 raft_consensus.cc:3060] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:22:56.569074 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 214388us
I20260709 12:22:56.538923 22897 raft_consensus.cc:493] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:56.592697 17688 raft_consensus.cc:2468] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 12.
I20260709 12:22:56.628859 22700 raft_consensus.cc:515] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:56.716486 18367 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59886 (request call id 1472) took 1250 ms. Trace:
I20260709 12:22:56.717065 18367 rpcz_store.cc:276] 0709 12:22:55.466252 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:56.596222 (+1129970us) service_pool.cc:224] Handling call
0709 12:22:56.610498 (+ 14276us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:56.610810 (+   312us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:56.610852 (+    42us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:56.624654 (+ 13802us) log.cc:844] Serialized 26 byte log entry
0709 12:22:56.713216 (+ 88562us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:22:56.713263 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:56.713415 (+   152us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:56.716225 (+  2810us) raft_consensus.cc:1685] finished
0709 12:22:56.716248 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:56.716411 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":8256,"raft.run_cpu_time_us":139,"raft.run_wall_time_us":5823,"thread_start_us":73788,"threads_started":1,"wal-append.queue_time_us":74185}
I20260709 12:22:56.594390 18364 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 1603) took 1479 ms. Trace:
I20260709 12:22:56.563092 22734 raft_consensus.cc:2749] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Leader election lost for term 11. Reason: Vote denied by peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837) with higher term. Message: Invalid argument: T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 11. Current term is 12.
W20260709 12:22:56.596549 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:56.719596 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 16 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:56.539243 22907 consensus_queue.cc:1048] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.104s
I20260709 12:22:56.597312 17838 raft_consensus.cc:606] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Received request to transfer leadership to TS e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:56.720137 18367 raft_consensus.cc:1240] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 10. Current term is 11. Ops: []
I20260709 12:22:56.720466 17838 raft_consensus.cc:614] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:22:56.721408 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:56.538841 22901 consensus_queue.cc:1048] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.153s
I20260709 12:22:56.720732 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:56.618516 17689 raft_consensus.cc:1240] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 11. Current term is 12. Ops: [11.7-11.7]
I20260709 12:22:56.619736 18361 raft_consensus.cc:1275] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 9 index: 8. (term mismatch)
I20260709 12:22:56.723910 18361 pending_rounds.cc:85] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da: Aborting all ops after (but not including) 7
I20260709 12:22:56.724328 18361 pending_rounds.cc:107] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da: Aborting uncommitted NO_OP operation due to leader change: 8.8
I20260709 12:22:56.724692 18361 raft_consensus.cc:2889] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:56.725862 18361 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59886 (request call id 1473) took 1259 ms. Trace:
I20260709 12:22:56.726343 18361 rpcz_store.cc:276] 0709 12:22:55.466562 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:56.609453 (+1142891us) service_pool.cc:224] Handling call
0709 12:22:56.609762 (+   309us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:56.725544 (+115782us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:56.725792 (+   248us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:56.625263 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:56.615105 18365 rpcz_store.cc:276] 0709 12:22:55.466858 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:56.613660 (+1146802us) service_pool.cc:224] Handling call
0709 12:22:56.614000 (+   340us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:56.614224 (+   224us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:56.614261 (+    37us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:56.614331 (+    70us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:22:56.614385 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:56.614457 (+    72us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:56.614582 (+   125us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:56.624346 17760 raft_consensus.cc:2368] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 11. Current term is 12.
I20260709 12:22:56.631632 17803 leader_election.cc:304] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:56.719439 18364 rpcz_store.cc:276] 0709 12:22:55.115376 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:56.228125 (+1112749us) service_pool.cc:224] Handling call
0709 12:22:56.228431 (+   306us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:56.593056 (+364625us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:56.594217 (+  1161us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:56.634727 22777 raft_consensus.cc:493] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:56.637809 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:56.728359 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:56.728245 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:56.729063 17756 raft_consensus.cc:2393] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 12: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:22:56.729332 17762 raft_consensus.cc:2393] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 14: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:22:56.636155 17802 leader_election.cc:304] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:56.728564 22673 raft_consensus.cc:2804] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:22:56.730185 22673 raft_consensus.cc:493] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:22:56.730252 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 94164us
I20260709 12:22:56.730620 22673 raft_consensus.cc:3060] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:56.731065 17802 leader_election.cc:304] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:56.716202 22897 raft_consensus.cc:515] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:56.732743 22672 raft_consensus.cc:2749] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 12:22:56.733407 22672 raft_consensus.cc:2749] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:22:56.733963 22672 consensus_queue.cc:1059] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 11.7, Next index: 8, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:22:56.734288 22897 leader_election.cc:290] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:56.641973 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:56.734961 17830 raft_consensus.cc:2468] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 13.
I20260709 12:22:56.500257 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:56.736445 17829 raft_consensus.cc:2393] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 14: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:22:56.645299 18363 pending_rounds.cc:107] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da: Aborting uncommitted NO_OP operation due to leader change: 8.9
I20260709 12:22:56.737172 17835 raft_consensus.cc:2468] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 12.
I20260709 12:22:56.737303 18363 raft_consensus.cc:2889] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:56.738247 18363 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59886 (request call id 1471) took 1272 ms. Trace:
I20260709 12:22:56.738406 17835 raft_consensus.cc:1240] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 10. Current term is 11. Ops: []
I20260709 12:22:56.738688 18363 rpcz_store.cc:276] 0709 12:22:55.465945 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:56.296493 (+830548us) service_pool.cc:224] Handling call
0709 12:22:56.296799 (+   306us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:56.737958 (+441159us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:56.738183 (+   225us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:56.641299 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:56.648689 18362 raft_consensus.cc:2468] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 11.
I20260709 12:22:56.626798 22896 raft_consensus.cc:2749] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:22:56.636616 22738 raft_consensus.cc:2749] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:22:56.752076 22852 raft_consensus.cc:2749] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:22:56.739926 22777 raft_consensus.cc:515] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:56.756484 22777 leader_election.cc:290] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:22:56.744525 22734 raft_consensus.cc:670] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da: failed to trigger leader election: Illegal state: only voting members can start elections: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:22:56.758217 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:56.758934 17830 raft_consensus.cc:2468] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 9.
I20260709 12:22:56.609838 22845 raft_consensus.cc:515] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:56.649722 22695 consensus_queue.cc:237] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 9.10, Last appended by leader: 10, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:56.757632 22734 raft_consensus.cc:493] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:56.761018 22734 raft_consensus.cc:515] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:56.761823 22845 leader_election.cc:290] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 12 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:22:56.648607 22895 log.cc:927] Time spent T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.160s	user 0.000s	sys 0.001s
I20260709 12:22:56.665202 22218 raft_consensus.cc:515] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:56.763800 22734 leader_election.cc:290] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:56.766114 17838 raft_consensus.cc:2468] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 9.
I20260709 12:22:56.766355 17686 raft_consensus.cc:1240] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 11. Current term is 12. Ops: []
W20260709 12:22:56.666363 22910 log.cc:927] Time spent T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.065s	user 0.000s	sys 0.000s
W20260709 12:22:56.669502 17764 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38786 (request call id 3017) took 1147 ms (client timeout 1519 ms). Trace:
W20260709 12:22:56.771718 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 88037.6us
I20260709 12:22:56.673719 22900 leader_election.cc:290] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:56.674260 22833 raft_consensus.cc:493] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:56.680158 22909 log.cc:927] Time spent T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.140s	user 0.000s	sys 0.001s
I20260709 12:22:56.658591 22757 leader_election.cc:290] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:56.682042 22912 consensus_queue.cc:1048] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:22:56.682806 22745 consensus_queue.cc:1059] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 11.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 12:22:56.683096 22890 leader_election.cc:290] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:56.658936 22069 raft_consensus.cc:515] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:56.684721 22360 consensus_queue.cc:1048] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 12:22:56.652271 22653 raft_consensus.cc:515] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:56.687248 22668 leader_election.cc:290] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:56.654892 22755 raft_consensus.cc:515] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:56.688033 22383 consensus_queue.cc:260] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.9, Last appended by leader: 9, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:56.658200 22763 raft_consensus.cc:515] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:56.664443 22709 consensus_queue.cc:237] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:56.690907 22619 leader_election.cc:290] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:56.642920 22676 raft_consensus.cc:493] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:56.643491 22898 raft_consensus.cc:493] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:56.810503 22676 raft_consensus.cc:3060] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:22:56.671500 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:56.564368 22696 raft_consensus.cc:493] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:56.812166 22696 raft_consensus.cc:515] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:56.729627 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:56.814574 22696 leader_election.cc:290] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:56.648316 22873 raft_consensus.cc:515] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:56.815080 17759 raft_consensus.cc:2468] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 10.
I20260709 12:22:56.810729 22898 raft_consensus.cc:515] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:56.816751 22873 leader_election.cc:290] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:56.723910 17836 raft_consensus.cc:2393] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 13: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:22:56.817499 22898 leader_election.cc:290] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:56.640319 17680 raft_consensus.cc:2468] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 10.
I20260709 12:22:56.628299 22749 raft_consensus.cc:2749] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
W20260709 12:22:56.728394 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 96805.2us
I20260709 12:22:56.820835 17803 leader_election.cc:304] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:22:56.749096 22729 leader_election.cc:290] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:22:56.750278 22902 log.cc:927] Time spent T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.097s	user 0.000s	sys 0.000s
I20260709 12:22:56.659960 22495 consensus_queue.cc:237] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.8, Last appended by leader: 8, Current term: 12, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:56.823177 18374 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59886 (request call id 1470) took 1939 ms. Trace:
I20260709 12:22:56.824530 22900 raft_consensus.cc:493] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:56.823621 18374 rpcz_store.cc:276] 0709 12:22:54.884257 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:55.787224 (+902967us) service_pool.cc:224] Handling call
0709 12:22:55.787575 (+   351us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:55.788444 (+   869us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:55.788485 (+    41us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:55.789043 (+   558us) log.cc:844] Serialized 26 byte log entry
0709 12:22:56.111961 (+322918us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:22:56.112016 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:56.112110 (+    94us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:56.822828 (+710718us) raft_consensus.cc:1685] finished
0709 12:22:56.822852 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:56.823109 (+   257us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":5042,"raft.run_cpu_time_us":128,"raft.run_wall_time_us":6345,"thread_start_us":254174,"threads_started":1,"wal-append.queue_time_us":806069}
I20260709 12:22:56.825205 22900 raft_consensus.cc:515] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:22:56.772042 17764 rpcz_store.cc:269] 0709 12:22:55.521770 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:55.521904 (+   134us) service_pool.cc:224] Handling call
0709 12:22:56.669148 (+1147244us) spinlock_profiling.cc:243] Waited 269 ms on lock 0x7b54000fc8d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:22:56.669339 (+   191us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":619103616}
I20260709 12:22:56.775141 17652 leader_election.cc:304] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:22:56.820643 22676 raft_consensus.cc:515] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:56.735209 22844 raft_consensus.cc:493] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:56.828034 22729 raft_consensus.cc:493] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:56.828397 22844 raft_consensus.cc:515] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:56.768133 22878 leader_election.cc:290] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:56.680521 22491 consensus_queue.cc:260] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 12.10, Last appended by leader: 10, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:56.830093 22491 raft_consensus.cc:3060] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:56.805285 22383 raft_consensus.cc:3060] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:22:56.830327 22885 leader_election.cc:290] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:56.831019 18367 raft_consensus.cc:1240] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 14. Current term is 15. Ops: []
W20260709 12:22:56.784798 22860 consensus_meta.cc:287] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.578s	user 0.014s	sys 0.000s
W20260709 12:22:56.827458 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 52373.6us
I20260709 12:22:56.827570 22619 raft_consensus.cc:2804] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:22:56.828492 22729 raft_consensus.cc:515] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:56.829371 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 12 candidate_status { last_received { term: 11 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:56.807263 22763 leader_election.cc:290] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:56.788697 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:56.837865 17803 leader_election.cc:304] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:56.841630 17652 leader_election.cc:304] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:22:56.850365 17803 leader_election.cc:304] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
W20260709 12:22:56.852061 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 214913us
W20260709 12:22:56.853731 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 121519us
I20260709 12:22:56.769593 22866 raft_consensus.cc:515] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:56.864382 17729 leader_election.cc:304] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:22:56.866029 17803 leader_election.cc:304] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:56.869006 17803 leader_election.cc:304] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:56.870792 18333 leader_election.cc:400] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 11 election: Vote denied by peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651) with higher term. Message: Invalid argument: T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 11. Current term is 12.
I20260709 12:22:56.865322 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:56.878242 17727 leader_election.cc:304] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:56.783901 22833 raft_consensus.cc:515] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:56.835709 22518 raft_consensus.cc:493] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:56.886893 17837 raft_consensus.cc:2368] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for earlier term 12. Current term is 13.
I20260709 12:22:56.793684 22673 raft_consensus.cc:515] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:56.770831 22376 consensus_queue.cc:260] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 14.12, Last appended by leader: 12, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:56.891341 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:56.891990 17680 raft_consensus.cc:2468] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 15.
I20260709 12:22:56.881134 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:56.882002 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:56.893280 17689 raft_consensus.cc:2468] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 13.
I20260709 12:22:56.885532 18333 leader_election.cc:304] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:56.882689 17835 raft_consensus.cc:1240] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 14. Current term is 15. Ops: []
I20260709 12:22:56.835120 22619 raft_consensus.cc:493] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:56.895812 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:56.889240 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:56.874045 22894 raft_consensus.cc:2749] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:22:56.896584 17683 raft_consensus.cc:2468] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 14.
I20260709 12:22:56.875627 22757 raft_consensus.cc:2804] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:22:56.896750 17833 raft_consensus.cc:2468] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 14.
I20260709 12:22:56.900277 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:56.872115 22696 raft_consensus.cc:2749] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:22:56.901068 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:56.901757 18363 raft_consensus.cc:2393] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 14: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:22:56.881161 17836 raft_consensus.cc:2393] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 12: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
W20260709 12:22:56.897620 17801 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.16.29.66:39651 timed out after 1.519s (SENT) [suppressed 18 similar messages]
I20260709 12:22:56.899848 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:56.883333 22896 raft_consensus.cc:2749] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 12:22:56.901659 17652 leader_election.cc:304] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:22:56.867409 22860 consensus_queue.cc:1059] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 10.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.030s
I20260709 12:22:56.918521 22408 raft_consensus.cc:2804] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:22:56.909101 17833 raft_consensus.cc:2393] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 13: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:22:56.887516 22840 raft_consensus.cc:2749] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:22:56.887140 22808 leader_election.cc:290] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:56.918915 22408 raft_consensus.cc:493] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:56.889941 22898 raft_consensus.cc:493] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:56.887336 22897 raft_consensus.cc:493] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:56.928115 22878 raft_consensus.cc:493] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:56.928746 22734 raft_consensus.cc:493] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:56.928833 22923 raft_consensus.cc:493] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:56.929774 17831 raft_consensus.cc:1240] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 10 LEADER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 9. Current term is 10. Ops: []
I20260709 12:22:56.907687 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:56.898717 22619 raft_consensus.cc:3060] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:56.933192 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:56.946651 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:22:56.947144 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 69257.3us
I20260709 12:22:56.948362 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:56.949038 17833 raft_consensus.cc:2468] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 11.
I20260709 12:22:56.950037 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:22:56.915766 22416 consensus_meta.cc:287] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.699s	user 0.010s	sys 0.001s
I20260709 12:22:56.889658 22900 leader_election.cc:290] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:56.900473 22192 raft_consensus.cc:2804] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:22:56.923583 22693 raft_consensus.cc:2749] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
W20260709 12:22:56.935643 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 71352.2us
I20260709 12:22:56.899747 22757 raft_consensus.cc:493] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:56.930428 22901 raft_consensus.cc:2804] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:22:56.937060 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:56.889318 22925 consensus_queue.cc:1048] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.060s
I20260709 12:22:56.950165 17803 leader_election.cc:304] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:56.892730 22518 raft_consensus.cc:515] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:56.946107 22890 raft_consensus.cc:2804] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:22:56.930238 22854 leader_election.cc:290] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:56.967701 18371 raft_consensus.cc:2393] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 12: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:22:56.933861 22926 raft_consensus.cc:493] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:56.933341 22808 raft_consensus.cc:493] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:56.981911 22495 consensus_queue.cc:1059] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 9.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
W20260709 12:22:56.982926 22924 log.cc:927] Time spent T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.052s	user 0.000s	sys 0.000s
I20260709 12:22:56.982653 22926 raft_consensus.cc:515] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:56.932940 22408 raft_consensus.cc:3060] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:22:56.942523 22727 raft_consensus.cc:493] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:56.981786 22808 raft_consensus.cc:515] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:56.951768 17836 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06"
dest_uuid: "e063541542a94969898868bf6eb5ce1b"
mode: GRACEFUL
new_leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
 from {username='slave'} at 127.0.0.1:59820
I20260709 12:22:56.983210 22495 raft_consensus.cc:3055] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:22:56.942888 22218 leader_election.cc:290] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:56.944831 22898 raft_consensus.cc:515] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:57.007184 17683 raft_consensus.cc:1275] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 14 index: 12. (index mismatch)
W20260709 12:22:57.004812 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 107198us
I20260709 12:22:56.959005 22192 raft_consensus.cc:493] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:57.032655 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:56.951145 22734 raft_consensus.cc:515] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:56.958915 22416 raft_consensus.cc:515] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:57.022348 18367 raft_consensus.cc:1275] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 11 index: 9. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
I20260709 12:22:56.990576 18368 raft_consensus.cc:1275] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 8 index: 7. Preceding OpId from leader: term: 10 index: 8. (index mismatch)
I20260709 12:22:57.022962 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 12 candidate_status { last_received { term: 11 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:57.045006 18365 raft_consensus.cc:3060] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Advancing to term 12
W20260709 12:22:56.998442 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 48330.7us
I20260709 12:22:57.025866 17836 raft_consensus.cc:606] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Received request to transfer leadership to TS e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:57.054239 17836 raft_consensus.cc:614] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:22:57.023543 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:57.056552 18361 raft_consensus.cc:2393] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 10: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:22:57.057896 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 11 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:56.939745 22897 raft_consensus.cc:515] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:57.062258 22897 leader_election.cc:290] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:57.063555 16501 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260709 12:22:56.966282 22873 consensus_queue.cc:1059] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 10.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.062s
I20260709 12:22:56.956166 22709 raft_consensus.cc:493] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:57.071916 22709 raft_consensus.cc:515] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:56.991750 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:56.946632 22878 raft_consensus.cc:515] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:57.078627 22878 leader_election.cc:290] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:57.080101 22878 raft_consensus.cc:493] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:57.080605 22878 raft_consensus.cc:515] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:57.075198 22709 leader_election.cc:290] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:57.033907 17683 raft_consensus.cc:2468] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 13.
I20260709 12:22:57.083563 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 12 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:57.084901 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:57.085786 17683 raft_consensus.cc:2468] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 9.
I20260709 12:22:57.086699 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:56.987217 18371 raft_consensus.cc:1240] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 9. Current term is 10. Ops: []
I20260709 12:22:56.985791 22926 leader_election.cc:290] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:56.973183 22757 raft_consensus.cc:3060] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:57.087709 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:57.021502 22900 raft_consensus.cc:493] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:57.023648 22695 raft_consensus.cc:2749] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:22:57.092978 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 11 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:57.097945 18371 raft_consensus.cc:2468] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 10.
I20260709 12:22:57.094059 17683 raft_consensus.cc:2468] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 14.
I20260709 12:22:57.099450 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:57.033064 22192 raft_consensus.cc:3060] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:57.111774 18365 raft_consensus.cc:2468] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 12.
I20260709 12:22:57.028066 22495 raft_consensus.cc:740] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:22:57.114095 22676 leader_election.cc:290] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 15 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:57.099923 22878 leader_election.cc:290] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:57.025243 22536 consensus_queue.cc:1048] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.073s
I20260709 12:22:57.125044 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:22:57.138386 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 208327us
I20260709 12:22:57.170679 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:57.203189 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:57.029999 22727 raft_consensus.cc:515] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:57.114624 22900 raft_consensus.cc:515] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:57.127156 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:22:57.293749 22929 log.cc:927] Time spent T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.232s	user 0.000s	sys 0.001s
I20260709 12:22:57.097487 22755 leader_election.cc:290] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:22:57.120720 22930 log.cc:927] Time spent T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.092s	user 0.000s	sys 0.000s
I20260709 12:22:57.120790 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:56.952785 22923 raft_consensus.cc:515] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:57.198309 17837 raft_consensus.cc:1240] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 12. Current term is 13. Ops: []
I20260709 12:22:57.122334 22913 raft_consensus.cc:3055] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:22:57.114250 22069 leader_election.cc:290] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:57.355665 17837 raft_consensus.cc:1275] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 13 index: 10. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
I20260709 12:22:57.357510 17681 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37402 (request call id 2753) took 1225 ms. Trace:
I20260709 12:22:57.033102 22668 raft_consensus.cc:493] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:57.127492 22723 raft_consensus.cc:3055] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:22:57.204771 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:57.126399 22898 leader_election.cc:290] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:57.120434 22878 raft_consensus.cc:493] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:57.228686 22901 raft_consensus.cc:697] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:22:57.248013 22727 leader_election.cc:290] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:57.248136 17829 raft_consensus.cc:1275] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 13 index: 11. Preceding OpId from leader: term: 15 index: 12. (index mismatch)
I20260709 12:22:57.248284 22416 leader_election.cc:290] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 15 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:57.250713 22673 leader_election.cc:290] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 14 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:57.179204 22536 raft_consensus.cc:3055] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:22:57.251748 22518 leader_election.cc:290] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:57.252462 22866 leader_election.cc:290] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 13 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:57.115146 22934 consensus_queue.cc:1048] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.201s
I20260709 12:22:57.253291 22734 leader_election.cc:290] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:22:57.254112 22928 log.cc:927] Time spent T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.067s	user 0.000s	sys 0.001s
I20260709 12:22:57.256280 22808 leader_election.cc:290] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:57.178434 22729 leader_election.cc:290] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:57.188151 22833 leader_election.cc:290] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:22:57.261525 22892 log.cc:927] Time spent T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 1.388s	user 0.009s	sys 0.007s
I20260709 12:22:57.126848 22192 raft_consensus.cc:515] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:22:57.189531 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 234304us
I20260709 12:22:57.292575 17654 leader_election.cc:304] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:22:57.293152 22874 consensus_queue.cc:1048] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.000s
W20260709 12:22:57.208052 18334 leader_election.cc:336] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Timed out: RequestConsensusVote RPC to 127.16.29.65:33071 timed out after 1.663s (SENT)
I20260709 12:22:57.405076 18334 leader_election.cc:304] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:22:57.405972 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 197964us
W20260709 12:22:57.407024 17685 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37402 (request call id 2747) took 1391 ms (client timeout 1698 ms). Trace:
W20260709 12:22:57.407430 17685 rpcz_store.cc:269] 0709 12:22:56.015685 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:56.015975 (+   290us) service_pool.cc:224] Handling call
0709 12:22:57.406797 (+1390822us) spinlock_profiling.cc:243] Waited 15.8 ms on lock 0x7b5400338954. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:22:57.406969 (+   172us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":36395264}
I20260709 12:22:57.408594 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:57.409112 17685 raft_consensus.cc:3060] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Advancing to term 13
I20260709 12:22:57.177793 17758 raft_consensus.cc:2393] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 15: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:22:57.410593 17758 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36572 (request call id 876) took 1258 ms. Trace:
I20260709 12:22:57.410908 17758 rpcz_store.cc:276] 0709 12:22:56.152546 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:56.299150 (+146604us) service_pool.cc:224] Handling call
0709 12:22:57.410413 (+1111263us) spinlock_profiling.cc:243] Waited 551 ms on lock 0x7b54002b6454. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:22:57.410537 (+   124us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1267341568}
I20260709 12:22:57.293816 22860 consensus_queue.cc:1048] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.002s
I20260709 12:22:57.412317 22729 raft_consensus.cc:2749] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:22:57.358017 17681 rpcz_store.cc:276] 0709 12:22:56.131797 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:56.131943 (+   146us) service_pool.cc:224] Handling call
0709 12:22:56.132326 (+   383us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:56.628794 (+496468us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 12:22:56.780462 (+151668us) log.cc:844] Serialized 14 byte log entry
0709 12:22:56.931767 (+151305us) log.cc:844] Serialized 14 byte log entry
0709 12:22:56.964963 (+ 33196us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:56.965022 (+    59us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:22:56.965073 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:57.357211 (+392138us) spinlock_profiling.cc:243] Waited 496 ms on lock 0x7b5400426dd4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:22:57.357276 (+    65us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:57.357438 (+   162us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":141126,"prepare.queue_time_us":275327,"prepare.run_cpu_time_us":69,"prepare.run_wall_time_us":71,"raft.queue_time_us":46292,"raft.run_cpu_time_us":382,"raft.run_wall_time_us":29259,"spinlock_wait_cycles":1141261056,"thread_start_us":269046,"threads_started":2,"wal-append.queue_time_us":311174}
I20260709 12:22:57.413007 22729 raft_consensus.cc:493] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:57.413441 22729 raft_consensus.cc:515] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:57.413775 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:57.415023 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
W20260709 12:22:57.353974 22491 consensus_meta.cc:287] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.519s	user 0.011s	sys 0.000s
I20260709 12:22:57.415833 17652 leader_election.cc:304] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:57.416108 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:57.417292 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:57.417954 22491 raft_consensus.cc:493] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:22:57.418339 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:57.418529 22491 raft_consensus.cc:515] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:57.420094 22652 raft_consensus.cc:493] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:22:57.385908 22536 raft_consensus.cc:740] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 10 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:22:57.366714 22723 raft_consensus.cc:740] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 14 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:22:57.401214 18363 raft_consensus.cc:2368] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for earlier term 10. Current term is 11.
W20260709 12:22:57.402640 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 110146us
I20260709 12:22:57.403343 22607 raft_consensus.cc:2804] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:22:57.437559 22607 raft_consensus.cc:697] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 12 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:22:57.438862 22607 consensus_queue.cc:237] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 11.12, Last appended by leader: 12, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:57.440709 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:57.442349 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:57.443107 17762 raft_consensus.cc:2468] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 14.
I20260709 12:22:57.354444 22913 raft_consensus.cc:740] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:22:57.418738 22495 consensus_queue.cc:260] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 9.10, Last appended by leader: 10, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:57.381657 22878 raft_consensus.cc:515] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:57.423028 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:57.457208 17764 raft_consensus.cc:2468] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 10.
I20260709 12:22:57.424592 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:57.425474 22823 leader_election.cc:290] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 11 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:57.426257 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:57.432664 18368 raft_consensus.cc:1240] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 9. Current term is 10. Ops: []
I20260709 12:22:57.434657 22860 raft_consensus.cc:493] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:57.446687 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 16 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:57.447145 17652 leader_election.cc:304] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:57.449653 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 11 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:57.421231 22854 raft_consensus.cc:2749] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 12:22:57.451541 17762 raft_consensus.cc:1275] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 8 index: 7. Preceding OpId from leader: term: 12 index: 9. (index mismatch)
I20260709 12:22:57.419495 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:57.453138 22940 consensus_queue.cc:1048] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 12:22:57.453303 22518 consensus_queue.cc:1059] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 12.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 12:22:57.379735 22900 leader_election.cc:290] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:57.415756 22729 leader_election.cc:290] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:57.461577 22901 consensus_queue.cc:237] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:57.463012 22536 consensus_queue.cc:260] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:57.512339 17681 raft_consensus.cc:2468] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 10.
I20260709 12:22:57.522176 18365 raft_consensus.cc:1275] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 13 index: 10. Preceding OpId from leader: term: 15 index: 11. (index mismatch)
I20260709 12:22:57.373512 22668 raft_consensus.cc:515] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:57.512692 18368 raft_consensus.cc:1275] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 10 index: 8. Preceding OpId from leader: term: 12 index: 9. (index mismatch)
W20260709 12:22:57.514115 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 47597us
W20260709 12:22:57.514206 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 67158.4us
W20260709 12:22:57.466140 22928 log.cc:927] Time spent T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.065s	user 0.000s	sys 0.000s
I20260709 12:22:57.515601 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:57.461494 22495 raft_consensus.cc:3060] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:22:57.510881 17727 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:57.526662 17685 raft_consensus.cc:2468] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 13.
W20260709 12:22:57.530524 17729 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.561s	user 0.022s	sys 0.031s
W20260709 12:22:57.530854 17729 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.561s	user 0.022s	sys 0.031s
W20260709 12:22:57.531304 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 562343us
I20260709 12:22:57.529597 17652 leader_election.cc:304] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:57.444332 22652 raft_consensus.cc:515] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:22:57.534957 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:57.537408 22652 leader_election.cc:290] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:22:57.539841 22408 consensus_meta.cc:287] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.556s	user 0.007s	sys 0.004s
I20260709 12:22:57.540352 22408 raft_consensus.cc:515] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:57.542522 22408 leader_election.cc:290] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 15 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:57.543471 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 12 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:57.544863 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 10 candidate_status { last_received { term: 9 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:22:57.545877 17729 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:57.546094 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:57.547395 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:57.548679 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:57.549114 17652 leader_election.cc:304] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:57.535315 22192 leader_election.cc:290] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 11 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:57.552520 17685 raft_consensus.cc:2468] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 10.
I20260709 12:22:57.547430 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:57.553369 17683 raft_consensus.cc:3060] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:22:57.553853 17681 raft_consensus.cc:3060] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:57.553532 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:57.521685 22874 raft_consensus.cc:2804] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:22:57.555104 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:57.556927 18334 leader_election.cc:304] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:22:57.557772 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:57.560634 17687 raft_consensus.cc:2393] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 13: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:22:57.560679 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:57.562601 17683 raft_consensus.cc:2468] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 15.
I20260709 12:22:57.562812 18334 leader_election.cc:304] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:57.561437 17763 raft_consensus.cc:2468] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 13.
I20260709 12:22:57.566233 18334 leader_election.cc:304] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:22:57.560495 22491 leader_election.cc:290] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:57.567224 22940 raft_consensus.cc:493] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:57.517985 22860 raft_consensus.cc:515] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:57.568502 17757 raft_consensus.cc:1240] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 14. Current term is 15. Ops: []
I20260709 12:22:57.568310 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:57.552438 22906 raft_consensus.cc:2749] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:22:57.581965 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:57.582106 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:57.582579 17757 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38776 (request call id 2319) took 1766 ms. Trace:
I20260709 12:22:57.582798 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:57.559775 22536 raft_consensus.cc:3060] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:22:57.611236 22903 log.cc:927] Time spent T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 1.014s	user 0.000s	sys 0.006s
I20260709 12:22:57.582739 22940 raft_consensus.cc:515] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:57.612232 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:57.558911 22874 raft_consensus.cc:493] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:22:57.631930 17681 raft_consensus.cc:2468] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 11.
W20260709 12:22:57.633574 22938 log.cc:927] Time spent T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.278s	user 0.000s	sys 0.002s
I20260709 12:22:57.634189 22695 raft_consensus.cc:493] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:57.634896 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:57.634773 22695 raft_consensus.cc:515] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:57.636008 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:57.638664 22940 leader_election.cc:290] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:57.639123 22695 leader_election.cc:290] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:22:57.639950 22944 log.cc:927] Time spent T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.121s	user 0.000s	sys 0.001s
I20260709 12:22:57.643270 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 16 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:57.614038 22875 raft_consensus.cc:3055] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:22:57.614240 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:57.585160 17757 rpcz_store.cc:276] 0709 12:22:55.815901 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:55.817245 (+  1344us) service_pool.cc:224] Handling call
0709 12:22:57.568157 (+1750912us) spinlock_profiling.cc:243] Waited 21.6 ms on lock 0x7b5800033398. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b474fbb71 00007f7b474fba22 00007f7b504148d9 00007f7b4f56eee0 00007f7b4f6380ab 00007f7b4f638131 00007f7b4f5f9fc0 00007f7b4f5e4929 00007f7b4f5e43d2 00007f7b48b5bbd4
0709 12:22:57.568348 (+   191us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:57.582296 (+ 13948us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:57.582518 (+   222us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":49776000}
I20260709 12:22:57.678834 17684 raft_consensus.cc:2468] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 15.
I20260709 12:22:57.616109 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:57.679741 17759 raft_consensus.cc:2468] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 14.
I20260709 12:22:57.680812 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:57.616890 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:22:57.617233 22619 consensus_meta.cc:287] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.659s	user 0.009s	sys 0.000s
I20260709 12:22:57.616819 22913 consensus_queue.cc:260] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:57.578150 22218 raft_consensus.cc:493] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:57.617558 17758 raft_consensus.cc:1240] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 LEADER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 14. Current term is 15. Ops: [14.13-14.13]
I20260709 12:22:57.699024 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:57.618180 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:57.619352 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 11 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:57.563679 22408 raft_consensus.cc:3060] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:57.577118 22900 raft_consensus.cc:2749] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:22:57.623174 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:57.553145 22763 consensus_queue.cc:1048] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.030s
I20260709 12:22:57.627447 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:57.628221 17760 raft_consensus.cc:2468] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 12.
I20260709 12:22:57.630775 17686 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37402 (request call id 2757) took 1372 ms. Trace:
I20260709 12:22:57.565419 22192 raft_consensus.cc:2749] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:22:57.612690 22476 consensus_queue.cc:260] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:22:57.724363 22476 raft_consensus.cc:3060] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:57.644009 22874 raft_consensus.cc:3060] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:22:57.645462 17831 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35040 (request call id 2575) took 1340 ms. Trace:
I20260709 12:22:57.728252 17831 rpcz_store.cc:276] 0709 12:22:56.305423 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:56.306344 (+   921us) service_pool.cc:224] Handling call
0709 12:22:56.306666 (+   322us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:56.947516 (+640850us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:57.645267 (+697751us) spinlock_profiling.cc:243] Waited 616 ms on lock 0x7b54000aafd4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:22:57.645408 (+   141us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1417544704}
I20260709 12:22:57.647028 22590 raft_consensus.cc:3060] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:57.649760 22844 leader_election.cc:290] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:57.552245 22757 raft_consensus.cc:515] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:57.660532 22722 raft_consensus.cc:493] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:57.661749 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:57.734726 17763 raft_consensus.cc:3060] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:57.622143 22923 leader_election.cc:290] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:22:57.664508 17804 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:57.736581 22722 raft_consensus.cc:515] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:57.738039 22757 leader_election.cc:290] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 11 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:57.665887 22833 raft_consensus.cc:493] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:57.738730 22722 leader_election.cc:290] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:57.739178 17837 raft_consensus.cc:1275] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 10 index: 8. Preceding OpId from leader: term: 12 index: 9. (index mismatch)
I20260709 12:22:57.739046 22833 raft_consensus.cc:515] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:22:57.734782 22864 consensus_queue.cc:1175] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" responder_term: 11 status { last_received { term: 11 index: 7 } last_committed_idx: 7 last_received_current_leader { term: 11 index: 7 } }
I20260709 12:22:57.671151 18366 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59886 (request call id 1469) took 2864 ms. Trace:
I20260709 12:22:57.741374 22833 leader_election.cc:290] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:57.742041 22866 raft_consensus.cc:993] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b: : Instructing follower e2909cff1da44ad387f96a8ae3aa5afb to start an election
I20260709 12:22:57.742507 22866 raft_consensus.cc:1081] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 14 LEADER]: Signalling peer e2909cff1da44ad387f96a8ae3aa5afb to start an election
W20260709 12:22:57.742923 22948 log.cc:927] Time spent T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.165s	user 0.001s	sys 0.000s
I20260709 12:22:57.712634 22619 raft_consensus.cc:515] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:57.744861 17834 raft_consensus.cc:1240] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 6. Current term is 7. Ops: []
I20260709 12:22:57.746345 22619 leader_election.cc:290] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 14 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:57.747141 17682 raft_consensus.cc:2393] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 14: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:22:57.671259 22305 raft_consensus.cc:3055] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:22:57.747844 22305 raft_consensus.cc:740] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:22:57.748369 22913 raft_consensus.cc:3060] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:57.748812 22305 consensus_queue.cc:260] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.9, Last appended by leader: 9, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:57.750130 22305 raft_consensus.cc:3060] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:57.751458 22860 leader_election.cc:290] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:22:57.687668 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 54355.7us
I20260709 12:22:57.753844 17801 leader_election.cc:304] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:57.579326 22729 raft_consensus.cc:2804] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:22:57.755425 22729 raft_consensus.cc:697] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 10 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:22:57.755837 22673 raft_consensus.cc:2804] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:22:57.756546 22729 consensus_queue.cc:237] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:57.702051 17689 raft_consensus.cc:2468] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 10.
I20260709 12:22:57.764402 22668 leader_election.cc:290] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:22:57.764918 17728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:57.740873 18366 rpcz_store.cc:276] 0709 12:22:54.806960 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:55.785676 (+978716us) service_pool.cc:224] Handling call
0709 12:22:55.786038 (+   362us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:55.798818 (+ 12780us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:55.798864 (+    46us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:55.809711 (+ 10847us) log.cc:844] Serialized 26 byte log entry
0709 12:22:56.257140 (+447429us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:22:56.285418 (+ 28278us) log.cc:844] Serialized 14 byte log entry
0709 12:22:56.286227 (+   809us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:56.286337 (+   110us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:56.769075 (+482738us) spinlock_profiling.cc:243] Waited 2.34 ms on lock 0x7b18004908a0. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b47533522 00007f7b475357c1 00007f7b47535779 00007f7b47535709 00007f7b475356d1 00007f7b475349cd 0000000000388634 0000000000388479 00007f7b4751683a
0709 12:22:56.806587 (+ 37512us) spinlock_profiling.cc:243] Waited 4.17 ms on lock 0x7b4800220a58. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4f549aa1 00007f7b4f540b37 00007f7b4f5409ca 00007f7b4f63826f 00007f7b4f637f37 00007f7b4f64c55c 00007f7b4f64c508 00007f7b4f64c488 00007f7b4f64c443 00007f7b4f64b6ec 00007f7b4e575a96
0709 12:22:57.610674 (+804087us) spinlock_profiling.cc:243] Waited 3.94 ms on lock 0x7b4400022b4c. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b47459c85 00007f7b4745a6a2 00007f7b4a1cc1db 00007f7b4a1c7aaf 00007f7b4a1bb86e 00007f7b4a1babb6 00007f7b4a1ba7d8 00007f7b4a1c4221 00007f7b4a1c41d9 00007f7b4a1c4169
0709 12:22:57.670917 (+ 60243us) raft_consensus.cc:1685] finished
0709 12:22:57.670945 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:57.671084 (+   139us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":12369,"prepare.queue_time_us":21635,"prepare.run_cpu_time_us":81,"prepare.run_wall_time_us":7195,"raft.queue_time_us":14958,"raft.run_cpu_time_us":5371,"raft.run_wall_time_us":1008377,"spinlock_wait_cycles":24019456,"thread_start_us":382278,"threads_started":1,"wal-append.queue_time_us":703846}
W20260709 12:22:57.776665 17728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:57.679113 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:57.623987 18364 raft_consensus.cc:2468] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 14.
I20260709 12:22:57.613829 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:57.677640 22653 leader_election.cc:290] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:57.717245 17686 rpcz_store.cc:276] 0709 12:22:56.258691 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:56.258835 (+   144us) service_pool.cc:224] Handling call
0709 12:22:56.286934 (+ 28099us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:56.771504 (+484570us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:57.630532 (+859028us) spinlock_profiling.cc:243] Waited 479 ms on lock 0x7b54004263d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:22:57.630709 (+   177us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1102128512}
I20260709 12:22:57.677098 22875 raft_consensus.cc:740] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
W20260709 12:22:57.802484 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 48698.7us
I20260709 12:22:57.806020 17801 leader_election.cc:304] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:57.816627 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 15 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:57.818204 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:57.818897 17686 raft_consensus.cc:2468] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 12.
I20260709 12:22:57.756585 22673 raft_consensus.cc:493] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:57.715402 22218 raft_consensus.cc:515] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:22:57.788888 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:57.820042 17729 leader_election.cc:304] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:57.789449 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:57.789907 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:57.789644 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:57.712669 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:57.797688 18364 raft_consensus.cc:1275] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 13 index: 11. Preceding OpId from leader: term: 15 index: 12. (index mismatch)
I20260709 12:22:57.799476 18366 raft_consensus.cc:1275] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 11 index: 11. Preceding OpId from leader: term: 15 index: 13. (index mismatch)
I20260709 12:22:57.722666 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:57.820935 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:57.821094 22901 raft_consensus.cc:493] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:22:57.822583 22408 raft_consensus.cc:2749] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:22:57.815174 22866 consensus_queue.cc:1059] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 13.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.014s
I20260709 12:22:57.833029 17683 raft_consensus.cc:2468] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 10.
I20260709 12:22:57.833945 18364 raft_consensus.cc:1240] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 13. Current term is 14. Ops: [13.11-13.11]
I20260709 12:22:57.848599 17728 leader_election.cc:304] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:22:57.837075 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:57.828920 17759 raft_consensus.cc:2468] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 14.
W20260709 12:22:57.850394 17833 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.677s	user 0.005s	sys 0.009s
I20260709 12:22:57.838410 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:22:57.852375 17833 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.679s	user 0.005s	sys 0.009s
I20260709 12:22:57.849828 17680 raft_consensus.cc:2468] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 14.
I20260709 12:22:57.835443 22755 raft_consensus.cc:493] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:57.872742 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 323696us
I20260709 12:22:57.872565 22755 raft_consensus.cc:515] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:57.871742 17763 raft_consensus.cc:2468] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 13.
I20260709 12:22:57.873317 22652 consensus_queue.cc:1048] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 12:22:57.874068 17759 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690"
dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
 from {username='slave'} at 127.0.0.1:38786
I20260709 12:22:57.967000 17759 raft_consensus.cc:493] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:22:57.967500 17759 raft_consensus.cc:3060] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:22:57.874651 22925 consensus_queue.cc:1048] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 12:22:57.875695 17688 raft_consensus.cc:1240] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 12. Current term is 13. Ops: [12.10-12.10]
I20260709 12:22:57.875248 22940 consensus_queue.cc:1059] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 11.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:22:57.875171 22518 consensus_queue.cc:1048] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:22:57.879676 17728 leader_election.cc:304] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:22:57.881150 18364 raft_consensus.cc:1275] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20260709 12:22:58.025661 17803 leader_election.cc:304] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:57.881711 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
W20260709 12:22:58.026273 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 146708us
I20260709 12:22:58.026903 17685 raft_consensus.cc:2393] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 14: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:22:57.882578 17683 raft_consensus.cc:1240] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 14. Current term is 15. Ops: []
I20260709 12:22:58.027025 17728 leader_election.cc:304] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:57.818989 22898 raft_consensus.cc:2804] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:22:58.030015 17685 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37396 (request call id 2410) took 1051 ms. Trace:
I20260709 12:22:58.030458 17685 rpcz_store.cc:276] 0709 12:22:56.978995 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:57.825599 (+846604us) service_pool.cc:224] Handling call
0709 12:22:58.029925 (+204326us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:57.826589 22899 raft_consensus.cc:3055] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:22:57.827203 22673 raft_consensus.cc:3060] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:57.920104 17680 raft_consensus.cc:1275] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20260709 12:22:57.923908 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:58.125679 22360 raft_consensus.cc:2749] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:22:57.885172 17801 leader_election.cc:304] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:57.963429 17652 leader_election.cc:304] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:57.824923 22722 raft_consensus.cc:493] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:58.008667 17688 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37396 (request call id 2413) took 1029 ms. Trace:
I20260709 12:22:58.007577 17761 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38776 (request call id 2317) took 2192 ms. Trace:
I20260709 12:22:57.914127 22491 raft_consensus.cc:2804] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:22:57.883081 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:58.027823 17683 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37396 (request call id 2406) took 1049 ms. Trace:
I20260709 12:22:58.156454 17683 rpcz_store.cc:276] 0709 12:22:56.978059 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:57.882107 (+904048us) service_pool.cc:224] Handling call
0709 12:22:57.882379 (+   272us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:58.027493 (+145114us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:58.027731 (+   238us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:22:58.143849 22959 log.cc:927] Time spent T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.077s	user 0.000s	sys 0.005s
I20260709 12:22:58.065356 22590 raft_consensus.cc:493] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:58.158103 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:57.883805 22866 consensus_queue.cc:1243] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer e2909cff1da44ad387f96a8ae3aa5afb log is divergent from this leader: its last log entry 15.13 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 12
I20260709 12:22:57.878049 22383 raft_consensus.cc:2804] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:22:57.933542 22875 consensus_queue.cc:260] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:57.871300 22901 raft_consensus.cc:515] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:58.129503 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:58.173849 18361 raft_consensus.cc:2393] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 15: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:22:58.143565 17728 leader_election.cc:304] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:22:58.146363 17685 raft_consensus.cc:1217] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 11 LEADER]: Deduplicated request from leader. Original: 8.8->[9.9-10.10]   Dedup: 10.10->[]
I20260709 12:22:58.174767 17685 raft_consensus.cc:1240] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 11 LEADER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 10. Current term is 11. Ops: [9.9-10.10]
I20260709 12:22:57.822539 22874 raft_consensus.cc:515] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:22:58.148530 18333 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:58.029950 22898 raft_consensus.cc:493] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:58.069181 22935 raft_consensus.cc:2749] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:22:58.152895 17687 raft_consensus.cc:2468] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 13.
I20260709 12:22:58.157793 22672 raft_consensus.cc:2804] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:22:57.934676 17686 raft_consensus.cc:2468] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 10.
W20260709 12:22:58.155476 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 270372us
W20260709 12:22:58.210841 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 67464.2us
I20260709 12:22:58.069566 18334 leader_election.cc:304] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:22:58.212247 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 142734us
I20260709 12:22:58.155593 17688 rpcz_store.cc:276] 0709 12:22:56.979680 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:57.875235 (+895555us) service_pool.cc:224] Handling call
0709 12:22:57.875502 (+   267us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:58.008310 (+132808us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:58.008595 (+   285us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:58.215129 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:58.216123 17688 raft_consensus.cc:2393] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 15: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:22:58.216874 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:22:58.136534 22960 log.cc:927] Time spent T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.069s	user 0.000s	sys 0.005s
W20260709 12:22:58.152316 22951 log.cc:927] Time spent T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.118s	user 0.000s	sys 0.000s
W20260709 12:22:58.155017 22536 consensus_meta.cc:287] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.530s	user 0.012s	sys 0.000s
I20260709 12:22:58.156046 17761 rpcz_store.cc:276] 0709 12:22:55.815266 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:55.816965 (+  1699us) service_pool.cc:224] Handling call
0709 12:22:55.971118 (+154153us) spinlock_profiling.cc:243] Waited 21.8 ms on lock 0x7b5800033398. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b474fbb71 00007f7b474fba22 00007f7b504148d9 00007f7b4f56eee0 00007f7b4f6380ab 00007f7b4f638131 00007f7b4f5f9fc0 00007f7b4f5e4929 00007f7b4f5e43d2 00007f7b48b5bbd4
0709 12:22:55.971319 (+   201us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:56.098591 (+127272us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 12:22:56.130929 (+ 32338us) log.cc:844] Serialized 14 byte log entry
0709 12:22:56.296000 (+165071us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:56.296051 (+    51us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:22:56.296094 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:58.007236 (+1711142us) spinlock_profiling.cc:243] Waited 127 ms on lock 0x7b54000fbed4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:22:58.007311 (+    75us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:58.007497 (+   186us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":42596,"prepare.queue_time_us":32757,"prepare.run_cpu_time_us":56,"prepare.run_wall_time_us":58,"raft.queue_time_us":17548,"raft.run_cpu_time_us":141,"raft.run_wall_time_us":7509,"spinlock_wait_cycles":342138368,"thread_start_us":148268,"threads_started":1,"wal-append.queue_time_us":377326,"wal-append.run_cpu_time_us":2509,"wal-append.run_wall_time_us":1285620}
W20260709 12:22:58.164808 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 201520us
I20260709 12:22:58.223601 17652 leader_election.cc:304] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:58.219079 22923 raft_consensus.cc:2749] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:22:58.224568 22954 raft_consensus.cc:2749] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:22:58.224756 22874 leader_election.cc:290] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 15 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:22:58.225540 22962 log.cc:927] Time spent T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.157s	user 0.001s	sys 0.000s
I20260709 12:22:58.225437 22874 consensus_queue.cc:1059] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:22:58.172614 22383 raft_consensus.cc:493] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:58.165290 17683 raft_consensus.cc:2393] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 12: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:22:58.226831 22383 raft_consensus.cc:3060] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:22:58.227504 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:58.228294 17683 raft_consensus.cc:2393] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 12: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:22:58.229007 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:58.229555 17683 raft_consensus.cc:2468] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 14.
I20260709 12:22:58.230286 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:58.231199 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 16 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:58.152180 17680 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37396 (request call id 2408) took 1173 ms. Trace:
I20260709 12:22:58.196609 17687 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37402 (request call id 2767) took 1128 ms. Trace:
I20260709 12:22:58.232167 17687 rpcz_store.cc:276] 0709 12:22:57.068760 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:57.870603 (+801843us) service_pool.cc:224] Handling call
0709 12:22:58.196542 (+325939us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:58.213137 18334 leader_election.cc:304] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:22:58.233455 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 11 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:58.234496 18334 leader_election.cc:304] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:58.221752 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:58.236405 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:58.235811 18334 leader_election.cc:304] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:22:58.238004 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:58.238449 17801 leader_election.cc:304] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:58.238767 17836 raft_consensus.cc:2393] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 7: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:22:58.226428 22874 raft_consensus.cc:3055] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:22:58.172878 22626 raft_consensus.cc:2749] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:22:58.232043 17680 rpcz_store.cc:276] 0709 12:22:56.978560 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:57.884037 (+905477us) service_pool.cc:224] Handling call
0709 12:22:57.884642 (+   605us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:58.151927 (+267285us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:58.152125 (+   198us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:58.240224 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 13 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:58.239871 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:58.159219 22491 raft_consensus.cc:493] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:58.241052 17760 raft_consensus.cc:2468] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 11.
I20260709 12:22:58.240947 18334 leader_election.cc:304] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:22:58.241128 17762 raft_consensus.cc:2410] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for term 13 because replica has last-logged OpId of term: 11 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 6.
I20260709 12:22:58.238507 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 16 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:58.242172 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:58.242759 17831 raft_consensus.cc:3060] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:58.241122 17834 raft_consensus.cc:2468] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 15.
I20260709 12:22:58.239447 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:58.241632 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:58.239440 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:58.248456 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:58.249608 17838 raft_consensus.cc:3060] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:22:58.149255 22899 raft_consensus.cc:740] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 13 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:22:58.252303 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 13 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:58.253232 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:58.257056 17837 raft_consensus.cc:3060] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:58.254427 17687 raft_consensus.cc:2393] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 11: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:22:58.218731 22755 leader_election.cc:290] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:58.255842 17758 raft_consensus.cc:2393] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 14: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:22:58.260795 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 10 candidate_status { last_received { term: 9 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:58.261883 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:58.251967 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:58.212001 22898 raft_consensus.cc:3060] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:58.262578 17836 raft_consensus.cc:3060] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Advancing to term 11
I20260709 12:22:58.242106 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:58.212703 22607 raft_consensus.cc:493] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:58.162542 22722 raft_consensus.cc:515] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:58.220458 22672 raft_consensus.cc:493] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:58.222493 22518 raft_consensus.cc:2749] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 12:22:58.224025 22965 raft_consensus.cc:2804] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:22:58.274377 17756 raft_consensus.cc:3060] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:22:58.275480 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 249891us
I20260709 12:22:58.173080 22590 raft_consensus.cc:515] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:58.276646 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:58.304196 17837 raft_consensus.cc:2468] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 14.
I20260709 12:22:58.304774 17763 raft_consensus.cc:2468] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 15.
W20260709 12:22:58.257292 22476 consensus_meta.cc:287] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.532s	user 0.016s	sys 0.004s
I20260709 12:22:58.279208 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:58.242015 22866 raft_consensus.cc:493] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:58.262007 22491 raft_consensus.cc:3060] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:58.288975 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:58.240394 22874 raft_consensus.cc:740] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:22:58.255403 22923 raft_consensus.cc:2749] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:22:58.225361 22954 consensus_queue.cc:1048] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.051s
I20260709 12:22:58.294724 22965 raft_consensus.cc:493] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:58.306051 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:58.307809 18365 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 1615) took 1020 ms. Trace:
I20260709 12:22:58.356935 18365 rpcz_store.cc:276] 0709 12:22:57.287520 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:57.287646 (+   126us) service_pool.cc:224] Handling call
0709 12:22:57.287963 (+   317us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:57.522660 (+234697us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:58.307468 (+784808us) spinlock_profiling.cc:243] Waited 234 ms on lock 0x7b54007602d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:22:58.307727 (+   259us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":537840896}
I20260709 12:22:58.313063 17835 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35042 (request call id 2348) took 1354 ms. Trace:
I20260709 12:22:58.334947 22517 raft_consensus.cc:3055] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:22:58.323010 17682 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37402 (request call id 2762) took 1443 ms. Trace:
I20260709 12:22:58.365105 17682 rpcz_store.cc:276] 0709 12:22:56.879997 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:56.880822 (+   825us) service_pool.cc:224] Handling call
0709 12:22:58.322715 (+1441893us) spinlock_profiling.cc:243] Waited 854 ms on lock 0x7b5400206f54. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:22:58.322922 (+   207us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1963832704}
I20260709 12:22:58.324165 17755 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38776 (request call id 2320) took 2508 ms. Trace:
I20260709 12:22:58.364077 17756 raft_consensus.cc:2468] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 11.
I20260709 12:22:58.348485 17836 raft_consensus.cc:2468] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 11.
I20260709 12:22:58.366439 17755 rpcz_store.cc:276] 0709 12:22:55.816179 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:55.816283 (+   104us) service_pool.cc:224] Handling call
0709 12:22:55.972410 (+156127us) spinlock_profiling.cc:243] Waited 22.7 ms on lock 0x7b5800033398. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b474fbb71 00007f7b474fba22 00007f7b504148d9 00007f7b4f56eee0 00007f7b4f6380ab 00007f7b4f638131 00007f7b4f5f9fc0 00007f7b4f5e4929 00007f7b4f5e43d2 00007f7b48b5bbd4
0709 12:22:56.676645 (+704235us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:56.676879 (+   234us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:56.676918 (+    39us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:56.676970 (+    52us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:22:56.677012 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:56.677075 (+    63us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:58.323946 (+1646871us) spinlock_profiling.cc:243] Waited 700 ms on lock 0x7b54000fc8d0. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a61f8ad 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:22:58.324093 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1662456832}
I20260709 12:22:58.324903 17681 raft_consensus.cc:1240] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 13. Current term is 14. Ops: []
I20260709 12:22:58.327454 22832 leader_election.cc:290] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:22:58.378821 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 203055us
I20260709 12:22:58.272867 22755 raft_consensus.cc:2804] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:22:58.332600 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:22:58.333768 17834 raft_consensus.cc:2410] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for term 13 because replica has last-logged OpId of term: 11 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 6.
I20260709 12:22:58.234889 22912 consensus_queue.cc:1059] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 14.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.060s
I20260709 12:22:58.275321 22218 leader_election.cc:290] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:58.287040 22672 raft_consensus.cc:3060] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:22:58.348819 17804 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:58.261834 22900 raft_consensus.cc:2749] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:22:58.400179 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:58.358218 17835 rpcz_store.cc:276] 0709 12:22:56.958878 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:57.298492 (+339614us) service_pool.cc:224] Handling call
0709 12:22:57.439901 (+141409us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:57.455793 (+ 15892us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:57.455825 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:57.456315 (+   490us) log.cc:844] Serialized 26 byte log entry
0709 12:22:57.712381 (+256066us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:22:57.712419 (+    38us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:57.712480 (+    61us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:57.757805 (+ 45325us) raft_consensus.cc:1685] finished
0709 12:22:57.757825 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:58.312685 (+554860us) spinlock_profiling.cc:243] Waited 141 ms on lock 0x7b540008cad0. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a61f8ad 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:22:58.312973 (+   288us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":305,"raft.run_cpu_time_us":129,"raft.run_wall_time_us":7755,"spinlock_wait_cycles":324254592,"thread_start_us":184344,"threads_started":1,"wal-append.queue_time_us":239715}
I20260709 12:22:58.315675 18363 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59918 (request call id 1610) took 1325 ms. Trace:
I20260709 12:22:58.401196 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:58.401276 18363 rpcz_store.cc:276] 0709 12:22:56.990427 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:56.991441 (+  1014us) service_pool.cc:224] Handling call
0709 12:22:58.315397 (+1323956us) spinlock_profiling.cc:243] Waited 324 ms on lock 0x7b54005730d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:22:58.315599 (+   202us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":746340096}
I20260709 12:22:58.363175 22965 raft_consensus.cc:3060] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:22:58.337494 22729 raft_consensus.cc:2804] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:22:58.368028 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:58.345176 22899 consensus_queue.cc:260] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.11, Last appended by leader: 11, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:22:58.376204 22953 log.cc:927] Time spent T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.209s	user 0.000s	sys 0.001s
I20260709 12:22:58.345940 22866 raft_consensus.cc:515] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:58.344084 22476 raft_consensus.cc:493] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:58.278488 22607 raft_consensus.cc:515] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:58.420200 22607 leader_election.cc:290] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:58.422840 22866 leader_election.cc:290] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:58.408489 22668 raft_consensus.cc:493] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:58.428408 22668 raft_consensus.cc:515] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:58.434216 22700 leader_election.cc:290] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:58.434980 22476 raft_consensus.cc:515] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:58.438367 22476 leader_election.cc:290] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:58.440399 17682 raft_consensus.cc:1275] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 9 index: 10. Preceding OpId from leader: term: 10 index: 11. (index mismatch)
I20260709 12:22:58.441390 22376 raft_consensus.cc:3060] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:22:58.377285 22590 leader_election.cc:290] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:58.444605 17688 raft_consensus.cc:1275] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 13 index: 10. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
I20260709 12:22:58.445925 17688 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44898 (request call id 987) took 1299 ms. Trace:
I20260709 12:22:58.446321 17688 rpcz_store.cc:276] 0709 12:22:57.147062 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:58.255794 (+1108732us) service_pool.cc:224] Handling call
0709 12:22:58.256115 (+   321us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:58.445025 (+188910us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:58.445670 (+   645us) spinlock_profiling.cc:243] Waited 188 ms on lock 0x7b5400137ed4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:22:58.445862 (+   192us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":432570240}
I20260709 12:22:58.261270 22833 raft_consensus.cc:2804] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:22:58.381901 18333 leader_election.cc:304] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:58.451325 22833 raft_consensus.cc:697] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 11 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
W20260709 12:22:58.451915 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 70121.8us
I20260709 12:22:58.453202 18333 leader_election.cc:304] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:58.343786 22874 consensus_queue.cc:260] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.11, Last appended by leader: 11, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:58.454424 22874 raft_consensus.cc:3060] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:22:58.454892 18333 leader_election.cc:304] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:58.388983 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:58.456418 22971 raft_consensus.cc:2749] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 12:22:58.456746 17829 raft_consensus.cc:2468] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 12.
I20260709 12:22:58.457433 22971 raft_consensus.cc:3060] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:58.458369 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 13 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:58.459089 17756 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 12.
I20260709 12:22:58.431325 22668 leader_election.cc:290] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:58.452522 22833 consensus_queue.cc:237] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:58.465149 18333 leader_election.cc:304] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:22:58.466277 22832 raft_consensus.cc:2749] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:22:58.466739 18333 leader_election.cc:304] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:22:58.392257 17838 raft_consensus.cc:2468] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 15.
I20260709 12:22:58.402369 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:58.473263 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 11 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:58.403167 22755 raft_consensus.cc:697] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 13 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:22:58.474010 22901 leader_election.cc:290] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:58.474709 22755 consensus_queue.cc:237] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.8, Last appended by leader: 8, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:58.476470 22383 raft_consensus.cc:515] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:58.477126 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:22:58.477414 17728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:58.477847 17728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:58.474040 17835 raft_consensus.cc:2393] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 11: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
W20260709 12:22:58.404248 17802 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:58.404316 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 163419us
I20260709 12:22:58.364948 22517 raft_consensus.cc:740] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:22:58.480532 17728 leader_election.cc:304] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:58.489018 17728 leader_election.cc:304] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:58.489599 17831 raft_consensus.cc:2468] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 11.
I20260709 12:22:58.471863 22971 raft_consensus.cc:2749] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:22:58.479727 18334 leader_election.cc:304] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:22:58.410928 22729 raft_consensus.cc:493] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:58.457787 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:58.498464 17728 leader_election.cc:304] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
W20260709 12:22:58.470551 17729 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:58.492179 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:58.499699 17689 raft_consensus.cc:2468] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 11.
W20260709 12:22:58.499788 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 95367.7us
I20260709 12:22:58.493359 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:58.385514 17759 raft_consensus.cc:515] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:58.494992 17835 raft_consensus.cc:1240] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 11. Current term is 12. Ops: []
I20260709 12:22:58.502336 17758 raft_consensus.cc:1240] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 14. Current term is 15. Ops: []
I20260709 12:22:58.495987 17838 raft_consensus.cc:1240] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 10. Current term is 11. Ops: []
I20260709 12:22:58.500874 17687 raft_consensus.cc:2393] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 11: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:22:58.514115 17802 leader_election.cc:304] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:58.516813 17835 raft_consensus.cc:1240] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 14. Current term is 15. Ops: []
W20260709 12:22:58.561455 17652 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:58.566540 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 63883.4us
I20260709 12:22:58.497524 22557 raft_consensus.cc:2804] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:22:58.506325 22971 raft_consensus.cc:2749] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:22:58.515614 22832 raft_consensus.cc:2749] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:22:58.523269 22966 raft_consensus.cc:493] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:58.506753 22745 raft_consensus.cc:2749] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:22:58.500875 22729 raft_consensus.cc:3060] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:58.593232 17654 leader_election.cc:400] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 10 election: Vote denied by peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577) with higher term. Message: Invalid argument: T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for earlier term 10. Current term is 11.
I20260709 12:22:58.597278 17654 leader_election.cc:403] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 10 election: Cancelling election due to peer responding with higher term
I20260709 12:22:58.596298 17652 leader_election.cc:400] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 12 pre-election: Vote denied by peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837) with higher term. Message: Invalid argument: T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for earlier term 12. Current term is 13.
I20260709 12:22:58.524160 17759 leader_election.cc:290] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 15 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:58.598843 17652 leader_election.cc:403] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 12 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:22:58.514685 22912 consensus_queue.cc:1048] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.086s
I20260709 12:22:58.508288 22590 raft_consensus.cc:493] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:58.506893 22673 raft_consensus.cc:515] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:58.601318 22590 raft_consensus.cc:515] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:58.601841 17836 raft_consensus.cc:2468] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 14.
I20260709 12:22:58.604461 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:58.604714 22673 leader_election.cc:290] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 14 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:58.605360 17759 raft_consensus.cc:2368] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 14. Current term is 15.
I20260709 12:22:58.609223 22700 raft_consensus.cc:2749] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:22:58.616012 22383 leader_election.cc:290] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 15 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:58.616912 22590 leader_election.cc:290] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:58.561388 22476 raft_consensus.cc:493] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:58.617883 22476 raft_consensus.cc:515] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:58.618791 22723 consensus_queue.cc:260] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 14.13, Last appended by leader: 13, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:58.619712 22517 consensus_queue.cc:260] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:58.630234 17759 raft_consensus.cc:1240] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 LEADER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 14. Current term is 15. Ops: [14.13-14.13]
W20260709 12:22:58.630491 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 58717.2us
I20260709 12:22:58.572528 22557 raft_consensus.cc:493] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:22:58.598821 22964 raft_consensus.cc:2804] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:22:58.625026 22668 raft_consensus.cc:2749] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Leader election lost for term 10. Reason: Vote denied by peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577) with higher term. Message: Invalid argument: T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for earlier term 10. Current term is 11.
I20260709 12:22:58.594893 22839 consensus_queue.cc:1048] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.045s
W20260709 12:22:58.657802 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 156811us
I20260709 12:22:58.661557 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:58.661753 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:58.570353 22966 raft_consensus.cc:515] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:58.638435 22557 raft_consensus.cc:3060] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:22:58.663461 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:58.666953 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:58.667948 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 16 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:22:58.674453 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 194783us
I20260709 12:22:58.680640 22729 raft_consensus.cc:515] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:58.662948 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:58.640312 22964 raft_consensus.cc:493] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:58.700598 18334 leader_election.cc:304] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:22:58.701361 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:22:58.704835 18361 raft_consensus.cc:3060] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:22:58.632910 22491 raft_consensus.cc:515] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:58.654325 22972 raft_consensus.cc:2749] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Leader pre-election lost for term 12. Reason: Vote denied by peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837) with higher term. Message: Invalid argument: T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for earlier term 12. Current term is 13.
I20260709 12:22:58.707412 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:58.708542 18334 leader_election.cc:304] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:22:58.701333 17762 raft_consensus.cc:3060] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:58.726390 18334 leader_election.cc:304] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
W20260709 12:22:58.732441 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 218438us
I20260709 12:22:58.734129 17802 leader_election.cc:304] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:22:58.735589 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 268921us
I20260709 12:22:58.739864 18361 raft_consensus.cc:2468] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 15.
I20260709 12:22:58.740953 17729 leader_election.cc:304] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:58.742867 18333 leader_election.cc:304] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:58.744577 18333 leader_election.cc:304] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:22:58.707607 22964 raft_consensus.cc:3060] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:58.698849 22476 leader_election.cc:290] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:22:58.797065 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 134402us
I20260709 12:22:58.799043 22954 consensus_queue.cc:1059] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
W20260709 12:22:58.827081 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 93053.7us
I20260709 12:22:58.828997 17802 leader_election.cc:304] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:58.720817 22984 raft_consensus.cc:2749] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:22:58.842615 17802 leader_election.cc:304] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:58.705437 22607 consensus_queue.cc:1059] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 14.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.069s
I20260709 12:22:58.768301 22833 raft_consensus.cc:2804] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:22:58.861819 17802 leader_election.cc:304] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:58.708441 22668 consensus_queue.cc:1048] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.117s
I20260709 12:22:58.743211 22981 raft_consensus.cc:2804] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:22:58.866878 22985 raft_consensus.cc:2804] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:22:58.838011 22476 raft_consensus.cc:2764] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Leader pre-election decision vote started in defunct term 9: won
I20260709 12:22:58.837011 22590 raft_consensus.cc:493] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:58.836159 22982 raft_consensus.cc:2804] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:22:58.835713 22989 raft_consensus.cc:493] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:58.861946 22948 log.cc:927] Time spent T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.095s	user 0.001s	sys 0.000s
I20260709 12:22:58.850528 22866 raft_consensus.cc:2804] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:22:58.765291 22971 consensus_queue.cc:1048] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.100s
I20260709 12:22:58.893855 17832 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35042 (request call id 2336) took 2800 ms. Trace:
I20260709 12:22:58.863530 22673 raft_consensus.cc:2764] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Leader pre-election decision vote started in defunct term 14: won
I20260709 12:22:58.862563 22833 raft_consensus.cc:493] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:58.870146 22981 raft_consensus.cc:493] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:58.929227 17832 rpcz_store.cc:276] 0709 12:22:56.078706 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:56.079853 (+  1147us) service_pool.cc:224] Handling call
0709 12:22:56.080589 (+   736us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:56.645418 (+564829us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:56.645451 (+    33us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:56.645493 (+    42us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:22:56.645531 (+    38us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:58.878344 (+2232813us) spinlock_profiling.cc:243] Waited 565 ms on lock 0x7b54004416d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:22:58.878407 (+    63us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:58.878584 (+   177us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1298389376}
I20260709 12:22:58.893568 22590 raft_consensus.cc:515] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:58.961768 22755 raft_consensus.cc:493] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:58.962635 22590 leader_election.cc:290] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:58.962440 22755 raft_consensus.cc:515] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:58.889945 22984 raft_consensus.cc:2749] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:22:58.959620 22981 raft_consensus.cc:3060] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:58.874367 22985 raft_consensus.cc:493] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:58.977706 22985 raft_consensus.cc:3060] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:22:58.969836 22833 raft_consensus.cc:3060] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:22:58.925081 22866 raft_consensus.cc:493] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
W20260709 12:22:58.982076 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 151006us
W20260709 12:22:58.992380 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 493966us
I20260709 12:22:58.991498 17652 leader_election.cc:304] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:22:59.007018 17759 raft_consensus.cc:1275] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 10 index: 7. Preceding OpId from leader: term: 13 index: 9. (index mismatch)
I20260709 12:22:58.986392 17762 raft_consensus.cc:2468] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 11.
I20260709 12:22:58.929920 22982 raft_consensus.cc:697] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:22:58.992524 22866 raft_consensus.cc:3060] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Advancing to term 16
W20260709 12:22:59.031963 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 166311us
W20260709 12:22:59.030980 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 404910us
I20260709 12:22:59.020773 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:59.012010 17652 leader_election.cc:304] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:22:59.035130 17652 leader_election.cc:304] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:22:59.038039 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:58.926226 22989 raft_consensus.cc:515] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:59.039021 17759 raft_consensus.cc:2468] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 13.
I20260709 12:22:59.044222 17762 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 12.
I20260709 12:22:59.044962 17802 leader_election.cc:304] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:59.009361 22998 raft_consensus.cc:2804] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:22:59.068782 22590 raft_consensus.cc:2749] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:22:59.070803 22590 consensus_queue.cc:1048] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:22:59.006738 22901 consensus_queue.cc:1048] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.041s
I20260709 12:22:59.073081 17727 leader_election.cc:304] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
W20260709 12:22:59.085072 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 340561us
W20260709 12:22:59.096120 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 61743us
I20260709 12:22:59.051292 22491 leader_election.cc:290] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 13 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:22:59.095062 17802 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:59.097349 18364 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.583s	user 0.006s	sys 0.019s
W20260709 12:22:59.108317 18364 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.598s	user 0.007s	sys 0.019s
I20260709 12:22:59.098791 18333 leader_election.cc:304] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:22:59.065951 22839 raft_consensus.cc:2804] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:22:59.106881 17727 leader_election.cc:304] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:22:59.109707 18333 leader_election.cc:304] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:22:59.066437 22729 leader_election.cc:290] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 13 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:22:59.115677 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 112806us
I20260709 12:22:59.115756 18333 leader_election.cc:304] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:22:59.116276 17727 leader_election.cc:304] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
W20260709 12:22:59.126595 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 56467.8us
I20260709 12:22:59.128710 18334 leader_election.cc:304] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:22:59.129142 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:59.129824 17831 raft_consensus.cc:2468] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 12.
I20260709 12:22:59.131860 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:59.133208 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:22:59.133899 17831 raft_consensus.cc:2468] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 14.
I20260709 12:22:59.133911 17728 leader_election.cc:304] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:22:59.143234 17728 leader_election.cc:304] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:22:59.147910 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 14 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:22:59.148562 18364 raft_consensus.cc:2410] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for term 14 because replica has last-logged OpId of term: 12 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 9.
I20260709 12:22:59.123405 22832 raft_consensus.cc:2804] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:22:59.125134 22383 raft_consensus.cc:3060] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:59.127170 22854 raft_consensus.cc:2804] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:22:59.167943 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 14 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:59.171182 17759 raft_consensus.cc:2410] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for term 14 because replica has last-logged OpId of term: 12 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 9.
I20260709 12:22:59.126968 22945 raft_consensus.cc:2749] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:22:59.120553 22986 raft_consensus.cc:2749] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:22:59.161113 17762 raft_consensus.cc:1240] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 14. Current term is 15. Ops: [14.11-14.11]
I20260709 12:22:59.122777 22971 raft_consensus.cc:2804] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:22:59.127427 22966 leader_election.cc:290] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:59.156599 22536 consensus_queue.cc:1059] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 14.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.036s
I20260709 12:22:59.080572 23005 consensus_queue.cc:1059] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 11.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.088s
I20260709 12:22:59.152136 22992 raft_consensus.cc:2804] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:22:59.182703 22839 raft_consensus.cc:697] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 11 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:22:59.144820 22729 raft_consensus.cc:2804] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:22:59.142515 23004 raft_consensus.cc:2749] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:22:59.192466 23008 consensus_queue.cc:1048] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
W20260709 12:22:59.188354 18362 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59902 (request call id 1296) took 3078 ms (client timeout 1949 ms). Trace:
I20260709 12:22:59.159773 22984 raft_consensus.cc:2804] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Leader pre-election won for term 11
W20260709 12:22:59.208016 18362 rpcz_store.cc:269] 0709 12:22:56.099824 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:56.211834 (+112010us) service_pool.cc:224] Handling call
0709 12:22:59.178081 (+2966247us) spinlock_profiling.cc:243] Waited 436 ms on lock 0x7b5400467b54. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:22:59.178222 (+   141us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1002668416}
W20260709 12:22:59.210269 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 69620.1us
I20260709 12:22:59.184866 22971 raft_consensus.cc:493] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:59.214836 22839 consensus_queue.cc:237] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:59.223004 22971 raft_consensus.cc:3060] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Advancing to term 16
W20260709 12:22:59.223459 22376 consensus_meta.cc:287] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.782s	user 0.014s	sys 0.001s
I20260709 12:22:59.188210 22994 consensus_queue.cc:1059] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 10.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.029s
I20260709 12:22:59.210325 22984 raft_consensus.cc:493] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:59.193670 22673 raft_consensus.cc:493] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:59.195550 22854 raft_consensus.cc:697] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 15 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
W20260709 12:22:59.232774 22961 log.cc:927] Time spent T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 1.098s	user 0.004s	sys 0.000s
I20260709 12:22:59.234470 22964 raft_consensus.cc:515] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:59.203586 22945 raft_consensus.cc:2749] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:22:59.203925 22992 raft_consensus.cc:493] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:22:59.283011 22992 raft_consensus.cc:3060] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Advancing to term 12
W20260709 12:22:59.252178 23002 log.cc:927] Time spent T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.051s	user 0.001s	sys 0.000s
I20260709 12:22:59.253015 22984 raft_consensus.cc:3060] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:22:59.256253 22976 log.cc:927] Time spent T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.588s	user 0.002s	sys 0.000s
W20260709 12:22:59.256987 22898 consensus_meta.cc:287] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.969s	user 0.006s	sys 0.007s
W20260709 12:22:59.258977 22672 consensus_meta.cc:287] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.848s	user 0.011s	sys 0.000s
I20260709 12:22:59.289858 22672 raft_consensus.cc:515] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:59.258574 22673 raft_consensus.cc:515] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:59.259035 22833 raft_consensus.cc:515] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:59.213308 22755 leader_election.cc:290] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:59.212561 22729 raft_consensus.cc:697] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 11 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
W20260709 12:22:59.289397 23009 log.cc:927] Time spent T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.076s	user 0.000s	sys 0.000s
I20260709 12:22:59.133266 22982 consensus_queue.cc:237] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 13.12, Last appended by leader: 12, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:22:59.265697 22987 log.cc:927] Time spent T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.129s	user 0.001s	sys 0.000s
I20260709 12:22:59.299101 22989 leader_election.cc:290] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:59.300609 22723 raft_consensus.cc:3060] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:22:59.291639 22898 raft_consensus.cc:515] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:59.317973 22898 leader_election.cc:290] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 14 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:22:59.299245 22997 log.cc:927] Time spent T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.155s	user 0.000s	sys 0.001s
I20260709 12:22:59.301689 22964 leader_election.cc:290] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 12 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:59.317685 22854 consensus_queue.cc:237] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.9, Last appended by leader: 9, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:22:59.294199 22977 log.cc:927] Time spent T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.577s	user 0.002s	sys 0.000s
W20260709 12:22:59.308276 22993 log.cc:927] Time spent T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.114s	user 0.000s	sys 0.001s
W20260709 12:22:59.322319 22980 log.cc:927] Time spent T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.644s	user 0.001s	sys 0.001s
I20260709 12:22:59.322796 22833 leader_election.cc:290] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 15 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:22:59.546798 17652 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:59.549079 17655 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:59.567934 17688 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37396 (request call id 2423) took 1083 ms. Trace:
W20260709 12:22:59.569691 17654 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:59.591192 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 443251us
I20260709 12:22:59.602748 17762 raft_consensus.cc:1275] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 15 index: 11. (index mismatch)
W20260709 12:22:59.606019 17802 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:59.614392 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 15 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:59.614045 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:59.616216 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:22:59.625583 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
W20260709 12:22:59.664202 17728 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.510s	user 0.002s	sys 0.010s
I20260709 12:22:59.675529 17838 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35042 (request call id 2356) took 1310 ms. Trace:
W20260709 12:22:59.720956 18333 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.601s	user 0.006s	sys 0.006s
W20260709 12:22:59.721426 17835 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.560s	user 0.004s	sys 0.010s
I20260709 12:22:59.723760 18366 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59886 (request call id 1504) took 1016 ms. Trace:
W20260709 12:22:59.724077 22536 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.531s	user 0.005s	sys 0.002s
I20260709 12:22:59.730243 22723 raft_consensus.cc:493] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:59.730587 22672 leader_election.cc:290] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 11 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:59.295753 22673 leader_election.cc:290] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:59.862354 17764 raft_consensus.cc:3060] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:59.731936 17830 raft_consensus.cc:2410] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for term 16 because replica has last-logged OpId of term: 14 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 11.
I20260709 12:22:59.865944 17652 leader_election.cc:304] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:22:59.838248 17838 rpcz_store.cc:276] 0709 12:22:58.364976 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:58.505192 (+140216us) service_pool.cc:224] Handling call
0709 12:22:58.505538 (+   346us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:58.672775 (+167237us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:58.672837 (+    62us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:58.673424 (+   587us) log.cc:844] Serialized 26 byte log entry
0709 12:22:58.989824 (+316400us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:22:58.989870 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:58.989942 (+    72us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:59.674760 (+684818us) raft_consensus.cc:1685] finished
0709 12:22:59.674816 (+    56us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:59.675292 (+   476us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":56925,"raft.queue_time_us":99429,"raft.run_cpu_time_us":80,"raft.run_wall_time_us":80,"thread_start_us":418210,"threads_started":2,"wal-append.queue_time_us":534465}
I20260709 12:22:59.870550 22495 raft_consensus.cc:2804] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:22:59.871130 22495 raft_consensus.cc:493] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:59.872251 23018 raft_consensus.cc:493] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:59.873445 22866 consensus_meta.cc:287] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.837s	user 0.013s	sys 0.005s
W20260709 12:22:59.298543 22965 consensus_meta.cc:287] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.885s	user 0.002s	sys 0.008s
I20260709 12:22:59.874712 22965 raft_consensus.cc:515] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:59.876585 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:59.877631 17682 raft_consensus.cc:2468] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 14.
I20260709 12:22:59.879061 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:59.324810 22971 raft_consensus.cc:515] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:59.880411 22898 raft_consensus.cc:493] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:59.880568 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:59.881345 22898 raft_consensus.cc:515] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:59.882254 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: true dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:59.882506 22971 leader_election.cc:290] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 16 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:59.883267 17837 raft_consensus.cc:2368] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for earlier term 14. Current term is 15.
W20260709 12:22:59.324543 22979 log.cc:927] Time spent T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.647s	user 0.002s	sys 0.001s
I20260709 12:22:59.319517 22729 consensus_queue.cc:237] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 10.9, Last appended by leader: 9, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:59.290454 22985 raft_consensus.cc:515] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:59.886220 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 13 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:22:59.888685 22956 log.cc:927] Time spent T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 1.719s	user 0.014s	sys 0.014s
I20260709 12:22:59.891256 18367 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59886 (request call id 1500) took 1416 ms. Trace:
I20260709 12:22:59.891654 17758 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38786 (request call id 3045) took 1615 ms. Trace:
W20260709 12:22:59.318285 22990 log.cc:927] Time spent T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.644s	user 0.002s	sys 0.000s
I20260709 12:22:59.892477 23012 raft_consensus.cc:493] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:59.894073 18368 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 1624) took 2448 ms. Trace:
I20260709 12:22:59.893702 23012 raft_consensus.cc:515] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:22:59.432736 23006 log.cc:927] Time spent T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.078s	user 0.000s	sys 0.000s
I20260709 12:22:59.896308 18365 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59886 (request call id 1499) took 1424 ms. Trace:
I20260709 12:22:59.897547 22985 leader_election.cc:290] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 15 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:59.898557 17681 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37396 (request call id 2403) took 2921 ms. Trace:
W20260709 12:22:59.899029 22992 consensus_meta.cc:287] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.615s	user 0.011s	sys 0.003s
I20260709 12:22:59.899528 23012 leader_election.cc:290] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:59.896867 18365 rpcz_store.cc:276] 0709 12:22:58.474495 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:58.478264 (+  3769us) service_pool.cc:224] Handling call
0709 12:22:58.478580 (+   316us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:22:58.478931 (+   351us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:58.478961 (+    30us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:22:58.500619 (+ 21658us) log.cc:844] Serialized 52 byte log entry
0709 12:22:58.671112 (+170493us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:22:59.086309 (+415197us) log.cc:844] Serialized 14 byte log entry
0709 12:22:59.086987 (+   678us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:59.087273 (+   286us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:59.317625 (+230352us) spinlock_profiling.cc:243] Waited 24.6 ms on lock 0x7b4400022b4c. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b47459c85 00007f7b4745a6a2 00007f7b4a1cc1db 00007f7b4a1c7aaf 00007f7b4a1bb86e 00007f7b4a1babb6 00007f7b4a1ba7d8 00007f7b4a1c4221 00007f7b4a1c41d9 00007f7b4a1c4169
0709 12:22:59.895503 (+577878us) raft_consensus.cc:1685] finished
0709 12:22:59.895701 (+   198us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:59.896110 (+   409us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":48264,"prepare.queue_time_us":343883,"prepare.run_cpu_time_us":68,"prepare.run_wall_time_us":4947,"raft.queue_time_us":16469,"raft.run_cpu_time_us":419,"raft.run_wall_time_us":36692,"spinlock_wait_cycles":56580864,"thread_start_us":421494,"threads_started":2,"wal-append.queue_time_us":150018}
I20260709 12:22:59.900233 23012 raft_consensus.cc:493] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:59.857407 22536 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.662s	user 0.006s	sys 0.003s
I20260709 12:22:59.901098 23012 raft_consensus.cc:515] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:59.871582 22495 raft_consensus.cc:3060] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:22:59.845660 22652 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.583s	user 0.006s	sys 0.018s
W20260709 12:22:59.848789 18333 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.729s	user 0.006s	sys 0.006s
W20260709 12:22:59.853055 17835 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.692s	user 0.005s	sys 0.010s
W20260709 12:22:59.848294 17728 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.694s	user 0.002s	sys 0.010s
W20260709 12:22:59.857031 17802 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:59.791738 22875 raft_consensus.cc:3060] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:59.856535 22723 raft_consensus.cc:515] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:59.860447 17759 raft_consensus.cc:2468] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 14.
I20260709 12:22:59.840447 18366 rpcz_store.cc:276] 0709 12:22:58.706710 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:58.706851 (+   141us) service_pool.cc:224] Handling call
0709 12:22:58.707156 (+   305us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:58.707523 (+   367us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:58.707550 (+    27us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:58.708650 (+  1100us) log.cc:844] Serialized 26 byte log entry
0709 12:22:59.078588 (+369938us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:22:59.078761 (+   173us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:59.078872 (+   111us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:59.722958 (+644086us) raft_consensus.cc:1685] finished
0709 12:22:59.723005 (+    47us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:59.723476 (+   471us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":287,"raft.run_cpu_time_us":133,"raft.run_wall_time_us":4672,"thread_start_us":305383,"threads_started":1,"wal-append.queue_time_us":413985}
I20260709 12:22:59.861303 17755 raft_consensus.cc:2468] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 12.
I20260709 12:22:59.721002 23026 raft_consensus.cc:493] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:59.873502 17832 raft_consensus.cc:3055] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:22:59.883375 17689 raft_consensus.cc:2393] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 15: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:22:59.936030 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 15 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:59.936843 22723 leader_election.cc:290] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:22:59.937042 17686 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37396 (request call id 2404) took 2959 ms (client timeout 2954 ms). Trace:
I20260709 12:22:59.393378 22981 raft_consensus.cc:515] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:22:59.937449 17686 rpcz_store.cc:269] 0709 12:22:56.977612 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:57.820647 (+843035us) service_pool.cc:224] Handling call
0709 12:22:59.936608 (+2115961us) spinlock_profiling.cc:243] Waited 81.4 ms on lock 0x7b54003002d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:22:59.936826 (+   218us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":187194624}
I20260709 12:22:59.938158 17683 raft_consensus.cc:1275] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20260709 12:22:59.899789 22992 raft_consensus.cc:515] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:59.873535 23018 raft_consensus.cc:515] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:59.942412 23018 leader_election.cc:290] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:59.781948 17688 rpcz_store.cc:276] 0709 12:22:58.484508 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:58.484629 (+   121us) service_pool.cc:224] Handling call
0709 12:22:58.484938 (+   309us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:22:58.493573 (+  8635us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:58.493609 (+    36us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:22:58.504347 (+ 10738us) log.cc:844] Serialized 52 byte log entry
0709 12:22:58.785188 (+280841us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:22:58.785247 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:58.785488 (+   241us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:59.566897 (+781409us) raft_consensus.cc:1685] finished
0709 12:22:59.566993 (+    96us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:59.567597 (+   604us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":118,"raft.run_cpu_time_us":115,"raft.run_wall_time_us":8062,"thread_start_us":214702,"threads_started":1,"wal-append.queue_time_us":500887}
I20260709 12:22:59.937225 17689 raft_consensus.cc:2468] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 14.
I20260709 12:22:59.905397 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:59.903146 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:59.894609 18368 rpcz_store.cc:276] 0709 12:22:57.445697 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:57.524037 (+ 78340us) service_pool.cc:224] Handling call
0709 12:22:57.524505 (+   468us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:22:57.524866 (+   361us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:57.524896 (+    30us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:22:57.528599 (+  3703us) log.cc:844] Serialized 52 byte log entry
0709 12:22:57.961555 (+432956us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:22:57.961604 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:57.961683 (+    79us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:59.887579 (+1925896us) spinlock_profiling.cc:243] Waited 13.9 ms on lock 0x7b4400022b4c. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b47459c85 00007f7b4745a6a2 00007f7b4a1cc1db 00007f7b4a1c7aaf 00007f7b4a1bb86e 00007f7b4a1babb6 00007f7b4a1ba7d8 00007f7b4a1c4221 00007f7b4a1c41d9 00007f7b4a1c4169
0709 12:22:59.893361 (+  5782us) raft_consensus.cc:1685] finished
0709 12:22:59.893394 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:59.893836 (+   442us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":579,"raft.run_cpu_time_us":125,"raft.run_wall_time_us":2711,"spinlock_wait_cycles":32051584,"thread_start_us":393454,"threads_started":1,"wal-append.queue_time_us":498499}
W20260709 12:22:59.946419 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 830685us
I20260709 12:22:59.892140 17758 rpcz_store.cc:276] 0709 12:22:58.276043 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:58.276160 (+   117us) service_pool.cc:224] Handling call
0709 12:22:58.276479 (+   319us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:58.509385 (+232906us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:59.891173 (+1381788us) spinlock_profiling.cc:243] Waited 226 ms on lock 0x7b540019fad4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:22:59.891472 (+   299us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":519041792}
I20260709 12:22:59.899163 17681 rpcz_store.cc:276] 0709 12:22:56.977374 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:57.879968 (+902594us) service_pool.cc:224] Handling call
0709 12:22:58.324658 (+444690us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:58.367892 (+ 43234us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:59.898099 (+1530207us) spinlock_profiling.cc:243] Waited 346 ms on lock 0x7b5400206f50. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a61f8ad 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:22:59.898380 (+   281us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":796013952}
I20260709 12:22:59.913179 17652 leader_election.cc:304] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:22:59.950464 22981 leader_election.cc:290] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 11 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:59.916733 22852 raft_consensus.cc:493] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:59.952059 22866 raft_consensus.cc:515] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:22:59.952631 22383 consensus_meta.cc:287] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.766s	user 0.007s	sys 0.016s
I20260709 12:22:59.952488 22852 raft_consensus.cc:515] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:22:59.953923 22723 raft_consensus.cc:493] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:59.954233 22864 raft_consensus.cc:493] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:59.954753 22723 raft_consensus.cc:515] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:59.954947 22864 raft_consensus.cc:515] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:59.955430 22994 raft_consensus.cc:2804] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:22:59.956214 22994 raft_consensus.cc:493] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:59.956677 22994 raft_consensus.cc:3060] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:22:59.957844 22723 leader_election.cc:290] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:22:59.934661 23029 log.cc:927] Time spent T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.077s	user 0.002s	sys 0.000s
I20260709 12:22:59.958464 22864 leader_election.cc:290] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:59.904719 22518 raft_consensus.cc:3055] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:22:59.960397 22518 raft_consensus.cc:740] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 14 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:22:59.961799 22866 leader_election.cc:290] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 16 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:59.962046 22890 raft_consensus.cc:493] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:22:59.961719 22518 consensus_queue.cc:260] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.9, Last appended by leader: 9, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:22:59.965251 17836 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35042 (request call id 2351) took 1635 ms (client timeout 1583 ms). Trace:
W20260709 12:22:59.965854 17836 rpcz_store.cc:269] 0709 12:22:58.330238 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:58.367745 (+ 37507us) service_pool.cc:224] Handling call
0709 12:22:59.964610 (+1596865us) spinlock_profiling.cc:243] Waited 192 ms on lock 0x7b5400158bd4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:22:59.964970 (+   360us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":442386688}
I20260709 12:22:59.918974 17685 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44898 (request call id 980) took 2875 ms. Trace:
I20260709 12:22:59.891750 18367 rpcz_store.cc:276] 0709 12:22:58.474825 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:58.475233 (+   408us) service_pool.cc:224] Handling call
0709 12:22:58.475548 (+   315us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:58.475927 (+   379us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:58.475958 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:58.476437 (+   479us) log.cc:844] Serialized 26 byte log entry
0709 12:22:58.737143 (+260706us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:22:58.737205 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:58.737303 (+    98us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:59.320892 (+583589us) spinlock_profiling.cc:243] Waited 4.29 ms on lock 0x7b4800220a58. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4f549aa1 00007f7b4f540b37 00007f7b4f5409ca 00007f7b4f63826f 00007f7b4f637f37 00007f7b4f64c55c 00007f7b4f64c508 00007f7b4f64c488 00007f7b4f64c443 00007f7b4f64b6ec 00007f7b4e575a96
0709 12:22:59.323138 (+  2246us) spinlock_profiling.cc:243] Waited 31.2 ms on lock 0x7b4400022b4c. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b47459c85 00007f7b4745a6a2 00007f7b4a1cc1db 00007f7b4a1c7aaf 00007f7b4a1bb86e 00007f7b4a1babb6 00007f7b4a1ba7d8 00007f7b4a1c4221 00007f7b4a1c41d9 00007f7b4a1c4169
0709 12:22:59.890746 (+567608us) raft_consensus.cc:1685] finished
0709 12:22:59.890783 (+    37us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:59.891049 (+   266us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4853,"raft.queue_time_us":3320,"raft.run_cpu_time_us":3469,"raft.run_wall_time_us":832280,"spinlock_wait_cycles":81604736,"thread_start_us":260330,"threads_started":1,"wal-append.queue_time_us":199681}
I20260709 12:22:59.967466 17685 rpcz_store.cc:276] 0709 12:22:57.094040 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:58.126652 (+1032612us) service_pool.cc:224] Handling call
0709 12:22:58.146236 (+ 19584us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:22:58.175162 (+ 28926us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:59.918449 (+1743287us) spinlock_profiling.cc:243] Waited 98.1 ms on lock 0x7b540030bc20. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a597dc0 00007f7b4a623a02 00007f7b4a61fdd6 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2
0709 12:22:59.918802 (+   353us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":225664896}
I20260709 12:22:59.922237 17680 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37402 (request call id 2769) took 2614 ms. Trace:
I20260709 12:22:59.970037 22832 raft_consensus.cc:493] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:22:59.970250 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:22:59.970729 22832 raft_consensus.cc:3060] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:59.971668 23026 raft_consensus.cc:515] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:59.953492 22383 raft_consensus.cc:2749] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:22:59.972826 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:22:59.917906 22970 raft_consensus.cc:493] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:59.934510 17832 raft_consensus.cc:740] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 11 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:22:59.974159 22970 raft_consensus.cc:515] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:59.974838 18374 raft_consensus.cc:1275] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 11 index: 12. Preceding OpId from leader: term: 12 index: 13. (index mismatch)
I20260709 12:22:59.974675 17832 consensus_queue.cc:260] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 11.12, Last appended by leader: 12, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:22:59.975998 17832 raft_consensus.cc:3060] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:59.962651 22890 raft_consensus.cc:3060] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:59.944478 17683 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37396 (request call id 2421) took 1794 ms. Trace:
W20260709 12:22:59.953207 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 809993us
I20260709 12:22:59.942472 22992 leader_election.cc:290] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 12 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:59.918715 22898 leader_election.cc:290] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:22:59.877872 22965 leader_election.cc:290] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 15 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:22:59.969434 17680 rpcz_store.cc:276] 0709 12:22:57.307966 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:58.261741 (+953775us) service_pool.cc:224] Handling call
0709 12:22:59.920810 (+1659069us) spinlock_profiling.cc:243] Waited 7.29 ms on lock 0x7b5800002a98. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b474fbb71 00007f7b474fba22 00007f7b504148d9 00007f7b4f56eee0 00007f7b4f6380ab 00007f7b4f638131 00007f7b4f5f9fc0 00007f7b4f5e4929 00007f7b4f5e43d2 00007f7b48b5bbd4
0709 12:22:59.921229 (+   419us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:59.921641 (+   412us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:59.921668 (+    27us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:59.921743 (+    75us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:22:59.921791 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:59.921855 (+    64us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:59.922011 (+   156us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":16757504}
I20260709 12:22:59.929824 17764 raft_consensus.cc:2468] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 14.
W20260709 12:22:59.978161 22652 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.715s	user 0.006s	sys 0.019s
I20260709 12:22:59.984133 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:22:59.976579 17683 rpcz_store.cc:276] 0709 12:22:58.150120 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:58.260434 (+110314us) service_pool.cc:224] Handling call
0709 12:22:59.937556 (+1677122us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:59.938584 (+  1028us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:59.944269 (+  5685us) spinlock_profiling.cc:243] Waited 1.67 s on lock 0x7b54003002d0. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a61f8ad 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:22:59.944422 (+   153us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3836259072}
I20260709 12:22:59.961612 22852 leader_election.cc:290] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:22:59.959543 22722 leader_election.cc:290] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:00.004570 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:00.013831 22518 raft_consensus.cc:3060] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:00.014868 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:00.015693 17683 raft_consensus.cc:1275] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 11 index: 8. Preceding OpId from leader: term: 13 index: 9. (index mismatch)
I20260709 12:23:00.015870 17829 raft_consensus.cc:2393] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 15: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:23:00.025381 23012 leader_election.cc:290] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:00.025738 23026 leader_election.cc:290] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:23:00.026227 22984 consensus_meta.cc:287] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.614s	user 0.012s	sys 0.004s
I20260709 12:23:00.013057 18333 leader_election.cc:304] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:00.026818 22984 raft_consensus.cc:515] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:00.029191 17802 leader_election.cc:304] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:23:00.030162 22722 raft_consensus.cc:493] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:00.030920 22722 raft_consensus.cc:515] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:00.032095 22965 raft_consensus.cc:2804] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:23:00.032604 22965 raft_consensus.cc:493] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:00.032956 22965 raft_consensus.cc:3060] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:00.036049 23026 raft_consensus.cc:3060] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:00.039317 22970 leader_election.cc:290] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:23:00.040252 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 92373.6us
I20260709 12:23:00.041357 18333 leader_election.cc:304] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:00.042510 22900 raft_consensus.cc:3060] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:00.045683 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:00.046607 17830 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37994 (request call id 1016) took 1808 ms. Trace:
I20260709 12:23:00.002367 22971 raft_consensus.cc:493] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:00.047412 22722 leader_election.cc:290] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:00.046571 17838 raft_consensus.cc:2468] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 10.
I20260709 12:23:00.002599 22981 raft_consensus.cc:493] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:00.064003 22981 raft_consensus.cc:515] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:00.066471 22981 leader_election.cc:290] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:00.031706 17802 leader_election.cc:304] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
W20260709 12:23:00.072495 17804 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:00.072715 22973 raft_consensus.cc:2804] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:23:00.073124 22973 raft_consensus.cc:493] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:00.073465 22973 raft_consensus.cc:3060] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:00.074501 22722 raft_consensus.cc:493] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:23:00.075177 22722 raft_consensus.cc:515] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:00.076232 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 44723.2us
I20260709 12:23:00.077906 22722 leader_election.cc:290] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:00.078801 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:00.079157 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:00.079795 17680 raft_consensus.cc:3060] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Advancing to term 15
I20260709 12:23:00.080611 17838 raft_consensus.cc:2393] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 14: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:23:00.083031 22972 raft_consensus.cc:3055] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:23:00.083478 22972 raft_consensus.cc:740] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:23:00.047075 17830 rpcz_store.cc:276] 0709 12:22:58.238142 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:58.238264 (+   122us) service_pool.cc:224] Handling call
0709 12:23:00.046379 (+1808115us) spinlock_profiling.cc:243] Waited 1.49 s on lock 0x7b5400089dd4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:23:00.046554 (+   175us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3423427840}
I20260709 12:23:00.084631 22972 consensus_queue.cc:260] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:00.085801 22972 raft_consensus.cc:3060] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:00.101112 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:00.102245 17764 raft_consensus.cc:2393] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 14: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:23:00.103576 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:00.096531 17687 raft_consensus.cc:2468] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 13.
W20260709 12:23:00.115228 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 53331.2us
I20260709 12:23:00.119184 17652 leader_election.cc:304] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:23:00.121995 17832 raft_consensus.cc:1275] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 11 index: 12. Preceding OpId from leader: term: 12 index: 13. (index mismatch)
W20260709 12:23:00.116778 18364 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.804s	user 0.027s	sys 0.010s
W20260709 12:23:00.123212 18364 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.914s	user 0.027s	sys 0.010s
I20260709 12:23:00.123644 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:00.087349 22890 raft_consensus.cc:515] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:00.129735 22877 raft_consensus.cc:2804] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:23:00.130291 22877 raft_consensus.cc:493] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:00.130618 22877 raft_consensus.cc:3060] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:00.131163 22890 leader_election.cc:290] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 11 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:00.132510 17686 raft_consensus.cc:1240] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 12 LEADER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 11. Current term is 12. Ops: []
I20260709 12:23:00.134183 17829 raft_consensus.cc:1240] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 10. Current term is 11. Ops: []
I20260709 12:23:00.127785 22994 raft_consensus.cc:515] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:00.136363 18333 leader_election.cc:304] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:23:00.138597 22994 leader_election.cc:290] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 15 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:00.140429 22981 raft_consensus.cc:2804] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:23:00.140974 22981 raft_consensus.cc:493] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:23:00.141381 17684 raft_consensus.cc:2468] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 7.
I20260709 12:23:00.143105 17838 raft_consensus.cc:1240] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 13. Current term is 14. Ops: []
I20260709 12:23:00.144321 22853 raft_consensus.cc:493] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:00.145231 17835 raft_consensus.cc:1240] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 13. Current term is 14. Ops: [13.11-13.11]
I20260709 12:23:00.145128 22853 raft_consensus.cc:515] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:00.147590 17836 raft_consensus.cc:1275] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
W20260709 12:23:00.148365 17684 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44898 (request call id 997) took 2697 ms (client timeout 1958 ms). Trace:
W20260709 12:23:00.148921 17684 rpcz_store.cc:269] 0709 12:22:57.451033 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:57.679830 (+228797us) service_pool.cc:224] Handling call
0709 12:23:00.148132 (+2468302us) spinlock_profiling.cc:243] Waited 2.31 s on lock 0x7b54003471d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:23:00.148281 (+   149us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5322379520}
W20260709 12:23:00.139451 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 378938us
I20260709 12:23:00.150527 17833 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35042 (request call id 2342) took 3270 ms. Trace:
I20260709 12:23:00.152858 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:00.154609 17764 raft_consensus.cc:2468] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 15.
W20260709 12:23:00.158070 17728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:00.127614 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:00.169286 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 9 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:00.172124 22607 raft_consensus.cc:493] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:23:00.172982 22607 raft_consensus.cc:515] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:00.047394 22971 raft_consensus.cc:515] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:00.175577 17683 raft_consensus.cc:1240] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 14. Current term is 15. Ops: []
I20260709 12:23:00.177280 22971 leader_election.cc:290] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:00.178694 22984 leader_election.cc:290] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 11 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:23:00.123231 22590 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 12 current term: 12 is higher.
I20260709 12:23:00.132510 17830 raft_consensus.cc:1275] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 14 index: 13. Preceding OpId from leader: term: 15 index: 13. (term mismatch)
I20260709 12:23:00.154853 22899 raft_consensus.cc:3060] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Advancing to term 14
W20260709 12:23:00.179456 17729 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:00.183686 17832 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35040 (request call id 2606) took 1054 ms. Trace:
I20260709 12:23:00.181283 17830 pending_rounds.cc:85] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b: Aborting all ops after (but not including) 12
I20260709 12:23:00.185834 17832 rpcz_store.cc:276] 0709 12:22:59.129697 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:59.129852 (+   155us) service_pool.cc:224] Handling call
0709 12:22:59.130134 (+   282us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:59.873160 (+743026us) spinlock_profiling.cc:243] Waited 508 ms on lock 0x7b54000eeaa0. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a597dc0 00007f7b4a6221c0 00007f7b4a623231 00007f7b4a61fd8f 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c
0709 12:23:00.122416 (+249256us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:00.183625 (+ 61209us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1168513920}
I20260709 12:23:00.190088 17681 raft_consensus.cc:2468] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 12.
I20260709 12:23:00.141431 22981 raft_consensus.cc:3060] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:00.186512 17830 pending_rounds.cc:107] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b: Aborting uncommitted NO_OP operation due to leader change: 14.13
I20260709 12:23:00.197620 17830 raft_consensus.cc:2889] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:00.196172 17680 raft_consensus.cc:2468] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 15.
I20260709 12:23:00.198915 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:00.203739 22518 raft_consensus.cc:493] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:00.208889 22518 raft_consensus.cc:515] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:00.150908 17833 rpcz_store.cc:276] 0709 12:22:56.880302 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:56.954424 (+ 74122us) service_pool.cc:224] Handling call
0709 12:22:56.954695 (+   271us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:57.106485 (+151790us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 12:22:57.173548 (+ 67063us) log.cc:844] Serialized 14 byte log entry
0709 12:22:58.025584 (+852036us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:58.025633 (+    49us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:22:58.025684 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:00.149518 (+2123834us) spinlock_profiling.cc:243] Waited 152 ms on lock 0x7b5400334ad4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:23:00.149598 (+    80us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:00.150279 (+   681us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":60068,"prepare.queue_time_us":67283,"prepare.run_cpu_time_us":56,"prepare.run_wall_time_us":11297,"raft.queue_time_us":211362,"raft.run_cpu_time_us":109,"raft.run_wall_time_us":110,"spinlock_wait_cycles":348576128,"thread_start_us":819981,"threads_started":2,"wal-append.queue_time_us":890749,"wal-append.run_cpu_time_us":2312,"wal-append.run_wall_time_us":1360051}
I20260709 12:23:00.217470 22984 raft_consensus.cc:493] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
W20260709 12:23:00.218189 17834 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35040 (request call id 2599) took 1930 ms (client timeout 2343 ms). Trace:
I20260709 12:23:00.218318 22984 raft_consensus.cc:515] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:00.218609 17834 rpcz_store.cc:269] 0709 12:22:58.250827 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:58.251981 (+  1154us) service_pool.cc:224] Handling call
0709 12:23:00.155644 (+1903663us) spinlock_profiling.cc:243] Waited 76.6 ms on lock 0x7b54000aa5d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:23:00.155861 (+   217us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":176263552}
I20260709 12:23:00.219000 22518 leader_election.cc:290] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:00.221395 23023 raft_consensus.cc:493] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:00.221706 22984 leader_election.cc:290] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:00.222193 23023 raft_consensus.cc:515] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:00.222821 22878 leader_election.cc:290] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:00.225297 18333 leader_election.cc:304] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:00.228559 18374 raft_consensus.cc:1240] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 10. Current term is 11. Ops: []
I20260709 12:23:00.229386 17838 raft_consensus.cc:2368] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for earlier term 13. Current term is 14.
I20260709 12:23:00.191000 22973 raft_consensus.cc:515] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:00.192024 22965 raft_consensus.cc:515] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:00.232558 22518 raft_consensus.cc:493] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:00.192785 22832 raft_consensus.cc:515] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:00.241495 18366 raft_consensus.cc:1240] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 14. Current term is 15. Ops: []
I20260709 12:23:00.238710 22878 raft_consensus.cc:2749] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
W20260709 12:23:00.254802 17655 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:00.255734 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: true dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:00.257428 17829 raft_consensus.cc:3055] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:23:00.258301 17829 raft_consensus.cc:740] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 14 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:23:00.200389 22900 raft_consensus.cc:2749] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:23:00.259034 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:00.260313 22518 raft_consensus.cc:515] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:00.264719 17829 consensus_queue.cc:260] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.8, Last appended by leader: 8, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:00.282863 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:00.292121 17829 raft_consensus.cc:3060] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:00.293015 18368 raft_consensus.cc:1275] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
I20260709 12:23:00.272095 22607 leader_election.cc:290] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:00.294790 22518 leader_election.cc:290] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:00.300189 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
W20260709 12:23:00.303740 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 120873us
I20260709 12:23:00.299274 17830 raft_consensus.cc:2468] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 14.
I20260709 12:23:00.317569 17838 raft_consensus.cc:2468] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 12.
W20260709 12:23:00.342319 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 48938.6us
I20260709 12:23:00.340917 18333 leader_election.cc:304] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:00.325973 23023 leader_election.cc:290] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:00.357050 17830 raft_consensus.cc:1240] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 14. Current term is 15. Ops: []
I20260709 12:23:00.339743 23026 raft_consensus.cc:2749] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:23:00.368232 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:00.372975 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:00.355723 22518 raft_consensus.cc:493] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:00.374969 17830 raft_consensus.cc:1240] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 15. Current term is 16. Ops: []
W20260709 12:23:00.378068 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 248306us
I20260709 12:23:00.368208 22984 raft_consensus.cc:2749] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:23:00.392141 17681 raft_consensus.cc:2468] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 12.
I20260709 12:23:00.394519 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:00.381199 22832 leader_election.cc:290] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 10 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:00.396978 17680 raft_consensus.cc:3060] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Advancing to term 16
W20260709 12:23:00.403787 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 459197us
W20260709 12:23:00.404198 17802 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:00.386801 22973 leader_election.cc:290] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 15 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:00.406162 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
W20260709 12:23:00.411309 22982 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.998s	user 0.014s	sys 0.046s
W20260709 12:23:00.400974 22839 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.084s	user 0.023s	sys 0.048s
W20260709 12:23:00.411722 22982 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.019s	user 0.014s	sys 0.047s
W20260709 12:23:00.411852 22839 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.187s	user 0.023s	sys 0.049s
W20260709 12:23:00.412940 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 106773us
W20260709 12:23:00.396744 22998 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.231s	user 0.000s	sys 0.026s
I20260709 12:23:00.420900 18364 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 1639) took 1237 ms. Trace:
W20260709 12:23:00.422099 17655 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:00.410267 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 15 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:00.424129 17686 raft_consensus.cc:3060] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:00.435257 17688 raft_consensus.cc:1240] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 15 LEADER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 14. Current term is 15. Ops: []
I20260709 12:23:00.413523 22965 leader_election.cc:290] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 15 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:23:00.425774 22557 consensus_meta.cc:287] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 1.625s	user 0.022s	sys 0.004s
W20260709 12:23:00.425125 22998 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.284s	user 0.000s	sys 0.027s
I20260709 12:23:00.407538 22992 consensus_queue.cc:1059] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 11.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.034s
I20260709 12:23:00.406525 22518 raft_consensus.cc:515] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:00.421440 18364 rpcz_store.cc:276] 0709 12:22:59.185484 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:59.188966 (+  3482us) service_pool.cc:224] Handling call
0709 12:22:59.189348 (+   382us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:59.193577 (+  4229us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 12:22:59.209069 (+ 15492us) log.cc:844] Serialized 14 byte log entry
0709 12:23:00.124028 (+914959us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:00.124658 (+   630us) log.cc:844] Serialized 26 byte log entry
0709 12:23:00.124867 (+   209us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:23:00.124906 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:00.125014 (+   108us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:00.409792 (+284778us) raft_consensus.cc:1685] finished
0709 12:23:00.409814 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:00.420819 (+ 11005us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3727,"prepare.queue_time_us":15787,"prepare.run_cpu_time_us":53,"prepare.run_wall_time_us":54,"raft.queue_time_us":63117,"raft.run_cpu_time_us":368,"raft.run_wall_time_us":24445,"thread_start_us":914427,"threads_started":1,"wal-append.queue_time_us":1084707}
W20260709 12:23:00.467180 17804 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:00.460893 22745 consensus_queue.cc:1048] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.029s
I20260709 12:23:00.417435 23023 consensus_queue.cc:1059] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 10.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.032s
W20260709 12:23:00.469580 17804 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:00.416792 22495 raft_consensus.cc:515] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:23:00.477339 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:00.477671 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:00.478276 17833 raft_consensus.cc:2393] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 15: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:23:00.484644 17802 leader_election.cc:304] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
W20260709 12:23:00.488293 17801 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:00.452971 22998 raft_consensus.cc:697] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 14 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:23:00.490249 22973 raft_consensus.cc:2804] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:23:00.502591 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:00.504479 17802 leader_election.cc:304] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:23:00.505594 17802 leader_election.cc:304] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:23:00.507400 22476 raft_consensus.cc:2804] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:23:00.507586 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:00.508200 22965 raft_consensus.cc:2804] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:23:00.508390 22923 raft_consensus.cc:2749] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:23:00.508689 17684 raft_consensus.cc:2393] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 15: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:23:00.510691 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:00.512868 17829 raft_consensus.cc:2468] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 15.
I20260709 12:23:00.514663 17680 raft_consensus.cc:2468] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 16.
I20260709 12:23:00.508737 22965 raft_consensus.cc:493] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:00.455782 22557 raft_consensus.cc:515] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:23:00.548029 17802 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:00.485476 22652 consensus_queue.cc:1059] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 13.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.033s
I20260709 12:23:00.490469 23018 raft_consensus.cc:493] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:00.561074 18366 raft_consensus.cc:3060] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Advancing to term 14
W20260709 12:23:00.564457 17652 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:00.521188 22973 raft_consensus.cc:697] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 15 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:23:00.550204 22965 raft_consensus.cc:3060] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:23:00.569300 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 168133us
I20260709 12:23:00.518018 22476 raft_consensus.cc:697] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 12 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:23:00.508054 23007 consensus_queue.cc:1048] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 6, Time since last communication: 0.116s
I20260709 12:23:00.582999 22518 leader_election.cc:290] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:23:00.583865 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 146957us
I20260709 12:23:00.586217 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:00.594756 18361 raft_consensus.cc:1240] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 11. Current term is 12. Ops: []
I20260709 12:23:00.594774 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:00.605618 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:00.616602 22832 raft_consensus.cc:493] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:00.617308 22832 raft_consensus.cc:515] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:00.617136 22998 consensus_queue.cc:237] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.10, Last appended by leader: 10, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:00.618074 22557 leader_election.cc:290] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 15 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:00.618961 22557 consensus_queue.cc:1059] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 10.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 12:23:00.619729 22832 leader_election.cc:290] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:00.621085 17760 raft_consensus.cc:1240] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 14. Current term is 15. Ops: []
I20260709 12:23:00.645727 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:00.634325 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:00.683946 17835 raft_consensus.cc:3060] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:00.648620 18361 raft_consensus.cc:1240] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 10. Current term is 11. Ops: []
I20260709 12:23:00.627910 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 13 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:00.672966 18364 raft_consensus.cc:2393] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 14: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
W20260709 12:23:00.625535 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 707943us
I20260709 12:23:00.647676 18366 raft_consensus.cc:2410] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 14 because replica has last-logged OpId of term: 13 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 10.
I20260709 12:23:00.638114 17801 leader_election.cc:400] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 14 election: Vote denied by peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651) with higher term. Message: Invalid argument: T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 14. Current term is 15.
I20260709 12:23:00.690065 17801 leader_election.cc:403] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 14 election: Cancelling election due to peer responding with higher term
I20260709 12:23:00.637408 18371 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59886 (request call id 1489) took 3050 ms. Trace:
I20260709 12:23:00.628223 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:00.649988 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:23:00.621383 17837 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35040 (request call id 2602) took 2367 ms (client timeout 3092 ms). Trace:
I20260709 12:23:00.650105 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:00.698938 18374 raft_consensus.cc:2468] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 12.
W20260709 12:23:00.713798 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 134975us
W20260709 12:23:00.635362 22877 consensus_meta.cc:287] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.504s	user 0.014s	sys 0.002s
I20260709 12:23:00.710696 18363 raft_consensus.cc:2468] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 12.
W20260709 12:23:00.700173 17837 rpcz_store.cc:269] 0709 12:22:58.254101 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:58.305534 (+ 51433us) service_pool.cc:224] Handling call
0709 12:23:00.621053 (+2315519us) spinlock_profiling.cc:243] Waited 1.53 s on lock 0x7b54003354d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:23:00.621233 (+   180us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3509693824}
W20260709 12:23:00.620752 23025 log.cc:927] Time spent T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.757s	user 0.003s	sys 0.002s
I20260709 12:23:00.622987 22982 raft_consensus.cc:3055] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:23:00.650259 22878 raft_consensus.cc:493] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:00.601794 23018 raft_consensus.cc:515] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:00.737102 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:00.621717 22495 leader_election.cc:290] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 13 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:00.618010 22652 raft_consensus.cc:3055] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:23:00.628985 22536 raft_consensus.cc:3055] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:23:00.729039 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
W20260709 12:23:00.742295 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 149647us
I20260709 12:23:00.658442 22517 raft_consensus.cc:3060] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:23:00.757126 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 119269us
I20260709 12:23:00.622838 23023 consensus_queue.cc:1048] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.117s
I20260709 12:23:00.764739 17686 raft_consensus.cc:2468] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 15.
I20260709 12:23:00.729027 22982 raft_consensus.cc:740] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:23:00.697860 18371 rpcz_store.cc:276] 0709 12:22:57.544602 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:57.720639 (+176037us) service_pool.cc:224] Handling call
0709 12:22:57.733912 (+ 13273us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:57.734773 (+   861us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:57.734813 (+    40us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:57.735256 (+   443us) log.cc:844] Serialized 26 byte log entry
0709 12:23:00.157302 (+2422046us) spinlock_profiling.cc:243] Waited 1.29 ms on lock 0x7b1800495280. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b47515e07 00007f7b503709a9 00007f7b4753269b 00007f7b4753093f 00007f7b47532c02 00007f7b4a1ba20b 00007f7b4a1bfd23 00007f7b4a1c00d1 00007f7b4a5dba0f
0709 12:23:00.436815 (+279513us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:23:00.436858 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:00.450598 (+ 13740us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:00.583778 (+133180us) raft_consensus.cc:1685] finished
0709 12:23:00.583800 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:00.594956 (+ 11156us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":310,"raft.queue_time_us":533,"raft.run_cpu_time_us":91,"raft.run_wall_time_us":91,"spinlock_wait_cycles":2956032,"thread_start_us":268401,"threads_started":1,"wal-append.queue_time_us":2843898}
I20260709 12:23:00.733523 22832 raft_consensus.cc:493] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:00.778509 18361 raft_consensus.cc:2368] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 14. Current term is 15.
I20260709 12:23:00.784557 18364 raft_consensus.cc:2468] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 6.
I20260709 12:23:00.749994 22652 raft_consensus.cc:740] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:23:00.788439 22898 raft_consensus.cc:493] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:00.750329 23036 log.cc:927] Time spent T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.095s	user 0.001s	sys 0.003s
I20260709 12:23:00.797853 17652 leader_election.cc:304] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:23:00.798482 23033 log.cc:927] Time spent T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.109s	user 0.000s	sys 0.004s
I20260709 12:23:00.756086 22536 raft_consensus.cc:740] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 13 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:23:00.630450 22476 consensus_queue.cc:237] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:00.659464 22973 consensus_queue.cc:237] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.7, Last appended by leader: 7, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:00.801539 18333 leader_election.cc:304] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:00.802423 22729 raft_consensus.cc:493] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:00.802803 22899 consensus_meta.cc:287] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.610s	user 0.011s	sys 0.004s
I20260709 12:23:00.776371 22878 raft_consensus.cc:515] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:00.805019 23018 leader_election.cc:290] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:23:00.805588 17803 connection.cc:256] RPC call timeout handler was delayed by -1.5109950609999032s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22260000000000002s for a response
I20260709 12:23:00.818719 22984 raft_consensus.cc:493] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
W20260709 12:23:00.809151 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 70104.7us
I20260709 12:23:00.771241 22923 raft_consensus.cc:493] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:00.728459 22877 raft_consensus.cc:515] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:00.800293 22981 consensus_meta.cc:287] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.590s	user 0.006s	sys 0.008s
W20260709 12:23:00.837381 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 52044.8us
W20260709 12:23:00.843983 17803 connection.cc:256] RPC call timeout handler was delayed by -1.3768619849997776s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16990000000000002s for a response
I20260709 12:23:00.778113 17727 leader_election.cc:400] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 13 election: Vote denied by peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837) with higher term. Message: Invalid argument: T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for earlier term 13. Current term is 14.
I20260709 12:23:00.846041 22913 raft_consensus.cc:493] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:00.852108 17727 leader_election.cc:403] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 13 election: Cancelling election due to peer responding with higher term
W20260709 12:23:00.851667 17803 connection.cc:256] RPC call timeout handler was delayed by -1.3568070150004132s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1836s for a response
W20260709 12:23:00.856006 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 117111us
I20260709 12:23:00.822944 22832 raft_consensus.cc:515] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:00.863695 17803 connection.cc:256] RPC call timeout handler was delayed by -1.2811568560000524s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.23210000000000003s for a response
I20260709 12:23:00.910544 22832 leader_election.cc:290] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:00.848193 22839 raft_consensus.cc:2749] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:23:00.850354 22729 raft_consensus.cc:515] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:00.911808 22853 leader_election.cc:290] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:00.913126 22729 leader_election.cc:290] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:00.853888 22981 raft_consensus.cc:515] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:00.848119 23026 raft_consensus.cc:2749] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:23:00.870383 22652 consensus_queue.cc:260] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.10, Last appended by leader: 10, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:00.874652 23011 consensus_queue.cc:1048] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:23:00.875341 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:23:00.875340 17804 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:00.876926 22984 raft_consensus.cc:515] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:00.836563 22898 raft_consensus.cc:515] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:00.857357 22913 raft_consensus.cc:515] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:00.960451 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 107143us
I20260709 12:23:00.858924 23051 consensus_queue.cc:1059] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 14.12, Next index: 13, Last known committed idx: 10, Time since last communication: 0.029s
I20260709 12:23:00.888497 22877 leader_election.cc:290] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 11 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:23:00.896181 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 94759.7us
I20260709 12:23:00.897352 22536 consensus_queue.cc:260] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.11, Last appended by leader: 11, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:00.957332 23054 log.cc:927] Time spent T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.050s	user 0.001s	sys 0.001s
I20260709 12:23:00.898532 22982 consensus_queue.cc:260] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:00.900388 17802 leader_election.cc:304] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:23:00.900708 17688 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37396 (request call id 2430) took 1023 ms. Trace:
I20260709 12:23:00.900954 22878 leader_election.cc:290] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:23:00.901835 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 104175us
I20260709 12:23:00.903017 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:00.859593 22923 raft_consensus.cc:515] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:00.911003 17803 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:00.974090 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 211908us
I20260709 12:23:00.870627 17801 leader_election.cc:304] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:00.972327 17688 rpcz_store.cc:276] 0709 12:22:59.877781 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:59.944054 (+ 66273us) service_pool.cc:224] Handling call
0709 12:22:59.944437 (+   383us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:00.435800 (+491363us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:00.900413 (+464613us) spinlock_profiling.cc:243] Waited 489 ms on lock 0x7b5400054bd4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:23:00.900611 (+   198us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1125681536}
I20260709 12:23:00.973410 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:00.974920 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:00.973940 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:00.971388 22536 raft_consensus.cc:3060] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:00.980252 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:00.981578 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:00.984416 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:00.853546 22965 raft_consensus.cc:515] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:23:00.984752 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 84429us
I20260709 12:23:00.991447 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:00.984161 17755 raft_consensus.cc:2468] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 12.
I20260709 12:23:00.993345 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:00.996086 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:00.996635 17727 leader_election.cc:304] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:23:00.997033 17761 raft_consensus.cc:3060] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:00.998023 17803 leader_election.cc:304] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:00.998834 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:00.999511 18367 raft_consensus.cc:2468] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 11.
I20260709 12:23:01.000105 23007 raft_consensus.cc:493] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:23:01.001327 23007 raft_consensus.cc:515] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:01.001749 22984 leader_election.cc:290] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:01.004683 22590 raft_consensus.cc:2749] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:23:01.005266 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:01.005935 22652 raft_consensus.cc:3060] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Advancing to term 14
W20260709 12:23:01.006261 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 135729us
I20260709 12:23:01.007928 23050 raft_consensus.cc:2749] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Leader election lost for term 13. Reason: Vote denied by peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837) with higher term. Message: Invalid argument: T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for earlier term 13. Current term is 14.
I20260709 12:23:01.008167 22890 raft_consensus.cc:2749] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:23:01.009200 22890 consensus_queue.cc:1048] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:23:01.009639 16501 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:23:01.011524 16501 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:23:01.008023 17685 raft_consensus.cc:2468] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 11.
I20260709 12:23:01.013710 23051 raft_consensus.cc:3055] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:23:01.014137 23051 raft_consensus.cc:740] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 14 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:23:01.014252 22854 raft_consensus.cc:493] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:01.015100 23051 consensus_queue.cc:260] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 14.12, Last appended by leader: 12, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:01.007710 17801 leader_election.cc:304] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:01.016232 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:01.015107 22854 raft_consensus.cc:515] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:00.996034 18363 raft_consensus.cc:3060] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:01.017277 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:01.018612 22981 leader_election.cc:290] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 13 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:01.018474 23048 consensus_queue.cc:1048] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:23:01.019492 22933 raft_consensus.cc:3055] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:23:01.020046 22982 raft_consensus.cc:3060] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:01.021071 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:01.022104 23007 leader_election.cc:290] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:01.011868 18364 raft_consensus.cc:1217] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Deduplicated request from leader. Original: 11.11->[13.12-14.13]   Dedup: 13.12->[14.13-14.13]
I20260709 12:23:01.012883 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:01.012028 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 15 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:01.011606 22898 leader_election.cc:290] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:01.048722 17802 leader_election.cc:304] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:23:01.049713 18365 raft_consensus.cc:2393] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 11: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:23:01.049541 17761 raft_consensus.cc:2468] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 15.
I20260709 12:23:01.061813 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:01.062002 18364 raft_consensus.cc:1240] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 14. Current term is 15. Ops: [13.12-14.13]
I20260709 12:23:01.060304 17756 raft_consensus.cc:3060] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:01.046391 18366 raft_consensus.cc:2368] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 12. Current term is 13.
I20260709 12:23:01.061691 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:01.063052 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:01.059473 18371 raft_consensus.cc:2468] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 13.
I20260709 12:23:01.045961 17835 raft_consensus.cc:2468] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 15.
I20260709 12:23:01.053488 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 9 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:01.063782 18374 raft_consensus.cc:2468] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 15.
I20260709 12:23:01.032663 22973 raft_consensus.cc:2804] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:23:01.063627 18361 raft_consensus.cc:2468] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 11.
I20260709 12:23:01.058941 18362 raft_consensus.cc:2468] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 10.
W20260709 12:23:01.065233 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 57765.8us
I20260709 12:23:01.055514 22933 raft_consensus.cc:740] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:23:01.090330 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:01.080217 17803 leader_election.cc:304] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:01.091224 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:01.112812 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:01.114888 18371 raft_consensus.cc:3060] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:01.064970 22722 raft_consensus.cc:2749] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Leader election lost for term 14. Reason: Vote denied by peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651) with higher term. Message: Invalid argument: T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 14. Current term is 15.
I20260709 12:23:01.115806 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:01.116343 18368 raft_consensus.cc:2468] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 9.
W20260709 12:23:01.118469 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 52948.1us
W20260709 12:23:01.119922 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 43925.1us
W20260709 12:23:01.120141 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 43608.5us
I20260709 12:23:01.124897 18367 raft_consensus.cc:2393] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 15: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:23:01.119360 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:01.118816 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:01.125619 22890 raft_consensus.cc:2749] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:23:01.085222 22898 raft_consensus.cc:2804] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:23:01.079298 23050 raft_consensus.cc:2804] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Leader pre-election won for term 12
W20260709 12:23:01.115061 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 153365us
I20260709 12:23:01.132218 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:01.135059 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:01.126487 23007 raft_consensus.cc:2804] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:23:01.142019 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:01.113129 22973 raft_consensus.cc:697] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 16 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:23:01.138921 18361 raft_consensus.cc:2393] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 10: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:23:01.139941 17801 leader_election.cc:304] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:01.136498 22899 consensus_queue.cc:1048] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.002s
W20260709 12:23:01.146438 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 149899us
W20260709 12:23:01.150080 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 93861.8us
I20260709 12:23:01.134249 23050 raft_consensus.cc:493] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:23:01.135133 23051 raft_consensus.cc:3060] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:23:01.229249 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 158084us
I20260709 12:23:01.143752 22898 raft_consensus.cc:697] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 15 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:23:01.231216 17831 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35040 (request call id 2610) took 2081 ms. Trace:
I20260709 12:23:01.231729 17831 rpcz_store.cc:276] 0709 12:22:59.149305 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:59.149522 (+   217us) service_pool.cc:224] Handling call
0709 12:22:59.149827 (+   305us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:59.156143 (+  6316us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 12:22:59.156290 (+   147us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:59.157433 (+  1143us) log.cc:844] Serialized 26 byte log entry
0709 12:23:00.899058 (+1741625us) spinlock_profiling.cc:243] Waited 3.96 ms on lock 0x7b1800306340. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b47515e07 00007f7b503709a9 00007f7b4753269b 00007f7b4753093f 00007f7b47532c02 00007f7b4a1ba20b 00007f7b4a1bfd23 00007f7b4a1c00d1 00007f7b4a5dba0f
0709 12:23:01.079807 (+180749us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:23:01.079861 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:01.079985 (+   124us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:01.230643 (+150658us) raft_consensus.cc:1685] finished
0709 12:23:01.230678 (+    35us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:01.230947 (+   269us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1277,"raft.run_cpu_time_us":157,"raft.run_wall_time_us":16552,"spinlock_wait_cycles":9118976,"thread_start_us":79644,"threads_started":1,"wal-append.queue_time_us":1983640}
I20260709 12:23:01.232971 22898 consensus_queue.cc:237] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 12.9, Last appended by leader: 9, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:01.234769 23068 raft_consensus.cc:493] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:01.141619 23007 raft_consensus.cc:493] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:01.244359 23007 raft_consensus.cc:3060] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:01.251684 23011 raft_consensus.cc:493] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:01.252595 23011 raft_consensus.cc:515] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:01.254165 22965 leader_election.cc:290] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 13 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:01.255159 23011 leader_election.cc:290] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:01.255688 17652 leader_election.cc:304] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:01.258387 17652 leader_election.cc:304] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:01.259503 22885 raft_consensus.cc:2749] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:23:01.261415 23022 consensus_queue.cc:1048] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:23:01.261745 22954 raft_consensus.cc:3060] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:01.265139 22973 consensus_queue.cc:237] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.8, Last appended by leader: 8, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:01.267107 22590 raft_consensus.cc:2804] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:23:01.267661 22590 raft_consensus.cc:493] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:01.268097 22590 raft_consensus.cc:3060] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:01.268733 22923 leader_election.cc:290] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:01.213256 23050 raft_consensus.cc:3060] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Advancing to term 12
W20260709 12:23:01.273541 17655 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:01.273753 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 43518.6us
I20260709 12:23:01.249181 22933 consensus_queue.cc:260] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:01.292264 17803 leader_election.cc:304] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:01.259722 17652 leader_election.cc:304] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:01.294378 17652 leader_election.cc:304] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:01.295440 22877 raft_consensus.cc:3060] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:01.296537 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:01.297185 17761 raft_consensus.cc:2468] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 15.
I20260709 12:23:01.298094 22899 raft_consensus.cc:493] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:01.298702 22899 raft_consensus.cc:515] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:01.299031 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:01.294651 23048 raft_consensus.cc:493] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:01.301468 22899 leader_election.cc:290] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:01.301158 23048 raft_consensus.cc:515] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:01.302141 22899 raft_consensus.cc:493] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:01.302806 22899 raft_consensus.cc:515] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:01.304289 23021 raft_consensus.cc:2804] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:23:01.304848 23021 raft_consensus.cc:493] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:01.305276 23021 raft_consensus.cc:3060] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:01.308286 23048 leader_election.cc:290] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:01.280966 22965 consensus_queue.cc:1059] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.028s
I20260709 12:23:01.235536 23068 raft_consensus.cc:515] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:01.312012 23068 leader_election.cc:290] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:01.312590 23068 raft_consensus.cc:3060] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:01.314428 22590 raft_consensus.cc:515] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:01.317610 22590 leader_election.cc:290] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 13 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:01.320098 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:01.320195 17762 raft_consensus.cc:1275] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 15 index: 10. (index mismatch)
I20260709 12:23:01.359827 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:01.360803 17762 raft_consensus.cc:2468] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 15.
I20260709 12:23:01.362165 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:01.364800 17762 raft_consensus.cc:2468] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 11.
I20260709 12:23:01.329072 17803 leader_election.cc:304] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:01.333250 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:01.368382 17836 raft_consensus.cc:3060] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:01.335230 22853 raft_consensus.cc:493] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:23:01.375234 22853 raft_consensus.cc:515] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:01.379014 22853 leader_election.cc:290] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:01.336319 17832 raft_consensus.cc:1275] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
I20260709 12:23:01.337807 17837 raft_consensus.cc:1275] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 13 index: 10. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
I20260709 12:23:01.309571 23011 raft_consensus.cc:493] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:01.339109 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:01.388876 17835 raft_consensus.cc:2468] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 7.
I20260709 12:23:01.339318 17834 raft_consensus.cc:1240] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 10. Current term is 11. Ops: []
I20260709 12:23:01.339871 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:01.390969 17831 raft_consensus.cc:3060] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:01.392124 23011 raft_consensus.cc:515] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:01.394855 23011 leader_election.cc:290] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:01.347934 22923 raft_consensus.cc:493] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:01.395747 22590 raft_consensus.cc:2764] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Leader pre-election decision vote started in defunct term 11: won
I20260709 12:23:01.396234 22923 raft_consensus.cc:515] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:01.328454 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:01.399173 17831 raft_consensus.cc:2468] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 13.
I20260709 12:23:01.400444 22923 leader_election.cc:290] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:23:01.401419 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 107116us
I20260709 12:23:01.402879 17803 leader_election.cc:304] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:01.404124 22590 raft_consensus.cc:2749] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:23:01.405040 22899 leader_election.cc:290] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:01.406270 17803 leader_election.cc:304] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:23:01.410833 17756 raft_consensus.cc:2468] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 15.
W20260709 12:23:01.346915 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 214194us
W20260709 12:23:01.465034 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 320494us
I20260709 12:23:01.466156 17801 leader_election.cc:304] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:23:01.467599 17801 leader_election.cc:304] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:01.291589 18371 raft_consensus.cc:2468] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 11.
I20260709 12:23:01.414139 18363 raft_consensus.cc:2468] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 11.
I20260709 12:23:01.432791 23007 raft_consensus.cc:515] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:01.434971 22590 raft_consensus.cc:2804] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:23:01.436527 17803 leader_election.cc:304] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:01.442122 22877 raft_consensus.cc:2749] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:23:01.442294 23050 raft_consensus.cc:515] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:23:01.455914 22652 consensus_queue.cc:1059] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 15.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 12:23:01.456746 17727 leader_election.cc:304] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:01.455669 22899 consensus_queue.cc:1059] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 14.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.003s
I20260709 12:23:01.458549 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:01.460206 17688 raft_consensus.cc:1240] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 10. Current term is 11. Ops: []
I20260709 12:23:01.460275 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:01.461305 17684 raft_consensus.cc:1240] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 14. Current term is 15. Ops: [14.11-14.11]
I20260709 12:23:01.461169 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:01.461687 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:01.461925 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:01.488719 17686 raft_consensus.cc:3060] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:01.488993 17680 raft_consensus.cc:3060] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:01.462124 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 13 candidate_status { last_received { term: 11 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:01.387560 17836 raft_consensus.cc:2468] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 11.
I20260709 12:23:01.390316 17761 raft_consensus.cc:2468] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 14.
I20260709 12:23:01.488752 17681 raft_consensus.cc:3060] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:01.494038 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:01.494550 23050 leader_election.cc:290] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 12 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:01.494874 17684 raft_consensus.cc:2468] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 10.
I20260709 12:23:01.494936 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:01.483003 22590 raft_consensus.cc:493] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:01.496035 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:01.496284 22590 raft_consensus.cc:3060] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:01.494277 23070 raft_consensus.cc:2749] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
W20260709 12:23:01.494072 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 57734.4us
I20260709 12:23:01.496330 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:01.504603 17681 raft_consensus.cc:2468] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 11.
I20260709 12:23:01.496765 17684 raft_consensus.cc:2468] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 7.
I20260709 12:23:01.505672 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:01.505843 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:01.506318 17681 raft_consensus.cc:2376] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 10. Re-sending same reply.
I20260709 12:23:01.491912 17763 raft_consensus.cc:2468] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 12.
I20260709 12:23:01.506916 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:01.506910 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:01.509150 17803 leader_election.cc:304] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:01.510095 17756 raft_consensus.cc:1275] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20260709 12:23:01.483208 22898 raft_consensus.cc:2804] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:23:01.514482 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:01.515491 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:01.492345 22476 raft_consensus.cc:2749] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:23:01.518006 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:23:01.517822 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 61151.9us
I20260709 12:23:01.516337 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
W20260709 12:23:01.516649 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 181255us
I20260709 12:23:01.519698 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:01.517829 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:01.522096 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:01.522850 17834 raft_consensus.cc:1240] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 10. Current term is 11. Ops: []
I20260709 12:23:01.521982 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 14 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:01.523927 17831 raft_consensus.cc:2410] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for term 14 because replica has last-logged OpId of term: 13 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 9.
I20260709 12:23:01.507946 22954 raft_consensus.cc:2749] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:23:01.525360 17686 raft_consensus.cc:2468] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 13.
I20260709 12:23:01.581286 17683 raft_consensus.cc:2468] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 15.
W20260709 12:23:01.582526 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 63505.1us
W20260709 12:23:01.584203 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 75175.3us
I20260709 12:23:01.589764 17680 raft_consensus.cc:2468] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 15.
I20260709 12:23:01.516314 23070 raft_consensus.cc:2749] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:23:01.605710 17834 raft_consensus.cc:1240] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 14. Current term is 15. Ops: [14.11-14.11]
I20260709 12:23:01.606515 17764 raft_consensus.cc:2468] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 11.
I20260709 12:23:01.607983 17688 raft_consensus.cc:1275] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20260709 12:23:01.609575 17682 raft_consensus.cc:2393] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 13: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:23:01.611783 17681 raft_consensus.cc:1240] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 10. Current term is 11. Ops: [8.8-10.9]
I20260709 12:23:01.581694 22898 raft_consensus.cc:697] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 14 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:23:01.607440 23056 raft_consensus.cc:2804] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:23:01.622165 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:01.604494 22923 raft_consensus.cc:493] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:01.520918 23021 raft_consensus.cc:515] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:01.624549 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:01.624534 17764 raft_consensus.cc:2468] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 15.
I20260709 12:23:01.632333 22923 raft_consensus.cc:515] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:01.641861 22933 raft_consensus.cc:3060] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:01.641001 22898 consensus_queue.cc:237] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.8, Last appended by leader: 8, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:01.648164 17756 raft_consensus.cc:2468] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 15.
I20260709 12:23:01.655552 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:01.660055 23007 leader_election.cc:290] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 14 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:23:01.664335 17727 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:01.665278 17687 raft_consensus.cc:2393] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 16: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:23:01.665911 17652 leader_election.cc:400] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 14 pre-election: Vote denied by peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837) with higher term. Message: Invalid argument: T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for earlier term 14. Current term is 15.
I20260709 12:23:01.666234 17652 leader_election.cc:403] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 14 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:23:01.667564 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:01.668154 17688 raft_consensus.cc:2468] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 13.
I20260709 12:23:01.668705 17803 leader_election.cc:304] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:23:01.669747 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:01.667209 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:01.662767 23056 raft_consensus.cc:697] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:23:01.672147 17762 raft_consensus.cc:2468] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 11.
I20260709 12:23:01.672056 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:01.673062 22921 raft_consensus.cc:2749] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Leader pre-election lost for term 14. Reason: Vote denied by peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837) with higher term. Message: Invalid argument: T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for earlier term 14. Current term is 15.
I20260709 12:23:01.673188 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:01.673820 17762 raft_consensus.cc:3060] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:01.674379 22854 leader_election.cc:290] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:01.677522 22923 leader_election.cc:290] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:01.680229 23021 leader_election.cc:290] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 16 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:01.681869 22305 raft_consensus.cc:493] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:01.682508 22305 raft_consensus.cc:515] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:01.682878 17686 raft_consensus.cc:1240] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 15. Current term is 16. Ops: []
I20260709 12:23:01.684250 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:01.684569 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:01.684712 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:01.685371 17832 raft_consensus.cc:2393] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 11: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:23:01.686066 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:01.686672 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:01.687402 17829 raft_consensus.cc:2393] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 14: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:23:01.684871 17682 raft_consensus.cc:3060] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:01.685344 17757 raft_consensus.cc:3060] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:01.620919 22954 consensus_queue.cc:1048] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.044s
I20260709 12:23:01.681782 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:01.694150 17762 raft_consensus.cc:2468] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 13.
I20260709 12:23:01.694321 18333 leader_election.cc:304] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:01.693614 17681 raft_consensus.cc:2468] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 15.
I20260709 12:23:01.696774 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 13 candidate_status { last_received { term: 11 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:01.697438 17834 raft_consensus.cc:2468] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 12.
I20260709 12:23:01.672868 23056 consensus_queue.cc:237] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.8, Last appended by leader: 8, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:01.700661 17803 leader_election.cc:400] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 14 election: Vote denied by peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577) with higher term. Message: Invalid argument: T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 14. Current term is 15.
I20260709 12:23:01.701135 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:01.702397 22965 raft_consensus.cc:2804] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:23:01.703121 22981 raft_consensus.cc:2749] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:23:01.692780 22590 raft_consensus.cc:515] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:01.703773 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:01.703861 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:01.747295 17681 raft_consensus.cc:3060] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:01.687906 22305 leader_election.cc:290] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:01.704217 23070 raft_consensus.cc:493] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:01.756141 23070 raft_consensus.cc:515] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:01.757110 22590 leader_election.cc:290] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 13 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:01.749416 23072 raft_consensus.cc:493] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:01.758052 23072 raft_consensus.cc:515] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:01.758313 23070 leader_election.cc:290] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:01.688819 23068 raft_consensus.cc:2749] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:23:01.705469 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:01.760653 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:01.761498 17838 raft_consensus.cc:2368] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for earlier term 12. Current term is 13.
I20260709 12:23:01.761219 22981 consensus_queue.cc:1059] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 13.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 12:23:01.745524 22965 raft_consensus.cc:493] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:01.762436 22965 raft_consensus.cc:3060] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:01.762501 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:01.684250 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:01.707213 22923 consensus_queue.cc:1059] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 12:23:01.765048 17837 raft_consensus.cc:3060] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:01.709280 22913 leader_election.cc:290] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:23:01.710364 23067 log.cc:927] Time spent T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.474s	user 0.003s	sys 0.002s
I20260709 12:23:01.710863 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:01.711613 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:01.711716 17682 raft_consensus.cc:2468] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 15.
I20260709 12:23:01.712472 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:01.713061 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:01.713732 17757 raft_consensus.cc:2468] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 13.
I20260709 12:23:01.714418 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:01.720584 17680 raft_consensus.cc:1275] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 15 index: 8. (index mismatch)
I20260709 12:23:01.725519 18334 leader_election.cc:304] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
W20260709 12:23:01.726651 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 69075.9us
I20260709 12:23:01.727634 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:01.734980 17689 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37402 (request call id 2779) took 1856 ms. Trace:
I20260709 12:23:01.739202 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:01.748805 17654 leader_election.cc:304] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:01.762739 17838 raft_consensus.cc:1275] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 13 index: 12. Preceding OpId from leader: term: 15 index: 13. (index mismatch)
I20260709 12:23:01.706422 17833 raft_consensus.cc:1240] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 13. Current term is 14. Ops: [13.11-13.11]
I20260709 12:23:01.788587 17836 raft_consensus.cc:2368] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 12. Current term is 13.
I20260709 12:23:01.786545 17832 raft_consensus.cc:2393] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 11: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:23:01.753324 17801 leader_election.cc:304] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:01.765774 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:01.810030 17763 raft_consensus.cc:2468] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 13.
I20260709 12:23:01.813522 17758 raft_consensus.cc:2368] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 12. Current term is 13.
I20260709 12:23:01.813650 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:23:01.814258 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 113747us
I20260709 12:23:01.813505 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:23:01.823660 17655 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:01.832337 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:01.833312 17835 raft_consensus.cc:2468] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 15.
W20260709 12:23:01.840937 17729 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:01.831588 17762 raft_consensus.cc:2468] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 13.
I20260709 12:23:01.787233 17689 rpcz_store.cc:276] 0709 12:22:59.878833 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:59.946880 (+ 68047us) service_pool.cc:224] Handling call
0709 12:22:59.947812 (+   932us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:59.971755 (+ 23943us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:59.971785 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:59.986960 (+ 15175us) log.cc:844] Serialized 26 byte log entry
0709 12:23:00.379179 (+392219us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:23:00.379225 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:01.734577 (+1355352us) spinlock_profiling.cc:243] Waited 21 ms on lock 0x7b54003467d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:23:01.734624 (+    47us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:01.734675 (+    51us) raft_consensus.cc:1685] finished
0709 12:23:01.734689 (+    14us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:01.734920 (+   231us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":2028,"raft.run_cpu_time_us":146,"raft.run_wall_time_us":145,"spinlock_wait_cycles":48316288,"thread_start_us":354821,"threads_started":1,"wal-append.queue_time_us":575159}
I20260709 12:23:01.860893 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:01.864173 17833 raft_consensus.cc:1240] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 10. Current term is 11. Ops: []
I20260709 12:23:01.866035 22915 raft_consensus.cc:3060] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:01.866120 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:01.868825 17652 leader_election.cc:304] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:23:01.870023 17652 leader_election.cc:304] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:23:01.870141 22854 consensus_queue.cc:1048] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 12:23:01.870499 22839 consensus_queue.cc:1048] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 12:23:01.870857 17652 leader_election.cc:304] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:23:01.871528 22854 raft_consensus.cc:2804] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:23:01.874905 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:01.874926 17757 raft_consensus.cc:2468] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 11.
I20260709 12:23:01.885552 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:23:01.890004 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 74910.3us
I20260709 12:23:01.892575 17757 raft_consensus.cc:2468] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 13.
W20260709 12:23:01.893055 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 139799us
I20260709 12:23:01.871696 22972 raft_consensus.cc:2804] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Leader election won for term 13
W20260709 12:23:01.892827 17728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:01.895012 17728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:01.816461 23070 raft_consensus.cc:2764] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Leader pre-election decision vote started in defunct term 10: won
I20260709 12:23:01.896361 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:01.898041 17764 raft_consensus.cc:2468] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 14.
I20260709 12:23:01.900940 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:01.901906 17757 raft_consensus.cc:2468] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 15.
I20260709 12:23:01.904310 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:01.894203 17801 leader_election.cc:304] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:23:01.908620 17803 leader_election.cc:304] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:01.924424 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:01.925082 17833 raft_consensus.cc:2468] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 14.
I20260709 12:23:01.925665 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:01.926331 17757 raft_consensus.cc:3060] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:01.884234 22854 raft_consensus.cc:493] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:01.871546 22998 consensus_queue.cc:1059] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 10.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.021s
I20260709 12:23:01.938817 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 16 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:01.938923 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:01.952720 17681 raft_consensus.cc:2468] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 13.
I20260709 12:23:01.937417 17801 leader_election.cc:304] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:23:01.955442 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 13 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:01.963817 17687 raft_consensus.cc:1240] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 14. Current term is 15. Ops: []
I20260709 12:23:01.966640 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:23:01.971215 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 245777us
I20260709 12:23:01.963496 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:01.973398 17836 raft_consensus.cc:2393] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 13: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:23:01.974467 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:01.955606 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:01.895355 22853 raft_consensus.cc:2804] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:23:01.960579 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: true dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:01.977326 17682 raft_consensus.cc:2393] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 15: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:23:01.976301 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:01.923645 22972 raft_consensus.cc:697] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 13 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:23:01.982105 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:01.975942 17689 raft_consensus.cc:2410] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for term 16 because replica has last-logged OpId of term: 14 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 10.
I20260709 12:23:01.943945 22854 raft_consensus.cc:3060] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:01.978679 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:01.983433 17832 raft_consensus.cc:2468] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 11.
I20260709 12:23:01.960994 22590 raft_consensus.cc:2804] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:23:01.993485 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 11 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:01.994378 17759 raft_consensus.cc:2393] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 11: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
W20260709 12:23:01.995818 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 58501us
I20260709 12:23:01.981004 17757 raft_consensus.cc:2468] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 14.
I20260709 12:23:01.962621 23070 raft_consensus.cc:2749] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:23:01.997002 17801 leader_election.cc:304] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:01.991973 23082 raft_consensus.cc:3055] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:23:01.998044 23082 raft_consensus.cc:740] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:23:01.999241 23082 consensus_queue.cc:260] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.11, Last appended by leader: 11, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:01.982685 17758 raft_consensus.cc:3060] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:02.003364 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:02.006234 17760 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38786 (request call id 3047) took 3434 ms. Trace:
I20260709 12:23:02.004082 17764 raft_consensus.cc:2468] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 15.
I20260709 12:23:02.006688 17760 rpcz_store.cc:276] 0709 12:22:58.567282 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:58.567439 (+   157us) service_pool.cc:224] Handling call
0709 12:22:58.567838 (+   399us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:00.715043 (+2147205us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:01.987037 (+1271994us) spinlock_profiling.cc:243] Waited 2.02 s on lock 0x7b54003e3fd4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:23:02.001755 (+ 14718us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4638154496}
I20260709 12:23:02.013238 17757 raft_consensus.cc:1275] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
W20260709 12:23:02.014232 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 311496us
I20260709 12:23:02.017552 17761 raft_consensus.cc:1275] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 15 index: 8. (index mismatch)
I20260709 12:23:01.990196 22590 raft_consensus.cc:493] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:23:02.025063 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 133958us
I20260709 12:23:02.026805 17764 raft_consensus.cc:1275] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
W20260709 12:23:02.032886 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 56091us
I20260709 12:23:02.041855 17757 raft_consensus.cc:1275] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 13 index: 10. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
I20260709 12:23:02.034021 17728 leader_election.cc:304] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:02.060834 17727 leader_election.cc:400] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 12 election: Vote denied by peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837) with higher term. Message: Invalid argument: T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for earlier term 12. Current term is 13.
I20260709 12:23:02.061290 17727 leader_election.cc:403] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 12 election: Cancelling election due to peer responding with higher term
I20260709 12:23:02.063632 17727 leader_election.cc:304] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:02.028002 22590 raft_consensus.cc:3060] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:02.071756 23056 raft_consensus.cc:2804] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:23:02.072456 23056 raft_consensus.cc:493] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:02.072927 23056 raft_consensus.cc:3060] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:02.100155 17652 leader_election.cc:304] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:02.102144 17758 raft_consensus.cc:2468] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 16.
W20260709 12:23:02.109089 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 66296.6us
I20260709 12:23:02.109973 18334 leader_election.cc:304] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
W20260709 12:23:02.110495 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 76598us
I20260709 12:23:02.110937 17803 leader_election.cc:400] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 12 election: Vote denied by peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577) with higher term. Message: Invalid argument: T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 12. Current term is 13.
I20260709 12:23:02.068501 22745 raft_consensus.cc:3060] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:02.065786 23068 raft_consensus.cc:2804] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Leader election won for term 13
W20260709 12:23:02.117131 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 120261us
I20260709 12:23:02.133143 17803 leader_election.cc:304] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:23:02.131625 17801 leader_election.cc:304] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:02.068305 22305 consensus_queue.cc:1048] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.048s
I20260709 12:23:02.116659 22972 consensus_queue.cc:237] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.7, Last appended by leader: 7, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:23:02.145800 17831 raft_consensus.cc:3060] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:02.136727 17728 leader_election.cc:304] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:02.116855 22899 raft_consensus.cc:2749] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 12:23:02.109391 22923 raft_consensus.cc:2749] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
W20260709 12:23:02.156292 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 68072.9us
I20260709 12:23:02.149670 17728 leader_election.cc:304] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:02.154321 17803 leader_election.cc:304] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:02.127475 22853 raft_consensus.cc:697] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 11 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:23:02.159585 17728 leader_election.cc:304] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:23:02.159960 17652 leader_election.cc:304] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:23:02.208945 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 49059.9us
I20260709 12:23:02.210251 17652 leader_election.cc:304] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:02.160277 17831 raft_consensus.cc:2468] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 16.
I20260709 12:23:02.160586 17803 leader_election.cc:304] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:23:02.159528 17801 leader_election.cc:304] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:23:02.192703 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 133107us
I20260709 12:23:02.188063 23046 raft_consensus.cc:2804] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:23:02.153460 23068 raft_consensus.cc:697] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 13 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:23:02.159719 22898 raft_consensus.cc:2804] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:23:02.156584 23076 raft_consensus.cc:2804] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:23:02.157284 23085 raft_consensus.cc:493] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:23:02.134379 22998 raft_consensus.cc:2764] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Leader pre-election decision vote started in defunct term 14: won
I20260709 12:23:02.237622 23085 raft_consensus.cc:515] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:02.238888 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 79443.6us
I20260709 12:23:02.239770 23087 raft_consensus.cc:2804] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:23:02.194283 23007 raft_consensus.cc:2804] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Leader election won for term 13
W20260709 12:23:02.229674 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 70179.9us
I20260709 12:23:02.229923 22982 raft_consensus.cc:3055] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:23:02.232025 23076 raft_consensus.cc:493] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:02.232693 23056 raft_consensus.cc:515] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:02.159386 23051 consensus_queue.cc:1059] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 15.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.046s
I20260709 12:23:02.234341 22853 consensus_queue.cc:237] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:02.225436 22898 raft_consensus.cc:493] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:23:02.239212 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 78721.6us
I20260709 12:23:02.190390 22890 raft_consensus.cc:2749] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:23:02.212814 18333 leader_election.cc:304] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:23:02.243463 17837 raft_consensus.cc:2468] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 10.
I20260709 12:23:02.182786 22839 consensus_queue.cc:1048] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.042s
I20260709 12:23:02.199980 23084 raft_consensus.cc:493] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:23:02.259272 17803 leader_election.cc:304] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:02.224167 23046 raft_consensus.cc:493] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:23:02.259263 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 46515us
I20260709 12:23:02.232053 23050 raft_consensus.cc:2804] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:23:02.204561 22923 consensus_queue.cc:1059] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 13.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.053s
I20260709 12:23:02.273375 23046 raft_consensus.cc:3060] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:02.273258 23080 raft_consensus.cc:2804] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:23:02.273320 23050 raft_consensus.cc:697] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:23:02.243542 23018 raft_consensus.cc:2804] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:23:02.282390 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:02.247668 22982 raft_consensus.cc:740] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 14 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:23:02.283217 17837 raft_consensus.cc:2468] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 14.
I20260709 12:23:02.247646 22921 raft_consensus.cc:2804] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:23:02.244601 23085 leader_election.cc:290] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:02.260524 22898 raft_consensus.cc:3060] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:02.249119 23076 raft_consensus.cc:3060] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:02.274035 22890 consensus_queue.cc:1059] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 14.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 12:23:02.271921 23088 raft_consensus.cc:2804] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Leader pre-election won for term 12
W20260709 12:23:02.317010 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 57838.9us
I20260709 12:23:02.286801 23018 raft_consensus.cc:493] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:02.288213 22921 raft_consensus.cc:493] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:23:02.321805 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:02.320657 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 80550us
I20260709 12:23:02.321388 17803 leader_election.cc:304] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:23:02.312235 22923 raft_consensus.cc:2804] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:23:02.314759 23085 raft_consensus.cc:2804] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:23:02.282843 23045 consensus_queue.cc:1059] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 14.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.037s
I20260709 12:23:02.382321 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:23:02.376924 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 76351.2us
I20260709 12:23:02.322003 22921 raft_consensus.cc:3060] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:02.314329 23088 raft_consensus.cc:493] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
W20260709 12:23:02.319805 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 209901us
I20260709 12:23:02.330639 23085 raft_consensus.cc:493] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:02.324530 23018 raft_consensus.cc:3060] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:02.280779 23084 raft_consensus.cc:515] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:02.401513 18362 raft_consensus.cc:2468] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 6.
I20260709 12:23:02.411733 18334 leader_election.cc:304] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:02.373683 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
W20260709 12:23:02.413151 17685 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37402 (request call id 2780) took 2444 ms (client timeout 3081 ms). Trace:
I20260709 12:23:02.331106 22923 raft_consensus.cc:493] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
W20260709 12:23:02.425768 17685 rpcz_store.cc:269] 0709 12:22:59.968918 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:59.972411 (+  3493us) service_pool.cc:224] Handling call
0709 12:23:02.412927 (+2440516us) spinlock_profiling.cc:243] Waited 1.03 s on lock 0x7b5400347bd4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:23:02.413076 (+   149us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2379027328}
W20260709 12:23:02.433177 17727 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:02.455909 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:02.457813 18363 raft_consensus.cc:1240] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 10. Current term is 11. Ops: []
I20260709 12:23:02.423863 23085 raft_consensus.cc:3060] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:02.415242 23088 raft_consensus.cc:3060] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:02.428990 22923 raft_consensus.cc:3060] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:02.322676 23068 consensus_queue.cc:237] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.9, Last appended by leader: 9, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:02.479990 22915 consensus_meta.cc:287] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.613s	user 0.013s	sys 0.000s
W20260709 12:23:02.488746 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 77091.5us
I20260709 12:23:02.498529 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 11 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:02.499349 18371 raft_consensus.cc:2393] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 11: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:23:02.427583 23050 consensus_queue.cc:237] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.9, Last appended by leader: 9, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:23:02.505486 23091 log.cc:927] Time spent T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.164s	user 0.000s	sys 0.001s
I20260709 12:23:02.473371 22476 consensus_queue.cc:1048] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.028s
W20260709 12:23:02.507489 23090 log.cc:927] Time spent T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.194s	user 0.000s	sys 0.001s
I20260709 12:23:02.512996 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:02.501338 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:02.479941 23093 raft_consensus.cc:493] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
W20260709 12:23:02.531541 23097 log.cc:927] Time spent T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.094s	user 0.000s	sys 0.000s
W20260709 12:23:02.491683 22933 consensus_meta.cc:287] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.833s	user 0.006s	sys 0.006s
I20260709 12:23:02.495091 23056 leader_election.cc:290] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 16 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:02.537328 17832 raft_consensus.cc:1240] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 16 LEADER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 15. Current term is 16. Ops: []
I20260709 12:23:02.489168 23108 raft_consensus.cc:493] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:02.568370 17654 leader_election.cc:304] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:02.492285 23101 raft_consensus.cc:2749] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:23:02.548898 18334 leader_election.cc:304] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:23:02.568466 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:02.503490 23100 raft_consensus.cc:3055] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:23:02.499507 22915 raft_consensus.cc:2749] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20260709 12:23:02.570822 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 249531us
I20260709 12:23:02.588025 17803 leader_election.cc:304] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:23:02.571506 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 245680us
I20260709 12:23:02.572348 23087 raft_consensus.cc:697] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 14 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:23:02.570188 17654 leader_election.cc:304] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:02.576256 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:02.592433 18371 raft_consensus.cc:3060] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Advancing to term 16
W20260709 12:23:02.577241 22854 consensus_meta.cc:287] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.580s	user 0.011s	sys 0.000s
I20260709 12:23:02.580806 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:02.581554 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:02.582227 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:02.572407 23093 raft_consensus.cc:515] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:02.603533 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:02.604332 18374 raft_consensus.cc:2468] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 15.
I20260709 12:23:02.573572 17832 raft_consensus.cc:1240] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 14. Current term is 15. Ops: []
I20260709 12:23:02.604516 23103 consensus_queue.cc:1048] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.095s
W20260709 12:23:02.606645 23094 log.cc:927] Time spent T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.134s	user 0.000s	sys 0.000s
I20260709 12:23:02.589516 18363 raft_consensus.cc:1240] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 9. Current term is 10. Ops: []
I20260709 12:23:02.586339 22626 raft_consensus.cc:2749] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
W20260709 12:23:02.608026 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 366501us
I20260709 12:23:02.608688 22626 raft_consensus.cc:2749] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:23:02.598913 18367 raft_consensus.cc:1240] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 15. Current term is 16. Ops: []
I20260709 12:23:02.611984 17652 leader_election.cc:304] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:02.598937 18361 raft_consensus.cc:2468] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 13.
I20260709 12:23:02.599501 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:02.569435 23099 raft_consensus.cc:2804] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:23:02.653674 17755 raft_consensus.cc:2468] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 10.
W20260709 12:23:02.672608 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 60720us
I20260709 12:23:02.587709 23100 raft_consensus.cc:740] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 14 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:23:02.570272 23007 raft_consensus.cc:697] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 13 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
W20260709 12:23:02.665091 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 77192.6us
I20260709 12:23:02.685501 17803 leader_election.cc:304] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:02.676954 17763 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38786 (request call id 3067) took 1113 ms. Trace:
W20260709 12:23:02.680049 17728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:02.692073 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:02.687220 17763 rpcz_store.cc:276] 0709 12:23:01.515101 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:02.000389 (+485288us) service_pool.cc:224] Handling call
0709 12:23:02.000792 (+   403us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:23:02.016442 (+ 15650us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:23:02.016492 (+    50us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:23:02.060652 (+ 44160us) log.cc:844] Serialized 52 byte log entry
0709 12:23:02.160508 (+ 99856us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:23:02.160551 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:02.160619 (+    68us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:02.627993 (+467374us) raft_consensus.cc:1685] finished
0709 12:23:02.628017 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:02.628204 (+   187us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1925,"raft.queue_time_us":10091,"raft.run_cpu_time_us":163,"raft.run_wall_time_us":7502,"thread_start_us":76221,"threads_started":1,"wal-append.queue_time_us":290067}
I20260709 12:23:02.679806 23083 raft_consensus.cc:2749] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 12:23:02.584682 23108 raft_consensus.cc:515] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:02.672891 23099 raft_consensus.cc:493] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:02.714874 18366 raft_consensus.cc:1275] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 13 index: 12. Preceding OpId from leader: term: 15 index: 13. (index mismatch)
I20260709 12:23:02.672564 22626 consensus_queue.cc:1048] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.029s
I20260709 12:23:02.712623 23102 raft_consensus.cc:493] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:23:02.713089 23104 consensus_queue.cc:1059] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.024s
I20260709 12:23:02.652247 22854 raft_consensus.cc:515] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:02.724179 18371 raft_consensus.cc:2468] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 16.
I20260709 12:23:02.725785 17836 raft_consensus.cc:1275] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 14 index: 8. Preceding OpId from leader: term: 15 index: 9. (index mismatch)
W20260709 12:23:02.712277 22965 consensus_meta.cc:287] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.949s	user 0.011s	sys 0.009s
W20260709 12:23:02.718016 23096 log.cc:927] Time spent T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.305s	user 0.003s	sys 0.003s
I20260709 12:23:02.740803 23120 raft_consensus.cc:2804] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:23:02.721673 23083 raft_consensus.cc:493] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:02.741824 23102 raft_consensus.cc:515] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:02.721397 23099 raft_consensus.cc:3060] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:23:02.743007 22745 consensus_meta.cc:287] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.613s	user 0.010s	sys 0.007s
W20260709 12:23:02.799381 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 123803us
I20260709 12:23:02.804966 17652 leader_election.cc:304] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:02.796690 23084 leader_election.cc:290] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:02.798800 22745 raft_consensus.cc:2749] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Leader election lost for term 12. Reason: Vote denied by peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837) with higher term. Message: Invalid argument: T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for earlier term 12. Current term is 13.
I20260709 12:23:02.774224 22965 raft_consensus.cc:515] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:02.707647 23087 consensus_queue.cc:237] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:02.817665 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:02.824412 18371 raft_consensus.cc:2410] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for term 16 because replica has last-logged OpId of term: 14 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 10.
I20260709 12:23:02.801971 23116 raft_consensus.cc:2804] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:23:02.795544 23083 raft_consensus.cc:515] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:02.841506 17727 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:02.821798 23118 raft_consensus.cc:493] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:02.808415 23120 raft_consensus.cc:697] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 11 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
W20260709 12:23:02.860713 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 311354us
I20260709 12:23:02.860438 22745 raft_consensus.cc:493] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:02.867729 22745 raft_consensus.cc:515] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:02.868331 22854 leader_election.cc:290] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 8 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:02.870343 22745 leader_election.cc:290] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:02.871124 23114 consensus_queue.cc:1048] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.150s
I20260709 12:23:02.873754 23112 raft_consensus.cc:493] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:02.874396 23112 raft_consensus.cc:515] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:02.874699 23018 raft_consensus.cc:515] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:02.876163 23106 raft_consensus.cc:2749] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:23:02.876585 23120 consensus_queue.cc:237] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:02.879513 23093 leader_election.cc:290] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:02.880386 18334 leader_election.cc:304] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:23:02.881035 23124 raft_consensus.cc:493] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:02.881624 23124 raft_consensus.cc:515] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:02.882750 18334 leader_election.cc:304] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:23:02.883733 23104 raft_consensus.cc:2804] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:23:02.884227 23104 raft_consensus.cc:493] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:02.884673 23104 raft_consensus.cc:3060] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:23:02.885426 22965 leader_election.cc:290] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 8 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:02.795712 23100 consensus_queue.cc:260] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.9, Last appended by leader: 9, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:02.887585 23100 raft_consensus.cc:3060] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:02.792455 23007 consensus_queue.cc:237] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:02.894845 23072 leader_election.cc:290] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:02.884479 18334 leader_election.cc:304] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:02.897816 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:02.898555 18362 raft_consensus.cc:2468] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 13.
I20260709 12:23:02.911600 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 16 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:02.895668 23121 raft_consensus.cc:2804] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Leader election won for term 13
W20260709 12:23:02.919852 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 115011us
I20260709 12:23:02.860852 23084 raft_consensus.cc:2804] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:23:02.932006 17652 leader_election.cc:304] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:23:02.938966 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 286463us
I20260709 12:23:02.865664 23116 raft_consensus.cc:697] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 16 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
W20260709 12:23:02.951509 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 61175.4us
I20260709 12:23:02.954188 22853 raft_consensus.cc:2749] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
W20260709 12:23:02.955106 18333 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.16.29.67:36837 timed out after 1.797s (SENT) [suppressed 20 similar messages]
I20260709 12:23:02.956624 17652 leader_election.cc:304] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:23:02.907249 23109 raft_consensus.cc:493] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:02.914709 22745 raft_consensus.cc:493] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:02.881407 22923 raft_consensus.cc:515] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:02.926877 23096 log.cc:927] Time spent T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.064s	user 0.000s	sys 0.000s
I20260709 12:23:02.946825 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:02.971935 18367 raft_consensus.cc:2468] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 14.
I20260709 12:23:02.921093 23093 raft_consensus.cc:493] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:23:02.958112 22853 raft_consensus.cc:2804] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:23:02.937603 23084 raft_consensus.cc:493] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:23:02.898620 23118 raft_consensus.cc:515] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:02.978437 23124 leader_election.cc:290] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:02.990854 18361 raft_consensus.cc:1240] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 10. Current term is 11. Ops: []
I20260709 12:23:02.989038 23018 leader_election.cc:290] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 16 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:02.978916 22853 raft_consensus.cc:493] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:02.990224 22923 leader_election.cc:290] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 11 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:02.983150 23084 raft_consensus.cc:3060] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:03.010140 22973 raft_consensus.cc:3055] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:23:02.981977 23093 raft_consensus.cc:515] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:03.005308 23124 raft_consensus.cc:3060] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:23:03.032639 23125 log.cc:927] Time spent T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.052s	user 0.001s	sys 0.000s
I20260709 12:23:03.033855 23093 leader_election.cc:290] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:03.033953 23116 consensus_queue.cc:237] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 14.12, Last appended by leader: 12, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:03.003559 23127 consensus_queue.cc:1048] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.046s
I20260709 12:23:03.037484 23128 consensus_queue.cc:1059] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 10.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.008s
I20260709 12:23:02.981129 23109 raft_consensus.cc:515] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:03.029657 22923 raft_consensus.cc:493] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:03.042085 23112 leader_election.cc:290] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:03.007925 23108 leader_election.cc:290] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:23:03.042528 23119 log.cc:927] Time spent T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.289s	user 0.001s	sys 0.001s
I20260709 12:23:03.045971 23124 raft_consensus.cc:2749] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:23:03.039692 22973 raft_consensus.cc:740] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 10 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:23:03.049197 22973 consensus_queue.cc:260] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 10.11, Last appended by leader: 11, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:03.049706 23118 leader_election.cc:290] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:03.050287 22973 raft_consensus.cc:3060] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:03.012476 22853 raft_consensus.cc:3060] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:03.042140 22923 raft_consensus.cc:515] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:03.041648 23109 leader_election.cc:290] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:03.061878 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:03.062794 17833 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 12.
I20260709 12:23:03.068871 22923 leader_election.cc:290] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:03.039076 23048 raft_consensus.cc:3055] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:23:03.069949 23048 raft_consensus.cc:740] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:23:03.070996 23048 consensus_queue.cc:260] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.8, Last appended by leader: 8, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:03.071925 23011 raft_consensus.cc:3055] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:23:03.072389 23011 raft_consensus.cc:740] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:23:03.071012 17830 raft_consensus.cc:1240] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 15. Current term is 16. Ops: []
W20260709 12:23:03.067593 17831 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37994 (request call id 1022) took 1409 ms (client timeout 1846 ms). Trace:
W20260709 12:23:03.074965 17831 rpcz_store.cc:269] 0709 12:23:01.658572 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:01.659080 (+   508us) service_pool.cc:224] Handling call
0709 12:23:03.067303 (+1408223us) spinlock_profiling.cc:243] Waited 473 ms on lock 0x7b54000893d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:23:03.067492 (+   189us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1087476992}
I20260709 12:23:02.975912 22745 raft_consensus.cc:515] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:03.077090 17831 raft_consensus.cc:1240] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 10. Current term is 11. Ops: []
W20260709 12:23:03.088200 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 203778us
I20260709 12:23:03.091267 18334 leader_election.cc:304] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
W20260709 12:23:03.009501 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 323911us
I20260709 12:23:03.079165 17654 leader_election.cc:304] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:03.092962 17803 leader_election.cc:304] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:23:03.106220 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 151143us
I20260709 12:23:03.107266 18333 leader_election.cc:304] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
W20260709 12:23:03.092774 23046 consensus_meta.cc:287] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.782s	user 0.007s	sys 0.013s
I20260709 12:23:03.092861 23048 raft_consensus.cc:3060] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:03.110522 18334 leader_election.cc:304] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:23:03.123493 17829 raft_consensus.cc:1240] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 10. Current term is 11. Ops: [10.10-10.10]
W20260709 12:23:03.127347 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 48274.2us
I20260709 12:23:03.129208 18333 leader_election.cc:304] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:03.115079 22923 raft_consensus.cc:2804] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:23:03.105847 23121 raft_consensus.cc:697] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 13 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:23:03.120131 23131 raft_consensus.cc:493] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:03.138808 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 178951us
I20260709 12:23:03.140635 18333 leader_election.cc:400] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 12 pre-election: Vote denied by peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837) with higher term. Message: Invalid argument: T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 12. Current term is 13.
I20260709 12:23:03.141057 18333 leader_election.cc:403] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 12 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:23:03.131466 23122 raft_consensus.cc:2749] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:23:03.132258 23109 raft_consensus.cc:2749] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20260709 12:23:03.137152 22898 consensus_meta.cc:287] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.822s	user 0.005s	sys 0.010s
I20260709 12:23:03.133116 22923 raft_consensus.cc:493] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:03.130651 23135 raft_consensus.cc:2749] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:23:03.136508 23118 raft_consensus.cc:2804] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Leader election won for term 15
W20260709 12:23:03.168092 17728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:03.159344 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:03.169570 18371 raft_consensus.cc:3060] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:03.127787 23046 raft_consensus.cc:515] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:03.166713 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:03.174356 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:03.165560 22923 raft_consensus.cc:3060] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:03.140733 23124 consensus_queue.cc:1048] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.032s
I20260709 12:23:03.192195 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:03.192911 23082 raft_consensus.cc:3060] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:03.193643 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:03.194434 17829 raft_consensus.cc:2393] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 8: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:23:03.196826 23121 consensus_queue.cc:237] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:03.147351 23131 raft_consensus.cc:515] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:03.211167 18371 raft_consensus.cc:2468] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 16.
I20260709 12:23:03.195231 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:03.180078 22517 raft_consensus.cc:493] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:03.162580 22898 raft_consensus.cc:515] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:03.180887 23118 raft_consensus.cc:697] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 15 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
W20260709 12:23:03.214257 23076 consensus_meta.cc:287] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.894s	user 0.012s	sys 0.006s
I20260709 12:23:03.206411 23083 leader_election.cc:290] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:03.232683 23076 raft_consensus.cc:515] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:03.224838 17728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:03.218233 23085 consensus_meta.cc:287] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.727s	user 0.013s	sys 0.005s
I20260709 12:23:03.234319 22745 leader_election.cc:290] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:23:03.228415 22590 consensus_meta.cc:287] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 1.133s	user 0.024s	sys 0.004s
W20260709 12:23:03.240670 23130 log.cc:927] Time spent T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.111s	user 0.001s	sys 0.000s
I20260709 12:23:03.226748 22517 raft_consensus.cc:515] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:03.241240 23141 raft_consensus.cc:3060] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Advancing to term 13
I20260709 12:23:03.243417 23118 consensus_queue.cc:237] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.12, Last appended by leader: 12, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:03.222579 17829 raft_consensus.cc:2393] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 13: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:23:03.251788 22517 leader_election.cc:290] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:03.253072 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:03.253906 17838 raft_consensus.cc:2393] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 15: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:23:03.219558 23099 raft_consensus.cc:515] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:03.254731 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:03.255254 17838 raft_consensus.cc:2376] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 10. Re-sending same reply.
I20260709 12:23:03.255810 23099 leader_election.cc:290] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:03.256626 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 10 candidate_status { last_received { term: 9 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:23:03.257225 22921 consensus_meta.cc:287] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.825s	user 0.008s	sys 0.008s
I20260709 12:23:03.258096 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 16 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:03.257927 22921 raft_consensus.cc:515] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:03.259159 17756 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36572 (request call id 896) took 1746 ms (client timeout 1983 ms). Trace:
I20260709 12:23:03.257048 22982 consensus_queue.cc:260] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:23:03.259603 17756 rpcz_store.cc:269] 0709 12:23:01.512607 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:01.514225 (+  1618us) service_pool.cc:224] Handling call
0709 12:23:03.258984 (+1744759us) spinlock_profiling.cc:243] Waited 121 ms on lock 0x7b54003e44d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:23:03.259078 (+    94us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":278149376}
I20260709 12:23:03.260336 22982 raft_consensus.cc:3060] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:03.261899 17683 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37396 (request call id 2448) took 1321 ms. Trace:
I20260709 12:23:03.262280 17683 rpcz_store.cc:276] 0709 12:23:01.940062 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:01.940182 (+   120us) service_pool.cc:224] Handling call
0709 12:23:03.261687 (+1321505us) spinlock_profiling.cc:243] Waited 403 ms on lock 0x7b5400207954. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:23:03.261819 (+   132us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":926571776}
I20260709 12:23:03.266322 23080 raft_consensus.cc:493] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:03.266848 23080 raft_consensus.cc:3060] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Advancing to term 12
W20260709 12:23:03.272038 17728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:03.273334 23131 leader_election.cc:290] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:23:03.274828 23089 log.cc:927] Time spent T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.957s	user 0.000s	sys 0.003s
I20260709 12:23:03.246783 23085 raft_consensus.cc:515] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:03.280375 23098 log.cc:927] Time spent T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.865s	user 0.000s	sys 0.003s
I20260709 12:23:03.281834 18364 raft_consensus.cc:1240] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 12. Current term is 13. Ops: [12.13-12.13]
I20260709 12:23:03.282819 17833 raft_consensus.cc:1240] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 9. Current term is 10. Ops: []
I20260709 12:23:03.283255 22898 leader_election.cc:290] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 12 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:03.249800 22590 raft_consensus.cc:515] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:23:03.285168 23088 consensus_meta.cc:287] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.791s	user 0.006s	sys 0.009s
I20260709 12:23:03.285704 23088 raft_consensus.cc:515] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:03.286365 22590 leader_election.cc:290] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 16 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:03.286938 17836 raft_consensus.cc:1240] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 11. Current term is 12. Ops: []
I20260709 12:23:03.287075 23046 leader_election.cc:290] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 16 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:03.287789 17835 raft_consensus.cc:1240] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 15. Current term is 16. Ops: []
I20260709 12:23:03.288915 17835 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35040 (request call id 2638) took 1316 ms. Trace:
I20260709 12:23:03.289122 17836 raft_consensus.cc:1275] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20260709 12:23:03.289278 17835 rpcz_store.cc:276] 0709 12:23:01.972965 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:01.977804 (+  4839us) service_pool.cc:224] Handling call
0709 12:23:01.978058 (+   254us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:03.288203 (+1310145us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:03.288699 (+   496us) spinlock_profiling.cc:243] Waited 1.31 s on lock 0x7b54003340d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:23:03.288859 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3011705088}
I20260709 12:23:03.294037 23076 leader_election.cc:290] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 14 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:03.318042 17584 catalog_manager.cc:5697] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb reported cstate change: term changed from 4 to 5, leader changed from 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) to e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66). New cstate: current_term: 5 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: UNKNOWN } } }
I20260709 12:23:03.295837 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 10 candidate_status { last_received { term: 9 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:03.303853 23083 raft_consensus.cc:493] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:03.304131 17757 raft_consensus.cc:1240] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 11. Current term is 12. Ops: []
I20260709 12:23:03.359278 17755 raft_consensus.cc:1240] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 15. Current term is 16. Ops: [15.10-15.10]
I20260709 12:23:03.304807 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:03.360575 17764 raft_consensus.cc:2468] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 14.
I20260709 12:23:03.309448 23077 raft_consensus.cc:3055] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:23:03.312572 22921 leader_election.cc:290] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 12 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:03.321188 17762 raft_consensus.cc:1275] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 15 index: 8. Preceding OpId from leader: term: 16 index: 9. (index mismatch)
I20260709 12:23:03.367923 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:03.328375 17756 raft_consensus.cc:1275] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 12 index: 8. Preceding OpId from leader: term: 14 index: 9. (index mismatch)
I20260709 12:23:03.370375 18361 raft_consensus.cc:2468] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 11.
I20260709 12:23:03.328860 18333 leader_election.cc:304] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:23:03.329633 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:03.295039 23102 leader_election.cc:290] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:03.282317 23085 leader_election.cc:290] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 13 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:23:03.331831 17801 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:03.332755 17834 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35042 (request call id 2377) took 1628 ms. Trace:
I20260709 12:23:03.375754 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:03.334882 23084 raft_consensus.cc:515] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:03.393654 23084 leader_election.cc:290] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 15 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:03.394443 23141 raft_consensus.cc:2749] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Leader pre-election lost for term 12. Reason: Vote denied by peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837) with higher term. Message: Invalid argument: T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 12. Current term is 13.
I20260709 12:23:03.345049 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:03.348975 17835 raft_consensus.cc:1275] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 13 index: 8. (index mismatch)
I20260709 12:23:03.365274 17680 raft_consensus.cc:1240] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 14. Current term is 15. Ops: []
I20260709 12:23:03.366268 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 13 candidate_status { last_received { term: 11 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:03.366295 17684 raft_consensus.cc:1240] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 15. Current term is 16. Ops: []
I20260709 12:23:03.373692 17688 raft_consensus.cc:1240] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 11 LEADER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 10. Current term is 11. Ops: [10.10-10.10]
I20260709 12:23:03.373836 22853 raft_consensus.cc:515] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:03.375166 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:03.377066 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:03.379523 17801 leader_election.cc:304] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:23:03.380590 17727 leader_election.cc:304] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:23:03.380686 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 51920.3us
I20260709 12:23:03.374727 23135 raft_consensus.cc:493] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:03.381687 22476 raft_consensus.cc:493] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:03.383361 17729 leader_election.cc:304] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:03.376050 17834 rpcz_store.cc:276] 0709 12:23:01.704459 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:01.728123 (+ 23664us) service_pool.cc:224] Handling call
0709 12:23:01.728441 (+   318us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:01.748235 (+ 19794us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:01.748278 (+    43us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:01.764334 (+ 16056us) log.cc:844] Serialized 26 byte log entry
0709 12:23:02.111805 (+347471us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:23:02.111864 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:02.111975 (+   111us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:03.274110 (+1162135us) spinlock_profiling.cc:243] Waited 2.25 ms on lock 0x7b4400022b4c. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b47459c85 00007f7b4745a6a2 00007f7b4a1cc1db 00007f7b4a1c7aaf 00007f7b4a1bb86e 00007f7b4a1babb6 00007f7b4a1ba7d8 00007f7b4a1c4221 00007f7b4a1c41d9 00007f7b4a1c4169
0709 12:23:03.332460 (+ 58350us) raft_consensus.cc:1685] finished
0709 12:23:03.332484 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:03.332699 (+   215us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":24890,"raft.queue_time_us":20458,"raft.run_cpu_time_us":109,"raft.run_wall_time_us":111,"spinlock_wait_cycles":5185792,"thread_start_us":346769,"threads_started":1,"wal-append.queue_time_us":504823}
I20260709 12:23:03.460474 17837 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35042 (request call id 2388) took 1402 ms. Trace:
I20260709 12:23:03.384351 17681 raft_consensus.cc:1240] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 11. Current term is 12. Ops: [11.10-11.10]
I20260709 12:23:03.374863 23104 raft_consensus.cc:515] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:03.384835 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:03.384747 22954 consensus_queue.cc:1059] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 10.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:23:03.386703 23046 raft_consensus.cc:493] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:03.387290 23076 raft_consensus.cc:493] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:03.387826 22652 consensus_queue.cc:1059] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 9.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:23:03.436494 22854 consensus_queue.cc:1048] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.032s
I20260709 12:23:03.388062 22745 consensus_queue.cc:1059] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 15.12, Next index: 13, Last known committed idx: 11, Time since last communication: 0.002s
I20260709 12:23:03.460539 23048 raft_consensus.cc:2804] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:23:03.436272 22839 consensus_queue.cc:1059] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 11.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.035s
I20260709 12:23:03.445559 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:03.367583 23077 raft_consensus.cc:740] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:23:03.389902 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:03.391147 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:03.397193 23088 leader_election.cc:290] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 12 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:03.370137 23083 raft_consensus.cc:515] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:03.403024 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:03.404021 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:03.404327 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:03.467388 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:03.404960 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:03.470032 17681 raft_consensus.cc:1275] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 15 index: 10. (index mismatch)
I20260709 12:23:03.404598 23108 raft_consensus.cc:2804] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:23:03.406541 17830 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35042 (request call id 2380) took 1701 ms. Trace:
I20260709 12:23:03.421222 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:03.429472 18333 leader_election.cc:304] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:23:03.478473 17830 rpcz_store.cc:276] 0709 12:23:01.705549 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:01.706717 (+  1168us) service_pool.cc:224] Handling call
0709 12:23:03.070746 (+1364029us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:03.073771 (+  3025us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:03.406319 (+332548us) spinlock_profiling.cc:243] Waited 1.36 s on lock 0x7b54000893d0. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a61f8ad 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:23:03.406476 (+   157us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3136229120}
W20260709 12:23:03.476320 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 96882.2us
W20260709 12:23:03.482235 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 52948.2us
I20260709 12:23:03.468590 22652 raft_consensus.cc:3055] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:23:03.483050 18333 leader_election.cc:304] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:23:03.483465 22652 raft_consensus.cc:740] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:23:03.421762 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:03.483836 23111 raft_consensus.cc:2804] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:23:03.484164 17680 raft_consensus.cc:3060] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:03.484342 23111 raft_consensus.cc:493] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:03.484848 23111 raft_consensus.cc:3060] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:03.484493 22652 consensus_queue.cc:260] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:03.485940 22652 raft_consensus.cc:3060] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:03.465291 22890 raft_consensus.cc:2749] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:23:03.472316 22982 raft_consensus.cc:2749] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:23:03.488745 17758 raft_consensus.cc:1275] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 13 index: 8. (index mismatch)
I20260709 12:23:03.462074 17682 raft_consensus.cc:2468] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 15.
I20260709 12:23:03.460938 17837 rpcz_store.cc:276] 0709 12:23:02.058398 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:02.430073 (+371675us) service_pool.cc:224] Handling call
0709 12:23:02.430408 (+   335us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:02.592079 (+161671us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:02.592113 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:02.600072 (+  7959us) log.cc:844] Serialized 26 byte log entry
0709 12:23:02.960339 (+360267us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:23:02.970065 (+  9726us) log.cc:844] Serialized 14 byte log entry
0709 12:23:03.460162 (+490097us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:03.460244 (+    82us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:03.460284 (+    40us) raft_consensus.cc:1685] finished
0709 12:23:03.460297 (+    13us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:03.460418 (+   121us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":515692,"prepare.queue_time_us":13418,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":41,"raft.queue_time_us":77474,"raft.run_cpu_time_us":83,"raft.run_wall_time_us":88,"thread_start_us":400805,"threads_started":2,"wal-append.queue_time_us":301172}
W20260709 12:23:03.459563 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 76268.8us
I20260709 12:23:03.490242 17729 leader_election.cc:304] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:03.460254 18363 raft_consensus.cc:3060] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [term 11 LEARNER]: Advancing to term 13
W20260709 12:23:03.462998 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 370103us
W20260709 12:23:03.446141 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 65610.3us
I20260709 12:23:03.463687 22476 raft_consensus.cc:515] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:03.465787 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:03.476934 18368 raft_consensus.cc:3060] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:03.472177 18367 raft_consensus.cc:3060] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:03.473448 17686 raft_consensus.cc:2393] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 16: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:23:03.472883 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:03.259168 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 15 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:03.472775 17684 raft_consensus.cc:1240] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 14. Current term is 15. Ops: []
I20260709 12:23:03.465236 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:03.480983 17832 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35042 (request call id 2393) took 1219 ms. Trace:
I20260709 12:23:03.472589 17835 raft_consensus.cc:2468] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 11.
I20260709 12:23:03.491339 17654 leader_election.cc:304] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:23:03.522541 17832 rpcz_store.cc:276] 0709 12:23:02.261820 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:02.606739 (+344919us) service_pool.cc:224] Handling call
0709 12:23:02.607077 (+   338us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:02.679477 (+ 72400us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 12:23:02.692071 (+ 12594us) log.cc:844] Serialized 14 byte log entry
0709 12:23:03.356616 (+664545us) spinlock_profiling.cc:243] Waited 6.29 ms on lock 0x7b1800481840. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b474adc0a 00007f7b50bed219 00007f7b4e4f458f 00007f7b47533571 00007f7b475357c1 00007f7b47535779 00007f7b47535709 00007f7b475356d1 00007f7b475349cd
0709 12:23:03.407045 (+ 50429us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:03.407541 (+   496us) log.cc:844] Serialized 26 byte log entry
0709 12:23:03.407698 (+   157us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:23:03.407734 (+    36us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:03.407800 (+    66us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:03.472583 (+ 64783us) raft_consensus.cc:1685] finished
0709 12:23:03.480774 (+  8191us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:03.480914 (+   140us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":719938,"prepare.queue_time_us":13728,"prepare.run_cpu_time_us":67,"prepare.run_wall_time_us":2115,"raft.queue_time_us":710822,"raft.run_cpu_time_us":368,"raft.run_wall_time_us":25810,"spinlock_wait_cycles":14465920,"thread_start_us":92901,"threads_started":1,"wal-append.queue_time_us":199260}
I20260709 12:23:03.524446 17803 leader_election.cc:304] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:23:03.526197 17687 raft_consensus.cc:3060] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:03.527101 17831 raft_consensus.cc:2410] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for term 15 because replica has last-logged OpId of term: 13 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 10.
I20260709 12:23:03.527664 18362 raft_consensus.cc:2468] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 14.
I20260709 12:23:03.476428 17833 raft_consensus.cc:2468] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 15.
I20260709 12:23:03.531519 17680 raft_consensus.cc:2468] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 16.
W20260709 12:23:03.532224 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 54130.5us
W20260709 12:23:03.533607 23152 log.cc:927] Time spent T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.059s	user 0.000s	sys 0.000s
I20260709 12:23:03.533797 23011 consensus_queue.cc:260] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 10.8, Last appended by leader: 8, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:03.535117 23011 raft_consensus.cc:3060] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:03.535295 23077 consensus_queue.cc:260] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.10, Last appended by leader: 10, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:03.536070 23095 raft_consensus.cc:2804] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:23:03.536480 23077 raft_consensus.cc:3060] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:03.537504 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:03.478462 23135 raft_consensus.cc:515] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:03.541716 23083 leader_election.cc:290] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:03.465874 22853 leader_election.cc:290] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 12 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:03.542578 23135 leader_election.cc:290] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:23:03.543423 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 60856.3us
I20260709 12:23:03.543712 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 13 candidate_status { last_received { term: 11 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
W20260709 12:23:03.545828 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 55694.8us
I20260709 12:23:03.547078 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:03.536514 23095 raft_consensus.cc:493] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:03.550519 17833 raft_consensus.cc:1240] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 12. Current term is 13. Ops: []
I20260709 12:23:03.519115 18333 leader_election.cc:304] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:03.464548 23121 raft_consensus.cc:493] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:03.555387 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:03.556533 17756 raft_consensus.cc:2393] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 12: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:23:03.555441 23121 raft_consensus.cc:515] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:03.469552 23048 raft_consensus.cc:493] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:03.565618 17681 raft_consensus.cc:1275] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 14 index: 8. Preceding OpId from leader: term: 15 index: 9. (index mismatch)
I20260709 12:23:03.557716 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:03.567082 17756 raft_consensus.cc:3060] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:03.468990 22745 raft_consensus.cc:3055] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:23:03.568496 17729 leader_election.cc:304] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:03.569370 23056 raft_consensus.cc:2749] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:23:03.471285 23104 leader_election.cc:290] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 7 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:03.464982 23133 raft_consensus.cc:2749] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:23:03.479174 23108 raft_consensus.cc:493] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:03.562546 22933 raft_consensus.cc:2749] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20260709 12:23:03.573760 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 54782.3us
I20260709 12:23:03.574215 23118 raft_consensus.cc:2749] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:23:03.467612 23046 raft_consensus.cc:515] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:03.570999 23048 raft_consensus.cc:3060] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:03.470624 22923 raft_consensus.cc:515] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:03.682204 18371 raft_consensus.cc:2393] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 16: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:23:03.681044 23080 raft_consensus.cc:515] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:03.684681 22923 leader_election.cc:290] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 16 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:03.686009 17830 raft_consensus.cc:1240] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 9. Current term is 10. Ops: []
I20260709 12:23:03.685863 22923 consensus_queue.cc:1048] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:23:03.688315 17829 raft_consensus.cc:1240] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 11. Current term is 12. Ops: []
I20260709 12:23:03.688611 18333 leader_election.cc:304] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:03.574448 23121 leader_election.cc:290] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:03.581593 18367 raft_consensus.cc:2468] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 8.
I20260709 12:23:03.558152 23095 raft_consensus.cc:3060] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:03.584399 18374 raft_consensus.cc:1240] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 10. Current term is 11. Ops: []
W20260709 12:23:03.584841 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 60496.4us
I20260709 12:23:03.467860 23076 raft_consensus.cc:515] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:03.602120 17801 leader_election.cc:304] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:03.600158 17682 raft_consensus.cc:1240] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 10. Current term is 11. Ops: []
I20260709 12:23:03.615787 17834 raft_consensus.cc:2468] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 15.
I20260709 12:23:03.595747 17689 raft_consensus.cc:1275] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 12 index: 8. Preceding OpId from leader: term: 14 index: 9. (index mismatch)
W20260709 12:23:03.596946 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:03.601612 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 184156us
W20260709 12:23:03.600872 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 109613us
I20260709 12:23:03.590821 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:03.703775 17758 raft_consensus.cc:2393] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 16: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:23:03.618255 17684 raft_consensus.cc:1275] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 12 index: 9. Preceding OpId from leader: term: 15 index: 10. (index mismatch)
W20260709 12:23:03.703838 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 101818us
W20260709 12:23:03.618906 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 46926.2us
I20260709 12:23:03.623816 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:03.594048 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:03.707511 17760 raft_consensus.cc:3060] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:03.705469 17729 leader_election.cc:304] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:23:03.720517 17803 leader_election.cc:304] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:03.624866 23108 raft_consensus.cc:3060] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:03.693823 18333 leader_election.cc:304] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:03.732470 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:03.572602 23052 consensus_queue.cc:1048] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.166s
I20260709 12:23:03.650147 17687 raft_consensus.cc:2468] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 14.
I20260709 12:23:03.626987 17756 raft_consensus.cc:2468] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 12.
I20260709 12:23:03.659304 18368 raft_consensus.cc:2468] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 14.
I20260709 12:23:03.740078 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:03.731679 18363 raft_consensus.cc:1275] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [term 13 LEARNER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 13 index: 8. (index mismatch)
W20260709 12:23:03.677783 23157 log.cc:927] Time spent T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.054s	user 0.000s	sys 0.001s
I20260709 12:23:03.750662 17687 raft_consensus.cc:2393] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 12: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
W20260709 12:23:03.767825 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 44350.5us
I20260709 12:23:03.601365 22854 raft_consensus.cc:2804] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:23:03.602799 23088 raft_consensus.cc:2804] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:23:03.570518 22745 raft_consensus.cc:740] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 15 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:23:03.702736 23021 raft_consensus.cc:493] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:03.772542 23121 consensus_queue.cc:1059] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 9.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:23:03.773869 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:03.775688 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 9 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:03.619903 23120 raft_consensus.cc:2804] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Leader pre-election won for term 15
W20260709 12:23:03.780426 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 87249.2us
W20260709 12:23:03.732991 23144 log.cc:927] Time spent T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.520s	user 0.005s	sys 0.000s
I20260709 12:23:03.818325 23120 raft_consensus.cc:493] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:03.818841 23120 raft_consensus.cc:3060] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:03.785985 18334 leader_election.cc:304] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:23:03.819087 18333 leader_election.cc:304] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:03.786573 18364 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 1662) took 1385 ms. Trace:
I20260709 12:23:03.820413 18364 rpcz_store.cc:276] 0709 12:23:02.401563 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:02.401687 (+   124us) service_pool.cc:224] Handling call
0709 12:23:02.424325 (+ 22638us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:03.282268 (+857943us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:03.786304 (+504036us) spinlock_profiling.cc:243] Waited 857 ms on lock 0x7b54001efd54. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:23:03.786508 (+   204us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1971512576}
I20260709 12:23:03.824391 23022 consensus_queue.cc:1059] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 9.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:23:03.825791 23022 raft_consensus.cc:3055] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:23:03.826454 23022 raft_consensus.cc:740] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:23:03.826768 23121 raft_consensus.cc:3055] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:23:03.827163 23121 raft_consensus.cc:740] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:23:03.829934 23111 raft_consensus.cc:515] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:03.741823 23133 raft_consensus.cc:2804] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:23:03.861475 23133 raft_consensus.cc:493] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:03.861577 17833 raft_consensus.cc:1240] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 10. Current term is 11. Ops: []
I20260709 12:23:03.861967 23133 raft_consensus.cc:3060] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:03.863034 23111 leader_election.cc:290] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 11 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:03.602731 23103 consensus_queue.cc:1059] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 11.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.177s
I20260709 12:23:03.801332 23113 raft_consensus.cc:2804] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:23:03.864493 23113 raft_consensus.cc:697] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 11 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:23:03.864612 23103 raft_consensus.cc:3055] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:23:03.865134 23103 raft_consensus.cc:740] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 11 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:23:03.866062 23103 consensus_queue.cc:260] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 11.8, Last appended by leader: 8, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:03.865871 23113 consensus_queue.cc:237] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 10.9, Last appended by leader: 9, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:03.796178 23021 raft_consensus.cc:515] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:03.802253 22898 raft_consensus.cc:493] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:03.872228 22898 raft_consensus.cc:515] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:23:03.601812 22915 consensus_queue.cc:1059] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.182s
I20260709 12:23:03.768221 23087 consensus_queue.cc:1048] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.133s
I20260709 12:23:03.876066 22898 leader_election.cc:290] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:03.881434 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:03.881958 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:03.882130 17829 raft_consensus.cc:2468] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 15.
I20260709 12:23:03.806599 23148 raft_consensus.cc:493] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:03.782622 23141 raft_consensus.cc:2749] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:23:03.887512 17760 raft_consensus.cc:2468] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 16.
I20260709 12:23:03.811117 23088 raft_consensus.cc:697] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 11 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:23:03.812394 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
W20260709 12:23:03.816076 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 95695.9us
I20260709 12:23:03.816844 23047 raft_consensus.cc:2804] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:23:03.817866 23080 leader_election.cc:290] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 12 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:03.830255 23093 raft_consensus.cc:2804] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Leader pre-election won for term 15
W20260709 12:23:03.830243 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 44330us
I20260709 12:23:03.836608 22745 consensus_queue.cc:260] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.12, Last appended by leader: 12, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:03.839033 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:03.832123 23083 raft_consensus.cc:3060] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:03.840200 22476 leader_election.cc:290] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:03.841738 23076 leader_election.cc:290] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:03.715044 23118 raft_consensus.cc:2804] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:23:03.843345 23022 consensus_queue.cc:260] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 9.11, Last appended by leader: 11, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:03.850601 23046 leader_election.cc:290] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:03.852102 23121 consensus_queue.cc:260] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.10, Last appended by leader: 10, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:03.804821 23099 consensus_queue.cc:1048] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 12:23:03.883335 17829 raft_consensus.cc:3055] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:23:03.982211 17829 raft_consensus.cc:740] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 12 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:23:03.879647 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:03.832707 22933 raft_consensus.cc:493] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:04.000561 22933 raft_consensus.cc:515] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:04.003099 22933 leader_election.cc:290] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:03.898953 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:03.899363 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:03.899655 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:04.009087 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 9 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:04.009377 17803 leader_election.cc:304] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:04.009691 18368 raft_consensus.cc:3060] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:04.010238 23113 raft_consensus.cc:2749] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
W20260709 12:23:03.898138 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 192759us
I20260709 12:23:04.012430 17729 leader_election.cc:304] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:03.908461 17685 raft_consensus.cc:2393] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 8: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:23:03.907279 23093 raft_consensus.cc:493] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:04.015297 23093 raft_consensus.cc:3060] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:03.907270 23047 raft_consensus.cc:493] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:04.018167 23047 raft_consensus.cc:3060] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:03.913295 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:03.914994 18374 raft_consensus.cc:1240] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 10. Current term is 11. Ops: [10.11-10.11]
I20260709 12:23:04.025596 17803 leader_election.cc:304] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:03.919440 17831 raft_consensus.cc:2376] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 16. Re-sending same reply.
I20260709 12:23:04.026849 23113 raft_consensus.cc:2749] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260709 12:23:03.920415 17801 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:04.028738 18333 leader_election.cc:304] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:04.029124 18365 raft_consensus.cc:1275] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 11 index: 10. (term mismatch)
I20260709 12:23:04.029537 18365 pending_rounds.cc:85] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da: Aborting all ops after (but not including) 9
I20260709 12:23:04.029810 18365 pending_rounds.cc:107] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da: Aborting uncommitted NO_OP operation due to leader change: 10.10
I20260709 12:23:04.030016 18365 raft_consensus.cc:2889] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:04.031138 23022 raft_consensus.cc:3060] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:03.931598 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:03.890206 23148 raft_consensus.cc:515] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:04.032194 18371 raft_consensus.cc:2393] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 8: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:23:04.033910 23148 leader_election.cc:290] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:03.881052 23103 raft_consensus.cc:3060] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:04.036161 23056 raft_consensus.cc:2749] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:23:03.944233 18362 raft_consensus.cc:1240] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 13 LEADER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 12. Current term is 13. Ops: []
I20260709 12:23:03.946923 23095 raft_consensus.cc:515] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:04.039711 23095 leader_election.cc:290] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 14 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:03.961360 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:03.963384 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:04.040534 18366 raft_consensus.cc:3060] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:04.040627 18367 raft_consensus.cc:3060] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:03.983072 17829 consensus_queue.cc:260] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.11, Last appended by leader: 11, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:04.042264 17829 raft_consensus.cc:3060] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:04.042068 23088 consensus_queue.cc:237] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 10.8, Last appended by leader: 8, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:04.043126 23113 raft_consensus.cc:493] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:04.043678 23113 raft_consensus.cc:515] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:04.045791 23113 leader_election.cc:290] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:04.047993 22745 raft_consensus.cc:3060] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:04.052603 23093 raft_consensus.cc:515] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:04.052561 23047 raft_consensus.cc:515] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:04.056010 23048 raft_consensus.cc:515] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:04.057376 17829 raft_consensus.cc:1275] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 13 index: 11. (term mismatch)
I20260709 12:23:04.057950 17829 pending_rounds.cc:85] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b: Aborting all ops after (but not including) 10
I20260709 12:23:04.058328 23048 leader_election.cc:290] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 15 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:04.058286 17829 pending_rounds.cc:107] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b: Aborting uncommitted NO_OP operation due to leader change: 12.11
I20260709 12:23:04.059306 17829 raft_consensus.cc:2889] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:04.060740 22854 raft_consensus.cc:697] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 16 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:23:04.062026 22854 consensus_queue.cc:237] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.11, Last appended by leader: 11, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:04.064208 23114 raft_consensus.cc:493] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:04.064827 23114 raft_consensus.cc:515] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:04.067245 23021 leader_election.cc:290] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:23:03.996733 23160 log.cc:927] Time spent T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.074s	user 0.000s	sys 0.000s
I20260709 12:23:04.069275 18363 raft_consensus.cc:2393] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 14: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
W20260709 12:23:04.069478 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 53491.4us
I20260709 12:23:04.069830 17830 raft_consensus.cc:2368] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 7. Current term is 8.
I20260709 12:23:04.078049 17684 raft_consensus.cc:1275] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 13 index: 11. (index mismatch)
I20260709 12:23:04.080639 18367 raft_consensus.cc:2468] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 12.
W20260709 12:23:04.091571 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 62822.2us
I20260709 12:23:04.081317 18366 raft_consensus.cc:2468] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 12.
I20260709 12:23:04.100005 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:04.100826 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:04.102110 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:04.106837 17680 raft_consensus.cc:2368] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 7. Current term is 8.
I20260709 12:23:04.098213 17729 leader_election.cc:304] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:04.111068 17729 leader_election.cc:304] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:04.114537 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:04.114965 18361 raft_consensus.cc:2468] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 13.
I20260709 12:23:04.116325 18364 raft_consensus.cc:1275] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 11 index: 9. Preceding OpId from leader: term: 13 index: 10. (index mismatch)
I20260709 12:23:04.117008 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:04.118191 17756 raft_consensus.cc:2393] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 16: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:23:04.121317 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:03.980994 23118 raft_consensus.cc:697] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 10 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:23:04.123307 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:04.127496 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:04.128253 17681 raft_consensus.cc:2468] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 15.
I20260709 12:23:04.129761 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:04.129163 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:04.134681 17756 raft_consensus.cc:1240] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 11. Current term is 12. Ops: [9.9-11.10]
I20260709 12:23:04.136085 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:04.122964 23093 leader_election.cc:290] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 15 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:04.039402 23056 consensus_queue.cc:1059] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 10.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.055s
I20260709 12:23:04.139868 17689 raft_consensus.cc:2393] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 16: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:23:04.137670 23048 raft_consensus.cc:493] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:04.142465 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 15 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:04.175345 17759 raft_consensus.cc:2393] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 15: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:23:04.144897 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 16 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:04.141438 18368 raft_consensus.cc:2468] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 10.
I20260709 12:23:04.177210 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:04.152146 23047 leader_election.cc:290] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 14 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:04.153117 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:04.154569 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:04.123795 23099 raft_consensus.cc:493] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:04.173692 17683 raft_consensus.cc:1275] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
W20260709 12:23:04.182343 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:04.067830 23021 raft_consensus.cc:493] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:04.137264 23113 consensus_queue.cc:1048] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.005s
I20260709 12:23:04.128280 23124 raft_consensus.cc:2749] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:23:04.179816 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:04.132858 22933 raft_consensus.cc:2804] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:23:04.187960 23021 raft_consensus.cc:515] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:04.173550 23093 raft_consensus.cc:493] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:04.189083 23093 raft_consensus.cc:515] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:04.190155 23021 leader_election.cc:290] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:04.193646 22853 raft_consensus.cc:3055] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:23:04.194224 22853 raft_consensus.cc:740] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
W20260709 12:23:04.194979 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 397632us
I20260709 12:23:04.196096 22476 raft_consensus.cc:493] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:23:04.196626 22476 raft_consensus.cc:515] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:04.196918 17652 leader_election.cc:304] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
W20260709 12:23:04.197891 17728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:04.198520 17652 leader_election.cc:304] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:23:04.198848 22476 leader_election.cc:290] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:04.186182 23048 raft_consensus.cc:515] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:04.200186 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
W20260709 12:23:04.201465 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 677193us
I20260709 12:23:04.067745 23114 leader_election.cc:290] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:04.204382 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:04.205299 17761 raft_consensus.cc:2376] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 11. Re-sending same reply.
I20260709 12:23:04.207643 22954 consensus_queue.cc:1048] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.006s
I20260709 12:23:04.213979 22933 raft_consensus.cc:697] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 14 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:23:04.214485 23048 leader_election.cc:290] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:04.214985 22933 consensus_queue.cc:237] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 12.10, Last appended by leader: 10, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:04.216502 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:04.216727 23070 raft_consensus.cc:493] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:04.217119 17764 raft_consensus.cc:2376] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 15. Re-sending same reply.
I20260709 12:23:04.217337 23070 raft_consensus.cc:515] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:04.218402 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:04.219105 17764 raft_consensus.cc:2393] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 16: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:23:04.131745 22652 raft_consensus.cc:2804] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:23:04.143304 23118 consensus_queue.cc:237] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:04.204001 17727 leader_election.cc:304] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:04.230549 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:04.220458 23148 raft_consensus.cc:3055] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:23:04.249081 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:04.250434 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:04.251312 17760 raft_consensus.cc:2468] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 11.
W20260709 12:23:04.241681 17728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:04.243757 17759 raft_consensus.cc:2368] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 12. Current term is 13.
I20260709 12:23:04.253464 17728 leader_election.cc:304] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:04.191867 23093 leader_election.cc:290] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:04.195184 22853 consensus_queue.cc:260] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 11.8, Last appended by leader: 8, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:04.258543 17728 leader_election.cc:304] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
W20260709 12:23:04.282413 17654 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:04.274614 17802 leader_election.cc:304] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:23:04.289979 23170 log.cc:927] Time spent T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.070s	user 0.001s	sys 0.001s
I20260709 12:23:04.291195 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:04.225112 22652 raft_consensus.cc:493] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:23:04.297149 17763 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.511s	user 0.010s	sys 0.063s
W20260709 12:23:04.297457 17763 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.512s	user 0.010s	sys 0.063s
I20260709 12:23:04.291970 17762 raft_consensus.cc:2468] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 13.
I20260709 12:23:04.298815 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:04.300314 17801 leader_election.cc:304] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:04.302402 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:04.246553 17682 raft_consensus.cc:2468] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 15.
I20260709 12:23:04.254588 23011 consensus_queue.cc:1059] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 11.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.040s
I20260709 12:23:04.208820 23099 raft_consensus.cc:515] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:04.268822 23076 consensus_queue.cc:1048] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.037s
I20260709 12:23:04.296193 23166 raft_consensus.cc:2804] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:23:04.220754 23133 raft_consensus.cc:515] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:04.236601 22877 consensus_queue.cc:1059] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 12.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.034s
I20260709 12:23:04.184216 23108 raft_consensus.cc:515] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:04.294086 17761 raft_consensus.cc:1275] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 14 index: 12. (index mismatch)
I20260709 12:23:04.300488 17802 leader_election.cc:304] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:04.247665 23148 raft_consensus.cc:740] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:23:04.313211 18368 raft_consensus.cc:1240] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 9. Current term is 10. Ops: []
I20260709 12:23:04.254773 22972 raft_consensus.cc:2804] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:23:04.296101 23047 raft_consensus.cc:2749] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:23:04.323534 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:04.319113 17759 raft_consensus.cc:1275] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 11 index: 9. Preceding OpId from leader: term: 13 index: 10. (index mismatch)
I20260709 12:23:04.247726 23056 raft_consensus.cc:2749] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:23:04.302510 22652 raft_consensus.cc:3060] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:04.324455 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:04.335762 18368 raft_consensus.cc:1275] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 14 index: 12. Preceding OpId from leader: term: 16 index: 13. (index mismatch)
I20260709 12:23:04.336061 18365 raft_consensus.cc:2393] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 12: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:23:04.332077 17762 raft_consensus.cc:1275] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 13 index: 8. (index mismatch)
I20260709 12:23:04.258123 23102 consensus_queue.cc:1059] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 10.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.060s
I20260709 12:23:04.316283 22998 raft_consensus.cc:2804] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:23:04.313477 22890 raft_consensus.cc:2804] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:23:04.315145 23080 raft_consensus.cc:2749] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:23:04.331198 23114 raft_consensus.cc:3055] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:23:04.326784 22972 raft_consensus.cc:493] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:04.331918 22853 raft_consensus.cc:3060] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:04.269305 23120 raft_consensus.cc:515] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:04.325196 23166 raft_consensus.cc:697] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 16 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:23:04.311609 23111 consensus_queue.cc:1048] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.030s
I20260709 12:23:04.335234 23070 leader_election.cc:290] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:23:04.363485 17729 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:04.366719 18365 raft_consensus.cc:1240] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 11. Current term is 12. Ops: []
I20260709 12:23:04.339560 23087 raft_consensus.cc:2749] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 12:23:04.371218 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:04.348714 22890 raft_consensus.cc:493] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:04.375625 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
W20260709 12:23:04.380124 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:04.340260 23048 consensus_queue.cc:1059] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 14.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.028s
I20260709 12:23:04.387822 17654 leader_election.cc:304] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:23:04.353492 22972 raft_consensus.cc:3060] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:04.340837 23113 consensus_queue.cc:1048] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.032s
I20260709 12:23:04.351414 23114 raft_consensus.cc:740] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:23:04.382181 17759 raft_consensus.cc:2393] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 14: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:23:04.365561 23104 raft_consensus.cc:493] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:23:04.404479 23104 raft_consensus.cc:515] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:04.382042 22890 raft_consensus.cc:3060] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:04.392709 23088 consensus_queue.cc:1048] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.008s
I20260709 12:23:04.398885 23166 consensus_queue.cc:237] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.9, Last appended by leader: 9, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:04.381832 23070 consensus_queue.cc:1048] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.019s
I20260709 12:23:04.426829 22921 consensus_queue.cc:1048] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:23:04.431170 23099 leader_election.cc:290] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:04.374589 23148 consensus_queue.cc:260] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.10, Last appended by leader: 10, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:04.436563 23148 raft_consensus.cc:3060] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:04.397643 22915 consensus_queue.cc:1048] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.040s
I20260709 12:23:04.440392 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:04.441582 23108 leader_election.cc:290] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 14 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:04.443171 23120 leader_election.cc:290] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 15 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:23:04.446096 23083 consensus_meta.cc:287] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.512s	user 0.009s	sys 0.000s
I20260709 12:23:04.446609 23083 raft_consensus.cc:2749] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:23:04.395697 23145 consensus_queue.cc:1059] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 11.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.052s
I20260709 12:23:04.448603 23145 raft_consensus.cc:3055] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:23:04.412952 23018 raft_consensus.cc:2804] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:23:04.449129 23145 raft_consensus.cc:740] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:23:04.449415 23018 raft_consensus.cc:697] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 8 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:23:04.450093 23145 consensus_queue.cc:260] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 12.13, Last appended by leader: 13, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:04.450559 23018 consensus_queue.cc:237] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:04.448526 23114 consensus_queue.cc:260] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:04.412129 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:04.419927 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:04.421329 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:04.448506 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:04.453934 18366 raft_consensus.cc:3060] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:04.453001 23162 raft_consensus.cc:3055] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:23:04.452636 23104 leader_election.cc:290] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:23:04.440629 23169 log.cc:927] Time spent T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.086s	user 0.000s	sys 0.000s
I20260709 12:23:04.408952 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:04.479123 17684 raft_consensus.cc:1275] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 15 index: 8. Preceding OpId from leader: term: 16 index: 9. (index mismatch)
I20260709 12:23:04.457579 17802 leader_election.cc:304] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:23:04.474512 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 49274.5us
I20260709 12:23:04.475812 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:04.483251 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:04.483745 23070 raft_consensus.cc:2749] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 12:23:04.484412 22877 consensus_queue.cc:1048] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 12:23:04.484706 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:04.486028 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:04.451772 23145 raft_consensus.cc:3060] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:04.486836 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:04.492152 18334 leader_election.cc:400] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 7 election: Vote denied by peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071) with higher term. Message: Invalid argument: T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 7. Current term is 8.
I20260709 12:23:04.492573 18334 leader_election.cc:403] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 7 election: Cancelling election due to peer responding with higher term
I20260709 12:23:04.493916 18334 leader_election.cc:304] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:23:04.497828 23108 raft_consensus.cc:2749] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Leader election lost for term 7. Reason: Vote denied by peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071) with higher term. Message: Invalid argument: T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 7. Current term is 8.
I20260709 12:23:04.498575 23108 consensus_queue.cc:1048] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 12:23:04.499197 22972 raft_consensus.cc:515] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:04.478691 17682 raft_consensus.cc:1275] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 13 index: 8. (index mismatch)
I20260709 12:23:04.472465 23162 raft_consensus.cc:740] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 14 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:23:04.500123 23108 raft_consensus.cc:2804] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:23:04.500658 23083 consensus_queue.cc:1048] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 12:23:04.501681 18334 leader_election.cc:304] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:23:04.507447 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:04.453446 23114 raft_consensus.cc:3060] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:04.545593 17686 raft_consensus.cc:2393] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 16: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
W20260709 12:23:04.545404 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 345973us
I20260709 12:23:04.547814 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
W20260709 12:23:04.521554 17654 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:04.550333 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:04.551021 17755 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38776 (request call id 2387) took 1012 ms. Trace:
I20260709 12:23:04.477705 17758 raft_consensus.cc:2468] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 16.
W20260709 12:23:04.521787 17655 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:04.554260 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:04.526216 18366 raft_consensus.cc:2468] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 15.
I20260709 12:23:04.524132 23148 raft_consensus.cc:493] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:04.541314 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:04.555478 17689 raft_consensus.cc:2468] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 15.
I20260709 12:23:04.499694 22998 raft_consensus.cc:493] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:04.551456 17755 rpcz_store.cc:276] 0709 12:23:03.538244 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:03.546086 (+  7842us) service_pool.cc:224] Handling call
0709 12:23:03.546481 (+   395us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:03.783443 (+236962us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:03.783481 (+    38us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:03.793836 (+ 10355us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:23:03.793867 (+    31us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:04.550772 (+756905us) spinlock_profiling.cc:243] Waited 220 ms on lock 0x7b54000fc8d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:23:04.550828 (+    56us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:04.550965 (+   137us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":506413952}
I20260709 12:23:04.561592 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:23:04.545511 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 372170us
I20260709 12:23:04.559477 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:04.566509 17837 raft_consensus.cc:2393] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 12: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:23:04.540876 22898 raft_consensus.cc:493] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:23:04.563287 23148 raft_consensus.cc:515] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:04.564149 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:04.564389 23021 raft_consensus.cc:3055] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:23:04.542575 23108 raft_consensus.cc:493] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:04.562207 17755 raft_consensus.cc:2468] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 15.
I20260709 12:23:04.540541 23007 raft_consensus.cc:493] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:23:04.563505 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:04.642174 17681 raft_consensus.cc:2468] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 15.
I20260709 12:23:04.566816 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:04.643049 18361 raft_consensus.cc:1275] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 14 index: 12. (index mismatch)
I20260709 12:23:04.545174 23104 raft_consensus.cc:2804] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Leader election won for term 15
W20260709 12:23:04.596419 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 326735us
I20260709 12:23:04.644131 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:04.602698 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:04.610309 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:04.548979 23070 consensus_queue.cc:1048] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.081s
I20260709 12:23:04.653280 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:04.661271 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:04.662269 17837 raft_consensus.cc:2468] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 15.
I20260709 12:23:04.544049 22890 raft_consensus.cc:515] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:23:04.637825 17802 leader_election.cc:304] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:23:04.565408 22998 raft_consensus.cc:3060] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:04.638929 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:04.547456 23113 raft_consensus.cc:493] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:04.662581 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:23:04.650489 23160 log.cc:927] Time spent T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.100s	user 0.008s	sys 0.001s
I20260709 12:23:04.672675 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:04.687373 17830 raft_consensus.cc:2468] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 13.
I20260709 12:23:04.565598 23099 consensus_queue.cc:1048] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 4, Time since last communication: 0.077s
W20260709 12:23:04.625216 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 357925us
I20260709 12:23:04.681195 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:04.668496 17834 raft_consensus.cc:1240] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 15. Current term is 16. Ops: [15.10-15.10]
I20260709 12:23:04.685753 18361 raft_consensus.cc:1275] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20260709 12:23:04.639590 23021 raft_consensus.cc:740] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:23:04.688994 18366 raft_consensus.cc:1240] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 13. Current term is 14. Ops: []
I20260709 12:23:04.641891 23108 raft_consensus.cc:3060] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:04.705369 17836 raft_consensus.cc:1240] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 13. Current term is 14. Ops: [13.8-13.8]
I20260709 12:23:04.713332 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:04.632131 22898 raft_consensus.cc:515] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:04.717828 17837 raft_consensus.cc:3060] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:04.717803 17652 leader_election.cc:304] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:04.723120 18333 leader_election.cc:304] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:23:04.682776 23087 raft_consensus.cc:2804] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:23:04.730809 17830 raft_consensus.cc:1240] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 14. Current term is 15. Ops: []
I20260709 12:23:04.610313 23162 consensus_queue.cc:260] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.8, Last appended by leader: 8, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:04.743026 23162 raft_consensus.cc:3060] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:04.742923 22954 raft_consensus.cc:2749] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:23:04.655805 23007 raft_consensus.cc:515] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:04.752085 23007 leader_election.cc:290] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:04.757534 22890 leader_election.cc:290] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 14 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:04.760124 17838 raft_consensus.cc:3060] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:23:04.764600 23172 log.cc:927] Time spent T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.090s	user 0.000s	sys 0.001s
I20260709 12:23:04.686506 23104 raft_consensus.cc:697] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 15 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:23:04.655303 22745 consensus_queue.cc:1048] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.061s
I20260709 12:23:04.693291 23080 raft_consensus.cc:493] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:04.769894 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:04.771064 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:04.772356 18361 raft_consensus.cc:1275] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 10 index: 8. Preceding OpId from leader: term: 11 index: 9. (index mismatch)
I20260709 12:23:04.773808 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:04.783061 17830 raft_consensus.cc:2468] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 12.
I20260709 12:23:04.780759 17834 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 12.
I20260709 12:23:04.787355 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:23:04.802312 17757 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.505s	user 0.059s	sys 0.083s
I20260709 12:23:04.812532 17684 raft_consensus.cc:2393] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 15: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
W20260709 12:23:04.766502 22933 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.548s	user 0.014s	sys 0.033s
I20260709 12:23:04.736346 23087 raft_consensus.cc:493] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:04.697685 23113 raft_consensus.cc:515] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:04.828560 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:04.831691 17681 raft_consensus.cc:1275] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 10 index: 8. Preceding OpId from leader: term: 11 index: 9. (index mismatch)
I20260709 12:23:04.798379 23022 raft_consensus.cc:493] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:04.824970 22933 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.607s	user 0.015s	sys 0.034s
I20260709 12:23:04.838282 17837 raft_consensus.cc:2468] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 15.
I20260709 12:23:04.857882 17833 raft_consensus.cc:2468] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 15.
W20260709 12:23:04.862052 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 139107us
I20260709 12:23:04.831466 23087 raft_consensus.cc:3060] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:04.865835 18333 leader_election.cc:400] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 7 election: Vote denied by peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837) with higher term. Message: Invalid argument: T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 7. Current term is 8.
I20260709 12:23:04.838438 22972 leader_election.cc:290] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 16 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:04.796926 23080 raft_consensus.cc:515] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:04.859853 22877 raft_consensus.cc:493] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:04.853345 23070 raft_consensus.cc:493] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:23:04.868167 22476 raft_consensus.cc:493] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:04.900242 22476 raft_consensus.cc:515] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:04.872670 23120 raft_consensus.cc:2804] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:23:04.911242 17584 catalog_manager.cc:5697] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb reported cstate change: term changed from 10 to 12. New cstate: current_term: 12 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:23:04.863159 23007 consensus_queue.cc:1048] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.024s
I20260709 12:23:04.855827 23022 raft_consensus.cc:515] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:23:04.944428 17760 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.554s	user 0.022s	sys 0.031s
W20260709 12:23:04.963008 17760 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.579s	user 0.022s	sys 0.031s
I20260709 12:23:04.962074 23148 leader_election.cc:290] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:04.873157 23083 consensus_queue.cc:1048] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.072s
I20260709 12:23:04.853122 23021 consensus_queue.cc:260] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 12.9, Last appended by leader: 9, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:04.967084 23120 raft_consensus.cc:697] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 16 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:23:04.898666 22877 raft_consensus.cc:515] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:04.905490 23070 raft_consensus.cc:515] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:04.998978 23021 raft_consensus.cc:3060] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:04.879767 23104 consensus_queue.cc:237] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:05.004733 17830 raft_consensus.cc:1275] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 15 index: 10. (index mismatch)
W20260709 12:23:04.985646 22854 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.534s	user 0.022s	sys 0.010s
I20260709 12:23:05.013682 23011 raft_consensus.cc:3055] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:23:05.014219 23011 raft_consensus.cc:740] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:23:05.015074 23011 consensus_queue.cc:260] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.7, Last appended by leader: 7, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:05.016631 23011 raft_consensus.cc:3060] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:23:05.017761 22652 consensus_meta.cc:287] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.674s	user 0.007s	sys 0.002s
I20260709 12:23:05.018170 22652 raft_consensus.cc:515] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:05.019323 17763 raft_consensus.cc:2393] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 12: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:23:05.019570 23099 raft_consensus.cc:493] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:23:05.020514 17757 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38786 (request call id 3081) took 1429 ms. Trace:
I20260709 12:23:05.025354 17688 raft_consensus.cc:2468] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 12.
I20260709 12:23:05.020979 17757 rpcz_store.cc:276] 0709 12:23:03.590744 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:03.590856 (+   112us) service_pool.cc:224] Handling call
0709 12:23:03.591490 (+   634us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:04.221225 (+629735us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:04.221256 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:04.221773 (+   517us) log.cc:844] Serialized 26 byte log entry
0709 12:23:04.804612 (+582839us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:23:04.804674 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:04.804774 (+   100us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:05.020196 (+215422us) raft_consensus.cc:1685] finished
0709 12:23:05.020225 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:05.020419 (+   194us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":117756,"raft.queue_time_us":224555,"raft.run_cpu_time_us":93,"raft.run_wall_time_us":93,"thread_start_us":996017,"threads_started":2,"wal-append.queue_time_us":716179}
I20260709 12:23:05.077193 17729 leader_election.cc:304] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:05.084656 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:05.020151 23099 raft_consensus.cc:515] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:05.032788 22877 leader_election.cc:290] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:05.000145 22476 leader_election.cc:290] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:05.037214 22898 leader_election.cc:290] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:05.036635 23120 consensus_queue.cc:237] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.12, Last appended by leader: 12, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:05.036464 23070 leader_election.cc:290] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:05.041059 23022 leader_election.cc:290] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:05.047061 23046 raft_consensus.cc:3055] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:23:05.098016 23046 raft_consensus.cc:740] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:23:05.099375 23046 consensus_queue.cc:260] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:05.100754 23083 raft_consensus.cc:493] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:05.100839 23046 raft_consensus.cc:3060] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:05.101506 23083 raft_consensus.cc:515] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:05.053660 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:05.106266 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:05.055740 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:05.109532 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:05.110589 17756 raft_consensus.cc:2468] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 15.
I20260709 12:23:05.114591 17801 leader_election.cc:304] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:05.069027 23076 raft_consensus.cc:493] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:05.069664 23056 raft_consensus.cc:493] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:05.070147 23168 consensus_queue.cc:1048] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 12:23:05.080623 23135 raft_consensus.cc:493] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:23:05.071359 23133 leader_election.cc:290] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 15 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:05.134289 23135 raft_consensus.cc:515] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:05.084120 22652 leader_election.cc:290] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 12 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:05.073026 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:05.084029 23124 raft_consensus.cc:493] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:05.135594 23124 raft_consensus.cc:515] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:05.077618 23007 raft_consensus.cc:493] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:05.136998 23007 raft_consensus.cc:515] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:05.078943 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:05.101330 23080 leader_election.cc:290] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:05.137972 17764 raft_consensus.cc:3060] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:05.138101 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:05.139287 23007 leader_election.cc:290] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:05.140347 23099 leader_election.cc:290] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:05.143505 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 13 candidate_status { last_received { term: 12 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:05.146204 17764 raft_consensus.cc:2468] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 14.
I20260709 12:23:05.146684 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:05.147329 17758 raft_consensus.cc:2468] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 16.
W20260709 12:23:05.149734 17655 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:05.081923 23118 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.595s	user 0.006s	sys 0.007s
I20260709 12:23:05.090029 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:05.090780 23179 raft_consensus.cc:2804] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:23:05.092140 22890 raft_consensus.cc:493] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:05.120041 22998 raft_consensus.cc:515] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:05.112039 23180 raft_consensus.cc:493] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:05.120528 23202 log.cc:927] Time spent T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.121s	user 0.001s	sys 0.000s
W20260709 12:23:05.008970 22854 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.561s	user 0.022s	sys 0.010s
I20260709 12:23:05.119432 17803 leader_election.cc:304] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3
W20260709 12:23:05.088747 22590 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.585s	user 0.006s	sys 0.007s
W20260709 12:23:05.083793 23145 consensus_meta.cc:287] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.597s	user 0.009s	sys 0.003s
I20260709 12:23:05.077148 23111 raft_consensus.cc:493] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:05.121832 23121 raft_consensus.cc:3060] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:05.123584 17801 leader_election.cc:304] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:05.126691 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:05.189472 17686 raft_consensus.cc:2468] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 15.
I20260709 12:23:05.128623 17802 leader_election.cc:304] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:23:05.165751 22590 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.668s	user 0.006s	sys 0.007s
I20260709 12:23:05.134649 23113 leader_election.cc:290] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:23:05.076706 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:05.150542 23118 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.664s	user 0.007s	sys 0.007s
I20260709 12:23:05.111784 17756 raft_consensus.cc:1275] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 12 index: 9. Preceding OpId from leader: term: 15 index: 10. (index mismatch)
I20260709 12:23:05.202641 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:05.167860 23099 raft_consensus.cc:2804] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:23:05.151280 22890 raft_consensus.cc:515] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:05.158453 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 440754us
I20260709 12:23:05.160434 17803 leader_election.cc:304] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:23:05.106221 23083 leader_election.cc:290] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:05.120663 22898 raft_consensus.cc:2749] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
W20260709 12:23:05.169586 17654 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:05.177860 17683 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.694s	user 0.010s	sys 0.005s
I20260709 12:23:05.183697 18333 leader_election.cc:304] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:23:05.216161 17683 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.732s	user 0.011s	sys 0.006s
I20260709 12:23:05.216797 18366 raft_consensus.cc:1240] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 10. Current term is 11. Ops: []
W20260709 12:23:05.190865 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 62333.9us
I20260709 12:23:05.177445 23179 raft_consensus.cc:697] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 15 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:23:05.214406 17652 leader_election.cc:304] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:05.132046 23076 raft_consensus.cc:515] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:05.217002 18333 leader_election.cc:304] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:23:05.188719 17832 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.817s	user 0.023s	sys 0.061s
I20260709 12:23:05.187698 17681 raft_consensus.cc:2468] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 13.
W20260709 12:23:05.167596 23018 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.682s	user 0.001s	sys 0.014s
W20260709 12:23:05.188638 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 65162.9us
I20260709 12:23:05.072677 17838 raft_consensus.cc:2468] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 15.
I20260709 12:23:05.197144 23080 raft_consensus.cc:3060] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:05.142405 23056 raft_consensus.cc:515] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:05.215474 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:05.232100 18367 raft_consensus.cc:1240] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 12. Current term is 13. Ops: []
I20260709 12:23:05.238102 18361 raft_consensus.cc:3060] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:05.232911 17801 leader_election.cc:304] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
W20260709 12:23:05.240495 17654 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:05.157675 23180 raft_consensus.cc:515] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:23:05.221488 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 61189.5us
I20260709 12:23:05.242802 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:05.219062 17802 leader_election.cc:304] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:05.209607 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 12 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:05.243748 17688 raft_consensus.cc:1275] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 14 index: 12. Preceding OpId from leader: term: 15 index: 13. (index mismatch)
I20260709 12:23:05.231698 23174 raft_consensus.cc:2749] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:23:05.240345 18333 leader_election.cc:304] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:05.210605 23175 raft_consensus.cc:493] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:05.245414 17652 leader_election.cc:304] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:05.247144 17803 leader_election.cc:304] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:23:05.253798 17757 raft_consensus.cc:3060] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:23:05.231982 23018 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.746s	user 0.002s	sys 0.014s
W20260709 12:23:05.220810 17832 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.854s	user 0.024s	sys 0.062s
I20260709 12:23:05.261971 22998 leader_election.cc:290] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 16 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:05.262894 23180 leader_election.cc:290] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:05.234951 23007 raft_consensus.cc:2749] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:23:05.257035 23175 raft_consensus.cc:515] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:05.268381 23124 leader_election.cc:290] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:05.269254 23124 raft_consensus.cc:2749] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 12:23:05.269704 23175 leader_election.cc:290] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:05.246343 23148 raft_consensus.cc:493] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:23:05.270143 23118 consensus_queue.cc:1048] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 12:23:05.270772 23148 raft_consensus.cc:515] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:05.271639 17803 leader_election.cc:304] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:05.221010 23099 raft_consensus.cc:697] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 12 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:23:05.271096 18363 raft_consensus.cc:2468] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 11.
I20260709 12:23:05.197428 23111 raft_consensus.cc:515] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:05.276171 17756 raft_consensus.cc:3060] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:05.290234 17757 raft_consensus.cc:2468] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 15.
I20260709 12:23:05.318061 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:05.318945 18367 raft_consensus.cc:2468] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 11.
I20260709 12:23:05.251783 23083 raft_consensus.cc:2749] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:23:05.273959 17652 leader_election.cc:304] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
W20260709 12:23:05.309875 17835 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.619s	user 0.027s	sys 0.038s
W20260709 12:23:05.331456 17835 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.641s	user 0.028s	sys 0.038s
I20260709 12:23:05.328703 18361 raft_consensus.cc:2468] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 12.
W20260709 12:23:05.344697 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 70738.8us
I20260709 12:23:05.255033 22898 raft_consensus.cc:493] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:05.253587 23133 raft_consensus.cc:2804] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:23:05.266981 23135 leader_election.cc:290] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:05.239537 23084 consensus_queue.cc:1048] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.077s
I20260709 12:23:05.346993 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:05.257572 23070 raft_consensus.cc:2804] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:23:05.285385 22915 raft_consensus.cc:3060] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Advancing to term 12
W20260709 12:23:05.361776 23108 consensus_meta.cc:287] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.594s	user 0.003s	sys 0.010s
I20260709 12:23:05.333629 23164 raft_consensus.cc:493] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:05.358493 23133 raft_consensus.cc:697] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 12 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:23:05.369663 23133 consensus_queue.cc:237] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:05.368186 23164 raft_consensus.cc:515] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:05.371991 23088 raft_consensus.cc:2804] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:23:05.368105 23108 raft_consensus.cc:515] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:05.372598 23088 raft_consensus.cc:697] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 10 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:23:05.373740 23088 consensus_queue.cc:237] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 9.11, Last appended by leader: 11, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:05.375500 23108 leader_election.cc:290] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 16 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:05.376825 22898 raft_consensus.cc:515] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:05.379283 23164 leader_election.cc:290] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:05.250448 22921 consensus_queue.cc:1059] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 13.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.105s
I20260709 12:23:05.340751 23139 raft_consensus.cc:493] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:05.350904 22998 raft_consensus.cc:2804] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Leader election won for term 12
W20260709 12:23:05.359488 23184 log.cc:927] Time spent T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.125s	user 0.000s	sys 0.001s
I20260709 12:23:05.384971 23080 raft_consensus.cc:2749] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
W20260709 12:23:05.407732 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 704227us
I20260709 12:23:05.416582 17762 raft_consensus.cc:2393] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 14: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:23:05.359114 23113 raft_consensus.cc:2804] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:23:05.468990 17760 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38786 (request call id 3090) took 1208 ms. Trace:
I20260709 12:23:05.431715 22853 raft_consensus.cc:2749] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:23:05.463447 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 12 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:05.529675 17684 raft_consensus.cc:2468] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 11.
W20260709 12:23:05.461690 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 51081.1us
W20260709 12:23:05.433117 23185 log.cc:927] Time spent T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.068s	user 0.000s	sys 0.000s
I20260709 12:23:05.510916 17727 leader_election.cc:304] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:05.455236 23167 raft_consensus.cc:493] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
W20260709 12:23:05.534176 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 86359.9us
I20260709 12:23:05.527463 17760 rpcz_store.cc:276] 0709 12:23:04.260622 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:04.313049 (+ 52427us) service_pool.cc:224] Handling call
0709 12:23:04.334448 (+ 21399us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:04.343061 (+  8613us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:04.343107 (+    46us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:04.346171 (+  3064us) log.cc:844] Serialized 26 byte log entry
0709 12:23:04.963436 (+617265us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:23:04.963482 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:04.963585 (+   103us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:05.468695 (+505110us) raft_consensus.cc:1685] finished
0709 12:23:05.468731 (+    36us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:05.468915 (+   184us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":8103,"raft.queue_time_us":16921,"raft.run_cpu_time_us":321,"raft.run_wall_time_us":17866,"thread_start_us":579288,"threads_started":1,"wal-append.queue_time_us":832607}
I20260709 12:23:05.535511 17727 leader_election.cc:304] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:05.469166 23148 leader_election.cc:290] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:23:05.461257 23199 log.cc:927] Time spent T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.057s	user 0.000s	sys 0.000s
I20260709 12:23:05.522352 17756 raft_consensus.cc:2468] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 15.
W20260709 12:23:05.471807 23162 consensus_meta.cc:287] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.728s	user 0.011s	sys 0.000s
W20260709 12:23:05.507529 23186 log.cc:927] Time spent T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.166s	user 0.000s	sys 0.001s
I20260709 12:23:05.478241 17755 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38786 (request call id 3093) took 1057 ms. Trace:
I20260709 12:23:05.462352 22921 raft_consensus.cc:3055] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:23:05.455240 23180 raft_consensus.cc:493] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:05.473074 23076 leader_election.cc:290] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:05.490132 22898 leader_election.cc:290] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:05.531421 17728 leader_election.cc:304] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:05.482198 23056 leader_election.cc:290] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:05.481915 23048 raft_consensus.cc:493] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
W20260709 12:23:05.513459 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 280584us
I20260709 12:23:05.538111 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:23:05.519912 23087 consensus_meta.cc:287] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.645s	user 0.011s	sys 0.000s
I20260709 12:23:05.480001 23164 raft_consensus.cc:493] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:05.597851 17801 leader_election.cc:304] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:05.538707 17652 leader_election.cc:304] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:23:05.599448 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 60846.7us
I20260709 12:23:05.604486 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:05.604809 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:05.605999 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 12 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:05.606587 17830 raft_consensus.cc:2468] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 11.
W20260709 12:23:05.539007 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 267423us
W20260709 12:23:05.417956 23192 log.cc:927] Time spent T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.138s	user 0.000s	sys 0.001s
I20260709 12:23:05.545006 22652 consensus_queue.cc:1048] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 12:23:05.465669 22998 raft_consensus.cc:697] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 12 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
W20260709 12:23:05.622761 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 403743us
I20260709 12:23:05.364494 23179 consensus_queue.cc:237] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.11, Last appended by leader: 11, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:05.412858 23139 raft_consensus.cc:515] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:05.541821 17755 rpcz_store.cc:276] 0709 12:23:04.421038 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:04.640591 (+219553us) service_pool.cc:224] Handling call
0709 12:23:04.640923 (+   332us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:04.651223 (+ 10300us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:04.651269 (+    46us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:04.654374 (+  3105us) log.cc:844] Serialized 26 byte log entry
0709 12:23:05.152520 (+498146us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:23:05.152562 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:05.152628 (+    66us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:05.477993 (+325365us) raft_consensus.cc:1685] finished
0709 12:23:05.478016 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:05.478160 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":78,"raft.queue_time_us":497,"raft.run_cpu_time_us":123,"raft.run_wall_time_us":4940,"thread_start_us":437052,"threads_started":1,"wal-append.queue_time_us":701255}
I20260709 12:23:05.516484 23080 consensus_queue.cc:1048] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.033s
I20260709 12:23:05.540273 23206 raft_consensus.cc:2804] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:23:05.648825 23206 raft_consensus.cc:493] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:23:05.649891 23204 log.cc:927] Time spent T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.112s	user 0.000s	sys 0.000s
I20260709 12:23:05.547896 23047 raft_consensus.cc:2749] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:23:05.600241 22890 leader_election.cc:290] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:05.651686 17685 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37402 (request call id 2824) took 1174 ms. Trace:
I20260709 12:23:05.653004 17760 raft_consensus.cc:1275] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 14 index: 12. Preceding OpId from leader: term: 15 index: 13. (index mismatch)
I20260709 12:23:05.652396 17685 rpcz_store.cc:276] 0709 12:23:04.477409 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:04.477631 (+   222us) service_pool.cc:224] Handling call
0709 12:23:04.477969 (+   338us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:04.478333 (+   364us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:04.478361 (+    28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:04.519379 (+ 41018us) log.cc:844] Serialized 26 byte log entry
0709 12:23:04.994097 (+474718us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:23:04.994146 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:04.994230 (+    84us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:05.651457 (+657227us) raft_consensus.cc:1685] finished
0709 12:23:05.651479 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:05.651624 (+   145us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1099,"raft.run_cpu_time_us":106,"raft.run_wall_time_us":33954,"thread_start_us":474234,"threads_started":1,"wal-append.queue_time_us":696022}
I20260709 12:23:05.605425 23087 raft_consensus.cc:515] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:05.659310 23087 leader_election.cc:290] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 16 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:23:05.564193 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 309433us
I20260709 12:23:05.597276 23148 raft_consensus.cc:493] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:05.661096 23148 raft_consensus.cc:515] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:05.662114 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 130939us
I20260709 12:23:05.630187 17836 raft_consensus.cc:1240] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 12. Current term is 13. Ops: []
I20260709 12:23:05.662909 17728 leader_election.cc:304] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:05.664538 23148 leader_election.cc:290] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:05.608769 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:05.635365 17829 raft_consensus.cc:2468] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 16.
I20260709 12:23:05.477799 23099 consensus_queue.cc:237] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 11.8, Last appended by leader: 8, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:05.548172 22921 raft_consensus.cc:740] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:23:05.615568 23201 raft_consensus.cc:2804] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:23:05.640962 17755 raft_consensus.cc:1240] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 11. Current term is 12. Ops: []
W20260709 12:23:05.645978 23046 consensus_meta.cc:287] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.544s	user 0.012s	sys 0.000s
I20260709 12:23:05.676214 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:05.677151 17682 raft_consensus.cc:2393] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 16: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:23:05.676847 18365 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59886 (request call id 1537) took 1257 ms. Trace:
I20260709 12:23:05.684319 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:05.685144 17688 raft_consensus.cc:3060] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:05.556478 22933 raft_consensus.cc:493] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:23:05.647891 23139 leader_election.cc:290] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:05.649618 23206 raft_consensus.cc:3060] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:05.621253 17652 leader_election.cc:304] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
W20260709 12:23:05.713747 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 92582.6us
I20260709 12:23:05.715310 22877 raft_consensus.cc:2749] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 12:23:05.608517 23164 raft_consensus.cc:515] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:05.715770 17652 leader_election.cc:304] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da
W20260709 12:23:05.716643 23187 log.cc:927] Time spent T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.079s	user 0.000s	sys 0.000s
I20260709 12:23:05.716856 23018 raft_consensus.cc:2804] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:23:05.717363 23018 raft_consensus.cc:493] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:23:05.717777 23018 raft_consensus.cc:3060] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:05.717985 23164 leader_election.cc:290] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:05.718030 17683 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37402 (request call id 2821) took 1259 ms. Trace:
I20260709 12:23:05.718892 17683 rpcz_store.cc:276] 0709 12:23:04.458541 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:04.459120 (+   579us) service_pool.cc:224] Handling call
0709 12:23:04.482627 (+ 23507us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:04.483126 (+   499us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:04.483159 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:04.483752 (+   593us) log.cc:844] Serialized 26 byte log entry
0709 12:23:05.216525 (+732773us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:23:05.216565 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:05.216628 (+    63us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:05.717821 (+501193us) raft_consensus.cc:1685] finished
0709 12:23:05.717847 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:05.717972 (+   125us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":74,"raft.queue_time_us":2484,"raft.run_cpu_time_us":90,"raft.run_wall_time_us":91,"thread_start_us":732333,"threads_started":1,"wal-append.queue_time_us":1055832}
I20260709 12:23:05.720809 18374 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 1697) took 1065 ms. Trace:
I20260709 12:23:05.721204 18374 rpcz_store.cc:276] 0709 12:23:04.655829 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:04.655967 (+   138us) service_pool.cc:224] Handling call
0709 12:23:04.689728 (+ 33761us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:04.703590 (+ 13862us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:04.703633 (+    43us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:04.715726 (+ 12093us) log.cc:844] Serialized 26 byte log entry
0709 12:23:05.148041 (+432315us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:23:05.148075 (+    34us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:05.148151 (+    76us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:05.720567 (+572416us) raft_consensus.cc:1685] finished
0709 12:23:05.720591 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:05.720748 (+   157us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":13374,"raft.queue_time_us":14016,"raft.run_cpu_time_us":136,"raft.run_wall_time_us":17875,"thread_start_us":382350,"threads_started":1,"wal-append.queue_time_us":900427}
I20260709 12:23:05.667677 17728 leader_election.cc:304] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:05.724162 23111 leader_election.cc:290] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:23:05.725054 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 57453.7us
I20260709 12:23:05.621784 17762 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38776 (request call id 2392) took 1075 ms. Trace:
I20260709 12:23:05.728819 23216 raft_consensus.cc:2804] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:23:05.729355 23216 raft_consensus.cc:493] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:05.729704 23216 raft_consensus.cc:3060] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:05.732096 23080 consensus_queue.cc:1048] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 12:23:05.674185 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 14 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:05.695184 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:05.732656 23087 consensus_queue.cc:1048] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
W20260709 12:23:05.733722 23211 log.cc:927] Time spent T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.057s	user 0.000s	sys 0.000s
I20260709 12:23:05.734519 17684 raft_consensus.cc:1275] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 12 index: 10. (index mismatch)
I20260709 12:23:05.671571 17759 raft_consensus.cc:2468] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 15.
I20260709 12:23:05.728168 17762 rpcz_store.cc:276] 0709 12:23:04.546890 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:05.621041 (+1074151us) service_pool.cc:224] Handling call
0709 12:23:05.621312 (+   271us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:05.621487 (+   175us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:05.621511 (+    24us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:05.621550 (+    39us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:23:05.621580 (+    30us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:05.621632 (+    52us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:05.621731 (+    99us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:05.592767 23180 raft_consensus.cc:515] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:05.736476 23164 raft_consensus.cc:2749] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:23:05.737857 23180 leader_election.cc:290] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:23:05.738632 23021 consensus_meta.cc:287] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.739s	user 0.011s	sys 0.000s
I20260709 12:23:05.738806 17757 raft_consensus.cc:1275] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 14 index: 12. Preceding OpId from leader: term: 16 index: 13. (index mismatch)
I20260709 12:23:05.675946 23221 raft_consensus.cc:2749] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:23:05.708305 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 12 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:05.710847 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:05.741303 17836 raft_consensus.cc:3060] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:05.667418 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:05.543224 23167 raft_consensus.cc:515] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:05.732283 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:23:05.758159 17655 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:05.680912 18365 rpcz_store.cc:276] 0709 12:23:04.418945 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:04.429509 (+ 10564us) service_pool.cc:224] Handling call
0709 12:23:04.429867 (+   358us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:04.430091 (+   224us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 12:23:04.975514 (+545423us) log.cc:844] Serialized 14 byte log entry
0709 12:23:05.675317 (+699803us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:05.675367 (+    50us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:23:05.675418 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:05.676655 (+  1237us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:05.676793 (+   138us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":46147,"prepare.queue_time_us":79455,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":41,"raft.queue_time_us":27696,"raft.run_cpu_time_us":79,"raft.run_wall_time_us":80,"thread_start_us":1170078,"threads_started":3,"wal-append.queue_time_us":481749}
I20260709 12:23:05.741808 17687 raft_consensus.cc:2410] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for term 12 because replica has last-logged OpId of term: 10 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 10.
I20260709 12:23:05.751319 17652 leader_election.cc:304] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:05.753445 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:05.744632 17830 raft_consensus.cc:1240] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 15. Current term is 16. Ops: [15.9-15.9]
I20260709 12:23:05.602681 23048 raft_consensus.cc:515] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:05.765987 17832 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37994 (request call id 1050) took 1919 ms. Trace:
I20260709 12:23:05.770684 23164 raft_consensus.cc:493] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:23:05.772336 23164 raft_consensus.cc:515] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:05.771165 17832 rpcz_store.cc:276] 0709 12:23:03.847135 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:03.927215 (+ 80080us) service_pool.cc:224] Handling call
0709 12:23:03.927930 (+   715us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:03.928450 (+   520us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 12:23:04.366082 (+437632us) log.cc:844] Serialized 14 byte log entry
0709 12:23:05.765535 (+1399453us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:05.765611 (+    76us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:23:05.765681 (+    70us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:05.765775 (+    94us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:05.765920 (+   145us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":65200,"prepare.queue_time_us":787456,"prepare.run_cpu_time_us":61,"prepare.run_wall_time_us":61,"raft.queue_time_us":334688,"raft.run_cpu_time_us":180,"raft.run_wall_time_us":7188,"thread_start_us":1792816,"threads_started":3,"wal-append.queue_time_us":1014858}
I20260709 12:23:05.638171 23056 raft_consensus.cc:2749] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
W20260709 12:23:05.760043 18333 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.507s	user 0.011s	sys 0.027s
I20260709 12:23:05.762742 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:05.776293 23164 leader_election.cc:290] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:05.777269 23046 raft_consensus.cc:493] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:23:05.707473 17688 raft_consensus.cc:2468] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 16.
W20260709 12:23:05.781129 18333 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.528s	user 0.011s	sys 0.027s
I20260709 12:23:05.781575 17830 raft_consensus.cc:2393] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 15: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
W20260709 12:23:05.676246 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 54340.7us
I20260709 12:23:05.782668 23046 raft_consensus.cc:515] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:23:05.783334 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 542989us
I20260709 12:23:05.784382 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:05.785240 17830 raft_consensus.cc:2393] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 11: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:23:05.784772 18333 leader_election.cc:304] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:05.786033 23135 raft_consensus.cc:2749] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:23:05.783457 17803 leader_election.cc:304] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:05.788143 23046 leader_election.cc:290] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:05.789688 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:05.789657 18333 leader_election.cc:304] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:05.792068 23046 raft_consensus.cc:493] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:05.796679 18333 leader_election.cc:304] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:05.792742 17832 raft_consensus.cc:2468] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 14.
I20260709 12:23:05.802325 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:05.802027 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:05.680860 23201 raft_consensus.cc:493] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:05.798403 17802 leader_election.cc:304] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:23:05.679780 23076 raft_consensus.cc:2804] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:23:05.808955 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:05.804255 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:05.809134 23077 raft_consensus.cc:2749] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
W20260709 12:23:05.813560 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 214562us
I20260709 12:23:05.827517 17686 raft_consensus.cc:1240] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 15. Current term is 16. Ops: [15.9-15.9]
I20260709 12:23:05.811844 17802 leader_election.cc:304] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:23:05.761515 23021 raft_consensus.cc:493] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:05.767305 23180 raft_consensus.cc:493] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:05.839491 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:05.794147 23221 raft_consensus.cc:493] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:23:05.882927 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:05.893175 17687 raft_consensus.cc:2468] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 11.
I20260709 12:23:05.760815 23070 raft_consensus.cc:697] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 14 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:23:05.833910 18333 leader_election.cc:304] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:05.801868 23139 raft_consensus.cc:2749] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:23:05.793673 23135 raft_consensus.cc:2749] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:23:05.802371 23056 raft_consensus.cc:493] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:23:05.817992 23076 raft_consensus.cc:493] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:05.802371 23087 raft_consensus.cc:493] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:23:05.791448 23164 raft_consensus.cc:493] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:05.767983 22933 raft_consensus.cc:515] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:05.840535 17760 raft_consensus.cc:1240] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 13. Current term is 14. Ops: []
I20260709 12:23:05.838935 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 14 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:23:05.832399 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 81201.3us
I20260709 12:23:05.911545 17758 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38786 (request call id 3098) took 1422 ms. Trace:
I20260709 12:23:05.819679 23201 raft_consensus.cc:3060] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:05.817431 22745 consensus_queue.cc:1048] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.121s
I20260709 12:23:05.974525 17835 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35040 (request call id 2664) took 1328 ms. Trace:
I20260709 12:23:05.996673 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:05.997323 17836 raft_consensus.cc:2468] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 16.
I20260709 12:23:05.882421 23113 raft_consensus.cc:697] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 12 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
W20260709 12:23:06.026396 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 192565us
I20260709 12:23:05.903996 23111 raft_consensus.cc:2804] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Leader pre-election won for term 16
W20260709 12:23:05.972054 23214 log.cc:927] Time spent T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.056s	user 0.003s	sys 0.000s
W20260709 12:23:06.021157 23226 log.cc:927] Time spent T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.093s	user 0.000s	sys 0.003s
I20260709 12:23:05.898427 23021 raft_consensus.cc:515] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:05.813151 23046 raft_consensus.cc:515] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:05.901892 22998 consensus_queue.cc:237] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:05.924839 23217 consensus_queue.cc:1048] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.116s
I20260709 12:23:05.886759 22921 consensus_queue.cc:260] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 13.12, Last appended by leader: 12, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:05.926702 22898 raft_consensus.cc:493] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:05.907171 23221 raft_consensus.cc:515] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:06.035027 17652 leader_election.cc:304] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:05.924549 23227 consensus_queue.cc:1048] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.136s
I20260709 12:23:05.900278 23180 raft_consensus.cc:515] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:06.022604 23011 raft_consensus.cc:493] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:23:06.037398 17762 raft_consensus.cc:2468] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 11.
I20260709 12:23:06.053539 17764 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38786 (request call id 3097) took 1596 ms. Trace:
I20260709 12:23:06.068018 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
W20260709 12:23:06.027495 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 215674us
I20260709 12:23:06.106945 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:06.039120 18333 leader_election.cc:304] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:23:06.114861 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:06.037178 17835 rpcz_store.cc:276] 0709 12:23:04.645835 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:04.646688 (+   853us) service_pool.cc:224] Handling call
0709 12:23:04.646950 (+   262us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:04.647150 (+   200us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 12:23:05.331959 (+684809us) log.cc:844] Serialized 14 byte log entry
0709 12:23:05.524411 (+192452us) log.cc:844] Serialized 14 byte log entry
0709 12:23:05.974112 (+449701us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:05.974179 (+    67us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:23:05.974240 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:05.974317 (+    77us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:05.974456 (+   139us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":101320,"prepare.run_cpu_time_us":82,"prepare.run_wall_time_us":83,"raft.queue_time_us":409425,"raft.run_cpu_time_us":289,"raft.run_wall_time_us":290,"thread_start_us":1280172,"threads_started":3,"wal-append.queue_time_us":428864}
I20260709 12:23:06.066951 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:06.034262 17758 rpcz_store.cc:276] 0709 12:23:04.489246 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:05.148027 (+658781us) service_pool.cc:224] Handling call
0709 12:23:05.148340 (+   313us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:05.201330 (+ 52990us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:05.201386 (+    56us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:05.212718 (+ 11332us) log.cc:844] Serialized 26 byte log entry
0709 12:23:05.626166 (+413448us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:23:05.626229 (+    63us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:05.626331 (+   102us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:05.911303 (+284972us) raft_consensus.cc:1685] finished
0709 12:23:05.911325 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:05.911479 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":39959,"raft.queue_time_us":48931,"raft.run_cpu_time_us":206,"raft.run_wall_time_us":11197,"thread_start_us":371366,"threads_started":1,"wal-append.queue_time_us":653517}
I20260709 12:23:06.117017 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:06.118702 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:06.139642 17762 raft_consensus.cc:3060] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Advancing to term 16
W20260709 12:23:06.244493 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 137658us
W20260709 12:23:06.244632 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 129773us
I20260709 12:23:06.242668 17833 raft_consensus.cc:2368] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 15. Current term is 16.
W20260709 12:23:06.118283 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 79221.1us
I20260709 12:23:06.114203 17764 rpcz_store.cc:276] 0709 12:23:04.457088 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:05.147262 (+690174us) service_pool.cc:224] Handling call
0709 12:23:05.147583 (+   321us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:05.166919 (+ 19336us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:05.166960 (+    41us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:05.176431 (+  9471us) log.cc:844] Serialized 26 byte log entry
0709 12:23:05.556232 (+379801us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:23:05.556297 (+    65us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:05.556411 (+   114us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:06.053264 (+496853us) raft_consensus.cc:1685] finished
0709 12:23:06.053292 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:06.053461 (+   169us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":19297,"raft.queue_time_us":29778,"raft.run_cpu_time_us":336,"raft.run_wall_time_us":8102,"thread_start_us":323749,"threads_started":1,"wal-append.queue_time_us":607471}
I20260709 12:23:06.249773 17801 leader_election.cc:304] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:23:06.251242 17758 raft_consensus.cc:1240] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 13. Current term is 14. Ops: [13.8-13.8]
I20260709 12:23:06.117261 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 15 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:06.058828 23133 raft_consensus.cc:2749] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:23:06.025444 23076 raft_consensus.cc:3060] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:05.917420 23070 consensus_queue.cc:237] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:23:06.251407 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:06.340165 17830 raft_consensus.cc:3060] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:06.052193 23111 raft_consensus.cc:493] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:06.257120 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:23:06.259156 17655 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:06.252947 18333 leader_election.cc:400] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 12 pre-election: Vote denied by peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651) with higher term. Message: Invalid argument: T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 12. Current term is 13.
W20260709 12:23:06.341614 17655 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:06.341665 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 88869.9us
I20260709 12:23:06.342494 18333 leader_election.cc:304] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:23:06.047714 22915 consensus_meta.cc:287] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.664s	user 0.007s	sys 0.001s
W20260709 12:23:06.044548 23222 log.cc:927] Time spent T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.157s	user 0.000s	sys 0.000s
W20260709 12:23:06.242447 22590 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.622s	user 0.011s	sys 0.046s
I20260709 12:23:06.369004 17764 raft_consensus.cc:1240] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 14. Current term is 15. Ops: []
I20260709 12:23:06.104835 23114 raft_consensus.cc:2804] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:23:06.033339 23225 raft_consensus.cc:2749] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:23:06.047996 22854 consensus_queue.cc:1059] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 14.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.300s
I20260709 12:23:06.408388 22854 raft_consensus.cc:3055] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 14 LEADER]: Stepping down as leader of term 14
W20260709 12:23:06.408667 23229 log.cc:927] Time spent T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.290s	user 0.000s	sys 0.000s
I20260709 12:23:06.060029 23018 raft_consensus.cc:515] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:06.413694 23018 leader_election.cc:290] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 14 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:06.040781 23167 leader_election.cc:290] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:06.042847 23048 leader_election.cc:290] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:06.145720 23021 leader_election.cc:290] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:06.244386 23088 raft_consensus.cc:493] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:23:06.252296 23104 raft_consensus.cc:2804] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:23:06.256291 22652 raft_consensus.cc:493] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:23:06.370270 22915 raft_consensus.cc:2749] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:23:06.344422 23207 raft_consensus.cc:493] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:06.251674 23240 raft_consensus.cc:2804] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Leader pre-election won for term 12
W20260709 12:23:06.414939 23213 log.cc:927] Time spent T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.384s	user 0.000s	sys 0.001s
I20260709 12:23:06.346937 23111 raft_consensus.cc:3060] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:06.379817 23113 consensus_queue.cc:237] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.7, Last appended by leader: 7, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:06.031303 23056 raft_consensus.cc:515] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:06.345038 22921 raft_consensus.cc:3060] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:06.374817 17762 raft_consensus.cc:2468] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 16.
W20260709 12:23:06.427631 17838 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35040 (request call id 2660) took 1880 ms (client timeout 1659 ms). Trace:
W20260709 12:23:06.607733 17838 rpcz_store.cc:269] 0709 12:23:04.547369 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:04.547538 (+   169us) service_pool.cc:224] Handling call
0709 12:23:06.427425 (+1879887us) spinlock_profiling.cc:243] Waited 211 ms on lock 0x7b5400267354. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:23:06.427552 (+   127us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":486310656}
W20260709 12:23:06.369256 22590 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.754s	user 0.012s	sys 0.048s
W20260709 12:23:06.478842 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 371732us
I20260709 12:23:06.252202 23139 consensus_queue.cc:1059] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true }, Status: INVALID_TERM, Last received: 11.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.145s
I20260709 12:23:06.375648 23099 raft_consensus.cc:493] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:06.520685 23224 raft_consensus.cc:493] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:06.523864 23021 raft_consensus.cc:493] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:06.529201 17683 raft_consensus.cc:2468] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 12.
I20260709 12:23:06.389060 23114 raft_consensus.cc:493] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:23:06.695181 17680 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37402 (request call id 2829) took 1907 ms. Trace:
I20260709 12:23:06.070544 23121 raft_consensus.cc:493] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:23:06.151835 23011 raft_consensus.cc:515] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:06.408854 22854 raft_consensus.cc:740] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
W20260709 12:23:06.637032 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 387855us
I20260709 12:23:06.610275 17652 leader_election.cc:304] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
W20260709 12:23:06.722642 17837 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.755s	user 0.000s	sys 0.013s
I20260709 12:23:06.484935 23225 raft_consensus.cc:493] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:06.528707 23104 raft_consensus.cc:493] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
W20260709 12:23:06.645149 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 395402us
I20260709 12:23:06.533782 23240 raft_consensus.cc:493] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:06.069694 22898 raft_consensus.cc:515] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:06.526950 23207 raft_consensus.cc:515] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:23:06.736322 17655 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:06.563665 23217 raft_consensus.cc:2804] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:23:06.033171 23087 raft_consensus.cc:515] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:06.037250 23164 raft_consensus.cc:515] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:06.697952 17680 rpcz_store.cc:276] 0709 12:23:04.787250 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:04.788052 (+   802us) service_pool.cc:224] Handling call
0709 12:23:04.788361 (+   309us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:23:04.983102 (+194741us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:23:05.308615 (+325513us) log.cc:844] Serialized 14 byte log entry
0709 12:23:05.626805 (+318190us) log.cc:844] Serialized 14 byte log entry
0709 12:23:05.627803 (+   998us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:23:05.672440 (+ 44637us) log.cc:844] Serialized 52 byte log entry
0709 12:23:05.672644 (+   204us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:23:05.672987 (+   343us) log.cc:844] Serialized 14 byte log entry
0709 12:23:05.675923 (+  2936us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:06.677951 (+1002028us) spinlock_profiling.cc:243] Waited 190 ms on lock 0x7b54003507d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:23:06.694750 (+ 16799us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:06.694836 (+    86us) raft_consensus.cc:1685] finished
0709 12:23:06.694862 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:06.695096 (+   234us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":65069,"prepare.run_cpu_time_us":145,"prepare.run_wall_time_us":45303,"raft.queue_time_us":26159,"raft.run_cpu_time_us":563,"raft.run_wall_time_us":24701,"spinlock_wait_cycles":436797184,"thread_start_us":520479,"threads_started":2,"wal-append.queue_time_us":571369}
I20260709 12:23:06.642583 23235 raft_consensus.cc:493] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:06.534096 22652 raft_consensus.cc:515] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:06.528450 23088 raft_consensus.cc:515] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:06.695204 23206 consensus_meta.cc:287] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.972s	user 0.013s	sys 0.002s
W20260709 12:23:06.547501 23222 log.cc:927] Time spent T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.155s	user 0.000s	sys 0.000s
W20260709 12:23:06.724126 17837 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.756s	user 0.000s	sys 0.014s
I20260709 12:23:06.703382 23114 raft_consensus.cc:3060] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:06.647841 23099 raft_consensus.cc:515] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:06.732345 23104 raft_consensus.cc:3060] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Advancing to term 12
W20260709 12:23:06.823915 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 99544.3us
I20260709 12:23:06.701938 23021 raft_consensus.cc:515] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:06.692551 23224 raft_consensus.cc:515] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:06.734474 23240 raft_consensus.cc:3060] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Advancing to term 12
W20260709 12:23:06.879427 17655 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:06.918490 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 308279us
W20260709 12:23:06.933079 17830 consensus_meta.cc:287] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.592s	user 0.011s	sys 0.000s
W20260709 12:23:06.765406 23231 log.cc:927] Time spent T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.105s	user 0.000s	sys 0.000s
I20260709 12:23:06.933864 17830 raft_consensus.cc:2468] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 15.
I20260709 12:23:06.932896 17652 leader_election.cc:304] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:23:06.679867 23120 raft_consensus.cc:493] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:06.758944 23046 leader_election.cc:290] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:06.732117 23225 raft_consensus.cc:515] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:06.749051 23076 raft_consensus.cc:515] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:06.960250 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 230991us
W20260709 12:23:06.978569 17756 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36572 (request call id 935) took 1806 ms (client timeout 1528 ms). Trace:
W20260709 12:23:06.979193 17756 rpcz_store.cc:269] 0709 12:23:05.171783 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:05.202258 (+ 30475us) service_pool.cc:224] Handling call
0709 12:23:06.978349 (+1776091us) spinlock_profiling.cc:243] Waited 72.7 ms on lock 0x7b54002b5f54. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:23:06.978494 (+   145us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":167180672}
W20260709 12:23:06.981237 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 210966us
W20260709 12:23:06.964879 23236 log.cc:927] Time spent T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.556s	user 0.001s	sys 0.000s
I20260709 12:23:06.991578 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:06.953913 23251 raft_consensus.cc:493] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:06.994488 23139 raft_consensus.cc:2804] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:23:06.989696 23248 raft_consensus.cc:493] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:23:06.921267 22854 consensus_queue.cc:260] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:06.762313 23235 raft_consensus.cc:515] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:06.760530 23247 raft_consensus.cc:2749] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:23:06.962318 23120 raft_consensus.cc:515] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:06.940062 23180 leader_election.cc:290] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:06.737934 23121 raft_consensus.cc:515] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:07.072961 23022 raft_consensus.cc:2804] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:23:07.073601 23022 raft_consensus.cc:697] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 16 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:23:07.074579 23022 consensus_queue.cc:237] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.8, Last appended by leader: 8, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:06.767926 23206 raft_consensus.cc:515] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:07.076300 17728 leader_election.cc:304] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:07.076534 18334 leader_election.cc:304] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:23:07.077299 23046 consensus_queue.cc:1059] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 14.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 12:23:07.077728 23206 leader_election.cc:290] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 13 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:05.961396 16501 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
W20260709 12:23:07.075053 17654 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:07.113763 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 180959us
I20260709 12:23:07.028923 23139 raft_consensus.cc:493] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:07.115412 17652 leader_election.cc:304] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:23:07.054905 23056 leader_election.cc:290] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:23:07.108253 23237 log.cc:927] Time spent T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.394s	user 0.000s	sys 0.003s
I20260709 12:23:07.079771 22898 leader_election.cc:290] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:07.011957 23251 raft_consensus.cc:515] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:07.053812 23248 raft_consensus.cc:515] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:07.146762 17804 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.892s	user 0.000s	sys 0.013s
I20260709 12:23:07.109277 23206 raft_consensus.cc:2804] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Leader pre-election won for term 11
W20260709 12:23:07.175489 17804 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.921s	user 0.000s	sys 0.014s
I20260709 12:23:07.079674 23011 leader_election.cc:290] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:07.086877 22854 raft_consensus.cc:3060] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:07.122802 23139 raft_consensus.cc:3060] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Advancing to term 12
W20260709 12:23:07.155628 23216 consensus_meta.cc:287] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 1.425s	user 0.008s	sys 0.008s
I20260709 12:23:07.196187 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:23:07.197264 18363 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59918 (request call id 1704) took 1985 ms (client timeout 2011 ms). Trace:
W20260709 12:23:07.177122 23201 consensus_meta.cc:287] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 1.118s	user 0.003s	sys 0.009s
W20260709 12:23:07.197779 18363 rpcz_store.cc:269] 0709 12:23:05.207693 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:05.209282 (+  1589us) service_pool.cc:224] Handling call
0709 12:23:07.192490 (+1983208us) spinlock_profiling.cc:243] Waited 27 ms on lock 0x7b5400552654. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:23:07.192654 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":62010240}
W20260709 12:23:07.217034 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 234503us
W20260709 12:23:07.223255 17655 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:07.225314 17837 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35042 (request call id 2406) took 1483 ms. Trace:
I20260709 12:23:07.175858 23174 raft_consensus.cc:493] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:07.177873 23206 raft_consensus.cc:493] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:07.250159 23234 raft_consensus.cc:493] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:07.250932 23234 raft_consensus.cc:515] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:07.252638 17757 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38776 (request call id 2396) took 2579 ms. Trace:
I20260709 12:23:07.253082 17757 rpcz_store.cc:276] 0709 12:23:04.673164 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:05.639386 (+966222us) service_pool.cc:224] Handling call
0709 12:23:05.639827 (+   441us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:05.739157 (+ 99330us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:07.252357 (+1513200us) spinlock_profiling.cc:243] Waited 98.4 ms on lock 0x7b54001b9254. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:23:07.252570 (+   213us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":226347776}
I20260709 12:23:07.256464 23088 leader_election.cc:290] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:07.256528 18362 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 1703) took 2049 ms. Trace:
W20260709 12:23:07.265321 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 46554.9us
W20260709 12:23:07.269500 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 425859us
I20260709 12:23:07.257519 18362 rpcz_store.cc:276] 0709 12:23:05.207405 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:05.222646 (+ 15241us) service_pool.cc:224] Handling call
0709 12:23:05.246216 (+ 23570us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:07.173429 (+1927213us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 12:23:07.173464 (+    35us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:07.173705 (+   241us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:23:07.173754 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:07.256321 (+ 82567us) spinlock_profiling.cc:243] Waited 1.91 s on lock 0x7b5400466ed4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:23:07.256364 (+    43us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:07.256475 (+   111us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":15830,"spinlock_wait_cycles":4394388352}
I20260709 12:23:07.199079 23265 raft_consensus.cc:2804] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:23:07.201252 23207 leader_election.cc:290] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:07.242007 23087 leader_election.cc:290] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:07.241082 23099 leader_election.cc:290] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:07.257906 23206 raft_consensus.cc:3060] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:07.286737 17802 leader_election.cc:304] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:23:07.272089 23021 leader_election.cc:290] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:07.202363 23216 raft_consensus.cc:515] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:07.355786 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 77187.6us
I20260709 12:23:07.241807 23007 consensus_queue.cc:1059] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.087s
I20260709 12:23:07.273388 23080 raft_consensus.cc:493] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
W20260709 12:23:07.330868 23241 log.cc:927] Time spent T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.392s	user 0.000s	sys 0.002s
I20260709 12:23:07.242233 23201 raft_consensus.cc:515] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:07.280910 22898 raft_consensus.cc:493] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:07.269848 23174 raft_consensus.cc:515] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:07.337690 23224 leader_election.cc:290] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:07.334627 23266 raft_consensus.cc:2804] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:23:07.355333 23225 leader_election.cc:290] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:23:07.393591 18333 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.050s	user 0.008s	sys 0.023s
W20260709 12:23:07.408422 18333 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.065s	user 0.011s	sys 0.023s
W20260709 12:23:07.409325 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 1.06683e+06us
I20260709 12:23:07.410394 18333 leader_election.cc:304] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:07.423307 18333 leader_election.cc:304] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:23:07.368235 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 291610us
I20260709 12:23:07.378739 23263 raft_consensus.cc:2804] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Leader pre-election won for term 13
W20260709 12:23:07.398913 23256 log.cc:927] Time spent T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.339s	user 0.001s	sys 0.001s
W20260709 12:23:07.397828 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 282498us
I20260709 12:23:07.377599 23076 leader_election.cc:290] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 16 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:07.437757 18333 leader_election.cc:304] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:07.440230 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:07.443148 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:07.442628 17652 leader_election.cc:304] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
W20260709 12:23:07.457441 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 95127.1us
I20260709 12:23:07.459057 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:07.459825 17680 raft_consensus.cc:2393] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 16: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:23:07.466341 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 17 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:23:07.438488 23255 log.cc:927] Time spent T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.330s	user 0.000s	sys 0.001s
W20260709 12:23:07.478546 17652 connection.cc:256] RPC call timeout handler was delayed by -1.4121103599995877s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16590000000000002s for a response
I20260709 12:23:07.423190 23266 raft_consensus.cc:697] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 16 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:23:07.476498 17680 raft_consensus.cc:1275] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 9 index: 11. Preceding OpId from leader: term: 10 index: 12. (index mismatch)
I20260709 12:23:07.476837 17682 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37396 (request call id 2465) took 1746 ms. Trace:
I20260709 12:23:07.429312 23225 raft_consensus.cc:493] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:07.473141 17760 raft_consensus.cc:1275] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 13 index: 10. Preceding OpId from leader: term: 16 index: 12. (index mismatch)
I20260709 12:23:07.486575 17682 rpcz_store.cc:276] 0709 12:23:05.730174 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:05.762762 (+ 32588us) service_pool.cc:224] Handling call
0709 12:23:05.763412 (+   650us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:05.766128 (+  2716us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:05.766157 (+    29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:05.766719 (+   562us) log.cc:844] Serialized 26 byte log entry
0709 12:23:06.056136 (+289417us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:23:06.056388 (+   252us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:06.056464 (+    76us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:07.476634 (+1420170us) raft_consensus.cc:1685] finished
0709 12:23:07.476653 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:07.476772 (+   119us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2160,"raft.queue_time_us":30345,"raft.run_cpu_time_us":236,"raft.run_wall_time_us":28394,"thread_start_us":168301,"threads_started":1,"wal-append.queue_time_us":928424}
I20260709 12:23:07.481278 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:07.381793 22898 raft_consensus.cc:515] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:07.450462 23263 raft_consensus.cc:493] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:07.494506 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:07.513028 17652 leader_election.cc:304] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:07.513363 17680 raft_consensus.cc:2468] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 11.
I20260709 12:23:07.379335 23080 raft_consensus.cc:515] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:07.514106 17761 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36572 (request call id 930) took 3172 ms (client timeout 2418 ms). Trace:
I20260709 12:23:07.500487 17689 raft_consensus.cc:1275] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 12 index: 10. (index mismatch)
W20260709 12:23:07.435110 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 356502us
I20260709 12:23:07.521389 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:07.456372 23121 leader_election.cc:290] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:07.458376 23120 leader_election.cc:290] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:23:07.514598 17761 rpcz_store.cc:269] 0709 12:23:04.317794 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:04.323275 (+  5481us) service_pool.cc:224] Handling call
0709 12:23:07.490136 (+3166861us) spinlock_profiling.cc:243] Waited 1.24 s on lock 0x7b54000fcdd4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:23:07.490297 (+   161us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2846585216}
I20260709 12:23:07.458741 23257 raft_consensus.cc:2804] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Leader election won for term 15
W20260709 12:23:07.523201 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 85788.7us
I20260709 12:23:07.477062 23011 raft_consensus.cc:493] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:07.475630 23244 raft_consensus.cc:3055] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:23:07.537562 18333 leader_election.cc:304] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:07.525468 17834 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37994 (request call id 1056) took 1419 ms. Trace:
I20260709 12:23:07.482517 22853 raft_consensus.cc:493] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:07.548879 17834 rpcz_store.cc:276] 0709 12:23:06.106291 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:06.106409 (+   118us) service_pool.cc:224] Handling call
0709 12:23:06.256423 (+150014us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:06.649493 (+393070us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:06.649530 (+    37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:06.650685 (+  1155us) log.cc:844] Serialized 26 byte log entry
0709 12:23:06.793325 (+142640us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:23:06.793364 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:06.793445 (+    81us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:07.525228 (+731783us) raft_consensus.cc:1685] finished
0709 12:23:07.525251 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:07.525392 (+   141us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":29300,"raft.queue_time_us":387633,"raft.run_cpu_time_us":183,"raft.run_wall_time_us":20069,"spinlock_wait_cycles":742656,"thread_start_us":534140,"threads_started":2,"wal-append.queue_time_us":353150}
I20260709 12:23:07.499009 23265 raft_consensus.cc:697] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 16 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:23:07.558917 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:07.577906 18333 leader_election.cc:304] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:23:07.578732 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 65794us
I20260709 12:23:07.580075 17652 leader_election.cc:304] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:07.530043 23220 raft_consensus.cc:2749] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 12:23:07.526376 23263 raft_consensus.cc:3060] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:07.534085 23076 raft_consensus.cc:493] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:07.541312 23124 raft_consensus.cc:2749] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20260709 12:23:07.577440 23254 log.cc:927] Time spent T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.355s	user 0.001s	sys 0.001s
I20260709 12:23:07.553361 23120 raft_consensus.cc:493] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:07.540452 23046 raft_consensus.cc:493] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:23:07.548426 23244 raft_consensus.cc:740] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 11 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:23:07.578456 23168 raft_consensus.cc:493] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:07.554667 23011 raft_consensus.cc:515] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:07.512727 23225 raft_consensus.cc:515] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:07.546838 23121 raft_consensus.cc:493] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:23:07.559098 23251 leader_election.cc:290] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:07.558022 23257 raft_consensus.cc:697] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 15 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
W20260709 12:23:07.599011 17759 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38776 (request call id 2393) took 3051 ms (client timeout 1659 ms). Trace:
I20260709 12:23:07.608759 17801 leader_election.cc:304] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:07.590826 23234 leader_election.cc:290] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:07.581035 23276 raft_consensus.cc:493] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
W20260709 12:23:07.846761 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 244885us
I20260709 12:23:07.848060 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:07.581619 23248 leader_election.cc:290] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:23:07.609618 23118 raft_consensus.cc:670] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da: failed to trigger leader election: Illegal state: only voting members can start elections: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } [suppressed 2 similar messages]
W20260709 12:23:07.859802 17759 rpcz_store.cc:269] 0709 12:23:04.547316 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:04.548583 (+  1267us) service_pool.cc:224] Handling call
0709 12:23:07.598788 (+3050205us) spinlock_profiling.cc:243] Waited 874 ms on lock 0x7b54002977d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:23:07.598931 (+   143us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2011157888}
I20260709 12:23:07.579490 22853 raft_consensus.cc:515] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:07.913827 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:07.772893 23220 raft_consensus.cc:2749] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 12:23:07.596836 23282 raft_consensus.cc:2749] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:23:07.595811 23277 raft_consensus.cc:493] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:07.891431 23271 log.cc:927] Time spent T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.286s	user 0.001s	sys 0.000s
W20260709 12:23:07.905983 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 46268.4us
I20260709 12:23:07.842128 23264 raft_consensus.cc:493] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:07.916782 18333 leader_election.cc:304] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:07.947561 18333 leader_election.cc:304] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:07.950251 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:23:07.951398 23283 log.cc:927] Time spent T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.369s	user 0.000s	sys 0.001s
I20260709 12:23:07.953768 23118 raft_consensus.cc:2804] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:23:07.954555 23118 raft_consensus.cc:697] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 15 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
W20260709 12:23:07.958539 17728 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.16.29.65:33071 timed out after 1.757s (SENT) [suppressed 19 similar messages]
I20260709 12:23:07.598445 23076 raft_consensus.cc:515] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:07.859864 23207 raft_consensus.cc:493] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:23:07.847481 23235 leader_election.cc:290] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:08.029441 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:07.605738 23120 raft_consensus.cc:515] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:07.782284 23046 raft_consensus.cc:515] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:08.025017 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:23:07.933756 17654 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.806s	user 0.003s	sys 0.029s
W20260709 12:23:08.073506 17654 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.946s	user 0.003s	sys 0.030s
W20260709 12:23:08.069746 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 461044us
W20260709 12:23:08.040375 18335 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.595s	user 0.007s	sys 0.002s
W20260709 12:23:08.074689 18335 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.638s	user 0.007s	sys 0.003s
W20260709 12:23:08.022701 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 64139.5us
W20260709 12:23:07.956180 23245 log.cc:927] Time spent T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.601s	user 0.001s	sys 0.001s
I20260709 12:23:08.073490 17759 raft_consensus.cc:2468] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 11.
I20260709 12:23:07.891499 23124 consensus_queue.cc:1048] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.135s
I20260709 12:23:07.917002 23289 raft_consensus.cc:2749] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:23:07.931753 23216 leader_election.cc:290] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 17 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:08.069944 23207 raft_consensus.cc:515] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:07.884675 23121 raft_consensus.cc:515] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:08.108784 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:23:08.125104 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 50409.3us
I20260709 12:23:08.125440 17686 raft_consensus.cc:2468] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 13.
I20260709 12:23:08.108381 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:07.839359 23168 raft_consensus.cc:515] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:08.118444 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:08.127745 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 17 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:07.609282 23266 consensus_queue.cc:237] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:08.124295 18333 connection.cc:256] RPC call timeout handler was delayed by -1.7068693730002451s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15280000000000002s for a response
I20260709 12:23:08.042289 23201 leader_election.cc:290] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 16 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:23:08.155565 18333 connection.cc:256] RPC call timeout handler was delayed by -1.5322693730004175s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17220000000000002s for a response
W20260709 12:23:08.156345 18333 connection.cc:256] RPC call timeout handler was delayed by -1.4782693730003302s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17820000000000003s for a response
W20260709 12:23:08.156764 18333 connection.cc:256] RPC call timeout handler was delayed by -1.4512693730002866s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18120000000000003s for a response
I20260709 12:23:08.122375 23120 leader_election.cc:290] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:08.081414 23174 leader_election.cc:290] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:08.181350 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:08.035302 23264 raft_consensus.cc:515] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:07.945870 23277 raft_consensus.cc:515] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:07.918025 23276 raft_consensus.cc:515] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:08.110852 23282 raft_consensus.cc:493] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:08.209002 23297 raft_consensus.cc:2804] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:23:08.118245 23056 raft_consensus.cc:493] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:08.208451 23258 raft_consensus.cc:2749] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
W20260709 12:23:08.287031 18336 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.789s	user 0.011s	sys 0.008s
W20260709 12:23:08.287732 17689 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.687s	user 0.011s	sys 0.005s
W20260709 12:23:08.385531 17689 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.789s	user 0.012s	sys 0.005s
W20260709 12:23:08.197036 23022 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.607s	user 0.000s	sys 0.015s
I20260709 12:23:08.133684 23179 raft_consensus.cc:493] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:08.294191 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:08.294415 23201 raft_consensus.cc:493] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:07.930742 23265 consensus_queue.cc:237] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.11, Last appended by leader: 11, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:08.302107 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 12 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:08.303774 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:08.304843 17833 raft_consensus.cc:1240] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 16. Current term is 17. Ops: []
I20260709 12:23:07.955873 23118 consensus_queue.cc:237] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.11, Last appended by leader: 11, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:08.305755 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:08.458587 17835 raft_consensus.cc:2468] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 11.
I20260709 12:23:08.307905 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:08.454454 17834 raft_consensus.cc:2468] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 15.
I20260709 12:23:08.459980 17688 raft_consensus.cc:2393] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 14: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:23:08.323506 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:08.462116 17686 raft_consensus.cc:2393] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 12: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:23:08.370261 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:08.370730 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:23:08.325378 17681 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.583s	user 0.000s	sys 0.020s
I20260709 12:23:08.375710 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
W20260709 12:23:08.462687 17681 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.863s	user 0.002s	sys 0.029s
I20260709 12:23:08.376312 17836 raft_consensus.cc:1240] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 15. Current term is 16. Ops: []
I20260709 12:23:08.462889 17831 raft_consensus.cc:3060] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Advancing to term 16
W20260709 12:23:08.376505 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 91998us
I20260709 12:23:07.957522 23244 consensus_queue.cc:260] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 11.10, Last appended by leader: 10, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:08.381448 23076 leader_election.cc:290] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:08.382481 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:08.469134 17829 raft_consensus.cc:2468] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 13.
W20260709 12:23:08.386584 18336 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.888s	user 0.011s	sys 0.008s
I20260709 12:23:08.389609 17685 raft_consensus.cc:2468] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 15.
W20260709 12:23:08.398797 17804 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.217s	user 0.006s	sys 0.038s
I20260709 12:23:08.464185 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:08.377223 22898 leader_election.cc:290] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:08.462749 17838 raft_consensus.cc:2468] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 13.
I20260709 12:23:08.381667 23168 leader_election.cc:290] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:08.464053 18333 leader_election.cc:304] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:08.470160 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:08.471273 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 16 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:23:08.471762 17804 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.290s	user 0.006s	sys 0.039s
W20260709 12:23:08.408422 23022 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.823s	user 0.001s	sys 0.015s
I20260709 12:23:08.376345 23120 raft_consensus.cc:493] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:08.543324 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:08.559772 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:08.541790 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:08.512753 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:23:08.562213 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 98231.4us
I20260709 12:23:08.514698 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:08.515436 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:08.468489 23011 leader_election.cc:290] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:08.519500 17758 raft_consensus.cc:3060] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Advancing to term 12
W20260709 12:23:08.521103 17801 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:08.389535 23133 raft_consensus.cc:493] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:08.523805 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:08.531530 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:08.583828 17681 raft_consensus.cc:2410] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for term 12 because replica has last-logged OpId of term: 11 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 9.
I20260709 12:23:08.529497 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:08.536868 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 11 candidate_status { last_received { term: 9 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:08.537961 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:08.376190 23297 raft_consensus.cc:493] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:23:08.564059 18365 raft_consensus.cc:2468] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 17.
I20260709 12:23:08.183341 23257 consensus_queue.cc:237] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.8, Last appended by leader: 8, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:08.515959 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:08.541033 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 17 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:08.590813 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:08.593803 18374 raft_consensus.cc:2410] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for term 17 because replica has last-logged OpId of term: 16 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 12.
I20260709 12:23:08.594455 18361 raft_consensus.cc:2468] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 11.
I20260709 12:23:08.597087 18367 raft_consensus.cc:1275] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 12 index: 10. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
I20260709 12:23:08.563596 18333 leader_election.cc:304] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:08.564437 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 17 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:08.600284 17685 raft_consensus.cc:2410] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for term 17 because replica has last-logged OpId of term: 15 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 9.
I20260709 12:23:08.403697 23056 raft_consensus.cc:515] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:08.559150 23291 log.cc:927] Time spent T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.123s	user 0.001s	sys 0.001s
I20260709 12:23:08.389828 23021 raft_consensus.cc:493] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:08.568719 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:08.571871 17689 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37402 (request call id 2848) took 1039 ms. Trace:
I20260709 12:23:08.578541 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:08.560083 17802 leader_election.cc:304] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:23:08.633869 17759 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 12.
W20260709 12:23:08.634006 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 74011.1us
I20260709 12:23:08.579353 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:08.635751 17801 leader_election.cc:304] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:08.636251 17829 raft_consensus.cc:2468] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 13.
I20260709 12:23:08.580127 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:08.460119 23179 raft_consensus.cc:515] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:08.650923 23179 leader_election.cc:290] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:08.606046 23281 consensus_queue.cc:1048] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.061s
I20260709 12:23:08.653414 23281 consensus_queue.cc:1048] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
W20260709 12:23:08.655553 23104 consensus_meta.cc:287] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 1.713s	user 0.011s	sys 0.000s
I20260709 12:23:08.558626 23124 raft_consensus.cc:493] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:08.578126 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:08.591414 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:08.566962 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:08.603577 17684 raft_consensus.cc:1240] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 16 LEADER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 15. Current term is 16. Ops: [15.9-15.9]
I20260709 12:23:08.678285 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 17 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:08.526479 23287 raft_consensus.cc:493] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:08.608175 17831 raft_consensus.cc:2468] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 16.
I20260709 12:23:08.678102 18362 raft_consensus.cc:2376] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 12. Re-sending same reply.
I20260709 12:23:08.524297 23260 raft_consensus.cc:493] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:08.655596 23139 consensus_meta.cc:287] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 1.470s	user 0.011s	sys 0.000s
I20260709 12:23:08.590806 23120 raft_consensus.cc:515] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:08.639382 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
W20260709 12:23:08.639585 23288 log.cc:927] Time spent T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.508s	user 0.001s	sys 0.000s
I20260709 12:23:08.639827 23304 raft_consensus.cc:493] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:08.639575 23227 raft_consensus.cc:2804] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Leader pre-election won for term 12
W20260709 12:23:08.640550 23301 log.cc:927] Time spent T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.105s	user 0.000s	sys 0.000s
W20260709 12:23:08.640894 23298 log.cc:927] Time spent T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.175s	user 0.000s	sys 0.001s
I20260709 12:23:08.641141 23293 raft_consensus.cc:493] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
W20260709 12:23:08.641863 23278 log.cc:927] Time spent T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.571s	user 0.001s	sys 0.001s
I20260709 12:23:08.603544 23133 raft_consensus.cc:515] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:08.642300 23275 raft_consensus.cc:493] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:08.642515 23276 leader_election.cc:290] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:23:08.643383 23302 log.cc:927] Time spent T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.107s	user 0.000s	sys 0.000s
I20260709 12:23:08.644176 23121 leader_election.cc:290] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:08.567097 22898 raft_consensus.cc:493] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:08.539979 23306 raft_consensus.cc:493] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:08.566599 23224 raft_consensus.cc:493] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:08.648416 23284 raft_consensus.cc:493] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:08.590695 23174 raft_consensus.cc:493] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:08.383956 23282 raft_consensus.cc:515] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:08.595136 23250 raft_consensus.cc:2804] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:23:08.464977 23201 raft_consensus.cc:515] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:08.601188 23297 raft_consensus.cc:3060] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:08.540621 23113 raft_consensus.cc:493] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:08.524492 23290 raft_consensus.cc:2749] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
W20260709 12:23:08.659814 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 96388.6us
I20260709 12:23:08.660207 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:08.582863 17764 raft_consensus.cc:1275] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 14 index: 8. (index mismatch)
I20260709 12:23:08.561045 23168 raft_consensus.cc:493] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:23:08.680359 23104 raft_consensus.cc:515] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:08.672067 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 93190.9us
I20260709 12:23:08.676903 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:08.636623 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:23:08.535701 23286 log.cc:927] Time spent T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.694s	user 0.002s	sys 0.000s
I20260709 12:23:08.681646 17762 raft_consensus.cc:3060] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:08.682425 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:08.685261 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:08.687194 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 16 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:08.605165 23080 leader_election.cc:290] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:08.688640 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:08.690259 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:08.691768 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:23:08.693037 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 57332.5us
I20260709 12:23:08.694180 17758 raft_consensus.cc:1275] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 12 index: 8. (index mismatch)
I20260709 12:23:08.701508 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:08.672453 23021 raft_consensus.cc:515] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:09.310638 17683 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37396 (request call id 2464) took 3581 ms (client timeout 2080 ms). Trace:
W20260709 12:23:09.774923 17683 rpcz_store.cc:269] 0709 12:23:05.729392 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:05.732020 (+  2628us) service_pool.cc:224] Handling call
0709 12:23:09.310363 (+3578343us) spinlock_profiling.cc:243] Waited 768 ms on lock 0x7b54000c5b54. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:23:09.310546 (+   183us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1767342976}
W20260709 12:23:09.677009 17757 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.858s	user 0.004s	sys 0.019s
I20260709 12:23:09.775493 17758 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38786 (request call id 3122) took 1256 ms. Trace:
W20260709 12:23:09.030339 18335 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.514s	user 0.023s	sys 0.014s
I20260709 12:23:08.681003 23251 raft_consensus.cc:2749] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:23:09.863248 17685 raft_consensus.cc:2468] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 13.
I20260709 12:23:09.862211 17727 leader_election.cc:304] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:09.762318 17764 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38786 (request call id 3124) took 1215 ms. Trace:
W20260709 12:23:09.306291 18334 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.631s	user 0.006s	sys 0.014s
I20260709 12:23:07.232926 17837 rpcz_store.cc:276] 0709 12:23:05.741838 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:05.742069 (+   231us) service_pool.cc:224] Handling call
0709 12:23:05.742474 (+   405us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:05.742744 (+   270us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 12:23:05.967519 (+224775us) log.cc:844] Serialized 14 byte log entry
0709 12:23:07.224818 (+1257299us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:07.224903 (+    85us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:23:07.224975 (+    72us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:07.225069 (+    94us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:07.225233 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":36658,"prepare.queue_time_us":88360,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":46,"raft.queue_time_us":354087,"raft.run_cpu_time_us":124,"raft.run_wall_time_us":7606,"thread_start_us":1426827,"threads_started":3,"wal-append.queue_time_us":1045615}
W20260709 12:23:09.776515 17757 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.957s	user 0.006s	sys 0.019s
W20260709 12:23:08.901124 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 50086us
W20260709 12:23:09.861850 18335 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.346s	user 0.027s	sys 0.022s
I20260709 12:23:09.776708 17758 rpcz_store.cc:276] 0709 12:23:08.518983 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:08.519107 (+   124us) service_pool.cc:224] Handling call
0709 12:23:08.519378 (+   271us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:09.775037 (+1255659us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:09.775413 (+   376us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:23:09.805325 17654 connection.cc:256] RPC call timeout handler was delayed by -1.7740703250001388s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2s for a response
I20260709 12:23:09.762861 17831 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35042 (request call id 2418) took 1078 ms. Trace:
W20260709 12:23:08.795544 17652 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.212s	user 0.002s	sys 0.021s
I20260709 12:23:09.868662 17829 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35042 (request call id 2419) took 1184 ms. Trace:
I20260709 12:23:09.768489 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:23:09.672711 18368 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 1.042s	user 0.008s	sys 0.012s
I20260709 12:23:08.728008 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:23:09.693886 17802 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:09.870390 17832 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35042 (request call id 2422) took 1183 ms. Trace:
I20260709 12:23:09.762754 18333 leader_election.cc:400] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 15 election: Vote denied by peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837) with higher term. Message: Invalid argument: T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 15. Current term is 16.
I20260709 12:23:09.777318 17680 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44898 (request call id 1059) took 1667 ms. Trace:
I20260709 12:23:08.688621 23314 raft_consensus.cc:493] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:08.904215 23111 consensus_meta.cc:287] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 2.266s	user 0.014s	sys 0.005s
I20260709 12:23:09.880345 17687 raft_consensus.cc:2393] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 12: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
W20260709 12:23:09.889683 17688 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37396 (request call id 2491) took 1208 ms (client timeout 1550 ms). Trace:
W20260709 12:23:09.277915 23243 log.cc:927] Time spent T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 2.539s	user 0.006s	sys 0.015s
W20260709 12:23:09.027058 23240 consensus_meta.cc:287] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 2.073s	user 0.009s	sys 0.003s
I20260709 12:23:08.708495 23216 raft_consensus.cc:493] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
W20260709 12:23:09.362787 23285 log.cc:927] Time spent T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.822s	user 0.001s	sys 0.000s
I20260709 12:23:08.726776 23308 raft_consensus.cc:493] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:09.892225 17831 rpcz_store.cc:276] 0709 12:23:08.684131 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:08.684249 (+   118us) service_pool.cc:224] Handling call
0709 12:23:09.762056 (+1077807us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:09.762233 (+   177us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:09.762258 (+    25us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:09.762299 (+    41us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:23:09.762326 (+    27us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:09.762434 (+   108us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:09.762803 (+   369us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:23:09.894032 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 131681us
I20260709 12:23:09.893441 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:23:09.893523 17652 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 2.310s	user 0.021s	sys 0.062s
W20260709 12:23:09.929345 17688 rpcz_store.cc:269] 0709 12:23:08.681326 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:08.682203 (+   877us) service_pool.cc:224] Handling call
0709 12:23:09.889573 (+1207370us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:23:09.900509 17685 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37396 (request call id 2488) took 1268 ms (client timeout 1653 ms). Trace:
W20260709 12:23:09.965742 17685 rpcz_store.cc:269] 0709 12:23:08.632157 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:08.632341 (+   184us) service_pool.cc:224] Handling call
0709 12:23:09.900449 (+1268108us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:09.895498 17680 rpcz_store.cc:276] 0709 12:23:08.109738 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:08.130459 (+ 20721us) service_pool.cc:224] Handling call
0709 12:23:08.283668 (+153209us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:08.294578 (+ 10910us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 12:23:08.294613 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:08.336978 (+ 42365us) log.cc:844] Serialized 26 byte log entry
0709 12:23:08.630198 (+293220us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:23:08.630241 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:08.630311 (+    70us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:09.777053 (+1146742us) raft_consensus.cc:1685] finished
0709 12:23:09.777077 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:09.777243 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":19659,"raft.queue_time_us":162467,"raft.run_cpu_time_us":137,"raft.run_wall_time_us":9153,"thread_start_us":177404,"threads_started":1,"wal-append.queue_time_us":649498}
I20260709 12:23:09.901332 17757 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38786 (request call id 3121) took 1383 ms. Trace:
I20260709 12:23:09.906580 23220 raft_consensus.cc:493] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:09.974444 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:09.975308 17838 raft_consensus.cc:1240] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 14 LEADER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 13. Current term is 14. Ops: [13.8-13.8]
W20260709 12:23:09.902096 18334 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.227s	user 0.011s	sys 0.027s
I20260709 12:23:08.695739 23046 leader_election.cc:290] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:08.710278 23281 raft_consensus.cc:2749] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
W20260709 12:23:09.228609 23265 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.570s	user 0.015s	sys 0.033s
I20260709 12:23:09.906847 17681 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37402 (request call id 2855) took 1215 ms. Trace:
I20260709 12:23:09.985702 17681 rpcz_store.cc:276] 0709 12:23:08.691130 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:08.727676 (+ 36546us) service_pool.cc:224] Handling call
0709 12:23:09.906761 (+1179085us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:09.894124 17829 rpcz_store.cc:276] 0709 12:23:08.684411 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:08.684544 (+   133us) service_pool.cc:224] Handling call
0709 12:23:09.868579 (+1184035us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:23:09.986765 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:09.701887 23263 consensus_meta.cc:287] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 2.101s	user 0.013s	sys 0.000s
I20260709 12:23:09.920887 18363 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59918 (request call id 1709) took 1244 ms. Trace:
I20260709 12:23:09.877976 23168 raft_consensus.cc:515] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:09.880084 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:23:09.929283 17687 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37396 (request call id 2489) took 1269 ms (client timeout 1633 ms). Trace:
I20260709 12:23:09.867916 17684 raft_consensus.cc:2468] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 13.
W20260709 12:23:09.147544 23206 consensus_meta.cc:287] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 1.771s	user 0.010s	sys 0.003s
W20260709 12:23:09.153919 23310 log.cc:927] Time spent T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.088s	user 0.000s	sys 0.001s
I20260709 12:23:09.895166 22898 raft_consensus.cc:515] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:09.932595 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 17 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:23:09.864854 17801 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:09.927173 17727 leader_election.cc:304] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:23:09.254240 22854 consensus_meta.cc:287] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 2.068s	user 0.012s	sys 0.000s
I20260709 12:23:09.118359 23311 raft_consensus.cc:493] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:08.768911 23227 raft_consensus.cc:493] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:09.019667 23277 leader_election.cc:290] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:23:09.894492 18368 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.289s	user 0.012s	sys 0.014s
W20260709 12:23:09.217346 23114 consensus_meta.cc:287] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 2.289s	user 0.015s	sys 0.000s
I20260709 12:23:09.950007 18371 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 12.
I20260709 12:23:09.021812 23056 leader_election.cc:290] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:08.849148 23303 raft_consensus.cc:2749] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:23:09.247328 23120 leader_election.cc:290] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:10.032899 23282 leader_election.cc:290] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:09.279947 22933 leader_election.cc:290] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:08.771909 23207 leader_election.cc:290] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:09.254415 23047 raft_consensus.cc:3055] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:23:09.893667 18367 raft_consensus.cc:2393] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 11: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:23:08.734481 23225 leader_election.cc:290] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:08.772092 23221 leader_election.cc:290] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:08.688359 23124 raft_consensus.cc:515] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:09.954852 18333 leader_election.cc:304] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:09.893290 17832 rpcz_store.cc:276] 0709 12:23:08.686843 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:08.686954 (+   111us) service_pool.cc:224] Handling call
0709 12:23:09.870313 (+1183359us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:09.959614 23250 raft_consensus.cc:697] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 16 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:23:08.696249 23287 raft_consensus.cc:515] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:08.709815 23139 raft_consensus.cc:515] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:09.925630 23276 raft_consensus.cc:2749] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:23:09.968281 17757 rpcz_store.cc:276] 0709 12:23:08.517692 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:08.518205 (+   513us) service_pool.cc:224] Handling call
0709 12:23:08.518492 (+   287us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:08.518695 (+   203us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 12:23:08.818993 (+300298us) log.cc:844] Serialized 14 byte log entry
0709 12:23:09.901008 (+1082015us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:09.901058 (+    50us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:23:09.901105 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:09.901172 (+    67us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:09.901278 (+   106us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":14533,"prepare.queue_time_us":381167,"prepare.run_cpu_time_us":30,"prepare.run_wall_time_us":31,"thread_start_us":1318949,"threads_started":2}
I20260709 12:23:10.058212 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 17 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:09.879501 17764 rpcz_store.cc:276] 0709 12:23:08.547217 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:08.581844 (+ 34627us) service_pool.cc:224] Handling call
0709 12:23:08.582473 (+   629us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:09.762017 (+1179544us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:09.762255 (+   238us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:09.117766 23264 leader_election.cc:290] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:09.906078 23179 raft_consensus.cc:493] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
W20260709 12:23:09.982882 18334 connection.cc:256] RPC call timeout handler was delayed by -1.1487063430004127s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15510000000000002s for a response
I20260709 12:23:09.984849 23133 leader_election.cc:290] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:08.900784 23318 consensus_queue.cc:1059] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 14.8, Next index: 9, Last known committed idx: 8, Time since last communication: 1.071s
I20260709 12:23:09.992774 18363 rpcz_store.cc:276] 0709 12:23:08.676407 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:08.676585 (+   178us) service_pool.cc:224] Handling call
0709 12:23:09.920786 (+1244201us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:23:09.695420 23309 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.527s	user 0.000s	sys 0.013s
I20260709 12:23:09.878994 23224 raft_consensus.cc:515] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:08.712271 23260 raft_consensus.cc:515] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:09.997673 17833 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 12.
I20260709 12:23:09.894932 23174 raft_consensus.cc:515] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:09.997421 23217 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 3.175s	user 0.008s	sys 0.028s
W20260709 12:23:09.029425 22921 consensus_meta.cc:287] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 2.386s	user 0.014s	sys 0.013s
I20260709 12:23:10.118120 23303 raft_consensus.cc:493] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
W20260709 12:23:09.985635 23265 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.327s	user 0.017s	sys 0.048s
I20260709 12:23:08.774238 23304 raft_consensus.cc:515] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:10.003168 23118 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.767s	user 0.020s	sys 0.014s
I20260709 12:23:09.948807 23111 raft_consensus.cc:515] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:23:10.128516 23228 log.cc:927] Time spent T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 3.856s	user 0.027s	sys 0.019s
I20260709 12:23:10.129601 23111 leader_election.cc:290] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 16 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:10.130640 17837 raft_consensus.cc:2468] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 16.
I20260709 12:23:10.130709 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:10.131790 18367 raft_consensus.cc:2468] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 12.
I20260709 12:23:09.891104 23306 raft_consensus.cc:515] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:09.892048 23113 raft_consensus.cc:515] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:09.952265 23240 raft_consensus.cc:515] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:10.122062 23217 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 3.329s	user 0.009s	sys 0.028s
I20260709 12:23:10.118381 23120 raft_consensus.cc:493] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:10.000880 23266 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.883s	user 0.030s	sys 0.048s
W20260709 12:23:10.148694 23266 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.031s	user 0.031s	sys 0.048s
I20260709 12:23:09.956426 23216 raft_consensus.cc:515] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:10.127708 23250 consensus_queue.cc:237] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:10.150957 23216 leader_election.cc:290] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:23:09.993335 23076 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.710s	user 0.001s	sys 0.012s
W20260709 12:23:10.151682 23076 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.903s	user 0.002s	sys 0.012s
W20260709 12:23:10.002460 22915 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 3.267s	user 0.014s	sys 0.024s
I20260709 12:23:10.152025 23104 leader_election.cc:290] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 12 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:23:10.152272 22915 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 3.423s	user 0.015s	sys 0.025s
I20260709 12:23:09.891469 23275 raft_consensus.cc:515] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:10.123443 23056 consensus_queue.cc:1059] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 14.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.034s
I20260709 12:23:10.154857 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:10.155093 23275 leader_election.cc:290] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:09.931969 23293 raft_consensus.cc:515] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:10.155529 18367 raft_consensus.cc:2468] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 13.
I20260709 12:23:10.156495 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:10.122354 22898 leader_election.cc:290] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:10.160274 17838 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35040 (request call id 2688) took 1574 ms. Trace:
I20260709 12:23:10.160822 17838 rpcz_store.cc:276] 0709 12:23:08.586298 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:08.586582 (+   284us) service_pool.cc:224] Handling call
0709 12:23:08.586927 (+   345us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:09.975066 (+1388139us) spinlock_profiling.cc:243] Waited 4.75 ms on lock 0x7b54000ef4a0. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a597dc0 00007f7b4a6221c0 00007f7b4a623231 00007f7b4a61fd8f 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c
0709 12:23:09.975753 (+   687us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:10.159670 (+183917us) spinlock_profiling.cc:243] Waited 1.33 ms on lock 0x7b54000ef4a0. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a597dc0 00007f7b4a623a02 00007f7b4a61fdd6 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2
0709 12:23:10.160193 (+   523us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":13997312}
I20260709 12:23:09.965382 23308 raft_consensus.cc:515] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:23:10.006626 23257 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.791s	user 0.010s	sys 0.019s
I20260709 12:23:09.943365 23314 raft_consensus.cc:515] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:10.021762 23227 raft_consensus.cc:3060] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Advancing to term 12
W20260709 12:23:10.054770 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 100037us
I20260709 12:23:10.061456 23021 leader_election.cc:290] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:10.062274 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 17 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:10.012244 23249 consensus_queue.cc:1048] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.044s
I20260709 12:23:10.066042 23201 leader_election.cc:290] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:10.005339 23206 raft_consensus.cc:515] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:10.000701 23263 raft_consensus.cc:515] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:09.988659 23220 raft_consensus.cc:515] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:10.116572 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:10.057262 23046 raft_consensus.cc:2764] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Leader pre-election decision vote started in defunct term 14: won
I20260709 12:23:10.053540 23251 raft_consensus.cc:493] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:10.049680 23047 raft_consensus.cc:740] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:23:10.057693 23225 raft_consensus.cc:2749] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:23:10.039997 23114 raft_consensus.cc:515] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:10.191006 23260 leader_election.cc:290] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:10.066983 23325 raft_consensus.cc:493] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:10.018464 23311 raft_consensus.cc:515] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:10.116708 23309 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.972s	user 0.000s	sys 0.015s
I20260709 12:23:09.990926 23148 raft_consensus.cc:493] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
W20260709 12:23:10.001338 23272 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.704s	user 0.004s	sys 0.023s
W20260709 12:23:10.133448 23118 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.902s	user 0.020s	sys 0.014s
I20260709 12:23:09.894780 23284 raft_consensus.cc:515] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:10.196528 23087 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.937s	user 0.024s	sys 0.099s
I20260709 12:23:10.119822 23321 raft_consensus.cc:493] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:10.148001 23217 raft_consensus.cc:697] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 15 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:23:10.112457 23179 raft_consensus.cc:515] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:10.168807 23257 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.953s	user 0.011s	sys 0.020s
I20260709 12:23:10.122630 23303 raft_consensus.cc:515] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:10.160075 23120 raft_consensus.cc:515] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:10.206162 23272 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.909s	user 0.004s	sys 0.027s
I20260709 12:23:10.204453 23246 raft_consensus.cc:3055] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:23:10.241681 23246 raft_consensus.cc:740] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:23:10.229553 23287 leader_election.cc:290] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:10.214288 23293 leader_election.cc:290] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:10.193634 23251 raft_consensus.cc:515] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:10.240074 23232 log.cc:927] Time spent T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 3.501s	user 0.013s	sys 0.026s
W20260709 12:23:10.256901 23323 log.cc:927] Time spent T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.137s	user 0.001s	sys 0.000s
I20260709 12:23:10.251214 23240 leader_election.cc:290] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 12 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:10.252560 23311 leader_election.cc:290] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:10.254338 23303 leader_election.cc:290] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:10.233922 23321 raft_consensus.cc:515] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:10.259435 23284 leader_election.cc:290] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:23:10.247964 23326 log.cc:927] Time spent T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.125s	user 0.000s	sys 0.001s
I20260709 12:23:10.266961 23308 leader_election.cc:290] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:10.267958 23306 leader_election.cc:290] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:10.268215 23314 leader_election.cc:290] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:10.206681 23325 raft_consensus.cc:515] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:23:10.270483 23325 leader_election.cc:290] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:10.271417 23304 leader_election.cc:290] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:10.272708 23321 leader_election.cc:290] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:23:09.958031 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 2.37786e+06us
I20260709 12:23:09.993606 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:10.328517 17680 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37396 (request call id 2494) took 1558 ms. Trace:
I20260709 12:23:10.328939 17680 rpcz_store.cc:276] 0709 12:23:08.769508 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:09.975796 (+1206288us) service_pool.cc:224] Handling call
0709 12:23:10.328253 (+352457us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:10.330865 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 17 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:09.993422 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:09.997679 17684 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37402 (request call id 2854) took 1324 ms. Trace:
I20260709 12:23:10.332484 17684 rpcz_store.cc:276] 0709 12:23:08.673706 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:08.691460 (+ 17754us) service_pool.cc:224] Handling call
0709 12:23:09.997615 (+1306155us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:10.332942 17681 raft_consensus.cc:2393] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 16: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:23:10.334995 17684 raft_consensus.cc:1275] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 11 index: 8. Preceding OpId from leader: term: 12 index: 9. (index mismatch)
W20260709 12:23:10.324548 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 431880us
I20260709 12:23:10.324666 17683 raft_consensus.cc:2468] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 14.
I20260709 12:23:10.326073 17685 raft_consensus.cc:2468] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 14.
I20260709 12:23:10.324702 17682 raft_consensus.cc:1240] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 16 LEADER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 15. Current term is 16. Ops: []
I20260709 12:23:10.335840 17684 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37402 (request call id 2852) took 1672 ms. Trace:
W20260709 12:23:10.338640 17654 connection.cc:256] RPC call timeout handler was delayed by -1.3709384720000344s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2014s for a response
I20260709 12:23:10.338634 17684 rpcz_store.cc:276] 0709 12:23:08.663357 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:10.333667 (+1670310us) service_pool.cc:224] Handling call
0709 12:23:10.334408 (+   741us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:10.335494 (+  1086us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:10.335740 (+   246us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:23:09.996752 17687 rpcz_store.cc:269] 0709 12:23:08.660401 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:08.660643 (+   242us) service_pool.cc:224] Handling call
0709 12:23:09.929215 (+1268572us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:09.977201 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 17 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:08.632510 17689 rpcz_store.cc:276] 0709 12:23:07.532746 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:07.532963 (+   217us) service_pool.cc:224] Handling call
0709 12:23:07.553407 (+ 20444us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:07.553618 (+   211us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 12:23:08.386148 (+832530us) log.cc:844] Serialized 14 byte log entry
0709 12:23:08.571536 (+185388us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:08.571590 (+    54us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:23:08.571639 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:08.571704 (+    65us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:08.571816 (+   112us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":65073,"prepare.queue_time_us":364222,"prepare.run_cpu_time_us":58,"prepare.run_wall_time_us":61,"raft.queue_time_us":37001,"raft.run_cpu_time_us":208,"raft.run_wall_time_us":50995,"thread_start_us":938308,"threads_started":2}
I20260709 12:23:10.399755 17763 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38786 (request call id 3095) took 5977 ms. Trace:
I20260709 12:23:10.401553 17689 raft_consensus.cc:1275] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 12 index: 8. (index mismatch)
I20260709 12:23:10.400202 17763 rpcz_store.cc:276] 0709 12:23:04.421718 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:05.116065 (+694347us) service_pool.cc:224] Handling call
0709 12:23:05.116385 (+   320us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:05.274554 (+158169us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:05.274599 (+    45us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:05.275188 (+   589us) log.cc:844] Serialized 26 byte log entry
0709 12:23:05.822085 (+546897us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:23:05.822126 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:07.071306 (+1249180us) spinlock_profiling.cc:243] Waited 158 ms on lock 0x7b54003eb054. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:23:07.071358 (+    52us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:10.112080 (+3040722us) spinlock_profiling.cc:243] Waited 65.7 ms on lock 0x7b4400022b4c. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b47459c85 00007f7b4745a6a2 00007f7b4a1cc1db 00007f7b4a1c7aaf 00007f7b4a1bb86e 00007f7b4a1babb6 00007f7b4a1ba7d8 00007f7b4a1c4221 00007f7b4a1c41d9 00007f7b4a1c4169
0709 12:23:10.399130 (+287050us) raft_consensus.cc:1685] finished
0709 12:23:10.399181 (+    51us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:10.399487 (+   306us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1899,"raft.run_cpu_time_us":306,"raft.run_wall_time_us":36448,"spinlock_wait_cycles":514599040,"thread_start_us":494401,"threads_started":1,"wal-append.queue_time_us":731612}
I20260709 12:23:10.402241 17689 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37402 (request call id 2858) took 1653 ms. Trace:
I20260709 12:23:10.402691 17689 rpcz_store.cc:276] 0709 12:23:08.749274 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:10.400441 (+1651167us) service_pool.cc:224] Handling call
0709 12:23:10.400922 (+   481us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:10.401946 (+  1024us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:10.402166 (+   220us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:23:10.007115 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 80020.8us
I20260709 12:23:10.000494 17756 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38786 (request call id 3110) took 3010 ms. Trace:
I20260709 12:23:10.012334 17759 raft_consensus.cc:2468] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 11.
I20260709 12:23:10.110340 17761 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36572 (request call id 942) took 2078 ms. Trace:
I20260709 12:23:10.008591 17762 raft_consensus.cc:2468] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 17.
I20260709 12:23:10.136576 17834 raft_consensus.cc:1240] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 15. Current term is 16. Ops: [15.10-15.10]
I20260709 12:23:10.143157 17836 raft_consensus.cc:3060] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:10.117347 18334 leader_election.cc:304] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:23:10.189193 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:10.180469 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:10.159137 18367 raft_consensus.cc:2393] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 12: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:23:10.120172 22652 leader_election.cc:290] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:10.169430 23056 consensus_queue.cc:1243] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer 0da169ff82674dc1947055ce43de68a3 log is divergent from this leader: its last log entry 16.9 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 8
W20260709 12:23:10.382958 17760 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.292s	user 0.006s	sys 0.008s
W20260709 12:23:10.387353 17729 connection.cc:256] RPC call timeout handler was delayed by -1.0436697799996182s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.163s for a response
I20260709 12:23:10.389549 17755 raft_consensus.cc:2468] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 11.
I20260709 12:23:10.398581 17830 raft_consensus.cc:2393] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 12: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:23:10.225984 23148 raft_consensus.cc:515] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:10.407181 22915 consensus_queue.cc:1048] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
W20260709 12:23:10.227905 23087 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.968s	user 0.024s	sys 0.100s
I20260709 12:23:10.652663 17685 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37396 (request call id 2493) took 1907 ms. Trace:
I20260709 12:23:10.656843 17685 rpcz_store.cc:276] 0709 12:23:08.745723 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:09.973896 (+1228173us) service_pool.cc:224] Handling call
0709 12:23:10.652380 (+678484us) spinlock_profiling.cc:243] Waited 350 ms on lock 0x7b54003016d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:23:10.652592 (+   212us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":805011840}
W20260709 12:23:10.651605 17683 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37396 (request call id 2495) took 1803 ms (client timeout 2348 ms). Trace:
W20260709 12:23:10.622635 23297 consensus_meta.cc:287] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.694s	user 0.005s	sys 0.007s
I20260709 12:23:10.659999 17686 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37396 (request call id 2479) took 2273 ms. Trace:
I20260709 12:23:10.653868 18364 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 1711) took 1883 ms. Trace:
I20260709 12:23:10.638546 18374 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 1712) took 1732 ms. Trace:
I20260709 12:23:10.660785 18364 rpcz_store.cc:276] 0709 12:23:08.770099 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:08.770229 (+   130us) service_pool.cc:224] Handling call
0709 12:23:08.889125 (+118896us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:10.112999 (+1223874us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:10.113044 (+    45us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:10.113102 (+    58us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:23:10.113157 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:10.653307 (+540150us) spinlock_profiling.cc:243] Waited 1.22 s on lock 0x7b54007e7cd4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:23:10.653386 (+    79us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:10.653676 (+   290us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2814050048}
I20260709 12:23:10.661484 18374 rpcz_store.cc:276] 0709 12:23:08.905671 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:08.905826 (+   155us) service_pool.cc:224] Handling call
0709 12:23:09.110949 (+205123us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:10.035324 (+924375us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:10.035374 (+    50us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:10.035452 (+    78us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:23:10.035497 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:10.638013 (+602516us) spinlock_profiling.cc:243] Waited 924 ms on lock 0x7b54006af9d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:23:10.638086 (+    73us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:10.638346 (+   260us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2125133184}
W20260709 12:23:10.659293 17683 rpcz_store.cc:269] 0709 12:23:08.848573 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:09.879774 (+1031201us) service_pool.cc:224] Handling call
0709 12:23:10.651219 (+771445us) spinlock_profiling.cc:243] Waited 323 ms on lock 0x7b5400380254. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:23:10.651476 (+   257us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":743005568}
I20260709 12:23:10.633883 17837 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35042 (request call id 2424) took 1939 ms. Trace:
I20260709 12:23:10.665287 23108 raft_consensus.cc:493] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:10.664539 17584 catalog_manager.cc:5697] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb reported cstate change: term changed from 5 to 7. New cstate: current_term: 7 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } }
I20260709 12:23:10.666222 23108 raft_consensus.cc:515] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:10.665042 17837 rpcz_store.cc:276] 0709 12:23:08.694455 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:09.932314 (+1237859us) service_pool.cc:224] Handling call
0709 12:23:10.633314 (+701000us) spinlock_profiling.cc:243] Waited 124 ms on lock 0x7b54003340d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:23:10.633618 (+   304us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":286034176}
I20260709 12:23:10.650643 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 14 candidate_status { last_received { term: 12 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:10.408622 23139 leader_election.cc:290] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 12 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:10.537257 17682 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44898 (request call id 1055) took 2436 ms. Trace:
W20260709 12:23:10.436825 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 47168.5us
I20260709 12:23:10.399799 17688 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37396 (request call id 2492) took 1718 ms. Trace:
I20260709 12:23:10.548400 18362 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 1710) took 1871 ms. Trace:
W20260709 12:23:10.680696 17760 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.603s	user 0.015s	sys 0.010s
W20260709 12:23:10.545889 17833 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35042 (request call id 2420) took 1860 ms (client timeout 1757 ms). Trace:
W20260709 12:23:10.550012 18371 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59886 (request call id 1551) took 2031 ms (client timeout 1757 ms). Trace:
I20260709 12:23:10.672659 17688 rpcz_store.cc:276] 0709 12:23:08.681664 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:09.976824 (+1295160us) service_pool.cc:224] Handling call
0709 12:23:10.399622 (+422798us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:10.547032 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 11 candidate_status { last_received { term: 9 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:10.671118 17682 rpcz_store.cc:276] 0709 12:23:08.100690 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:08.100842 (+   152us) service_pool.cc:224] Handling call
0709 12:23:08.101138 (+   296us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:10.337941 (+2236803us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:10.536744 (+198803us) spinlock_profiling.cc:243] Waited 2.22 s on lock 0x7b54002526d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:23:10.537135 (+   391us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5111922048}
W20260709 12:23:10.432791 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 468965us
I20260709 12:23:10.447009 23174 leader_election.cc:290] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:10.633466 23120 leader_election.cc:290] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:10.673875 18362 rpcz_store.cc:276] 0709 12:23:08.676905 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:08.708169 (+ 31264us) service_pool.cc:224] Handling call
0709 12:23:08.768885 (+ 60716us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:08.849860 (+ 80975us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:08.849891 (+    31us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:08.849934 (+    43us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:23:08.849965 (+    31us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:10.547603 (+1697638us) spinlock_profiling.cc:243] Waited 80.7 ms on lock 0x7b54006c6954. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:23:10.547692 (+    89us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:10.548042 (+   350us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":185683712}
I20260709 12:23:10.437502 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:10.441998 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:23:10.692442 17833 rpcz_store.cc:269] 0709 12:23:08.684906 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:08.685021 (+   115us) service_pool.cc:224] Handling call
0709 12:23:10.545321 (+1860300us) spinlock_profiling.cc:243] Waited 130 ms on lock 0x7b540022ae54. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:23:10.545593 (+   272us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":298012544}
W20260709 12:23:10.691725 18371 rpcz_store.cc:269] 0709 12:23:08.518928 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:08.520710 (+  1782us) service_pool.cc:224] Handling call
0709 12:23:10.549492 (+2028782us) spinlock_profiling.cc:243] Waited 1.39 s on lock 0x7b54006e23d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:23:10.549735 (+   243us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3193194752}
I20260709 12:23:10.700376 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:10.434162 23220 leader_election.cc:290] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:10.414981 23263 leader_election.cc:290] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 13 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:10.716136 18361 raft_consensus.cc:2468] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 10.
I20260709 12:23:10.424445 23224 leader_election.cc:290] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:10.415830 23124 leader_election.cc:290] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:10.719261 23179 leader_election.cc:290] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:10.720366 23244 raft_consensus.cc:3060] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:10.716030 17761 rpcz_store.cc:276] 0709 12:23:08.031922 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:08.032078 (+   156us) service_pool.cc:224] Handling call
0709 12:23:08.032719 (+   641us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:08.033095 (+   376us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:08.033133 (+    38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:08.033705 (+   572us) log.cc:844] Serialized 26 byte log entry
0709 12:23:08.465466 (+431761us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:23:08.638541 (+173075us) log.cc:844] Serialized 14 byte log entry
0709 12:23:08.725502 (+ 86961us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:08.725584 (+    82us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:10.068850 (+1343266us) raft_consensus.cc:1685] finished
0709 12:23:10.068879 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:10.110243 (+ 41364us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":383656,"prepare.run_cpu_time_us":125,"prepare.run_wall_time_us":955,"raft.queue_time_us":603935,"raft.run_cpu_time_us":98,"raft.run_wall_time_us":98,"thread_start_us":541705,"threads_started":3,"wal-append.queue_time_us":935991}
I20260709 12:23:10.716742 17831 raft_consensus.cc:2468] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 16.
I20260709 12:23:10.722442 17756 rpcz_store.cc:276] 0709 12:23:06.990009 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:06.990451 (+   442us) service_pool.cc:224] Handling call
0709 12:23:06.990769 (+   318us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:07.000768 (+  9999us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:07.000815 (+    47us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:07.009254 (+  8439us) log.cc:844] Serialized 26 byte log entry
0709 12:23:07.380843 (+371589us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:23:07.380901 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:07.380972 (+    71us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:09.995677 (+2614705us) raft_consensus.cc:1685] finished
0709 12:23:09.995710 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:10.000426 (+  4716us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":3190,"raft.run_cpu_time_us":122,"raft.run_wall_time_us":3263,"thread_start_us":298250,"threads_started":1,"wal-append.queue_time_us":820433}
W20260709 12:23:10.725422 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 307882us
I20260709 12:23:10.660511 17686 rpcz_store.cc:276] 0709 12:23:08.386818 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:08.518051 (+131233us) service_pool.cc:224] Handling call
0709 12:23:08.518398 (+   347us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:23:09.942692 (+1424294us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:09.942724 (+    32us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:23:09.957183 (+ 14459us) log.cc:844] Serialized 52 byte log entry
0709 12:23:10.385036 (+427853us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:23:10.385145 (+   109us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:10.628970 (+243825us) spinlock_profiling.cc:243] Waited 1.42 s on lock 0x7b54003480d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:23:10.629131 (+   161us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:10.629202 (+    71us) raft_consensus.cc:1685] finished
0709 12:23:10.629215 (+    13us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:10.659780 (+ 30565us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":41232,"raft.run_cpu_time_us":118,"raft.run_wall_time_us":118,"spinlock_wait_cycles":3274625920,"thread_start_us":388482,"threads_started":1,"wal-append.queue_time_us":407904}
I20260709 12:23:10.531388 23206 leader_election.cc:290] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 11 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:10.722025 22853 leader_election.cc:290] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:10.728544 17727 leader_election.cc:304] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
W20260709 12:23:10.732372 17758 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.678s	user 0.006s	sys 0.005s
W20260709 12:23:10.448823 17728 leader_election.cc:336] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Timed out: RequestConsensusVote RPC to 127.16.29.65:33071 timed out after 1.633s (SENT)
W20260709 12:23:10.733613 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 284961us
W20260709 12:23:10.717711 18361 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59918 (request call id 1714) took 1468 ms (client timeout 1516 ms). Trace:
W20260709 12:23:10.734436 17728 leader_election.cc:336] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 14 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Timed out: RequestConsensusVote RPC to 127.16.29.65:33071 timed out after 1.653s (SENT)
W20260709 12:23:10.734862 18361 rpcz_store.cc:269] 0709 12:23:09.249825 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:09.250461 (+   636us) service_pool.cc:224] Handling call
0709 12:23:10.546523 (+1296062us) spinlock_profiling.cc:243] Waited 8.06 ms on lock 0x7b5800102398. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b474fbb71 00007f7b474fba22 00007f7b504148d9 00007f7b4f56eee0 00007f7b4f6380ab 00007f7b4f638131 00007f7b4f5e60a0 00007f7b4f5e5984 00007f7b48b606d4 00007f7b48b60662
0709 12:23:10.717627 (+171104us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":18539264}
I20260709 12:23:10.436244 23293 raft_consensus.cc:493] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:10.736624 23293 raft_consensus.cc:515] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:10.738976 23251 leader_election.cc:290] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:10.240918 23168 leader_election.cc:290] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:10.739835 18374 raft_consensus.cc:2468] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 14.
I20260709 12:23:10.741942 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:10.743120 18362 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 12.
I20260709 12:23:10.521179 23217 consensus_queue.cc:237] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:10.748802 23139 raft_consensus.cc:493] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
W20260709 12:23:10.749282 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:10.749647 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 17 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:10.751070 17681 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37402 (request call id 2853) took 2086 ms. Trace:
I20260709 12:23:10.751581 17681 rpcz_store.cc:276] 0709 12:23:08.664728 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:10.334450 (+1669722us) service_pool.cc:224] Handling call
0709 12:23:10.334963 (+   513us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:10.335510 (+   547us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 12:23:10.383654 (+ 48144us) log.cc:844] Serialized 14 byte log entry
0709 12:23:10.750396 (+366742us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:10.750465 (+    69us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:23:10.750514 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:10.750585 (+    71us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:10.750745 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":75027,"prepare.queue_time_us":47954,"prepare.run_cpu_time_us":109,"prepare.run_wall_time_us":109,"raft.queue_time_us":77001,"raft.run_cpu_time_us":134,"raft.run_wall_time_us":135,"thread_start_us":333386,"threads_started":2}
I20260709 12:23:10.757118 23113 leader_election.cc:290] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:10.759568 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 12 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:10.436406 23114 leader_election.cc:290] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 16 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:10.435240 23266 raft_consensus.cc:493] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:23:10.702947 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:23:10.715508 17801 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:10.747445 17760 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38786 (request call id 3120) took 2230 ms. Trace:
I20260709 12:23:10.776860 23148 leader_election.cc:290] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:10.780491 23164 leader_election.cc:290] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:10.414772 23246 consensus_queue.cc:260] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 13.8, Last appended by leader: 8, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
W20260709 12:23:10.784291 17803 connection.cc:256] RPC call timeout handler was delayed by -2.5039199910002026s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1811s for a response
W20260709 12:23:10.784991 17803 connection.cc:256] RPC call timeout handler was delayed by -2.0863199909999821s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2275s for a response
W20260709 12:23:10.786212 17803 connection.cc:256] RPC call timeout handler was delayed by -1.9460324279998531s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.32220000000000004s for a response
W20260709 12:23:10.785307 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 57364.8us
I20260709 12:23:10.714018 17832 raft_consensus.cc:2468] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 13.
I20260709 12:23:10.796912 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:23:10.807049 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 75245.8us
I20260709 12:23:10.810115 18365 raft_consensus.cc:2410] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for term 17 because replica has last-logged OpId of term: 14 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 10.
I20260709 12:23:10.801438 18363 raft_consensus.cc:1275] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 13 index: 11. (index mismatch)
I20260709 12:23:10.660622 23297 raft_consensus.cc:515] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:10.804816 23266 raft_consensus.cc:515] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:10.857477 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:10.849304 18364 raft_consensus.cc:2468] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 14.
W20260709 12:23:10.860895 17759 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38786 (request call id 3125) took 2215 ms (client timeout 1836 ms). Trace:
W20260709 12:23:10.861825 17759 rpcz_store.cc:269] 0709 12:23:08.636045 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:08.636267 (+   222us) service_pool.cc:224] Handling call
0709 12:23:10.817742 (+2181475us) spinlock_profiling.cc:243] Waited 250 ms on lock 0x7b54000fdcd4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:23:10.851509 (+ 33767us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":574971648}
W20260709 12:23:10.863925 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 122124us
I20260709 12:23:10.865337 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:10.866618 17729 leader_election.cc:304] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:10.869752 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:10.552188 23257 consensus_queue.cc:1048] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.290s
I20260709 12:23:10.845062 23293 leader_election.cc:290] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:10.774905 17760 rpcz_store.cc:276] 0709 12:23:08.516636 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:08.516762 (+   126us) service_pool.cc:224] Handling call
0709 12:23:08.517077 (+   315us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:08.517294 (+   217us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 12:23:08.659706 (+142412us) log.cc:844] Serialized 14 byte log entry
0709 12:23:10.746670 (+2086964us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:10.746818 (+   148us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:23:10.746943 (+   125us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:10.747071 (+   128us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:10.747348 (+   277us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":185001,"prepare.queue_time_us":382234,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":44,"raft.queue_time_us":347966,"raft.run_cpu_time_us":203,"raft.run_wall_time_us":201,"thread_start_us":1963069,"threads_started":3,"wal-append.queue_time_us":1478429}
I20260709 12:23:10.897718 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:10.899296 17686 raft_consensus.cc:2468] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 12.
I20260709 12:23:10.844154 23179 raft_consensus.cc:2749] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:23:10.914636 17834 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35042 (request call id 2417) took 2159 ms. Trace:
I20260709 12:23:10.905260 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 17 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:10.916236 17834 rpcz_store.cc:276] 0709 12:23:08.683591 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:08.683781 (+   190us) service_pool.cc:224] Handling call
0709 12:23:08.725803 (+ 42022us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:10.788451 (+2062648us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:10.791463 (+  3012us) spinlock_profiling.cc:243] Waited 1.41 s on lock 0x7b54001581d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:23:10.842896 (+ 51433us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3238822144}
I20260709 12:23:10.925769 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 17 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:10.914666 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:10.934391 17761 raft_consensus.cc:2393] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 16: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:23:10.926782 17837 raft_consensus.cc:2468] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 16.
I20260709 12:23:10.890323 23220 raft_consensus.cc:493] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:10.953449 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 13 candidate_status { last_received { term: 12 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:10.965679 18371 raft_consensus.cc:1240] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 11. Current term is 12. Ops: []
I20260709 12:23:10.968340 17689 raft_consensus.cc:2410] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 17 because replica has last-logged OpId of term: 15 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 9.
W20260709 12:23:10.972026 17652 connection.cc:256] RPC call timeout handler was delayed by -1.508344502s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1501s for a response
W20260709 12:23:10.972609 17652 connection.cc:256] RPC call timeout handler was delayed by -1.4948445019999781s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1516s for a response
I20260709 12:23:10.921489 23139 raft_consensus.cc:515] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:10.973527 17652 connection.cc:256] RPC call timeout handler was delayed by -1.3391445019997263s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16890000000000002s for a response
W20260709 12:23:10.974077 17652 connection.cc:256] RPC call timeout handler was delayed by -1.0547445019997213s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2005s for a response
W20260709 12:23:10.974450 17652 connection.cc:256] RPC call timeout handler was delayed by -1.0259445020001294s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2037s for a response
I20260709 12:23:10.972728 23267 raft_consensus.cc:2804] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:23:10.975628 23267 raft_consensus.cc:493] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:10.976240 23267 raft_consensus.cc:3060] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:10.978207 23297 leader_election.cc:290] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 11 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:23:10.978391 23227 consensus_meta.cc:287] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.791s	user 0.011s	sys 0.003s
I20260709 12:23:10.979024 23227 raft_consensus.cc:515] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:10.994359 18371 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 1720) took 1736 ms. Trace:
I20260709 12:23:10.994982 18371 rpcz_store.cc:276] 0709 12:23:09.261276 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:10.813349 (+1552073us) service_pool.cc:224] Handling call
0709 12:23:10.906134 (+ 92785us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:10.966255 (+ 60121us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:10.988211 (+ 21956us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:10.967760 17759 raft_consensus.cc:2468] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 12.
I20260709 12:23:10.993608 17758 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38786 (request call id 3127) took 1095 ms. Trace:
W20260709 12:23:10.999698 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 69841.3us
I20260709 12:23:10.975715 23139 leader_election.cc:290] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:23:10.999198 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 58741us
I20260709 12:23:11.016484 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:11.018420 17757 raft_consensus.cc:2468] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 16.
I20260709 12:23:11.019408 23108 leader_election.cc:290] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:11.019940 23346 consensus_queue.cc:1059] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 11.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.147s
W20260709 12:23:11.020850 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 881982us
I20260709 12:23:11.021454 23168 raft_consensus.cc:2749] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:23:11.024477 18362 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 1718) took 1773 ms. Trace:
I20260709 12:23:11.024940 18362 rpcz_store.cc:276] 0709 12:23:09.251248 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:10.806550 (+1555302us) service_pool.cc:224] Handling call
0709 12:23:10.911221 (+104671us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:11.024004 (+112783us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:11.024056 (+    52us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:11.024125 (+    69us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:23:11.024185 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:11.024263 (+    78us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:11.024399 (+   136us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:11.027273 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:11.026660 23088 consensus_queue.cc:1048] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.025s
I20260709 12:23:11.036073 23220 raft_consensus.cc:515] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:11.036293 23267 raft_consensus.cc:515] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:11.056100 18333 leader_election.cc:304] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:23:11.062685 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:11.065073 23267 leader_election.cc:290] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 14 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:11.067065 23267 raft_consensus.cc:2804] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:23:11.050585 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 15 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:11.068696 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:11.069489 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:11.070433 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 17 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:11.067793 23267 raft_consensus.cc:493] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:10.998034 17758 rpcz_store.cc:276] 0709 12:23:09.891204 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:09.979048 (+ 87844us) service_pool.cc:224] Handling call
0709 12:23:09.979507 (+   459us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:09.979730 (+   223us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 12:23:09.997158 (+ 17428us) log.cc:844] Serialized 14 byte log entry
0709 12:23:10.840016 (+842858us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:10.874471 (+ 34455us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:23:10.895362 (+ 20891us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:10.911722 (+ 16360us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:10.986771 (+ 75049us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":73509,"prepare.queue_time_us":207513,"prepare.run_cpu_time_us":571,"prepare.run_wall_time_us":10651,"raft.queue_time_us":117786,"raft.run_cpu_time_us":716,"raft.run_wall_time_us":79605,"thread_start_us":735687,"threads_started":1}
I20260709 12:23:11.071621 17757 raft_consensus.cc:2468] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 16.
I20260709 12:23:11.065169 18333 leader_election.cc:304] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
W20260709 12:23:11.092849 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 81197.4us
W20260709 12:23:11.067703 17830 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37994 (request call id 1059) took 2766 ms (client timeout 2914 ms). Trace:
W20260709 12:23:11.110656 17830 rpcz_store.cc:269] 0709 12:23:08.301678 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:08.301823 (+   145us) service_pool.cc:224] Handling call
0709 12:23:11.067129 (+2765306us) spinlock_profiling.cc:243] Waited 1.93 s on lock 0x7b54000aafd4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:23:11.067516 (+   387us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4429111680}
I20260709 12:23:11.044307 23246 raft_consensus.cc:3060] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:11.121925 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:23:11.082528 17680 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.526s	user 0.007s	sys 0.012s
W20260709 12:23:11.082391 17684 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.738s	user 0.010s	sys 0.011s
W20260709 12:23:11.130715 17684 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.787s	user 0.010s	sys 0.011s
I20260709 12:23:11.076406 23267 raft_consensus.cc:3060] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:11.137513 23139 raft_consensus.cc:493] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:11.137966 18368 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59886 (request call id 1557) took 2607 ms. Trace:
I20260709 12:23:11.059458 23266 leader_election.cc:290] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:23:11.165522 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 87662.6us
W20260709 12:23:11.159075 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 80341.2us
W20260709 12:23:11.156044 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 91113us
W20260709 12:23:11.163578 17680 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.610s	user 0.008s	sys 0.012s
W20260709 12:23:11.184543 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 44205.4us
I20260709 12:23:11.142933 18368 rpcz_store.cc:276] 0709 12:23:08.530416 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:08.604805 (+ 74389us) service_pool.cc:224] Handling call
0709 12:23:08.605172 (+   367us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:08.605427 (+   255us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 12:23:10.024776 (+1419349us) log.cc:844] Serialized 14 byte log entry
0709 12:23:10.631905 (+607129us) spinlock_profiling.cc:243] Waited 14.8 ms on lock 0x7b18003e0500. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b47515e07 00007f7b503709a9 00007f7b4753269b 00007f7b4753093f 00007f7b47532c02 00007f7b4a1ba20b 00007f7b4a1bfd23 00007f7b4a1c023a 00007f7b4a62aa2a
0709 12:23:11.136631 (+504726us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:11.136724 (+    93us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:23:11.137137 (+   413us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:11.137244 (+   107us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:11.137750 (+   506us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":1406468,"prepare.run_cpu_time_us":62,"prepare.run_wall_time_us":2904,"raft.queue_time_us":171,"raft.run_cpu_time_us":142,"raft.run_wall_time_us":144,"spinlock_wait_cycles":34049920,"thread_start_us":1868363,"threads_started":2}
I20260709 12:23:11.076336 23118 consensus_queue.cc:1059] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.101s
W20260709 12:23:11.197041 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 330724us
W20260709 12:23:11.208134 17755 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38786 (request call id 3114) took 3768 ms (client timeout 2149 ms). Trace:
I20260709 12:23:11.217363 17835 raft_consensus.cc:1240] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 15 LEADER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 14. Current term is 15. Ops: []
I20260709 12:23:11.213284 17802 leader_election.cc:304] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:23:11.240423 17760 raft_consensus.cc:2393] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 13: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
W20260709 12:23:11.221040 17755 rpcz_store.cc:269] 0709 12:23:07.439530 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:07.439975 (+   445us) service_pool.cc:224] Handling call
0709 12:23:11.182474 (+3742499us) spinlock_profiling.cc:243] Waited 2.93 s on lock 0x7b54000fbed4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:23:11.208050 (+ 25576us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":6737868672}
I20260709 12:23:11.217440 23118 raft_consensus.cc:3055] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:23:11.251721 23118 raft_consensus.cc:740] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:23:11.251457 23251 consensus_queue.cc:1059] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:23:11.252843 23251 raft_consensus.cc:3055] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:23:11.253432 23251 raft_consensus.cc:740] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:23:11.253281 23118 consensus_queue.cc:260] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.13, Last appended by leader: 13, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:11.254175 17729 leader_election.cc:304] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:23:11.255335 23118 raft_consensus.cc:3060] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:11.254932 23251 consensus_queue.cc:260] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.12, Last appended by leader: 12, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:11.255965 17652 leader_election.cc:304] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:11.256745 23251 raft_consensus.cc:3060] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:11.257866 17652 leader_election.cc:304] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:23:11.259979 23139 raft_consensus.cc:515] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:11.262432 23139 leader_election.cc:290] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:23:11.276137 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 294119us
I20260709 12:23:11.277359 17654 leader_election.cc:304] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b
W20260709 12:23:11.284220 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:11.282459 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 63880.9us
I20260709 12:23:11.286790 17654 leader_election.cc:304] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:23:11.300113 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 84916.5us
I20260709 12:23:11.305387 18333 leader_election.cc:304] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:23:11.306164 23047 consensus_queue.cc:260] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.9, Last appended by leader: 9, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:11.307276 23108 raft_consensus.cc:2804] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:23:11.307901 23108 raft_consensus.cc:493] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
W20260709 12:23:11.308138 17762 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38786 (request call id 3119) took 2792 ms (client timeout 2860 ms). Trace:
I20260709 12:23:11.308317 23108 raft_consensus.cc:3060] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Advancing to term 14
W20260709 12:23:11.308620 17762 rpcz_store.cc:269] 0709 12:23:08.515551 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:08.515691 (+   140us) service_pool.cc:224] Handling call
0709 12:23:11.241464 (+2725773us) spinlock_profiling.cc:243] Waited 88.6 ms on lock 0x7b54001b8354. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:23:11.307977 (+ 66513us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":203782528}
I20260709 12:23:11.311069 17680 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37402 (request call id 2857) took 2595 ms. Trace:
I20260709 12:23:11.311491 17680 rpcz_store.cc:276] 0709 12:23:08.715268 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:10.341688 (+1626420us) service_pool.cc:224] Handling call
0709 12:23:10.342417 (+   729us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:10.343230 (+   813us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:10.343272 (+    42us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:10.398114 (+ 54842us) log.cc:844] Serialized 26 byte log entry
0709 12:23:11.188747 (+790633us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:23:11.188866 (+   119us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:11.189003 (+   137us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:11.310817 (+121814us) raft_consensus.cc:1685] finished
0709 12:23:11.310836 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:11.310969 (+   133us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":91805,"raft.run_cpu_time_us":140,"raft.run_wall_time_us":140,"thread_start_us":634878,"threads_started":1,"wal-append.queue_time_us":755262}
I20260709 12:23:11.315968 23121 raft_consensus.cc:493] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:23:11.317121 23121 raft_consensus.cc:515] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:11.324182 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 246037us
W20260709 12:23:11.326520 17583 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:37172 (request call id 255) took 35829 ms (client timeout 15000 ms). Trace:
I20260709 12:23:11.329165 23220 leader_election.cc:290] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:11.330199 23047 raft_consensus.cc:3060] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:11.332062 23266 raft_consensus.cc:2804] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:23:11.332520 23266 raft_consensus.cc:493] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:11.332911 23266 raft_consensus.cc:3060] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:11.281869 23120 raft_consensus.cc:2804] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:23:11.336304 23120 raft_consensus.cc:493] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:23:11.336794 23120 raft_consensus.cc:3060] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:11.298969 17652 leader_election.cc:304] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:23:11.298811 23297 raft_consensus.cc:2804] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Leader pre-election won for term 13
W20260709 12:23:11.398265 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:11.388477 23148 raft_consensus.cc:2749] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:23:11.390237 17685 raft_consensus.cc:2468] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 13.
I20260709 12:23:11.392102 23217 consensus_queue.cc:1048] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
W20260709 12:23:11.393503 17764 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.760s	user 0.005s	sys 0.009s
W20260709 12:23:11.403167 17764 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.986s	user 0.006s	sys 0.009s
W20260709 12:23:11.400208 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 112165us
W20260709 12:23:11.388556 23087 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.676s	user 0.007s	sys 0.011s
W20260709 12:23:11.416999 17836 consensus_meta.cc:287] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.721s	user 0.013s	sys 0.004s
I20260709 12:23:11.375723 23139 raft_consensus.cc:2749] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:23:11.397529 17801 leader_election.cc:304] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:11.417431 17836 raft_consensus.cc:2468] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 13.
I20260709 12:23:11.397970 23297 raft_consensus.cc:493] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:23:11.417021 23087 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.748s	user 0.008s	sys 0.011s
I20260709 12:23:11.409554 22853 raft_consensus.cc:2804] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:23:11.401582 23227 leader_election.cc:290] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 12 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:11.378696 23088 consensus_queue.cc:1048] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.052s
I20260709 12:23:11.409926 23108 raft_consensus.cc:515] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:11.434499 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 154658us
I20260709 12:23:11.484668 23087 raft_consensus.cc:2749] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:23:11.485025 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:23:11.457909 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:11.434333 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 13 candidate_status { last_received { term: 12 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:23:11.491917 17682 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.561s	user 0.024s	sys 0.082s
W20260709 12:23:11.492321 17682 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.564s	user 0.024s	sys 0.082s
I20260709 12:23:11.486399 18374 raft_consensus.cc:2468] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 12.
I20260709 12:23:11.485294 17654 leader_election.cc:304] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:11.429881 23224 raft_consensus.cc:2804] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:23:11.498225 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:11.427503 23297 raft_consensus.cc:3060] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:11.498139 23224 raft_consensus.cc:493] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:11.501639 17764 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38786 (request call id 3128) took 1588 ms. Trace:
I20260709 12:23:11.436194 22853 raft_consensus.cc:493] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:11.511826 17681 raft_consensus.cc:1275] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 14 index: 8. (term mismatch)
I20260709 12:23:11.456350 23251 raft_consensus.cc:493] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:23:11.516860 18368 raft_consensus.cc:2393] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 14: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:23:11.520326 18361 raft_consensus.cc:2393] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 13: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:23:11.516788 22853 raft_consensus.cc:3060] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:11.432391 23168 raft_consensus.cc:2804] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:23:11.515066 17681 pending_rounds.cc:85] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3: Aborting all ops after (but not including) 7
I20260709 12:23:11.509816 17764 rpcz_store.cc:276] 0709 12:23:09.905499 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:10.059123 (+153624us) service_pool.cc:224] Handling call
0709 12:23:10.059435 (+   312us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:10.059641 (+   206us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 12:23:10.416404 (+356763us) log.cc:844] Serialized 14 byte log entry
0709 12:23:11.422990 (+1006586us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:11.431809 (+  8819us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:23:11.448952 (+ 17143us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:11.466272 (+ 17320us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:11.493838 (+ 27566us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":170603,"prepare.run_cpu_time_us":48,"prepare.run_wall_time_us":49,"raft.queue_time_us":6525,"raft.run_cpu_time_us":150,"raft.run_wall_time_us":4723,"thread_start_us":1291114,"threads_started":2,"wal-append.queue_time_us":863516}
I20260709 12:23:11.529966 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:11.531247 18374 raft_consensus.cc:2410] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 17 because replica has last-logged OpId of term: 16 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 12.
I20260709 12:23:11.512038 23224 raft_consensus.cc:3060] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Advancing to term 14
W20260709 12:23:11.521270 23244 consensus_meta.cc:287] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.800s	user 0.012s	sys 0.005s
W20260709 12:23:11.539120 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 318868us
I20260709 12:23:11.528877 17681 pending_rounds.cc:107] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3: Aborting uncommitted NO_OP operation due to leader change: 13.8
I20260709 12:23:11.541662 17681 raft_consensus.cc:2889] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:23:11.539983 17727 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.622s	user 0.016s	sys 0.027s
W20260709 12:23:11.543856 17727 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.633s	user 0.016s	sys 0.028s
W20260709 12:23:11.544356 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 672536us
I20260709 12:23:11.553690 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:11.554662 18374 raft_consensus.cc:2410] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 16 because replica has last-logged OpId of term: 15 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 11.
I20260709 12:23:11.556677 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:23:11.327124 17583 rpcz_store.cc:269] 0709 12:22:35.490653 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:35.490906 (+   253us) service_pool.cc:224] Handling call
0709 12:23:11.319555 (+35828649us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:22:51.780146 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:22:51.964275 (+184129us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:22:51.980562 (+ 16287us) write_op.cc:435] Acquired schema lock
0709 12:22:51.988373 (+  7811us) tablet.cc:662] Decoding operations
0709 12:22:52.046103 (+ 57730us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:22:52.046226 (+   123us) write_op.cc:641] Partition lock acquired for write op
0709 12:22:52.046260 (+    34us) tablet.cc:685] Acquiring locks for 21 operations
0709 12:22:52.047759 (+  1499us) tablet.cc:701] Row locks acquired
0709 12:22:52.047807 (+    48us) write_op.cc:260] PREPARE: finished
0709 12:22:52.082692 (+ 34885us) write_op.cc:270] Start()
0709 12:22:52.132804 (+ 50112us) write_op.cc:276] Timestamp: P: 1783599772065063 usec, L: 0
0709 12:22:52.137371 (+  4567us) op_driver.cc:348] REPLICATION: starting
0709 12:22:52.368559 (+231188us) log.cc:844] Serialized 8192 byte log entry
0709 12:22:53.077419 (+708860us) spinlock_profiling.cc:243] Waited 9.81 ms on lock 0x7b4400022b4c. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b47459c85 00007f7b4745a6a2 00007f7b4a1cc1db 00007f7b4a1c7aaf 00007f7b475160ac 00007f7b503709a9 00007f7b4753269b 00007f7b4753093f 00007f7b47532c02 00007f7b4a1ba20b
0709 12:22:53.444099 (+366680us) op_driver.cc:464] REPLICATION: finished
0709 12:22:53.572767 (+128668us) write_op.cc:301] APPLY: starting
0709 12:22:53.604866 (+ 32099us) tablet.cc:1367] starting BulkCheckPresence
0709 12:22:54.779007 (+1174141us) tablet.cc:1370] finished BulkCheckPresence
0709 12:22:54.779076 (+    69us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:22:55.417391 (+638315us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:22:55.417768 (+   377us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=42,key_file_lookups=42,delta_file_lookups=42,mrs_lookups=0
0709 12:22:55.417812 (+    44us) write_op.cc:312] APPLY: finished
0709 12:22:55.418302 (+   490us) log.cc:844] Serialized 188 byte log entry
0709 12:22:55.713221 (+294919us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:22:55.795504 (+ 82283us) write_op.cc:454] Released schema lock
0709 12:22:55.842605 (+ 47101us) write_op.cc:341] FINISH: Updating metrics
0709 12:22:55.986519 (+143914us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:22:55.992708 (+  6189us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":4216374,"reported_tablets":44,"child_traces":[["op",{"apply.queue_time_us":80914,"apply.run_cpu_time_us":47424,"apply.run_wall_time_us":2331455,"cfile_cache_hit":86,"cfile_cache_hit_bytes":91991,"num_ops":21,"prepare.queue_time_us":177493,"prepare.run_cpu_time_us":19437,"prepare.run_wall_time_us":1567053,"raft.queue_time_us":282896,"raft.run_cpu_time_us":6597,"raft.run_wall_time_us":74509,"replication_time_us":1283313,"spinlock_wait_cycles":22565760,"thread_start_us":1146388,"threads_started":5,"wal-append.queue_time_us":1161164,"wal-append.run_cpu_time_us":22874,"wal-append.run_wall_time_us":2559201}]]}
I20260709 12:23:11.526248 23168 raft_consensus.cc:493] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
W20260709 12:23:11.559994 17584 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.575s	user 0.017s	sys 0.019s
I20260709 12:23:11.563741 23220 raft_consensus.cc:493] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
W20260709 12:23:11.565573 23356 log.cc:927] Time spent T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.073s	user 0.000s	sys 0.000s
I20260709 12:23:11.534317 23251 raft_consensus.cc:515] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:11.564756 17584 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.580s	user 0.018s	sys 0.019s
W20260709 12:23:11.561458 23114 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.580s	user 0.010s	sys 0.028s
I20260709 12:23:11.576144 23305 raft_consensus.cc:493] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:11.578254 23201 raft_consensus.cc:2749] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
W20260709 12:23:11.579180 18334 connection.cc:256] RPC call timeout handler was delayed by -1.2994936700001745s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1791s for a response
I20260709 12:23:11.566547 17729 leader_election.cc:304] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:11.587042 18361 raft_consensus.cc:2468] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 11.
I20260709 12:23:11.590577 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:23:11.585121 23114 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.603s	user 0.010s	sys 0.028s
I20260709 12:23:11.564781 23168 raft_consensus.cc:3060] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:23:11.585731 23293 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.592s	user 0.013s	sys 0.018s
I20260709 12:23:11.599182 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 12 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:11.565677 23220 raft_consensus.cc:515] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:11.600545 17803 leader_election.cc:336] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Timed out: RequestConsensusVote RPC to 127.16.29.68:42577 timed out after 1.593s (SENT)
W20260709 12:23:11.600319 23293 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.609s	user 0.014s	sys 0.018s
I20260709 12:23:11.632043 17687 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37402 (request call id 2862) took 1656 ms. Trace:
I20260709 12:23:11.633119 17727 leader_election.cc:304] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:11.628877 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:11.636109 17687 rpcz_store.cc:276] 0709 12:23:09.975526 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:10.412396 (+436870us) service_pool.cc:224] Handling call
0709 12:23:10.451056 (+ 38660us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:10.922229 (+471173us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:10.922264 (+    35us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:10.922383 (+   119us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:23:10.922430 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:11.631641 (+709211us) spinlock_profiling.cc:243] Waited 454 ms on lock 0x7b5400348ad4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:23:11.631685 (+    44us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:11.631830 (+   145us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1044202752}
I20260709 12:23:11.640272 18374 raft_consensus.cc:2468] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 11.
I20260709 12:23:11.628047 17803 leader_election.cc:304] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:11.651618 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 15 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:11.654546 17688 raft_consensus.cc:2393] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 16: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
W20260709 12:23:11.654433 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 47913.3us
I20260709 12:23:11.628430 23118 raft_consensus.cc:493] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:11.655992 23056 raft_consensus.cc:2804] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:23:11.656564 23056 raft_consensus.cc:493] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:11.656991 23056 raft_consensus.cc:3060] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Advancing to term 12
W20260709 12:23:11.657390 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 67895.6us
W20260709 12:23:11.659176 23021 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.598s	user 0.017s	sys 0.024s
W20260709 12:23:11.659231 17803 leader_election.cc:336] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Timed out: RequestConsensusVote RPC to 127.16.29.68:42577 timed out after 1.672s (SENT)
W20260709 12:23:11.659710 23021 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.677s	user 0.017s	sys 0.025s
I20260709 12:23:11.660079 17729 leader_election.cc:304] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:11.661123 23120 raft_consensus.cc:515] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:11.593602 23305 raft_consensus.cc:515] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:11.663087 23088 raft_consensus.cc:2749] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:23:11.664662 18364 raft_consensus.cc:1240] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 11. Current term is 12. Ops: [11.10-11.10]
I20260709 12:23:11.655726 23118 raft_consensus.cc:515] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:11.670356 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:11.671455 23118 leader_election.cc:290] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:11.672003 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:11.672840 18365 raft_consensus.cc:2468] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 12.
I20260709 12:23:11.673980 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:11.674686 18365 raft_consensus.cc:2468] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 9.
I20260709 12:23:11.681089 18362 raft_consensus.cc:1275] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 9 index: 11. Preceding OpId from leader: term: 10 index: 12. (index mismatch)
I20260709 12:23:11.683320 18362 raft_consensus.cc:1275] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 11 index: 8. Preceding OpId from leader: term: 12 index: 9. (index mismatch)
I20260709 12:23:11.672122 22652 raft_consensus.cc:2749] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:23:11.686053 18367 raft_consensus.cc:2468] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 11.
I20260709 12:23:11.675664 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:11.691643 23351 consensus_queue.cc:1048] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.027s
W20260709 12:23:11.698490 23360 log.cc:927] Time spent T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.177s	user 0.001s	sys 0.000s
I20260709 12:23:11.700289 23364 raft_consensus.cc:2749] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:23:11.652792 23108 leader_election.cc:290] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 14 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:11.716374 17838 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35040 (request call id 2689) took 1303 ms. Trace:
I20260709 12:23:11.716977 17838 rpcz_store.cc:276] 0709 12:23:10.412971 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:10.548683 (+135712us) service_pool.cc:224] Handling call
0709 12:23:10.549367 (+   684us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:10.549904 (+   537us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:10.549938 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:10.550665 (+   727us) log.cc:844] Serialized 26 byte log entry
0709 12:23:11.187063 (+636398us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:23:11.187117 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:11.187212 (+    95us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:11.697662 (+510450us) spinlock_profiling.cc:243] Waited 4.5 ms on lock 0x7b4400022b4c. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b47459c85 00007f7b4745a6a2 00007f7b4a1cc1db 00007f7b4a1c7aaf 00007f7b4a1bb86e 00007f7b4a1babb6 00007f7b4a1ba7d8 00007f7b4a1c4221 00007f7b4a1c41d9 00007f7b4a1c4169
0709 12:23:11.702953 (+  5291us) raft_consensus.cc:1685] finished
0709 12:23:11.702985 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:11.716113 (+ 13128us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":10339584,"thread_start_us":461693,"threads_started":1,"wal-append.queue_time_us":774677}
I20260709 12:23:11.720413 17684 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37402 (request call id 2856) took 3011 ms. Trace:
I20260709 12:23:11.720628 23251 leader_election.cc:290] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:11.720925 17684 rpcz_store.cc:276] 0709 12:23:08.709384 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:10.340153 (+1630769us) service_pool.cc:224] Handling call
0709 12:23:10.341575 (+  1422us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:10.342218 (+   643us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:10.342246 (+    28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:10.342937 (+   691us) log.cc:844] Serialized 26 byte log entry
0709 12:23:11.131434 (+788497us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:23:11.131472 (+    38us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:11.131650 (+   178us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:11.720139 (+588489us) raft_consensus.cc:1685] finished
0709 12:23:11.720158 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:11.720343 (+   185us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":92259,"raft.run_cpu_time_us":283,"raft.run_wall_time_us":284,"thread_start_us":788024,"threads_started":1,"wal-append.queue_time_us":1075725}
I20260709 12:23:11.730108 18365 raft_consensus.cc:2393] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 16: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:23:11.709458 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 12 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:11.734961 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 12 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:11.735581 18362 raft_consensus.cc:2468] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 11.
I20260709 12:23:11.736429 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:11.736984 18362 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 12.
I20260709 12:23:11.737857 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:23:11.738005 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 78832.9us
I20260709 12:23:11.738590 18367 raft_consensus.cc:2393] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 14: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:23:11.738204 23056 raft_consensus.cc:515] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:11.739243 17803 leader_election.cc:336] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Timed out: RequestConsensusVote RPC to 127.16.29.68:42577 timed out after 1.729s (SENT)
I20260709 12:23:11.740355 17683 raft_consensus.cc:2393] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 12: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
W20260709 12:23:11.744649 23267 consensus_meta.cc:287] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.554s	user 0.011s	sys 0.001s
I20260709 12:23:11.745029 23120 leader_election.cc:290] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 16 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:11.745117 23267 raft_consensus.cc:515] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:11.749991 22853 raft_consensus.cc:515] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:11.729164 23220 leader_election.cc:290] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:11.768357 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:11.768740 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:11.769476 18362 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 12.
I20260709 12:23:11.773814 18368 raft_consensus.cc:2393] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 12: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:23:11.774896 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 13 candidate_status { last_received { term: 12 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:11.775729 18365 raft_consensus.cc:2393] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 13: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:23:11.791250 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:11.793309 18362 raft_consensus.cc:2410] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 17 because replica has last-logged OpId of term: 16 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 12.
I20260709 12:23:11.791257 17835 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35040 (request call id 2685) took 3252 ms. Trace:
I20260709 12:23:11.791849 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:11.793867 17835 rpcz_store.cc:276] 0709 12:23:08.538691 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:08.538819 (+   128us) service_pool.cc:224] Handling call
0709 12:23:08.539136 (+   317us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:11.239227 (+2700091us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:11.789052 (+549825us) spinlock_profiling.cc:243] Waited 2.67 s on lock 0x7b54003345d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:23:11.791200 (+  2148us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":6151720832}
I20260709 12:23:11.759961 23267 leader_election.cc:290] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 12 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:11.797413 18374 raft_consensus.cc:3060] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [term 13 LEARNER]: Advancing to term 14
I20260709 12:23:11.801677 22853 leader_election.cc:290] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 17 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:11.809039 23121 leader_election.cc:290] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:11.755391 23297 raft_consensus.cc:515] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:11.814667 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:11.809886 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:11.796795 23056 leader_election.cc:290] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 12 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:11.805735 23224 raft_consensus.cc:515] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:11.830214 18366 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59886 (request call id 1546) took 3309 ms. Trace:
I20260709 12:23:11.833801 18365 raft_consensus.cc:1275] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 12 index: 10. (index mismatch)
I20260709 12:23:11.821836 23266 raft_consensus.cc:515] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:11.834841 18368 raft_consensus.cc:2468] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 15.
W20260709 12:23:11.842029 17836 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35042 (request call id 2423) took 3154 ms (client timeout 2037 ms). Trace:
W20260709 12:23:11.853945 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:11.854072 17836 rpcz_store.cc:269] 0709 12:23:08.687831 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:08.701188 (+ 13357us) service_pool.cc:224] Handling call
0709 12:23:11.832386 (+3131198us) spinlock_profiling.cc:243] Waited 276 ms on lock 0x7b5400267854. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:23:11.841940 (+  9554us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":635912960}
I20260709 12:23:11.834812 18367 raft_consensus.cc:2393] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 14: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
W20260709 12:23:11.860963 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 375933us
W20260709 12:23:11.868171 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 275751us
W20260709 12:23:11.850275 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 468797us
W20260709 12:23:11.869169 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 375096us
W20260709 12:23:11.881363 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 225319us
I20260709 12:23:11.884728 17652 leader_election.cc:304] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:23:11.851773 18366 rpcz_store.cc:276] 0709 12:23:08.514629 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:08.514784 (+   155us) service_pool.cc:224] Handling call
0709 12:23:08.540340 (+ 25556us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:09.953086 (+1412746us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 12:23:10.156641 (+203555us) spinlock_profiling.cc:243] Waited 3.77 ms on lock 0x7b18003f9b80. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b47515e07 00007f7b503709a9 00007f7b4753269b 00007f7b4753093f 00007f7b47530455 00007f7b4e0183c7 00007f7b4a62a9a7 00007f7b4a63a0bf 00007f7b4a63a038
0709 12:23:10.457881 (+301240us) log.cc:844] Serialized 14 byte log entry
0709 12:23:10.895363 (+437482us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:10.918386 (+ 23023us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:23:10.920543 (+  2157us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:11.821048 (+900505us) spinlock_profiling.cc:243] Waited 1.37 s on lock 0x7b54006ae0d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:23:11.821144 (+    96us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:11.823982 (+  2838us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":47311,"prepare.queue_time_us":92100,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":35,"raft.queue_time_us":74591,"raft.run_cpu_time_us":880,"raft.run_wall_time_us":88191,"spinlock_wait_cycles":3148312576,"thread_start_us":682870,"threads_started":2,"wal-append.queue_time_us":699586}
W20260709 12:23:11.887820 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 240672us
I20260709 12:23:11.831974 23168 raft_consensus.cc:515] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:11.897089 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 518161us
I20260709 12:23:11.900169 18374 raft_consensus.cc:1275] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [term 14 LEARNER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 14 index: 8. (term mismatch)
I20260709 12:23:11.900591 18374 pending_rounds.cc:85] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da: Aborting all ops after (but not including) 7
I20260709 12:23:11.901181 18374 pending_rounds.cc:107] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da: Aborting uncommitted NO_OP operation due to leader change: 13.8
I20260709 12:23:11.901458 18374 raft_consensus.cc:2889] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [term 14 LEARNER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:23:11.901866 17728 leader_election.cc:336] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Timed out: RequestConsensusVote RPC to 127.16.29.65:33071 timed out after 1.757s (SENT)
I20260709 12:23:11.907723 23374 raft_consensus.cc:2804] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:23:11.909163 23374 raft_consensus.cc:493] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:11.910138 23374 raft_consensus.cc:3060] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:11.916311 17652 leader_election.cc:304] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:23:11.918367 23168 leader_election.cc:290] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 18 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:11.918814 17727 leader_election.cc:304] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
W20260709 12:23:11.919366 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:11.922082 17803 leader_election.cc:336] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 14 pre-election: RPC error from VoteRequest() call to peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Timed out: RequestConsensusVote RPC to 127.16.29.68:42577 timed out after 1.884s (SENT)
I20260709 12:23:11.922425 17803 leader_election.cc:304] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:23:11.927956 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 282349us
I20260709 12:23:11.928858 23056 raft_consensus.cc:493] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:11.930892 23056 raft_consensus.cc:515] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:11.931291 23224 leader_election.cc:290] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 14 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:11.932404 23297 leader_election.cc:290] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 13 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:11.932145 23305 leader_election.cc:290] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:11.932296 23372 raft_consensus.cc:2804] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:23:11.933586 23372 raft_consensus.cc:493] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:11.933889 23372 raft_consensus.cc:3060] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:11.940522 23244 raft_consensus.cc:2749] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:23:11.950650 17802 leader_election.cc:304] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:23:11.954309 22915 consensus_queue.cc:1059] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 11.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:23:11.956328 23246 raft_consensus.cc:3055] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:23:11.960630 23246 raft_consensus.cc:740] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:23:11.961776 23246 consensus_queue.cc:260] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:11.967406 23379 log.cc:927] Time spent T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.051s	user 0.001s	sys 0.000s
I20260709 12:23:11.966938 23372 raft_consensus.cc:515] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:11.966416 23088 raft_consensus.cc:2764] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Leader pre-election decision vote started in defunct term 12: won
I20260709 12:23:11.963671 23246 raft_consensus.cc:3060] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:11.972785 17729 leader_election.cc:304] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
W20260709 12:23:11.972770 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 53762.7us
I20260709 12:23:11.975726 17727 leader_election.cc:304] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:11.969975 23372 leader_election.cc:290] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 17 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:11.981865 23056 leader_election.cc:290] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:11.983976 23380 raft_consensus.cc:2749] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:23:11.989547 17801 leader_election.cc:304] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:11.993129 23168 raft_consensus.cc:2749] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:23:11.997485 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:11.996753 23056 raft_consensus.cc:2749] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:23:11.999293 23113 raft_consensus.cc:2804] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:23:11.999738 23113 raft_consensus.cc:493] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:12.000101 23113 raft_consensus.cc:3060] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:12.002336 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:12.002533 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:12.003024 18367 raft_consensus.cc:2468] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 11.
I20260709 12:23:12.006698 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:12.006706 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 12 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:12.008718 18365 raft_consensus.cc:3060] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:12.011551 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:12.013635 17836 raft_consensus.cc:2393] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 16: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:23:12.014681 18374 raft_consensus.cc:2468] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 12.
I20260709 12:23:12.016731 17760 raft_consensus.cc:1275] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 12 index: 10. (index mismatch)
I20260709 12:23:12.017722 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:12.017961 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:12.018698 17838 raft_consensus.cc:2393] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 13: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:23:12.018591 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:12.019424 17762 raft_consensus.cc:2468] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 11.
I20260709 12:23:12.019605 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:12.020673 17837 raft_consensus.cc:1275] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 14 index: 8. Preceding OpId from leader: term: 16 index: 9. (index mismatch)
I20260709 12:23:12.020932 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:12.021615 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:12.021958 17759 raft_consensus.cc:2393] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 16: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:23:12.023072 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:12.031773 17802 leader_election.cc:304] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:23:12.029305 17764 raft_consensus.cc:2468] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 11.
I20260709 12:23:12.032958 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:12.033429 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:12.034268 17682 raft_consensus.cc:2410] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for term 13 because replica has last-logged OpId of term: 12 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 9.
I20260709 12:23:12.035449 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:12.034133 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
W20260709 12:23:12.041106 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 51688.7us
I20260709 12:23:12.040251 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:12.036306 18366 raft_consensus.cc:2468] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 15.
I20260709 12:23:12.034897 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 13 candidate_status { last_received { term: 12 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:12.046940 18368 raft_consensus.cc:2393] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 13: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:23:12.047336 17756 raft_consensus.cc:2393] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 13: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:23:12.031021 23266 leader_election.cc:290] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 14 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:12.052130 23113 raft_consensus.cc:515] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:12.054467 17833 raft_consensus.cc:2393] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 17: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:23:12.035326 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:12.056665 17836 raft_consensus.cc:1275] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 14 index: 8. Preceding OpId from leader: term: 15 index: 9. (index mismatch)
I20260709 12:23:12.057858 17801 leader_election.cc:304] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:12.058378 17759 raft_consensus.cc:1275] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 15 index: 9. Preceding OpId from leader: term: 16 index: 9. (term mismatch)
I20260709 12:23:12.058799 23373 raft_consensus.cc:2804] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:23:12.059345 23373 raft_consensus.cc:697] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 17 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:23:12.064706 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:12.065512 17762 raft_consensus.cc:3060] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:12.058815 17759 pending_rounds.cc:85] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb: Aborting all ops after (but not including) 8
I20260709 12:23:12.068874 17759 pending_rounds.cc:107] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb: Aborting uncommitted NO_OP operation due to leader change: 15.9
I20260709 12:23:12.069216 17759 raft_consensus.cc:2889] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:12.072101 23113 leader_election.cc:290] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 17 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:12.072504 17760 raft_consensus.cc:1240] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 17. Current term is 18. Ops: []
I20260709 12:23:12.072762 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:12.073714 17764 raft_consensus.cc:2468] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 15.
I20260709 12:23:12.074929 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:23:12.070887 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 104909us
I20260709 12:23:12.082381 17762 raft_consensus.cc:2468] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 17.
W20260709 12:23:12.088399 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 56728.3us
W20260709 12:23:12.097939 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 140363us
I20260709 12:23:12.099249 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:12.101532 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 14 candidate_status { last_received { term: 11 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:12.105244 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 17 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:12.106146 17833 raft_consensus.cc:3060] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:12.107476 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:12.108361 17837 raft_consensus.cc:2393] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 14: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:23:12.115265 17802 leader_election.cc:304] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:12.116185 23378 raft_consensus.cc:493] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:12.116602 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:12.128750 23224 raft_consensus.cc:2749] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:23:12.129262 18371 raft_consensus.cc:2410] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 16 because replica has last-logged OpId of term: 15 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 11.
I20260709 12:23:12.096781 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:12.117311 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 15 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:12.117992 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:12.118603 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:12.133275 18364 raft_consensus.cc:2468] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 9.
I20260709 12:23:12.118690 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:12.134187 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:23:12.116605 17727 connection.cc:256] RPC call timeout handler was delayed by -2.405259408999882s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1757s for a response
W20260709 12:23:12.135545 17727 connection.cc:256] RPC call timeout handler was delayed by -2.1532594089999293s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2037s for a response
I20260709 12:23:12.119365 17833 raft_consensus.cc:2468] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 17.
W20260709 12:23:12.136154 17727 connection.cc:256] RPC call timeout handler was delayed by -1.2136594089997743s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.30810000000000004s for a response
I20260709 12:23:12.119333 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:12.120247 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:12.120977 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:12.121007 23257 raft_consensus.cc:493] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:23:12.139685 18362 raft_consensus.cc:2393] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 14: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:23:12.137826 18368 raft_consensus.cc:2410] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 17 because replica has last-logged OpId of term: 16 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 12.
I20260709 12:23:12.137557 18367 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 12.
I20260709 12:23:12.140384 18365 raft_consensus.cc:2468] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 12.
I20260709 12:23:12.121917 17652 leader_election.cc:304] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:12.130743 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:12.130816 18363 raft_consensus.cc:2468] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 11.
I20260709 12:23:12.144583 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:12.141671 18371 raft_consensus.cc:2468] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 12.
I20260709 12:23:12.060415 23373 consensus_queue.cc:237] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.12, Last appended by leader: 12, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:12.101050 23113 raft_consensus.cc:2804] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:23:12.151420 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:12.154079 17681 raft_consensus.cc:2393] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 13: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:23:12.153661 17756 raft_consensus.cc:2468] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 11.
I20260709 12:23:12.071499 23374 raft_consensus.cc:515] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:12.182958 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 61146.2us
I20260709 12:23:12.185352 17652 leader_election.cc:304] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:12.133108 23378 raft_consensus.cc:515] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:12.174232 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:12.147352 23305 consensus_queue.cc:1048] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.022s
I20260709 12:23:12.193740 17682 raft_consensus.cc:2393] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 14: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:23:12.196910 23266 raft_consensus.cc:2804] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:23:12.197146 17652 leader_election.cc:304] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:12.197465 23266 raft_consensus.cc:493] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:12.197876 23266 raft_consensus.cc:3060] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:12.200078 22915 raft_consensus.cc:2749] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:23:12.200947 17762 raft_consensus.cc:2468] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 15.
I20260709 12:23:12.200665 23376 consensus_queue.cc:1048] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:23:12.158190 23372 raft_consensus.cc:2749] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:23:12.147850 23257 raft_consensus.cc:515] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:12.199662 17652 leader_election.cc:304] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:12.214162 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:12.202383 23378 leader_election.cc:290] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:12.232007 17652 leader_election.cc:304] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:23:12.236811 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 70120.5us
I20260709 12:23:12.234745 23376 raft_consensus.cc:2804] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:23:12.233791 23113 raft_consensus.cc:697] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 13 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:23:12.243237 23297 raft_consensus.cc:2804] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:23:12.266045 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:12.266772 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:12.262991 17729 leader_election.cc:304] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:23:12.267236 17652 leader_election.cc:304] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:12.269795 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:12.270587 17764 raft_consensus.cc:3060] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:12.277683 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 17 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:12.280915 17652 leader_election.cc:304] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:12.282296 22998 raft_consensus.cc:2804] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:23:12.282680 23367 consensus_queue.cc:1048] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 12:23:12.283042 22853 raft_consensus.cc:2804] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:23:12.283458 22853 raft_consensus.cc:493] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:12.283851 22853 raft_consensus.cc:3060] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Advancing to term 16
W20260709 12:23:12.288409 17728 leader_election.cc:336] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Timed out: RequestConsensusVote RPC to 127.16.29.65:33071 timed out after 1.550s (SENT)
I20260709 12:23:12.289621 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:12.298822 23113 consensus_queue.cc:237] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 12.13, Last appended by leader: 13, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:12.300292 17763 raft_consensus.cc:2393] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 17: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:23:12.304718 17758 raft_consensus.cc:2468] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 12.
I20260709 12:23:12.322324 17683 raft_consensus.cc:1240] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 16. Current term is 17. Ops: []
I20260709 12:23:12.266853 23376 raft_consensus.cc:697] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 11 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
W20260709 12:23:12.328513 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 46255us
W20260709 12:23:12.332353 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 69501.6us
I20260709 12:23:12.336076 17801 leader_election.cc:304] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:12.337957 17652 leader_election.cc:304] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:12.322466 23378 raft_consensus.cc:2804] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:23:12.339921 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:12.335996 22998 raft_consensus.cc:697] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 17 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:23:12.290457 23293 consensus_queue.cc:1059] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 17.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.046s
I20260709 12:23:12.352088 17683 raft_consensus.cc:1240] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 13. Current term is 14. Ops: []
I20260709 12:23:12.363602 17652 leader_election.cc:304] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:12.373958 17757 raft_consensus.cc:1240] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 11. Current term is 12. Ops: []
I20260709 12:23:12.348205 23374 leader_election.cc:290] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 14 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:23:12.382225 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 46215.7us
I20260709 12:23:12.388013 17801 leader_election.cc:304] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:12.383785 17681 raft_consensus.cc:2468] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 12.
I20260709 12:23:12.384603 17764 raft_consensus.cc:2468] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 12.
W20260709 12:23:12.385473 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:12.386698 17652 leader_election.cc:304] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:12.354058 23372 raft_consensus.cc:2749] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:23:12.390282 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:23:12.366358 23389 log.cc:927] Time spent T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.161s	user 0.000s	sys 0.001s
I20260709 12:23:12.354964 23378 raft_consensus.cc:697] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:23:12.390986 23257 leader_election.cc:290] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:12.420655 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:23:12.435351 23398 log.cc:927] Time spent T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.082s	user 0.000s	sys 0.000s
I20260709 12:23:12.433243 23293 raft_consensus.cc:3060] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:23:12.435916 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 48026.9us
I20260709 12:23:12.436679 17801 leader_election.cc:304] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:12.437747 17801 leader_election.cc:304] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:12.439436 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:12.440790 23395 raft_consensus.cc:2804] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:23:12.441241 23395 raft_consensus.cc:493] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:12.441589 23395 raft_consensus.cc:3060] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:12.442865 22853 raft_consensus.cc:515] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:12.435290 23402 raft_consensus.cc:493] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:12.391395 22915 raft_consensus.cc:2749] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:23:12.448491 23400 raft_consensus.cc:2804] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:23:12.448812 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:12.449412 23402 raft_consensus.cc:515] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:12.448997 23400 raft_consensus.cc:493] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:12.451274 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 11 candidate_status { last_received { term: 9 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:12.454598 17652 leader_election.cc:304] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:23:12.467258 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:12.468124 17764 raft_consensus.cc:2468] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 13.
I20260709 12:23:12.470496 17729 leader_election.cc:304] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:12.412772 23266 raft_consensus.cc:515] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:12.475920 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:12.476565 17689 raft_consensus.cc:2468] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 14.
I20260709 12:23:12.462141 23400 raft_consensus.cc:3060] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:12.479148 23139 consensus_queue.cc:1059] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 11.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.003s
I20260709 12:23:12.483724 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:12.463299 23224 raft_consensus.cc:2804] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Leader election won for term 12
W20260709 12:23:12.488998 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 353535us
I20260709 12:23:12.453249 23401 raft_consensus.cc:2804] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:23:12.492779 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 14 candidate_status { last_received { term: 11 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:12.467269 23372 raft_consensus.cc:2804] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:23:12.495975 17833 raft_consensus.cc:2468] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 11.
I20260709 12:23:12.490226 17652 leader_election.cc:304] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:12.502096 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:12.499037 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:12.463490 23374 consensus_queue.cc:1048] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.026s
I20260709 12:23:12.486770 23351 raft_consensus.cc:3060] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:12.467554 23407 consensus_queue.cc:1048] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.037s
I20260709 12:23:12.522511 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:12.432236 23376 consensus_queue.cc:237] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:12.526247 17689 raft_consensus.cc:2410] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for term 15 because replica has last-logged OpId of term: 14 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 10.
I20260709 12:23:12.532318 17729 leader_election.cc:304] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:12.533557 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:12.534356 17833 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 12.
I20260709 12:23:12.539079 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:12.504001 23224 raft_consensus.cc:697] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 12 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:23:12.464462 22998 consensus_queue.cc:237] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.10, Last appended by leader: 10, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:12.543064 17689 raft_consensus.cc:2393] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 14: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:23:12.496907 23401 raft_consensus.cc:493] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:23:12.555528 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:12.556397 17833 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 12.
I20260709 12:23:12.555972 23224 consensus_queue.cc:237] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.7, Last appended by leader: 7, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:23:12.556555 23409 log.cc:927] Time spent T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.052s	user 0.000s	sys 0.000s
I20260709 12:23:12.557693 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:12.559020 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 14 candidate_status { last_received { term: 12 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
W20260709 12:23:12.563859 17728 leader_election.cc:336] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Timed out: RequestConsensusVote RPC to 127.16.29.65:33071 timed out after 1.817s (SENT)
I20260709 12:23:12.501327 23403 consensus_queue.cc:1048] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.043s
I20260709 12:23:12.565901 23056 raft_consensus.cc:2804] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:23:12.566526 23056 raft_consensus.cc:493] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:23:12.567150 23056 raft_consensus.cc:3060] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:12.536892 23378 consensus_queue.cc:237] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:12.568583 22853 leader_election.cc:290] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 16 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:12.570161 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:12.571396 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 17 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:12.571943 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:12.572346 17682 raft_consensus.cc:2468] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 16.
I20260709 12:23:12.572427 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:12.572880 17688 raft_consensus.cc:2410] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for term 13 because replica has last-logged OpId of term: 12 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 9.
I20260709 12:23:12.571313 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:12.573810 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:12.574043 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 14 candidate_status { last_received { term: 11 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:12.574671 17687 raft_consensus.cc:2393] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 13: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:23:12.574797 17681 raft_consensus.cc:2393] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 14: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:23:12.575517 23266 leader_election.cc:290] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 13 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:12.576249 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 16 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:12.577049 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:12.577776 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 11 candidate_status { last_received { term: 9 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:12.582701 23293 raft_consensus.cc:2749] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:23:12.584528 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:12.594460 17836 raft_consensus.cc:3060] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:12.585052 23401 raft_consensus.cc:3060] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:12.585565 23407 raft_consensus.cc:2804] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:23:12.602474 23407 raft_consensus.cc:493] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:23:12.603076 23407 raft_consensus.cc:3060] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:12.586153 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 14 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:12.609489 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:12.610457 17838 raft_consensus.cc:2393] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 14: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:23:12.611389 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:12.587731 23402 leader_election.cc:290] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:12.627897 17835 raft_consensus.cc:2410] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for term 16 because replica has last-logged OpId of term: 13 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 10.
I20260709 12:23:12.628058 17836 raft_consensus.cc:2468] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 14.
I20260709 12:23:12.628840 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:12.630426 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:12.644881 17833 raft_consensus.cc:2468] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 9.
I20260709 12:23:12.651213 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:12.644917 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:12.623749 23297 raft_consensus.cc:493] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:23:12.708735 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:12.709471 17835 raft_consensus.cc:3060] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Advancing to term 16
W20260709 12:23:12.709564 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 86870.7us
I20260709 12:23:12.718786 23407 raft_consensus.cc:515] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:12.722198 23407 leader_election.cc:290] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 14 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:12.626078 23113 consensus_queue.cc:1048] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.090s
W20260709 12:23:12.724344 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 85703.2us
W20260709 12:23:13.349535 17728 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.16.29.65:33071 timed out after 2.071s (SENT) [suppressed 38 similar messages]
I20260709 12:23:12.756378 17803 leader_election.cc:304] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:23:13.354077 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 597818us
I20260709 12:23:12.756219 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:12.763326 17835 raft_consensus.cc:2468] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 16.
I20260709 12:23:12.737346 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:13.358940 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:13.359549 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:13.360697 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:13.360837 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:13.361356 17833 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 12.
I20260709 12:23:13.361785 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:13.362167 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:13.362855 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:13.363197 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:12.734453 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:13.460716 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:13.462005 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:13.463096 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 15 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:12.947265 23437 raft_consensus.cc:493] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:23:13.464224 23437 raft_consensus.cc:515] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:12.981143 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:23:13.054344 17652 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.514s	user 0.040s	sys 0.038s
W20260709 12:23:13.466639 17652 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.926s	user 0.041s	sys 0.038s
I20260709 12:23:13.466678 17688 raft_consensus.cc:2468] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 15.
W20260709 12:23:13.467020 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 926400us
I20260709 12:23:13.468803 23305 consensus_queue.cc:1048] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:23:13.060040 17681 raft_consensus.cc:1240] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 15. Current term is 16. Ops: []
I20260709 12:23:13.470816 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:13.471580 17681 raft_consensus.cc:2393] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 16: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:23:13.472416 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:13.473563 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:13.474298 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:13.474679 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:13.475026 17688 raft_consensus.cc:2468] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 16.
I20260709 12:23:13.080175 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:13.476694 23437 leader_election.cc:290] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:13.476835 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:13.477866 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:13.478211 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:13.479398 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:13.133322 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:13.139570 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:23:13.158939 17727 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.654s	user 0.021s	sys 0.033s
I20260709 12:23:13.181862 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 12 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:13.182221 18362 raft_consensus.cc:1275] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 12 index: 10. (term mismatch)
W20260709 12:23:13.202530 18333 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.671s	user 0.006s	sys 0.018s
I20260709 12:23:13.481715 18362 pending_rounds.cc:85] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da: Aborting all ops after (but not including) 9
I20260709 12:23:13.204988 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:13.482098 18362 pending_rounds.cc:107] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da: Aborting uncommitted NO_OP operation due to leader change: 11.10
I20260709 12:23:13.215518 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:23:13.227063 17729 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.688s	user 0.088s	sys 0.074s
W20260709 12:23:13.482988 17729 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.944s	user 0.088s	sys 0.074s
I20260709 12:23:13.483191 18374 raft_consensus.cc:2410] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for term 12 because replica has last-logged OpId of term: 11 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 8.
W20260709 12:23:13.232679 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:13.481910 18333 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.951s	user 0.007s	sys 0.018s
I20260709 12:23:13.265144 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 17 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:23:13.484373 18333 connection.cc:256] RPC call timeout handler was delayed by -2.7133666429995174s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16190000000000002s for a response
I20260709 12:23:13.266897 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 15 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:23:13.484727 18333 connection.cc:256] RPC call timeout handler was delayed by -2.614366642999812s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17290000000000003s for a response
W20260709 12:23:13.275679 17755 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.863s	user 0.008s	sys 0.008s
W20260709 12:23:13.485051 18333 connection.cc:256] RPC call timeout handler was delayed by -2.5999666429997887s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17450000000000002s for a response
W20260709 12:23:13.485373 18333 connection.cc:256] RPC call timeout handler was delayed by -2.4478666429995428s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19140000000000001s for a response
W20260709 12:23:13.485690 18333 connection.cc:256] RPC call timeout handler was delayed by -1.5478666429994519s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.29140000000000005s for a response
I20260709 12:23:13.277312 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:13.278736 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:13.279352 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:13.281514 17763 raft_consensus.cc:1275] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
I20260709 12:23:13.289027 17758 raft_consensus.cc:1275] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 16 index: 12. (index mismatch)
I20260709 12:23:13.289934 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 12 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:13.318245 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:13.321439 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:12.930256 23056 raft_consensus.cc:515] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:13.332650 17655 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.829s	user 0.029s	sys 0.032s
W20260709 12:23:13.647464 17655 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.144s	user 0.029s	sys 0.033s
I20260709 12:23:13.333603 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 13 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:13.653209 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:13.653980 17685 raft_consensus.cc:2468] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 14.
I20260709 12:23:13.654794 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:13.655311 17685 raft_consensus.cc:2468] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 14.
W20260709 12:23:13.335153 17829 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.758s	user 0.026s	sys 0.034s
W20260709 12:23:13.656214 17829 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.079s	user 0.027s	sys 0.035s
I20260709 12:23:13.656991 17829 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37994 (request call id 1080) took 1082 ms. Trace:
I20260709 12:23:13.656810 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:13.657878 18374 raft_consensus.cc:2393] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 14: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:23:13.657375 17829 rpcz_store.cc:276] 0709 12:23:12.574099 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:12.575499 (+  1400us) service_pool.cc:224] Handling call
0709 12:23:12.575807 (+   308us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:12.576112 (+   305us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:12.576136 (+    24us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:12.576635 (+   499us) log.cc:844] Serialized 26 byte log entry
0709 12:23:13.656651 (+1080016us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:23:13.656690 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:13.656756 (+    66us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:13.656803 (+    47us) raft_consensus.cc:1685] finished
0709 12:23:13.656816 (+    13us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:13.656941 (+   125us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":1079578,"threads_started":1,"wal-append.queue_time_us":348299}
I20260709 12:23:13.337550 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 11 candidate_status { last_received { term: 9 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:13.338776 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:13.660151 18365 raft_consensus.cc:2468] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 12.
I20260709 12:23:13.660984 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:13.662060 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:13.662668 18365 raft_consensus.cc:3060] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:13.357543 17830 raft_consensus.cc:2468] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 15.
I20260709 12:23:13.363855 17833 raft_consensus.cc:2376] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 13. Re-sending same reply.
I20260709 12:23:13.208526 23257 raft_consensus.cc:3055] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:23:13.724144 23257 raft_consensus.cc:740] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:23:12.815027 23351 raft_consensus.cc:2749] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:23:12.781528 23372 raft_consensus.cc:697] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 17 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:23:13.728219 23257 consensus_queue.cc:260] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:12.952838 23293 raft_consensus.cc:493] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:23:13.733800 23293 raft_consensus.cc:515] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:12.713157 23297 raft_consensus.cc:3060] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Advancing to term 12
W20260709 12:23:13.111122 23374 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.526s	user 0.002s	sys 0.002s
W20260709 12:23:13.166956 23378 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.592s	user 0.005s	sys 0.000s
I20260709 12:23:13.363637 23445 consensus_queue.cc:1059] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.004s
W20260709 12:23:13.747059 23378 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.173s	user 0.008s	sys 0.000s
I20260709 12:23:13.747442 23445 raft_consensus.cc:3055] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:23:13.747920 23445 raft_consensus.cc:740] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:23:13.748859 23445 consensus_queue.cc:260] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.12, Last appended by leader: 12, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:12.830189 23395 raft_consensus.cc:515] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:12.826092 23400 raft_consensus.cc:515] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:13.754652 17835 raft_consensus.cc:2393] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 16: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:23:12.938316 23401 raft_consensus.cc:515] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:13.081552 23405 log.cc:927] Time spent T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.521s	user 0.003s	sys 0.000s
W20260709 12:23:13.188741 23415 log.cc:927] Time spent T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.624s	user 0.005s	sys 0.000s
W20260709 12:23:12.934664 23414 log.cc:927] Time spent T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.370s	user 0.004s	sys 0.001s
I20260709 12:23:12.927289 23433 raft_consensus.cc:493] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:13.770303 17762 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38776 (request call id 2431) took 1400 ms. Trace:
I20260709 12:23:13.770572 23433 raft_consensus.cc:515] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:12.951114 23434 raft_consensus.cc:493] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:13.374572 23449 consensus_queue.cc:1059] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 13.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 12:23:13.772972 17756 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38786 (request call id 3140) took 1511 ms. Trace:
I20260709 12:23:13.375560 18334 leader_election.cc:304] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:23:13.387831 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:13.389276 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 14 candidate_status { last_received { term: 12 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:13.772960 23434 raft_consensus.cc:515] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:13.774374 18371 raft_consensus.cc:2468] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 13.
I20260709 12:23:13.774540 23456 raft_consensus.cc:2804] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:23:13.775102 23456 raft_consensus.cc:493] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:13.775434 23456 raft_consensus.cc:3060] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:13.775980 23434 leader_election.cc:290] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:13.401126 23394 raft_consensus.cc:2804] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:23:13.776590 23394 raft_consensus.cc:493] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:23:13.777038 23394 raft_consensus.cc:3060] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:13.773494 17756 rpcz_store.cc:276] 0709 12:23:12.261156 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:12.261304 (+   148us) service_pool.cc:224] Handling call
0709 12:23:12.269436 (+  8132us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:12.318156 (+ 48720us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:12.318201 (+    45us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:12.346919 (+ 28718us) log.cc:844] Serialized 26 byte log entry
0709 12:23:12.485590 (+138671us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:23:12.485646 (+    56us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:12.485736 (+    90us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:12.863799 (+378063us) spinlock_profiling.cc:243] Waited 48.4 ms on lock 0x7b4400022b4c. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b47459c85 00007f7b4745a6a2 00007f7b4a1cc1db 00007f7b4a1c7aaf 00007f7b4a1bb86e 00007f7b4a1babb6 00007f7b4a1ba7d8 00007f7b4a1c4221 00007f7b4a1c41d9 00007f7b4a1c4169
0709 12:23:13.772656 (+908857us) raft_consensus.cc:1685] finished
0709 12:23:13.772681 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:13.772914 (+   233us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":8248,"raft.queue_time_us":23134,"raft.run_cpu_time_us":130,"raft.run_wall_time_us":5076,"spinlock_wait_cycles":111372672,"thread_start_us":113337,"threads_started":1,"wal-append.queue_time_us":208321}
I20260709 12:23:13.402989 23403 consensus_queue.cc:1048] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:23:13.442574 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:13.781023 17756 raft_consensus.cc:1275] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 14 index: 8. Preceding OpId from leader: term: 15 index: 9. (index mismatch)
I20260709 12:23:13.781910 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:13.783126 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:13.480558 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 17 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:13.773243 23449 raft_consensus.cc:3055] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:23:13.784230 23449 raft_consensus.cc:740] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:23:13.784386 17836 raft_consensus.cc:2393] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 17: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:23:13.784619 17688 raft_consensus.cc:2468] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 11.
I20260709 12:23:13.785112 23449 consensus_queue.cc:260] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.11, Last appended by leader: 11, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:13.482461 18362 raft_consensus.cc:2889] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:13.487619 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:13.489049 17689 raft_consensus.cc:2393] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 12: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:23:13.787161 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:13.760200 23293 leader_election.cc:290] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:13.787753 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 17 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:13.489310 18368 raft_consensus.cc:2410] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for term 15 because replica has last-logged OpId of term: 12 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 9.
I20260709 12:23:13.788864 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:13.789354 23378 raft_consensus.cc:493] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:23:13.486320 17680 raft_consensus.cc:2468] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 16.
I20260709 12:23:13.789400 17689 raft_consensus.cc:3060] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:13.791544 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:13.793692 17680 raft_consensus.cc:1240] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 12. Current term is 13. Ops: [12.8-12.8]
I20260709 12:23:13.794220 17688 raft_consensus.cc:1240] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 12. Current term is 13. Ops: [12.9-12.9]
W20260709 12:23:13.489406 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 139904us
W20260709 12:23:13.774016 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 398494us
I20260709 12:23:13.489331 18364 raft_consensus.cc:2393] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 12: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
W20260709 12:23:13.483443 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 955991us
I20260709 12:23:13.498320 17831 raft_consensus.cc:1240] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 13. Current term is 14. Ops: []
I20260709 12:23:13.499125 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:13.795908 18334 leader_election.cc:304] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:23:13.500398 17757 raft_consensus.cc:1240] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 17. Current term is 18. Ops: []
I20260709 12:23:13.502689 18366 raft_consensus.cc:2393] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 11: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:23:13.562682 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:23:13.550761 17755 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.157s	user 0.009s	sys 0.008s
I20260709 12:23:13.541059 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 15 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:13.797502 17684 raft_consensus.cc:2468] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 14.
I20260709 12:23:13.560961 23446 raft_consensus.cc:493] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:13.540416 17727 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.035s	user 0.021s	sys 0.035s
I20260709 12:23:13.533790 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:13.798619 17760 raft_consensus.cc:3060] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:13.503427 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:23:13.573114 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 68176.6us
I20260709 12:23:13.565838 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:13.532570 17759 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 12.
W20260709 12:23:13.598321 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 110189us
I20260709 12:23:13.603247 23452 raft_consensus.cc:493] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:23:13.581468 17834 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "22fa38faae58422e9cecc99973cab688"
dest_uuid: "e063541542a94969898868bf6eb5ce1b"
mode: GRACEFUL
new_leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
 from {username='slave'} at 127.0.0.1:59820
I20260709 12:23:13.605590 23451 consensus_queue.cc:1048] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
W20260709 12:23:13.538944 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 161927us
I20260709 12:23:13.645927 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 14 candidate_status { last_received { term: 12 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:13.659004 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:13.659655 18367 raft_consensus.cc:2468] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 10.
I20260709 12:23:13.821332 18361 raft_consensus.cc:3060] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:13.666883 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:13.822652 17685 raft_consensus.cc:2393] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 14: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:23:13.680111 17761 raft_consensus.cc:1240] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 14. Current term is 15. Ops: []
I20260709 12:23:13.774008 17683 raft_consensus.cc:2468] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 13.
I20260709 12:23:13.686728 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 17 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
W20260709 12:23:13.700263 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 98386.4us
I20260709 12:23:13.704790 23056 leader_election.cc:290] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 15 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:13.728627 23372 consensus_queue.cc:237] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.10, Last appended by leader: 10, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:13.118709 23246 raft_consensus.cc:3055] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 17 LEADER]: Stepping down as leader of term 17
W20260709 12:23:13.743913 23374 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.159s	user 0.002s	sys 0.003s
I20260709 12:23:13.751085 23445 raft_consensus.cc:3060] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:13.755463 23395 leader_election.cc:290] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 16 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:13.759037 17829 raft_consensus.cc:1240] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 15. Current term is 16. Ops: [15.9-15.9]
I20260709 12:23:13.758499 17654 leader_election.cc:304] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
W20260709 12:23:14.137948 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 385347us
I20260709 12:23:14.138682 17654 leader_election.cc:304] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:14.136461 23394 raft_consensus.cc:515] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:14.139794 17654 leader_election.cc:304] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:14.140193 23374 raft_consensus.cc:2749] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:23:14.140591 23470 raft_consensus.cc:2749] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 12:23:14.140967 17654 leader_election.cc:304] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:23:14.141196 23374 raft_consensus.cc:2749] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 12:23:14.140635 17830 raft_consensus.cc:1240] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 15. Current term is 16. Ops: [15.12-15.12]
I20260709 12:23:13.760746 17837 raft_consensus.cc:1240] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 11. Current term is 12. Ops: []
I20260709 12:23:14.141999 23374 raft_consensus.cc:2804] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:23:14.142832 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:13.757524 17587 catalog_manager.cc:5697] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 12 to 17, leader changed from e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66) to 0da169ff82674dc1947055ce43de68a3 (127.16.29.65). New cstate: current_term: 17 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:23:13.761828 23400 leader_election.cc:290] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 13 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:23:13.762667 17584 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35962 (request call id 356) took 58165 ms (client timeout 15000 ms). Trace:
I20260709 12:23:13.768718 23401 leader_election.cc:290] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 12 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:13.770794 17762 rpcz_store.cc:276] 0709 12:23:12.369705 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:12.371998 (+  2293us) service_pool.cc:224] Handling call
0709 12:23:12.372298 (+   300us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:12.372625 (+   327us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:12.372652 (+    27us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:12.373073 (+   421us) log.cc:844] Serialized 26 byte log entry
0709 12:23:12.463777 (+ 90704us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:23:12.463825 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:12.463907 (+    82us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:13.188036 (+724129us) spinlock_profiling.cc:243] Waited 48 ms on lock 0x7b4400022b4c. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b47459c85 00007f7b4745a6a2 00007f7b4a1cc1db 00007f7b4a1c7aaf 00007f7b4a1bb86e 00007f7b4a1babb6 00007f7b4a1ba7d8 00007f7b4a1c4221 00007f7b4a1c41d9 00007f7b4a1c4169
0709 12:23:13.769993 (+581957us) raft_consensus.cc:1685] finished
0709 12:23:13.770019 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:13.770238 (+   219us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":2569,"raft.run_cpu_time_us":51,"raft.run_wall_time_us":51,"spinlock_wait_cycles":110290688,"thread_start_us":89729,"threads_started":1,"wal-append.queue_time_us":190807}
I20260709 12:23:13.390547 17832 raft_consensus.cc:1275] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 15 index: 12. (index mismatch)
I20260709 12:23:13.786502 23449 raft_consensus.cc:3060] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:14.147336 17587 catalog_manager.cc:5697] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 9 to 14. New cstate: current_term: 14 leader_uuid: "334a4e1f59944544b3b3ff36616b57da" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:23:14.151149 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:14.151690 17587 catalog_manager.cc:5697] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 7 to 11. New cstate: current_term: 11 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:23:14.141827 23394 leader_election.cc:290] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 16 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:14.152499 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:14.142863 23374 raft_consensus.cc:493] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:23:14.153334 23374 raft_consensus.cc:3060] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:14.153079 17587 catalog_manager.cc:5697] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 10 to 12. New cstate: current_term: 12 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
W20260709 12:23:14.153725 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:13.775534 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 17 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:14.154351 17587 catalog_manager.cc:5697] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 9 to 11, leader changed from 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) to 0da169ff82674dc1947055ce43de68a3 (127.16.29.65). New cstate: current_term: 11 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:23:13.818898 23297 raft_consensus.cc:515] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:13.819471 18334 leader_election.cc:304] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:23:13.789973 23378 raft_consensus.cc:515] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:14.157284 23297 leader_election.cc:290] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 12 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:14.157487 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:14.157915 23297 consensus_queue.cc:1048] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 12:23:14.158073 23378 leader_election.cc:290] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:13.751782 23257 raft_consensus.cc:3060] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:13.826164 17729 leader_election.cc:304] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:23:14.159192 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 339276us
I20260709 12:23:13.842048 18362 raft_consensus.cc:1240] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 11. Current term is 12. Ops: []
I20260709 12:23:13.843271 18333 leader_election.cc:304] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:13.686640 18365 raft_consensus.cc:2468] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 13.
I20260709 12:23:13.821075 17803 leader_election.cc:304] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:14.157248 17654 leader_election.cc:304] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:13.864643 17686 raft_consensus.cc:1275] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 16 index: 12. (index mismatch)
I20260709 12:23:13.821079 23351 raft_consensus.cc:493] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:23:13.875818 17689 raft_consensus.cc:2468] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 12.
I20260709 12:23:13.879943 17761 raft_consensus.cc:1240] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 11. Current term is 12. Ops: []
I20260709 12:23:13.827304 23433 leader_election.cc:290] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:13.885684 17758 raft_consensus.cc:1275] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 17 index: 11. (index mismatch)
I20260709 12:23:13.888484 23468 consensus_queue.cc:1059] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 17.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:23:13.894769 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:13.841744 23403 consensus_queue.cc:1059] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 11.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.050s
I20260709 12:23:13.905036 17687 raft_consensus.cc:1240] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 13. Current term is 14. Ops: []
W20260709 12:23:14.168916 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 339811us
I20260709 12:23:13.906210 17764 raft_consensus.cc:1275] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 10 index: 7. Preceding OpId from leader: term: 12 index: 8. (index mismatch)
I20260709 12:23:13.913316 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:13.847733 23224 consensus_queue.cc:1059] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 11.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.049s
I20260709 12:23:14.166489 17587 catalog_manager.cc:5697] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da reported cstate change: config changed from index -1 to 5, term changed from 1 to 11, leader changed from e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66) to e063541542a94969898868bf6eb5ce1b (127.16.29.67), NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New cstate: current_term: 11 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } }
W20260709 12:23:14.173079 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 352085us
I20260709 12:23:14.170564 23351 raft_consensus.cc:515] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:14.173789 17803 leader_election.cc:304] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:23:14.173462 17587 catalog_manager.cc:5697] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 9 to 12. New cstate: current_term: 12 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:23:14.174769 17587 catalog_manager.cc:5697] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 11 to 12, leader changed from 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) to 0da169ff82674dc1947055ce43de68a3 (127.16.29.65). New cstate: current_term: 12 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:23:14.175341 23351 leader_election.cc:290] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:23:13.921545 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 65002.6us
I20260709 12:23:13.852948 23437 consensus_queue.cc:1048] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.047s
I20260709 12:23:13.926303 18367 raft_consensus.cc:1275] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 16 index: 12. (term mismatch)
I20260709 12:23:13.937197 17760 raft_consensus.cc:2468] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 16.
I20260709 12:23:14.176978 23454 raft_consensus.cc:3055] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:23:14.177237 18367 pending_rounds.cc:85] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da: Aborting all ops after (but not including) 11
I20260709 12:23:14.177438 23454 raft_consensus.cc:740] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 12 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:23:13.841183 23446 raft_consensus.cc:515] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:14.169670 23461 raft_consensus.cc:2804] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:23:14.178745 23372 raft_consensus.cc:2749] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:23:13.946512 17682 raft_consensus.cc:1240] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 15. Current term is 16. Ops: []
I20260709 12:23:13.939474 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:13.950244 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:14.179515 23400 raft_consensus.cc:3055] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 12 LEADER]: Stepping down as leader of term 12
W20260709 12:23:13.979504 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:14.180238 17763 raft_consensus.cc:2468] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 13.
I20260709 12:23:14.180786 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:14.180967 17803 leader_election.cc:304] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:14.181350 17757 raft_consensus.cc:3060] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:14.181782 23395 raft_consensus.cc:2804] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:23:14.182176 23395 raft_consensus.cc:493] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:14.182786 23395 raft_consensus.cc:3060] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:14.183225 17763 raft_consensus.cc:1275] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20260709 12:23:13.946249 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:14.183861 17652 leader_election.cc:304] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:14.184173 17756 raft_consensus.cc:3060] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:14.180233 23400 raft_consensus.cc:740] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 12 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:23:14.185540 17652 leader_election.cc:304] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:14.185644 23376 consensus_queue.cc:1059] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 14.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:23:14.186545 23400 consensus_queue.cc:260] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.9, Last appended by leader: 9, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:13.848781 23452 raft_consensus.cc:515] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:14.077440 23469 consensus_queue.cc:1059] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 14.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:23:14.078373 17727 leader_election.cc:304] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:14.071511 23056 raft_consensus.cc:493] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:23:13.994027 23453 raft_consensus.cc:493] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:13.998819 23451 consensus_queue.cc:1059] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 14.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.030s
I20260709 12:23:14.094991 18361 raft_consensus.cc:2468] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 14.
I20260709 12:23:14.143002 17837 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37994 (request call id 1074) took 1571 ms. Trace:
I20260709 12:23:13.762493 17838 raft_consensus.cc:2393] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 13: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:23:14.151365 17832 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37994 (request call id 1084) took 1576 ms. Trace:
I20260709 12:23:14.152266 17829 raft_consensus.cc:1275] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 17 index: 11. (index mismatch)
W20260709 12:23:13.818216 17728 leader_election.cc:336] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Timed out: RequestConsensusVote RPC to 127.16.29.65:33071 timed out after 2.080s (SENT)
I20260709 12:23:14.158172 17762 raft_consensus.cc:3060] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:13.801777 17834 raft_consensus.cc:606] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 16 LEADER]: Received request to transfer leadership to TS e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:13.847708 23305 consensus_queue.cc:1048] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 12:23:13.841775 23458 raft_consensus.cc:2804] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:23:13.868741 17685 raft_consensus.cc:1240] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 13. Current term is 14. Ops: []
I20260709 12:23:14.192662 23434 raft_consensus.cc:3055] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 15 LEADER]: Stepping down as leader of term 15
W20260709 12:23:14.155825 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 336407us
I20260709 12:23:14.190913 23297 raft_consensus.cc:2749] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:23:14.192922 23378 consensus_queue.cc:1048] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.014s
I20260709 12:23:14.198067 17832 rpcz_store.cc:276] 0709 12:23:12.575237 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:12.631447 (+ 56210us) service_pool.cc:224] Handling call
0709 12:23:12.763086 (+131639us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:14.149909 (+1386823us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:14.151297 (+  1388us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:14.167392 17680 raft_consensus.cc:1275] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 10 index: 7. Preceding OpId from leader: term: 12 index: 8. (index mismatch)
I20260709 12:23:14.179783 23446 leader_election.cc:290] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:23:14.145231 17584 rpcz_store.cc:269] 0709 12:22:15.597451 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:15.597630 (+   179us) service_pool.cc:224] Handling call
0709 12:22:16.416358 (+818728us) spinlock_profiling.cc:243] Waited 547 ms on lock 0x7b540010d454. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a58a090 00007f7b4a628f3a 00007f7b503dee9c 00007f7b504dd04e 00007f7b4e99e2a4 00007f7b4e99e232 00007f7b4e99e161 00007f7b4e99e0dc 00007f7b4e99d392 00007f7b4895b48c 00007f7b4895a8c6
0709 12:23:13.762545 (+57346187us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:23:11.797243 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:23:11.798064 (+   821us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:23:11.798129 (+    65us) write_op.cc:435] Acquired schema lock
0709 12:23:11.798152 (+    23us) tablet.cc:662] Decoding operations
0709 12:23:11.798820 (+   668us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:23:11.798925 (+   105us) write_op.cc:641] Partition lock acquired for write op
0709 12:23:11.798953 (+    28us) tablet.cc:685] Acquiring locks for 3 operations
0709 12:23:11.799361 (+   408us) tablet.cc:701] Row locks acquired
0709 12:23:11.799406 (+    45us) write_op.cc:260] PREPARE: finished
0709 12:23:11.799605 (+   199us) write_op.cc:270] Start()
0709 12:23:11.799770 (+   165us) write_op.cc:276] Timestamp: P: 1783599791799521 usec, L: 0
0709 12:23:11.799810 (+    40us) op_driver.cc:348] REPLICATION: starting
0709 12:23:11.884384 (+ 84574us) log.cc:844] Serialized 1303 byte log entry
0709 12:23:11.983656 (+ 99272us) spinlock_profiling.cc:243] Waited 11.2 ms on lock 0x7b540010d454. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a61be28 00007f7b4e056581 00007f7b4e055d58 00007f7b4e05a9f1 00007f7b4e05a9a9 00007f7b4e05a939 00007f7b4e05a901 00007f7b4e059b2d 0000000000388634 0000000000388479
0709 12:23:11.991058 (+  7402us) spinlock_profiling.cc:243] Waited 2.42 ms on lock 0x7b1800032200. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b47515e07 00007f7b503709a9 00007f7b4753269b 00007f7b4753093f 00007f7b47530455 00007f7b4a59d190 00007f7b4a596ae7 00007f7b4a595dd9 00007f7b4a5a0930
0709 12:23:12.230933 (+239875us) op_driver.cc:464] REPLICATION: finished
0709 12:23:12.488453 (+257520us) write_op.cc:301] APPLY: starting
0709 12:23:12.488581 (+   128us) tablet.cc:1367] starting BulkCheckPresence
0709 12:23:12.747096 (+258515us) tablet.cc:1370] finished BulkCheckPresence
0709 12:23:12.747157 (+    61us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:23:12.931447 (+184290us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:23:12.931816 (+   369us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=6,key_file_lookups=6,delta_file_lookups=6,mrs_lookups=0
0709 12:23:12.931925 (+   109us) write_op.cc:312] APPLY: finished
0709 12:23:12.932352 (+   427us) log.cc:844] Serialized 41 byte log entry
0709 12:23:12.932635 (+   283us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:23:12.932785 (+   150us) write_op.cc:454] Released schema lock
0709 12:23:12.932885 (+   100us) write_op.cc:341] FINISH: Updating metrics
0709 12:23:12.933501 (+   616us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:23:12.933542 (+    41us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":7849914,"reported_tablets":44,"spinlock_wait_cycles":1256990592,"child_traces":[["op",{"apply.queue_time_us":231125,"apply.run_cpu_time_us":10436,"apply.run_wall_time_us":454061,"cfile_cache_hit":14,"cfile_cache_hit_bytes":16679,"num_ops":3,"prepare.queue_time_us":812839,"prepare.run_cpu_time_us":21793,"prepare.run_wall_time_us":186674,"raft.queue_time_us":198749,"raft.run_cpu_time_us":10689,"raft.run_wall_time_us":163757,"replication_time_us":406150,"spinlock_wait_cycles":31362688,"thread_start_us":822191,"threads_started":4,"wal-append.queue_time_us":78112,"wal-append.run_cpu_time_us":61706,"wal-append.run_wall_time_us":1965455}]]}
W20260709 12:23:14.167737 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 175195us
I20260709 12:23:14.143251 23246 raft_consensus.cc:740] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:23:14.187788 23400 raft_consensus.cc:3060] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:14.177563 18367 pending_rounds.cc:107] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da: Aborting uncommitted NO_OP operation due to leader change: 15.12
I20260709 12:23:14.172742 23403 raft_consensus.cc:2804] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:23:14.178696 23461 raft_consensus.cc:493] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:23:14.178404 23454 consensus_queue.cc:260] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.8, Last appended by leader: 8, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:14.179720 18333 leader_election.cc:304] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:23:14.186904 23224 raft_consensus.cc:2804] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:23:13.869753 17802 leader_election.cc:304] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:23:14.247190 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 377538us
W20260709 12:23:14.077143 23474 log.cc:927] Time spent T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.147s	user 0.001s	sys 0.000s
I20260709 12:23:14.002605 17755 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38786 (request call id 3146) took 1633 ms. Trace:
I20260709 12:23:14.188628 17759 raft_consensus.cc:1240] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 15. Current term is 16. Ops: []
I20260709 12:23:14.188862 17652 leader_election.cc:304] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:14.190845 17587 catalog_manager.cc:5697] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 8 to 10, leader changed from e063541542a94969898868bf6eb5ce1b (127.16.29.67) to e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66). New cstate: current_term: 10 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:23:14.197875 17837 rpcz_store.cc:276] 0709 12:23:12.572167 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:12.575334 (+  3167us) service_pool.cc:224] Handling call
0709 12:23:12.575700 (+   366us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:14.142215 (+1566515us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:14.142781 (+   566us) spinlock_profiling.cc:243] Waited 1.18 s on lock 0x7b5400159ad4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:23:14.142950 (+   169us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2720782720}
W20260709 12:23:14.263161 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 74419.4us
I20260709 12:23:14.197564 18361 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59918 (request call id 1746) took 1018 ms. Trace:
W20260709 12:23:14.201890 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 406196us
I20260709 12:23:14.263763 18361 rpcz_store.cc:276] 0709 12:23:13.179268 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:13.645650 (+466382us) service_pool.cc:224] Handling call
0709 12:23:14.197503 (+551853us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:14.264711 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:14.251014 17757 raft_consensus.cc:2468] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 12.
I20260709 12:23:14.217535 17762 raft_consensus.cc:2468] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 16.
I20260709 12:23:14.221503 18367 raft_consensus.cc:2889] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:14.223110 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:14.200567 17756 raft_consensus.cc:2468] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 12.
I20260709 12:23:14.202729 17838 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37994 (request call id 1083) took 1627 ms. Trace:
I20260709 12:23:14.200925 17803 leader_election.cc:304] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:14.266779 17838 rpcz_store.cc:276] 0709 12:23:12.574994 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:12.611087 (+ 36093us) service_pool.cc:224] Handling call
0709 12:23:14.202524 (+1591437us) spinlock_profiling.cc:243] Waited 1.14 s on lock 0x7b54003d4fd4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:23:14.202671 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2631977216}
I20260709 12:23:14.267524 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:14.268764 17652 leader_election.cc:304] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:23:14.269025 17801 leader_election.cc:304] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
W20260709 12:23:14.271446 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 74220.9us
I20260709 12:23:14.271070 17587 catalog_manager.cc:5697] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 7 to 9. New cstate: current_term: 9 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:23:14.272689 17654 leader_election.cc:304] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:14.274595 17654 leader_election.cc:304] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:23:14.172137 23427 raft_consensus.cc:2749] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:23:14.227934 23305 consensus_queue.cc:1048] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.049s
I20260709 12:23:14.221015 23246 consensus_queue.cc:260] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 17.15, Last appended by leader: 15, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:14.277182 17587 catalog_manager.cc:5697] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 12 to 14. New cstate: current_term: 14 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:23:14.176571 23456 raft_consensus.cc:515] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:14.202987 23056 raft_consensus.cc:515] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:14.264197 23372 raft_consensus.cc:2749] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:23:14.203343 23451 raft_consensus.cc:3055] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:23:14.224483 23446 raft_consensus.cc:2804] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:23:14.205217 23434 raft_consensus.cc:740] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:23:14.240294 23374 raft_consensus.cc:515] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:14.224068 23224 raft_consensus.cc:493] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
W20260709 12:23:14.279441 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 202397us
I20260709 12:23:14.280362 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:14.221913 23461 raft_consensus.cc:3060] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:14.200346 23475 raft_consensus.cc:2804] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:23:14.281070 23468 raft_consensus.cc:3060] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Advancing to term 16
W20260709 12:23:14.281760 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 80953.1us
I20260709 12:23:14.282061 17801 leader_election.cc:304] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:14.293896 18367 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 1744) took 1115 ms. Trace:
I20260709 12:23:14.262467 17755 rpcz_store.cc:276] 0709 12:23:12.369048 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:12.376479 (+  7431us) service_pool.cc:224] Handling call
0709 12:23:12.376791 (+   312us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:12.377038 (+   247us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 12:23:13.822256 (+1445218us) log.cc:844] Serialized 14 byte log entry
0709 12:23:14.001865 (+179609us) log.cc:844] Serialized 14 byte log entry
0709 12:23:14.002063 (+   198us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:14.002114 (+    51us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:23:14.002163 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:14.002260 (+    97us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:14.002384 (+   124us) inbound_call.cc:177] Queueing success response
0709 12:23:14.010487 (+  8103us) spinlock_profiling.cc:243] Waited 10.3 ms on lock 0x7b18002bf300. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b475333f3 00007f7b475357c1 00007f7b47535779 00007f7b47535709 00007f7b475356d1 00007f7b475349cd 0000000000388634 0000000000388479 00007f7b4751683a
Metrics: {"prepare.queue_time_us":206268,"prepare.run_cpu_time_us":7608,"prepare.run_wall_time_us":77403,"raft.queue_time_us":72755,"raft.run_cpu_time_us":163,"raft.run_wall_time_us":166,"spinlock_wait_cycles":23586688,"thread_start_us":1579907,"threads_started":3,"wal-append.queue_time_us":104772}
I20260709 12:23:14.245512 23454 raft_consensus.cc:3060] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:14.296970 17652 leader_election.cc:304] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:14.320760 18367 rpcz_store.cc:276] 0709 12:23:13.178912 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:13.893345 (+714433us) service_pool.cc:224] Handling call
0709 12:23:13.910293 (+ 16948us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:14.283359 (+373066us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:14.293823 (+ 10464us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:14.204742 23452 leader_election.cc:290] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:23:14.299731 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 139170us
I20260709 12:23:14.204411 23376 raft_consensus.cc:2749] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 12:23:14.202527 23458 raft_consensus.cc:697] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 12 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:23:14.221746 23403 raft_consensus.cc:493] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:23:14.260596 23401 raft_consensus.cc:2749] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 12:23:14.204866 23453 raft_consensus.cc:515] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:14.299432 17587 catalog_manager.cc:5697] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 10 to 12. New cstate: current_term: 12 leader_uuid: "334a4e1f59944544b3b3ff36616b57da" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:23:14.281697 23476 raft_consensus.cc:2749] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:23:14.294248 23297 consensus_queue.cc:1048] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.038s
W20260709 12:23:14.338024 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 56048us
I20260709 12:23:14.301823 23437 raft_consensus.cc:2804] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:23:14.339846 17727 leader_election.cc:304] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:14.302667 23446 raft_consensus.cc:493] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:14.299767 23266 raft_consensus.cc:2804] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:23:14.301328 23400 raft_consensus.cc:2804] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:23:14.317317 23378 raft_consensus.cc:2749] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:23:14.321862 23224 raft_consensus.cc:3060] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:14.302812 23451 raft_consensus.cc:740] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 14 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:23:14.321523 23475 raft_consensus.cc:493] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:14.334334 23246 raft_consensus.cc:3060] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:14.342412 23403 raft_consensus.cc:3060] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:14.296134 23395 raft_consensus.cc:515] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:14.360366 23372 raft_consensus.cc:2804] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:23:14.361956 23452 raft_consensus.cc:2804] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:23:14.362282 17587 catalog_manager.cc:5697] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 8 to 12, leader changed from e063541542a94969898868bf6eb5ce1b (127.16.29.67) to 0da169ff82674dc1947055ce43de68a3 (127.16.29.65). New cstate: current_term: 12 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
W20260709 12:23:14.384279 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 88427.5us
W20260709 12:23:14.414058 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 95965.4us
I20260709 12:23:14.363118 23446 raft_consensus.cc:3060] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:14.374428 17803 leader_election.cc:304] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:14.459223 17652 leader_election.cc:304] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:23:14.413295 18406 debug-util.cc:398] Leaking SignalData structure 0x7b08002fe820 after lost signal to thread 16504
I20260709 12:23:14.461905 17587 catalog_manager.cc:5697] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 6 to 11, leader changed from 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) to 0da169ff82674dc1947055ce43de68a3 (127.16.29.65). New cstate: current_term: 11 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
W20260709 12:23:14.464993 18406 debug-util.cc:398] Leaking SignalData structure 0x7b0800212680 after lost signal to thread 17621
I20260709 12:23:14.465286 17587 catalog_manager.cc:5697] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 12 to 14. New cstate: current_term: 14 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
W20260709 12:23:14.466773 18406 debug-util.cc:398] Leaking SignalData structure 0x7b0800208200 after lost signal to thread 17713
W20260709 12:23:14.467190 18406 debug-util.cc:398] Leaking SignalData structure 0x7b0800236b00 after lost signal to thread 17714
W20260709 12:23:14.472597 18406 debug-util.cc:398] Leaking SignalData structure 0x7b08002fe680 after lost signal to thread 17788
I20260709 12:23:14.371742 23266 raft_consensus.cc:697] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 16 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:23:14.374693 23475 raft_consensus.cc:3060] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:14.477667 17587 catalog_manager.cc:5697] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 7 to 8, leader changed from e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66) to 334a4e1f59944544b3b3ff36616b57da (127.16.29.68). New cstate: current_term: 8 leader_uuid: "334a4e1f59944544b3b3ff36616b57da" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:23:14.413852 23372 raft_consensus.cc:493] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
W20260709 12:23:14.476616 18406 debug-util.cc:398] Leaking SignalData structure 0x7b08002fe660 after lost signal to thread 17789
W20260709 12:23:14.490680 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 158253us
I20260709 12:23:14.461180 23488 raft_consensus.cc:493] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:14.500867 23492 raft_consensus.cc:2804] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:23:14.505786 17727 leader_election.cc:304] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
W20260709 12:23:14.505249 18406 debug-util.cc:398] Leaking SignalData structure 0x7b08001b64c0 after lost signal to thread 17862
W20260709 12:23:14.510161 18406 debug-util.cc:398] Leaking SignalData structure 0x7b080038d700 after lost signal to thread 17863
W20260709 12:23:14.513408 18406 debug-util.cc:398] Leaking SignalData structure 0x7b080003b780 after lost signal to thread 18409
I20260709 12:23:14.511155 17587 catalog_manager.cc:5697] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 9 to 11. New cstate: current_term: 11 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:23:14.508723 23492 raft_consensus.cc:697] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 12 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:23:14.518924 23480 raft_consensus.cc:2749] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
W20260709 12:23:14.520184 23484 log.cc:927] Time spent T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.161s	user 0.001s	sys 0.000s
I20260709 12:23:14.504211 23372 raft_consensus.cc:3060] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:14.519392 23492 consensus_queue.cc:237] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:14.528517 23479 log.cc:927] Time spent T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.191s	user 0.000s	sys 0.000s
I20260709 12:23:14.530071 18363 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 1738) took 1352 ms. Trace:
I20260709 12:23:14.530501 18363 rpcz_store.cc:276] 0709 12:23:13.177778 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:13.864904 (+687126us) service_pool.cc:224] Handling call
0709 12:23:13.892788 (+ 27884us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:13.900198 (+  7410us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:13.900238 (+    40us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:13.910972 (+ 10734us) log.cc:844] Serialized 26 byte log entry
0709 12:23:14.002572 (+ 91600us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:23:14.002612 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:14.002703 (+    91us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:14.529834 (+527131us) raft_consensus.cc:1685] finished
0709 12:23:14.529854 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:14.530006 (+   152us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":43652,"threads_started":1,"wal-append.queue_time_us":333881}
I20260709 12:23:14.532456 17681 raft_consensus.cc:2468] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 12.
I20260709 12:23:14.369817 23434 consensus_queue.cc:260] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.9, Last appended by leader: 9, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:14.507017 17728 leader_election.cc:304] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:23:14.493224 23452 raft_consensus.cc:697] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 12 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:23:14.468286 23293 consensus_queue.cc:1048] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.038s
I20260709 12:23:14.563478 17587 catalog_manager.cc:5697] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 1 to 8. New cstate: current_term: 8 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
W20260709 12:23:14.566200 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 191854us
I20260709 12:23:14.402522 23400 raft_consensus.cc:697] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 16 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:23:14.541574 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:14.537788 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:23:14.585269 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 51386.5us
I20260709 12:23:14.586046 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:14.585750 17587 catalog_manager.cc:5697] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 4 to 13. New cstate: current_term: 13 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:23:14.586383 18366 raft_consensus.cc:1240] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 13. Current term is 14. Ops: []
I20260709 12:23:14.585302 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 13 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:14.603739 18367 raft_consensus.cc:2410] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for term 13 because replica has last-logged OpId of term: 9 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 8.
I20260709 12:23:14.503728 23266 consensus_queue.cc:237] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:14.587253 23488 raft_consensus.cc:515] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:14.587627 23468 raft_consensus.cc:2749] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:23:14.589133 23351 raft_consensus.cc:3055] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 LEADER]: Stepping down as leader of term 15
W20260709 12:23:14.590785 17681 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37396 (request call id 2509) took 1635 ms (client timeout 2045 ms). Trace:
I20260709 12:23:14.506031 23469 raft_consensus.cc:2749] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
W20260709 12:23:14.592988 23477 log.cc:927] Time spent T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.459s	user 0.003s	sys 0.008s
W20260709 12:23:14.614603 17681 rpcz_store.cc:269] 0709 12:23:12.955031 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:13.475654 (+520623us) service_pool.cc:224] Handling call
0709 12:23:14.590566 (+1114912us) spinlock_profiling.cc:243] Waited 726 ms on lock 0x7b54004268d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:23:14.590684 (+   118us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1670033792}
I20260709 12:23:14.593727 23452 consensus_queue.cc:237] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:14.595871 17760 raft_consensus.cc:1240] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 15. Current term is 16. Ops: [15.9-15.9]
I20260709 12:23:14.596532 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:14.597139 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:14.599354 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:14.621959 18365 raft_consensus.cc:2468] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 13.
I20260709 12:23:14.605358 18361 raft_consensus.cc:2468] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 11.
W20260709 12:23:14.607051 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 131861us
I20260709 12:23:14.608162 18371 raft_consensus.cc:2410] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for term 17 because replica has last-logged OpId of term: 16 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 10.
I20260709 12:23:14.608160 17803 leader_election.cc:304] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:14.592661 23374 leader_election.cc:290] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 13 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:14.618530 17587 catalog_manager.cc:5697] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 6 to 9, leader changed from 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) to 334a4e1f59944544b3b3ff36616b57da (127.16.29.68). New cstate: current_term: 9 leader_uuid: "334a4e1f59944544b3b3ff36616b57da" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:23:14.627956 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:14.628808 18364 raft_consensus.cc:3060] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:14.629396 17755 raft_consensus.cc:1240] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 17. Current term is 18. Ops: []
I20260709 12:23:14.632922 18365 raft_consensus.cc:1240] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 15. Current term is 16. Ops: []
W20260709 12:23:14.639468 17652 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:14.641597 17652 leader_election.cc:304] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:14.613725 23378 raft_consensus.cc:2804] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:23:14.642294 17727 leader_election.cc:304] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:14.615602 23351 raft_consensus.cc:740] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:23:14.664353 17587 catalog_manager.cc:5697] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 8 to 13, leader changed from e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66) to 0da169ff82674dc1947055ce43de68a3 (127.16.29.65). New cstate: current_term: 13 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:23:14.635506 23492 raft_consensus.cc:493] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:14.644114 23434 raft_consensus.cc:3060] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:14.485554 23458 consensus_queue.cc:237] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:14.641371 23056 leader_election.cc:290] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:14.617332 23372 raft_consensus.cc:515] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:14.534392 23451 consensus_queue.cc:260] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 14.12, Last appended by leader: 12, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:14.648324 23378 raft_consensus.cc:493] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
W20260709 12:23:14.699596 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 58082.3us
I20260709 12:23:14.709623 17755 raft_consensus.cc:1240] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 15. Current term is 16. Ops: [15.13-15.13]
W20260709 12:23:14.711387 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:14.705840 23468 raft_consensus.cc:2749] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:23:14.678707 23469 raft_consensus.cc:2804] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:23:14.707453 17654 leader_election.cc:304] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:23:14.689220 23472 raft_consensus.cc:493] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:14.694303 23378 raft_consensus.cc:3060] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:23:14.776410 18333 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.527s	user 0.009s	sys 0.013s
W20260709 12:23:14.777067 18333 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.528s	user 0.009s	sys 0.013s
W20260709 12:23:14.777384 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 549587us
W20260709 12:23:14.783380 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 141162us
W20260709 12:23:14.783783 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 54418.2us
I20260709 12:23:14.710867 23456 leader_election.cc:290] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 15 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:23:14.788878 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 81563.8us
I20260709 12:23:14.712126 23453 leader_election.cc:290] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:14.681061 23492 raft_consensus.cc:515] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:14.792814 17655 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:14.790273 17654 leader_election.cc:304] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b
W20260709 12:23:14.798418 18333 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:14.798200 23507 consensus_queue.cc:1048] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:23:14.800040 23490 raft_consensus.cc:2749] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:23:14.800575 18333 leader_election.cc:304] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:14.801445 23468 raft_consensus.cc:2804] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:23:14.801981 23468 raft_consensus.cc:697] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 13 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:23:14.802903 23468 consensus_queue.cc:237] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 11.8, Last appended by leader: 8, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:14.804414 23481 raft_consensus.cc:993] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b: : Instructing follower e2909cff1da44ad387f96a8ae3aa5afb to start an election
I20260709 12:23:14.804987 23481 raft_consensus.cc:2749] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:23:14.806269 23224 raft_consensus.cc:515] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:14.807009 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:14.807773 23401 raft_consensus.cc:1081] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 16 LEADER]: Signalling peer e2909cff1da44ad387f96a8ae3aa5afb to start an election
I20260709 12:23:14.808174 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:14.808966 18367 raft_consensus.cc:2393] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 16: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
W20260709 12:23:14.809159 23486 log.cc:927] Time spent T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.191s	user 0.001s	sys 0.000s
I20260709 12:23:14.809892 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:14.810647 18365 raft_consensus.cc:2393] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 14: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:23:14.811748 17727 leader_election.cc:304] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:23:14.807801 18363 raft_consensus.cc:2393] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 12: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:23:14.734877 23469 raft_consensus.cc:493] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
W20260709 12:23:14.829576 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 44399.8us
I20260709 12:23:14.837960 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:23:14.839222 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 49088.4us
I20260709 12:23:14.834299 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:14.841887 23506 raft_consensus.cc:2804] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Leader election won for term 13
W20260709 12:23:14.842128 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 258688us
I20260709 12:23:14.842752 23506 raft_consensus.cc:697] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:23:14.842964 17803 leader_election.cc:304] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:14.844441 17803 leader_election.cc:304] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:14.844615 23453 raft_consensus.cc:2749] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:23:14.844048 23506 consensus_queue.cc:237] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 11.8, Last appended by leader: 8, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:14.845260 23480 raft_consensus.cc:2804] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:23:14.845718 23480 raft_consensus.cc:493] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:23:14.846091 17803 leader_election.cc:304] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:14.846119 23480 raft_consensus.cc:3060] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:14.850934 17803 leader_election.cc:304] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:23:14.855974 18371 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59918 (request call id 1741) took 1677 ms. Trace:
I20260709 12:23:14.856642 18371 rpcz_store.cc:276] 0709 12:23:13.178372 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:13.775243 (+596871us) service_pool.cc:224] Handling call
0709 12:23:14.851933 (+1076690us) spinlock_profiling.cc:243] Waited 453 ms on lock 0x7b540034c1d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:23:14.855909 (+  3976us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1040800000}
I20260709 12:23:14.731299 23376 consensus_queue.cc:1059] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 11.10, Next index: 11, Last known committed idx: 9, Time since last communication: 0.099s
I20260709 12:23:14.872424 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:14.705873 23400 consensus_queue.cc:237] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.8, Last appended by leader: 8, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:23:14.877332 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 76822.8us
I20260709 12:23:14.880919 18333 leader_election.cc:304] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:14.828680 23445 raft_consensus.cc:493] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:23:14.831359 23427 raft_consensus.cc:2749] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:23:14.891047 18371 raft_consensus.cc:1275] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
W20260709 12:23:14.881927 17802 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.600s	user 0.000s	sys 0.008s
W20260709 12:23:14.892230 17802 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.610s	user 0.001s	sys 0.009s
I20260709 12:23:14.734697 17587 catalog_manager.cc:5697] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 8 to 10. New cstate: current_term: 10 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:23:14.831806 23469 raft_consensus.cc:3060] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:14.886075 17762 raft_consensus.cc:1240] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 11. Current term is 12. Ops: []
W20260709 12:23:14.892859 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 619782us
I20260709 12:23:14.895974 17837 raft_consensus.cc:1240] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 12. Current term is 13. Ops: []
I20260709 12:23:14.781684 23472 raft_consensus.cc:515] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:14.927901 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 77102.9us
I20260709 12:23:14.929481 17803 leader_election.cc:304] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:23:14.854568 23293 raft_consensus.cc:2804] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Leader election won for term 14
W20260709 12:23:14.932909 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 44029.5us
I20260709 12:23:14.930812 17587 catalog_manager.cc:5697] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 6 to 10, leader changed from e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66) to 334a4e1f59944544b3b3ff36616b57da (127.16.29.68). New cstate: current_term: 10 leader_uuid: "334a4e1f59944544b3b3ff36616b57da" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:23:14.899205 18333 leader_election.cc:304] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
W20260709 12:23:14.936553 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 124885us
I20260709 12:23:14.938149 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:14.934743 17587 catalog_manager.cc:5697] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 6 to 9, leader changed from 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) to 0da169ff82674dc1947055ce43de68a3 (127.16.29.65). New cstate: current_term: 9 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:23:14.936851 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:14.969151 17727 leader_election.cc:304] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:14.896854 23445 raft_consensus.cc:515] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:14.898999 23453 raft_consensus.cc:2804] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Leader election won for term 12
W20260709 12:23:14.932147 23483 log.cc:927] Time spent T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.173s	user 0.000s	sys 0.001s
I20260709 12:23:14.976583 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:14.977068 18371 raft_consensus.cc:1275] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 16 index: 12. Preceding OpId from leader: term: 17 index: 11. (term mismatch)
I20260709 12:23:14.883306 23508 raft_consensus.cc:3060] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:14.899731 23456 raft_consensus.cc:2749] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:23:14.981721 18371 pending_rounds.cc:85] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da: Aborting all ops after (but not including) 10
I20260709 12:23:14.982151 18371 pending_rounds.cc:107] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da: Aborting uncommitted NO_OP operation due to leader change: 14.11
I20260709 12:23:14.982450 18371 raft_consensus.cc:2889] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:14.984359 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:14.988709 18371 pending_rounds.cc:107] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da: Aborting uncommitted NO_OP operation due to leader change: 16.12
I20260709 12:23:14.989156 18371 raft_consensus.cc:2889] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:23:14.990204 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 99359.7us
I20260709 12:23:14.991183 18333 leader_election.cc:304] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
W20260709 12:23:14.993824 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 64447.9us
I20260709 12:23:14.994470 17803 leader_election.cc:304] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:14.973299 23293 raft_consensus.cc:697] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 14 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
W20260709 12:23:15.021884 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 83972.5us
I20260709 12:23:14.978030 23488 leader_election.cc:290] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:15.024914 18368 raft_consensus.cc:3060] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:15.027522 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
W20260709 12:23:15.031657 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 524675us
W20260709 12:23:15.063750 23500 log.cc:927] Time spent T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.282s	user 0.001s	sys 0.000s
I20260709 12:23:15.066011 18364 raft_consensus.cc:2468] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 15.
I20260709 12:23:15.068360 23297 raft_consensus.cc:493] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:15.023741 23462 log.cc:927] Time spent T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.061s	user 0.001s	sys 0.000s
I20260709 12:23:15.068428 17728 leader_election.cc:304] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:15.061928 23498 raft_consensus.cc:493] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:23:15.018090 23224 leader_election.cc:290] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 16 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:23:15.031629 23487 log.cc:927] Time spent T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.334s	user 0.000s	sys 0.002s
I20260709 12:23:15.116442 17681 raft_consensus.cc:1240] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 15. Current term is 16. Ops: []
I20260709 12:23:15.116683 17684 raft_consensus.cc:1240] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 12 LEADER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 11. Current term is 12. Ops: []
W20260709 12:23:15.118898 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 50563.4us
I20260709 12:23:15.121749 17728 leader_election.cc:304] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:15.123204 17728 leader_election.cc:304] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:23:15.124172 23490 raft_consensus.cc:2749] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:23:15.127208 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:15.139052 17762 raft_consensus.cc:2468] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 12.
W20260709 12:23:15.140219 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:15.156092 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:15.082676 23456 raft_consensus.cc:2804] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:23:15.129951 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:15.161705 17685 raft_consensus.cc:3060] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:23:15.171257 17801 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.505s	user 0.005s	sys 0.001s
I20260709 12:23:15.131541 23056 raft_consensus.cc:2749] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
W20260709 12:23:15.172787 17801 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.507s	user 0.006s	sys 0.001s
W20260709 12:23:15.175400 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 509500us
I20260709 12:23:15.176964 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 17 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:15.156965 23490 raft_consensus.cc:2804] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Leader pre-election won for term 17
W20260709 12:23:15.183820 17655 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:15.118031 23504 consensus_queue.cc:1048] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.299s
I20260709 12:23:15.212939 17728 leader_election.cc:304] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:23:15.129191 23511 raft_consensus.cc:2804] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:23:15.116894 23297 raft_consensus.cc:515] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:15.166946 23224 raft_consensus.cc:2749] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 12:23:15.179777 18368 raft_consensus.cc:2468] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 13.
W20260709 12:23:15.208487 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 214049us
W20260709 12:23:15.208925 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 268599us
W20260709 12:23:15.209432 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 270420us
I20260709 12:23:15.207231 17685 raft_consensus.cc:2468] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 15.
W20260709 12:23:15.221166 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 230071us
I20260709 12:23:15.221431 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:15.223789 17728 leader_election.cc:304] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:15.164963 23456 raft_consensus.cc:493] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:23:15.226830 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 16 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:15.234687 18333 leader_election.cc:304] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:15.212702 23490 raft_consensus.cc:493] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:23:15.127979 23498 raft_consensus.cc:515] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:15.197183 17757 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "22fa38faae58422e9cecc99973cab688"
dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
 from {username='slave'} at 127.0.0.1:38786
I20260709 12:23:15.245772 17757 raft_consensus.cc:493] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:23:15.246201 17757 raft_consensus.cc:3060] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:15.254210 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:15.226464 23511 raft_consensus.cc:493] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:15.263824 23445 leader_election.cc:290] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:15.244755 23488 raft_consensus.cc:2804] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:23:15.255326 23506 raft_consensus.cc:2749] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
W20260709 12:23:15.265252 23496 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.509s	user 0.007s	sys 0.000s
W20260709 12:23:15.265669 23496 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.577s	user 0.007s	sys 0.000s
I20260709 12:23:15.243028 23456 raft_consensus.cc:3060] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:15.266637 23521 consensus_queue.cc:1059] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 11.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.054s
I20260709 12:23:15.267306 23493 raft_consensus.cc:493] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:15.267819 23493 raft_consensus.cc:515] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:15.270076 23493 leader_election.cc:290] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:15.264752 23525 raft_consensus.cc:493] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:15.272635 23519 raft_consensus.cc:2749] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:23:15.264030 23513 consensus_queue.cc:1059] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 13.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.061s
I20260709 12:23:15.275008 23513 raft_consensus.cc:3055] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:23:15.275420 23513 raft_consensus.cc:740] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 13 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:23:15.276394 23513 consensus_queue.cc:260] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.10, Last appended by leader: 10, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:15.277704 23513 raft_consensus.cc:3060] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:15.278445 23485 raft_consensus.cc:2749] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:23:15.264815 23488 raft_consensus.cc:493] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
W20260709 12:23:15.280390 23512 log.cc:927] Time spent T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.207s	user 0.001s	sys 0.000s
I20260709 12:23:15.280344 23488 raft_consensus.cc:3060] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:15.272277 23525 raft_consensus.cc:515] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:15.284382 23403 consensus_meta.cc:287] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.825s	user 0.004s	sys 0.007s
W20260709 12:23:15.284621 23516 log.cc:927] Time spent T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.170s	user 0.000s	sys 0.001s
I20260709 12:23:15.257746 23511 raft_consensus.cc:3060] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:15.261909 17728 leader_election.cc:304] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
W20260709 12:23:15.289909 17803 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:15.284870 23403 raft_consensus.cc:515] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:15.285830 23525 leader_election.cc:290] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:23:15.306561 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 74681.8us
I20260709 12:23:15.243577 23490 raft_consensus.cc:3060] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:15.303040 17757 raft_consensus.cc:515] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:15.312463 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:23:15.286609 23495 log.cc:927] Time spent T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.340s	user 0.000s	sys 0.001s
I20260709 12:23:15.320349 17833 raft_consensus.cc:1275] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 13 index: 10. Preceding OpId from leader: term: 17 index: 11. (index mismatch)
I20260709 12:23:15.317423 17832 raft_consensus.cc:2468] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 16.
I20260709 12:23:15.202952 23293 consensus_queue.cc:237] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 12.13, Last appended by leader: 13, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:15.312882 17803 connection.cc:256] RPC call timeout handler was delayed by -1.1862758709999071s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2102s for a response
I20260709 12:23:15.312829 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:15.245355 23224 consensus_queue.cc:1059] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.066s
I20260709 12:23:15.341758 17757 leader_election.cc:290] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 17 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:15.310676 23445 raft_consensus.cc:2749] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:23:15.320791 23506 raft_consensus.cc:2749] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:23:15.303324 23472 leader_election.cc:290] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:23:15.350047 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 88228.5us
I20260709 12:23:15.415702 17755 raft_consensus.cc:1240] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 16. Current term is 17. Ops: []
I20260709 12:23:15.415813 17728 leader_election.cc:304] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:23:15.361693 23456 raft_consensus.cc:515] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:15.363463 23535 log.cc:927] Time spent T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.061s	user 0.000s	sys 0.000s
I20260709 12:23:15.328433 23502 consensus_queue.cc:1059] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 15.10, Next index: 11, Last known committed idx: 9, Time since last communication: 0.240s
I20260709 12:23:15.368575 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:15.369236 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:23:15.371230 23454 consensus_meta.cc:287] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 1.034s	user 0.009s	sys 0.000s
W20260709 12:23:15.414714 23434 consensus_meta.cc:287] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.732s	user 0.010s	sys 0.000s
I20260709 12:23:15.332962 23521 consensus_queue.cc:1243] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer 0da169ff82674dc1947055ce43de68a3 log is divergent from this leader: its last log entry 12.11 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 10
I20260709 12:23:15.372473 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:15.373519 23403 leader_election.cc:290] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 12 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:15.373533 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:23:15.424463 23378 consensus_meta.cc:287] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.594s	user 0.006s	sys 0.003s
I20260709 12:23:15.401311 23224 raft_consensus.cc:493] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:15.386214 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:15.386921 23469 raft_consensus.cc:515] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:15.393249 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 17 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:23:15.440675 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 207647us
I20260709 12:23:15.395282 18363 raft_consensus.cc:1275] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 12 index: 10. (index mismatch)
I20260709 12:23:15.396571 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:15.311395 23530 consensus_queue.cc:1059] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 15.13, Next index: 14, Last known committed idx: 10, Time since last communication: 0.102s
I20260709 12:23:15.452427 17757 raft_consensus.cc:2393] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 18: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:23:15.401295 23437 raft_consensus.cc:493] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
W20260709 12:23:15.402025 23514 log.cc:927] Time spent T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.284s	user 0.000s	sys 0.001s
I20260709 12:23:15.405078 23498 leader_election.cc:290] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:15.395577 17729 leader_election.cc:304] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
W20260709 12:23:15.462530 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 67037.1us
I20260709 12:23:15.464301 23506 raft_consensus.cc:2749] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 12:23:15.465236 23469 leader_election.cc:290] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 16 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:15.474087 23351 consensus_queue.cc:260] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.10, Last appended by leader: 10, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:15.475435 23351 raft_consensus.cc:3060] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:15.477135 23456 leader_election.cc:290] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 12 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:15.477782 23437 raft_consensus.cc:3060] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:15.484297 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 17 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:23:15.483948 17803 leader_election.cc:336] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Timed out: RequestConsensusVote RPC to 127.16.29.68:42577 timed out after 2.102s (SENT)
I20260709 12:23:15.409371 23509 raft_consensus.cc:3055] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:23:15.485527 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:23:15.437004 23461 consensus_meta.cc:287] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 1.098s	user 0.012s	sys 0.000s
W20260709 12:23:15.367671 23446 consensus_meta.cc:287] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.892s	user 0.007s	sys 0.000s
W20260709 12:23:15.419173 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 102024us
I20260709 12:23:15.415120 17763 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38786 (request call id 3168) took 1191 ms. Trace:
I20260709 12:23:15.486968 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:15.487315 17763 rpcz_store.cc:276] 0709 12:23:14.223819 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:14.223978 (+   159us) service_pool.cc:224] Handling call
0709 12:23:14.224287 (+   309us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:14.294045 (+ 69758us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:14.294079 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:14.321125 (+ 27046us) log.cc:844] Serialized 26 byte log entry
0709 12:23:14.514969 (+193844us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:23:14.515018 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:14.515091 (+    73us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:15.414886 (+899795us) raft_consensus.cc:1685] finished
0709 12:23:15.414908 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:15.415062 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":46419,"raft.queue_time_us":148985,"raft.run_cpu_time_us":199,"raft.run_wall_time_us":86471,"thread_start_us":155548,"threads_started":1,"wal-append.queue_time_us":272929}
I20260709 12:23:15.372356 23297 leader_election.cc:290] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:15.431224 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:15.432351 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:15.429728 17728 leader_election.cc:304] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:23:15.363826 23246 consensus_meta.cc:287] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.987s	user 0.006s	sys 0.002s
I20260709 12:23:15.385861 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 14 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:15.496235 17764 raft_consensus.cc:2410] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for term 14 because replica has last-logged OpId of term: 13 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 9.
I20260709 12:23:15.496026 23446 raft_consensus.cc:515] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:15.425143 23056 raft_consensus.cc:2804] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:23:15.496975 23056 raft_consensus.cc:493] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:23:15.497378 23056 raft_consensus.cc:3060] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:15.498163 23446 leader_election.cc:290] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 12 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:15.498734 23469 raft_consensus.cc:2804] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:23:15.499287 23469 raft_consensus.cc:697] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 12 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:23:15.500164 23469 consensus_queue.cc:237] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:15.500972 23434 consensus_queue.cc:1059] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 12.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 12:23:15.436076 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:15.501968 23493 raft_consensus.cc:3055] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 12 LEADER]: Stepping down as leader of term 12
W20260709 12:23:15.443027 17801 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:15.502332 23493 raft_consensus.cc:740] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
W20260709 12:23:15.436235 23475 consensus_meta.cc:287] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.928s	user 0.014s	sys 0.000s
I20260709 12:23:15.440204 23378 raft_consensus.cc:515] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:15.408303 23395 leader_election.cc:290] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 13 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:15.515709 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:23:15.389876 23480 consensus_meta.cc:287] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.543s	user 0.011s	sys 0.000s
I20260709 12:23:15.518903 23480 raft_consensus.cc:515] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:15.524802 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:15.491430 17803 leader_election.cc:304] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:15.525947 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:15.509687 23493 consensus_queue.cc:260] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 12.12, Last appended by leader: 12, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:15.526625 18368 raft_consensus.cc:2468] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 14.
I20260709 12:23:15.527740 18374 raft_consensus.cc:1240] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 11. Current term is 12. Ops: []
I20260709 12:23:15.527871 23480 leader_election.cc:290] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:23:15.495088 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 65464.3us
I20260709 12:23:15.510509 18334 leader_election.cc:304] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
W20260709 12:23:15.541509 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 306945us
I20260709 12:23:15.544262 17729 leader_election.cc:304] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:15.548777 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:23:15.559759 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 53898.7us
I20260709 12:23:15.482093 23518 raft_consensus.cc:493] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:23:15.488200 23509 raft_consensus.cc:740] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 15 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:23:15.560737 17652 leader_election.cc:304] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
W20260709 12:23:15.572566 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 84078.1us
W20260709 12:23:15.573376 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 62952.5us
W20260709 12:23:15.590368 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 87087.8us
I20260709 12:23:15.512562 23508 raft_consensus.cc:2749] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:23:15.590849 17729 leader_election.cc:304] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:15.454663 23224 raft_consensus.cc:515] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:23:15.559386 23537 log.cc:927] Time spent T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.101s	user 0.001s	sys 0.000s
I20260709 12:23:15.556061 23475 raft_consensus.cc:515] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:15.629810 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 69195.1us
W20260709 12:23:15.654004 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 111076us
I20260709 12:23:15.654855 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:15.655499 18333 leader_election.cc:304] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:15.657899 17836 raft_consensus.cc:2468] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 14.
I20260709 12:23:15.603144 23446 raft_consensus.cc:2804] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:23:15.591986 23374 raft_consensus.cc:493] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:23:15.673241 18333 leader_election.cc:304] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:15.586375 23434 raft_consensus.cc:493] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:23:15.670867 17762 raft_consensus.cc:1275] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
I20260709 12:23:15.628472 23480 consensus_queue.cc:1048] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.067s
W20260709 12:23:15.678442 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 85260.8us
W20260709 12:23:15.680934 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 197023us
I20260709 12:23:15.681813 23491 raft_consensus.cc:2749] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
W20260709 12:23:15.682034 17803 leader_election.cc:336] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Timed out: RequestConsensusVote RPC to 127.16.29.68:42577 timed out after 2.217s (SENT)
I20260709 12:23:15.682765 17803 leader_election.cc:304] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:15.682945 23395 consensus_queue.cc:1059] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20260709 12:23:15.683723 17760 raft_consensus.cc:2393] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 15: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:23:15.506294 23461 raft_consensus.cc:515] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:15.673523 23488 raft_consensus.cc:515] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:15.685811 23461 leader_election.cc:290] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 17 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:15.656718 23266 raft_consensus.cc:2804] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:23:15.688257 18361 raft_consensus.cc:1240] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 16. Current term is 17. Ops: [16.10-16.10]
I20260709 12:23:15.628242 23246 raft_consensus.cc:493] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:23:15.652675 23506 raft_consensus.cc:2749] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
W20260709 12:23:15.679100 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 88342.5us
I20260709 12:23:15.539445 23056 raft_consensus.cc:515] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:23:15.705955 17729 leader_election.cc:304] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:15.703498 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:15.707086 18333 leader_election.cc:304] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:15.707993 17762 raft_consensus.cc:1240] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 16. Current term is 17. Ops: []
I20260709 12:23:15.709512 23544 raft_consensus.cc:493] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:15.710065 23544 raft_consensus.cc:515] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:15.712035 23412 raft_consensus.cc:2749] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
W20260709 12:23:15.714178 17803 leader_election.cc:336] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 16 pre-election: RPC error from VoteRequest() call to peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Timed out: RequestConsensusVote RPC to 127.16.29.68:42577 timed out after 2.284s (SENT)
I20260709 12:23:15.714700 17803 leader_election.cc:304] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:15.715754 23551 raft_consensus.cc:2749] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 12:23:15.716073 23454 raft_consensus.cc:2749] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:23:15.716742 23526 raft_consensus.cc:2749] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:23:15.717761 23305 consensus_queue.cc:1059] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 13.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 12:23:15.718842 23305 raft_consensus.cc:3055] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:23:15.719229 23305 raft_consensus.cc:740] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 13 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:23:15.718966 23528 consensus_queue.cc:1048] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:23:15.600620 23518 raft_consensus.cc:515] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:15.720584 23472 raft_consensus.cc:3055] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:23:15.720978 23472 raft_consensus.cc:740] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 13 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:23:15.721853 23472 consensus_queue.cc:260] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 13.9, Last appended by leader: 9, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:15.722148 23475 leader_election.cc:290] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 15 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:15.722990 23472 raft_consensus.cc:3060] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:15.725853 23509 consensus_queue.cc:260] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.12, Last appended by leader: 12, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:15.701603 23547 raft_consensus.cc:2749] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:23:15.730057 23452 consensus_queue.cc:1048] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 12:23:15.701887 23434 raft_consensus.cc:515] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
W20260709 12:23:15.732390 17802 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:15.735098 23434 leader_election.cc:290] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:15.738240 23378 leader_election.cc:290] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 13 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:15.701818 23446 raft_consensus.cc:697] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 15 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:23:15.740533 23446 consensus_queue.cc:237] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 13.12, Last appended by leader: 12, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:15.743235 17837 raft_consensus.cc:1240] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 15. Current term is 16. Ops: []
W20260709 12:23:15.715839 17803 leader_election.cc:336] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 14 pre-election: RPC error from VoteRequest() call to peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Timed out: RequestConsensusVote RPC to 127.16.29.68:42577 timed out after 2.304s (SENT)
I20260709 12:23:15.689687 23488 leader_election.cc:290] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 15 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:15.746815 23525 raft_consensus.cc:3055] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:23:15.747359 23525 raft_consensus.cc:740] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:23:15.748327 17838 raft_consensus.cc:1240] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 11. Current term is 12. Ops: []
I20260709 12:23:15.749274 23509 raft_consensus.cc:3060] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:15.755856 17835 raft_consensus.cc:1240] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 12. Current term is 13. Ops: []
W20260709 12:23:15.758229 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:15.768708 17830 raft_consensus.cc:1240] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 9. Current term is 10. Ops: []
I20260709 12:23:15.770558 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
W20260709 12:23:15.771291 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 51160.2us
I20260709 12:23:15.765748 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:15.775282 17832 raft_consensus.cc:1240] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 15. Current term is 16. Ops: [15.10-15.10]
W20260709 12:23:15.678328 23453 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.614s	user 0.008s	sys 0.063s
I20260709 12:23:15.777204 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:15.781862 17762 raft_consensus.cc:2468] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 12.
I20260709 12:23:15.778432 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:15.782744 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: true dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:15.783568 17831 raft_consensus.cc:3055] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:23:15.784981 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:15.746750 23056 leader_election.cc:290] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 15 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:23:15.786535 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 79486.2us
I20260709 12:23:15.787348 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:15.788314 17832 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d"
dest_uuid: "e063541542a94969898868bf6eb5ce1b"
mode: GRACEFUL
new_leader_uuid: "0da169ff82674dc1947055ce43de68a3"
 from {username='slave'} at 127.0.0.1:59820
W20260709 12:23:15.788731 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 133954us
W20260709 12:23:15.794902 23458 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.727s	user 0.005s	sys 0.014s
I20260709 12:23:15.795083 17832 raft_consensus.cc:606] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Received request to transfer leadership to TS 0da169ff82674dc1947055ce43de68a3
W20260709 12:23:15.795475 23458 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.728s	user 0.005s	sys 0.015s
I20260709 12:23:15.795547 17832 raft_consensus.cc:614] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:23:15.783607 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:15.789685 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:15.797318 17833 raft_consensus.cc:2468] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 13.
W20260709 12:23:15.792091 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 76204us
W20260709 12:23:15.798398 17803 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:15.784116 17831 raft_consensus.cc:740] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 16 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:23:15.792500 17829 raft_consensus.cc:1240] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 15. Current term is 16. Ops: []
I20260709 12:23:15.798206 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:15.703786 23266 raft_consensus.cc:493] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:15.722298 23518 leader_election.cc:290] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:15.802457 17757 raft_consensus.cc:1275] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
W20260709 12:23:15.804215 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 98348.4us
W20260709 12:23:15.781535 23453 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.751s	user 0.009s	sys 0.063s
I20260709 12:23:15.806104 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:15.778904 23469 consensus_queue.cc:1059] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 13.11, Next index: 12, Last known committed idx: 10, Time since last communication: 0.028s
I20260709 12:23:15.688684 23374 raft_consensus.cc:515] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:15.808457 23490 raft_consensus.cc:515] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:15.799274 17831 consensus_queue.cc:260] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.9, Last appended by leader: 9, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:15.809581 17764 raft_consensus.cc:1240] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 12. Current term is 13. Ops: []
I20260709 12:23:15.811631 17729 leader_election.cc:304] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:23:15.814289 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:15.801679 23305 consensus_queue.cc:260] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 13.14, Last appended by leader: 14, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:15.814743 23519 raft_consensus.cc:3055] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:23:15.690464 23504 consensus_queue.cc:1048] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.113s
I20260709 12:23:15.815395 23497 raft_consensus.cc:3055] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:23:15.815889 23494 consensus_queue.cc:1059] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 15.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
W20260709 12:23:15.816250 23511 consensus_meta.cc:287] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.501s	user 0.010s	sys 0.000s
W20260709 12:23:15.817648 23513 consensus_meta.cc:287] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.539s	user 0.008s	sys 0.004s
I20260709 12:23:15.818867 23502 consensus_queue.cc:1059] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 11.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:23:15.818924 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:15.819660 23488 consensus_queue.cc:1059] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 12:23:15.822680 23437 raft_consensus.cc:515] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:15.822898 23475 consensus_queue.cc:1059] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 9.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:23:15.774979 23491 raft_consensus.cc:2749] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:23:15.825584 23451 raft_consensus.cc:3060] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:15.828097 23372 leader_election.cc:290] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 14 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:23:15.751040 23395 consensus_queue.cc:1175] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "0da169ff82674dc1947055ce43de68a3" responder_term: 16 status { last_received { term: 14 index: 11 } last_committed_idx: 11 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 15. Current term is 16. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260709 12:23:15.830566 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:15.831584 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:15.785202 23547 raft_consensus.cc:2749] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:23:15.945870 17763 raft_consensus.cc:2393] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 15: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:23:15.836982 23224 leader_election.cc:290] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:23:15.791785 23537 log.cc:927] Time spent T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.094s	user 0.000s	sys 0.000s
I20260709 12:23:15.809634 23525 consensus_queue.cc:260] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:15.754769 23246 raft_consensus.cc:515] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:23:15.846843 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 52182.9us
I20260709 12:23:15.848177 17681 raft_consensus.cc:1240] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 13. Current term is 14. Ops: []
I20260709 12:23:15.956295 18333 leader_election.cc:304] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:23:15.957482 18333 leader_election.cc:304] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:23:15.958293 23427 raft_consensus.cc:2804] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:23:15.958441 18333 leader_election.cc:304] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:15.848260 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 17 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:15.849443 23493 raft_consensus.cc:3060] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:15.849274 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:15.849956 23523 raft_consensus.cc:493] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:15.850587 23544 leader_election.cc:290] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:23:15.853312 23515 log.cc:927] Time spent T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.784s	user 0.000s	sys 0.004s
I20260709 12:23:15.837038 17755 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36572 (request call id 945) took 1064 ms. Trace:
I20260709 12:23:15.857831 17727 leader_election.cc:304] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:15.888005 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:15.892153 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 14 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:15.893455 17689 raft_consensus.cc:1240] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 12. Current term is 13. Ops: []
I20260709 12:23:15.897950 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:15.902148 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 14 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:15.912128 17688 raft_consensus.cc:1275] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 15 index: 9. Preceding OpId from leader: term: 16 index: 9. (term mismatch)
W20260709 12:23:15.913574 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 102016us
I20260709 12:23:15.882825 23453 raft_consensus.cc:697] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 12 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:23:15.931150 17756 raft_consensus.cc:1240] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 16. Current term is 17. Ops: []
I20260709 12:23:15.928879 17652 leader_election.cc:304] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:15.935086 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:15.927830 23056 raft_consensus.cc:2804] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:23:15.810410 23266 raft_consensus.cc:3060] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:15.906127 23519 raft_consensus.cc:740] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:23:15.920164 23497 raft_consensus.cc:740] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:23:15.941437 23305 raft_consensus.cc:3060] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Advancing to term 14
W20260709 12:23:15.996367 17801 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:15.995246 17729 leader_election.cc:304] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:23:15.983400 17755 rpcz_store.cc:276] 0709 12:23:14.772299 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:14.773362 (+  1063us) service_pool.cc:224] Handling call
0709 12:23:15.415519 (+642157us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:15.416219 (+   700us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:15.836790 (+420571us) spinlock_profiling.cc:243] Waited 628 ms on lock 0x7b54001b8350. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a61f8ad 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:23:15.836963 (+   173us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1443992448}
I20260709 12:23:15.984169 17830 raft_consensus.cc:1275] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 16 index: 12. (term mismatch)
I20260709 12:23:15.982669 23445 raft_consensus.cc:2749] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:23:15.997303 17830 pending_rounds.cc:85] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b: Aborting all ops after (but not including) 11
I20260709 12:23:15.997701 17830 pending_rounds.cc:107] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b: Aborting uncommitted NO_OP operation due to leader change: 15.12
I20260709 12:23:15.998030 17830 raft_consensus.cc:2889] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:15.904193 23526 consensus_queue.cc:1048] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.056s
W20260709 12:23:16.002435 23495 log.cc:927] Time spent T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.050s	user 0.000s	sys 0.000s
I20260709 12:23:16.003868 23490 leader_election.cc:290] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 17 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:16.004413 23497 consensus_queue.cc:260] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:16.006156 23497 raft_consensus.cc:3060] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:16.007642 23525 raft_consensus.cc:3060] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:15.991988 23456 raft_consensus.cc:2804] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:23:16.008626 23456 raft_consensus.cc:493] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:23:15.988165 17762 raft_consensus.cc:1240] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 15. Current term is 16. Ops: []
I20260709 12:23:16.009058 23456 raft_consensus.cc:3060] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:16.010172 17762 raft_consensus.cc:1240] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 14. Current term is 15. Ops: []
I20260709 12:23:15.992599 23056 raft_consensus.cc:493] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:23:16.012494 23056 raft_consensus.cc:3060] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:15.960548 18333 leader_election.cc:304] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:23:16.016661 23453 consensus_queue.cc:237] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.11, Last appended by leader: 11, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:15.937189 23511 raft_consensus.cc:515] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:16.019043 23351 consensus_meta.cc:287] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.543s	user 0.008s	sys 0.004s
I20260709 12:23:16.000931 17835 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad"
dest_uuid: "e063541542a94969898868bf6eb5ce1b"
mode: GRACEFUL
new_leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
 from {username='slave'} at 127.0.0.1:59820
I20260709 12:23:15.999225 17688 pending_rounds.cc:85] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3: Aborting all ops after (but not including) 8
I20260709 12:23:16.020092 17688 pending_rounds.cc:107] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3: Aborting uncommitted NO_OP operation due to leader change: 15.9
I20260709 12:23:16.020413 17688 raft_consensus.cc:2889] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:16.020561 17758 raft_consensus.cc:2410] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 17 because replica has last-logged OpId of term: 15 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 9.
W20260709 12:23:15.993605 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 64817.6us
I20260709 12:23:16.019762 17835 raft_consensus.cc:606] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 16 LEADER]: Received request to transfer leadership to TS e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:16.002142 23492 leader_election.cc:290] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:15.986688 17757 raft_consensus.cc:1240] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 14. Current term is 15. Ops: []
I20260709 12:23:16.024111 23547 raft_consensus.cc:2804] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:23:16.024708 23547 raft_consensus.cc:697] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 16 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:23:15.982329 23523 raft_consensus.cc:515] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:16.025803 23547 consensus_queue.cc:237] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 15.13, Last appended by leader: 13, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:16.026757 23523 leader_election.cc:290] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:16.027472 23523 consensus_queue.cc:1059] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 12.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 12:23:16.028661 23528 raft_consensus.cc:3055] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:23:16.029059 23528 raft_consensus.cc:740] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:23:16.029114 23513 raft_consensus.cc:3055] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:23:16.000727 23519 consensus_queue.cc:260] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 15.13, Last appended by leader: 13, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:16.029651 23513 raft_consensus.cc:740] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 11 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:23:16.030083 23519 raft_consensus.cc:3060] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:16.030076 23528 consensus_queue.cc:260] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 12.9, Last appended by leader: 9, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:16.030573 23513 consensus_queue.cc:260] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 11.8, Last appended by leader: 8, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:16.001547 23491 raft_consensus.cc:2804] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:23:16.033047 23491 raft_consensus.cc:493] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:23:16.033489 23491 raft_consensus.cc:3060] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:16.034123 17729 leader_election.cc:304] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:23:16.000768 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 142998us
W20260709 12:23:16.035665 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 75159.5us
I20260709 12:23:16.036537 18333 leader_election.cc:304] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:16.000715 23468 raft_consensus.cc:2804] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:23:16.037559 23468 raft_consensus.cc:493] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:16.037928 23468 raft_consensus.cc:3060] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:16.039446 18333 leader_election.cc:304] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:23:15.996519 17684 raft_consensus.cc:2468] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 13.
I20260709 12:23:16.017339 23246 leader_election.cc:290] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:16.045994 17761 raft_consensus.cc:2468] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 14.
I20260709 12:23:16.048486 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:15.958904 23427 raft_consensus.cc:697] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 14 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:23:16.061474 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:16.070236 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 19 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:23:16.070847 17652 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:16.070905 17762 raft_consensus.cc:2468] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 18.
I20260709 12:23:16.072909 17755 raft_consensus.cc:1275] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 14 index: 8. Preceding OpId from leader: term: 16 index: 9. (index mismatch)
I20260709 12:23:16.073370 18333 leader_election.cc:304] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:16.023453 23497 consensus_queue.cc:1048] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.017s
I20260709 12:23:16.075220 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:16.077378 17652 leader_election.cc:304] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:16.078671 23572 consensus_queue.cc:1059] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 15.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20260709 12:23:16.079375 23472 consensus_queue.cc:1059] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 13.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:23:16.080087 17802 leader_election.cc:304] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:23:16.080299 23403 consensus_queue.cc:1059] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 12.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:23:16.080710 23522 consensus_queue.cc:1048] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:23:15.964625 23488 consensus_queue.cc:1243] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer e063541542a94969898868bf6eb5ce1b log is divergent from this leader: its last log entry 11.10 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 8
I20260709 12:23:16.069841 23551 raft_consensus.cc:2804] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:23:16.057906 23458 raft_consensus.cc:2749] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 12:23:16.060070 23506 raft_consensus.cc:2804] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Leader pre-election won for term 12
W20260709 12:23:16.100900 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 446479us
W20260709 12:23:16.105166 17801 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:16.073149 23513 raft_consensus.cc:3060] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:16.073520 23502 raft_consensus.cc:2804] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:23:16.110638 17833 raft_consensus.cc:1275] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 12 index: 9. Preceding OpId from leader: term: 13 index: 9. (term mismatch)
I20260709 12:23:16.170352 17833 pending_rounds.cc:85] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b: Aborting all ops after (but not including) 8
I20260709 12:23:16.170609 17833 pending_rounds.cc:107] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b: Aborting uncommitted NO_OP operation due to leader change: 12.9
I20260709 12:23:16.170859 17833 raft_consensus.cc:2889] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:16.092554 17652 leader_election.cc:304] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
W20260709 12:23:16.172235 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 79799us
I20260709 12:23:16.172519 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:16.101383 23372 raft_consensus.cc:2749] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:23:16.060837 23351 consensus_queue.cc:1059] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 15.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.046s
I20260709 12:23:16.165845 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 18 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:16.103142 23427 consensus_queue.cc:237] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.10, Last appended by leader: 10, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:16.109576 23551 raft_consensus.cc:493] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:23:16.073241 23530 consensus_queue.cc:1048] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.090s
I20260709 12:23:16.158594 23506 raft_consensus.cc:493] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:16.178205 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 12 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:16.092386 23461 raft_consensus.cc:2749] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
W20260709 12:23:16.169513 23516 log.cc:927] Time spent T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.080s	user 0.000s	sys 0.000s
I20260709 12:23:16.160356 17654 leader_election.cc:304] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:23:16.186875 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:16.177876 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:16.207042 17833 raft_consensus.cc:2393] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 12: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
W20260709 12:23:16.207342 23554 log.cc:927] Time spent T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.075s	user 0.000s	sys 0.000s
I20260709 12:23:16.174106 23403 raft_consensus.cc:3055] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:23:16.207899 23528 raft_consensus.cc:3060] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:16.207980 23403 raft_consensus.cc:740] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 12 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:23:16.209017 23403 consensus_queue.cc:260] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.10, Last appended by leader: 10, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:16.210188 23403 raft_consensus.cc:3060] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:16.181121 22853 raft_consensus.cc:493] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:23:16.212649 22853 raft_consensus.cc:515] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:16.213251 23546 raft_consensus.cc:2804] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:23:16.213719 23546 raft_consensus.cc:493] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:16.214046 23546 raft_consensus.cc:3060] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:16.217948 23496 raft_consensus.cc:3055] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:23:16.218457 23496 raft_consensus.cc:740] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
W20260709 12:23:16.199577 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 162246us
I20260709 12:23:16.092865 23446 consensus_queue.cc:1059] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 11.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.091s
I20260709 12:23:16.177191 23499 raft_consensus.cc:2804] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:23:16.178624 23502 raft_consensus.cc:697] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
W20260709 12:23:16.244585 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 84358.7us
I20260709 12:23:16.242272 23499 raft_consensus.cc:493] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:23:16.246430 23543 log.cc:927] Time spent T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.072s	user 0.000s	sys 0.000s
I20260709 12:23:16.195122 23551 raft_consensus.cc:3060] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:16.203852 23506 raft_consensus.cc:3060] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Advancing to term 12
W20260709 12:23:16.246129 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 187809us
I20260709 12:23:16.248271 23499 raft_consensus.cc:3060] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:16.255151 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 11 candidate_status { last_received { term: 10 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
W20260709 12:23:16.249017 23567 log.cc:927] Time spent T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.052s	user 0.000s	sys 0.000s
I20260709 12:23:16.220286 23555 consensus_queue.cc:1048] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.092s
I20260709 12:23:16.302989 23555 raft_consensus.cc:493] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:16.249366 23523 raft_consensus.cc:2804] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:23:16.304368 23523 raft_consensus.cc:697] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 13 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:23:16.305266 23523 consensus_queue.cc:237] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.10, Last appended by leader: 10, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:16.271981 23468 raft_consensus.cc:515] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:16.262137 23437 leader_election.cc:290] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 11 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:16.247505 23496 consensus_queue.cc:260] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:16.289507 23056 raft_consensus.cc:515] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:16.337694 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 84560.8us
I20260709 12:23:16.296623 23546 raft_consensus.cc:515] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:23:16.257572 23542 log.cc:927] Time spent T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.597s	user 0.002s	sys 0.002s
I20260709 12:23:16.303643 23555 raft_consensus.cc:515] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:16.308977 23502 consensus_queue.cc:237] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 14.12, Last appended by leader: 12, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:16.260613 16501 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 3
I20260709 12:23:16.310523 23520 raft_consensus.cc:493] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:16.311272 23485 raft_consensus.cc:3055] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 15 LEADER]: Stepping down as leader of term 15
W20260709 12:23:16.311959 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 48430.7us
I20260709 12:23:16.270856 23266 raft_consensus.cc:515] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:16.313944 23494 consensus_queue.cc:1048] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.003s
I20260709 12:23:16.307513 17837 raft_consensus.cc:2468] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 15.
I20260709 12:23:16.357347 23468 leader_election.cc:290] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 13 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:16.361398 23555 leader_election.cc:290] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:16.362622 23491 raft_consensus.cc:515] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:16.319116 23456 raft_consensus.cc:515] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:16.376677 23576 log.cc:927] Time spent T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.060s	user 0.000s	sys 0.000s
I20260709 12:23:16.378110 17759 raft_consensus.cc:1240] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 13. Current term is 14. Ops: []
I20260709 12:23:16.413632 23456 leader_election.cc:290] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 10 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:16.380401 23056 leader_election.cc:290] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 14 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:23:16.354584 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 169685us
W20260709 12:23:16.382682 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 133648us
I20260709 12:23:16.385988 16501 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:23:16.438123 17829 raft_consensus.cc:1240] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 15. Current term is 16. Ops: [15.12-15.12]
I20260709 12:23:16.439045 16501 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:23:16.441032 17652 leader_election.cc:304] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:16.394706 23485 raft_consensus.cc:740] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 15 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:23:16.442754 17652 leader_election.cc:304] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:16.442651 23485 consensus_queue.cc:260] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.12, Last appended by leader: 12, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:16.393725 17803 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:16.444221 23485 raft_consensus.cc:3060] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:16.444343 23573 raft_consensus.cc:493] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:23:16.445202 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:16.445204 23573 raft_consensus.cc:515] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:16.446696 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:16.447278 17838 raft_consensus.cc:2376] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 15. Re-sending same reply.
I20260709 12:23:16.397153 23520 raft_consensus.cc:515] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:16.447635 23573 leader_election.cc:290] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:16.447686 23468 raft_consensus.cc:3060] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:16.310940 23499 raft_consensus.cc:515] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:16.415496 23546 leader_election.cc:290] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 14 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:16.416823 23266 leader_election.cc:290] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 17 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:23:16.417416 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:16.419411 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:16.419822 23577 consensus_queue.cc:1059] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 14.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:23:16.420183 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:16.420812 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:16.423200 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 17 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:16.426653 23437 consensus_queue.cc:1048] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 12:23:16.428313 17831 raft_consensus.cc:3060] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:16.431810 23551 raft_consensus.cc:515] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:16.432003 23478 raft_consensus.cc:3055] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 LEADER]: Stepping down as leader of term 15
W20260709 12:23:16.414273 23529 log.cc:927] Time spent T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.756s	user 0.004s	sys 0.000s
I20260709 12:23:16.415328 17680 raft_consensus.cc:2393] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 17: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:23:16.432428 23525 raft_consensus.cc:3055] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:23:16.455762 17685 raft_consensus.cc:2468] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 13.
I20260709 12:23:16.438123 17836 raft_consensus.cc:1240] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 13. Current term is 14. Ops: []
W20260709 12:23:16.486804 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 52001.9us
I20260709 12:23:16.453325 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:23:16.478731 17803 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:16.483753 23504 raft_consensus.cc:2749] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
W20260709 12:23:16.439049 23566 log.cc:927] Time spent T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.184s	user 0.001s	sys 0.000s
I20260709 12:23:16.449437 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:16.356045 17654 leader_election.cc:304] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:23:16.490473 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:16.551393 23520 leader_election.cc:290] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:16.426549 23469 raft_consensus.cc:3055] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:23:16.553912 23551 leader_election.cc:290] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 16 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:16.497673 23478 raft_consensus.cc:740] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:23:16.421563 23506 raft_consensus.cc:515] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:16.451629 17652 leader_election.cc:304] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:23:16.566008 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:16.574968 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:16.575193 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:16.575690 18363 raft_consensus.cc:3060] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:16.577271 18366 raft_consensus.cc:1240] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 9. Current term is 10. Ops: []
I20260709 12:23:16.577584 18361 raft_consensus.cc:1275] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
I20260709 12:23:16.561764 17838 raft_consensus.cc:2468] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 10.
W20260709 12:23:16.567375 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 64756.3us
W20260709 12:23:16.567646 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 211681us
I20260709 12:23:16.566301 17755 raft_consensus.cc:2468] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 12.
I20260709 12:23:16.506628 23525 raft_consensus.cc:740] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
W20260709 12:23:16.581751 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 130209us
I20260709 12:23:16.583482 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:23:16.584419 17652 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:16.584736 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:16.585258 18362 raft_consensus.cc:3060] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:16.585178 17652 leader_election.cc:304] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:16.569464 23469 raft_consensus.cc:740] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:23:16.571210 23504 raft_consensus.cc:2749] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:23:16.592463 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:16.600536 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 17 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:16.601716 17689 raft_consensus.cc:2410] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for term 17 because replica has last-logged OpId of term: 15 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 12.
I20260709 12:23:16.600371 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:16.621670 17684 raft_consensus.cc:2393] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 14: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
W20260709 12:23:16.634970 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:16.625168 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:16.626399 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:16.636744 17688 raft_consensus.cc:2393] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 13: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:23:16.633132 18361 raft_consensus.cc:1275] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 11 index: 12. Preceding OpId from leader: term: 14 index: 14. (index mismatch)
I20260709 12:23:16.588948 23520 raft_consensus.cc:2749] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:23:16.623800 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:16.633546 17728 leader_election.cc:304] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:16.628382 23493 raft_consensus.cc:2804] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:23:16.642539 23351 raft_consensus.cc:2749] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:23:16.642839 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:16.643997 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:16.644126 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 19 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:16.644663 18367 raft_consensus.cc:2468] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 17.
I20260709 12:23:16.645005 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: true dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:16.645551 17687 raft_consensus.cc:3060] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:16.645648 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:16.646684 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 15 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:16.647181 18367 raft_consensus.cc:2468] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 14.
I20260709 12:23:16.642560 23493 raft_consensus.cc:493] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:16.648581 23493 raft_consensus.cc:3060] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:16.650660 23511 leader_election.cc:290] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 12 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:16.593024 23374 leader_election.cc:290] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:16.656193 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:16.657599 17684 raft_consensus.cc:2393] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 13: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:23:16.637923 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 11 candidate_status { last_received { term: 10 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:16.659052 18361 raft_consensus.cc:3060] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:16.639587 17728 leader_election.cc:304] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:16.640875 18374 raft_consensus.cc:1275] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 11 index: 8. Preceding OpId from leader: term: 13 index: 9. (index mismatch)
I20260709 12:23:16.641207 18366 raft_consensus.cc:1275] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
I20260709 12:23:16.624440 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:23:16.665886 18333 connection.cc:256] RPC call timeout handler was delayed by -1.2537731960005658s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.30200000000000005s for a response
W20260709 12:23:16.666658 18333 connection.cc:256] RPC call timeout handler was delayed by -1.0953731960003097s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.31960000000000005s for a response
W20260709 12:23:16.713617 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 74109.6us
W20260709 12:23:16.724047 17802 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:16.785068 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:16.799162 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
W20260709 12:23:16.882153 17761 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36572 (request call id 963) took 1512 ms (client timeout 1519 ms). Trace:
I20260709 12:23:16.799736 17760 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36572 (request call id 959) took 1304 ms. Trace:
I20260709 12:23:16.854503 23546 raft_consensus.cc:493] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
W20260709 12:23:16.882807 17761 rpcz_store.cc:269] 0709 12:23:15.369834 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:15.385967 (+ 16133us) service_pool.cc:224] Handling call
0709 12:23:16.881962 (+1495995us) spinlock_profiling.cc:243] Waited 500 ms on lock 0x7b54004150d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:23:16.882091 (+   129us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1150501760}
I20260709 12:23:16.883076 17760 rpcz_store.cc:276] 0709 12:23:15.368565 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:15.372129 (+  3564us) service_pool.cc:224] Handling call
0709 12:23:16.672625 (+1300496us) spinlock_profiling.cc:243] Waited 248 ms on lock 0x7b54000fc3d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:23:16.672786 (+   161us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":570150272}
I20260709 12:23:16.806974 17689 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 13.
I20260709 12:23:16.808028 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 14 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
W20260709 12:23:16.808657 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 172103us
I20260709 12:23:16.809616 18362 raft_consensus.cc:2468] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 14.
W20260709 12:23:16.814383 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 126591us
W20260709 12:23:16.824791 17801 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:16.669914 23351 raft_consensus.cc:3060] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:16.636152 23555 raft_consensus.cc:493] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:16.829766 17802 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:16.672349 23499 leader_election.cc:290] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 15 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:16.811115 23427 raft_consensus.cc:493] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:16.839602 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 14 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:16.836066 23528 raft_consensus.cc:493] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:16.830677 23520 raft_consensus.cc:493] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:16.836364 23257 raft_consensus.cc:493] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:16.855825 23586 log.cc:927] Time spent T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.243s	user 0.001s	sys 0.001s
I20260709 12:23:16.871397 17687 raft_consensus.cc:2468] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 17.
I20260709 12:23:16.857259 23588 raft_consensus.cc:493] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:16.875782 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:16.928841 18374 raft_consensus.cc:3060] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:16.671211 23478 consensus_queue.cc:260] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 15.13, Last appended by leader: 13, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:16.890193 17802 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:16.887710 18361 raft_consensus.cc:2468] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 11.
I20260709 12:23:16.872867 23506 leader_election.cc:290] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 12 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:16.892846 17682 raft_consensus.cc:3060] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:16.895355 23555 raft_consensus.cc:515] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:16.918027 17830 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35040 (request call id 2748) took 1112 ms. Trace:
I20260709 12:23:16.932760 17830 rpcz_store.cc:276] 0709 12:23:15.805186 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:15.805403 (+   217us) service_pool.cc:224] Handling call
0709 12:23:15.805701 (+   298us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:15.998671 (+192970us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:16.917770 (+919099us) spinlock_profiling.cc:243] Waited 156 ms on lock 0x7b54003345d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:23:16.917964 (+   194us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":359513344}
I20260709 12:23:16.934026 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:16.925691 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 17 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:16.935038 17687 raft_consensus.cc:3060] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:16.935111 17688 raft_consensus.cc:2393] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 12: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:23:16.895450 23481 raft_consensus.cc:493] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:16.937813 23555 leader_election.cc:290] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:23:16.954118 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:16.937930 23590 log.cc:927] Time spent T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.121s	user 0.001s	sys 0.000s
I20260709 12:23:16.940100 23499 consensus_queue.cc:1048] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 12:23:16.942071 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:16.932724 23588 raft_consensus.cc:515] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:16.960994 23528 raft_consensus.cc:515] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:16.961539 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:16.962242 17683 raft_consensus.cc:2393] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 15: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:23:16.962886 23588 leader_election.cc:290] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:16.927992 23551 consensus_queue.cc:1059] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 11.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.017s
W20260709 12:23:16.944483 23587 log.cc:927] Time spent T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.114s	user 0.001s	sys 0.000s
I20260709 12:23:16.922953 23546 raft_consensus.cc:515] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:16.948518 23475 raft_consensus.cc:3055] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 LEADER]: Stepping down as leader of term 12
W20260709 12:23:16.948625 23589 log.cc:927] Time spent T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.096s	user 0.000s	sys 0.000s
I20260709 12:23:16.946425 17682 raft_consensus.cc:2468] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 14.
I20260709 12:23:17.017163 17688 raft_consensus.cc:1240] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 13. Current term is 14. Ops: []
I20260709 12:23:17.018098 17686 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37402 (request call id 2901) took 1149 ms. Trace:
I20260709 12:23:17.018493 17686 rpcz_store.cc:276] 0709 12:23:15.868602 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:15.868748 (+   146us) service_pool.cc:224] Handling call
0709 12:23:15.869200 (+   452us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:15.901144 (+ 31944us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:15.901192 (+    48us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:16.420574 (+519382us) spinlock_profiling.cc:243] Waited 14.4 ms on lock 0x7b180027dc00. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b475012a6 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b48f19a11 00007f7b48f29372 00007f7b4a65ba01 00007f7b4a62043a
0709 12:23:16.421702 (+  1128us) spinlock_profiling.cc:243] Waited 7.16 ms on lock 0x7b2c00000c88. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b48f19a11 00007f7b48f29372 00007f7b4a65ba01 00007f7b4a62043a 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2
0709 12:23:16.422336 (+   634us) log.cc:844] Serialized 26 byte log entry
0709 12:23:16.569938 (+147602us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:23:16.582165 (+ 12227us) log.cc:844] Serialized 14 byte log entry
0709 12:23:16.583808 (+  1643us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:16.583910 (+   102us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:17.017911 (+434001us) raft_consensus.cc:1685] finished
0709 12:23:17.017929 (+    18us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:17.018041 (+   112us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":46296,"prepare.queue_time_us":12073,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":39,"raft.queue_time_us":38855,"raft.run_cpu_time_us":325,"raft.run_wall_time_us":24961,"spinlock_wait_cycles":49518208,"thread_start_us":146992,"threads_started":1,"wal-append.queue_time_us":376755}
W20260709 12:23:17.020153 17730 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:17.018802 17682 raft_consensus.cc:1275] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
I20260709 12:23:17.025933 17682 raft_consensus.cc:1240] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 13. Current term is 14. Ops: []
I20260709 12:23:16.846648 23469 consensus_queue.cc:260] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 15.13, Last appended by leader: 13, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:16.957567 18363 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 13.
W20260709 12:23:16.958062 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 98990us
W20260709 12:23:17.031076 17803 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:16.963164 23528 leader_election.cc:290] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:16.963975 17831 raft_consensus.cc:2468] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 17.
I20260709 12:23:17.038254 17831 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35042 (request call id 2455) took 1283 ms. Trace:
I20260709 12:23:16.970939 23530 raft_consensus.cc:493] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:16.957013 23481 raft_consensus.cc:515] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:16.974517 23468 raft_consensus.cc:2749] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:23:16.953290 23520 raft_consensus.cc:515] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:16.916994 23427 raft_consensus.cc:515] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:16.992331 17728 leader_election.cc:304] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:16.992901 18374 raft_consensus.cc:2468] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 15.
I20260709 12:23:17.006615 17687 raft_consensus.cc:2410] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for term 17 because replica has last-logged OpId of term: 16 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 9.
I20260709 12:23:17.015707 23475 raft_consensus.cc:740] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:23:16.951560 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 18 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:23:17.034132 17730 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:17.055042 17730 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:16.925360 23257 raft_consensus.cc:515] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:17.038726 17831 rpcz_store.cc:276] 0709 12:23:15.754752 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:15.754972 (+   220us) service_pool.cc:224] Handling call
0709 12:23:16.427957 (+672985us) spinlock_profiling.cc:243] Waited 512 ms on lock 0x7b58004ff4d0. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a57c0aa 00007f7b4a5fe447 00007f7b4a616fe7 00007f7b4a6187de 00007f7b4a62427c 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c
0709 12:23:17.038170 (+610213us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1177582336}
I20260709 12:23:16.933210 23544 consensus_queue.cc:1059] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 15.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.075s
W20260709 12:23:17.043733 17803 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:16.999847 23478 raft_consensus.cc:3060] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Advancing to term 16
W20260709 12:23:17.003389 23485 consensus_meta.cc:287] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.558s	user 0.021s	sys 0.001s
I20260709 12:23:17.048491 17683 raft_consensus.cc:1240] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 16. Current term is 17. Ops: []
W20260709 12:23:17.050582 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 58350.4us
I20260709 12:23:17.055963 17682 raft_consensus.cc:1240] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 14. Current term is 15. Ops: []
I20260709 12:23:17.064746 17687 raft_consensus.cc:1240] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 16. Current term is 17. Ops: []
I20260709 12:23:17.058727 17684 raft_consensus.cc:1240] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 15. Current term is 16. Ops: []
I20260709 12:23:17.040437 23494 raft_consensus.cc:493] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:17.008147 23531 consensus_queue.cc:1059] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 14.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.053s
I20260709 12:23:17.045411 23521 raft_consensus.cc:493] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:17.066147 17728 leader_election.cc:304] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:17.070078 17684 raft_consensus.cc:1240] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 12. Current term is 13. Ops: []
I20260709 12:23:17.067679 23544 raft_consensus.cc:3055] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:23:17.008893 23493 raft_consensus.cc:515] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:17.077865 17684 raft_consensus.cc:1275] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 13 index: 12. Preceding OpId from leader: term: 15 index: 13. (index mismatch)
I20260709 12:23:17.078616 17687 raft_consensus.cc:1240] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 12 LEADER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 11. Current term is 12. Ops: []
I20260709 12:23:17.056823 23588 raft_consensus.cc:2749] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 12:23:17.083698 23506 raft_consensus.cc:2749] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:23:17.086102 17728 leader_election.cc:304] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:17.046181 23530 raft_consensus.cc:515] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:17.103449 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 186849us
I20260709 12:23:17.104851 18333 leader_election.cc:304] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:17.106393 17728 leader_election.cc:304] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:17.109050 17728 leader_election.cc:304] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:17.078272 23544 raft_consensus.cc:740] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 15 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:23:17.111433 23506 raft_consensus.cc:2749] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:23:17.114917 23506 raft_consensus.cc:2749] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:23:17.118600 23546 leader_election.cc:290] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:17.075798 23475 consensus_queue.cc:260] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 12.10, Last appended by leader: 10, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:17.120011 23475 raft_consensus.cc:3060] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:17.123281 23520 leader_election.cc:290] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:17.125415 23257 leader_election.cc:290] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:17.080886 23521 raft_consensus.cc:515] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:17.086884 23494 raft_consensus.cc:515] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:17.134364 23544 consensus_queue.cc:260] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.8, Last appended by leader: 8, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:17.112682 23488 raft_consensus.cc:2749] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
W20260709 12:23:17.142488 17727 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:17.143095 17727 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:17.142771 23481 leader_election.cc:290] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:17.154222 17727 leader_election.cc:304] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:23:17.156611 17728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:17.158032 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 224134us
I20260709 12:23:17.165323 23454 consensus_queue.cc:1048] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:23:17.162758 17727 leader_election.cc:304] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:17.172328 17687 raft_consensus.cc:1275] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 10 index: 11. Preceding OpId from leader: term: 12 index: 12. (index mismatch)
I20260709 12:23:17.179399 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 15 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:17.179581 23456 raft_consensus.cc:2749] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:23:17.180253 17687 raft_consensus.cc:2468] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 14.
W20260709 12:23:17.180670 23600 log.cc:927] Time spent T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.075s	user 0.001s	sys 0.000s
I20260709 12:23:17.181424 23494 leader_election.cc:290] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:17.179783 23521 leader_election.cc:290] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:17.172384 23378 raft_consensus.cc:2804] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:23:17.182546 23378 raft_consensus.cc:697] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:23:17.183493 23378 consensus_queue.cc:237] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.9, Last appended by leader: 9, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:17.168111 23506 raft_consensus.cc:2804] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:23:17.188468 23506 raft_consensus.cc:493] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:23:17.189055 23506 raft_consensus.cc:3060] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:17.166925 23481 consensus_queue.cc:1059] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 16.12, Next index: 13, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:23:17.204548 23481 raft_consensus.cc:3055] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:23:17.205096 23481 raft_consensus.cc:740] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 16 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:23:17.206295 23481 consensus_queue.cc:260] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.12, Last appended by leader: 12, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:17.164187 23546 consensus_queue.cc:1059] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 14.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.045s
I20260709 12:23:17.209497 23546 raft_consensus.cc:3055] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:23:17.210466 23546 raft_consensus.cc:740] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 14 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:23:17.179502 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:23:17.221590 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 116812us
I20260709 12:23:17.158419 23490 consensus_queue.cc:1059] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.057s
I20260709 12:23:17.230894 23606 raft_consensus.cc:493] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:17.230950 23403 raft_consensus.cc:493] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:17.232126 23403 raft_consensus.cc:515] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:17.199604 23351 raft_consensus.cc:2749] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:23:17.247617 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:17.211820 23546 consensus_queue.cc:260] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.8, Last appended by leader: 8, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:23:17.248304 18368 raft_consensus.cc:2468] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 16.
I20260709 12:23:17.249114 23546 raft_consensus.cc:3060] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:17.250634 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 17 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:17.252193 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:17.252768 17832 raft_consensus.cc:3060] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:17.253764 17831 raft_consensus.cc:1240] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 11. Current term is 12. Ops: []
I20260709 12:23:17.228657 22853 leader_election.cc:290] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:17.245857 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:17.250763 23490 raft_consensus.cc:3055] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:23:17.254839 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 15 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:17.220301 23481 raft_consensus.cc:3060] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:17.240228 17728 leader_election.cc:304] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:17.257629 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:17.274358 17762 raft_consensus.cc:3060] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:17.254853 17761 raft_consensus.cc:1240] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 13. Current term is 14. Ops: []
I20260709 12:23:17.227684 18333 leader_election.cc:304] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:17.269856 17830 raft_consensus.cc:2468] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 14.
I20260709 12:23:17.256723 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
W20260709 12:23:17.245218 17654 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:17.268872 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:17.276403 17758 raft_consensus.cc:3060] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:17.276594 18367 raft_consensus.cc:2468] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 12.
W20260709 12:23:17.276007 17803 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:17.271829 22853 consensus_queue.cc:1059] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 11.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.006s
I20260709 12:23:17.278164 23456 consensus_queue.cc:1059] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 12:23:17.278745 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:17.279690 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:17.279983 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 17 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:17.281153 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:17.287147 17830 raft_consensus.cc:2468] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 14.
W20260709 12:23:17.288960 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 48884.2us
I20260709 12:23:17.288888 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:23:17.292138 17802 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:17.291790 17758 raft_consensus.cc:2468] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 13.
I20260709 12:23:17.294884 17829 raft_consensus.cc:2393] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 17: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:23:17.298954 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:17.299582 17834 raft_consensus.cc:3060] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:17.295356 17652 leader_election.cc:304] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:23:17.299167 17835 pending_rounds.cc:85] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b: Aborting all ops after (but not including) 13
I20260709 12:23:17.301709 17835 pending_rounds.cc:107] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b: Aborting uncommitted NO_OP operation due to leader change: 13.14
I20260709 12:23:17.302031 17835 raft_consensus.cc:2889] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:17.242028 23530 leader_election.cc:290] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:23:17.304855 17802 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:17.298133 23488 consensus_queue.cc:1048] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.010s
W20260709 12:23:17.309763 17802 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:17.268028 23490 raft_consensus.cc:740] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:23:17.314337 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:17.314817 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:17.315424 17758 raft_consensus.cc:2468] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 16.
I20260709 12:23:17.232062 23606 raft_consensus.cc:515] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:17.319394 23606 leader_election.cc:290] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:23:17.320034 17804 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:17.320305 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:17.321110 17831 raft_consensus.cc:2410] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for term 14 because replica has last-logged OpId of term: 12 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 7.
I20260709 12:23:17.286427 23427 leader_election.cc:290] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:17.295446 23351 raft_consensus.cc:2749] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 12:23:17.322111 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:17.340453 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 19 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:17.341090 18367 raft_consensus.cc:3060] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Advancing to term 19
I20260709 12:23:17.342094 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:17.347177 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:17.313851 23493 leader_election.cc:290] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 19 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:17.353308 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:17.354507 17758 raft_consensus.cc:2410] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for term 14 because replica has last-logged OpId of term: 12 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 7.
I20260709 12:23:17.355608 17832 raft_consensus.cc:2468] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 15.
I20260709 12:23:17.356302 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:17.357338 17762 raft_consensus.cc:2468] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 17.
I20260709 12:23:17.325160 23606 raft_consensus.cc:493] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
W20260709 12:23:17.418433 17729 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:17.357148 17652 leader_election.cc:304] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:23:17.338609 23530 raft_consensus.cc:2749] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:23:17.363328 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:17.425724 17729 leader_election.cc:304] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:17.425868 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 17 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:17.426229 18363 raft_consensus.cc:1240] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 16. Current term is 17. Ops: [16.10-16.10]
I20260709 12:23:17.350176 23470 raft_consensus.cc:2804] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:23:17.427546 23470 raft_consensus.cc:493] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:17.428079 23470 raft_consensus.cc:3060] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:17.428004 23526 raft_consensus.cc:3055] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:23:17.428864 23526 raft_consensus.cc:740] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 13 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:23:17.429751 17729 leader_election.cc:304] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:23:17.430114 23502 raft_consensus.cc:2749] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:23:17.430761 23573 consensus_queue.cc:1048] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 12:23:17.364555 23490 consensus_queue.cc:260] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.9, Last appended by leader: 9, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:17.405153 23403 leader_election.cc:290] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:17.342434 23608 consensus_queue.cc:1048] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.057s
I20260709 12:23:17.434216 17729 leader_election.cc:304] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:23:17.435422 23572 raft_consensus.cc:2804] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:23:17.435938 23572 raft_consensus.cc:697] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:23:17.436264 23506 raft_consensus.cc:515] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:17.437183 23572 consensus_queue.cc:237] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 12.9, Last appended by leader: 9, Current term: 13, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:17.438815 23506 leader_election.cc:290] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 11 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:17.441964 23490 raft_consensus.cc:3060] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:17.359006 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:23:17.447587 17730 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:17.360131 23351 consensus_queue.cc:1059] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 16.10, Next index: 11, Last known committed idx: 9, Time since last communication: 0.029s
I20260709 12:23:17.408325 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 19 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:17.457396 17758 raft_consensus.cc:3060] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Advancing to term 19
W20260709 12:23:17.408866 23611 log.cc:927] Time spent T 00000000000000000000000000000000 P 2193af31cb894ec9b3d65d939a2a6851: Append to log took a long time: real 0.057s	user 0.002s	sys 0.000s
I20260709 12:23:17.409538 17764 raft_consensus.cc:1275] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 16 index: 12. (index mismatch)
I20260709 12:23:17.411298 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:23:17.411345 17685 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37402 (request call id 2896) took 1564 ms (client timeout 2080 ms). Trace:
W20260709 12:23:17.460472 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 103397us
W20260709 12:23:17.460633 17685 rpcz_store.cc:269] 0709 12:23:15.847044 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:15.848987 (+  1943us) service_pool.cc:224] Handling call
0709 12:23:17.411199 (+1562212us) spinlock_profiling.cc:243] Waited 474 ms on lock 0x7b54000c4c54. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:23:17.411291 (+    92us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1090583936}
I20260709 12:23:17.412204 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:17.463012 17689 raft_consensus.cc:2468] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 14.
I20260709 12:23:17.463245 17652 leader_election.cc:304] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:17.413717 17759 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38786 (request call id 3189) took 1556 ms. Trace:
I20260709 12:23:17.417608 18365 raft_consensus.cc:1240] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 16 LEADER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 15. Current term is 16. Ops: []
I20260709 12:23:17.418380 23475 raft_consensus.cc:493] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:23:17.467152 17759 rpcz_store.cc:276] 0709 12:23:15.857667 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:16.012230 (+154563us) service_pool.cc:224] Handling call
0709 12:23:16.012583 (+   353us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:16.408043 (+395460us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:17.413503 (+1005460us) spinlock_profiling.cc:243] Waited 365 ms on lock 0x7b54000fe1d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:23:17.413660 (+   157us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":840176000}
I20260709 12:23:17.419515 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:17.420162 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:17.418195 23606 raft_consensus.cc:515] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:17.420763 23525 consensus_queue.cc:260] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:17.421638 23598 log.cc:927] Time spent T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.480s	user 0.002s	sys 0.000s
I20260709 12:23:17.431700 23502 raft_consensus.cc:2804] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:23:17.359956 23609 raft_consensus.cc:493] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:17.412983 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:17.458220 17834 raft_consensus.cc:2468] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 17.
I20260709 12:23:17.412961 17680 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37402 (request call id 2895) took 1566 ms. Trace:
W20260709 12:23:17.465179 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 237567us
I20260709 12:23:17.516750 18333 leader_election.cc:304] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:17.418665 18361 raft_consensus.cc:1240] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 15. Current term is 16. Ops: []
I20260709 12:23:17.470299 17652 leader_election.cc:304] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:17.516513 17680 rpcz_store.cc:276] 0709 12:23:15.846655 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:15.847928 (+  1273us) service_pool.cc:224] Handling call
0709 12:23:17.412777 (+1564849us) spinlock_profiling.cc:243] Waited 456 ms on lock 0x7b5400206554. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:23:17.412902 (+   125us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1048191872}
I20260709 12:23:17.498905 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:17.510784 17758 raft_consensus.cc:2468] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 19.
I20260709 12:23:17.456651 23488 raft_consensus.cc:493] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:23:17.519276 18333 leader_election.cc:304] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:17.521633 23507 raft_consensus.cc:493] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:17.525827 23507 raft_consensus.cc:515] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:17.526170 18333 leader_election.cc:304] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:17.527310 23530 raft_consensus.cc:2749] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 12:23:17.529309 18333 leader_election.cc:304] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:17.521340 17654 leader_election.cc:304] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:23:17.482887 22853 raft_consensus.cc:2804] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Leader election won for term 15
W20260709 12:23:17.534154 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 63951.9us
I20260709 12:23:17.485204 23266 raft_consensus.cc:2749] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
W20260709 12:23:17.569878 17803 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:17.528687 23608 raft_consensus.cc:2749] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:23:17.575698 17652 leader_election.cc:304] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
W20260709 12:23:17.568286 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 47075.7us
I20260709 12:23:17.485837 23513 consensus_queue.cc:1059] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 14.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.065s
I20260709 12:23:17.550209 23525 raft_consensus.cc:3060] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:17.485281 23475 raft_consensus.cc:515] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:17.520331 23246 consensus_queue.cc:1048] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.053s
I20260709 12:23:17.533565 23622 raft_consensus.cc:493] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:17.535923 23605 raft_consensus.cc:493] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:17.544453 23569 raft_consensus.cc:2749] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
W20260709 12:23:17.572922 23598 log.cc:927] Time spent T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.051s	user 0.001s	sys 0.000s
I20260709 12:23:17.535126 23488 raft_consensus.cc:515] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:17.513890 23293 consensus_queue.cc:1048] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.089s
I20260709 12:23:17.533638 23502 raft_consensus.cc:697] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:23:17.429816 23526 consensus_queue.cc:260] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.8, Last appended by leader: 8, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:17.532166 23555 consensus_queue.cc:1048] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.054s
W20260709 12:23:17.582885 17730 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:17.569631 23573 raft_consensus.cc:493] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:17.522260 23546 consensus_queue.cc:1059] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 13.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.057s
I20260709 12:23:17.536059 23574 raft_consensus.cc:3055] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:23:17.574723 23504 raft_consensus.cc:2804] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:23:17.522140 23609 raft_consensus.cc:515] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:17.602523 18333 leader_election.cc:304] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:23:17.667050 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 64621.2us
I20260709 12:23:17.528648 23551 raft_consensus.cc:3060] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:17.592068 22853 raft_consensus.cc:697] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 15 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
W20260709 12:23:17.652406 17728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:17.656561 17756 raft_consensus.cc:2468] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 13.
I20260709 12:23:17.608844 23468 raft_consensus.cc:2804] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:23:17.678273 18374 raft_consensus.cc:1240] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 10. Current term is 11. Ops: [10.12-10.12]
I20260709 12:23:17.679405 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:17.681473 18363 raft_consensus.cc:1240] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 13. Current term is 14. Ops: []
I20260709 12:23:17.682353 18366 raft_consensus.cc:1240] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 14. Current term is 15. Ops: []
I20260709 12:23:17.683138 18362 raft_consensus.cc:1275] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 16 index: 13. Preceding OpId from leader: term: 17 index: 13. (term mismatch)
W20260709 12:23:17.684697 17803 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:17.699412 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:17.709326 18362 pending_rounds.cc:85] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da: Aborting all ops after (but not including) 12
I20260709 12:23:17.711194 18362 pending_rounds.cc:107] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da: Aborting uncommitted NO_OP operation due to leader change: 16.13
I20260709 12:23:17.711674 18361 raft_consensus.cc:2393] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 13: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:23:17.711637 18362 raft_consensus.cc:2889] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:17.654084 23507 leader_election.cc:290] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:17.667322 23526 raft_consensus.cc:3060] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:17.715909 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:17.728699 18362 raft_consensus.cc:3060] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Advancing to term 12
W20260709 12:23:17.726922 23478 consensus_meta.cc:287] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.652s	user 0.006s	sys 0.009s
I20260709 12:23:17.717417 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:17.715946 18365 raft_consensus.cc:1240] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 12. Current term is 13. Ops: [12.9-12.9]
I20260709 12:23:17.721599 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:17.886618 18366 raft_consensus.cc:2393] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 10: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:23:17.887661 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:17.852941 22853 consensus_queue.cc:237] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.11, Last appended by leader: 11, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:17.888809 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:17.889557 18366 raft_consensus.cc:2393] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 13: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:23:17.890306 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:17.891338 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:17.724844 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:17.715106 23496 raft_consensus.cc:3060] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:17.678187 23530 raft_consensus.cc:2804] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Leader election won for term 12
W20260709 12:23:17.717410 17803 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:17.842226 23624 raft_consensus.cc:493] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:17.732266 17801 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:17.896044 17801 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:17.897261 17801 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:17.752108 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:17.898677 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:17.715633 23468 raft_consensus.cc:697] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 13 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:23:17.667186 23573 raft_consensus.cc:515] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:17.916142 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:17.712698 23513 raft_consensus.cc:493] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:17.603469 23622 raft_consensus.cc:515] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:17.811370 18362 raft_consensus.cc:2468] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 12.
I20260709 12:23:17.920567 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:17.728400 23506 raft_consensus.cc:493] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:17.841666 16501 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:23:17.800052 23620 raft_consensus.cc:493] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:17.828869 23606 leader_election.cc:290] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:17.857642 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:17.859946 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:23:17.861567 17802 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:17.788009 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:17.698457 23504 raft_consensus.cc:697] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 11 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:23:17.934363 23468 consensus_queue.cc:237] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:17.938128 23504 consensus_queue.cc:237] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 10.12, Last appended by leader: 12, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:17.938491 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:17.610574 23605 raft_consensus.cc:515] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:17.940500 23488 leader_election.cc:290] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:23:17.940663 23623 log.cc:927] Time spent T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.113s	user 0.000s	sys 0.000s
I20260709 12:23:17.941561 23605 leader_election.cc:290] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:23:17.893265 18367 consensus_meta.cc:287] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.552s	user 0.012s	sys 0.000s
I20260709 12:23:17.942562 18371 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 1777) took 1308 ms. Trace:
I20260709 12:23:17.942691 18367 raft_consensus.cc:2468] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 19.
I20260709 12:23:17.894073 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:17.945156 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
W20260709 12:23:17.919524 23625 log.cc:927] Time spent T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.072s	user 0.000s	sys 0.001s
I20260709 12:23:17.894672 18366 raft_consensus.cc:2393] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 16: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:23:17.954329 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
W20260709 12:23:17.852481 23619 log.cc:927] Time spent T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.138s	user 0.001s	sys 0.000s
I20260709 12:23:17.732805 23604 raft_consensus.cc:2749] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 12:23:17.901801 17758 raft_consensus.cc:1240] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 14. Current term is 15. Ops: []
W20260709 12:23:17.912801 17804 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:17.920693 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:17.967088 18362 raft_consensus.cc:2393] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 16: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:23:17.967933 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:17.968928 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 15 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:17.969624 18362 raft_consensus.cc:2468] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 14.
I20260709 12:23:17.971585 18362 raft_consensus.cc:1240] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 13. Current term is 14. Ops: []
I20260709 12:23:17.728752 23494 raft_consensus.cc:493] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:23:17.664765 23574 raft_consensus.cc:740] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 16 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:23:17.972147 23494 raft_consensus.cc:515] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:17.892271 23502 consensus_queue.cc:237] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.10, Last appended by leader: 10, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:17.972611 23574 consensus_queue.cc:260] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.10, Last appended by leader: 10, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:17.921306 18365 raft_consensus.cc:2468] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 8.
I20260709 12:23:17.947029 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:17.974619 18368 raft_consensus.cc:2393] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 10: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:23:17.974434 23574 raft_consensus.cc:3060] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:17.911224 23546 raft_consensus.cc:493] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:17.913218 23530 raft_consensus.cc:697] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 12 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
W20260709 12:23:17.934424 17802 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:17.975555 23546 raft_consensus.cc:515] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:17.976187 23530 consensus_queue.cc:237] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 11.8, Last appended by leader: 8, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:17.976715 17802 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:17.977717 17802 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:17.977753 23546 leader_election.cc:290] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:17.942965 18371 rpcz_store.cc:276] 0709 12:23:16.633773 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:16.652762 (+ 18989us) service_pool.cc:224] Handling call
0709 12:23:16.672538 (+ 19776us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:16.830453 (+157915us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 12:23:17.214028 (+383575us) spinlock_profiling.cc:243] Waited 3.43 ms on lock 0x7b18003f9a00. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b47515e07 00007f7b503709a9 00007f7b4753269b 00007f7b4753093f 00007f7b47530455 00007f7b4e0183c7 00007f7b4a62a9a7 00007f7b4a63a0bf 00007f7b4a63a038
0709 12:23:17.271982 (+ 57954us) log.cc:844] Serialized 14 byte log entry
0709 12:23:17.576941 (+304959us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:17.577454 (+   513us) log.cc:844] Serialized 26 byte log entry
0709 12:23:17.577589 (+   135us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:23:17.577624 (+    35us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:17.577890 (+   266us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:17.942279 (+364389us) raft_consensus.cc:1685] finished
0709 12:23:17.942383 (+   104us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:17.942503 (+   120us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":162945,"prepare.queue_time_us":36649,"prepare.run_cpu_time_us":23,"prepare.run_wall_time_us":24,"raft.queue_time_us":173380,"raft.run_cpu_time_us":350,"raft.run_wall_time_us":20306,"spinlock_wait_cycles":7891200,"thread_start_us":355280,"threads_started":2,"wal-append.queue_time_us":407014}
W20260709 12:23:17.980358 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:17.981206 18371 raft_consensus.cc:1240] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 12. Current term is 13. Ops: [12.10-12.10]
I20260709 12:23:17.981913 17802 leader_election.cc:304] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:17.956120 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:17.958873 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:17.984273 18367 raft_consensus.cc:2468] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 13.
I20260709 12:23:17.960660 23609 leader_election.cc:290] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:17.986423 18367 raft_consensus.cc:1275] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 9 index: 10. Preceding OpId from leader: term: 12 index: 12. (index mismatch)
W20260709 12:23:17.960877 23621 log.cc:927] Time spent T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.145s	user 0.001s	sys 0.000s
I20260709 12:23:17.944675 23620 raft_consensus.cc:515] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:17.994082 23620 leader_election.cc:290] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:17.903097 23624 raft_consensus.cc:515] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:17.996917 23626 log.cc:927] Time spent T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.197s	user 0.000s	sys 0.000s
I20260709 12:23:17.998268 23624 leader_election.cc:290] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:17.964916 23622 leader_election.cc:290] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:17.966859 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:18.000634 18366 raft_consensus.cc:2468] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 16.
I20260709 12:23:18.001484 23546 consensus_queue.cc:1048] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:23:17.989215 23605 raft_consensus.cc:2804] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:23:18.005018 23605 raft_consensus.cc:697] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 14 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:23:17.955268 17833 raft_consensus.cc:2393] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 10: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:23:18.005836 18361 raft_consensus.cc:2468] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 12.
I20260709 12:23:18.003641 17802 leader_election.cc:304] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:23:18.011693 17837 raft_consensus.cc:1275] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 15 index: 13. Preceding OpId from leader: term: 16 index: 14. (index mismatch)
I20260709 12:23:18.017787 17802 leader_election.cc:304] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:18.020869 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:18.023161 17802 leader_election.cc:304] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:23:18.026386 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:18.026525 23605 consensus_queue.cc:237] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 11.8, Last appended by leader: 8, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:18.032335 23622 raft_consensus.cc:2749] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:23:18.036562 18362 raft_consensus.cc:1240] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 16. Current term is 17. Ops: [16.10-16.10]
I20260709 12:23:18.039465 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:18.005999 23475 leader_election.cc:290] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:23:18.041728 17729 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:17.932575 23513 raft_consensus.cc:515] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:17.935231 23506 raft_consensus.cc:515] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:18.035362 23552 raft_consensus.cc:2804] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:23:18.034823 17835 raft_consensus.cc:1240] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 12 LEADER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 11. Current term is 12. Ops: []
I20260709 12:23:18.065632 17836 raft_consensus.cc:2393] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 16: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:23:18.165969 17681 raft_consensus.cc:1240] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 11 LEADER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 10. Current term is 11. Ops: []
W20260709 12:23:18.061462 17803 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:18.181252 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 158200us
I20260709 12:23:18.113036 17833 raft_consensus.cc:1275] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 13 index: 10. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
W20260709 12:23:18.176512 17652 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.578s	user 0.012s	sys 0.003s
W20260709 12:23:18.186208 17652 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.588s	user 0.012s	sys 0.003s
W20260709 12:23:18.187348 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 594169us
I20260709 12:23:18.189994 17652 leader_election.cc:304] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:18.186256 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:18.194321 17758 raft_consensus.cc:2376] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 14. Re-sending same reply.
I20260709 12:23:18.099246 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:18.196079 17837 raft_consensus.cc:2468] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 15.
W20260709 12:23:18.168716 17654 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.565s	user 0.012s	sys 0.045s
W20260709 12:23:18.197299 17654 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.594s	user 0.012s	sys 0.046s
W20260709 12:23:18.198004 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 616304us
I20260709 12:23:18.183023 17802 leader_election.cc:304] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:23:18.199275 17761 raft_consensus.cc:1240] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 12. Current term is 13. Ops: []
I20260709 12:23:18.202107 18333 leader_election.cc:304] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:18.202107 17654 leader_election.cc:304] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:23:18.122268 23470 consensus_meta.cc:287] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.693s	user 0.012s	sys 0.000s
W20260709 12:23:18.131860 23490 consensus_meta.cc:287] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.689s	user 0.005s	sys 0.005s
I20260709 12:23:18.214349 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:18.222209 18333 leader_election.cc:304] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:18.045195 23573 leader_election.cc:290] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:18.219177 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:18.227684 17762 raft_consensus.cc:2368] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 18. Current term is 19.
I20260709 12:23:18.219892 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:18.228436 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:18.218922 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:18.229302 17757 raft_consensus.cc:2393] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 12: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:23:18.229635 17755 raft_consensus.cc:2376] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 12. Re-sending same reply.
I20260709 12:23:18.136797 23494 leader_election.cc:290] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:23:18.234436 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 51526.5us
I20260709 12:23:18.176753 23620 raft_consensus.cc:493] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:18.232189 17803 leader_election.cc:304] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:18.133028 23427 raft_consensus.cc:3055] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:23:18.213155 17652 leader_election.cc:304] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:18.284442 17654 leader_election.cc:304] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:18.117578 23491 leader_election.cc:290] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 13 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:18.245504 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:18.135330 23469 raft_consensus.cc:3060] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:18.331297 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:18.358876 17762 raft_consensus.cc:2368] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 12. Current term is 13.
I20260709 12:23:18.203495 23508 raft_consensus.cc:2804] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:23:18.120623 23520 raft_consensus.cc:3055] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:23:18.357110 17802 leader_election.cc:304] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:23:18.357517 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 126227us
I20260709 12:23:18.368497 17803 leader_election.cc:304] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:18.331980 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:18.195267 23552 raft_consensus.cc:493] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:18.346207 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:23:18.315191 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 80897.1us
W20260709 12:23:18.371268 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 139021us
W20260709 12:23:18.371747 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 158692us
I20260709 12:23:18.219355 23492 raft_consensus.cc:2804] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:23:18.233044 22853 raft_consensus.cc:2749] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:23:18.379734 17654 leader_election.cc:304] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:23:18.331207 23630 raft_consensus.cc:2804] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:23:18.384665 17759 raft_consensus.cc:1240] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 LEADER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 13. Current term is 14. Ops: []
I20260709 12:23:18.388082 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 14 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:18.214596 23511 consensus_queue.cc:1048] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.163s
I20260709 12:23:18.392488 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:18.345089 23427 raft_consensus.cc:740] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:23:18.203145 23546 consensus_queue.cc:1059] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 9.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.163s
I20260709 12:23:18.409044 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:18.214962 23521 raft_consensus.cc:493] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:23:18.415920 17757 raft_consensus.cc:1275] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 13 index: 12. Preceding OpId from leader: term: 15 index: 13. (index mismatch)
I20260709 12:23:18.236065 23609 raft_consensus.cc:2749] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:23:18.366665 23520 raft_consensus.cc:740] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
W20260709 12:23:18.393500 23551 consensus_meta.cc:287] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.705s	user 0.012s	sys 0.002s
I20260709 12:23:18.363298 23631 raft_consensus.cc:2749] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:23:18.381707 23552 raft_consensus.cc:3060] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:18.405922 23606 raft_consensus.cc:2749] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:23:18.233109 23470 raft_consensus.cc:515] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:18.417114 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 195102us
I20260709 12:23:18.417268 17757 raft_consensus.cc:1275] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 13 index: 10. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
I20260709 12:23:18.398207 23636 raft_consensus.cc:2804] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:23:18.407573 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 17 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:18.359863 23526 consensus_queue.cc:1059] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 14.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.042s
W20260709 12:23:18.422693 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 44079.5us
W20260709 12:23:18.426502 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:18.404929 23610 log.cc:927] Time spent T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 1.127s	user 0.003s	sys 0.011s
I20260709 12:23:18.362530 23620 raft_consensus.cc:515] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:18.428575 17804 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:18.400455 23504 raft_consensus.cc:2749] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:23:18.402844 23630 raft_consensus.cc:493] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:18.377224 23530 raft_consensus.cc:2749] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:23:18.438000 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:18.438725 17756 raft_consensus.cc:2468] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 16.
I20260709 12:23:18.404304 23293 raft_consensus.cc:2804] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:23:18.418591 23481 raft_consensus.cc:3060] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:18.443177 17755 raft_consensus.cc:2410] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for term 17 because replica has last-logged OpId of term: 16 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 9.
I20260709 12:23:18.444067 17652 leader_election.cc:304] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:18.388983 23492 raft_consensus.cc:493] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:18.443125 18333 leader_election.cc:304] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:18.442464 23638 raft_consensus.cc:493] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
W20260709 12:23:18.458529 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:18.420789 23521 raft_consensus.cc:515] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:18.492720 17727 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:18.473084 17756 raft_consensus.cc:1240] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 10. Current term is 11. Ops: []
I20260709 12:23:18.493644 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:18.494240 18362 raft_consensus.cc:2468] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 13.
I20260709 12:23:18.494405 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:23:18.470937 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 94517.8us
I20260709 12:23:18.441708 23605 raft_consensus.cc:3055] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:23:18.438859 23551 raft_consensus.cc:2749] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:23:18.445855 23636 raft_consensus.cc:493] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:23:18.452853 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:18.501156 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:18.414768 23427 consensus_queue.cc:260] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.11, Last appended by leader: 11, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:18.491822 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:18.502331 18366 raft_consensus.cc:2393] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 12: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:23:18.471731 23630 raft_consensus.cc:3060] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:23:18.485404 23637 log.cc:927] Time spent T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.082s	user 0.000s	sys 0.001s
I20260709 12:23:18.505878 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:18.509001 17834 raft_consensus.cc:2468] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 8.
I20260709 12:23:18.501086 23569 consensus_queue.cc:1048] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.010s
I20260709 12:23:18.500247 23638 raft_consensus.cc:515] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:23:18.512418 23639 log.cc:927] Time spent T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.096s	user 0.000s	sys 0.001s
I20260709 12:23:18.513326 23521 leader_election.cc:290] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:18.513815 23638 leader_election.cc:290] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:18.513967 23506 leader_election.cc:290] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:18.514634 18365 raft_consensus.cc:2468] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 12.
I20260709 12:23:18.484375 23492 raft_consensus.cc:3060] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:18.508181 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:18.491192 23468 raft_consensus.cc:2804] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:23:18.480710 23520 consensus_queue.cc:260] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.9, Last appended by leader: 9, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:18.531306 23520 raft_consensus.cc:3060] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:18.503356 23636 raft_consensus.cc:3060] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:18.533440 18366 raft_consensus.cc:1275] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 14 index: 12. Preceding OpId from leader: term: 15 index: 13. (index mismatch)
I20260709 12:23:18.534758 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:18.535437 18362 raft_consensus.cc:2468] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 17.
I20260709 12:23:18.499290 23609 consensus_queue.cc:1048] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.037s
I20260709 12:23:18.523154 23427 raft_consensus.cc:3060] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:18.518780 17757 raft_consensus.cc:1240] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 11. Current term is 12. Ops: []
I20260709 12:23:18.519781 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 15 candidate_status { last_received { term: 11 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:18.545446 23544 raft_consensus.cc:3060] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Advancing to term 16
W20260709 12:23:18.518385 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 150009us
I20260709 12:23:18.524789 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:18.499993 23605 raft_consensus.cc:740] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 10 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:23:18.528716 23620 leader_election.cc:290] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:23:18.529959 23496 consensus_meta.cc:287] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.634s	user 0.012s	sys 0.000s
I20260709 12:23:18.550773 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:23:18.549439 18334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:18.558504 17830 raft_consensus.cc:2468] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 11.
I20260709 12:23:18.558641 17803 leader_election.cc:304] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:18.553706 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:18.565383 17834 raft_consensus.cc:2468] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 12.
I20260709 12:23:18.556681 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:18.573349 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:18.574829 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:18.579213 17757 raft_consensus.cc:1275] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 11 index: 8. Preceding OpId from leader: term: 12 index: 9. (index mismatch)
I20260709 12:23:18.581012 17729 leader_election.cc:304] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:18.578347 17803 leader_election.cc:304] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:23:18.587814 18333 leader_election.cc:400] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 18 pre-election: Vote denied by peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651) with higher term. Message: Invalid argument: T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 18. Current term is 19.
I20260709 12:23:18.588541 18333 leader_election.cc:403] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 18 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:23:18.594985 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 15 candidate_status { last_received { term: 11 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:18.596122 17689 raft_consensus.cc:1240] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 13 LEADER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 12. Current term is 13. Ops: [12.11-12.11]
I20260709 12:23:18.598398 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:18.599396 17689 raft_consensus.cc:2393] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 13: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:23:18.600795 17686 raft_consensus.cc:1275] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 16 index: 9. Preceding OpId from leader: term: 17 index: 10. (index mismatch)
I20260709 12:23:18.562296 23502 raft_consensus.cc:493] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:18.604089 17730 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:18.562139 23056 raft_consensus.cc:493] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:18.570753 23496 raft_consensus.cc:493] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:18.610203 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 131456us
I20260709 12:23:18.561899 23468 raft_consensus.cc:697] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 17 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:23:18.599953 23508 raft_consensus.cc:697] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 13 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:23:18.622784 23648 raft_consensus.cc:2804] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:23:18.615311 17803 leader_election.cc:304] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:18.617591 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:18.617007 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:18.582350 23403 raft_consensus.cc:2749] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:23:18.626947 18333 leader_election.cc:304] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:18.602422 23620 raft_consensus.cc:2749] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Leader pre-election lost for term 18. Reason: Vote denied by peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651) with higher term. Message: Invalid argument: T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 18. Current term is 19.
I20260709 12:23:18.547808 23630 raft_consensus.cc:515] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:18.589077 23490 raft_consensus.cc:2804] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:23:18.637574 18333 leader_election.cc:400] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 12 election: Vote denied by peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651) with higher term. Message: Invalid argument: T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 12. Current term is 13.
I20260709 12:23:18.605710 23475 raft_consensus.cc:493] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:18.635116 17680 raft_consensus.cc:2468] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 12.
I20260709 12:23:18.584815 23494 consensus_queue.cc:1048] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.052s
I20260709 12:23:18.643082 23454 raft_consensus.cc:3055] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:23:18.643474 23454 raft_consensus.cc:740] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 13 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:23:18.644892 23630 leader_election.cc:290] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 15 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:18.645211 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:18.647908 23454 consensus_queue.cc:260] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.10, Last appended by leader: 10, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:18.649647 23454 raft_consensus.cc:3060] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:18.626174 23496 raft_consensus.cc:515] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:18.654973 23496 leader_election.cc:290] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:18.655645 17760 raft_consensus.cc:2468] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 16.
I20260709 12:23:18.655941 23470 leader_election.cc:290] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 15 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:23:18.603448 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 246412us
I20260709 12:23:18.658703 18366 raft_consensus.cc:1275] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 13 index: 10. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
I20260709 12:23:18.659479 17803 leader_election.cc:304] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:18.661587 17764 raft_consensus.cc:2468] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 10.
I20260709 12:23:18.641075 23624 raft_consensus.cc:3060] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:18.623390 23636 raft_consensus.cc:515] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:18.668505 23644 log.cc:927] Time spent T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.097s	user 0.000s	sys 0.001s
I20260709 12:23:18.656430 23650 raft_consensus.cc:2749] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 12:23:18.674292 18333 leader_election.cc:304] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:18.683357 18366 raft_consensus.cc:1275] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 13 index: 11. (index mismatch)
I20260709 12:23:18.638161 23293 raft_consensus.cc:697] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 19 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:23:18.656773 23648 raft_consensus.cc:697] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 12 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:23:18.688952 23469 consensus_queue.cc:1048] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:23:18.620164 23502 raft_consensus.cc:515] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:18.689723 23481 raft_consensus.cc:2749] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
W20260709 12:23:18.693591 17655 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:18.708225 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:18.699826 23395 raft_consensus.cc:3055] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:23:18.700680 18361 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59902 (request call id 1411) took 1015 ms. Trace:
I20260709 12:23:18.709318 18361 rpcz_store.cc:276] 0709 12:23:17.685852 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:17.898454 (+212602us) service_pool.cc:224] Handling call
0709 12:23:18.700475 (+802021us) spinlock_profiling.cc:243] Waited 65 ms on lock 0x7b5400672d54. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:23:18.700614 (+   139us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":149557504}
I20260709 12:23:18.689014 23508 consensus_queue.cc:237] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 12.12, Last appended by leader: 12, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:18.701474 18363 raft_consensus.cc:1275] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
I20260709 12:23:18.624827 23056 raft_consensus.cc:515] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:18.702639 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:23:18.703503 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 45163.3us
I20260709 12:23:18.642256 23490 raft_consensus.cc:493] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:23:18.731292 18334 leader_election.cc:304] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
W20260709 12:23:18.731205 17802 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:18.736759 17838 raft_consensus.cc:1240] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 15. Current term is 16. Ops: []
I20260709 12:23:18.736912 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 17 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:18.739444 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:18.740612 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:18.740851 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:18.706024 17728 leader_election.cc:304] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:18.706943 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 13 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:18.741748 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:18.741930 18364 raft_consensus.cc:2468] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 12.
I20260709 12:23:18.713037 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:18.742928 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:18.716130 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:18.722108 23648 consensus_queue.cc:237] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.9, Last appended by leader: 9, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:18.728483 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:23:18.748155 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 88770.7us
I20260709 12:23:18.748489 17652 leader_election.cc:304] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:18.743418 17687 raft_consensus.cc:2468] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 12.
I20260709 12:23:18.744261 17685 raft_consensus.cc:3060] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:18.750860 17803 leader_election.cc:304] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:23:18.743868 17832 raft_consensus.cc:2368] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for earlier term 14. Current term is 15.
I20260709 12:23:18.761256 17761 raft_consensus.cc:1275] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 13 index: 11. (index mismatch)
I20260709 12:23:18.761690 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:18.762230 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 13 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:18.767588 18371 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59902 (request call id 1415) took 1058 ms. Trace:
I20260709 12:23:18.745342 17728 leader_election.cc:304] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:18.746474 18367 raft_consensus.cc:1240] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 11. Current term is 12. Ops: []
I20260709 12:23:18.771173 17685 raft_consensus.cc:2468] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 15.
I20260709 12:23:18.772043 17803 leader_election.cc:304] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:23:18.773743 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 99559.3us
W20260709 12:23:18.734246 23653 log.cc:927] Time spent T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.066s	user 0.001s	sys 0.001s
I20260709 12:23:18.704687 23620 raft_consensus.cc:2749] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:23:18.785701 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:18.670665 23475 raft_consensus.cc:515] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:18.768355 18371 rpcz_store.cc:276] 0709 12:23:17.687456 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:17.982653 (+295197us) service_pool.cc:224] Handling call
0709 12:23:17.982974 (+   321us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:17.983185 (+   211us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 12:23:18.366971 (+383786us) log.cc:844] Serialized 14 byte log entry
0709 12:23:18.728695 (+361724us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:18.728747 (+    52us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:23:18.728797 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:18.745777 (+ 16980us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:18.745929 (+   152us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":488481,"prepare.run_cpu_time_us":216,"prepare.run_wall_time_us":4070,"raft.queue_time_us":4112,"thread_start_us":744395,"threads_started":2}
I20260709 12:23:18.791782 17803 leader_election.cc:304] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:23:18.792135 17832 raft_consensus.cc:1275] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 14 index: 12. Preceding OpId from leader: term: 15 index: 13. (index mismatch)
I20260709 12:23:18.808326 17803 leader_election.cc:304] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:23:18.808467 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:18.809505 17832 raft_consensus.cc:2468] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 16.
I20260709 12:23:18.810998 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 15 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:18.732218 23525 raft_consensus.cc:2749] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:23:18.794412 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:18.776347 18333 leader_election.cc:304] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:18.797250 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:18.803352 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:18.728204 23395 raft_consensus.cc:740] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 14 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:23:18.732219 23650 raft_consensus.cc:493] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:18.768394 23649 raft_consensus.cc:493] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:18.815153 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 15 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:18.774093 23604 raft_consensus.cc:2804] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:23:18.747893 23490 raft_consensus.cc:3060] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:18.746654 23630 raft_consensus.cc:493] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:18.817943 17757 raft_consensus.cc:2393] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 17: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:23:18.797036 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:18.824743 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:23:18.798826 23572 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.884s	user 0.008s	sys 0.037s
I20260709 12:23:18.784734 23257 raft_consensus.cc:2764] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Leader pre-election decision vote started in defunct term 14: won
I20260709 12:23:18.747082 23469 raft_consensus.cc:2749] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
W20260709 12:23:18.829784 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 53495.2us
I20260709 12:23:18.800269 23631 raft_consensus.cc:493] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:18.830405 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:18.828128 23546 raft_consensus.cc:2804] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:23:18.834017 23521 raft_consensus.cc:2749] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:23:18.834196 23475 leader_election.cc:290] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:18.834939 17803 leader_election.cc:304] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
W20260709 12:23:18.886421 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 51540.8us
I20260709 12:23:18.847909 23604 raft_consensus.cc:493] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:18.835744 23502 leader_election.cc:290] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:18.835969 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:18.836313 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:18.837505 17727 leader_election.cc:304] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:18.837980 23513 leader_election.cc:290] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:18.838649 23056 leader_election.cc:290] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:18.799433 23297 raft_consensus.cc:493] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:23:18.839462 23293 consensus_queue.cc:237] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 17.15, Last appended by leader: 15, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:18.801502 23468 consensus_queue.cc:237] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.10, Last appended by leader: 10, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:18.798978 23378 raft_consensus.cc:2749] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:23:18.796392 23481 raft_consensus.cc:2749] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:23:18.786865 23403 consensus_queue.cc:1048] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.055s
I20260709 12:23:18.841392 23650 raft_consensus.cc:515] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:18.775650 23491 raft_consensus.cc:493] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:18.854839 17830 raft_consensus.cc:2468] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 17.
I20260709 12:23:18.787204 23651 raft_consensus.cc:2804] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:23:18.705379 23552 raft_consensus.cc:515] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:18.657379 23605 consensus_queue.cc:260] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 10.12, Last appended by leader: 12, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:18.775900 23652 raft_consensus.cc:2749] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:23:18.866048 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:18.868216 18333 leader_election.cc:304] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:18.829672 18364 raft_consensus.cc:3060] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:18.827911 23531 raft_consensus.cc:2804] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:23:18.883308 17758 raft_consensus.cc:1240] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 11. Current term is 12. Ops: []
I20260709 12:23:18.883366 17686 raft_consensus.cc:2368] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for earlier term 14. Current term is 15.
I20260709 12:23:18.835346 23494 raft_consensus.cc:493] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
W20260709 12:23:18.858683 23572 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.962s	user 0.008s	sys 0.038s
W20260709 12:23:18.896685 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 98626.3us
W20260709 12:23:18.972585 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 104466us
I20260709 12:23:18.898113 23604 raft_consensus.cc:3060] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:23:18.898716 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:18.977234 23658 log.cc:927] Time spent T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.149s	user 0.000s	sys 0.000s
I20260709 12:23:18.973567 23531 raft_consensus.cc:493] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:18.980057 23531 raft_consensus.cc:3060] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:18.933447 23470 consensus_queue.cc:1048] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.039s
I20260709 12:23:18.926990 17802 leader_election.cc:304] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da
W20260709 12:23:18.981844 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 54970us
I20260709 12:23:18.964093 23491 raft_consensus.cc:515] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:18.884809 23546 raft_consensus.cc:493] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:18.983062 23546 raft_consensus.cc:3060] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:18.984915 23491 leader_election.cc:290] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:18.938382 23668 raft_consensus.cc:493] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:18.944260 23297 raft_consensus.cc:515] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:18.986526 23668 raft_consensus.cc:515] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:18.989578 23668 leader_election.cc:290] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:18.990515 23622 raft_consensus.cc:2804] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:23:18.991039 23622 raft_consensus.cc:493] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:18.991279 17832 raft_consensus.cc:2393] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 15: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:23:18.965728 23651 raft_consensus.cc:493] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:18.994550 23651 raft_consensus.cc:3060] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:18.895808 23472 raft_consensus.cc:2804] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Leader pre-election won for term 14
W20260709 12:23:18.948173 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 110750us
I20260709 12:23:18.855957 23649 raft_consensus.cc:515] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:23:18.953810 23663 log.cc:927] Time spent T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.058s	user 0.000s	sys 0.000s
I20260709 12:23:18.949725 23513 raft_consensus.cc:2749] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:23:18.867904 23630 raft_consensus.cc:515] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:23:18.944219 23609 raft_consensus.cc:493] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:23:18.973352 18333 leader_election.cc:304] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:18.973996 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:18.887300 23631 raft_consensus.cc:515] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:18.972046 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:18.976363 23494 raft_consensus.cc:515] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:18.816040 23634 consensus_queue.cc:1048] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.068s
I20260709 12:23:18.950337 23469 raft_consensus.cc:493] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:18.982709 17802 leader_election.cc:304] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:23:19.005066 23427 raft_consensus.cc:2749] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:23:19.003785 23498 raft_consensus.cc:493] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:19.007551 17654 leader_election.cc:304] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
W20260709 12:23:19.057916 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 50487.6us
I20260709 12:23:19.008556 17727 leader_election.cc:304] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
W20260709 12:23:19.059202 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 50759.7us
I20260709 12:23:19.060654 17727 leader_election.cc:304] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:19.063203 17727 leader_election.cc:400] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 14 pre-election: Vote denied by peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837) with higher term. Message: Invalid argument: T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for earlier term 14. Current term is 15.
I20260709 12:23:19.063527 17727 leader_election.cc:403] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 14 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:23:19.010999 17681 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37402 (request call id 2923) took 1150 ms. Trace:
I20260709 12:23:19.014048 18364 raft_consensus.cc:2468] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 15.
I20260709 12:23:19.022292 23649 leader_election.cc:290] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:19.022944 23297 leader_election.cc:290] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:19.024943 23650 leader_election.cc:290] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:18.968144 23475 raft_consensus.cc:493] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:18.966303 23638 raft_consensus.cc:493] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:18.964761 23620 raft_consensus.cc:493] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:19.102830 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:19.036412 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:18.968641 23518 raft_consensus.cc:493] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:18.991513 23622 raft_consensus.cc:3060] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:19.040277 18366 raft_consensus.cc:2468] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 13.
W20260709 12:23:19.044693 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 62068.8us
I20260709 12:23:18.995307 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:19.112560 17652 leader_election.cc:304] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:23:19.125711 17832 raft_consensus.cc:2468] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 13.
I20260709 12:23:19.126993 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:23:19.046239 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:19.007730 23472 raft_consensus.cc:493] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:19.064205 17681 rpcz_store.cc:276] 0709 12:23:17.860607 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:17.860783 (+   176us) service_pool.cc:224] Handling call
0709 12:23:17.861200 (+   417us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:18.181784 (+320584us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:19.010762 (+828978us) spinlock_profiling.cc:243] Waited 304 ms on lock 0x7b54004254d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:23:19.010939 (+   177us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":700131584}
I20260709 12:23:19.025852 23511 raft_consensus.cc:493] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:19.040086 23605 raft_consensus.cc:3060] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:19.049676 23551 raft_consensus.cc:2749] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:23:19.082120 17833 raft_consensus.cc:2410] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for term 17 because replica has last-logged OpId of term: 16 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 11.
I20260709 12:23:19.036085 23374 raft_consensus.cc:493] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:19.078541 17727 leader_election.cc:304] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:23:19.141433 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 63419.4us
W20260709 12:23:19.142733 17727 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:19.000172 23492 raft_consensus.cc:515] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:19.094221 17837 raft_consensus.cc:2468] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 13.
W20260709 12:23:19.144968 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 171673us
I20260709 12:23:19.105829 17830 raft_consensus.cc:2468] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 16.
I20260709 12:23:19.050388 23636 leader_election.cc:290] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 13 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:19.091137 23257 raft_consensus.cc:493] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:23:19.114451 23620 raft_consensus.cc:515] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:19.114496 23660 log.cc:927] Time spent T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.075s	user 0.001s	sys 0.000s
I20260709 12:23:19.154633 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 15 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:19.116222 23494 leader_election.cc:290] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:23:19.156180 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 43716.1us
I20260709 12:23:19.069139 23544 raft_consensus.cc:2804] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:23:19.156515 17681 raft_consensus.cc:2393] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 15: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
W20260709 12:23:19.080320 23520 consensus_meta.cc:287] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.548s	user 0.013s	sys 0.000s
I20260709 12:23:19.076560 23656 raft_consensus.cc:2804] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:23:19.161058 23656 raft_consensus.cc:493] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:23:19.161515 23656 raft_consensus.cc:3060] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:19.076182 23491 raft_consensus.cc:3060] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Advancing to term 15
I20260709 12:23:19.075440 23506 raft_consensus.cc:2749] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:23:19.141280 23511 raft_consensus.cc:515] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:19.074915 23513 raft_consensus.cc:2749] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 12:23:19.109630 23475 raft_consensus.cc:515] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:19.132994 23518 raft_consensus.cc:515] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:19.173596 23518 leader_election.cc:290] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:19.175258 23531 raft_consensus.cc:515] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:19.181488 17830 raft_consensus.cc:1240] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 16. Current term is 17. Ops: []
I20260709 12:23:19.184345 23531 leader_election.cc:290] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 17 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:19.146611 18333 leader_election.cc:304] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:19.150630 17727 leader_election.cc:304] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:19.151417 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:19.185515 17687 raft_consensus.cc:2468] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 14.
I20260709 12:23:19.185835 17727 leader_election.cc:304] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:19.186368 23513 raft_consensus.cc:2804] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Leader pre-election won for term 14
W20260709 12:23:19.151919 17802 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:19.186798 23513 raft_consensus.cc:493] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:19.152132 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:19.187529 17682 raft_consensus.cc:2468] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 13.
I20260709 12:23:19.127512 23624 raft_consensus.cc:2749] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:23:19.058231 23469 raft_consensus.cc:515] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:19.194857 23469 leader_election.cc:290] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:19.157321 17654 leader_election.cc:304] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:23:19.049484 23609 raft_consensus.cc:515] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:19.147114 23374 raft_consensus.cc:515] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:19.200331 23669 consensus_queue.cc:1048] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 12:23:19.158790 23496 raft_consensus.cc:493] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:19.201918 23609 leader_election.cc:290] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:19.202340 23496 raft_consensus.cc:515] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:19.202765 18367 raft_consensus.cc:2468] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 15.
I20260709 12:23:19.152647 23521 consensus_queue.cc:1048] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.007s
I20260709 12:23:19.159493 23470 raft_consensus.cc:2804] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:23:19.205917 23470 raft_consensus.cc:493] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:23:19.206439 23470 raft_consensus.cc:3060] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:19.147245 23573 raft_consensus.cc:493] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:19.152582 23056 raft_consensus.cc:2804] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:23:19.207350 17802 leader_election.cc:304] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:23:19.207650 23056 raft_consensus.cc:493] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:19.208124 23056 raft_consensus.cc:3060] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:19.208748 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:19.152473 23427 consensus_queue.cc:1048] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.008s
I20260709 12:23:19.207381 23573 raft_consensus.cc:515] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:19.159780 23492 leader_election.cc:290] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 17 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:19.147333 23522 raft_consensus.cc:493] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:19.189910 23513 raft_consensus.cc:3060] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:19.212456 23573 leader_election.cc:290] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:19.212237 23522 raft_consensus.cc:515] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:19.160212 23257 raft_consensus.cc:515] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:19.216012 23544 raft_consensus.cc:697] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 15 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:23:19.216560 17802 leader_election.cc:304] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:23:19.217077 23544 consensus_queue.cc:237] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 13.9, Last appended by leader: 9, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:19.217675 23257 leader_election.cc:290] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:19.218410 23652 raft_consensus.cc:2804] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Leader pre-election won for term 15
W20260709 12:23:19.179978 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:19.218854 23652 raft_consensus.cc:493] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:19.219264 23652 raft_consensus.cc:3060] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:19.219812 23522 leader_election.cc:290] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:19.139252 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:19.131160 23502 raft_consensus.cc:493] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:19.223196 23649 raft_consensus.cc:2804] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:23:19.223515 23502 raft_consensus.cc:515] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:19.177507 23546 raft_consensus.cc:515] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:19.225327 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:19.190563 18333 leader_election.cc:304] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:19.226019 23502 leader_election.cc:290] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:19.226238 17758 raft_consensus.cc:3060] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:19.136075 23472 raft_consensus.cc:3060] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:19.220077 18371 raft_consensus.cc:1240] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 11. Current term is 12. Ops: []
I20260709 12:23:19.227476 18333 leader_election.cc:304] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:19.228941 17760 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38786 (request call id 3206) took 1208 ms. Trace:
I20260709 12:23:19.229218 17755 raft_consensus.cc:1275] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 15 index: 13. Preceding OpId from leader: term: 16 index: 14. (index mismatch)
I20260709 12:23:19.229321 17760 rpcz_store.cc:276] 0709 12:23:18.020176 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:18.020528 (+   352us) service_pool.cc:224] Handling call
0709 12:23:19.228768 (+1208240us) spinlock_profiling.cc:243] Waited 633 ms on lock 0x7b54002b5554. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:23:19.228889 (+   121us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1456636288}
I20260709 12:23:19.222025 23293 raft_consensus.cc:2804] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:23:19.230441 23609 raft_consensus.cc:2749] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:23:19.230557 23293 raft_consensus.cc:697] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 15 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:23:19.230787 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:19.192385 23530 raft_consensus.cc:2804] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:23:19.231833 23293 consensus_queue.cc:237] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 12.9, Last appended by leader: 9, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:19.232553 17801 leader_election.cc:304] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:19.232796 23530 raft_consensus.cc:697] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 16 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:23:19.233757 23650 raft_consensus.cc:2804] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:23:19.234288 23650 raft_consensus.cc:493] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:23:19.233795 23530 consensus_queue.cc:237] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 15.13, Last appended by leader: 13, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:19.234661 23650 raft_consensus.cc:3060] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:19.236449 18363 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59902 (request call id 1422) took 1546 ms. Trace:
I20260709 12:23:19.237084 23521 raft_consensus.cc:2749] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:23:19.222491 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:19.223770 23649 raft_consensus.cc:493] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:19.238927 23649 raft_consensus.cc:3060] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:19.226058 23546 leader_election.cc:290] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 9 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:19.230787 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:23:19.221458 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 64206.3us
I20260709 12:23:19.240134 17760 raft_consensus.cc:2468] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 17.
I20260709 12:23:19.241648 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:19.242250 17755 raft_consensus.cc:2468] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 14.
I20260709 12:23:19.231284 23609 consensus_queue.cc:1059] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 10.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.012s
I20260709 12:23:19.229517 18333 leader_election.cc:304] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:19.248310 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:19.243669 23655 consensus_queue.cc:1059] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 11.10, Next index: 11, Last known committed idx: 9, Time since last communication: 0.013s
I20260709 12:23:19.246827 23573 raft_consensus.cc:493] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:19.201972 23374 leader_election.cc:290] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:19.257984 23573 raft_consensus.cc:515] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:23:19.257246 23655 raft_consensus.cc:3055] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:23:19.238639 17835 raft_consensus.cc:2468] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 14.
I20260709 12:23:19.250556 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:19.265681 18366 raft_consensus.cc:1275] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 9 index: 11. Preceding OpId from leader: term: 11 index: 13. (index mismatch)
I20260709 12:23:19.243759 23609 raft_consensus.cc:3055] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:23:19.267827 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 15 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:19.254118 17652 leader_election.cc:304] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:19.248564 17759 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38786 (request call id 3203) took 1348 ms. Trace:
I20260709 12:23:19.301852 17759 rpcz_store.cc:276] 0709 12:23:17.900132 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:17.900326 (+   194us) service_pool.cc:224] Handling call
0709 12:23:18.119773 (+219447us) spinlock_profiling.cc:243] Waited 1.94 ms on lock 0x7b5800033398. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b474fbb71 00007f7b474fba22 00007f7b504148d9 00007f7b4f56eee0 00007f7b4f6380ab 00007f7b4f638131 00007f7b4f5f9fc0 00007f7b4f5e4929 00007f7b4f5e43d2 00007f7b48b5bbd4
0709 12:23:18.132946 (+ 13173us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:18.402949 (+270003us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:19.248201 (+845252us) spinlock_profiling.cc:243] Waited 251 ms on lock 0x7b54000fe1d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:23:19.248446 (+   245us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":582903808}
I20260709 12:23:19.253366 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:19.275414 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:19.237349 18363 rpcz_store.cc:276] 0709 12:23:17.689911 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:17.985077 (+295166us) service_pool.cc:224] Handling call
0709 12:23:18.701088 (+716011us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:18.729024 (+ 27936us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:19.236213 (+507189us) spinlock_profiling.cc:243] Waited 716 ms on lock 0x7b5400672d50. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a61f8ad 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:23:19.236390 (+   177us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1645831552}
I20260709 12:23:19.304093 17836 raft_consensus.cc:2468] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 12.
I20260709 12:23:19.304105 18361 raft_consensus.cc:3060] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Advancing to term 17
W20260709 12:23:19.308954 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 79533.8us
I20260709 12:23:19.103492 23665 consensus_queue.cc:1048] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.087s
I20260709 12:23:19.088762 23498 raft_consensus.cc:515] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:23:19.281605 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:19.317441 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:23:19.322145 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 67997.6us
I20260709 12:23:19.126543 23638 raft_consensus.cc:515] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:19.328544 18333 leader_election.cc:304] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:19.343442 17727 leader_election.cc:304] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:19.197795 23494 raft_consensus.cc:2804] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:23:19.347793 18363 raft_consensus.cc:2468] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 17.
I20260709 12:23:19.246299 23552 leader_election.cc:290] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 15 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:19.348527 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:19.300218 23502 raft_consensus.cc:493] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:19.320807 23522 raft_consensus.cc:493] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:19.249290 23496 leader_election.cc:290] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:19.349293 17652 leader_election.cc:304] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:19.351594 18361 raft_consensus.cc:2468] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 17.
I20260709 12:23:19.370407 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:19.371340 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:19.347026 17758 raft_consensus.cc:2468] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 17.
I20260709 12:23:19.310276 23609 raft_consensus.cc:740] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:23:19.308645 23682 raft_consensus.cc:2749] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:23:19.373232 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:19.344519 23654 raft_consensus.cc:2804] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Leader pre-election won for term 18
W20260709 12:23:19.344241 23660 log.cc:927] Time spent T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.127s	user 0.000s	sys 0.000s
I20260709 12:23:19.365897 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:19.309648 23655 raft_consensus.cc:740] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:23:19.354935 23624 raft_consensus.cc:2749] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:23:19.351915 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:19.374461 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:19.388281 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:19.377233 17681 raft_consensus.cc:1240] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 12. Current term is 13. Ops: []
I20260709 12:23:19.297888 22853 consensus_queue.cc:1059] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 11.10, Next index: 11, Last known committed idx: 9, Time since last communication: 0.065s
I20260709 12:23:19.342161 23506 raft_consensus.cc:493] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:19.256980 23650 raft_consensus.cc:515] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:19.354384 23494 raft_consensus.cc:493] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:19.380339 17654 leader_election.cc:304] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:19.417858 17654 leader_election.cc:304] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:23:19.374030 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:19.382627 17652 leader_election.cc:304] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:19.395867 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:19.422778 17758 raft_consensus.cc:2468] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 13.
I20260709 12:23:19.362668 23572 raft_consensus.cc:493] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:19.371031 23634 raft_consensus.cc:2804] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:23:19.357980 23525 consensus_queue.cc:1048] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.035s
I20260709 12:23:19.376874 23654 raft_consensus.cc:493] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:19.413679 17764 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36572 (request call id 975) took 1228 ms. Trace:
I20260709 12:23:19.434022 17764 rpcz_store.cc:276] 0709 12:23:18.184773 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:18.184879 (+   106us) service_pool.cc:224] Handling call
0709 12:23:19.413509 (+1228630us) spinlock_profiling.cc:243] Waited 332 ms on lock 0x7b54003e4ed4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:23:19.413622 (+   113us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":764339456}
W20260709 12:23:19.437323 23691 log.cc:927] Time spent T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.053s	user 0.000s	sys 0.000s
I20260709 12:23:19.384717 23496 raft_consensus.cc:493] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:19.384805 23669 raft_consensus.cc:2749] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:23:19.384693 23573 leader_election.cc:290] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:19.439625 23496 raft_consensus.cc:515] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:19.427405 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:19.441174 17758 raft_consensus.cc:2468] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 14.
I20260709 12:23:19.427898 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:19.442963 23496 leader_election.cc:290] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:19.443244 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:19.443900 23496 raft_consensus.cc:493] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:23:19.444177 17835 raft_consensus.cc:2468] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 16.
I20260709 12:23:19.444692 23496 raft_consensus.cc:515] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:19.374655 23502 raft_consensus.cc:515] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:19.448359 23502 leader_election.cc:290] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:19.450246 17760 raft_consensus.cc:2410] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for term 15 because replica has last-logged OpId of term: 13 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 8.
I20260709 12:23:19.445422 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 15 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:19.372716 23522 raft_consensus.cc:515] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:19.454344 23522 leader_election.cc:290] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:19.457444 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:19.463217 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:19.466444 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:19.466714 23496 leader_election.cc:290] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:19.467274 17835 raft_consensus.cc:2468] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 15.
I20260709 12:23:19.470472 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:23:19.470912 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 88380.7us
I20260709 12:23:19.471086 17758 raft_consensus.cc:2468] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 15.
I20260709 12:23:19.471911 17835 raft_consensus.cc:1240] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 8. Current term is 9. Ops: []
I20260709 12:23:19.472168 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:19.473337 17801 leader_election.cc:304] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:19.474339 23522 raft_consensus.cc:2804] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:23:19.474715 23522 raft_consensus.cc:493] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:19.475117 23522 raft_consensus.cc:3060] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Advancing to term 16
W20260709 12:23:19.475870 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 58097.1us
I20260709 12:23:19.419315 23494 raft_consensus.cc:3060] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:19.479861 18364 raft_consensus.cc:1275] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 13 index: 11. (term mismatch)
I20260709 12:23:19.481760 18364 pending_rounds.cc:85] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da: Aborting all ops after (but not including) 10
I20260709 12:23:19.481644 17652 leader_election.cc:304] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:19.482189 18364 pending_rounds.cc:107] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da: Aborting uncommitted NO_OP operation due to leader change: 12.11
I20260709 12:23:19.482553 18364 raft_consensus.cc:2889] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:19.424700 23520 raft_consensus.cc:2749] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:23:19.487449 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:19.428723 23665 raft_consensus.cc:493] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:19.492939 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 12 candidate_status { last_received { term: 9 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:19.428467 23630 leader_election.cc:290] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:19.428792 23634 raft_consensus.cc:493] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:19.465258 23654 raft_consensus.cc:3060] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:23:19.502262 23685 log.cc:927] Time spent T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.078s	user 0.000s	sys 0.001s
I20260709 12:23:19.503326 23395 consensus_queue.cc:260] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.9, Last appended by leader: 9, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:23:19.505776 23684 log.cc:927] Time spent T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.072s	user 0.000s	sys 0.001s
I20260709 12:23:19.464973 23594 raft_consensus.cc:2804] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:23:19.509264 23636 consensus_queue.cc:1048] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
W20260709 12:23:19.514333 23679 log.cc:927] Time spent T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.166s	user 0.001s	sys 0.000s
I20260709 12:23:19.476177 23504 raft_consensus.cc:2804] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:23:19.414466 23506 raft_consensus.cc:515] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:23:19.533474 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:19.554961 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 68755.9us
I20260709 12:23:19.487268 23631 leader_election.cc:290] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:19.489404 23669 raft_consensus.cc:2749] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20260709 12:23:19.562036 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 166806us
I20260709 12:23:19.432183 23572 raft_consensus.cc:515] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:19.520277 23634 raft_consensus.cc:3060] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:19.561510 23497 consensus_queue.cc:1059] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.004s
I20260709 12:23:19.496457 23552 raft_consensus.cc:493] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:19.577513 18363 raft_consensus.cc:1275] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 13 index: 11. Preceding OpId from leader: term: 15 index: 12. (index mismatch)
I20260709 12:23:19.558068 23594 raft_consensus.cc:493] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:19.584787 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 20 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:19.590631 18364 raft_consensus.cc:2468] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 19.
W20260709 12:23:19.586431 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 99745.2us
I20260709 12:23:19.568301 23620 leader_election.cc:290] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:19.548476 23665 raft_consensus.cc:515] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:19.592624 17729 leader_election.cc:304] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:19.478428 23609 consensus_queue.cc:260] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:19.585309 23631 raft_consensus.cc:493] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:23:19.583863 23504 raft_consensus.cc:697] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 17 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:23:19.573385 23056 raft_consensus.cc:515] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:23:19.604995 23688 log.cc:927] Time spent T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.109s	user 0.000s	sys 0.001s
I20260709 12:23:19.606674 23496 raft_consensus.cc:2804] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:23:19.606750 23656 raft_consensus.cc:515] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:19.607600 23681 log.cc:927] Time spent T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.192s	user 0.000s	sys 0.001s
I20260709 12:23:19.588181 23472 raft_consensus.cc:515] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:19.611706 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:19.592677 23594 raft_consensus.cc:3060] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:19.629393 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 14 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:19.646055 18364 raft_consensus.cc:2468] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 12.
I20260709 12:23:19.631294 23504 consensus_queue.cc:237] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.13, Last appended by leader: 13, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:19.632091 17762 raft_consensus.cc:1240] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 16. Current term is 17. Ops: []
I20260709 12:23:19.608327 23638 leader_election.cc:290] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:19.641775 17758 raft_consensus.cc:2468] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 15.
I20260709 12:23:19.629725 23609 raft_consensus.cc:3060] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:19.648144 17654 leader_election.cc:304] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
W20260709 12:23:19.655709 23693 log.cc:927] Time spent T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.087s	user 0.000s	sys 0.000s
I20260709 12:23:19.659521 17680 raft_consensus.cc:1240] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 12. Current term is 13. Ops: []
I20260709 12:23:19.669916 23472 leader_election.cc:290] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 14 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:23:19.677446 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 99263.3us
W20260709 12:23:19.678813 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 297624us
I20260709 12:23:19.695034 23652 raft_consensus.cc:515] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:19.679545 23530 consensus_queue.cc:1048] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20260709 12:23:19.681373 23056 leader_election.cc:290] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 17 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:19.681802 23656 leader_election.cc:290] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 13 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:23:19.683243 17727 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:19.636183 23498 leader_election.cc:290] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:19.594472 23552 raft_consensus.cc:515] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:19.697391 17834 raft_consensus.cc:1240] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 15. Current term is 17. Ops: []
I20260709 12:23:19.697386 17831 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37994 (request call id 1109) took 1142 ms. Trace:
I20260709 12:23:19.718997 17831 rpcz_store.cc:276] 0709 12:23:18.554808 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:18.554944 (+   136us) service_pool.cc:224] Handling call
0709 12:23:18.580243 (+ 25299us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:18.588199 (+  7956us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:18.588249 (+    50us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:18.588892 (+   643us) log.cc:844] Serialized 26 byte log entry
0709 12:23:19.012971 (+424079us) spinlock_profiling.cc:243] Waited 2.38 ms on lock 0x7b180032b540. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b47515e07 00007f7b503709a9 00007f7b4753269b 00007f7b4753093f 00007f7b47532c02 00007f7b4a1ba20b 00007f7b4a1bfd23 00007f7b4a1c00d1 00007f7b4a5dba0f
0709 12:23:19.123511 (+110540us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:23:19.123571 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:19.123649 (+    78us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:19.697014 (+573365us) raft_consensus.cc:1685] finished
0709 12:23:19.697042 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:19.697316 (+   274us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":52008,"raft.queue_time_us":31569,"raft.run_cpu_time_us":211,"raft.run_wall_time_us":15425,"spinlock_wait_cycles":5476224,"thread_start_us":110347,"threads_started":1,"wal-append.queue_time_us":737595}
I20260709 12:23:19.721699 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:19.697875 17764 raft_consensus.cc:1275] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 13 index: 9. Preceding OpId from leader: term: 14 index: 9. (term mismatch)
I20260709 12:23:19.722791 17764 pending_rounds.cc:85] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb: Aborting all ops after (but not including) 8
I20260709 12:23:19.723116 17764 pending_rounds.cc:107] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb: Aborting uncommitted NO_OP operation due to leader change: 13.9
I20260709 12:23:19.723397 17764 raft_consensus.cc:2889] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:19.699795 17728 leader_election.cc:304] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:19.647641 23470 raft_consensus.cc:515] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:19.634014 23496 raft_consensus.cc:493] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:19.674916 17835 raft_consensus.cc:1240] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 13. Current term is 14. Ops: []
W20260709 12:23:19.708350 23605 consensus_meta.cc:287] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.568s	user 0.010s	sys 0.000s
W20260709 12:23:19.679203 23490 consensus_meta.cc:287] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.811s	user 0.005s	sys 0.005s
W20260709 12:23:19.717563 23491 consensus_meta.cc:287] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.553s	user 0.009s	sys 0.000s
I20260709 12:23:19.716007 17838 raft_consensus.cc:1240] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 12 LEADER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 11. Current term is 12. Ops: []
W20260709 12:23:19.691732 23604 consensus_meta.cc:287] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.714s	user 0.010s	sys 0.001s
W20260709 12:23:19.745138 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 97065us
I20260709 12:23:19.747957 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:19.748776 18366 raft_consensus.cc:3060] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:19.736182 17832 raft_consensus.cc:2393] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 15: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:23:19.733516 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:19.632490 23631 raft_consensus.cc:515] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:19.642046 23636 consensus_queue.cc:1048] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.077s
I20260709 12:23:19.712258 23521 raft_consensus.cc:493] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:19.726495 17803 leader_election.cc:304] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:19.719167 23650 leader_election.cc:290] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 17 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:19.732506 23496 raft_consensus.cc:3060] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:19.807200 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:19.807909 17761 raft_consensus.cc:3060] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:19.808266 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:19.713390 18333 leader_election.cc:304] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:19.753182 23491 raft_consensus.cc:2749] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Leader pre-election lost for term 14. Reason: Vote denied by peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837) with higher term. Message: Invalid argument: T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for earlier term 14. Current term is 15.
I20260709 12:23:19.780897 23297 raft_consensus.cc:2804] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Leader pre-election won for term 14
W20260709 12:23:19.788635 23649 consensus_meta.cc:287] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.549s	user 0.006s	sys 0.005s
W20260709 12:23:19.810035 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 96747.7us
I20260709 12:23:19.810611 23649 raft_consensus.cc:515] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:19.812084 23530 raft_consensus.cc:2804] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:23:19.812536 23530 raft_consensus.cc:493] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:23:19.812593 23475 leader_election.cc:290] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:19.810083 23297 raft_consensus.cc:493] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:19.813621 23297 raft_consensus.cc:3060] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:19.813669 23620 consensus_queue.cc:1048] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:23:19.813006 23649 leader_election.cc:290] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 14 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:19.755136 23604 raft_consensus.cc:515] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:19.790273 23665 leader_election.cc:290] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:19.819154 23604 leader_election.cc:290] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 13 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:19.795204 23652 leader_election.cc:290] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 15 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:19.797118 23521 raft_consensus.cc:515] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:19.819864 23665 raft_consensus.cc:493] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:19.792488 23572 leader_election.cc:290] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:19.799638 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:19.820680 23665 raft_consensus.cc:515] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:19.822073 17835 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35040 (request call id 2777) took 1055 ms. Trace:
I20260709 12:23:19.822220 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:19.822872 17684 raft_consensus.cc:2468] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 14.
I20260709 12:23:19.823014 23521 leader_election.cc:290] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:19.799237 23552 leader_election.cc:290] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:19.813560 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:19.825116 23470 leader_election.cc:290] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 17 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:23:19.767690 23427 raft_consensus.cc:670] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da: failed to trigger leader election: Illegal state: only voting members can start elections: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } [suppressed 3 similar messages]
I20260709 12:23:19.803436 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:19.826749 17836 raft_consensus.cc:2393] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 16: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:23:19.818992 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:19.827654 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:19.805225 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:23:19.796594 23622 consensus_meta.cc:287] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.656s	user 0.009s	sys 0.001s
I20260709 12:23:19.783816 23494 raft_consensus.cc:515] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:19.813079 23530 raft_consensus.cc:3060] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:19.814075 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 20 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:19.809561 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:19.828771 17838 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35040 (request call id 2773) took 1076 ms. Trace:
I20260709 12:23:19.831823 23297 raft_consensus.cc:515] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:19.848201 23622 raft_consensus.cc:515] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:19.838456 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:23:19.840008 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:19.850422 17689 raft_consensus.cc:2468] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 15.
I20260709 12:23:19.850793 17832 raft_consensus.cc:2468] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 17.
I20260709 12:23:19.850381 23297 leader_election.cc:290] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 14 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:19.851722 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:19.832468 18366 raft_consensus.cc:2468] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 13.
I20260709 12:23:19.852869 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 14 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:23:19.840021 23651 consensus_meta.cc:287] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.845s	user 0.016s	sys 0.006s
I20260709 12:23:19.853626 23651 raft_consensus.cc:515] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:19.854027 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 12 candidate_status { last_received { term: 9 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:19.854276 17834 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35042 (request call id 2471) took 1116 ms. Trace:
I20260709 12:23:19.854787 17832 raft_consensus.cc:2410] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for term 12 because replica has last-logged OpId of term: 10 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 11.
I20260709 12:23:19.855820 17763 raft_consensus.cc:2368] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 16. Current term is 17.
I20260709 12:23:19.836231 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 16 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:23:19.858031 17829 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37994 (request call id 1108) took 1303 ms (client timeout 1719 ms). Trace:
I20260709 12:23:19.831408 23665 leader_election.cc:290] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:19.854687 17834 rpcz_store.cc:276] 0709 12:23:18.738145 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:18.738281 (+   136us) service_pool.cc:224] Handling call
0709 12:23:19.697197 (+958916us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:19.718558 (+ 21361us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:19.854092 (+135534us) spinlock_profiling.cc:243] Waited 959 ms on lock 0x7b5400089dd0. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a61f8ad 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:23:19.854219 (+   127us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2204605824}
I20260709 12:23:19.864928 17834 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3"
dest_uuid: "e063541542a94969898868bf6eb5ce1b"
mode: GRACEFUL
new_leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
 from {username='slave'} at 127.0.0.1:59820
I20260709 12:23:19.865514 17834 raft_consensus.cc:606] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 15 LEADER]: Received request to transfer leadership to TS e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:23:19.858412 17829 rpcz_store.cc:269] 0709 12:23:18.554142 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:18.554264 (+   122us) service_pool.cc:224] Handling call
0709 12:23:19.857874 (+1303610us) spinlock_profiling.cc:243] Waited 1.02 s on lock 0x7b540022b354. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:23:19.857981 (+   107us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2340711680}
I20260709 12:23:19.853096 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:19.867000 17758 raft_consensus.cc:2393] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 17: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:23:19.859485 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:19.811712 18333 leader_election.cc:304] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3
W20260709 12:23:19.868045 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 56413.5us
I20260709 12:23:19.868716 18333 leader_election.cc:304] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:19.870494 18333 leader_election.cc:304] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:19.850906 17833 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35042 (request call id 2484) took 1031 ms. Trace:
I20260709 12:23:19.839788 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:19.801023 23655 consensus_queue.cc:260] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:19.855614 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:19.871064 17833 rpcz_store.cc:276] 0709 12:23:18.819629 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:18.835984 (+ 16355us) service_pool.cc:224] Handling call
0709 12:23:19.850732 (+1014748us) spinlock_profiling.cc:243] Waited 185 ms on lock 0x7b54003345d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:23:19.850853 (+   121us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":425048448}
I20260709 12:23:19.822536 17835 rpcz_store.cc:276] 0709 12:23:18.766777 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:19.475205 (+708428us) service_pool.cc:224] Handling call
0709 12:23:19.475475 (+   270us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:19.732818 (+257343us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:19.821820 (+ 89002us) spinlock_profiling.cc:243] Waited 199 ms on lock 0x7b540008b6d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:23:19.822017 (+   197us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":458052352}
I20260709 12:23:19.871809 17832 raft_consensus.cc:2468] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 17.
I20260709 12:23:19.828967 17836 raft_consensus.cc:3060] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:19.851054 17838 rpcz_store.cc:276] 0709 12:23:18.752367 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:18.785474 (+ 33107us) service_pool.cc:224] Handling call
0709 12:23:18.785774 (+   300us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:19.743577 (+957803us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:19.828525 (+ 84948us) spinlock_profiling.cc:243] Waited 930 ms on lock 0x7b54000ac3d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:23:19.828717 (+   192us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2138740352}
I20260709 12:23:19.873781 17757 raft_consensus.cc:2468] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 15.
I20260709 12:23:19.862071 17652 leader_election.cc:304] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:19.735194 23695 raft_consensus.cc:2749] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
W20260709 12:23:19.886327 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 87150.7us
I20260709 12:23:19.887446 17838 raft_consensus.cc:1240] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 13. Current term is 14. Ops: []
I20260709 12:23:19.887629 17832 raft_consensus.cc:1275] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 13 index: 10. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
I20260709 12:23:19.873884 18333 leader_election.cc:304] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:19.888608 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:19.890372 17835 raft_consensus.cc:1275] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 16 index: 9. Preceding OpId from leader: term: 17 index: 10. (index mismatch)
I20260709 12:23:19.891098 17835 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35042 (request call id 2482) took 1091 ms. Trace:
I20260709 12:23:19.885059 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:19.891968 17727 leader_election.cc:304] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:19.892391 17758 raft_consensus.cc:3060] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:19.893777 17727 leader_election.cc:304] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:19.894971 17727 leader_election.cc:304] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:19.885421 17833 raft_consensus.cc:1275] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 10 index: 12. Preceding OpId from leader: term: 11 index: 13. (index mismatch)
I20260709 12:23:19.896411 17833 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35040 (request call id 2780) took 1098 ms. Trace:
I20260709 12:23:19.896777 17833 rpcz_store.cc:276] 0709 12:23:18.798033 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:19.884797 (+1086764us) service_pool.cc:224] Handling call
0709 12:23:19.885099 (+   302us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:19.896095 (+ 10996us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:19.896355 (+   260us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:19.886529 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 14 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:19.891474 17835 rpcz_store.cc:276] 0709 12:23:18.799910 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:19.889632 (+1089722us) service_pool.cc:224] Handling call
0709 12:23:19.889933 (+   301us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:19.890808 (+   875us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:19.891036 (+   228us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:19.899708 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:19.900352 17684 raft_consensus.cc:2468] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 15.
I20260709 12:23:19.897506 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:19.878372 17761 raft_consensus.cc:2468] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 17.
I20260709 12:23:19.902032 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:19.902426 23665 raft_consensus.cc:2804] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:23:19.903041 23665 raft_consensus.cc:493] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:19.903525 23665 raft_consensus.cc:3060] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:19.901885 23475 raft_consensus.cc:2804] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:23:19.888155 17832 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35042 (request call id 2483) took 1074 ms. Trace:
I20260709 12:23:19.908772 17683 raft_consensus.cc:1275] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 11 index: 8. Preceding OpId from leader: term: 14 index: 9. (index mismatch)
W20260709 12:23:19.839118 23689 log.cc:927] Time spent T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.089s	user 0.000s	sys 0.001s
W20260709 12:23:19.916392 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 49240.4us
I20260709 12:23:19.918028 18371 raft_consensus.cc:1240] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 12. Current term is 13. Ops: [10.11-12.12]
I20260709 12:23:19.918706 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:19.908483 17832 rpcz_store.cc:276] 0709 12:23:18.814357 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:19.887104 (+1072747us) service_pool.cc:224] Handling call
0709 12:23:19.887341 (+   237us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:19.887958 (+   617us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:19.888113 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:19.920872 17832 raft_consensus.cc:1275] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 13 index: 11. Preceding OpId from leader: term: 15 index: 12. (index mismatch)
I20260709 12:23:19.913509 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:19.922418 18333 leader_election.cc:304] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:23:19.922757 17758 raft_consensus.cc:2468] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 15.
I20260709 12:23:19.923421 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:19.708912 23513 raft_consensus.cc:515] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:19.924145 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:23:19.928473 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 66478.7us
I20260709 12:23:19.929471 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:19.929740 17652 leader_election.cc:304] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:19.933559 17681 raft_consensus.cc:1240] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 13. Current term is 14. Ops: []
I20260709 12:23:19.925973 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:19.832746 23631 leader_election.cc:290] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:19.843600 23511 leader_election.cc:290] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:19.744485 23490 raft_consensus.cc:515] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:19.935521 23525 raft_consensus.cc:2749] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20260709 12:23:19.976293 23702 log.cc:927] Time spent T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.189s	user 0.000s	sys 0.002s
I20260709 12:23:19.977178 23525 raft_consensus.cc:2804] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:23:19.977769 23525 raft_consensus.cc:697] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 10 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:23:19.952301 23672 raft_consensus.cc:2749] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:23:19.978731 23525 consensus_queue.cc:237] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:19.942564 23257 raft_consensus.cc:2804] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:23:19.936803 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:19.884146 23604 raft_consensus.cc:2804] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:23:19.938670 17801 leader_election.cc:304] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:19.938971 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:19.923360 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:19.913863 17727 leader_election.cc:304] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:23:19.925896 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 62340.1us
I20260709 12:23:19.876273 23526 raft_consensus.cc:2804] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:23:19.855845 23651 leader_election.cc:290] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 14 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:19.884308 23521 raft_consensus.cc:2804] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:23:19.875105 23427 raft_consensus.cc:2749] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:23:19.959564 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:19.875453 23605 consensus_queue.cc:1048] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.117s
I20260709 12:23:20.001892 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:20.002691 17683 raft_consensus.cc:3060] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:19.989625 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
W20260709 12:23:19.928006 23704 log.cc:927] Time spent T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.287s	user 0.000s	sys 0.004s
W20260709 12:23:20.003654 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 65066.3us
I20260709 12:23:19.990603 17836 raft_consensus.cc:2468] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 17.
I20260709 12:23:19.980142 23257 raft_consensus.cc:493] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:20.008718 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:23:19.990833 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 77023.4us
I20260709 12:23:19.987601 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 15 candidate_status { last_received { term: 14 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:19.917162 23475 raft_consensus.cc:493] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:19.916071 23624 raft_consensus.cc:493] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:20.010468 17727 leader_election.cc:304] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:19.997376 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 15 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:19.998880 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 15 candidate_status { last_received { term: 14 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:23:19.999119 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 43924.3us
W20260709 12:23:19.987653 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:20.006515 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 278645us
I20260709 12:23:20.019279 17728 leader_election.cc:304] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:20.007330 17836 raft_consensus.cc:1275] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 17 index: 11. (index mismatch)
I20260709 12:23:19.952991 23655 raft_consensus.cc:3060] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:20.010908 17833 raft_consensus.cc:1240] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 16. Current term is 17. Ops: []
I20260709 12:23:20.030860 18366 raft_consensus.cc:2393] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 15: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:23:20.004944 17801 leader_election.cc:304] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:23:20.032299 17655 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:20.000916 23056 raft_consensus.cc:2804] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:23:20.027141 17802 leader_election.cc:304] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:20.025166 17836 raft_consensus.cc:1275] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 12 index: 9. Preceding OpId from leader: term: 15 index: 10. (index mismatch)
I20260709 12:23:19.989704 23526 raft_consensus.cc:493] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:23:19.990010 23604 raft_consensus.cc:493] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:19.936642 23648 consensus_queue.cc:1048] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.074s
I20260709 12:23:20.017138 23257 raft_consensus.cc:3060] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:19.994084 23506 leader_election.cc:290] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:20.037321 17728 leader_election.cc:304] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:20.012121 17654 leader_election.cc:304] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:23:20.040081 23297 consensus_queue.cc:1059] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 13.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:23:20.016805 23546 raft_consensus.cc:2804] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:23:20.016716 23475 raft_consensus.cc:3060] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:20.040565 17836 raft_consensus.cc:1240] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 8. Current term is 9. Ops: []
I20260709 12:23:20.004041 23521 raft_consensus.cc:493] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:23:20.046900 17727 leader_election.cc:304] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:20.037072 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:20.066677 17836 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "d48b7de191684318aa08f60a7a395387"
dest_uuid: "e063541542a94969898868bf6eb5ce1b"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:35032
I20260709 12:23:19.981348 23518 consensus_queue.cc:1059] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 16.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.048s
I20260709 12:23:20.034672 23650 raft_consensus.cc:493] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:20.072494 17728 leader_election.cc:400] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 14 pre-election: Vote denied by peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071) with higher term. Message: Invalid argument: T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for earlier term 14. Current term is 15.
I20260709 12:23:20.076745 17728 leader_election.cc:304] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:20.009985 23672 consensus_queue.cc:1059] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.048s
I20260709 12:23:20.079631 17728 leader_election.cc:304] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:20.038772 23056 raft_consensus.cc:493] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:20.046573 23604 raft_consensus.cc:3060] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:20.042248 23469 raft_consensus.cc:2804] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:23:20.046444 23526 raft_consensus.cc:3060] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:23:20.093688 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 81654.8us
I20260709 12:23:20.117198 18366 raft_consensus.cc:1240] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 11. Current term is 12. Ops: []
I20260709 12:23:20.119114 18366 raft_consensus.cc:1275] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 12 index: 9. Preceding OpId from leader: term: 15 index: 10. (index mismatch)
I20260709 12:23:20.127777 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:20.128439 17687 raft_consensus.cc:3060] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:20.038817 23454 raft_consensus.cc:2804] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:23:20.008558 23246 consensus_queue.cc:1048] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.054s
I20260709 12:23:20.074682 23521 raft_consensus.cc:3060] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:20.028811 23624 raft_consensus.cc:515] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:20.137960 17837 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35042 (request call id 2468) took 1402 ms. Trace:
I20260709 12:23:20.068696 23705 raft_consensus.cc:2749] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:23:20.144732 17837 rpcz_store.cc:276] 0709 12:23:18.735773 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:18.735961 (+   188us) service_pool.cc:224] Handling call
0709 12:23:20.137754 (+1401793us) spinlock_profiling.cc:243] Waited 200 ms on lock 0x7b54002eb654. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:23:20.137888 (+   134us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":461064832}
I20260709 12:23:20.092427 23544 raft_consensus.cc:2749] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:23:20.150794 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:20.108131 23492 consensus_queue.cc:1048] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.118s
I20260709 12:23:20.042732 23491 consensus_queue.cc:1048] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.070s
I20260709 12:23:20.108196 23573 raft_consensus.cc:2804] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:23:20.066561 23502 consensus_queue.cc:1048] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.066s
I20260709 12:23:20.110445 23469 raft_consensus.cc:493] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:23:20.222337 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 128886us
I20260709 12:23:20.092747 23636 consensus_queue.cc:1048] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.074s
I20260709 12:23:20.103176 23652 consensus_queue.cc:1059] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.122s
I20260709 12:23:20.104979 23552 consensus_queue.cc:1048] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.129s
I20260709 12:23:20.143146 23673 raft_consensus.cc:2804] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:23:20.106313 23650 raft_consensus.cc:515] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:20.241640 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:20.299896 17758 raft_consensus.cc:3060] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Advancing to term 14
W20260709 12:23:20.242847 18367 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59886 (request call id 1615) took 1403 ms (client timeout 1734 ms). Trace:
I20260709 12:23:20.107743 23056 raft_consensus.cc:3060] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:20.243914 17687 raft_consensus.cc:2468] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 13.
I20260709 12:23:20.217479 23622 leader_election.cc:290] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 12 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:20.106590 23594 raft_consensus.cc:515] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:20.103710 23671 consensus_queue.cc:1048] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.154s
W20260709 12:23:20.271718 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 349700us
I20260709 12:23:20.235666 23506 raft_consensus.cc:2749] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:23:20.241179 23705 raft_consensus.cc:2804] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:23:20.234979 23494 leader_election.cc:290] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 17 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:20.227957 23374 consensus_queue.cc:1048] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.051s
I20260709 12:23:20.245059 23498 raft_consensus.cc:493] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:23:20.358505 17680 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37402 (request call id 2939) took 1153 ms. Trace:
I20260709 12:23:20.359028 17680 rpcz_store.cc:276] 0709 12:23:19.205516 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:19.208138 (+  2622us) service_pool.cc:224] Handling call
0709 12:23:19.208507 (+   369us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:19.660183 (+451676us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:20.358180 (+697997us) spinlock_profiling.cc:243] Waited 451 ms on lock 0x7b54003467d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:23:20.358416 (+   236us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1036615552}
W20260709 12:23:20.300858 18367 rpcz_store.cc:269] 0709 12:23:18.838998 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:18.971695 (+132697us) service_pool.cc:224] Handling call
0709 12:23:20.242651 (+1270956us) spinlock_profiling.cc:243] Waited 161 ms on lock 0x7b540047ca54. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:23:20.242781 (+   130us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":370876160}
I20260709 12:23:20.230916 23297 consensus_queue.cc:1059] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.052s
W20260709 12:23:20.310326 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 230882us
I20260709 12:23:20.317147 17758 raft_consensus.cc:2468] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 14.
W20260709 12:23:20.317839 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 270993us
I20260709 12:23:20.365432 17688 raft_consensus.cc:2393] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 15: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:23:20.282302 23469 raft_consensus.cc:3060] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:20.331538 18333 leader_election.cc:400] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 16 pre-election: Vote denied by peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651) with higher term. Message: Invalid argument: T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 16. Current term is 17.
I20260709 12:23:20.369164 18333 leader_election.cc:403] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 16 pre-election: Cancelling election due to peer responding with higher term
W20260709 12:23:20.321792 23679 log.cc:927] Time spent T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.052s	user 0.001s	sys 0.000s
W20260709 12:23:20.337452 23496 consensus_meta.cc:287] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.531s	user 0.011s	sys 0.000s
I20260709 12:23:20.297909 23513 leader_election.cc:290] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 14 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:20.337795 23673 raft_consensus.cc:697] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 13 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:23:20.343786 17838 raft_consensus.cc:2468] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 17.
I20260709 12:23:20.346804 17835 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35040 (request call id 2787) took 1223 ms. Trace:
W20260709 12:23:20.370990 17727 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.16.29.67:36837 timed out after 1.916s (SENT) [suppressed 29 similar messages]
I20260709 12:23:20.371737 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:23:20.379110 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 48284.2us
I20260709 12:23:20.289357 23530 raft_consensus.cc:515] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:20.429992 17835 rpcz_store.cc:276] 0709 12:23:19.122969 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:19.906277 (+783308us) service_pool.cc:224] Handling call
0709 12:23:19.906614 (+   337us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:19.906834 (+   220us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 12:23:20.036505 (+129671us) log.cc:844] Serialized 14 byte log entry
0709 12:23:20.346339 (+309834us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:20.346414 (+    75us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:23:20.346478 (+    64us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:20.346573 (+    95us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:20.346733 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":18588,"prepare.queue_time_us":234454,"prepare.run_cpu_time_us":60,"prepare.run_wall_time_us":2668,"thread_start_us":419355,"threads_started":3}
I20260709 12:23:20.430233 17683 raft_consensus.cc:2468] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 14.
I20260709 12:23:20.451859 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 14 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:20.452693 18367 raft_consensus.cc:3060] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Advancing to term 14
I20260709 12:23:20.454669 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:20.444017 23454 raft_consensus.cc:697] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 15 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:23:20.455956 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:20.456791 17837 raft_consensus.cc:2468] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 15.
I20260709 12:23:20.455394 17838 raft_consensus.cc:3060] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:20.457523 23454 consensus_queue.cc:237] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.8, Last appended by leader: 8, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:20.467474 17838 raft_consensus.cc:2468] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 13.
W20260709 12:23:20.444908 23696 log.cc:927] Time spent T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.224s	user 0.001s	sys 0.000s
I20260709 12:23:20.472249 17835 raft_consensus.cc:1275] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 11 index: 8. Preceding OpId from leader: term: 12 index: 9. (index mismatch)
I20260709 12:23:20.440872 23494 raft_consensus.cc:2749] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:23:20.473470 18374 raft_consensus.cc:1240] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 15. Current term is 16. Ops: []
I20260709 12:23:20.426064 23723 raft_consensus.cc:3060] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Advancing to term 17
I20260709 12:23:20.441758 23709 raft_consensus.cc:2749] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:23:20.435765 23496 raft_consensus.cc:515] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:20.484930 18362 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59886 (request call id 1609) took 1930 ms. Trace:
I20260709 12:23:20.499038 18362 rpcz_store.cc:276] 0709 12:23:18.554966 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:18.555384 (+   418us) service_pool.cc:224] Handling call
0709 12:23:19.264021 (+708637us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:19.264273 (+   252us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:19.264311 (+    38us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:19.264372 (+    61us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:23:19.264440 (+    68us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:19.264525 (+    85us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:20.484730 (+1220205us) spinlock_profiling.cc:243] Waited 708 ms on lock 0x7b5400467b50. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a61f8ad 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:23:20.484864 (+   134us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1628830080}
I20260709 12:23:20.501382 18362 raft_consensus.cc:1275] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 17 index: 15. Preceding OpId from leader: term: 19 index: 16. (index mismatch)
I20260709 12:23:20.370282 23498 raft_consensus.cc:515] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:20.507082 17835 raft_consensus.cc:1240] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 13. Current term is 14. Ops: [13.11-13.11]
I20260709 12:23:20.508266 17652 leader_election.cc:304] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:23:20.493876 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 130616us
I20260709 12:23:20.509442 23297 raft_consensus.cc:2804] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:23:20.510138 23297 raft_consensus.cc:697] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 17 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:23:20.510052 17728 leader_election.cc:304] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:20.511022 23293 consensus_queue.cc:1059] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 12.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 12:23:20.511365 23297 consensus_queue.cc:237] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.12, Last appended by leader: 12, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:20.483232 18367 raft_consensus.cc:2468] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 14.
I20260709 12:23:20.517885 23624 leader_election.cc:290] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:23:20.521952 23711 log.cc:927] Time spent T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.355s	user 0.000s	sys 0.001s
W20260709 12:23:20.526369 23708 log.cc:927] Time spent T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.300s	user 0.001s	sys 0.000s
I20260709 12:23:20.530009 23594 leader_election.cc:290] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 15 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:20.531656 23496 leader_election.cc:290] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 20 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:20.504891 23671 raft_consensus.cc:493] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:23:20.536345 23671 raft_consensus.cc:515] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:20.534173 17728 leader_election.cc:304] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:23:20.522321 23494 raft_consensus.cc:493] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:20.540385 23494 raft_consensus.cc:515] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:20.541550 23469 raft_consensus.cc:515] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:20.542815 23498 leader_election.cc:290] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:20.543224 23630 raft_consensus.cc:993] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b: : Instructing follower e2909cff1da44ad387f96a8ae3aa5afb to start an election
I20260709 12:23:20.543494 23494 leader_election.cc:290] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:20.543824 23650 leader_election.cc:290] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:20.545318 23671 leader_election.cc:290] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:23:20.518937 23654 consensus_meta.cc:287] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.964s	user 0.012s	sys 0.000s
I20260709 12:23:20.561110 23654 raft_consensus.cc:515] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:20.561479 23475 raft_consensus.cc:515] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:20.562902 17758 raft_consensus.cc:2393] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 17: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:23:20.563316 23654 leader_election.cc:290] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 18 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:20.563470 23604 raft_consensus.cc:515] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:20.564188 23530 leader_election.cc:290] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 16 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:20.564015 23737 consensus_queue.cc:1059] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 12.11, Next index: 12, Last known committed idx: 10, Time since last communication: 0.042s
I20260709 12:23:20.565394 18364 raft_consensus.cc:1240] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 8. Current term is 9. Ops: []
I20260709 12:23:20.565414 23732 raft_consensus.cc:3055] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:23:20.566046 23732 raft_consensus.cc:740] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 12 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
W20260709 12:23:20.527673 23609 consensus_meta.cc:287] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.843s	user 0.005s	sys 0.005s
I20260709 12:23:20.543496 23739 raft_consensus.cc:493] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:20.548871 17760 raft_consensus.cc:1240] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 17. Current term is 18. Ops: []
I20260709 12:23:20.543865 23630 raft_consensus.cc:1081] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 15 LEADER]: Signalling peer e2909cff1da44ad387f96a8ae3aa5afb to start an election
W20260709 12:23:20.549525 23634 consensus_meta.cc:287] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.956s	user 0.010s	sys 0.002s
I20260709 12:23:20.518414 23673 consensus_queue.cc:237] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.12, Last appended by leader: 12, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:20.555305 23665 consensus_meta.cc:287] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.651s	user 0.010s	sys 0.000s
I20260709 12:23:20.555919 23728 raft_consensus.cc:2749] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:23:20.558231 23056 raft_consensus.cc:515] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:20.559194 23521 raft_consensus.cc:515] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:20.545369 23520 raft_consensus.cc:3055] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:23:20.597025 18333 leader_election.cc:304] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:20.555517 23469 leader_election.cc:290] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 18 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:20.613835 23737 raft_consensus.cc:2804] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Leader pre-election won for term 15
W20260709 12:23:20.617247 23522 consensus_meta.cc:287] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 1.142s	user 0.006s	sys 0.006s
I20260709 12:23:20.656980 23522 raft_consensus.cc:515] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:23:20.618098 23546 raft_consensus.cc:697] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 17 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
W20260709 12:23:20.642071 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 45176.4us
I20260709 12:23:20.658710 23546 consensus_queue.cc:237] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 16.14, Last appended by leader: 14, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:20.659569 18333 leader_election.cc:304] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:23:20.660535 18333 leader_election.cc:304] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:23:20.661206 23620 raft_consensus.cc:2804] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:23:20.661700 23620 raft_consensus.cc:697] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 13 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:23:20.646323 23609 raft_consensus.cc:2749] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:23:20.616961 23634 raft_consensus.cc:515] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:20.652057 23526 consensus_meta.cc:287] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.506s	user 0.010s	sys 0.003s
I20260709 12:23:20.685587 23526 raft_consensus.cc:515] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:20.685393 23257 consensus_meta.cc:287] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.599s	user 0.009s	sys 0.002s
W20260709 12:23:20.654153 23655 consensus_meta.cc:287] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.618s	user 0.010s	sys 0.002s
I20260709 12:23:20.566861 23732 consensus_queue.cc:260] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:20.686888 23257 raft_consensus.cc:515] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:23:20.654932 23521 leader_election.cc:290] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 13 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:20.650219 23737 raft_consensus.cc:493] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:20.638352 23665 raft_consensus.cc:515] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:23:20.616350 23739 raft_consensus.cc:515] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:20.661870 23706 consensus_queue.cc:1048] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.015s
I20260709 12:23:20.677688 23620 consensus_queue.cc:237] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.10, Last appended by leader: 10, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:20.641314 23520 raft_consensus.cc:740] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:23:20.941298 23546 raft_consensus.cc:493] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:20.942389 23604 leader_election.cc:290] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 9 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:20.943161 23521 raft_consensus.cc:493] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
W20260709 12:23:20.812517 17764 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.670s	user 0.001s	sys 0.002s
W20260709 12:23:20.944542 17764 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.802s	user 0.002s	sys 0.002s
I20260709 12:23:20.945405 23522 leader_election.cc:290] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 16 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:20.943758 23604 raft_consensus.cc:493] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:20.813516 17761 raft_consensus.cc:1240] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 17. Current term is 18. Ops: []
W20260709 12:23:20.813799 17757 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38786 (request call id 3209) took 1503 ms (client timeout 1722 ms). Trace:
I20260709 12:23:20.946597 23604 raft_consensus.cc:515] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:20.570870 17728 leader_election.cc:304] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:20.942902 23546 raft_consensus.cc:515] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:20.950640 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 379782us
I20260709 12:23:20.818579 17759 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38776 (request call id 2517) took 1011 ms. Trace:
W20260709 12:23:20.727934 23692 log.cc:927] Time spent T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 1.139s	user 0.002s	sys 0.003s
I20260709 12:23:20.837157 17756 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38776 (request call id 2502) took 1740 ms. Trace:
I20260709 12:23:20.835063 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:20.845571 17832 raft_consensus.cc:1240] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 12. Current term is 13. Ops: []
I20260709 12:23:20.568166 17801 leader_election.cc:304] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:20.848294 17833 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35040 (request call id 2782) took 1755 ms. Trace:
W20260709 12:23:20.848409 17802 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.743s	user 0.000s	sys 0.010s
I20260709 12:23:20.855901 17836 raft_consensus.cc:606] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Received request to transfer leadership
I20260709 12:23:20.856284 17837 raft_consensus.cc:1240] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 17 LEADER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 16. Current term is 17. Ops: []
I20260709 12:23:20.856295 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:20.857707 17830 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35040 (request call id 2774) took 2104 ms. Trace:
I20260709 12:23:20.891522 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:20.894418 18365 raft_consensus.cc:1275] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 11 index: 9. Preceding OpId from leader: term: 12 index: 10. (index mismatch)
I20260709 12:23:20.894294 17654 leader_election.cc:304] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b
W20260709 12:23:20.894804 17587 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:37172 (request call id 256) took 23691 ms (client timeout 15000 ms). Trace:
I20260709 12:23:20.905498 18367 raft_consensus.cc:1240] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 13. Current term is 14. Ops: []
I20260709 12:23:20.657420 23056 leader_election.cc:290] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 18 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:20.912315 23293 raft_consensus.cc:493] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:21.008760 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 440507us
I20260709 12:23:21.008457 23293 raft_consensus.cc:515] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:21.009725 23594 raft_consensus.cc:493] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:21.012043 23748 raft_consensus.cc:2804] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:23:20.914125 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:21.013609 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 20 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:21.014693 18365 raft_consensus.cc:3060] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:21.015020 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:20.918298 23257 leader_election.cc:290] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 16 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:21.016096 22853 raft_consensus.cc:3055] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:23:21.016727 22853 raft_consensus.cc:740] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:23:21.017081 23257 consensus_queue.cc:1048] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:23:20.941111 17803 leader_election.cc:304] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:23:20.946997 17757 rpcz_store.cc:269] 0709 12:23:19.310670 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:19.350381 (+ 39711us) service_pool.cc:224] Handling call
0709 12:23:20.813282 (+1462901us) spinlock_profiling.cc:243] Waited 420 ms on lock 0x7b5400415fd4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:23:20.813568 (+   286us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":966547200}
I20260709 12:23:20.814966 17762 raft_consensus.cc:1240] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 14. Current term is 15. Ops: [14.11-14.11]
I20260709 12:23:20.998143 17756 rpcz_store.cc:276] 0709 12:23:19.097078 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:19.097270 (+   192us) service_pool.cc:224] Handling call
0709 12:23:19.114218 (+ 16948us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:20.533817 (+1419599us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:20.533849 (+    32us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:20.533895 (+    46us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:23:20.533936 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:20.836511 (+302575us) spinlock_profiling.cc:243] Waited 1.42 s on lock 0x7b54000fc8d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:23:20.836594 (+    83us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:20.836919 (+   325us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3264287104}
I20260709 12:23:20.951213 23546 leader_election.cc:290] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:20.952260 23526 leader_election.cc:290] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 13 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:20.815380 17760 raft_consensus.cc:1275] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 13 index: 9. Preceding OpId from leader: term: 15 index: 10. (index mismatch)
I20260709 12:23:20.953728 23630 consensus_queue.cc:1059] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 9.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:23:20.712847 23665 leader_election.cc:290] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 16 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:20.732326 23668 raft_consensus.cc:3055] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:23:21.026892 23656 consensus_queue.cc:1048] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:23:21.027648 23668 raft_consensus.cc:740] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:23:21.029855 23690 raft_consensus.cc:493] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:21.029630 23668 consensus_queue.cc:260] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:20.720419 23686 log.cc:927] Time spent T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 1.169s	user 0.004s	sys 0.000s
I20260709 12:23:21.030972 23690 raft_consensus.cc:515] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:20.821664 17688 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37396 (request call id 2556) took 1013 ms. Trace:
I20260709 12:23:21.033236 17688 rpcz_store.cc:276] 0709 12:23:19.807342 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:19.808014 (+   672us) service_pool.cc:224] Handling call
0709 12:23:20.806860 (+998846us) spinlock_profiling.cc:243] Waited 556 ms on lock 0x7b54003507d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:23:20.820872 (+ 14012us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1279564032}
W20260709 12:23:20.728884 23703 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.540s	user 0.005s	sys 0.006s
I20260709 12:23:21.034386 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
W20260709 12:23:21.034722 23703 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.900s	user 0.005s	sys 0.007s
I20260709 12:23:21.018299 22853 consensus_queue.cc:260] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.9, Last appended by leader: 9, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:23:21.035694 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 163856us
W20260709 12:23:20.718940 23707 log.cc:927] Time spent T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.672s	user 0.000s	sys 0.001s
W20260709 12:23:21.037169 17655 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.643s	user 0.012s	sys 0.056s
I20260709 12:23:21.037698 23740 raft_consensus.cc:2749] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
W20260709 12:23:21.037664 17655 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.708s	user 0.013s	sys 0.056s
W20260709 12:23:20.738234 23713 log.cc:927] Time spent T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.379s	user 0.001s	sys 0.001s
I20260709 12:23:21.039047 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 12 candidate_status { last_received { term: 10 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:21.040002 18363 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 1817) took 1409 ms. Trace:
I20260709 12:23:21.040060 18367 raft_consensus.cc:2468] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 11.
W20260709 12:23:20.741617 23714 log.cc:927] Time spent T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.299s	user 0.002s	sys 0.000s
I20260709 12:23:21.042024 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
W20260709 12:23:20.741870 23721 log.cc:927] Time spent T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.385s	user 0.002s	sys 0.000s
I20260709 12:23:20.725937 23723 raft_consensus.cc:2749] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Leader pre-election lost for term 16. Reason: Vote denied by peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651) with higher term. Message: Invalid argument: T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 16. Current term is 17.
I20260709 12:23:21.053328 18361 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59886 (request call id 1629) took 1191 ms. Trace:
I20260709 12:23:21.053412 23624 raft_consensus.cc:2804] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:23:21.036922 22853 raft_consensus.cc:3060] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:20.689683 23732 raft_consensus.cc:3060] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:20.690860 23737 raft_consensus.cc:3060] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:20.716342 23739 leader_election.cc:290] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:20.965797 23695 raft_consensus.cc:2749] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:23:20.955910 17759 rpcz_store.cc:276] 0709 12:23:19.806663 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:20.495021 (+688358us) service_pool.cc:224] Handling call
0709 12:23:20.495319 (+   298us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:20.497851 (+  2532us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:20.497908 (+    57us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:20.497964 (+    56us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:23:20.498022 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:20.817805 (+319783us) spinlock_profiling.cc:243] Waited 2.28 ms on lock 0x7b540009f1d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:23:20.817877 (+    72us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:20.818201 (+   324us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5246208}
I20260709 12:23:20.990252 23521 raft_consensus.cc:515] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:20.947744 23475 leader_election.cc:290] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 17 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:21.001812 17836 raft_consensus.cc:614] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:23:20.999688 17830 rpcz_store.cc:276] 0709 12:23:18.752632 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:19.152706 (+400074us) service_pool.cc:224] Handling call
0709 12:23:19.160666 (+  7960us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:19.181936 (+ 21270us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:20.857026 (+1675090us) spinlock_profiling.cc:243] Waited 20.6 ms on lock 0x7b540008a2d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:23:20.857467 (+   441us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":47392512}
I20260709 12:23:21.007336 23749 raft_consensus.cc:493] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:23:21.012971 23748 raft_consensus.cc:697] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 14 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
W20260709 12:23:21.004719 17802 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.899s	user 0.000s	sys 0.011s
I20260709 12:23:20.951159 23604 leader_election.cc:290] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:21.018072 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:20.938648 23520 consensus_queue.cc:260] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 13.9, Last appended by leader: 9, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:20.999209 17833 rpcz_store.cc:276] 0709 12:23:19.092651 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:19.914046 (+821395us) service_pool.cc:224] Handling call
0709 12:23:20.005152 (+ 91106us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:20.031071 (+ 25919us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:20.847311 (+816240us) spinlock_profiling.cc:243] Waited 90.8 ms on lock 0x7b54000abed0. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a61f8ad 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:23:20.847913 (+   602us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":208811776}
I20260709 12:23:21.026082 23630 raft_consensus.cc:3055] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:23:21.071367 23630 raft_consensus.cc:740] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 12 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
W20260709 12:23:21.071658 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 997912us
I20260709 12:23:21.038031 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:21.073028 17802 leader_election.cc:304] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:21.073393 17838 raft_consensus.cc:2468] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 16.
W20260709 12:23:20.720788 23710 log.cc:927] Time spent T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.480s	user 0.002s	sys 0.000s
I20260709 12:23:21.073180 23630 consensus_queue.cc:260] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 12.12, Last appended by leader: 12, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:20.736303 23719 log.cc:927] Time spent T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.372s	user 0.003s	sys 0.000s
I20260709 12:23:21.076473 17831 raft_consensus.cc:2393] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 16: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:23:21.038787 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:21.017958 23530 raft_consensus.cc:493] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:21.037245 17803 leader_election.cc:304] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:21.037011 18368 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59886 (request call id 1614) took 2298 ms. Trace:
I20260709 12:23:21.011777 23293 leader_election.cc:290] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:21.046906 17764 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36572 (request call id 1005) took 1338 ms. Trace:
W20260709 12:23:20.740018 23725 log.cc:927] Time spent T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.248s	user 0.000s	sys 0.001s
W20260709 12:23:21.053279 17727 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.571s	user 0.001s	sys 0.017s
I20260709 12:23:21.040678 18363 rpcz_store.cc:276] 0709 12:23:19.630314 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:19.630479 (+   165us) service_pool.cc:224] Handling call
0709 12:23:19.630811 (+   332us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:19.677841 (+ 47030us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:19.677871 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:19.678355 (+   484us) log.cc:844] Serialized 26 byte log entry
0709 12:23:19.909239 (+230884us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:23:19.909282 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:19.909343 (+    61us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:20.718077 (+808734us) spinlock_profiling.cc:243] Waited 5.58 ms on lock 0x7b4400022b4c. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b47459c85 00007f7b4745a6a2 00007f7b4a1cc1db 00007f7b4a1c7aaf 00007f7b4a1bb86e 00007f7b4a1babb6 00007f7b4a1ba7d8 00007f7b4a1c4221 00007f7b4a1c41d9 00007f7b4a1c4169
0709 12:23:21.039285 (+321208us) raft_consensus.cc:1685] finished
0709 12:23:21.039322 (+    37us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:21.039756 (+   434us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":46617,"raft.queue_time_us":48146,"raft.run_cpu_time_us":84,"raft.run_wall_time_us":85,"spinlock_wait_cycles":12828032,"thread_start_us":230612,"threads_started":1,"wal-append.queue_time_us":328591}
I20260709 12:23:21.058355 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:21.048564 23668 raft_consensus.cc:3060] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:21.054116 18361 rpcz_store.cc:276] 0709 12:23:19.861913 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:19.862059 (+   146us) service_pool.cc:224] Handling call
0709 12:23:19.862389 (+   330us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:19.862576 (+   187us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:19.862598 (+    22us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:19.886742 (+ 24144us) log.cc:844] Serialized 26 byte log entry
0709 12:23:20.108902 (+222160us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:23:20.108956 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:20.109047 (+    91us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:20.741016 (+631969us) spinlock_profiling.cc:243] Waited 117 ms on lock 0x7b4400022b4c. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b47459c85 00007f7b4745a6a2 00007f7b4a1cc1db 00007f7b4a1c7aaf 00007f7b4a1bb86e 00007f7b4a1babb6 00007f7b4a1ba7d8 00007f7b4a1c4221 00007f7b4a1c41d9 00007f7b4a1c4169
0709 12:23:21.052585 (+311569us) raft_consensus.cc:1685] finished
0709 12:23:21.052634 (+    49us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:21.053138 (+   504us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":269043968,"thread_start_us":131105,"threads_started":1,"wal-append.queue_time_us":391452}
W20260709 12:23:21.081224 23760 log.cc:927] Time spent T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.084s	user 0.003s	sys 0.000s
I20260709 12:23:21.010823 23594 raft_consensus.cc:515] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:21.079282 23526 raft_consensus.cc:2749] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:23:21.104291 17834 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35042 (request call id 2492) took 1682 ms. Trace:
I20260709 12:23:21.097975 17755 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38776 (request call id 2511) took 1832 ms. Trace:
I20260709 12:23:21.034354 23690 leader_election.cc:290] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:23:21.069113 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 54371.6us
I20260709 12:23:21.067054 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:21.073604 17758 raft_consensus.cc:2393] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 15: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:23:21.042964 18367 raft_consensus.cc:3060] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:23:21.083003 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 189270us
I20260709 12:23:21.085893 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
W20260709 12:23:21.060388 23705 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.598s	user 0.018s	sys 0.018s
W20260709 12:23:21.122865 23705 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.681s	user 0.019s	sys 0.019s
I20260709 12:23:21.123500 17759 raft_consensus.cc:2393] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 15: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:23:21.124682 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:21.124913 23521 leader_election.cc:290] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:21.080993 18368 rpcz_store.cc:276] 0709 12:23:18.738711 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:19.137381 (+398670us) service_pool.cc:224] Handling call
0709 12:23:19.153798 (+ 16417us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:19.154157 (+   359us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 12:23:19.172919 (+ 18762us) log.cc:844] Serialized 14 byte log entry
0709 12:23:19.876802 (+703883us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:19.884595 (+  7793us) log.cc:844] Serialized 26 byte log entry
0709 12:23:19.884917 (+   322us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:23:19.884968 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:19.885073 (+   105us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:20.719672 (+834599us) spinlock_profiling.cc:243] Waited 2.94 ms on lock 0x7b4400022b4c. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b47459c85 00007f7b4745a6a2 00007f7b4a1cc1db 00007f7b4a1c7aaf 00007f7b4a1bb86e 00007f7b4a1babb6 00007f7b4a1ba7d8 00007f7b4a1c4221 00007f7b4a1c41d9 00007f7b4a1c4169
0709 12:23:21.036105 (+316433us) raft_consensus.cc:1685] finished
0709 12:23:21.036256 (+   151us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:21.036668 (+   412us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":308276,"prepare.queue_time_us":19059,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":40,"raft.queue_time_us":290189,"raft.run_cpu_time_us":188,"raft.run_wall_time_us":4085,"spinlock_wait_cycles":6754560,"thread_start_us":413513,"threads_started":2,"wal-append.queue_time_us":258666}
W20260709 12:23:21.095513 17727 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.665s	user 0.001s	sys 0.017s
I20260709 12:23:21.087540 23530 raft_consensus.cc:515] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:21.123353 23705 raft_consensus.cc:697] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:23:21.097710 23741 raft_consensus.cc:493] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:23:21.128872 23497 raft_consensus.cc:2804] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Leader election won for term 14
W20260709 12:23:21.129115 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 758110us
I20260709 12:23:21.128916 17759 raft_consensus.cc:2393] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 16: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:23:21.130642 23752 raft_consensus.cc:493] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:23:21.131760 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:21.132128 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:21.131629 23497 raft_consensus.cc:697] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 14 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:23:21.132714 17759 raft_consensus.cc:3060] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:21.109337 17755 rpcz_store.cc:276] 0709 12:23:19.255326 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:19.276884 (+ 21558us) service_pool.cc:224] Handling call
0709 12:23:19.277340 (+   456us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:19.301179 (+ 23839us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:19.301221 (+    42us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:19.321837 (+ 20616us) log.cc:844] Serialized 26 byte log entry
0709 12:23:19.466444 (+144607us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:23:19.466490 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:19.466557 (+    67us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:20.715252 (+1248695us) spinlock_profiling.cc:243] Waited 6.72 ms on lock 0x7b4400022b4c. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b47459c85 00007f7b4745a6a2 00007f7b4a1cc1db 00007f7b4a1c7aaf 00007f7b4a1bb86e 00007f7b4a1babb6 00007f7b4a1ba7d8 00007f7b4a1c4221 00007f7b4a1c41d9 00007f7b4a1c4169
0709 12:23:21.082636 (+367384us) raft_consensus.cc:1685] finished
0709 12:23:21.082686 (+    50us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:21.088059 (+  5373us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":30435,"raft.queue_time_us":32737,"raft.run_cpu_time_us":236,"raft.run_wall_time_us":5509,"spinlock_wait_cycles":15457152,"thread_start_us":93464,"threads_started":1,"wal-append.queue_time_us":155058}
I20260709 12:23:21.133948 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:21.133744 23497 consensus_queue.cc:237] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 12.12, Last appended by leader: 12, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:21.130242 23705 consensus_queue.cc:237] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.9, Last appended by leader: 9, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:21.134783 17755 raft_consensus.cc:2468] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 10.
I20260709 12:23:21.107561 17834 rpcz_store.cc:276] 0709 12:23:19.421629 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:19.930898 (+509269us) service_pool.cc:224] Handling call
0709 12:23:19.931308 (+   410us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:19.931709 (+   401us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:19.931747 (+    38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:19.932936 (+  1189us) log.cc:844] Serialized 26 byte log entry
0709 12:23:20.170900 (+237964us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:23:20.170959 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:20.171060 (+   101us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:20.735468 (+564408us) spinlock_profiling.cc:243] Waited 98.1 ms on lock 0x7b4400022b4c. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b47459c85 00007f7b4745a6a2 00007f7b4a1cc1db 00007f7b4a1c7aaf 00007f7b4a1bb86e 00007f7b4a1babb6 00007f7b4a1ba7d8 00007f7b4a1c4221 00007f7b4a1c41d9 00007f7b4a1c4169
0709 12:23:21.103623 (+368155us) raft_consensus.cc:1685] finished
0709 12:23:21.103689 (+    66us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:21.104083 (+   394us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":5710,"raft.run_cpu_time_us":153,"raft.run_wall_time_us":9257,"spinlock_wait_cycles":225561344,"thread_start_us":237471,"threads_started":1,"wal-append.queue_time_us":363538}
I20260709 12:23:21.135223 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:21.132155 18364 raft_consensus.cc:3060] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:21.137605 17757 raft_consensus.cc:2393] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 16: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:23:21.138111 23594 leader_election.cc:290] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:21.112394 23604 raft_consensus.cc:493] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:21.140444 23604 raft_consensus.cc:515] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:21.140705 17757 raft_consensus.cc:1240] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 16. Current term is 17. Ops: [16.14-16.14]
I20260709 12:23:21.139560 17755 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38786 (request call id 3228) took 1096 ms. Trace:
I20260709 12:23:21.142994 23604 leader_election.cc:290] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:21.144331 23630 raft_consensus.cc:3060] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:23:21.151257 23759 log.cc:927] Time spent T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.166s	user 0.000s	sys 0.002s
I20260709 12:23:21.087387 17764 rpcz_store.cc:276] 0709 12:23:19.709175 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:19.855909 (+146734us) service_pool.cc:224] Handling call
0709 12:23:19.856231 (+   322us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:19.856611 (+   380us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:19.856641 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:19.857089 (+   448us) log.cc:844] Serialized 26 byte log entry
0709 12:23:20.078348 (+221259us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:23:20.737291 (+658943us) spinlock_profiling.cc:243] Waited 115 ms on lock 0x7b4400022b4c. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b47459c85 00007f7b4745a6a2 00007f7b4a1cc1db 00007f7b4a1c7aaf 00007f7b4a1bb86e 00007f7b4a1babb6 00007f7b4a1ba7d8 00007f7b4a1c4221 00007f7b4a1c41d9 00007f7b4a1c4169
0709 12:23:20.945195 (+207904us) log.cc:844] Serialized 14 byte log entry
0709 12:23:20.945885 (+   690us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:20.946069 (+   184us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:21.043099 (+ 97030us) raft_consensus.cc:1685] finished
0709 12:23:21.046537 (+  3438us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:21.046836 (+   299us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":6842,"prepare.queue_time_us":199831,"prepare.run_cpu_time_us":28,"prepare.run_wall_time_us":29,"raft.queue_time_us":20724,"raft.run_cpu_time_us":419,"raft.run_wall_time_us":5096,"spinlock_wait_cycles":264057088,"thread_start_us":1023721,"threads_started":2,"wal-append.queue_time_us":428485}
I20260709 12:23:21.143711 17755 rpcz_store.cc:276] 0709 12:23:20.043626 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:21.136889 (+1093263us) service_pool.cc:224] Handling call
0709 12:23:21.138795 (+  1906us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:21.139136 (+   341us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:21.139168 (+    32us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:21.139235 (+    67us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:23:21.139270 (+    35us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:21.139349 (+    79us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:21.139472 (+   123us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:21.118860 17763 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36572 (request call id 1006) took 1372 ms. Trace:
W20260709 12:23:21.153524 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 108109us
W20260709 12:23:21.155613 23758 log.cc:927] Time spent T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.132s	user 0.002s	sys 0.000s
W20260709 12:23:21.160030 17652 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:21.166457 17829 raft_consensus.cc:1240] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 8. Current term is 9. Ops: []
I20260709 12:23:21.169448 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 16 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:23:21.172633 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 135749us
I20260709 12:23:21.172168 23748 consensus_queue.cc:237] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.8, Last appended by leader: 8, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:21.173625 17803 leader_election.cc:304] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:21.099254 23749 raft_consensus.cc:515] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:21.189313 17801 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:21.193094 17757 raft_consensus.cc:1275] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 15 index: 13. Preceding OpId from leader: term: 16 index: 14. (index mismatch)
I20260709 12:23:21.194334 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:21.125540 23521 raft_consensus.cc:493] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:23:21.154843 17763 rpcz_store.cc:276] 0709 12:23:19.709693 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:19.857276 (+147583us) service_pool.cc:224] Handling call
0709 12:23:19.857603 (+   327us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:19.857922 (+   319us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:19.857946 (+    24us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:19.858371 (+   425us) log.cc:844] Serialized 26 byte log entry
0709 12:23:20.093607 (+235236us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:23:20.093650 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:20.093722 (+    72us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:20.740647 (+646925us) spinlock_profiling.cc:243] Waited 25.9 ms on lock 0x7b4400022b4c. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b47459c85 00007f7b4745a6a2 00007f7b4a1cc1db 00007f7b4a1c7aaf 00007f7b4a1bb86e 00007f7b4a1babb6 00007f7b4a1ba7d8 00007f7b4a1c4221 00007f7b4a1c41d9 00007f7b4a1c4169
0709 12:23:21.057672 (+317025us) raft_consensus.cc:1685] finished
0709 12:23:21.057739 (+    67us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:21.081599 (+ 23860us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":80,"raft.run_cpu_time_us":76,"raft.run_wall_time_us":73,"spinlock_wait_cycles":59467520,"thread_start_us":234442,"threads_started":1,"wal-append.queue_time_us":448638}
W20260709 12:23:21.212812 17655 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:21.219247 23697 consensus_queue.cc:1059] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 16.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.031s
I20260709 12:23:21.219973 23739 raft_consensus.cc:2749] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 12:23:21.220871 23697 raft_consensus.cc:3055] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:23:21.221458 23697 raft_consensus.cc:740] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
W20260709 12:23:21.224059 23761 log.cc:927] Time spent T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.095s	user 0.001s	sys 0.000s
I20260709 12:23:21.226392 17728 leader_election.cc:304] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:21.226604 18371 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 1824) took 1110 ms. Trace:
I20260709 12:23:21.228174 17758 raft_consensus.cc:2368] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for earlier term 16. Current term is 17.
I20260709 12:23:21.228394 23742 raft_consensus.cc:493] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:21.229714 17758 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3"
dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
 from {username='slave'} at 127.0.0.1:38786
I20260709 12:23:21.229530 23742 raft_consensus.cc:515] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:21.230584 17758 raft_consensus.cc:493] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:23:21.231026 17758 raft_consensus.cc:3060] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:21.214471 17652 leader_election.cc:304] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:23:21.208494 23546 raft_consensus.cc:2749] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:23:21.238859 23741 raft_consensus.cc:515] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:21.222968 23697 consensus_queue.cc:260] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.12, Last appended by leader: 12, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:21.258690 18364 raft_consensus.cc:2468] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 14.
I20260709 12:23:21.131791 23752 raft_consensus.cc:515] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:21.255242 18374 raft_consensus.cc:2468] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 17.
I20260709 12:23:21.262567 18365 raft_consensus.cc:2468] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 20.
I20260709 12:23:21.263473 17759 raft_consensus.cc:2468] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 18.
I20260709 12:23:21.227198 18371 rpcz_store.cc:276] 0709 12:23:20.116357 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:20.121139 (+  4782us) service_pool.cc:224] Handling call
0709 12:23:20.121736 (+   597us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:20.142494 (+ 20758us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:20.142538 (+    44us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:20.143168 (+   630us) log.cc:844] Serialized 26 byte log entry
0709 12:23:20.908559 (+765391us) spinlock_profiling.cc:243] Waited 7.6 ms on lock 0x7b1800332140. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b47515e07 00007f7b503709a9 00007f7b4753269b 00007f7b4753093f 00007f7b47532c02 00007f7b4a1ba20b 00007f7b4a1bfd23 00007f7b4a1c00d1 00007f7b4a5dba0f
0709 12:23:21.051435 (+142876us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:23:21.051524 (+    89us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:21.051672 (+   148us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:21.225976 (+174304us) raft_consensus.cc:1685] finished
0709 12:23:21.226008 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:21.226386 (+   378us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":20271,"raft.queue_time_us":93295,"spinlock_wait_cycles":24108160,"thread_start_us":87216,"threads_started":1,"wal-append.queue_time_us":851165}
W20260709 12:23:21.268831 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 78516.8us
I20260709 12:23:21.267220 18363 raft_consensus.cc:1275] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 15 index: 13. Preceding OpId from leader: term: 17 index: 14. (index mismatch)
W20260709 12:23:21.275445 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 200239us
W20260709 12:23:21.283140 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 109684us
I20260709 12:23:21.296815 23427 raft_consensus.cc:493] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:21.301779 23741 leader_election.cc:290] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:21.284322 17803 leader_election.cc:304] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:21.308638 18367 raft_consensus.cc:2468] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 18.
I20260709 12:23:21.311170 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:21.312070 17759 raft_consensus.cc:2468] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 12.
I20260709 12:23:21.315721 23697 raft_consensus.cc:3060] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:21.318298 23764 raft_consensus.cc:2804] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:23:21.318794 23764 raft_consensus.cc:493] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:23:21.319192 23764 raft_consensus.cc:3060] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:21.319460 23749 leader_election.cc:290] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:21.322079 17764 raft_consensus.cc:1240] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 LEADER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 16. Current term is 17. Ops: []
I20260709 12:23:21.261806 23521 raft_consensus.cc:515] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:21.000604 17587 rpcz_store.cc:269] 0709 12:22:57.202896 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:57.203121 (+   225us) service_pool.cc:224] Handling call
0709 12:23:20.894530 (+23691409us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:23:15.460664 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:23:15.461287 (+   623us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:23:15.461321 (+    34us) write_op.cc:435] Acquired schema lock
0709 12:23:15.461338 (+    17us) tablet.cc:662] Decoding operations
0709 12:23:15.463067 (+  1729us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:23:15.463119 (+    52us) write_op.cc:641] Partition lock acquired for write op
0709 12:23:15.463146 (+    27us) tablet.cc:685] Acquiring locks for 24 operations
0709 12:23:15.676766 (+213620us) tablet.cc:701] Row locks acquired
0709 12:23:15.676791 (+    25us) write_op.cc:260] PREPARE: finished
0709 12:23:15.677011 (+   220us) write_op.cc:270] Start()
0709 12:23:15.677168 (+   157us) write_op.cc:276] Timestamp: P: 1783599795676946 usec, L: 0
0709 12:23:15.677195 (+    27us) op_driver.cc:348] REPLICATION: starting
0709 12:23:15.703987 (+ 26792us) log.cc:844] Serialized 9460 byte log entry
0709 12:23:16.063266 (+359279us) op_driver.cc:464] REPLICATION: finished
0709 12:23:16.065032 (+  1766us) write_op.cc:301] APPLY: starting
0709 12:23:16.065124 (+    92us) tablet.cc:1367] starting BulkCheckPresence
0709 12:23:16.559259 (+494135us) tablet.cc:1370] finished BulkCheckPresence
0709 12:23:16.559361 (+   102us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:23:17.139401 (+580040us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:23:17.149146 (+  9745us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=48,key_file_lookups=48,delta_file_lookups=48,mrs_lookups=0
0709 12:23:17.157473 (+  8327us) write_op.cc:312] APPLY: finished
0709 12:23:17.165305 (+  7832us) log.cc:844] Serialized 212 byte log entry
0709 12:23:17.408211 (+242906us) spinlock_profiling.cc:243] Waited 3.66 ms on lock 0x7b4400022b4c. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b47459c85 00007f7b4745a6a2 00007f7b4a1cc1db 00007f7b4a1c7aaf 00007f7b4a1bb86e 00007f7b4a1babb6 00007f7b4a1ba7d8 00007f7b4a1c4221 00007f7b4a1c41d9 00007f7b4a1c4169
0709 12:23:18.691508 (+1283297us) spinlock_profiling.cc:243] Waited 15 ms on lock 0x7b1800395ce0. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b4a1c84bc 00007f7b4751648b 00007f7b503709a9 00007f7b4753269b 00007f7b4753093f 00007f7b47532c02 00007f7b4a1ba20b 00007f7b4a1bfd23 00007f7b4a1c023a
0709 12:23:18.691930 (+   422us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:23:18.692215 (+   285us) write_op.cc:454] Released schema lock
0709 12:23:18.692303 (+    88us) write_op.cc:341] FINISH: Updating metrics
0709 12:23:18.692969 (+   666us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:23:18.692999 (+    30us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":3685220,"reported_tablets":44,"spinlock_wait_cycles":36608,"child_traces":[["op",{"apply.queue_time_us":1492,"apply.run_cpu_time_us":110073,"apply.run_wall_time_us":2627871,"cfile_cache_hit":98,"cfile_cache_hit_bytes":104543,"num_ops":24,"prepare.queue_time_us":196213,"prepare.run_cpu_time_us":20366,"prepare.run_wall_time_us":434575,"raft.queue_time_us":74947,"raft.run_cpu_time_us":1571,"raft.run_wall_time_us":1862,"replication_time_us":385874,"spinlock_wait_cycles":42898688,"thread_start_us":434234,"threads_started":5,"wal-append.queue_time_us":384876,"wal-append.run_cpu_time_us":15060,"wal-append.run_wall_time_us":2293843}]]}
I20260709 12:23:21.321561 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 16 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:21.323423 17755 raft_consensus.cc:2468] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 15.
I20260709 12:23:21.320770 17728 leader_election.cc:304] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:21.280289 23634 leader_election.cc:290] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 15 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:21.299698 23752 leader_election.cc:290] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:21.340677 23546 raft_consensus.cc:2749] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 12:23:21.343866 23573 raft_consensus.cc:493] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:23:21.344501 23573 raft_consensus.cc:3060] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:21.345598 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:21.347761 23763 raft_consensus.cc:493] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:21.348618 23763 raft_consensus.cc:515] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:21.350538 23604 raft_consensus.cc:493] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:23:21.351423 23763 leader_election.cc:290] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:21.351228 23604 raft_consensus.cc:515] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:21.353518 23521 leader_election.cc:290] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:21.354094 23521 raft_consensus.cc:493] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:21.354645 23521 raft_consensus.cc:515] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:21.358337 23634 raft_consensus.cc:2749] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:23:21.359045 23604 leader_election.cc:290] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:23:21.364476 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 80314.1us
I20260709 12:23:21.366225 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:21.366525 17803 leader_election.cc:304] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:21.367321 17688 raft_consensus.cc:2368] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 12. Current term is 13.
I20260709 12:23:21.330489 23742 leader_election.cc:290] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:21.369433 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 12 candidate_status { last_received { term: 10 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:21.370512 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:21.371414 17683 raft_consensus.cc:2393] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 16: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:23:21.371512 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:21.372208 17682 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 13.
I20260709 12:23:21.374490 17832 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35042 (request call id 2495) took 1735 ms. Trace:
I20260709 12:23:21.374887 17832 rpcz_store.cc:276] 0709 12:23:19.639335 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:19.935270 (+295935us) service_pool.cc:224] Handling call
0709 12:23:19.940403 (+  5133us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:21.026084 (+1085681us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:21.374203 (+348119us) spinlock_profiling.cc:243] Waited 904 ms on lock 0x7b540028d054. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:23:21.374430 (+   227us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2079935232}
I20260709 12:23:21.378515 17762 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36572 (request call id 1004) took 1670 ms. Trace:
I20260709 12:23:21.378970 17762 rpcz_store.cc:276] 0709 12:23:19.707585 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:19.836294 (+128709us) service_pool.cc:224] Handling call
0709 12:23:19.836669 (+   375us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:21.026438 (+1189769us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:21.377820 (+351382us) spinlock_profiling.cc:243] Waited 977 ms on lock 0x7b54004150d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:23:21.378404 (+   584us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2248078720}
I20260709 12:23:21.379624 17834 raft_consensus.cc:1275] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 15 index: 13. Preceding OpId from leader: term: 16 index: 14. (index mismatch)
I20260709 12:23:21.380920 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:21.381795 17834 raft_consensus.cc:2393] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 9: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:23:21.382462 17758 raft_consensus.cc:515] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:21.382817 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:21.383435 23690 raft_consensus.cc:2749] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:23:21.390919 17758 leader_election.cc:290] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 16 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:21.384704 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:23:21.393978 23652 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.066s	user 0.004s	sys 0.001s
I20260709 12:23:21.328823 23668 raft_consensus.cc:493] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
W20260709 12:23:21.392733 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 120361us
I20260709 12:23:21.395769 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
W20260709 12:23:21.400799 23652 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.071s	user 0.005s	sys 0.002s
I20260709 12:23:21.395514 23668 raft_consensus.cc:515] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:23:21.401899 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 87697.7us
I20260709 12:23:21.404120 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:21.426203 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:21.426587 17761 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38776 (request call id 2515) took 1620 ms. Trace:
I20260709 12:23:21.423074 17836 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:35032 (request call id 101) took 1671 ms. Trace:
I20260709 12:23:21.411747 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:21.432608 17829 raft_consensus.cc:2393] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 13: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:23:21.411517 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:21.320312 23427 raft_consensus.cc:515] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:21.434635 17682 raft_consensus.cc:2468] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 9.
I20260709 12:23:21.432965 17761 rpcz_store.cc:276] 0709 12:23:19.806059 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:20.448734 (+642675us) service_pool.cc:224] Handling call
0709 12:23:20.449404 (+   670us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:20.946600 (+497196us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:21.426347 (+479747us) spinlock_profiling.cc:243] Waited 363 ms on lock 0x7b54003e3fd4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:23:21.426514 (+   167us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":835226496}
W20260709 12:23:21.425021 17831 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35040 (request call id 2793) took 1774 ms (client timeout 1600 ms). Trace:
I20260709 12:23:21.433180 17836 rpcz_store.cc:276] 0709 12:23:19.751461 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:20.066468 (+315007us) service_pool.cc:224] Handling call
0709 12:23:21.422812 (+1356344us) spinlock_profiling.cc:243] Waited 787 ms on lock 0x7b540008a7d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a61a853 00007f7b4f5e860a 00007f7b48b6e7d4 00007f7b48b6e762 00007f7b48b6e691 00007f7b48b6e60c 00007f7b48b6d8c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:23:21.422956 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1809484672}
I20260709 12:23:21.438030 17683 raft_consensus.cc:2468] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 12.
I20260709 12:23:21.413446 23470 raft_consensus.cc:3055] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 13 LEADER]: Stepping down as leader of term 13
W20260709 12:23:21.439152 17831 rpcz_store.cc:269] 0709 12:23:19.650325 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:19.721375 (+ 71050us) service_pool.cc:224] Handling call
0709 12:23:21.424769 (+1703394us) spinlock_profiling.cc:243] Waited 1.35 s on lock 0x7b540008a7d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:23:21.424916 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3112245504}
I20260709 12:23:21.415107 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:21.415647 23606 raft_consensus.cc:493] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:21.445027 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:21.417789 23668 leader_election.cc:290] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:21.442883 23470 raft_consensus.cc:740] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:23:21.414273 23631 raft_consensus.cc:2749] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:23:21.447788 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:21.446540 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:23:21.451568 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 85189us
I20260709 12:23:21.454362 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:21.453317 23470 consensus_queue.cc:260] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.11, Last appended by leader: 11, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:21.455983 17833 raft_consensus.cc:2393] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 14: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:23:21.447640 23606 raft_consensus.cc:515] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:21.460755 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:21.461220 23606 leader_election.cc:290] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:21.462045 17683 raft_consensus.cc:2468] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 11.
I20260709 12:23:21.462286 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:21.463078 17838 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 13.
I20260709 12:23:21.464764 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:21.465484 17832 raft_consensus.cc:3060] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:21.466521 17803 leader_election.cc:304] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:21.465644 23606 consensus_queue.cc:1059] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 17.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:23:21.467708 17801 leader_election.cc:304] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:21.468446 18334 leader_election.cc:304] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:23:21.469913 23604 raft_consensus.cc:2804] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:23:21.470410 23604 raft_consensus.cc:493] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:21.470757 23604 raft_consensus.cc:3060] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:21.471248 23727 raft_consensus.cc:3055] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:23:21.479765 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:21.484969 23620 raft_consensus.cc:493] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:23:21.479665 23690 consensus_queue.cc:1059] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 12.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:23:21.481417 17830 raft_consensus.cc:2468] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 16.
I20260709 12:23:21.483098 17688 raft_consensus.cc:2393] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 13: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:23:21.471688 23727 raft_consensus.cc:740] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 13 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:23:21.485774 23620 raft_consensus.cc:515] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:21.490096 17833 raft_consensus.cc:2393] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 14: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:23:21.495530 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:21.500855 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:21.501663 17688 raft_consensus.cc:2468] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 14.
I20260709 12:23:21.500873 18334 leader_election.cc:304] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:23:21.496905 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:21.504148 17803 leader_election.cc:304] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:21.504679 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:21.506181 17832 raft_consensus.cc:2468] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 14.
I20260709 12:23:21.503441 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:21.509624 18334 leader_election.cc:304] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:23:21.510888 17836 raft_consensus.cc:3060] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:21.512082 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:21.513139 23498 raft_consensus.cc:493] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:21.513947 23762 raft_consensus.cc:2804] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:23:21.513741 23498 raft_consensus.cc:515] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:21.498769 23631 raft_consensus.cc:2749] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 12:23:21.518105 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:21.527541 23498 leader_election.cc:290] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:23:21.523488 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 55855.1us
I20260709 12:23:21.530217 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:21.523016 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:21.515584 18374 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185"
dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
mode: GRACEFUL
new_leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
 from {username='slave'} at 127.0.0.1:50468
I20260709 12:23:21.533325 18374 raft_consensus.cc:606] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Received request to transfer leadership to TS e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:21.533805 18374 raft_consensus.cc:614] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:23:21.533932 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:21.495545 23573 raft_consensus.cc:515] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:21.537338 17838 raft_consensus.cc:3060] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:21.538645 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:21.540763 23551 consensus_queue.cc:1059] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 15.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.016s
I20260709 12:23:21.542336 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:21.543061 23682 raft_consensus.cc:2804] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:23:21.543488 23682 raft_consensus.cc:493] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:23:21.543916 23682 raft_consensus.cc:3060] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:21.545053 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 16 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:21.546762 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 20 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:21.547509 17831 raft_consensus.cc:2393] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 16: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:23:21.529290 23606 raft_consensus.cc:2749] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
W20260709 12:23:21.549492 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 103118us
I20260709 12:23:21.533579 23521 leader_election.cc:290] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:21.555428 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:21.556649 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:21.556665 23655 raft_consensus.cc:493] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:23:21.557209 23620 leader_election.cc:290] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:23:21.559315 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 55274.1us
I20260709 12:23:21.559492 17833 raft_consensus.cc:3060] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:21.563671 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:21.570577 17681 raft_consensus.cc:2393] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 16: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:23:21.572628 23727 consensus_queue.cc:260] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 13.13, Last appended by leader: 13, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:21.574237 17801 leader_election.cc:304] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:21.576133 23573 leader_election.cc:290] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 13 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:21.560494 23655 raft_consensus.cc:515] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:21.577176 17803 leader_election.cc:304] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:21.579442 23530 leader_election.cc:290] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:21.575001 23522 consensus_queue.cc:1048] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 12:23:21.574618 23727 raft_consensus.cc:3060] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:21.579324 23655 leader_election.cc:290] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:21.580546 17802 leader_election.cc:400] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 12 election: Vote denied by peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071) with higher term. Message: Invalid argument: T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 12. Current term is 13.
I20260709 12:23:21.621413 17802 leader_election.cc:403] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 12 election: Cancelling election due to peer responding with higher term
I20260709 12:23:21.569429 23737 raft_consensus.cc:515] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:21.586166 17652 leader_election.cc:304] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:21.584304 23620 consensus_queue.cc:1048] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.002s
I20260709 12:23:21.587318 23624 raft_consensus.cc:493] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:21.574584 17832 raft_consensus.cc:2468] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 10.
W20260709 12:23:21.592381 17729 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:21.592761 23774 log.cc:927] Time spent T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.084s	user 0.001s	sys 0.000s
I20260709 12:23:21.644012 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 18 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:21.598335 23764 raft_consensus.cc:515] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:21.600443 17803 leader_election.cc:304] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:21.601697 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:21.602717 23508 raft_consensus.cc:2749] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:23:21.603367 23650 raft_consensus.cc:2749] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:23:21.603752 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: true dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:21.603902 17728 leader_election.cc:304] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:21.606547 23502 raft_consensus.cc:493] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:23:21.607513 17759 raft_consensus.cc:1240] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 19. Current term is 20. Ops: []
I20260709 12:23:21.611083 17837 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37994 (request call id 1117) took 1156 ms. Trace:
I20260709 12:23:21.652150 17681 raft_consensus.cc:2468] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 12.
I20260709 12:23:21.616559 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:21.599264 23604 raft_consensus.cc:515] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:21.638509 17801 leader_election.cc:304] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:21.658888 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:23:21.650398 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 97563.4us
I20260709 12:23:21.653087 17838 raft_consensus.cc:2468] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 17.
W20260709 12:23:21.653892 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 144247us
I20260709 12:23:21.653255 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:21.655288 17686 raft_consensus.cc:3060] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:21.656282 17837 rpcz_store.cc:276] 0709 12:23:20.454909 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:20.457568 (+  2659us) service_pool.cc:224] Handling call
0709 12:23:20.457834 (+   266us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:21.018524 (+560690us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:21.609598 (+591074us) spinlock_profiling.cc:243] Waited 398 ms on lock 0x7b5400089dd4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:23:21.611021 (+  1423us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":914737536}
I20260709 12:23:21.661891 23649 raft_consensus.cc:2749] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Leader election lost for term 12. Reason: Vote denied by peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071) with higher term. Message: Invalid argument: T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 12. Current term is 13.
I20260709 12:23:21.662899 23749 raft_consensus.cc:2804] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:23:21.647074 23624 raft_consensus.cc:3060] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:21.663765 23749 raft_consensus.cc:493] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:23:21.664232 23723 consensus_queue.cc:1059] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 13.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.024s
I20260709 12:23:21.664863 23749 raft_consensus.cc:3060] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:23:21.665202 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 65760.4us
I20260709 12:23:21.667150 17682 raft_consensus.cc:3060] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:21.668730 23502 raft_consensus.cc:515] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:21.671648 23470 raft_consensus.cc:3060] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Advancing to term 14
W20260709 12:23:21.673298 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 87547.2us
I20260709 12:23:21.637475 23762 raft_consensus.cc:697] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 9 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:23:21.677373 23762 consensus_queue.cc:237] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:21.684255 23454 raft_consensus.cc:2749] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:23:21.686476 23502 leader_election.cc:290] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:21.687829 23723 raft_consensus.cc:3055] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:23:21.688400 23723 raft_consensus.cc:740] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
W20260709 12:23:21.688598 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 84822.2us
I20260709 12:23:21.689137 23257 raft_consensus.cc:2749] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:23:21.689517 23573 raft_consensus.cc:2804] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:23:21.690099 23573 raft_consensus.cc:493] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:21.694271 23573 raft_consensus.cc:3060] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:21.663064 17756 raft_consensus.cc:1240] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 17. Current term is 18. Ops: []
I20260709 12:23:21.693225 23690 consensus_queue.cc:1059] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 15.13, Next index: 14, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:23:21.677855 23737 leader_election.cc:290] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 15 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:21.697042 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:21.684562 23651 raft_consensus.cc:3055] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:23:21.633361 17836 raft_consensus.cc:2468] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 17.
I20260709 12:23:21.714186 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 15 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:21.716820 23690 raft_consensus.cc:3055] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:23:21.717487 23690 raft_consensus.cc:740] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:23:21.718454 23690 consensus_queue.cc:260] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 15.13, Last appended by leader: 13, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:21.719435 23690 raft_consensus.cc:3060] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:21.716897 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: true dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:21.721967 17836 raft_consensus.cc:3055] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:23:21.722595 17836 raft_consensus.cc:740] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 15 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
W20260709 12:23:21.724452 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 49980.1us
I20260709 12:23:21.729920 17652 leader_election.cc:304] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:21.733464 17682 raft_consensus.cc:2468] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 17.
W20260709 12:23:21.736853 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 136571us
I20260709 12:23:21.735862 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:21.663759 23764 leader_election.cc:290] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 16 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:23:21.747444 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 56783.1us
I20260709 12:23:21.760670 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:21.723495 23772 raft_consensus.cc:2749] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:23:21.797606 23427 leader_election.cc:290] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:23:21.826634 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 76271.6us
I20260709 12:23:21.723922 23651 raft_consensus.cc:740] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 17 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
W20260709 12:23:21.828135 17802 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:21.800487 23504 raft_consensus.cc:2749] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:23:21.802186 23521 raft_consensus.cc:3055] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:23:21.802894 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:21.838575 17759 raft_consensus.cc:2468] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 12.
I20260709 12:23:21.803766 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:21.804559 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:23:21.839432 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 48431.7us
I20260709 12:23:21.840324 17764 raft_consensus.cc:2468] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 13.
I20260709 12:23:21.805424 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:21.806277 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:21.808230 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 18 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:21.809193 23475 raft_consensus.cc:3055] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:23:21.813884 23490 leader_election.cc:290] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 18 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:21.689603 23723 consensus_queue.cc:260] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:21.857461 17681 raft_consensus.cc:1275] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 13 index: 9. Preceding OpId from leader: term: 15 index: 10. (index mismatch)
I20260709 12:23:21.756145 17836 consensus_queue.cc:260] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.10, Last appended by leader: 10, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:21.731755 23620 raft_consensus.cc:493] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:21.833688 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:21.834574 17830 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343"
dest_uuid: "e063541542a94969898868bf6eb5ce1b"
mode: GRACEFUL
new_leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
 from {username='slave'} at 127.0.0.1:59820
I20260709 12:23:21.683761 23682 raft_consensus.cc:515] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:21.837224 18333 leader_election.cc:304] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:21.743101 23530 raft_consensus.cc:493] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:23:21.841866 17760 raft_consensus.cc:2468] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 11.
I20260709 12:23:21.791347 23498 raft_consensus.cc:493] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:23:21.857015 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:21.842788 17763 raft_consensus.cc:2468] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 17.
I20260709 12:23:21.881067 17761 raft_consensus.cc:1275] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 15 index: 9. (index mismatch)
W20260709 12:23:21.816720 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 86715us
W20260709 12:23:21.815256 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 55720.7us
I20260709 12:23:21.871078 17830 raft_consensus.cc:606] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Received request to transfer leadership to TS e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:21.882337 17652 leader_election.cc:400] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 16 pre-election: Vote denied by peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651) with higher term. Message: Invalid argument: T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for earlier term 16. Current term is 17.
I20260709 12:23:21.882506 17830 raft_consensus.cc:614] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:23:21.838084 23521 raft_consensus.cc:740] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:23:21.844861 23475 raft_consensus.cc:740] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:23:21.902040 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:23:21.822780 23594 consensus_queue.cc:1175] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "334a4e1f59944544b3b3ff36616b57da" responder_term: 13 status { last_received { term: 13 index: 9 } last_committed_idx: 9 last_received_current_leader { term: 13 index: 9 } }
I20260709 12:23:21.910724 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 18 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:21.906448 17728 leader_election.cc:304] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:21.904306 23631 consensus_queue.cc:1048] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.027s
I20260709 12:23:21.922749 17727 leader_election.cc:304] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:23:21.938830 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 44075.6us
W20260709 12:23:21.944226 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 80351.7us
I20260709 12:23:21.950264 17803 leader_election.cc:304] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:21.935598 23490 raft_consensus.cc:2749] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:23:21.952605 23572 raft_consensus.cc:2804] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:23:21.943521 23781 raft_consensus.cc:493] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:21.986521 17761 raft_consensus.cc:1240] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 14. Current term is 15. Ops: [14.9-14.9]
W20260709 12:23:21.992429 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 110262us
I20260709 12:23:21.993670 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 15 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:21.995765 17686 raft_consensus.cc:2468] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 16.
I20260709 12:23:22.003190 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:22.004245 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:22.018672 17687 raft_consensus.cc:3060] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:22.008369 23572 raft_consensus.cc:697] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:23:21.932333 23723 raft_consensus.cc:3060] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:22.007534 17727 leader_election.cc:304] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:21.910034 23530 raft_consensus.cc:515] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:22.037544 17838 raft_consensus.cc:1240] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 13. Current term is 14. Ops: [13.11-13.11]
I20260709 12:23:21.924129 23620 raft_consensus.cc:515] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:22.044580 23551 raft_consensus.cc:2749] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:23:22.048537 17652 leader_election.cc:304] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:22.056038 17756 raft_consensus.cc:2468] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 16.
I20260709 12:23:21.927939 23498 raft_consensus.cc:515] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:22.009168 23781 raft_consensus.cc:515] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:22.024130 23522 raft_consensus.cc:2749] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
W20260709 12:23:22.062215 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 112051us
I20260709 12:23:22.066563 17803 leader_election.cc:304] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3
W20260709 12:23:22.071291 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 176325us
I20260709 12:23:22.075493 17728 leader_election.cc:304] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
W20260709 12:23:22.085960 17654 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:22.065591 23604 leader_election.cc:290] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 13 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:22.073185 23763 raft_consensus.cc:2749] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:23:22.091665 23749 raft_consensus.cc:515] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:22.119117 23551 raft_consensus.cc:493] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:23:22.119774 23551 raft_consensus.cc:515] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:22.120733 23781 leader_election.cc:290] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:23:22.121956 23793 log.cc:927] Time spent T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.097s	user 0.000s	sys 0.000s
I20260709 12:23:22.122912 23551 leader_election.cc:290] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:22.104426 23502 raft_consensus.cc:2749] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
W20260709 12:23:22.114291 23782 log.cc:927] Time spent T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.174s	user 0.000s	sys 0.004s
I20260709 12:23:22.124943 23620 leader_election.cc:290] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:22.125147 23749 leader_election.cc:290] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 11 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:23:22.126372 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 59941.5us
I20260709 12:23:22.080859 23690 raft_consensus.cc:493] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
W20260709 12:23:22.133054 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 125592us
I20260709 12:23:22.107350 23740 raft_consensus.cc:2804] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:23:22.134315 17727 leader_election.cc:304] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:22.141112 17727 leader_election.cc:304] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
W20260709 12:23:22.151742 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 76362.4us
I20260709 12:23:22.152657 17728 leader_election.cc:304] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
W20260709 12:23:22.161525 17803 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:22.140673 23740 raft_consensus.cc:493] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:22.074709 23475 consensus_queue.cc:260] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:22.142721 23763 raft_consensus.cc:493] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:22.184543 23763 raft_consensus.cc:515] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:22.187281 23763 leader_election.cc:290] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:23:22.191605 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 50592.8us
I20260709 12:23:22.147656 23502 raft_consensus.cc:2804] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:23:22.168485 23682 leader_election.cc:290] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 12 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:23:22.179462 23786 log.cc:927] Time spent T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.093s	user 0.000s	sys 0.004s
I20260709 12:23:22.176687 23742 raft_consensus.cc:2804] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:23:22.188441 23634 raft_consensus.cc:2749] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:23:22.179132 23740 raft_consensus.cc:3060] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:22.207393 17687 raft_consensus.cc:2468] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 13.
W20260709 12:23:22.213202 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 60620.8us
W20260709 12:23:22.219084 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 381948us
I20260709 12:23:22.220106 23498 leader_election.cc:290] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:23:22.221092 23727 consensus_meta.cc:287] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.584s	user 0.012s	sys 0.000s
W20260709 12:23:22.223989 23525 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.776s	user 0.022s	sys 0.024s
W20260709 12:23:22.224424 23525 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.867s	user 0.022s	sys 0.024s
I20260709 12:23:22.224391 17727 leader_election.cc:304] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:22.225628 17727 leader_election.cc:304] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:23:22.227325 17833 consensus_meta.cc:287] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.655s	user 0.008s	sys 0.000s
W20260709 12:23:22.227393 23573 consensus_meta.cc:287] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.532s	user 0.005s	sys 0.006s
I20260709 12:23:22.195302 23502 raft_consensus.cc:493] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:22.214677 17728 leader_election.cc:304] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:22.227799 17833 raft_consensus.cc:2468] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 18.
I20260709 12:23:22.230446 17832 raft_consensus.cc:1240] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 17. Current term is 18. Ops: []
I20260709 12:23:22.163488 23572 consensus_queue.cc:237] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 12.10, Last appended by leader: 10, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:22.236425 23763 raft_consensus.cc:2804] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:23:22.237010 23763 raft_consensus.cc:697] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 17 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:23:22.163929 23690 raft_consensus.cc:515] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:22.241678 23763 consensus_queue.cc:237] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.12, Last appended by leader: 12, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:22.210920 23475 raft_consensus.cc:3060] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:22.230867 23502 raft_consensus.cc:3060] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:22.249141 17831 raft_consensus.cc:1240] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 12. Current term is 13. Ops: []
I20260709 12:23:22.252226 23530 leader_election.cc:290] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:22.207029 23742 raft_consensus.cc:493] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:23:22.279439 23742 raft_consensus.cc:3060] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Advancing to term 17
W20260709 12:23:22.260006 23784 log.cc:927] Time spent T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.071s	user 0.000s	sys 0.001s
W20260709 12:23:22.261260 23624 consensus_meta.cc:287] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.597s	user 0.012s	sys 0.000s
I20260709 12:23:22.262686 23740 raft_consensus.cc:515] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:22.240309 23634 raft_consensus.cc:2804] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:23:22.234668 23573 raft_consensus.cc:515] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:23:22.286253 23797 log.cc:927] Time spent T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.113s	user 0.001s	sys 0.001s
W20260709 12:23:22.300731 23789 log.cc:927] Time spent T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.098s	user 0.002s	sys 0.002s
I20260709 12:23:22.308509 17758 raft_consensus.cc:1240] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 14. Current term is 16. Ops: []
I20260709 12:23:22.293049 23624 raft_consensus.cc:515] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:22.322374 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:22.324182 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:22.326694 18367 raft_consensus.cc:2410] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 14 because replica has last-logged OpId of term: 13 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 9.
I20260709 12:23:22.327857 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:22.331084 18362 raft_consensus.cc:1240] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 11. Current term is 12. Ops: []
I20260709 12:23:22.336649 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:22.337651 18361 raft_consensus.cc:2393] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 13: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:23:22.338860 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:22.342190 23624 leader_election.cc:290] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 16 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:22.345238 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:22.346206 18364 raft_consensus.cc:2410] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for term 14 because replica has last-logged OpId of term: 13 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 9.
I20260709 12:23:22.343729 18365 raft_consensus.cc:1240] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 19. Current term is 20. Ops: [19.16-19.16]
I20260709 12:23:22.350481 18365 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 1838) took 1082 ms. Trace:
I20260709 12:23:22.351006 18365 rpcz_store.cc:276] 0709 12:23:21.265877 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:22.343160 (+1077283us) service_pool.cc:224] Handling call
0709 12:23:22.343472 (+   312us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:22.348176 (+  4704us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:22.348428 (+   252us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:23:22.354254 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 128703us
I20260709 12:23:22.355723 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:22.356683 18363 raft_consensus.cc:2468] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 14.
I20260709 12:23:22.357878 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:22.359491 18361 raft_consensus.cc:2393] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 13: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
W20260709 12:23:22.364720 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 150131us
I20260709 12:23:22.349344 23530 consensus_queue.cc:1048] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.010s
I20260709 12:23:22.367770 18333 leader_election.cc:304] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:22.370711 18367 raft_consensus.cc:1275] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 9 index: 10. Preceding OpId from leader: term: 13 index: 13. (index mismatch)
I20260709 12:23:22.373368 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:22.359503 23690 leader_election.cc:290] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:22.372426 23624 raft_consensus.cc:3060] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:22.380129 23763 raft_consensus.cc:2804] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:23:22.397809 23763 raft_consensus.cc:493] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:23:22.397377 17730 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:22.399375 17803 leader_election.cc:304] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:22.398438 23763 raft_consensus.cc:3060] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:22.401357 23522 consensus_queue.cc:1059] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 11.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.006s
I20260709 12:23:22.401446 23740 leader_election.cc:290] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 12 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:22.402720 23634 raft_consensus.cc:697] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:23:22.410554 23634 consensus_queue.cc:237] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.8, Last appended by leader: 8, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:22.410948 23785 consensus_queue.cc:1059] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 16.14, Next index: 15, Last known committed idx: 13, Time since last communication: 0.025s
I20260709 12:23:22.392438 23573 leader_election.cc:290] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 16 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:23:22.415675 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 59926.4us
I20260709 12:23:22.412117 23690 raft_consensus.cc:493] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:22.424117 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 12 candidate_status { last_received { term: 10 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:22.421988 23739 raft_consensus.cc:2749] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:23:22.429440 17727 leader_election.cc:304] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:23:22.425895 23749 raft_consensus.cc:493] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:22.440905 23573 raft_consensus.cc:2804] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:23:22.440685 23749 raft_consensus.cc:515] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:22.441478 23573 raft_consensus.cc:697] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 18 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:23:22.442624 23573 consensus_queue.cc:237] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.12, Last appended by leader: 12, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:22.417855 23502 raft_consensus.cc:515] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:22.445577 23502 leader_election.cc:290] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 12 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:22.425169 23785 consensus_queue.cc:1243] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer e063541542a94969898868bf6eb5ce1b log is divergent from this leader: its last log entry 17.15 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 13
I20260709 12:23:22.443660 23749 leader_election.cc:290] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:22.492077 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:22.492244 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:22.496523 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:23:22.497839 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 481088us
W20260709 12:23:22.498220 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:22.498682 23293 raft_consensus.cc:2804] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:23:22.501178 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:22.502031 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:22.508911 17688 raft_consensus.cc:3060] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Advancing to term 16
I20260709 12:23:22.502317 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:23:22.503283 17655 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:22.504187 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:22.504567 17686 raft_consensus.cc:1275] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 16 index: 9. Preceding OpId from leader: term: 17 index: 10. (index mismatch)
I20260709 12:23:22.504834 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:22.427129 23690 raft_consensus.cc:515] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:22.514570 17652 leader_election.cc:304] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:22.518873 17652 leader_election.cc:304] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:23:22.522290 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:22.512445 23490 raft_consensus.cc:493] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:22.523617 23490 raft_consensus.cc:515] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:22.524111 17829 raft_consensus.cc:1275] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 13 index: 11. (index mismatch)
I20260709 12:23:22.526216 23490 leader_election.cc:290] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:22.515390 23293 raft_consensus.cc:697] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 18 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:23:22.527626 23293 consensus_queue.cc:237] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.10, Last appended by leader: 10, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:22.528754 23690 leader_election.cc:290] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:22.530474 23804 raft_consensus.cc:493] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:23:22.520859 23700 raft_consensus.cc:2764] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Leader pre-election decision vote started in defunct term 12: won
I20260709 12:23:22.504842 23749 consensus_queue.cc:1048] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.030s
I20260709 12:23:22.532953 23804 raft_consensus.cc:515] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:22.536036 23742 raft_consensus.cc:515] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:22.538327 23742 leader_election.cc:290] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 17 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:22.539237 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:22.539961 17833 raft_consensus.cc:2468] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 10.
I20260709 12:23:22.540930 23763 raft_consensus.cc:515] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:22.533156 17831 raft_consensus.cc:1275] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 16 index: 9. Preceding OpId from leader: term: 17 index: 10. (index mismatch)
I20260709 12:23:22.545476 23470 raft_consensus.cc:2804] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:23:22.548444 23470 raft_consensus.cc:493] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:22.548918 23470 raft_consensus.cc:3060] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Advancing to term 16
W20260709 12:23:22.550040 17729 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:22.553743 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:22.556586 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:22.555534 23490 raft_consensus.cc:493] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:22.558851 17729 leader_election.cc:304] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:23:22.563347 17755 raft_consensus.cc:1240] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 12. Current term is 13. Ops: []
I20260709 12:23:22.564193 23737 raft_consensus.cc:2804] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:23:22.564827 23737 raft_consensus.cc:697] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:23:22.565800 23737 consensus_queue.cc:237] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 17.15, Last appended by leader: 15, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:22.566417 23470 raft_consensus.cc:515] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:22.566810 23654 consensus_queue.cc:1059] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.022s
I20260709 12:23:22.568542 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:22.572173 17757 raft_consensus.cc:1240] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 16. Current term is 17. Ops: [16.14-16.14]
I20260709 12:23:22.582504 17654 leader_election.cc:304] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:23:22.572706 23470 leader_election.cc:290] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 16 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:22.550529 23700 consensus_queue.cc:1059] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.035s
I20260709 12:23:22.586884 23804 leader_election.cc:290] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:22.590945 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:22.588688 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:22.596060 17755 raft_consensus.cc:2393] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 13: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:23:22.555737 23475 raft_consensus.cc:493] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:22.592047 17757 raft_consensus.cc:2468] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 12.
I20260709 12:23:22.602954 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:22.606998 23636 raft_consensus.cc:2804] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:23:22.607502 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:22.610239 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:22.575022 23594 consensus_queue.cc:1048] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.036s
I20260709 12:23:22.611137 17759 raft_consensus.cc:2468] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 15.
I20260709 12:23:22.613351 17688 raft_consensus.cc:2468] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 16.
I20260709 12:23:22.617033 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 12 candidate_status { last_received { term: 10 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:22.607502 23636 raft_consensus.cc:493] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:23:22.625998 23636 raft_consensus.cc:3060] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:22.558900 23490 raft_consensus.cc:515] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:22.628369 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:22.632754 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:22.637631 17682 raft_consensus.cc:1275] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 14 index: 9. (index mismatch)
I20260709 12:23:22.641242 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:22.650856 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:22.653301 17838 raft_consensus.cc:1240] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 10. Current term is 11. Ops: [10.9-10.9]
I20260709 12:23:22.616179 23763 leader_election.cc:290] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 13 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:22.657516 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:23:22.658612 17801 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:22.659507 17801 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:22.660697 17801 leader_election.cc:304] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:22.664881 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:22.666652 17759 raft_consensus.cc:1275] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 12 index: 8. Preceding OpId from leader: term: 14 index: 9. (index mismatch)
I20260709 12:23:22.655746 18333 leader_election.cc:304] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:23:22.618669 23475 raft_consensus.cc:515] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:22.680156 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:22.693109 17755 raft_consensus.cc:2393] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 18: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:23:22.680671 23749 raft_consensus.cc:2749] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:23:22.680840 23796 raft_consensus.cc:2804] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:23:22.679755 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:22.681214 23481 consensus_queue.cc:1048] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:23:22.682869 17838 raft_consensus.cc:1275] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260709 12:23:22.640743 23498 consensus_queue.cc:1059] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 17.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.039s
I20260709 12:23:22.685079 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:22.686214 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 18 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:22.704192 18367 raft_consensus.cc:2410] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for term 18 because replica has last-logged OpId of term: 15 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 12.
I20260709 12:23:22.687016 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:22.687767 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:22.689023 17756 raft_consensus.cc:1275] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 15 index: 13. Preceding OpId from leader: term: 17 index: 15. (index mismatch)
I20260709 12:23:22.641763 23594 consensus_queue.cc:1048] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 10, Time since last communication: 0.053s
I20260709 12:23:22.708010 18366 raft_consensus.cc:2393] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 13: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:23:22.709600 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 18 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:22.710884 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:22.711982 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:22.712955 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:22.693014 18374 raft_consensus.cc:2376] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 17. Re-sending same reply.
I20260709 12:23:22.675148 23723 raft_consensus.cc:493] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:22.714275 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:22.714881 18374 raft_consensus.cc:2468] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 12.
I20260709 12:23:22.715559 23764 raft_consensus.cc:3055] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:23:22.715842 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:22.717419 23740 raft_consensus.cc:3055] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:23:22.717880 23740 raft_consensus.cc:740] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 11 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:23:22.698735 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:22.689554 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:22.697497 23796 raft_consensus.cc:493] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:22.713209 23697 raft_consensus.cc:493] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:22.714558 23723 raft_consensus.cc:515] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:22.716009 23764 raft_consensus.cc:740] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:23:22.720161 18361 raft_consensus.cc:2410] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for term 15 because replica has last-logged OpId of term: 14 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 10.
I20260709 12:23:22.718643 17761 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 12.
I20260709 12:23:22.697484 18333 leader_election.cc:304] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:22.682256 23521 consensus_queue.cc:260] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 16.14, Last appended by leader: 14, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:22.730280 18333 leader_election.cc:304] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:23:22.690810 17801 leader_election.cc:304] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:22.721792 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:22.732470 18374 raft_consensus.cc:1240] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 12. Current term is 13. Ops: []
I20260709 12:23:22.724208 23796 raft_consensus.cc:3060] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:22.725620 17729 leader_election.cc:304] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
W20260709 12:23:22.726369 17803 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:22.661669 23785 consensus_queue.cc:1048] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.046s
I20260709 12:23:22.738147 17729 leader_election.cc:304] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:22.749480 17689 raft_consensus.cc:1275] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 15 index: 9. (index mismatch)
I20260709 12:23:22.735554 23804 raft_consensus.cc:2804] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:23:22.735425 23780 raft_consensus.cc:2804] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:23:22.753773 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:22.753885 17801 leader_election.cc:304] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:22.753979 17759 raft_consensus.cc:2468] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 13.
W20260709 12:23:22.754653 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 212282us
I20260709 12:23:22.726691 23697 raft_consensus.cc:515] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:22.745038 23573 raft_consensus.cc:2749] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:23:22.750480 23813 raft_consensus.cc:2749] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 12:23:22.739852 23749 consensus_queue.cc:1059] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 11.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.021s
I20260709 12:23:22.758342 17652 leader_election.cc:304] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:22.763558 17729 leader_election.cc:304] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:22.770303 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:22.756174 23780 raft_consensus.cc:493] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:22.755961 23804 raft_consensus.cc:493] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:23:22.785276 18367 raft_consensus.cc:2393] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 13: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:23:22.786381 17729 leader_election.cc:304] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
W20260709 12:23:22.764539 23817 log.cc:927] Time spent T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.148s	user 0.000s	sys 0.001s
I20260709 12:23:22.829547 17830 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:59820 (request call id 47) took 1063 ms. Trace:
I20260709 12:23:22.829936 17830 rpcz_store.cc:276] 0709 12:23:21.711486 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:21.751221 (+ 39735us) service_pool.cc:224] Handling call
0709 12:23:22.774374 (+1023153us) spinlock_profiling.cc:243] Waited 11.3 ms on lock 0x7b54001581d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a61a853 00007f7b4f5e860a 00007f7b48b6e7d4 00007f7b48b6e762 00007f7b48b6e691 00007f7b48b6e60c 00007f7b48b6d8c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:23:22.774504 (+   130us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":25894272}
W20260709 12:23:22.831607 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 45355.3us
I20260709 12:23:22.764601 23772 raft_consensus.cc:2749] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 12:23:22.835894 17729 leader_election.cc:304] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:22.839905 17729 leader_election.cc:304] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
W20260709 12:23:22.840240 23818 log.cc:927] Time spent T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.185s	user 0.000s	sys 0.001s
I20260709 12:23:22.833287 23624 raft_consensus.cc:2749] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:23:22.841370 23573 raft_consensus.cc:2749] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:23:22.841603 18365 raft_consensus.cc:1240] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 13. Current term is 14. Ops: []
I20260709 12:23:22.836810 23697 leader_election.cc:290] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:22.840874 23502 raft_consensus.cc:2804] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:23:22.842445 23502 raft_consensus.cc:493] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:22.843123 23502 raft_consensus.cc:3060] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:22.844041 23573 raft_consensus.cc:493] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:22.844539 23573 raft_consensus.cc:515] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:22.847105 23573 leader_election.cc:290] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:22.848140 23812 raft_consensus.cc:493] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:23:22.848709 23812 raft_consensus.cc:515] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:22.849570 23651 consensus_queue.cc:260] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.13, Last appended by leader: 13, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:22.850884 23651 raft_consensus.cc:3060] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:22.850904 23812 leader_election.cc:290] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:22.851650 23812 raft_consensus.cc:493] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:22.852178 23812 raft_consensus.cc:515] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:22.854436 23812 leader_election.cc:290] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:22.855795 23573 raft_consensus.cc:493] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:22.855954 23594 consensus_queue.cc:1059] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 12:23:22.760231 23740 consensus_queue.cc:260] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.9, Last appended by leader: 9, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:22.836707 23636 raft_consensus.cc:515] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:22.773823 23521 raft_consensus.cc:3060] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:22.774461 23780 raft_consensus.cc:3060] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:22.785727 23804 raft_consensus.cc:3060] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:22.866117 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:22.828009 23742 raft_consensus.cc:2749] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:23:22.831745 23293 raft_consensus.cc:2749] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:23:22.833487 23813 raft_consensus.cc:2749] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:23:22.869652 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:22.856815 23697 raft_consensus.cc:3055] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:23:22.861963 23812 raft_consensus.cc:493] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:23:22.856432 23573 raft_consensus.cc:515] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:22.867687 23740 raft_consensus.cc:3060] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:22.889063 17652 leader_election.cc:304] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:22.871479 23470 raft_consensus.cc:493] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:22.774930 23764 consensus_queue.cc:260] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.11, Last appended by leader: 11, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:23:22.921305 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 287621us
I20260709 12:23:22.874015 23697 raft_consensus.cc:740] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:23:22.884263 23475 leader_election.cc:290] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:22.887322 23671 raft_consensus.cc:493] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:23:22.942844 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:23:22.945677 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 211991us
I20260709 12:23:22.930719 23481 raft_consensus.cc:2749] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:23:22.934708 23293 raft_consensus.cc:493] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:22.928254 23490 leader_election.cc:290] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:22.942878 23828 raft_consensus.cc:493] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:22.965801 18333 leader_election.cc:304] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:22.865635 23502 raft_consensus.cc:515] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:22.949196 23723 leader_election.cc:290] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:22.972457 18333 leader_election.cc:304] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:22.974624 23511 raft_consensus.cc:493] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:22.977866 23706 consensus_queue.cc:1059] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 12.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.019s
I20260709 12:23:22.979736 23530 raft_consensus.cc:2749] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:23:22.937409 23812 raft_consensus.cc:515] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:22.948331 23831 raft_consensus.cc:2749] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:23:22.934183 23470 raft_consensus.cc:515] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:22.985885 18333 leader_election.cc:304] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:23:22.995532 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:22.970057 23764 raft_consensus.cc:3060] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:23:23.009145 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 62771.5us
I20260709 12:23:22.946805 23671 raft_consensus.cc:515] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:22.991123 23511 raft_consensus.cc:515] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:23.016709 23723 raft_consensus.cc:3055] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:23:23.017159 23502 leader_election.cc:290] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 13 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:23.018038 23573 leader_election.cc:290] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:23.018745 23636 leader_election.cc:290] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 15 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:23.030592 23511 leader_election.cc:290] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:22.971745 23293 raft_consensus.cc:515] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:23.026566 18333 leader_election.cc:304] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
W20260709 12:23:23.035274 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 146307us
I20260709 12:23:23.021044 23706 raft_consensus.cc:2804] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:23:23.025913 23723 raft_consensus.cc:740] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:23:22.984674 23828 raft_consensus.cc:515] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:23.043740 18366 raft_consensus.cc:1275] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 12 index: 12. Preceding OpId from leader: term: 14 index: 13. (index mismatch)
W20260709 12:23:23.044878 17652 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:22.991360 23697 consensus_queue.cc:260] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.13, Last appended by leader: 13, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:23.050112 18363 raft_consensus.cc:1240] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 15. Current term is 16. Ops: []
I20260709 12:23:23.050155 18333 leader_election.cc:304] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:23:23.048743 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:23.055053 17681 raft_consensus.cc:2468] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 16.
I20260709 12:23:23.055145 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:23.047714 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:23.050024 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:23.057356 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:23.057574 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:23.050912 23395 raft_consensus.cc:3060] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:23.062122 17836 raft_consensus.cc:3060] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:23.042359 23835 raft_consensus.cc:2804] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:23:23.038872 23706 raft_consensus.cc:493] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:23.064990 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:23.066568 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:23.064361 23835 raft_consensus.cc:493] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:23.041708 23796 raft_consensus.cc:515] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:23.066229 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:23.073547 17682 raft_consensus.cc:2393] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 13: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:23:23.067767 23835 raft_consensus.cc:3060] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:23.075413 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:23.079030 18364 raft_consensus.cc:2393] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 12: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:23:23.045228 23511 raft_consensus.cc:493] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:23.080088 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:23.077242 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:23.082410 17681 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 13.
W20260709 12:23:23.083182 17635 auto_rebalancer.cc:893] Removing replace marker failed: Replica e063541542a94969898868bf6eb5ce1b is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 16 committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } }
I20260709 12:23:23.083516 17687 raft_consensus.cc:1275] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260709 12:23:23.084183 23838 raft_consensus.cc:2804] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:23:23.086957 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 15 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:23.087117 17761 raft_consensus.cc:1240] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 12. Current term is 13. Ops: []
W20260709 12:23:23.083647 17635 auto_rebalancer.cc:901] Could not move replica: Illegal state: not currently leader
I20260709 12:23:23.090811 23839 raft_consensus.cc:493] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:23.090867 17838 raft_consensus.cc:2410] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for term 15 because replica has last-logged OpId of term: 13 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 12.
I20260709 12:23:23.049292 23682 raft_consensus.cc:2804] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:23:23.099282 17802 leader_election.cc:304] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:23.101905 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:23.068128 23706 raft_consensus.cc:3060] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:23:23.088147 17635 auto_rebalancer.cc:330] scheduled replica move failed to complete: Illegal state: not currently leader
I20260709 12:23:23.108192 17832 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35040 (request call id 2817) took 1657 ms. Trace:
I20260709 12:23:23.108635 17832 rpcz_store.cc:276] 0709 12:23:21.451060 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:21.895953 (+444893us) service_pool.cc:224] Handling call
0709 12:23:22.230220 (+334267us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:22.230923 (+   703us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:23.108006 (+877083us) spinlock_profiling.cc:243] Waited 314 ms on lock 0x7b5400335ed0. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a61f8ad 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:23:23.108132 (+   126us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":722852224}
I20260709 12:23:23.111034 17681 raft_consensus.cc:1275] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 13 index: 11. (index mismatch)
I20260709 12:23:23.115442 17836 raft_consensus.cc:2468] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 16.
I20260709 12:23:23.116004 17836 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35042 (request call id 2514) took 1435 ms. Trace:
I20260709 12:23:23.116295 17836 rpcz_store.cc:276] 0709 12:23:21.681022 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:21.716506 (+ 35484us) service_pool.cc:224] Handling call
0709 12:23:23.061476 (+1344970us) spinlock_profiling.cc:243] Waited 4.32 ms on lock 0x7b54001f1e20. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a595a1d 00007f7b4a57c1a4 00007f7b4a5fe447 00007f7b4a616fe7 00007f7b4a6187de 00007f7b4a62427c 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591
0709 12:23:23.115944 (+ 54468us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":9947008}
I20260709 12:23:23.101385 23839 raft_consensus.cc:515] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:23.118106 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 14 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:23.118440 17687 raft_consensus.cc:1240] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 15. Current term is 16. Ops: [15.10-15.10]
I20260709 12:23:23.118957 23812 leader_election.cc:290] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:23.119907 17759 raft_consensus.cc:1240] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 16. Current term is 17. Ops: []
I20260709 12:23:23.120114 23839 leader_election.cc:290] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:23.121842 23796 leader_election.cc:290] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 12 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:23.122948 23676 raft_consensus.cc:2749] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
W20260709 12:23:23.123202 23842 log.cc:927] Time spent T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.078s	user 0.000s	sys 0.000s
I20260709 12:23:23.123379 23804 raft_consensus.cc:515] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:23.124087 18368 raft_consensus.cc:1275] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 16 index: 14. Preceding OpId from leader: term: 17 index: 15. (index mismatch)
W20260709 12:23:23.124365 23833 log.cc:927] Time spent T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.081s	user 0.000s	sys 0.001s
I20260709 12:23:23.125771 23804 leader_election.cc:290] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 13 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:23.128309 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 15 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:23.129416 23520 raft_consensus.cc:3060] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:23.133487 23828 leader_election.cc:290] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:23.136174 23668 raft_consensus.cc:3055] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:23:23.136603 23668 raft_consensus.cc:740] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:23:23.137518 23668 consensus_queue.cc:260] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 19.16, Last appended by leader: 16, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:23.138604 23668 raft_consensus.cc:3060] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Advancing to term 20
W20260709 12:23:23.140336 23829 log.cc:927] Time spent T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.151s	user 0.000s	sys 0.001s
W20260709 12:23:23.141206 17803 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:23.144517 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:23.068882 23502 consensus_queue.cc:1048] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.038s
I20260709 12:23:23.147519 17834 raft_consensus.cc:3060] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Advancing to term 12
I20260709 12:23:23.091439 23749 raft_consensus.cc:493] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
W20260709 12:23:23.152308 17801 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:23.148595 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 14 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:23.092365 23697 raft_consensus.cc:3060] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:23.155588 17802 leader_election.cc:304] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:23.150697 17728 leader_election.cc:304] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:23.165053 17728 leader_election.cc:304] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
W20260709 12:23:23.165484 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 79787.2us
I20260709 12:23:23.166602 18333 leader_election.cc:304] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:23.145057 23812 raft_consensus.cc:493] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:23.112440 23682 raft_consensus.cc:697] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 16 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:23:23.170995 18334 leader_election.cc:304] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:23.093852 23511 raft_consensus.cc:515] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:23:23.187590 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 141701us
I20260709 12:23:23.184711 18334 leader_election.cc:304] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:23.143443 23530 consensus_queue.cc:1059] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 9.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.034s
I20260709 12:23:23.194146 18334 leader_election.cc:304] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:23.180049 17728 leader_election.cc:304] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:23.174572 23521 raft_consensus.cc:493] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:23.199275 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:23.176357 23804 raft_consensus.cc:2749] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:23:23.210912 17728 leader_election.cc:304] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:23.179646 23737 raft_consensus.cc:2749] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:23:23.214134 18363 raft_consensus.cc:1275] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 18 index: 11. (index mismatch)
I20260709 12:23:23.180032 23838 raft_consensus.cc:697] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 13 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:23:23.160422 23739 consensus_queue.cc:1048] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.039s
I20260709 12:23:23.234484 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:23.248001 17759 raft_consensus.cc:2393] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 12: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:23:23.170533 23828 consensus_queue.cc:1059] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 15.12, Next index: 13, Last known committed idx: 11, Time since last communication: 0.031s
I20260709 12:23:23.252478 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:23.168685 23749 raft_consensus.cc:515] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:23:23.242242 17654 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:23.153273 23723 consensus_queue.cc:260] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 12.9, Last appended by leader: 9, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:23.243047 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 87502.9us
I20260709 12:23:23.179265 23796 consensus_queue.cc:1048] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.047s
I20260709 12:23:23.205603 23470 leader_election.cc:290] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:23.257366 17833 raft_consensus.cc:2393] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 17: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:23:23.180315 23812 raft_consensus.cc:515] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:23.235621 23490 raft_consensus.cc:2749] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:23:23.248425 23502 raft_consensus.cc:2749] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:23:23.225793 23813 consensus_queue.cc:1048] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.039s
I20260709 12:23:23.279752 23511 leader_election.cc:290] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:23.283322 17837 raft_consensus.cc:2468] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 18.
I20260709 12:23:23.256146 23671 leader_election.cc:290] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:23.284366 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:23.225033 23521 raft_consensus.cc:515] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:23.256723 23772 raft_consensus.cc:2749] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:23:23.289063 23682 consensus_queue.cc:237] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.13, Last appended by leader: 13, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:23.290757 23749 leader_election.cc:290] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:23.275116 23723 raft_consensus.cc:3060] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:23.294571 23526 raft_consensus.cc:3055] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:23:23.325611 23526 raft_consensus.cc:740] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 14 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:23:23.326538 23526 consensus_queue.cc:260] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.9, Last appended by leader: 9, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:23.295501 23852 raft_consensus.cc:493] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:23.327710 23852 raft_consensus.cc:515] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:23.330587 23852 leader_election.cc:290] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:23.277174 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:23.291733 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:23.332759 17684 raft_consensus.cc:2393] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 13: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:23:23.325649 23844 consensus_queue.cc:1048] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.166s
I20260709 12:23:23.335516 17835 raft_consensus.cc:2468] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 15.
I20260709 12:23:23.321846 23511 raft_consensus.cc:493] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:23.337172 23812 leader_election.cc:290] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:23.337492 23511 raft_consensus.cc:515] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:23.340070 23521 leader_election.cc:290] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:23.340894 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:23.341733 17830 raft_consensus.cc:2468] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 17.
I20260709 12:23:23.296248 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:23.342888 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:23.283136 23835 raft_consensus.cc:515] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:23.344179 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 16 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:23.346494 23605 raft_consensus.cc:493] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:23.347111 23605 raft_consensus.cc:515] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:23.330631 17761 raft_consensus.cc:2468] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 13.
I20260709 12:23:23.298779 17687 raft_consensus.cc:2468] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 17.
W20260709 12:23:23.274381 23839 consensus_queue.cc:1175] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "e063541542a94969898868bf6eb5ce1b" responder_term: 12 status { last_received { term: 12 index: 9 } last_committed_idx: 9 last_received_current_leader { term: 12 index: 9 } }
I20260709 12:23:23.345628 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:23.352655 23855 raft_consensus.cc:2749] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:23:23.352949 17830 raft_consensus.cc:2368] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for earlier term 16. Current term is 17.
I20260709 12:23:23.354327 23816 raft_consensus.cc:2804] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:23:23.276487 23293 leader_election.cc:290] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:23.355084 23749 consensus_queue.cc:1048] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 12:23:23.296011 23838 consensus_queue.cc:237] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:23.279649 23848 raft_consensus.cc:2804] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:23:23.353652 23855 raft_consensus.cc:3055] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:23:23.320379 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:23:23.365950 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 171817us
I20260709 12:23:23.354020 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 21 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:23.371757 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:23.383315 17833 raft_consensus.cc:2393] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 16: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:23:23.383459 17761 raft_consensus.cc:3060] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:23.346204 23835 leader_election.cc:290] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 16 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:23.374042 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:23.375803 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:23.370147 23816 raft_consensus.cc:697] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:23:23.354069 23530 raft_consensus.cc:493] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:23.373920 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:23.387283 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:23.399856 17833 raft_consensus.cc:2468] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 17.
I20260709 12:23:23.397418 17832 raft_consensus.cc:2468] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 14.
I20260709 12:23:23.404573 17832 raft_consensus.cc:1240] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 13. Current term is 14. Ops: []
I20260709 12:23:23.386827 18334 leader_election.cc:304] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
W20260709 12:23:23.379189 23850 log.cc:927] Time spent T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.091s	user 0.000s	sys 0.001s
I20260709 12:23:23.410423 17833 raft_consensus.cc:1275] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 12 index: 12. Preceding OpId from leader: term: 13 index: 13. (index mismatch)
I20260709 12:23:23.415912 18334 leader_election.cc:304] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
W20260709 12:23:23.417481 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 156638us
I20260709 12:23:23.378976 23739 raft_consensus.cc:493] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:23.382709 23855 raft_consensus.cc:740] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:23:23.441439 18334 leader_election.cc:304] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:23.442354 17802 leader_election.cc:304] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:23.366865 17580 catalog_manager.cc:5697] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb reported cstate change: term changed from 10 to 11, leader changed from 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) to 0da169ff82674dc1947055ce43de68a3 (127.16.29.65). New cstate: current_term: 11 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:23:23.452123 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:23.398144 23848 raft_consensus.cc:697] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 13 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:23:23.456215 17761 raft_consensus.cc:2468] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 16.
I20260709 12:23:23.417112 23839 raft_consensus.cc:2749] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 12:23:23.418372 23511 leader_election.cc:290] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:23.443145 23796 raft_consensus.cc:2749] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:23:23.464147 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:23.414661 23395 consensus_queue.cc:1048] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.042s
W20260709 12:23:23.469977 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 268561us
I20260709 12:23:23.469462 17761 raft_consensus.cc:2468] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 13.
I20260709 12:23:23.418115 23521 consensus_queue.cc:1048] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.041s
W20260709 12:23:23.461968 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 251165us
I20260709 12:23:23.414564 23530 raft_consensus.cc:515] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:23.475000 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:23.476151 17728 leader_election.cc:304] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:23:23.477708 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:23.478569 17652 leader_election.cc:304] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:23:23.482050 17652 leader_election.cc:400] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 16 election: Vote denied by peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837) with higher term. Message: Invalid argument: T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for earlier term 16. Current term is 17.
I20260709 12:23:23.482496 17652 leader_election.cc:403] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 16 election: Cancelling election due to peer responding with higher term
I20260709 12:23:23.447192 23835 consensus_queue.cc:1048] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.022s
I20260709 12:23:23.461449 23634 raft_consensus.cc:493] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:23.469655 23780 consensus_meta.cc:287] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.600s	user 0.012s	sys 0.000s
I20260709 12:23:23.457820 23723 raft_consensus.cc:2749] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:23:23.494366 17728 leader_election.cc:304] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:23:23.531313 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 82168us
I20260709 12:23:23.495594 17763 raft_consensus.cc:1240] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 12. Current term is 13. Ops: []
I20260709 12:23:23.498438 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 16 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:23.533056 17757 raft_consensus.cc:3060] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:23.499161 17756 raft_consensus.cc:1275] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 14 index: 12. Preceding OpId from leader: term: 17 index: 14. (index mismatch)
I20260709 12:23:23.451473 23739 raft_consensus.cc:515] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:23.533677 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:23.500993 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:23.534963 17756 raft_consensus.cc:1240] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 LEADER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 19. Current term is 20. Ops: [19.16-19.16]
I20260709 12:23:23.535192 17755 raft_consensus.cc:2468] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 16.
I20260709 12:23:23.502931 17761 raft_consensus.cc:1240] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 15. Current term is 16. Ops: []
W20260709 12:23:23.506043 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 60302.8us
I20260709 12:23:23.478960 23772 raft_consensus.cc:2749] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:23:23.507632 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:23.535542 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 18 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:23.491070 23293 raft_consensus.cc:2804] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:23:23.489606 23594 raft_consensus.cc:2749] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Leader election lost for term 16. Reason: Vote denied by peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837) with higher term. Message: Invalid argument: T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for earlier term 16. Current term is 17.
W20260709 12:23:23.537977 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 43736.1us
I20260709 12:23:23.508541 17758 raft_consensus.cc:1240] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 18 LEADER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 17. Current term is 18. Ops: [17.13-17.13]
I20260709 12:23:23.539705 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:23.531955 17652 leader_election.cc:304] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:23:23.505824 23858 raft_consensus.cc:2804] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:23:23.546645 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:23.486251 23511 consensus_queue.cc:1059] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 13.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.029s
I20260709 12:23:23.549831 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:23.544970 17728 leader_election.cc:304] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:23.513168 23796 raft_consensus.cc:493] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:23.552207 17728 leader_election.cc:304] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:23.484926 23861 consensus_queue.cc:1048] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.065s
I20260709 12:23:23.553568 17652 leader_election.cc:304] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:23.504473 23634 raft_consensus.cc:515] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:23.557035 23634 leader_election.cc:290] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:23.559542 17728 leader_election.cc:304] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:23.510650 23848 consensus_queue.cc:237] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.10, Last appended by leader: 10, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:23.578534 23831 consensus_queue.cc:1048] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
W20260709 12:23:23.582963 17652 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:23.583366 17652 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:23.540915 23293 raft_consensus.cc:493] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:23:23.504968 23780 raft_consensus.cc:515] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:23.608188 17757 raft_consensus.cc:2410] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for term 16 because replica has last-logged OpId of term: 15 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 9.
I20260709 12:23:23.546518 23763 raft_consensus.cc:2804] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:23:23.592219 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:23:23.610169 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 50691.7us
I20260709 12:23:23.611222 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:23.614622 17757 raft_consensus.cc:1240] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 17. Current term is 18. Ops: []
I20260709 12:23:23.620659 17728 leader_election.cc:304] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:23.616426 17834 raft_consensus.cc:2468] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 12.
I20260709 12:23:23.621783 17834 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35042 (request call id 2523) took 1130 ms. Trace:
I20260709 12:23:23.553306 23810 raft_consensus.cc:2749] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:23:23.622278 17834 rpcz_store.cc:276] 0709 12:23:22.491250 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:23.144199 (+652949us) service_pool.cc:224] Handling call
0709 12:23:23.621708 (+477509us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:23.589311 23594 raft_consensus.cc:2804] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:23:23.629424 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:23.634759 18367 raft_consensus.cc:1240] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 17. Current term is 18. Ops: [17.14-17.14]
I20260709 12:23:23.572475 23764 raft_consensus.cc:2804] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:23:23.639513 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:23:23.614745 23860 log.cc:927] Time spent T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.070s	user 0.001s	sys 0.000s
I20260709 12:23:23.617949 23737 raft_consensus.cc:2749] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:23:23.614967 23293 raft_consensus.cc:3060] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:23.657001 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:23.657519 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:23.657730 18362 raft_consensus.cc:2468] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 16.
I20260709 12:23:23.658280 17684 raft_consensus.cc:2468] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 18.
I20260709 12:23:23.659600 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:23.530278 17580 catalog_manager.cc:5697] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb reported cstate change: term changed from 12 to 13, leader changed from 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) to e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66). New cstate: current_term: 13 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:23:23.634859 23739 leader_election.cc:290] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:23.664592 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:23.665324 17683 raft_consensus.cc:2468] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 9.
I20260709 12:23:23.623044 23521 raft_consensus.cc:493] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:23:23.666379 23521 raft_consensus.cc:515] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:23.667249 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 18 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:23.667850 17761 raft_consensus.cc:3060] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:23.668916 23521 leader_election.cc:290] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:23.670198 23861 raft_consensus.cc:493] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:23.665132 17580 catalog_manager.cc:5697] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb reported cstate change: term changed from 8 to 15, leader changed from e063541542a94969898868bf6eb5ce1b (127.16.29.67) to e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66). New cstate: current_term: 15 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: UNKNOWN } } }
I20260709 12:23:23.670697 23861 raft_consensus.cc:515] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:23.671684 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:23.672370 17757 raft_consensus.cc:2468] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 15.
I20260709 12:23:23.588562 23634 raft_consensus.cc:2749] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:23:23.673130 23861 leader_election.cc:290] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:23.672313 17580 catalog_manager.cc:5697] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb reported cstate change: term changed from 9 to 12, leader changed from 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) to e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66). New cstate: current_term: 12 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:23:23.673870 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:23.634631 23594 raft_consensus.cc:493] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:23.674786 23594 raft_consensus.cc:3060] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:23.677825 23739 raft_consensus.cc:493] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:23:23.678393 23739 raft_consensus.cc:515] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:23.646384 17652 leader_election.cc:304] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:23:23.683040 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 61464.8us
I20260709 12:23:23.641384 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:23.643043 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:23.506142 23816 consensus_queue.cc:237] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.10, Last appended by leader: 10, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:23.558732 23858 raft_consensus.cc:697] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:23:23.694649 18366 raft_consensus.cc:1240] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 15. Current term is 16. Ops: []
W20260709 12:23:23.645342 17654 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:23.648949 23863 raft_consensus.cc:2749] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
W20260709 12:23:23.650324 23520 consensus_meta.cc:287] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.520s	user 0.009s	sys 0.003s
I20260709 12:23:23.653364 23780 leader_election.cc:290] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 18 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:23.617619 23763 raft_consensus.cc:493] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:23:23.530117 23855 consensus_queue.cc:260] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 10.9, Last appended by leader: 9, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:23.655045 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:23.655260 23530 leader_election.cc:290] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:23.573840 23796 raft_consensus.cc:515] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:23.572178 23706 raft_consensus.cc:515] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:23.656155 23866 raft_consensus.cc:493] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:23.656662 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:23.715703 23855 raft_consensus.cc:3060] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:23.720028 23518 raft_consensus.cc:493] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:23:23.720839 23518 raft_consensus.cc:515] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:23.720122 17580 catalog_manager.cc:5697] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb reported cstate change: term changed from 6 to 13. New cstate: current_term: 13 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } }
I20260709 12:23:23.673751 23844 consensus_queue.cc:1048] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.033s
I20260709 12:23:23.724152 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:23.724825 17834 raft_consensus.cc:2468] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 16.
I20260709 12:23:23.725660 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:23.726193 17834 raft_consensus.cc:2468] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 14.
I20260709 12:23:23.726588 23706 leader_election.cc:290] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 13 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:23.727020 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:23.727245 23843 raft_consensus.cc:2749] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:23:23.690979 23861 raft_consensus.cc:493] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:23.728112 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:23.728981 17755 raft_consensus.cc:2393] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 13: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:23:23.729252 23293 raft_consensus.cc:515] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:23.687822 17761 raft_consensus.cc:2468] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 18.
I20260709 12:23:23.690164 18365 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 1848) took 1319 ms. Trace:
W20260709 12:23:23.691355 17803 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:23.688755 17727 leader_election.cc:304] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
W20260709 12:23:23.732483 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 43849us
I20260709 12:23:23.733558 17727 leader_election.cc:304] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:23.693315 17762 raft_consensus.cc:1240] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 15. Current term is 16. Ops: []
I20260709 12:23:23.639040 23511 raft_consensus.cc:493] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:23.640121 23764 raft_consensus.cc:493] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:23:23.705112 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 58800.3us
I20260709 12:23:23.706370 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:23:23.706707 17654 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:23.706370 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:23.708498 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:23.692010 23650 raft_consensus.cc:493] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:23.730862 18365 rpcz_store.cc:276] 0709 12:23:22.371216 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:22.373164 (+  1948us) service_pool.cc:224] Handling call
0709 12:23:22.373496 (+   332us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:22.842098 (+468602us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:23.689852 (+847754us) spinlock_profiling.cc:243] Waited 468 ms on lock 0x7b540034b554. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:23:23.690089 (+   237us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1076083968}
I20260709 12:23:23.709738 23656 raft_consensus.cc:493] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:23.736447 17727 leader_election.cc:304] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:23.707231 23763 raft_consensus.cc:3060] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:23.741820 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:23.768007 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:23.768718 18362 raft_consensus.cc:2468] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 15.
I20260709 12:23:23.768761 23518 leader_election.cc:290] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:23.741729 17652 leader_election.cc:304] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:23.743923 17761 raft_consensus.cc:1275] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 16 index: 10. Preceding OpId from leader: term: 18 index: 11. (index mismatch)
I20260709 12:23:23.739684 23866 raft_consensus.cc:515] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:23.773626 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:23.774333 17688 raft_consensus.cc:2468] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 16.
I20260709 12:23:23.774463 23866 leader_election.cc:290] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:23.728142 23861 raft_consensus.cc:515] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:23.775312 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:23.723086 17580 catalog_manager.cc:5697] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb reported cstate change: term changed from 11 to 15. New cstate: current_term: 15 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:23:23.776749 23861 leader_election.cc:290] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:23.777720 23796 leader_election.cc:290] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:23.752868 23739 leader_election.cc:290] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:23.752626 23511 raft_consensus.cc:515] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:23.780870 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 21 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:23.781075 18368 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59902 (request call id 1462) took 1466 ms. Trace:
I20260709 12:23:23.781689 23511 leader_election.cc:290] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:23.737357 23634 raft_consensus.cc:2749] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:23:23.782603 18364 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59886 (request call id 1638) took 1116 ms. Trace:
I20260709 12:23:23.783663 23650 raft_consensus.cc:515] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:23.782079 17683 raft_consensus.cc:2410] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for term 21 because replica has last-logged OpId of term: 19 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 15.
W20260709 12:23:23.786356 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 44755.6us
I20260709 12:23:23.781446 18368 rpcz_store.cc:276] 0709 12:23:22.314235 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:22.316430 (+  2195us) service_pool.cc:224] Handling call
0709 12:23:22.324194 (+  7764us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:23.124496 (+800302us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:23.780850 (+656354us) spinlock_profiling.cc:243] Waited 799 ms on lock 0x7b54008039d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:23:23.781019 (+   169us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1838611712}
I20260709 12:23:23.790736 17803 leader_election.cc:304] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:23:23.783142 18364 rpcz_store.cc:276] 0709 12:23:22.665855 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:22.666046 (+   191us) service_pool.cc:224] Handling call
0709 12:23:23.782288 (+1116242us) spinlock_profiling.cc:243] Waited 374 ms on lock 0x7b54006c5cd4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:23:23.782413 (+   125us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":859645184}
I20260709 12:23:23.799579 17728 leader_election.cc:304] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:23.740469 23863 raft_consensus.cc:2749] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 12:23:23.749776 23764 raft_consensus.cc:3060] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:23.847981 17834 raft_consensus.cc:2468] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 17.
I20260709 12:23:23.851505 17728 leader_election.cc:304] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:23.857884 18371 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59902 (request call id 1466) took 1529 ms. Trace:
I20260709 12:23:23.858345 18371 rpcz_store.cc:276] 0709 12:23:22.328258 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:22.328372 (+   114us) service_pool.cc:224] Handling call
0709 12:23:22.331236 (+  2864us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:22.336310 (+  5074us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:22.336354 (+    44us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:22.336874 (+   520us) log.cc:844] Serialized 26 byte log entry
0709 12:23:23.472992 (+1136118us) spinlock_profiling.cc:243] Waited 2.72 ms on lock 0x7b1800442ae0. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b47515e07 00007f7b503709a9 00007f7b4753269b 00007f7b4753093f 00007f7b47532c02 00007f7b4a1ba20b 00007f7b4a1bfd23 00007f7b4a1c00d1 00007f7b4a5dba0f
0709 12:23:23.594934 (+121942us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:23:23.816318 (+221384us) log.cc:844] Serialized 14 byte log entry
0709 12:23:23.816759 (+   441us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:23.816861 (+   102us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:23.816925 (+    64us) raft_consensus.cc:1685] finished
0709 12:23:23.857578 (+ 40653us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:23.857806 (+   228us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":270,"raft.run_cpu_time_us":309,"raft.run_wall_time_us":33857,"spinlock_wait_cycles":6246272,"thread_start_us":315388,"threads_started":2,"wal-append.queue_time_us":1309754}
I20260709 12:23:23.778872 23293 leader_election.cc:290] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 18 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:23.905519 18363 raft_consensus.cc:1240] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 15. Current term is 16. Ops: []
I20260709 12:23:23.888578 17652 leader_election.cc:304] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:23:23.785722 23737 raft_consensus.cc:2804] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:23:23.918359 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:23.785172 23656 raft_consensus.cc:515] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:23.917778 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:23.922343 23849 raft_consensus.cc:2749] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:23:23.923166 17832 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995"
dest_uuid: "e063541542a94969898868bf6eb5ce1b"
mode: GRACEFUL
new_leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
 from {username='slave'} at 127.0.0.1:59820
I20260709 12:23:23.923895 17832 raft_consensus.cc:606] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 12 LEADER]: Received request to transfer leadership to TS e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:23.868129 23520 consensus_queue.cc:1059] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 12.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.057s
I20260709 12:23:23.925732 23520 raft_consensus.cc:3055] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:23:23.925959 17832 raft_consensus.cc:1275] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
I20260709 12:23:23.914367 23864 raft_consensus.cc:2804] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:23:23.868376 23812 raft_consensus.cc:493] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:23.930706 17680 raft_consensus.cc:2468] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 14.
I20260709 12:23:23.930253 23812 raft_consensus.cc:515] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:23:23.932955 17680 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44898 (request call id 1114) took 2550 ms (client timeout 1816 ms). Trace:
I20260709 12:23:23.933619 23812 leader_election.cc:290] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:23.934906 23749 raft_consensus.cc:493] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:23.935523 23749 raft_consensus.cc:515] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:23:23.936239 23470 raft_consensus.cc:3055] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:23:23.936841 23470 raft_consensus.cc:740] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:23:23.937873 23749 leader_election.cc:290] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:23.937955 23470 consensus_queue.cc:260] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.12, Last appended by leader: 12, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:23.927676 23864 raft_consensus.cc:493] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:23:23.939538 23470 raft_consensus.cc:3060] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:23.942318 17831 raft_consensus.cc:1275] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 13 index: 11. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
I20260709 12:23:23.778779 17580 catalog_manager.cc:5697] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb reported cstate change: term changed from 9 to 11, leader changed from 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) to e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66). New cstate: current_term: 11 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:23:23.927562 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 19 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:23:23.933518 17680 rpcz_store.cc:269] 0709 12:23:21.382772 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:21.384241 (+  1469us) service_pool.cc:224] Handling call
0709 12:23:23.932687 (+2548446us) spinlock_profiling.cc:243] Waited 2.54 s on lock 0x7b54004277d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:23:23.932834 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5837308800}
I20260709 12:23:23.815279 23518 raft_consensus.cc:2804] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:23:23.954632 17760 raft_consensus.cc:2410] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for term 19 because replica has last-logged OpId of term: 16 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 9.
I20260709 12:23:23.856935 23739 raft_consensus.cc:2804] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:23:23.904227 23810 raft_consensus.cc:2749] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:23:23.939724 23864 raft_consensus.cc:3060] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:23.973261 17652 leader_election.cc:304] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:23.976680 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:23.977545 18363 raft_consensus.cc:2468] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 12.
I20260709 12:23:23.843168 23511 consensus_queue.cc:1059] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 15.10, Next index: 11, Last known committed idx: 9, Time since last communication: 0.121s
I20260709 12:23:23.979187 17652 leader_election.cc:304] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:23.926234 23520 raft_consensus.cc:740] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:23:23.873376 23828 consensus_queue.cc:1048] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.091s
I20260709 12:23:23.855827 23808 raft_consensus.cc:493] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:23.965032 23518 raft_consensus.cc:493] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:23.989688 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 19 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:23.996352 23520 consensus_queue.cc:260] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:24.005813 17688 raft_consensus.cc:2410] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for term 19 because replica has last-logged OpId of term: 17 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 10.
I20260709 12:23:23.968096 23737 raft_consensus.cc:697] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:23:23.981220 23293 raft_consensus.cc:2749] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:23:23.970536 23739 raft_consensus.cc:493] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:23:24.013082 23520 raft_consensus.cc:3060] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:24.014427 17688 raft_consensus.cc:1275] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 15 index: 8. Preceding OpId from leader: term: 17 index: 9. (index mismatch)
W20260709 12:23:23.996352 23870 log.cc:927] Time spent T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.082s	user 0.000s	sys 0.000s
I20260709 12:23:24.008026 23518 raft_consensus.cc:3060] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:24.008912 23808 raft_consensus.cc:515] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:23.943902 23858 consensus_queue.cc:237] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.9, Last appended by leader: 9, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:24.017637 23650 leader_election.cc:290] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:24.017732 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:24.018349 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:24.019593 18361 raft_consensus.cc:1275] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
I20260709 12:23:24.021251 17580 catalog_manager.cc:5697] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb reported cstate change: term changed from 7 to 15, leader changed from e063541542a94969898868bf6eb5ce1b (127.16.29.67) to e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66). New cstate: current_term: 15 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: UNKNOWN } } }
I20260709 12:23:23.986984 23697 raft_consensus.cc:2749] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:23:23.981225 23843 consensus_queue.cc:1059] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 15.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.043s
I20260709 12:23:24.025782 17683 raft_consensus.cc:1240] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 17. Current term is 18. Ops: []
I20260709 12:23:24.028825 23851 consensus_queue.cc:1048] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 12:23:24.028858 23737 consensus_queue.cc:237] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 10.9, Last appended by leader: 9, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:24.031152 23511 consensus_queue.cc:1048] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20260709 12:23:24.033257 17834 raft_consensus.cc:1275] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 13 index: 13. Preceding OpId from leader: term: 14 index: 13. (term mismatch)
I20260709 12:23:24.037839 17682 raft_consensus.cc:1275] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 19 index: 16. Preceding OpId from leader: term: 20 index: 16. (term mismatch)
I20260709 12:23:24.137658 17682 pending_rounds.cc:85] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3: Aborting all ops after (but not including) 15
I20260709 12:23:24.042104 23735 raft_consensus.cc:3055] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:23:23.999670 23594 raft_consensus.cc:515] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:24.137298 17834 pending_rounds.cc:85] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b: Aborting all ops after (but not including) 12
I20260709 12:23:24.047923 17761 raft_consensus.cc:1240] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 17. Current term is 18. Ops: [15.13-17.14]
I20260709 12:23:24.055047 17685 raft_consensus.cc:1240] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 14. Current term is 15. Ops: []
I20260709 12:23:24.049953 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:24.078884 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:24.079360 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:24.079728 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:24.080283 18367 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 1858) took 1041 ms. Trace:
I20260709 12:23:24.086585 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:24.090988 17837 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37994 (request call id 1136) took 1510 ms. Trace:
I20260709 12:23:24.092456 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:24.093318 17759 raft_consensus.cc:1240] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 14. Current term is 15. Ops: [14.9-14.9]
I20260709 12:23:24.110380 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:24.111433 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:24.112625 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:24.115633 18334 leader_election.cc:304] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:23:24.077967 17652 leader_election.cc:304] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:23:24.101724 17654 leader_election.cc:304] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:24.123924 17729 leader_election.cc:304] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:24.047708 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:24.076252 17836 raft_consensus.cc:1240] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 11. Current term is 12. Ops: [11.13-11.13]
W20260709 12:23:24.107991 17801 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:24.053260 23763 raft_consensus.cc:515] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:24.105240 23808 leader_election.cc:290] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:24.133134 18371 raft_consensus.cc:1240] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 12. Current term is 13. Ops: []
I20260709 12:23:24.137951 17682 pending_rounds.cc:107] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3: Aborting uncommitted NO_OP operation due to leader change: 19.16
I20260709 12:23:24.049973 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 15 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:24.150837 17834 pending_rounds.cc:107] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b: Aborting uncommitted NO_OP operation due to leader change: 13.13
W20260709 12:23:24.125574 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 78818.7us
I20260709 12:23:24.330345 17834 raft_consensus.cc:2889] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:24.331374 18333 leader_election.cc:304] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:23:24.332212 17682 raft_consensus.cc:2889] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:23:24.337759 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 259878us
I20260709 12:23:24.341918 18364 raft_consensus.cc:2393] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 16: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:23:24.343434 18368 raft_consensus.cc:2468] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 13.
W20260709 12:23:24.190069 17655 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:24.170081 17803 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:24.349603 17837 rpcz_store.cc:276] 0709 12:23:22.584765 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:22.585076 (+   311us) service_pool.cc:224] Handling call
0709 12:23:24.090811 (+1505735us) spinlock_profiling.cc:243] Waited 679 ms on lock 0x7b54000893d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:23:24.090921 (+   110us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1562473216}
I20260709 12:23:24.352221 17762 raft_consensus.cc:2393] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 15: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:23:24.120278 23843 raft_consensus.cc:3055] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:23:24.354069 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:23:24.354461 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 252852us
W20260709 12:23:24.355824 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 231979us
I20260709 12:23:24.363682 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:24.364470 17833 raft_consensus.cc:2468] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 14.
I20260709 12:23:24.356230 17764 raft_consensus.cc:2393] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 18: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:23:24.342305 18367 rpcz_store.cc:276] 0709 12:23:23.038674 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:23.038789 (+   115us) service_pool.cc:224] Handling call
0709 12:23:23.043581 (+  4792us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:23.638417 (+594836us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:24.080001 (+441584us) spinlock_profiling.cc:243] Waited 591 ms on lock 0x7b54007e8954. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:23:24.080215 (+   214us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1358730624}
W20260709 12:23:24.356756 17681 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.693s	user 0.054s	sys 0.119s
I20260709 12:23:24.064626 23864 raft_consensus.cc:515] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:24.366636 17758 raft_consensus.cc:1240] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 16. Current term is 17. Ops: []
I20260709 12:23:24.366748 17689 raft_consensus.cc:2393] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 15: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:23:24.372985 17727 leader_election.cc:304] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
W20260709 12:23:24.365813 17681 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.702s	user 0.054s	sys 0.119s
I20260709 12:23:24.111969 23624 raft_consensus.cc:493] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:24.075613 23838 raft_consensus.cc:493] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:24.380951 17654 leader_election.cc:304] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:24.363782 17729 leader_election.cc:304] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:24.047178 23739 raft_consensus.cc:3060] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:24.141306 23742 raft_consensus.cc:493] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:23:24.096127 23796 raft_consensus.cc:493] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:24.367532 18333 leader_election.cc:304] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:24.145118 23735 raft_consensus.cc:740] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 15 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:23:24.334262 23868 raft_consensus.cc:493] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:24.335286 23697 raft_consensus.cc:493] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:24.397622 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 17 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:24.048883 23835 consensus_queue.cc:1059] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 16.14, Next index: 15, Last known committed idx: 13, Time since last communication: 0.308s
I20260709 12:23:24.404376 17761 raft_consensus.cc:2468] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 16.
I20260709 12:23:24.354130 23293 raft_consensus.cc:493] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:23:24.355170 23869 raft_consensus.cc:493] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:23:24.047974 23572 consensus_queue.cc:1048] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.334s
I20260709 12:23:24.070935 23656 leader_election.cc:290] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:24.359642 23808 raft_consensus.cc:2749] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:23:24.070631 23764 raft_consensus.cc:515] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:24.360265 23843 raft_consensus.cc:740] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:23:24.366111 23676 raft_consensus.cc:493] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:23:24.385887 23785 raft_consensus.cc:2804] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:23:24.385254 23810 raft_consensus.cc:2749] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:23:24.421123 18371 raft_consensus.cc:1275] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 17 index: 15. Preceding OpId from leader: term: 20 index: 16. (index mismatch)
I20260709 12:23:24.429191 17834 raft_consensus.cc:1275] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 16 index: 12. Preceding OpId from leader: term: 18 index: 13. (index mismatch)
I20260709 12:23:24.404012 23844 raft_consensus.cc:2749] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:23:24.357506 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:23:24.434578 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 44562.1us
I20260709 12:23:24.407235 23796 raft_consensus.cc:515] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:24.451023 17687 raft_consensus.cc:2393] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 18: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
W20260709 12:23:24.430225 23816 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.655s	user 0.041s	sys 0.062s
W20260709 12:23:24.452972 23816 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.681s	user 0.041s	sys 0.062s
I20260709 12:23:24.400121 23863 consensus_queue.cc:1048] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 8, Time since last communication: 0.030s
I20260709 12:23:24.392247 23624 raft_consensus.cc:515] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:24.428239 23804 raft_consensus.cc:2804] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:23:24.430881 23849 raft_consensus.cc:2804] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:23:24.408655 23742 raft_consensus.cc:515] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:24.400650 23838 raft_consensus.cc:515] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:24.431924 23852 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.502s	user 0.000s	sys 0.012s
I20260709 12:23:24.412596 23697 raft_consensus.cc:515] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:24.502115 23697 leader_election.cc:290] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:24.431247 23676 raft_consensus.cc:515] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:24.502933 17682 raft_consensus.cc:1275] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 17 index: 13. Preceding OpId from leader: term: 18 index: 13. (term mismatch)
I20260709 12:23:24.503302 17682 pending_rounds.cc:85] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3: Aborting all ops after (but not including) 12
I20260709 12:23:24.503638 17682 pending_rounds.cc:107] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3: Aborting uncommitted NO_OP operation due to leader change: 17.13
I20260709 12:23:24.503902 17682 raft_consensus.cc:2889] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:24.504590 23676 leader_election.cc:290] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:24.505115 23676 raft_consensus.cc:493] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:24.505450 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 15 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:24.505537 23676 raft_consensus.cc:515] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:24.506668 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:24.507234 17682 raft_consensus.cc:2468] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 16.
I20260709 12:23:24.507565 23676 leader_election.cc:290] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:24.433501 23656 consensus_queue.cc:1059] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 15.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.051s
I20260709 12:23:24.484751 16501 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 4
W20260709 12:23:24.485354 23855 consensus_meta.cc:287] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.769s	user 0.012s	sys 0.000s
I20260709 12:23:24.486462 23605 leader_election.cc:290] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:24.487226 23864 leader_election.cc:290] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 17 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:24.487419 23468 raft_consensus.cc:2804] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:23:24.459533 23690 raft_consensus.cc:2749] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:23:24.118587 17580 catalog_manager.cc:5697] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb reported cstate change: term changed from 11 to 15, leader changed from 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) to 0da169ff82674dc1947055ce43de68a3 (127.16.29.65). New cstate: current_term: 15 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
W20260709 12:23:24.488972 17835 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35040 (request call id 2808) took 3634 ms (client timeout 1628 ms). Trace:
I20260709 12:23:24.491130 17838 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35040 (request call id 2823) took 1330 ms. Trace:
W20260709 12:23:24.496474 23884 log.cc:927] Time spent T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.136s	user 0.000s	sys 0.001s
I20260709 12:23:24.538407 17838 rpcz_store.cc:276] 0709 12:23:23.161159 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:23.675636 (+514477us) service_pool.cc:224] Handling call
0709 12:23:23.680048 (+  4412us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:23.785711 (+105663us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 12:23:23.821937 (+ 36226us) log.cc:844] Serialized 14 byte log entry
0709 12:23:24.037962 (+216025us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:24.038013 (+    51us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:23:24.038055 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:24.490880 (+452825us) spinlock_profiling.cc:243] Waited 105 ms on lock 0x7b54000aaad4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:23:24.490931 (+    51us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:24.491066 (+   135us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":61245,"prepare.run_cpu_time_us":72,"prepare.run_wall_time_us":62,"raft.queue_time_us":4253,"raft.run_cpu_time_us":121,"raft.run_wall_time_us":4860,"spinlock_wait_cycles":242367360,"thread_start_us":185760,"threads_started":1,"wal-append.queue_time_us":615403}
W20260709 12:23:24.508417 23852 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.579s	user 0.001s	sys 0.012s
I20260709 12:23:24.508882 23650 raft_consensus.cc:493] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:23:24.414322 23868 raft_consensus.cc:515] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:24.509260 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:24.511513 17836 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35040 (request call id 2825) took 1343 ms. Trace:
I20260709 12:23:24.584043 17836 rpcz_store.cc:276] 0709 12:23:23.167822 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:23.188769 (+ 20947us) service_pool.cc:224] Handling call
0709 12:23:23.190660 (+  1891us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:24.348845 (+1158185us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:24.511282 (+162437us) spinlock_profiling.cc:243] Waited 885 ms on lock 0x7b5400265a54. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:23:24.511447 (+   165us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2035998592}
I20260709 12:23:24.512851 23594 leader_election.cc:290] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 15 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:24.513032 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 17 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:24.585794 17688 raft_consensus.cc:2468] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 16.
I20260709 12:23:24.457792 23831 consensus_queue.cc:1059] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 14.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.057s
I20260709 12:23:24.428611 23869 raft_consensus.cc:515] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:24.420264 23293 raft_consensus.cc:515] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:24.450829 23785 raft_consensus.cc:697] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 13 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:23:24.522110 23849 raft_consensus.cc:697] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 16 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
W20260709 12:23:24.519678 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 66768.4us
I20260709 12:23:24.464280 23863 consensus_queue.cc:1243] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer e063541542a94969898868bf6eb5ce1b log is divergent from this leader: its last log entry 11.10 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 8
W20260709 12:23:24.537143 17835 rpcz_store.cc:269] 0709 12:23:20.855064 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:20.986966 (+131902us) service_pool.cc:224] Handling call
0709 12:23:24.488725 (+3501759us) spinlock_profiling.cc:243] Waited 2.18 s on lock 0x7b54003cc8d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:23:24.488849 (+   124us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5009698688}
I20260709 12:23:24.467855 23804 raft_consensus.cc:493] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:24.582793 18374 raft_consensus.cc:1275] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 12 index: 10. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
W20260709 12:23:24.592244 17830 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.650s	user 0.012s	sys 0.086s
W20260709 12:23:24.595695 18365 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.568s	user 0.015s	sys 0.019s
W20260709 12:23:24.674582 17830 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.732s	user 0.012s	sys 0.086s
W20260709 12:23:24.674685 18365 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.647s	user 0.015s	sys 0.019s
I20260709 12:23:24.675410 17830 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35042 (request call id 2531) took 1129 ms. Trace:
I20260709 12:23:24.676359 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:24.675762 17830 rpcz_store.cc:276] 0709 12:23:23.545901 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:23.941506 (+395605us) service_pool.cc:224] Handling call
0709 12:23:23.941816 (+   310us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:23.942025 (+   209us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 12:23:23.942302 (+   277us) log.cc:844] Serialized 14 byte log entry
0709 12:23:24.675085 (+732783us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:24.675136 (+    51us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:23:24.675178 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:24.675240 (+    62us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:24.675357 (+   117us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":6468,"prepare.run_cpu_time_us":66,"prepare.run_wall_time_us":4042,"raft.queue_time_us":460,"raft.run_cpu_time_us":86,"raft.run_wall_time_us":86,"thread_start_us":732445,"threads_started":1}
I20260709 12:23:24.677835 16501 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:23:24.679100 16501 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:23:24.677836 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:24.682396 17652 leader_election.cc:304] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:24.682550 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:24.687674 17680 raft_consensus.cc:1240] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 16. Current term is 17. Ops: []
I20260709 12:23:24.689071 17680 raft_consensus.cc:1240] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 15. Current term is 16. Ops: [15.10-15.10]
I20260709 12:23:24.692723 23796 leader_election.cc:290] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:24.694207 23697 raft_consensus.cc:2749] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 12:23:24.697022 23742 leader_election.cc:290] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:24.699440 23293 leader_election.cc:290] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:24.700835 23764 leader_election.cc:290] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 18 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:24.531672 23841 raft_consensus.cc:493] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:24.471577 17832 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.529s	user 0.029s	sys 0.069s
I20260709 12:23:24.533910 23468 raft_consensus.cc:493] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:24.466958 23812 consensus_queue.cc:1048] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 10, Time since last communication: 0.114s
W20260709 12:23:24.707188 17831 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.699s	user 0.020s	sys 0.012s
W20260709 12:23:24.705654 17832 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.763s	user 0.029s	sys 0.069s
I20260709 12:23:24.711733 17832 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35042 (request call id 2530) took 1169 ms. Trace:
I20260709 12:23:24.717175 17832 rpcz_store.cc:276] 0709 12:23:23.542217 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:23.926907 (+384690us) service_pool.cc:224] Handling call
0709 12:23:23.940967 (+ 14060us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:23.941178 (+   211us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 12:23:23.942292 (+  1114us) log.cc:844] Serialized 14 byte log entry
0709 12:23:24.710882 (+768590us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:24.711146 (+   264us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:23:24.711232 (+    86us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:24.711396 (+   164us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:24.711645 (+   249us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":561,"prepare.queue_time_us":670,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":37,"raft.queue_time_us":109,"thread_start_us":768070,"threads_started":1}
I20260709 12:23:24.534062 23851 consensus_queue.cc:1048] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.140s
I20260709 12:23:24.531795 23518 raft_consensus.cc:515] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:24.712210 23468 raft_consensus.cc:3060] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:24.740299 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:24.741218 17762 raft_consensus.cc:2393] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 15: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:23:24.720618 23841 raft_consensus.cc:515] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:24.744525 23841 leader_election.cc:290] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:24.746872 17681 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37402 (request call id 2981) took 1091 ms. Trace:
I20260709 12:23:24.747272 17681 rpcz_store.cc:276] 0709 12:23:23.655862 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:23.662522 (+  6660us) service_pool.cc:224] Handling call
0709 12:23:23.662911 (+   389us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:23.663272 (+   361us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:23.663303 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:23.663731 (+   428us) log.cc:844] Serialized 26 byte log entry
0709 12:23:24.376110 (+712379us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:23:24.376224 (+   114us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:24.376317 (+    93us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:24.746624 (+370307us) raft_consensus.cc:1685] finished
0709 12:23:24.746644 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:24.746775 (+   131us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":11295,"raft.run_cpu_time_us":77,"raft.run_wall_time_us":78,"thread_start_us":711928,"threads_started":1,"wal-append.queue_time_us":1043067}
W20260709 12:23:24.724771 17684 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.575s	user 0.012s	sys 0.024s
W20260709 12:23:24.748823 17684 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.599s	user 0.012s	sys 0.024s
I20260709 12:23:24.750118 17684 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37396 (request call id 2593) took 1008 ms. Trace:
W20260709 12:23:24.710163 17831 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.706s	user 0.020s	sys 0.012s
I20260709 12:23:24.748387 23468 raft_consensus.cc:515] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:24.754830 23468 leader_election.cc:290] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 18 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:24.750550 17684 rpcz_store.cc:276] 0709 12:23:23.741760 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:23.752375 (+ 10615us) service_pool.cc:224] Handling call
0709 12:23:23.752681 (+   306us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:23.752883 (+   202us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 12:23:24.149001 (+396118us) log.cc:844] Serialized 14 byte log entry
0709 12:23:24.749737 (+600736us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:24.749795 (+    58us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:23:24.749853 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:24.749924 (+    71us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:24.750056 (+   132us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":312230,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":46,"raft.queue_time_us":105,"raft.run_cpu_time_us":92,"raft.run_wall_time_us":94,"thread_start_us":989790,"threads_started":2}
I20260709 12:23:24.765198 23838 leader_election.cc:290] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:24.766103 23838 raft_consensus.cc:493] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:24.766754 23838 raft_consensus.cc:515] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:24.703442 23804 raft_consensus.cc:3060] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:24.773437 17688 raft_consensus.cc:2393] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 18: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:23:24.715400 23869 leader_election.cc:290] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:24.789397 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
W20260709 12:23:24.800362 17763 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.716s	user 0.012s	sys 0.034s
I20260709 12:23:24.704967 23650 raft_consensus.cc:515] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:24.800879 17763 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.716s	user 0.012s	sys 0.034s
I20260709 12:23:24.812553 23763 leader_election.cc:290] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 17 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:23:24.827800 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 428777us
W20260709 12:23:24.817456 23737 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.686s	user 0.015s	sys 0.019s
W20260709 12:23:24.834975 17756 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.753s	user 0.025s	sys 0.021s
I20260709 12:23:24.777403 23735 consensus_queue.cc:260] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.10, Last appended by leader: 10, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:23:24.842183 17756 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.760s	user 0.026s	sys 0.021s
W20260709 12:23:24.851362 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 449774us
W20260709 12:23:24.914656 18363 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.582s	user 0.021s	sys 0.053s
W20260709 12:23:24.915169 18363 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.586s	user 0.021s	sys 0.053s
I20260709 12:23:24.853063 18333 leader_election.cc:304] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:23:24.961781 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 108836us
I20260709 12:23:25.048978 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: true dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:25.049741 17833 raft_consensus.cc:2376] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 16. Re-sending same reply.
I20260709 12:23:25.050086 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 19 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:25.051048 17832 raft_consensus.cc:2410] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for term 19 because replica has last-logged OpId of term: 17 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 10.
I20260709 12:23:25.053810 17801 leader_election.cc:304] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:25.068233 17836 raft_consensus.cc:1275] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 15 index: 8. Preceding OpId from leader: term: 17 index: 9. (index mismatch)
I20260709 12:23:24.849125 23763 raft_consensus.cc:493] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:25.084457 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:24.848976 23810 raft_consensus.cc:493] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:23:24.851790 23742 raft_consensus.cc:493] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:24.907495 23737 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.776s	user 0.015s	sys 0.020s
I20260709 12:23:25.077785 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:25.050518 23735 raft_consensus.cc:3060] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:24.826977 23785 consensus_queue.cc:237] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:25.059973 23676 raft_consensus.cc:493] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:25.073244 17761 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.611s	user 0.016s	sys 0.026s
W20260709 12:23:25.111032 17761 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.649s	user 0.017s	sys 0.026s
I20260709 12:23:25.070278 23749 raft_consensus.cc:493] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:25.071291 23526 raft_consensus.cc:3060] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:25.101671 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:24.831665 23849 consensus_queue.cc:237] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.11, Last appended by leader: 11, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:25.113409 17832 raft_consensus.cc:2468] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 15.
I20260709 12:23:25.133723 17833 raft_consensus.cc:2393] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 11: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:23:25.126415 17758 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38786 (request call id 3254) took 1048 ms. Trace:
I20260709 12:23:25.083357 18333 leader_election.cc:304] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:25.087535 17802 leader_election.cc:304] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:23:25.141050 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 57849.9us
W20260709 12:23:25.141462 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 54057.5us
I20260709 12:23:25.142594 17802 leader_election.cc:304] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:25.131736 23518 leader_election.cc:290] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 14 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:25.131341 23676 raft_consensus.cc:515] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:25.088694 23851 raft_consensus.cc:493] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:25.134649 17831 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35040 (request call id 2841) took 1549 ms. Trace:
I20260709 12:23:25.105584 23868 leader_election.cc:290] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:25.108781 23838 leader_election.cc:290] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:23:25.148378 17829 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.764s	user 0.012s	sys 0.037s
W20260709 12:23:25.162159 17829 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.784s	user 0.013s	sys 0.038s
I20260709 12:23:25.159472 17832 raft_consensus.cc:1275] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 13 index: 12. (index mismatch)
I20260709 12:23:25.156912 17758 rpcz_store.cc:276] 0709 12:23:24.077926 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:24.078281 (+   355us) service_pool.cc:224] Handling call
0709 12:23:24.078588 (+   307us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:24.379997 (+301409us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:25.126152 (+746155us) spinlock_profiling.cc:243] Waited 288 ms on lock 0x7b54002977d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:23:25.126343 (+   191us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":662002560}
I20260709 12:23:25.154487 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:23:25.148556 23858 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.028s	user 0.016s	sys 0.073s
I20260709 12:23:25.159180 17831 rpcz_store.cc:276] 0709 12:23:23.584812 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:23.952250 (+367438us) service_pool.cc:224] Handling call
0709 12:23:23.976224 (+ 23974us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:23.976737 (+   513us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:23.976814 (+    77us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:23.980805 (+  3991us) log.cc:844] Serialized 26 byte log entry
0709 12:23:24.752259 (+771454us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:23:24.752302 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:24.752456 (+   154us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:25.133303 (+380847us) raft_consensus.cc:1685] finished
0709 12:23:25.133336 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:25.134571 (+  1235us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1620,"raft.queue_time_us":2672,"raft.run_cpu_time_us":143,"raft.run_wall_time_us":4335,"thread_start_us":747979,"threads_started":1,"wal-append.queue_time_us":1123569}
I20260709 12:23:25.166224 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:25.168977 17836 raft_consensus.cc:2468] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 12.
I20260709 12:23:25.091398 23763 raft_consensus.cc:515] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:25.167079 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:25.169150 17833 raft_consensus.cc:3060] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:25.107542 23742 raft_consensus.cc:515] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:25.148347 23828 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.046s	user 0.020s	sys 0.078s
I20260709 12:23:25.172670 17832 raft_consensus.cc:2468] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 14.
W20260709 12:23:25.177345 17653 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.758s	user 0.023s	sys 0.059s
W20260709 12:23:25.168448 23858 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.048s	user 0.016s	sys 0.073s
I20260709 12:23:25.129998 23810 raft_consensus.cc:515] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:25.180114 17764 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.680s	user 0.019s	sys 0.039s
W20260709 12:23:25.178660 18362 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.810s	user 0.018s	sys 0.054s
W20260709 12:23:25.174829 23886 log.cc:927] Time spent T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.357s	user 0.001s	sys 0.001s
I20260709 12:23:25.170204 23868 raft_consensus.cc:2804] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:23:25.227981 23868 raft_consensus.cc:493] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:25.228420 23868 raft_consensus.cc:3060] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:25.228508 23742 leader_election.cc:290] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:25.229180 23742 raft_consensus.cc:493] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:25.229671 23742 raft_consensus.cc:515] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:25.170778 23869 raft_consensus.cc:493] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:25.232203 23869 raft_consensus.cc:515] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:25.234488 23869 leader_election.cc:290] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:25.210593 17834 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37994 (request call id 1150) took 1171 ms. Trace:
I20260709 12:23:25.236430 23650 leader_election.cc:290] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:25.236357 17834 rpcz_store.cc:276] 0709 12:23:24.039561 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:24.469770 (+430209us) service_pool.cc:224] Handling call
0709 12:23:24.486032 (+ 16262us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:24.695833 (+209801us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:24.695895 (+    62us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:24.695949 (+    54us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:23:24.695994 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:25.210325 (+514331us) spinlock_profiling.cc:243] Waited 210 ms on lock 0x7b540008bbd4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:23:25.210381 (+    56us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:25.210516 (+   135us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":481955072}
I20260709 12:23:25.207480 23843 consensus_queue.cc:260] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.11, Last appended by leader: 11, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:25.189427 18374 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59886 (request call id 1663) took 1171 ms. Trace:
W20260709 12:23:25.240068 18362 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.872s	user 0.019s	sys 0.054s
I20260709 12:23:25.223088 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:23:25.245391 17801 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:25.158133 18333 connection.cc:256] RPC call timeout handler was delayed by -1.0482973519997358s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.202s for a response
W20260709 12:23:25.176548 23828 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.074s	user 0.020s	sys 0.078s
I20260709 12:23:25.275415 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:25.276081 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:25.133153 23749 raft_consensus.cc:515] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:25.178862 17653 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.763s	user 0.023s	sys 0.060s
W20260709 12:23:25.196497 23891 log.cc:927] Time spent T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.405s	user 0.000s	sys 0.001s
W20260709 12:23:25.196583 23892 log.cc:927] Time spent T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.356s	user 0.000s	sys 0.000s
W20260709 12:23:25.285569 18336 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.698s	user 0.007s	sys 0.033s
W20260709 12:23:25.299589 18336 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.713s	user 0.011s	sys 0.033s
I20260709 12:23:25.196900 23863 raft_consensus.cc:493] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:23:25.197849 23835 raft_consensus.cc:2804] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:23:25.198849 23866 raft_consensus.cc:493] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:23:24.848527 17580 catalog_manager.cc:5697] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb reported cstate change: term changed from 9 to 15, leader changed from 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) to e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66). New cstate: current_term: 15 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:23:25.200570 23511 raft_consensus.cc:493] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:25.202397 23605 raft_consensus.cc:2804] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:23:25.205061 23810 leader_election.cc:290] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:25.209291 23676 leader_election.cc:290] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:25.163329 23851 raft_consensus.cc:515] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:23:25.210589 23739 consensus_meta.cc:287] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.799s	user 0.003s	sys 0.007s
I20260709 12:23:25.217386 23864 raft_consensus.cc:3055] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:23:25.220885 23905 raft_consensus.cc:2804] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:23:25.222450 17763 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38786 (request call id 3256) took 1144 ms. Trace:
I20260709 12:23:25.224092 18363 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 1880) took 1252 ms. Trace:
I20260709 12:23:25.285847 23906 raft_consensus.cc:493] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:23:25.226099 17756 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38786 (request call id 3255) took 1147 ms. Trace:
I20260709 12:23:25.251223 17833 raft_consensus.cc:1275] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 15 index: 13. Preceding OpId from leader: term: 16 index: 14. (index mismatch)
W20260709 12:23:25.252079 17654 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.841s	user 0.012s	sys 0.008s
W20260709 12:23:25.252524 17729 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.859s	user 0.017s	sys 0.033s
W20260709 12:23:25.251669 18365 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.576s	user 0.006s	sys 0.012s
I20260709 12:23:25.240091 18374 rpcz_store.cc:276] 0709 12:23:24.018464 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:24.390980 (+372516us) service_pool.cc:224] Handling call
0709 12:23:24.391297 (+   317us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:24.674485 (+283188us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:25.189077 (+514592us) spinlock_profiling.cc:243] Waited 150 ms on lock 0x7b540051aed4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:23:25.189339 (+   262us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":345341824}
W20260709 12:23:25.242779 17764 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.743s	user 0.019s	sys 0.039s
I20260709 12:23:25.378868 18363 rpcz_store.cc:276] 0709 12:23:23.978640 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:23.983394 (+  4754us) service_pool.cc:224] Handling call
0709 12:23:24.009463 (+ 26069us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:23:24.009932 (+   469us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:24.009974 (+    42us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:23:24.120972 (+110998us) log.cc:844] Serialized 52 byte log entry
0709 12:23:25.047303 (+926331us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:23:25.047353 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:25.047433 (+    80us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:25.223792 (+176359us) raft_consensus.cc:1685] finished
0709 12:23:25.223820 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:25.224020 (+   200us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":2052,"raft.run_cpu_time_us":85,"raft.run_wall_time_us":85,"thread_start_us":585964,"threads_started":1,"wal-append.queue_time_us":1047317}
W20260709 12:23:25.387085 17654 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.976s	user 0.013s	sys 0.008s
W20260709 12:23:25.387512 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 976176us
I20260709 12:23:25.172749 23838 raft_consensus.cc:2804] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:23:25.387795 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:25.388168 23838 raft_consensus.cc:493] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:25.388425 17836 raft_consensus.cc:3060] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:25.388746 23838 raft_consensus.cc:3060] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:25.389101 17654 leader_election.cc:304] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:23:25.390647 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:25.391136 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:25.391263 17831 raft_consensus.cc:2468] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 15.
I20260709 12:23:25.282104 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:25.391696 17834 raft_consensus.cc:2468] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 18.
I20260709 12:23:25.392112 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:25.392256 18366 raft_consensus.cc:2468] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 12.
I20260709 12:23:25.379699 23742 leader_election.cc:290] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:23:25.393680 18365 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.718s	user 0.008s	sys 0.012s
I20260709 12:23:25.393510 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:25.394426 18365 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59886 (request call id 1658) took 1378 ms. Trace:
I20260709 12:23:25.394928 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 15 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:25.303397 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:25.241928 18362 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 1876) took 1587 ms. Trace:
I20260709 12:23:25.398597 18362 rpcz_store.cc:276] 0709 12:23:23.654354 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:23.769520 (+115166us) service_pool.cc:224] Handling call
0709 12:23:23.769828 (+   308us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:23.770037 (+   209us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 12:23:23.952163 (+182126us) log.cc:844] Serialized 14 byte log entry
0709 12:23:25.241448 (+1289285us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:25.241526 (+    78us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:23:25.241591 (+    65us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:25.241692 (+   101us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:25.241855 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":8122,"prepare.queue_time_us":230155,"prepare.run_cpu_time_us":50,"prepare.run_wall_time_us":51,"raft.queue_time_us":41284,"raft.run_cpu_time_us":150,"raft.run_wall_time_us":9803,"thread_start_us":1462393,"threads_started":3,"wal-append.queue_time_us":719089}
W20260709 12:23:25.181458 18334 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.813s	user 0.022s	sys 0.081s
W20260709 12:23:25.300779 23852 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.617s	user 0.008s	sys 0.028s
W20260709 12:23:25.402137 18334 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.034s	user 0.022s	sys 0.081s
W20260709 12:23:25.402200 23852 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.718s	user 0.008s	sys 0.028s
I20260709 12:23:25.274672 23740 raft_consensus.cc:493] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:25.402779 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 1.28717e+06us
I20260709 12:23:25.402908 23740 raft_consensus.cc:515] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:25.403512 17830 raft_consensus.cc:2468] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 16.
I20260709 12:23:25.404467 18366 raft_consensus.cc:2410] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 15 because replica has last-logged OpId of term: 12 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 10.
I20260709 12:23:25.403968 17580 catalog_manager.cc:5697] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb reported cstate change: term changed from 9 to 14, leader changed from e063541542a94969898868bf6eb5ce1b (127.16.29.67) to e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66). New cstate: current_term: 14 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: HEALTHY } } }
I20260709 12:23:25.405035 23740 leader_election.cc:290] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:25.405925 18334 leader_election.cc:304] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:25.406165 17580 catalog_manager.cc:5697] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb reported cstate change: term changed from 8 to 10, leader changed from 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) to e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66). New cstate: current_term: 10 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:23:25.276408 17757 raft_consensus.cc:3060] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:25.408308 23763 leader_election.cc:290] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:25.386663 17756 rpcz_store.cc:276] 0709 12:23:24.078195 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:24.079844 (+  1649us) service_pool.cc:224] Handling call
0709 12:23:24.080282 (+   438us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:24.080729 (+   447us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:24.080769 (+    40us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:24.081571 (+   802us) log.cc:844] Serialized 26 byte log entry
0709 12:23:24.851810 (+770239us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:23:24.851858 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:24.852734 (+   876us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:25.225858 (+373124us) raft_consensus.cc:1685] finished
0709 12:23:25.225881 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:25.226037 (+   156us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":2589,"raft.run_cpu_time_us":94,"raft.run_wall_time_us":96,"thread_start_us":768812,"threads_started":1,"wal-append.queue_time_us":1142398}
I20260709 12:23:25.300508 17758 raft_consensus.cc:1240] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 15. Current term is 16. Ops: []
I20260709 12:23:25.282143 23904 raft_consensus.cc:493] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:25.415445 23904 raft_consensus.cc:515] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:25.394769 18365 rpcz_store.cc:276] 0709 12:23:24.016368 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:24.020017 (+  3649us) service_pool.cc:224] Handling call
0709 12:23:24.020409 (+   392us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:24.020674 (+   265us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 12:23:24.027446 (+  6772us) log.cc:844] Serialized 14 byte log entry
0709 12:23:25.394049 (+1366603us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:25.394101 (+    52us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:23:25.394143 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:25.394204 (+    61us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:25.394300 (+    96us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":6971,"prepare.queue_time_us":7151,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":42,"raft.queue_time_us":460432,"raft.run_cpu_time_us":111,"raft.run_wall_time_us":14017,"thread_start_us":1366154,"threads_started":2,"wal-append.queue_time_us":665913}
I20260709 12:23:25.387826 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:25.417642 23904 leader_election.cc:290] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:25.417974 23526 raft_consensus.cc:493] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:25.302558 23908 raft_consensus.cc:493] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:25.418514 23526 raft_consensus.cc:515] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:25.419001 23908 raft_consensus.cc:515] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:25.420624 23526 leader_election.cc:290] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:25.336242 23605 raft_consensus.cc:493] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
W20260709 12:23:25.346869 23804 consensus_meta.cc:287] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.562s	user 0.010s	sys 0.000s
I20260709 12:23:25.338541 23866 raft_consensus.cc:515] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:25.424012 23804 raft_consensus.cc:515] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:25.425480 23866 leader_election.cc:290] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:25.333933 23863 raft_consensus.cc:515] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:25.425976 23804 leader_election.cc:290] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 17 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:25.426569 23804 raft_consensus.cc:2749] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:23:25.427211 17836 raft_consensus.cc:2468] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 15.
I20260709 12:23:25.350343 23868 raft_consensus.cc:515] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:23:25.321401 23650 consensus_queue.cc:1059] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 16.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.032s
I20260709 12:23:25.355312 23851 leader_election.cc:290] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:25.360618 23518 raft_consensus.cc:493] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:23:25.366618 23855 raft_consensus.cc:493] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:25.372696 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:25.436367 17757 raft_consensus.cc:2468] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 14.
I20260709 12:23:25.374279 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:25.437415 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:25.225755 17689 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37402 (request call id 2979) took 1570 ms. Trace:
W20260709 12:23:25.386765 17729 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.993s	user 0.018s	sys 0.034s
I20260709 12:23:25.282477 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:25.396245 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:25.396525 23749 leader_election.cc:290] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:25.397418 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:25.380079 23905 raft_consensus.cc:493] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:23:25.459971 23905 raft_consensus.cc:3060] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:25.459971 17835 raft_consensus.cc:3060] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:23:25.460991 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 1.09722e+06us
I20260709 12:23:25.462034 23763 consensus_queue.cc:1048] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 12:23:25.252127 23858 raft_consensus.cc:493] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:25.464114 18371 raft_consensus.cc:2468] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 13.
I20260709 12:23:25.464160 23858 raft_consensus.cc:515] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:25.464953 17689 rpcz_store.cc:276] 0709 12:23:23.655226 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:23.659337 (+  4111us) service_pool.cc:224] Handling call
0709 12:23:25.225207 (+1565870us) spinlock_profiling.cc:243] Waited 699 ms on lock 0x7b5400347bd4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:23:25.225576 (+   369us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1608060288}
I20260709 12:23:25.428350 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:25.466168 17756 raft_consensus.cc:2468] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 15.
I20260709 12:23:25.466506 23858 leader_election.cc:290] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:25.428660 18364 raft_consensus.cc:2468] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 17.
I20260709 12:23:25.277022 23737 raft_consensus.cc:493] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:25.467123 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 18 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:25.467381 23737 raft_consensus.cc:515] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:25.468343 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:25.469296 17764 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38776 (request call id 2561) took 1128 ms. Trace:
I20260709 12:23:25.469595 23737 leader_election.cc:290] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:25.378059 17763 rpcz_store.cc:276] 0709 12:23:24.078508 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:24.082286 (+  3778us) service_pool.cc:224] Handling call
0709 12:23:24.082722 (+   436us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:23:24.083251 (+   529us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:24.083292 (+    41us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:23:24.084105 (+   813us) log.cc:844] Serialized 52 byte log entry
0709 12:23:24.801378 (+717273us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:23:24.801435 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:24.801540 (+   105us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:25.221931 (+420391us) raft_consensus.cc:1685] finished
0709 12:23:25.221963 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:25.222363 (+   400us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":346,"raft.run_cpu_time_us":39,"raft.run_wall_time_us":39,"thread_start_us":716625,"threads_started":1,"wal-append.queue_time_us":1135827}
I20260709 12:23:25.468930 17756 raft_consensus.cc:2468] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 18.
I20260709 12:23:25.470549 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:25.471036 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 15 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:25.471314 17763 raft_consensus.cc:2468] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 13.
I20260709 12:23:25.469928 17764 rpcz_store.cc:276] 0709 12:23:24.341088 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:24.498757 (+157669us) service_pool.cc:224] Handling call
0709 12:23:24.499061 (+   304us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:24.499410 (+   349us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:24.499440 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:24.499900 (+   460us) log.cc:844] Serialized 26 byte log entry
0709 12:23:25.386681 (+886781us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:23:25.386739 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:25.386835 (+    96us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:25.469045 (+ 82210us) raft_consensus.cc:1685] finished
0709 12:23:25.469073 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:25.469229 (+   156us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":2070,"raft.run_cpu_time_us":107,"raft.run_wall_time_us":107,"thread_start_us":886347,"threads_started":1,"wal-append.queue_time_us":967012}
I20260709 12:23:25.473096 17759 raft_consensus.cc:2393] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 13: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:23:25.376188 23864 raft_consensus.cc:740] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:23:25.474192 23864 consensus_queue.cc:260] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 17.14, Last appended by leader: 14, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:25.475384 23864 raft_consensus.cc:3060] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:25.421428 23908 leader_election.cc:290] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:25.476362 17801 leader_election.cc:304] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:25.478129 17756 raft_consensus.cc:1275] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 13 index: 11. (index mismatch)
I20260709 12:23:25.478052 17801 leader_election.cc:304] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:25.377205 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 19 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:25.479163 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:25.441747 17755 raft_consensus.cc:2393] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 16: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:23:25.480207 23917 raft_consensus.cc:2804] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:23:25.480656 23917 raft_consensus.cc:493] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:23:25.481009 23917 raft_consensus.cc:3060] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:25.478976 23737 raft_consensus.cc:493] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:25.419991 23906 raft_consensus.cc:515] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:25.481863 23737 raft_consensus.cc:515] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:25.432231 23605 raft_consensus.cc:3060] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:25.341143 23511 raft_consensus.cc:515] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:25.484292 23906 leader_election.cc:290] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:25.427558 23863 leader_election.cc:290] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:25.485235 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:25.486310 23511 leader_election.cc:290] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:23:25.487751 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 159063us
I20260709 12:23:25.487248 17762 raft_consensus.cc:1275] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 13 index: 12. (index mismatch)
I20260709 12:23:25.480829 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:25.495359 17763 raft_consensus.cc:1275] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 15 index: 13. Preceding OpId from leader: term: 16 index: 14. (index mismatch)
I20260709 12:23:25.496335 23913 raft_consensus.cc:2804] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:23:25.498406 23869 raft_consensus.cc:2804] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:23:25.498891 23919 raft_consensus.cc:493] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:25.432667 23804 raft_consensus.cc:493] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:25.378562 23739 raft_consensus.cc:515] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:25.517653 17761 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38786 (request call id 3258) took 1426 ms. Trace:
I20260709 12:23:25.500754 17686 raft_consensus.cc:1240] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 17. Current term is 18. Ops: [17.10-17.10]
I20260709 12:23:25.506768 23913 raft_consensus.cc:493] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:23:25.443825 23855 raft_consensus.cc:515] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:25.507634 23869 raft_consensus.cc:493] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:25.509025 23919 raft_consensus.cc:515] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:25.537417 23739 leader_election.cc:290] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 15 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:25.512456 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 18 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:25.587594 17833 raft_consensus.cc:2468] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 17.
I20260709 12:23:25.513113 17685 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37396 (request call id 2596) took 1770 ms. Trace:
I20260709 12:23:25.518716 17757 raft_consensus.cc:2393] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 17: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:23:25.441277 23518 raft_consensus.cc:515] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:25.500231 17580 catalog_manager.cc:5697] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb reported cstate change: term changed from 7 to 9. New cstate: current_term: 9 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 12:23:25.500427 17687 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37402 (request call id 2976) took 1846 ms (client timeout 1720 ms). Trace:
I20260709 12:23:25.525943 23470 raft_consensus.cc:493] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:23:25.442795 23526 raft_consensus.cc:493] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
W20260709 12:23:25.536607 17655 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:25.537042 17835 raft_consensus.cc:2468] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 18.
I20260709 12:23:25.542888 17761 rpcz_store.cc:276] 0709 12:23:24.091572 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:24.408384 (+316812us) service_pool.cc:224] Handling call
0709 12:23:24.423534 (+ 15150us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:24.423773 (+   239us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 12:23:25.111710 (+687937us) log.cc:844] Serialized 14 byte log entry
0709 12:23:25.517167 (+405457us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:25.517237 (+    70us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:23:25.517304 (+    67us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:25.517390 (+    86us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:25.517585 (+   195us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":792282,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":41,"raft.queue_time_us":12185,"raft.run_cpu_time_us":97,"raft.run_wall_time_us":98,"thread_start_us":1054482,"threads_started":2,"wal-append.queue_time_us":374637}
W20260709 12:23:25.545956 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 403470us
I20260709 12:23:25.500840 17801 leader_election.cc:304] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:23:25.596397 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 96369.6us
I20260709 12:23:25.565852 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 18 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:23:25.511137 23899 log.cc:927] Time spent T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.328s	user 0.001s	sys 0.001s
I20260709 12:23:25.588411 17685 rpcz_store.cc:276] 0709 12:23:23.742593 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:24.000447 (+257854us) service_pool.cc:224] Handling call
0709 12:23:24.000771 (+   324us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:24.332172 (+331401us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:25.512816 (+1180644us) spinlock_profiling.cc:243] Waited 54 ms on lock 0x7b54003480d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:23:25.513036 (+   220us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":124285696}
I20260709 12:23:25.544296 23902 raft_consensus.cc:493] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
W20260709 12:23:25.593308 17687 rpcz_store.cc:269] 0709 12:23:23.654276 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:23.656392 (+  2116us) service_pool.cc:224] Handling call
0709 12:23:25.500231 (+1843839us) spinlock_profiling.cc:243] Waited 735 ms on lock 0x7b5400426dd4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:23:25.500362 (+   131us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1691629312}
I20260709 12:23:25.535611 23838 raft_consensus.cc:515] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:25.598210 17760 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38776 (request call id 2560) took 1257 ms. Trace:
I20260709 12:23:25.598548 17802 leader_election.cc:304] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:25.576761 23868 leader_election.cc:290] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 17 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:25.537628 23804 raft_consensus.cc:515] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:25.600309 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:25.576256 23913 raft_consensus.cc:3060] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Advancing to term 16
W20260709 12:23:25.605232 17635 auto_rebalancer.cc:629] Failed to schedule move for tablet 53e726e4ebbb4ca58b6a6102c9b0850f: Illegal state: Replica e2909cff1da44ad387f96a8ae3aa5afb is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 14 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } }
I20260709 12:23:25.585974 23869 raft_consensus.cc:3060] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:25.601660 17760 rpcz_store.cc:276] 0709 12:23:24.340399 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:24.496414 (+156015us) service_pool.cc:224] Handling call
0709 12:23:24.496718 (+   304us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:24.695106 (+198388us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:24.695152 (+    46us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:24.695207 (+    55us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:23:24.695260 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:25.597967 (+902707us) spinlock_profiling.cc:243] Waited 198 ms on lock 0x7b54001b9254. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:23:25.598012 (+    45us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:25.598144 (+   132us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":455553408}
I20260709 12:23:25.603642 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 17 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:25.647142 17761 raft_consensus.cc:3060] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:25.591055 23737 leader_election.cc:290] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:23:25.632581 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 314208us
I20260709 12:23:25.590739 23573 raft_consensus.cc:493] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
W20260709 12:23:25.660138 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 269848us
W20260709 12:23:25.670809 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 72329.6us
I20260709 12:23:25.596203 23835 raft_consensus.cc:697] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 18 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:23:25.680085 17757 raft_consensus.cc:2410] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 12 because replica has last-logged OpId of term: 10 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 8.
I20260709 12:23:25.684476 18333 leader_election.cc:304] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:25.692910 17802 leader_election.cc:304] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:25.602171 23651 raft_consensus.cc:2804] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:23:25.701752 17802 leader_election.cc:304] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:25.702620 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:25.707715 18333 leader_election.cc:304] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:25.709753 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:25.710029 17802 leader_election.cc:304] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:25.705689 17685 raft_consensus.cc:2468] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 14.
I20260709 12:23:25.710565 17760 raft_consensus.cc:2393] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 15: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:23:25.711344 17761 raft_consensus.cc:2468] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 17.
I20260709 12:23:25.711937 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:25.713438 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 19 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:25.713882 17802 leader_election.cc:304] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
W20260709 12:23:25.708979 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 217233us
I20260709 12:23:25.717253 17652 leader_election.cc:304] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:23:25.717634 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 18 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:25.683370 23868 raft_consensus.cc:2749] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:23:25.599787 23470 raft_consensus.cc:515] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:25.577054 23605 raft_consensus.cc:515] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:25.725642 17682 raft_consensus.cc:1275] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 13 index: 11. (index mismatch)
I20260709 12:23:25.736513 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:25.705497 23481 raft_consensus.cc:2749] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:23:25.741755 18333 leader_election.cc:304] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:25.705185 23651 raft_consensus.cc:493] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:25.602622 23526 raft_consensus.cc:515] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:25.710981 23908 raft_consensus.cc:493] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:25.769712 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 14 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:25.717983 23511 raft_consensus.cc:2749] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:23:25.761754 17802 leader_election.cc:304] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:25.738292 17684 raft_consensus.cc:2468] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 16.
I20260709 12:23:25.628551 23902 raft_consensus.cc:515] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:25.766471 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:25.722630 23916 raft_consensus.cc:2749] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:23:25.777202 23931 raft_consensus.cc:493] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:25.777666 23737 consensus_queue.cc:1048] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 12:23:25.719748 23915 raft_consensus.cc:2749] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 12:23:25.701174 23573 raft_consensus.cc:515] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:25.733543 23700 raft_consensus.cc:2749] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
W20260709 12:23:25.735165 23785 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.530s	user 0.011s	sys 0.013s
I20260709 12:23:25.711062 23917 raft_consensus.cc:515] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:25.723227 23922 raft_consensus.cc:2804] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:23:25.733522 23920 raft_consensus.cc:2749] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:23:25.764516 23651 raft_consensus.cc:3060] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:25.736843 23749 consensus_queue.cc:1059] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 16.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.090s
I20260709 12:23:25.853770 23855 leader_election.cc:290] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:25.835748 23835 consensus_queue.cc:237] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 15.13, Last appended by leader: 13, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:25.856693 23948 raft_consensus.cc:493] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:25.838260 23931 raft_consensus.cc:515] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:25.857053 23929 raft_consensus.cc:493] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:23:25.858290 17683 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44898 (request call id 1136) took 2111 ms. Trace:
I20260709 12:23:25.845388 23922 raft_consensus.cc:493] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
W20260709 12:23:25.840377 23897 log.cc:927] Time spent T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.069s	user 0.000s	sys 0.000s
I20260709 12:23:25.773874 23863 consensus_queue.cc:1059] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 17.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.067s
I20260709 12:23:25.833221 23926 raft_consensus.cc:493] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:25.826177 23481 raft_consensus.cc:2749] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:23:25.781898 23906 consensus_queue.cc:1048] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.062s
W20260709 12:23:25.843524 23785 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.643s	user 0.011s	sys 0.014s
W20260709 12:23:25.844053 23896 log.cc:927] Time spent T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.058s	user 0.000s	sys 0.000s
I20260709 12:23:25.784907 23908 raft_consensus.cc:515] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:25.845988 23916 raft_consensus.cc:493] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:25.858381 23914 raft_consensus.cc:2804] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:23:25.827605 23911 consensus_queue.cc:1059] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 14.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.060s
W20260709 12:23:25.904896 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 146194us
I20260709 12:23:25.904227 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:25.941288 17682 raft_consensus.cc:2393] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 16: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:23:25.904630 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 19 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:25.907012 23922 raft_consensus.cc:3060] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:25.834013 23905 raft_consensus.cc:515] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:25.910871 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:25.944968 17681 raft_consensus.cc:2468] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 11.
I20260709 12:23:25.911854 23650 raft_consensus.cc:493] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:25.915060 23932 log.cc:927] Time spent T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.073s	user 0.001s	sys 0.000s
I20260709 12:23:25.904637 23851 raft_consensus.cc:493] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:23:25.941900 17684 raft_consensus.cc:2468] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 18.
I20260709 12:23:25.900828 17683 rpcz_store.cc:276] 0709 12:23:23.747314 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:24.120213 (+372899us) service_pool.cc:224] Handling call
0709 12:23:24.120704 (+   491us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:24.128077 (+  7373us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:24.128127 (+    50us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:24.706231 (+578104us) spinlock_profiling.cc:243] Waited 5.59 ms on lock 0x7b18004c2880. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b475012a6 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b48f19a11 00007f7b48f29372 00007f7b4a65ba01 00007f7b4a62043a
0709 12:23:25.533345 (+827114us) spinlock_profiling.cc:243] Waited 6.78 ms on lock 0x7b2c00000c88. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b48f19a11 00007f7b48f29372 00007f7b4a65ba01 00007f7b4a62043a 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2
0709 12:23:25.533795 (+   450us) log.cc:844] Serialized 26 byte log entry
0709 12:23:25.730428 (+196633us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:23:25.730480 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:25.730575 (+    95us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:25.858060 (+127485us) raft_consensus.cc:1685] finished
0709 12:23:25.858080 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:25.858229 (+   149us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":10819,"raft.run_cpu_time_us":154,"raft.run_wall_time_us":15471,"spinlock_wait_cycles":28443264,"thread_start_us":196191,"threads_started":1,"wal-append.queue_time_us":322021}
W20260709 12:23:25.942848 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 225675us
I20260709 12:23:25.916136 23518 leader_election.cc:290] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:25.900594 23948 raft_consensus.cc:515] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:23:25.943692 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 202022us
I20260709 12:23:25.938248 23914 raft_consensus.cc:493] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:25.935686 23737 raft_consensus.cc:493] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
W20260709 12:23:25.972697 17654 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:25.976380 23948 leader_election.cc:290] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:25.947714 23838 leader_election.cc:290] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 16 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:23:25.962280 23924 log.cc:927] Time spent T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.102s	user 0.000s	sys 0.001s
I20260709 12:23:25.953312 23804 leader_election.cc:290] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:25.918713 23926 raft_consensus.cc:515] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:25.977140 23923 log.cc:927] Time spent T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.192s	user 0.000s	sys 0.000s
I20260709 12:23:25.968278 23914 raft_consensus.cc:3060] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:25.994861 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:25.981513 23943 raft_consensus.cc:493] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:25.920763 23929 raft_consensus.cc:515] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:25.952947 23650 raft_consensus.cc:515] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:25.995826 17683 raft_consensus.cc:2468] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 16.
I20260709 12:23:26.005535 17685 raft_consensus.cc:2468] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 13.
I20260709 12:23:25.986701 23470 leader_election.cc:290] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:25.988096 23934 raft_consensus.cc:493] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:26.014163 23956 consensus_queue.cc:1059] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 10.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:23:26.016520 23518 raft_consensus.cc:3055] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:23:26.017192 23518 raft_consensus.cc:740] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:23:26.018859 23902 leader_election.cc:290] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:25.944649 23916 raft_consensus.cc:515] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:26.022354 23908 leader_election.cc:290] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:26.024883 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:26.027448 23929 leader_election.cc:290] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:26.027980 23934 raft_consensus.cc:515] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:26.028862 17829 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35042 (request call id 2535) took 2389 ms. Trace:
I20260709 12:23:26.031905 23934 leader_election.cc:290] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:26.007702 23828 raft_consensus.cc:493] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:25.969636 23851 raft_consensus.cc:515] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:26.032618 23624 leader_election.cc:290] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:26.032701 23828 raft_consensus.cc:515] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:26.034150 23841 raft_consensus.cc:3055] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:23:26.034577 23841 raft_consensus.cc:740] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 14 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:23:26.035439 23841 consensus_queue.cc:260] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:26.011716 23943 raft_consensus.cc:515] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:26.036784 23841 raft_consensus.cc:3060] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:25.983461 23737 raft_consensus.cc:515] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:23:26.020052 23942 log.cc:927] Time spent T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.076s	user 0.000s	sys 0.001s
I20260709 12:23:25.982990 23869 raft_consensus.cc:515] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:26.030792 17654 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.16.29.68:42577 timed out after 1.813s (SENT) [suppressed 6 similar messages]
I20260709 12:23:26.050174 17688 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37402 (request call id 2990) took 1567 ms. Trace:
I20260709 12:23:26.036796 23926 leader_election.cc:290] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:26.053912 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:26.029235 17829 rpcz_store.cc:276] 0709 12:23:23.639127 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:24.377168 (+738041us) service_pool.cc:224] Handling call
0709 12:23:24.377583 (+   415us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:25.162678 (+785095us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:25.162723 (+    45us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:25.184113 (+ 21390us) log.cc:844] Serialized 26 byte log entry
0709 12:23:25.542184 (+358071us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:23:25.542219 (+    35us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:25.542281 (+    62us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:26.028658 (+486377us) raft_consensus.cc:1685] finished
0709 12:23:26.028678 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:26.028808 (+   130us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":79804,"raft.run_cpu_time_us":150,"raft.run_wall_time_us":9949,"thread_start_us":1142160,"threads_started":2,"wal-append.queue_time_us":525352}
I20260709 12:23:26.056124 17688 rpcz_store.cc:276] 0709 12:23:24.482730 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:24.676039 (+193309us) service_pool.cc:224] Handling call
0709 12:23:26.049793 (+1373754us) spinlock_profiling.cc:243] Waited 92.1 ms on lock 0x7b54002526d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:23:26.050070 (+   277us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":211943680}
I20260709 12:23:26.058548 17683 raft_consensus.cc:2393] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 17: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:23:25.943794 17580 catalog_manager.cc:5697] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb reported cstate change: term changed from 7 to 14, leader changed from e063541542a94969898868bf6eb5ce1b (127.16.29.67) to e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66). New cstate: current_term: 14 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: HEALTHY } } }
I20260709 12:23:26.042043 23735 raft_consensus.cc:493] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:26.057425 23573 leader_election.cc:290] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:23:26.068711 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 71556.7us
I20260709 12:23:26.058894 23925 raft_consensus.cc:493] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:23:26.077925 18364 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59902 (request call id 1480) took 2047 ms. Trace:
I20260709 12:23:26.078454 18364 rpcz_store.cc:276] 0709 12:23:24.030123 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:24.357210 (+327087us) service_pool.cc:224] Handling call
0709 12:23:26.077727 (+1720517us) spinlock_profiling.cc:243] Waited 997 ms on lock 0x7b54006144d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:23:26.077863 (+   136us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2292525824}
I20260709 12:23:26.084668 17580 catalog_manager.cc:5697] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb reported cstate change: term changed from 10 to 15, leader changed from e063541542a94969898868bf6eb5ce1b (127.16.29.67) to e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66). New cstate: current_term: 15 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: HEALTHY } } }
I20260709 12:23:26.086153 23843 raft_consensus.cc:3060] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:26.086586 17580 catalog_manager.cc:5697] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb reported cstate change: term changed from 10 to 12. New cstate: current_term: 12 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:23:26.088904 23916 leader_election.cc:290] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:26.089493 23796 raft_consensus.cc:3055] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:23:26.089881 17580 catalog_manager.cc:5697] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb reported cstate change: term changed from 13 to 15, leader changed from e063541542a94969898868bf6eb5ce1b (127.16.29.67) to e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66). New cstate: current_term: 15 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:23:26.092500 23737 leader_election.cc:290] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:26.092105 17580 catalog_manager.cc:5697] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb reported cstate change: term changed from 8 to 10, leader changed from 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) to e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66). New cstate: current_term: 10 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:23:26.094013 17580 catalog_manager.cc:5697] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb reported cstate change: term changed from 5 to 10, leader changed from e063541542a94969898868bf6eb5ce1b (127.16.29.67) to 0da169ff82674dc1947055ce43de68a3 (127.16.29.65). New cstate: current_term: 10 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } }
W20260709 12:23:26.102921 18334 leader_election.cc:336] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 14 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Timed out: RequestConsensusVote RPC to 127.16.29.65:33071 timed out after 1.816s (SENT)
I20260709 12:23:26.103559 18334 leader_election.cc:304] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:26.101181 17652 leader_election.cc:304] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:26.110600 17652 leader_election.cc:304] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:26.111943 17652 leader_election.cc:304] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:26.110431 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:23:26.114605 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 88868.3us
W20260709 12:23:26.117955 17728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:26.114092 17652 leader_election.cc:304] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:23:26.119014 17652 leader_election.cc:304] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:26.091048 23925 raft_consensus.cc:515] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:26.134322 17652 leader_election.cc:304] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
W20260709 12:23:26.111580 23864 consensus_meta.cc:287] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.636s	user 0.008s	sys 0.006s
I20260709 12:23:26.097066 23796 raft_consensus.cc:740] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:23:26.147465 17689 raft_consensus.cc:2410] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for term 18 because replica has last-logged OpId of term: 17 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 13.
I20260709 12:23:26.107695 23737 raft_consensus.cc:493] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:26.091135 23651 raft_consensus.cc:515] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:26.145736 17652 leader_election.cc:304] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:23:26.069453 23735 raft_consensus.cc:515] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:26.120280 23470 raft_consensus.cc:2804] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:23:26.117324 23929 raft_consensus.cc:2749] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 12:23:26.120687 23934 raft_consensus.cc:2804] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:23:26.144734 23915 raft_consensus.cc:493] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:26.166941 17652 leader_election.cc:304] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:26.139981 23941 raft_consensus.cc:2804] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Leader election won for term 15
W20260709 12:23:26.168974 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 205880us
I20260709 12:23:26.173803 18333 leader_election.cc:304] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:23:26.139724 23950 raft_consensus.cc:2749] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 12:23:26.171663 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 16 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:26.180950 23904 raft_consensus.cc:493] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:23:26.195089 23904 raft_consensus.cc:515] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:26.176492 23864 raft_consensus.cc:2804] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:23:26.180794 23838 consensus_queue.cc:1048] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.016s
I20260709 12:23:26.197197 23904 leader_election.cc:290] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:26.181988 23957 raft_consensus.cc:493] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:26.198087 23957 raft_consensus.cc:515] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:26.198294 23861 raft_consensus.cc:493] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:23:26.159927 23927 raft_consensus.cc:2804] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:23:26.163633 23573 raft_consensus.cc:493] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:26.169942 23934 raft_consensus.cc:493] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:26.164067 23470 raft_consensus.cc:493] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:23:26.207558 17686 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37396 (request call id 2595) took 2458 ms. Trace:
I20260709 12:23:26.199031 23861 raft_consensus.cc:515] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:26.169667 23955 raft_consensus.cc:2804] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:23:26.159795 23737 raft_consensus.cc:515] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:26.172359 23919 leader_election.cc:290] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:26.365931 17686 rpcz_store.cc:276] 0709 12:23:23.742300 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:23.742423 (+   123us) service_pool.cc:224] Handling call
0709 12:23:25.500544 (+1758121us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:25.543099 (+ 42555us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:26.200199 (+657100us) spinlock_profiling.cc:243] Waited 1.72 s on lock 0x7b5400426dd0. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a61f8ad 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:23:26.200414 (+   215us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3965412096}
I20260709 12:23:26.206980 23864 raft_consensus.cc:697] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 14 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:23:26.389784 23864 consensus_queue.cc:237] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.11, Last appended by leader: 11, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:26.186094 23526 leader_election.cc:290] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:26.176285 23947 raft_consensus.cc:493] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:26.168040 23906 consensus_queue.cc:1048] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.191s
I20260709 12:23:26.200268 23941 raft_consensus.cc:697] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 15 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:23:26.182353 23931 leader_election.cc:290] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:23:26.407727 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 236671us
I20260709 12:23:26.204890 23927 raft_consensus.cc:493] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:23:26.415225 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 17 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:26.417153 17688 raft_consensus.cc:3060] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:26.369524 23866 raft_consensus.cc:3055] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:23:26.424594 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:23:26.420768 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 317880us
I20260709 12:23:26.425721 18334 leader_election.cc:304] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:26.367239 23934 raft_consensus.cc:3060] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:26.372221 23470 raft_consensus.cc:3060] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:26.440811 23955 raft_consensus.cc:697] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 18 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:23:26.441921 23955 consensus_queue.cc:237] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.12, Last appended by leader: 12, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:26.419128 23927 raft_consensus.cc:3060] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:23:26.423836 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 250124us
I20260709 12:23:26.447196 18333 leader_election.cc:304] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:23:26.421015 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 14 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:26.368149 23573 raft_consensus.cc:515] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:26.432662 23835 raft_consensus.cc:3060] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:26.433550 23941 consensus_queue.cc:237] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.11, Last appended by leader: 11, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:26.177347 23915 raft_consensus.cc:515] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:26.434746 23904 raft_consensus.cc:2804] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:23:26.435287 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:26.479211 17683 raft_consensus.cc:2468] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 15.
I20260709 12:23:26.453454 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:26.469293 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:23:26.476784 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 51154.5us
W20260709 12:23:26.478403 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 53006.6us
I20260709 12:23:26.416100 23911 raft_consensus.cc:493] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:26.491228 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:26.480608 17689 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 13.
I20260709 12:23:26.492113 18374 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 13.
I20260709 12:23:26.492835 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:26.423584 23804 raft_consensus.cc:493] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:26.424532 23866 raft_consensus.cc:740] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 16 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:23:26.498454 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:26.500864 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 21 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:26.501847 18374 raft_consensus.cc:2468] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 20.
I20260709 12:23:26.502835 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:26.476984 23838 raft_consensus.cc:2804] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:23:26.510080 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:26.510624 18368 raft_consensus.cc:3060] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:23:26.507722 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 58956.8us
I20260709 12:23:26.483901 23948 raft_consensus.cc:2804] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:23:26.515038 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:23:26.525018 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 167229us
I20260709 12:23:26.515518 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:26.526289 18371 raft_consensus.cc:3060] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:26.479158 23957 leader_election.cc:290] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:26.478952 23904 raft_consensus.cc:493] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:26.516968 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:26.518754 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:26.518685 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:26.538810 17832 raft_consensus.cc:2468] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 18.
I20260709 12:23:26.521035 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:26.540113 18367 raft_consensus.cc:2468] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 14.
I20260709 12:23:26.522428 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:26.540622 17688 raft_consensus.cc:2468] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 17.
I20260709 12:23:26.527204 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:26.528928 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:26.525780 17727 leader_election.cc:304] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:26.529541 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:26.541870 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 13 candidate_status { last_received { term: 10 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:26.542333 17758 raft_consensus.cc:2468] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 15.
I20260709 12:23:26.543421 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:23:26.543963 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 61783.7us
I20260709 12:23:26.544684 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:26.529897 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:26.530893 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 21 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:26.532477 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:26.545341 18334 leader_election.cc:304] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:23:26.545982 17755 raft_consensus.cc:2468] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 13.
I20260709 12:23:26.546041 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:26.538461 17759 raft_consensus.cc:2468] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 18.
I20260709 12:23:26.548094 17727 leader_election.cc:304] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
W20260709 12:23:26.535548 17652 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:26.537191 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:26.489254 23858 consensus_queue.cc:1048] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.024s
I20260709 12:23:26.557471 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:26.412464 23796 consensus_queue.cc:260] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 16.14, Last appended by leader: 14, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:26.497143 23953 raft_consensus.cc:2749] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:23:26.507575 23838 raft_consensus.cc:493] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:26.550733 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 17 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:26.551297 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 17 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:26.575387 17833 raft_consensus.cc:2468] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 15.
I20260709 12:23:26.561264 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:26.576733 17687 raft_consensus.cc:2468] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 14.
I20260709 12:23:26.559345 17803 leader_election.cc:304] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:23:26.551828 18374 raft_consensus.cc:2468] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 11.
I20260709 12:23:26.470646 23518 consensus_queue.cc:260] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 11.13, Last appended by leader: 13, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:26.549893 17727 leader_election.cc:304] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:26.513377 23855 raft_consensus.cc:493] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:26.549561 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:26.602838 17838 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 13.
I20260709 12:23:26.563162 23651 leader_election.cc:290] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 14 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:23:26.570931 23841 consensus_meta.cc:287] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.534s	user 0.006s	sys 0.002s
I20260709 12:23:26.574592 18334 leader_election.cc:304] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:23:26.584990 18333 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:26.419713 23947 raft_consensus.cc:515] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:26.586270 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:26.587566 23573 leader_election.cc:290] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:26.587956 23470 raft_consensus.cc:515] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:26.556545 23906 raft_consensus.cc:2749] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:23:26.498418 23911 raft_consensus.cc:515] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:26.589386 17652 leader_election.cc:304] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:23:26.592914 17803 leader_election.cc:304] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:26.591015 23957 raft_consensus.cc:2804] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:23:26.511286 23804 raft_consensus.cc:515] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:26.588611 23735 leader_election.cc:290] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:26.559830 23916 consensus_queue.cc:1048] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.188s
I20260709 12:23:26.771160 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:26.580403 23861 leader_election.cc:290] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:23:26.682866 23913 consensus_meta.cc:287] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.979s	user 0.011s	sys 0.000s
I20260709 12:23:26.776221 23913 raft_consensus.cc:515] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:26.778450 23913 leader_election.cc:290] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 16 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:26.744904 23470 leader_election.cc:290] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 15 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:26.577099 23838 raft_consensus.cc:3060] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:23:26.748442 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 173925us
I20260709 12:23:26.785790 23804 leader_election.cc:290] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:26.748567 17681 raft_consensus.cc:2410] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for term 16 because replica has last-logged OpId of term: 14 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 8.
I20260709 12:23:26.752151 17764 raft_consensus.cc:2468] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 15.
W20260709 12:23:26.754782 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 76708.7us
I20260709 12:23:26.788152 23911 leader_election.cc:290] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:26.789050 23906 raft_consensus.cc:2804] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:23:26.789467 23906 raft_consensus.cc:493] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:26.789819 23906 raft_consensus.cc:3060] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:26.789865 18333 leader_election.cc:304] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:23:26.790262 23843 consensus_meta.cc:287] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.704s	user 0.008s	sys 0.000s
I20260709 12:23:26.790870 17684 raft_consensus.cc:2468] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 16.
I20260709 12:23:26.791492 23737 leader_election.cc:290] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:23:26.755260 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 162426us
I20260709 12:23:26.754079 23866 consensus_queue.cc:260] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.9, Last appended by leader: 9, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:26.792750 23915 leader_election.cc:290] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:26.792821 17803 leader_election.cc:304] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:26.793103 23866 raft_consensus.cc:3060] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:26.793422 23915 consensus_queue.cc:1048] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 12:23:26.794303 17803 leader_election.cc:304] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:23:26.794785 23915 raft_consensus.cc:2749] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:23:26.544725 23904 raft_consensus.cc:3060] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:26.759482 23957 raft_consensus.cc:493] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:26.673672 23796 raft_consensus.cc:3060] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:26.674680 23812 raft_consensus.cc:3055] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:23:26.674996 23852 raft_consensus.cc:2804] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:23:26.761145 23526 raft_consensus.cc:2749] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:23:26.767014 18368 raft_consensus.cc:2468] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 15.
I20260709 12:23:26.768348 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:26.852720 18361 raft_consensus.cc:2468] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 12.
I20260709 12:23:26.577903 23901 consensus_queue.cc:1048] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.184s
I20260709 12:23:26.774565 23947 leader_election.cc:290] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:26.763079 23930 raft_consensus.cc:2804] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:23:26.795619 23735 raft_consensus.cc:2749] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
W20260709 12:23:26.760346 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 171086us
I20260709 12:23:26.885310 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:26.885718 16501 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:23:26.886135 18361 raft_consensus.cc:2368] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 15. Current term is 16.
I20260709 12:23:26.751173 23573 raft_consensus.cc:2749] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 12:23:26.797134 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 18 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:26.798842 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:26.847328 18371 raft_consensus.cc:2468] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 18.
I20260709 12:23:26.892740 18367 raft_consensus.cc:2410] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 18 because replica has last-logged OpId of term: 15 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 11.
I20260709 12:23:26.893262 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:26.854187 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 14 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:26.857165 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 13 candidate_status { last_received { term: 10 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:26.680830 23855 raft_consensus.cc:515] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:26.772168 23841 raft_consensus.cc:493] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:26.880204 17684 raft_consensus.cc:1240] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 17. Current term is 18. Ops: []
I20260709 12:23:26.878178 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:26.895314 17755 raft_consensus.cc:2468] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 10.
W20260709 12:23:26.779807 23922 consensus_meta.cc:287] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.824s	user 0.002s	sys 0.005s
I20260709 12:23:26.888444 17687 raft_consensus.cc:1275] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 15 index: 10. Preceding OpId from leader: term: 16 index: 11. (index mismatch)
W20260709 12:23:26.890655 17728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:26.898401 18371 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 13.
I20260709 12:23:26.868048 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:26.900707 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:26.892724 17652 leader_election.cc:304] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:23:26.901234 17681 raft_consensus.cc:1240] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 12. Current term is 13. Ops: []
I20260709 12:23:26.844103 23651 raft_consensus.cc:2764] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Leader pre-election decision vote started in defunct term 17: won
I20260709 12:23:26.899478 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:26.904510 17762 raft_consensus.cc:1240] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 16. Current term is 17. Ops: []
I20260709 12:23:26.850464 23812 raft_consensus.cc:740] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:23:26.907940 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:26.856324 23852 raft_consensus.cc:493] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:26.902290 17652 leader_election.cc:304] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:26.911234 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:26.843806 23957 raft_consensus.cc:3060] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Advancing to term 17
W20260709 12:23:26.914517 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:26.914685 23852 raft_consensus.cc:3060] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:23:26.902055 23914 consensus_meta.cc:287] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.895s	user 0.012s	sys 0.000s
I20260709 12:23:26.923012 17680 raft_consensus.cc:1240] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 16. Current term is 17. Ops: []
I20260709 12:23:26.926543 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:26.923599 17652 leader_election.cc:304] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:26.923755 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:26.927804 18361 raft_consensus.cc:2410] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 12 because replica has last-logged OpId of term: 10 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 8.
I20260709 12:23:26.929544 17803 leader_election.cc:304] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:26.850963 23969 consensus_queue.cc:1048] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.116s
I20260709 12:23:26.882264 23915 consensus_queue.cc:1048] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.060s
I20260709 12:23:26.938452 17652 leader_election.cc:304] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:23:26.911121 23928 raft_consensus.cc:2804] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:23:26.936158 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:23:26.920966 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 371163us
I20260709 12:23:26.901808 23737 raft_consensus.cc:493] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:26.901276 23930 raft_consensus.cc:697] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 13 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:23:26.935374 17803 leader_election.cc:400] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 15 election: Vote denied by peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577) with higher term. Message: Invalid argument: T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 15. Current term is 16.
I20260709 12:23:26.955528 17803 leader_election.cc:403] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 15 election: Cancelling election due to peer responding with higher term
W20260709 12:23:26.956106 17802 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.765s	user 0.014s	sys 0.044s
I20260709 12:23:26.942425 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
W20260709 12:23:26.956578 17802 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.766s	user 0.014s	sys 0.045s
I20260709 12:23:26.940310 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:26.941859 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:26.926558 23470 raft_consensus.cc:2749] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
W20260709 12:23:26.956985 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 766337us
I20260709 12:23:26.967178 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:26.956714 17727 leader_election.cc:304] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:26.934626 23651 raft_consensus.cc:2749] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:23:26.947542 17652 leader_election.cc:304] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:26.908754 23624 raft_consensus.cc:493] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:23:26.960637 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 18 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:26.909951 23922 raft_consensus.cc:515] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:26.912086 23841 raft_consensus.cc:515] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:26.964915 17755 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 12.
I20260709 12:23:26.990574 23955 raft_consensus.cc:2804] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:23:26.969821 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:26.958483 23928 raft_consensus.cc:493] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:23:26.991921 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:26.934448 23934 raft_consensus.cc:515] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:26.960664 23735 raft_consensus.cc:3060] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:27.003419 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:27.017143 23651 raft_consensus.cc:493] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:23:27.017912 23651 raft_consensus.cc:515] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:26.985039 23858 raft_consensus.cc:2804] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:23:26.966413 23737 raft_consensus.cc:515] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:27.026400 18366 raft_consensus.cc:2410] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for term 19 because replica has last-logged OpId of term: 18 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 10.
I20260709 12:23:27.033041 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:27.034775 17759 raft_consensus.cc:3060] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:27.038955 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:27.048187 17755 raft_consensus.cc:2468] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 12.
I20260709 12:23:27.009953 23955 raft_consensus.cc:493] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:27.010824 23928 raft_consensus.cc:3060] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:26.937389 23914 raft_consensus.cc:515] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:26.957544 23838 raft_consensus.cc:515] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:27.062783 17682 raft_consensus.cc:1240] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 18. Current term is 19. Ops: [18.13-18.13]
W20260709 12:23:27.040704 23932 log.cc:927] Time spent T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.052s	user 0.000s	sys 0.000s
I20260709 12:23:27.041878 23858 raft_consensus.cc:493] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:26.991045 23624 raft_consensus.cc:515] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:27.067342 17763 raft_consensus.cc:2468] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 16.
I20260709 12:23:27.032908 23940 raft_consensus.cc:493] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:27.034186 23979 raft_consensus.cc:493] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:27.064309 23955 raft_consensus.cc:3060] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Advancing to term 14
W20260709 12:23:27.089955 17803 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:27.093595 17683 consensus_meta.cc:287] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.548s	user 0.010s	sys 0.000s
I20260709 12:23:26.968140 23812 consensus_queue.cc:260] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 17.10, Last appended by leader: 10, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:27.082291 23858 raft_consensus.cc:3060] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:27.118786 23922 leader_election.cc:290] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 16 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:27.123001 23852 raft_consensus.cc:515] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:27.125103 23914 leader_election.cc:290] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 15 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:27.125519 23624 leader_election.cc:290] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:27.088184 23911 consensus_queue.cc:1048] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.038s
I20260709 12:23:27.127036 23911 consensus_queue.cc:1048] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 12:23:27.130203 23841 leader_election.cc:290] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:27.130982 23841 raft_consensus.cc:493] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:27.132197 23855 leader_election.cc:290] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:27.131453 23841 raft_consensus.cc:515] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:27.097358 23940 raft_consensus.cc:515] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:27.133296 23812 raft_consensus.cc:3060] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:27.133879 23737 leader_election.cc:290] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:27.134274 23841 leader_election.cc:290] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:27.134588 23940 leader_election.cc:290] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:27.083575 23992 raft_consensus.cc:493] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
W20260709 12:23:27.132470 23977 log.cc:927] Time spent T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.096s	user 0.000s	sys 0.001s
I20260709 12:23:27.141578 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:27.142428 17830 raft_consensus.cc:2468] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 12.
W20260709 12:23:27.142751 17652 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:27.148154 17830 raft_consensus.cc:3060] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:27.136070 17683 raft_consensus.cc:2468] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 18.
I20260709 12:23:27.138971 17832 raft_consensus.cc:1217] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Deduplicated request from leader. Original: 7.8->[10.9-14.11]   Dedup: 10.9->[12.10-14.11]
I20260709 12:23:27.160439 17832 pending_rounds.cc:85] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b: Aborting all ops after (but not including) 9
I20260709 12:23:27.160832 17832 pending_rounds.cc:107] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b: Aborting uncommitted NO_OP operation due to leader change: 11.10
I20260709 12:23:27.066113 17759 raft_consensus.cc:2468] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 16.
I20260709 12:23:27.140435 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:27.161224 17832 raft_consensus.cc:2889] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:27.139267 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:27.161882 17829 raft_consensus.cc:2393] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 16: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:23:27.167678 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:27.168330 17836 raft_consensus.cc:1275] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 12 index: 10. (index mismatch)
I20260709 12:23:27.161015 17686 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37396 (request call id 2613) took 1044 ms. Trace:
I20260709 12:23:27.167773 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:27.108330 23979 raft_consensus.cc:515] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:27.168555 17835 raft_consensus.cc:2393] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 14: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:23:27.172307 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:27.163563 23700 raft_consensus.cc:2804] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:23:27.173125 17829 raft_consensus.cc:2468] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 16.
I20260709 12:23:27.170960 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:27.140580 23949 raft_consensus.cc:2749] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:23:27.176445 17833 raft_consensus.cc:2393] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 19: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:23:27.174785 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:27.176470 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:27.180790 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:27.181012 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 14 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:27.185043 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 16 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:27.185353 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:27.186636 17836 raft_consensus.cc:2410] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for term 16 because replica has last-logged OpId of term: 14 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 8.
I20260709 12:23:27.186532 17835 raft_consensus.cc:2468] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 17.
I20260709 12:23:27.185164 17837 raft_consensus.cc:2368] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 17. Current term is 18.
I20260709 12:23:27.169317 17686 rpcz_store.cc:276] 0709 12:23:26.116130 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:26.844527 (+728397us) service_pool.cc:224] Handling call
0709 12:23:26.885205 (+ 40678us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:26.887997 (+  2792us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:26.888030 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:26.888496 (+   466us) log.cc:844] Serialized 26 byte log entry
0709 12:23:27.068805 (+180309us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:23:27.068880 (+    75us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:27.068960 (+    80us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:27.160714 (+ 91754us) raft_consensus.cc:1685] finished
0709 12:23:27.160740 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:27.160939 (+   199us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1689,"raft.queue_time_us":2405,"raft.run_cpu_time_us":116,"raft.run_wall_time_us":7526,"thread_start_us":151802,"threads_started":1,"wal-append.queue_time_us":153988}
I20260709 12:23:27.188625 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:27.188719 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:27.189531 17834 raft_consensus.cc:2468] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 11.
I20260709 12:23:27.169997 23913 raft_consensus.cc:493] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:23:27.192991 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:27.193549 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:27.194172 17833 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 13.
I20260709 12:23:27.199903 18374 raft_consensus.cc:2468] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 17.
I20260709 12:23:27.201251 23852 leader_election.cc:290] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 13 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:27.082791 23930 consensus_queue.cc:237] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:27.208247 23838 leader_election.cc:290] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 15 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:27.209823 23928 raft_consensus.cc:515] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:27.213250 23979 leader_election.cc:290] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:27.213466 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 18 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:27.214083 17833 raft_consensus.cc:2468] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 17.
I20260709 12:23:27.216454 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:27.217161 17833 raft_consensus.cc:2393] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 15: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:23:27.217037 23955 raft_consensus.cc:515] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:27.219548 23869 leader_election.cc:290] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 19 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:27.220203 23934 leader_election.cc:290] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 19 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:27.228817 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:27.233671 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:27.234369 17835 raft_consensus.cc:3060] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:27.235749 17728 leader_election.cc:304] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:27.236653 17728 leader_election.cc:304] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:27.240175 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:27.242764 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:27.255478 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:27.253281 17728 leader_election.cc:304] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:23:27.256567 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 170796us
I20260709 12:23:27.258093 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:27.256316 17683 raft_consensus.cc:2468] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 16.
I20260709 12:23:27.268566 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 19 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:27.269902 17680 raft_consensus.cc:2410] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for term 19 because replica has last-logged OpId of term: 17 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 12.
I20260709 12:23:27.252307 23928 leader_election.cc:290] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 17 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:27.273545 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:27.274317 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:27.286180 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:27.289194 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:27.253391 23828 leader_election.cc:290] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:27.258114 23841 raft_consensus.cc:2804] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:23:27.258184 23979 raft_consensus.cc:2804] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:23:27.253136 23925 leader_election.cc:290] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:27.325536 23841 raft_consensus.cc:493] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:23:27.326421 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:27.326131 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:27.328250 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:27.328713 23967 consensus_queue.cc:1048] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 12:23:27.333297 17802 leader_election.cc:304] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:27.326133 23841 raft_consensus.cc:3060] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:27.336113 23991 raft_consensus.cc:2749] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
W20260709 12:23:27.337852 23988 log.cc:927] Time spent T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.054s	user 0.000s	sys 0.000s
W20260709 12:23:27.324779 23835 consensus_meta.cc:287] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.845s	user 0.011s	sys 0.000s
I20260709 12:23:27.339049 23835 raft_consensus.cc:2749] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:23:27.254525 23573 consensus_queue.cc:1059] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 16.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.034s
I20260709 12:23:27.262523 23941 raft_consensus.cc:493] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:27.340247 18365 raft_consensus.cc:2468] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 14.
I20260709 12:23:27.340308 23941 raft_consensus.cc:515] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:27.342829 23941 leader_election.cc:290] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:27.330098 17680 raft_consensus.cc:1275] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 15 index: 11. Preceding OpId from leader: term: 16 index: 12. (index mismatch)
I20260709 12:23:27.296422 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:27.324685 23972 raft_consensus.cc:2804] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:23:27.297685 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 19 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:27.287678 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:27.303109 23998 raft_consensus.cc:2804] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Leader pre-election won for term 17
W20260709 12:23:27.304009 18333 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:27.349929 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:27.362951 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:27.261670 17729 leader_election.cc:304] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:23:27.363905 17833 raft_consensus.cc:2468] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 14.
W20260709 12:23:27.364585 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 103233us
W20260709 12:23:27.308808 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:27.335642 17802 leader_election.cc:304] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:27.374552 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:27.374687 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 18 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:27.356851 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:27.376713 17683 raft_consensus.cc:2393] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 13: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
W20260709 12:23:27.357026 23904 consensus_meta.cc:287] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.515s	user 0.003s	sys 0.007s
I20260709 12:23:27.382225 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:27.382695 23904 raft_consensus.cc:515] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:27.383430 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 15 candidate_status { last_received { term: 14 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:23:27.356040 23990 log.cc:927] Time spent T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.098s	user 0.000s	sys 0.001s
I20260709 12:23:27.385174 23904 leader_election.cc:290] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 14 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:27.359366 23917 leader_election.cc:290] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 13 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:27.360208 23735 raft_consensus.cc:2749] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Leader election lost for term 15. Reason: Vote denied by peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577) with higher term. Message: Invalid argument: T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 15. Current term is 16.
I20260709 12:23:27.325405 23979 raft_consensus.cc:493] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:23:27.386849 23979 raft_consensus.cc:3060] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:27.378942 17727 leader_election.cc:304] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:27.388720 17686 raft_consensus.cc:1240] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 14 LEADER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 13. Current term is 14. Ops: []
I20260709 12:23:27.394013 17687 raft_consensus.cc:2410] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for term 18 because replica has last-logged OpId of term: 17 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 13.
I20260709 12:23:27.397066 18371 raft_consensus.cc:2393] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 14: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:23:27.397274 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:27.400398 17831 raft_consensus.cc:2468] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 15.
I20260709 12:23:27.170305 23992 raft_consensus.cc:515] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:27.237697 23913 raft_consensus.cc:515] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:23:27.323303 23927 consensus_meta.cc:287] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.879s	user 0.011s	sys 0.000s
I20260709 12:23:27.298059 17684 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37402 (request call id 3002) took 1102 ms. Trace:
I20260709 12:23:27.346076 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:27.343868 23858 raft_consensus.cc:515] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:27.352787 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:27.430001 17684 rpcz_store.cc:276] 0709 12:23:26.195252 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:26.903142 (+707890us) service_pool.cc:224] Handling call
0709 12:23:26.903464 (+   322us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:26.903783 (+   319us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 12:23:27.048202 (+144419us) log.cc:844] Serialized 14 byte log entry
0709 12:23:27.297694 (+249492us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:27.297748 (+    54us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:23:27.297800 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:27.297872 (+    72us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:27.298003 (+   131us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":194072,"prepare.run_cpu_time_us":66,"prepare.run_wall_time_us":52,"raft.queue_time_us":2649,"raft.run_cpu_time_us":137,"raft.run_wall_time_us":139,"thread_start_us":374512,"threads_started":2,"wal-append.queue_time_us":249310}
I20260709 12:23:27.401803 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:27.402590 17829 raft_consensus.cc:2468] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 16.
I20260709 12:23:27.297364 17835 raft_consensus.cc:2468] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 15.
I20260709 12:23:27.436241 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 17 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:27.437594 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 15 candidate_status { last_received { term: 14 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:27.439011 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:23:27.349188 23906 consensus_meta.cc:287] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.559s	user 0.011s	sys 0.001s
I20260709 12:23:27.348053 23650 leader_election.cc:290] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:27.410326 17654 leader_election.cc:304] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:27.374017 23998 raft_consensus.cc:493] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:23:27.412461 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 146164us
I20260709 12:23:27.351017 23955 leader_election.cc:290] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 14 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:27.360060 23605 leader_election.cc:290] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 18 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:27.420426 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:27.403719 23740 raft_consensus.cc:2804] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:23:27.432072 17681 raft_consensus.cc:2468] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 13.
W20260709 12:23:27.455175 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 76365.4us
I20260709 12:23:27.362118 17801 leader_election.cc:304] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:27.345736 23957 raft_consensus.cc:515] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:27.440146 17684 raft_consensus.cc:1275] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 17 index: 14. Preceding OpId from leader: term: 18 index: 14. (term mismatch)
I20260709 12:23:27.458388 17684 pending_rounds.cc:85] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3: Aborting all ops after (but not including) 13
I20260709 12:23:27.458647 17684 pending_rounds.cc:107] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3: Aborting uncommitted NO_OP operation due to leader change: 17.14
I20260709 12:23:27.354457 23841 raft_consensus.cc:515] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:27.441835 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:27.443132 17654 leader_election.cc:304] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:27.445017 17830 raft_consensus.cc:2468] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 16.
I20260709 12:23:27.461774 23961 raft_consensus.cc:2804] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:23:27.462196 23961 raft_consensus.cc:493] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:23:27.462656 23961 raft_consensus.cc:3060] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:27.468339 17654 leader_election.cc:304] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:23:27.469867 17654 leader_election.cc:304] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:27.427759 23735 raft_consensus.cc:493] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:27.454784 17831 raft_consensus.cc:2393] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 10: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:23:27.458906 17684 raft_consensus.cc:2889] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:23:27.460400 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 128621us
I20260709 12:23:27.471458 23735 raft_consensus.cc:515] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:27.472289 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:27.473191 17801 leader_election.cc:304] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:27.474485 23735 leader_election.cc:290] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:27.445547 17835 raft_consensus.cc:2468] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 17.
I20260709 12:23:27.470644 23929 raft_consensus.cc:2804] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:23:27.475612 23929 raft_consensus.cc:493] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:27.476126 23929 raft_consensus.cc:3060] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:27.451129 23925 raft_consensus.cc:2804] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:23:27.447845 17687 raft_consensus.cc:1240] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 13. Current term is 14. Ops: [13.11-13.11]
I20260709 12:23:27.451550 23998 raft_consensus.cc:3060] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:27.482982 18333 leader_election.cc:304] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:27.479983 23605 raft_consensus.cc:2749] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:23:27.488833 17833 raft_consensus.cc:1240] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 13. Current term is 14. Ops: [13.12-13.12]
I20260709 12:23:27.444263 23906 raft_consensus.cc:515] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:27.490257 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:27.491001 17833 raft_consensus.cc:3060] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Advancing to term 14
I20260709 12:23:27.432936 23927 raft_consensus.cc:515] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:27.493669 23927 leader_election.cc:290] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 18 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:27.484623 23925 raft_consensus.cc:493] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:23:27.494530 23925 raft_consensus.cc:3060] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:27.494735 18362 raft_consensus.cc:2468] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 18.
I20260709 12:23:27.497010 18333 leader_election.cc:304] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:23:27.497887 18333 leader_election.cc:304] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:23:27.501554 18333 leader_election.cc:304] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:27.504069 18333 leader_election.cc:400] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 17 election: Vote denied by peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837) with higher term. Message: Invalid argument: T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 17. Current term is 18.
I20260709 12:23:27.504508 18333 leader_election.cc:403] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 17 election: Cancelling election due to peer responding with higher term
I20260709 12:23:27.506181 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:27.508545 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:27.517869 18333 leader_election.cc:304] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:23:27.518862 17838 raft_consensus.cc:1240] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 9. Current term is 10. Ops: []
I20260709 12:23:27.519837 18333 leader_election.cc:304] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:27.517204 18366 raft_consensus.cc:1275] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 16 index: 12. Preceding OpId from leader: term: 18 index: 13. (index mismatch)
I20260709 12:23:27.485392 23650 raft_consensus.cc:2804] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:23:27.494051 23904 raft_consensus.cc:2749] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:23:27.517514 23927 raft_consensus.cc:2804] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:23:27.495592 23961 raft_consensus.cc:515] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:27.487672 23812 raft_consensus.cc:2804] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Leader pre-election won for term 11
W20260709 12:23:27.524487 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 68298us
I20260709 12:23:27.491173 23906 leader_election.cc:290] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 17 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:27.497273 23941 raft_consensus.cc:2749] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:23:27.529946 18333 leader_election.cc:304] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:27.526734 17727 leader_election.cc:304] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:27.518800 23930 raft_consensus.cc:2749] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:23:27.493551 23979 raft_consensus.cc:515] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:27.517172 23951 raft_consensus.cc:2804] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:23:27.531847 17687 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01"
dest_uuid: "0da169ff82674dc1947055ce43de68a3"
mode: GRACEFUL
new_leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
 from {username='slave'} at 127.0.0.1:54540
I20260709 12:23:27.545325 17687 raft_consensus.cc:606] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Received request to transfer leadership to TS e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:27.546113 17687 raft_consensus.cc:614] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:23:27.523334 23927 raft_consensus.cc:493] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:27.565116 23651 leader_election.cc:290] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:27.525761 23812 raft_consensus.cc:493] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:23:27.539400 18333 leader_election.cc:304] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:23:27.523653 23838 raft_consensus.cc:2749] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Leader election lost for term 17. Reason: Vote denied by peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837) with higher term. Message: Invalid argument: T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 17. Current term is 18.
I20260709 12:23:27.539425 17833 raft_consensus.cc:2468] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 14.
I20260709 12:23:27.528817 24013 raft_consensus.cc:2804] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:23:27.531690 23947 raft_consensus.cc:2749] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:23:27.532657 23796 raft_consensus.cc:2804] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:23:27.573128 17833 raft_consensus.cc:1240] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 15. Current term is 16. Ops: []
I20260709 12:23:27.532099 23650 raft_consensus.cc:697] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 17 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:23:27.524870 23835 consensus_queue.cc:1048] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.042s
I20260709 12:23:27.552423 23855 raft_consensus.cc:2749] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:23:27.544627 23951 raft_consensus.cc:493] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:27.544306 23624 consensus_queue.cc:1048] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 10, Time since last communication: 0.043s
I20260709 12:23:27.552942 23906 consensus_queue.cc:1048] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.047s
W20260709 12:23:27.597882 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 71217.7us
I20260709 12:23:27.661693 23957 leader_election.cc:290] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 17 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:23:27.644680 17804 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:27.681423 17727 leader_election.cc:304] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
W20260709 12:23:27.661243 24008 log.cc:927] Time spent T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.116s	user 0.000s	sys 0.000s
W20260709 12:23:27.652787 24007 log.cc:927] Time spent T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.152s	user 0.000s	sys 0.001s
I20260709 12:23:27.683017 23951 raft_consensus.cc:3060] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Advancing to term 12
W20260709 12:23:27.667587 24001 log.cc:927] Time spent T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.190s	user 0.001s	sys 0.000s
I20260709 12:23:27.687230 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:27.687960 17836 raft_consensus.cc:3060] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:27.689314 23851 leader_election.cc:290] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:27.691150 23948 raft_consensus.cc:493] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:23:27.581591 24013 raft_consensus.cc:493] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:27.677383 23913 leader_election.cc:290] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:27.692487 24013 raft_consensus.cc:3060] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:27.694681 23979 leader_election.cc:290] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 15 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:27.695580 23979 raft_consensus.cc:2749] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
W20260709 12:23:27.662624 24003 log.cc:927] Time spent T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.125s	user 0.000s	sys 0.000s
I20260709 12:23:27.589036 23904 raft_consensus.cc:2804] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:23:27.697535 23904 raft_consensus.cc:493] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:23:27.697924 23904 raft_consensus.cc:3060] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:27.679361 23841 leader_election.cc:290] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 16 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:27.673274 23961 leader_election.cc:290] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 21 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:27.700932 23700 raft_consensus.cc:493] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:23:27.701191 24005 log.cc:927] Time spent T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.120s	user 0.000s	sys 0.001s
I20260709 12:23:27.701505 23700 raft_consensus.cc:3060] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:27.704200 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:27.569789 23927 raft_consensus.cc:3060] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:23:27.678443 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 138010us
I20260709 12:23:27.719514 17836 raft_consensus.cc:2468] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 17.
I20260709 12:23:27.707201 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:27.721288 17829 raft_consensus.cc:3060] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:27.570760 23812 raft_consensus.cc:3060] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:23:27.724861 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 185592us
I20260709 12:23:27.726938 18333 leader_election.cc:304] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
W20260709 12:23:27.674571 23966 log.cc:927] Time spent T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.051s	user 0.001s	sys 0.000s
W20260709 12:23:27.674156 24004 log.cc:927] Time spent T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.169s	user 0.000s	sys 0.000s
I20260709 12:23:27.745270 17829 raft_consensus.cc:2468] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 17.
I20260709 12:23:27.746728 18333 leader_election.cc:304] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:27.572994 23804 consensus_queue.cc:1059] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 11.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.159s
I20260709 12:23:27.692348 23948 raft_consensus.cc:3060] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:27.676031 23858 leader_election.cc:290] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 17 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:27.582890 23796 raft_consensus.cc:493] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:23:27.748914 23951 raft_consensus.cc:515] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:27.784909 18333 leader_election.cc:304] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:27.673408 24019 raft_consensus.cc:2749] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:23:27.790304 23972 raft_consensus.cc:493] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:23:27.638641 23992 leader_election.cc:290] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:23:27.780462 24014 log.cc:927] Time spent T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.124s	user 0.000s	sys 0.001s
I20260709 12:23:27.782428 23785 raft_consensus.cc:2749] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:23:27.786098 23624 raft_consensus.cc:3060] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:27.815784 23650 consensus_queue.cc:237] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.8, Last appended by leader: 8, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:23:27.821439 23951 leader_election.cc:290] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 12 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:27.825807 23913 raft_consensus.cc:493] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:27.826363 23913 raft_consensus.cc:515] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:27.829697 23913 leader_election.cc:290] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:27.792855 23796 raft_consensus.cc:3060] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:23:27.794209 23964 log.cc:927] Time spent T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 1.345s	user 0.008s	sys 0.005s
I20260709 12:23:27.832333 23905 leader_election.cc:290] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:23:27.841150 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 44537.5us
I20260709 12:23:27.834401 23624 raft_consensus.cc:2749] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:23:27.849023 18333 leader_election.cc:304] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:23:27.853960 18333 leader_election.cc:304] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:23:27.806545 23804 raft_consensus.cc:2749] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:23:27.839083 23972 raft_consensus.cc:3060] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:27.862912 17764 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38776 (request call id 2582) took 1334 ms. Trace:
W20260709 12:23:27.865132 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 175551us
I20260709 12:23:27.863323 17764 rpcz_store.cc:276] 0709 12:23:26.528424 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:26.528663 (+   239us) service_pool.cc:224] Handling call
0709 12:23:27.862737 (+1334074us) spinlock_profiling.cc:243] Waited 210 ms on lock 0x7b54003eb554. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:23:27.862852 (+   115us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":483091840}
W20260709 12:23:27.865669 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 499637us
I20260709 12:23:27.865767 24026 raft_consensus.cc:3055] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:23:27.866411 24026 raft_consensus.cc:740] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:23:27.866559 17729 leader_election.cc:304] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:27.867022 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:27.867828 17729 leader_election.cc:304] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:23:27.825145 23700 raft_consensus.cc:515] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:27.809132 23957 consensus_queue.cc:1059] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 12.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.052s
W20260709 12:23:27.941735 17803 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:27.939553 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:27.945626 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:27.946869 17764 raft_consensus.cc:2393] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 15: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:23:27.938385 17756 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38786 (request call id 3275) took 2278 ms. Trace:
I20260709 12:23:27.889747 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:27.947455 17756 rpcz_store.cc:276] 0709 12:23:25.601387 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:25.601536 (+   149us) service_pool.cc:224] Handling call
0709 12:23:25.601793 (+   257us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:25.871679 (+269886us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 12:23:26.041976 (+170297us) log.cc:844] Serialized 14 byte log entry
0709 12:23:26.042480 (+   504us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:26.042536 (+    56us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:23:26.042583 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:26.172930 (+130347us) spinlock_profiling.cc:243] Waited 5.19 ms on lock 0x7b180007b1c0. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b475012a6 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419
0709 12:23:27.880012 (+1707082us) spinlock_profiling.cc:243] Waited 270 ms on lock 0x7b5400415ad4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:23:27.880080 (+    68us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:27.880222 (+   142us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":161334,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":34,"raft.queue_time_us":165,"raft.run_cpu_time_us":108,"raft.run_wall_time_us":110,"spinlock_wait_cycles":631993984,"thread_start_us":140003,"threads_started":1}
W20260709 12:23:27.879671 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:27.886679 17830 raft_consensus.cc:1240] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 16. Current term is 17. Ops: [16.14-16.14]
I20260709 12:23:27.887594 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:27.945433 17835 raft_consensus.cc:2368] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 18. Current term is 19.
I20260709 12:23:27.951671 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:27.959179 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:27.853200 24019 raft_consensus.cc:493] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:23:27.887190 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:27.973896 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:27.974417 17838 raft_consensus.cc:3060] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:27.952266 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:27.867282 24026 consensus_queue.cc:260] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 17.10, Last appended by leader: 10, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:27.958959 17829 raft_consensus.cc:3060] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:27.959659 18361 raft_consensus.cc:2368] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 16. Current term is 17.
I20260709 12:23:27.964939 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 17 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:27.962453 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 19 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
W20260709 12:23:27.879451 23970 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.770s	user 0.006s	sys 0.022s
I20260709 12:23:27.974494 18368 raft_consensus.cc:3060] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:27.860491 23951 raft_consensus.cc:2804] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:23:27.975154 18364 raft_consensus.cc:2468] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 17.
I20260709 12:23:27.973584 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:27.948459 23858 raft_consensus.cc:3060] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:27.853443 23785 consensus_queue.cc:1059] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 9.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.095s
W20260709 12:23:27.946486 23931 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 1.410s	user 0.014s	sys 0.037s
W20260709 12:23:27.959628 24027 log.cc:927] Time spent T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.086s	user 0.000s	sys 0.008s
I20260709 12:23:27.860307 23838 raft_consensus.cc:2804] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:23:27.989537 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:27.992340 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:27.973562 23997 raft_consensus.cc:493] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:23:28.006448 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:27.859716 23855 consensus_queue.cc:1059] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 15.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.126s
I20260709 12:23:27.998654 17829 raft_consensus.cc:2468] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 12.
I20260709 12:23:28.004148 17760 raft_consensus.cc:3060] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:28.004357 17762 raft_consensus.cc:2468] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 18.
W20260709 12:23:27.983939 23970 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.888s	user 0.007s	sys 0.022s
I20260709 12:23:28.007759 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:28.015064 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:28.019210 18368 raft_consensus.cc:2468] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 13.
W20260709 12:23:27.998868 23931 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.485s	user 0.014s	sys 0.038s
I20260709 12:23:28.021260 17838 raft_consensus.cc:2468] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 17.
I20260709 12:23:28.013489 23997 raft_consensus.cc:515] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:28.004810 23969 raft_consensus.cc:3055] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 13 LEADER]: Stepping down as leader of term 13
W20260709 12:23:28.032061 24016 log.cc:927] Time spent T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.292s	user 0.001s	sys 0.000s
I20260709 12:23:28.032416 23957 raft_consensus.cc:493] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:28.033828 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:27.996402 23951 raft_consensus.cc:697] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 14 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:23:28.013155 23785 raft_consensus.cc:3055] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 9 LEADER]: Stepping down as leader of term 9
W20260709 12:23:28.057263 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 83329.3us
I20260709 12:23:28.000768 24031 raft_consensus.cc:493] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:28.060961 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:28.124974 17756 raft_consensus.cc:1240] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 16. Current term is 17. Ops: []
I20260709 12:23:28.094729 17762 raft_consensus.cc:3060] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Advancing to term 19
W20260709 12:23:28.067395 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 199560us
I20260709 12:23:27.983120 24019 raft_consensus.cc:515] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:28.137202 24019 leader_election.cc:290] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:28.096997 18333 leader_election.cc:400] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 18 election: Vote denied by peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837) with higher term. Message: Invalid argument: T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 18. Current term is 19.
W20260709 12:23:28.128211 24029 log.cc:927] Time spent T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.105s	user 0.000s	sys 0.001s
I20260709 12:23:28.140089 18333 leader_election.cc:403] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 18 election: Cancelling election due to peer responding with higher term
I20260709 12:23:28.146140 23700 leader_election.cc:290] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 19 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:28.021026 24032 raft_consensus.cc:2804] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:23:28.146919 24032 raft_consensus.cc:493] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:23:28.147287 24032 raft_consensus.cc:3060] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:28.020076 23838 raft_consensus.cc:697] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 16 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:23:28.148643 23838 consensus_queue.cc:237] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:28.153398 23997 leader_election.cc:290] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:28.160758 23951 consensus_queue.cc:237] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 12.9, Last appended by leader: 9, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:28.064949 23992 raft_consensus.cc:493] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:28.167371 17760 raft_consensus.cc:2468] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 14.
I20260709 12:23:28.168597 23992 raft_consensus.cc:515] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:23:28.172806 23929 consensus_meta.cc:287] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.696s	user 0.011s	sys 0.002s
I20260709 12:23:28.173429 23929 raft_consensus.cc:515] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:28.096261 23785 raft_consensus.cc:740] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:23:28.180560 23785 consensus_queue.cc:260] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:28.129793 17729 leader_election.cc:304] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:28.122527 23739 raft_consensus.cc:493] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:28.186856 23739 raft_consensus.cc:515] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:28.188710 23997 raft_consensus.cc:3060] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:28.096089 24022 consensus_queue.cc:1059] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 15.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.094s
I20260709 12:23:28.189747 23739 leader_election.cc:290] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:23:28.193930 23925 consensus_meta.cc:287] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.699s	user 0.011s	sys 0.002s
I20260709 12:23:28.194628 23925 raft_consensus.cc:515] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:28.197582 23998 consensus_meta.cc:287] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.695s	user 0.006s	sys 0.006s
I20260709 12:23:28.198276 23998 raft_consensus.cc:515] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:28.198522 23904 raft_consensus.cc:515] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:28.200052 18371 raft_consensus.cc:2393] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 15: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:23:28.201037 23904 leader_election.cc:290] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 15 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:23:28.134588 24024 log.cc:927] Time spent T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.183s	user 0.001s	sys 0.000s
I20260709 12:23:28.134857 23972 raft_consensus.cc:515] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:28.197455 23858 raft_consensus.cc:2749] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
W20260709 12:23:28.198577 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 68902.3us
W20260709 12:23:28.203366 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 106563us
W20260709 12:23:28.133387 24034 log.cc:927] Time spent T 00000000000000000000000000000000 P 2193af31cb894ec9b3d65d939a2a6851: Append to log took a long time: real 0.100s	user 0.002s	sys 0.000s
I20260709 12:23:28.207499 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:28.209909 18333 leader_election.cc:304] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:23:28.210492 17729 leader_election.cc:304] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:28.211634 17729 leader_election.cc:304] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:28.098258 23651 raft_consensus.cc:493] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:28.102089 23905 raft_consensus.cc:493] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:28.058915 23969 raft_consensus.cc:740] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:23:28.217988 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:28.218743 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:28.218384 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:23:28.216554 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 265590us
W20260709 12:23:28.237881 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 277165us
I20260709 12:23:28.216838 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:28.208853 23947 raft_consensus.cc:3060] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:28.253786 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:28.193822 23796 raft_consensus.cc:515] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:28.254033 17803 leader_election.cc:400] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 16 election: Vote denied by peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577) with higher term. Message: Invalid argument: T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 16. Current term is 17.
I20260709 12:23:28.270298 17803 leader_election.cc:403] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 16 election: Cancelling election due to peer responding with higher term
I20260709 12:23:28.259822 17831 raft_consensus.cc:3060] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:28.221117 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:28.274008 17757 raft_consensus.cc:2393] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 18: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:23:28.259684 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:28.256995 17727 leader_election.cc:304] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:28.275224 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:28.275705 17761 raft_consensus.cc:2468] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 16.
I20260709 12:23:28.260865 17755 raft_consensus.cc:1240] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 13. Current term is 14. Ops: []
I20260709 12:23:28.264444 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 18 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:28.267704 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:28.207314 23943 leader_election.cc:290] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:28.272327 17803 leader_election.cc:304] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:28.269188 17728 leader_election.cc:304] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:28.279978 17758 raft_consensus.cc:1275] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 14 index: 12. Preceding OpId from leader: term: 18 index: 14. (index mismatch)
I20260709 12:23:28.218387 23972 leader_election.cc:290] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 16 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:28.220395 24019 raft_consensus.cc:2749] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:23:28.207723 23957 raft_consensus.cc:515] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:28.197702 23997 raft_consensus.cc:2749] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:23:28.335117 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:28.315287 17763 raft_consensus.cc:1240] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 13. Current term is 14. Ops: [13.11-13.11]
I20260709 12:23:28.335940 18366 raft_consensus.cc:2468] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 12.
I20260709 12:23:28.259233 23905 raft_consensus.cc:515] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:28.269937 23855 raft_consensus.cc:2804] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:23:28.129724 24031 raft_consensus.cc:515] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:28.321650 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
W20260709 12:23:28.308099 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 51178.7us
I20260709 12:23:28.219661 23863 raft_consensus.cc:2749] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:23:28.335237 23739 raft_consensus.cc:2804] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:23:28.325313 24038 raft_consensus.cc:2804] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:23:28.314309 17803 leader_election.cc:304] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:23:28.255211 23929 leader_election.cc:290] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 16 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:28.337353 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:28.275403 23901 raft_consensus.cc:493] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:28.376688 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:23:28.377668 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:28.322353 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:28.362524 17763 raft_consensus.cc:2468] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 18.
I20260709 12:23:28.346989 24037 raft_consensus.cc:2804] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:23:28.304824 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:28.350315 17729 leader_election.cc:304] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:23:28.269241 23804 raft_consensus.cc:493] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:23:28.275337 23650 raft_consensus.cc:2749] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Leader election lost for term 16. Reason: Vote denied by peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577) with higher term. Message: Invalid argument: T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 16. Current term is 17.
I20260709 12:23:28.355329 17831 raft_consensus.cc:2468] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 19.
I20260709 12:23:28.306450 23925 leader_election.cc:290] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 14 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:28.402439 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:28.315512 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:28.262253 23992 leader_election.cc:290] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:28.356848 17727 leader_election.cc:304] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:23:28.406839 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 50099.3us
I20260709 12:23:28.286664 23904 raft_consensus.cc:493] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:28.338438 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 19 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:23:28.392601 24013 consensus_meta.cc:287] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.699s	user 0.009s	sys 0.004s
I20260709 12:23:28.256276 23812 raft_consensus.cc:515] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:28.367096 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:28.358549 23739 raft_consensus.cc:493] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:23:28.286993 23858 raft_consensus.cc:493] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:28.309638 23796 leader_election.cc:290] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 18 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:23:28.354406 23927 consensus_meta.cc:287] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.624s	user 0.005s	sys 0.008s
I20260709 12:23:28.314735 17757 raft_consensus.cc:1240] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 16. Current term is 17. Ops: []
I20260709 12:23:28.392750 18371 raft_consensus.cc:2468] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 17.
I20260709 12:23:28.427871 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:28.399915 17762 raft_consensus.cc:2468] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 19.
W20260709 12:23:28.400507 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 50329us
I20260709 12:23:28.256652 23651 raft_consensus.cc:515] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:28.364287 23855 raft_consensus.cc:697] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 12 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:23:28.368886 24038 raft_consensus.cc:697] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:23:28.412325 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:28.412603 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:28.414776 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:28.416828 17801 leader_election.cc:304] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:28.393989 24037 raft_consensus.cc:493] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:23:28.452883 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 179867us
I20260709 12:23:28.405218 23997 raft_consensus.cc:2804] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:23:28.453236 17763 raft_consensus.cc:2468] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 17.
I20260709 12:23:28.428978 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:23:28.454967 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 240581us
I20260709 12:23:28.456349 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:28.434906 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:28.457036 17762 raft_consensus.cc:3060] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:28.440908 17761 raft_consensus.cc:3060] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:28.456162 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:28.461740 17757 raft_consensus.cc:3060] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:28.464277 18333 leader_election.cc:304] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:28.453236 18366 raft_consensus.cc:2468] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 16.
I20260709 12:23:28.467712 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:28.468683 18366 raft_consensus.cc:2468] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 16.
I20260709 12:23:28.470443 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:28.454586 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:28.358773 23969 consensus_queue.cc:260] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.12, Last appended by leader: 12, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:28.499909 18368 raft_consensus.cc:1240] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 17. Current term is 18. Ops: []
W20260709 12:23:28.500252 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 83549.5us
I20260709 12:23:28.500473 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:28.393512 23901 raft_consensus.cc:515] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:28.412218 23992 raft_consensus.cc:2804] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:23:28.402097 23804 raft_consensus.cc:515] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:28.423267 23739 raft_consensus.cc:3060] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:28.506132 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:28.511471 17762 raft_consensus.cc:2468] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 16.
I20260709 12:23:28.504976 18366 raft_consensus.cc:1240] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 14. Current term is 15. Ops: [14.11-14.11]
I20260709 12:23:28.472693 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
W20260709 12:23:28.464388 23948 consensus_meta.cc:287] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.670s	user 0.011s	sys 0.002s
I20260709 12:23:28.523442 17761 raft_consensus.cc:2468] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 12.
I20260709 12:23:28.515297 17757 raft_consensus.cc:2468] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 19.
W20260709 12:23:28.529659 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 65469.3us
I20260709 12:23:28.529868 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:28.534780 18333 leader_election.cc:304] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:23:28.536319 17763 raft_consensus.cc:2393] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 16: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:23:28.463233 24037 raft_consensus.cc:3060] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:28.542048 18333 leader_election.cc:304] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:23:28.415495 23904 raft_consensus.cc:515] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:28.463454 23997 raft_consensus.cc:493] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:28.411823 24039 raft_consensus.cc:2804] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:23:28.506147 23998 leader_election.cc:290] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 17 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:28.551591 18333 leader_election.cc:304] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:23:28.429893 23858 raft_consensus.cc:515] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:28.411986 24013 raft_consensus.cc:515] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:28.558869 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 96883us
I20260709 12:23:28.540747 24022 raft_consensus.cc:2749] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:23:28.536299 23740 raft_consensus.cc:493] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:23:28.464515 23927 raft_consensus.cc:515] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:28.567088 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 136637us
W20260709 12:23:28.574332 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 72809.4us
W20260709 12:23:28.576093 17759 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36572 (request call id 1054) took 1702 ms (client timeout 1665 ms). Trace:
I20260709 12:23:28.578179 17801 leader_election.cc:304] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:23:28.578200 17759 rpcz_store.cc:269] 0709 12:23:26.866075 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:26.867712 (+  1637us) service_pool.cc:224] Handling call
0709 12:23:28.568561 (+1700849us) spinlock_profiling.cc:243] Waited 135 ms on lock 0x7b54004150d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:23:28.568703 (+   142us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":310535424}
I20260709 12:23:28.549578 23997 raft_consensus.cc:3060] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:28.564746 17729 leader_election.cc:304] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
W20260709 12:23:28.584411 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 269982us
I20260709 12:23:28.560818 23969 raft_consensus.cc:3060] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:28.536152 23948 raft_consensus.cc:515] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:28.554252 23650 raft_consensus.cc:2749] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:23:28.548843 24042 raft_consensus.cc:2804] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:23:28.590086 17756 raft_consensus.cc:2393] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 14: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:23:28.566109 23796 raft_consensus.cc:2804] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:23:28.592813 17729 leader_election.cc:304] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:28.574427 23740 raft_consensus.cc:3060] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:28.616158 24052 consensus_queue.cc:1059] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.019s
W20260709 12:23:28.616657 24043 log.cc:927] Time spent T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.060s	user 0.000s	sys 0.000s
I20260709 12:23:28.617691 24022 raft_consensus.cc:3055] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:23:28.618103 24022 raft_consensus.cc:740] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:23:28.618979 24022 consensus_queue.cc:260] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.12, Last appended by leader: 12, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:28.620242 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:28.620617 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:28.621070 17835 raft_consensus.cc:2393] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 14: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:23:28.602448 23927 leader_election.cc:290] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 18 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:28.605268 24013 leader_election.cc:290] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 14 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:28.607537 24050 raft_consensus.cc:2749] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:23:28.546060 24038 consensus_queue.cc:237] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 15.13, Last appended by leader: 13, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:28.609217 23901 leader_election.cc:290] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:28.611433 23948 leader_election.cc:290] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 18 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:28.612695 23651 leader_election.cc:290] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:28.613097 23904 leader_election.cc:290] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:28.613919 23812 leader_election.cc:290] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 11 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:28.613575 23997 raft_consensus.cc:515] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:28.576229 23838 raft_consensus.cc:2804] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:23:28.594561 24042 raft_consensus.cc:493] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:28.621321 17832 raft_consensus.cc:2368] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for earlier term 11. Current term is 12.
I20260709 12:23:28.623654 17729 leader_election.cc:304] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:28.615571 23804 leader_election.cc:290] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:23:28.627245 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 96508us
I20260709 12:23:28.649147 17652 leader_election.cc:304] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:28.650125 17652 leader_election.cc:304] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:28.613144 24031 leader_election.cc:290] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:28.621227 23796 raft_consensus.cc:493] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:28.656252 17652 leader_election.cc:304] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:28.626144 23970 raft_consensus.cc:2749] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:23:28.659155 17652 leader_election.cc:304] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:28.662459 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:28.639853 23838 raft_consensus.cc:493] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
W20260709 12:23:28.670882 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 47374.6us
I20260709 12:23:28.572680 23855 consensus_queue.cc:237] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 10.9, Last appended by leader: 9, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:28.660306 24036 log.cc:927] Time spent T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.266s	user 0.000s	sys 0.001s
W20260709 12:23:28.657924 24040 log.cc:927] Time spent T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.263s	user 0.000s	sys 0.001s
I20260709 12:23:28.639238 24061 raft_consensus.cc:2749] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:23:28.673403 17729 leader_election.cc:304] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:23:28.651911 24042 raft_consensus.cc:3060] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:28.672042 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
W20260709 12:23:28.704285 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:28.651067 23992 raft_consensus.cc:697] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 13 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:23:28.659816 23700 raft_consensus.cc:2804] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:23:28.725984 17835 raft_consensus.cc:1275] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 12 index: 10. Preceding OpId from leader: term: 15 index: 12. (index mismatch)
W20260709 12:23:28.727641 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 54343.3us
I20260709 12:23:28.728605 17729 leader_election.cc:304] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:23:28.658895 23796 raft_consensus.cc:3060] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:28.731652 23906 raft_consensus.cc:2804] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:23:28.732142 23930 raft_consensus.cc:3055] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:23:28.674700 23838 raft_consensus.cc:3060] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:28.660445 23948 raft_consensus.cc:2804] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:23:28.660161 23928 raft_consensus.cc:2804] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:23:28.670522 23925 raft_consensus.cc:2804] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:23:28.670840 23947 raft_consensus.cc:2749] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Leader election lost for term 18. Reason: Vote denied by peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837) with higher term. Message: Invalid argument: T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 18. Current term is 19.
W20260709 12:23:28.718330 24007 log.cc:927] Time spent T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.051s	user 0.000s	sys 0.000s
I20260709 12:23:28.726089 23700 raft_consensus.cc:493] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:23:28.717994 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 157681us
I20260709 12:23:28.778064 23700 raft_consensus.cc:3060] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:28.778469 18333 leader_election.cc:304] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:28.732158 23906 raft_consensus.cc:493] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:28.733515 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:28.733608 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:28.674816 23812 raft_consensus.cc:2749] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
W20260709 12:23:28.742974 17655 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:28.749430 23785 raft_consensus.cc:3060] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:23:28.755788 24012 log.cc:927] Time spent T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 1.228s	user 0.003s	sys 0.005s
I20260709 12:23:28.669930 24051 consensus_queue.cc:1059] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 12.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.082s
W20260709 12:23:28.757509 24045 log.cc:927] Time spent T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.186s	user 0.000s	sys 0.001s
I20260709 12:23:28.757726 23997 leader_election.cc:290] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 13 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:28.759806 17682 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37396 (request call id 2612) took 2736 ms. Trace:
I20260709 12:23:28.760057 23948 raft_consensus.cc:697] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 14 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:23:28.893059 17682 rpcz_store.cc:276] 0709 12:23:26.023349 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:26.854460 (+831111us) service_pool.cc:224] Handling call
0709 12:23:26.854923 (+   463us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:27.068417 (+213494us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:28.759610 (+1691193us) spinlock_profiling.cc:243] Waited 203 ms on lock 0x7b54000c6054. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:23:28.759758 (+   148us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":467467008}
I20260709 12:23:28.763242 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:28.868247 23950 raft_consensus.cc:493] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:23:28.767102 17833 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35042 (request call id 2566) took 1434 ms. Trace:
I20260709 12:23:28.780196 23947 consensus_queue.cc:1048] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.107s
I20260709 12:23:28.807471 23700 raft_consensus.cc:515] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:28.768074 17652 leader_election.cc:304] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:23:28.892777 24006 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.312s	user 0.019s	sys 0.030s
I20260709 12:23:28.772213 23925 raft_consensus.cc:697] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 19 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:23:28.775435 23740 raft_consensus.cc:515] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:28.741076 24031 raft_consensus.cc:2804] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:23:28.776469 23992 consensus_queue.cc:237] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.9, Last appended by leader: 9, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:28.741407 23960 raft_consensus.cc:493] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:28.739780 23931 raft_consensus.cc:493] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:28.783080 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:23:28.785583 18333 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:28.850069 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:28.835139 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:28.925127 17684 raft_consensus.cc:2468] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 13.
I20260709 12:23:28.828517 17729 leader_election.cc:304] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:23:28.784099 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:28.925784 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:28.926102 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:28.926663 17684 raft_consensus.cc:3060] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:28.790688 18367 raft_consensus.cc:1240] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 12. Current term is 13. Ops: []
I20260709 12:23:28.871587 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:28.780057 23916 raft_consensus.cc:2749] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:23:28.904935 23804 raft_consensus.cc:493] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:23:28.876176 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:28.931329 17829 raft_consensus.cc:1240] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 18. Current term is 19. Ops: []
I20260709 12:23:28.877902 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 18 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:28.876220 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:28.801852 24021 raft_consensus.cc:3055] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:23:28.978798 24021 raft_consensus.cc:740] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:23:28.980193 23700 leader_election.cc:290] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 19 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:28.979830 24021 consensus_queue.cc:260] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.12, Last appended by leader: 12, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:28.980851 23740 leader_election.cc:290] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 19 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:28.981436 24021 raft_consensus.cc:3060] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Advancing to term 16
W20260709 12:23:28.982198 24025 log.cc:927] Time spent T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.096s	user 0.000s	sys 0.000s
I20260709 12:23:28.780575 23906 raft_consensus.cc:3060] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:28.849164 23915 consensus_queue.cc:1059] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.034s
I20260709 12:23:28.869449 23943 raft_consensus.cc:493] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:28.984372 23943 raft_consensus.cc:515] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:28.986037 23925 consensus_queue.cc:237] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.11, Last appended by leader: 11, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:28.919534 24031 raft_consensus.cc:493] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:28.989382 24031 raft_consensus.cc:3060] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:28.932118 23998 raft_consensus.cc:2804] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:23:28.992877 23998 raft_consensus.cc:493] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:28.981455 23931 raft_consensus.cc:515] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:28.993335 23998 raft_consensus.cc:3060] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:23:28.993741 24065 log.cc:927] Time spent T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.061s	user 0.001s	sys 0.000s
I20260709 12:23:28.995997 23913 raft_consensus.cc:493] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:28.996066 23804 raft_consensus.cc:515] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:28.996726 23913 raft_consensus.cc:515] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:28.905216 23950 raft_consensus.cc:515] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:28.999338 23913 leader_election.cc:290] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:28.912582 24057 raft_consensus.cc:2749] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:23:28.812925 23518 raft_consensus.cc:3060] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Advancing to term 12
W20260709 12:23:28.886870 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 299983us
I20260709 12:23:29.018291 17803 leader_election.cc:304] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:28.864042 23957 leader_election.cc:290] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:28.868687 23972 raft_consensus.cc:493] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:28.762046 23739 raft_consensus.cc:515] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:28.764421 18365 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59918 (request call id 1900) took 1536 ms (client timeout 1813 ms). Trace:
I20260709 12:23:28.873365 23956 raft_consensus.cc:493] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:28.781836 23901 consensus_queue.cc:1059] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 11.10, Next index: 11, Last known committed idx: 9, Time since last communication: 0.104s
W20260709 12:23:28.896723 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 318586us
I20260709 12:23:29.030802 17801 leader_election.cc:304] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:28.785677 23904 consensus_queue.cc:1059] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.101s
I20260709 12:23:28.900995 17833 rpcz_store.cc:276] 0709 12:23:27.332920 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:27.681742 (+348822us) service_pool.cc:224] Handling call
0709 12:23:27.803605 (+121863us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:27.803946 (+   341us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:27.803996 (+    50us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:27.837745 (+ 33749us) log.cc:844] Serialized 26 byte log entry
0709 12:23:28.052553 (+214808us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:23:28.052622 (+    69us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:28.053045 (+   423us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:28.765847 (+712802us) raft_consensus.cc:1685] finished
0709 12:23:28.765870 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:28.766810 (+   940us) spinlock_profiling.cc:243] Waited 121 ms on lock 0x7b54002ebb50. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a61f8ad 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:23:28.767036 (+   226us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":279423872,"thread_start_us":172521,"threads_started":1,"wal-append.queue_time_us":413042}
I20260709 12:23:28.738567 23930 raft_consensus.cc:740] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:23:28.903061 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:29.033389 17682 raft_consensus.cc:3060] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:28.769668 23928 raft_consensus.cc:697] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 16 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
W20260709 12:23:28.904316 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 136334us
I20260709 12:23:29.036664 17652 leader_election.cc:304] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:28.770521 23835 raft_consensus.cc:3055] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:23:28.904645 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:28.767114 24037 raft_consensus.cc:515] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:28.904874 18363 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59902 (request call id 1501) took 1666 ms. Trace:
I20260709 12:23:28.930603 23929 raft_consensus.cc:493] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:23:28.893443 23955 raft_consensus.cc:493] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:23:28.882266 23796 raft_consensus.cc:515] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:28.909060 24006 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.364s	user 0.019s	sys 0.030s
I20260709 12:23:28.853703 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:23:28.926756 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 98374.7us
I20260709 12:23:29.047710 17685 raft_consensus.cc:2368] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 18. Current term is 19.
I20260709 12:23:28.926589 18368 raft_consensus.cc:2468] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 14.
I20260709 12:23:29.044422 17652 leader_election.cc:304] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:23:28.872617 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:28.957353 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:29.050815 18366 raft_consensus.cc:3060] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Advancing to term 16
I20260709 12:23:28.978297 23960 raft_consensus.cc:515] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:28.981053 23948 consensus_queue.cc:237] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:28.877214 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:29.008975 17802 leader_election.cc:304] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:29.011301 17687 raft_consensus.cc:2393] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 18: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:23:29.013476 17683 raft_consensus.cc:3060] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Advancing to term 13
W20260709 12:23:28.779858 24032 consensus_meta.cc:287] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.632s	user 0.012s	sys 0.000s
I20260709 12:23:29.023613 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 17 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:28.987108 23943 leader_election.cc:290] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:23:29.027681 18365 rpcz_store.cc:269] 0709 12:23:27.227592 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:27.228496 (+   904us) service_pool.cc:224] Handling call
0709 12:23:28.764260 (+1535764us) spinlock_profiling.cc:243] Waited 110 ms on lock 0x7b54004687d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:23:28.764366 (+   106us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":254102784}
I20260709 12:23:29.037523 18374 raft_consensus.cc:3060] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:28.911450 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:29.061794 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 14 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:29.064254 23943 raft_consensus.cc:2804] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:23:29.064891 23943 raft_consensus.cc:697] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 15 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:23:29.065634 23928 consensus_queue.cc:237] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:28.995579 23931 leader_election.cc:290] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:29.037636 18363 rpcz_store.cc:276] 0709 12:23:27.238957 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:27.239623 (+   666us) service_pool.cc:224] Handling call
0709 12:23:27.262762 (+ 23139us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:27.263010 (+   248us) raft_consensus.cc:1534] Early marking committed up to index 15
0709 12:23:27.293186 (+ 30176us) log.cc:844] Serialized 14 byte log entry
0709 12:23:28.784610 (+1491424us) spinlock_profiling.cc:243] Waited 3.56 ms on lock 0x7b180041e900. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b47515e07 00007f7b503709a9 00007f7b4753269b 00007f7b4753093f 00007f7b47532c02 00007f7b4a1ba20b 00007f7b4a1bfd23 00007f7b4a1c023a 00007f7b4a62aa2a
0709 12:23:28.904502 (+119892us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:28.904560 (+    58us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:23:28.904610 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:28.904697 (+    87us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:28.904817 (+   120us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":28198,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":40,"raft.queue_time_us":8178,"raft.run_cpu_time_us":165,"raft.run_wall_time_us":6949,"spinlock_wait_cycles":8185600,"thread_start_us":115486,"threads_started":1,"wal-append.queue_time_us":1688910}
I20260709 12:23:29.068809 18365 raft_consensus.cc:1240] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 14. Current term is 15. Ops: []
I20260709 12:23:29.059463 17686 raft_consensus.cc:2376] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 17. Re-sending same reply.
I20260709 12:23:29.057343 23997 consensus_queue.cc:1059] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 17.13, Next index: 14, Last known committed idx: 10, Time since last communication: 0.011s
I20260709 12:23:29.072085 24069 raft_consensus.cc:2749] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
W20260709 12:23:29.073203 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 64365us
I20260709 12:23:29.074033 17802 leader_election.cc:304] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:29.043123 23835 raft_consensus.cc:740] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:23:29.075922 24069 raft_consensus.cc:2749] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:23:29.076409 23835 consensus_queue.cc:260] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.14, Last appended by leader: 14, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:29.077227 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 46546.6us
I20260709 12:23:29.077594 23835 raft_consensus.cc:3060] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:29.078091 17801 leader_election.cc:304] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:29.037699 23972 raft_consensus.cc:515] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:29.080072 24069 raft_consensus.cc:3060] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:29.081791 23972 leader_election.cc:290] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:29.082518 17764 raft_consensus.cc:1240] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 13. Current term is 14. Ops: []
I20260709 12:23:29.022336 23915 consensus_queue.cc:1243] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Peer 0da169ff82674dc1947055ce43de68a3 log is divergent from this leader: its last log entry 14.12 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 10
I20260709 12:23:29.055649 24046 raft_consensus.cc:2804] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:23:29.084163 24037 leader_election.cc:290] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 14 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:29.084607 24046 raft_consensus.cc:697] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 15 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:23:29.086800 24046 consensus_queue.cc:237] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.9, Last appended by leader: 9, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:29.087589 23796 leader_election.cc:290] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 17 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:29.090619 23960 leader_election.cc:290] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:29.065838 23943 consensus_queue.cc:237] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 13.9, Last appended by leader: 9, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:29.075354 23740 raft_consensus.cc:2804] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:23:29.094527 23740 raft_consensus.cc:493] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:23:29.094990 23740 raft_consensus.cc:3060] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:29.096573 18365 raft_consensus.cc:1240] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 16. Current term is 17. Ops: []
I20260709 12:23:29.100539 17684 raft_consensus.cc:2468] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 17.
W20260709 12:23:29.008708 24045 log.cc:927] Time spent T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.103s	user 0.000s	sys 0.001s
I20260709 12:23:29.056950 17729 leader_election.cc:304] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:29.041844 17682 raft_consensus.cc:2468] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 15.
I20260709 12:23:29.101318 17801 leader_election.cc:304] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:29.108168 17689 raft_consensus.cc:2468] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 16.
W20260709 12:23:29.100692 17802 leader_election.cc:336] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 18 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Timed out: RequestConsensusVote RPC to 127.16.29.65:33071 timed out after 1.720s (SENT)
I20260709 12:23:29.112020 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:29.104727 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:29.112238 17802 leader_election.cc:304] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:29.113708 17685 raft_consensus.cc:2468] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 17.
I20260709 12:23:29.114905 18374 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 14.
I20260709 12:23:29.112368 17801 leader_election.cc:304] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:29.117072 18368 raft_consensus.cc:1240] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 13. Current term is 14. Ops: []
I20260709 12:23:29.119081 18374 raft_consensus.cc:1240] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 17. Current term is 18. Ops: []
I20260709 12:23:29.116119 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:29.120230 17682 raft_consensus.cc:3060] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:23:29.122236 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 65358.9us
I20260709 12:23:29.123775 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:29.124079 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:29.124370 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:29.125130 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:29.126178 17762 raft_consensus.cc:1275] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 13 index: 10. Preceding OpId from leader: term: 16 index: 12. (index mismatch)
W20260709 12:23:29.127290 18362 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59886 (request call id 1672) took 2312 ms (client timeout 2202 ms). Trace:
W20260709 12:23:29.127709 18362 rpcz_store.cc:269] 0709 12:23:26.801450 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:26.900475 (+ 99025us) service_pool.cc:224] Handling call
0709 12:23:29.113260 (+2212785us) spinlock_profiling.cc:243] Waited 593 ms on lock 0x7b5400305f54. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:23:29.113362 (+   102us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1364628864}
I20260709 12:23:29.128201 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:29.132244 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:29.133106 17680 raft_consensus.cc:2468] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 12.
I20260709 12:23:29.146032 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:29.043260 23956 raft_consensus.cc:515] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:29.056989 23913 raft_consensus.cc:2749] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
W20260709 12:23:29.152208 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 107882us
I20260709 12:23:29.175093 17652 leader_election.cc:304] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:29.152159 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:29.122056 24056 raft_consensus.cc:2804] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:23:29.181538 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:23:29.181905 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 136127us
I20260709 12:23:29.183315 17683 raft_consensus.cc:2468] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 13.
I20260709 12:23:29.189851 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:29.092597 24042 raft_consensus.cc:515] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:29.186905 17652 leader_election.cc:304] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:29.206717 17682 raft_consensus.cc:2468] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 10.
I20260709 12:23:29.209484 23913 raft_consensus.cc:493] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:29.210318 23913 raft_consensus.cc:515] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:29.216185 23950 leader_election.cc:290] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:29.217471 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 12 candidate_status { last_received { term: 11 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:29.055294 23929 raft_consensus.cc:515] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:29.227321 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 13 candidate_status { last_received { term: 10 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:29.145334 23970 raft_consensus.cc:3055] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:23:29.055483 23955 raft_consensus.cc:515] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:29.105762 23911 consensus_queue.cc:1059] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 12.10, Next index: 11, Last known committed idx: 9, Time since last communication: 0.069s
I20260709 12:23:29.218547 24059 raft_consensus.cc:3055] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:23:29.197173 17654 leader_election.cc:304] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
W20260709 12:23:29.221386 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 120727us
W20260709 12:23:29.246682 17802 leader_election.cc:336] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 17 election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Timed out: RequestConsensusVote RPC to 127.16.29.65:33071 timed out after 1.558s (SENT)
I20260709 12:23:29.247890 18361 raft_consensus.cc:2393] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 13: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:23:29.223867 17652 leader_election.cc:304] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:29.147984 23997 raft_consensus.cc:2749] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:23:29.153591 23651 raft_consensus.cc:2749] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:23:29.242177 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 14 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:29.192754 23992 raft_consensus.cc:2804] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:23:29.248225 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:29.110066 24032 raft_consensus.cc:515] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:29.194389 24077 raft_consensus.cc:2749] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 12:23:29.258039 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:29.265059 17685 raft_consensus.cc:3060] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:29.201202 24056 raft_consensus.cc:493] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:23:29.217401 23901 raft_consensus.cc:3055] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:23:29.236318 23931 raft_consensus.cc:2804] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:23:29.231321 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:29.186889 24037 consensus_queue.cc:1059] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 14.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.069s
I20260709 12:23:29.245802 24059 raft_consensus.cc:740] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 12 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:23:29.274844 23913 leader_election.cc:290] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:29.277705 24042 leader_election.cc:290] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 13 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:29.278728 24078 raft_consensus.cc:2749] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:23:29.285983 23740 raft_consensus.cc:515] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:29.287187 23905 leader_election.cc:290] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:29.232272 23970 raft_consensus.cc:740] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 17 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:23:29.288586 17727 leader_election.cc:304] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
W20260709 12:23:29.403488 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 114998us
I20260709 12:23:29.404374 17727 leader_election.cc:400] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 11 election: Vote denied by peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837) with higher term. Message: Invalid argument: T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for earlier term 11. Current term is 12.
I20260709 12:23:29.404690 17727 leader_election.cc:403] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 11 election: Cancelling election due to peer responding with higher term
I20260709 12:23:29.348127 24037 raft_consensus.cc:3055] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:23:29.409996 24037 raft_consensus.cc:740] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:23:29.410784 24037 consensus_queue.cc:260] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:29.273598 23929 leader_election.cc:290] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:23:29.348537 24073 log.cc:927] Time spent T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.101s	user 0.000s	sys 0.001s
I20260709 12:23:29.296327 23901 raft_consensus.cc:740] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:23:29.412789 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 15 candidate_status { last_received { term: 14 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:29.412938 23955 leader_election.cc:290] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:29.413900 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:29.413614 23901 consensus_queue.cc:260] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.12, Last appended by leader: 12, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:29.414598 23901 raft_consensus.cc:3060] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:29.414932 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:29.415628 18364 raft_consensus.cc:2393] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 18: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:23:29.350641 17760 raft_consensus.cc:1240] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 12. Current term is 13. Ops: []
I20260709 12:23:29.351970 24032 leader_election.cc:290] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 13 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:23:29.339264 17728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:29.253439 23976 raft_consensus.cc:3055] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:23:29.418058 23976 raft_consensus.cc:740] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:23:29.418996 23976 consensus_queue.cc:260] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:29.423657 23906 raft_consensus.cc:515] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:29.369827 17680 raft_consensus.cc:1240] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 16. Current term is 17. Ops: [16.10-16.10]
I20260709 12:23:29.295447 18366 raft_consensus.cc:2468] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 16.
I20260709 12:23:29.425277 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:29.425989 23906 leader_election.cc:290] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 17 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:29.426970 24059 consensus_queue.cc:260] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.10, Last appended by leader: 10, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:29.385205 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:29.266510 23992 raft_consensus.cc:697] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 19 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:23:29.428339 24059 raft_consensus.cc:3060] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:29.428882 17688 raft_consensus.cc:2410] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 13 because replica has last-logged OpId of term: 11 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 12.
I20260709 12:23:29.429100 23992 consensus_queue.cc:237] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.13, Last appended by leader: 13, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:29.297886 24056 raft_consensus.cc:3060] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:29.432511 24047 raft_consensus.cc:493] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:29.433132 24047 raft_consensus.cc:515] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:29.394505 17764 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36572 (request call id 1072) took 1060 ms. Trace:
I20260709 12:23:29.434870 17764 rpcz_store.cc:276] 0709 12:23:28.333944 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:28.334544 (+   600us) service_pool.cc:224] Handling call
0709 12:23:28.334972 (+   428us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:29.083307 (+748335us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:29.394176 (+310869us) spinlock_profiling.cc:243] Waited 747 ms on lock 0x7b540009f1d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:23:29.394423 (+   247us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1718763264}
I20260709 12:23:29.400462 17803 leader_election.cc:304] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:23:29.301501 24077 raft_consensus.cc:2749] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:23:29.436805 18362 raft_consensus.cc:3060] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [term 14 LEARNER]: Advancing to term 17
I20260709 12:23:29.405500 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 16 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:29.408509 24069 raft_consensus.cc:2749] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
W20260709 12:23:29.408795 23838 consensus_meta.cc:287] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.663s	user 0.005s	sys 0.006s
I20260709 12:23:29.408586 23970 consensus_queue.cc:260] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.13, Last appended by leader: 13, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:29.409608 24028 raft_consensus.cc:2749] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 12:23:29.363164 23931 raft_consensus.cc:493] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:23:29.422245 23956 leader_election.cc:290] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:29.365315 17685 raft_consensus.cc:2468] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 17.
I20260709 12:23:29.444746 18374 raft_consensus.cc:2468] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 14.
W20260709 12:23:29.503103 18334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:29.504987 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 104652us
I20260709 12:23:29.527066 17685 raft_consensus.cc:1240] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 13. Current term is 14. Ops: []
I20260709 12:23:29.530902 17680 raft_consensus.cc:1240] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 20. Current term is 21. Ops: []
I20260709 12:23:29.445730 23906 raft_consensus.cc:2749] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Leader election lost for term 11. Reason: Vote denied by peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837) with higher term. Message: Invalid argument: T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for earlier term 11. Current term is 12.
I20260709 12:23:29.443262 24052 raft_consensus.cc:493] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:29.452085 23976 raft_consensus.cc:3060] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:29.501121 23838 raft_consensus.cc:515] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:29.485129 23835 raft_consensus.cc:493] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:29.461865 24037 raft_consensus.cc:3060] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:29.459317 18366 raft_consensus.cc:2468] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 17.
I20260709 12:23:29.545285 17803 leader_election.cc:304] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:23:29.573207 17680 raft_consensus.cc:1240] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 16. Current term is 17. Ops: [16.12-16.12]
I20260709 12:23:29.504340 23931 raft_consensus.cc:3060] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:29.502466 23785 raft_consensus.cc:493] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:29.545782 23998 consensus_meta.cc:287] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.552s	user 0.012s	sys 0.001s
W20260709 12:23:29.575837 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 112610us
I20260709 12:23:29.544252 24080 raft_consensus.cc:2804] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:23:29.574218 17802 leader_election.cc:400] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 18 pre-election: Vote denied by peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071) with higher term. Message: Invalid argument: T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 18. Current term is 19.
I20260709 12:23:29.599548 17802 leader_election.cc:304] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:29.449944 24042 consensus_queue.cc:1048] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.114s
W20260709 12:23:29.585731 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 388609us
I20260709 12:23:29.598322 24080 raft_consensus.cc:493] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:29.602713 24080 raft_consensus.cc:3060] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:29.572957 23905 raft_consensus.cc:493] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:29.604296 23905 raft_consensus.cc:515] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:29.606966 23905 leader_election.cc:290] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:29.581899 23651 raft_consensus.cc:493] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:29.612514 23651 raft_consensus.cc:515] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:29.616073 23956 consensus_queue.cc:1048] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:23:29.618405 23651 leader_election.cc:290] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:29.622532 18362 raft_consensus.cc:1275] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [term 17 LEARNER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 14 index: 8. Preceding OpId from leader: term: 17 index: 9. (index mismatch)
I20260709 12:23:29.624354 23740 leader_election.cc:290] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 14 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:29.573076 23911 raft_consensus.cc:493] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:29.625571 23911 raft_consensus.cc:515] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:29.636983 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:29.565555 24028 raft_consensus.cc:493] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:29.577831 23913 raft_consensus.cc:493] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:29.638628 17802 leader_election.cc:304] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:23:29.581663 24067 raft_consensus.cc:2804] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:23:29.646611 17802 leader_election.cc:304] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da
W20260709 12:23:29.648485 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 46811.8us
I20260709 12:23:29.653317 17802 leader_election.cc:304] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:29.565878 24052 raft_consensus.cc:515] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:29.601307 23948 raft_consensus.cc:493] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:29.642937 24021 consensus_meta.cc:287] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.661s	user 0.009s	sys 0.002s
I20260709 12:23:29.642619 23740 raft_consensus.cc:2749] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:23:29.582578 23835 raft_consensus.cc:515] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:29.648156 23905 raft_consensus.cc:493] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:29.600159 23785 raft_consensus.cc:515] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:29.650077 23651 raft_consensus.cc:2804] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:23:29.650370 24067 raft_consensus.cc:493] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:29.636550 23998 raft_consensus.cc:515] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:29.677201 24069 raft_consensus.cc:2804] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Leader election won for term 10
W20260709 12:23:29.691776 24031 consensus_meta.cc:287] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.702s	user 0.013s	sys 0.001s
I20260709 12:23:29.648484 24028 raft_consensus.cc:515] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:29.708832 24026 raft_consensus.cc:3060] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:29.720435 23785 leader_election.cc:290] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:29.721385 23835 leader_election.cc:290] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:23:29.724054 17801 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.602s	user 0.013s	sys 0.027s
W20260709 12:23:29.735918 17801 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.614s	user 0.015s	sys 0.027s
I20260709 12:23:29.699016 23651 raft_consensus.cc:493] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:29.697925 23905 raft_consensus.cc:515] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:29.725206 23838 leader_election.cc:290] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 17 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:29.693955 17654 leader_election.cc:304] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:29.651484 23913 raft_consensus.cc:515] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:29.716470 24067 raft_consensus.cc:3060] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Advancing to term 16
W20260709 12:23:29.729197 24048 thread.cc:641] apply [worker] (thread pool) Time spent creating pthread: real 1.120s	user 0.031s	sys 0.039s
W20260709 12:23:29.736940 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 624605us
I20260709 12:23:29.712641 24082 raft_consensus.cc:2749] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
W20260709 12:23:29.725544 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 72327.1us
I20260709 12:23:29.780620 17802 leader_election.cc:304] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:29.741137 23651 raft_consensus.cc:3060] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:29.786891 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 12 candidate_status { last_received { term: 11 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:23:29.737416 23943 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.508s	user 0.009s	sys 0.040s
I20260709 12:23:29.715586 23997 consensus_queue.cc:1059] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 18.13, Next index: 14, Last known committed idx: 11, Time since last communication: 0.024s
W20260709 12:23:29.749801 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 526006us
I20260709 12:23:29.728682 24069 raft_consensus.cc:697] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 10 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
W20260709 12:23:29.748916 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 99036us
I20260709 12:23:29.774056 23956 raft_consensus.cc:493] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:29.689455 23948 raft_consensus.cc:515] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:29.805604 17832 raft_consensus.cc:2393] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 12: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:23:29.770691 24085 raft_consensus.cc:2749] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 12:23:29.791671 17830 raft_consensus.cc:1240] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 13. Current term is 14. Ops: []
I20260709 12:23:29.793025 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:29.779559 23740 raft_consensus.cc:493] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:29.788502 24061 raft_consensus.cc:2749] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 12:23:29.798079 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 14 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:29.816565 17834 raft_consensus.cc:1275] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 12 index: 9. Preceding OpId from leader: term: 14 index: 10. (index mismatch)
W20260709 12:23:29.799671 17801 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:29.796211 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:29.798823 17652 leader_election.cc:304] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:29.728833 24031 raft_consensus.cc:515] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:29.794202 17836 raft_consensus.cc:1275] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 16 index: 12. (index mismatch)
I20260709 12:23:29.778750 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:29.821457 17838 raft_consensus.cc:3060] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:23:29.773073 24048 thread.cc:608] apply [worker] (thread pool) Time spent starting thread: real 1.164s	user 0.032s	sys 0.039s
W20260709 12:23:29.801710 23928 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.513s	user 0.027s	sys 0.047s
I20260709 12:23:29.808002 24017 raft_consensus.cc:3055] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:23:29.809291 24082 raft_consensus.cc:493] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:29.836609 17652 leader_election.cc:304] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:29.839102 24052 leader_election.cc:290] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:29.849812 24082 raft_consensus.cc:515] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:29.854270 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 280186us
I20260709 12:23:29.855619 17837 raft_consensus.cc:1240] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 16. Current term is 17. Ops: [16.14-16.14]
W20260709 12:23:29.860994 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 290071us
I20260709 12:23:29.863345 17803 leader_election.cc:304] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:23:29.835711 23943 raft_consensus.cc:2749] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
W20260709 12:23:29.842820 23928 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.554s	user 0.028s	sys 0.048s
I20260709 12:23:29.809624 23956 raft_consensus.cc:515] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:29.848402 24080 raft_consensus.cc:515] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:29.869477 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:29.842077 24017 raft_consensus.cc:740] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 18 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
W20260709 12:23:29.873764 18333 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:29.874704 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 19 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:29.878284 17838 raft_consensus.cc:2468] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 18.
I20260709 12:23:29.878588 17829 raft_consensus.cc:2468] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 18.
I20260709 12:23:29.860349 23972 raft_consensus.cc:2749] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 12:23:29.820541 23740 raft_consensus.cc:515] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:29.880178 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:29.882867 17833 raft_consensus.cc:2468] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 15.
I20260709 12:23:29.864987 23911 leader_election.cc:290] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:29.883265 18333 leader_election.cc:304] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:23:29.896248 23901 raft_consensus.cc:2804] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:23:29.897404 23901 raft_consensus.cc:697] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 17 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:23:29.896479 24067 raft_consensus.cc:515] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:29.897634 24082 leader_election.cc:290] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:29.898690 23905 leader_election.cc:290] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:29.898481 23901 consensus_queue.cc:237] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.12, Last appended by leader: 12, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:29.899901 24067 leader_election.cc:290] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 16 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:29.864754 24046 consensus_queue.cc:1048] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.018s
I20260709 12:23:29.901834 18333 leader_election.cc:304] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:23:29.903307 24084 log.cc:927] Time spent T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.061s	user 0.000s	sys 0.001s
I20260709 12:23:29.903577 23838 raft_consensus.cc:2749] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:23:29.905674 23838 consensus_queue.cc:1059] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 13.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:23:29.907402 23835 raft_consensus.cc:3055] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:23:29.908296 23835 raft_consensus.cc:740] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:23:29.910430 23785 raft_consensus.cc:493] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:29.911435 23785 raft_consensus.cc:515] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:29.914855 23785 leader_election.cc:290] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:23:29.915138 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:29.916162 17728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:29.916360 18334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:29.885743 17756 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36572 (request call id 1069) took 1571 ms (client timeout 1736 ms). Trace:
I20260709 12:23:29.888998 23913 leader_election.cc:290] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:29.890626 24080 leader_election.cc:290] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 15 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:29.918490 18333 leader_election.cc:304] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:29.877274 24052 raft_consensus.cc:493] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:29.873227 23998 leader_election.cc:290] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 18 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:23:29.918087 17756 rpcz_store.cc:269] 0709 12:23:28.314752 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:28.314990 (+   238us) service_pool.cc:224] Handling call
0709 12:23:29.885510 (+1570520us) spinlock_profiling.cc:243] Waited 184 ms on lock 0x7b540019f5d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:23:29.885645 (+   135us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":422720768}
I20260709 12:23:29.926548 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:29.933694 24080 raft_consensus.cc:493] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:23:29.933754 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 19 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:29.934113 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:29.934470 24080 raft_consensus.cc:515] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:29.931249 24042 consensus_queue.cc:1048] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.005s
I20260709 12:23:29.936353 23956 leader_election.cc:290] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:29.936946 24080 leader_election.cc:290] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:29.937363 23948 leader_election.cc:290] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:29.934684 18364 raft_consensus.cc:2468] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 18.
I20260709 12:23:29.922179 23785 raft_consensus.cc:493] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:29.922541 24081 raft_consensus.cc:2804] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:23:29.937633 24042 raft_consensus.cc:493] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:29.871770 24069 consensus_queue.cc:237] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.9, Last appended by leader: 9, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:29.938898 24083 log.cc:927] Time spent T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.098s	user 0.001s	sys 0.001s
I20260709 12:23:29.951756 23785 raft_consensus.cc:515] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:29.952275 23931 raft_consensus.cc:515] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:29.954037 24031 leader_election.cc:290] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 17 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:29.955539 17689 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37402 (request call id 3007) took 1105 ms. Trace:
I20260709 12:23:29.956004 24028 leader_election.cc:290] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:29.964160 17687 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37402 (request call id 3008) took 1113 ms. Trace:
I20260709 12:23:29.985981 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 16 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:29.924222 24021 raft_consensus.cc:2749] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:23:29.983178 17803 leader_election.cc:304] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:23:29.965262 17682 raft_consensus.cc:1240] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 17. Current term is 18. Ops: [17.10-17.10]
I20260709 12:23:29.968545 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:29.942404 23835 consensus_queue.cc:260] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.11, Last appended by leader: 11, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:29.965502 24039 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.356s	user 0.021s	sys 0.033s
W20260709 12:23:30.024377 24039 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.431s	user 0.021s	sys 0.033s
I20260709 12:23:30.024677 24039 raft_consensus.cc:697] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:23:30.024881 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:30.025599 24039 consensus_queue.cc:237] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.10, Last appended by leader: 10, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:30.026117 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:30.028106 17756 raft_consensus.cc:1240] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 LEADER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 16. Current term is 17. Ops: [16.14-16.14]
I20260709 12:23:30.027143 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:29.986308 17687 rpcz_store.cc:276] 0709 12:23:28.850857 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:28.851002 (+   145us) service_pool.cc:224] Handling call
0709 12:23:29.963962 (+1112960us) spinlock_profiling.cc:243] Waited 77.4 ms on lock 0x7b5400426dd4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:23:29.964102 (+   140us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":177993984}
I20260709 12:23:29.988595 17689 rpcz_store.cc:276] 0709 12:23:28.850368 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:28.850555 (+   187us) service_pool.cc:224] Handling call
0709 12:23:29.955348 (+1104793us) spinlock_profiling.cc:243] Waited 42.3 ms on lock 0x7b54003458d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:23:29.955471 (+   123us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":97222400}
I20260709 12:23:29.932582 24052 raft_consensus.cc:515] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:30.014320 18333 leader_election.cc:304] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:30.018368 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 18 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:30.030869 18363 raft_consensus.cc:2410] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 18 because replica has last-logged OpId of term: 16 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 13.
I20260709 12:23:29.973896 24081 raft_consensus.cc:697] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 17 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:23:30.023904 23835 raft_consensus.cc:3060] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:29.986759 18374 raft_consensus.cc:3060] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Advancing to term 16
W20260709 12:23:29.986657 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 385784us
I20260709 12:23:30.032908 17758 raft_consensus.cc:1275] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 12 index: 9. Preceding OpId from leader: term: 14 index: 10. (index mismatch)
I20260709 12:23:30.034219 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:30.034480 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:30.040432 17763 raft_consensus.cc:2393] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 13: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:23:30.023690 23906 raft_consensus.cc:493] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:30.037770 17759 raft_consensus.cc:1240] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 16. Current term is 17. Ops: [16.12-16.12]
I20260709 12:23:30.041606 17762 raft_consensus.cc:2393] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 17: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:23:30.020049 23905 raft_consensus.cc:2804] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:23:30.052053 17763 raft_consensus.cc:1240] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 13. Current term is 14. Ops: [13.11-13.11]
W20260709 12:23:30.046679 17727 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:30.051332 18364 raft_consensus.cc:2393] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 17: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:23:30.018509 23737 raft_consensus.cc:493] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:29.939986 24017 consensus_queue.cc:260] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.13, Last appended by leader: 13, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:30.055475 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:30.055487 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:30.061156 18364 raft_consensus.cc:3060] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:30.033078 24021 raft_consensus.cc:2749] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 12:23:30.104509 24081 consensus_queue.cc:237] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 16.14, Last appended by leader: 14, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:30.108815 23740 leader_election.cc:290] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:29.981033 24042 raft_consensus.cc:515] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:30.057246 23905 raft_consensus.cc:493] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:30.118597 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 15 candidate_status { last_received { term: 14 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:30.039700 23953 consensus_queue.cc:1059] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.033s
W20260709 12:23:30.133019 24059 consensus_meta.cc:287] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.704s	user 0.006s	sys 0.006s
I20260709 12:23:30.135784 18363 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 12.
I20260709 12:23:30.128084 23905 raft_consensus.cc:3060] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:30.067378 23737 raft_consensus.cc:515] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:23:30.142647 17802 leader_election.cc:336] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 14 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Timed out: RequestConsensusVote RPC to 127.16.29.65:33071 timed out after 1.707s (SENT)
I20260709 12:23:30.056211 23906 raft_consensus.cc:515] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:23:30.143414 17802 leader_election.cc:304] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:30.148025 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:30.149767 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:30.137912 24021 raft_consensus.cc:3055] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:23:30.162287 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:30.164288 17758 raft_consensus.cc:2393] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 17: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
W20260709 12:23:30.167865 18334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:30.154142 23852 raft_consensus.cc:2749] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:23:30.158654 23913 raft_consensus.cc:493] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:30.172029 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:30.157665 17802 leader_election.cc:304] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:30.173317 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:30.172919 18374 raft_consensus.cc:2468] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 16.
I20260709 12:23:30.173749 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:30.177487 17758 raft_consensus.cc:2468] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 18.
I20260709 12:23:30.175403 18364 raft_consensus.cc:2468] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 18.
I20260709 12:23:30.178997 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:30.179718 17762 raft_consensus.cc:2393] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 14: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:23:30.182426 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:30.183310 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:30.178083 17803 leader_election.cc:304] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:23:30.157651 24021 raft_consensus.cc:740] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:23:30.184489 17802 leader_election.cc:304] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:30.190289 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:30.190927 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:30.191360 17833 raft_consensus.cc:2393] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 14: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:23:30.192183 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 19 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:30.192931 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:30.194032 17833 raft_consensus.cc:2468] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 13.
I20260709 12:23:30.207823 18374 raft_consensus.cc:1275] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 16 index: 14. Preceding OpId from leader: term: 17 index: 14. (term mismatch)
I20260709 12:23:30.210331 18374 pending_rounds.cc:85] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da: Aborting all ops after (but not including) 13
I20260709 12:23:30.171986 23953 raft_consensus.cc:493] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:30.208455 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:30.172130 24067 raft_consensus.cc:493] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
W20260709 12:23:30.202776 24056 consensus_meta.cc:287] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.771s	user 0.008s	sys 0.006s
I20260709 12:23:30.212314 18374 pending_rounds.cc:107] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da: Aborting uncommitted NO_OP operation due to leader change: 16.14
I20260709 12:23:30.238302 18374 raft_consensus.cc:2889] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:23:30.221804 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:30.189738 17729 leader_election.cc:304] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:23:30.240188 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 50570.1us
I20260709 12:23:30.240581 24053 raft_consensus.cc:2804] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:23:30.241379 24053 raft_consensus.cc:697] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:23:30.241750 23998 raft_consensus.cc:493] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:30.242309 23998 raft_consensus.cc:515] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:30.242539 24053 consensus_queue.cc:237] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 16.14, Last appended by leader: 14, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:30.205089 23852 raft_consensus.cc:2749] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:23:30.207672 24059 raft_consensus.cc:2804] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:23:30.245333 24080 raft_consensus.cc:2749] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:23:30.186281 23913 raft_consensus.cc:515] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:30.277282 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 98090.9us
I20260709 12:23:30.281862 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:30.282644 18374 raft_consensus.cc:1275] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 11 index: 9. Preceding OpId from leader: term: 13 index: 10. (index mismatch)
I20260709 12:23:30.282660 17758 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 12.
I20260709 12:23:30.266260 23785 leader_election.cc:290] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:30.248169 24067 raft_consensus.cc:515] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:30.288239 17835 raft_consensus.cc:2393] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 17: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:23:30.246958 24056 raft_consensus.cc:515] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:30.244772 23953 raft_consensus.cc:515] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:30.290026 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:30.291697 17762 raft_consensus.cc:2468] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 14.
I20260709 12:23:30.291620 23785 consensus_queue.cc:1048] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.004s
I20260709 12:23:30.297377 23785 consensus_queue.cc:1059] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 11.10, Next index: 11, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 12:23:30.291841 23852 raft_consensus.cc:493] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:23:30.298707 17685 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44898 (request call id 1167) took 1395 ms. Trace:
I20260709 12:23:30.298919 23852 raft_consensus.cc:515] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:30.299181 17685 rpcz_store.cc:276] 0709 12:23:28.903701 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:29.528169 (+624468us) service_pool.cc:224] Handling call
0709 12:23:29.560212 (+ 32043us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:29.560645 (+   433us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 12:23:29.873386 (+312741us) log.cc:844] Serialized 14 byte log entry
0709 12:23:30.172817 (+299431us) log.cc:844] Serialized 14 byte log entry
0709 12:23:30.173298 (+   481us) log.cc:844] Serialized 14 byte log entry
0709 12:23:30.173581 (+   283us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:30.174186 (+   605us) log.cc:844] Serialized 26 byte log entry
0709 12:23:30.174363 (+   177us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:23:30.174415 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:30.174513 (+    98us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:30.297848 (+123335us) spinlock_profiling.cc:243] Waited 5.73 ms on lock 0x7b1800314920. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b475333f3 00007f7b475357c1 00007f7b47535779 00007f7b47535709 00007f7b475356d1 00007f7b475349cd 0000000000388634 0000000000388479 00007f7b4751683a
0709 12:23:30.298290 (+   442us) raft_consensus.cc:1685] finished
0709 12:23:30.298310 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:30.298631 (+   321us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":627198,"prepare.run_cpu_time_us":85,"prepare.run_wall_time_us":90,"raft.queue_time_us":348,"raft.run_cpu_time_us":270,"raft.run_wall_time_us":14263,"spinlock_wait_cycles":32883712,"thread_start_us":577850,"threads_started":2,"wal-append.queue_time_us":402063}
I20260709 12:23:30.301832 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:30.302464 23852 leader_election.cc:290] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:30.309944 18368 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59886 (request call id 1695) took 1479 ms. Trace:
I20260709 12:23:30.310634 18368 rpcz_store.cc:276] 0709 12:23:28.830936 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:29.048594 (+217658us) service_pool.cc:224] Handling call
0709 12:23:29.116854 (+ 68260us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:29.117604 (+   750us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:30.309715 (+1192111us) spinlock_profiling.cc:243] Waited 67.9 ms on lock 0x7b54006e23d0. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a61f8ad 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:23:30.309873 (+   158us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":156124928}
I20260709 12:23:30.316200 23913 leader_election.cc:290] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:30.328680 24067 leader_election.cc:290] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:30.330125 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:30.334208 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:30.263068 24021 consensus_queue.cc:260] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.12, Last appended by leader: 12, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:30.335702 24021 raft_consensus.cc:3060] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:30.336702 17760 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38786 (request call id 3282) took 2119 ms. Trace:
I20260709 12:23:30.337075 17760 rpcz_store.cc:276] 0709 12:23:28.216974 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:28.221348 (+  4374us) service_pool.cc:224] Handling call
0709 12:23:28.221897 (+   549us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:29.416427 (+1194530us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:30.336461 (+920034us) spinlock_profiling.cc:243] Waited 1.13 s on lock 0x7b54000fdcd4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:23:30.336643 (+   182us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2595309824}
I20260709 12:23:30.339071 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:30.341197 24056 leader_election.cc:290] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 18 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:30.341821 17829 raft_consensus.cc:3060] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:30.343098 17682 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37396 (request call id 2627) took 1477 ms. Trace:
I20260709 12:23:30.343461 17682 rpcz_store.cc:276] 0709 12:23:28.865361 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:29.373025 (+507664us) service_pool.cc:224] Handling call
0709 12:23:29.964959 (+591934us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:30.009025 (+ 44066us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:30.342930 (+333905us) spinlock_profiling.cc:243] Waited 591 ms on lock 0x7b5400426dd0. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a61f8ad 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:23:30.343043 (+   113us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1360122752}
I20260709 12:23:30.344588 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:30.345098 17682 raft_consensus.cc:3060] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:30.349741 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:30.350525 17837 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 12.
W20260709 12:23:30.350773 24037 consensus_meta.cc:287] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.762s	user 0.011s	sys 0.000s
I20260709 12:23:30.354939 18371 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 1912) took 1525 ms. Trace:
I20260709 12:23:30.355171 18367 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59902 (request call id 1504) took 2466 ms. Trace:
I20260709 12:23:30.356071 24042 leader_election.cc:290] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:30.355404 18371 rpcz_store.cc:276] 0709 12:23:28.829836 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:29.003766 (+173930us) service_pool.cc:224] Handling call
0709 12:23:29.049709 (+ 45943us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:29.050021 (+   312us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:29.050061 (+    40us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:29.050132 (+    71us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:23:29.050193 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:29.050277 (+    84us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:30.354751 (+1304474us) spinlock_profiling.cc:243] Waited 42.2 ms on lock 0x7b54007e7cd0. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a61f8ad 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:23:30.354875 (+   124us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":97000192}
W20260709 12:23:30.357118 18361 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59886 (request call id 1696) took 1525 ms (client timeout 1877 ms). Trace:
I20260709 12:23:30.357273 23737 leader_election.cc:290] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:30.355839 18367 rpcz_store.cc:276] 0709 12:23:27.888948 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:27.889062 (+   114us) service_pool.cc:224] Handling call
0709 12:23:27.889333 (+   271us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:28.931036 (+1041703us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:30.354798 (+1423762us) spinlock_profiling.cc:243] Waited 901 ms on lock 0x7b54006af9d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:23:30.355062 (+   264us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2072447744}
W20260709 12:23:30.361979 24026 consensus_meta.cc:287] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.633s	user 0.011s	sys 0.000s
I20260709 12:23:30.364228 23953 leader_election.cc:290] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:30.364500 17687 raft_consensus.cc:2393] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 19: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
W20260709 12:23:30.364759 24072 log.cc:927] Time spent T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 1.387s	user 0.008s	sys 0.007s
I20260709 12:23:30.365515 23998 leader_election.cc:290] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:30.335037 17681 raft_consensus.cc:2393] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 16: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:23:30.339752 17832 raft_consensus.cc:2468] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 14.
I20260709 12:23:30.380748 17832 raft_consensus.cc:1240] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 17. Current term is 18. Ops: []
W20260709 12:23:30.383966 17681 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37396 (request call id 2629) took 1518 ms (client timeout 1630 ms). Trace:
W20260709 12:23:30.384452 17681 rpcz_store.cc:269] 0709 12:23:28.865974 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:29.054663 (+188689us) service_pool.cc:224] Handling call
0709 12:23:30.334051 (+1279388us) spinlock_profiling.cc:243] Waited 40 ms on lock 0x7b5800002a98. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b474fbb71 00007f7b474fba22 00007f7b504148d9 00007f7b4f56eee0 00007f7b4f6380ab 00007f7b4f638131 00007f7b4f5e60a0 00007f7b4f5e5984 00007f7b48b606d4 00007f7b48b60662
0709 12:23:30.383857 (+ 49806us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":91962240}
I20260709 12:23:30.385946 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:23:30.386478 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 580027us
I20260709 12:23:30.391968 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:30.351596 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:30.403734 17802 leader_election.cc:304] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:30.406721 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:23:30.357630 18361 rpcz_store.cc:269] 0709 12:23:28.831412 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:28.894933 (+ 63521us) service_pool.cc:224] Handling call
0709 12:23:30.356929 (+1461996us) spinlock_profiling.cc:243] Waited 185 ms on lock 0x7b54005730d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:23:30.357057 (+   128us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":425139968}
I20260709 12:23:30.412026 18374 raft_consensus.cc:1275] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 17 index: 13. Preceding OpId from leader: term: 19 index: 14. (index mismatch)
I20260709 12:23:30.409600 18334 leader_election.cc:304] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:30.414666 17829 raft_consensus.cc:2468] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 19.
I20260709 12:23:30.418859 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:30.419404 17802 leader_election.cc:304] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
W20260709 12:23:30.419710 17684 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37396 (request call id 2635) took 1531 ms (client timeout 1877 ms). Trace:
W20260709 12:23:30.420158 17684 rpcz_store.cc:269] 0709 12:23:28.890083 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:29.101496 (+211413us) service_pool.cc:224] Handling call
0709 12:23:30.418695 (+1317199us) spinlock_profiling.cc:243] Waited 1.19 ms on lock 0x7b5800002a98. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b474fbb71 00007f7b474fba22 00007f7b504148d9 00007f7b4f56eee0 00007f7b4f6380ab 00007f7b4f638131 00007f7b4f5e60a0 00007f7b4f5e5984 00007f7b48b606d4 00007f7b48b60662
0709 12:23:30.419643 (+   948us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2729216}
I20260709 12:23:30.421032 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:30.421373 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:30.421514 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:30.421768 17681 raft_consensus.cc:2393] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 17: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:23:30.422109 17684 raft_consensus.cc:2368] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 18. Current term is 19.
I20260709 12:23:30.421999 17689 raft_consensus.cc:2468] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 17.
W20260709 12:23:30.335031 23651 consensus_meta.cc:287] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.531s	user 0.009s	sys 0.000s
I20260709 12:23:30.430392 18334 leader_election.cc:304] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:30.341197 24047 leader_election.cc:290] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:30.445055 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:30.429845 17728 leader_election.cc:304] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:30.447425 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 18 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:30.433509 17682 raft_consensus.cc:2468] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 14.
I20260709 12:23:30.450047 17686 raft_consensus.cc:2393] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 14: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:23:30.452054 17685 raft_consensus.cc:2393] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 18: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:23:30.418533 24056 raft_consensus.cc:2749] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
W20260709 12:23:30.468849 17686 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37402 (request call id 3016) took 1473 ms (client timeout 1707 ms). Trace:
I20260709 12:23:30.432025 24098 raft_consensus.cc:2749] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
W20260709 12:23:30.471012 17686 rpcz_store.cc:269] 0709 12:23:28.995136 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:29.071034 (+ 75898us) service_pool.cc:224] Handling call
0709 12:23:30.444820 (+1373786us) spinlock_profiling.cc:243] Waited 34.4 ms on lock 0x7b5800002a98. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b474fbb71 00007f7b474fba22 00007f7b504148d9 00007f7b4f56eee0 00007f7b4f6380ab 00007f7b4f638131 00007f7b4f5e60a0 00007f7b4f5e5984 00007f7b48b606d4 00007f7b48b60662
0709 12:23:30.468758 (+ 23938us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":79038208}
I20260709 12:23:30.473317 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:30.474108 24052 leader_election.cc:290] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:30.474639 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:30.475271 23998 raft_consensus.cc:2749] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:23:30.443641 23651 raft_consensus.cc:515] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:30.411180 23700 consensus_queue.cc:1059] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 17.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.057s
I20260709 12:23:30.451697 23953 raft_consensus.cc:2749] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:23:30.413096 24026 consensus_queue.cc:1059] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 16.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.056s
W20260709 12:23:30.477778 17730 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:30.493238 17728 leader_election.cc:304] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:30.479734 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:30.494854 17680 raft_consensus.cc:2376] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 16. Re-sending same reply.
I20260709 12:23:30.480906 24098 consensus_queue.cc:1059] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 12.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.014s
I20260709 12:23:30.496038 24098 raft_consensus.cc:3055] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:23:30.496595 24098 raft_consensus.cc:740] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 12 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:23:30.475857 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:30.486359 17830 raft_consensus.cc:2393] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 18: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:23:30.495991 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
W20260709 12:23:30.488607 17688 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37402 (request call id 2996) took 4327 ms (client timeout 2310 ms). Trace:
W20260709 12:23:30.501449 17688 rpcz_store.cc:269] 0709 12:23:26.160678 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:26.541527 (+380849us) service_pool.cc:224] Handling call
0709 12:23:30.488198 (+3946671us) spinlock_profiling.cc:243] Waited 2.89 s on lock 0x7b54004254d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:23:30.488505 (+   307us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":6637640448}
I20260709 12:23:30.506852 17728 leader_election.cc:304] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
W20260709 12:23:30.509482 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:30.510808 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:30.513458 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 19 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:30.515921 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 19 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:30.517256 17760 raft_consensus.cc:2410] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for term 19 because replica has last-logged OpId of term: 17 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 9.
I20260709 12:23:30.520268 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:30.521188 17763 raft_consensus.cc:2468] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 17.
I20260709 12:23:30.523689 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:30.524127 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:30.525378 17761 raft_consensus.cc:2393] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 17: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:23:30.527465 17763 raft_consensus.cc:1275] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 14 index: 12. (index mismatch)
I20260709 12:23:30.530664 17680 raft_consensus.cc:2393] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 16: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:23:30.500329 17681 raft_consensus.cc:2376] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 15. Re-sending same reply.
I20260709 12:23:30.531337 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:30.537891 17686 raft_consensus.cc:2376] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 17. Re-sending same reply.
W20260709 12:23:30.539166 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 250789us
I20260709 12:23:30.540529 17762 raft_consensus.cc:1240] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 16. Current term is 17. Ops: []
I20260709 12:23:30.568111 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:30.546782 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:30.568564 18333 leader_election.cc:304] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:30.569221 17756 raft_consensus.cc:2393] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 14: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:23:30.483784 24111 raft_consensus.cc:2749] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:23:30.530774 24042 raft_consensus.cc:493] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:30.570312 18333 leader_election.cc:304] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:30.540616 24006 raft_consensus.cc:2804] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:23:30.488111 23700 raft_consensus.cc:3055] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:23:30.553982 17728 leader_election.cc:304] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:23:30.481102 23998 raft_consensus.cc:493] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:30.584528 17728 leader_election.cc:304] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:30.485584 23858 leader_election.cc:290] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:30.588653 18362 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59902 (request call id 1515) took 1184 ms. Trace:
I20260709 12:23:30.497646 23930 consensus_queue.cc:260] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 13.11, Last appended by leader: 11, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:30.467540 18366 raft_consensus.cc:2468] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 14.
I20260709 12:23:30.497910 23804 leader_election.cc:290] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:30.531344 24028 raft_consensus.cc:493] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:23:30.547998 24037 consensus_queue.cc:1059] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 20.16, Next index: 17, Last known committed idx: 15, Time since last communication: 0.036s
I20260709 12:23:30.598050 17757 raft_consensus.cc:1275] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 14 index: 8. Preceding OpId from leader: term: 17 index: 9. (index mismatch)
I20260709 12:23:30.598745 24037 raft_consensus.cc:3055] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:23:30.578794 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:30.531693 23979 consensus_queue.cc:1059] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 14.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.040s
I20260709 12:23:30.582988 17756 raft_consensus.cc:1275] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 15 index: 10. Preceding OpId from leader: term: 17 index: 11. (index mismatch)
W20260709 12:23:30.570793 24106 log.cc:927] Time spent T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.073s	user 0.000s	sys 0.001s
W20260709 12:23:30.558271 17729 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:30.585731 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:30.553064 24038 raft_consensus.cc:493] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:30.608403 24038 raft_consensus.cc:515] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:30.610110 17759 raft_consensus.cc:1275] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
I20260709 12:23:30.611130 24038 leader_election.cc:290] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:30.587539 17760 raft_consensus.cc:1240] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 15. Current term is 16. Ops: []
I20260709 12:23:30.611140 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:30.614137 23651 leader_election.cc:290] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 18 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:30.614292 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:30.552690 24059 raft_consensus.cc:697] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 16 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:23:30.616648 24059 consensus_queue.cc:237] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.9, Last appended by leader: 9, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:30.592331 23998 raft_consensus.cc:515] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:30.619860 17758 raft_consensus.cc:3060] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:30.601213 24028 raft_consensus.cc:515] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:30.625936 24028 leader_election.cc:290] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:30.621363 23998 leader_election.cc:290] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:30.599611 23901 raft_consensus.cc:493] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:23:30.628880 23901 raft_consensus.cc:515] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:30.589118 18362 rpcz_store.cc:276] 0709 12:23:29.404010 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:29.404134 (+   124us) service_pool.cc:224] Handling call
0709 12:23:29.404445 (+   311us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:29.622999 (+218554us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:30.588391 (+965392us) spinlock_profiling.cc:243] Waited 32.2 ms on lock 0x7b5400acf3d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:23:30.588587 (+   196us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":74060032}
I20260709 12:23:30.530913 23943 raft_consensus.cc:493] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:30.591375 18334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:30.487236 23905 raft_consensus.cc:515] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:30.497539 24098 consensus_queue.cc:260] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:30.576718 18333 leader_election.cc:304] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:30.572523 24052 raft_consensus.cc:2749] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 12:23:30.597431 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:30.596144 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:23:30.645601 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 69040.6us
I20260709 12:23:30.540207 23737 consensus_queue.cc:1059] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 13.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.038s
I20260709 12:23:30.649211 23737 raft_consensus.cc:3055] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:23:30.649605 23737 raft_consensus.cc:740] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:23:30.650084 23905 leader_election.cc:290] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 19 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:30.612521 24051 consensus_queue.cc:1048] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.039s
I20260709 12:23:30.579528 23700 raft_consensus.cc:740] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:23:30.653067 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 16 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:30.653270 23700 consensus_queue.cc:260] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.11, Last appended by leader: 11, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:30.654084 17689 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37402 (request call id 3026) took 1059 ms. Trace:
I20260709 12:23:30.655134 23700 raft_consensus.cc:3060] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:30.586658 24006 raft_consensus.cc:697] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 17 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
W20260709 12:23:30.656705 17683 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37402 (request call id 3009) took 1805 ms (client timeout 1705 ms). Trace:
W20260709 12:23:30.657205 17683 rpcz_store.cc:269] 0709 12:23:28.851195 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:28.851324 (+   129us) service_pool.cc:224] Handling call
0709 12:23:30.656515 (+1805191us) spinlock_profiling.cc:243] Waited 83.7 ms on lock 0x7b5400207e54. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:23:30.656634 (+   119us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":192456064}
I20260709 12:23:30.658907 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:30.659497 17683 raft_consensus.cc:3060] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:30.660650 17831 raft_consensus.cc:1240] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 18. Current term is 19. Ops: []
I20260709 12:23:30.604748 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:30.633960 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 18 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:30.663178 17682 raft_consensus.cc:2468] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 16.
I20260709 12:23:30.664638 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:30.665830 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:30.667294 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:30.635214 23943 raft_consensus.cc:515] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:23:30.668301 24056 consensus_queue.cc:1059] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 15.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.024s
I20260709 12:23:30.669866 24056 raft_consensus.cc:3055] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:23:30.670228 24056 raft_consensus.cc:740] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 15 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:23:30.670660 23943 leader_election.cc:290] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:30.671082 24056 consensus_queue.cc:260] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.9, Last appended by leader: 9, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:30.627432 24009 raft_consensus.cc:2749] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:23:30.695868 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:30.696657 17764 raft_consensus.cc:2468] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 17.
I20260709 12:23:30.636814 17801 leader_election.cc:304] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:30.637688 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:30.632362 17758 raft_consensus.cc:2468] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 19.
I20260709 12:23:30.639963 18365 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 1913) took 1747 ms. Trace:
I20260709 12:23:30.579929 24042 raft_consensus.cc:515] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:30.706640 23979 consensus_queue.cc:1048] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:23:30.650557 23737 consensus_queue.cc:260] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.10, Last appended by leader: 10, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:30.579728 23835 raft_consensus.cc:2804] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:23:30.656780 24006 consensus_queue.cc:237] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.11, Last appended by leader: 11, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:30.718327 23906 leader_election.cc:290] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:30.721997 23739 leader_election.cc:290] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 17 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:30.672605 17802 leader_election.cc:304] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:30.676607 23997 raft_consensus.cc:493] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:30.678910 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:30.679759 17687 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37402 (request call id 3024) took 1085 ms. Trace:
W20260709 12:23:30.728881 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:30.729936 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:30.731024 17682 raft_consensus.cc:3060] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:30.732693 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 19 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:30.733546 24111 raft_consensus.cc:493] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:30.715152 23835 raft_consensus.cc:493] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:30.734072 24111 raft_consensus.cc:515] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:30.734946 24021 raft_consensus.cc:493] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:30.683245 24102 raft_consensus.cc:2804] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:23:30.736135 24102 raft_consensus.cc:493] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:30.724512 24028 consensus_queue.cc:1059] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 13.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.021s
I20260709 12:23:30.688279 23901 leader_election.cc:290] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:30.603672 23930 raft_consensus.cc:3060] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:30.736690 24102 raft_consensus.cc:3060] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:30.690249 23931 leader_election.cc:290] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 17 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:30.634379 24039 raft_consensus.cc:3055] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:23:30.738876 24039 raft_consensus.cc:740] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:23:30.687709 23651 consensus_queue.cc:1048] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:23:30.697819 17681 raft_consensus.cc:2468] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 10.
I20260709 12:23:30.739858 24039 consensus_queue.cc:260] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.10, Last appended by leader: 10, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:30.740772 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:30.741287 24039 raft_consensus.cc:3060] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:30.741626 17681 raft_consensus.cc:3060] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:30.638805 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:23:30.698179 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 61475.9us
I20260709 12:23:30.744819 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:30.745048 17757 raft_consensus.cc:2468] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 15.
I20260709 12:23:30.746201 18362 raft_consensus.cc:2368] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for earlier term 12. Current term is 13.
I20260709 12:23:30.662107 17688 raft_consensus.cc:2468] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 12.
I20260709 12:23:30.749127 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 18 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:30.599313 24037 raft_consensus.cc:740] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:23:30.701254 17729 leader_election.cc:304] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:30.755239 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:23:30.756481 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 55326.5us
I20260709 12:23:30.757004 17836 raft_consensus.cc:2468] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 13.
I20260709 12:23:30.704190 17727 leader_election.cc:304] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:30.704447 17683 raft_consensus.cc:2468] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 14.
I20260709 12:23:30.705762 24061 raft_consensus.cc:2749] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:23:30.772871 17764 raft_consensus.cc:1275] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 17 index: 13. Preceding OpId from leader: term: 19 index: 14. (index mismatch)
I20260709 12:23:30.698764 23905 consensus_queue.cc:1048] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.008s
I20260709 12:23:30.706470 23998 raft_consensus.cc:493] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:30.780385 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:30.698818 24056 raft_consensus.cc:3060] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:30.595278 24096 raft_consensus.cc:2804] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:23:30.732797 23997 raft_consensus.cc:515] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:30.734625 23835 raft_consensus.cc:3060] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:30.735774 24021 raft_consensus.cc:515] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:30.750322 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:30.797392 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 19 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:30.727582 17687 rpcz_store.cc:276] 0709 12:23:29.593973 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:30.178120 (+584147us) service_pool.cc:224] Handling call
0709 12:23:30.679585 (+501465us) spinlock_profiling.cc:243] Waited 148 ms on lock 0x7b54000c6054. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:23:30.679704 (+   119us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":341282176}
I20260709 12:23:30.753756 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:30.654533 17689 rpcz_store.cc:276] 0709 12:23:29.594524 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:30.652732 (+1058208us) service_pool.cc:224] Handling call
0709 12:23:30.654011 (+  1279us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:30.754555 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 14 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:30.798892 17837 raft_consensus.cc:2468] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 13.
W20260709 12:23:30.682726 18333 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:30.800607 18333 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:30.756148 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:30.759138 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:30.759654 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:30.802407 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:30.803257 17683 raft_consensus.cc:2393] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 17: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:23:30.803148 18367 raft_consensus.cc:2468] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 13.
I20260709 12:23:30.763094 17836 raft_consensus.cc:1275] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
I20260709 12:23:30.764910 23979 raft_consensus.cc:2749] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:23:30.765122 24037 consensus_queue.cc:260] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 20.16, Last appended by leader: 16, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:30.766759 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:30.813815 17834 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 12.
W20260709 12:23:30.766491 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 62394.3us
I20260709 12:23:30.816219 17727 leader_election.cc:304] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:30.816990 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:30.768098 24031 raft_consensus.cc:2749] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:23:30.768873 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 19 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:30.739609 24042 leader_election.cc:290] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:23:30.772946 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 100392us
I20260709 12:23:30.775239 18366 raft_consensus.cc:1240] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 12. Current term is 13. Ops: []
I20260709 12:23:30.679947 23740 consensus_queue.cc:1048] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
W20260709 12:23:30.779783 17804 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:30.796788 24085 raft_consensus.cc:493] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:30.783416 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:30.783212 18363 raft_consensus.cc:1275] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 13 index: 11. Preceding OpId from leader: term: 15 index: 12. (index mismatch)
I20260709 12:23:30.849893 18374 raft_consensus.cc:2393] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 14: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:23:30.784793 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:30.783354 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:30.850493 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 13 candidate_status { last_received { term: 11 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:30.787292 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:30.851265 18366 raft_consensus.cc:2468] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 11.
I20260709 12:23:30.851672 17760 raft_consensus.cc:2468] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 15.
I20260709 12:23:30.788161 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 15 candidate_status { last_received { term: 14 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:30.698688 18365 rpcz_store.cc:276] 0709 12:23:28.893388 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:29.069914 (+176526us) service_pool.cc:224] Handling call
0709 12:23:29.070198 (+   284us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:29.097005 (+ 26807us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:30.635278 (+1538273us) spinlock_profiling.cc:243] Waited 26.1 ms on lock 0x7b5400466ed4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:23:30.639854 (+  4576us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":60110080}
I20260709 12:23:30.801666 24096 raft_consensus.cc:697] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 15 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:23:30.756383 24111 leader_election.cc:290] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:30.787664 24051 raft_consensus.cc:3060] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:30.818868 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 18 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:30.819160 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:30.819303 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:30.867182 18361 raft_consensus.cc:2410] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for term 17 because replica has last-logged OpId of term: 16 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 11.
I20260709 12:23:30.867208 18371 raft_consensus.cc:2393] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 15: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:23:30.820952 17682 raft_consensus.cc:2468] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 18.
I20260709 12:23:30.824489 23855 raft_consensus.cc:493] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:30.833411 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:30.868685 17763 raft_consensus.cc:3060] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:30.835726 18364 raft_consensus.cc:2393] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 18: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:23:30.762987 17801 leader_election.cc:304] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:30.839148 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:30.838158 24032 raft_consensus.cc:2804] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:23:30.887116 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:30.836855 17802 leader_election.cc:304] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:23:30.803927 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
W20260709 12:23:30.889328 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 52558.5us
I20260709 12:23:30.858032 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:30.889822 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 15 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:30.812556 23804 raft_consensus.cc:493] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:30.812194 23901 raft_consensus.cc:493] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:30.890395 17687 raft_consensus.cc:2393] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 19: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:23:30.547793 24026 raft_consensus.cc:2749] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 12:23:30.874078 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:30.875907 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:30.896178 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:30.896893 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 16 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
W20260709 12:23:30.879724 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 116812us
I20260709 12:23:30.880061 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:30.886870 17681 raft_consensus.cc:2468] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 17.
I20260709 12:23:30.886793 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 19 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:30.796192 23998 raft_consensus.cc:515] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:30.848410 17654 leader_election.cc:400] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 12 pre-election: Vote denied by peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577) with higher term. Message: Invalid argument: T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for earlier term 12. Current term is 13.
I20260709 12:23:30.857591 24037 raft_consensus.cc:3060] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:30.873508 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:30.889801 17762 raft_consensus.cc:2368] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 18. Current term is 19.
I20260709 12:23:30.890406 17802 leader_election.cc:304] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:30.855634 24022 raft_consensus.cc:3060] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:30.899480 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:30.947636 17681 raft_consensus.cc:2468] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 13.
W20260709 12:23:30.840903 24009 consensus_queue.cc:1175] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "334a4e1f59944544b3b3ff36616b57da" responder_term: 20 status { last_received { term: 20 index: 16 } last_committed_idx: 16 last_received_current_leader { term: 20 index: 16 } }
I20260709 12:23:30.902453 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 18 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:23:30.981796 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 91476.5us
I20260709 12:23:30.899438 17801 leader_election.cc:304] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
W20260709 12:23:30.941255 17728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:30.984699 17728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:30.921862 18333 leader_election.cc:304] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:23:30.983560 17802 leader_election.cc:304] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
W20260709 12:23:30.986559 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 64811.5us
I20260709 12:23:30.988168 18333 leader_election.cc:304] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:23:30.989713 18333 leader_election.cc:304] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:23:30.941619 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:30.992730 18361 raft_consensus.cc:2410] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for term 14 because replica has last-logged OpId of term: 13 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 11.
I20260709 12:23:30.992928 17755 raft_consensus.cc:1240] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 LEADER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 17. Current term is 18. Ops: []
I20260709 12:23:30.991608 18333 leader_election.cc:304] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:30.994652 17802 leader_election.cc:400] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 18 election: Vote denied by peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071) with higher term. Message: Invalid argument: T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 18. Current term is 19.
I20260709 12:23:30.902261 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:30.995163 17802 leader_election.cc:403] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 18 election: Cancelling election due to peer responding with higher term
I20260709 12:23:30.950636 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:30.996704 17682 raft_consensus.cc:2393] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 21: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:23:30.886720 24059 raft_consensus.cc:2749] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:23:30.953540 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 14 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:30.902217 23901 raft_consensus.cc:515] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:30.904410 24111 raft_consensus.cc:493] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:30.955865 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 19 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:31.011302 24111 raft_consensus.cc:515] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:31.012316 17681 raft_consensus.cc:1240] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 13. Current term is 14. Ops: []
I20260709 12:23:31.012462 18371 raft_consensus.cc:2468] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 16.
I20260709 12:23:31.012845 23901 leader_election.cc:290] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:31.013319 17681 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44898 (request call id 1171) took 2098 ms. Trace:
I20260709 12:23:31.013576 24111 leader_election.cc:290] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:31.013717 17681 rpcz_store.cc:276] 0709 12:23:28.914830 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:31.011786 (+2096956us) service_pool.cc:224] Handling call
0709 12:23:31.012131 (+   345us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:31.013070 (+   939us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:31.013263 (+   193us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:31.014448 24111 raft_consensus.cc:3060] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:31.015486 17681 raft_consensus.cc:1240] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 14 LEADER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 13. Current term is 14. Ops: [13.11-13.11]
I20260709 12:23:30.923749 23785 consensus_queue.cc:1048] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.033s
I20260709 12:23:30.956403 23998 leader_election.cc:290] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:23:30.940908 24052 raft_consensus.cc:670] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da: failed to trigger leader election: Illegal state: only voting members can start elections: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } [suppressed 5 similar messages]
I20260709 12:23:30.957571 23997 leader_election.cc:290] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:30.901587 23804 raft_consensus.cc:515] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:31.017858 24021 leader_election.cc:290] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:31.018005 23737 raft_consensus.cc:3060] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:30.973474 24096 consensus_queue.cc:237] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.9, Last appended by leader: 9, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:23:30.979106 24110 log.cc:927] Time spent T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.086s	user 0.000s	sys 0.000s
I20260709 12:23:30.946483 17654 leader_election.cc:403] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 12 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:23:30.953192 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 15 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:30.860821 24085 raft_consensus.cc:515] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:30.888200 24032 raft_consensus.cc:493] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
W20260709 12:23:31.023480 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 181903us
I20260709 12:23:30.902051 24047 raft_consensus.cc:493] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:31.032598 24032 raft_consensus.cc:3060] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:31.037015 23804 leader_election.cc:290] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:31.038044 17688 raft_consensus.cc:1240] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 15. Current term is 16. Ops: []
I20260709 12:23:31.032372 23700 raft_consensus.cc:2749] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Leader pre-election lost for term 12. Reason: Vote denied by peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577) with higher term. Message: Invalid argument: T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for earlier term 12. Current term is 13.
I20260709 12:23:31.038715 17688 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37402 (request call id 3029) took 1443 ms. Trace:
I20260709 12:23:31.039036 17688 rpcz_store.cc:276] 0709 12:23:29.595371 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:31.034612 (+1439241us) service_pool.cc:224] Handling call
0709 12:23:31.037882 (+  3270us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:31.038468 (+   586us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:31.038661 (+   193us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:31.005071 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:23:31.003316 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 104002us
I20260709 12:23:31.007386 17682 raft_consensus.cc:1240] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 13. Current term is 14. Ops: []
I20260709 12:23:31.001263 23931 raft_consensus.cc:493] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:31.009104 23992 raft_consensus.cc:2804] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:23:31.010332 23970 raft_consensus.cc:3060] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:31.001180 24117 raft_consensus.cc:2804] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:23:30.899405 24061 raft_consensus.cc:2804] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:23:31.019407 18371 raft_consensus.cc:1275] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 16 index: 12. (index mismatch)
I20260709 12:23:31.019698 17681 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44898 (request call id 1172) took 2098 ms. Trace:
I20260709 12:23:31.021003 17729 leader_election.cc:304] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:31.011646 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:31.001616 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:30.997728 23905 raft_consensus.cc:3060] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:31.031701 17654 leader_election.cc:304] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:23:30.892674 23855 raft_consensus.cc:515] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:30.998088 23651 raft_consensus.cc:2804] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:23:31.040643 17688 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37402 (request call id 3034) took 1233 ms. Trace:
I20260709 12:23:31.006582 17680 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119"
dest_uuid: "0da169ff82674dc1947055ce43de68a3"
mode: GRACEFUL
new_leader_uuid: "334a4e1f59944544b3b3ff36616b57da"
 from {username='slave'} at 127.0.0.1:54540
I20260709 12:23:31.043186 17682 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44898 (request call id 1170) took 2129 ms. Trace:
I20260709 12:23:31.079638 17682 rpcz_store.cc:276] 0709 12:23:28.913458 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:31.006639 (+2093181us) service_pool.cc:224] Handling call
0709 12:23:31.007136 (+   497us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:31.042877 (+ 35741us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:31.043118 (+   241us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:31.080200 18363 raft_consensus.cc:1240] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 16. Current term is 17. Ops: []
I20260709 12:23:31.041332 17801 leader_election.cc:304] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:31.013490 23901 raft_consensus.cc:2804] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:23:31.008695 23739 raft_consensus.cc:2749] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Leader election lost for term 18. Reason: Vote denied by peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071) with higher term. Message: Invalid argument: T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 18. Current term is 19.
I20260709 12:23:31.058701 17833 raft_consensus.cc:1275] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 17 index: 11. Preceding OpId from leader: term: 19 index: 12. (index mismatch)
W20260709 12:23:31.085634 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 44385.8us
I20260709 12:23:31.060148 18361 raft_consensus.cc:2468] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 16.
I20260709 12:23:31.046597 17686 raft_consensus.cc:1275] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 11 index: 9. Preceding OpId from leader: term: 13 index: 10. (index mismatch)
I20260709 12:23:31.066810 17654 leader_election.cc:304] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:23:30.981829 17687 raft_consensus.cc:2468] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 17.
W20260709 12:23:31.070904 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 207519us
W20260709 12:23:31.090101 17803 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:31.040532 17802 leader_election.cc:304] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
W20260709 12:23:31.076530 17729 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:31.073393 17688 rpcz_store.cc:276] 0709 12:23:29.806984 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:31.039809 (+1232825us) service_pool.cc:224] Handling call
0709 12:23:31.040159 (+   350us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:31.040321 (+   162us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:31.040352 (+    31us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:31.040412 (+    60us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:23:31.040447 (+    35us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:31.040498 (+    51us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:31.040596 (+    98us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:31.047423 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 13 candidate_status { last_received { term: 11 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:31.073750 17680 raft_consensus.cc:606] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 15 LEADER]: Received request to transfer leadership to TS 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:31.092515 17838 raft_consensus.cc:2468] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 12.
I20260709 12:23:31.050403 17836 raft_consensus.cc:1240] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 14. Current term is 15. Ops: []
I20260709 12:23:31.052474 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 18 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:31.054515 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:31.056885 17681 rpcz_store.cc:276] 0709 12:23:28.922165 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:31.014897 (+2092732us) service_pool.cc:224] Handling call
0709 12:23:31.015237 (+   340us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:31.019408 (+  4171us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:31.019641 (+   233us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:31.117614 17729 leader_election.cc:304] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:31.128368 17682 raft_consensus.cc:1275] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 14 index: 8. Preceding OpId from leader: term: 17 index: 9. (index mismatch)
I20260709 12:23:31.153437 18366 raft_consensus.cc:1275] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
W20260709 12:23:31.153056 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 232748us
I20260709 12:23:31.155792 17688 raft_consensus.cc:1240] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 17. Current term is 18. Ops: []
W20260709 12:23:31.157509 17652 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:31.159011 18334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:31.055780 23992 raft_consensus.cc:493] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:31.159909 17680 raft_consensus.cc:1275] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
W20260709 12:23:31.161216 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 126767us
W20260709 12:23:31.171322 17727 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:31.161854 17763 raft_consensus.cc:2468] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 18.
I20260709 12:23:31.082491 23651 raft_consensus.cc:493] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:31.082667 24042 raft_consensus.cc:2804] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:23:31.167351 17687 raft_consensus.cc:1240] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 16. Current term is 17. Ops: []
I20260709 12:23:31.072567 24088 raft_consensus.cc:2804] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:23:31.076115 23997 raft_consensus.cc:2804] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:23:31.171749 17802 leader_election.cc:304] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:31.061225 24117 raft_consensus.cc:493] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:31.069104 24061 raft_consensus.cc:697] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 14 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:23:31.172891 24088 raft_consensus.cc:493] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:31.124025 24059 raft_consensus.cc:2749] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:23:31.040287 24047 raft_consensus.cc:515] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:31.185889 17688 raft_consensus.cc:1275] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20260709 12:23:31.124632 23901 raft_consensus.cc:697] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 18 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:23:31.172947 23997 raft_consensus.cc:493] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:31.158088 23906 raft_consensus.cc:2804] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:23:31.056445 23931 raft_consensus.cc:515] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:31.084510 23925 consensus_queue.cc:1059] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 12.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.094s
I20260709 12:23:31.142203 23740 consensus_queue.cc:1048] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.048s
I20260709 12:23:31.173605 23739 raft_consensus.cc:2804] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Leader pre-election won for term 18
W20260709 12:23:31.196312 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 204772us
I20260709 12:23:31.173992 23651 raft_consensus.cc:3060] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:23:31.212376 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:31.143194 23700 consensus_queue.cc:1048] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.064s
I20260709 12:23:31.176304 23992 raft_consensus.cc:3060] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:31.194919 23997 raft_consensus.cc:3060] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Advancing to term 17
W20260709 12:23:31.239629 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 240851us
W20260709 12:23:31.263144 17728 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.16.29.65:33071 timed out after 1.664s (SENT) [suppressed 26 similar messages]
I20260709 12:23:31.194532 23906 raft_consensus.cc:493] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:31.264043 23906 raft_consensus.cc:3060] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:31.265264 23700 raft_consensus.cc:2804] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:23:31.265727 23700 raft_consensus.cc:493] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:31.266078 23700 raft_consensus.cc:3060] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:31.247486 23739 raft_consensus.cc:493] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:31.268517 23739 raft_consensus.cc:3060] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:31.188342 24088 raft_consensus.cc:3060] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:31.179607 24117 raft_consensus.cc:3060] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:31.281704 18333 leader_election.cc:304] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:31.282974 24052 raft_consensus.cc:2804] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:23:31.283599 24052 raft_consensus.cc:493] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:23:31.284446 24052 raft_consensus.cc:3060] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:23:31.294356 24125 log.cc:927] Time spent T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.338s	user 0.003s	sys 0.000s
I20260709 12:23:31.300835 24085 leader_election.cc:290] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:31.302146 23992 raft_consensus.cc:515] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:31.302999 24021 raft_consensus.cc:493] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:31.304077 24061 consensus_queue.cc:237] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.11, Last appended by leader: 11, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:31.304522 23739 raft_consensus.cc:515] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:31.291307 23700 raft_consensus.cc:515] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:31.347436 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:31.318043 24116 raft_consensus.cc:493] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:31.310151 23855 leader_election.cc:290] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:31.356436 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:31.381161 17837 raft_consensus.cc:2468] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 16.
I20260709 12:23:31.358278 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:31.348464 23901 consensus_queue.cc:237] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.11, Last appended by leader: 11, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:31.346845 24067 raft_consensus.cc:2749] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
W20260709 12:23:31.354303 24102 consensus_meta.cc:287] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.616s	user 0.013s	sys 0.000s
I20260709 12:23:31.350416 24111 raft_consensus.cc:2749] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:23:31.408706 24102 raft_consensus.cc:515] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:31.354303 24021 raft_consensus.cc:515] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:31.414855 24051 consensus_meta.cc:287] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.542s	user 0.012s	sys 0.000s
I20260709 12:23:31.475278 24051 raft_consensus.cc:2749] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:23:31.477461 23931 leader_election.cc:290] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:31.403239 24117 raft_consensus.cc:515] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:31.409931 23905 raft_consensus.cc:2749] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:23:31.390209 24116 raft_consensus.cc:515] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:31.371749 24032 raft_consensus.cc:515] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:31.553964 24102 leader_election.cc:290] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 15 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:31.556792 24047 leader_election.cc:290] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:31.501842 23992 leader_election.cc:290] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 16 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:31.487527 24117 leader_election.cc:290] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 16 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:23:31.802305 17727 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:31.806315 17728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:31.809094 17681 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.657s	user 0.012s	sys 0.005s
W20260709 12:23:31.809706 17681 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.658s	user 0.012s	sys 0.005s
I20260709 12:23:31.811287 17681 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44898 (request call id 1188) took 1315 ms. Trace:
W20260709 12:23:31.828644 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 456316us
I20260709 12:23:31.813910 17682 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37402 (request call id 3040) took 1610 ms. Trace:
I20260709 12:23:31.829599 17682 rpcz_store.cc:276] 0709 12:23:30.203319 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:31.080465 (+877146us) service_pool.cc:224] Handling call
0709 12:23:31.127820 (+ 47355us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:31.131204 (+  3384us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:31.813434 (+682230us) spinlock_profiling.cc:243] Waited 47 ms on lock 0x7b5400207950. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a61f8ad 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:23:31.813726 (+   292us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":108074240}
I20260709 12:23:31.816555 17686 raft_consensus.cc:1240] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 14. Current term is 15. Ops: []
I20260709 12:23:31.811172 17684 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.BulkChangeConfig from 127.0.0.1:33458 (request call id 0) took 2215 ms. Trace:
W20260709 12:23:31.819936 17729 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:31.349354 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 232529us
I20260709 12:23:31.831982 17684 rpcz_store.cc:276] 0709 12:23:29.595849 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:31.176813 (+1580964us) service_pool.cc:224] Handling call
0709 12:23:31.810490 (+633677us) spinlock_profiling.cc:243] Waited 620 ms on lock 0x7b54003016d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a626735 00007f7b4f5e6d29 00007f7b48b651d4 00007f7b48b65162 00007f7b48b65091 00007f7b48b6500c 00007f7b48b642c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:23:31.810873 (+   383us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1426019200}
W20260709 12:23:31.832005 17803 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:31.834100 17686 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37402 (request call id 3044) took 1452 ms. Trace:
W20260709 12:23:31.834372 17804 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:31.837908 17830 raft_consensus.cc:1240] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 17. Current term is 18. Ops: []
W20260709 12:23:31.839820 18333 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:31.841030 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:31.842686 17831 raft_consensus.cc:2393] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 19: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:23:31.844942 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:23:31.843705 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:31.839864 17729 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:31.866313 23905 consensus_queue.cc:1048] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:23:31.869695 18362 raft_consensus.cc:1275] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 16 index: 14. Preceding OpId from leader: term: 18 index: 15. (index mismatch)
I20260709 12:23:31.870013 18366 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59902 (request call id 1537) took 1013 ms. Trace:
W20260709 12:23:31.871133 23737 consensus_meta.cc:287] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.852s	user 0.011s	sys 0.002s
I20260709 12:23:31.866582 23979 raft_consensus.cc:493] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:23:31.872416 23979 raft_consensus.cc:515] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:31.877107 23739 leader_election.cc:290] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 18 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:31.878127 23739 raft_consensus.cc:493] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:31.878844 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:31.878798 23739 raft_consensus.cc:515] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:31.870517 18366 rpcz_store.cc:276] 0709 12:23:30.856628 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:31.154246 (+297618us) service_pool.cc:224] Handling call
0709 12:23:31.154581 (+   335us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:31.159155 (+  4574us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:31.159189 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:31.181831 (+ 22642us) log.cc:844] Serialized 26 byte log entry
0709 12:23:31.381280 (+199449us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:23:31.381502 (+   222us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:31.381804 (+   302us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:31.863832 (+482028us) raft_consensus.cc:1685] finished
0709 12:23:31.869407 (+  5575us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:31.869757 (+   350us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":4045,"raft.run_cpu_time_us":477,"raft.run_wall_time_us":8468,"thread_start_us":147291,"threads_started":1,"wal-append.queue_time_us":344804}
W20260709 12:23:31.894567 17654 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.768s	user 0.003s	sys 0.006s
W20260709 12:23:31.895051 17654 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.768s	user 0.003s	sys 0.006s
W20260709 12:23:31.895825 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 829020us
I20260709 12:23:31.834661 17686 rpcz_store.cc:276] 0709 12:23:30.382214 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:31.087328 (+705114us) service_pool.cc:224] Handling call
0709 12:23:31.087620 (+   292us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:31.831983 (+744363us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:31.833756 (+  1773us) spinlock_profiling.cc:243] Waited 728 ms on lock 0x7b540014dcd4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:23:31.834042 (+   286us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1674395008}
W20260709 12:23:31.909999 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:31.916158 24032 leader_election.cc:290] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 13 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:31.925273 24148 raft_consensus.cc:493] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:23:31.926365 24148 raft_consensus.cc:515] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:31.916703 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 20 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:23:31.938189 23906 consensus_meta.cc:287] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.673s	user 0.009s	sys 0.000s
I20260709 12:23:31.921273 24102 raft_consensus.cc:493] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:31.938885 23906 raft_consensus.cc:515] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:31.939743 24102 raft_consensus.cc:515] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:31.940376 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:31.945189 23979 leader_election.cc:290] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:23:31.948673 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 90841.5us
W20260709 12:23:31.952857 24135 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.528s	user 0.008s	sys 0.014s
I20260709 12:23:31.945523 24116 leader_election.cc:290] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:23:31.955792 17635 auto_rebalancer.cc:629] Failed to schedule move for tablet 00a02ab39fcd4078ad49697693cf3be8: Illegal state: Replica 0da169ff82674dc1947055ce43de68a3 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 17 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } }
I20260709 12:23:31.956139 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 18 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:31.957651 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:31.828979 17681 rpcz_store.cc:276] 0709 12:23:30.496140 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:31.111058 (+614918us) service_pool.cc:224] Handling call
0709 12:23:31.111564 (+   506us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:31.112126 (+   562us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 12:23:31.151488 (+ 39362us) log.cc:844] Serialized 14 byte log entry
0709 12:23:31.810322 (+658834us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:31.810526 (+   204us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:23:31.810684 (+   158us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:31.810846 (+   162us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:31.811153 (+   307us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":139077,"prepare.run_cpu_time_us":84,"prepare.run_wall_time_us":87,"raft.queue_time_us":138544,"raft.run_cpu_time_us":103,"raft.run_wall_time_us":104,"thread_start_us":697091,"threads_started":2}
W20260709 12:23:31.956158 24153 log.cc:927] Time spent T 00000000000000000000000000000000 P 2193af31cb894ec9b3d65d939a2a6851: Append to log took a long time: real 0.059s	user 0.000s	sys 0.002s
I20260709 12:23:31.972718 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 18 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:31.978173 17760 raft_consensus.cc:1275] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 12 index: 10. (index mismatch)
I20260709 12:23:31.978842 24148 leader_election.cc:290] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:23:31.979593 24135 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.595s	user 0.009s	sys 0.015s
I20260709 12:23:31.974330 17830 raft_consensus.cc:2468] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 17.
I20260709 12:23:31.986369 23700 leader_election.cc:290] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 13 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:31.989560 24021 leader_election.cc:290] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:31.997215 17762 raft_consensus.cc:2393] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 17: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
W20260709 12:23:31.994831 17652 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.760s	user 0.014s	sys 0.010s
I20260709 12:23:32.020710 17759 raft_consensus.cc:2468] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 19.
W20260709 12:23:32.024756 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 76400us
I20260709 12:23:32.027624 24148 raft_consensus.cc:493] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:32.028813 24148 raft_consensus.cc:515] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:32.032366 24148 leader_election.cc:290] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:23:32.040253 24052 consensus_meta.cc:287] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.754s	user 0.010s	sys 0.006s
W20260709 12:23:32.018105 17652 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.858s	user 0.015s	sys 0.010s
I20260709 12:23:32.040128 23855 raft_consensus.cc:493] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:32.049225 24137 consensus_queue.cc:1059] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 16.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.047s
I20260709 12:23:32.049865 23739 leader_election.cc:290] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:32.050387 23855 raft_consensus.cc:515] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:32.051198 23928 raft_consensus.cc:493] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:31.999166 24117 raft_consensus.cc:493] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:32.053583 23928 raft_consensus.cc:515] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:32.057977 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 109056us
I20260709 12:23:32.069408 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:32.075150 17836 raft_consensus.cc:1275] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 15 index: 13. Preceding OpId from leader: term: 17 index: 14. (index mismatch)
W20260709 12:23:32.063614 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 905012us
W20260709 12:23:32.078445 23901 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.598s	user 0.004s	sys 0.015s
I20260709 12:23:32.079775 23906 leader_election.cc:290] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 17 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:32.069001 17801 leader_election.cc:304] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:32.082376 23739 raft_consensus.cc:2804] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:23:32.082940 23739 raft_consensus.cc:493] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:23:32.046085 24052 raft_consensus.cc:515] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:32.091410 24022 consensus_meta.cc:287] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 1.138s	user 0.017s	sys 0.006s
W20260709 12:23:32.092674 17728 leader_election.cc:336] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 17 election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Timed out: RequestConsensusVote RPC to 127.16.29.65:33071 timed out after 1.566s (SENT)
I20260709 12:23:32.093236 23838 raft_consensus.cc:493] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:32.093390 24017 raft_consensus.cc:3060] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Advancing to term 19
W20260709 12:23:32.082970 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 67020.3us
I20260709 12:23:32.083634 23739 raft_consensus.cc:3060] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:32.113096 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 15 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:32.065234 24135 raft_consensus.cc:493] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:32.117848 24028 consensus_queue.cc:1048] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:23:32.118193 24042 raft_consensus.cc:493] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:32.118880 24042 raft_consensus.cc:3060] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Advancing to term 19
W20260709 12:23:32.119980 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 178435us
I20260709 12:23:32.122040 17801 leader_election.cc:400] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 18 election: Vote denied by peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651) with higher term. Message: Invalid argument: T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 18. Current term is 19.
I20260709 12:23:32.122253 24098 raft_consensus.cc:3060] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:23:32.126096 23835 consensus_meta.cc:287] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 1.314s	user 0.017s	sys 0.002s
I20260709 12:23:32.126827 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:32.126665 23835 raft_consensus.cc:515] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:32.128326 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:32.128414 24154 consensus_queue.cc:1048] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.061s
I20260709 12:23:32.129567 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:32.130677 18333 leader_election.cc:304] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:32.130787 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:32.131088 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:32.128535 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 18 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:32.132843 17686 raft_consensus.cc:3060] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:32.133169 24117 raft_consensus.cc:515] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:32.136371 18371 raft_consensus.cc:1240] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 18. Current term is 19. Ops: []
I20260709 12:23:32.136667 24022 consensus_queue.cc:1048] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:23:32.137755 24155 raft_consensus.cc:493] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:32.138603 24022 raft_consensus.cc:2804] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:23:32.138542 24155 raft_consensus.cc:515] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:32.140324 24117 leader_election.cc:290] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:32.139264 24022 raft_consensus.cc:493] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:23:32.141767 24022 raft_consensus.cc:3060] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Advancing to term 20
W20260709 12:23:32.143441 24037 consensus_meta.cc:287] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 1.195s	user 0.016s	sys 0.007s
I20260709 12:23:32.128857 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:32.123673 17801 leader_election.cc:304] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:32.094237 23838 raft_consensus.cc:515] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:32.078467 23931 raft_consensus.cc:493] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:32.155670 23931 raft_consensus.cc:515] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:32.158700 23931 leader_election.cc:290] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:32.165616 23855 leader_election.cc:290] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:23:32.167543 23970 consensus_meta.cc:287] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 1.105s	user 0.011s	sys 0.015s
I20260709 12:23:32.169615 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 18 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:32.171060 18367 raft_consensus.cc:2410] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for term 15 because replica has last-logged OpId of term: 13 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 11.
W20260709 12:23:32.171739 23930 consensus_meta.cc:287] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 1.373s	user 0.017s	sys 0.008s
I20260709 12:23:32.171860 17758 raft_consensus.cc:2468] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 21.
I20260709 12:23:32.172362 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:32.181862 17755 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38786 (request call id 3314) took 1391 ms. Trace:
W20260709 12:23:32.184847 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 69432.6us
W20260709 12:23:32.193010 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 66255.4us
W20260709 12:23:32.203828 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 80337.6us
I20260709 12:23:32.185181 17755 rpcz_store.cc:276] 0709 12:23:30.791462 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:30.855371 (+ 63909us) service_pool.cc:224] Handling call
0709 12:23:30.855636 (+   265us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:30.993567 (+137931us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:32.181299 (+1187732us) spinlock_profiling.cc:243] Waited 137 ms on lock 0x7b54004155d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:23:32.181649 (+   350us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":315176832}
I20260709 12:23:32.206313 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:32.210670 18362 raft_consensus.cc:3060] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Advancing to term 17
W20260709 12:23:32.206346 24056 consensus_meta.cc:287] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 1.416s	user 0.016s	sys 0.009s
I20260709 12:23:32.218818 17684 raft_consensus.cc:2393] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 18: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:23:32.200790 24135 raft_consensus.cc:515] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:32.264328 24156 log.cc:927] Time spent T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.057s	user 0.000s	sys 0.001s
I20260709 12:23:32.264743 24056 raft_consensus.cc:493] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:32.265431 24056 raft_consensus.cc:515] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:32.269044 17652 leader_election.cc:304] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:32.270129 23976 raft_consensus.cc:493] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:32.270380 17652 leader_election.cc:304] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:32.270860 23976 raft_consensus.cc:515] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:32.272526 17764 raft_consensus.cc:2468] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 18.
I20260709 12:23:32.273864 24154 raft_consensus.cc:2764] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Leader pre-election decision vote started in defunct term 17: won
I20260709 12:23:32.274816 24047 raft_consensus.cc:2749] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:23:32.275996 23930 raft_consensus.cc:493] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:32.276134 23740 raft_consensus.cc:2804] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:23:32.276762 23740 raft_consensus.cc:697] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 18 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:23:32.261358 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:32.278110 23740 consensus_queue.cc:237] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.9, Last appended by leader: 9, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:32.278770 17755 raft_consensus.cc:3060] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:32.281104 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 15 candidate_status { last_received { term: 14 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:32.281258 24155 leader_election.cc:290] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:32.282637 23930 raft_consensus.cc:515] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:32.260870 17727 leader_election.cc:304] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:32.293453 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:32.260839 23953 raft_consensus.cc:493] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:32.298166 23930 leader_election.cc:290] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:32.298570 23953 raft_consensus.cc:515] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:32.300168 24139 consensus_queue.cc:1059] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 16.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:23:32.301867 23835 leader_election.cc:290] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 19 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:32.262068 23906 raft_consensus.cc:493] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:32.302457 23998 consensus_queue.cc:1059] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 17.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:23:32.303025 23953 leader_election.cc:290] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:32.292984 24056 leader_election.cc:290] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:32.303109 23906 raft_consensus.cc:515] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:32.292688 17727 leader_election.cc:304] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:32.304132 24056 raft_consensus.cc:493] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:32.304824 24056 raft_consensus.cc:515] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:32.305629 23906 leader_election.cc:290] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:32.310557 18364 raft_consensus.cc:2410] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for term 15 because replica has last-logged OpId of term: 13 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 10.
I20260709 12:23:32.318467 18371 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 1931) took 1495 ms. Trace:
I20260709 12:23:32.318869 18371 rpcz_store.cc:276] 0709 12:23:30.822692 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:31.078626 (+255934us) service_pool.cc:224] Handling call
0709 12:23:31.142635 (+ 64009us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:32.136895 (+994260us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:32.318160 (+181265us) spinlock_profiling.cc:243] Waited 986 ms on lock 0x7b54006ae0d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:23:32.318387 (+   227us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2266675968}
I20260709 12:23:32.321223 24144 raft_consensus.cc:3055] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:23:32.321682 24144 raft_consensus.cc:740] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:23:32.322310 24116 raft_consensus.cc:493] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:32.322822 24144 consensus_queue.cc:260] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.12, Last appended by leader: 12, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:32.323012 24116 raft_consensus.cc:515] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:32.324286 23928 leader_election.cc:290] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:32.328460 23700 raft_consensus.cc:3055] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:23:32.328913 23700 raft_consensus.cc:740] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:23:32.330034 23700 consensus_queue.cc:260] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 16.13, Last appended by leader: 13, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:32.331569 23700 raft_consensus.cc:3060] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:32.334160 18361 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59886 (request call id 1713) took 1498 ms. Trace:
I20260709 12:23:32.312541 17756 raft_consensus.cc:1240] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 15 LEADER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 14. Current term is 15. Ops: []
W20260709 12:23:32.314693 18334 leader_election.cc:336] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 17 election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Timed out: RequestConsensusVote RPC to 127.16.29.65:33071 timed out after 1.913s (SENT)
I20260709 12:23:32.273887 24135 leader_election.cc:290] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:32.311797 23953 raft_consensus.cc:493] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:32.315833 24085 raft_consensus.cc:3055] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:23:32.341080 17755 raft_consensus.cc:2468] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 15.
I20260709 12:23:32.343955 17686 raft_consensus.cc:2468] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 18.
W20260709 12:23:32.341142 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 246685us
I20260709 12:23:32.349244 17761 raft_consensus.cc:2468] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 19.
I20260709 12:23:32.360972 24056 leader_election.cc:290] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:32.360662 23953 raft_consensus.cc:515] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:32.362526 24116 leader_election.cc:290] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:32.366710 23739 raft_consensus.cc:515] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:32.370608 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 62714.1us
I20260709 12:23:32.375036 24042 raft_consensus.cc:515] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:32.368224 23970 raft_consensus.cc:493] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:32.376468 23970 raft_consensus.cc:515] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:32.371582 17802 leader_election.cc:304] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:32.377605 23998 raft_consensus.cc:3055] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:23:32.377990 23998 raft_consensus.cc:740] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:23:32.379076 23998 consensus_queue.cc:260] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 17.9, Last appended by leader: 9, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:23:32.380029 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 87530.3us
I20260709 12:23:32.380594 18333 leader_election.cc:304] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:23:32.381160 24017 raft_consensus.cc:493] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:32.381534 24169 raft_consensus.cc:493] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:32.381747 17728 leader_election.cc:336] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Timed out: RequestConsensusVote RPC to 127.16.29.65:33071 timed out after 1.786s (SENT)
I20260709 12:23:32.382126 24028 raft_consensus.cc:2804] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:23:32.382820 24052 leader_election.cc:290] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 18 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:32.381207 24032 raft_consensus.cc:493] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:32.384665 24144 raft_consensus.cc:3060] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Advancing to term 17
W20260709 12:23:32.385159 24088 consensus_meta.cc:287] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 1.111s	user 0.014s	sys 0.009s
W20260709 12:23:32.385639 24039 consensus_meta.cc:287] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 1.643s	user 0.032s	sys 0.046s
I20260709 12:23:32.334614 18361 rpcz_store.cc:276] 0709 12:23:30.836039 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:31.116269 (+280230us) service_pool.cc:224] Handling call
0709 12:23:31.168873 (+ 52604us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:31.169448 (+   575us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 12:23:31.848067 (+678619us) spinlock_profiling.cc:243] Waited 3.55 ms on lock 0x7b1800321ee0. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b47515e07 00007f7b503709a9 00007f7b4753269b 00007f7b4753093f 00007f7b47530455 00007f7b4e0183c7 00007f7b4a62a9a7 00007f7b4a63a0bf 00007f7b4a63a038
0709 12:23:31.874703 (+ 26636us) spinlock_profiling.cc:243] Waited 9.9 ms on lock 0x7b1800321ee0. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b475333f3 00007f7b475357c1 00007f7b47535779 00007f7b47535709 00007f7b475356d1 00007f7b475349cd 0000000000388634 0000000000388479 00007f7b4751683a
0709 12:23:32.046365 (+171662us) log.cc:844] Serialized 14 byte log entry
0709 12:23:32.211212 (+164847us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:32.263228 (+ 52016us) log.cc:844] Serialized 26 byte log entry
0709 12:23:32.263412 (+   184us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:23:32.263452 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:32.263583 (+   131us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:32.333785 (+ 70202us) raft_consensus.cc:1685] finished
0709 12:23:32.333806 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:32.333977 (+   171us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":11961,"prepare.queue_time_us":64664,"prepare.run_cpu_time_us":101,"prepare.run_wall_time_us":103,"raft.queue_time_us":15530,"raft.run_cpu_time_us":277,"raft.run_wall_time_us":7700,"spinlock_wait_cycles":30955136,"thread_start_us":282458,"threads_started":2,"wal-append.queue_time_us":265038}
I20260709 12:23:32.386277 18367 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59886 (request call id 1710) took 1561 ms. Trace:
I20260709 12:23:32.397079 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 19 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:23:32.399091 17803 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:32.342849 24085 raft_consensus.cc:740] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:23:32.336369 24155 raft_consensus.cc:2749] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:23:32.401407 18362 raft_consensus.cc:2468] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 17.
I20260709 12:23:32.380086 24116 raft_consensus.cc:2749] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:23:32.350579 23855 raft_consensus.cc:493] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:32.426491 23855 raft_consensus.cc:515] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:32.429153 23855 leader_election.cc:290] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:32.405072 18367 rpcz_store.cc:276] 0709 12:23:30.824604 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:30.875579 (+ 50975us) service_pool.cc:224] Handling call
0709 12:23:32.386000 (+1510421us) spinlock_profiling.cc:243] Waited 1.27 s on lock 0x7b54006946d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:23:32.386189 (+   189us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2932206464}
I20260709 12:23:32.408353 17728 leader_election.cc:304] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:32.411119 23739 leader_election.cc:290] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 15 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:32.411617 23953 leader_election.cc:290] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:32.412251 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 15 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:23:32.430859 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 49195.8us
W20260709 12:23:32.414188 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 99573.4us
W20260709 12:23:32.433684 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 84061.9us
I20260709 12:23:32.434263 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:32.434587 18333 leader_election.cc:304] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:23:32.435101 17759 raft_consensus.cc:2468] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 14.
I20260709 12:23:32.436215 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:32.437192 24131 raft_consensus.cc:2804] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:23:32.437652 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:32.437793 24131 raft_consensus.cc:493] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:32.438436 24131 raft_consensus.cc:3060] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:32.414992 23998 raft_consensus.cc:3060] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:32.438448 17759 raft_consensus.cc:3060] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:32.440582 24135 raft_consensus.cc:2804] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:23:32.441051 24135 raft_consensus.cc:493] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:32.441457 24135 raft_consensus.cc:3060] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:32.403010 24169 raft_consensus.cc:515] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:32.396697 24028 raft_consensus.cc:493] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:32.415872 24032 raft_consensus.cc:515] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:32.444465 24096 raft_consensus.cc:2749] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:23:32.445873 23739 raft_consensus.cc:493] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:32.446579 23739 raft_consensus.cc:515] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:32.412251 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:32.449203 17755 raft_consensus.cc:3060] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:32.419531 24037 consensus_queue.cc:1048] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20260709 12:23:32.420358 24155 raft_consensus.cc:493] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
W20260709 12:23:32.421667 23651 consensus_meta.cc:287] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 1.108s	user 0.018s	sys 0.000s
I20260709 12:23:32.422122 24056 consensus_queue.cc:1048] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 12:23:32.422937 23992 consensus_queue.cc:1048] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:23:32.424985 24042 leader_election.cc:290] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 19 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:32.419543 24059 raft_consensus.cc:493] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:32.433651 17728 leader_election.cc:304] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:32.443733 24028 raft_consensus.cc:3060] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:32.415421 24009 raft_consensus.cc:493] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:32.477732 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 65342.5us
W20260709 12:23:32.481544 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 45941.7us
I20260709 12:23:32.461370 24154 consensus_queue.cc:1048] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.025s
I20260709 12:23:32.486400 24085 consensus_queue.cc:260] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.10, Last appended by leader: 10, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:32.487627 24085 raft_consensus.cc:3060] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:32.481353 24059 raft_consensus.cc:515] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:32.450212 24098 consensus_queue.cc:1048] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.034s
I20260709 12:23:32.450014 24088 raft_consensus.cc:515] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:32.493566 24169 leader_election.cc:290] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:32.494393 24088 leader_election.cc:290] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 14 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:32.494735 24032 leader_election.cc:290] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:32.496061 24009 raft_consensus.cc:515] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:32.498250 24009 leader_election.cc:290] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:32.499336 24042 consensus_queue.cc:1048] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 12:23:32.502830 17652 leader_election.cc:304] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:32.504534 17652 leader_election.cc:304] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:32.506318 17652 leader_election.cc:304] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:32.507308 24088 raft_consensus.cc:2749] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:23:32.479820 17757 raft_consensus.cc:2468] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 17.
I20260709 12:23:32.508549 24088 raft_consensus.cc:2749] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:23:32.499467 18333 leader_election.cc:304] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
W20260709 12:23:32.509702 23997 consensus_meta.cc:287] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 1.162s	user 0.019s	sys 0.002s
I20260709 12:23:32.510190 23997 raft_consensus.cc:515] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:32.511425 17763 raft_consensus.cc:2393] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 17: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:23:32.512086 24155 raft_consensus.cc:515] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:32.512846 24151 raft_consensus.cc:2804] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:23:32.513417 24151 raft_consensus.cc:493] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:32.513778 24151 raft_consensus.cc:3060] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:32.514910 24155 leader_election.cc:290] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:23:32.515543 17655 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:32.509763 24170 log.cc:927] Time spent T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.133s	user 0.002s	sys 0.000s
I20260709 12:23:32.520993 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:32.480094 23651 raft_consensus.cc:515] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:32.526324 23651 leader_election.cc:290] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 18 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:32.527027 17755 raft_consensus.cc:2468] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 18.
W20260709 12:23:32.529717 18333 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:32.530517 24042 raft_consensus.cc:493] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:23:32.531152 24042 raft_consensus.cc:515] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:32.532016 23953 raft_consensus.cc:493] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:32.532531 23953 raft_consensus.cc:515] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:32.495432 24022 raft_consensus.cc:515] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:32.533324 24042 leader_election.cc:290] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:32.534534 23953 leader_election.cc:290] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:32.534662 24022 leader_election.cc:290] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 20 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:32.536163 18363 raft_consensus.cc:1240] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 17. Current term is 18. Ops: []
I20260709 12:23:32.541380 23739 leader_election.cc:290] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:32.541473 24154 raft_consensus.cc:493] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:23:32.541810 23970 leader_election.cc:290] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:32.542109 24154 raft_consensus.cc:515] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:32.546376 23838 leader_election.cc:290] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:32.546566 24154 leader_election.cc:290] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:32.549517 18366 raft_consensus.cc:1240] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 19. Current term is 20. Ops: []
I20260709 12:23:32.531700 18364 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59886 (request call id 1711) took 1706 ms. Trace:
I20260709 12:23:32.550127 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:32.494670 17758 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36572 (request call id 1096) took 1631 ms. Trace:
I20260709 12:23:32.409368 24017 raft_consensus.cc:515] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:32.553066 17760 raft_consensus.cc:2410] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 15 because replica has last-logged OpId of term: 14 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 9.
I20260709 12:23:32.553062 17759 raft_consensus.cc:2468] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 16.
I20260709 12:23:32.513799 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:32.551990 17758 rpcz_store.cc:276] 0709 12:23:30.862987 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:30.879746 (+ 16759us) service_pool.cc:224] Handling call
0709 12:23:32.494446 (+1614700us) spinlock_profiling.cc:243] Waited 1.25 s on lock 0x7b54003f2854. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:23:32.494590 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2868398464}
I20260709 12:23:32.550659 18364 rpcz_store.cc:276] 0709 12:23:30.824918 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:30.889394 (+ 64476us) service_pool.cc:224] Handling call
0709 12:23:32.531484 (+1642090us) spinlock_profiling.cc:243] Waited 1.42 s on lock 0x7b5400305554. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:23:32.531646 (+   162us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3264463616}
I20260709 12:23:32.471081 24144 raft_consensus.cc:493] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:32.556814 17837 raft_consensus.cc:1240] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 17. Current term is 18. Ops: []
W20260709 12:23:32.559685 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 47484.3us
I20260709 12:23:32.550171 23906 raft_consensus.cc:493] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:32.569787 17729 leader_election.cc:304] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
W20260709 12:23:32.571689 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:32.572358 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 18 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:32.573395 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:32.562637 24059 leader_election.cc:290] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:32.575151 24056 raft_consensus.cc:493] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:32.577839 18363 raft_consensus.cc:1275] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 13 index: 11. Preceding OpId from leader: term: 14 index: 12. (index mismatch)
I20260709 12:23:32.564153 23976 leader_election.cc:290] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:23:32.579237 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 145698us
I20260709 12:23:32.568439 24087 raft_consensus.cc:2749] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:23:32.571719 24009 raft_consensus.cc:2749] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:23:32.586005 17802 leader_election.cc:304] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:32.587275 17759 raft_consensus.cc:2410] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for term 18 because replica has last-logged OpId of term: 17 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 8.
I20260709 12:23:32.590284 18367 raft_consensus.cc:2393] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 16: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:23:32.588301 23992 consensus_queue.cc:1059] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 14.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.002s
I20260709 12:23:32.591233 24139 consensus_queue.cc:1048] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:23:32.573968 23906 raft_consensus.cc:515] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:23:32.593216 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:32.596518 23976 raft_consensus.cc:2749] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:23:32.589073 17763 raft_consensus.cc:2468] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 16.
I20260709 12:23:32.580943 24037 raft_consensus.cc:493] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:32.602510 24037 raft_consensus.cc:515] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:32.603375 18364 raft_consensus.cc:1275] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 13 index: 12. Preceding OpId from leader: term: 14 index: 12. (term mismatch)
I20260709 12:23:32.603858 18364 pending_rounds.cc:85] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da: Aborting all ops after (but not including) 11
I20260709 12:23:32.604159 18364 pending_rounds.cc:107] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da: Aborting uncommitted NO_OP operation due to leader change: 13.12
I20260709 12:23:32.604465 18364 raft_consensus.cc:2889] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:32.605166 24037 leader_election.cc:290] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:32.611742 17728 leader_election.cc:304] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:32.613478 17802 leader_election.cc:304] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:32.616508 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:32.617093 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 16 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:32.617700 18374 raft_consensus.cc:2393] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 15: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:23:32.618168 17758 raft_consensus.cc:2410] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for term 16 because replica has last-logged OpId of term: 14 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 10.
W20260709 12:23:32.621831 17757 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38786 (request call id 3304) took 1866 ms (client timeout 2202 ms). Trace:
I20260709 12:23:32.622116 23997 leader_election.cc:290] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 17 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:23:32.622272 17757 rpcz_store.cc:269] 0709 12:23:30.754974 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:30.755737 (+   763us) service_pool.cc:224] Handling call
0709 12:23:32.621616 (+1865879us) spinlock_profiling.cc:243] Waited 1.65 s on lock 0x7b54003bd3d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:23:32.621774 (+   158us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3793503616}
I20260709 12:23:32.625831 17757 raft_consensus.cc:1240] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 16. Current term is 17. Ops: []
I20260709 12:23:32.637126 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:32.637699 17756 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38786 (request call id 3299) took 2032 ms. Trace:
I20260709 12:23:32.638192 17831 raft_consensus.cc:2393] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 15: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:23:32.638564 18371 raft_consensus.cc:1240] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 15. Current term is 16. Ops: []
I20260709 12:23:32.585285 24032 raft_consensus.cc:493] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:23:32.637190 24022 raft_consensus.cc:493] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
W20260709 12:23:32.639207 17764 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38786 (request call id 3308) took 1860 ms (client timeout 1848 ms). Trace:
W20260709 12:23:32.643730 17764 rpcz_store.cc:269] 0709 12:23:30.779163 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:30.779311 (+   148us) service_pool.cc:224] Handling call
0709 12:23:32.638994 (+1859683us) spinlock_profiling.cc:243] Waited 1.49 s on lock 0x7b54002b5a54. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:23:32.639113 (+   119us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3426165120}
I20260709 12:23:32.643951 24022 raft_consensus.cc:515] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:32.645190 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:32.645838 17764 raft_consensus.cc:3060] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:32.647377 24022 leader_election.cc:290] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:32.628300 24139 raft_consensus.cc:493] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:32.648454 24139 raft_consensus.cc:515] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:32.649252 18362 raft_consensus.cc:1240] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 12. Current term is 13. Ops: []
I20260709 12:23:32.650712 24139 leader_election.cc:290] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:32.650951 17757 raft_consensus.cc:1240] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 12. Current term is 13. Ops: []
I20260709 12:23:32.631493 23740 consensus_queue.cc:1059] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 17.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.020s
I20260709 12:23:32.651902 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:32.652007 24087 raft_consensus.cc:493] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:32.652575 18361 raft_consensus.cc:2468] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 12.
I20260709 12:23:32.591171 24056 raft_consensus.cc:515] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:32.611474 23838 consensus_queue.cc:1048] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.047s
I20260709 12:23:32.643175 18368 raft_consensus.cc:1240] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 17. Current term is 18. Ops: []
I20260709 12:23:32.642911 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 13 candidate_status { last_received { term: 11 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:32.656230 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 15 candidate_status { last_received { term: 14 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:32.674650 17836 raft_consensus.cc:2468] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 14.
I20260709 12:23:32.675067 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:32.656386 23739 raft_consensus.cc:2804] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:23:32.676069 24154 consensus_queue.cc:1048] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 12:23:32.676393 23739 raft_consensus.cc:697] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 18 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:23:32.656700 24061 raft_consensus.cc:2749] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
W20260709 12:23:32.677227 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 66475.5us
I20260709 12:23:32.678013 23739 consensus_queue.cc:237] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.12, Last appended by leader: 12, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:32.638139 17756 rpcz_store.cc:276] 0709 12:23:30.605223 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:30.605335 (+   112us) service_pool.cc:224] Handling call
0709 12:23:30.605611 (+   276us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:32.386000 (+1780389us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:32.637411 (+251411us) spinlock_profiling.cc:243] Waited 1.71 s on lock 0x7b5400415ad4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:23:32.637643 (+   232us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3924408192}
W20260709 12:23:32.679793 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 66733.1us
I20260709 12:23:32.664168 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:32.639870 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 13 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:32.658879 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 20 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:32.664260 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:32.669184 18365 raft_consensus.cc:3060] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:32.664937 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:32.664594 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:32.671218 17831 raft_consensus.cc:1275] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 12 index: 10. (index mismatch)
I20260709 12:23:32.690316 18364 raft_consensus.cc:1240] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 15. Current term is 16. Ops: [15.12-15.12]
I20260709 12:23:32.661257 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:32.589910 24028 raft_consensus.cc:515] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:32.657629 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:32.679152 17652 leader_election.cc:304] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:32.677644 17803 leader_election.cc:304] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:23:32.661599 23905 raft_consensus.cc:1064] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b: attempting to promote NON_VOTER 334a4e1f59944544b3b3ff36616b57da to VOTER
I20260709 12:23:32.665573 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:32.658408 24059 raft_consensus.cc:1064] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b: attempting to promote NON_VOTER 334a4e1f59944544b3b3ff36616b57da to VOTER
I20260709 12:23:32.664914 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:32.667227 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:32.666378 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:32.588073 24144 raft_consensus.cc:515] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:32.708823 17687 raft_consensus.cc:2468] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 10.
I20260709 12:23:32.662626 23740 raft_consensus.cc:3055] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:23:32.706266 17688 raft_consensus.cc:2468] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 14.
I20260709 12:23:32.708537 17834 raft_consensus.cc:2468] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 15.
I20260709 12:23:32.657452 23992 raft_consensus.cc:493] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:32.658385 24017 leader_election.cc:290] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:32.657166 24056 leader_election.cc:290] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:32.667579 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 18 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:32.713240 18374 raft_consensus.cc:1275] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 13 index: 11. Preceding OpId from leader: term: 14 index: 11. (term mismatch)
I20260709 12:23:32.722982 18374 pending_rounds.cc:85] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da: Aborting all ops after (but not including) 10
I20260709 12:23:32.723245 18374 pending_rounds.cc:107] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da: Aborting uncommitted NO_OP operation due to leader change: 13.11
I20260709 12:23:32.711476 17838 raft_consensus.cc:2468] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 16.
I20260709 12:23:32.723459 18371 raft_consensus.cc:1240] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 16. Current term is 17. Ops: [16.12-16.12]
I20260709 12:23:32.657627 23651 raft_consensus.cc:493] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:32.715144 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:32.709283 24037 raft_consensus.cc:2749] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:23:32.720288 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:32.727396 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:23:32.720865 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 135061us
I20260709 12:23:32.721524 17762 raft_consensus.cc:2368] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 18. Current term is 19.
I20260709 12:23:32.722350 17686 raft_consensus.cc:3055] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:23:32.660885 24148 raft_consensus.cc:3055] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:23:32.736713 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:32.712090 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:32.723467 18374 raft_consensus.cc:2889] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:32.718070 17681 raft_consensus.cc:2468] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 17.
I20260709 12:23:32.716087 24061 raft_consensus.cc:2804] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:23:32.641685 24135 raft_consensus.cc:515] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:32.729249 17686 raft_consensus.cc:740] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:23:32.730137 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:32.654314 24146 consensus_queue.cc:1048] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.074s
I20260709 12:23:32.727377 23928 raft_consensus.cc:2804] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:23:32.757702 23928 raft_consensus.cc:697] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 19 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:23:32.758818 23928 consensus_queue.cc:237] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 17.10, Last appended by leader: 10, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:32.759817 24135 leader_election.cc:290] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 14 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:32.673086 24183 consensus_queue.cc:1059] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.064s
I20260709 12:23:32.761354 24183 raft_consensus.cc:3055] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:23:32.761834 24183 raft_consensus.cc:740] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:23:32.761970 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:32.762769 17760 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 14.
I20260709 12:23:32.762874 24183 consensus_queue.cc:260] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 17.11, Last appended by leader: 11, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:32.677618 24182 consensus_queue.cc:1059] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.060s
I20260709 12:23:32.763729 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:32.764415 24182 raft_consensus.cc:3055] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:23:32.764791 24182 raft_consensus.cc:740] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 17 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:23:32.765017 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:32.765991 24182 consensus_queue.cc:260] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 17.15, Last appended by leader: 15, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:32.766443 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:32.767138 17760 raft_consensus.cc:2468] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 13.
I20260709 12:23:32.767335 24182 raft_consensus.cc:3060] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:32.652884 24087 raft_consensus.cc:515] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:32.769585 24087 leader_election.cc:290] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:32.769814 17652 leader_election.cc:304] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:32.656522 24032 raft_consensus.cc:515] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:32.718813 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:32.738757 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
W20260709 12:23:32.740294 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:32.737979 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:32.727392 17832 raft_consensus.cc:2468] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 17.
I20260709 12:23:32.745015 17836 raft_consensus.cc:1240] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 16. Current term is 17. Ops: []
I20260709 12:23:32.715999 23740 raft_consensus.cc:740] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 17 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:23:32.740321 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:32.737362 17687 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 14.
I20260709 12:23:32.740437 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:32.747282 17682 raft_consensus.cc:2468] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 13.
I20260709 12:23:32.740453 17756 raft_consensus.cc:2468] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 14.
I20260709 12:23:32.730197 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:32.790807 17680 raft_consensus.cc:2468] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 17.
I20260709 12:23:32.749568 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 15 candidate_status { last_received { term: 14 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:32.708395 17837 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d"
dest_uuid: "e063541542a94969898868bf6eb5ce1b"
mode: GRACEFUL
new_leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
 from {username='slave'} at 127.0.0.1:59820
I20260709 12:23:32.750097 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:32.750237 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:32.750881 17686 consensus_queue.cc:260] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.10, Last appended by leader: 10, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:32.751760 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:32.752460 17728 leader_election.cc:304] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:23:32.761277 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:32.790627 17729 leader_election.cc:304] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:32.773546 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:32.774068 23905 consensus_queue.cc:237] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 17.9, Last appended by leader: 8, Current term: 17, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:32.776829 18333 leader_election.cc:304] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:32.732049 23992 raft_consensus.cc:515] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:32.746007 24061 raft_consensus.cc:493] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:32.736289 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 22 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:32.748530 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:32.737062 24148 raft_consensus.cc:740] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 14 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:23:32.731534 24131 raft_consensus.cc:515] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:32.738440 23651 raft_consensus.cc:515] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:32.790318 17835 raft_consensus.cc:2468] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 12.
I20260709 12:23:32.849176 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:32.870198 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:32.875850 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:32.849252 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 15 candidate_status { last_received { term: 14 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:32.865804 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:32.908437 17829 raft_consensus.cc:3060] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:32.791726 17834 raft_consensus.cc:3060] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:32.913887 17764 raft_consensus.cc:2468] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 19.
I20260709 12:23:32.832791 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 15 candidate_status { last_received { term: 14 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:32.912133 17760 raft_consensus.cc:2468] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 15.
I20260709 12:23:32.856850 17684 raft_consensus.cc:3060] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:32.864833 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 19 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:32.912361 17763 raft_consensus.cc:2393] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 13: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:23:32.916402 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:32.911397 17831 raft_consensus.cc:2468] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 14.
I20260709 12:23:32.918349 18366 raft_consensus.cc:1240] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 18. Current term is 19. Ops: []
I20260709 12:23:32.777863 24144 leader_election.cc:290] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:32.918231 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:32.915257 17837 raft_consensus.cc:606] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Received request to transfer leadership to TS e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:32.916592 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:32.922784 17837 raft_consensus.cc:614] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:23:32.917433 18361 raft_consensus.cc:2468] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 20.
W20260709 12:23:32.923415 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 146686us
W20260709 12:23:32.925438 17804 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:32.928268 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 137743us
I20260709 12:23:32.929639 24111 raft_consensus.cc:2804] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:23:32.929787 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 14 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:32.930485 24111 raft_consensus.cc:697] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 15 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:23:32.932934 24179 raft_consensus.cc:493] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:32.933485 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 181006us
I20260709 12:23:32.933995 24102 leader_election.cc:290] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:32.934692 23740 consensus_queue.cc:260] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 17.15, Last appended by leader: 15, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:32.935333 24009 raft_consensus.cc:2804] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:23:32.936177 24009 raft_consensus.cc:697] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 17 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:23:32.937836 23976 raft_consensus.cc:2804] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:23:32.937249 24009 consensus_queue.cc:237] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:32.938352 23976 raft_consensus.cc:493] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
W20260709 12:23:32.938742 24181 log.cc:927] Time spent T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.312s	user 0.003s	sys 0.000s
I20260709 12:23:32.938838 23976 raft_consensus.cc:3060] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:32.939173 23970 consensus_queue.cc:1059] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 16.11, Next index: 12, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 12:23:32.940351 24155 raft_consensus.cc:493] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:32.940461 23998 raft_consensus.cc:3055] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:23:32.940930 23998 raft_consensus.cc:740] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:23:32.940877 24155 raft_consensus.cc:515] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:32.941674 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:32.942013 23998 consensus_queue.cc:260] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.11, Last appended by leader: 11, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:32.946099 17729 leader_election.cc:304] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
W20260709 12:23:32.948634 17761 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38786 (request call id 3296) took 2351 ms (client timeout 1831 ms). Trace:
W20260709 12:23:32.949085 17761 rpcz_store.cc:269] 0709 12:23:30.597060 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:30.597188 (+   128us) service_pool.cc:224] Handling call
0709 12:23:32.948381 (+2351193us) spinlock_profiling.cc:243] Waited 1.62 s on lock 0x7b54003e62d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:23:32.948545 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3732896000}
I20260709 12:23:32.950213 17687 raft_consensus.cc:1275] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 17 index: 11. Preceding OpId from leader: term: 18 index: 12. (index mismatch)
I20260709 12:23:32.950326 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:32.951145 17761 raft_consensus.cc:2468] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 19.
I20260709 12:23:32.951761 17687 raft_consensus.cc:1240] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 17. Current term is 18. Ops: [17.13-17.13]
I20260709 12:23:32.952165 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:32.953737 17729 leader_election.cc:304] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:32.956127 24032 leader_election.cc:290] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:23:32.960041 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 190322us
I20260709 12:23:32.960253 24039 raft_consensus.cc:2749] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:23:32.960902 17652 leader_election.cc:304] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:32.961846 17830 raft_consensus.cc:2468] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 15.
I20260709 12:23:32.923784 17764 raft_consensus.cc:2468] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 13.
I20260709 12:23:32.933691 24179 raft_consensus.cc:515] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:32.963797 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 13 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:32.965234 17686 raft_consensus.cc:3060] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:32.902810 24037 raft_consensus.cc:493] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:32.919872 24184 log.cc:927] Time spent T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.142s	user 0.000s	sys 0.001s
I20260709 12:23:32.977959 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:32.826146 23931 raft_consensus.cc:2749] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:23:32.876626 24183 raft_consensus.cc:3060] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:32.901023 23739 raft_consensus.cc:493] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:32.970500 17802 leader_election.cc:304] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:32.973483 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:32.974767 17727 leader_election.cc:304] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:32.979893 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:32.918958 24061 raft_consensus.cc:3060] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:32.981519 17756 raft_consensus.cc:2468] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 15.
I20260709 12:23:32.851011 24151 raft_consensus.cc:515] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:32.980036 17757 raft_consensus.cc:2468] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 14.
I20260709 12:23:32.974340 17832 raft_consensus.cc:2468] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 17.
I20260709 12:23:32.984349 17834 raft_consensus.cc:2468] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 15.
I20260709 12:23:32.986312 17829 raft_consensus.cc:2410] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for term 16 because replica has last-logged OpId of term: 15 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 11.
I20260709 12:23:32.988283 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:33.013875 18333 leader_election.cc:304] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:33.019747 17835 raft_consensus.cc:2393] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 19: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:23:32.967330 23998 raft_consensus.cc:3060] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:33.020538 18368 raft_consensus.cc:2393] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 13: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:23:33.021587 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:33.022735 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:33.023767 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:33.025311 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:33.020730 24096 raft_consensus.cc:2749] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:23:33.023448 17832 raft_consensus.cc:2468] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 16.
I20260709 12:23:33.029611 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:23:33.031399 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 70589.7us
I20260709 12:23:32.989830 24088 raft_consensus.cc:2749] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:23:32.984418 24116 raft_consensus.cc:493] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:33.041766 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:33.042732 17727 leader_election.cc:304] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:33.043387 17652 leader_election.cc:304] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:33.038925 17763 raft_consensus.cc:2393] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 13: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:23:33.035681 18367 raft_consensus.cc:2468] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 15.
I20260709 12:23:33.018690 24017 raft_consensus.cc:2804] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:23:33.048139 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:33.045306 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:33.046175 17682 raft_consensus.cc:2468] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 18.
I20260709 12:23:33.046851 18365 raft_consensus.cc:2468] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 13.
I20260709 12:23:33.050675 17688 raft_consensus.cc:2468] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 14.
I20260709 12:23:32.982028 24037 raft_consensus.cc:515] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:33.056926 17838 raft_consensus.cc:2468] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 19.
I20260709 12:23:33.027724 24135 raft_consensus.cc:2804] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:23:33.027441 24141 raft_consensus.cc:2749] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:23:33.061056 17757 raft_consensus.cc:1240] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 17. Current term is 18. Ops: []
I20260709 12:23:33.061190 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:33.064610 18365 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 14.
I20260709 12:23:33.065102 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:33.063474 23970 raft_consensus.cc:2804] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:23:33.066356 23970 raft_consensus.cc:493] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:33.062217 24088 raft_consensus.cc:2749] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:23:33.066783 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:33.067414 17832 raft_consensus.cc:3060] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:33.071041 17727 leader_election.cc:304] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:33.055406 24017 raft_consensus.cc:493] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:23:33.072221 17686 raft_consensus.cc:1275] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 15 index: 10. Preceding OpId from leader: term: 16 index: 10. (term mismatch)
I20260709 12:23:33.072279 24017 raft_consensus.cc:3060] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:33.072700 17686 pending_rounds.cc:85] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3: Aborting all ops after (but not including) 9
I20260709 12:23:33.073081 17686 pending_rounds.cc:107] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3: Aborting uncommitted NO_OP operation due to leader change: 15.10
I20260709 12:23:33.073323 17686 raft_consensus.cc:2889] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:33.075206 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:33.077534 24151 leader_election.cc:290] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 13 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:33.079013 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:33.079679 17686 raft_consensus.cc:2468] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 18.
I20260709 12:23:32.931777 24111 consensus_queue.cc:237] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.12, Last appended by leader: 12, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:33.080610 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:33.081511 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:33.081863 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:33.083006 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:33.082852 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:33.083721 18366 raft_consensus.cc:2468] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 12.
I20260709 12:23:33.084228 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
W20260709 12:23:33.084729 24189 log.cc:927] Time spent T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.140s	user 0.002s	sys 0.000s
I20260709 12:23:33.084920 18364 raft_consensus.cc:3060] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:33.085954 18374 raft_consensus.cc:1275] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 16 index: 12. Preceding OpId from leader: term: 17 index: 12. (term mismatch)
I20260709 12:23:33.082705 24116 raft_consensus.cc:515] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:33.086333 18374 pending_rounds.cc:85] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da: Aborting all ops after (but not including) 11
I20260709 12:23:33.086694 18374 pending_rounds.cc:107] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da: Aborting uncommitted NO_OP operation due to leader change: 16.12
I20260709 12:23:33.086992 18374 raft_consensus.cc:2889] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:33.088258 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:33.082706 18371 raft_consensus.cc:2468] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 18.
I20260709 12:23:33.089133 18362 raft_consensus.cc:2393] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 19: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:23:33.088539 24116 leader_election.cc:290] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:33.090070 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:33.090617 17835 raft_consensus.cc:3060] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Advancing to term 19
I20260709 12:23:33.090351 17686 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 14.
I20260709 12:23:33.066272 17833 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 13.
I20260709 12:23:33.092200 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:33.092868 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:33.066983 23970 raft_consensus.cc:3060] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:33.097481 17729 leader_election.cc:304] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:33.100073 17729 leader_election.cc:304] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:33.049052 24028 leader_election.cc:290] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 19 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:33.065681 17683 raft_consensus.cc:2376] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 14. Re-sending same reply.
I20260709 12:23:33.102747 17802 leader_election.cc:304] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
W20260709 12:23:33.087000 24186 log.cc:927] Time spent T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.154s	user 0.001s	sys 0.000s
I20260709 12:23:33.106990 18333 leader_election.cc:304] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:33.107856 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:33.101891 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:33.112671 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:33.072307 24039 raft_consensus.cc:2804] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:23:33.064606 24135 raft_consensus.cc:493] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:33.113073 17729 leader_election.cc:304] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:33.117825 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:33.122210 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:33.116389 24039 raft_consensus.cc:493] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:23:33.132030 24039 raft_consensus.cc:3060] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:33.132613 18333 leader_election.cc:304] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:23:33.133224 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 16 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:33.106693 24032 raft_consensus.cc:2749] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:23:33.104089 23997 raft_consensus.cc:2804] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:23:33.122879 17833 raft_consensus.cc:2468] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 16.
I20260709 12:23:33.134333 17830 raft_consensus.cc:2410] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for term 16 because replica has last-logged OpId of term: 14 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 10.
I20260709 12:23:33.129549 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:33.146821 23997 raft_consensus.cc:493] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:33.177347 17685 raft_consensus.cc:2468] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 19.
I20260709 12:23:33.177742 23997 raft_consensus.cc:3060] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:33.144567 24155 leader_election.cc:290] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:33.178546 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:33.156184 23906 leader_election.cc:290] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:33.150556 24102 raft_consensus.cc:2804] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:23:33.180354 24102 raft_consensus.cc:493] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:23:33.180783 17685 raft_consensus.cc:1275] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 15 index: 10. (index mismatch)
I20260709 12:23:33.180838 24102 raft_consensus.cc:3060] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:33.182793 17680 raft_consensus.cc:1275] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 13 index: 11. Preceding OpId from leader: term: 14 index: 12. (index mismatch)
I20260709 12:23:33.178546 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:33.134204 17835 raft_consensus.cc:2468] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 19.
I20260709 12:23:33.183714 17688 raft_consensus.cc:2468] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 18.
I20260709 12:23:33.116271 23838 raft_consensus.cc:493] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:33.114776 24116 consensus_queue.cc:1048] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.031s
I20260709 12:23:33.129750 24051 raft_consensus.cc:2749] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:23:33.122665 24028 raft_consensus.cc:2804] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:23:33.138419 18364 raft_consensus.cc:2468] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 17.
I20260709 12:23:33.205163 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:33.059286 23739 raft_consensus.cc:515] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:33.142707 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 99343.9us
I20260709 12:23:33.190632 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 19 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:33.193569 17686 raft_consensus.cc:2468] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 17.
I20260709 12:23:33.197824 17685 raft_consensus.cc:1275] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 16 index: 9. Preceding OpId from leader: term: 18 index: 10. (index mismatch)
I20260709 12:23:33.141306 24151 raft_consensus.cc:2804] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:23:33.213582 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:33.188186 18333 leader_election.cc:304] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:33.119519 24135 raft_consensus.cc:3060] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:23:33.212137 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 59799.7us
I20260709 12:23:33.212818 17686 raft_consensus.cc:1275] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 15 index: 11. Preceding OpId from leader: term: 17 index: 12. (index mismatch)
I20260709 12:23:33.199506 17832 raft_consensus.cc:2468] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 17.
I20260709 12:23:33.220157 17832 raft_consensus.cc:1240] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 18 LEADER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 17. Current term is 18. Ops: [17.13-17.13]
W20260709 12:23:33.217439 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 320197us
W20260709 12:23:33.200253 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 82717.5us
I20260709 12:23:33.221031 18333 leader_election.cc:304] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:33.221951 17830 raft_consensus.cc:1275] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 17 index: 11. Preceding OpId from leader: term: 18 index: 12. (index mismatch)
I20260709 12:23:33.224277 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:33.223110 17729 leader_election.cc:304] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:33.213032 17802 leader_election.cc:304] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:23:33.207669 24028 raft_consensus.cc:493] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:33.227342 18333 leader_election.cc:304] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:33.113619 24148 consensus_queue.cc:260] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.9, Last appended by leader: 9, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:33.232712 17801 leader_election.cc:304] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:23:33.234033 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:23:33.235445 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:33.241896 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:33.245743 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:33.236117 24051 raft_consensus.cc:2804] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:23:33.252609 24148 raft_consensus.cc:3060] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:33.253058 24051 raft_consensus.cc:493] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:33.230304 24199 consensus_queue.cc:1048] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.027s
I20260709 12:23:33.259667 17836 raft_consensus.cc:2410] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for term 19 because replica has last-logged OpId of term: 17 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 14.
I20260709 12:23:33.261444 17829 raft_consensus.cc:1275] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 17 index: 15. Preceding OpId from leader: term: 18 index: 15. (term mismatch)
I20260709 12:23:33.261799 17829 pending_rounds.cc:85] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b: Aborting all ops after (but not including) 14
I20260709 12:23:33.262694 17829 pending_rounds.cc:107] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b: Aborting uncommitted NO_OP operation due to leader change: 17.15
I20260709 12:23:33.263304 17829 raft_consensus.cc:2889] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:33.262336 23976 raft_consensus.cc:515] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:23:33.265815 23739 leader_election.cc:290] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:33.261775 17833 raft_consensus.cc:2468] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 16.
I20260709 12:23:33.253544 24051 raft_consensus.cc:3060] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:33.267299 23651 leader_election.cc:290] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:33.269068 17801 leader_election.cc:304] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:23:33.225055 24111 raft_consensus.cc:2804] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:23:33.240202 23906 raft_consensus.cc:2804] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Leader election won for term 17
W20260709 12:23:33.252807 24190 log.cc:927] Time spent T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.195s	user 0.000s	sys 0.001s
I20260709 12:23:33.280309 23992 leader_election.cc:290] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:33.236474 24028 raft_consensus.cc:3060] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:33.242758 17689 raft_consensus.cc:2468] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 16.
I20260709 12:23:33.225216 24131 leader_election.cc:290] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 14 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:33.285182 17684 raft_consensus.cc:2468] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 18.
I20260709 12:23:33.228540 23953 raft_consensus.cc:2804] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Leader pre-election won for term 15
W20260709 12:23:33.310978 17728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:33.315117 23953 raft_consensus.cc:493] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:33.315740 23953 raft_consensus.cc:3060] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:33.317171 17687 raft_consensus.cc:1240] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 17. Current term is 18. Ops: []
I20260709 12:23:33.320390 24131 raft_consensus.cc:2804] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:23:33.320863 24131 raft_consensus.cc:493] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:33.321399 24131 raft_consensus.cc:3060] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:33.280854 24177 raft_consensus.cc:2804] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:23:33.220407 23838 raft_consensus.cc:515] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:33.313829 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 44911.2us
I20260709 12:23:33.310313 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:33.329234 17764 raft_consensus.cc:1240] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 17. Current term is 18. Ops: []
I20260709 12:23:33.329926 24177 raft_consensus.cc:697] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 16 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:23:33.300518 24111 raft_consensus.cc:493] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:33.326620 23906 raft_consensus.cc:697] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 17 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:23:33.334194 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:33.335039 17680 raft_consensus.cc:2468] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 17.
I20260709 12:23:33.335107 23906 consensus_queue.cc:237] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.12, Last appended by leader: 12, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:33.335872 23740 raft_consensus.cc:3060] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:33.336782 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:33.337507 17684 raft_consensus.cc:2468] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 17.
I20260709 12:23:33.334211 24111 raft_consensus.cc:3060] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:33.334221 24177 consensus_queue.cc:237] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.9, Last appended by leader: 9, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:33.342129 23838 leader_election.cc:290] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:23:33.342360 17802 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:33.328534 23992 raft_consensus.cc:2804] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:23:33.344123 23992 raft_consensus.cc:493] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:33.344513 23992 raft_consensus.cc:3060] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:33.344635 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 19 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:33.345681 17680 raft_consensus.cc:2468] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 18.
I20260709 12:23:33.346737 17688 raft_consensus.cc:1275] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 18 index: 13. (index mismatch)
I20260709 12:23:33.348351 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:33.348857 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:33.349334 17727 leader_election.cc:304] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:33.349818 17684 raft_consensus.cc:2468] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 13.
I20260709 12:23:33.352082 24199 raft_consensus.cc:2804] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:23:33.352519 24199 raft_consensus.cc:493] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:33.352979 24199 raft_consensus.cc:3060] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:33.360018 17801 leader_election.cc:304] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:23:33.361370 17801 leader_election.cc:304] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:33.362671 24141 raft_consensus.cc:2804] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:23:33.363051 24141 raft_consensus.cc:493] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:23:33.363409 24141 raft_consensus.cc:3060] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:33.363497 23739 raft_consensus.cc:2804] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:23:33.364248 17801 leader_election.cc:304] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:33.365561 17801 leader_election.cc:304] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:23:33.365823 24116 raft_consensus.cc:2749] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:23:33.367301 24182 raft_consensus.cc:2804] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:23:33.367429 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:33.367740 24182 raft_consensus.cc:493] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:33.368198 18368 raft_consensus.cc:2468] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 13.
I20260709 12:23:33.369357 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:33.370034 18367 raft_consensus.cc:2468] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 17.
I20260709 12:23:33.368224 24182 raft_consensus.cc:3060] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:33.364393 23739 raft_consensus.cc:493] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:23:33.383864 24017 raft_consensus.cc:515] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:33.355724 24039 raft_consensus.cc:515] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:33.396843 17803 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:33.398343 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:23:33.417361 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 51857.2us
I20260709 12:23:33.453668 17801 leader_election.cc:304] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:33.454615 24098 raft_consensus.cc:2749] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
W20260709 12:23:33.418911 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 45993.7us
I20260709 12:23:33.379526 17727 leader_election.cc:304] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:33.455363 18334 leader_election.cc:304] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b
W20260709 12:23:33.424255 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 197738us
W20260709 12:23:33.455734 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 103928us
I20260709 12:23:33.456352 18333 leader_election.cc:304] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:33.456604 17727 leader_election.cc:304] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:33.457154 24179 leader_election.cc:290] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:33.457573 18333 leader_election.cc:304] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:33.457789 24179 raft_consensus.cc:2804] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:23:33.442595 17683 raft_consensus.cc:2393] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 14: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:23:33.458316 24179 raft_consensus.cc:493] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:33.420680 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:33.458742 24179 raft_consensus.cc:3060] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:33.459293 18368 raft_consensus.cc:2393] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 14: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:23:33.452852 17689 raft_consensus.cc:2468] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 16.
I20260709 12:23:33.446280 17728 leader_election.cc:304] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:33.455482 17729 leader_election.cc:304] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:33.454936 17801 leader_election.cc:304] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:33.461382 18364 raft_consensus.cc:1240] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 13. Current term is 14. Ops: []
I20260709 12:23:33.463960 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:33.464715 18365 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 14.
W20260709 12:23:33.450194 17654 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:33.466699 17654 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:33.476624 17801 leader_election.cc:304] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:33.475188 17729 leader_election.cc:304] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:33.419205 23905 raft_consensus.cc:493] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:33.478212 17727 leader_election.cc:304] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:33.482285 17728 leader_election.cc:304] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:23:33.483376 23979 raft_consensus.cc:2804] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:23:33.484019 23979 raft_consensus.cc:493] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:33.484375 23979 raft_consensus.cc:3060] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:33.478852 17682 raft_consensus.cc:1275] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 17 index: 9. Preceding OpId from leader: term: 17 index: 10. (index mismatch)
I20260709 12:23:33.442656 23739 raft_consensus.cc:3060] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:33.480473 17755 raft_consensus.cc:1240] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 16. Current term is 17. Ops: []
I20260709 12:23:33.436932 24151 raft_consensus.cc:697] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 18 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
W20260709 12:23:33.452253 24171 log.cc:927] Time spent T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.084s	user 0.001s	sys 0.006s
W20260709 12:23:33.494012 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 178426us
I20260709 12:23:33.479149 24155 raft_consensus.cc:2804] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:23:33.480968 24098 raft_consensus.cc:2804] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:23:33.507046 17689 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0"
dest_uuid: "0da169ff82674dc1947055ce43de68a3"
mode: GRACEFUL
new_leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
 from {username='slave'} at 127.0.0.1:54540
I20260709 12:23:33.486001 23998 raft_consensus.cc:2804] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:23:33.509159 17689 raft_consensus.cc:606] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 13 LEADER]: Received request to transfer leadership to TS e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:33.510200 17801 leader_election.cc:400] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 18 pre-election: Vote denied by peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651) with higher term. Message: Invalid argument: T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 18. Current term is 19.
I20260709 12:23:33.512708 17801 leader_election.cc:403] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 18 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:23:33.461223 24102 raft_consensus.cc:515] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:33.460638 23970 raft_consensus.cc:515] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:33.492835 24056 raft_consensus.cc:3060] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:33.480556 24032 consensus_queue.cc:1048] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.029s
I20260709 12:23:33.463261 23997 raft_consensus.cc:515] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:33.514714 23998 raft_consensus.cc:493] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:33.536726 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:33.536561 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:33.512966 24155 raft_consensus.cc:697] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
W20260709 12:23:33.540879 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 65777us
I20260709 12:23:33.511009 23906 raft_consensus.cc:2804] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:23:33.513087 24009 raft_consensus.cc:2804] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:23:33.546610 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:33.547688 17761 raft_consensus.cc:2393] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 16: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
W20260709 12:23:33.548427 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 70286.1us
I20260709 12:23:33.491153 24144 consensus_queue.cc:1059] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 18.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.043s
I20260709 12:23:33.535331 23998 raft_consensus.cc:3060] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:33.562572 18361 raft_consensus.cc:1240] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 12. Current term is 13. Ops: []
W20260709 12:23:33.558169 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 75990.8us
I20260709 12:23:33.532325 23651 raft_consensus.cc:2749] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Leader pre-election lost for term 18. Reason: Vote denied by peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651) with higher term. Message: Invalid argument: T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 18. Current term is 19.
I20260709 12:23:33.504850 23905 raft_consensus.cc:515] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:23:33.556316 24183 consensus_meta.cc:287] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.563s	user 0.012s	sys 0.000s
W20260709 12:23:33.559167 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 101650us
I20260709 12:23:33.546447 24202 raft_consensus.cc:2804] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Leader pre-election won for term 15
W20260709 12:23:33.561017 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 105739us
I20260709 12:23:33.535483 24098 raft_consensus.cc:697] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 15 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:23:33.535818 24203 raft_consensus.cc:2804] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:23:33.545971 23906 raft_consensus.cc:493] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:23:33.546033 24169 raft_consensus.cc:2804] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:23:33.527647 24182 raft_consensus.cc:515] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:33.553094 24009 raft_consensus.cc:493] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:33.573379 17728 leader_election.cc:304] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:33.581928 17760 raft_consensus.cc:2410] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 18 because replica has last-logged OpId of term: 17 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 10.
I20260709 12:23:33.573570 23976 leader_election.cc:290] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 18 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:33.572377 23838 raft_consensus.cc:2804] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:23:33.583138 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:33.591002 18334 leader_election.cc:304] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:33.536903 23953 raft_consensus.cc:515] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:33.596378 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:33.630030 17830 raft_consensus.cc:3060] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:33.578991 24202 raft_consensus.cc:493] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:33.634651 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:33.565382 24116 raft_consensus.cc:493] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
W20260709 12:23:33.580790 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 74143.4us
I20260709 12:23:33.626173 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:33.626287 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:33.597026 24111 raft_consensus.cc:515] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:33.594769 23906 raft_consensus.cc:3060] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:33.646036 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
W20260709 12:23:33.626719 24061 consensus_meta.cc:287] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.569s	user 0.010s	sys 0.004s
I20260709 12:23:33.596727 24039 leader_election.cc:290] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 17 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:33.598272 23928 raft_consensus.cc:3055] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:23:33.638823 17761 raft_consensus.cc:2393] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 13: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:23:33.649215 17729 leader_election.cc:304] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:33.596431 24017 leader_election.cc:290] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 13 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:33.598953 24169 raft_consensus.cc:493] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:33.653026 17837 raft_consensus.cc:1275] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 15 index: 10. (index mismatch)
I20260709 12:23:33.653028 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:33.627032 24009 raft_consensus.cc:3060] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:33.631734 23838 raft_consensus.cc:493] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:23:33.638651 23651 raft_consensus.cc:493] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:33.627465 24144 consensus_queue.cc:1048] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.030s
I20260709 12:23:33.593236 24206 consensus_queue.cc:1048] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.081s
I20260709 12:23:33.712915 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:33.637712 24202 raft_consensus.cc:3060] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:23:33.724203 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 133280us
W20260709 12:23:33.718683 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 69595.6us
I20260709 12:23:33.591377 24151 consensus_queue.cc:237] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 17.15, Last appended by leader: 15, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:33.715453 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 14 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:33.715417 17761 raft_consensus.cc:1275] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 16 index: 9. Preceding OpId from leader: term: 18 index: 10. (index mismatch)
I20260709 12:23:33.726814 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:33.738535 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:33.646567 24199 raft_consensus.cc:515] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:33.744436 17757 raft_consensus.cc:2468] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 17.
I20260709 12:23:33.645720 24135 raft_consensus.cc:515] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:33.720372 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 80900.2us
I20260709 12:23:33.650967 23992 raft_consensus.cc:515] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:33.755616 17829 raft_consensus.cc:3060] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Advancing to term 14
W20260709 12:23:33.748364 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 111497us
I20260709 12:23:33.639142 24051 raft_consensus.cc:515] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:33.714775 24169 raft_consensus.cc:3060] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:33.668885 24148 raft_consensus.cc:493] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:33.650540 24028 raft_consensus.cc:515] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:33.661985 23928 raft_consensus.cc:740] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:23:33.709192 23838 raft_consensus.cc:3060] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:33.662171 24061 raft_consensus.cc:515] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:33.715329 24214 raft_consensus.cc:2804] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:23:33.782503 24219 consensus_queue.cc:1048] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.010s
I20260709 12:23:33.707190 24116 raft_consensus.cc:515] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:33.768242 17834 raft_consensus.cc:2468] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 18.
I20260709 12:23:33.664278 24177 consensus_queue.cc:1059] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 19.14, Next index: 15, Last known committed idx: 13, Time since last communication: 0.093s
I20260709 12:23:33.743275 24183 raft_consensus.cc:2749] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 12:23:33.645577 24155 consensus_queue.cc:237] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 17.10, Last appended by leader: 10, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:33.653470 24141 raft_consensus.cc:515] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:33.725903 23740 raft_consensus.cc:493] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:33.780128 17762 raft_consensus.cc:2368] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 19. Current term is 20.
I20260709 12:23:33.715536 24146 raft_consensus.cc:493] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:33.756353 24203 raft_consensus.cc:697] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 19 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:23:33.771610 24102 leader_election.cc:290] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 21 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:33.822418 24061 leader_election.cc:290] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 14 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:33.718099 24131 raft_consensus.cc:515] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:33.748904 24098 consensus_queue.cc:237] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:33.835237 24215 consensus_queue.cc:1059] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 14.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:23:33.835983 24211 raft_consensus.cc:2804] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:23:33.841163 23838 raft_consensus.cc:515] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:33.918990 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 434422us
W20260709 12:23:33.843601 24209 log.cc:927] Time spent T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.179s	user 0.000s	sys 0.000s
W20260709 12:23:33.844702 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 279720us
I20260709 12:23:33.738055 23651 raft_consensus.cc:515] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:33.845396 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:33.925037 17761 raft_consensus.cc:2393] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 14: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:23:33.845662 17831 raft_consensus.cc:2368] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 13. Current term is 14.
I20260709 12:23:33.851678 17829 raft_consensus.cc:2468] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 14.
I20260709 12:23:33.866153 17830 raft_consensus.cc:2468] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 13.
I20260709 12:23:33.928052 23838 leader_election.cc:290] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 11 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:23:33.880578 17803 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:33.784677 24214 raft_consensus.cc:697] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:23:33.929744 24214 consensus_queue.cc:237] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:33.890168 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 148400us
W20260709 12:23:33.932377 24216 log.cc:927] Time spent T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.189s	user 0.001s	sys 0.000s
I20260709 12:23:33.897480 17836 raft_consensus.cc:3060] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Advancing to term 14
W20260709 12:23:33.898999 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:33.933000 18334 leader_election.cc:304] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:33.936666 23992 leader_election.cc:290] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 17 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:33.920182 24102 raft_consensus.cc:493] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:23:33.905618 17652 leader_election.cc:304] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:33.844416 24111 leader_election.cc:290] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 14 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:33.792685 24148 raft_consensus.cc:515] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:23:33.842716 24208 log.cc:927] Time spent T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.192s	user 0.000s	sys 0.000s
I20260709 12:23:33.904765 24056 raft_consensus.cc:2749] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:23:33.948592 17835 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37994 (request call id 1216) took 1184 ms. Trace:
I20260709 12:23:33.922830 17728 leader_election.cc:304] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:33.897794 23740 raft_consensus.cc:515] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:33.949237 17835 rpcz_store.cc:276] 0709 12:23:32.764345 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:33.219196 (+454851us) service_pool.cc:224] Handling call
0709 12:23:33.228162 (+  8966us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:33.246755 (+ 18593us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 12:23:33.478349 (+231594us) log.cc:844] Serialized 14 byte log entry
0709 12:23:33.948144 (+469795us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:33.948219 (+    75us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:23:33.948306 (+    87us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:33.948397 (+    91us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:33.948521 (+   124us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":271637,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":40,"raft.queue_time_us":24914,"raft.run_cpu_time_us":140,"raft.run_wall_time_us":140,"spinlock_wait_cycles":1920,"thread_start_us":668676,"threads_started":3,"wal-append.queue_time_us":431613}
I20260709 12:23:33.937615 24102 raft_consensus.cc:515] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:33.924094 23905 leader_election.cc:290] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:33.964217 24183 raft_consensus.cc:2749] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:23:33.946578 23992 raft_consensus.cc:493] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:33.953856 23953 leader_election.cc:290] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 15 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:33.910746 24146 raft_consensus.cc:515] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:33.971787 17654 leader_election.cc:304] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:23:33.994288 24228 raft_consensus.cc:493] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:33.994954 24228 raft_consensus.cc:515] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:33.995980 24146 leader_election.cc:290] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:33.909605 23928 consensus_queue.cc:260] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.11, Last appended by leader: 11, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:33.993046 18333 leader_election.cc:304] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
W20260709 12:23:33.999637 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 66722.2us
I20260709 12:23:34.001130 18334 leader_election.cc:304] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:33.970925 23838 raft_consensus.cc:2804] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:23:34.004854 17728 leader_election.cc:304] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:34.005507 17833 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 13.
W20260709 12:23:33.998435 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 99408.5us
I20260709 12:23:34.009755 24051 leader_election.cc:290] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 19 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:34.010690 23740 leader_election.cc:290] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:34.010882 24028 leader_election.cc:290] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 14 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:23:34.011585 24179 consensus_meta.cc:287] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.552s	user 0.012s	sys 0.000s
I20260709 12:23:34.011847 24141 leader_election.cc:290] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 17 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:34.013346 23651 leader_election.cc:290] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:34.015995 24179 raft_consensus.cc:515] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:34.016814 24131 leader_election.cc:290] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 16 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:23:34.016929 24059 raft_consensus.cc:1068] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b: Unable to promote non-voter 334a4e1f59944544b3b3ff36616b57da: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
  Committed config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }.
  Pending config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:33.998281 23992 raft_consensus.cc:515] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:34.018622 24179 leader_election.cc:290] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 17 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:34.019556 17759 raft_consensus.cc:2393] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 17: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:23:34.020022 24199 leader_election.cc:290] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 20 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:34.020372 24116 leader_election.cc:290] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:33.967684 23931 consensus_queue.cc:1059] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 14.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.039s
I20260709 12:23:34.006307 24228 leader_election.cc:290] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:34.021533 24148 leader_election.cc:290] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:34.009537 24227 consensus_queue.cc:1059] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 13.11, Next index: 12, Last known committed idx: 10, Time since last communication: 0.060s
I20260709 12:23:34.020895 23970 leader_election.cc:290] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 19 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:34.027149 18334 leader_election.cc:304] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:34.024940 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 20 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:33.994601 24236 consensus_queue.cc:1048] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.006s
I20260709 12:23:34.028507 18333 leader_election.cc:304] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:33.987362 24215 raft_consensus.cc:2749] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 12:23:33.971707 24151 consensus_queue.cc:1048] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.038s
I20260709 12:23:34.034248 17761 raft_consensus.cc:2468] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 19.
I20260709 12:23:34.021732 24182 leader_election.cc:290] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 20 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:34.035061 24111 raft_consensus.cc:2804] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:23:34.035275 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:34.035557 24199 raft_consensus.cc:2804] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:23:34.033218 23838 raft_consensus.cc:697] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 13 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:23:34.036198 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:34.050820 17757 raft_consensus.cc:3060] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:34.049190 23953 consensus_queue.cc:1059] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 15.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.005s
I20260709 12:23:34.052192 24102 leader_election.cc:290] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:34.053135 23998 raft_consensus.cc:515] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:34.040499 24235 raft_consensus.cc:2804] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:23:34.021711 23992 leader_election.cc:290] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:33.990278 24183 consensus_queue.cc:1048] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.040s
I20260709 12:23:34.047840 17758 raft_consensus.cc:3060] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:34.048228 18334 leader_election.cc:304] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:23:34.022310 24135 leader_election.cc:290] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 15 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:23:34.048375 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 157248us
W20260709 12:23:34.056346 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 64995.3us
I20260709 12:23:33.970973 24203 consensus_queue.cc:237] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.11, Last appended by leader: 11, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:34.058467 17759 raft_consensus.cc:1275] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 11 index: 8. Preceding OpId from leader: term: 16 index: 10. (index mismatch)
I20260709 12:23:34.068967 17801 leader_election.cc:304] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:34.049079 24111 raft_consensus.cc:493] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:23:34.074533 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 46147.5us
I20260709 12:23:34.076058 18333 leader_election.cc:304] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:23:34.077212 17836 raft_consensus.cc:2468] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 14.
I20260709 12:23:34.041400 24205 consensus_queue.cc:1048] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.017s
I20260709 12:23:34.052203 23953 raft_consensus.cc:3055] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:23:34.078435 17728 leader_election.cc:304] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:34.070138 24102 raft_consensus.cc:2804] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:23:34.080322 17837 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35042 (request call id 2606) took 1369 ms. Trace:
I20260709 12:23:34.083712 17801 leader_election.cc:400] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 19 election: Vote denied by peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651) with higher term. Message: Invalid argument: T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 19. Current term is 20.
I20260709 12:23:34.084450 17801 leader_election.cc:403] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 19 election: Cancelling election due to peer responding with higher term
I20260709 12:23:34.042260 23906 raft_consensus.cc:515] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:34.088253 24234 log.cc:927] Time spent T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.130s	user 0.001s	sys 0.000s
I20260709 12:23:34.088403 17763 raft_consensus.cc:1240] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 18. Current term is 19. Ops: []
I20260709 12:23:34.070060 24199 raft_consensus.cc:697] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 17 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:23:34.089572 23906 leader_election.cc:290] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 19 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:34.056233 24215 raft_consensus.cc:2804] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:23:34.085287 24182 consensus_queue.cc:1048] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.000s
W20260709 12:23:34.085956 24230 log.cc:927] Time spent T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.130s	user 0.001s	sys 0.000s
I20260709 12:23:34.085669 23838 consensus_queue.cc:237] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.10, Last appended by leader: 10, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:34.086818 23739 consensus_meta.cc:287] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.580s	user 0.008s	sys 0.003s
W20260709 12:23:34.087306 24225 log.cc:927] Time spent T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.135s	user 0.001s	sys 0.000s
I20260709 12:23:34.084023 24102 raft_consensus.cc:493] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:34.108094 24102 raft_consensus.cc:3060] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:34.084208 17837 rpcz_store.cc:276] 0709 12:23:32.710592 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:33.144357 (+433765us) service_pool.cc:224] Handling call
0709 12:23:33.183353 (+ 38996us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:33.707316 (+523963us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:34.074437 (+367121us) spinlock_profiling.cc:243] Waited 456 ms on lock 0x7b54002eb654. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:23:34.080232 (+  5795us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1048746368}
I20260709 12:23:34.076247 24111 raft_consensus.cc:3060] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:34.101819 24148 raft_consensus.cc:3060] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:34.101971 24231 raft_consensus.cc:2804] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:23:34.102501 24222 raft_consensus.cc:2764] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Leader pre-election decision vote started in defunct term 16: won
I20260709 12:23:34.103201 17728 leader_election.cc:304] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:34.102442 24237 consensus_queue.cc:1059] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 18.13, Next index: 14, Last known committed idx: 12, Time since last communication: 0.037s
I20260709 12:23:34.089931 24199 consensus_queue.cc:237] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 16.13, Last appended by leader: 13, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:34.083133 23953 raft_consensus.cc:740] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
W20260709 12:23:34.134554 17728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:34.135528 24179 raft_consensus.cc:493] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:23:34.135318 23953 consensus_queue.cc:260] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.13, Last appended by leader: 13, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:34.105098 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:34.137401 17759 raft_consensus.cc:2468] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 18.
I20260709 12:23:34.089686 24202 raft_consensus.cc:515] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:34.106360 23979 consensus_meta.cc:287] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.615s	user 0.010s	sys 0.001s
I20260709 12:23:34.138315 23979 raft_consensus.cc:515] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:34.139724 17801 leader_election.cc:304] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:34.140357 24202 leader_election.cc:290] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 15 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:34.105392 23998 leader_election.cc:290] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 14 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:34.140667 24059 raft_consensus.cc:2804] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:23:34.140775 23979 leader_election.cc:290] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 18 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:34.080760 23992 raft_consensus.cc:2804] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:23:34.091522 24228 raft_consensus.cc:2804] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:23:34.141702 23992 raft_consensus.cc:493] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:34.141783 24228 raft_consensus.cc:493] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:34.136216 24179 raft_consensus.cc:515] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:34.142273 23992 raft_consensus.cc:3060] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:34.121229 18333 leader_election.cc:304] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:34.144321 24179 leader_election.cc:290] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:34.127357 17757 raft_consensus.cc:2468] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 17.
I20260709 12:23:34.141244 24059 raft_consensus.cc:493] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:34.143272 23953 raft_consensus.cc:3060] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:34.142328 24228 raft_consensus.cc:3060] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:34.145609 18333 leader_election.cc:304] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:34.146926 18333 leader_election.cc:304] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:34.148008 24205 raft_consensus.cc:2749] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:23:34.148523 24202 raft_consensus.cc:2749] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:23:34.148814 24205 raft_consensus.cc:2749] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:23:34.149104 17801 leader_election.cc:304] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:34.150157 24116 raft_consensus.cc:2804] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:23:34.151379 24116 raft_consensus.cc:697] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 17 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:23:34.153072 24116 consensus_queue.cc:237] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.12, Last appended by leader: 12, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:34.138628 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:34.090551 24226 raft_consensus.cc:493] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:34.156859 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:34.079412 24155 consensus_queue.cc:1048] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.030s
I20260709 12:23:34.084507 24235 raft_consensus.cc:697] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 17 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:23:34.097873 24215 raft_consensus.cc:493] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:34.121042 23906 raft_consensus.cc:493] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:34.145146 24059 raft_consensus.cc:3060] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:34.171555 17757 raft_consensus.cc:2468] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 17.
I20260709 12:23:34.155640 23970 raft_consensus.cc:2749] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:23:34.155119 24214 raft_consensus.cc:2749] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:23:34.180755 17801 leader_election.cc:304] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:34.158205 24231 raft_consensus.cc:493] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:34.109158 23739 raft_consensus.cc:515] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:34.185096 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:34.154161 24009 raft_consensus.cc:515] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:34.166024 24215 raft_consensus.cc:3060] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Advancing to term 22
W20260709 12:23:34.193279 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 46441us
I20260709 12:23:34.192543 17757 raft_consensus.cc:2468] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 17.
I20260709 12:23:34.193738 17758 raft_consensus.cc:2468] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 20.
W20260709 12:23:34.196386 24194 log.cc:927] Time spent T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 1.052s	user 0.005s	sys 0.000s
I20260709 12:23:34.196468 17831 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37994 (request call id 1222) took 1430 ms. Trace:
I20260709 12:23:34.201478 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 20 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:34.204975 17801 leader_election.cc:304] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:34.205753 18333 leader_election.cc:304] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:34.207862 17831 rpcz_store.cc:276] 0709 12:23:32.766509 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:32.924253 (+157744us) service_pool.cc:224] Handling call
0709 12:23:34.196247 (+1271994us) spinlock_profiling.cc:243] Waited 845 ms on lock 0x7b5400159ad4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:23:34.196389 (+   142us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1944217984}
I20260709 12:23:34.209816 17838 raft_consensus.cc:2468] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 18.
I20260709 12:23:34.187043 24098 raft_consensus.cc:2804] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:23:34.213965 17759 raft_consensus.cc:2368] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 17. Current term is 18.
I20260709 12:23:34.176532 23906 raft_consensus.cc:515] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:34.216823 18333 leader_election.cc:304] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:34.214418 17801 leader_election.cc:304] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:34.215674 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 18 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:34.166448 23931 consensus_queue.cc:1059] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.044s
I20260709 12:23:34.188755 24231 raft_consensus.cc:3060] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:34.221159 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:34.258591 17756 raft_consensus.cc:1240] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 17. Current term is 18. Ops: []
I20260709 12:23:34.261451 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:34.262089 18367 raft_consensus.cc:3060] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Advancing to term 14
I20260709 12:23:34.225636 24141 raft_consensus.cc:2804] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:23:34.216754 24222 raft_consensus.cc:2749] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 12:23:34.282258 24141 raft_consensus.cc:493] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:23:34.282541 24207 log.cc:927] Time spent T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.724s	user 0.002s	sys 0.001s
I20260709 12:23:34.282776 24141 raft_consensus.cc:3060] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:34.257829 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:34.283965 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:34.285005 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:34.285590 24236 raft_consensus.cc:3055] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:23:34.285991 24236 raft_consensus.cc:740] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:23:34.286839 24236 consensus_queue.cc:260] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.12, Last appended by leader: 12, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:34.287137 24215 raft_consensus.cc:515] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:34.226603 24151 raft_consensus.cc:2749] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:23:34.215749 24098 raft_consensus.cc:493] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:34.326812 24098 raft_consensus.cc:3060] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:34.326916 24215 leader_election.cc:290] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 22 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:34.243713 23997 leader_election.cc:290] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 16 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:34.178263 24226 raft_consensus.cc:515] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:34.329349 24226 leader_election.cc:290] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:34.245265 24211 raft_consensus.cc:493] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:23:34.330037 24211 raft_consensus.cc:3060] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:34.333235 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:34.335525 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:34.336457 17832 raft_consensus.cc:2468] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 16.
I20260709 12:23:34.247373 24061 raft_consensus.cc:3055] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:23:34.337913 24061 raft_consensus.cc:740] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 19 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:23:34.339008 24061 consensus_queue.cc:260] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 19.14, Last appended by leader: 14, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:34.341046 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 117657us
I20260709 12:23:34.251245 23906 leader_election.cc:290] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:34.253259 17761 raft_consensus.cc:1240] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 17 LEADER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 16. Current term is 17. Ops: []
I20260709 12:23:34.265139 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:34.268215 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:34.360694 18361 raft_consensus.cc:3060] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:34.268925 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:34.231006 24144 raft_consensus.cc:493] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:23:34.228427 24088 raft_consensus.cc:3055] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:23:34.269761 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:34.364555 17689 raft_consensus.cc:3060] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:34.270905 18371 raft_consensus.cc:1275] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 17 index: 12. (index mismatch)
I20260709 12:23:34.229077 24037 leader_election.cc:290] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:34.273881 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:34.275241 17834 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37994 (request call id 1217) took 1510 ms. Trace:
I20260709 12:23:34.263492 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:34.275827 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:34.226145 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 20 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:23:34.287925 17760 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38786 (request call id 3323) took 1459 ms (client timeout 1918 ms). Trace:
I20260709 12:23:34.289184 24009 leader_election.cc:290] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 18 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:34.291934 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:34.292516 17836 raft_consensus.cc:1240] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 16 LEADER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 15. Current term is 16. Ops: []
I20260709 12:23:34.292655 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:34.375393 17830 raft_consensus.cc:2393] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 14: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:23:34.373245 17758 raft_consensus.cc:2393] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 20: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:23:34.376259 18374 raft_consensus.cc:2393] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 16: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:23:34.376596 17837 raft_consensus.cc:2393] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 19: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:23:34.293099 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 18 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:34.377319 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:34.377841 17829 raft_consensus.cc:2410] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for term 18 because replica has last-logged OpId of term: 16 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 9.
I20260709 12:23:34.217561 23651 raft_consensus.cc:2804] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:23:34.294723 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 13 candidate_status { last_received { term: 11 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:34.295343 24177 raft_consensus.cc:493] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:34.273926 24169 raft_consensus.cc:515] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:34.280896 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 20 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:34.281404 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 14 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:34.297695 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 22 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:34.281404 17755 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36572 (request call id 1123) took 1304 ms. Trace:
I20260709 12:23:34.278249 17587 catalog_manager.cc:5697] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 7 to 13, leader changed from 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) to 334a4e1f59944544b3b3ff36616b57da (127.16.29.68). New cstate: current_term: 13 leader_uuid: "334a4e1f59944544b3b3ff36616b57da" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:23:34.422799 17755 rpcz_store.cc:276] 0709 12:23:32.977127 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:33.059181 (+ 82054us) service_pool.cc:224] Handling call
0709 12:23:33.079921 (+ 20740us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:33.488387 (+408466us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:34.281174 (+792787us) spinlock_profiling.cc:243] Waited 400 ms on lock 0x7b54004150d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:23:34.281331 (+   157us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":920675072}
I20260709 12:23:34.299558 24236 raft_consensus.cc:3060] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:34.300310 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:34.424363 17838 raft_consensus.cc:2468] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 15.
I20260709 12:23:34.228034 23739 leader_election.cc:290] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 18 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:34.304889 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:34.359653 23740 raft_consensus.cc:493] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:34.428674 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:34.311543 24231 raft_consensus.cc:515] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:34.321710 24244 raft_consensus.cc:3055] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:23:34.324076 24235 consensus_queue.cc:237] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.12, Last appended by leader: 12, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:34.233827 23928 raft_consensus.cc:3060] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:34.446694 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:34.447633 17758 raft_consensus.cc:2368] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 18. Current term is 19.
I20260709 12:23:34.448594 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:34.449244 17758 raft_consensus.cc:2468] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 18.
I20260709 12:23:34.451601 24177 raft_consensus.cc:515] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:34.454197 24177 leader_election.cc:290] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:34.249749 17764 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38786 (request call id 3320) took 1467 ms. Trace:
I20260709 12:23:34.456807 17764 rpcz_store.cc:276] 0709 12:23:32.781896 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:32.983467 (+201571us) service_pool.cc:224] Handling call
0709 12:23:32.983915 (+   448us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:33.329703 (+345788us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:34.249468 (+919765us) spinlock_profiling.cc:243] Waited 345 ms on lock 0x7b5400410854. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:23:34.249683 (+   215us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":793692928}
I20260709 12:23:34.458362 17764 raft_consensus.cc:1240] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 16. Current term is 17. Ops: []
I20260709 12:23:34.458547 24111 raft_consensus.cc:515] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:34.456085 17831 raft_consensus.cc:3060] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:34.363853 24151 consensus_queue.cc:1048] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.069s
I20260709 12:23:34.465358 24231 leader_election.cc:290] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 16 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:34.439172 24244 raft_consensus.cc:740] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:23:34.466728 24244 consensus_queue.cc:260] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.13, Last appended by leader: 13, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:34.433744 23651 raft_consensus.cc:697] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 20 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:23:34.468276 24244 raft_consensus.cc:3060] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:34.468636 23651 consensus_queue.cc:237] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.11, Last appended by leader: 11, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:34.469904 17587 catalog_manager.cc:5697] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 14 to 17, leader changed from 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) to 0da169ff82674dc1947055ce43de68a3 (127.16.29.65). New cstate: current_term: 17 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
W20260709 12:23:34.340857 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 370529us
I20260709 12:23:34.472318 17654 leader_election.cc:304] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:23:34.473033 17829 raft_consensus.cc:1240] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 15. Current term is 16. Ops: []
I20260709 12:23:34.472957 24228 raft_consensus.cc:515] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:34.475067 24228 leader_election.cc:290] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 15 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:34.343761 18333 leader_election.cc:304] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:34.475804 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:23:34.476570 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 132910us
I20260709 12:23:34.477324 18333 leader_election.cc:304] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:23:34.478716 23953 raft_consensus.cc:2749] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:23:34.478796 18333 leader_election.cc:304] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:23:34.479499 23953 raft_consensus.cc:2804] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:23:34.374830 24144 raft_consensus.cc:515] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:34.480275 24215 consensus_queue.cc:1059] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 13.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20260709 12:23:34.481779 24215 consensus_queue.cc:1059] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 12.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 12:23:34.279088 17727 leader_election.cc:304] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:34.482262 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
W20260709 12:23:34.482873 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 203863us
I20260709 12:23:34.483349 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:34.365164 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:34.483995 24085 raft_consensus.cc:3055] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:23:34.346505 17801 leader_election.cc:400] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 17 election: Vote denied by peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651) with higher term. Message: Invalid argument: T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 17. Current term is 18.
I20260709 12:23:34.484436 24085 raft_consensus.cc:740] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:23:34.294098 17833 raft_consensus.cc:1240] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 16 LEADER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 15. Current term is 16. Ops: []
I20260709 12:23:34.485627 24246 raft_consensus.cc:2804] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Leader election won for term 14
W20260709 12:23:34.373816 17760 rpcz_store.cc:269] 0709 12:23:32.828632 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:32.916135 (+ 87503us) service_pool.cc:224] Handling call
0709 12:23:34.287687 (+1371552us) spinlock_profiling.cc:243] Waited 664 ms on lock 0x7b5400415fd4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:23:34.287828 (+   141us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1527836032}
I20260709 12:23:34.486150 24085 consensus_queue.cc:260] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 12.9, Last appended by leader: 9, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:34.373447 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:34.488090 17760 raft_consensus.cc:1240] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 19. Current term is 20. Ops: [17.11-19.12]
I20260709 12:23:34.414582 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:23:34.490586 17803 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:34.415428 17682 raft_consensus.cc:2393] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 17: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:23:34.371220 17834 rpcz_store.cc:276] 0709 12:23:32.764695 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:33.023425 (+258730us) service_pool.cc:224] Handling call
0709 12:23:34.275058 (+1251633us) spinlock_profiling.cc:243] Waited 694 ms on lock 0x7b5400089dd4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:23:34.275181 (+   123us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1596751232}
I20260709 12:23:34.414780 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:34.374452 17802 leader_election.cc:304] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:23:34.492139 17684 raft_consensus.cc:2468] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 17.
I20260709 12:23:34.424664 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 20 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:34.484438 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:34.368593 24214 raft_consensus.cc:493] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:34.505993 18333 leader_election.cc:400] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 13 pre-election: Vote denied by peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837) with higher term. Message: Invalid argument: T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 13. Current term is 14.
I20260709 12:23:34.431074 17652 leader_election.cc:304] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:34.378077 24061 raft_consensus.cc:3060] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:34.363802 24088 raft_consensus.cc:740] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:23:34.439162 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:34.375511 24148 raft_consensus.cc:2749] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Leader election lost for term 19. Reason: Vote denied by peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651) with higher term. Message: Invalid argument: T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 19. Current term is 20.
W20260709 12:23:34.442788 24245 log.cc:927] Time spent T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.093s	user 0.000s	sys 0.000s
I20260709 12:23:34.368664 24238 consensus_queue.cc:1048] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.076s
I20260709 12:23:34.480060 23931 raft_consensus.cc:2804] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:23:34.480018 23953 raft_consensus.cc:697] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 14 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:23:34.482017 24144 leader_election.cc:290] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:34.357626 18367 raft_consensus.cc:2468] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 14.
I20260709 12:23:34.486178 24246 raft_consensus.cc:697] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 14 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:23:34.438779 23740 raft_consensus.cc:515] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:34.313835 17681 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37402 (request call id 3060) took 1619 ms. Trace:
I20260709 12:23:34.493678 17686 raft_consensus.cc:2393] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 14: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:23:34.541791 17833 raft_consensus.cc:1240] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 17 LEADER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 16. Current term is 17. Ops: []
I20260709 12:23:34.420033 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:23:34.495152 18334 leader_election.cc:336] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 17 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Timed out: RequestConsensusVote RPC to 127.16.29.65:33071 timed out after 1.509s (SENT)
I20260709 12:23:34.542831 17688 raft_consensus.cc:2468] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 18.
I20260709 12:23:34.542956 18334 leader_election.cc:304] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:34.495829 17756 raft_consensus.cc:2393] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 15: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:23:34.429445 17836 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 13.
I20260709 12:23:34.498589 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:34.471386 17587 catalog_manager.cc:5697] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 11 to 16, leader changed from e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66) to 0da169ff82674dc1947055ce43de68a3 (127.16.29.65). New cstate: current_term: 16 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:23:34.499847 17764 raft_consensus.cc:1240] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 18. Current term is 19. Ops: []
I20260709 12:23:34.493739 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:34.506382 18333 leader_election.cc:403] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 13 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:23:34.507319 18361 raft_consensus.cc:2468] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 19.
I20260709 12:23:34.511801 17689 raft_consensus.cc:2468] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 19.
I20260709 12:23:34.539299 17729 leader_election.cc:304] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:23:34.514076 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 139695us
I20260709 12:23:34.482791 24215 raft_consensus.cc:3055] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:23:34.522959 17755 raft_consensus.cc:2393] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 17: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
W20260709 12:23:34.523381 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 102166us
I20260709 12:23:34.526901 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:34.455077 24102 raft_consensus.cc:515] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:34.510679 24009 raft_consensus.cc:2804] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:23:34.484609 17801 leader_election.cc:403] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 17 election: Cancelling election due to peer responding with higher term
I20260709 12:23:34.557830 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:34.533834 17835 raft_consensus.cc:2410] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for term 20 because replica has last-logged OpId of term: 18 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 12.
I20260709 12:23:34.517439 23906 consensus_queue.cc:1048] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.017s
I20260709 12:23:34.424646 24232 consensus_queue.cc:1048] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.110s
I20260709 12:23:34.511255 24206 raft_consensus.cc:2749] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:23:34.434855 23992 raft_consensus.cc:515] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:34.541028 17681 rpcz_store.cc:276] 0709 12:23:32.694793 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:32.935509 (+240716us) service_pool.cc:224] Handling call
0709 12:23:32.936096 (+   587us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:32.937069 (+   973us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:32.937099 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:32.937959 (+   860us) log.cc:844] Serialized 26 byte log entry
0709 12:23:33.100437 (+162478us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:23:33.100477 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:33.100552 (+    75us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:34.196081 (+1095529us) spinlock_profiling.cc:243] Waited 2.47 ms on lock 0x7b2400172cb8. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b473b99c4 00007f7b473b7acb 00007f7b47572dc9 00007f7b47572a55 00007f7b4757209d 00007f7b4a1a410e 00007f7b4a1bb88c 00007f7b4a1babb6 00007f7b4a1ba7d8 00007f7b4a1c4221
0709 12:23:34.313331 (+117250us) raft_consensus.cc:1685] finished
0709 12:23:34.313359 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:34.313745 (+   386us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":389,"raft.run_cpu_time_us":49,"raft.run_wall_time_us":50,"spinlock_wait_cycles":5684096,"thread_start_us":162058,"threads_started":1,"wal-append.queue_time_us":206249}
I20260709 12:23:34.522433 24146 raft_consensus.cc:2804] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:23:34.545120 17683 raft_consensus.cc:2468] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 17.
I20260709 12:23:34.555284 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:34.552397 17729 leader_election.cc:304] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:34.552166 18367 raft_consensus.cc:1275] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
W20260709 12:23:34.547559 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 53434.3us
I20260709 12:23:34.586288 17688 raft_consensus.cc:3060] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:34.587311 18333 leader_election.cc:304] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:34.554582 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 15 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:34.524554 24155 consensus_queue.cc:1048] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.034s
I20260709 12:23:34.591059 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:34.540880 23931 raft_consensus.cc:697] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 13 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:23:34.527542 24111 leader_election.cc:290] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 15 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:34.561519 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
W20260709 12:23:34.561990 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 215615us
I20260709 12:23:34.573797 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
W20260709 12:23:34.578156 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 83020.2us
I20260709 12:23:34.556603 17684 raft_consensus.cc:2368] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 21. Current term is 22.
I20260709 12:23:34.515961 24214 raft_consensus.cc:515] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:34.547634 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:34.557945 24037 raft_consensus.cc:2804] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:23:34.554114 17802 leader_election.cc:304] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:34.555264 24215 raft_consensus.cc:740] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:23:34.595562 18361 raft_consensus.cc:1275] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 13 index: 9. Preceding OpId from leader: term: 16 index: 10. (index mismatch)
I20260709 12:23:34.596518 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 15 candidate_status { last_received { term: 14 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:34.600148 17728 leader_election.cc:304] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:23:34.602145 18367 raft_consensus.cc:1240] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 14. Current term is 15. Ops: [14.12-14.12]
I20260709 12:23:34.576377 24238 raft_consensus.cc:3060] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:34.604301 17829 raft_consensus.cc:1240] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 19. Current term is 20. Ops: []
I20260709 12:23:34.597396 18368 raft_consensus.cc:1240] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 10. Current term is 11. Ops: []
I20260709 12:23:34.623711 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 22 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:34.623683 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:34.624936 17801 leader_election.cc:400] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 18 election: Vote denied by peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651) with higher term. Message: Invalid argument: T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 18. Current term is 19.
I20260709 12:23:34.618083 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:34.580354 24151 raft_consensus.cc:2749] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:23:34.626230 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 17 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:34.627332 18374 raft_consensus.cc:1240] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 14. Current term is 15. Ops: [14.12-14.12]
I20260709 12:23:34.629755 18363 raft_consensus.cc:1240] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 14. Current term is 15. Ops: [14.11-14.11]
I20260709 12:23:34.631570 17686 raft_consensus.cc:1240] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 18. Current term is 19. Ops: [18.12-18.12]
I20260709 12:23:34.562455 24148 consensus_queue.cc:1048] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.042s
I20260709 12:23:34.587817 24009 raft_consensus.cc:697] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:23:34.633395 18361 raft_consensus.cc:3060] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:23:34.638345 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:34.641312 18368 raft_consensus.cc:2393] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 16: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:23:34.632820 17763 raft_consensus.cc:2468] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 21.
I20260709 12:23:34.634246 17801 leader_election.cc:403] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 18 election: Cancelling election due to peer responding with higher term
W20260709 12:23:34.628408 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 74478.6us
I20260709 12:23:34.643816 18371 raft_consensus.cc:1240] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 14. Current term is 15. Ops: []
I20260709 12:23:34.596020 24146 raft_consensus.cc:697] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 18 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:23:34.558812 24211 raft_consensus.cc:515] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:34.642107 18334 leader_election.cc:304] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:23:34.590481 23838 raft_consensus.cc:2764] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Leader pre-election decision vote started in defunct term 18: won
I20260709 12:23:34.605667 24032 raft_consensus.cc:2804] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:23:34.653460 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:34.654536 18365 raft_consensus.cc:1240] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 18. Current term is 19. Ops: []
I20260709 12:23:34.669644 17681 raft_consensus.cc:3060] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:23:34.657388 17654 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:34.661293 18368 raft_consensus.cc:1275] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 17 index: 13. Preceding OpId from leader: term: 18 index: 13. (term mismatch)
I20260709 12:23:34.670473 18368 pending_rounds.cc:85] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da: Aborting all ops after (but not including) 12
I20260709 12:23:34.670960 18368 pending_rounds.cc:107] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da: Aborting uncommitted NO_OP operation due to leader change: 17.13
I20260709 12:23:34.671386 18368 raft_consensus.cc:2889] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:34.668144 17831 raft_consensus.cc:2468] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 13.
I20260709 12:23:34.672361 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:34.673444 17686 raft_consensus.cc:3060] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:34.675407 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:34.682386 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:34.688825 24009 consensus_queue.cc:237] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.10, Last appended by leader: 10, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:34.632117 24085 raft_consensus.cc:3060] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:34.648046 23970 raft_consensus.cc:2804] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:23:34.692709 23970 raft_consensus.cc:493] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:23:34.693208 23970 raft_consensus.cc:3060] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:34.693256 24148 raft_consensus.cc:3060] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:34.694717 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:34.696007 18362 raft_consensus.cc:1240] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 16. Current term is 17. Ops: [16.12-16.12]
I20260709 12:23:34.703398 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:34.704418 17689 raft_consensus.cc:2468] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 18.
I20260709 12:23:34.699389 17587 catalog_manager.cc:5697] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 11 to 13. New cstate: current_term: 13 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
W20260709 12:23:34.696743 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 54707us
I20260709 12:23:34.681006 18364 raft_consensus.cc:1275] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [term 17 LEARNER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 17 index: 9. Preceding OpId from leader: term: 17 index: 10. (index mismatch)
I20260709 12:23:34.699553 17830 raft_consensus.cc:1240] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 20 LEADER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 19. Current term is 20. Ops: [19.12-19.12]
W20260709 12:23:34.700618 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 74382.6us
I20260709 12:23:34.638234 24182 raft_consensus.cc:2804] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:23:34.723935 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:34.725864 17682 raft_consensus.cc:3060] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:34.637506 24037 raft_consensus.cc:697] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 14 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:23:34.724411 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:34.745325 17683 raft_consensus.cc:3060] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:34.728359 17688 raft_consensus.cc:2468] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 15.
I20260709 12:23:34.725359 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:34.747406 17684 raft_consensus.cc:3060] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:34.742957 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:34.750435 18361 raft_consensus.cc:2468] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 22.
I20260709 12:23:34.750897 18368 raft_consensus.cc:2468] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 14.
I20260709 12:23:34.743988 18366 raft_consensus.cc:1240] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 17. Current term is 18. Ops: []
I20260709 12:23:34.744163 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:34.621791 23997 consensus_queue.cc:1048] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.075s
W20260709 12:23:34.780412 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 135149us
W20260709 12:23:34.774348 17655 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:34.754206 17689 raft_consensus.cc:3060] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:34.781816 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
W20260709 12:23:34.625954 24227 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.568s	user 0.000s	sys 0.025s
I20260709 12:23:34.784418 17802 leader_election.cc:304] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:23:34.705178 24203 raft_consensus.cc:2804] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:23:34.630324 24116 consensus_queue.cc:1059] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.116s
I20260709 12:23:34.807730 17652 leader_election.cc:304] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:34.811352 17686 raft_consensus.cc:2468] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 20.
I20260709 12:23:34.813478 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:34.724426 24232 raft_consensus.cc:493] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:34.727617 24032 raft_consensus.cc:697] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 19 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:23:34.820789 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
W20260709 12:23:34.796236 24059 consensus_meta.cc:287] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.593s	user 0.010s	sys 0.001s
W20260709 12:23:34.791539 24227 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.737s	user 0.000s	sys 0.026s
I20260709 12:23:34.823058 17684 raft_consensus.cc:2468] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 18.
I20260709 12:23:34.732303 24182 raft_consensus.cc:493] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:34.639931 24088 consensus_queue.cc:260] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 14.13, Last appended by leader: 13, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:34.791987 24203 raft_consensus.cc:493] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:34.843307 17685 raft_consensus.cc:2410] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for term 17 because replica has last-logged OpId of term: 16 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 11.
I20260709 12:23:34.824028 17686 raft_consensus.cc:2468] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 18.
I20260709 12:23:34.864068 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:34.869573 17684 raft_consensus.cc:3060] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:34.865935 17689 raft_consensus.cc:2468] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 14.
I20260709 12:23:34.870316 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 20 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:34.825475 24229 raft_consensus.cc:2804] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:23:34.875581 18368 raft_consensus.cc:2410] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 20 because replica has last-logged OpId of term: 18 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 12.
I20260709 12:23:34.837615 24210 raft_consensus.cc:2749] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Leader pre-election lost for term 13. Reason: Vote denied by peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837) with higher term. Message: Invalid argument: T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 13. Current term is 14.
W20260709 12:23:34.881417 24252 log.cc:927] Time spent T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.052s	user 0.000s	sys 0.001s
I20260709 12:23:34.883935 24146 consensus_queue.cc:237] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.10, Last appended by leader: 10, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:34.837988 24087 consensus_queue.cc:1059] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 16.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.047s
W20260709 12:23:34.885900 24249 log.cc:927] Time spent T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.051s	user 0.000s	sys 0.000s
I20260709 12:23:34.699139 23953 consensus_queue.cc:237] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.10, Last appended by leader: 10, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:34.886870 24169 leader_election.cc:290] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 15 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:34.829301 23651 raft_consensus.cc:2804] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:23:34.890285 24214 leader_election.cc:290] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:34.966841 18365 raft_consensus.cc:2393] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 14: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
W20260709 12:23:34.933413 24141 consensus_meta.cc:287] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.650s	user 0.006s	sys 0.006s
I20260709 12:23:34.881047 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 13 candidate_status { last_received { term: 10 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:34.891683 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 17 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:34.835278 24232 raft_consensus.cc:515] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:34.896392 17587 catalog_manager.cc:5697] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 8 to 11, leader changed from 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) to e063541542a94969898868bf6eb5ce1b (127.16.29.67). New cstate: current_term: 11 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:23:34.901182 17687 raft_consensus.cc:1275] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 16 index: 12. Preceding OpId from leader: term: 17 index: 12. (term mismatch)
I20260709 12:23:34.915690 18334 leader_election.cc:400] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 21 election: Vote denied by peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071) with higher term. Message: Invalid argument: T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 21. Current term is 22.
I20260709 12:23:34.977478 17687 pending_rounds.cc:85] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3: Aborting all ops after (but not including) 11
I20260709 12:23:34.977596 18334 leader_election.cc:403] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 21 election: Cancelling election due to peer responding with higher term
I20260709 12:23:34.977974 17687 pending_rounds.cc:107] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3: Aborting uncommitted NO_OP operation due to leader change: 16.12
I20260709 12:23:34.927798 17681 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 15.
I20260709 12:23:34.978389 17687 raft_consensus.cc:2889] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:34.922861 17682 raft_consensus.cc:2468] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 17.
I20260709 12:23:34.742384 24215 consensus_queue.cc:260] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 13.13, Last appended by leader: 13, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:34.875809 24203 raft_consensus.cc:3060] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:34.871220 24182 raft_consensus.cc:3060] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:34.938663 17654 leader_election.cc:304] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:23:34.982352 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
W20260709 12:23:34.981940 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 74209us
I20260709 12:23:34.983160 17686 raft_consensus.cc:3060] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:34.947278 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:34.989116 18368 raft_consensus.cc:2393] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 15: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:23:34.989884 24183 raft_consensus.cc:493] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:23:34.905596 24229 raft_consensus.cc:493] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:34.752269 23931 consensus_queue.cc:237] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 11.13, Last appended by leader: 13, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:34.945623 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 14 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:34.948611 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 22 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:34.990609 24183 raft_consensus.cc:515] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:34.890506 17580 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35962 (request call id 358) took 29150 ms (client timeout 15000 ms). Trace:
I20260709 12:23:34.752656 24246 consensus_queue.cc:237] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 13.13, Last appended by leader: 13, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:34.876475 24236 consensus_queue.cc:1059] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 15.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.077s
I20260709 12:23:34.981462 17689 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b"
dest_uuid: "0da169ff82674dc1947055ce43de68a3"
mode: GRACEFUL
new_leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
 from {username='slave'} at 127.0.0.1:54540
I20260709 12:23:34.892558 23992 leader_election.cc:290] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 20 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:34.992204 18361 raft_consensus.cc:2393] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 22: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:23:35.000289 17728 leader_election.cc:304] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:35.000900 24229 raft_consensus.cc:3060] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Advancing to term 19
W20260709 12:23:34.990242 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 365487us
I20260709 12:23:35.001283 24237 raft_consensus.cc:3055] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:23:35.001977 24237 raft_consensus.cc:740] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:23:35.002915 24237 consensus_queue.cc:260] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.12, Last appended by leader: 12, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:35.004318 24237 raft_consensus.cc:3060] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:35.010496 23737 raft_consensus.cc:2749] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:23:35.012079 17728 leader_election.cc:304] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:35.025911 17681 raft_consensus.cc:3055] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:23:35.026376 17681 raft_consensus.cc:740] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
W20260709 12:23:35.028309 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 43601us
I20260709 12:23:35.030053 18334 leader_election.cc:304] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:23:35.031279 17684 raft_consensus.cc:2468] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 18.
I20260709 12:23:35.040727 17587 catalog_manager.cc:5697] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 2 to 15, leader changed from e063541542a94969898868bf6eb5ce1b (127.16.29.67) to 334a4e1f59944544b3b3ff36616b57da (127.16.29.68). New cstate: current_term: 15 leader_uuid: "334a4e1f59944544b3b3ff36616b57da" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 12:23:35.044211 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 105642us
W20260709 12:23:35.047413 17654 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:34.876973 24059 raft_consensus.cc:515] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:34.972812 23997 raft_consensus.cc:493] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:35.064613 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 19 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:34.971457 24102 leader_election.cc:290] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 15 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:35.028940 17728 leader_election.cc:304] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:23:35.079727 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 50898.6us
I20260709 12:23:35.080530 17728 leader_election.cc:304] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:23:35.061579 17681 consensus_queue.cc:260] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.10, Last appended by leader: 10, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:35.077184 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 20 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:35.081727 17756 raft_consensus.cc:3060] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:35.069502 17587 catalog_manager.cc:5697] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 11 to 14. New cstate: current_term: 14 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } }
I20260709 12:23:35.069809 17686 raft_consensus.cc:2468] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 15.
I20260709 12:23:35.090737 18365 raft_consensus.cc:1240] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 19. Current term is 20. Ops: []
W20260709 12:23:35.093291 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:35.096210 24226 raft_consensus.cc:2804] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:23:35.096973 24226 raft_consensus.cc:697] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 20 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:23:35.098130 24226 consensus_queue.cc:237] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.13, Last appended by leader: 13, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:35.099722 23906 raft_consensus.cc:2804] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:23:35.100307 23906 raft_consensus.cc:697] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:23:35.101497 23906 consensus_queue.cc:237] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 17.9, Last appended by leader: 9, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:23:35.104506 17763 raft_consensus.cc:1275] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 15 index: 12. (index mismatch)
I20260709 12:23:35.105861 17681 raft_consensus.cc:3060] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:35.100226 17728 leader_election.cc:304] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:34.944725 23928 consensus_queue.cc:1059] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 11.10, Next index: 11, Last known committed idx: 9, Time since last communication: 0.119s
I20260709 12:23:35.110235 17728 leader_election.cc:304] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:35.109236 17687 raft_consensus.cc:1240] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 14. Current term is 15. Ops: [14.12-14.12]
I20260709 12:23:35.113162 17686 raft_consensus.cc:3060] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:35.042755 24210 raft_consensus.cc:2749] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Leader election lost for term 21. Reason: Vote denied by peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071) with higher term. Message: Invalid argument: T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 21. Current term is 22.
I20260709 12:23:35.116462 17587 catalog_manager.cc:5697] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 12 to 15. New cstate: current_term: 15 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:23:34.886891 24037 consensus_queue.cc:237] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:35.125849 17756 raft_consensus.cc:2468] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 20.
I20260709 12:23:34.973063 24155 consensus_queue.cc:1059] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.110s
I20260709 12:23:35.056103 23737 raft_consensus.cc:2749] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:23:35.067188 24247 raft_consensus.cc:2804] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:23:35.136950 17832 raft_consensus.cc:2393] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 19: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
W20260709 12:23:35.136806 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 316921us
I20260709 12:23:35.025553 24227 consensus_queue.cc:1059] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 12.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.137s
I20260709 12:23:35.150271 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 13 candidate_status { last_received { term: 10 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:35.157414 17587 catalog_manager.cc:5697] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 6 to 8. New cstate: current_term: 8 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:23:35.158388 18361 raft_consensus.cc:2393] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 13: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:23:34.981902 24141 raft_consensus.cc:515] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:35.153770 17687 raft_consensus.cc:1275] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 18 index: 11. Preceding OpId from leader: term: 19 index: 12. (index mismatch)
I20260709 12:23:35.154599 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:35.159685 17831 raft_consensus.cc:3060] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:35.116698 24214 raft_consensus.cc:2804] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:23:35.065761 24151 raft_consensus.cc:2804] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:23:35.110884 24250 raft_consensus.cc:3055] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:23:35.058157 24231 consensus_queue.cc:1059] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 17.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.077s
I20260709 12:23:35.072518 23740 leader_election.cc:290] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:35.148834 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:35.136543 24169 raft_consensus.cc:2804] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:23:35.166641 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:35.182747 18365 raft_consensus.cc:2368] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 18. Current term is 19.
I20260709 12:23:35.118680 17728 leader_election.cc:304] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:35.169401 17836 raft_consensus.cc:1275] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
I20260709 12:23:35.172243 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:35.177596 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:35.041091 24032 consensus_queue.cc:237] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.13, Last appended by leader: 13, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:35.110306 23997 raft_consensus.cc:515] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:35.184154 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 65539us
I20260709 12:23:35.195566 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:35.194981 17587 catalog_manager.cc:5697] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 15 to 16, leader changed from e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66) to 334a4e1f59944544b3b3ff36616b57da (127.16.29.68). New cstate: current_term: 16 leader_uuid: "334a4e1f59944544b3b3ff36616b57da" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:23:35.208256 17829 raft_consensus.cc:2393] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 14: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:23:35.155468 23928 raft_consensus.cc:3055] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:23:35.197422 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:35.207454 17728 leader_election.cc:304] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:35.157248 24247 raft_consensus.cc:697] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 22 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:23:35.211122 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:35.163120 24214 raft_consensus.cc:493] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:35.214872 17587 catalog_manager.cc:5697] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 14 to 16, leader changed from e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66) to 0da169ff82674dc1947055ce43de68a3 (127.16.29.65). New cstate: current_term: 16 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:23:35.134522 23739 consensus_queue.cc:1048] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.046s
I20260709 12:23:35.223703 17683 raft_consensus.cc:2468] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 18.
I20260709 12:23:35.165115 24155 consensus_queue.cc:1243] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer 334a4e1f59944544b3b3ff36616b57da log is divergent from this leader: its last log entry 13.12 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 11
I20260709 12:23:35.175120 23992 raft_consensus.cc:493] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:23:35.232614 18333 leader_election.cc:304] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:35.249621 17587 catalog_manager.cc:5697] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 13 to 14, leader changed from e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66) to 334a4e1f59944544b3b3ff36616b57da (127.16.29.68). New cstate: current_term: 14 leader_uuid: "334a4e1f59944544b3b3ff36616b57da" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:23:35.233529 18365 raft_consensus.cc:1275] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 12 index: 10. Preceding OpId from leader: term: 15 index: 12. (index mismatch)
I20260709 12:23:35.179956 24151 raft_consensus.cc:697] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 15 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:23:35.175621 24250 raft_consensus.cc:740] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:23:35.241055 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 17 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:23:35.227082 24254 log.cc:927] Time spent T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.051s	user 0.000s	sys 0.000s
I20260709 12:23:35.242844 17681 raft_consensus.cc:1275] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 15 index: 10. Preceding OpId from leader: term: 16 index: 10. (term mismatch)
I20260709 12:23:35.238248 17833 raft_consensus.cc:2468] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 20.
I20260709 12:23:35.208662 23740 raft_consensus.cc:493] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:35.252682 17587 catalog_manager.cc:5697] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 9 to 14, leader changed from 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) to 0da169ff82674dc1947055ce43de68a3 (127.16.29.65). New cstate: current_term: 14 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:23:35.228636 24214 raft_consensus.cc:3060] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:35.210693 23928 raft_consensus.cc:740] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:23:35.286330 17681 pending_rounds.cc:85] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3: Aborting all ops after (but not including) 9
I20260709 12:23:35.245529 17686 raft_consensus.cc:1275] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 16 index: 9. Preceding OpId from leader: term: 19 index: 11. (index mismatch)
I20260709 12:23:35.303155 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:35.278278 17831 raft_consensus.cc:2468] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 16.
I20260709 12:23:35.303942 17836 raft_consensus.cc:3060] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Advancing to term 15
I20260709 12:23:35.303689 17681 pending_rounds.cc:107] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3: Aborting uncommitted NO_OP operation due to leader change: 15.10
I20260709 12:23:35.311309 17681 raft_consensus.cc:2889] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:35.303193 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:35.316612 17833 raft_consensus.cc:2468] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 14.
I20260709 12:23:35.254463 24102 raft_consensus.cc:2749] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
W20260709 12:23:35.346565 17801 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:35.267997 24263 raft_consensus.cc:2804] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:23:35.268682 24266 raft_consensus.cc:493] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:23:35.285617 24155 raft_consensus.cc:2804] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:23:35.368402 17836 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 15.
I20260709 12:23:35.380666 17727 leader_election.cc:304] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:35.381362 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:35.382056 17836 raft_consensus.cc:3060] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:35.384140 17756 raft_consensus.cc:1275] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 16 index: 12. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
I20260709 12:23:35.390422 17727 leader_election.cc:304] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:35.268884 23992 raft_consensus.cc:515] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:35.397251 17836 raft_consensus.cc:2468] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 16.
I20260709 12:23:35.394348 17587 catalog_manager.cc:5697] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 11 to 13. New cstate: current_term: 13 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:23:35.354573 24087 raft_consensus.cc:3055] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:23:35.368603 17833 raft_consensus.cc:1275] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 17 index: 10. Preceding OpId from leader: term: 19 index: 11. (index mismatch)
I20260709 12:23:35.415954 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 18 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:35.415177 17587 catalog_manager.cc:5697] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 15 to 16, leader changed from e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66) to 334a4e1f59944544b3b3ff36616b57da (127.16.29.68). New cstate: current_term: 16 leader_uuid: "334a4e1f59944544b3b3ff36616b57da" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:23:35.382427 24009 raft_consensus.cc:493] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:35.373679 17654 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:35.316658 23838 consensus_queue.cc:1059] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 18.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.086s
I20260709 12:23:35.305199 23740 raft_consensus.cc:515] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:35.431514 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:35.437040 17681 raft_consensus.cc:2393] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 15: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:23:35.388976 24263 raft_consensus.cc:697] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 17 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:23:35.458096 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:35.392058 24155 raft_consensus.cc:697] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
W20260709 12:23:35.427042 24255 log.cc:927] Time spent T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.354s	user 0.001s	sys 0.000s
I20260709 12:23:35.458520 17801 leader_election.cc:304] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:35.489792 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
W20260709 12:23:35.488235 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 314193us
I20260709 12:23:35.492233 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:35.418781 24087 raft_consensus.cc:740] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:23:35.502709 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:23:35.462769 24098 consensus_meta.cc:287] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 1.135s	user 0.013s	sys 0.000s
W20260709 12:23:35.464267 24253 log.cc:927] Time spent T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.336s	user 0.001s	sys 0.000s
I20260709 12:23:35.518764 18368 raft_consensus.cc:1240] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 14. Current term is 15. Ops: []
I20260709 12:23:35.518502 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 22 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:35.506784 17836 raft_consensus.cc:2393] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 15: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
W20260709 12:23:35.508931 17652 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:35.510106 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 22 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
W20260709 12:23:35.511773 17688 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.598s	user 0.000s	sys 0.012s
I20260709 12:23:35.436545 23651 raft_consensus.cc:697] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 19 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:23:35.388576 24266 raft_consensus.cc:515] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:35.429797 24059 leader_election.cc:290] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 19 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:35.422055 24102 consensus_queue.cc:1059] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 16.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.082s
I20260709 12:23:35.354599 24247 consensus_queue.cc:237] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 20.16, Last appended by leader: 16, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:35.473565 17587 catalog_manager.cc:5697] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 14 to 16. New cstate: current_term: 16 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:23:35.520885 17756 raft_consensus.cc:3060] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:23:35.535410 17681 raft_consensus.cc:1240] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 14. Current term is 15. Ops: []
I20260709 12:23:35.458094 23905 raft_consensus.cc:493] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
W20260709 12:23:35.521077 17688 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.607s	user 0.000s	sys 0.013s
W20260709 12:23:35.527413 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 241917us
I20260709 12:23:35.528332 17686 raft_consensus.cc:1275] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 16 index: 13. Preceding OpId from leader: term: 17 index: 14. (index mismatch)
W20260709 12:23:35.522936 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 64514.2us
I20260709 12:23:35.466670 24183 leader_election.cc:290] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:35.530454 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 15 candidate_status { last_received { term: 14 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
W20260709 12:23:35.538766 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 148474us
I20260709 12:23:35.421584 23928 consensus_queue.cc:260] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 19.12, Last appended by leader: 12, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:35.541824 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:35.462881 24009 raft_consensus.cc:515] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:35.559692 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 19 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:35.553994 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 17 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:35.561872 17756 raft_consensus.cc:2468] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 22.
I20260709 12:23:35.503335 24227 raft_consensus.cc:3055] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:23:35.534986 24267 consensus_queue.cc:1048] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.034s
I20260709 12:23:35.547713 24275 raft_consensus.cc:2804] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Leader election won for term 20
W20260709 12:23:35.563992 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:35.459856 24151 consensus_queue.cc:237] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.12, Last appended by leader: 12, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:35.548179 17727 leader_election.cc:304] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:35.538941 23931 raft_consensus.cc:493] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:23:35.568136 17587 catalog_manager.cc:5697] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 8 to 14, leader changed from 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) to e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66). New cstate: current_term: 14 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:23:35.563481 23928 raft_consensus.cc:3060] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:35.554963 24183 raft_consensus.cc:2749] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:23:35.598577 17587 catalog_manager.cc:5697] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 10 to 11, leader changed from e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66) to 0da169ff82674dc1947055ce43de68a3 (127.16.29.65). New cstate: current_term: 11 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:23:35.555817 24232 leader_election.cc:290] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:35.510702 23906 consensus_queue.cc:1048] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.039s
I20260709 12:23:35.582808 24276 raft_consensus.cc:2804] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:23:35.557957 24211 leader_election.cc:290] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 18 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:35.557706 23997 leader_election.cc:290] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:35.589650 17762 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38786 (request call id 3341) took 1540 ms. Trace:
I20260709 12:23:35.526444 24098 raft_consensus.cc:515] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:23:35.585314 24259 log.cc:927] Time spent T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.330s	user 0.000s	sys 0.001s
W20260709 12:23:35.648185 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 231560us
I20260709 12:23:35.607177 23740 leader_election.cc:290] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:35.608702 24009 leader_election.cc:290] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:35.576047 24263 consensus_queue.cc:237] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.9, Last appended by leader: 9, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:35.583031 24087 consensus_queue.cc:260] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.11, Last appended by leader: 11, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:35.610361 24085 consensus_meta.cc:287] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.918s	user 0.010s	sys 0.001s
W20260709 12:23:35.612192 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 64125.3us
W20260709 12:23:35.568137 23970 consensus_meta.cc:287] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.862s	user 0.008s	sys 0.004s
I20260709 12:23:35.555140 23905 raft_consensus.cc:515] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:35.505069 24250 consensus_queue.cc:260] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.12, Last appended by leader: 12, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:35.679984 24250 raft_consensus.cc:3060] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Advancing to term 17
W20260709 12:23:35.680388 24281 log.cc:927] Time spent T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.084s	user 0.000s	sys 0.001s
I20260709 12:23:35.588037 24155 consensus_queue.cc:237] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.10, Last appended by leader: 10, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:35.684185 17837 raft_consensus.cc:1240] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 17. Current term is 18. Ops: []
I20260709 12:23:35.686760 23905 leader_election.cc:290] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:23:34.994863 17580 rpcz_store.cc:269] 0709 12:23:05.739839 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:05.740151 (+   312us) service_pool.cc:224] Handling call
0709 12:23:34.890249 (+29150098us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:23:27.500126 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:23:27.668096 (+167970us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:23:27.668173 (+    77us) write_op.cc:435] Acquired schema lock
0709 12:23:27.668201 (+    28us) tablet.cc:662] Decoding operations
0709 12:23:27.670438 (+  2237us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:23:27.670559 (+   121us) write_op.cc:641] Partition lock acquired for write op
0709 12:23:27.670601 (+    42us) tablet.cc:685] Acquiring locks for 19 operations
0709 12:23:27.831150 (+160549us) tablet.cc:701] Row locks acquired
0709 12:23:27.831202 (+    52us) write_op.cc:260] PREPARE: finished
0709 12:23:27.831396 (+   194us) write_op.cc:270] Start()
0709 12:23:27.831560 (+   164us) write_op.cc:276] Timestamp: P: 1783599807831328 usec, L: 0
0709 12:23:27.831595 (+    35us) op_driver.cc:348] REPLICATION: starting
0709 12:23:27.832414 (+   819us) log.cc:844] Serialized 7467 byte log entry
0709 12:23:28.608356 (+775942us) op_driver.cc:464] REPLICATION: finished
0709 12:23:29.283786 (+675430us) spinlock_profiling.cc:243] Waited 16.2 ms on lock 0x7b4400022b4c. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b47459c85 00007f7b4745a6a2 00007f7b4a1cc1db 00007f7b4a1c7aaf 00007f7b475160ac 00007f7b503709a9 00007f7b4753269b 00007f7b4753093f 00007f7b47532c02 00007f7b4e056ef0
0709 12:23:30.369342 (+1085556us) spinlock_profiling.cc:243] Waited 3.11 ms on lock 0x7b180035a040. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b475333f3 00007f7b475357c1 00007f7b47535779 00007f7b47535709 00007f7b475356d1 00007f7b475349cd 0000000000388634 0000000000388479 00007f7b4751683a
0709 12:23:30.369562 (+   220us) write_op.cc:301] APPLY: starting
0709 12:23:30.369668 (+   106us) tablet.cc:1367] starting BulkCheckPresence
0709 12:23:30.665169 (+295501us) tablet.cc:1370] finished BulkCheckPresence
0709 12:23:30.665219 (+    50us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:23:31.706709 (+1041490us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:23:31.707290 (+   581us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=38,key_file_lookups=38,delta_file_lookups=38,mrs_lookups=0
0709 12:23:31.707602 (+   312us) write_op.cc:312] APPLY: finished
0709 12:23:31.708384 (+   782us) log.cc:844] Serialized 172 byte log entry
0709 12:23:31.896834 (+188450us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:23:31.912942 (+ 16108us) write_op.cc:454] Released schema lock
0709 12:23:31.924862 (+ 11920us) write_op.cc:341] FINISH: Updating metrics
0709 12:23:32.036475 (+111613us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:23:32.036531 (+    56us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":3043095,"reported_tablets":45,"child_traces":[["op",{"apply.queue_time_us":1245623,"apply.run_cpu_time_us":45614,"apply.run_wall_time_us":1630206,"cfile_cache_hit":78,"cfile_cache_hit_bytes":83623,"num_ops":19,"prepare.queue_time_us":303043,"prepare.run_cpu_time_us":20629,"prepare.run_wall_time_us":498420,"raft.queue_time_us":369359,"raft.run_cpu_time_us":71976,"raft.run_wall_time_us":1262690,"replication_time_us":776513,"spinlock_wait_cycles":44335744,"thread_start_us":1987677,"threads_started":5,"wal-append.queue_time_us":359994,"wal-append.run_cpu_time_us":31642,"wal-append.run_wall_time_us":2631417}]]}
W20260709 12:23:35.695318 24148 consensus_meta.cc:287] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.961s	user 0.010s	sys 0.000s
I20260709 12:23:35.604822 23992 leader_election.cc:290] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:35.638653 24169 raft_consensus.cc:697] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:23:35.582827 24227 raft_consensus.cc:740] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:23:35.629004 17762 rpcz_store.cc:276] 0709 12:23:34.049172 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:34.238218 (+189046us) service_pool.cc:224] Handling call
0709 12:23:34.238605 (+   387us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:34.239035 (+   430us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:34.239067 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:34.239541 (+   474us) log.cc:844] Serialized 26 byte log entry
0709 12:23:34.627616 (+388075us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:23:34.627673 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:34.627764 (+    91us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:35.589418 (+961654us) raft_consensus.cc:1685] finished
0709 12:23:35.589444 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:35.589591 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":861,"raft.run_cpu_time_us":126,"raft.run_wall_time_us":127,"thread_start_us":313948,"threads_started":1,"wal-append.queue_time_us":702650}
I20260709 12:23:35.688629 17689 raft_consensus.cc:606] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Received request to transfer leadership to TS e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:35.602286 23931 raft_consensus.cc:515] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:35.697994 24244 consensus_meta.cc:287] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 1.229s	user 0.014s	sys 0.000s
I20260709 12:23:35.706060 24088 raft_consensus.cc:3060] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:35.707752 24266 leader_election.cc:290] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:23:35.708036 24061 consensus_meta.cc:287] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 1.193s	user 0.014s	sys 0.000s
I20260709 12:23:35.626394 17587 catalog_manager.cc:5697] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 9 to 12, leader changed from 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) to 0da169ff82674dc1947055ce43de68a3 (127.16.29.65). New cstate: current_term: 12 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:23:35.714740 17689 raft_consensus.cc:614] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260709 12:23:35.714735 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 62826.1us
I20260709 12:23:35.715636 17728 leader_election.cc:304] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:35.665542 24288 raft_consensus.cc:2804] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Leader pre-election won for term 15
W20260709 12:23:35.709733 24237 consensus_meta.cc:287] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.705s	user 0.013s	sys 0.000s
I20260709 12:23:35.722537 24141 leader_election.cc:290] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 18 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:35.719947 24037 consensus_queue.cc:1059] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 17.12, Next index: 13, Last known committed idx: 11, Time since last communication: 0.003s
I20260709 12:23:35.748258 24276 raft_consensus.cc:697] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:23:35.749599 24037 raft_consensus.cc:3055] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:23:35.749320 24276 consensus_queue.cc:237] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:35.750011 24037 raft_consensus.cc:740] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 17 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:23:35.750859 24037 consensus_queue.cc:260] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.12, Last appended by leader: 12, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:35.713366 24148 raft_consensus.cc:2749] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Leader election lost for term 18. Reason: Vote denied by peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651) with higher term. Message: Invalid argument: T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 18. Current term is 19.
I20260709 12:23:35.588464 23651 consensus_queue.cc:237] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.11, Last appended by leader: 11, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:35.733359 17680 raft_consensus.cc:2410] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 20 because replica has last-logged OpId of term: 18 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 12.
I20260709 12:23:35.714035 24087 raft_consensus.cc:3060] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:35.677304 23970 raft_consensus.cc:515] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:23:35.735960 24229 consensus_meta.cc:287] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.734s	user 0.004s	sys 0.011s
I20260709 12:23:35.722373 24227 consensus_queue.cc:260] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.12, Last appended by leader: 12, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:35.826848 17830 raft_consensus.cc:2468] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 19.
I20260709 12:23:35.765195 17587 catalog_manager.cc:5697] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 13 to 15, leader changed from 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) to 334a4e1f59944544b3b3ff36616b57da (127.16.29.68). New cstate: current_term: 15 leader_uuid: "334a4e1f59944544b3b3ff36616b57da" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:23:35.824575 18371 raft_consensus.cc:1240] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 13. Current term is 14. Ops: []
W20260709 12:23:35.826293 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 89233.9us
W20260709 12:23:35.826642 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 111120us
I20260709 12:23:35.835150 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 19 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:23:35.781494 24271 log.cc:927] Time spent T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.315s	user 0.003s	sys 0.000s
I20260709 12:23:35.838055 17760 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38776 (request call id 2637) took 1539 ms. Trace:
I20260709 12:23:35.752089 24037 raft_consensus.cc:3060] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:35.736011 23997 raft_consensus.cc:493] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:35.785068 24203 consensus_meta.cc:287] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.705s	user 0.011s	sys 0.001s
I20260709 12:23:35.843468 17587 catalog_manager.cc:5697] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 11 to 12, leader changed from 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) to e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66). New cstate: current_term: 12 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:23:35.834447 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:35.736797 24288 raft_consensus.cc:493] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:23:35.856315 17587 catalog_manager.cc:5697] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 7 to 15, leader changed from e063541542a94969898868bf6eb5ce1b (127.16.29.67) to e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66). New cstate: current_term: 15 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:23:35.748072 24009 consensus_queue.cc:1048] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.119s
I20260709 12:23:35.852458 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:23:35.822610 24238 consensus_meta.cc:287] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 1.193s	user 0.013s	sys 0.000s
W20260709 12:23:35.782277 24291 log.cc:927] Time spent T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.073s	user 0.000s	sys 0.000s
W20260709 12:23:35.827404 24214 consensus_meta.cc:287] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.503s	user 0.009s	sys 0.003s
W20260709 12:23:35.781652 24289 log.cc:927] Time spent T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.062s	user 0.000s	sys 0.000s
I20260709 12:23:35.862771 24214 raft_consensus.cc:515] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:35.867748 17757 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38786 (request call id 3347) took 1640 ms. Trace:
I20260709 12:23:35.868250 17757 rpcz_store.cc:276] 0709 12:23:34.227534 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:34.268689 (+ 41155us) service_pool.cc:224] Handling call
0709 12:23:34.269017 (+   328us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:23:34.336361 (+ 67344us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:34.336394 (+    33us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:23:34.337107 (+   713us) log.cc:844] Serialized 52 byte log entry
0709 12:23:34.597922 (+260815us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:23:34.948520 (+350598us) log.cc:844] Serialized 14 byte log entry
0709 12:23:35.387283 (+438763us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:35.867183 (+479900us) spinlock_profiling.cc:243] Waited 66.7 ms on lock 0x7b5400310e54. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:23:35.867256 (+    73us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:35.867318 (+    62us) raft_consensus.cc:1685] finished
0709 12:23:35.867422 (+   104us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:35.867675 (+   253us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":39489,"prepare.queue_time_us":232134,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":44,"raft.queue_time_us":48706,"raft.run_cpu_time_us":429,"raft.run_wall_time_us":11723,"spinlock_wait_cycles":153452800,"thread_start_us":938736,"threads_started":3,"wal-append.queue_time_us":727833}
I20260709 12:23:35.784349 23931 leader_election.cc:290] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:35.872395 17759 raft_consensus.cc:1240] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 15. Current term is 16. Ops: []
I20260709 12:23:35.838488 17760 rpcz_store.cc:276] 0709 12:23:34.298266 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:34.489038 (+190772us) service_pool.cc:224] Handling call
0709 12:23:34.489332 (+   294us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:34.489780 (+   448us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 12:23:34.804246 (+314466us) log.cc:844] Serialized 14 byte log entry
0709 12:23:35.179993 (+375747us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:35.180528 (+   535us) log.cc:844] Serialized 26 byte log entry
0709 12:23:35.180684 (+   156us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:23:35.180735 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:35.180818 (+    83us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:35.780777 (+599959us) spinlock_profiling.cc:243] Waited 68.5 ms on lock 0x7b4400022b4c. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b47459c85 00007f7b4745a6a2 00007f7b4a1cc1db 00007f7b4a1c7aaf 00007f7b4a1bb86e 00007f7b4a1babb6 00007f7b4a1ba7d8 00007f7b4a1c4221 00007f7b4a1c41d9 00007f7b4a1c4169
0709 12:23:35.837633 (+ 56856us) raft_consensus.cc:1685] finished
0709 12:23:35.837767 (+   134us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:35.837973 (+   206us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":13832,"prepare.queue_time_us":262224,"prepare.run_cpu_time_us":73,"prepare.run_wall_time_us":7072,"raft.queue_time_us":5088,"raft.run_cpu_time_us":299,"raft.run_wall_time_us":47114,"spinlock_wait_cycles":157531520,"thread_start_us":620913,"threads_started":2,"wal-append.queue_time_us":501073}
I20260709 12:23:35.873565 17728 leader_election.cc:304] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:35.875988 24116 raft_consensus.cc:3055] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:23:35.876447 24116 raft_consensus.cc:740] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 17 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
W20260709 12:23:35.857245 24286 log.cc:927] Time spent T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.204s	user 0.000s	sys 0.001s
I20260709 12:23:35.860358 18362 raft_consensus.cc:2468] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 15.
I20260709 12:23:35.736685 24293 consensus_queue.cc:1048] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.127s
I20260709 12:23:35.868366 24227 raft_consensus.cc:3060] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:35.890069 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:35.861315 24183 raft_consensus.cc:2804] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:23:35.894726 17831 raft_consensus.cc:2393] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 16: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:23:35.841198 24229 raft_consensus.cc:515] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:35.860669 24288 raft_consensus.cc:3060] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:35.872303 24238 raft_consensus.cc:2749] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Leader election lost for term 17. Reason: Vote denied by peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651) with higher term. Message: Invalid argument: T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 17. Current term is 18.
W20260709 12:23:35.901000 24290 log.cc:927] Time spent T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.148s	user 0.001s	sys 0.000s
I20260709 12:23:35.877373 24116 consensus_queue.cc:260] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 17.10, Last appended by leader: 10, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:35.873505 24214 leader_election.cc:290] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 19 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:35.903247 24098 leader_election.cc:290] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 18 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:35.878614 23906 raft_consensus.cc:493] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:35.782653 24169 consensus_queue.cc:237] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.10, Last appended by leader: 10, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:35.860150 24203 raft_consensus.cc:515] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:35.859402 23997 raft_consensus.cc:515] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:35.901093 24212 raft_consensus.cc:3055] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 LEADER]: Stepping down as leader of term 14
W20260709 12:23:35.926625 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 55223.8us
I20260709 12:23:35.926842 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:35.928817 17831 raft_consensus.cc:2393] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 15: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:23:35.930066 17831 raft_consensus.cc:1240] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 12. Current term is 13. Ops: [10.9-12.10]
I20260709 12:23:35.917282 24183 raft_consensus.cc:697] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:23:35.938380 17832 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35040 (request call id 2923) took 1601 ms. Trace:
I20260709 12:23:35.938856 17832 rpcz_store.cc:276] 0709 12:23:34.337391 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:34.340860 (+  3469us) service_pool.cc:224] Handling call
0709 12:23:35.938174 (+1597314us) spinlock_profiling.cc:243] Waited 639 ms on lock 0x7b54003354d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:23:35.938322 (+   148us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1470123904}
I20260709 12:23:35.911147 24155 consensus_queue.cc:1048] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.034s
W20260709 12:23:35.945300 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 71823.6us
I20260709 12:23:35.945708 17832 raft_consensus.cc:1275] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 17 index: 15. Preceding OpId from leader: term: 18 index: 16. (index mismatch)
I20260709 12:23:35.946208 17728 leader_election.cc:304] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:35.953392 24229 leader_election.cc:290] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 19 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:35.956269 24210 raft_consensus.cc:3055] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:23:35.956781 24210 raft_consensus.cc:740] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:23:35.957743 24210 consensus_queue.cc:260] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.12, Last appended by leader: 12, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:35.958995 24210 raft_consensus.cc:3060] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:35.960316 23838 raft_consensus.cc:3055] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:23:35.960740 23838 raft_consensus.cc:740] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:23:35.961268 17652 leader_election.cc:304] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:35.961593 23838 consensus_queue.cc:260] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.12, Last appended by leader: 12, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:35.962432 17652 leader_election.cc:304] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:35.964963 24237 raft_consensus.cc:493] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:23:35.965090 23997 leader_election.cc:290] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:35.965653 23931 raft_consensus.cc:2749] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:23:35.965476 24237 raft_consensus.cc:515] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:35.966284 23931 raft_consensus.cc:2764] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Leader pre-election decision vote started in defunct term 16: won
I20260709 12:23:35.967258 24183 consensus_queue.cc:237] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.12, Last appended by leader: 12, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:35.971174 24313 raft_consensus.cc:2749] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:23:35.927359 23906 raft_consensus.cc:515] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:35.962587 23838 raft_consensus.cc:3060] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Advancing to term 17
W20260709 12:23:35.972743 24270 log.cc:927] Time spent T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.544s	user 0.001s	sys 0.001s
I20260709 12:23:35.974004 17685 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37396 (request call id 2686) took 1245 ms. Trace:
I20260709 12:23:35.974326 23906 leader_election.cc:290] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:35.951236 24226 raft_consensus.cc:2749] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260709 12:23:35.975353 24269 log.cc:927] Time spent T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.509s	user 0.002s	sys 0.000s
I20260709 12:23:35.976416 23970 leader_election.cc:290] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 19 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:35.977792 24203 leader_election.cc:290] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 18 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:35.967475 24237 leader_election.cc:290] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:35.979470 17688 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37396 (request call id 2685) took 1294 ms. Trace:
I20260709 12:23:35.981577 24235 raft_consensus.cc:3055] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:23:35.982048 24235 raft_consensus.cc:740] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:23:35.971285 18367 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59886 (request call id 1756) took 1033 ms. Trace:
W20260709 12:23:35.980185 17654 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:35.983809 18367 rpcz_store.cc:276] 0709 12:23:34.937591 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:34.937768 (+   177us) service_pool.cc:224] Handling call
0709 12:23:34.991968 (+ 54200us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:35.388504 (+396536us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:35.388577 (+    73us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:35.526206 (+137629us) log.cc:844] Serialized 26 byte log entry
0709 12:23:35.614234 (+ 88028us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:23:35.614286 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:35.614391 (+   105us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:35.703675 (+ 89284us) raft_consensus.cc:1685] finished
0709 12:23:35.703702 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:35.971039 (+267337us) spinlock_profiling.cc:243] Waited 53.7 ms on lock 0x7b5400802d50. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a61f8ad 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:23:35.971225 (+   186us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":63473,"raft.queue_time_us":83527,"raft.run_cpu_time_us":145,"raft.run_wall_time_us":10000,"spinlock_wait_cycles":123558656,"thread_start_us":376506,"threads_started":2,"wal-append.queue_time_us":160117}
I20260709 12:23:35.985623 17763 raft_consensus.cc:2468] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 15.
I20260709 12:23:35.987008 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:35.988610 24215 raft_consensus.cc:3060] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:35.989766 17681 raft_consensus.cc:1240] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 17. Current term is 18. Ops: []
I20260709 12:23:35.988718 17682 raft_consensus.cc:1240] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 13. Current term is 14. Ops: []
I20260709 12:23:35.992970 17652 leader_election.cc:304] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:35.993682 17836 raft_consensus.cc:1275] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 13 index: 11. (index mismatch)
I20260709 12:23:35.974367 17685 rpcz_store.cc:276] 0709 12:23:34.728907 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:34.887762 (+158855us) service_pool.cc:224] Handling call
0709 12:23:34.888214 (+   452us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:34.888605 (+   391us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:34.888698 (+    93us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:34.889271 (+   573us) log.cc:844] Serialized 26 byte log entry
0709 12:23:35.160748 (+271477us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:23:35.160803 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:35.162620 (+  1817us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:35.972060 (+809440us) spinlock_profiling.cc:243] Waited 126 ms on lock 0x7b4400022b4c. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b47459c85 00007f7b4745a6a2 00007f7b4a1cc1db 00007f7b4a1c7aaf 00007f7b4a1bb86e 00007f7b4a1babb6 00007f7b4a1ba7d8 00007f7b4a1c4221 00007f7b4a1c41d9 00007f7b4a1c4169
0709 12:23:35.973734 (+  1674us) raft_consensus.cc:1685] finished
0709 12:23:35.973756 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:35.973945 (+   189us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":23148,"raft.queue_time_us":29037,"raft.run_cpu_time_us":168,"raft.run_wall_time_us":12762,"spinlock_wait_cycles":290945280,"thread_start_us":270331,"threads_started":1,"wal-append.queue_time_us":478229}
I20260709 12:23:35.997191 18374 raft_consensus.cc:1240] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 15. Current term is 16. Ops: []
I20260709 12:23:35.935647 24212 raft_consensus.cc:740] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:23:35.998449 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:35.999650 18368 raft_consensus.cc:2393] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 11: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:23:35.997795 18334 leader_election.cc:304] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:23:35.981042 17688 rpcz_store.cc:276] 0709 12:23:34.685417 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:34.769851 (+ 84434us) service_pool.cc:224] Handling call
0709 12:23:34.770389 (+   538us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:34.790320 (+ 19931us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:34.790617 (+   297us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:34.829098 (+ 38481us) log.cc:844] Serialized 26 byte log entry
0709 12:23:35.544907 (+715809us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:23:35.544951 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:35.545019 (+    68us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:35.978595 (+433576us) spinlock_profiling.cc:243] Waited 6.05 ms on lock 0x7b2400172c28. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b473b99c4 00007f7b473b7acb 00007f7b47572dc9 00007f7b47572a55 00007f7b47571fb0 00007f7b4a1e8488 00007f7b4a1e8be8 00007f7b4a1c0398 00007f7b4a1bb76a 00007f7b4a1babb6
0709 12:23:35.979205 (+   610us) raft_consensus.cc:1685] finished
0709 12:23:35.979225 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:35.979414 (+   189us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":19256,"raft.run_cpu_time_us":186,"raft.run_wall_time_us":27785,"spinlock_wait_cycles":13921024,"thread_start_us":630794,"threads_started":1,"wal-append.queue_time_us":813981}
I20260709 12:23:36.002064 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:36.003027 18367 raft_consensus.cc:2468] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 16.
I20260709 12:23:36.004393 17836 raft_consensus.cc:1240] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 15. Current term is 16. Ops: []
I20260709 12:23:36.009405 17652 leader_election.cc:304] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
W20260709 12:23:36.010712 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:36.011684 24111 raft_consensus.cc:2804] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:23:36.012456 24111 raft_consensus.cc:697] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 15 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:23:36.012876 24278 raft_consensus.cc:993] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3: : Instructing follower e2909cff1da44ad387f96a8ae3aa5afb to start an election
I20260709 12:23:36.013834 24111 consensus_queue.cc:237] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.11, Last appended by leader: 11, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:36.014791 24283 raft_consensus.cc:493] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:36.006255 24214 consensus_queue.cc:1059] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.012s
I20260709 12:23:36.006443 23737 consensus_queue.cc:1048] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.010s
I20260709 12:23:36.017063 17680 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37402 (request call id 3082) took 1647 ms. Trace:
I20260709 12:23:36.017470 17680 rpcz_store.cc:276] 0709 12:23:34.369406 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:34.369905 (+   499us) service_pool.cc:224] Handling call
0709 12:23:36.016814 (+1646909us) spinlock_profiling.cc:243] Waited 1.24 s on lock 0x7b54002526d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:23:36.016984 (+   170us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2850459008}
I20260709 12:23:36.018581 24244 raft_consensus.cc:2749] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:23:36.020125 24237 raft_consensus.cc:493] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:36.020741 24237 raft_consensus.cc:515] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:36.027107 17730 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:36.028276 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 481775us
I20260709 12:23:36.028681 24272 consensus_queue.cc:1048] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:23:36.029446 24236 raft_consensus.cc:493] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:36.030290 24236 raft_consensus.cc:515] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:36.027001 23737 raft_consensus.cc:493] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:23:36.032440 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:36.039228 17760 raft_consensus.cc:3060] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:36.035485 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:35.982975 24235 consensus_queue.cc:260] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:36.046437 24235 raft_consensus.cc:3060] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:36.036670 24203 consensus_queue.cc:1048] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.014s
I20260709 12:23:36.037854 17652 leader_election.cc:304] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:23:36.045722 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 478113us
I20260709 12:23:36.015347 24283 raft_consensus.cc:515] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:36.048867 17755 raft_consensus.cc:1240] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 LEADER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 17. Current term is 18. Ops: [17.10-17.10]
I20260709 12:23:36.060149 17836 raft_consensus.cc:1275] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 13 index: 13. Preceding OpId from leader: term: 14 index: 14. (index mismatch)
I20260709 12:23:36.052510 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:36.062702 17756 raft_consensus.cc:3060] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:36.052587 17757 raft_consensus.cc:1275] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 15 index: 10. Preceding OpId from leader: term: 20 index: 12. (index mismatch)
I20260709 12:23:36.049149 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 18 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:36.075939 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:36.091632 17838 raft_consensus.cc:2468] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 18.
W20260709 12:23:36.054384 24287 log.cc:927] Time spent T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.532s	user 0.002s	sys 0.000s
I20260709 12:23:36.108647 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:23:36.115808 17804 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:36.120066 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 53824.4us
I20260709 12:23:36.057574 24250 raft_consensus.cc:2749] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 12:23:36.056499 24238 raft_consensus.cc:493] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:36.058120 24247 raft_consensus.cc:493] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:36.130518 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:36.124634 18367 raft_consensus.cc:1240] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 17. Current term is 18. Ops: []
I20260709 12:23:36.131090 17764 raft_consensus.cc:3060] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:36.124926 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 16 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:36.123046 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 19 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:36.126658 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:36.051925 24211 consensus_queue.cc:1059] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.077s
I20260709 12:23:36.049970 24275 raft_consensus.cc:697] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 20 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:23:36.047814 24236 leader_election.cc:290] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:36.068989 24051 raft_consensus.cc:493] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:36.114265 23928 raft_consensus.cc:2749] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:23:36.106112 24088 raft_consensus.cc:1081] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 13 LEADER]: Signalling peer e2909cff1da44ad387f96a8ae3aa5afb to start an election
I20260709 12:23:36.136262 17832 raft_consensus.cc:1240] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 19 LEADER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 18. Current term is 19. Ops: [18.12-18.12]
I20260709 12:23:36.105847 23992 raft_consensus.cc:493] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:36.138098 17756 raft_consensus.cc:2468] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 18.
I20260709 12:23:36.137461 17729 leader_election.cc:304] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:36.144146 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 15 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
W20260709 12:23:36.145218 17655 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:36.118391 24293 raft_consensus.cc:493] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:36.134644 24231 raft_consensus.cc:493] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:36.055492 24288 raft_consensus.cc:515] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:36.126181 17803 leader_election.cc:304] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:36.068136 23737 raft_consensus.cc:515] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:36.148597 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:36.155709 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:36.178208 18367 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 13.
W20260709 12:23:36.178972 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:36.156412 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 118703us
W20260709 12:23:36.159626 17655 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:36.180160 17655 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:36.182785 17652 leader_election.cc:304] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:36.164603 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:36.181986 17801 leader_election.cc:304] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:23:36.184218 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 46820.3us
I20260709 12:23:36.156841 24051 raft_consensus.cc:515] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:36.139999 23905 consensus_queue.cc:1059] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 10.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.044s
W20260709 12:23:36.187664 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 61606.9us
I20260709 12:23:36.186190 24265 raft_consensus.cc:2804] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:23:36.129688 24232 consensus_queue.cc:1048] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.036s
I20260709 12:23:36.191321 17652 leader_election.cc:304] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:36.146423 24141 consensus_queue.cc:1048] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.040s
W20260709 12:23:36.194504 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 47283.5us
I20260709 12:23:36.195350 17803 leader_election.cc:304] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:23:36.196905 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:36.138572 24238 raft_consensus.cc:515] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:36.142488 24247 raft_consensus.cc:515] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:36.218080 24146 raft_consensus.cc:2804] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Leader election won for term 18
W20260709 12:23:36.252137 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 51627.2us
I20260709 12:23:36.207859 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 20 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:36.218451 23970 consensus_queue.cc:1059] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:23:36.182214 24293 raft_consensus.cc:515] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:36.200625 24265 raft_consensus.cc:493] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:23:36.222195 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
W20260709 12:23:36.237481 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:36.292327 17802 leader_election.cc:304] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:36.181232 23992 raft_consensus.cc:515] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:36.241384 17803 leader_election.cc:304] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:36.218955 23651 consensus_queue.cc:1048] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:23:36.242812 17652 leader_election.cc:304] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:36.184201 24236 raft_consensus.cc:493] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:36.231948 24273 raft_consensus.cc:493] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:36.307466 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:36.206126 17764 raft_consensus.cc:2468] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 18.
W20260709 12:23:36.259912 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 58971.7us
I20260709 12:23:36.264775 18333 leader_election.cc:304] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:36.267540 17834 raft_consensus.cc:1275] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 15 index: 13. (index mismatch)
I20260709 12:23:36.197468 24098 raft_consensus.cc:2749] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:23:36.217106 24177 consensus_queue.cc:1048] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:23:36.292769 17836 raft_consensus.cc:2468] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 18.
I20260709 12:23:36.217561 24311 raft_consensus.cc:2804] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:23:36.343638 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:36.346824 17755 raft_consensus.cc:2393] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 17: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:23:36.184675 24231 raft_consensus.cc:515] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:36.350879 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 86190us
I20260709 12:23:36.352087 18333 leader_election.cc:304] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:23:36.306432 17757 raft_consensus.cc:2468] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 18.
W20260709 12:23:36.254479 24182 consensus_meta.cc:287] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 1.188s	user 0.014s	sys 0.012s
I20260709 12:23:36.240572 24262 raft_consensus.cc:3055] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:23:36.335068 17680 raft_consensus.cc:2393] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 19: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:23:36.241227 23928 raft_consensus.cc:2764] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Leader pre-election decision vote started in defunct term 14: won
I20260709 12:23:36.244338 24148 raft_consensus.cc:2804] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:23:36.296633 24265 raft_consensus.cc:3060] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:36.343673 24311 raft_consensus.cc:493] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:23:36.293308 23905 raft_consensus.cc:493] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:36.368870 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:36.374872 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:36.390869 17757 raft_consensus.cc:2393] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 15: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:23:36.383069 18333 leader_election.cc:304] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:23:36.383339 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 49188.4us
I20260709 12:23:36.384765 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:36.394927 17764 raft_consensus.cc:2368] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 21. Current term is 22.
I20260709 12:23:36.291567 24277 consensus_queue.cc:1059] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 13.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.256s
W20260709 12:23:36.398926 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 156174us
I20260709 12:23:36.395412 17801 leader_election.cc:304] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:36.374341 24262 raft_consensus.cc:740] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 10 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:23:36.378988 24304 raft_consensus.cc:493] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:36.418010 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:36.373739 23970 raft_consensus.cc:493] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:23:36.436760 23970 raft_consensus.cc:515] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:36.439460 23970 leader_election.cc:290] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:23:36.425451 17635 auto_rebalancer.cc:629] Failed to schedule move for tablet a9aeb14eca4a43dcbeaccffdf7e28995: Illegal state: Replica 0da169ff82674dc1947055ce43de68a3 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 12 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } }
I20260709 12:23:36.439661 17760 raft_consensus.cc:2468] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 19.
I20260709 12:23:36.379428 24148 raft_consensus.cc:493] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:36.442814 24148 raft_consensus.cc:3060] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:36.443174 17760 raft_consensus.cc:1240] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 14. Current term is 15. Ops: []
I20260709 12:23:36.445178 17760 raft_consensus.cc:1275] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 17 index: 15. Preceding OpId from leader: term: 18 index: 16. (index mismatch)
I20260709 12:23:36.381894 24250 raft_consensus.cc:493] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:36.376864 24088 raft_consensus.cc:2749] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:23:36.428406 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:36.433326 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:36.398541 24203 raft_consensus.cc:2804] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:23:36.401134 24141 raft_consensus.cc:2749] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
W20260709 12:23:36.446873 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 51558.2us
I20260709 12:23:36.391687 23931 raft_consensus.cc:493] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:36.459832 17801 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:36.393857 24311 raft_consensus.cc:3060] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:36.390241 24146 raft_consensus.cc:697] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 18 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:23:36.382167 24246 raft_consensus.cc:2749] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:23:36.343182 24236 raft_consensus.cc:515] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:36.400761 24009 consensus_queue.cc:1048] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 8, Time since last communication: 0.060s
I20260709 12:23:36.380149 24182 raft_consensus.cc:515] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:36.420526 24264 raft_consensus.cc:2749] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:23:36.461220 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
W20260709 12:23:36.483713 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 148653us
I20260709 12:23:36.460964 17801 leader_election.cc:304] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:36.428272 24283 leader_election.cc:290] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:36.474391 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:36.487804 17834 raft_consensus.cc:2393] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 19: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:23:36.488267 17682 raft_consensus.cc:2468] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 16.
I20260709 12:23:36.489585 17764 raft_consensus.cc:1275] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 13 index: 12. Preceding OpId from leader: term: 15 index: 13. (index mismatch)
I20260709 12:23:36.401181 23905 raft_consensus.cc:515] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:36.492542 23737 leader_election.cc:290] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:36.492826 23905 leader_election.cc:290] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:36.493803 17835 raft_consensus.cc:2393] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 17: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:23:36.494275 24182 leader_election.cc:290] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 17 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:36.483265 17652 leader_election.cc:304] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:36.461966 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:36.348229 24275 consensus_queue.cc:237] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 17.10, Last appended by leader: 10, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:36.495903 24283 raft_consensus.cc:2804] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Leader pre-election won for term 16
W20260709 12:23:36.496376 24315 log.cc:927] Time spent T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.233s	user 0.000s	sys 0.001s
I20260709 12:23:36.496711 24283 raft_consensus.cc:493] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:23:36.497184 24283 raft_consensus.cc:3060] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Advancing to term 16
W20260709 12:23:36.484962 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 191131us
I20260709 12:23:36.373596 24273 raft_consensus.cc:515] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:36.434728 23928 consensus_queue.cc:1059] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.037s
I20260709 12:23:36.489507 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:36.500481 17682 raft_consensus.cc:3060] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:36.476397 24278 raft_consensus.cc:493] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:23:36.500686 17761 raft_consensus.cc:3060] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:36.501116 24278 raft_consensus.cc:515] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:36.504179 24331 raft_consensus.cc:493] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:36.505085 24182 raft_consensus.cc:2804] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:23:36.504741 24331 raft_consensus.cc:515] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:36.505765 24182 raft_consensus.cc:697] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 19 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
W20260709 12:23:36.482468 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 241166us
I20260709 12:23:36.508421 24238 leader_election.cc:290] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:36.509260 17803 leader_election.cc:304] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:36.464684 24088 consensus_queue.cc:1059] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 15.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.045s
I20260709 12:23:36.511206 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 18 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:36.470382 24203 raft_consensus.cc:697] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 16 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:23:36.511821 24088 raft_consensus.cc:3055] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:23:36.473379 23931 raft_consensus.cc:515] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:36.512302 24088 raft_consensus.cc:740] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
W20260709 12:23:36.495714 24227 consensus_meta.cc:287] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.590s	user 0.011s	sys 0.000s
I20260709 12:23:36.450884 24304 raft_consensus.cc:515] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:36.497299 24316 log.cc:927] Time spent T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.190s	user 0.001s	sys 0.000s
I20260709 12:23:36.464672 24211 raft_consensus.cc:2749] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:23:36.518023 17803 leader_election.cc:304] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:36.462826 24051 leader_election.cc:290] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:36.518841 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:36.528000 17683 raft_consensus.cc:2393] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 17: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:23:36.462633 24059 consensus_queue.cc:1059] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 13.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.060s
I20260709 12:23:36.531736 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:36.532296 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 15 candidate_status { last_received { term: 14 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:36.532418 17762 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 13.
I20260709 12:23:36.533439 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 19 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:36.533681 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 20 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:36.533998 17762 raft_consensus.cc:2468] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 18.
I20260709 12:23:36.534252 24237 leader_election.cc:290] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:36.534260 17683 raft_consensus.cc:2468] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 19.
I20260709 12:23:36.534873 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 20 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:36.535471 17832 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37994 (request call id 1239) took 1183 ms. Trace:
I20260709 12:23:36.508649 24182 consensus_queue.cc:237] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 17.9, Last appended by leader: 9, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:36.536242 24328 raft_consensus.cc:2749] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 12:23:36.536005 17832 rpcz_store.cc:276] 0709 12:23:35.352536 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:35.946758 (+594222us) service_pool.cc:224] Handling call
0709 12:23:35.947072 (+   314us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:36.158010 (+210938us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:36.535191 (+377181us) spinlock_profiling.cc:243] Waited 189 ms on lock 0x7b54002eac54. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:23:36.535398 (+   207us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":434483072}
I20260709 12:23:36.537631 24328 raft_consensus.cc:2749] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:23:36.538163 17759 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38776 (request call id 2628) took 2281 ms. Trace:
I20260709 12:23:36.538594 17759 rpcz_store.cc:276] 0709 12:23:34.256946 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:34.272041 (+ 15095us) service_pool.cc:224] Handling call
0709 12:23:35.872130 (+1600089us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:35.874706 (+  2576us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:36.537930 (+663224us) spinlock_profiling.cc:243] Waited 1.6 s on lock 0x7b5400310e50. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a61f8ad 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:23:36.538073 (+   143us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3679108352}
I20260709 12:23:36.540484 17761 raft_consensus.cc:2468] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 14.
I20260709 12:23:36.538921 17803 leader_election.cc:400] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 18 election: Vote denied by peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577) with higher term. Message: Invalid argument: T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 18. Current term is 19.
I20260709 12:23:36.543105 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:36.544559 17759 raft_consensus.cc:2393] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 19: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:23:36.544692 17682 raft_consensus.cc:2468] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 18.
W20260709 12:23:36.544809 17730 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:36.511025 24288 leader_election.cc:290] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 15 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:36.548264 17803 leader_election.cc:304] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:36.455339 24250 raft_consensus.cc:515] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:36.488639 24141 raft_consensus.cc:2804] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:23:36.558902 17682 raft_consensus.cc:1275] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 19 index: 12. Preceding OpId from leader: term: 20 index: 12. (term mismatch)
I20260709 12:23:36.559574 17682 pending_rounds.cc:85] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3: Aborting all ops after (but not including) 11
I20260709 12:23:36.559981 17682 pending_rounds.cc:107] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3: Aborting uncommitted NO_OP operation due to leader change: 19.12
I20260709 12:23:36.552852 23905 consensus_queue.cc:1048] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 9, Time since last communication: 0.009s
I20260709 12:23:36.562279 24331 leader_election.cc:290] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:36.564852 24247 leader_election.cc:290] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:23:36.579706 24333 log.cc:927] Time spent T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.207s	user 0.000s	sys 0.001s
I20260709 12:23:36.570325 24146 consensus_queue.cc:237] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.11, Last appended by leader: 11, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:36.627132 17685 raft_consensus.cc:1275] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 16 index: 12. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
I20260709 12:23:36.571687 17757 raft_consensus.cc:1240] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 LEADER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 12. Current term is 13. Ops: []
I20260709 12:23:36.560490 17682 raft_consensus.cc:2889] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:36.629598 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:36.583129 24212 consensus_queue.cc:260] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.12, Last appended by leader: 12, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:36.583761 24273 leader_election.cc:290] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:23:36.586694 24256 log.cc:927] Time spent T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 1.605s	user 0.000s	sys 0.011s
I20260709 12:23:36.589133 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
W20260709 12:23:36.589257 18334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:36.633252 17652 leader_election.cc:304] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:36.593492 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:36.593948 24236 leader_election.cc:290] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:23:36.596177 23838 consensus_meta.cc:287] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.623s	user 0.003s	sys 0.008s
W20260709 12:23:36.596519 24235 consensus_meta.cc:287] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.550s	user 0.005s	sys 0.005s
I20260709 12:23:36.573990 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:23:36.598793 24210 consensus_meta.cc:287] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.639s	user 0.003s	sys 0.010s
I20260709 12:23:36.563516 24278 leader_election.cc:290] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:36.568768 24293 leader_election.cc:290] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:36.604816 24203 consensus_queue.cc:237] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.10, Last appended by leader: 10, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:36.606297 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 58120us
I20260709 12:23:36.610450 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:36.612802 17756 raft_consensus.cc:1275] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 18 index: 11. Preceding OpId from leader: term: 19 index: 12. (index mismatch)
I20260709 12:23:36.615547 24141 raft_consensus.cc:697] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 18 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:23:36.504782 24262 consensus_queue.cc:260] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:36.615547 24331 raft_consensus.cc:2749] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:23:36.576514 24231 leader_election.cc:290] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:36.624243 24304 leader_election.cc:290] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:36.569307 17835 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35042 (request call id 2626) took 1405 ms. Trace:
I20260709 12:23:36.628137 23906 raft_consensus.cc:3055] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 17 LEADER]: Stepping down as leader of term 17
W20260709 12:23:36.630923 24215 consensus_meta.cc:287] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.642s	user 0.006s	sys 0.004s
I20260709 12:23:36.695827 17831 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 13.
I20260709 12:23:36.718240 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:36.700671 17837 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35040 (request call id 2928) took 2361 ms. Trace:
I20260709 12:23:36.631243 24247 raft_consensus.cc:493] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:36.646943 24229 raft_consensus.cc:2749] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:23:36.703154 24148 raft_consensus.cc:515] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:36.704886 17834 raft_consensus.cc:2468] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 19.
W20260709 12:23:36.711742 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 328731us
I20260709 12:23:36.711388 17835 rpcz_store.cc:276] 0709 12:23:35.163645 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:35.178730 (+ 15085us) service_pool.cc:224] Handling call
0709 12:23:36.569073 (+1390343us) spinlock_profiling.cc:243] Waited 1.28 s on lock 0x7b54000aa5d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:23:36.569224 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2947486464}
I20260709 12:23:36.729213 18333 leader_election.cc:304] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:36.713688 17756 raft_consensus.cc:1240] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 18. Current term is 19. Ops: []
I20260709 12:23:36.646096 24311 raft_consensus.cc:515] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:36.715209 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:36.719398 17837 rpcz_store.cc:276] 0709 12:23:34.339079 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:34.480739 (+141660us) service_pool.cc:224] Handling call
0709 12:23:34.481163 (+   424us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:35.684732 (+1203569us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:36.700320 (+1015588us) spinlock_profiling.cc:243] Waited 1.2 s on lock 0x7b54003340d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:23:36.700559 (+   239us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2752025344}
I20260709 12:23:36.707576 24212 raft_consensus.cc:3060] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:36.720853 17652 leader_election.cc:304] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:36.655316 24273 raft_consensus.cc:493] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:36.702129 24244 raft_consensus.cc:493] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:36.726569 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:23:36.729763 17804 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:36.732327 18333 leader_election.cc:304] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:36.735591 17652 leader_election.cc:304] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:23:36.732337 18366 raft_consensus.cc:1240] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 15 LEADER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 14. Current term is 15. Ops: [14.12-14.12]
I20260709 12:23:36.733681 17684 raft_consensus.cc:1275] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 13 index: 13. Preceding OpId from leader: term: 14 index: 14. (index mismatch)
I20260709 12:23:36.714838 23906 raft_consensus.cc:740] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 17 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:23:36.724419 24304 raft_consensus.cc:493] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:23:36.743451 17757 raft_consensus.cc:1240] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 17. Current term is 18. Ops: []
I20260709 12:23:36.743165 17758 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38776 (request call id 2630) took 2446 ms. Trace:
I20260709 12:23:36.752118 17757 raft_consensus.cc:1240] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 19. Current term is 20. Ops: []
I20260709 12:23:36.751281 17758 rpcz_store.cc:276] 0709 12:23:34.296381 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:34.449822 (+153441us) service_pool.cc:224] Handling call
0709 12:23:34.450162 (+   340us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:34.450532 (+   370us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:34.450563 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:34.451063 (+   500us) log.cc:844] Serialized 26 byte log entry
0709 12:23:34.801253 (+350190us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:23:34.801311 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:34.801406 (+    95us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:36.454807 (+1653401us) spinlock_profiling.cc:243] Waited 42.9 ms on lock 0x7b4400022b4c. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b47459c85 00007f7b4745a6a2 00007f7b4a1cc1db 00007f7b4a1c7aaf 00007f7b4a1bb86e 00007f7b4a1babb6 00007f7b4a1ba7d8 00007f7b4a1c4221 00007f7b4a1c41d9 00007f7b4a1c4169
0709 12:23:36.586352 (+131545us) spinlock_profiling.cc:243] Waited 25.1 ms on lock 0x7b24000f7388. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b473b99c4 00007f7b473b7acb 00007f7b47572dc9 00007f7b47572a55 00007f7b4757209d 00007f7b4a1a410e 00007f7b4a1bb88c 00007f7b4a1babb6 00007f7b4a1ba7d8 00007f7b4a1c4221
0709 12:23:36.742923 (+156571us) raft_consensus.cc:1685] finished
0709 12:23:36.742949 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:36.743104 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":19899,"raft.run_cpu_time_us":82,"raft.run_wall_time_us":84,"spinlock_wait_cycles":156396800,"thread_start_us":349750,"threads_started":1,"wal-append.queue_time_us":425189}
I20260709 12:23:36.747072 24229 raft_consensus.cc:2804] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:23:36.754160 24229 raft_consensus.cc:493] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:36.754515 24229 raft_consensus.cc:3060] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:36.754910 17758 raft_consensus.cc:1275] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 20 index: 16. Preceding OpId from leader: term: 22 index: 17. (index mismatch)
I20260709 12:23:36.743790 18333 leader_election.cc:304] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:36.755972 17758 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0"
dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
 from {username='slave'} at 127.0.0.1:38776
I20260709 12:23:36.757318 18333 leader_election.cc:304] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:36.758924 18333 leader_election.cc:304] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
W20260709 12:23:36.762370 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:36.764222 24141 consensus_queue.cc:237] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.11, Last appended by leader: 11, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:36.739208 24215 raft_consensus.cc:2804] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:23:36.768970 24215 raft_consensus.cc:493] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:36.769446 24215 raft_consensus.cc:3060] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:36.771983 24236 consensus_queue.cc:1059] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 17.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:23:36.736711 24337 raft_consensus.cc:2749] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 12:23:36.774712 24279 raft_consensus.cc:3055] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:23:36.776739 24148 leader_election.cc:290] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 16 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:36.744910 17831 raft_consensus.cc:2393] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 16: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:23:36.753429 17757 raft_consensus.cc:1240] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 13. Current term is 14. Ops: []
I20260709 12:23:36.773176 24236 raft_consensus.cc:3055] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:23:36.742662 24151 raft_consensus.cc:2749] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:23:36.787659 17758 raft_consensus.cc:493] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:23:36.788424 17758 raft_consensus.cc:3060] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:36.730639 24247 raft_consensus.cc:515] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:36.778725 24311 leader_election.cc:290] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 16 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:36.787698 24235 raft_consensus.cc:2749] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:23:36.810690 24265 raft_consensus.cc:515] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:36.785342 24210 raft_consensus.cc:3060] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:36.784972 24231 raft_consensus.cc:2749] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:23:36.741036 24244 raft_consensus.cc:515] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:36.739429 24273 raft_consensus.cc:515] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:36.786419 24236 raft_consensus.cc:740] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:23:36.823171 17758 raft_consensus.cc:515] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:36.832767 17835 raft_consensus.cc:1240] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 15. Current term is 16. Ops: []
I20260709 12:23:36.793179 24279 raft_consensus.cc:740] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
W20260709 12:23:36.831308 23953 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.284s	user 0.005s	sys 0.003s
I20260709 12:23:36.823241 24276 raft_consensus.cc:493] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:36.780275 24304 raft_consensus.cc:515] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:36.846109 23953 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.313s	user 0.005s	sys 0.003s
I20260709 12:23:36.846380 24276 raft_consensus.cc:515] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:36.847872 24304 leader_election.cc:290] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:36.848832 24276 leader_election.cc:290] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:36.790958 23838 consensus_queue.cc:1059] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 9.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.045s
I20260709 12:23:36.833272 24278 raft_consensus.cc:493] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:36.841560 24322 log.cc:927] Time spent T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.402s	user 0.000s	sys 0.002s
I20260709 12:23:36.856896 17759 raft_consensus.cc:1240] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 18. Current term is 19. Ops: []
I20260709 12:23:36.831802 24151 consensus_queue.cc:1048] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.019s
I20260709 12:23:36.867420 17763 raft_consensus.cc:2410] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for term 15 because replica has last-logged OpId of term: 14 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 12.
I20260709 12:23:36.834877 24211 consensus_queue.cc:1048] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.025s
I20260709 12:23:36.853811 24231 raft_consensus.cc:493] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:36.870683 24265 leader_election.cc:290] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 16 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:36.852989 24235 raft_consensus.cc:493] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:36.830442 24337 consensus_queue.cc:1048] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.031s
W20260709 12:23:36.917313 17728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:36.828634 23906 consensus_queue.cc:260] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.14, Last appended by leader: 14, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:36.919689 24349 log.cc:927] Time spent T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.053s	user 0.000s	sys 0.000s
I20260709 12:23:36.922459 24329 raft_consensus.cc:493] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:23:36.923653 24329 raft_consensus.cc:515] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:36.917368 24235 raft_consensus.cc:515] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:36.876847 24340 log.cc:927] Time spent T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.164s	user 0.001s	sys 0.000s
I20260709 12:23:36.862006 24278 raft_consensus.cc:515] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:36.869099 23928 consensus_queue.cc:1059] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 15.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.062s
I20260709 12:23:36.927560 24246 raft_consensus.cc:493] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
W20260709 12:23:36.940037 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:36.942085 24278 leader_election.cc:290] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:36.876968 24231 raft_consensus.cc:515] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:36.942454 23992 leader_election.cc:290] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:36.943324 24235 leader_election.cc:290] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:36.945572 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:36.937852 24283 raft_consensus.cc:515] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:36.952764 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:36.964708 17683 raft_consensus.cc:3060] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:36.960281 17764 raft_consensus.cc:2468] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 19.
I20260709 12:23:36.960947 23928 consensus_queue.cc:1059] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 14.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20260709 12:23:36.961784 24339 consensus_queue.cc:1059] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 12:23:36.963274 17686 raft_consensus.cc:1275] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 16 index: 12. Preceding OpId from leader: term: 18 index: 13. (index mismatch)
I20260709 12:23:36.876363 24236 consensus_queue.cc:260] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.11, Last appended by leader: 11, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:36.969414 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:36.970578 17686 raft_consensus.cc:2468] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 19.
I20260709 12:23:36.931634 24279 consensus_queue.cc:260] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.13, Last appended by leader: 13, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:36.973045 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:36.973888 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:36.974591 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:36.978856 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:36.966449 24329 leader_election.cc:290] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:36.966953 24227 consensus_queue.cc:1059] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 13.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.013s
I20260709 12:23:36.959484 24246 raft_consensus.cc:515] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:36.986728 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 20 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:36.984213 24339 consensus_queue.cc:1059] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 15.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.016s
I20260709 12:23:36.983354 24279 raft_consensus.cc:3060] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:37.020855 17682 raft_consensus.cc:1275] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 17 index: 9. Preceding OpId from leader: term: 18 index: 10. (index mismatch)
I20260709 12:23:37.023720 17684 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44898 (request call id 1240) took 1032 ms. Trace:
I20260709 12:23:37.032524 17684 rpcz_store.cc:276] 0709 12:23:35.991487 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:35.993803 (+  2316us) service_pool.cc:224] Handling call
0709 12:23:35.994050 (+   247us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:36.738104 (+744054us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:37.023475 (+285371us) spinlock_profiling.cc:243] Waited 738 ms on lock 0x7b54003467d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:23:37.023658 (+   183us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1697189632}
I20260709 12:23:37.023265 17829 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35042 (request call id 2631) took 1840 ms. Trace:
W20260709 12:23:37.024534 18334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:37.026957 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:37.038213 17831 raft_consensus.cc:3060] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:37.027781 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:37.040040 17682 raft_consensus.cc:1275] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 15 index: 9. Preceding OpId from leader: term: 17 index: 10. (index mismatch)
W20260709 12:23:37.030081 17730 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:36.967782 24237 consensus_queue.cc:1048] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.017s
I20260709 12:23:37.039911 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:36.972117 24278 consensus_queue.cc:1059] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 17.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.016s
I20260709 12:23:37.051534 17758 leader_election.cc:290] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 15 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:37.036285 17829 rpcz_store.cc:276] 0709 12:23:35.182980 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:35.217227 (+ 34247us) service_pool.cc:224] Handling call
0709 12:23:35.217601 (+   374us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:35.433028 (+215427us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 12:23:35.614427 (+181399us) log.cc:844] Serialized 14 byte log entry
0709 12:23:36.052278 (+437851us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:36.052328 (+    50us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:23:36.052370 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:37.019822 (+967452us) spinlock_profiling.cc:243] Waited 215 ms on lock 0x7b54003359d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:23:37.023003 (+  3181us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:37.023160 (+   157us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":401076,"prepare.queue_time_us":64925,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":37,"raft.queue_time_us":401462,"raft.run_cpu_time_us":92,"raft.run_wall_time_us":92,"spinlock_wait_cycles":494816640,"thread_start_us":152746,"threads_started":2,"wal-append.queue_time_us":212656,"wal-append.run_cpu_time_us":1944,"wal-append.run_wall_time_us":1078042}
I20260709 12:23:36.986636 24273 leader_election.cc:290] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:36.986467 24247 leader_election.cc:290] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:23:37.051460 24351 log.cc:927] Time spent T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.115s	user 0.001s	sys 0.000s
I20260709 12:23:37.073235 17683 raft_consensus.cc:2468] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 19.
I20260709 12:23:37.023005 24215 raft_consensus.cc:515] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:37.071127 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 19 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:23:37.079232 17803 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:37.030705 24304 raft_consensus.cc:3055] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 12 LEADER]: Stepping down as leader of term 12
W20260709 12:23:37.072791 24354 log.cc:927] Time spent T 00000000000000000000000000000000 P 2193af31cb894ec9b3d65d939a2a6851: Append to log took a long time: real 0.150s	user 0.005s	sys 0.005s
I20260709 12:23:37.073828 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
W20260709 12:23:37.064059 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 97592.2us
I20260709 12:23:37.087795 17654 leader_election.cc:304] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:23:37.048019 24229 raft_consensus.cc:515] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:37.084589 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:37.056877 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 17 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:23:37.078223 18333 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:37.058934 24246 leader_election.cc:290] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:37.074630 24250 leader_election.cc:290] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:37.109519 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 18 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:37.109669 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:37.110569 18368 raft_consensus.cc:2410] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 18 because replica has last-logged OpId of term: 17 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 12.
I20260709 12:23:37.107335 17756 raft_consensus.cc:2410] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for term 17 because replica has last-logged OpId of term: 16 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 9.
I20260709 12:23:37.108105 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:37.110206 17686 raft_consensus.cc:2393] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 14: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:23:37.111511 24151 consensus_queue.cc:1059] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:23:37.072079 24265 consensus_queue.cc:1059] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 13.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.020s
I20260709 12:23:37.113163 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:37.085520 24235 raft_consensus.cc:493] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:37.115363 18367 raft_consensus.cc:2468] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 16.
I20260709 12:23:37.119038 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:37.119235 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:37.119799 18371 raft_consensus.cc:2468] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 17.
I20260709 12:23:37.119781 24211 consensus_queue.cc:1048] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:23:37.120409 24237 raft_consensus.cc:2804] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:23:37.081760 24304 raft_consensus.cc:740] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:23:37.083328 17831 raft_consensus.cc:2468] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 16.
I20260709 12:23:37.122843 17760 raft_consensus.cc:1240] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 LEADER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 17. Current term is 18. Ops: []
I20260709 12:23:37.123965 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:37.124835 18362 raft_consensus.cc:2410] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 19 because replica has last-logged OpId of term: 18 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 11.
I20260709 12:23:37.128759 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 19 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:37.129520 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:37.129959 17764 raft_consensus.cc:2393] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 19: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:23:37.132601 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 15 candidate_status { last_received { term: 14 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:37.135818 18365 raft_consensus.cc:2393] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 15: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:23:37.135895 17682 raft_consensus.cc:1275] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 13 index: 12. (index mismatch)
I20260709 12:23:37.137621 17689 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:54540 (request call id 41) took 2221 ms. Trace:
I20260709 12:23:37.120796 24237 raft_consensus.cc:493] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:23:37.138057 17689 rpcz_store.cc:276] 0709 12:23:34.910273 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:34.981283 (+ 71010us) service_pool.cc:224] Handling call
0709 12:23:37.131054 (+2149771us) spinlock_profiling.cc:243] Waited 691 ms on lock 0x7b54004259d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a61a853 00007f7b4f5e860a 00007f7b48b6e7d4 00007f7b48b6e762 00007f7b48b6e691 00007f7b48b6e60c 00007f7b48b6d8c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:23:37.131209 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1588518656}
I20260709 12:23:37.140537 18367 raft_consensus.cc:1275] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 18 index: 12. Preceding OpId from leader: term: 19 index: 12. (term mismatch)
I20260709 12:23:37.140956 18367 pending_rounds.cc:85] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da: Aborting all ops after (but not including) 11
I20260709 12:23:37.141316 18367 pending_rounds.cc:107] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da: Aborting uncommitted NO_OP operation due to leader change: 18.12
I20260709 12:23:37.141654 18367 raft_consensus.cc:2889] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:37.149466 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:37.150837 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 14 candidate_status { last_received { term: 11 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:37.125129 24283 leader_election.cc:290] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 16 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:37.156105 17682 raft_consensus.cc:1240] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 17. Current term is 18. Ops: [17.13-17.13]
I20260709 12:23:37.157838 17682 raft_consensus.cc:1275] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 13 index: 11. Preceding OpId from leader: term: 15 index: 12. (index mismatch)
I20260709 12:23:37.155766 17652 leader_election.cc:304] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:37.159319 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 19 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:37.164552 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
W20260709 12:23:37.168889 17830 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35042 (request call id 2624) took 2016 ms (client timeout 1559 ms). Trace:
I20260709 12:23:37.169253 24087 raft_consensus.cc:3055] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:23:37.169739 24087 raft_consensus.cc:740] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:23:37.170717 24215 leader_election.cc:290] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 14 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:37.171013 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:37.171499 24215 raft_consensus.cc:2804] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:23:37.155099 24358 raft_consensus.cc:493] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:37.171769 18362 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 13.
I20260709 12:23:37.172297 24358 raft_consensus.cc:515] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:37.172631 24088 consensus_queue.cc:260] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.10, Last appended by leader: 10, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:37.174430 24088 raft_consensus.cc:3060] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:37.175316 24358 leader_election.cc:290] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:37.151839 23928 raft_consensus.cc:493] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:23:37.155221 24359 raft_consensus.cc:493] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:37.176342 23928 raft_consensus.cc:515] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:37.176576 24359 raft_consensus.cc:515] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:37.177503 24229 leader_election.cc:290] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 20 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:37.178898 23928 leader_election.cc:290] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:37.179214 24236 raft_consensus.cc:3060] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:37.179484 24359 leader_election.cc:290] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:37.154943 24356 raft_consensus.cc:3055] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:23:37.179739 16501 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260709 12:23:37.149365 24237 raft_consensus.cc:3060] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Advancing to term 17
W20260709 12:23:37.169385 17830 rpcz_store.cc:269] 0709 12:23:35.152344 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:35.156744 (+  4400us) service_pool.cc:224] Handling call
0709 12:23:37.168701 (+2011957us) spinlock_profiling.cc:243] Waited 592 ms on lock 0x7b540022b854. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:23:37.168813 (+   112us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1362026752}
I20260709 12:23:37.122694 24235 raft_consensus.cc:515] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:37.189527 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:37.192315 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 15 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:37.192648 17829 raft_consensus.cc:3060] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:37.192966 17830 raft_consensus.cc:2468] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 14.
I20260709 12:23:37.191330 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:37.180338 24304 consensus_queue.cc:260] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.10, Last appended by leader: 10, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:37.170718 24087 consensus_queue.cc:260] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 19.11, Last appended by leader: 11, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:37.186626 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:37.195405 17831 raft_consensus.cc:3060] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:37.179211 24210 raft_consensus.cc:2749] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:23:37.155120 24231 leader_election.cc:290] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:37.201370 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 21 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:37.189307 24356 raft_consensus.cc:740] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:23:37.229092 17836 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35042 (request call id 2640) took 1106 ms. Trace:
I20260709 12:23:37.229621 17836 rpcz_store.cc:276] 0709 12:23:36.122510 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:36.122677 (+   167us) service_pool.cc:224] Handling call
0709 12:23:37.228871 (+1106194us) spinlock_profiling.cc:243] Waited 160 ms on lock 0x7b5400089dd4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:23:37.229020 (+   149us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":367363840}
I20260709 12:23:37.273830 17831 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 14.
I20260709 12:23:37.211978 24304 raft_consensus.cc:3060] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:37.188983 24211 consensus_queue.cc:1048] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.031s
I20260709 12:23:37.202757 24203 consensus_queue.cc:1048] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.083s
W20260709 12:23:37.348783 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:37.350104 17838 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35042 (request call id 2632) took 2166 ms (client timeout 1614 ms). Trace:
W20260709 12:23:37.350538 17838 rpcz_store.cc:269] 0709 12:23:35.183261 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:35.197115 (+ 13854us) service_pool.cc:224] Handling call
0709 12:23:37.349902 (+2152787us) spinlock_profiling.cc:243] Waited 708 ms on lock 0x7b54003cc8d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:23:37.350017 (+   115us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1628520832}
I20260709 12:23:37.281392 17829 raft_consensus.cc:2468] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 20.
I20260709 12:23:37.293860 18333 leader_election.cc:304] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:37.294246 16501 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:23:37.295313 24362 raft_consensus.cc:493] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:37.352591 24362 raft_consensus.cc:515] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:37.353230 16501 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:23:37.355050 24227 raft_consensus.cc:3055] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:23:37.355500 24227 raft_consensus.cc:740] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:23:37.296406 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:37.356607 18363 raft_consensus.cc:3060] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:37.357456 24227 consensus_queue.cc:260] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.11, Last appended by leader: 11, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:37.358510 24283 raft_consensus.cc:493] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:37.358687 24227 raft_consensus.cc:3060] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:37.359053 24283 raft_consensus.cc:515] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:37.230306 24087 raft_consensus.cc:3060] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:37.360143 24366 raft_consensus.cc:493] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:37.360630 24366 raft_consensus.cc:515] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:37.361265 24283 leader_election.cc:290] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:37.362723 24366 leader_election.cc:290] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:37.312731 17755 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38786 (request call id 3362) took 1268 ms. Trace:
I20260709 12:23:37.312888 17757 raft_consensus.cc:1240] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 16. Current term is 17. Ops: []
I20260709 12:23:37.363489 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:37.214336 24215 raft_consensus.cc:697] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 16 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:23:37.364593 24215 consensus_queue.cc:237] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.10, Last appended by leader: 10, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:37.365168 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:37.365721 17757 raft_consensus.cc:3060] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:37.363507 17755 rpcz_store.cc:276] 0709 12:23:36.044553 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:36.069426 (+ 24873us) service_pool.cc:224] Handling call
0709 12:23:37.312440 (+1243014us) spinlock_profiling.cc:243] Waited 229 ms on lock 0x7b54004150d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:23:37.312647 (+   207us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":526911744}
I20260709 12:23:37.370076 24116 raft_consensus.cc:3060] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:37.370393 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 21 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:37.372421 24362 leader_election.cc:290] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:23:37.373494 17727 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.16.29.67:36837 timed out after 1.614s (SENT) [suppressed 17 similar messages]
I20260709 12:23:37.374871 24247 raft_consensus.cc:3055] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:23:37.375278 24247 raft_consensus.cc:740] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:23:37.379083 24262 raft_consensus.cc:3060] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:37.329890 24235 leader_election.cc:290] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:37.331282 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 19 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:37.320516 24356 consensus_queue.cc:260] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.11, Last appended by leader: 11, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:37.389709 24247 consensus_queue.cc:260] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 14.14, Last appended by leader: 14, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:37.371652 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:37.402420 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 16 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:23:37.424041 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:37.441169 17685 raft_consensus.cc:2410] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for term 19 because replica has last-logged OpId of term: 18 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 12.
W20260709 12:23:37.440492 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 146778us
I20260709 12:23:37.445740 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 23 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:37.446084 18333 leader_election.cc:304] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:23:37.440989 24356 raft_consensus.cc:3060] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:37.441256 17689 raft_consensus.cc:1275] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 15 index: 10. Preceding OpId from leader: term: 16 index: 11. (index mismatch)
I20260709 12:23:37.443398 17763 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36572 (request call id 1133) took 1327 ms. Trace:
I20260709 12:23:37.367664 18367 raft_consensus.cc:2468] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 20.
W20260709 12:23:37.447219 18334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:37.450644 18363 raft_consensus.cc:2468] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 16.
I20260709 12:23:37.447901 17763 rpcz_store.cc:276] 0709 12:23:36.116318 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:36.221765 (+105447us) service_pool.cc:224] Handling call
0709 12:23:37.388861 (+1167096us) spinlock_profiling.cc:243] Waited 575 ms on lock 0x7b540009f1d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:23:37.443340 (+ 54479us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1321601792}
I20260709 12:23:37.458631 18333 leader_election.cc:400] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 21 election: Vote denied by peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651) with higher term. Message: Invalid argument: T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 21. Current term is 22.
I20260709 12:23:37.460386 18333 leader_election.cc:304] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:37.461051 18366 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59886 (request call id 1766) took 1625 ms. Trace:
I20260709 12:23:37.462718 18366 rpcz_store.cc:276] 0709 12:23:35.835196 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:35.835352 (+   156us) service_pool.cc:224] Handling call
0709 12:23:35.861265 (+ 25913us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:36.737805 (+876540us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:37.460793 (+722988us) spinlock_profiling.cc:243] Waited 871 ms on lock 0x7b54006aed54. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:23:37.460990 (+   197us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2002762496}
W20260709 12:23:37.465597 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:37.467494 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 94030.3us
I20260709 12:23:37.469184 17757 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 14.
I20260709 12:23:37.443182 24277 raft_consensus.cc:3055] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:23:37.446848 24235 raft_consensus.cc:2749] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:23:37.443231 23953 raft_consensus.cc:493] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:23:37.482811 17760 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36572 (request call id 1138) took 1361 ms. Trace:
W20260709 12:23:37.484390 18334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:37.483271 17760 rpcz_store.cc:276] 0709 12:23:36.121389 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:36.452171 (+330782us) service_pool.cc:224] Handling call
0709 12:23:36.468441 (+ 16270us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:37.123347 (+654906us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:37.482548 (+359201us) spinlock_profiling.cc:243] Waited 654 ms on lock 0x7b54002b6454. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:23:37.482744 (+   196us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1504482688}
I20260709 12:23:37.467689 24203 raft_consensus.cc:2804] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:23:37.476925 24244 leader_election.cc:290] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:37.479611 17832 raft_consensus.cc:2468] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 11.
I20260709 12:23:37.472461 24277 raft_consensus.cc:740] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 14 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:23:37.472667 18333 leader_election.cc:304] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:37.483253 23953 raft_consensus.cc:515] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:37.470918 24210 raft_consensus.cc:2749] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:23:37.440512 24237 raft_consensus.cc:515] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:37.513578 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:37.516327 18363 raft_consensus.cc:1275] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 11 index: 13. Preceding OpId from leader: term: 13 index: 14. (index mismatch)
I20260709 12:23:37.532251 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:23:37.537688 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:37.538420 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:37.493000 24231 raft_consensus.cc:493] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:37.563756 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:37.564790 18363 raft_consensus.cc:2393] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 16: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:23:37.572557 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:37.574460 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:37.575455 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:37.575829 18363 raft_consensus.cc:3060] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Advancing to term 14
W20260709 12:23:37.578263 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 63735.2us
I20260709 12:23:37.473870 24279 consensus_queue.cc:1059] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 14.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.046s
I20260709 12:23:37.576514 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 18 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:23:37.585956 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 113393us
I20260709 12:23:37.520259 24244 raft_consensus.cc:493] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:37.596083 18366 raft_consensus.cc:2410] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for term 18 because replica has last-logged OpId of term: 17 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 11.
I20260709 12:23:37.519767 24235 raft_consensus.cc:493] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:23:37.517158 24203 raft_consensus.cc:697] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 18 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
W20260709 12:23:37.583834 24378 log.cc:927] Time spent T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.068s	user 0.001s	sys 0.000s
I20260709 12:23:37.501842 24151 consensus_queue.cc:1048] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.080s
W20260709 12:23:37.615177 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:37.615276 18364 raft_consensus.cc:1240] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 15 LEADER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 14. Current term is 15. Ops: [14.12-14.12]
I20260709 12:23:37.616891 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:37.557614 24032 raft_consensus.cc:493] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:23:37.605525 24244 raft_consensus.cc:515] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:37.601061 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:37.595580 24265 raft_consensus.cc:493] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:37.557859 24339 raft_consensus.cc:493] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:23:37.605028 18363 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 14.
I20260709 12:23:37.606065 24235 raft_consensus.cc:515] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:37.610673 24381 log.cc:927] Time spent T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.098s	user 0.000s	sys 0.000s
I20260709 12:23:37.618180 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
W20260709 12:23:37.618356 18334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:37.603157 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:37.635557 18367 raft_consensus.cc:2468] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 17.
I20260709 12:23:37.593904 24210 raft_consensus.cc:2749] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:23:37.595741 24215 raft_consensus.cc:493] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:37.630728 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:37.592087 24087 raft_consensus.cc:493] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:23:37.660171 24087 raft_consensus.cc:515] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:37.663038 24087 leader_election.cc:290] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:37.625581 24032 raft_consensus.cc:515] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:37.666985 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:37.667824 17760 raft_consensus.cc:2393] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 18: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:23:37.640928 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:37.669289 18364 raft_consensus.cc:2393] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 20: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:23:37.670150 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:37.671178 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 15 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:37.672039 24235 leader_election.cc:290] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:37.672432 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:37.610704 24203 consensus_queue.cc:237] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.12, Last appended by leader: 12, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:37.673264 18367 raft_consensus.cc:2393] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 14: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:23:37.674081 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:37.674614 18367 raft_consensus.cc:2468] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 19.
I20260709 12:23:37.675426 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:37.676646 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:37.677371 18363 raft_consensus.cc:2393] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 18: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:23:37.678144 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 16 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:37.679404 18363 raft_consensus.cc:1275] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 20 index: 16. Preceding OpId from leader: term: 22 index: 17. (index mismatch)
W20260709 12:23:37.643708 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 48588.5us
I20260709 12:23:37.581873 24231 raft_consensus.cc:515] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:37.680392 18333 leader_election.cc:304] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:23:37.681630 24231 leader_election.cc:290] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:37.682165 18333 leader_election.cc:304] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:37.684994 23953 leader_election.cc:290] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:37.685396 24210 consensus_queue.cc:1048] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:23:37.685779 23953 raft_consensus.cc:2804] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:23:37.686185 23953 raft_consensus.cc:493] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:37.686579 23953 raft_consensus.cc:3060] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:37.689450 24237 leader_election.cc:290] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 17 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:37.691411 24339 raft_consensus.cc:515] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:37.693753 24339 leader_election.cc:290] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:37.700251 18368 raft_consensus.cc:2468] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 19.
I20260709 12:23:37.683540 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:37.646373 17654 leader_election.cc:304] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:37.646950 24236 consensus_queue.cc:1048] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20260709 12:23:37.649210 24244 leader_election.cc:290] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:37.649719 24390 consensus_queue.cc:1048] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 12:23:37.656682 24215 raft_consensus.cc:515] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:37.659139 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:23:37.707834 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 61552.4us
I20260709 12:23:37.715477 17654 leader_election.cc:304] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:37.716969 24250 raft_consensus.cc:3060] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:37.717111 17654 leader_election.cc:304] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:37.707912 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:37.718257 17759 raft_consensus.cc:2393] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 19: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:23:37.682199 24231 raft_consensus.cc:2804] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:23:37.724928 18371 raft_consensus.cc:3060] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [term 17 LEARNER]: Advancing to term 18
I20260709 12:23:37.727113 17654 leader_election.cc:304] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:37.710660 24339 raft_consensus.cc:2749] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:23:37.730635 17654 leader_election.cc:304] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:37.740279 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:23:37.743203 17727 leader_election.cc:336] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 19 election: RPC error from VoteRequest() call to peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837): Timed out: RequestConsensusVote RPC to 127.16.29.67:36837 timed out after 1.559s (SENT)
I20260709 12:23:37.745424 17803 leader_election.cc:304] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:37.771574 17803 leader_election.cc:304] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:37.773533 17803 leader_election.cc:304] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:37.642083 24265 raft_consensus.cc:515] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:37.776495 17684 raft_consensus.cc:2393] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 20: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:23:37.779299 17803 leader_election.cc:304] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:23:37.785329 17803 leader_election.cc:304] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:37.732582 24237 raft_consensus.cc:2749] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:23:37.801303 24215 leader_election.cc:290] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:37.787657 23992 raft_consensus.cc:2749] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:23:37.802727 23992 raft_consensus.cc:2749] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:23:37.740262 24231 raft_consensus.cc:697] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 14 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:23:37.781674 24032 leader_election.cc:290] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:37.627007 24277 consensus_queue.cc:260] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.12, Last appended by leader: 12, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:37.732878 24244 raft_consensus.cc:2749] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:23:37.793241 24265 leader_election.cc:290] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:37.776186 24329 raft_consensus.cc:2749] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:23:37.735987 24087 raft_consensus.cc:2749] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:23:37.813545 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 19 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:37.770951 24262 raft_consensus.cc:2749] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:23:37.806769 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:37.845856 17759 raft_consensus.cc:2468] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 14.
I20260709 12:23:37.807049 24061 raft_consensus.cc:2804] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:23:37.846839 24061 raft_consensus.cc:493] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:37.847236 24061 raft_consensus.cc:3060] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:37.857326 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:37.858678 17686 raft_consensus.cc:2468] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 18.
I20260709 12:23:37.859678 17759 raft_consensus.cc:3055] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:23:37.860229 17759 raft_consensus.cc:740] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 15 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:23:37.873523 17684 raft_consensus.cc:1275] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 12 index: 10. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
I20260709 12:23:37.876276 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 19 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:37.875449 17728 leader_election.cc:304] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:37.825459 24236 raft_consensus.cc:493] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:37.878105 18362 raft_consensus.cc:2410] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for term 19 because replica has last-logged OpId of term: 17 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 12.
I20260709 12:23:37.804380 24231 consensus_queue.cc:237] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.11, Last appended by leader: 11, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:37.854607 24277 raft_consensus.cc:3060] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:37.886042 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: true dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:37.893563 17685 raft_consensus.cc:3060] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:37.890719 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:37.880400 17759 consensus_queue.cc:260] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.10, Last appended by leader: 10, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:37.896181 17759 raft_consensus.cc:3060] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:37.897675 17728 leader_election.cc:304] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:23:37.895143 17687 raft_consensus.cc:2468] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 16.
I20260709 12:23:37.899842 24263 consensus_queue.cc:1048] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:23:37.902128 24300 raft_consensus.cc:2804] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:23:37.902789 24300 raft_consensus.cc:697] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
W20260709 12:23:37.880265 24381 log.cc:927] Time spent T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.055s	user 0.001s	sys 0.000s
I20260709 12:23:37.905196 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:37.877439 24183 consensus_queue.cc:1048] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.016s
I20260709 12:23:37.892592 24037 raft_consensus.cc:2804] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:23:37.912088 17687 raft_consensus.cc:1275] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 18 index: 11. (index mismatch)
W20260709 12:23:37.913203 18334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:37.897915 23997 consensus_queue.cc:1175] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "0da169ff82674dc1947055ce43de68a3" responder_term: 15 status { last_received { term: 15 index: 10 } last_committed_idx: 9 last_received_current_leader { term: 15 index: 10 } }
I20260709 12:23:37.922592 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:37.924031 18362 raft_consensus.cc:2468] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 15.
I20260709 12:23:37.925426 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:37.925810 24061 raft_consensus.cc:515] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:37.927054 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:37.911945 24037 raft_consensus.cc:493] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:23:37.933733 17801 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:37.935599 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:37.936684 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:37.941089 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:37.927847 18362 raft_consensus.cc:2468] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 19.
I20260709 12:23:37.942104 17757 raft_consensus.cc:2393] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 14: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:23:37.942396 18371 raft_consensus.cc:1275] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [term 18 LEARNER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 17 index: 9. Preceding OpId from leader: term: 18 index: 10. (index mismatch)
I20260709 12:23:37.931115 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:37.943733 18365 raft_consensus.cc:2468] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 15.
I20260709 12:23:37.946389 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:37.947467 17758 raft_consensus.cc:2468] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 19.
I20260709 12:23:37.903864 24300 consensus_queue.cc:237] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 17.11, Last appended by leader: 11, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:37.894136 24296 consensus_queue.cc:1048] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.039s
I20260709 12:23:37.954535 18371 raft_consensus.cc:1275] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 12 index: 10. Preceding OpId from leader: term: 13 index: 11. (index mismatch)
I20260709 12:23:37.928285 24037 raft_consensus.cc:3060] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:37.894661 24236 raft_consensus.cc:515] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:23:37.962514 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:37.927860 24061 leader_election.cc:290] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 17 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:37.964480 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:37.975469 17833 raft_consensus.cc:2468] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 20.
I20260709 12:23:37.966281 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 20 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:37.977047 17838 raft_consensus.cc:2468] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 19.
I20260709 12:23:37.967497 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:37.932575 23931 leader_election.cc:290] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:37.970672 17755 raft_consensus.cc:1275] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 17 index: 9. Preceding OpId from leader: term: 19 index: 10. (index mismatch)
W20260709 12:23:37.955761 24116 consensus_meta.cc:287] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.583s	user 0.013s	sys 0.000s
I20260709 12:23:37.956480 17760 raft_consensus.cc:1240] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 15. Current term is 16. Ops: []
I20260709 12:23:37.978154 17831 raft_consensus.cc:2468] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 17.
I20260709 12:23:37.978868 17763 raft_consensus.cc:1240] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 18. Current term is 19. Ops: []
I20260709 12:23:37.953599 24087 raft_consensus.cc:493] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:37.955418 24329 raft_consensus.cc:493] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:37.993502 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:38.002390 17685 raft_consensus.cc:2468] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 15.
I20260709 12:23:38.003577 18368 raft_consensus.cc:1275] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 17 index: 13. Preceding OpId from leader: term: 19 index: 14. (index mismatch)
I20260709 12:23:37.972383 23905 raft_consensus.cc:493] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:37.971800 24250 raft_consensus.cc:2749] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:23:37.992111 24032 raft_consensus.cc:493] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:38.020350 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 14 candidate_status { last_received { term: 11 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:38.001403 24262 raft_consensus.cc:493] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:38.024998 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 217450us
W20260709 12:23:38.034102 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:38.023870 24032 raft_consensus.cc:515] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:38.003104 24087 raft_consensus.cc:515] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:38.007498 24329 raft_consensus.cc:515] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:38.046396 17762 raft_consensus.cc:1240] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 LEADER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 13. Current term is 14. Ops: []
I20260709 12:23:38.018404 23905 raft_consensus.cc:515] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:38.048907 17833 raft_consensus.cc:1240] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 12. Current term is 13. Ops: [10.9-12.10]
I20260709 12:23:38.057221 24151 raft_consensus.cc:3055] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:23:38.057857 24151 raft_consensus.cc:740] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:23:38.060081 24236 leader_election.cc:290] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:38.061412 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:38.062449 24087 leader_election.cc:290] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:38.031083 24262 raft_consensus.cc:515] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:38.068468 18334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:38.072414 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 287162us
W20260709 12:23:38.078754 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 179585us
I20260709 12:23:38.086649 17803 leader_election.cc:304] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:23:38.088625 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:38.089308 17833 raft_consensus.cc:2468] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 14.
I20260709 12:23:38.032207 24394 raft_consensus.cc:493] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:38.071430 24087 raft_consensus.cc:493] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:38.103173 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 19 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:38.105636 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:38.106473 17687 raft_consensus.cc:2468] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 18.
I20260709 12:23:38.101502 17803 leader_election.cc:304] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:23:38.113528 17803 leader_election.cc:304] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:23:38.100725 24116 raft_consensus.cc:2804] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:23:38.119275 17833 raft_consensus.cc:1275] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 15 index: 10. Preceding OpId from leader: term: 16 index: 11. (index mismatch)
I20260709 12:23:38.099615 23905 leader_election.cc:290] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:38.101904 24262 leader_election.cc:290] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:38.127162 17834 raft_consensus.cc:1275] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 10 index: 12. Preceding OpId from leader: term: 13 index: 14. (index mismatch)
I20260709 12:23:38.111191 24032 leader_election.cc:290] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:38.116601 24061 raft_consensus.cc:2804] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:23:38.107177 23992 consensus_queue.cc:1048] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.021s
I20260709 12:23:38.119830 24116 raft_consensus.cc:493] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:23:38.103669 24087 raft_consensus.cc:515] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:38.151839 17762 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36572 (request call id 1140) took 2030 ms. Trace:
I20260709 12:23:38.152746 17762 rpcz_store.cc:276] 0709 12:23:36.121870 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:36.539930 (+418060us) service_pool.cc:224] Handling call
0709 12:23:37.025910 (+485980us) spinlock_profiling.cc:243] Waited 3.33 ms on lock 0x7b5800033398. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b474fbb71 00007f7b474fba22 00007f7b504148d9 00007f7b4f56eee0 00007f7b4f6380ab 00007f7b4f638131 00007f7b4f5f9fc0 00007f7b4f5e4929 00007f7b4f5e43d2 00007f7b48b5bbd4
0709 12:23:37.026120 (+   210us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:38.052204 (+1026084us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:38.151396 (+ 99192us) spinlock_profiling.cc:243] Waited 1.02 s on lock 0x7b54004164d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:23:38.151748 (+   352us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2351337984}
I20260709 12:23:38.166723 17759 raft_consensus.cc:1275] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 15 index: 10. Preceding OpId from leader: term: 16 index: 11. (index mismatch)
I20260709 12:23:38.101377 24394 raft_consensus.cc:515] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:38.138103 23931 raft_consensus.cc:493] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
W20260709 12:23:38.145741 24399 log.cc:927] Time spent T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.059s	user 0.000s	sys 0.000s
W20260709 12:23:38.150540 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:38.169946 23997 raft_consensus.cc:493] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:23:38.236665 23997 raft_consensus.cc:515] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:38.174491 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:38.174610 23953 raft_consensus.cc:515] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:38.177392 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 16 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:38.180552 18368 raft_consensus.cc:1275] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 16 index: 12. (index mismatch)
I20260709 12:23:38.182039 24329 leader_election.cc:290] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:38.182492 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 19 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:38.265975 17762 raft_consensus.cc:2468] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 18.
I20260709 12:23:38.183692 17801 leader_election.cc:304] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:38.098695 24151 consensus_queue.cc:260] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.10, Last appended by leader: 10, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:38.184865 24116 raft_consensus.cc:3060] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Advancing to term 17
W20260709 12:23:38.194269 24395 log.cc:927] Time spent T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.265s	user 0.002s	sys 0.000s
I20260709 12:23:38.198031 17652 leader_election.cc:304] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:23:38.199189 17832 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35040 (request call id 2933) took 1738 ms (client timeout 2183 ms). Trace:
W20260709 12:23:38.281544 17832 rpcz_store.cc:269] 0709 12:23:36.460891 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:36.593164 (+132273us) service_pool.cc:224] Handling call
0709 12:23:38.199017 (+1605853us) spinlock_profiling.cc:243] Waited 845 ms on lock 0x7b54000ab9d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:23:38.199130 (+   113us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1942944000}
I20260709 12:23:38.283468 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:38.199910 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: true dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:38.136974 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:38.153935 24087 leader_election.cc:290] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:38.208499 17764 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38776 (request call id 2658) took 1053 ms. Trace:
I20260709 12:23:38.219252 24250 raft_consensus.cc:493] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:38.229339 17728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:38.243449 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:38.292533 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:38.293390 17830 raft_consensus.cc:2468] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 18.
I20260709 12:23:38.294418 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:38.295511 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:38.267616 18365 raft_consensus.cc:1275] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 14 index: 12. (index mismatch)
I20260709 12:23:38.187541 24037 raft_consensus.cc:515] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:38.275957 24183 raft_consensus.cc:493] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:38.133186 24401 raft_consensus.cc:2804] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:23:38.240700 17837 raft_consensus.cc:2376] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 19. Re-sending same reply.
W20260709 12:23:38.279145 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 95555.5us
I20260709 12:23:38.286904 18361 raft_consensus.cc:3055] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:23:38.304394 18361 raft_consensus.cc:740] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
W20260709 12:23:38.288367 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 90405us
I20260709 12:23:38.307189 17652 leader_election.cc:304] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:38.286558 17764 rpcz_store.cc:276] 0709 12:23:37.154654 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:37.165716 (+ 11062us) service_pool.cc:224] Handling call
0709 12:23:37.166053 (+   337us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:37.166424 (+   371us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:37.166454 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:37.166926 (+   472us) log.cc:844] Serialized 26 byte log entry
0709 12:23:38.163663 (+996737us) spinlock_profiling.cc:243] Waited 6.76 ms on lock 0x7b18003c5ec0. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b4a1c84bc 00007f7b4751648b 00007f7b503709a9 00007f7b4753269b 00007f7b4753093f 00007f7b47532c02 00007f7b4a1ba20b 00007f7b4a1bfd23 00007f7b4a1c00d1
0709 12:23:38.201274 (+ 37611us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:23:38.201647 (+   373us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:38.201711 (+    64us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:38.201745 (+    34us) raft_consensus.cc:1685] finished
0709 12:23:38.201752 (+     7us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:38.208387 (+  6635us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":10702,"raft.run_cpu_time_us":89,"raft.run_wall_time_us":88,"spinlock_wait_cycles":15540096,"thread_start_us":277524,"threads_started":1,"wal-append.queue_time_us":276201}
I20260709 12:23:38.285696 17833 raft_consensus.cc:2468] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 12.
I20260709 12:23:38.309033 17652 leader_election.cc:304] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:38.294435 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:38.309648 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:38.310497 17681 raft_consensus.cc:2393] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 16: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:23:38.275163 24169 raft_consensus.cc:493] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:38.269717 24296 raft_consensus.cc:493] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:38.137012 24061 raft_consensus.cc:493] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:38.291100 24236 raft_consensus.cc:3060] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:38.313995 17833 raft_consensus.cc:2393] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 20: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:23:38.314222 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:38.258440 23931 raft_consensus.cc:515] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:23:38.315438 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 199169us
I20260709 12:23:38.261993 23997 leader_election.cc:290] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:38.274924 24313 raft_consensus.cc:493] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:23:38.314584 23905 consensus_queue.cc:1059] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.007s
I20260709 12:23:38.321031 24313 raft_consensus.cc:515] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:38.286245 24329 raft_consensus.cc:2804] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:23:38.316212 17829 raft_consensus.cc:1275] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 14 index: 12. Preceding OpId from leader: term: 15 index: 12. (term mismatch)
I20260709 12:23:38.331960 17829 pending_rounds.cc:85] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b: Aborting all ops after (but not including) 11
I20260709 12:23:38.332336 17829 pending_rounds.cc:107] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b: Aborting uncommitted NO_OP operation due to leader change: 14.12
I20260709 12:23:38.305569 18361 consensus_queue.cc:260] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.12, Last appended by leader: 12, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:38.332850 17729 leader_election.cc:304] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:38.333518 18361 raft_consensus.cc:3060] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:38.334475 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:38.317567 17652 leader_election.cc:304] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:38.318383 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:38.338744 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:38.318818 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:38.340590 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:38.341275 17837 raft_consensus.cc:3060] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:38.319583 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:38.344619 17833 raft_consensus.cc:2393] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 14: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:23:38.322774 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 23 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:38.324433 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:38.351163 17763 raft_consensus.cc:1275] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 18 index: 13. (index mismatch)
I20260709 12:23:38.325815 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:38.327716 17760 raft_consensus.cc:1240] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 19. Current term is 20. Ops: [19.12-19.12]
I20260709 12:23:38.314507 24237 raft_consensus.cc:2749] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:23:38.359217 17652 leader_election.cc:304] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:38.314872 24061 raft_consensus.cc:3060] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:38.362778 17729 leader_election.cc:304] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:38.362989 17829 raft_consensus.cc:2889] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:38.321650 23905 consensus_queue.cc:1243] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer e2909cff1da44ad387f96a8ae3aa5afb log is divergent from this leader: its last log entry 17.11 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 9
I20260709 12:23:38.295132 24250 raft_consensus.cc:515] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:38.319821 24227 raft_consensus.cc:2749] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:23:38.317394 24401 raft_consensus.cc:697] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 16 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
W20260709 12:23:38.377955 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 58228.8us
W20260709 12:23:38.362335 24397 log.cc:927] Time spent T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.252s	user 0.000s	sys 0.001s
I20260709 12:23:38.322898 24394 leader_election.cc:290] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:38.388502 18333 leader_election.cc:304] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:38.304057 24183 raft_consensus.cc:515] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:38.392745 18333 leader_election.cc:304] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:23:38.356832 24329 raft_consensus.cc:493] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:23:38.363442 23997 raft_consensus.cc:2804] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:23:38.395159 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 16 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:23:38.398919 18334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:38.367635 24237 raft_consensus.cc:2749] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:23:38.391937 17729 leader_election.cc:304] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:38.317051 24169 raft_consensus.cc:515] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:38.321249 24262 consensus_queue.cc:1059] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 15.12, Next index: 13, Last known committed idx: 11, Time since last communication: 0.065s
I20260709 12:23:38.316743 24296 raft_consensus.cc:515] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:38.406883 18333 leader_election.cc:304] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:38.438282 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:38.406570 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:38.368530 24244 raft_consensus.cc:2804] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:23:38.414784 17684 raft_consensus.cc:2468] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 19.
I20260709 12:23:38.405310 17652 leader_election.cc:304] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:23:38.475440 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 106104us
I20260709 12:23:38.471992 18361 raft_consensus.cc:2410] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for term 15 because replica has last-logged OpId of term: 14 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 11.
W20260709 12:23:38.459252 24406 log.cc:927] Time spent T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.247s	user 0.004s	sys 0.028s
W20260709 12:23:38.476713 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 69922us
I20260709 12:23:38.491319 17763 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 15.
I20260709 12:23:38.402293 24304 raft_consensus.cc:493] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:38.492304 17652 leader_election.cc:304] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:38.406525 24362 raft_consensus.cc:2804] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:23:38.492435 18333 leader_election.cc:304] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:23:38.407400 24404 raft_consensus.cc:2749] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:23:38.456663 24032 raft_consensus.cc:493] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:23:38.501549 18333 leader_election.cc:304] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:38.416323 24394 raft_consensus.cc:2804] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:23:38.496145 17759 raft_consensus.cc:2410] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for term 15 because replica has last-logged OpId of term: 14 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 12.
W20260709 12:23:38.510021 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:38.474968 23997 raft_consensus.cc:493] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:38.385842 23905 consensus_queue.cc:1059] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 18.10, Next index: 11, Last known committed idx: 9, Time since last communication: 0.106s
I20260709 12:23:38.473907 24329 raft_consensus.cc:3060] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:38.483029 24418 raft_consensus.cc:3055] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:23:38.439438 24227 consensus_queue.cc:1048] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.083s
I20260709 12:23:38.492805 24237 raft_consensus.cc:2804] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:23:38.481828 24392 raft_consensus.cc:2749] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:23:38.515419 24304 raft_consensus.cc:515] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:38.524534 23997 raft_consensus.cc:3060] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:38.515198 24362 raft_consensus.cc:493] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:38.492813 24417 raft_consensus.cc:2804] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Leader pre-election won for term 20
W20260709 12:23:38.554812 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 236260us
I20260709 12:23:38.519400 24394 raft_consensus.cc:493] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:23:38.522771 24203 raft_consensus.cc:2804] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Leader pre-election won for term 15
W20260709 12:23:38.580536 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 188471us
I20260709 12:23:38.516249 23953 leader_election.cc:290] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 21 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:38.596603 17837 raft_consensus.cc:2468] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 15.
I20260709 12:23:38.601125 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:38.525606 24037 leader_election.cc:290] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 20 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:38.529454 24237 raft_consensus.cc:493] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:38.615631 24237 raft_consensus.cc:3060] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:38.616194 23953 consensus_queue.cc:1048] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 12:23:38.596655 23905 consensus_queue.cc:1048] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.023s
I20260709 12:23:38.492702 24401 consensus_queue.cc:237] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 14.13, Last appended by leader: 13, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:38.602412 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:38.541983 24313 leader_election.cc:290] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:38.524755 24391 raft_consensus.cc:2749] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:23:38.611346 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:38.542936 24335 raft_consensus.cc:493] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:38.620747 17837 raft_consensus.cc:2376] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 18. Re-sending same reply.
I20260709 12:23:38.526754 24418 raft_consensus.cc:740] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 18 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:23:38.621923 18362 raft_consensus.cc:1275] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 17 index: 13. Preceding OpId from leader: term: 18 index: 13. (term mismatch)
W20260709 12:23:38.624943 17803 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:38.614367 17689 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44898 (request call id 1251) took 1316 ms. Trace:
I20260709 12:23:38.626252 17836 raft_consensus.cc:2393] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 17: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:23:38.580292 23931 leader_election.cc:290] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:38.522115 24032 raft_consensus.cc:515] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:23:38.628068 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 135810us
I20260709 12:23:38.599252 24362 raft_consensus.cc:3060] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:38.722158 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:23:38.654575 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:38.678997 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 158498us
I20260709 12:23:38.722980 17689 rpcz_store.cc:276] 0709 12:23:37.298113 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:37.298484 (+   371us) service_pool.cc:224] Handling call
0709 12:23:37.298845 (+   361us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:37.447594 (+148749us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:38.614035 (+1166441us) spinlock_profiling.cc:243] Waited 84.5 ms on lock 0x7b54001aa254. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:23:38.614286 (+   251us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":194271104}
W20260709 12:23:38.699518 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 97350.1us
I20260709 12:23:38.595909 24394 raft_consensus.cc:3060] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:23:38.628103 24412 log.cc:927] Time spent T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.316s	user 0.001s	sys 0.001s
I20260709 12:23:38.598351 24203 raft_consensus.cc:493] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:38.598875 24417 raft_consensus.cc:493] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:23:38.724828 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:38.727460 17764 raft_consensus.cc:2393] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 19: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:23:38.726135 18333 leader_election.cc:304] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:23:38.720214 18362 pending_rounds.cc:85] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da: Aborting all ops after (but not including) 12
W20260709 12:23:38.691151 24423 log.cc:927] Time spent T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.071s	user 0.001s	sys 0.000s
I20260709 12:23:38.732251 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:23:38.704685 24410 log.cc:927] Time spent T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.336s	user 0.000s	sys 0.002s
I20260709 12:23:38.737902 18362 pending_rounds.cc:107] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da: Aborting uncommitted NO_OP operation due to leader change: 17.13
I20260709 12:23:38.742522 18362 raft_consensus.cc:2889] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:38.743209 17836 raft_consensus.cc:1275] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 16 index: 10. Preceding OpId from leader: term: 18 index: 11. (index mismatch)
I20260709 12:23:38.630507 23740 raft_consensus.cc:3055] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:23:38.680891 17837 raft_consensus.cc:1275] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 18 index: 13. Preceding OpId from leader: term: 20 index: 14. (index mismatch)
I20260709 12:23:38.702706 24391 raft_consensus.cc:493] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:38.722232 24250 leader_election.cc:290] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:38.629087 24392 raft_consensus.cc:493] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:23:38.721006 24244 raft_consensus.cc:697] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 14 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:23:38.789454 17802 leader_election.cc:304] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:38.737375 24203 raft_consensus.cc:3060] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:38.793864 17802 leader_election.cc:304] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:38.796342 17836 raft_consensus.cc:1275] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 16 index: 12. (index mismatch)
I20260709 12:23:38.740500 23931 raft_consensus.cc:2804] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:23:38.704078 24428 consensus_queue.cc:1048] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.065s
I20260709 12:23:38.737234 24417 raft_consensus.cc:3060] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:38.750810 24431 raft_consensus.cc:493] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:38.808434 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 437078us
I20260709 12:23:38.814778 17837 raft_consensus.cc:1240] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 15. Current term is 16. Ops: []
I20260709 12:23:38.669420 24335 raft_consensus.cc:515] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:23:38.816717 17804 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:38.749810 23740 raft_consensus.cc:740] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 18 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:23:38.806195 17802 leader_election.cc:304] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:38.784493 24183 leader_election.cc:290] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:38.824244 17654 leader_election.cc:304] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:38.741380 24404 consensus_queue.cc:1048] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.076s
I20260709 12:23:38.741060 24313 consensus_queue.cc:1048] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.070s
I20260709 12:23:38.828886 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:38.797868 24434 raft_consensus.cc:493] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:38.804406 24433 raft_consensus.cc:493] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:38.831266 17654 leader_election.cc:304] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:38.834967 17654 leader_election.cc:304] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:38.836030 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:38.818329 24420 raft_consensus.cc:2749] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 12:23:38.806658 23931 raft_consensus.cc:493] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:38.837935 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:38.838887 18365 raft_consensus.cc:2393] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 15: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:23:38.840018 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:38.828120 24429 raft_consensus.cc:493] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:23:38.798007 24425 consensus_queue.cc:1048] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 10, Time since last communication: 0.044s
I20260709 12:23:38.784607 24391 raft_consensus.cc:515] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:38.880604 24244 consensus_queue.cc:237] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 12.9, Last appended by leader: 9, Current term: 14, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:38.895421 24391 leader_election.cc:290] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:38.821398 24431 raft_consensus.cc:515] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:38.818712 24432 consensus_queue.cc:1048] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.039s
I20260709 12:23:38.896806 24263 raft_consensus.cc:493] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:38.897625 24431 leader_election.cc:290] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:38.897424 24263 raft_consensus.cc:515] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:38.831851 24183 consensus_queue.cc:1048] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.055s
I20260709 12:23:38.822687 24418 consensus_queue.cc:260] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.11, Last appended by leader: 11, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:38.883581 24032 leader_election.cc:290] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:38.900285 24418 raft_consensus.cc:3060] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:38.900409 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 19 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:38.899745 24263 leader_election.cc:290] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:38.901321 24420 consensus_queue.cc:1048] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 12:23:38.902935 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 17 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:38.903498 24087 raft_consensus.cc:2804] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:23:38.904285 17756 raft_consensus.cc:1275] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 14 index: 12. Preceding OpId from leader: term: 15 index: 13. (index mismatch)
I20260709 12:23:38.867811 24428 raft_consensus.cc:2749] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:23:38.904978 24087 raft_consensus.cc:493] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:23:38.906867 24236 consensus_meta.cc:287] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.589s	user 0.009s	sys 0.002s
I20260709 12:23:38.888199 24430 consensus_queue.cc:1048] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 12:23:38.845587 24169 leader_election.cc:290] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:23:38.891893 24396 log.cc:927] Time spent T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 1.047s	user 0.000s	sys 0.003s
I20260709 12:23:38.835865 23905 raft_consensus.cc:2804] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Leader pre-election won for term 20
W20260709 12:23:38.876435 24422 log.cc:927] Time spent T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.138s	user 0.001s	sys 0.000s
I20260709 12:23:38.905658 24087 raft_consensus.cc:3060] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:38.906867 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:23:38.887269 24061 consensus_meta.cc:287] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.504s	user 0.006s	sys 0.003s
I20260709 12:23:38.892006 24429 raft_consensus.cc:515] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:38.837807 24433 raft_consensus.cc:515] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:38.926551 24116 consensus_meta.cc:287] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.636s	user 0.012s	sys 0.000s
I20260709 12:23:38.916498 24424 raft_consensus.cc:3055] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:23:38.927294 24236 raft_consensus.cc:2749] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:23:38.910567 24335 leader_election.cc:290] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:23:38.886274 24408 log.cc:927] Time spent T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.623s	user 0.000s	sys 0.002s
I20260709 12:23:38.928558 24236 raft_consensus.cc:493] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:23:38.886933 24296 leader_election.cc:290] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:38.929095 24236 raft_consensus.cc:515] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:23:38.872664 24426 log.cc:927] Time spent T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.128s	user 0.000s	sys 0.001s
I20260709 12:23:38.931285 24236 leader_election.cc:290] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:38.927122 23905 raft_consensus.cc:493] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:23:38.905517 24428 raft_consensus.cc:3060] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:38.931978 24429 leader_election.cc:290] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:38.926810 24061 raft_consensus.cc:515] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:38.932680 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:38.954964 23905 raft_consensus.cc:3060] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:38.960435 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:38.961462 17831 raft_consensus.cc:2410] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for term 19 because replica has last-logged OpId of term: 17 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 9.
I20260709 12:23:38.962579 17831 raft_consensus.cc:1275] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 17 index: 10. Preceding OpId from leader: term: 18 index: 10. (term mismatch)
I20260709 12:23:38.962874 17831 pending_rounds.cc:85] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b: Aborting all ops after (but not including) 9
I20260709 12:23:38.963088 17831 pending_rounds.cc:107] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b: Aborting uncommitted CHANGE_CONFIG_OP operation due to leader change: 17.10
I20260709 12:23:38.963403 17831 raft_consensus.cc:2917] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Aborting config change with OpId 17.10: Aborted: Op aborted by new leader
I20260709 12:23:38.905555 24329 raft_consensus.cc:515] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:38.927119 24116 raft_consensus.cc:515] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:38.966377 24329 leader_election.cc:290] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 19 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:38.838768 24434 raft_consensus.cc:515] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:38.967958 24116 leader_election.cc:290] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 17 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:38.969861 24434 leader_election.cc:290] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:38.970606 17830 raft_consensus.cc:2468] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 17.
I20260709 12:23:38.829809 24427 raft_consensus.cc:3060] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:38.977813 24237 raft_consensus.cc:515] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:38.837069 24437 raft_consensus.cc:2804] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:23:38.980894 24437 raft_consensus.cc:493] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:23:38.981287 24437 raft_consensus.cc:3060] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:38.795717 24392 raft_consensus.cc:515] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:38.929440 24433 leader_election.cc:290] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:23:38.938616 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:38.956650 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:38.841430 23931 raft_consensus.cc:3060] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:38.988793 17831 raft_consensus.cc:2889] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:38.904177 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:39.005993 17837 raft_consensus.cc:2468] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 16.
I20260709 12:23:38.914779 23906 raft_consensus.cc:3060] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:38.877753 23997 raft_consensus.cc:515] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:39.015452 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:39.019320 17831 raft_consensus.cc:1275] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 14 index: 12. (index mismatch)
I20260709 12:23:39.012949 17832 raft_consensus.cc:2368] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for earlier term 20. Current term is 21.
I20260709 12:23:39.033123 17831 raft_consensus.cc:1275] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 15 index: 9. Preceding OpId from leader: term: 17 index: 10. (index mismatch)
I20260709 12:23:39.033252 17683 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44898 (request call id 1254) took 1686 ms. Trace:
I20260709 12:23:38.848203 24304 leader_election.cc:290] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:39.033663 17683 rpcz_store.cc:276] 0709 12:23:37.346653 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:37.447156 (+100503us) service_pool.cc:224] Handling call
0709 12:23:37.447483 (+   327us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:37.460898 (+ 13415us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:37.460942 (+    44us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:37.461609 (+   667us) log.cc:844] Serialized 26 byte log entry
0709 12:23:37.714614 (+253005us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:23:37.714671 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:37.714755 (+    84us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:38.891131 (+1176376us) spinlock_profiling.cc:243] Waited 10.5 ms on lock 0x7b4400022b4c. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b47459c85 00007f7b4745a6a2 00007f7b4a1cc1db 00007f7b4a1c7aaf 00007f7b4a1bb86e 00007f7b4a1babb6 00007f7b4a1ba7d8 00007f7b4a1c4221 00007f7b4a1c41d9 00007f7b4a1c4169
0709 12:23:39.015980 (+124849us) raft_consensus.cc:1685] finished
0709 12:23:39.016013 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:39.033187 (+ 17174us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":7418,"raft.run_cpu_time_us":128,"raft.run_wall_time_us":8475,"spinlock_wait_cycles":24118144,"thread_start_us":220347,"threads_started":1,"wal-append.queue_time_us":372231}
I20260709 12:23:38.957904 23740 consensus_queue.cc:260] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.10, Last appended by leader: 10, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:38.927460 24424 raft_consensus.cc:740] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 15 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:23:39.082389 24424 consensus_queue.cc:260] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.12, Last appended by leader: 12, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:39.083297 17729 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:39.084062 24424 raft_consensus.cc:3060] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Advancing to term 16
W20260709 12:23:38.987107 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 123413us
I20260709 12:23:39.015748 24427 raft_consensus.cc:2749] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
W20260709 12:23:39.012774 24414 log.cc:927] Time spent T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.750s	user 0.007s	sys 0.004s
I20260709 12:23:39.057216 23997 leader_election.cc:290] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 18 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:23:39.058802 24413 log.cc:927] Time spent T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.612s	user 0.001s	sys 0.000s
I20260709 12:23:39.064313 24247 raft_consensus.cc:3060] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:23:39.069075 24405 log.cc:927] Time spent T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.856s	user 0.000s	sys 0.016s
W20260709 12:23:39.070269 24409 log.cc:927] Time spent T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.806s	user 0.002s	sys 0.002s
W20260709 12:23:39.074539 24384 log.cc:927] Time spent T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 1.146s	user 0.002s	sys 0.008s
I20260709 12:23:39.077701 17833 raft_consensus.cc:1240] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 20 LEADER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 19. Current term is 20. Ops: [19.11-19.11]
I20260709 12:23:39.042856 24392 leader_election.cc:290] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:39.126004 17763 raft_consensus.cc:1275] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 16 index: 11. Preceding OpId from leader: term: 18 index: 12. (index mismatch)
I20260709 12:23:39.126091 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:39.127079 18374 raft_consensus.cc:3060] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:39.128315 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:39.083760 23905 raft_consensus.cc:515] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:23:39.085937 24416 log.cc:927] Time spent T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.864s	user 0.000s	sys 0.008s
I20260709 12:23:39.142560 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 19 candidate_status { last_received { term: 18 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
W20260709 12:23:39.087040 24415 log.cc:927] Time spent T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.813s	user 0.000s	sys 0.003s
I20260709 12:23:39.087781 17829 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37994 (request call id 1249) took 1122 ms. Trace:
W20260709 12:23:39.089530 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 363461us
I20260709 12:23:39.091759 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:39.095085 18368 pending_rounds.cc:85] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da: Aborting all ops after (but not including) 10
I20260709 12:23:39.143760 18333 leader_election.cc:304] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:23:39.144374 18368 pending_rounds.cc:107] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da: Aborting uncommitted NO_OP operation due to leader change: 13.11
I20260709 12:23:39.144708 18368 raft_consensus.cc:2889] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:39.144913 18333 leader_election.cc:304] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:39.096467 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:39.146324 18367 raft_consensus.cc:2468] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 22.
I20260709 12:23:39.012437 24061 leader_election.cc:290] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 21 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:39.099552 17837 raft_consensus.cc:1275] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 17 index: 13. Preceding OpId from leader: term: 19 index: 14. (index mismatch)
I20260709 12:23:39.148126 24391 raft_consensus.cc:2804] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:23:39.148566 24391 raft_consensus.cc:493] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:23:39.148953 24391 raft_consensus.cc:3060] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:39.008858 24429 raft_consensus.cc:493] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:39.149757 24337 raft_consensus.cc:3060] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:39.016625 24237 leader_election.cc:290] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 16 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:39.104820 17834 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35040 (request call id 2937) took 2499 ms. Trace:
I20260709 12:23:39.147578 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:39.153486 17834 rpcz_store.cc:276] 0709 12:23:36.605312 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:36.734357 (+129045us) service_pool.cc:224] Handling call
0709 12:23:36.734654 (+   297us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:38.128180 (+1393526us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:39.104568 (+976388us) spinlock_profiling.cc:243] Waited 1.39 s on lock 0x7b5400265a54. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:23:39.104753 (+   185us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3201737600}
I20260709 12:23:39.109575 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:39.132649 17687 raft_consensus.cc:1275] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 17 index: 11. Preceding OpId from leader: term: 19 index: 12. (index mismatch)
I20260709 12:23:39.110152 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:39.111034 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:39.114630 17680 raft_consensus.cc:1240] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 14. Current term is 15. Ops: [14.12-14.12]
I20260709 12:23:39.157882 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:39.118667 17831 raft_consensus.cc:1240] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 14. Current term is 15. Ops: []
I20260709 12:23:39.123649 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:39.150023 24429 raft_consensus.cc:515] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:39.137635 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:39.143419 17829 rpcz_store.cc:276] 0709 12:23:37.965461 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:37.965601 (+   140us) service_pool.cc:224] Handling call
0709 12:23:37.967608 (+  2007us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:38.365795 (+398187us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:39.087517 (+721722us) spinlock_profiling.cc:243] Waited 348 ms on lock 0x7b54003d4ad4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:23:39.087720 (+   203us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":800832256}
I20260709 12:23:39.154322 17683 raft_consensus.cc:3060] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:39.117244 17686 raft_consensus.cc:1240] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 18. Current term is 19. Ops: []
I20260709 12:23:39.136264 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:39.170079 17759 raft_consensus.cc:2368] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 17. Current term is 18.
I20260709 12:23:39.160012 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:39.157549 18365 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 15.
I20260709 12:23:39.175972 17689 raft_consensus.cc:2468] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 16.
I20260709 12:23:39.158583 17687 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 15.
I20260709 12:23:39.117890 23740 raft_consensus.cc:3060] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:39.178191 17763 raft_consensus.cc:3060] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Advancing to term 21
W20260709 12:23:39.180222 17803 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:39.180308 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:23:39.182194 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:39.198503 18374 raft_consensus.cc:2468] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 18.
I20260709 12:23:39.202514 17688 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37396 (request call id 2714) took 1109 ms. Trace:
I20260709 12:23:39.203128 17688 rpcz_store.cc:276] 0709 12:23:38.095459 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:38.096236 (+   777us) service_pool.cc:224] Handling call
0709 12:23:38.096664 (+   428us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:38.097759 (+  1095us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:38.097800 (+    41us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:38.098336 (+   536us) log.cc:844] Serialized 26 byte log entry
0709 12:23:38.172380 (+ 74044us) spinlock_profiling.cc:243] Waited 5.25 ms on lock 0x7b48000030d8. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4f549aa1 00007f7b4f540b37 00007f7b4f5409ca 00007f7b4f63826f 00007f7b4f637f37 00007f7b4f64c55c 00007f7b4f64c508 00007f7b4f64c488 00007f7b4f64c443 00007f7b4f64b6ec 00007f7b4e575a96
0709 12:23:38.203331 (+ 30951us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:23:38.203389 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:38.203477 (+    88us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:39.069727 (+866250us) spinlock_profiling.cc:243] Waited 11.4 ms on lock 0x7b4400022b4c. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b47459c85 00007f7b4745a6a2 00007f7b4a1cc1db 00007f7b4a1c7aaf 00007f7b4a1bb86e 00007f7b4a1babb6 00007f7b4a1ba7d8 00007f7b4a1c4221 00007f7b4a1c41d9 00007f7b4a1c4169
0709 12:23:39.202220 (+132493us) raft_consensus.cc:1685] finished
0709 12:23:39.202254 (+    34us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:39.202431 (+   177us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":608,"raft.queue_time_us":1646,"raft.run_cpu_time_us":1403,"raft.run_wall_time_us":71990,"spinlock_wait_cycles":38200320,"thread_start_us":104632,"threads_started":1,"wal-append.queue_time_us":165749}
I20260709 12:23:39.222723 17683 raft_consensus.cc:2468] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 17.
I20260709 12:23:39.206079 24392 raft_consensus.cc:493] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:39.235236 24362 consensus_meta.cc:287] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.503s	user 0.007s	sys 0.003s
I20260709 12:23:39.248524 17682 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44898 (request call id 1253) took 1909 ms. Trace:
I20260709 12:23:39.250664 17682 rpcz_store.cc:276] 0709 12:23:37.339009 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:37.378942 (+ 39933us) service_pool.cc:224] Handling call
0709 12:23:37.379402 (+   460us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:23:37.442367 (+ 62965us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:37.442404 (+    37us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:23:37.443271 (+   867us) log.cc:844] Serialized 52 byte log entry
0709 12:23:37.814999 (+371728us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:23:37.815070 (+    71us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:37.815178 (+   108us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:39.073948 (+1258770us) spinlock_profiling.cc:243] Waited 2.75 ms on lock 0x7b4400022b4c. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b47459c85 00007f7b4745a6a2 00007f7b4a1cc1db 00007f7b4a1c7aaf 00007f7b4a1bb86e 00007f7b4a1babb6 00007f7b4a1ba7d8 00007f7b4a1c4221 00007f7b4a1c41d9 00007f7b4a1c4169
0709 12:23:39.242593 (+168645us) raft_consensus.cc:1685] finished
0709 12:23:39.242618 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:39.248450 (+  5832us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":36960,"raft.run_cpu_time_us":94,"raft.run_wall_time_us":95,"spinlock_wait_cycles":6316928,"thread_start_us":342885,"threads_started":1,"wal-append.queue_time_us":466558}
I20260709 12:23:39.246768 17835 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37994 (request call id 1253) took 1276 ms. Trace:
I20260709 12:23:39.257071 17835 rpcz_store.cc:276] 0709 12:23:37.970206 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:37.973165 (+  2959us) service_pool.cc:224] Handling call
0709 12:23:38.005612 (+ 32447us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:38.005928 (+   316us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:38.005956 (+    28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:38.014495 (+  8539us) log.cc:844] Serialized 26 byte log entry
0709 12:23:38.115107 (+100612us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:23:38.115161 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:38.115401 (+   240us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:39.068481 (+953080us) spinlock_profiling.cc:243] Waited 35.2 ms on lock 0x7b4400022b4c. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b47459c85 00007f7b4745a6a2 00007f7b4a1cc1db 00007f7b4a1c7aaf 00007f7b4a1bb86e 00007f7b4a1babb6 00007f7b4a1ba7d8 00007f7b4a1c4221 00007f7b4a1c41d9 00007f7b4a1c4169
0709 12:23:39.246483 (+178002us) raft_consensus.cc:1685] finished
0709 12:23:39.246508 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:39.246704 (+   196us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":80945408,"thread_start_us":71903,"threads_started":1,"wal-append.queue_time_us":181521}
W20260709 12:23:39.263818 17803 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:39.212951 24437 raft_consensus.cc:515] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:39.273712 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 438752us
I20260709 12:23:39.261379 24362 raft_consensus.cc:515] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:39.259536 24392 raft_consensus.cc:515] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:39.265642 24203 raft_consensus.cc:515] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:39.274624 24441 raft_consensus.cc:493] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
W20260709 12:23:39.289819 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 222961us
I20260709 12:23:39.273871 24417 raft_consensus.cc:515] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:39.313970 24417 leader_election.cc:290] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 20 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:39.315675 24362 leader_election.cc:290] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 20 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:39.316792 18371 raft_consensus.cc:1240] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 19. Current term is 20. Ops: [19.14-19.14]
W20260709 12:23:39.317656 24394 consensus_meta.cc:287] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.577s	user 0.004s	sys 0.005s
I20260709 12:23:39.318159 24394 raft_consensus.cc:515] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:39.319660 17836 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35042 (request call id 2657) took 1021 ms. Trace:
I20260709 12:23:39.320116 24394 leader_election.cc:290] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 18 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:39.317632 24337 raft_consensus.cc:2749] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:23:39.320086 17836 rpcz_store.cc:276] 0709 12:23:38.298078 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:38.801786 (+503708us) service_pool.cc:224] Handling call
0709 12:23:38.802295 (+   509us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 12:23:38.814382 (+ 12087us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:38.814410 (+    28us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 12:23:38.829930 (+ 15520us) log.cc:844] Serialized 78 byte log entry
0709 12:23:39.110276 (+280346us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:23:39.110321 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:39.110390 (+    69us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:39.319426 (+209036us) raft_consensus.cc:1685] finished
0709 12:23:39.319449 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:39.319600 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":12502,"raft.queue_time_us":27806,"raft.run_cpu_time_us":107,"raft.run_wall_time_us":4862,"thread_start_us":260856,"threads_started":1,"wal-append.queue_time_us":429338}
I20260709 12:23:39.321007 24429 leader_election.cc:290] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:39.326910 23931 raft_consensus.cc:515] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:39.333599 24428 raft_consensus.cc:2749] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:23:39.333990 24087 raft_consensus.cc:515] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:39.336213 18364 raft_consensus.cc:1240] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 14. Current term is 15. Ops: [14.12-14.12]
I20260709 12:23:39.306464 24392 leader_election.cc:290] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:39.307237 23905 leader_election.cc:290] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 20 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:39.338625 24437 leader_election.cc:290] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 19 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:39.339015 23931 leader_election.cc:290] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 19 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:39.342032 17761 raft_consensus.cc:2410] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for term 18 because replica has last-logged OpId of term: 17 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 13.
I20260709 12:23:39.336297 24087 leader_election.cc:290] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 16 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:39.342461 23997 raft_consensus.cc:493] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:39.342482 17838 raft_consensus.cc:2468] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 19.
I20260709 12:23:39.343007 23997 raft_consensus.cc:515] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:39.340066 17763 raft_consensus.cc:2468] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 21.
I20260709 12:23:39.331396 24441 raft_consensus.cc:515] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:39.336627 24391 raft_consensus.cc:515] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:39.337314 24203 leader_election.cc:290] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 15 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:39.356611 24441 leader_election.cc:290] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:39.366442 24250 raft_consensus.cc:493] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:39.367027 24250 raft_consensus.cc:515] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:39.367476 23997 leader_election.cc:290] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:39.372061 24151 raft_consensus.cc:3060] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:39.375602 17654 leader_election.cc:304] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:39.387630 17729 leader_election.cc:304] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:39.392258 18333 leader_election.cc:304] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:23:39.396476 17652 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.672s	user 0.010s	sys 0.008s
W20260709 12:23:39.397222 17652 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.673s	user 0.011s	sys 0.008s
I20260709 12:23:39.399358 18333 leader_election.cc:304] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:39.399567 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:23:39.302234 17727 leader_election.cc:336] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 18 election: RPC error from VoteRequest() call to peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837): Timed out: RequestConsensusVote RPC to 127.16.29.67:36837 timed out after 1.614s (SENT)
I20260709 12:23:39.400681 17683 raft_consensus.cc:1275] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 17 index: 9. Preceding OpId from leader: term: 19 index: 10. (index mismatch)
I20260709 12:23:39.401139 17682 raft_consensus.cc:1240] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 18. Current term is 19. Ops: []
W20260709 12:23:39.400647 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 98436.4us
I20260709 12:23:39.402864 17763 raft_consensus.cc:1240] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 14. Current term is 15. Ops: []
I20260709 12:23:39.406087 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:39.409169 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:39.409983 17759 raft_consensus.cc:2393] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 16: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:23:39.404254 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:39.412217 17756 raft_consensus.cc:2393] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 21: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:23:39.414016 24421 raft_consensus.cc:493] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:39.414651 24421 raft_consensus.cc:515] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:39.415493 24430 consensus_queue.cc:1048] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 12:23:39.415690 17801 leader_election.cc:400] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 17 election: Vote denied by peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651) with higher term. Message: Invalid argument: T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 17. Current term is 18.
I20260709 12:23:39.416481 17801 leader_election.cc:403] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 17 election: Cancelling election due to peer responding with higher term
I20260709 12:23:39.417331 24430 raft_consensus.cc:3060] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:39.417445 24421 leader_election.cc:290] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:39.418704 24421 raft_consensus.cc:493] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:39.419437 24421 raft_consensus.cc:515] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:23:39.420210 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:39.420754 17801 leader_election.cc:304] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:39.406877 17652 leader_election.cc:304] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:39.424582 17652 leader_election.cc:304] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:39.425100 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:39.427692 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:39.428071 17756 raft_consensus.cc:1275] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 17 index: 10. Preceding OpId from leader: term: 20 index: 11. (index mismatch)
I20260709 12:23:39.429392 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:39.430590 17838 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 15.
I20260709 12:23:39.388418 24247 raft_consensus.cc:2804] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:23:39.434080 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:39.432145 17652 leader_election.cc:304] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:39.436410 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 21 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:39.398391 24313 raft_consensus.cc:2804] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:23:39.438946 17727 leader_election.cc:304] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:39.405481 24432 raft_consensus.cc:493] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:39.439200 17838 raft_consensus.cc:1275] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 18 index: 11. Preceding OpId from leader: term: 19 index: 12. (index mismatch)
I20260709 12:23:39.403343 24087 raft_consensus.cc:493] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:23:39.403913 24203 raft_consensus.cc:2749] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:23:39.434638 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:39.438035 17728 leader_election.cc:304] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:39.443688 17727 leader_election.cc:304] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:39.445650 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 21 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:39.447539 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:39.447661 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:39.449451 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:39.426210 24441 raft_consensus.cc:2749] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:23:39.451251 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:39.463099 17652 leader_election.cc:304] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:39.451624 17834 raft_consensus.cc:1240] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 14. Current term is 15. Ops: []
I20260709 12:23:39.431571 24448 raft_consensus.cc:493] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:39.456635 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 21 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:39.456883 17686 raft_consensus.cc:1275] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 19 index: 11. Preceding OpId from leader: term: 20 index: 11. (term mismatch)
I20260709 12:23:39.478397 17686 pending_rounds.cc:85] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3: Aborting all ops after (but not including) 10
I20260709 12:23:39.478555 17836 raft_consensus.cc:1240] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 18. Current term is 19. Ops: []
I20260709 12:23:39.478734 17686 pending_rounds.cc:107] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3: Aborting uncommitted NO_OP operation due to leader change: 19.11
I20260709 12:23:39.479036 17686 raft_consensus.cc:2889] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:39.430259 24424 raft_consensus.cc:2804] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:23:39.430477 24244 raft_consensus.cc:2804] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:23:39.388120 23997 consensus_queue.cc:1048] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.049s
I20260709 12:23:39.460857 17829 raft_consensus.cc:2468] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 11.
W20260709 12:23:39.487355 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 49908.4us
I20260709 12:23:39.461102 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:39.454671 17687 raft_consensus.cc:1275] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 14 index: 13. Preceding OpId from leader: term: 16 index: 14. (index mismatch)
I20260709 12:23:39.465754 17802 leader_election.cc:304] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:39.467623 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:39.482028 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:39.434154 24247 raft_consensus.cc:493] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:23:39.485450 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:39.490533 17837 raft_consensus.cc:2468] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 16.
W20260709 12:23:39.493577 17761 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36572 (request call id 1145) took 2649 ms (client timeout 3167 ms). Trace:
W20260709 12:23:39.494108 17761 rpcz_store.cc:269] 0709 12:23:36.843736 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:37.027554 (+183818us) service_pool.cc:224] Handling call
0709 12:23:39.493370 (+2465816us) spinlock_profiling.cc:243] Waited 2.3 s on lock 0x7b54001b9254. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:23:39.493513 (+   143us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5294951680}
I20260709 12:23:39.440254 24313 raft_consensus.cc:493] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:39.498106 18334 leader_election.cc:304] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:23:39.457790 24169 raft_consensus.cc:2804] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:23:39.491843 24247 raft_consensus.cc:3060] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:39.448017 24425 consensus_queue.cc:1048] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.051s
I20260709 12:23:39.443441 24437 consensus_queue.cc:1059] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 18.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.054s
I20260709 12:23:39.529929 17802 leader_election.cc:304] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:39.458856 24432 raft_consensus.cc:515] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:39.491458 24244 raft_consensus.cc:493] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:23:39.462703 23740 consensus_queue.cc:1048] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.045s
I20260709 12:23:39.572608 17802 leader_election.cc:304] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:39.503286 24169 raft_consensus.cc:493] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
W20260709 12:23:39.573901 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 65270.4us
I20260709 12:23:39.585517 17802 leader_election.cc:304] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:39.588025 18333 leader_election.cc:304] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:39.573490 24328 raft_consensus.cc:2749] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:23:39.461299 24087 raft_consensus.cc:515] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:39.497809 23906 raft_consensus.cc:2804] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Leader pre-election won for term 17
W20260709 12:23:39.575627 24455 log.cc:927] Time spent T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.152s	user 0.000s	sys 0.001s
I20260709 12:23:39.566717 24441 raft_consensus.cc:2804] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:23:39.569216 24421 leader_election.cc:290] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:23:39.631793 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 43835.2us
I20260709 12:23:39.500509 24313 raft_consensus.cc:3060] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:23:39.618427 17761 raft_consensus.cc:1275] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 17 index: 11. Preceding OpId from leader: term: 18 index: 12. (index mismatch)
I20260709 12:23:39.486102 24448 raft_consensus.cc:515] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:39.618338 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:39.496951 24458 consensus_queue.cc:1048] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.143s
W20260709 12:23:39.598464 24456 log.cc:927] Time spent T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.112s	user 0.000s	sys 0.000s
I20260709 12:23:39.508733 24442 raft_consensus.cc:2749] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:23:39.634266 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:39.575911 23838 raft_consensus.cc:493] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:23:39.614954 17802 leader_election.cc:304] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:23:39.584669 24439 raft_consensus.cc:2749] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:23:39.506263 24424 raft_consensus.cc:697] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 21 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:23:39.569602 24418 consensus_queue.cc:1048] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.062s
I20260709 12:23:39.588989 24169 raft_consensus.cc:3060] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:39.707448 17683 raft_consensus.cc:2468] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 18.
W20260709 12:23:39.644821 17729 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:39.680275 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:39.629289 23906 raft_consensus.cc:493] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:23:39.642680 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 19 candidate_status { last_received { term: 18 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:39.661649 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:39.638410 24231 raft_consensus.cc:2749] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 12:23:39.508476 24394 consensus_queue.cc:1048] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.151s
I20260709 12:23:39.604301 24437 raft_consensus.cc:493] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:39.702503 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 87679.3us
I20260709 12:23:39.602713 24429 consensus_queue.cc:1048] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.104s
I20260709 12:23:39.740348 17802 leader_election.cc:304] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:39.741537 17802 leader_election.cc:304] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:39.704187 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:39.629299 24441 raft_consensus.cc:493] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:39.744678 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:39.630765 24262 raft_consensus.cc:2749] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:23:39.706631 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:39.659349 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 16 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:39.597371 24432 leader_election.cc:290] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:39.708472 18371 raft_consensus.cc:3060] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:39.592150 24098 raft_consensus.cc:2804] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:23:39.707192 23740 raft_consensus.cc:2804] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:23:39.712514 18363 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 15.
I20260709 12:23:39.591235 24244 raft_consensus.cc:3060] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:23:39.712380 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 274430us
I20260709 12:23:39.788820 17728 leader_election.cc:304] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:39.719177 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 15 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:39.720160 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 20 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:39.720502 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 20 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:39.790145 17763 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 14.
I20260709 12:23:39.722054 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:39.726339 17803 leader_election.cc:304] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:39.730777 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:39.791445 24432 consensus_queue.cc:1048] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:23:39.728470 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:39.734773 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 16 candidate_status { last_received { term: 14 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:39.793221 17834 raft_consensus.cc:2468] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 15.
I20260709 12:23:39.735816 18361 raft_consensus.cc:1275] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 14 index: 13. Preceding OpId from leader: term: 16 index: 14. (index mismatch)
I20260709 12:23:39.673825 24391 leader_election.cc:290] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 13 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:23:39.737763 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 294167us
I20260709 12:23:39.694974 23931 raft_consensus.cc:2749] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:23:39.742791 18362 raft_consensus.cc:2368] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 17. Current term is 18.
I20260709 12:23:39.743836 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:39.751793 18367 raft_consensus.cc:2393] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 21: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:23:39.797278 17833 raft_consensus.cc:3060] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:39.659801 24427 consensus_queue.cc:1048] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.078s
I20260709 12:23:39.784595 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 15 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:39.799898 17836 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 14.
I20260709 12:23:39.733419 23906 raft_consensus.cc:3060] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:39.786695 24421 consensus_queue.cc:1059] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.029s
I20260709 12:23:39.790172 17838 raft_consensus.cc:3060] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:39.786949 24231 consensus_queue.cc:1048] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.029s
I20260709 12:23:39.783871 24098 raft_consensus.cc:493] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:39.818181 24098 raft_consensus.cc:3060] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:39.820245 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:39.821503 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:39.822109 17836 raft_consensus.cc:3060] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:39.795377 17728 leader_election.cc:304] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:39.824896 24424 consensus_queue.cc:237] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 19.14, Last appended by leader: 14, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:39.701484 23838 raft_consensus.cc:515] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:39.836028 17838 raft_consensus.cc:2468] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 20.
I20260709 12:23:39.798306 18371 raft_consensus.cc:2468] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 17.
I20260709 12:23:39.786305 23905 raft_consensus.cc:2749] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 12:23:39.743642 24437 raft_consensus.cc:515] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:39.804117 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:23:39.805619 24465 log.cc:927] Time spent T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.148s	user 0.001s	sys 0.000s
W20260709 12:23:39.806680 24468 log.cc:927] Time spent T 00000000000000000000000000000000 P 2193af31cb894ec9b3d65d939a2a6851: Append to log took a long time: real 0.096s	user 0.000s	sys 0.000s
W20260709 12:23:39.807520 24467 log.cc:927] Time spent T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.067s	user 0.000s	sys 0.000s
W20260709 12:23:39.808203 24464 log.cc:927] Time spent T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.112s	user 0.000s	sys 0.001s
I20260709 12:23:39.808697 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
W20260709 12:23:39.809713 24469 log.cc:927] Time spent T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.108s	user 0.000s	sys 0.000s
I20260709 12:23:39.783237 24441 raft_consensus.cc:3060] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:39.783419 24328 consensus_queue.cc:1048] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.029s
I20260709 12:23:39.846285 17727 leader_election.cc:304] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
W20260709 12:23:39.881484 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 51929.3us
I20260709 12:23:39.882270 17728 leader_election.cc:304] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:39.882506 17727 leader_election.cc:304] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:39.840821 17836 raft_consensus.cc:2468] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 15.
W20260709 12:23:39.885067 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 53340.4us
I20260709 12:23:39.800897 24262 raft_consensus.cc:2749] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:23:39.751431 24429 consensus_queue.cc:1243] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer e2909cff1da44ad387f96a8ae3aa5afb log is divergent from this leader: its last log entry 14.12 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 11
I20260709 12:23:39.844036 24051 raft_consensus.cc:493] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:39.847388 24394 raft_consensus.cc:493] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:39.848307 24427 raft_consensus.cc:2749] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:23:39.844990 24432 raft_consensus.cc:2804] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:23:39.877750 24392 raft_consensus.cc:493] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:39.844323 23740 raft_consensus.cc:697] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 17 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:23:39.843554 24475 raft_consensus.cc:2804] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Leader pre-election won for term 19
W20260709 12:23:39.898433 24410 log.cc:927] Time spent T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.051s	user 0.000s	sys 0.000s
I20260709 12:23:39.892875 23997 raft_consensus.cc:2749] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
W20260709 12:23:39.933503 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 207226us
I20260709 12:23:39.898182 24328 raft_consensus.cc:3055] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 18 LEADER]: Stepping down as leader of term 18
W20260709 12:23:39.937876 17635 auto_rebalancer.cc:629] Failed to schedule move for tablet 22fa38faae58422e9cecc99973cab688: Illegal state: Replica e2909cff1da44ad387f96a8ae3aa5afb is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 20 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } }
I20260709 12:23:39.896701 24471 raft_consensus.cc:3060] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:39.807541 24250 leader_election.cc:290] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:39.939476 17833 raft_consensus.cc:2468] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 18.
W20260709 12:23:39.947571 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 221306us
W20260709 12:23:39.949249 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 207754us
I20260709 12:23:39.947705 18333 leader_election.cc:304] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:39.960237 24262 consensus_queue.cc:1048] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 12:23:39.962244 24437 leader_election.cc:290] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:39.963143 24262 consensus_queue.cc:1048] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:23:39.963064 23740 consensus_queue.cc:237] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.12, Last appended by leader: 12, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:39.930622 24394 raft_consensus.cc:515] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:39.975564 24421 raft_consensus.cc:493] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:39.975858 24394 leader_election.cc:290] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:39.977022 24247 raft_consensus.cc:515] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:39.977068 24421 raft_consensus.cc:515] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:39.979490 24247 leader_election.cc:290] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 19 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:39.980242 24421 leader_election.cc:290] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:39.980577 24443 raft_consensus.cc:493] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:23:39.981204 24443 raft_consensus.cc:515] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:39.981637 17685 raft_consensus.cc:1275] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 18 index: 13. (index mismatch)
I20260709 12:23:39.983122 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:39.983472 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:39.936661 24425 raft_consensus.cc:2804] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:23:39.984001 17687 raft_consensus.cc:2468] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 18.
I20260709 12:23:39.984191 17681 raft_consensus.cc:2468] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 13.
I20260709 12:23:39.991004 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:39.991755 17681 raft_consensus.cc:2393] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 17: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:23:39.985131 17689 raft_consensus.cc:1240] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 15. Current term is 16. Ops: []
W20260709 12:23:39.994212 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 46601.3us
I20260709 12:23:39.941618 24328 raft_consensus.cc:740] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 18 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
W20260709 12:23:40.011821 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 548749us
I20260709 12:23:40.023319 17652 leader_election.cc:304] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:40.024295 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:40.025955 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:40.026786 17682 raft_consensus.cc:3060] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:40.027060 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:40.027771 18365 raft_consensus.cc:3060] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:40.013835 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 16 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:40.002277 17729 leader_election.cc:304] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:40.020732 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 16 candidate_status { last_received { term: 14 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:23:40.028693 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:40.021339 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:23:40.029148 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:39.946410 24448 leader_election.cc:290] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:40.029421 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:40.030534 18368 raft_consensus.cc:3060] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:40.029125 17681 raft_consensus.cc:2393] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 16: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:23:40.023762 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:40.029417 17686 raft_consensus.cc:2468] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 10.
I20260709 12:23:39.926199 24051 raft_consensus.cc:515] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:40.019289 18333 leader_election.cc:304] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:40.029131 17680 raft_consensus.cc:2468] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 15.
I20260709 12:23:40.038610 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 15 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:40.026039 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:39.956384 24401 raft_consensus.cc:493] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:23:39.942180 24392 raft_consensus.cc:515] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:40.045894 18361 raft_consensus.cc:1240] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 12. Current term is 13. Ops: []
I20260709 12:23:40.047667 18367 raft_consensus.cc:2393] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 17: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:23:40.049674 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:40.057121 18362 raft_consensus.cc:3060] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:40.057386 18364 raft_consensus.cc:1240] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 18. Current term is 19. Ops: []
I20260709 12:23:40.051542 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:40.057801 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:39.999284 24087 leader_election.cc:290] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:40.039012 18333 leader_election.cc:304] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:23:40.058282 17652 leader_election.cc:304] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:40.009176 24431 raft_consensus.cc:2749] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
W20260709 12:23:40.041550 24430 consensus_meta.cc:287] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.624s	user 0.012s	sys 0.000s
I20260709 12:23:40.073266 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:40.057458 24247 raft_consensus.cc:493] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:23:40.041864 24237 raft_consensus.cc:2804] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:23:40.075109 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:40.093528 17685 raft_consensus.cc:2368] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for earlier term 15. Current term is 16.
I20260709 12:23:40.076135 17680 raft_consensus.cc:1275] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 16 index: 12. Preceding OpId from leader: term: 18 index: 13. (index mismatch)
I20260709 12:23:40.019626 24425 raft_consensus.cc:697] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 15 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:23:40.079523 17681 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 15.
I20260709 12:23:40.096702 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:40.032701 24236 raft_consensus.cc:2749] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:23:40.050259 24393 raft_consensus.cc:2804] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:23:40.081538 18365 raft_consensus.cc:2468] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 16.
I20260709 12:23:40.086103 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:40.102200 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:40.103502 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:40.104655 17688 raft_consensus.cc:2468] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 18.
I20260709 12:23:40.105880 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:40.106590 17688 raft_consensus.cc:3060] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Advancing to term 20
W20260709 12:23:40.109174 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 226721us
I20260709 12:23:40.051158 24250 raft_consensus.cc:493] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:23:40.092630 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:40.097564 17681 raft_consensus.cc:2468] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 18.
I20260709 12:23:40.074643 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:40.090250 17682 raft_consensus.cc:2410] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for term 20 because replica has last-logged OpId of term: 19 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 11.
I20260709 12:23:40.122071 18366 raft_consensus.cc:2468] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 18.
I20260709 12:23:40.118081 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:40.034370 24458 consensus_queue.cc:1059] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 18.16, Next index: 17, Last known committed idx: 15, Time since last communication: 0.060s
I20260709 12:23:40.131467 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 21 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:40.132385 17685 raft_consensus.cc:2468] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 20.
I20260709 12:23:40.094935 24430 raft_consensus.cc:2749] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Leader election lost for term 17. Reason: Vote denied by peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651) with higher term. Message: Invalid argument: T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 17. Current term is 18.
I20260709 12:23:40.146667 24495 raft_consensus.cc:493] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:40.139561 24250 raft_consensus.cc:515] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:40.147362 24495 raft_consensus.cc:515] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:40.149116 24250 leader_election.cc:290] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:40.149852 24495 leader_election.cc:290] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:23:40.150203 24481 log.cc:927] Time spent T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.065s	user 0.001s	sys 0.000s
I20260709 12:23:40.150631 24495 raft_consensus.cc:2804] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:23:40.151165 24495 raft_consensus.cc:697] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 18 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:23:40.166916 24169 raft_consensus.cc:515] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:40.110327 24328 consensus_queue.cc:260] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.12, Last appended by leader: 12, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:40.167985 24443 leader_election.cc:290] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:40.176807 24458 raft_consensus.cc:493] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:40.177696 23838 leader_election.cc:290] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:40.095970 24247 raft_consensus.cc:515] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:40.190828 24425 consensus_queue.cc:237] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.10, Last appended by leader: 10, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:40.141227 24211 raft_consensus.cc:2804] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:23:40.192199 24211 raft_consensus.cc:493] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:40.192725 24211 raft_consensus.cc:3060] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:40.139806 23740 raft_consensus.cc:493] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:40.196341 23740 raft_consensus.cc:515] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:40.197475 24098 raft_consensus.cc:515] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:40.198993 24051 leader_election.cc:290] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:40.199263 23740 leader_election.cc:290] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:40.201227 18366 raft_consensus.cc:1275] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 12 index: 9. Preceding OpId from leader: term: 14 index: 10. (index mismatch)
I20260709 12:23:40.201783 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 16 candidate_status { last_received { term: 14 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:40.202293 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
W20260709 12:23:40.159698 24313 consensus_meta.cc:287] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.500s	user 0.003s	sys 0.005s
I20260709 12:23:40.062063 24401 raft_consensus.cc:515] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:40.204646 24313 raft_consensus.cc:515] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:40.205830 24227 raft_consensus.cc:2804] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:23:40.206224 24227 raft_consensus.cc:493] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:23:40.206671 24227 raft_consensus.cc:3060] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:40.207175 24401 leader_election.cc:290] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:40.188423 24458 raft_consensus.cc:515] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:40.208802 24098 leader_election.cc:290] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 16 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:40.207347 24313 leader_election.cc:290] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 23 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:40.207791 24244 raft_consensus.cc:515] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:40.210557 24458 leader_election.cc:290] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:40.191632 24247 leader_election.cc:290] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:40.211741 24244 leader_election.cc:290] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 15 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:40.186987 17688 raft_consensus.cc:2468] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 20.
W20260709 12:23:40.216311 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 158104us
I20260709 12:23:40.217056 17652 leader_election.cc:304] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:40.113227 24393 raft_consensus.cc:493] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:23:40.202385 18364 raft_consensus.cc:2468] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 15.
I20260709 12:23:40.226850 18362 raft_consensus.cc:2468] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 16.
I20260709 12:23:40.230567 17728 leader_election.cc:400] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 15 pre-election: Vote denied by peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071) with higher term. Message: Invalid argument: T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for earlier term 15. Current term is 16.
I20260709 12:23:40.230712 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:40.231295 23740 raft_consensus.cc:493] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:40.230929 17728 leader_election.cc:403] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 15 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:23:40.241236 18334 leader_election.cc:304] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:23:40.242426 18334 leader_election.cc:304] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:40.119848 23906 raft_consensus.cc:515] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:40.248979 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:40.257982 17831 raft_consensus.cc:2468] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 18.
W20260709 12:23:40.227766 24489 log.cc:927] Time spent T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.083s	user 0.000s	sys 0.001s
I20260709 12:23:40.224702 24393 raft_consensus.cc:3060] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:40.267326 18334 leader_election.cc:304] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:23:40.268983 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:40.228178 24087 raft_consensus.cc:2749] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
W20260709 12:23:40.267443 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 237907us
I20260709 12:23:40.248600 24237 raft_consensus.cc:697] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 20 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
W20260709 12:23:40.287755 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 66189.4us
I20260709 12:23:40.261896 24432 raft_consensus.cc:697] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:23:40.276472 17654 leader_election.cc:304] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:23:40.277451 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 60487.7us
W20260709 12:23:40.271873 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 133142us
I20260709 12:23:40.259120 23953 consensus_queue.cc:1048] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.027s
I20260709 12:23:40.296348 17652 leader_election.cc:304] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:40.301656 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:40.302367 17683 raft_consensus.cc:2468] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 15.
I20260709 12:23:40.296293 24494 consensus_queue.cc:1048] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 11, Time since last communication: 0.015s
I20260709 12:23:40.303285 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:40.281049 24458 raft_consensus.cc:2804] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:23:40.304953 17683 raft_consensus.cc:2368] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for earlier term 15. Current term is 16.
I20260709 12:23:40.307078 18334 leader_election.cc:304] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:23:40.307188 17652 leader_election.cc:304] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:23:40.230897 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 191997us
I20260709 12:23:40.258992 23740 raft_consensus.cc:515] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:40.261821 23838 raft_consensus.cc:2804] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:23:40.289434 17802 leader_election.cc:304] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:40.289673 17728 leader_election.cc:304] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:40.311481 18333 leader_election.cc:304] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:23:40.290249 23931 raft_consensus.cc:2749] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:23:40.319553 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 14 candidate_status { last_received { term: 13 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:40.274142 24437 consensus_queue.cc:1059] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 15.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.027s
I20260709 12:23:40.289064 24436 raft_consensus.cc:493] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:23:40.330873 18368 raft_consensus.cc:2468] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 20.
I20260709 12:23:40.332516 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 15 candidate_status { last_received { term: 14 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:40.278367 24490 consensus_queue.cc:1048] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.047s
I20260709 12:23:40.294971 24501 raft_consensus.cc:2804] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:23:40.289973 24434 consensus_queue.cc:1048] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.033s
I20260709 12:23:40.316679 23838 raft_consensus.cc:493] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:23:40.309212 24491 raft_consensus.cc:3060] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Advancing to term 16
I20260709 12:23:40.390013 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:40.324020 24483 raft_consensus.cc:493] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:40.317909 24496 raft_consensus.cc:493] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:40.315670 24482 raft_consensus.cc:2804] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:23:40.318243 24448 raft_consensus.cc:2749] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:23:40.332228 24485 raft_consensus.cc:2804] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:23:40.340054 24430 raft_consensus.cc:493] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
W20260709 12:23:40.459681 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 150297us
I20260709 12:23:40.333642 24391 raft_consensus.cc:493] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:40.459475 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 161780us
I20260709 12:23:40.388827 24501 raft_consensus.cc:493] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:23:40.465859 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 176244us
I20260709 12:23:40.283630 24495 consensus_queue.cc:237] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.12, Last appended by leader: 12, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:40.466514 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 155088us
I20260709 12:23:40.506014 18333 leader_election.cc:304] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:40.396960 23838 raft_consensus.cc:3060] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:40.397768 24428 raft_consensus.cc:2749] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:23:40.489621 17728 leader_election.cc:304] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:40.491384 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:40.394399 24304 raft_consensus.cc:2804] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:23:40.408803 24169 leader_election.cc:290] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 19 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:23:40.550155 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 60584.3us
W20260709 12:23:40.518905 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:40.553166 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:40.506994 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 199914us
I20260709 12:23:40.405962 24431 raft_consensus.cc:3055] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:23:40.399646 23740 leader_election.cc:290] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:40.551406 17728 leader_election.cc:304] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:40.556412 17652 leader_election.cc:304] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:40.477490 24485 raft_consensus.cc:493] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:40.472615 24482 raft_consensus.cc:493] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:40.338002 24436 raft_consensus.cc:515] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:40.608028 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 56768.4us
I20260709 12:23:40.609349 24476 raft_consensus.cc:2804] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:23:40.609509 17728 leader_election.cc:304] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:40.610205 24476 raft_consensus.cc:493] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:40.567790 17652 leader_election.cc:304] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:40.610793 24476 raft_consensus.cc:3060] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Advancing to term 19
W20260709 12:23:40.611320 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 43668.1us
I20260709 12:23:40.503818 23997 raft_consensus.cc:2804] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:23:40.614082 23931 consensus_queue.cc:1048] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 12:23:40.597038 17685 raft_consensus.cc:3060] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:40.392387 24510 consensus_queue.cc:1048] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.218s
I20260709 12:23:40.607998 24490 consensus_queue.cc:1048] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:23:40.612134 24428 raft_consensus.cc:2804] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:23:40.506198 24501 raft_consensus.cc:3060] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:40.619725 17728 leader_election.cc:304] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:40.619784 17652 leader_election.cc:304] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:23:40.550097 24448 raft_consensus.cc:493] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:40.392630 24237 consensus_queue.cc:237] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.13, Last appended by leader: 13, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:40.606086 24482 raft_consensus.cc:3060] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:40.559350 24431 raft_consensus.cc:740] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:23:40.640996 24425 raft_consensus.cc:2749] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 12:23:40.483058 24439 consensus_queue.cc:1059] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 17.13, Next index: 14, Last known committed idx: 10, Time since last communication: 0.120s
I20260709 12:23:40.615375 23997 raft_consensus.cc:493] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:40.597609 23740 raft_consensus.cc:493] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:23:40.600733 24509 raft_consensus.cc:2749] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:23:40.467792 24483 raft_consensus.cc:515] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:40.512161 24391 raft_consensus.cc:515] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:40.604360 24485 raft_consensus.cc:3060] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:40.668964 24515 raft_consensus.cc:493] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:23:40.707968 24518 raft_consensus.cc:2804] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:23:40.737289 24524 raft_consensus.cc:1064] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb: attempting to promote NON_VOTER 334a4e1f59944544b3b3ff36616b57da to VOTER
I20260709 12:23:40.548141 24474 raft_consensus.cc:493] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
W20260709 12:23:40.671523 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 165675us
I20260709 12:23:40.702984 24304 raft_consensus.cc:697] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 20 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:23:40.605023 24475 raft_consensus.cc:493] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:23:40.635565 24203 raft_consensus.cc:3060] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Advancing to term 17
W20260709 12:23:40.702977 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 83273.4us
I20260709 12:23:40.618177 24087 raft_consensus.cc:2749] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:23:40.625314 24418 raft_consensus.cc:493] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:23:40.636368 24428 raft_consensus.cc:493] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:23:40.607211 24227 raft_consensus.cc:515] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:40.787129 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 19 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:40.712709 17685 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 16.
I20260709 12:23:40.672006 24037 raft_consensus.cc:2804] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:23:40.505260 24430 raft_consensus.cc:515] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:40.804164 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:40.511368 24496 raft_consensus.cc:515] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:40.624715 24169 consensus_queue.cc:1059] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 18.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.079s
W20260709 12:23:40.721557 24499 log.cc:927] Time spent T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.116s	user 0.000s	sys 0.000s
I20260709 12:23:40.704849 23997 raft_consensus.cc:3060] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:40.725677 24425 raft_consensus.cc:2804] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:23:40.642163 24507 consensus_queue.cc:1059] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 12.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.105s
I20260709 12:23:40.728271 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:40.512679 24417 consensus_queue.cc:1059] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 12.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.121s
I20260709 12:23:40.730913 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 14 candidate_status { last_received { term: 10 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:40.831746 17681 raft_consensus.cc:1275] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 17 index: 11. Preceding OpId from leader: term: 18 index: 12. (index mismatch)
I20260709 12:23:40.559844 24432 consensus_queue.cc:237] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.11, Last appended by leader: 11, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:40.696410 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 84494.8us
I20260709 12:23:40.835952 17728 leader_election.cc:304] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:40.744367 24431 consensus_queue.cc:260] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 18.16, Last appended by leader: 16, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:40.746393 24458 raft_consensus.cc:493] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:23:40.838249 24458 raft_consensus.cc:3060] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:40.839486 24496 leader_election.cc:290] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:40.756325 17654 leader_election.cc:304] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
W20260709 12:23:40.842315 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 86124.9us
W20260709 12:23:40.824628 24512 log.cc:927] Time spent T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.124s	user 0.000s	sys 0.001s
I20260709 12:23:40.844007 17654 leader_election.cc:304] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:23:39.738831 16501 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:23:40.845628 24244 raft_consensus.cc:2804] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Leader election won for term 16
W20260709 12:23:40.734622 24441 consensus_meta.cc:287] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.860s	user 0.012s	sys 0.000s
I20260709 12:23:40.848204 17654 leader_election.cc:304] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:23:40.848690 24441 raft_consensus.cc:515] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:40.761000 18333 leader_election.cc:304] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
W20260709 12:23:40.853324 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 92405.2us
I20260709 12:23:40.854242 18374 raft_consensus.cc:1240] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 19. Current term is 20. Ops: [19.12-19.12]
W20260709 12:23:40.734128 24211 consensus_meta.cc:287] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.541s	user 0.010s	sys 0.000s
I20260709 12:23:40.849647 23931 consensus_queue.cc:1048] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.006s
I20260709 12:23:40.855031 24211 raft_consensus.cc:515] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:40.731472 24523 raft_consensus.cc:493] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:23:40.856406 24523 raft_consensus.cc:515] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:40.760432 24474 raft_consensus.cc:515] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:40.824172 24425 raft_consensus.cc:493] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:23:40.858546 24425 raft_consensus.cc:3060] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:40.858660 24523 leader_election.cc:290] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:40.735184 24518 raft_consensus.cc:493] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:23:40.785917 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 16 candidate_status { last_received { term: 14 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:40.859133 24474 leader_election.cc:290] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:40.865024 24474 raft_consensus.cc:2804] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:23:40.864689 24304 consensus_queue.cc:237] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.13, Last appended by leader: 13, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:40.865844 24474 raft_consensus.cc:697] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 16 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:23:40.704077 24448 raft_consensus.cc:515] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:40.859628 24362 raft_consensus.cc:3055] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:23:40.868583 24362 raft_consensus.cc:740] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:23:40.869505 24448 leader_election.cc:290] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:40.870229 24362 consensus_queue.cc:260] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 19.12, Last appended by leader: 12, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:40.870855 24430 leader_election.cc:290] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:40.806813 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:40.873821 17682 raft_consensus.cc:1240] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 19. Current term is 20. Ops: []
I20260709 12:23:40.875106 17683 raft_consensus.cc:1275] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
I20260709 12:23:40.808944 17833 raft_consensus.cc:1275] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 15 index: 10. Preceding OpId from leader: term: 16 index: 11. (index mismatch)
I20260709 12:23:40.809640 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:40.810585 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:40.766958 24475 raft_consensus.cc:3060] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:40.818763 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 19 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:40.787662 24428 raft_consensus.cc:3060] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:40.733791 24393 raft_consensus.cc:515] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:40.803282 24037 raft_consensus.cc:493] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:40.836809 17681 raft_consensus.cc:1240] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 16. Current term is 17. Ops: []
I20260709 12:23:40.730262 24515 raft_consensus.cc:515] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:40.854384 18333 leader_election.cc:304] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:23:40.856705 23931 raft_consensus.cc:2804] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:23:40.715303 23740 raft_consensus.cc:515] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:40.868803 24483 leader_election.cc:290] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:40.881917 23906 leader_election.cc:290] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 17 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:40.789004 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:40.878532 17836 raft_consensus.cc:2468] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 15.
I20260709 12:23:40.850271 24476 raft_consensus.cc:515] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:40.879684 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 15 candidate_status { last_received { term: 14 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
W20260709 12:23:40.880962 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 45130.8us
I20260709 12:23:40.787285 24418 raft_consensus.cc:515] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:40.857254 24211 leader_election.cc:290] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 12 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:40.859278 24523 raft_consensus.cc:2804] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:23:40.848980 24313 raft_consensus.cc:3055] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:23:40.952656 24313 raft_consensus.cc:740] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 19 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:23:40.952706 24523 raft_consensus.cc:697] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 15 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:23:40.954525 24523 consensus_queue.cc:237] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:40.871775 24362 raft_consensus.cc:3060] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:40.885583 24439 raft_consensus.cc:3055] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:23:40.941586 17728 leader_election.cc:304] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:40.879484 24518 raft_consensus.cc:3060] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:40.921188 17652 leader_election.cc:304] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:40.940761 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:40.960544 17837 raft_consensus.cc:2468] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 17.
I20260709 12:23:40.907194 24037 raft_consensus.cc:3060] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:40.970429 17728 leader_election.cc:400] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 15 pre-election: Vote denied by peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071) with higher term. Message: Invalid argument: T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for earlier term 15. Current term is 16.
I20260709 12:23:40.970983 17728 leader_election.cc:403] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 15 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:23:40.954331 24313 consensus_queue.cc:260] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 19.14, Last appended by leader: 14, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:40.955715 24439 raft_consensus.cc:740] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:23:40.937150 17834 raft_consensus.cc:2393] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 20: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:23:40.942148 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:40.866978 24474 consensus_queue.cc:237] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.11, Last appended by leader: 11, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:40.903856 24417 raft_consensus.cc:2749] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
W20260709 12:23:40.965175 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 44086.3us
I20260709 12:23:40.919111 23931 raft_consensus.cc:493] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:40.994848 17830 raft_consensus.cc:2468] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 14.
W20260709 12:23:40.964282 24516 log.cc:927] Time spent T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.057s	user 0.000s	sys 0.001s
I20260709 12:23:40.926774 24496 consensus_queue.cc:1048] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.072s
I20260709 12:23:40.979401 24448 raft_consensus.cc:2749] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:23:40.968837 24441 leader_election.cc:290] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 20 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:40.929620 23740 leader_election.cc:290] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:41.004408 23931 raft_consensus.cc:3060] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Advancing to term 16
W20260709 12:23:41.037523 24491 consensus_meta.cc:287] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.571s	user 0.007s	sys 0.005s
I20260709 12:23:41.068305 24491 raft_consensus.cc:2749] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Leader pre-election lost for term 15. Reason: Vote denied by peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071) with higher term. Message: Invalid argument: T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for earlier term 15. Current term is 16.
I20260709 12:23:41.052695 24391 leader_election.cc:290] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:23:41.073822 24471 consensus_meta.cc:287] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 1.090s	user 0.009s	sys 0.004s
I20260709 12:23:41.081064 24495 raft_consensus.cc:2804] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:23:41.085676 23906 raft_consensus.cc:3060] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Advancing to term 16
I20260709 12:23:41.041194 24418 leader_election.cc:290] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:41.022593 24313 raft_consensus.cc:3060] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Advancing to term 20
W20260709 12:23:41.107146 17729 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.617s	user 0.016s	sys 0.010s
W20260709 12:23:41.127766 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:41.137846 17729 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.648s	user 0.016s	sys 0.010s
W20260709 12:23:41.133126 24508 log.cc:927] Time spent T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.645s	user 0.001s	sys 0.001s
W20260709 12:23:41.131636 24506 log.cc:927] Time spent T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.648s	user 0.000s	sys 0.002s
W20260709 12:23:41.148005 17727 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.652s	user 0.025s	sys 0.044s
W20260709 12:23:41.173292 17727 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.677s	user 0.026s	sys 0.045s
W20260709 12:23:41.174065 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 686760us
W20260709 12:23:41.138535 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 648894us
W20260709 12:23:41.118691 23838 consensus_meta.cc:287] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.560s	user 0.006s	sys 0.006s
W20260709 12:23:41.184356 17801 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:41.187973 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:41.189639 18361 raft_consensus.cc:2368] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for earlier term 15. Current term is 16.
I20260709 12:23:41.195379 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 19 candidate_status { last_received { term: 18 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
W20260709 12:23:41.149169 18335 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.525s	user 0.005s	sys 0.023s
W20260709 12:23:41.202003 18335 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.583s	user 0.006s	sys 0.023s
W20260709 12:23:41.206648 17729 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:41.117362 24441 raft_consensus.cc:2804] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:23:41.122277 24471 raft_consensus.cc:2749] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:23:41.236516 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:41.148330 24495 raft_consensus.cc:697] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 20 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
W20260709 12:23:41.177982 17803 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.537s	user 0.017s	sys 0.013s
W20260709 12:23:41.231833 24482 consensus_meta.cc:287] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.518s	user 0.013s	sys 0.000s
W20260709 12:23:41.220415 17686 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.732s	user 0.012s	sys 0.019s
I20260709 12:23:41.152360 24418 raft_consensus.cc:493] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:23:41.079746 24425 raft_consensus.cc:515] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:23:41.293324 17686 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.811s	user 0.012s	sys 0.019s
W20260709 12:23:41.249516 17803 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.608s	user 0.017s	sys 0.014s
I20260709 12:23:41.229969 24227 leader_election.cc:290] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 11 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:23:41.301352 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 660523us
W20260709 12:23:41.236246 24494 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.833s	user 0.007s	sys 0.022s
I20260709 12:23:41.242488 24393 leader_election.cc:290] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 16 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:41.242978 24441 raft_consensus.cc:697] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 18 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:23:41.108582 24439 consensus_queue.cc:260] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.10, Last appended by leader: 10, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:41.227491 23838 raft_consensus.cc:515] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:41.240847 24485 raft_consensus.cc:515] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:23:41.438284 17689 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.565s	user 0.022s	sys 0.026s
I20260709 12:23:41.374540 17686 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37402 (request call id 3124) took 1077 ms. Trace:
W20260709 12:23:41.402565 17654 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.553s	user 0.008s	sys 0.019s
W20260709 12:23:41.452322 17654 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.603s	user 0.009s	sys 0.021s
I20260709 12:23:41.450675 17686 rpcz_store.cc:276] 0709 12:23:40.297103 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:40.297245 (+   142us) service_pool.cc:224] Handling call
0709 12:23:40.313603 (+ 16358us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:40.336703 (+ 23100us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:40.336753 (+    50us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:40.337601 (+   848us) log.cc:844] Serialized 26 byte log entry
0709 12:23:41.300655 (+963054us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:23:41.300726 (+    71us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:41.300819 (+    93us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:41.374250 (+ 73431us) raft_consensus.cc:1685] finished
0709 12:23:41.374281 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:41.374461 (+   180us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":24517,"raft.queue_time_us":31284,"raft.run_cpu_time_us":290,"raft.run_wall_time_us":21552,"thread_start_us":817984,"threads_started":1,"wal-append.queue_time_us":952646}
W20260709 12:23:41.452780 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 605015us
W20260709 12:23:41.450060 17689 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.577s	user 0.023s	sys 0.026s
I20260709 12:23:41.296247 24515 leader_election.cc:290] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:23:41.456838 17688 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.585s	user 0.012s	sys 0.020s
W20260709 12:23:41.463213 17688 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.591s	user 0.013s	sys 0.020s
I20260709 12:23:41.291631 24244 raft_consensus.cc:697] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 16 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
W20260709 12:23:41.361899 24458 consensus_meta.cc:287] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.523s	user 0.010s	sys 0.003s
W20260709 12:23:41.326725 24437 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.933s	user 0.005s	sys 0.028s
W20260709 12:23:41.474467 24437 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.086s	user 0.006s	sys 0.029s
W20260709 12:23:41.448184 24494 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.045s	user 0.007s	sys 0.024s
I20260709 12:23:41.424300 24430 raft_consensus.cc:493] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:41.476107 24458 raft_consensus.cc:515] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:41.476482 24430 raft_consensus.cc:515] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:41.480247 24430 leader_election.cc:290] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:41.467947 17688 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44898 (request call id 1274) took 1171 ms. Trace:
I20260709 12:23:41.468650 17829 raft_consensus.cc:2393] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 19: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:23:41.482898 24437 raft_consensus.cc:493] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:41.390329 24503 log.cc:927] Time spent T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.925s	user 0.003s	sys 0.000s
W20260709 12:23:41.470118 23997 consensus_meta.cc:287] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.646s	user 0.015s	sys 0.000s
I20260709 12:23:41.484716 23997 raft_consensus.cc:515] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:41.393324 24434 consensus_queue.cc:1048] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.077s
I20260709 12:23:41.370980 24518 raft_consensus.cc:515] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:41.486878 23931 raft_consensus.cc:515] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:41.487458 23997 leader_election.cc:290] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 16 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:41.483791 24437 raft_consensus.cc:515] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:41.488751 24518 leader_election.cc:290] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 19 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:23:41.400475 24517 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.519s	user 0.000s	sys 0.033s
W20260709 12:23:41.489655 24517 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.612s	user 0.000s	sys 0.040s
I20260709 12:23:41.373799 24362 raft_consensus.cc:493] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:41.399595 24425 leader_election.cc:290] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 21 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:41.312868 24418 raft_consensus.cc:515] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:41.490489 24362 raft_consensus.cc:515] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:41.491216 24437 leader_election.cc:290] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:41.493007 24362 leader_election.cc:290] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:41.424389 24476 leader_election.cc:290] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 19 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:41.389192 24530 consensus_queue.cc:1048] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.105s
I20260709 12:23:41.481302 17688 rpcz_store.cc:276] 0709 12:23:40.296548 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:40.298408 (+  1860us) service_pool.cc:224] Handling call
0709 12:23:40.298715 (+   307us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:40.298930 (+   215us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 12:23:40.871534 (+572604us) log.cc:844] Serialized 14 byte log entry
0709 12:23:41.466113 (+594579us) log.cc:844] Serialized 14 byte log entry
0709 12:23:41.466318 (+   205us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:41.466373 (+    55us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:23:41.467220 (+   847us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:41.467314 (+    94us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:41.467655 (+   341us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":572187,"prepare.queue_time_us":610614,"prepare.run_cpu_time_us":79,"prepare.run_wall_time_us":80,"raft.queue_time_us":502,"raft.run_cpu_time_us":122,"raft.run_wall_time_us":3938,"spinlock_wait_cycles":1348352,"thread_start_us":593788,"threads_started":1}
I20260709 12:23:41.296447 24482 raft_consensus.cc:515] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:41.495652 24504 log.cc:927] Time spent T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 1.163s	user 0.004s	sys 0.000s
I20260709 12:23:41.497198 24418 leader_election.cc:290] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:41.472352 23838 leader_election.cc:290] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 14 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:23:41.497725 24475 consensus_meta.cc:287] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.595s	user 0.009s	sys 0.003s
I20260709 12:23:41.395635 24495 consensus_queue.cc:237] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:41.498180 24475 raft_consensus.cc:515] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:41.496282 24482 leader_election.cc:290] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 16 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:41.493064 18374 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59902 (request call id 1599) took 1858 ms. Trace:
I20260709 12:23:41.454561 17654 leader_election.cc:304] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:41.501888 24537 raft_consensus.cc:2749] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
W20260709 12:23:41.502012 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 47544.9us
I20260709 12:23:41.500064 18374 rpcz_store.cc:276] 0709 12:23:39.634814 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:39.635145 (+   331us) service_pool.cc:224] Handling call
0709 12:23:39.635554 (+   409us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:40.854715 (+1219161us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:41.492762 (+638047us) spinlock_profiling.cc:243] Waited 1.22 s on lock 0x7b540034c1d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:23:41.492995 (+   233us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2802072320}
I20260709 12:23:41.396895 24328 raft_consensus.cc:3060] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:41.462298 24484 raft_consensus.cc:493] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:41.510352 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:41.512950 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:41.514307 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 21 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:41.515053 17680 raft_consensus.cc:2468] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 20.
I20260709 12:23:41.512948 24484 raft_consensus.cc:515] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:41.518823 24484 leader_election.cc:290] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:23:41.520948 17728 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.548s	user 0.004s	sys 0.008s
W20260709 12:23:41.522385 17728 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.550s	user 0.005s	sys 0.009s
I20260709 12:23:41.472438 24244 consensus_queue.cc:237] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.12, Last appended by leader: 12, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:41.522857 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 550242us
I20260709 12:23:41.524003 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
W20260709 12:23:41.524791 17835 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.509s	user 0.007s	sys 0.004s
W20260709 12:23:41.525225 17835 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.509s	user 0.007s	sys 0.004s
I20260709 12:23:41.366753 24486 raft_consensus.cc:493] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:41.525022 24304 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.588s	user 0.015s	sys 0.019s
W20260709 12:23:41.535645 17730 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.650s	user 0.006s	sys 0.025s
W20260709 12:23:41.541656 17728 connection.cc:256] RPC call timeout handler was delayed by -1.1772209639998437s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 1.5s for a response
I20260709 12:23:41.512391 24250 raft_consensus.cc:493] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:41.564366 17684 raft_consensus.cc:1240] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 14. Current term is 15. Ops: []
I20260709 12:23:41.501026 24037 raft_consensus.cc:515] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:41.520452 24483 raft_consensus.cc:493] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:41.568089 24483 raft_consensus.cc:515] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:41.570823 24483 leader_election.cc:290] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:41.573704 24037 leader_election.cc:290] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 19 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:23:41.533473 24501 consensus_meta.cc:287] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.839s	user 0.017s	sys 0.000s
I20260709 12:23:41.576433 17684 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44898 (request call id 1262) took 1595 ms. Trace:
I20260709 12:23:41.576989 17684 rpcz_store.cc:276] 0709 12:23:39.981081 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:39.982803 (+  1722us) service_pool.cc:224] Handling call
0709 12:23:41.563857 (+1581054us) spinlock_profiling.cc:243] Waited 2.9 ms on lock 0x7b5800002a98. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b474fbb71 00007f7b474fbeee 000000000036eb39 00007f7b4f56f1e0 00007f7b4f6380c4 00007f7b4f638131 00007f7b4f5f9fc0 00007f7b4f5e4929 00007f7b4f5e43d2 00007f7b48b5bbd4
0709 12:23:41.564115 (+   258us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:41.576070 (+ 11955us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:41.576346 (+   276us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":6706304}
W20260709 12:23:41.549470 24523 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.590s	user 0.000s	sys 0.013s
W20260709 12:23:41.549546 17687 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.667s	user 0.003s	sys 0.008s
W20260709 12:23:41.562180 17730 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.677s	user 0.007s	sys 0.025s
W20260709 12:23:41.594856 17687 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.716s	user 0.004s	sys 0.009s
I20260709 12:23:41.596654 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:41.597313 18334 leader_election.cc:304] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:23:41.543069 24531 raft_consensus.cc:493] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:41.561913 24304 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.628s	user 0.015s	sys 0.020s
I20260709 12:23:41.567178 24485 leader_election.cc:290] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 19 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:23:41.574115 24203 consensus_meta.cc:287] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.804s	user 0.004s	sys 0.009s
I20260709 12:23:41.582731 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:41.584260 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 21 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:41.585856 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 17 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:41.586858 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
W20260709 12:23:41.549396 17685 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.649s	user 0.000s	sys 0.012s
I20260709 12:23:41.524822 17686 raft_consensus.cc:3060] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:41.575037 24539 raft_consensus.cc:493] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:23:41.591027 24098 raft_consensus.cc:493] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:23:41.591797 23931 leader_election.cc:290] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 16 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:41.593420 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
W20260709 12:23:41.633843 18363 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.508s	user 0.025s	sys 0.021s
W20260709 12:23:41.634160 18363 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.509s	user 0.025s	sys 0.021s
I20260709 12:23:41.598498 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:41.599370 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:41.613998 24515 raft_consensus.cc:2804] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:23:41.578891 24250 raft_consensus.cc:515] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:41.600513 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:41.602424 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:41.610630 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:41.638381 17759 raft_consensus.cc:2468] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 12.
I20260709 12:23:41.611650 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 19 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:23:41.594645 24523 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.635s	user 0.000s	sys 0.013s
I20260709 12:23:41.622740 17684 raft_consensus.cc:3060] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Advancing to term 16
W20260709 12:23:41.622937 17685 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.722s	user 0.000s	sys 0.013s
W20260709 12:23:41.640218 18333 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.726s	user 0.003s	sys 0.064s
I20260709 12:23:41.628358 17760 raft_consensus.cc:2468] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 16.
I20260709 12:23:41.596266 24418 raft_consensus.cc:493] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:41.597757 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 19 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:41.646356 17763 raft_consensus.cc:3060] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Advancing to term 19
W20260709 12:23:41.566022 24428 consensus_meta.cc:287] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.658s	user 0.013s	sys 0.002s
I20260709 12:23:41.662220 17681 raft_consensus.cc:3060] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:41.667735 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:41.669101 18368 raft_consensus.cc:2368] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for earlier term 11. Current term is 12.
I20260709 12:23:41.669413 17686 raft_consensus.cc:2468] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 12.
I20260709 12:23:41.671015 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
W20260709 12:23:41.640525 18333 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.726s	user 0.003s	sys 0.064s
I20260709 12:23:41.673180 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 16 candidate_status { last_received { term: 14 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
W20260709 12:23:41.673444 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 819068us
I20260709 12:23:41.641260 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:41.674331 18333 leader_election.cc:304] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:41.638760 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:41.644727 17757 raft_consensus.cc:2393] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 20: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:23:41.594141 24501 raft_consensus.cc:515] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:41.680068 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
W20260709 12:23:41.647794 17728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:41.652491 17756 raft_consensus.cc:1240] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 22. Current term is 23. Ops: []
I20260709 12:23:41.653429 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:41.654168 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:41.621869 23838 consensus_queue.cc:1059] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 14.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.033s
I20260709 12:23:41.596490 24441 consensus_queue.cc:237] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.13, Last appended by leader: 13, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:41.615268 24531 raft_consensus.cc:515] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:41.633424 24539 raft_consensus.cc:515] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:41.658622 17684 raft_consensus.cc:2468] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 16.
I20260709 12:23:41.659005 24250 leader_election.cc:290] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:41.637534 24515 raft_consensus.cc:493] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:41.631955 24203 raft_consensus.cc:2749] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:23:41.674379 17764 raft_consensus.cc:2468] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 16.
I20260709 12:23:41.641894 17760 raft_consensus.cc:1275] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 12 index: 9. Preceding OpId from leader: term: 14 index: 10. (index mismatch)
I20260709 12:23:41.676703 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 19 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:41.694557 17762 raft_consensus.cc:2393] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 19: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:23:41.682797 18334 leader_election.cc:304] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:23:41.719936 18368 raft_consensus.cc:2393] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 16: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:23:41.678881 24485 raft_consensus.cc:493] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:41.701749 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:41.563855 24486 raft_consensus.cc:515] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:41.727169 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:41.612491 24458 leader_election.cc:290] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 19 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:41.713667 17681 raft_consensus.cc:2468] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 19.
I20260709 12:23:41.714015 17763 raft_consensus.cc:2410] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for term 19 because replica has last-logged OpId of term: 18 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 11.
I20260709 12:23:41.635124 24098 raft_consensus.cc:515] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:41.746524 24098 leader_election.cc:290] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:23:41.746838 17801 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:41.747534 17759 raft_consensus.cc:1240] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 18. Current term is 19. Ops: []
W20260709 12:23:41.723064 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 48845.1us
I20260709 12:23:41.714550 24515 raft_consensus.cc:3060] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Advancing to term 21
W20260709 12:23:41.725699 24313 consensus_meta.cc:287] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.548s	user 0.010s	sys 0.001s
I20260709 12:23:41.722349 24482 raft_consensus.cc:3055] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:23:41.750622 24203 raft_consensus.cc:2749] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 12:23:41.766153 17687 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44898 (request call id 1277) took 1244 ms. Trace:
I20260709 12:23:41.680341 24418 raft_consensus.cc:515] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
W20260709 12:23:41.763965 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 470504us
W20260709 12:23:41.709045 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 452828us
I20260709 12:23:41.765381 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:41.679996 24428 raft_consensus.cc:515] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:41.724861 24362 raft_consensus.cc:2804] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:23:41.769578 17803 leader_election.cc:304] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:41.780609 17686 raft_consensus.cc:3060] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:41.791471 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:41.784083 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:41.784083 17729 leader_election.cc:400] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 15 pre-election: Vote denied by peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577) with higher term. Message: Invalid argument: T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for earlier term 15. Current term is 16.
I20260709 12:23:41.784757 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 14 candidate_status { last_received { term: 13 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:41.786232 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:41.793330 17729 leader_election.cc:304] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:41.792513 17837 raft_consensus.cc:2468] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 17.
I20260709 12:23:41.797286 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:41.798209 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:41.799216 17762 raft_consensus.cc:1240] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 16. Current term is 17. Ops: []
I20260709 12:23:41.768990 17687 rpcz_store.cc:276] 0709 12:23:40.522200 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:40.834586 (+312386us) service_pool.cc:224] Handling call
0709 12:23:40.834867 (+   281us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:40.836333 (+  1466us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 12:23:40.836782 (+   449us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:40.878355 (+ 41573us) log.cc:844] Serialized 26 byte log entry
0709 12:23:41.595263 (+716908us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:23:41.595302 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:41.595367 (+    65us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:41.765824 (+170457us) raft_consensus.cc:1685] finished
0709 12:23:41.765857 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:41.766069 (+   212us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1248,"raft.queue_time_us":4857,"raft.run_cpu_time_us":90,"raft.run_wall_time_us":91,"thread_start_us":716495,"threads_started":1,"wal-append.queue_time_us":782134}
I20260709 12:23:41.800575 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:41.801437 17687 raft_consensus.cc:2393] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 16: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:23:41.802299 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:41.806529 17727 leader_election.cc:304] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:41.749779 24485 raft_consensus.cc:515] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:41.808156 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:23:41.770356 24538 log.cc:927] Time spent T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.065s	user 0.000s	sys 0.001s
I20260709 12:23:41.816890 17833 raft_consensus.cc:2468] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 16.
W20260709 12:23:41.824425 17831 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35042 (request call id 2674) took 2377 ms (client timeout 2317 ms). Trace:
I20260709 12:23:41.767930 24458 raft_consensus.cc:493] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:41.847924 24458 raft_consensus.cc:515] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:41.848712 24510 raft_consensus.cc:3055] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:23:41.849880 24510 raft_consensus.cc:740] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
W20260709 12:23:41.827512 17831 rpcz_store.cc:269] 0709 12:23:39.447165 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:39.447429 (+   264us) service_pool.cc:224] Handling call
0709 12:23:41.824178 (+2376749us) spinlock_profiling.cc:243] Waited 724 ms on lock 0x7b54001586d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:23:41.824336 (+   158us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1664388480}
I20260709 12:23:41.825060 17688 raft_consensus.cc:1240] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 16. Current term is 17. Ops: []
I20260709 12:23:41.828470 18333 leader_election.cc:304] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:23:41.828770 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:41.772863 24418 leader_election.cc:290] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:41.768013 24482 raft_consensus.cc:740] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:23:41.769678 24501 leader_election.cc:290] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 16 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:41.838577 17832 raft_consensus.cc:2410] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for term 19 because replica has last-logged OpId of term: 18 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 11.
I20260709 12:23:41.802423 24430 consensus_queue.cc:1048] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.037s
I20260709 12:23:41.841079 24428 leader_election.cc:290] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 17 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:41.796353 24484 raft_consensus.cc:2749] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:23:41.823770 24551 raft_consensus.cc:2749] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:23:41.824083 24362 raft_consensus.cc:697] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 12 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:23:41.850342 24458 leader_election.cc:290] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:41.802831 24250 raft_consensus.cc:493] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:23:41.866631 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:23:41.876019 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 82760.6us
I20260709 12:23:41.873469 24486 leader_election.cc:290] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:23:41.873106 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 63764.7us
I20260709 12:23:41.913519 17764 raft_consensus.cc:1275] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 19 index: 14. Preceding OpId from leader: term: 21 index: 15. (index mismatch)
I20260709 12:23:41.914618 17727 leader_election.cc:304] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:41.916790 18371 raft_consensus.cc:1240] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 15 LEADER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 14. Current term is 15. Ops: [14.12-14.12]
I20260709 12:23:41.918406 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:41.919180 18367 raft_consensus.cc:3060] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:41.924659 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:41.870573 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 21 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:41.926255 17837 raft_consensus.cc:2468] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 20.
I20260709 12:23:41.882217 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:41.896083 24433 raft_consensus.cc:493] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:23:41.894876 17838 raft_consensus.cc:2468] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 16.
I20260709 12:23:41.896549 17834 raft_consensus.cc:3060] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:41.898777 24559 raft_consensus.cc:493] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:41.899030 24483 raft_consensus.cc:493] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:41.901676 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:41.902107 17686 raft_consensus.cc:2468] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 19.
W20260709 12:23:41.904379 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 76057.8us
I20260709 12:23:41.937017 18333 leader_election.cc:304] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:41.937134 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:41.851246 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 17 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:41.808075 24401 consensus_queue.cc:1059] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 17.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.099s
I20260709 12:23:41.927927 17763 raft_consensus.cc:2393] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 16: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:23:41.888778 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:41.898612 24476 raft_consensus.cc:2749] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:23:41.906424 17689 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37396 (request call id 2735) took 1605 ms. Trace:
I20260709 12:23:41.945403 18333 leader_election.cc:304] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:41.947917 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:41.948904 18365 raft_consensus.cc:3060] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:41.941327 17689 rpcz_store.cc:276] 0709 12:23:40.301052 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:40.746679 (+445627us) service_pool.cc:224] Handling call
0709 12:23:40.747071 (+   392us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:40.747487 (+   416us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 12:23:41.454701 (+707214us) log.cc:844] Serialized 14 byte log entry
0709 12:23:41.905962 (+451261us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:41.906035 (+    73us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:23:41.906098 (+    63us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:41.906193 (+    95us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:41.906349 (+   156us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":125583,"prepare.queue_time_us":246370,"prepare.run_cpu_time_us":442,"prepare.run_wall_time_us":3965,"raft.queue_time_us":269,"raft.run_cpu_time_us":104,"raft.run_wall_time_us":19138,"thread_start_us":1031841,"threads_started":2}
I20260709 12:23:41.951508 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:41.955580 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:41.961792 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:41.963099 18367 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 16.
I20260709 12:23:41.966413 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:41.968163 18367 raft_consensus.cc:2393] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 16: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:23:41.981791 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:41.984678 18362 raft_consensus.cc:2468] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 15.
I20260709 12:23:41.986166 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:41.965607 17683 raft_consensus.cc:2468] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 19.
I20260709 12:23:41.897259 24037 consensus_queue.cc:1048] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.037s
I20260709 12:23:42.005151 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:42.040522 18366 raft_consensus.cc:1240] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 13. Current term is 14. Ops: []
I20260709 12:23:41.866132 24510 consensus_queue.cc:260] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 18.15, Last appended by leader: 15, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:41.903479 24418 consensus_queue.cc:1048] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.056s
I20260709 12:23:41.941731 24169 raft_consensus.cc:2749] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:23:42.053752 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:42.055204 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:42.004190 24458 raft_consensus.cc:2749] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
W20260709 12:23:42.033481 24535 log.cc:927] Time spent T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.123s	user 0.000s	sys 0.001s
I20260709 12:23:42.055356 18365 raft_consensus.cc:2468] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 19.
I20260709 12:23:42.067519 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:42.067812 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:42.090162 17686 raft_consensus.cc:2468] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 19.
I20260709 12:23:42.091500 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 21 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:42.092937 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
W20260709 12:23:42.070319 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 170721us
I20260709 12:23:42.072757 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:42.059582 17834 raft_consensus.cc:2468] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 12.
I20260709 12:23:42.085119 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:42.086021 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:42.096444 18362 raft_consensus.cc:2468] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 16.
I20260709 12:23:41.962579 24429 raft_consensus.cc:493] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:23:42.102638 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:42.039070 24531 leader_election.cc:290] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:41.927564 24250 raft_consensus.cc:515] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:41.951507 24337 raft_consensus.cc:2749] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:23:42.097590 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:42.043838 24037 consensus_queue.cc:1243] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer e063541542a94969898868bf6eb5ce1b log is divergent from this leader: its last log entry 18.11 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 10
I20260709 12:23:41.959326 24559 raft_consensus.cc:515] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:42.031994 24501 raft_consensus.cc:2804] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:23:41.950730 24483 raft_consensus.cc:515] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:42.111657 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:42.121481 18364 raft_consensus.cc:2468] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 13.
I20260709 12:23:42.121999 18371 raft_consensus.cc:2393] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 13: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:23:42.122722 18366 raft_consensus.cc:2393] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 19: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:23:42.071048 23931 raft_consensus.cc:493] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:42.115682 23906 consensus_meta.cc:287] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.979s	user 0.013s	sys 0.001s
I20260709 12:23:42.103467 18365 raft_consensus.cc:2393] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 21: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:23:42.116921 18362 raft_consensus.cc:2393] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 16: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:23:42.103637 17764 raft_consensus.cc:1240] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 20. Current term is 21. Ops: []
I20260709 12:23:42.128252 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:42.103819 23997 raft_consensus.cc:493] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:42.105832 24517 raft_consensus.cc:493] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:23:41.951622 24433 raft_consensus.cc:515] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:42.139060 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
W20260709 12:23:42.121095 24546 log.cc:927] Time spent T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.344s	user 0.000s	sys 0.001s
I20260709 12:23:42.139978 18364 raft_consensus.cc:2368] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 17. Current term is 18.
I20260709 12:23:42.106575 24551 raft_consensus.cc:493] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:23:42.127350 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:42.088936 24539 leader_election.cc:290] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:42.111557 24510 raft_consensus.cc:3060] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:42.128830 17762 raft_consensus.cc:2468] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 16.
I20260709 12:23:42.132360 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 21 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:42.155390 18362 raft_consensus.cc:2410] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 21 because replica has last-logged OpId of term: 18 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 12.
I20260709 12:23:42.133549 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:42.156133 18365 raft_consensus.cc:2376] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 17. Re-sending same reply.
I20260709 12:23:42.003382 24482 consensus_queue.cc:260] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 14.14, Last appended by leader: 14, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:42.148746 17803 leader_election.cc:304] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:23:42.162238 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:42.088601 24362 consensus_queue.cc:237] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:42.108764 24429 raft_consensus.cc:515] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:42.122889 24558 consensus_queue.cc:1048] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.043s
W20260709 12:23:42.222471 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 48998.5us
I20260709 12:23:42.189041 17835 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35040 (request call id 2969) took 1300 ms. Trace:
I20260709 12:23:42.214120 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:42.165000 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 14 candidate_status { last_received { term: 10 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:42.166952 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
W20260709 12:23:42.316846 17728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:42.329560 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:42.322870 17652 leader_election.cc:304] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:23:42.196606 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 20 candidate_status { last_received { term: 18 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:42.333290 17756 raft_consensus.cc:2468] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 19.
I20260709 12:23:42.331900 17757 raft_consensus.cc:2468] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 19.
I20260709 12:23:42.335232 17652 leader_election.cc:304] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:23:42.227980 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 129190us
I20260709 12:23:42.225638 17803 leader_election.cc:304] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:23:42.224296 17835 rpcz_store.cc:276] 0709 12:23:40.889734 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:40.890361 (+   627us) service_pool.cc:224] Handling call
0709 12:23:40.948295 (+ 57934us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:23:40.949408 (+  1113us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:40.949441 (+    33us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:23:40.971504 (+ 22063us) log.cc:844] Serialized 52 byte log entry
0709 12:23:41.553083 (+581579us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:23:41.904971 (+351888us) log.cc:844] Serialized 14 byte log entry
0709 12:23:42.055639 (+150668us) log.cc:844] Serialized 14 byte log entry
0709 12:23:42.097435 (+ 41796us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:42.097527 (+    92us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:42.188790 (+ 91263us) raft_consensus.cc:1685] finished
0709 12:23:42.188811 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:42.188970 (+   159us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":82040,"prepare.queue_time_us":578133,"prepare.run_cpu_time_us":58,"prepare.run_wall_time_us":58,"raft.queue_time_us":138017,"raft.run_cpu_time_us":338,"raft.run_wall_time_us":37888,"thread_start_us":1000698,"threads_started":3,"wal-append.queue_time_us":704057}
I20260709 12:23:42.191449 18333 leader_election.cc:304] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:42.392144 24518 raft_consensus.cc:493] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
W20260709 12:23:42.367338 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:42.227299 18364 raft_consensus.cc:2410] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 14 because replica has last-logged OpId of term: 13 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 12.
I20260709 12:23:42.393035 24392 leader_election.cc:290] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:42.163038 18362 raft_consensus.cc:2468] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 17.
I20260709 12:23:42.188858 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:42.139379 23906 raft_consensus.cc:2749] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Leader pre-election lost for term 15. Reason: Vote denied by peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071) with higher term. Message: Invalid argument: T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for earlier term 15. Current term is 16.
I20260709 12:23:42.130774 23931 raft_consensus.cc:515] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:42.403378 24428 raft_consensus.cc:493] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:23:42.404670 24428 raft_consensus.cc:515] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:42.406539 18365 raft_consensus.cc:1240] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 16. Current term is 17. Ops: []
I20260709 12:23:42.406926 23931 leader_election.cc:290] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:42.138959 23997 raft_consensus.cc:515] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:42.162598 24250 leader_election.cc:290] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:42.242256 24433 leader_election.cc:290] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:42.191116 24485 leader_election.cc:290] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:42.412189 24401 raft_consensus.cc:2804] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Leader pre-election won for term 17
W20260709 12:23:42.255079 24493 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.868s	user 0.005s	sys 0.006s
I20260709 12:23:42.248347 24515 raft_consensus.cc:515] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:42.150909 24517 raft_consensus.cc:515] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:42.263345 24513 log.cc:927] Time spent T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 1.856s	user 0.004s	sys 0.000s
W20260709 12:23:42.255649 24488 log.cc:927] Time spent T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 1.984s	user 0.005s	sys 0.001s
I20260709 12:23:42.163214 24486 raft_consensus.cc:2804] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:23:42.418725 23997 leader_election.cc:290] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:23:42.243150 24569 log.cc:927] Time spent T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.052s	user 0.000s	sys 0.000s
I20260709 12:23:42.422938 24517 leader_election.cc:290] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:42.423130 24515 leader_election.cc:290] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 21 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:23:42.153604 24553 log.cc:927] Time spent T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.337s	user 0.001s	sys 0.001s
I20260709 12:23:42.253623 24559 leader_election.cc:290] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:23:42.187873 24560 log.cc:927] Time spent T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.117s	user 0.001s	sys 0.000s
I20260709 12:23:42.424985 24087 raft_consensus.cc:2749] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:23:42.391637 24581 consensus_queue.cc:1059] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 16.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.034s
W20260709 12:23:42.262811 24566 log.cc:927] Time spent T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.086s	user 0.001s	sys 0.001s
I20260709 12:23:42.427381 24581 raft_consensus.cc:3055] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 16 LEADER]: Stepping down as leader of term 16
W20260709 12:23:42.187359 24562 log.cc:927] Time spent T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.089s	user 0.000s	sys 0.001s
W20260709 12:23:42.243160 24573 log.cc:927] Time spent T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.076s	user 0.000s	sys 0.000s
W20260709 12:23:42.187120 24563 log.cc:927] Time spent T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.064s	user 0.000s	sys 0.000s
W20260709 12:23:42.243804 24575 log.cc:927] Time spent T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.070s	user 0.000s	sys 0.000s
W20260709 12:23:42.186380 24570 log.cc:927] Time spent T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.078s	user 0.000s	sys 0.000s
I20260709 12:23:42.237869 24579 consensus_queue.cc:1059] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 13.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.017s
I20260709 12:23:42.396438 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 20 candidate_status { last_received { term: 18 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:42.443651 24579 raft_consensus.cc:3055] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:23:42.444268 24579 raft_consensus.cc:740] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 13 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:23:42.396481 24244 raft_consensus.cc:2804] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:23:42.445214 24244 raft_consensus.cc:493] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:23:42.445837 24244 raft_consensus.cc:3060] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:42.446045 24579 consensus_queue.cc:260] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.10, Last appended by leader: 10, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:42.447090 17682 raft_consensus.cc:1240] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 18. Current term is 19. Ops: [18.13-18.13]
I20260709 12:23:42.407657 24428 leader_election.cc:290] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:23:42.411768 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 186029us
I20260709 12:23:42.413468 24401 raft_consensus.cc:493] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:42.393640 24518 raft_consensus.cc:515] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:42.393780 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 202271us
I20260709 12:23:42.163707 24482 raft_consensus.cc:3060] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:42.156069 24551 raft_consensus.cc:515] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:23:42.415222 24493 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 2.070s	user 0.005s	sys 0.007s
I20260709 12:23:42.418751 18362 raft_consensus.cc:1275] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 19 index: 14. Preceding OpId from leader: term: 21 index: 15. (index mismatch)
I20260709 12:23:42.427956 18363 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 1992) took 2227 ms. Trace:
W20260709 12:23:42.188400 24561 log.cc:927] Time spent T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.132s	user 0.000s	sys 0.000s
I20260709 12:23:42.459329 18363 rpcz_store.cc:276] 0709 12:23:40.200782 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:40.203739 (+  2957us) service_pool.cc:224] Handling call
0709 12:23:40.204186 (+   447us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:40.240177 (+ 35991us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:40.240227 (+    50us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:40.249838 (+  9611us) log.cc:844] Serialized 26 byte log entry
0709 12:23:41.125287 (+875449us) spinlock_profiling.cc:243] Waited 1.07 ms on lock 0x7b1800418d80. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b47515e07 00007f7b503709a9 00007f7b4753269b 00007f7b4753093f 00007f7b47532c02 00007f7b4a1ba20b 00007f7b4a1bfd23 00007f7b4a1c00d1 00007f7b4a5dba0f
0709 12:23:41.635561 (+510274us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:23:41.635634 (+    73us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:41.635726 (+    92us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:42.427137 (+791411us) raft_consensus.cc:1685] finished
0709 12:23:42.427188 (+    51us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:42.427684 (+   496us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":35431,"raft.queue_time_us":36547,"raft.run_cpu_time_us":205,"raft.run_wall_time_us":21502,"spinlock_wait_cycles":2465664,"thread_start_us":510024,"threads_started":1,"wal-append.queue_time_us":1496530}
I20260709 12:23:42.419946 24486 raft_consensus.cc:493] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:42.462427 24486 raft_consensus.cc:3060] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:42.462935 24551 leader_election.cc:290] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:42.464730 18333 leader_election.cc:304] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:23:42.429770 24362 raft_consensus.cc:493] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:42.409412 24392 raft_consensus.cc:2749] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:23:42.466404 18333 leader_election.cc:304] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:42.466708 24392 raft_consensus.cc:2804] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:23:42.467211 24392 raft_consensus.cc:493] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:23:42.450551 24401 raft_consensus.cc:3060] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:42.467760 24527 raft_consensus.cc:2749] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 12:23:42.467800 24392 raft_consensus.cc:3060] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:42.450959 24579 raft_consensus.cc:3060] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:42.434974 24524 consensus_queue.cc:237] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.10, Last appended by leader: 9, Current term: 18, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:42.458159 24518 leader_election.cc:290] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:42.466264 24362 raft_consensus.cc:515] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:42.449239 17803 leader_election.cc:304] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:23:42.478988 17758 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38786 (request call id 3397) took 2589 ms. Trace:
I20260709 12:23:42.479364 24362 leader_election.cc:290] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:42.481164 17803 leader_election.cc:400] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 17 election: Vote denied by peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577) with higher term. Message: Invalid argument: T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 17. Current term is 18.
I20260709 12:23:42.481457 17763 raft_consensus.cc:2468] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 19.
I20260709 12:23:42.479691 17758 rpcz_store.cc:276] 0709 12:23:39.889097 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:39.889309 (+   212us) service_pool.cc:224] Handling call
0709 12:23:39.889747 (+   438us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:39.929992 (+ 40245us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:39.930138 (+   146us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:39.938053 (+  7915us) log.cc:844] Serialized 26 byte log entry
0709 12:23:40.409986 (+471933us) spinlock_profiling.cc:243] Waited 2.96 ms on lock 0x7b180048b920. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b47515e07 00007f7b503709a9 00007f7b4753269b 00007f7b4753093f 00007f7b47532c02 00007f7b4a1ba20b 00007f7b4a1bfd23 00007f7b4a1c00d1 00007f7b4a5dba0f
0709 12:23:40.604834 (+194848us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:23:40.604874 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:42.478149 (+1873275us) spinlock_profiling.cc:243] Waited 39.7 ms on lock 0x7b540019f0d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:23:42.478273 (+   124us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:42.478391 (+   118us) raft_consensus.cc:1685] finished
0709 12:23:42.478410 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:42.478740 (+   330us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":54286,"raft.run_cpu_time_us":144,"raft.run_wall_time_us":21933,"spinlock_wait_cycles":98234368,"thread_start_us":194633,"threads_started":1,"wal-append.queue_time_us":761403,"wal-append.run_cpu_time_us":2544,"wal-append.run_wall_time_us":1004397}
I20260709 12:23:42.449754 24581 raft_consensus.cc:740] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 16 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:23:42.481823 24485 raft_consensus.cc:2804] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:23:42.488586 24485 raft_consensus.cc:493] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:23:42.489209 24485 raft_consensus.cc:3060] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:42.493214 24581 consensus_queue.cc:260] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.10, Last appended by leader: 10, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:42.495395 24581 raft_consensus.cc:3060] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:42.483897 17802 leader_election.cc:304] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:42.485818 18365 raft_consensus.cc:1275] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 13 index: 11. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
W20260709 12:23:42.483072 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 148231us
I20260709 12:23:42.500384 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 19 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:42.503153 24098 raft_consensus.cc:3055] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:23:42.503665 24098 raft_consensus.cc:740] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 19 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:23:42.505287 24244 raft_consensus.cc:515] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:42.507038 24098 consensus_queue.cc:260] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 19.12, Last appended by leader: 12, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:42.508827 18333 leader_election.cc:304] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:42.509193 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:42.510748 18333 leader_election.cc:304] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:23:42.511399 24429 leader_election.cc:290] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:42.510819 24151 raft_consensus.cc:2749] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 12:23:42.521757 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:42.513175 24486 raft_consensus.cc:515] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:42.514389 24482 raft_consensus.cc:2804] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:23:42.527061 24482 raft_consensus.cc:493] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:23:42.527690 24482 raft_consensus.cc:3060] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:42.528313 24486 leader_election.cc:290] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 16 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:42.512069 18333 leader_election.cc:304] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:42.512936 24418 raft_consensus.cc:2749] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:23:42.485704 24401 raft_consensus.cc:515] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:42.523299 17759 raft_consensus.cc:2468] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 15.
I20260709 12:23:42.513774 17652 leader_election.cc:304] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:42.530455 24244 leader_election.cc:290] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 13 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:42.537106 24394 raft_consensus.cc:2749] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:23:42.537807 24507 raft_consensus.cc:3055] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:23:42.509176 24098 raft_consensus.cc:3060] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:42.541816 24443 raft_consensus.cc:2749] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:23:42.533419 24501 raft_consensus.cc:493] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:42.533423 24401 leader_election.cc:290] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 17 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:42.546121 24500 raft_consensus.cc:3055] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:23:42.546574 24500 raft_consensus.cc:740] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:23:42.541863 24518 raft_consensus.cc:493] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:42.551496 24518 raft_consensus.cc:515] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:42.553254 24501 raft_consensus.cc:3060] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:23:42.553725 24583 log.cc:927] Time spent T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.320s	user 0.002s	sys 0.000s
I20260709 12:23:42.557829 24518 leader_election.cc:290] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:42.540946 24507 raft_consensus.cc:740] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 14 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:23:42.560479 24507 consensus_queue.cc:260] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.12, Last appended by leader: 12, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:42.561635 17761 raft_consensus.cc:2468] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 16.
I20260709 12:23:42.561832 24507 raft_consensus.cc:3060] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:42.562404 24500 consensus_queue.cc:260] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 15.13, Last appended by leader: 13, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:42.563736 24500 raft_consensus.cc:3060] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:42.564348 24485 raft_consensus.cc:515] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:42.584491 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 70909.1us
I20260709 12:23:42.586248 24518 consensus_queue.cc:1059] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 22.17, Next index: 18, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:23:42.587530 24518 raft_consensus.cc:3055] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:23:42.588016 24518 raft_consensus.cc:740] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:23:42.589193 24518 consensus_queue.cc:260] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 22.17, Last appended by leader: 17, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:42.587695 17688 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44898 (request call id 1285) took 1006 ms. Trace:
I20260709 12:23:42.586869 17652 leader_election.cc:304] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:23:42.597368 17832 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35040 (request call id 2962) took 1813 ms (client timeout 1918 ms). Trace:
W20260709 12:23:42.597950 17832 rpcz_store.cc:269] 0709 12:23:40.783901 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:40.791688 (+  7787us) service_pool.cc:224] Handling call
0709 12:23:42.596810 (+1805122us) spinlock_profiling.cc:243] Waited 957 ms on lock 0x7b54002ebb54. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:23:42.597118 (+   308us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2201834368}
I20260709 12:23:42.591723 24518 raft_consensus.cc:3060] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:23:42.592267 17688 rpcz_store.cc:276] 0709 12:23:41.581396 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:41.586543 (+  5147us) service_pool.cc:224] Handling call
0709 12:23:41.587562 (+  1019us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:41.877974 (+290412us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:42.587132 (+709158us) spinlock_profiling.cc:243] Waited 237 ms on lock 0x7b54001aa254. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:23:42.587484 (+   352us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":545729920}
I20260709 12:23:42.593643 24392 raft_consensus.cc:515] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:42.612298 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 170035us
I20260709 12:23:42.620322 24485 leader_election.cc:290] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 14 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:42.620553 24475 leader_election.cc:290] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 19 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:42.623859 24244 raft_consensus.cc:2804] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:23:42.627717 18333 leader_election.cc:304] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:23:42.623997 24443 consensus_queue.cc:1048] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.006s
I20260709 12:23:42.625145 24244 raft_consensus.cc:493] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:42.634781 17802 leader_election.cc:304] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:23:42.637799 24436 leader_election.cc:290] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:42.638412 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 16 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:42.639544 17758 raft_consensus.cc:2393] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 16: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:23:42.637928 17759 raft_consensus.cc:1240] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 18. Current term is 19. Ops: [18.13-18.13]
I20260709 12:23:42.626009 24392 leader_election.cc:290] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 18 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:42.648504 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 21 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:42.650395 17758 raft_consensus.cc:2410] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 21 because replica has last-logged OpId of term: 19 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 13.
I20260709 12:23:42.650157 24434 raft_consensus.cc:493] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:42.652930 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:42.654433 24394 raft_consensus.cc:2804] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:23:42.654625 17759 raft_consensus.cc:2393] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 16: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:23:42.655022 24394 raft_consensus.cc:493] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:42.641784 24244 raft_consensus.cc:3060] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:42.655151 18333 leader_election.cc:304] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:23:42.666466 24485 raft_consensus.cc:2804] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:23:42.666695 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:42.661751 24394 raft_consensus.cc:3060] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:42.663620 17802 leader_election.cc:304] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:23:42.672029 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:42.667109 24485 raft_consensus.cc:493] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:42.673666 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:42.673629 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:42.663997 23838 raft_consensus.cc:493] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:42.679126 17755 raft_consensus.cc:3060] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Advancing to term 17
W20260709 12:23:42.679317 17804 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:42.679337 24484 raft_consensus.cc:2804] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:23:42.685151 24484 raft_consensus.cc:493] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:42.686364 24484 raft_consensus.cc:3060] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:42.690722 17685 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37396 (request call id 2736) took 2389 ms. Trace:
W20260709 12:23:42.695771 17730 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:42.696650 17730 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:42.695801 24571 log.cc:927] Time spent T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.576s	user 0.000s	sys 0.004s
I20260709 12:23:42.697986 17654 leader_election.cc:400] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 11 election: Vote denied by peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577) with higher term. Message: Invalid argument: T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for earlier term 11. Current term is 12.
I20260709 12:23:42.698315 17654 leader_election.cc:403] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 11 election: Cancelling election due to peer responding with higher term
I20260709 12:23:42.676715 24485 raft_consensus.cc:3060] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:42.704224 18333 leader_election.cc:304] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
W20260709 12:23:42.707605 17801 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:42.710421 17755 raft_consensus.cc:2468] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 17.
I20260709 12:23:42.679186 23838 raft_consensus.cc:515] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:42.710431 24443 raft_consensus.cc:493] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:23:42.712738 24443 raft_consensus.cc:515] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:23:42.714963 24461 raft_consensus.cc:670] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da: failed to trigger leader election: Illegal state: only voting members can start elections: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } [suppressed 4 similar messages]
I20260709 12:23:42.716111 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:42.717170 17830 raft_consensus.cc:1275] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 12 index: 9. Preceding OpId from leader: term: 14 index: 10. (index mismatch)
I20260709 12:23:42.717290 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:42.717667 23838 leader_election.cc:290] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:42.718566 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:42.718698 23740 raft_consensus.cc:493] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:42.716216 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 16 candidate_status { last_received { term: 14 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:42.719580 23740 raft_consensus.cc:515] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:42.671324 24434 raft_consensus.cc:515] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:42.722151 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:42.729044 24098 raft_consensus.cc:493] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:42.724498 24443 leader_election.cc:290] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:42.726022 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:42.732009 17829 raft_consensus.cc:3060] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:42.721302 24392 raft_consensus.cc:2804] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:23:42.719745 24486 raft_consensus.cc:493] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:23:42.740221 24486 raft_consensus.cc:515] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:42.724656 24436 raft_consensus.cc:2749] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Leader election lost for term 11. Reason: Vote denied by peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577) with higher term. Message: Invalid argument: T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for earlier term 11. Current term is 12.
W20260709 12:23:42.753580 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:42.751317 17652 leader_election.cc:304] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:23:42.757252 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 46133.1us
I20260709 12:23:42.735190 24098 raft_consensus.cc:515] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:42.727768 24482 raft_consensus.cc:515] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:42.691210 17685 rpcz_store.cc:276] 0709 12:23:40.301293 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:40.804258 (+502965us) service_pool.cc:224] Handling call
0709 12:23:40.805386 (+  1128us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:40.864175 (+ 58789us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 12:23:41.640306 (+776131us) log.cc:844] Serialized 14 byte log entry
0709 12:23:41.960551 (+320245us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:41.960620 (+    69us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:23:41.960688 (+    68us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:42.690157 (+729469us) spinlock_profiling.cc:243] Waited 58.5 ms on lock 0x7b54003507d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:23:42.690266 (+   109us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:42.690512 (+   246us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":16871,"prepare.queue_time_us":169185,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":46,"raft.queue_time_us":94,"raft.run_cpu_time_us":130,"raft.run_wall_time_us":14423,"spinlock_wait_cycles":134549376,"thread_start_us":1059347,"threads_started":2,"wal-append.queue_time_us":533370}
I20260709 12:23:42.756511 24486 leader_election.cc:290] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:42.755285 24392 raft_consensus.cc:697] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 16 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
W20260709 12:23:42.763434 24599 log.cc:927] Time spent T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.053s	user 0.001s	sys 0.000s
I20260709 12:23:42.758787 24500 raft_consensus.cc:2749] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:23:42.768700 17758 raft_consensus.cc:1240] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 15. Current term is 16. Ops: []
I20260709 12:23:42.770323 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:42.768782 24087 raft_consensus.cc:2749] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:23:42.773985 17759 pending_rounds.cc:85] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb: Aborting all ops after (but not including) 11
I20260709 12:23:42.779361 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:42.778805 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:42.778220 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 21 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:42.776274 17759 pending_rounds.cc:107] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb: Aborting uncommitted NO_OP operation due to leader change: 14.12
I20260709 12:23:42.778120 17801 leader_election.cc:304] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:42.782717 17759 raft_consensus.cc:2889] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:23:42.787844 17728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:42.785223 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:42.792016 17755 raft_consensus.cc:2368] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 16. Current term is 17.
I20260709 12:23:42.792586 23740 leader_election.cc:290] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:23:42.793345 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 49082.4us
I20260709 12:23:42.795001 17652 leader_election.cc:304] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:42.796756 24227 raft_consensus.cc:2804] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:23:42.797202 24227 raft_consensus.cc:493] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:23:42.797652 24227 raft_consensus.cc:3060] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:42.797737 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:42.798518 17835 raft_consensus.cc:2468] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 17.
I20260709 12:23:42.802057 24431 raft_consensus.cc:3060] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:42.807721 18334 leader_election.cc:304] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:23:42.808981 24098 leader_election.cc:290] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:42.809090 24500 raft_consensus.cc:2804] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:23:42.809950 24500 raft_consensus.cc:697] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 16 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:23:42.809993 17833 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37994 (request call id 1297) took 1016 ms. Trace:
I20260709 12:23:42.810863 17834 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37994 (request call id 1287) took 1025 ms. Trace:
I20260709 12:23:42.811340 18371 raft_consensus.cc:1275] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
I20260709 12:23:42.811558 17834 rpcz_store.cc:276] 0709 12:23:41.785773 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:41.785963 (+   190us) service_pool.cc:224] Handling call
0709 12:23:42.810465 (+1024502us) spinlock_profiling.cc:243] Waited 103 ms on lock 0x7b54000ab9d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:23:42.810639 (+   174us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":237166976}
W20260709 12:23:42.812301 17761 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38776 (request call id 2674) took 1216 ms (client timeout 1513 ms). Trace:
W20260709 12:23:42.812726 17761 rpcz_store.cc:269] 0709 12:23:41.595967 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:41.598183 (+  2216us) service_pool.cc:224] Handling call
0709 12:23:42.812084 (+1213901us) spinlock_profiling.cc:243] Waited 772 ms on lock 0x7b54003e5dd4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:23:42.812226 (+   142us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1776391936}
I20260709 12:23:42.779492 24434 leader_election.cc:290] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:42.823607 17829 raft_consensus.cc:2468] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 16.
W20260709 12:23:42.827829 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 151328us
I20260709 12:23:42.810635 17833 rpcz_store.cc:276] 0709 12:23:41.793235 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:41.927824 (+134589us) service_pool.cc:224] Handling call
0709 12:23:42.003278 (+ 75454us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:42.226648 (+223370us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:42.226678 (+    30us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:42.226718 (+    40us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:23:42.226753 (+    35us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:42.809512 (+582759us) spinlock_profiling.cc:243] Waited 137 ms on lock 0x7b540022b854. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:23:42.809578 (+    66us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:42.809828 (+   250us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":37756,"raft.queue_time_us":41598,"raft.run_cpu_time_us":93,"raft.run_wall_time_us":93,"spinlock_wait_cycles":315428224}
I20260709 12:23:42.834049 17761 raft_consensus.cc:1275] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 14 index: 12. Preceding OpId from leader: term: 16 index: 13. (index mismatch)
I20260709 12:23:42.835191 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:42.836678 24483 leader_election.cc:290] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:42.837288 24087 raft_consensus.cc:493] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:23:42.839695 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:42.840044 24087 raft_consensus.cc:515] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:42.841503 24501 raft_consensus.cc:515] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:42.837769 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 59873.6us
I20260709 12:23:42.811290 24500 consensus_queue.cc:237] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.12, Last appended by leader: 12, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:42.842649 24087 leader_election.cc:290] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:42.843027 24439 raft_consensus.cc:3060] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:42.843325 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:42.856122 23740 consensus_queue.cc:1059] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 16.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:23:42.844381 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:42.844705 24482 leader_election.cc:290] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 17 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:42.845218 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:42.850812 24428 raft_consensus.cc:493] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:23:42.869037 17758 raft_consensus.cc:1240] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 18. Current term is 19. Ops: []
I20260709 12:23:42.848806 24098 raft_consensus.cc:2749] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:23:42.876684 17755 raft_consensus.cc:2393] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 23: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:23:42.881560 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 19 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:42.890445 24501 leader_election.cc:290] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 15 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:42.892383 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:42.894311 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:42.895037 24588 consensus_queue.cc:1048] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.020s
I20260709 12:23:42.894402 17729 leader_election.cc:304] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:42.900345 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:42.901386 17683 raft_consensus.cc:2393] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 12: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:23:42.901806 24430 consensus_queue.cc:1059] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 20.11, Next index: 12, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:23:42.908797 17801 leader_election.cc:304] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:42.909319 17729 leader_election.cc:304] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:42.910261 17801 leader_election.cc:304] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:42.911201 24579 raft_consensus.cc:2749] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:23:42.911985 24579 raft_consensus.cc:2749] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:23:42.912171 24471 raft_consensus.cc:2749] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:23:42.914163 24471 raft_consensus.cc:2804] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:23:42.914811 24471 raft_consensus.cc:697] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 19 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:23:42.908092 24392 consensus_queue.cc:237] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.12, Last appended by leader: 12, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:42.918284 18365 raft_consensus.cc:1240] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 20. Current term is 21. Ops: []
I20260709 12:23:42.881271 24428 raft_consensus.cc:515] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:42.918272 24484 raft_consensus.cc:515] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:42.921082 17831 raft_consensus.cc:2368] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for earlier term 19. Current term is 20.
I20260709 12:23:42.911774 17801 leader_election.cc:304] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:23:42.922115 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:42.922951 18371 raft_consensus.cc:1240] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 18 LEADER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 17. Current term is 18. Ops: []
W20260709 12:23:42.929319 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 49028.5us
I20260709 12:23:42.905036 24430 raft_consensus.cc:3055] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:23:42.920707 24244 raft_consensus.cc:515] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:42.944600 24588 raft_consensus.cc:2804] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:23:42.945220 24588 raft_consensus.cc:493] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:42.945712 24588 raft_consensus.cc:3060] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:42.950205 24483 raft_consensus.cc:3055] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:23:42.951035 24483 raft_consensus.cc:740] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 16 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
W20260709 12:23:42.954895 24601 log.cc:927] Time spent T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.091s	user 0.000s	sys 0.000s
I20260709 12:23:42.957398 18367 raft_consensus.cc:1275] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 16 index: 12. Preceding OpId from leader: term: 18 index: 13. (index mismatch)
I20260709 12:23:42.959236 18366 raft_consensus.cc:1240] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 16 LEADER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 15. Current term is 16. Ops: []
I20260709 12:23:42.960690 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:42.965029 24484 leader_election.cc:290] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 20 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:42.965271 17803 leader_election.cc:304] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:42.966759 24579 raft_consensus.cc:2749] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:23:42.966946 17803 leader_election.cc:304] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:42.967473 24579 raft_consensus.cc:493] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:42.967660 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:42.968724 18333 leader_election.cc:304] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:42.968935 24527 raft_consensus.cc:493] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:42.969549 24527 raft_consensus.cc:515] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:42.972225 24527 leader_election.cc:290] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:42.968205 24579 raft_consensus.cc:515] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:42.974104 17764 raft_consensus.cc:1275] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 19 index: 14. Preceding OpId from leader: term: 20 index: 14. (term mismatch)
I20260709 12:23:42.974504 17764 pending_rounds.cc:85] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb: Aborting all ops after (but not including) 13
I20260709 12:23:42.974623 24579 leader_election.cc:290] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:42.974802 17764 pending_rounds.cc:107] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb: Aborting uncommitted NO_OP operation due to leader change: 19.14
I20260709 12:23:42.975106 17764 raft_consensus.cc:2889] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:42.976305 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:23:42.976584 17728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:42.969298 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 18 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:42.920737 24428 leader_election.cc:290] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:42.938861 24430 raft_consensus.cc:740] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 20 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:23:42.990970 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:42.994570 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:42.990881 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:42.987313 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:43.000654 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 21 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:43.000771 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:42.986789 17835 raft_consensus.cc:2376] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 18. Re-sending same reply.
I20260709 12:23:43.002207 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 17 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:43.003176 17758 raft_consensus.cc:2410] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for term 17 because replica has last-logged OpId of term: 14 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 10.
I20260709 12:23:43.004292 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:43.004890 17833 raft_consensus.cc:2368] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for earlier term 19. Current term is 20.
I20260709 12:23:43.003924 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:43.006848 17757 raft_consensus.cc:2468] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 20.
I20260709 12:23:42.936816 24394 raft_consensus.cc:515] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:43.013404 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:42.934753 24227 raft_consensus.cc:515] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:43.011451 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 15 candidate_status { last_received { term: 14 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:43.011747 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:43.018069 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:43.018095 17829 raft_consensus.cc:1275] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 16 index: 12. Preceding OpId from leader: term: 20 index: 14. (index mismatch)
I20260709 12:23:43.018886 17836 raft_consensus.cc:2368] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for earlier term 16. Current term is 17.
I20260709 12:23:43.016289 18368 raft_consensus.cc:1275] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [term 18 LEARNER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 18 index: 10. Preceding OpId from leader: term: 18 index: 11. (index mismatch)
I20260709 12:23:43.023425 18366 raft_consensus.cc:1275] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 14 index: 10. Preceding OpId from leader: term: 15 index: 11. (index mismatch)
I20260709 12:23:43.027170 24483 consensus_queue.cc:260] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.10, Last appended by leader: 10, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:43.020350 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:43.028817 24483 raft_consensus.cc:3060] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:43.036798 18363 raft_consensus.cc:1240] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 18. Current term is 19. Ops: []
I20260709 12:23:43.039001 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:43.043416 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:43.046903 17688 raft_consensus.cc:2393] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 15: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:23:43.005946 23838 raft_consensus.cc:2749] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:23:43.056306 24244 leader_election.cc:290] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 17 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:42.990918 24098 raft_consensus.cc:2749] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:23:42.934900 24471 consensus_queue.cc:237] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.14, Last appended by leader: 14, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:43.061672 24430 consensus_queue.cc:260] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 20.11, Last appended by leader: 11, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:43.064054 24475 raft_consensus.cc:493] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:42.996362 24434 consensus_queue.cc:1059] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.069s
I20260709 12:23:43.064807 24475 raft_consensus.cc:515] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:43.067224 23906 raft_consensus.cc:3055] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:23:43.067797 23906 raft_consensus.cc:740] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 20 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:23:43.069022 23906 consensus_queue.cc:260] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 20.15, Last appended by leader: 15, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:42.999039 24518 raft_consensus.cc:493] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:23:43.070837 24518 raft_consensus.cc:515] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:23:43.020535 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:23:43.110108 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 143255us
I20260709 12:23:43.111081 18363 raft_consensus.cc:1240] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 12. Current term is 13. Ops: []
W20260709 12:23:43.120602 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:43.063696 24430 raft_consensus.cc:3060] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Advancing to term 21
W20260709 12:23:43.122871 17729 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.16.29.68:42577 timed out after 1.870s (SENT) [suppressed 15 similar messages]
I20260709 12:23:43.093974 24231 raft_consensus.cc:493] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:43.139232 24231 raft_consensus.cc:515] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:43.141654 24231 leader_election.cc:290] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:43.124563 24518 leader_election.cc:290] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:43.141743 24485 raft_consensus.cc:515] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:43.127321 24475 leader_election.cc:290] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:23:43.130456 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 256739us
I20260709 12:23:43.143193 17652 leader_election.cc:304] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:43.144299 24394 leader_election.cc:290] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 21 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:43.144582 17652 leader_election.cc:304] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:43.145526 24244 raft_consensus.cc:2804] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:23:43.145900 24244 raft_consensus.cc:493] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:23:43.146261 24244 raft_consensus.cc:3060] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:23:43.135555 17730 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:43.147979 24098 raft_consensus.cc:493] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:43.136466 24606 log.cc:927] Time spent T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.117s	user 0.000s	sys 0.000s
W20260709 12:23:43.137797 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 143570us
I20260709 12:23:43.126163 23906 raft_consensus.cc:3060] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Advancing to term 21
W20260709 12:23:43.148257 17730 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:43.137279 24600 log.cc:927] Time spent T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.277s	user 0.000s	sys 0.001s
I20260709 12:23:43.151489 17689 raft_consensus.cc:2468] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 19.
I20260709 12:23:43.171420 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 21 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:23:43.166791 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 50725.8us
W20260709 12:23:43.174752 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 206139us
I20260709 12:23:43.176906 17803 leader_election.cc:304] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:43.178133 18368 raft_consensus.cc:2410] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for term 21 because replica has last-logged OpId of term: 19 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 11.
I20260709 12:23:43.148948 24434 raft_consensus.cc:493] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:43.182145 18334 leader_election.cc:304] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:23:43.160799 24227 leader_election.cc:290] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 16 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:43.184722 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:43.166512 24518 raft_consensus.cc:2749] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
W20260709 12:23:43.193060 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 48593.4us
I20260709 12:23:43.197216 18363 raft_consensus.cc:1240] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 17. Current term is 18. Ops: []
I20260709 12:23:43.203132 17652 leader_election.cc:304] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:43.208223 17831 raft_consensus.cc:1240] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 17. Current term is 18. Ops: []
I20260709 12:23:43.209969 18366 raft_consensus.cc:1240] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 16. Current term is 17. Ops: []
W20260709 12:23:43.207793 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 84979.1us
W20260709 12:23:43.214648 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:43.193949 24485 leader_election.cc:290] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 20 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:43.220949 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:43.222218 17803 leader_election.cc:304] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:43.203416 24484 raft_consensus.cc:2804] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:23:43.202333 24436 raft_consensus.cc:493] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:23:43.187775 24098 raft_consensus.cc:515] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:43.222277 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:43.185361 24434 raft_consensus.cc:515] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:43.225553 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:43.224061 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:43.216459 24527 raft_consensus.cc:2804] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:23:43.226866 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:43.225621 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:43.220767 24227 raft_consensus.cc:2749] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:23:43.235342 18368 raft_consensus.cc:1240] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 20. Current term is 21. Ops: []
I20260709 12:23:43.227739 18362 raft_consensus.cc:2468] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 17.
I20260709 12:23:43.238595 18361 raft_consensus.cc:2468] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 19.
W20260709 12:23:43.238772 18334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:43.240859 24211 raft_consensus.cc:493] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:43.241144 24559 raft_consensus.cc:2749] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:23:43.241418 24211 raft_consensus.cc:515] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:43.242769 24417 consensus_queue.cc:1059] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 17.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:23:43.243665 24417 consensus_queue.cc:1243] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer 334a4e1f59944544b3b3ff36616b57da log is divergent from this leader: its last log entry 18.14 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 13
I20260709 12:23:43.244300 24556 consensus_queue.cc:1048] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:23:43.244592 24313 raft_consensus.cc:3055] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:23:43.245023 24313 raft_consensus.cc:740] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 17 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:23:43.245932 24313 consensus_queue.cc:260] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.13, Last appended by leader: 13, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:43.246699 24556 consensus_queue.cc:1048] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 12:23:43.247225 24313 raft_consensus.cc:3060] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:43.239544 18365 raft_consensus.cc:2468] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 16.
I20260709 12:23:43.216611 23838 consensus_queue.cc:1059] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 17.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.017s
I20260709 12:23:43.224531 24484 raft_consensus.cc:493] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:23:43.255234 18371 raft_consensus.cc:1275] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 16 index: 13. (index mismatch)
I20260709 12:23:43.252162 17801 leader_election.cc:304] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:43.237013 24087 raft_consensus.cc:493] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:23:43.267031 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 16 candidate_status { last_received { term: 14 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:43.268234 17685 raft_consensus.cc:2393] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 16: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:23:43.268878 17689 raft_consensus.cc:1240] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 14. Current term is 15. Ops: []
I20260709 12:23:43.269171 24098 leader_election.cc:290] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:43.229862 24392 consensus_queue.cc:1059] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 16.11, Next index: 12, Last known committed idx: 10, Time since last communication: 0.031s
I20260709 12:23:43.274044 24491 consensus_queue.cc:1048] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.036s
W20260709 12:23:43.276057 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 57016.3us
W20260709 12:23:43.276748 18361 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59886 (request call id 1814) took 2081 ms (client timeout 1870 ms). Trace:
W20260709 12:23:43.277158 18361 rpcz_store.cc:269] 0709 12:23:41.194846 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:41.195050 (+   204us) service_pool.cc:224] Handling call
0709 12:23:43.276513 (+2081463us) spinlock_profiling.cc:243] Waited 1.78 s on lock 0x7b54008039d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:23:43.276631 (+   118us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4094478592}
I20260709 12:23:43.277647 17652 leader_election.cc:304] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:43.278460 24579 raft_consensus.cc:2804] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:23:43.278975 24579 raft_consensus.cc:493] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:23:43.279062 17652 leader_election.cc:304] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:43.279474 24579 raft_consensus.cc:3060] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:43.280086 24428 raft_consensus.cc:3060] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:43.284601 24392 raft_consensus.cc:3055] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:23:43.285058 24392 raft_consensus.cc:740] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:23:43.286020 24392 consensus_queue.cc:260] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.11, Last appended by leader: 11, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:43.289068 24231 raft_consensus.cc:3055] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:23:43.289598 24231 raft_consensus.cc:740] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:23:43.289502 24485 consensus_queue.cc:1048] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:23:43.290832 24231 consensus_queue.cc:260] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.13, Last appended by leader: 13, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:43.292444 24227 raft_consensus.cc:3060] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:43.293452 17801 leader_election.cc:304] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:43.291761 24485 consensus_queue.cc:1048] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 10, Time since last communication: 0.002s
I20260709 12:23:43.295085 24486 raft_consensus.cc:2804] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:23:43.295666 24486 raft_consensus.cc:697] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 17 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:23:43.296002 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 19 candidate_status { last_received { term: 18 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:43.254026 24436 raft_consensus.cc:515] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:43.296667 24486 consensus_queue.cc:237] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.12, Last appended by leader: 12, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:43.297760 24434 leader_election.cc:290] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:43.298405 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 16 candidate_status { last_received { term: 14 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:43.298870 24436 leader_election.cc:290] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:43.299230 17829 raft_consensus.cc:2393] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 16: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:23:43.299810 17760 raft_consensus.cc:2468] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 13.
I20260709 12:23:43.300104 24231 raft_consensus.cc:3060] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:43.301056 17760 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38786 (request call id 3411) took 1228 ms. Trace:
I20260709 12:23:43.301367 17760 rpcz_store.cc:276] 0709 12:23:42.072242 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:42.072468 (+   226us) service_pool.cc:224] Handling call
0709 12:23:43.300893 (+1228425us) spinlock_profiling.cc:243] Waited 1.16 s on lock 0x7b54000fe6d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:23:43.301005 (+   112us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2657877888}
I20260709 12:23:43.296797 24211 leader_election.cc:290] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:43.302814 17686 raft_consensus.cc:1240] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 15. Current term is 17. Ops: []
I20260709 12:23:43.304031 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 21 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:43.304706 17688 raft_consensus.cc:3060] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:43.304615 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:43.305660 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 17 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:43.267061 24087 raft_consensus.cc:515] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:43.310515 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:43.310370 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:43.311622 17687 raft_consensus.cc:2468] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 14.
I20260709 12:23:43.312219 18333 leader_election.cc:400] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 16 election: Vote denied by peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651) with higher term. Message: Invalid argument: T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 16. Current term is 17.
I20260709 12:23:43.312810 17684 raft_consensus.cc:1240] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 15. Current term is 16. Ops: []
W20260709 12:23:43.312129 17803 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:43.314363 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 21 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:23:43.315307 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:43.255736 24527 raft_consensus.cc:697] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 19 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:23:43.258314 24484 raft_consensus.cc:3060] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:43.324622 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:43.244335 24490 consensus_queue.cc:1059] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 18.13, Next index: 14, Last known committed idx: 12, Time since last communication: 0.072s
I20260709 12:23:43.287654 24392 raft_consensus.cc:3060] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:43.238528 24524 consensus_queue.cc:1059] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.080s
I20260709 12:23:43.326589 18368 raft_consensus.cc:2468] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 13.
I20260709 12:23:43.324142 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 18 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:43.336889 18374 raft_consensus.cc:2468] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 17.
I20260709 12:23:43.310415 24087 leader_election.cc:290] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:43.334079 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:43.332592 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:43.363274 17833 raft_consensus.cc:2393] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 16: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:23:43.332756 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 16 candidate_status { last_received { term: 14 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:43.333635 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 19 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:43.333602 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 18 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:43.334631 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:43.338209 24500 consensus_queue.cc:1048] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:23:43.338882 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:23:43.365324 17729 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:43.341089 24611 log.cc:927] Time spent T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.115s	user 0.000s	sys 0.003s
W20260709 12:23:43.345741 17587 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:37172 (request call id 257) took 20329 ms (client timeout 15000 ms). Trace:
I20260709 12:23:43.348259 24524 consensus_queue.cc:1243] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer 334a4e1f59944544b3b3ff36616b57da log is divergent from this leader: its last log entry 16.13 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 11
W20260709 12:23:43.350349 24615 log.cc:927] Time spent T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.077s	user 0.001s	sys 0.000s
I20260709 12:23:43.377274 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:43.350080 24434 consensus_queue.cc:1059] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 16.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 12:23:43.262688 24417 consensus_queue.cc:1059] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 17.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.070s
I20260709 12:23:43.379185 24417 raft_consensus.cc:3055] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:23:43.379555 24417 raft_consensus.cc:740] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
W20260709 12:23:43.368242 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 89247.2us
I20260709 12:23:43.353454 24475 raft_consensus.cc:3055] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:23:43.355523 17688 raft_consensus.cc:2468] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 21.
I20260709 12:23:43.330531 23906 consensus_queue.cc:1059] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 16.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.026s
I20260709 12:23:43.337184 24244 raft_consensus.cc:515] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:23:43.361686 24612 log.cc:927] Time spent T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.118s	user 0.000s	sys 0.001s
I20260709 12:23:43.327612 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:43.397254 17652 leader_election.cc:304] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:43.365444 17689 raft_consensus.cc:2468] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 23.
I20260709 12:23:43.340289 18333 leader_election.cc:304] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:23:43.398483 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 58285us
I20260709 12:23:43.366355 17829 raft_consensus.cc:2468] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 18.
I20260709 12:23:43.366278 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 19 candidate_status { last_received { term: 18 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:43.366848 17835 raft_consensus.cc:1240] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 18. Current term is 19. Ops: [18.11-18.11]
I20260709 12:23:43.372225 24491 raft_consensus.cc:3055] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:23:43.373070 17838 raft_consensus.cc:1240] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 19. Current term is 20. Ops: []
I20260709 12:23:43.400255 18368 raft_consensus.cc:2376] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 19. Re-sending same reply.
I20260709 12:23:43.384873 17729 leader_election.cc:304] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:43.367537 17681 raft_consensus.cc:2468] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 15.
I20260709 12:23:43.321604 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 21 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:43.398293 17686 raft_consensus.cc:2468] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 20.
I20260709 12:23:43.404525 17685 raft_consensus.cc:2468] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 17.
I20260709 12:23:43.410645 24527 consensus_queue.cc:237] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.13, Last appended by leader: 13, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:43.414135 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:43.415947 17652 leader_election.cc:304] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:43.419384 24250 raft_consensus.cc:2749] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:23:43.420280 24250 consensus_queue.cc:1048] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 12:23:43.421371 24244 leader_election.cc:290] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 18 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:43.405870 24500 raft_consensus.cc:2804] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:23:43.428239 24500 raft_consensus.cc:493] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:43.428622 24500 raft_consensus.cc:3060] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:43.401935 24491 raft_consensus.cc:740] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:23:43.409632 24087 raft_consensus.cc:2804] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:23:43.431643 24417 consensus_queue.cc:260] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 17.14, Last appended by leader: 14, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:43.432319 24491 consensus_queue.cc:260] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.12, Last appended by leader: 12, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:43.433496 24417 raft_consensus.cc:3060] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:43.434191 23740 raft_consensus.cc:493] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:23:43.434810 23740 raft_consensus.cc:515] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:43.436003 17728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:43.436180 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:43.436858 17829 raft_consensus.cc:2468] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 19.
I20260709 12:23:43.437741 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:43.439615 17829 raft_consensus.cc:1240] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 14. Current term is 15. Ops: [14.12-14.12]
I20260709 12:23:43.432464 24087 raft_consensus.cc:493] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:23:43.441104 24087 raft_consensus.cc:3060] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:43.389247 24475 raft_consensus.cc:740] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:23:43.444635 24475 consensus_queue.cc:260] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.13, Last appended by leader: 13, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:43.445375 23740 leader_election.cc:290] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:43.446098 24475 raft_consensus.cc:3060] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:43.415616 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:43.447191 17838 raft_consensus.cc:2393] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 16: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
W20260709 12:23:43.368418 17587 rpcz_store.cc:269] 0709 12:23:23.016414 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:23.032360 (+ 15946us) service_pool.cc:224] Handling call
0709 12:23:43.345440 (+20313080us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:23:36.420680 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:23:36.589584 (+168904us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:23:36.589639 (+    55us) write_op.cc:435] Acquired schema lock
0709 12:23:36.589658 (+    19us) tablet.cc:662] Decoding operations
0709 12:23:36.591411 (+  1753us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:23:36.591526 (+   115us) write_op.cc:641] Partition lock acquired for write op
0709 12:23:36.591551 (+    25us) tablet.cc:685] Acquiring locks for 22 operations
0709 12:23:36.812615 (+221064us) tablet.cc:701] Row locks acquired
0709 12:23:36.812690 (+    75us) write_op.cc:260] PREPARE: finished
0709 12:23:36.812936 (+   246us) write_op.cc:270] Start()
0709 12:23:36.813131 (+   195us) write_op.cc:276] Timestamp: P: 1783599816812847 usec, L: 0
0709 12:23:36.813177 (+    46us) op_driver.cc:348] REPLICATION: starting
0709 12:23:36.814180 (+  1003us) log.cc:844] Serialized 8576 byte log entry
0709 12:23:37.046011 (+231831us) spinlock_profiling.cc:243] Waited 4.76 ms on lock 0x7b4400022b4c. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b47459c85 00007f7b4745a6a2 00007f7b4a1cc1db 00007f7b4a1c7aaf 00007f7b4a1bb86e 00007f7b4a1babb6 00007f7b4a1ba7d8 00007f7b4a1c4221 00007f7b4a1c41d9 00007f7b4a1c4169
0709 12:23:37.442276 (+396265us) op_driver.cc:464] REPLICATION: finished
0709 12:23:38.155474 (+713198us) spinlock_profiling.cc:243] Waited 14.5 ms on lock 0x7b180009a0c0. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b475333f3 00007f7b475357c1 00007f7b47535779 00007f7b47535709 00007f7b475356d1 00007f7b475349cd 0000000000388634 0000000000388479 00007f7b4751683a
0709 12:23:38.155653 (+   179us) write_op.cc:301] APPLY: starting
0709 12:23:38.189154 (+ 33501us) tablet.cc:1367] starting BulkCheckPresence
0709 12:23:38.980260 (+791106us) tablet.cc:1370] finished BulkCheckPresence
0709 12:23:38.980303 (+    43us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:23:39.311252 (+330949us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:23:39.311774 (+   522us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=44,key_file_lookups=44,delta_file_lookups=44,mrs_lookups=0
0709 12:23:39.311899 (+   125us) write_op.cc:312] APPLY: finished
0709 12:23:39.312799 (+   900us) log.cc:844] Serialized 196 byte log entry
0709 12:23:39.433822 (+121023us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:23:39.472227 (+ 38405us) write_op.cc:454] Released schema lock
0709 12:23:39.481331 (+  9104us) write_op.cc:341] FINISH: Updating metrics
0709 12:23:39.510326 (+ 28995us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:23:39.567948 (+ 57622us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":1588399,"reported_tablets":44,"child_traces":[["op",{"apply.queue_time_us":51500,"apply.run_cpu_time_us":47460,"apply.run_wall_time_us":1345100,"cfile_cache_hit":90,"cfile_cache_hit_bytes":96175,"num_ops":22,"prepare.queue_time_us":190721,"prepare.run_cpu_time_us":17030,"prepare.run_wall_time_us":488683,"raft.queue_time_us":257357,"raft.run_cpu_time_us":8452,"raft.run_wall_time_us":59225,"replication_time_us":628903,"spinlock_wait_cycles":44476416,"thread_start_us":521236,"threads_started":5,"wal-append.queue_time_us":459295,"wal-append.run_cpu_time_us":34721,"wal-append.run_wall_time_us":2761098}]]}
I20260709 12:23:43.433576 24491 raft_consensus.cc:3060] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:43.456599 17833 raft_consensus.cc:2468] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 17.
I20260709 12:23:43.448300 17838 raft_consensus.cc:1240] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 13. Current term is 14. Ops: []
I20260709 12:23:43.458877 17688 raft_consensus.cc:1240] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 15. Current term is 16. Ops: []
I20260709 12:23:43.461175 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:43.461311 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:43.409713 24556 raft_consensus.cc:2804] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:23:43.419546 17652 leader_election.cc:304] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:23:43.470569 17838 raft_consensus.cc:1275] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 16 index: 12. Preceding OpId from leader: term: 18 index: 13. (index mismatch)
W20260709 12:23:43.470752 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 51299.4us
I20260709 12:23:43.427644 24588 raft_consensus.cc:515] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:43.466539 24524 raft_consensus.cc:493] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:43.469614 24556 raft_consensus.cc:493] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:43.489220 24250 raft_consensus.cc:2804] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:23:43.491777 24250 raft_consensus.cc:493] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:43.492183 24250 raft_consensus.cc:3060] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:43.492058 24579 raft_consensus.cc:515] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:43.493661 17830 raft_consensus.cc:1240] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 18. Current term is 19. Ops: []
I20260709 12:23:43.494304 24556 raft_consensus.cc:3060] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Advancing to term 14
W20260709 12:23:43.494774 17727 leader_election.cc:336] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 18 pre-election: RPC error from VoteRequest() call to peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837): Timed out: RequestConsensusVote RPC to 127.16.29.67:36837 timed out after 1.557s (SENT)
I20260709 12:23:43.495820 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:43.497225 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:43.497886 17680 raft_consensus.cc:2468] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 19.
I20260709 12:23:43.499096 17838 raft_consensus.cc:1275] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 17 index: 9. Preceding OpId from leader: term: 18 index: 11. (index mismatch)
I20260709 12:23:43.499217 24484 raft_consensus.cc:515] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:43.500303 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:43.500535 17831 raft_consensus.cc:1240] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 17. Current term is 18. Ops: []
I20260709 12:23:43.502595 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:43.503165 17831 raft_consensus.cc:3060] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:23:43.503333 24614 log.cc:927] Time spent T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.262s	user 0.001s	sys 0.001s
I20260709 12:23:43.503960 24524 raft_consensus.cc:515] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:43.504809 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:43.505584 18366 raft_consensus.cc:2393] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 16: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:23:43.506278 24524 leader_election.cc:290] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:43.508144 17834 raft_consensus.cc:2368] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for earlier term 19. Current term is 20.
I20260709 12:23:43.509153 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:43.511109 17834 raft_consensus.cc:1240] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 15. Current term is 16. Ops: []
I20260709 12:23:43.511059 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:43.511812 17687 raft_consensus.cc:2468] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 23.
I20260709 12:23:43.512018 17832 raft_consensus.cc:1240] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 18. Current term is 20. Ops: []
I20260709 12:23:43.512204 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 21 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:43.513075 17685 raft_consensus.cc:2410] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for term 21 because replica has last-logged OpId of term: 19 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 11.
I20260709 12:23:43.514158 17681 raft_consensus.cc:1240] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 20. Current term is 21. Ops: []
I20260709 12:23:43.515278 17680 raft_consensus.cc:1275] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 13 index: 9. Preceding OpId from leader: term: 19 index: 12. (index mismatch)
I20260709 12:23:43.516345 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:43.517606 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 16 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:43.518790 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:43.518791 24500 raft_consensus.cc:515] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:43.506299 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 15 candidate_status { last_received { term: 14 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:43.522505 18366 raft_consensus.cc:2468] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 15.
I20260709 12:23:43.523430 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:43.525468 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:43.526167 17764 raft_consensus.cc:2468] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 19.
I20260709 12:23:43.489653 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:43.528299 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:43.529881 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:43.532265 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 21 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:43.488318 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:43.514253 17832 raft_consensus.cc:1240] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 19. Current term is 20. Ops: [17.11-19.12]
I20260709 12:23:43.489858 17652 leader_election.cc:304] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:43.533488 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:43.533913 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
W20260709 12:23:43.534279 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:43.534435 18366 raft_consensus.cc:3060] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:43.541313 17729 leader_election.cc:304] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:43.552567 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:43.563361 17763 raft_consensus.cc:3060] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:43.552834 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:43.532315 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:43.564644 17756 raft_consensus.cc:2468] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 18.
I20260709 12:23:43.556746 17764 raft_consensus.cc:1240] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 18. Current term is 19. Ops: []
I20260709 12:23:43.553078 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:43.553681 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 16 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:43.558064 17831 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 15.
I20260709 12:23:43.553906 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:43.556147 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 21 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:43.577139 17683 raft_consensus.cc:1275] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 18 index: 10. Preceding OpId from leader: term: 18 index: 11. (index mismatch)
I20260709 12:23:43.577677 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:43.578253 18334 leader_election.cc:304] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:43.578951 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:43.571270 17728 leader_election.cc:304] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:43.581954 18365 raft_consensus.cc:2393] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 19: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:23:43.580214 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:43.592146 24588 leader_election.cc:290] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 20 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:43.580123 24428 raft_consensus.cc:2749] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:23:43.596009 24436 raft_consensus.cc:2804] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:23:43.596854 24443 consensus_queue.cc:1048] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 12:23:43.597849 17684 raft_consensus.cc:1240] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 19. Current term is 20. Ops: []
I20260709 12:23:43.598306 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:43.597683 18374 raft_consensus.cc:2393] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 14: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
W20260709 12:23:43.599666 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 58430.7us
I20260709 12:23:43.600742 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:43.603158 17803 leader_election.cc:304] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:43.604135 17761 raft_consensus.cc:2468] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 19.
I20260709 12:23:43.612764 17654 leader_election.cc:304] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:43.522498 24500 leader_election.cc:290] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 21 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:43.537957 24484 leader_election.cc:290] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 18 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:43.613291 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 21 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:43.539634 24579 leader_election.cc:290] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 19 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:43.625360 18371 raft_consensus.cc:1240] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 14. Current term is 15. Ops: []
I20260709 12:23:43.623339 17761 raft_consensus.cc:1275] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 17 index: 13. Preceding OpId from leader: term: 18 index: 14. (index mismatch)
W20260709 12:23:43.634230 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 63063.5us
I20260709 12:23:43.636330 18334 leader_election.cc:304] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:43.665086 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:43.667277 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:43.671769 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 20 candidate_status { last_received { term: 18 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:23:43.676154 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 191099us
I20260709 12:23:43.616087 24524 raft_consensus.cc:2804] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:23:43.679430 18334 leader_election.cc:304] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:43.668107 18363 raft_consensus.cc:2468] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 14.
W20260709 12:23:43.682864 17727 leader_election.cc:336] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 16 election: RPC error from VoteRequest() call to peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837): Timed out: RequestConsensusVote RPC to 127.16.29.67:36837 timed out after 1.654s (SENT)
I20260709 12:23:43.666133 17830 raft_consensus.cc:2468] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 12.
I20260709 12:23:43.685869 17758 raft_consensus.cc:1240] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 16. Current term is 17. Ops: []
I20260709 12:23:43.690524 17835 raft_consensus.cc:2468] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 21.
W20260709 12:23:43.693543 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 80878.1us
I20260709 12:23:43.695081 17654 leader_election.cc:304] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:43.620045 24211 raft_consensus.cc:2749] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:23:43.719311 17762 pending_rounds.cc:85] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb: Aborting all ops after (but not including) 11
I20260709 12:23:43.719668 17762 pending_rounds.cc:107] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb: Aborting uncommitted NO_OP operation due to leader change: 14.12
I20260709 12:23:43.719923 17762 raft_consensus.cc:2889] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:43.720600 17680 raft_consensus.cc:1240] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 12. Current term is 13. Ops: []
I20260709 12:23:43.716846 17727 leader_election.cc:304] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:43.708607 17654 leader_election.cc:304] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
W20260709 12:23:43.717324 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 46781.8us
I20260709 12:23:43.616252 24436 raft_consensus.cc:697] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 16 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
W20260709 12:23:43.745754 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 66402.2us
I20260709 12:23:43.747067 18374 raft_consensus.cc:1240] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 16 LEADER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 15. Current term is 16. Ops: [15.12-15.12]
I20260709 12:23:43.673451 24313 raft_consensus.cc:2804] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:23:43.728905 17654 leader_election.cc:304] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:23:43.716519 24490 raft_consensus.cc:3055] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:23:43.760417 18363 raft_consensus.cc:1240] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 16. Current term is 17. Ops: []
W20260709 12:23:43.761000 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 157965us
W20260709 12:23:43.761656 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 178946us
I20260709 12:23:43.762259 24250 raft_consensus.cc:515] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:43.763198 24441 raft_consensus.cc:2804] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:23:43.730263 24394 raft_consensus.cc:493] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:43.766371 18334 leader_election.cc:304] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:43.767059 23838 consensus_queue.cc:1059] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 15.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:23:43.769105 24244 raft_consensus.cc:2749] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:23:43.730183 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
W20260709 12:23:43.773989 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:43.717765 24428 raft_consensus.cc:2749] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
W20260709 12:23:43.788556 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 152833us
I20260709 12:23:43.789170 24524 raft_consensus.cc:697] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:23:43.789306 24439 raft_consensus.cc:2749] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:23:43.680879 24588 raft_consensus.cc:2749] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
W20260709 12:23:43.802714 17728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:43.803529 24313 raft_consensus.cc:493] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:43.813884 17652 leader_election.cc:400] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 19 election: Vote denied by peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837) with higher term. Message: Invalid argument: T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for earlier term 19. Current term is 20.
I20260709 12:23:43.856851 17652 leader_election.cc:403] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 19 election: Cancelling election due to peer responding with higher term
I20260709 12:23:43.692340 24518 raft_consensus.cc:2749] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 12:23:43.785051 17582 catalog_manager.cc:5697] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b reported cstate change: term changed from 6 to 9, leader changed from e063541542a94969898868bf6eb5ce1b (127.16.29.67) to e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66). New cstate: current_term: 9 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } }
W20260709 12:23:43.862465 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:43.867060 17803 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:43.866845 18333 leader_election.cc:304] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
W20260709 12:23:43.821260 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 131985us
I20260709 12:23:43.826948 17681 raft_consensus.cc:3060] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:43.732242 24227 raft_consensus.cc:2749] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
W20260709 12:23:43.836133 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 69859.2us
I20260709 12:23:43.808283 24490 raft_consensus.cc:740] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:23:43.837247 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 21 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:43.900403 18366 raft_consensus.cc:2468] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 17.
I20260709 12:23:43.812402 24441 raft_consensus.cc:493] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:23:43.822952 24394 raft_consensus.cc:515] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:43.697561 24392 raft_consensus.cc:2749] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:23:43.844324 17763 raft_consensus.cc:2468] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 20.
I20260709 12:23:43.845070 24244 raft_consensus.cc:2804] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:23:43.855930 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:43.926663 17680 raft_consensus.cc:3060] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:43.858464 23838 raft_consensus.cc:3055] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:23:43.928453 17689 raft_consensus.cc:1240] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 18. Current term is 19. Ops: []
W20260709 12:23:43.888317 17730 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:43.903332 17802 leader_election.cc:304] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:43.864403 24313 raft_consensus.cc:3060] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:43.840265 24443 raft_consensus.cc:493] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:43.908690 24441 raft_consensus.cc:3060] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:43.880676 24524 consensus_queue.cc:237] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 18.16, Last appended by leader: 16, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:43.928681 18334 leader_election.cc:304] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:43.888902 24500 raft_consensus.cc:2804] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Leader election won for term 21
W20260709 12:23:43.950519 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 47408.6us
W20260709 12:23:43.936757 17727 leader_election.cc:336] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837): Timed out: RequestConsensusVote RPC to 127.16.29.67:36837 timed out after 1.820s (SENT)
I20260709 12:23:43.951854 17727 leader_election.cc:304] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
W20260709 12:23:43.942435 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 90937.5us
I20260709 12:23:43.917171 17687 raft_consensus.cc:2410] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 21 because replica has last-logged OpId of term: 20 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 13.
W20260709 12:23:43.943934 17730 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:43.928428 24211 raft_consensus.cc:2749] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
W20260709 12:23:43.948295 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 199139us
I20260709 12:23:43.956131 24443 raft_consensus.cc:515] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:43.958539 24443 leader_election.cc:290] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:43.928467 23838 raft_consensus.cc:740] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:23:43.960202 24394 leader_election.cc:290] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:43.916528 24439 consensus_queue.cc:1059] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 16.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.044s
I20260709 12:23:43.960843 23838 consensus_queue.cc:260] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 15.13, Last appended by leader: 13, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:43.962355 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:43.962785 23838 raft_consensus.cc:3060] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:43.963135 17835 raft_consensus.cc:2393] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 17: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:23:43.966288 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:43.966961 17835 raft_consensus.cc:2468] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 18.
I20260709 12:23:43.925729 24244 raft_consensus.cc:493] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
W20260709 12:23:43.941289 24231 consensus_meta.cc:287] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.641s	user 0.013s	sys 0.000s
W20260709 12:23:43.950078 17729 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:43.970896 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:43.873996 24087 raft_consensus.cc:515] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:43.973064 17728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:43.972877 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:43.976809 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:44.006840 17830 raft_consensus.cc:2468] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 14.
I20260709 12:23:43.952119 17802 leader_election.cc:304] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:23:43.999950 18333 leader_election.cc:304] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:44.009649 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:44.010967 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:44.011782 17757 raft_consensus.cc:3060] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:44.012871 18364 raft_consensus.cc:2468] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 19.
I20260709 12:23:44.014938 17763 raft_consensus.cc:2393] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 19: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:23:44.015257 17758 raft_consensus.cc:1275] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 16 index: 12. (index mismatch)
W20260709 12:23:44.017715 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 65656.5us
W20260709 12:23:44.035247 24621 log.cc:927] Time spent T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.067s	user 0.000s	sys 0.000s
I20260709 12:23:44.035578 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:44.036257 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:44.038759 17759 raft_consensus.cc:2393] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 17: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:23:43.978322 24588 raft_consensus.cc:2749] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:23:44.005577 24244 raft_consensus.cc:3060] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:44.047470 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
W20260709 12:23:44.049418 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 49607.3us
I20260709 12:23:44.045528 17830 raft_consensus.cc:1240] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 14. Current term is 15. Ops: [14.10-14.10]
I20260709 12:23:44.052834 24394 raft_consensus.cc:3060] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Advancing to term 16
W20260709 12:23:44.053103 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 125350us
I20260709 12:23:44.016366 24211 raft_consensus.cc:2804] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:23:44.036326 24471 raft_consensus.cc:2749] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:23:44.013077 24500 raft_consensus.cc:697] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 21 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:23:44.071797 17681 raft_consensus.cc:2468] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 20.
I20260709 12:23:44.037819 24227 raft_consensus.cc:493] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:44.037487 24428 raft_consensus.cc:493] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:23:44.040174 24484 raft_consensus.cc:2804] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Leader pre-election won for term 20
W20260709 12:23:44.074009 17829 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.577s	user 0.016s	sys 0.020s
I20260709 12:23:44.069571 24250 leader_election.cc:290] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 18 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:43.999547 24443 consensus_queue.cc:1048] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.072s
W20260709 12:23:44.081348 17829 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.584s	user 0.017s	sys 0.020s
I20260709 12:23:44.086361 18333 leader_election.cc:304] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:44.082448 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 20 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:43.963155 24490 consensus_queue.cc:260] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.12, Last appended by leader: 12, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:44.095366 17652 leader_election.cc:304] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:44.097043 17652 leader_election.cc:304] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:44.000228 24436 consensus_queue.cc:237] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 14.14, Last appended by leader: 14, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:44.115509 17801 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.502s	user 0.009s	sys 0.036s
W20260709 12:23:44.119676 17801 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.507s	user 0.009s	sys 0.036s
W20260709 12:23:44.119706 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 47514.3us
I20260709 12:23:44.120859 18333 leader_election.cc:304] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:44.074738 24211 raft_consensus.cc:493] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:44.087239 24484 raft_consensus.cc:493] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:23:44.120191 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 544962us
I20260709 12:23:44.129225 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:44.129472 18364 raft_consensus.cc:1275] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 16 index: 13. Preceding OpId from leader: term: 19 index: 15. (index mismatch)
I20260709 12:23:44.177647 17801 leader_election.cc:304] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:44.081976 24250 raft_consensus.cc:493] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:23:44.186059 17801 leader_election.cc:304] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:44.187044 17757 raft_consensus.cc:2468] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 17.
I20260709 12:23:44.080443 24518 raft_consensus.cc:2749] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Leader election lost for term 19. Reason: Vote denied by peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837) with higher term. Message: Invalid argument: T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for earlier term 19. Current term is 20.
W20260709 12:23:44.187297 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 66546us
I20260709 12:23:44.200356 17727 leader_election.cc:304] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:44.118933 24392 raft_consensus.cc:2804] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:23:44.209625 17803 leader_election.cc:304] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:44.210217 24250 raft_consensus.cc:515] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:44.213007 24250 leader_election.cc:290] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:44.220360 24431 consensus_queue.cc:1048] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:23:44.221740 24437 consensus_queue.cc:1059] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.001s
W20260709 12:23:44.224486 24623 log.cc:927] Time spent T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.187s	user 0.000s	sys 0.000s
I20260709 12:23:44.225096 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:44.227367 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 20 candidate_status { last_received { term: 18 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:44.250088 18361 raft_consensus.cc:3060] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:44.128172 24087 leader_election.cc:290] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 15 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:44.230952 17755 raft_consensus.cc:1275] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 18 index: 13. Preceding OpId from leader: term: 19 index: 14. (index mismatch)
I20260709 12:23:44.231727 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:44.255671 17763 raft_consensus.cc:3060] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:44.232748 24491 consensus_queue.cc:1048] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:23:44.233553 24475 raft_consensus.cc:493] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:44.231673 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:44.235271 24500 consensus_queue.cc:237] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:44.172677 24211 raft_consensus.cc:3060] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:23:44.230082 17758 raft_consensus.cc:1275] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 16 index: 13. (index mismatch)
I20260709 12:23:44.184114 24231 raft_consensus.cc:2749] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 12:23:44.180962 24484 raft_consensus.cc:3060] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:44.100574 24227 raft_consensus.cc:515] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:44.200163 24579 raft_consensus.cc:2749] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:23:44.243551 24490 raft_consensus.cc:3060] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:44.257187 17764 raft_consensus.cc:2468] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 19.
I20260709 12:23:44.087898 24428 raft_consensus.cc:515] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:44.249086 23906 raft_consensus.cc:2749] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
W20260709 12:23:44.255254 24556 consensus_meta.cc:287] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.760s	user 0.011s	sys 0.001s
I20260709 12:23:44.252842 24250 raft_consensus.cc:493] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:23:44.290736 18364 raft_consensus.cc:2393] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 18: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:23:44.280313 24437 raft_consensus.cc:3055] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:23:44.309195 18361 raft_consensus.cc:2468] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 20.
I20260709 12:23:44.311151 17763 raft_consensus.cc:2468] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 21.
I20260709 12:23:44.293200 24231 raft_consensus.cc:493] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:44.332609 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 159711us
W20260709 12:23:44.333072 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 604210us
W20260709 12:23:44.335618 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 406997us
I20260709 12:23:44.337759 18334 leader_election.cc:304] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:23:44.341444 18333 leader_election.cc:304] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:44.343165 17654 leader_election.cc:304] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b
W20260709 12:23:44.367194 17730 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:44.288761 24087 consensus_queue.cc:1048] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 10, Time since last communication: 0.044s
I20260709 12:23:44.286780 24475 raft_consensus.cc:515] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:44.372112 18333 leader_election.cc:304] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:44.302603 24437 raft_consensus.cc:740] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:23:44.291085 24556 raft_consensus.cc:515] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:44.410967 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 67923.5us
I20260709 12:23:44.412065 17654 leader_election.cc:304] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:44.404839 18334 leader_election.cc:304] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:44.337495 23906 raft_consensus.cc:493] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:44.339797 24443 raft_consensus.cc:2749] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:23:44.344265 24527 raft_consensus.cc:2804] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Leader pre-election won for term 16
W20260709 12:23:44.374809 24623 log.cc:927] Time spent T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.057s	user 0.000s	sys 0.000s
I20260709 12:23:44.410110 17727 leader_election.cc:400] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 19 election: Vote denied by peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837) with higher term. Message: Invalid argument: T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for earlier term 19. Current term is 20.
I20260709 12:23:44.421550 17727 leader_election.cc:403] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 19 election: Cancelling election due to peer responding with higher term
W20260709 12:23:44.421691 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 372532us
I20260709 12:23:44.339546 24231 raft_consensus.cc:515] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:44.415910 24638 raft_consensus.cc:2749] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:23:44.421025 24527 raft_consensus.cc:493] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:44.370817 24250 raft_consensus.cc:515] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:44.419920 23906 raft_consensus.cc:515] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:44.435281 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:44.452430 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:44.435525 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 62484.9us
W20260709 12:23:44.466092 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 271385us
W20260709 12:23:44.466971 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 370021us
I20260709 12:23:44.429559 24491 raft_consensus.cc:2749] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Leader election lost for term 19. Reason: Vote denied by peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837) with higher term. Message: Invalid argument: T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for earlier term 19. Current term is 20.
W20260709 12:23:44.486619 17680 consensus_meta.cc:287] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.541s	user 0.010s	sys 0.000s
I20260709 12:23:44.487085 17680 raft_consensus.cc:2468] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 20.
I20260709 12:23:44.482178 17801 leader_election.cc:304] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:23:44.491037 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 107198us
I20260709 12:23:44.443609 24443 raft_consensus.cc:2804] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:23:44.449628 24087 raft_consensus.cc:2749] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:23:44.427995 24431 consensus_queue.cc:1048] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.054s
I20260709 12:23:44.478180 24527 raft_consensus.cc:3060] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:44.445143 24642 raft_consensus.cc:2804] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:23:44.475656 24417 raft_consensus.cc:493] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:44.480293 24418 raft_consensus.cc:2749] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
W20260709 12:23:44.547260 17730 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:44.428771 24639 consensus_queue.cc:1059] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.090s
W20260709 12:23:44.557305 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 347768us
W20260709 12:23:44.549679 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 177672us
I20260709 12:23:44.493171 24227 leader_election.cc:290] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:23:44.557168 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 74229.1us
I20260709 12:23:44.562961 17727 leader_election.cc:304] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:44.531764 23740 raft_consensus.cc:2749] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:23:44.522027 24443 raft_consensus.cc:493] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
W20260709 12:23:44.568449 24628 log.cc:927] Time spent T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.256s	user 0.000s	sys 0.001s
I20260709 12:23:44.568784 17837 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35042 (request call id 2689) took 1061 ms. Trace:
I20260709 12:23:44.569104 24637 consensus_queue.cc:1048] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.134s
I20260709 12:23:44.569229 17837 rpcz_store.cc:276] 0709 12:23:43.372811 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:43.373476 (+   665us) service_pool.cc:224] Handling call
0709 12:23:43.373830 (+   354us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:43.374228 (+   398us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:43.374255 (+    27us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:43.374739 (+   484us) log.cc:844] Serialized 26 byte log entry
0709 12:23:43.667354 (+292615us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:23:43.821894 (+154540us) log.cc:844] Serialized 14 byte log entry
0709 12:23:43.878933 (+ 57039us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:43.879054 (+   121us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:44.434324 (+555270us) raft_consensus.cc:1685] finished
0709 12:23:44.434343 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:44.434483 (+   140us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":56609,"prepare.queue_time_us":155122,"prepare.run_cpu_time_us":55,"prepare.run_wall_time_us":57,"raft.queue_time_us":57940,"raft.run_cpu_time_us":227,"raft.run_wall_time_us":22109,"thread_start_us":291665,"threads_started":1,"wal-append.queue_time_us":541841}
I20260709 12:23:44.574395 24556 leader_election.cc:290] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 14 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:23:44.575464 24627 log.cc:927] Time spent T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.122s	user 0.000s	sys 0.000s
W20260709 12:23:44.577739 24635 log.cc:927] Time spent T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.083s	user 0.000s	sys 0.000s
I20260709 12:23:44.548727 24428 leader_election.cc:290] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:44.580205 17829 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35042 (request call id 2693) took 1206 ms. Trace:
W20260709 12:23:44.568629 24636 log.cc:927] Time spent T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.141s	user 0.000s	sys 0.001s
I20260709 12:23:44.549973 24642 raft_consensus.cc:493] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:44.444870 24437 consensus_queue.cc:260] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.13, Last appended by leader: 13, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:44.586803 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:44.580643 17829 rpcz_store.cc:276] 0709 12:23:43.374034 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:43.440356 (+ 66322us) service_pool.cc:224] Handling call
0709 12:23:43.457670 (+ 17314us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:43.469451 (+ 11781us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:43.469486 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:43.473587 (+  4101us) log.cc:844] Serialized 26 byte log entry
0709 12:23:44.089073 (+615486us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:23:44.122683 (+ 33610us) log.cc:844] Serialized 14 byte log entry
0709 12:23:44.123117 (+   434us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:44.123200 (+    83us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:44.579946 (+456746us) raft_consensus.cc:1685] finished
0709 12:23:44.579969 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:44.580136 (+   167us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":33072,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":41,"raft.queue_time_us":5230,"raft.run_cpu_time_us":224,"raft.run_wall_time_us":2235,"thread_start_us":591903,"threads_started":1,"wal-append.queue_time_us":920308}
I20260709 12:23:44.592713 18366 raft_consensus.cc:3060] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:44.584518 17803 leader_election.cc:304] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:44.556286 24654 raft_consensus.cc:2804] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Leader election won for term 21
W20260709 12:23:44.594164 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:44.595690 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:44.609951 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 197986us
I20260709 12:23:44.611106 17654 leader_election.cc:304] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:44.567620 24443 raft_consensus.cc:3060] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:44.568527 24639 raft_consensus.cc:3055] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:23:44.626642 17803 leader_election.cc:304] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:44.584996 24491 raft_consensus.cc:2749] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:23:44.595624 17758 consensus_queue.cc:237] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 20.14, Last appended by leader: 13, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:23:44.581108 24588 raft_consensus.cc:493] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:23:44.548817 24643 consensus_queue.cc:1048] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.063s
I20260709 12:23:44.563941 24087 consensus_queue.cc:1048] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.051s
I20260709 12:23:44.584619 24556 raft_consensus.cc:493] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:23:44.613991 17802 leader_election.cc:304] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:23:44.591271 24642 raft_consensus.cc:3060] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:44.563403 24417 raft_consensus.cc:515] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:44.645530 17682 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44898 (request call id 1303) took 1337 ms. Trace:
I20260709 12:23:44.629432 24437 raft_consensus.cc:3060] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:44.647563 24475 leader_election.cc:290] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:44.659477 24436 consensus_queue.cc:1059] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 14.10, Next index: 11, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:23:44.648268 17761 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36572 (request call id 1202) took 1024 ms. Trace:
I20260709 12:23:44.656389 18366 raft_consensus.cc:2468] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 14.
I20260709 12:23:44.663206 17761 rpcz_store.cc:276] 0709 12:23:43.623773 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:43.660228 (+ 36455us) service_pool.cc:224] Handling call
0709 12:23:43.690207 (+ 29979us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:43.690542 (+   335us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:43.690576 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:43.691034 (+   458us) log.cc:844] Serialized 26 byte log entry
0709 12:23:44.061990 (+370956us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:23:44.062029 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:44.062095 (+    66us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:44.648041 (+585946us) raft_consensus.cc:1685] finished
0709 12:23:44.648065 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:44.648208 (+   143us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":9647,"raft.queue_time_us":741,"raft.run_cpu_time_us":147,"raft.run_wall_time_us":13726,"thread_start_us":253054,"threads_started":1,"wal-append.queue_time_us":613168}
I20260709 12:23:44.628048 24639 raft_consensus.cc:740] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:23:44.641372 24643 consensus_queue.cc:1243] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer e2909cff1da44ad387f96a8ae3aa5afb log is divergent from this leader: its last log entry 15.12 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 11
W20260709 12:23:44.686388 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 131687us
I20260709 12:23:44.644951 24392 raft_consensus.cc:493] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:23:44.692950 17654 leader_election.cc:304] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
W20260709 12:23:44.699697 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 73133.5us
I20260709 12:23:44.649158 17682 rpcz_store.cc:276] 0709 12:23:43.308051 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:43.314337 (+  6286us) service_pool.cc:224] Handling call
0709 12:23:43.314707 (+   370us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:43.314940 (+   233us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 12:23:43.315305 (+   365us) log.cc:844] Serialized 14 byte log entry
0709 12:23:44.420645 (+1105340us) log.cc:844] Serialized 14 byte log entry
0709 12:23:44.645110 (+224465us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:44.645169 (+    59us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:23:44.645229 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:44.645309 (+    80us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:44.645461 (+   152us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":32501,"prepare.queue_time_us":599424,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":35,"raft.queue_time_us":134163,"raft.run_cpu_time_us":266,"raft.run_wall_time_us":30904,"thread_start_us":1296008,"threads_started":4,"wal-append.queue_time_us":639868}
W20260709 12:23:44.709031 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 226928us
I20260709 12:23:44.709908 17801 leader_election.cc:304] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:44.645671 24556 raft_consensus.cc:515] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:44.712890 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 130691us
I20260709 12:23:44.712374 17654 leader_election.cc:304] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:23:44.648092 24657 consensus_queue.cc:1059] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 12.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.067s
I20260709 12:23:44.692982 24392 raft_consensus.cc:3060] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:44.725188 17654 leader_election.cc:304] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:23:44.733803 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:44.734596 17680 raft_consensus.cc:3060] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:44.567006 17582 catalog_manager.cc:5697] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b reported cstate change: term changed from 7 to 10, leader changed from e063541542a94969898868bf6eb5ce1b (127.16.29.67) to e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66). New cstate: current_term: 10 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } }
I20260709 12:23:44.704115 24648 raft_consensus.cc:493] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:44.752789 17680 raft_consensus.cc:2468] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 21.
W20260709 12:23:44.765645 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 50350.5us
I20260709 12:23:44.684423 24588 raft_consensus.cc:515] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:44.814392 24646 log.cc:927] Time spent T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.229s	user 0.000s	sys 0.001s
I20260709 12:23:44.715838 24661 raft_consensus.cc:2804] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Leader pre-election won for term 15
W20260709 12:23:44.839675 24484 consensus_meta.cc:287] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.560s	user 0.011s	sys 0.000s
I20260709 12:23:44.812549 24647 raft_consensus.cc:2749] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:23:44.811734 24649 raft_consensus.cc:2749] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 12:23:44.743606 24227 raft_consensus.cc:2804] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:23:44.891954 17686 raft_consensus.cc:1240] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 14. Current term is 15. Ops: []
I20260709 12:23:44.716219 24231 leader_election.cc:290] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:23:44.892145 17728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:44.703843 24650 raft_consensus.cc:2804] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:23:44.893980 17836 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35042 (request call id 2684) took 1575 ms. Trace:
I20260709 12:23:44.947449 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:44.930048 17682 raft_consensus.cc:1275] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 13 index: 11. Preceding OpId from leader: term: 16 index: 13. (index mismatch)
W20260709 12:23:44.931562 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 230999us
I20260709 12:23:44.932649 17681 raft_consensus.cc:1275] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
I20260709 12:23:44.933790 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 20 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:44.739204 24637 raft_consensus.cc:2804] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Leader election won for term 17
W20260709 12:23:44.830049 24635 log.cc:927] Time spent T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.114s	user 0.000s	sys 0.000s
W20260709 12:23:44.938416 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 343789us
I20260709 12:23:44.941183 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
W20260709 12:23:44.896574 24313 consensus_meta.cc:287] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.929s	user 0.003s	sys 0.020s
I20260709 12:23:44.952251 18374 raft_consensus.cc:3060] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:44.931144 24654 raft_consensus.cc:697] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 21 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
W20260709 12:23:44.953286 17729 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:44.893203 24250 leader_election.cc:290] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:44.893042 23906 leader_election.cc:290] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:44.943092 17728 leader_election.cc:304] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:44.962275 17687 raft_consensus.cc:1240] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 15. Current term is 16. Ops: []
I20260709 12:23:44.970149 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 17 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:44.973719 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:44.974483 17688 raft_consensus.cc:2468] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 19.
I20260709 12:23:44.975523 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 17 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:44.934751 24661 raft_consensus.cc:493] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:23:44.941746 24518 raft_consensus.cc:493] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:44.947098 17836 rpcz_store.cc:276] 0709 12:23:43.318416 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:43.318585 (+   169us) service_pool.cc:224] Handling call
0709 12:23:43.331478 (+ 12893us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:43.354564 (+ 23086us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 12:23:43.659695 (+305131us) log.cc:844] Serialized 14 byte log entry
0709 12:23:44.465331 (+805636us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:44.477190 (+ 11859us) log.cc:844] Serialized 26 byte log entry
0709 12:23:44.477345 (+   155us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:23:44.707405 (+230060us) log.cc:844] Serialized 14 byte log entry
0709 12:23:44.762283 (+ 54878us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:44.762361 (+    78us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:44.762405 (+    44us) raft_consensus.cc:1685] finished
0709 12:23:44.762420 (+    15us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:44.893928 (+131508us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":90142,"prepare.queue_time_us":155623,"prepare.run_cpu_time_us":90,"prepare.run_wall_time_us":91,"raft.queue_time_us":319101,"raft.run_cpu_time_us":335,"raft.run_wall_time_us":25049,"thread_start_us":1187494,"threads_started":4,"wal-append.queue_time_us":688006}
W20260709 12:23:44.959996 24658 log.cc:927] Time spent T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.269s	user 0.002s	sys 0.003s
W20260709 12:23:44.981225 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 271394us
I20260709 12:23:44.982780 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:23:44.984300 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:44.941543 24227 raft_consensus.cc:493] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:23:44.946928 24650 raft_consensus.cc:493] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:44.988196 17728 leader_election.cc:304] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:44.993710 17689 raft_consensus.cc:1240] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 17. Current term is 18. Ops: []
I20260709 12:23:44.997557 17801 leader_election.cc:304] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:44.999022 17801 leader_election.cc:304] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:44.960835 24657 raft_consensus.cc:3055] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:23:45.002233 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 17 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:45.003129 18371 raft_consensus.cc:2468] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 16.
I20260709 12:23:45.004170 17686 raft_consensus.cc:1275] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 17 index: 14. Preceding OpId from leader: term: 19 index: 15. (index mismatch)
I20260709 12:23:44.971158 24524 raft_consensus.cc:493] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:44.993551 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:44.975615 23906 raft_consensus.cc:493] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:44.953228 24484 raft_consensus.cc:515] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:44.980182 24637 raft_consensus.cc:697] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 17 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:23:44.930897 24648 raft_consensus.cc:515] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:45.024572 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:45.007915 17803 leader_election.cc:304] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:23:44.988711 24231 raft_consensus.cc:493] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:45.005331 24439 raft_consensus.cc:493] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:44.960973 24313 raft_consensus.cc:515] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:44.989557 24672 raft_consensus.cc:493] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:44.992461 24661 raft_consensus.cc:3060] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:44.996575 24227 raft_consensus.cc:3060] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:45.012171 24250 raft_consensus.cc:493] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:45.006151 24657 raft_consensus.cc:740] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 12 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:23:45.005842 24650 raft_consensus.cc:3060] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Advancing to term 19
W20260709 12:23:45.050551 24652 log.cc:927] Time spent T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.300s	user 0.003s	sys 0.003s
I20260709 12:23:45.050806 24662 raft_consensus.cc:493] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:45.051203 24656 raft_consensus.cc:2804] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:23:45.052347 24688 consensus_queue.cc:1059] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 16.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:23:45.053666 24639 consensus_queue.cc:260] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.14, Last appended by leader: 14, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:45.052383 23740 raft_consensus.cc:2804] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:23:45.120843 23740 raft_consensus.cc:697] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 19 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:23:45.032908 23906 raft_consensus.cc:515] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:45.069741 17729 leader_election.cc:304] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:45.077872 18374 raft_consensus.cc:2468] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 18.
I20260709 12:23:45.077131 24471 raft_consensus.cc:2804] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:23:45.027150 24579 consensus_queue.cc:1059] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 18.12, Next index: 13, Last known committed idx: 11, Time since last communication: 0.054s
W20260709 12:23:45.099098 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 97621.6us
W20260709 12:23:45.106317 24658 log.cc:927] Time spent T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.070s	user 0.000s	sys 0.000s
I20260709 12:23:45.035048 24659 raft_consensus.cc:2804] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Leader pre-election won for term 18
W20260709 12:23:45.105075 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 491154us
I20260709 12:23:45.139956 17802 leader_election.cc:304] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:23:45.119493 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 143585us
I20260709 12:23:45.089692 24250 raft_consensus.cc:515] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:45.130373 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:45.104259 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 21 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:45.064585 24682 raft_consensus.cc:2749] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:23:45.027954 24518 raft_consensus.cc:515] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:45.118129 24441 consensus_meta.cc:287] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 1.155s	user 0.013s	sys 0.000s
I20260709 12:23:45.139371 23740 consensus_queue.cc:237] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 18.15, Last appended by leader: 15, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:45.143790 17680 raft_consensus.cc:2410] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for term 21 because replica has last-logged OpId of term: 19 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 11.
I20260709 12:23:45.047613 24231 raft_consensus.cc:515] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:45.057700 24439 raft_consensus.cc:515] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:45.012787 17681 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8"
dest_uuid: "0da169ff82674dc1947055ce43de68a3"
mode: GRACEFUL
new_leader_uuid: "e063541542a94969898868bf6eb5ce1b"
 from {username='slave'} at 127.0.0.1:54540
I20260709 12:23:45.203502 17681 raft_consensus.cc:606] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Received request to transfer leadership to TS e063541542a94969898868bf6eb5ce1b
I20260709 12:23:45.121217 24500 raft_consensus.cc:493] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:23:45.125752 24644 raft_consensus.cc:2749] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:23:45.130630 24471 raft_consensus.cc:493] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:45.152827 23906 leader_election.cc:290] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:45.131619 24656 raft_consensus.cc:697] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 20 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
W20260709 12:23:45.149798 24651 log.cc:927] Time spent T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.426s	user 0.000s	sys 0.001s
I20260709 12:23:45.204069 17681 raft_consensus.cc:614] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260709 12:23:45.204705 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 154702us
W20260709 12:23:45.194919 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 252514us
I20260709 12:23:45.031942 24654 consensus_queue.cc:237] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 19.12, Last appended by leader: 12, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:45.143741 24659 raft_consensus.cc:493] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:23:45.188499 24579 raft_consensus.cc:2804] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:23:45.145984 24639 raft_consensus.cc:3060] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:45.187575 24250 leader_election.cc:290] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:45.149626 24556 leader_election.cc:290] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:45.035168 24524 raft_consensus.cc:515] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:45.210285 23906 raft_consensus.cc:2749] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:23:45.088683 24672 raft_consensus.cc:515] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:45.236025 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:45.327117 17836 raft_consensus.cc:2376] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 17. Re-sending same reply.
I20260709 12:23:45.210927 24655 raft_consensus.cc:2749] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:23:45.228608 24471 raft_consensus.cc:3060] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:23:45.328457 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 258641us
I20260709 12:23:45.123016 24662 raft_consensus.cc:515] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:45.342118 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 215974us
W20260709 12:23:45.240020 24394 consensus_meta.cc:287] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 1.170s	user 0.008s	sys 0.005s
W20260709 12:23:45.245599 24652 log.cc:927] Time spent T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.062s	user 0.000s	sys 0.000s
W20260709 12:23:45.345866 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 56808.4us
I20260709 12:23:45.346031 23906 raft_consensus.cc:3055] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:23:45.245143 24659 raft_consensus.cc:3060] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:45.187258 24441 raft_consensus.cc:515] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:45.392264 23906 raft_consensus.cc:740] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:23:45.239614 24417 leader_election.cc:290] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:45.393375 17727 leader_election.cc:304] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:45.380637 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:45.395699 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 20 candidate_status { last_received { term: 18 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:45.349167 17729 leader_election.cc:304] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:45.361495 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:45.254269 24484 leader_election.cc:290] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 20 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:45.395943 18365 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59902 (request call id 1640) took 1845 ms. Trace:
I20260709 12:23:45.334002 24579 raft_consensus.cc:697] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 20 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:23:45.342648 24685 raft_consensus.cc:3055] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:23:45.351478 24644 raft_consensus.cc:2804] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:23:45.404260 24394 raft_consensus.cc:2749] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:23:45.441913 18363 raft_consensus.cc:1240] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 14. Current term is 15. Ops: []
I20260709 12:23:45.352002 24682 raft_consensus.cc:493] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:23:45.419176 18365 rpcz_store.cc:276] 0709 12:23:43.507090 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:43.620880 (+113790us) service_pool.cc:224] Handling call
0709 12:23:43.670991 (+ 50111us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:23:43.745435 (+ 74444us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 12:23:44.266260 (+520825us) log.cc:844] Serialized 14 byte log entry
0709 12:23:44.649743 (+383483us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:23:44.667563 (+ 17820us) log.cc:844] Serialized 52 byte log entry
0709 12:23:44.667755 (+   192us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:23:44.667802 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:44.667899 (+    97us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:45.351738 (+683839us) raft_consensus.cc:1685] finished
0709 12:23:45.351770 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:45.351984 (+   214us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":51919,"prepare.queue_time_us":663595,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":41,"raft.queue_time_us":497019,"raft.run_cpu_time_us":415,"raft.run_wall_time_us":60340,"thread_start_us":842066,"threads_started":3,"wal-append.queue_time_us":350763}
I20260709 12:23:45.229697 24500 raft_consensus.cc:515] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:45.459170 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 65891.6us
I20260709 12:23:45.442272 23906 consensus_queue.cc:260] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.11, Last appended by leader: 11, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:45.351161 24556 raft_consensus.cc:493] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:45.404944 24655 raft_consensus.cc:493] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:45.405439 24663 raft_consensus.cc:493] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:45.460093 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 111091us
W20260709 12:23:45.474743 17652 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.990s	user 0.014s	sys 0.018s
W20260709 12:23:45.513970 17652 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.030s	user 0.015s	sys 0.019s
W20260709 12:23:45.514401 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 1.03013e+06us
I20260709 12:23:45.422233 24313 leader_election.cc:290] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 20 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:45.423048 24588 leader_election.cc:290] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:23:45.464305 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 117087us
I20260709 12:23:45.468132 17763 raft_consensus.cc:2468] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 18.
I20260709 12:23:45.518719 18361 raft_consensus.cc:1240] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 15. Current term is 16. Ops: []
I20260709 12:23:45.516801 17652 leader_election.cc:304] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:45.246104 24657 consensus_queue.cc:260] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.8, Last appended by leader: 8, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:45.228897 24637 consensus_queue.cc:237] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 15.13, Last appended by leader: 13, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:45.529349 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:45.449513 24648 leader_election.cc:290] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:45.453156 24692 raft_consensus.cc:493] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:45.530236 17763 raft_consensus.cc:2468] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 20.
I20260709 12:23:45.532902 18333 leader_election.cc:304] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:45.533351 24665 raft_consensus.cc:493] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:45.533782 24500 leader_election.cc:290] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:45.533944 24690 consensus_queue.cc:1048] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 9, Time since last communication: 0.190s
I20260709 12:23:45.534543 24675 consensus_queue.cc:1059] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 18.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.057s
I20260709 12:23:45.534858 24439 leader_election.cc:290] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:45.240725 17582 catalog_manager.cc:5697] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b reported cstate change: term changed from 9 to 12. New cstate: current_term: 12 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: HEALTHY } } }
I20260709 12:23:45.535485 17727 leader_election.cc:304] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
W20260709 12:23:45.538646 24490 consensus_meta.cc:287] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 1.259s	user 0.003s	sys 0.007s
I20260709 12:23:45.539685 24676 raft_consensus.cc:2804] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:23:45.541101 17681 raft_consensus.cc:1275] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 18 index: 14. Preceding OpId from leader: term: 21 index: 15. (index mismatch)
I20260709 12:23:45.469069 24668 raft_consensus.cc:493] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:45.546056 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:45.546942 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:45.551647 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:45.557085 17687 raft_consensus.cc:1275] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 16 index: 12. Preceding OpId from leader: term: 19 index: 14. (index mismatch)
I20260709 12:23:45.518335 24644 raft_consensus.cc:697] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 20 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:23:45.607021 24704 consensus_queue.cc:1048] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20260709 12:23:45.607527 24644 consensus_queue.cc:237] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.11, Last appended by leader: 11, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:45.522197 24524 leader_election.cc:290] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:45.567245 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:45.609059 18374 raft_consensus.cc:1240] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 14. Current term is 15. Ops: []
I20260709 12:23:45.610857 18374 raft_consensus.cc:1275] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 14 index: 14. Preceding OpId from leader: term: 16 index: 15. (index mismatch)
I20260709 12:23:45.440527 24685 raft_consensus.cc:740] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 18 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:23:45.565515 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 17 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:45.575320 18362 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59902 (request call id 1639) took 2068 ms. Trace:
W20260709 12:23:45.558334 24670 log.cc:927] Time spent T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.511s	user 0.000s	sys 0.001s
W20260709 12:23:45.582960 24244 consensus_meta.cc:287] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 1.507s	user 0.014s	sys 0.000s
I20260709 12:23:45.562500 24692 raft_consensus.cc:515] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:45.585481 24660 consensus_queue.cc:1048] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 12:23:45.449694 24698 consensus_queue.cc:1059] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 19.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.124s
I20260709 12:23:45.519970 24556 raft_consensus.cc:515] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:45.592667 24673 consensus_queue.cc:1059] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 14.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.114s
I20260709 12:23:45.593688 24705 raft_consensus.cc:493] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:23:45.568354 24500 raft_consensus.cc:493] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:45.522733 24663 raft_consensus.cc:515] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:23:45.563020 24527 consensus_meta.cc:287] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 1.003s	user 0.006s	sys 0.006s
I20260709 12:23:45.597894 24672 leader_election.cc:290] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:23:45.603435 23838 consensus_meta.cc:287] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 1.640s	user 0.008s	sys 0.007s
W20260709 12:23:45.604672 24678 log.cc:927] Time spent T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.277s	user 0.000s	sys 0.001s
I20260709 12:23:45.529234 24394 consensus_queue.cc:1048] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.054s
I20260709 12:23:45.605321 18365 raft_consensus.cc:1240] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 20. Current term is 21. Ops: [20.14-20.14]
I20260709 12:23:45.562063 24702 raft_consensus.cc:2749] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
W20260709 12:23:45.605952 24686 log.cc:927] Time spent T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.400s	user 0.001s	sys 0.000s
I20260709 12:23:45.606120 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 16 candidate_status { last_received { term: 14 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:45.606369 24643 raft_consensus.cc:2749] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 12:23:45.703136 24666 raft_consensus.cc:493] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:45.445943 24699 consensus_queue.cc:1048] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.140s
I20260709 12:23:45.704195 17762 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36572 (request call id 1208) took 1118 ms. Trace:
I20260709 12:23:45.704017 24666 raft_consensus.cc:515] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:45.562453 24417 raft_consensus.cc:2804] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:23:45.705370 24417 raft_consensus.cc:493] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:45.704591 17762 rpcz_store.cc:276] 0709 12:23:44.585965 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:44.586107 (+   142us) service_pool.cc:224] Handling call
0709 12:23:44.624485 (+ 38378us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:44.723034 (+ 98549us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:44.723069 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:44.824615 (+101546us) log.cc:844] Serialized 26 byte log entry
0709 12:23:45.006129 (+181514us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:23:45.006172 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:45.006246 (+    74us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:45.703948 (+697702us) raft_consensus.cc:1685] finished
0709 12:23:45.703970 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:45.704114 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":69968,"raft.run_cpu_time_us":125,"raft.run_wall_time_us":10022,"thread_start_us":162659,"threads_started":2,"wal-append.queue_time_us":205498}
I20260709 12:23:45.705773 24417 raft_consensus.cc:3060] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:45.706483 24666 leader_election.cc:290] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:45.706552 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:45.707465 17688 raft_consensus.cc:2393] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 18: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:23:45.626298 17760 raft_consensus.cc:2393] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 21: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:23:45.562638 24649 raft_consensus.cc:493] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:23:45.708556 24649 raft_consensus.cc:515] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:45.709550 24556 leader_election.cc:290] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:45.710618 24649 leader_election.cc:290] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:45.522163 24656 consensus_queue.cc:237] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.10, Last appended by leader: 10, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:45.711130 24688 consensus_queue.cc:1059] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 19.11, Next index: 12, Last known committed idx: 10, Time since last communication: 0.033s
I20260709 12:23:45.712225 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:45.712841 17686 raft_consensus.cc:3060] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Advancing to term 20
W20260709 12:23:45.641108 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 105705us
W20260709 12:23:45.643235 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 404585us
I20260709 12:23:45.715226 17728 leader_election.cc:304] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:23:45.715464 17727 leader_election.cc:304] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:45.716138 24721 raft_consensus.cc:493] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:45.618172 18362 rpcz_store.cc:276] 0709 12:23:43.506795 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:43.594949 (+ 88154us) service_pool.cc:224] Handling call
0709 12:23:43.595300 (+   351us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:43.595862 (+   562us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 12:23:44.174118 (+578256us) log.cc:844] Serialized 14 byte log entry
0709 12:23:44.593311 (+419193us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:44.647325 (+ 54014us) log.cc:844] Serialized 26 byte log entry
0709 12:23:44.649399 (+  2074us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:23:44.649455 (+    56us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:44.649547 (+    92us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:45.575049 (+925502us) raft_consensus.cc:1685] finished
0709 12:23:45.575077 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:45.575239 (+   162us) inbound_call.cc:177] Queueing success response
0709 12:23:45.585192 (+  9953us) spinlock_profiling.cc:243] Waited 34.2 ms on lock 0x7b4800220a58. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4f549aa1 00007f7b4f540b37 00007f7b4f5409ca 00007f7b4f63826f 00007f7b4f637f37 00007f7b4f64c55c 00007f7b4f64c508 00007f7b4f64c488 00007f7b4f64c443 00007f7b4f64b6ec 00007f7b4e575a96
Metrics: {"mutex_wait_us":2764,"prepare.queue_time_us":776585,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":38,"raft.queue_time_us":360297,"raft.run_cpu_time_us":9650,"raft.run_wall_time_us":770120,"spinlock_wait_cycles":78712704,"thread_start_us":835022,"threads_started":3,"wal-append.queue_time_us":375623}
W20260709 12:23:45.650658 24428 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.596s	user 0.002s	sys 0.011s
I20260709 12:23:45.652110 17652 leader_election.cc:304] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:23:45.715680 24437 consensus_meta.cc:287] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 1.048s	user 0.008s	sys 0.002s
I20260709 12:23:45.478938 24682 raft_consensus.cc:515] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:45.517980 24655 raft_consensus.cc:515] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:45.636847 24392 consensus_meta.cc:287] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.845s	user 0.012s	sys 0.000s
I20260709 12:23:45.671043 17761 raft_consensus.cc:2410] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for term 17 because replica has last-logged OpId of term: 15 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 11.
I20260709 12:23:45.631235 24439 raft_consensus.cc:2749] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:23:45.638090 24250 raft_consensus.cc:2749] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:23:45.673296 24527 raft_consensus.cc:515] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:45.636572 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:23:45.651638 24434 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.624s	user 0.006s	sys 0.007s
W20260709 12:23:45.692587 17654 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:45.745057 17654 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:45.677325 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:23:45.695096 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 162262us
I20260709 12:23:45.748847 17682 raft_consensus.cc:2393] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 15: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:23:45.750332 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:23:45.717422 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 65449.9us
I20260709 12:23:45.634531 24244 raft_consensus.cc:515] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:45.640280 24676 raft_consensus.cc:697] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:23:45.771634 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 21 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:45.740794 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 18 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:45.741312 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
W20260709 12:23:45.672526 24211 consensus_meta.cc:287] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 1.393s	user 0.013s	sys 0.002s
I20260709 12:23:45.742100 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
W20260709 12:23:45.772629 18371 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.623s	user 0.022s	sys 0.031s
I20260709 12:23:45.772892 17829 raft_consensus.cc:3060] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:45.772738 17836 raft_consensus.cc:3060] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:23:45.773578 18371 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.624s	user 0.022s	sys 0.031s
W20260709 12:23:45.674034 24691 log.cc:927] Time spent T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.468s	user 0.000s	sys 0.001s
I20260709 12:23:45.751339 17652 leader_election.cc:304] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:45.566347 24665 raft_consensus.cc:515] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:45.725402 24428 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.670s	user 0.002s	sys 0.011s
I20260709 12:23:45.782743 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 15 candidate_status { last_received { term: 14 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:45.785420 17834 raft_consensus.cc:2410] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for term 21 because replica has last-logged OpId of term: 19 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 11.
I20260709 12:23:45.657809 24705 raft_consensus.cc:515] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:45.762794 24680 consensus_queue.cc:1048] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.046s
I20260709 12:23:45.763257 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:45.763459 24695 raft_consensus.cc:493] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:23:45.763947 24696 consensus_queue.cc:1059] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 13.11, Next index: 12, Last known committed idx: 8, Time since last communication: 0.034s
I20260709 12:23:45.768013 24701 raft_consensus.cc:2804] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:23:45.768224 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 20 candidate_status { last_received { term: 18 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:45.769819 24675 consensus_queue.cc:1059] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:23:45.775627 17686 raft_consensus.cc:2468] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 20.
I20260709 12:23:45.616021 24668 raft_consensus.cc:515] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:45.826663 17683 raft_consensus.cc:1240] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 16. Current term is 17. Ops: []
I20260709 12:23:45.783982 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 21 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
W20260709 12:23:45.753963 24434 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.734s	user 0.007s	sys 0.007s
I20260709 12:23:45.730938 24556 consensus_queue.cc:1048] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.060s
I20260709 12:23:45.788417 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 16 candidate_status { last_received { term: 14 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:45.834915 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:45.835695 17835 raft_consensus.cc:2468] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 14.
I20260709 12:23:45.835840 17686 raft_consensus.cc:2468] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 21.
I20260709 12:23:45.789309 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:45.633174 24579 consensus_queue.cc:237] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 18.16, Last appended by leader: 16, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:45.793087 17838 raft_consensus.cc:1275] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
I20260709 12:23:45.666734 24500 raft_consensus.cc:515] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:45.794502 18364 raft_consensus.cc:2393] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 16: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:23:45.794539 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 20 candidate_status { last_received { term: 18 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:45.751359 24649 raft_consensus.cc:493] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:45.751228 24524 raft_consensus.cc:2749] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:23:45.831780 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:45.783257 24436 raft_consensus.cc:493] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:23:45.846331 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:45.847579 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 16 candidate_status { last_received { term: 14 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:45.850669 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:45.857993 17830 raft_consensus.cc:2468] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 14.
I20260709 12:23:45.858083 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:45.858541 17759 raft_consensus.cc:2393] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 20: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:23:45.859200 17830 raft_consensus.cc:1240] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 17 LEADER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 16. Current term is 17. Ops: []
I20260709 12:23:45.751844 24392 raft_consensus.cc:515] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:45.858695 17837 raft_consensus.cc:2376] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 15. Re-sending same reply.
I20260709 12:23:45.771230 24698 consensus_queue.cc:1048] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.080s
W20260709 12:23:45.845078 24642 consensus_meta.cc:287] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 1.169s	user 0.013s	sys 0.001s
I20260709 12:23:45.866132 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:45.879141 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:45.879669 17835 raft_consensus.cc:2468] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 20.
W20260709 12:23:45.872466 18335 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.521s	user 0.044s	sys 0.084s
I20260709 12:23:45.881294 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:45.753257 24313 consensus_queue.cc:1059] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 17.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.077s
I20260709 12:23:45.879989 17838 raft_consensus.cc:2468] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 14.
I20260709 12:23:45.810549 24701 raft_consensus.cc:493] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:23:45.730685 24721 raft_consensus.cc:515] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:23:45.859097 24227 consensus_meta.cc:287] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.754s	user 0.006s	sys 0.006s
W20260709 12:23:45.876992 24443 consensus_meta.cc:287] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 1.232s	user 0.008s	sys 0.004s
W20260709 12:23:45.881827 18335 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.532s	user 0.044s	sys 0.084s
I20260709 12:23:45.881953 17759 raft_consensus.cc:1275] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
I20260709 12:23:45.882942 17803 leader_election.cc:304] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
W20260709 12:23:45.858632 24650 consensus_meta.cc:287] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.727s	user 0.005s	sys 0.006s
W20260709 12:23:45.886415 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 135147us
I20260709 12:23:45.936371 17829 raft_consensus.cc:2468] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 15.
I20260709 12:23:45.937076 17652 leader_election.cc:304] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:45.887461 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:45.782879 24211 raft_consensus.cc:515] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:45.782721 24702 consensus_queue.cc:1059] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.099s
I20260709 12:23:45.927002 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:45.946337 17686 raft_consensus.cc:2468] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 19.
W20260709 12:23:45.947666 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 64824.8us
I20260709 12:23:45.927855 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:45.873456 18333 leader_election.cc:304] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:45.931489 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:45.863232 24723 raft_consensus.cc:3060] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:45.885254 24672 raft_consensus.cc:493] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:45.860337 24649 raft_consensus.cc:515] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:45.959605 24672 raft_consensus.cc:515] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:45.926642 24701 raft_consensus.cc:3060] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:45.961838 24672 leader_election.cc:290] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:45.940172 24227 raft_consensus.cc:515] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:45.870833 24428 raft_consensus.cc:2804] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:23:45.941339 17837 raft_consensus.cc:1240] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 14. Current term is 15. Ops: []
I20260709 12:23:45.952212 17680 raft_consensus.cc:2393] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 20: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:23:45.876192 24695 raft_consensus.cc:515] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:45.842551 24659 raft_consensus.cc:515] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:45.892532 24715 raft_consensus.cc:493] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:23:45.868144 24436 raft_consensus.cc:515] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:45.964874 17833 raft_consensus.cc:2393] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 21: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
W20260709 12:23:45.984829 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 111486us
I20260709 12:23:45.986433 18333 leader_election.cc:304] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:45.942869 17652 leader_election.cc:304] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:45.966185 17762 raft_consensus.cc:2468] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 15.
W20260709 12:23:45.968123 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:45.751632 24685 consensus_queue.cc:260] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.12, Last appended by leader: 12, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:45.885921 24484 raft_consensus.cc:493] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:45.974236 18368 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59886 (request call id 1853) took 1388 ms. Trace:
W20260709 12:23:45.976930 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 207906us
I20260709 12:23:45.859273 24471 raft_consensus.cc:515] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:45.979844 24700 log.cc:927] Time spent T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.074s	user 0.000s	sys 0.000s
I20260709 12:23:45.980149 24697 consensus_queue.cc:1048] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.247s
I20260709 12:23:45.980979 24708 raft_consensus.cc:3055] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:23:45.996671 17832 raft_consensus.cc:1275] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 14 index: 14. Preceding OpId from leader: term: 16 index: 15. (index mismatch)
I20260709 12:23:45.993345 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:45.977480 24250 raft_consensus.cc:2749] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 12:23:45.951315 24675 raft_consensus.cc:3055] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:23:45.971329 24394 raft_consensus.cc:493] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:45.992810 24661 consensus_meta.cc:287] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.903s	user 0.012s	sys 0.001s
I20260709 12:23:45.999256 17830 raft_consensus.cc:2468] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 16.
I20260709 12:23:45.985245 17728 leader_election.cc:304] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:45.990108 24437 raft_consensus.cc:2749] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:23:45.886857 24642 raft_consensus.cc:515] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:46.000720 17686 raft_consensus.cc:2468] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 16.
I20260709 12:23:45.968703 24644 raft_consensus.cc:3060] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:23:45.999289 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 56517.5us
I20260709 12:23:46.032303 24729 raft_consensus.cc:493] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:45.947842 24443 raft_consensus.cc:515] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:46.004230 24673 raft_consensus.cc:493] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:46.033185 18371 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 2028) took 1077 ms. Trace:
W20260709 12:23:46.033248 24689 log.cc:927] Time spent T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.207s	user 0.000s	sys 0.001s
I20260709 12:23:46.005120 24436 leader_election.cc:290] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:46.006139 24663 leader_election.cc:290] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:46.006501 24668 leader_election.cc:290] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:46.007333 24655 leader_election.cc:290] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:46.033569 24673 raft_consensus.cc:515] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:45.991422 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:46.035019 17680 raft_consensus.cc:2393] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 16: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:23:46.005282 24692 leader_election.cc:290] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:46.008306 24313 raft_consensus.cc:3055] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:23:45.965878 24676 consensus_queue.cc:237] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.10, Last appended by leader: 10, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:46.035616 24313 raft_consensus.cc:740] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 15 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:23:46.035965 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 14 candidate_status { last_received { term: 13 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:46.036525 24313 consensus_queue.cc:260] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.11, Last appended by leader: 11, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:46.037340 17680 raft_consensus.cc:1275] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 20 index: 14. Preceding OpId from leader: term: 20 index: 15. (index mismatch)
I20260709 12:23:46.037675 24313 raft_consensus.cc:3060] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:45.996847 24715 raft_consensus.cc:515] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:46.038975 17680 raft_consensus.cc:1240] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 19. Current term is 20. Ops: [15.10-19.12]
W20260709 12:23:46.025115 24722 log.cc:927] Time spent T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.160s	user 0.001s	sys 0.000s
I20260709 12:23:46.024605 24682 leader_election.cc:290] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:46.026727 24717 raft_consensus.cc:493] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:46.040493 24471 leader_election.cc:290] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 24 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:46.041064 24500 leader_election.cc:290] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:46.040838 24717 raft_consensus.cc:515] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:46.008277 24527 leader_election.cc:290] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 16 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:46.042248 18367 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 2023) took 1144 ms. Trace:
I20260709 12:23:46.042520 24697 raft_consensus.cc:2804] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:23:45.987789 18368 rpcz_store.cc:276] 0709 12:23:44.585822 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:44.586504 (+   682us) service_pool.cc:224] Handling call
0709 12:23:44.615195 (+ 28691us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:23:44.760140 (+144945us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:44.895971 (+135831us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:23:44.939829 (+ 43858us) log.cc:844] Serialized 52 byte log entry
0709 12:23:45.046612 (+106783us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:23:45.046743 (+   131us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:45.047196 (+   453us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:45.974032 (+926836us) raft_consensus.cc:1685] finished
0709 12:23:45.974051 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:45.974176 (+   125us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":19447,"raft.queue_time_us":344757,"raft.run_cpu_time_us":169,"raft.run_wall_time_us":23490,"thread_start_us":177082,"threads_started":2,"wal-append.queue_time_us":159439}
I20260709 12:23:46.021185 24394 raft_consensus.cc:515] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:46.044013 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:46.042706 18367 rpcz_store.cc:276] 0709 12:23:44.898320 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:44.933452 (+ 35132us) service_pool.cc:224] Handling call
0709 12:23:44.955508 (+ 22056us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:23:44.975175 (+ 19667us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 12:23:45.017469 (+ 42294us) log.cc:844] Serialized 14 byte log entry
0709 12:23:45.843104 (+825635us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:23:45.861655 (+ 18551us) log.cc:844] Serialized 52 byte log entry
0709 12:23:45.861878 (+   223us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:23:45.861927 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:45.862011 (+    84us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:46.042007 (+179996us) raft_consensus.cc:1685] finished
0709 12:23:46.042039 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:46.042184 (+   145us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":40174,"prepare.queue_time_us":36730,"prepare.run_cpu_time_us":85,"prepare.run_wall_time_us":6305,"raft.queue_time_us":263418,"raft.run_cpu_time_us":231,"raft.run_wall_time_us":8250,"thread_start_us":714561,"threads_started":2,"wal-append.queue_time_us":554066}
I20260709 12:23:46.045265 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:46.045596 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 17 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:45.998857 24708 raft_consensus.cc:740] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:23:46.046281 18367 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 16.
I20260709 12:23:46.046404 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 14 candidate_status { last_received { term: 13 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:46.047608 18367 raft_consensus.cc:1275] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 16 index: 13. Preceding OpId from leader: term: 17 index: 13. (term mismatch)
W20260709 12:23:46.028678 24725 log.cc:927] Time spent T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.078s	user 0.000s	sys 0.000s
I20260709 12:23:46.048066 18367 pending_rounds.cc:85] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da: Aborting all ops after (but not including) 12
I20260709 12:23:46.048322 18367 pending_rounds.cc:107] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da: Aborting uncommitted NO_OP operation due to leader change: 16.13
I20260709 12:23:46.027433 17727 leader_election.cc:304] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:46.033046 24729 raft_consensus.cc:515] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:45.947458 24650 raft_consensus.cc:515] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:46.033598 18371 rpcz_store.cc:276] 0709 12:23:44.955462 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:45.034651 (+ 79189us) service_pool.cc:224] Handling call
0709 12:23:45.051492 (+ 16841us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:23:45.059603 (+  8111us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:45.059653 (+    50us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:23:45.081307 (+ 21654us) log.cc:844] Serialized 52 byte log entry
0709 12:23:45.773975 (+692668us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:23:45.774012 (+    37us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:45.774073 (+    61us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:46.032955 (+258882us) raft_consensus.cc:1685] finished
0709 12:23:46.032976 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:46.033124 (+   148us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":7478,"raft.queue_time_us":12614,"raft.run_cpu_time_us":161,"raft.run_wall_time_us":20319,"thread_start_us":623901,"threads_started":1,"wal-append.queue_time_us":852376}
I20260709 12:23:46.039750 24417 raft_consensus.cc:515] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:46.026151 24660 raft_consensus.cc:2749] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 12:23:45.996609 24702 consensus_queue.cc:1243] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer e063541542a94969898868bf6eb5ce1b log is divergent from this leader: its last log entry 18.11 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 10
I20260709 12:23:46.043133 24697 raft_consensus.cc:493] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:46.048439 24673 leader_election.cc:290] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:46.047072 24708 consensus_queue.cc:260] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.10, Last appended by leader: 10, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:46.053711 24490 raft_consensus.cc:3055] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:23:45.976339 24428 raft_consensus.cc:493] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:23:46.054162 24490 raft_consensus.cc:740] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:23:46.054503 24428 raft_consensus.cc:3060] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:46.051774 24715 leader_election.cc:290] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:46.027273 24695 leader_election.cc:290] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:46.054222 24729 leader_election.cc:290] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:46.055019 24702 raft_consensus.cc:2749] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:23:46.055977 17689 raft_consensus.cc:1240] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 17. Current term is 18. Ops: []
I20260709 12:23:46.048588 18367 raft_consensus.cc:2889] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:46.048626 18333 leader_election.cc:304] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:23:46.057159 18361 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59886 (request call id 1852) took 1714 ms. Trace:
I20260709 12:23:46.057582 18361 rpcz_store.cc:276] 0709 12:23:44.342669 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:44.349231 (+  6562us) service_pool.cc:224] Handling call
0709 12:23:44.349656 (+   425us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:45.520685 (+1171029us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:46.057001 (+536316us) spinlock_profiling.cc:243] Waited 1.17 s on lock 0x7b5400570154. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:23:46.057114 (+   113us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2688162688}
I20260709 12:23:46.031088 24642 leader_election.cc:290] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 20 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:46.009968 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:46.068760 17835 raft_consensus.cc:1240] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 18. Current term is 19. Ops: []
I20260709 12:23:46.069188 18364 raft_consensus.cc:2393] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 18: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:23:46.067973 24650 leader_election.cc:290] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 19 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:46.049326 24661 raft_consensus.cc:515] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:46.070796 17756 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36572 (request call id 1199) took 2448 ms (client timeout 1547 ms). Trace:
W20260709 12:23:46.071228 17756 rpcz_store.cc:269] 0709 12:23:43.622793 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:43.622910 (+   117us) service_pool.cc:224] Handling call
0709 12:23:46.070561 (+2447651us) spinlock_profiling.cc:243] Waited 1.29 s on lock 0x7b54002b5554. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:23:46.070683 (+   122us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2974538112}
I20260709 12:23:46.072432 24661 leader_election.cc:290] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 15 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:46.073048 17764 raft_consensus.cc:1240] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 16. Current term is 17. Ops: [16.13-16.13]
I20260709 12:23:46.073282 17685 raft_consensus.cc:2393] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 15: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:23:46.073735 24680 consensus_queue.cc:1048] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 12:23:46.074241 24417 leader_election.cc:290] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 17 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:46.053941 24708 raft_consensus.cc:3060] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:46.059157 24211 leader_election.cc:290] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 23 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:46.002465 24675 raft_consensus.cc:740] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 17 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:23:46.055276 24490 consensus_queue.cc:260] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 19.11, Last appended by leader: 11, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:46.076956 23906 raft_consensus.cc:3060] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:46.077047 24490 raft_consensus.cc:3060] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:46.045836 24394 leader_election.cc:290] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:46.080293 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:46.002604 24733 raft_consensus.cc:2804] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:23:46.081647 24733 raft_consensus.cc:493] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:23:46.082052 24733 raft_consensus.cc:3060] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:46.048879 24659 leader_election.cc:290] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 18 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:46.085577 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 20 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:46.087347 24711 raft_consensus.cc:493] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:23:46.065413 24227 leader_election.cc:290] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 19 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:46.065907 24443 leader_election.cc:290] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 18 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:46.067978 24697 raft_consensus.cc:3060] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:23:46.050982 24518 leader_election.cc:290] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:46.088953 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:46.091298 17762 raft_consensus.cc:2468] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 17.
I20260709 12:23:46.091482 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:46.050726 24717 leader_election.cc:290] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:46.096779 17836 raft_consensus.cc:2468] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 18.
W20260709 12:23:46.105945 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 57414.8us
I20260709 12:23:46.111586 17803 leader_election.cc:304] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:23:46.113888 17728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:46.118643 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:46.119571 18367 raft_consensus.cc:2468] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 20.
I20260709 12:23:46.121409 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:46.122448 17686 raft_consensus.cc:2410] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for term 20 because replica has last-logged OpId of term: 18 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 12.
I20260709 12:23:46.123503 17756 raft_consensus.cc:2410] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 20 because replica has last-logged OpId of term: 18 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 10.
I20260709 12:23:46.126618 18334 leader_election.cc:304] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:46.127187 17803 leader_election.cc:304] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:46.130759 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:46.131057 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 17 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:46.136849 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:46.137791 18365 raft_consensus.cc:2393] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 17: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:23:46.138343 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:46.132019 17832 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 16.
I20260709 12:23:46.002024 24484 raft_consensus.cc:515] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:46.130625 24673 raft_consensus.cc:493] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:46.104830 24659 raft_consensus.cc:493] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:23:46.151671 17803 leader_election.cc:304] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
W20260709 12:23:46.146188 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 113277us
I20260709 12:23:46.139003 18367 raft_consensus.cc:2468] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 17.
I20260709 12:23:46.149897 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 14 candidate_status { last_received { term: 13 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:46.129139 24394 raft_consensus.cc:2804] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:23:46.153736 17832 raft_consensus.cc:2468] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 13.
I20260709 12:23:46.153576 17652 leader_election.cc:304] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:23:46.132999 24682 raft_consensus.cc:2749] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:23:46.156250 17652 leader_election.cc:304] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:23:46.157608 17803 leader_election.cc:304] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:23:46.158677 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 16 candidate_status { last_received { term: 14 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
W20260709 12:23:46.151196 24639 consensus_meta.cc:287] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.826s	user 0.009s	sys 0.003s
I20260709 12:23:46.159449 18333 leader_election.cc:304] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:23:46.165201 24654 raft_consensus.cc:2804] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:23:46.167718 24654 raft_consensus.cc:493] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:46.169245 17652 leader_election.cc:304] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:46.168396 24654 raft_consensus.cc:3060] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:46.157289 24655 raft_consensus.cc:2804] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:23:46.155658 24394 raft_consensus.cc:493] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:23:46.180219 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 123699us
I20260709 12:23:46.176043 24709 raft_consensus.cc:2749] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:23:46.182021 24639 raft_consensus.cc:2804] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:23:46.182260 24672 consensus_queue.cc:1048] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:23:46.183750 17652 leader_election.cc:304] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:23:46.163509 24656 raft_consensus.cc:2804] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:23:46.187740 24654 raft_consensus.cc:515] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:46.200201 24705 leader_election.cc:290] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:46.159268 24443 raft_consensus.cc:2804] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:23:46.157248 24588 consensus_queue.cc:1048] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.059s
I20260709 12:23:46.234200 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:46.154227 24673 raft_consensus.cc:515] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:46.214468 24655 raft_consensus.cc:493] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:46.199397 24428 raft_consensus.cc:515] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:23:46.156890 24659 raft_consensus.cc:515] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:46.231072 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 17 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:46.239015 17829 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 16.
I20260709 12:23:46.227239 24639 raft_consensus.cc:697] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 15 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:23:46.217263 24394 raft_consensus.cc:3060] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:46.231287 24701 raft_consensus.cc:515] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:46.243903 24639 consensus_queue.cc:237] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:46.251296 24669 raft_consensus.cc:493] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:46.252267 24701 leader_election.cc:290] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 20 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:46.223058 24656 raft_consensus.cc:493] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:23:46.253012 24656 raft_consensus.cc:3060] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:46.253552 24673 leader_election.cc:290] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:46.254278 24669 raft_consensus.cc:515] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:46.228636 24443 raft_consensus.cc:493] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:46.255290 24443 raft_consensus.cc:3060] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:46.257143 24669 leader_election.cc:290] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:46.215668 24717 raft_consensus.cc:493] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:46.261361 24428 leader_election.cc:290] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 19 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:46.261389 17680 raft_consensus.cc:1240] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 18. Current term is 19. Ops: [18.11-18.11]
I20260709 12:23:46.254287 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:46.262880 17829 raft_consensus.cc:2468] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 19.
I20260709 12:23:46.263859 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:46.265273 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:46.260277 24717 raft_consensus.cc:515] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:46.266348 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:46.266505 24441 leader_election.cc:290] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 15 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:46.234984 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 17 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:46.267716 17759 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 16.
I20260709 12:23:46.268625 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:46.090245 24711 raft_consensus.cc:515] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:46.269101 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:46.238008 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:46.270546 17755 raft_consensus.cc:2393] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 19: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:23:46.270242 17838 raft_consensus.cc:2468] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 20.
I20260709 12:23:46.271485 24711 leader_election.cc:290] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:46.227159 24729 raft_consensus.cc:2804] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:23:46.240203 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 21 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:46.240712 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:46.076488 24675 consensus_queue.cc:260] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 17.10, Last appended by leader: 10, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:46.272368 17763 raft_consensus.cc:2468] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 20.
I20260709 12:23:46.272466 17762 raft_consensus.cc:2468] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 20.
I20260709 12:23:46.272811 24675 raft_consensus.cc:3060] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:46.273370 24687 raft_consensus.cc:493] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:46.273859 24687 raft_consensus.cc:515] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:46.238494 24655 raft_consensus.cc:3060] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:46.276635 24687 leader_election.cc:290] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:46.248410 17652 leader_election.cc:304] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:23:46.277096 24654 leader_election.cc:290] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 14 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:23:46.279331 24738 log.cc:927] Time spent T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.065s	user 0.000s	sys 0.000s
I20260709 12:23:46.281783 24659 leader_election.cc:290] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:46.288000 17801 leader_election.cc:304] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:46.288373 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:46.283309 24733 raft_consensus.cc:515] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:46.305289 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 57034.5us
I20260709 12:23:46.306653 17801 leader_election.cc:304] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:46.314968 17683 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37402 (request call id 3167) took 1125 ms. Trace:
I20260709 12:23:46.325376 17801 leader_election.cc:304] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:46.315658 17683 rpcz_store.cc:276] 0709 12:23:45.189580 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:45.189729 (+   149us) service_pool.cc:224] Handling call
0709 12:23:45.236477 (+ 46748us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:45.830296 (+593819us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:46.314620 (+484324us) spinlock_profiling.cc:243] Waited 557 ms on lock 0x7b54000c4c54. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:23:46.314881 (+   261us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1281651968}
I20260709 12:23:46.392592 23740 raft_consensus.cc:2804] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:23:46.393246 23740 raft_consensus.cc:493] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:23:46.393682 23740 raft_consensus.cc:3060] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Advancing to term 17
W20260709 12:23:46.393869 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 78703.8us
W20260709 12:23:46.392652 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 67413.1us
I20260709 12:23:46.398670 17801 leader_election.cc:304] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:46.382395 24736 raft_consensus.cc:2804] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:23:46.411212 24717 leader_election.cc:290] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:46.412797 24484 leader_election.cc:290] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:23:46.414286 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 129598us
I20260709 12:23:46.304456 24709 raft_consensus.cc:2804] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:23:46.432475 17652 leader_election.cc:304] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:46.305830 24642 raft_consensus.cc:2749] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:23:46.415331 24588 raft_consensus.cc:2804] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:23:46.438445 24588 raft_consensus.cc:493] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:23:46.438835 24588 raft_consensus.cc:3060] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:46.438917 24227 raft_consensus.cc:2804] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:23:46.439584 24227 raft_consensus.cc:493] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:46.439694 24484 raft_consensus.cc:2749] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:23:46.450737 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:46.320611 24428 raft_consensus.cc:2804] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:23:46.433683 17652 leader_election.cc:304] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:46.304575 24723 raft_consensus.cc:2749] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:23:46.409401 24736 raft_consensus.cc:493] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:23:46.304761 24729 raft_consensus.cc:697] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 18 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:23:46.457754 17727 leader_election.cc:304] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:23:46.452168 17838 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 15.
W20260709 12:23:46.514847 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 57187.3us
I20260709 12:23:46.416833 24705 raft_consensus.cc:493] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
W20260709 12:23:46.443187 24658 log.cc:927] Time spent T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.060s	user 0.000s	sys 0.000s
I20260709 12:23:46.440120 24227 raft_consensus.cc:3060] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:46.457366 17652 leader_election.cc:304] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:46.541463 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
W20260709 12:23:46.546085 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 88844.6us
I20260709 12:23:46.544260 24705 raft_consensus.cc:515] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:46.546907 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:46.548100 17762 raft_consensus.cc:3055] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:23:46.550915 17762 raft_consensus.cc:740] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:23:46.320078 24682 consensus_queue.cc:1048] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.137s
I20260709 12:23:46.548517 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 21 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:46.549186 17652 leader_election.cc:304] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:23:46.543993 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 257080us
I20260709 12:23:46.570600 18363 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59886 (request call id 1845) took 3047 ms. Trace:
I20260709 12:23:46.570832 24662 leader_election.cc:290] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:46.456812 24428 raft_consensus.cc:493] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:23:46.547447 24708 raft_consensus.cc:2804] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:23:46.455801 24709 raft_consensus.cc:697] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 18 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:23:46.467793 24638 raft_consensus.cc:2804] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:23:46.547500 24313 raft_consensus.cc:2804] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:23:46.433310 24518 consensus_queue.cc:1059] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 14.12, Next index: 13, Last known committed idx: 10, Time since last communication: 0.109s
I20260709 12:23:46.685096 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:46.471055 24736 raft_consensus.cc:3060] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Advancing to term 21
W20260709 12:23:46.612223 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 63131.3us
I20260709 12:23:46.439407 24697 raft_consensus.cc:515] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:46.630292 18363 rpcz_store.cc:276] 0709 12:23:43.522702 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:43.816902 (+294200us) service_pool.cc:224] Handling call
0709 12:23:43.838570 (+ 21668us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:45.459353 (+1620783us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:46.570230 (+1110877us) spinlock_profiling.cc:243] Waited 1.6 s on lock 0x7b54005730d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:23:46.570527 (+   297us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3686788992}
I20260709 12:23:46.677356 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:46.688819 17836 raft_consensus.cc:2468] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 18.
I20260709 12:23:46.679687 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:46.692068 24729 consensus_queue.cc:237] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.10, Last appended by leader: 10, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:46.696172 24723 raft_consensus.cc:493] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:46.700218 24707 raft_consensus.cc:493] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:46.700666 24733 leader_election.cc:290] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 14 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:46.700767 24707 raft_consensus.cc:515] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:46.702185 24313 raft_consensus.cc:697] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:23:46.703111 24313 consensus_queue.cc:237] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.10, Last appended by leader: 10, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:46.705269 24697 leader_election.cc:290] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 22 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:46.628479 24647 raft_consensus.cc:2804] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:23:46.712236 24647 raft_consensus.cc:493] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:46.712793 24647 raft_consensus.cc:3060] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:46.717080 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 14 candidate_status { last_received { term: 13 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:46.717890 17836 raft_consensus.cc:3060] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:46.726305 24443 raft_consensus.cc:515] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:46.681123 24708 raft_consensus.cc:493] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:46.728580 24708 raft_consensus.cc:3060] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:46.729914 24709 consensus_queue.cc:237] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 17.14, Last appended by leader: 14, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:46.730613 24443 leader_election.cc:290] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 16 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:46.732457 24443 raft_consensus.cc:493] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:46.736150 24443 raft_consensus.cc:515] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:46.682832 24428 raft_consensus.cc:3060] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:46.748690 23740 raft_consensus.cc:515] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:46.682338 24638 raft_consensus.cc:493] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:46.747334 17836 raft_consensus.cc:2468] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 14.
I20260709 12:23:46.811061 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:46.703060 24707 leader_election.cc:290] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:23:46.746582 24739 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.503s	user 0.001s	sys 0.026s
W20260709 12:23:46.846498 18364 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.535s	user 0.019s	sys 0.021s
I20260709 12:23:46.812201 17838 raft_consensus.cc:2468] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 12.
W20260709 12:23:46.848037 18364 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.537s	user 0.020s	sys 0.021s
W20260709 12:23:46.843212 24655 consensus_meta.cc:287] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.567s	user 0.005s	sys 0.005s
I20260709 12:23:46.849280 24655 raft_consensus.cc:515] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:46.850342 24683 log.cc:927] Time spent T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 1.644s	user 0.009s	sys 0.004s
I20260709 12:23:46.851749 17838 raft_consensus.cc:1240] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 13. Current term is 14. Ops: []
I20260709 12:23:46.852380 23740 leader_election.cc:290] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 17 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:46.851816 24655 leader_election.cc:290] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 21 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:46.682464 24677 raft_consensus.cc:493] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:23:46.854650 24677 raft_consensus.cc:515] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:46.856762 24677 leader_election.cc:290] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:46.815305 24588 raft_consensus.cc:515] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:46.859800 24588 leader_election.cc:290] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 16 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:46.815562 24638 raft_consensus.cc:3060] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Advancing to term 21
W20260709 12:23:46.847841 24644 consensus_meta.cc:287] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.746s	user 0.011s	sys 0.000s
I20260709 12:23:46.739637 24708 raft_consensus.cc:515] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:46.808233 24662 raft_consensus.cc:493] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:46.875527 24227 raft_consensus.cc:515] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:23:46.879261 17757 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38786 (request call id 3441) took 1501 ms (client timeout 1674 ms). Trace:
W20260709 12:23:46.887064 17757 rpcz_store.cc:269] 0709 12:23:45.378332 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:45.380259 (+  1927us) service_pool.cc:224] Handling call
0709 12:23:46.879043 (+1498784us) spinlock_profiling.cc:243] Waited 737 ms on lock 0x7b54000fe6d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:23:46.879194 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1695999232}
I20260709 12:23:46.728539 24647 raft_consensus.cc:515] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:46.882682 24644 raft_consensus.cc:2749] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
W20260709 12:23:46.891752 24745 log.cc:927] Time spent T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.210s	user 0.001s	sys 0.000s
I20260709 12:23:46.699924 24723 raft_consensus.cc:515] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:46.808512 24394 raft_consensus.cc:515] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:46.902510 17832 raft_consensus.cc:3060] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:46.889072 24662 raft_consensus.cc:515] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:46.931814 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:46.933566 17757 raft_consensus.cc:2393] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 24: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:23:46.923861 24639 raft_consensus.cc:493] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:46.924695 24443 leader_election.cc:290] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:46.927102 24705 leader_election.cc:290] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:46.930639 24711 raft_consensus.cc:493] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:46.975708 24662 leader_election.cc:290] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:46.980211 24711 raft_consensus.cc:515] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:46.977063 17728 leader_election.cc:304] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:46.983942 17728 leader_election.cc:304] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:46.986349 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:46.987726 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:46.986474 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:46.988485 18363 raft_consensus.cc:2393] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 20: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:23:46.989513 24687 raft_consensus.cc:2804] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:23:46.989950 24687 raft_consensus.cc:493] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:46.990300 24687 raft_consensus.cc:3060] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:46.994553 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:47.000468 18361 raft_consensus.cc:2468] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 19.
I20260709 12:23:47.001716 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 21 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:47.002557 17757 raft_consensus.cc:3060] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:47.003410 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:47.005013 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:47.008987 18363 raft_consensus.cc:2393] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 15: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:23:47.010140 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 16 candidate_status { last_received { term: 14 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:47.011238 18362 raft_consensus.cc:2393] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 16: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:23:47.007827 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:47.015031 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:46.970624 24649 leader_election.cc:290] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:47.019093 18368 raft_consensus.cc:2368] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for earlier term 14. Current term is 15.
I20260709 12:23:47.019271 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:46.951700 24639 raft_consensus.cc:515] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:46.972563 24697 raft_consensus.cc:493] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:23:47.021235 17832 raft_consensus.cc:2468] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 16.
W20260709 12:23:47.022678 17654 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:47.024413 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:46.999768 24688 raft_consensus.cc:2804] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:23:47.032526 18362 raft_consensus.cc:2468] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 23.
W20260709 12:23:47.051182 17654 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:47.052223 24662 raft_consensus.cc:493] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:23:47.053491 24639 leader_election.cc:290] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:47.053318 24662 raft_consensus.cc:515] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:47.054234 24639 raft_consensus.cc:493] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:47.057561 24227 leader_election.cc:290] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 21 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:47.058337 24662 leader_election.cc:290] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:23:47.058640 24752 log.cc:927] Time spent T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.054s	user 0.000s	sys 0.000s
I20260709 12:23:47.059482 24723 leader_election.cc:290] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:47.061794 24697 raft_consensus.cc:515] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:23:47.063642 24748 log.cc:927] Time spent T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.113s	user 0.001s	sys 0.000s
I20260709 12:23:47.028097 24753 raft_consensus.cc:493] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:47.067698 24753 raft_consensus.cc:515] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:47.070422 24753 leader_election.cc:290] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:47.034276 24688 raft_consensus.cc:493] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:47.072168 24688 raft_consensus.cc:3060] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Advancing to term 20
W20260709 12:23:47.077773 24675 consensus_meta.cc:287] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.805s	user 0.012s	sys 0.001s
I20260709 12:23:47.079003 17758 raft_consensus.cc:2468] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 18.
I20260709 12:23:47.080657 24675 raft_consensus.cc:493] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:23:47.085858 24439 raft_consensus.cc:3055] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:23:47.086843 24662 raft_consensus.cc:493] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:47.088016 24313 consensus_queue.cc:1048] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20260709 12:23:47.084167 24647 leader_election.cc:290] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 20 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:47.091162 24394 leader_election.cc:290] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 15 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:47.091322 24711 leader_election.cc:290] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:23:47.098560 24656 consensus_meta.cc:287] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.845s	user 0.012s	sys 0.000s
I20260709 12:23:47.117053 24687 raft_consensus.cc:515] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:47.284912 17757 raft_consensus.cc:2468] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 21.
I20260709 12:23:47.054780 24639 raft_consensus.cc:515] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:47.299059 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 312804us
W20260709 12:23:47.313827 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 142170us
I20260709 12:23:47.167739 24439 raft_consensus.cc:740] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:23:47.286116 24723 raft_consensus.cc:493] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:23:47.167177 24753 raft_consensus.cc:493] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:23:47.327210 24759 consensus_queue.cc:1059] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 13.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:23:47.328701 24639 leader_election.cc:290] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:23:47.325773 24736 consensus_meta.cc:287] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.590s	user 0.006s	sys 0.007s
I20260709 12:23:47.359830 17654 leader_election.cc:304] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:47.296209 24662 raft_consensus.cc:515] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:47.295527 24675 raft_consensus.cc:515] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:47.374611 17837 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37994 (request call id 1314) took 1632 ms. Trace:
I20260709 12:23:47.384325 17837 rpcz_store.cc:276] 0709 12:23:45.741990 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:45.880616 (+138626us) service_pool.cc:224] Handling call
0709 12:23:45.937410 (+ 56794us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:45.977309 (+ 39899us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:47.374282 (+1396973us) spinlock_profiling.cc:243] Waited 3.34 ms on lock 0x7b54000ac3d0. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a61f8ad 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:23:47.374518 (+   236us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":7682432}
W20260709 12:23:47.376250 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 91248us
I20260709 12:23:47.301187 24656 raft_consensus.cc:515] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:47.387741 18333 leader_election.cc:336] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 20 pre-election: RPC error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Timed out: RequestConsensusVote RPC to 127.16.29.66:39651 timed out after 1.547s (SENT)
I20260709 12:23:47.361138 24394 raft_consensus.cc:493] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:47.388336 18333 leader_election.cc:304] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:47.370015 24697 leader_election.cc:290] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:47.376727 24759 raft_consensus.cc:493] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:47.343981 24723 raft_consensus.cc:515] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:47.350930 24753 raft_consensus.cc:515] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:47.390697 24677 raft_consensus.cc:2749] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:23:47.414822 24687 leader_election.cc:290] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 17 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:23:47.411214 24428 consensus_meta.cc:287] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.606s	user 0.009s	sys 0.005s
I20260709 12:23:47.395550 24394 raft_consensus.cc:515] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:47.408475 24708 leader_election.cc:290] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 17 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:47.386497 24439 consensus_queue.cc:260] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.11, Last appended by leader: 11, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:47.415243 24729 raft_consensus.cc:2749] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 12:23:47.421334 24723 leader_election.cc:290] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:47.415537 24721 leader_election.cc:290] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:47.328722 24638 raft_consensus.cc:515] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:47.373031 24736 raft_consensus.cc:515] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:47.399055 24759 raft_consensus.cc:515] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:47.433287 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 73543.8us
I20260709 12:23:47.434502 17654 leader_election.cc:304] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:47.422564 24677 raft_consensus.cc:493] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:47.438760 17654 leader_election.cc:304] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:47.423148 24428 raft_consensus.cc:515] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:47.450076 17654 leader_election.cc:304] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:47.436476 24439 raft_consensus.cc:3060] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:47.452106 17654 leader_election.cc:400] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 14 election: Vote denied by peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577) with higher term. Message: Invalid argument: T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for earlier term 14. Current term is 15.
I20260709 12:23:47.452519 17654 leader_election.cc:403] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 14 election: Cancelling election due to peer responding with higher term
I20260709 12:23:47.439052 24729 raft_consensus.cc:2804] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:23:47.454576 17654 leader_election.cc:304] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:47.454725 24753 leader_election.cc:290] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:23:47.463618 17685 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44898 (request call id 1317) took 1934 ms (client timeout 2166 ms). Trace:
I20260709 12:23:47.463992 24662 leader_election.cc:290] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:47.464386 24753 raft_consensus.cc:2749] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:23:47.464754 24662 raft_consensus.cc:493] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:47.465744 24753 raft_consensus.cc:3060] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:23:47.464335 17685 rpcz_store.cc:269] 0709 12:23:45.523425 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:45.523609 (+   184us) service_pool.cc:224] Handling call
0709 12:23:47.457341 (+1933732us) spinlock_profiling.cc:243] Waited 492 ms on lock 0x7b5400190354. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:23:47.457478 (+   137us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1131217792}
I20260709 12:23:47.452927 24708 raft_consensus.cc:2749] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:23:47.465649 24662 raft_consensus.cc:515] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:47.438661 24688 raft_consensus.cc:515] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:47.468559 24708 raft_consensus.cc:2804] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:23:47.469604 24708 raft_consensus.cc:493] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:47.470155 24708 raft_consensus.cc:3060] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:23:47.470839 24688 leader_election.cc:290] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 20 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:47.437959 24677 raft_consensus.cc:515] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:47.480551 24758 log.cc:927] Time spent T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.070s	user 0.000s	sys 0.001s
I20260709 12:23:47.500324 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:47.501546 17681 raft_consensus.cc:2468] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 17.
W20260709 12:23:47.507513 17761 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38776 (request call id 2706) took 1963 ms (client timeout 1841 ms). Trace:
W20260709 12:23:47.508116 17761 rpcz_store.cc:269] 0709 12:23:45.544203 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:45.544328 (+   125us) service_pool.cc:224] Handling call
0709 12:23:47.507188 (+1962860us) spinlock_profiling.cc:243] Waited 52.9 ms on lock 0x7b5400194e54. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:23:47.507407 (+   219us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":121622528}
I20260709 12:23:47.457095 24656 leader_election.cc:290] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 17 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:47.490969 24675 leader_election.cc:290] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:47.533954 24736 leader_election.cc:290] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 21 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:47.491482 24231 leader_election.cc:290] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:47.556453 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 16 candidate_status { last_received { term: 14 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:47.556522 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:47.561167 17689 raft_consensus.cc:2393] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 16: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:23:47.564594 24662 leader_election.cc:290] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:47.564481 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:47.566380 17689 raft_consensus.cc:3060] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:47.565953 24729 raft_consensus.cc:697] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 19 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:23:47.567369 24759 leader_election.cc:290] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:47.568017 24394 leader_election.cc:290] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:47.598129 24638 leader_election.cc:290] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 21 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:47.606501 24428 leader_election.cc:290] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 21 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:47.618384 24394 raft_consensus.cc:493] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:23:47.625736 24677 leader_election.cc:290] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:47.640067 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:47.640753 18361 raft_consensus.cc:2468] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 17.
I20260709 12:23:47.641026 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:47.641630 18366 raft_consensus.cc:2468] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 16.
I20260709 12:23:47.641939 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 17 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:47.642493 18361 raft_consensus.cc:3060] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:47.648872 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:47.650480 18366 raft_consensus.cc:2393] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 18: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:23:47.650534 18361 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 17.
I20260709 12:23:47.651105 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:47.652024 17654 leader_election.cc:304] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:23:47.651770 18368 raft_consensus.cc:2468] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 21.
I20260709 12:23:47.655700 18367 raft_consensus.cc:1275] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 16 index: 10. Preceding OpId from leader: term: 18 index: 11. (index mismatch)
I20260709 12:23:47.639101 24729 consensus_queue.cc:237] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.13, Last appended by leader: 13, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:47.653628 17654 leader_election.cc:304] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:23:47.666275 17689 raft_consensus.cc:2468] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 18.
I20260709 12:23:47.673107 17654 leader_election.cc:304] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:23:47.675729 17831 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35040 (request call id 2995) took 1944 ms. Trace:
I20260709 12:23:47.662642 24767 raft_consensus.cc:2804] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:23:47.676669 24767 raft_consensus.cc:493] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:47.639424 24394 raft_consensus.cc:515] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:47.680114 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 17 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:47.676374 17831 rpcz_store.cc:276] 0709 12:23:45.731622 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:45.731743 (+   121us) service_pool.cc:224] Handling call
0709 12:23:45.732198 (+   455us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:23:45.739636 (+  7438us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:45.739681 (+    45us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:23:45.740375 (+   694us) log.cc:844] Serialized 52 byte log entry
0709 12:23:45.964775 (+224400us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:23:45.988560 (+ 23785us) log.cc:844] Serialized 14 byte log entry
0709 12:23:45.991384 (+  2824us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:45.991434 (+    50us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:47.661334 (+1669900us) spinlock_profiling.cc:243] Waited 4.32 ms on lock 0x7b18000f7300. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b475333f3 00007f7b475357c1 00007f7b47535779 00007f7b47535709 00007f7b475356d1 00007f7b475349cd 0000000000388634 0000000000388479 00007f7b4751683a
0709 12:23:47.675154 (+ 13820us) raft_consensus.cc:1685] finished
0709 12:23:47.675316 (+   162us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:47.675570 (+   254us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":20214,"prepare.queue_time_us":23550,"prepare.run_wall_time_us":42,"raft.queue_time_us":10651,"raft.run_cpu_time_us":369,"raft.run_wall_time_us":22252,"spinlock_wait_cycles":22769792,"thread_start_us":139699,"threads_started":1,"wal-append.queue_time_us":225342}
I20260709 12:23:47.683149 24769 raft_consensus.cc:2804] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:23:47.683794 24769 raft_consensus.cc:493] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:47.684285 24769 raft_consensus.cc:3060] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:23:47.686496 24227 consensus_queue.cc:1048] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:23:47.687381 17756 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38786 (request call id 3448) took 1608 ms. Trace:
I20260709 12:23:47.687846 17756 rpcz_store.cc:276] 0709 12:23:46.079051 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:46.085140 (+  6089us) service_pool.cc:224] Handling call
0709 12:23:47.687139 (+1601999us) spinlock_profiling.cc:243] Waited 21.7 ms on lock 0x7b54003bd3d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:23:47.687307 (+   168us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":49905536}
I20260709 12:23:47.692387 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:47.693883 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:47.694780 17684 raft_consensus.cc:2393] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 13: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:23:47.670120 24637 raft_consensus.cc:2804] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:23:47.697544 17680 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37396 (request call id 2775) took 1745 ms. Trace:
I20260709 12:23:47.697707 24637 raft_consensus.cc:493] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:47.697985 17680 rpcz_store.cc:276] 0709 12:23:45.952315 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:46.040248 (+ 87933us) service_pool.cc:224] Handling call
0709 12:23:46.040722 (+   474us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:46.262712 (+221990us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:47.697210 (+1434498us) spinlock_profiling.cc:243] Waited 220 ms on lock 0x7b5400207954. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:23:47.697488 (+   278us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":506301824}
I20260709 12:23:47.699306 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:47.700119 17835 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37994 (request call id 1315) took 1937 ms. Trace:
I20260709 12:23:47.700342 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 14 candidate_status { last_received { term: 13 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:47.700551 17835 rpcz_store.cc:276] 0709 12:23:45.762609 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:45.940329 (+177720us) service_pool.cc:224] Handling call
0709 12:23:45.959609 (+ 19280us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:46.069394 (+109785us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:47.699895 (+1630501us) spinlock_profiling.cc:243] Waited 109 ms on lock 0x7b540008bbd4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:23:47.700059 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":250066176}
I20260709 12:23:47.701538 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 18 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:47.701812 24643 raft_consensus.cc:3055] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:23:47.702277 24643 raft_consensus.cc:740] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:23:47.702200 24723 consensus_queue.cc:1048] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:23:47.702620 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:47.703739 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 14 candidate_status { last_received { term: 13 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:47.704363 24677 raft_consensus.cc:493] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:47.704859 24677 raft_consensus.cc:515] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:47.705480 18368 raft_consensus.cc:1240] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 13. Current term is 14. Ops: []
I20260709 12:23:47.705317 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 21 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:47.705710 24244 leader_election.cc:290] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 17 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:23:47.706568 17830 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37994 (request call id 1324) took 1835 ms (client timeout 1506 ms). Trace:
I20260709 12:23:47.707106 24677 leader_election.cc:290] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:23:47.720338 17830 rpcz_store.cc:269] 0709 12:23:45.870739 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:45.887173 (+ 16434us) service_pool.cc:224] Handling call
0709 12:23:47.706380 (+1819207us) spinlock_profiling.cc:243] Waited 60.4 ms on lock 0x7b54000aaad4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:23:47.706471 (+    91us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":138895232}
I20260709 12:23:47.700053 17684 raft_consensus.cc:3060] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:47.707456 17760 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38786 (request call id 3444) took 1848 ms. Trace:
W20260709 12:23:47.707917 24773 log.cc:927] Time spent T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.055s	user 0.001s	sys 0.000s
I20260709 12:23:47.728526 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:47.703404 24769 raft_consensus.cc:515] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:47.712035 24231 raft_consensus.cc:493] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:47.712244 24394 leader_election.cc:290] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:47.677500 24767 raft_consensus.cc:3060] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:47.703272 24643 consensus_queue.cc:260] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:47.714068 24662 raft_consensus.cc:493] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:23:47.715647 18334 leader_election.cc:304] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:23:47.724820 17760 rpcz_store.cc:276] 0709 12:23:45.858901 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:45.868791 (+  9890us) service_pool.cc:224] Handling call
0709 12:23:45.869124 (+   333us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:46.253158 (+384034us) raft_consensus.cc:1534] Early marking committed up to index 15
0709 12:23:46.382182 (+129024us) log.cc:844] Serialized 14 byte log entry
0709 12:23:46.833396 (+451214us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:46.833703 (+   307us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:23:46.833782 (+    79us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:47.707222 (+873440us) spinlock_profiling.cc:243] Waited 384 ms on lock 0x7b54001b8354. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:23:47.707266 (+    44us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:47.707400 (+   134us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":128591,"prepare.run_cpu_time_us":51,"prepare.run_wall_time_us":52,"raft.queue_time_us":244,"raft.run_cpu_time_us":124,"raft.run_wall_time_us":2189,"spinlock_wait_cycles":882626432,"thread_start_us":446051,"threads_started":2,"wal-append.queue_time_us":440988}
I20260709 12:23:47.736742 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:47.737557 17689 raft_consensus.cc:2393] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 19: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:23:47.720121 17680 raft_consensus.cc:2393] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 21: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:23:47.715230 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:47.727392 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:47.730664 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:47.737205 18334 leader_election.cc:304] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:23:47.739295 18361 raft_consensus.cc:3060] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:47.741888 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:47.699478 24637 raft_consensus.cc:3060] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:47.748785 18361 raft_consensus.cc:2468] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 21.
I20260709 12:23:47.729307 24656 raft_consensus.cc:493] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:23:47.772006 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:47.772883 17689 raft_consensus.cc:2468] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 19.
I20260709 12:23:47.755668 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 17 candidate_status { last_received { term: 14 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:47.748306 24394 raft_consensus.cc:2804] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:23:47.748550 24681 raft_consensus.cc:2804] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:23:47.757927 24244 consensus_queue.cc:1059] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 13.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.035s
I20260709 12:23:47.793766 24729 raft_consensus.cc:3055] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 13 LEADER]: Stepping down as leader of term 13
W20260709 12:23:47.773506 24518 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.070s	user 0.004s	sys 0.015s
I20260709 12:23:47.793907 24665 leader_election.cc:290] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:47.771693 24643 raft_consensus.cc:3060] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:47.772109 24439 raft_consensus.cc:493] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:23:47.794260 24729 raft_consensus.cc:740] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:23:47.738842 24231 raft_consensus.cc:515] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:47.804628 18361 raft_consensus.cc:1240] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 19. Current term is 20. Ops: []
W20260709 12:23:47.808073 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:47.809568 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:47.809675 24392 leader_election.cc:290] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 19 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:47.810891 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 17 candidate_status { last_received { term: 14 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:47.813122 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:47.813256 24769 leader_election.cc:290] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 22 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:23:47.813439 17832 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37994 (request call id 1327) took 1703 ms (client timeout 2054 ms). Trace:
I20260709 12:23:47.826030 17830 raft_consensus.cc:3060] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Advancing to term 19
W20260709 12:23:47.826279 17832 rpcz_store.cc:269] 0709 12:23:46.110209 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:46.158428 (+ 48219us) service_pool.cc:224] Handling call
0709 12:23:47.813251 (+1654823us) spinlock_profiling.cc:243] Waited 743 ms on lock 0x7b54000aafd4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:23:47.813384 (+   133us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1709244800}
I20260709 12:23:47.801223 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:47.815672 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 16 candidate_status { last_received { term: 14 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:47.743554 24662 raft_consensus.cc:515] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:47.803596 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:47.828549 17831 raft_consensus.cc:2393] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 16: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:23:47.795521 24394 raft_consensus.cc:493] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:47.767621 24708 raft_consensus.cc:515] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:47.832748 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:47.833678 18364 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59902 (request call id 1660) took 1597 ms. Trace:
I20260709 12:23:47.833487 24439 raft_consensus.cc:515] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:47.835000 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:47.862445 17682 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 15.
W20260709 12:23:47.836118 17833 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35040 (request call id 2997) took 2057 ms (client timeout 1656 ms). Trace:
W20260709 12:23:47.863336 17833 rpcz_store.cc:269] 0709 12:23:45.778890 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:45.808797 (+ 29907us) service_pool.cc:224] Handling call
0709 12:23:47.835925 (+2027128us) spinlock_profiling.cc:243] Waited 107 ms on lock 0x7b54004416d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:23:47.836054 (+   129us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":246222080}
I20260709 12:23:47.836258 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:47.864853 17837 raft_consensus.cc:2468] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 15.
I20260709 12:23:47.836897 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:47.837147 18371 raft_consensus.cc:2393] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 17: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:23:47.840425 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:47.873828 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 17 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:47.874593 17688 raft_consensus.cc:2393] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 17: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:23:47.778484 24656 raft_consensus.cc:515] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:47.860503 17681 raft_consensus.cc:2468] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 18.
I20260709 12:23:47.872673 18363 raft_consensus.cc:2468] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 16.
W20260709 12:23:47.799500 24518 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.096s	user 0.004s	sys 0.016s
I20260709 12:23:47.886557 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:47.883699 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:47.890072 17686 raft_consensus.cc:2368] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 23. Current term is 24.
I20260709 12:23:47.861109 24394 raft_consensus.cc:3060] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:47.839318 24729 consensus_queue.cc:260] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 13.14, Last appended by leader: 14, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:47.893325 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:47.861131 18364 rpcz_store.cc:276] 0709 12:23:46.236021 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:46.236141 (+   120us) service_pool.cc:224] Handling call
0709 12:23:46.237118 (+   977us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:46.309732 (+ 72614us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:46.309780 (+    48us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:46.310684 (+   904us) log.cc:844] Serialized 26 byte log entry
0709 12:23:46.860214 (+549530us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:23:46.860288 (+    74us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:47.494495 (+634207us) spinlock_profiling.cc:243] Waited 62 ms on lock 0x7b4800220a58. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4f549aa1 00007f7b4f540b37 00007f7b4f5409ca 00007f7b4f63826f 00007f7b4f637f37 00007f7b4f64c55c 00007f7b4f64c508 00007f7b4f64c488 00007f7b4f64c443 00007f7b4f64b6ec 00007f7b4e575a96
0709 12:23:47.833233 (+338738us) spinlock_profiling.cc:243] Waited 71 ms on lock 0x7b540051aed4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:23:47.833311 (+    78us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:47.833381 (+    70us) raft_consensus.cc:1685] finished
0709 12:23:47.833406 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:47.833599 (+   193us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":939,"raft.queue_time_us":1774,"raft.run_cpu_time_us":13268,"raft.run_wall_time_us":1181322,"spinlock_wait_cycles":305917952,"thread_start_us":537637,"threads_started":1,"wal-append.queue_time_us":629657}
I20260709 12:23:47.895133 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:47.896324 18364 raft_consensus.cc:2410] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for term 20 because replica has last-logged OpId of term: 18 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 12.
I20260709 12:23:47.897194 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:47.898118 18362 raft_consensus.cc:2393] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 15: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:23:47.899035 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:47.875317 24753 raft_consensus.cc:2749] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Leader election lost for term 14. Reason: Vote denied by peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577) with higher term. Message: Invalid argument: T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for earlier term 14. Current term is 15.
I20260709 12:23:47.901715 17681 raft_consensus.cc:1240] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 15 LEADER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 14. Current term is 15. Ops: []
I20260709 12:23:47.902263 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:47.873687 24681 raft_consensus.cc:697] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 18 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:23:47.908123 17681 raft_consensus.cc:1240] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 17. Current term is 18. Ops: []
W20260709 12:23:47.909206 17729 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:47.913475 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:47.914156 18364 raft_consensus.cc:3060] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:23:47.916934 17835 raft_consensus.cc:2468] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 14.
I20260709 12:23:47.918435 24231 leader_election.cc:290] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:23:47.925729 17729 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:47.926952 24675 raft_consensus.cc:493] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:23:47.929435 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 17 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:47.930155 18363 raft_consensus.cc:3060] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:47.931272 17688 raft_consensus.cc:1275] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 19 index: 12. Preceding OpId from leader: term: 21 index: 13. (index mismatch)
W20260709 12:23:47.933097 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 90199.5us
I20260709 12:23:47.952637 24759 raft_consensus.cc:493] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:23:47.960696 24759 raft_consensus.cc:515] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:47.962620 24681 consensus_queue.cc:237] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.12, Last appended by leader: 12, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:47.965853 24637 raft_consensus.cc:515] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:47.967908 17683 raft_consensus.cc:2393] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 17: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:23:47.968506 24687 raft_consensus.cc:493] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:47.969123 24687 raft_consensus.cc:515] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:47.972216 24688 raft_consensus.cc:493] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:47.973366 24688 raft_consensus.cc:515] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:47.975062 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:47.976574 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:47.927511 24675 raft_consensus.cc:515] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:47.977501 17755 raft_consensus.cc:1240] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 21. Current term is 22. Ops: []
I20260709 12:23:47.978621 24675 leader_election.cc:290] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:47.978770 17755 raft_consensus.cc:1240] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 19. Current term is 20. Ops: [19.14-19.14]
I20260709 12:23:47.979419 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:47.976758 24708 leader_election.cc:290] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 24 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:47.980400 17760 raft_consensus.cc:1240] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 16. Current term is 17. Ops: [14.12-16.13]
I20260709 12:23:47.985306 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:47.988529 17760 raft_consensus.cc:2468] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 15.
I20260709 12:23:47.989663 18363 raft_consensus.cc:2410] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for term 17 because replica has last-logged OpId of term: 16 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 12.
I20260709 12:23:47.990451 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:47.986965 18333 leader_election.cc:304] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:23:47.990151 17830 raft_consensus.cc:2468] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 19.
I20260709 12:23:47.996165 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:47.996873 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 20 candidate_status { last_received { term: 18 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:23:47.998276 17801 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:47.999513 18362 raft_consensus.cc:1275] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 12 index: 10. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
I20260709 12:23:47.945278 17762 consensus_queue.cc:260] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 19.12, Last appended by leader: 12, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:48.002264 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:48.003038 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 17 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:48.003314 24688 leader_election.cc:290] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:47.990113 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
W20260709 12:23:47.998694 17801 connection.cc:256] RPC call timeout handler was delayed by -1.316785594999601s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1674s for a response
W20260709 12:23:48.008288 17801 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:48.009228 18368 raft_consensus.cc:1240] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 18. Current term is 19. Ops: []
W20260709 12:23:48.011152 17654 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:48.014075 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:48.014745 17759 raft_consensus.cc:3060] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:47.953253 24439 leader_election.cc:290] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:48.006023 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:48.018085 17760 raft_consensus.cc:2368] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 23. Current term is 24.
I20260709 12:23:47.987262 17680 raft_consensus.cc:2468] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 15.
I20260709 12:23:48.007959 17755 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 16.
I20260709 12:23:48.020252 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:48.021044 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:48.021476 17755 raft_consensus.cc:2468] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 19.
I20260709 12:23:48.023298 17757 raft_consensus.cc:2468] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 21.
I20260709 12:23:48.029129 17762 raft_consensus.cc:3060] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:48.030686 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:48.031145 17801 leader_election.cc:304] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:48.027338 18333 leader_election.cc:304] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:23:47.997073 24729 raft_consensus.cc:3060] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:48.034565 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:48.037079 18334 leader_election.cc:304] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:23:48.041532 18363 raft_consensus.cc:2393] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 20: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:23:48.004566 24767 raft_consensus.cc:515] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:48.043104 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:48.016064 24766 raft_consensus.cc:2804] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:23:48.044977 17757 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 14.
I20260709 12:23:48.002182 24687 leader_election.cc:290] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:48.045725 17755 raft_consensus.cc:1240] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 19. Current term is 20. Ops: []
I20260709 12:23:48.017560 24716 raft_consensus.cc:493] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:48.047981 24656 leader_election.cc:290] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:48.029804 24665 raft_consensus.cc:493] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:48.043857 24231 raft_consensus.cc:2804] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:23:48.064695 18368 raft_consensus.cc:1240] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 19. Current term is 20. Ops: []
I20260709 12:23:48.082122 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:48.067804 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:48.068982 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:48.084834 17684 raft_consensus.cc:2468] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 20.
I20260709 12:23:48.073120 24703 raft_consensus.cc:2804] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:23:48.075723 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:23:48.077354 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 46278.5us
I20260709 12:23:48.043257 18334 leader_election.cc:304] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:23:48.081156 24709 raft_consensus.cc:3060] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:48.093816 17801 leader_election.cc:304] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:48.088742 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:48.080041 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 18 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:23:48.104880 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 61675.5us
I20260709 12:23:48.106168 18334 leader_election.cc:304] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:23:48.109265 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 18 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:48.109680 18364 raft_consensus.cc:2468] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 24.
I20260709 12:23:48.033973 24677 consensus_queue.cc:1059] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.060s
I20260709 12:23:48.112502 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:48.119637 17755 raft_consensus.cc:2468] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 18.
I20260709 12:23:48.119940 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:48.120589 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:48.092527 24656 raft_consensus.cc:493] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:48.118028 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 18 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:48.122553 17757 raft_consensus.cc:2468] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 17.
I20260709 12:23:48.121560 24721 consensus_queue.cc:1059] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 19.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.003s
I20260709 12:23:48.124428 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 21 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:48.091338 24703 raft_consensus.cc:493] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
W20260709 12:23:48.130487 17801 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.16.29.66:39651 timed out after 1.723s (SENT) [suppressed 20 similar messages]
I20260709 12:23:48.128712 24759 leader_election.cc:290] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:48.102222 24766 raft_consensus.cc:697] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 14 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:23:48.104421 24486 raft_consensus.cc:2804] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:23:48.128090 24637 leader_election.cc:290] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 17 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:48.089505 24716 raft_consensus.cc:515] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:48.137893 24703 raft_consensus.cc:3060] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:48.186223 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:48.132747 17761 raft_consensus.cc:1275] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 18 index: 11. Preceding OpId from leader: term: 21 index: 13. (index mismatch)
I20260709 12:23:48.141898 23838 raft_consensus.cc:2804] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:23:48.209183 23838 raft_consensus.cc:493] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:48.138023 24656 raft_consensus.cc:515] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:48.209612 23838 raft_consensus.cc:3060] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:48.152635 24766 consensus_queue.cc:237] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 13.14, Last appended by leader: 14, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:48.143151 24677 raft_consensus.cc:2804] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:23:48.212193 24677 raft_consensus.cc:493] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:23:48.212699 24677 raft_consensus.cc:3060] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:48.213332 24712 consensus_queue.cc:1059] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 18.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
W20260709 12:23:48.214973 17758 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38786 (request call id 3449) took 2135 ms (client timeout 1723 ms). Trace:
I20260709 12:23:48.215672 24643 raft_consensus.cc:3055] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:23:48.216056 24643 raft_consensus.cc:740] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:23:48.157745 24394 raft_consensus.cc:515] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:48.217754 24643 consensus_queue.cc:260] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 19.14, Last appended by leader: 14, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:48.218770 18366 raft_consensus.cc:2368] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 17. Current term is 18.
W20260709 12:23:48.215391 17758 rpcz_store.cc:269] 0709 12:23:46.079534 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:46.079973 (+   439us) service_pool.cc:224] Handling call
0709 12:23:48.214785 (+2134812us) spinlock_profiling.cc:243] Waited 998 ms on lock 0x7b54003eb054. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:23:48.214898 (+   113us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2294756224}
I20260709 12:23:48.226999 17836 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37994 (request call id 1331) took 2111 ms. Trace:
I20260709 12:23:48.112694 24788 consensus_queue.cc:1059] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.044s
I20260709 12:23:48.164973 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:48.236542 24788 consensus_queue.cc:1243] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer e2909cff1da44ad387f96a8ae3aa5afb log is divergent from this leader: its last log entry 20.15 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 14
I20260709 12:23:48.141397 24486 raft_consensus.cc:493] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:48.169623 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 18 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:48.237915 18374 raft_consensus.cc:2468] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 17.
I20260709 12:23:48.170208 24716 leader_election.cc:290] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:48.172382 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:48.244741 18363 raft_consensus.cc:2468] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 18.
I20260709 12:23:48.180827 24685 raft_consensus.cc:3060] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:48.100378 24665 raft_consensus.cc:515] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:48.068145 17762 raft_consensus.cc:1275] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 19 index: 12. Preceding OpId from leader: term: 20 index: 12. (term mismatch)
I20260709 12:23:48.251089 17762 pending_rounds.cc:85] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb: Aborting all ops after (but not including) 11
I20260709 12:23:48.251497 17762 pending_rounds.cc:107] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb: Aborting uncommitted NO_OP operation due to leader change: 19.12
W20260709 12:23:48.199479 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:48.251832 17762 raft_consensus.cc:2889] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:48.207078 17759 raft_consensus.cc:2468] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 20.
W20260709 12:23:48.211752 24747 log.cc:927] Time spent T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 1.495s	user 0.002s	sys 0.001s
W20260709 12:23:48.228492 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 1.21259e+06us
I20260709 12:23:48.230840 24657 raft_consensus.cc:3060] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:48.269886 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:48.237149 17836 rpcz_store.cc:276] 0709 12:23:46.115769 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:46.833268 (+717499us) service_pool.cc:224] Handling call
0709 12:23:46.848513 (+ 15245us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:46.848989 (+   476us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:46.849028 (+    39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:46.849635 (+   607us) log.cc:844] Serialized 26 byte log entry
0709 12:23:46.971897 (+122262us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:23:46.972085 (+   188us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:46.972203 (+   118us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:47.125800 (+153597us) raft_consensus.cc:1685] finished
0709 12:23:47.125834 (+    34us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:48.226657 (+1100823us) spinlock_profiling.cc:243] Waited 14.7 ms on lock 0x7b54000ab9d0. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a61f8ad 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:23:48.226918 (+   261us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1547,"raft.run_cpu_time_us":128,"raft.run_wall_time_us":323,"spinlock_wait_cycles":33919744,"thread_start_us":121527,"threads_started":1,"wal-append.queue_time_us":165480,"wal-append.run_cpu_time_us":1780,"wal-append.run_wall_time_us":1040696}
I20260709 12:23:48.158937 17756 raft_consensus.cc:2468] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 17.
I20260709 12:23:48.245306 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 22 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:48.270967 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:48.246860 23838 raft_consensus.cc:515] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:48.237278 24486 raft_consensus.cc:3060] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:23:48.235616 24733 raft_consensus.cc:3055] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:23:48.273291 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:48.218786 24656 leader_election.cc:290] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:48.274195 17831 raft_consensus.cc:2468] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 17.
I20260709 12:23:48.292608 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:48.273999 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:48.294346 18363 raft_consensus.cc:2393] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 24: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:23:48.271040 17838 raft_consensus.cc:3060] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:48.275980 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:48.295013 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:48.295305 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:48.295794 18374 raft_consensus.cc:3060] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:48.290529 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 21 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:48.294224 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:48.298671 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:48.298732 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:48.299687 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 17 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:48.300199 17829 raft_consensus.cc:3060] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:48.283417 17762 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38786 (request call id 3452) took 1834 ms. Trace:
I20260709 12:23:48.299358 18364 raft_consensus.cc:3060] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:23:48.299256 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:48.300765 17762 rpcz_store.cc:276] 0709 12:23:46.448594 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:46.448783 (+   189us) service_pool.cc:224] Handling call
0709 12:23:46.515658 (+ 66875us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:47.944495 (+1428837us) spinlock_profiling.cc:243] Waited 55.1 ms on lock 0x7b4c004146c0. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b488eca20 00007f7b488fda29 00007f7b488c3644 00007f7b488eaf2d 00007f7b488eab93 00007f7b4a61b18b 00007f7b4a61bcb5 00007f7b4a616f30 00007f7b4a6187de 00007f7b4a622b49 00007f7b4a623416
0709 12:23:48.283018 (+338523us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:48.283338 (+   320us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":126705024}
I20260709 12:23:48.295922 17831 raft_consensus.cc:3060] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:48.300132 17761 raft_consensus.cc:2468] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 15.
W20260709 12:23:48.277716 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 147268us
I20260709 12:23:48.288530 17756 raft_consensus.cc:2393] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 20: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
W20260709 12:23:48.273595 17729 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:48.303674 17729 connection.cc:256] RPC call timeout handler was delayed by -1.1684541519998675s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1535s for a response
W20260709 12:23:48.304116 17729 connection.cc:256] RPC call timeout handler was delayed by -1.1162541519997831s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1593s for a response
I20260709 12:23:48.348690 17838 raft_consensus.cc:2468] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 20.
W20260709 12:23:48.350760 17729 connection.cc:256] RPC call timeout handler was delayed by -1.1072541519997685s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1603s for a response
I20260709 12:23:48.340875 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:23:48.351229 17729 connection.cc:256] RPC call timeout handler was delayed by -1.1036541519997627s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1607s for a response
W20260709 12:23:48.351575 17729 connection.cc:256] RPC call timeout handler was delayed by -1.0685541519997059s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1646s for a response
I20260709 12:23:48.348914 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:48.308313 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 17 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:48.352588 18365 raft_consensus.cc:2393] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 18: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:23:48.352411 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
W20260709 12:23:48.346865 17802 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:48.353416 17835 raft_consensus.cc:3060] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Advancing to term 21
W20260709 12:23:48.356057 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 249972us
W20260709 12:23:48.279685 17652 connection.cc:256] RPC call timeout handler was delayed by -1.23982560200011s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1656s for a response
W20260709 12:23:48.358647 17652 connection.cc:256] RPC call timeout handler was delayed by -1.0733256020002955s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1841s for a response
I20260709 12:23:48.356874 17729 leader_election.cc:304] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:48.281100 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:48.358940 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:48.359591 17831 raft_consensus.cc:2468] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 21.
I20260709 12:23:48.304464 23838 leader_election.cc:290] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 16 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:48.360293 17829 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 17.
I20260709 12:23:48.360196 17759 raft_consensus.cc:2468] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 12.
I20260709 12:23:48.303795 17801 leader_election.cc:304] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:48.366191 17831 raft_consensus.cc:1275] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 16 index: 10. Preceding OpId from leader: term: 18 index: 11. (index mismatch)
I20260709 12:23:48.352753 17758 raft_consensus.cc:3060] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:48.370736 18374 raft_consensus.cc:2468] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 17.
I20260709 12:23:48.372499 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:48.256927 24709 raft_consensus.cc:2749] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:23:48.373603 17759 raft_consensus.cc:3060] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:48.287436 24733 raft_consensus.cc:740] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:23:48.387749 23838 consensus_queue.cc:1048] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:23:48.363339 18364 raft_consensus.cc:2468] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 22.
W20260709 12:23:48.309122 24524 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 20 current term: 20 is higher.
W20260709 12:23:48.374688 17652 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:48.375154 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:48.237641 24703 raft_consensus.cc:515] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:48.378162 17729 leader_election.cc:304] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:23:48.306597 24704 raft_consensus.cc:493] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:23:48.400956 17654 leader_election.cc:304] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:23:48.364130 24518 raft_consensus.cc:493] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
W20260709 12:23:48.402258 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 300067us
I20260709 12:23:48.404247 17652 leader_election.cc:304] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:48.404407 17835 raft_consensus.cc:2468] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 21.
I20260709 12:23:48.378453 24759 raft_consensus.cc:2749] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:23:48.406714 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:48.407385 17757 raft_consensus.cc:2393] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 24: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:23:48.419606 17764 raft_consensus.cc:3060] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Advancing to term 17
I20260709 12:23:48.408209 17830 raft_consensus.cc:1275] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 15 index: 12. (index mismatch)
W20260709 12:23:48.289868 17572 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet d522f70beb1f417cb98940d90b594823 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 0da169ff82674dc1947055ce43de68a3 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 17 committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:23:48.389247 24723 raft_consensus.cc:493] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:23:48.389822 24231 raft_consensus.cc:697] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 16 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:23:48.417495 17759 raft_consensus.cc:2468] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 21.
I20260709 12:23:48.402695 24439 raft_consensus.cc:493] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:48.420909 17572 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 163801us
I20260709 12:23:48.430130 18333 leader_election.cc:304] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:23:48.435781 23838 raft_consensus.cc:2764] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Leader pre-election decision vote started in defunct term 16: won
I20260709 12:23:48.407127 24637 raft_consensus.cc:493] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:48.407799 24769 raft_consensus.cc:493] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:48.388130 24675 consensus_queue.cc:1059] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 14.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.029s
I20260709 12:23:48.436560 18333 leader_election.cc:304] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:23:48.447993 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:48.449214 17757 raft_consensus.cc:3060] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Advancing to term 21
W20260709 12:23:48.450978 17834 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37994 (request call id 1319) took 2676 ms (client timeout 1549 ms). Trace:
W20260709 12:23:48.451648 17834 rpcz_store.cc:269] 0709 12:23:45.763728 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:45.767775 (+  4047us) service_pool.cc:224] Handling call
0709 12:23:48.432645 (+2664870us) spinlock_profiling.cc:243] Waited 6.92 ms on lock 0x7b54002eac54. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:23:48.440517 (+  7872us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":15913856}
I20260709 12:23:48.423709 24753 raft_consensus.cc:2804] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Leader election won for term 22
W20260709 12:23:48.462605 18333 leader_election.cc:336] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837): Timed out: RequestConsensusVote RPC to 127.16.29.67:36837 timed out after 1.549s (SENT)
W20260709 12:23:48.463176 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 159468us
W20260709 12:23:48.460211 17802 connection.cc:256] RPC call timeout handler was delayed by -1.1752230490001239s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15400000000000003s for a response
I20260709 12:23:48.463193 18333 leader_election.cc:304] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:48.467224 23838 raft_consensus.cc:2749] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:23:48.471164 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:48.471083 24231 consensus_queue.cc:237] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 14.14, Last appended by leader: 14, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:48.472160 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:48.475809 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:48.476706 17831 raft_consensus.cc:3060] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:48.475620 17764 raft_consensus.cc:2468] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 17.
I20260709 12:23:48.439779 24703 leader_election.cc:290] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 20 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:48.507232 17757 raft_consensus.cc:2468] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 21.
I20260709 12:23:48.421069 24704 raft_consensus.cc:515] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:48.454213 24656 raft_consensus.cc:2804] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:23:48.531467 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:48.516269 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:48.534329 17829 raft_consensus.cc:2393] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 15: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:23:48.517150 17764 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690"
dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
mode: GRACEFUL
new_leader_uuid: "e063541542a94969898868bf6eb5ce1b"
 from {username='slave'} at 127.0.0.1:49786
I20260709 12:23:48.535529 17764 raft_consensus.cc:606] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Received request to transfer leadership to TS e063541542a94969898868bf6eb5ce1b
I20260709 12:23:48.536044 17764 raft_consensus.cc:614] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:23:48.426600 24518 raft_consensus.cc:515] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:48.522245 17683 raft_consensus.cc:1275] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 16 index: 10. Preceding OpId from leader: term: 17 index: 11. (index mismatch)
I20260709 12:23:48.523185 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:48.538362 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:48.477922 24796 raft_consensus.cc:2804] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:23:48.524950 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 19 candidate_status { last_received { term: 18 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
W20260709 12:23:48.515563 17728 connection.cc:256] RPC call timeout handler was delayed by -1.053475877999972s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1535s for a response
W20260709 12:23:48.541687 24805 log.cc:927] Time spent T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.065s	user 0.001s	sys 0.000s
I20260709 12:23:48.542413 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 22 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:48.542960 17728 leader_election.cc:304] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:23:48.526789 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 57415.9us
I20260709 12:23:48.529147 17681 raft_consensus.cc:1240] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 14. Current term is 15. Ops: []
I20260709 12:23:48.548900 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:48.549813 18363 raft_consensus.cc:2410] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 17 because replica has last-logged OpId of term: 15 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 10.
W20260709 12:23:48.529816 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:48.479396 24759 raft_consensus.cc:2764] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Leader pre-election decision vote started in defunct term 15: won
I20260709 12:23:48.543540 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:48.447146 24637 raft_consensus.cc:515] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:48.552474 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:48.551978 17688 raft_consensus.cc:2393] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 21: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:23:48.557461 17836 raft_consensus.cc:2393] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 20: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:23:48.479151 24394 leader_election.cc:290] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 18 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:48.554626 18366 raft_consensus.cc:1275] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 17 index: 14. Preceding OpId from leader: term: 18 index: 15. (index mismatch)
I20260709 12:23:48.465883 24441 consensus_queue.cc:1059] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.072s
I20260709 12:23:48.445377 24439 raft_consensus.cc:515] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:48.453642 24769 raft_consensus.cc:515] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:48.561548 17687 raft_consensus.cc:1240] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 20. Current term is 21. Ops: [20.15-20.15]
W20260709 12:23:48.574988 24797 log.cc:927] Time spent T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.128s	user 0.000s	sys 0.001s
I20260709 12:23:48.518484 24675 raft_consensus.cc:3055] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:23:48.442094 24733 consensus_queue.cc:260] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 16.13, Last appended by leader: 13, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:48.563606 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:48.564481 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:48.564805 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:48.551687 24796 raft_consensus.cc:493] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:48.561105 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 18 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:48.476325 24723 raft_consensus.cc:515] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:48.533367 24709 raft_consensus.cc:493] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:48.576294 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 172159us
I20260709 12:23:48.569774 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:48.577471 24441 consensus_queue.cc:1243] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer e2909cff1da44ad387f96a8ae3aa5afb log is divergent from this leader: its last log entry 15.12 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 11
I20260709 12:23:48.577903 24675 raft_consensus.cc:740] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:23:48.565127 24677 raft_consensus.cc:515] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:48.581385 17831 raft_consensus.cc:2468] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 14.
I20260709 12:23:48.647280 17831 raft_consensus.cc:1240] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 20 LEADER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 19. Current term is 20. Ops: []
I20260709 12:23:48.592409 17836 raft_consensus.cc:3060] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:48.593703 17829 raft_consensus.cc:2468] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 15.
W20260709 12:23:48.586879 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 200246us
I20260709 12:23:48.595796 18365 raft_consensus.cc:2468] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 13.
I20260709 12:23:48.596601 17652 leader_election.cc:304] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:48.596908 24753 raft_consensus.cc:697] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 22 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:23:48.601871 24816 raft_consensus.cc:2749] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 12:23:48.602160 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:48.593240 24796 raft_consensus.cc:3060] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:48.605388 24705 raft_consensus.cc:493] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:48.613466 24642 raft_consensus.cc:3055] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:23:48.620039 18366 raft_consensus.cc:1275] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 18 index: 16. Preceding OpId from leader: term: 20 index: 17. (index mismatch)
I20260709 12:23:48.619741 24524 consensus_queue.cc:1059] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 17.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:23:48.623600 24313 raft_consensus.cc:2804] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:23:48.623911 24518 leader_election.cc:290] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:23:48.626711 24806 log.cc:927] Time spent T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.061s	user 0.000s	sys 0.000s
I20260709 12:23:48.572240 24656 raft_consensus.cc:697] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 17 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:23:48.688761 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:48.689450 17687 raft_consensus.cc:2468] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 20.
I20260709 12:23:48.628386 24802 raft_consensus.cc:3055] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:23:48.632561 17684 raft_consensus.cc:2410] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for term 22 because replica has last-logged OpId of term: 19 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 11.
I20260709 12:23:48.637554 24769 leader_election.cc:290] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:48.638298 24733 raft_consensus.cc:3060] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:48.639922 24704 leader_election.cc:290] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:48.640350 24579 raft_consensus.cc:493] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:48.641163 24665 leader_election.cc:290] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:48.642549 24637 leader_election.cc:290] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:48.566089 24687 consensus_queue.cc:1059] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 18.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.037s
I20260709 12:23:48.642262 24486 raft_consensus.cc:515] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:48.653076 17831 raft_consensus.cc:1240] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 18. Current term is 19. Ops: [18.10-18.11]
I20260709 12:23:48.653581 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 21 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:23:48.658702 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 63333us
I20260709 12:23:48.662011 17689 raft_consensus.cc:2468] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 17.
I20260709 12:23:48.658529 17729 leader_election.cc:304] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:48.676062 17829 raft_consensus.cc:1240] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 15. Current term is 16. Ops: [15.11-15.11]
I20260709 12:23:48.677312 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 20 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:48.724759 17652 leader_election.cc:304] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:48.727748 17830 raft_consensus.cc:1240] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 20. Current term is 21. Ops: []
I20260709 12:23:48.729055 17830 raft_consensus.cc:1240] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 13. Current term is 14. Ops: []
I20260709 12:23:48.729149 17652 leader_election.cc:304] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:48.730521 24769 raft_consensus.cc:2804] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:23:48.732861 17683 raft_consensus.cc:2393] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 20: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:23:48.685315 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 17 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:48.733556 18365 raft_consensus.cc:2393] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 17: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:23:48.733497 17831 raft_consensus.cc:1275] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 12 index: 10. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
I20260709 12:23:48.687935 18363 raft_consensus.cc:1240] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 18. Current term is 19. Ops: []
I20260709 12:23:48.699265 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 17 candidate_status { last_received { term: 14 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:48.731068 24769 raft_consensus.cc:493] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:48.701164 17758 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 17.
I20260709 12:23:48.679179 24642 raft_consensus.cc:740] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:23:48.600171 24709 raft_consensus.cc:515] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:48.673722 24818 raft_consensus.cc:3060] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:48.709618 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:48.717630 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:48.701840 24753 consensus_queue.cc:237] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.14, Last appended by leader: 14, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:48.732528 17652 leader_election.cc:304] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:48.680539 17836 raft_consensus.cc:2468] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 18.
I20260709 12:23:48.690702 24802 raft_consensus.cc:740] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:23:48.737299 17835 raft_consensus.cc:2368] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for earlier term 21. Current term is 22.
I20260709 12:23:48.738818 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:48.745733 17688 raft_consensus.cc:2393] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 18: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:23:48.702824 24313 raft_consensus.cc:697] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:23:48.683049 24705 raft_consensus.cc:515] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:48.747128 17652 leader_election.cc:304] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:48.752178 17652 leader_election.cc:304] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:48.731199 24812 raft_consensus.cc:2749] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:23:48.753923 24518 raft_consensus.cc:2804] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:23:48.754037 24313 consensus_queue.cc:237] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 19.10, Last appended by leader: 10, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:48.754748 24518 raft_consensus.cc:697] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 21 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:23:48.755616 24518 consensus_queue.cc:237] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.10, Last appended by leader: 10, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:23:48.759099 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:48.768127 24486 leader_election.cc:290] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 22 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:48.769289 24802 consensus_queue.cc:260] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 16.13, Last appended by leader: 13, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:48.749293 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:48.772001 24637 raft_consensus.cc:3060] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:48.773595 17833 raft_consensus.cc:2468] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 13.
I20260709 12:23:48.778992 24801 raft_consensus.cc:493] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:48.779570 24801 raft_consensus.cc:515] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:48.783821 17680 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44898 (request call id 1340) took 1063 ms. Trace:
I20260709 12:23:48.784267 17680 rpcz_store.cc:276] 0709 12:23:47.720498 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:47.809304 (+ 88806us) service_pool.cc:224] Handling call
0709 12:23:48.783615 (+974311us) spinlock_profiling.cc:243] Waited 114 ms on lock 0x7b5400207e54. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:23:48.783754 (+   139us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":261337984}
I20260709 12:23:48.737854 24769 raft_consensus.cc:3060] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:23:48.793756 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 57651.5us
I20260709 12:23:48.796747 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:48.797916 17680 raft_consensus.cc:2368] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 17. Current term is 18.
I20260709 12:23:48.720701 24579 raft_consensus.cc:515] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:48.803962 18334 leader_election.cc:304] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:23:48.803576 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:48.803524 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:48.809202 17832 raft_consensus.cc:2468] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 17.
I20260709 12:23:48.807788 17802 leader_election.cc:304] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:48.810720 18365 raft_consensus.cc:1275] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 15 index: 12. (index mismatch)
I20260709 12:23:48.711673 24675 consensus_queue.cc:260] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:48.748282 24819 raft_consensus.cc:2749] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:23:48.818764 17680 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba"
dest_uuid: "0da169ff82674dc1947055ce43de68a3"
mode: GRACEFUL
new_leader_uuid: "e063541542a94969898868bf6eb5ce1b"
 from {username='slave'} at 127.0.0.1:54540
I20260709 12:23:48.806843 17687 raft_consensus.cc:2468] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 17.
I20260709 12:23:48.824888 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:48.827854 17683 raft_consensus.cc:1275] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 13 index: 9. Preceding OpId from leader: term: 20 index: 12. (index mismatch)
I20260709 12:23:48.836206 17837 raft_consensus.cc:2468] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 19.
I20260709 12:23:48.800733 24812 raft_consensus.cc:2749] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:23:48.800743 24657 raft_consensus.cc:493] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:23:48.817775 24704 raft_consensus.cc:2804] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:23:48.824780 24685 raft_consensus.cc:2749] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:23:48.827957 24716 consensus_queue.cc:1059] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 17.13, Next index: 14, Last known committed idx: 11, Time since last communication: 0.038s
I20260709 12:23:48.865875 17682 raft_consensus.cc:1240] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 19. Current term is 20. Ops: []
I20260709 12:23:48.866611 17802 leader_election.cc:304] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:48.870533 17802 leader_election.cc:400] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 17 pre-election: Vote denied by peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071) with higher term. Message: Invalid argument: T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 17. Current term is 18.
I20260709 12:23:48.872637 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:48.844954 24813 consensus_queue.cc:1059] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 16.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.039s
I20260709 12:23:48.900673 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:48.867620 24637 raft_consensus.cc:2749] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:23:48.860730 24677 leader_election.cc:290] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 19 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:48.914846 17683 raft_consensus.cc:1240] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 21 LEADER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 20. Current term is 21. Ops: []
I20260709 12:23:48.799820 24656 consensus_queue.cc:237] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.12, Last appended by leader: 12, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:48.920125 24721 raft_consensus.cc:3055] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:23:48.920562 24721 raft_consensus.cc:740] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 16 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:23:48.900756 24486 raft_consensus.cc:2749] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:23:48.919811 24687 consensus_queue.cc:1059] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 14.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:23:48.860034 24688 consensus_queue.cc:1048] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.054s
W20260709 12:23:48.921815 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 263398us
I20260709 12:23:48.928264 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:48.872818 24704 raft_consensus.cc:697] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 15 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:23:48.907815 24723 leader_election.cc:290] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:23:48.930294 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 125801us
I20260709 12:23:48.828377 24642 consensus_queue.cc:260] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 21.15, Last appended by leader: 15, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:48.858960 24657 raft_consensus.cc:515] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:48.935788 17682 raft_consensus.cc:3060] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:23:48.910600 24809 consensus_queue.cc:1048] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 9, Time since last communication: 0.026s
I20260709 12:23:48.935968 17729 leader_election.cc:304] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:48.940271 18364 raft_consensus.cc:1275] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 15 index: 13. Preceding OpId from leader: term: 17 index: 14. (index mismatch)
I20260709 12:23:48.949884 17803 leader_election.cc:304] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:48.953787 17803 leader_election.cc:304] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:23:48.929082 24800 consensus_queue.cc:1059] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 19.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.022s
I20260709 12:23:48.960786 17803 leader_election.cc:304] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:48.959345 24807 raft_consensus.cc:2804] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:23:48.959731 24813 consensus_queue.cc:1059] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.007s
W20260709 12:23:48.967233 24822 log.cc:927] Time spent T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.100s	user 0.001s	sys 0.000s
I20260709 12:23:48.970257 24716 raft_consensus.cc:2749] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
W20260709 12:23:48.976830 18333 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:48.980064 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:48.982470 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:48.952334 24313 raft_consensus.cc:3060] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:48.986012 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 17 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:48.986739 17684 raft_consensus.cc:2393] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 17: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
W20260709 12:23:48.989480 24821 log.cc:927] Time spent T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.050s	user 0.000s	sys 0.000s
I20260709 12:23:48.988291 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:23:48.993646 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:48.993528 17802 leader_election.cc:304] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:48.994397 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:48.995630 17684 raft_consensus.cc:2368] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 17. Current term is 18.
I20260709 12:23:48.999513 24657 leader_election.cc:290] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:49.000160 24704 consensus_queue.cc:237] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.10, Last appended by leader: 10, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:48.959735 24642 raft_consensus.cc:3060] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:23:48.963689 24721 consensus_queue.cc:260] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 16.14, Last appended by leader: 14, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:48.954456 24650 raft_consensus.cc:2749] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:23:49.002804 24709 leader_election.cc:290] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:49.003649 24705 leader_election.cc:290] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:48.948416 24687 consensus_queue.cc:1059] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 20.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.061s
I20260709 12:23:49.007076 17682 raft_consensus.cc:2468] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 22.
I20260709 12:23:49.008458 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:49.023526 17758 raft_consensus.cc:2393] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 19: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:23:49.008561 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:49.024255 17760 raft_consensus.cc:2468] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 21.
I20260709 12:23:49.012765 17762 raft_consensus.cc:1275] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 19 index: 14. (index mismatch)
I20260709 12:23:49.016479 24807 raft_consensus.cc:697] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 21 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
W20260709 12:23:49.017804 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 50459.9us
W20260709 12:23:49.004803 17801 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:49.026695 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:49.028081 17684 raft_consensus.cc:2393] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 17: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:23:49.030723 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:49.031635 17834 raft_consensus.cc:2393] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 14: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:23:49.033890 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:49.034587 24643 raft_consensus.cc:3060] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:49.034847 17836 raft_consensus.cc:3060] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:49.036655 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:49.036903 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 17 candidate_status { last_received { term: 14 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:49.038808 24675 raft_consensus.cc:3060] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:49.044591 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:49.045591 17682 raft_consensus.cc:2393] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 18: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:23:49.026552 17803 leader_election.cc:304] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:49.047046 17833 raft_consensus.cc:1240] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 15. Current term is 16. Ops: []
I20260709 12:23:48.999733 24662 leader_election.cc:290] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:49.050851 17803 leader_election.cc:400] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 17 election: Vote denied by peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577) with higher term. Message: Invalid argument: T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 17. Current term is 18.
I20260709 12:23:49.052306 17803 leader_election.cc:403] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 17 election: Cancelling election due to peer responding with higher term
I20260709 12:23:49.054250 17836 raft_consensus.cc:2468] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 15.
I20260709 12:23:49.013326 24819 raft_consensus.cc:493] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:49.055580 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:49.055454 24819 raft_consensus.cc:515] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:49.056353 17832 raft_consensus.cc:2393] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 18: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:23:49.056733 24392 raft_consensus.cc:3055] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:23:49.057513 17763 raft_consensus.cc:2468] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 18.
I20260709 12:23:49.058606 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:49.056412 24807 consensus_queue.cc:237] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 19.12, Last appended by leader: 12, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:49.063169 24818 raft_consensus.cc:2749] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:23:48.994917 24796 raft_consensus.cc:515] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:49.067008 17727 connection.cc:256] RPC call timeout handler was delayed by -1.0962774069998886s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1593s for a response
I20260709 12:23:49.067894 18364 raft_consensus.cc:1240] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 19. Current term is 20. Ops: []
I20260709 12:23:49.069023 17727 leader_election.cc:304] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:49.069743 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:49.071084 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:49.071952 17836 raft_consensus.cc:2468] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 16.
I20260709 12:23:49.073968 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:49.075634 17682 raft_consensus.cc:3060] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:49.065368 17688 raft_consensus.cc:2468] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 20.
I20260709 12:23:49.063072 24769 raft_consensus.cc:515] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:49.080947 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:49.082156 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:49.084055 17684 raft_consensus.cc:2393] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 18: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:23:49.084685 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:49.085188 17836 raft_consensus.cc:2468] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 14.
I20260709 12:23:49.018169 24835 raft_consensus.cc:2749] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:23:49.057334 24815 raft_consensus.cc:493] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:49.097910 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:49.074035 24662 raft_consensus.cc:493] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:49.094786 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:49.111698 17832 raft_consensus.cc:2468] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 18.
I20260709 12:23:49.094523 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:49.112897 17758 raft_consensus.cc:2468] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 14.
I20260709 12:23:49.098255 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:49.100422 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:49.051757 24810 raft_consensus.cc:493] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:23:49.102973 17682 raft_consensus.cc:2468] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 19.
I20260709 12:23:49.105028 17684 raft_consensus.cc:3060] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:49.092272 17756 raft_consensus.cc:1275] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 17 index: 14. Preceding OpId from leader: term: 18 index: 15. (index mismatch)
I20260709 12:23:49.072599 24392 raft_consensus.cc:740] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:23:49.057296 24709 consensus_queue.cc:1048] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 9, Time since last communication: 0.047s
I20260709 12:23:49.073186 24694 raft_consensus.cc:2749] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:23:49.114643 17680 raft_consensus.cc:606] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Received request to transfer leadership to TS e063541542a94969898868bf6eb5ce1b
I20260709 12:23:49.083144 24721 raft_consensus.cc:3060] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Advancing to term 17
W20260709 12:23:49.119633 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 126172us
I20260709 12:23:49.122221 18334 leader_election.cc:304] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:23:49.122460 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:49.080212 24688 raft_consensus.cc:493] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:49.134793 17838 raft_consensus.cc:1240] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 18. Current term is 19. Ops: [18.14-18.14]
I20260709 12:23:49.092175 24828 raft_consensus.cc:493] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:49.135349 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:49.091984 24656 raft_consensus.cc:2749] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Leader election lost for term 17. Reason: Vote denied by peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577) with higher term. Message: Invalid argument: T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 17. Current term is 18.
I20260709 12:23:49.137511 17756 raft_consensus.cc:1240] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 20. Current term is 21. Ops: []
I20260709 12:23:49.163738 17684 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 16.
I20260709 12:23:49.157833 17682 raft_consensus.cc:2468] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 18.
I20260709 12:23:49.156713 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:49.110749 24687 raft_consensus.cc:2749] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 12:23:49.159617 17680 raft_consensus.cc:614] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260709 12:23:49.161008 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:49.169685 17681 raft_consensus.cc:2393] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 13: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:23:49.169843 17838 raft_consensus.cc:1275] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 16 index: 10. Preceding OpId from leader: term: 17 index: 11. (index mismatch)
I20260709 12:23:49.169963 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:49.169843 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:49.171707 17831 raft_consensus.cc:2393] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 19: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:23:49.171739 17802 leader_election.cc:400] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 17 election: Vote denied by peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071) with higher term. Message: Invalid argument: T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 17. Current term is 18.
I20260709 12:23:49.172405 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:49.172679 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:49.173444 17832 raft_consensus.cc:2393] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 18: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:23:49.174013 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:49.158939 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:49.117352 24830 raft_consensus.cc:3055] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:23:49.157754 24709 consensus_queue.cc:1243] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer e2909cff1da44ad387f96a8ae3aa5afb log is divergent from this leader: its last log entry 17.11 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 9
I20260709 12:23:49.166045 17834 raft_consensus.cc:2393] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 19: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:23:49.174865 17836 raft_consensus.cc:3060] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:49.174211 17728 leader_election.cc:304] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:49.172405 17802 leader_election.cc:304] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:23:49.179378 24819 leader_election.cc:290] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:49.180254 24392 consensus_queue.cc:260] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:49.163465 24688 raft_consensus.cc:515] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:49.184340 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:49.184506 18333 leader_election.cc:304] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:49.165620 24431 raft_consensus.cc:2804] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:23:49.186728 24769 leader_election.cc:290] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 13 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:49.187506 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:49.187743 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:49.133387 24810 raft_consensus.cc:515] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:49.193516 24796 leader_election.cc:290] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 20 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:49.119256 24815 raft_consensus.cc:515] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:49.165443 17684 raft_consensus.cc:1240] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 18. Current term is 19. Ops: []
W20260709 12:23:49.195451 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 126470us
W20260709 12:23:49.217090 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 166411us
I20260709 12:23:49.225030 17727 leader_election.cc:304] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:49.196266 18361 raft_consensus.cc:1240] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 18 LEADER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 17. Current term is 18. Ops: []
I20260709 12:23:49.215694 18333 leader_election.cc:304] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:23:49.218672 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:49.194428 24830 raft_consensus.cc:740] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 20 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:23:49.228909 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:49.227773 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:49.122203 24662 raft_consensus.cc:515] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:49.226907 18333 leader_election.cc:304] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:23:49.234002 18333 leader_election.cc:304] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:23:49.234763 17803 leader_election.cc:304] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:23:49.235491 17727 leader_election.cc:304] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:49.237393 17756 raft_consensus.cc:1240] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 20. Current term is 21. Ops: []
I20260709 12:23:49.238605 18333 leader_election.cc:304] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:49.217092 24687 raft_consensus.cc:2804] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:23:49.217244 24656 raft_consensus.cc:2804] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:23:49.241336 18333 leader_election.cc:304] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:49.217864 24681 raft_consensus.cc:2749] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:23:49.242324 17803 leader_election.cc:304] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:49.226486 24392 raft_consensus.cc:3060] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:49.244866 17762 raft_consensus.cc:2410] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 17 because replica has last-logged OpId of term: 15 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 10.
I20260709 12:23:49.245793 24796 raft_consensus.cc:3060] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:49.248179 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:49.248462 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:49.170791 24828 raft_consensus.cc:515] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:49.248057 17836 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 16.
I20260709 12:23:49.253520 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:49.234143 24431 raft_consensus.cc:697] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 19 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:23:49.280766 24688 leader_election.cc:290] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:49.281004 24431 consensus_queue.cc:237] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.13, Last appended by leader: 13, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:49.239782 24650 raft_consensus.cc:2804] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Leader pre-election won for term 15
W20260709 12:23:49.284353 24824 log.cc:927] Time spent T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.278s	user 0.002s	sys 0.000s
I20260709 12:23:49.284617 24650 raft_consensus.cc:493] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:49.285156 24650 raft_consensus.cc:3060] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:49.252624 24834 consensus_queue.cc:1059] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 18.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.044s
I20260709 12:23:49.288926 24665 raft_consensus.cc:3055] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:23:49.289428 24665 raft_consensus.cc:740] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 17 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:23:49.290299 24665 consensus_queue.cc:260] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.13, Last appended by leader: 13, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:49.291766 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:49.292536 17760 raft_consensus.cc:3060] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:23:49.293314 17756 raft_consensus.cc:1240] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 19. Current term is 20. Ops: []
I20260709 12:23:49.233708 24790 raft_consensus.cc:2804] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:23:49.248719 24809 raft_consensus.cc:2749] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:23:49.295465 24790 raft_consensus.cc:493] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:49.297631 24790 raft_consensus.cc:3060] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:49.299563 24767 leader_election.cc:290] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 18 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:23:49.301045 24825 log.cc:927] Time spent T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.304s	user 0.000s	sys 0.001s
I20260709 12:23:49.272518 24313 raft_consensus.cc:2749] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:23:49.272184 24844 consensus_queue.cc:1059] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.018s
I20260709 12:23:49.242658 24677 raft_consensus.cc:2804] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:23:49.243332 24687 raft_consensus.cc:493] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:49.244647 24807 consensus_queue.cc:1048] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.029s
I20260709 12:23:49.240386 24694 consensus_queue.cc:1048] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.034s
W20260709 12:23:49.275662 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:49.241686 24819 raft_consensus.cc:2804] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:23:49.244740 24231 raft_consensus.cc:2749] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 12:23:49.242311 24659 raft_consensus.cc:2804] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:23:49.233129 24713 raft_consensus.cc:493] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:49.293309 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:49.287145 17682 raft_consensus.cc:1240] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 20. Current term is 21. Ops: []
I20260709 12:23:49.310722 24834 raft_consensus.cc:3055] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:23:49.279145 24829 raft_consensus.cc:2749] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:23:49.312503 24815 leader_election.cc:290] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:49.274881 24830 consensus_queue.cc:260] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 20.15, Last appended by leader: 15, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:23:49.314838 24828 leader_election.cc:290] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:49.315596 24665 raft_consensus.cc:3060] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:49.316143 24839 consensus_queue.cc:1059] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 18.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:23:49.318562 24439 leader_election.cc:290] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:49.319828 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:23:49.321141 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 85718.7us
I20260709 12:23:49.326932 17760 raft_consensus.cc:2468] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 22.
I20260709 12:23:49.275928 24656 raft_consensus.cc:697] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 22 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:23:49.330076 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 17 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:49.343104 17803 leader_election.cc:304] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:49.383648 17687 raft_consensus.cc:2393] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 20: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:23:49.345167 24844 consensus_queue.cc:1243] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer e063541542a94969898868bf6eb5ce1b log is divergent from this leader: its last log entry 20.17 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 15
I20260709 12:23:49.328318 24677 raft_consensus.cc:493] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:23:49.387094 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 83145.3us
I20260709 12:23:49.388417 18363 raft_consensus.cc:1240] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 19. Current term is 20. Ops: [19.16-19.16]
I20260709 12:23:49.352494 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:49.332816 24687 raft_consensus.cc:3060] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:49.297648 24833 consensus_queue.cc:1059] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.053s
I20260709 12:23:49.383972 17758 raft_consensus.cc:2393] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 17: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:23:49.389940 17682 raft_consensus.cc:2376] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 20. Re-sending same reply.
I20260709 12:23:49.349519 17838 raft_consensus.cc:2468] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 17.
I20260709 12:23:49.342181 24819 raft_consensus.cc:493] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:23:49.390902 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:49.392782 17682 raft_consensus.cc:2376] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 22. Re-sending same reply.
I20260709 12:23:49.393769 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:49.394618 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:49.349597 24659 raft_consensus.cc:493] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:49.396554 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:49.398630 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:49.399101 17682 raft_consensus.cc:2393] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 18: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:23:49.400416 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 21 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:49.362792 24834 raft_consensus.cc:740] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 18 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:23:49.352270 24709 consensus_queue.cc:1048] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.041s
I20260709 12:23:49.400998 17687 raft_consensus.cc:2468] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 20.
I20260709 12:23:49.403844 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:49.406018 17728 leader_election.cc:304] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:49.392426 24677 raft_consensus.cc:3060] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:49.406764 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:49.387506 24853 raft_consensus.cc:493] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:49.400642 24659 raft_consensus.cc:3060] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:49.409706 17727 leader_election.cc:304] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:49.395687 24819 raft_consensus.cc:3060] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:49.407599 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:49.418154 17727 leader_election.cc:304] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:49.417100 17687 raft_consensus.cc:2468] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 18.
I20260709 12:23:49.367024 24713 raft_consensus.cc:515] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:49.420158 17728 leader_election.cc:304] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:49.405797 24809 raft_consensus.cc:2749] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:23:49.385098 24829 consensus_queue.cc:1059] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true }, Status: INVALID_TERM, Last received: 17.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.024s
I20260709 12:23:49.422925 17687 raft_consensus.cc:1240] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 18. Current term is 19. Ops: [18.11-18.11]
W20260709 12:23:49.427572 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:49.395368 24838 consensus_queue.cc:1059] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 20.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.034s
I20260709 12:23:49.428369 24814 raft_consensus.cc:2804] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:23:49.428994 24814 raft_consensus.cc:493] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:49.429445 24814 raft_consensus.cc:3060] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:49.431077 18361 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59886 (request call id 1859) took 1592 ms. Trace:
I20260709 12:23:49.432821 24662 leader_election.cc:290] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:23:49.433096 24849 log.cc:927] Time spent T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.091s	user 0.000s	sys 0.000s
I20260709 12:23:49.414369 24807 consensus_queue.cc:1059] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 18.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.021s
I20260709 12:23:49.420572 24439 raft_consensus.cc:3060] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:23:49.439807 18371 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59902 (request call id 1664) took 1713 ms (client timeout 1923 ms). Trace:
W20260709 12:23:49.440477 18371 rpcz_store.cc:269] 0709 12:23:47.726830 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:47.728115 (+  1285us) service_pool.cc:224] Handling call
0709 12:23:49.439522 (+1711407us) spinlock_profiling.cc:243] Waited 107 ms on lock 0x7b54007701d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:23:49.439720 (+   198us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":247001856}
W20260709 12:23:49.442735 24850 log.cc:927] Time spent T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.057s	user 0.000s	sys 0.000s
I20260709 12:23:49.442960 17680 raft_consensus.cc:1275] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 19 index: 10. Preceding OpId from leader: term: 20 index: 11. (index mismatch)
I20260709 12:23:49.445575 24695 raft_consensus.cc:3055] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:23:49.446157 24695 raft_consensus.cc:740] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 19 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:23:49.431509 18361 rpcz_store.cc:276] 0709 12:23:47.838301 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:47.862605 (+ 24304us) service_pool.cc:224] Handling call
0709 12:23:47.862916 (+   311us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:49.225862 (+1362946us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:49.430864 (+205002us) spinlock_profiling.cc:243] Waited 1.33 s on lock 0x7b5400802d54. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:23:49.431020 (+   156us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3065868544}
I20260709 12:23:49.449384 24759 raft_consensus.cc:3055] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:23:49.455281 17727 leader_election.cc:400] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 21 election: Vote denied by peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837) with higher term. Message: Invalid argument: T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for earlier term 21. Current term is 22.
I20260709 12:23:49.398140 24815 consensus_queue.cc:1059] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.021s
I20260709 12:23:49.465140 17727 leader_election.cc:403] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 21 election: Cancelling election due to peer responding with higher term
I20260709 12:23:49.464639 17687 raft_consensus.cc:1240] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 17. Current term is 18. Ops: []
I20260709 12:23:49.467114 17680 raft_consensus.cc:1275] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 18 index: 13. (index mismatch)
I20260709 12:23:49.469713 17689 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9"
dest_uuid: "0da169ff82674dc1947055ce43de68a3"
mode: GRACEFUL
new_leader_uuid: "e063541542a94969898868bf6eb5ce1b"
 from {username='slave'} at 127.0.0.1:54540
I20260709 12:23:49.470494 17689 raft_consensus.cc:606] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 22 LEADER]: Received request to transfer leadership to TS e063541542a94969898868bf6eb5ce1b
I20260709 12:23:49.472009 24834 consensus_queue.cc:260] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.13, Last appended by leader: 13, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:49.482105 24716 raft_consensus.cc:3055] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:23:49.482547 24716 raft_consensus.cc:740] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 19 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:23:49.484493 24856 raft_consensus.cc:3055] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:23:49.484905 24856 raft_consensus.cc:740] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 20 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:23:49.485742 24856 consensus_queue.cc:260] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 20.12, Last appended by leader: 12, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:49.488282 17652 leader_election.cc:304] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da
W20260709 12:23:49.490774 24827 log.cc:927] Time spent T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.489s	user 0.001s	sys 0.001s
I20260709 12:23:49.491150 24713 leader_election.cc:290] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:49.483428 24819 raft_consensus.cc:515] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:49.426195 24313 raft_consensus.cc:2749] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:23:49.497181 17685 raft_consensus.cc:1240] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 12. Current term is 13. Ops: [12.11-12.11]
I20260709 12:23:49.418099 24844 raft_consensus.cc:2804] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Leader pre-election won for term 21
W20260709 12:23:49.492952 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 96738.1us
I20260709 12:23:49.414613 24828 consensus_queue.cc:1048] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.056s
I20260709 12:23:49.560343 24815 consensus_queue.cc:1048] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:23:49.561630 24829 raft_consensus.cc:2749] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Leader election lost for term 21. Reason: Vote denied by peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837) with higher term. Message: Invalid argument: T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for earlier term 21. Current term is 22.
I20260709 12:23:49.565001 17654 leader_election.cc:304] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:23:49.469151 24759 raft_consensus.cc:740] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 17 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:23:49.423206 24675 consensus_queue.cc:1048] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.074s
I20260709 12:23:49.552380 24694 raft_consensus.cc:493] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:49.447324 24695 consensus_queue.cc:260] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 19.10, Last appended by leader: 10, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:49.552613 24844 raft_consensus.cc:493] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:23:49.587709 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 138229us
I20260709 12:23:49.591127 17728 leader_election.cc:304] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
W20260709 12:23:49.596223 17686 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44898 (request call id 1325) took 2047 ms (client timeout 1847 ms). Trace:
W20260709 12:23:49.596901 17686 rpcz_store.cc:269] 0709 12:23:47.549133 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:47.549296 (+   163us) service_pool.cc:224] Handling call
0709 12:23:49.595928 (+2046632us) spinlock_profiling.cc:243] Waited 332 ms on lock 0x7b5400208354. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:23:49.596129 (+   201us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":764684800}
I20260709 12:23:49.417141 24853 raft_consensus.cc:515] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:49.609158 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 212752us
I20260709 12:23:49.578080 24439 raft_consensus.cc:2749] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:23:49.451155 24656 consensus_queue.cc:237] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 20.11, Last appended by leader: 11, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:49.605286 24833 consensus_queue.cc:1243] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer 0da169ff82674dc1947055ce43de68a3 log is divergent from this leader: its last log entry 21.11 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 10
I20260709 12:23:49.589401 24844 raft_consensus.cc:3060] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:49.568696 24858 consensus_queue.cc:1059] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 20.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.115s
I20260709 12:23:49.577900 24694 raft_consensus.cc:515] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:49.618278 24861 consensus_queue.cc:1059] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:23:49.609683 24313 raft_consensus.cc:2804] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:23:49.625485 24853 leader_election.cc:290] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:49.627050 24796 raft_consensus.cc:2749] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:23:49.589182 24659 raft_consensus.cc:515] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:49.629653 24659 leader_election.cc:290] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 19 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:49.574420 24848 consensus_queue.cc:1059] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 18.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.057s
W20260709 12:23:49.656704 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:49.662261 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 45716.1us
I20260709 12:23:49.664074 24861 consensus_queue.cc:1059] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20260709 12:23:49.643150 24814 raft_consensus.cc:515] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:23:49.668246 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 213159us
I20260709 12:23:49.670950 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:49.671464 17836 raft_consensus.cc:2468] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 16.
W20260709 12:23:49.672662 17802 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:49.673786 17801 leader_election.cc:304] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:49.675768 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:49.633353 24694 leader_election.cc:290] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:49.623871 24313 raft_consensus.cc:493] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
W20260709 12:23:49.667383 24859 log.cc:927] Time spent T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.058s	user 0.001s	sys 0.000s
W20260709 12:23:49.667559 24854 log.cc:927] Time spent T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.186s	user 0.000s	sys 0.000s
I20260709 12:23:49.678352 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:49.681635 17836 raft_consensus.cc:1217] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 20 LEADER]: Deduplicated request from leader. Original: 17.15->[18.16-19.17]   Dedup: 18.16->[19.17-19.17]
I20260709 12:23:49.728143 17836 raft_consensus.cc:1240] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 20 LEADER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 19. Current term is 20. Ops: [18.16-19.17]
I20260709 12:23:49.665782 24677 raft_consensus.cc:515] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:49.729197 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:49.729923 17684 raft_consensus.cc:2468] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 18.
I20260709 12:23:49.730437 24677 leader_election.cc:290] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 15 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:49.634163 24790 raft_consensus.cc:515] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:49.731215 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:49.732064 17835 raft_consensus.cc:2393] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 20: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:23:49.732623 17688 raft_consensus.cc:1240] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 18. Current term is 19. Ops: []
I20260709 12:23:49.684795 24579 leader_election.cc:290] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:49.679488 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:49.730181 17802 leader_election.cc:304] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:49.686136 24801 leader_election.cc:290] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:49.734045 24801 raft_consensus.cc:2804] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:23:49.734509 24801 raft_consensus.cc:697] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 20 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:23:49.734637 24790 leader_election.cc:290] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 17 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:49.689134 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 21 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:49.734927 17802 leader_election.cc:304] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:23:49.692687 24826 log.cc:927] Time spent T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.695s	user 0.007s	sys 0.008s
I20260709 12:23:49.735348 24801 consensus_queue.cc:237] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.12, Last appended by leader: 12, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:49.736073 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:49.737310 17802 leader_election.cc:304] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:49.738044 24685 raft_consensus.cc:2749] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:23:49.695847 24814 leader_election.cc:290] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 20 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:49.697834 24819 leader_election.cc:290] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 16 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:49.739871 17802 leader_election.cc:304] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:49.698679 24650 raft_consensus.cc:515] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:49.647078 24759 consensus_queue.cc:260] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.12, Last appended by leader: 12, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:49.704005 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 215814us
I20260709 12:23:49.743213 17652 leader_election.cc:304] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:23:49.704449 18333 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:49.705436 24705 raft_consensus.cc:2804] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:23:49.705868 24723 raft_consensus.cc:493] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:23:49.705978 24695 raft_consensus.cc:3060] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Advancing to term 20
W20260709 12:23:49.706655 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 141790us
I20260709 12:23:49.707020 24823 raft_consensus.cc:2804] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:23:49.707767 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:49.709528 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 17 candidate_status { last_received { term: 14 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:49.631800 24687 raft_consensus.cc:515] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:49.713479 17829 raft_consensus.cc:1240] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 20 LEADER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 19. Current term is 20. Ops: [17.11-19.12]
I20260709 12:23:49.739319 24313 raft_consensus.cc:3060] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:49.674991 24844 raft_consensus.cc:515] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:49.776096 24639 consensus_queue.cc:1059] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20260709 12:23:49.718281 24828 raft_consensus.cc:493] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:49.723001 24721 raft_consensus.cc:3055] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 20 LEADER]: Stepping down as leader of term 20
W20260709 12:23:49.726083 24855 log.cc:927] Time spent T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.120s	user 0.001s	sys 0.000s
I20260709 12:23:49.732836 17727 leader_election.cc:304] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:23:49.801056 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 68304.7us
I20260709 12:23:49.782274 24721 raft_consensus.cc:740] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 20 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:23:49.801720 24815 raft_consensus.cc:493] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:23:49.802230 24815 raft_consensus.cc:515] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:49.802438 24721 consensus_queue.cc:260] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 20.11, Last appended by leader: 11, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:49.802907 24814 raft_consensus.cc:2804] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:23:49.803377 24814 raft_consensus.cc:493] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:49.803807 24814 raft_consensus.cc:3060] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:49.732102 17681 raft_consensus.cc:2468] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 13.
W20260709 12:23:49.691483 17729 consensus_peers.cc:597] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb -> Peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Couldn't send request to peer 334a4e1f59944544b3b3ff36616b57da. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e2bcfe1b94464b6d9620cc3b8a17a2ad. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:23:49.806013 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 859460us
I20260709 12:23:49.738672 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:49.733595 17833 raft_consensus.cc:2468] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 17.
I20260709 12:23:49.748569 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:49.809368 18374 raft_consensus.cc:2368] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 21. Current term is 22.
I20260709 12:23:49.751773 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 20 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:49.810375 18371 raft_consensus.cc:2410] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 20 because replica has last-logged OpId of term: 18 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 13.
I20260709 12:23:49.753922 18364 raft_consensus.cc:1275] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 17 index: 11. Preceding OpId from leader: term: 21 index: 13. (index mismatch)
W20260709 12:23:49.755244 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 81544us
I20260709 12:23:49.758883 17652 leader_election.cc:304] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:49.750124 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 17 candidate_status { last_received { term: 14 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:49.811061 17801 leader_election.cc:304] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:49.763279 17802 leader_election.cc:304] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:49.765553 17689 raft_consensus.cc:2468] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 16.
I20260709 12:23:49.687743 17685 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44898 (request call id 1323) took 2198 ms. Trace:
I20260709 12:23:49.775717 24823 raft_consensus.cc:697] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 17 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:23:49.779429 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:49.818573 17684 raft_consensus.cc:3060] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Advancing to term 20
I20260709 12:23:49.780174 17687 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9"
dest_uuid: "0da169ff82674dc1947055ce43de68a3"
mode: GRACEFUL
new_leader_uuid: "334a4e1f59944544b3b3ff36616b57da"
 from {username='slave'} at 127.0.0.1:54540
I20260709 12:23:49.819237 17687 raft_consensus.cc:606] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Received request to transfer leadership to TS 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:49.819480 17687 raft_consensus.cc:614] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:23:49.760262 24486 raft_consensus.cc:2749] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:23:49.766528 24820 raft_consensus.cc:2804] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:23:49.777069 24639 raft_consensus.cc:3055] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:23:49.756644 24829 consensus_queue.cc:1059] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 17.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.027s
I20260709 12:23:49.768654 24816 consensus_queue.cc:1048] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.015s
I20260709 12:23:49.777285 24723 raft_consensus.cc:515] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:49.789692 24650 leader_election.cc:290] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 15 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:49.780303 24638 consensus_queue.cc:1048] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.013s
W20260709 12:23:49.791155 17728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:49.796200 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:49.797868 24687 leader_election.cc:290] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 19 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:49.756999 24705 raft_consensus.cc:493] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:23:49.799021 18365 raft_consensus.cc:1240] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 19. Current term is 20. Ops: [19.16-19.16]
I20260709 12:23:49.799613 24844 leader_election.cc:290] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 21 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:49.800009 24828 raft_consensus.cc:515] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:49.809587 17729 leader_election.cc:304] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:49.803622 24721 raft_consensus.cc:3060] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:49.812371 17685 rpcz_store.cc:276] 0709 12:23:47.489648 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:47.492613 (+  2965us) service_pool.cc:224] Handling call
0709 12:23:49.450674 (+1958061us) spinlock_profiling.cc:243] Waited 10.1 ms on lock 0x7b5800002a98. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b474fbb71 00007f7b474fba22 00007f7b504148d9 00007f7b4f56eee0 00007f7b4f6380ab 00007f7b4f638131 00007f7b4f5f9fc0 00007f7b4f5e4929 00007f7b4f5e43d2 00007f7b48b5bbd4
0709 12:23:49.495145 (+ 44471us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:49.552253 (+ 57108us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:49.687480 (+135227us) spinlock_profiling.cc:243] Waited 44.2 ms on lock 0x7b54003471d0. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a61f8ad 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:23:49.687650 (+   170us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":124962432}
I20260709 12:23:49.813104 17803 leader_election.cc:400] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 21 pre-election: Vote denied by peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577) with higher term. Message: Invalid argument: T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 21. Current term is 22.
W20260709 12:23:49.816718 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 57876.3us
W20260709 12:23:49.848110 17804 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:49.848114 17803 leader_election.cc:403] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 21 pre-election: Cancelling election due to peer responding with higher term
W20260709 12:23:49.817391 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 57346.4us
I20260709 12:23:49.815990 24838 raft_consensus.cc:3060] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:49.829838 18363 raft_consensus.cc:3060] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Advancing to term 15
I20260709 12:23:49.832239 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:49.854328 17687 raft_consensus.cc:3060] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:49.854631 17836 raft_consensus.cc:1275] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 13 index: 10. Preceding OpId from leader: term: 15 index: 11. (index mismatch)
I20260709 12:23:49.848979 17652 leader_election.cc:304] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:23:49.857627 17652 leader_election.cc:304] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:23:49.861060 17684 raft_consensus.cc:2468] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 20.
W20260709 12:23:49.862264 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 49298.6us
I20260709 12:23:49.862811 17802 leader_election.cc:304] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:23:49.864969 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 21 candidate_status { last_received { term: 18 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:49.866291 17835 raft_consensus.cc:1275] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 13 index: 14. Preceding OpId from leader: term: 14 index: 15. (index mismatch)
W20260709 12:23:49.868592 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 59094.3us
I20260709 12:23:49.821414 24639 raft_consensus.cc:740] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:23:49.871376 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:49.821267 24656 raft_consensus.cc:2749] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:23:49.821465 24769 raft_consensus.cc:2804] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:23:49.873106 17681 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6"
dest_uuid: "0da169ff82674dc1947055ce43de68a3"
mode: GRACEFUL
new_leader_uuid: "334a4e1f59944544b3b3ff36616b57da"
 from {username='slave'} at 127.0.0.1:54540
I20260709 12:23:49.876706 23838 consensus_queue.cc:1059] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 10.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 12:23:49.822713 24685 raft_consensus.cc:2749] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:23:49.824066 24815 leader_election.cc:290] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:49.876919 17681 raft_consensus.cc:606] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Received request to transfer leadership to TS 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:49.877501 23838 raft_consensus.cc:3055] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:23:49.880270 17681 raft_consensus.cc:614] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:23:49.880350 23838 raft_consensus.cc:740] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:23:49.836534 24705 raft_consensus.cc:3060] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:49.876999 17727 leader_election.cc:304] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:49.822355 24820 raft_consensus.cc:697] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 19 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:23:49.874294 17833 raft_consensus.cc:3060] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:49.890784 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:49.882395 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:49.875531 24687 raft_consensus.cc:2749] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 12:23:49.891391 17836 raft_consensus.cc:3060] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:49.885699 18363 raft_consensus.cc:2468] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 15.
I20260709 12:23:49.870081 24708 raft_consensus.cc:2804] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:23:49.875463 24694 raft_consensus.cc:3060] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Advancing to term 22
I20260709 12:23:49.882109 24486 consensus_queue.cc:1048] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.026s
I20260709 12:23:49.910509 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 21 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:49.911504 23838 consensus_queue.cc:260] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:49.914484 23838 raft_consensus.cc:3060] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:49.892113 24579 raft_consensus.cc:2804] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:23:49.928962 17836 raft_consensus.cc:2468] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 17.
I20260709 12:23:49.951495 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:23:49.958062 17804 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:49.886479 24865 consensus_queue.cc:1059] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.053s
I20260709 12:23:49.895299 24818 raft_consensus.cc:2804] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:23:49.908924 24864 raft_consensus.cc:3060] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Advancing to term 20
W20260709 12:23:49.938655 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:49.943387 17833 raft_consensus.cc:2468] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 19.
I20260709 12:23:49.886193 24823 consensus_queue.cc:237] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 15.13, Last appended by leader: 13, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:49.929869 24814 raft_consensus.cc:515] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:49.933447 24579 raft_consensus.cc:493] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:49.934206 24708 raft_consensus.cc:697] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 21 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
W20260709 12:23:49.977936 17803 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:49.952507 17836 raft_consensus.cc:2393] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 16: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:23:49.947229 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:49.886153 24716 consensus_queue.cc:260] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 19.16, Last appended by leader: 16, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:49.932664 24687 consensus_queue.cc:1048] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.043s
I20260709 12:23:49.955384 24638 raft_consensus.cc:493] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:49.987756 24853 consensus_queue.cc:1059] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 18.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:23:49.990300 24830 raft_consensus.cc:3060] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Advancing to term 21
W20260709 12:23:49.992975 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:49.931499 24639 consensus_queue.cc:260] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 19.14, Last appended by leader: 14, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:50.004474 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:50.005023 17683 raft_consensus.cc:1275] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 13 index: 14. Preceding OpId from leader: term: 14 index: 15. (index mismatch)
I20260709 12:23:49.975853 24865 consensus_queue.cc:1243] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer e063541542a94969898868bf6eb5ce1b log is divergent from this leader: its last log entry 20.17 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 15
I20260709 12:23:49.974869 24839 raft_consensus.cc:2749] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:23:49.984266 24579 raft_consensus.cc:3060] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:50.011739 17687 raft_consensus.cc:2468] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 19.
I20260709 12:23:50.032579 17803 leader_election.cc:304] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:23:49.987345 24818 raft_consensus.cc:697] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 17 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:23:50.036777 24486 raft_consensus.cc:2804] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:23:50.037492 24486 raft_consensus.cc:697] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 15 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:23:50.038517 24486 consensus_queue.cc:237] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.11, Last appended by leader: 11, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:50.040596 24818 consensus_queue.cc:237] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.10, Last appended by leader: 10, Current term: 17, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:50.043151 24759 raft_consensus.cc:3060] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:50.052883 24814 leader_election.cc:290] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 17 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:50.046092 24864 raft_consensus.cc:2749] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:23:50.054842 17837 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35040 (request call id 3029) took 1253 ms. Trace:
I20260709 12:23:50.055466 17837 rpcz_store.cc:276] 0709 12:23:48.801767 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:48.801983 (+   216us) service_pool.cc:224] Handling call
0709 12:23:50.054605 (+1252622us) spinlock_profiling.cc:243] Waited 10.3 ms on lock 0x7b54003340d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:23:50.054765 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":23728896}
I20260709 12:23:50.056536 17838 raft_consensus.cc:1275] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 14 index: 14. Preceding OpId from leader: term: 16 index: 15. (index mismatch)
I20260709 12:23:50.032567 24639 raft_consensus.cc:3060] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:50.058583 24524 raft_consensus.cc:3055] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:23:50.059116 24524 raft_consensus.cc:740] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:23:50.032894 24723 leader_election.cc:290] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:50.060209 24524 consensus_queue.cc:260] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.11, Last appended by leader: 11, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:50.063326 17838 raft_consensus.cc:1217] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 20 LEADER]: Deduplicated request from leader. Original: 17.15->[18.16-19.17]   Dedup: 18.16->[19.17-19.17]
I20260709 12:23:50.064888 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:50.066118 17835 raft_consensus.cc:2393] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 17: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:23:50.067966 24643 raft_consensus.cc:493] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:50.068678 24643 raft_consensus.cc:515] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:50.047161 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 21 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:50.075204 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:50.077651 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:50.097705 17687 raft_consensus.cc:1275] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 14 index: 14. Preceding OpId from leader: term: 16 index: 15. (index mismatch)
I20260709 12:23:50.081624 17838 raft_consensus.cc:1240] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 20 LEADER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 19. Current term is 20. Ops: [18.16-19.17]
I20260709 12:23:50.098716 17802 leader_election.cc:304] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:23:50.079412 17685 raft_consensus.cc:1275] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 13 index: 10. Preceding OpId from leader: term: 15 index: 11. (index mismatch)
I20260709 12:23:50.072036 23838 raft_consensus.cc:493] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:50.099792 17802 leader_election.cc:304] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:50.099866 23838 raft_consensus.cc:515] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:50.107319 17680 raft_consensus.cc:3060] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:50.098053 17831 raft_consensus.cc:2468] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 15.
W20260709 12:23:50.111563 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 254035us
I20260709 12:23:50.111474 17687 raft_consensus.cc:1240] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 21 LEADER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 20. Current term is 21. Ops: [20.11-20.11]
I20260709 12:23:50.115020 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:23:50.116595 17728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:50.120939 17652 leader_election.cc:304] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
W20260709 12:23:50.074606 24868 log.cc:927] Time spent T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.068s	user 0.000s	sys 0.000s
I20260709 12:23:50.136808 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:50.138484 17687 raft_consensus.cc:2468] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 21.
I20260709 12:23:50.142405 23838 leader_election.cc:290] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:50.072151 24431 raft_consensus.cc:493] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:50.111501 24816 raft_consensus.cc:493] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:50.112326 24723 raft_consensus.cc:493] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:50.151041 17755 raft_consensus.cc:1217] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Deduplicated request from leader. Original: 13.9->[15.10-20.12]   Dedup: 15.10->[18.11-20.12]
I20260709 12:23:50.073455 24810 leader_election.cc:290] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:50.146261 17763 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38776 (request call id 2738) took 1124 ms. Trace:
I20260709 12:23:50.146869 17680 raft_consensus.cc:1275] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 19 index: 12. Preceding OpId from leader: term: 21 index: 13. (index mismatch)
I20260709 12:23:50.149775 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:50.155723 17757 raft_consensus.cc:3060] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:50.149858 17756 raft_consensus.cc:1240] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 LEADER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 19. Current term is 20. Ops: []
I20260709 12:23:50.149701 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:50.158622 17764 raft_consensus.cc:2393] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 20: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:23:50.110750 24643 leader_election.cc:290] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:50.168203 17832 raft_consensus.cc:1275] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 19 index: 10. Preceding OpId from leader: term: 20 index: 11. (index mismatch)
I20260709 12:23:50.151584 17755 raft_consensus.cc:1240] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 20. Current term is 21. Ops: [15.10-20.12]
I20260709 12:23:50.073251 24856 raft_consensus.cc:3060] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:50.071794 24708 consensus_queue.cc:237] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 20.14, Last appended by leader: 14, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:50.013744 24313 raft_consensus.cc:515] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:50.135272 24864 consensus_queue.cc:1059] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.033s
I20260709 12:23:50.114724 24713 raft_consensus.cc:2804] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:23:50.116058 24807 raft_consensus.cc:2804] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:23:50.016242 24638 raft_consensus.cc:515] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:50.154881 17760 raft_consensus.cc:1275] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 16 index: 12. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
I20260709 12:23:50.154383 17763 rpcz_store.cc:276] 0709 12:23:49.021662 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:49.021794 (+   132us) service_pool.cc:224] Handling call
0709 12:23:50.146091 (+1124297us) spinlock_profiling.cc:243] Waited 20.2 ms on lock 0x7b54003eb054. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:23:50.146204 (+   113us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":46404864}
I20260709 12:23:50.165737 24828 leader_election.cc:290] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:50.141124 24814 raft_consensus.cc:493] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:23:49.981952 24820 consensus_queue.cc:237] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.10, Last appended by leader: 10, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:23:50.170661 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:50.201865 17762 raft_consensus.cc:2468] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 13.
I20260709 12:23:50.171490 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 20 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:50.203189 17758 raft_consensus.cc:2410] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 20 because replica has last-logged OpId of term: 18 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 13.
I20260709 12:23:50.137691 24524 raft_consensus.cc:3060] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:50.182703 17680 raft_consensus.cc:1275] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 18 index: 13. Preceding OpId from leader: term: 19 index: 14. (index mismatch)
I20260709 12:23:50.204815 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:23:50.201377 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 331784us
I20260709 12:23:50.206223 17729 leader_election.cc:304] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:50.180575 24769 raft_consensus.cc:697] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 20 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:23:50.186200 24865 raft_consensus.cc:493] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:50.173763 24802 raft_consensus.cc:3060] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:50.193848 24807 raft_consensus.cc:493] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:50.198714 24864 consensus_queue.cc:1243] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer e063541542a94969898868bf6eb5ce1b log is divergent from this leader: its last log entry 20.17 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 15
I20260709 12:23:50.192548 17757 raft_consensus.cc:2468] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 15.
I20260709 12:23:50.193679 24713 raft_consensus.cc:493] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:50.187201 24819 raft_consensus.cc:493] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:50.220225 17728 leader_election.cc:304] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
W20260709 12:23:50.220346 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:50.224216 17802 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:50.225203 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:50.172776 24723 raft_consensus.cc:515] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:50.172304 24816 raft_consensus.cc:515] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:50.240054 17801 leader_election.cc:304] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:50.168141 24431 raft_consensus.cc:515] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:50.241454 17763 raft_consensus.cc:1240] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 17. Current term is 18. Ops: []
I20260709 12:23:50.241683 23838 consensus_queue.cc:1048] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:23:50.209357 17729 leader_election.cc:304] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:50.243355 16501 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 6
W20260709 12:23:50.256335 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:50.199331 24695 consensus_queue.cc:1048] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.040s
I20260709 12:23:50.211275 24828 raft_consensus.cc:2749] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:23:50.236289 24807 raft_consensus.cc:3060] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Advancing to term 17
W20260709 12:23:50.266175 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 142899us
I20260709 12:23:50.238674 24713 raft_consensus.cc:3060] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:50.204563 24814 raft_consensus.cc:515] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:50.271057 18365 raft_consensus.cc:2468] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 20.
I20260709 12:23:50.247342 24864 raft_consensus.cc:2804] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:23:50.229154 24865 raft_consensus.cc:515] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:50.214005 24815 consensus_queue.cc:1059] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 18.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.055s
I20260709 12:23:50.215842 24853 consensus_queue.cc:1059] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.054s
I20260709 12:23:50.259676 24810 raft_consensus.cc:2749] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:23:50.241328 24643 consensus_queue.cc:1048] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.032s
W20260709 12:23:50.282063 17803 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:50.268187 24439 raft_consensus.cc:2749] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:23:50.252682 24392 consensus_queue.cc:1048] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.035s
I20260709 12:23:50.260426 24835 consensus_queue.cc:1048] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.029s
I20260709 12:23:50.245105 24819 raft_consensus.cc:515] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:50.246912 24705 raft_consensus.cc:515] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:50.325891 17804 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:50.324357 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:50.329156 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 91062.9us
I20260709 12:23:50.264998 24875 consensus_queue.cc:1048] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.048s
W20260709 12:23:50.335836 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 66353.1us
I20260709 12:23:50.303203 24853 consensus_queue.cc:1243] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer e2909cff1da44ad387f96a8ae3aa5afb log is divergent from this leader: its last log entry 17.11 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 9
I20260709 12:23:50.336741 17652 leader_election.cc:304] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:50.271278 24790 consensus_queue.cc:1048] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.036s
I20260709 12:23:50.282665 24876 raft_consensus.cc:2804] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:23:50.332536 17728 leader_election.cc:304] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:50.341276 17728 leader_election.cc:304] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:50.340889 17652 leader_election.cc:304] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:23:50.303510 24864 raft_consensus.cc:697] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 20 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
W20260709 12:23:50.346275 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 93033.3us
I20260709 12:23:50.347517 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
W20260709 12:23:50.340871 24869 log.cc:927] Time spent T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.199s	user 0.000s	sys 0.001s
W20260709 12:23:50.348565 17728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:50.351644 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 99538us
I20260709 12:23:50.354341 17729 leader_election.cc:304] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:50.356170 17729 leader_election.cc:304] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:50.304659 24877 consensus_queue.cc:1059] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.059s
I20260709 12:23:50.366993 17729 leader_election.cc:304] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:23:50.316354 24818 consensus_queue.cc:1048] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.035s
I20260709 12:23:50.326607 24801 consensus_queue.cc:1059] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 17.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.019s
I20260709 12:23:50.376081 24638 leader_election.cc:290] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:50.376466 24801 raft_consensus.cc:3055] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:23:50.376869 24801 raft_consensus.cc:740] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 17 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:23:50.377728 24801 consensus_queue.cc:260] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.13, Last appended by leader: 13, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:50.383791 24694 raft_consensus.cc:2749] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Leader pre-election lost for term 21. Reason: Vote denied by peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577) with higher term. Message: Invalid argument: T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 21. Current term is 22.
I20260709 12:23:50.350138 24439 raft_consensus.cc:2804] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:23:50.386008 24819 leader_election.cc:290] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:23:50.387565 24721 consensus_meta.cc:287] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.535s	user 0.010s	sys 0.000s
I20260709 12:23:50.374198 17761 raft_consensus.cc:3055] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:23:50.391579 17761 raft_consensus.cc:740] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:23:50.324544 24769 consensus_queue.cc:237] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.11, Last appended by leader: 11, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:50.392553 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:50.388033 24883 raft_consensus.cc:2749] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 12:23:50.389067 17763 raft_consensus.cc:1275] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 18 index: 12. Preceding OpId from leader: term: 20 index: 13. (index mismatch)
I20260709 12:23:50.389317 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:50.397241 18371 raft_consensus.cc:2468] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 17.
W20260709 12:23:50.389976 24874 log.cc:927] Time spent T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.122s	user 0.000s	sys 0.001s
W20260709 12:23:50.366767 24872 log.cc:927] Time spent T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.167s	user 0.001s	sys 0.001s
W20260709 12:23:50.369968 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 124573us
I20260709 12:23:50.398859 17654 leader_election.cc:304] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:23:50.400677 24816 leader_election.cc:290] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:50.402065 24814 leader_election.cc:290] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:50.413103 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:50.402441 24819 raft_consensus.cc:3060] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:50.404436 24431 leader_election.cc:290] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:50.410302 18334 leader_election.cc:304] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:23:50.415798 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:50.413775 18371 raft_consensus.cc:2468] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 15.
I20260709 12:23:50.408694 24877 raft_consensus.cc:2749] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 12:23:50.424930 17684 raft_consensus.cc:2468] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 15.
I20260709 12:23:50.428205 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:50.444932 17685 raft_consensus.cc:2468] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 19.
I20260709 12:23:50.436612 24643 consensus_queue.cc:1048] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:23:50.397246 24313 leader_election.cc:290] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 18 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:50.446514 17728 leader_election.cc:304] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:50.437685 24716 raft_consensus.cc:3060] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:50.438172 24392 raft_consensus.cc:2804] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Leader pre-election won for term 23
W20260709 12:23:50.452301 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 92180.1us
I20260709 12:23:50.447517 18334 leader_election.cc:304] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:23:50.399237 24882 raft_consensus.cc:2804] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:23:50.407481 24875 raft_consensus.cc:2764] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Leader pre-election decision vote started in defunct term 18: won
I20260709 12:23:50.458514 17652 leader_election.cc:304] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:23:50.460390 17654 leader_election.cc:304] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:23:50.420680 24816 raft_consensus.cc:2804] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:23:50.457497 24759 raft_consensus.cc:2804] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:23:50.466437 24431 raft_consensus.cc:2804] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:23:50.441280 24885 raft_consensus.cc:2804] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:23:50.465969 17652 leader_election.cc:304] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:23:50.459496 24392 raft_consensus.cc:493] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:50.420012 24721 consensus_queue.cc:1048] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.047s
I20260709 12:23:50.468329 24759 raft_consensus.cc:493] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:50.467401 24816 raft_consensus.cc:493] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:50.478367 24759 raft_consensus.cc:3060] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:50.478580 24816 raft_consensus.cc:3060] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:50.478195 24885 raft_consensus.cc:697] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:23:50.490513 24885 consensus_queue.cc:237] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 22.17, Last appended by leader: 17, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:50.471182 24431 raft_consensus.cc:493] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:50.477710 24834 raft_consensus.cc:3060] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Advancing to term 19
W20260709 12:23:50.521641 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 55755.9us
I20260709 12:23:50.453256 24864 consensus_queue.cc:237] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.12, Last appended by leader: 12, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:50.484792 24392 raft_consensus.cc:3060] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Advancing to term 23
W20260709 12:23:50.496625 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:50.540151 17760 raft_consensus.cc:1217] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Deduplicated request from leader. Original: 13.9->[15.10-20.12]   Dedup: 15.10->[18.11-20.12]
I20260709 12:23:50.560374 17760 raft_consensus.cc:1240] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 20. Current term is 21. Ops: [15.10-20.12]
W20260709 12:23:50.476830 24881 log.cc:927] Time spent T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.057s	user 0.000s	sys 0.000s
I20260709 12:23:50.563961 24643 raft_consensus.cc:493] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:50.530400 24816 raft_consensus.cc:515] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:50.564947 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:50.565305 24643 raft_consensus.cc:515] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:50.559121 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:50.566843 24816 leader_election.cc:290] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 18 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:23:50.563114 17804 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:50.569072 17836 raft_consensus.cc:1275] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 20 index: 11. Preceding OpId from leader: term: 21 index: 11. (term mismatch)
I20260709 12:23:50.569455 17836 pending_rounds.cc:85] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b: Aborting all ops after (but not including) 10
I20260709 12:23:50.569397 24883 raft_consensus.cc:493] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:50.569919 17836 pending_rounds.cc:107] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b: Aborting uncommitted NO_OP operation due to leader change: 20.11
I20260709 12:23:50.570248 17836 raft_consensus.cc:2889] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:50.570111 24883 raft_consensus.cc:515] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:50.571563 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:50.520558 24639 raft_consensus.cc:2804] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:23:50.572183 24639 raft_consensus.cc:493] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:50.572670 24639 raft_consensus.cc:3060] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:50.573055 24883 leader_election.cc:290] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:50.578626 17831 raft_consensus.cc:1240] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 18. Current term is 19. Ops: [18.11-18.11]
I20260709 12:23:50.581772 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:50.582808 17836 raft_consensus.cc:2468] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 17.
I20260709 12:23:50.585035 17830 raft_consensus.cc:1275] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 20 index: 14. Preceding OpId from leader: term: 22 index: 15. (index mismatch)
I20260709 12:23:50.543576 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:50.543087 17764 raft_consensus.cc:1275] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 20 index: 11. Preceding OpId from leader: term: 22 index: 12. (index mismatch)
I20260709 12:23:50.594643 17760 raft_consensus.cc:1275] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 18 index: 10. Preceding OpId from leader: term: 21 index: 11. (index mismatch)
I20260709 12:23:50.598320 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
W20260709 12:23:50.601135 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 234209us
I20260709 12:23:50.602052 17729 leader_election.cc:304] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:50.534477 24431 raft_consensus.cc:3060] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:50.531546 24705 leader_election.cc:290] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 16 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:50.610929 17762 raft_consensus.cc:1275] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 15 index: 13. Preceding OpId from leader: term: 17 index: 14. (index mismatch)
W20260709 12:23:50.604362 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 144045us
I20260709 12:23:50.604713 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:50.576133 16501 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:23:50.484820 24865 leader_election.cc:290] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:50.530623 24880 raft_consensus.cc:2764] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Leader pre-election decision vote started in defunct term 18: won
I20260709 12:23:50.494588 24892 raft_consensus.cc:2804] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:23:50.544049 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:50.605577 17652 leader_election.cc:304] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:50.613070 17759 raft_consensus.cc:2393] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 20: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:23:50.594728 24883 raft_consensus.cc:493] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:50.521162 24579 raft_consensus.cc:515] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:23:50.618794 24889 log.cc:927] Time spent T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.232s	user 0.002s	sys 0.002s
I20260709 12:23:50.618645 24807 raft_consensus.cc:515] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:50.607190 24818 raft_consensus.cc:493] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:50.567315 24643 leader_election.cc:290] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:50.632328 17729 leader_election.cc:304] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:50.644579 17652 leader_election.cc:304] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:50.608121 24439 raft_consensus.cc:697] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 19 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:23:50.638208 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:50.633126 24844 raft_consensus.cc:2749] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 12:23:50.648605 24887 raft_consensus.cc:2749] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:23:50.632545 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:50.645758 17729 leader_election.cc:304] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:23:50.645912 17685 raft_consensus.cc:1240] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 15. Current term is 16. Ops: []
I20260709 12:23:50.648398 24882 raft_consensus.cc:697] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 18 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:23:50.648624 24880 raft_consensus.cc:2749] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:23:50.650336 17764 raft_consensus.cc:2468] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 17.
W20260709 12:23:50.650542 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:50.645224 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 22 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:50.631243 24892 raft_consensus.cc:493] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:50.656446 17654 leader_election.cc:304] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
W20260709 12:23:50.653024 24888 log.cc:927] Time spent T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.090s	user 0.001s	sys 0.000s
I20260709 12:23:50.664654 17652 leader_election.cc:304] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:50.665696 17729 leader_election.cc:304] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:23:50.667833 17729 leader_election.cc:304] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:50.620317 24863 consensus_queue.cc:1059] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.031s
I20260709 12:23:50.671867 17652 leader_election.cc:304] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:23:50.669380 24887 raft_consensus.cc:2804] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:23:50.675558 24887 raft_consensus.cc:493] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:50.676118 24887 raft_consensus.cc:3060] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:50.678969 24759 raft_consensus.cc:515] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:50.673357 24863 consensus_queue.cc:1243] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer e2909cff1da44ad387f96a8ae3aa5afb log is divergent from this leader: its last log entry 17.11 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 9
I20260709 12:23:50.673492 24823 raft_consensus.cc:2749] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260709 12:23:50.680159 24439 consensus_queue.cc:237] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 17.10, Last appended by leader: 10, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:50.638532 24709 consensus_queue.cc:1048] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.029s
I20260709 12:23:50.690043 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:50.685858 18363 raft_consensus.cc:1240] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 19. Current term is 20. Ops: []
I20260709 12:23:50.688663 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:50.638813 24883 raft_consensus.cc:515] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:50.668177 24705 raft_consensus.cc:2804] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:23:50.662453 24892 raft_consensus.cc:3060] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:50.696672 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:50.696445 17756 raft_consensus.cc:2393] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 16: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:23:50.697978 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:50.698724 17764 raft_consensus.cc:2368] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 14. Current term is 15.
I20260709 12:23:50.688335 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:50.699272 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:50.700057 18364 raft_consensus.cc:2468] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 17.
I20260709 12:23:50.700292 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:50.694314 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:50.693141 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:50.701146 17758 raft_consensus.cc:2393] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 17: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
W20260709 12:23:50.695654 17728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:50.701151 18361 raft_consensus.cc:2393] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 17: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:23:50.702504 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:50.653843 24818 raft_consensus.cc:515] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:50.683986 24865 raft_consensus.cc:2749] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:23:50.705361 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:50.706482 18364 raft_consensus.cc:1240] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 19. Current term is 20. Ops: [19.17-19.17]
I20260709 12:23:50.710952 18361 raft_consensus.cc:2468] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 15.
I20260709 12:23:50.708345 18363 raft_consensus.cc:1240] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 18. Current term is 19. Ops: []
I20260709 12:23:50.695096 24816 raft_consensus.cc:2804] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:23:50.714047 18365 raft_consensus.cc:1240] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 14. Current term is 15. Ops: [14.11-14.11]
I20260709 12:23:50.711637 17652 leader_election.cc:304] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:50.696493 24705 raft_consensus.cc:493] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:23:50.705476 24863 raft_consensus.cc:493] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:50.698323 24885 raft_consensus.cc:493] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:50.733781 24579 leader_election.cc:290] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 20 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:50.723052 24705 raft_consensus.cc:3060] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:50.732257 24885 raft_consensus.cc:515] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:23:50.736562 24838 consensus_meta.cc:287] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.866s	user 0.010s	sys 0.003s
W20260709 12:23:50.752344 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:50.698103 24882 consensus_queue.cc:237] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.11, Last appended by leader: 11, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:50.738184 24807 leader_election.cc:290] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 17 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:50.755441 24838 raft_consensus.cc:2749] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:23:50.762017 24713 raft_consensus.cc:515] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:50.770558 24899 raft_consensus.cc:2749] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:23:50.771629 17829 raft_consensus.cc:3060] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:50.773272 24713 leader_election.cc:290] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 14 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:50.745409 24816 raft_consensus.cc:697] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 21 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:23:50.780287 17829 raft_consensus.cc:2468] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 19.
I20260709 12:23:50.780870 24816 consensus_queue.cc:237] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 20.12, Last appended by leader: 12, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:50.760604 24890 raft_consensus.cc:2804] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:23:50.783190 24890 raft_consensus.cc:493] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:23:50.783759 24890 raft_consensus.cc:3060] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:23:50.788249 24431 raft_consensus.cc:515] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:50.792505 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:50.794134 24431 leader_election.cc:290] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 21 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:50.780290 17729 leader_election.cc:304] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:50.796027 24705 raft_consensus.cc:515] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:50.796731 24524 raft_consensus.cc:2749] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:23:50.797283 17680 raft_consensus.cc:3060] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:50.798102 17729 leader_election.cc:304] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:50.798866 24524 raft_consensus.cc:3060] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:50.805610 17680 raft_consensus.cc:2468] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 21.
I20260709 12:23:50.721199 24639 raft_consensus.cc:515] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:50.815457 17729 leader_election.cc:304] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
W20260709 12:23:50.817766 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 59950.2us
I20260709 12:23:50.820112 24227 raft_consensus.cc:493] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:23:50.824859 17729 leader_election.cc:304] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:50.832170 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:50.829687 24227 raft_consensus.cc:515] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:50.837723 24639 leader_election.cc:290] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 16 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:50.839046 17682 raft_consensus.cc:2393] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 16: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:23:50.841077 24227 leader_election.cc:290] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:50.740955 24863 raft_consensus.cc:515] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:50.851135 17652 leader_election.cc:304] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:50.855482 17831 raft_consensus.cc:1275] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 13 index: 10. Preceding OpId from leader: term: 17 index: 11. (index mismatch)
I20260709 12:23:50.825070 24687 raft_consensus.cc:2804] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:23:50.848840 24865 raft_consensus.cc:2749] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 12:23:50.818565 24814 consensus_queue.cc:1059] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 19.15, Next index: 16, Last known committed idx: 13, Time since last communication: 0.039s
I20260709 12:23:50.875051 17652 leader_election.cc:304] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:50.826607 24820 consensus_queue.cc:1048] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.039s
I20260709 12:23:50.863647 24687 raft_consensus.cc:493] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:23:50.870276 24880 raft_consensus.cc:2749] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:23:50.884771 17652 leader_election.cc:304] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:23:50.881453 24759 leader_election.cc:290] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 16 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:50.859139 24769 consensus_queue.cc:1048] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.021s
W20260709 12:23:50.915422 24856 consensus_meta.cc:287] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.731s	user 0.007s	sys 0.004s
I20260709 12:23:50.915767 24818 leader_election.cc:290] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:23:50.910748 24906 log.cc:927] Time spent T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.155s	user 0.000s	sys 0.001s
I20260709 12:23:50.881709 24897 raft_consensus.cc:2749] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
W20260709 12:23:50.910758 24907 log.cc:927] Time spent T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.062s	user 0.000s	sys 0.001s
I20260709 12:23:50.882699 24687 raft_consensus.cc:3060] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:23:50.975006 24650 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.729s	user 0.014s	sys 0.023s
I20260709 12:23:50.981659 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 21 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:23:50.969763 24802 consensus_meta.cc:287] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.741s	user 0.010s	sys 0.003s
I20260709 12:23:50.971738 24880 consensus_queue.cc:1059] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 16.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.014s
I20260709 12:23:50.881848 24823 consensus_queue.cc:1048] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.100s
W20260709 12:23:50.983953 24650 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.745s	user 0.015s	sys 0.023s
I20260709 12:23:50.975720 24708 raft_consensus.cc:493] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:50.996903 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:50.972993 24818 raft_consensus.cc:2804] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:23:50.986713 24677 raft_consensus.cc:493] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:50.976476 24903 consensus_queue.cc:1048] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.013s
I20260709 12:23:50.987327 24880 raft_consensus.cc:3055] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:23:50.992694 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:51.001049 24818 raft_consensus.cc:493] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:51.007256 24818 raft_consensus.cc:3060] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:51.008970 24677 raft_consensus.cc:515] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:51.017746 24863 leader_election.cc:290] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:50.986078 24897 consensus_queue.cc:1048] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.018s
I20260709 12:23:51.026403 24887 raft_consensus.cc:515] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:51.006313 24880 raft_consensus.cc:740] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:23:51.047640 24880 consensus_queue.cc:260] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.11, Last appended by leader: 11, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:51.049306 24880 raft_consensus.cc:3060] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:51.010780 24524 raft_consensus.cc:2749] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:23:51.031152 24892 raft_consensus.cc:515] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:51.046620 24657 raft_consensus.cc:493] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:51.057000 24657 raft_consensus.cc:515] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:51.058676 24892 leader_election.cc:290] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 14 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:51.066911 24890 raft_consensus.cc:515] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:51.070943 24885 leader_election.cc:290] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:51.029431 24721 raft_consensus.cc:493] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:51.072600 24890 leader_election.cc:290] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 22 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:23:51.064554 24909 log.cc:927] Time spent T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.075s	user 0.000s	sys 0.001s
I20260709 12:23:51.072561 24721 raft_consensus.cc:515] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:51.073908 17833 raft_consensus.cc:1275] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 18 index: 13. Preceding OpId from leader: term: 19 index: 14. (index mismatch)
I20260709 12:23:51.000162 24708 raft_consensus.cc:515] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:51.077364 24815 raft_consensus.cc:3055] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:23:51.077745 24815 raft_consensus.cc:740] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 18 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:23:51.078106 24708 leader_election.cc:290] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:51.078632 24815 consensus_queue.cc:260] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.13, Last appended by leader: 13, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:51.059692 24657 leader_election.cc:290] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:51.081638 17687 raft_consensus.cc:1275] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 22 index: 17. Preceding OpId from leader: term: 24 index: 18. (index mismatch)
I20260709 12:23:51.046208 17759 raft_consensus.cc:2468] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 18.
I20260709 12:23:51.087598 18333 leader_election.cc:400] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 14 pre-election: Vote denied by peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651) with higher term. Message: Invalid argument: T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 14. Current term is 15.
I20260709 12:23:51.091262 18333 leader_election.cc:403] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 14 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:23:51.095968 24887 leader_election.cc:290] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 18 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:51.092764 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:51.104171 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:51.105175 17832 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35042 (request call id 2739) took 1374 ms. Trace:
I20260709 12:23:51.100179 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:51.098636 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:51.107120 17762 raft_consensus.cc:2393] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 17: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:23:51.107833 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:51.105566 17832 rpcz_store.cc:276] 0709 12:23:49.729449 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:49.737044 (+  7595us) service_pool.cc:224] Handling call
0709 12:23:49.737338 (+   294us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:50.168692 (+431354us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:51.103692 (+935000us) spinlock_profiling.cc:243] Waited 430 ms on lock 0x7b54003354d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:23:51.103837 (+   145us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":989876992}
I20260709 12:23:51.109189 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:51.105547 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 18 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:51.110837 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:51.112335 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 18 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:51.106199 17755 raft_consensus.cc:2468] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 19.
W20260709 12:23:51.080651 24819 consensus_meta.cc:287] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.639s	user 0.007s	sys 0.006s
I20260709 12:23:51.115269 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:51.115082 24767 raft_consensus.cc:493] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:51.117790 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 18 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:51.117455 24767 raft_consensus.cc:515] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:51.118619 17755 raft_consensus.cc:2393] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 18: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:23:51.120112 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:51.120483 24767 leader_election.cc:290] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:51.120957 17831 raft_consensus.cc:2393] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 14: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:23:51.121196 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:23:51.121803 24392 consensus_meta.cc:287] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.527s	user 0.012s	sys 0.000s
I20260709 12:23:51.121842 17836 raft_consensus.cc:2468] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 18.
I20260709 12:23:51.122228 24392 raft_consensus.cc:515] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:51.123677 17652 leader_election.cc:304] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:23:51.119514 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:51.126765 24431 raft_consensus.cc:493] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:23:51.126994 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:51.079849 24815 raft_consensus.cc:3060] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:51.127835 17755 raft_consensus.cc:3060] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:51.127290 24431 raft_consensus.cc:515] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:51.090049 24642 consensus_queue.cc:1048] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.044s
I20260709 12:23:51.129968 24431 leader_election.cc:290] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:51.130697 24887 raft_consensus.cc:493] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:51.131197 24887 raft_consensus.cc:515] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:51.132555 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:51.133109 17685 raft_consensus.cc:3060] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Advancing to term 23
I20260709 12:23:51.115235 24802 raft_consensus.cc:2749] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Leader pre-election lost for term 14. Reason: Vote denied by peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651) with higher term. Message: Invalid argument: T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 14. Current term is 15.
I20260709 12:23:51.131614 24392 leader_election.cc:290] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 23 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:51.136670 24687 raft_consensus.cc:515] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:51.139096 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:51.142204 24708 raft_consensus.cc:2804] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:23:51.142690 24708 raft_consensus.cc:493] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:51.143198 24708 raft_consensus.cc:3060] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:51.149088 17836 raft_consensus.cc:1275] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 17 index: 9. Preceding OpId from leader: term: 19 index: 11. (index mismatch)
I20260709 12:23:51.151002 24819 raft_consensus.cc:2749] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:23:51.133442 24887 leader_election.cc:290] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:23:51.163226 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:51.194766 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:51.195240 17833 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37994 (request call id 1369) took 1359 ms. Trace:
I20260709 12:23:51.197602 17833 rpcz_store.cc:276] 0709 12:23:49.835465 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:49.975767 (+140302us) service_pool.cc:224] Handling call
0709 12:23:49.976127 (+   360us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:51.074229 (+1098102us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:51.194827 (+120598us) spinlock_profiling.cc:243] Waited 1.1 s on lock 0x7b54003cc3d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:23:51.195177 (+   350us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2523930496}
I20260709 12:23:51.201287 18333 leader_election.cc:304] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:51.210145 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:51.214184 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:51.211958 17758 raft_consensus.cc:2393] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 17: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:23:51.230217 18333 leader_election.cc:304] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:51.232741 18333 leader_election.cc:304] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:51.238749 18333 leader_election.cc:304] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:23:51.240283 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:51.241081 18333 leader_election.cc:304] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:51.212181 24802 raft_consensus.cc:2749] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:23:51.215906 24892 raft_consensus.cc:493] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:23:51.251219 17755 raft_consensus.cc:2468] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 21.
I20260709 12:23:51.237862 24431 raft_consensus.cc:2749] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:23:51.255833 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:51.242451 24790 raft_consensus.cc:2749] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:23:51.240386 24642 raft_consensus.cc:2749] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:23:51.259099 17755 raft_consensus.cc:2393] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 17: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:23:51.213349 24875 consensus_queue.cc:1048] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.037s
I20260709 12:23:51.255163 24650 raft_consensus.cc:3055] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:23:51.267206 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:23:51.273887 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:51.268332 24650 raft_consensus.cc:740] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 20 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:23:51.276779 24650 consensus_queue.cc:260] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 20.12, Last appended by leader: 12, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:51.280778 24650 raft_consensus.cc:3060] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:51.282531 17755 raft_consensus.cc:2468] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 22.
I20260709 12:23:51.263145 24802 raft_consensus.cc:3060] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:51.258885 24892 raft_consensus.cc:515] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:51.254897 24818 raft_consensus.cc:515] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:51.290982 24890 raft_consensus.cc:493] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:51.338491 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
W20260709 12:23:51.308410 24876 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.937s	user 0.004s	sys 0.020s
I20260709 12:23:51.344908 17755 raft_consensus.cc:3060] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:51.346374 17762 raft_consensus.cc:2468] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 19.
W20260709 12:23:51.345786 24876 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.980s	user 0.005s	sys 0.020s
I20260709 12:23:51.344599 24227 raft_consensus.cc:493] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:51.361285 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 120283us
I20260709 12:23:51.362294 18333 leader_election.cc:304] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:51.362349 17685 raft_consensus.cc:2468] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 23.
I20260709 12:23:51.364559 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:51.365269 17760 raft_consensus.cc:2468] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 19.
I20260709 12:23:51.365167 18333 leader_election.cc:304] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:51.366649 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:51.366719 24392 raft_consensus.cc:2749] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:23:51.367709 17760 raft_consensus.cc:3060] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:51.354346 24657 raft_consensus.cc:493] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:51.371726 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:51.353777 24876 raft_consensus.cc:697] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 21 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:23:51.372545 18374 raft_consensus.cc:2393] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 19: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:23:51.373602 18367 raft_consensus.cc:1275] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 16 index: 12. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
I20260709 12:23:51.375200 17755 raft_consensus.cc:2468] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 20.
I20260709 12:23:51.376885 17760 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 16.
I20260709 12:23:51.377482 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 17 candidate_status { last_received { term: 14 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:51.377939 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:51.377498 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:23:51.378783 24922 log.cc:927] Time spent T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.096s	user 0.000s	sys 0.001s
I20260709 12:23:51.378747 17764 raft_consensus.cc:2376] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 16. Re-sending same reply.
I20260709 12:23:51.379078 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:51.379858 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 21 candidate_status { last_received { term: 18 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:51.380219 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:51.345920 24890 raft_consensus.cc:515] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:51.368844 24431 raft_consensus.cc:2749] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 12:23:51.382128 17764 raft_consensus.cc:1240] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 18. Current term is 19. Ops: []
I20260709 12:23:51.382277 17755 raft_consensus.cc:1240] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 21. Current term is 22. Ops: [21.15-21.15]
I20260709 12:23:51.383576 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:51.362040 24227 raft_consensus.cc:515] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:51.385172 17764 raft_consensus.cc:1240] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 15. Current term is 16. Ops: []
I20260709 12:23:51.386300 24227 leader_election.cc:290] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:51.389081 24708 raft_consensus.cc:515] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:51.390762 18333 leader_election.cc:304] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:51.391196 24892 leader_election.cc:290] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:51.391743 24708 leader_election.cc:290] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 19 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:51.381085 17760 raft_consensus.cc:2376] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 20. Re-sending same reply.
I20260709 12:23:51.395751 24431 raft_consensus.cc:3060] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:51.396636 24876 consensus_queue.cc:237] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 19.11, Last appended by leader: 11, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:51.397643 18333 leader_election.cc:304] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:23:51.400234 24712 raft_consensus.cc:2804] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:23:51.400758 24712 raft_consensus.cc:697] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 16 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:23:51.401225 24642 consensus_queue.cc:1059] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 15.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
W20260709 12:23:51.404805 17804 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:51.415441 18333 leader_election.cc:304] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:23:51.414366 17755 raft_consensus.cc:1275] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 18 index: 13. (index mismatch)
I20260709 12:23:51.373541 24657 raft_consensus.cc:515] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:51.412541 24929 consensus_queue.cc:1048] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.011s
I20260709 12:23:51.425570 24929 consensus_queue.cc:1048] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:23:51.429445 24929 raft_consensus.cc:2804] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:23:51.436295 24929 raft_consensus.cc:697] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 17 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:23:51.437613 24723 leader_election.cc:290] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:51.439783 24705 leader_election.cc:290] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 20 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:51.446715 24834 raft_consensus.cc:493] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:23:51.447434 24834 raft_consensus.cc:515] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:51.401628 24712 consensus_queue.cc:237] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.10, Last appended by leader: 10, Current term: 16, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:51.453660 17760 raft_consensus.cc:1275] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 13 index: 10. Preceding OpId from leader: term: 17 index: 11. (index mismatch)
I20260709 12:23:51.466498 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:51.470022 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 21 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:51.475939 17682 raft_consensus.cc:1275] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 17 index: 10. Preceding OpId from leader: term: 19 index: 11. (index mismatch)
I20260709 12:23:51.478250 17760 raft_consensus.cc:1240] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 17. Current term is 18. Ops: [17.14-17.14]
I20260709 12:23:51.481618 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:51.482419 17755 raft_consensus.cc:3060] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:51.491542 17760 raft_consensus.cc:1275] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 18 index: 11. Preceding OpId from leader: term: 19 index: 11. (term mismatch)
I20260709 12:23:51.492143 17760 pending_rounds.cc:85] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb: Aborting all ops after (but not including) 10
I20260709 12:23:51.495566 24864 consensus_queue.cc:1048] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 12:23:51.496417 24642 raft_consensus.cc:493] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:51.496867 17761 consensus_queue.cc:260] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 19.17, Last appended by leader: 17, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:51.497901 17761 raft_consensus.cc:3060] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:51.497177 24642 raft_consensus.cc:515] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:51.464635 24807 raft_consensus.cc:493] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:51.481081 24924 log.cc:927] Time spent T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.061s	user 0.000s	sys 0.000s
W20260709 12:23:51.510421 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 95034.6us
I20260709 12:23:51.515177 18333 leader_election.cc:304] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:23:51.516484 18333 leader_election.cc:304] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:23:51.492449 17760 pending_rounds.cc:107] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb: Aborting uncommitted CHANGE_CONFIG_OP operation due to leader change: 18.11
I20260709 12:23:51.524652 17760 raft_consensus.cc:2917] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Aborting config change with OpId 18.11: Aborted: Op aborted by new leader
I20260709 12:23:51.525271 17760 raft_consensus.cc:2889] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:51.530683 17761 raft_consensus.cc:1275] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 19 index: 17. Preceding OpId from leader: term: 20 index: 17. (term mismatch)
I20260709 12:23:51.531226 17761 pending_rounds.cc:85] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb: Aborting all ops after (but not including) 16
I20260709 12:23:51.488737 24890 leader_election.cc:290] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:51.508898 24834 leader_election.cc:290] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:51.513651 24431 raft_consensus.cc:2749] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:23:51.484673 24885 consensus_queue.cc:1048] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.037s
I20260709 12:23:51.538216 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:51.531486 17761 pending_rounds.cc:107] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb: Aborting uncommitted NO_OP operation due to leader change: 19.17
I20260709 12:23:51.545929 17760 raft_consensus.cc:3060] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:23:51.547684 24815 raft_consensus.cc:493] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:51.547840 17755 raft_consensus.cc:2468] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 18.
I20260709 12:23:51.548237 24815 raft_consensus.cc:515] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:51.548949 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:51.549472 17755 raft_consensus.cc:3060] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:51.550320 24815 leader_election.cc:290] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:51.552030 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 23 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:51.552596 17682 raft_consensus.cc:2468] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 22.
I20260709 12:23:51.553707 17728 leader_election.cc:304] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:51.545907 17761 raft_consensus.cc:2889] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:51.554646 17761 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38786 (request call id 3469) took 2275 ms. Trace:
I20260709 12:23:51.555070 17761 rpcz_store.cc:276] 0709 12:23:49.278779 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:49.278941 (+   162us) service_pool.cc:224] Handling call
0709 12:23:49.288730 (+  9789us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:50.374109 (+1085379us) spinlock_profiling.cc:243] Waited 97.3 ms on lock 0x7b5400390ba0. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a597dc0 00007f7b4a6221c0 00007f7b4a623231 00007f7b4a61fd8f 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c
0709 12:23:51.496538 (+1122429us) spinlock_profiling.cc:243] Waited 1.87 ms on lock 0x7b4c00414340. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b488eca20 00007f7b488fda29 00007f7b488c3644 00007f7b488eaf2d 00007f7b488eab93 00007f7b4a61b18b 00007f7b4a61bcb5 00007f7b4a616f30 00007f7b4a6187de 00007f7b4a622b49 00007f7b4a623416
0709 12:23:51.554388 (+ 57850us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:51.554591 (+   203us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":228144256}
I20260709 12:23:51.519696 24818 leader_election.cc:290] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 18 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:51.554580 24815 raft_consensus.cc:2804] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:23:51.562297 24815 raft_consensus.cc:493] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:51.562645 24815 raft_consensus.cc:3060] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:23:51.580848 24883 leader_election.cc:290] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:51.582959 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 18 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:51.579586 17760 raft_consensus.cc:2468] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 22.
I20260709 12:23:51.587731 17760 raft_consensus.cc:1275] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 15 index: 13. Preceding OpId from leader: term: 17 index: 14. (index mismatch)
I20260709 12:23:51.515170 24807 raft_consensus.cc:515] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:23:51.561463 24688 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 20 current term: 20 is higher.
I20260709 12:23:51.589660 17760 raft_consensus.cc:1275] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 18 index: 11. Preceding OpId from leader: term: 20 index: 12. (index mismatch)
I20260709 12:23:51.590226 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:51.584376 17755 raft_consensus.cc:2468] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 14.
I20260709 12:23:51.583858 24431 raft_consensus.cc:2804] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:23:51.516474 24929 consensus_queue.cc:237] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.11, Last appended by leader: 11, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:51.598412 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 82015.5us
I20260709 12:23:51.643488 17761 raft_consensus.cc:1275] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 20 index: 15. Preceding OpId from leader: term: 21 index: 15. (term mismatch)
I20260709 12:23:51.644016 17761 pending_rounds.cc:85] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb: Aborting all ops after (but not including) 14
I20260709 12:23:51.644393 17761 pending_rounds.cc:107] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb: Aborting uncommitted CHANGE_CONFIG_OP operation due to leader change: 20.15
I20260709 12:23:51.583751 24934 raft_consensus.cc:2804] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:23:51.588799 24642 leader_election.cc:290] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:51.647044 18333 leader_election.cc:304] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:23:51.644948 17761 raft_consensus.cc:2917] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Aborting config change with OpId 20.15: Aborted: Op aborted by new leader
I20260709 12:23:51.652951 17803 leader_election.cc:304] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:51.598888 17755 raft_consensus.cc:1275] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 13 index: 11. Preceding OpId from leader: term: 15 index: 12. (index mismatch)
I20260709 12:23:51.600363 24431 raft_consensus.cc:493] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:23:51.655745 17761 raft_consensus.cc:2889] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:51.648252 24802 raft_consensus.cc:2749] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:23:51.657943 24858 raft_consensus.cc:3060] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:51.660018 24807 leader_election.cc:290] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:23:51.661654 17727 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:51.708173 17727 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:51.662056 24935 raft_consensus.cc:2804] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:23:51.710040 24935 raft_consensus.cc:493] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:51.710505 24935 raft_consensus.cc:3060] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Advancing to term 16
W20260709 12:23:51.711371 24886 log.cc:927] Time spent T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 1.243s	user 0.004s	sys 0.001s
I20260709 12:23:51.712368 24657 leader_election.cc:290] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:51.681327 24431 raft_consensus.cc:3060] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:51.684754 24943 consensus_queue.cc:1048] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.062s
I20260709 12:23:51.714287 17761 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.BulkChangeConfig from 127.0.0.1:37500 (request call id 3) took 7272 ms. Trace:
I20260709 12:23:51.714672 17761 rpcz_store.cc:276] 0709 12:23:44.436636 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:44.436778 (+   142us) service_pool.cc:224] Handling call
0709 12:23:44.706075 (+269297us) log.cc:844] Serialized 499 byte log entry
0709 12:23:46.005331 (+1299256us) spinlock_profiling.cc:243] Waited 16.1 ms on lock 0x7b1800448ba0. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b4a1c84bc 00007f7b4751648b 00007f7b503709a9 00007f7b4753269b 00007f7b4753093f 00007f7b47530455 00007f7b4a579b0b 00007f7b4a5feb63 00007f7b4a626752
0709 12:23:46.260411 (+255080us) spinlock_profiling.cc:243] Waited 5.75 ms on lock 0x7b54003aaaa0. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a5a0b26 00007f7b4a59a49a 00007f7b4a599714 00007f7b4a579eaa 00007f7b4a580d53 00007f7b4a580cd9 00007f7b4a580c69 00007f7b4a580c31 00007f7b4a57fdfd 0000000000388634
0709 12:23:46.849289 (+588878us) spinlock_profiling.cc:243] Waited 10.5 ms on lock 0x7b4400022b4c. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b47459c85 00007f7b4745a6a2 00007f7b4a1cc1db 00007f7b4a1c7aaf 00007f7b4a1bb86e 00007f7b4a1babb6 00007f7b4a1ba7d8 00007f7b4a1c4221 00007f7b4a1c41d9 00007f7b4a1c4169
0709 12:23:51.708913 (+4859624us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":100885,"raft.queue_time_us":685015,"raft.run_cpu_time_us":7131,"raft.run_wall_time_us":1353407,"spinlock_wait_cycles":74268672,"thread_start_us":426766,"threads_started":3,"wal-append.queue_time_us":415417,"wal-append.run_cpu_time_us":16161,"wal-append.run_wall_time_us":2720873}
I20260709 12:23:51.662794 17729 leader_election.cc:304] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
W20260709 12:23:51.716527 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 53815.9us
I20260709 12:23:51.717494 24887 raft_consensus.cc:2749] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:23:51.663573 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:51.718854 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:51.719704 17761 raft_consensus.cc:2393] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 14: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:23:51.665822 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:51.720039 24883 consensus_queue.cc:1059] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 12.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 12:23:51.720511 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:51.720846 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 18 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:51.721084 24883 raft_consensus.cc:3055] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:23:51.721151 17761 raft_consensus.cc:3060] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:23:51.664677 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:51.722071 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:51.722631 18371 raft_consensus.cc:2468] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 20.
I20260709 12:23:51.722900 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:51.666474 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:51.724182 18361 raft_consensus.cc:3060] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:51.668397 18366 raft_consensus.cc:1275] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 13 index: 10. Preceding OpId from leader: term: 17 index: 11. (index mismatch)
I20260709 12:23:51.728929 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:51.728703 24815 raft_consensus.cc:515] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:51.729926 17760 raft_consensus.cc:1240] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 22. Current term is 23. Ops: [22.12-22.12]
I20260709 12:23:51.730020 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:51.730645 18366 raft_consensus.cc:3060] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:51.669682 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:51.732184 18368 raft_consensus.cc:3060] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:51.733107 17760 raft_consensus.cc:1240] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 21. Current term is 22. Ops: [19.12-21.13]
I20260709 12:23:51.733856 24858 raft_consensus.cc:2749] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:23:51.734401 17760 raft_consensus.cc:1240] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 15. Current term is 16. Ops: []
I20260709 12:23:51.721540 24883 raft_consensus.cc:740] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
W20260709 12:23:51.670794 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:51.735966 24885 raft_consensus.cc:493] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:51.736136 24883 consensus_queue.cc:260] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:51.736477 24885 raft_consensus.cc:515] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:51.737432 24883 raft_consensus.cc:3060] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:51.737869 24815 leader_election.cc:290] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 23 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:51.717542 24677 leader_election.cc:290] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:51.663352 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:51.747869 17759 raft_consensus.cc:2468] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 16.
I20260709 12:23:51.723529 17755 raft_consensus.cc:2468] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 19.
I20260709 12:23:51.667455 18362 raft_consensus.cc:1275] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 18 index: 13. Preceding OpId from leader: term: 22 index: 15. (index mismatch)
I20260709 12:23:51.738688 24885 leader_election.cc:290] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:51.740119 17688 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37396 (request call id 2811) took 1095 ms. Trace:
I20260709 12:23:51.740420 17729 leader_election.cc:304] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:51.740901 24887 raft_consensus.cc:493] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:51.741326 17654 leader_election.cc:304] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:51.723448 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
W20260709 12:23:51.742782 17635 auto_rebalancer.cc:629] Failed to schedule move for tablet e2bcfe1b94464b6d9620cc3b8a17a2ad: Aborted: Op aborted by new leader
I20260709 12:23:51.675781 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:51.755834 18364 raft_consensus.cc:2393] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 20: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:23:51.746897 17761 raft_consensus.cc:2468] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 23.
I20260709 12:23:51.755913 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:51.756754 18362 raft_consensus.cc:2368] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for earlier term 20. Current term is 21.
I20260709 12:23:51.757841 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:51.750303 18368 raft_consensus.cc:2468] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 16.
I20260709 12:23:51.758898 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:51.752760 18361 raft_consensus.cc:2468] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 18.
I20260709 12:23:51.753921 17688 rpcz_store.cc:276] 0709 12:23:50.644598 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:50.644966 (+   368us) service_pool.cc:224] Handling call
0709 12:23:51.739929 (+1094963us) spinlock_profiling.cc:243] Waited 749 ms on lock 0x7b5400426dd4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:23:51.740061 (+   132us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1721732992}
I20260709 12:23:51.761111 17654 leader_election.cc:304] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:23:51.760973 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:51.762099 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:51.764369 17654 leader_election.cc:304] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:23:51.767338 24823 raft_consensus.cc:2804] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:23:51.790179 24823 raft_consensus.cc:493] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:51.790601 24823 raft_consensus.cc:3060] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:51.791962 18371 raft_consensus.cc:1275] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 16 index: 11. Preceding OpId from leader: term: 18 index: 12. (index mismatch)
I20260709 12:23:51.794307 18371 raft_consensus.cc:1275] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 22 index: 17. Preceding OpId from leader: term: 24 index: 18. (index mismatch)
I20260709 12:23:51.795110 24885 raft_consensus.cc:2749] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
W20260709 12:23:51.787540 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:51.801391 24801 raft_consensus.cc:3060] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:51.802042 24823 raft_consensus.cc:515] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:51.804382 24823 leader_election.cc:290] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 21 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:51.777333 18368 raft_consensus.cc:1275] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 18 index: 11. Preceding OpId from leader: term: 19 index: 11. (term mismatch)
W20260709 12:23:51.778904 17728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:51.779285 18366 raft_consensus.cc:2468] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 19.
I20260709 12:23:51.768729 18362 raft_consensus.cc:1240] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 17. Current term is 18. Ops: []
W20260709 12:23:51.811057 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 158221us
I20260709 12:23:51.813716 18368 pending_rounds.cc:85] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da: Aborting all ops after (but not including) 10
I20260709 12:23:51.766223 17654 leader_election.cc:304] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:23:51.815632 18368 pending_rounds.cc:107] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da: Aborting uncommitted CHANGE_CONFIG_OP operation due to leader change: 18.11
I20260709 12:23:51.807590 18364 raft_consensus.cc:1275] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 15 index: 13. Preceding OpId from leader: term: 17 index: 14. (index mismatch)
W20260709 12:23:51.815646 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 148767us
W20260709 12:23:51.820323 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 54201.3us
I20260709 12:23:51.827731 17654 leader_election.cc:304] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:51.767503 24818 consensus_queue.cc:1048] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 12, Time since last communication: 0.061s
I20260709 12:23:51.829281 24816 raft_consensus.cc:2749] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:23:51.765941 24891 raft_consensus.cc:2749] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:23:51.768973 24898 raft_consensus.cc:2804] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:23:51.825017 24899 raft_consensus.cc:2804] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:23:51.817425 18368 raft_consensus.cc:2917] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Aborting config change with OpId 18.11: Aborted: Op aborted by new leader
I20260709 12:23:51.761727 24887 raft_consensus.cc:515] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:51.846681 24923 log.cc:927] Time spent T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.446s	user 0.002s	sys 0.000s
I20260709 12:23:51.848102 17689 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44898 (request call id 1371) took 1420 ms. Trace:
I20260709 12:23:51.848405 24898 raft_consensus.cc:697] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 16 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:23:51.849709 24898 consensus_queue.cc:237] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:51.848583 17689 rpcz_store.cc:276] 0709 12:23:50.427393 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:50.427544 (+   151us) service_pool.cc:224] Handling call
0709 12:23:50.428034 (+   490us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:50.458878 (+ 30844us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:50.458928 (+    50us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:50.459433 (+   505us) log.cc:844] Serialized 26 byte log entry
0709 12:23:51.097309 (+637876us) spinlock_profiling.cc:243] Waited 2.13 ms on lock 0x7b18001b8620. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b47515e07 00007f7b503709a9 00007f7b4753269b 00007f7b4753093f 00007f7b47532c02 00007f7b4a1ba20b 00007f7b4a1bfd23 00007f7b4a1c00d1 00007f7b4a5dba0f
0709 12:23:51.291501 (+194192us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:23:51.291564 (+    63us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:51.291669 (+   105us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:51.846049 (+554380us) spinlock_profiling.cc:243] Waited 1.93 ms on lock 0x7b4400022b4c. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b47459c85 00007f7b4745a6a2 00007f7b4a1cc1db 00007f7b4a1c7aaf 00007f7b4a1bb86e 00007f7b4a1babb6 00007f7b4a1ba7d8 00007f7b4a1c4221 00007f7b4a1c41d9 00007f7b4a1c4169
0709 12:23:51.847709 (+  1660us) raft_consensus.cc:1685] finished
0709 12:23:51.847741 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:51.848020 (+   279us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":51158,"raft.queue_time_us":35741,"raft.run_cpu_time_us":100,"raft.run_wall_time_us":3430,"spinlock_wait_cycles":9344256,"thread_start_us":193918,"threads_started":1,"wal-append.queue_time_us":907700}
W20260709 12:23:51.858042 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:51.767620 24890 consensus_queue.cc:1048] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.067s
W20260709 12:23:51.841643 18333 leader_election.cc:336] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 15 election: RPC error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Timed out: RequestConsensusVote RPC to 127.16.29.66:39651 timed out after 2.166s (SENT)
I20260709 12:23:51.840358 18368 raft_consensus.cc:2889] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [term 19 LEARNER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:23:51.842841 17803 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:51.865900 18333 leader_election.cc:304] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:51.831961 17654 leader_election.cc:400] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 20 election: Vote denied by peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577) with higher term. Message: Invalid argument: T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for earlier term 20. Current term is 21.
I20260709 12:23:51.867316 24687 leader_election.cc:290] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 18 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:51.871932 17654 leader_election.cc:403] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 20 election: Cancelling election due to peer responding with higher term
I20260709 12:23:51.877480 17688 raft_consensus.cc:1275] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 20 index: 11. Preceding OpId from leader: term: 22 index: 12. (index mismatch)
I20260709 12:23:51.824401 24858 consensus_queue.cc:1048] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.040s
I20260709 12:23:51.878995 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 15 candidate_status { last_received { term: 13 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:23:51.879536 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 48947.2us
I20260709 12:23:51.885206 24934 raft_consensus.cc:697] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 19 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:23:51.887390 17654 leader_election.cc:304] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:23:51.889724 24934 consensus_queue.cc:237] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.11, Last appended by leader: 11, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:51.865847 24899 raft_consensus.cc:697] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 18 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
W20260709 12:23:51.899082 18333 leader_election.cc:336] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 16 pre-election: RPC error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Timed out: RequestConsensusVote RPC to 127.16.29.66:39651 timed out after 2.435s (SENT)
W20260709 12:23:51.888223 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 147896us
W20260709 12:23:51.902915 17804 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:51.904572 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:51.905237 18363 raft_consensus.cc:3060] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:51.904901 18333 leader_election.cc:304] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:51.877277 24441 raft_consensus.cc:2749] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:23:51.908466 18333 leader_election.cc:304] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:51.882419 24650 raft_consensus.cc:493] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:23:51.882570 24927 raft_consensus.cc:2749] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Leader election lost for term 20. Reason: Vote denied by peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577) with higher term. Message: Invalid argument: T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for earlier term 20. Current term is 21.
I20260709 12:23:51.929745 24650 raft_consensus.cc:515] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:51.930187 17762 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36572 (request call id 1237) took 1226 ms. Trace:
I20260709 12:23:51.933080 24951 raft_consensus.cc:2804] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:23:51.933861 24951 raft_consensus.cc:493] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:51.934361 24951 raft_consensus.cc:3060] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:51.935935 24899 consensus_queue.cc:237] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.13, Last appended by leader: 13, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:51.937672 24935 raft_consensus.cc:515] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:51.938956 17829 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35042 (request call id 2742) took 2058 ms (client timeout 1893 ms). Trace:
I20260709 12:23:51.913296 24929 raft_consensus.cc:2804] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:23:51.940346 24929 raft_consensus.cc:697] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 21 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:23:51.895509 24816 consensus_queue.cc:1048] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.045s
I20260709 12:23:51.941183 24929 consensus_queue.cc:237] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 19.14, Last appended by leader: 14, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:51.941789 18361 raft_consensus.cc:1240] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 15. Current term is 16. Ops: []
I20260709 12:23:51.943727 24650 leader_election.cc:290] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:23:51.944583 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 89691.1us
I20260709 12:23:51.908735 24431 raft_consensus.cc:515] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:51.948407 17829 rpcz_store.cc:269] 0709 12:23:49.880554 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:49.944721 (+ 64167us) service_pool.cc:224] Handling call
0709 12:23:51.938719 (+1993998us) spinlock_profiling.cc:243] Waited 786 ms on lock 0x7b54000aa5d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:23:51.938859 (+   140us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1806942592}
I20260709 12:23:51.922657 17680 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44898 (request call id 1377) took 1130 ms. Trace:
I20260709 12:23:51.949106 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:51.949657 17680 rpcz_store.cc:276] 0709 12:23:50.792016 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:50.792201 (+   185us) service_pool.cc:224] Handling call
0709 12:23:51.922289 (+1130088us) spinlock_profiling.cc:243] Waited 4.02 ms on lock 0x7b54001aac54. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:23:51.922559 (+   270us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":9255808}
I20260709 12:23:51.950198 18364 raft_consensus.cc:3060] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:51.931411 17762 rpcz_store.cc:276] 0709 12:23:50.703933 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:51.115059 (+411126us) service_pool.cc:224] Handling call
0709 12:23:51.929705 (+814646us) spinlock_profiling.cc:243] Waited 222 ms on lock 0x7b54004150d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:23:51.930065 (+   360us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":510219648}
I20260709 12:23:51.951704 18363 raft_consensus.cc:2468] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 18.
I20260709 12:23:51.955003 18334 leader_election.cc:304] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:23:51.956974 24890 consensus_queue.cc:1048] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.003s
I20260709 12:23:51.933391 24887 leader_election.cc:290] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:51.962198 18374 raft_consensus.cc:2468] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 17.
I20260709 12:23:51.946220 24898 raft_consensus.cc:2804] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:23:51.966776 24790 raft_consensus.cc:2804] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:23:51.971722 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:51.972759 17682 raft_consensus.cc:2468] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 16.
I20260709 12:23:51.974145 18364 raft_consensus.cc:2468] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 21.
I20260709 12:23:51.954568 24769 consensus_queue.cc:1048] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.014s
I20260709 12:23:51.954708 24816 consensus_queue.cc:1059] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 17.14, Next index: 15, Last known committed idx: 13, Time since last communication: 0.014s
I20260709 12:23:51.955384 24677 consensus_queue.cc:1059] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 15.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.017s
I20260709 12:23:51.979566 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:51.980084 18364 raft_consensus.cc:1240] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 17. Current term is 18. Ops: [17.13-17.13]
I20260709 12:23:51.981359 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 23 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:51.985955 17688 raft_consensus.cc:3060] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:23:51.995441 17654 leader_election.cc:304] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
W20260709 12:23:52.009444 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 53175.8us
I20260709 12:23:52.011268 18334 leader_election.cc:304] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:23:52.018689 17688 raft_consensus.cc:2468] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 23.
I20260709 12:23:51.981451 24898 raft_consensus.cc:697] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 19 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:23:52.024382 17728 leader_election.cc:304] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:52.009009 24891 raft_consensus.cc:2804] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Leader election won for term 21
W20260709 12:23:52.028712 17730 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:52.006297 24887 raft_consensus.cc:493] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:52.007655 24834 raft_consensus.cc:493] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:52.019635 24844 raft_consensus.cc:2804] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:23:52.025542 24952 raft_consensus.cc:493] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:52.028657 24829 raft_consensus.cc:2804] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:23:52.039849 24834 raft_consensus.cc:515] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:52.044099 17834 raft_consensus.cc:2468] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 18.
W20260709 12:23:52.044595 24947 log.cc:927] Time spent T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.062s	user 0.000s	sys 0.000s
W20260709 12:23:52.045794 17834 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35040 (request call id 3032) took 1508 ms (client timeout 1903 ms). Trace:
I20260709 12:23:52.046052 24721 leader_election.cc:290] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:23:52.046344 17834 rpcz_store.cc:269] 0709 12:23:50.537796 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:50.538049 (+   253us) service_pool.cc:224] Handling call
0709 12:23:52.045618 (+1507569us) spinlock_profiling.cc:243] Waited 1.48 s on lock 0x7b54003345d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:23:52.045734 (+   116us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3399990016}
I20260709 12:23:52.043556 24687 raft_consensus.cc:493] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:23:52.047778 24687 raft_consensus.cc:515] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:52.021541 24801 consensus_queue.cc:1059] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 16.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.020s
I20260709 12:23:52.051678 17758 raft_consensus.cc:1217] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Deduplicated request from leader. Original: 13.9->[15.10-20.12]   Dedup: 15.10->[18.11-20.12]
I20260709 12:23:52.052232 17758 raft_consensus.cc:1240] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 20. Current term is 21. Ops: [15.10-20.12]
I20260709 12:23:52.055203 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:52.056507 17762 raft_consensus.cc:2468] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 20.
I20260709 12:23:52.057727 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:52.054275 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:52.059711 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:52.060679 17689 raft_consensus.cc:1275] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 16 index: 11. Preceding OpId from leader: term: 20 index: 13. (index mismatch)
I20260709 12:23:52.061602 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 15 candidate_status { last_received { term: 13 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:52.062590 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 17 candidate_status { last_received { term: 14 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:52.063153 17688 raft_consensus.cc:3060] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:52.065353 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:52.068392 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 18 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:52.069005 17684 raft_consensus.cc:2393] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 14: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:23:52.071328 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 22 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:52.071988 17757 raft_consensus.cc:2468] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 21.
I20260709 12:23:52.073874 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:52.075006 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 22 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:52.077052 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:52.070842 24848 raft_consensus.cc:493] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:52.077191 24898 consensus_queue.cc:237] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.13, Last appended by leader: 13, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:52.078508 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:52.078369 24848 raft_consensus.cc:515] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:52.042054 24952 raft_consensus.cc:515] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:52.084348 24801 raft_consensus.cc:493] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:52.085093 24801 raft_consensus.cc:515] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:52.088259 24957 raft_consensus.cc:3055] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:23:52.090422 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:52.094053 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 21 candidate_status { last_received { term: 18 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:52.097059 17688 raft_consensus.cc:2410] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 17 because replica has last-logged OpId of term: 16 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 14.
I20260709 12:23:52.110908 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:52.111140 17729 leader_election.cc:304] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:52.111696 17758 raft_consensus.cc:2376] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 19. Re-sending same reply.
I20260709 12:23:52.117256 18368 raft_consensus.cc:1240] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 19. Current term is 20. Ops: []
I20260709 12:23:52.088749 24957 raft_consensus.cc:740] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 17 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:23:52.038592 24887 raft_consensus.cc:515] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:52.124574 17729 leader_election.cc:304] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:52.130045 17729 leader_election.cc:304] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:52.092971 24883 raft_consensus.cc:493] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:52.093146 24858 raft_consensus.cc:493] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:52.093026 24650 raft_consensus.cc:493] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:52.121711 24813 raft_consensus.cc:493] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:52.117190 24829 raft_consensus.cc:697] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:23:52.154740 17757 raft_consensus.cc:2468] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 17.
I20260709 12:23:52.104417 24721 raft_consensus.cc:493] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:52.070858 24953 consensus_queue.cc:1048] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.061s
I20260709 12:23:52.125599 24807 raft_consensus.cc:493] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:23:52.125379 24885 raft_consensus.cc:2749] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:23:52.149536 24815 raft_consensus.cc:2749] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:23:52.129153 24790 raft_consensus.cc:697] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 23 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:23:52.147584 24688 raft_consensus.cc:2804] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:23:52.147923 24964 raft_consensus.cc:3055] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:23:52.132707 24675 consensus_queue.cc:1048] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 15, Time since last communication: 0.098s
I20260709 12:23:52.154456 24943 raft_consensus.cc:2804] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:23:52.156919 24801 leader_election.cc:290] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:52.230501 24807 raft_consensus.cc:515] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:52.316941 24807 leader_election.cc:290] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:50.644138 16501 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:23:52.325798 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:52.326272 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:52.363823 17761 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 16.
I20260709 12:23:52.328150 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:52.365012 17757 raft_consensus.cc:2468] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 20.
I20260709 12:23:52.333345 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:52.366043 17759 raft_consensus.cc:3060] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:52.156719 24650 raft_consensus.cc:515] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:52.131404 24883 raft_consensus.cc:515] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:52.241322 24657 raft_consensus.cc:493] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:52.272150 24963 log.cc:927] Time spent T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.234s	user 0.001s	sys 0.000s
W20260709 12:23:52.280367 24959 log.cc:927] Time spent T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.192s	user 0.001s	sys 0.000s
I20260709 12:23:52.154044 24858 raft_consensus.cc:515] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:52.247741 24943 raft_consensus.cc:493] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:52.347867 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:52.243916 24964 raft_consensus.cc:740] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:23:52.250567 24688 raft_consensus.cc:697] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:23:52.355420 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:52.464155 17838 raft_consensus.cc:2468] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 21.
W20260709 12:23:52.406260 24950 log.cc:927] Time spent T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.057s	user 0.000s	sys 0.000s
I20260709 12:23:52.356163 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:52.470409 17833 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 16.
I20260709 12:23:52.471306 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:52.472509 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:52.473052 17833 raft_consensus.cc:2468] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 20.
I20260709 12:23:52.473762 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 23 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:52.159051 24813 raft_consensus.cc:515] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:52.231792 24721 raft_consensus.cc:515] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:52.371711 17829 raft_consensus.cc:1275] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 16 index: 11. Preceding OpId from leader: term: 18 index: 12. (index mismatch)
W20260709 12:23:52.372167 17801 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:52.384331 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:52.346709 24801 raft_consensus.cc:493] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:23:52.339985 24848 leader_election.cc:290] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:52.401535 17756 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38776 (request call id 2746) took 1769 ms. Trace:
I20260709 12:23:52.498088 17756 rpcz_store.cc:276] 0709 12:23:50.632009 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:50.632303 (+   294us) service_pool.cc:224] Handling call
0709 12:23:52.401293 (+1768990us) spinlock_profiling.cc:243] Waited 41.5 ms on lock 0x7b54004164d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:23:52.401459 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":95508224}
I20260709 12:23:52.397383 24815 raft_consensus.cc:493] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:23:52.353068 24894 raft_consensus.cc:3055] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:23:52.416580 17755 raft_consensus.cc:2368] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for earlier term 20. Current term is 21.
I20260709 12:23:52.354668 24687 leader_election.cc:290] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:52.421262 24943 raft_consensus.cc:3060] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:52.395242 17759 raft_consensus.cc:2468] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 21.
I20260709 12:23:52.428613 17762 raft_consensus.cc:2393] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 21: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:23:52.355895 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:52.541953 17830 raft_consensus.cc:3060] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:52.466369 17835 raft_consensus.cc:1275] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 20 index: 14. Preceding OpId from leader: term: 21 index: 15. (index mismatch)
I20260709 12:23:52.474758 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 22 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:52.543140 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:52.475593 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:52.544004 18361 raft_consensus.cc:2468] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 15.
I20260709 12:23:52.476155 17836 raft_consensus.cc:3060] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:52.508554 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:52.469051 24891 raft_consensus.cc:697] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 21 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:23:52.510048 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:52.516479 18367 raft_consensus.cc:1240] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 19 LEADER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 18. Current term is 19. Ops: []
I20260709 12:23:52.524714 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:52.548504 17686 raft_consensus.cc:2393] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 16: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:23:52.552176 17763 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38786 (request call id 3487) took 2028 ms. Trace:
I20260709 12:23:52.250916 24957 consensus_queue.cc:260] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.13, Last appended by leader: 13, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:52.535069 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:52.348991 24829 consensus_queue.cc:237] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 20.11, Last appended by leader: 11, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:52.545552 17680 raft_consensus.cc:2410] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 20 because replica has last-logged OpId of term: 19 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 13.
I20260709 12:23:52.509361 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 19 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:52.545763 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:52.552561 17763 rpcz_store.cc:276] 0709 12:23:50.524072 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:50.524284 (+   212us) service_pool.cc:224] Handling call
0709 12:23:50.533012 (+  8728us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:50.559715 (+ 26703us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:50.559757 (+    42us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:50.567635 (+  7878us) log.cc:844] Serialized 26 byte log entry
0709 12:23:51.932440 (+1364805us) spinlock_profiling.cc:243] Waited 6.8 ms on lock 0x7b18004360c0. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b47515e07 00007f7b503709a9 00007f7b4753269b 00007f7b4753093f 00007f7b47532c02 00007f7b4a1ba20b 00007f7b4a1bfd23 00007f7b4a1c00d1 00007f7b4a5dba0f
0709 12:23:52.007762 (+ 75322us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:23:52.007825 (+    63us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:52.021899 (+ 14074us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:52.544778 (+522879us) raft_consensus.cc:1685] finished
0709 12:23:52.544798 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:52.552114 (+  7316us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":54383,"raft.queue_time_us":30521,"raft.run_cpu_time_us":144,"raft.run_wall_time_us":20593,"spinlock_wait_cycles":15649024,"thread_start_us":75057,"threads_started":1,"wal-append.queue_time_us":1473909}
I20260709 12:23:52.548470 17683 raft_consensus.cc:2468] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 17.
I20260709 12:23:52.493924 24981 raft_consensus.cc:493] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:52.555364 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:23:52.409363 17803 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:52.404496 24657 raft_consensus.cc:515] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:52.540483 24898 raft_consensus.cc:493] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:52.531749 24894 raft_consensus.cc:740] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 19 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:23:52.563545 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
W20260709 12:23:52.554332 24963 log.cc:927] Time spent T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.058s	user 0.000s	sys 0.000s
I20260709 12:23:52.383822 24790 consensus_queue.cc:237] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 21.15, Last appended by leader: 15, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:52.577731 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:52.569029 17836 raft_consensus.cc:2468] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 18.
I20260709 12:23:52.578783 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:52.580053 24883 leader_election.cc:290] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:52.584757 24650 leader_election.cc:290] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:52.586226 24721 leader_election.cc:290] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:23:52.594837 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:52.595081 17764 raft_consensus.cc:2393] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 16: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:23:52.595469 17835 raft_consensus.cc:2376] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 19. Re-sending same reply.
I20260709 12:23:52.602238 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:52.602917 17833 raft_consensus.cc:2468] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 21.
I20260709 12:23:52.602917 17835 raft_consensus.cc:2468] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 21.
I20260709 12:23:52.604063 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 23 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:52.597121 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 21 candidate_status { last_received { term: 18 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:52.606869 17835 raft_consensus.cc:1217] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 20 LEADER]: Deduplicated request from leader. Original: 17.15->[18.16-19.17]   Dedup: 19.17->[]
I20260709 12:23:52.607631 17835 raft_consensus.cc:1240] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 20 LEADER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 19. Current term is 20. Ops: [18.16-19.17]
I20260709 12:23:52.587687 24807 consensus_queue.cc:1048] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.024s
I20260709 12:23:52.612514 24986 raft_consensus.cc:493] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:52.613425 17764 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38776 (request call id 2749) took 1159 ms. Trace:
I20260709 12:23:52.537910 24815 raft_consensus.cc:515] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:52.619397 17830 raft_consensus.cc:2468] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 19.
I20260709 12:23:52.531678 24801 raft_consensus.cc:515] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:52.621914 17838 raft_consensus.cc:2468] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 18.
I20260709 12:23:52.572059 24816 raft_consensus.cc:493] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:23:52.613832 17764 rpcz_store.cc:276] 0709 12:23:51.454461 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:52.576093 (+1121632us) service_pool.cc:224] Handling call
0709 12:23:52.613363 (+ 37270us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:52.561048 24983 raft_consensus.cc:493] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:52.572907 24887 leader_election.cc:290] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:52.636287 17830 raft_consensus.cc:1275] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 18 index: 12. Preceding OpId from leader: term: 20 index: 13. (index mismatch)
I20260709 12:23:52.613106 24986 raft_consensus.cc:515] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:52.683960 17830 raft_consensus.cc:1275] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 20 index: 12. Preceding OpId from leader: term: 21 index: 13. (index mismatch)
I20260709 12:23:52.684298 24986 leader_election.cc:290] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:23:52.684518 24951 consensus_meta.cc:287] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.750s	user 0.013s	sys 0.000s
I20260709 12:23:52.685153 17838 raft_consensus.cc:1275] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 16 index: 10. Preceding OpId from leader: term: 20 index: 12. (index mismatch)
I20260709 12:23:52.685093 24951 raft_consensus.cc:515] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:52.600723 24894 consensus_queue.cc:260] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 19.15, Last appended by leader: 15, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:52.689033 24894 raft_consensus.cc:3060] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:52.623406 24650 raft_consensus.cc:493] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:52.692416 24650 raft_consensus.cc:515] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:52.694972 24650 leader_election.cc:290] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:23:52.696835 24982 log.cc:927] Time spent T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.171s	user 0.000s	sys 0.000s
I20260709 12:23:52.700516 24657 leader_election.cc:290] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:52.701239 24891 consensus_queue.cc:237] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 19.14, Last appended by leader: 14, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:52.705510 24815 leader_election.cc:290] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:23:52.664811 24988 log.cc:927] Time spent T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.063s	user 0.000s	sys 0.000s
I20260709 12:23:52.588987 24898 raft_consensus.cc:515] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:52.594429 24688 consensus_queue.cc:237] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.13, Last appended by leader: 13, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:52.673496 24858 leader_election.cc:290] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:23:52.674515 24973 log.cc:927] Time spent T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.211s	user 0.000s	sys 0.000s
I20260709 12:23:52.674953 24816 raft_consensus.cc:515] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:52.677419 24801 leader_election.cc:290] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:52.680449 24983 raft_consensus.cc:515] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:52.682157 24813 leader_election.cc:290] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:52.602232 24957 raft_consensus.cc:3060] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:23:52.721092 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 67883.1us
I20260709 12:23:52.720702 24927 raft_consensus.cc:493] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:52.763471 18364 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.613s	user 0.000s	sys 0.012s
W20260709 12:23:52.763940 18364 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.614s	user 0.000s	sys 0.012s
I20260709 12:23:52.758955 18333 leader_election.cc:304] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:52.577785 24981 raft_consensus.cc:515] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:52.721357 24986 raft_consensus.cc:493] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:52.767980 18333 leader_election.cc:304] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:52.722713 24951 leader_election.cc:290] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 20 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:23:52.756181 24677 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.516s	user 0.006s	sys 0.005s
I20260709 12:23:52.777997 24934 raft_consensus.cc:2749] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:23:52.778738 18374 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59918 (request call id 2074) took 1117 ms. Trace:
I20260709 12:23:52.755239 24998 raft_consensus.cc:493] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:52.779225 18374 rpcz_store.cc:276] 0709 12:23:51.661047 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:51.663335 (+  2288us) service_pool.cc:224] Handling call
0709 12:23:52.778552 (+1115217us) spinlock_profiling.cc:243] Waited 244 ms on lock 0x7b54007e7cd4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:23:52.778673 (+   121us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":560559488}
I20260709 12:23:52.765772 24989 raft_consensus.cc:493] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:52.782790 24677 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.551s	user 0.007s	sys 0.005s
I20260709 12:23:52.789564 24898 leader_election.cc:290] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:52.778949 24990 raft_consensus.cc:2749] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
W20260709 12:23:52.781997 24980 log.cc:927] Time spent T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.373s	user 0.000s	sys 0.001s
I20260709 12:23:52.766139 24927 raft_consensus.cc:515] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:52.787277 24998 raft_consensus.cc:515] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:52.780036 24986 raft_consensus.cc:515] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:52.810161 24943 raft_consensus.cc:515] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:52.804217 24976 log.cc:927] Time spent T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.172s	user 0.001s	sys 0.000s
I20260709 12:23:52.796772 24989 raft_consensus.cc:515] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:52.818317 24981 leader_election.cc:290] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:52.832774 24989 leader_election.cc:290] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:52.822038 24986 leader_election.cc:290] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:23:52.824926 24994 log.cc:927] Time spent T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.056s	user 0.000s	sys 0.001s
I20260709 12:23:52.825788 24983 leader_election.cc:290] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:52.829916 24998 leader_election.cc:290] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:52.886075 17684 raft_consensus.cc:1240] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 16. Current term is 17. Ops: []
W20260709 12:23:52.885957 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 115314us
I20260709 12:23:52.890384 17801 leader_election.cc:304] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:52.901193 17801 leader_election.cc:304] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:52.777179 24657 consensus_queue.cc:1048] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.023s
I20260709 12:23:52.908651 18333 leader_election.cc:304] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:23:52.909914 17801 leader_election.cc:304] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:52.910549 24657 consensus_queue.cc:1048] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:23:52.913772 24657 consensus_queue.cc:1059] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 20.11, Next index: 12, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:23:52.915290 24657 raft_consensus.cc:2804] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:23:52.915840 24657 raft_consensus.cc:493] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:52.916453 24657 raft_consensus.cc:3060] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:52.916730 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:52.918049 17832 raft_consensus.cc:2468] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 19.
I20260709 12:23:52.920526 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:52.921769 24677 consensus_queue.cc:1059] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 15.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:23:52.924975 24677 raft_consensus.cc:2804] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:23:52.925642 24677 raft_consensus.cc:697] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 20 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:23:52.927364 24677 consensus_queue.cc:237] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 18.15, Last appended by leader: 15, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:52.818890 24927 leader_election.cc:290] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:52.829425 24943 leader_election.cc:290] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 18 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:23:52.951838 24946 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 1.019s	user 0.006s	sys 0.022s
I20260709 12:23:52.952286 24981 raft_consensus.cc:2804] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:23:52.956701 24981 raft_consensus.cc:493] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:23:52.957386 24981 raft_consensus.cc:3060] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:52.964007 24989 consensus_queue.cc:1059] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 18.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:23:52.964659 24986 raft_consensus.cc:2804] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:23:52.965232 24986 raft_consensus.cc:493] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:52.965655 24990 raft_consensus.cc:3055] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:23:52.967862 24829 raft_consensus.cc:493] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:52.968191 24983 raft_consensus.cc:3055] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:23:52.968722 24983 raft_consensus.cc:740] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 22 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:23:52.969220 24829 raft_consensus.cc:515] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:52.970351 24983 consensus_queue.cc:260] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 22.12, Last appended by leader: 12, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:52.966181 24990 raft_consensus.cc:740] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:23:52.974027 24987 raft_consensus.cc:493] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:23:52.965649 24986 raft_consensus.cc:3060] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:23:52.975052 24987 raft_consensus.cc:515] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:52.973501 24990 consensus_queue.cc:260] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 21.13, Last appended by leader: 13, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:52.982335 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 43822.2us
I20260709 12:23:52.987418 25003 raft_consensus.cc:493] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:52.988173 25005 consensus_queue.cc:1059] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 15.12, Next index: 13, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:23:52.988519 25003 raft_consensus.cc:515] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:52.989667 25005 raft_consensus.cc:3055] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:23:52.990155 25005 raft_consensus.cc:740] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:23:52.991564 25005 consensus_queue.cc:260] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.12, Last appended by leader: 12, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:52.993008 17652 leader_election.cc:304] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:52.993621 25005 raft_consensus.cc:3060] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:52.994386 24898 raft_consensus.cc:3060] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:52.994869 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:23:53.007041 24946 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.083s	user 0.007s	sys 0.022s
I20260709 12:23:52.938189 24790 consensus_queue.cc:1048] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.017s
I20260709 12:23:52.997421 17652 leader_election.cc:304] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:52.992340 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:52.993186 24891 consensus_queue.cc:1048] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.018s
I20260709 12:23:53.037178 24983 raft_consensus.cc:3060] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:23:53.038434 18367 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 2076) took 1373 ms. Trace:
I20260709 12:23:53.038944 18367 rpcz_store.cc:276] 0709 12:23:51.664613 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:51.664748 (+   135us) service_pool.cc:224] Handling call
0709 12:23:51.665143 (+   395us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:52.552268 (+887125us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:53.037778 (+485510us) spinlock_profiling.cc:243] Waited 851 ms on lock 0x7b54004687d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:23:53.038205 (+   427us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1957230976}
I20260709 12:23:53.040366 17686 raft_consensus.cc:2368] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 17. Current term is 18.
I20260709 12:23:53.044222 24392 raft_consensus.cc:3055] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:23:52.991812 25003 leader_election.cc:290] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:23:53.032725 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 436154us
I20260709 12:23:53.061636 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:53.053819 24392 raft_consensus.cc:740] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:23:53.080717 24642 consensus_queue.cc:1059] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 15.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.023s
I20260709 12:23:53.082540 24642 raft_consensus.cc:3055] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:23:53.091432 24823 raft_consensus.cc:3055] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:23:53.092087 24823 raft_consensus.cc:740] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:23:53.093917 24823 consensus_queue.cc:260] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 17.14, Last appended by leader: 14, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:53.095412 24823 raft_consensus.cc:3060] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:23:53.099080 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 102249us
I20260709 12:23:53.103147 24705 raft_consensus.cc:493] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:23:53.097810 17802 leader_election.cc:304] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:53.104044 24705 raft_consensus.cc:515] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:53.099375 24721 consensus_queue.cc:1048] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 10, Time since last communication: 0.006s
I20260709 12:23:53.105317 24829 leader_election.cc:290] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:53.107558 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:53.110728 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:53.111465 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 22 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:53.111985 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 21 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:53.113006 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 21 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:53.113126 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:23:53.113790 17803 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:53.114607 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:53.077863 25003 raft_consensus.cc:2749] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:23:53.131340 17763 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 17.
I20260709 12:23:53.088467 24642 raft_consensus.cc:740] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
W20260709 12:23:53.140182 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 76383us
I20260709 12:23:53.148473 17761 raft_consensus.cc:2468] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 20.
I20260709 12:23:53.088832 24657 raft_consensus.cc:515] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:53.156958 17758 raft_consensus.cc:2410] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 21 because replica has last-logged OpId of term: 18 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 10.
I20260709 12:23:53.156888 25008 raft_consensus.cc:2749] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:23:53.159229 18333 leader_election.cc:304] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:23:53.162002 17652 leader_election.cc:304] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:53.162207 24898 raft_consensus.cc:2749] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:23:53.162835 25003 consensus_queue.cc:1048] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:23:53.160023 17829 raft_consensus.cc:2468] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 18.
I20260709 12:23:53.174618 24952 leader_election.cc:290] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:53.175308 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 21 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:53.176244 24642 consensus_queue.cc:260] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.12, Last appended by leader: 12, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:53.177510 24834 leader_election.cc:290] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:53.178223 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 22 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:53.178746 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 18 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:53.179193 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 18 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:53.179774 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:53.180802 24392 consensus_queue.cc:260] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 15.13, Last appended by leader: 13, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:53.182597 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 85156us
W20260709 12:23:53.182786 17728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:53.184913 17764 raft_consensus.cc:2468] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 21.
I20260709 12:23:53.217458 17757 raft_consensus.cc:2468] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 21.
W20260709 12:23:53.230537 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:53.230607 17681 raft_consensus.cc:1240] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 19. Current term is 20. Ops: []
I20260709 12:23:53.234169 17759 raft_consensus.cc:2468] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 21.
I20260709 12:23:53.235399 17680 raft_consensus.cc:2468] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 20.
W20260709 12:23:53.237191 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 67693.9us
W20260709 12:23:53.248849 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 89768.5us
I20260709 12:23:53.219194 24829 raft_consensus.cc:493] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:53.247449 17802 leader_election.cc:304] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:53.256965 17682 raft_consensus.cc:2410] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for term 18 because replica has last-logged OpId of term: 16 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 12.
I20260709 12:23:53.265909 17801 leader_election.cc:304] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:53.268102 24829 raft_consensus.cc:515] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:53.270066 24981 raft_consensus.cc:515] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:53.271143 24894 consensus_meta.cc:287] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.581s	user 0.008s	sys 0.013s
I20260709 12:23:53.271229 24829 leader_election.cc:290] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:53.272152 24790 raft_consensus.cc:2804] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:23:53.272475 24981 leader_election.cc:290] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 21 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:53.258343 24898 raft_consensus.cc:2804] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:23:53.272373 17728 leader_election.cc:304] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:53.273912 24829 raft_consensus.cc:2804] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:23:53.274364 24829 raft_consensus.cc:493] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:23:53.274804 24829 raft_consensus.cc:3060] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:53.286821 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:53.291579 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:53.292492 17829 raft_consensus.cc:3060] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:53.295104 17831 raft_consensus.cc:1240] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 16. Current term is 17. Ops: [16.15-16.15]
I20260709 12:23:53.296919 17837 raft_consensus.cc:1240] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 20. Current term is 21. Ops: []
I20260709 12:23:53.303149 24829 raft_consensus.cc:515] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:53.308482 24964 consensus_queue.cc:260] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.12, Last appended by leader: 12, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:53.309818 25000 log.cc:927] Time spent T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.518s	user 0.004s	sys 0.000s
I20260709 12:23:53.264722 24642 raft_consensus.cc:3060] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:53.309902 24964 raft_consensus.cc:3060] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Advancing to term 16
W20260709 12:23:53.311527 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 149746us
I20260709 12:23:53.311977 24898 raft_consensus.cc:697] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 22 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:23:53.310320 24829 leader_election.cc:290] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 21 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:23:53.321231 17730 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:53.324906 25019 raft_consensus.cc:2749] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 12:23:53.325443 17652 leader_election.cc:304] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
W20260709 12:23:53.326050 25022 log.cc:927] Time spent T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.107s	user 0.001s	sys 0.000s
W20260709 12:23:53.326663 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 62521.1us
W20260709 12:23:53.325459 17730 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:53.315202 24790 raft_consensus.cc:697] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 19 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:23:53.325901 24801 raft_consensus.cc:493] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:53.333004 17803 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:53.334692 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 19 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:23:53.344971 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 110248us
I20260709 12:23:53.345754 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:53.348260 24876 consensus_queue.cc:1048] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.013s
I20260709 12:23:53.350407 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:53.350687 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:53.351480 18365 raft_consensus.cc:2468] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 13.
I20260709 12:23:53.364614 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:53.365012 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:53.365091 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:53.366250 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:53.366653 18364 raft_consensus.cc:1275] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 13 index: 11. Preceding OpId from leader: term: 15 index: 12. (index mismatch)
I20260709 12:23:53.355790 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 21 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:53.352291 24813 raft_consensus.cc:2804] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:23:53.369122 24813 raft_consensus.cc:493] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:53.364627 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:53.356369 24431 leader_election.cc:290] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 17 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:23:53.375824 25025 log.cc:927] Time spent T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.106s	user 0.001s	sys 0.000s
I20260709 12:23:53.361243 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 17 candidate_status { last_received { term: 16 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:53.351567 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 20 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:53.369515 24813 raft_consensus.cc:3060] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:53.357436 24818 raft_consensus.cc:2804] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:23:53.390239 17801 leader_election.cc:304] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:53.342306 24801 raft_consensus.cc:515] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:53.352471 25005 consensus_queue.cc:1048] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.040s
I20260709 12:23:53.391105 24657 leader_election.cc:290] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 17 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:53.399195 25003 consensus_queue.cc:1048] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.013s
W20260709 12:23:53.412189 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:53.414651 17652 leader_election.cc:400] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 20 election: Vote denied by peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651) with higher term. Message: Invalid argument: T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for earlier term 20. Current term is 21.
I20260709 12:23:53.416831 24929 consensus_queue.cc:1048] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.034s
I20260709 12:23:53.418325 17802 leader_election.cc:304] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:53.420288 24802 raft_consensus.cc:493] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:53.403985 17762 raft_consensus.cc:2468] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 18.
W20260709 12:23:53.424319 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 44640.9us
I20260709 12:23:53.385735 24876 raft_consensus.cc:493] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:23:53.422662 18366 raft_consensus.cc:2468] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 16.
I20260709 12:23:53.403358 24818 raft_consensus.cc:697] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 14 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:23:53.420318 24935 leader_election.cc:290] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 16 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:53.440968 17801 leader_election.cc:304] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:53.444480 24876 raft_consensus.cc:515] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:53.445075 24802 raft_consensus.cc:515] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:53.439461 25019 raft_consensus.cc:2749] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 12:23:53.464478 17801 leader_election.cc:304] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:53.446894 24834 raft_consensus.cc:2804] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Leader pre-election won for term 22
W20260709 12:23:53.464545 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 81558.7us
I20260709 12:23:53.464560 18362 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 2098) took 1512 ms. Trace:
I20260709 12:23:53.472890 25019 raft_consensus.cc:2804] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:23:53.473363 25019 raft_consensus.cc:493] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:53.473786 25019 raft_consensus.cc:3060] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:23:53.475144 24818 consensus_queue.cc:237] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.8, Last appended by leader: 8, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:23:53.476276 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 58081.3us
I20260709 12:23:53.477103 17802 leader_election.cc:304] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:53.477178 24802 leader_election.cc:290] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:53.478626 17802 leader_election.cc:304] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:53.485193 17829 raft_consensus.cc:2468] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 16.
I20260709 12:23:53.484683 18333 leader_election.cc:304] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:53.486197 24957 raft_consensus.cc:2749] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:23:53.486948 24957 raft_consensus.cc:2749] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 12:23:53.487146 24801 leader_election.cc:290] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:53.488065 24935 raft_consensus.cc:2749] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:23:53.488831 24935 raft_consensus.cc:493] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:23:53.489490 24935 raft_consensus.cc:515] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:53.492194 24712 consensus_queue.cc:1059] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 20.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:23:53.472559 17801 leader_election.cc:304] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:53.492322 24962 consensus_queue.cc:1059] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 14.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.003s
W20260709 12:23:53.498153 24986 consensus_meta.cc:287] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.523s	user 0.006s	sys 0.009s
I20260709 12:23:53.498674 17652 leader_election.cc:304] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:53.498831 24986 raft_consensus.cc:515] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:53.501117 24723 raft_consensus.cc:2804] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:23:53.501628 24723 raft_consensus.cc:493] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:23:53.502170 24723 raft_consensus.cc:3060] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:53.506057 25005 raft_consensus.cc:493] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:23:53.440424 24898 consensus_queue.cc:237] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 21.13, Last appended by leader: 13, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:53.469501 24721 raft_consensus.cc:2804] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:23:53.468189 24834 raft_consensus.cc:493] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:23:53.510425 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:53.508934 24721 raft_consensus.cc:493] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:53.516482 24721 raft_consensus.cc:3060] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:23:53.516855 24657 raft_consensus.cc:2804] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:23:53.517413 24657 raft_consensus.cc:493] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:53.517819 24657 raft_consensus.cc:3060] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:53.437500 24790 consensus_queue.cc:237] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.13, Last appended by leader: 13, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:53.512552 24935 leader_election.cc:290] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:53.524344 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 18 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:53.525461 17758 raft_consensus.cc:2410] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 18 because replica has last-logged OpId of term: 15 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 10.
I20260709 12:23:53.520789 24823 raft_consensus.cc:493] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:23:53.532401 17838 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35040 (request call id 3061) took 1037 ms. Trace:
I20260709 12:23:53.532573 24823 raft_consensus.cc:515] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:53.532893 17838 rpcz_store.cc:276] 0709 12:23:52.495148 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:52.637225 (+142077us) service_pool.cc:224] Handling call
0709 12:23:52.637592 (+   367us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:52.685705 (+ 48113us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:53.531871 (+846166us) spinlock_profiling.cc:243] Waited 47.1 ms on lock 0x7b54003cc8d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:23:53.532212 (+   341us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":108386048}
I20260709 12:23:53.535045 24823 leader_election.cc:290] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:53.535116 24987 leader_election.cc:290] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:53.536072 17834 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35040 (request call id 3062) took 1004 ms. Trace:
I20260709 12:23:53.536634 17833 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35042 (request call id 2752) took 1068 ms. Trace:
I20260709 12:23:53.537310 17652 leader_election.cc:304] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:53.537117 17833 rpcz_store.cc:276] 0709 12:23:52.468525 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:52.474513 (+  5988us) service_pool.cc:224] Handling call
0709 12:23:53.536239 (+1061726us) spinlock_profiling.cc:243] Waited 59.2 ms on lock 0x7b54000abed4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:23:53.536399 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":136206080}
I20260709 12:23:53.538667 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:53.539294 17762 raft_consensus.cc:3060] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:53.539428 17759 raft_consensus.cc:1240] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 20. Current term is 21. Ops: []
I20260709 12:23:53.540263 24712 raft_consensus.cc:493] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:53.540916 24712 raft_consensus.cc:515] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:53.541941 24431 raft_consensus.cc:493] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:53.542563 24431 raft_consensus.cc:515] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:53.543349 25002 raft_consensus.cc:2804] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:23:53.543823 25002 raft_consensus.cc:493] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:23:53.544270 25002 raft_consensus.cc:3060] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:53.536492 17834 rpcz_store.cc:276] 0709 12:23:52.531144 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:52.655319 (+124175us) service_pool.cc:224] Handling call
0709 12:23:52.655808 (+   489us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:52.693679 (+ 37871us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:52.693727 (+    48us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:52.693787 (+    60us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:23:52.693833 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:53.535612 (+841779us) spinlock_profiling.cc:243] Waited 37.6 ms on lock 0x7b54000ab4d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:23:53.535678 (+    66us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:53.535842 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":86498560}
I20260709 12:23:53.511204 24834 raft_consensus.cc:3060] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:23:53.541577 24990 raft_consensus.cc:3060] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:23:53.507274 25005 raft_consensus.cc:515] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:53.467957 18362 rpcz_store.cc:276] 0709 12:23:51.951638 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:51.951763 (+   125us) service_pool.cc:224] Handling call
0709 12:23:52.062501 (+110738us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:23:52.154649 (+ 92148us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 12:23:52.520598 (+365949us) log.cc:844] Serialized 14 byte log entry
0709 12:23:52.723190 (+202592us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:23:52.752377 (+ 29187us) log.cc:844] Serialized 52 byte log entry
0709 12:23:52.752539 (+   162us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:23:52.752574 (+    35us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:52.752643 (+    69us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:53.308871 (+556228us) spinlock_profiling.cc:243] Waited 2.39 ms on lock 0x7b4400022b4c. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b47459c85 00007f7b4745a6a2 00007f7b4a1cc1db 00007f7b4a1c7aaf 00007f7b4a1bb86e 00007f7b4a1babb6 00007f7b4a1ba7d8 00007f7b4a1c4221 00007f7b4a1c41d9 00007f7b4a1c4169
0709 12:23:53.441406 (+132535us) raft_consensus.cc:1685] finished
0709 12:23:53.447763 (+  6357us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:53.464352 (+ 16589us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":180797,"prepare.queue_time_us":213718,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":44,"raft.queue_time_us":214979,"raft.run_cpu_time_us":311,"raft.run_wall_time_us":10098,"spinlock_wait_cycles":5498624,"thread_start_us":525193,"threads_started":3,"wal-append.queue_time_us":258297}
I20260709 12:23:53.576717 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 18 candidate_status { last_received { term: 16 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:53.577248 18368 raft_consensus.cc:1275] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 17 index: 13. Preceding OpId from leader: term: 19 index: 14. (index mismatch)
I20260709 12:23:53.578280 24431 leader_election.cc:290] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:53.578517 24823 raft_consensus.cc:493] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:53.543371 24712 leader_election.cc:290] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:53.580011 17652 leader_election.cc:304] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
W20260709 12:23:53.592644 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 114160us
W20260709 12:23:53.594065 17727 connection.cc:256] RPC call timeout handler was delayed by -1.1889293269996415s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1958s for a response
I20260709 12:23:53.594182 24829 raft_consensus.cc:493] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:53.596201 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:53.596387 24816 leader_election.cc:290] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:53.599774 24986 leader_election.cc:290] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 22 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:53.601166 24723 raft_consensus.cc:515] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:53.603490 24723 leader_election.cc:290] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 18 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:53.596457 24952 raft_consensus.cc:493] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:53.604501 24952 raft_consensus.cc:515] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:53.606820 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 14 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:53.607106 24952 leader_election.cc:290] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:53.607671 24823 raft_consensus.cc:515] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:53.608510 17803 leader_election.cc:304] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:23:53.609952 24650 raft_consensus.cc:2804] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:23:53.610591 24650 raft_consensus.cc:493] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:23:53.610973 24650 raft_consensus.cc:3060] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:53.611332 24983 consensus_queue.cc:1048] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 12:23:53.615851 18362 raft_consensus.cc:1240] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 17. Current term is 18. Ops: []
I20260709 12:23:53.617180 24876 leader_election.cc:290] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:53.609987 24823 leader_election.cc:290] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:53.607757 18367 raft_consensus.cc:2410] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for term 14 because replica has last-logged OpId of term: 12 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 10.
I20260709 12:23:53.595176 24829 raft_consensus.cc:515] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:53.596225 17727 leader_election.cc:304] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:53.620370 17803 leader_election.cc:304] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:53.607487 24721 raft_consensus.cc:515] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:53.629405 24986 raft_consensus.cc:2749] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 12:23:53.640481 24844 raft_consensus.cc:493] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:23:53.645418 18366 raft_consensus.cc:1275] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 16 index: 12. (index mismatch)
W20260709 12:23:53.646714 17582 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35966 (request call id 355) took 77640 ms (client timeout 15000 ms). Trace:
I20260709 12:23:53.648432 24894 raft_consensus.cc:2749] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
W20260709 12:23:53.649163 25035 log.cc:927] Time spent T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.058s	user 0.000s	sys 0.000s
I20260709 12:23:53.657976 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:53.658161 17762 raft_consensus.cc:2468] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 17.
I20260709 12:23:53.658015 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:53.659096 17802 leader_election.cc:304] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:53.641901 24844 raft_consensus.cc:3060] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:53.662670 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 22 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:53.663033 25005 leader_election.cc:290] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:23:53.663518 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 118458us
I20260709 12:23:53.659574 18362 raft_consensus.cc:2468] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 24.
I20260709 12:23:53.642393 25003 raft_consensus.cc:2804] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:23:53.673197 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:53.673843 18362 raft_consensus.cc:2468] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 21.
I20260709 12:23:53.677150 24721 leader_election.cc:290] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 22 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:53.681308 24642 raft_consensus.cc:3055] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:23:53.681751 24891 raft_consensus.cc:1064] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3: attempting to promote NON_VOTER 334a4e1f59944544b3b3ff36616b57da to VOTER
I20260709 12:23:53.681962 24642 raft_consensus.cc:740] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:23:53.683027 17652 leader_election.cc:304] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da
W20260709 12:23:53.683823 24891 raft_consensus.cc:1068] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3: Unable to promote non-voter 334a4e1f59944544b3b3ff36616b57da: Illegal state: Leader has not yet committed an operation in its own term
I20260709 12:23:53.684082 24767 consensus_queue.cc:1048] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:23:53.686744 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 18 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:53.669926 25003 raft_consensus.cc:493] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:53.689831 24974 raft_consensus.cc:3055] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 20 LEADER]: Stepping down as leader of term 20
W20260709 12:23:53.690083 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:53.689885 25003 raft_consensus.cc:3060] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:53.690356 24974 raft_consensus.cc:740] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:23:53.694466 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 18 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:53.694909 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 18 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:53.695269 24946 raft_consensus.cc:2804] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:23:53.695333 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:53.670405 25005 consensus_queue.cc:1048] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.026s
I20260709 12:23:53.696173 18374 raft_consensus.cc:2468] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 15.
I20260709 12:23:53.697264 24712 raft_consensus.cc:493] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:53.697489 24986 raft_consensus.cc:493] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:53.698062 24986 raft_consensus.cc:515] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:53.697902 24712 raft_consensus.cc:515] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:53.684788 24723 raft_consensus.cc:2804] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:23:53.692111 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:53.703421 24712 leader_election.cc:290] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:53.703904 24723 raft_consensus.cc:697] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 18 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
W20260709 12:23:53.694805 17729 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:53.690670 24705 leader_election.cc:290] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:53.700486 24986 leader_election.cc:290] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:53.708657 18365 raft_consensus.cc:2410] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for term 18 because replica has last-logged OpId of term: 17 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 10.
W20260709 12:23:53.708767 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 65583.7us
W20260709 12:23:53.715552 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 119576us
I20260709 12:23:53.695782 24946 raft_consensus.cc:493] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:23:53.719434 24952 raft_consensus.cc:493] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:53.729668 17802 leader_election.cc:304] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:53.720494 17803 leader_election.cc:304] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:23:53.725813 24946 raft_consensus.cc:3060] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:53.734596 24819 raft_consensus.cc:493] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:53.735199 24819 raft_consensus.cc:515] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:23:53.737112 24723 consensus_queue.cc:237] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 16.14, Last appended by leader: 14, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:53.737694 24829 leader_election.cc:290] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:53.738210 24829 raft_consensus.cc:493] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:53.738708 24829 raft_consensus.cc:515] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:53.747246 18367 raft_consensus.cc:2393] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 18: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:23:53.751068 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 18 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:23:53.754881 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 48987.2us
W20260709 12:23:53.756922 17803 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:53.757858 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:53.759548 18365 raft_consensus.cc:1275] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 16 index: 12. Preceding OpId from leader: term: 18 index: 14. (index mismatch)
I20260709 12:23:53.762516 18367 raft_consensus.cc:2393] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 15: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:23:53.767946 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:53.716090 24891 consensus_queue.cc:1048] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.037s
I20260709 12:23:53.769747 24721 raft_consensus.cc:2804] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:23:53.770004 25005 consensus_queue.cc:1048] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
W20260709 12:23:53.772589 17727 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.16.29.67:36837 timed out after 1.503s (SENT) [suppressed 21 similar messages]
I20260709 12:23:53.774756 17757 raft_consensus.cc:1275] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 15 index: 11. Preceding OpId from leader: term: 17 index: 12. (index mismatch)
I20260709 12:23:53.776413 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:53.777875 24819 leader_election.cc:290] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:53.793953 17758 raft_consensus.cc:3060] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Advancing to term 20
W20260709 12:23:53.794185 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 67724.4us
I20260709 12:23:53.794617 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:53.795308 17802 leader_election.cc:304] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:53.778524 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:53.796290 17764 raft_consensus.cc:3060] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:53.730882 24952 raft_consensus.cc:515] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:53.726725 24818 consensus_queue.cc:1048] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.042s
I20260709 12:23:53.780246 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:53.784025 17654 leader_election.cc:304] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:23:53.784884 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:53.786443 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:53.786475 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:53.787186 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 25 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:53.788489 17760 raft_consensus.cc:1275] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 18 index: 15. Preceding OpId from leader: term: 20 index: 16. (index mismatch)
I20260709 12:23:53.790279 17729 leader_election.cc:304] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:53.791157 24974 consensus_queue.cc:260] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 20.14, Last appended by leader: 14, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:53.795753 17757 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 17.
W20260709 12:23:53.781687 17728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:53.801988 24392 raft_consensus.cc:3060] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:53.804709 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:53.805488 24952 leader_election.cc:290] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:53.782214 24721 raft_consensus.cc:493] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:53.807464 24801 raft_consensus.cc:2749] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:23:53.807736 17830 raft_consensus.cc:1275] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 14 index: 10. Preceding OpId from leader: term: 17 index: 12. (index mismatch)
I20260709 12:23:53.808210 17836 raft_consensus.cc:1275] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 17 index: 10. Preceding OpId from leader: term: 19 index: 11. (index mismatch)
I20260709 12:23:53.808382 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:53.803426 24642 consensus_queue.cc:260] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 16.15, Last appended by leader: 15, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:53.809574 17838 raft_consensus.cc:1275] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 12 index: 9. Preceding OpId from leader: term: 16 index: 11. (index mismatch)
I20260709 12:23:53.810218 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:53.811013 24834 raft_consensus.cc:515] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:53.811443 17831 raft_consensus.cc:1275] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 16 index: 10. Preceding OpId from leader: term: 19 index: 12. (index mismatch)
I20260709 12:23:53.812287 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:53.766635 24986 raft_consensus.cc:2749] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:23:53.818403 24829 leader_election.cc:290] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:53.816957 24657 raft_consensus.cc:515] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:23:53.822815 17730 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:53.829732 24894 raft_consensus.cc:2804] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:23:53.831149 17758 raft_consensus.cc:2468] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 20.
I20260709 12:23:53.831408 17829 raft_consensus.cc:3060] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:53.833098 17759 raft_consensus.cc:2368] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 20. Current term is 21.
W20260709 12:23:53.832921 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 60358us
I20260709 12:23:53.833315 17755 raft_consensus.cc:2368] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 17. Current term is 18.
I20260709 12:23:53.836683 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:23:53.855041 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 73529.8us
I20260709 12:23:53.822242 24650 raft_consensus.cc:515] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:53.836229 17587 catalog_manager.cc:5697] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 13 to 15. New cstate: current_term: 15 leader_uuid: "334a4e1f59944544b3b3ff36616b57da" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:23:53.824379 17756 raft_consensus.cc:2376] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 16. Re-sending same reply.
I20260709 12:23:53.837337 24721 raft_consensus.cc:3060] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:53.841046 18333 leader_election.cc:304] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:23:53.868722 24960 raft_consensus.cc:2804] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:23:53.869451 24960 raft_consensus.cc:697] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 16 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
W20260709 12:23:53.869623 24964 consensus_meta.cc:287] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.549s	user 0.011s	sys 0.000s
W20260709 12:23:53.870458 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 71695.7us
I20260709 12:23:53.843151 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:53.848198 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:53.848198 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:53.855755 17763 raft_consensus.cc:2468] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 20.
W20260709 12:23:53.834501 24967 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 1.267s	user 0.005s	sys 0.007s
I20260709 12:23:53.843502 17801 leader_election.cc:304] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:53.837524 18334 leader_election.cc:304] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:53.853768 17836 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35042 (request call id 2757) took 1379 ms. Trace:
I20260709 12:23:53.851954 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:53.874433 17760 raft_consensus.cc:2468] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 15.
I20260709 12:23:53.870293 18363 raft_consensus.cc:2468] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 16.
I20260709 12:23:53.860139 24767 raft_consensus.cc:2804] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:23:53.861163 17832 raft_consensus.cc:2368] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 17. Current term is 18.
I20260709 12:23:53.856091 17654 leader_election.cc:304] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:23:53.828234 25019 raft_consensus.cc:515] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:53.873456 17836 rpcz_store.cc:276] 0709 12:23:52.470303 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:52.606282 (+135979us) service_pool.cc:224] Handling call
0709 12:23:53.787640 (+1181358us) spinlock_profiling.cc:243] Waited 872 us on lock 0x7b580013fe98. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b474fbb71 00007f7b474fba22 00007f7b504148d9 00007f7b4f56eee0 00007f7b4f6380ab 00007f7b4f638131 00007f7b4f5f9fc0 00007f7b4f5e4929 00007f7b4f5e43d2 00007f7b48b5bbd4
0709 12:23:53.792680 (+  5040us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:53.838014 (+ 45334us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:53.849917 (+ 11903us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2005888}
I20260709 12:23:53.883246 17838 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 17.
I20260709 12:23:53.877887 24767 raft_consensus.cc:493] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:23:53.883558 17729 leader_election.cc:304] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:53.884063 17835 raft_consensus.cc:1240] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 21. Current term is 22. Ops: []
I20260709 12:23:53.884683 17831 raft_consensus.cc:1240] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 20. Current term is 21. Ops: []
I20260709 12:23:53.891667 17758 raft_consensus.cc:1275] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 20 index: 15. Preceding OpId from leader: term: 23 index: 16. (index mismatch)
I20260709 12:23:53.885281 17727 leader_election.cc:304] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:53.884739 24960 consensus_queue.cc:237] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.12, Last appended by leader: 12, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:53.893880 24974 raft_consensus.cc:3060] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:53.894888 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:53.895695 17763 raft_consensus.cc:2468] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 19.
I20260709 12:23:53.885564 17761 raft_consensus.cc:2468] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 18.
I20260709 12:23:53.886051 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:53.897691 17756 raft_consensus.cc:3060] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:23:53.887567 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:53.892428 17757 raft_consensus.cc:3060] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:23:53.893898 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 20 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:23:53.877010 24967 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.509s	user 0.006s	sys 0.008s
I20260709 12:23:53.902354 17683 raft_consensus.cc:2410] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 20 because replica has last-logged OpId of term: 18 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 10.
W20260709 12:23:53.902410 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 46441.3us
I20260709 12:23:53.854736 24894 raft_consensus.cc:493] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:23:53.904538 17587 catalog_manager.cc:5697] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 9 to 18, leader changed from 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) to e063541542a94969898868bf6eb5ce1b (127.16.29.67). New cstate: current_term: 18 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:23:53.905478 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:53.898254 24767 raft_consensus.cc:3060] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:23:53.906492 17729 leader_election.cc:304] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:53.906194 24981 consensus_queue.cc:1059] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 20.13, Next index: 14, Last known committed idx: 11, Time since last communication: 0.011s
I20260709 12:23:53.890705 25005 consensus_queue.cc:1048] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.024s
I20260709 12:23:53.913687 17835 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35040 (request call id 3060) took 1443 ms. Trace:
I20260709 12:23:53.916061 24964 raft_consensus.cc:2749] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [term 19 LEARNER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:23:53.908603 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:53.933166 17685 raft_consensus.cc:3060] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:53.890905 24642 raft_consensus.cc:3060] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:53.883867 24807 raft_consensus.cc:2749] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:23:53.935765 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:53.903684 17834 raft_consensus.cc:2468] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 13.
I20260709 12:23:53.929551 17835 rpcz_store.cc:276] 0709 12:23:52.470707 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:52.636037 (+165330us) service_pool.cc:224] Handling call
0709 12:23:53.536563 (+900526us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:53.912079 (+375516us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:53.912761 (+   682us) spinlock_profiling.cc:243] Waited 340 ms on lock 0x7b54000abed4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:23:53.913476 (+   715us) spinlock_profiling.cc:243] Waited 900 ms on lock 0x7b54000abed0. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a61f8ad 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:23:53.913624 (+   148us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2852642432}
I20260709 12:23:53.932727 17764 raft_consensus.cc:2468] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 18.
I20260709 12:23:53.918146 17801 leader_election.cc:304] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:53.934533 17729 leader_election.cc:304] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:53.937364 17838 raft_consensus.cc:1240] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 16. Current term is 17. Ops: [16.15-16.15]
I20260709 12:23:53.904650 24819 raft_consensus.cc:2804] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:23:53.945358 17829 raft_consensus.cc:2468] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 20.
I20260709 12:23:53.950569 17801 leader_election.cc:304] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:53.919672 24894 raft_consensus.cc:3060] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:53.954747 17803 leader_election.cc:304] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:23:53.950527 17587 catalog_manager.cc:5697] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 16 to 19, leader changed from 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) to 334a4e1f59944544b3b3ff36616b57da (127.16.29.68). New cstate: current_term: 19 leader_uuid: "334a4e1f59944544b3b3ff36616b57da" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:23:53.959115 17801 leader_election.cc:304] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:53.961378 17727 leader_election.cc:304] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:23:53.961901 25019 leader_election.cc:290] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 22 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:53.962687 25019 raft_consensus.cc:2804] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Leader pre-election won for term 20
W20260709 12:23:53.962770 17714 heartbeater.cc:646] Failed to heartbeat to 127.16.29.126:39149 (4 consecutive failures): Timed out: Failed to send heartbeat to master: TSHeartbeat RPC to 127.16.29.126:39149 timed out after 15.000s (SENT)
I20260709 12:23:53.963209 25019 raft_consensus.cc:493] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:53.964228 17688 raft_consensus.cc:2393] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 13: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
W20260709 12:23:53.965195 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 127895us
I20260709 12:23:53.967115 18334 leader_election.cc:400] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 17 election: Vote denied by peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071) with higher term. Message: Invalid argument: T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 17. Current term is 18.
I20260709 12:23:53.967412 18334 leader_election.cc:403] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 17 election: Cancelling election due to peer responding with higher term
I20260709 12:23:53.968010 24962 raft_consensus.cc:3060] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:23:53.647359 17582 rpcz_store.cc:269] 0709 12:22:35.979205 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:36.007221 (+ 28016us) service_pool.cc:224] Handling call
0709 12:23:53.619229 (+77612008us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:23:46.170482 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:23:46.232400 (+ 61918us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:23:46.232449 (+    49us) write_op.cc:435] Acquired schema lock
0709 12:23:46.232466 (+    17us) tablet.cc:662] Decoding operations
0709 12:23:46.233074 (+   608us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:23:46.233168 (+    94us) write_op.cc:641] Partition lock acquired for write op
0709 12:23:46.233194 (+    26us) tablet.cc:685] Acquiring locks for 3 operations
0709 12:23:46.233595 (+   401us) tablet.cc:701] Row locks acquired
0709 12:23:46.233642 (+    47us) write_op.cc:260] PREPARE: finished
0709 12:23:46.233890 (+   248us) write_op.cc:270] Start()
0709 12:23:46.234034 (+   144us) write_op.cc:276] Timestamp: P: 1783599826233758 usec, L: 0
0709 12:23:46.234073 (+    39us) op_driver.cc:348] REPLICATION: starting
0709 12:23:46.243499 (+  9426us) log.cc:844] Serialized 1304 byte log entry
0709 12:23:47.434088 (+1190589us) op_driver.cc:464] REPLICATION: finished
0709 12:23:47.638747 (+204659us) write_op.cc:301] APPLY: starting
0709 12:23:47.648913 (+ 10166us) tablet.cc:1367] starting BulkCheckPresence
0709 12:23:47.659100 (+ 10187us) tablet.cc:1370] finished BulkCheckPresence
0709 12:23:47.659220 (+   120us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:23:47.676377 (+ 17157us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:23:47.676857 (+   480us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=6,key_file_lookups=6,delta_file_lookups=6,mrs_lookups=0
0709 12:23:47.676929 (+    72us) write_op.cc:312] APPLY: finished
0709 12:23:47.677319 (+   390us) log.cc:844] Serialized 41 byte log entry
0709 12:23:47.677612 (+   293us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:23:47.680336 (+  2724us) write_op.cc:454] Released schema lock
0709 12:23:47.680443 (+   107us) write_op.cc:341] FINISH: Updating metrics
0709 12:23:47.700735 (+ 20292us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:23:47.700766 (+    31us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":26480,"reported_tablets":46,"child_traces":[["op",{"apply.queue_time_us":184145,"apply.run_cpu_time_us":9316,"apply.run_wall_time_us":72361,"cfile_cache_hit":14,"cfile_cache_hit_bytes":16679,"num_ops":3,"prepare.queue_time_us":119750,"prepare.run_cpu_time_us":36158,"prepare.run_wall_time_us":715156,"raft.queue_time_us":405587,"raft.run_cpu_time_us":45605,"raft.run_wall_time_us":186473,"replication_time_us":1184244,"thread_start_us":1167584,"threads_started":4,"wal-append.queue_time_us":672970,"wal-append.run_cpu_time_us":79821,"wal-append.run_wall_time_us":1762288}]]}
W20260709 12:23:53.972750 25040 log.cc:927] Time spent T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.097s	user 0.000s	sys 0.000s
I20260709 12:23:53.968020 25006 raft_consensus.cc:2804] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:23:53.974606 25006 raft_consensus.cc:493] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:23:53.975106 25006 raft_consensus.cc:3060] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:23:53.938751 24952 raft_consensus.cc:2749] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:23:53.980345 24987 raft_consensus.cc:2804] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:23:53.937070 24829 raft_consensus.cc:2749] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 12:23:53.980844 24987 raft_consensus.cc:493] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:23:53.981287 24987 raft_consensus.cc:3060] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:23:53.981937 24813 consensus_meta.cc:287] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.589s	user 0.006s	sys 0.006s
I20260709 12:23:53.982368 24813 raft_consensus.cc:515] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:53.983449 17830 raft_consensus.cc:1240] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 20. Current term is 21. Ops: []
I20260709 12:23:53.938570 24983 raft_consensus.cc:3055] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:23:53.988083 24983 raft_consensus.cc:740] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 20 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:23:53.989048 24983 consensus_queue.cc:260] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.13, Last appended by leader: 13, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:53.990257 24983 raft_consensus.cc:3060] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:53.992281 17682 raft_consensus.cc:1275] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 20 index: 11. Preceding OpId from leader: term: 23 index: 12. (index mismatch)
W20260709 12:23:53.993629 25041 log.cc:927] Time spent T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.088s	user 0.000s	sys 0.000s
I20260709 12:23:53.994697 24962 raft_consensus.cc:2749] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Leader election lost for term 17. Reason: Vote denied by peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071) with higher term. Message: Invalid argument: T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 17. Current term is 18.
I20260709 12:23:53.915540 25002 raft_consensus.cc:515] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:53.992815 25006 raft_consensus.cc:515] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:53.980392 24957 raft_consensus.cc:2804] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:23:53.952452 24819 raft_consensus.cc:493] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:54.024961 25003 raft_consensus.cc:515] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:54.025828 24819 raft_consensus.cc:3060] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:54.025883 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 22 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:54.028661 24657 leader_election.cc:290] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 19 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:54.000442 25005 raft_consensus.cc:493] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:54.029816 25005 raft_consensus.cc:515] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:23:54.032042 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:54.033455 17830 raft_consensus.cc:1240] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 15. Current term is 16. Ops: []
I20260709 12:23:54.033576 24957 raft_consensus.cc:697] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 17 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:23:54.034729 24957 consensus_queue.cc:237] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 15.13, Last appended by leader: 13, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:54.034904 17587 catalog_manager.cc:5697] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 11 to 14, leader changed from 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) to 334a4e1f59944544b3b3ff36616b57da (127.16.29.68). New cstate: current_term: 14 leader_uuid: "334a4e1f59944544b3b3ff36616b57da" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:23:54.037272 17587 catalog_manager.cc:5697] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 12 to 16, leader changed from 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) to e063541542a94969898868bf6eb5ce1b (127.16.29.67). New cstate: current_term: 16 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:23:54.040777 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:54.041335 24981 raft_consensus.cc:2804] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:23:54.041718 24981 raft_consensus.cc:493] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:23:54.042089 24981 raft_consensus.cc:3060] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:54.044513 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:54.043794 24834 leader_election.cc:290] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 22 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:54.046584 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 18 candidate_status { last_received { term: 16 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:54.047343 17838 raft_consensus.cc:2468] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 17.
I20260709 12:23:54.051479 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 21 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:54.053776 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:23:54.055207 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 120764us
W20260709 12:23:54.056139 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 111065us
I20260709 12:23:54.018636 17756 raft_consensus.cc:2468] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 22.
I20260709 12:23:54.020002 24952 raft_consensus.cc:2749] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:23:54.023496 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:54.025120 17829 raft_consensus.cc:1275] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 17 index: 13. Preceding OpId from leader: term: 19 index: 14. (index mismatch)
I20260709 12:23:54.025300 25006 leader_election.cc:290] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 22 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:53.963825 25019 raft_consensus.cc:3060] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:54.057436 17835 raft_consensus.cc:2468] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 19.
I20260709 12:23:54.039155 17587 catalog_manager.cc:5697] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 13 to 16, leader changed from 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) to 334a4e1f59944544b3b3ff36616b57da (127.16.29.68). New cstate: current_term: 16 leader_uuid: "334a4e1f59944544b3b3ff36616b57da" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:23:54.069185 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 17 candidate_status { last_received { term: 16 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:54.082911 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:54.086416 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 14 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:54.087201 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:54.082969 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:54.088191 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:54.113552 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:54.113534 17833 raft_consensus.cc:2410] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for term 14 because replica has last-logged OpId of term: 12 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 10.
I20260709 12:23:54.027866 25003 leader_election.cc:290] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 17 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:23:54.118091 18333 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:54.063036 24829 raft_consensus.cc:2764] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Leader pre-election decision vote started in defunct term 21: won
I20260709 12:23:54.119396 17684 raft_consensus.cc:2468] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 13.
I20260709 12:23:54.086513 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:54.120285 17686 raft_consensus.cc:3060] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Advancing to term 18
I20260709 12:23:54.128731 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:54.136560 17681 raft_consensus.cc:1275] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 19 index: 14. Preceding OpId from leader: term: 21 index: 15. (index mismatch)
I20260709 12:23:54.136787 24650 leader_election.cc:290] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 13 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:54.137689 17831 raft_consensus.cc:1240] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 21. Current term is 22. Ops: [21.15-21.15]
I20260709 12:23:54.126137 24898 raft_consensus.cc:493] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:54.139681 24898 raft_consensus.cc:515] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:54.141991 24898 leader_election.cc:290] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:54.143270 17830 raft_consensus.cc:1240] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 21. Current term is 22. Ops: []
I20260709 12:23:54.143672 25005 leader_election.cc:290] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:54.127961 24834 raft_consensus.cc:493] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:54.144711 17830 raft_consensus.cc:1240] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 21. Current term is 22. Ops: [21.13-21.13]
I20260709 12:23:54.073058 24990 raft_consensus.cc:493] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:54.124758 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:23:54.148167 18333 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:54.148943 17836 raft_consensus.cc:2368] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for earlier term 18. Current term is 19.
I20260709 12:23:54.157785 17685 raft_consensus.cc:2468] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 21.
I20260709 12:23:54.163772 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:54.164615 18368 raft_consensus.cc:3060] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:54.166514 17757 raft_consensus.cc:2468] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 22.
I20260709 12:23:54.168164 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:54.170018 17681 raft_consensus.cc:1240] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 16. Current term is 17. Ops: []
I20260709 12:23:54.172156 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:54.174671 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:54.181136 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:54.176338 17833 raft_consensus.cc:2468] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 20.
W20260709 12:23:54.208971 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 171887us
I20260709 12:23:54.206672 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:54.144626 24834 raft_consensus.cc:515] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:54.212201 17587 catalog_manager.cc:5697] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 15 to 18. New cstate: current_term: 18 leader_uuid: "334a4e1f59944544b3b3ff36616b57da" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 12:23:54.218277 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 259246us
W20260709 12:23:54.225703 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:54.226369 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 25 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:54.231277 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:54.232084 17835 raft_consensus.cc:3060] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:54.240239 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:54.241570 17683 raft_consensus.cc:1240] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 21. Current term is 22. Ops: []
I20260709 12:23:54.243848 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:54.244469 17685 raft_consensus.cc:2468] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 18.
I20260709 12:23:54.184053 24813 leader_election.cc:290] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 21 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:54.243356 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:54.165690 24952 consensus_queue.cc:1048] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.067s
I20260709 12:23:54.212029 24891 raft_consensus.cc:493] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:54.250530 17833 raft_consensus.cc:2393] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 21: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:23:54.162710 24990 raft_consensus.cc:515] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:54.256815 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 193682us
I20260709 12:23:54.263427 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:54.267582 17684 raft_consensus.cc:2468] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 20.
I20260709 12:23:54.267972 17689 raft_consensus.cc:1240] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 15. Current term is 16. Ops: []
I20260709 12:23:54.269963 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:54.273486 17688 raft_consensus.cc:1240] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 20. Current term is 21. Ops: [18.12-20.13]
I20260709 12:23:54.269150 17652 leader_election.cc:304] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:54.233106 25003 consensus_queue.cc:1059] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.039s
I20260709 12:23:54.278239 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:54.242421 17587 catalog_manager.cc:5697] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 14 to 18, leader changed from e063541542a94969898868bf6eb5ce1b (127.16.29.67) to e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66), now has a pending config: VOTER 0da169ff82674dc1947055ce43de68a3 (127.16.29.65), VOTER e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66), VOTER e063541542a94969898868bf6eb5ce1b (127.16.29.67), VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68). New cstate: current_term: 18 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } } pending_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:23:54.279078 17833 raft_consensus.cc:2393] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 22: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:23:54.277719 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:54.279835 17684 raft_consensus.cc:2468] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 18.
W20260709 12:23:54.287067 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 325761us
I20260709 12:23:54.287423 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 22 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:54.287808 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 21 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:54.266791 24650 raft_consensus.cc:493] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:54.288592 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:54.299923 17829 raft_consensus.cc:2410] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for term 22 because replica has last-logged OpId of term: 20 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 11.
I20260709 12:23:54.299827 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:54.300688 17684 raft_consensus.cc:1275] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 18 index: 11. Preceding OpId from leader: term: 19 index: 12. (index mismatch)
W20260709 12:23:54.293243 24844 consensus_meta.cc:287] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.592s	user 0.012s	sys 0.000s
I20260709 12:23:54.305606 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:54.320892 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:54.322053 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:54.317783 18334 leader_election.cc:304] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:23:54.328333 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:23:54.331235 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 83178us
I20260709 12:23:54.335768 17587 catalog_manager.cc:5697] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 10 to 15, leader changed from e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66) to 334a4e1f59944544b3b3ff36616b57da (127.16.29.68). New cstate: current_term: 15 leader_uuid: "334a4e1f59944544b3b3ff36616b57da" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:23:54.338218 17652 leader_election.cc:304] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da
W20260709 12:23:54.316111 24946 consensus_meta.cc:287] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.583s	user 0.014s	sys 0.000s
I20260709 12:23:54.341722 17757 raft_consensus.cc:1275] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 14 index: 10. Preceding OpId from leader: term: 16 index: 11. (index mismatch)
I20260709 12:23:54.343238 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:54.340876 18334 leader_election.cc:304] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:23:54.320132 24650 raft_consensus.cc:515] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:23:54.347983 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
W20260709 12:23:54.348855 25048 log.cc:927] Time spent T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.101s	user 0.001s	sys 0.000s
I20260709 12:23:54.349213 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 21 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:23:54.350179 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 48100.9us
I20260709 12:23:54.350349 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 24 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:54.327668 24818 raft_consensus.cc:2804] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:23:54.351178 17727 leader_election.cc:304] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:54.351495 24818 raft_consensus.cc:493] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:54.351938 24818 raft_consensus.cc:3060] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:54.352061 24823 raft_consensus.cc:2804] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:23:54.352520 24823 raft_consensus.cc:493] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:54.352932 24823 raft_consensus.cc:3060] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:54.356148 24952 raft_consensus.cc:2749] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:23:54.356894 24952 consensus_queue.cc:1048] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 12:23:54.357343 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:54.358201 24952 raft_consensus.cc:493] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:54.358441 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:54.359045 18365 raft_consensus.cc:2468] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 16.
I20260709 12:23:54.363988 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:54.364176 24962 consensus_queue.cc:1048] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:23:54.365134 18365 raft_consensus.cc:2393] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 20: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:23:54.365713 24962 consensus_queue.cc:1059] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 12:23:54.366550 24964 raft_consensus.cc:3055] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:23:54.366888 24964 raft_consensus.cc:740] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:23:54.368136 24964 consensus_queue.cc:260] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.11, Last appended by leader: 11, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:54.370083 24964 raft_consensus.cc:3060] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:54.327606 24967 raft_consensus.cc:493] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:54.371596 24967 raft_consensus.cc:515] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:54.372560 24818 raft_consensus.cc:515] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:54.373494 24392 raft_consensus.cc:2804] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:23:54.373991 24392 raft_consensus.cc:493] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:54.374395 24392 raft_consensus.cc:3060] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:54.374423 24967 leader_election.cc:290] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:54.370111 17654 leader_election.cc:304] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:23:54.318962 25002 leader_election.cc:290] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 21 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:54.344034 17587 catalog_manager.cc:5697] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 8 to 12, leader changed from 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) to 334a4e1f59944544b3b3ff36616b57da (127.16.29.68). New cstate: current_term: 12 leader_uuid: "334a4e1f59944544b3b3ff36616b57da" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:23:54.376242 25053 raft_consensus.cc:493] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:54.376814 25053 raft_consensus.cc:515] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:54.377319 24927 raft_consensus.cc:2804] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:23:54.278609 24891 raft_consensus.cc:515] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:54.314957 25056 raft_consensus.cc:493] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:54.377763 24927 raft_consensus.cc:493] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:54.377202 17587 catalog_manager.cc:5697] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 9 to 13. New cstate: current_term: 13 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:23:54.370831 18363 raft_consensus.cc:2468] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 13.
I20260709 12:23:54.356184 24990 leader_election.cc:290] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:54.388720 24677 raft_consensus.cc:2804] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Leader pre-election won for term 14
W20260709 12:23:54.389151 25051 log.cc:927] Time spent T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.102s	user 0.000s	sys 0.000s
W20260709 12:23:54.390146 25046 log.cc:927] Time spent T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.074s	user 0.000s	sys 0.000s
I20260709 12:23:54.390378 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:54.391273 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 18 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:54.392817 18364 raft_consensus.cc:1275] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 16 index: 13. Preceding OpId from leader: term: 18 index: 14. (index mismatch)
I20260709 12:23:54.394568 17756 raft_consensus.cc:1275] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 18 index: 13. Preceding OpId from leader: term: 21 index: 15. (index mismatch)
I20260709 12:23:54.395015 17683 raft_consensus.cc:1275] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 21 index: 15. Preceding OpId from leader: term: 23 index: 16. (index mismatch)
I20260709 12:23:54.388325 24927 raft_consensus.cc:3060] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:54.371546 24819 raft_consensus.cc:515] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:54.396875 17832 raft_consensus.cc:2410] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for term 24 because replica has last-logged OpId of term: 22 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 11.
I20260709 12:23:54.372886 18334 leader_election.cc:304] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:23:54.413451 18365 raft_consensus.cc:1275] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 21 index: 13. Preceding OpId from leader: term: 22 index: 14. (index mismatch)
W20260709 12:23:54.414772 18334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:54.400207 18368 raft_consensus.cc:2468] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 17.
W20260709 12:23:54.400833 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 174224us
I20260709 12:23:54.358680 24952 raft_consensus.cc:515] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:54.412838 17587 catalog_manager.cc:5697] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 12 to 15, leader changed from e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66) to 334a4e1f59944544b3b3ff36616b57da (127.16.29.68). New cstate: current_term: 15 leader_uuid: "334a4e1f59944544b3b3ff36616b57da" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:23:54.410290 17686 raft_consensus.cc:2468] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 18.
W20260709 12:23:54.420212 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:54.410730 17835 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 17.
I20260709 12:23:54.411461 17762 raft_consensus.cc:1275] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 14 index: 12. Preceding OpId from leader: term: 16 index: 13. (index mismatch)
W20260709 12:23:54.426828 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:54.411409 17652 leader_election.cc:304] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:54.416055 17801 leader_election.cc:304] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:54.417557 17761 raft_consensus.cc:1275] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 16 index: 12. (index mismatch)
I20260709 12:23:54.302218 24721 raft_consensus.cc:515] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:54.397827 17728 leader_election.cc:304] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:54.435848 17728 leader_election.cc:304] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:54.440371 17756 raft_consensus.cc:1240] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 21. Current term is 22. Ops: []
W20260709 12:23:54.407976 25049 log.cc:927] Time spent T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.114s	user 0.000s	sys 0.000s
I20260709 12:23:54.442509 17835 raft_consensus.cc:1275] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 17 index: 13. Preceding OpId from leader: term: 18 index: 14. (index mismatch)
I20260709 12:23:54.311932 25043 consensus_queue.cc:1048] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.109s
I20260709 12:23:54.444168 17801 leader_election.cc:304] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:54.447476 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 18 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:54.448446 17835 raft_consensus.cc:2468] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 17.
W20260709 12:23:54.454113 17801 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:54.379179 25053 leader_election.cc:290] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:54.396579 25003 consensus_queue.cc:1048] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.104s
I20260709 12:23:54.439466 24989 raft_consensus.cc:2804] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:23:54.469308 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:54.328025 24844 raft_consensus.cc:515] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:54.470520 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 59241.4us
I20260709 12:23:54.482801 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:54.456532 24986 raft_consensus.cc:2804] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:23:54.408494 24946 raft_consensus.cc:515] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:54.440057 24962 consensus_queue.cc:1048] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.044s
I20260709 12:23:54.533820 17762 raft_consensus.cc:1240] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 21. Current term is 22. Ops: []
I20260709 12:23:54.465281 25002 raft_consensus.cc:2749] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:23:54.529320 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:54.388273 25056 raft_consensus.cc:515] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:54.483536 24989 raft_consensus.cc:493] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:54.470651 24834 leader_election.cc:290] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:54.552598 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:54.553196 17762 raft_consensus.cc:1275] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 18 index: 13. Preceding OpId from leader: term: 19 index: 14. (index mismatch)
I20260709 12:23:54.553503 17835 raft_consensus.cc:2393] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 20: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:23:54.554308 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:54.554483 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 22 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:54.555444 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:54.555995 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:54.555445 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:54.556563 17686 raft_consensus.cc:3060] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:54.556866 17838 raft_consensus.cc:2376] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 18. Re-sending same reply.
I20260709 12:23:54.558012 17838 raft_consensus.cc:1275] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 22 index: 12. Preceding OpId from leader: term: 23 index: 12. (term mismatch)
I20260709 12:23:54.558281 17838 pending_rounds.cc:85] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b: Aborting all ops after (but not including) 11
I20260709 12:23:54.558503 17838 pending_rounds.cc:107] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b: Aborting uncommitted NO_OP operation due to leader change: 22.12
I20260709 12:23:54.559624 24650 leader_election.cc:290] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:54.561508 24844 leader_election.cc:290] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 19 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:54.565137 24723 raft_consensus.cc:3055] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:23:54.565590 24723 raft_consensus.cc:740] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:23:54.565857 25059 raft_consensus.cc:2749] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:23:54.565977 24952 leader_election.cc:290] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:54.552623 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:54.570488 17763 raft_consensus.cc:2468] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 16.
I20260709 12:23:54.555397 17835 raft_consensus.cc:2393] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 22: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
W20260709 12:23:54.572395 24642 consensus_meta.cc:287] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.626s	user 0.009s	sys 0.005s
I20260709 12:23:54.536361 25006 consensus_queue.cc:1059] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 20.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.039s
I20260709 12:23:54.566469 24723 consensus_queue.cc:260] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 21.12, Last appended by leader: 12, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:54.576581 24723 raft_consensus.cc:3060] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:23:54.575695 17587 catalog_manager.cc:5697] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 10 to 16, leader changed from e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66) to 334a4e1f59944544b3b3ff36616b57da (127.16.29.68). New cstate: current_term: 16 leader_uuid: "334a4e1f59944544b3b3ff36616b57da" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:23:54.439126 24894 raft_consensus.cc:515] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:54.589167 24894 leader_election.cc:290] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 16 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:54.484229 25053 raft_consensus.cc:2804] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:23:54.558756 17838 raft_consensus.cc:2889] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:54.579784 24952 consensus_queue.cc:1048] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:23:54.580807 17727 leader_election.cc:304] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:54.530655 24813 raft_consensus.cc:493] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:54.586959 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:54.592365 17837 raft_consensus.cc:1240] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 17. Current term is 18. Ops: []
I20260709 12:23:54.547871 24992 raft_consensus.cc:2804] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:23:54.593531 17587 catalog_manager.cc:5697] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 14 to 17. New cstate: current_term: 17 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:23:54.600744 17803 leader_election.cc:304] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:23:54.603545 17686 raft_consensus.cc:2468] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 19.
I20260709 12:23:54.618963 17759 raft_consensus.cc:2393] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 18: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:23:54.619426 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 22 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:54.619684 18333 leader_election.cc:304] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:54.620148 17762 raft_consensus.cc:2468] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 21.
I20260709 12:23:54.621384 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 22 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:54.621559 24957 raft_consensus.cc:2804] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:23:54.621836 24819 leader_election.cc:290] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 16 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:54.622205 17680 raft_consensus.cc:2410] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 22 because replica has last-logged OpId of term: 21 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 11.
I20260709 12:23:54.622118 24957 raft_consensus.cc:697] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 13 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:23:54.624375 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:54.625497 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:54.626101 17763 raft_consensus.cc:2468] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 22.
I20260709 12:23:54.626766 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 20 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:23:54.627573 24987 consensus_meta.cc:287] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.646s	user 0.007s	sys 0.000s
W20260709 12:23:54.627976 24767 consensus_meta.cc:287] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.683s	user 0.010s	sys 0.003s
I20260709 12:23:54.620622 25005 consensus_queue.cc:1059] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 21.11, Next index: 12, Last known committed idx: 10, Time since last communication: 0.008s
I20260709 12:23:54.629194 25006 raft_consensus.cc:3055] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 21 LEADER]: Stepping down as leader of term 21
W20260709 12:23:54.647544 24983 consensus_meta.cc:287] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.657s	user 0.007s	sys 0.000s
I20260709 12:23:54.632944 24946 leader_election.cc:290] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 18 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:54.634017 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:54.697986 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:54.634202 24677 raft_consensus.cc:493] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:54.634740 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:23:54.635416 25054 log.cc:927] Time spent T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.288s	user 0.001s	sys 0.000s
I20260709 12:23:54.635463 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
W20260709 12:23:54.701953 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 121227us
I20260709 12:23:54.702410 18366 raft_consensus.cc:3060] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:54.636543 17587 catalog_manager.cc:5697] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 12 to 14, leader changed from 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) to e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66). New cstate: current_term: 14 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:23:54.640306 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:54.640374 17802 leader_election.cc:304] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:54.641290 24642 consensus_queue.cc:1048] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:23:54.623265 24957 consensus_queue.cc:237] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:54.703188 17834 raft_consensus.cc:2468] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 21.
I20260709 12:23:54.648047 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:54.648209 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:54.649276 17759 raft_consensus.cc:1275] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 21 index: 15. (index mismatch)
I20260709 12:23:54.699177 18371 raft_consensus.cc:2468] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 16.
I20260709 12:23:54.649688 24818 leader_election.cc:290] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 18 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:54.652441 18334 leader_election.cc:304] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:23:54.652932 24816 raft_consensus.cc:493] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:23:54.654662 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 18 candidate_status { last_received { term: 16 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:54.709571 18363 raft_consensus.cc:3060] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:54.655684 18364 raft_consensus.cc:1275] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 16 index: 14. Preceding OpId from leader: term: 18 index: 15. (index mismatch)
I20260709 12:23:54.656483 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 24 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:23:54.656920 25061 log.cc:927] Time spent T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.108s	user 0.001s	sys 0.000s
I20260709 12:23:54.651629 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:54.660172 24898 raft_consensus.cc:493] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:54.660435 24790 raft_consensus.cc:2749] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:23:54.661334 18333 leader_election.cc:400] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 17 election: Vote denied by peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837) with higher term. Message: Invalid argument: T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 17. Current term is 18.
I20260709 12:23:54.661676 24721 leader_election.cc:290] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 17 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:54.544821 24986 raft_consensus.cc:697] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 22 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:23:54.661334 24960 consensus_queue.cc:1048] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:23:54.627400 24927 raft_consensus.cc:515] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:54.666091 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:54.736944 17687 raft_consensus.cc:2393] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 13: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:23:54.666320 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:54.667016 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:54.666837 24813 raft_consensus.cc:515] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:54.591509 25056 leader_election.cc:290] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:54.670804 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:54.593637 24989 raft_consensus.cc:3060] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:54.626814 24992 raft_consensus.cc:697] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 22 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
W20260709 12:23:54.669143 24981 consensus_meta.cc:287] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.617s	user 0.007s	sys 0.000s
W20260709 12:23:54.706344 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:54.665315 24392 raft_consensus.cc:515] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:54.707090 17727 leader_election.cc:304] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:54.684634 24819 raft_consensus.cc:2749] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:23:54.713125 17763 raft_consensus.cc:3060] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Advancing to term 19
W20260709 12:23:54.718623 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 78352us
I20260709 12:23:54.690016 25006 raft_consensus.cc:740] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:23:54.695472 24767 raft_consensus.cc:515] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:54.721473 17587 catalog_manager.cc:5697] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 15 to 18, leader changed from 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) to 334a4e1f59944544b3b3ff36616b57da (127.16.29.68). New cstate: current_term: 18 leader_uuid: "334a4e1f59944544b3b3ff36616b57da" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:23:54.724773 17729 leader_election.cc:304] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:54.737092 24974 raft_consensus.cc:2804] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:23:54.702467 24677 raft_consensus.cc:3060] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:54.693125 24723 consensus_queue.cc:1048] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 10, Time since last communication: 0.036s
I20260709 12:23:54.743548 17762 raft_consensus.cc:3060] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:54.746985 24927 leader_election.cc:290] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 17 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:23:54.748176 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 87000.6us
I20260709 12:23:54.689497 24987 raft_consensus.cc:515] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:54.749702 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 221125us
I20260709 12:23:54.752591 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 21 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:54.695385 24823 raft_consensus.cc:515] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:54.771997 17759 raft_consensus.cc:3060] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:54.773713 24823 leader_election.cc:290] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 20 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:54.690253 25053 raft_consensus.cc:697] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
W20260709 12:23:54.727810 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 75444.2us
I20260709 12:23:54.719444 24946 raft_consensus.cc:2749] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:23:54.761960 17802 leader_election.cc:304] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:54.777386 17685 raft_consensus.cc:1240] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 19. Current term is 20. Ops: [19.12-19.12]
I20260709 12:23:54.776859 25053 consensus_queue.cc:237] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 20.13, Last appended by leader: 13, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:54.787595 17652 leader_election.cc:400] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 18 election: Vote denied by peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837) with higher term. Message: Invalid argument: T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for earlier term 18. Current term is 19.
I20260709 12:23:54.788267 17652 leader_election.cc:403] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 18 election: Cancelling election due to peer responding with higher term
I20260709 12:23:54.763187 17834 raft_consensus.cc:1240] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 17 LEADER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 15. Current term is 17. Ops: []
I20260709 12:23:54.730050 24891 leader_election.cc:290] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:54.768522 17682 raft_consensus.cc:1240] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 24. Current term is 25. Ops: []
I20260709 12:23:54.732266 24898 raft_consensus.cc:515] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:54.792354 17587 catalog_manager.cc:5697] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 6 to 15. New cstate: current_term: 15 leader_uuid: "334a4e1f59944544b3b3ff36616b57da" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:23:54.797024 17652 leader_election.cc:304] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:23:54.775866 24823 raft_consensus.cc:1064] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3: attempting to promote NON_VOTER 334a4e1f59944544b3b3ff36616b57da to VOTER
W20260709 12:23:54.769223 25019 consensus_meta.cc:287] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.653s	user 0.009s	sys 0.002s
I20260709 12:23:54.824172 25019 raft_consensus.cc:515] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:54.826892 25019 leader_election.cc:290] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 20 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:54.827977 17832 raft_consensus.cc:2393] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 20: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:23:54.828176 24927 consensus_queue.cc:1059] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 20.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 12:23:54.829046 24891 raft_consensus.cc:2749] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Leader election lost for term 18. Reason: Vote denied by peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837) with higher term. Message: Invalid argument: T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for earlier term 18. Current term is 19.
I20260709 12:23:54.829339 17652 leader_election.cc:304] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:54.830447 17652 leader_election.cc:304] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:54.831171 25003 raft_consensus.cc:3055] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:23:54.831516 25003 raft_consensus.cc:740] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:23:54.832370 25003 consensus_queue.cc:260] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 21.13, Last appended by leader: 13, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:54.747905 24986 consensus_queue.cc:237] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 20.12, Last appended by leader: 12, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:54.798017 18363 raft_consensus.cc:2468] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 18.
I20260709 12:23:54.806674 18374 raft_consensus.cc:2410] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for term 17 because replica has last-logged OpId of term: 16 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 10.
I20260709 12:23:54.806974 17829 raft_consensus.cc:2468] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 15.
I20260709 12:23:54.770434 25067 raft_consensus.cc:2804] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:23:54.770778 25056 raft_consensus.cc:3060] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:54.806531 17802 leader_election.cc:304] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:54.757076 24981 raft_consensus.cc:515] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:54.787604 24946 raft_consensus.cc:2749] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:23:54.787206 24974 raft_consensus.cc:697] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 18 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:23:54.820624 18333 leader_election.cc:304] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
W20260709 12:23:54.821877 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 118929us
W20260709 12:23:54.823160 25071 log.cc:927] Time spent T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.070s	user 0.000s	sys 0.001s
I20260709 12:23:54.720322 24816 raft_consensus.cc:515] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:54.836215 17759 raft_consensus.cc:2468] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 21.
I20260709 12:23:54.859474 17727 leader_election.cc:304] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:54.840054 17763 raft_consensus.cc:2468] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 19.
I20260709 12:23:54.853760 17684 raft_consensus.cc:1275] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 16 index: 13. (index mismatch)
I20260709 12:23:54.850304 17587 catalog_manager.cc:5697] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 15 to 20, leader changed from e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66) to 334a4e1f59944544b3b3ff36616b57da (127.16.29.68). New cstate: current_term: 20 leader_uuid: "334a4e1f59944544b3b3ff36616b57da" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:23:54.860904 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:54.852821 24801 raft_consensus.cc:493] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:54.862622 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 20 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:54.848345 24927 raft_consensus.cc:2804] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:23:54.862998 17587 catalog_manager.cc:5697] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 9 to 13. New cstate: current_term: 13 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:23:54.848800 25067 raft_consensus.cc:493] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:23:54.865531 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 59070.6us
I20260709 12:23:54.865002 24801 raft_consensus.cc:515] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:54.854846 25003 raft_consensus.cc:3060] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:23:54.870167 17802 leader_election.cc:304] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:54.880234 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 17 candidate_status { last_received { term: 16 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:54.887203 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:54.888960 17727 leader_election.cc:304] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:54.892679 17762 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 17.
I20260709 12:23:54.823415 24823 consensus_queue.cc:237] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 19.11, Last appended by leader: 10, Current term: 19, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:54.899742 18365 raft_consensus.cc:1275] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 15 index: 13. Preceding OpId from leader: term: 17 index: 14. (index mismatch)
I20260709 12:23:54.899956 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:54.868728 24964 raft_consensus.cc:2804] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:23:54.865999 24927 raft_consensus.cc:493] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:23:54.918869 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:54.919673 17759 raft_consensus.cc:3060] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:54.913277 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 22 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:54.920650 17763 raft_consensus.cc:3060] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:23:54.926190 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 24 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:54.870720 25082 raft_consensus.cc:2749] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:23:54.871469 25067 raft_consensus.cc:3060] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:54.863235 24642 consensus_queue.cc:1059] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.040s
I20260709 12:23:54.925571 18333 leader_election.cc:304] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:23:54.948906 24642 consensus_queue.cc:1243] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer e063541542a94969898868bf6eb5ce1b log is divergent from this leader: its last log entry 19.16 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 15
I20260709 12:23:54.949415 24962 raft_consensus.cc:3055] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:23:54.949546 24642 raft_consensus.cc:2804] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:23:54.949826 24962 raft_consensus.cc:740] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:23:54.950047 18333 leader_election.cc:304] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:54.950084 24642 raft_consensus.cc:493] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:54.950816 24962 consensus_queue.cc:260] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 20.16, Last appended by leader: 16, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:54.951592 24642 raft_consensus.cc:3060] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:54.934465 24990 consensus_queue.cc:1048] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.022s
I20260709 12:23:54.952330 24962 raft_consensus.cc:3060] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Advancing to term 21
W20260709 12:23:54.952036 24967 consensus_queue.cc:1175] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" responder_term: 20 status { last_received { term: 18 index: 15 } last_committed_idx: 15 error { code: PRECEDING_ENTRY_DIDNT_MATCH status { code: ILLEGAL_STATE message: "Log matching property violated. Preceding OpId in replica: term: 18 index: 15. Preceding OpId from leader: term: 20 index: 16. (index mismatch)" } } last_received_current_leader { term: 0 index: 0 } }
I20260709 12:23:54.953389 24990 raft_consensus.cc:2749] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:23:54.954780 24898 leader_election.cc:290] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:54.959077 25080 raft_consensus.cc:2749] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:23:54.959700 18366 raft_consensus.cc:2468] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 16.
I20260709 12:23:54.960093 25078 raft_consensus.cc:3060] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Advancing to term 22
W20260709 12:23:54.964421 25074 log.cc:927] Time spent T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.096s	user 0.000s	sys 0.000s
I20260709 12:23:54.965184 24989 raft_consensus.cc:515] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:54.967324 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:54.969764 17757 raft_consensus.cc:1275] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 18 index: 11. Preceding OpId from leader: term: 22 index: 14. (index mismatch)
I20260709 12:23:54.970333 17801 leader_election.cc:304] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:54.971032 17757 raft_consensus.cc:1240] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 15. Current term is 16. Ops: []
I20260709 12:23:54.932348 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
W20260709 12:23:54.940212 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 69113.1us
I20260709 12:23:54.971581 17801 leader_election.cc:304] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:54.855427 24992 consensus_queue.cc:237] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 21.11, Last appended by leader: 11, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:54.896733 24723 raft_consensus.cc:2749] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:23:54.940011 25050 raft_consensus.cc:493] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:54.988916 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 319176us
I20260709 12:23:54.945350 24801 leader_election.cc:290] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:54.946859 24816 leader_election.cc:290] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:54.948132 24767 leader_election.cc:290] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 25 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:54.853507 25006 consensus_queue.cc:260] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 21.15, Last appended by leader: 15, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:54.952029 24392 leader_election.cc:290] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 21 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:54.964252 17729 leader_election.cc:304] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:23:55.013789 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 49639.9us
I20260709 12:23:54.934285 24974 consensus_queue.cc:237] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 16.13, Last appended by leader: 13, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:54.944643 24987 leader_election.cc:290] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 15 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:54.972251 17755 raft_consensus.cc:2368] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 15. Current term is 16.
I20260709 12:23:54.911686 25019 raft_consensus.cc:493] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:54.973590 17757 raft_consensus.cc:1240] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 18. Current term is 19. Ops: []
I20260709 12:23:54.976364 17762 raft_consensus.cc:1275] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 15 index: 13. Preceding OpId from leader: term: 17 index: 14. (index mismatch)
I20260709 12:23:54.978024 17727 leader_election.cc:304] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:54.981383 17837 raft_consensus.cc:1240] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 19. Current term is 20. Ops: []
W20260709 12:23:54.978066 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 108008us
I20260709 12:23:54.923437 24927 raft_consensus.cc:3060] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:55.000679 17728 leader_election.cc:304] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:55.025427 25077 raft_consensus.cc:2749] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:23:55.026389 24952 consensus_queue.cc:1059] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 24.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
W20260709 12:23:55.008270 25071 log.cc:927] Time spent T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.078s	user 0.000s	sys 0.000s
I20260709 12:23:54.977603 25002 raft_consensus.cc:3055] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:23:55.002640 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:55.004689 17763 raft_consensus.cc:2468] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 22.
W20260709 12:23:55.007727 18334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:55.032083 25079 raft_consensus.cc:2804] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:23:55.032729 25079 raft_consensus.cc:697] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:23:55.033692 25079 consensus_queue.cc:237] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:55.037163 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:55.037685 16501 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:23:55.037967 17682 raft_consensus.cc:2393] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 17: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:23:55.038853 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:54.986505 24964 raft_consensus.cc:697] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 20 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:23:54.931453 25053 raft_consensus.cc:2749] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:23:55.016378 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:55.055227 18361 raft_consensus.cc:3060] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:54.974575 24834 raft_consensus.cc:2804] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:23:54.971858 25076 raft_consensus.cc:2804] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:23:55.008127 24946 raft_consensus.cc:2749] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:23:55.082906 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:55.027442 24952 raft_consensus.cc:3055] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:23:55.057113 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:55.042604 25002 raft_consensus.cc:740] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:23:55.053416 17801 leader_election.cc:400] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 15 election: Vote denied by peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651) with higher term. Message: Invalid argument: T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 15. Current term is 16.
I20260709 12:23:55.102551 17801 leader_election.cc:403] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 15 election: Cancelling election due to peer responding with higher term
W20260709 12:23:55.098484 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 79384us
W20260709 12:23:55.103106 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 58725.3us
I20260709 12:23:54.978307 24998 consensus_queue.cc:1059] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 15.12, Next index: 13, Last known committed idx: 8, Time since last communication: 0.063s
I20260709 12:23:55.105849 18366 raft_consensus.cc:1275] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [term 19 LEARNER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 19 index: 11. Preceding OpId from leader: term: 19 index: 12. (index mismatch)
W20260709 12:23:55.112946 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 134984us
I20260709 12:23:55.032147 25006 raft_consensus.cc:3060] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:23:55.129042 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
W20260709 12:23:55.161296 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:55.174852 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:55.177789 17687 raft_consensus.cc:3060] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:55.177822 17688 raft_consensus.cc:3060] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:55.180718 17727 leader_election.cc:304] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
W20260709 12:23:55.183492 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 350379us
W20260709 12:23:55.172399 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 68617.2us
I20260709 12:23:55.104225 24894 raft_consensus.cc:2749] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:23:55.101732 24952 raft_consensus.cc:740] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:23:55.105926 25080 raft_consensus.cc:3060] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:55.008766 25050 raft_consensus.cc:515] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:55.101277 24834 raft_consensus.cc:697] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 19 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
W20260709 12:23:55.187145 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 237156us
I20260709 12:23:55.022666 25019 raft_consensus.cc:515] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:55.108570 24981 leader_election.cc:290] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 16 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:55.136196 25076 raft_consensus.cc:697] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 21 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:23:55.189350 17727 leader_election.cc:304] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:55.190224 17654 leader_election.cc:304] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:23:55.057405 24392 consensus_queue.cc:1059] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 18.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.126s
I20260709 12:23:55.190307 17802 leader_election.cc:304] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:55.195770 17759 raft_consensus.cc:2468] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 20.
I20260709 12:23:55.057099 24790 consensus_queue.cc:1048] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.126s
I20260709 12:23:55.191581 17755 consensus_queue.cc:237] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 19.11, Last appended by leader: 10, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:23:55.197255 17757 raft_consensus.cc:1240] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 21. Current term is 22. Ops: []
I20260709 12:23:55.197994 18333 leader_election.cc:304] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:55.040156 25075 consensus_queue.cc:1048] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.172s
W20260709 12:23:55.193033 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:55.201376 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:55.204933 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 24 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:55.205868 17836 raft_consensus.cc:1240] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 16. Current term is 17. Ops: []
W20260709 12:23:55.208240 17652 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:55.210489 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 74661.5us
I20260709 12:23:55.210783 17829 raft_consensus.cc:1240] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 21. Current term is 22. Ops: [21.15-21.15]
I20260709 12:23:55.235637 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:55.237534 17763 raft_consensus.cc:2468] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 23.
I20260709 12:23:55.237922 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:55.239007 17829 raft_consensus.cc:3060] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Advancing to term 20
W20260709 12:23:55.240782 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 51518.4us
I20260709 12:23:55.247251 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 18 candidate_status { last_received { term: 16 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:55.247856 17838 raft_consensus.cc:3060] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:55.248399 17762 raft_consensus.cc:2393] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 21: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:23:55.237406 17801 leader_election.cc:304] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:55.196666 25053 raft_consensus.cc:2804] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:23:55.192159 24983 raft_consensus.cc:3055] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 15 LEADER]: Stepping down as leader of term 15
W20260709 12:23:55.255838 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 65660.9us
I20260709 12:23:55.185083 24650 consensus_queue.cc:1048] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.032s
I20260709 12:23:55.263684 17832 raft_consensus.cc:1275] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 16 index: 14. Preceding OpId from leader: term: 18 index: 15. (index mismatch)
W20260709 12:23:55.270190 17801 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:55.187723 24767 consensus_queue.cc:1048] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.059s
I20260709 12:23:55.253080 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:55.186151 25077 consensus_queue.cc:1048] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.067s
I20260709 12:23:55.214745 24989 leader_election.cc:290] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 19 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:55.188530 25095 consensus_queue.cc:1048] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.094s
I20260709 12:23:55.217283 24721 consensus_queue.cc:1059] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 18.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.069s
I20260709 12:23:55.276381 17829 raft_consensus.cc:2468] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 20.
I20260709 12:23:55.287628 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:23:55.253955 25083 log.cc:927] Time spent T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.226s	user 0.001s	sys 0.001s
I20260709 12:23:55.373777 17830 raft_consensus.cc:2468] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 16.
I20260709 12:23:55.315063 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:55.309954 17833 raft_consensus.cc:1275] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 19 index: 11. Preceding OpId from leader: term: 19 index: 12. (index mismatch)
I20260709 12:23:55.374037 17831 raft_consensus.cc:3060] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:55.277068 17760 raft_consensus.cc:1275] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 12 index: 8. Preceding OpId from leader: term: 14 index: 9. (index mismatch)
I20260709 12:23:55.336790 17801 leader_election.cc:304] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:23:55.387200 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 50525.4us
W20260709 12:23:55.308682 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 123717us
I20260709 12:23:55.389726 17802 leader_election.cc:304] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:55.279973 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:23:55.290071 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 92163.4us
W20260709 12:23:55.313687 25090 log.cc:927] Time spent T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.128s	user 0.002s	sys 0.000s
I20260709 12:23:55.249507 24819 raft_consensus.cc:2749] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
W20260709 12:23:55.405314 17654 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:55.310498 25094 log.cc:927] Time spent T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.206s	user 0.001s	sys 0.000s
I20260709 12:23:55.259650 24983 raft_consensus.cc:740] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 15 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:23:55.264669 25082 raft_consensus.cc:2804] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:23:55.260401 25053 raft_consensus.cc:493] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:55.265024 24981 raft_consensus.cc:2749] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:23:55.276886 24801 raft_consensus.cc:2804] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:23:55.324328 25019 leader_election.cc:290] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:55.327824 24894 raft_consensus.cc:493] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:55.326402 24813 leader_election.cc:290] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:55.424897 17728 leader_election.cc:304] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:55.428666 18374 raft_consensus.cc:1275] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 18 index: 13. Preceding OpId from leader: term: 21 index: 15. (index mismatch)
I20260709 12:23:55.428100 17837 raft_consensus.cc:2468] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 21.
I20260709 12:23:55.453706 17838 raft_consensus.cc:2468] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 18.
I20260709 12:23:55.456451 17802 leader_election.cc:304] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:55.468976 17764 raft_consensus.cc:1275] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 19 index: 11. Preceding OpId from leader: term: 19 index: 12. (index mismatch)
I20260709 12:23:55.416201 24650 raft_consensus.cc:2804] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Leader election won for term 20
W20260709 12:23:55.470474 17727 leader_election.cc:336] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 16 pre-election: RPC error from VoteRequest() call to peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837): Timed out: RequestConsensusVote RPC to 127.16.29.67:36837 timed out after 1.503s (SENT)
I20260709 12:23:55.419752 24992 raft_consensus.cc:2749] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
W20260709 12:23:55.424214 24677 consensus_meta.cc:287] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.638s	user 0.006s	sys 0.022s
I20260709 12:23:55.253701 24964 consensus_queue.cc:237] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:55.416504 24657 raft_consensus.cc:493] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:55.473907 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:55.452577 25053 raft_consensus.cc:3060] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:55.277374 25002 consensus_queue.cc:260] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 21.11, Last appended by leader: 11, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:55.301419 24952 consensus_queue.cc:260] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 24.18, Last appended by leader: 18, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:55.411376 25089 consensus_queue.cc:1048] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.070s
I20260709 12:23:55.324781 25076 consensus_queue.cc:237] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 19.16, Last appended by leader: 16, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:55.473629 24819 raft_consensus.cc:493] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:55.491043 17802 leader_election.cc:304] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:55.317602 24834 consensus_queue.cc:237] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.13, Last appended by leader: 13, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:55.518703 17761 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38776 (request call id 2774) took 1059 ms. Trace:
I20260709 12:23:55.481510 24801 raft_consensus.cc:697] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 22 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:23:55.530889 17727 leader_election.cc:304] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:55.498171 24650 raft_consensus.cc:697] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 20 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:23:55.494910 24981 raft_consensus.cc:493] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:23:55.511044 24986 raft_consensus.cc:2749] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
W20260709 12:23:55.536123 25101 log.cc:927] Time spent T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.054s	user 0.001s	sys 0.000s
I20260709 12:23:55.536593 25109 raft_consensus.cc:2749] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
W20260709 12:23:55.536931 25093 log.cc:927] Time spent T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.056s	user 0.000s	sys 0.000s
W20260709 12:23:55.534778 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 113892us
I20260709 12:23:55.519155 17761 rpcz_store.cc:276] 0709 12:23:54.455369 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:54.455500 (+   131us) service_pool.cc:224] Handling call
0709 12:23:54.472262 (+ 16762us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:54.547713 (+ 75451us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:54.547760 (+    47us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:54.548370 (+   610us) log.cc:844] Serialized 26 byte log entry
0709 12:23:54.753994 (+205624us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:23:54.754035 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:54.754098 (+    63us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:55.514438 (+760340us) raft_consensus.cc:1685] finished
0709 12:23:55.514459 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:55.514605 (+   146us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":17533,"raft.run_cpu_time_us":64,"raft.run_wall_time_us":64,"thread_start_us":205189,"threads_started":1,"wal-append.queue_time_us":379575}
I20260709 12:23:55.513228 25079 raft_consensus.cc:2749] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:23:55.486866 24894 raft_consensus.cc:515] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:55.485788 25019 consensus_queue.cc:1059] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 21.13, Next index: 14, Last known committed idx: 11, Time since last communication: 0.040s
I20260709 12:23:55.469465 24642 raft_consensus.cc:515] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:55.587591 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 51021.7us
I20260709 12:23:55.608372 17727 leader_election.cc:304] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:55.593812 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:55.586205 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:55.497279 24677 raft_consensus.cc:515] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:55.596529 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 181447us
I20260709 12:23:55.534680 24813 raft_consensus.cc:2749] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:23:55.525213 25077 raft_consensus.cc:493] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:55.596577 18333 leader_election.cc:400] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 17 pre-election: Vote denied by peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651) with higher term. Message: Invalid argument: T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 17. Current term is 18.
I20260709 12:23:55.658588 18333 leader_election.cc:403] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 17 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:23:55.585016 24392 raft_consensus.cc:493] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:55.611936 17729 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:55.589775 25087 log.cc:927] Time spent T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.072s	user 0.000s	sys 0.000s
I20260709 12:23:55.532881 24819 raft_consensus.cc:515] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:55.583870 24989 raft_consensus.cc:493] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:55.511803 24657 raft_consensus.cc:515] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:55.498106 24983 consensus_queue.cc:260] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.12, Last appended by leader: 12, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:55.584746 25050 leader_election.cc:290] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:23:55.697774 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 272970us
W20260709 12:23:55.700192 17652 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:55.703573 18361 consensus_meta.cc:287] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.643s	user 0.011s	sys 0.001s
I20260709 12:23:55.704088 18365 raft_consensus.cc:1240] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 17. Current term is 18. Ops: []
W20260709 12:23:55.696516 17688 consensus_meta.cc:287] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.514s	user 0.007s	sys 0.004s
I20260709 12:23:55.706166 17688 raft_consensus.cc:2468] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 21.
I20260709 12:23:55.708663 18361 raft_consensus.cc:2468] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 17.
I20260709 12:23:55.710325 17654 leader_election.cc:304] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:23:55.591077 24981 raft_consensus.cc:515] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:55.725852 17760 raft_consensus.cc:2468] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 14.
I20260709 12:23:55.709898 25082 raft_consensus.cc:697] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 18 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
W20260709 12:23:55.734675 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:55.742017 25072 raft_consensus.cc:2749] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260709 12:23:55.751384 17802 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:55.766889 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:55.769421 17759 raft_consensus.cc:1275] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 21 index: 11. Preceding OpId from leader: term: 22 index: 12. (index mismatch)
I20260709 12:23:55.771716 17755 raft_consensus.cc:1275] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 20 index: 12. Preceding OpId from leader: term: 22 index: 13. (index mismatch)
I20260709 12:23:55.778885 24967 raft_consensus.cc:3055] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:23:55.782922 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 19 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:55.783896 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:55.844048 17685 raft_consensus.cc:2368] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 16. Current term is 17.
I20260709 12:23:55.785117 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:55.785635 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 17 candidate_status { last_received { term: 16 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:55.786990 24721 raft_consensus.cc:3055] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:23:55.793805 24987 raft_consensus.cc:3055] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:23:55.787130 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 20 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:23:55.799101 25091 log.cc:927] Time spent T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.501s	user 0.001s	sys 0.001s
I20260709 12:23:55.801031 25075 raft_consensus.cc:2749] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Leader pre-election lost for term 17. Reason: Vote denied by peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651) with higher term. Message: Invalid argument: T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 17. Current term is 18.
W20260709 12:23:55.804963 25092 log.cc:927] Time spent T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.270s	user 0.002s	sys 0.000s
I20260709 12:23:55.854059 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
W20260709 12:23:55.869177 18334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:55.856850 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:55.871057 17838 raft_consensus.cc:2393] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 17: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:23:55.857905 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:55.859083 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:55.863809 17758 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38776 (request call id 2772) took 1443 ms. Trace:
W20260709 12:23:55.865098 17687 consensus_meta.cc:287] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.681s	user 0.009s	sys 0.001s
W20260709 12:23:55.833811 24962 raft_consensus.cc:670] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da: failed to trigger leader election: Illegal state: only voting members can start elections: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:23:55.873515 17687 raft_consensus.cc:2468] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 19.
I20260709 12:23:55.872550 17758 rpcz_store.cc:276] 0709 12:23:54.420764 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:54.439370 (+ 18606us) service_pool.cc:224] Handling call
0709 12:23:54.439675 (+   305us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:54.760061 (+320386us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:54.760109 (+    48us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:54.794244 (+ 34135us) log.cc:844] Serialized 26 byte log entry
0709 12:23:55.138078 (+343834us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:23:55.138136 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:55.138232 (+    96us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:55.863509 (+725277us) raft_consensus.cc:1685] finished
0709 12:23:55.863552 (+    43us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:55.863735 (+   183us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":15380,"raft.queue_time_us":107645,"raft.run_cpu_time_us":76,"raft.run_wall_time_us":76,"thread_start_us":612570,"threads_started":2,"wal-append.queue_time_us":576376}
I20260709 12:23:55.692801 24392 raft_consensus.cc:515] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:55.877853 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 175475us
W20260709 12:23:55.880950 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 272672us
I20260709 12:23:55.879683 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:55.884605 17802 leader_election.cc:304] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:55.705137 24818 consensus_queue.cc:1048] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.161s
I20260709 12:23:55.697022 24989 raft_consensus.cc:515] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:55.851939 24983 raft_consensus.cc:3060] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:55.691171 25077 raft_consensus.cc:515] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:55.887558 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 61362.8us
I20260709 12:23:55.873414 24960 raft_consensus.cc:2749] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:23:55.891058 17802 leader_election.cc:304] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:55.842765 24998 raft_consensus.cc:493] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:55.862440 24957 raft_consensus.cc:493] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:55.843030 24967 raft_consensus.cc:740] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:23:55.851737 24987 raft_consensus.cc:740] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 18 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:23:55.851507 24721 raft_consensus.cc:740] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 21 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:23:55.882937 17833 raft_consensus.cc:2468] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 15.
I20260709 12:23:55.868193 24946 raft_consensus.cc:493] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
W20260709 12:23:55.890407 17729 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:55.886919 24767 raft_consensus.cc:2804] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Leader election won for term 17
W20260709 12:23:55.921994 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 45153.4us
W20260709 12:23:55.931432 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 334962us
I20260709 12:23:55.933070 18333 leader_election.cc:400] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 20 election: Vote denied by peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651) with higher term. Message: Invalid argument: T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 20. Current term is 21.
I20260709 12:23:55.933452 18333 leader_election.cc:403] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 20 election: Cancelling election due to peer responding with higher term
I20260709 12:23:55.934448 18334 leader_election.cc:304] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:23:55.939575 17682 raft_consensus.cc:1240] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 19. Current term is 20. Ops: [19.12-19.12]
I20260709 12:23:55.942088 24952 raft_consensus.cc:3060] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Advancing to term 25
W20260709 12:23:55.943348 25067 consensus_meta.cc:287] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.945s	user 0.010s	sys 0.006s
I20260709 12:23:55.842818 24650 consensus_queue.cc:237] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.13, Last appended by leader: 13, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:55.911898 24962 raft_consensus.cc:493] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:55.916347 24992 raft_consensus.cc:493] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
W20260709 12:23:55.953893 25115 log.cc:927] Time spent T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.241s	user 0.001s	sys 0.000s
I20260709 12:23:55.956965 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:55.967746 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
W20260709 12:23:55.971352 17831 consensus_meta.cc:287] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.585s	user 0.007s	sys 0.000s
I20260709 12:23:55.966001 18334 leader_election.cc:304] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:23:55.912842 25019 raft_consensus.cc:2749] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:23:55.893574 24986 consensus_queue.cc:1048] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.038s
I20260709 12:23:55.982095 17831 raft_consensus.cc:2468] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 17.
I20260709 12:23:55.897607 24813 consensus_queue.cc:1048] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.068s
W20260709 12:23:55.959591 25003 consensus_meta.cc:287] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 1.083s	user 0.011s	sys 0.008s
I20260709 12:23:56.033285 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:56.034148 17683 raft_consensus.cc:3060] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:56.034778 17756 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38776 (request call id 2773) took 1593 ms. Trace:
I20260709 12:23:55.922950 24974 raft_consensus.cc:493] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:23:55.863085 24801 consensus_queue.cc:237] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 19.11, Last appended by leader: 11, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:23:55.961122 25078 consensus_meta.cc:287] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 1.000s	user 0.011s	sys 0.000s
W20260709 12:23:55.948652 24927 consensus_meta.cc:287] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.895s	user 0.012s	sys 0.000s
W20260709 12:23:56.044291 17654 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:55.952795 25114 log.cc:927] Time spent T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.089s	user 0.000s	sys 0.001s
I20260709 12:23:56.035172 17756 rpcz_store.cc:276] 0709 12:23:54.441529 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:54.441667 (+   138us) service_pool.cc:224] Handling call
0709 12:23:54.455237 (+ 13570us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:54.706062 (+250825us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:54.706109 (+    47us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:54.726018 (+ 19909us) log.cc:844] Serialized 26 byte log entry
0709 12:23:55.085315 (+359297us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:23:55.085372 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:55.085469 (+    97us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:55.804118 (+718649us) spinlock_profiling.cc:243] Waited 4.55 ms on lock 0x7b4400022b4c. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b47459c85 00007f7b4745a6a2 00007f7b4a1cc1db 00007f7b4a1c7aaf 00007f7b4a1bb86e 00007f7b4a1babb6 00007f7b4a1ba7d8 00007f7b4a1c4221 00007f7b4a1c41d9 00007f7b4a1c4169
0709 12:23:56.002309 (+198191us) raft_consensus.cc:1685] finished
0709 12:23:56.002338 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:56.034713 (+ 32375us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":13060,"raft.queue_time_us":101156,"raft.run_cpu_time_us":62,"raft.run_wall_time_us":62,"spinlock_wait_cycles":10461568,"thread_start_us":510917,"threads_started":2,"wal-append.queue_time_us":707781}
I20260709 12:23:55.950033 24894 leader_election.cc:290] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:55.955336 24657 leader_election.cc:290] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:55.924656 24957 raft_consensus.cc:515] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:55.963557 24642 leader_election.cc:290] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 18 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:23:56.002940 25056 consensus_meta.cc:287] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 1.147s	user 0.006s	sys 0.007s
I20260709 12:23:55.960489 25077 leader_election.cc:290] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:55.962345 24392 leader_election.cc:290] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:23:56.102550 17654 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:55.964588 24677 leader_election.cc:290] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 14 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:55.943845 25067 raft_consensus.cc:515] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:56.117596 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 151648us
W20260709 12:23:56.117630 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 151612us
I20260709 12:23:56.122682 17802 leader_election.cc:400] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 16 election: Vote denied by peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071) with higher term. Message: Invalid argument: T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 16. Current term is 17.
I20260709 12:23:55.923698 24998 raft_consensus.cc:515] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:55.964125 24992 raft_consensus.cc:515] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:55.960623 24946 raft_consensus.cc:515] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:56.188098 17802 leader_election.cc:403] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 16 election: Cancelling election due to peer responding with higher term
I20260709 12:23:56.099542 24767 raft_consensus.cc:697] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 17 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
W20260709 12:23:56.204797 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 83345.7us
I20260709 12:23:55.955976 25113 consensus_queue.cc:1048] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 10, Time since last communication: 0.204s
W20260709 12:23:56.080406 25006 consensus_meta.cc:287] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.906s	user 0.013s	sys 0.000s
I20260709 12:23:56.001220 25088 consensus_queue.cc:1048] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.255s
I20260709 12:23:55.965098 24721 consensus_queue.cc:260] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 21.13, Last appended by leader: 13, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:55.954926 24962 raft_consensus.cc:515] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:56.001067 25116 consensus_queue.cc:1059] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 17.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.261s
W20260709 12:23:56.248371 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 56071.6us
I20260709 12:23:56.111115 25078 raft_consensus.cc:2749] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260709 12:23:56.248371 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 55118.9us
W20260709 12:23:56.231029 17802 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:56.175319 25056 raft_consensus.cc:2749] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:23:56.186635 24657 raft_consensus.cc:493] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:23:56.098498 24974 raft_consensus.cc:515] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:56.128261 24927 raft_consensus.cc:515] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:23:56.282907 17762 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.616s	user 0.012s	sys 0.018s
W20260709 12:23:56.285470 17762 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.619s	user 0.012s	sys 0.019s
I20260709 12:23:56.292250 18333 leader_election.cc:304] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
W20260709 12:23:56.297108 25107 log.cc:927] Time spent T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.286s	user 0.001s	sys 0.001s
W20260709 12:23:56.298537 25105 log.cc:927] Time spent T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.403s	user 0.000s	sys 0.003s
I20260709 12:23:56.299008 25067 leader_election.cc:290] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 17 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:56.299420 25127 raft_consensus.cc:2749] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 12:23:56.301013 24957 leader_election.cc:290] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:23:56.303277 25110 log.cc:927] Time spent T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.404s	user 0.002s	sys 0.004s
I20260709 12:23:56.306741 25144 raft_consensus.cc:2749] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Leader election lost for term 16. Reason: Vote denied by peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071) with higher term. Message: Invalid argument: T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 16. Current term is 17.
I20260709 12:23:56.293916 17757 raft_consensus.cc:2468] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 21.
I20260709 12:23:56.312433 17680 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8"
dest_uuid: "0da169ff82674dc1947055ce43de68a3"
mode: GRACEFUL
new_leader_uuid: "e063541542a94969898868bf6eb5ce1b"
 from {username='slave'} at 127.0.0.1:54540
I20260709 12:23:56.313393 17680 raft_consensus.cc:606] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 20 LEADER]: Received request to transfer leadership to TS e063541542a94969898868bf6eb5ce1b
I20260709 12:23:56.315156 18368 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 2137) took 1304 ms. Trace:
I20260709 12:23:56.317978 24927 leader_election.cc:290] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 21 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:56.319032 17681 raft_consensus.cc:1240] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 20. Current term is 21. Ops: []
I20260709 12:23:56.321448 17681 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37396 (request call id 2851) took 1313 ms. Trace:
I20260709 12:23:56.322283 24998 leader_election.cc:290] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:55.982733 25082 consensus_queue.cc:237] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 16.15, Last appended by leader: 15, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:56.315630 18368 rpcz_store.cc:276] 0709 12:23:55.011261 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:55.011555 (+   294us) service_pool.cc:224] Handling call
0709 12:23:55.031784 (+ 20229us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:55.179654 (+147870us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 12:23:55.179707 (+    53us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:55.180321 (+   614us) log.cc:844] Serialized 26 byte log entry
0709 12:23:55.507759 (+327438us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:23:55.507800 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:55.507868 (+    68us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:56.314955 (+807087us) raft_consensus.cc:1685] finished
0709 12:23:56.314975 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:56.315102 (+   127us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":23418,"raft.run_cpu_time_us":149,"raft.run_wall_time_us":9034,"thread_start_us":260494,"threads_started":1,"wal-append.queue_time_us":421708}
I20260709 12:23:56.327366 24767 consensus_queue.cc:237] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.12, Last appended by leader: 12, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:56.336318 25149 raft_consensus.cc:493] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:56.336748 24981 leader_election.cc:290] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:56.336975 25149 raft_consensus.cc:515] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:56.321892 17681 rpcz_store.cc:276] 0709 12:23:55.008378 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:55.022140 (+ 13762us) service_pool.cc:224] Handling call
0709 12:23:55.022410 (+   270us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:56.319576 (+1297166us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:56.321128 (+  1552us) spinlock_profiling.cc:243] Waited 1.3 s on lock 0x7b54001aa754. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:23:56.321344 (+   216us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2981434752}
I20260709 12:23:56.264670 25006 raft_consensus.cc:493] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:56.345534 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 53393.1us
I20260709 12:23:56.264822 24392 raft_consensus.cc:2804] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:23:56.353483 18333 leader_election.cc:304] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:23:56.272238 24677 raft_consensus.cc:2804] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:23:56.143126 24987 consensus_queue.cc:260] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 18.15, Last appended by leader: 15, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:56.339696 25111 log.cc:927] Time spent T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.653s	user 0.009s	sys 0.000s
I20260709 12:23:56.182137 24967 consensus_queue.cc:260] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 19.12, Last appended by leader: 12, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:56.264492 24642 consensus_queue.cc:1048] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.099s
I20260709 12:23:56.287645 24657 raft_consensus.cc:515] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:23:56.585974 18363 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.509s	user 0.044s	sys 0.129s
W20260709 12:23:56.568946 17727 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.666s	user 0.013s	sys 0.015s
W20260709 12:23:56.610743 17727 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.708s	user 0.014s	sys 0.016s
W20260709 12:23:56.591375 17683 consensus_meta.cc:287] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.535s	user 0.011s	sys 0.000s
I20260709 12:23:56.612387 17683 raft_consensus.cc:2468] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 14.
W20260709 12:23:56.584089 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 452697us
W20260709 12:23:56.615129 17803 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:56.615705 17682 raft_consensus.cc:1240] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 13. Current term is 14. Ops: []
W20260709 12:23:56.617352 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 715033us
W20260709 12:23:56.404277 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:56.396052 17652 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.653s	user 0.035s	sys 0.058s
W20260709 12:23:56.622941 17652 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.910s	user 0.044s	sys 0.069s
W20260709 12:23:56.623544 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 910936us
I20260709 12:23:56.626209 17652 leader_election.cc:304] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:56.363706 24927 raft_consensus.cc:493] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:56.471855 25125 log.cc:927] Time spent T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.268s	user 0.000s	sys 0.000s
W20260709 12:23:56.635653 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 289894us
I20260709 12:23:56.635987 17652 leader_election.cc:304] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:23:56.486577 25080 consensus_meta.cc:287] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 1.272s	user 0.013s	sys 0.000s
W20260709 12:23:56.475737 24952 consensus_meta.cc:287] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.533s	user 0.003s	sys 0.008s
W20260709 12:23:56.482105 24983 consensus_meta.cc:287] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.573s	user 0.008s	sys 0.000s
W20260709 12:23:56.472927 25117 log.cc:927] Time spent T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.559s	user 0.000s	sys 0.001s
I20260709 12:23:56.379767 25006 raft_consensus.cc:515] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:56.642477 17652 leader_election.cc:304] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:56.470847 25140 raft_consensus.cc:2749] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Leader election lost for term 20. Reason: Vote denied by peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651) with higher term. Message: Invalid argument: T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 20. Current term is 21.
I20260709 12:23:56.473656 25002 raft_consensus.cc:3060] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:23:56.609072 24960 raft_consensus.cc:2804] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:23:56.472394 24891 raft_consensus.cc:493] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:56.645156 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 24 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:56.645967 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:56.621158 24642 consensus_queue.cc:1243] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer e2909cff1da44ad387f96a8ae3aa5afb log is divergent from this leader: its last log entry 20.15 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 14
I20260709 12:23:56.558482 25146 raft_consensus.cc:493] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:56.597730 24989 leader_election.cc:290] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:56.477439 25149 leader_election.cc:290] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:56.650195 24816 raft_consensus.cc:2749] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:23:56.421113 24819 leader_election.cc:290] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:56.557964 24974 leader_election.cc:290] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:56.621960 24964 raft_consensus.cc:2804] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:23:56.656148 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:23:56.639719 25112 log.cc:927] Time spent T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.502s	user 0.000s	sys 0.002s
I20260709 12:23:56.640313 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:56.767609 17763 raft_consensus.cc:2468] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 25.
W20260709 12:23:56.722925 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 80526.3us
I20260709 12:23:56.723416 18334 leader_election.cc:304] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:23:56.644948 24981 raft_consensus.cc:493] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:56.777788 17652 leader_election.cc:304] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:56.778012 24981 raft_consensus.cc:515] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:56.779829 17652 leader_election.cc:304] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:56.780381 24981 leader_election.cc:290] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:56.644793 24998 raft_consensus.cc:493] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:23:56.780782 24767 consensus_queue.cc:1048] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 12:23:56.781250 24998 raft_consensus.cc:515] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:23:56.782758 25154 log.cc:927] Time spent T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.053s	user 0.000s	sys 0.000s
I20260709 12:23:56.783011 25003 consensus_queue.cc:1048] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 12:23:56.740000 24891 raft_consensus.cc:515] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:56.645951 24927 raft_consensus.cc:515] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:56.477669 25132 consensus_queue.cc:1048] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.296s
W20260709 12:23:56.728384 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 374998us
I20260709 12:23:56.644624 25095 consensus_queue.cc:1048] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 12, Time since last communication: 0.096s
I20260709 12:23:56.548557 24677 raft_consensus.cc:697] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 19 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:23:56.544297 25153 consensus_queue.cc:1048] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.167s
I20260709 12:23:56.652719 24392 raft_consensus.cc:697] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 21 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:23:56.615790 25142 consensus_queue.cc:1059] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 18.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.107s
I20260709 12:23:56.714102 25080 raft_consensus.cc:2749] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Leader election lost for term 15. Reason: Vote denied by peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651) with higher term. Message: Invalid argument: T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 15. Current term is 16.
I20260709 12:23:56.746819 25124 raft_consensus.cc:2804] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:23:56.747130 25146 raft_consensus.cc:515] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:56.765744 17837 raft_consensus.cc:2468] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 16.
W20260709 12:23:56.739964 25125 log.cc:927] Time spent T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.086s	user 0.000s	sys 0.001s
I20260709 12:23:56.769215 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:56.770650 25133 raft_consensus.cc:493] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:23:56.771440 25119 raft_consensus.cc:2804] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:23:56.747418 25164 raft_consensus.cc:493] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:23:56.773602 17683 raft_consensus.cc:2468] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 19.
W20260709 12:23:56.671836 17654 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:56.727005 24960 raft_consensus.cc:493] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:56.796974 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:56.777885 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 15 candidate_status { last_received { term: 14 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:23:56.794549 17729 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:56.786203 17833 raft_consensus.cc:1275] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 12 index: 8. Preceding OpId from leader: term: 14 index: 9. (index mismatch)
W20260709 12:23:56.792536 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 902132us
I20260709 12:23:56.794080 17834 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35040 (request call id 3089) took 1588 ms. Trace:
I20260709 12:23:56.778673 25126 raft_consensus.cc:2749] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
W20260709 12:23:56.791736 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 70799.2us
I20260709 12:23:56.798832 17727 leader_election.cc:304] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
W20260709 12:23:56.837947 17801 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.572s	user 0.054s	sys 0.085s
I20260709 12:23:56.869311 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 19 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:23:57.018455 17803 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:57.085943 17836 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35040 (request call id 3094) took 1828 ms. Trace:
I20260709 12:23:56.877756 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 22 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:23:56.851943 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:57.121870 18366 raft_consensus.cc:1240] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 17. Current term is 18. Ops: []
I20260709 12:23:56.873183 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:57.122890 18374 raft_consensus.cc:3060] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:56.906317 17680 raft_consensus.cc:1275] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 21 index: 13. Preceding OpId from leader: term: 22 index: 13. (term mismatch)
I20260709 12:23:57.125137 17680 pending_rounds.cc:85] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3: Aborting all ops after (but not including) 12
I20260709 12:23:57.125485 17680 pending_rounds.cc:107] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3: Aborting uncommitted NO_OP operation due to leader change: 21.13
I20260709 12:23:57.125821 17680 raft_consensus.cc:2889] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:56.957654 25003 consensus_queue.cc:1059] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 16.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.169s
I20260709 12:23:56.797233 25081 raft_consensus.cc:493] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:23:57.127019 25081 raft_consensus.cc:515] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:56.919737 17829 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35042 (request call id 2791) took 1064 ms. Trace:
I20260709 12:23:56.784157 25113 consensus_queue.cc:1048] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.345s
I20260709 12:23:57.129503 17682 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37402 (request call id 3253) took 1121 ms. Trace:
I20260709 12:23:56.798755 25077 consensus_queue.cc:1048] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.331s
I20260709 12:23:57.129891 17682 rpcz_store.cc:276] 0709 12:23:56.023259 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:56.036651 (+ 13392us) service_pool.cc:224] Handling call
0709 12:23:56.612966 (+576315us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:56.616315 (+  3349us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:57.129323 (+513008us) spinlock_profiling.cc:243] Waited 576 ms on lock 0x7b54003471d0. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a61f8ad 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:23:57.129450 (+   127us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1324270464}
W20260709 12:23:56.989869 17802 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:57.131210 17689 raft_consensus.cc:1275] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 21 index: 11. Preceding OpId from leader: term: 22 index: 12. (index mismatch)
I20260709 12:23:57.121666 17685 raft_consensus.cc:2468] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 14.
I20260709 12:23:56.793812 25131 raft_consensus.cc:493] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:56.879648 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:57.129163 25081 leader_election.cc:290] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:57.135390 17831 raft_consensus.cc:3060] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:56.782613 24767 raft_consensus.cc:2804] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:23:56.791999 24974 raft_consensus.cc:2804] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Leader election won for term 14
W20260709 12:23:57.120350 17801 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.855s	user 0.061s	sys 0.092s
I20260709 12:23:56.773336 24989 raft_consensus.cc:493] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:23:56.791252 25140 raft_consensus.cc:493] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
W20260709 12:23:57.142409 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 343642us
I20260709 12:23:56.792920 24962 leader_election.cc:290] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:23:56.793970 25053 consensus_meta.cc:287] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 1.269s	user 0.010s	sys 0.002s
I20260709 12:23:57.119031 17834 rpcz_store.cc:276] 0709 12:23:55.205177 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:55.205304 (+   127us) service_pool.cc:224] Handling call
0709 12:23:55.235111 (+ 29807us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:55.262609 (+ 27498us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:55.262650 (+    41us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:55.263173 (+   523us) log.cc:844] Serialized 26 byte log entry
0709 12:23:55.421481 (+158308us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:23:55.421560 (+    79us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:55.421696 (+   136us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:56.793864 (+1372168us) raft_consensus.cc:1685] finished
0709 12:23:56.793886 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:56.794024 (+   138us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":9742,"raft.queue_time_us":24137,"raft.run_cpu_time_us":148,"raft.run_wall_time_us":6097,"thread_start_us":118473,"threads_started":1,"wal-append.queue_time_us":405315}
I20260709 12:23:56.790943 25149 raft_consensus.cc:493] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:23:56.795732 25163 raft_consensus.cc:493] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:57.175300 17832 raft_consensus.cc:1275] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 16 index: 13. (index mismatch)
W20260709 12:23:57.001441 25155 log.cc:927] Time spent T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.077s	user 0.000s	sys 0.001s
I20260709 12:23:56.791092 25055 raft_consensus.cc:493] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:57.128649 17829 rpcz_store.cc:276] 0709 12:23:55.855265 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:55.855392 (+   127us) service_pool.cc:224] Handling call
0709 12:23:55.855776 (+   384us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:55.873905 (+ 18129us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:55.873938 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:55.880302 (+  6364us) log.cc:844] Serialized 26 byte log entry
0709 12:23:56.345973 (+465671us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:23:56.390170 (+ 44197us) log.cc:844] Serialized 14 byte log entry
0709 12:23:56.480136 (+ 89966us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:56.480211 (+    75us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:56.919524 (+439313us) raft_consensus.cc:1685] finished
0709 12:23:56.919546 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:56.919675 (+   129us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":96327,"prepare.queue_time_us":42707,"prepare.run_cpu_time_us":67,"prepare.run_wall_time_us":2655,"raft.queue_time_us":107793,"raft.run_cpu_time_us":289,"raft.run_wall_time_us":19510,"thread_start_us":428444,"threads_started":1,"wal-append.queue_time_us":713938}
I20260709 12:23:57.216445 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:57.217247 17829 raft_consensus.cc:2468] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 21.
I20260709 12:23:57.218500 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:57.219213 17829 raft_consensus.cc:3060] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:57.175498 25081 raft_consensus.cc:493] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:57.189396 17762 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.701s	user 0.007s	sys 0.014s
I20260709 12:23:56.796448 24657 leader_election.cc:290] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:56.783793 24998 leader_election.cc:290] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:56.791832 24946 leader_election.cc:290] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:57.084913 24818 raft_consensus.cc:3055] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:23:56.905444 25082 raft_consensus.cc:493] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:57.120041 17836 rpcz_store.cc:276] 0709 12:23:55.257845 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:55.270013 (+ 12168us) service_pool.cc:224] Handling call
0709 12:23:55.270335 (+   322us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:55.482405 (+212070us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 12:23:55.712137 (+229732us) log.cc:844] Serialized 14 byte log entry
0709 12:23:56.208798 (+496661us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:56.208860 (+    62us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:23:56.208907 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:57.085603 (+876696us) spinlock_profiling.cc:243] Waited 212 ms on lock 0x7b540022ae54. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:23:57.085668 (+    65us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:57.085851 (+   183us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":58620,"prepare.queue_time_us":441632,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":38,"raft.queue_time_us":147288,"raft.run_cpu_time_us":137,"raft.run_wall_time_us":21180,"spinlock_wait_cycles":487109376,"thread_start_us":525363,"threads_started":2,"wal-append.queue_time_us":676511}
I20260709 12:23:56.954069 24721 raft_consensus.cc:3060] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:23:56.918879 25134 raft_consensus.cc:493] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:23:56.953205 24891 leader_election.cc:290] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:57.132937 24723 raft_consensus.cc:493] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:57.172209 24952 consensus_queue.cc:1059] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 20.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.067s
I20260709 12:23:57.015120 25165 raft_consensus.cc:493] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:56.979702 24992 leader_election.cc:290] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:56.970089 25006 leader_election.cc:290] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:57.133544 24927 leader_election.cc:290] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:57.141958 25119 raft_consensus.cc:493] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:23:57.147715 25133 raft_consensus.cc:515] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:57.273635 25119 raft_consensus.cc:3060] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Advancing to term 22
W20260709 12:23:57.206733 17729 consensus_peers.cc:597] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb -> Peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Couldn't send request to peer 334a4e1f59944544b3b3ff36616b57da. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a5732e71b9d54f8786b8c511d69943bf. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:23:57.237380 25056 raft_consensus.cc:493] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
W20260709 12:23:57.276180 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 141808us
I20260709 12:23:57.276520 25056 raft_consensus.cc:515] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:57.239593 25081 raft_consensus.cc:515] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:57.279642 25056 leader_election.cc:290] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:57.279913 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:57.280586 25081 leader_election.cc:290] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:57.281172 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:57.281939 17759 raft_consensus.cc:2468] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 21.
I20260709 12:23:57.285754 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 22 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:57.281365 24964 raft_consensus.cc:493] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:57.251394 25082 raft_consensus.cc:515] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:57.212327 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:57.224635 17834 raft_consensus.cc:1275] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 19 index: 11. Preceding OpId from leader: term: 19 index: 12. (index mismatch)
I20260709 12:23:57.225423 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:57.300262 17838 raft_consensus.cc:3060] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Advancing to term 17
I20260709 12:23:57.145073 25137 raft_consensus.cc:2749] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:23:57.172278 25168 raft_consensus.cc:493] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:57.146736 25002 raft_consensus.cc:493] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:57.159360 24767 raft_consensus.cc:493] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:23:57.233768 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:23:57.235358 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 426146us
I20260709 12:23:57.173620 25124 raft_consensus.cc:697] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 17 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:23:57.324914 17802 leader_election.cc:304] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:23:57.239598 17801 leader_election.cc:304] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:23:57.325873 17802 leader_election.cc:304] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:23:57.206271 25147 raft_consensus.cc:493] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:57.227898 25163 raft_consensus.cc:515] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:57.250976 17760 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa"
dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
mode: GRACEFUL
new_leader_uuid: "334a4e1f59944544b3b3ff36616b57da"
 from {username='slave'} at 127.0.0.1:49786
I20260709 12:23:57.333083 17760 raft_consensus.cc:606] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Received request to transfer leadership to TS 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:57.333519 17760 raft_consensus.cc:614] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:23:57.251996 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:57.148783 25131 raft_consensus.cc:515] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:56.798627 24819 consensus_queue.cc:1059] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 17.11, Next index: 12, Last known committed idx: 10, Time since last communication: 0.361s
I20260709 12:23:57.255193 18374 raft_consensus.cc:2468] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 17.
I20260709 12:23:57.258697 17829 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 18.
I20260709 12:23:57.210021 25121 raft_consensus.cc:493] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:57.213274 24962 raft_consensus.cc:493] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:23:57.232932 24818 raft_consensus.cc:740] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
W20260709 12:23:57.260217 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 116028us
I20260709 12:23:57.254146 25134 raft_consensus.cc:515] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:57.358913 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:57.261484 24723 raft_consensus.cc:515] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:57.247900 25079 raft_consensus.cc:493] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:57.233269 25078 raft_consensus.cc:493] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:57.141273 24960 raft_consensus.cc:3060] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:23:57.322847 24767 raft_consensus.cc:3060] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:23:57.275555 25133 leader_election.cc:290] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:57.292776 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:57.176079 25164 raft_consensus.cc:515] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:57.241226 25067 raft_consensus.cc:493] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
W20260709 12:23:57.406018 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 44690.9us
I20260709 12:23:57.405961 25067 raft_consensus.cc:515] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:23:57.407558 25164 leader_election.cc:290] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:57.208057 25053 raft_consensus.cc:515] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:57.181425 24989 raft_consensus.cc:515] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:57.421680 17830 raft_consensus.cc:2468] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 16.
W20260709 12:23:57.220409 17762 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.732s	user 0.010s	sys 0.020s
I20260709 12:23:57.213964 25149 raft_consensus.cc:515] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:57.308528 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:57.183947 25140 raft_consensus.cc:515] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:57.319723 17683 raft_consensus.cc:2393] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 21: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:23:57.321374 17727 leader_election.cc:304] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:57.293452 24964 raft_consensus.cc:3060] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:57.290854 25006 raft_consensus.cc:3055] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:23:57.236223 25055 raft_consensus.cc:515] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:57.250291 17831 raft_consensus.cc:2468] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 21.
I20260709 12:23:57.330782 24721 raft_consensus.cc:2804] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Leader pre-election won for term 20
W20260709 12:23:57.345273 17763 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38776 (request call id 2791) took 2121 ms (client timeout 2132 ms). Trace:
I20260709 12:23:57.359006 25076 consensus_queue.cc:1048] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.079s
I20260709 12:23:57.349956 17652 leader_election.cc:304] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da
W20260709 12:23:57.357368 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 117819us
I20260709 12:23:57.361666 17838 raft_consensus.cc:2468] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 17.
I20260709 12:23:57.362126 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:23:57.440240 17763 rpcz_store.cc:269] 0709 12:23:55.207170 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:55.314768 (+107598us) service_pool.cc:224] Handling call
0709 12:23:57.328143 (+2013375us) spinlock_profiling.cc:243] Waited 1.34 s on lock 0x7b54003f2854. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:23:57.328255 (+   112us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3087853440}
I20260709 12:23:57.442996 17829 raft_consensus.cc:2468] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 21.
I20260709 12:23:57.238417 24650 consensus_queue.cc:1048] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.051s
I20260709 12:23:57.359594 17834 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 17.
W20260709 12:23:57.446130 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 96279.9us
I20260709 12:23:57.446817 17652 leader_election.cc:304] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:57.291679 25165 raft_consensus.cc:515] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:57.443202 17801 leader_election.cc:304] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:57.450130 17652 leader_election.cc:304] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:57.402578 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:23:57.294586 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 86932.7us
W20260709 12:23:57.408505 25170 log.cc:927] Time spent T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.118s	user 0.000s	sys 0.000s
I20260709 12:23:57.364900 25121 raft_consensus.cc:515] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:57.360687 24992 consensus_queue.cc:1059] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.049s
I20260709 12:23:57.290004 24677 consensus_queue.cc:237] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.14, Last appended by leader: 14, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:57.409482 25124 consensus_queue.cc:237] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.10, Last appended by leader: 10, Current term: 17, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:57.411019 17688 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44898 (request call id 1412) took 1473 ms. Trace:
I20260709 12:23:57.464810 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 20 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:57.465337 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:57.441506 24819 raft_consensus.cc:2749] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:23:57.360749 25179 consensus_queue.cc:1048] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.052s
I20260709 12:23:57.415796 25156 raft_consensus.cc:493] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:57.417294 17837 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35042 (request call id 2788) took 1563 ms. Trace:
I20260709 12:23:57.373199 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 22 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:57.298197 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:57.422842 17762 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38786 (request call id 3542) took 2084 ms. Trace:
I20260709 12:23:57.323061 24974 raft_consensus.cc:697] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 14 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:23:57.458046 24891 raft_consensus.cc:2804] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:23:57.428529 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:23:57.433475 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 112219us
I20260709 12:23:57.360674 25080 raft_consensus.cc:493] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:57.492291 25080 raft_consensus.cc:515] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:57.493037 24974 consensus_queue.cc:237] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:57.495293 25080 leader_election.cc:290] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:57.322907 25002 raft_consensus.cc:515] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:57.347676 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:57.318555 25168 raft_consensus.cc:515] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:57.498322 25002 leader_election.cc:290] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:57.498759 17680 raft_consensus.cc:2468] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 21.
I20260709 12:23:57.495818 25119 raft_consensus.cc:515] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:57.501049 25168 leader_election.cc:290] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:23:57.501768 25169 log.cc:927] Time spent T 00000000000000000000000000000000 P 2193af31cb894ec9b3d65d939a2a6851: Append to log took a long time: real 0.156s	user 0.001s	sys 0.000s
I20260709 12:23:57.502271 25080 raft_consensus.cc:3055] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:23:57.502940 25080 raft_consensus.cc:740] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 13 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:23:57.328195 25137 consensus_queue.cc:1048] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.106s
I20260709 12:23:57.444005 17756 raft_consensus.cc:1240] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 17. Current term is 18. Ops: [17.12-17.12]
W20260709 12:23:57.503793 25178 log.cc:927] Time spent T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.057s	user 0.000s	sys 0.000s
I20260709 12:23:57.444175 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:57.507272 17831 raft_consensus.cc:1275] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 20 index: 13. Preceding OpId from leader: term: 21 index: 14. (index mismatch)
I20260709 12:23:57.510023 17831 raft_consensus.cc:1240] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 19. Current term is 20. Ops: [18.11-19.12]
I20260709 12:23:57.451269 17685 raft_consensus.cc:2468] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 20.
I20260709 12:23:57.453529 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:57.513182 18363 raft_consensus.cc:2368] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 16. Current term is 17.
I20260709 12:23:57.514330 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:57.515331 18363 raft_consensus.cc:2410] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 19 because replica has last-logged OpId of term: 18 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 13.
I20260709 12:23:57.454180 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:57.454629 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 19 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:57.455415 17652 leader_election.cc:304] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:57.491343 18333 leader_election.cc:304] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:57.518970 25002 raft_consensus.cc:2804] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:23:57.519496 25002 raft_consensus.cc:697] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 21 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:23:57.520080 25168 raft_consensus.cc:2804] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:23:57.520514 25168 raft_consensus.cc:493] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:57.521075 25168 raft_consensus.cc:3060] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:23:57.520470 25002 consensus_queue.cc:237] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 20.12, Last appended by leader: 12, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:57.524061 24818 consensus_queue.cc:260] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.11, Last appended by leader: 11, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:57.525168 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 69812.2us
I20260709 12:23:57.525383 24818 raft_consensus.cc:3060] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:57.526381 17803 leader_election.cc:304] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:57.457636 17758 raft_consensus.cc:3060] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:57.460237 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:57.365837 24962 raft_consensus.cc:515] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:57.437877 25006 raft_consensus.cc:740] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:23:57.401211 25078 raft_consensus.cc:515] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:57.300575 24392 consensus_queue.cc:237] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 19.15, Last appended by leader: 15, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:57.467173 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:57.439996 24721 raft_consensus.cc:493] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:23:57.468350 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:57.442512 24952 raft_consensus.cc:493] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
W20260709 12:23:57.546610 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 139515us
I20260709 12:23:57.547063 17687 raft_consensus.cc:1275] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 21 index: 12. Preceding OpId from leader: term: 22 index: 12. (term mismatch)
I20260709 12:23:57.547506 17687 pending_rounds.cc:85] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3: Aborting all ops after (but not including) 11
I20260709 12:23:57.547757 17687 pending_rounds.cc:107] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3: Aborting uncommitted NO_OP operation due to leader change: 21.12
I20260709 12:23:57.548018 17687 raft_consensus.cc:2889] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:57.400748 25079 raft_consensus.cc:515] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:57.465093 17688 rpcz_store.cc:276] 0709 12:23:55.937616 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:55.952256 (+ 14640us) service_pool.cc:224] Handling call
0709 12:23:55.952740 (+   484us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:56.050889 (+ 98149us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:56.050934 (+    45us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:56.051541 (+   607us) log.cc:844] Serialized 26 byte log entry
0709 12:23:56.327035 (+275494us) spinlock_profiling.cc:243] Waited 18.3 ms on lock 0x7b48000030d8. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4f549aa1 00007f7b4f540b37 00007f7b4f5409ca 00007f7b4f63826f 00007f7b4f637f37 00007f7b4f64c55c 00007f7b4f64c508 00007f7b4f64c488 00007f7b4f64c443 00007f7b4f64b6ec 00007f7b4e575a96
0709 12:23:57.270750 (+943715us) spinlock_profiling.cc:243] Waited 1.75 ms on lock 0x7b180033a600. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b47515e07 00007f7b503709a9 00007f7b4753269b 00007f7b4753093f 00007f7b47532c02 00007f7b4a1ba20b 00007f7b4a1bfd23 00007f7b4a1c00d1 00007f7b4a5dba0f
0709 12:23:57.343754 (+ 73004us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:23:57.343815 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:57.343948 (+   133us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:57.410825 (+ 66877us) raft_consensus.cc:1685] finished
0709 12:23:57.410843 (+    18us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:57.410966 (+   123us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":97671,"raft.queue_time_us":123997,"raft.run_cpu_time_us":2597,"raft.run_wall_time_us":226620,"spinlock_wait_cycles":46189696,"thread_start_us":56117,"threads_started":1,"wal-append.queue_time_us":1325756}
I20260709 12:23:57.470765 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:57.353243 25147 raft_consensus.cc:515] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:57.473320 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:57.560016 18368 raft_consensus.cc:2368] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 15. Current term is 16.
I20260709 12:23:57.477483 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:57.546550 24721 raft_consensus.cc:3060] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:57.478173 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:57.562543 17761 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 17.
I20260709 12:23:57.474563 17837 rpcz_store.cc:276] 0709 12:23:55.854239 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:55.857638 (+  3399us) service_pool.cc:224] Handling call
0709 12:23:57.417148 (+1559510us) spinlock_profiling.cc:243] Waited 790 ms on lock 0x7b5400334fd4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:23:57.417246 (+    98us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1817155328}
I20260709 12:23:57.453305 24927 raft_consensus.cc:2804] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:23:57.346153 24998 raft_consensus.cc:2804] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:23:57.474992 17762 rpcz_store.cc:276] 0709 12:23:55.338377 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:55.601361 (+262984us) service_pool.cc:224] Handling call
0709 12:23:55.601771 (+   410us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:56.339069 (+737298us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:56.339124 (+    55us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:56.352351 (+ 13227us) log.cc:844] Serialized 26 byte log entry
0709 12:23:57.422452 (+1070101us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:23:57.422495 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:57.422589 (+    94us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:57.422635 (+    46us) raft_consensus.cc:1685] finished
0709 12:23:57.422650 (+    15us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:57.422781 (+   131us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":28416,"raft.queue_time_us":180806,"raft.run_cpu_time_us":92,"raft.run_wall_time_us":92,"thread_start_us":1553284,"threads_started":2,"wal-append.queue_time_us":757078}
I20260709 12:23:57.563859 17755 raft_consensus.cc:3060] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:57.482714 24891 raft_consensus.cc:493] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:23:57.482489 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:57.486286 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 17 candidate_status { last_received { term: 16 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:57.486756 25149 leader_election.cc:290] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:57.365178 24946 raft_consensus.cc:2804] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:23:57.503180 25119 leader_election.cc:290] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 22 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:57.517024 18374 raft_consensus.cc:2410] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 19 because replica has last-logged OpId of term: 18 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 14.
I20260709 12:23:57.517978 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:57.575721 18363 raft_consensus.cc:2368] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 16. Current term is 17.
I20260709 12:23:57.575629 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 20 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:57.517009 18366 raft_consensus.cc:2410] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 19 because replica has last-logged OpId of term: 18 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 13.
I20260709 12:23:57.466784 24834 raft_consensus.cc:2804] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:23:57.529922 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:57.581578 18361 raft_consensus.cc:2376] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 17. Re-sending same reply.
I20260709 12:23:57.542141 18333 leader_election.cc:304] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:57.544446 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 19 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:23:57.545120 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 89445.8us
I20260709 12:23:57.584851 18364 raft_consensus.cc:2410] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 19 because replica has last-logged OpId of term: 18 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 14.
I20260709 12:23:57.548995 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 24 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:57.586109 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:57.577956 17836 raft_consensus.cc:2468] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 14.
I20260709 12:23:57.492347 25072 raft_consensus.cc:493] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:23:57.564277 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:57.594365 17762 raft_consensus.cc:2468] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 17.
I20260709 12:23:57.458952 25089 raft_consensus.cc:2749] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:23:57.568344 17763 raft_consensus.cc:2368] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 16. Current term is 17.
W20260709 12:23:57.597404 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 113037us
I20260709 12:23:57.598659 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:57.599417 18368 raft_consensus.cc:2368] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 16. Current term is 17.
W20260709 12:23:57.597883 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 55863us
I20260709 12:23:57.480274 25184 consensus_queue.cc:1048] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.121s
I20260709 12:23:57.587579 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:57.573617 17758 raft_consensus.cc:2468] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 19.
I20260709 12:23:57.586803 18366 raft_consensus.cc:1275] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 20 index: 16. Preceding OpId from leader: term: 21 index: 17. (index mismatch)
I20260709 12:23:57.480249 25156 raft_consensus.cc:515] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:23:57.554589 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 111455us
I20260709 12:23:57.566128 24998 raft_consensus.cc:493] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:23:57.594529 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 15 candidate_status { last_received { term: 14 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:57.574955 24891 raft_consensus.cc:3060] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:57.601459 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 23 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:57.585026 24834 raft_consensus.cc:493] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:57.602267 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 18 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:57.611451 17684 raft_consensus.cc:2410] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 23 because replica has last-logged OpId of term: 21 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 14.
I20260709 12:23:57.603394 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:57.534477 25183 consensus_queue.cc:1059] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.134s
I20260709 12:23:57.589680 17764 raft_consensus.cc:2468] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 21.
I20260709 12:23:57.586745 17802 leader_election.cc:304] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:23:57.609019 18333 leader_election.cc:304] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:57.601672 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:57.627754 18365 raft_consensus.cc:2368] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 15. Current term is 16.
I20260709 12:23:57.559612 24952 raft_consensus.cc:515] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:57.609884 17727 leader_election.cc:304] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:57.593019 24946 raft_consensus.cc:493] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:57.624473 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:57.637418 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:57.640486 17763 raft_consensus.cc:2468] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 22.
I20260709 12:23:57.605517 25182 raft_consensus.cc:2749] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 12:23:57.653364 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 20 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:57.655807 17762 raft_consensus.cc:2468] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 19.
I20260709 12:23:57.606690 25146 leader_election.cc:290] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:23:57.662073 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 75381.5us
I20260709 12:23:57.606211 25076 raft_consensus.cc:2749] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
W20260709 12:23:57.660176 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 104486us
I20260709 12:23:57.611189 24998 raft_consensus.cc:3060] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:57.625752 24834 raft_consensus.cc:3060] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:23:57.663069 17802 leader_election.cc:304] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:23:57.615391 25153 raft_consensus.cc:493] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:57.666868 18362 raft_consensus.cc:1240] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 14 LEADER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 13. Current term is 14. Ops: []
I20260709 12:23:57.627504 24819 raft_consensus.cc:493] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:57.686678 24819 raft_consensus.cc:515] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:57.652702 25077 raft_consensus.cc:2804] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:23:57.687959 25078 leader_election.cc:290] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:23:57.671428 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 62479.5us
I20260709 12:23:57.649125 24946 raft_consensus.cc:3060] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:57.657698 25121 leader_election.cc:290] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:57.685775 25119 raft_consensus.cc:493] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:57.625857 25149 consensus_queue.cc:1048] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.051s
I20260709 12:23:57.689260 24819 leader_election.cc:290] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:57.662132 25163 leader_election.cc:290] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:57.689344 25055 leader_election.cc:290] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:57.689344 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:57.707505 17761 raft_consensus.cc:3060] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:23:57.688755 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:57.696128 18333 leader_election.cc:304] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:23:57.708977 17729 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:57.696767 17801 leader_election.cc:304] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:57.596671 25080 consensus_queue.cc:260] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.12, Last appended by leader: 12, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:57.664551 25075 raft_consensus.cc:2804] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:23:57.661146 25082 leader_election.cc:290] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:57.625204 25136 raft_consensus.cc:493] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:57.694427 25134 leader_election.cc:290] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:23:57.744168 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:57.703747 24650 consensus_queue.cc:1048] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 12:23:57.670267 25131 leader_election.cc:290] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:57.727329 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:57.728526 17760 raft_consensus.cc:2393] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 19: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:23:57.728734 17801 leader_election.cc:304] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:57.730717 17687 raft_consensus.cc:1240] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 24. Current term is 25. Ops: []
I20260709 12:23:57.730845 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 19 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:57.750384 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:57.731369 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 22 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:57.732257 17685 raft_consensus.cc:1275] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 19 index: 11. Preceding OpId from leader: term: 19 index: 12. (index mismatch)
I20260709 12:23:57.733189 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:57.693720 25146 raft_consensus.cc:2804] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:23:57.625646 25072 raft_consensus.cc:515] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:57.728302 25075 raft_consensus.cc:493] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:57.692361 24767 raft_consensus.cc:515] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:57.699666 25121 raft_consensus.cc:493] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:57.700002 25076 raft_consensus.cc:2804] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:23:57.762586 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 19 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:57.700397 24927 raft_consensus.cc:697] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 20 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:23:57.699635 25119 raft_consensus.cc:515] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:57.700076 25078 consensus_queue.cc:1048] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.051s
I20260709 12:23:57.737886 25149 raft_consensus.cc:2749] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:23:57.727160 24962 leader_election.cc:290] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:57.776276 17685 raft_consensus.cc:1275] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 16 index: 11. Preceding OpId from leader: term: 21 index: 14. (index mismatch)
I20260709 12:23:57.777544 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 19 candidate_status { last_received { term: 16 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:57.762699 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:57.764554 25075 raft_consensus.cc:3060] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:57.758078 17831 raft_consensus.cc:2393] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 16: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:23:57.768569 25079 leader_election.cc:290] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:57.770198 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:57.781517 17758 raft_consensus.cc:2393] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 16: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:23:57.771994 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:57.693955 25153 raft_consensus.cc:515] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:57.756695 25146 raft_consensus.cc:493] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:57.780459 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:57.756880 25134 raft_consensus.cc:493] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:57.760088 25182 raft_consensus.cc:493] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:57.753751 18333 leader_election.cc:304] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:23:57.782258 17682 raft_consensus.cc:2393] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 25: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:23:57.787945 17760 raft_consensus.cc:2393] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 25: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:23:57.789768 17834 raft_consensus.cc:1275] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 17 index: 13. Preceding OpId from leader: term: 20 index: 15. (index mismatch)
I20260709 12:23:57.797204 17680 raft_consensus.cc:2410] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for term 19 because replica has last-logged OpId of term: 16 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 12.
I20260709 12:23:57.765249 25076 raft_consensus.cc:493] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
W20260709 12:23:57.790202 25185 log.cc:927] Time spent T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.165s	user 0.001s	sys 0.000s
W20260709 12:23:57.791721 25130 log.cc:927] Time spent T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 1.019s	user 0.007s	sys 0.009s
I20260709 12:23:57.792245 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:57.826442 17758 raft_consensus.cc:2468] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 17.
W20260709 12:23:57.821501 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 67862.6us
I20260709 12:23:57.746089 25136 raft_consensus.cc:515] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:57.783113 25164 raft_consensus.cc:2749] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:23:57.840710 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 18 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:57.838838 17755 raft_consensus.cc:2468] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 21.
I20260709 12:23:57.841255 17652 leader_election.cc:304] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:57.851053 24767 leader_election.cc:290] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 24 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:57.852375 25156 leader_election.cc:290] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:57.852800 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 18 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:57.853655 17760 raft_consensus.cc:2410] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for term 18 because replica has last-logged OpId of term: 17 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 13.
I20260709 12:23:57.854439 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:57.854974 17760 raft_consensus.cc:2468] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 22.
I20260709 12:23:57.856256 24927 consensus_queue.cc:237] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.11, Last appended by leader: 11, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:57.857012 25072 leader_election.cc:290] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:57.821024 25182 raft_consensus.cc:515] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:57.862444 25182 leader_election.cc:290] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:57.863157 25147 leader_election.cc:290] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:57.867100 17684 raft_consensus.cc:1240] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 23. Current term is 24. Ops: []
I20260709 12:23:57.867798 25153 leader_election.cc:290] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:57.821689 25134 raft_consensus.cc:515] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:57.870760 25134 leader_election.cc:290] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:57.793428 25077 raft_consensus.cc:697] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 17 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:23:57.871843 24891 raft_consensus.cc:515] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:57.873037 25077 consensus_queue.cc:237] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.12, Last appended by leader: 12, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:57.881929 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:57.771040 25121 raft_consensus.cc:515] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:57.792699 25146 raft_consensus.cc:3060] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Advancing to term 22
W20260709 12:23:57.885735 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 157081us
W20260709 12:23:57.886026 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 44888.4us
I20260709 12:23:57.887290 17687 raft_consensus.cc:2393] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 22: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:23:57.822091 25076 raft_consensus.cc:3060] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:57.831498 25149 raft_consensus.cc:2804] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:23:57.884596 17760 pending_rounds.cc:85] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb: Aborting all ops after (but not including) 14
I20260709 12:23:57.898691 17760 pending_rounds.cc:107] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb: Aborting uncommitted NO_OP operation due to leader change: 20.15
I20260709 12:23:57.877712 24891 leader_election.cc:290] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 17 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:57.877749 25201 raft_consensus.cc:2749] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:23:57.855259 24960 raft_consensus.cc:515] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:57.900668 17760 raft_consensus.cc:2889] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:57.893730 17801 leader_election.cc:400] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 16 pre-election: Vote denied by peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651) with higher term. Message: Invalid argument: T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 16. Current term is 17.
W20260709 12:23:57.905149 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 263793us
I20260709 12:23:57.912648 17801 leader_election.cc:403] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 16 pre-election: Cancelling election due to peer responding with higher term
W20260709 12:23:57.912513 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 386175us
I20260709 12:23:57.894170 24767 raft_consensus.cc:2804] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:23:57.915046 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 18 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:23:57.915270 17803 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:57.922325 18333 leader_election.cc:304] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:57.929574 18333 leader_election.cc:304] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:57.932102 17652 leader_election.cc:304] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:57.897948 25149 raft_consensus.cc:493] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:57.936841 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 20 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:57.899055 25002 raft_consensus.cc:493] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:23:57.946660 17831 raft_consensus.cc:1275] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
I20260709 12:23:57.947762 25206 raft_consensus.cc:493] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:23:57.941959 24974 raft_consensus.cc:2749] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:23:57.948515 25206 raft_consensus.cc:515] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:57.950845 25206 leader_election.cc:290] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:23:57.952322 25193 log.cc:927] Time spent T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.203s	user 0.000s	sys 0.000s
I20260709 12:23:57.952489 25002 raft_consensus.cc:515] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:57.953806 25212 raft_consensus.cc:493] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:57.954327 25212 raft_consensus.cc:515] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:57.954748 25002 leader_election.cc:290] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:57.952330 24819 raft_consensus.cc:493] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:23:57.955741 24819 raft_consensus.cc:515] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:57.956508 25198 log.cc:927] Time spent T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.195s	user 0.000s	sys 0.001s
I20260709 12:23:57.957928 24819 leader_election.cc:290] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:57.947839 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 21 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:57.959295 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 22 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:57.960372 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 19 candidate_status { last_received { term: 16 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:57.960882 17831 raft_consensus.cc:2468] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 18.
I20260709 12:23:57.961652 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:57.962627 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:57.963207 17831 raft_consensus.cc:2468] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 22.
I20260709 12:23:57.964550 25204 raft_consensus.cc:993] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3: : Instructing follower e063541542a94969898868bf6eb5ce1b to start an election
I20260709 12:23:57.965095 25003 raft_consensus.cc:1081] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 20 LEADER]: Signalling peer e063541542a94969898868bf6eb5ce1b to start an election
I20260709 12:23:57.946508 25082 raft_consensus.cc:2804] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:23:57.967742 25082 raft_consensus.cc:697] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 21 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:23:57.968544 25121 leader_election.cc:290] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:57.968703 25082 consensus_queue.cc:237] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 20.11, Last appended by leader: 11, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:57.969307 25121 raft_consensus.cc:493] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:57.969882 25121 raft_consensus.cc:515] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:57.931181 24657 raft_consensus.cc:2749] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Leader pre-election lost for term 16. Reason: Vote denied by peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651) with higher term. Message: Invalid argument: T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 16. Current term is 17.
I20260709 12:23:57.923924 25072 raft_consensus.cc:493] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:57.972757 25199 log.cc:927] Time spent T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.076s	user 0.000s	sys 0.000s
I20260709 12:23:57.972375 25072 raft_consensus.cc:515] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:57.962635 24946 raft_consensus.cc:515] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:57.975078 25072 leader_election.cc:290] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:57.979074 25075 raft_consensus.cc:515] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:57.980098 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 48052.4us
I20260709 12:23:57.981743 25136 leader_election.cc:290] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:57.983408 24952 leader_election.cc:290] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:57.983958 17652 leader_election.cc:304] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:57.985713 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:57.992960 18368 raft_consensus.cc:2393] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 18: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:23:57.895612 24721 raft_consensus.cc:515] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:57.995644 17652 leader_election.cc:304] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:58.001689 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:57.938057 25134 raft_consensus.cc:3060] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Advancing to term 16
W20260709 12:23:58.002626 17803 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:58.000252 17652 leader_election.cc:304] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:57.941248 25149 raft_consensus.cc:3060] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:57.914207 24818 consensus_queue.cc:1048] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.084s
I20260709 12:23:57.975106 25119 leader_election.cc:290] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:57.939195 24767 raft_consensus.cc:697] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 25 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:23:58.009473 17803 leader_election.cc:304] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:58.020901 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:23:58.022578 25203 log.cc:927] Time spent T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.083s	user 0.000s	sys 0.000s
I20260709 12:23:58.023552 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:58.024366 17834 raft_consensus.cc:2468] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 18.
I20260709 12:23:58.024856 17803 leader_election.cc:304] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:58.025705 25156 raft_consensus.cc:2749] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:23:58.025446 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:58.026331 25156 raft_consensus.cc:2749] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 12:23:58.026878 17761 raft_consensus.cc:2468] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 22.
I20260709 12:23:58.026993 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:58.013835 25204 raft_consensus.cc:2804] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:23:58.040215 25204 raft_consensus.cc:493] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:23:58.040491 17833 raft_consensus.cc:1275] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 16 index: 12. (index mismatch)
I20260709 12:23:58.040679 25204 raft_consensus.cc:3060] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:23:58.040791 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:58.041929 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:58.042461 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:58.043011 17833 raft_consensus.cc:2376] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 16. Re-sending same reply.
I20260709 12:23:58.010658 25164 raft_consensus.cc:493] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:58.043998 25076 raft_consensus.cc:515] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:58.044369 25164 raft_consensus.cc:515] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:58.046105 25076 leader_election.cc:290] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 21 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:58.046922 25164 leader_election.cc:290] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:58.043829 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:58.048312 17833 raft_consensus.cc:2376] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 16. Re-sending same reply.
I20260709 12:23:58.028259 17652 leader_election.cc:304] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:58.003883 25002 raft_consensus.cc:2804] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:23:58.034224 25056 raft_consensus.cc:3055] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:23:58.034051 24767 consensus_queue.cc:237] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 24.18, Last appended by leader: 18, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:58.027769 17803 leader_election.cc:400] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 16 pre-election: Vote denied by peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577) with higher term. Message: Invalid argument: T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 16. Current term is 17.
I20260709 12:23:58.035741 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:58.059376 17836 raft_consensus.cc:2468] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 19.
I20260709 12:23:58.060492 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:58.061122 17836 raft_consensus.cc:2468] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 22.
I20260709 12:23:58.061973 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:23:58.038619 17830 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35042 (request call id 2790) took 2183 ms (client timeout 2148 ms). Trace:
I20260709 12:23:57.983840 25168 raft_consensus.cc:515] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:58.042461 17834 raft_consensus.cc:2468] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 19.
I20260709 12:23:57.941408 25182 consensus_queue.cc:1048] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 12, Time since last communication: 0.072s
I20260709 12:23:58.027837 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 23 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:58.051721 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:58.052102 25184 raft_consensus.cc:2804] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:23:58.051780 25121 leader_election.cc:290] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:58.010475 25206 raft_consensus.cc:493] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:58.004772 24891 raft_consensus.cc:2804] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:23:58.076877 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 18 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:58.064800 17761 raft_consensus.cc:2410] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 23 because replica has last-logged OpId of term: 21 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 14.
I20260709 12:23:58.078372 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:58.078521 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:58.078974 17833 raft_consensus.cc:3060] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:58.079043 17834 raft_consensus.cc:2468] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 22.
I20260709 12:23:58.082926 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 22 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:58.083832 17836 raft_consensus.cc:2410] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for term 22 because replica has last-logged OpId of term: 20 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 11.
I20260709 12:23:58.064855 17803 leader_election.cc:304] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
W20260709 12:23:58.063006 17830 rpcz_store.cc:269] 0709 12:23:55.854931 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:55.858753 (+  3822us) service_pool.cc:224] Handling call
0709 12:23:58.038033 (+2179280us) spinlock_profiling.cc:243] Waited 1.31 s on lock 0x7b540008cad4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:23:58.038517 (+   484us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3006289152}
I20260709 12:23:58.001494 25208 raft_consensus.cc:493] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:58.087778 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 23 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:58.070138 17652 leader_election.cc:304] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:23:58.084758 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:58.085566 17803 leader_election.cc:304] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:58.057725 25002 raft_consensus.cc:493] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:23:58.089848 17652 leader_election.cc:304] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:23:58.057005 25056 raft_consensus.cc:740] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:23:58.094153 18367 raft_consensus.cc:2393] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 22: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:23:58.091645 25146 raft_consensus.cc:515] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:58.103360 25211 raft_consensus.cc:493] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:58.105804 25146 leader_election.cc:290] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 22 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:58.103982 25211 raft_consensus.cc:515] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:58.107693 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:58.107604 17689 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37402 (request call id 3252) took 2099 ms. Trace:
I20260709 12:23:58.108944 17689 rpcz_store.cc:276] 0709 12:23:56.007635 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:56.021892 (+ 14257us) service_pool.cc:224] Handling call
0709 12:23:56.059099 (+ 37207us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:57.132435 (+1073336us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:58.107150 (+974715us) spinlock_profiling.cc:243] Waited 1.07 s on lock 0x7b54000c5654. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:23:58.107418 (+   268us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2464812416}
I20260709 12:23:58.073642 24818 consensus_queue.cc:1048] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.039s
I20260709 12:23:58.075225 24891 raft_consensus.cc:493] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:58.019152 24834 raft_consensus.cc:515] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:58.113828 17803 leader_election.cc:304] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:58.079814 25184 raft_consensus.cc:697] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 22 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:23:58.129980 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:58.130267 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:58.130960 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:58.132128 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:58.132747 17764 raft_consensus.cc:2468] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 19.
I20260709 12:23:58.091527 24657 raft_consensus.cc:2749] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:23:58.094347 25088 raft_consensus.cc:2804] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:23:58.135162 17803 leader_election.cc:400] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 16 election: Vote denied by peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577) with higher term. Message: Invalid argument: T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 16. Current term is 17.
I20260709 12:23:58.097041 25002 raft_consensus.cc:3060] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:58.163806 24721 leader_election.cc:290] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 20 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:58.165072 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:58.166287 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:58.166718 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 19 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:58.167349 17759 raft_consensus.cc:2468] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 18.
I20260709 12:23:58.167953 25184 consensus_queue.cc:237] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 21.15, Last appended by leader: 15, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:58.094264 25208 raft_consensus.cc:515] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:58.171303 25208 leader_election.cc:290] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:58.128233 24891 raft_consensus.cc:3060] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:23:58.080698 25206 raft_consensus.cc:515] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:58.172065 17830 raft_consensus.cc:2468] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 22.
I20260709 12:23:58.172906 25168 leader_election.cc:290] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 22 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:58.173455 25206 leader_election.cc:290] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:23:58.120183 24964 consensus_meta.cc:287] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.674s	user 0.012s	sys 0.000s
I20260709 12:23:58.166841 17761 raft_consensus.cc:2468] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 22.
I20260709 12:23:58.176213 17803 leader_election.cc:400] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 15 election: Vote denied by peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577) with higher term. Message: Invalid argument: T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 15. Current term is 16.
I20260709 12:23:58.098044 24927 raft_consensus.cc:2804] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:23:58.176636 25147 raft_consensus.cc:493] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:23:58.152617 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:58.136647 25126 raft_consensus.cc:493] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:58.136422 25212 leader_election.cc:290] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:23:58.187175 25161 log.cc:927] Time spent T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 1.410s	user 0.000s	sys 0.003s
I20260709 12:23:58.181432 24964 raft_consensus.cc:515] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:58.162026 25088 raft_consensus.cc:493] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:58.190838 25088 raft_consensus.cc:3060] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:23:58.190738 25149 raft_consensus.cc:515] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:58.161669 25146 raft_consensus.cc:2749] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:23:58.183704 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 18 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:58.119771 24801 raft_consensus.cc:2749] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:23:58.198186 17759 raft_consensus.cc:2410] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 18 because replica has last-logged OpId of term: 14 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 9.
I20260709 12:23:58.185688 25147 raft_consensus.cc:515] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:58.201290 24964 leader_election.cc:290] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 17 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:58.211288 25147 leader_election.cc:290] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:58.211054 25204 raft_consensus.cc:515] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:58.213368 24723 leader_election.cc:290] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:58.214053 25088 raft_consensus.cc:515] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:58.215268 17838 raft_consensus.cc:1275] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 14 index: 14. Preceding OpId from leader: term: 18 index: 16. (index mismatch)
I20260709 12:23:58.216064 25088 leader_election.cc:290] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 23 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:58.211709 17833 raft_consensus.cc:2468] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 17.
I20260709 12:23:58.216771 17838 raft_consensus.cc:1240] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 21. Current term is 22. Ops: [21.14-21.14]
I20260709 12:23:58.217518 17834 raft_consensus.cc:1240] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 16. Current term is 17. Ops: [16.13-16.13]
I20260709 12:23:58.197962 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:58.223716 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:58.224260 17759 raft_consensus.cc:1275] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 18 index: 14. Preceding OpId from leader: term: 20 index: 15. (index mismatch)
I20260709 12:23:58.221557 17760 raft_consensus.cc:2468] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 21.
W20260709 12:23:58.223007 25221 log.cc:927] Time spent T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.104s	user 0.000s	sys 0.001s
I20260709 12:23:58.231081 17838 raft_consensus.cc:1240] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 17. Current term is 18. Ops: []
I20260709 12:23:58.234010 25149 leader_election.cc:290] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 18 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:58.197655 25126 raft_consensus.cc:515] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:58.236892 25126 leader_election.cc:290] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:58.216724 25211 leader_election.cc:290] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:58.180485 25206 consensus_queue.cc:1048] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.045s
I20260709 12:23:58.245820 25204 leader_election.cc:290] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 23 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:58.179970 24927 raft_consensus.cc:493] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:23:58.256939 17838 raft_consensus.cc:1240] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 20. Current term is 21. Ops: []
I20260709 12:23:58.251523 25206 consensus_queue.cc:1243] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer 334a4e1f59944544b3b3ff36616b57da log is divergent from this leader: its last log entry 20.16 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 15
I20260709 12:23:58.184835 25056 consensus_queue.cc:260] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 20.11, Last appended by leader: 11, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:58.259176 24927 raft_consensus.cc:3060] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:58.275908 25149 raft_consensus.cc:493] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:58.276518 18365 raft_consensus.cc:1275] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 16 index: 15. Preceding OpId from leader: term: 18 index: 16. (index mismatch)
I20260709 12:23:58.289183 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
W20260709 12:23:58.280840 25172 log.cc:927] Time spent T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 1.020s	user 0.004s	sys 0.000s
I20260709 12:23:58.284231 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:58.284380 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:58.291955 17687 raft_consensus.cc:2468] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 22.
I20260709 12:23:58.284996 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:58.287142 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:58.294234 18374 raft_consensus.cc:2468] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 21.
I20260709 12:23:58.295521 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:58.287722 24834 leader_election.cc:290] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 22 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:23:58.287997 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:58.277774 18363 raft_consensus.cc:1240] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 22. Current term is 23. Ops: []
I20260709 12:23:58.297699 18333 leader_election.cc:304] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:23:58.304908 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:58.305371 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:58.305610 18333 leader_election.cc:304] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:23:58.308326 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 18 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:58.303270 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 19 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:58.302405 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:58.307762 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:58.299671 17652 leader_election.cc:304] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:23:58.311108 17652 leader_election.cc:304] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:23:58.311283 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:58.312188 18374 raft_consensus.cc:2468] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 21.
I20260709 12:23:58.316661 17687 raft_consensus.cc:2393] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 21: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:23:58.309207 18333 leader_election.cc:304] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:23:58.317047 18365 raft_consensus.cc:3055] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:23:58.312286 17652 leader_election.cc:304] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:58.308914 18363 raft_consensus.cc:2468] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 17.
I20260709 12:23:58.320335 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 22 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:58.321098 17684 raft_consensus.cc:2393] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 19: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:23:58.317489 18365 raft_consensus.cc:740] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:23:58.322325 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 21 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:58.317617 25182 raft_consensus.cc:2804] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:23:58.322408 17803 leader_election.cc:304] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:23:58.297036 25206 raft_consensus.cc:493] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:58.332835 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 23 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:58.302736 24790 raft_consensus.cc:493] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:23:58.328152 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:58.329761 17756 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36572 (request call id 1289) took 1051 ms. Trace:
I20260709 12:23:58.331204 17803 leader_election.cc:304] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:23:58.334842 17761 raft_consensus.cc:3060] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:23:58.335500 18374 pending_rounds.cc:85] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da: Aborting all ops after (but not including) 15
I20260709 12:23:58.334802 17756 rpcz_store.cc:276] 0709 12:23:57.278323 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:57.279063 (+   740us) service_pool.cc:224] Handling call
0709 12:23:57.279451 (+   388us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:57.504126 (+224675us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:58.329442 (+825316us) spinlock_profiling.cc:243] Waited 164 ms on lock 0x7b540019f5d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:23:58.329704 (+   262us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":377756032}
I20260709 12:23:58.335840 18374 pending_rounds.cc:107] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da: Aborting uncommitted NO_OP operation due to leader change: 20.16
I20260709 12:23:58.337352 18374 raft_consensus.cc:2889] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:58.332751 25121 raft_consensus.cc:493] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:58.341113 18363 raft_consensus.cc:1275] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
I20260709 12:23:58.330622 17728 leader_election.cc:304] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:58.342250 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:58.328507 18365 consensus_queue.cc:260] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 19.14, Last appended by leader: 14, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:58.345330 18365 raft_consensus.cc:3060] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:58.320890 24723 raft_consensus.cc:2804] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:23:58.321007 25126 raft_consensus.cc:2804] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:23:58.324484 24818 raft_consensus.cc:2804] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:23:58.325709 25182 raft_consensus.cc:493] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:58.324705 25141 raft_consensus.cc:2804] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:23:58.323669 25214 raft_consensus.cc:493] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:23:58.341302 24834 raft_consensus.cc:2804] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:23:58.353711 17728 leader_election.cc:304] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:58.324066 25099 raft_consensus.cc:2804] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:23:58.348677 24819 raft_consensus.cc:2749] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:23:58.348120 25211 raft_consensus.cc:2804] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:23:58.358161 25126 raft_consensus.cc:493] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:58.304467 25149 raft_consensus.cc:515] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:58.361981 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:58.353554 18363 raft_consensus.cc:1240] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 16. Current term is 17. Ops: []
I20260709 12:23:58.320739 25088 consensus_queue.cc:1048] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.035s
I20260709 12:23:58.368580 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 22 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:58.356532 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
W20260709 12:23:58.359112 24998 consensus_meta.cc:287] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.666s	user 0.012s	sys 0.000s
I20260709 12:23:58.348810 24952 raft_consensus.cc:493] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:58.373939 24834 raft_consensus.cc:493] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:58.348465 25206 raft_consensus.cc:515] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:58.358948 18364 raft_consensus.cc:1275] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 12 index: 10. Preceding OpId from leader: term: 16 index: 12. (index mismatch)
I20260709 12:23:58.361367 24818 raft_consensus.cc:493] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:58.360100 25182 raft_consensus.cc:3060] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:58.387470 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 19 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:58.348234 25121 raft_consensus.cc:515] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:58.351722 25002 raft_consensus.cc:515] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:58.404919 24818 raft_consensus.cc:3060] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:58.389534 17689 raft_consensus.cc:2393] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 23: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:23:58.363297 24723 raft_consensus.cc:697] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 17 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:23:58.416162 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:58.417099 17832 raft_consensus.cc:3060] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:23:58.393658 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:58.420092 17684 raft_consensus.cc:2393] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 24: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:23:58.387809 25126 raft_consensus.cc:3060] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:23:58.352983 25136 consensus_queue.cc:1048] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.031s
I20260709 12:23:58.372291 25099 raft_consensus.cc:493] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:58.383486 25211 raft_consensus.cc:493] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:58.404140 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:58.362905 24891 raft_consensus.cc:515] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:58.428812 25099 raft_consensus.cc:3060] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:58.429204 24392 raft_consensus.cc:493] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
W20260709 12:23:58.476748 25230 log.cc:927] Time spent T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.090s	user 0.000s	sys 0.001s
W20260709 12:23:58.462368 17727 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:58.408365 17761 raft_consensus.cc:2468] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 23.
I20260709 12:23:58.410552 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:58.478528 18364 raft_consensus.cc:2468] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 16.
I20260709 12:23:58.351141 24790 raft_consensus.cc:515] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:58.411666 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 18 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:58.412035 17680 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81"
dest_uuid: "0da169ff82674dc1947055ce43de68a3"
mode: GRACEFUL
new_leader_uuid: "e063541542a94969898868bf6eb5ce1b"
 from {username='slave'} at 127.0.0.1:54540
I20260709 12:23:58.526324 17680 raft_consensus.cc:606] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 18 LEADER]: Received request to transfer leadership to TS e063541542a94969898868bf6eb5ce1b
I20260709 12:23:58.394735 25072 raft_consensus.cc:2749] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:23:58.401870 24834 raft_consensus.cc:3060] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:23:58.476543 24392 raft_consensus.cc:515] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:58.364751 25141 raft_consensus.cc:493] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:23:58.403278 25204 raft_consensus.cc:493] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:58.397867 25131 consensus_queue.cc:1048] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 10, Time since last communication: 0.013s
I20260709 12:23:58.375476 25214 raft_consensus.cc:515] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:58.404132 24952 raft_consensus.cc:515] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:58.474661 25211 raft_consensus.cc:3060] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:58.416474 24723 consensus_queue.cc:237] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.13, Last appended by leader: 13, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:58.537490 25226 log.cc:927] Time spent T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.163s	user 0.001s	sys 0.000s
I20260709 12:23:58.544484 17759 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38786 (request call id 3551) took 1239 ms. Trace:
I20260709 12:23:58.545735 25206 leader_election.cc:290] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:23:58.547922 25231 log.cc:927] Time spent T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.187s	user 0.001s	sys 0.000s
I20260709 12:23:58.473385 25146 raft_consensus.cc:493] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
W20260709 12:23:58.550940 25228 log.cc:927] Time spent T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.164s	user 0.000s	sys 0.000s
I20260709 12:23:58.399255 24998 raft_consensus.cc:515] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:58.638916 17832 raft_consensus.cc:2468] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 23.
I20260709 12:23:58.683068 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:58.683866 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:58.682142 25141 raft_consensus.cc:3060] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:23:58.479133 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:59.119765 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:59.130116 17763 raft_consensus.cc:3060] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:59.018736 18362 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59902 (request call id 1745) took 1564 ms. Trace:
I20260709 12:23:59.132154 18362 rpcz_store.cc:276] 0709 12:23:57.454160 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:57.455914 (+  1754us) service_pool.cc:224] Handling call
0709 12:23:57.456279 (+   365us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:57.686281 (+230002us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:59.018325 (+1332044us) spinlock_profiling.cc:243] Waited 208 ms on lock 0x7b540069e1d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:23:59.018646 (+   321us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":478455040}
W20260709 12:23:59.006444 17727 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:58.844383 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:59.129235 17759 rpcz_store.cc:276] 0709 12:23:57.305528 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:58.248647 (+943119us) service_pool.cc:224] Handling call
0709 12:23:58.264133 (+ 15486us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:58.268131 (+  3998us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 12:23:58.268166 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:58.295255 (+ 27089us) log.cc:844] Serialized 26 byte log entry
0709 12:23:58.401820 (+106565us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:23:58.401862 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:58.401931 (+    69us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:58.544260 (+142329us) raft_consensus.cc:1685] finished
0709 12:23:58.544280 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:58.544418 (+   138us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3670,"raft.queue_time_us":2714,"raft.run_cpu_time_us":89,"raft.run_wall_time_us":88,"thread_start_us":73358,"threads_started":1,"wal-append.queue_time_us":226047}
W20260709 12:23:58.960255 18365 consensus_meta.cc:287] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.615s	user 0.012s	sys 0.000s
I20260709 12:23:59.168164 18365 raft_consensus.cc:1275] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 19 index: 14. Preceding OpId from leader: term: 20 index: 14. (term mismatch)
I20260709 12:23:59.168687 18365 pending_rounds.cc:85] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da: Aborting all ops after (but not including) 13
I20260709 12:23:59.169063 18365 pending_rounds.cc:107] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da: Aborting uncommitted NO_OP operation due to leader change: 19.14
I20260709 12:23:59.169430 18365 raft_consensus.cc:2889] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:23:58.685907 25232 log.cc:927] Time spent T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.135s	user 0.001s	sys 0.000s
W20260709 12:23:59.187359 17802 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.16.29.65:33071 timed out after 1.595s (SENT) [suppressed 15 similar messages]
W20260709 12:23:58.829113 25219 log.cc:927] Time spent T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.104s	user 0.000s	sys 0.000s
W20260709 12:23:58.945572 25233 log.cc:927] Time spent T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.223s	user 0.000s	sys 0.001s
I20260709 12:23:59.119418 24987 raft_consensus.cc:3060] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:59.226358 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:59.063009 25075 leader_election.cc:290] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 18 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:59.051118 24790 leader_election.cc:290] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:58.896610 25182 raft_consensus.cc:515] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:59.227118 18364 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.541s	user 0.010s	sys 0.015s
I20260709 12:23:59.064684 24989 leader_election.cc:290] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:59.165448 25002 leader_election.cc:290] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 20 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:58.684767 25204 raft_consensus.cc:515] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:58.687579 24964 raft_consensus.cc:493] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:58.855435 25189 raft_consensus.cc:493] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
W20260709 12:23:59.194700 18361 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.507s	user 0.010s	sys 0.031s
I20260709 12:23:59.317540 24392 leader_election.cc:290] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:59.318434 25189 raft_consensus.cc:515] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:59.298673 24964 raft_consensus.cc:515] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:59.322433 24964 leader_election.cc:290] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:59.302055 25121 leader_election.cc:290] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:59.253669 17756 raft_consensus.cc:3060] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:59.324813 25006 consensus_queue.cc:260] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 16.15, Last appended by leader: 15, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:59.325891 25006 raft_consensus.cc:3060] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:59.254182 18363 raft_consensus.cc:2468] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 22.
I20260709 12:23:59.016950 25081 raft_consensus.cc:3055] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:23:59.328472 25081 raft_consensus.cc:740] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:23:59.329221 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:59.329890 17758 raft_consensus.cc:3060] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:59.329627 25081 consensus_queue.cc:260] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.11, Last appended by leader: 11, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:59.330695 25081 raft_consensus.cc:3060] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:59.256809 17755 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36572 (request call id 1288) took 1979 ms. Trace:
I20260709 12:23:59.331923 17755 rpcz_store.cc:276] 0709 12:23:57.277948 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:57.292363 (+ 14415us) service_pool.cc:224] Handling call
0709 12:23:59.081139 (+1788776us) spinlock_profiling.cc:243] Waited 159 ms on lock 0x7b54001b9254. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:23:59.256757 (+175618us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":365126016}
I20260709 12:23:59.333238 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 20 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:23:59.334918 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 21 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:58.687382 25149 leader_election.cc:290] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:59.256258 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:59.261586 18368 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59902 (request call id 1762) took 1338 ms. Trace:
I20260709 12:23:59.353924 18368 rpcz_store.cc:276] 0709 12:23:57.914684 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:57.914802 (+   118us) service_pool.cc:224] Handling call
0709 12:23:59.116980 (+1202178us) spinlock_profiling.cc:243] Waited 63.8 ms on lock 0x7b54006c6954. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:23:59.253269 (+136289us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":146737536}
I20260709 12:23:59.355405 17680 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba"
dest_uuid: "0da169ff82674dc1947055ce43de68a3"
mode: GRACEFUL
new_leader_uuid: "e063541542a94969898868bf6eb5ce1b"
 from {username='slave'} at 127.0.0.1:54540
I20260709 12:23:59.355950 17687 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37396 (request call id 2877) took 1030 ms. Trace:
I20260709 12:23:59.356829 17680 raft_consensus.cc:606] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Received request to transfer leadership to TS e063541542a94969898868bf6eb5ce1b
I20260709 12:23:59.357218 17680 raft_consensus.cc:614] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:23:59.263062 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:23:59.128650 17803 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:59.268110 17837 raft_consensus.cc:1240] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 14. Current term is 15. Ops: []
W20260709 12:23:59.269506 18364 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.583s	user 0.010s	sys 0.015s
W20260709 12:23:59.357962 17803 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:59.357962 17832 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35040 (request call id 3122) took 1077 ms. Trace:
I20260709 12:23:59.358953 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:59.359232 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:59.359534 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 19 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:59.279727 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 20 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:59.360348 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:59.356312 17687 rpcz_store.cc:276] 0709 12:23:58.323859 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:59.113589 (+789730us) service_pool.cc:224] Handling call
0709 12:23:59.354568 (+240979us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:23:59.267032 25142 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 20 current term: 20 is higher.
I20260709 12:23:59.281610 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:59.325902 25189 leader_election.cc:290] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:59.358461 17832 rpcz_store.cc:276] 0709 12:23:58.280945 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:59.262803 (+981858us) service_pool.cc:224] Handling call
0709 12:23:59.357904 (+ 95101us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:59.363067 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:59.363201 18366 raft_consensus.cc:1240] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 17. Current term is 18. Ops: []
W20260709 12:23:59.318970 24818 consensus_meta.cc:287] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.895s	user 0.007s	sys 0.001s
I20260709 12:23:59.363963 24818 raft_consensus.cc:515] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:59.364720 25099 consensus_meta.cc:287] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.890s	user 0.001s	sys 0.008s
I20260709 12:23:59.269910 24967 raft_consensus.cc:3060] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:59.365796 17680 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5"
dest_uuid: "0da169ff82674dc1947055ce43de68a3"
mode: GRACEFUL
new_leader_uuid: "334a4e1f59944544b3b3ff36616b57da"
 from {username='slave'} at 127.0.0.1:54540
I20260709 12:23:59.366130 24818 leader_election.cc:290] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 19 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:59.366369 17680 raft_consensus.cc:606] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 22 LEADER]: Received request to transfer leadership to TS 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:59.366703 24818 consensus_queue.cc:1059] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 16.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 12:23:59.367467 24818 raft_consensus.cc:3055] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:23:59.367776 24818 raft_consensus.cc:740] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:23:59.262354 25146 raft_consensus.cc:515] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:59.369733 25146 leader_election.cc:290] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:59.337253 18361 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 2160) took 1062 ms. Trace:
I20260709 12:23:59.360569 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 22 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:59.340394 17833 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35040 (request call id 3111) took 1333 ms. Trace:
I20260709 12:23:59.370968 25141 raft_consensus.cc:515] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:59.371124 17833 rpcz_store.cc:276] 0709 12:23:58.007321 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:58.223547 (+216226us) service_pool.cc:224] Handling call
0709 12:23:58.224382 (+   835us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:58.224838 (+   456us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:58.224886 (+    48us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:58.228982 (+  4096us) log.cc:844] Serialized 556 byte log entry
0709 12:23:58.259629 (+ 30647us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:23:58.259671 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:58.259729 (+    58us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:59.340187 (+1080458us) raft_consensus.cc:1685] finished
0709 12:23:59.340209 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:59.340337 (+   128us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":30303,"threads_started":1,"wal-append.queue_time_us":98458}
I20260709 12:23:59.372910 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:59.342203 24952 leader_election.cc:290] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:59.342895 17686 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44898 (request call id 1418) took 2705 ms. Trace:
I20260709 12:23:59.343142 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 22 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:59.343840 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 22 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:59.256825 17763 raft_consensus.cc:2468] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 18.
I20260709 12:23:59.391173 17763 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36572 (request call id 1313) took 1088 ms. Trace:
I20260709 12:23:59.256240 17761 raft_consensus.cc:3060] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:23:59.347698 17756 raft_consensus.cc:2468] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 17.
I20260709 12:23:59.391633 17763 rpcz_store.cc:276] 0709 12:23:58.302310 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:58.483030 (+180720us) service_pool.cc:224] Handling call
0709 12:23:59.391096 (+908066us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:59.349000 17758 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 18.
I20260709 12:23:59.392419 17756 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36572 (request call id 1312) took 1103 ms. Trace:
I20260709 12:23:59.392804 17756 rpcz_store.cc:276] 0709 12:23:58.288761 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:58.429955 (+141194us) service_pool.cc:224] Handling call
0709 12:23:59.392366 (+962411us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:23:59.319746 24927 consensus_meta.cc:287] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 1.015s	user 0.010s	sys 0.000s
I20260709 12:23:59.352140 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:59.286590 17834 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37994 (request call id 1414) took 1345 ms. Trace:
I20260709 12:23:59.363270 18374 raft_consensus.cc:1240] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 17. Current term is 18. Ops: []
I20260709 12:23:59.394059 17834 rpcz_store.cc:276] 0709 12:23:57.941530 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:58.108911 (+167381us) service_pool.cc:224] Handling call
0709 12:23:58.217217 (+108306us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:58.218000 (+   783us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:59.286376 (+1068376us) spinlock_profiling.cc:243] Waited 108 ms on lock 0x7b54000ac3d0. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a61f8ad 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:23:59.286529 (+   153us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":248052480}
W20260709 12:23:59.335237 25134 consensus_meta.cc:287] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 1.321s	user 0.003s	sys 0.008s
I20260709 12:23:59.373092 25141 leader_election.cc:290] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 22 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:59.373581 17680 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9"
dest_uuid: "0da169ff82674dc1947055ce43de68a3"
mode: GRACEFUL
new_leader_uuid: "e063541542a94969898868bf6eb5ce1b"
 from {username='slave'} at 127.0.0.1:54540
I20260709 12:23:59.397440 17680 raft_consensus.cc:606] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 22 LEADER]: Received request to transfer leadership to TS e063541542a94969898868bf6eb5ce1b
I20260709 12:23:59.342027 17838 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37994 (request call id 1420) took 1327 ms. Trace:
I20260709 12:23:59.397948 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:59.398826 17763 raft_consensus.cc:2468] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 18.
I20260709 12:23:59.374303 17832 raft_consensus.cc:2393] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 22: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:23:59.370307 18361 rpcz_store.cc:276] 0709 12:23:58.274987 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:58.285986 (+ 10999us) service_pool.cc:224] Handling call
0709 12:23:58.286312 (+   326us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:58.404312 (+118000us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 12:23:58.413374 (+  9062us) log.cc:844] Serialized 14 byte log entry
0709 12:23:59.309210 (+895836us) log.cc:844] Serialized 14 byte log entry
0709 12:23:59.335759 (+ 26549us) log.cc:844] Serialized 14 byte log entry
0709 12:23:59.335944 (+   185us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:59.335990 (+    46us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:23:59.336033 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:59.336959 (+   926us) spinlock_profiling.cc:243] Waited 117 ms on lock 0x7b54006946d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:23:59.336998 (+    39us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:59.337197 (+   199us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":48738,"prepare.queue_time_us":71291,"prepare.run_cpu_time_us":126,"prepare.run_wall_time_us":134,"raft.queue_time_us":52229,"raft.run_cpu_time_us":221,"raft.run_wall_time_us":221,"spinlock_wait_cycles":269519232,"thread_start_us":870815,"threads_started":2,"wal-append.queue_time_us":856875}
I20260709 12:23:59.375594 24801 consensus_queue.cc:1048] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20260709 12:23:59.375942 24392 raft_consensus.cc:493] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:59.376515 25146 raft_consensus.cc:493] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:59.365114 25099 raft_consensus.cc:515] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:59.406373 17756 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38786 (request call id 3556) took 2098 ms. Trace:
I20260709 12:23:59.406913 17756 rpcz_store.cc:276] 0709 12:23:57.307766 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:59.400863 (+2093097us) service_pool.cc:224] Handling call
0709 12:23:59.401144 (+   281us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:59.405851 (+  4707us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:59.405909 (+    58us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:59.405990 (+    81us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:23:59.406053 (+    63us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:59.406148 (+    95us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:59.406300 (+   152us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:59.377347 25182 leader_election.cc:290] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 19 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:59.377851 17831 raft_consensus.cc:2468] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 19.
I20260709 12:23:59.377566 24818 consensus_queue.cc:260] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.12, Last appended by leader: 12, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:59.378777 24998 leader_election.cc:290] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 15 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:59.354115 25088 consensus_queue.cc:1048] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.025s
I20260709 12:23:59.420230 16501 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 6
I20260709 12:23:59.380561 18366 raft_consensus.cc:1275] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 19 index: 12. Preceding OpId from leader: term: 20 index: 12. (term mismatch)
I20260709 12:23:59.420881 18366 pending_rounds.cc:85] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da: Aborting all ops after (but not including) 11
I20260709 12:23:59.421150 18366 pending_rounds.cc:107] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da: Aborting uncommitted NO_OP operation due to leader change: 19.12
I20260709 12:23:59.421404 18366 raft_consensus.cc:2889] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:59.421959 24946 leader_election.cc:290] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 15 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:59.422703 16501 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
W20260709 12:23:59.352190 24834 consensus_meta.cc:287] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.814s	user 0.010s	sys 0.000s
I20260709 12:23:59.423946 24834 raft_consensus.cc:515] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:59.426223 24834 leader_election.cc:290] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 23 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:59.426923 18362 raft_consensus.cc:1240] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 18. Current term is 19. Ops: []
I20260709 12:23:59.427322 25099 leader_election.cc:290] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 19 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:59.386345 17760 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38786 (request call id 3554) took 2079 ms. Trace:
I20260709 12:23:59.372644 25095 raft_consensus.cc:493] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:23:59.329279 25204 leader_election.cc:290] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:59.348323 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 19 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:59.392868 17758 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36572 (request call id 1309) took 1105 ms. Trace:
I20260709 12:23:59.439903 17758 rpcz_store.cc:276] 0709 12:23:58.287808 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:59.328962 (+1041154us) service_pool.cc:224] Handling call
0709 12:23:59.392793 (+ 63831us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:59.441032 17756 raft_consensus.cc:1240] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 19. Current term is 20. Ops: [19.14-19.14]
I20260709 12:23:59.393409 18363 raft_consensus.cc:3060] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:23:59.442020 17756 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36572 (request call id 1299) took 1949 ms. Trace:
I20260709 12:23:59.360549 17759 raft_consensus.cc:2468] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 19.
I20260709 12:23:59.442752 17758 raft_consensus.cc:1240] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 21. Current term is 22. Ops: [19.14-21.15]
I20260709 12:23:59.443559 17758 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36572 (request call id 1300) took 1951 ms. Trace:
I20260709 12:23:59.444113 17758 rpcz_store.cc:276] 0709 12:23:57.492574 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:59.440917 (+1948343us) service_pool.cc:224] Handling call
0709 12:23:59.441333 (+   416us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:23:59.443233 (+  1900us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:59.443495 (+   262us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:59.323109 25165 leader_election.cc:290] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:59.400942 17833 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35040 (request call id 3124) took 1105 ms. Trace:
I20260709 12:23:59.442970 17759 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38776 (request call id 2817) took 1156 ms. Trace:
W20260709 12:23:59.412890 25126 consensus_meta.cc:287] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.941s	user 0.010s	sys 0.000s
I20260709 12:23:59.404628 17832 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37994 (request call id 1422) took 1119 ms. Trace:
I20260709 12:23:59.417487 25146 raft_consensus.cc:515] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:59.447041 17759 rpcz_store.cc:276] 0709 12:23:58.286836 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:59.261308 (+974472us) service_pool.cc:224] Handling call
0709 12:23:59.442886 (+181578us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:59.451709 24960 leader_election.cc:290] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 23 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:59.453028 25146 leader_election.cc:290] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:59.396863 24927 raft_consensus.cc:515] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:59.453691 25126 raft_consensus.cc:515] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:59.453790 17756 rpcz_store.cc:276] 0709 12:23:57.492270 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:59.416226 (+1923956us) service_pool.cc:224] Handling call
0709 12:23:59.440717 (+ 24491us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:59.441663 (+   946us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:59.441945 (+   282us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:59.454949 24927 leader_election.cc:290] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 20 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:59.460079 17763 raft_consensus.cc:1275] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 19 index: 11. Preceding OpId from leader: term: 22 index: 12. (index mismatch)
I20260709 12:23:59.462348 17832 rpcz_store.cc:276] 0709 12:23:58.285282 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:59.362865 (+1077583us) service_pool.cc:224] Handling call
0709 12:23:59.404545 (+ 41680us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:59.462864 17763 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38786 (request call id 3561) took 2093 ms. Trace:
I20260709 12:23:59.432520 16501 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:23:59.463543 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:59.402748 25134 raft_consensus.cc:2749] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:23:59.410382 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:59.410193 25076 raft_consensus.cc:493] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:59.469336 17654 leader_election.cc:304] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:59.469851 17834 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8"
dest_uuid: "e063541542a94969898868bf6eb5ce1b"
 from {username='slave'} at 127.0.0.1:35040
I20260709 12:23:59.469942 25076 raft_consensus.cc:515] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:59.470611 17834 raft_consensus.cc:493] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:23:59.471040 17834 raft_consensus.cc:3060] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Advancing to term 21
W20260709 12:23:59.471388 25177 log.cc:927] Time spent T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 2.106s	user 0.008s	sys 0.018s
I20260709 12:23:59.354316 24767 consensus_queue.cc:1059] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 22.14, Next index: 15, Last known committed idx: 13, Time since last communication: 0.030s
I20260709 12:23:59.473402 17835 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35042 (request call id 2792) took 3617 ms. Trace:
I20260709 12:23:59.387878 17686 rpcz_store.cc:276] 0709 12:23:56.637654 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:56.638080 (+   426us) service_pool.cc:224] Handling call
0709 12:23:56.655840 (+ 17760us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:57.770756 (+1114916us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:57.770784 (+    28us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:57.770821 (+    37us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:23:57.770845 (+    24us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:59.342638 (+1571793us) spinlock_profiling.cc:243] Waited 1.11 s on lock 0x7b54003480d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:23:59.342684 (+    46us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:59.342800 (+   116us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2563584768}
I20260709 12:23:59.386906 25184 consensus_queue.cc:1048] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.026s
I20260709 12:23:59.475744 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:59.446767 17833 rpcz_store.cc:276] 0709 12:23:58.295723 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:59.372663 (+1076940us) service_pool.cc:224] Handling call
0709 12:23:59.400876 (+ 28213us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:59.398320 17838 rpcz_store.cc:276] 0709 12:23:58.014457 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:58.258127 (+243670us) service_pool.cc:224] Handling call
0709 12:23:58.267034 (+  8907us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:58.267278 (+   244us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 12:23:58.302129 (+ 34851us) log.cc:844] Serialized 14 byte log entry
0709 12:23:59.341326 (+1039197us) spinlock_profiling.cc:243] Waited 10.7 ms on lock 0x7b18002a8e60. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b474adc0a 00007f7b50bed219 00007f7b47535a40 00007f7b475304e4 00007f7b47530455 00007f7b4e0183c7 00007f7b4a62a9a7 00007f7b4a63a0bf 00007f7b4a63a038
0709 12:23:59.341520 (+   194us) log.cc:844] Serialized 14 byte log entry
0709 12:23:59.341711 (+   191us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:59.341758 (+    47us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:23:59.341799 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:59.341861 (+    62us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:59.341974 (+   113us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":24084,"prepare.queue_time_us":35118,"prepare.run_wall_time_us":43,"raft.queue_time_us":2320,"raft.run_cpu_time_us":221,"raft.run_wall_time_us":9285,"spinlock_wait_cycles":38670976}
I20260709 12:23:59.477852 18362 raft_consensus.cc:3060] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:23:59.464582 17832 raft_consensus.cc:3060] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:59.456810 17684 raft_consensus.cc:1240] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 19. Current term is 20. Ops: [19.12-19.12]
I20260709 12:23:59.408581 17755 raft_consensus.cc:1275] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 18 index: 15. Preceding OpId from leader: term: 19 index: 16. (index mismatch)
I20260709 12:23:59.483916 17755 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38786 (request call id 3559) took 2149 ms. Trace:
I20260709 12:23:59.484467 17755 rpcz_store.cc:276] 0709 12:23:57.336081 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:59.402004 (+2065923us) service_pool.cc:224] Handling call
0709 12:23:59.403261 (+  1257us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:59.479632 (+ 76371us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:59.479831 (+   199us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:59.463277 17763 rpcz_store.cc:276] 0709 12:23:57.369471 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:59.403979 (+2034508us) service_pool.cc:224] Handling call
0709 12:23:59.439785 (+ 35806us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:59.462610 (+ 22825us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:59.462810 (+   200us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:59.456322 17756 raft_consensus.cc:1240] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 16. Current term is 17. Ops: [16.12-16.12]
I20260709 12:23:59.491752 17756 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38776 (request call id 2801) took 1912 ms. Trace:
I20260709 12:23:59.492192 17756 rpcz_store.cc:276] 0709 12:23:57.579315 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:59.455638 (+1876323us) service_pool.cc:224] Handling call
0709 12:23:59.456076 (+   438us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:59.491085 (+ 35009us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:59.491312 (+   227us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:59.492645 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 19 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:59.473832 17761 raft_consensus.cc:2468] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 22.
I20260709 12:23:59.430229 17760 rpcz_store.cc:276] 0709 12:23:57.306511 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:58.230718 (+924207us) service_pool.cc:224] Handling call
0709 12:23:58.256404 (+ 25686us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:58.265137 (+  8733us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:58.265175 (+    38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:58.283438 (+ 18263us) log.cc:844] Serialized 26 byte log entry
0709 12:23:58.360894 (+ 77456us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:23:58.360943 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:58.361038 (+    95us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:59.386132 (+1025094us) raft_consensus.cc:1685] finished
0709 12:23:59.386153 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:59.386284 (+   131us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":98,"raft.run_cpu_time_us":132,"raft.run_wall_time_us":4235,"thread_start_us":77014,"threads_started":1,"wal-append.queue_time_us":139877}
I20260709 12:23:59.493525 17761 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36572 (request call id 1311) took 1205 ms. Trace:
I20260709 12:23:59.493896 17761 rpcz_store.cc:276] 0709 12:23:58.288447 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:58.478816 (+190369us) service_pool.cc:224] Handling call
0709 12:23:59.493466 (+1014650us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:59.455791 25126 leader_election.cc:290] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 23 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:59.473803 17835 rpcz_store.cc:276] 0709 12:23:55.855653 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:55.855785 (+   132us) service_pool.cc:224] Handling call
0709 12:23:55.856204 (+   419us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:55.884007 (+ 27803us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 12:23:56.806706 (+922699us) spinlock_profiling.cc:243] Waited 3.39 ms on lock 0x7b18000ae460. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b4a1c84bc 00007f7b4751648b 00007f7b503709a9 00007f7b4753269b 00007f7b4753093f 00007f7b47530455 00007f7b4e0183c7 00007f7b4a62a9a7 00007f7b4a6335e5
0709 12:23:56.806845 (+   139us) log.cc:844] Serialized 14 byte log entry
0709 12:23:57.183668 (+376823us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:57.214235 (+ 30567us) log.cc:844] Serialized 26 byte log entry
0709 12:23:57.214434 (+   199us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:23:57.227028 (+ 12594us) log.cc:844] Serialized 14 byte log entry
0709 12:23:57.263001 (+ 35973us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:57.263103 (+   102us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:57.791502 (+528399us) spinlock_profiling.cc:243] Waited 4.95 ms on lock 0x7b18000ae460. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b47533522 00007f7b475357c1 00007f7b47535779 00007f7b47535709 00007f7b475356d1 00007f7b475349cd 0000000000388634 0000000000388479 00007f7b4751683a
0709 12:23:59.471056 (+1679554us) spinlock_profiling.cc:243] Waited 35.8 ms on lock 0x7b2400238f88. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b473b99c4 00007f7b473b7acb 00007f7b47572dc9 00007f7b47572a55 00007f7b4757209d 00007f7b4a1a410e 00007f7b4a1bb88c 00007f7b4a1babb6 00007f7b4a1ba7d8 00007f7b4a1c4221
0709 12:23:59.473168 (+  2112us) raft_consensus.cc:1685] finished
0709 12:23:59.473191 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:59.473317 (+   126us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":42249,"prepare.queue_time_us":78806,"prepare.run_cpu_time_us":72,"prepare.run_wall_time_us":74,"raft.queue_time_us":68543,"raft.run_cpu_time_us":551,"raft.run_wall_time_us":36611,"spinlock_wait_cycles":103168256,"thread_start_us":397244,"threads_started":2,"wal-append.queue_time_us":478241}
I20260709 12:23:59.495536 17761 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36572 (request call id 1306) took 1760 ms. Trace:
I20260709 12:23:59.497218 18365 raft_consensus.cc:1240] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 18. Current term is 19. Ops: [18.16-18.16]
I20260709 12:23:59.497381 17761 rpcz_store.cc:276] 0709 12:23:57.735681 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:59.494739 (+1759058us) service_pool.cc:224] Handling call
0709 12:23:59.495061 (+   322us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:59.495238 (+   177us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:59.495263 (+    25us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:59.495306 (+    43us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:23:59.495342 (+    36us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:59.495391 (+    49us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:59.495483 (+    92us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:59.493875 17832 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 18.
I20260709 12:23:59.499873 17832 raft_consensus.cc:1240] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 16. Current term is 17. Ops: []
I20260709 12:23:59.500536 17832 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35040 (request call id 3118) took 1442 ms. Trace:
I20260709 12:23:59.502027 17832 rpcz_store.cc:276] 0709 12:23:58.058413 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:59.499428 (+1441015us) service_pool.cc:224] Handling call
0709 12:23:59.499705 (+   277us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:59.500325 (+   620us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:59.500489 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:59.507501 17838 raft_consensus.cc:1240] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 14. Current term is 15. Ops: [14.9-14.9]
I20260709 12:23:59.508461 17838 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35040 (request call id 3116) took 1458 ms. Trace:
I20260709 12:23:59.508863 17838 rpcz_store.cc:276] 0709 12:23:58.049984 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:59.478372 (+1428388us) service_pool.cc:224] Handling call
0709 12:23:59.507174 (+ 28802us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:59.508097 (+   923us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:59.508381 (+   284us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:59.521835 17756 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38776 (request call id 2805) took 1909 ms. Trace:
I20260709 12:23:59.522431 17756 rpcz_store.cc:276] 0709 12:23:57.612950 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:59.493019 (+1880069us) service_pool.cc:224] Handling call
0709 12:23:59.521180 (+ 28161us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:59.521416 (+   236us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:59.521454 (+    38us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:59.521510 (+    56us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:23:59.521553 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:59.521623 (+    70us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:59.521761 (+   138us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:59.410955 24392 raft_consensus.cc:515] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:59.533226 17833 raft_consensus.cc:1240] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 17. Current term is 18. Ops: []
I20260709 12:23:59.533704 17763 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38776 (request call id 2804) took 1926 ms. Trace:
I20260709 12:23:59.533941 17833 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35040 (request call id 3119) took 1253 ms. Trace:
I20260709 12:23:59.534368 18363 raft_consensus.cc:1275] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 24 index: 18. Preceding OpId from leader: term: 25 index: 19. (index mismatch)
I20260709 12:23:59.534334 17763 rpcz_store.cc:276] 0709 12:23:57.607370 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:59.490678 (+1883308us) service_pool.cc:224] Handling call
0709 12:23:59.533021 (+ 42343us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:59.533269 (+   248us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:59.533306 (+    37us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:59.533362 (+    56us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:23:59.533406 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:59.533493 (+    87us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:59.533631 (+   138us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:59.479924 24818 raft_consensus.cc:3060] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:59.473415 25076 leader_election.cc:290] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:59.464105 25095 raft_consensus.cc:515] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:59.559870 17760 raft_consensus.cc:1275] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 18 index: 14. (index mismatch)
I20260709 12:23:59.554970 17833 rpcz_store.cc:276] 0709 12:23:58.280227 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:59.532765 (+1252538us) service_pool.cc:224] Handling call
0709 12:23:59.533045 (+   280us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:59.533687 (+   642us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:59.533876 (+   189us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:59.496343 25006 raft_consensus.cc:2749] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260709 12:23:59.572124 17838 raft_consensus.cc:1275] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 20 index: 12. Preceding OpId from leader: term: 21 index: 13. (index mismatch)
I20260709 12:23:59.571991 17763 raft_consensus.cc:1240] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 17. Current term is 18. Ops: []
I20260709 12:23:59.574462 17838 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35040 (request call id 3120) took 1294 ms. Trace:
I20260709 12:23:59.574505 17763 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38776 (request call id 2811) took 1643 ms. Trace:
I20260709 12:23:59.575104 17763 rpcz_store.cc:276] 0709 12:23:57.930949 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:59.535761 (+1604812us) service_pool.cc:224] Handling call
0709 12:23:59.571817 (+ 36056us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:59.574285 (+  2468us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:59.574456 (+   171us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:59.571964 17760 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36572 (request call id 1305) took 1947 ms. Trace:
I20260709 12:23:59.574914 17838 rpcz_store.cc:276] 0709 12:23:58.280497 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:59.536343 (+1255846us) service_pool.cc:224] Handling call
0709 12:23:59.571788 (+ 35445us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:59.574174 (+  2386us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:59.574391 (+   217us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:59.576548 17760 rpcz_store.cc:276] 0709 12:23:57.624422 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:59.493660 (+1869238us) service_pool.cc:224] Handling call
0709 12:23:59.559393 (+ 65733us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:59.570423 (+ 11030us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:59.571902 (+  1479us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:59.475117 25211 raft_consensus.cc:515] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:59.583101 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 260337us
W20260709 12:23:59.583575 17727 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:59.584939 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:59.585724 18365 raft_consensus.cc:2393] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 17: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
W20260709 12:23:59.585646 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 398334us
I20260709 12:23:59.587931 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:59.590750 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:59.597214 17680 raft_consensus.cc:1240] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 22. Current term is 23. Ops: [22.12-22.12]
I20260709 12:23:59.600431 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:59.600805 17763 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38786 (request call id 3565) took 1657 ms. Trace:
I20260709 12:23:59.601224 17763 rpcz_store.cc:276] 0709 12:23:57.943351 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:59.575906 (+1632555us) service_pool.cc:224] Handling call
0709 12:23:59.600342 (+ 24436us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:59.600506 (+   164us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:59.600528 (+    22us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:59.600567 (+    39us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:23:59.600598 (+    31us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:59.600649 (+    51us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:59.600750 (+   101us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:59.599612 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:59.599311 17727 leader_election.cc:304] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:23:59.614781 17833 raft_consensus.cc:1240] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 17. Current term is 18. Ops: [17.13-17.13]
I20260709 12:23:59.613744 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 21 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:59.619285 18366 raft_consensus.cc:2468] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 20.
I20260709 12:23:59.619441 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:59.619735 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:59.620398 17685 raft_consensus.cc:3060] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:59.619369 17686 raft_consensus.cc:1240] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 19. Current term is 20. Ops: [19.14-19.14]
I20260709 12:23:59.622041 17729 leader_election.cc:304] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:23:59.565126 25126 consensus_queue.cc:1059] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 17.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.038s
I20260709 12:23:59.623649 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:59.621028 17834 raft_consensus.cc:515] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:23:59.624111 17727 leader_election.cc:304] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:59.624253 17687 raft_consensus.cc:3060] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:23:59.624785 17835 raft_consensus.cc:1240] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 20. Current term is 21. Ops: []
I20260709 12:23:59.566920 25088 consensus_queue.cc:1059] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.042s
I20260709 12:23:59.628630 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:59.629467 17683 raft_consensus.cc:2468] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 17.
I20260709 12:23:59.630517 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 18 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:59.631366 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:59.632071 17689 raft_consensus.cc:3060] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:59.636075 17728 leader_election.cc:304] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
W20260709 12:23:59.638796 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 65723.1us
I20260709 12:23:59.639611 17763 raft_consensus.cc:1240] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 17. Current term is 18. Ops: []
I20260709 12:23:59.574705 25179 consensus_queue.cc:1059] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 18.15, Next index: 16, Last known committed idx: 14, Time since last communication: 0.049s
I20260709 12:23:59.636955 17803 leader_election.cc:304] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:23:59.575139 25002 consensus_queue.cc:1048] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.047s
I20260709 12:23:59.648862 17763 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38776 (request call id 2814) took 1596 ms. Trace:
I20260709 12:23:59.615159 25165 raft_consensus.cc:493] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:59.647100 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:59.650044 17834 leader_election.cc:290] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 21 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:23:59.653357 17685 raft_consensus.cc:2468] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 20.
I20260709 12:23:59.667371 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:59.667259 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:59.671206 17686 raft_consensus.cc:2393] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 23: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:23:59.671206 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:59.653692 17803 leader_election.cc:304] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:59.656211 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:59.674710 17830 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37994 (request call id 1416) took 1687 ms. Trace:
I20260709 12:23:59.676497 18365 raft_consensus.cc:2393] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 18: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:23:59.675132 17830 rpcz_store.cc:276] 0709 12:23:57.986955 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:58.174166 (+187211us) service_pool.cc:224] Handling call
0709 12:23:58.174480 (+   314us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:58.248061 (+ 73581us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:58.248107 (+    46us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:58.248167 (+    60us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:23:58.248214 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:59.674477 (+1426263us) spinlock_profiling.cc:243] Waited 73.3 ms on lock 0x7b5400266e54. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:23:59.674527 (+    50us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:59.674656 (+   129us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":168597888}
I20260709 12:23:59.654886 17763 rpcz_store.cc:276] 0709 12:23:58.052091 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:59.615672 (+1563581us) service_pool.cc:224] Handling call
0709 12:23:59.639364 (+ 23692us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:59.640196 (+   832us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:59.648786 (+  8590us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:59.678341 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 17 candidate_status { last_received { term: 16 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:23:59.682176 17831 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37994 (request call id 1411) took 1742 ms (client timeout 1788 ms). Trace:
I20260709 12:23:59.607939 25243 consensus_queue.cc:1059] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 16.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.047s
I20260709 12:23:59.683650 17830 raft_consensus.cc:2393] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 17: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:23:59.685911 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 15 candidate_status { last_received { term: 14 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
W20260709 12:23:59.697690 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 45823.4us
W20260709 12:23:59.704046 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:59.787616 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 163560us
W20260709 12:23:59.901010 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 278386us
I20260709 12:23:59.902624 18362 raft_consensus.cc:2468] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 23.
I20260709 12:23:59.902902 17727 leader_election.cc:304] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:59.702389 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:23:59.687914 17687 raft_consensus.cc:2468] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 22.
I20260709 12:23:59.786201 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:23:59.797425 25078 raft_consensus.cc:2804] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:23:59.684387 18374 raft_consensus.cc:2468] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 21.
I20260709 12:23:59.796368 17802 leader_election.cc:304] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:23:59.908077 17838 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37994 (request call id 1421) took 1623 ms. Trace:
W20260709 12:23:59.911490 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 115236us
W20260709 12:23:59.682742 17831 rpcz_store.cc:269] 0709 12:23:57.940041 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:57.985053 (+ 45012us) service_pool.cc:224] Handling call
0709 12:23:59.681970 (+1696917us) spinlock_profiling.cc:243] Waited 1.39 s on lock 0x7b54003359d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:23:59.682102 (+   132us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3199705472}
I20260709 12:23:59.703125 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:23:59.806151 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:59.640900 24952 consensus_queue.cc:1048] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.070s
I20260709 12:23:59.748970 25121 raft_consensus.cc:2804] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:23:59.723892 25247 raft_consensus.cc:493] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:59.672605 25252 raft_consensus.cc:493] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:59.763533 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 154937us
I20260709 12:23:59.916806 17802 leader_election.cc:304] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:59.918637 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:23:59.704130 17685 raft_consensus.cc:3060] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:59.919476 18363 raft_consensus.cc:2393] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 23: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
W20260709 12:23:59.716105 25181 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 2.266s	user 0.021s	sys 0.034s
I20260709 12:23:59.851573 17762 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38786 (request call id 3571) took 1799 ms. Trace:
I20260709 12:23:59.920173 17762 rpcz_store.cc:276] 0709 12:23:58.052359 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:59.816978 (+1764619us) service_pool.cc:224] Handling call
0709 12:23:59.850853 (+ 33875us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:59.851167 (+   314us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:59.851202 (+    35us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:59.851252 (+    50us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:23:59.851288 (+    36us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:59.851353 (+    65us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:59.851498 (+   145us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:59.640728 25067 leader_election.cc:290] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:59.623257 25146 consensus_queue.cc:1048] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.052s
I20260709 12:23:59.729276 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:59.798686 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: true dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:59.749369 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:59.724607 24946 raft_consensus.cc:2749] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:23:59.698571 25088 consensus_queue.cc:1243] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer 334a4e1f59944544b3b3ff36616b57da log is divergent from this leader: its last log entry 16.11 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 9
I20260709 12:23:59.748605 25246 raft_consensus.cc:2804] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:23:59.710842 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:59.705155 25206 raft_consensus.cc:493] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:59.669931 25076 raft_consensus.cc:2804] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:23:59.669751 24767 raft_consensus.cc:3055] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:23:59.675176 25165 raft_consensus.cc:515] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:23:59.753213 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 99231.2us
I20260709 12:23:59.863492 25140 leader_election.cc:290] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:59.865217 25179 raft_consensus.cc:3055] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:23:59.866811 25080 raft_consensus.cc:3060] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:59.868485 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:23:59.868621 25251 log.cc:927] Time spent T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.097s	user 0.000s	sys 0.000s
I20260709 12:23:59.869139 24392 leader_election.cc:290] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:23:59.874558 24642 raft_consensus.cc:493] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:59.851603 25053 leader_election.cc:290] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 18 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:23:59.885224 18334 connection.cc:256] RPC call timeout handler was delayed by -1.707869189000121s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1695s for a response
I20260709 12:23:59.751688 24801 raft_consensus.cc:493] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
W20260709 12:23:59.765614 17803 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:59.924602 25247 raft_consensus.cc:515] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:59.890175 17837 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37994 (request call id 1408) took 2197 ms. Trace:
W20260709 12:23:59.892045 25254 log.cc:927] Time spent T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.167s	user 0.000s	sys 0.001s
I20260709 12:23:59.893879 25095 leader_election.cc:290] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:59.894754 25211 leader_election.cc:290] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 18 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:59.802037 25253 raft_consensus.cc:493] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:23:59.704931 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 21 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:59.701483 17834 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RunLeaderElection from 127.0.0.1:35040 (request call id 3115) took 1633 ms. Trace:
I20260709 12:23:59.794924 25184 consensus_queue.cc:1048] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.075s
I20260709 12:23:59.907344 17829 raft_consensus.cc:3060] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:59.904430 17729 leader_election.cc:304] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:59.909101 17727 leader_election.cc:304] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
W20260709 12:23:59.914009 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:59.914855 17687 raft_consensus.cc:1240] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 22. Current term is 23. Ops: []
W20260709 12:23:59.921231 25181 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 2.480s	user 0.028s	sys 0.040s
I20260709 12:23:59.912081 17838 rpcz_store.cc:276] 0709 12:23:58.284983 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:59.576884 (+1291901us) service_pool.cc:224] Handling call
0709 12:23:59.600397 (+ 23513us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:23:59.600804 (+   407us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 12:23:59.623084 (+ 22280us) log.cc:844] Serialized 14 byte log entry
0709 12:23:59.816110 (+193026us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:23:59.857180 (+ 41070us) log.cc:844] Serialized 52 byte log entry
0709 12:23:59.857394 (+   214us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:23:59.893298 (+ 35904us) spinlock_profiling.cc:243] Waited 13.9 ms on lock 0x7b180016f720. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b47533522 00007f7b475357c1 00007f7b47535779 00007f7b47535709 00007f7b475356d1 00007f7b475349cd 0000000000388634 0000000000388479 00007f7b4751683a
0709 12:23:59.901119 (+  7821us) log.cc:844] Serialized 14 byte log entry
0709 12:23:59.901470 (+   351us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:59.901582 (+   112us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:59.907851 (+  6269us) raft_consensus.cc:1685] finished
0709 12:23:59.907869 (+    18us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:59.908024 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":64269,"prepare.queue_time_us":52693,"prepare.run_cpu_time_us":82,"prepare.run_wall_time_us":85,"raft.queue_time_us":65784,"raft.run_cpu_time_us":292,"raft.run_wall_time_us":38333,"spinlock_wait_cycles":32056576,"thread_start_us":54257,"threads_started":1,"wal-append.queue_time_us":149693}
W20260709 12:23:59.924821 17681 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44898 (request call id 1419) took 3286 ms (client timeout 1695 ms). Trace:
I20260709 12:23:59.930090 25252 raft_consensus.cc:515] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:23:59.932003 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:59.934854 17683 raft_consensus.cc:3055] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:23:59.922230 17689 raft_consensus.cc:2468] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 21.
I20260709 12:23:59.924798 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:23:59.936493 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 21 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:23:59.936367 25099 consensus_queue.cc:1059] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 15.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 12:23:59.935634 17685 raft_consensus.cc:2468] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 15.
I20260709 12:23:59.937402 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 18 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:23:59.937670 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 20 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:59.930487 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:00.054895 17683 raft_consensus.cc:740] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:24:00.055166 17762 raft_consensus.cc:2468] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 19.
I20260709 12:24:00.055706 18365 raft_consensus.cc:2468] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 18.
I20260709 12:23:59.985749 17758 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38776 (request call id 2797) took 2448 ms. Trace:
I20260709 12:23:59.940614 17682 raft_consensus.cc:2393] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 23: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:23:59.900360 25264 raft_consensus.cc:2804] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:23:59.938920 17802 leader_election.cc:304] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:24:00.069334 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 19 candidate_status { last_received { term: 16 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
W20260709 12:24:00.070232 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 131611us
I20260709 12:23:59.944792 25246 raft_consensus.cc:493] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:59.929522 25158 raft_consensus.cc:2749] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:23:59.916473 25210 raft_consensus.cc:2764] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Leader pre-election decision vote started in defunct term 22: won
I20260709 12:23:59.946863 18333 leader_election.cc:304] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:23:59.918226 25142 raft_consensus.cc:493] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:23:59.921682 25078 raft_consensus.cc:697] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
W20260709 12:23:59.969686 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 333670us
I20260709 12:23:59.928984 25214 leader_election.cc:290] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:23:59.930691 25164 consensus_queue.cc:1059] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 19.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.047s
W20260709 12:24:00.058723 17654 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:59.950814 25076 raft_consensus.cc:493] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
W20260709 12:24:00.126641 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 58830.1us
I20260709 12:24:00.058833 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:00.055979 17837 rpcz_store.cc:276] 0709 12:23:57.692401 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:57.693240 (+   839us) service_pool.cc:224] Handling call
0709 12:23:57.693559 (+   319us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:59.357969 (+1664410us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:59.889821 (+531852us) spinlock_profiling.cc:243] Waited 1.57 s on lock 0x7b5400265a54. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:23:59.890091 (+   270us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3620702464}
W20260709 12:24:00.061061 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 156714us
I20260709 12:23:59.976908 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 23 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:00.064345 18363 raft_consensus.cc:2468] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 16.
W20260709 12:24:00.000717 18367 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59902 (request call id 1746) took 2543 ms (client timeout 2921 ms). Trace:
W20260709 12:24:00.066569 17681 rpcz_store.cc:269] 0709 12:23:56.638001 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:56.768776 (+130775us) service_pool.cc:224] Handling call
0709 12:23:59.922250 (+3153474us) spinlock_profiling.cc:243] Waited 484 ms on lock 0x7b54004259d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:23:59.924727 (+  2477us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1112263424}
I20260709 12:24:00.070261 17831 raft_consensus.cc:3060] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:59.942268 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 15 candidate_status { last_received { term: 14 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:00.056535 17758 rpcz_store.cc:276] 0709 12:23:57.537219 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:59.445053 (+1907834us) service_pool.cc:224] Handling call
0709 12:23:59.445462 (+   409us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:59.464972 (+ 19510us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:59.465019 (+    47us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:59.494514 (+ 29495us) log.cc:844] Serialized 26 byte log entry
0709 12:23:59.731597 (+237083us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:23:59.731658 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:59.731751 (+    93us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:59.985433 (+253682us) raft_consensus.cc:1685] finished
0709 12:23:59.985466 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:59.985673 (+   207us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":927,"spinlock_wait_cycles":2434944,"thread_start_us":191924,"threads_started":1,"wal-append.queue_time_us":373527}
I20260709 12:23:59.945493 24987 consensus_queue.cc:1059] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 17.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.057s
I20260709 12:24:00.071715 17802 leader_election.cc:304] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:23:59.950027 24767 raft_consensus.cc:740] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:24:00.073292 17829 raft_consensus.cc:2468] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 19.
I20260709 12:24:00.069226 17683 consensus_queue.cc:260] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 20.12, Last appended by leader: 12, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:00.123358 17684 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37396 (request call id 2881) took 1581 ms. Trace:
I20260709 12:24:00.205426 17683 raft_consensus.cc:3060] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:00.075019 17835 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35040 (request call id 3117) took 2024 ms. Trace:
I20260709 12:24:00.123809 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:23:59.991007 25267 ts_tablet_manager.cc:933] T a5732e71b9d54f8786b8c511d69943bf P 334a4e1f59944544b3b3ff36616b57da: Initiating tablet copy from peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:00.208355 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
W20260709 12:24:00.133222 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 224216us
I20260709 12:24:00.207170 17835 rpcz_store.cc:276] 0709 12:23:58.050448 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:59.520035 (+1469587us) service_pool.cc:224] Handling call
0709 12:23:59.558727 (+ 38692us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:59.625185 (+ 66458us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:00.074794 (+449609us) spinlock_profiling.cc:243] Waited 65.8 ms on lock 0x7b54003cc8d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:24:00.074962 (+   168us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":151413504}
I20260709 12:24:00.124552 17801 leader_election.cc:304] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:24:00.205662 17684 rpcz_store.cc:276] 0709 12:23:58.541805 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:59.259993 (+718188us) service_pool.cc:224] Handling call
0709 12:23:59.260449 (+   456us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:59.479368 (+218919us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:00.122994 (+643626us) spinlock_profiling.cc:243] Waited 196 ms on lock 0x7b54003507d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:24:00.123293 (+   299us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":450839808}
I20260709 12:24:00.127213 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:00.126746 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:00.209549 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 17 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:00.134066 17728 leader_election.cc:304] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:23:59.973697 25179 raft_consensus.cc:740] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:23:59.956452 25137 consensus_queue.cc:1048] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.174s
I20260709 12:24:00.137781 25140 raft_consensus.cc:493] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:24:00.225163 17762 raft_consensus.cc:2393] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 22: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:24:00.226583 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 17 candidate_status { last_received { term: 16 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:00.141379 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:00.144502 17729 leader_election.cc:304] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:00.148602 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:00.228189 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:00.144500 25076 raft_consensus.cc:3060] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:00.149868 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:00.069854 24801 raft_consensus.cc:515] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:00.146742 18367 rpcz_store.cc:269] 0709 12:23:57.459719 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:57.459898 (+   179us) service_pool.cc:224] Handling call
0709 12:24:00.000419 (+2540521us) spinlock_profiling.cc:243] Waited 564 ms on lock 0x7b540047ca54. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:24:00.000602 (+   183us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1297900928}
I20260709 12:24:00.230952 18367 raft_consensus.cc:3055] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:24:00.231416 18367 raft_consensus.cc:740] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
W20260709 12:24:00.128773 25251 log.cc:927] Time spent T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.078s	user 0.001s	sys 0.000s
I20260709 12:24:00.122782 25189 raft_consensus.cc:2804] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:24:00.149537 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:00.161536 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:00.162999 17688 raft_consensus.cc:2393] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 23: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:24:00.124322 24992 raft_consensus.cc:2749] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:24:00.069032 25264 raft_consensus.cc:493] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:24:00.073342 25263 raft_consensus.cc:2749] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:24:00.147017 18334 leader_election.cc:304] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:00.064946 17834 rpcz_store.cc:276] 0709 12:23:58.036037 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:59.469700 (+1433663us) service_pool.cc:224] Handling call
0709 12:23:59.669542 (+199842us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:00.204921 24946 raft_consensus.cc:493] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:00.130195 25246 raft_consensus.cc:3060] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:00.210148 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:00.259325 18365 raft_consensus.cc:1240] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 22. Current term is 23. Ops: []
I20260709 12:24:00.259491 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 21 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:00.259979 17837 raft_consensus.cc:2368] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 22. Current term is 23.
I20260709 12:24:00.260339 17764 raft_consensus.cc:2410] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 21 because replica has last-logged OpId of term: 18 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 13.
I20260709 12:24:00.210422 17835 raft_consensus.cc:2410] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for term 17 because replica has last-logged OpId of term: 14 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 10.
I20260709 12:24:00.140688 25248 raft_consensus.cc:3055] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 16 LEADER]: Stepping down as leader of term 16
W20260709 12:24:00.210652 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 138996us
I20260709 12:24:00.247756 18367 consensus_queue.cc:260] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 21.15, Last appended by leader: 15, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:00.272768 17764 raft_consensus.cc:1275] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 15 index: 13. Preceding OpId from leader: term: 19 index: 15. (index mismatch)
I20260709 12:24:00.211695 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:00.143816 25142 raft_consensus.cc:515] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:00.213182 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 79271.3us
W20260709 12:24:00.214512 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 267564us
I20260709 12:24:00.049337 24642 raft_consensus.cc:515] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:00.215775 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 91300.4us
I20260709 12:24:00.222116 17654 leader_election.cc:304] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:24:00.222988 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:00.232677 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: true dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:00.227394 17684 raft_consensus.cc:2468] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 17.
I20260709 12:24:00.145478 25268 raft_consensus.cc:2749] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 12:24:00.248471 17833 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35040 (request call id 3121) took 1967 ms. Trace:
I20260709 12:24:00.248842 17683 raft_consensus.cc:2468] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 21.
I20260709 12:24:00.068866 25253 raft_consensus.cc:515] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:00.285048 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 21 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:24:00.230201 25254 log.cc:927] Time spent T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.082s	user 0.000s	sys 0.000s
I20260709 12:24:00.267743 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:00.273025 18367 raft_consensus.cc:3060] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:00.212925 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:00.273121 17802 leader_election.cc:304] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:24:00.274415 17835 raft_consensus.cc:1240] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 20 LEADER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 19. Current term is 20. Ops: [19.14-19.14]
I20260709 12:24:00.224925 25131 raft_consensus.cc:2749] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:24:00.226684 24960 raft_consensus.cc:2804] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:24:00.309059 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:00.283152 17728 leader_election.cc:304] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:00.309906 17683 raft_consensus.cc:2368] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 22. Current term is 23.
I20260709 12:23:59.960983 25206 raft_consensus.cc:515] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:00.284672 17833 rpcz_store.cc:276] 0709 12:23:58.280724 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:59.570782 (+1290058us) service_pool.cc:224] Handling call
0709 12:23:59.571065 (+   283us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:59.615396 (+ 44331us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:00.248217 (+632821us) spinlock_profiling.cc:243] Waited 43.4 ms on lock 0x7b54003d4ad4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:24:00.248407 (+   190us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":99760000}
I20260709 12:24:00.317613 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:00.317695 17728 leader_election.cc:304] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:00.317906 17829 raft_consensus.cc:1275] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
I20260709 12:24:00.317965 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:00.319278 17689 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 18.
I20260709 12:24:00.249686 25189 raft_consensus.cc:493] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:00.298643 17757 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38776 (request call id 2796) took 3481 ms. Trace:
I20260709 12:24:00.304648 17835 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37994 (request call id 1423) took 2019 ms. Trace:
I20260709 12:24:00.331702 17835 rpcz_store.cc:276] 0709 12:23:58.285549 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:00.273840 (+1988291us) service_pool.cc:224] Handling call
0709 12:24:00.274160 (+   320us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:00.304385 (+ 30225us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:00.304573 (+   188us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:24:00.310508 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 88490us
W20260709 12:24:00.313081 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 168637us
I20260709 12:24:00.333280 17729 leader_election.cc:304] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:24:00.318621 17829 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35042 (request call id 2820) took 1768 ms. Trace:
I20260709 12:24:00.334532 17829 rpcz_store.cc:276] 0709 12:23:58.550075 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:00.317285 (+1767210us) service_pool.cc:224] Handling call
0709 12:24:00.317609 (+   324us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:00.318360 (+   751us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:00.318559 (+   199us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:00.328399 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
W20260709 12:24:00.329376 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 47157.6us
I20260709 12:24:00.260594 25264 raft_consensus.cc:3060] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:00.271222 25248 raft_consensus.cc:740] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:24:00.241920 25147 consensus_queue.cc:1059] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 20.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.058s
I20260709 12:24:00.340930 17729 leader_election.cc:304] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:24:00.379130 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 61527.6us
I20260709 12:24:00.379719 17834 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35040 (request call id 3125) took 1984 ms. Trace:
I20260709 12:24:00.379933 17728 leader_election.cc:304] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:00.380118 17834 rpcz_store.cc:276] 0709 12:23:58.395283 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:00.316370 (+1921087us) service_pool.cc:224] Handling call
0709 12:24:00.379210 (+ 62840us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:00.379389 (+   179us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:00.379415 (+    26us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:00.379454 (+    39us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:24:00.379489 (+    35us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:00.379553 (+    64us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:00.379667 (+   114us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:00.250537 25140 raft_consensus.cc:515] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:00.355647 17831 raft_consensus.cc:2468] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 19.
I20260709 12:24:00.331717 17757 rpcz_store.cc:276] 0709 12:23:56.816849 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:56.868333 (+ 51484us) service_pool.cc:224] Handling call
0709 12:23:56.905975 (+ 37642us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 12:23:56.920487 (+ 14512us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:56.920531 (+    44us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 12:23:58.177811 (+1257280us) spinlock_profiling.cc:243] Waited 4.51 ms on lock 0x7b2c0000e5c8. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b48f19a11 00007f7b48f29372 00007f7b4a65ba01 00007f7b4a62043a 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2
0709 12:23:58.180490 (+  2679us) log.cc:844] Serialized 78 byte log entry
0709 12:23:58.226104 (+ 45614us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:23:58.246705 (+ 20601us) log.cc:844] Serialized 14 byte log entry
0709 12:23:59.861971 (+1615266us) spinlock_profiling.cc:243] Waited 7.51 ms on lock 0x7b180011bea0. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b475333f3 00007f7b475357c1 00007f7b47535779 00007f7b47535709 00007f7b475356d1 00007f7b475349cd 0000000000388634 0000000000388479 00007f7b4751683a
0709 12:24:00.075688 (+213717us) spinlock_profiling.cc:243] Waited 4.02 ms on lock 0x7b180011bea0. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b474adc0a 00007f7b50bed219 00007f7b47535a40 00007f7b475304e4 00007f7b47530455 00007f7b4e0183c7 00007f7b4a62a9a7 00007f7b4a63a0bf 00007f7b4a63a038
0709 12:24:00.075915 (+   227us) log.cc:844] Serialized 14 byte log entry
0709 12:24:00.298169 (+222254us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:00.298264 (+    95us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:00.298316 (+    52us) raft_consensus.cc:1685] finished
0709 12:24:00.298327 (+    11us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:00.298524 (+   197us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2002088,"prepare.queue_time_us":1901628,"prepare.run_cpu_time_us":113,"prepare.run_wall_time_us":1648,"raft.queue_time_us":66779,"raft.run_cpu_time_us":448,"raft.run_wall_time_us":30826,"spinlock_wait_cycles":37003648,"thread_start_us":229048,"threads_started":3,"wal-append.queue_time_us":69889,"wal-append.run_cpu_time_us":2990,"wal-append.run_wall_time_us":1010003}
I20260709 12:24:00.331041 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 19 candidate_status { last_received { term: 16 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:00.316637 24927 raft_consensus.cc:3055] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:24:00.329910 24723 raft_consensus.cc:2804] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Leader election won for term 23
W20260709 12:24:00.401916 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 61075.4us
I20260709 12:24:00.402218 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:00.402405 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:00.275341 24946 raft_consensus.cc:515] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:00.403331 17728 leader_election.cc:304] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:00.399300 17830 raft_consensus.cc:2393] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 18: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:24:00.378594 25164 raft_consensus.cc:2804] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:24:00.407555 25268 raft_consensus.cc:2749] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:24:00.408356 25284 raft_consensus.cc:3055] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 LEADER]: Stepping down as leader of term 21
W20260709 12:24:00.411692 25269 log.cc:927] Time spent T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.123s	user 0.000s	sys 0.000s
I20260709 12:24:00.317332 24960 raft_consensus.cc:493] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:00.340794 25053 raft_consensus.cc:2749] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:24:00.403488 17683 raft_consensus.cc:1240] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 22 LEADER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 21. Current term is 22. Ops: []
W20260709 12:24:00.412614 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 193734us
I20260709 12:24:00.341418 25189 raft_consensus.cc:3060] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:00.427223 18367 raft_consensus.cc:1275] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 21 index: 15. Preceding OpId from leader: term: 22 index: 16. (index mismatch)
I20260709 12:24:00.411773 25067 raft_consensus.cc:2804] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:24:00.455780 17683 raft_consensus.cc:1240] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 21. Current term is 22. Ops: [18.12-21.14]
I20260709 12:24:00.382994 25131 raft_consensus.cc:2804] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:24:00.464936 17689 raft_consensus.cc:1275] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 19 index: 15. Preceding OpId from leader: term: 21 index: 16. (index mismatch)
W20260709 12:24:00.438076 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:00.448165 24960 raft_consensus.cc:3060] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:00.398597 25211 raft_consensus.cc:2804] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:24:00.444778 17728 leader_election.cc:304] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
W20260709 12:24:00.444873 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:00.403709 25146 raft_consensus.cc:493] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:00.420450 24392 raft_consensus.cc:2749] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:24:00.436376 24723 raft_consensus.cc:697] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 23 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
W20260709 12:24:00.496537 25280 log.cc:927] Time spent T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.084s	user 0.000s	sys 0.000s
I20260709 12:24:00.496640 24723 consensus_queue.cc:237] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 21.13, Last appended by leader: 13, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:00.497884 25206 leader_election.cc:290] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:00.498287 17755 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38776 (request call id 2803) took 2897 ms. Trace:
I20260709 12:24:00.498677 17755 rpcz_store.cc:276] 0709 12:23:57.600779 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:59.485217 (+1884438us) service_pool.cc:224] Handling call
0709 12:23:59.485725 (+   508us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:59.486073 (+   348us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 12:24:00.211467 (+725394us) log.cc:844] Serialized 14 byte log entry
0709 12:24:00.256806 (+ 45339us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:00.258093 (+  1287us) log.cc:844] Serialized 556 byte log entry
0709 12:24:00.258294 (+   201us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:24:00.258336 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:00.258423 (+    87us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:00.498081 (+239658us) raft_consensus.cc:1685] finished
0709 12:24:00.498103 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:00.498228 (+   125us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":660284,"prepare.queue_time_us":830844,"raft.queue_time_us":64873,"raft.run_cpu_time_us":115,"raft.run_wall_time_us":40304,"spinlock_wait_cycles":142136192,"thread_start_us":97858,"threads_started":2,"wal-append.queue_time_us":99739}
I20260709 12:24:00.456111 17688 raft_consensus.cc:1275] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 14 index: 12. (index mismatch)
I20260709 12:24:00.403395 24927 raft_consensus.cc:740] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:24:00.501214 24927 consensus_queue.cc:260] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 22.14, Last appended by leader: 14, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:00.502368 24927 raft_consensus.cc:3060] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:00.429041 25214 raft_consensus.cc:2749] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:24:00.506038 25056 raft_consensus.cc:3060] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:00.474538 25268 raft_consensus.cc:2749] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:24:00.474893 25211 raft_consensus.cc:493] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:00.509375 25211 raft_consensus.cc:3060] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:00.454653 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 22 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:00.306200 25078 consensus_queue.cc:237] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.12, Last appended by leader: 12, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:00.470585 25126 raft_consensus.cc:3055] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:24:00.513315 25126 raft_consensus.cc:740] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:24:00.476266 17728 leader_election.cc:304] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:00.516149 17728 leader_election.cc:304] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:00.517987 25002 consensus_queue.cc:1048] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:24:00.477247 24801 leader_election.cc:290] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:00.477353 24642 leader_election.cc:290] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:24:00.477949 25271 log.cc:927] Time spent T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.079s	user 0.000s	sys 0.000s
I20260709 12:24:00.444260 25284 raft_consensus.cc:740] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:24:00.521422 25284 consensus_queue.cc:260] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 21.14, Last appended by leader: 14, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:00.523120 25284 raft_consensus.cc:3060] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:00.481810 25247 leader_election.cc:290] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:00.482203 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:00.482553 25253 leader_election.cc:290] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:00.443792 25147 raft_consensus.cc:493] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:00.444051 25164 raft_consensus.cc:697] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 20 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:24:00.452842 25292 consensus_queue.cc:1048] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.039s
W20260709 12:24:00.483628 25080 consensus_meta.cc:287] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.511s	user 0.007s	sys 0.000s
I20260709 12:24:00.484231 25252 leader_election.cc:290] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:00.455458 25067 raft_consensus.cc:493] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:24:00.485076 25142 leader_election.cc:290] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:00.485664 24891 leader_election.cc:290] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 23 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:24:00.485816 25270 log.cc:927] Time spent T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.105s	user 0.000s	sys 0.000s
I20260709 12:24:00.398001 24767 consensus_queue.cc:260] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 17.14, Last appended by leader: 14, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:00.489121 25165 leader_election.cc:290] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:00.489809 24992 raft_consensus.cc:2804] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Leader election won for term 22
W20260709 12:24:00.490195 24818 consensus_meta.cc:287] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.925s	user 0.009s	sys 0.005s
I20260709 12:24:00.490891 25146 raft_consensus.cc:515] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:00.491964 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 19 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:00.492722 17836 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35042 (request call id 2803) took 2571 ms. Trace:
I20260709 12:24:00.493654 25248 consensus_queue.cc:260] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 16.13, Last appended by leader: 13, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:00.524783 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 251570us
I20260709 12:24:00.589708 17836 rpcz_store.cc:276] 0709 12:23:57.921373 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:58.098289 (+176916us) service_pool.cc:224] Handling call
0709 12:23:58.134452 (+ 36163us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:58.159067 (+ 24615us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:58.159100 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:59.874515 (+1715415us) spinlock_profiling.cc:243] Waited 6.69 ms on lock 0x7b4c0041c840. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b488eca20 00007f7b488fda29 00007f7b488c3644 00007f7b488eaf2d 00007f7b488eab93 00007f7b4a61b18b 00007f7b4a61bcb5 00007f7b4a61d004 00007f7b4a61c729 00007f7b4a621d96 00007f7b4a616adb
0709 12:23:59.875311 (+   796us) log.cc:844] Serialized 499 byte log entry
0709 12:24:00.082788 (+207477us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:24:00.082846 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:00.082961 (+   115us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:00.492326 (+409365us) raft_consensus.cc:1685] finished
0709 12:24:00.492348 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:00.492656 (+   308us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":15385472,"thread_start_us":207132,"threads_started":1,"wal-append.queue_time_us":354391}
I20260709 12:24:00.559180 25067 raft_consensus.cc:3060] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:00.522356 17759 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38776 (request call id 2800) took 2957 ms. Trace:
I20260709 12:24:00.529704 17728 leader_election.cc:304] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:00.468348 25131 raft_consensus.cc:493] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:24:00.493193 25053 raft_consensus.cc:2749] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:24:00.557415 17837 raft_consensus.cc:1275] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 13 index: 12. Preceding OpId from leader: term: 14 index: 12. (term mismatch)
I20260709 12:24:00.602340 17837 pending_rounds.cc:85] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b: Aborting all ops after (but not including) 11
I20260709 12:24:00.602671 17837 pending_rounds.cc:107] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b: Aborting uncommitted NO_OP operation due to leader change: 13.12
I20260709 12:24:00.602977 17837 raft_consensus.cc:2889] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:24:00.573189 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 426222us
I20260709 12:24:00.539206 25214 raft_consensus.cc:2749] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:24:00.584447 18371 raft_consensus.cc:1240] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 18. Current term is 19. Ops: [18.16-18.16]
I20260709 12:24:00.542411 25247 raft_consensus.cc:3060] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:00.538373 25121 raft_consensus.cc:493] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:00.610495 25121 raft_consensus.cc:3060] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:00.611313 25290 raft_consensus.cc:493] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:00.611858 25290 raft_consensus.cc:515] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:00.613155 17654 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:00.524828 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 244530us
I20260709 12:24:00.621488 25252 consensus_queue.cc:1048] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.000s
I20260709 12:24:00.622936 25252 consensus_queue.cc:1048] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 12:24:00.614113 25290 leader_election.cc:290] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:00.595041 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:00.627707 25165 consensus_queue.cc:1059] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 23.16, Next index: 17, Last known committed idx: 15, Time since last communication: 0.000s
I20260709 12:24:00.511610 25076 raft_consensus.cc:515] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:00.597060 17759 rpcz_store.cc:276] 0709 12:23:57.565103 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:59.452421 (+1887318us) service_pool.cc:224] Handling call
0709 12:23:59.453167 (+   746us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 12:23:59.467907 (+ 14740us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:59.467939 (+    32us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 12:23:59.480992 (+ 13053us) log.cc:844] Serialized 78 byte log entry
0709 12:23:59.703820 (+222828us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:24:00.518880 (+815060us) log.cc:844] Serialized 14 byte log entry
0709 12:24:00.519941 (+  1061us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:00.520304 (+   363us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:00.520446 (+   142us) raft_consensus.cc:1685] finished
0709 12:24:00.520573 (+   127us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:00.522074 (+  1501us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":570001,"prepare.queue_time_us":703522,"prepare.run_cpu_time_us":32,"prepare.run_wall_time_us":34,"raft.queue_time_us":4703,"raft.run_cpu_time_us":144,"raft.run_wall_time_us":5237,"thread_start_us":356369,"threads_started":2,"wal-append.queue_time_us":401837}
I20260709 12:24:00.629720 17832 raft_consensus.cc:2393] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 23: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:24:00.513335 25179 consensus_queue.cc:260] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.16, Last appended by leader: 16, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:00.632031 17832 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37994 (request call id 1428) took 1022 ms. Trace:
I20260709 12:24:00.632354 25179 raft_consensus.cc:3060] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Advancing to term 19
W20260709 12:24:00.643204 17654 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:00.633153 18334 leader_election.cc:304] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:00.644575 25246 raft_consensus.cc:515] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:00.645889 18334 leader_election.cc:304] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:24:00.639760 25247 raft_consensus.cc:2749] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:24:00.649680 25146 leader_election.cc:290] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:00.642001 25076 leader_election.cc:290] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 23 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:00.651201 25140 leader_election.cc:290] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:00.602169 24992 raft_consensus.cc:697] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 22 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:24:00.632440 17832 rpcz_store.cc:276] 0709 12:23:59.609754 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:59.868184 (+258430us) service_pool.cc:224] Handling call
0709 12:24:00.631698 (+763514us) spinlock_profiling.cc:243] Waited 659 ms on lock 0x7b540008a2d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:24:00.631833 (+   135us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1516373760}
I20260709 12:24:00.652884 24992 consensus_queue.cc:237] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.13, Last appended by leader: 13, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:00.638870 25121 raft_consensus.cc:515] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:00.659549 25121 leader_election.cc:290] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 15 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:00.603775 24834 raft_consensus.cc:493] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:00.608970 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 79537.6us
I20260709 12:24:00.609588 18365 raft_consensus.cc:1275] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
I20260709 12:24:00.609823 25214 raft_consensus.cc:493] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:00.628805 25133 raft_consensus.cc:3055] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:24:00.557396 25126 consensus_queue.cc:260] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 18.15, Last appended by leader: 15, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:00.603533 25131 raft_consensus.cc:3060] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:00.545593 25147 raft_consensus.cc:515] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:00.640520 17838 raft_consensus.cc:1240] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 20. Current term is 21. Ops: []
I20260709 12:24:00.709015 17760 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38786 (request call id 3567) took 2750 ms. Trace:
I20260709 12:24:00.670250 17686 raft_consensus.cc:2468] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 17.
I20260709 12:24:00.703800 18334 leader_election.cc:400] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 22 election: Vote denied by peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071) with higher term. Message: Invalid argument: T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 22. Current term is 23.
I20260709 12:24:00.643371 25189 raft_consensus.cc:515] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:24:00.715945 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 378936us
W20260709 12:24:00.682287 17654 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:00.707975 17727 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:00.721458 18334 leader_election.cc:403] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 22 election: Cancelling election due to peer responding with higher term
I20260709 12:24:00.698716 25140 raft_consensus.cc:2804] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:24:00.704535 24767 raft_consensus.cc:3060] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:24:00.730209 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 63306.9us
I20260709 12:24:00.731003 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:00.731990 17832 raft_consensus.cc:2393] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 15: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:24:00.732474 17680 raft_consensus.cc:1240] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 20. Current term is 21. Ops: []
I20260709 12:24:00.732945 17727 leader_election.cc:304] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:00.647328 25246 leader_election.cc:290] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 17 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:00.733060 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:00.733814 18333 leader_election.cc:304] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:00.713301 25263 raft_consensus.cc:2749] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:24:00.712940 25133 raft_consensus.cc:740] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:24:00.720588 17756 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38776 (request call id 2810) took 2814 ms. Trace:
I20260709 12:24:00.737079 17761 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38776 (request call id 2809) took 2835 ms. Trace:
I20260709 12:24:00.671231 25165 raft_consensus.cc:2749] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:24:00.741690 17761 rpcz_store.cc:276] 0709 12:23:57.906107 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:59.528104 (+1621997us) service_pool.cc:224] Handling call
0709 12:23:59.569100 (+ 40996us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:59.569309 (+   209us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 12:24:00.524997 (+955688us) log.cc:844] Serialized 14 byte log entry
0709 12:24:00.733737 (+208740us) log.cc:844] Serialized 14 byte log entry
0709 12:24:00.734240 (+   503us) log.cc:844] Serialized 14 byte log entry
0709 12:24:00.736695 (+  2455us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:00.736754 (+    59us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:24:00.736813 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:00.736886 (+    73us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:00.737014 (+   128us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":641266,"prepare.queue_time_us":832732,"prepare.run_cpu_time_us":63,"prepare.run_wall_time_us":62,"raft.queue_time_us":5608,"raft.run_cpu_time_us":385,"raft.run_wall_time_us":5297,"thread_start_us":462920,"threads_started":2}
I20260709 12:24:00.723235 25076 raft_consensus.cc:493] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:00.723331 25278 raft_consensus.cc:493] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:00.720148 25283 raft_consensus.cc:2749] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:24:00.744694 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:00.744769 17687 raft_consensus.cc:1275] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 17 index: 14. Preceding OpId from leader: term: 19 index: 15. (index mismatch)
I20260709 12:24:00.758250 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:00.745508 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:00.747433 17680 raft_consensus.cc:1240] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 18. Current term is 19. Ops: []
I20260709 12:24:00.749629 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 21 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:00.740945 17756 rpcz_store.cc:276] 0709 12:23:57.906455 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:59.536532 (+1630077us) service_pool.cc:224] Handling call
0709 12:23:59.536804 (+   272us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:59.536986 (+   182us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 12:24:00.251680 (+714694us) log.cc:844] Serialized 14 byte log entry
0709 12:24:00.716489 (+464809us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:00.716540 (+    51us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:24:00.716583 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:00.719728 (+  3145us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:00.719850 (+   122us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":669082,"prepare.queue_time_us":837089,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":42,"raft.queue_time_us":82637,"raft.run_cpu_time_us":83,"raft.run_wall_time_us":83,"thread_start_us":383083,"threads_started":3,"wal-append.queue_time_us":226385}
I20260709 12:24:00.713827 25210 consensus_queue.cc:1059] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 19.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.023s
I20260709 12:24:00.715693 24834 raft_consensus.cc:515] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:00.735258 25142 raft_consensus.cc:2749] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Leader election lost for term 22. Reason: Vote denied by peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071) with higher term. Message: Invalid argument: T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 22. Current term is 23.
I20260709 12:24:00.730636 25140 raft_consensus.cc:493] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:00.754760 18333 leader_election.cc:304] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:24:00.715538 25214 raft_consensus.cc:515] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:00.771608 17682 raft_consensus.cc:1240] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 17 LEADER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 16. Current term is 17. Ops: []
W20260709 12:24:00.777000 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 159444us
I20260709 12:24:00.632956 25164 consensus_queue.cc:237] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 19.14, Last appended by leader: 14, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:00.813081 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 58396.8us
I20260709 12:24:00.804880 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 17 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:00.763676 25146 raft_consensus.cc:493] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:00.744871 25053 consensus_queue.cc:1059] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 20.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.063s
I20260709 12:24:00.804651 25189 leader_election.cc:290] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 19 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:00.742990 25293 consensus_queue.cc:1059] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 15.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.076s
I20260709 12:24:00.808465 17801 leader_election.cc:304] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:24:00.832624 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 63566.4us
I20260709 12:24:00.816040 18333 leader_election.cc:304] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:00.743824 24677 consensus_queue.cc:1048] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.066s
I20260709 12:24:00.763273 24790 consensus_queue.cc:1048] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.052s
W20260709 12:24:00.804520 24987 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.585s	user 0.009s	sys 0.012s
I20260709 12:24:00.773371 25294 raft_consensus.cc:2749] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:24:00.771937 25078 raft_consensus.cc:2749] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:24:00.763711 25076 raft_consensus.cc:515] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:00.837389 17801 leader_election.cc:304] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:00.811088 25140 raft_consensus.cc:3060] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:00.772236 25267 tablet_copy_client.cc:323] T a5732e71b9d54f8786b8c511d69943bf P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Beginning tablet copy session from remote peer at address 127.16.29.66:39651
I20260709 12:24:00.770501 25278 raft_consensus.cc:515] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:00.841218 17727 leader_election.cc:304] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
W20260709 12:24:00.842854 24987 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.632s	user 0.010s	sys 0.013s
I20260709 12:24:00.810647 24989 consensus_queue.cc:1059] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.028s
I20260709 12:24:00.828415 25142 raft_consensus.cc:2804] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:24:00.851130 17801 leader_election.cc:304] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:00.845263 25132 raft_consensus.cc:2804] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:24:00.832924 25297 raft_consensus.cc:493] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:24:00.805714 25165 consensus_queue.cc:1048] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.036s
I20260709 12:24:00.842999 18333 leader_election.cc:304] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:00.857357 25268 raft_consensus.cc:2749] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:24:00.854962 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:00.842135 25189 raft_consensus.cc:2804] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:24:00.865218 25210 consensus_queue.cc:1059] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 20.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.003s
I20260709 12:24:00.868920 25278 leader_election.cc:290] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:00.873654 25204 raft_consensus.cc:3055] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:24:00.883108 17837 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37994 (request call id 1424) took 2514 ms. Trace:
I20260709 12:24:00.971129 17837 rpcz_store.cc:276] 0709 12:23:58.368575 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:00.312267 (+1943692us) service_pool.cc:224] Handling call
0709 12:24:00.378637 (+ 66370us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:00.603543 (+224906us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:00.882660 (+279117us) spinlock_profiling.cc:243] Waited 178 ms on lock 0x7b54000ab9d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:24:00.883005 (+   345us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":410205952}
I20260709 12:24:00.884536 25147 leader_election.cc:290] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:00.829463 25133 consensus_queue.cc:260] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 23.16, Last appended by leader: 16, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:00.901551 25211 raft_consensus.cc:515] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:24:00.902617 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:00.979926 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:00.980448 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:00.980963 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:00.974120 24818 raft_consensus.cc:493] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:24:00.981585 24818 raft_consensus.cc:515] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:00.879541 24946 leader_election.cc:290] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:00.974949 25292 raft_consensus.cc:2749] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:24:00.966197 25142 raft_consensus.cc:697] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 18 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:24:00.989161 25142 consensus_queue.cc:237] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 17.14, Last appended by leader: 14, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:00.989769 25214 leader_election.cc:290] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:00.991272 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 23 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:00.983796 24818 leader_election.cc:290] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:00.992326 17759 raft_consensus.cc:3060] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:00.993610 17832 raft_consensus.cc:1240] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 18. Current term is 19. Ops: [16.15-18.16]
I20260709 12:24:00.996913 17834 raft_consensus.cc:1240] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 18. Current term is 19. Ops: [18.15-18.15]
I20260709 12:24:00.998342 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:00.998948 17831 raft_consensus.cc:3060] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:00.715785 17760 rpcz_store.cc:276] 0709 12:23:57.958309 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:59.577445 (+1619136us) service_pool.cc:224] Handling call
0709 12:23:59.608954 (+ 31509us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:59.609172 (+   218us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 12:24:00.432189 (+823017us) log.cc:844] Serialized 14 byte log entry
0709 12:24:00.708468 (+276279us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:00.708596 (+   128us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:24:00.708676 (+    80us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:00.708784 (+   108us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:00.708945 (+   161us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":631145,"prepare.queue_time_us":797940,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":41,"raft.queue_time_us":4741,"raft.run_cpu_time_us":127,"raft.run_wall_time_us":5353,"thread_start_us":437993,"threads_started":2}
I20260709 12:24:01.001739 25165 raft_consensus.cc:493] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:01.002297 25165 raft_consensus.cc:515] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:01.003171 24960 raft_consensus.cc:515] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Starting leader election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:01.004269 17761 raft_consensus.cc:1275] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 24 index: 18. Preceding OpId from leader: term: 25 index: 19. (index mismatch)
I20260709 12:24:01.004443 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:00.910009 17802 leader_election.cc:304] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:24:01.005070 25076 leader_election.cc:290] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:24:01.005254 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 95869us
I20260709 12:24:01.005486 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 22 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:01.005839 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:01.006453 17764 raft_consensus.cc:2468] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 21.
I20260709 12:24:00.903666 17729 leader_election.cc:304] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:01.007529 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 19 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:01.008620 25211 leader_election.cc:290] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 22 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:01.008311 17755 raft_consensus.cc:3060] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:00.843231 25293 raft_consensus.cc:3055] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:24:01.009673 25211 consensus_queue.cc:1059] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 22.13, Next index: 14, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:24:01.010236 25293 raft_consensus.cc:740] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:24:01.012177 25298 raft_consensus.cc:3055] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:24:01.012401 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:01.012719 25298 raft_consensus.cc:740] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 20 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:24:00.976788 25285 raft_consensus.cc:493] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:01.014046 25285 raft_consensus.cc:515] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:01.014129 25298 consensus_queue.cc:260] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.13, Last appended by leader: 13, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:24:00.928673 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 77663.1us
I20260709 12:24:01.015591 17801 leader_election.cc:304] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:00.929589 25132 raft_consensus.cc:697] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 17 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:24:01.015859 17802 leader_election.cc:304] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:00.831295 25279 consensus_queue.cc:1059] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 16.13, Next index: 14, Last known committed idx: 12, Time since last communication: 0.078s
I20260709 12:24:01.016767 25279 consensus_queue.cc:1243] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Peer 0da169ff82674dc1947055ce43de68a3 log is divergent from this leader: its last log entry 17.14 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 12
I20260709 12:24:00.845649 25053 raft_consensus.cc:493] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:01.017016 25132 consensus_queue.cc:237] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.12, Last appended by leader: 12, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:01.017468 25053 raft_consensus.cc:515] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:00.928565 24989 raft_consensus.cc:3055] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:24:01.019260 24989 raft_consensus.cc:740] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:24:00.828855 25146 raft_consensus.cc:515] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:01.020226 24989 consensus_queue.cc:260] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.14, Last appended by leader: 14, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:00.841976 25136 consensus_queue.cc:1048] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.078s
I20260709 12:24:01.021312 24989 raft_consensus.cc:3060] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:00.918669 24834 leader_election.cc:290] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:00.974905 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:24:00.955509 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 114289us
I20260709 12:24:01.022547 17683 raft_consensus.cc:2468] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 15.
I20260709 12:24:00.930306 25297 raft_consensus.cc:515] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:01.004597 18366 raft_consensus.cc:1240] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 19. Current term is 20. Ops: []
I20260709 12:24:01.023165 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:01.023454 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:00.970245 25204 raft_consensus.cc:740] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:24:00.976611 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:00.953903 25308 raft_consensus.cc:2749] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:24:01.024771 17687 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 18.
I20260709 12:24:01.014346 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:01.026245 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 18 candidate_status { last_received { term: 16 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:01.023193 18333 leader_election.cc:304] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:24:01.027072 17686 raft_consensus.cc:2468] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 17.
I20260709 12:24:01.024478 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 20 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:01.028614 17686 raft_consensus.cc:1240] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 22. Current term is 23. Ops: [22.12-22.12]
I20260709 12:24:01.028788 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:01.029989 17829 raft_consensus.cc:2393] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 23: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:24:01.030452 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:00.966328 17682 raft_consensus.cc:2468] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 23.
I20260709 12:24:01.031406 18362 raft_consensus.cc:2393] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 23: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:24:01.024472 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:01.034336 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:01.034914 17728 leader_election.cc:304] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:24:01.036718 17654 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:00.853838 24992 raft_consensus.cc:2749] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:24:01.038383 17764 raft_consensus.cc:1275] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 20 index: 11. Preceding OpId from leader: term: 21 index: 12. (index mismatch)
I20260709 12:24:01.039681 18333 leader_election.cc:304] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:24:01.040771 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:01.041096 17801 leader_election.cc:304] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:01.048694 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:00.968190 25189 raft_consensus.cc:493] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:24:01.043470 17652 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:01.056334 18333 leader_election.cc:304] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
W20260709 12:24:01.056975 17652 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:01.045526 17831 raft_consensus.cc:2410] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for term 20 because replica has last-logged OpId of term: 19 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 11.
I20260709 12:24:01.054428 17802 leader_election.cc:304] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:01.046625 25298 raft_consensus.cc:3060] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:01.058048 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 20 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:01.058943 17834 raft_consensus.cc:1275] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 20 index: 11. Preceding OpId from leader: term: 21 index: 12. (index mismatch)
I20260709 12:24:01.059988 17757 raft_consensus.cc:1275] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 17 index: 11. Preceding OpId from leader: term: 21 index: 13. (index mismatch)
I20260709 12:24:00.937860 25002 raft_consensus.cc:493] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:24:01.055645 17829 raft_consensus.cc:1240] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 22. Current term is 23. Ops: []
I20260709 12:24:01.061913 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 23 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:24:01.063581 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 160006us
I20260709 12:24:01.064162 18333 leader_election.cc:304] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:01.068889 17729 leader_election.cc:304] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
W20260709 12:24:01.064317 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 206725us
I20260709 12:24:01.069892 17836 raft_consensus.cc:1275] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 16 index: 10. Preceding OpId from leader: term: 20 index: 12. (index mismatch)
I20260709 12:24:01.070114 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:24:01.067842 17652 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:01.065109 17801 leader_election.cc:304] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:01.064498 17761 raft_consensus.cc:1275] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 18 index: 13. Preceding OpId from leader: term: 22 index: 16. (index mismatch)
I20260709 12:24:01.067551 17802 leader_election.cc:304] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:24:01.073025 17652 connection.cc:256] RPC call timeout handler was delayed by -1.1079309580004519s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17700000000000002s for a response
W20260709 12:24:01.073478 17652 connection.cc:256] RPC call timeout handler was delayed by -1.1079309580004519s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17700000000000002s for a response
I20260709 12:24:01.073618 17682 raft_consensus.cc:2393] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 23: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:24:01.005667 25076 raft_consensus.cc:2804] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:24:01.070258 17727 leader_election.cc:304] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
W20260709 12:24:01.073947 17652 connection.cc:256] RPC call timeout handler was delayed by -1.0611309580003763s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18220000000000003s for a response
W20260709 12:24:01.078251 17652 connection.cc:256] RPC call timeout handler was delayed by -1.0611309580003763s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18220000000000003s for a response
I20260709 12:24:01.078572 17802 leader_election.cc:304] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:24:01.080894 17802 leader_election.cc:304] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:24:01.081625 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 23 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:01.088943 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:01.054306 25099 raft_consensus.cc:2804] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Leader pre-election won for term 19
W20260709 12:24:01.104701 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 69908.7us
I20260709 12:24:01.091706 17803 leader_election.cc:304] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:01.089959 17764 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 17.
I20260709 12:24:01.079221 17652 leader_election.cc:304] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:01.059827 25189 raft_consensus.cc:3060] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:01.080677 24992 raft_consensus.cc:2749] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:24:01.095562 25165 leader_election.cc:290] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:01.059291 24677 raft_consensus.cc:2804] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:24:01.097477 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:01.024541 25204 consensus_queue.cc:260] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 17.11, Last appended by leader: 11, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:01.064615 24834 raft_consensus.cc:2749] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:24:01.080348 25002 raft_consensus.cc:515] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:01.102737 25146 leader_election.cc:290] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:01.295614 18366 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 2192) took 1076 ms. Trace:
W20260709 12:24:01.123202 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 54380.3us
I20260709 12:24:01.297894 17729 leader_election.cc:304] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:01.068409 25279 raft_consensus.cc:2804] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:24:01.114666 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 18 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:01.314028 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:01.145519 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:01.108049 25099 raft_consensus.cc:493] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:01.011368 25293 consensus_queue.cc:260] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.12, Last appended by leader: 12, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:01.239655 17687 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 18.
I20260709 12:24:01.152196 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 23 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:01.068830 25278 raft_consensus.cc:2749] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:24:01.074636 25308 raft_consensus.cc:2804] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Leader election won for term 21
W20260709 12:24:01.240145 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 148573us
I20260709 12:24:01.109452 25076 raft_consensus.cc:697] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 15 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:24:01.346995 18333 leader_election.cc:304] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:01.365499 17803 leader_election.cc:304] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:01.150926 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 18 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:01.167294 17755 raft_consensus.cc:2468] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 19.
I20260709 12:24:01.074877 24987 raft_consensus.cc:2749] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:24:01.243086 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:01.244393 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:01.075281 25137 raft_consensus.cc:2749] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:24:01.156129 17759 raft_consensus.cc:2468] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 23.
I20260709 12:24:01.149439 25247 raft_consensus.cc:2749] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:24:01.249500 24998 raft_consensus.cc:3055] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:24:01.254990 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 18 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:01.378614 17728 leader_election.cc:304] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:01.397766 18368 raft_consensus.cc:2468] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 17.
I20260709 12:24:01.398984 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:01.262035 24818 raft_consensus.cc:2804] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:24:01.400384 24818 raft_consensus.cc:697] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 23 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:24:01.401561 24818 consensus_queue.cc:237] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 21.15, Last appended by leader: 15, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:24:01.266291 17654 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:01.277524 17830 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35042 (request call id 2823) took 1635 ms (client timeout 1784 ms). Trace:
W20260709 12:24:01.405735 17830 rpcz_store.cc:269] 0709 12:23:59.612230 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:59.702897 (+ 90667us) service_pool.cc:224] Handling call
0709 12:24:01.247587 (+1544690us) spinlock_profiling.cc:243] Waited 485 ms on lock 0x7b540022ae54. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:24:01.247813 (+   226us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1116577024}
W20260709 12:24:01.407666 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:01.408646 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:01.409770 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:00.601222 16501 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:24:01.293956 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:01.297930 18363 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 2188) took 1217 ms. Trace:
I20260709 12:24:01.411738 18363 rpcz_store.cc:276] 0709 12:24:00.080981 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:00.146145 (+ 65164us) service_pool.cc:224] Handling call
0709 12:24:00.146469 (+   324us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:00.339850 (+193381us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:00.339914 (+    64us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:00.339960 (+    46us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:24:00.340003 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:01.297617 (+957614us) spinlock_profiling.cc:243] Waited 193 ms on lock 0x7b540034b554. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:24:01.297681 (+    64us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:01.297852 (+   171us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":444251904}
I20260709 12:24:01.104240 25080 raft_consensus.cc:2804] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:24:01.417114 25080 raft_consensus.cc:493] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:24:01.417829 25080 raft_consensus.cc:3060] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:01.308745 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:01.296260 18366 rpcz_store.cc:276] 0709 12:24:00.219222 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:00.254690 (+ 35468us) service_pool.cc:224] Handling call
0709 12:24:00.255196 (+   506us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:01.023472 (+768276us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:01.295282 (+271810us) spinlock_profiling.cc:243] Waited 749 ms on lock 0x7b5400acf3d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:24:01.295530 (+   248us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1723081600}
I20260709 12:24:01.311005 17834 raft_consensus.cc:1240] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 19. Current term is 20. Ops: []
I20260709 12:24:01.420840 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:01.103329 25053 leader_election.cc:290] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:01.104727 25252 raft_consensus.cc:2804] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:24:01.421873 18366 raft_consensus.cc:2468] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 16.
I20260709 12:24:01.422333 25252 raft_consensus.cc:493] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:01.423120 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 20 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:24:01.242358 25264 consensus_meta.cc:287] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.846s	user 0.014s	sys 0.000s
I20260709 12:24:01.424329 25264 raft_consensus.cc:515] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:01.146368 25056 raft_consensus.cc:2749] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
W20260709 12:24:01.338975 25131 consensus_meta.cc:287] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.618s	user 0.008s	sys 0.002s
I20260709 12:24:01.428777 25131 raft_consensus.cc:515] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:01.430058 18366 raft_consensus.cc:2410] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 20 because replica has last-logged OpId of term: 18 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 14.
I20260709 12:24:01.430678 17764 raft_consensus.cc:1240] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 20. Current term is 21. Ops: []
I20260709 12:24:01.341513 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:01.431228 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 18 candidate_status { last_received { term: 16 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:01.432564 18371 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 18.
I20260709 12:24:01.342403 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:01.317040 24801 raft_consensus.cc:2749] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
W20260709 12:24:01.324676 24767 consensus_meta.cc:287] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.588s	user 0.010s	sys 0.003s
I20260709 12:24:01.377152 18333 leader_election.cc:304] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:24:01.428769 17762 raft_consensus.cc:1240] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 22. Current term is 23. Ops: []
I20260709 12:24:01.349850 17729 leader_election.cc:304] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
W20260709 12:24:01.439939 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 90187.1us
I20260709 12:24:01.440979 17729 leader_election.cc:304] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:01.390018 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:01.108184 25290 consensus_queue.cc:1059] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 19.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.224s
I20260709 12:24:01.442114 18365 raft_consensus.cc:2468] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 18.
I20260709 12:24:01.103590 24790 raft_consensus.cc:2749] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:24:01.444868 25053 raft_consensus.cc:2749] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:24:01.447247 25053 raft_consensus.cc:2749] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 12:24:01.146114 25284 raft_consensus.cc:2804] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:24:01.450680 25284 raft_consensus.cc:493] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:01.451272 25284 raft_consensus.cc:3060] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:01.329221 17757 raft_consensus.cc:2393] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 18: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:24:01.428946 25264 leader_election.cc:290] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 23 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:01.104503 25294 consensus_queue.cc:1048] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.255s
I20260709 12:24:01.308766 24677 raft_consensus.cc:697] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 18 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
W20260709 12:24:01.458954 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 93562.4us
I20260709 12:24:01.335047 25099 raft_consensus.cc:3060] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Advancing to term 19
W20260709 12:24:01.461174 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 82677.6us
I20260709 12:24:01.422904 25252 raft_consensus.cc:3060] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:24:01.460479 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 83437.6us
I20260709 12:24:01.399709 18368 raft_consensus.cc:3060] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:01.334836 25309 raft_consensus.cc:2804] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:24:01.415441 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:24:01.373869 25286 log.cc:927] Time spent T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.079s	user 0.001s	sys 0.000s
W20260709 12:24:01.463039 25067 consensus_meta.cc:287] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.856s	user 0.012s	sys 0.000s
W20260709 12:24:01.479837 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:01.486943 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:01.489080 25076 consensus_queue.cc:237] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 14.15, Last appended by leader: 15, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:01.374006 25279 raft_consensus.cc:697] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 22 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:24:01.465914 25293 raft_consensus.cc:3060] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:01.494048 25002 leader_election.cc:290] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:01.397406 24998 raft_consensus.cc:740] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 22 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:24:01.396174 25275 raft_consensus.cc:2749] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:24:01.391978 25147 raft_consensus.cc:493] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:01.371028 25315 raft_consensus.cc:2749] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:24:01.498512 18333 leader_election.cc:400] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 22 election: Vote denied by peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837) with higher term. Message: Invalid argument: T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 22. Current term is 23.
I20260709 12:24:01.535072 17685 raft_consensus.cc:1275] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 16 index: 11. Preceding OpId from leader: term: 22 index: 14. (index mismatch)
W20260709 12:24:01.535738 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 51957us
I20260709 12:24:01.370859 25263 raft_consensus.cc:493] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:01.491165 24677 consensus_queue.cc:237] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.11, Last appended by leader: 11, Current term: 18, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:01.508147 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 137050us
I20260709 12:24:01.466732 24642 consensus_queue.cc:1059] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 14.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.051s
W20260709 12:24:01.511168 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 432004us
I20260709 12:24:01.554219 17652 leader_election.cc:304] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:01.555927 24987 consensus_queue.cc:1048] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 12:24:01.556429 17831 raft_consensus.cc:1275] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 16 index: 13. (index mismatch)
I20260709 12:24:01.513506 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 22 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:01.558050 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:01.560268 25002 raft_consensus.cc:2804] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:24:01.560833 25002 raft_consensus.cc:493] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:24:01.515411 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 18 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:01.507153 25131 leader_election.cc:290] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 21 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:01.477141 24790 raft_consensus.cc:2804] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:24:01.475544 25137 raft_consensus.cc:2749] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:24:01.482574 25136 raft_consensus.cc:493] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:01.371757 25210 consensus_queue.cc:1048] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.136s
I20260709 12:24:01.527503 25309 raft_consensus.cc:697] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 21 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:24:01.536818 18333 leader_election.cc:304] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:01.546140 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:01.606052 17761 raft_consensus.cc:2468] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 17.
I20260709 12:24:01.508711 25081 raft_consensus.cc:493] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:01.608614 17831 raft_consensus.cc:1240] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 20 LEADER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 19. Current term is 20. Ops: []
W20260709 12:24:01.609444 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 72756.2us
W20260709 12:24:01.567513 25140 consensus_meta.cc:287] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.691s	user 0.010s	sys 0.002s
I20260709 12:24:01.465234 25308 raft_consensus.cc:697] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 21 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:24:01.567399 17682 raft_consensus.cc:1240] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 19. Current term is 20. Ops: [19.14-19.14]
I20260709 12:24:01.560190 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 21 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:01.515018 25067 raft_consensus.cc:515] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:01.584704 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:01.586449 24790 raft_consensus.cc:697] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 19 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:24:01.587529 17652 leader_election.cc:304] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:24:01.644981 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 57530us
I20260709 12:24:01.505715 25080 raft_consensus.cc:515] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:01.588588 17774 tablet_copy_service.cc:140] P e2909cff1da44ad387f96a8ae3aa5afb: Received BeginTabletCopySession request for tablet a5732e71b9d54f8786b8c511d69943bf from peer 334a4e1f59944544b3b3ff36616b57da ({username='slave'} at 127.0.0.1:36572)
I20260709 12:24:01.589665 25181 consensus_queue.cc:1059] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:24:01.589608 25131 consensus_queue.cc:1048] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:24:01.595803 25136 raft_consensus.cc:515] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:01.596393 17760 raft_consensus.cc:1240] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 22. Current term is 23. Ops: [22.12-22.12]
I20260709 12:24:01.597986 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 21 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:01.503757 25248 raft_consensus.cc:3060] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:01.561193 25002 raft_consensus.cc:3060] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:01.610935 18333 leader_election.cc:304] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:24:01.611546 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:01.562937 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:01.614568 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 21 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:01.612282 24992 raft_consensus.cc:493] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:01.618029 25298 consensus_meta.cc:287] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.560s	user 0.011s	sys 0.000s
I20260709 12:24:01.617806 25279 consensus_queue.cc:237] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 20.14, Last appended by leader: 14, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:01.618808 18368 raft_consensus.cc:2468] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 23.
I20260709 12:24:01.620344 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:01.620793 25081 raft_consensus.cc:515] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:01.545634 25147 raft_consensus.cc:515] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:01.621734 24723 raft_consensus.cc:2955] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 19 LEADER]: Committing config change with OpId 19.12: config changed from index 5 to 12, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:24:01.627154 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:01.636490 17687 raft_consensus.cc:1240] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 22. Current term is 23. Ops: [22.12-22.12]
I20260709 12:24:01.651357 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 18 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:01.647147 17652 leader_election.cc:304] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:01.604328 24801 raft_consensus.cc:493] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:01.654363 24767 raft_consensus.cc:2749] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
W20260709 12:24:01.662217 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 51365.3us
I20260709 12:24:01.689394 18333 leader_election.cc:304] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:01.563578 25263 raft_consensus.cc:515] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:01.662118 17755 raft_consensus.cc:3060] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:01.672192 17833 raft_consensus.cc:1240] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 15. Current term is 16. Ops: []
I20260709 12:24:01.576948 25264 consensus_queue.cc:1059] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true }, Status: INVALID_TERM, Last received: 19.12, Next index: 13, Last known committed idx: 11, Time since last communication: 0.080s
I20260709 12:24:01.670214 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:01.791729 17757 raft_consensus.cc:3060] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:01.653522 25211 raft_consensus.cc:2804] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Leader pre-election won for term 17
W20260709 12:24:01.662088 18334 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.518s	user 0.004s	sys 0.016s
W20260709 12:24:01.795320 18334 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.651s	user 0.005s	sys 0.017s
W20260709 12:24:01.793313 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 101890us
W20260709 12:24:01.793596 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 146554us
I20260709 12:24:01.797932 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:01.685945 17774 tablet_copy_service.cc:161] P e2909cff1da44ad387f96a8ae3aa5afb: Beginning new tablet copy session on tablet a5732e71b9d54f8786b8c511d69943bf from peer 334a4e1f59944544b3b3ff36616b57da at {username='slave'} at 127.0.0.1:36572: session id = 334a4e1f59944544b3b3ff36616b57da-a5732e71b9d54f8786b8c511d69943bf
I20260709 12:24:01.798758 17835 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 18.
I20260709 12:24:01.668241 25168 raft_consensus.cc:2804] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Leader election won for term 19
W20260709 12:24:01.805510 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 119501us
W20260709 12:24:01.807722 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:01.837994 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:01.635249 25140 raft_consensus.cc:515] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:01.817464 18362 raft_consensus.cc:2468] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 18.
I20260709 12:24:01.794721 25132 raft_consensus.cc:2749] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:24:01.820108 24790 consensus_queue.cc:237] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.15, Last appended by leader: 15, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:01.820780 25319 log.cc:927] Time spent T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.168s	user 0.000s	sys 0.000s
I20260709 12:24:01.822098 25080 leader_election.cc:290] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 16 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:01.822331 25136 leader_election.cc:290] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:01.869967 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:01.829983 25067 leader_election.cc:290] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 22 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:01.832742 17757 raft_consensus.cc:2468] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 20.
I20260709 12:24:01.841117 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:01.807806 17803 leader_election.cc:304] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:01.843585 17759 raft_consensus.cc:3060] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:01.848097 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:01.853704 18366 raft_consensus.cc:2468] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 17.
I20260709 12:24:01.796295 25181 raft_consensus.cc:2749] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:24:01.893649 18367 raft_consensus.cc:2468] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 21.
I20260709 12:24:01.615417 24998 consensus_queue.cc:260] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 22.12, Last appended by leader: 12, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:01.892488 18371 raft_consensus.cc:3060] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:01.892679 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:01.903836 17757 raft_consensus.cc:2468] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 17.
I20260709 12:24:01.905310 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:01.906814 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:01.897363 17729 leader_election.cc:304] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:01.693061 25206 consensus_queue.cc:1059] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 17.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.176s
I20260709 12:24:01.909513 17757 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 17.
I20260709 12:24:01.679628 24992 raft_consensus.cc:515] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:01.867226 25322 log.cc:927] Time spent T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.202s	user 0.001s	sys 0.000s
W20260709 12:24:01.866744 25303 log.cc:927] Time spent T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.078s	user 0.001s	sys 0.000s
I20260709 12:24:01.861302 24987 raft_consensus.cc:493] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:01.796247 25182 consensus_queue.cc:1048] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.096s
W20260709 12:24:01.916282 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 108598us
I20260709 12:24:01.787923 24801 raft_consensus.cc:515] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:01.805066 25165 consensus_queue.cc:1059] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 19.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.092s
I20260709 12:24:01.943449 17803 leader_election.cc:304] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:24:01.937431 17757 raft_consensus.cc:1240] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 23. Current term is 24. Ops: []
I20260709 12:24:01.946686 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:01.885157 17755 raft_consensus.cc:2468] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 22.
I20260709 12:24:01.958384 17757 raft_consensus.cc:1240] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 17. Current term is 18. Ops: [17.12-17.12]
I20260709 12:24:01.957540 17759 raft_consensus.cc:2468] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 19.
I20260709 12:24:01.961233 17757 raft_consensus.cc:1275] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 21 index: 14. Preceding OpId from leader: term: 22 index: 14. (term mismatch)
I20260709 12:24:01.962411 17832 raft_consensus.cc:2468] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 20.
I20260709 12:24:01.951397 17803 leader_election.cc:304] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:24:01.861869 24767 consensus_queue.cc:1048] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.068s
I20260709 12:24:01.860293 25179 consensus_queue.cc:1059] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.069s
I20260709 12:24:01.969111 17833 raft_consensus.cc:2393] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 18: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
W20260709 12:24:01.965487 25284 consensus_meta.cc:287] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.513s	user 0.012s	sys 0.000s
I20260709 12:24:01.901304 25292 raft_consensus.cc:2749] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:24:01.868410 25323 consensus_queue.cc:1059] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 12.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.111s
I20260709 12:24:01.979734 17803 leader_election.cc:304] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
W20260709 12:24:02.014202 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 149720us
I20260709 12:24:01.975867 17757 pending_rounds.cc:85] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb: Aborting all ops after (but not including) 13
I20260709 12:24:01.933691 25131 raft_consensus.cc:2804] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:24:01.972357 25080 raft_consensus.cc:2804] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:24:01.933459 25329 raft_consensus.cc:2749] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 12:24:02.027611 17757 pending_rounds.cc:107] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb: Aborting uncommitted NO_OP operation due to leader change: 21.14
I20260709 12:24:02.029436 17757 raft_consensus.cc:2889] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:01.930804 25326 consensus_queue.cc:1059] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 19.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.102s
W20260709 12:24:02.032437 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 52800.6us
I20260709 12:24:02.033914 17803 leader_election.cc:304] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:24:02.036273 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 23 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:02.037593 17833 raft_consensus.cc:2468] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 22.
I20260709 12:24:01.949033 25181 consensus_queue.cc:1059] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 15.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.071s
I20260709 12:24:02.046167 17803 leader_election.cc:304] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:02.045440 18333 leader_election.cc:304] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:01.863338 25309 consensus_queue.cc:237] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 20.12, Last appended by leader: 12, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:02.053122 18333 leader_election.cc:304] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:24:02.051384 17803 leader_election.cc:304] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:24:01.950111 24987 raft_consensus.cc:515] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:24:02.019915 25131 raft_consensus.cc:493] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
W20260709 12:24:02.049283 24989 consensus_meta.cc:287] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 1.027s	user 0.015s	sys 0.001s
I20260709 12:24:02.067024 18364 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 2186) took 1989 ms. Trace:
I20260709 12:24:02.044955 25278 raft_consensus.cc:2804] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:24:02.029351 25080 raft_consensus.cc:493] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:02.075240 18333 leader_election.cc:304] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:01.890914 25308 consensus_queue.cc:237] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.13, Last appended by leader: 13, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:02.063208 25329 raft_consensus.cc:2804] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:24:02.089694 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:24:02.074873 25252 consensus_meta.cc:287] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.565s	user 0.008s	sys 0.004s
I20260709 12:24:02.016079 25284 raft_consensus.cc:515] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:02.061847 25298 raft_consensus.cc:493] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:02.068241 25281 raft_consensus.cc:2804] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:24:02.061956 24834 raft_consensus.cc:2749] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:24:02.076434 18364 rpcz_store.cc:276] 0709 12:24:00.077264 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:00.087327 (+ 10063us) service_pool.cc:224] Handling call
0709 12:24:00.087804 (+   477us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:00.589199 (+501395us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:00.589229 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:00.596343 (+  7114us) log.cc:844] Serialized 26 byte log entry
0709 12:24:00.811102 (+214759us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:24:00.811139 (+    37us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:02.053947 (+1242808us) spinlock_profiling.cc:243] Waited 252 ms on lock 0x7b54004687d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:24:02.054017 (+    70us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:02.054083 (+    66us) raft_consensus.cc:1685] finished
0709 12:24:02.054098 (+    15us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:02.066921 (+ 12823us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":111452,"raft.run_cpu_time_us":192,"raft.run_wall_time_us":18300,"spinlock_wait_cycles":580366464,"thread_start_us":364147,"threads_started":2,"wal-append.queue_time_us":271139,"wal-append.run_cpu_time_us":1962,"wal-append.run_wall_time_us":1015817}
I20260709 12:24:02.064277 24392 raft_consensus.cc:2749] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:24:02.108695 18367 raft_consensus.cc:2393] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 20: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:24:02.073812 25131 raft_consensus.cc:3060] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:02.111974 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:02.081661 25278 raft_consensus.cc:493] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:02.092747 25080 raft_consensus.cc:3060] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:02.145321 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:02.110847 25168 raft_consensus.cc:697] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 19 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:24:02.181542 17833 raft_consensus.cc:2393] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 19: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
W20260709 12:24:02.144428 25320 log.cc:927] Time spent T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.339s	user 0.000s	sys 0.001s
I20260709 12:24:02.094702 25329 raft_consensus.cc:493] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:02.093359 24989 raft_consensus.cc:2804] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:24:02.103720 25140 leader_election.cc:290] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 17 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:02.212073 25329 raft_consensus.cc:3060] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:02.230470 25284 leader_election.cc:290] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 23 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:02.197072 25346 raft_consensus.cc:493] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:24:02.232400 25346 raft_consensus.cc:515] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:02.236423 25346 leader_election.cc:290] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:24:02.200402 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 160845us
I20260709 12:24:02.266829 17652 leader_election.cc:304] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:02.152920 25147 leader_election.cc:290] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:02.117657 25165 consensus_queue.cc:1243] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer e063541542a94969898868bf6eb5ce1b log is divergent from this leader: its last log entry 20.15 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 14
I20260709 12:24:02.187976 25081 leader_election.cc:290] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:02.184363 25002 raft_consensus.cc:515] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:02.239081 24987 leader_election.cc:290] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:24:02.241151 25332 log.cc:927] Time spent T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.258s	user 0.001s	sys 0.001s
W20260709 12:24:02.242161 25335 log.cc:927] Time spent T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.156s	user 0.000s	sys 0.000s
W20260709 12:24:02.243175 25328 log.cc:927] Time spent T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.259s	user 0.000s	sys 0.001s
I20260709 12:24:02.247296 24801 leader_election.cc:290] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:24:02.292867 25321 log.cc:927] Time spent T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.418s	user 0.000s	sys 0.002s
I20260709 12:24:02.114585 25294 raft_consensus.cc:2804] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:24:02.256486 18371 raft_consensus.cc:2468] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 22.
I20260709 12:24:02.180140 25278 raft_consensus.cc:3060] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:02.300626 25323 raft_consensus.cc:3055] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:24:02.289302 17652 leader_election.cc:304] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:02.153872 25298 raft_consensus.cc:515] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:02.198074 25281 raft_consensus.cc:697] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 23 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:24:02.257174 24989 raft_consensus.cc:493] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:24:02.308406 24992 leader_election.cc:290] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:02.305760 17680 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37402 (request call id 3289) took 1334 ms. Trace:
W20260709 12:24:02.248857 25339 log.cc:927] Time spent T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.146s	user 0.000s	sys 0.000s
I20260709 12:24:02.310036 17833 raft_consensus.cc:1240] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 19. Current term is 20. Ops: []
I20260709 12:24:02.294227 25179 raft_consensus.cc:2749] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:24:02.187500 25181 raft_consensus.cc:3055] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:24:02.344502 18368 raft_consensus.cc:1240] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 21. Current term is 22. Ops: []
I20260709 12:24:02.116675 25252 raft_consensus.cc:515] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:02.265143 25168 consensus_queue.cc:237] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 16.15, Last appended by leader: 15, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:02.337025 25294 raft_consensus.cc:493] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:24:02.408007 17763 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38786 (request call id 3588) took 1255 ms. Trace:
W20260709 12:24:02.359856 25334 log.cc:927] Time spent T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.315s	user 0.002s	sys 0.000s
W20260709 12:24:02.399555 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 48225.9us
I20260709 12:24:02.340087 25323 raft_consensus.cc:740] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:24:02.412619 17689 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44898 (request call id 1450) took 1496 ms. Trace:
W20260709 12:24:02.363425 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 74217.4us
I20260709 12:24:02.409087 17763 rpcz_store.cc:276] 0709 12:24:01.152720 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:01.152841 (+   121us) service_pool.cc:224] Handling call
0709 12:24:01.320779 (+167938us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:01.321166 (+   387us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:01.321207 (+    41us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:01.348907 (+ 27700us) log.cc:844] Serialized 26 byte log entry
0709 12:24:01.648996 (+300089us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:24:01.649048 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:01.649135 (+    87us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:02.407671 (+758536us) raft_consensus.cc:1685] finished
0709 12:24:02.407701 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:02.407923 (+   222us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":169747,"threads_started":1,"wal-append.queue_time_us":620601}
I20260709 12:24:02.412721 17836 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35042 (request call id 2831) took 1260 ms. Trace:
I20260709 12:24:02.350584 17680 rpcz_store.cc:276] 0709 12:24:00.971526 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:00.971701 (+   175us) service_pool.cc:224] Handling call
0709 12:24:00.980886 (+  9185us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:01.038677 (+ 57791us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:01.038724 (+    47us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:01.039321 (+   597us) log.cc:844] Serialized 26 byte log entry
0709 12:24:01.499356 (+460035us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:24:01.499394 (+    38us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:01.499460 (+    66us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:02.305507 (+806047us) raft_consensus.cc:1685] finished
0709 12:24:02.305526 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:02.305699 (+   173us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":57326,"raft.queue_time_us":57501,"spinlock_wait_cycles":21374336,"thread_start_us":444709,"threads_started":1,"wal-append.queue_time_us":625580}
I20260709 12:24:02.351313 24989 raft_consensus.cc:3060] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:02.413388 17689 rpcz_store.cc:276] 0709 12:24:00.915964 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:00.916124 (+   160us) service_pool.cc:224] Handling call
0709 12:24:00.955694 (+ 39570us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:00.956195 (+   501us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:00.956239 (+    44us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:01.026957 (+ 70718us) log.cc:844] Serialized 26 byte log entry
0709 12:24:01.519603 (+492646us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:24:01.519642 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:01.519719 (+    77us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:02.412346 (+892627us) raft_consensus.cc:1685] finished
0709 12:24:02.412372 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:02.412544 (+   172us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":9830,"raft.run_cpu_time_us":174,"raft.run_wall_time_us":7610,"thread_start_us":423334,"threads_started":1,"wal-append.queue_time_us":617616}
I20260709 12:24:02.414521 17836 rpcz_store.cc:276] 0709 12:24:01.152685 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:01.152843 (+   158us) service_pool.cc:224] Handling call
0709 12:24:01.322437 (+169594us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:01.330634 (+  8197us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:01.330686 (+    52us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:01.331407 (+   721us) log.cc:844] Serialized 26 byte log entry
0709 12:24:01.636134 (+304727us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:24:01.636191 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:01.636287 (+    96us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:02.412516 (+776229us) raft_consensus.cc:1685] finished
0709 12:24:02.412544 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:02.412659 (+   115us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":72551,"raft.queue_time_us":27299,"raft.run_cpu_time_us":179,"raft.run_wall_time_us":13650,"thread_start_us":304329,"threads_started":1,"wal-append.queue_time_us":569521}
I20260709 12:24:02.400048 25067 raft_consensus.cc:493] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:02.408005 25206 raft_consensus.cc:2804] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:24:02.411670 25294 raft_consensus.cc:3060] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:02.441226 25002 leader_election.cc:290] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 20 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:24:02.434827 25341 log.cc:927] Time spent T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.092s	user 0.000s	sys 0.000s
I20260709 12:24:02.402552 25354 raft_consensus.cc:2749] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 12:24:02.409034 25181 raft_consensus.cc:740] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
W20260709 12:24:02.450294 25338 log.cc:927] Time spent T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.297s	user 0.002s	sys 0.000s
W20260709 12:24:02.487263 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 66254.6us
I20260709 12:24:02.428313 25327 consensus_queue.cc:1048] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.046s
I20260709 12:24:02.425585 25352 consensus_queue.cc:1059] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 19.12, Next index: 13, Last known committed idx: 11, Time since last communication: 0.060s
I20260709 12:24:02.467602 24677 raft_consensus.cc:493] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:02.495560 17686 raft_consensus.cc:1240] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 14. Current term is 15. Ops: []
I20260709 12:24:02.507503 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:02.500087 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 20 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:02.500669 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:02.501210 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 17 candidate_status { last_received { term: 16 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:02.504522 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:02.512936 17682 raft_consensus.cc:2393] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 20: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:24:02.518674 17686 raft_consensus.cc:1240] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 19. Current term is 20. Ops: [19.12-19.12]
I20260709 12:24:02.509939 17681 raft_consensus.cc:2468] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 18.
I20260709 12:24:02.523743 17680 raft_consensus.cc:1275] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 17 index: 14. Preceding OpId from leader: term: 18 index: 15. (index mismatch)
I20260709 12:24:02.449770 25067 raft_consensus.cc:515] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:02.542743 17689 raft_consensus.cc:1240] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 19. Current term is 20. Ops: []
I20260709 12:24:02.469488 24818 consensus_queue.cc:1059] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 22.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.037s
I20260709 12:24:02.467797 25281 consensus_queue.cc:237] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 22.15, Last appended by leader: 15, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:02.521193 25081 raft_consensus.cc:493] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
W20260709 12:24:02.572543 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 70990.1us
I20260709 12:24:02.565776 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:24:02.568567 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 143819us
I20260709 12:24:02.574641 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:02.479069 25323 consensus_queue.cc:260] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.9, Last appended by leader: 9, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:02.574312 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:02.589569 17689 raft_consensus.cc:2468] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 17.
I20260709 12:24:02.586617 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 23 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:02.584144 25308 raft_consensus.cc:493] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:02.591462 25308 raft_consensus.cc:515] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:02.593767 25308 leader_election.cc:290] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:02.545668 25352 raft_consensus.cc:493] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:02.598533 17728 leader_election.cc:304] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
W20260709 12:24:02.576136 25099 consensus_meta.cc:287] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 1.069s	user 0.012s	sys 0.003s
I20260709 12:24:02.599852 17684 raft_consensus.cc:2468] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 17.
I20260709 12:24:02.601217 17728 leader_election.cc:304] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:02.602289 25099 raft_consensus.cc:515] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:02.608796 17762 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38786 (request call id 3587) took 1457 ms. Trace:
I20260709 12:24:02.609549 17762 rpcz_store.cc:276] 0709 12:24:01.151165 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:01.151293 (+   128us) service_pool.cc:224] Handling call
0709 12:24:01.323688 (+172395us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:01.438453 (+114765us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:02.608441 (+1169988us) spinlock_profiling.cc:243] Waited 105 ms on lock 0x7b54002977d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:24:02.608702 (+   261us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":241025408}
I20260709 12:24:02.609037 25252 leader_election.cc:290] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 18 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:02.562160 25356 raft_consensus.cc:493] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:02.598789 17682 raft_consensus.cc:2468] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 23.
I20260709 12:24:02.525837 24677 raft_consensus.cc:515] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:02.638967 25357 log.cc:927] Time spent T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.090s	user 0.000s	sys 0.001s
I20260709 12:24:02.640894 25285 leader_election.cc:290] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:02.641845 25285 raft_consensus.cc:2804] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:24:02.642508 25285 raft_consensus.cc:493] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:24:02.643041 25285 raft_consensus.cc:3060] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:02.644520 17755 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36572 (request call id 1326) took 1057 ms. Trace:
W20260709 12:24:02.645742 25248 consensus_meta.cc:287] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.977s	user 0.009s	sys 0.005s
I20260709 12:24:02.646613 18361 raft_consensus.cc:1275] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 16 index: 13. Preceding OpId from leader: term: 17 index: 14. (index mismatch)
I20260709 12:24:02.646744 25353 consensus_queue.cc:1048] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.001s
W20260709 12:24:02.648084 25359 log.cc:927] Time spent T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.086s	user 0.001s	sys 0.000s
I20260709 12:24:02.597882 25298 leader_election.cc:290] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:02.649544 24960 leader_election.cc:290] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 20 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:02.645157 17755 rpcz_store.cc:276] 0709 12:24:01.586726 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:01.955778 (+369052us) service_pool.cc:224] Handling call
0709 12:24:01.956187 (+   409us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:01.970802 (+ 14615us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 12:24:02.142823 (+172021us) log.cc:844] Serialized 14 byte log entry
0709 12:24:02.352053 (+209230us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:02.363340 (+ 11287us) log.cc:844] Serialized 26 byte log entry
0709 12:24:02.399347 (+ 36007us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:24:02.399412 (+    65us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:02.399489 (+    77us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:02.643969 (+244480us) raft_consensus.cc:1685] finished
0709 12:24:02.643991 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:02.644410 (+   419us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":13763,"prepare.queue_time_us":48467,"prepare.run_cpu_time_us":312,"prepare.run_wall_time_us":6577,"raft.queue_time_us":30301,"raft.run_cpu_time_us":320,"raft.run_wall_time_us":44095,"thread_start_us":259547,"threads_started":2,"wal-append.queue_time_us":300902}
W20260709 12:24:02.656055 25189 consensus_meta.cc:287] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 1.509s	user 0.017s	sys 0.000s
I20260709 12:24:02.656481 25158 raft_consensus.cc:3055] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:24:02.656769 25189 raft_consensus.cc:515] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:02.657227 25158 raft_consensus.cc:740] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 19 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:24:02.658125 17834 raft_consensus.cc:2368] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 19. Current term is 20.
I20260709 12:24:02.659168 25189 leader_election.cc:290] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 20 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:02.658540 25158 consensus_queue.cc:260] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 19.14, Last appended by leader: 14, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:02.618141 25284 raft_consensus.cc:2749] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
W20260709 12:24:02.646878 25363 log.cc:927] Time spent T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.125s	user 0.001s	sys 0.000s
I20260709 12:24:02.581178 25081 raft_consensus.cc:515] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:02.660671 25293 raft_consensus.cc:493] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:02.684497 25293 raft_consensus.cc:515] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:02.524227 25181 consensus_queue.cc:260] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.13, Last appended by leader: 13, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:02.685904 25081 leader_election.cc:290] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:02.687922 25067 leader_election.cc:290] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:02.695842 24960 raft_consensus.cc:493] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:02.696926 24960 raft_consensus.cc:515] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:02.661552 25323 raft_consensus.cc:3060] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:02.698791 17682 raft_consensus.cc:1275] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 16 index: 12. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
I20260709 12:24:02.697455 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:02.643483 25131 raft_consensus.cc:515] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:02.720721 25211 raft_consensus.cc:493] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:02.721431 25211 raft_consensus.cc:3060] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:02.721582 25131 leader_election.cc:290] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 22 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:02.725278 18366 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59902 (request call id 1795) took 1430 ms. Trace:
I20260709 12:24:02.726317 18366 rpcz_store.cc:276] 0709 12:24:01.294536 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:01.430906 (+136370us) service_pool.cc:224] Handling call
0709 12:24:02.725039 (+1294133us) spinlock_profiling.cc:243] Waited 421 ms on lock 0x7b54001efd54. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:24:02.725193 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":967635712}
I20260709 12:24:02.732424 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 20 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:02.732882 17727 leader_election.cc:304] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
W20260709 12:24:02.734479 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:02.734934 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:02.736230 17729 leader_election.cc:304] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:24:02.736662 25364 log.cc:927] Time spent T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.171s	user 0.002s	sys 0.000s
I20260709 12:24:02.737064 25298 raft_consensus.cc:2804] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:24:02.737675 25298 raft_consensus.cc:697] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 22 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:24:02.738423 18367 raft_consensus.cc:1240] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 22. Current term is 23. Ops: []
I20260709 12:24:02.703207 24677 leader_election.cc:290] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:02.701225 25350 raft_consensus.cc:493] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:02.739651 25350 raft_consensus.cc:515] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:02.703084 24960 leader_election.cc:290] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:02.706835 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 23 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:02.707451 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 19 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:02.707517 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:02.708487 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:02.709455 17758 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38786 (request call id 3589) took 1399 ms. Trace:
I20260709 12:24:02.712380 25284 raft_consensus.cc:493] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:02.614363 25147 consensus_queue.cc:1059] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 19.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.062s
I20260709 12:24:02.749483 18366 raft_consensus.cc:1275] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 21 index: 13. Preceding OpId from leader: term: 23 index: 14. (index mismatch)
W20260709 12:24:02.712757 24723 consensus_meta.cc:287] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.750s	user 0.010s	sys 0.003s
I20260709 12:24:02.700067 25158 raft_consensus.cc:3060] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:02.755262 25327 consensus_queue.cc:1048] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20260709 12:24:02.712920 25297 leader_election.cc:290] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:02.713377 25278 raft_consensus.cc:515] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:02.756366 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 23 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:24:02.715318 17655 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:02.716862 18362 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59902 (request call id 1789) took 1601 ms (client timeout 1913 ms). Trace:
W20260709 12:24:02.758424 18362 rpcz_store.cc:269] 0709 12:24:01.115921 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:01.313685 (+197764us) service_pool.cc:224] Handling call
0709 12:24:02.716608 (+1402923us) spinlock_profiling.cc:243] Waited 502 ms on lock 0x7b5400570154. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:24:02.716775 (+   167us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1155661696}
I20260709 12:24:02.718585 24642 raft_consensus.cc:3055] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:24:02.719394 24946 raft_consensus.cc:3055] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:24:02.734829 25131 raft_consensus.cc:2749] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 12:24:02.738808 18361 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 2199) took 1686 ms. Trace:
I20260709 12:24:02.739866 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:02.754998 25368 consensus_queue.cc:1059] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 22.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.002s
I20260709 12:24:02.741605 17833 raft_consensus.cc:2468] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 16.
I20260709 12:24:02.741545 17835 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 18.
I20260709 12:24:02.668125 25356 raft_consensus.cc:515] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:02.696161 18334 leader_election.cc:304] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:02.745491 17758 rpcz_store.cc:276] 0709 12:24:01.309751 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:01.310089 (+   338us) service_pool.cc:224] Handling call
0709 12:24:01.358746 (+ 48657us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:24:01.522864 (+164118us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:01.522907 (+    43us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:24:01.534653 (+ 11746us) log.cc:844] Serialized 52 byte log entry
0709 12:24:01.869270 (+334617us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:24:01.911109 (+ 41839us) log.cc:844] Serialized 14 byte log entry
0709 12:24:01.914380 (+  3271us) log.cc:844] Serialized 14 byte log entry
0709 12:24:01.914686 (+   306us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:02.708860 (+794174us) spinlock_profiling.cc:243] Waited 139 ms on lock 0x7b54004155d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:24:02.708939 (+    79us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:02.709127 (+   188us) raft_consensus.cc:1685] finished
0709 12:24:02.709153 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:02.709372 (+   219us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":67975,"prepare.queue_time_us":170964,"prepare.run_cpu_time_us":105,"prepare.run_wall_time_us":3710,"raft.queue_time_us":51860,"raft.run_cpu_time_us":161,"raft.run_wall_time_us":27512,"spinlock_wait_cycles":355724672,"thread_start_us":244383,"threads_started":1,"wal-append.queue_time_us":662794}
I20260709 12:24:02.750244 18364 raft_consensus.cc:1240] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 18. Current term is 19. Ops: []
I20260709 12:24:02.741091 17727 leader_election.cc:304] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:02.741605 17836 raft_consensus.cc:2468] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 21.
W20260709 12:24:02.752095 17802 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:02.778652 17727 leader_election.cc:304] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:02.752204 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:02.752406 25284 raft_consensus.cc:515] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:02.764621 17801 leader_election.cc:304] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:02.770954 18361 rpcz_store.cc:276] 0709 12:24:01.052110 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:01.052443 (+   333us) service_pool.cc:224] Handling call
0709 12:24:01.053032 (+   589us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:02.647073 (+1594041us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:02.738562 (+ 91489us) spinlock_profiling.cc:243] Waited 1.59 s on lock 0x7b54006af9d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:24:02.738754 (+   192us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3663858048}
W20260709 12:24:02.701699 25337 log.cc:927] Time spent T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.279s	user 0.000s	sys 0.002s
I20260709 12:24:02.618269 25352 raft_consensus.cc:515] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:02.766621 25329 consensus_meta.cc:287] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.543s	user 0.012s	sys 0.000s
W20260709 12:24:02.804961 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 108889us
W20260709 12:24:02.810552 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 209394us
I20260709 12:24:02.821542 17832 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37994 (request call id 1437) took 1210 ms. Trace:
I20260709 12:24:02.821997 17832 rpcz_store.cc:276] 0709 12:24:01.610777 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:01.613982 (+  3205us) service_pool.cc:224] Handling call
0709 12:24:02.821360 (+1207378us) spinlock_profiling.cc:243] Waited 300 ms on lock 0x7b540008cfd4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:24:02.821463 (+   103us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":689746816}
I20260709 12:24:02.739367 25298 consensus_queue.cc:237] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 21.15, Last appended by leader: 15, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:02.825291 17774 tablet_copy_source_session.cc:215] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:24:02.825417 24790 consensus_queue.cc:1059] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 14.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20260709 12:24:02.826822 17774 rpcz_store.cc:275] Call kudu.tserver.TabletCopyService.BeginTabletCopySession from 127.0.0.1:36572 (request call id 1325) took 1308 ms. Trace:
W20260709 12:24:02.820025 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 55491.1us
I20260709 12:24:02.828397 17801 leader_election.cc:304] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:24:02.766353 24642 raft_consensus.cc:740] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:24:02.821463 17728 leader_election.cc:304] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:24:02.769357 24946 raft_consensus.cc:740] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 22 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:24:02.812769 25147 raft_consensus.cc:3060] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:02.773778 25099 leader_election.cc:290] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 19 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:02.827260 17774 rpcz_store.cc:276] 0709 12:24:01.577349 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:01.588310 (+ 10961us) service_pool.cc:224] Handling call
0709 12:24:02.824803 (+1236493us) spinlock_profiling.cc:243] Waited 3.96 ms on lock 0x7b540036f4a0. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a597dc0 00007f7b4a62a4a5 00007f7b4a62a3ee 00007f7b4f5c0036 00007f7b4f5c9a3e 00007f7b46e8d27b 00007f7b4f5c99ba 00007f7b4f5c4acc 00007f7b4f5bfc7b 00007f7b4f5b48d1
0709 12:24:02.826728 (+  1925us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":9099136}
I20260709 12:24:02.814301 24960 raft_consensus.cc:2804] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:24:02.832690 25246 raft_consensus.cc:3055] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:24:02.830498 25356 leader_election.cc:290] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:02.832991 25275 raft_consensus.cc:3055] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:24:02.831686 25297 raft_consensus.cc:2804] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:24:02.835250 25189 raft_consensus.cc:2749] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:24:02.833635 25165 raft_consensus.cc:493] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:02.842060 24801 raft_consensus.cc:2804] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:24:02.873904 17829 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35042 (request call id 2832) took 1720 ms. Trace:
I20260709 12:24:02.826823 25279 consensus_queue.cc:1059] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 19.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.037s
I20260709 12:24:02.920886 17829 rpcz_store.cc:276] 0709 12:24:01.153155 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:01.153276 (+   121us) service_pool.cc:224] Handling call
0709 12:24:01.153676 (+   400us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:01.503703 (+350027us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:01.503732 (+    29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:01.510356 (+  6624us) log.cc:844] Serialized 26 byte log entry
0709 12:24:01.925535 (+415179us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:24:01.925741 (+   206us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:01.926468 (+   727us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:02.873552 (+947084us) raft_consensus.cc:1685] finished
0709 12:24:02.873590 (+    38us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:02.873812 (+   222us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":91277,"raft.queue_time_us":157114,"raft.run_cpu_time_us":162,"raft.run_wall_time_us":14797,"thread_start_us":631439,"threads_started":2,"wal-append.queue_time_us":763542}
I20260709 12:24:02.849388 24960 raft_consensus.cc:493] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:24:02.920240 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 143043us
I20260709 12:24:02.927659 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:02.834291 25329 raft_consensus.cc:515] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:02.931756 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:02.854331 25246 raft_consensus.cc:740] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 19 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:24:02.933501 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 19 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:24:02.922345 25080 consensus_meta.cc:287] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.693s	user 0.009s	sys 0.004s
I20260709 12:24:02.849678 24992 consensus_queue.cc:1048] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 11, Time since last communication: 0.078s
I20260709 12:24:02.872591 25275 raft_consensus.cc:740] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:24:02.875170 25297 raft_consensus.cc:493] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:02.849685 25375 consensus_queue.cc:1059] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 18.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.090s
W20260709 12:24:02.949688 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 127122us
W20260709 12:24:02.956159 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 125731us
I20260709 12:24:02.851487 25315 consensus_queue.cc:1059] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true }, Status: INVALID_TERM, Last received: 19.12, Next index: 13, Last known committed idx: 11, Time since last communication: 0.085s
I20260709 12:24:02.929746 24960 raft_consensus.cc:3060] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:02.975428 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
W20260709 12:24:02.979272 17803 leader_election.cc:336] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Timed out: RequestConsensusVote RPC to 127.16.29.68:42577 timed out after 2.921s (SENT)
I20260709 12:24:02.938050 24801 raft_consensus.cc:697] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 22 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:24:02.946341 25099 raft_consensus.cc:493] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
W20260709 12:24:02.986387 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 170824us
I20260709 12:24:02.937762 25168 raft_consensus.cc:493] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:02.946019 25121 raft_consensus.cc:2764] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Leader pre-election decision vote started in defunct term 18: won
I20260709 12:24:02.930837 25165 raft_consensus.cc:515] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:03.008128 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 21 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:03.003461 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:02.978888 25297 raft_consensus.cc:3060] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:02.980079 25376 raft_consensus.cc:493] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:03.024567 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:03.023427 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:03.005714 25327 raft_consensus.cc:493] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:02.956007 25080 raft_consensus.cc:515] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:03.033049 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:03.025954 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:03.034584 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 18 candidate_status { last_received { term: 16 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:03.002985 25081 raft_consensus.cc:493] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:24:02.983809 25308 consensus_queue.cc:1059] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 22.12, Next index: 13, Last known committed idx: 10, Time since last communication: 0.042s
I20260709 12:24:03.040650 17685 raft_consensus.cc:3060] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:03.020460 24818 raft_consensus.cc:493] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:03.018543 25350 leader_election.cc:290] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:03.018272 25278 leader_election.cc:290] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 19 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:02.994254 25368 consensus_queue.cc:1048] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.040s
W20260709 12:24:03.028885 24989 consensus_meta.cc:287] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.600s	user 0.012s	sys 0.000s
I20260709 12:24:03.056160 24801 consensus_queue.cc:237] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 20.12, Last appended by leader: 12, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:03.036568 25376 raft_consensus.cc:515] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:03.020628 25002 raft_consensus.cc:493] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:24:03.059944 25081 raft_consensus.cc:515] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:03.060276 25002 raft_consensus.cc:515] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:03.061410 25376 leader_election.cc:290] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:03.031790 25275 consensus_queue.cc:260] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 14.15, Last appended by leader: 15, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:03.062230 25081 leader_election.cc:290] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:03.062541 25352 leader_election.cc:290] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:03.042295 25380 raft_consensus.cc:493] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:03.062784 25284 leader_election.cc:290] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:03.063261 25380 raft_consensus.cc:515] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:03.065594 25380 leader_election.cc:290] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:03.042193 25327 raft_consensus.cc:515] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:03.062577 25275 raft_consensus.cc:3060] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:03.068845 25308 raft_consensus.cc:3055] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:24:03.069257 25308 raft_consensus.cc:740] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 22 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:24:03.070045 25308 consensus_queue.cc:260] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 22.12, Last appended by leader: 12, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:03.070653 25327 leader_election.cc:290] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:03.053843 25080 leader_election.cc:290] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 18 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:02.953579 24946 consensus_queue.cc:260] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 22.13, Last appended by leader: 13, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:03.072935 24946 raft_consensus.cc:3060] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:03.073113 25329 leader_election.cc:290] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 19 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:03.026170 25168 raft_consensus.cc:515] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:03.074636 25379 log.cc:927] Time spent T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.071s	user 0.000s	sys 0.000s
I20260709 12:24:03.007378 25099 raft_consensus.cc:515] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:03.076131 25374 log.cc:927] Time spent T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.093s	user 0.001s	sys 0.000s
I20260709 12:24:03.077431 25099 leader_election.cc:290] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:03.039564 17682 raft_consensus.cc:2468] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 20.
I20260709 12:24:03.039494 17686 raft_consensus.cc:2468] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 23.
I20260709 12:24:03.079264 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:03.079591 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:03.057173 24989 raft_consensus.cc:515] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:03.080402 25381 log.cc:927] Time spent T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.074s	user 0.000s	sys 0.000s
W20260709 12:24:03.081278 25377 log.cc:927] Time spent T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.127s	user 0.000s	sys 0.001s
I20260709 12:24:03.082119 24989 leader_election.cc:290] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 18 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:24:03.082896 25378 log.cc:927] Time spent T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.058s	user 0.000s	sys 0.000s
I20260709 12:24:03.086479 17688 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44898 (request call id 1449) took 2172 ms. Trace:
I20260709 12:24:03.087045 17688 rpcz_store.cc:276] 0709 12:24:00.914432 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:00.914848 (+   416us) service_pool.cc:224] Handling call
0709 12:24:00.957808 (+ 42960us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:24:00.966449 (+  8641us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:00.966484 (+    35us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:24:00.973994 (+  7510us) log.cc:844] Serialized 52 byte log entry
0709 12:24:02.655821 (+1681827us) spinlock_profiling.cc:243] Waited 44.1 ms on lock 0x7b1800461da0. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b47515e07 00007f7b503709a9 00007f7b4753269b 00007f7b4753093f 00007f7b47532c02 00007f7b4a1ba20b 00007f7b4a1bfd23 00007f7b4a1c00d1 00007f7b4a5dba0f
0709 12:24:02.835178 (+179357us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:24:02.980856 (+145678us) log.cc:844] Serialized 14 byte log entry
0709 12:24:03.017555 (+ 36699us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:03.017721 (+   166us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:03.085939 (+ 68218us) raft_consensus.cc:1685] finished
0709 12:24:03.086066 (+   127us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:03.086392 (+   326us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":72797,"prepare.queue_time_us":27428,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":44,"raft.queue_time_us":67505,"raft.run_cpu_time_us":212,"raft.run_wall_time_us":6825,"spinlock_wait_cycles":101417344,"thread_start_us":308434,"threads_started":2,"wal-append.queue_time_us":1969799}
I20260709 12:24:03.090107 25211 raft_consensus.cc:515] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:03.077782 25168 leader_election.cc:290] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:03.100162 25165 leader_election.cc:290] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:24:03.110090 25294 consensus_meta.cc:287] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.638s	user 0.011s	sys 0.001s
I20260709 12:24:03.117733 17685 raft_consensus.cc:2468] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 18.
I20260709 12:24:03.071250 25308 raft_consensus.cc:3060] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:03.082901 17686 raft_consensus.cc:1240] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 22. Current term is 23. Ops: []
I20260709 12:24:03.130532 17688 raft_consensus.cc:1240] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 19. Current term is 20. Ops: []
I20260709 12:24:03.135967 25285 raft_consensus.cc:515] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:03.119930 25294 raft_consensus.cc:515] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:03.107385 25211 leader_election.cc:290] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 17 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:03.144600 25294 leader_election.cc:290] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 18 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:03.144924 25181 raft_consensus.cc:3060] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:24:03.146452 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:03.149317 17838 raft_consensus.cc:2410] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for term 20 because replica has last-logged OpId of term: 19 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 13.
W20260709 12:24:03.150020 17802 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:03.120965 24989 raft_consensus.cc:493] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
W20260709 12:24:03.153733 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 135620us
I20260709 12:24:03.157946 17727 leader_election.cc:304] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:03.132037 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:03.126049 25327 raft_consensus.cc:493] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:03.173285 25327 raft_consensus.cc:515] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:03.174070 17834 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37994 (request call id 1434) took 1591 ms (client timeout 1788 ms). Trace:
I20260709 12:24:03.158013 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:24:03.174749 17834 rpcz_store.cc:269] 0709 12:24:01.582681 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:01.611242 (+ 28561us) service_pool.cc:224] Handling call
0709 12:24:03.169891 (+1558649us) spinlock_profiling.cc:243] Waited 996 ms on lock 0x7b54003359d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:24:03.173985 (+  4094us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2291312512}
I20260709 12:24:03.176177 24989 raft_consensus.cc:515] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:03.176980 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:03.159389 17802 leader_election.cc:304] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:24:03.176679 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:03.178627 25285 leader_election.cc:290] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 19 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:03.179709 17760 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38786 (request call id 3591) took 1677 ms. Trace:
I20260709 12:24:03.188035 17760 rpcz_store.cc:276] 0709 12:24:01.502067 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:01.502760 (+   693us) service_pool.cc:224] Handling call
0709 12:24:01.526502 (+ 23742us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:01.660619 (+134117us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:03.179351 (+1518732us) spinlock_profiling.cc:243] Waited 69.4 ms on lock 0x7b540019fad4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:24:03.179626 (+   275us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":159552384}
I20260709 12:24:03.188258 17727 leader_election.cc:304] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:03.156589 25354 raft_consensus.cc:493] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:24:03.180351 17764 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38786 (request call id 3585) took 2034 ms. Trace:
I20260709 12:24:03.181843 17685 raft_consensus.cc:2468] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 21.
I20260709 12:24:03.183281 25387 raft_consensus.cc:2804] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:24:03.192775 17764 rpcz_store.cc:276] 0709 12:24:01.145941 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:01.146207 (+   266us) service_pool.cc:224] Handling call
0709 12:24:01.146495 (+   288us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:01.431180 (+284685us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:03.180054 (+1748874us) spinlock_profiling.cc:243] Waited 284 ms on lock 0x7b54002b5554. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:24:03.180282 (+   228us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":653144832}
I20260709 12:24:03.154924 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:03.196652 17802 leader_election.cc:304] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:24:03.154673 25211 raft_consensus.cc:493] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:03.125504 24818 raft_consensus.cc:515] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:03.199220 17684 raft_consensus.cc:3060] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:03.208277 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:03.208498 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:03.209028 17829 raft_consensus.cc:2468] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 23.
I20260709 12:24:03.199813 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:03.204421 24946 raft_consensus.cc:2804] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:24:03.201390 25211 raft_consensus.cc:515] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:03.201917 25158 raft_consensus.cc:2804] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:24:03.196415 25388 raft_consensus.cc:2804] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:24:03.219118 17834 raft_consensus.cc:1240] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 17. Current term is 18. Ops: []
I20260709 12:24:03.238947 17727 leader_election.cc:304] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:03.198899 25376 raft_consensus.cc:493] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:03.246563 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:03.199195 25298 raft_consensus.cc:493] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:03.195863 25387 raft_consensus.cc:493] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:03.254336 25387 raft_consensus.cc:3060] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:03.240433 25206 raft_consensus.cc:493] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:03.255344 25206 raft_consensus.cc:3060] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:03.254249 24960 raft_consensus.cc:515] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:03.255920 25275 raft_consensus.cc:493] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:03.248607 25081 raft_consensus.cc:2804] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:24:03.244614 24767 consensus_queue.cc:1059] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 16.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.018s
I20260709 12:24:03.256470 25081 raft_consensus.cc:493] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:24:03.226950 24818 leader_election.cc:290] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:24:03.259218 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:03.257136 25081 raft_consensus.cc:3060] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:03.260779 25298 raft_consensus.cc:515] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:03.238158 25388 raft_consensus.cc:493] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:03.265412 25388 raft_consensus.cc:3060] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:03.246312 24946 raft_consensus.cc:697] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 18 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:24:03.256457 25397 consensus_queue.cc:1059] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 17.13, Next index: 14, Last known committed idx: 12, Time since last communication: 0.014s
I20260709 12:24:03.257802 25301 raft_consensus.cc:493] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:03.238317 25158 raft_consensus.cc:493] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:03.266335 25298 leader_election.cc:290] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:03.267228 24998 raft_consensus.cc:3060] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:03.199476 25354 raft_consensus.cc:515] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:03.271265 25264 raft_consensus.cc:3055] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:24:03.277994 25264 raft_consensus.cc:740] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 17 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:24:03.271685 25301 raft_consensus.cc:515] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:03.279001 25264 consensus_queue.cc:260] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.13, Last appended by leader: 13, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:03.280254 25264 raft_consensus.cc:3060] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:03.293816 25002 leader_election.cc:290] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:03.272167 25158 raft_consensus.cc:3060] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:03.253438 25327 leader_election.cc:290] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:03.258478 25376 raft_consensus.cc:515] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:03.330654 25132 raft_consensus.cc:493] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:24:03.334456 25132 raft_consensus.cc:515] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:03.289456 24989 leader_election.cc:290] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:03.273057 25275 raft_consensus.cc:515] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:03.347707 25376 leader_election.cc:290] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:03.343106 25132 leader_election.cc:290] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:24:03.349448 25147 consensus_meta.cc:287] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.500s	user 0.006s	sys 0.006s
I20260709 12:24:03.350929 25147 raft_consensus.cc:2749] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:24:03.270150 24946 consensus_queue.cc:237] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 16.15, Last appended by leader: 15, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:03.351439 24960 leader_election.cc:290] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 19 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:03.356837 24723 raft_consensus.cc:3055] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:24:03.357498 24723 raft_consensus.cc:740] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:24:03.358878 24723 consensus_queue.cc:260] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 19.12, Last appended by leader: 12, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:03.348773 25081 raft_consensus.cc:515] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:03.381909 25081 leader_election.cc:290] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 23 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:03.362543 25297 raft_consensus.cc:515] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:03.385049 25297 leader_election.cc:290] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 23 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:03.376150 25158 raft_consensus.cc:515] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Starting leader election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:24:03.387871 25158 leader_election.cc:290] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 21 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:03.389389 24723 raft_consensus.cc:3060] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:03.393442 25133 raft_consensus.cc:3060] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:24:03.393522 18374 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59902 (request call id 1785) took 2279 ms (client timeout 2921 ms). Trace:
W20260709 12:24:03.398360 18374 rpcz_store.cc:269] 0709 12:24:01.114660 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:01.114848 (+   188us) service_pool.cc:224] Handling call
0709 12:24:03.393305 (+2278457us) spinlock_profiling.cc:243] Waited 984 ms on lock 0x7b540047ca54. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:24:03.393454 (+   149us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2263331072}
I20260709 12:24:03.399303 25370 raft_consensus.cc:3055] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:24:03.402240 25370 raft_consensus.cc:740] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:24:03.401072 25293 leader_election.cc:290] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:03.403901 25370 consensus_queue.cc:260] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 22.15, Last appended by leader: 15, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:03.404258 25206 raft_consensus.cc:515] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:03.404875 25275 leader_election.cc:290] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:03.405448 25370 raft_consensus.cc:3060] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:03.410245 25211 leader_election.cc:290] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:03.412905 25204 raft_consensus.cc:3060] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:03.414374 25302 raft_consensus.cc:493] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:03.414884 25302 raft_consensus.cc:515] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:03.418119 25302 leader_election.cc:290] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:03.421376 25246 consensus_queue.cc:260] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 19.12, Last appended by leader: 12, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:24:03.422658 25246 raft_consensus.cc:3060] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:03.424201 25384 raft_consensus.cc:3055] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:24:03.424670 25384 raft_consensus.cc:740] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:24:03.425583 25206 leader_election.cc:290] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 17 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:03.425441 25384 consensus_queue.cc:260] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:03.426329 25263 leader_election.cc:290] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:03.426651 25384 raft_consensus.cc:3060] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Advancing to term 19
W20260709 12:24:03.467371 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:03.469210 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:03.469475 17652 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.882s	user 0.018s	sys 0.026s
W20260709 12:24:03.469883 17652 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.883s	user 0.018s	sys 0.026s
W20260709 12:24:03.470368 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 887562us
W20260709 12:24:03.471343 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:03.485375 17682 raft_consensus.cc:2468] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 18.
I20260709 12:24:03.485746 17687 raft_consensus.cc:2410] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for term 19 because replica has last-logged OpId of term: 17 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 10.
I20260709 12:24:03.220036 17684 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 19.
I20260709 12:24:03.247172 17686 raft_consensus.cc:2468] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 19.
I20260709 12:24:03.487160 17688 raft_consensus.cc:1275] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 18 index: 16. Preceding OpId from leader: term: 19 index: 16. (term mismatch)
I20260709 12:24:03.488065 17688 pending_rounds.cc:85] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3: Aborting all ops after (but not including) 15
I20260709 12:24:03.488641 17688 pending_rounds.cc:107] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3: Aborting uncommitted NO_OP operation due to leader change: 18.16
I20260709 12:24:03.489115 17688 raft_consensus.cc:2889] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:03.489288 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 23 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:24:03.494055 17730 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:03.495594 18334 leader_election.cc:304] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:24:03.496693 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:03.497786 17686 raft_consensus.cc:2393] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 23: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:24:03.499012 18334 leader_election.cc:304] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:24:03.504980 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:03.506286 17688 raft_consensus.cc:3060] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:03.509104 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 19 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:03.495636 17832 raft_consensus.cc:1240] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 19. Current term is 21. Ops: []
I20260709 12:24:03.509104 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:03.511854 25206 raft_consensus.cc:493] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:03.513398 25182 raft_consensus.cc:2804] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:24:03.513161 25206 raft_consensus.cc:515] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:03.514348 25182 raft_consensus.cc:697] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 17 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:24:03.526043 25301 leader_election.cc:290] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:03.511699 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:03.527940 18374 raft_consensus.cc:2393] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 19: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:24:03.528357 25354 leader_election.cc:290] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:03.523272 25133 consensus_queue.cc:1059] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 19.15, Next index: 16, Last known committed idx: 13, Time since last communication: 0.006s
I20260709 12:24:03.530771 25133 raft_consensus.cc:3055] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:24:03.531411 25133 raft_consensus.cc:740] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:24:03.520649 17686 raft_consensus.cc:1240] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 19. Current term is 20. Ops: [19.12-19.12]
I20260709 12:24:03.542783 25264 raft_consensus.cc:493] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:24:03.543123 25301 raft_consensus.cc:493] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:03.544214 25264 raft_consensus.cc:515] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:03.544355 25301 raft_consensus.cc:515] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:03.546877 25263 raft_consensus.cc:493] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:03.547962 25263 raft_consensus.cc:515] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:03.550348 25133 consensus_queue.cc:260] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 19.15, Last appended by leader: 15, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:03.551945 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:03.518635 25275 consensus_queue.cc:1048] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.036s
I20260709 12:24:03.535025 24989 raft_consensus.cc:2804] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:24:03.559649 24989 raft_consensus.cc:493] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:03.560204 24989 raft_consensus.cc:3060] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:03.561484 18366 raft_consensus.cc:1240] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 19. Current term is 20. Ops: []
I20260709 12:24:03.542846 25182 consensus_queue.cc:237] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 16.15, Last appended by leader: 15, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:03.563342 17759 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38786 (request call id 3592) took 2042 ms (client timeout 1563 ms). Trace:
I20260709 12:24:03.565179 17764 raft_consensus.cc:1240] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 22 LEADER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 21. Current term is 22. Ops: []
I20260709 12:24:03.566847 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:03.551457 25264 leader_election.cc:290] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:03.578645 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:03.580637 25281 raft_consensus.cc:493] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:03.581988 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:03.583156 17834 raft_consensus.cc:2393] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 16: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:24:03.587002 17829 raft_consensus.cc:1275] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 21 index: 13. Preceding OpId from leader: term: 23 index: 14. (index mismatch)
I20260709 12:24:03.585645 24642 consensus_queue.cc:260] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 19.14, Last appended by leader: 14, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:03.558372 25301 leader_election.cc:290] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:03.574252 25133 raft_consensus.cc:3060] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:03.589751 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:03.581648 25281 raft_consensus.cc:515] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:03.597206 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:03.598201 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
W20260709 12:24:03.598999 17801 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:03.600349 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:03.601408 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:03.604864 17755 raft_consensus.cc:3060] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Advancing to term 20
W20260709 12:24:03.606194 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:03.606688 17836 raft_consensus.cc:1240] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 22 LEADER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 21. Current term is 22. Ops: [21.13-21.13]
I20260709 12:24:03.602499 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:03.619550 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:03.620294 25206 leader_election.cc:290] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:03.618225 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:03.608877 17756 raft_consensus.cc:2376] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 19. Re-sending same reply.
W20260709 12:24:03.569285 17759 rpcz_store.cc:269] 0709 12:24:01.520898 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:01.557905 (+ 37007us) service_pool.cc:224] Handling call
0709 12:24:03.562876 (+2004971us) spinlock_profiling.cc:243] Waited 209 ms on lock 0x7b5400310e54. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:24:03.563076 (+   200us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":479730944}
I20260709 12:24:03.637068 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:03.638896 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:03.638343 17836 raft_consensus.cc:2955] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Committing config change with OpId 19.12: config changed from index 5 to 12, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:24:03.663520 17761 raft_consensus.cc:2393] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 20: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:24:03.663442 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 23 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:24:03.664289 17761 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38786 (request call id 3595) took 2080 ms (client timeout 1568 ms). Trace:
W20260709 12:24:03.664736 17761 rpcz_store.cc:269] 0709 12:24:01.583711 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:01.641937 (+ 58226us) service_pool.cc:224] Handling call
0709 12:24:03.596772 (+1954835us) spinlock_profiling.cc:243] Waited 878 us on lock 0x7b5800033398. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b474fbb71 00007f7b474fba22 00007f7b504148d9 00007f7b4f56eee0 00007f7b4f6380ab 00007f7b4f638131 00007f7b4f5e60a0 00007f7b4f5e5984 00007f7b48b606d4 00007f7b48b60662
0709 12:24:03.664203 (+ 67431us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2018560}
I20260709 12:24:03.665076 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 23 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:03.667300 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:03.667505 17728 leader_election.cc:304] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:03.676906 17757 raft_consensus.cc:3060] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:03.679154 18362 raft_consensus.cc:2468] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 24.
I20260709 12:24:03.680604 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:03.682206 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:03.684506 25131 raft_consensus.cc:2749] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:24:03.689397 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 20 candidate_status { last_received { term: 18 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:03.694429 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:03.694602 25281 leader_election.cc:290] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:03.661893 24642 raft_consensus.cc:3060] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:03.697211 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:03.698102 17832 raft_consensus.cc:2468] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 15.
I20260709 12:24:03.698570 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:03.699487 17835 raft_consensus.cc:2468] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 20.
I20260709 12:24:03.699844 17831 raft_consensus.cc:1240] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 16. Current term is 17. Ops: []
W20260709 12:24:03.682387 17727 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:03.701993 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 113271us
I20260709 12:24:03.677529 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 23 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:03.703576 18362 raft_consensus.cc:1275] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 16 index: 13. (index mismatch)
I20260709 12:24:03.695261 18368 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59902 (request call id 1799) took 1605 ms. Trace:
I20260709 12:24:03.643930 25288 consensus_queue.cc:1059] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 20.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.114s
W20260709 12:24:03.721449 17655 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:03.728085 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 61829.1us
I20260709 12:24:03.730608 17729 leader_election.cc:304] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:03.732162 25264 raft_consensus.cc:3060] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:03.736980 17838 raft_consensus.cc:3055] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:24:03.737483 17838 raft_consensus.cc:740] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 21 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:24:03.738574 17838 consensus_queue.cc:260] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 21.15, Last appended by leader: 15, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:03.739270 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 64552.1us
I20260709 12:24:03.739830 17838 raft_consensus.cc:3060] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:03.742041 17682 raft_consensus.cc:1240] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 19. Current term is 20. Ops: []
I20260709 12:24:03.743969 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:03.744453 18334 leader_election.cc:304] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:24:03.746308 25182 raft_consensus.cc:2804] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:24:03.746718 25182 raft_consensus.cc:493] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:03.747023 25182 raft_consensus.cc:3060] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:03.741247 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 24 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:03.741482 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:03.743518 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:03.751502 17686 raft_consensus.cc:2468] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 24.
I20260709 12:24:03.753013 25275 consensus_queue.cc:1048] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:24:03.756038 17829 raft_consensus.cc:3060] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Advancing to term 23
W20260709 12:24:03.759518 18334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:03.761164 17727 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:03.767721 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:03.722886 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 19 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:03.759263 17762 raft_consensus.cc:2468] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 18.
I20260709 12:24:03.713876 18368 rpcz_store.cc:276] 0709 12:24:02.089241 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:02.089406 (+   165us) service_pool.cc:224] Handling call
0709 12:24:02.312957 (+223551us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:02.403961 (+ 91004us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:03.694630 (+1290669us) spinlock_profiling.cc:243] Waited 223 ms on lock 0x7b5400305f50. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a61f8ad 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:24:03.695001 (+   371us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":513059840}
I20260709 12:24:03.786001 17834 raft_consensus.cc:2468] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 23.
I20260709 12:24:03.780054 17838 raft_consensus.cc:1275] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 21 index: 15. Preceding OpId from leader: term: 22 index: 16. (index mismatch)
W20260709 12:24:03.787333 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:03.801934 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 19 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:03.803557 17688 raft_consensus.cc:2468] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 19.
I20260709 12:24:03.800199 17757 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 19.
I20260709 12:24:03.814610 17728 leader_election.cc:304] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:03.815933 18367 raft_consensus.cc:2410] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for term 19 because replica has last-logged OpId of term: 17 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 11.
I20260709 12:24:03.795331 25264 raft_consensus.cc:2749] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:24:03.818343 17680 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44898 (request call id 1455) took 1307 ms. Trace:
I20260709 12:24:03.821628 18334 leader_election.cc:304] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:24:03.825909 17834 raft_consensus.cc:2410] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for term 19 because replica has last-logged OpId of term: 17 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 11.
I20260709 12:24:03.827277 17838 raft_consensus.cc:1240] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 15 LEADER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 14. Current term is 15. Ops: []
W20260709 12:24:03.828370 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 109051us
I20260709 12:24:03.822645 17680 rpcz_store.cc:276] 0709 12:24:02.494652 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:02.494789 (+   137us) service_pool.cc:224] Handling call
0709 12:24:02.523280 (+ 28491us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:02.524236 (+   956us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:03.801428 (+1277192us) spinlock_profiling.cc:243] Waited 10.3 ms on lock 0x7b54000cf150. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a61f8ad 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:24:03.801734 (+   306us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":23761280}
I20260709 12:24:03.832029 25131 raft_consensus.cc:2804] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:24:03.835250 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:03.836113 17686 raft_consensus.cc:2468] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 24.
I20260709 12:24:03.837085 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 17 candidate_status { last_received { term: 16 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:03.837623 17728 leader_election.cc:304] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:03.841614 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:03.843295 17688 raft_consensus.cc:2393] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 21: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:24:03.804874 25182 raft_consensus.cc:515] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:03.848186 17760 raft_consensus.cc:2468] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 20.
I20260709 12:24:03.848402 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:03.848948 25354 raft_consensus.cc:493] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:03.849180 17680 raft_consensus.cc:2393] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 23: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:24:03.849545 25354 raft_consensus.cc:515] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:03.854532 17727 leader_election.cc:304] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:03.855711 25278 raft_consensus.cc:3055] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:24:03.856287 25278 raft_consensus.cc:740] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 20 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
W20260709 12:24:03.858235 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 127866us
I20260709 12:24:03.861644 25182 leader_election.cc:290] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 20 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:24:03.871577 17655 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:03.841477 25131 raft_consensus.cc:697] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 23 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:24:03.877746 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:24:03.882954 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 80857.8us
I20260709 12:24:03.885161 17762 raft_consensus.cc:1275] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 19 index: 15. Preceding OpId from leader: term: 21 index: 16. (index mismatch)
I20260709 12:24:03.893705 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:03.897337 17680 raft_consensus.cc:2468] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 15.
I20260709 12:24:03.914482 17729 leader_election.cc:304] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:03.917037 25081 raft_consensus.cc:2749] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:24:03.920223 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
W20260709 12:24:03.924345 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 86820us
I20260709 12:24:03.926996 17728 leader_election.cc:304] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:24:03.932888 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:03.933902 17680 raft_consensus.cc:2468] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 23.
I20260709 12:24:03.934409 25278 consensus_queue.cc:260] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 20.17, Last appended by leader: 17, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:03.935010 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 20 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:24:03.936378 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 82126.8us
I20260709 12:24:03.937557 17727 leader_election.cc:304] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:03.938849 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:24:03.941309 25410 log.cc:927] Time spent T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.073s	user 0.000s	sys 0.000s
W20260709 12:24:03.941759 25409 log.cc:927] Time spent T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.124s	user 0.000s	sys 0.000s
I20260709 12:24:03.941589 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:03.942633 18365 raft_consensus.cc:2393] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 19: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:24:03.947059 24989 raft_consensus.cc:515] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:03.881203 25264 raft_consensus.cc:3060] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:03.953902 17829 raft_consensus.cc:2468] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 23.
I20260709 12:24:03.954743 17686 raft_consensus.cc:1275] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 20 index: 12. Preceding OpId from leader: term: 21 index: 13. (index mismatch)
I20260709 12:24:03.885142 25354 leader_election.cc:290] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:03.960378 17762 raft_consensus.cc:1275] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
W20260709 12:24:03.963424 25370 consensus_meta.cc:287] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.557s	user 0.010s	sys 0.004s
W20260709 12:24:03.964929 25407 log.cc:927] Time spent T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.099s	user 0.000s	sys 0.001s
I20260709 12:24:03.976840 17688 raft_consensus.cc:1275] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 22 index: 15. Preceding OpId from leader: term: 23 index: 16. (index mismatch)
I20260709 12:24:03.977900 17727 leader_election.cc:304] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:03.987854 17755 raft_consensus.cc:2468] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 20.
I20260709 12:24:03.988034 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:03.991056 18363 raft_consensus.cc:2368] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for earlier term 23. Current term is 24.
I20260709 12:24:03.995087 17802 leader_election.cc:304] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:24:03.999416 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:24:03.999818 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 48052.4us
W20260709 12:24:04.000159 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:04.002310 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 18 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:04.004586 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:04.009640 25136 raft_consensus.cc:2804] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:24:04.022663 17802 leader_election.cc:304] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:04.023659 17680 raft_consensus.cc:3060] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:04.001441 25293 raft_consensus.cc:493] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:04.028319 25293 raft_consensus.cc:515] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:04.030941 25179 raft_consensus.cc:493] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:04.031304 25293 leader_election.cc:290] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:04.003258 25182 raft_consensus.cc:2804] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:24:04.031672 25179 raft_consensus.cc:515] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:04.035126 25067 raft_consensus.cc:2804] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:24:04.035297 24992 raft_consensus.cc:2804] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:24:04.036091 25297 raft_consensus.cc:2804] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:24:04.010429 25136 raft_consensus.cc:493] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:24:04.055248 17801 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:04.039250 17727 leader_election.cc:304] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:04.057638 17686 raft_consensus.cc:1275] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 19 index: 14. Preceding OpId from leader: term: 20 index: 15. (index mismatch)
I20260709 12:24:04.040648 17834 raft_consensus.cc:2393] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 18: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
W20260709 12:24:04.071722 17801 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:04.039834 25301 raft_consensus.cc:2804] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:24:04.038403 25182 raft_consensus.cc:493] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:04.084430 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 20 candidate_status { last_received { term: 18 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:04.081971 17829 raft_consensus.cc:2393] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 19: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
W20260709 12:24:04.094333 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:04.047772 24992 raft_consensus.cc:493] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:04.107512 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:04.095009 25182 raft_consensus.cc:3060] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:04.095178 25301 raft_consensus.cc:493] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:24:04.116215 25388 consensus_meta.cc:287] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.845s	user 0.006s	sys 0.012s
I20260709 12:24:04.116315 24989 leader_election.cc:290] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 24 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:04.117435 25388 raft_consensus.cc:515] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:04.118906 25179 leader_election.cc:290] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:04.119484 17833 raft_consensus.cc:1275] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 12 index: 9. Preceding OpId from leader: term: 18 index: 12. (index mismatch)
I20260709 12:24:04.119792 25278 raft_consensus.cc:3060] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:04.122870 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:24:04.124898 25387 consensus_meta.cc:287] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.869s	user 0.015s	sys 0.004s
I20260709 12:24:04.125573 25387 raft_consensus.cc:515] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:04.127089 17764 raft_consensus.cc:2393] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 17: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:24:04.129061 17758 raft_consensus.cc:2393] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 22: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:24:04.115628 17830 raft_consensus.cc:1240] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 23 LEADER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 22. Current term is 23. Ops: []
I20260709 12:24:04.117210 17687 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37396 (request call id 2904) took 1187 ms. Trace:
I20260709 12:24:04.042526 25067 raft_consensus.cc:493] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:24:04.052594 25297 raft_consensus.cc:493] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:04.066812 25136 raft_consensus.cc:3060] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 15 FOLLOWER]: Advancing to term 16
W20260709 12:24:04.137174 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:04.131590 17687 rpcz_store.cc:276] 0709 12:24:02.929704 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:02.929923 (+   219us) service_pool.cc:224] Handling call
0709 12:24:04.116734 (+1186811us) spinlock_profiling.cc:243] Waited 546 ms on lock 0x7b54004259d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:24:04.117122 (+   388us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1254691712}
I20260709 12:24:04.141701 17837 raft_consensus.cc:1275] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 16 index: 14. Preceding OpId from leader: term: 19 index: 16. (index mismatch)
W20260709 12:24:04.147239 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 221062us
I20260709 12:24:04.154395 17728 leader_election.cc:304] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
W20260709 12:24:04.155027 25378 log.cc:927] Time spent T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.071s	user 0.000s	sys 0.000s
W20260709 12:24:04.156217 25408 log.cc:927] Time spent T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.280s	user 0.000s	sys 0.002s
I20260709 12:24:04.156985 25315 consensus_queue.cc:1059] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 16.12, Next index: 13, Last known committed idx: 11, Time since last communication: 0.049s
I20260709 12:24:04.157502 17728 leader_election.cc:304] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:04.158308 25281 raft_consensus.cc:2749] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:24:04.160480 25326 raft_consensus.cc:493] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:04.161266 25326 raft_consensus.cc:515] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:04.164690 17803 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:04.168898 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:04.173226 25387 leader_election.cc:290] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 17 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:04.174304 25388 leader_election.cc:290] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 22 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:04.143321 25067 raft_consensus.cc:3060] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:04.109676 24992 raft_consensus.cc:3060] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:04.144747 17838 raft_consensus.cc:2468] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 19.
I20260709 12:24:04.183872 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:04.190738 17833 raft_consensus.cc:1275] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 20 index: 14. Preceding OpId from leader: term: 23 index: 16. (index mismatch)
I20260709 12:24:04.148042 25297 raft_consensus.cc:3060] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:04.195192 17803 leader_election.cc:304] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:04.141297 25301 raft_consensus.cc:3060] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:04.201930 17686 raft_consensus.cc:3060] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Advancing to term 20
W20260709 12:24:04.235653 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 69087.8us
I20260709 12:24:04.236747 17830 raft_consensus.cc:1275] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 14 index: 14. Preceding OpId from leader: term: 19 index: 16. (index mismatch)
I20260709 12:24:04.089155 25131 consensus_queue.cc:237] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 22.14, Last appended by leader: 14, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:04.249607 24801 raft_consensus.cc:2749] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:24:04.175786 25081 consensus_queue.cc:1048] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.049s
W20260709 12:24:04.243465 24642 consensus_meta.cc:287] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.548s	user 0.015s	sys 0.001s
W20260709 12:24:04.260550 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 288151us
W20260709 12:24:04.256140 25133 consensus_meta.cc:287] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.647s	user 0.017s	sys 0.000s
I20260709 12:24:04.250308 25263 leader_election.cc:290] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:04.260143 25326 leader_election.cc:290] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:04.296952 17688 raft_consensus.cc:1275] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 20 index: 13. Preceding OpId from leader: term: 21 index: 14. (index mismatch)
I20260709 12:24:04.311914 25388 raft_consensus.cc:2749] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:24:04.326503 17680 raft_consensus.cc:2468] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 21.
W20260709 12:24:04.327057 17655 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:04.328763 17655 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:04.360734 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:04.362228 17763 raft_consensus.cc:3060] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:04.308529 25387 raft_consensus.cc:2804] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:24:04.373787 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:04.376863 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:04.381103 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:04.384917 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:04.387229 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:04.387262 17762 raft_consensus.cc:2468] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 25.
W20260709 12:24:04.394610 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 397047us
I20260709 12:24:04.396211 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:04.402707 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:04.403764 17756 raft_consensus.cc:2393] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 20: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:24:04.405408 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:04.407141 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:04.408368 25246 raft_consensus.cc:2804] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:24:04.412698 17761 raft_consensus.cc:1275] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 14 index: 10. Preceding OpId from leader: term: 18 index: 12. (index mismatch)
I20260709 12:24:04.414072 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:04.414880 17761 raft_consensus.cc:3060] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:04.416155 24767 raft_consensus.cc:493] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:04.416874 24767 raft_consensus.cc:515] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:04.420552 17760 raft_consensus.cc:2468] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 22.
W20260709 12:24:04.420727 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 421064us
I20260709 12:24:04.421509 25420 raft_consensus.cc:493] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:04.421850 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:04.422132 25420 raft_consensus.cc:515] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:04.422971 25387 raft_consensus.cc:697] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
W20260709 12:24:04.423586 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 129488us
I20260709 12:24:04.424783 25387 consensus_queue.cc:237] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 22.13, Last appended by leader: 13, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:04.422659 17760 raft_consensus.cc:3060] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:24:04.400413 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 45866.1us
I20260709 12:24:04.384112 25081 raft_consensus.cc:493] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:24:04.439623 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:24:04.443645 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:04.450205 17686 raft_consensus.cc:2468] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 20.
W20260709 12:24:04.451772 25413 log.cc:927] Time spent T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.237s	user 0.001s	sys 0.002s
I20260709 12:24:04.452284 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:04.453994 17755 raft_consensus.cc:2468] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 19.
I20260709 12:24:04.419319 24767 leader_election.cc:290] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:04.444131 18334 leader_election.cc:304] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:24:04.455102 17802 leader_election.cc:304] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:24:04.458431 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 24 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:04.459089 17836 raft_consensus.cc:2468] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 23.
I20260709 12:24:04.447521 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 21 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:04.460575 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:04.453409 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:24:04.467010 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 309572us
W20260709 12:24:04.474036 17801 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:04.465360 24642 raft_consensus.cc:2804] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:24:04.475948 24642 raft_consensus.cc:493] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:04.476563 24642 raft_consensus.cc:3060] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:04.465186 25081 raft_consensus.cc:515] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:04.480472 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:04.470021 25388 consensus_queue.cc:1048] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.011s
I20260709 12:24:04.481267 17652 leader_election.cc:304] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:04.482661 25081 leader_election.cc:290] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:04.482999 25383 consensus_queue.cc:1048] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:24:04.483922 25133 raft_consensus.cc:493] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:04.484596 25421 consensus_queue.cc:1048] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.038s
I20260709 12:24:04.484593 25133 raft_consensus.cc:515] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:04.486344 25421 raft_consensus.cc:2804] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:24:04.487020 25421 raft_consensus.cc:697] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 21 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:24:04.443308 25326 consensus_queue.cc:1048] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.028s
W20260709 12:24:04.490864 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:04.438781 24801 consensus_queue.cc:1048] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.036s
I20260709 12:24:04.496327 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:04.497267 17755 raft_consensus.cc:3060] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:04.432564 25288 consensus_queue.cc:1048] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.040s
I20260709 12:24:04.484469 25151 raft_consensus.cc:2804] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:24:04.499293 17830 raft_consensus.cc:2393] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 19: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:24:04.499857 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:04.498137 17831 raft_consensus.cc:2468] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 21.
I20260709 12:24:04.443029 25182 raft_consensus.cc:515] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:04.499857 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:04.419390 25136 raft_consensus.cc:515] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:04.487741 25133 leader_election.cc:290] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:24:04.513365 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 45358.5us
I20260709 12:24:04.517787 17728 leader_election.cc:304] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:04.504154 25151 raft_consensus.cc:493] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:24:04.556962 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:24:04.565125 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 47429.5us
W20260709 12:24:04.556177 25264 consensus_meta.cc:287] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.574s	user 0.010s	sys 0.004s
I20260709 12:24:04.562197 18366 raft_consensus.cc:2468] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 21.
I20260709 12:24:04.568636 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:24:04.570590 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 134190us
I20260709 12:24:04.554247 25151 raft_consensus.cc:3060] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Advancing to term 22
W20260709 12:24:04.574759 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 102935us
W20260709 12:24:04.576916 17803 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.16.29.68:42577 timed out after 1.510s (SENT) [suppressed 14 similar messages]
I20260709 12:24:04.577464 17682 raft_consensus.cc:3055] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 21 LEADER]: Stepping down as leader of term 21
W20260709 12:24:04.612022 18333 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:04.633994 17760 raft_consensus.cc:2468] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 18.
I20260709 12:24:04.611544 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:04.592890 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 24 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:04.582031 17729 leader_election.cc:304] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:24:04.608714 17728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:04.642876 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 60951.3us
I20260709 12:24:04.611100 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
W20260709 12:24:04.579125 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 97922.8us
I20260709 12:24:04.603596 17757 raft_consensus.cc:2468] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 19.
I20260709 12:24:04.600057 18367 raft_consensus.cc:1275] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 21 index: 15. Preceding OpId from leader: term: 22 index: 16. (index mismatch)
W20260709 12:24:04.621727 25411 log.cc:927] Time spent T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.602s	user 0.002s	sys 0.002s
I20260709 12:24:04.606271 17756 raft_consensus.cc:2468] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 23.
I20260709 12:24:04.649398 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:04.574399 25264 raft_consensus.cc:2749] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:24:04.579596 17755 raft_consensus.cc:2468] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 20.
W20260709 12:24:04.653342 25426 log.cc:927] Time spent T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.072s	user 0.000s	sys 0.000s
I20260709 12:24:04.654214 25425 raft_consensus.cc:493] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:24:04.654848 25425 raft_consensus.cc:515] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:04.655365 18371 raft_consensus.cc:1240] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 19 LEADER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 18. Current term is 19. Ops: []
W20260709 12:24:04.657780 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 81954.3us
W20260709 12:24:04.641057 18333 connection.cc:256] RPC call timeout handler was delayed by -1.7144038120000005s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17880000000000001s for a response
W20260709 12:24:04.658442 18333 connection.cc:256] RPC call timeout handler was delayed by -1.6622038119999161s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18460000000000001s for a response
W20260709 12:24:04.660001 18333 connection.cc:256] RPC call timeout handler was delayed by -1.6469038119998913s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18630000000000002s for a response
W20260709 12:24:04.660426 18333 connection.cc:256] RPC call timeout handler was delayed by -1.4741038120000667s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20550000000000002s for a response
W20260709 12:24:04.660768 18333 connection.cc:256] RPC call timeout handler was delayed by -1.3373038119998455s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2207s for a response
I20260709 12:24:04.658308 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:04.659271 17681 raft_consensus.cc:1275] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 16 index: 15. Preceding OpId from leader: term: 18 index: 16. (index mismatch)
I20260709 12:24:04.607136 25422 raft_consensus.cc:3060] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:04.663039 17755 raft_consensus.cc:2393] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 19: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:24:04.656627 25136 leader_election.cc:290] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 16 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:04.671248 17761 raft_consensus.cc:2468] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 21.
I20260709 12:24:04.665707 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:04.675665 17686 raft_consensus.cc:3060] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:04.675624 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 18 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:04.666244 17682 raft_consensus.cc:740] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:24:04.562986 25246 raft_consensus.cc:697] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:24:04.677898 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 21 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:04.623286 25297 raft_consensus.cc:515] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:04.681476 18363 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59886 (request call id 1961) took 1175 ms. Trace:
I20260709 12:24:04.682542 17801 leader_election.cc:304] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:24:04.599633 24989 consensus_queue.cc:1048] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.067s
W20260709 12:24:04.688241 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 248193us
I20260709 12:24:04.651662 25420 leader_election.cc:290] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:04.685573 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:04.689605 17688 raft_consensus.cc:2393] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 20: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:24:04.685622 18363 rpcz_store.cc:276] 0709 12:24:03.506498 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:03.536111 (+ 29613us) service_pool.cc:224] Handling call
0709 12:24:04.681231 (+1145120us) spinlock_profiling.cc:243] Waited 429 ms on lock 0x7b54006ae0d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:24:04.681381 (+   150us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":985773440}
I20260709 12:24:04.692667 17652 leader_election.cc:304] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:04.694413 17756 raft_consensus.cc:2393] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 21: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:24:04.694712 25433 raft_consensus.cc:493] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:04.695441 25433 raft_consensus.cc:515] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:04.682858 25431 raft_consensus.cc:2804] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:24:04.694610 25434 raft_consensus.cc:493] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:04.697324 17761 raft_consensus.cc:1240] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 17. Current term is 18. Ops: []
W20260709 12:24:04.699783 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:04.703163 17764 raft_consensus.cc:2393] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 25: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:24:04.704873 17727 leader_election.cc:304] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:04.695815 25126 raft_consensus.cc:3060] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:04.707938 25383 raft_consensus.cc:2749] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
W20260709 12:24:04.713251 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 46388.3us
I20260709 12:24:04.713919 25297 leader_election.cc:290] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 25 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:04.704317 25387 raft_consensus.cc:493] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:04.719998 17727 leader_election.cc:304] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:04.712191 25432 raft_consensus.cc:493] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:04.700110 25431 raft_consensus.cc:493] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:24:04.707592 24992 raft_consensus.cc:515] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:04.699267 25425 leader_election.cc:290] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:04.663120 25067 raft_consensus.cc:515] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:04.734827 24992 leader_election.cc:290] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 24 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:04.735677 25067 leader_election.cc:290] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 21 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:04.740617 25428 consensus_queue.cc:1048] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:24:04.724582 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:04.743583 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:04.744021 17763 raft_consensus.cc:2468] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 18.
I20260709 12:24:04.744220 17681 raft_consensus.cc:3060] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:04.744379 17762 raft_consensus.cc:3060] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:04.744740 17759 raft_consensus.cc:1240] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 17. Current term is 18. Ops: []
I20260709 12:24:04.746578 25432 raft_consensus.cc:515] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:04.725987 25424 raft_consensus.cc:493] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:04.748544 25424 raft_consensus.cc:515] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:04.748890 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:04.743677 25267 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a5732e71b9d54f8786b8c511d69943bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:04.750171 25432 leader_election.cc:290] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:04.750450 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:04.751212 17763 raft_consensus.cc:2468] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 17.
I20260709 12:24:04.726058 25387 raft_consensus.cc:515] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:04.726713 25136 raft_consensus.cc:2749] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:24:04.727232 18361 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 2213) took 1113 ms. Trace:
I20260709 12:24:04.760030 18361 rpcz_store.cc:276] 0709 12:24:03.614253 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:03.615093 (+   840us) service_pool.cc:224] Handling call
0709 12:24:03.619991 (+  4898us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:03.620668 (+   677us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:03.620707 (+    39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:03.621526 (+   819us) log.cc:844] Serialized 26 byte log entry
0709 12:24:03.802452 (+180926us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:24:03.802495 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:03.802561 (+    66us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:04.621017 (+818456us) spinlock_profiling.cc:243] Waited 7.16 ms on lock 0x7b4400022b4c. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b47459c85 00007f7b4745a6a2 00007f7b4a1cc1db 00007f7b4a1c7aaf 00007f7b4a1bb86e 00007f7b4a1babb6 00007f7b4a1ba7d8 00007f7b4a1c4221 00007f7b4a1c41d9 00007f7b4a1c4169
0709 12:24:04.724365 (+103348us) raft_consensus.cc:1685] finished
0709 12:24:04.724395 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:04.727159 (+  2764us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":12063,"raft.run_cpu_time_us":276,"raft.run_wall_time_us":14515,"spinlock_wait_cycles":16474496,"thread_start_us":180331,"threads_started":1,"wal-append.queue_time_us":327564}
I20260709 12:24:04.727394 24801 raft_consensus.cc:2804] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:24:04.763162 24801 raft_consensus.cc:697] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 19 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:24:04.724538 25431 raft_consensus.cc:3060] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:04.764425 24801 consensus_queue.cc:237] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 17.11, Last appended by leader: 11, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:04.761708 25387 leader_election.cc:290] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:04.730638 25429 consensus_queue.cc:1048] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:24:04.628446 25421 consensus_queue.cc:237] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 19.12, Last appended by leader: 12, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:24:04.722250 25081 raft_consensus.cc:493] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:04.721912 25264 raft_consensus.cc:493] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:04.736449 25384 consensus_meta.cc:287] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 1.309s	user 0.019s	sys 0.004s
I20260709 12:24:04.705557 25182 leader_election.cc:290] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 18 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:24:04.753050 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:04.756071 17686 raft_consensus.cc:2468] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 16.
W20260709 12:24:04.722606 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 145661us
I20260709 12:24:04.757761 17762 raft_consensus.cc:2468] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 20.
I20260709 12:24:04.751542 25424 leader_election.cc:290] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:04.769047 25246 consensus_queue.cc:237] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 17.14, Last appended by leader: 14, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:04.729979 25433 leader_election.cc:290] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:04.705971 25434 raft_consensus.cc:515] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:04.767045 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:04.755420 17755 raft_consensus.cc:1275] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 17 index: 11. Preceding OpId from leader: term: 22 index: 13. (index mismatch)
W20260709 12:24:04.780246 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 51748.4us
I20260709 12:24:04.791138 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:04.795113 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:24:04.796427 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 76546.8us
W20260709 12:24:04.792371 25301 consensus_meta.cc:287] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.567s	user 0.010s	sys 0.003s
I20260709 12:24:04.792363 25136 raft_consensus.cc:2804] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:24:04.812218 25301 raft_consensus.cc:515] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:04.812548 25136 raft_consensus.cc:697] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 23 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:24:04.813261 17761 raft_consensus.cc:1240] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 21. Current term is 22. Ops: []
I20260709 12:24:04.814085 25136 consensus_queue.cc:237] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 20.14, Last appended by leader: 14, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:04.815163 25301 leader_election.cc:290] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 22 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:04.815997 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:04.792402 25081 raft_consensus.cc:515] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:04.818982 25081 leader_election.cc:290] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:04.820943 17681 raft_consensus.cc:2468] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 25.
I20260709 12:24:04.807989 24642 raft_consensus.cc:515] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:04.808764 25438 log.cc:927] Time spent T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.052s	user 0.000s	sys 0.001s
W20260709 12:24:04.827746 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 111510us
I20260709 12:24:04.809476 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:04.810959 25151 raft_consensus.cc:515] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:04.811228 17652 leader_election.cc:304] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:04.821800 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 24 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:04.806529 25434 leader_election.cc:290] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:04.808897 25323 raft_consensus.cc:493] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:04.828927 17755 raft_consensus.cc:2393] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 19: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
W20260709 12:24:04.836697 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 123050us
I20260709 12:24:04.839912 17688 raft_consensus.cc:2468] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 22.
W20260709 12:24:04.842357 17803 leader_election.cc:336] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 18 pre-election: RPC error from VoteRequest() call to peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Timed out: RequestConsensusVote RPC to 127.16.29.68:42577 timed out after 1.913s (SENT)
I20260709 12:24:04.842870 17803 leader_election.cc:304] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:04.847303 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:04.848728 17681 raft_consensus.cc:2468] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 22.
I20260709 12:24:04.845409 25126 raft_consensus.cc:2764] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Leader election decision vote started in defunct term 19: won
I20260709 12:24:04.794330 25383 consensus_queue.cc:1048] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 13, Time since last communication: 0.050s
I20260709 12:24:04.795958 25264 raft_consensus.cc:515] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:04.857049 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:04.859238 25345 consensus_queue.cc:1048] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 9, Time since last communication: 0.071s
I20260709 12:24:04.861387 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:04.861742 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:04.862253 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:04.862761 17685 raft_consensus.cc:3060] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:04.861747 25345 consensus_queue.cc:1243] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer e2909cff1da44ad387f96a8ae3aa5afb log is divergent from this leader: its last log entry 17.11 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 9
I20260709 12:24:04.855999 25281 raft_consensus.cc:3060] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:04.844250 25323 raft_consensus.cc:515] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:04.868156 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:04.868919 17831 raft_consensus.cc:3060] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:04.870234 17683 raft_consensus.cc:2410] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 20 because replica has last-logged OpId of term: 18 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 14.
I20260709 12:24:04.858579 17801 leader_election.cc:304] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:04.885528 17685 raft_consensus.cc:2468] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 24.
I20260709 12:24:04.886798 18333 leader_election.cc:304] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:24:04.908723 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:04.909914 17834 raft_consensus.cc:3060] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:04.861454 25263 raft_consensus.cc:3055] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:24:04.917593 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 19 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:04.848502 25433 consensus_queue.cc:1048] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.039s
W20260709 12:24:04.920106 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 68657.8us
I20260709 12:24:04.921842 17681 raft_consensus.cc:2410] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 19 because replica has last-logged OpId of term: 17 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 12.
I20260709 12:24:04.922465 17830 raft_consensus.cc:1240] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 16. Current term is 17. Ops: []
I20260709 12:24:04.922959 24801 raft_consensus.cc:2804] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:24:04.920526 17728 leader_election.cc:304] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
W20260709 12:24:04.962503 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 161662us
I20260709 12:24:04.925890 25182 raft_consensus.cc:2804] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:24:04.923470 25445 consensus_queue.cc:1048] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.080s
W20260709 12:24:05.011768 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:05.012255 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 173760us
I20260709 12:24:04.921077 25263 raft_consensus.cc:740] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:24:05.014952 25281 raft_consensus.cc:2749] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:24:05.024780 17831 raft_consensus.cc:2468] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 17.
I20260709 12:24:05.024036 17801 leader_election.cc:304] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:05.026752 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:05.027623 17830 raft_consensus.cc:3060] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:05.030364 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 22 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:05.032403 17652 leader_election.cc:304] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:24:05.020943 25429 raft_consensus.cc:493] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:05.033831 25441 log.cc:927] Time spent T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.173s	user 0.000s	sys 0.001s
I20260709 12:24:05.034128 25429 raft_consensus.cc:515] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:05.037225 17727 leader_election.cc:304] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:05.031041 17833 raft_consensus.cc:2468] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 21.
W20260709 12:24:05.038527 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 151802us
I20260709 12:24:05.039677 18333 leader_election.cc:304] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:24:05.040334 25435 consensus_queue.cc:1048] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 12:24:05.040805 25264 leader_election.cc:290] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:05.041220 25081 consensus_queue.cc:1059] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 16.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20260709 12:24:05.042229 25447 raft_consensus.cc:2804] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:24:05.042872 25422 raft_consensus.cc:2749] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:24:05.043673 17652 leader_election.cc:304] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:24:05.043779 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 123338us
I20260709 12:24:05.017824 25182 raft_consensus.cc:493] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:05.044080 25384 raft_consensus.cc:2764] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Leader election decision vote started in defunct term 23: won
I20260709 12:24:05.075987 25182 raft_consensus.cc:3060] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:05.076522 25323 leader_election.cc:290] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:05.077908 17682 consensus_queue.cc:260] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 21.13, Last appended by leader: 13, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:05.079037 17682 raft_consensus.cc:3060] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:05.039731 24642 leader_election.cc:290] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 25 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:05.043819 17801 leader_election.cc:304] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:05.082630 25326 raft_consensus.cc:2804] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:24:05.083263 25326 raft_consensus.cc:697] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 20 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:24:05.071600 25447 raft_consensus.cc:697] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 18 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:24:05.049991 25263 consensus_queue.cc:260] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.12, Last appended by leader: 12, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:05.050326 25431 raft_consensus.cc:515] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:05.055357 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:05.020895 24801 raft_consensus.cc:697] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 19 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:24:05.057127 25081 raft_consensus.cc:3055] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 LEADER]: Stepping down as leader of term 16
W20260709 12:24:05.057993 25412 log.cc:927] Time spent T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.929s	user 0.006s	sys 0.000s
I20260709 12:24:05.029803 25246 raft_consensus.cc:2749] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:24:05.060137 25301 raft_consensus.cc:2804] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:24:05.114327 25301 raft_consensus.cc:697] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 17 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:24:05.114602 18364 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 2214) took 1486 ms. Trace:
I20260709 12:24:05.115352 25301 consensus_queue.cc:237] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.11, Last appended by leader: 11, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:05.115010 18364 rpcz_store.cc:276] 0709 12:24:03.627825 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:03.641539 (+ 13714us) service_pool.cc:224] Handling call
0709 12:24:03.692975 (+ 51436us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:03.694074 (+  1099us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 12:24:03.694570 (+   496us) log.cc:844] Serialized 14 byte log entry
0709 12:24:03.873599 (+179029us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:03.900204 (+ 26605us) log.cc:844] Serialized 26 byte log entry
0709 12:24:03.900411 (+   207us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:24:03.900453 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:03.900554 (+   101us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:03.943791 (+ 43237us) spinlock_profiling.cc:243] Waited 1.59 ms on lock 0x7b180004b5e0. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b475333f3 00007f7b475357c1 00007f7b47535779 00007f7b47535709 00007f7b475356d1 00007f7b475349cd 0000000000388634 0000000000388479 00007f7b4751683a
0709 12:24:05.057631 (+1113840us) spinlock_profiling.cc:243] Waited 30 ms on lock 0x7b2400308408. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b473b99c4 00007f7b473b7acb 00007f7b47572dc9 00007f7b47572a55 00007f7b4757209d 00007f7b4a1a410e 00007f7b4a1bb88c 00007f7b4a1babb6 00007f7b4a1ba7d8 00007f7b4a1c4221
0709 12:24:05.114256 (+ 56625us) raft_consensus.cc:1685] finished
0709 12:24:05.114278 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:05.114429 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4487,"prepare.queue_time_us":9046,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":42,"raft.queue_time_us":8885,"raft.run_cpu_time_us":333,"raft.run_wall_time_us":15346,"spinlock_wait_cycles":72634624,"thread_start_us":178070,"threads_started":1,"wal-append.queue_time_us":401036}
I20260709 12:24:05.060467 17758 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38786 (request call id 3600) took 1494 ms. Trace:
I20260709 12:24:05.060880 25435 raft_consensus.cc:493] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:05.117770 17758 rpcz_store.cc:276] 0709 12:24:03.566375 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:03.573693 (+  7318us) service_pool.cc:224] Handling call
0709 12:24:05.060212 (+1486519us) spinlock_profiling.cc:243] Waited 549 ms on lock 0x7b5400216e54. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:24:05.060380 (+   168us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1262793472}
I20260709 12:24:05.118364 25435 raft_consensus.cc:515] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:05.119433 17758 raft_consensus.cc:1240] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 19. Current term is 20. Ops: []
I20260709 12:24:05.120654 17758 raft_consensus.cc:1240] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 21. Current term is 22. Ops: []
I20260709 12:24:05.121691 17758 raft_consensus.cc:1240] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 16. Current term is 17. Ops: []
I20260709 12:24:05.121940 25435 leader_election.cc:290] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:05.122655 17758 raft_consensus.cc:1240] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 18. Current term is 19. Ops: []
I20260709 12:24:05.061194 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 24 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:05.063138 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:05.068629 17727 leader_election.cc:304] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:05.124193 17833 raft_consensus.cc:2468] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 23.
W20260709 12:24:05.124747 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 56240.3us
I20260709 12:24:05.125164 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:05.125435 25246 raft_consensus.cc:2804] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:24:05.125876 25246 raft_consensus.cc:493] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:05.126461 25246 raft_consensus.cc:3060] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:05.128594 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:05.129869 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:05.130349 25433 raft_consensus.cc:493] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:05.084632 25326 consensus_queue.cc:237] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 18.15, Last appended by leader: 15, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:05.130868 25433 raft_consensus.cc:515] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:24:05.047544 17803 leader_election.cc:336] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 20 pre-election: RPC error from VoteRequest() call to peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Timed out: RequestConsensusVote RPC to 127.16.29.68:42577 timed out after 1.568s (SENT)
W20260709 12:24:05.085546 25446 log.cc:927] Time spent T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.080s	user 0.002s	sys 0.000s
I20260709 12:24:05.092695 17760 raft_consensus.cc:1240] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 14. Current term is 16. Ops: []
I20260709 12:24:05.131850 17803 leader_election.cc:304] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:05.092612 17728 leader_election.cc:304] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
W20260709 12:24:05.132611 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 92937.7us
I20260709 12:24:05.133175 25433 leader_election.cc:290] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:05.133749 25433 raft_consensus.cc:2749] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:24:05.084059 25323 raft_consensus.cc:2749] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:24:05.102504 24642 raft_consensus.cc:2804] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:24:05.103266 17762 raft_consensus.cc:1240] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 21. Current term is 22. Ops: []
I20260709 12:24:05.106107 17802 leader_election.cc:304] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:24:05.101877 25263 raft_consensus.cc:3060] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:05.108276 25081 raft_consensus.cc:740] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:24:05.113540 17763 raft_consensus.cc:2468] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 21.
I20260709 12:24:05.124243 17831 raft_consensus.cc:2468] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 23.
I20260709 12:24:05.129942 25444 raft_consensus.cc:493] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:05.135454 25264 raft_consensus.cc:2749] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:24:05.143211 25444 raft_consensus.cc:515] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:05.134261 17728 leader_election.cc:304] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:05.134719 24642 raft_consensus.cc:493] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:05.128983 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:05.135572 17802 leader_election.cc:304] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:24:05.109333 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:05.147500 17836 raft_consensus.cc:2468] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 17.
I20260709 12:24:05.147845 17830 raft_consensus.cc:2468] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 21.
W20260709 12:24:05.091818 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 48292.8us
I20260709 12:24:05.146015 17728 leader_election.cc:304] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:05.148809 17652 leader_election.cc:304] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:05.149459 17728 leader_election.cc:304] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:05.150725 24642 raft_consensus.cc:3060] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:05.151296 25264 raft_consensus.cc:2749] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:24:05.152037 25264 raft_consensus.cc:2804] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:24:05.152518 25264 raft_consensus.cc:697] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:24:05.153409 25264 consensus_queue.cc:237] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.9, Last appended by leader: 9, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:05.150398 25081 consensus_queue.cc:260] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 16.13, Last appended by leader: 13, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:05.155202 17727 leader_election.cc:304] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
W20260709 12:24:05.156944 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 117284us
I20260709 12:24:05.158108 17728 leader_election.cc:304] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:05.158329 17834 raft_consensus.cc:2468] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 16.
I20260709 12:24:05.160967 25323 raft_consensus.cc:2804] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:24:05.161428 17833 raft_consensus.cc:1275] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 22 index: 13. Preceding OpId from leader: term: 23 index: 14. (index mismatch)
I20260709 12:24:05.161502 25323 raft_consensus.cc:493] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:24:05.161972 25323 raft_consensus.cc:3060] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:05.162635 25424 raft_consensus.cc:2804] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:24:05.163269 25387 raft_consensus.cc:2804] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:24:05.163743 25387 raft_consensus.cc:493] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:05.164184 25387 raft_consensus.cc:3060] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:05.165257 25447 consensus_queue.cc:237] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.13, Last appended by leader: 13, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:05.167330 17727 leader_election.cc:304] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:24:05.167623 17652 leader_election.cc:304] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:05.185168 25081 raft_consensus.cc:3060] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:05.147245 25281 raft_consensus.cc:2804] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:24:05.192499 25439 raft_consensus.cc:2804] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:24:05.193404 17727 leader_election.cc:304] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:05.197714 25444 leader_election.cc:290] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:05.160903 25431 leader_election.cc:290] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 22 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:05.168071 25067 raft_consensus.cc:2749] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
W20260709 12:24:05.212684 17728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:05.168792 25433 raft_consensus.cc:2804] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:24:05.209493 25136 consensus_queue.cc:1048] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.002s
I20260709 12:24:05.215981 17758 raft_consensus.cc:1240] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 17 LEADER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 16. Current term is 17. Ops: []
W20260709 12:24:05.215977 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 58158us
I20260709 12:24:05.187089 25424 raft_consensus.cc:697] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 25 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:24:05.232209 25420 raft_consensus.cc:493] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:05.231726 25424 consensus_queue.cc:237] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 23.16, Last appended by leader: 16, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:05.232879 25420 raft_consensus.cc:515] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:05.235569 17683 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37402 (request call id 3298) took 4218 ms (client timeout 2628 ms). Trace:
I20260709 12:24:05.236133 25429 leader_election.cc:290] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:05.200302 25281 raft_consensus.cc:493] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:05.237114 25281 raft_consensus.cc:3060] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:05.237604 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:05.238472 18367 raft_consensus.cc:2393] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 25: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
W20260709 12:24:05.238677 17655 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:05.239161 17655 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:05.209838 25297 raft_consensus.cc:2804] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:24:05.208315 25179 raft_consensus.cc:2804] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:24:05.237627 17689 raft_consensus.cc:1275] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 18 index: 15. Preceding OpId from leader: term: 19 index: 16. (index mismatch)
I20260709 12:24:05.208506 25439 raft_consensus.cc:493] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:24:05.220746 25417 log.cc:927] Time spent T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 1.184s	user 0.011s	sys 0.000s
I20260709 12:24:05.221033 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
W20260709 12:24:05.225314 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 143230us
I20260709 12:24:05.228295 24801 consensus_queue.cc:237] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.11, Last appended by leader: 11, Current term: 19, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:05.229732 25131 raft_consensus.cc:493] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:05.230394 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 62869.2us
I20260709 12:24:05.249347 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:05.250365 17832 raft_consensus.cc:2468] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 19.
I20260709 12:24:05.250444 17652 leader_election.cc:304] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:05.251338 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 22 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:05.231493 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 18 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:05.252167 18364 raft_consensus.cc:2468] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 20.
I20260709 12:24:05.253338 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:05.254673 25301 raft_consensus.cc:493] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:24:05.255241 25301 raft_consensus.cc:515] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:05.255405 25067 consensus_queue.cc:1048] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20260709 12:24:05.219676 25433 raft_consensus.cc:697] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 24 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:24:05.261849 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 24 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:24:05.240538 17728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:05.219313 17682 raft_consensus.cc:1275] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 21 index: 13. Preceding OpId from leader: term: 22 index: 13. (term mismatch)
I20260709 12:24:05.245131 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:24:05.247025 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:05.236055 17683 rpcz_store.cc:269] 0709 12:24:01.017383 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:01.024202 (+  6819us) service_pool.cc:224] Handling call
0709 12:24:05.235298 (+4211096us) spinlock_profiling.cc:243] Waited 3.84 s on lock 0x7b54000c4c54. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:24:05.235468 (+   170us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":8832947584}
W20260709 12:24:05.263761 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:05.232461 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:05.264894 18371 raft_consensus.cc:2468] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 20.
I20260709 12:24:05.247617 18333 leader_election.cc:304] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:05.257652 25439 raft_consensus.cc:3060] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:05.264501 17682 pending_rounds.cc:85] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3: Aborting all ops after (but not including) 12
I20260709 12:24:05.276229 17729 leader_election.cc:400] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 23 election: Vote denied by peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577) with higher term. Message: Invalid argument: T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for earlier term 23. Current term is 24.
I20260709 12:24:05.276620 17682 pending_rounds.cc:107] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3: Aborting uncommitted NO_OP operation due to leader change: 21.13
I20260709 12:24:05.277143 17682 raft_consensus.cc:2889] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:05.263144 17684 raft_consensus.cc:2468] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 20.
I20260709 12:24:05.280319 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 22 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:05.287556 17835 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35040 (request call id 3142) took 1452 ms. Trace:
I20260709 12:24:05.239848 25387 raft_consensus.cc:515] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:05.292817 25301 leader_election.cc:290] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:05.293862 25387 leader_election.cc:290] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 24 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:05.294898 25433 consensus_queue.cc:237] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 22.12, Last appended by leader: 12, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:05.296607 18371 raft_consensus.cc:1240] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 23. Current term is 24. Ops: []
I20260709 12:24:05.296322 24642 raft_consensus.cc:515] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:05.263442 25297 raft_consensus.cc:697] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 21 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:24:05.299484 24642 leader_election.cc:290] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 23 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:05.299078 25297 consensus_queue.cc:237] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 20.15, Last appended by leader: 15, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:05.304514 17801 leader_election.cc:304] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:05.252261 25429 raft_consensus.cc:2749] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:24:05.305320 17756 raft_consensus.cc:1275] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 20 index: 12. Preceding OpId from leader: term: 21 index: 13. (index mismatch)
W20260709 12:24:05.307765 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 66492.8us
W20260709 12:24:05.317564 17730 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:05.332162 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 66853.5us
I20260709 12:24:05.317472 18333 leader_election.cc:304] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:24:05.249601 25131 raft_consensus.cc:515] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:05.336638 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:05.288233 17835 rpcz_store.cc:276] 0709 12:24:03.835247 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:03.835409 (+   162us) service_pool.cc:224] Handling call
0709 12:24:03.835796 (+   387us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:03.838886 (+  3090us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:03.838921 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:03.839589 (+   668us) log.cc:844] Serialized 26 byte log entry
0709 12:24:03.883298 (+ 43709us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:24:03.883344 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:03.883418 (+    74us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:05.219762 (+1336344us) spinlock_profiling.cc:243] Waited 31.2 ms on lock 0x7b4400022b4c. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b47459c85 00007f7b4745a6a2 00007f7b4a1cc1db 00007f7b4a1c7aaf 00007f7b4a1bb86e 00007f7b4a1babb6 00007f7b4a1ba7d8 00007f7b4a1c4221 00007f7b4a1c41d9 00007f7b4a1c4169
0709 12:24:05.286404 (+ 66642us) raft_consensus.cc:1685] finished
0709 12:24:05.286991 (+   587us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:05.287456 (+   465us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":152,"raft.run_cpu_time_us":108,"raft.run_wall_time_us":108,"spinlock_wait_cycles":71721856,"thread_start_us":43286,"threads_started":1,"wal-append.queue_time_us":196516}
I20260709 12:24:05.345973 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:05.346834 18366 raft_consensus.cc:3060] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:05.265753 25179 raft_consensus.cc:697] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 20 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:24:05.265642 25420 leader_election.cc:290] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:05.360158 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:05.361625 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:05.349418 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:24:05.351117 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 94089.9us
I20260709 12:24:05.337617 25281 raft_consensus.cc:515] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:05.378602 17729 leader_election.cc:304] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:05.381888 18361 raft_consensus.cc:2393] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 24: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:24:05.299507 25323 raft_consensus.cc:515] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:05.363399 25126 raft_consensus.cc:493] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:05.397665 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 80288.1us
I20260709 12:24:05.397753 25264 raft_consensus.cc:493] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:05.398135 25448 raft_consensus.cc:493] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:05.398588 25435 raft_consensus.cc:493] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:05.401124 25151 leader_election.cc:290] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 22 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:05.393277 25420 raft_consensus.cc:2749] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:24:05.407824 25179 consensus_queue.cc:237] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 19.15, Last appended by leader: 15, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:05.373287 24642 raft_consensus.cc:493] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:05.410502 17652 leader_election.cc:304] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:05.373641 25428 raft_consensus.cc:2804] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Leader election won for term 18
W20260709 12:24:05.412730 17655 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:05.390556 17729 leader_election.cc:304] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:24:05.414729 18333 leader_election.cc:304] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:05.333889 25246 raft_consensus.cc:515] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:05.350198 25345 consensus_queue.cc:1059] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 21.15, Next index: 16, Last known committed idx: 13, Time since last communication: 0.046s
I20260709 12:24:05.426608 25264 raft_consensus.cc:515] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:05.505553 25345 raft_consensus.cc:3055] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:24:05.506003 25345 raft_consensus.cc:740] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:24:05.440412 25433 raft_consensus.cc:2749] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:24:05.507225 25264 leader_election.cc:290] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:05.506920 25345 consensus_queue.cc:260] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 21.15, Last appended by leader: 15, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:05.420641 17832 raft_consensus.cc:2393] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 24: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
W20260709 12:24:05.508381 25459 log.cc:927] Time spent T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.093s	user 0.000s	sys 0.000s
I20260709 12:24:05.508926 25151 consensus_queue.cc:1059] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 16.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 12:24:05.509758 25151 consensus_queue.cc:1243] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer e2909cff1da44ad387f96a8ae3aa5afb log is divergent from this leader: its last log entry 17.12 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 11
I20260709 12:24:05.430934 25448 raft_consensus.cc:515] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:05.510313 25151 raft_consensus.cc:3055] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:24:05.374989 25354 consensus_queue.cc:1059] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 16.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.048s
I20260709 12:24:05.514655 25354 raft_consensus.cc:3055] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:24:05.515134 25354 raft_consensus.cc:740] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:24:05.516278 25354 consensus_queue.cc:260] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.12, Last appended by leader: 12, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:05.451352 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 60932.1us
I20260709 12:24:05.475116 17681 raft_consensus.cc:1240] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 21. Current term is 22. Ops: []
I20260709 12:24:05.475970 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:05.528203 17685 raft_consensus.cc:3060] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:05.477234 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:05.479753 17684 raft_consensus.cc:1275] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 17 index: 11. Preceding OpId from leader: term: 18 index: 12. (index mismatch)
I20260709 12:24:05.481302 17802 leader_election.cc:304] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
W20260709 12:24:05.531075 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 49903.4us
I20260709 12:24:05.531409 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:05.484117 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
W20260709 12:24:05.503177 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 198767us
I20260709 12:24:05.443548 18333 leader_election.cc:304] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:05.416651 25434 raft_consensus.cc:2804] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:24:05.519627 17680 raft_consensus.cc:1240] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 17. Current term is 18. Ops: []
I20260709 12:24:05.510702 25151 raft_consensus.cc:740] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:24:05.450549 25370 raft_consensus.cc:2804] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:24:05.551724 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 23 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:05.556619 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:05.558560 17757 raft_consensus.cc:1217] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Deduplicated request from leader. Original: 13.9->[15.10-21.13]   Dedup: 15.10->[18.11-21.13]
I20260709 12:24:05.559115 17757 raft_consensus.cc:1240] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 21. Current term is 22. Ops: [15.10-21.13]
I20260709 12:24:05.549201 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:05.559973 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:05.517295 25431 raft_consensus.cc:2764] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Leader pre-election decision vote started in defunct term 19: won
W20260709 12:24:05.563642 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 120196us
I20260709 12:24:05.550184 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:05.573325 18371 raft_consensus.cc:2468] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 20.
I20260709 12:24:05.567732 17687 raft_consensus.cc:2393] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 25: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:24:05.422434 25126 raft_consensus.cc:515] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:05.582571 18365 raft_consensus.cc:3055] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:24:05.558773 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:24:05.569295 17652 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:05.526446 25428 raft_consensus.cc:697] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 18 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:24:05.571332 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:05.542677 24992 raft_consensus.cc:2804] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:24:05.568279 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:05.583822 18367 raft_consensus.cc:2468] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 20.
I20260709 12:24:05.570089 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 21 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:05.596439 17762 raft_consensus.cc:2468] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 20.
I20260709 12:24:05.582635 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
W20260709 12:24:05.546788 25442 log.cc:927] Time spent T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.104s	user 0.001s	sys 0.000s
I20260709 12:24:05.582918 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:05.503643 25439 raft_consensus.cc:515] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:05.574805 17689 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44898 (request call id 1461) took 2866 ms. Trace:
I20260709 12:24:05.582986 18365 raft_consensus.cc:740] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:24:05.607208 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:05.536756 25323 leader_election.cc:290] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 18 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:05.441346 25435 raft_consensus.cc:515] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:05.613659 17759 raft_consensus.cc:1240] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 21. Current term is 22. Ops: [21.12-21.12]
I20260709 12:24:05.597851 17760 raft_consensus.cc:3060] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:05.601897 17764 raft_consensus.cc:3060] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:05.599305 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:05.518998 24642 raft_consensus.cc:515] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:05.549086 25370 raft_consensus.cc:493] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:24:05.620188 17684 raft_consensus.cc:2468] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 20.
I20260709 12:24:05.607288 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 22 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:24:05.589534 25421 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.548s	user 0.018s	sys 0.013s
I20260709 12:24:05.606757 17689 rpcz_store.cc:276] 0709 12:24:02.695039 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:02.695166 (+   127us) service_pool.cc:224] Handling call
0709 12:24:05.237085 (+2541919us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:05.243042 (+  5957us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:05.539807 (+296765us) spinlock_profiling.cc:243] Waited 2.54 s on lock 0x7b54000c4c50. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a61f8ad 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:24:05.561827 (+ 22020us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5844998528}
I20260709 12:24:05.577863 25448 leader_election.cc:290] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:05.610040 18333 leader_election.cc:304] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:05.621249 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 23 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:05.648249 17688 raft_consensus.cc:2410] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for term 23 because replica has last-logged OpId of term: 20 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 11.
W20260709 12:24:05.622130 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 81263.4us
I20260709 12:24:05.626026 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:05.649148 17681 raft_consensus.cc:3060] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Advancing to term 24
I20260709 12:24:05.568519 25444 raft_consensus.cc:493] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
W20260709 12:24:05.605388 25182 consensus_meta.cc:287] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.528s	user 0.009s	sys 0.004s
I20260709 12:24:05.629563 17680 raft_consensus.cc:2393] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 22: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:24:05.618659 17685 raft_consensus.cc:2468] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 24.
I20260709 12:24:05.632246 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:05.635978 25436 raft_consensus.cc:2804] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:24:05.636896 25126 leader_election.cc:290] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:05.628803 18365 consensus_queue.cc:260] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.15, Last appended by leader: 15, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:05.637708 25151 consensus_queue.cc:260] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.11, Last appended by leader: 11, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:24:05.639195 25462 log.cc:927] Time spent T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.054s	user 0.000s	sys 0.000s
I20260709 12:24:05.605340 25293 raft_consensus.cc:2749] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
W20260709 12:24:05.640648 25460 log.cc:927] Time spent T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.063s	user 0.000s	sys 0.000s
I20260709 12:24:05.640791 25345 raft_consensus.cc:3060] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:05.545945 25424 consensus_queue.cc:1059] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.063s
I20260709 12:24:05.641642 25353 raft_consensus.cc:493] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:05.642493 25246 leader_election.cc:290] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 22 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:05.590173 25434 raft_consensus.cc:697] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 19 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:24:05.630367 25370 raft_consensus.cc:3060] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:05.644505 25281 leader_election.cc:290] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 24 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:05.625300 25354 raft_consensus.cc:3060] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:05.601181 24992 raft_consensus.cc:493] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:05.600674 25136 raft_consensus.cc:493] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:24:05.549254 25464 consensus_queue.cc:1048] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.086s
I20260709 12:24:05.657682 18366 raft_consensus.cc:2468] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 22.
I20260709 12:24:05.720630 25267 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.StartTabletCopy from 127.0.0.1:59886 (request call id 1941) took 6056 ms. Trace:
I20260709 12:24:05.661341 17689 raft_consensus.cc:1240] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 23. Current term is 24. Ops: []
I20260709 12:24:05.661387 17686 raft_consensus.cc:2468] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 18.
I20260709 12:24:05.666240 17838 raft_consensus.cc:1275] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 17 index: 14. (index mismatch)
W20260709 12:24:05.629161 25421 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.588s	user 0.019s	sys 0.013s
W20260709 12:24:05.667138 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 78905.6us
I20260709 12:24:05.667784 17764 raft_consensus.cc:2468] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 22.
I20260709 12:24:05.670331 18365 raft_consensus.cc:3060] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:05.671166 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:05.675436 25151 raft_consensus.cc:3060] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:05.678927 25428 consensus_queue.cc:237] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 17.14, Last appended by leader: 14, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:05.680927 25439 leader_election.cc:290] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 24 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:05.682715 25435 leader_election.cc:290] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:05.684163 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:05.783087 17835 raft_consensus.cc:2393] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 16: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:24:05.712311 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 21 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:24:05.714291 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:05.715104 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 22 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:05.784559 18367 raft_consensus.cc:3060] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Advancing to term 22
I20260709 12:24:05.715631 17729 leader_election.cc:304] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:24:05.785495 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 69951.1us
I20260709 12:24:05.787079 17729 leader_election.cc:304] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:24:05.716503 17688 raft_consensus.cc:1275] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 22 index: 13. Preceding OpId from leader: term: 23 index: 14. (index mismatch)
I20260709 12:24:05.684640 24642 leader_election.cc:290] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:05.720943 17760 raft_consensus.cc:2468] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 18.
W20260709 12:24:05.721772 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 72553.9us
W20260709 12:24:05.790525 17655 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:05.727298 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:05.791263 17832 raft_consensus.cc:2468] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 19.
I20260709 12:24:05.735316 25445 raft_consensus.cc:2804] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Leader election won for term 18
W20260709 12:24:05.740537 25463 log.cc:927] Time spent T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.192s	user 0.000s	sys 0.001s
I20260709 12:24:05.740208 17585 catalog_manager.cc:5697] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 reported cstate change: term changed from 5 to 10. New cstate: current_term: 10 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: UNKNOWN } } }
I20260709 12:24:05.747864 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:05.670148 25444 raft_consensus.cc:515] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:05.752826 17681 raft_consensus.cc:2468] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 24.
I20260709 12:24:05.773056 18371 raft_consensus.cc:3055] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:24:05.668754 25182 raft_consensus.cc:515] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:05.725687 25446 log.cc:927] Time spent T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.059s	user 0.000s	sys 0.000s
I20260709 12:24:05.759482 24992 raft_consensus.cc:3060] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:05.800501 18371 raft_consensus.cc:740] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:24:05.809039 17729 leader_election.cc:304] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:24:05.772354 25131 leader_election.cc:290] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:05.814848 25439 raft_consensus.cc:493] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:05.733440 25353 raft_consensus.cc:515] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:05.765038 25267 rpcz_store.cc:276] 0709 12:23:59.593495 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:59.677239 (+ 83744us) service_pool.cc:224] Handling call
0709 12:24:00.225938 (+548699us) ts_tablet_manager.cc:934] T a5732e71b9d54f8786b8c511d69943bf P 334a4e1f59944544b3b3ff36616b57da: Initiating tablet copy from peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
0709 12:24:04.728899 (+4502961us) tablet_copy_client.cc:368] Tablet copy session begun
0709 12:24:05.088731 (+359832us) tablet_metadata.cc:665] Metadata flushed
0709 12:24:05.088889 (+   158us) tablet_copy_client.cc:497] Wrote new tablet metadata
0709 12:24:05.387608 (+298719us) tablet_copy_client.cc:418] Wrote new consensus metadata
0709 12:24:05.544502 (+156894us) tablet_replica.cc:206] Creating consensus instance
0709 12:24:05.649774 (+105272us) ts_tablet_manager.cc:989] Replica registered.
0709 12:24:05.649947 (+   173us) inbound_call.cc:177] Queueing success response
Metrics: {"dns_us":22933,"spinlock_wait_cycles":1280,"tablet-copy.queue_time_us":189801,"thread_start_us":381423,"threads_started":2}
W20260709 12:24:05.830349 17655 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:05.797825 25424 raft_consensus.cc:2804] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Leader pre-election won for term 21
W20260709 12:24:05.829346 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 216871us
I20260709 12:24:05.839133 18374 raft_consensus.cc:1240] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 25. Current term is 26. Ops: []
I20260709 12:24:05.813387 25301 raft_consensus.cc:493] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:05.815907 25315 raft_consensus.cc:2804] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:24:05.815768 25420 raft_consensus.cc:2804] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:24:05.839459 18371 consensus_queue.cc:260] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 22.15, Last appended by leader: 15, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:05.853106 17683 raft_consensus.cc:1240] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 18. Current term is 19. Ops: [18.15-18.15]
I20260709 12:24:05.813633 25323 raft_consensus.cc:493] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
W20260709 12:24:05.854432 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:05.854460 18333 leader_election.cc:304] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:05.855786 17801 leader_election.cc:304] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:05.859856 18371 raft_consensus.cc:3060] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:05.836099 25067 raft_consensus.cc:493] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:05.834702 25179 raft_consensus.cc:493] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:05.795740 25136 raft_consensus.cc:515] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:05.839025 25436 raft_consensus.cc:697] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 20 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:24:05.841723 25424 raft_consensus.cc:493] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:05.842350 25445 raft_consensus.cc:697] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 18 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:24:05.850879 25315 raft_consensus.cc:493] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:24:05.862679 17585 catalog_manager.cc:5697] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 reported cstate change: term changed from 9 to 12, leader changed from e063541542a94969898868bf6eb5ce1b (127.16.29.67) to 0da169ff82674dc1947055ce43de68a3 (127.16.29.65). New cstate: current_term: 12 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: UNKNOWN } } }
I20260709 12:24:05.835122 25447 consensus_queue.cc:1048] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 9, Time since last communication: 0.029s
W20260709 12:24:05.935470 17728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:05.862648 25420 raft_consensus.cc:697] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:24:05.842522 25439 raft_consensus.cc:515] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:05.959525 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:05.964119 17587 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:37172 (request call id 258) took 18245 ms (client timeout 15000 ms). Trace:
I20260709 12:24:05.964906 18366 raft_consensus.cc:1275] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 23 index: 16. Preceding OpId from leader: term: 25 index: 17. (index mismatch)
I20260709 12:24:05.842862 25476 consensus_queue.cc:1059] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 17.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.172s
I20260709 12:24:05.960484 17585 catalog_manager.cc:5697] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 reported cstate change: term changed from 3 to 11, leader changed from e063541542a94969898868bf6eb5ce1b (127.16.29.67) to 0da169ff82674dc1947055ce43de68a3 (127.16.29.65). New cstate: current_term: 11 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: UNKNOWN } } }
I20260709 12:24:05.908682 25433 raft_consensus.cc:3060] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:05.908806 25424 raft_consensus.cc:3060] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:05.911331 25293 raft_consensus.cc:2749] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:24:05.837746 25434 consensus_queue.cc:237] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:05.853353 25301 raft_consensus.cc:515] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:05.947223 25315 raft_consensus.cc:3060] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Advancing to term 21
W20260709 12:24:05.996616 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 132913us
W20260709 12:24:05.997694 25482 log.cc:927] Time spent T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.051s	user 0.000s	sys 0.000s
I20260709 12:24:05.956037 25447 consensus_queue.cc:1243] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer e2909cff1da44ad387f96a8ae3aa5afb log is divergent from this leader: its last log entry 17.11 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 9
W20260709 12:24:06.005007 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 158324us
W20260709 12:24:05.998533 25477 log.cc:927] Time spent T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.239s	user 0.000s	sys 0.001s
W20260709 12:24:06.008270 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 164518us
I20260709 12:24:05.872154 25067 raft_consensus.cc:515] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:06.013190 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.StartTabletCopy -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 44377.1us
I20260709 12:24:05.869256 25323 raft_consensus.cc:515] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:05.986060 25388 raft_consensus.cc:493] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:24:05.945603 25429 raft_consensus.cc:493] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:06.037843 17652 leader_election.cc:304] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
W20260709 12:24:05.999322 25467 log.cc:927] Time spent T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.281s	user 0.001s	sys 0.000s
W20260709 12:24:06.059358 17655 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:06.059607 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 224577us
I20260709 12:24:06.026196 18371 raft_consensus.cc:1275] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 22 index: 15. Preceding OpId from leader: term: 23 index: 15. (term mismatch)
I20260709 12:24:06.112747 18371 pending_rounds.cc:85] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da: Aborting all ops after (but not including) 14
I20260709 12:24:06.113042 18371 pending_rounds.cc:107] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da: Aborting uncommitted NO_OP operation due to leader change: 22.15
I20260709 12:24:06.113298 18371 raft_consensus.cc:2889] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:05.961195 25267 tablet_copy_client.cc:806] T a5732e71b9d54f8786b8c511d69943bf P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Starting download of 0 data blocks...
I20260709 12:24:06.076503 25182 leader_election.cc:290] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 26 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:06.077436 25323 leader_election.cc:290] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:06.078469 25067 leader_election.cc:290] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:06.078675 25133 consensus_queue.cc:1059] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 21.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20260709 12:24:06.157397 17585 catalog_manager.cc:5697] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 reported cstate change: term changed from 8 to 11, leader changed from e063541542a94969898868bf6eb5ce1b (127.16.29.67) to 0da169ff82674dc1947055ce43de68a3 (127.16.29.65). New cstate: current_term: 11 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: UNKNOWN } } }
I20260709 12:24:06.079452 25478 consensus_queue.cc:1048] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:24:06.080756 25420 consensus_queue.cc:237] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 21.17, Last appended by leader: 17, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:06.082860 25353 leader_election.cc:290] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:05.899192 25179 raft_consensus.cc:515] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:06.083163 18367 raft_consensus.cc:2468] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 22.
I20260709 12:24:06.085628 18365 raft_consensus.cc:1275] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 18 index: 15. Preceding OpId from leader: term: 19 index: 15. (term mismatch)
I20260709 12:24:06.040827 25422 raft_consensus.cc:3055] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 23 LEADER]: Stepping down as leader of term 23
W20260709 12:24:06.127198 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 47292.9us
I20260709 12:24:06.128436 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 24 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:06.204069 17838 raft_consensus.cc:3060] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:06.129505 17728 leader_election.cc:304] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
W20260709 12:24:06.205238 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 75857us
I20260709 12:24:06.206653 17728 leader_election.cc:304] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:06.133636 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:06.139250 18374 raft_consensus.cc:1240] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 22. Current term is 23. Ops: []
I20260709 12:24:06.146106 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 16 candidate_status { last_received { term: 14 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:06.149550 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 22 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:06.152765 25136 leader_election.cc:290] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:06.213114 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:06.214540 17835 raft_consensus.cc:1275] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 20 index: 14. Preceding OpId from leader: term: 23 index: 15. (index mismatch)
I20260709 12:24:06.215376 18365 pending_rounds.cc:85] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da: Aborting all ops after (but not including) 14
W20260709 12:24:06.160315 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:06.162791 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:06.163120 17832 raft_consensus.cc:1275] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 16 index: 11. Preceding OpId from leader: term: 17 index: 12. (index mismatch)
I20260709 12:24:06.208920 17833 raft_consensus.cc:3060] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:06.215677 18365 pending_rounds.cc:107] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da: Aborting uncommitted NO_OP operation due to leader change: 18.15
I20260709 12:24:06.217758 18365 raft_consensus.cc:2889] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:06.220263 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:24:06.258595 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:06.286890 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:06.287745 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 21 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:06.213841 18363 raft_consensus.cc:2468] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 20.
I20260709 12:24:06.164965 25432 consensus_queue.cc:1059] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 17.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.128s
I20260709 12:24:06.294567 17684 raft_consensus.cc:1240] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 17. Current term is 18. Ops: []
W20260709 12:24:06.293550 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 330266us
I20260709 12:24:06.111361 25278 consensus_queue.cc:1059] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.103s
W20260709 12:24:06.301028 17655 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:06.203289 18334 leader_election.cc:304] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:24:05.964537 17587 rpcz_store.cc:269] 0709 12:23:47.710076 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:47.710242 (+   166us) service_pool.cc:224] Handling call
0709 12:24:05.944000 (+18233758us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:23:55.513884 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:23:55.921454 (+407570us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:23:55.976748 (+ 55294us) write_op.cc:435] Acquired schema lock
0709 12:23:55.982475 (+  5727us) tablet.cc:662] Decoding operations
0709 12:23:56.778418 (+795943us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:23:56.778516 (+    98us) write_op.cc:641] Partition lock acquired for write op
0709 12:23:56.778548 (+    32us) tablet.cc:685] Acquiring locks for 19 operations
0709 12:23:56.780050 (+  1502us) tablet.cc:701] Row locks acquired
0709 12:23:56.780100 (+    50us) write_op.cc:260] PREPARE: finished
0709 12:23:56.780318 (+   218us) write_op.cc:270] Start()
0709 12:23:56.780481 (+   163us) write_op.cc:276] Timestamp: P: 1783599836780234 usec, L: 0
0709 12:23:56.780518 (+    37us) op_driver.cc:348] REPLICATION: starting
0709 12:23:56.781456 (+   938us) log.cc:844] Serialized 7679 byte log entry
0709 12:23:57.785816 (+1004360us) op_driver.cc:464] REPLICATION: finished
0709 12:23:58.184355 (+398539us) write_op.cc:301] APPLY: starting
0709 12:23:58.201753 (+ 17398us) tablet.cc:1367] starting BulkCheckPresence
0709 12:23:59.374613 (+1172860us) tablet.cc:1370] finished BulkCheckPresence
0709 12:23:59.374736 (+   123us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:23:59.993526 (+618790us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:23:59.994120 (+   594us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=38,key_file_lookups=38,delta_file_lookups=38,mrs_lookups=0
0709 12:23:59.994217 (+    97us) write_op.cc:312] APPLY: finished
0709 12:23:59.995208 (+   991us) log.cc:844] Serialized 172 byte log entry
0709 12:24:00.223922 (+228714us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:24:00.234628 (+ 10706us) write_op.cc:454] Released schema lock
0709 12:24:00.264810 (+ 30182us) write_op.cc:341] FINISH: Updating metrics
0709 12:24:00.398671 (+133861us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:24:00.402876 (+  4205us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":1395230,"reported_tablets":44,"child_traces":[["op",{"apply.queue_time_us":322338,"apply.run_cpu_time_us":102310,"apply.run_wall_time_us":2150185,"cfile_cache_hit":78,"cfile_cache_hit_bytes":83623,"num_ops":19,"prepare.queue_time_us":394826,"prepare.run_cpu_time_us":19142,"prepare.run_wall_time_us":1733313,"raft.queue_time_us":237279,"raft.run_cpu_time_us":69276,"raft.run_wall_time_us":256076,"replication_time_us":979759,"thread_start_us":1072333,"threads_started":5,"wal-append.queue_time_us":914179,"wal-append.run_cpu_time_us":20317,"wal-append.run_wall_time_us":2456131}]]}
I20260709 12:24:06.304860 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:06.305388 17683 raft_consensus.cc:1275] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 19 index: 11. Preceding OpId from leader: term: 24 index: 13. (index mismatch)
I20260709 12:24:06.110862 25297 consensus_queue.cc:1048] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.058s
W20260709 12:24:06.308851 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 97250.8us
I20260709 12:24:06.320374 18333 leader_election.cc:304] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:06.170974 25478 consensus_queue.cc:1243] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer 0da169ff82674dc1947055ce43de68a3 log is divergent from this leader: its last log entry 17.11 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 10
I20260709 12:24:06.202509 25422 raft_consensus.cc:740] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 23 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:24:06.120723 25485 consensus_queue.cc:1059] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.184s
W20260709 12:24:06.326444 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 123211us
I20260709 12:24:06.327404 18371 raft_consensus.cc:2468] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 18.
I20260709 12:24:06.329180 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:06.121762 25429 raft_consensus.cc:515] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:06.351805 18334 leader_election.cc:304] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:24:06.222401 25267 tablet_copy_client.cc:670] T a5732e71b9d54f8786b8c511d69943bf P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Starting download of 1 WAL segments...
I20260709 12:24:06.221841 25067 raft_consensus.cc:2749] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
W20260709 12:24:06.342720 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 305036us
I20260709 12:24:06.110513 25388 raft_consensus.cc:515] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:06.343792 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:06.343182 17585 catalog_manager.cc:5697] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 reported cstate change: term changed from 5 to 7. New cstate: current_term: 7 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:24:06.308545 25131 raft_consensus.cc:2749] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:24:06.076099 25445 consensus_queue.cc:237] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.13, Last appended by leader: 13, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:06.168628 25293 consensus_queue.cc:1048] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.153s
I20260709 12:24:06.310604 25136 consensus_queue.cc:1048] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.080s
W20260709 12:24:06.396941 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 57832us
I20260709 12:24:06.295264 25263 consensus_queue.cc:1059] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.051s
I20260709 12:24:06.084860 25436 consensus_queue.cc:237] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.14, Last appended by leader: 14, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:06.395444 17688 raft_consensus.cc:2468] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 17.
W20260709 12:24:06.400799 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 49244.1us
I20260709 12:24:06.342191 25474 raft_consensus.cc:2749] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
W20260709 12:24:06.392520 24992 consensus_meta.cc:287] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.560s	user 0.005s	sys 0.005s
I20260709 12:24:06.342573 25448 raft_consensus.cc:2804] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Leader election won for term 24
W20260709 12:24:06.431896 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:06.321426 25281 consensus_queue.cc:1059] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 21.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.072s
W20260709 12:24:06.456749 25493 log.cc:927] Time spent T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.131s	user 0.001s	sys 0.000s
I20260709 12:24:06.527613 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:06.498983 24992 raft_consensus.cc:515] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:06.530797 17834 raft_consensus.cc:2393] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 21: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:24:06.532922 24992 leader_election.cc:290] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 21 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:24:06.534210 25433 consensus_meta.cc:287] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.534s	user 0.011s	sys 0.000s
I20260709 12:24:06.535342 25433 raft_consensus.cc:2749] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:24:06.545589 25429 leader_election.cc:290] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:06.548588 25492 raft_consensus.cc:493] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:06.461191 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 21 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:24:06.461966 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 248220us
I20260709 12:24:06.550166 25448 raft_consensus.cc:697] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 24 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:24:06.462949 17688 raft_consensus.cc:1275] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 23 index: 16. Preceding OpId from leader: term: 25 index: 17. (index mismatch)
I20260709 12:24:06.463057 25480 raft_consensus.cc:3055] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:24:06.564613 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:06.565559 17688 raft_consensus.cc:3060] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:06.463418 25487 raft_consensus.cc:3060] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:06.464071 17683 raft_consensus.cc:1275] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 17 index: 14. Preceding OpId from leader: term: 19 index: 15. (index mismatch)
I20260709 12:24:06.464746 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 24 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
W20260709 12:24:06.465233 25491 log.cc:927] Time spent T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.126s	user 0.000s	sys 0.001s
I20260709 12:24:06.465808 25473 raft_consensus.cc:493] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:06.467201 17829 raft_consensus.cc:1275] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 12 index: 8. Preceding OpId from leader: term: 16 index: 10. (index mismatch)
W20260709 12:24:06.472498 25481 log.cc:927] Time spent T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.435s	user 0.001s	sys 0.000s
I20260709 12:24:06.473341 17680 raft_consensus.cc:1275] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 16 index: 10. (index mismatch)
W20260709 12:24:06.476215 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 78821.4us
I20260709 12:24:06.476898 25490 consensus_queue.cc:1048] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:24:06.479147 25179 leader_election.cc:290] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:24:06.483234 25345 consensus_meta.cc:287] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.764s	user 0.007s	sys 0.002s
I20260709 12:24:06.484083 25301 leader_election.cc:290] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:24:06.484684 25151 consensus_meta.cc:287] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.705s	user 0.009s	sys 0.002s
I20260709 12:24:06.412319 24642 raft_consensus.cc:2804] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:24:06.485306 25388 leader_election.cc:290] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:06.486091 25439 leader_election.cc:290] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:06.493891 25126 consensus_queue.cc:1059] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 25.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.101s
I20260709 12:24:06.457000 25288 consensus_queue.cc:1048] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.035s
W20260709 12:24:06.525525 25458 log.cc:927] Time spent T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.091s	user 0.000s	sys 0.000s
I20260709 12:24:06.496029 24801 raft_consensus.cc:3055] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 21 LEADER]: Stepping down as leader of term 21
W20260709 12:24:06.548894 25496 log.cc:927] Time spent T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.055s	user 0.000s	sys 0.000s
I20260709 12:24:06.549319 25492 raft_consensus.cc:515] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:06.457293 17835 raft_consensus.cc:1275] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 20 index: 15. Preceding OpId from leader: term: 21 index: 16. (index mismatch)
W20260709 12:24:06.564635 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 47719.3us
I20260709 12:24:06.589771 25297 consensus_queue.cc:1048] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.041s
I20260709 12:24:06.579227 17838 raft_consensus.cc:2468] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 24.
I20260709 12:24:06.569943 17729 leader_election.cc:304] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:06.612672 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 20 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:06.650439 18366 raft_consensus.cc:2468] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 19.
I20260709 12:24:06.564251 25480 raft_consensus.cc:740] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:24:06.627437 17684 raft_consensus.cc:1275] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 16 index: 11. Preceding OpId from leader: term: 17 index: 12. (index mismatch)
I20260709 12:24:06.399011 25476 consensus_queue.cc:1048] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.091s
I20260709 12:24:06.495922 25384 consensus_queue.cc:1059] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 22.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.109s
I20260709 12:24:06.630075 18364 raft_consensus.cc:1275] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 19 index: 15. Preceding OpId from leader: term: 20 index: 16. (index mismatch)
W20260709 12:24:06.630681 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 215190us
I20260709 12:24:06.630779 17682 raft_consensus.cc:2468] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 22.
W20260709 12:24:06.634826 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 311250us
W20260709 12:24:06.638854 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 87638.6us
I20260709 12:24:06.575799 17833 raft_consensus.cc:2468] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 22.
W20260709 12:24:06.639103 17763 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.771s	user 0.006s	sys 0.009s
W20260709 12:24:06.731839 17763 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.868s	user 0.006s	sys 0.010s
I20260709 12:24:06.642506 17688 raft_consensus.cc:2468] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 21.
I20260709 12:24:06.551297 25448 consensus_queue.cc:237] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 22.14, Last appended by leader: 14, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:06.672559 17655 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:06.745184 17652 leader_election.cc:304] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da
W20260709 12:24:06.745385 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 534955us
I20260709 12:24:06.747088 17728 leader_election.cc:304] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:24:06.634147 24642 raft_consensus.cc:697] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 18 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:24:06.749364 17680 pending_rounds.cc:85] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3: Aborting all ops after (but not including) 10
I20260709 12:24:06.749926 17680 pending_rounds.cc:107] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3: Aborting uncommitted NO_OP operation due to leader change: 17.11
I20260709 12:24:06.750155 17680 raft_consensus.cc:2889] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:24:06.749433 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 179946us
I20260709 12:24:06.638619 24801 raft_consensus.cc:740] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 21 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:24:06.753638 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:06.751999 17729 leader_election.cc:304] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:06.760936 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:06.762398 17761 raft_consensus.cc:2393] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 24: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:24:06.757230 17654 leader_election.cc:304] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:24:06.771174 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:06.775334 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:06.776047 17764 raft_consensus.cc:3060] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:06.775816 24642 consensus_queue.cc:237] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.13, Last appended by leader: 13, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:06.777663 25502 log.cc:927] Time spent T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.100s	user 0.000s	sys 0.000s
I20260709 12:24:06.778513 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:06.779664 17836 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35040 (request call id 3157) took 1052 ms. Trace:
I20260709 12:24:06.772703 25067 raft_consensus.cc:493] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:06.780436 24801 consensus_queue.cc:260] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 21.14, Last appended by leader: 14, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:24:06.781610 25424 consensus_meta.cc:287] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.742s	user 0.006s	sys 0.006s
I20260709 12:24:06.781284 25067 raft_consensus.cc:515] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:24:06.612600 25473 raft_consensus.cc:515] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:06.782037 24801 raft_consensus.cc:3060] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:06.676466 25485 consensus_queue.cc:1059] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 17.13, Next index: 14, Last known committed idx: 12, Time since last communication: 0.130s
I20260709 12:24:06.784380 25480 consensus_queue.cc:260] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 21.16, Last appended by leader: 16, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:06.784961 25370 consensus_meta.cc:287] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 1.034s	user 0.012s	sys 0.000s
I20260709 12:24:06.786098 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:06.785782 25370 raft_consensus.cc:515] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:06.787851 17760 raft_consensus.cc:1217] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Deduplicated request from leader. Original: 13.9->[15.10-22.13]   Dedup: 15.10->[18.11-22.13]
I20260709 12:24:06.788130 17685 raft_consensus.cc:1240] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 21. Current term is 22. Ops: []
I20260709 12:24:06.788717 17760 pending_rounds.cc:85] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb: Aborting all ops after (but not including) 10
I20260709 12:24:06.789057 17760 pending_rounds.cc:107] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb: Aborting uncommitted NO_OP operation due to leader change: 17.11
I20260709 12:24:06.789325 17760 raft_consensus.cc:2889] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:06.780117 17836 rpcz_store.cc:276] 0709 12:24:05.727437 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:05.735737 (+  8300us) service_pool.cc:224] Handling call
0709 12:24:05.736203 (+   466us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:24:05.760934 (+ 24731us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:05.760967 (+    33us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:24:05.773115 (+ 12148us) log.cc:844] Serialized 52 byte log entry
0709 12:24:05.998849 (+225734us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:24:06.062526 (+ 63677us) log.cc:844] Serialized 14 byte log entry
0709 12:24:06.090432 (+ 27906us) log.cc:844] Serialized 14 byte log entry
0709 12:24:06.090799 (+   367us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:06.090914 (+   115us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:06.476217 (+385303us) spinlock_profiling.cc:243] Waited 16.2 ms on lock 0x7b180044a400. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b47533522 00007f7b475357c1 00007f7b47535779 00007f7b47535709 00007f7b475356d1 00007f7b475349cd 0000000000388634 0000000000388479 00007f7b4751683a
0709 12:24:06.779432 (+303215us) raft_consensus.cc:1685] finished
0709 12:24:06.779458 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:06.779601 (+   143us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":54207,"prepare.queue_time_us":70751,"prepare.run_cpu_time_us":79,"prepare.run_wall_time_us":82,"raft.queue_time_us":104883,"raft.run_cpu_time_us":266,"raft.run_wall_time_us":41305,"spinlock_wait_cycles":91268992,"thread_start_us":160279,"threads_started":1,"wal-append.queue_time_us":341616}
I20260709 12:24:06.792174 17762 raft_consensus.cc:1275] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 17 index: 11. Preceding OpId from leader: term: 19 index: 12. (index mismatch)
I20260709 12:24:06.792493 25353 raft_consensus.cc:2804] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:24:06.792858 25353 raft_consensus.cc:493] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:06.793193 25353 raft_consensus.cc:3060] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:06.793736 17762 raft_consensus.cc:1275] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 17 index: 13. Preceding OpId from leader: term: 18 index: 14. (index mismatch)
I20260709 12:24:06.782095 25424 raft_consensus.cc:515] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Starting leader election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:06.785807 25480 raft_consensus.cc:3060] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:06.799955 25504 consensus_queue.cc:1048] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.214s
I20260709 12:24:06.801556 25439 raft_consensus.cc:493] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:06.802086 25439 raft_consensus.cc:515] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:06.804327 25473 leader_election.cc:290] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:24:06.809259 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 78072.7us
I20260709 12:24:06.813839 17761 raft_consensus.cc:1240] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 17. Current term is 18. Ops: []
I20260709 12:24:06.772051 17758 raft_consensus.cc:2468] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 20.
W20260709 12:24:06.807338 17803 leader_election.cc:336] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 20 pre-election: RPC error from VoteRequest() call to peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Timed out: RequestConsensusVote RPC to 127.16.29.68:42577 timed out after 1.510s (SENT)
I20260709 12:24:06.823416 17803 leader_election.cc:304] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:06.827330 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:06.828888 17758 raft_consensus.cc:3060] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:06.830524 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 24 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:06.831175 18333 leader_election.cc:304] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3
W20260709 12:24:06.840229 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 95155.3us
I20260709 12:24:06.841838 17652 leader_election.cc:304] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:06.783864 25067 leader_election.cc:290] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:06.831347 17759 raft_consensus.cc:2393] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 24: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:24:06.892161 17764 raft_consensus.cc:2468] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 24.
I20260709 12:24:06.828583 25151 raft_consensus.cc:2804] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:24:06.832538 25133 raft_consensus.cc:3055] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 17 LEADER]: Stepping down as leader of term 17
W20260709 12:24:06.852432 25497 log.cc:927] Time spent T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.085s	user 0.000s	sys 0.001s
W20260709 12:24:07.008647 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 201353us
W20260709 12:24:07.012881 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 171138us
W20260709 12:24:07.022779 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:06.980904 25434 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.635s	user 0.017s	sys 0.038s
W20260709 12:24:07.052286 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 221258us
I20260709 12:24:06.869527 25490 raft_consensus.cc:2749] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:24:06.870545 25474 raft_consensus.cc:2804] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:24:06.825425 25436 consensus_queue.cc:1059] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 15.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.196s
W20260709 12:24:07.065055 17803 leader_election.cc:336] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 17 election: RPC error from VoteRequest() call to peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Timed out: RequestConsensusVote RPC to 127.16.29.68:42577 timed out after 1.547s (SENT)
I20260709 12:24:07.001979 25301 raft_consensus.cc:2804] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Leader election won for term 22
W20260709 12:24:07.029421 25498 log.cc:927] Time spent T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.191s	user 0.001s	sys 0.000s
W20260709 12:24:07.054165 25445 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.503s	user 0.021s	sys 0.069s
I20260709 12:24:06.832438 25179 consensus_queue.cc:1048] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.214s
I20260709 12:24:07.002784 25151 raft_consensus.cc:493] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:24:07.126320 25509 raft_consensus.cc:2804] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Leader pre-election won for term 21
W20260709 12:24:07.057196 25315 consensus_meta.cc:287] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 1.000s	user 0.011s	sys 0.001s
I20260709 12:24:07.007509 25439 leader_election.cc:290] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:24:07.131716 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 92832.6us
I20260709 12:24:07.112707 17803 leader_election.cc:304] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:07.014490 25133 raft_consensus.cc:740] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:24:07.139796 18333 leader_election.cc:400] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 19 pre-election: Vote denied by peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837) with higher term. Message: Invalid argument: T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 19. Current term is 20.
I20260709 12:24:07.142148 18333 leader_election.cc:403] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 19 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:24:07.008757 25354 consensus_queue.cc:1048] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.139s
I20260709 12:24:07.107264 25474 raft_consensus.cc:493] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
W20260709 12:24:07.129935 25445 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.579s	user 0.022s	sys 0.069s
I20260709 12:24:07.049877 25297 consensus_queue.cc:1048] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.091s
I20260709 12:24:07.129418 25370 leader_election.cc:290] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 22 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:07.057360 25504 consensus_queue.cc:1048] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.087s
I20260709 12:24:07.135515 25151 raft_consensus.cc:3060] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:07.132040 25126 raft_consensus.cc:493] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:07.131265 25301 raft_consensus.cc:697] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:24:07.134114 25509 raft_consensus.cc:493] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:07.149940 25474 raft_consensus.cc:3060] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:07.179414 25281 raft_consensus.cc:2749] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
W20260709 12:24:07.203778 25511 log.cc:927] Time spent T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.083s	user 0.001s	sys 0.000s
I20260709 12:24:07.205698 25492 leader_election.cc:290] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:07.143649 25315 raft_consensus.cc:515] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:07.187188 25436 raft_consensus.cc:3060] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Advancing to term 20
W20260709 12:24:07.256844 17655 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:07.215566 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 463779us
W20260709 12:24:07.273082 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 208024us
W20260709 12:24:07.276341 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 135519us
I20260709 12:24:07.277601 17652 leader_election.cc:304] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:07.207957 25512 raft_consensus.cc:2749] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:24:07.296463 25297 raft_consensus.cc:493] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:07.297143 25297 raft_consensus.cc:515] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:07.298655 25494 log.cc:927] Time spent T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.912s	user 0.000s	sys 0.012s
I20260709 12:24:07.188134 25509 raft_consensus.cc:3060] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:07.284356 17763 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38776 (request call id 2844) took 1663 ms. Trace:
I20260709 12:24:07.227352 25420 raft_consensus.cc:493] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:07.292659 25517 raft_consensus.cc:2804] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:24:07.306959 17756 raft_consensus.cc:1240] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 20. Current term is 21. Ops: []
I20260709 12:24:07.186383 25126 raft_consensus.cc:515] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:24:07.311367 17803 leader_election.cc:336] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 19 election: RPC error from VoteRequest() call to peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Timed out: RequestConsensusVote RPC to 127.16.29.68:42577 timed out after 1.901s (SENT)
I20260709 12:24:07.328987 17803 leader_election.cc:304] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:07.302335 17763 rpcz_store.cc:276] 0709 12:24:05.556821 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:05.556980 (+   159us) service_pool.cc:224] Handling call
0709 12:24:05.582927 (+ 25947us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:05.629686 (+ 46759us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:05.629718 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:05.652343 (+ 22625us) log.cc:844] Serialized 26 byte log entry
0709 12:24:05.835326 (+182983us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:24:06.748694 (+913368us) log.cc:844] Serialized 14 byte log entry
0709 12:24:07.219731 (+471037us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:07.220012 (+   281us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:07.220191 (+   179us) raft_consensus.cc:1685] finished
0709 12:24:07.220210 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:07.220480 (+   270us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":242584,"prepare.queue_time_us":760674,"prepare.run_cpu_time_us":32,"prepare.run_wall_time_us":34,"raft.queue_time_us":387646,"raft.run_cpu_time_us":234,"raft.run_wall_time_us":11125,"thread_start_us":1285037,"threads_started":3,"wal-append.queue_time_us":245192}
W20260709 12:24:07.330482 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 361626us
W20260709 12:24:07.333758 17804 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:07.336800 18334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:07.336254 17755 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38776 (request call id 2845) took 1779 ms. Trace:
W20260709 12:24:07.310561 25480 consensus_meta.cc:287] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.512s	user 0.011s	sys 0.000s
W20260709 12:24:07.340435 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 62963.3us
W20260709 12:24:07.341246 17727 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.520s	user 0.012s	sys 0.012s
W20260709 12:24:07.341708 17727 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.530s	user 0.012s	sys 0.012s
I20260709 12:24:07.327450 25281 raft_consensus.cc:493] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:07.345499 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 533852us
I20260709 12:24:07.345835 25281 raft_consensus.cc:515] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:07.344739 17689 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37402 (request call id 3349) took 1183 ms. Trace:
I20260709 12:24:07.348412 17688 raft_consensus.cc:1240] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 17. Current term is 18. Ops: []
I20260709 12:24:07.348902 25281 leader_election.cc:290] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:07.337437 17755 rpcz_store.cc:276] 0709 12:24:05.557145 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:05.559164 (+  2019us) service_pool.cc:224] Handling call
0709 12:24:05.583483 (+ 24319us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 12:24:05.630657 (+ 47174us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:05.630683 (+    26us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 12:24:05.664733 (+ 34050us) log.cc:844] Serialized 78 byte log entry
0709 12:24:06.072247 (+407514us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:24:06.344649 (+272402us) log.cc:844] Serialized 14 byte log entry
0709 12:24:06.344985 (+   336us) log.cc:844] Serialized 14 byte log entry
0709 12:24:06.677028 (+332043us) log.cc:844] Serialized 14 byte log entry
0709 12:24:07.334912 (+657884us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:07.335992 (+  1080us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:07.336037 (+    45us) raft_consensus.cc:1685] finished
0709 12:24:07.336048 (+    11us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:07.336197 (+   149us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":313428,"prepare.queue_time_us":778243,"prepare.run_cpu_time_us":111,"prepare.run_wall_time_us":4968,"raft.queue_time_us":724421,"raft.run_cpu_time_us":256,"raft.run_wall_time_us":12224,"spinlock_wait_cycles":11931904,"thread_start_us":1241412,"threads_started":4,"wal-append.queue_time_us":791490}
W20260709 12:24:07.350420 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 603523us
I20260709 12:24:07.328332 25179 raft_consensus.cc:2749] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:24:07.363181 25281 raft_consensus.cc:2749] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:24:07.331478 25424 leader_election.cc:290] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 21 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:07.361863 17652 leader_election.cc:304] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:07.314447 25517 raft_consensus.cc:697] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 24 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:24:07.307991 25505 raft_consensus.cc:493] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
W20260709 12:24:07.365674 17680 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.544s	user 0.014s	sys 0.022s
W20260709 12:24:07.363204 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 51887.7us
W20260709 12:24:07.367797 17680 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.546s	user 0.014s	sys 0.022s
W20260709 12:24:07.368646 17803 leader_election.cc:336] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 18 election: RPC error from VoteRequest() call to peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Timed out: RequestConsensusVote RPC to 127.16.29.68:42577 timed out after 1.906s (SENT)
I20260709 12:24:07.348287 17689 rpcz_store.cc:276] 0709 12:24:06.161724 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:06.162004 (+   280us) service_pool.cc:224] Handling call
0709 12:24:06.212414 (+ 50410us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:06.212595 (+   181us) raft_consensus.cc:1534] Early marking committed up to index 16
0709 12:24:06.608564 (+395969us) log.cc:844] Serialized 14 byte log entry
0709 12:24:07.344370 (+735806us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:07.344426 (+    56us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:24:07.344479 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:07.344544 (+    65us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:07.344685 (+   141us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":146777,"prepare.queue_time_us":247056,"prepare.run_cpu_time_us":58,"prepare.run_wall_time_us":1988,"raft.queue_time_us":75451,"raft.run_cpu_time_us":148,"raft.run_wall_time_us":13938,"thread_start_us":926402,"threads_started":3,"wal-append.queue_time_us":693483}
I20260709 12:24:07.340416 24801 raft_consensus.cc:493] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:07.338200 25439 raft_consensus.cc:493] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:07.380657 17728 leader_election.cc:304] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:24:07.372579 25323 raft_consensus.cc:2804] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Leader pre-election won for term 20
W20260709 12:24:07.385229 17758 consensus_meta.cc:287] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.555s	user 0.010s	sys 0.006s
I20260709 12:24:07.309639 25420 raft_consensus.cc:515] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:07.389169 18371 raft_consensus.cc:1240] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 24 LEADER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 23. Current term is 24. Ops: [23.15-23.15]
I20260709 12:24:07.387261 17758 raft_consensus.cc:2468] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 21.
I20260709 12:24:07.364001 25297 leader_election.cc:290] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:07.391515 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:24:07.412000 17803 leader_election.cc:336] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 19 election: RPC error from VoteRequest() call to peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Timed out: RequestConsensusVote RPC to 127.16.29.68:42577 timed out after 1.938s (SENT)
I20260709 12:24:07.423365 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:07.424149 17836 raft_consensus.cc:2468] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 19.
I20260709 12:24:07.423668 17758 raft_consensus.cc:2393] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 21: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:24:07.427768 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 20 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:07.429628 17758 raft_consensus.cc:1275] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 19 index: 15. Preceding OpId from leader: term: 20 index: 16. (index mismatch)
I20260709 12:24:07.366472 25492 consensus_queue.cc:1048] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.038s
I20260709 12:24:07.433835 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:07.434332 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:07.434984 17758 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 18.
I20260709 12:24:07.436074 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:07.437287 17761 raft_consensus.cc:2468] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 19.
I20260709 12:24:07.393186 25387 raft_consensus.cc:493] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:07.440138 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:07.446939 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:07.448604 17835 raft_consensus.cc:1240] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 18. Current term is 19. Ops: []
W20260709 12:24:07.440552 17803 leader_election.cc:336] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 17 election: RPC error from VoteRequest() call to peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Timed out: RequestConsensusVote RPC to 127.16.29.68:42577 timed out after 1.982s (SENT)
I20260709 12:24:07.449863 17803 leader_election.cc:304] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:07.397222 25246 raft_consensus.cc:493] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:07.389398 25323 raft_consensus.cc:493] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:24:07.448493 17832 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 18.
I20260709 12:24:07.451222 17763 raft_consensus.cc:1240] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 23. Current term is 24. Ops: []
I20260709 12:24:07.451387 17652 leader_election.cc:304] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:07.401043 25126 leader_election.cc:290] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:07.446862 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:07.453282 25486 raft_consensus.cc:493] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:07.465065 17652 leader_election.cc:304] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:07.465348 17830 raft_consensus.cc:1240] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 23. Current term is 24. Ops: [21.15-23.16]
W20260709 12:24:07.473263 17803 leader_election.cc:336] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 19 pre-election: RPC error from VoteRequest() call to peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Timed out: RequestConsensusVote RPC to 127.16.29.68:42577 timed out after 1.969s (SENT)
W20260709 12:24:07.466005 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 343338us
I20260709 12:24:07.466996 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:07.473734 17803 leader_election.cc:304] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:07.474130 17762 raft_consensus.cc:3060] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:07.385090 24801 raft_consensus.cc:515] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:07.392284 25531 consensus_queue.cc:1048] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.072s
I20260709 12:24:07.449331 25384 raft_consensus.cc:493] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:07.371685 25505 raft_consensus.cc:515] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:24:07.453569 25516 log.cc:927] Time spent T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.272s	user 0.001s	sys 0.000s
I20260709 12:24:07.397341 25151 raft_consensus.cc:515] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:07.476485 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 95941.7us
I20260709 12:24:07.439349 25525 raft_consensus.cc:2804] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:24:07.477262 17652 leader_election.cc:304] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:07.398239 25315 leader_election.cc:290] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 21 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:07.479542 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 21 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:07.478206 25126 raft_consensus.cc:2749] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:24:07.507434 17758 raft_consensus.cc:2468] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 20.
I20260709 12:24:07.462939 25246 raft_consensus.cc:515] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:07.462631 25323 raft_consensus.cc:3060] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:07.513131 25538 consensus_queue.cc:1048] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20260709 12:24:07.514078 25151 leader_election.cc:290] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 20 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:07.387080 25439 raft_consensus.cc:515] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:07.515554 25538 consensus_queue.cc:1048] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:24:07.515928 24801 leader_election.cc:290] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:07.517912 25246 leader_election.cc:290] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:07.518630 25529 consensus_queue.cc:1048] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.001s
W20260709 12:24:07.494069 25523 log.cc:927] Time spent T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.112s	user 0.000s	sys 0.000s
I20260709 12:24:07.495790 25384 raft_consensus.cc:515] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:07.521162 25527 raft_consensus.cc:493] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:07.521704 25384 leader_election.cc:290] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:24:07.521950 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:07.471907 25326 raft_consensus.cc:493] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:07.521697 25527 raft_consensus.cc:515] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:07.493177 25513 log.cc:927] Time spent T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.103s	user 0.000s	sys 0.000s
I20260709 12:24:07.523275 25326 raft_consensus.cc:515] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:07.524276 17686 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37402 (request call id 3351) took 1346 ms. Trace:
I20260709 12:24:07.525406 25326 leader_election.cc:290] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:07.524641 17686 rpcz_store.cc:276] 0709 12:24:06.177823 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:06.178103 (+   280us) service_pool.cc:224] Handling call
0709 12:24:06.178769 (+   666us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:06.206279 (+ 27510us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:06.206310 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:06.206804 (+   494us) log.cc:844] Serialized 26 byte log entry
0709 12:24:06.633964 (+427160us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:24:06.634014 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:06.636523 (+  2509us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:07.524077 (+887554us) raft_consensus.cc:1685] finished
0709 12:24:07.524100 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:07.524218 (+   118us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":45033,"raft.queue_time_us":36618,"raft.run_cpu_time_us":130,"raft.run_wall_time_us":74352,"thread_start_us":287141,"threads_started":1,"wal-append.queue_time_us":1003840}
I20260709 12:24:07.369059 25301 consensus_queue.cc:237] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 20.17, Last appended by leader: 17, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:07.504890 17652 leader_election.cc:304] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:07.527595 17652 leader_election.cc:304] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:07.528481 25151 raft_consensus.cc:2749] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:24:07.528708 17652 leader_election.cc:304] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:07.529191 25151 consensus_queue.cc:1059] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 20.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 12:24:07.529883 17652 leader_election.cc:304] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:07.530669 25179 raft_consensus.cc:2804] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:24:07.531160 25179 raft_consensus.cc:697] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 22 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:24:07.531288 25490 raft_consensus.cc:3055] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:24:07.531510 25136 raft_consensus.cc:493] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:07.509033 18333 leader_election.cc:304] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:24:07.485844 25521 log.cc:927] Time spent T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.151s	user 0.001s	sys 0.000s
I20260709 12:24:07.501624 25525 raft_consensus.cc:493] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:07.479135 25370 raft_consensus.cc:2749] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:24:07.519369 25246 raft_consensus.cc:493] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:07.519279 25528 raft_consensus.cc:493] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:07.478304 25281 raft_consensus.cc:2749] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 12:24:07.492353 25444 leader_election.cc:290] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:07.526049 25527 leader_election.cc:290] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:07.530001 25151 raft_consensus.cc:2749] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
W20260709 12:24:07.505640 17803 leader_election.cc:336] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 20 election: RPC error from VoteRequest() call to peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Timed out: RequestConsensusVote RPC to 127.16.29.68:42577 timed out after 2.166s (SENT)
W20260709 12:24:07.482735 17682 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44898 (request call id 1485) took 2009 ms (client timeout 2230 ms). Trace:
I20260709 12:24:07.531714 25354 raft_consensus.cc:2804] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:24:07.531801 25490 raft_consensus.cc:740] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:24:07.533505 17684 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37396 (request call id 2934) took 1121 ms. Trace:
W20260709 12:24:07.511190 25518 log.cc:927] Time spent T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.149s	user 0.001s	sys 0.000s
I20260709 12:24:07.532088 25179 consensus_queue.cc:237] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 21.16, Last appended by leader: 16, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:07.542534 25537 consensus_queue.cc:1048] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:24:07.543913 17652 leader_election.cc:304] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:07.544992 17760 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce"
dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
mode: GRACEFUL
new_leader_uuid: "334a4e1f59944544b3b3ff36616b57da"
 from {username='slave'} at 127.0.0.1:49786
I20260709 12:24:07.496706 25433 raft_consensus.cc:2749] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:24:07.489622 25517 consensus_queue.cc:237] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 22.12, Last appended by leader: 12, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:24:07.548066 25487 consensus_meta.cc:287] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.942s	user 0.012s	sys 0.001s
I20260709 12:24:07.549889 25505 leader_election.cc:290] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:07.454282 25387 raft_consensus.cc:515] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:07.471640 25353 consensus_meta.cc:287] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.678s	user 0.013s	sys 0.001s
I20260709 12:24:07.512115 25492 raft_consensus.cc:2764] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Leader pre-election decision vote started in defunct term 20: won
I20260709 12:24:07.478928 25486 raft_consensus.cc:515] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:07.632658 17760 raft_consensus.cc:606] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Received request to transfer leadership to TS 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:07.642643 17760 raft_consensus.cc:614] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260709 12:24:07.633306 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 89494.7us
W20260709 12:24:07.632573 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 126971us
W20260709 12:24:07.587275 17682 rpcz_store.cc:269] 0709 12:24:05.473113 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:05.476887 (+  3774us) service_pool.cc:224] Handling call
0709 12:24:07.482446 (+2005559us) spinlock_profiling.cc:243] Waited 1.1 s on lock 0x7b54002530d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:24:07.482642 (+   196us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2533083904}
I20260709 12:24:07.630313 17684 rpcz_store.cc:276] 0709 12:24:06.412174 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:06.413101 (+   927us) service_pool.cc:224] Handling call
0709 12:24:06.468230 (+ 55129us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:06.653890 (+185660us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:07.533142 (+879252us) spinlock_profiling.cc:243] Waited 159 ms on lock 0x7b540014dcd4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:24:07.533350 (+   208us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":365064064}
I20260709 12:24:07.629590 25448 raft_consensus.cc:2749] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
W20260709 12:24:07.634688 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 125748us
I20260709 12:24:07.577770 25525 raft_consensus.cc:3060] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:07.636463 17762 raft_consensus.cc:2468] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 22.
I20260709 12:24:07.644632 18333 leader_election.cc:304] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:07.644717 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 25 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:07.645859 17758 raft_consensus.cc:2468] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 24.
I20260709 12:24:07.635466 25384 raft_consensus.cc:2804] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:24:07.632483 25354 raft_consensus.cc:697] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 18 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:24:07.635640 25527 raft_consensus.cc:493] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:07.646931 17652 leader_election.cc:304] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:07.637375 25487 raft_consensus.cc:2749] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:24:07.579509 25136 raft_consensus.cc:515] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:07.584674 25246 raft_consensus.cc:515] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:07.586531 25528 raft_consensus.cc:515] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:07.660234 17652 leader_election.cc:304] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:07.647821 25384 raft_consensus.cc:493] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:07.695786 17687 raft_consensus.cc:1240] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 15. Current term is 16. Ops: []
I20260709 12:24:07.696092 25384 raft_consensus.cc:3060] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Advancing to term 20
W20260709 12:24:07.697067 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 52525us
I20260709 12:24:07.698412 18333 leader_election.cc:304] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:07.673269 25433 raft_consensus.cc:493] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
W20260709 12:24:07.683728 25533 log.cc:927] Time spent T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.100s	user 0.001s	sys 0.000s
W20260709 12:24:07.685655 25539 log.cc:927] Time spent T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.051s	user 0.000s	sys 0.000s
I20260709 12:24:07.685577 25474 raft_consensus.cc:515] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:07.702123 25448 consensus_queue.cc:1048] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 9, Time since last communication: 0.001s
W20260709 12:24:07.703159 25495 log.cc:927] Time spent T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 1.252s	user 0.022s	sys 0.025s
W20260709 12:24:07.703482 17834 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35040 (request call id 3154) took 2048 ms (client timeout 2514 ms). Trace:
I20260709 12:24:07.692481 25354 consensus_queue.cc:237] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.13, Last appended by leader: 13, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:07.703958 17834 rpcz_store.cc:269] 0709 12:24:05.655435 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:05.662864 (+  7429us) service_pool.cc:224] Handling call
0709 12:24:07.703266 (+2040402us) spinlock_profiling.cc:243] Waited 747 ms on lock 0x7b54003345d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:24:07.703398 (+   132us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1717739904}
I20260709 12:24:07.703734 25474 leader_election.cc:290] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 21 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:07.693264 25323 raft_consensus.cc:515] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:07.707350 17803 leader_election.cc:336] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 18 election: RPC error from VoteRequest() call to peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Timed out: RequestConsensusVote RPC to 127.16.29.68:42577 timed out after 2.353s (SENT)
I20260709 12:24:07.662407 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 22 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:07.694584 25486 leader_election.cc:290] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:07.699720 25535 raft_consensus.cc:3060] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:07.656982 25527 raft_consensus.cc:515] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:07.711927 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 51777us
I20260709 12:24:07.643748 25353 raft_consensus.cc:515] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:07.716472 25353 leader_election.cc:290] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 20 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:07.652021 25445 raft_consensus.cc:2749] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 12:24:07.729422 18361 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59886 (request call id 1988) took 2190 ms. Trace:
I20260709 12:24:07.672269 25151 raft_consensus.cc:2804] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:24:07.700598 25433 raft_consensus.cc:515] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:07.733721 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 26 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:07.729828 18361 rpcz_store.cc:276] 0709 12:24:05.530356 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:05.540810 (+ 10454us) service_pool.cc:224] Handling call
0709 12:24:05.568694 (+ 27884us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:05.846994 (+278300us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:05.847027 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:05.865119 (+ 18092us) log.cc:844] Serialized 26 byte log entry
0709 12:24:06.147193 (+282074us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:24:06.147812 (+   619us) log.cc:844] Serialized 14 byte log entry
0709 12:24:06.165166 (+ 17354us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:06.548432 (+383266us) spinlock_profiling.cc:243] Waited 95.4 ms on lock 0x7b5400305f54. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:24:06.548491 (+    59us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:07.717874 (+1169383us) spinlock_profiling.cc:243] Waited 3.07 ms on lock 0x7b5400306720. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a595dbe 00007f7b4a5a0930 00007f7b4a5a08d8 00007f7b4a5a0858 00007f7b4a5a0813 00007f7b4a59fa9c 00007f7b5048b2f6 00007f7b50485aa8 00007f7b4a5dc7cb 00007f7b4a5df2c0
0709 12:24:07.720578 (+  2704us) raft_consensus.cc:1685] finished
0709 12:24:07.720601 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:07.720731 (+   130us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":21634,"prepare.queue_time_us":152,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":48,"raft.queue_time_us":154525,"raft.run_cpu_time_us":295,"raft.run_wall_time_us":81282,"spinlock_wait_cycles":226479232,"thread_start_us":352875,"threads_started":2,"wal-append.queue_time_us":204518}
I20260709 12:24:07.736526 18333 leader_election.cc:304] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:24:07.716571 25182 raft_consensus.cc:3055] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:24:07.748409 25323 leader_election.cc:290] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 20 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:07.714785 25474 consensus_queue.cc:1048] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.012s
I20260709 12:24:07.742580 25534 raft_consensus.cc:2804] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:24:07.727065 25486 raft_consensus.cc:2804] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:24:07.740185 17758 raft_consensus.cc:1240] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 21. Current term is 22. Ops: [21.12-21.12]
I20260709 12:24:07.721359 25370 raft_consensus.cc:493] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:24:07.730769 25435 consensus_queue.cc:1048] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.030s
I20260709 12:24:07.731613 25387 leader_election.cc:290] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:07.744159 25182 raft_consensus.cc:740] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:24:07.760819 25490 consensus_queue.cc:260] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 21.12, Last appended by leader: 12, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:07.762799 25246 leader_election.cc:290] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:07.763558 25433 leader_election.cc:290] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:07.764748 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 20 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:07.728138 25525 raft_consensus.cc:515] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:07.729060 25436 raft_consensus.cc:2749] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Leader pre-election lost for term 19. Reason: Vote denied by peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837) with higher term. Message: Invalid argument: T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 19. Current term is 20.
I20260709 12:24:07.734395 25151 raft_consensus.cc:493] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:07.800887 25509 raft_consensus.cc:515] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:07.820986 25151 raft_consensus.cc:3060] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:07.822749 25509 leader_election.cc:290] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 21 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:24:07.794747 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 58303.5us
I20260709 12:24:07.797848 25527 leader_election.cc:290] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:07.824513 18333 leader_election.cc:304] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:07.791944 25370 raft_consensus.cc:515] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:07.826282 18333 leader_election.cc:304] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:07.799063 25182 consensus_queue.cc:260] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 17.16, Last appended by leader: 16, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:07.829563 25445 raft_consensus.cc:2749] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:24:07.804992 25528 leader_election.cc:290] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:07.808321 25489 raft_consensus.cc:3055] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:24:07.703811 25384 raft_consensus.cc:515] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:07.814028 25136 leader_election.cc:290] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:07.772535 25486 raft_consensus.cc:493] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:24:07.777513 25534 raft_consensus.cc:493] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:07.800308 25490 raft_consensus.cc:3060] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:07.829825 25182 raft_consensus.cc:3060] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:07.827203 25370 leader_election.cc:290] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:24:07.833777 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 66123.2us
I20260709 12:24:07.830461 25435 raft_consensus.cc:2749] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:24:07.842679 17759 raft_consensus.cc:2368] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for earlier term 18. Current term is 20.
W20260709 12:24:07.844767 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:07.846774 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:07.847816 18361 raft_consensus.cc:2468] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 17.
I20260709 12:24:07.852783 25422 consensus_queue.cc:260] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 23.15, Last appended by leader: 15, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:07.854007 25422 raft_consensus.cc:3060] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:24:07.856869 25543 log.cc:927] Time spent T 00000000000000000000000000000000 P 2193af31cb894ec9b3d65d939a2a6851: Append to log took a long time: real 0.151s	user 0.003s	sys 0.000s
I20260709 12:24:07.857409 25206 raft_consensus.cc:3055] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:24:07.857811 25206 raft_consensus.cc:740] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 17 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:24:07.859844 25429 raft_consensus.cc:3055] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:24:07.860472 25429 raft_consensus.cc:740] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
W20260709 12:24:07.860949 25532 log.cc:927] Time spent T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.352s	user 0.000s	sys 0.000s
I20260709 12:24:07.861698 25429 consensus_queue.cc:260] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 22.16, Last appended by leader: 16, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:07.864771 17804 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:07.866107 25384 leader_election.cc:290] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 20 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:07.868487 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:07.869215 17762 raft_consensus.cc:3060] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:07.832193 25489 raft_consensus.cc:740] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 17 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:24:07.874017 17833 raft_consensus.cc:1240] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 21. Current term is 22. Ops: []
W20260709 12:24:07.875679 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 46978us
I20260709 12:24:07.876900 18333 leader_election.cc:304] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:07.877633 25133 consensus_queue.cc:260] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.13, Last appended by leader: 13, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:07.840538 25486 raft_consensus.cc:3060] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:07.878851 25133 raft_consensus.cc:3060] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:07.863539 25429 raft_consensus.cc:3060] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:07.880360 25434 raft_consensus.cc:2764] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Leader pre-election decision vote started in defunct term 20: won
I20260709 12:24:07.878190 18333 leader_election.cc:304] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:24:07.883164 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 21 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:07.884050 18367 raft_consensus.cc:2393] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 21: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:24:07.884322 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 20 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:07.884933 17832 raft_consensus.cc:3060] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:07.886059 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 18 candidate_status { last_received { term: 16 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:07.887017 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:07.887645 18361 raft_consensus.cc:2468] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 18.
I20260709 12:24:07.845662 25179 raft_consensus.cc:493] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:24:07.888334 17688 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44898 (request call id 1505) took 1141 ms. Trace:
I20260709 12:24:07.888746 17688 rpcz_store.cc:276] 0709 12:24:06.746529 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:06.746739 (+   210us) service_pool.cc:224] Handling call
0709 12:24:07.348208 (+601469us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:07.349274 (+  1066us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:07.888143 (+538869us) spinlock_profiling.cc:243] Waited 601 ms on lock 0x7b54004263d0. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a61f8ad 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:24:07.888272 (+   129us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1382131584}
I20260709 12:24:07.891000 18363 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59886 (request call id 1998) took 1375 ms. Trace:
I20260709 12:24:07.842307 25534 raft_consensus.cc:3060] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:07.891426 18363 rpcz_store.cc:276] 0709 12:24:06.515653 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:06.515812 (+   159us) service_pool.cc:224] Handling call
0709 12:24:07.890307 (+1374495us) spinlock_profiling.cc:243] Waited 2.24 ms on lock 0x7b5800102398. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b474fbb71 00007f7b474fbeee 000000000036eb39 00007f7b4f56f1e0 00007f7b4f6380c4 00007f7b4f638131 00007f7b4f5f9fc0 00007f7b4f5e4929 00007f7b4f5e43d2 00007f7b48b5bbd4
0709 12:24:07.890477 (+   170us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:07.890668 (+   191us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:07.890698 (+    30us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:07.890743 (+    45us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:24:07.890776 (+    33us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:07.890834 (+    58us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:07.890942 (+   108us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5154560}
I20260709 12:24:07.894124 18367 raft_consensus.cc:2410] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 18 because replica has last-logged OpId of term: 17 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 15.
I20260709 12:24:07.894307 25384 raft_consensus.cc:493] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:07.894862 25384 raft_consensus.cc:515] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:07.898133 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:07.898754 17755 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 19.
W20260709 12:24:07.899580 17655 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:07.899933 25421 raft_consensus.cc:3055] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:24:07.899343 25489 consensus_queue.cc:260] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 17.14, Last appended by leader: 14, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:07.900720 25489 raft_consensus.cc:3060] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:07.901530 25384 leader_election.cc:290] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:07.901676 17685 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37402 (request call id 3345) took 2176 ms. Trace:
I20260709 12:24:07.902243 17685 rpcz_store.cc:276] 0709 12:24:05.725437 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:05.726035 (+   598us) service_pool.cc:224] Handling call
0709 12:24:05.726403 (+   368us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:06.788860 (+1062457us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:07.901452 (+1112592us) spinlock_profiling.cc:243] Waited 1.06 s on lock 0x7b54003016d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:24:07.901619 (+   167us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2441177984}
I20260709 12:24:07.904246 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 25 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:07.904637 25535 raft_consensus.cc:2749] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:24:07.881096 17757 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38786 (request call id 3625) took 1128 ms. Trace:
I20260709 12:24:07.906838 17762 raft_consensus.cc:2468] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 20.
I20260709 12:24:07.878149 25525 leader_election.cc:290] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 20 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:07.907258 17757 rpcz_store.cc:276] 0709 12:24:06.753263 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:06.758239 (+  4976us) service_pool.cc:224] Handling call
0709 12:24:06.758684 (+   445us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:06.806667 (+ 47983us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 12:24:06.806711 (+    44us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:06.806771 (+    60us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:24:06.806826 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:07.880844 (+1074018us) spinlock_profiling.cc:243] Waited 47.6 ms on lock 0x7b54003eb054. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:24:07.880893 (+    49us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:07.881026 (+   133us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":946,"raft.queue_time_us":206847,"raft.run_cpu_time_us":171,"raft.run_wall_time_us":79342,"spinlock_wait_cycles":109429632}
I20260709 12:24:07.913875 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:07.859045 25206 consensus_queue.cc:260] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.13, Last appended by leader: 13, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:07.919775 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 97596.7us
I20260709 12:24:07.920830 18374 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 2235) took 1769 ms. Trace:
I20260709 12:24:07.925213 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:07.926357 18363 raft_consensus.cc:2468] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 21.
W20260709 12:24:07.932159 18334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:07.900321 25421 raft_consensus.cc:740] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 21 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:24:07.934155 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:07.934877 25421 consensus_queue.cc:260] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 21.17, Last appended by leader: 17, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:07.929229 25434 raft_consensus.cc:2804] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:24:07.936391 25421 raft_consensus.cc:3060] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:07.937044 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 22 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:07.913595 25504 consensus_queue.cc:1048] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.025s
I20260709 12:24:07.938971 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:07.940289 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:07.940526 25206 raft_consensus.cc:3060] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:07.940968 18366 raft_consensus.cc:2468] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 16.
I20260709 12:24:07.941952 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:07.942487 18366 raft_consensus.cc:3060] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:07.888360 25179 raft_consensus.cc:515] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:07.935941 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:07.946933 25370 raft_consensus.cc:493] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:07.947606 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:07.947547 25370 raft_consensus.cc:515] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:07.948378 18368 raft_consensus.cc:2468] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 16.
I20260709 12:24:07.948879 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:07.949465 17757 raft_consensus.cc:2468] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 18.
I20260709 12:24:07.949563 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:07.950232 18368 raft_consensus.cc:2468] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 20.
I20260709 12:24:07.952208 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:07.952594 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:07.952978 17688 raft_consensus.cc:2468] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 22.
I20260709 12:24:07.953393 17686 raft_consensus.cc:1240] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 23. Current term is 24. Ops: [23.15-23.15]
I20260709 12:24:07.954430 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:07.956336 18364 raft_consensus.cc:2468] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 20.
I20260709 12:24:07.957528 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:07.959028 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:07.960295 18364 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 19.
I20260709 12:24:07.963548 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 25 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:07.969594 17686 raft_consensus.cc:1240] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 19. Current term is 20. Ops: [19.15-19.15]
I20260709 12:24:07.950043 25151 raft_consensus.cc:515] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:07.976934 17681 raft_consensus.cc:2393] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 21: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:24:07.978741 25151 leader_election.cc:290] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 21 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:07.947006 25179 leader_election.cc:290] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:07.979341 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:07.980021 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:07.980242 17757 raft_consensus.cc:2393] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 21: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:24:07.980537 25433 raft_consensus.cc:493] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:07.980695 18363 raft_consensus.cc:3060] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:07.980909 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:07.980172 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:07.981760 17689 raft_consensus.cc:2393] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 20: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:24:07.982261 18364 raft_consensus.cc:1275] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 21 index: 17. Preceding OpId from leader: term: 22 index: 18. (index mismatch)
I20260709 12:24:07.971758 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:07.950152 25370 leader_election.cc:290] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:07.981854 17684 raft_consensus.cc:2468] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 21.
I20260709 12:24:07.983811 18368 raft_consensus.cc:2468] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 16.
I20260709 12:24:07.985006 17729 leader_election.cc:304] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:07.984201 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:07.985426 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 20 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:07.993727 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:07.994289 17833 raft_consensus.cc:3060] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:07.985944 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 19 candidate_status { last_received { term: 18 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:07.994729 17686 raft_consensus.cc:2468] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 17.
I20260709 12:24:07.988184 18366 raft_consensus.cc:2468] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 21.
I20260709 12:24:07.986644 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:07.981098 25433 raft_consensus.cc:515] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:07.997321 17685 raft_consensus.cc:2468] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 18.
I20260709 12:24:07.998142 17832 raft_consensus.cc:2468] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 20.
I20260709 12:24:08.000288 18365 raft_consensus.cc:3055] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:24:08.000701 18365 raft_consensus.cc:740] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:24:07.981104 25525 raft_consensus.cc:493] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:08.002331 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:08.002656 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:08.000602 17729 leader_election.cc:304] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:08.004159 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 25 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:08.004987 17831 raft_consensus.cc:2468] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 24.
I20260709 12:24:08.005087 17729 leader_election.cc:304] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:08.008456 17729 leader_election.cc:304] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:08.003024 17689 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 19.
I20260709 12:24:08.019747 17729 leader_election.cc:304] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:08.025878 18365 consensus_queue.cc:260] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 20.15, Last appended by leader: 15, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:08.029452 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:08.030282 17682 raft_consensus.cc:3060] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Advancing to term 20
I20260709 12:24:08.028399 17729 leader_election.cc:304] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:24:08.039441 18334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:08.041936 17833 raft_consensus.cc:2468] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 20.
I20260709 12:24:08.045027 17682 raft_consensus.cc:2468] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 20.
I20260709 12:24:08.026119 25527 raft_consensus.cc:2804] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:24:08.069924 25527 raft_consensus.cc:493] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:08.070793 25527 raft_consensus.cc:3060] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:08.046948 18365 raft_consensus.cc:3060] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:08.047930 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 20 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:08.050697 17729 leader_election.cc:304] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:08.056222 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:08.003396 25517 raft_consensus.cc:2804] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:24:08.057260 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:08.082177 17836 raft_consensus.cc:3060] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Advancing to term 21
I20260709 12:24:08.061902 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 23 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:24:08.064591 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:08.019481 25528 raft_consensus.cc:2804] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:24:08.009667 25136 raft_consensus.cc:2804] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:24:08.067600 18363 raft_consensus.cc:2468] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 21.
I20260709 12:24:08.082690 17682 raft_consensus.cc:1240] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 18 LEADER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 17. Current term is 18. Ops: []
I20260709 12:24:08.036692 25509 raft_consensus.cc:2804] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:24:08.001708 25534 raft_consensus.cc:515] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:08.099345 18334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:08.066431 25529 raft_consensus.cc:2804] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:24:08.021648 25525 raft_consensus.cc:515] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:08.092126 18365 raft_consensus.cc:1275] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 20 index: 15. Preceding OpId from leader: term: 21 index: 16. (index mismatch)
I20260709 12:24:08.106403 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:08.107857 17687 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44898 (request call id 1497) took 1971 ms. Trace:
I20260709 12:24:08.112525 17728 leader_election.cc:304] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:08.112358 17687 rpcz_store.cc:276] 0709 12:24:06.136765 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:06.137126 (+   361us) service_pool.cc:224] Handling call
0709 12:24:06.137475 (+   349us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:07.696259 (+1558784us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:08.107254 (+410995us) spinlock_profiling.cc:243] Waited 1.56 s on lock 0x7b5400207e54. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:24:08.107684 (+   430us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3583278848}
I20260709 12:24:08.084023 25504 raft_consensus.cc:2804] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:24:08.057233 25246 consensus_queue.cc:1048] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.041s
I20260709 12:24:08.082002 25517 raft_consensus.cc:493] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:08.111311 17689 raft_consensus.cc:1275] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 16 index: 13. Preceding OpId from leader: term: 20 index: 15. (index mismatch)
I20260709 12:24:08.113875 17728 leader_election.cc:304] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:08.093569 25528 raft_consensus.cc:493] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:08.115255 17836 raft_consensus.cc:2468] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 21.
I20260709 12:24:08.108680 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:08.118764 17728 leader_election.cc:304] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:08.093374 25136 raft_consensus.cc:493] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:08.119869 17682 raft_consensus.cc:2468] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 22.
I20260709 12:24:08.093699 25433 leader_election.cc:290] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:08.092935 25509 raft_consensus.cc:493] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:08.122725 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:08.110203 25345 raft_consensus.cc:3055] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:24:08.123081 17728 leader_election.cc:304] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:08.121160 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:08.123368 18368 raft_consensus.cc:3060] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:08.124118 18365 raft_consensus.cc:2368] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 20. Current term is 21.
I20260709 12:24:08.102730 25529 raft_consensus.cc:493] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:08.128790 17728 leader_election.cc:304] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:08.122087 25387 raft_consensus.cc:2749] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:24:08.119397 25517 raft_consensus.cc:3060] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:08.142597 25315 raft_consensus.cc:2804] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:24:08.126286 25509 raft_consensus.cc:3060] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Advancing to term 21
W20260709 12:24:08.193229 17803 leader_election.cc:336] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Timed out: RequestConsensusVote RPC to 127.16.29.68:42577 timed out after 1.536s (SENT)
I20260709 12:24:08.122304 25528 raft_consensus.cc:3060] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:24:08.144172 17655 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:08.138955 25480 consensus_queue.cc:1059] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 15.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:24:08.131441 18334 leader_election.cc:304] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:24:08.128496 25529 raft_consensus.cc:3060] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:08.183348 17756 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38786 (request call id 3622) took 1432 ms. Trace:
W20260709 12:24:08.183386 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 51079.7us
I20260709 12:24:08.112293 25424 consensus_queue.cc:1048] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.031s
I20260709 12:24:08.185189 17684 raft_consensus.cc:2410] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for term 23 because replica has last-logged OpId of term: 21 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 11.
I20260709 12:24:08.185566 17689 raft_consensus.cc:1275] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 22 index: 14. Preceding OpId from leader: term: 24 index: 15. (index mismatch)
I20260709 12:24:08.116673 25388 consensus_queue.cc:1059] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 17.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.077s
I20260709 12:24:08.126585 25345 raft_consensus.cc:740] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:24:08.122105 25136 raft_consensus.cc:3060] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Advancing to term 23
W20260709 12:24:08.204761 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:08.205634 17803 leader_election.cc:304] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:08.118737 25264 raft_consensus.cc:2804] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:24:08.127463 25301 raft_consensus.cc:2804] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:24:08.221783 25301 raft_consensus.cc:493] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:24:08.222188 17689 raft_consensus.cc:1240] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 19. Current term is 20. Ops: []
W20260709 12:24:08.222913 25540 log.cc:927] Time spent T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.501s	user 0.007s	sys 0.000s
I20260709 12:24:08.224093 25434 raft_consensus.cc:493] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:08.224730 25434 raft_consensus.cc:3060] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:08.232106 17680 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44898 (request call id 1502) took 1588 ms. Trace:
I20260709 12:24:08.232568 17680 rpcz_store.cc:276] 0709 12:24:06.644065 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:06.645070 (+  1005us) service_pool.cc:224] Handling call
0709 12:24:06.723524 (+ 78454us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:24:07.369233 (+645709us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:07.369303 (+    70us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:24:07.381793 (+ 12490us) log.cc:844] Serialized 52 byte log entry
0709 12:24:07.586722 (+204929us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:24:07.586759 (+    37us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:07.586815 (+    56us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:08.222278 (+635463us) spinlock_profiling.cc:243] Waited 7.74 ms on lock 0x7b4400022b4c. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b47459c85 00007f7b4745a6a2 00007f7b4a1cc1db 00007f7b4a1c7aaf 00007f7b4a1bb86e 00007f7b4a1babb6 00007f7b4a1ba7d8 00007f7b4a1c4221 00007f7b4a1c41d9 00007f7b4a1c4169
0709 12:24:08.231639 (+  9361us) raft_consensus.cc:1685] finished
0709 12:24:08.231662 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:08.231965 (+   303us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":65617,"raft.queue_time_us":51967,"raft.run_cpu_time_us":183,"raft.run_wall_time_us":13158,"spinlock_wait_cycles":17794944,"thread_start_us":701927,"threads_started":2,"wal-append.queue_time_us":274117}
W20260709 12:24:08.240513 17801 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:08.242096 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 48922.4us
W20260709 12:24:08.250365 17802 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:08.206672 25428 raft_consensus.cc:3055] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 15 LEADER]: Stepping down as leader of term 15
W20260709 12:24:08.216594 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 85199.3us
I20260709 12:24:08.215008 17687 raft_consensus.cc:1240] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 19. Current term is 20. Ops: []
I20260709 12:24:08.212393 17756 rpcz_store.cc:276] 0709 12:24:06.750889 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:06.770759 (+ 19870us) service_pool.cc:224] Handling call
0709 12:24:06.805564 (+ 34805us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:07.307595 (+502031us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:08.148522 (+840927us) spinlock_profiling.cc:243] Waited 498 ms on lock 0x7b54004150d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:24:08.183268 (+ 34746us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1145504128}
I20260709 12:24:08.265936 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:08.148635 25131 raft_consensus.cc:2804] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:24:08.222249 25301 raft_consensus.cc:3060] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:08.123531 25504 raft_consensus.cc:697] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 21 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:24:08.188563 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:08.262023 25447 raft_consensus.cc:2749] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:24:08.266661 17683 raft_consensus.cc:2468] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 16.
I20260709 12:24:08.271585 17764 raft_consensus.cc:1240] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 21. Current term is 22. Ops: [21.13-21.13]
I20260709 12:24:08.272168 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:08.276714 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:08.272230 18365 raft_consensus.cc:2468] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 19.
I20260709 12:24:08.273885 17757 raft_consensus.cc:1240] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 LEADER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 21. Current term is 22. Ops: []
W20260709 12:24:08.274084 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:08.273948 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:24:08.277328 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 253593us
I20260709 12:24:08.220201 25133 raft_consensus.cc:2804] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:24:08.288311 17652 leader_election.cc:304] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:08.288650 17763 raft_consensus.cc:1240] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 21. Current term is 22. Ops: [21.14-21.14]
I20260709 12:24:08.292634 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:08.297200 17652 leader_election.cc:304] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:24:08.298225 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:08.298977 17755 raft_consensus.cc:2468] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 18.
I20260709 12:24:08.293372 17687 raft_consensus.cc:2468] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 23.
I20260709 12:24:08.301447 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:08.305305 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:08.305797 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:08.305006 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:08.266688 25428 raft_consensus.cc:740] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:24:08.310165 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:08.266841 25264 raft_consensus.cc:493] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:24:08.276491 25531 raft_consensus.cc:3055] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:24:08.313531 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 24 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:08.316995 18365 raft_consensus.cc:2393] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 24: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:24:08.304678 17652 leader_election.cc:304] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:08.217504 25486 raft_consensus.cc:515] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:08.321805 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:08.322472 18362 raft_consensus.cc:3060] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:08.314405 25429 raft_consensus.cc:2804] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:24:08.309123 25179 raft_consensus.cc:2804] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:24:08.327802 25179 raft_consensus.cc:493] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:08.328271 25179 raft_consensus.cc:3060] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Advancing to term 21
W20260709 12:24:08.328771 25560 log.cc:927] Time spent T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.142s	user 0.000s	sys 0.001s
I20260709 12:24:08.300238 25388 consensus_queue.cc:1048] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.031s
I20260709 12:24:08.331825 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:08.333729 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:24:08.335786 25555 log.cc:927] Time spent T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.116s	user 0.000s	sys 0.000s
I20260709 12:24:08.291783 25471 raft_consensus.cc:493] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:08.344929 25534 leader_election.cc:290] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 24 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:08.316497 25531 raft_consensus.cc:740] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:24:08.349723 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:08.350528 17833 raft_consensus.cc:2468] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 18.
I20260709 12:24:08.350171 25531 consensus_queue.cc:260] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 19.15, Last appended by leader: 15, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:08.349252 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:08.351959 17831 raft_consensus.cc:2393] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 21: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:24:08.355700 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:08.287694 25384 raft_consensus.cc:493] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:24:08.358021 25384 raft_consensus.cc:515] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:08.358409 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:08.358999 17834 raft_consensus.cc:2468] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 17.
I20260709 12:24:08.360653 25384 leader_election.cc:290] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:08.360826 25525 leader_election.cc:290] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:08.361450 25504 consensus_queue.cc:237] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 20.14, Last appended by leader: 14, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:08.363596 25486 leader_election.cc:290] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 19 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:08.272751 25131 raft_consensus.cc:493] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:08.377996 18368 raft_consensus.cc:2468] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 17.
I20260709 12:24:08.377996 18362 raft_consensus.cc:2468] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 18.
W20260709 12:24:08.379920 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 61406us
I20260709 12:24:08.293967 25133 raft_consensus.cc:493] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:08.385200 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:08.385731 17652 leader_election.cc:304] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:08.271957 25485 consensus_queue.cc:1048] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.046s
I20260709 12:24:08.314141 25264 raft_consensus.cc:3060] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:08.387104 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 19 candidate_status { last_received { term: 18 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:08.378171 25429 raft_consensus.cc:697] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 22 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
W20260709 12:24:08.391098 25559 log.cc:927] Time spent T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.111s	user 0.000s	sys 0.000s
I20260709 12:24:08.392712 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:08.393088 25384 raft_consensus.cc:2804] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:24:08.392742 17652 leader_election.cc:304] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:08.393580 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:08.340355 25471 raft_consensus.cc:515] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:08.359386 25527 raft_consensus.cc:515] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:08.396229 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:08.306641 25151 consensus_queue.cc:1059] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.081s
I20260709 12:24:08.400393 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:08.439805 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 24 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:08.440678 17836 raft_consensus.cc:2376] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 24. Re-sending same reply.
I20260709 12:24:08.404404 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 21 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:08.443442 17837 raft_consensus.cc:1275] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 21 index: 17. Preceding OpId from leader: term: 22 index: 18. (index mismatch)
I20260709 12:24:08.410270 18368 raft_consensus.cc:2468] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 18.
I20260709 12:24:08.444742 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 24 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:08.445749 18368 raft_consensus.cc:2393] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 24: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:24:08.446738 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:08.411870 17838 raft_consensus.cc:2468] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 18.
W20260709 12:24:08.447697 17727 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:08.386375 25133 raft_consensus.cc:3060] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:08.411919 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:08.386698 25531 raft_consensus.cc:3060] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:08.414147 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 23 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:08.415329 17829 raft_consensus.cc:2468] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 17.
W20260709 12:24:08.456651 17803 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:08.457100 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 64426.4us
I20260709 12:24:08.458529 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:08.460091 17834 raft_consensus.cc:2410] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for term 23 because replica has last-logged OpId of term: 21 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 11.
I20260709 12:24:08.461208 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 20 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:08.462791 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:08.463968 25486 raft_consensus.cc:2804] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:24:08.464416 25486 raft_consensus.cc:493] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:24:08.464814 25486 raft_consensus.cc:3060] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:08.464931 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:08.465682 18368 raft_consensus.cc:3060] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:08.413415 17802 leader_election.cc:304] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:24:08.424307 25429 consensus_queue.cc:237] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 21.13, Last appended by leader: 13, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:08.383729 25345 consensus_queue.cc:260] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 25.19, Last appended by leader: 19, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:08.467207 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 53918.2us
I20260709 12:24:08.479514 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 23 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:08.490234 17836 raft_consensus.cc:2410] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for term 23 because replica has last-logged OpId of term: 21 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 13.
W20260709 12:24:08.485533 17804 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:08.433565 25471 leader_election.cc:290] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:08.416356 25315 raft_consensus.cc:697] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:24:08.494517 25464 consensus_queue.cc:1059] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 21.13, Next index: 14, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:24:08.494571 25315 consensus_queue.cc:237] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.11, Last appended by leader: 11, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:08.495492 25464 raft_consensus.cc:3055] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:24:08.495862 25464 raft_consensus.cc:740] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 21 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:24:08.497177 17759 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38776 (request call id 2853) took 1071 ms. Trace:
I20260709 12:24:08.497751 17759 rpcz_store.cc:276] 0709 12:24:07.425598 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:07.425768 (+   170us) service_pool.cc:224] Handling call
0709 12:24:08.496907 (+1071139us) spinlock_profiling.cc:243] Waited 370 ms on lock 0x7b5400310e54. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:24:08.497100 (+   193us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":850237824}
I20260709 12:24:08.436228 25527 leader_election.cc:290] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 18 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:08.438079 17686 raft_consensus.cc:1240] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 15. Current term is 16. Ops: []
I20260709 12:24:08.434414 25517 raft_consensus.cc:515] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:08.506963 17727 leader_election.cc:304] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:08.512380 17727 leader_election.cc:304] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:08.413902 25131 raft_consensus.cc:3060] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:08.513875 25525 raft_consensus.cc:2749] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
W20260709 12:24:08.491794 17804 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:08.478571 17730 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:08.480325 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:08.432760 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:08.493204 18362 raft_consensus.cc:2468] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 19.
I20260709 12:24:08.530748 18365 raft_consensus.cc:2393] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 19: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:24:08.468132 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 21 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:08.478791 25512 raft_consensus.cc:2804] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:24:08.501621 17654 leader_election.cc:304] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:24:08.496845 25464 consensus_queue.cc:260] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 21.13, Last appended by leader: 13, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:08.477104 25485 raft_consensus.cc:493] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:08.530033 17804 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:08.531337 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
W20260709 12:24:08.532044 17804 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:08.532080 17836 raft_consensus.cc:3060] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:24:08.555449 18334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:08.566656 18368 raft_consensus.cc:2468] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 17.
I20260709 12:24:08.570734 17727 leader_election.cc:304] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:24:08.573770 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:08.575311 17727 leader_election.cc:304] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:08.580768 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:08.581979 18367 raft_consensus.cc:2393] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 21: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:24:08.582891 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:08.589354 25388 raft_consensus.cc:2804] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:24:08.589864 25388 raft_consensus.cc:493] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:08.590250 25388 raft_consensus.cc:3060] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Advancing to term 19
W20260709 12:24:08.592809 25562 log.cc:927] Time spent T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.318s	user 0.001s	sys 0.000s
I20260709 12:24:08.592963 25528 raft_consensus.cc:515] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:08.595306 25528 leader_election.cc:290] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 18 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:08.594789 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:08.602550 18361 raft_consensus.cc:2376] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 18. Re-sending same reply.
I20260709 12:24:08.604086 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:08.604894 18361 raft_consensus.cc:2468] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 21.
I20260709 12:24:08.596835 25517 leader_election.cc:290] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 19 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:08.608350 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:08.608560 25517 consensus_queue.cc:1048] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:24:08.597467 25464 raft_consensus.cc:3060] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:08.609189 25529 raft_consensus.cc:515] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:08.613000 25422 consensus_meta.cc:287] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.759s	user 0.011s	sys 0.000s
I20260709 12:24:08.599458 17803 leader_election.cc:304] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:08.613559 25527 raft_consensus.cc:493] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:08.614920 25179 raft_consensus.cc:515] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:08.481091 25509 raft_consensus.cc:515] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:08.568171 25525 raft_consensus.cc:2804] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:24:08.616266 25471 raft_consensus.cc:2749] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:24:08.616653 25525 raft_consensus.cc:493] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:08.617161 25525 raft_consensus.cc:3060] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:08.619830 17803 leader_election.cc:304] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:24:08.598258 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 20 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:08.621867 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:24:08.609681 18334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:08.624279 25527 raft_consensus.cc:515] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:08.625828 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:08.626444 18362 raft_consensus.cc:2376] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 17. Re-sending same reply.
I20260709 12:24:08.612536 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:08.628711 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:08.629920 18361 raft_consensus.cc:2393] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 19: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:24:08.637282 25439 leader_election.cc:290] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:08.594817 25133 raft_consensus.cc:515] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:08.585520 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:08.643666 18365 raft_consensus.cc:2468] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 18.
I20260709 12:24:08.634285 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:08.646610 17836 raft_consensus.cc:2468] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 18.
I20260709 12:24:08.560801 25370 raft_consensus.cc:2804] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:24:08.644951 18361 raft_consensus.cc:2376] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 20. Re-sending same reply.
I20260709 12:24:08.652812 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:08.651813 17837 raft_consensus.cc:1275] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 22 index: 12. Preceding OpId from leader: term: 24 index: 13. (index mismatch)
I20260709 12:24:08.656179 17686 raft_consensus.cc:1240] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 20. Current term is 21. Ops: [18.14-20.15]
I20260709 12:24:08.661649 17727 leader_election.cc:304] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:08.663496 17829 raft_consensus.cc:1275] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 20 index: 17. Preceding OpId from leader: term: 22 index: 18. (index mismatch)
W20260709 12:24:08.664809 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:08.607280 25136 raft_consensus.cc:515] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:08.669811 18362 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 19.
W20260709 12:24:08.673141 18334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:08.672605 17803 leader_election.cc:304] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:24:08.586846 25424 raft_consensus.cc:2804] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:24:08.580539 25512 raft_consensus.cc:697] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 21 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:24:08.680639 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:08.681399 17836 raft_consensus.cc:3060] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:08.581681 18368 raft_consensus.cc:2376] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 19. Re-sending same reply.
W20260709 12:24:08.682605 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 223280us
I20260709 12:24:08.573227 25485 raft_consensus.cc:515] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:08.691172 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 21 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:08.694854 18365 raft_consensus.cc:3060] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:08.678860 25384 raft_consensus.cc:697] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 20 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:24:08.518429 25428 consensus_queue.cc:260] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.11, Last appended by leader: 11, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:08.673964 25439 raft_consensus.cc:2804] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:24:08.680730 25471 raft_consensus.cc:2804] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:24:08.726598 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 18 candidate_status { last_received { term: 16 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:08.728652 18368 raft_consensus.cc:2410] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 18 because replica has last-logged OpId of term: 17 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 15.
I20260709 12:24:08.727413 17836 raft_consensus.cc:2468] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 18.
I20260709 12:24:08.690099 25424 raft_consensus.cc:697] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 24 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:24:08.679122 25528 consensus_queue.cc:1048] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.042s
I20260709 12:24:08.662851 25264 raft_consensus.cc:515] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Starting leader election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:24:08.702906 25464 raft_consensus.cc:2804] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:24:08.707563 25529 leader_election.cc:290] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 18 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:08.692180 25517 consensus_queue.cc:1048] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.045s
I20260709 12:24:08.756078 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 18 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:08.731185 25439 raft_consensus.cc:697] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:24:08.738942 25428 raft_consensus.cc:3060] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:08.723131 25151 consensus_queue.cc:1059] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 18.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.036s
I20260709 12:24:08.743976 25464 raft_consensus.cc:493] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:24:08.786437 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 284921us
I20260709 12:24:08.791433 17654 leader_election.cc:304] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:08.788718 25464 raft_consensus.cc:3060] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:08.800918 18365 raft_consensus.cc:2468] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 21.
I20260709 12:24:08.746266 25301 raft_consensus.cc:515] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:08.792584 25490 raft_consensus.cc:3055] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:24:08.814967 18371 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59886 (request call id 1996) took 2391 ms. Trace:
I20260709 12:24:08.825256 18371 rpcz_store.cc:276] 0709 12:24:06.439277 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:06.469865 (+ 30588us) service_pool.cc:224] Handling call
0709 12:24:06.470272 (+   407us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:07.389625 (+919353us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:08.814530 (+1424905us) spinlock_profiling.cc:243] Waited 919 ms on lock 0x7b54006ae0d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:24:08.814807 (+   277us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2112695552}
I20260709 12:24:08.809654 17727 leader_election.cc:304] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:08.827181 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 18 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:08.761160 25512 consensus_queue.cc:237] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 19.14, Last appended by leader: 14, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:08.794250 25179 leader_election.cc:290] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 21 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:24:08.858867 18333 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:08.859212 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 26 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:24:08.858091 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 66783us
W20260709 12:24:08.861860 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:08.862972 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:08.866277 18333 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:08.875226 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:08.878880 17729 leader_election.cc:304] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:24:08.881973 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 19 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:24:08.884225 17802 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:08.824972 25490 raft_consensus.cc:740] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:24:08.886767 17654 leader_election.cc:304] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:08.884207 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 21 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:08.890336 18371 raft_consensus.cc:2393] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 21: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
W20260709 12:24:08.907955 17804 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:08.859855 25528 raft_consensus.cc:2804] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:24:08.923187 17654 leader_election.cc:304] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:08.931143 18371 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639"
dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
mode: GRACEFUL
new_leader_uuid: "0da169ff82674dc1947055ce43de68a3"
 from {username='slave'} at 127.0.0.1:50468
W20260709 12:24:08.932011 17804 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:08.932201 18371 raft_consensus.cc:606] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Received request to transfer leadership to TS 0da169ff82674dc1947055ce43de68a3
I20260709 12:24:08.932808 18371 raft_consensus.cc:614] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:24:08.938081 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:08.939399 17764 raft_consensus.cc:2368] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 22. Current term is 23.
I20260709 12:24:07.921702 18374 rpcz_store.cc:276] 0709 12:24:06.150934 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:06.320510 (+169576us) service_pool.cc:224] Handling call
0709 12:24:06.341068 (+ 20558us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:06.341299 (+   231us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 12:24:06.341619 (+   320us) log.cc:844] Serialized 14 byte log entry
0709 12:24:07.742817 (+1401198us) spinlock_profiling.cc:243] Waited 5.43 ms on lock 0x7b1800499f60. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b47515e07 00007f7b503709a9 00007f7b4753269b 00007f7b4753093f 00007f7b47532c02 00007f7b4a1ba20b 00007f7b4a1bfd23 00007f7b4a1c023a 00007f7b4a62aa2a
0709 12:24:07.919929 (+177112us) log.cc:844] Serialized 14 byte log entry
0709 12:24:07.920208 (+   279us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:07.920276 (+    68us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:24:07.920330 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:07.920571 (+   241us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:07.920772 (+   201us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":123,"prepare.run_cpu_time_us":78,"prepare.run_wall_time_us":10273,"raft.queue_time_us":409,"spinlock_wait_cycles":12484352,"thread_start_us":176615,"threads_started":2,"wal-append.queue_time_us":1467208}
I20260709 12:24:08.857194 25575 raft_consensus.cc:2804] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Leader pre-election won for term 19
W20260709 12:24:08.979945 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 296639us
I20260709 12:24:08.927501 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:08.876647 25509 leader_election.cc:290] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 21 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:08.800618 25388 raft_consensus.cc:515] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:08.928324 18365 raft_consensus.cc:1240] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 25. Current term is 26. Ops: []
I20260709 12:24:08.984635 17759 raft_consensus.cc:1240] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 21 LEADER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 20. Current term is 21. Ops: []
I20260709 12:24:08.984966 18365 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 2242) took 1038 ms. Trace:
I20260709 12:24:08.835685 25525 raft_consensus.cc:515] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:08.812202 25424 consensus_queue.cc:237] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 21.15, Last appended by leader: 15, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:08.984129 17757 raft_consensus.cc:1240] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 17. Current term is 18. Ops: [16.12-17.13]
I20260709 12:24:08.928427 25136 leader_election.cc:290] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 23 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:08.989535 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:08.924316 25179 raft_consensus.cc:3055] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:24:08.925123 25429 raft_consensus.cc:2749] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 12:24:09.012995 25421 consensus_queue.cc:1048] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:24:09.000216 18374 raft_consensus.cc:1240] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 22. Current term is 23. Ops: []
I20260709 12:24:09.015548 18374 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 2243) took 1043 ms. Trace:
I20260709 12:24:09.015722 17801 leader_election.cc:304] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:09.016139 18374 rpcz_store.cc:276] 0709 12:24:07.972220 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:08.941808 (+969588us) service_pool.cc:224] Handling call
0709 12:24:08.999973 (+ 58165us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:09.015203 (+ 15230us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:09.015469 (+   266us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:24:09.018838 17804 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:08.985718 18365 rpcz_store.cc:276] 0709 12:24:07.946089 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:08.927777 (+981688us) service_pool.cc:224] Handling call
0709 12:24:08.928125 (+   348us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:08.984705 (+ 56580us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:08.984912 (+   207us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:09.018708 17801 leader_election.cc:304] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:24:09.020751 17804 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:08.925720 25315 raft_consensus.cc:2804] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:24:09.020668 18365 raft_consensus.cc:1240] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 18. Current term is 19. Ops: []
I20260709 12:24:09.027315 18365 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 2251) took 1052 ms. Trace:
I20260709 12:24:08.980168 25574 raft_consensus.cc:2749] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 12:24:09.027688 18365 rpcz_store.cc:276] 0709 12:24:07.974889 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:09.019989 (+1045100us) service_pool.cc:224] Handling call
0709 12:24:09.020397 (+   408us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:09.027039 (+  6642us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:09.027258 (+   219us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:09.018713 18374 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 2246) took 1045 ms. Trace:
I20260709 12:24:09.028865 18374 rpcz_store.cc:276] 0709 12:24:07.973283 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:09.017561 (+1044278us) service_pool.cc:224] Handling call
0709 12:24:09.018041 (+   480us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:09.018291 (+   250us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:09.018332 (+    41us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:09.018398 (+    66us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:24:09.018446 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:09.018522 (+    76us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:09.018624 (+   102us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:09.005795 25581 raft_consensus.cc:2804] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:24:09.031070 25581 raft_consensus.cc:493] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:09.031425 25581 raft_consensus.cc:3060] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:09.032032 25505 consensus_queue.cc:1048] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20260709 12:24:09.034902 17681 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37396 (request call id 2944) took 1081 ms. Trace:
I20260709 12:24:09.035307 17681 rpcz_store.cc:276] 0709 12:24:07.953097 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:07.954231 (+  1134us) service_pool.cc:224] Handling call
0709 12:24:09.034708 (+1080477us) spinlock_profiling.cc:243] Waited 21.8 ms on lock 0x7b5400207954. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:24:09.034843 (+   135us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":50036480}
W20260709 12:24:09.038193 25569 log.cc:927] Time spent T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.244s	user 0.000s	sys 0.001s
I20260709 12:24:09.038439 25067 raft_consensus.cc:493] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:24:08.858788 25384 consensus_queue.cc:237] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:09.039147 25067 raft_consensus.cc:515] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:09.044238 25067 leader_election.cc:290] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:24:09.060480 25577 log.cc:927] Time spent T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.198s	user 0.000s	sys 0.000s
I20260709 12:24:08.834856 25439 consensus_queue.cc:237] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.12, Last appended by leader: 12, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:08.980499 25528 raft_consensus.cc:697] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:24:08.980402 25575 raft_consensus.cc:493] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
W20260709 12:24:09.075807 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 52598.5us
I20260709 12:24:09.100785 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:09.101688 17764 raft_consensus.cc:3060] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:09.092597 17652 leader_election.cc:304] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:09.088872 17688 raft_consensus.cc:2468] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 22.
I20260709 12:24:09.119455 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:09.123232 17762 raft_consensus.cc:1240] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 21. Current term is 22. Ops: [21.16-21.16]
I20260709 12:24:09.124533 18374 raft_consensus.cc:1275] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 14 index: 15. Preceding OpId from leader: term: 15 index: 16. (index mismatch)
I20260709 12:24:09.131244 18365 raft_consensus.cc:1240] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 20. Current term is 21. Ops: []
I20260709 12:24:09.132943 18365 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 2252) took 1157 ms. Trace:
I20260709 12:24:09.133472 18365 rpcz_store.cc:276] 0709 12:24:07.975206 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:09.080698 (+1105492us) service_pool.cc:224] Handling call
0709 12:24:09.130975 (+ 50277us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:09.132444 (+  1469us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:09.132864 (+   420us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:09.135502 18365 raft_consensus.cc:1240] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 20. Current term is 22. Ops: []
I20260709 12:24:09.136335 18374 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59902 (request call id 1812) took 1106 ms. Trace:
I20260709 12:24:09.138590 18363 raft_consensus.cc:1275] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 22 index: 15. Preceding OpId from leader: term: 23 index: 16. (index mismatch)
I20260709 12:24:09.144295 18363 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59902 (request call id 1814) took 1112 ms. Trace:
I20260709 12:24:09.144743 18363 rpcz_store.cc:276] 0709 12:24:08.031838 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:09.101336 (+1069498us) service_pool.cc:224] Handling call
0709 12:24:09.101733 (+   397us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:09.143918 (+ 42185us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:09.144216 (+   298us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:09.023144 25179 raft_consensus.cc:740] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
W20260709 12:24:09.142484 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 50087.5us
I20260709 12:24:09.147285 17652 leader_election.cc:304] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:09.051713 25433 raft_consensus.cc:2749] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:24:09.148378 18374 rpcz_store.cc:276] 0709 12:24:08.030476 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:09.030037 (+999561us) service_pool.cc:224] Handling call
0709 12:24:09.105071 (+ 75034us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:09.125028 (+ 19957us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:09.136248 (+ 11220us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:09.157158 17652 leader_election.cc:304] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:09.165050 18363 raft_consensus.cc:1275] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 19 index: 14. Preceding OpId from leader: term: 20 index: 15. (index mismatch)
I20260709 12:24:09.151965 25264 leader_election.cc:290] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 22 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:09.155598 25574 raft_consensus.cc:2749] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:24:09.091027 25315 raft_consensus.cc:697] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
W20260709 12:24:09.140393 25578 log.cc:927] Time spent T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.207s	user 0.000s	sys 0.002s
I20260709 12:24:09.104126 25575 raft_consensus.cc:3060] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:09.186594 18363 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59902 (request call id 1818) took 1152 ms. Trace:
I20260709 12:24:09.165278 17764 raft_consensus.cc:2468] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 23.
W20260709 12:24:09.191064 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:09.162869 25354 raft_consensus.cc:2804] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:24:09.090590 25490 consensus_queue.cc:260] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 23.16, Last appended by leader: 16, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:09.198511 18365 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59902 (request call id 1817) took 1164 ms. Trace:
I20260709 12:24:09.201959 18363 rpcz_store.cc:276] 0709 12:24:08.034954 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:09.145702 (+1110748us) service_pool.cc:224] Handling call
0709 12:24:09.164630 (+ 18928us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:09.186194 (+ 21564us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:09.186517 (+   323us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:09.187388 25505 raft_consensus.cc:2749] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
W20260709 12:24:09.214879 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 57818.2us
I20260709 12:24:09.211345 18365 rpcz_store.cc:276] 0709 12:24:08.034124 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:09.134863 (+1100739us) service_pool.cc:224] Handling call
0709 12:24:09.135267 (+   404us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:09.197686 (+ 62419us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:09.198435 (+   749us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:09.222257 17652 leader_election.cc:304] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:09.221220 18365 raft_consensus.cc:1240] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 18 LEADER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 17. Current term is 18. Ops: []
I20260709 12:24:09.224406 25354 raft_consensus.cc:697] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 20 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:24:09.225768 18365 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59902 (request call id 1821) took 1133 ms. Trace:
I20260709 12:24:09.225718 25354 consensus_queue.cc:237] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 19.16, Last appended by leader: 16, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:09.228075 18363 raft_consensus.cc:1240] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 18 LEADER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 17. Current term is 18. Ops: []
I20260709 12:24:09.244689 18363 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59902 (request call id 1819) took 1205 ms. Trace:
I20260709 12:24:09.245023 18363 rpcz_store.cc:276] 0709 12:24:08.039443 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:09.211141 (+1171698us) service_pool.cc:224] Handling call
0709 12:24:09.227808 (+ 16667us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:09.244413 (+ 16605us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:09.244633 (+   220us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:24:09.208770 25579 log.cc:927] Time spent T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.405s	user 0.002s	sys 0.000s
I20260709 12:24:09.236778 17654 leader_election.cc:304] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:09.240049 18365 rpcz_store.cc:276] 0709 12:24:08.092251 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:09.220548 (+1128297us) service_pool.cc:224] Handling call
0709 12:24:09.220970 (+   422us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:09.224831 (+  3861us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:09.225691 (+   860us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:09.228381 25420 leader_election.cc:290] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:09.211786 25301 leader_election.cc:290] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 19 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:09.220106 25586 raft_consensus.cc:2804] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Leader pre-election won for term 25
W20260709 12:24:09.251482 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:09.243336 25572 log.cc:927] Time spent T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.382s	user 0.000s	sys 0.003s
I20260709 12:24:09.212487 25151 consensus_queue.cc:1059] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 20.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.036s
I20260709 12:24:09.242940 25429 raft_consensus.cc:2804] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:24:09.261852 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:09.277611 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:09.253430 25586 raft_consensus.cc:493] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:09.197002 25528 consensus_queue.cc:237] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 16.13, Last appended by leader: 13, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:09.206959 25179 consensus_queue.cc:260] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.12, Last appended by leader: 12, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:09.274618 25587 raft_consensus.cc:2749] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:24:09.286679 25586 raft_consensus.cc:3060] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Advancing to term 25
W20260709 12:24:09.297786 25583 log.cc:927] Time spent T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.090s	user 0.000s	sys 0.000s
I20260709 12:24:09.298406 25345 raft_consensus.cc:3060] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Advancing to term 26
W20260709 12:24:09.298777 25585 log.cc:927] Time spent T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.081s	user 0.000s	sys 0.000s
W20260709 12:24:09.300393 25584 log.cc:927] Time spent T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.116s	user 0.001s	sys 0.000s
I20260709 12:24:09.300856 18361 raft_consensus.cc:1275] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 17 index: 16. Preceding OpId from leader: term: 18 index: 16. (term mismatch)
I20260709 12:24:09.301263 18361 pending_rounds.cc:85] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da: Aborting all ops after (but not including) 15
I20260709 12:24:09.301502 18361 pending_rounds.cc:107] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da: Aborting uncommitted NO_OP operation due to leader change: 17.16
I20260709 12:24:09.301741 18361 raft_consensus.cc:2889] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:09.302613 18361 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59902 (request call id 1830) took 1033 ms. Trace:
I20260709 12:24:09.302973 18361 rpcz_store.cc:276] 0709 12:24:08.273778 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:09.300114 (+1026336us) service_pool.cc:224] Handling call
0709 12:24:09.300466 (+   352us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:09.302321 (+  1855us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:09.302548 (+   227us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:09.286758 25429 raft_consensus.cc:697] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 20 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:24:09.304867 25429 consensus_queue.cc:237] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 19.16, Last appended by leader: 16, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:09.309257 25586 raft_consensus.cc:515] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:09.316612 25586 leader_election.cc:290] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 25 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:24:09.317839 25434 consensus_meta.cc:287] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 1.093s	user 0.011s	sys 0.004s
I20260709 12:24:09.318342 25434 raft_consensus.cc:515] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:09.321671 25133 leader_election.cc:290] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 22 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:09.285923 25388 leader_election.cc:290] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 19 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:24:09.321843 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:09.329933 25586 raft_consensus.cc:493] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:24:09.369278 17680 raft_consensus.cc:1275] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 19 index: 11. Preceding OpId from leader: term: 21 index: 13. (index mismatch)
W20260709 12:24:09.387308 17804 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:09.332227 18368 raft_consensus.cc:2393] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 18: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:24:09.351102 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:09.382122 17687 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37402 (request call id 3363) took 1105 ms. Trace:
I20260709 12:24:09.369390 17689 raft_consensus.cc:1240] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 19. Current term is 20. Ops: []
W20260709 12:24:09.401790 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 383142us
I20260709 12:24:09.404366 17687 rpcz_store.cc:276] 0709 12:24:08.276371 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:08.310569 (+ 34198us) service_pool.cc:224] Handling call
0709 12:24:08.310858 (+   289us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:08.376941 (+ 66083us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:08.376991 (+    50us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:08.377055 (+    64us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:24:08.377098 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:09.381750 (+1004652us) spinlock_profiling.cc:243] Waited 65.8 ms on lock 0x7b54003458d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:24:09.381807 (+    57us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:09.382043 (+   236us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":151309056}
I20260709 12:24:09.400835 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:24:09.329818 25531 consensus_meta.cc:287] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.877s	user 0.012s	sys 0.002s
W20260709 12:24:09.379237 25428 consensus_meta.cc:287] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.607s	user 0.010s	sys 0.002s
W20260709 12:24:09.370761 25131 consensus_meta.cc:287] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.857s	user 0.011s	sys 0.001s
I20260709 12:24:09.291293 25525 leader_election.cc:290] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 19 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:09.340425 17802 leader_election.cc:304] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
W20260709 12:24:09.406220 25486 consensus_meta.cc:287] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.905s	user 0.016s	sys 0.003s
I20260709 12:24:09.410575 18374 raft_consensus.cc:1240] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 20. Current term is 21. Ops: []
I20260709 12:24:09.414254 17760 raft_consensus.cc:1240] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 18. Current term is 20. Ops: [18.15-18.15]
I20260709 12:24:09.416109 18367 raft_consensus.cc:2468] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 16.
I20260709 12:24:09.407931 17756 raft_consensus.cc:2468] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 18.
W20260709 12:24:09.416626 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 76317.1us
I20260709 12:24:09.425670 17802 leader_election.cc:304] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:24:09.377209 25471 raft_consensus.cc:493] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
W20260709 12:24:09.416672 17804 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:09.296633 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:09.384449 25370 raft_consensus.cc:493] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:24:09.377689 25434 leader_election.cc:290] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 18 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:09.430534 18365 raft_consensus.cc:2468] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 20.
I20260709 12:24:09.434410 18365 raft_consensus.cc:1275] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 17 index: 13. Preceding OpId from leader: term: 18 index: 14. (index mismatch)
I20260709 12:24:09.435200 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:09.435835 18365 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59902 (request call id 1839) took 1038 ms. Trace:
I20260709 12:24:09.436028 17756 raft_consensus.cc:3060] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:09.436237 18365 rpcz_store.cc:276] 0709 12:24:08.397243 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:09.432655 (+1035412us) service_pool.cc:224] Handling call
0709 12:24:09.433446 (+   791us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:09.435450 (+  2004us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:09.435781 (+   331us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:09.452629 18374 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59902 (request call id 1837) took 1073 ms. Trace:
I20260709 12:24:09.453066 18374 rpcz_store.cc:276] 0709 12:24:08.379542 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:09.368126 (+988584us) service_pool.cc:224] Handling call
0709 12:24:09.408264 (+ 40138us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:09.452322 (+ 44058us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:09.452570 (+   248us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:09.296615 25573 consensus_queue.cc:1059] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 18.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.119s
I20260709 12:24:09.423498 25131 raft_consensus.cc:515] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:09.452162 25370 raft_consensus.cc:3060] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:09.452277 25421 raft_consensus.cc:493] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:09.409339 25594 raft_consensus.cc:493] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:09.453323 17728 leader_election.cc:304] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:09.462359 17684 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37396 (request call id 2952) took 1447 ms. Trace:
I20260709 12:24:09.494786 17684 rpcz_store.cc:276] 0709 12:24:08.014848 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:08.022864 (+  8016us) service_pool.cc:224] Handling call
0709 12:24:09.457021 (+1434157us) spinlock_profiling.cc:243] Waited 60.8 ms on lock 0x7b54001aa754. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:24:09.462286 (+  5265us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":139831680}
I20260709 12:24:09.402419 25586 raft_consensus.cc:515] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:09.451278 25206 raft_consensus.cc:493] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:09.451910 25489 raft_consensus.cc:493] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:09.451980 25447 raft_consensus.cc:493] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:09.451735 25471 raft_consensus.cc:3060] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:09.520619 18374 raft_consensus.cc:1275] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 17 index: 11. Preceding OpId from leader: term: 19 index: 12. (index mismatch)
W20260709 12:24:09.525251 17804 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:09.528887 17804 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:09.529150 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 292088us
I20260709 12:24:09.427045 25588 raft_consensus.cc:2804] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:24:09.458266 25433 raft_consensus.cc:2804] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:24:09.532670 17759 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d"
dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
mode: GRACEFUL
new_leader_uuid: "e063541542a94969898868bf6eb5ce1b"
 from {username='slave'} at 127.0.0.1:49786
I20260709 12:24:09.533921 17759 raft_consensus.cc:606] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 22 LEADER]: Received request to transfer leadership to TS e063541542a94969898868bf6eb5ce1b
I20260709 12:24:09.558091 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:09.566305 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:09.509745 25525 raft_consensus.cc:2749] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 12:24:09.566308 17762 raft_consensus.cc:1240] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 18. Current term is 19. Ops: []
I20260709 12:24:09.515712 25587 raft_consensus.cc:493] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:09.451512 25486 raft_consensus.cc:515] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:09.330786 25315 consensus_queue.cc:237] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 18.15, Last appended by leader: 15, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:09.580488 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 21 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:09.576650 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:09.578033 18374 raft_consensus.cc:1240] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 23. Current term is 24. Ops: []
I20260709 12:24:09.586374 17832 raft_consensus.cc:3060] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:09.554982 25588 raft_consensus.cc:493] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:09.579807 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 25 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:09.587298 17763 raft_consensus.cc:3060] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:09.555207 25433 raft_consensus.cc:493] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:09.576644 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 21 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:09.590912 17762 raft_consensus.cc:1240] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 20. Current term is 21. Ops: []
I20260709 12:24:09.601464 17682 raft_consensus.cc:1240] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 18. Current term is 19. Ops: []
I20260709 12:24:09.519716 25421 raft_consensus.cc:515] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:09.604887 17758 raft_consensus.cc:1275] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 19 index: 15. Preceding OpId from leader: term: 22 index: 17. (index mismatch)
W20260709 12:24:09.609827 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 170831us
I20260709 12:24:09.558974 25489 raft_consensus.cc:515] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:09.608019 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:09.555904 25206 raft_consensus.cc:515] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:09.515831 25594 raft_consensus.cc:515] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:09.605794 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 484181us
I20260709 12:24:09.582773 25151 raft_consensus.cc:493] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:24:09.611373 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 25 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:09.557518 25447 raft_consensus.cc:515] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:09.612547 17762 raft_consensus.cc:1275] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 18 index: 14. (index mismatch)
I20260709 12:24:09.592857 25433 raft_consensus.cc:3060] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:09.610635 17801 leader_election.cc:304] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:09.587520 25588 raft_consensus.cc:3060] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Advancing to term 17
W20260709 12:24:09.606721 25592 log.cc:927] Time spent T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.182s	user 0.001s	sys 0.000s
I20260709 12:24:09.631572 18374 raft_consensus.cc:1275] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 17 index: 13. Preceding OpId from leader: term: 18 index: 14. (index mismatch)
I20260709 12:24:09.668783 25594 leader_election.cc:290] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:24:09.632045 25464 consensus_meta.cc:287] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.801s	user 0.014s	sys 0.000s
I20260709 12:24:09.669857 25464 raft_consensus.cc:515] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:09.584682 25587 raft_consensus.cc:515] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:24:09.632608 17803 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:09.674957 25581 consensus_meta.cc:287] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.643s	user 0.011s	sys 0.000s
W20260709 12:24:09.675402 17803 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.16.29.68:42577 timed out after 4.706s (SENT) [suppressed 16 similar messages]
I20260709 12:24:09.670797 18374 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0"
dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
mode: GRACEFUL
new_leader_uuid: "0da169ff82674dc1947055ce43de68a3"
 from {username='slave'} at 127.0.0.1:50468
I20260709 12:24:09.675657 25587 leader_election.cc:290] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:09.612979 25434 raft_consensus.cc:493] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:09.677597 17803 leader_election.cc:304] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:09.677877 18371 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 2245) took 1704 ms. Trace:
I20260709 12:24:09.678943 17803 leader_election.cc:304] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:09.678550 18371 rpcz_store.cc:276] 0709 12:24:07.972975 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:09.002844 (+1029869us) service_pool.cc:224] Handling call
0709 12:24:09.003282 (+   438us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:09.216836 (+213554us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:09.216932 (+    96us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:09.224554 (+  7622us) log.cc:844] Serialized 26 byte log entry
0709 12:24:09.517168 (+292614us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:24:09.517215 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:09.517304 (+    89us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:09.677554 (+160250us) raft_consensus.cc:1685] finished
0709 12:24:09.677615 (+    61us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:09.677803 (+   188us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":76946,"raft.run_cpu_time_us":377,"raft.run_wall_time_us":33075,"thread_start_us":504798,"threads_started":2,"wal-append.queue_time_us":406382}
I20260709 12:24:09.671898 25206 leader_election.cc:290] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:09.673668 17763 raft_consensus.cc:2468] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 21.
W20260709 12:24:09.619321 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 356221us
I20260709 12:24:09.675719 25447 leader_election.cc:290] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:09.676465 18374 raft_consensus.cc:606] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 20 LEADER]: Received request to transfer leadership to TS 0da169ff82674dc1947055ce43de68a3
W20260709 12:24:09.682488 18333 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:09.672731 25596 log.cc:927] Time spent T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.115s	user 0.000s	sys 0.000s
I20260709 12:24:09.604521 25354 raft_consensus.cc:493] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
W20260709 12:24:09.695765 18333 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:09.696769 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:09.704968 17759 raft_consensus.cc:1240] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 19. Current term is 20. Ops: []
I20260709 12:24:09.697706 17756 raft_consensus.cc:2468] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 26.
I20260709 12:24:09.614475 25126 raft_consensus.cc:493] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:09.677014 25434 raft_consensus.cc:515] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:09.605876 25512 consensus_queue.cc:1059] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 20.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.092s
I20260709 12:24:09.714773 17832 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 19.
I20260709 12:24:09.702339 25602 raft_consensus.cc:493] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:09.653124 25151 raft_consensus.cc:515] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:09.675459 25581 raft_consensus.cc:515] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:09.730393 17762 raft_consensus.cc:1240] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 18. Current term is 19. Ops: []
I20260709 12:24:09.713995 25447 raft_consensus.cc:2749] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
W20260709 12:24:09.733843 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 53529.3us
I20260709 12:24:09.724469 25206 raft_consensus.cc:3060] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Advancing to term 21
W20260709 12:24:09.729902 25597 log.cc:927] Time spent T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.125s	user 0.000s	sys 0.001s
I20260709 12:24:09.706225 25354 raft_consensus.cc:515] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:09.746206 17683 raft_consensus.cc:1240] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 19. Current term is 20. Ops: []
I20260709 12:24:09.749126 18374 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f"
dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
mode: GRACEFUL
new_leader_uuid: "e063541542a94969898868bf6eb5ce1b"
 from {username='slave'} at 127.0.0.1:50468
I20260709 12:24:09.751528 18374 raft_consensus.cc:606] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Received request to transfer leadership to TS e063541542a94969898868bf6eb5ce1b
I20260709 12:24:09.752036 18374 raft_consensus.cc:614] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260709 12:24:09.754709 17802 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:09.714058 25126 raft_consensus.cc:515] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:09.757496 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:09.738744 25611 raft_consensus.cc:2749] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
W20260709 12:24:09.773774 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 62464us
I20260709 12:24:09.775149 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:09.775827 17684 raft_consensus.cc:2468] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 21.
W20260709 12:24:09.759781 25595 log.cc:927] Time spent T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.225s	user 0.000s	sys 0.001s
I20260709 12:24:09.733215 25602 raft_consensus.cc:515] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:09.774986 17654 leader_election.cc:304] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:24:09.753216 25586 leader_election.cc:290] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:09.768946 25599 raft_consensus.cc:3055] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 20 LEADER]: Stepping down as leader of term 20
W20260709 12:24:09.774557 25575 consensus_meta.cc:287] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.557s	user 0.015s	sys 0.000s
I20260709 12:24:09.777282 17652 leader_election.cc:400] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 18 pre-election: Vote denied by peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651) with higher term. Message: Invalid argument: T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for earlier term 18. Current term is 20.
I20260709 12:24:09.784948 25622 raft_consensus.cc:2804] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:24:09.785425 25622 raft_consensus.cc:493] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:24:09.785807 25622 raft_consensus.cc:3060] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:09.745327 25613 consensus_queue.cc:1048] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.039s
I20260709 12:24:09.792088 17759 raft_consensus.cc:1240] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 20. Current term is 21. Ops: []
I20260709 12:24:09.779371 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:09.793103 17686 raft_consensus.cc:1275] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 17 index: 13. Preceding OpId from leader: term: 21 index: 15. (index mismatch)
I20260709 12:24:09.793429 17762 raft_consensus.cc:2368] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 22. Current term is 23.
I20260709 12:24:09.781984 25603 consensus_queue.cc:1059] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 20.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.015s
I20260709 12:24:09.797008 18365 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59886 (request call id 2018) took 1295 ms. Trace:
I20260709 12:24:09.797420 18365 rpcz_store.cc:276] 0709 12:24:08.501302 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:09.437960 (+936658us) service_pool.cc:224] Handling call
0709 12:24:09.438386 (+   426us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:09.452647 (+ 14261us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:09.452680 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:09.453313 (+   633us) log.cc:844] Serialized 26 byte log entry
0709 12:24:09.632747 (+179434us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:24:09.632812 (+    65us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:09.632919 (+   107us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:09.796784 (+163865us) raft_consensus.cc:1685] finished
0709 12:24:09.796811 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:09.796949 (+   138us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":13749,"raft.queue_time_us":13880,"raft.run_cpu_time_us":112,"raft.run_wall_time_us":4614,"thread_start_us":118395,"threads_started":1,"wal-append.queue_time_us":274517}
I20260709 12:24:09.782250 25599 raft_consensus.cc:740] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 20 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:24:09.799396 25599 consensus_queue.cc:260] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 20.14, Last appended by leader: 14, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:09.781838 25512 consensus_queue.cc:1048] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.019s
I20260709 12:24:09.800128 25617 consensus_queue.cc:1059] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 23.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20260709 12:24:09.800745 25599 raft_consensus.cc:3060] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:09.801192 25620 consensus_queue.cc:1048] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.011s
I20260709 12:24:09.802651 16501 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260709 12:24:09.789464 25575 raft_consensus.cc:515] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:09.804539 25607 log.cc:927] Time spent T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.067s	user 0.000s	sys 0.000s
W20260709 12:24:09.806056 25612 log.cc:927] Time spent T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.057s	user 0.000s	sys 0.000s
I20260709 12:24:09.806391 25575 leader_election.cc:290] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 19 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:09.807176 25575 consensus_queue.cc:1048] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20260709 12:24:09.796128 25421 leader_election.cc:290] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:09.810516 25464 leader_election.cc:290] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 19 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:09.810806 25621 consensus_queue.cc:1048] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:24:09.817795 25581 leader_election.cc:290] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 20 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:09.819635 25354 leader_election.cc:290] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:09.821354 25151 leader_election.cc:290] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:24:09.777683 25608 log.cc:927] Time spent T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.068s	user 0.001s	sys 0.000s
I20260709 12:24:09.823014 17756 raft_consensus.cc:1275] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 23 index: 15. Preceding OpId from leader: term: 24 index: 15. (term mismatch)
I20260709 12:24:09.823410 17756 pending_rounds.cc:85] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb: Aborting all ops after (but not including) 14
W20260709 12:24:09.784591 17803 consensus_peers.cc:597] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b -> Peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Couldn't send request to peer 334a4e1f59944544b3b3ff36616b57da. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:24:09.821000 25370 raft_consensus.cc:515] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:09.817903 17833 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35042 (request call id 2903) took 1464 ms. Trace:
I20260709 12:24:09.811429 25464 consensus_queue.cc:1059] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 12:24:09.839938 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:24:09.847898 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 69538.4us
W20260709 12:24:09.843950 18334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:09.847306 17804 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:09.824286 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:24:09.849498 18334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:09.826884 17687 raft_consensus.cc:1240] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 18. Current term is 19. Ops: []
W20260709 12:24:09.840368 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:09.850869 18334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:09.823689 17756 pending_rounds.cc:107] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb: Aborting uncommitted NO_OP operation due to leader change: 23.15
I20260709 12:24:09.851527 17756 raft_consensus.cc:2889] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:09.788544 17652 leader_election.cc:403] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 18 pre-election: Cancelling election due to peer responding with higher term
W20260709 12:24:09.837651 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:09.835642 17755 raft_consensus.cc:1240] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 18. Current term is 19. Ops: []
I20260709 12:24:09.829589 17831 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35042 (request call id 2902) took 1476 ms. Trace:
I20260709 12:24:09.819082 25131 leader_election.cc:290] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 20 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:09.848985 17833 rpcz_store.cc:276] 0709 12:24:08.353441 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:08.360565 (+  7124us) service_pool.cc:224] Handling call
0709 12:24:08.360882 (+   317us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:08.361259 (+   377us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:08.361287 (+    28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:09.383077 (+1021790us) spinlock_profiling.cc:243] Waited 6.21 ms on lock 0x7b2c000472a8. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b48f19a11 00007f7b48f29372 00007f7b4a65ba01 00007f7b4a62043a 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2
0709 12:24:09.383550 (+   473us) log.cc:844] Serialized 26 byte log entry
0709 12:24:09.598965 (+215415us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:24:09.599009 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:09.599082 (+    73us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:09.817543 (+218461us) raft_consensus.cc:1685] finished
0709 12:24:09.817571 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:09.817833 (+   262us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":412,"raft.run_cpu_time_us":73,"raft.run_wall_time_us":74,"spinlock_wait_cycles":14290304,"thread_start_us":214935,"threads_started":1,"wal-append.queue_time_us":375225}
I20260709 12:24:09.862373 17831 rpcz_store.cc:276] 0709 12:24:08.353124 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:08.354223 (+  1099us) service_pool.cc:224] Handling call
0709 12:24:08.354544 (+   321us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:08.354913 (+   369us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:08.354944 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:09.042813 (+687869us) spinlock_profiling.cc:243] Waited 12.4 ms on lock 0x7b2c000472a8. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b48f19a11 00007f7b48f29372 00007f7b4a65ba01 00007f7b4a62043a 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2
0709 12:24:09.044342 (+  1529us) log.cc:844] Serialized 26 byte log entry
0709 12:24:09.421083 (+376741us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:24:09.421139 (+    56us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:09.421234 (+    95us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:09.789399 (+368165us) raft_consensus.cc:1685] finished
0709 12:24:09.789428 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:09.829517 (+ 40089us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":369,"raft.run_cpu_time_us":83,"raft.run_wall_time_us":83,"spinlock_wait_cycles":28541696,"thread_start_us":375830,"threads_started":1,"wal-append.queue_time_us":521761}
I20260709 12:24:09.849155 18362 raft_consensus.cc:2468] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 22.
W20260709 12:24:09.870474 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 97834.8us
I20260709 12:24:09.870826 18362 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59902 (request call id 1845) took 1471 ms. Trace:
I20260709 12:24:09.871592 17652 leader_election.cc:304] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:09.823540 25489 leader_election.cc:290] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:09.783028 25586 raft_consensus.cc:493] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:09.874274 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:24:09.851565 18334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:09.875370 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:09.837466 25434 leader_election.cc:290] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:09.871342 18362 rpcz_store.cc:276] 0709 12:24:08.399803 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:08.676720 (+276917us) service_pool.cc:224] Handling call
0709 12:24:09.839740 (+1163020us) spinlock_profiling.cc:243] Waited 8.27 ms on lock 0x7b5800102398. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b474fbb71 00007f7b474fba22 00007f7b504148d9 00007f7b4f56eee0 00007f7b4f6380ab 00007f7b4f638131 00007f7b4f5e60a0 00007f7b4f5e5984 00007f7b48b606d4 00007f7b48b60662
0709 12:24:09.870744 (+ 31004us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":19022208}
I20260709 12:24:09.883028 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:24:09.868463 25345 consensus_meta.cc:287] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.566s	user 0.008s	sys 0.004s
I20260709 12:24:09.777860 25447 consensus_queue.cc:1048] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.085s
I20260709 12:24:09.878891 17762 raft_consensus.cc:2468] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 22.
I20260709 12:24:09.877712 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:09.874236 25587 raft_consensus.cc:493] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:09.841917 25485 leader_election.cc:290] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:09.859961 25182 raft_consensus.cc:493] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:09.825919 25126 leader_election.cc:290] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:09.865746 25613 consensus_queue.cc:1059] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.028s
I20260709 12:24:09.868098 25527 leader_election.cc:290] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:09.879637 18365 raft_consensus.cc:2468] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 19.
I20260709 12:24:09.894811 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:09.901190 18333 leader_election.cc:400] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 22 pre-election: Vote denied by peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651) with higher term. Message: Invalid argument: T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 22. Current term is 23.
I20260709 12:24:09.914093 18333 leader_election.cc:304] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:09.877519 25421 raft_consensus.cc:3055] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:24:09.908316 18371 raft_consensus.cc:1275] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 20 index: 17. Preceding OpId from leader: term: 22 index: 18. (index mismatch)
I20260709 12:24:09.865515 25611 consensus_queue.cc:1048] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.029s
I20260709 12:24:09.911095 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:09.918316 18365 raft_consensus.cc:3060] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:09.881264 25512 raft_consensus.cc:493] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:09.893649 25603 raft_consensus.cc:2749] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Leader pre-election lost for term 18. Reason: Vote denied by peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651) with higher term. Message: Invalid argument: T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for earlier term 18. Current term is 20.
I20260709 12:24:09.920392 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 23 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:09.921873 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:09.924955 25428 raft_consensus.cc:2749] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 12:24:09.925014 18374 raft_consensus.cc:2468] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 22.
I20260709 12:24:09.926337 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:09.903295 17803 leader_election.cc:304] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:24:09.927153 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:09.927977 18374 raft_consensus.cc:3060] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:09.928045 25620 raft_consensus.cc:2804] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:24:09.928788 25620 raft_consensus.cc:493] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:09.929337 25620 raft_consensus.cc:3060] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:09.927198 18371 raft_consensus.cc:3060] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:24:09.931733 17804 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:09.933524 17803 leader_election.cc:304] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:24:09.934806 25571 raft_consensus.cc:2804] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:24:09.935375 25571 raft_consensus.cc:493] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:09.936038 25571 raft_consensus.cc:3060] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:09.939569 18333 leader_election.cc:304] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:09.944208 25428 raft_consensus.cc:2749] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:24:09.946368 18333 leader_election.cc:304] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:24:09.880355 25616 consensus_queue.cc:1048] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.037s
W20260709 12:24:09.950501 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 68180.5us
I20260709 12:24:09.911262 25345 raft_consensus.cc:2749] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:24:09.881726 25586 raft_consensus.cc:515] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:09.948881 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 165235us
I20260709 12:24:09.895539 25434 consensus_queue.cc:1048] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.052s
I20260709 12:24:09.916216 25602 leader_election.cc:290] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:09.917364 25421 raft_consensus.cc:740] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 19 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:24:09.958357 17728 leader_election.cc:304] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
W20260709 12:24:09.958966 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 148634us
W20260709 12:24:09.960143 17804 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:09.960923 18365 raft_consensus.cc:2468] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 19.
I20260709 12:24:09.958096 18333 leader_election.cc:304] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:09.964356 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
W20260709 12:24:09.963390 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 88109us
I20260709 12:24:09.972646 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:09.968039 18371 raft_consensus.cc:2468] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 18.
W20260709 12:24:09.969251 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 358686us
I20260709 12:24:09.974498 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:09.968585 25428 raft_consensus.cc:2804] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:24:09.916321 25182 raft_consensus.cc:515] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:09.917912 25126 consensus_queue.cc:1059] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 20.16, Next index: 17, Last known committed idx: 14, Time since last communication: 0.044s
I20260709 12:24:10.036442 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:10.036870 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:10.037252 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:09.976715 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:10.037621 17834 raft_consensus.cc:3060] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:10.038096 17836 raft_consensus.cc:3060] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:10.038760 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:10.039288 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:10.040011 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:09.976939 25509 raft_consensus.cc:2749] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:24:09.955138 25603 raft_consensus.cc:493] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:09.981268 25602 consensus_queue.cc:1048] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:24:09.982671 25433 raft_consensus.cc:515] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:09.983001 25421 consensus_queue.cc:260] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 19.12, Last appended by leader: 12, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:10.058113 25421 raft_consensus.cc:3060] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:10.059165 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:09.987588 18361 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59886 (request call id 2017) took 1716 ms. Trace:
I20260709 12:24:10.060053 18361 rpcz_store.cc:276] 0709 12:24:08.271012 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:09.303841 (+1032829us) service_pool.cc:224] Handling call
0709 12:24:09.356443 (+ 52602us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:09.397556 (+ 41113us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:09.397599 (+    43us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:09.414551 (+ 16952us) log.cc:844] Serialized 26 byte log entry
0709 12:24:09.882690 (+468139us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:24:09.882731 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:09.882791 (+    60us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:09.987324 (+104533us) raft_consensus.cc:1685] finished
0709 12:24:09.987347 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:09.987513 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":42834,"raft.run_cpu_time_us":188,"raft.run_wall_time_us":7453,"thread_start_us":392180,"threads_started":1,"wal-append.queue_time_us":509720}
I20260709 12:24:10.062151 25433 leader_election.cc:290] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 24 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:09.976745 25133 consensus_queue.cc:1059] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 19.16, Next index: 17, Last known committed idx: 14, Time since last communication: 0.012s
I20260709 12:24:10.064592 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:09.990214 25471 raft_consensus.cc:515] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:09.992789 25631 raft_consensus.cc:2804] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:24:09.993364 25267 tablet_copy_client.cc:538] T a5732e71b9d54f8786b8c511d69943bf P 334a4e1f59944544b3b3ff36616b57da: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:24:09.993616 25485 consensus_queue.cc:1059] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20260709 12:24:10.072804 25485 raft_consensus.cc:3055] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:24:10.073199 25485 raft_consensus.cc:740] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:24:10.074162 25485 consensus_queue.cc:260] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.15, Last appended by leader: 15, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:10.075508 25485 raft_consensus.cc:3060] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:09.994382 18333 leader_election.cc:304] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:24:09.995076 18334 leader_election.cc:304] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
W20260709 12:24:10.077070 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 82053.2us
I20260709 12:24:10.077219 25133 raft_consensus.cc:3055] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:24:10.077675 25133 raft_consensus.cc:740] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:24:09.995589 25434 consensus_queue.cc:1059] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 19.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20260709 12:24:09.996119 25586 leader_election.cc:290] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:10.079038 25594 raft_consensus.cc:2804] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:24:09.911085 25587 raft_consensus.cc:515] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:10.002049 18366 raft_consensus.cc:1240] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 21. Current term is 22. Ops: []
I20260709 12:24:10.080363 18366 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59902 (request call id 1826) took 1872 ms. Trace:
I20260709 12:24:10.002534 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:10.003564 25370 leader_election.cc:290] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 25 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:09.951028 25512 raft_consensus.cc:515] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:10.004068 25622 raft_consensus.cc:515] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:10.006969 25582 log.cc:927] Time spent T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.796s	user 0.004s	sys 0.000s
I20260709 12:24:10.007375 17729 leader_election.cc:304] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
W20260709 12:24:10.102316 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 95005.2us
W20260709 12:24:10.008164 25570 log.cc:927] Time spent T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 1.204s	user 0.000s	sys 0.003s
I20260709 12:24:10.009097 25525 consensus_queue.cc:1048] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:24:09.951784 25486 leader_election.cc:290] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 19 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:24:10.015966 18367 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59902 (request call id 1841) took 1617 ms (client timeout 1998 ms). Trace:
I20260709 12:24:10.016448 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:10.112776 17681 raft_consensus.cc:2468] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 24.
I20260709 12:24:10.114187 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:10.114953 17681 raft_consensus.cc:3060] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:10.022291 25206 raft_consensus.cc:2749] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:24:10.026538 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 21 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:10.026871 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:10.033927 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:10.046769 17832 raft_consensus.cc:1275] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 16 index: 13. Preceding OpId from leader: term: 18 index: 14. (index mismatch)
I20260709 12:24:10.047699 25603 raft_consensus.cc:515] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:10.094439 25620 raft_consensus.cc:515] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:10.130327 25620 leader_election.cc:290] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 23 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:10.130470 17829 raft_consensus.cc:1240] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 22. Current term is 23. Ops: []
I20260709 12:24:10.051345 17834 raft_consensus.cc:2468] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 19.
I20260709 12:24:10.052971 25571 raft_consensus.cc:515] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:10.054229 25490 raft_consensus.cc:3060] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:10.056839 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:10.141161 17689 raft_consensus.cc:3060] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:10.062247 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:09.988004 25588 raft_consensus.cc:515] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:10.067685 17728 leader_election.cc:304] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:10.065564 17683 raft_consensus.cc:2468] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 26.
W20260709 12:24:10.145018 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 77441.3us
I20260709 12:24:10.145394 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:10.044091 25428 raft_consensus.cc:697] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 23 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:24:10.069746 25631 raft_consensus.cc:493] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:10.078599 25133 consensus_queue.cc:260] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 19.16, Last appended by leader: 16, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:10.082026 18371 raft_consensus.cc:2393] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 26: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:24:09.998865 18374 raft_consensus.cc:2468] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 18.
I20260709 12:24:10.110908 25528 raft_consensus.cc:2804] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:24:10.080823 18366 rpcz_store.cc:276] 0709 12:24:08.207580 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:10.001507 (+1793927us) service_pool.cc:224] Handling call
0709 12:24:10.001833 (+   326us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:10.079996 (+ 78163us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:10.080286 (+   290us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:10.175034 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 19 candidate_status { last_received { term: 18 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:10.175912 18366 raft_consensus.cc:3060] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:10.031600 17801 leader_election.cc:304] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:09.974545 17652 leader_election.cc:304] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:10.098475 17836 raft_consensus.cc:2468] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 19.
W20260709 12:24:10.177671 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 203262us
I20260709 12:24:10.181810 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:10.127194 18365 raft_consensus.cc:3060] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:10.185667 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:10.186776 17756 raft_consensus.cc:2393] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 20: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:24:10.187484 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:10.131975 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:10.132266 17685 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37396 (request call id 2950) took 2132 ms. Trace:
I20260709 12:24:10.142729 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:10.189543 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 19 candidate_status { last_received { term: 18 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:10.190274 17688 raft_consensus.cc:3060] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:10.189347 17685 rpcz_store.cc:276] 0709 12:24:08.000250 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:08.000461 (+   211us) service_pool.cc:224] Handling call
0709 12:24:08.021158 (+ 20697us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:08.027264 (+  6106us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 12:24:08.280067 (+252803us) log.cc:844] Serialized 14 byte log entry
0709 12:24:08.839571 (+559504us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:08.857600 (+ 18029us) log.cc:844] Serialized 26 byte log entry
0709 12:24:08.857756 (+   156us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:24:09.045929 (+188173us) spinlock_profiling.cc:243] Waited 2.18 ms on lock 0x7b18003a2dc0. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b474adc0a 00007f7b50bed219 00007f7b47535a40 00007f7b475304e4 00007f7b47530455 00007f7b4e0183c7 00007f7b4a62a9a7 00007f7b4a63a0bf 00007f7b4a63a038
0709 12:24:09.338989 (+293060us) log.cc:844] Serialized 14 byte log entry
0709 12:24:09.367702 (+ 28713us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:09.367820 (+   118us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:10.007542 (+639722us) spinlock_profiling.cc:243] Waited 7.19 ms on lock 0x7b4400022b4c. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b47459c85 00007f7b4745a6a2 00007f7b4a1cc1db 00007f7b4a1c7aaf 00007f7b4a1bb86e 00007f7b4a1babb6 00007f7b4a1ba7d8 00007f7b4a1c4221 00007f7b4a1c41d9 00007f7b4a1c4169
0709 12:24:10.132043 (+124501us) raft_consensus.cc:1685] finished
0709 12:24:10.132067 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:10.132208 (+   141us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":37494,"prepare.queue_time_us":677180,"prepare.run_cpu_time_us":105,"prepare.run_wall_time_us":1338,"raft.queue_time_us":217603,"raft.run_cpu_time_us":517,"raft.run_wall_time_us":70455,"spinlock_wait_cycles":21538560,"thread_start_us":1003844,"threads_started":4,"wal-append.queue_time_us":456429}
W20260709 12:24:10.142760 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 64895.3us
I20260709 12:24:10.194254 17762 raft_consensus.cc:1275] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 18 index: 14. Preceding OpId from leader: term: 20 index: 15. (index mismatch)
I20260709 12:24:10.146207 17728 leader_election.cc:304] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:24:10.196293 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 164809us
I20260709 12:24:10.160431 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:10.198356 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:10.117398 25434 consensus_queue.cc:1059] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 16.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.055s
W20260709 12:24:10.083071 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 52149.1us
I20260709 12:24:10.161744 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:24:10.202144 17803 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:10.162916 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:10.165290 25512 leader_election.cc:290] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:10.130327 25603 leader_election.cc:290] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:10.127123 25182 leader_election.cc:290] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:10.206493 18365 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 20.
I20260709 12:24:10.207435 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:10.176761 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
W20260709 12:24:10.111797 18367 rpcz_store.cc:269] 0709 12:24:08.398316 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:08.652533 (+254217us) service_pool.cc:224] Handling call
0709 12:24:10.015727 (+1363194us) spinlock_profiling.cc:243] Waited 763 ms on lock 0x7b54008020d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:24:10.015863 (+   136us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1753773952}
I20260709 12:24:10.148046 25638 raft_consensus.cc:3055] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:24:10.202060 17683 raft_consensus.cc:2468] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 23.
I20260709 12:24:10.162086 17830 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35042 (request call id 2922) took 1547 ms. Trace:
I20260709 12:24:10.219599 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:24:10.220413 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 226375us
I20260709 12:24:10.220489 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:10.208822 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 19 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:10.221601 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:10.215121 18365 raft_consensus.cc:3060] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Advancing to term 18
I20260709 12:24:10.215981 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:10.159086 25387 raft_consensus.cc:2749] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 12:24:10.217841 17688 raft_consensus.cc:2468] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 23.
I20260709 12:24:10.205816 17681 raft_consensus.cc:2468] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 24.
I20260709 12:24:10.223696 17755 raft_consensus.cc:2468] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 22.
I20260709 12:24:10.224243 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 23 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:10.225539 17688 raft_consensus.cc:2410] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for term 23 because replica has last-logged OpId of term: 21 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 13.
I20260709 12:24:10.224313 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:10.226473 17681 raft_consensus.cc:3060] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:10.199483 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:10.217911 17830 rpcz_store.cc:276] 0709 12:24:08.614315 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:08.617214 (+  2899us) service_pool.cc:224] Handling call
0709 12:24:08.617620 (+   406us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:08.617964 (+   344us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:08.617996 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:08.618464 (+   468us) log.cc:844] Serialized 26 byte log entry
0709 12:24:08.856110 (+237646us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:24:08.856152 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:08.856223 (+    71us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:10.006448 (+1150225us) spinlock_profiling.cc:243] Waited 3.36 ms on lock 0x7b4400022b4c. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b47459c85 00007f7b4745a6a2 00007f7b4a1cc1db 00007f7b4a1c7aaf 00007f7b4a1bb86e 00007f7b4a1babb6 00007f7b4a1ba7d8 00007f7b4a1c4221 00007f7b4a1c41d9 00007f7b4a1c4169
0709 12:24:10.161710 (+155262us) raft_consensus.cc:1685] finished
0709 12:24:10.161733 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:10.162002 (+   269us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1161,"raft.run_cpu_time_us":84,"raft.run_wall_time_us":85,"spinlock_wait_cycles":7722880,"thread_start_us":237115,"threads_started":1,"wal-append.queue_time_us":545801}
I20260709 12:24:10.158481 25631 raft_consensus.cc:3060] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:10.228950 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:10.229360 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:10.230219 17759 raft_consensus.cc:2393] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 22: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:24:10.220338 17757 raft_consensus.cc:2468] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 22.
I20260709 12:24:10.230739 17688 raft_consensus.cc:1240] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 20 LEADER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 19. Current term is 20. Ops: []
I20260709 12:24:10.231724 18366 raft_consensus.cc:2468] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 19.
I20260709 12:24:10.232477 17688 raft_consensus.cc:1240] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 24. Current term is 25. Ops: []
I20260709 12:24:10.233002 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:10.233912 17688 raft_consensus.cc:1240] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 23. Current term is 24. Ops: []
I20260709 12:24:10.233966 18365 raft_consensus.cc:2468] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 18.
I20260709 12:24:10.197031 25486 raft_consensus.cc:2804] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:24:10.236855 18367 raft_consensus.cc:1240] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 18. Current term is 19. Ops: [18.16-18.16]
W20260709 12:24:10.237113 17652 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:10.237442 17757 raft_consensus.cc:2393] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 19: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:24:10.258535 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:10.259099 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:10.261883 17654 leader_election.cc:304] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:24:10.263226 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 22 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:24:10.216449 25639 log.cc:927] Time spent T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.059s	user 0.000s	sys 0.000s
I20260709 12:24:10.263983 17764 raft_consensus.cc:2468] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 21.
I20260709 12:24:10.265298 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:10.202752 25594 raft_consensus.cc:697] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 22 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:24:10.206277 25528 raft_consensus.cc:697] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 19 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:24:10.217321 25133 raft_consensus.cc:3060] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:10.265610 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:10.216243 25206 raft_consensus.cc:2749] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:24:10.265748 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:10.271276 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:10.272380 17830 raft_consensus.cc:2468] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 22.
I20260709 12:24:10.264688 17729 leader_election.cc:304] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:10.269049 17681 raft_consensus.cc:2468] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 18.
I20260709 12:24:10.197239 25370 consensus_queue.cc:1048] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 11, Time since last communication: 0.066s
I20260709 12:24:10.266073 17832 raft_consensus.cc:2468] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 17.
W20260709 12:24:10.266762 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:10.277050 17761 raft_consensus.cc:1240] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 23 LEADER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 22. Current term is 23. Ops: []
I20260709 12:24:10.271216 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:10.217285 25638 raft_consensus.cc:740] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:24:10.219250 25589 raft_consensus.cc:493] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:10.272305 17757 raft_consensus.cc:2468] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 23.
W20260709 12:24:10.269490 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 48049.6us
I20260709 12:24:10.277029 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:10.287802 17764 raft_consensus.cc:3060] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:10.288055 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:10.280543 17835 raft_consensus.cc:2468] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 22.
I20260709 12:24:10.282958 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:10.288931 18364 raft_consensus.cc:1240] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 20. Current term is 21. Ops: []
I20260709 12:24:10.289158 17758 raft_consensus.cc:2468] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 22.
I20260709 12:24:10.239519 25424 raft_consensus.cc:2804] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:24:10.292124 25424 raft_consensus.cc:493] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:10.292709 25424 raft_consensus.cc:3060] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:10.298239 17652 leader_election.cc:304] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:10.268353 25602 raft_consensus.cc:2804] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:24:10.277666 25571 leader_election.cc:290] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 20 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:10.280226 25538 raft_consensus.cc:2749] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 12:24:10.287997 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 22 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:10.289772 18364 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59902 (request call id 1827) took 2076 ms. Trace:
I20260709 12:24:10.318336 18364 rpcz_store.cc:276] 0709 12:24:08.213711 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:10.288312 (+2074601us) service_pool.cc:224] Handling call
0709 12:24:10.288720 (+   408us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:10.289476 (+   756us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:10.289707 (+   231us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:10.319753 18364 raft_consensus.cc:1275] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 16 index: 12. Preceding OpId from leader: term: 18 index: 13. (index mismatch)
I20260709 12:24:10.304281 17764 raft_consensus.cc:2468] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 25.
I20260709 12:24:10.306141 17689 raft_consensus.cc:2468] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 22.
I20260709 12:24:10.273756 25182 raft_consensus.cc:2804] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:24:10.275403 25471 leader_election.cc:290] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 22 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:10.266887 25486 raft_consensus.cc:697] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 21 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:24:10.265543 25642 raft_consensus.cc:493] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:10.328243 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:10.329182 17832 raft_consensus.cc:2393] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 19: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
W20260709 12:24:10.315045 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 168949us
I20260709 12:24:10.320749 17757 raft_consensus.cc:1240] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 24. Current term is 25. Ops: []
I20260709 12:24:10.332538 17728 leader_election.cc:304] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:10.333452 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:10.334234 17836 raft_consensus.cc:2393] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 18: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:24:10.321448 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:10.314508 18333 leader_election.cc:400] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 22 election: Vote denied by peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651) with higher term. Message: Invalid argument: T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 22. Current term is 23.
I20260709 12:24:10.325459 17689 raft_consensus.cc:1240] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 24. Current term is 25. Ops: []
I20260709 12:24:10.285849 25434 raft_consensus.cc:3055] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:24:10.339604 17764 raft_consensus.cc:2468] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 22.
W20260709 12:24:10.343335 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:10.268815 25428 consensus_queue.cc:237] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 22.16, Last appended by leader: 16, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:10.375842 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 16 candidate_status { last_received { term: 14 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:10.333592 25512 raft_consensus.cc:493] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:24:10.382014 17801 leader_election.cc:304] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:24:10.336699 25182 raft_consensus.cc:697] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 26 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:24:10.336575 25642 raft_consensus.cc:515] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:10.383176 25471 raft_consensus.cc:2804] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:24:10.340530 25603 raft_consensus.cc:2749] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 12:24:10.383574 25471 raft_consensus.cc:493] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:10.384372 25471 raft_consensus.cc:3060] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:10.385003 25588 leader_election.cc:290] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 17 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:10.385612 25638 consensus_queue.cc:260] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 19.16, Last appended by leader: 16, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:10.378309 17689 raft_consensus.cc:1240] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 21. Current term is 22. Ops: [19.12-21.13]
I20260709 12:24:10.387493 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:10.382134 25512 raft_consensus.cc:515] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:10.388134 17764 raft_consensus.cc:3060] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:10.379700 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:10.388943 17832 raft_consensus.cc:2393] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 23: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:24:10.387319 17801 leader_election.cc:304] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:10.340304 25434 raft_consensus.cc:740] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:24:10.309543 25589 raft_consensus.cc:515] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:10.396845 25588 raft_consensus.cc:2804] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:24:10.383490 25182 consensus_queue.cc:237] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 25.19, Last appended by leader: 19, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:10.400660 25588 raft_consensus.cc:493] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:10.401301 25588 raft_consensus.cc:3060] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:10.405150 25643 raft_consensus.cc:493] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:10.406008 25643 raft_consensus.cc:515] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:10.408733 17689 raft_consensus.cc:1275] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 17 index: 11. Preceding OpId from leader: term: 20 index: 12. (index mismatch)
W20260709 12:24:10.410122 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:10.399183 25587 leader_election.cc:290] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:10.416514 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:10.417354 17836 raft_consensus.cc:2468] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 18.
I20260709 12:24:10.418532 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:10.326258 25640 consensus_queue.cc:1059] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.069s
I20260709 12:24:10.429590 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:10.436002 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:10.436884 17836 raft_consensus.cc:2468] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 23.
I20260709 12:24:10.430694 17764 raft_consensus.cc:2468] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 19.
I20260709 12:24:10.456560 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:10.457518 17836 raft_consensus.cc:3060] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:10.384126 25622 leader_election.cc:290] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 21 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:10.341075 25637 consensus_queue.cc:1059] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.087s
I20260709 12:24:10.397133 25422 raft_consensus.cc:2764] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Leader pre-election decision vote started in defunct term 25: won
W20260709 12:24:10.499511 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:10.509290 17836 raft_consensus.cc:2468] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 18.
I20260709 12:24:10.510735 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:10.437888 25602 raft_consensus.cc:697] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 19 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:24:10.512516 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:10.514405 25538 raft_consensus.cc:493] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:10.491667 25571 raft_consensus.cc:493] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:10.520236 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 239234us
I20260709 12:24:10.524945 25643 leader_election.cc:290] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:24:10.522761 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 224235us
I20260709 12:24:10.539549 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:10.517395 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:10.549794 18366 raft_consensus.cc:2468] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 16.
I20260709 12:24:10.540406 17801 leader_election.cc:304] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:10.541797 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
W20260709 12:24:10.544013 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 342139us
I20260709 12:24:10.371783 25594 consensus_queue.cc:237] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 21.15, Last appended by leader: 15, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:10.556488 17729 leader_election.cc:304] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:10.557940 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:10.377408 25528 consensus_queue.cc:237] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.13, Last appended by leader: 13, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:10.558826 17836 raft_consensus.cc:2393] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 17: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
W20260709 12:24:10.543686 25635 log.cc:927] Time spent T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.235s	user 0.001s	sys 0.000s
I20260709 12:24:10.562391 17836 raft_consensus.cc:1240] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 22. Current term is 23. Ops: []
I20260709 12:24:10.563179 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 23 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:10.560729 17729 leader_election.cc:304] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
W20260709 12:24:10.546108 25636 log.cc:927] Time spent T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.281s	user 0.001s	sys 0.000s
I20260709 12:24:10.568820 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:10.570021 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:10.571560 17764 raft_consensus.cc:1240] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 LEADER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 19. Current term is 20. Ops: [19.12-19.12]
I20260709 12:24:10.572222 17836 raft_consensus.cc:1240] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 18 LEADER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 17. Current term is 18. Ops: []
I20260709 12:24:10.573280 17652 leader_election.cc:304] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:24:10.576166 17764 raft_consensus.cc:1240] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 19. Current term is 20. Ops: []
I20260709 12:24:10.574522 17836 raft_consensus.cc:1240] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 18. Current term is 19. Ops: []
I20260709 12:24:10.576694 25603 raft_consensus.cc:2804] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:24:10.581503 25603 raft_consensus.cc:493] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:10.582026 25603 raft_consensus.cc:3060] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:10.589113 17836 raft_consensus.cc:1240] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 15. Current term is 17. Ops: []
I20260709 12:24:10.593267 17836 raft_consensus.cc:1240] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 19. Current term is 20. Ops: [18.15-19.16]
I20260709 12:24:10.557660 25620 raft_consensus.cc:2804] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Leader election won for term 19
W20260709 12:24:10.600329 17801 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:10.601737 18334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:10.603804 18334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:10.604668 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 266221us
I20260709 12:24:10.497057 25486 consensus_queue.cc:237] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 16.13, Last appended by leader: 13, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:10.613952 17836 raft_consensus.cc:1240] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 19. Current term is 20. Ops: [16.15-19.16]
I20260709 12:24:10.615430 18363 raft_consensus.cc:1275] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 16 index: 13. Preceding OpId from leader: term: 18 index: 14. (index mismatch)
I20260709 12:24:10.558172 25428 consensus_queue.cc:1048] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.049s
I20260709 12:24:10.553140 25571 raft_consensus.cc:515] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:10.546607 25538 raft_consensus.cc:515] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:10.629285 17652 leader_election.cc:304] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:10.630782 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:10.632201 17681 raft_consensus.cc:2393] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 20: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:24:10.538496 25434 consensus_queue.cc:260] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.15, Last appended by leader: 15, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:10.637168 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 19 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:10.648975 17836 raft_consensus.cc:1240] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 18. Current term is 19. Ops: []
I20260709 12:24:10.617733 25642 leader_election.cc:290] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:24:10.651001 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 90670.5us
I20260709 12:24:10.602274 25206 consensus_queue.cc:1059] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.033s
I20260709 12:24:10.648797 17652 leader_election.cc:304] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:10.627625 25656 raft_consensus.cc:3055] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:24:10.672014 17836 raft_consensus.cc:1240] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 18. Current term is 19. Ops: [14.10-18.12]
I20260709 12:24:10.672426 17652 leader_election.cc:304] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:24:10.636909 25370 raft_consensus.cc:2749] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:24:10.633616 25653 raft_consensus.cc:2804] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:24:10.686779 18333 leader_election.cc:304] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:24:10.711769 25206 raft_consensus.cc:3055] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:24:10.712436 25206 raft_consensus.cc:740] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 19 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:24:10.713680 25206 consensus_queue.cc:260] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 19.12, Last appended by leader: 12, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:10.715798 25206 raft_consensus.cc:3060] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:10.716470 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 25 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:10.661723 25656 raft_consensus.cc:740] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
W20260709 12:24:10.736722 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 50212.5us
W20260709 12:24:10.742339 18333 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:10.744222 17803 leader_election.cc:304] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:10.681653 25650 raft_consensus.cc:493] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:24:10.662251 25622 consensus_queue.cc:1048] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.038s
I20260709 12:24:10.658746 25662 consensus_queue.cc:1059] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 11, Time since last communication: 0.076s
I20260709 12:24:10.632120 25602 consensus_queue.cc:237] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 18.16, Last appended by leader: 16, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:10.697293 25647 raft_consensus.cc:2749] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
W20260709 12:24:10.767967 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 95622.3us
I20260709 12:24:10.755316 17803 leader_election.cc:304] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:10.695307 25512 leader_election.cc:290] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:10.696529 25370 raft_consensus.cc:2749] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 12:24:10.668798 25424 raft_consensus.cc:515] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:10.699271 25653 raft_consensus.cc:697] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
W20260709 12:24:10.773389 18333 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:10.774011 17652 leader_election.cc:304] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:24:10.783450 18333 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:10.778656 25182 raft_consensus.cc:493] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:10.750166 25645 consensus_queue.cc:1059] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 18.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.031s
I20260709 12:24:10.788213 25659 raft_consensus.cc:2749] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
W20260709 12:24:10.788558 25654 log.cc:927] Time spent T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.101s	user 0.000s	sys 0.001s
I20260709 12:24:10.786638 17652 leader_election.cc:304] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:10.790097 17652 leader_election.cc:304] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:10.776582 25662 consensus_queue.cc:1243] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Peer e2909cff1da44ad387f96a8ae3aa5afb log is divergent from this leader: its last log entry 15.12 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 11
I20260709 12:24:10.797562 18333 leader_election.cc:304] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:10.767678 25649 raft_consensus.cc:2804] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:24:10.792609 17652 leader_election.cc:304] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:10.767524 25637 raft_consensus.cc:2804] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:24:10.755232 25471 raft_consensus.cc:515] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:10.810230 17764 raft_consensus.cc:2393] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 25: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:24:10.803701 17682 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44898 (request call id 1519) took 1817 ms. Trace:
I20260709 12:24:10.812538 17682 rpcz_store.cc:276] 0709 12:24:08.986892 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:08.987332 (+   440us) service_pool.cc:224] Handling call
0709 12:24:08.988198 (+   866us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:09.601968 (+613770us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:10.803373 (+1201405us) spinlock_profiling.cc:243] Waited 613 ms on lock 0x7b54003480d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:24:10.803591 (+   218us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1409843456}
I20260709 12:24:10.757336 25603 raft_consensus.cc:515] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:10.789002 25645 raft_consensus.cc:3055] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:24:10.759536 25682 consensus_queue.cc:1048] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.043s
I20260709 12:24:10.793273 25659 raft_consensus.cc:2804] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:24:10.797143 25643 raft_consensus.cc:493] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:10.803148 25637 raft_consensus.cc:493] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:10.806144 25649 raft_consensus.cc:493] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:24:10.788240 25182 raft_consensus.cc:515] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:10.794497 25370 raft_consensus.cc:2804] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:24:10.820397 25573 raft_consensus.cc:2804] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:24:10.779702 25650 raft_consensus.cc:515] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:10.859794 25573 raft_consensus.cc:697] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 25 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
W20260709 12:24:10.861315 25606 log.cc:927] Time spent T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 1.131s	user 0.000s	sys 0.013s
I20260709 12:24:10.777727 25588 raft_consensus.cc:515] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:10.848176 25471 leader_election.cc:290] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 23 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:10.848682 25267 tablet_bootstrap.cc:492] T a5732e71b9d54f8786b8c511d69943bf P 334a4e1f59944544b3b3ff36616b57da: Bootstrap starting.
W20260709 12:24:10.848971 25631 consensus_meta.cc:287] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.582s	user 0.011s	sys 0.000s
W20260709 12:24:10.849797 25668 log.cc:927] Time spent T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.058s	user 0.000s	sys 0.000s
W20260709 12:24:10.850813 25667 log.cc:927] Time spent T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.159s	user 0.000s	sys 0.001s
W20260709 12:24:10.850935 25490 consensus_meta.cc:287] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.701s	user 0.006s	sys 0.003s
I20260709 12:24:10.820631 25645 raft_consensus.cc:740] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:24:10.853895 25182 leader_election.cc:290] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:10.835810 25659 raft_consensus.cc:493] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:10.856913 25589 leader_election.cc:290] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:10.796742 25656 consensus_queue.cc:260] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.14, Last appended by leader: 14, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:10.860893 25573 consensus_queue.cc:237] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 24.13, Last appended by leader: 13, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:10.843063 25643 raft_consensus.cc:515] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:10.816718 25674 raft_consensus.cc:2749] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
W20260709 12:24:10.845526 25661 log.cc:927] Time spent T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.084s	user 0.001s	sys 0.000s
I20260709 12:24:10.827157 25666 raft_consensus.cc:2749] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
W20260709 12:24:10.881424 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:10.884804 18363 raft_consensus.cc:1275] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 20 index: 14. Preceding OpId from leader: term: 21 index: 15. (index mismatch)
I20260709 12:24:10.843691 25637 raft_consensus.cc:3060] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:10.864845 25571 leader_election.cc:290] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:10.863396 25370 raft_consensus.cc:493] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:10.917464 25370 raft_consensus.cc:3060] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:10.920157 25490 raft_consensus.cc:493] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:24:10.862208 25649 raft_consensus.cc:3060] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:10.824055 18333 leader_election.cc:304] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:10.893837 25659 raft_consensus.cc:3060] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:10.887410 25538 leader_election.cc:290] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:10.891965 25345 raft_consensus.cc:493] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:10.934842 25643 leader_election.cc:290] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:10.905845 18362 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59902 (request call id 1852) took 1007 ms. Trace:
I20260709 12:24:10.891685 25471 raft_consensus.cc:493] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
W20260709 12:24:10.908349 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 212696us
I20260709 12:24:10.884981 25631 raft_consensus.cc:515] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:10.974748 17761 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36572 (request call id 1351) took 2050 ms. Trace:
I20260709 12:24:10.899834 25620 raft_consensus.cc:697] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 19 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:24:10.912492 17836 raft_consensus.cc:1275] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
I20260709 12:24:10.916836 18363 raft_consensus.cc:1275] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 18 index: 15. Preceding OpId from leader: term: 21 index: 16. (index mismatch)
I20260709 12:24:10.920833 25490 raft_consensus.cc:515] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:10.921326 25588 leader_election.cc:290] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 23 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:10.935039 25345 raft_consensus.cc:515] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:10.945497 25471 raft_consensus.cc:515] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:24:10.956058 25421 consensus_meta.cc:287] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.897s	user 0.014s	sys 0.000s
I20260709 12:24:10.944645 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 19 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:10.956749 25424 leader_election.cc:290] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 25 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:10.914286 25645 consensus_queue.cc:260] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.13, Last appended by leader: 13, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:10.940140 25672 raft_consensus.cc:493] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:24:10.896195 25653 consensus_queue.cc:237] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.12, Last appended by leader: 12, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:10.968813 17652 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:10.972268 25603 leader_election.cc:290] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 23 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:10.975189 17761 rpcz_store.cc:276] 0709 12:24:08.924538 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:08.924696 (+   158us) service_pool.cc:224] Handling call
0709 12:24:08.925057 (+   361us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:10.276791 (+1351734us) spinlock_profiling.cc:243] Waited 73.9 ms on lock 0x7b54003c6a20. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a597dc0 00007f7b4a6221c0 00007f7b4a623231 00007f7b4a61fd8f 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c
0709 12:24:10.281992 (+  5201us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:10.974423 (+692431us) spinlock_profiling.cc:243] Waited 35.6 ms on lock 0x7b54003c6a20. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a597dc0 00007f7b4a623a02 00007f7b4a61fdd6 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2
0709 12:24:10.974688 (+   265us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":251800320}
I20260709 12:24:10.982136 18362 rpcz_store.cc:276] 0709 12:24:09.898864 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:09.904168 (+  5304us) service_pool.cc:224] Handling call
0709 12:24:09.904619 (+   451us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:09.907906 (+  3287us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:09.907937 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:09.915952 (+  8015us) log.cc:844] Serialized 26 byte log entry
0709 12:24:09.999588 (+ 83636us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:24:10.097842 (+ 98254us) log.cc:844] Serialized 14 byte log entry
0709 12:24:10.850376 (+752534us) spinlock_profiling.cc:243] Waited 1.83 ms on lock 0x7b18001e8080. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b47515e07 00007f7b503709a9 00007f7b4753269b 00007f7b4753093f 00007f7b47530455 00007f7b4a618060 00007f7b4a62a716 00007f7b4a63b287 00007f7b4a63b208
0709 12:24:10.896208 (+ 45832us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:10.896332 (+   124us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:10.896404 (+    72us) raft_consensus.cc:1685] finished
0709 12:24:10.902671 (+  6267us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:10.905746 (+  3075us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":25971,"prepare.queue_time_us":98178,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":47,"raft.queue_time_us":185777,"raft.run_cpu_time_us":252,"raft.run_wall_time_us":39802,"spinlock_wait_cycles":5761024,"thread_start_us":189381,"threads_started":3,"wal-append.queue_time_us":167021}
I20260709 12:24:10.939795 25650 leader_election.cc:290] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:11.002076 18374 raft_consensus.cc:2393] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 19: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:24:11.002260 17831 raft_consensus.cc:1275] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 19 index: 12. Preceding OpId from leader: term: 20 index: 12. (term mismatch)
I20260709 12:24:11.004742 25620 consensus_queue.cc:237] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.12, Last appended by leader: 12, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:11.012570 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:11.012238 17831 pending_rounds.cc:85] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b: Aborting all ops after (but not including) 11
I20260709 12:24:11.013195 17831 pending_rounds.cc:107] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b: Aborting uncommitted NO_OP operation due to leader change: 19.12
I20260709 12:24:11.013248 17761 raft_consensus.cc:2468] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 18.
I20260709 12:24:11.013636 17831 raft_consensus.cc:2889] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:24:10.987988 25485 consensus_meta.cc:287] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.900s	user 0.013s	sys 0.000s
I20260709 12:24:11.015458 25599 raft_consensus.cc:3055] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:24:11.015692 18371 raft_consensus.cc:2410] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for term 19 because replica has last-logged OpId of term: 18 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 14.
I20260709 12:24:11.015945 25599 raft_consensus.cc:740] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 20 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:24:11.016901 25599 consensus_queue.cc:260] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 20.16, Last appended by leader: 16, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:11.017975 25599 raft_consensus.cc:3060] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:11.018036 18371 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59886 (request call id 2032) took 1069 ms. Trace:
I20260709 12:24:11.018743 18371 rpcz_store.cc:276] 0709 12:24:09.948956 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:10.216709 (+267753us) service_pool.cc:224] Handling call
0709 12:24:11.017791 (+801082us) spinlock_profiling.cc:243] Waited 756 ms on lock 0x7b5400467b54. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:24:11.017942 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1738967424}
I20260709 12:24:11.014372 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:11.022154 17761 raft_consensus.cc:3060] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:11.027417 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 25 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:11.029345 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:11.031755 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 21 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:11.026787 25424 raft_consensus.cc:493] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:11.033891 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:11.034757 17684 raft_consensus.cc:2468] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 20.
I20260709 12:24:11.033964 25424 raft_consensus.cc:515] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:11.047984 17758 raft_consensus.cc:1240] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 19 LEADER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 18. Current term is 19. Ops: [18.14-18.14]
I20260709 12:24:11.005018 25672 raft_consensus.cc:515] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:11.050158 17759 raft_consensus.cc:1240] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 21. Current term is 22. Ops: [21.13-21.13]
I20260709 12:24:11.026731 25630 raft_consensus.cc:493] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:11.054716 17759 raft_consensus.cc:1275] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 17 index: 13. Preceding OpId from leader: term: 21 index: 15. (index mismatch)
I20260709 12:24:11.059561 17761 raft_consensus.cc:2468] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 23.
I20260709 12:24:11.052610 25645 raft_consensus.cc:3060] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Advancing to term 19
W20260709 12:24:11.070015 18368 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59902 (request call id 1820) took 3023 ms (client timeout 2715 ms). Trace:
I20260709 12:24:11.070791 25672 leader_election.cc:290] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:24:11.070667 18368 rpcz_store.cc:269] 0709 12:24:08.039771 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:08.735829 (+696058us) service_pool.cc:224] Handling call
0709 12:24:11.063351 (+2327522us) spinlock_profiling.cc:243] Waited 575 ms on lock 0x7b54006af9d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:24:11.063494 (+   143us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1322697984}
I20260709 12:24:11.077294 25370 raft_consensus.cc:515] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:11.078269 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:11.078956 17682 raft_consensus.cc:2468] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 22.
I20260709 12:24:11.080504 25581 raft_consensus.cc:3055] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:24:11.080969 25581 raft_consensus.cc:740] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:24:11.081774 25581 consensus_queue.cc:260] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 23.14, Last appended by leader: 14, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:11.083132 25581 raft_consensus.cc:3060] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:11.039126 25637 raft_consensus.cc:515] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:11.088083 25615 log.cc:927] Time spent T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 1.310s	user 0.002s	sys 0.002s
I20260709 12:24:11.088330 25637 leader_election.cc:290] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 23 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:11.091090 17654 leader_election.cc:304] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:24:11.092743 25627 raft_consensus.cc:2804] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:24:11.093297 25471 leader_election.cc:290] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:11.093425 25627 raft_consensus.cc:493] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:11.094600 25627 raft_consensus.cc:3060] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Advancing to term 17
W20260709 12:24:11.105108 17801 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:11.103858 25652 log.cc:927] Time spent T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.474s	user 0.002s	sys 0.000s
I20260709 12:24:11.079826 25370 leader_election.cc:290] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 24 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:24:11.123899 17801 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:11.122484 25490 leader_election.cc:290] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:11.057180 25630 raft_consensus.cc:515] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:11.119535 25643 consensus_queue.cc:1048] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.008s
W20260709 12:24:11.136848 17801 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:11.138114 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 23 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:11.131765 25653 consensus_queue.cc:1048] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.008s
I20260709 12:24:11.142408 25630 leader_election.cc:290] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:11.144217 17758 raft_consensus.cc:1240] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 24. Current term is 25. Ops: [24.15-24.15]
I20260709 12:24:11.144542 17686 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37402 (request call id 3375) took 1374 ms. Trace:
I20260709 12:24:11.144376 25659 raft_consensus.cc:515] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:11.144949 17686 rpcz_store.cc:276] 0709 12:24:09.770229 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:09.770390 (+   161us) service_pool.cc:224] Handling call
0709 12:24:09.772410 (+  2020us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:09.793478 (+ 21068us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:11.144324 (+1350846us) spinlock_profiling.cc:243] Waited 20.4 ms on lock 0x7b5400206f54. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:24:11.144489 (+   165us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":46813952}
I20260709 12:24:11.146795 25659 leader_election.cc:290] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 22 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:11.147416 25631 leader_election.cc:290] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 19 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:11.127131 25531 consensus_queue.cc:1048] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.021s
W20260709 12:24:11.148855 25655 log.cc:927] Time spent T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.293s	user 0.002s	sys 0.000s
I20260709 12:24:11.159934 25179 raft_consensus.cc:3060] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:11.132694 25424 leader_election.cc:290] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:11.129216 25649 raft_consensus.cc:515] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:11.172287 16501 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:24:11.173456 16501 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
W20260709 12:24:11.178409 25665 log.cc:927] Time spent T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.327s	user 0.001s	sys 0.013s
I20260709 12:24:11.185061 25627 raft_consensus.cc:515] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:11.185722 18334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:11.186163 18334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:11.186482 18334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:11.186755 18334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:11.190721 17680 raft_consensus.cc:2468] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 19.
I20260709 12:24:11.192334 25434 raft_consensus.cc:3060] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:11.193444 25649 leader_election.cc:290] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 23 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:11.196897 17686 raft_consensus.cc:1240] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 18. Current term is 19. Ops: []
I20260709 12:24:11.187237 25627 leader_election.cc:290] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 17 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:11.191665 25384 raft_consensus.cc:493] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:11.198491 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
W20260709 12:24:11.198828 25698 log.cc:927] Time spent T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.063s	user 0.000s	sys 0.001s
I20260709 12:24:11.201373 25384 raft_consensus.cc:515] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:11.205364 17760 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36572 (request call id 1359) took 1987 ms. Trace:
I20260709 12:24:11.205765 17760 rpcz_store.cc:276] 0709 12:24:09.218344 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:09.218483 (+   139us) service_pool.cc:224] Handling call
0709 12:24:09.239648 (+ 21165us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:09.424384 (+184736us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:11.205113 (+1780729us) spinlock_profiling.cc:243] Waited 174 ms on lock 0x7b540009f1d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:24:11.205308 (+   195us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":400905088}
W20260709 12:24:11.211262 25644 log.cc:927] Time spent T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.831s	user 0.002s	sys 0.000s
I20260709 12:24:11.216217 25354 raft_consensus.cc:3055] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:24:11.217471 25354 raft_consensus.cc:740] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:24:11.218413 25354 consensus_queue.cc:260] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 20.12, Last appended by leader: 12, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:11.219743 25354 raft_consensus.cc:3060] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:11.224975 17686 raft_consensus.cc:1240] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 20. Current term is 21. Ops: []
I20260709 12:24:11.230202 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:11.230934 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:11.232097 17685 raft_consensus.cc:2393] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 17: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:24:11.233314 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:11.235014 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:11.236218 17682 raft_consensus.cc:2393] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 23: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:24:11.246749 17680 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37402 (request call id 3374) took 1489 ms. Trace:
I20260709 12:24:11.247408 17680 rpcz_store.cc:276] 0709 12:24:09.756925 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:09.757155 (+   230us) service_pool.cc:224] Handling call
0709 12:24:11.246493 (+1489338us) spinlock_profiling.cc:243] Waited 1.42 s on lock 0x7b54001aa254. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:24:11.246660 (+   167us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3267886464}
I20260709 12:24:11.251310 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:11.251619 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:11.252146 17680 raft_consensus.cc:3060] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Advancing to term 23
W20260709 12:24:11.233376 17802 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:11.254096 17802 leader_election.cc:304] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:11.264351 18367 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59902 (request call id 1853) took 1364 ms. Trace:
I20260709 12:24:11.264653 17680 raft_consensus.cc:2468] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 23.
I20260709 12:24:11.264967 18367 rpcz_store.cc:276] 0709 12:24:09.899759 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:10.229523 (+329764us) service_pool.cc:224] Handling call
0709 12:24:10.232307 (+  2784us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:10.237316 (+  5009us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:11.264063 (+1026747us) spinlock_profiling.cc:243] Waited 2.45 ms on lock 0x7b54001efd50. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a61f8ad 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:24:11.264266 (+   203us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5634944}
I20260709 12:24:11.300904 17802 leader_election.cc:304] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:24:11.302695 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 101852us
I20260709 12:24:11.267701 25464 raft_consensus.cc:2749] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:24:11.310390 25688 consensus_queue.cc:1059] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 20.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:24:11.309193 25588 consensus_queue.cc:1048] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.002s
I20260709 12:24:11.309139 25421 consensus_queue.cc:1059] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 19.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.004s
I20260709 12:24:11.317202 17802 leader_election.cc:304] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:24:11.318765 25571 raft_consensus.cc:3055] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:24:11.319391 25571 raft_consensus.cc:740] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 20 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:24:11.320295 25571 consensus_queue.cc:260] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 20.15, Last appended by leader: 15, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:11.323364 17836 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37994 (request call id 1477) took 1006 ms. Trace:
I20260709 12:24:11.323247 17652 leader_election.cc:304] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:11.323779 17836 rpcz_store.cc:276] 0709 12:24:10.317434 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:10.886719 (+569285us) service_pool.cc:224] Handling call
0709 12:24:10.887150 (+   431us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:10.975515 (+ 88365us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:11.323131 (+347616us) spinlock_profiling.cc:243] Waited 24.8 ms on lock 0x7b54003d4ad4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:24:11.323306 (+   175us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":57054464}
I20260709 12:24:11.318836 25421 raft_consensus.cc:3055] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:24:11.324723 17652 leader_election.cc:304] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:11.328200 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:11.309324 25599 raft_consensus.cc:2804] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:24:11.333905 17680 raft_consensus.cc:2376] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 22. Re-sending same reply.
I20260709 12:24:11.339687 17831 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35042 (request call id 2929) took 1307 ms. Trace:
I20260709 12:24:11.340121 17831 rpcz_store.cc:276] 0709 12:24:10.032283 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:10.032409 (+   126us) service_pool.cc:224] Handling call
0709 12:24:10.032691 (+   282us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:11.014217 (+981526us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:11.339401 (+325184us) spinlock_profiling.cc:243] Waited 969 ms on lock 0x7b5400267854. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:24:11.339631 (+   230us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2228787456}
I20260709 12:24:11.355244 17728 leader_election.cc:304] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:11.357478 17728 leader_election.cc:304] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:11.332684 25464 raft_consensus.cc:493] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:11.332767 25681 raft_consensus.cc:493] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
W20260709 12:24:11.400928 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 43551.3us
I20260709 12:24:11.423457 17728 leader_election.cc:304] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:11.402390 25678 raft_consensus.cc:2804] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Leader election won for term 23
W20260709 12:24:11.405324 25703 log.cc:927] Time spent T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.072s	user 0.000s	sys 0.000s
I20260709 12:24:11.405892 18364 raft_consensus.cc:3055] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 19 LEADER]: Stepping down as leader of term 19
W20260709 12:24:11.416188 25701 log.cc:927] Time spent T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.078s	user 0.000s	sys 0.000s
I20260709 12:24:11.309095 25620 consensus_queue.cc:1059] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 24.13, Next index: 14, Last known committed idx: 10, Time since last communication: 0.031s
I20260709 12:24:11.336589 25354 raft_consensus.cc:2804] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:24:11.337663 25421 raft_consensus.cc:740] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 21 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:24:11.358354 25586 raft_consensus.cc:2764] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Leader pre-election decision vote started in defunct term 21: won
I20260709 12:24:11.449901 18364 raft_consensus.cc:740] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:24:11.336611 25571 raft_consensus.cc:3060] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:11.359515 25688 raft_consensus.cc:493] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:11.394735 25422 raft_consensus.cc:2764] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Leader pre-election decision vote started in defunct term 22: won
I20260709 12:24:11.346563 25471 raft_consensus.cc:2804] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:24:11.357527 25674 raft_consensus.cc:493] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:11.346817 25599 raft_consensus.cc:697] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 24 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
W20260709 12:24:11.459703 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 142566us
I20260709 12:24:11.460470 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:11.461714 17831 raft_consensus.cc:1240] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 17. Current term is 18. Ops: []
I20260709 12:24:11.462805 17836 raft_consensus.cc:1275] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 14 index: 14. Preceding OpId from leader: term: 20 index: 17. (index mismatch)
I20260709 12:24:11.448586 25527 raft_consensus.cc:2749] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:24:11.464267 25686 consensus_queue.cc:1048] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:24:11.467329 17728 leader_election.cc:304] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:11.460078 18364 consensus_queue.cc:260] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 19.15, Last appended by leader: 15, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:11.453171 25354 raft_consensus.cc:493] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:11.453091 25678 raft_consensus.cc:697] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 23 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:24:11.447638 25464 raft_consensus.cc:515] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:11.479271 18364 raft_consensus.cc:3060] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:11.483600 17836 raft_consensus.cc:1275] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 20 index: 14. Preceding OpId from leader: term: 21 index: 15. (index mismatch)
I20260709 12:24:11.480135 17728 leader_election.cc:304] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:11.478366 17831 raft_consensus.cc:1240] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 21. Current term is 22. Ops: []
I20260709 12:24:11.459107 25131 consensus_queue.cc:1059] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.018s
I20260709 12:24:11.470427 25420 consensus_queue.cc:1059] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 23.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.012s
I20260709 12:24:11.449892 25681 raft_consensus.cc:515] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:11.476508 25471 raft_consensus.cc:697] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 23 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:24:11.471442 25688 raft_consensus.cc:515] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:11.484122 25422 raft_consensus.cc:2804] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:24:11.481354 25586 consensus_queue.cc:1048] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 14, Time since last communication: 0.018s
I20260709 12:24:11.477514 25708 raft_consensus.cc:493] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:11.490873 17728 leader_election.cc:304] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:24:11.482956 25354 raft_consensus.cc:3060] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:11.492506 25527 raft_consensus.cc:3055] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:24:11.508315 18364 raft_consensus.cc:1275] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 19 index: 15. Preceding OpId from leader: term: 20 index: 15. (term mismatch)
I20260709 12:24:11.480366 25575 consensus_queue.cc:1059] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 17.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.033s
I20260709 12:24:11.489607 25315 raft_consensus.cc:2804] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:24:11.516454 18364 pending_rounds.cc:85] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da: Aborting all ops after (but not including) 14
I20260709 12:24:11.517838 18364 pending_rounds.cc:107] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da: Aborting uncommitted NO_OP operation due to leader change: 19.15
I20260709 12:24:11.518235 18364 raft_consensus.cc:2889] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:11.518976 17834 raft_consensus.cc:1275] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 16 index: 12. Preceding OpId from leader: term: 18 index: 13. (index mismatch)
I20260709 12:24:11.519168 18364 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 2263) took 1508 ms. Trace:
I20260709 12:24:11.501219 25131 raft_consensus.cc:3055] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:24:11.494256 25677 raft_consensus.cc:493] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:24:11.496887 25688 leader_election.cc:290] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:11.518850 25315 raft_consensus.cc:493] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
W20260709 12:24:11.520738 25133 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 20 current term: 20 is higher.
I20260709 12:24:11.515631 25527 raft_consensus.cc:740] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 24 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:24:11.521597 17830 raft_consensus.cc:1275] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 20 index: 14. Preceding OpId from leader: term: 24 index: 16. (index mismatch)
W20260709 12:24:11.522845 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:11.501643 25678 consensus_queue.cc:237] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 21.15, Last appended by leader: 15, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:11.501978 25708 raft_consensus.cc:515] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:24:11.524161 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 700237us
I20260709 12:24:11.561430 18333 leader_election.cc:304] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:11.562672 18333 leader_election.cc:304] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:11.524663 25662 raft_consensus.cc:3060] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Advancing to term 23
W20260709 12:24:11.525316 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:11.568245 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:11.527813 18365 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59902 (request call id 1857) took 1614 ms. Trace:
I20260709 12:24:11.519625 18364 rpcz_store.cc:276] 0709 12:24:10.010976 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:10.320523 (+309547us) service_pool.cc:224] Handling call
0709 12:24:10.320759 (+   236us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:11.405785 (+1085026us) spinlock_profiling.cc:243] Waited 335 ms on lock 0x7b5400694ea0. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a597dc0 00007f7b4a6221c0 00007f7b4a623231 00007f7b4a61fd8f 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c
0709 12:24:11.518915 (+113130us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:11.519115 (+   200us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":770961152}
I20260709 12:24:11.528803 17682 raft_consensus.cc:1240] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 23. Current term is 24. Ops: []
I20260709 12:24:11.498697 25422 raft_consensus.cc:493] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:11.503311 25464 leader_election.cc:290] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:11.519042 17683 raft_consensus.cc:1275] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 23 index: 16. Preceding OpId from leader: term: 24 index: 16. (term mismatch)
I20260709 12:24:11.572955 17683 pending_rounds.cc:85] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3: Aborting all ops after (but not including) 15
I20260709 12:24:11.573372 17683 pending_rounds.cc:107] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3: Aborting uncommitted NO_OP operation due to leader change: 23.16
I20260709 12:24:11.573664 17683 raft_consensus.cc:2889] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:11.498433 25681 leader_election.cc:290] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:11.545037 25315 raft_consensus.cc:3060] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:11.534672 17801 leader_election.cc:304] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:24:11.577365 18333 leader_election.cc:304] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
W20260709 12:24:11.578461 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 43854.3us
I20260709 12:24:11.535974 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:11.537958 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:11.538756 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:11.539316 25690 raft_consensus.cc:493] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:11.585595 25681 consensus_queue.cc:1059] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:24:11.586360 25690 raft_consensus.cc:515] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:11.539577 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:11.588438 17760 raft_consensus.cc:2468] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 22.
I20260709 12:24:11.540179 25707 consensus_queue.cc:1048] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:24:11.588973 25485 consensus_queue.cc:1059] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:24:11.589954 17801 leader_election.cc:304] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:11.541380 17835 raft_consensus.cc:1275] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 20 index: 16. Preceding OpId from leader: term: 21 index: 16. (term mismatch)
I20260709 12:24:11.588565 17683 raft_consensus.cc:1240] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 22. Current term is 23. Ops: []
I20260709 12:24:11.547982 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 23 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:11.592120 17833 raft_consensus.cc:2410] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for term 23 because replica has last-logged OpId of term: 21 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 12.
I20260709 12:24:11.592370 17682 raft_consensus.cc:1240] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 21. Current term is 22. Ops: [19.12-21.13]
I20260709 12:24:11.517002 25657 raft_consensus.cc:2804] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:24:11.553651 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:11.556327 17686 raft_consensus.cc:1240] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 22. Current term is 23. Ops: []
W20260709 12:24:11.558001 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:11.597324 17688 raft_consensus.cc:2468] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 24.
I20260709 12:24:11.563360 17685 raft_consensus.cc:1240] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 21. Current term is 22. Ops: [19.14-21.15]
I20260709 12:24:11.565729 17687 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37402 (request call id 3371) took 2103 ms. Trace:
I20260709 12:24:11.480201 25674 raft_consensus.cc:515] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:11.532279 25131 raft_consensus.cc:740] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:24:11.569149 18365 rpcz_store.cc:276] 0709 12:24:09.913731 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:10.235188 (+321457us) service_pool.cc:224] Handling call
0709 12:24:10.235501 (+   313us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:10.324065 (+ 88564us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:10.324116 (+    51us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:10.337360 (+ 13244us) log.cc:844] Serialized 26 byte log entry
0709 12:24:10.536519 (+199159us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:24:10.561170 (+ 24651us) log.cc:844] Serialized 14 byte log entry
0709 12:24:11.103268 (+542098us) spinlock_profiling.cc:243] Waited 1.8 ms on lock 0x7b4400022b4c. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b47459c85 00007f7b4745a6a2 00007f7b4a1cc1db 00007f7b4a1c7aaf 00007f7b4a1bb86e 00007f7b4a1babb6 00007f7b4a1ba7d8 00007f7b4a1c4221 00007f7b4a1c41d9 00007f7b4a1c4169
0709 12:24:11.181292 (+ 78024us) spinlock_profiling.cc:243] Waited 29.2 ms on lock 0x7b1800165fa0. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b474adc0a 00007f7b50bed219 00007f7b4a1d6d70 00007f7b4a1ca293 00007f7b4a1bfcea 00007f7b4a1c023a 00007f7b4a62aa2a 00007f7b4a63a0bf 00007f7b4a63a038
0709 12:24:11.527460 (+346168us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:11.527569 (+   109us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:11.527611 (+    42us) raft_consensus.cc:1685] finished
0709 12:24:11.527629 (+    18us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:11.527749 (+   120us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":450942,"prepare.queue_time_us":27506,"prepare.run_cpu_time_us":51,"prepare.run_wall_time_us":41,"raft.queue_time_us":432498,"raft.run_cpu_time_us":190,"raft.run_wall_time_us":1598,"spinlock_wait_cycles":71302784,"thread_start_us":174200,"threads_started":2,"wal-append.queue_time_us":181131}
I20260709 12:24:11.580245 17759 raft_consensus.cc:3060] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:11.582397 25631 consensus_queue.cc:1059] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 18.12, Next index: 13, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:24:11.492424 25421 consensus_queue.cc:260] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 11, Last appended: 21.13, Last appended by leader: 13, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:24:11.584173 25464 consensus_queue.cc:1059] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 17.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20260709 12:24:11.584673 25594 raft_consensus.cc:2749] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260709 12:24:11.552434 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:11.590821 17835 pending_rounds.cc:85] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b: Aborting all ops after (but not including) 15
I20260709 12:24:11.634929 17835 pending_rounds.cc:107] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b: Aborting uncommitted NO_OP operation due to leader change: 20.16
I20260709 12:24:11.635306 17835 raft_consensus.cc:2889] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:11.594465 17683 raft_consensus.cc:1275] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 25 index: 19. Preceding OpId from leader: term: 26 index: 20. (index mismatch)
I20260709 12:24:11.600059 17687 rpcz_store.cc:276] 0709 12:24:09.462514 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:09.462682 (+   168us) service_pool.cc:224] Handling call
0709 12:24:09.463054 (+   372us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:09.866222 (+403168us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:11.565306 (+1699084us) spinlock_profiling.cc:243] Waited 364 ms on lock 0x7b54004263d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:24:11.565640 (+   334us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":836251008}
I20260709 12:24:11.599478 17802 leader_election.cc:304] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:24:11.590441 25637 raft_consensus.cc:3055] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:24:11.622889 17801 leader_election.cc:304] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:11.572189 25642 raft_consensus.cc:2804] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:24:11.597043 25688 raft_consensus.cc:2804] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:24:11.629304 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 19 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:24:11.672319 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 49510.9us
W20260709 12:24:11.673888 17801 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:11.674434 17801 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:11.554064 25599 consensus_queue.cc:237] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 23.12, Last appended by leader: 12, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:24:11.632406 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 55116.5us
I20260709 12:24:11.598642 25512 raft_consensus.cc:2804] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:24:11.577394 25422 raft_consensus.cc:3060] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:11.601222 25681 raft_consensus.cc:2804] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:24:11.668442 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:11.690680 18368 raft_consensus.cc:2468] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 19.
I20260709 12:24:11.624878 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 25 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:11.692767 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 25 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:11.684361 25588 raft_consensus.cc:2749] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:24:11.695207 25641 raft_consensus.cc:2749] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 12:24:11.695456 25588 raft_consensus.cc:493] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:11.695797 25681 raft_consensus.cc:493] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:11.696240 25681 raft_consensus.cc:3060] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Advancing to term 23
W20260709 12:24:11.700421 18361 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59886 (request call id 2028) took 1799 ms (client timeout 2158 ms). Trace:
W20260709 12:24:11.700850 18361 rpcz_store.cc:269] 0709 12:24:09.901279 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:10.176418 (+275139us) service_pool.cc:224] Handling call
0709 12:24:11.700247 (+1523829us) spinlock_profiling.cc:243] Waited 789 ms on lock 0x7b54007602d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:24:11.700366 (+   119us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1815559552}
I20260709 12:24:11.630152 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 21 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:11.703073 18365 raft_consensus.cc:2468] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 20.
I20260709 12:24:11.703495 25206 raft_consensus.cc:3055] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:24:11.703923 25206 raft_consensus.cc:740] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 24 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:24:11.704324 18365 raft_consensus.cc:1275] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 17 index: 13. Preceding OpId from leader: term: 19 index: 14. (index mismatch)
I20260709 12:24:11.704694 25206 consensus_queue.cc:260] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 24.13, Last appended by leader: 13, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:11.705853 25674 leader_election.cc:290] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:11.706528 25674 raft_consensus.cc:493] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:11.696069 25588 raft_consensus.cc:515] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:11.708814 25588 leader_election.cc:290] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:24:11.709112 25709 log.cc:927] Time spent T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.075s	user 0.000s	sys 0.000s
I20260709 12:24:11.709354 25588 raft_consensus.cc:2804] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:24:11.563797 25471 consensus_queue.cc:237] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 21.14, Last appended by leader: 14, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:11.710153 25588 raft_consensus.cc:493] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:11.710861 25588 raft_consensus.cc:3060] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:24:11.713040 18374 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59902 (request call id 1854) took 1812 ms (client timeout 2285 ms). Trace:
W20260709 12:24:11.713753 18374 rpcz_store.cc:269] 0709 12:24:09.900520 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:10.199265 (+298745us) service_pool.cc:224] Handling call
0709 12:24:11.712777 (+1513512us) spinlock_profiling.cc:243] Waited 774 ms on lock 0x7b5400570154. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:24:11.712950 (+   173us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1779237120}
I20260709 12:24:11.706569 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:11.716087 18368 raft_consensus.cc:3060] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Advancing to term 21
W20260709 12:24:11.638056 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 44669.7us
I20260709 12:24:11.546007 25677 raft_consensus.cc:515] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:11.664463 25434 raft_consensus.cc:3055] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:24:11.672894 18364 raft_consensus.cc:2410] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for term 19 because replica has last-logged OpId of term: 17 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 11.
I20260709 12:24:11.675698 18333 leader_election.cc:304] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:11.705969 25206 raft_consensus.cc:3060] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Advancing to term 25
W20260709 12:24:11.718991 18334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:11.621841 25657 raft_consensus.cc:697] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 22 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:24:11.669054 25642 raft_consensus.cc:493] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:11.667573 25666 raft_consensus.cc:493] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:11.735491 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 59931.2us
W20260709 12:24:11.736675 18334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:11.673424 25464 consensus_queue.cc:1243] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer e2909cff1da44ad387f96a8ae3aa5afb log is divergent from this leader: its last log entry 19.14 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 13
I20260709 12:24:11.665035 25637 raft_consensus.cc:740] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:24:11.707235 25674 raft_consensus.cc:515] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:11.759541 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:11.634086 25580 consensus_queue.cc:1048] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.096s
I20260709 12:24:11.761266 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 21 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:11.815178 18333 leader_election.cc:304] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:24:11.815938 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:11.804745 18368 raft_consensus.cc:2468] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 21.
I20260709 12:24:11.817624 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
W20260709 12:24:11.827386 18334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:11.835251 18364 raft_consensus.cc:3060] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:11.720323 25688 raft_consensus.cc:697] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 23 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:24:11.630923 25527 consensus_queue.cc:260] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 24.13, Last appended by leader: 13, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:24:11.838764 17804 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:11.838672 18333 leader_election.cc:304] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:11.730265 25690 leader_election.cc:290] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:11.844763 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 16 candidate_status { last_received { term: 15 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:11.853704 18368 raft_consensus.cc:2468] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 15.
I20260709 12:24:11.861780 17755 raft_consensus.cc:1240] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 LEADER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 20. Current term is 21. Ops: []
I20260709 12:24:11.863480 17755 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38786 (request call id 3642) took 1584 ms. Trace:
I20260709 12:24:11.864202 17755 rpcz_store.cc:276] 0709 12:24:10.278696 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:10.278821 (+   125us) service_pool.cc:224] Handling call
0709 12:24:10.308545 (+ 29724us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:11.861565 (+1553020us) spinlock_profiling.cc:243] Waited 9.06 ms on lock 0x7b54003c6f20. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a597dc0 00007f7b4a6221c0 00007f7b4a623231 00007f7b4a61fd8f 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c
0709 12:24:11.863196 (+  1631us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:11.863424 (+   228us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":20850304}
I20260709 12:24:11.866938 18364 raft_consensus.cc:2468] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 17.
W20260709 12:24:11.835700 25267 log.cc:927] Time spent T a5732e71b9d54f8786b8c511d69943bf P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.090s	user 0.003s	sys 0.002s
I20260709 12:24:11.762542 25642 raft_consensus.cc:3060] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:11.883968 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:24:11.890626 17804 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:11.897078 17654 leader_election.cc:304] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:11.901444 18368 raft_consensus.cc:1240] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 20. Current term is 21. Ops: []
W20260709 12:24:11.869799 25679 log.cc:927] Time spent T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.069s	user 0.000s	sys 0.000s
I20260709 12:24:11.863101 25680 raft_consensus.cc:2804] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:24:11.908432 17831 raft_consensus.cc:1240] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 24 LEADER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 23. Current term is 24. Ops: []
I20260709 12:24:11.862030 25428 raft_consensus.cc:493] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:11.913497 18368 raft_consensus.cc:1275] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 19 index: 16. Preceding OpId from leader: term: 20 index: 17. (index mismatch)
I20260709 12:24:11.918525 18368 raft_consensus.cc:1240] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 22. Current term is 23. Ops: []
I20260709 12:24:11.861747 25645 raft_consensus.cc:2749] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
W20260709 12:24:11.891956 25665 log.cc:927] Time spent T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.055s	user 0.000s	sys 0.001s
I20260709 12:24:11.871608 25388 raft_consensus.cc:493] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:11.902745 25527 raft_consensus.cc:3060] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:11.852130 25664 consensus_queue.cc:1048] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.092s
I20260709 12:24:11.845960 25666 raft_consensus.cc:515] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:11.869163 25594 consensus_queue.cc:1059] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true }, Status: INVALID_TERM, Last received: 12.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.073s
I20260709 12:24:11.942642 25680 raft_consensus.cc:697] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 18 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
W20260709 12:24:11.968536 25655 log.cc:927] Time spent T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.055s	user 0.001s	sys 0.000s
I20260709 12:24:11.853269 25131 consensus_queue.cc:260] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 23.12, Last appended by leader: 12, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:24:12.029392 17730 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:11.938560 25485 consensus_queue.cc:1048] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.067s
I20260709 12:24:12.031502 18363 raft_consensus.cc:2468] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 21.
I20260709 12:24:11.986464 25677 leader_election.cc:290] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:24:11.663143 25589 consensus_queue.cc:1175] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" responder_term: 22 status { last_received { term: 19 index: 12 } last_committed_idx: 11 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 21. Current term is 22. Ops: [21.13-21.13]" } } last_received_current_leader { term: 0 index: 0 } }
I20260709 12:24:11.942304 25428 raft_consensus.cc:515] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:11.730623 25434 raft_consensus.cc:740] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
W20260709 12:24:12.049926 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 450491us
I20260709 12:24:11.998348 25669 consensus_queue.cc:1059] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.381s
I20260709 12:24:11.921195 25637 consensus_queue.cc:260] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 23.17, Last appended by leader: 17, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:11.962718 25388 raft_consensus.cc:515] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:12.037532 25571 consensus_meta.cc:287] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.567s	user 0.009s	sys 0.002s
I20260709 12:24:11.758915 25641 raft_consensus.cc:2804] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Leader pre-election won for term 24
W20260709 12:24:12.043236 25715 log.cc:927] Time spent T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.102s	user 0.000s	sys 0.001s
I20260709 12:24:12.043551 25708 leader_election.cc:290] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:12.032282 25681 raft_consensus.cc:515] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:12.063553 25131 raft_consensus.cc:3060] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:12.077844 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:12.100819 17682 raft_consensus.cc:2468] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 21.
I20260709 12:24:12.079943 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:12.101851 25345 leader_election.cc:290] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:24:12.102355 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:12.059345 25594 raft_consensus.cc:493] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:12.104465 25674 leader_election.cc:290] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:12.105023 25674 raft_consensus.cc:493] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:12.104326 25594 raft_consensus.cc:515] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:12.105459 25674 raft_consensus.cc:515] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:12.103238 25434 consensus_queue.cc:260] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.12, Last appended by leader: 12, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:12.107685 25434 raft_consensus.cc:3060] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:11.956048 25657 consensus_queue.cc:237] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 19.12, Last appended by leader: 12, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:24:12.042575 25574 raft_consensus.cc:3055] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 20 LEADER]: Stepping down as leader of term 20
W20260709 12:24:12.095738 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 228878us
I20260709 12:24:12.099582 25721 raft_consensus.cc:493] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:12.107848 25674 leader_election.cc:290] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:12.118803 25663 consensus_queue.cc:1048] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:24:12.119136 25721 raft_consensus.cc:515] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:12.122359 25721 leader_election.cc:290] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:12.100121 25571 raft_consensus.cc:493] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:12.101888 17680 raft_consensus.cc:2393] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 25: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:24:12.123459 25571 raft_consensus.cc:515] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:12.126950 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:12.127322 25571 leader_election.cc:290] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:12.129513 25428 leader_election.cc:290] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:12.130321 25428 raft_consensus.cc:3055] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:24:12.130798 25428 raft_consensus.cc:740] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
W20260709 12:24:12.132332 25711 log.cc:927] Time spent T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.297s	user 0.000s	sys 0.001s
I20260709 12:24:12.134418 17838 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35042 (request call id 2928) took 2102 ms. Trace:
I20260709 12:24:12.134881 17838 rpcz_store.cc:276] 0709 12:24:10.031823 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:10.032019 (+   196us) service_pool.cc:224] Handling call
0709 12:24:10.032354 (+   335us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:10.032820 (+   466us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:10.032862 (+    42us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:10.033436 (+   574us) log.cc:844] Serialized 26 byte log entry
0709 12:24:11.533342 (+1499906us) spinlock_profiling.cc:243] Waited 17.6 ms on lock 0x7b18002dc100. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b47515e07 00007f7b503709a9 00007f7b4753269b 00007f7b4753093f 00007f7b47532c02 00007f7b4a1ba20b 00007f7b4a1bfd23 00007f7b4a1c00d1 00007f7b4a5dba0f
0709 12:24:11.596399 (+ 63057us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:24:11.596443 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:11.596558 (+   115us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:12.134026 (+537468us) raft_consensus.cc:1685] finished
0709 12:24:12.134056 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:12.134317 (+   261us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":3777,"raft.run_cpu_time_us":185,"raft.run_wall_time_us":8129,"spinlock_wait_cycles":40452992,"thread_start_us":62824,"threads_started":1,"wal-append.queue_time_us":1617181}
W20260709 12:24:12.140614 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:12.145645 25388 leader_election.cc:290] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:12.082165 25575 raft_consensus.cc:493] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:12.146868 25575 raft_consensus.cc:515] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:12.150924 25575 leader_election.cc:290] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:24:12.085939 25354 consensus_meta.cc:287] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.571s	user 0.010s	sys 0.001s
I20260709 12:24:12.160328 25354 raft_consensus.cc:515] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:12.163249 25354 leader_election.cc:290] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 18 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:12.006738 25688 consensus_queue.cc:237] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 21.12, Last appended by leader: 12, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:12.095008 17759 raft_consensus.cc:2468] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 23.
W20260709 12:24:12.092836 17802 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:12.075821 25677 raft_consensus.cc:493] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:12.052377 25620 consensus_queue.cc:1175] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "0da169ff82674dc1947055ce43de68a3" responder_term: 22 status { last_received { term: 19 index: 11 } last_committed_idx: 11 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 21. Current term is 22. Ops: [19.12-21.13]" } } last_received_current_leader { term: 0 index: 0 } }
I20260709 12:24:12.092846 25641 raft_consensus.cc:493] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:12.109297 25594 leader_election.cc:290] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:24:12.171590 17802 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:12.111089 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:12.097622 25599 raft_consensus.cc:493] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:12.191965 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 21 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:24:12.178234 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 320406us
I20260709 12:24:12.116000 25574 raft_consensus.cc:740] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:24:12.117152 25669 raft_consensus.cc:493] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:12.187609 17687 raft_consensus.cc:2468] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 21.
I20260709 12:24:12.191759 25641 raft_consensus.cc:3060] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:24:12.209079 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:12.214890 25574 consensus_queue.cc:260] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 20.16, Last appended by leader: 16, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:12.216317 25574 raft_consensus.cc:3060] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Advancing to term 21
W20260709 12:24:12.217901 25662 consensus_meta.cc:287] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.642s	user 0.011s	sys 0.000s
I20260709 12:24:12.218377 25662 raft_consensus.cc:2749] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:24:12.196343 25464 consensus_queue.cc:1059] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 17.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.024s
I20260709 12:24:12.220914 25681 leader_election.cc:290] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 23 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:12.221581 25681 consensus_queue.cc:1059] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 22.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20260709 12:24:12.222563 25681 consensus_queue.cc:1059] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 18.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20260709 12:24:12.223097 18371 raft_consensus.cc:1240] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 22. Current term is 23. Ops: []
I20260709 12:24:12.226250 17837 raft_consensus.cc:2468] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 21.
W20260709 12:24:12.235044 18334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:12.184475 25677 raft_consensus.cc:515] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:12.246445 25677 leader_election.cc:290] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:12.200631 25599 raft_consensus.cc:515] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:12.260538 25720 log.cc:927] Time spent T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.247s	user 0.001s	sys 0.000s
I20260709 12:24:12.261225 25641 raft_consensus.cc:515] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:12.264492 25641 leader_election.cc:290] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 24 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:12.182255 25680 consensus_queue.cc:237] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 17.14, Last appended by leader: 14, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:12.273202 25571 raft_consensus.cc:3055] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:24:12.281536 25421 raft_consensus.cc:3060] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:12.199694 25650 raft_consensus.cc:3060] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:12.208520 25428 consensus_queue.cc:260] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 22.16, Last appended by leader: 16, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:12.235529 25315 consensus_meta.cc:287] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.613s	user 0.011s	sys 0.000s
W20260709 12:24:12.294122 18367 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.655s	user 0.023s	sys 0.035s
I20260709 12:24:12.296623 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 23 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:24:12.304633 18367 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.665s	user 0.023s	sys 0.035s
I20260709 12:24:12.301689 25133 raft_consensus.cc:493] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:12.292065 25674 raft_consensus.cc:3055] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:24:12.295297 25599 leader_election.cc:290] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:12.305620 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:12.307307 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:12.294535 25721 raft_consensus.cc:493] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:12.200901 25669 raft_consensus.cc:515] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:12.318228 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 19 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:24:12.318570 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:12.298890 25571 raft_consensus.cc:740] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 22 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:24:12.315104 18374 raft_consensus.cc:3060] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:12.322036 25428 raft_consensus.cc:3060] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Advancing to term 23
W20260709 12:24:12.306334 25422 consensus_meta.cc:287] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.596s	user 0.013s	sys 0.000s
I20260709 12:24:12.308274 18333 leader_election.cc:304] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:24:12.305245 25662 raft_consensus.cc:493] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:12.337893 17804 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:12.339385 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:12.308636 25674 raft_consensus.cc:740] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 18 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:24:12.317634 25677 raft_consensus.cc:493] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:12.348948 17801 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.674s	user 0.015s	sys 0.009s
I20260709 12:24:12.320396 25574 raft_consensus.cc:493] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
W20260709 12:24:12.351493 17801 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.677s	user 0.015s	sys 0.009s
W20260709 12:24:12.353700 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 45527.6us
I20260709 12:24:12.358083 18333 leader_election.cc:304] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:12.359237 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:12.360365 17759 raft_consensus.cc:2468] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 21.
I20260709 12:24:12.310755 25315 raft_consensus.cc:515] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:12.325516 25725 raft_consensus.cc:493] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:12.341919 25664 raft_consensus.cc:493] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:12.308655 25133 raft_consensus.cc:515] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:12.370833 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:12.354518 18374 raft_consensus.cc:2468] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 22.
I20260709 12:24:12.377779 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:12.324635 25599 consensus_queue.cc:1059] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 16.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.044s
I20260709 12:24:12.323396 25721 raft_consensus.cc:515] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:12.378667 18374 raft_consensus.cc:2393] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 17: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:24:12.393579 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:24:12.381275 17801 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:12.409551 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 20 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:12.342346 25422 raft_consensus.cc:515] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:12.414834 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 20 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:12.415621 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 21 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:12.413946 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:12.363293 25574 raft_consensus.cc:515] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:12.347086 25662 raft_consensus.cc:515] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:12.407819 18374 raft_consensus.cc:2393] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 24: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:24:12.411121 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:12.425215 17834 raft_consensus.cc:2468] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 20.
I20260709 12:24:12.423497 18368 raft_consensus.cc:2410] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 20 because replica has last-logged OpId of term: 18 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 14.
I20260709 12:24:12.416143 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:12.368582 25594 raft_consensus.cc:3060] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:12.427160 17838 raft_consensus.cc:2468] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 21.
I20260709 12:24:12.428642 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:12.426136 17835 raft_consensus.cc:2468] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 19.
I20260709 12:24:12.429678 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:12.431257 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
W20260709 12:24:12.432008 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 241403us
W20260709 12:24:12.415774 25727 log.cc:927] Time spent T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.095s	user 0.000s	sys 0.000s
I20260709 12:24:12.354504 25677 raft_consensus.cc:515] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:24:12.435207 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:12.436719 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 19 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:12.371439 25728 raft_consensus.cc:2804] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Leader pre-election won for term 19
W20260709 12:24:12.438433 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:12.375345 25725 raft_consensus.cc:515] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:12.417140 25131 raft_consensus.cc:493] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:12.441576 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:12.438828 25728 raft_consensus.cc:493] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:12.445448 25133 leader_election.cc:290] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:12.454082 25315 leader_election.cc:290] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 23 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:12.457652 25574 leader_election.cc:290] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:12.459517 25662 leader_election.cc:290] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:12.460194 25388 raft_consensus.cc:493] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
W20260709 12:24:12.461853 25642 consensus_meta.cc:287] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.548s	user 0.011s	sys 0.001s
W20260709 12:24:12.466392 25588 consensus_meta.cc:287] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.755s	user 0.000s	sys 0.016s
I20260709 12:24:12.384651 25571 consensus_queue.cc:260] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 22.13, Last appended by leader: 13, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:12.461786 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:12.378331 25664 raft_consensus.cc:515] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:12.466838 18374 raft_consensus.cc:2468] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 21.
I20260709 12:24:12.496057 18367 raft_consensus.cc:2468] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 21.
I20260709 12:24:12.498832 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 21 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:12.500207 18374 raft_consensus.cc:2410] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 21 because replica has last-logged OpId of term: 19 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 14.
I20260709 12:24:12.501675 18374 raft_consensus.cc:1240] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 21. Current term is 22. Ops: []
I20260709 12:24:12.462108 25586 raft_consensus.cc:3055] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 17 LEADER]: Stepping down as leader of term 17
W20260709 12:24:12.468431 17654 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.570s	user 0.011s	sys 0.026s
W20260709 12:24:12.489749 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 202273us
W20260709 12:24:12.488900 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 228262us
I20260709 12:24:12.470204 17802 leader_election.cc:304] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
W20260709 12:24:12.506462 17654 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.608s	user 0.012s	sys 0.026s
I20260709 12:24:12.467288 25728 raft_consensus.cc:3060] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:12.508184 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:12.510306 17680 raft_consensus.cc:1275] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 23 index: 12. Preceding OpId from leader: term: 24 index: 13. (index mismatch)
W20260709 12:24:12.511961 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 614984us
I20260709 12:24:12.519382 17684 raft_consensus.cc:1240] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 19 LEADER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 18. Current term is 19. Ops: []
I20260709 12:24:12.486366 25354 raft_consensus.cc:493] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:12.523983 17838 raft_consensus.cc:3055] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:24:12.524443 17838 raft_consensus.cc:740] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 18 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:24:12.474247 25666 leader_election.cc:290] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:12.567426 25421 raft_consensus.cc:493] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:12.568087 25422 leader_election.cc:290] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 21 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:12.568210 25421 raft_consensus.cc:515] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:12.495476 25527 consensus_meta.cc:287] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.515s	user 0.010s	sys 0.000s
I20260709 12:24:12.569865 25725 leader_election.cc:290] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:12.494853 25669 leader_election.cc:290] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:12.529870 25354 raft_consensus.cc:515] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:12.570969 25721 leader_election.cc:290] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:12.572072 25721 consensus_queue.cc:1048] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 12:24:12.572687 25354 leader_election.cc:290] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:12.498251 25642 raft_consensus.cc:515] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:12.573503 25721 consensus_queue.cc:1059] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 18.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20260709 12:24:12.574294 25721 consensus_queue.cc:1243] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer 0da169ff82674dc1947055ce43de68a3 log is divergent from this leader: its last log entry 19.17 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 16
I20260709 12:24:12.575227 25642 leader_election.cc:290] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 19 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:12.576004 18365 raft_consensus.cc:1240] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 18. Current term is 19. Ops: []
I20260709 12:24:12.578576 25677 leader_election.cc:290] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:12.579452 18368 raft_consensus.cc:3060] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:12.522174 25464 raft_consensus.cc:2804] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:24:12.588227 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:12.505319 25588 raft_consensus.cc:515] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:12.504009 25388 raft_consensus.cc:515] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:12.542151 17831 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35042 (request call id 2939) took 1052 ms. Trace:
I20260709 12:24:12.602084 17683 raft_consensus.cc:1240] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 17. Current term is 18. Ops: []
I20260709 12:24:12.525296 17838 consensus_queue.cc:260] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.14, Last appended by leader: 14, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:12.602191 17831 rpcz_store.cc:276] 0709 12:24:11.490169 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:11.490630 (+   461us) service_pool.cc:224] Handling call
0709 12:24:11.490944 (+   314us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:11.917347 (+426403us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:12.541780 (+624433us) spinlock_profiling.cc:243] Waited 417 ms on lock 0x7b54004416d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:24:12.542048 (+   268us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":959216896}
I20260709 12:24:12.603469 17838 raft_consensus.cc:3060] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:12.604897 17837 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35040 (request call id 3194) took 1144 ms. Trace:
I20260709 12:24:12.606369 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 22 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:12.607218 17683 raft_consensus.cc:2468] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 21.
I20260709 12:24:12.609848 17685 raft_consensus.cc:1240] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 23. Current term is 24. Ops: []
W20260709 12:24:12.610015 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 247726us
I20260709 12:24:12.605377 17837 rpcz_store.cc:276] 0709 12:24:11.460077 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:11.460209 (+   132us) service_pool.cc:224] Handling call
0709 12:24:12.604697 (+1144488us) spinlock_profiling.cc:243] Waited 607 ms on lock 0x7b54003cc3d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:24:12.604818 (+   121us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1396101888}
I20260709 12:24:12.561249 17689 raft_consensus.cc:1275] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 16 index: 12. Preceding OpId from leader: term: 18 index: 13. (index mismatch)
I20260709 12:24:12.562881 18363 raft_consensus.cc:1240] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 18. Current term is 19. Ops: [18.14-18.14]
I20260709 12:24:12.489382 25133 consensus_queue.cc:1059] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 22.14, Next index: 15, Last known committed idx: 13, Time since last communication: 0.045s
I20260709 12:24:12.625727 17727 leader_election.cc:304] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:12.491899 25734 raft_consensus.cc:493] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:12.626598 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:12.626881 18368 raft_consensus.cc:2468] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 20.
I20260709 12:24:12.627341 17837 raft_consensus.cc:2468] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 21.
I20260709 12:24:12.504599 25586 raft_consensus.cc:740] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:24:12.466621 25131 raft_consensus.cc:515] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:12.581044 17730 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:12.635990 17730 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:12.593869 25151 raft_consensus.cc:493] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:12.583850 25512 raft_consensus.cc:493] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:12.587807 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:12.588074 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:12.643913 17834 raft_consensus.cc:2468] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 14.
I20260709 12:24:12.588604 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:12.588886 17801 leader_election.cc:304] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:12.589320 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 20 candidate_status { last_received { term: 16 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:24:12.589617 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 65050us
W20260709 12:24:12.590039 17655 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:12.589967 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:12.593181 17684 raft_consensus.cc:2468] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 25.
I20260709 12:24:12.597260 25677 raft_consensus.cc:493] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:12.592912 25464 raft_consensus.cc:493] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:12.600471 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 16 candidate_status { last_received { term: 15 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:12.596714 17654 leader_election.cc:304] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:24:12.615267 17689 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37396 (request call id 2983) took 1083 ms. Trace:
I20260709 12:24:12.661175 17689 rpcz_store.cc:276] 0709 12:24:11.531325 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:11.531456 (+   131us) service_pool.cc:224] Handling call
0709 12:24:12.530942 (+999486us) spinlock_profiling.cc:243] Waited 1 ms on lock 0x7b5800002a98. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b474fbb71 00007f7b474fba22 00007f7b504148d9 00007f7b4f56eee0 00007f7b4f6380ab 00007f7b4f638131 00007f7b4f5f9fc0 00007f7b4f5e4929 00007f7b4f5e43d2 00007f7b48b5bbd4
0709 12:24:12.560840 (+ 29898us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:12.614900 (+ 54060us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:12.615201 (+   301us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2306944}
I20260709 12:24:12.625938 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 22 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:24:12.661971 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 73165us
I20260709 12:24:12.565943 25664 leader_election.cc:290] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:24:12.628690 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:12.628552 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 111721us
W20260709 12:24:12.602578 25729 log.cc:927] Time spent T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.138s	user 0.000s	sys 0.001s
W20260709 12:24:12.664640 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 68015.3us
I20260709 12:24:12.665401 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:12.665666 17654 leader_election.cc:304] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:24:12.666630 17654 leader_election.cc:304] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:24:12.630404 17727 leader_election.cc:304] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:12.603399 25656 raft_consensus.cc:3060] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:12.633100 25422 raft_consensus.cc:2804] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:24:12.669416 17838 raft_consensus.cc:1275] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 18 index: 14. Preceding OpId from leader: term: 19 index: 14. (term mismatch)
I20260709 12:24:12.633246 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:12.633999 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:12.639379 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:12.640692 17763 raft_consensus.cc:1275] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 20 index: 12. Preceding OpId from leader: term: 22 index: 14. (index mismatch)
I20260709 12:24:12.677209 17838 pending_rounds.cc:85] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b: Aborting all ops after (but not including) 13
I20260709 12:24:12.680506 17838 pending_rounds.cc:107] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b: Aborting uncommitted NO_OP operation due to leader change: 18.14
I20260709 12:24:12.677510 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:12.678929 17761 raft_consensus.cc:3060] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:12.681911 17757 raft_consensus.cc:2468] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 21.
I20260709 12:24:12.645165 17688 raft_consensus.cc:2468] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 21.
I20260709 12:24:12.661785 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
W20260709 12:24:12.583213 17729 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:12.684062 17755 raft_consensus.cc:2393] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 23: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:24:12.662770 17760 raft_consensus.cc:1240] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 20. Current term is 21. Ops: []
I20260709 12:24:12.668155 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 22 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:12.660399 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 22 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:12.678653 17687 raft_consensus.cc:3060] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:12.666262 18363 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 19.
I20260709 12:24:12.677418 17727 leader_election.cc:304] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:12.680862 17838 raft_consensus.cc:2889] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:12.690256 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:12.692355 17757 raft_consensus.cc:1275] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 19 index: 11. Preceding OpId from leader: term: 25 index: 14. (index mismatch)
W20260709 12:24:12.694687 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:12.695725 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:12.659502 25151 raft_consensus.cc:515] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:12.701035 17832 raft_consensus.cc:2393] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 17: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:24:12.701184 17830 raft_consensus.cc:1275] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 16 index: 14. Preceding OpId from leader: term: 19 index: 16. (index mismatch)
I20260709 12:24:12.701526 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:12.702028 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 27 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:12.634415 25527 consensus_queue.cc:1048] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.049s
I20260709 12:24:12.703637 17761 raft_consensus.cc:2468] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 24.
W20260709 12:24:12.716611 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:12.654318 25512 raft_consensus.cc:3060] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:12.720933 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:12.722223 18363 raft_consensus.cc:2468] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 25.
I20260709 12:24:12.663352 25464 raft_consensus.cc:3060] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:12.680006 25575 raft_consensus.cc:2804] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:24:12.676378 25264 raft_consensus.cc:2749] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:24:12.731853 17760 raft_consensus.cc:1275] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 21 index: 14. (index mismatch)
I20260709 12:24:12.739077 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:12.664312 25574 raft_consensus.cc:493] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:12.679941 25622 raft_consensus.cc:2804] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:24:12.689432 25740 raft_consensus.cc:2804] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Leader pre-election won for term 22
W20260709 12:24:12.733806 17801 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:12.727157 25707 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 19 current term: 19 is higher.
I20260709 12:24:12.676700 25731 raft_consensus.cc:2804] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:24:12.633630 25734 raft_consensus.cc:515] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:12.663286 25677 raft_consensus.cc:515] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:24:12.747109 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 69813.4us
I20260709 12:24:12.756268 25657 raft_consensus.cc:2804] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:24:12.749123 25574 raft_consensus.cc:515] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:12.756924 25638 raft_consensus.cc:3060] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:12.757138 25421 leader_election.cc:290] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:12.685446 25315 consensus_queue.cc:1059] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 22.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.058s
I20260709 12:24:12.759836 25388 leader_election.cc:290] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:12.748237 25622 raft_consensus.cc:493] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
W20260709 12:24:12.761685 25743 log.cc:927] Time spent T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.061s	user 0.000s	sys 0.000s
I20260709 12:24:12.732508 25602 consensus_queue.cc:1059] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.032s
I20260709 12:24:12.743249 25586 consensus_queue.cc:260] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 17.14, Last appended by leader: 14, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:12.749276 25746 raft_consensus.cc:493] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:12.767122 25267 tablet_bootstrap.cc:492] T a5732e71b9d54f8786b8c511d69943bf P 334a4e1f59944544b3b3ff36616b57da: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=11 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20260709 12:24:12.769201 25588 leader_election.cc:290] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 24 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:12.732582 25735 consensus_queue.cc:1048] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.037s
I20260709 12:24:12.771330 17687 raft_consensus.cc:2468] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 21.
I20260709 12:24:12.777251 17830 raft_consensus.cc:1240] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 22. Current term is 23. Ops: []
I20260709 12:24:12.855082 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
W20260709 12:24:12.777891 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 256155us
I20260709 12:24:12.787966 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:24:12.856428 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:12.784440 25731 raft_consensus.cc:697] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 21 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:24:12.816242 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 20 candidate_status { last_received { term: 16 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:12.819048 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:12.820855 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:24:12.881575 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:12.881836 17837 raft_consensus.cc:2393] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 20: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:24:12.827303 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:12.882095 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:12.882555 17756 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 18.
I20260709 12:24:12.750924 25740 raft_consensus.cc:493] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:12.746659 25736 raft_consensus.cc:493] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:12.835450 17762 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38786 (request call id 3666) took 1161 ms. Trace:
I20260709 12:24:12.849628 25746 raft_consensus.cc:515] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:12.852977 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 27 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:12.895821 17762 rpcz_store.cc:276] 0709 12:24:11.674303 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:11.674513 (+   210us) service_pool.cc:224] Handling call
0709 12:24:11.728091 (+ 53578us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:24:11.746168 (+ 18077us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:11.746216 (+    48us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:24:11.771059 (+ 24843us) log.cc:844] Serialized 52 byte log entry
0709 12:24:12.300207 (+529148us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:24:12.300271 (+    64us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:12.300371 (+   100us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:12.835195 (+534824us) raft_consensus.cc:1685] finished
0709 12:24:12.835222 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:12.835386 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":161,"raft.run_cpu_time_us":120,"raft.run_wall_time_us":24186,"thread_start_us":422877,"threads_started":1,"wal-append.queue_time_us":581958}
I20260709 12:24:12.832054 25424 consensus_queue.cc:1048] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.053s
W20260709 12:24:12.879984 17802 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:12.902678 17802 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:12.903208 17802 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:12.838600 25622 raft_consensus.cc:3060] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:12.887645 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:12.889775 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:12.895534 25267 tablet_bootstrap.cc:492] T a5732e71b9d54f8786b8c511d69943bf P 334a4e1f59944544b3b3ff36616b57da: Bootstrap complete.
I20260709 12:24:12.894517 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:12.723491 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:12.898502 17763 raft_consensus.cc:2468] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 26.
I20260709 12:24:12.898105 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:24:12.891170 25206 consensus_meta.cc:287] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 1.132s	user 0.012s	sys 0.003s
I20260709 12:24:12.892334 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 26 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:12.908241 17688 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 20.
I20260709 12:24:12.914712 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 21 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:24:12.896129 25650 consensus_meta.cc:287] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.595s	user 0.008s	sys 0.003s
I20260709 12:24:12.913369 17755 raft_consensus.cc:1275] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 15 index: 11. Preceding OpId from leader: term: 19 index: 13. (index mismatch)
I20260709 12:24:12.915812 17681 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37402 (request call id 3393) took 1361 ms. Trace:
I20260709 12:24:12.914767 17761 raft_consensus.cc:2410] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for term 26 because replica has last-logged OpId of term: 24 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 12.
I20260709 12:24:12.916302 17681 rpcz_store.cc:276] 0709 12:24:11.554137 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:11.554272 (+   135us) service_pool.cc:224] Handling call
0709 12:24:12.914562 (+1360290us) spinlock_profiling.cc:243] Waited 997 us on lock 0x7b5800002a98. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b474fbb71 00007f7b474fbeee 000000000036eb39 00007f7b4f56f1e0 00007f7b4f6380c4 00007f7b4f638131 00007f7b4f5e60a0 00007f7b4f5e5984 00007f7b48b606d4 00007f7b48b60662
0709 12:24:12.915739 (+  1177us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2292992}
I20260709 12:24:12.935537 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:12.938074 17802 leader_election.cc:304] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:12.896589 25586 raft_consensus.cc:3060] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:12.941955 17837 raft_consensus.cc:1275] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 24 index: 13. Preceding OpId from leader: term: 25 index: 14. (index mismatch)
I20260709 12:24:12.941658 17681 raft_consensus.cc:1240] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 21. Current term is 22. Ops: [19.14-21.15]
I20260709 12:24:12.936290 17762 raft_consensus.cc:2468] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 25.
I20260709 12:24:12.898458 25384 leader_election.cc:290] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:12.895578 25740 raft_consensus.cc:3060] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:12.855126 25527 consensus_queue.cc:1048] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.058s
I20260709 12:24:12.958293 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 20 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:12.961388 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:24:12.963155 17801 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:12.907363 25745 raft_consensus.cc:493] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:12.964231 18367 raft_consensus.cc:1240] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 18 LEADER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 17. Current term is 18. Ops: []
I20260709 12:24:12.962368 17830 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 19.
I20260709 12:24:12.941399 25650 raft_consensus.cc:2749] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
W20260709 12:24:12.981557 17655 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:12.969877 18333 leader_election.cc:304] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:24:12.986717 17689 raft_consensus.cc:2468] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 24.
I20260709 12:24:12.988469 25151 leader_election.cc:290] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:12.974989 25745 raft_consensus.cc:515] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:12.990453 25723 raft_consensus.cc:3055] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:24:12.990952 25723 raft_consensus.cc:740] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 21 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:24:12.977288 25206 raft_consensus.cc:2749] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:24:12.977870 25267 ts_tablet_manager.cc:1403] T a5732e71b9d54f8786b8c511d69943bf P 334a4e1f59944544b3b3ff36616b57da: Time spent bootstrapping tablet: real 2.129s	user 0.068s	sys 0.044s
I20260709 12:24:12.993752 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 21 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:24:12.995577 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 277486us
I20260709 12:24:12.901566 25736 raft_consensus.cc:515] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:12.960086 25384 raft_consensus.cc:493] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:13.001492 17802 leader_election.cc:304] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:24:13.004709 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:13.005314 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:13.001473 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:13.006476 17835 raft_consensus.cc:2468] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 24.
I20260709 12:24:13.007781 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:13.021858 17654 leader_election.cc:304] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:24:13.031644 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:13.033324 18361 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59886 (request call id 2053) took 1407 ms. Trace:
W20260709 12:24:13.033442 17655 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:13.033902 18361 rpcz_store.cc:276] 0709 12:24:11.625889 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:11.701668 (+ 75779us) service_pool.cc:224] Handling call
0709 12:24:11.702067 (+   399us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:11.702411 (+   344us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:11.702439 (+    28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:12.563917 (+861478us) spinlock_profiling.cc:243] Waited 9.57 ms on lock 0x7b2c000123a8. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b48f19a11 00007f7b48f29372 00007f7b4a65ba01 00007f7b4a62043a 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2
0709 12:24:12.564264 (+   347us) log.cc:844] Serialized 26 byte log entry
0709 12:24:12.878059 (+313795us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:24:12.878365 (+   306us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:12.878468 (+   103us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:13.033005 (+154537us) raft_consensus.cc:1685] finished
0709 12:24:13.033027 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:13.033256 (+   229us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":316,"raft.run_cpu_time_us":64,"raft.run_wall_time_us":64,"spinlock_wait_cycles":22009216,"thread_start_us":313326,"threads_started":1,"wal-append.queue_time_us":427743}
I20260709 12:24:13.036412 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:13.038157 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:13.039534 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:13.039834 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:13.041244 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:13.042155 18361 raft_consensus.cc:2376] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 22. Re-sending same reply.
I20260709 12:24:13.045807 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 19 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:13.043346 17756 raft_consensus.cc:2393] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 19: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:24:13.047067 17652 leader_election.cc:304] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:24:13.008850 18333 leader_election.cc:304] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:24:12.993623 25388 consensus_queue.cc:1059] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 21.16, Next index: 17, Last known committed idx: 14, Time since last communication: 0.036s
I20260709 12:24:13.009289 25688 raft_consensus.cc:493] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:13.087173 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 24 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:13.013558 25206 raft_consensus.cc:493] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:13.090314 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:13.090747 17758 raft_consensus.cc:1275] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 25 index: 19. Preceding OpId from leader: term: 26 index: 20. (index mismatch)
I20260709 12:24:13.091820 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:13.119127 17684 raft_consensus.cc:2468] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 19.
I20260709 12:24:13.055207 18334 leader_election.cc:304] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:24:13.092913 17652 leader_election.cc:304] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:13.105775 18361 raft_consensus.cc:2410] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for term 24 because replica has last-logged OpId of term: 22 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 15.
I20260709 12:24:13.106323 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 19 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:13.124351 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 26 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:13.011951 25749 raft_consensus.cc:2804] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:24:13.119804 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 20 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:13.130471 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:13.140614 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:13.141379 17689 raft_consensus.cc:2468] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 19.
I20260709 12:24:13.140744 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:13.012513 25384 raft_consensus.cc:515] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:13.155553 17758 raft_consensus.cc:1240] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 22. Current term is 23. Ops: []
I20260709 12:24:13.084185 25422 raft_consensus.cc:697] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
W20260709 12:24:13.145722 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 321669us
I20260709 12:24:13.155970 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
W20260709 12:24:13.145324 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 90254.9us
I20260709 12:24:13.083242 25131 leader_election.cc:290] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:13.085534 25589 raft_consensus.cc:493] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:13.174054 18361 raft_consensus.cc:2393] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 24: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:24:13.174746 17727 leader_election.cc:304] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:13.016727 25744 raft_consensus.cc:2804] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:24:13.184693 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:13.184830 25744 raft_consensus.cc:697] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 24 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:24:13.186265 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:13.186976 17760 raft_consensus.cc:2468] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 19.
I20260709 12:24:13.186540 25744 consensus_queue.cc:237] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 23.14, Last appended by leader: 14, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:13.014364 25745 leader_election.cc:290] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:13.188629 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:13.189333 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 21 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:13.190114 17681 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 20.
I20260709 12:24:13.178606 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:24:13.181102 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:13.182677 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:13.192673 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:13.141320 25734 leader_election.cc:290] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:13.125582 25688 raft_consensus.cc:515] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:24:13.183908 25384 leader_election.cc:290] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:13.106370 25575 raft_consensus.cc:697] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 17 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:24:13.147954 25677 leader_election.cc:290] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:13.209964 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:13.211458 17729 leader_election.cc:304] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:13.211869 25736 leader_election.cc:290] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:13.212469 25745 raft_consensus.cc:493] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:13.212663 17830 raft_consensus.cc:2376] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 21. Re-sending same reply.
I20260709 12:24:13.214213 18367 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 2284) took 1318 ms. Trace:
I20260709 12:24:13.214922 18365 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59886 (request call id 2056) took 1588 ms. Trace:
I20260709 12:24:13.234781 18365 rpcz_store.cc:276] 0709 12:24:11.626579 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:11.705903 (+ 79324us) service_pool.cc:224] Handling call
0709 12:24:11.706147 (+   244us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:12.576480 (+870333us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:13.214721 (+638241us) spinlock_profiling.cc:243] Waited 870 ms on lock 0x7b54007701d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:24:13.214868 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1999946624}
I20260709 12:24:13.236701 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:13.237663 18365 raft_consensus.cc:2393] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 17: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:24:13.179956 25388 raft_consensus.cc:3055] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:24:13.216629 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:13.217869 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:13.003453 25723 consensus_queue.cc:260] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 21.15, Last appended by leader: 15, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:24:13.143258 25594 consensus_meta.cc:287] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.712s	user 0.007s	sys 0.004s
I20260709 12:24:13.219259 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:24:13.183089 25747 log.cc:927] Time spent T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.179s	user 0.000s	sys 0.001s
I20260709 12:24:13.221148 18361 raft_consensus.cc:2468] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 23.
I20260709 12:24:13.262993 17758 raft_consensus.cc:2468] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 21.
I20260709 12:24:13.266448 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:13.267251 18361 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 20.
I20260709 12:24:13.193872 25133 raft_consensus.cc:493] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:13.165243 17762 raft_consensus.cc:1240] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 23. Current term is 24. Ops: []
I20260709 12:24:13.227715 18334 leader_election.cc:304] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:24:13.139086 25206 raft_consensus.cc:515] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:13.136590 25749 raft_consensus.cc:493] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:13.234714 18367 rpcz_store.cc:276] 0709 12:24:11.895838 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:12.499299 (+603461us) service_pool.cc:224] Handling call
0709 12:24:12.499672 (+   373us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:12.967278 (+467606us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:13.213954 (+246676us) spinlock_profiling.cc:243] Waited 464 ms on lock 0x7b54006af9d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:24:13.214147 (+   193us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1067719936}
I20260709 12:24:13.159664 25657 raft_consensus.cc:697] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:24:13.249734 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:13.320636 18365 raft_consensus.cc:2393] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 24: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:24:13.173250 25664 raft_consensus.cc:2804] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Leader pre-election won for term 26
W20260709 12:24:13.234659 25728 consensus_meta.cc:287] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.704s	user 0.013s	sys 0.000s
I20260709 12:24:13.240762 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 23 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:13.269065 17654 leader_election.cc:304] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:24:13.268883 17756 raft_consensus.cc:2468] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 21.
I20260709 12:24:13.260761 25384 raft_consensus.cc:2804] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:24:13.321838 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 20 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:13.271821 25761 raft_consensus.cc:2804] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:24:13.230365 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:13.261361 25388 raft_consensus.cc:740] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 21 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:24:13.345855 25388 consensus_queue.cc:260] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 21.16, Last appended by leader: 16, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:13.347043 25388 raft_consensus.cc:3060] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Advancing to term 22
W20260709 12:24:13.347438 25512 consensus_meta.cc:287] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.606s	user 0.002s	sys 0.006s
W20260709 12:24:13.281620 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 53969.9us
I20260709 12:24:13.322041 25746 leader_election.cc:290] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:13.221393 16501 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:24:13.234443 25760 consensus_queue.cc:1048] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.106s
I20260709 12:24:13.286130 25666 raft_consensus.cc:2764] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Leader pre-election decision vote started in defunct term 19: won
I20260709 12:24:13.355628 25766 consensus_queue.cc:1048] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 14, Time since last communication: 0.041s
I20260709 12:24:13.335451 25131 raft_consensus.cc:2749] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:24:13.367831 25760 consensus_queue.cc:1059] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 17.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 12:24:13.368080 17757 raft_consensus.cc:3060] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Advancing to term 23
W20260709 12:24:13.358928 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 89975.3us
I20260709 12:24:13.360549 25771 raft_consensus.cc:2804] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:24:13.369304 25771 raft_consensus.cc:493] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:24:13.369755 25771 raft_consensus.cc:3060] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:13.263990 17835 raft_consensus.cc:2393] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 20: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:24:13.356869 25746 consensus_queue.cc:1048] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.016s
W20260709 12:24:13.317904 17801 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:13.326606 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:13.009619 25731 consensus_queue.cc:237] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 20.15, Last appended by leader: 15, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:13.263248 25677 raft_consensus.cc:2804] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:24:13.226356 25589 raft_consensus.cc:515] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:13.407510 25666 raft_consensus.cc:2804] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:24:13.336194 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 22 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:13.280007 25594 raft_consensus.cc:2749] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:24:13.338632 25384 raft_consensus.cc:493] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:13.338929 25761 raft_consensus.cc:493] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
W20260709 12:24:13.340404 25748 log.cc:927] Time spent T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.107s	user 0.000s	sys 0.000s
W20260709 12:24:13.341674 25638 consensus_meta.cc:287] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.509s	user 0.005s	sys 0.005s
I20260709 12:24:13.424489 25677 raft_consensus.cc:493] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:13.441598 18364 raft_consensus.cc:1275] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 19 index: 16. Preceding OpId from leader: term: 20 index: 17. (index mismatch)
I20260709 12:24:13.320025 25420 consensus_queue.cc:1048] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.024s
I20260709 12:24:13.342185 25575 consensus_queue.cc:237] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.11, Last appended by leader: 11, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:13.322835 25133 raft_consensus.cc:515] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:13.348070 17802 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:13.431463 25666 raft_consensus.cc:493] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:13.366112 25645 raft_consensus.cc:493] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:13.356418 25770 consensus_queue.cc:1059] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.012s
I20260709 12:24:13.270099 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:24:13.401944 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 393137us
I20260709 12:24:13.454290 18361 raft_consensus.cc:2393] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 20: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:24:13.454629 18333 leader_election.cc:304] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:24:13.456990 18333 leader_election.cc:304] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:13.436372 25761 raft_consensus.cc:3060] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:13.442207 25690 raft_consensus.cc:493] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:13.368217 17680 raft_consensus.cc:1240] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 19 LEADER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 18. Current term is 19. Ops: []
I20260709 12:24:13.367954 25131 raft_consensus.cc:2749] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 12:24:13.361455 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:13.461665 17689 raft_consensus.cc:2468] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 22.
W20260709 12:24:13.461797 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 250408us
I20260709 12:24:13.462937 17689 raft_consensus.cc:1275] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 19 index: 11. Preceding OpId from leader: term: 22 index: 13. (index mismatch)
I20260709 12:24:13.463250 17729 leader_election.cc:304] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:24:13.406070 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:24:13.352623 25755 log.cc:927] Time spent T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.122s	user 0.000s	sys 0.000s
W20260709 12:24:13.353212 25756 log.cc:927] Time spent T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.174s	user 0.000s	sys 0.000s
I20260709 12:24:13.347859 25512 raft_consensus.cc:515] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:13.354074 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 24 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:13.377586 17760 raft_consensus.cc:1275] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 20 index: 13. Preceding OpId from leader: term: 21 index: 14. (index mismatch)
I20260709 12:24:13.399015 18334 leader_election.cc:304] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:24:13.327728 25749 raft_consensus.cc:3060] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:13.325978 25664 raft_consensus.cc:493] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:13.323705 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:13.471796 18367 raft_consensus.cc:2376] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 20. Re-sending same reply.
I20260709 12:24:13.472913 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:13.460784 25594 raft_consensus.cc:493] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:13.355515 25206 leader_election.cc:290] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:13.411854 17757 raft_consensus.cc:2468] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 23.
I20260709 12:24:13.412484 25734 consensus_queue.cc:1048] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:24:13.414213 17801 leader_election.cc:304] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:24:13.466572 25746 raft_consensus.cc:2804] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Leader pre-election won for term 21
W20260709 12:24:13.471171 25464 consensus_meta.cc:287] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.728s	user 0.011s	sys 0.003s
I20260709 12:24:13.473809 25664 raft_consensus.cc:3060] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:13.410261 17682 raft_consensus.cc:2468] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 19.
W20260709 12:24:13.422654 25656 consensus_meta.cc:287] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.723s	user 0.007s	sys 0.006s
I20260709 12:24:13.306713 25745 raft_consensus.cc:515] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:13.423171 25740 raft_consensus.cc:515] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Starting leader election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:13.335022 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 20 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:13.467120 25690 raft_consensus.cc:515] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:13.430459 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 213397us
I20260709 12:24:13.372991 25728 raft_consensus.cc:515] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:13.363468 25622 raft_consensus.cc:515] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:13.427384 25315 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.576s	user 0.022s	sys 0.047s
I20260709 12:24:13.342060 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:13.443588 18363 raft_consensus.cc:3060] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Advancing to term 21
W20260709 12:24:13.500236 25315 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.658s	user 0.022s	sys 0.047s
I20260709 12:24:13.447372 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:13.350567 18368 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59902 (request call id 1866) took 1116 ms. Trace:
I20260709 12:24:13.532732 17685 raft_consensus.cc:1240] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 15. Current term is 16. Ops: []
I20260709 12:24:13.437029 25384 raft_consensus.cc:3060] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:13.534859 25512 leader_election.cc:290] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 23 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:13.366114 25657 consensus_queue.cc:237] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.13, Last appended by leader: 13, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:13.437218 25767 raft_consensus.cc:3055] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:24:13.540091 25767 raft_consensus.cc:740] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 17 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:24:13.540536 25133 leader_election.cc:290] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:13.541155 25133 raft_consensus.cc:2804] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:24:13.541556 25133 raft_consensus.cc:493] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:13.541014 25767 consensus_queue.cc:260] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.12, Last appended by leader: 12, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:13.541879 25133 raft_consensus.cc:3060] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:13.542197 25767 raft_consensus.cc:3060] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:24:13.452455 17802 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:13.545725 17802 leader_election.cc:304] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:24:13.443274 25677 raft_consensus.cc:3060] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:13.549598 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:13.550415 17681 raft_consensus.cc:2393] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 18: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:24:13.550530 25739 consensus_queue.cc:1059] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 18.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20260709 12:24:13.551306 25739 consensus_queue.cc:1243] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer 0da169ff82674dc1947055ce43de68a3 log is divergent from this leader: its last log entry 19.17 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 16
W20260709 12:24:13.403436 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 270186us
I20260709 12:24:13.464170 17761 raft_consensus.cc:2468] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 19.
I20260709 12:24:13.464812 17689 raft_consensus.cc:1275] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 17 index: 14. Preceding OpId from leader: term: 18 index: 15. (index mismatch)
I20260709 12:24:13.496266 25778 raft_consensus.cc:2804] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:24:13.461922 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:13.552553 17652 leader_election.cc:304] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
W20260709 12:24:13.553082 25758 log.cc:927] Time spent T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.088s	user 0.000s	sys 0.001s
I20260709 12:24:13.553550 17652 leader_election.cc:304] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:24:13.553917 25640 raft_consensus.cc:3055] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:24:13.554342 25640 raft_consensus.cc:740] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:24:13.460714 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 19 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:13.556207 18361 raft_consensus.cc:2468] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 18.
I20260709 12:24:13.557159 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 16 candidate_status { last_received { term: 15 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:13.558303 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:13.452412 25599 raft_consensus.cc:493] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:13.560171 25599 raft_consensus.cc:515] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:13.561671 25740 leader_election.cc:290] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 22 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:13.561587 25664 raft_consensus.cc:515] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:13.562232 25599 leader_election.cc:290] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:13.455462 25666 raft_consensus.cc:3060] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:13.467816 17729 leader_election.cc:304] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:13.469537 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:24:13.473076 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 260900us
I20260709 12:24:13.473960 18367 raft_consensus.cc:2393] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 19: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:24:13.565429 18367 raft_consensus.cc:1240] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 19. Current term is 20. Ops: [19.12-19.12]
I20260709 12:24:13.264710 25588 consensus_queue.cc:1059] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 17.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.142s
W20260709 12:24:13.481585 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 67519.8us
W20260709 12:24:13.416545 25750 log.cc:927] Time spent T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.228s	user 0.000s	sys 0.004s
W20260709 12:24:13.418619 25757 log.cc:927] Time spent T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.231s	user 0.000s	sys 0.001s
I20260709 12:24:13.457413 25645 raft_consensus.cc:515] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:13.485805 25681 raft_consensus.cc:2804] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:24:13.271855 25422 consensus_queue.cc:237] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 17.14, Last appended by leader: 14, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:13.496985 17756 raft_consensus.cc:1275] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 23 index: 12. Preceding OpId from leader: term: 24 index: 13. (index mismatch)
I20260709 12:24:13.482661 25746 raft_consensus.cc:493] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:13.459610 25267 raft_consensus.cc:359] T a5732e71b9d54f8786b8c511d69943bf P 334a4e1f59944544b3b3ff36616b57da [term 19 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:24:13.524616 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:13.525519 18371 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59886 (request call id 2049) took 1900 ms. Trace:
I20260709 12:24:13.525848 18366 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59886 (request call id 2051) took 1900 ms. Trace:
I20260709 12:24:13.473317 25768 consensus_queue.cc:1048] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.030s
I20260709 12:24:13.529013 17838 raft_consensus.cc:2393] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 22: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
W20260709 12:24:13.530262 25775 log.cc:927] Time spent T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.059s	user 0.000s	sys 0.000s
I20260709 12:24:13.530344 25656 raft_consensus.cc:2749] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:24:13.495276 25641 raft_consensus.cc:493] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:13.540122 25622 leader_election.cc:290] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 16 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:24:13.531296 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 158036us
I20260709 12:24:13.429352 25688 leader_election.cc:290] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:13.532835 18368 rpcz_store.cc:276] 0709 12:24:12.233996 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:12.431038 (+197042us) service_pool.cc:224] Handling call
0709 12:24:13.350360 (+919322us) spinlock_profiling.cc:243] Waited 102 ms on lock 0x7b540047de54. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:24:13.350499 (+   139us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":235642112}
I20260709 12:24:13.534272 17758 raft_consensus.cc:1275] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 20 index: 11. Preceding OpId from leader: term: 23 index: 13. (index mismatch)
I20260709 12:24:13.546187 25728 leader_election.cc:290] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 19 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:13.552387 25778 raft_consensus.cc:493] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:13.671150 17829 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35042 (request call id 2950) took 1154 ms. Trace:
I20260709 12:24:13.565335 17654 leader_election.cc:304] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:24:13.366055 25574 leader_election.cc:290] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:13.566504 18363 raft_consensus.cc:2468] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 21.
I20260709 12:24:13.567142 17762 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 20.
I20260709 12:24:13.569831 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 19 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:13.572132 17652 leader_election.cc:304] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:13.487663 25464 raft_consensus.cc:515] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:13.573834 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 106103us
I20260709 12:24:13.582590 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:13.643529 25681 raft_consensus.cc:493] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
W20260709 12:24:13.629666 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 172732us
I20260709 12:24:13.555183 25640 consensus_queue.cc:260] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 24.15, Last appended by leader: 15, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:13.632148 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:13.688577 17688 raft_consensus.cc:2468] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 18.
I20260709 12:24:13.609967 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
W20260709 12:24:13.590915 17730 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:13.533036 17728 leader_election.cc:304] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:24:13.636211 17655 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:13.631104 17801 leader_election.cc:304] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:24:13.691466 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 60519.4us
I20260709 12:24:13.692584 17801 leader_election.cc:304] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:13.694044 17801 leader_election.cc:304] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:13.551749 25739 raft_consensus.cc:2804] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:24:13.651293 18371 rpcz_store.cc:276] 0709 12:24:11.624794 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:11.628353 (+  3559us) service_pool.cc:224] Handling call
0709 12:24:11.628614 (+   261us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:12.223502 (+594888us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:13.525317 (+1301815us) spinlock_profiling.cc:243] Waited 594 ms on lock 0x7b5400305f54. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:24:13.525465 (+   148us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1366765952}
I20260709 12:24:13.652088 18366 rpcz_store.cc:276] 0709 12:24:11.625359 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:11.625480 (+   121us) service_pool.cc:224] Handling call
0709 12:24:11.663821 (+ 38341us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:11.673228 (+  9407us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:11.673274 (+    46us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:12.586172 (+912898us) spinlock_profiling.cc:243] Waited 23.3 ms on lock 0x7b2c000123a8. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b48f19a11 00007f7b48f29372 00007f7b4a65ba01 00007f7b4a62043a 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2
0709 12:24:12.586655 (+   483us) log.cc:844] Serialized 26 byte log entry
0709 12:24:12.914640 (+327985us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:24:12.914674 (+    34us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:12.914737 (+    63us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:13.498163 (+583426us) raft_consensus.cc:1685] finished
0709 12:24:13.498192 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:13.525803 (+ 27611us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":8976,"raft.queue_time_us":20479,"raft.run_cpu_time_us":118,"raft.run_wall_time_us":5485,"spinlock_wait_cycles":53671296,"thread_start_us":313971,"threads_started":1,"wal-append.queue_time_us":487323}
W20260709 12:24:13.577232 25772 log.cc:927] Time spent T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.121s	user 0.000s	sys 0.000s
I20260709 12:24:13.579617 25735 raft_consensus.cc:3055] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:24:13.480036 25594 raft_consensus.cc:515] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:13.629935 25740 raft_consensus.cc:493] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:13.576777 25770 raft_consensus.cc:2804] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:24:13.700080 25708 raft_consensus.cc:2749] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:24:13.574445 25777 raft_consensus.cc:2804] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:24:13.629629 25680 raft_consensus.cc:493] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:13.573807 18367 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59902 (request call id 1861) took 1371 ms. Trace:
I20260709 12:24:13.647292 25589 leader_election.cc:290] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:13.656637 25746 raft_consensus.cc:3060] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:13.696378 17833 raft_consensus.cc:1275] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 21 index: 13. Preceding OpId from leader: term: 22 index: 13. (term mismatch)
I20260709 12:24:13.702555 25620 raft_consensus.cc:2749] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:24:13.725649 17833 pending_rounds.cc:85] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b: Aborting all ops after (but not including) 12
I20260709 12:24:13.702215 25728 raft_consensus.cc:493] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:13.576609 25773 consensus_queue.cc:1048] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.101s
I20260709 12:24:13.701074 18371 raft_consensus.cc:1275] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 19 index: 13. (index mismatch)
I20260709 12:24:13.691615 17802 leader_election.cc:304] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:24:13.675938 25641 raft_consensus.cc:515] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:13.704612 17801 leader_election.cc:304] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:24:13.705178 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:13.728873 18371 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59902 (request call id 1873) took 1474 ms. Trace:
I20260709 12:24:13.679365 25151 raft_consensus.cc:2804] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:24:13.574288 25315 raft_consensus.cc:2749] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:24:13.702713 25783 raft_consensus.cc:2804] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:24:13.675946 25778 raft_consensus.cc:3060] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:13.686277 25681 raft_consensus.cc:3060] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:13.688772 25574 raft_consensus.cc:2749] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:24:13.700527 25688 raft_consensus.cc:2749] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:24:13.700304 25735 raft_consensus.cc:740] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:24:13.729333 18371 rpcz_store.cc:276] 0709 12:24:12.254687 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:13.695212 (+1440525us) service_pool.cc:224] Handling call
0709 12:24:13.695546 (+   334us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:13.728618 (+ 33072us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:13.728820 (+   202us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:13.748689 18371 raft_consensus.cc:1275] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 16 index: 12. Preceding OpId from leader: term: 18 index: 13. (index mismatch)
I20260709 12:24:13.749372 18371 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59886 (request call id 2058) took 1157 ms. Trace:
I20260709 12:24:13.711180 25740 raft_consensus.cc:515] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:13.749687 18371 rpcz_store.cc:276] 0709 12:24:12.591480 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:13.748027 (+1156547us) service_pool.cc:224] Handling call
0709 12:24:13.748367 (+   340us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:13.749068 (+   701us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:13.749317 (+   249us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:13.750974 18371 raft_consensus.cc:1240] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 25. Current term is 26. Ops: []
I20260709 12:24:13.751186 25464 leader_election.cc:290] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 22 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:13.751509 18371 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59886 (request call id 2060) took 1159 ms. Trace:
I20260709 12:24:13.758841 25787 raft_consensus.cc:493] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:13.760823 25787 raft_consensus.cc:515] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:13.716185 25777 raft_consensus.cc:493] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:13.730520 25151 raft_consensus.cc:493] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:13.761968 25777 raft_consensus.cc:3060] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:13.739711 25384 raft_consensus.cc:515] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:13.744352 25779 log.cc:927] Time spent T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.093s	user 0.001s	sys 0.000s
I20260709 12:24:13.736639 25512 raft_consensus.cc:2804] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:24:13.763723 25512 raft_consensus.cc:493] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:13.764154 25512 raft_consensus.cc:3060] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:13.759985 18371 rpcz_store.cc:276] 0709 12:24:12.591974 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:13.750500 (+1158526us) service_pool.cc:224] Handling call
0709 12:24:13.750792 (+   292us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:13.751302 (+   510us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:13.751460 (+   158us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:13.765589 18368 raft_consensus.cc:1240] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 20. Current term is 21. Ops: [20.16-20.16]
I20260709 12:24:13.709049 25599 consensus_queue.cc:1059] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 22.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.023s
I20260709 12:24:13.727591 25680 raft_consensus.cc:515] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:13.732576 25728 raft_consensus.cc:515] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:13.745128 25594 leader_election.cc:290] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:13.761379 18374 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59902 (request call id 1859) took 1559 ms. Trace:
I20260709 12:24:13.788981 18374 rpcz_store.cc:276] 0709 12:24:12.201751 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:12.516108 (+314357us) service_pool.cc:224] Handling call
0709 12:24:12.519585 (+  3477us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:12.533429 (+ 13844us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:12.585374 (+ 51945us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:12.585876 (+   502us) log.cc:844] Serialized 26 byte log entry
0709 12:24:12.911395 (+325519us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:24:12.947645 (+ 36250us) log.cc:844] Serialized 14 byte log entry
0709 12:24:12.953329 (+  5684us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:12.953508 (+   179us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:13.417720 (+464212us) spinlock_profiling.cc:243] Waited 41.8 ms on lock 0x7b4400022b4c. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b47459c85 00007f7b4745a6a2 00007f7b4a1cc1db 00007f7b4a1c7aaf 00007f7b4a1bb86e 00007f7b4a1babb6 00007f7b4a1ba7d8 00007f7b4a1c4221 00007f7b4a1c41d9 00007f7b4a1c4169
0709 12:24:13.761051 (+343331us) raft_consensus.cc:1685] finished
0709 12:24:13.761164 (+   113us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:13.761314 (+   150us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":6787,"prepare.queue_time_us":36017,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":47,"raft.queue_time_us":18776,"raft.run_cpu_time_us":192,"raft.run_wall_time_us":194,"spinlock_wait_cycles":96105728,"thread_start_us":325162,"threads_started":1,"wal-append.queue_time_us":601213}
I20260709 12:24:13.707831 25770 raft_consensus.cc:493] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:13.762058 25151 raft_consensus.cc:3060] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:13.764552 25745 leader_election.cc:290] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:13.734284 25267 raft_consensus.cc:740] T a5732e71b9d54f8786b8c511d69943bf P 334a4e1f59944544b3b3ff36616b57da [term 19 LEARNER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Initialized, Role: LEARNER
I20260709 12:24:13.751209 25740 leader_election.cc:290] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:13.746027 25645 leader_election.cc:290] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:13.748214 25735 consensus_queue.cc:260] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 22.17, Last appended by leader: 17, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:13.767050 25663 raft_consensus.cc:3055] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:24:13.685606 25785 consensus_queue.cc:1048] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.039s
I20260709 12:24:13.730587 25664 leader_election.cc:290] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 26 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:13.796419 25770 raft_consensus.cc:3060] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:13.677245 25736 consensus_queue.cc:1048] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.035s
I20260709 12:24:13.767120 25571 raft_consensus.cc:3060] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:13.767757 25787 leader_election.cc:290] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:13.804903 25663 raft_consensus.cc:740] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 22 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:24:13.808728 25728 leader_election.cc:290] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:13.801313 25674 consensus_queue.cc:260] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 18.16, Last appended by leader: 16, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:13.809176 25735 raft_consensus.cc:3060] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:13.806350 25681 raft_consensus.cc:515] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:13.736776 25783 raft_consensus.cc:493] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:13.822307 25663 consensus_queue.cc:260] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 22.16, Last appended by leader: 16, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:13.823081 25783 raft_consensus.cc:3060] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:13.823344 25681 leader_election.cc:290] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 25 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:13.823767 25663 raft_consensus.cc:3060] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:13.811004 25666 raft_consensus.cc:515] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:13.754801 25771 raft_consensus.cc:515] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:13.828238 25674 raft_consensus.cc:3060] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:13.828873 25666 leader_election.cc:290] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 22 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:13.830106 25771 leader_election.cc:290] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 24 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:13.759601 25677 raft_consensus.cc:515] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:13.832373 25746 raft_consensus.cc:515] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:13.830705 25680 leader_election.cc:290] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:13.833592 25641 leader_election.cc:290] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:13.834697 25746 leader_election.cc:290] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 21 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:13.834528 25749 raft_consensus.cc:515] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:13.838193 25677 leader_election.cc:290] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 22 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:13.841275 25723 raft_consensus.cc:3060] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:13.817628 25761 raft_consensus.cc:515] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:13.847489 25761 leader_election.cc:290] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 15 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:13.847849 25690 leader_election.cc:290] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:13.836038 25777 raft_consensus.cc:515] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:13.856909 25777 leader_election.cc:290] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 25 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:24:13.858031 25752 log.cc:927] Time spent T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.525s	user 0.001s	sys 0.003s
I20260709 12:24:13.860705 25778 raft_consensus.cc:515] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:13.862650 25763 raft_consensus.cc:3055] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:24:13.863319 25763 raft_consensus.cc:740] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:24:13.863351 25778 leader_election.cc:290] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 20 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:13.864925 25763 consensus_queue.cc:260] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 22.14, Last appended by leader: 14, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:13.866096 25763 raft_consensus.cc:3060] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:13.866390 25749 leader_election.cc:290] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 21 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:24:13.868363 25744 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.658s	user 0.022s	sys 0.006s
W20260709 12:24:13.868858 25744 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.658s	user 0.022s	sys 0.006s
W20260709 12:24:13.680828 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 109004us
W20260709 12:24:13.673728 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 108460us
W20260709 12:24:13.698380 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 165407us
W20260709 12:24:13.697924 17730 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:13.708665 17764 raft_consensus.cc:1240] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 LEADER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 18. Current term is 19. Ops: [18.14-18.14]
I20260709 12:24:13.673295 17829 rpcz_store.cc:276] 0709 12:24:12.516282 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:12.516418 (+   136us) service_pool.cc:224] Handling call
0709 12:24:12.516767 (+   349us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:12.517037 (+   270us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 12:24:12.854296 (+337259us) log.cc:844] Serialized 14 byte log entry
0709 12:24:13.548415 (+694119us) spinlock_profiling.cc:243] Waited 20.2 ms on lock 0x7b1800043f00. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b474adc0a 00007f7b50bed219 00007f7b47535a40 00007f7b475304e4 00007f7b47530455 00007f7b4e0183c7 00007f7b4a62a9a7 00007f7b4a63a0bf 00007f7b4a63a038
0709 12:24:13.670592 (+122177us) log.cc:844] Serialized 14 byte log entry
0709 12:24:13.670809 (+   217us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:13.670863 (+    54us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:24:13.670907 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:13.670975 (+    68us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:13.671090 (+   115us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":65018,"prepare.queue_time_us":306241,"prepare.run_cpu_time_us":95,"prepare.run_wall_time_us":13383,"raft.queue_time_us":6976,"raft.run_cpu_time_us":97,"raft.run_wall_time_us":3953,"spinlock_wait_cycles":46518528,"thread_start_us":836350,"threads_started":3,"wal-append.queue_time_us":577360}
I20260709 12:24:13.706290 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 21 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:13.726006 17833 pending_rounds.cc:107] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b: Aborting uncommitted NO_OP operation due to leader change: 21.13
I20260709 12:24:13.769512 18365 raft_consensus.cc:2410] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 20 because replica has last-logged OpId of term: 18 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 13.
I20260709 12:24:13.766031 18371 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59886 (request call id 2061) took 1173 ms. Trace:
I20260709 12:24:13.726505 18367 rpcz_store.cc:276] 0709 12:24:12.206510 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:13.564772 (+1358262us) service_pool.cc:224] Handling call
0709 12:24:13.565150 (+   378us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:13.573265 (+  8115us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:13.573708 (+   443us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:13.764140 25384 leader_election.cc:290] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 26 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:13.913733 17687 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37402 (request call id 3406) took 1056 ms. Trace:
I20260709 12:24:13.916115 17683 raft_consensus.cc:2393] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 22: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:24:13.914793 17681 raft_consensus.cc:1240] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 23. Current term is 24. Ops: []
I20260709 12:24:13.937980 17834 raft_consensus.cc:2410] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for term 21 because replica has last-logged OpId of term: 19 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 11.
I20260709 12:24:13.941807 17836 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35042 (request call id 2940) took 2449 ms. Trace:
I20260709 12:24:13.935081 17685 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37402 (request call id 3408) took 1045 ms. Trace:
I20260709 12:24:13.945907 18366 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 2286) took 1368 ms. Trace:
I20260709 12:24:13.957949 18363 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59886 (request call id 2062) took 1365 ms. Trace:
I20260709 12:24:13.961234 25264 consensus_queue.cc:1048] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.024s
I20260709 12:24:13.939630 17837 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35040 (request call id 3209) took 1088 ms. Trace:
I20260709 12:24:13.988127 17837 rpcz_store.cc:276] 0709 12:24:12.850524 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:12.886359 (+ 35835us) service_pool.cc:224] Handling call
0709 12:24:12.886739 (+   380us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:12.942653 (+ 55914us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:13.938866 (+996213us) spinlock_profiling.cc:243] Waited 54.8 ms on lock 0x7b54000abed4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:24:13.939319 (+   453us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":126102784}
I20260709 12:24:13.987936 25267 consensus_queue.cc:260] T a5732e71b9d54f8786b8c511d69943bf P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 19.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:24:13.982646 18363 rpcz_store.cc:276] 0709 12:24:12.592415 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:12.723173 (+130758us) service_pool.cc:224] Handling call
0709 12:24:13.957253 (+1234080us) spinlock_profiling.cc:243] Waited 531 ms on lock 0x7b54006aed54. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:24:13.957555 (+   302us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1220689280}
I20260709 12:24:13.993753 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:13.995219 18363 raft_consensus.cc:2393] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 26: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:24:13.961285 18362 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59886 (request call id 2052) took 2335 ms. Trace:
I20260709 12:24:13.955071 18368 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59902 (request call id 1862) took 1748 ms. Trace:
I20260709 12:24:13.944132 18371 rpcz_store.cc:276] 0709 12:24:12.592205 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:13.765233 (+1173028us) service_pool.cc:224] Handling call
0709 12:24:13.765532 (+   299us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:13.765705 (+   173us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:13.765741 (+    36us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:13.765786 (+    45us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:24:13.765822 (+    36us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:13.765874 (+    52us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:13.765980 (+   106us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:24:13.913890 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:13.921475 17652 leader_election.cc:304] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:14.004350 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:14.005203 18371 raft_consensus.cc:3060] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:13.927855 17727 leader_election.cc:304] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
W20260709 12:24:13.933723 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 248407us
I20260709 12:24:13.964449 25471 raft_consensus.cc:2749] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:24:13.982156 17685 rpcz_store.cc:276] 0709 12:24:12.889677 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:12.889808 (+   131us) service_pool.cc:224] Handling call
0709 12:24:12.900717 (+ 10909us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:13.544098 (+643381us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:13.934419 (+390321us) spinlock_profiling.cc:243] Waited 632 ms on lock 0x7b54004254d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:24:13.934839 (+   420us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1453057408}
I20260709 12:24:14.013118 25133 raft_consensus.cc:515] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:14.015254 18361 raft_consensus.cc:2393] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 20: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:24:14.016546 17803 leader_election.cc:304] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:13.936332 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 26 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:13.940488 17833 raft_consensus.cc:2889] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:24:14.024396 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 97060.2us
I20260709 12:24:13.997978 18368 rpcz_store.cc:276] 0709 12:24:12.206957 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:13.678664 (+1471707us) service_pool.cc:224] Handling call
0709 12:24:13.679026 (+   362us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:13.765989 (+ 86963us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:13.954252 (+188263us) spinlock_profiling.cc:243] Waited 86.3 ms on lock 0x7b540051a254. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:24:13.954808 (+   556us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":198383360}
I20260709 12:24:14.025755 25776 raft_consensus.cc:2749] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:24:13.996622 18362 rpcz_store.cc:276] 0709 12:24:11.625662 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:11.693921 (+ 68259us) service_pool.cc:224] Handling call
0709 12:24:11.694326 (+   405us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:11.694789 (+   463us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:11.694832 (+    43us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:11.702439 (+  7607us) spinlock_profiling.cc:243] Waited 5.5 ms on lock 0x7b4800220a58. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4f549aa1 00007f7b4f540b37 00007f7b4f5409ca 00007f7b4f63826f 00007f7b4f637f37 00007f7b4f64c55c 00007f7b4f64c508 00007f7b4f64c488 00007f7b4f64c443 00007f7b4f64b6ec 00007f7b4e575a96
0709 12:24:12.541258 (+838819us) spinlock_profiling.cc:243] Waited 10.6 ms on lock 0x7b2c000123a8. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b48f19a11 00007f7b48f29372 00007f7b4a65ba01 00007f7b4a62043a 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2
0709 12:24:12.563374 (+ 22116us) log.cc:844] Serialized 26 byte log entry
0709 12:24:12.900323 (+336949us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:24:12.900365 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:12.900429 (+    64us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:13.857463 (+957034us) spinlock_profiling.cc:243] Waited 2.07 ms on lock 0x7b4400022b4c. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b47459c85 00007f7b4745a6a2 00007f7b4a1cc1db 00007f7b4a1c7aaf 00007f7b4a1bb86e 00007f7b4a1babb6 00007f7b4a1ba7d8 00007f7b4a1c4221 00007f7b4a1c41d9 00007f7b4a1c4169
0709 12:24:13.960489 (+103026us) raft_consensus.cc:1685] finished
0709 12:24:13.960551 (+    62us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:13.960973 (+   422us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":594,"raft.run_cpu_time_us":1873,"raft.run_wall_time_us":7256,"spinlock_wait_cycles":41885184,"thread_start_us":336598,"threads_started":1,"wal-append.queue_time_us":676735}
W20260709 12:24:13.935657 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 230981us
I20260709 12:24:13.926925 17728 leader_election.cc:304] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:14.024230 17830 raft_consensus.cc:2410] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for term 26 because replica has last-logged OpId of term: 24 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 11.
I20260709 12:24:14.028324 25736 consensus_queue.cc:1059] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 18.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.005s
I20260709 12:24:14.033221 25778 raft_consensus.cc:493] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:14.033916 25739 raft_consensus.cc:493] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:14.034219 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:14.034086 25778 raft_consensus.cc:515] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:14.037739 25778 leader_election.cc:290] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:14.037803 25770 raft_consensus.cc:515] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:14.038837 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 16 candidate_status { last_received { term: 15 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:14.040107 25749 raft_consensus.cc:493] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:13.963294 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:13.948936 18374 raft_consensus.cc:1275] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 16 index: 15. Preceding OpId from leader: term: 19 index: 17. (index mismatch)
I20260709 12:24:14.034489 25739 raft_consensus.cc:3060] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:14.038872 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:14.040891 25749 raft_consensus.cc:515] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:14.025777 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:14.060608 17830 raft_consensus.cc:2468] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 23.
I20260709 12:24:14.062314 18367 raft_consensus.cc:3060] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:13.982574 18366 rpcz_store.cc:276] 0709 12:24:12.577562 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:13.695223 (+1117661us) service_pool.cc:224] Handling call
0709 12:24:13.706298 (+ 11075us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:13.710990 (+  4692us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:13.711038 (+    48us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:13.711518 (+   480us) log.cc:844] Serialized 26 byte log entry
0709 12:24:13.944974 (+233456us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:24:13.945087 (+   113us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:13.945243 (+   156us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:13.945322 (+    79us) raft_consensus.cc:1685] finished
0709 12:24:13.945336 (+    14us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:13.945588 (+   252us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4297,"raft.queue_time_us":5212,"raft.run_cpu_time_us":248,"raft.run_wall_time_us":14566,"thread_start_us":232773,"threads_started":1,"wal-append.queue_time_us":112600}
I20260709 12:24:13.981510 17687 rpcz_store.cc:276] 0709 12:24:12.857049 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:12.857246 (+   197us) service_pool.cc:224] Handling call
0709 12:24:12.857687 (+   441us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:13.147904 (+290217us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:13.147938 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:13.149147 (+  1209us) log.cc:844] Serialized 26 byte log entry
0709 12:24:13.436168 (+287021us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:24:13.436212 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:13.436289 (+    77us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:13.912668 (+476379us) raft_consensus.cc:1685] finished
0709 12:24:13.912763 (+    95us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:13.913284 (+   521us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":248628,"raft.run_cpu_time_us":201,"raft.run_wall_time_us":17098,"thread_start_us":541559,"threads_started":2,"wal-append.queue_time_us":324397}
I20260709 12:24:14.073439 18374 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 2290) took 1042 ms. Trace:
I20260709 12:24:14.080506 25637 raft_consensus.cc:3060] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:14.082494 17830 raft_consensus.cc:1240] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 22. Current term is 23. Ops: []
W20260709 12:24:14.084508 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 68392.2us
I20260709 12:24:14.085050 17838 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37994 (request call id 1488) took 1266 ms. Trace:
I20260709 12:24:14.085834 17803 leader_election.cc:304] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:14.087131 18365 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59902 (request call id 1884) took 1124 ms. Trace:
I20260709 12:24:14.087697 18365 rpcz_store.cc:276] 0709 12:24:12.962474 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:13.321548 (+359074us) service_pool.cc:224] Handling call
0709 12:24:14.086643 (+765095us) spinlock_profiling.cc:243] Waited 434 ms on lock 0x7b540051aed4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:24:14.086886 (+   243us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":998971648}
W20260709 12:24:14.095486 17802 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:14.098459 25640 raft_consensus.cc:3060] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Advancing to term 25
W20260709 12:24:14.099710 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 178855us
W20260709 12:24:14.101086 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 174672us
I20260709 12:24:14.126605 25749 leader_election.cc:290] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:14.102208 25527 raft_consensus.cc:2804] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:24:14.104678 25464 raft_consensus.cc:2749] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:24:13.982095 17836 rpcz_store.cc:276] 0709 12:24:11.491734 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:11.491901 (+   167us) service_pool.cc:224] Handling call
0709 12:24:11.492201 (+   300us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:11.598947 (+106746us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 12:24:11.818388 (+219441us) log.cc:844] Serialized 14 byte log entry
0709 12:24:12.045551 (+227163us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:12.045622 (+    71us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:24:12.045691 (+    69us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:12.283155 (+237464us) spinlock_profiling.cc:243] Waited 13.4 ms on lock 0x7b1800395e00. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b475012a6 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b475333f3 00007f7b475357c1
0709 12:24:12.284260 (+  1105us) spinlock_profiling.cc:243] Waited 4.97 ms on lock 0x7b1800395e00. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b475333f3 00007f7b475357c1 00007f7b47535779 00007f7b47535709 00007f7b475356d1 00007f7b475349cd 0000000000388634 0000000000388479 00007f7b4751683a
0709 12:24:13.940966 (+1656706us) spinlock_profiling.cc:243] Waited 107 ms on lock 0x7b5400267354. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:24:13.941166 (+   200us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:13.941558 (+   392us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":14127,"prepare.queue_time_us":460325,"prepare.run_cpu_time_us":57,"prepare.run_wall_time_us":59,"raft.queue_time_us":830,"raft.run_cpu_time_us":162,"raft.run_wall_time_us":9114,"spinlock_wait_cycles":287360768,"thread_start_us":412918,"threads_started":2,"wal-append.queue_time_us":479378,"wal-append.run_cpu_time_us":2874,"wal-append.run_wall_time_us":1138085}
I20260709 12:24:14.128472 17687 raft_consensus.cc:1240] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 14. Current term is 15. Ops: []
I20260709 12:24:14.106060 17755 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38786 (request call id 3670) took 1169 ms. Trace:
I20260709 12:24:14.085634 17838 rpcz_store.cc:276] 0709 12:24:12.818402 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:12.818674 (+   272us) service_pool.cc:224] Handling call
0709 12:24:14.084510 (+1265836us) spinlock_profiling.cc:243] Waited 647 ms on lock 0x7b540008a7d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:24:14.084802 (+   292us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1489164032}
I20260709 12:24:13.873097 25783 raft_consensus.cc:515] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:14.115923 18363 raft_consensus.cc:1275] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 20 index: 15. Preceding OpId from leader: term: 21 index: 14. (term mismatch)
I20260709 12:24:14.120854 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:14.121444 25770 leader_election.cc:290] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 20 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:14.122391 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 18 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:14.124068 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:14.124338 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:14.127158 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:14.076623 18374 rpcz_store.cc:276] 0709 12:24:13.030779 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:13.791738 (+760959us) service_pool.cc:224] Handling call
0709 12:24:13.793343 (+  1605us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:14.053669 (+260326us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:14.073366 (+ 19697us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:14.130146 18363 pending_rounds.cc:85] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da: Aborting all ops after (but not including) 13
I20260709 12:24:14.135082 18371 raft_consensus.cc:2468] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 24.
I20260709 12:24:14.136022 18363 pending_rounds.cc:107] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da: Aborting uncommitted NO_OP operation due to leader change: 18.14
I20260709 12:24:14.143718 17652 leader_election.cc:304] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:14.145874 25777 raft_consensus.cc:493] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:14.146111 25783 leader_election.cc:290] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 22 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:14.147161 18368 raft_consensus.cc:1240] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 20. Current term is 21. Ops: [20.17-20.17]
I20260709 12:24:14.146965 25777 raft_consensus.cc:515] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:14.148186 18368 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 2293) took 1069 ms. Trace:
I20260709 12:24:14.149241 25735 raft_consensus.cc:2804] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:24:14.150143 25735 raft_consensus.cc:697] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 24 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:24:14.148723 25133 leader_election.cc:290] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 20 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:14.145872 25761 consensus_queue.cc:1048] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 11, Time since last communication: 0.007s
I20260709 12:24:14.153257 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:14.154690 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 21 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:14.156256 17686 raft_consensus.cc:2468] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 24.
I20260709 12:24:14.158566 17728 leader_election.cc:304] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:14.160120 17689 raft_consensus.cc:2468] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 21.
I20260709 12:24:14.160315 17687 raft_consensus.cc:1275] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 23 index: 14. Preceding OpId from leader: term: 24 index: 15. (index mismatch)
I20260709 12:24:14.129848 17755 rpcz_store.cc:276] 0709 12:24:12.936250 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:12.936710 (+   460us) service_pool.cc:224] Handling call
0709 12:24:14.105676 (+1168966us) spinlock_profiling.cc:243] Waited 1.09 s on lock 0x7b540042c054. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:24:14.105857 (+   181us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2515936000}
I20260709 12:24:14.161075 17652 leader_election.cc:304] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:14.162439 25677 consensus_queue.cc:1048] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:24:14.162966 25354 raft_consensus.cc:2749] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 12:24:14.163478 25677 consensus_queue.cc:1243] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer e2909cff1da44ad387f96a8ae3aa5afb log is divergent from this leader: its last log entry 15.12 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 11
I20260709 12:24:14.164664 18374 raft_consensus.cc:1240] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 18. Current term is 19. Ops: []
I20260709 12:24:14.148701 18368 rpcz_store.cc:276] 0709 12:24:13.079073 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:14.075119 (+996046us) service_pool.cc:224] Handling call
0709 12:24:14.146611 (+ 71492us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:14.147788 (+  1177us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:14.148085 (+   297us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:14.166420 18374 raft_consensus.cc:1240] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 18. Current term is 19. Ops: [18.16-18.16]
W20260709 12:24:14.168069 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 67272.4us
I20260709 12:24:14.169631 17801 leader_election.cc:304] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:14.169759 25744 consensus_queue.cc:1048] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:24:14.170811 18366 raft_consensus.cc:1240] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 22. Current term is 23. Ops: [22.17-22.17]
I20260709 12:24:14.140893 18363 raft_consensus.cc:2889] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:24:14.172832 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 68981.8us
I20260709 12:24:14.175526 25424 raft_consensus.cc:2804] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:24:14.176110 25424 raft_consensus.cc:493] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:14.176486 25424 raft_consensus.cc:3060] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:14.177466 25657 consensus_queue.cc:1048] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:24:14.180109 25575 raft_consensus.cc:2749] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:24:14.164561 25735 consensus_queue.cc:237] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 22.16, Last appended by leader: 16, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:14.181483 25588 raft_consensus.cc:3055] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:24:14.181957 25588 raft_consensus.cc:740] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 22 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:24:14.185158 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 21 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:14.173493 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:14.193252 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:14.194633 17680 raft_consensus.cc:2468] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 23.
I20260709 12:24:14.195093 17757 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38786 (request call id 3675) took 1022 ms. Trace:
I20260709 12:24:14.196118 25744 raft_consensus.cc:2749] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:24:14.195796 17757 rpcz_store.cc:276] 0709 12:24:13.172475 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:13.172777 (+   302us) service_pool.cc:224] Handling call
0709 12:24:14.194645 (+1021868us) spinlock_profiling.cc:243] Waited 32.9 ms on lock 0x7b54003bd3d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:24:14.194860 (+   215us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":75624832}
I20260709 12:24:14.197422 25527 raft_consensus.cc:697] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:24:14.198391 18334 leader_election.cc:304] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:24:14.200428 17801 leader_election.cc:304] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:14.201133 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:24:14.203529 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:14.204085 17759 raft_consensus.cc:1240] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 19 LEADER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 18. Current term is 19. Ops: [18.14-18.14]
I20260709 12:24:14.204950 25674 consensus_queue.cc:1048] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:24:14.164433 25264 consensus_queue.cc:1048] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.043s
I20260709 12:24:14.207626 25690 consensus_queue.cc:1059] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 14.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:24:14.208024 25471 raft_consensus.cc:493] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:14.208876 25471 raft_consensus.cc:515] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:14.209818 25485 raft_consensus.cc:3055] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:24:14.210273 25485 raft_consensus.cc:740] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:24:14.211519 25464 raft_consensus.cc:2804] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:24:14.211973 25464 raft_consensus.cc:493] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:14.212348 25267 ts_tablet_manager.cc:1434] T a5732e71b9d54f8786b8c511d69943bf P 334a4e1f59944544b3b3ff36616b57da: Time spent starting tablet: real 1.219s	user 0.012s	sys 0.006s
W20260709 12:24:14.212764 25267 ts_tablet_manager.cc:1467] T a5732e71b9d54f8786b8c511d69943bf P 334a4e1f59944544b3b3ff36616b57da: Tablet startup took 1219ms
I20260709 12:24:14.212700 25760 consensus_queue.cc:1048] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.001s
W20260709 12:24:14.213543 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:14.215083 18334 leader_election.cc:304] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:24:14.201730 18363 pending_rounds.cc:107] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da: Aborting uncommitted NO_OP operation due to leader change: 20.15
I20260709 12:24:14.202903 25783 consensus_queue.cc:1059] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 22.18, Next index: 19, Last known committed idx: 17, Time since last communication: 0.000s
I20260709 12:24:14.201429 25133 raft_consensus.cc:2804] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:24:14.216264 18363 raft_consensus.cc:2889] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:14.216547 25133 raft_consensus.cc:493] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:14.217105 25133 raft_consensus.cc:3060] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:14.211408 25485 consensus_queue.cc:260] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 14.12, Last appended by leader: 12, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:14.217324 18363 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 2291) took 1153 ms. Trace:
I20260709 12:24:14.219856 18363 rpcz_store.cc:276] 0709 12:24:13.064464 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:14.059969 (+995505us) service_pool.cc:224] Handling call
0709 12:24:14.115254 (+ 55285us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:14.217054 (+101800us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:14.217270 (+   216us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:14.188109 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
W20260709 12:24:14.202343 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 58622.8us
I20260709 12:24:14.136655 17832 raft_consensus.cc:2468] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 23.
I20260709 12:24:14.221249 17685 raft_consensus.cc:3060] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Advancing to term 22
W20260709 12:24:14.222247 17802 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:14.222350 18367 raft_consensus.cc:2468] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 22.
I20260709 12:24:14.137923 25749 raft_consensus.cc:493] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:14.240957 17801 leader_election.cc:304] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:14.241117 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:14.246555 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:14.212359 25464 raft_consensus.cc:3060] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Advancing to term 21
W20260709 12:24:14.249810 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 108109us
I20260709 12:24:14.252126 18362 raft_consensus.cc:3060] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:14.250806 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 26 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:14.183343 25588 consensus_queue.cc:260] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 22.14, Last appended by leader: 14, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:14.231832 25777 leader_election.cc:290] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:14.284591 17681 raft_consensus.cc:1275] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 21 index: 15. Preceding OpId from leader: term: 23 index: 16. (index mismatch)
I20260709 12:24:14.217603 25783 raft_consensus.cc:3055] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:24:14.198859 25527 consensus_queue.cc:237] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 19.15, Last appended by leader: 15, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:14.289364 17682 raft_consensus.cc:2393] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 25: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:24:14.232920 25681 raft_consensus.cc:493] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:14.293463 17687 raft_consensus.cc:3060] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:14.298482 25384 raft_consensus.cc:493] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:14.299196 25384 raft_consensus.cc:515] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:24:14.300002 25807 log.cc:927] Time spent T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.190s	user 0.004s	sys 0.000s
I20260709 12:24:14.300364 25588 raft_consensus.cc:3060] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:14.301472 25749 raft_consensus.cc:515] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:14.303947 17831 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35040 (request call id 3207) took 1381 ms. Trace:
I20260709 12:24:14.304569 17831 rpcz_store.cc:276] 0709 12:24:12.826876 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:12.827014 (+   138us) service_pool.cc:224] Handling call
0709 12:24:12.827490 (+   476us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:14.205852 (+1378362us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:14.205933 (+    81us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:14.206017 (+    84us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:24:14.206062 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:14.207526 (+  1464us) spinlock_profiling.cc:243] Waited 1.38 s on lock 0x7b54000ab4d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:24:14.207587 (+    61us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:14.207804 (+   217us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":6750,"raft.run_cpu_time_us":131,"raft.run_wall_time_us":132,"spinlock_wait_cycles":3168887168}
W20260709 12:24:14.310810 25808 log.cc:927] Time spent T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.058s	user 0.000s	sys 0.000s
W20260709 12:24:14.312183 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 226537us
W20260709 12:24:14.312860 25151 consensus_meta.cc:287] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.510s	user 0.014s	sys 0.000s
I20260709 12:24:14.313443 17803 leader_election.cc:304] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:14.313539 25151 raft_consensus.cc:515] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:14.315352 17803 leader_election.cc:304] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:14.320137 25744 raft_consensus.cc:2749] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:24:14.322042 25424 raft_consensus.cc:515] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:14.328419 17830 raft_consensus.cc:1275] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 17 index: 13. Preceding OpId from leader: term: 19 index: 14. (index mismatch)
I20260709 12:24:14.287854 25778 raft_consensus.cc:2804] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Leader pre-election won for term 21
W20260709 12:24:14.330904 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 86354.6us
I20260709 12:24:14.242383 25674 consensus_queue.cc:1048] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.053s
W20260709 12:24:14.295234 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 134327us
I20260709 12:24:14.249670 25264 consensus_queue.cc:1059] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 25.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.092s
W20260709 12:24:14.350464 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 145190us
I20260709 12:24:14.293459 25571 raft_consensus.cc:2804] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:24:14.351503 25571 raft_consensus.cc:697] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 22 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:24:14.351536 17801 leader_election.cc:304] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:24:14.352913 25571 consensus_queue.cc:237] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 21.16, Last appended by leader: 16, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:14.357709 25620 consensus_queue.cc:1059] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 15.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:24:14.359402 25620 raft_consensus.cc:2804] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Leader election won for term 23
W20260709 12:24:14.213315 25267 ts_tablet_manager.cc:1469] T a5732e71b9d54f8786b8c511d69943bf P 334a4e1f59944544b3b3ff36616b57da: Trace:
0709 12:23:59.593495 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:59.677239 (+ 83744us) service_pool.cc:224] Handling call
0709 12:24:00.225938 (+548699us) ts_tablet_manager.cc:934] T a5732e71b9d54f8786b8c511d69943bf P 334a4e1f59944544b3b3ff36616b57da: Initiating tablet copy from peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
0709 12:24:04.728899 (+4502961us) tablet_copy_client.cc:368] Tablet copy session begun
0709 12:24:05.088731 (+359832us) tablet_metadata.cc:665] Metadata flushed
0709 12:24:05.088889 (+   158us) tablet_copy_client.cc:497] Wrote new tablet metadata
0709 12:24:05.387608 (+298719us) tablet_copy_client.cc:418] Wrote new consensus metadata
0709 12:24:05.544502 (+156894us) tablet_replica.cc:206] Creating consensus instance
0709 12:24:05.649774 (+105272us) ts_tablet_manager.cc:989] Replica registered.
0709 12:24:05.649947 (+   173us) inbound_call.cc:177] Queueing success response
0709 12:24:10.848236 (+5198289us) ts_tablet_manager.cc:1379] Bootstrapping tablet
0709 12:24:12.993007 (+2144771us) ts_tablet_manager.cc:1435] Starting tablet replica
0709 12:24:13.262101 (+269094us) tablet_replica.cc:261] Starting instrumentation
0709 12:24:13.357315 (+ 95214us) tablet_replica.cc:280] Starting consensus
Metrics: {"dns_us":22933,"mutex_wait_us":15368,"raft.queue_time_us":396,"raft.run_cpu_time_us":386,"raft.run_wall_time_us":388,"spinlock_wait_cycles":1280,"tablet-copy.queue_time_us":189801,"tablet-download-pool.queue_time_us":73950,"tablet-download-pool.run_cpu_time_us":1311,"tablet-download-pool.run_wall_time_us":3522351,"thread_start_us":381423,"threads_started":2}
I20260709 12:24:14.350546 25264 raft_consensus.cc:3055] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:24:14.377717 17654 leader_election.cc:304] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:24:14.377969 18364 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 2271) took 2698 ms. Trace:
I20260709 12:24:14.331570 25485 raft_consensus.cc:3060] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:24:14.378366 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 124662us
I20260709 12:24:14.332784 25778 raft_consensus.cc:493] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:14.360291 25740 raft_consensus.cc:3055] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:24:14.396966 25740 raft_consensus.cc:740] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 15 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:24:14.398414 25740 consensus_queue.cc:260] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 15.16, Last appended by leader: 16, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:14.399977 25778 raft_consensus.cc:3060] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:14.400914 25768 raft_consensus.cc:493] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:24:14.401522 25768 raft_consensus.cc:515] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:14.402985 25575 consensus_queue.cc:1059] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 18.14, Next index: 15, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:24:14.404004 25575 consensus_queue.cc:1059] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 19.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20260709 12:24:14.344368 25681 raft_consensus.cc:515] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:14.405577 25575 consensus_queue.cc:1048] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:24:14.408576 25761 raft_consensus.cc:3055] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:24:14.408953 25761 raft_consensus.cc:740] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:24:14.409154 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:14.410022 18361 raft_consensus.cc:1275] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 22 index: 16. Preceding OpId from leader: term: 23 index: 16. (term mismatch)
I20260709 12:24:14.410418 18361 pending_rounds.cc:85] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da: Aborting all ops after (but not including) 15
I20260709 12:24:14.410705 18361 pending_rounds.cc:107] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da: Aborting uncommitted NO_OP operation due to leader change: 22.16
I20260709 12:24:14.410962 18361 raft_consensus.cc:2889] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:14.410002 18371 raft_consensus.cc:2393] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 23: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:24:14.412145 25688 consensus_queue.cc:1048] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
W20260709 12:24:14.413025 17801 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:14.349473 17652 leader_election.cc:304] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:14.379987 25680 raft_consensus.cc:493] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:14.351580 25471 leader_election.cc:290] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:14.415973 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:14.371551 25264 raft_consensus.cc:740] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 25 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:24:14.416819 25731 raft_consensus.cc:2804] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:24:14.334450 25151 leader_election.cc:290] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 19 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:14.417625 25264 consensus_queue.cc:260] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 25.17, Last appended by leader: 17, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:14.417861 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 23 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:14.419018 17728 leader_election.cc:304] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:14.390255 18364 rpcz_store.cc:276] 0709 12:24:11.634844 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:11.871068 (+236224us) service_pool.cc:224] Handling call
0709 12:24:11.871447 (+   379us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:13.441990 (+1570543us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:14.250151 (+808161us) spinlock_profiling.cc:243] Waited 1.57 s on lock 0x7b5400552654. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:24:14.333074 (+ 82923us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3610149760}
I20260709 12:24:14.411986 25677 consensus_queue.cc:1048] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.008s
I20260709 12:24:14.420365 25784 raft_consensus.cc:493] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:14.409978 25761 consensus_queue.cc:260] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 20.17, Last appended by leader: 17, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:14.415902 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 66615.7us
I20260709 12:24:14.292905 25783 raft_consensus.cc:740] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:24:14.417724 18334 leader_election.cc:304] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:24:14.419689 25264 raft_consensus.cc:3060] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:14.415786 25424 leader_election.cc:290] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 20 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:14.456686 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:14.457497 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 22 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:14.458211 17681 raft_consensus.cc:3060] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:14.409207 25384 leader_election.cc:290] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:14.425978 25770 raft_consensus.cc:2749] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:24:14.380848 25744 raft_consensus.cc:493] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:14.420921 25575 raft_consensus.cc:3055] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:24:14.470526 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:14.426151 25731 raft_consensus.cc:697] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 24 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
W20260709 12:24:14.503985 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 85088.7us
I20260709 12:24:14.505344 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 21 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:14.477800 17688 raft_consensus.cc:1240] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 18. Current term is 19. Ops: [18.13-18.13]
I20260709 12:24:14.434803 25749 leader_election.cc:290] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:24:14.519471 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 70543.8us
W20260709 12:24:14.525280 17801 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:14.533828 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:14.527364 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:14.534858 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:14.525156 17685 raft_consensus.cc:2468] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 22.
W20260709 12:24:14.517445 17832 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35040 (request call id 3205) took 1780 ms (client timeout 1852 ms). Trace:
I20260709 12:24:14.416405 25680 raft_consensus.cc:515] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:14.532678 17684 raft_consensus.cc:1240] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 24. Current term is 25. Ops: []
I20260709 12:24:14.546026 17687 raft_consensus.cc:2468] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 20.
I20260709 12:24:14.538298 17652 leader_election.cc:304] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
W20260709 12:24:14.536907 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 119268us
W20260709 12:24:14.537572 17832 rpcz_store.cc:269] 0709 12:24:12.737196 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:12.738776 (+  1580us) service_pool.cc:224] Handling call
0709 12:24:14.472108 (+1733332us) spinlock_profiling.cc:243] Waited 1.36 s on lock 0x7b540008c5d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:24:14.517363 (+ 45255us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3130837760}
I20260709 12:24:14.502403 25620 raft_consensus.cc:697] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 23 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:24:14.539620 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 19 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:14.522864 25740 raft_consensus.cc:3060] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:14.558661 18367 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 20.
I20260709 12:24:14.534373 25384 raft_consensus.cc:493] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:14.523204 25575 raft_consensus.cc:740] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:24:14.559933 18361 raft_consensus.cc:2468] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 21.
I20260709 12:24:14.551699 25749 raft_consensus.cc:2804] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:24:14.566637 17759 raft_consensus.cc:2468] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 18.
I20260709 12:24:14.566540 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:14.549403 25688 raft_consensus.cc:2749] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:24:14.566665 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:14.604785 18364 raft_consensus.cc:3060] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:14.640863 25422 consensus_queue.cc:1059] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 16.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.054s
I20260709 12:24:14.566749 25663 raft_consensus.cc:2749] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:24:14.524714 25739 raft_consensus.cc:515] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Starting leader election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
W20260709 12:24:14.647436 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 332148us
I20260709 12:24:14.563186 25761 raft_consensus.cc:3060] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:14.651197 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 23 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:24:14.654390 18333 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:14.655921 25723 consensus_queue.cc:1059] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 23.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.236s
W20260709 12:24:14.657084 25512 consensus_meta.cc:287] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.753s	user 0.004s	sys 0.017s
I20260709 12:24:14.658238 25133 raft_consensus.cc:515] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:14.658699 17760 raft_consensus.cc:1240] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 22. Current term is 23. Ops: []
I20260709 12:24:14.659430 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:14.660532 17829 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35042 (request call id 2957) took 1027 ms. Trace:
I20260709 12:24:14.741053 18367 raft_consensus.cc:2393] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 20: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:24:14.527590 25744 raft_consensus.cc:515] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:14.641731 17757 raft_consensus.cc:3060] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:14.640978 25721 raft_consensus.cc:493] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:14.756294 25721 raft_consensus.cc:515] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:14.757815 25814 log.cc:927] Time spent T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.057s	user 0.000s	sys 0.001s
I20260709 12:24:14.642212 25781 raft_consensus.cc:3055] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:24:14.758589 25781 raft_consensus.cc:740] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:24:14.758788 25133 leader_election.cc:290] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 25 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:14.759758 25781 consensus_queue.cc:260] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:14.743947 25640 consensus_meta.cc:287] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.617s	user 0.013s	sys 0.000s
I20260709 12:24:14.761018 25781 raft_consensus.cc:3060] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:14.566476 25388 consensus_queue.cc:1048] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.191s
W20260709 12:24:14.756914 25637 consensus_meta.cc:287] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.676s	user 0.013s	sys 0.001s
I20260709 12:24:14.590355 25749 raft_consensus.cc:493] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:14.780292 17757 raft_consensus.cc:2468] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 22.
I20260709 12:24:14.589251 25599 raft_consensus.cc:2749] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:24:14.551249 25784 raft_consensus.cc:515] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:14.736758 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 230645us
I20260709 12:24:14.794718 17831 raft_consensus.cc:1240] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 22. Current term is 23. Ops: []
I20260709 12:24:14.648363 25464 raft_consensus.cc:515] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:14.742885 18333 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:14.721992 25422 raft_consensus.cc:3055] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:24:14.740795 17829 rpcz_store.cc:276] 0709 12:24:13.633053 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:13.937420 (+304367us) service_pool.cc:224] Handling call
0709 12:24:13.938334 (+   914us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:13.939468 (+  1134us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 12:24:14.036840 (+ 97372us) log.cc:844] Serialized 14 byte log entry
0709 12:24:14.067488 (+ 30648us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:14.219456 (+151968us) log.cc:844] Serialized 26 byte log entry
0709 12:24:14.227475 (+  8019us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:24:14.233888 (+  6413us) log.cc:844] Serialized 14 byte log entry
0709 12:24:14.236157 (+  2269us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:14.236231 (+    74us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:14.300026 (+ 63795us) spinlock_profiling.cc:243] Waited 3.1 ms on lock 0x7b4800190158. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4f549aa1 00007f7b4f540b37 00007f7b4f5409ca 00007f7b4f63826f 00007f7b4f637f37 00007f7b4f64c55c 00007f7b4f64c508 00007f7b4f64c488 00007f7b4f64c443 00007f7b4f64b6ec 00007f7b4e575a96
0709 12:24:14.660079 (+360053us) raft_consensus.cc:1685] finished
0709 12:24:14.660119 (+    40us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:14.660307 (+   188us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":12983,"prepare.queue_time_us":71156,"prepare.run_cpu_time_us":153,"prepare.run_wall_time_us":3850,"raft.queue_time_us":33403,"raft.run_cpu_time_us":2233,"raft.run_wall_time_us":91112,"spinlock_wait_cycles":7132032,"thread_start_us":30105,"threads_started":1,"wal-append.queue_time_us":137406}
I20260709 12:24:14.752704 17763 raft_consensus.cc:2468] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 25.
I20260709 12:24:14.752585 25512 raft_consensus.cc:515] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:14.764680 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 213743us
I20260709 12:24:14.559669 25151 raft_consensus.cc:2804] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:24:14.743520 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 23 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:14.641779 25763 consensus_queue.cc:1048] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.155s
I20260709 12:24:14.671003 25384 raft_consensus.cc:515] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:14.743470 25688 raft_consensus.cc:3055] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:24:14.812160 17833 raft_consensus.cc:1240] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 22. Current term is 23. Ops: [22.13-22.13]
W20260709 12:24:14.818107 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:14.823253 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:14.787294 25749 raft_consensus.cc:3060] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:24:14.827535 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:14.829519 17830 raft_consensus.cc:2393] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 23: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:24:14.844096 17834 raft_consensus.cc:1240] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 18. Current term is 19. Ops: []
I20260709 12:24:14.846582 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 18 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:14.798393 25776 raft_consensus.cc:493] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:24:14.818151 25151 raft_consensus.cc:493] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:24:14.808984 25422 raft_consensus.cc:740] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:24:14.853045 17652 leader_election.cc:304] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:14.712445 25783 consensus_queue.cc:260] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 22.18, Last appended by leader: 18, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:14.844553 18333 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.16.29.66:39651 timed out after 1.847s (SENT) [suppressed 12 similar messages]
I20260709 12:24:14.856515 17681 raft_consensus.cc:2468] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 22.
I20260709 12:24:14.826249 25688 raft_consensus.cc:740] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 23 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:24:14.859404 25638 raft_consensus.cc:493] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:14.864202 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:14.828836 25620 consensus_queue.cc:237] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 21.13, Last appended by leader: 13, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:14.865790 25512 leader_election.cc:290] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 19 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:14.866374 25784 leader_election.cc:290] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:14.867113 25681 leader_election.cc:290] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:14.852550 25131 raft_consensus.cc:493] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:24:14.857352 25151 raft_consensus.cc:3060] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:14.869303 17829 raft_consensus.cc:2468] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 17.
I20260709 12:24:14.873653 25384 leader_election.cc:290] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:14.874080 25776 raft_consensus.cc:515] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:14.875765 25763 raft_consensus.cc:2749] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:24:14.876075 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 23 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:14.877406 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 23 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:14.877997 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:14.880539 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:14.882777 25464 leader_election.cc:290] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 21 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:14.884804 17729 leader_election.cc:304] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:14.884923 25739 leader_election.cc:290] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 23 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:14.889710 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:14.891922 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 20 candidate_status { last_received { term: 18 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:14.895360 18334 leader_election.cc:304] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:24:14.897058 17654 leader_election.cc:304] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
W20260709 12:24:14.954193 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 57555.9us
I20260709 12:24:14.908427 17764 raft_consensus.cc:2468] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 25.
I20260709 12:24:14.745529 25575 consensus_queue.cc:260] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:14.909610 18362 consensus_meta.cc:287] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.653s	user 0.012s	sys 0.000s
I20260709 12:24:14.855193 25760 raft_consensus.cc:493] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:14.952198 17756 raft_consensus.cc:3060] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Advancing to term 19
W20260709 12:24:14.905216 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 125404us
I20260709 12:24:14.785876 25731 consensus_queue.cc:237] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 23.16, Last appended by leader: 16, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:14.969017 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 89191.5us
W20260709 12:24:14.976687 25820 log.cc:927] Time spent T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.130s	user 0.000s	sys 0.001s
I20260709 12:24:14.976323 25131 raft_consensus.cc:515] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:14.977053 25637 raft_consensus.cc:2804] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:24:14.977694 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:14.980046 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:14.985512 18362 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 20.
I20260709 12:24:15.009672 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:15.016115 17763 raft_consensus.cc:1240] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 24. Current term is 25. Ops: [24.15-24.15]
I20260709 12:24:14.961177 25677 raft_consensus.cc:2764] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Leader pre-election decision vote started in defunct term 21: won
I20260709 12:24:15.014670 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:24:15.022410 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 59599us
I20260709 12:24:15.023814 17803 leader_election.cc:304] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:14.953246 25744 leader_election.cc:290] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:15.028825 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
W20260709 12:24:14.980612 25778 consensus_meta.cc:287] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.575s	user 0.010s	sys 0.004s
W20260709 12:24:15.051739 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 207191us
I20260709 12:24:15.014338 25736 raft_consensus.cc:3060] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:15.053066 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:24:15.068972 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 184241us
I20260709 12:24:15.072505 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
W20260709 12:24:15.053875 25588 consensus_meta.cc:287] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.753s	user 0.013s	sys 0.001s
I20260709 12:24:15.075310 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:15.077198 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:15.018249 25680 leader_election.cc:290] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:14.912612 25638 raft_consensus.cc:515] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:15.080204 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:15.078490 17729 leader_election.cc:304] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:24:15.083750 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:15.084422 17835 raft_consensus.cc:2468] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 19.
I20260709 12:24:15.085210 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:15.085372 25688 consensus_queue.cc:260] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 23.16, Last appended by leader: 16, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:15.086954 25133 raft_consensus.cc:493] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:24:15.087664 25133 raft_consensus.cc:515] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:15.089087 25131 leader_election.cc:290] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:24:15.089654 25809 log.cc:927] Time spent T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.394s	user 0.000s	sys 0.002s
W20260709 12:24:15.091635 25828 log.cc:927] Time spent T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.144s	user 0.001s	sys 0.000s
I20260709 12:24:15.092131 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 22 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:15.099951 17837 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35042 (request call id 2960) took 1432 ms. Trace:
I20260709 12:24:15.100425 17837 rpcz_store.cc:276] 0709 12:24:13.667426 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:13.990106 (+322680us) service_pool.cc:224] Handling call
0709 12:24:13.990941 (+   835us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:14.040648 (+ 49707us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 12:24:14.041101 (+   453us) log.cc:844] Serialized 14 byte log entry
0709 12:24:14.235853 (+194752us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:14.256785 (+ 20932us) log.cc:844] Serialized 26 byte log entry
0709 12:24:14.256959 (+   174us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:24:14.257000 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:14.257091 (+    91us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:15.099661 (+842570us) raft_consensus.cc:1685] finished
0709 12:24:15.099686 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:15.099856 (+   170us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":14639,"prepare.queue_time_us":149,"prepare.run_cpu_time_us":61,"prepare.run_wall_time_us":62,"raft.queue_time_us":15237,"raft.run_cpu_time_us":300,"raft.run_wall_time_us":3417,"thread_start_us":179585,"threads_started":1,"wal-append.queue_time_us":349503}
I20260709 12:24:15.104079 25767 consensus_queue.cc:1048] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:24:15.098156 18374 raft_consensus.cc:2410] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 22 because replica has last-logged OpId of term: 21 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 15.
W20260709 12:24:15.087671 17730 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:15.107080 17730 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:15.103821 25422 consensus_queue.cc:260] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.13, Last appended by leader: 13, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:15.055255 25637 raft_consensus.cc:697] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 20 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:24:15.061582 25739 raft_consensus.cc:2749] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:24:15.120513 17755 raft_consensus.cc:2368] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 21. Current term is 22.
I20260709 12:24:15.130503 18364 raft_consensus.cc:2468] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 19.
I20260709 12:24:15.130883 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:15.105412 17684 raft_consensus.cc:2468] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 23.
I20260709 12:24:15.133734 17652 leader_election.cc:304] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:15.141686 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 23 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:15.142590 18368 raft_consensus.cc:3060] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:15.144531 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:15.150012 18362 raft_consensus.cc:1275] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 20 index: 15. Preceding OpId from leader: term: 21 index: 16. (index mismatch)
I20260709 12:24:15.153406 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:15.048938 25778 raft_consensus.cc:515] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:15.162436 17688 raft_consensus.cc:3060] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Advancing to term 21
W20260709 12:24:15.163524 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 149441us
I20260709 12:24:15.034174 25760 raft_consensus.cc:515] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:15.129495 25575 raft_consensus.cc:3060] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Advancing to term 19
W20260709 12:24:15.169510 17728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:15.170008 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:15.172079 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 23 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:15.172567 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:15.200932 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
W20260709 12:24:15.203059 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 53291.8us
W20260709 12:24:15.202246 17728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:15.209830 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:15.212517 18374 raft_consensus.cc:2393] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 22: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:24:15.216040 17755 raft_consensus.cc:2393] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 20: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:24:15.209879 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:15.214361 17728 leader_election.cc:304] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:15.221465 25784 consensus_queue.cc:1059] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 24.16, Next index: 17, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:24:15.222179 25422 raft_consensus.cc:3060] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:15.222640 25784 raft_consensus.cc:3055] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:24:15.223049 25784 raft_consensus.cc:740] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 24 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:24:15.223487 25776 leader_election.cc:290] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:24:15.224050 17655 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:15.221838 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 143436us
I20260709 12:24:15.225626 17764 raft_consensus.cc:2468] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 22.
I20260709 12:24:15.217000 17837 raft_consensus.cc:2468] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 19.
I20260709 12:24:15.217971 18362 raft_consensus.cc:3060] T a5732e71b9d54f8786b8c511d69943bf P 334a4e1f59944544b3b3ff36616b57da [term 19 LEARNER]: Advancing to term 22
I20260709 12:24:15.172019 25677 raft_consensus.cc:2749] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:24:15.221946 25354 raft_consensus.cc:493] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:15.249516 17803 leader_election.cc:304] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:15.232156 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:15.250604 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 22 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:15.251284 18364 raft_consensus.cc:3060] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:15.256588 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 21 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:15.211684 25133 leader_election.cc:290] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:15.268945 17835 raft_consensus.cc:2410] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for term 21 because replica has last-logged OpId of term: 20 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 16.
I20260709 12:24:15.275004 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:15.276789 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:15.277279 18371 raft_consensus.cc:3060] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Advancing to term 22
W20260709 12:24:15.280452 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 66165.6us
I20260709 12:24:15.280609 17756 raft_consensus.cc:2468] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 19.
I20260709 12:24:15.284327 18364 raft_consensus.cc:2468] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 22.
I20260709 12:24:15.286067 18364 raft_consensus.cc:1240] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 18. Current term is 19. Ops: [18.13-18.13]
I20260709 12:24:15.235175 25131 raft_consensus.cc:2804] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:24:15.288477 25131 raft_consensus.cc:493] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:15.288743 25767 raft_consensus.cc:2804] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:24:15.288856 25131 raft_consensus.cc:3060] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:15.289440 25767 raft_consensus.cc:697] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:24:15.290251 17680 raft_consensus.cc:2468] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 22.
I20260709 12:24:15.290450 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:15.290766 25767 consensus_queue.cc:237] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 20.16, Last appended by leader: 16, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:15.291436 25824 raft_consensus.cc:493] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:15.292066 25824 raft_consensus.cc:515] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:15.292728 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
W20260709 12:24:15.294319 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 58960.7us
I20260709 12:24:15.294926 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:15.295954 17837 raft_consensus.cc:3060] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Advancing to term 20
W20260709 12:24:15.328346 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 103062us
I20260709 12:24:15.301741 18361 raft_consensus.cc:2468] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 25.
I20260709 12:24:15.342562 18374 raft_consensus.cc:1240] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 21. Current term is 22. Ops: []
I20260709 12:24:15.359820 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 21 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:15.336388 25674 raft_consensus.cc:2749] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
W20260709 12:24:15.377585 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 256014us
W20260709 12:24:15.377743 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 81055.6us
I20260709 12:24:15.395648 18362 raft_consensus.cc:1275] T a5732e71b9d54f8786b8c511d69943bf P 334a4e1f59944544b3b3ff36616b57da [term 22 LEARNER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 19 index: 12. Preceding OpId from leader: term: 22 index: 13. (index mismatch)
I20260709 12:24:15.402853 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:24:15.406855 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:15.407402 18374 raft_consensus.cc:1275] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 18 index: 14. Preceding OpId from leader: term: 19 index: 14. (term mismatch)
I20260709 12:24:15.407826 18374 pending_rounds.cc:85] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da: Aborting all ops after (but not including) 13
I20260709 12:24:15.408098 18374 pending_rounds.cc:107] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da: Aborting uncommitted NO_OP operation due to leader change: 18.14
I20260709 12:24:15.408358 18374 raft_consensus.cc:2889] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:15.276338 25784 consensus_queue.cc:260] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 24.16, Last appended by leader: 16, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:15.416157 17652 leader_election.cc:304] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:15.338624 25354 raft_consensus.cc:515] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:15.365074 25637 consensus_queue.cc:237] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.13, Last appended by leader: 13, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:15.399400 25657 raft_consensus.cc:493] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:15.432324 18364 raft_consensus.cc:1240] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 19. Current term is 20. Ops: []
I20260709 12:24:15.436142 17835 raft_consensus.cc:2468] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 21.
W20260709 12:24:15.431809 25420 consensus_queue.cc:1175] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "e063541542a94969898868bf6eb5ce1b" responder_term: 24 status { last_received { term: 24 index: 16 } last_committed_idx: 14 last_received_current_leader { term: 24 index: 16 } }
I20260709 12:24:15.450024 25354 leader_election.cc:290] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:15.450810 25677 raft_consensus.cc:2764] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Leader pre-election decision vote started in defunct term 19: won
I20260709 12:24:15.456701 25776 raft_consensus.cc:493] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:15.457036 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:15.457517 25776 raft_consensus.cc:515] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:15.458739 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 223500us
I20260709 12:24:15.456696 18362 raft_consensus.cc:1240] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 18. Current term is 19. Ops: []
I20260709 12:24:15.473129 18371 raft_consensus.cc:2468] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 22.
I20260709 12:24:15.475317 18362 raft_consensus.cc:1240] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 23. Current term is 24. Ops: []
W20260709 12:24:15.477499 25151 consensus_meta.cc:287] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.517s	user 0.010s	sys 0.000s
I20260709 12:24:15.480654 25315 consensus_queue.cc:1048] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.142s
W20260709 12:24:15.539384 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 133278us
I20260709 12:24:15.553498 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 25 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:15.554697 18362 raft_consensus.cc:2410] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for term 25 because replica has last-logged OpId of term: 23 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 16.
I20260709 12:24:15.564666 25760 leader_election.cc:290] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:24:15.566291 25845 log.cc:927] Time spent T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.077s	user 0.000s	sys 0.000s
W20260709 12:24:15.607556 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:15.630403 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:15.631529 18371 raft_consensus.cc:2368] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for earlier term 19. Current term is 20.
W20260709 12:24:15.658043 17728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:15.535939 25824 leader_election.cc:290] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:24:15.622918 25736 consensus_meta.cc:287] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.539s	user 0.011s	sys 0.000s
W20260709 12:24:15.666481 25781 consensus_meta.cc:287] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.905s	user 0.009s	sys 0.002s
W20260709 12:24:15.666591 25836 log.cc:927] Time spent T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.241s	user 0.000s	sys 0.001s
I20260709 12:24:15.478006 25151 raft_consensus.cc:515] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:15.537047 25826 raft_consensus.cc:493] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
W20260709 12:24:15.674531 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 45430.7us
W20260709 12:24:15.675271 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 270046us
W20260709 12:24:15.688452 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 236642us
I20260709 12:24:15.458954 25657 raft_consensus.cc:515] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:15.669056 25674 raft_consensus.cc:493] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:15.701583 17686 raft_consensus.cc:1240] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 26. Current term is 27. Ops: []
I20260709 12:24:15.702633 17801 leader_election.cc:304] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:24:15.668856 25620 raft_consensus.cc:493] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
W20260709 12:24:15.708122 18368 consensus_meta.cc:287] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.565s	user 0.011s	sys 0.010s
I20260709 12:24:15.708671 18368 raft_consensus.cc:2468] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 23.
I20260709 12:24:15.701288 17803 leader_election.cc:304] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:24:15.699798 25749 consensus_meta.cc:287] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.808s	user 0.010s	sys 0.003s
I20260709 12:24:15.678503 25736 raft_consensus.cc:2749] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
W20260709 12:24:15.732659 25740 consensus_meta.cc:287] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 1.168s	user 0.013s	sys 0.001s
W20260709 12:24:15.753765 17727 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:15.689232 25826 raft_consensus.cc:515] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:24:15.761454 17727 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:15.766124 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 67480us
I20260709 12:24:15.775928 17758 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38786 (request call id 3694) took 1155 ms. Trace:
I20260709 12:24:15.776263 25721 leader_election.cc:290] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:15.722221 25620 raft_consensus.cc:515] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:15.778779 17685 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.501s	user 0.022s	sys 0.063s
I20260709 12:24:15.779493 17757 raft_consensus.cc:2410] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for term 23 because replica has last-logged OpId of term: 22 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 16.
I20260709 12:24:15.776615 17758 rpcz_store.cc:276] 0709 12:24:14.662166 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:14.761976 (+ 99810us) service_pool.cc:224] Handling call
0709 12:24:14.859808 (+ 97832us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:14.868152 (+  8344us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:14.868188 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:14.868739 (+   551us) log.cc:844] Serialized 26 byte log entry
0709 12:24:15.002573 (+133834us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:24:15.002628 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:15.002716 (+    88us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:15.757914 (+755198us) raft_consensus.cc:1685] finished
0709 12:24:15.757937 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:15.758085 (+   148us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1157,"raft.queue_time_us":2768,"raft.run_cpu_time_us":139,"raft.run_wall_time_us":139,"thread_start_us":133366,"threads_started":1,"wal-append.queue_time_us":281029}
I20260709 12:24:15.735940 25749 raft_consensus.cc:515] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:15.736284 25422 consensus_queue.cc:1059] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 22.18, Next index: 19, Last known committed idx: 17, Time since last communication: 0.035s
I20260709 12:24:15.756721 25739 raft_consensus.cc:2804] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:24:15.735785 25760 consensus_queue.cc:1059] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true }, Status: INVALID_TERM, Last received: 19.12, Next index: 13, Last known committed idx: 11, Time since last communication: 0.035s
I20260709 12:24:15.737187 25674 raft_consensus.cc:515] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:15.795387 25638 leader_election.cc:290] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:15.780315 25781 raft_consensus.cc:493] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:15.796245 17832 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37994 (request call id 1491) took 1234 ms. Trace:
I20260709 12:24:15.808388 17832 rpcz_store.cc:276] 0709 12:24:14.561819 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:14.566840 (+  5021us) service_pool.cc:224] Handling call
0709 12:24:14.701342 (+134502us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:24:14.780654 (+ 79312us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:14.780687 (+    33us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:24:14.794321 (+ 13634us) log.cc:844] Serialized 52 byte log entry
0709 12:24:15.018072 (+223751us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:24:15.060872 (+ 42800us) log.cc:844] Serialized 14 byte log entry
0709 12:24:15.079583 (+ 18711us) log.cc:844] Serialized 14 byte log entry
0709 12:24:15.102962 (+ 23379us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:15.103041 (+    79us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:15.795955 (+692914us) raft_consensus.cc:1685] finished
0709 12:24:15.795977 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:15.796165 (+   188us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":27398,"prepare.queue_time_us":54004,"prepare.run_cpu_time_us":64,"prepare.run_wall_time_us":70,"raft.queue_time_us":93171,"raft.run_cpu_time_us":312,"raft.run_wall_time_us":28948,"thread_start_us":154750,"threads_started":1,"wal-append.queue_time_us":903719}
W20260709 12:24:15.782986 25761 consensus_meta.cc:287] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 1.038s	user 0.013s	sys 0.002s
I20260709 12:24:15.784291 25740 raft_consensus.cc:3060] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:15.818979 25826 leader_election.cc:290] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:15.822279 18364 raft_consensus.cc:1240] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 23. Current term is 24. Ops: [23.16-23.16]
W20260709 12:24:15.823021 17837 consensus_meta.cc:287] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.527s	user 0.011s	sys 0.002s
W20260709 12:24:15.825794 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 127698us
I20260709 12:24:15.828634 17837 raft_consensus.cc:2468] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 20.
I20260709 12:24:15.790516 25739 raft_consensus.cc:493] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:15.831578 25739 raft_consensus.cc:3060] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:24:15.835543 25833 log.cc:927] Time spent T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.066s	user 0.001s	sys 0.000s
I20260709 12:24:15.841248 18363 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 2309) took 1326 ms. Trace:
I20260709 12:24:15.843319 17801 leader_election.cc:304] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:24:15.837075 25784 raft_consensus.cc:3060] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:15.822278 25620 leader_election.cc:290] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:24:15.839956 18367 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.543s	user 0.069s	sys 0.066s
W20260709 12:24:15.848702 18367 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.551s	user 0.069s	sys 0.066s
W20260709 12:24:15.829936 25485 consensus_meta.cc:287] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 1.395s	user 0.014s	sys 0.001s
I20260709 12:24:15.841652 18363 rpcz_store.cc:276] 0709 12:24:14.511112 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:14.511263 (+   151us) service_pool.cc:224] Handling call
0709 12:24:14.511609 (+   346us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:14.556655 (+ 45046us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:14.556691 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:14.569904 (+ 13213us) log.cc:844] Serialized 26 byte log entry
0709 12:24:14.840840 (+270936us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:24:14.840888 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:14.841141 (+   253us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:15.837277 (+996136us) raft_consensus.cc:1685] finished
0709 12:24:15.837304 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:15.837464 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3129,"raft.queue_time_us":223261,"raft.run_cpu_time_us":106,"raft.run_wall_time_us":107,"thread_start_us":121090,"threads_started":1,"wal-append.queue_time_us":1084601}
I20260709 12:24:15.857702 25674 leader_election.cc:290] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:15.857491 17801 leader_election.cc:304] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:24:15.855813 25776 leader_election.cc:290] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:15.864372 25620 raft_consensus.cc:2804] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:24:15.866931 25620 raft_consensus.cc:493] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:15.867312 25620 raft_consensus.cc:3060] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:15.871224 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:15.871349 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:15.825279 25781 raft_consensus.cc:515] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:15.840154 25736 raft_consensus.cc:3055] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:24:15.871670 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:15.876361 17831 raft_consensus.cc:3060] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:15.877413 17838 raft_consensus.cc:1275] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 20 index: 15. Preceding OpId from leader: term: 21 index: 16. (index mismatch)
W20260709 12:24:15.859611 25131 consensus_meta.cc:287] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.570s	user 0.006s	sys 0.017s
I20260709 12:24:15.880012 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:15.878672 25151 leader_election.cc:290] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 27 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:15.886741 17832 raft_consensus.cc:2393] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 22: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:24:15.890040 25674 raft_consensus.cc:2804] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:24:15.890726 25674 raft_consensus.cc:697] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 25 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
W20260709 12:24:15.891214 17802 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:15.891783 17802 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:15.877861 25736 raft_consensus.cc:740] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 22 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
W20260709 12:24:15.893815 17802 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:15.890177 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
W20260709 12:24:15.882175 17652 connection.cc:256] RPC call timeout handler was delayed by -1.631872076000036s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18520000000000003s for a response
W20260709 12:24:15.896214 17652 connection.cc:256] RPC call timeout handler was delayed by -1.2977861839999605s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16810000000000003s for a response
W20260709 12:24:15.896543 17652 connection.cc:256] RPC call timeout handler was delayed by -1.0745861840000543s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19290000000000002s for a response
W20260709 12:24:15.898317 17688 consensus_meta.cc:287] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.731s	user 0.008s	sys 0.000s
I20260709 12:24:15.898641 17688 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 21.
I20260709 12:24:15.900118 17829 raft_consensus.cc:1275] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 15 index: 11. Preceding OpId from leader: term: 17 index: 12. (index mismatch)
I20260709 12:24:15.903785 17686 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44898 (request call id 1564) took 1022 ms. Trace:
I20260709 12:24:15.904183 17686 rpcz_store.cc:276] 0709 12:24:14.881577 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:14.905652 (+ 24075us) service_pool.cc:224] Handling call
0709 12:24:14.933333 (+ 27681us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:15.705976 (+772643us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:15.903547 (+197571us) spinlock_profiling.cc:243] Waited 766 ms on lock 0x7b5400208354. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:24:15.903727 (+   180us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1760812800}
W20260709 12:24:15.905016 25264 consensus_meta.cc:287] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 1.449s	user 0.013s	sys 0.004s
I20260709 12:24:15.906693 25781 leader_election.cc:290] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:15.906765 25739 raft_consensus.cc:515] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:15.909265 25739 leader_election.cc:290] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 18 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:24:15.911695 25575 consensus_meta.cc:287] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.707s	user 0.001s	sys 0.010s
I20260709 12:24:15.912071 17831 raft_consensus.cc:2468] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 25.
I20260709 12:24:15.913359 25768 leader_election.cc:290] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:24:15.915228 17730 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:15.916316 17834 raft_consensus.cc:3060] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:15.917258 25778 leader_election.cc:290] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 21 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:15.892165 25721 raft_consensus.cc:493] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:15.918159 25721 raft_consensus.cc:515] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:15.901055 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:15.920246 25721 leader_election.cc:290] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:15.920673 17838 raft_consensus.cc:2393] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 22: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:24:15.891659 25674 consensus_queue.cc:237] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 24.13, Last appended by leader: 13, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:15.926571 17761 raft_consensus.cc:2468] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 26.
I20260709 12:24:15.926887 17757 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36572 (request call id 1405) took 1045 ms. Trace:
W20260709 12:24:15.927455 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 230145us
W20260709 12:24:15.915823 17730 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:15.927532 17757 rpcz_store.cc:276] 0709 12:24:14.881635 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:15.119118 (+237483us) service_pool.cc:224] Handling call
0709 12:24:15.926600 (+807482us) spinlock_profiling.cc:243] Waited 634 ms on lock 0x7b54001b9254. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:24:15.926802 (+   202us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1457526528}
I20260709 12:24:15.939260 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:15.940169 17832 raft_consensus.cc:2468] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 20.
I20260709 12:24:15.940829 17838 raft_consensus.cc:1240] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 19. Current term is 20. Ops: []
I20260709 12:24:15.945195 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:15.946345 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:15.947525 17764 raft_consensus.cc:2393] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 22: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:24:15.951398 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:15.952219 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:15.953019 17755 raft_consensus.cc:3060] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Advancing to term 20
I20260709 12:24:15.960811 17829 raft_consensus.cc:2393] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 23: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:24:15.961082 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:15.961632 17803 leader_election.cc:304] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:24:15.954412 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:15.884683 25131 raft_consensus.cc:515] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:15.965189 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:15.965809 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:24:15.966730 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 109364us
I20260709 12:24:15.967206 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 19 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:15.969686 17764 raft_consensus.cc:2410] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for term 19 because replica has last-logged OpId of term: 18 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 12.
I20260709 12:24:15.935398 25721 raft_consensus.cc:493] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
W20260709 12:24:15.957064 25637 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.524s	user 0.008s	sys 0.047s
I20260709 12:24:15.973681 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:15.938691 25657 leader_election.cc:290] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:15.976331 25862 raft_consensus.cc:493] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:15.980039 25862 raft_consensus.cc:515] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:15.982486 25862 leader_election.cc:290] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:15.983959 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:15.985730 17762 pending_rounds.cc:85] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb: Aborting all ops after (but not including) 11
I20260709 12:24:15.986114 17762 pending_rounds.cc:107] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb: Aborting uncommitted NO_OP operation due to leader change: 15.12
I20260709 12:24:15.986462 17762 raft_consensus.cc:2889] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:24:15.975867 25637 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.549s	user 0.008s	sys 0.047s
I20260709 12:24:15.988248 17757 raft_consensus.cc:1240] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 22. Current term is 23. Ops: [21.13-22.14]
I20260709 12:24:15.984665 17758 raft_consensus.cc:2468] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 25.
I20260709 12:24:15.985003 25749 leader_election.cc:290] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 24 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:16.000138 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 22 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:24:16.004902 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 60987.8us
I20260709 12:24:16.006630 17757 raft_consensus.cc:1240] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 20. Current term is 21. Ops: []
I20260709 12:24:15.988219 25721 raft_consensus.cc:515] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:16.012414 25721 leader_election.cc:290] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:16.012308 25778 raft_consensus.cc:493] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:16.013804 17834 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 20.
I20260709 12:24:16.014930 17761 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36572 (request call id 1396) took 1136 ms. Trace:
I20260709 12:24:16.017187 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:16.024431 25778 raft_consensus.cc:515] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:16.027477 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:16.027041 25778 leader_election.cc:290] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:16.028694 25778 consensus_queue.cc:1059] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 26.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.000s
I20260709 12:24:16.015477 17761 rpcz_store.cc:276] 0709 12:24:14.878759 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:14.880086 (+  1327us) service_pool.cc:224] Handling call
0709 12:24:16.014702 (+1134616us) spinlock_profiling.cc:243] Waited 973 ms on lock 0x7b54003e44d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:24:16.014856 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2238016384}
I20260709 12:24:16.033150 18367 raft_consensus.cc:1275] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 23 index: 17. Preceding OpId from leader: term: 24 index: 17. (term mismatch)
W20260709 12:24:16.041390 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:16.033596 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 22 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:16.034020 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 27 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:16.044389 18367 pending_rounds.cc:85] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da: Aborting all ops after (but not including) 16
I20260709 12:24:16.048952 18367 pending_rounds.cc:107] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da: Aborting uncommitted NO_OP operation due to leader change: 23.17
I20260709 12:24:16.049284 18367 raft_consensus.cc:2889] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:16.048157 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 27 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:16.050693 17763 raft_consensus.cc:3060] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:16.053068 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:16.053817 17652 leader_election.cc:304] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:16.054170 17687 raft_consensus.cc:2368] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for earlier term 19. Current term is 20.
I20260709 12:24:16.056244 18366 raft_consensus.cc:2410] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for term 22 because replica has last-logged OpId of term: 19 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 14.
I20260709 12:24:16.083431 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 25 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:16.082423 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:16.092357 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:16.091572 17758 raft_consensus.cc:2468] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 22.
I20260709 12:24:16.029587 25778 raft_consensus.cc:3055] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:24:16.029614 25131 leader_election.cc:290] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 21 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:16.099314 17683 raft_consensus.cc:1275] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 17 index: 14. Preceding OpId from leader: term: 21 index: 16. (index mismatch)
I20260709 12:24:16.093271 17684 raft_consensus.cc:2468] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 25.
I20260709 12:24:16.096490 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 16 candidate_status { last_received { term: 15 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:16.101270 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:16.102073 17682 raft_consensus.cc:2393] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 23: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:24:16.102262 25740 raft_consensus.cc:2749] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:24:16.102842 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:16.094732 25862 raft_consensus.cc:2804] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:24:16.104735 25862 raft_consensus.cc:697] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 20 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:24:16.106005 25862 consensus_queue.cc:237] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.12, Last appended by leader: 12, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:16.106451 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:16.106968 17802 leader_election.cc:304] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:24:16.107107 17829 raft_consensus.cc:2468] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 22.
I20260709 12:24:16.108027 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 21 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:16.108685 17829 raft_consensus.cc:2393] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 21: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:24:16.109154 17681 raft_consensus.cc:2393] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 16: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:24:16.103487 17834 raft_consensus.cc:2468] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 23.
W20260709 12:24:16.120599 17729 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.679s	user 0.008s	sys 0.017s
W20260709 12:24:16.121202 17729 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.680s	user 0.008s	sys 0.017s
I20260709 12:24:16.120445 17830 raft_consensus.cc:1240] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 23. Current term is 24. Ops: []
I20260709 12:24:16.125730 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:16.128703 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 27 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:16.122176 17802 leader_election.cc:304] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:24:16.131803 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 26 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:16.132936 18368 raft_consensus.cc:2410] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for term 26 because replica has last-logged OpId of term: 24 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 14.
I20260709 12:24:16.131855 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 23 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:15.998534 25736 consensus_queue.cc:260] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 22.13, Last appended by leader: 13, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:24:16.136974 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:16.120361 25384 raft_consensus.cc:3060] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:16.140461 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:16.096150 25778 raft_consensus.cc:740] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:24:16.141333 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:16.162475 17687 raft_consensus.cc:2368] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 21. Current term is 22.
I20260709 12:24:16.162657 18367 raft_consensus.cc:2468] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 22.
I20260709 12:24:16.150000 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:16.155898 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 21 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:16.156183 17729 leader_election.cc:304] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:16.135219 17758 raft_consensus.cc:2393] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 21: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:24:16.163841 18371 raft_consensus.cc:2468] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 23.
I20260709 12:24:16.164683 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 24 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:16.163954 18362 raft_consensus.cc:3060] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:16.141130 25131 raft_consensus.cc:493] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
W20260709 12:24:16.174957 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 127583us
I20260709 12:24:16.176256 17764 raft_consensus.cc:1275] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 21 index: 16. Preceding OpId from leader: term: 22 index: 17. (index mismatch)
I20260709 12:24:16.179455 17727 leader_election.cc:304] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:16.187868 25746 raft_consensus.cc:493] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:16.163530 25420 consensus_queue.cc:1059] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.025s
I20260709 12:24:16.188957 25420 raft_consensus.cc:3055] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:24:16.188611 25746 raft_consensus.cc:515] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:16.189334 25420 raft_consensus.cc:740] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:24:16.190233 25420 consensus_queue.cc:260] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 19.15, Last appended by leader: 15, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:16.204737 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:24:16.214342 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 58289.3us
I20260709 12:24:16.192914 17652 leader_election.cc:304] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:16.215158 18362 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 21.
I20260709 12:24:16.203796 17727 leader_election.cc:304] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:16.205678 17654 leader_election.cc:304] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:24:16.255604 17729 leader_election.cc:304] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
W20260709 12:24:16.194504 25834 log.cc:927] Time spent T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 1.052s	user 0.001s	sys 0.007s
W20260709 12:24:16.258378 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 54657.6us
I20260709 12:24:16.180820 25765 raft_consensus.cc:2749] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 12:24:16.261896 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:16.200296 25678 raft_consensus.cc:2804] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:24:16.191695 25420 raft_consensus.cc:3060] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Advancing to term 20
W20260709 12:24:16.261487 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 55884.1us
I20260709 12:24:16.191215 25746 leader_election.cc:290] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:24:16.266185 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 73351.9us
W20260709 12:24:16.278448 17728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:16.281384 25846 log.cc:927] Time spent T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.430s	user 0.003s	sys 0.002s
W20260709 12:24:16.281493 17585 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35958 (request call id 412) took 100782 ms (client timeout 15000 ms). Trace:
I20260709 12:24:16.195396 25858 consensus_queue.cc:1048] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.088s
I20260709 12:24:16.284403 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:16.285115 17834 raft_consensus.cc:2468] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 23.
I20260709 12:24:16.286777 17689 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37396 (request call id 3004) took 1863 ms. Trace:
I20260709 12:24:16.267983 25663 consensus_queue.cc:1059] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 23.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.020s
I20260709 12:24:16.289070 25663 raft_consensus.cc:3055] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:24:16.289489 25663 raft_consensus.cc:740] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 23 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:24:16.290553 25663 consensus_queue.cc:260] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 23.15, Last appended by leader: 15, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:16.250859 25131 raft_consensus.cc:515] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:16.264369 25657 raft_consensus.cc:2804] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:24:16.300313 25657 raft_consensus.cc:493] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:16.300767 25657 raft_consensus.cc:3060] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:16.300412 25778 consensus_queue.cc:260] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 26.20, Last appended by leader: 20, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:16.302898 17652 leader_election.cc:304] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:16.304101 25731 raft_consensus.cc:2804] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:24:16.287325 17689 rpcz_store.cc:276] 0709 12:24:14.423203 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:14.429814 (+  6611us) service_pool.cc:224] Handling call
0709 12:24:14.475186 (+ 45372us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:24:14.640701 (+165515us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:14.646712 (+  6011us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:24:14.721749 (+ 75037us) log.cc:844] Serialized 525 byte log entry
0709 12:24:14.906536 (+184787us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:24:14.906599 (+    63us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:14.906681 (+    82us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:16.193762 (+1287081us) spinlock_profiling.cc:243] Waited 10.1 ms on lock 0x7b4400022b4c. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b47459c85 00007f7b4745a6a2 00007f7b4a1cc1db 00007f7b4a1c7aaf 00007f7b4a1bb86e 00007f7b4a1babb6 00007f7b4a1ba7d8 00007f7b4a1c4221 00007f7b4a1c41d9 00007f7b4a1c4169
0709 12:24:16.286191 (+ 92429us) raft_consensus.cc:1685] finished
0709 12:24:16.286225 (+    34us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:16.286678 (+   453us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":9621,"raft.queue_time_us":23168,"raft.run_cpu_time_us":128,"raft.run_wall_time_us":8499,"spinlock_wait_cycles":23247744,"thread_start_us":183809,"threads_started":1,"wal-append.queue_time_us":419841}
I20260709 12:24:16.307232 17654 leader_election.cc:304] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:24:16.307245 17763 raft_consensus.cc:2468] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 27.
I20260709 12:24:16.309574 17755 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 20.
W20260709 12:24:16.307230 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 328015us
I20260709 12:24:16.323902 25663 raft_consensus.cc:3060] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:16.331106 17759 raft_consensus.cc:2468] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 25.
I20260709 12:24:16.350584 25384 raft_consensus.cc:2749] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:24:16.353274 18365 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 2314) took 1271 ms. Trace:
W20260709 12:24:16.359741 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 51716us
I20260709 12:24:16.212471 25620 raft_consensus.cc:515] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:16.304548 25731 raft_consensus.cc:493] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:16.305951 25777 consensus_queue.cc:1048] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.062s
I20260709 12:24:16.368554 25731 raft_consensus.cc:3060] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:16.264289 25678 raft_consensus.cc:493] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:24:16.266642 25151 raft_consensus.cc:2804] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:24:16.193141 17581 catalog_manager.cc:5697] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b reported cstate change: term changed from 15 to 16, leader changed from e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66) to e063541542a94969898868bf6eb5ce1b (127.16.29.67). New cstate: current_term: 16 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: HEALTHY } } }
I20260709 12:24:16.378283 17652 leader_election.cc:304] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:16.269042 25768 raft_consensus.cc:2749] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 12:24:16.268067 25761 raft_consensus.cc:2749] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:24:16.381127 17729 leader_election.cc:400] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 19 pre-election: Vote denied by peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577) with higher term. Message: Invalid argument: T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for earlier term 19. Current term is 20.
I20260709 12:24:16.353754 18365 rpcz_store.cc:276] 0709 12:24:15.081732 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:15.112598 (+ 30866us) service_pool.cc:224] Handling call
0709 12:24:15.137884 (+ 25286us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:15.144394 (+  6510us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:15.144429 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:15.144966 (+   537us) log.cc:844] Serialized 26 byte log entry
0709 12:24:15.227742 (+ 82776us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:24:15.227796 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:15.227896 (+   100us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:16.251072 (+1023176us) spinlock_profiling.cc:243] Waited 1.07 ms on lock 0x7b4400022b4c. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b47459c85 00007f7b4745a6a2 00007f7b4a1cc1db 00007f7b4a1c7aaf 00007f7b4a1bb86e 00007f7b4a1babb6 00007f7b4a1ba7d8 00007f7b4a1c4221 00007f7b4a1c41d9 00007f7b4a1c4169
0709 12:24:16.304646 (+ 53574us) raft_consensus.cc:1685] finished
0709 12:24:16.304666 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:16.353204 (+ 48538us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":3984,"raft.run_cpu_time_us":60,"raft.run_wall_time_us":60,"spinlock_wait_cycles":2454784,"thread_start_us":82045,"threads_started":1,"wal-append.queue_time_us":661760}
I20260709 12:24:16.260473 25760 consensus_queue.cc:1059] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 21.15, Next index: 16, Last known committed idx: 14, Time since last communication: 0.105s
I20260709 12:24:16.387228 17729 leader_election.cc:403] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 19 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:24:16.358119 25858 raft_consensus.cc:2804] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:24:16.263024 25264 consensus_queue.cc:1048] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.108s
W20260709 12:24:16.389691 18333 connection.cc:256] RPC call timeout handler was delayed by -1.8944086050000806s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1892s for a response
I20260709 12:24:16.383590 25678 raft_consensus.cc:3060] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Advancing to term 22
W20260709 12:24:16.419293 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 474398us
I20260709 12:24:16.398802 25858 raft_consensus.cc:493] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:16.422940 17729 leader_election.cc:304] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:24:16.406687 25424 raft_consensus.cc:2749] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Leader pre-election lost for term 19. Reason: Vote denied by peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577) with higher term. Message: Invalid argument: T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for earlier term 19. Current term is 20.
I20260709 12:24:16.421844 25777 raft_consensus.cc:493] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:16.429693 25858 raft_consensus.cc:3060] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Advancing to term 26
W20260709 12:24:16.282178 17585 rpcz_store.cc:269] 0709 12:22:35.498965 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:35.499168 (+   203us) service_pool.cc:224] Handling call
0709 12:22:35.741818 (+242650us) spinlock_profiling.cc:243] Waited 72 ms on lock 0x7b5400002210. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b474fbb71 00007f7b474fc0b2 00007f7b50475759 00007f7b5040da76 00007f7b503ded58 00007f7b504dd04e 00007f7b4e99e2a4 00007f7b4e99e232 00007f7b4e99e161 00007f7b4e99e0dc
0709 12:24:16.281188 (+100539370us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:24:07.395948 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:24:07.396713 (+   765us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:24:07.396763 (+    50us) write_op.cc:435] Acquired schema lock
0709 12:24:07.396783 (+    20us) tablet.cc:662] Decoding operations
0709 12:24:07.397795 (+  1012us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:24:07.397904 (+   109us) write_op.cc:641] Partition lock acquired for write op
0709 12:24:07.397944 (+    40us) tablet.cc:685] Acquiring locks for 5 operations
0709 12:24:07.398745 (+   801us) tablet.cc:701] Row locks acquired
0709 12:24:07.398805 (+    60us) write_op.cc:260] PREPARE: finished
0709 12:24:07.399043 (+   238us) write_op.cc:270] Start()
0709 12:24:07.399197 (+   154us) write_op.cc:276] Timestamp: P: 1783599847398967 usec, L: 0
0709 12:24:07.399232 (+    35us) op_driver.cc:348] REPLICATION: starting
0709 12:24:07.444944 (+ 45712us) log.cc:844] Serialized 2074 byte log entry
0709 12:24:07.705081 (+260137us) spinlock_profiling.cc:243] Waited 8.04 ms on lock 0x7b180039e380. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b475333f3 00007f7b475357c1 00007f7b47535779 00007f7b47535709 00007f7b475356d1 00007f7b475349cd 0000000000388634 0000000000388479 00007f7b4751683a
0709 12:24:07.839861 (+134780us) spinlock_profiling.cc:243] Waited 22 ms on lock 0x7b4400022b4c. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b47459c85 00007f7b4745a6a2 00007f7b4a1cc1db 00007f7b4a1c7aaf 00007f7b4a1bb86e 00007f7b4a1babb6 00007f7b4a1ba7d8 00007f7b4a1c4221 00007f7b4a1c41d9 00007f7b4a1c4169
0709 12:24:08.085625 (+245764us) op_driver.cc:464] REPLICATION: finished
0709 12:24:08.134984 (+ 49359us) write_op.cc:301] APPLY: starting
0709 12:24:08.183443 (+ 48459us) tablet.cc:1367] starting BulkCheckPresence
0709 12:24:08.326509 (+143066us) tablet.cc:1370] finished BulkCheckPresence
0709 12:24:08.326585 (+    76us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:24:08.391705 (+ 65120us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:24:08.428051 (+ 36346us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=10,key_file_lookups=10,delta_file_lookups=10,mrs_lookups=0
0709 12:24:08.428130 (+    79us) write_op.cc:312] APPLY: finished
0709 12:24:08.428599 (+   469us) log.cc:844] Serialized 57 byte log entry
0709 12:24:08.428864 (+   265us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:24:08.429014 (+   150us) write_op.cc:454] Released schema lock
0709 12:24:08.429112 (+    98us) write_op.cc:341] FINISH: Updating metrics
0709 12:24:08.429661 (+   549us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:24:08.429710 (+    49us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":461702,"reported_tablets":45,"spinlock_wait_cycles":165697920,"child_traces":[["op",{"apply.queue_time_us":38755,"apply.run_cpu_time_us":17230,"apply.run_wall_time_us":294638,"cfile_cache_hit":22,"cfile_cache_hit_bytes":25047,"num_ops":5,"prepare.queue_time_us":553624,"prepare.run_cpu_time_us":14256,"prepare.run_wall_time_us":279446,"raft.queue_time_us":142912,"raft.run_cpu_time_us":3361,"raft.run_wall_time_us":102300,"replication_time_us":627105,"spinlock_wait_cycles":68994048,"thread_start_us":664986,"threads_started":4,"wal-append.queue_time_us":173834,"wal-append.run_cpu_time_us":14837,"wal-append.run_wall_time_us":1757681}]]}
I20260709 12:24:16.444595 25768 raft_consensus.cc:2804] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:24:16.444443 25867 raft_consensus.cc:2804] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:24:16.462160 25866 raft_consensus.cc:2804] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:24:16.486435 25866 raft_consensus.cc:493] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:16.486873 25866 raft_consensus.cc:3060] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:16.445318 25777 raft_consensus.cc:515] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:16.489373 25777 leader_election.cc:290] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:16.488777 25674 consensus_queue.cc:1048] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:24:16.490097 25777 raft_consensus.cc:493] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:16.461042 25681 raft_consensus.cc:493] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:24:16.498587 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:16.500522 25867 raft_consensus.cc:697] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 22 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:24:16.501801 25867 consensus_queue.cc:237] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 20.12, Last appended by leader: 12, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:16.490655 25777 raft_consensus.cc:515] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:16.512446 25777 leader_election.cc:290] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:16.490607 25485 raft_consensus.cc:493] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:16.475018 25768 raft_consensus.cc:493] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:16.452913 25767 consensus_queue.cc:1059] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 21.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.040s
W20260709 12:24:16.516841 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 206035us
I20260709 12:24:16.524660 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:16.564481 17763 raft_consensus.cc:2468] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 25.
I20260709 12:24:16.525004 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:16.497483 25826 raft_consensus.cc:2804] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:24:16.525835 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:16.573820 17757 raft_consensus.cc:2468] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 24.
I20260709 12:24:16.530521 17581 catalog_manager.cc:5697] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b reported cstate change: term changed from 9 to 13, leader changed from e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66) to e063541542a94969898868bf6eb5ce1b (127.16.29.67). New cstate: current_term: 13 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: HEALTHY } } }
I20260709 12:24:16.517426 25783 raft_consensus.cc:3060] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Advancing to term 23
W20260709 12:24:16.544615 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:16.529467 25637 raft_consensus.cc:493] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:16.575578 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 261076us
I20260709 12:24:16.553108 25420 raft_consensus.cc:493] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:24:16.527223 25681 raft_consensus.cc:515] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:16.573321 25768 raft_consensus.cc:3060] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:16.563225 25777 raft_consensus.cc:493] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:24:16.591377 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 24 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:16.591506 18334 leader_election.cc:304] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
W20260709 12:24:16.580276 25865 log.cc:927] Time spent T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.093s	user 0.000s	sys 0.001s
I20260709 12:24:16.593565 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:16.530555 17774 tablet_copy_service.cc:342] P e2909cff1da44ad387f96a8ae3aa5afb: Request end of tablet copy session 334a4e1f59944544b3b3ff36616b57da-a5732e71b9d54f8786b8c511d69943bf received from {username='slave'} at 127.0.0.1:36572
I20260709 12:24:16.579869 25485 raft_consensus.cc:515] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:16.597963 17774 tablet_copy_service.cc:434] P e2909cff1da44ad387f96a8ae3aa5afb: ending tablet copy session 334a4e1f59944544b3b3ff36616b57da-a5732e71b9d54f8786b8c511d69943bf on tablet a5732e71b9d54f8786b8c511d69943bf with peer 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:16.597577 17762 raft_consensus.cc:2393] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 21: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:24:16.596493 17728 leader_election.cc:304] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
W20260709 12:24:16.600131 25874 log.cc:927] Time spent T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.070s	user 0.001s	sys 0.000s
I20260709 12:24:16.600227 25485 leader_election.cc:290] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:16.600942 25485 raft_consensus.cc:2804] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:24:16.600986 17728 leader_election.cc:304] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:16.601394 25485 raft_consensus.cc:493] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:16.601748 25485 raft_consensus.cc:3060] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:16.602263 17728 leader_election.cc:400] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 19 pre-election: Vote denied by peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071) with higher term. Message: Invalid argument: T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for earlier term 19. Current term is 20.
I20260709 12:24:16.604312 25663 raft_consensus.cc:2749] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:24:16.604977 25663 raft_consensus.cc:3060] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:16.586751 25637 raft_consensus.cc:515] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:16.608171 25767 consensus_queue.cc:1048] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.000s
I20260709 12:24:16.609078 17762 raft_consensus.cc:1275] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 18 index: 13. Preceding OpId from leader: term: 20 index: 14. (index mismatch)
I20260709 12:24:16.610495 25637 leader_election.cc:290] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:16.612334 25866 raft_consensus.cc:515] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:16.613653 17581 catalog_manager.cc:5697] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b reported cstate change: term changed from 10 to 14, leader changed from 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) to e063541542a94969898868bf6eb5ce1b (127.16.29.67). New cstate: current_term: 14 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: HEALTHY } } }
I20260709 12:24:16.614822 25866 leader_election.cc:290] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 24 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:16.584414 25620 leader_election.cc:290] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 26 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:16.586432 25760 raft_consensus.cc:3055] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 23 LEADER]: Stepping down as leader of term 23
W20260709 12:24:16.621348 17655 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:16.585489 25826 raft_consensus.cc:697] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 23 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:24:16.625780 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 27 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:24:16.618188 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 300366us
I20260709 12:24:16.624514 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
W20260709 12:24:16.628345 17802 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:16.629408 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:16.629522 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:16.630494 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 26 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:16.631402 17686 raft_consensus.cc:1275] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 20 index: 16. Preceding OpId from leader: term: 22 index: 17. (index mismatch)
I20260709 12:24:16.632485 17680 raft_consensus.cc:2393] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 27: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
W20260709 12:24:16.635727 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:16.634047 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:16.635131 17684 raft_consensus.cc:2468] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 22.
I20260709 12:24:16.637581 17683 raft_consensus.cc:2393] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 22: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:24:16.638170 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:24:16.639386 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 273595us
I20260709 12:24:16.622763 25663 raft_consensus.cc:2749] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
W20260709 12:24:16.653246 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 232347us
I20260709 12:24:16.620813 25760 raft_consensus.cc:740] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 23 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:24:16.644714 17728 leader_election.cc:304] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:16.616329 25420 raft_consensus.cc:515] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:16.658245 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:16.654361 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:16.653666 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 21 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:16.662654 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:16.622660 25151 raft_consensus.cc:697] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 19 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:24:16.663928 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
W20260709 12:24:16.664075 17801 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:16.669250 17682 raft_consensus.cc:2393] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 21: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
W20260709 12:24:16.669437 17801 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:16.621173 25721 consensus_queue.cc:1048] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.031s
I20260709 12:24:16.621591 25777 raft_consensus.cc:515] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:16.650485 25776 raft_consensus.cc:493] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:16.677367 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 17 candidate_status { last_received { term: 15 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:16.679970 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:16.679060 17684 raft_consensus.cc:2468] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 16.
I20260709 12:24:16.697115 25760 consensus_queue.cc:260] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 23.16, Last appended by leader: 16, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:16.698401 17803 leader_election.cc:304] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:16.700752 17581 catalog_manager.cc:5697] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b reported cstate change: term changed from 10 to 11, leader changed from e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66) to e063541542a94969898868bf6eb5ce1b (127.16.29.67). New cstate: current_term: 11 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: HEALTHY } } }
I20260709 12:24:16.702888 25657 raft_consensus.cc:515] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:16.675050 25889 consensus_queue.cc:1048] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.035s
I20260709 12:24:16.705232 25777 leader_election.cc:290] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:16.705669 25420 leader_election.cc:290] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:16.705917 25777 raft_consensus.cc:493] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:16.706480 25777 raft_consensus.cc:515] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:16.708143 17687 raft_consensus.cc:2393] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 18: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:24:16.708629 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:16.709197 17762 raft_consensus.cc:3060] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:16.682229 25776 raft_consensus.cc:515] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:16.711154 25776 leader_election.cc:290] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:24:16.712759 25878 log.cc:927] Time spent T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.051s	user 0.000s	sys 0.000s
I20260709 12:24:16.713651 17652 leader_election.cc:304] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:16.714628 25681 leader_election.cc:290] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:16.715490 17755 raft_consensus.cc:2393] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 24: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:24:16.715080 25678 raft_consensus.cc:515] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:16.717896 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 16 candidate_status { last_received { term: 15 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:16.718590 17834 raft_consensus.cc:2468] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 16.
I20260709 12:24:16.719664 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:16.720593 17803 leader_election.cc:304] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:16.668467 25746 raft_consensus.cc:2804] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:24:16.721035 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:16.721242 25746 raft_consensus.cc:493] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:16.721364 25721 raft_consensus.cc:2749] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:24:16.721697 25746 raft_consensus.cc:3060] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:16.722067 25721 raft_consensus.cc:2749] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:24:16.722234 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:16.720207 25826 consensus_queue.cc:237] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 22.13, Last appended by leader: 13, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:16.722846 17830 raft_consensus.cc:2468] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 23.
W20260709 12:24:16.724586 25877 log.cc:927] Time spent T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.052s	user 0.000s	sys 0.000s
I20260709 12:24:16.725236 17803 leader_election.cc:304] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:16.709204 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 21 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:16.726382 17684 raft_consensus.cc:2468] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 21.
I20260709 12:24:16.726218 25151 consensus_queue.cc:237] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.12, Last appended by leader: 12, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:16.727218 17764 raft_consensus.cc:2468] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 22.
I20260709 12:24:16.727487 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 21 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:16.698582 25760 raft_consensus.cc:3060] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:16.650184 17689 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8"
dest_uuid: "0da169ff82674dc1947055ce43de68a3"
mode: GRACEFUL
new_leader_uuid: "e063541542a94969898868bf6eb5ce1b"
 from {username='slave'} at 127.0.0.1:54540
I20260709 12:24:16.734580 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:16.735010 17689 raft_consensus.cc:606] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Received request to transfer leadership to TS e063541542a94969898868bf6eb5ce1b
I20260709 12:24:16.735476 17689 raft_consensus.cc:614] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260709 12:24:16.740882 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 327897us
I20260709 12:24:16.732960 25862 raft_consensus.cc:2749] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
W20260709 12:24:16.752466 25873 log.cc:927] Time spent T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.125s	user 0.000s	sys 0.001s
I20260709 12:24:16.735025 25731 raft_consensus.cc:515] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:16.746362 25777 leader_election.cc:290] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:16.746004 17581 catalog_manager.cc:5697] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b reported cstate change: term changed from 7 to 10, leader changed from 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) to e063541542a94969898868bf6eb5ce1b (127.16.29.67). New cstate: current_term: 10 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:24:16.746853 25765 raft_consensus.cc:3055] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:24:16.748622 25657 leader_election.cc:290] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 24 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:24:16.784423 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 59252.1us
W20260709 12:24:16.799404 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 208003us
I20260709 12:24:16.805819 17803 leader_election.cc:304] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:24:16.810025 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 144597us
I20260709 12:24:16.812183 25862 raft_consensus.cc:2804] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:24:16.812817 25862 raft_consensus.cc:697] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 23 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:24:16.813721 25862 consensus_queue.cc:237] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 22.18, Last appended by leader: 18, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:16.817713 17727 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:16.818435 17727 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:16.782469 25765 raft_consensus.cc:740] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 21 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:24:16.822607 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 22 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:16.824036 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 21 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:16.824894 17688 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 20.
I20260709 12:24:16.825946 18334 leader_election.cc:400] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 21 election: Vote denied by peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071) with higher term. Message: Invalid argument: T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 21. Current term is 22.
I20260709 12:24:16.826505 18334 leader_election.cc:403] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 21 election: Cancelling election due to peer responding with higher term
I20260709 12:24:16.823414 17689 raft_consensus.cc:2468] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 21.
I20260709 12:24:16.828303 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:16.829850 17756 raft_consensus.cc:2468] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 18.
I20260709 12:24:16.835268 17762 raft_consensus.cc:2468] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 24.
I20260709 12:24:16.837114 25884 raft_consensus.cc:2749] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:24:16.839649 25884 raft_consensus.cc:493] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:16.841629 25884 raft_consensus.cc:515] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:16.856473 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 50736.7us
I20260709 12:24:16.759867 25858 raft_consensus.cc:515] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:16.816771 25781 raft_consensus.cc:493] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:16.861718 17681 raft_consensus.cc:2393] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 26: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:24:16.722870 25768 raft_consensus.cc:515] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:16.862445 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:16.820437 25776 raft_consensus.cc:493] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
W20260709 12:24:16.876130 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 162533us
I20260709 12:24:16.872815 17689 raft_consensus.cc:2468] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 18.
I20260709 12:24:16.886461 25678 leader_election.cc:290] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 22 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:16.893734 25657 consensus_queue.cc:1048] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20260709 12:24:16.895509 25768 leader_election.cc:290] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 23 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:24:16.896790 18333 connection.cc:256] RPC call timeout handler was delayed by -1.5839086050000333s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2237s for a response
W20260709 12:24:16.897403 18333 connection.cc:256] RPC call timeout handler was delayed by -1.5785086050000245s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2243s for a response
W20260709 12:24:16.897811 18333 connection.cc:256] RPC call timeout handler was delayed by -1.4952454110002691s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.27530000000000004s for a response
I20260709 12:24:16.845978 25424 raft_consensus.cc:493] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:16.902969 17652 leader_election.cc:304] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:24:16.917320 17802 leader_election.cc:304] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:24:16.863489 25777 raft_consensus.cc:493] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:16.953893 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 68697.2us
I20260709 12:24:16.844805 25575 consensus_queue.cc:1048] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.058s
I20260709 12:24:16.954162 25761 raft_consensus.cc:2804] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Leader election won for term 25
W20260709 12:24:16.922508 25893 log.cc:927] Time spent T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.235s	user 0.001s	sys 0.000s
I20260709 12:24:16.883620 25864 raft_consensus.cc:493] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:16.843015 25887 consensus_queue.cc:1048] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.128s
I20260709 12:24:16.966225 17762 raft_consensus.cc:1240] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 22. Current term is 23. Ops: [22.17-22.17]
I20260709 12:24:16.956645 17652 leader_election.cc:304] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:24:16.968166 17802 leader_election.cc:304] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:16.998358 17758 raft_consensus.cc:1275] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 20 index: 12. Preceding OpId from leader: term: 23 index: 14. (index mismatch)
W20260709 12:24:17.001575 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 154634us
I20260709 12:24:16.880242 25781 raft_consensus.cc:515] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:16.998828 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 140015us
I20260709 12:24:17.005612 17802 leader_election.cc:304] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
W20260709 12:24:17.005843 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 49240.2us
I20260709 12:24:17.006944 17652 leader_election.cc:304] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:24:16.965029 25867 raft_consensus.cc:2804] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Leader pre-election won for term 23
W20260709 12:24:17.005861 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 132116us
W20260709 12:24:17.014477 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:16.952006 25424 raft_consensus.cc:515] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:16.971211 25761 raft_consensus.cc:697] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 25 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:24:16.958135 25777 raft_consensus.cc:515] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:17.019917 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:17.023164 17687 raft_consensus.cc:2393] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 22: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:24:16.901814 25776 raft_consensus.cc:515] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:17.007865 25739 raft_consensus.cc:493] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:17.036335 25739 raft_consensus.cc:515] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:17.037286 25784 raft_consensus.cc:493] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:17.037775 25784 raft_consensus.cc:515] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:17.038509 25870 log.cc:927] Time spent T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.452s	user 0.000s	sys 0.005s
I20260709 12:24:17.038756 25739 leader_election.cc:290] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:24:17.039518 25880 log.cc:927] Time spent T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.300s	user 0.001s	sys 0.001s
I20260709 12:24:17.025404 25866 raft_consensus.cc:2804] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:24:17.040122 17760 raft_consensus.cc:2468] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 23.
I20260709 12:24:17.040325 25866 raft_consensus.cc:493] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:17.007983 25721 raft_consensus.cc:2749] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:24:17.020211 25886 raft_consensus.cc:2804] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:24:17.013602 25867 raft_consensus.cc:493] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:17.041350 25761 consensus_queue.cc:237] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 24.16, Last appended by leader: 16, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:17.012341 17581 catalog_manager.cc:5697] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b reported cstate change: term changed from 5 to 9, leader changed from e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66) to 0da169ff82674dc1947055ce43de68a3 (127.16.29.65). New cstate: current_term: 9 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } }
I20260709 12:24:17.042522 25781 leader_election.cc:290] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:17.026196 17685 raft_consensus.cc:1240] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 23. Current term is 24. Ops: []
I20260709 12:24:17.013219 25826 raft_consensus.cc:2804] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Leader pre-election won for term 21
W20260709 12:24:17.028168 25888 log.cc:927] Time spent T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.158s	user 0.000s	sys 0.001s
W20260709 12:24:17.028483 25896 log.cc:927] Time spent T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.127s	user 0.000s	sys 0.001s
I20260709 12:24:17.030882 17831 raft_consensus.cc:1240] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 25. Current term is 26. Ops: [25.14-25.14]
W20260709 12:24:17.032884 25900 log.cc:927] Time spent T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.063s	user 0.000s	sys 0.000s
I20260709 12:24:17.059546 25915 raft_consensus.cc:493] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:17.060302 25915 raft_consensus.cc:515] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:17.062327 17838 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35042 (request call id 2977) took 1091 ms. Trace:
I20260709 12:24:17.060804 25688 raft_consensus.cc:3060] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:17.041323 25886 raft_consensus.cc:697] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 20 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:24:17.063256 25915 leader_election.cc:290] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:17.064411 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:17.064175 25886 consensus_queue.cc:237] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.12, Last appended by leader: 12, Current term: 20, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:17.065658 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:17.066746 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:17.069722 25736 raft_consensus.cc:3060] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:17.073251 25131 leader_election.cc:290] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:17.035125 25638 raft_consensus.cc:493] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:17.074705 25778 raft_consensus.cc:3060] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:17.047097 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:17.005379 25864 raft_consensus.cc:515] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:17.009568 25485 raft_consensus.cc:515] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:17.062712 17838 rpcz_store.cc:276] 0709 12:24:15.971164 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:15.971328 (+   164us) service_pool.cc:224] Handling call
0709 12:24:15.977461 (+  6133us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:16.400540 (+423079us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:16.400652 (+   112us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:16.401237 (+   585us) log.cc:844] Serialized 26 byte log entry
0709 12:24:16.656743 (+255506us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:24:16.657488 (+   745us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:16.657569 (+    81us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:17.062105 (+404536us) raft_consensus.cc:1685] finished
0709 12:24:17.062125 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:17.062265 (+   140us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":11911,"raft.queue_time_us":39591,"raft.run_cpu_time_us":122,"raft.run_wall_time_us":122,"thread_start_us":626002,"threads_started":2,"wal-append.queue_time_us":384289}
I20260709 12:24:17.045487 25826 raft_consensus.cc:493] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:17.019038 25899 raft_consensus.cc:2749] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Leader election lost for term 21. Reason: Vote denied by peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071) with higher term. Message: Invalid argument: T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 21. Current term is 22.
I20260709 12:24:17.049292 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:17.041303 25867 raft_consensus.cc:3060] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:16.965111 25765 consensus_queue.cc:260] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 21.15, Last appended by leader: 15, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:17.049629 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:17.057776 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:17.057807 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:17.059485 25777 leader_election.cc:290] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:17.033025 25420 raft_consensus.cc:493] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:17.074874 25638 raft_consensus.cc:515] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:17.077600 17760 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36572 (request call id 1394) took 2200 ms (client timeout 1726 ms). Trace:
I20260709 12:24:17.040769 25866 raft_consensus.cc:3060] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Advancing to term 17
W20260709 12:24:17.081576 17652 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:17.123250 17652 leader_election.cc:304] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:17.124085 25638 leader_election.cc:290] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:17.124933 25420 raft_consensus.cc:515] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:17.125820 25887 raft_consensus.cc:493] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:24:17.126412 25887 raft_consensus.cc:515] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:17.082523 25826 raft_consensus.cc:3060] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:17.128036 25777 raft_consensus.cc:493] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:17.128545 25777 raft_consensus.cc:515] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:17.130640 25777 leader_election.cc:290] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:17.131125 25777 raft_consensus.cc:493] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:17.131633 25777 raft_consensus.cc:515] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:17.134866 25777 leader_election.cc:290] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:17.136868 25776 leader_election.cc:290] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:17.139402 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:17.084038 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:17.142187 17762 raft_consensus.cc:2468] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 19.
I20260709 12:24:17.088088 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:17.143277 17681 raft_consensus.cc:2468] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 25.
I20260709 12:24:17.089118 17834 raft_consensus.cc:2468] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 18.
W20260709 12:24:17.103356 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:17.081506 25868 raft_consensus.cc:493] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:17.150725 18333 connection.cc:256] RPC call timeout handler was delayed by -1.1069086050001715s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2767s for a response
W20260709 12:24:17.108577 17760 rpcz_store.cc:269] 0709 12:24:14.876972 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:14.877139 (+   167us) service_pool.cc:224] Handling call
0709 12:24:17.077379 (+2200240us) spinlock_profiling.cc:243] Waited 2.11 s on lock 0x7b54002b5554. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:24:17.077490 (+   111us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4842284416}
W20260709 12:24:17.152184 18333 connection.cc:256] RPC call timeout handler was delayed by -1.1069086050001715s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2767s for a response
W20260709 12:24:17.157567 18333 connection.cc:256] RPC call timeout handler was delayed by -1.1034578520002469s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16040000000000001s for a response
I20260709 12:24:17.157732 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:24:17.158630 18333 connection.cc:256] RPC call timeout handler was delayed by -1.0429646170000524s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1577s for a response
I20260709 12:24:17.158876 17760 raft_consensus.cc:2468] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 19.
I20260709 12:24:17.159639 17801 leader_election.cc:304] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:17.161473 17837 raft_consensus.cc:2468] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 24.
I20260709 12:24:17.131464 25424 leader_election.cc:290] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:17.106804 25884 leader_election.cc:290] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:17.176795 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:17.177742 17681 raft_consensus.cc:2468] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 19.
I20260709 12:24:17.168288 25777 raft_consensus.cc:2749] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:24:17.180045 25866 raft_consensus.cc:515] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:17.152129 25420 leader_election.cc:290] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:24:17.184455 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 61282.8us
W20260709 12:24:17.184882 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 179325us
I20260709 12:24:17.186636 17652 leader_election.cc:304] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:17.187162 17802 leader_election.cc:304] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
W20260709 12:24:17.187521 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:17.176414 17801 leader_election.cc:304] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:17.171154 25899 raft_consensus.cc:493] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:17.193327 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:17.194203 17681 raft_consensus.cc:2468] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 23.
I20260709 12:24:17.200068 17652 leader_election.cc:304] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:17.201243 17652 leader_election.cc:304] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:17.184127 25885 raft_consensus.cc:2749] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:24:17.206434 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:17.205471 17652 leader_election.cc:304] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:17.209210 18368 raft_consensus.cc:1275] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 19 index: 15. Preceding OpId from leader: term: 21 index: 16. (index mismatch)
I20260709 12:24:17.209427 18361 raft_consensus.cc:1275] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 20 index: 16. Preceding OpId from leader: term: 22 index: 17. (index mismatch)
I20260709 12:24:17.211404 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:17.190883 25681 raft_consensus.cc:2804] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:24:17.212142 18371 raft_consensus.cc:3060] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:17.212249 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:17.213493 25887 leader_election.cc:290] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:17.193557 25895 raft_consensus.cc:2804] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:24:17.218122 18366 raft_consensus.cc:3060] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:17.195807 25638 raft_consensus.cc:493] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:17.178745 25746 raft_consensus.cc:515] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:17.218556 18365 raft_consensus.cc:2468] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 19.
I20260709 12:24:17.202131 25761 raft_consensus.cc:2749] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:24:17.206580 25884 consensus_queue.cc:1059] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 20.15, Next index: 16, Last known committed idx: 14, Time since last communication: 0.016s
I20260709 12:24:17.199293 25883 raft_consensus.cc:493] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:17.275935 25884 raft_consensus.cc:3055] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:24:17.275636 25883 raft_consensus.cc:515] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:17.276328 25884 raft_consensus.cc:740] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:24:17.171533 25868 raft_consensus.cc:515] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:17.226023 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:24:17.231519 25922 log.cc:927] Time spent T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.054s	user 0.001s	sys 0.000s
I20260709 12:24:17.231722 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:17.220744 25681 raft_consensus.cc:493] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:17.231726 25638 raft_consensus.cc:515] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:17.200879 25899 raft_consensus.cc:515] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:17.246719 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:17.248979 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:17.299320 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:17.308991 18367 raft_consensus.cc:2468] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 19.
I20260709 12:24:17.311432 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 18 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:17.255864 17728 leader_election.cc:304] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:17.256974 25887 raft_consensus.cc:2804] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:24:17.259805 25864 leader_election.cc:290] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:17.262441 25776 consensus_queue.cc:1048] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:24:17.206619 25917 raft_consensus.cc:493] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:17.223295 25919 consensus_queue.cc:1048] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.021s
W20260709 12:24:17.264384 17803 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:17.265866 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:17.266734 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:17.209424 25575 raft_consensus.cc:2749] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:24:17.207568 25777 raft_consensus.cc:2749] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:24:17.272071 25912 consensus_queue.cc:1048] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.004s
I20260709 12:24:17.272774 17837 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35040 (request call id 3222) took 1403 ms. Trace:
I20260709 12:24:17.273182 25746 leader_election.cc:290] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 24 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:17.274019 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:17.274225 18334 leader_election.cc:304] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:24:17.329890 17684 raft_consensus.cc:2393] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 26: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:24:17.296521 25681 raft_consensus.cc:3060] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:17.280753 17832 raft_consensus.cc:1275] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 23 index: 16. Preceding OpId from leader: term: 24 index: 17. (index mismatch)
I20260709 12:24:17.283360 17683 raft_consensus.cc:1275] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 18 index: 12. Preceding OpId from leader: term: 20 index: 13. (index mismatch)
I20260709 12:24:17.284008 25868 leader_election.cc:290] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:17.232117 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:24:17.409297 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 77470.5us
I20260709 12:24:17.289335 18366 raft_consensus.cc:2468] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 23.
I20260709 12:24:17.410735 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:17.411419 18362 raft_consensus.cc:2468] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 23.
I20260709 12:24:17.293597 18371 raft_consensus.cc:2468] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 24.
I20260709 12:24:17.413594 25826 raft_consensus.cc:515] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:17.286439 17682 raft_consensus.cc:2376] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 21. Re-sending same reply.
I20260709 12:24:17.294836 17688 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119"
dest_uuid: "0da169ff82674dc1947055ce43de68a3"
mode: GRACEFUL
new_leader_uuid: "e063541542a94969898868bf6eb5ce1b"
 from {username='slave'} at 127.0.0.1:54540
I20260709 12:24:17.270093 17652 leader_election.cc:304] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:17.415297 17688 raft_consensus.cc:606] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Received request to transfer leadership to TS e063541542a94969898868bf6eb5ce1b
I20260709 12:24:17.415630 17688 raft_consensus.cc:614] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:24:17.296638 17681 raft_consensus.cc:2393] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 15: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:24:17.301463 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:17.303133 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:17.304989 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:17.417083 17761 raft_consensus.cc:2410] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 23 because replica has last-logged OpId of term: 22 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 12.
I20260709 12:24:17.299767 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:17.408046 18365 raft_consensus.cc:2468] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 19.
I20260709 12:24:17.309023 17830 raft_consensus.cc:3060] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:17.307817 18333 leader_election.cc:304] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:17.321842 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:24:17.324227 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 68461.8us
I20260709 12:24:17.325942 18368 raft_consensus.cc:2468] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 19.
I20260709 12:24:17.325965 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 23 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:17.328310 17837 rpcz_store.cc:276] 0709 12:24:15.869435 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:15.879738 (+ 10303us) service_pool.cc:224] Handling call
0709 12:24:17.272605 (+1392867us) spinlock_profiling.cc:243] Waited 1.26 s on lock 0x7b540008a2d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:24:17.272718 (+   113us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2896550784}
W20260709 12:24:17.280836 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 56820.8us
I20260709 12:24:17.419843 18367 raft_consensus.cc:2468] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 22.
W20260709 12:24:17.332458 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 58311.5us
I20260709 12:24:17.296085 25895 raft_consensus.cc:697] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 24 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:24:17.285694 25918 consensus_queue.cc:1059] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 21.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.036s
I20260709 12:24:17.409535 18364 raft_consensus.cc:2468] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 19.
I20260709 12:24:17.298979 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:17.305789 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:17.321812 25674 raft_consensus.cc:493] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:24:17.420994 17801 leader_election.cc:304] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:17.325403 25485 leader_election.cc:290] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 26 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:17.323913 25887 raft_consensus.cc:697] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 27 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
W20260709 12:24:17.485289 17727 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:17.484966 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 214909us
I20260709 12:24:17.506208 25151 raft_consensus.cc:2804] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:24:17.503685 25690 raft_consensus.cc:2749] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:24:17.506757 25151 raft_consensus.cc:493] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:17.330758 25864 raft_consensus.cc:2749] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:24:17.507234 25151 raft_consensus.cc:3060] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:17.330090 25917 raft_consensus.cc:515] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:17.511830 25917 leader_election.cc:290] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:17.512529 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 24 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:17.516651 17689 raft_consensus.cc:1240] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 20. Current term is 21. Ops: []
I20260709 12:24:17.280464 25884 consensus_queue.cc:260] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 20.15, Last appended by leader: 15, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:17.517843 17728 leader_election.cc:304] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:17.518181 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:17.503350 17829 raft_consensus.cc:2468] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 18.
I20260709 12:24:17.519127 17837 raft_consensus.cc:2468] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 18.
I20260709 12:24:17.411545 25867 raft_consensus.cc:515] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:17.522912 18367 raft_consensus.cc:1275] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 15 index: 11. Preceding OpId from leader: term: 17 index: 12. (index mismatch)
I20260709 12:24:17.524284 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:17.524945 17760 raft_consensus.cc:2468] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 23.
I20260709 12:24:17.527379 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:17.502359 18368 raft_consensus.cc:1275] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 23 index: 16. Preceding OpId from leader: term: 24 index: 17. (index mismatch)
I20260709 12:24:17.530272 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 26 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:17.530846 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:17.531348 17757 raft_consensus.cc:2410] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for term 26 because replica has last-logged OpId of term: 24 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 12.
I20260709 12:24:17.531675 17758 raft_consensus.cc:3060] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:17.528082 17760 raft_consensus.cc:2468] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 23.
I20260709 12:24:17.533241 17652 leader_election.cc:304] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:17.533166 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:17.534119 17836 raft_consensus.cc:3060] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:24:17.512526 25927 log.cc:927] Time spent T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.098s	user 0.000s	sys 0.001s
I20260709 12:24:17.423692 25637 raft_consensus.cc:2804] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:24:17.542089 17728 leader_election.cc:304] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:17.542701 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:17.544521 17831 raft_consensus.cc:2368] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for earlier term 22. Current term is 23.
I20260709 12:24:17.544641 17756 raft_consensus.cc:1275] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 22 index: 16. Preceding OpId from leader: term: 24 index: 17. (index mismatch)
I20260709 12:24:17.536991 17652 leader_election.cc:304] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:17.485100 25131 raft_consensus.cc:493] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:17.547528 25777 raft_consensus.cc:2749] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:24:17.548175 25914 raft_consensus.cc:493] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
W20260709 12:24:17.549690 25923 log.cc:927] Time spent T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.129s	user 0.001s	sys 0.000s
I20260709 12:24:17.551488 17755 raft_consensus.cc:1240] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 23. Current term is 24. Ops: []
I20260709 12:24:17.552596 17728 leader_election.cc:304] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:17.528358 25721 raft_consensus.cc:493] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:17.585912 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:17.601919 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:17.532312 25663 raft_consensus.cc:493] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:17.541770 25784 leader_election.cc:290] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:17.605341 25424 raft_consensus.cc:2804] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Leader pre-election won for term 19
W20260709 12:24:17.605422 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 52924.2us
I20260709 12:24:17.637531 17728 leader_election.cc:304] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:17.547508 25864 raft_consensus.cc:2749] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
W20260709 12:24:17.612957 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 343546us
W20260709 12:24:17.597966 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 177037us
I20260709 12:24:17.641690 17801 leader_election.cc:304] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:24:17.618854 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 23 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:17.541957 25637 raft_consensus.cc:493] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:17.613502 25918 raft_consensus.cc:2804] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:24:17.630682 25131 raft_consensus.cc:515] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:24:17.553151 25884 raft_consensus.cc:3060] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:17.633136 17652 leader_election.cc:304] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:17.634414 17830 raft_consensus.cc:2468] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 24.
I20260709 12:24:17.676635 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:17.543968 25575 raft_consensus.cc:2749] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:24:17.554945 25912 raft_consensus.cc:2749] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:24:17.528615 25674 raft_consensus.cc:515] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:17.540544 25384 consensus_queue.cc:1059] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 24.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.094s
W20260709 12:24:17.632803 25926 log.cc:927] Time spent T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.107s	user 0.003s	sys 0.000s
I20260709 12:24:17.640496 17728 leader_election.cc:304] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:17.601477 25914 raft_consensus.cc:515] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:17.699242 25914 leader_election.cc:290] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:17.700166 25638 leader_election.cc:290] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:17.633994 25895 consensus_queue.cc:237] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 22.13, Last appended by leader: 13, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:17.712450 25867 leader_election.cc:290] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 23 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:24:17.684509 25931 log.cc:927] Time spent T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.084s	user 0.000s	sys 0.001s
W20260709 12:24:17.638319 25888 log.cc:927] Time spent T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.095s	user 0.005s	sys 0.000s
I20260709 12:24:17.637622 25424 raft_consensus.cc:493] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:24:17.654209 25778 consensus_meta.cc:287] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.577s	user 0.009s	sys 0.003s
I20260709 12:24:17.647136 25881 raft_consensus.cc:493] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:17.744552 25881 raft_consensus.cc:515] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:17.644450 25637 raft_consensus.cc:3060] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:17.605309 25721 raft_consensus.cc:515] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:17.643784 25919 raft_consensus.cc:493] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:17.748320 25919 raft_consensus.cc:515] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:17.748420 25885 consensus_queue.cc:1048] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:24:17.670166 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:17.751770 17761 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38786 (request call id 3698) took 1620 ms. Trace:
I20260709 12:24:17.647868 25918 raft_consensus.cc:493] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:17.632933 25663 raft_consensus.cc:515] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:17.755110 25663 leader_election.cc:290] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:17.641160 25767 consensus_queue.cc:1059] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 23.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.025s
W20260709 12:24:17.751780 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 88313.5us
I20260709 12:24:17.756558 25767 raft_consensus.cc:3055] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:24:17.756975 25767 raft_consensus.cc:740] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:24:17.758355 25131 leader_election.cc:290] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:17.758003 25767 consensus_queue.cc:260] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 23.14, Last appended by leader: 14, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:17.757230 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:24:17.680982 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 47957.8us
W20260709 12:24:17.693687 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 53265.9us
W20260709 12:24:17.760646 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 480360us
I20260709 12:24:17.760860 17728 leader_election.cc:304] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:17.693868 25912 raft_consensus.cc:2804] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:24:17.694353 25890 raft_consensus.cc:2804] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:24:17.761773 17728 leader_election.cc:304] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:17.696416 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 26 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:17.677083 25887 consensus_queue.cc:237] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 26.20, Last appended by leader: 20, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:17.781514 17836 raft_consensus.cc:2468] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 24.
I20260709 12:24:17.727136 25826 leader_election.cc:290] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 21 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:24:17.727715 25939 log.cc:927] Time spent T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.122s	user 0.000s	sys 0.004s
I20260709 12:24:17.731112 25867 raft_consensus.cc:493] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
W20260709 12:24:17.739471 25940 log.cc:927] Time spent T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.135s	user 0.003s	sys 0.000s
I20260709 12:24:17.686376 25761 raft_consensus.cc:2804] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:24:17.743671 25424 raft_consensus.cc:3060] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:17.752489 17761 rpcz_store.cc:276] 0709 12:24:16.131427 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:16.131610 (+   183us) service_pool.cc:224] Handling call
0709 12:24:17.751565 (+1619955us) spinlock_profiling.cc:243] Waited 1.21 s on lock 0x7b54003eb054. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:24:17.751706 (+   141us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2772827008}
I20260709 12:24:17.731786 25858 leader_election.cc:290] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 26 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:17.766803 25918 raft_consensus.cc:3060] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Advancing to term 19
W20260709 12:24:17.695464 25935 log.cc:927] Time spent T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.059s	user 0.000s	sys 0.000s
W20260709 12:24:17.775770 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:17.777158 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:17.786686 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 24 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:17.822952 18371 raft_consensus.cc:2468] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 23.
I20260709 12:24:17.804039 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:17.776800 25575 raft_consensus.cc:493] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:17.798555 25131 raft_consensus.cc:2804] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:24:17.838624 25131 raft_consensus.cc:493] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:24:17.839071 25131 raft_consensus.cc:3060] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:17.841322 25919 leader_election.cc:290] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:17.785295 25739 raft_consensus.cc:493] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:17.843261 18364 raft_consensus.cc:2393] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 26: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:24:17.843686 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:17.845328 17836 raft_consensus.cc:2468] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 23.
W20260709 12:24:17.811968 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 48885.7us
I20260709 12:24:17.818014 17830 raft_consensus.cc:1240] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 24 LEADER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 23. Current term is 24. Ops: []
I20260709 12:24:17.758016 18333 leader_election.cc:304] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:17.777910 25768 raft_consensus.cc:2804] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:24:17.766680 25912 raft_consensus.cc:493] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:17.805717 18361 raft_consensus.cc:1275] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 16 index: 12. Preceding OpId from leader: term: 19 index: 13. (index mismatch)
I20260709 12:24:17.819310 25761 raft_consensus.cc:493] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:24:17.874722 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 88847.9us
I20260709 12:24:17.815719 25867 raft_consensus.cc:515] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:17.833967 25731 leader_election.cc:290] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 24 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:17.837807 25862 raft_consensus.cc:3055] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:24:17.809326 25777 raft_consensus.cc:493] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:17.847456 17729 leader_election.cc:304] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:17.850783 25768 raft_consensus.cc:493] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:17.852283 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:17.855504 25575 raft_consensus.cc:515] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:24:17.858764 25883 leader_election.cc:290] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:17.875200 25912 raft_consensus.cc:3060] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:17.864763 25866 leader_election.cc:290] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 17 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:17.864921 25881 leader_election.cc:290] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:17.867108 25765 raft_consensus.cc:3060] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:17.877449 17758 raft_consensus.cc:2468] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 26.
I20260709 12:24:17.852187 25781 consensus_queue.cc:1059] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 20.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.035s
I20260709 12:24:17.877655 25862 raft_consensus.cc:740] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 22 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:24:17.821854 25890 raft_consensus.cc:697] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 18 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:24:17.824858 25767 raft_consensus.cc:3060] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:17.879284 17729 leader_election.cc:304] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:17.886896 17832 raft_consensus.cc:2393] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 26: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:24:17.848551 25858 raft_consensus.cc:493] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:17.860781 25681 raft_consensus.cc:515] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:17.876243 25761 raft_consensus.cc:3060] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:17.900327 17729 leader_election.cc:304] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:24:17.893987 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 122658us
I20260709 12:24:17.888137 25768 raft_consensus.cc:3060] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:17.887816 25663 raft_consensus.cc:2804] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:24:17.891546 25731 raft_consensus.cc:493] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:24:17.876469 25915 raft_consensus.cc:2749] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:24:17.876950 25424 raft_consensus.cc:515] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:17.874600 25739 raft_consensus.cc:515] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:17.914296 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 156417us
I20260709 12:24:17.888813 25899 leader_election.cc:290] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:17.919391 17729 leader_election.cc:304] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:24:17.899550 25784 raft_consensus.cc:2749] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:24:17.932782 25151 raft_consensus.cc:515] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:17.887682 25777 raft_consensus.cc:515] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:17.907071 25905 raft_consensus.cc:2804] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:24:17.912894 25858 raft_consensus.cc:515] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:17.956012 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:17.958711 17729 leader_election.cc:304] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:24:17.986873 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 90920us
I20260709 12:24:17.965656 17652 leader_election.cc:304] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
W20260709 12:24:18.001570 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 43908.9us
I20260709 12:24:17.985380 17729 leader_election.cc:304] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:18.002040 18333 leader_election.cc:304] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:18.004187 17652 leader_election.cc:304] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:18.007349 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 17 candidate_status { last_received { term: 15 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:18.008669 17689 raft_consensus.cc:3060] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:17.935848 25151 leader_election.cc:290] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 20 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:17.950769 25777 leader_election.cc:290] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:17.957962 25905 raft_consensus.cc:697] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 24 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:24:17.979910 25914 raft_consensus.cc:2804] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:24:17.949838 25901 raft_consensus.cc:3055] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:24:17.958058 25781 raft_consensus.cc:2804] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Leader pre-election won for term 20
W20260709 12:24:18.013188 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 43834.6us
I20260709 12:24:17.954612 25858 leader_election.cc:290] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:24:18.016451 25946 log.cc:927] Time spent T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.119s	user 0.000s	sys 0.000s
I20260709 12:24:18.006184 25884 raft_consensus.cc:2804] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:24:17.956171 25731 raft_consensus.cc:515] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:18.007364 25663 raft_consensus.cc:697] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 23 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:24:17.982805 25637 raft_consensus.cc:515] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:18.012893 25881 raft_consensus.cc:493] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:18.067845 18366 raft_consensus.cc:1240] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 21. Current term is 22. Ops: [21.16-21.16]
I20260709 12:24:18.068394 25881 raft_consensus.cc:515] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:18.072867 25424 leader_election.cc:290] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 19 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:18.075028 25131 raft_consensus.cc:515] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:18.009467 25890 consensus_queue.cc:237] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.12, Last appended by leader: 12, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:18.051263 25954 raft_consensus.cc:2804] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:24:18.085219 18333 leader_election.cc:400] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 21 pre-election: Vote denied by peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651) with higher term. Message: Invalid argument: T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 21. Current term is 22.
I20260709 12:24:18.085415 25954 raft_consensus.cc:493] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:18.085932 25954 raft_consensus.cc:3060] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:18.023576 25914 raft_consensus.cc:493] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:18.052798 25784 raft_consensus.cc:493] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:18.052313 25905 consensus_queue.cc:237] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 22.16, Last appended by leader: 16, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:18.046633 25151 consensus_queue.cc:1048] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.007s
I20260709 12:24:18.014756 25917 raft_consensus.cc:2764] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Leader pre-election decision vote started in defunct term 21: won
I20260709 12:24:18.048391 25884 raft_consensus.cc:493] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:18.023147 25777 raft_consensus.cc:2749] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260709 12:24:18.064316 25912 raft_consensus.cc:515] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:18.066820 25637 leader_election.cc:290] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 22 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:18.025198 25781 raft_consensus.cc:493] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:18.075836 25867 leader_election.cc:290] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:24:18.080718 25948 log.cc:927] Time spent T 00000000000000000000000000000000 P 2193af31cb894ec9b3d65d939a2a6851: Append to log took a long time: real 0.125s	user 0.002s	sys 0.000s
I20260709 12:24:18.081858 25663 consensus_queue.cc:237] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 21.16, Last appended by leader: 16, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:18.044498 25957 raft_consensus.cc:2804] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Leader pre-election won for term 20
W20260709 12:24:18.046690 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:18.084563 25681 leader_election.cc:290] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 21 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:18.087831 18333 leader_election.cc:304] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:24:18.090663 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:17.948160 25862 consensus_queue.cc:260] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 22.17, Last appended by leader: 17, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:18.022646 25901 raft_consensus.cc:740] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:24:18.116496 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:18.121589 25674 leader_election.cc:290] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:18.121959 25912 leader_election.cc:290] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 26 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:18.088544 25914 raft_consensus.cc:3060] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:18.093286 25884 raft_consensus.cc:3060] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:18.195189 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:18.088030 25575 leader_election.cc:290] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:18.090261 25131 leader_election.cc:290] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 24 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:18.119230 25781 raft_consensus.cc:3060] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:18.169126 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:18.204146 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 23 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:24:18.192031 17803 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:18.088855 25784 raft_consensus.cc:515] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:18.198565 17757 raft_consensus.cc:1240] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 25. Current term is 26. Ops: []
I20260709 12:24:18.199258 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:18.200150 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:18.205330 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:18.128610 25957 raft_consensus.cc:493] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:24:18.205341 17762 raft_consensus.cc:2468] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 23.
I20260709 12:24:18.204272 17760 raft_consensus.cc:2468] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 24.
I20260709 12:24:18.220790 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:18.221480 17756 raft_consensus.cc:3060] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:18.224545 17689 raft_consensus.cc:2468] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 17.
I20260709 12:24:18.226147 17802 leader_election.cc:304] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:24:18.193271 25637 raft_consensus.cc:2804] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:24:18.195363 25862 raft_consensus.cc:3060] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Advancing to term 23
W20260709 12:24:18.234818 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 147117us
I20260709 12:24:18.193482 25854 raft_consensus.cc:493] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:18.193372 25264 raft_consensus.cc:493] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:18.241020 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:18.216665 25912 raft_consensus.cc:493] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:18.249516 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 21 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:18.217370 25957 raft_consensus.cc:3060] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Advancing to term 20
W20260709 12:24:18.255654 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 249767us
I20260709 12:24:18.234941 25890 raft_consensus.cc:2804] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:24:18.261931 17831 raft_consensus.cc:1240] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 25. Current term is 26. Ops: [25.14-25.14]
W20260709 12:24:18.263823 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:18.264518 17652 leader_election.cc:304] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:18.237758 25767 raft_consensus.cc:493] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:18.269757 25914 raft_consensus.cc:515] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:18.272125 17758 raft_consensus.cc:1240] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 19. Current term is 20. Ops: []
I20260709 12:24:18.273272 25914 leader_election.cc:290] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 20 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:18.278211 17756 raft_consensus.cc:2468] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 23.
I20260709 12:24:18.282794 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:18.245200 25637 raft_consensus.cc:697] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 19 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:24:18.285616 17756 raft_consensus.cc:1240] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 23. Current term is 24. Ops: []
I20260709 12:24:18.280587 17652 leader_election.cc:304] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:18.222739 25954 raft_consensus.cc:515] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:18.291651 17834 raft_consensus.cc:2393] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 23: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
W20260709 12:24:18.293121 17803 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:18.295667 17652 leader_election.cc:304] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:18.306015 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 21 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:18.309178 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 20 candidate_status { last_received { term: 19 index: 17 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:18.279269 25575 raft_consensus.cc:2749] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:24:18.243384 25264 raft_consensus.cc:515] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:18.300819 18333 leader_election.cc:304] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:18.295930 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:18.301545 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:18.316932 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:18.320511 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 22 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:18.307724 17680 raft_consensus.cc:2468] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 20.
I20260709 12:24:18.327121 17684 raft_consensus.cc:3060] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Advancing to term 24
I20260709 12:24:18.281260 25890 raft_consensus.cc:697] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 17 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:24:18.324715 17755 raft_consensus.cc:1240] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 23. Current term is 24. Ops: []
I20260709 12:24:18.245523 25854 raft_consensus.cc:515] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:18.334306 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:18.335165 17685 raft_consensus.cc:2468] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 22.
I20260709 12:24:18.341080 25954 leader_election.cc:290] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 24 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:18.342661 17762 raft_consensus.cc:2410] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 21 because replica has last-logged OpId of term: 19 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 14.
I20260709 12:24:18.338490 17683 raft_consensus.cc:2468] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 15.
I20260709 12:24:18.348861 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:18.333999 18333 leader_election.cc:304] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:18.265053 25912 raft_consensus.cc:515] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:18.353668 17652 leader_election.cc:304] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:18.360338 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:18.284413 25767 raft_consensus.cc:515] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:18.370926 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:18.336644 25917 raft_consensus.cc:2764] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Leader pre-election decision vote started in defunct term 22: won
I20260709 12:24:18.383407 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:18.384419 17683 raft_consensus.cc:3060] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:18.385864 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:18.366544 17801 leader_election.cc:304] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:18.341118 25854 leader_election.cc:290] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:18.257794 25901 consensus_queue.cc:260] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 25.14, Last appended by leader: 14, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:18.388499 17654 leader_election.cc:304] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:24:18.395002 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 24 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:18.341567 25890 consensus_queue.cc:237] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 15.16, Last appended by leader: 16, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:18.398610 17757 raft_consensus.cc:2468] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 25.
I20260709 12:24:18.294483 25958 consensus_queue.cc:1048] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.098s
I20260709 12:24:18.405817 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:18.409453 17831 raft_consensus.cc:2468] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 20.
I20260709 12:24:18.363788 25264 leader_election.cc:290] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:18.367414 25731 leader_election.cc:290] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:18.385325 25131 raft_consensus.cc:493] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:18.395371 25915 raft_consensus.cc:2749] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:24:18.396662 25688 raft_consensus.cc:2804] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Leader election won for term 23
W20260709 12:24:18.407044 25765 consensus_meta.cc:287] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.501s	user 0.011s	sys 0.000s
W20260709 12:24:18.428153 25768 consensus_meta.cc:287] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.507s	user 0.004s	sys 0.009s
I20260709 12:24:18.404057 25868 consensus_queue.cc:1059] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 23.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.039s
I20260709 12:24:18.492291 17683 raft_consensus.cc:2468] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 26.
I20260709 12:24:18.517185 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:18.502200 17757 raft_consensus.cc:1275] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 24 index: 16. Preceding OpId from leader: term: 25 index: 17. (index mismatch)
I20260709 12:24:18.451845 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 23 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:18.455638 17684 raft_consensus.cc:2468] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 24.
I20260709 12:24:18.516044 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:24:18.554983 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 197365us
I20260709 12:24:18.556818 17756 raft_consensus.cc:2410] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 23 because replica has last-logged OpId of term: 22 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 12.
I20260709 12:24:18.557114 17758 raft_consensus.cc:2468] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 24.
I20260709 12:24:18.489110 17801 leader_election.cc:304] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:24:18.431545 25761 consensus_meta.cc:287] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.515s	user 0.008s	sys 0.000s
I20260709 12:24:18.557973 17652 leader_election.cc:304] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:24:18.457700 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 130292us
I20260709 12:24:18.560150 17763 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38776 (request call id 2940) took 1011 ms. Trace:
W20260709 12:24:18.599848 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 110811us
W20260709 12:24:18.433492 25918 consensus_meta.cc:287] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.584s	user 0.013s	sys 0.000s
W20260709 12:24:18.451367 25963 log.cc:927] Time spent T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.100s	user 0.000s	sys 0.000s
W20260709 12:24:18.608115 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 50246us
W20260709 12:24:18.601692 17803 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:18.433144 25912 leader_election.cc:290] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:18.613377 17652 leader_election.cc:304] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:24:18.455986 25926 log.cc:927] Time spent T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.055s	user 0.000s	sys 0.000s
W20260709 12:24:18.456223 25961 log.cc:927] Time spent T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.261s	user 0.003s	sys 0.000s
I20260709 12:24:18.433101 25951 raft_consensus.cc:2804] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:24:18.424357 25777 raft_consensus.cc:2804] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:24:18.608322 17801 leader_election.cc:304] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:24:18.627949 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 239565us
I20260709 12:24:18.414130 25858 consensus_queue.cc:1059] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.181s
W20260709 12:24:18.462249 25964 log.cc:927] Time spent T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.079s	user 0.002s	sys 0.000s
W20260709 12:24:18.613922 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 248159us
I20260709 12:24:18.593990 17763 rpcz_store.cc:276] 0709 12:24:17.548310 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:17.549911 (+  1601us) service_pool.cc:224] Handling call
0709 12:24:17.550234 (+   323us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:17.759489 (+209255us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 12:24:17.766435 (+  6946us) log.cc:844] Serialized 14 byte log entry
0709 12:24:17.920479 (+154044us) log.cc:844] Serialized 14 byte log entry
0709 12:24:18.003372 (+ 82893us) log.cc:844] Serialized 14 byte log entry
0709 12:24:18.003535 (+   163us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:18.003578 (+    43us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:24:18.003647 (+    69us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:18.533234 (+529587us) spinlock_profiling.cc:243] Waited 209 ms on lock 0x7b540019fad4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:24:18.533301 (+    67us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:18.560066 (+ 26765us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":34911,"prepare.queue_time_us":12306,"prepare.run_cpu_time_us":154,"prepare.run_wall_time_us":4890,"raft.queue_time_us":154499,"raft.run_cpu_time_us":147,"raft.run_wall_time_us":147,"spinlock_wait_cycles":486160128,"thread_start_us":219317,"threads_started":2,"wal-append.queue_time_us":242340}
I20260709 12:24:18.432984 25784 leader_election.cc:290] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:18.481279 25721 leader_election.cc:290] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:18.416924 25901 raft_consensus.cc:3060] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:18.450589 25953 raft_consensus.cc:2749] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:24:18.614785 25959 raft_consensus.cc:493] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:18.433974 25131 raft_consensus.cc:515] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:18.617059 18333 leader_election.cc:304] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:24:18.599601 25688 raft_consensus.cc:697] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 23 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:24:18.592338 25768 raft_consensus.cc:515] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:18.423943 25637 consensus_queue.cc:237] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.14, Last appended by leader: 14, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:18.617234 17652 leader_election.cc:304] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:18.443003 25767 leader_election.cc:290] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:18.431629 25957 raft_consensus.cc:515] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:18.432605 25886 raft_consensus.cc:2804] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:24:18.502641 25866 consensus_queue.cc:1059] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.114s
I20260709 12:24:18.593837 25761 raft_consensus.cc:515] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:18.612485 25674 raft_consensus.cc:2804] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:24:18.674851 17761 raft_consensus.cc:2393] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 20: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:24:18.616672 25967 raft_consensus.cc:2804] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:24:18.645949 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:18.647804 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:18.677840 18364 raft_consensus.cc:2468] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 20.
I20260709 12:24:18.647845 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 24 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:18.650347 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:18.649288 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:18.629045 25951 raft_consensus.cc:493] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:18.688272 17764 raft_consensus.cc:1240] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 19. Current term is 20. Ops: []
I20260709 12:24:18.656009 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:18.518586 25884 raft_consensus.cc:515] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:18.644266 25868 raft_consensus.cc:2804] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:24:18.641685 25915 raft_consensus.cc:493] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:18.613113 25918 raft_consensus.cc:515] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:18.644119 25952 raft_consensus.cc:3055] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:24:18.760849 17680 raft_consensus.cc:2393] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 19: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:24:18.671746 17755 raft_consensus.cc:1275] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 18 index: 12. Preceding OpId from leader: term: 20 index: 13. (index mismatch)
W20260709 12:24:18.672286 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 55145.5us
I20260709 12:24:18.643162 18365 raft_consensus.cc:1275] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 22 index: 16. Preceding OpId from leader: term: 24 index: 17. (index mismatch)
I20260709 12:24:18.678947 18361 raft_consensus.cc:3060] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:18.679046 18362 raft_consensus.cc:3060] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:18.660904 25777 raft_consensus.cc:697] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 26 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:24:18.665657 25886 raft_consensus.cc:493] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:18.674891 25674 raft_consensus.cc:493] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:18.716055 18374 raft_consensus.cc:2468] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 24.
I20260709 12:24:18.695787 17729 leader_election.cc:304] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:24:18.740483 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 140202us
I20260709 12:24:18.744458 25909 consensus_queue.cc:1048] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.079s
I20260709 12:24:18.648429 25959 raft_consensus.cc:515] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:18.746239 25969 log.cc:927] Time spent T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.129s	user 0.000s	sys 0.000s
I20260709 12:24:18.414012 25944 consensus_queue.cc:1059] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.292s
W20260709 12:24:18.793427 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 97733.6us
I20260709 12:24:18.688114 25953 raft_consensus.cc:2804] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:24:18.676623 25967 raft_consensus.cc:493] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:18.692377 25951 raft_consensus.cc:3060] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:18.692296 25917 raft_consensus.cc:2804] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:24:18.716945 25868 raft_consensus.cc:493] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:18.670605 25826 consensus_queue.cc:1048] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.067s
I20260709 12:24:18.849802 25957 leader_election.cc:290] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 20 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:18.850001 25868 raft_consensus.cc:3060] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Advancing to term 25
W20260709 12:24:18.806578 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 401444us
I20260709 12:24:18.818825 18362 raft_consensus.cc:2468] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 20.
W20260709 12:24:18.857887 25862 consensus_meta.cc:287] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.616s	user 0.010s	sys 0.001s
I20260709 12:24:18.819952 17760 raft_consensus.cc:1240] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 23 LEADER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 22. Current term is 23. Ops: [22.17-22.17]
I20260709 12:24:18.858182 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:18.859280 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:18.861524 25918 leader_election.cc:290] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 19 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:18.861821 17835 raft_consensus.cc:1275] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 22 index: 16. Preceding OpId from leader: term: 24 index: 17. (index mismatch)
I20260709 12:24:18.862941 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 21 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:18.863687 17838 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 20.
I20260709 12:24:18.822270 25953 raft_consensus.cc:493] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:18.864666 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:18.791179 25886 raft_consensus.cc:3060] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:18.796787 25674 raft_consensus.cc:3060] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:18.871341 17757 raft_consensus.cc:1240] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 23. Current term is 24. Ops: [23.14-23.14]
I20260709 12:24:18.757827 25915 raft_consensus.cc:515] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:18.874398 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:18.804212 25954 raft_consensus.cc:2804] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Leader pre-election won for term 21
W20260709 12:24:18.882164 17803 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:18.835711 25972 raft_consensus.cc:3055] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:24:18.837955 25881 leader_election.cc:290] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:18.864976 25953 raft_consensus.cc:3060] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:18.841380 17837 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35040 (request call id 3240) took 1335 ms. Trace:
I20260709 12:24:18.898025 17837 rpcz_store.cc:276] 0709 12:24:17.506432 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:17.526471 (+ 20039us) service_pool.cc:224] Handling call
0709 12:24:17.543119 (+ 16648us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:17.543643 (+   524us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:17.543689 (+    46us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:18.546155 (+1002466us) spinlock_profiling.cc:243] Waited 8.85 ms on lock 0x7b2c000472a8. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b48f19a11 00007f7b48f29372 00007f7b4a65ba01 00007f7b4a62043a 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2
0709 12:24:18.592318 (+ 46163us) log.cc:844] Serialized 26 byte log entry
0709 12:24:18.674827 (+ 82509us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:24:18.674861 (+    34us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:18.674946 (+    85us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:18.841073 (+166127us) raft_consensus.cc:1685] finished
0709 12:24:18.841094 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:18.841320 (+   226us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":46904,"raft.queue_time_us":554,"raft.run_cpu_time_us":70,"raft.run_wall_time_us":70,"spinlock_wait_cycles":20365568,"thread_start_us":55271,"threads_started":1,"wal-append.queue_time_us":227985}
W20260709 12:24:18.842787 17804 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:18.843297 25131 leader_election.cc:290] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:18.843591 17801 leader_election.cc:304] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:18.845659 25884 leader_election.cc:290] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 24 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:18.849260 25866 raft_consensus.cc:3055] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:24:18.760208 25952 raft_consensus.cc:740] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
W20260709 12:24:18.866545 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 192543us
W20260709 12:24:18.869158 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 66369.5us
I20260709 12:24:18.827837 25967 raft_consensus.cc:3060] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:18.885133 17761 raft_consensus.cc:1275] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 25 index: 19. Preceding OpId from leader: term: 27 index: 21. (index mismatch)
I20260709 12:24:18.828961 25767 consensus_queue.cc:1048] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.062s
W20260709 12:24:18.887593 17655 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:18.829465 25970 raft_consensus.cc:2749] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 12:24:18.838317 25777 consensus_queue.cc:237] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 24.15, Last appended by leader: 15, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:18.896610 25901 consensus_queue.cc:1059] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.036s
I20260709 12:24:18.933043 25952 consensus_queue.cc:260] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 20.12, Last appended by leader: 12, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:18.934338 25952 raft_consensus.cc:3060] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:18.935570 17689 raft_consensus.cc:1240] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 25. Current term is 26. Ops: []
I20260709 12:24:18.872201 25979 raft_consensus.cc:493] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:18.940304 25979 raft_consensus.cc:515] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:18.828141 25958 consensus_queue.cc:1059] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.061s
I20260709 12:24:18.924717 25881 raft_consensus.cc:493] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:18.956154 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:18.956266 25881 raft_consensus.cc:515] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:18.959084 25881 leader_election.cc:290] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:18.936913 18333 leader_election.cc:304] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:24:18.937608 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
W20260709 12:24:18.938021 25781 consensus_meta.cc:287] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.717s	user 0.011s	sys 0.002s
I20260709 12:24:18.937902 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:18.938670 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:18.837978 25739 leader_election.cc:290] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:24:18.924511 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 80986.1us
I20260709 12:24:18.916966 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:18.919368 17835 raft_consensus.cc:1275] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 12 index: 9. Preceding OpId from leader: term: 20 index: 13. (index mismatch)
I20260709 12:24:18.943209 18365 raft_consensus.cc:1275] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 21 index: 16. Preceding OpId from leader: term: 23 index: 17. (index mismatch)
I20260709 12:24:18.928676 18366 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59886 (request call id 2104) took 1193 ms. Trace:
I20260709 12:24:18.945855 17729 leader_election.cc:304] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:19.026623 18366 rpcz_store.cc:276] 0709 12:24:17.756105 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:17.757428 (+  1323us) service_pool.cc:224] Handling call
0709 12:24:17.811127 (+ 53699us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:18.068359 (+257232us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:18.928393 (+860034us) spinlock_profiling.cc:243] Waited 256 ms on lock 0x7b54006aed54. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:24:18.928611 (+   218us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":589680384}
W20260709 12:24:19.027416 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 81604.5us
I20260709 12:24:19.028195 17729 leader_election.cc:304] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:18.948156 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 21 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:19.028913 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:19.029860 18362 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 20.
I20260709 12:24:18.880821 25954 raft_consensus.cc:493] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:18.952680 17689 raft_consensus.cc:1240] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 23. Current term is 24. Ops: []
I20260709 12:24:18.954587 25979 leader_election.cc:290] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:18.827304 25688 consensus_queue.cc:237] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 21.15, Last appended by leader: 15, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:18.924005 25915 leader_election.cc:290] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:18.896068 25991 raft_consensus.cc:493] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:24:18.889432 25972 raft_consensus.cc:740] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:24:18.961683 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:18.962603 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:18.923403 25951 raft_consensus.cc:515] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:18.904619 25384 consensus_queue.cc:1048] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.074s
I20260709 12:24:18.980654 25740 raft_consensus.cc:3055] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 23 LEADER]: Stepping down as leader of term 23
W20260709 12:24:19.082368 17803 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:18.983431 18361 raft_consensus.cc:2468] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 24.
I20260709 12:24:18.949360 25681 raft_consensus.cc:2749] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
W20260709 12:24:18.992619 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 51449.5us
I20260709 12:24:18.990607 25868 raft_consensus.cc:515] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:18.945777 17727 leader_election.cc:304] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:19.124397 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:19.125659 18361 raft_consensus.cc:2393] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 24: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:24:19.020581 17683 raft_consensus.cc:3060] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:19.004769 18374 raft_consensus.cc:3060] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:19.021631 25918 raft_consensus.cc:493] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:19.021776 25958 consensus_queue.cc:1243] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer e2909cff1da44ad387f96a8ae3aa5afb log is divergent from this leader: its last log entry 18.13 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 12
I20260709 12:24:19.037557 17760 raft_consensus.cc:1240] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 23. Current term is 24. Ops: []
I20260709 12:24:19.044950 17681 raft_consensus.cc:2393] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 24: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:24:19.047292 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:19.137679 17837 raft_consensus.cc:2393] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 19: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:24:18.941452 25866 raft_consensus.cc:740] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 25 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:24:18.992328 25901 consensus_queue.cc:1243] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer e063541542a94969898868bf6eb5ce1b log is divergent from this leader: its last log entry 24.14 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 13
I20260709 12:24:19.056394 17689 raft_consensus.cc:1240] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 18. Current term is 19. Ops: []
I20260709 12:24:19.061399 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:19.149164 17759 raft_consensus.cc:2468] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 24.
I20260709 12:24:19.062578 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:19.062788 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:19.151078 17756 raft_consensus.cc:2393] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 19: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:24:19.065687 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 22 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:18.985853 25959 leader_election.cc:290] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:19.052412 25954 raft_consensus.cc:3060] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:19.077761 25974 raft_consensus.cc:493] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:18.988474 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 21 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:19.081223 25917 raft_consensus.cc:697] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 26 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:24:19.091202 25951 leader_election.cc:290] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 24 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:19.085848 25761 leader_election.cc:290] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 19 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:19.105002 25768 leader_election.cc:290] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 20 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:19.082226 25740 raft_consensus.cc:740] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 23 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:24:19.107725 17728 leader_election.cc:304] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:18.983816 25967 raft_consensus.cc:515] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:24:19.113160 25776 consensus_queue.cc:1059] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 19.15, Next index: 16, Last known committed idx: 14, Time since last communication: 0.000s
I20260709 12:24:19.109026 25890 raft_consensus.cc:3055] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:24:19.116072 25915 raft_consensus.cc:2804] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:24:19.061365 25264 raft_consensus.cc:2804] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:24:19.117014 17729 leader_election.cc:304] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:19.117419 25978 consensus_queue.cc:1048] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 19, Time since last communication: 0.000s
I20260709 12:24:19.068441 25767 raft_consensus.cc:2804] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:24:19.118464 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 23 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:19.118858 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:19.082103 25944 consensus_queue.cc:1059] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 19.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.041s
I20260709 12:24:19.119519 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:19.119596 18364 raft_consensus.cc:1275] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 26 index: 20. Preceding OpId from leader: term: 27 index: 21. (index mismatch)
I20260709 12:24:19.075316 25991 raft_consensus.cc:515] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:19.075177 25979 consensus_queue.cc:1048] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.051s
W20260709 12:24:19.125494 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 179790us
I20260709 12:24:19.018175 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:19.240397 17687 raft_consensus.cc:2468] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 26.
I20260709 12:24:19.030874 25781 raft_consensus.cc:515] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:19.152027 17832 raft_consensus.cc:2468] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 21.
I20260709 12:24:19.153656 18365 raft_consensus.cc:1275] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 18 index: 12. Preceding OpId from leader: term: 20 index: 13. (index mismatch)
I20260709 12:24:19.100600 25674 raft_consensus.cc:515] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:19.173185 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:19.267735 18361 raft_consensus.cc:2468] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 22.
I20260709 12:24:19.123438 25952 consensus_queue.cc:1048] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.020s
I20260709 12:24:19.174654 17830 raft_consensus.cc:2393] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 25: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:24:19.141451 25887 consensus_queue.cc:1048] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.033s
I20260709 12:24:19.153194 25960 raft_consensus.cc:2804] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:24:19.070405 25867 consensus_queue.cc:1059] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.092s
W20260709 12:24:19.178531 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 71025.7us
I20260709 12:24:19.302206 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 22 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:19.302955 17728 leader_election.cc:304] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:19.303154 18365 raft_consensus.cc:2468] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 21.
I20260709 12:24:19.305042 17728 leader_election.cc:304] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:19.184669 18374 raft_consensus.cc:2468] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 20.
W20260709 12:24:19.217667 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 100706us
I20260709 12:24:19.307955 17687 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7"
dest_uuid: "0da169ff82674dc1947055ce43de68a3"
mode: GRACEFUL
new_leader_uuid: "e063541542a94969898868bf6eb5ce1b"
 from {username='slave'} at 127.0.0.1:54540
I20260709 12:24:19.219552 17683 raft_consensus.cc:2468] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 23.
I20260709 12:24:19.308558 17687 raft_consensus.cc:606] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 19 LEADER]: Received request to transfer leadership to TS e063541542a94969898868bf6eb5ce1b
I20260709 12:24:19.309041 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:19.309731 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 21 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:19.310189 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:19.310719 17683 raft_consensus.cc:2376] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 24. Re-sending same reply.
I20260709 12:24:19.310786 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:19.311471 17687 raft_consensus.cc:2393] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 20: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
W20260709 12:24:19.271456 17803 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:19.278895 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 23 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:19.279192 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:19.280097 17686 raft_consensus.cc:1240] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 20. Current term is 21. Ops: []
W20260709 12:24:19.287384 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 47484.6us
I20260709 12:24:19.163952 25731 consensus_queue.cc:1059] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 22.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.056s
I20260709 12:24:19.184428 25264 raft_consensus.cc:493] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:19.184520 25959 raft_consensus.cc:2804] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:24:19.184329 25890 raft_consensus.cc:740] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 23 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:24:19.184314 25915 raft_consensus.cc:493] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:19.184662 25884 raft_consensus.cc:493] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:19.236759 25776 raft_consensus.cc:2804] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:24:19.334513 17761 raft_consensus.cc:2368] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 19. Current term is 20.
I20260709 12:24:19.342722 17727 leader_election.cc:304] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:19.344748 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:19.345526 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:19.175832 25974 raft_consensus.cc:515] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:19.327865 25420 raft_consensus.cc:2804] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:24:19.352965 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:19.353778 18363 raft_consensus.cc:3060] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:19.339594 17831 raft_consensus.cc:2393] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 26: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:24:19.177270 25886 raft_consensus.cc:515] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:19.164842 25918 raft_consensus.cc:515] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:19.286957 25761 raft_consensus.cc:493] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:19.287204 25767 raft_consensus.cc:697] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:24:19.356817 17728 leader_election.cc:304] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:19.331004 25678 raft_consensus.cc:2749] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:24:19.337034 25944 raft_consensus.cc:493] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:19.332511 25959 raft_consensus.cc:493] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:19.332433 25264 raft_consensus.cc:3060] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:19.337257 25915 raft_consensus.cc:3060] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:19.337213 25960 raft_consensus.cc:697] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:24:19.343816 25638 raft_consensus.cc:493] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:19.373987 17728 leader_election.cc:304] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:24:19.374328 17654 leader_election.cc:304] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:24:19.360800 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 423943us
I20260709 12:24:19.216033 25740 consensus_queue.cc:260] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 23.13, Last appended by leader: 13, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:19.380527 18333 leader_election.cc:304] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:24:19.362848 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 310134us
I20260709 12:24:19.364655 17727 leader_election.cc:304] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:19.370158 25959 raft_consensus.cc:3060] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:19.361864 25776 raft_consensus.cc:697] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:24:19.392834 25776 consensus_queue.cc:237] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 22.18, Last appended by leader: 18, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:19.405876 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:19.406715 17756 raft_consensus.cc:3060] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:24:19.362828 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 289487us
I20260709 12:24:19.426630 17685 raft_consensus.cc:1240] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 22. Current term is 23. Ops: []
I20260709 12:24:19.353621 25905 raft_consensus.cc:2749] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
W20260709 12:24:19.429159 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 55300.7us
I20260709 12:24:19.432389 25746 raft_consensus.cc:2749] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:24:19.432595 25952 raft_consensus.cc:2804] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:24:19.433077 25952 raft_consensus.cc:493] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:19.433441 25952 raft_consensus.cc:3060] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:19.433607 25960 consensus_queue.cc:237] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 22.14, Last appended by leader: 14, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:19.436647 25967 leader_election.cc:290] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 23 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:19.349216 25884 raft_consensus.cc:515] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:19.439146 25884 leader_election.cc:290] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:19.376716 25944 raft_consensus.cc:515] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:19.441597 25918 leader_election.cc:290] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:19.441903 25944 leader_election.cc:290] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:24:19.446275 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 81737.7us
I20260709 12:24:19.447723 17682 raft_consensus.cc:2468] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 21.
I20260709 12:24:19.321038 25953 raft_consensus.cc:515] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:19.449101 17801 leader_election.cc:304] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:19.454457 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:19.356900 25867 raft_consensus.cc:493] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
W20260709 12:24:19.461817 17763 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.597s	user 0.015s	sys 0.049s
W20260709 12:24:19.452494 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 74974.8us
I20260709 12:24:19.451781 17680 raft_consensus.cc:2468] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 21.
W20260709 12:24:19.462293 17763 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.597s	user 0.015s	sys 0.049s
W20260709 12:24:19.457428 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 385044us
I20260709 12:24:19.462754 18333 leader_election.cc:304] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:24:19.458495 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 84250.3us
I20260709 12:24:19.365036 25420 raft_consensus.cc:697] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 26 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
W20260709 12:24:19.482941 17803 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:19.486929 17654 leader_election.cc:304] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:19.298695 25917 consensus_queue.cc:237] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 25.17, Last appended by leader: 17, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:19.487012 17829 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.546s	user 0.017s	sys 0.048s
W20260709 12:24:19.545991 17829 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.605s	user 0.017s	sys 0.048s
W20260709 12:24:19.537472 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 50615.4us
W20260709 12:24:19.468572 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 418757us
I20260709 12:24:19.549384 17654 leader_election.cc:304] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
W20260709 12:24:19.558159 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 69060.5us
I20260709 12:24:19.560226 17652 leader_election.cc:304] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:19.563297 17756 raft_consensus.cc:2468] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 24.
I20260709 12:24:19.589121 17652 leader_election.cc:304] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:19.485939 25944 raft_consensus.cc:2804] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:24:19.467069 25884 raft_consensus.cc:2749] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:24:19.468719 25905 raft_consensus.cc:2804] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:24:19.605410 17652 leader_election.cc:304] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:19.486769 25688 raft_consensus.cc:493] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:19.428234 25761 raft_consensus.cc:515] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:19.779443 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:19.796811 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:19.454346 25638 raft_consensus.cc:515] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:19.613067 25767 consensus_queue.cc:237] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.13, Last appended by leader: 13, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:19.783812 17760 raft_consensus.cc:1240] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 19. Current term is 20. Ops: []
I20260709 12:24:19.594509 25951 consensus_queue.cc:1048] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.094s
I20260709 12:24:19.730294 25420 consensus_queue.cc:237] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 24.13, Last appended by leader: 13, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:19.592458 25973 raft_consensus.cc:2749] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
W20260709 12:24:19.729244 25994 log.cc:927] Time spent T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.276s	user 0.001s	sys 0.000s
I20260709 12:24:19.615857 25905 raft_consensus.cc:493] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:19.729820 25886 leader_election.cc:290] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 23 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:19.445490 25663 consensus_queue.cc:1048] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.160s
I20260709 12:24:19.499421 25909 consensus_queue.cc:1048] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.141s
I20260709 12:24:19.621168 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:24:19.721820 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 233577us
I20260709 12:24:19.460593 25890 consensus_queue.cc:260] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 23.19, Last appended by leader: 19, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:19.606868 25944 raft_consensus.cc:493] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:19.593693 25901 raft_consensus.cc:2804] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Leader pre-election won for term 22
W20260709 12:24:19.714964 25997 log.cc:927] Time spent T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.217s	user 0.000s	sys 0.000s
I20260709 12:24:19.731178 25918 raft_consensus.cc:493] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:19.593060 25868 leader_election.cc:290] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 25 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:24:19.815577 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 352867us
W20260709 12:24:19.825917 17804 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:19.828394 18367 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.530s	user 0.007s	sys 0.010s
I20260709 12:24:19.680873 26004 raft_consensus.cc:493] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:19.841125 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:19.885986 25991 leader_election.cc:290] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:19.885548 25967 raft_consensus.cc:493] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:19.830010 26006 consensus_queue.cc:1059] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.215s
I20260709 12:24:19.883514 25688 raft_consensus.cc:515] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:19.876914 25959 raft_consensus.cc:515] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Starting leader election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:24:19.553730 25958 raft_consensus.cc:2749] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 12:24:19.498020 25867 raft_consensus.cc:515] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:19.871985 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 320132us
I20260709 12:24:19.709401 25637 raft_consensus.cc:2749] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260709 12:24:19.795688 25953 leader_election.cc:290] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 24 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:19.936445 18333 leader_election.cc:304] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:19.933254 17802 leader_election.cc:304] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
W20260709 12:24:20.005818 17728 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.16.29.65:33071 timed out after 1.571s (SENT) [suppressed 20 similar messages]
W20260709 12:24:19.953688 18363 consensus_meta.cc:287] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.599s	user 0.011s	sys 0.000s
I20260709 12:24:20.015030 18363 raft_consensus.cc:2468] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 24.
W20260709 12:24:19.869433 17803 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:19.878701 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 273363us
I20260709 12:24:19.797840 25131 raft_consensus.cc:493] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:19.923537 25905 raft_consensus.cc:3060] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:20.025115 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:20.025952 17760 raft_consensus.cc:2368] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for earlier term 23. Current term is 24.
I20260709 12:24:20.026484 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
W20260709 12:24:20.027074 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 90758.2us
I20260709 12:24:19.940196 26004 raft_consensus.cc:515] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:20.028383 25867 leader_election.cc:290] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:20.028997 18333 leader_election.cc:304] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:20.029882 26004 leader_election.cc:290] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:19.933529 25901 raft_consensus.cc:493] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:20.032408 25901 raft_consensus.cc:3060] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:19.709103 25881 consensus_queue.cc:1059] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.168s
I20260709 12:24:19.844705 25674 leader_election.cc:290] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 26 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:19.934177 25918 raft_consensus.cc:515] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:20.035708 26017 raft_consensus.cc:2764] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Leader pre-election decision vote started in defunct term 22: won
I20260709 12:24:19.885874 25858 raft_consensus.cc:2749] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:24:19.920807 17681 raft_consensus.cc:2468] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 26.
W20260709 12:24:19.937397 18367 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.639s	user 0.007s	sys 0.010s
I20260709 12:24:20.038841 25918 leader_election.cc:290] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:20.025379 25131 raft_consensus.cc:515] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:19.933339 25944 raft_consensus.cc:3060] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:20.027256 17832 raft_consensus.cc:2393] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 23: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:24:19.728261 25995 raft_consensus.cc:3060] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:20.036239 26014 raft_consensus.cc:2749] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:24:19.884862 25781 leader_election.cc:290] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 20 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:20.044420 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:20.049137 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:20.051153 26025 raft_consensus.cc:3055] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:24:20.050330 17758 raft_consensus.cc:1275] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 22 index: 14. Preceding OpId from leader: term: 26 index: 16. (index mismatch)
I20260709 12:24:19.933034 25890 raft_consensus.cc:3060] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:20.054865 26009 raft_consensus.cc:493] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:24:20.055330 17652 leader_election.cc:304] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:20.055521 26009 raft_consensus.cc:515] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:20.057228 17832 raft_consensus.cc:1240] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 24 LEADER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 23. Current term is 24. Ops: [23.14-23.14]
I20260709 12:24:20.058240 25736 raft_consensus.cc:3055] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:24:20.058642 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 23 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:19.929308 25970 consensus_queue.cc:1048] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 9, Time since last communication: 0.130s
I20260709 12:24:20.058665 25736 raft_consensus.cc:740] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:24:20.000501 18334 leader_election.cc:304] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:24:20.063265 26009 leader_election.cc:290] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:20.064864 25995 raft_consensus.cc:2749] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:24:20.067379 18366 raft_consensus.cc:2468] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 21.
I20260709 12:24:20.073473 17685 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37402 (request call id 3457) took 1139 ms. Trace:
I20260709 12:24:20.057971 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:20.074106 17685 rpcz_store.cc:276] 0709 12:24:18.933780 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:18.937204 (+  3424us) service_pool.cc:224] Handling call
0709 12:24:18.937480 (+   276us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:19.427458 (+489978us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:20.073156 (+645698us) spinlock_profiling.cc:243] Waited 489 ms on lock 0x7b54004277d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:24:20.073393 (+   237us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1124273024}
I20260709 12:24:20.074605 17760 raft_consensus.cc:2468] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 21.
W20260709 12:24:20.076159 17831 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35042 (request call id 3018) took 1220 ms (client timeout 1579 ms). Trace:
I20260709 12:24:19.985244 25974 leader_election.cc:290] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:24:20.077289 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 76882.6us
I20260709 12:24:20.077762 17838 raft_consensus.cc:1240] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 25. Current term is 26. Ops: []
I20260709 12:24:20.078492 18334 leader_election.cc:304] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:24:20.079442 17838 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35040 (request call id 3259) took 1167 ms. Trace:
I20260709 12:24:20.079995 26014 raft_consensus.cc:2804] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:24:20.080595 26014 raft_consensus.cc:493] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:20.080952 26014 raft_consensus.cc:3060] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:20.081040 26026 raft_consensus.cc:2804] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:24:20.081663 26026 raft_consensus.cc:493] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:20.082018 26026 raft_consensus.cc:3060] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:19.634649 25384 raft_consensus.cc:3060] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Advancing to term 19
W20260709 12:24:19.932471 26001 log.cc:927] Time spent T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.111s	user 0.000s	sys 0.000s
W20260709 12:24:20.076710 17831 rpcz_store.cc:269] 0709 12:24:18.856227 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:18.859008 (+  2781us) service_pool.cc:224] Handling call
0709 12:24:20.075913 (+1216905us) spinlock_profiling.cc:243] Waited 474 ms on lock 0x7b5400335ed4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:24:20.076090 (+   177us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1090315008}
I20260709 12:24:20.087466 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:20.104252 18333 leader_election.cc:304] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:24:20.051640 26025 raft_consensus.cc:740] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 18 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:24:20.100932 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:20.107383 18368 raft_consensus.cc:2393] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 25: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
W20260709 12:24:20.090167 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 156990us
I20260709 12:24:20.112526 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:20.090173 17838 rpcz_store.cc:276] 0709 12:24:18.942387 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:18.996037 (+ 53650us) service_pool.cc:224] Handling call
0709 12:24:20.077526 (+1081489us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:20.078264 (+   738us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:20.079221 (+   957us) spinlock_profiling.cc:243] Waited 1.05 s on lock 0x7b5400335ed0. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a61f8ad 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:24:20.079375 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2420083968}
W20260709 12:24:20.092057 17801 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.603s	user 0.006s	sys 0.014s
W20260709 12:24:20.115242 17801 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.627s	user 0.006s	sys 0.014s
W20260709 12:24:19.943789 26011 log.cc:927] Time spent T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.231s	user 0.000s	sys 0.001s
W20260709 12:24:20.092375 18365 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.649s	user 0.004s	sys 0.016s
I20260709 12:24:20.102164 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 23 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
W20260709 12:24:20.121166 18365 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.678s	user 0.004s	sys 0.016s
I20260709 12:24:20.102663 18363 raft_consensus.cc:2393] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 24: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:24:20.120059 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:20.121529 18362 raft_consensus.cc:3060] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:20.128384 17836 raft_consensus.cc:3060] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:20.118412 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
W20260709 12:24:20.144203 17801 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:20.154294 17836 raft_consensus.cc:2468] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 20.
I20260709 12:24:20.160279 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:20.161000 17834 raft_consensus.cc:2468] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 21.
W20260709 12:24:20.162436 25952 consensus_meta.cc:287] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.729s	user 0.010s	sys 0.000s
I20260709 12:24:20.162879 25952 raft_consensus.cc:515] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:20.105639 25858 raft_consensus.cc:2804] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:24:20.134418 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:19.964483 25967 raft_consensus.cc:515] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:20.005034 26024 consensus_queue.cc:1048] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.175s
I20260709 12:24:20.163825 26002 consensus_queue.cc:1059] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.022s
I20260709 12:24:20.186743 26002 raft_consensus.cc:3055] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:24:20.187134 26002 raft_consensus.cc:740] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:24:20.187968 26002 consensus_queue.cc:260] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 18.15, Last appended by leader: 15, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:20.190232 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:20.093820 25995 raft_consensus.cc:493] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:20.192220 25995 raft_consensus.cc:515] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:20.192864 25881 consensus_queue.cc:1243] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer 0da169ff82674dc1947055ce43de68a3 log is divergent from this leader: its last log entry 22.14 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 12
I20260709 12:24:20.100401 25867 raft_consensus.cc:2804] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:24:20.107511 26025 consensus_queue.cc:260] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.13, Last appended by leader: 13, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:20.193063 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 23 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:20.194602 26025 raft_consensus.cc:3060] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:20.194357 25995 leader_election.cc:290] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:20.197013 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 25 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:20.201728 17683 raft_consensus.cc:1275] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 17 index: 13. Preceding OpId from leader: term: 23 index: 16. (index mismatch)
I20260709 12:24:20.204253 17684 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37402 (request call id 3462) took 1257 ms. Trace:
I20260709 12:24:20.204682 17684 rpcz_store.cc:276] 0709 12:24:18.946737 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:18.946977 (+   240us) service_pool.cc:224] Handling call
0709 12:24:18.947406 (+   429us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:18.992758 (+ 45352us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:18.992787 (+    29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:18.993277 (+   490us) log.cc:844] Serialized 26 byte log entry
0709 12:24:19.376369 (+383092us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:24:19.376411 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:19.376493 (+    82us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:20.204037 (+827544us) raft_consensus.cc:1685] finished
0709 12:24:20.204059 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:20.204196 (+   137us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":44846,"raft.queue_time_us":45406,"raft.run_cpu_time_us":118,"raft.run_wall_time_us":119,"thread_start_us":382497,"threads_started":1,"wal-append.queue_time_us":626761}
I20260709 12:24:20.209343 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 25 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:20.210362 17683 raft_consensus.cc:2468] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 24.
I20260709 12:24:20.170436 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:20.213295 17727 leader_election.cc:304] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:20.176589 17760 raft_consensus.cc:1275] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 21 index: 15. Preceding OpId from leader: term: 23 index: 16. (index mismatch)
I20260709 12:24:20.182080 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:20.088840 17755 raft_consensus.cc:2468] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 24.
I20260709 12:24:20.217598 17801 leader_election.cc:400] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 19 election: Vote denied by peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651) with higher term. Message: Invalid argument: T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 19. Current term is 20.
I20260709 12:24:20.218140 17801 leader_election.cc:403] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 19 election: Cancelling election due to peer responding with higher term
I20260709 12:24:20.221854 17685 raft_consensus.cc:1275] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 15 index: 16. Preceding OpId from leader: term: 17 index: 17. (index mismatch)
I20260709 12:24:20.222576 17652 leader_election.cc:400] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 23 election: Vote denied by peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651) with higher term. Message: Invalid argument: T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for earlier term 23. Current term is 24.
I20260709 12:24:20.222958 17652 leader_election.cc:403] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 23 election: Cancelling election due to peer responding with higher term
I20260709 12:24:20.223147 17831 raft_consensus.cc:1275] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 22 index: 17. Preceding OpId from leader: term: 23 index: 17. (term mismatch)
I20260709 12:24:20.223496 17831 pending_rounds.cc:85] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b: Aborting all ops after (but not including) 16
I20260709 12:24:20.223788 17831 pending_rounds.cc:107] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b: Aborting uncommitted NO_OP operation due to leader change: 22.17
I20260709 12:24:20.224089 17831 raft_consensus.cc:2889] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:24:20.170434 26000 log.cc:927] Time spent T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.223s	user 0.000s	sys 0.001s
W20260709 12:24:20.183807 25954 consensus_meta.cc:287] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 1.008s	user 0.009s	sys 0.001s
I20260709 12:24:20.238447 25954 raft_consensus.cc:515] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:20.239404 17833 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35040 (request call id 3253) took 1335 ms. Trace:
I20260709 12:24:20.240165 17829 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35040 (request call id 3254) took 1336 ms. Trace:
I20260709 12:24:20.185292 25959 leader_election.cc:290] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 24 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:20.240556 17829 rpcz_store.cc:276] 0709 12:24:18.904132 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:18.915676 (+ 11544us) service_pool.cc:224] Handling call
0709 12:24:18.916062 (+   386us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:18.940035 (+ 23973us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:18.940064 (+    29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:18.940738 (+   674us) log.cc:844] Serialized 26 byte log entry
0709 12:24:19.546953 (+606215us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:24:19.546998 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:19.547081 (+    83us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:20.239959 (+692878us) raft_consensus.cc:1685] finished
0709 12:24:20.239980 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:20.240108 (+   128us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":54680,"raft.queue_time_us":25652,"raft.run_cpu_time_us":165,"raft.run_wall_time_us":53246,"thread_start_us":605588,"threads_started":1,"wal-append.queue_time_us":879315}
I20260709 12:24:20.174641 25905 raft_consensus.cc:515] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:20.177706 25967 leader_election.cc:290] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:20.244741 25972 consensus_queue.cc:260] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 23.14, Last appended by leader: 14, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:24:20.180279 25264 consensus_meta.cc:287] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.737s	user 0.010s	sys 0.000s
I20260709 12:24:20.246496 25972 raft_consensus.cc:3060] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:20.247023 25264 raft_consensus.cc:515] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:20.227384 26024 raft_consensus.cc:2804] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:24:20.248139 17757 raft_consensus.cc:1240] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 20. Current term is 21. Ops: [20.13-20.13]
I20260709 12:24:20.248328 26024 raft_consensus.cc:493] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:20.248754 26024 raft_consensus.cc:3060] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:20.250116 17756 raft_consensus.cc:2393] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 25: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:24:20.251708 25264 leader_election.cc:290] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 25 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:20.230854 25867 raft_consensus.cc:697] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 21 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:24:20.256875 25739 raft_consensus.cc:3055] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:24:20.256278 25867 consensus_queue.cc:237] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 20.17, Last appended by leader: 17, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:20.257297 25739 raft_consensus.cc:740] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 19 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:24:20.258315 25739 consensus_queue.cc:260] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 19.13, Last appended by leader: 13, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:20.259275 25739 raft_consensus.cc:3060] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:20.260645 17729 leader_election.cc:304] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:20.260829 26014 raft_consensus.cc:515] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:20.261777 17729 leader_election.cc:304] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:24:20.263172 25740 raft_consensus.cc:3060] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:20.233335 26022 raft_consensus.cc:3055] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:24:20.234660 25688 leader_election.cc:290] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:20.234737 25151 raft_consensus.cc:3055] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:24:20.220887 25862 raft_consensus.cc:493] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:20.221625 25858 raft_consensus.cc:697] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 24 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:24:19.891757 25866 consensus_queue.cc:260] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 25.14, Last appended by leader: 14, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:20.169976 25638 leader_election.cc:290] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:20.237480 25761 leader_election.cc:290] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:20.167747 25944 raft_consensus.cc:515] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:20.264025 26014 leader_election.cc:290] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 21 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:20.264438 25952 leader_election.cc:290] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 26 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:20.265439 25959 raft_consensus.cc:2804] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:24:20.266718 25974 raft_consensus.cc:2804] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Leader election won for term 20
W20260709 12:24:20.267406 25915 consensus_meta.cc:287] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.820s	user 0.010s	sys 0.000s
W20260709 12:24:20.231906 17803 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:20.268729 25912 raft_consensus.cc:493] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:20.272516 17832 raft_consensus.cc:2410] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for term 19 because replica has last-logged OpId of term: 18 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 12.
I20260709 12:24:20.263335 17729 leader_election.cc:304] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:20.286434 25151 raft_consensus.cc:740] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:24:20.281137 26022 raft_consensus.cc:740] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 23 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
W20260709 12:24:20.308538 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 45291.1us
I20260709 12:24:20.289932 25575 raft_consensus.cc:3060] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:24:20.288777 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 133417us
W20260709 12:24:20.292701 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 188520us
I20260709 12:24:20.277179 17833 rpcz_store.cc:276] 0709 12:24:18.903790 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:18.912260 (+  8470us) service_pool.cc:224] Handling call
0709 12:24:18.912703 (+   443us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:18.912955 (+   252us) raft_consensus.cc:1534] Early marking committed up to index 17
0709 12:24:19.344485 (+431530us) log.cc:844] Serialized 14 byte log entry
0709 12:24:20.238670 (+894185us) spinlock_profiling.cc:243] Waited 29.5 ms on lock 0x7b180004bca0. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b4a1c84bc 00007f7b4751648b 00007f7b503709a9 00007f7b4753269b 00007f7b4753093f 00007f7b47532c02 00007f7b4a1ba20b 00007f7b4a1bfd23 00007f7b4a1c023a
0709 12:24:20.238930 (+   260us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:20.238998 (+    68us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:24:20.239075 (+    77us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:20.239163 (+    88us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:20.239327 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":768184,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":37,"raft.queue_time_us":14971,"raft.run_cpu_time_us":66,"raft.run_wall_time_us":66,"spinlock_wait_cycles":67743104,"thread_start_us":874045,"threads_started":2,"wal-append.queue_time_us":688577}
W20260709 12:24:20.298169 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 152999us
I20260709 12:24:20.279778 18362 raft_consensus.cc:2468] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 23.
I20260709 12:24:20.287251 25784 raft_consensus.cc:3060] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:20.291057 25688 raft_consensus.cc:493] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:20.313369 17802 leader_election.cc:304] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:24:20.310737 17652 leader_election.cc:304] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:20.332638 25868 raft_consensus.cc:2804] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:24:20.333201 17727 leader_election.cc:304] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:20.333210 25868 raft_consensus.cc:493] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:20.334048 25868 raft_consensus.cc:3060] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:20.336751 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:20.337195 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:20.337903 17835 raft_consensus.cc:3060] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:20.337885 17833 raft_consensus.cc:2468] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 18.
I20260709 12:24:20.217213 25901 raft_consensus.cc:515] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:20.337251 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:20.340669 17681 raft_consensus.cc:2393] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 21: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:24:20.341536 25901 leader_election.cc:290] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 22 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:20.343446 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:20.344098 17763 raft_consensus.cc:2468] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 26.
I20260709 12:24:20.344939 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:20.346274 25909 raft_consensus.cc:2804] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:24:20.346665 25909 raft_consensus.cc:493] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:20.347026 25909 raft_consensus.cc:3060] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:20.347462 25868 raft_consensus.cc:515] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:20.352077 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:20.352990 17831 raft_consensus.cc:2393] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 25: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:24:20.351464 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 21 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:20.352680 25978 consensus_queue.cc:1059] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 0.0, Next index: 17, Last known committed idx: 14, Time since last communication: 0.000s
I20260709 12:24:20.347101 17835 raft_consensus.cc:2468] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 24.
I20260709 12:24:20.353965 25826 raft_consensus.cc:3055] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:24:20.354437 25826 raft_consensus.cc:740] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
W20260709 12:24:20.351415 17680 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.556s	user 0.017s	sys 0.016s
I20260709 12:24:20.355438 25826 consensus_queue.cc:260] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 25.17, Last appended by leader: 17, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:20.356985 25826 raft_consensus.cc:3060] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Advancing to term 26
W20260709 12:24:20.355827 17680 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.560s	user 0.018s	sys 0.016s
I20260709 12:24:20.357414 25979 raft_consensus.cc:993] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3: : Instructing follower e063541542a94969898868bf6eb5ce1b to start an election
I20260709 12:24:20.357895 25979 raft_consensus.cc:1081] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 19 LEADER]: Signalling peer e063541542a94969898868bf6eb5ce1b to start an election
I20260709 12:24:20.355201 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:20.360073 17652 leader_election.cc:304] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:24:20.363309 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 22 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:20.365177 25901 raft_consensus.cc:2804] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:24:20.365710 25901 raft_consensus.cc:493] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:24:20.366176 25901 raft_consensus.cc:3060] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:20.368150 17833 raft_consensus.cc:3060] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:20.383306 17831 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7"
dest_uuid: "e063541542a94969898868bf6eb5ce1b"
 from {username='slave'} at 127.0.0.1:35040
I20260709 12:24:20.384447 17831 raft_consensus.cc:493] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:24:20.384987 17831 raft_consensus.cc:3060] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:20.298916 25384 raft_consensus.cc:2749] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
W20260709 12:24:20.395612 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 65101.7us
W20260709 12:24:20.394447 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 82594.5us
W20260709 12:24:20.400213 17652 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:20.400882 17759 raft_consensus.cc:1240] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 LEADER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 19. Current term is 20. Ops: [19.13-19.13]
I20260709 12:24:20.301081 26006 raft_consensus.cc:2749] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Leader election lost for term 19. Reason: Vote denied by peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651) with higher term. Message: Invalid argument: T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 19. Current term is 20.
I20260709 12:24:20.405983 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:20.406807 18362 raft_consensus.cc:3060] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:20.292658 26026 raft_consensus.cc:515] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:20.294899 25881 consensus_queue.cc:1059] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.101s
I20260709 12:24:20.295742 25862 raft_consensus.cc:515] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:20.304108 25959 raft_consensus.cc:493] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:20.296626 25890 consensus_queue.cc:1048] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 13, Time since last communication: 0.100s
I20260709 12:24:20.421380 18362 raft_consensus.cc:2468] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 20.
I20260709 12:24:20.423041 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:20.423753 18362 raft_consensus.cc:3060] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:20.330250 25264 raft_consensus.cc:2749] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 12:24:20.426544 25881 raft_consensus.cc:3055] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:24:20.427182 26026 leader_election.cc:290] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 16 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:20.399360 25905 leader_election.cc:290] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 21 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:20.430644 18333 leader_election.cc:304] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:20.432848 17833 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 21.
I20260709 12:24:20.477146 25909 raft_consensus.cc:515] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:20.394702 25915 raft_consensus.cc:515] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:20.417408 25974 raft_consensus.cc:697] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 20 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:24:20.396517 25912 raft_consensus.cc:515] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:20.396206 25688 raft_consensus.cc:515] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:20.489887 25868 leader_election.cc:290] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 25 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:24:20.472672 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 466849us
I20260709 12:24:20.438057 25917 raft_consensus.cc:493] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:20.477133 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 21 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:20.477833 17831 raft_consensus.cc:515] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:20.486481 18362 raft_consensus.cc:2468] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 25.
W20260709 12:24:20.491295 26034 log.cc:927] Time spent T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.073s	user 0.000s	sys 0.000s
I20260709 12:24:20.395195 25761 consensus_queue.cc:1048] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 13, Time since last communication: 0.034s
I20260709 12:24:20.430409 25151 consensus_queue.cc:260] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 19.13, Last appended by leader: 13, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:20.431995 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 21 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:20.505184 25862 leader_election.cc:290] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:20.467911 25858 consensus_queue.cc:237] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 22.17, Last appended by leader: 17, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:20.424757 25959 raft_consensus.cc:3060] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Advancing to term 23
W20260709 12:24:20.553725 18361 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.822s	user 0.000s	sys 0.030s
W20260709 12:24:20.549962 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 119388us
I20260709 12:24:20.514768 25944 leader_election.cc:290] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 24 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:20.582261 18366 raft_consensus.cc:3060] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:20.594403 18333 leader_election.cc:304] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:24:20.595743 18361 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.864s	user 0.000s	sys 0.032s
I20260709 12:24:20.585352 17835 raft_consensus.cc:2393] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 26: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:24:20.596804 18361 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59886 (request call id 2131) took 1197 ms. Trace:
I20260709 12:24:20.597880 17831 leader_election.cc:290] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 20 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:20.597939 18364 raft_consensus.cc:1240] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 24. Current term is 25. Ops: []
I20260709 12:24:20.597295 18361 rpcz_store.cc:276] 0709 12:24:19.399331 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:19.402277 (+  2946us) service_pool.cc:224] Handling call
0709 12:24:19.402768 (+   491us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:19.705752 (+302984us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:19.705800 (+    48us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:19.730897 (+ 25097us) log.cc:844] Serialized 26 byte log entry
0709 12:24:20.596346 (+865449us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:24:20.596407 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:20.596519 (+   112us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:20.596583 (+    64us) raft_consensus.cc:1685] finished
0709 12:24:20.596601 (+    18us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:20.596740 (+   139us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":228566,"raft.run_cpu_time_us":181,"raft.run_wall_time_us":34077,"thread_start_us":1167399,"threads_started":2,"wal-append.queue_time_us":759449}
I20260709 12:24:20.518131 25881 raft_consensus.cc:740] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 22 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:24:20.615895 18366 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 21.
I20260709 12:24:20.555065 26026 raft_consensus.cc:2749] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:24:20.615780 18333 leader_election.cc:304] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:20.528784 25826 raft_consensus.cc:493] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:24:20.629164 17688 raft_consensus.cc:1275] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 23 index: 14. Preceding OpId from leader: term: 24 index: 14. (term mismatch)
I20260709 12:24:20.597709 26018 raft_consensus.cc:2749] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:24:20.632068 17688 pending_rounds.cc:85] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3: Aborting all ops after (but not including) 13
I20260709 12:24:20.633980 17688 pending_rounds.cc:107] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3: Aborting uncommitted NO_OP operation due to leader change: 23.14
I20260709 12:24:20.634271 17688 raft_consensus.cc:2889] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:20.603229 25688 leader_election.cc:290] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:20.610284 25151 raft_consensus.cc:3060] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:20.599931 25917 raft_consensus.cc:515] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:20.633498 25638 raft_consensus.cc:2749] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:24:20.649076 26022 consensus_queue.cc:260] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 23.14, Last appended by leader: 14, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:20.650527 25901 raft_consensus.cc:515] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:20.656344 17686 raft_consensus.cc:1240] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 19. Current term is 20. Ops: []
I20260709 12:24:20.656947 25951 raft_consensus.cc:493] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:24:20.657347 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:20.657518 25951 raft_consensus.cc:515] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:20.658823 25972 raft_consensus.cc:493] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:20.650570 26022 raft_consensus.cc:3060] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:20.660292 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 20 candidate_status { last_received { term: 19 index: 17 } } ignore_live_leader: true dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:20.661051 17687 raft_consensus.cc:3055] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:24:20.661425 17687 raft_consensus.cc:740] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:24:20.629011 25784 raft_consensus.cc:2749] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:24:20.663605 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 21 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:20.662875 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 22 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:20.660324 17728 leader_election.cc:304] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:20.664629 17686 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 21.
I20260709 12:24:20.664273 18333 leader_election.cc:304] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:24:20.656991 17683 raft_consensus.cc:1240] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 22. Current term is 23. Ops: [22.13-22.13]
I20260709 12:24:20.665820 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:20.666368 17686 raft_consensus.cc:3060] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:20.666699 17683 raft_consensus.cc:1275] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 25 index: 17. Preceding OpId from leader: term: 26 index: 18. (index mismatch)
W20260709 12:24:20.666913 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 267223us
I20260709 12:24:20.656379 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:20.667795 17683 raft_consensus.cc:1240] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 18. Current term is 19. Ops: []
I20260709 12:24:20.669332 17801 leader_election.cc:304] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:20.669813 17682 raft_consensus.cc:2393] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 24: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:24:20.669415 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 20 candidate_status { last_received { term: 19 index: 17 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:20.662915 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 25 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:20.672493 17681 raft_consensus.cc:3060] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:20.659346 26024 raft_consensus.cc:515] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:20.677718 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 27 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:20.678839 17684 raft_consensus.cc:2410] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 27 because replica has last-logged OpId of term: 25 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 13.
I20260709 12:24:20.659689 25951 leader_election.cc:290] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:20.665985 17687 consensus_queue.cc:260] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 19.17, Last appended by leader: 17, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:20.682689 17654 leader_election.cc:304] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:20.686178 18333 leader_election.cc:304] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:20.688045 17687 raft_consensus.cc:3060] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Advancing to term 20
W20260709 12:24:20.689721 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:20.694687 17681 raft_consensus.cc:2468] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 25.
I20260709 12:24:20.690182 25954 leader_election.cc:290] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 21 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:20.635824 25826 raft_consensus.cc:515] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:20.678654 25944 raft_consensus.cc:2749] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:24:20.697387 17654 leader_election.cc:304] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:24:20.694357 25638 raft_consensus.cc:2804] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:24:20.659523 25972 raft_consensus.cc:515] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:20.699684 18333 leader_election.cc:304] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:20.701298 25638 raft_consensus.cc:493] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:20.701442 25967 raft_consensus.cc:3060] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:20.701785 25638 raft_consensus.cc:3060] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:20.704008 26018 raft_consensus.cc:2749] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:24:20.687476 26017 raft_consensus.cc:2804] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:24:20.713770 26017 raft_consensus.cc:493] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:20.714689 26017 raft_consensus.cc:3060] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:20.708751 25979 raft_consensus.cc:2804] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:24:20.712649 25575 raft_consensus.cc:2749] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Leader election lost for term 23. Reason: Vote denied by peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651) with higher term. Message: Invalid argument: T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for earlier term 23. Current term is 24.
I20260709 12:24:20.713122 17686 raft_consensus.cc:2468] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 16.
I20260709 12:24:20.713937 25912 leader_election.cc:290] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:24:20.717159 26041 log.cc:927] Time spent T 00000000000000000000000000000000 P 2193af31cb894ec9b3d65d939a2a6851: Append to log took a long time: real 0.134s	user 0.000s	sys 0.001s
I20260709 12:24:20.716228 25979 raft_consensus.cc:697] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 23 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:24:20.711921 25972 leader_election.cc:290] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:20.718756 25979 consensus_queue.cc:237] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 20.12, Last appended by leader: 12, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:20.719182 25638 raft_consensus.cc:515] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:20.720477 25915 leader_election.cc:290] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 25 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:20.721374 25909 leader_election.cc:290] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 22 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:20.722288 17687 raft_consensus.cc:2468] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 20.
I20260709 12:24:20.715626 26024 leader_election.cc:290] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 18 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:20.636523 25881 consensus_queue.cc:260] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 22.13, Last appended by leader: 13, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:20.728497 25881 raft_consensus.cc:3060] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:20.729183 25826 leader_election.cc:290] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:20.729734 25638 leader_election.cc:290] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 20 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:20.665668 25974 consensus_queue.cc:237] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 19.13, Last appended by leader: 13, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:20.742771 18333 leader_election.cc:304] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:20.747098 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 21 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:24:20.757223 26050 log.cc:927] Time spent T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.119s	user 0.000s	sys 0.001s
I20260709 12:24:20.730108 25736 consensus_queue.cc:260] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 21.16, Last appended by leader: 16, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:20.759416 25131 leader_election.cc:290] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:20.779038 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:20.784451 25740 consensus_queue.cc:1048] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:24:20.760447 25678 raft_consensus.cc:2749] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:24:20.777460 25424 raft_consensus.cc:2804] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:24:20.778751 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 25 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:20.830546 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:20.830967 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:20.831324 17833 raft_consensus.cc:3060] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:20.779469 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
W20260709 12:24:20.776785 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 116537us
I20260709 12:24:20.777757 25890 raft_consensus.cc:493] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:20.777338 25957 raft_consensus.cc:493] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:20.769667 25952 raft_consensus.cc:493] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:20.760452 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 427277us
I20260709 12:24:20.854871 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:20.785233 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:20.790288 17762 raft_consensus.cc:1275] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 22 index: 13. Preceding OpId from leader: term: 24 index: 14. (index mismatch)
I20260709 12:24:20.857267 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:20.779646 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 27 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:20.857856 17830 raft_consensus.cc:3060] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:20.857879 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:20.858568 17763 raft_consensus.cc:2410] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 27 because replica has last-logged OpId of term: 25 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 13.
I20260709 12:24:20.858776 17764 raft_consensus.cc:3060] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:20.800783 25901 leader_election.cc:290] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 19 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:20.861578 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:20.806356 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:20.865835 17761 raft_consensus.cc:2468] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 20.
I20260709 12:24:20.809890 17831 raft_consensus.cc:1240] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 20. Current term is 21. Ops: [14.10-20.13]
I20260709 12:24:20.768860 17729 leader_election.cc:304] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:20.817209 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:20.778864 25967 raft_consensus.cc:2749] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:24:20.756229 17687 raft_consensus.cc:2410] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for term 21 because replica has last-logged OpId of term: 20 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 13.
I20260709 12:24:20.828372 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 28 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:24:20.835350 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 92697.4us
I20260709 12:24:20.867766 17836 raft_consensus.cc:2468] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 20.
I20260709 12:24:20.841703 17652 leader_election.cc:304] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:20.866948 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:20.871922 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 21 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:20.825330 25951 raft_consensus.cc:493] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:20.759630 25736 raft_consensus.cc:3060] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:20.868372 18333 leader_election.cc:304] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3
W20260709 12:24:20.876636 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 107922us
I20260709 12:24:20.829233 25826 raft_consensus.cc:493] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:20.873881 17681 raft_consensus.cc:2468] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 24.
I20260709 12:24:20.853261 17833 raft_consensus.cc:2468] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 19.
I20260709 12:24:20.879729 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:20.883230 17687 raft_consensus.cc:2393] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 22: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:24:20.835143 25424 raft_consensus.cc:493] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:20.844199 25909 raft_consensus.cc:493] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:20.835502 25638 raft_consensus.cc:2749] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 12:24:20.878136 17729 leader_election.cc:304] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:20.875541 17652 leader_election.cc:304] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:20.904389 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:20.904803 25637 consensus_queue.cc:1059] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:24:20.891609 17764 raft_consensus.cc:2468] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 22.
I20260709 12:24:20.910903 17683 raft_consensus.cc:2468] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 26.
I20260709 12:24:20.913336 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:20.915400 17755 raft_consensus.cc:2393] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 24: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:24:20.901520 17830 raft_consensus.cc:2468] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 21.
I20260709 12:24:20.899434 25901 raft_consensus.cc:2804] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:24:20.918170 18333 leader_election.cc:304] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:20.898491 25424 raft_consensus.cc:3060] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:20.918709 17728 leader_election.cc:304] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:20.859948 25890 raft_consensus.cc:515] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:20.864405 25957 raft_consensus.cc:515] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:20.905650 26024 raft_consensus.cc:2804] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:24:20.905913 25663 raft_consensus.cc:2804] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:24:20.908995 25967 raft_consensus.cc:2804] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:24:20.925349 17728 leader_election.cc:304] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:20.873859 25952 raft_consensus.cc:515] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:20.929838 18334 leader_election.cc:304] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:24:20.920539 25746 raft_consensus.cc:2804] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:24:20.902525 25826 raft_consensus.cc:515] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:20.939781 18333 leader_election.cc:304] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:24:20.938104 17757 raft_consensus.cc:2468] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 21.
I20260709 12:24:20.939297 17829 raft_consensus.cc:2468] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 24.
I20260709 12:24:20.913439 25972 consensus_queue.cc:1059] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 18.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.019s
I20260709 12:24:20.924237 26024 raft_consensus.cc:493] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:20.906486 25909 raft_consensus.cc:515] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:20.925231 25901 raft_consensus.cc:697] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 24 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:24:20.923761 25264 consensus_queue.cc:1048] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.030s
I20260709 12:24:20.900784 25951 raft_consensus.cc:515] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:20.921886 26026 consensus_queue.cc:1048] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.029s
I20260709 12:24:20.930462 25967 raft_consensus.cc:493] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:20.960553 17728 leader_election.cc:304] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:20.942802 25781 raft_consensus.cc:2804] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:24:20.939042 25637 raft_consensus.cc:2804] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:24:20.947844 26014 raft_consensus.cc:2804] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:24:20.951457 25678 raft_consensus.cc:2749] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:24:20.953987 25746 raft_consensus.cc:697] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 25 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:24:20.974123 17683 raft_consensus.cc:1275] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 22 index: 12. Preceding OpId from leader: term: 26 index: 14. (index mismatch)
I20260709 12:24:20.959964 26024 raft_consensus.cc:3060] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:20.975703 25637 raft_consensus.cc:697] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 20 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:24:20.981740 25909 leader_election.cc:290] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:20.982209 25776 raft_consensus.cc:2804] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:24:20.982788 25901 consensus_queue.cc:237] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 23.16, Last appended by leader: 16, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:20.984200 25781 raft_consensus.cc:697] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 24 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:24:20.985419 25826 leader_election.cc:290] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:20.981808 25951 leader_election.cc:290] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:20.979537 26017 raft_consensus.cc:515] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:20.990933 26013 log.cc:927] Time spent T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 1.005s	user 0.003s	sys 0.001s
I20260709 12:24:20.993132 25915 raft_consensus.cc:3055] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:24:20.994072 25957 leader_election.cc:290] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:20.996438 25959 raft_consensus.cc:515] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:20.959777 26018 raft_consensus.cc:2804] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:24:20.997712 17687 raft_consensus.cc:1240] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 24. Current term is 25. Ops: []
W20260709 12:24:20.998255 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 74499.6us
I20260709 12:24:20.999442 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:21.018745 26014 raft_consensus.cc:697] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 16 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:24:21.045322 17681 raft_consensus.cc:2468] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 21.
W20260709 12:24:21.024837 17802 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:20.959335 25638 consensus_queue.cc:1059] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 18.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.040s
I20260709 12:24:20.973899 25967 raft_consensus.cc:3060] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:24:21.048924 17801 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:20.967628 25761 consensus_queue.cc:1059] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true }, Status: INVALID_TERM, Last received: 12.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.068s
W20260709 12:24:21.040861 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 101165us
I20260709 12:24:21.046937 26014 consensus_queue.cc:237] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 14.12, Last appended by leader: 12, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:21.053514 18333 leader_election.cc:304] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:21.054679 26026 raft_consensus.cc:3055] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:24:21.055186 26026 raft_consensus.cc:740] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:24:21.056197 18333 leader_election.cc:304] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:21.056093 26026 consensus_queue.cc:260] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:21.057603 26026 raft_consensus.cc:3060] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:21.057730 26059 raft_consensus.cc:2749] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 12:24:21.061403 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 23 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:21.061851 26059 raft_consensus.cc:2764] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Leader pre-election decision vote started in defunct term 23: won
I20260709 12:24:21.062120 17755 raft_consensus.cc:2468] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 22.
W20260709 12:24:21.064205 17803 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.669s	user 0.022s	sys 0.027s
W20260709 12:24:21.064545 17803 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.670s	user 0.022s	sys 0.028s
W20260709 12:24:21.064803 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 757425us
I20260709 12:24:21.072659 17802 leader_election.cc:304] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:21.035346 25663 raft_consensus.cc:697] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:24:21.075528 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 25 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:21.076205 17764 raft_consensus.cc:2468] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 24.
I20260709 12:24:21.072799 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:24:21.077961 17655 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:21.079077 17802 leader_election.cc:304] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:21.083901 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:21.084599 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 23 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:21.085616 25740 raft_consensus.cc:3060] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Advancing to term 21
W20260709 12:24:21.088513 26052 log.cc:927] Time spent T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.140s	user 0.000s	sys 0.001s
I20260709 12:24:21.088989 17759 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38776 (request call id 2960) took 1064 ms. Trace:
I20260709 12:24:21.089402 17759 rpcz_store.cc:276] 0709 12:24:20.024364 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:20.030578 (+  6214us) service_pool.cc:224] Handling call
0709 12:24:20.030973 (+   395us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:20.401427 (+370454us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:21.088679 (+687252us) spinlock_profiling.cc:243] Waited 370 ms on lock 0x7b54003e5dd4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:24:21.088896 (+   217us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":849975680}
W20260709 12:24:21.091079 18371 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59886 (request call id 2112) took 2455 ms (client timeout 2182 ms). Trace:
W20260709 12:24:21.091519 18371 rpcz_store.cc:269] 0709 12:24:18.635714 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:18.635921 (+   207us) service_pool.cc:224] Handling call
0709 12:24:21.090837 (+2454916us) spinlock_profiling.cc:243] Waited 1.72 s on lock 0x7b54007e8954. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:24:21.090969 (+   132us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3948665856}
I20260709 12:24:21.081645 25881 consensus_queue.cc:1048] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.015s
I20260709 12:24:21.093604 25746 consensus_queue.cc:237] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 23.15, Last appended by leader: 15, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:21.100806 25781 consensus_queue.cc:237] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 22.13, Last appended by leader: 13, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:24:21.103379 17688 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37402 (request call id 3469) took 2114 ms. Trace:
I20260709 12:24:21.103835 17688 rpcz_store.cc:276] 0709 12:24:18.988905 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:18.996192 (+  7287us) service_pool.cc:224] Handling call
0709 12:24:19.027273 (+ 31081us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:20.634726 (+1607453us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:21.103110 (+468384us) spinlock_profiling.cc:243] Waited 1.6 s on lock 0x7b5400338954. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:24:21.103289 (+   179us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3678372224}
I20260709 12:24:21.038895 25776 raft_consensus.cc:697] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:24:21.106690 25776 consensus_queue.cc:237] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 25.14, Last appended by leader: 14, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:21.111232 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:21.112049 26017 leader_election.cc:290] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 20 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:21.112287 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:21.112691 18364 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 2342) took 1791 ms. Trace:
I20260709 12:24:21.113014 18374 raft_consensus.cc:2393] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 25: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:24:21.113137 18364 rpcz_store.cc:276] 0709 12:24:19.321830 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:19.330801 (+  8971us) service_pool.cc:224] Handling call
0709 12:24:19.342306 (+ 11505us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:20.599348 (+1257042us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:21.112374 (+513026us) spinlock_profiling.cc:243] Waited 1.25 s on lock 0x7b5400466ed4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:24:21.112627 (+   253us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2877449088}
I20260709 12:24:21.069686 26022 raft_consensus.cc:493] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:21.038216 25915 raft_consensus.cc:740] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
W20260709 12:24:21.074255 17803 leader_election.cc:336] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Timed out: RequestConsensusVote RPC to 127.16.29.68:42577 timed out after 1.531s (SENT)
I20260709 12:24:21.084625 17830 raft_consensus.cc:2468] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 24.
I20260709 12:24:21.120450 18365 raft_consensus.cc:1240] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 21. Current term is 22. Ops: [21.16-21.16]
I20260709 12:24:21.123466 17802 leader_election.cc:304] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:21.124323 17801 leader_election.cc:304] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:21.121166 17803 leader_election.cc:304] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:21.129355 17652 leader_election.cc:304] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:21.129606 17760 raft_consensus.cc:2468] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 24.
I20260709 12:24:21.132493 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 25 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:21.133267 17652 leader_election.cc:304] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:21.137734 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 28 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:21.139134 18366 raft_consensus.cc:2410] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for term 28 because replica has last-logged OpId of term: 26 index: 20, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 19.
I20260709 12:24:21.063688 26018 raft_consensus.cc:493] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:24:21.141811 17802 leader_election.cc:304] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
W20260709 12:24:21.148295 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 76590.3us
I20260709 12:24:21.140844 25151 raft_consensus.cc:2804] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:24:21.149386 17727 leader_election.cc:304] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:21.134825 25959 leader_election.cc:290] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 23 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:21.149901 17652 leader_election.cc:304] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:21.145601 26018 raft_consensus.cc:3060] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:21.150241 26021 raft_consensus.cc:2749] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:24:21.162686 25424 raft_consensus.cc:515] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:21.115824 25967 raft_consensus.cc:515] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:21.116261 25952 leader_election.cc:290] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:21.132984 25881 raft_consensus.cc:2749] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
W20260709 12:24:21.181643 17727 leader_election.cc:336] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 19 pre-election: RPC error from VoteRequest() call to peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837): Timed out: RequestConsensusVote RPC to 127.16.29.67:36837 timed out after 1.696s (SENT)
I20260709 12:24:21.150591 25151 raft_consensus.cc:493] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:24:21.200510 25151 raft_consensus.cc:3060] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:21.185087 17652 leader_election.cc:304] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:24:21.206063 17652 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:21.198277 18366 raft_consensus.cc:1240] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 19. Current term is 20. Ops: []
I20260709 12:24:21.149842 26064 raft_consensus.cc:493] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:21.230257 26064 raft_consensus.cc:515] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:21.187415 26062 raft_consensus.cc:2804] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:24:21.232712 26062 raft_consensus.cc:697] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:24:21.233271 26064 leader_election.cc:290] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:21.233817 26062 consensus_queue.cc:237] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 19.15, Last appended by leader: 15, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:21.236011 25761 raft_consensus.cc:2804] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:24:21.236714 25761 raft_consensus.cc:493] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:21.237135 25761 raft_consensus.cc:3060] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:21.238917 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 23 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:21.239940 17688 raft_consensus.cc:2468] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 22.
W20260709 12:24:21.241901 17729 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:21.242851 17729 leader_election.cc:304] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:21.157881 26003 raft_consensus.cc:2804] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:24:21.212615 18366 raft_consensus.cc:1275] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 16 index: 12. Preceding OpId from leader: term: 20 index: 14. (index mismatch)
I20260709 12:24:21.150295 25688 raft_consensus.cc:2804] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:24:21.218384 18367 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59886 (request call id 2123) took 2099 ms. Trace:
I20260709 12:24:21.172870 26049 consensus_queue.cc:1048] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.071s
I20260709 12:24:21.196630 25384 raft_consensus.cc:2804] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:24:21.116906 26022 raft_consensus.cc:515] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:21.247550 18367 rpcz_store.cc:276] 0709 12:24:19.118829 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:19.127066 (+  8237us) service_pool.cc:224] Handling call
0709 12:24:19.160834 (+ 33768us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:19.161272 (+   438us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:19.161314 (+    42us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:19.172734 (+ 11420us) log.cc:844] Serialized 26 byte log entry
0709 12:24:20.038749 (+866015us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:24:20.038794 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:20.038868 (+    74us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:20.990066 (+951198us) spinlock_profiling.cc:243] Waited 2.26 ms on lock 0x7b4400022b4c. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b47459c85 00007f7b4745a6a2 00007f7b4a1cc1db 00007f7b4a1c7aaf 00007f7b4a1bb86e 00007f7b4a1babb6 00007f7b4a1ba7d8 00007f7b4a1c4221 00007f7b4a1c41d9 00007f7b4a1c4169
0709 12:24:21.217958 (+227892us) raft_consensus.cc:1685] finished
0709 12:24:21.217991 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:21.218304 (+   313us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":164158,"raft.run_cpu_time_us":169,"raft.run_wall_time_us":20255,"spinlock_wait_cycles":5201152,"thread_start_us":740753,"threads_started":1,"wal-append.queue_time_us":768965}
I20260709 12:24:21.277971 25890 leader_election.cc:290] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:21.189944 25918 raft_consensus.cc:2804] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:24:21.117834 25915 consensus_queue.cc:260] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 19.15, Last appended by leader: 15, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:21.209421 26021 raft_consensus.cc:2749] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:24:21.271934 17729 leader_election.cc:304] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
W20260709 12:24:21.300457 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 244334us
I20260709 12:24:21.268908 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 18 candidate_status { last_received { term: 15 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:21.209354 26059 raft_consensus.cc:493] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
W20260709 12:24:21.315354 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 170288us
W20260709 12:24:21.318279 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 46489.3us
I20260709 12:24:21.324783 17835 raft_consensus.cc:2468] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 22.
I20260709 12:24:21.264700 17652 leader_election.cc:304] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
W20260709 12:24:21.327847 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 63262.9us
I20260709 12:24:21.330382 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:21.342666 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 21 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:21.342952 25151 raft_consensus.cc:515] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:21.330648 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:21.344743 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:21.333241 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:21.174630 25637 consensus_queue.cc:237] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.15, Last appended by leader: 15, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:21.371449 25151 leader_election.cc:290] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 21 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:21.378434 17764 raft_consensus.cc:1240] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 19. Current term is 20. Ops: []
I20260709 12:24:21.378782 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 23 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:21.332649 25959 raft_consensus.cc:2804] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:24:21.397387 25959 raft_consensus.cc:697] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 21 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:24:21.340747 25918 raft_consensus.cc:697] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 20 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:24:21.398614 25918 consensus_queue.cc:237] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 19.17, Last appended by leader: 17, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:21.385600 25784 raft_consensus.cc:493] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:21.334867 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 22 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:21.336624 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:21.261410 25663 consensus_queue.cc:237] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:21.280943 25688 raft_consensus.cc:697] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 25 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:24:21.282181 26003 raft_consensus.cc:697] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 22 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
W20260709 12:24:21.341082 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 199395us
I20260709 12:24:21.370854 17652 leader_election.cc:304] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:21.385708 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 23 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:21.413209 18371 raft_consensus.cc:2468] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 22.
I20260709 12:24:21.331573 25915 raft_consensus.cc:3060] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:21.387540 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:21.388015 25424 leader_election.cc:290] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 27 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:21.388214 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 22 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:21.356670 17801 leader_election.cc:304] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:24:21.260494 25384 raft_consensus.cc:493] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:24:21.390534 17804 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:21.392396 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 22 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:21.424458 17764 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 21.
I20260709 12:24:21.398509 25959 consensus_queue.cc:237] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.12, Last appended by leader: 12, Current term: 21, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:21.402829 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:21.403038 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:21.403491 17756 raft_consensus.cc:2468] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 22.
I20260709 12:24:21.273216 25960 raft_consensus.cc:2749] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:24:21.327239 25974 raft_consensus.cc:493] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:24:21.335510 25858 raft_consensus.cc:2804] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:24:21.405930 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:21.437923 18361 raft_consensus.cc:3060] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:21.410434 18366 raft_consensus.cc:1240] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 23. Current term is 24. Ops: []
I20260709 12:24:21.410787 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:21.401417 25784 raft_consensus.cc:515] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:24:21.440948 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 86236.8us
I20260709 12:24:21.441401 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 20 candidate_status { last_received { term: 18 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:24:21.415262 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 44500.6us
I20260709 12:24:21.444451 18367 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639"
dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
mode: GRACEFUL
new_leader_uuid: "0da169ff82674dc1947055ce43de68a3"
 from {username='slave'} at 127.0.0.1:50468
I20260709 12:24:21.445081 18367 raft_consensus.cc:606] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Received request to transfer leadership to TS 0da169ff82674dc1947055ce43de68a3
I20260709 12:24:21.445446 18367 raft_consensus.cc:614] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:24:21.446364 18374 raft_consensus.cc:1240] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 24. Current term is 25. Ops: []
I20260709 12:24:21.446604 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 21 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:21.446946 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:21.447504 17836 raft_consensus.cc:2393] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 21: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:24:21.448501 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:21.448884 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 20 candidate_status { last_received { term: 19 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:21.332617 26059 raft_consensus.cc:515] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:21.430866 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:21.438764 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:21.428726 25384 raft_consensus.cc:3060] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Advancing to term 25
W20260709 12:24:21.459102 17802 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:21.452832 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:21.449558 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 21 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:21.450587 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 22 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:21.461615 17833 raft_consensus.cc:2410] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for term 22 because replica has last-logged OpId of term: 19 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 11.
I20260709 12:24:21.462747 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:21.463574 17833 raft_consensus.cc:3060] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Advancing to term 23
I20260709 12:24:21.455379 18363 raft_consensus.cc:2393] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 21: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:24:21.426311 25978 raft_consensus.cc:2804] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:24:21.456616 17838 raft_consensus.cc:1275] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 22 index: 13. Preceding OpId from leader: term: 23 index: 13. (term mismatch)
I20260709 12:24:21.467295 17838 pending_rounds.cc:85] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b: Aborting all ops after (but not including) 12
I20260709 12:24:21.467221 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:21.467833 17838 pending_rounds.cc:107] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b: Aborting uncommitted NO_OP operation due to leader change: 22.13
I20260709 12:24:21.459766 17759 raft_consensus.cc:2468] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 24.
I20260709 12:24:21.468267 17838 raft_consensus.cc:2889] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:24:21.468488 17655 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:21.450330 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:21.466344 18361 raft_consensus.cc:2468] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 23.
I20260709 12:24:21.469602 17836 raft_consensus.cc:2468] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 14.
I20260709 12:24:21.469743 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 23 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:21.471943 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 23 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:21.472620 17681 raft_consensus.cc:2468] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 22.
I20260709 12:24:21.478222 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:21.478524 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:21.479104 17831 raft_consensus.cc:2393] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 20: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:24:21.479283 25784 leader_election.cc:290] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:21.480778 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:21.481473 17685 raft_consensus.cc:2468] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 23.
I20260709 12:24:21.482158 25979 raft_consensus.cc:493] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:21.482725 25979 raft_consensus.cc:515] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:21.448767 25974 raft_consensus.cc:515] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:21.485251 25979 leader_election.cc:290] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:21.489337 25688 consensus_queue.cc:237] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 22.14, Last appended by leader: 14, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:21.479426 18366 raft_consensus.cc:2393] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 20: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:24:21.497782 18361 raft_consensus.cc:1240] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 19. Current term is 20. Ops: []
I20260709 12:24:21.451045 25858 raft_consensus.cc:697] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 19 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:24:21.500321 17830 raft_consensus.cc:1275] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 19 index: 13. Preceding OpId from leader: term: 20 index: 14. (index mismatch)
I20260709 12:24:21.505285 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 21 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:21.505544 18363 raft_consensus.cc:1240] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 19. Current term is 20. Ops: [19.13-19.13]
W20260709 12:24:21.417753 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 343500us
I20260709 12:24:21.505788 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:21.466109 26021 raft_consensus.cc:2804] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:24:21.473029 25978 raft_consensus.cc:493] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:21.511273 18368 raft_consensus.cc:1240] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 20. Current term is 21. Ops: [20.13-20.13]
I20260709 12:24:21.507129 17802 leader_election.cc:304] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
W20260709 12:24:21.506352 17803 leader_election.cc:336] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 21 election: RPC error from VoteRequest() call to peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Timed out: RequestConsensusVote RPC to 127.16.29.68:42577 timed out after 1.547s (SENT)
I20260709 12:24:21.512466 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:21.513242 18366 raft_consensus.cc:2468] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 19.
I20260709 12:24:21.511703 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:21.507295 18374 raft_consensus.cc:1275] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 24 index: 15. Preceding OpId from leader: term: 26 index: 16. (index mismatch)
I20260709 12:24:21.513386 17803 leader_election.cc:304] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:21.516207 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:21.517845 17687 raft_consensus.cc:1275] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 19 index: 13. Preceding OpId from leader: term: 20 index: 14. (index mismatch)
I20260709 12:24:21.518703 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:21.518915 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 23 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:21.519304 17686 raft_consensus.cc:2468] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 19.
I20260709 12:24:21.519409 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:21.519455 17684 raft_consensus.cc:2468] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 22.
I20260709 12:24:21.521034 17728 leader_election.cc:304] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:21.522224 17686 raft_consensus.cc:1240] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 26. Current term is 27. Ops: [26.18-26.18]
I20260709 12:24:21.523564 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 22 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:21.524209 18368 raft_consensus.cc:3060] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:21.523806 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:21.520934 17689 raft_consensus.cc:1240] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 18. Current term is 19. Ops: []
I20260709 12:24:21.525583 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:21.525768 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:21.526194 18367 raft_consensus.cc:2393] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 22: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:24:21.526464 18363 raft_consensus.cc:2368] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 22. Current term is 23.
I20260709 12:24:21.452854 26018 raft_consensus.cc:515] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:21.527258 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:21.529209 17686 raft_consensus.cc:1240] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 20. Current term is 21. Ops: []
I20260709 12:24:21.529445 17684 raft_consensus.cc:1240] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 19. Current term is 20. Ops: [19.15-19.15]
W20260709 12:24:21.536511 17803 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:21.537158 17803 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:21.482748 25917 leader_election.cc:290] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:21.540643 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:21.542737 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:21.472448 26049 consensus_queue.cc:1048] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.044s
I20260709 12:24:21.546311 17759 raft_consensus.cc:2393] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 27: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:24:21.546438 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:21.547430 17683 raft_consensus.cc:2393] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 27: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
W20260709 12:24:21.562819 17803 leader_election.cc:336] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 15 pre-election: RPC error from VoteRequest() call to peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Timed out: RequestConsensusVote RPC to 127.16.29.68:42577 timed out after 1.778s (SENT)
I20260709 12:24:21.563329 17803 leader_election.cc:304] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:21.547077 17728 leader_election.cc:304] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:21.545604 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 21 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:21.571607 17759 raft_consensus.cc:1240] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 26. Current term is 27. Ops: []
I20260709 12:24:21.515455 25978 raft_consensus.cc:3060] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:21.573761 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:21.499862 25979 consensus_queue.cc:1048] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.049s
I20260709 12:24:21.529985 25952 raft_consensus.cc:2804] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:24:21.524852 25918 raft_consensus.cc:2804] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Leader pre-election won for term 25
W20260709 12:24:21.595794 18334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:21.597239 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 17 candidate_status { last_received { term: 15 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:21.598181 18367 raft_consensus.cc:2468] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 16.
I20260709 12:24:21.579434 18368 raft_consensus.cc:2468] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 22.
I20260709 12:24:21.523984 25890 raft_consensus.cc:2749] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
W20260709 12:24:21.584195 17803 leader_election.cc:336] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 20 pre-election: RPC error from VoteRequest() call to peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Timed out: RequestConsensusVote RPC to 127.16.29.68:42577 timed out after 1.808s (SENT)
I20260709 12:24:21.517864 25967 leader_election.cc:290] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 27 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:21.534328 25784 consensus_queue.cc:1048] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.065s
I20260709 12:24:21.532851 25736 raft_consensus.cc:493] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:21.592974 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:21.510674 25960 consensus_queue.cc:1048] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.073s
I20260709 12:24:21.605120 17803 leader_election.cc:304] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:21.595844 26072 raft_consensus.cc:3060] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:21.613196 17683 raft_consensus.cc:1240] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 24. Current term is 25. Ops: []
I20260709 12:24:21.529812 25776 consensus_queue.cc:1059] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.064s
I20260709 12:24:21.579434 17833 raft_consensus.cc:2468] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 23.
I20260709 12:24:21.645655 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:21.499991 26003 consensus_queue.cc:237] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 20.15, Last appended by leader: 15, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:21.592895 25420 raft_consensus.cc:2804] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Leader pre-election won for term 20
W20260709 12:24:21.630491 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 46382.2us
I20260709 12:24:21.593858 25952 raft_consensus.cc:493] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:21.601004 25918 raft_consensus.cc:493] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:21.570173 25781 consensus_queue.cc:1048] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.046s
I20260709 12:24:21.648825 18366 raft_consensus.cc:1240] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 17. Current term is 18. Ops: []
W20260709 12:24:21.648969 17803 leader_election.cc:336] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 19 election: RPC error from VoteRequest() call to peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Timed out: RequestConsensusVote RPC to 127.16.29.68:42577 timed out after 1.839s (SENT)
I20260709 12:24:21.650308 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:21.579742 25637 consensus_queue.cc:1059] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 25.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.039s
I20260709 12:24:21.657333 17803 leader_election.cc:304] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:21.602551 25995 consensus_queue.cc:1059] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 20.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.044s
I20260709 12:24:21.621065 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 21 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:21.644263 25890 raft_consensus.cc:2749] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 20 LEADER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:24:21.604389 25917 consensus_queue.cc:1059] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 18.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.044s
I20260709 12:24:21.677914 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 23 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:21.706549 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
W20260709 12:24:21.689395 17803 leader_election.cc:336] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 17 pre-election: RPC error from VoteRequest() call to peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Timed out: RequestConsensusVote RPC to 127.16.29.68:42577 timed out after 1.853s (SENT)
I20260709 12:24:21.707639 18368 raft_consensus.cc:2368] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 21. Current term is 22.
W20260709 12:24:21.707720 17803 leader_election.cc:336] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Timed out: RequestConsensusVote RPC to 127.16.29.68:42577 timed out after 1.917s (SENT)
W20260709 12:24:21.709244 17803 leader_election.cc:336] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Timed out: RequestConsensusVote RPC to 127.16.29.68:42577 timed out after 1.919s (SENT)
I20260709 12:24:21.706935 18363 raft_consensus.cc:2468] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 22.
I20260709 12:24:21.709708 17803 leader_election.cc:304] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:21.690058 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:21.652273 25420 raft_consensus.cc:493] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:21.710817 18366 raft_consensus.cc:3060] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:21.715204 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:24:21.716450 18334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:21.716429 17803 leader_election.cc:336] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 20 pre-election: RPC error from VoteRequest() call to peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Timed out: RequestConsensusVote RPC to 127.16.29.68:42577 timed out after 1.948s (SENT)
I20260709 12:24:21.718266 17803 leader_election.cc:304] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:21.719010 18368 raft_consensus.cc:2368] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 19. Current term is 20.
I20260709 12:24:21.631094 25736 raft_consensus.cc:515] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:21.648612 26022 leader_election.cc:290] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:21.671626 26021 raft_consensus.cc:697] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 24 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:24:21.656047 25952 raft_consensus.cc:3060] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:21.661046 25781 consensus_queue.cc:1243] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer 0da169ff82674dc1947055ce43de68a3 log is divergent from this leader: its last log entry 19.13 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 12
I20260709 12:24:21.656349 25918 raft_consensus.cc:3060] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Advancing to term 25
W20260709 12:24:21.740845 17803 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:21.708912 25890 raft_consensus.cc:2749] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:24:21.654946 25761 raft_consensus.cc:515] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:21.753319 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 18 candidate_status { last_received { term: 15 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:21.690392 26070 consensus_queue.cc:1059] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 24.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.076s
I20260709 12:24:21.723992 26071 raft_consensus.cc:2749] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:24:21.784948 18363 raft_consensus.cc:2468] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 16.
I20260709 12:24:21.744503 26068 raft_consensus.cc:3055] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:24:21.787125 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
W20260709 12:24:21.790338 17729 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:22.059595 17729 leader_election.cc:304] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
W20260709 12:24:22.063007 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:22.069068 26081 log.cc:927] Time spent T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.417s	user 0.000s	sys 0.001s
I20260709 12:24:22.070058 25761 leader_election.cc:290] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 22 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:24:22.057904 26078 log.cc:927] Time spent T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.550s	user 0.001s	sys 0.007s
I20260709 12:24:22.062166 26068 raft_consensus.cc:740] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:24:22.076618 26068 consensus_queue.cc:260] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 26.18, Last appended by leader: 18, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:22.078593 26068 raft_consensus.cc:3060] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:21.795998 18366 raft_consensus.cc:2468] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 22.
I20260709 12:24:21.888399 17838 raft_consensus.cc:1240] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 25 LEADER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 24. Current term is 25. Ops: []
I20260709 12:24:22.018122 25575 consensus_queue.cc:1059] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 18.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.003s
W20260709 12:24:21.938222 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 429760us
W20260709 12:24:21.952589 26069 log.cc:927] Time spent T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.397s	user 0.001s	sys 0.001s
I20260709 12:24:21.772614 26021 consensus_queue.cc:237] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 23.14, Last appended by leader: 14, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:21.954420 26089 raft_consensus.cc:493] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:21.952318 25826 raft_consensus.cc:493] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:21.942193 26018 leader_election.cc:290] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 16 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:24:21.981056 26084 log.cc:927] Time spent T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.092s	user 0.000s	sys 0.000s
I20260709 12:24:21.940601 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
W20260709 12:24:21.996227 17804 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:21.990494 17655 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:21.929836 26071 raft_consensus.cc:493] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:22.119410 26071 raft_consensus.cc:515] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:24:22.120165 26093 log.cc:927] Time spent T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.079s	user 0.001s	sys 0.000s
I20260709 12:24:22.121857 26071 leader_election.cc:290] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:24:21.986035 26051 log.cc:927] Time spent T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 1.041s	user 0.002s	sys 0.002s
I20260709 12:24:21.858271 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:22.123381 18363 raft_consensus.cc:2368] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 22. Current term is 23.
I20260709 12:24:21.911473 26060 consensus_queue.cc:1059] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 24.17, Next index: 18, Last known committed idx: 16, Time since last communication: 0.041s
I20260709 12:24:22.124528 26060 raft_consensus.cc:3055] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:24:22.124883 26060 raft_consensus.cc:740] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:24:22.125741 26060 consensus_queue.cc:260] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 24.17, Last appended by leader: 17, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:22.127029 26060 raft_consensus.cc:3060] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:21.788659 25970 raft_consensus.cc:493] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:21.739153 25868 raft_consensus.cc:2749] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:24:21.860982 25974 leader_election.cc:290] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:21.773926 25736 leader_election.cc:290] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:24:21.983515 25740 consensus_meta.cc:287] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.897s	user 0.012s	sys 0.000s
I20260709 12:24:22.128146 25970 raft_consensus.cc:515] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:22.128834 25740 raft_consensus.cc:2749] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:24:21.649027 25858 consensus_queue.cc:237] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:22.130223 25970 leader_election.cc:290] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:22.129933 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:22.131229 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:24:21.941737 26079 log.cc:927] Time spent T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.347s	user 0.000s	sys 0.001s
I20260709 12:24:22.020447 17654 leader_election.cc:304] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:24:22.132411 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
W20260709 12:24:22.133010 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 112672us
I20260709 12:24:22.133489 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 17 candidate_status { last_received { term: 15 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:22.133968 18363 raft_consensus.cc:3060] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:22.134454 17654 leader_election.cc:304] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:22.135514 17654 leader_election.cc:304] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:24:21.855337 26059 leader_election.cc:290] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:24:22.136060 26024 consensus_meta.cc:287] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 1.102s	user 0.014s	sys 0.000s
I20260709 12:24:22.136237 25761 raft_consensus.cc:2804] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:24:21.771828 25978 raft_consensus.cc:515] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:22.136716 26024 raft_consensus.cc:515] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:22.138448 25978 leader_election.cc:290] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 25 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:22.139024 25978 consensus_queue.cc:1059] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 17.12, Next index: 13, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 12:24:21.716683 25420 raft_consensus.cc:3060] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:22.025919 25901 consensus_queue.cc:1048] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.017s
W20260709 12:24:21.945009 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 469661us
I20260709 12:24:22.144136 17832 raft_consensus.cc:2468] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 21.
I20260709 12:24:22.144779 25909 raft_consensus.cc:2804] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:24:22.080090 25637 raft_consensus.cc:493] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:22.145653 25637 raft_consensus.cc:515] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:22.081254 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:22.147779 25637 leader_election.cc:290] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:22.148552 26060 raft_consensus.cc:3055] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:24:21.793774 18374 raft_consensus.cc:2468] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 19.
I20260709 12:24:22.100975 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
W20260709 12:24:22.103747 17728 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.516s	user 0.014s	sys 0.020s
W20260709 12:24:22.149878 17728 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.562s	user 0.015s	sys 0.020s
W20260709 12:24:22.150195 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 567738us
I20260709 12:24:22.150447 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 25 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:22.151268 17830 raft_consensus.cc:2410] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for term 25 because replica has last-logged OpId of term: 23 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 14.
W20260709 12:24:22.094333 17801 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:22.152235 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:22.153231 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:22.153954 17830 raft_consensus.cc:2410] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for term 20 because replica has last-logged OpId of term: 18 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 12.
I20260709 12:24:22.154284 25970 raft_consensus.cc:2804] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:24:22.131672 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 20 candidate_status { last_received { term: 18 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:22.154642 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:22.154968 25868 raft_consensus.cc:493] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:22.155323 17683 raft_consensus.cc:1275] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 25 index: 14. Preceding OpId from leader: term: 26 index: 15. (index mismatch)
I20260709 12:24:22.155418 18365 raft_consensus.cc:2410] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 20 because replica has last-logged OpId of term: 19 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 16.
I20260709 12:24:22.154673 25970 raft_consensus.cc:493] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:24:22.067589 25866 raft_consensus.cc:3060] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:22.109786 18367 raft_consensus.cc:2393] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 21: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:24:22.156409 25970 raft_consensus.cc:3060] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:22.159710 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 21 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:22.109594 26089 raft_consensus.cc:515] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:22.160794 18367 raft_consensus.cc:2368] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 21. Current term is 22.
I20260709 12:24:22.161660 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 21 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:22.162314 18367 raft_consensus.cc:2368] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 21. Current term is 22.
I20260709 12:24:22.162642 25776 raft_consensus.cc:3055] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 LEADER]: Stepping down as leader of term 24
W20260709 12:24:22.163012 26088 log.cc:927] Time spent T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.105s	user 0.000s	sys 0.000s
I20260709 12:24:22.163132 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 21 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:22.157763 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:22.163928 17727 leader_election.cc:304] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:22.164338 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:22.165396 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:22.111017 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:22.145198 25909 raft_consensus.cc:493] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:22.148815 18368 raft_consensus.cc:2393] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 22: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
W20260709 12:24:22.082427 26087 log.cc:927] Time spent T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.153s	user 0.001s	sys 0.000s
W20260709 12:24:22.149207 17652 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:22.156399 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 17 candidate_status { last_received { term: 15 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:22.170358 17652 leader_election.cc:304] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:22.171250 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:22.147750 18363 raft_consensus.cc:2468] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 17.
I20260709 12:24:22.173346 17689 pending_rounds.cc:85] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3: Aborting all ops after (but not including) 12
I20260709 12:24:22.173743 17689 pending_rounds.cc:107] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3: Aborting uncommitted NO_OP operation due to leader change: 19.13
I20260709 12:24:22.174026 17689 raft_consensus.cc:2889] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:22.175625 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:22.177409 18363 raft_consensus.cc:2393] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 20: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:24:22.182192 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:22.187291 25151 raft_consensus.cc:493] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:22.187170 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 21 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:22.188055 25151 raft_consensus.cc:515] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:22.189873 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:22.192440 18363 raft_consensus.cc:2393] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 25: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:24:22.198098 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:22.198897 18366 raft_consensus.cc:2393] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 19: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:24:22.199563 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 18 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:22.201164 17654 leader_election.cc:304] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:22.201627 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:22.202487 18368 raft_consensus.cc:2368] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 19. Current term is 20.
I20260709 12:24:22.202510 18363 raft_consensus.cc:2393] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 18: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:24:22.208956 18367 raft_consensus.cc:1240] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 20. Current term is 21. Ops: []
I20260709 12:24:22.209179 18363 raft_consensus.cc:1240] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 23. Current term is 24. Ops: []
I20260709 12:24:22.139412 26024 leader_election.cc:290] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 22 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:22.211287 18365 raft_consensus.cc:1275] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 21 index: 16. Preceding OpId from leader: term: 24 index: 18. (index mismatch)
I20260709 12:24:22.211144 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:22.211799 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:22.166810 25970 raft_consensus.cc:515] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:22.186542 25740 raft_consensus.cc:493] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:22.169044 25909 raft_consensus.cc:3060] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Advancing to term 23
W20260709 12:24:22.213485 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 154007us
I20260709 12:24:22.214448 17683 raft_consensus.cc:2393] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 24: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:24:22.163085 25776 raft_consensus.cc:740] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:24:22.215317 18363 raft_consensus.cc:2393] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 20: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:24:22.167169 26060 raft_consensus.cc:740] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:24:22.215938 18367 raft_consensus.cc:1275] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 20 index: 12. Preceding OpId from leader: term: 23 index: 13. (index mismatch)
I20260709 12:24:22.227248 18368 raft_consensus.cc:1240] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 22. Current term is 23. Ops: []
I20260709 12:24:22.189973 25901 raft_consensus.cc:2749] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 12:24:22.155475 25868 raft_consensus.cc:515] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:22.111168 25826 raft_consensus.cc:515] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:22.117946 25918 raft_consensus.cc:515] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:22.259594 18367 raft_consensus.cc:1275] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 21 index: 15. Preceding OpId from leader: term: 24 index: 17. (index mismatch)
I20260709 12:24:22.234340 18366 raft_consensus.cc:1240] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 19. Current term is 20. Ops: [19.13-19.13]
I20260709 12:24:22.201201 26090 raft_consensus.cc:2804] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:24:22.224428 25663 raft_consensus.cc:493] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:22.212911 25978 raft_consensus.cc:2749] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 12:24:22.306915 25151 leader_election.cc:290] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:22.308465 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:22.309924 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:22.310809 18366 raft_consensus.cc:2393] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 25: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:24:22.256227 26089 leader_election.cc:290] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:24:22.264148 25915 consensus_meta.cc:287] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.835s	user 0.012s	sys 0.000s
W20260709 12:24:22.324960 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 89711.3us
I20260709 12:24:22.259809 26100 raft_consensus.cc:3060] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:22.266988 25761 raft_consensus.cc:697] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 22 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:24:22.253919 25740 raft_consensus.cc:515] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:22.333491 18366 raft_consensus.cc:1275] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 15 index: 16. Preceding OpId from leader: term: 17 index: 17. (index mismatch)
I20260709 12:24:22.330432 17729 leader_election.cc:304] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
W20260709 12:24:22.337240 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 108103us
I20260709 12:24:22.260394 26009 consensus_queue.cc:1048] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.068s
I20260709 12:24:22.321955 26090 raft_consensus.cc:697] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 23 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:24:22.342335 18367 raft_consensus.cc:1240] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 21. Current term is 22. Ops: []
I20260709 12:24:22.325994 25901 raft_consensus.cc:493] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:24:22.287014 25858 consensus_queue.cc:1048] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.044s
I20260709 12:24:22.343778 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:24:22.340906 25952 consensus_meta.cc:287] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.566s	user 0.007s	sys 0.005s
I20260709 12:24:22.307821 25663 raft_consensus.cc:515] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:22.350458 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 23 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:22.350924 18366 raft_consensus.cc:2468] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 22.
I20260709 12:24:22.357615 18367 raft_consensus.cc:1240] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 17. Current term is 18. Ops: []
I20260709 12:24:22.346590 25901 raft_consensus.cc:515] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:22.362614 17762 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36572 (request call id 1417) took 1033 ms. Trace:
I20260709 12:24:22.363052 17762 rpcz_store.cc:276] 0709 12:24:21.329061 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:21.329246 (+   185us) service_pool.cc:224] Handling call
0709 12:24:22.362381 (+1033135us) spinlock_profiling.cc:243] Waited 918 ms on lock 0x7b54000fc8d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:24:22.362552 (+   171us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2111711488}
I20260709 12:24:22.330788 26060 consensus_queue.cc:260] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.12, Last appended by leader: 12, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:22.364343 26115 consensus_queue.cc:1048] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.007s
I20260709 12:24:22.352767 25952 raft_consensus.cc:515] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:22.372227 26109 raft_consensus.cc:2749] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
W20260709 12:24:22.372908 26082 log.cc:927] Time spent T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.720s	user 0.000s	sys 0.001s
I20260709 12:24:22.374543 25952 leader_election.cc:290] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 22 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:24:22.375120 26102 log.cc:927] Time spent T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.109s	user 0.001s	sys 0.000s
I20260709 12:24:22.353880 25918 leader_election.cc:290] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 25 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:22.375535 26060 raft_consensus.cc:3060] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:22.375659 18374 raft_consensus.cc:2410] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for term 27 because replica has last-logged OpId of term: 26 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 17.
I20260709 12:24:22.378662 17688 raft_consensus.cc:1275] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 22 index: 13. Preceding OpId from leader: term: 24 index: 14. (index mismatch)
I20260709 12:24:22.379818 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 20 candidate_status { last_received { term: 19 index: 17 } } ignore_live_leader: true dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:22.381204 18366 raft_consensus.cc:3060] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:22.337319 25776 consensus_queue.cc:260] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 24.15, Last appended by leader: 15, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:22.398880 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:22.400499 17684 raft_consensus.cc:2393] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 22: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:24:22.405491 18366 raft_consensus.cc:2468] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 20.
I20260709 12:24:22.408669 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:22.409368 17834 raft_consensus.cc:2468] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 26.
I20260709 12:24:22.410694 25420 raft_consensus.cc:515] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:22.411464 17834 pending_rounds.cc:85] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b: Aborting all ops after (but not including) 18
I20260709 12:24:22.413045 17832 raft_consensus.cc:1240] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 20. Current term is 21. Ops: [14.10-20.13]
W20260709 12:24:22.417322 26111 log.cc:927] Time spent T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.065s	user 0.000s	sys 0.000s
I20260709 12:24:22.411748 17834 pending_rounds.cc:107] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b: Aborting uncommitted NO_OP operation due to leader change: 23.19
I20260709 12:24:22.420467 17834 raft_consensus.cc:2889] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:24:22.420468 17829 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35040 (request call id 3279) took 1593 ms (client timeout 1733 ms). Trace:
W20260709 12:24:22.424714 17829 rpcz_store.cc:269] 0709 12:24:20.826737 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:20.826922 (+   185us) service_pool.cc:224] Handling call
0709 12:24:22.420279 (+1593357us) spinlock_profiling.cc:243] Waited 77 ms on lock 0x7b54003354d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:24:22.420388 (+   109us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":177131264}
W20260709 12:24:22.432705 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 87489.8us
I20260709 12:24:22.436136 18366 raft_consensus.cc:1240] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 21. Current term is 22. Ops: []
I20260709 12:24:22.434533 17652 leader_election.cc:304] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
W20260709 12:24:22.439352 26072 consensus_meta.cc:287] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.794s	user 0.004s	sys 0.008s
W20260709 12:24:22.452077 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 70825.4us
I20260709 12:24:22.454227 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:22.413075 25420 leader_election.cc:290] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 20 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:22.455014 18366 raft_consensus.cc:2393] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 22: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:24:22.445673 25970 leader_election.cc:290] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 18 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:22.456655 25858 consensus_queue.cc:1059] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true }, Status: INVALID_TERM, Last received: 12.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 12:24:22.378605 25761 consensus_queue.cc:237] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 20.17, Last appended by leader: 17, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:22.459519 17835 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35040 (request call id 3282) took 1389 ms (client timeout 1833 ms). Trace:
I20260709 12:24:22.469846 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:22.470642 17689 raft_consensus.cc:3060] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:22.384584 26114 consensus_queue.cc:1048] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 11, Time since last communication: 0.068s
W20260709 12:24:22.464586 17756 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38776 (request call id 2967) took 2121 ms (client timeout 2001 ms). Trace:
W20260709 12:24:22.472211 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 141869us
W20260709 12:24:22.470013 17835 rpcz_store.cc:269] 0709 12:24:21.070213 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:21.070556 (+   343us) service_pool.cc:224] Handling call
0709 12:24:22.459334 (+1388778us) spinlock_profiling.cc:243] Waited 239 ms on lock 0x7b540008a7d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:24:22.459457 (+   123us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":550727552}
I20260709 12:24:22.475159 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:22.520625 17762 raft_consensus.cc:1275] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 22 index: 17. Preceding OpId from leader: term: 24 index: 18. (index mismatch)
I20260709 12:24:22.453450 26009 raft_consensus.cc:2804] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:24:22.524858 25663 leader_election.cc:290] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:24:22.525476 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 47867.1us
W20260709 12:24:22.472790 17756 rpcz_store.cc:269] 0709 12:24:20.342766 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:20.344701 (+  1935us) service_pool.cc:224] Handling call
0709 12:24:22.464407 (+2119706us) spinlock_profiling.cc:243] Waited 1.06 s on lock 0x7b540009f1d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:24:22.464522 (+   115us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2432933248}
I20260709 12:24:22.528024 17683 raft_consensus.cc:3055] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:24:22.536093 17683 raft_consensus.cc:740] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:24:22.536351 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:22.537613 17756 raft_consensus.cc:2393] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 20: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:24:22.474373 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 18 candidate_status { last_received { term: 17 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:22.464258 26112 raft_consensus.cc:493] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:24:22.540566 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:22.376155 26090 consensus_queue.cc:237] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 21.15, Last appended by leader: 15, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:22.526737 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:22.452678 25776 raft_consensus.cc:3060] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:22.457101 25917 raft_consensus.cc:3055] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:24:22.542637 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:22.473877 25957 raft_consensus.cc:493] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:22.544008 17689 raft_consensus.cc:2468] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 20.
W20260709 12:24:22.535065 25384 consensus_meta.cc:287] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 1.058s	user 0.013s	sys 0.001s
I20260709 12:24:22.535306 26009 raft_consensus.cc:493] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:22.545379 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:22.548631 25905 consensus_queue.cc:1059] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 19.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20260709 12:24:22.549280 25657 consensus_queue.cc:1048] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 12:24:22.537353 17683 consensus_queue.cc:260] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 20.13, Last appended by leader: 13, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:22.575510 17683 raft_consensus.cc:3060] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Advancing to term 21
W20260709 12:24:22.474164 26014 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.416s	user 0.013s	sys 0.017s
I20260709 12:24:22.469959 26072 raft_consensus.cc:2749] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:24:22.535092 25826 leader_election.cc:290] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:22.550936 25951 raft_consensus.cc:493] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:22.550107 25858 raft_consensus.cc:493] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:22.550333 26112 raft_consensus.cc:515] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:22.552767 25917 raft_consensus.cc:740] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:24:22.466027 25901 leader_election.cc:290] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:22.466146 25868 leader_election.cc:290] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:22.560142 17729 leader_election.cc:304] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:24:22.561408 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 23 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:22.578934 26100 raft_consensus.cc:2749] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:24:22.616067 18371 raft_consensus.cc:2468] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 22.
W20260709 12:24:22.616387 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 57543.6us
I20260709 12:24:22.562283 17728 leader_election.cc:304] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:22.464845 25740 leader_election.cc:290] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:22.566543 17762 raft_consensus.cc:2468] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 26.
I20260709 12:24:22.544970 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 25 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:22.610224 26068 raft_consensus.cc:493] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:22.617835 17756 raft_consensus.cc:2410] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for term 25 because replica has last-logged OpId of term: 23 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 14.
I20260709 12:24:22.573469 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 17 candidate_status { last_received { term: 15 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:22.576855 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 23 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:22.572650 26028 raft_consensus.cc:493] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:22.566035 26009 raft_consensus.cc:3060] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:22.565928 25957 raft_consensus.cc:515] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:22.551970 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 25 candidate_status { last_received { term: 23 index: 19 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:22.565985 25384 raft_consensus.cc:515] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:22.586634 17759 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38786 (request call id 3756) took 1143 ms. Trace:
I20260709 12:24:22.589269 17683 raft_consensus.cc:1275] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 20 index: 13. Preceding OpId from leader: term: 21 index: 13. (term mismatch)
I20260709 12:24:22.598335 25915 raft_consensus.cc:493] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:22.598935 25637 raft_consensus.cc:493] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:22.600771 26055 log.cc:927] Time spent T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 1.605s	user 0.008s	sys 0.000s
I20260709 12:24:22.602178 18333 leader_election.cc:304] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:22.608920 17831 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35042 (request call id 3051) took 1146 ms. Trace:
I20260709 12:24:22.612368 17833 raft_consensus.cc:3060] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 15 FOLLOWER]: Advancing to term 16
W20260709 12:24:22.579242 26014 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.523s	user 0.030s	sys 0.022s
I20260709 12:24:22.618505 18363 raft_consensus.cc:2376] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 17. Re-sending same reply.
I20260709 12:24:22.618602 17835 raft_consensus.cc:2468] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 22.
I20260709 12:24:22.619566 17761 raft_consensus.cc:1240] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 24. Current term is 25. Ops: []
I20260709 12:24:22.621596 17683 pending_rounds.cc:85] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3: Aborting all ops after (but not including) 12
I20260709 12:24:22.633086 17683 pending_rounds.cc:107] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3: Aborting uncommitted NO_OP operation due to leader change: 20.13
I20260709 12:24:22.630430 18333 leader_election.cc:304] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
W20260709 12:24:22.631635 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 69472.5us
I20260709 12:24:22.643524 17833 raft_consensus.cc:2468] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 16.
I20260709 12:24:22.628369 17831 rpcz_store.cc:276] 0709 12:24:21.462706 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:21.479756 (+ 17050us) service_pool.cc:224] Handling call
0709 12:24:21.480106 (+   350us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:21.480322 (+   216us) raft_consensus.cc:1534] Early marking committed up to index 17
0709 12:24:21.609389 (+129067us) log.cc:844] Serialized 14 byte log entry
0709 12:24:22.460564 (+851175us) spinlock_profiling.cc:243] Waited 6.33 ms on lock 0x7b18002311c0. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b47515e07 00007f7b503709a9 00007f7b4753269b 00007f7b4753093f 00007f7b47532c02 00007f7b4a1ba20b 00007f7b4a1bfd23 00007f7b4a1c023a 00007f7b4a62aa2a
0709 12:24:22.608392 (+147828us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:22.608466 (+    74us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:24:22.608534 (+    68us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:22.608629 (+    95us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:22.608803 (+   174us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":23898,"prepare.queue_time_us":81144,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":34,"raft.queue_time_us":2373,"raft.run_cpu_time_us":269,"raft.run_wall_time_us":2760,"spinlock_wait_cycles":14555520,"thread_start_us":233240,"threads_started":2}
I20260709 12:24:22.633404 17683 raft_consensus.cc:2889] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:24:22.639173 17727 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:22.634469 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:24:22.664963 17801 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:22.640010 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:22.668208 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:22.696728 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:22.697557 17762 raft_consensus.cc:3060] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:22.621469 17759 rpcz_store.cc:276] 0709 12:24:21.443345 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:22.339029 (+895684us) service_pool.cc:224] Handling call
0709 12:24:22.339350 (+   321us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:22.356211 (+ 16861us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:22.356317 (+   106us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:22.369028 (+ 12711us) log.cc:844] Serialized 26 byte log entry
0709 12:24:22.453386 (+ 84358us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:24:22.453427 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:22.453507 (+    80us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:22.586317 (+132810us) raft_consensus.cc:1685] finished
0709 12:24:22.586351 (+    34us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:22.586556 (+   205us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":33374,"raft.run_cpu_time_us":105,"raft.run_wall_time_us":106,"thread_start_us":57261,"threads_started":2,"wal-append.queue_time_us":184571}
I20260709 12:24:22.699209 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:22.614172 25951 raft_consensus.cc:515] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:22.703420 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 21 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:22.700495 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 23 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:24:22.701792 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 71429.1us
I20260709 12:24:22.633581 26024 raft_consensus.cc:2749] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:24:22.704095 17760 raft_consensus.cc:2468] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 20.
I20260709 12:24:22.704506 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 16 candidate_status { last_received { term: 15 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:22.723986 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:22.724540 17760 raft_consensus.cc:3060] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:22.724869 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 23 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:22.705869 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:22.706362 18333 leader_election.cc:304] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:22.692652 26117 raft_consensus.cc:2804] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:24:22.614871 25858 raft_consensus.cc:515] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:22.619632 26120 raft_consensus.cc:2804] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:24:22.729733 26120 raft_consensus.cc:493] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:22.730382 26120 raft_consensus.cc:3060] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:22.732103 17760 raft_consensus.cc:2468] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 25.
I20260709 12:24:22.733135 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 23 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:22.734213 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:22.734788 17760 raft_consensus.cc:2468] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 24.
I20260709 12:24:22.735703 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:22.736876 17760 raft_consensus.cc:2393] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 22: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:24:22.737689 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 23 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:22.738636 17760 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d"
dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
mode: GRACEFUL
new_leader_uuid: "e063541542a94969898868bf6eb5ce1b"
 from {username='slave'} at 127.0.0.1:49786
I20260709 12:24:22.739202 17760 raft_consensus.cc:606] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 24 LEADER]: Received request to transfer leadership to TS e063541542a94969898868bf6eb5ce1b
I20260709 12:24:22.738950 26120 raft_consensus.cc:515] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Starting leader election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:22.706514 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:22.730783 25858 leader_election.cc:290] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:22.741600 17759 raft_consensus.cc:2393] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 23: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:24:22.742055 17760 raft_consensus.cc:1240] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 21 LEADER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 20. Current term is 21. Ops: [20.13-20.13]
I20260709 12:24:22.743566 26120 leader_election.cc:290] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 23 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:22.743830 17760 raft_consensus.cc:1275] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 23 index: 16. Preceding OpId from leader: term: 24 index: 17. (index mismatch)
I20260709 12:24:22.746063 26117 raft_consensus.cc:697] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 20 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:24:22.746986 26117 consensus_queue.cc:237] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.12, Last appended by leader: 12, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:22.747397 25959 consensus_queue.cc:1048] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:24:22.750669 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 23 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:22.691272 25776 raft_consensus.cc:493] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:24:22.692328 26058 raft_consensus.cc:493] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:22.755596 17834 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35042 (request call id 3057) took 1280 ms. Trace:
I20260709 12:24:22.763032 17762 raft_consensus.cc:2468] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 25.
I20260709 12:24:22.756170 17834 rpcz_store.cc:276] 0709 12:24:21.475636 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:22.410953 (+935317us) service_pool.cc:224] Handling call
0709 12:24:22.411298 (+   345us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:22.425893 (+ 14595us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:22.425923 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:22.426331 (+   408us) log.cc:844] Serialized 26 byte log entry
0709 12:24:22.563103 (+136772us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:24:22.563155 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:22.563247 (+    92us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:22.735667 (+172420us) raft_consensus.cc:1685] finished
0709 12:24:22.735699 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:22.755517 (+ 19818us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":136431,"threads_started":1,"wal-append.queue_time_us":195267}
W20260709 12:24:22.770645 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 64335us
I20260709 12:24:22.703151 26014 raft_consensus.cc:2749] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:24:22.770560 26131 raft_consensus.cc:2749] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 12:24:22.631420 25915 raft_consensus.cc:515] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:22.634076 25637 raft_consensus.cc:515] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:22.772289 18333 leader_election.cc:304] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:22.621625 26068 raft_consensus.cc:515] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:22.628201 26028 raft_consensus.cc:515] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:22.783740 18333 leader_election.cc:304] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:22.792258 18333 leader_election.cc:304] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:24:22.793327 26131 raft_consensus.cc:2804] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:24:22.793701 26131 raft_consensus.cc:493] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:22.794020 26131 raft_consensus.cc:3060] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:22.785868 25384 leader_election.cc:290] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 25 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:22.798527 17830 raft_consensus.cc:1275] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 25 index: 14. Preceding OpId from leader: term: 26 index: 14. (term mismatch)
I20260709 12:24:22.799021 17830 pending_rounds.cc:85] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b: Aborting all ops after (but not including) 13
I20260709 12:24:22.799319 17830 pending_rounds.cc:107] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b: Aborting uncommitted NO_OP operation due to leader change: 25.14
I20260709 12:24:22.799611 17830 raft_consensus.cc:2889] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:22.802311 25957 leader_election.cc:290] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:22.802953 17830 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35042 (request call id 3054) took 1339 ms. Trace:
I20260709 12:24:22.804067 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 21 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:22.804704 17688 raft_consensus.cc:2468] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 20.
I20260709 12:24:22.805593 26024 raft_consensus.cc:2749] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 12:24:22.807700 25951 leader_election.cc:290] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:22.773344 26058 raft_consensus.cc:515] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:22.698381 25917 consensus_queue.cc:260] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 20.14, Last appended by leader: 14, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:22.812613 25917 raft_consensus.cc:3060] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Advancing to term 21
W20260709 12:24:22.813989 18334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:22.814559 18334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:22.815980 26028 leader_election.cc:290] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:22.786793 26014 raft_consensus.cc:2749] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 12:24:22.768605 25776 raft_consensus.cc:515] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:24:22.781075 26055 log.cc:927] Time spent T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.076s	user 0.000s	sys 0.000s
I20260709 12:24:22.820288 17728 leader_election.cc:304] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:22.826861 17758 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38776 (request call id 2969) took 2049 ms. Trace:
I20260709 12:24:22.705682 25909 raft_consensus.cc:515] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:22.823460 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 16 candidate_status { last_received { term: 15 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:22.811547 26058 leader_election.cc:290] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:24:22.822216 26128 log.cc:927] Time spent T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.117s	user 0.001s	sys 0.000s
I20260709 12:24:22.803431 17830 rpcz_store.cc:276] 0709 12:24:21.463131 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:22.155557 (+692426us) service_pool.cc:224] Handling call
0709 12:24:22.155797 (+   240us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:22.800170 (+644373us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:22.802653 (+  2483us) spinlock_profiling.cc:243] Waited 642 ms on lock 0x7b5400267354. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:24:22.802879 (+   226us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1477343488}
I20260709 12:24:22.830453 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
W20260709 12:24:22.817305 26121 log.cc:927] Time spent T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.202s	user 0.001s	sys 0.000s
I20260709 12:24:22.832327 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 25 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:22.827265 17758 rpcz_store.cc:276] 0709 12:24:20.777467 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:20.782184 (+  4717us) service_pool.cc:224] Handling call
0709 12:24:20.782640 (+   456us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:24:20.783124 (+   484us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 12:24:20.783465 (+   341us) log.cc:844] Serialized 14 byte log entry
0709 12:24:20.907925 (+124460us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:24:21.113664 (+205739us) spinlock_profiling.cc:243] Waited 6.57 ms on lock 0x7b2c0000e5c8. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b48f19a11 00007f7b48f29372 00007f7b4a65ba01 00007f7b4a62043a 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2
0709 12:24:21.114123 (+   459us) log.cc:844] Serialized 52 byte log entry
0709 12:24:21.114281 (+   158us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:24:21.114514 (+   233us) log.cc:844] Serialized 14 byte log entry
0709 12:24:21.156598 (+ 42084us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:21.156682 (+    84us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:22.599827 (+1443145us) spinlock_profiling.cc:243] Waited 2.45 ms on lock 0x7b4400022b4c. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b47459c85 00007f7b4745a6a2 00007f7b4a1cc1db 00007f7b4a1c7aaf 00007f7b4a1bb86e 00007f7b4a1babb6 00007f7b4a1ba7d8 00007f7b4a1c4221 00007f7b4a1c41d9 00007f7b4a1c4169
0709 12:24:22.826641 (+226814us) raft_consensus.cc:1685] finished
0709 12:24:22.826662 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:22.826803 (+   141us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":44746,"prepare.queue_time_us":490,"prepare.run_cpu_time_us":62,"prepare.run_wall_time_us":64,"raft.queue_time_us":50843,"raft.run_cpu_time_us":397,"raft.run_wall_time_us":3438,"spinlock_wait_cycles":20750976,"thread_start_us":123943,"threads_started":1,"wal-append.queue_time_us":148309}
I20260709 12:24:22.835500 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:22.842456 26131 raft_consensus.cc:515] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Starting leader election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:22.845083 26131 leader_election.cc:290] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 23 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:22.845278 17833 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37994 (request call id 1520) took 1346 ms. Trace:
I20260709 12:24:22.845675 26131 raft_consensus.cc:493] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:22.845806 17833 rpcz_store.cc:276] 0709 12:24:21.499326 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:21.645381 (+146055us) service_pool.cc:224] Handling call
0709 12:24:22.845052 (+1199671us) spinlock_profiling.cc:243] Waited 966 ms on lock 0x7b54000ab9d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:24:22.845202 (+   150us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2221762944}
I20260709 12:24:22.846307 26131 raft_consensus.cc:515] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:22.848752 26131 leader_election.cc:290] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:24:22.849079 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:22.849946 26131 raft_consensus.cc:493] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:22.851253 26131 raft_consensus.cc:515] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:22.853530 26131 leader_election.cc:290] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:22.854805 26060 consensus_queue.cc:1048] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:24:22.857736 26060 consensus_queue.cc:1059] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 21.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:24:22.857761 17764 raft_consensus.cc:1275] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 24 index: 15. Preceding OpId from leader: term: 25 index: 15. (term mismatch)
I20260709 12:24:22.862910 17764 pending_rounds.cc:85] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb: Aborting all ops after (but not including) 14
I20260709 12:24:22.863173 17764 pending_rounds.cc:107] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb: Aborting uncommitted NO_OP operation due to leader change: 24.15
I20260709 12:24:22.863435 17764 raft_consensus.cc:2889] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:22.843663 17755 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36572 (request call id 1428) took 1465 ms. Trace:
I20260709 12:24:22.865463 17755 rpcz_store.cc:276] 0709 12:24:21.378302 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:21.379615 (+  1313us) service_pool.cc:224] Handling call
0709 12:24:21.380016 (+   401us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:21.467493 (+ 87477us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:21.467525 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:21.477454 (+  9929us) log.cc:844] Serialized 26 byte log entry
0709 12:24:21.676593 (+199139us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:24:21.721063 (+ 44470us) log.cc:844] Serialized 14 byte log entry
0709 12:24:22.270308 (+549245us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:22.843121 (+572813us) spinlock_profiling.cc:243] Waited 73.4 ms on lock 0x7b54003eb554. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:24:22.843187 (+    66us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:22.843252 (+    65us) raft_consensus.cc:1685] finished
0709 12:24:22.843386 (+   134us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:22.843568 (+   182us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":463977,"prepare.queue_time_us":41201,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":43,"raft.queue_time_us":495569,"raft.run_cpu_time_us":240,"raft.run_wall_time_us":41437,"spinlock_wait_cycles":168781184,"thread_start_us":254439,"threads_started":2,"wal-append.queue_time_us":522738}
W20260709 12:24:22.870211 17804 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:22.873876 17838 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35042 (request call id 3052) took 1411 ms. Trace:
I20260709 12:24:22.874306 17838 rpcz_store.cc:276] 0709 12:24:21.462848 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:21.494145 (+ 31297us) service_pool.cc:224] Handling call
0709 12:24:21.494419 (+   274us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:22.099957 (+605538us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:22.873661 (+773704us) spinlock_profiling.cc:243] Waited 393 ms on lock 0x7b54002ebb54. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:24:22.873814 (+   153us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":903921408}
I20260709 12:24:22.875988 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 18 candidate_status { last_received { term: 17 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:22.876644 18366 raft_consensus.cc:3060] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:22.877739 18361 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 2353) took 1388 ms. Trace:
I20260709 12:24:22.878118 18361 rpcz_store.cc:276] 0709 12:24:21.489757 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:21.491956 (+  2199us) service_pool.cc:224] Handling call
0709 12:24:21.497572 (+  5616us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:21.498220 (+   648us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:22.877574 (+1379354us) spinlock_profiling.cc:243] Waited 5.28 ms on lock 0x7b540034b550. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a61f8ad 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:24:22.877690 (+   116us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":12135296}
I20260709 12:24:22.880013 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:22.894384 18368 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59902 (request call id 1911) took 1116 ms. Trace:
I20260709 12:24:22.895447 18361 raft_consensus.cc:1240] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 20. Current term is 21. Ops: [20.13-20.13]
I20260709 12:24:22.895008 18368 rpcz_store.cc:276] 0709 12:24:21.777945 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:22.203339 (+425394us) service_pool.cc:224] Handling call
0709 12:24:22.227009 (+ 23670us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:22.227745 (+   736us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:22.894093 (+666348us) spinlock_profiling.cc:243] Waited 11.1 ms on lock 0x7b54004687d0. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a61f8ad 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:24:22.894287 (+   194us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":25513856}
W20260709 12:24:22.899479 26067 log.cc:927] Time spent T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 1.842s	user 0.003s	sys 0.000s
W20260709 12:24:22.901571 26066 log.cc:927] Time spent T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 1.844s	user 0.003s	sys 0.000s
I20260709 12:24:22.902066 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:22.899497 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 21 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:22.906075 26060 raft_consensus.cc:493] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:22.906492 18361 raft_consensus.cc:2410] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for term 21 because replica has last-logged OpId of term: 19 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 14.
I20260709 12:24:22.906770 26060 raft_consensus.cc:515] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:22.909292 26114 raft_consensus.cc:3055] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:24:22.909598 26060 leader_election.cc:290] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:22.909901 26114 raft_consensus.cc:740] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:24:22.911028 26114 consensus_queue.cc:260] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 19.14, Last appended by leader: 14, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:22.912995 26114 raft_consensus.cc:3060] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:22.913654 25688 raft_consensus.cc:493] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:22.914175 25688 raft_consensus.cc:515] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:22.916815 25688 leader_election.cc:290] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:22.918100 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:22.918713 17758 raft_consensus.cc:2468] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 23.
I20260709 12:24:22.919620 17682 raft_consensus.cc:1240] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 24. Current term is 25. Ops: [24.14-24.14]
I20260709 12:24:22.920895 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:22.921623 17682 raft_consensus.cc:2393] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 20: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:24:22.922492 17682 raft_consensus.cc:1240] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 22 LEADER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 21. Current term is 22. Ops: []
W20260709 12:24:22.932132 17803 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:22.935685 17803 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:22.941431 25881 raft_consensus.cc:3055] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:24:22.942041 25881 raft_consensus.cc:740] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 24 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:24:22.947854 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:22.948786 18361 raft_consensus.cc:2468] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 25.
W20260709 12:24:22.949906 17652 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:22.953503 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:22.954341 17833 raft_consensus.cc:2368] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for earlier term 24. Current term is 25.
W20260709 12:24:22.956393 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:22.957104 18362 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59886 (request call id 2151) took 1533 ms. Trace:
I20260709 12:24:22.957600 18362 rpcz_store.cc:276] 0709 12:24:21.424947 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:21.425125 (+   178us) service_pool.cc:224] Handling call
0709 12:24:21.425527 (+   402us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:21.493537 (+ 68010us) raft_consensus.cc:1534] Early marking committed up to index 17
0709 12:24:21.671208 (+177671us) log.cc:844] Serialized 14 byte log entry
0709 12:24:21.738485 (+ 67277us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:21.738568 (+    83us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:24:21.738647 (+    79us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:22.956803 (+1218156us) spinlock_profiling.cc:243] Waited 67.8 ms on lock 0x7b54005460d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:24:22.956848 (+    45us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:22.957019 (+   171us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":27930,"prepare.queue_time_us":415543,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":45,"raft.queue_time_us":14994,"raft.run_cpu_time_us":173,"raft.run_wall_time_us":9022,"spinlock_wait_cycles":155840896,"thread_start_us":216709,"threads_started":2,"wal-append.queue_time_us":241174}
I20260709 12:24:22.960088 18365 raft_consensus.cc:1240] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 22. Current term is 23. Ops: []
I20260709 12:24:22.961800 18365 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59902 (request call id 1915) took 1182 ms. Trace:
I20260709 12:24:22.962193 18365 rpcz_store.cc:276] 0709 12:24:21.779496 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:22.214296 (+434800us) service_pool.cc:224] Handling call
0709 12:24:22.959851 (+745555us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:22.960595 (+   744us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:22.961579 (+   984us) spinlock_profiling.cc:243] Waited 745 ms on lock 0x7b54005460d0. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a61f8ad 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:24:22.961735 (+   156us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1713933696}
I20260709 12:24:22.955432 25384 consensus_queue.cc:1059] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 19.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.014s
I20260709 12:24:22.976737 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 25 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
W20260709 12:24:22.980638 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 138593us
I20260709 12:24:22.985594 17838 raft_consensus.cc:1275] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 18 index: 14. Preceding OpId from leader: term: 19 index: 15. (index mismatch)
I20260709 12:24:22.987468 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 26 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:22.989009 17683 raft_consensus.cc:2393] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 23: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:24:22.943152 25881 consensus_queue.cc:260] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 24.15, Last appended by leader: 15, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:22.987358 26004 raft_consensus.cc:493] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:22.997866 25881 raft_consensus.cc:3060] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:22.997660 26004 raft_consensus.cc:515] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:22.833868 25784 raft_consensus.cc:2804] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:24:22.833369 25761 raft_consensus.cc:493] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:22.995563 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:24:23.003474 26075 log.cc:927] Time spent T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 1.351s	user 0.003s	sys 0.000s
W20260709 12:24:23.005331 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 213147us
I20260709 12:24:23.021970 17680 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37402 (request call id 3480) took 2368 ms. Trace:
I20260709 12:24:23.022359 17680 rpcz_store.cc:276] 0709 12:24:20.653546 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:20.671214 (+ 17668us) service_pool.cc:224] Handling call
0709 12:24:20.685445 (+ 14231us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 12:24:20.698242 (+ 12797us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:20.698265 (+    23us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 12:24:20.716222 (+ 17957us) log.cc:844] Serialized 78 byte log entry
0709 12:24:21.218982 (+502760us) spinlock_profiling.cc:243] Waited 15.7 ms on lock 0x7b180033d900. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b47515e07 00007f7b503709a9 00007f7b4753269b 00007f7b4753093f 00007f7b47532c02 00007f7b4a1ba20b 00007f7b4a1bfd23 00007f7b4a1c00d1 00007f7b4a5dba0f
0709 12:24:21.444218 (+225236us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:24:21.444424 (+   206us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:21.444604 (+   180us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:23.002763 (+1558159us) spinlock_profiling.cc:243] Waited 4.15 ms on lock 0x7b4400022b4c. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b47459c85 00007f7b4745a6a2 00007f7b4a1cc1db 00007f7b4a1c7aaf 00007f7b4a1bb86e 00007f7b4a1babb6 00007f7b4a1ba7d8 00007f7b4a1c4221 00007f7b4a1c41d9 00007f7b4a1c4169
0709 12:24:23.021727 (+ 18964us) raft_consensus.cc:1685] finished
0709 12:24:23.021748 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:23.021910 (+   162us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4384,"raft.queue_time_us":9686,"raft.run_cpu_time_us":134,"raft.run_wall_time_us":7169,"spinlock_wait_cycles":45611264,"thread_start_us":216485,"threads_started":1,"wal-append.queue_time_us":902876}
I20260709 12:24:23.027038 18366 raft_consensus.cc:2468] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 18.
I20260709 12:24:23.027675 26004 leader_election.cc:290] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:24:23.032100 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 113440us
I20260709 12:24:23.033794 17803 leader_election.cc:304] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:22.955389 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:23.035663 17832 raft_consensus.cc:2468] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 19.
W20260709 12:24:23.037561 17730 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:23.035310 17801 leader_election.cc:304] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:22.985301 26112 leader_election.cc:290] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:23.039968 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 25 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:23.040807 17803 leader_election.cc:304] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:23.040949 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 26 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:23.041623 17757 raft_consensus.cc:2468] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 25.
I20260709 12:24:23.034142 18333 leader_election.cc:304] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:22.999639 25951 raft_consensus.cc:493] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:23.044299 18366 raft_consensus.cc:1275] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 22 index: 18. Preceding OpId from leader: term: 24 index: 19. (index mismatch)
I20260709 12:24:23.044636 18362 raft_consensus.cc:1275] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 23 index: 15. Preceding OpId from leader: term: 25 index: 16. (index mismatch)
I20260709 12:24:23.052213 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 25 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:23.045730 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:23.053898 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:23.054181 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:23.046568 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:23.006350 25784 raft_consensus.cc:493] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:23.020301 25761 raft_consensus.cc:515] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:23.047518 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:23.048130 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 24 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:23.050107 18365 raft_consensus.cc:3060] T a5732e71b9d54f8786b8c511d69943bf P 334a4e1f59944544b3b3ff36616b57da [term 22 LEARNER]: Advancing to term 24
I20260709 12:24:23.039533 25736 raft_consensus.cc:493] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:24:23.058116 18371 raft_consensus.cc:1275] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 20 index: 14. (index mismatch)
I20260709 12:24:23.059051 18368 raft_consensus.cc:1275] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 18 index: 14. Preceding OpId from leader: term: 20 index: 15. (index mismatch)
I20260709 12:24:23.053941 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 21 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:23.059607 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:23.076208 18366 raft_consensus.cc:2410] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for term 21 because replica has last-logged OpId of term: 19 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 14.
I20260709 12:24:23.047590 25688 raft_consensus.cc:2804] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Leader election won for term 25
W20260709 12:24:23.047030 26062 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.499s	user 0.009s	sys 0.006s
I20260709 12:24:23.052007 17729 leader_election.cc:304] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:23.062337 18363 raft_consensus.cc:1240] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 19. Current term is 20. Ops: []
I20260709 12:24:23.065464 25657 raft_consensus.cc:993] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb: : Instructing follower e063541542a94969898868bf6eb5ce1b to start an election
W20260709 12:24:23.039453 25972 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 21 current term: 21 is higher.
I20260709 12:24:23.048300 25740 raft_consensus.cc:2749] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:24:23.069753 25909 leader_election.cc:290] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 23 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:23.054739 17760 raft_consensus.cc:2468] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 22.
I20260709 12:24:23.071750 17680 raft_consensus.cc:3060] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Advancing to term 25
W20260709 12:24:23.073137 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 45332.9us
I20260709 12:24:23.045433 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 23 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:23.039971 25944 consensus_queue.cc:1059] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.041s
I20260709 12:24:23.050261 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 23 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:23.070156 25951 raft_consensus.cc:515] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:23.045640 26014 raft_consensus.cc:2749] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:24:23.047785 18334 leader_election.cc:304] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:23.081821 18362 raft_consensus.cc:3055] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:24:23.083950 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:23.235118 18362 raft_consensus.cc:740] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:24:23.084666 17654 leader_election.cc:304] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
W20260709 12:24:23.110757 17655 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:23.122349 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:23.238013 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 153479us
I20260709 12:24:23.046031 18333 leader_election.cc:304] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:24:23.239102 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:23.070842 25784 raft_consensus.cc:3060] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Advancing to term 21
W20260709 12:24:23.237430 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 194113us
I20260709 12:24:23.236817 18361 raft_consensus.cc:2393] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 23: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:24:23.134742 18365 raft_consensus.cc:1275] T a5732e71b9d54f8786b8c511d69943bf P 334a4e1f59944544b3b3ff36616b57da [term 24 LEARNER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 19 index: 12. Preceding OpId from leader: term: 24 index: 14. (index mismatch)
I20260709 12:24:23.236083 18362 consensus_queue.cc:260] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 20.16, Last appended by leader: 16, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:23.081656 26062 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.551s	user 0.009s	sys 0.007s
W20260709 12:24:23.142055 26136 log.cc:927] Time spent T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.057s	user 0.000s	sys 0.001s
W20260709 12:24:23.172915 26083 log.cc:927] Time spent T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 1.400s	user 0.003s	sys 0.002s
I20260709 12:24:23.241573 17687 raft_consensus.cc:2393] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 23: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:24:23.253545 17654 leader_election.cc:304] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:24:23.056386 26072 raft_consensus.cc:493] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
W20260709 12:24:23.082070 26003 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.307s	user 0.013s	sys 0.021s
W20260709 12:24:23.219277 26009 consensus_meta.cc:287] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.586s	user 0.004s	sys 0.004s
W20260709 12:24:23.236817 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 192873us
I20260709 12:24:23.240756 25995 raft_consensus.cc:2749] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
W20260709 12:24:23.282984 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 234382us
I20260709 12:24:23.288262 26009 raft_consensus.cc:515] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:23.288705 18334 leader_election.cc:304] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:23.240495 26117 raft_consensus.cc:1081] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 24 LEADER]: Signalling peer e063541542a94969898868bf6eb5ce1b to start an election
I20260709 12:24:23.213881 25915 leader_election.cc:290] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:23.275065 17764 raft_consensus.cc:1240] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 LEADER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 20. Current term is 21. Ops: []
I20260709 12:24:23.290988 26009 leader_election.cc:290] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 27 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:23.227718 26068 leader_election.cc:290] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:23.240584 26107 raft_consensus.cc:2804] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:24:23.276520 17684 raft_consensus.cc:1275] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 14 index: 12. Preceding OpId from leader: term: 16 index: 13. (index mismatch)
I20260709 12:24:23.266150 18362 raft_consensus.cc:3060] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:23.154068 25970 raft_consensus.cc:3055] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:24:23.213505 25637 leader_election.cc:290] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:23.227156 25776 leader_election.cc:290] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:24:23.241339 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 189464us
I20260709 12:24:23.249857 26114 raft_consensus.cc:2749] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 12:24:23.143347 25688 raft_consensus.cc:697] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 25 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:24:23.305430 25740 raft_consensus.cc:2804] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:24:23.341640 25740 raft_consensus.cc:493] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:23.241065 25761 leader_election.cc:290] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:23.306622 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:23.342149 25688 consensus_queue.cc:237] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 23.13, Last appended by leader: 13, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:23.342170 25740 raft_consensus.cc:3060] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:23.267800 26137 raft_consensus.cc:2749] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260709 12:24:23.321316 26002 raft_consensus.cc:3060] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:23.080622 25736 raft_consensus.cc:515] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:23.331954 18362 raft_consensus.cc:1275] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 20 index: 16. Preceding OpId from leader: term: 19 index: 15. (term mismatch)
I20260709 12:24:23.348527 25736 leader_election.cc:290] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:23.348762 18362 pending_rounds.cc:85] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da: Aborting all ops after (but not including) 14
I20260709 12:24:23.349135 18362 pending_rounds.cc:107] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da: Aborting uncommitted NO_OP operation due to leader change: 18.15
I20260709 12:24:23.349390 18362 raft_consensus.cc:2889] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:23.349745 18362 pending_rounds.cc:107] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da: Aborting uncommitted NO_OP operation due to leader change: 20.16
I20260709 12:24:23.350052 18362 raft_consensus.cc:2889] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:23.351418 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 21 candidate_status { last_received { term: 19 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:23.352190 18362 raft_consensus.cc:2468] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 20.
I20260709 12:24:23.351823 17683 raft_consensus.cc:2955] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Committing config change with OpId 19.12: config changed from index -1 to 12, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } }
I20260709 12:24:23.353205 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 23 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:23.353929 18362 raft_consensus.cc:2393] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 23: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:24:23.355207 18362 raft_consensus.cc:1275] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 18 index: 14. Preceding OpId from leader: term: 19 index: 15. (index mismatch)
I20260709 12:24:23.336763 26068 raft_consensus.cc:493] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:23.313959 17804 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:23.294694 26003 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.520s	user 0.026s	sys 0.027s
I20260709 12:24:23.356482 26068 raft_consensus.cc:515] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:23.357299 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:23.357324 17727 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:23.358634 26068 leader_election.cc:290] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:23.091773 25663 consensus_queue.cc:1048] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.236s
I20260709 12:24:23.306000 25970 raft_consensus.cc:740] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:24:23.043454 26004 consensus_queue.cc:1048] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.249s
I20260709 12:24:23.363224 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 21 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:23.364102 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 25 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:23.364498 18362 raft_consensus.cc:1275] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 19 index: 16. Preceding OpId from leader: term: 22 index: 18. (index mismatch)
W20260709 12:24:23.365746 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:23.128278 25866 consensus_queue.cc:1048] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.174s
I20260709 12:24:23.368100 18363 raft_consensus.cc:1240] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 21. Current term is 22. Ops: []
I20260709 12:24:23.296434 25979 raft_consensus.cc:2804] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:24:23.369614 17654 leader_election.cc:304] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:24:23.325796 25915 raft_consensus.cc:2749] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:24:23.378917 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:23.379102 26003 raft_consensus.cc:2804] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:24:23.379609 26003 raft_consensus.cc:493] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:23.380108 26003 raft_consensus.cc:3060] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:23.381031 25951 leader_election.cc:290] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:23.382745 25866 consensus_queue.cc:1059] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 22.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20260709 12:24:23.379760 17687 raft_consensus.cc:2393] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 21: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:24:23.383896 26150 raft_consensus.cc:3055] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:24:23.384307 26150 raft_consensus.cc:740] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 24 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
W20260709 12:24:23.386130 26129 log.cc:927] Time spent T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.602s	user 0.001s	sys 0.001s
W20260709 12:24:23.387511 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 341501us
W20260709 12:24:23.388476 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 58042.4us
I20260709 12:24:23.312654 26107 raft_consensus.cc:493] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:23.385219 26150 consensus_queue.cc:260] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 24.14, Last appended by leader: 14, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:23.407544 25740 raft_consensus.cc:515] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:23.422499 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 381725us
W20260709 12:24:23.518164 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 244954us
I20260709 12:24:23.404325 26107 raft_consensus.cc:3060] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:23.300194 26072 raft_consensus.cc:515] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:23.629571 25784 raft_consensus.cc:515] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:23.631179 26096 raft_consensus.cc:493] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:23.631067 18333 leader_election.cc:304] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:23.632074 25638 raft_consensus.cc:493] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:23.440191 26100 consensus_queue.cc:1048] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
W20260709 12:24:23.632566 25881 consensus_meta.cc:287] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.634s	user 0.009s	sys 0.015s
I20260709 12:24:23.393533 26112 consensus_queue.cc:1048] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.049s
I20260709 12:24:23.680431 25915 raft_consensus.cc:2749] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:24:23.680644 17829 raft_consensus.cc:2468] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 25.
I20260709 12:24:23.633865 25740 leader_election.cc:290] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 23 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:24:23.681985 26143 log.cc:927] Time spent T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.516s	user 0.002s	sys 0.000s
W20260709 12:24:23.635668 18334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:23.683048 25951 raft_consensus.cc:493] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:23.683583 25951 raft_consensus.cc:515] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:23.685699 25951 leader_election.cc:290] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:24:23.636536 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 248741us
I20260709 12:24:23.386832 25858 raft_consensus.cc:3055] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:24:23.642004 17801 leader_election.cc:304] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:23.646457 25995 raft_consensus.cc:493] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:23.646437 17680 raft_consensus.cc:2468] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 25.
I20260709 12:24:23.672567 25979 raft_consensus.cc:697] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 18 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
W20260709 12:24:23.678751 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 47804us
I20260709 12:24:23.679183 26028 raft_consensus.cc:476] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 20 LEADER]: Not starting pre-election -- already a leader
I20260709 12:24:23.679267 25638 raft_consensus.cc:515] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:23.647135 26120 raft_consensus.cc:2955] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 24 LEADER]: Committing config change with OpId 19.12: config changed from index -1 to 12, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } }
I20260709 12:24:23.718451 25638 leader_election.cc:290] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:24:23.649474 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 256917us
I20260709 12:24:23.719141 17761 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38776 (request call id 2982) took 1244 ms. Trace:
I20260709 12:24:23.719410 17729 leader_election.cc:304] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:23.719545 17761 rpcz_store.cc:276] 0709 12:24:22.474911 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:22.475254 (+   343us) service_pool.cc:224] Handling call
0709 12:24:22.475590 (+   336us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:22.690591 (+215001us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:23.718862 (+1028271us) spinlock_profiling.cc:243] Waited 144 ms on lock 0x7b54002b6454. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:24:23.719067 (+   205us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":330641664}
I20260709 12:24:23.721349 17764 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36572 (request call id 1435) took 1082 ms. Trace:
W20260709 12:24:23.695469 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 53872us
I20260709 12:24:23.695899 18333 leader_election.cc:304] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:23.674816 17803 leader_election.cc:304] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
W20260709 12:24:23.698407 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 422410us
W20260709 12:24:23.724524 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 83707.1us
I20260709 12:24:23.724824 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:23.725320 17803 leader_election.cc:304] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:24:23.700309 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:23.727370 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:23.721769 17764 rpcz_store.cc:276] 0709 12:24:22.638974 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:22.864488 (+225514us) service_pool.cc:224] Handling call
0709 12:24:22.864849 (+   361us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:23.290878 (+426029us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:23.721138 (+430260us) spinlock_profiling.cc:243] Waited 410 ms on lock 0x7b540042c054. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:24:23.721294 (+   156us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":942982016}
I20260709 12:24:23.725488 17685 raft_consensus.cc:2468] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 20.
I20260709 12:24:23.729984 17688 raft_consensus.cc:2468] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 27.
W20260709 12:24:23.734555 17655 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:23.690753 25858 raft_consensus.cc:740] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:24:23.748689 18374 raft_consensus.cc:1240] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 19. Current term is 20. Ops: []
I20260709 12:24:23.744679 25663 raft_consensus.cc:2749] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:24:23.749455 25858 consensus_queue.cc:260] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 19.14, Last appended by leader: 14, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:23.750823 25858 raft_consensus.cc:3060] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:23.752256 26028 raft_consensus.cc:493] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:23.752945 26028 raft_consensus.cc:515] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:23.753602 18333 leader_election.cc:304] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:23.754758 18333 leader_election.cc:304] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:23.742162 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 23 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:23.755259 18334 leader_election.cc:304] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:24:23.756202 18334 leader_election.cc:304] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:24:23.757218 26068 raft_consensus.cc:2749] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:24:23.757967 26068 raft_consensus.cc:2804] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:24:23.758404 26068 raft_consensus.cc:493] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:24:23.758653 26130 log.cc:927] Time spent T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.979s	user 0.002s	sys 0.000s
I20260709 12:24:23.758843 26068 raft_consensus.cc:3060] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:23.743552 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:23.760082 26137 raft_consensus.cc:2804] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:24:23.760196 18371 raft_consensus.cc:2468] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 25.
I20260709 12:24:23.761250 18333 leader_election.cc:304] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:23.696964 25995 raft_consensus.cc:515] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:23.672200 25736 consensus_queue.cc:1048] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.022s
I20260709 12:24:23.686902 26096 raft_consensus.cc:515] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:23.744598 25979 consensus_queue.cc:237] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 17.17, Last appended by leader: 17, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:23.690819 26009 consensus_queue.cc:1048] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.056s
I20260709 12:24:23.690385 25915 raft_consensus.cc:493] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:23.721959 25740 raft_consensus.cc:2804] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:24:23.690402 26154 consensus_queue.cc:1059] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.061s
I20260709 12:24:23.738273 25638 consensus_queue.cc:1048] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.010s
I20260709 12:24:23.741513 25881 raft_consensus.cc:3060] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:23.754843 26072 leader_election.cc:290] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:23.759615 25784 leader_election.cc:290] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 21 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:23.740635 17729 leader_election.cc:304] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:23.762616 18333 leader_election.cc:304] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:23.763263 17684 raft_consensus.cc:1275] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 20 index: 13. (index mismatch)
I20260709 12:24:23.755288 25917 raft_consensus.cc:2749] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:24:23.763513 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 21 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:23.787623 17688 raft_consensus.cc:3060] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:23.760584 26137 raft_consensus.cc:493] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:23.741704 26002 raft_consensus.cc:2749] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:24:23.785053 17652 leader_election.cc:400] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 24 pre-election: Vote denied by peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837) with higher term. Message: Invalid argument: T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for earlier term 24. Current term is 25.
I20260709 12:24:23.787969 18333 leader_election.cc:304] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:24:23.782313 17654 leader_election.cc:304] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:24:23.789609 18364 raft_consensus.cc:2410] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 19 because replica has last-logged OpId of term: 18 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 14.
I20260709 12:24:23.784090 25740 raft_consensus.cc:493] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:24:23.789451 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 48938.9us
I20260709 12:24:23.765465 25637 raft_consensus.cc:2749] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:24:23.800170 25740 raft_consensus.cc:3060] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Advancing to term 25
W20260709 12:24:23.794135 25944 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.519s	user 0.008s	sys 0.033s
I20260709 12:24:23.794260 26097 raft_consensus.cc:2804] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:24:23.803342 17729 leader_election.cc:304] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:23.791103 17652 leader_election.cc:403] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 24 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:24:23.791970 26014 raft_consensus.cc:2749] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:24:23.793969 26100 raft_consensus.cc:2749] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
W20260709 12:24:23.803068 17802 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.514s	user 0.066s	sys 0.093s
I20260709 12:24:23.801005 26154 raft_consensus.cc:3055] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 21 LEADER]: Stepping down as leader of term 21
W20260709 12:24:23.820928 17802 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.532s	user 0.066s	sys 0.093s
I20260709 12:24:23.792191 26137 raft_consensus.cc:3060] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:24:23.797096 26072 raft_consensus.cc:2749] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:24:23.764014 25970 consensus_queue.cc:260] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 21.16, Last appended by leader: 16, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:23.837812 25970 raft_consensus.cc:3060] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:23.815259 17652 leader_election.cc:304] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:23.857825 17688 raft_consensus.cc:2468] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 21.
I20260709 12:24:23.858327 17652 leader_election.cc:304] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:23.814200 26097 raft_consensus.cc:493] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:23.823107 26154 raft_consensus.cc:740] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 21 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:24:23.821110 25826 raft_consensus.cc:2749] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Leader pre-election lost for term 24. Reason: Vote denied by peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837) with higher term. Message: Invalid argument: T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for earlier term 24. Current term is 25.
W20260709 12:24:23.819545 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:23.804314 25784 consensus_queue.cc:1048] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.016s
I20260709 12:24:23.807032 25131 consensus_queue.cc:1048] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.013s
W20260709 12:24:23.811352 25944 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.541s	user 0.009s	sys 0.033s
W20260709 12:24:23.821496 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 532237us
I20260709 12:24:23.784042 25915 raft_consensus.cc:515] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:23.803200 25917 raft_consensus.cc:2804] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:24:23.823233 17729 leader_election.cc:304] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
W20260709 12:24:23.809913 26119 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.634s	user 0.011s	sys 0.025s
I20260709 12:24:23.822427 25951 consensus_queue.cc:1048] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.013s
W20260709 12:24:23.917179 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 94056.9us
I20260709 12:24:23.822595 25974 raft_consensus.cc:2749] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:24:23.767130 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 17 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:24:23.869481 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:23.871032 17652 leader_election.cc:304] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:23.863059 25972 raft_consensus.cc:2804] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:24:23.880424 17802 leader_election.cc:304] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:23.875041 17728 leader_election.cc:304] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:23.864500 26097 raft_consensus.cc:3060] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Advancing to term 26
W20260709 12:24:23.919507 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 48606.7us
I20260709 12:24:23.968889 17652 leader_election.cc:304] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
W20260709 12:24:23.970566 18364 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59902 (request call id 1901) took 2546 ms (client timeout 1839 ms). Trace:
I20260709 12:24:23.970503 17652 leader_election.cc:304] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:24:23.854126 26002 raft_consensus.cc:493] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
W20260709 12:24:23.890358 17804 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:23.926769 25972 raft_consensus.cc:493] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:24:23.918588 26119 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.742s	user 0.011s	sys 0.026s
I20260709 12:24:23.927558 25951 raft_consensus.cc:2749] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:24:23.937680 26107 raft_consensus.cc:515] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:23.853708 25657 consensus_queue.cc:1048] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 11, Time since last communication: 0.033s
W20260709 12:24:23.854461 26112 consensus_queue.cc:1175] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "334a4e1f59944544b3b3ff36616b57da" responder_term: 22 status { last_received { term: 21 index: 16 } last_committed_idx: 16 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 21. Current term is 22. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260709 12:24:23.923264 25663 raft_consensus.cc:2749] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:24:23.886283 26114 consensus_queue.cc:1048] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.076s
I20260709 12:24:23.868331 26009 consensus_queue.cc:1048] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.089s
I20260709 12:24:23.880843 26154 consensus_queue.cc:260] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 21.18, Last appended by leader: 18, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:23.859445 26003 raft_consensus.cc:515] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:23.868907 25909 raft_consensus.cc:2749] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:24:23.960157 26028 leader_election.cc:290] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:24.048163 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 21 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:24.049098 25951 raft_consensus.cc:493] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:24.049693 25951 raft_consensus.cc:515] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:23.964781 26137 raft_consensus.cc:515] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:23.967099 26068 raft_consensus.cc:515] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:23.973742 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 93391.3us
W20260709 12:24:23.971268 18364 rpcz_store.cc:269] 0709 12:24:21.424409 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:21.424544 (+   135us) service_pool.cc:224] Handling call
0709 12:24:23.970193 (+2545649us) spinlock_profiling.cc:243] Waited 2.33 s on lock 0x7b54006af9d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:24:23.970451 (+   258us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5368541056}
I20260709 12:24:23.980587 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:23.981112 17652 leader_election.cc:304] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:24:23.916874 25917 raft_consensus.cc:493] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:24:23.983413 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 108471us
I20260709 12:24:24.059325 18362 raft_consensus.cc:2468] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 21.
I20260709 12:24:23.990216 18374 raft_consensus.cc:1275] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 21 index: 15. Preceding OpId from leader: term: 23 index: 16. (index mismatch)
I20260709 12:24:23.993297 25575 raft_consensus.cc:493] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:23.996690 26100 raft_consensus.cc:2804] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:24:23.978740 26060 raft_consensus.cc:2764] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Leader pre-election decision vote started in defunct term 20: won
I20260709 12:24:23.982154 25972 raft_consensus.cc:3060] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:23.985662 25826 raft_consensus.cc:2804] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:24:23.975225 26057 consensus_queue.cc:1048] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.046s
W20260709 12:24:23.953330 26158 log.cc:927] Time spent T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.146s	user 0.000s	sys 0.001s
I20260709 12:24:23.976529 26148 raft_consensus.cc:3060] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:24.067516 18368 raft_consensus.cc:2393] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 21: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:24:24.066587 17802 leader_election.cc:304] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:23.984508 26002 raft_consensus.cc:515] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:24.079676 17838 raft_consensus.cc:1275] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 23 index: 15. Preceding OpId from leader: term: 25 index: 16. (index mismatch)
I20260709 12:24:24.086086 17838 raft_consensus.cc:1275] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 19 index: 12. Preceding OpId from leader: term: 24 index: 14. (index mismatch)
I20260709 12:24:24.101294 17833 raft_consensus.cc:1240] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 22. Current term is 23. Ops: []
I20260709 12:24:24.061028 25917 raft_consensus.cc:3060] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:24.108588 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:24.057118 25995 leader_election.cc:290] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:24.116058 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:24.114935 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:24.118167 17838 raft_consensus.cc:3060] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:24.119349 17833 raft_consensus.cc:1275] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 24 index: 13. Preceding OpId from leader: term: 26 index: 15. (index mismatch)
I20260709 12:24:24.122789 17833 raft_consensus.cc:1275] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 18 index: 14. Preceding OpId from leader: term: 20 index: 15. (index mismatch)
I20260709 12:24:24.070787 25575 raft_consensus.cc:515] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:24.125211 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 23 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:24.129909 17833 raft_consensus.cc:3060] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:24.139297 17836 raft_consensus.cc:1275] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 21 index: 15. Preceding OpId from leader: term: 23 index: 16. (index mismatch)
I20260709 12:24:24.142897 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:24.146876 17836 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d"
dest_uuid: "e063541542a94969898868bf6eb5ce1b"
 from {username='slave'} at 127.0.0.1:35042
I20260709 12:24:24.147639 17836 raft_consensus.cc:493] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:24:24.148151 17836 raft_consensus.cc:3060] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:24.148622 25915 leader_election.cc:290] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:24.157186 26002 leader_election.cc:290] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:24.067859 26156 consensus_queue.cc:1059] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 0.0, Next index: 18, Last known committed idx: 15, Time since last communication: 0.096s
I20260709 12:24:24.158147 26156 consensus_queue.cc:1243] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Peer 0da169ff82674dc1947055ce43de68a3 log is divergent from this leader: its last log entry 22.18 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 15
I20260709 12:24:24.104499 26100 raft_consensus.cc:697] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:24:24.159214 26100 consensus_queue.cc:237] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:24.109182 25858 raft_consensus.cc:2804] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:24:24.110776 25688 raft_consensus.cc:2749] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:24:24.097438 26149 raft_consensus.cc:3055] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:24:24.169919 25858 raft_consensus.cc:493] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:24:24.114190 25826 raft_consensus.cc:697] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 25 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
W20260709 12:24:24.175192 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:24.176503 17834 raft_consensus.cc:1275] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 17 index: 14. Preceding OpId from leader: term: 19 index: 15. (index mismatch)
I20260709 12:24:24.176329 17833 raft_consensus.cc:2468] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 23.
I20260709 12:24:24.183408 17836 raft_consensus.cc:515] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Starting forced leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:24.204917 17655 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:24.201031 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 475745us
I20260709 12:24:24.173308 26137 leader_election.cc:290] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 28 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:24.171811 26003 leader_election.cc:290] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 21 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:24.232631 17803 leader_election.cc:304] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:24.190213 25858 raft_consensus.cc:3060] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:24.169864 26149 raft_consensus.cc:740] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 20 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
W20260709 12:24:24.236024 17801 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.501s	user 0.013s	sys 0.040s
I20260709 12:24:24.209684 25881 raft_consensus.cc:2749] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:24:24.243353 18367 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59886 (request call id 2167) took 1196 ms. Trace:
W20260709 12:24:24.244056 17801 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.510s	user 0.014s	sys 0.040s
W20260709 12:24:24.245060 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 457043us
I20260709 12:24:24.247839 18333 leader_election.cc:304] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:24.256856 18333 leader_election.cc:304] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:24.241262 26156 raft_consensus.cc:2749] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
W20260709 12:24:24.258095 18334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:24.245501 18367 rpcz_store.cc:276] 0709 12:24:23.046633 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:23.156692 (+110059us) service_pool.cc:224] Handling call
0709 12:24:23.157088 (+   396us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:24:23.157441 (+   353us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:23.157472 (+    31us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:24:23.158011 (+   539us) log.cc:844] Serialized 52 byte log entry
0709 12:24:23.389103 (+231092us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:24:23.810226 (+421123us) log.cc:844] Serialized 14 byte log entry
0709 12:24:23.810903 (+   677us) log.cc:844] Serialized 14 byte log entry
0709 12:24:23.944063 (+133160us) spinlock_profiling.cc:243] Waited 1.66 ms on lock 0x7b18000ae400. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b47533522 00007f7b475357c1 00007f7b47535779 00007f7b47535709 00007f7b475356d1 00007f7b475349cd 0000000000388634 0000000000388479 00007f7b4751683a
0709 12:24:24.131119 (+187056us) spinlock_profiling.cc:243] Waited 15.2 ms on lock 0x7b18000ae400. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b475333f3 00007f7b475357c1 00007f7b47535779 00007f7b47535709 00007f7b475356d1 00007f7b475349cd 0000000000388634 0000000000388479 00007f7b4751683a
0709 12:24:24.242933 (+111814us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:24.243050 (+   117us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:24.243107 (+    57us) raft_consensus.cc:1685] finished
0709 12:24:24.243125 (+    18us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:24.243287 (+   162us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":242938,"prepare.queue_time_us":242831,"prepare.run_cpu_time_us":73,"prepare.run_wall_time_us":76,"raft.queue_time_us":25150,"raft.run_cpu_time_us":405,"raft.run_wall_time_us":39847,"spinlock_wait_cycles":38732288,"thread_start_us":738963,"threads_started":3,"wal-append.queue_time_us":418488}
I20260709 12:24:24.295782 18333 leader_election.cc:304] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:24.244172 26107 leader_election.cc:290] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 26 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:24.302528 18333 leader_election.cc:304] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:24.306717 18333 leader_election.cc:304] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:24.258020 25881 raft_consensus.cc:493] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:24.259162 25688 raft_consensus.cc:493] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:24.259819 26137 raft_consensus.cc:2749] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 12:24:24.295117 25868 raft_consensus.cc:493] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:24.303032 26014 raft_consensus.cc:2749] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 12:24:24.319254 25638 raft_consensus.cc:2749] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:24:24.307706 26002 raft_consensus.cc:2749] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:24:24.321436 25915 raft_consensus.cc:2749] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:24:24.343797 26096 leader_election.cc:290] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:24.244834 25740 raft_consensus.cc:515] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:24.295477 25826 consensus_queue.cc:237] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 23.14, Last appended by leader: 14, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:24.327389 26068 leader_election.cc:290] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 21 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:24.311581 25917 raft_consensus.cc:515] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:24.326144 25881 raft_consensus.cc:515] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:24.344432 25972 raft_consensus.cc:515] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:24.330891 25868 raft_consensus.cc:515] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:24.327987 25688 raft_consensus.cc:515] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:24:24.320605 26149 consensus_queue.cc:260] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.13, Last appended by leader: 13, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:24.374362 26149 raft_consensus.cc:3060] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:24.360357 26097 raft_consensus.cc:515] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:24.476219 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:24:24.479076 17688 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.594s	user 0.010s	sys 0.000s
I20260709 12:24:24.481041 17684 raft_consensus.cc:1240] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 20. Current term is 21. Ops: []
I20260709 12:24:24.483222 17685 raft_consensus.cc:1240] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 18 LEADER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 17. Current term is 18. Ops: []
I20260709 12:24:24.485299 17687 raft_consensus.cc:1275] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 22 index: 14. Preceding OpId from leader: term: 25 index: 15. (index mismatch)
I20260709 12:24:24.499826 17837 raft_consensus.cc:2368] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for earlier term 24. Current term is 25.
I20260709 12:24:24.519733 25868 leader_election.cc:290] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:24.534664 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:24.535050 17680 raft_consensus.cc:2468] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 24.
I20260709 12:24:24.535394 25868 raft_consensus.cc:493] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:24.519577 25974 raft_consensus.cc:493] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:24.536732 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:24.536538 25868 raft_consensus.cc:515] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:24.537822 18362 raft_consensus.cc:2368] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 19. Current term is 20.
I20260709 12:24:24.537436 25974 raft_consensus.cc:515] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:24.538894 26097 leader_election.cc:290] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 26 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:24.525993 25917 leader_election.cc:290] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 22 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:24.530813 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 24 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:24.521874 26028 raft_consensus.cc:493] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:24.541254 26028 raft_consensus.cc:515] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:24.504488 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:24.533229 25688 leader_election.cc:290] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:24.542601 17758 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 21.
I20260709 12:24:24.542667 26120 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.BulkChangeConfig from 127.0.0.1:37500 (request call id 4) took 29823 ms. Trace:
I20260709 12:24:24.527037 17585 catalog_manager.cc:5697] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 15 to 18. New cstate: current_term: 18 leader_uuid: "334a4e1f59944544b3b3ff36616b57da" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:24:24.522629 25740 leader_election.cc:290] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 25 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:24.511119 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:24.543350 26120 rpcz_store.cc:276] 0709 12:23:54.718855 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:55.035633 (+316778us) service_pool.cc:224] Handling call
0709 12:23:55.303115 (+267482us) log.cc:844] Serialized 499 byte log entry
0709 12:23:56.789384 (+1486269us) spinlock_profiling.cc:243] Waited 6.52 ms on lock 0x7b18003dff60. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b474adc0a 00007f7b50bed219 00007f7b4e4f458f 00007f7b47533571 00007f7b475357c1 00007f7b47535779 00007f7b47535709 00007f7b475356d1 00007f7b475349cd
0709 12:24:24.542397 (+27753013us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1707461,"raft.queue_time_us":209016,"raft.run_cpu_time_us":3540,"raft.run_wall_time_us":576568,"spinlock_wait_cycles":15005056,"thread_start_us":335164,"threads_started":3,"wal-append.queue_time_us":504656,"wal-append.run_cpu_time_us":20663,"wal-append.run_wall_time_us":1975671}
I20260709 12:24:24.545511 25740 raft_consensus.cc:493] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:24.546118 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 21 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:24.546373 25740 raft_consensus.cc:515] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:24.547065 18365 raft_consensus.cc:2468] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 21.
W20260709 12:24:24.534389 17804 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:24.513247 18367 raft_consensus.cc:1275] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 19 index: 17. Preceding OpId from leader: term: 20 index: 18. (index mismatch)
I20260709 12:24:24.534366 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 21 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:24.515352 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:24.519018 26009 raft_consensus.cc:493] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:24.551220 26009 raft_consensus.cc:515] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:24.389314 26150 raft_consensus.cc:3060] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:24.508869 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 25 candidate_status { last_received { term: 23 index: 19 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:24.510521 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: true dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:24.545297 17585 catalog_manager.cc:5697] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 18 to 21, leader changed from e063541542a94969898868bf6eb5ce1b (127.16.29.67) to 0da169ff82674dc1947055ce43de68a3 (127.16.29.65). New cstate: current_term: 21 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:24:24.518756 25972 leader_election.cc:290] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 20 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:24.541553 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 23 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:24.503034 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:24.553725 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:24.541779 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:24.538823 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:24:24.558635 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 322376us
I20260709 12:24:24.539582 25868 leader_election.cc:290] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:24.565078 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:24.565950 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 18 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:24:24.562546 17688 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.678s	user 0.010s	sys 0.003s
W20260709 12:24:24.561601 17654 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.506s	user 0.005s	sys 0.008s
W20260709 12:24:24.587985 17654 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.532s	user 0.006s	sys 0.008s
I20260709 12:24:24.588860 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:24.589648 26148 raft_consensus.cc:2749] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:24:24.589978 17687 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 21.
I20260709 12:24:24.596799 25663 raft_consensus.cc:493] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:24.597656 25663 raft_consensus.cc:515] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:24.598289 17761 raft_consensus.cc:3055] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:24:24.598877 17761 raft_consensus.cc:740] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 24 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:24:24.600361 25131 raft_consensus.cc:493] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:24.600883 17761 consensus_queue.cc:260] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 24.17, Last appended by leader: 17, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:24.601715 25131 raft_consensus.cc:515] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:24.603065 17761 raft_consensus.cc:3060] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:24.603974 25917 raft_consensus.cc:493] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:24.604501 17829 raft_consensus.cc:1275] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 12 index: 9. Preceding OpId from leader: term: 21 index: 13. (index mismatch)
I20260709 12:24:24.604872 25917 raft_consensus.cc:515] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:24.605748 17802 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:24.605895 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:24.609670 25881 leader_election.cc:290] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:24.627748 25657 raft_consensus.cc:493] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:24.628153 26100 raft_consensus.cc:493] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:24:24.629161 18367 raft_consensus.cc:2393] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 18: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:24:24.628614 17830 raft_consensus.cc:2410] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for term 20 because replica has last-logged OpId of term: 18 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 12.
I20260709 12:24:24.627177 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:24.632872 18371 raft_consensus.cc:2393] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 20: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:24:24.633538 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 25 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:24.634311 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:24.636279 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 23 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:24.641597 26009 leader_election.cc:290] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:24.642194 18367 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f"
dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
mode: GRACEFUL
new_leader_uuid: "e063541542a94969898868bf6eb5ce1b"
 from {username='slave'} at 127.0.0.1:50468
I20260709 12:24:24.643168 18367 raft_consensus.cc:606] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Received request to transfer leadership to TS e063541542a94969898868bf6eb5ce1b
I20260709 12:24:24.643603 18367 raft_consensus.cc:614] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:24:24.644127 25740 leader_election.cc:290] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:24.644621 25951 leader_election.cc:290] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:24:24.622745 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 567273us
I20260709 12:24:24.645551 18367 raft_consensus.cc:1240] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 17. Current term is 18. Ops: [17.17-17.17]
I20260709 12:24:24.652763 17833 raft_consensus.cc:2393] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 23: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:24:24.646554 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:24.656896 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:24.655534 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:24.661199 17685 raft_consensus.cc:3060] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:24.645457 25974 leader_election.cc:290] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:24.628826 25657 raft_consensus.cc:515] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:24.664582 17728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:24.665335 18367 raft_consensus.cc:1275] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 18 index: 12. Preceding OpId from leader: term: 21 index: 13. (index mismatch)
I20260709 12:24:24.667357 25784 raft_consensus.cc:493] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:24.667817 16501 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 6
I20260709 12:24:24.668848 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 23 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:24.669036 25663 leader_election.cc:290] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:24.671010 26114 raft_consensus.cc:493] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:24.678071 26114 raft_consensus.cc:515] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:24.680532 26114 leader_election.cc:290] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:24.672326 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:24.652586 26148 consensus_queue.cc:1059] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.036s
I20260709 12:24:24.629235 26100 raft_consensus.cc:515] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:24.681389 25784 raft_consensus.cc:515] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:24.673470 25131 leader_election.cc:290] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:24.694489 17686 raft_consensus.cc:2468] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 20.
W20260709 12:24:24.699164 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 77464.3us
I20260709 12:24:24.681758 26028 leader_election.cc:290] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:24.705878 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:24.712416 18374 raft_consensus.cc:2393] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 22: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:24:24.717753 26100 leader_election.cc:290] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:24.718883 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:24.719544 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 25 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:24.721004 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:24.721352 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 21 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:24.722072 17758 raft_consensus.cc:2468] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 27.
W20260709 12:24:24.722198 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 740994us
I20260709 12:24:24.724365 17764 raft_consensus.cc:2393] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 23: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:24:24.730976 17761 raft_consensus.cc:2468] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 25.
I20260709 12:24:24.723773 17652 leader_election.cc:304] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:24.719594 25915 raft_consensus.cc:493] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:24.736744 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 24 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:24.736162 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:24.737800 17764 raft_consensus.cc:3060] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:24:24.740828 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:24.741304 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 25 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:24.742264 17760 raft_consensus.cc:2410] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for term 25 because replica has last-logged OpId of term: 23 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 15.
I20260709 12:24:24.742312 26163 consensus_queue.cc:1059] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 17.17, Next index: 18, Last known committed idx: 16, Time since last communication: 0.004s
I20260709 12:24:24.742780 25784 leader_election.cc:290] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:24.708640 25663 raft_consensus.cc:493] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:24:24.743186 26163 consensus_queue.cc:1243] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer 0da169ff82674dc1947055ce43de68a3 log is divergent from this leader: its last log entry 18.18 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 16
I20260709 12:24:24.743814 26163 raft_consensus.cc:3055] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:24:24.743746 25663 raft_consensus.cc:515] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:24.744338 26163 raft_consensus.cc:740] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 17 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:24:24.746160 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 25 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:24.745867 26163 consensus_queue.cc:260] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 17.17, Last appended by leader: 17, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:24.747661 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:24.748399 25575 leader_election.cc:290] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:24.748627 17832 raft_consensus.cc:2468] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 19.
I20260709 12:24:24.749895 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:24.751289 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 22 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:24.752676 26163 raft_consensus.cc:3060] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:24.754608 17802 leader_election.cc:304] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:24:24.754863 25657 leader_election.cc:290] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:24:24.755570 17655 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:24.756192 25868 raft_consensus.cc:2804] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:24:24.756438 17829 raft_consensus.cc:1240] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 20. Current term is 21. Ops: []
I20260709 12:24:24.756644 25868 raft_consensus.cc:493] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:24.757285 25868 raft_consensus.cc:3060] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:24.757688 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:24.757972 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 22 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:24.758522 17829 raft_consensus.cc:2468] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 19.
I20260709 12:24:24.758852 17686 raft_consensus.cc:2410] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 22 because replica has last-logged OpId of term: 21 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 9.
I20260709 12:24:24.759333 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 21 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:24.761031 25974 consensus_queue.cc:1048] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:24:24.760267 18364 raft_consensus.cc:2955] T a5732e71b9d54f8786b8c511d69943bf P 334a4e1f59944544b3b3ff36616b57da [term 24 LEARNER]: Committing config change with OpId 19.12: config changed from index -1 to 12, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } }
I20260709 12:24:24.744246 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:24.757231 25915 raft_consensus.cc:515] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:24.734445 26100 raft_consensus.cc:493] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:24.759372 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 24 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:24.767442 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:24.765785 26057 raft_consensus.cc:493] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:24.769173 26057 raft_consensus.cc:515] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:24.778185 25917 leader_election.cc:290] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:24.778457 17685 raft_consensus.cc:2468] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 25.
I20260709 12:24:24.761745 26109 consensus_queue.cc:1059] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 22.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.018s
I20260709 12:24:24.779989 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:24.781009 25131 raft_consensus.cc:3055] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:24:24.781474 25131 raft_consensus.cc:740] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:24:24.782686 25131 consensus_queue.cc:260] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 22.18, Last appended by leader: 18, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:24.763520 26114 consensus_queue.cc:1048] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 11, Time since last communication: 0.021s
I20260709 12:24:24.784175 25826 raft_consensus.cc:2749] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:24:24.791144 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:24.768472 17829 raft_consensus.cc:2468] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 23.
I20260709 12:24:24.784267 25131 raft_consensus.cc:3060] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:24.793844 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:24.795401 17684 raft_consensus.cc:1275] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 23 index: 13. Preceding OpId from leader: term: 25 index: 14. (index mismatch)
I20260709 12:24:24.800760 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 22 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:24.801452 17763 raft_consensus.cc:3060] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Advancing to term 22
W20260709 12:24:24.803030 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 79599.1us
I20260709 12:24:24.806134 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:24.806991 18374 raft_consensus.cc:3060] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:24.808225 26100 raft_consensus.cc:515] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:24.813190 17801 leader_election.cc:304] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:24.815019 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:24.815960 17760 raft_consensus.cc:2393] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 25: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:24:24.800639 18367 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 21.
I20260709 12:24:24.826542 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 21 candidate_status { last_received { term: 19 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:24.827231 18362 raft_consensus.cc:3060] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:24.827275 18363 raft_consensus.cc:2468] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 21.
I20260709 12:24:24.833074 26068 raft_consensus.cc:493] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:24.833586 26164 raft_consensus.cc:493] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:24.833837 26068 raft_consensus.cc:515] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:24.834220 26164 raft_consensus.cc:515] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:24.829805 17802 leader_election.cc:304] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:24:24.794605 25917 raft_consensus.cc:493] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:24.836434 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:24.836974 26120 consensus_queue.cc:1048] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.002s
W20260709 12:24:24.837553 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 71094.7us
W20260709 12:24:24.843500 17654 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:24.796681 26028 consensus_queue.cc:1048] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.039s
I20260709 12:24:24.844105 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:24.848455 17832 raft_consensus.cc:3060] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:24.854350 18367 raft_consensus.cc:3060] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Advancing to term 26
W20260709 12:24:24.859706 18334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:24.860350 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:24.861232 18368 raft_consensus.cc:3060] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:24.860219 25917 raft_consensus.cc:515] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:24.862068 18361 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 2405) took 1137 ms. Trace:
I20260709 12:24:24.863072 26155 consensus_queue.cc:1048] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.014s
I20260709 12:24:24.862409 18361 rpcz_store.cc:276] 0709 12:24:23.724135 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:23.725791 (+  1656us) service_pool.cc:224] Handling call
0709 12:24:23.726206 (+   415us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:23.977015 (+250809us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:23.977061 (+    46us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:23.983163 (+  6102us) log.cc:844] Serialized 26 byte log entry
0709 12:24:24.509239 (+526076us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:24:24.509329 (+    90us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:24.861497 (+352168us) spinlock_profiling.cc:243] Waited 250 ms on lock 0x7b54006b3854. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:24:24.861547 (+    50us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:24.861620 (+    73us) raft_consensus.cc:1685] finished
0709 12:24:24.861637 (+    17us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:24.861886 (+   249us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":570,"raft.run_cpu_time_us":130,"raft.run_wall_time_us":7715,"spinlock_wait_cycles":575709568,"thread_start_us":459159,"threads_started":1,"wal-append.queue_time_us":391171}
I20260709 12:24:24.864081 26149 raft_consensus.cc:2804] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:24:24.864368 25917 leader_election.cc:290] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:24.864710 26149 raft_consensus.cc:697] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 25 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:24:24.850348 25663 leader_election.cc:290] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:24.865886 26149 consensus_queue.cc:237] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 23.14, Last appended by leader: 14, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:24.858916 17801 leader_election.cc:304] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:24.878119 17831 raft_consensus.cc:1275] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 18 index: 13. Preceding OpId from leader: term: 20 index: 13. (term mismatch)
I20260709 12:24:24.878589 17831 pending_rounds.cc:85] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b: Aborting all ops after (but not including) 12
I20260709 12:24:24.879931 17729 leader_election.cc:304] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:24:24.879688 17831 pending_rounds.cc:107] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b: Aborting uncommitted NO_OP operation due to leader change: 18.13
I20260709 12:24:24.880609 17831 raft_consensus.cc:2889] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:24.880111 25637 raft_consensus.cc:493] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:24.881747 25637 raft_consensus.cc:515] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:24.883226 17803 leader_election.cc:336] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 17 election: RPC error from VoteRequest() call to peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Timed out: RequestConsensusVote RPC to 127.16.29.68:42577 timed out after 1.988s (SENT)
I20260709 12:24:24.874570 25917 raft_consensus.cc:493] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:24.891644 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:24.894289 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: true dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
W20260709 12:24:24.894795 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 50828.4us
W20260709 12:24:24.895737 25858 consensus_meta.cc:287] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.657s	user 0.011s	sys 0.010s
I20260709 12:24:24.895695 17801 leader_election.cc:304] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:24.898687 26156 raft_consensus.cc:2804] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:24:24.895709 26155 raft_consensus.cc:2749] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:24:24.899386 26156 raft_consensus.cc:697] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 25 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:24:24.894955 18371 raft_consensus.cc:3060] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:24.893409 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 22 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:24.904594 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 21 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:24.904570 26156 consensus_queue.cc:237] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 24.17, Last appended by leader: 17, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:24.904687 17757 raft_consensus.cc:1240] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 24. Current term is 25. Ops: []
I20260709 12:24:24.908332 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:24.909480 17759 raft_consensus.cc:3060] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Advancing to term 25
I20260709 12:24:24.896826 25858 raft_consensus.cc:515] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:24.910552 17680 raft_consensus.cc:1240] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 24. Current term is 25. Ops: []
I20260709 12:24:24.914515 18363 raft_consensus.cc:2410] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 22 because replica has last-logged OpId of term: 18 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 9.
I20260709 12:24:24.919904 26120 raft_consensus.cc:2804] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:24:24.920480 26120 raft_consensus.cc:493] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:24:24.920868 26120 raft_consensus.cc:3060] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:24.923069 25915 leader_election.cc:290] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:24.926661 26068 leader_election.cc:290] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:24:24.927366 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 79381.2us
I20260709 12:24:24.928493 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 21 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:24.929831 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 24 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:24.930526 18361 raft_consensus.cc:2468] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 23.
I20260709 12:24:24.931388 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 21 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:24.934507 25637 leader_election.cc:290] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:24.936682 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 25 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:24.937840 26057 leader_election.cc:290] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:24.938477 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 22 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:24.929508 18333 leader_election.cc:304] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:24.948467 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:24.949379 17830 raft_consensus.cc:2468] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 17.
I20260709 12:24:24.948661 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:24.950582 17834 raft_consensus.cc:2468] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 24.
I20260709 12:24:24.951896 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 24 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:24.926723 25858 leader_election.cc:290] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 25 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:24:24.957546 17804 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:24.958267 18368 raft_consensus.cc:2468] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 26.
I20260709 12:24:24.961804 18374 raft_consensus.cc:2468] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 20.
W20260709 12:24:24.964234 17803 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:24.964740 17803 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:24.970919 26100 leader_election.cc:290] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:24.971740 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:24:24.972425 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 167743us
I20260709 12:24:24.973837 25761 consensus_queue.cc:1048] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:24:24.974599 17652 leader_election.cc:304] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:24:24.884528 25663 consensus_queue.cc:1048] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.071s
I20260709 12:24:24.985430 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:24.988710 17685 raft_consensus.cc:1240] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 25 LEADER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 24. Current term is 25. Ops: []
I20260709 12:24:24.995323 17652 leader_election.cc:304] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:25.001394 17761 raft_consensus.cc:1275] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 23 index: 13. Preceding OpId from leader: term: 25 index: 14. (index mismatch)
I20260709 12:24:24.995541 17801 leader_election.cc:304] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:25.026412 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:24.985874 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 23 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:25.027753 18371 raft_consensus.cc:2468] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 25.
I20260709 12:24:25.028889 17759 raft_consensus.cc:2468] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 25.
I20260709 12:24:24.976457 17836 leader_election.cc:290] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 25 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:25.034653 17684 raft_consensus.cc:2393] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 27: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:24:24.991920 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:25.037314 17801 leader_election.cc:304] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:25.037501 18362 raft_consensus.cc:2468] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 21.
I20260709 12:24:24.989346 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 26 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:25.040412 17763 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 22.
I20260709 12:24:25.041179 17832 raft_consensus.cc:2410] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for term 22 because replica has last-logged OpId of term: 19 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 11.
I20260709 12:24:24.983193 25979 raft_consensus.cc:2804] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:24:25.043466 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:25.046025 17682 raft_consensus.cc:2468] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 25.
W20260709 12:24:25.042136 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 112953us
I20260709 12:24:25.039780 18361 raft_consensus.cc:2468] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 20.
W20260709 12:24:25.044010 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 85515.7us
I20260709 12:24:24.983467 26168 raft_consensus.cc:493] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:24.990871 26154 raft_consensus.cc:3060] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:25.045593 17654 leader_election.cc:304] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:24:24.984864 26112 raft_consensus.cc:493] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:25.053411 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:25.051635 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:25.054754 17681 raft_consensus.cc:2393] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 25: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:24:24.990665 25881 consensus_queue.cc:1059] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 24.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.059s
I20260709 12:24:25.056926 25881 raft_consensus.cc:3055] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:24:25.057312 25881 raft_consensus.cc:740] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 24 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:24:25.058213 25881 consensus_queue.cc:260] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 24.13, Last appended by leader: 13, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:24.992882 26155 consensus_queue.cc:1059] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.055s
I20260709 12:24:25.059706 26155 raft_consensus.cc:3055] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:24:25.060104 26155 raft_consensus.cc:740] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 24 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:24:25.061094 26155 consensus_queue.cc:260] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 24.15, Last appended by leader: 15, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:25.062310 26155 raft_consensus.cc:3060] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:25.045017 25979 raft_consensus.cc:493] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:25.064189 25979 raft_consensus.cc:3060] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:25.064402 26068 raft_consensus.cc:493] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:25.051115 25637 raft_consensus.cc:2749] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:24:25.064940 26068 raft_consensus.cc:515] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:25.064430 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:25.066789 26004 raft_consensus.cc:2764] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Leader pre-election decision vote started in defunct term 21: won
I20260709 12:24:25.068411 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 22 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
W20260709 12:24:25.067973 25688 consensus_queue.cc:1175] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "0da169ff82674dc1947055ce43de68a3" responder_term: 25 status { last_received { term: 25 index: 15 } last_committed_idx: 14 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 24. Current term is 25. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260709 12:24:25.069192 17682 raft_consensus.cc:3060] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Advancing to term 22
W20260709 12:24:25.069983 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 74794.5us
I20260709 12:24:25.055675 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:25.068037 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 28 candidate_status { last_received { term: 27 index: 21 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:25.073021 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:24.961552 25917 raft_consensus.cc:515] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:25.076896 17764 raft_consensus.cc:2468] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 28.
I20260709 12:24:25.046977 26149 raft_consensus.cc:2749] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:24:25.079983 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:25.081401 25967 raft_consensus.cc:2749] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 12:24:25.081851 25951 raft_consensus.cc:2804] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:24:25.082345 25951 raft_consensus.cc:493] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:25.082696 25951 raft_consensus.cc:3060] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:24.983956 26057 consensus_queue.cc:1048] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.070s
I20260709 12:24:25.084462 26114 consensus_queue.cc:1048] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20260709 12:24:25.085772 17686 raft_consensus.cc:2393] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 20: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:24:25.086807 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:25.087483 17686 raft_consensus.cc:3060] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:24:25.088088 26150 raft_consensus.cc:2749] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:24:25.087805 26112 raft_consensus.cc:515] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:25.087819 17728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:25.092875 26192 log.cc:927] Time spent T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.221s	user 0.001s	sys 0.000s
I20260709 12:24:25.096446 26168 raft_consensus.cc:515] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:24:25.099563 26168 leader_election.cc:290] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:25.104427 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:25.081715 17727 leader_election.cc:304] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:25.113359 18365 raft_consensus.cc:2393] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 20: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:24:25.114531 17681 raft_consensus.cc:2393] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 21: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:24:25.120321 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:25.129500 26149 consensus_queue.cc:1048] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
W20260709 12:24:25.131778 17728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:25.132872 25917 leader_election.cc:290] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:25.134728 17686 raft_consensus.cc:2468] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 28.
I20260709 12:24:25.135133 18367 raft_consensus.cc:2468] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 26.
I20260709 12:24:25.137218 17685 raft_consensus.cc:2468] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 21.
I20260709 12:24:25.138067 17652 leader_election.cc:304] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:25.138492 26164 leader_election.cc:290] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:25.141010 25967 raft_consensus.cc:2804] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:24:25.141423 25967 raft_consensus.cc:493] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:25.141780 25967 raft_consensus.cc:3060] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:25.142236 26120 raft_consensus.cc:515] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:25.144737 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 23 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:25.145565 17759 raft_consensus.cc:2393] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 23: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:24:25.146476 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:25.147179 17761 raft_consensus.cc:3060] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:25.147464 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:25.148864 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:25.149521 17758 raft_consensus.cc:2468] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 24.
I20260709 12:24:25.165660 17682 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 22.
I20260709 12:24:25.180280 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:25.181248 17685 raft_consensus.cc:2468] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 25.
I20260709 12:24:25.183132 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 17 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:25.183696 17831 raft_consensus.cc:1240] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 21. Current term is 22. Ops: []
I20260709 12:24:25.184988 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:25.185654 17829 raft_consensus.cc:3060] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:25.186301 17684 raft_consensus.cc:2393] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 26: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:24:25.187278 25868 raft_consensus.cc:515] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:25.188284 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:25.189530 25868 leader_election.cc:290] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 20 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:25.190171 17801 leader_election.cc:304] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:25.191220 25868 raft_consensus.cc:2804] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:24:25.191567 25868 raft_consensus.cc:493] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:25.191913 25868 raft_consensus.cc:3060] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:25.185420 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 21 candidate_status { last_received { term: 19 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:25.193552 26175 raft_consensus.cc:493] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:25.193921 25657 raft_consensus.cc:2749] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:24:25.194806 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:25.194315 26175 raft_consensus.cc:515] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:25.196264 26112 leader_election.cc:290] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:25.197721 25740 raft_consensus.cc:3055] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:24:25.198275 25740 raft_consensus.cc:740] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 20 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:24:25.199200 25740 consensus_queue.cc:260] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 20.18, Last appended by leader: 18, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:25.200627 25740 raft_consensus.cc:3060] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Advancing to term 21
W20260709 12:24:25.197615 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 53470.4us
I20260709 12:24:25.202674 26175 leader_election.cc:290] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:24:25.204681 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 123177us
I20260709 12:24:25.206194 17654 leader_election.cc:304] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:24:25.207561 17654 leader_election.cc:304] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:24:25.208796 18333 leader_election.cc:304] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
W20260709 12:24:25.209602 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 62187us
I20260709 12:24:25.209717 25917 raft_consensus.cc:2764] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Leader pre-election decision vote started in defunct term 19: won
I20260709 12:24:25.211053 25951 raft_consensus.cc:515] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:25.211261 25967 raft_consensus.cc:515] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:25.212101 17683 raft_consensus.cc:2468] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 18.
W20260709 12:24:25.213603 26201 log.cc:927] Time spent T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.175s	user 0.000s	sys 0.000s
I20260709 12:24:25.193240 17761 raft_consensus.cc:2468] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 27.
I20260709 12:24:25.215999 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:25.216730 17758 raft_consensus.cc:2468] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 25.
I20260709 12:24:25.213649 25967 leader_election.cc:290] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 22 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:25.222638 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:25.211385 18333 leader_election.cc:304] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:24:25.223178 17829 raft_consensus.cc:2468] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 27.
I20260709 12:24:25.229017 17761 raft_consensus.cc:2468] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 21.
I20260709 12:24:25.234728 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:25.237936 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:25.238471 17759 raft_consensus.cc:3060] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:25.240195 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 23 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:25.240803 17832 raft_consensus.cc:2393] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 23: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:24:25.241528 25967 raft_consensus.cc:2804] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:24:25.244122 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 21 candidate_status { last_received { term: 19 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:25.245409 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:25.245889 17834 raft_consensus.cc:3060] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:25.248732 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:25.248987 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:25.249361 17829 raft_consensus.cc:2468] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 17.
I20260709 12:24:25.240023 17654 leader_election.cc:304] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:24:25.246380 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:25.251374 18333 leader_election.cc:304] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:24:25.246711 26114 consensus_queue.cc:1048] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.010s
I20260709 12:24:25.256498 18333 leader_election.cc:304] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:24:25.257434 18333 leader_election.cc:304] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:24:25.261164 17830 raft_consensus.cc:1275] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 17 index: 17. Preceding OpId from leader: term: 18 index: 18. (index mismatch)
I20260709 12:24:25.262431 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:25.262977 17832 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 21.
I20260709 12:24:25.242187 26112 raft_consensus.cc:2804] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:24:25.267848 17803 leader_election.cc:304] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:25.279601 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:25.273336 17652 leader_election.cc:304] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:25.285847 17837 raft_consensus.cc:3060] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:25.270473 17654 leader_election.cc:304] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
W20260709 12:24:25.271226 17838 consensus_meta.cc:287] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.958s	user 0.011s	sys 0.011s
I20260709 12:24:25.310148 17759 raft_consensus.cc:2468] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 26.
I20260709 12:24:25.286967 16501 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:24:25.253841 25967 raft_consensus.cc:697] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 26 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
W20260709 12:24:25.321192 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 72179.4us
I20260709 12:24:25.322055 26168 raft_consensus.cc:2749] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:24:25.321985 25967 consensus_queue.cc:237] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 24.17, Last appended by leader: 17, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:25.266054 25637 raft_consensus.cc:2804] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:24:25.324120 25637 raft_consensus.cc:493] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:25.324457 25637 raft_consensus.cc:3060] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:25.326205 25657 consensus_queue.cc:1048] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 12:24:25.328576 17835 raft_consensus.cc:1275] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 21 index: 18. Preceding OpId from leader: term: 22 index: 18. (term mismatch)
I20260709 12:24:25.329010 17835 pending_rounds.cc:85] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b: Aborting all ops after (but not including) 17
I20260709 12:24:25.329265 17835 pending_rounds.cc:107] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b: Aborting uncommitted NO_OP operation due to leader change: 21.18
I20260709 12:24:25.329533 17835 raft_consensus.cc:2889] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:25.330065 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:25.330648 17832 raft_consensus.cc:2468] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 24.
I20260709 12:24:25.331264 26120 leader_election.cc:290] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 21 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:25.266338 25915 raft_consensus.cc:2804] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:24:25.333389 25915 raft_consensus.cc:493] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:25.333796 25915 raft_consensus.cc:3060] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:25.318648 17585 catalog_manager.cc:5697] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 14 to 18. New cstate: current_term: 18 leader_uuid: "334a4e1f59944544b3b3ff36616b57da" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 12:24:25.336097 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 65760.3us
I20260709 12:24:25.319785 17838 raft_consensus.cc:2468] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 22.
I20260709 12:24:25.300246 25917 raft_consensus.cc:2804] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:24:25.285674 25979 raft_consensus.cc:515] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:25.345243 17684 raft_consensus.cc:2468] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 20.
I20260709 12:24:25.270359 25826 raft_consensus.cc:2804] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:24:25.346580 17802 leader_election.cc:304] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
W20260709 12:24:25.314844 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 56646.4us
W20260709 12:24:25.353937 17801 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:25.354748 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:25.355656 17684 raft_consensus.cc:2368] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for earlier term 21. Current term is 22.
I20260709 12:24:25.356449 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:25.359658 17686 raft_consensus.cc:1240] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 25. Current term is 26. Ops: [25.15-25.15]
I20260709 12:24:25.359707 17834 raft_consensus.cc:2468] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 20.
I20260709 12:24:25.344386 25688 raft_consensus.cc:2764] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Leader pre-election decision vote started in defunct term 22: won
I20260709 12:24:25.315838 26112 raft_consensus.cc:697] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 20 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:24:25.366503 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:25.368254 17838 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35042 (request call id 3067) took 1238 ms. Trace:
I20260709 12:24:25.340982 25917 raft_consensus.cc:493] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:25.370434 17837 raft_consensus.cc:2468] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 26.
I20260709 12:24:25.346125 26068 leader_election.cc:290] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:25.370975 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:25.375820 17684 raft_consensus.cc:2468] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 20.
I20260709 12:24:25.368645 17838 rpcz_store.cc:276] 0709 12:24:24.114536 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:24.114662 (+   126us) service_pool.cc:224] Handling call
0709 12:24:25.346858 (+1232196us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:25.373958 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:25.374574 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
W20260709 12:24:25.374557 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 101341us
I20260709 12:24:25.363893 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 19 candidate_status { last_received { term: 15 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:25.374574 25917 raft_consensus.cc:3060] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Advancing to term 20
W20260709 12:24:25.381007 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 132773us
I20260709 12:24:25.402570 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:25.402997 17684 raft_consensus.cc:1275] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 22 index: 14. Preceding OpId from leader: term: 25 index: 15. (index mismatch)
I20260709 12:24:25.367066 25826 raft_consensus.cc:697] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 21 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:24:25.401013 17727 leader_election.cc:304] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:25.408919 18371 raft_consensus.cc:2468] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 20.
W20260709 12:24:25.407207 17804 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:25.411580 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:25.414417 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:25.418280 17837 raft_consensus.cc:2468] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 22.
I20260709 12:24:25.416529 17652 leader_election.cc:304] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:25.411319 25909 raft_consensus.cc:2804] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:24:25.420559 18374 raft_consensus.cc:2393] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 22: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:24:25.492961 25909 raft_consensus.cc:697] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 26 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:24:25.397697 26068 raft_consensus.cc:493] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:25.498926 17763 raft_consensus.cc:2468] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 22.
W20260709 12:24:25.501078 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 81858.4us
I20260709 12:24:25.503057 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:25.511727 26107 raft_consensus.cc:2749] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:24:25.511983 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:25.512725 18366 raft_consensus.cc:2468] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 22.
I20260709 12:24:25.516796 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:24:25.517189 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 281646us
W20260709 12:24:25.518416 26203 log.cc:927] Time spent T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.106s	user 0.000s	sys 0.000s
I20260709 12:24:25.519280 17803 leader_election.cc:304] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:24:25.520160 26202 log.cc:927] Time spent T a5732e71b9d54f8786b8c511d69943bf P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.118s	user 0.001s	sys 0.000s
I20260709 12:24:25.519253 25868 raft_consensus.cc:515] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:25.504141 25688 consensus_queue.cc:1048] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 12, Time since last communication: 0.019s
I20260709 12:24:25.523316 25688 consensus_queue.cc:1243] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer 0da169ff82674dc1947055ce43de68a3 log is divergent from this leader: its last log entry 22.14 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 12
W20260709 12:24:25.524622 26204 log.cc:927] Time spent T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.106s	user 0.000s	sys 0.000s
I20260709 12:24:25.526489 25881 raft_consensus.cc:3060] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:25.527594 17686 raft_consensus.cc:1217] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Deduplicated request from leader. Original: 20.12->[21.13-25.15]   Dedup: 21.13->[23.14-25.15]
I20260709 12:24:25.404026 25784 consensus_queue.cc:1048] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.100s
I20260709 12:24:25.528177 17686 pending_rounds.cc:85] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3: Aborting all ops after (but not including) 13
I20260709 12:24:25.568200 17686 pending_rounds.cc:107] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3: Aborting uncommitted NO_OP operation due to leader change: 22.14
I20260709 12:24:25.568549 17686 raft_consensus.cc:2889] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:24:25.549150 17804 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:25.497628 26156 raft_consensus.cc:493] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:25.575351 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 19 candidate_status { last_received { term: 15 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:25.576532 17837 raft_consensus.cc:2410] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for term 19 because replica has last-logged OpId of term: 17 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 16.
I20260709 12:24:25.577484 17830 raft_consensus.cc:1275] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 24 index: 15. Preceding OpId from leader: term: 25 index: 15. (term mismatch)
W20260709 12:24:25.583343 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 212204us
I20260709 12:24:25.577967 17830 pending_rounds.cc:85] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b: Aborting all ops after (but not including) 14
I20260709 12:24:25.584120 17830 pending_rounds.cc:107] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b: Aborting uncommitted NO_OP operation due to leader change: 24.15
I20260709 12:24:25.584352 17830 raft_consensus.cc:2889] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:24:25.587412 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 61473.7us
I20260709 12:24:25.569501 25951 leader_election.cc:290] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 22 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:24:25.623144 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 206693us
W20260709 12:24:25.586088 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 214180us
I20260709 12:24:25.627681 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:25.641886 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:25.642793 17688 raft_consensus.cc:3060] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:25.573005 26114 raft_consensus.cc:2749] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:24:25.510783 26068 raft_consensus.cc:515] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:25.662231 18334 leader_election.cc:304] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:24:25.662830 25866 raft_consensus.cc:2749] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
W20260709 12:24:25.664572 26211 log.cc:927] Time spent T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.134s	user 0.000s	sys 0.001s
W20260709 12:24:25.672510 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 153334us
I20260709 12:24:25.673851 17803 leader_election.cc:400] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 19 election: Vote denied by peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577) with higher term. Message: Invalid argument: T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 19. Current term is 20.
I20260709 12:24:25.675256 17803 leader_election.cc:304] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:25.676838 25826 consensus_queue.cc:237] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 19.17, Last appended by leader: 17, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:25.619349 25868 leader_election.cc:290] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 26 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:25.585573 26208 raft_consensus.cc:2804] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:24:25.575448 26156 raft_consensus.cc:515] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:25.643982 26072 raft_consensus.cc:2804] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:24:25.680687 26155 raft_consensus.cc:493] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
W20260709 12:24:25.706725 25131 consensus_meta.cc:287] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.887s	user 0.012s	sys 0.000s
I20260709 12:24:25.705870 18334 leader_election.cc:304] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:24:25.622522 26164 consensus_queue.cc:1059] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 22.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.088s
I20260709 12:24:25.651799 26148 raft_consensus.cc:493] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:25.647841 26107 raft_consensus.cc:493] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:25.511934 25909 consensus_queue.cc:237] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 24.15, Last appended by leader: 15, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:25.727705 26211 log.cc:927] Time spent T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.062s	user 0.000s	sys 0.001s
I20260709 12:24:25.759822 25905 consensus_queue.cc:1048] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.035s
I20260709 12:24:25.712862 26163 raft_consensus.cc:2749] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:24:25.711966 26208 raft_consensus.cc:493] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
W20260709 12:24:25.768321 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 143388us
I20260709 12:24:25.641153 26112 consensus_queue.cc:237] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 18.15, Last appended by leader: 15, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:25.761816 25866 raft_consensus.cc:493] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
W20260709 12:24:25.820933 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 145901us
I20260709 12:24:25.810518 17652 leader_election.cc:304] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:24:25.764760 25688 raft_consensus.cc:493] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:25.771715 26208 raft_consensus.cc:3060] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:25.760763 26155 raft_consensus.cc:515] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:25.853943 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 83068.6us
W20260709 12:24:25.859079 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RunLeaderElection -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 93952.9us
I20260709 12:24:25.861456 25384 raft_consensus.cc:2804] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:24:25.862644 25384 raft_consensus.cc:697] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 27 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:24:25.819552 25866 raft_consensus.cc:515] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:24:25.864719 26215 log.cc:927] Time spent T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.123s	user 0.001s	sys 0.000s
I20260709 12:24:25.859447 17652 leader_election.cc:304] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:25.892578 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:25.897730 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:25.898988 17834 raft_consensus.cc:2393] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 27: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:24:25.782339 26148 raft_consensus.cc:515] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:25.906186 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:25.928771 17803 leader_election.cc:304] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:24:25.930555 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 71239.6us
I20260709 12:24:25.932268 17652 leader_election.cc:304] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:24:25.940828 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:25.941815 17688 raft_consensus.cc:2468] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 26.
I20260709 12:24:25.938997 17652 leader_election.cc:304] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:24:25.806466 26107 raft_consensus.cc:515] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:25.948254 17833 raft_consensus.cc:2368] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for earlier term 25. Current term is 26.
W20260709 12:24:25.954866 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 249071us
I20260709 12:24:25.956423 17837 raft_consensus.cc:2468] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 19.
I20260709 12:24:25.958699 18364 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59886 (request call id 2186) took 1363 ms. Trace:
I20260709 12:24:25.934181 25951 raft_consensus.cc:2749] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:24:25.980093 17684 raft_consensus.cc:1275] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 18 index: 14. Preceding OpId from leader: term: 21 index: 15. (index mismatch)
I20260709 12:24:25.944859 26163 raft_consensus.cc:2749] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:24:25.948685 26009 raft_consensus.cc:2804] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:24:25.902611 25688 raft_consensus.cc:515] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:25.957996 25761 raft_consensus.cc:2804] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:24:25.987807 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:25.864790 25384 consensus_queue.cc:237] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 25.17, Last appended by leader: 17, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:25.990900 26218 raft_consensus.cc:493] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:25.994289 26216 log.cc:927] Time spent T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.170s	user 0.001s	sys 0.000s
W20260709 12:24:25.995030 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 90593.8us
I20260709 12:24:25.996006 17803 leader_election.cc:304] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:24:25.997038 26221 log.cc:927] Time spent T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.056s	user 0.000s	sys 0.000s
I20260709 12:24:25.999181 17803 leader_election.cc:304] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:26.000772 26213 raft_consensus.cc:2804] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:24:25.966673 26097 consensus_queue.cc:1048] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 15, Time since last communication: 0.035s
I20260709 12:24:26.001376 26213 raft_consensus.cc:697] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 28 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:24:25.963089 25967 consensus_queue.cc:1048] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.039s
I20260709 12:24:26.002452 26213 consensus_queue.cc:237] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 26.18, Last appended by leader: 18, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:26.004550 26163 raft_consensus.cc:2764] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Leader pre-election decision vote started in defunct term 20: won
I20260709 12:24:26.005257 26163 raft_consensus.cc:2749] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 25 LEADER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:24:25.986593 26009 raft_consensus.cc:493] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:26.008796 26009 raft_consensus.cc:3060] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:26.018710 17680 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37402 (request call id 3520) took 1033 ms. Trace:
I20260709 12:24:26.019408 17680 rpcz_store.cc:276] 0709 12:24:24.985569 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:25.026255 (+ 40686us) service_pool.cc:224] Handling call
0709 12:24:25.046461 (+ 20206us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:25.108743 (+ 62282us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:25.108779 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:25.146399 (+ 37620us) log.cc:844] Serialized 26 byte log entry
0709 12:24:25.314045 (+167646us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:24:25.315022 (+   977us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:25.315188 (+   166us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:25.528394 (+213206us) raft_consensus.cc:1685] finished
0709 12:24:25.528425 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:26.018303 (+489878us) spinlock_profiling.cc:243] Waited 1.15 ms on lock 0x7b54001aa750. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a61f8ad 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:24:26.018614 (+   311us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":6338,"raft.queue_time_us":9666,"raft.run_cpu_time_us":113,"raft.run_wall_time_us":113,"spinlock_wait_cycles":2647296,"thread_start_us":166672,"threads_started":1,"wal-append.queue_time_us":227691}
I20260709 12:24:26.036911 25979 leader_election.cc:290] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 26 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:26.038808 25761 raft_consensus.cc:493] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:26.039433 25761 raft_consensus.cc:3060] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:26.041057 26068 leader_election.cc:290] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:25.964728 25974 raft_consensus.cc:493] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:25.959146 18364 rpcz_store.cc:276] 0709 12:24:24.594839 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:24.688834 (+ 93995us) service_pool.cc:224] Handling call
0709 12:24:24.706303 (+ 17469us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:24:24.759997 (+ 53694us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 12:24:25.013042 (+253045us) log.cc:844] Serialized 14 byte log entry
0709 12:24:25.118361 (+105319us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:24:25.243298 (+124937us) log.cc:844] Serialized 52 byte log entry
0709 12:24:25.255050 (+ 11752us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:24:25.370314 (+115264us) log.cc:844] Serialized 14 byte log entry
0709 12:24:25.800708 (+430394us) log.cc:844] Serialized 14 byte log entry
0709 12:24:25.958383 (+157675us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:25.958468 (+    85us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:25.958508 (+    40us) raft_consensus.cc:1685] finished
0709 12:24:25.958518 (+    10us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:25.958649 (+   131us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":249123,"prepare.run_wall_time_us":34,"raft.queue_time_us":137839,"raft.run_cpu_time_us":419,"raft.run_wall_time_us":16946,"spinlock_wait_cycles":28734720,"thread_start_us":724448,"threads_started":4,"wal-append.queue_time_us":276074}
I20260709 12:24:25.991436 26218 raft_consensus.cc:515] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:26.060182 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 61125.1us
W20260709 12:24:26.061739 25915 consensus_meta.cc:287] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.727s	user 0.005s	sys 0.011s
I20260709 12:24:26.062196 17803 leader_election.cc:304] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:26.063686 26163 raft_consensus.cc:2749] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:24:26.059616 26155 leader_election.cc:290] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:26.065949 17803 leader_election.cc:304] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:26.067750 26155 raft_consensus.cc:2749] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:24:26.070173 17803 leader_election.cc:304] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:26.071123 26155 raft_consensus.cc:3060] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:26.075994 25974 raft_consensus.cc:515] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:26.076112 17585 catalog_manager.cc:5697] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 16 to 21, leader changed from 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) to 0da169ff82674dc1947055ce43de68a3 (127.16.29.65). New cstate: current_term: 21 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:24:25.522348 16501 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:24:26.079000 25974 leader_election.cc:290] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:26.078259 17803 leader_election.cc:304] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:26.081175 26148 leader_election.cc:290] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:24:26.082969 26212 log.cc:927] Time spent T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.424s	user 0.024s	sys 0.068s
I20260709 12:24:26.083550 26163 raft_consensus.cc:2749] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:24:26.086650 25866 leader_election.cc:290] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:26.086931 17803 leader_election.cc:304] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:26.087977 26155 raft_consensus.cc:2749] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:24:26.089155 26107 leader_election.cc:290] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:26.090205 17838 raft_consensus.cc:2955] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Committing config change with OpId 19.12: config changed from index -1 to 12, NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } }
I20260709 12:24:26.091897 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 26 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:25.970382 26072 raft_consensus.cc:697] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 20 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:24:26.093282 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:24:26.098630 18333 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.591s	user 0.021s	sys 0.014s
W20260709 12:24:26.103113 18333 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.595s	user 0.022s	sys 0.014s
I20260709 12:24:26.078354 17585 catalog_manager.cc:5697] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 9 to 18. New cstate: current_term: 18 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:24:26.115813 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:26.118101 18333 leader_election.cc:304] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:26.062242 25915 raft_consensus.cc:515] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:26.111817 26156 leader_election.cc:290] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:26.126435 18365 raft_consensus.cc:2393] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 19: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:24:26.108963 25688 leader_election.cc:290] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:26.114187 26228 raft_consensus.cc:2804] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:24:26.148311 26230 raft_consensus.cc:493] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:26.149452 26072 consensus_queue.cc:237] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 19.14, Last appended by leader: 14, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:26.128331 25637 consensus_meta.cc:287] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.803s	user 0.015s	sys 0.003s
I20260709 12:24:26.127660 25866 raft_consensus.cc:3060] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Advancing to term 20
W20260709 12:24:26.149441 25881 consensus_meta.cc:287] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.622s	user 0.010s	sys 0.006s
I20260709 12:24:26.161890 26228 raft_consensus.cc:493] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:24:26.180683 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 93838.2us
I20260709 12:24:26.183605 26156 raft_consensus.cc:493] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:26.184419 26156 raft_consensus.cc:515] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:26.187357 26156 leader_election.cc:290] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:26.178726 26228 raft_consensus.cc:3060] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Advancing to term 21
W20260709 12:24:26.183511 18336 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.600s	user 0.020s	sys 0.015s
W20260709 12:24:26.188874 18336 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.766s	user 0.021s	sys 0.015s
I20260709 12:24:26.189908 17654 leader_election.cc:304] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:24:26.192924 25826 raft_consensus.cc:2804] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:24:26.193760 25826 raft_consensus.cc:493] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:26.194201 25826 raft_consensus.cc:3060] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Advancing to term 21
W20260709 12:24:26.196317 17801 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:26.199265 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:26.200057 17683 raft_consensus.cc:2468] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 26.
I20260709 12:24:26.202437 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 23 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:26.204588 17727 leader_election.cc:304] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:26.205976 17727 leader_election.cc:304] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:26.206908 25905 raft_consensus.cc:2749] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:24:26.207269 26114 raft_consensus.cc:2804] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:24:26.207675 26114 raft_consensus.cc:493] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:26.208033 26114 raft_consensus.cc:3060] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:26.211251 25905 raft_consensus.cc:1064] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb: attempting to promote NON_VOTER 334a4e1f59944544b3b3ff36616b57da to VOTER
I20260709 12:24:26.212142 26227 raft_consensus.cc:493] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:26.170939 26230 raft_consensus.cc:515] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:26.212853 26227 raft_consensus.cc:515] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:26.214337 26230 leader_election.cc:290] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:26.215031 26227 leader_election.cc:290] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:26.215178 26230 raft_consensus.cc:2749] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:24:26.189188 26218 leader_election.cc:290] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:26.215389 25905 consensus_queue.cc:237] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 24.14, Last appended by leader: 13, Current term: 24, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:26.217962 25915 leader_election.cc:290] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 19 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:26.219904 26208 raft_consensus.cc:515] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:26.220352 17654 leader_election.cc:304] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:24:26.220974 18368 raft_consensus.cc:1240] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 24. Current term is 25. Ops: []
I20260709 12:24:26.222107 26112 raft_consensus.cc:2804] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:24:26.222635 26112 raft_consensus.cc:493] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:26.222703 26208 leader_election.cc:290] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 25 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:26.223201 26112 raft_consensus.cc:3060] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:26.206930 17727 leader_election.cc:400] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 25 pre-election: Vote denied by peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837) with higher term. Message: Invalid argument: T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for earlier term 25. Current term is 26.
I20260709 12:24:26.227154 17727 leader_election.cc:403] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 25 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:24:26.232827 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 21 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:26.230446 17802 leader_election.cc:304] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:24:26.245051 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 23 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:26.244427 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:26.246619 18365 raft_consensus.cc:2468] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 20.
I20260709 12:24:26.245779 17802 leader_election.cc:304] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:24:26.252408 17829 raft_consensus.cc:2468] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 25.
I20260709 12:24:26.255841 26155 raft_consensus.cc:2804] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:24:26.256136 17803 leader_election.cc:304] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:26.256345 26155 raft_consensus.cc:493] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:26.256853 26155 raft_consensus.cc:3060] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:26.260046 18367 raft_consensus.cc:2410] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for term 23 because replica has last-logged OpId of term: 22 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 15.
W20260709 12:24:26.272172 25740 consensus_meta.cc:287] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 1.071s	user 0.013s	sys 0.001s
I20260709 12:24:26.273074 17836 raft_consensus.cc:2410] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for term 21 because replica has last-logged OpId of term: 20 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 17.
I20260709 12:24:26.274333 17836 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35040 (request call id 3317) took 1056 ms. Trace:
I20260709 12:24:26.274694 17836 rpcz_store.cc:276] 0709 12:24:25.217652 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:25.233402 (+ 15750us) service_pool.cc:224] Handling call
0709 12:24:26.274134 (+1040732us) spinlock_profiling.cc:243] Waited 1.03 s on lock 0x7b54000ab4d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:24:26.274276 (+   142us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2364351232}
I20260709 12:24:26.242448 25131 raft_consensus.cc:3060] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Advancing to term 26
I20260709 12:24:26.280232 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 28 candidate_status { last_received { term: 27 index: 21 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:26.275812 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 26 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:26.282109 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:26.281067 18366 raft_consensus.cc:1275] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 19 index: 15. Preceding OpId from leader: term: 21 index: 15. (term mismatch)
I20260709 12:24:26.285827 18366 pending_rounds.cc:85] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da: Aborting all ops after (but not including) 14
I20260709 12:24:26.286126 18366 pending_rounds.cc:107] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da: Aborting uncommitted NO_OP operation due to leader change: 19.15
I20260709 12:24:26.286394 18366 raft_consensus.cc:2889] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:26.281391 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:26.285516 17836 raft_consensus.cc:2468] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 25.
I20260709 12:24:26.287370 18363 raft_consensus.cc:2468] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 22.
I20260709 12:24:26.293462 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:26.295749 18371 raft_consensus.cc:2393] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 26: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
W20260709 12:24:26.296948 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 342550us
I20260709 12:24:26.172070 25637 raft_consensus.cc:515] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:26.297989 17652 leader_election.cc:304] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:24:26.247395 25915 raft_consensus.cc:493] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:26.248968 26156 raft_consensus.cc:2804] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:24:26.258345 26208 raft_consensus.cc:493] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
W20260709 12:24:26.282069 25917 consensus_meta.cc:287] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.862s	user 0.007s	sys 0.003s
I20260709 12:24:26.372081 17829 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35040 (request call id 3325) took 1138 ms. Trace:
I20260709 12:24:26.372212 17727 leader_election.cc:304] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:24:26.303249 17760 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.725s	user 0.088s	sys 0.159s
W20260709 12:24:26.373452 17760 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.802s	user 0.088s	sys 0.159s
W20260709 12:24:26.376022 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 405943us
W20260709 12:24:26.372833 17686 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.690s	user 0.013s	sys 0.031s
W20260709 12:24:26.376977 17686 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.694s	user 0.013s	sys 0.031s
I20260709 12:24:26.385468 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:26.372745 17829 rpcz_store.cc:276] 0709 12:24:25.234118 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:25.254561 (+ 20443us) service_pool.cc:224] Handling call
0709 12:24:26.371917 (+1117356us) spinlock_profiling.cc:243] Waited 895 ms on lock 0x7b54004416d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:24:26.372028 (+   111us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2058548480}
I20260709 12:24:26.403919 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:26.406008 18361 raft_consensus.cc:1240] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 23. Current term is 24. Ops: [23.16-23.16]
I20260709 12:24:26.374109 25657 raft_consensus.cc:2804] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:24:26.425248 26228 raft_consensus.cc:515] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:26.425925 25657 raft_consensus.cc:493] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:26.426390 25657 raft_consensus.cc:3060] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:26.427824 26228 leader_election.cc:290] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 21 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:26.371429 25967 raft_consensus.cc:2804] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:24:26.370096 25740 raft_consensus.cc:2749] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:24:26.429188 17728 leader_election.cc:304] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
W20260709 12:24:26.445225 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 195161us
I20260709 12:24:26.279990 26112 raft_consensus.cc:515] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:26.391654 26156 raft_consensus.cc:697] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 26 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:24:26.513495 18368 raft_consensus.cc:2468] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 21.
W20260709 12:24:26.514894 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 85827.9us
I20260709 12:24:26.529597 17728 leader_election.cc:304] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
W20260709 12:24:26.532051 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 201028us
I20260709 12:24:26.532783 17728 leader_election.cc:304] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
W20260709 12:24:26.516009 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 397995us
I20260709 12:24:26.438581 25967 raft_consensus.cc:493] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:26.385077 26208 raft_consensus.cc:515] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:26.384259 25917 raft_consensus.cc:515] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:26.391342 25915 raft_consensus.cc:515] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:26.534789 18333 leader_election.cc:304] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
W20260709 12:24:26.549645 18334 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.570s	user 0.005s	sys 0.011s
W20260709 12:24:26.558012 18334 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.578s	user 0.006s	sys 0.011s
I20260709 12:24:26.574712 25826 raft_consensus.cc:515] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:26.577852 25826 leader_election.cc:290] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 21 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:26.574728 26114 raft_consensus.cc:515] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:26.582815 25131 raft_consensus.cc:2749] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Leader pre-election lost for term 25. Reason: Vote denied by peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837) with higher term. Message: Invalid argument: T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for earlier term 25. Current term is 26.
W20260709 12:24:26.550632 26175 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.524s	user 0.005s	sys 0.010s
I20260709 12:24:26.534363 25974 raft_consensus.cc:2749] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:24:26.559306 17576 maintenance_manager.cc:419] P 2193af31cb894ec9b3d65d939a2a6851: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=0.033398
I20260709 12:24:26.546833 25967 raft_consensus.cc:3060] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:26.599426 18374 raft_consensus.cc:1240] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 19. Current term is 20. Ops: []
W20260709 12:24:26.602051 17837 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.508s	user 0.036s	sys 0.021s
I20260709 12:24:26.603514 18333 leader_election.cc:304] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:24:26.616348 17837 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.523s	user 0.036s	sys 0.022s
W20260709 12:24:26.600059 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 50876.3us
I20260709 12:24:26.556787 26233 raft_consensus.cc:493] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
W20260709 12:24:26.598834 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 65195.9us
W20260709 12:24:26.599700 26175 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.573s	user 0.005s	sys 0.010s
W20260709 12:24:26.608708 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 76001.3us
W20260709 12:24:26.596535 26213 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.502s	user 0.009s	sys 0.044s
I20260709 12:24:26.610852 18362 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 2421) took 1245 ms. Trace:
I20260709 12:24:26.580601 26068 raft_consensus.cc:2749] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:24:26.557520 25979 raft_consensus.cc:493] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:26.622663 18364 raft_consensus.cc:1275] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 24 index: 17. Preceding OpId from leader: term: 26 index: 18. (index mismatch)
I20260709 12:24:26.627442 17728 leader_election.cc:304] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:26.609042 26220 raft_consensus.cc:493] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
W20260709 12:24:26.609834 26164 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.811s	user 0.016s	sys 0.013s
I20260709 12:24:26.644738 18362 rpcz_store.cc:276] 0709 12:24:25.365057 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:25.371022 (+  5965us) service_pool.cc:224] Handling call
0709 12:24:25.381539 (+ 10517us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:25.505678 (+124139us) raft_consensus.cc:1534] Early marking committed up to index 18
0709 12:24:25.616259 (+110581us) log.cc:844] Serialized 14 byte log entry
0709 12:24:26.109296 (+493037us) log.cc:844] Serialized 14 byte log entry
0709 12:24:26.608813 (+499517us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:26.608860 (+    47us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:24:26.608905 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:26.610624 (+  1719us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:26.610784 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":15617,"prepare.queue_time_us":151037,"prepare.run_cpu_time_us":117,"prepare.run_wall_time_us":104,"raft.queue_time_us":219248,"raft.run_cpu_time_us":114,"raft.run_wall_time_us":116,"thread_start_us":1022558,"threads_started":3,"wal-append.queue_time_us":745036}
W20260709 12:24:26.668231 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 47532.2us
I20260709 12:24:26.671720 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:26.675036 18364 raft_consensus.cc:1275] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 18 index: 15. Preceding OpId from leader: term: 20 index: 16. (index mismatch)
W20260709 12:24:26.681764 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 54416.1us
I20260709 12:24:26.683449 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
W20260709 12:24:26.681332 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 50961.7us
I20260709 12:24:26.684198 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:26.684167 18367 raft_consensus.cc:3060] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:26.694623 17803 leader_election.cc:304] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:26.682925 17728 leader_election.cc:304] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:26.700764 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:26.701574 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:26.702163 17685 raft_consensus.cc:3060] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:26.653175 26057 raft_consensus.cc:2804] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Leader pre-election won for term 26
W20260709 12:24:26.661492 26164 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.879s	user 0.016s	sys 0.013s
W20260709 12:24:26.693090 25761 consensus_meta.cc:287] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.653s	user 0.010s	sys 0.002s
W20260709 12:24:26.718014 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 114577us
I20260709 12:24:26.709389 26057 raft_consensus.cc:493] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:24:26.708033 26218 raft_consensus.cc:493] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:26.676710 25909 raft_consensus.cc:493] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:26.707757 25688 raft_consensus.cc:493] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:26.675249 26234 raft_consensus.cc:2749] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:24:26.692195 26228 raft_consensus.cc:493] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:26.630985 26233 raft_consensus.cc:515] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:26.675158 26220 raft_consensus.cc:515] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:26.728138 25761 raft_consensus.cc:515] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:26.769735 18333 leader_election.cc:304] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:26.755205 26218 raft_consensus.cc:515] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:26.756230 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 22 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:26.733297 26229 raft_consensus.cc:2749] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 12:24:26.758100 26114 leader_election.cc:290] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 21 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:26.759207 26206 raft_consensus.cc:2804] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Leader pre-election won for term 23
W20260709 12:24:26.759578 26223 log.cc:927] Time spent T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.232s	user 0.001s	sys 0.000s
W20260709 12:24:26.783854 17801 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:26.760695 26097 consensus_queue.cc:1048] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20260709 12:24:26.651276 26156 consensus_queue.cc:237] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 24.14, Last appended by leader: 14, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:24:26.761802 26222 log.cc:927] Time spent T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.136s	user 0.000s	sys 0.000s
I20260709 12:24:26.763767 26120 raft_consensus.cc:2749] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:24:26.762126 17685 raft_consensus.cc:2468] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 21.
W20260709 12:24:26.728216 26072 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.502s	user 0.010s	sys 0.019s
I20260709 12:24:26.727538 26057 raft_consensus.cc:3060] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:26.722098 25637 leader_election.cc:290] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 24 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:26.721462 26245 raft_consensus.cc:2749] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
W20260709 12:24:26.755297 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 76977.6us
I20260709 12:24:26.718667 25979 raft_consensus.cc:515] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:26.782550 17689 raft_consensus.cc:2410] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for term 22 because replica has last-logged OpId of term: 20 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 12.
I20260709 12:24:26.764031 26112 leader_election.cc:290] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 23 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:24:26.803656 26072 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.578s	user 0.010s	sys 0.020s
I20260709 12:24:26.796190 17729 leader_election.cc:304] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:26.789727 18367 raft_consensus.cc:2468] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 21.
I20260709 12:24:26.804356 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:26.739781 25657 raft_consensus.cc:515] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:26.742412 25909 raft_consensus.cc:515] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:26.789646 26206 raft_consensus.cc:493] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:26.812479 18333 leader_election.cc:304] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:26.761930 18374 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 2429) took 1346 ms. Trace:
I20260709 12:24:26.767868 25688 raft_consensus.cc:515] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:26.828312 18364 raft_consensus.cc:1240] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 20. Current term is 21. Ops: []
I20260709 12:24:26.828234 18362 raft_consensus.cc:3060] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:26.768046 26228 raft_consensus.cc:515] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:26.846671 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 50576.5us
I20260709 12:24:26.805691 26234 raft_consensus.cc:493] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:24:26.852396 18374 rpcz_store.cc:276] 0709 12:24:25.415046 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:25.506424 (+ 91378us) service_pool.cc:224] Handling call
0709 12:24:25.506687 (+   263us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:26.615538 (+1108851us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:26.761676 (+146138us) spinlock_profiling.cc:243] Waited 1.09 s on lock 0x7b54006b3854. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:24:26.761844 (+   168us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2512707456}
I20260709 12:24:26.864069 17729 leader_election.cc:304] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:26.824451 26208 leader_election.cc:290] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:24:26.851480 26155 consensus_meta.cc:287] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.594s	user 0.011s	sys 0.000s
W20260709 12:24:26.878968 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 196098us
I20260709 12:24:26.881582 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:26.824710 25917 leader_election.cc:290] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 20 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:26.851034 25784 raft_consensus.cc:493] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:26.860431 26206 raft_consensus.cc:3060] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:26.864403 26213 raft_consensus.cc:2749] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
W20260709 12:24:26.896756 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 214394us
W20260709 12:24:26.898809 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 86450.1us
I20260709 12:24:26.903820 17759 raft_consensus.cc:1275] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 23 index: 14. Preceding OpId from leader: term: 25 index: 15. (index mismatch)
W20260709 12:24:26.904405 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 234136us
I20260709 12:24:26.891516 17728 leader_election.cc:304] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:26.928709 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:26.833460 25951 consensus_queue.cc:1059] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 24.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.060s
W20260709 12:24:26.928210 26222 log.cc:927] Time spent T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.059s	user 0.000s	sys 0.000s
I20260709 12:24:26.938654 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:26.865561 26120 raft_consensus.cc:2749] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:24:26.861289 25915 leader_election.cc:290] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:27.014961 17654 leader_election.cc:304] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
W20260709 12:24:27.026713 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 161371us
W20260709 12:24:27.025686 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 129548us
I20260709 12:24:26.904640 26114 raft_consensus.cc:2749] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
W20260709 12:24:27.049862 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 158443us
I20260709 12:24:27.066951 18362 raft_consensus.cc:2468] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 21.
I20260709 12:24:26.927953 26208 raft_consensus.cc:493] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:27.035809 18333 leader_election.cc:304] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:27.073297 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:26.894328 26155 raft_consensus.cc:515] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:26.929358 25784 raft_consensus.cc:515] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:26.893389 26234 raft_consensus.cc:515] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:27.071858 26225 raft_consensus.cc:2749] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:24:27.124627 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:24:27.125342 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 114928us
I20260709 12:24:27.034968 26233 leader_election.cc:290] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:24:27.106182 26231 log.cc:927] Time spent T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.359s	user 0.000s	sys 0.001s
I20260709 12:24:27.126194 17652 leader_election.cc:304] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:24:27.129460 26114 consensus_queue.cc:1059] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 21.15, Next index: 16, Last known committed idx: 14, Time since last communication: 0.007s
I20260709 12:24:27.136163 26285 consensus_queue.cc:1048] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.019s
I20260709 12:24:27.137787 26285 raft_consensus.cc:2749] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:24:27.140069 26120 raft_consensus.cc:3055] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:24:27.140496 26120 raft_consensus.cc:740] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 23 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
W20260709 12:24:27.140762 26255 log.cc:927] Time spent T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.262s	user 0.000s	sys 0.000s
W20260709 12:24:27.140918 17730 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:27.141386 26120 consensus_queue.cc:260] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 23.16, Last appended by leader: 16, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:27.142715 26120 raft_consensus.cc:3060] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:27.143224 26234 leader_election.cc:290] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:27.143308 17838 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35042 (request call id 3090) took 1054 ms. Trace:
W20260709 12:24:27.144438 26242 log.cc:927] Time spent T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.329s	user 0.000s	sys 0.001s
I20260709 12:24:27.146066 25784 leader_election.cc:290] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:27.146615 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 21 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:27.148144 17837 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35042 (request call id 3092) took 1058 ms. Trace:
W20260709 12:24:27.149097 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 213490us
I20260709 12:24:27.149608 17764 raft_consensus.cc:1275] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 24 index: 17. Preceding OpId from leader: term: 25 index: 18. (index mismatch)
I20260709 12:24:27.133688 25915 raft_consensus.cc:493] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:27.151108 17581 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35966 (request call id 356) took 81425 ms (client timeout 15000 ms). Trace:
I20260709 12:24:27.151298 18333 leader_election.cc:304] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:27.118022 26213 raft_consensus.cc:2749] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:24:27.151491 25915 raft_consensus.cc:515] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:27.153379 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 55652.1us
I20260709 12:24:27.153106 17585 catalog_manager.cc:5697] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 15 to 21. New cstate: current_term: 21 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:24:27.154106 17803 leader_election.cc:304] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:27.155272 17803 leader_election.cc:304] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:27.154958 17585 catalog_manager.cc:5697] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 12 to 14. New cstate: current_term: 14 leader_uuid: "334a4e1f59944544b3b3ff36616b57da" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:24:27.156163 26164 raft_consensus.cc:2749] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:24:27.156050 26271 consensus_queue.cc:1048] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.027s
W20260709 12:24:27.157305 26258 log.cc:927] Time spent T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.213s	user 0.000s	sys 0.000s
W20260709 12:24:27.158205 26250 log.cc:927] Time spent T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.255s	user 0.001s	sys 0.000s
W20260709 12:24:27.158901 26257 log.cc:927] Time spent T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.272s	user 0.000s	sys 0.000s
I20260709 12:24:27.160719 26220 leader_election.cc:290] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:27.160373 26267 consensus_queue.cc:1048] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.054s
W20260709 12:24:27.162922 26253 log.cc:927] Time spent T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.258s	user 0.000s	sys 0.001s
I20260709 12:24:27.163035 17688 raft_consensus.cc:1240] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 20. Current term is 21. Ops: []
I20260709 12:24:27.164019 17689 raft_consensus.cc:1240] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 25. Current term is 26. Ops: [25.15-25.15]
I20260709 12:24:27.096341 25917 raft_consensus.cc:493] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
W20260709 12:24:27.112849 26247 log.cc:927] Time spent T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.384s	user 0.001s	sys 0.000s
W20260709 12:24:27.168246 17830 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.605s	user 0.039s	sys 0.019s
W20260709 12:24:27.222743 17830 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.665s	user 0.039s	sys 0.019s
I20260709 12:24:27.168218 17838 rpcz_store.cc:276] 0709 12:24:26.089181 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:26.089300 (+   119us) service_pool.cc:224] Handling call
0709 12:24:26.089644 (+   344us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:24:26.090091 (+   447us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 12:24:26.229127 (+139036us) log.cc:844] Serialized 14 byte log entry
0709 12:24:26.763245 (+534118us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:24:26.769865 (+  6620us) log.cc:844] Serialized 52 byte log entry
0709 12:24:26.770032 (+   167us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:24:27.015579 (+245547us) log.cc:844] Serialized 14 byte log entry
0709 12:24:27.102279 (+ 86700us) log.cc:844] Serialized 14 byte log entry
0709 12:24:27.102494 (+   215us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:27.102584 (+    90us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:27.143075 (+ 40491us) raft_consensus.cc:1685] finished
0709 12:24:27.143095 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:27.143226 (+   131us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":153827,"prepare.queue_time_us":492354,"prepare.run_cpu_time_us":54,"prepare.run_wall_time_us":56,"raft.queue_time_us":68283,"raft.run_cpu_time_us":544,"raft.run_wall_time_us":37902,"thread_start_us":626136,"threads_started":2,"wal-append.queue_time_us":538823}
I20260709 12:24:27.094144 26208 raft_consensus.cc:515] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:27.174630 17760 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38776 (request call id 3009) took 1799 ms. Trace:
W20260709 12:24:27.224431 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 98322.3us
I20260709 12:24:27.148514 17837 rpcz_store.cc:276] 0709 12:24:26.089792 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:26.091519 (+  1727us) service_pool.cc:224] Handling call
0709 12:24:26.091862 (+   343us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:24:26.092694 (+   832us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:26.092720 (+    26us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:24:26.093386 (+   666us) log.cc:844] Serialized 52 byte log entry
0709 12:24:26.617009 (+523623us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:24:26.842577 (+225568us) log.cc:844] Serialized 14 byte log entry
0709 12:24:26.842966 (+   389us) log.cc:844] Serialized 14 byte log entry
0709 12:24:26.851996 (+  9030us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:26.852081 (+    85us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:27.147935 (+295854us) raft_consensus.cc:1685] finished
0709 12:24:27.147956 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:27.148084 (+   128us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":8754,"prepare.queue_time_us":480631,"prepare.run_cpu_time_us":48,"prepare.run_wall_time_us":49,"raft.queue_time_us":19300,"raft.run_cpu_time_us":99,"raft.run_wall_time_us":99,"spinlock_wait_cycles":52064128,"thread_start_us":697909,"threads_started":2,"wal-append.queue_time_us":675396}
I20260709 12:24:27.117533 25637 raft_consensus.cc:493] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:27.215735 17585 catalog_manager.cc:5697] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 13 to 18. New cstate: current_term: 18 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:24:27.148465 26228 leader_election.cc:290] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:24:27.248734 17832 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.528s	user 0.000s	sys 0.010s
W20260709 12:24:27.249485 17832 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.536s	user 0.000s	sys 0.010s
I20260709 12:24:27.250789 17832 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35042 (request call id 3097) took 1159 ms. Trace:
I20260709 12:24:27.173445 26218 leader_election.cc:290] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:24:27.231586 25866 consensus_meta.cc:287] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 1.057s	user 0.009s	sys 0.002s
I20260709 12:24:27.216107 26163 raft_consensus.cc:2804] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Leader pre-election won for term 24
W20260709 12:24:27.233771 26251 log.cc:927] Time spent T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.429s	user 0.001s	sys 0.001s
W20260709 12:24:27.219740 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 52790.7us
I20260709 12:24:27.224306 17760 rpcz_store.cc:276] 0709 12:24:25.334606 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:25.344994 (+ 10388us) service_pool.cc:224] Handling call
0709 12:24:25.345351 (+   357us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:25.345560 (+   209us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:25.345593 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:25.404296 (+ 58703us) log.cc:844] Serialized 26 byte log entry
0709 12:24:26.375639 (+971343us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:24:26.429790 (+ 54151us) log.cc:844] Serialized 14 byte log entry
0709 12:24:26.432817 (+  3027us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:26.432897 (+    80us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:27.134086 (+701189us) raft_consensus.cc:1685] finished
0709 12:24:27.134107 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:27.134233 (+   126us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":58737,"prepare.queue_time_us":39543,"prepare.run_wall_time_us":6831,"raft.queue_time_us":403,"raft.run_cpu_time_us":126,"raft.run_wall_time_us":11248,"spinlock_wait_cycles":20280064,"thread_start_us":803805,"threads_started":1,"wal-append.queue_time_us":1114846}
I20260709 12:24:27.215641 26233 raft_consensus.cc:2804] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:24:27.227846 26213 raft_consensus.cc:2749] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:24:27.278659 17585 catalog_manager.cc:5697] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 16 to 22, leader changed from 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) to e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66). New cstate: current_term: 22 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:24:27.244675 26164 raft_consensus.cc:493] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:27.322458 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 307556us
W20260709 12:24:27.308254 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 143512us
W20260709 12:24:27.302968 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 151696us
I20260709 12:24:27.320861 17803 leader_election.cc:304] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:24:27.327509 17763 raft_consensus.cc:2468] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 26.
I20260709 12:24:27.260563 25384 raft_consensus.cc:2804] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:24:27.251853 26175 raft_consensus.cc:493] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:27.333316 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 253938us
I20260709 12:24:27.332437 18334 leader_election.cc:304] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:24:27.333691 17837 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35042 (request call id 3103) took 1076 ms. Trace:
I20260709 12:24:27.335429 18333 leader_election.cc:304] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:24:27.333985 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 280736us
I20260709 12:24:27.339571 17837 rpcz_store.cc:276] 0709 12:24:26.257725 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:27.238562 (+980837us) service_pool.cc:224] Handling call
0709 12:24:27.269140 (+ 30578us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:27.321727 (+ 52587us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:27.321764 (+    37us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:27.333334 (+ 11570us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:24:27.333397 (+    63us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:27.333479 (+    82us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:27.333623 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:27.342079 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 23 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:27.342715 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 24 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:27.343538 17682 raft_consensus.cc:2468] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 23.
I20260709 12:24:27.346132 17728 leader_election.cc:400] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 21 election: Vote denied by peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071) with higher term. Message: Invalid argument: T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for earlier term 21. Current term is 22.
I20260709 12:24:27.347072 17728 leader_election.cc:403] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 21 election: Cancelling election due to peer responding with higher term
I20260709 12:24:27.268885 25979 leader_election.cc:290] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:27.351464 17838 raft_consensus.cc:1240] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 26. Current term is 27. Ops: []
I20260709 12:24:27.251269 17832 rpcz_store.cc:276] 0709 12:24:26.091260 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:26.139190 (+ 47930us) service_pool.cc:224] Handling call
0709 12:24:26.139670 (+   480us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:26.153663 (+ 13993us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 12:24:26.653531 (+499868us) log.cc:844] Serialized 14 byte log entry
0709 12:24:27.250291 (+596760us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:27.250376 (+    85us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:24:27.250483 (+   107us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:27.250577 (+    94us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:27.250731 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":619979,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":42,"thread_start_us":1036372,"threads_started":2}
I20260709 12:24:27.356482 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:27.369618 18371 raft_consensus.cc:3060] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:27.322126 25866 raft_consensus.cc:2749] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:24:27.247848 25917 raft_consensus.cc:515] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:27.317335 26163 raft_consensus.cc:493] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
W20260709 12:24:27.339354 26258 log.cc:927] Time spent T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.081s	user 0.000s	sys 0.000s
I20260709 12:24:27.372615 17687 raft_consensus.cc:1275] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 25 index: 17. Preceding OpId from leader: term: 28 index: 19. (index mismatch)
I20260709 12:24:27.247546 25637 raft_consensus.cc:515] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:27.372864 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:27.372910 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:27.375044 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 19 candidate_status { last_received { term: 17 index: 17 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:27.378031 25131 consensus_queue.cc:1048] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20260709 12:24:27.328493 25657 leader_election.cc:290] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 26 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:27.338474 26233 raft_consensus.cc:697] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 21 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:24:27.408068 25131 raft_consensus.cc:493] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:27.356658 26218 raft_consensus.cc:2804] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Leader election won for term 21
W20260709 12:24:27.418087 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 79932.8us
I20260709 12:24:27.414754 25131 raft_consensus.cc:515] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:27.366430 25909 leader_election.cc:290] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:27.376986 26068 raft_consensus.cc:2749] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:24:27.373811 26234 raft_consensus.cc:3060] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:27.433156 17833 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35042 (request call id 3091) took 1343 ms. Trace:
I20260709 12:24:27.397145 26163 raft_consensus.cc:3060] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:24:27.450740 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 204655us
I20260709 12:24:27.334204 26164 raft_consensus.cc:515] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:27.392277 25688 leader_election.cc:290] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:27.462150 25131 leader_election.cc:290] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:27.390710 25384 raft_consensus.cc:697] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 25 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:24:27.458252 17585 catalog_manager.cc:5697] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 16 to 22, leader changed from 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) to e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66). New cstate: current_term: 22 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:24:27.523535 18371 raft_consensus.cc:2468] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 23.
I20260709 12:24:27.404423 25826 raft_consensus.cc:493] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
W20260709 12:24:27.524590 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 203792us
I20260709 12:24:27.402446 25761 leader_election.cc:290] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 22 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:24:27.537600 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 77563.9us
I20260709 12:24:27.538697 17585 catalog_manager.cc:5697] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 16 to 23, leader changed from 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) to e063541542a94969898868bf6eb5ce1b (127.16.29.67). New cstate: current_term: 23 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:24:27.530838 17829 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35042 (request call id 3100) took 1376 ms. Trace:
I20260709 12:24:27.376386 26175 raft_consensus.cc:515] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:27.455760 17833 rpcz_store.cc:276] 0709 12:24:26.089487 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:26.092379 (+  2892us) service_pool.cc:224] Handling call
0709 12:24:26.092771 (+   392us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:26.116983 (+ 24212us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:26.117070 (+    87us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:26.138007 (+ 20937us) log.cc:844] Serialized 26 byte log entry
0709 12:24:26.528911 (+390904us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:24:26.528962 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:26.529023 (+    61us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:27.432934 (+903911us) raft_consensus.cc:1685] finished
0709 12:24:27.432955 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:27.433096 (+   141us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":15893,"raft.queue_time_us":15225,"raft.run_cpu_time_us":120,"raft.run_wall_time_us":7508,"thread_start_us":390459,"threads_started":1,"wal-append.queue_time_us":528911}
I20260709 12:24:27.541790 17829 rpcz_store.cc:276] 0709 12:24:26.154396 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:26.662866 (+508470us) service_pool.cc:224] Handling call
0709 12:24:26.663320 (+   454us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:24:26.685041 (+ 21721us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:26.685073 (+    32us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:24:26.701222 (+ 16149us) log.cc:844] Serialized 52 byte log entry
0709 12:24:26.937252 (+236030us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:24:27.033414 (+ 96162us) log.cc:844] Serialized 14 byte log entry
0709 12:24:27.033840 (+   426us) log.cc:844] Serialized 14 byte log entry
0709 12:24:27.034040 (+   200us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:27.034144 (+   104us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:27.530604 (+496460us) raft_consensus.cc:1685] finished
0709 12:24:27.530624 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:27.530775 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5770,"prepare.queue_time_us":307592,"prepare.run_cpu_time_us":50,"prepare.run_wall_time_us":50,"raft.queue_time_us":41311,"raft.run_cpu_time_us":359,"raft.run_wall_time_us":34803,"spinlock_wait_cycles":7720320,"thread_start_us":156870,"threads_started":1,"wal-append.queue_time_us":792451}
W20260709 12:24:27.549408 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 225126us
I20260709 12:24:27.551321 17834 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35042 (request call id 3095) took 1460 ms. Trace:
I20260709 12:24:27.551942 17834 rpcz_store.cc:276] 0709 12:24:26.090733 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:26.117247 (+ 26514us) service_pool.cc:224] Handling call
0709 12:24:26.130808 (+ 13561us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:26.166787 (+ 35979us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:26.166933 (+   146us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:26.183006 (+ 16073us) log.cc:844] Serialized 26 byte log entry
0709 12:24:26.630682 (+447676us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:24:26.630738 (+    56us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:26.630828 (+    90us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:27.551042 (+920214us) raft_consensus.cc:1685] finished
0709 12:24:27.551074 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:27.551249 (+   175us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":2820,"raft.run_cpu_time_us":218,"raft.run_wall_time_us":59545,"thread_start_us":403349,"threads_started":1,"wal-append.queue_time_us":522219}
W20260709 12:24:27.528121 26252 log.cc:927] Time spent T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.140s	user 0.001s	sys 0.000s
I20260709 12:24:27.561455 26281 raft_consensus.cc:493] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:27.449242 26218 raft_consensus.cc:697] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 21 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
W20260709 12:24:27.565068 17730 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:27.575325 25384 consensus_queue.cc:237] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 24.13, Last appended by leader: 13, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:27.577365 17585 catalog_manager.cc:5697] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 15 to 22. New cstate: current_term: 22 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:24:27.578547 18333 leader_election.cc:304] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:27.564688 25915 leader_election.cc:290] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:24:27.579349 26249 log.cc:927] Time spent T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.482s	user 0.004s	sys 0.006s
I20260709 12:24:27.564949 26155 leader_election.cc:290] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 27 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:27.581903 25131 raft_consensus.cc:493] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:27.616817 25131 raft_consensus.cc:515] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:27.583076 17652 leader_election.cc:304] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
W20260709 12:24:27.555320 26265 log.cc:927] Time spent T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.438s	user 0.001s	sys 0.000s
I20260709 12:24:27.601511 17585 catalog_manager.cc:5697] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 15 to 20, leader changed from 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) to 0da169ff82674dc1947055ce43de68a3 (127.16.29.65). New cstate: current_term: 20 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:24:27.622721 25951 raft_consensus.cc:2804] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:24:27.623497 25951 raft_consensus.cc:697] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 22 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:24:27.622875 17585 catalog_manager.cc:5697] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 18 to 22. New cstate: current_term: 22 leader_uuid: "334a4e1f59944544b3b3ff36616b57da" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:24:27.624457 25951 consensus_queue.cc:237] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 21.16, Last appended by leader: 16, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:27.625470 17585 catalog_manager.cc:5697] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 5 to 19, leader changed from 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) to 0da169ff82674dc1947055ce43de68a3 (127.16.29.65). New cstate: current_term: 19 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:24:27.627650 17585 catalog_manager.cc:5697] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 11 to 25, leader changed from 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) to e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66). New cstate: current_term: 25 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
W20260709 12:24:27.563791 26057 consensus_meta.cc:287] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.743s	user 0.003s	sys 0.007s
I20260709 12:24:27.633304 26057 raft_consensus.cc:515] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:27.635860 17757 raft_consensus.cc:1240] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 25. Current term is 26. Ops: []
I20260709 12:24:27.629737 17585 catalog_manager.cc:5697] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 15 to 19, leader changed from 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) to e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66). New cstate: current_term: 19 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:24:27.637753 17585 catalog_manager.cc:5697] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 12 to 20, leader changed from e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66) to 0da169ff82674dc1947055ce43de68a3 (127.16.29.65). New cstate: current_term: 20 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:24:27.639777 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:27.642068 17759 raft_consensus.cc:1240] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 25. Current term is 26. Ops: []
I20260709 12:24:27.643544 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 26 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:27.647219 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:27.648012 25131 leader_election.cc:290] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:27.649696 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:27.579865 26289 raft_consensus.cc:2804] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:24:27.659471 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:27.581562 26164 leader_election.cc:290] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:27.672688 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:24:27.667167 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 248003us
I20260709 12:24:27.592136 25637 leader_election.cc:290] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:27.585906 26208 leader_election.cc:290] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:24:27.659749 26258 log.cc:927] Time spent T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.062s	user 0.000s	sys 0.000s
I20260709 12:24:27.677120 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:27.675992 26155 consensus_queue.cc:1059] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 22.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.005s
I20260709 12:24:27.640739 26057 leader_election.cc:290] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 26 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:27.603001 25688 raft_consensus.cc:493] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:27.683715 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 22 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:27.684772 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 26 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:27.685401 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:27.701026 17832 raft_consensus.cc:2393] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 27: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:24:27.666769 26289 raft_consensus.cc:493] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:27.590935 25881 consensus_queue.cc:1059] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.076s
I20260709 12:24:27.686412 17838 raft_consensus.cc:1275] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 20 index: 18. Preceding OpId from leader: term: 21 index: 18. (term mismatch)
I20260709 12:24:27.732096 17838 pending_rounds.cc:85] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b: Aborting all ops after (but not including) 17
I20260709 12:24:27.732355 17838 pending_rounds.cc:107] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b: Aborting uncommitted NO_OP operation due to leader change: 20.18
I20260709 12:24:27.732709 17838 raft_consensus.cc:2889] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:27.733214 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:27.681425 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 27 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:27.553915 25826 raft_consensus.cc:515] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:27.659308 25915 raft_consensus.cc:2749] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
W20260709 12:24:27.696772 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 118343us
I20260709 12:24:27.685820 17689 raft_consensus.cc:3060] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:27.724723 17835 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35042 (request call id 3094) took 1634 ms. Trace:
I20260709 12:24:27.721524 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:27.588711 26218 consensus_queue.cc:237] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 19.14, Last appended by leader: 14, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:27.742408 25967 consensus_meta.cc:287] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 1.114s	user 0.010s	sys 0.001s
I20260709 12:24:27.742309 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:27.742614 17838 raft_consensus.cc:1275] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 24 index: 15. Preceding OpId from leader: term: 26 index: 16. (index mismatch)
I20260709 12:24:27.742995 25967 raft_consensus.cc:515] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:24:27.731107 26301 raft_consensus.cc:493] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:27.745390 25967 leader_election.cc:290] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 25 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:27.683116 26233 consensus_queue.cc:237] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 20.14, Last appended by leader: 14, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:27.745976 26301 raft_consensus.cc:515] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:27.748322 18361 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 2431) took 1473 ms. Trace:
W20260709 12:24:27.748525 26276 log.cc:927] Time spent T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.193s	user 0.000s	sys 0.000s
W20260709 12:24:27.151575 17581 rpcz_store.cc:269] 0709 12:23:05.725967 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:05.726295 (+   328us) service_pool.cc:224] Handling call
0709 12:24:27.150857 (+81424562us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:24:17.412278 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:24:17.413041 (+   763us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:24:17.413089 (+    48us) write_op.cc:435] Acquired schema lock
0709 12:24:17.413106 (+    17us) tablet.cc:662] Decoding operations
0709 12:24:17.413968 (+   862us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:24:17.414072 (+   104us) write_op.cc:641] Partition lock acquired for write op
0709 12:24:17.414105 (+    33us) tablet.cc:685] Acquiring locks for 6 operations
0709 12:24:17.613089 (+198984us) tablet.cc:701] Row locks acquired
0709 12:24:17.643585 (+ 30496us) write_op.cc:260] PREPARE: finished
0709 12:24:17.670221 (+ 26636us) write_op.cc:270] Start()
0709 12:24:17.687186 (+ 16965us) write_op.cc:276] Timestamp: P: 1783599857662347 usec, L: 0
0709 12:24:17.717962 (+ 30776us) op_driver.cc:348] REPLICATION: starting
0709 12:24:17.720360 (+  2398us) log.cc:844] Serialized 2459 byte log entry
0709 12:24:18.835861 (+1115501us) op_driver.cc:464] REPLICATION: finished
0709 12:24:19.364245 (+528384us) write_op.cc:301] APPLY: starting
0709 12:24:19.448051 (+ 83806us) tablet.cc:1367] starting BulkCheckPresence
0709 12:24:20.013616 (+565565us) tablet.cc:1370] finished BulkCheckPresence
0709 12:24:20.102609 (+ 88993us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:24:20.220811 (+118202us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:24:20.249030 (+ 28219us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=12,key_file_lookups=12,delta_file_lookups=12,mrs_lookups=0
0709 12:24:20.249081 (+    51us) write_op.cc:312] APPLY: finished
0709 12:24:20.249434 (+   353us) log.cc:844] Serialized 65 byte log entry
0709 12:24:20.479038 (+229604us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:24:20.479184 (+   146us) write_op.cc:454] Released schema lock
0709 12:24:20.479259 (+    75us) write_op.cc:341] FINISH: Updating metrics
0709 12:24:20.479764 (+   505us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:24:20.479796 (+    32us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":2447675,"reported_tablets":46,"child_traces":[["op",{"apply.queue_time_us":484213,"apply.run_cpu_time_us":37013,"apply.run_wall_time_us":1130056,"cfile_cache_hit":26,"cfile_cache_hit_bytes":29231,"num_ops":6,"prepare.queue_time_us":127138,"prepare.run_cpu_time_us":18791,"prepare.run_wall_time_us":485038,"raft.queue_time_us":414641,"raft.run_cpu_time_us":31605,"raft.run_wall_time_us":266984,"replication_time_us":1092388,"thread_start_us":906427,"threads_started":5,"wal-append.queue_time_us":504660,"wal-append.run_cpu_time_us":57114,"wal-append.run_wall_time_us":2913667}]]}
W20260709 12:24:27.749712 26120 consensus_meta.cc:287] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.607s	user 0.011s	sys 0.002s
I20260709 12:24:27.721139 26289 raft_consensus.cc:3060] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Advancing to term 21
W20260709 12:24:27.750790 26282 log.cc:927] Time spent T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.050s	user 0.000s	sys 0.000s
I20260709 12:24:27.751819 26287 raft_consensus.cc:493] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:24:27.752430 26287 raft_consensus.cc:515] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:27.736652 17686 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37402 (request call id 3527) took 2198 ms. Trace:
I20260709 12:24:27.739735 17837 raft_consensus.cc:3060] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Advancing to term 23
W20260709 12:24:27.722501 26269 log.cc:927] Time spent T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.393s	user 0.001s	sys 0.000s
I20260709 12:24:27.754519 17836 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35042 (request call id 3101) took 1582 ms. Trace:
I20260709 12:24:27.671833 26281 raft_consensus.cc:515] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:27.748754 18361 rpcz_store.cc:276] 0709 12:24:26.274927 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:26.604404 (+329477us) service_pool.cc:224] Handling call
0709 12:24:26.756009 (+151605us) spinlock_profiling.cc:243] Waited 3.73 ms on lock 0x7b5800102398. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b474fbb71 00007f7b474fba22 00007f7b504148d9 00007f7b4f56eee0 00007f7b4f6380ab 00007f7b4f638131 00007f7b4f5f9fc0 00007f7b4f5e4929 00007f7b4f5e43d2 00007f7b48b5bbd4
0709 12:24:26.756233 (+   224us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:26.936526 (+180293us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:26.936558 (+    32us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:27.011416 (+ 74858us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:24:27.011468 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:27.748113 (+736645us) spinlock_profiling.cc:243] Waited 138 ms on lock 0x7b54006946d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:24:27.748158 (+    45us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:27.748269 (+   111us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":325102208}
I20260709 12:24:27.749315 26299 raft_consensus.cc:493] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:27.762225 18333 leader_election.cc:304] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:27.757414 17836 rpcz_store.cc:276] 0709 12:24:26.172039 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:26.815477 (+643438us) service_pool.cc:224] Handling call
0709 12:24:26.833551 (+ 18074us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:26.834007 (+   456us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:26.834057 (+    50us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:26.834702 (+   645us) log.cc:844] Serialized 26 byte log entry
0709 12:24:27.017860 (+183158us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:24:27.017897 (+    37us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:27.017964 (+    67us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:27.754314 (+736350us) raft_consensus.cc:1685] finished
0709 12:24:27.754334 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:27.754459 (+   125us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":7846,"raft.run_cpu_time_us":170,"raft.run_wall_time_us":18953,"thread_start_us":117440,"threads_started":1,"wal-append.queue_time_us":811546}
I20260709 12:24:27.754673 17686 rpcz_store.cc:276] 0709 12:24:25.526401 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:25.527018 (+   617us) service_pool.cc:224] Handling call
0709 12:24:25.527396 (+   378us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 12:24:25.623108 (+ 95712us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 12:24:26.389988 (+766880us) log.cc:844] Serialized 14 byte log entry
0709 12:24:26.928326 (+538338us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:24:26.944110 (+ 15784us) log.cc:844] Serialized 52 byte log entry
0709 12:24:27.014458 (+ 70348us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:24:27.017827 (+  3369us) log.cc:844] Serialized 14 byte log entry
0709 12:24:27.241134 (+223307us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:27.241220 (+    86us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:27.724781 (+483561us) raft_consensus.cc:1685] finished
0709 12:24:27.724802 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:27.724940 (+   138us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":118188,"prepare.queue_time_us":1047936,"prepare.run_cpu_time_us":153,"prepare.run_wall_time_us":31175,"raft.queue_time_us":363579,"raft.run_cpu_time_us":481,"raft.run_wall_time_us":23547,"thread_start_us":1365597,"threads_started":4,"wal-append.queue_time_us":460550}
I20260709 12:24:27.724257 26300 raft_consensus.cc:493] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:27.764870 18333 leader_election.cc:304] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:24:27.724674 26164 raft_consensus.cc:493] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:27.779062 17833 raft_consensus.cc:1275] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 24 index: 16. Preceding OpId from leader: term: 25 index: 17. (index mismatch)
I20260709 12:24:27.756439 17835 rpcz_store.cc:276] 0709 12:24:26.090420 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:26.093789 (+  3369us) service_pool.cc:224] Handling call
0709 12:24:26.094103 (+   314us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:26.094577 (+   474us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:26.094608 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:26.095186 (+   578us) log.cc:844] Serialized 26 byte log entry
0709 12:24:26.623593 (+528407us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:24:26.623783 (+   190us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:26.623993 (+   210us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:27.724478 (+1100485us) raft_consensus.cc:1685] finished
0709 12:24:27.724504 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:27.724668 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":104,"raft.run_cpu_time_us":88,"raft.run_wall_time_us":88,"thread_start_us":499984,"threads_started":1,"wal-append.queue_time_us":770184}
I20260709 12:24:27.731640 26271 raft_consensus.cc:493] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:27.722312 25688 raft_consensus.cc:515] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:27.763969 25967 raft_consensus.cc:493] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:27.757028 25979 raft_consensus.cc:493] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:24:27.769794 25917 leader_election.cc:290] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:27.770269 26208 raft_consensus.cc:2749] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:24:27.849902 17837 raft_consensus.cc:2468] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 23.
I20260709 12:24:27.795629 26155 raft_consensus.cc:493] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:27.836233 25740 raft_consensus.cc:3055] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:24:27.778141 26299 raft_consensus.cc:515] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:27.871126 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:27.872215 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:27.872382 18333 leader_election.cc:304] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:24:27.869364 26009 consensus_meta.cc:287] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 1.860s	user 0.014s	sys 0.005s
I20260709 12:24:27.876561 26009 raft_consensus.cc:515] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:27.857776 25967 raft_consensus.cc:515] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:27.888605 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:27.842042 25915 raft_consensus.cc:2804] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Leader election won for term 22
W20260709 12:24:27.867929 26206 consensus_meta.cc:287] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.852s	user 0.011s	sys 0.000s
I20260709 12:24:27.893147 17680 raft_consensus.cc:2393] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 19: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:24:27.896481 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:27.868193 25740 raft_consensus.cc:740] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 20 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:24:27.891680 26287 leader_election.cc:290] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:27.904592 25740 consensus_queue.cc:260] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 20.15, Last appended by leader: 15, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:27.839325 26271 raft_consensus.cc:515] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:27.841081 26300 raft_consensus.cc:515] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:27.907276 26271 leader_election.cc:290] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:27.908115 26300 leader_election.cc:290] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:27.909128 26299 leader_election.cc:290] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:27.910471 26281 leader_election.cc:290] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:27.867652 25881 raft_consensus.cc:493] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:27.911921 25881 raft_consensus.cc:515] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:27.913535 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 48862.7us
I20260709 12:24:27.914551 25881 leader_election.cc:290] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:24:27.901198 26262 log.cc:927] Time spent T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.058s	user 0.000s	sys 0.001s
I20260709 12:24:27.915191 17801 leader_election.cc:304] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:27.917218 26156 raft_consensus.cc:2749] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:24:27.870630 26175 leader_election.cc:290] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:27.919261 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 24 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:27.919901 17756 raft_consensus.cc:2468] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 23.
I20260709 12:24:27.839022 26164 raft_consensus.cc:515] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:27.914388 26298 consensus_queue.cc:1048] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
W20260709 12:24:27.925931 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 584651us
I20260709 12:24:27.922291 17689 raft_consensus.cc:2468] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 27.
I20260709 12:24:27.928774 26163 raft_consensus.cc:515] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:27.930725 26289 raft_consensus.cc:515] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:27.932876 26164 leader_election.cc:290] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:27.933497 25915 raft_consensus.cc:697] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 22 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:24:27.934032 25688 leader_election.cc:290] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:27.931080 26163 leader_election.cc:290] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 24 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:27.934621 25915 consensus_queue.cc:237] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.11, Last appended by leader: 11, Current term: 22, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:27.936380 17834 raft_consensus.cc:1240] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 20. Current term is 21. Ops: []
I20260709 12:24:27.938555 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:27.960821 25881 raft_consensus.cc:493] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:27.961513 25881 raft_consensus.cc:515] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:27.946332 26261 raft_consensus.cc:493] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:27.964489 26261 raft_consensus.cc:515] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:27.899663 26301 leader_election.cc:290] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:27.965408 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 25 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:27.891852 26155 raft_consensus.cc:515] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:27.966619 26261 leader_election.cc:290] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:27.967818 26155 leader_election.cc:290] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:27.943562 26163 raft_consensus.cc:493] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:24:27.900447 26206 raft_consensus.cc:515] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:27.969190 26163 raft_consensus.cc:515] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:27.971454 26163 leader_election.cc:290] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:27.965453 25826 leader_election.cc:290] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:27.964004 25881 leader_election.cc:290] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:27.969179 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:24:27.977246 26234 consensus_meta.cc:287] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.509s	user 0.009s	sys 0.000s
I20260709 12:24:27.977754 26234 raft_consensus.cc:2749] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Leader election lost for term 21. Reason: Vote denied by peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071) with higher term. Message: Invalid argument: T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for earlier term 21. Current term is 22.
I20260709 12:24:27.981925 17758 raft_consensus.cc:2468] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 24.
I20260709 12:24:27.971429 26206 leader_election.cc:290] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 23 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:27.996467 17764 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38786 (request call id 3789) took 1143 ms. Trace:
I20260709 12:24:27.935073 26289 leader_election.cc:290] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 21 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:27.997092 17764 rpcz_store.cc:276] 0709 12:24:26.853680 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:26.853832 (+   152us) service_pool.cc:224] Handling call
0709 12:24:26.878031 (+ 24199us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:27.149969 (+271938us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:27.996133 (+846164us) spinlock_profiling.cc:243] Waited 271 ms on lock 0x7b54001b8354. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:24:27.996387 (+   254us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":623800576}
I20260709 12:24:28.010427 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:28.011346 18371 raft_consensus.cc:2468] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 25.
W20260709 12:24:28.022619 26243 log.cc:927] Time spent T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 1.280s	user 0.004s	sys 0.000s
I20260709 12:24:28.041484 17760 raft_consensus.cc:2468] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 22.
W20260709 12:24:28.042604 17801 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.16.29.66:39651 timed out after 1.808s (SENT) [suppressed 20 similar messages]
I20260709 12:24:28.047673 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:24:28.054149 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 181862us
I20260709 12:24:28.055675 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:28.056097 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 24 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:28.056798 18333 leader_election.cc:304] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:28.036123 26233 raft_consensus.cc:493] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:28.059626 18363 raft_consensus.cc:2468] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 23.
I20260709 12:24:28.061115 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:28.061676 18362 raft_consensus.cc:3060] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:27.866261 25979 raft_consensus.cc:515] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:28.080233 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:28.079596 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 23 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:28.082921 17829 raft_consensus.cc:2468] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 22.
I20260709 12:24:28.084148 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:28.084893 17829 raft_consensus.cc:2468] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 26.
I20260709 12:24:28.086040 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:28.089912 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:28.114284 25909 consensus_queue.cc:1048] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:24:28.075954 26233 raft_consensus.cc:515] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:28.073599 26289 raft_consensus.cc:493] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:28.122566 26289 raft_consensus.cc:515] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:28.124182 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:28.124832 17837 raft_consensus.cc:3060] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:28.126575 17763 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38786 (request call id 3791) took 1272 ms. Trace:
I20260709 12:24:28.126962 17763 rpcz_store.cc:276] 0709 12:24:26.854455 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:26.860469 (+  6014us) service_pool.cc:224] Handling call
0709 12:24:28.126344 (+1265875us) spinlock_profiling.cc:243] Waited 330 ms on lock 0x7b5400415fd4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:24:28.126488 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":759272832}
I20260709 12:24:28.128257 18364 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59886 (request call id 2207) took 1347 ms. Trace:
I20260709 12:24:28.128346 17652 leader_election.cc:304] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:24:28.128731 18364 rpcz_store.cc:276] 0709 12:24:26.780908 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:26.791562 (+ 10654us) service_pool.cc:224] Handling call
0709 12:24:26.828132 (+ 36570us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:26.869213 (+ 41081us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:28.128061 (+1258848us) spinlock_profiling.cc:243] Waited 36.1 ms on lock 0x7b5400305550. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a61f8ad 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:24:28.128188 (+   127us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":83090816}
I20260709 12:24:28.130337 17652 leader_election.cc:304] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:24:28.131206 17652 leader_election.cc:304] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:24:28.132221 17761 raft_consensus.cc:1275] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 24 index: 14. Preceding OpId from leader: term: 26 index: 15. (index mismatch)
I20260709 12:24:28.114583 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:28.133494 17836 raft_consensus.cc:3060] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:28.136660 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 23 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:28.137333 17763 raft_consensus.cc:2468] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 22.
I20260709 12:24:28.137743 17837 raft_consensus.cc:2468] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 25.
I20260709 12:24:28.064036 26300 raft_consensus.cc:493] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:28.069044 26208 raft_consensus.cc:2749] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:24:28.055567 25917 raft_consensus.cc:2749] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:24:28.173748 18362 raft_consensus.cc:2468] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 23.
W20260709 12:24:28.175613 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 133038us
I20260709 12:24:28.178233 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 22 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:28.179066 17837 raft_consensus.cc:2410] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for term 22 because replica has last-logged OpId of term: 20 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 12.
I20260709 12:24:28.179857 17652 leader_election.cc:304] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:28.153225 26097 raft_consensus.cc:2804] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:24:28.087035 26287 consensus_queue.cc:1048] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.086s
I20260709 12:24:28.146096 25761 consensus_queue.cc:1048] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 14, Time since last communication: 0.034s
I20260709 12:24:28.186141 17652 leader_election.cc:304] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:28.187022 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:28.190083 18364 raft_consensus.cc:1275] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 24 index: 17. Preceding OpId from leader: term: 25 index: 18. (index mismatch)
I20260709 12:24:28.195241 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:28.167991 26218 raft_consensus.cc:493] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:28.195657 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:28.197624 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 27 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:28.180823 26260 raft_consensus.cc:2804] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:24:28.200678 18364 raft_consensus.cc:2468] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 26.
I20260709 12:24:28.201918 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:28.202276 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:28.205343 18363 raft_consensus.cc:2468] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 21.
I20260709 12:24:28.205869 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:28.180655 26208 raft_consensus.cc:493] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
W20260709 12:24:28.193512 26316 log.cc:927] Time spent T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.163s	user 0.002s	sys 0.000s
I20260709 12:24:28.210675 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:28.158411 26300 raft_consensus.cc:515] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:28.179924 26310 raft_consensus.cc:2804] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:24:28.213454 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:28.214051 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:28.216591 18371 raft_consensus.cc:2468] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 19.
I20260709 12:24:28.184514 26313 raft_consensus.cc:493] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:28.217849 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:28.218173 18364 raft_consensus.cc:2393] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 25: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:24:28.219123 18363 raft_consensus.cc:2468] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 18.
I20260709 12:24:28.186429 25826 raft_consensus.cc:2749] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:24:28.196760 25917 raft_consensus.cc:493] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:28.226832 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 23 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:28.228066 18371 raft_consensus.cc:2410] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 23 because replica has last-logged OpId of term: 22 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 17.
I20260709 12:24:28.229831 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:28.232136 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:28.218577 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:24:28.233572 17575 tablet_replica.cc:1411] Time spent flushing the Log queue took a long time: real 0.271s	user 0.009s	sys 0.026s
I20260709 12:24:28.234764 18362 raft_consensus.cc:2468] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 21.
I20260709 12:24:28.235203 18363 raft_consensus.cc:2393] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 24: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:24:28.235720 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:28.236002 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 24 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:28.184747 25384 consensus_queue.cc:1059] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.029s
I20260709 12:24:28.202526 26294 raft_consensus.cc:493] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:24:28.209398 26260 raft_consensus.cc:493] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:28.222802 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:28.218576 26310 raft_consensus.cc:493] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:28.217494 25979 leader_election.cc:290] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:28.236572 18363 raft_consensus.cc:2468] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 23.
I20260709 12:24:28.224291 25951 raft_consensus.cc:2749] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:24:28.241803 26289 leader_election.cc:290] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:28.243569 26310 raft_consensus.cc:3060] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:28.246160 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 25 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:28.246765 18374 raft_consensus.cc:2468] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 24.
I20260709 12:24:28.238513 26233 leader_election.cc:290] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:28.252074 26163 raft_consensus.cc:493] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:28.253089 18363 raft_consensus.cc:1240] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 21 LEADER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 20. Current term is 21. Ops: []
I20260709 12:24:28.252978 26163 raft_consensus.cc:515] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:28.256678 25384 raft_consensus.cc:3055] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:24:28.257210 25384 raft_consensus.cc:740] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:24:28.258078 25384 consensus_queue.cc:260] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 25.15, Last appended by leader: 15, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:28.258937 26310 raft_consensus.cc:515] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:28.261719 26310 leader_election.cc:290] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 23 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:28.266409 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 21 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:28.267210 17837 raft_consensus.cc:2468] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 20.
I20260709 12:24:28.243530 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:28.219318 26208 raft_consensus.cc:515] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:28.259614 25384 raft_consensus.cc:3060] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:28.264040 26163 leader_election.cc:290] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:28.264992 26300 leader_election.cc:290] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:28.209159 26218 raft_consensus.cc:515] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:28.268234 18364 raft_consensus.cc:2468] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 23.
I20260709 12:24:28.269210 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:28.240962 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 21 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:28.247824 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 21 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:28.274458 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 19 candidate_status { last_received { term: 17 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:28.290652 18362 raft_consensus.cc:2468] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 18.
I20260709 12:24:28.290824 18374 raft_consensus.cc:2410] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 21 because replica has last-logged OpId of term: 18 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 14.
I20260709 12:24:28.276971 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:28.280357 26112 raft_consensus.cc:3055] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:24:28.244537 26260 raft_consensus.cc:3060] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:28.284225 18365 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59886 (request call id 2194) took 2025 ms. Trace:
I20260709 12:24:28.222385 26313 raft_consensus.cc:515] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:28.238229 25917 raft_consensus.cc:515] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:28.285567 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:28.287174 18366 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59886 (request call id 2199) took 2016 ms. Trace:
I20260709 12:24:28.293910 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 23 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:28.301256 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 22 candidate_status { last_received { term: 19 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:28.302367 17837 raft_consensus.cc:2468] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 21.
I20260709 12:24:28.304702 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 24 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:28.305636 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 23 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:28.307739 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:28.308018 17838 raft_consensus.cc:2393] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 23: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:24:28.244678 26294 raft_consensus.cc:515] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:28.310546 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 24 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:28.294824 18365 rpcz_store.cc:276] 0709 12:24:26.258611 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:26.258763 (+   152us) service_pool.cc:224] Handling call
0709 12:24:26.259159 (+   396us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:26.600843 (+341684us) raft_consensus.cc:1534] Early marking committed up to index 16
0709 12:24:26.928409 (+327566us) log.cc:844] Serialized 14 byte log entry
0709 12:24:27.090203 (+161794us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:27.090269 (+    66us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:24:27.090335 (+    66us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:28.283099 (+1192764us) spinlock_profiling.cc:243] Waited 10.5 ms on lock 0x7b180042f220. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b475012a6 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419
0709 12:24:28.283909 (+   810us) spinlock_profiling.cc:243] Waited 341 ms on lock 0x7b540047ca54. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:24:28.283967 (+    58us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:28.284145 (+   178us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":52109,"prepare.queue_time_us":82804,"prepare.run_cpu_time_us":72,"prepare.run_wall_time_us":4760,"raft.queue_time_us":39412,"raft.run_cpu_time_us":169,"raft.run_wall_time_us":11272,"spinlock_wait_cycles":809447552,"thread_start_us":298036,"threads_started":2,"wal-append.queue_time_us":755981}
I20260709 12:24:28.311179 17834 raft_consensus.cc:2468] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 23.
I20260709 12:24:28.311487 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:24:28.312147 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 99399.8us
I20260709 12:24:28.293099 26112 raft_consensus.cc:740] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:24:28.297101 18366 rpcz_store.cc:276] 0709 12:24:26.270855 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:26.425309 (+154454us) service_pool.cc:224] Handling call
0709 12:24:26.425670 (+   361us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:26.596906 (+171236us) raft_consensus.cc:1534] Early marking committed up to index 15
0709 12:24:26.825038 (+228132us) log.cc:844] Serialized 14 byte log entry
0709 12:24:27.067583 (+242545us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:27.067636 (+    53us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:24:27.067673 (+    37us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:27.141836 (+ 74163us) spinlock_profiling.cc:243] Waited 10.7 ms on lock 0x7b180003b380. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b475333f3 00007f7b475357c1 00007f7b47535779 00007f7b47535709 00007f7b475356d1 00007f7b475349cd 0000000000388634 0000000000388479 00007f7b4751683a
0709 12:24:28.286972 (+1145136us) spinlock_profiling.cc:243] Waited 171 ms on lock 0x7b54007701d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:24:28.287019 (+    47us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:28.287124 (+   105us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":22652,"prepare.queue_time_us":30910,"prepare.run_cpu_time_us":83,"prepare.run_wall_time_us":23273,"raft.queue_time_us":78103,"raft.run_cpu_time_us":188,"raft.run_wall_time_us":20875,"spinlock_wait_cycles":417879168,"thread_start_us":385229,"threads_started":2,"wal-append.queue_time_us":267303,"wal-append.run_cpu_time_us":910,"wal-append.run_wall_time_us":1004200}
I20260709 12:24:28.315584 18371 raft_consensus.cc:3060] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Advancing to term 21
W20260709 12:24:28.328750 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 129674us
I20260709 12:24:28.330605 17803 leader_election.cc:304] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3
W20260709 12:24:28.330971 17727 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:28.334547 17831 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37994 (request call id 1545) took 2819 ms (client timeout 2638 ms). Trace:
I20260709 12:24:28.335220 17652 leader_election.cc:304] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:24:28.334988 17831 rpcz_store.cc:269] 0709 12:24:25.502769 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:25.508494 (+  5725us) service_pool.cc:224] Handling call
0709 12:24:28.322422 (+2813928us) spinlock_profiling.cc:243] Waited 1.34 s on lock 0x7b540008b6d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:24:28.322549 (+   127us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3074104192}
I20260709 12:24:28.340476 25826 raft_consensus.cc:2749] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:24:28.340579 26112 consensus_queue.cc:260] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 24.17, Last appended by leader: 17, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:28.342051 26112 raft_consensus.cc:3060] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:28.342690 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 24 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:28.343415 17764 raft_consensus.cc:2468] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 23.
I20260709 12:24:28.345567 25917 leader_election.cc:290] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:28.350834 26234 raft_consensus.cc:493] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:28.351559 26234 raft_consensus.cc:515] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:28.352164 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:28.352499 26163 raft_consensus.cc:493] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:28.352938 26163 raft_consensus.cc:515] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:28.354063 26218 leader_election.cc:290] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:28.317191 26300 consensus_queue.cc:1048] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.038s
I20260709 12:24:28.353992 26234 leader_election.cc:290] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:28.355854 26294 leader_election.cc:290] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:28.356444 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:28.357264 26208 leader_election.cc:290] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:28.358355 17803 leader_election.cc:304] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:28.352279 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 26 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:28.360625 17803 leader_election.cc:304] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:28.361070 17652 leader_election.cc:304] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:28.364916 17803 leader_election.cc:304] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:28.368156 17652 leader_election.cc:304] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:28.369385 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:28.371248 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 24 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:28.371832 18367 raft_consensus.cc:2468] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 22.
I20260709 12:24:28.374284 18371 raft_consensus.cc:2468] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 21.
I20260709 12:24:28.374785 18333 leader_election.cc:304] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:28.375980 17803 leader_election.cc:304] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:28.376433 18333 leader_election.cc:304] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:28.380157 17803 leader_election.cc:304] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:24:28.382296 17803 leader_election.cc:304] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:28.380398 18333 leader_election.cc:304] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:24:28.383191 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:28.365494 25881 raft_consensus.cc:2804] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:24:28.365785 26289 raft_consensus.cc:2804] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:24:28.362157 26097 raft_consensus.cc:697] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 23 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:24:28.365651 26300 raft_consensus.cc:2804] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:24:28.372386 26294 raft_consensus.cc:2749] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:24:28.387477 17803 leader_election.cc:304] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:24:28.393903 25688 raft_consensus.cc:2804] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:24:28.394309 25688 raft_consensus.cc:493] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:28.394693 25688 raft_consensus.cc:3060] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:28.395133 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 23 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:28.395785 17762 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 22.
I20260709 12:24:28.378428 26239 raft_consensus.cc:2804] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:24:28.401136 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 23 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:28.401307 26239 raft_consensus.cc:493] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:28.401714 17762 raft_consensus.cc:3060] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:28.380692 25917 raft_consensus.cc:2749] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:24:28.402626 25917 raft_consensus.cc:493] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:28.403118 25917 raft_consensus.cc:515] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:28.405807 25917 leader_election.cc:290] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:28.392339 26289 raft_consensus.cc:493] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:28.406674 26289 raft_consensus.cc:3060] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:28.407955 17729 leader_election.cc:304] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:28.410399 18365 raft_consensus.cc:1275] T a5732e71b9d54f8786b8c511d69943bf P 334a4e1f59944544b3b3ff36616b57da [term 24 LEARNER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 24 index: 14. Preceding OpId from leader: term: 24 index: 15. (index mismatch)
I20260709 12:24:28.411438 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:28.412070 18374 raft_consensus.cc:2468] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 24.
I20260709 12:24:28.412667 25784 raft_consensus.cc:2804] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:24:28.412827 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 23 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:28.413494 17829 raft_consensus.cc:2468] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 22.
I20260709 12:24:28.414176 17729 leader_election.cc:304] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:28.414299 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:28.415004 17829 raft_consensus.cc:2393] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 25: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:24:28.388255 25866 raft_consensus.cc:2804] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:24:28.367635 26234 consensus_queue.cc:1059] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 26.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.024s
I20260709 12:24:28.388351 26149 raft_consensus.cc:3055] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:24:28.401712 26239 raft_consensus.cc:3060] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:28.434549 17757 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38786 (request call id 3799) took 1295 ms. Trace:
I20260709 12:24:28.425233 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:28.417330 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:28.418247 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 26 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:28.426582 17836 raft_consensus.cc:2468] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 19.
I20260709 12:24:28.416533 26281 raft_consensus.cc:2804] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:24:28.388599 25881 raft_consensus.cc:493] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:28.438957 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:28.439122 26281 raft_consensus.cc:493] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:28.439383 25881 raft_consensus.cc:3060] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:28.423450 17652 leader_election.cc:304] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:24:28.392755 26218 raft_consensus.cc:2804] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:24:28.424503 26149 raft_consensus.cc:740] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 25 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:24:28.448562 26218 raft_consensus.cc:493] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:28.449061 26218 raft_consensus.cc:3060] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:28.449433 26149 consensus_queue.cc:260] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 25.15, Last appended by leader: 15, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:28.450495 26149 raft_consensus.cc:3060] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:28.389827 26301 raft_consensus.cc:2749] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:24:28.439617 26281 raft_consensus.cc:3060] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:28.437155 25784 raft_consensus.cc:697] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:24:28.453063 25967 leader_election.cc:290] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:28.453027 25784 consensus_queue.cc:237] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 19.14, Last appended by leader: 14, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:28.455003 26313 leader_election.cc:290] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:28.416409 25866 raft_consensus.cc:493] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:28.436529 17757 rpcz_store.cc:276] 0709 12:24:27.139159 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:27.139280 (+   121us) service_pool.cc:224] Handling call
0709 12:24:27.139595 (+   315us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:27.636323 (+496728us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:28.434261 (+797938us) spinlock_profiling.cc:243] Waited 496 ms on lock 0x7b5400216e54. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:24:28.434480 (+   219us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1140726144}
I20260709 12:24:28.380918 26155 raft_consensus.cc:2749] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:24:28.455837 25866 raft_consensus.cc:3060] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:28.368058 17575 maintenance_manager.cc:643] P 2193af31cb894ec9b3d65d939a2a6851: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 1.625s	user 0.035s	sys 0.037s Metrics: {"bytes_written":4389,"delete_count":0,"lbm_write_time_us":47942,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":4,"raft.queue_time_us":84094,"reinsert_count":0,"thread_start_us":220084,"threads_started":2,"update_count":153,"wal-append.queue_time_us":230938}
I20260709 12:24:28.423609 26234 raft_consensus.cc:3055] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:24:28.457290 26234 raft_consensus.cc:740] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 26 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:24:28.417703 26260 raft_consensus.cc:515] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:28.457947 17576 maintenance_manager.cc:419] P 2193af31cb894ec9b3d65d939a2a6851: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.240142
I20260709 12:24:28.392536 26300 raft_consensus.cc:493] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:28.458886 26300 raft_consensus.cc:3060] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:28.459465 26260 leader_election.cc:290] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 27 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:24:28.460676 17680 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44898 (request call id 1625) took 1789 ms (client timeout 1590 ms). Trace:
W20260709 12:24:28.461113 17680 rpcz_store.cc:269] 0709 12:24:26.670863 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:26.671419 (+   556us) service_pool.cc:224] Handling call
0709 12:24:28.460423 (+1789004us) spinlock_profiling.cc:243] Waited 1.22 s on lock 0x7b5400207e54. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:24:28.460571 (+   148us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2800754432}
I20260709 12:24:28.443833 17801 leader_election.cc:304] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:28.439551 17834 raft_consensus.cc:2468] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 16.
I20260709 12:24:28.415850 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:28.464385 17829 raft_consensus.cc:3060] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:28.464618 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 23 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:28.440142 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:28.465485 25384 raft_consensus.cc:2804] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:24:28.465349 17801 leader_election.cc:304] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:28.465166 17834 raft_consensus.cc:3060] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:28.422044 17762 raft_consensus.cc:2468] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 23.
I20260709 12:24:28.440773 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 23 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:28.466971 17836 raft_consensus.cc:2410] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for term 23 because replica has last-logged OpId of term: 19 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 11.
I20260709 12:24:28.465847 17755 raft_consensus.cc:2376] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 26. Re-sending same reply.
I20260709 12:24:28.468302 25384 raft_consensus.cc:697] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 25 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:24:28.463421 17727 leader_election.cc:304] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:24:28.469539 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 89195.1us
I20260709 12:24:28.469211 17801 leader_election.cc:304] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:28.488845 17652 leader_election.cc:304] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:24:28.482064 18333 leader_election.cc:304] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:24:28.494246 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:28.499150 18374 raft_consensus.cc:3060] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:28.437071 17803 leader_election.cc:304] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:24:28.392691 25915 raft_consensus.cc:2749] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
W20260709 12:24:28.505590 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 91484.1us
I20260709 12:24:28.491195 17652 leader_election.cc:304] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:24:28.506379 25384 consensus_queue.cc:237] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 24.14, Last appended by leader: 14, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
W20260709 12:24:28.516251 17652 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:28.514911 17829 raft_consensus.cc:2468] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 21.
I20260709 12:24:28.504724 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 24 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:28.496063 18334 leader_election.cc:304] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:28.427224 26322 raft_consensus.cc:2749] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:24:28.470361 26155 raft_consensus.cc:2804] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:24:28.469808 26263 raft_consensus.cc:493] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:24:28.525911 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 22 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:28.525336 18363 raft_consensus.cc:1240] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 20. Current term is 21. Ops: []
I20260709 12:24:28.531811 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:28.538223 17829 raft_consensus.cc:3060] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:28.538564 18334 leader_election.cc:304] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:24:28.533281 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 22 candidate_status { last_received { term: 19 index: 17 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:28.535624 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 21 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:28.483629 26164 raft_consensus.cc:2804] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:24:28.421865 26097 consensus_queue.cc:237] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 21.16, Last appended by leader: 16, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:28.554654 25740 raft_consensus.cc:3060] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:28.484187 26120 raft_consensus.cc:2764] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Leader pre-election decision vote started in defunct term 22: won
I20260709 12:24:28.513258 26260 raft_consensus.cc:2804] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:24:28.544464 17832 raft_consensus.cc:1240] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 24. Current term is 25. Ops: []
I20260709 12:24:28.548178 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:28.548674 25826 raft_consensus.cc:493] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:24:28.555562 17684 raft_consensus.cc:1275] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 19 index: 14. Preceding OpId from leader: term: 21 index: 15. (index mismatch)
I20260709 12:24:28.541867 17834 raft_consensus.cc:2468] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 23.
I20260709 12:24:28.527446 26295 raft_consensus.cc:2804] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:24:28.543519 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 28 candidate_status { last_received { term: 27 index: 21 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:28.561524 18374 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 22.
I20260709 12:24:28.522365 26208 raft_consensus.cc:2804] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:24:28.582013 26208 raft_consensus.cc:493] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:28.582535 26208 raft_consensus.cc:3060] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:28.584832 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:28.586077 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 23 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:28.568508 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 21 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:28.569114 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:28.570407 17689 raft_consensus.cc:1240] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 21. Current term is 22. Ops: []
I20260709 12:24:28.575350 17687 raft_consensus.cc:1240] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 26. Current term is 27. Ops: []
I20260709 12:24:28.591125 18367 raft_consensus.cc:1275] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 15 index: 16. Preceding OpId from leader: term: 18 index: 18. (index mismatch)
I20260709 12:24:28.592744 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 26 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:28.593583 17683 raft_consensus.cc:2410] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for term 26 because replica has last-logged OpId of term: 25 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 14.
I20260709 12:24:28.594146 18367 raft_consensus.cc:1240] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 24. Current term is 25. Ops: []
I20260709 12:24:28.580092 25826 raft_consensus.cc:515] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:28.572219 25917 raft_consensus.cc:2749] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:24:28.596738 26163 leader_election.cc:290] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:28.596910 25826 leader_election.cc:290] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:28.597322 26163 raft_consensus.cc:2804] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:24:28.597843 25637 consensus_queue.cc:1048] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.012s
I20260709 12:24:28.585209 26149 raft_consensus.cc:2804] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:24:28.612465 26149 raft_consensus.cc:493] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:28.612864 26149 raft_consensus.cc:3060] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:28.611457 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:28.580634 26260 raft_consensus.cc:697] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 19 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:24:28.572146 17729 leader_election.cc:304] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:28.576658 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:28.468020 26289 raft_consensus.cc:515] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:28.577682 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 26 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:24:28.620179 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 48060us
I20260709 12:24:28.580036 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:28.621060 17688 raft_consensus.cc:2410] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for term 26 because replica has last-logged OpId of term: 25 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 14.
I20260709 12:24:28.505714 26267 consensus_queue.cc:1059] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 20.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.064s
I20260709 12:24:28.537308 26155 raft_consensus.cc:493] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:28.576951 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:28.579243 26295 raft_consensus.cc:493] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:24:28.561878 26309 consensus_queue.cc:1059] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 19.15, Next index: 16, Last known committed idx: 14, Time since last communication: 0.039s
W20260709 12:24:28.618983 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 149824us
I20260709 12:24:28.496553 26301 consensus_queue.cc:1048] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.064s
I20260709 12:24:28.588541 25915 raft_consensus.cc:2804] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:24:28.523494 17652 leader_election.cc:304] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:28.599366 26163 raft_consensus.cc:493] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:28.627351 18367 raft_consensus.cc:1275] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 23 index: 14. Preceding OpId from leader: term: 26 index: 16. (index mismatch)
I20260709 12:24:28.561488 26263 raft_consensus.cc:515] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:28.627842 17829 raft_consensus.cc:2468] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 27.
I20260709 12:24:28.624507 18361 raft_consensus.cc:2393] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 21: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:24:28.661275 17834 raft_consensus.cc:1240] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 19. Current term is 20. Ops: []
I20260709 12:24:28.580570 26114 consensus_queue.cc:1059] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 20.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.042s
I20260709 12:24:28.597911 26120 consensus_queue.cc:1059] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.034s
I20260709 12:24:28.668757 18367 raft_consensus.cc:1240] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 22. Current term is 23. Ops: []
I20260709 12:24:28.669612 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 24 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:28.626253 26261 raft_consensus.cc:2804] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:24:28.630625 26155 raft_consensus.cc:3060] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:24:28.671337 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 147943us
I20260709 12:24:28.689149 17829 raft_consensus.cc:1275] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 17 index: 14. Preceding OpId from leader: term: 20 index: 16. (index mismatch)
I20260709 12:24:28.609267 25881 raft_consensus.cc:515] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:28.631772 26295 raft_consensus.cc:3060] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:28.684443 17688 raft_consensus.cc:2468] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 23.
I20260709 12:24:28.691239 17837 raft_consensus.cc:1275] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 24 index: 13. Preceding OpId from leader: term: 25 index: 14. (index mismatch)
W20260709 12:24:28.694222 17801 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:28.634663 25915 raft_consensus.cc:493] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:28.661453 26163 raft_consensus.cc:3060] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:28.703217 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:28.703902 17681 raft_consensus.cc:3060] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:28.704721 17802 leader_election.cc:304] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:28.704962 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:28.705608 17837 raft_consensus.cc:2468] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 21.
I20260709 12:24:28.706434 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 21 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:28.707067 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 21 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:28.708421 17687 raft_consensus.cc:1240] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 25 LEADER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 24. Current term is 25. Ops: []
I20260709 12:24:28.708962 17837 raft_consensus.cc:2393] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 21: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:24:28.706631 18333 leader_election.cc:304] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:24:28.709527 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 23 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:24:28.709821 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 272829us
W20260709 12:24:28.711645 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 88022.5us
I20260709 12:24:28.710896 17803 leader_election.cc:304] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:24:28.716583 17686 raft_consensus.cc:2393] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 23: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:24:28.717569 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:28.718936 17832 raft_consensus.cc:1240] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 26. Current term is 27. Ops: [26.16-26.16]
I20260709 12:24:28.693529 26175 raft_consensus.cc:2749] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:24:28.719945 17652 leader_election.cc:304] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:28.724197 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:28.724823 17688 raft_consensus.cc:2468] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 20.
I20260709 12:24:28.735276 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 26 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:28.735373 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:28.736135 17686 raft_consensus.cc:2410] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for term 26 because replica has last-logged OpId of term: 25 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 14.
I20260709 12:24:28.703722 25915 raft_consensus.cc:3060] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:28.737298 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 24 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:28.736306 17680 raft_consensus.cc:2468] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 25.
I20260709 12:24:28.709640 26120 raft_consensus.cc:3055] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:24:28.743001 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 21 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:28.737905 17682 raft_consensus.cc:2468] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 23.
I20260709 12:24:28.744566 17681 raft_consensus.cc:2468] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 26.
I20260709 12:24:28.686735 25866 raft_consensus.cc:515] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:28.738535 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 167557us
I20260709 12:24:28.740659 17652 leader_election.cc:304] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:28.746583 17729 leader_election.cc:304] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:28.716281 26323 raft_consensus.cc:2804] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:24:28.750300 18334 leader_election.cc:304] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:28.751827 18334 leader_election.cc:304] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:28.742125 26120 raft_consensus.cc:740] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 24 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:24:28.723754 26322 raft_consensus.cc:493] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:28.761684 17652 leader_election.cc:304] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:28.798802 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:28.805305 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:28.744199 26164 raft_consensus.cc:697] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 24 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:24:28.801796 17681 raft_consensus.cc:1240] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 20. Current term is 21. Ops: [20.15-20.15]
I20260709 12:24:28.764556 26114 consensus_queue.cc:1048] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.057s
I20260709 12:24:28.820976 26330 raft_consensus.cc:2749] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:24:28.822180 17833 raft_consensus.cc:2368] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for earlier term 21. Current term is 22.
W20260709 12:24:28.834164 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 72572.1us
I20260709 12:24:28.766404 25917 raft_consensus.cc:2749] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 12:24:28.713673 26334 consensus_queue.cc:1048] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.090s
I20260709 12:24:28.749254 25761 raft_consensus.cc:2749] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:24:28.850853 26287 raft_consensus.cc:493] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:28.762178 26323 raft_consensus.cc:493] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:28.853188 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:28.862360 26300 raft_consensus.cc:515] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:28.862989 26263 leader_election.cc:290] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:28.737216 26260 consensus_queue.cc:237] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.12, Last appended by leader: 12, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:28.863755 26218 raft_consensus.cc:515] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:28.865314 25688 raft_consensus.cc:515] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:28.886581 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 136564us
I20260709 12:24:28.809309 25951 raft_consensus.cc:2804] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:24:28.868501 26285 raft_consensus.cc:2804] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:24:28.871352 18334 leader_election.cc:304] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:28.871520 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:28.875247 17681 raft_consensus.cc:1240] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 26. Current term is 27. Ops: []
I20260709 12:24:28.921229 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:28.881168 17654 leader_election.cc:304] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:28.845260 26175 raft_consensus.cc:2804] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:24:28.892191 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:28.892488 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 24 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:28.898571 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:28.737388 26336 consensus_queue.cc:1059] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 25.14, Next index: 15, Last known committed idx: 13, Time since last communication: 0.152s
I20260709 12:24:28.910502 17685 raft_consensus.cc:1275] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 17 index: 11. Preceding OpId from leader: term: 22 index: 12. (index mismatch)
I20260709 12:24:28.914242 17682 raft_consensus.cc:1240] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 21 LEADER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 20. Current term is 21. Ops: []
I20260709 12:24:28.928040 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:24:28.931314 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 50237.3us
I20260709 12:24:28.932963 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:28.933601 17654 leader_election.cc:304] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:28.933679 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 24 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:28.934126 26310 raft_consensus.cc:2749] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:24:28.934487 17763 raft_consensus.cc:2468] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 23.
I20260709 12:24:28.934644 25826 consensus_queue.cc:1059] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 20.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20260709 12:24:28.936257 17756 raft_consensus.cc:1275] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 25 index: 17. Preceding OpId from leader: term: 26 index: 18. (index mismatch)
I20260709 12:24:28.938764 17836 raft_consensus.cc:1240] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 24. Current term is 25. Ops: []
I20260709 12:24:28.939674 18368 raft_consensus.cc:1240] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 20. Current term is 21. Ops: [20.16-20.16]
I20260709 12:24:28.936268 17761 raft_consensus.cc:2393] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 24: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:24:28.941334 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 23 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:28.941501 26175 raft_consensus.cc:697] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 23 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:24:28.941987 17762 raft_consensus.cc:2468] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 23.
I20260709 12:24:28.942392 26175 consensus_queue.cc:237] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 20.12, Last appended by leader: 12, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:28.948239 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:28.920812 26285 raft_consensus.cc:493] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:28.951915 26285 raft_consensus.cc:3060] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:28.957332 25688 leader_election.cc:290] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 25 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:24:28.960173 26281 consensus_meta.cc:287] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.508s	user 0.005s	sys 0.004s
I20260709 12:24:28.960362 26263 consensus_queue.cc:1059] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 26.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:24:28.960680 26281 raft_consensus.cc:515] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:28.961786 26164 consensus_queue.cc:237] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 23.16, Last appended by leader: 16, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:28.962832 26281 leader_election.cc:290] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 26 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:28.881000 26208 raft_consensus.cc:515] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:28.965224 26333 log.cc:927] Time spent T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.218s	user 0.000s	sys 0.001s
I20260709 12:24:28.966598 26208 leader_election.cc:290] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 22 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:28.884706 26323 raft_consensus.cc:3060] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:28.819790 26322 raft_consensus.cc:515] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:28.969956 17682 raft_consensus.cc:1240] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 25. Current term is 26. Ops: []
I20260709 12:24:28.970149 18374 raft_consensus.cc:1240] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 21. Current term is 22. Ops: []
I20260709 12:24:28.974542 17681 raft_consensus.cc:1240] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 25. Current term is 26. Ops: [25.15-25.15]
W20260709 12:24:28.974472 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 103205us
I20260709 12:24:28.980751 18334 leader_election.cc:304] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:28.935436 26310 raft_consensus.cc:2749] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:24:28.995837 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 28 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:28.930924 25951 raft_consensus.cc:697] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 22 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:24:28.930677 26289 leader_election.cc:290] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 20 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:28.997049 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:28.998203 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:29.003698 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 28 candidate_status { last_received { term: 27 index: 21 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:28.890489 26287 raft_consensus.cc:515] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:29.015264 17680 raft_consensus.cc:1275] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 22 index: 13. Preceding OpId from leader: term: 26 index: 15. (index mismatch)
I20260709 12:24:28.973928 26340 raft_consensus.cc:493] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:29.014941 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:28.976598 26218 leader_election.cc:290] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 21 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:28.991937 26208 raft_consensus.cc:493] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:28.991582 26339 raft_consensus.cc:2749] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
W20260709 12:24:29.085134 26097 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.530s	user 0.012s	sys 0.009s
I20260709 12:24:28.999768 25917 raft_consensus.cc:2749] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260709 12:24:29.112879 26097 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.558s	user 0.013s	sys 0.010s
I20260709 12:24:29.110839 25881 leader_election.cc:290] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 27 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:29.160522 18363 raft_consensus.cc:1240] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 21. Current term is 22. Ops: []
W20260709 12:24:29.165830 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 232286us
I20260709 12:24:29.168730 17654 leader_election.cc:304] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:29.121922 26208 raft_consensus.cc:515] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:29.171900 26112 raft_consensus.cc:2749] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:24:29.173043 26208 leader_election.cc:290] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:29.150413 26350 raft_consensus.cc:493] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:29.173909 26350 raft_consensus.cc:515] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:29.111487 26340 raft_consensus.cc:515] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:29.255800 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 50380.7us
I20260709 12:24:29.177224 25866 leader_election.cc:290] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 22 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:29.176937 26112 consensus_queue.cc:1048] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.082s
I20260709 12:24:29.290814 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 22 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:29.291937 18333 leader_election.cc:304] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:24:29.262253 26321 log.cc:927] Time spent T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.073s	user 0.000s	sys 0.000s
I20260709 12:24:29.293215 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 29 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:29.177884 26163 raft_consensus.cc:515] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:29.302958 18333 leader_election.cc:304] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:29.124214 26218 consensus_queue.cc:1048] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.174s
W20260709 12:24:29.320405 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 429211us
I20260709 12:24:29.321194 17728 leader_election.cc:304] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:29.321599 16501 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:24:29.317503 18333 leader_election.cc:304] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:29.258772 26267 consensus_queue.cc:1059] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 17, Last known committed idx: 14, Time since last communication: 0.048s
I20260709 12:24:29.255036 26097 consensus_queue.cc:1048] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.047s
I20260709 12:24:29.280390 26350 leader_election.cc:290] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:29.315762 25917 raft_consensus.cc:2749] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:24:29.331581 17728 leader_election.cc:304] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:29.177515 25951 consensus_queue.cc:237] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 20.14, Last appended by leader: 14, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:29.311494 26208 raft_consensus.cc:2749] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
W20260709 12:24:29.348534 26352 log.cc:927] Time spent T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.090s	user 0.000s	sys 0.000s
I20260709 12:24:29.349647 26332 raft_consensus.cc:3055] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:24:29.350086 26332 raft_consensus.cc:740] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:24:29.350955 26332 consensus_queue.cc:260] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 20.14, Last appended by leader: 14, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:29.352859 26351 log.cc:927] Time spent T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.175s	user 0.000s	sys 0.001s
W20260709 12:24:29.354636 26295 consensus_meta.cc:287] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.635s	user 0.010s	sys 0.000s
I20260709 12:24:29.304318 26330 consensus_queue.cc:1048] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.034s
I20260709 12:24:29.355046 26295 raft_consensus.cc:515] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:29.333761 26281 raft_consensus.cc:493] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:29.356235 26281 raft_consensus.cc:515] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:29.358331 26281 leader_election.cc:290] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:29.363251 26287 leader_election.cc:290] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:29.372020 17764 raft_consensus.cc:1275] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 17 index: 11. Preceding OpId from leader: term: 22 index: 12. (index mismatch)
I20260709 12:24:29.373539 17686 raft_consensus.cc:2393] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 17: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
W20260709 12:24:29.340240 26149 consensus_meta.cc:287] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.727s	user 0.008s	sys 0.003s
I20260709 12:24:29.336844 26206 raft_consensus.cc:2749] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:24:29.378615 17728 leader_election.cc:304] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:29.376852 26346 raft_consensus.cc:2749] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:24:29.386440 26009 leader_election.cc:290] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 19 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:24:29.386817 26155 consensus_meta.cc:287] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.677s	user 0.007s	sys 0.004s
W20260709 12:24:29.389343 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 71913.4us
I20260709 12:24:29.376060 26149 raft_consensus.cc:515] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:29.383358 25915 consensus_meta.cc:287] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.623s	user 0.014s	sys 0.005s
I20260709 12:24:29.395426 18333 leader_election.cc:304] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:24:29.396234 26208 raft_consensus.cc:2804] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:24:29.396615 26208 raft_consensus.cc:493] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:29.396965 26208 raft_consensus.cc:3060] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:29.398677 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:29.404033 18333 leader_election.cc:304] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:29.404850 26218 raft_consensus.cc:2749] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:24:29.407696 18333 leader_election.cc:304] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:29.415088 18333 leader_election.cc:304] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:29.416308 17728 leader_election.cc:304] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:29.418915 26340 leader_election.cc:290] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:29.393463 26359 raft_consensus.cc:2749] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:24:29.426198 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:29.420451 26354 raft_consensus.cc:493] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:29.428807 17681 raft_consensus.cc:2393] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 26: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:24:29.431546 18333 leader_election.cc:304] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:29.428325 17728 leader_election.cc:304] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:29.437512 17728 leader_election.cc:304] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:24:29.431828 25917 consensus_queue.cc:1048] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.006s
I20260709 12:24:29.420579 26218 raft_consensus.cc:2749] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:24:29.441051 26350 raft_consensus.cc:2804] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:24:29.432420 26149 leader_election.cc:290] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 24 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:29.441473 26350 raft_consensus.cc:493] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:29.441848 26350 raft_consensus.cc:3060] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:29.407069 25915 raft_consensus.cc:515] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Starting leader election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:29.430325 26340 raft_consensus.cc:2749] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260709 12:24:29.427480 26355 raft_consensus.cc:493] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:29.423480 26206 raft_consensus.cc:3060] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:29.452739 18333 leader_election.cc:304] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:29.453428 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:29.455991 18374 raft_consensus.cc:2393] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 19: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:24:29.402787 26155 raft_consensus.cc:515] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Starting leader election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:29.457796 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 28 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:29.460816 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 22 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:29.462044 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:29.462167 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:29.457841 26359 raft_consensus.cc:2749] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:24:29.463728 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:29.464457 17759 raft_consensus.cc:2393] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 26: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:24:29.465083 26218 consensus_queue.cc:1048] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 12:24:29.467526 26163 leader_election.cc:290] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 24 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:24:29.468245 26285 consensus_meta.cc:287] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.516s	user 0.010s	sys 0.000s
I20260709 12:24:29.468525 26295 leader_election.cc:290] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 17 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:29.468712 26285 raft_consensus.cc:515] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:29.433300 26234 consensus_queue.cc:260] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 26.14, Last appended by leader: 14, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:29.432835 26354 raft_consensus.cc:515] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:29.469698 26234 raft_consensus.cc:3060] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:29.470783 26285 leader_election.cc:290] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 25 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:29.471388 26354 leader_election.cc:290] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:29.471793 26346 raft_consensus.cc:2749] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:24:29.472533 26346 raft_consensus.cc:2804] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:24:29.473016 26346 raft_consensus.cc:697] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
W20260709 12:24:29.473279 26239 consensus_meta.cc:287] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.989s	user 0.011s	sys 0.000s
I20260709 12:24:29.473856 26346 consensus_queue.cc:237] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 25.14, Last appended by leader: 14, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:29.477227 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 22 candidate_status { last_received { term: 19 index: 17 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
W20260709 12:24:29.477476 26357 log.cc:927] Time spent T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.219s	user 0.002s	sys 0.000s
I20260709 12:24:29.478101 26155 leader_election.cc:290] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 24 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:29.478425 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:29.478977 17835 raft_consensus.cc:2393] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 24: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:24:29.458710 26355 raft_consensus.cc:515] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:29.461359 26340 raft_consensus.cc:2749] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:24:29.480584 26323 raft_consensus.cc:515] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:29.481185 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:29.473855 26239 raft_consensus.cc:515] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:29.488471 18374 raft_consensus.cc:1275] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 21 index: 16. Preceding OpId from leader: term: 23 index: 17. (index mismatch)
I20260709 12:24:29.451558 25915 leader_election.cc:290] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 24 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:29.494400 26368 consensus_queue.cc:1059] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 26.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:24:29.496271 18366 raft_consensus.cc:1275] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 20 index: 14. Preceding OpId from leader: term: 21 index: 15. (index mismatch)
I20260709 12:24:29.501194 18367 raft_consensus.cc:1240] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 21. Current term is 22. Ops: []
I20260709 12:24:29.496883 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:29.498522 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:29.499028 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 21 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:29.509976 17758 raft_consensus.cc:2410] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 28 because replica has last-logged OpId of term: 25 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 16.
W20260709 12:24:29.503948 26345 log.cc:927] Time spent T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.563s	user 0.000s	sys 0.002s
I20260709 12:24:29.513751 17833 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35040 (request call id 3352) took 1092 ms. Trace:
I20260709 12:24:29.514209 17833 rpcz_store.cc:276] 0709 12:24:28.421238 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:28.424984 (+  3746us) service_pool.cc:224] Handling call
0709 12:24:29.513567 (+1088583us) spinlock_profiling.cc:243] Waited 369 ms on lock 0x7b54000aaad4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:24:29.513694 (+   127us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":847769344}
I20260709 12:24:29.504415 26368 raft_consensus.cc:3055] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:24:29.482365 25951 consensus_queue.cc:1059] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.028s
I20260709 12:24:29.518038 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:29.500892 26323 leader_election.cc:290] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 22 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:29.523145 17835 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35042 (request call id 3115) took 1150 ms. Trace:
I20260709 12:24:29.513871 26355 leader_election.cc:290] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:29.524998 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 28 candidate_status { last_received { term: 27 index: 21 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:29.523581 17835 rpcz_store.cc:276] 0709 12:24:28.373263 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:28.463053 (+ 89790us) service_pool.cc:224] Handling call
0709 12:24:29.522918 (+1059865us) spinlock_profiling.cc:243] Waited 955 ms on lock 0x7b540008a7d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:24:29.523077 (+   159us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2196724480}
I20260709 12:24:29.525014 17763 raft_consensus.cc:1240] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 25. Current term is 26. Ops: [25.18-25.18]
I20260709 12:24:29.519740 26368 raft_consensus.cc:740] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:24:29.531311 26355 raft_consensus.cc:493] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:24:29.533253 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:29.509686 26175 consensus_queue.cc:1048] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.017s
I20260709 12:24:29.536525 18362 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95"
dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
mode: GRACEFUL
new_leader_uuid: "e063541542a94969898868bf6eb5ce1b"
 from {username='slave'} at 127.0.0.1:50468
I20260709 12:24:29.535261 26355 raft_consensus.cc:515] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:29.537679 18362 raft_consensus.cc:606] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Received request to transfer leadership to TS e063541542a94969898868bf6eb5ce1b
I20260709 12:24:29.538050 18362 raft_consensus.cc:614] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:24:29.540879 17728 leader_election.cc:304] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:29.544755 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 28 candidate_status { last_received { term: 27 index: 21 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:29.546051 26283 raft_consensus.cc:493] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:29.546715 26283 raft_consensus.cc:515] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:29.549013 26283 leader_election.cc:290] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:29.548926 26350 raft_consensus.cc:515] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Starting leader election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:29.550465 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 28 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:29.551173 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:29.551638 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 24 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:29.552212 17682 raft_consensus.cc:3060] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:29.544656 26263 raft_consensus.cc:2749] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:24:29.535917 26285 consensus_queue.cc:1059] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 21.13, Next index: 14, Last known committed idx: 12, Time since last communication: 0.020s
I20260709 12:24:29.555899 26239 leader_election.cc:290] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 19 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:29.540131 26323 raft_consensus.cc:493] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:29.545310 26208 raft_consensus.cc:515] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:29.559715 25967 raft_consensus.cc:3055] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:24:29.560173 25967 raft_consensus.cc:740] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:24:29.561074 25967 consensus_queue.cc:260] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 19.15, Last appended by leader: 15, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:29.562206 25967 raft_consensus.cc:3060] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:29.562780 26206 raft_consensus.cc:2749] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:24:29.563668 17755 raft_consensus.cc:1275] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 24 index: 13. Preceding OpId from leader: term: 25 index: 14. (index mismatch)
I20260709 12:24:29.558637 26323 raft_consensus.cc:515] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:29.564429 26363 raft_consensus.cc:3055] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:24:29.564980 26363 raft_consensus.cc:740] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 21 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:24:29.566051 26363 consensus_queue.cc:260] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 21.13, Last appended by leader: 13, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:29.567579 26363 raft_consensus.cc:3060] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:29.570870 25915 raft_consensus.cc:493] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:29.571702 25915 raft_consensus.cc:515] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:29.575105 17801 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:29.525121 26270 consensus_queue.cc:1048] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.015s
I20260709 12:24:29.554178 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 29 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:29.585897 17681 raft_consensus.cc:2468] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 28.
I20260709 12:24:29.589816 17728 leader_election.cc:304] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:29.601022 26285 raft_consensus.cc:493] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:29.599610 26322 leader_election.cc:290] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:29.603462 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:29.606868 17682 raft_consensus.cc:2468] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 24.
I20260709 12:24:29.602816 26285 raft_consensus.cc:515] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:29.608345 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 25 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:29.612357 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:29.613719 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:29.615015 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 24 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
W20260709 12:24:29.617652 26341 log.cc:927] Time spent T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.653s	user 0.003s	sys 0.000s
I20260709 12:24:29.620157 17684 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37396 (request call id 3122) took 1051 ms. Trace:
I20260709 12:24:29.620535 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 24 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:29.621325 17680 raft_consensus.cc:2393] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 24: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:24:29.620481 17684 rpcz_store.cc:276] 0709 12:24:28.569161 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:28.580238 (+ 11077us) service_pool.cc:224] Handling call
0709 12:24:28.580538 (+   300us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:28.590079 (+  9541us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:28.590114 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:28.594953 (+  4839us) log.cc:844] Serialized 26 byte log entry
0709 12:24:28.692551 (+ 97598us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:24:28.692602 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:28.692686 (+    84us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:29.616938 (+924252us) spinlock_profiling.cc:243] Waited 12.2 ms on lock 0x7b4400022b4c. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b47459c85 00007f7b4745a6a2 00007f7b4a1cc1db 00007f7b4a1c7aaf 00007f7b4a1bb86e 00007f7b4a1babb6 00007f7b4a1ba7d8 00007f7b4a1c4221 00007f7b4a1c41d9 00007f7b4a1c4169
0709 12:24:29.619915 (+  2977us) raft_consensus.cc:1685] finished
0709 12:24:29.619935 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:29.620109 (+   174us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":909,"raft.run_cpu_time_us":168,"raft.run_wall_time_us":2865,"spinlock_wait_cycles":28160256,"thread_start_us":97212,"threads_started":1,"wal-append.queue_time_us":369173}
I20260709 12:24:29.621658 26285 leader_election.cc:290] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:29.622268 17687 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37396 (request call id 3125) took 1003 ms. Trace:
I20260709 12:24:29.623196 26323 leader_election.cc:290] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:29.623628 26294 raft_consensus.cc:3055] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:24:29.602285 26206 raft_consensus.cc:2804] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:24:29.624135 26294 raft_consensus.cc:740] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:24:29.624267 26206 raft_consensus.cc:493] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:29.624877 26206 raft_consensus.cc:3060] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:24:29.627020 17763 raft_consensus.cc:2393] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 24: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:24:29.622704 17687 rpcz_store.cc:276] 0709 12:24:28.618640 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:28.618784 (+   144us) service_pool.cc:224] Handling call
0709 12:24:28.625329 (+  6545us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:28.711158 (+ 85829us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:29.622053 (+910895us) spinlock_profiling.cc:243] Waited 82.8 ms on lock 0x7b54003507d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:24:29.622209 (+   156us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":190545792}
I20260709 12:24:29.648252 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:29.632608 26355 leader_election.cc:290] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:24:29.651504 26261 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.940s	user 0.008s	sys 0.016s
W20260709 12:24:29.655045 26261 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.946s	user 0.008s	sys 0.017s
I20260709 12:24:29.655334 26261 raft_consensus.cc:697] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:24:29.656780 26261 consensus_queue.cc:237] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 21.16, Last appended by leader: 16, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:29.668219 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 24 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:29.668874 17763 raft_consensus.cc:3060] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:29.680835 17763 raft_consensus.cc:2468] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 24.
W20260709 12:24:29.685595 26366 log.cc:927] Time spent T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.281s	user 0.001s	sys 0.000s
I20260709 12:24:29.691036 26206 raft_consensus.cc:515] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:29.695804 18364 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d"
dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
mode: GRACEFUL
new_leader_uuid: "e063541542a94969898868bf6eb5ce1b"
 from {username='slave'} at 127.0.0.1:50468
I20260709 12:24:29.696620 18364 raft_consensus.cc:606] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Received request to transfer leadership to TS e063541542a94969898868bf6eb5ce1b
I20260709 12:24:29.697037 18364 raft_consensus.cc:614] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:24:29.699489 26206 leader_election.cc:290] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 29 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:29.645309 26368 consensus_queue.cc:260] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 26.15, Last appended by leader: 15, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:29.705360 26368 raft_consensus.cc:3060] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:24:29.713812 17729 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:29.714907 17729 leader_election.cc:304] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:29.721258 17834 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35042 (request call id 3114) took 1352 ms. Trace:
I20260709 12:24:29.721865 17834 rpcz_store.cc:276] 0709 12:24:28.368723 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:28.580739 (+212016us) service_pool.cc:224] Handling call
0709 12:24:28.581028 (+   289us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:28.662532 (+ 81504us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:29.720885 (+1058353us) spinlock_profiling.cc:243] Waited 80 ms on lock 0x7b5400159ad4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:24:29.721169 (+   284us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":184114048}
I20260709 12:24:29.726457 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:29.726864 17729 leader_election.cc:304] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:29.733809 26285 raft_consensus.cc:2749] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
W20260709 12:24:29.737336 17727 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:29.745685 17834 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35042 (request call id 3123) took 1281 ms. Trace:
I20260709 12:24:29.746838 17834 rpcz_store.cc:276] 0709 12:24:28.447035 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:29.727271 (+1280236us) service_pool.cc:224] Handling call
0709 12:24:29.727662 (+   391us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:29.727913 (+   251us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:29.727950 (+    37us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:29.728009 (+    59us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:24:29.728053 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:29.728122 (+    69us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:29.728292 (+   170us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:29.771173 17760 raft_consensus.cc:1240] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 28. Current term is 29. Ops: [26.18-28.19]
I20260709 12:24:29.772666 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:29.777884 26346 raft_consensus.cc:2749] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:24:29.778983 17682 raft_consensus.cc:1240] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 26. Current term is 27. Ops: []
I20260709 12:24:29.788587 26261 raft_consensus.cc:493] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:24:29.790611 17836 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35042 (request call id 3108) took 1457 ms. Trace:
I20260709 12:24:29.790980 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:29.791733 18364 raft_consensus.cc:2468] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 24.
I20260709 12:24:29.792387 26324 raft_consensus.cc:3055] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:24:29.792590 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:29.792809 26324 raft_consensus.cc:740] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:24:29.793637 26324 consensus_queue.cc:260] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 21.18, Last appended by leader: 18, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:29.794790 26324 raft_consensus.cc:3060] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Advancing to term 22
W20260709 12:24:29.788820 26383 log.cc:927] Time spent T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.059s	user 0.001s	sys 0.000s
I20260709 12:24:29.791054 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 29 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
W20260709 12:24:29.800426 26362 log.cc:927] Time spent T a5732e71b9d54f8786b8c511d69943bf P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.404s	user 0.000s	sys 0.002s
I20260709 12:24:29.803766 26353 raft_consensus.cc:3055] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:24:29.804337 26353 raft_consensus.cc:740] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:24:29.805318 26353 consensus_queue.cc:260] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 22.17, Last appended by leader: 17, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:29.806849 26353 raft_consensus.cc:3060] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:29.798553 17832 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35040 (request call id 3350) took 1377 ms. Trace:
I20260709 12:24:29.808401 17832 rpcz_store.cc:276] 0709 12:24:28.420698 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:28.582601 (+161903us) service_pool.cc:224] Handling call
0709 12:24:28.582956 (+   355us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:28.723809 (+140853us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:29.798343 (+1074534us) spinlock_profiling.cc:243] Waited 136 ms on lock 0x7b54002eac54. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:24:29.798502 (+   159us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":312194816}
I20260709 12:24:29.810091 17832 raft_consensus.cc:1275] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 20 index: 15. Preceding OpId from leader: term: 21 index: 15. (term mismatch)
I20260709 12:24:29.810415 17832 pending_rounds.cc:85] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b: Aborting all ops after (but not including) 14
I20260709 12:24:29.810638 17832 pending_rounds.cc:107] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b: Aborting uncommitted NO_OP operation due to leader change: 20.15
I20260709 12:24:29.810881 17832 raft_consensus.cc:2889] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:24:29.813103 26348 log.cc:927] Time spent T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.631s	user 0.001s	sys 0.000s
I20260709 12:24:29.798486 17834 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37994 (request call id 1565) took 1096 ms. Trace:
I20260709 12:24:29.790983 17836 rpcz_store.cc:276] 0709 12:24:28.333281 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:28.488097 (+154816us) service_pool.cc:224] Handling call
0709 12:24:28.488519 (+   422us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:28.939156 (+450637us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:29.790401 (+851245us) spinlock_profiling.cc:243] Waited 450 ms on lock 0x7b5400089dd4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:24:29.790559 (+   158us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1035037056}
I20260709 12:24:29.815689 17838 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35040 (request call id 3355) took 1393 ms. Trace:
I20260709 12:24:29.816809 17836 raft_consensus.cc:1240] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 23. Current term is 24. Ops: []
I20260709 12:24:29.816088 17838 rpcz_store.cc:276] 0709 12:24:28.421985 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:28.691612 (+269627us) service_pool.cc:224] Handling call
0709 12:24:28.692012 (+   400us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:28.714985 (+ 22973us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:28.715017 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:28.715460 (+   443us) log.cc:844] Serialized 26 byte log entry
0709 12:24:29.009466 (+294006us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:24:29.009593 (+   127us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:29.009726 (+   133us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:29.812383 (+802657us) spinlock_profiling.cc:243] Waited 2.13 ms on lock 0x7b4400022b4c. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b47459c85 00007f7b4745a6a2 00007f7b4a1cc1db 00007f7b4a1c7aaf 00007f7b4a1bb86e 00007f7b4a1babb6 00007f7b4a1ba7d8 00007f7b4a1c4221 00007f7b4a1c41d9 00007f7b4a1c4169
0709 12:24:29.815392 (+  3009us) raft_consensus.cc:1685] finished
0709 12:24:29.815414 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:29.815631 (+   217us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":22572,"spinlock_wait_cycles":4900096,"thread_start_us":202806,"threads_started":1,"wal-append.queue_time_us":410222}
I20260709 12:24:29.820046 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:29.821137 17764 raft_consensus.cc:2393] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 22: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:24:29.811743 17832 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37994 (request call id 1566) took 1109 ms. Trace:
I20260709 12:24:29.823741 17832 rpcz_store.cc:276] 0709 12:24:28.702668 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:29.809530 (+1106862us) service_pool.cc:224] Handling call
0709 12:24:29.809803 (+   273us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:29.811469 (+  1666us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:29.811695 (+   226us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:29.824638 17688 raft_consensus.cc:1240] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 16. Current term is 17. Ops: []
I20260709 12:24:29.826709 18368 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 2439) took 1335 ms. Trace:
I20260709 12:24:29.827066 18368 rpcz_store.cc:276] 0709 12:24:28.491514 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:28.491670 (+   156us) service_pool.cc:224] Handling call
0709 12:24:28.492013 (+   343us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:28.940084 (+448071us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:29.826409 (+886325us) spinlock_profiling.cc:243] Waited 447 ms on lock 0x7b54006af9d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:24:29.826650 (+   241us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1028989824}
I20260709 12:24:29.828480 17683 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37402 (request call id 3543) took 1118 ms. Trace:
I20260709 12:24:29.828836 17683 rpcz_store.cc:276] 0709 12:24:28.710477 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:28.710637 (+   160us) service_pool.cc:224] Handling call
0709 12:24:28.710892 (+   255us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:28.744969 (+ 34077us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:28.745000 (+    31us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:28.745041 (+    41us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:24:28.745072 (+    31us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:29.828253 (+1083181us) spinlock_profiling.cc:243] Waited 33.9 ms on lock 0x7b5400054bd4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:24:29.828296 (+    43us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:29.828400 (+   104us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":77972352}
I20260709 12:24:29.808470 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:29.837703 18333 leader_election.cc:304] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:29.789245 26261 raft_consensus.cc:515] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:29.843585 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 24 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:29.836136 17756 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 22.
I20260709 12:24:29.845705 18363 raft_consensus.cc:1275] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 19 index: 14. Preceding OpId from leader: term: 21 index: 15. (index mismatch)
I20260709 12:24:29.846890 17755 raft_consensus.cc:2393] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 24: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:24:29.813712 17834 rpcz_store.cc:276] 0709 12:24:28.702390 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:29.770117 (+1067727us) service_pool.cc:224] Handling call
0709 12:24:29.797988 (+ 27871us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:29.798174 (+   186us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:29.798202 (+    28us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:29.798248 (+    46us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:24:29.798280 (+    32us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:29.798338 (+    58us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:29.798436 (+    98us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:29.769145 26285 raft_consensus.cc:493] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:29.852478 17729 leader_election.cc:304] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:29.774387 26206 raft_consensus.cc:493] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:29.850013 26323 raft_consensus.cc:3060] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:29.810961 26120 consensus_queue.cc:260] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 24.15, Last appended by leader: 15, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:29.775398 26340 consensus_queue.cc:1059] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 25.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.086s
I20260709 12:24:29.920331 17652 leader_election.cc:304] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
W20260709 12:24:29.912017 17727 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:29.935802 25384 raft_consensus.cc:2804] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:24:29.936717 25384 raft_consensus.cc:697] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 27 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:24:29.864805 26355 raft_consensus.cc:2804] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:24:29.912237 26354 consensus_queue.cc:1059] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 20.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.023s
I20260709 12:24:29.871692 26300 leader_election.cc:290] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 22 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:24:29.885840 17728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:29.921986 26330 raft_consensus.cc:493] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:24:29.864552 26283 consensus_queue.cc:1059] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 21.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.078s
W20260709 12:24:29.938304 26373 log.cc:927] Time spent T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.071s	user 0.000s	sys 0.000s
I20260709 12:24:29.869225 26206 raft_consensus.cc:515] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:29.995982 26354 consensus_queue.cc:1243] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Peer 0da169ff82674dc1947055ce43de68a3 log is divergent from this leader: its last log entry 21.15 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 14
I20260709 12:24:29.996780 26206 leader_election.cc:290] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:29.954532 26270 consensus_queue.cc:1048] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.043s
I20260709 12:24:29.861629 26285 raft_consensus.cc:515] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:30.000669 26285 leader_election.cc:290] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:24:29.951978 26260 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.985s	user 0.024s	sys 0.021s
W20260709 12:24:30.002243 26260 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.047s	user 0.026s	sys 0.021s
I20260709 12:24:30.004753 17652 leader_election.cc:400] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 21 pre-election: Vote denied by peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837) with higher term. Message: Invalid argument: T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for earlier term 21. Current term is 22.
I20260709 12:24:30.005177 17652 leader_election.cc:403] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 21 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:24:30.006265 25761 raft_consensus.cc:3060] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Advancing to term 22
I20260709 12:24:30.008754 17652 leader_election.cc:304] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:30.014117 26261 leader_election.cc:290] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:30.015863 26295 consensus_queue.cc:1048] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:24:29.988137 26363 raft_consensus.cc:3055] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:24:30.017424 26363 raft_consensus.cc:740] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:24:30.018707 26363 consensus_queue.cc:260] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 21.16, Last appended by leader: 16, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:29.939638 26355 raft_consensus.cc:493] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:30.020116 26355 raft_consensus.cc:3060] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:29.979422 26330 raft_consensus.cc:515] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:30.022415 26330 leader_election.cc:290] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:30.025682 17833 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35042 (request call id 3121) took 1579 ms. Trace:
I20260709 12:24:30.026361 17759 raft_consensus.cc:1240] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 LEADER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 22. Current term is 23. Ops: []
W20260709 12:24:30.027010 17801 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:30.027966 25866 raft_consensus.cc:2804] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:24:30.028431 25866 raft_consensus.cc:493] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:30.028946 25866 raft_consensus.cc:3060] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:30.030355 17801 leader_election.cc:304] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:24:30.032665 26376 log.cc:927] Time spent T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.077s	user 0.000s	sys 0.000s
I20260709 12:24:30.015909 25881 raft_consensus.cc:2804] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:24:29.992180 25384 consensus_queue.cc:237] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 26.15, Last appended by leader: 15, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:30.063838 17835 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35042 (request call id 3122) took 1617 ms. Trace:
W20260709 12:24:30.057960 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 49302.3us
I20260709 12:24:30.026091 17833 rpcz_store.cc:276] 0709 12:24:28.446484 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:29.521544 (+1075060us) service_pool.cc:224] Handling call
0709 12:24:29.521829 (+   285us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:29.522028 (+   199us) raft_consensus.cc:1534] Early marking committed up to index 15
0709 12:24:29.522303 (+   275us) log.cc:844] Serialized 14 byte log entry
0709 12:24:30.025334 (+503031us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:30.025386 (+    52us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:24:30.025429 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:30.025509 (+    80us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:30.025623 (+   114us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":491942,"prepare.queue_time_us":149,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":41,"raft.queue_time_us":493209,"raft.run_cpu_time_us":80,"raft.run_wall_time_us":81,"spinlock_wait_cycles":43392,"thread_start_us":10718,"threads_started":1,"wal-append.queue_time_us":23473}
I20260709 12:24:30.019598 26337 raft_consensus.cc:3055] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:24:30.060417 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 22 candidate_status { last_received { term: 19 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:29.939757 26168 consensus_queue.cc:1048] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.092s
I20260709 12:24:30.064272 25637 raft_consensus.cc:493] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:30.075001 26337 raft_consensus.cc:740] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 25 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:24:30.077732 25637 raft_consensus.cc:515] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:30.078459 26337 consensus_queue.cc:260] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 25.14, Last appended by leader: 14, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:30.080202 26338 consensus_queue.cc:1059] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 24.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:24:30.081147 26338 raft_consensus.cc:2804] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:24:30.071107 17830 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35042 (request call id 3098) took 3966 ms. Trace:
I20260709 12:24:30.065878 17837 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35042 (request call id 3120) took 1619 ms. Trace:
I20260709 12:24:30.074313 17838 raft_consensus.cc:1275] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 19 index: 12. Preceding OpId from leader: term: 22 index: 12. (term mismatch)
I20260709 12:24:30.080523 25915 leader_election.cc:290] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:24:30.043323 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 184627us
I20260709 12:24:29.984887 17829 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35040 (request call id 3358) took 1562 ms. Trace:
I20260709 12:24:30.090878 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 21 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:30.067107 17835 rpcz_store.cc:276] 0709 12:24:28.446755 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:29.528110 (+1081355us) service_pool.cc:224] Handling call
0709 12:24:29.534826 (+  6716us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:29.534967 (+   141us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 12:24:29.541611 (+  6644us) log.cc:844] Serialized 14 byte log entry
0709 12:24:30.063364 (+521753us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:30.063436 (+    72us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:24:30.063495 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:30.063582 (+    87us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:30.063759 (+   177us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":416297,"prepare.queue_time_us":5630,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":39,"thread_start_us":105086,"threads_started":1,"wal-append.queue_time_us":139793}
I20260709 12:24:30.094282 17832 raft_consensus.cc:3060] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:30.095844 17758 raft_consensus.cc:1240] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 24. Current term is 25. Ops: []
I20260709 12:24:30.095199 17829 rpcz_store.cc:276] 0709 12:24:28.422705 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:28.691306 (+268601us) service_pool.cc:224] Handling call
0709 12:24:28.691631 (+   325us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:28.691850 (+   219us) raft_consensus.cc:1534] Early marking committed up to index 18
0709 12:24:28.692173 (+   323us) log.cc:844] Serialized 14 byte log entry
0709 12:24:29.918207 (+1226034us) spinlock_profiling.cc:243] Waited 11.2 ms on lock 0x7b18004ce700. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b474adc0a 00007f7b50bed219 00007f7b47535a40 00007f7b475304e4 00007f7b47530455 00007f7b4a618060 00007f7b4a62a716 00007f7b4a63b287 00007f7b4a63b208
0709 12:24:29.980572 (+ 62365us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:29.984596 (+  4024us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:24:29.984644 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:29.984715 (+    71us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:29.984831 (+   116us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":110865,"prepare.queue_time_us":6799,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":33,"raft.queue_time_us":1263586,"raft.run_cpu_time_us":176,"raft.run_wall_time_us":7225,"spinlock_wait_cycles":25802624,"thread_start_us":62057,"threads_started":1}
W20260709 12:24:30.100004 17728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:30.084601 17838 pending_rounds.cc:85] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b: Aborting all ops after (but not including) 11
I20260709 12:24:30.102006 17838 pending_rounds.cc:107] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b: Aborting uncommitted NO_OP operation due to leader change: 19.12
I20260709 12:24:30.102347 17838 raft_consensus.cc:2889] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:30.098031 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 27 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:30.079722 26337 raft_consensus.cc:3060] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:30.110121 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 22 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:30.110939 17829 raft_consensus.cc:2468] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 21.
I20260709 12:24:30.070891 26270 raft_consensus.cc:2749] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:24:30.081704 26363 raft_consensus.cc:3060] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:30.070732 25688 raft_consensus.cc:493] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:30.075138 26300 raft_consensus.cc:493] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:30.251822 26277 raft_consensus.cc:493] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
W20260709 12:24:30.196390 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 83910.3us
I20260709 12:24:30.084867 26330 raft_consensus.cc:493] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:30.197618 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 22 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:30.198194 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 24 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
W20260709 12:24:30.198413 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 88646.1us
I20260709 12:24:30.082795 17830 rpcz_store.cc:276] 0709 12:24:26.098203 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:26.179543 (+ 81340us) service_pool.cc:224] Handling call
0709 12:24:26.205052 (+ 25509us) raft_consensus.cc:1471] Updating replica for 4 ops
0709 12:24:26.205467 (+   415us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:26.205501 (+    34us) raft_consensus.cc:1539] Triggering prepare for 4 ops
0709 12:24:26.206336 (+   835us) log.cc:844] Serialized 104 byte log entry
0709 12:24:26.521893 (+315557us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:24:27.223107 (+701214us) log.cc:844] Serialized 14 byte log entry
0709 12:24:27.389713 (+166606us) log.cc:844] Serialized 14 byte log entry
0709 12:24:28.442369 (+1052656us) spinlock_profiling.cc:243] Waited 7.27 ms on lock 0x7b1800418240. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b47515e07 00007f7b503709a9 00007f7b4753269b 00007f7b4753093f 00007f7b47530455 00007f7b4a618060 00007f7b4a62a716 00007f7b4a63b287 00007f7b4a63b208
0709 12:24:28.616832 (+174463us) log.cc:844] Serialized 14 byte log entry
0709 12:24:29.618379 (+1001547us) spinlock_profiling.cc:243] Waited 3.4 ms on lock 0x7b1800418240. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b47515e07 00007f7b503709a9 00007f7b4753269b 00007f7b4753093f 00007f7b47532c02 00007f7b4a1ba20b 00007f7b4a1bfd23 00007f7b4a1c023a 00007f7b4a62aa2a
0709 12:24:30.064420 (+446041us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:30.064528 (+   108us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:30.064585 (+    57us) raft_consensus.cc:1685] finished
0709 12:24:30.064595 (+    10us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:30.064716 (+   121us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":533503,"prepare.queue_time_us":556512,"prepare.run_cpu_time_us":122,"prepare.run_wall_time_us":124,"raft.queue_time_us":611044,"raft.run_cpu_time_us":448,"raft.run_wall_time_us":29744,"spinlock_wait_cycles":24547456,"thread_start_us":1354607,"threads_started":5,"wal-append.queue_time_us":1782034,"wal-append.run_cpu_time_us":5944,"wal-append.run_wall_time_us":1692800}
I20260709 12:24:30.271412 17838 raft_consensus.cc:2393] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 24: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
W20260709 12:24:30.272172 17729 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:30.106246 26239 raft_consensus.cc:3055] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 25 LEADER]: Stepping down as leader of term 25
W20260709 12:24:30.200605 17727 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:30.198948 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 1.48802e+06us
I20260709 12:24:30.275447 17803 leader_election.cc:304] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:30.200634 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:30.276886 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:30.204334 17757 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38786 (request call id 3814) took 1738 ms. Trace:
I20260709 12:24:30.277578 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:30.278007 17757 rpcz_store.cc:276] 0709 12:24:28.466401 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:28.466648 (+   247us) service_pool.cc:224] Handling call
0709 12:24:28.490072 (+ 23424us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:28.693164 (+203092us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:28.693193 (+    29us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:28.693235 (+    42us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:24:28.693274 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:30.204052 (+1510778us) spinlock_profiling.cc:243] Waited 203 ms on lock 0x7b54000ff0d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:24:30.204114 (+    62us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:30.204265 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":466659584}
W20260709 12:24:30.215305 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 1.51055e+06us
I20260709 12:24:30.216121 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:30.217913 17835 raft_consensus.cc:1240] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 20. Current term is 21. Ops: []
I20260709 12:24:30.219203 25637 leader_election.cc:290] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:30.103749 26271 consensus_queue.cc:1059] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 24.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.097s
I20260709 12:24:30.198879 26164 raft_consensus.cc:493] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:30.230232 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 27 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:30.116734 26009 consensus_queue.cc:1059] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 23.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.106s
I20260709 12:24:30.123924 26263 consensus_queue.cc:1048] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.104s
I20260709 12:24:30.278981 26294 consensus_queue.cc:260] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 20.17, Last appended by leader: 17, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:30.297920 26294 raft_consensus.cc:3060] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:30.223263 26338 raft_consensus.cc:697] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 24 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:24:30.298764 26283 consensus_queue.cc:1059] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 19.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20260709 12:24:30.301214 17803 leader_election.cc:304] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:30.081588 26302 raft_consensus.cc:493] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
W20260709 12:24:30.162642 17728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:30.266229 17832 raft_consensus.cc:2468] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 21.
I20260709 12:24:30.299321 26338 consensus_queue.cc:237] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 22.18, Last appended by leader: 18, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:30.274119 26239 raft_consensus.cc:740] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 25 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:24:30.315062 26009 raft_consensus.cc:3055] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:24:30.315502 26009 raft_consensus.cc:740] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:24:30.274233 26330 raft_consensus.cc:515] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:30.316395 26009 consensus_queue.cc:260] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 23.13, Last appended by leader: 13, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:30.317561 26330 leader_election.cc:290] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:30.261439 26149 raft_consensus.cc:493] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:30.283273 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 27 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:30.318336 26149 raft_consensus.cc:515] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:30.283756 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:30.320538 26149 leader_election.cc:290] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:30.287626 17652 leader_election.cc:304] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:30.321265 26271 raft_consensus.cc:3055] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:24:30.321676 26271 raft_consensus.cc:740] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:24:30.322190 26260 raft_consensus.cc:2749] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 12:24:30.322445 26271 consensus_queue.cc:260] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 26.16, Last appended by leader: 16, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:30.289466 26389 log.cc:927] Time spent T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.072s	user 0.001s	sys 0.000s
I20260709 12:24:30.326859 26323 raft_consensus.cc:2749] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:24:30.110723 25881 raft_consensus.cc:697] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 27 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:24:30.329288 26350 leader_election.cc:290] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 24 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:30.329181 25881 consensus_queue.cc:237] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 26.14, Last appended by leader: 14, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:30.292343 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 27 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:30.289511 26285 raft_consensus.cc:493] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:30.331276 26285 raft_consensus.cc:515] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:30.334390 26285 leader_election.cc:290] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:30.336905 26208 leader_election.cc:290] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 23 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:24:30.282259 17802 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:30.307322 26396 raft_consensus.cc:2749] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
W20260709 12:24:30.340453 17802 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:30.304165 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 24 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:30.341259 17686 raft_consensus.cc:2393] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 24: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:24:30.341471 17835 raft_consensus.cc:1275] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 24 index: 14. Preceding OpId from leader: term: 25 index: 15. (index mismatch)
I20260709 12:24:30.263633 26322 raft_consensus.cc:493] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:30.310248 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:30.273542 25951 consensus_queue.cc:1059] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 24.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.069s
I20260709 12:24:30.269553 26300 raft_consensus.cc:515] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:30.265681 17764 raft_consensus.cc:1275] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 24 index: 17. Preceding OpId from leader: term: 26 index: 18. (index mismatch)
I20260709 12:24:30.317627 26009 raft_consensus.cc:3060] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:30.344776 26300 leader_election.cc:290] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:30.267665 25688 raft_consensus.cc:515] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:30.346167 26218 raft_consensus.cc:3055] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:24:30.346700 26218 raft_consensus.cc:740] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:24:30.346997 25688 leader_election.cc:290] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:30.348645 17829 raft_consensus.cc:2468] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 21.
I20260709 12:24:30.349323 25761 raft_consensus.cc:2749] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Leader pre-election lost for term 21. Reason: Vote denied by peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837) with higher term. Message: Invalid argument: T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for earlier term 21. Current term is 22.
I20260709 12:24:30.271278 26277 raft_consensus.cc:515] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:30.285957 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:30.351235 17830 raft_consensus.cc:2468] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 23.
I20260709 12:24:30.351657 26277 leader_election.cc:290] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:30.309747 26114 consensus_queue.cc:1059] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 24.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.043s
I20260709 12:24:30.296327 26164 raft_consensus.cc:515] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:30.314806 26239 consensus_queue.cc:260] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 25.18, Last appended by leader: 18, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:30.353267 26354 raft_consensus.cc:3055] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:24:30.353662 26354 raft_consensus.cc:740] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:24:30.353926 26239 raft_consensus.cc:3060] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:30.354746 26164 leader_election.cc:290] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:30.354451 26354 consensus_queue.cc:260] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 24.19, Last appended by leader: 19, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:30.355510 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:30.356170 17838 raft_consensus.cc:2468] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 20.
I20260709 12:24:30.359777 18366 raft_consensus.cc:3060] T a5732e71b9d54f8786b8c511d69943bf P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Advancing to term 25
W20260709 12:24:30.339661 26368 consensus_meta.cc:287] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.634s	user 0.009s	sys 0.000s
I20260709 12:24:30.342214 26330 consensus_queue.cc:1059] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 16.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20260709 12:24:30.365655 17682 raft_consensus.cc:2468] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 22.
I20260709 12:24:30.310956 17832 raft_consensus.cc:1275] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 21 index: 16. Preceding OpId from leader: term: 23 index: 17. (index mismatch)
I20260709 12:24:30.367274 26120 raft_consensus.cc:3060] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:30.371706 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 23 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:30.377660 17728 leader_election.cc:304] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:30.323757 26271 raft_consensus.cc:3060] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:30.384941 17652 leader_election.cc:304] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:30.389355 18362 raft_consensus.cc:1240] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 20. Current term is 21. Ops: []
I20260709 12:24:30.369890 26346 raft_consensus.cc:493] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:30.393894 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:30.395545 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:30.396804 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:30.401492 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 24 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:30.402534 18371 raft_consensus.cc:2393] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 24: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:24:30.343294 26322 raft_consensus.cc:515] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:30.405650 18365 raft_consensus.cc:1240] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 20. Current term is 21. Ops: []
I20260709 12:24:30.331683 26355 raft_consensus.cc:515] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:30.407640 17755 raft_consensus.cc:1240] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 28. Current term is 29. Ops: [26.18-28.19]
I20260709 12:24:30.407994 17756 raft_consensus.cc:2393] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 26: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:24:30.413235 17834 raft_consensus.cc:1240] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 24. Current term is 25. Ops: [24.15-24.15]
I20260709 12:24:30.414597 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:30.415792 17829 raft_consensus.cc:1275] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 19 index: 14. Preceding OpId from leader: term: 21 index: 15. (index mismatch)
I20260709 12:24:30.417970 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:24:30.418721 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 117565us
I20260709 12:24:30.419433 17803 leader_election.cc:304] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:24:30.419519 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 23 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:30.361853 25866 raft_consensus.cc:515] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:30.428465 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 24 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:30.431967 17761 raft_consensus.cc:2368] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 23. Current term is 24.
W20260709 12:24:30.477317 17730 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:30.471063 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 173479us
W20260709 12:24:30.544306 17728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:30.470081 18362 raft_consensus.cc:1275] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 19 index: 15. Preceding OpId from leader: term: 23 index: 17. (index mismatch)
I20260709 12:24:30.471889 25384 raft_consensus.cc:3055] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:24:30.557525 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 24 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:30.558393 17687 raft_consensus.cc:2410] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for term 24 because replica has last-logged OpId of term: 22 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 15.
I20260709 12:24:30.404562 25967 raft_consensus.cc:493] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:30.564975 17686 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119"
dest_uuid: "0da169ff82674dc1947055ce43de68a3"
mode: GRACEFUL
new_leader_uuid: "e063541542a94969898868bf6eb5ce1b"
 from {username='slave'} at 127.0.0.1:54540
W20260709 12:24:30.556046 17730 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:30.470870 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:30.565838 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:30.566426 17832 raft_consensus.cc:2468] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 26.
I20260709 12:24:30.566787 17755 raft_consensus.cc:1275] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 20 index: 14. Preceding OpId from leader: term: 22 index: 15. (index mismatch)
I20260709 12:24:30.568758 17756 raft_consensus.cc:1240] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 24. Current term is 25. Ops: []
I20260709 12:24:30.556360 25384 raft_consensus.cc:740] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
W20260709 12:24:30.569614 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 55118us
I20260709 12:24:30.561743 17838 raft_consensus.cc:1240] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 26. Current term is 27. Ops: []
I20260709 12:24:30.554749 17830 raft_consensus.cc:1240] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 21. Current term is 22. Ops: []
I20260709 12:24:30.409922 26344 raft_consensus.cc:2749] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260709 12:24:30.570670 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 201430us
I20260709 12:24:30.429229 26260 raft_consensus.cc:493] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:30.347838 26218 consensus_queue.cc:260] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 28.19, Last appended by leader: 19, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:30.432598 26353 raft_consensus.cc:493] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:30.428520 25688 raft_consensus.cc:493] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:30.570456 25384 consensus_queue.cc:260] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 24.17, Last appended by leader: 17, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:30.419032 26324 raft_consensus.cc:493] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:30.474177 26149 raft_consensus.cc:2749] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:24:30.467361 26365 raft_consensus.cc:2804] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:24:30.365111 26302 raft_consensus.cc:515] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:30.567274 26300 raft_consensus.cc:2804] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:24:30.407289 25761 consensus_queue.cc:1048] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.158s
I20260709 12:24:30.470520 26277 raft_consensus.cc:493] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:30.607551 18366 raft_consensus.cc:1275] T a5732e71b9d54f8786b8c511d69943bf P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 24 index: 15. Preceding OpId from leader: term: 25 index: 15. (term mismatch)
I20260709 12:24:30.609051 18333 leader_election.cc:304] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:30.467212 26399 raft_consensus.cc:493] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:30.410255 26346 raft_consensus.cc:515] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:30.597234 25967 raft_consensus.cc:515] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:30.564559 25915 raft_consensus.cc:493] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:30.617625 25384 raft_consensus.cc:3060] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:30.630297 18333 leader_election.cc:304] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:24:30.633212 17801 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:30.622571 18366 pending_rounds.cc:85] T a5732e71b9d54f8786b8c511d69943bf P 334a4e1f59944544b3b3ff36616b57da: Aborting all ops after (but not including) 14
I20260709 12:24:30.635843 18366 pending_rounds.cc:107] T a5732e71b9d54f8786b8c511d69943bf P 334a4e1f59944544b3b3ff36616b57da: Aborting uncommitted CHANGE_CONFIG_OP operation due to leader change: 24.15
I20260709 12:24:30.637667 18366 raft_consensus.cc:2917] T a5732e71b9d54f8786b8c511d69943bf P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Aborting config change with OpId 24.15: Aborted: Op aborted by new leader
I20260709 12:24:30.638193 18366 raft_consensus.cc:2889] T a5732e71b9d54f8786b8c511d69943bf P 334a4e1f59944544b3b3ff36616b57da [term 25 LEARNER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:30.476378 25951 consensus_queue.cc:1048] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.137s
I20260709 12:24:30.607896 26397 raft_consensus.cc:493] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:30.681790 25915 raft_consensus.cc:515] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:30.696604 17759 raft_consensus.cc:1275] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 24 index: 15. Preceding OpId from leader: term: 25 index: 15. (term mismatch)
I20260709 12:24:30.707211 17759 pending_rounds.cc:85] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb: Aborting all ops after (but not including) 14
I20260709 12:24:30.707557 17759 pending_rounds.cc:107] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb: Aborting uncommitted CHANGE_CONFIG_OP operation due to leader change: 24.15
I20260709 12:24:30.707866 17759 raft_consensus.cc:2917] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Aborting config change with OpId 24.15: Aborted: Op aborted by new leader
I20260709 12:24:30.604416 26398 raft_consensus.cc:493] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:30.618196 26365 raft_consensus.cc:493] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:24:30.715583 17759 raft_consensus.cc:2889] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:30.734354 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:30.739573 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 28 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:30.644757 26208 raft_consensus.cc:2749] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:24:30.742067 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 24 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:30.754912 17838 raft_consensus.cc:2393] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 24: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
W20260709 12:24:30.757526 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:30.638648 26300 raft_consensus.cc:697] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 21 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
W20260709 12:24:30.761966 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 342447us
I20260709 12:24:30.612723 26353 raft_consensus.cc:515] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:30.765945 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:30.769665 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:30.620427 26324 raft_consensus.cc:515] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:30.616822 25637 consensus_queue.cc:1059] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 25.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.098s
I20260709 12:24:30.719940 26365 raft_consensus.cc:3060] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:30.780536 17652 leader_election.cc:304] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:24:30.782836 26387 log.cc:927] Time spent T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.473s	user 0.000s	sys 0.001s
I20260709 12:24:30.784340 26346 leader_election.cc:290] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:30.786274 17686 raft_consensus.cc:606] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Received request to transfer leadership to TS e063541542a94969898868bf6eb5ce1b
I20260709 12:24:30.813620 17686 raft_consensus.cc:614] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:24:30.645264 26399 raft_consensus.cc:515] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:30.817075 26399 leader_election.cc:290] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:24:30.818915 26402 log.cc:927] Time spent T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.211s	user 0.000s	sys 0.000s
I20260709 12:24:30.792645 26355 leader_election.cc:290] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 25 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:30.822644 17683 raft_consensus.cc:2468] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 24.
I20260709 12:24:30.644937 26340 consensus_queue.cc:1048] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.119s
I20260709 12:24:30.617030 26359 consensus_queue.cc:1048] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.145s
I20260709 12:24:30.608462 26260 raft_consensus.cc:515] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:30.644611 26277 raft_consensus.cc:515] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:30.763460 26379 log.cc:927] Time spent T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.077s	user 0.001s	sys 0.000s
I20260709 12:24:30.721895 26397 raft_consensus.cc:515] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:24:30.719830 26350 raft_consensus.cc:493] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:30.617758 25688 raft_consensus.cc:515] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:30.748591 26398 raft_consensus.cc:515] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:30.772878 25917 raft_consensus.cc:2804] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:24:30.794456 26322 leader_election.cc:290] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:24:30.795573 26401 log.cc:927] Time spent T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.179s	user 0.000s	sys 0.000s
I20260709 12:24:30.796427 26300 consensus_queue.cc:237] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 20.17, Last appended by leader: 17, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:30.798240 26400 log.cc:927] Time spent T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.169s	user 0.000s	sys 0.000s
I20260709 12:24:30.799042 26323 raft_consensus.cc:3055] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:24:30.801301 26353 leader_election.cc:290] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:30.851835 17831 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35040 (request call id 3357) took 2429 ms. Trace:
W20260709 12:24:30.870420 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 300035us
I20260709 12:24:30.876204 17834 raft_consensus.cc:1240] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 18. Current term is 19. Ops: []
W20260709 12:24:30.876166 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 245963us
W20260709 12:24:30.877048 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 321697us
I20260709 12:24:30.878423 17802 leader_election.cc:304] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
W20260709 12:24:30.836359 26405 log.cc:927] Time spent T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.095s	user 0.000s	sys 0.001s
I20260709 12:24:30.916417 18333 leader_election.cc:304] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:24:30.859678 26239 consensus_meta.cc:287] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.505s	user 0.008s	sys 0.001s
I20260709 12:24:30.831707 25967 leader_election.cc:290] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:30.838269 25915 leader_election.cc:290] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:30.834995 25866 leader_election.cc:290] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 19 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:30.840977 26324 leader_election.cc:290] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:30.866137 25917 raft_consensus.cc:493] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:30.880316 17831 rpcz_store.cc:276] 0709 12:24:28.422444 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:28.692460 (+270016us) service_pool.cc:224] Handling call
0709 12:24:28.709038 (+ 16578us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:24:30.066676 (+1357638us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 12:24:30.066715 (+    39us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:24:30.090749 (+ 24034us) log.cc:844] Serialized 52 byte log entry
0709 12:24:30.266335 (+175586us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:24:30.266375 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:30.266437 (+    62us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:30.851487 (+585050us) raft_consensus.cc:1685] finished
0709 12:24:30.851522 (+    35us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:30.851745 (+   223us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1310871,"raft.queue_time_us":1270733,"raft.run_cpu_time_us":168,"raft.run_wall_time_us":23227,"thread_start_us":182516,"threads_started":2,"wal-append.queue_time_us":204966}
I20260709 12:24:30.911743 26323 raft_consensus.cc:740] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:24:30.845724 26260 leader_election.cc:290] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:30.857908 25688 leader_election.cc:290] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:30.860110 25761 raft_consensus.cc:2749] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:24:30.931079 17836 raft_consensus.cc:2410] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for term 26 because replica has last-logged OpId of term: 25 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 17.
W20260709 12:24:30.931991 18363 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.570s	user 0.010s	sys 0.022s
I20260709 12:24:30.933336 17830 raft_consensus.cc:1275] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 25 index: 14. Preceding OpId from leader: term: 26 index: 15. (index mismatch)
I20260709 12:24:30.863672 26398 leader_election.cc:290] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:24:30.934441 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 56112us
W20260709 12:24:30.971001 18363 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.609s	user 0.010s	sys 0.022s
I20260709 12:24:30.848461 26277 leader_election.cc:290] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:24:30.936502 17728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:30.929942 26359 raft_consensus.cc:2749] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
W20260709 12:24:30.938910 17727 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:30.971835 17802 leader_election.cc:304] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:30.855158 26397 leader_election.cc:290] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:30.960235 25740 raft_consensus.cc:493] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:30.999925 17802 leader_election.cc:304] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:30.962352 25915 raft_consensus.cc:493] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:30.928634 18333 leader_election.cc:304] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:30.947557 26239 raft_consensus.cc:2749] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:24:30.855507 26350 raft_consensus.cc:515] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:24:31.013190 18334 leader_election.cc:304] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:30.993575 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 27 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:31.015112 18367 raft_consensus.cc:1275] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 20 index: 14. Preceding OpId from leader: term: 22 index: 15. (index mismatch)
I20260709 12:24:31.000911 17763 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38786 (request call id 3820) took 1398 ms. Trace:
I20260709 12:24:31.016088 17763 rpcz_store.cc:276] 0709 12:24:29.602720 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:29.683351 (+ 80631us) service_pool.cc:224] Handling call
0709 12:24:29.708840 (+ 25489us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:29.735916 (+ 27076us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:29.735977 (+    61us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:29.736576 (+   599us) log.cc:844] Serialized 26 byte log entry
0709 12:24:30.102895 (+366319us) spinlock_profiling.cc:243] Waited 2 ms on lock 0x7b180018e0e0. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b47515e07 00007f7b503709a9 00007f7b4753269b 00007f7b4753093f 00007f7b47532c02 00007f7b4a1ba20b 00007f7b4a1bfd23 00007f7b4a1c00d1 00007f7b4a5dba0f
0709 12:24:30.261514 (+158619us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:24:30.261575 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:30.261660 (+    85us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:30.986166 (+724506us) raft_consensus.cc:1685] finished
0709 12:24:30.986195 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:31.000816 (+ 14621us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":26628,"raft.queue_time_us":57610,"raft.run_cpu_time_us":102,"raft.run_wall_time_us":4903,"spinlock_wait_cycles":4594432,"thread_start_us":158349,"threads_started":1,"wal-append.queue_time_us":640234}
I20260709 12:24:30.927027 26353 consensus_queue.cc:1048] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.040s
I20260709 12:24:31.004022 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:31.006069 25740 raft_consensus.cc:515] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:31.006742 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:30.965852 25917 raft_consensus.cc:3060] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:31.008255 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:31.030714 18366 raft_consensus.cc:2468] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 25.
W20260709 12:24:31.008764 17727 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:31.013489 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:24:31.031754 17727 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:31.032294 17680 raft_consensus.cc:2468] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 27.
I20260709 12:24:31.033457 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:31.012301 17685 raft_consensus.cc:1275] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 25 index: 14. Preceding OpId from leader: term: 26 index: 15. (index mismatch)
I20260709 12:24:30.974031 26334 raft_consensus.cc:493] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:31.021785 17684 raft_consensus.cc:2468] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 23.
I20260709 12:24:31.025100 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:31.025132 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 24 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:24:31.029213 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 100713us
I20260709 12:24:31.039258 18333 leader_election.cc:304] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:31.034245 17680 raft_consensus.cc:2468] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 20.
I20260709 12:24:31.003795 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 29 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:31.014994 26397 raft_consensus.cc:2749] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:24:31.009730 26399 raft_consensus.cc:2804] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:24:31.021579 26208 raft_consensus.cc:2749] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:24:31.042271 17684 raft_consensus.cc:1240] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 16. Current term is 17. Ops: []
I20260709 12:24:31.059748 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:31.012615 25688 raft_consensus.cc:493] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:31.072906 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 27 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:30.985170 26260 consensus_queue.cc:1059] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 18.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.051s
I20260709 12:24:31.047111 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:31.048676 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 24 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:31.077461 18371 raft_consensus.cc:3060] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:24:31.079114 26294 consensus_meta.cc:287] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.781s	user 0.008s	sys 0.003s
I20260709 12:24:31.085027 18365 raft_consensus.cc:3060] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:31.049810 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 24 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:31.050262 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:31.041749 17802 leader_election.cc:304] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:31.086436 18368 raft_consensus.cc:2393] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 24: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:24:31.051378 25637 raft_consensus.cc:3060] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:31.052395 25740 leader_election.cc:290] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:31.054648 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:30.084826 17837 rpcz_store.cc:276] 0709 12:24:28.446149 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:28.744089 (+297940us) service_pool.cc:224] Handling call
0709 12:24:28.745884 (+  1795us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:28.746075 (+   191us) raft_consensus.cc:1534] Early marking committed up to index 16
0709 12:24:28.797826 (+ 51751us) log.cc:844] Serialized 14 byte log entry
0709 12:24:29.568749 (+770923us) spinlock_profiling.cc:243] Waited 2.5 ms on lock 0x7b1800498fa0. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b47515e07 00007f7b503709a9 00007f7b4753269b 00007f7b4753093f 00007f7b47532c02 00007f7b4a1ba20b 00007f7b4a1bfd23 00007f7b4a1c023a 00007f7b4a62aa2a
0709 12:24:30.065579 (+496830us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:30.065628 (+    49us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:24:30.065668 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:30.065726 (+    58us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:30.065827 (+   101us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":434544,"prepare.queue_time_us":51451,"prepare.run_cpu_time_us":65,"prepare.run_wall_time_us":11421,"raft.queue_time_us":487208,"spinlock_wait_cycles":5756800,"thread_start_us":28496,"threads_started":1,"wal-append.queue_time_us":799584}
I20260709 12:24:31.042662 26334 raft_consensus.cc:515] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:31.062144 17687 raft_consensus.cc:1275] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 22 index: 17. Preceding OpId from leader: term: 23 index: 17. (term mismatch)
I20260709 12:24:31.070751 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 25 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:31.120783 17687 pending_rounds.cc:85] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3: Aborting all ops after (but not including) 16
I20260709 12:24:31.121198 17687 pending_rounds.cc:107] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3: Aborting uncommitted NO_OP operation due to leader change: 22.17
I20260709 12:24:31.121521 17687 raft_consensus.cc:2889] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:24:31.072176 26410 log.cc:927] Time spent T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.082s	user 0.000s	sys 0.001s
I20260709 12:24:31.035006 26340 raft_consensus.cc:2749] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:24:31.039707 26164 raft_consensus.cc:493] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:31.123893 26164 raft_consensus.cc:515] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:31.126796 26164 leader_election.cc:290] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:31.049002 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
W20260709 12:24:31.082894 26407 log.cc:927] Time spent T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.107s	user 0.000s	sys 0.001s
I20260709 12:24:31.128561 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:31.129492 18361 raft_consensus.cc:2393] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 20: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
W20260709 12:24:31.084160 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 45094.1us
W20260709 12:24:31.089702 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 48037.7us
I20260709 12:24:31.136255 17802 leader_election.cc:304] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:31.091528 26397 raft_consensus.cc:2749] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:24:31.143847 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:31.113041 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 22 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:31.116027 26419 raft_consensus.cc:493] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:31.119480 26354 raft_consensus.cc:3060] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:31.130381 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:31.046660 26399 raft_consensus.cc:493] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:31.142968 18371 raft_consensus.cc:2468] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 24.
I20260709 12:24:31.153236 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 19 candidate_status { last_received { term: 18 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:31.153844 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 24 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:31.155512 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:31.143852 17802 leader_election.cc:304] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:31.073864 26420 raft_consensus.cc:493] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:31.023666 25915 raft_consensus.cc:515] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:31.165779 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 23 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:31.161479 17802 leader_election.cc:304] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:24:31.136917 26363 consensus_meta.cc:287] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.856s	user 0.009s	sys 0.003s
I20260709 12:24:31.148284 26338 raft_consensus.cc:2749] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
W20260709 12:24:31.196139 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 60087us
I20260709 12:24:31.085563 25688 raft_consensus.cc:515] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:31.196921 17802 leader_election.cc:304] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:24:31.205075 18362 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 22.
I20260709 12:24:31.158915 26399 raft_consensus.cc:3060] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Advancing to term 25
W20260709 12:24:31.208858 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 113697us
I20260709 12:24:31.171118 25740 raft_consensus.cc:3060] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:31.213949 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 24 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:31.215859 17727 leader_election.cc:304] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:31.177537 26239 raft_consensus.cc:2749] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260709 12:24:31.218305 17727 leader_election.cc:304] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:31.217944 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:31.215566 17833 raft_consensus.cc:2468] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 23.
I20260709 12:24:31.218626 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:31.157603 26164 consensus_queue.cc:1048] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.049s
I20260709 12:24:31.090085 26323 consensus_queue.cc:260] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 16.13, Last appended by leader: 13, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:31.222196 17727 leader_election.cc:304] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:31.199998 26359 raft_consensus.cc:493] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:31.200038 26353 raft_consensus.cc:493] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:31.206099 25761 raft_consensus.cc:493] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:31.211588 26397 raft_consensus.cc:2804] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:24:31.214668 26332 raft_consensus.cc:3060] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:31.158687 26419 raft_consensus.cc:515] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:31.237876 17835 raft_consensus.cc:1240] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 27 LEADER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 26. Current term is 27. Ops: []
W20260709 12:24:31.183272 17730 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.616s	user 0.029s	sys 0.046s
I20260709 12:24:31.173951 26420 raft_consensus.cc:515] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:31.243067 17730 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.676s	user 0.029s	sys 0.053s
I20260709 12:24:31.223975 26392 raft_consensus.cc:2749] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:24:31.223631 26114 raft_consensus.cc:2749] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260709 12:24:31.225675 26218 raft_consensus.cc:3060] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:24:31.229586 26120 raft_consensus.cc:2749] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:24:31.216333 26234 consensus_queue.cc:1048] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.026s
W20260709 12:24:31.259088 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 128845us
W20260709 12:24:31.263687 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 53856.2us
W20260709 12:24:31.286859 18334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:31.233515 26398 raft_consensus.cc:493] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
W20260709 12:24:31.311685 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 72981.4us
I20260709 12:24:31.230247 17575 maintenance_manager.cc:643] P 2193af31cb894ec9b3d65d939a2a6851: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 2.770s	user 0.108s	sys 0.092s Metrics: {"cfile_cache_miss":8,"cfile_cache_miss_bytes":25615,"cfile_init":4,"delete_count":0,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"lbm_read_time_us":79734,"lbm_reads_10-100_ms":2,"lbm_reads_lt_1ms":22,"lbm_write_time_us":1390,"lbm_writes_lt_1ms":10,"peak_mem_usage":260771,"reinsert_count":0,"thread_start_us":476036,"threads_started":2,"update_count":175}
I20260709 12:24:31.220949 26391 consensus_queue.cc:1059] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 21.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.023s
I20260709 12:24:31.323436 18361 raft_consensus.cc:3060] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:31.252995 26323 raft_consensus.cc:3060] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:31.259781 26397 raft_consensus.cc:697] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 22 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:24:31.238240 26353 raft_consensus.cc:515] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:31.255311 26208 raft_consensus.cc:493] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:31.237218 26359 raft_consensus.cc:515] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:31.354053 17576 maintenance_manager.cc:419] P 2193af31cb894ec9b3d65d939a2a6851: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
W20260709 12:24:31.333171 26365 consensus_meta.cc:287] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.522s	user 0.008s	sys 0.007s
I20260709 12:24:31.249608 26346 consensus_queue.cc:1048] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.096s
I20260709 12:24:31.243090 26413 consensus_queue.cc:1059] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 18.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.097s
I20260709 12:24:31.239182 25761 raft_consensus.cc:515] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:31.262442 26270 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.628s	user 0.010s	sys 0.001s
I20260709 12:24:31.316850 26350 leader_election.cc:290] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:31.310530 26340 raft_consensus.cc:493] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:31.396391 26340 raft_consensus.cc:515] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:31.398669 26340 leader_election.cc:290] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:31.399209 26340 raft_consensus.cc:493] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:31.355520 17727 leader_election.cc:304] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:31.364850 26391 raft_consensus.cc:3055] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:24:31.365073 25917 raft_consensus.cc:515] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:31.347983 26412 raft_consensus.cc:493] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:31.369472 17654 leader_election.cc:304] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:24:31.310931 26239 consensus_queue.cc:1059] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.048s
I20260709 12:24:31.384390 17575 maintenance_manager.cc:643] P 2193af31cb894ec9b3d65d939a2a6851: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.014s	user 0.004s	sys 0.006s Metrics: {}
I20260709 12:24:31.388963 26260 consensus_queue.cc:1048] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.002s
I20260709 12:24:31.391638 26433 consensus_queue.cc:1048] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
W20260709 12:24:31.392765 26429 log.cc:927] Time spent T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.129s	user 0.000s	sys 0.000s
I20260709 12:24:31.394483 25761 leader_election.cc:290] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:31.502378 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:31.440503 17803 leader_election.cc:304] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:31.487787 17683 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37402 (request call id 3559) took 1204 ms. Trace:
I20260709 12:24:31.470590 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:31.491508 17832 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35042 (request call id 3133) took 1054 ms. Trace:
I20260709 12:24:31.418062 17835 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35042 (request call id 3130) took 1004 ms. Trace:
I20260709 12:24:31.460173 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:31.470983 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:31.505609 17838 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 21.
I20260709 12:24:31.442301 17836 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35042 (request call id 3127) took 1029 ms. Trace:
W20260709 12:24:31.505506 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 150067us
I20260709 12:24:31.434442 17686 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:54540 (request call id 55) took 1022 ms. Trace:
W20260709 12:24:31.506430 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 66068.9us
I20260709 12:24:31.395295 18365 raft_consensus.cc:2468] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 22.
I20260709 12:24:31.510169 17803 leader_election.cc:304] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:31.507177 17835 rpcz_store.cc:276] 0709 12:24:30.413754 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:30.428587 (+ 14833us) service_pool.cc:224] Handling call
0709 12:24:30.565112 (+136525us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:31.242724 (+677612us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:31.417833 (+175109us) spinlock_profiling.cc:243] Waited 671 ms on lock 0x7b5400267354. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:24:31.418001 (+   168us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1542265600}
I20260709 12:24:31.507050 17832 rpcz_store.cc:276] 0709 12:24:30.437391 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:30.571484 (+134093us) service_pool.cc:224] Handling call
0709 12:24:30.571729 (+   245us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:30.704699 (+132970us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:30.704745 (+    46us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:30.704800 (+    55us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:24:30.704845 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:31.491204 (+786359us) spinlock_profiling.cc:243] Waited 133 ms on lock 0x7b540008bbd4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:24:31.491271 (+    67us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:31.491435 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":305274624}
I20260709 12:24:31.454453 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:31.513195 18371 raft_consensus.cc:3060] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Advancing to term 25
W20260709 12:24:31.507114 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 137711us
I20260709 12:24:31.511834 18361 raft_consensus.cc:2468] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 19.
I20260709 12:24:31.506352 17683 rpcz_store.cc:276] 0709 12:24:30.283408 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:30.283524 (+   116us) service_pool.cc:224] Handling call
0709 12:24:31.487627 (+1204103us) spinlock_profiling.cc:243] Waited 493 ms on lock 0x7b54004277d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:24:31.487727 (+   100us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1134716672}
I20260709 12:24:31.516526 17728 leader_election.cc:304] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:31.508455 17836 rpcz_store.cc:276] 0709 12:24:30.412798 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:30.414362 (+  1564us) service_pool.cc:224] Handling call
0709 12:24:31.442045 (+1027683us) spinlock_profiling.cc:243] Waited 516 ms on lock 0x7b54000893d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:24:31.442223 (+   178us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1185695104}
I20260709 12:24:31.509681 17686 rpcz_store.cc:276] 0709 12:24:30.427908 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:30.428824 (+   916us) service_pool.cc:224] Handling call
0709 12:24:31.434224 (+1005400us) spinlock_profiling.cc:243] Waited 221 ms on lock 0x7b54004277d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a61a853 00007f7b4f5e860a 00007f7b48b6e7d4 00007f7b48b6e762 00007f7b48b6e691 00007f7b48b6e60c 00007f7b48b6d8c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:24:31.434342 (+   118us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":507615104}
I20260709 12:24:31.497089 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 19 candidate_status { last_received { term: 18 index: 18 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:31.524641 17834 raft_consensus.cc:2393] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 19: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:24:31.506626 17757 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 22.
I20260709 12:24:31.528481 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 24 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:31.529392 18367 raft_consensus.cc:2468] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 23.
I20260709 12:24:31.531178 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:31.535718 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:31.536901 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:31.563402 17831 raft_consensus.cc:2393] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 20: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:24:31.576033 17832 raft_consensus.cc:2468] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 25.
W20260709 12:24:31.563195 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 53980.4us
I20260709 12:24:31.478983 26334 leader_election.cc:290] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:31.440981 17759 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38776 (request call id 3050) took 1343 ms. Trace:
I20260709 12:24:31.608713 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:24:31.425098 26415 log.cc:927] Time spent T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.389s	user 0.000s	sys 0.001s
W20260709 12:24:31.395859 26270 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.776s	user 0.010s	sys 0.001s
I20260709 12:24:31.605506 17759 rpcz_store.cc:276] 0709 12:24:30.097938 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:30.199302 (+101364us) service_pool.cc:224] Handling call
0709 12:24:30.200665 (+  1363us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:30.716386 (+515721us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:31.440737 (+724351us) spinlock_profiling.cc:243] Waited 496 ms on lock 0x7b54003eb054. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:24:31.440917 (+   180us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1139654528}
W20260709 12:24:31.613561 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 97125.7us
I20260709 12:24:31.351999 26398 raft_consensus.cc:515] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:31.615072 17728 leader_election.cc:304] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:31.613690 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:31.611560 18334 leader_election.cc:304] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
W20260709 12:24:31.612732 17727 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:31.578697 17838 raft_consensus.cc:2468] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 26.
W20260709 12:24:31.617226 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 107136us
I20260709 12:24:31.610298 17654 leader_election.cc:304] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:24:31.466326 26283 raft_consensus.cc:493] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:31.626386 17728 leader_election.cc:304] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:24:31.627149 17830 raft_consensus.cc:2393] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 24: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:24:31.399806 26413 raft_consensus.cc:3055] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:24:31.429174 25688 leader_election.cc:290] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:31.628548 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:31.504588 26391 raft_consensus.cc:740] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:24:31.638257 17758 raft_consensus.cc:1275] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 20 index: 14. Preceding OpId from leader: term: 21 index: 15. (index mismatch)
I20260709 12:24:31.568931 26322 raft_consensus.cc:2804] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Leader pre-election won for term 26
W20260709 12:24:31.623348 25637 consensus_meta.cc:287] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.511s	user 0.007s	sys 0.003s
I20260709 12:24:31.469575 26377 raft_consensus.cc:493] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:31.626746 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 19 candidate_status { last_received { term: 18 index: 18 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:31.628269 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:31.659348 17683 raft_consensus.cc:2393] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 24: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:24:31.641358 18371 raft_consensus.cc:2468] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 25.
I20260709 12:24:31.638782 26263 raft_consensus.cc:2749] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:24:31.437091 25915 leader_election.cc:290] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:31.645090 17684 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 22.
I20260709 12:24:31.470283 26414 raft_consensus.cc:3055] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:24:31.427392 26353 leader_election.cc:290] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:31.561124 26164 raft_consensus.cc:493] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:31.667081 26427 log.cc:927] Time spent T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.134s	user 0.000s	sys 0.000s
I20260709 12:24:31.667238 26164 raft_consensus.cc:515] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:31.667783 26218 raft_consensus.cc:493] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:31.668366 26218 raft_consensus.cc:515] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:31.649979 26368 raft_consensus.cc:493] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:31.669656 26164 leader_election.cc:290] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:31.579577 26338 raft_consensus.cc:2749] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 12:24:31.669773 26368 raft_consensus.cc:515] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:31.561604 26412 raft_consensus.cc:515] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:31.671334 26354 consensus_meta.cc:287] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.500s	user 0.005s	sys 0.001s
I20260709 12:24:31.672827 26398 leader_election.cc:290] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:31.673019 25915 raft_consensus.cc:493] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:31.673502 26414 raft_consensus.cc:740] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:24:31.673542 25915 raft_consensus.cc:515] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:31.674798 26414 consensus_queue.cc:260] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 20.16, Last appended by leader: 16, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:31.399725 26340 raft_consensus.cc:515] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:31.644483 26413 raft_consensus.cc:740] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:24:31.685671 26413 consensus_queue.cc:260] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.13, Last appended by leader: 13, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:31.417484 26359 leader_election.cc:290] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:31.643788 26283 raft_consensus.cc:515] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:31.401587 26420 leader_election.cc:290] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:31.655347 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:31.660641 25637 raft_consensus.cc:2749] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:24:31.658237 26302 leader_election.cc:290] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:31.642263 17803 leader_election.cc:304] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:31.639350 26325 raft_consensus.cc:2804] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:24:31.389829 26365 raft_consensus.cc:515] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:31.638553 26334 raft_consensus.cc:2749] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:24:31.639042 26392 raft_consensus.cc:2749] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:24:31.664753 26234 raft_consensus.cc:2804] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:24:31.665170 26435 raft_consensus.cc:493] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:31.663069 26377 raft_consensus.cc:515] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:31.665601 26434 consensus_queue.cc:1048] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.058s
I20260709 12:24:31.666254 26416 raft_consensus.cc:493] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:31.384984 26208 raft_consensus.cc:515] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:31.579990 26344 raft_consensus.cc:2804] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:24:31.676170 26414 raft_consensus.cc:3060] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:31.665637 25917 leader_election.cc:290] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 22 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:31.672833 26412 leader_election.cc:290] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:31.777626 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 22 candidate_status { last_received { term: 21 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:31.778493 18361 raft_consensus.cc:2393] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 22: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:24:31.679344 26218 leader_election.cc:290] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:31.650945 26322 raft_consensus.cc:493] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:31.420455 26397 consensus_queue.cc:237] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.13, Last appended by leader: 13, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:31.658944 26421 raft_consensus.cc:493] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:31.684316 18374 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59886 (request call id 2253) took 1286 ms. Trace:
I20260709 12:24:31.785881 18374 rpcz_store.cc:276] 0709 12:24:30.398287 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:30.398430 (+   143us) service_pool.cc:224] Handling call
0709 12:24:30.398732 (+   302us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:30.408002 (+  9270us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:30.408049 (+    47us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:30.408562 (+   513us) log.cc:844] Serialized 26 byte log entry
0709 12:24:30.750656 (+342094us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:24:30.751698 (+  1042us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:30.753188 (+  1490us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:31.424536 (+671348us) spinlock_profiling.cc:243] Waited 2.06 ms on lock 0x7b4400022b4c. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b47459c85 00007f7b4745a6a2 00007f7b4a1cc1db 00007f7b4a1c7aaf 00007f7b4a1bb86e 00007f7b4a1babb6 00007f7b4a1ba7d8 00007f7b4a1c4221 00007f7b4a1c41d9 00007f7b4a1c4169
0709 12:24:31.683960 (+259424us) raft_consensus.cc:1685] finished
0709 12:24:31.683985 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:31.684227 (+   242us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":103,"raft.run_cpu_time_us":106,"raft.run_wall_time_us":7869,"spinlock_wait_cycles":4741376,"thread_start_us":103960,"threads_started":1,"wal-append.queue_time_us":528778}
W20260709 12:24:31.684649 26330 raft_consensus.cc:670] T a5732e71b9d54f8786b8c511d69943bf P 334a4e1f59944544b3b3ff36616b57da: failed to trigger leader election: Illegal state: only voting members can start elections: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:24:31.456177 26419 leader_election.cc:290] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:31.686969 26359 raft_consensus.cc:493] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
W20260709 12:24:31.692332 17655 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.572s	user 0.007s	sys 0.008s
W20260709 12:24:31.792328 17655 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.672s	user 0.007s	sys 0.008s
I20260709 12:24:31.762168 26399 raft_consensus.cc:515] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:31.619499 26355 raft_consensus.cc:493] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:31.641381 26114 consensus_queue.cc:1048] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 14, Time since last communication: 0.049s
I20260709 12:24:31.624547 25866 raft_consensus.cc:3055] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:24:31.704473 25384 consensus_queue.cc:1048] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20260709 12:24:31.819862 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 19 candidate_status { last_received { term: 18 index: 18 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
W20260709 12:24:31.705199 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 63007.1us
I20260709 12:24:31.726295 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:31.729202 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:24:31.694722 26409 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.611s	user 0.002s	sys 0.018s
I20260709 12:24:31.762281 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:31.822997 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 22 candidate_status { last_received { term: 21 index: 18 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:31.763799 18362 raft_consensus.cc:1240] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 24. Current term is 25. Ops: []
I20260709 12:24:31.766297 17688 raft_consensus.cc:1240] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 21. Current term is 22. Ops: []
I20260709 12:24:31.769129 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:31.703081 26325 raft_consensus.cc:493] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:31.844348 26325 raft_consensus.cc:3060] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:24:31.850977 26395 log.cc:927] Time spent T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 1.476s	user 0.006s	sys 0.002s
I20260709 12:24:31.769935 26435 raft_consensus.cc:515] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:31.853785 26435 leader_election.cc:290] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:31.860713 26283 leader_election.cc:290] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:31.862545 26208 leader_election.cc:290] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:24:31.768185 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 106932us
I20260709 12:24:31.768788 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:31.864444 17757 raft_consensus.cc:2468] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 25.
I20260709 12:24:31.771999 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:31.864841 17729 leader_election.cc:304] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:24:31.773769 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 23 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:31.700412 26261 raft_consensus.cc:493] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:31.704954 26420 raft_consensus.cc:2804] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:24:31.703235 25740 raft_consensus.cc:2749] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
W20260709 12:24:31.678625 26271 consensus_meta.cc:287] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 1.286s	user 0.012s	sys 0.000s
I20260709 12:24:31.742755 26164 raft_consensus.cc:2749] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 12:24:31.763104 26234 raft_consensus.cc:493] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:24:31.780216 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 153933us
I20260709 12:24:31.767827 26344 raft_consensus.cc:493] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:31.783918 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:31.888473 17761 raft_consensus.cc:2468] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 26.
I20260709 12:24:31.788270 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:31.795389 17727 leader_election.cc:304] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:24:31.889865 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 105087us
I20260709 12:24:31.890576 17727 leader_election.cc:304] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:24:31.795858 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 22 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:31.796824 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 24 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:31.797380 17802 leader_election.cc:304] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
W20260709 12:24:31.891798 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 94475.5us
I20260709 12:24:31.819921 25866 raft_consensus.cc:740] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 24 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:24:31.798040 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:31.916426 18361 raft_consensus.cc:1240] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 21. Current term is 22. Ops: []
I20260709 12:24:31.817759 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:31.818274 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 23 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:31.818665 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:31.917421 17689 raft_consensus.cc:2468] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 26.
I20260709 12:24:31.820812 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:31.917805 17686 raft_consensus.cc:2468] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 20.
I20260709 12:24:31.918108 17685 raft_consensus.cc:2468] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 26.
I20260709 12:24:31.821784 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:31.824285 17762 raft_consensus.cc:2468] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 25.
I20260709 12:24:31.829733 26365 leader_election.cc:290] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 22 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:31.798782 26359 raft_consensus.cc:515] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:31.830669 18333 leader_election.cc:304] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:31.832132 26399 leader_election.cc:290] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 25 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:31.770514 26416 raft_consensus.cc:515] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:31.817268 26355 raft_consensus.cc:515] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:31.936820 26114 consensus_queue.cc:1059] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 21.15, Next index: 16, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:24:31.937777 26355 leader_election.cc:290] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:31.836390 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:31.944088 17689 raft_consensus.cc:1275] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 26 index: 14. Preceding OpId from leader: term: 27 index: 15. (index mismatch)
I20260709 12:24:31.790508 26421 raft_consensus.cc:515] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:31.839457 17755 raft_consensus.cc:2393] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 26: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:24:31.870761 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 23 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:31.955010 17689 raft_consensus.cc:1275] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 20 index: 17. Preceding OpId from leader: term: 21 index: 18. (index mismatch)
I20260709 12:24:31.955861 18371 raft_consensus.cc:2468] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 22.
I20260709 12:24:31.871708 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:31.957222 18374 raft_consensus.cc:2468] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 22.
I20260709 12:24:31.880805 17760 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36572 (request call id 1471) took 2102 ms. Trace:
I20260709 12:24:31.958220 17760 rpcz_store.cc:276] 0709 12:24:29.770900 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:29.843565 (+ 72665us) service_pool.cc:224] Handling call
0709 12:24:29.844008 (+   443us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:29.849053 (+  5045us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:29.849083 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:29.852127 (+  3044us) log.cc:844] Serialized 26 byte log entry
0709 12:24:30.070796 (+218669us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:24:30.070855 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:30.070950 (+    95us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:31.414853 (+1343903us) spinlock_profiling.cc:243] Waited 4.87 ms on lock 0x7b4400022b4c. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b47459c85 00007f7b4745a6a2 00007f7b4a1cc1db 00007f7b4a1c7aaf 00007f7b4a1bb86e 00007f7b4a1babb6 00007f7b4a1ba7d8 00007f7b4a1c4221 00007f7b4a1c41d9 00007f7b4a1c4169
0709 12:24:31.850652 (+435799us) spinlock_profiling.cc:243] Waited 126 ms on lock 0x7b24000f72f8. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b473b99c4 00007f7b473b7acb 00007f7b47572dc9 00007f7b47572a55 00007f7b4757209d 00007f7b4a1a410e 00007f7b4a1bb88c 00007f7b4a1babb6 00007f7b4a1ba7d8 00007f7b4a1c4221
0709 12:24:31.872492 (+ 21840us) raft_consensus.cc:1685] finished
0709 12:24:31.872524 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:31.872824 (+   300us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":5642,"raft.run_cpu_time_us":88,"raft.run_wall_time_us":8413,"spinlock_wait_cycles":302122112,"thread_start_us":134194,"threads_started":1,"wal-append.queue_time_us":355685}
I20260709 12:24:31.960333 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:24:31.822817 26409 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.739s	user 0.002s	sys 0.019s
I20260709 12:24:31.884810 17728 leader_election.cc:304] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:24:31.883241 17682 raft_consensus.cc:2468] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 27.
I20260709 12:24:31.883277 17801 leader_election.cc:304] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:24:31.961730 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 78537us
I20260709 12:24:31.893051 17830 raft_consensus.cc:2393] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 27: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:24:31.783511 26322 raft_consensus.cc:3060] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:31.886253 26452 raft_consensus.cc:2804] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:24:31.894865 26451 consensus_queue.cc:1048] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.005s
I20260709 12:24:31.895022 26450 raft_consensus.cc:493] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:31.970844 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:31.970685 26450 raft_consensus.cc:515] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:31.896325 26448 log.cc:927] Time spent T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.102s	user 0.001s	sys 0.000s
W20260709 12:24:31.897169 26444 log.cc:927] Time spent T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.074s	user 0.000s	sys 0.000s
I20260709 12:24:31.897692 25784 raft_consensus.cc:2804] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:24:31.898013 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:31.900398 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:31.900929 26354 raft_consensus.cc:493] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:31.902441 26206 raft_consensus.cc:3055] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:24:31.904866 17758 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38776 (request call id 3048) took 1807 ms. Trace:
I20260709 12:24:31.910324 26377 leader_election.cc:290] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:32.016323 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 22 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:32.017032 18362 raft_consensus.cc:2468] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 21.
I20260709 12:24:32.018438 26340 leader_election.cc:290] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:31.868321 26208 raft_consensus.cc:493] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:31.916236 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:32.010353 26206 raft_consensus.cc:740] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 19 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:24:32.021015 17680 raft_consensus.cc:2393] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 22: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
W20260709 12:24:31.815274 17652 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:31.919814 17829 raft_consensus.cc:1240] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 21. Current term is 22. Ops: []
I20260709 12:24:32.021442 26206 consensus_queue.cc:260] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 19.14, Last appended by leader: 14, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:32.009989 26422 raft_consensus.cc:493] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:32.022899 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:32.023520 26422 raft_consensus.cc:515] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:32.024364 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:32.024986 17829 raft_consensus.cc:2468] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 25.
I20260709 12:24:32.025755 26422 leader_election.cc:290] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:32.025810 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:32.026508 17829 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 21.
I20260709 12:24:32.027329 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 27 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:31.830636 18365 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "28062a50142448aa97994888b4765920"
dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
mode: GRACEFUL
new_leader_uuid: "e063541542a94969898868bf6eb5ce1b"
 from {username='slave'} at 127.0.0.1:50468
I20260709 12:24:32.028123 18365 raft_consensus.cc:606] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Received request to transfer leadership to TS e063541542a94969898868bf6eb5ce1b
I20260709 12:24:32.010967 17758 rpcz_store.cc:276] 0709 12:24:30.097313 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:30.097447 (+   134us) service_pool.cc:224] Handling call
0709 12:24:30.097721 (+   274us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:31.649762 (+1552041us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:31.904603 (+254841us) spinlock_profiling.cc:243] Waited 1.54 s on lock 0x7b54003e5dd4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:24:31.904800 (+   197us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3542156544}
I20260709 12:24:32.028487 18365 raft_consensus.cc:614] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:24:31.922525 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:32.029080 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:32.029529 17835 raft_consensus.cc:2468] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 21.
I20260709 12:24:32.029620 17758 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 22.
I20260709 12:24:32.030408 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 27 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:24:31.924300 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 93716.8us
I20260709 12:24:32.031597 17835 raft_consensus.cc:1240] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 20. Current term is 21. Ops: [18.15-20.16]
I20260709 12:24:31.913640 26391 consensus_queue.cc:260] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 21.16, Last appended by leader: 16, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:31.925179 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 24 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:31.925323 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:31.877449 26420 raft_consensus.cc:493] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:32.043835 26420 raft_consensus.cc:3060] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:32.044060 26391 raft_consensus.cc:3060] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:31.886631 26234 raft_consensus.cc:3060] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:24:32.046545 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:32.047240 17757 raft_consensus.cc:2468] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 26.
I20260709 12:24:31.880298 26330 raft_consensus.cc:493] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:32.049645 26330 raft_consensus.cc:515] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:31.884274 26283 raft_consensus.cc:493] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:32.052205 26283 raft_consensus.cc:515] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:32.054287 26283 leader_election.cc:290] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:31.930945 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:32.055038 17762 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 22.
I20260709 12:24:32.056042 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:32.056605 17762 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 22.
I20260709 12:24:31.931865 26443 raft_consensus.cc:2804] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:24:31.932427 26447 raft_consensus.cc:2804] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:24:31.933369 17687 raft_consensus.cc:2468] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 22.
I20260709 12:24:31.934976 26416 leader_election.cc:290] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:31.917789 26323 raft_consensus.cc:493] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:31.885241 26261 raft_consensus.cc:515] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:31.961787 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:24:31.965189 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 80501.8us
W20260709 12:24:31.966465 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 354960us
I20260709 12:24:31.968043 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:32.069473 17728 leader_election.cc:304] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:31.932024 26404 raft_consensus.cc:2804] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:24:31.967228 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:31.970005 26452 raft_consensus.cc:697] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 25 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:24:31.890414 26344 raft_consensus.cc:3060] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:31.969211 25881 raft_consensus.cc:2804] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:24:31.969542 17729 leader_election.cc:304] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:31.971602 17830 raft_consensus.cc:2468] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 25.
I20260709 12:24:31.972581 26454 raft_consensus.cc:493] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:31.979342 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:31.979588 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:31.969259 26350 raft_consensus.cc:2804] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:24:31.981338 17801 leader_election.cc:304] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:31.985081 26359 leader_election.cc:290] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:32.064429 26447 raft_consensus.cc:493] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:31.986918 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:31.964970 25866 consensus_queue.cc:260] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 24.17, Last appended by leader: 17, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:31.989001 26325 raft_consensus.cc:515] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:31.990087 26368 leader_election.cc:290] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:31.991039 26421 leader_election.cc:290] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:31.980820 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:31.999475 18374 raft_consensus.cc:1240] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 18. Current term is 19. Ops: []
I20260709 12:24:31.975349 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:32.004905 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:32.008495 17683 raft_consensus.cc:2468] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 26.
I20260709 12:24:32.008131 25784 raft_consensus.cc:493] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:24:32.013657 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 23 candidate_status { last_received { term: 21 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:32.014320 26354 raft_consensus.cc:515] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:31.923949 17802 leader_election.cc:304] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:24:32.092774 26340 raft_consensus.cc:493] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:32.022907 26206 raft_consensus.cc:3060] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:32.081683 26323 raft_consensus.cc:515] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:32.061934 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:32.060700 26330 leader_election.cc:290] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:32.062079 26443 raft_consensus.cc:697] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 24 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:24:32.063105 17835 raft_consensus.cc:1275] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 20 index: 12. Preceding OpId from leader: term: 23 index: 13. (index mismatch)
I20260709 12:24:32.020417 26208 raft_consensus.cc:515] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:32.068720 17755 raft_consensus.cc:2468] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 22.
I20260709 12:24:32.075609 17831 raft_consensus.cc:2410] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for term 26 because replica has last-logged OpId of term: 25 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 17.
I20260709 12:24:32.071619 17652 leader_election.cc:304] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
W20260709 12:24:32.079798 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 258644us
I20260709 12:24:32.134306 17803 leader_election.cc:304] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:32.135444 17803 leader_election.cc:304] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:32.078220 17654 leader_election.cc:304] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:24:32.084934 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:32.086488 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 22 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:32.089082 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:31.985419 25915 leader_election.cc:290] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:24:32.095516 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 126786us
I20260709 12:24:32.144187 26149 raft_consensus.cc:2749] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:24:32.144305 17729 leader_election.cc:304] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:24:32.096657 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 115380us
I20260709 12:24:32.145522 17801 leader_election.cc:304] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:24:32.127122 26450 leader_election.cc:290] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:32.097884 17761 raft_consensus.cc:2468] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 23.
I20260709 12:24:32.080420 26409 raft_consensus.cc:493] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:32.148309 26409 raft_consensus.cc:515] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:32.149385 26354 leader_election.cc:290] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:32.085466 25881 raft_consensus.cc:493] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:32.083200 26283 raft_consensus.cc:2749] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260709 12:24:32.031913 18333 leader_election.cc:304] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:24:32.150779 26409 leader_election.cc:290] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:32.151328 26409 raft_consensus.cc:2804] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:24:32.108232 18363 raft_consensus.cc:2468] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 22.
I20260709 12:24:32.151691 26450 raft_consensus.cc:2804] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Leader pre-election won for term 21
W20260709 12:24:32.113005 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 192223us
I20260709 12:24:32.116058 26397 raft_consensus.cc:2804] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:24:32.092075 26350 raft_consensus.cc:493] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:24:32.110801 25784 raft_consensus.cc:3060] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:32.043434 17756 raft_consensus.cc:2393] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 25: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:24:32.100387 26447 raft_consensus.cc:3060] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:32.101670 26271 raft_consensus.cc:493] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:24:32.102535 26451 raft_consensus.cc:2804] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:24:32.084637 26120 consensus_queue.cc:1048] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.024s
I20260709 12:24:32.114130 26359 raft_consensus.cc:493] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:32.151197 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 79636.3us
I20260709 12:24:32.123862 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 23 candidate_status { last_received { term: 21 index: 18 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:32.111205 25866 raft_consensus.cc:3060] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Advancing to term 25
W20260709 12:24:32.151152 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 119351us
I20260709 12:24:32.140905 17803 leader_election.cc:304] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:32.142206 26443 consensus_queue.cc:237] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 23.13, Last appended by leader: 13, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:32.148696 25740 raft_consensus.cc:2804] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:24:32.150142 25881 raft_consensus.cc:3060] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:32.109644 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:32.097332 26454 raft_consensus.cc:515] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:32.151952 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 73850.2us
I20260709 12:24:32.201889 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 23 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:32.156992 17756 raft_consensus.cc:1275] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 26 index: 15. Preceding OpId from leader: term: 27 index: 16. (index mismatch)
I20260709 12:24:32.152554 17802 leader_election.cc:304] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:32.162155 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 26 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:32.163390 17652 leader_election.cc:304] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:24:32.164551 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:32.164834 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 25 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:32.206831 18371 raft_consensus.cc:2468] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 24.
I20260709 12:24:32.163832 17832 raft_consensus.cc:2468] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 21.
I20260709 12:24:32.168917 17831 raft_consensus.cc:1240] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 18. Current term is 19. Ops: []
I20260709 12:24:32.207944 26434 raft_consensus.cc:2804] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:24:32.208382 26434 raft_consensus.cc:493] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:32.178282 26325 leader_election.cc:290] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 24 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:32.172518 26446 raft_consensus.cc:2749] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:24:32.137006 26399 raft_consensus.cc:2749] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 12:24:32.172686 18333 leader_election.cc:304] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:24:32.176374 26404 raft_consensus.cc:493] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:32.177831 25740 raft_consensus.cc:493] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:32.210402 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:32.168579 26164 raft_consensus.cc:2804] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:24:32.172766 26261 leader_election.cc:290] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:32.162469 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:32.212065 18374 raft_consensus.cc:2468] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 26.
I20260709 12:24:32.120458 26340 raft_consensus.cc:515] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:32.152741 26350 raft_consensus.cc:3060] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:32.218317 26340 leader_election.cc:290] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:24:32.218894 26464 log.cc:927] Time spent T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.105s	user 0.001s	sys 0.000s
I20260709 12:24:32.160347 26451 raft_consensus.cc:493] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:32.152504 26397 raft_consensus.cc:493] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:32.220098 26451 raft_consensus.cc:3060] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:32.220317 26397 raft_consensus.cc:3060] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:32.220845 17838 raft_consensus.cc:1275] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 26 index: 15. Preceding OpId from leader: term: 27 index: 16. (index mismatch)
I20260709 12:24:32.161810 26359 raft_consensus.cc:515] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:32.158286 26283 raft_consensus.cc:2804] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:24:32.224838 26283 raft_consensus.cc:493] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:32.216264 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:24:32.226491 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 53880.4us
W20260709 12:24:32.226938 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 74478.4us
I20260709 12:24:32.227191 18333 leader_election.cc:304] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:32.228188 17802 leader_election.cc:304] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:32.204644 18364 raft_consensus.cc:2468] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 24.
I20260709 12:24:32.210703 25740 raft_consensus.cc:3060] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:32.196815 17803 leader_election.cc:304] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:32.234150 17652 leader_election.cc:304] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:32.152087 26277 raft_consensus.cc:2804] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:24:32.160175 26271 raft_consensus.cc:515] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:32.189908 26454 leader_election.cc:290] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:32.260874 26271 leader_election.cc:290] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:32.191512 26208 leader_election.cc:290] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:32.263983 26323 leader_election.cc:290] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:32.208753 26434 raft_consensus.cc:3060] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:32.243408 18333 leader_election.cc:304] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:32.267472 18333 leader_election.cc:304] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:32.269758 18333 leader_election.cc:400] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 23 election: Vote denied by peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651) with higher term. Message: Invalid argument: T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 23. Current term is 24.
I20260709 12:24:32.270112 18333 leader_election.cc:403] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 23 election: Cancelling election due to peer responding with higher term
I20260709 12:24:32.269933 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 28 candidate_status { last_received { term: 27 index: 21 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:32.270644 26323 raft_consensus.cc:3060] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:32.271111 26261 raft_consensus.cc:493] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:32.271358 26334 consensus_queue.cc:1048] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20260709 12:24:32.271833 26261 raft_consensus.cc:515] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:32.275048 25881 raft_consensus.cc:515] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:32.276854 17829 raft_consensus.cc:2393] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 27: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:24:32.210448 26404 raft_consensus.cc:3060] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:32.280107 26208 raft_consensus.cc:2749] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:24:32.255460 26350 raft_consensus.cc:515] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:32.284020 26261 leader_election.cc:290] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:32.244532 17652 leader_election.cc:304] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:32.152115 26450 raft_consensus.cc:493] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:32.245273 26398 raft_consensus.cc:2749] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:24:32.245985 26436 raft_consensus.cc:2749] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:24:32.240100 26283 raft_consensus.cc:3060] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:32.248235 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:32.241022 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 28 candidate_status { last_received { term: 27 index: 21 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:32.301784 18362 raft_consensus.cc:2468] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 27.
I20260709 12:24:32.302685 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:32.251830 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:32.245366 26419 raft_consensus.cc:2749] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:24:32.168076 26452 consensus_queue.cc:237] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 23.16, Last appended by leader: 16, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:32.297348 26450 raft_consensus.cc:3060] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:32.249395 26277 raft_consensus.cc:493] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:32.257464 17680 raft_consensus.cc:1240] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 23. Current term is 24. Ops: [23.17-23.17]
I20260709 12:24:32.257537 26399 raft_consensus.cc:2804] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:24:32.249876 25967 consensus_queue.cc:1048] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.008s
I20260709 12:24:32.258611 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:32.291208 18333 leader_election.cc:304] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:32.290397 26208 raft_consensus.cc:2749] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Leader election lost for term 23. Reason: Vote denied by peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651) with higher term. Message: Invalid argument: T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 23. Current term is 24.
W20260709 12:24:32.294402 17727 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:32.223771 26359 leader_election.cc:290] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:32.301069 26271 raft_consensus.cc:2749] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
W20260709 12:24:32.295334 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 50877.2us
I20260709 12:24:32.291105 17802 leader_election.cc:304] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:32.256829 25915 raft_consensus.cc:2804] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:24:32.303280 18362 raft_consensus.cc:2468] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 17.
W20260709 12:24:32.309474 17801 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:32.285725 26234 raft_consensus.cc:515] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:32.291801 26412 raft_consensus.cc:3055] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:24:32.293032 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:32.324370 26277 raft_consensus.cc:3060] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:32.332981 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 24 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:32.151871 26409 raft_consensus.cc:493] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:24:32.337901 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 46940.7us
I20260709 12:24:32.336651 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:32.328188 18333 leader_election.cc:304] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:32.298308 26334 consensus_queue.cc:1048] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.036s
I20260709 12:24:32.349736 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:32.350421 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:32.332522 26419 raft_consensus.cc:3060] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:32.361385 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:32.324645 26399 raft_consensus.cc:493] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
W20260709 12:24:32.355806 17802 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:32.353490 17680 raft_consensus.cc:2468] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 25.
I20260709 12:24:32.314661 26322 raft_consensus.cc:515] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:32.364454 17762 raft_consensus.cc:1240] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 27. Current term is 28. Ops: []
I20260709 12:24:32.343274 26302 raft_consensus.cc:2804] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:24:32.339215 25915 raft_consensus.cc:493] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:32.365512 17652 leader_election.cc:304] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:24:32.368266 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 22 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:32.367080 17727 leader_election.cc:304] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:32.373126 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 28 candidate_status { last_received { term: 27 index: 21 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:32.347069 26412 raft_consensus.cc:740] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:24:32.381319 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:32.386252 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:32.361611 26409 raft_consensus.cc:3060] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:24:32.388659 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 60591.4us
I20260709 12:24:32.396486 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:32.397168 18368 raft_consensus.cc:2468] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 24.
I20260709 12:24:32.390892 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:32.399716 17684 raft_consensus.cc:2368] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 25. Current term is 26.
I20260709 12:24:32.403015 17654 leader_election.cc:304] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:32.403990 17729 leader_election.cc:304] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:32.406816 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:32.371392 26164 raft_consensus.cc:493] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:32.349048 26433 consensus_queue.cc:1048] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.038s
I20260709 12:24:32.405113 17729 leader_election.cc:304] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:32.370636 26399 raft_consensus.cc:3060] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:32.348899 26365 consensus_queue.cc:1048] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.038s
I20260709 12:24:32.417869 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:32.380936 25915 raft_consensus.cc:3060] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:32.372252 26302 raft_consensus.cc:493] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:32.381987 26350 leader_election.cc:290] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 24 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:32.391551 26208 raft_consensus.cc:2749] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:24:32.400606 26363 raft_consensus.cc:2749] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:24:32.374112 26325 consensus_queue.cc:1059] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 18.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.051s
I20260709 12:24:32.413672 26436 raft_consensus.cc:2804] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:24:32.437460 17686 raft_consensus.cc:2468] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 28.
I20260709 12:24:32.419430 26164 raft_consensus.cc:3060] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:32.436849 17654 leader_election.cc:304] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:24:32.418816 26435 raft_consensus.cc:2749] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:24:32.418975 26354 raft_consensus.cc:2804] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:24:32.415483 25881 leader_election.cc:290] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 27 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:32.436178 26302 raft_consensus.cc:3060] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:24:32.480676 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 43907.9us
I20260709 12:24:32.482219 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:32.484225 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:32.484853 18371 raft_consensus.cc:2468] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 29.
I20260709 12:24:32.488160 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:32.488492 18334 leader_election.cc:304] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:32.492424 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:32.490901 17802 leader_election.cc:304] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:24:32.530719 17830 raft_consensus.cc:2393] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 19: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:24:32.488852 17829 raft_consensus.cc:2468] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 19.
I20260709 12:24:32.536972 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 25 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:32.561718 25784 raft_consensus.cc:515] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:32.562958 26416 raft_consensus.cc:2804] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:24:32.570582 26447 raft_consensus.cc:515] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:32.572196 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:32.628541 17686 raft_consensus.cc:3060] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:32.573395 17685 raft_consensus.cc:1275] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 20 index: 13. Preceding OpId from leader: term: 22 index: 14. (index mismatch)
I20260709 12:24:32.628742 17652 leader_election.cc:304] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:32.582383 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 26 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:32.629545 17838 raft_consensus.cc:2468] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 25.
I20260709 12:24:32.446398 26436 raft_consensus.cc:493] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:32.538866 18333 leader_election.cc:304] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
W20260709 12:24:32.633955 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 95206.4us
I20260709 12:24:32.586678 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:32.602090 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 22 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:32.589032 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 21 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:32.636325 17829 raft_consensus.cc:2376] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 21. Re-sending same reply.
W20260709 12:24:32.597453 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 192277us
I20260709 12:24:32.608222 26120 consensus_queue.cc:1059] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 21.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:24:32.609052 26421 raft_consensus.cc:2804] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:24:32.609460 17728 leader_election.cc:304] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:32.617734 26399 raft_consensus.cc:515] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:32.620951 26416 raft_consensus.cc:493] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:32.620810 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:32.669726 17835 raft_consensus.cc:2468] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 17.
I20260709 12:24:32.626909 26451 raft_consensus.cc:515] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:32.463624 26354 raft_consensus.cc:493] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:32.631273 17801 leader_election.cc:304] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:32.434973 26420 raft_consensus.cc:515] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:32.632846 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:24:32.633071 17730 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:32.688350 18367 raft_consensus.cc:1240] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 23. Current term is 24. Ops: []
I20260709 12:24:32.639463 17729 leader_election.cc:304] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:32.640417 26436 raft_consensus.cc:3060] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:32.649590 26377 raft_consensus.cc:2804] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:24:32.651799 26168 raft_consensus.cc:1064] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3: attempting to promote NON_VOTER 334a4e1f59944544b3b3ff36616b57da to VOTER
W20260709 12:24:32.653738 26467 log.cc:927] Time spent T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.528s	user 0.006s	sys 0.000s
I20260709 12:24:32.676818 17758 raft_consensus.cc:2410] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for term 27 because replica has last-logged OpId of term: 25 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 14.
I20260709 12:24:32.664383 17727 leader_election.cc:304] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:32.688190 17831 raft_consensus.cc:2468] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 27.
I20260709 12:24:32.599210 25881 raft_consensus.cc:2804] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:24:32.704751 25881 raft_consensus.cc:493] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:32.705178 25881 raft_consensus.cc:3060] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Advancing to term 22
W20260709 12:24:32.706701 26206 consensus_meta.cc:287] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.585s	user 0.003s	sys 0.004s
W20260709 12:24:32.679175 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 50520.2us
I20260709 12:24:32.708632 17652 leader_election.cc:304] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:24:32.473207 25967 consensus_queue.cc:1059] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 18.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.179s
I20260709 12:24:32.710721 26434 raft_consensus.cc:515] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:32.713161 26322 leader_election.cc:290] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 26 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:32.717671 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 24 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:32.719012 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:32.684471 26416 raft_consensus.cc:3060] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:32.719712 17758 raft_consensus.cc:2393] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 29: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
W20260709 12:24:32.727056 26344 consensus_meta.cc:287] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.634s	user 0.005s	sys 0.002s
I20260709 12:24:32.701607 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:32.728070 17755 raft_consensus.cc:3060] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:32.730528 26359 raft_consensus.cc:3060] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:32.730690 26409 raft_consensus.cc:515] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:32.723259 25740 raft_consensus.cc:515] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Starting leader election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
W20260709 12:24:32.692014 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 60836.2us
W20260709 12:24:32.735056 17801 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:32.713356 26309 raft_consensus.cc:3055] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:24:32.700408 17688 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44898 (request call id 1653) took 2229 ms. Trace:
I20260709 12:24:33.090432 17688 rpcz_store.cc:276] 0709 12:24:30.471618 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:30.471771 (+   153us) service_pool.cc:224] Handling call
0709 12:24:30.472046 (+   275us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:31.836678 (+1364632us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:32.700194 (+863516us) spinlock_profiling.cc:243] Waited 1.29 s on lock 0x7b5400347bd4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:24:32.700354 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2976017792}
I20260709 12:24:32.722733 17652 leader_election.cc:304] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:32.689960 26433 raft_consensus.cc:2749] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260709 12:24:33.092217 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 369568us
I20260709 12:24:33.094545 17652 leader_election.cc:304] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:32.723809 26271 raft_consensus.cc:2804] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:24:33.099184 26271 raft_consensus.cc:493] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:24:32.724926 26447 leader_election.cc:290] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 24 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:32.644030 26421 raft_consensus.cc:493] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:24:32.755270 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 90960.5us
I20260709 12:24:32.685043 26354 raft_consensus.cc:3060] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:32.736449 26377 raft_consensus.cc:493] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:32.699335 26446 raft_consensus.cc:2804] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Leader pre-election won for term 22
W20260709 12:24:32.754281 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 114928us
I20260709 12:24:33.101531 26309 raft_consensus.cc:740] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:24:33.102535 26309 consensus_queue.cc:260] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 18.18, Last appended by leader: 18, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:33.103844 17727 leader_election.cc:304] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:24:33.106204 26349 raft_consensus.cc:2804] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:24:33.106608 26349 raft_consensus.cc:493] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:33.106757 26446 raft_consensus.cc:493] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:33.107029 26349 raft_consensus.cc:3060] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:24:33.107317 26446 raft_consensus.cc:3060] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:33.108392 25740 leader_election.cc:290] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 26 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:24:33.111378 25915 consensus_meta.cc:287] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.670s	user 0.006s	sys 0.010s
I20260709 12:24:33.111177 26416 raft_consensus.cc:515] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:33.111796 25915 raft_consensus.cc:515] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:33.112180 26377 raft_consensus.cc:3060] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:33.113524 26416 leader_election.cc:290] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 18 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:33.113632 26421 raft_consensus.cc:3060] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:33.115458 17729 leader_election.cc:304] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:24:33.092404 26433 raft_consensus.cc:2749] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:24:32.747393 25881 raft_consensus.cc:515] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:32.829332 17762 raft_consensus.cc:1240] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 22. Current term is 23. Ops: [22.15-22.15]
W20260709 12:24:32.893520 17803 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.601s	user 0.019s	sys 0.010s
W20260709 12:24:32.768100 26368 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.600s	user 0.008s	sys 0.068s
I20260709 12:24:32.919157 17833 raft_consensus.cc:1240] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 25. Current term is 26. Ops: [25.15-25.15]
I20260709 12:24:32.692926 26398 raft_consensus.cc:2749] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:24:32.727481 26344 raft_consensus.cc:515] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:24:32.823612 26470 log.cc:927] Time spent T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.632s	user 0.002s	sys 0.002s
I20260709 12:24:32.925446 17686 raft_consensus.cc:2468] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 27.
W20260709 12:24:33.133251 17832 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.677s	user 0.011s	sys 0.004s
W20260709 12:24:33.133487 17727 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:33.133605 17832 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.678s	user 0.011s	sys 0.004s
I20260709 12:24:32.942528 17802 leader_election.cc:400] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 25 election: Vote denied by peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071) with higher term. Message: Invalid argument: T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 25. Current term is 26.
I20260709 12:24:33.133971 17802 leader_election.cc:403] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 25 election: Cancelling election due to peer responding with higher term
W20260709 12:24:32.835764 26168 raft_consensus.cc:1068] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3: Unable to promote non-voter 334a4e1f59944544b3b3ff36616b57da: Illegal state: Leader has not yet committed an operation in its own term
I20260709 12:24:32.728082 26234 leader_election.cc:290] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 28 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:32.835229 26350 raft_consensus.cc:2749] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:24:32.964386 26420 leader_election.cc:290] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 21 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:24:32.917806 26323 consensus_meta.cc:287] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.647s	user 0.017s	sys 0.001s
W20260709 12:24:32.969157 18366 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.682s	user 0.030s	sys 0.046s
W20260709 12:24:33.139693 18366 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.853s	user 0.030s	sys 0.046s
I20260709 12:24:32.960639 26149 consensus_queue.cc:1048] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.015s
I20260709 12:24:33.004820 26413 raft_consensus.cc:3060] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:33.127627 25915 leader_election.cc:290] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 23 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:33.145394 25881 leader_election.cc:290] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 22 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:24:33.138839 26277 consensus_meta.cc:287] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.774s	user 0.013s	sys 0.003s
I20260709 12:24:33.146656 26277 raft_consensus.cc:515] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:33.009056 26451 leader_election.cc:290] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 23 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:33.158854 26451 raft_consensus.cc:493] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:24:33.159394 26451 raft_consensus.cc:515] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:33.160945 26446 raft_consensus.cc:515] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:33.162192 26451 leader_election.cc:290] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:33.018034 26409 leader_election.cc:290] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 27 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:33.019416 25784 leader_election.cc:290] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 26 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:32.902292 26239 consensus_queue.cc:1059] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.068s
W20260709 12:24:33.166466 26475 log.cc:927] Time spent T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.520s	user 0.007s	sys 0.000s
W20260709 12:24:33.168150 26283 consensus_meta.cc:287] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.853s	user 0.002s	sys 0.011s
I20260709 12:24:33.163616 26239 raft_consensus.cc:3055] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:24:33.168673 26283 raft_consensus.cc:515] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:33.169308 26239 raft_consensus.cc:740] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 27 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:24:33.170733 26349 raft_consensus.cc:515] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:24:33.030031 26397 consensus_meta.cc:287] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.809s	user 0.014s	sys 0.003s
I20260709 12:24:32.961871 26322 raft_consensus.cc:493] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:33.062191 26419 consensus_meta.cc:287] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.683s	user 0.010s	sys 0.000s
W20260709 12:24:33.059839 26450 consensus_meta.cc:287] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.725s	user 0.011s	sys 0.002s
W20260709 12:24:33.068835 26164 consensus_meta.cc:287] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.567s	user 0.007s	sys 0.001s
I20260709 12:24:33.053629 26302 raft_consensus.cc:515] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:24:33.079080 26404 consensus_meta.cc:287] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.798s	user 0.009s	sys 0.003s
I20260709 12:24:32.821671 17755 raft_consensus.cc:2468] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 27.
I20260709 12:24:33.095806 26422 consensus_queue.cc:1048] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.003s
I20260709 12:24:33.099844 17652 leader_election.cc:304] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:24:33.121770 17763 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.823s	user 0.042s	sys 0.068s
W20260709 12:24:33.194664 17763 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.896s	user 0.043s	sys 0.068s
W20260709 12:24:33.195168 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 95406.5us
I20260709 12:24:33.195731 17763 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38786 (request call id 3841) took 1347 ms. Trace:
I20260709 12:24:33.196571 17763 rpcz_store.cc:276] 0709 12:24:31.848494 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:32.260231 (+411737us) service_pool.cc:224] Handling call
0709 12:24:32.260585 (+   354us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:32.260779 (+   194us) raft_consensus.cc:1534] Early marking committed up to index 15
0709 12:24:32.298841 (+ 38062us) log.cc:844] Serialized 14 byte log entry
0709 12:24:33.195243 (+896402us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:33.195318 (+    75us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:24:33.195383 (+    65us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:33.195480 (+    97us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:33.195658 (+   178us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4801,"prepare.queue_time_us":37879,"prepare.run_cpu_time_us":64,"prepare.run_wall_time_us":3277,"thread_start_us":895936,"threads_started":1}
I20260709 12:24:33.086354 26359 raft_consensus.cc:2749] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:24:33.091213 26412 consensus_queue.cc:260] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 21.15, Last appended by leader: 15, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:33.124909 17803 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.832s	user 0.019s	sys 0.011s
I20260709 12:24:33.114792 26271 raft_consensus.cc:3060] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:33.114462 26447 raft_consensus.cc:493] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:33.129254 17729 leader_election.cc:304] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
W20260709 12:24:33.122248 25866 consensus_meta.cc:287] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.958s	user 0.012s	sys 0.000s
W20260709 12:24:33.128924 26461 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.520s	user 0.019s	sys 0.061s
I20260709 12:24:33.131963 26399 leader_election.cc:290] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 20 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:24:33.128281 26368 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.958s	user 0.008s	sys 0.068s
W20260709 12:24:33.139271 18334 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.645s	user 0.002s	sys 0.014s
I20260709 12:24:33.141029 17764 raft_consensus.cc:1240] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 23. Current term is 24. Ops: []
W20260709 12:24:33.281158 18334 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.787s	user 0.002s	sys 0.014s
I20260709 12:24:33.148727 26277 leader_election.cc:290] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 22 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:33.149003 17762 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38776 (request call id 3070) took 1076 ms. Trace:
W20260709 12:24:33.281764 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 793299us
I20260709 12:24:33.149737 26340 raft_consensus.cc:2804] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:24:33.209847 26409 raft_consensus.cc:2749] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260709 12:24:33.150271 17833 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35040 (request call id 3385) took 1130 ms. Trace:
I20260709 12:24:33.281738 17762 rpcz_store.cc:276] 0709 12:24:32.072549 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:32.658482 (+585933us) service_pool.cc:224] Handling call
0709 12:24:32.658800 (+   318us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:33.124331 (+465531us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:33.148730 (+ 24399us) spinlock_profiling.cc:243] Waited 170 ms on lock 0x7b54003eb554. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:24:33.148949 (+   219us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":391422336}
I20260709 12:24:33.285153 17833 rpcz_store.cc:276] 0709 12:24:32.020403 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:32.231063 (+210660us) service_pool.cc:224] Handling call
0709 12:24:32.282654 (+ 51591us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:33.127587 (+844933us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:33.149725 (+ 22138us) spinlock_profiling.cc:243] Waited 634 ms on lock 0x7b54000aa5d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:24:33.150181 (+   456us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1458116992}
I20260709 12:24:33.150859 26334 raft_consensus.cc:3060] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:33.152024 26434 leader_election.cc:290] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 23 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:33.154642 26420 raft_consensus.cc:493] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:33.156239 18365 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 2499) took 1122 ms. Trace:
I20260709 12:24:33.157058 26344 leader_election.cc:290] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 24 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:33.134865 26309 raft_consensus.cc:3060] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:33.190336 26164 raft_consensus.cc:515] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:33.142660 26168 raft_consensus.cc:493] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
W20260709 12:24:33.285080 26461 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.690s	user 0.022s	sys 0.072s
I20260709 12:24:32.962191 26414 raft_consensus.cc:493] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:33.134253 26435 raft_consensus.cc:2749] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:24:33.175242 17832 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35040 (request call id 3387) took 1154 ms. Trace:
W20260709 12:24:33.232158 26330 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.665s	user 0.003s	sys 0.016s
I20260709 12:24:33.134749 26441 raft_consensus.cc:2804] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:24:33.143687 26323 raft_consensus.cc:2749] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:24:33.346042 17832 rpcz_store.cc:276] 0709 12:24:32.021370 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:32.284667 (+263297us) service_pool.cc:224] Handling call
0709 12:24:32.285021 (+   354us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:32.285259 (+   238us) raft_consensus.cc:1534] Early marking committed up to index 18
0709 12:24:32.455760 (+170501us) log.cc:844] Serialized 14 byte log entry
0709 12:24:33.134228 (+678468us) log.cc:844] Serialized 14 byte log entry
0709 12:24:33.174385 (+ 40157us) log.cc:844] Serialized 14 byte log entry
0709 12:24:33.174712 (+   327us) log.cc:844] Serialized 14 byte log entry
0709 12:24:33.174899 (+   187us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:33.174961 (+    62us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:24:33.175011 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:33.175073 (+    62us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:33.175187 (+   114us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":13365,"prepare.queue_time_us":634898,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":78,"raft.queue_time_us":22507,"raft.run_cpu_time_us":619,"raft.run_wall_time_us":18571,"spinlock_wait_cycles":11305344,"thread_start_us":848020,"threads_started":2,"wal-append.queue_time_us":866218}
I20260709 12:24:33.202939 17836 raft_consensus.cc:1240] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 18. Current term is 19. Ops: []
I20260709 12:24:33.207489 17757 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38776 (request call id 3063) took 1363 ms. Trace:
W20260709 12:24:33.209265 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 1.01567e+06us
I20260709 12:24:33.188055 26419 raft_consensus.cc:2749] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:24:33.211176 17835 raft_consensus.cc:1240] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 26. Current term is 27. Ops: []
W20260709 12:24:33.211268 17730 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:33.353147 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:33.353783 17836 raft_consensus.cc:3060] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:33.233871 26412 raft_consensus.cc:3060] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Advancing to term 22
W20260709 12:24:33.221971 17759 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.534s	user 0.000s	sys 0.003s
W20260709 12:24:33.354959 17759 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.709s	user 0.000s	sys 0.004s
W20260709 12:24:33.224555 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 95402us
I20260709 12:24:33.356215 17729 leader_election.cc:304] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
W20260709 12:24:33.231705 17682 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.571s	user 0.011s	sys 0.002s
W20260709 12:24:33.232540 17689 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.656s	user 0.014s	sys 0.004s
W20260709 12:24:33.357946 17682 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.698s	user 0.012s	sys 0.002s
W20260709 12:24:33.232266 17838 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.564s	user 0.017s	sys 0.019s
W20260709 12:24:33.359490 17838 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.691s	user 0.017s	sys 0.019s
I20260709 12:24:33.149389 17683 raft_consensus.cc:2393] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 22: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
W20260709 12:24:33.280208 18333 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.644s	user 0.007s	sys 0.011s
I20260709 12:24:33.288149 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
W20260709 12:24:33.360566 18333 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.726s	user 0.007s	sys 0.011s
I20260709 12:24:33.360921 17763 raft_consensus.cc:3060] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Advancing to term 23
W20260709 12:24:33.290148 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 361514us
I20260709 12:24:33.184306 26397 raft_consensus.cc:515] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:33.190373 26450 raft_consensus.cc:515] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:33.170524 26239 consensus_queue.cc:260] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 27.15, Last appended by leader: 15, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:33.233167 26283 leader_election.cc:290] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 23 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:33.194437 26322 raft_consensus.cc:515] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:33.282022 26452 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.647s	user 0.016s	sys 0.001s
I20260709 12:24:33.315675 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 22 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:33.316581 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:33.319568 26487 raft_consensus.cc:493] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:33.327487 26473 log.cc:927] Time spent T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.155s	user 0.001s	sys 0.000s
I20260709 12:24:33.328488 26325 raft_consensus.cc:493] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:33.330596 26416 consensus_queue.cc:1059] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true }, Status: INVALID_TERM, Last received: 24.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:24:33.333175 26349 leader_election.cc:290] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 28 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:33.334954 26302 leader_election.cc:290] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 27 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:33.341143 17837 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35040 (request call id 3389) took 1319 ms. Trace:
I20260709 12:24:33.428714 17837 rpcz_store.cc:276] 0709 12:24:32.022326 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:32.635280 (+612954us) service_pool.cc:224] Handling call
0709 12:24:32.635575 (+   295us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:32.635783 (+   208us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:32.635806 (+    23us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:32.645975 (+ 10169us) log.cc:844] Serialized 26 byte log entry
0709 12:24:33.121242 (+475267us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:24:33.121300 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:33.121397 (+    97us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:33.340842 (+219445us) raft_consensus.cc:1685] finished
0709 12:24:33.340872 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:33.341067 (+   195us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":474843,"threads_started":1,"wal-append.queue_time_us":674048}
I20260709 12:24:33.223403 26446 leader_election.cc:290] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 22 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:33.283957 26234 raft_consensus.cc:2804] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Leader pre-election won for term 28
W20260709 12:24:33.175649 17654 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.661s	user 0.044s	sys 0.041s
W20260709 12:24:33.433768 17654 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.920s	user 0.044s	sys 0.041s
I20260709 12:24:33.348793 26414 raft_consensus.cc:515] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:33.200516 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:33.349390 17803 leader_election.cc:304] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:33.211820 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
W20260709 12:24:33.358047 17689 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.782s	user 0.014s	sys 0.004s
I20260709 12:24:33.436498 17755 raft_consensus.cc:3060] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:33.438208 17836 raft_consensus.cc:2468] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 18.
I20260709 12:24:33.367564 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
W20260709 12:24:33.367861 17802 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:33.369858 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 22 candidate_status { last_received { term: 21 index: 18 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:33.371745 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
W20260709 12:24:33.439276 17802 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:33.442124 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:33.439499 17832 raft_consensus.cc:3060] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Advancing to term 23
I20260709 12:24:33.383347 17764 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38776 (request call id 3052) took 1606 ms. Trace:
I20260709 12:24:33.388407 17833 raft_consensus.cc:3060] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:33.347677 26441 raft_consensus.cc:493] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:33.552767 26441 raft_consensus.cc:3060] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:24:33.553068 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 203750us
I20260709 12:24:33.553835 17803 leader_election.cc:304] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:33.554443 26164 leader_election.cc:290] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 27 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:33.555215 26164 raft_consensus.cc:2749] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:24:33.387856 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:33.556052 26164 raft_consensus.cc:2749] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:24:33.560211 26368 raft_consensus.cc:3055] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:24:33.560760 26368 raft_consensus.cc:740] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:24:33.561208 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:33.561954 17835 raft_consensus.cc:2368] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for earlier term 26. Current term is 27.
I20260709 12:24:33.562819 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:33.563135 17802 leader_election.cc:304] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
W20260709 12:24:33.391530 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 363726us
I20260709 12:24:33.399559 18362 raft_consensus.cc:1240] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 20. Current term is 21. Ops: []
I20260709 12:24:33.244560 26447 raft_consensus.cc:515] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:33.564182 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 24 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:33.564853 17836 raft_consensus.cc:1240] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 18. Current term is 19. Ops: []
I20260709 12:24:33.565271 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:33.566136 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:33.566512 26447 leader_election.cc:290] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:24:33.391661 26452 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.786s	user 0.017s	sys 0.002s
I20260709 12:24:33.370950 26261 raft_consensus.cc:2749] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:24:33.406912 17838 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35040 (request call id 3388) took 1385 ms. Trace:
I20260709 12:24:33.567397 17833 raft_consensus.cc:2468] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 27.
I20260709 12:24:33.335624 26420 raft_consensus.cc:515] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:33.364363 26471 raft_consensus.cc:2749] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:24:33.408833 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:33.346606 26168 raft_consensus.cc:515] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:33.568243 17756 raft_consensus.cc:1240] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 25 LEADER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 24. Current term is 25. Ops: [24.17-24.17]
I20260709 12:24:33.406778 26239 raft_consensus.cc:3060] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:24:33.416694 26325 raft_consensus.cc:515] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:33.575166 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 23 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:33.575932 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:33.421187 26450 leader_election.cc:290] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 21 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:33.422709 26397 leader_election.cc:290] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 26 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:33.396940 26206 raft_consensus.cc:493] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:33.576668 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:33.577322 18368 raft_consensus.cc:2393] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 23: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:24:33.577087 26206 raft_consensus.cc:515] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:33.396826 26419 raft_consensus.cc:493] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:33.373423 26483 raft_consensus.cc:493] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:33.578249 26419 raft_consensus.cc:515] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:33.578368 26483 raft_consensus.cc:515] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:33.209575 26404 raft_consensus.cc:515] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Starting leader election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:24:33.577337 17682 raft_consensus.cc:2368] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for earlier term 26. Current term is 27.
I20260709 12:24:33.292428 18365 rpcz_store.cc:276] 0709 12:24:32.033837 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:32.033970 (+   133us) service_pool.cc:224] Handling call
0709 12:24:32.034252 (+   282us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:32.062445 (+ 28193us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:32.062478 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:32.062893 (+   415us) log.cc:844] Serialized 26 byte log entry
0709 12:24:32.121785 (+ 58892us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:24:32.121849 (+    64us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:32.121952 (+   103us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:32.659501 (+537549us) spinlock_profiling.cc:243] Waited 7.89 ms on lock 0x7b4400022b4c. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b47459c85 00007f7b4745a6a2 00007f7b4a1cc1db 00007f7b4a1c7aaf 00007f7b4a1bb86e 00007f7b4a1babb6 00007f7b4a1ba7d8 00007f7b4a1c4221 00007f7b4a1c41d9 00007f7b4a1c4169
0709 12:24:33.155949 (+496448us) raft_consensus.cc:1685] finished
0709 12:24:33.155972 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:33.156178 (+   206us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":27840,"raft.queue_time_us":28861,"raft.run_cpu_time_us":67,"raft.run_wall_time_us":67,"spinlock_wait_cycles":18150784,"thread_start_us":58467,"threads_started":1,"wal-append.queue_time_us":58811}
I20260709 12:24:33.348944 17757 rpcz_store.cc:276] 0709 12:24:31.844150 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:32.196613 (+352463us) service_pool.cc:224] Handling call
0709 12:24:32.224429 (+ 27816us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:32.241734 (+ 17305us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:32.241764 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:32.261878 (+ 20114us) log.cc:844] Serialized 26 byte log entry
0709 12:24:32.390297 (+128419us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:24:32.431553 (+ 41256us) log.cc:844] Serialized 14 byte log entry
0709 12:24:32.576519 (+144966us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:32.576592 (+    73us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:33.165821 (+589229us) spinlock_profiling.cc:243] Waited 31.6 ms on lock 0x7b4400022b4c. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b47459c85 00007f7b4745a6a2 00007f7b4a1cc1db 00007f7b4a1c7aaf 00007f7b4a1bb86e 00007f7b4a1babb6 00007f7b4a1ba7d8 00007f7b4a1c4221 00007f7b4a1c41d9 00007f7b4a1c4169
0709 12:24:33.207224 (+ 41403us) raft_consensus.cc:1685] finished
0709 12:24:33.207307 (+    83us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:33.207436 (+   129us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":161847,"prepare.queue_time_us":38695,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":39,"raft.queue_time_us":162298,"raft.run_cpu_time_us":203,"raft.run_wall_time_us":12160,"spinlock_wait_cycles":72735616,"thread_start_us":127757,"threads_started":1,"wal-append.queue_time_us":383991}
I20260709 12:24:33.580128 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:33.580914 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:33.581552 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:33.567610 17838 rpcz_store.cc:276] 0709 12:24:32.021829 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:32.630192 (+608363us) service_pool.cc:224] Handling call
0709 12:24:32.630492 (+   300us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:32.630674 (+   182us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 12:24:33.359996 (+729322us) log.cc:844] Serialized 14 byte log entry
0709 12:24:33.406583 (+ 46587us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:33.406640 (+    57us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:24:33.406686 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:33.406750 (+    64us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:33.406856 (+   106us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":11434,"prepare.queue_time_us":338103,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":41,"raft.queue_time_us":1927,"raft.run_cpu_time_us":108,"raft.run_wall_time_us":109,"thread_start_us":737761,"threads_started":2,"wal-append.queue_time_us":183376}
I20260709 12:24:33.582192 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:33.582832 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 24 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:33.443416 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 22 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
W20260709 12:24:33.350745 26330 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.783s	user 0.003s	sys 0.017s
I20260709 12:24:33.444834 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:33.444862 17764 rpcz_store.cc:276] 0709 12:24:31.777004 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:32.100603 (+323599us) service_pool.cc:224] Handling call
0709 12:24:32.100852 (+   249us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:33.281284 (+1180432us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:33.383000 (+101716us) spinlock_profiling.cc:243] Waited 1.04 s on lock 0x7b5400310e54. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:24:33.383268 (+   268us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2391748992}
I20260709 12:24:33.584043 17682 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d"
dest_uuid: "0da169ff82674dc1947055ce43de68a3"
mode: GRACEFUL
new_leader_uuid: "e063541542a94969898868bf6eb5ce1b"
 from {username='slave'} at 127.0.0.1:54540
I20260709 12:24:33.584396 17760 raft_consensus.cc:1240] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 25. Current term is 26. Ops: [25.15-25.15]
I20260709 12:24:33.584722 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 28 candidate_status { last_received { term: 27 index: 21 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:33.585213 18371 raft_consensus.cc:3060] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:33.585726 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:33.586854 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:33.490541 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:33.588317 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 23 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:33.589537 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:33.590039 26168 leader_election.cc:290] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:24:33.590204 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 154647us
I20260709 12:24:33.590662 17688 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 22.
I20260709 12:24:33.590822 18364 raft_consensus.cc:3060] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:33.431686 17763 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 23.
I20260709 12:24:33.568022 26499 raft_consensus.cc:493] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:33.433429 26234 raft_consensus.cc:493] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:33.592895 26234 raft_consensus.cc:3060] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:24:33.592702 26499 raft_consensus.cc:515] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:33.534233 17727 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:33.407433 26487 raft_consensus.cc:515] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:33.596560 18374 raft_consensus.cc:1240] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 21. Current term is 22. Ops: []
I20260709 12:24:33.597064 26450 raft_consensus.cc:493] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:33.597471 26487 leader_election.cc:290] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:33.597872 18365 raft_consensus.cc:1275] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 21 index: 15. Preceding OpId from leader: term: 23 index: 16. (index mismatch)
I20260709 12:24:33.597808 26450 raft_consensus.cc:515] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:33.599601 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:33.600734 26450 leader_election.cc:290] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:33.600924 26404 leader_election.cc:290] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 26 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:33.603012 18367 raft_consensus.cc:3060] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Advancing to term 23
W20260709 12:24:33.524642 26477 log.cc:927] Time spent T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.099s	user 0.000s	sys 0.000s
I20260709 12:24:33.541743 26414 leader_election.cc:290] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:33.541530 26421 raft_consensus.cc:515] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:33.605885 17761 raft_consensus.cc:1240] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 24. Current term is 25. Ops: []
W20260709 12:24:33.542392 26490 log.cc:927] Time spent T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.102s	user 0.000s	sys 0.001s
I20260709 12:24:33.446183 25784 consensus_queue.cc:1059] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 23.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.089s
I20260709 12:24:33.544021 26281 raft_consensus.cc:493] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:33.544549 26322 leader_election.cc:290] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:33.544713 17578 catalog_manager.cc:5697] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb reported cstate change: term changed from 11 to 20, leader changed from 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) to e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66). New cstate: current_term: 20 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: UNKNOWN } } }
I20260709 12:24:33.546149 17755 raft_consensus.cc:2468] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 27.
W20260709 12:24:33.547986 26481 log.cc:927] Time spent T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.138s	user 0.000s	sys 0.001s
I20260709 12:24:33.548086 26271 raft_consensus.cc:515] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:33.548686 17689 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37396 (request call id 3164) took 1343 ms. Trace:
I20260709 12:24:33.549364 26454 raft_consensus.cc:493] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:33.396750 26277 consensus_queue.cc:1059] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 20.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.049s
I20260709 12:24:33.585223 26483 leader_election.cc:290] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:33.640079 26271 leader_election.cc:290] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 24 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:33.641676 17681 raft_consensus.cc:1240] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 23. Current term is 24. Ops: []
I20260709 12:24:33.644030 26454 raft_consensus.cc:515] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:33.644887 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:33.645810 17687 raft_consensus.cc:2393] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 23: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:24:33.646616 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:33.646962 26454 leader_election.cc:290] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:33.648198 17682 raft_consensus.cc:606] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Received request to transfer leadership to TS e063541542a94969898868bf6eb5ce1b
I20260709 12:24:33.551867 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:33.649412 17758 raft_consensus.cc:2468] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 19.
I20260709 12:24:33.552331 26443 raft_consensus.cc:493] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:33.566805 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:33.576022 26325 leader_election.cc:290] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:33.419466 26283 raft_consensus.cc:493] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:33.578776 26420 leader_election.cc:290] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:33.592989 17652 leader_election.cc:304] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:24:33.587841 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:33.439344 26295 raft_consensus.cc:3055] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:24:33.608129 17803 leader_election.cc:304] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:24:33.405557 18334 leader_election.cc:304] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
W20260709 12:24:33.689487 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 283988us
I20260709 12:24:33.566809 17835 raft_consensus.cc:3060] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:33.690218 18334 leader_election.cc:304] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:24:33.691009 26483 raft_consensus.cc:2804] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:24:33.691382 26483 raft_consensus.cc:493] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:33.691764 26483 raft_consensus.cc:3060] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:33.583503 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:33.693292 17757 raft_consensus.cc:2468] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 29.
I20260709 12:24:33.694190 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:33.695246 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:33.695904 17757 raft_consensus.cc:2368] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 22. Current term is 23.
I20260709 12:24:33.696566 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:33.613412 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:33.614023 17686 raft_consensus.cc:2468] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 20.
I20260709 12:24:33.615095 18364 raft_consensus.cc:2468] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 26.
I20260709 12:24:33.698853 17755 raft_consensus.cc:2393] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 26: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:24:33.616165 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 23 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:33.617964 17832 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 23.
I20260709 12:24:33.699626 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:33.699283 26483 raft_consensus.cc:515] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:33.700198 17755 raft_consensus.cc:3060] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:33.702008 26483 leader_election.cc:290] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 27 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:33.701864 17835 raft_consensus.cc:2468] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 23.
I20260709 12:24:33.619674 18371 raft_consensus.cc:2468] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 22.
I20260709 12:24:33.619856 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:33.623399 26281 raft_consensus.cc:515] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:33.624902 25784 raft_consensus.cc:3055] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:24:33.704129 25784 raft_consensus.cc:740] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 23 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:24:33.625571 26334 raft_consensus.cc:2749] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Leader election lost for term 25. Reason: Vote denied by peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071) with higher term. Message: Invalid argument: T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 25. Current term is 26.
I20260709 12:24:33.704986 25784 consensus_queue.cc:260] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 23.17, Last appended by leader: 17, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:33.705467 26281 leader_election.cc:290] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:33.706214 25784 raft_consensus.cc:3060] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:33.706292 26281 raft_consensus.cc:493] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:33.626581 26421 leader_election.cc:290] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 25 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:33.626581 26419 leader_election.cc:290] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:33.629797 18367 raft_consensus.cc:2468] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 23.
I20260709 12:24:33.547329 26354 raft_consensus.cc:515] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:33.628232 17689 rpcz_store.cc:276] 0709 12:24:32.205581 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:32.213778 (+  8197us) service_pool.cc:224] Handling call
0709 12:24:32.219039 (+  5261us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:32.219378 (+   339us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:32.219406 (+    28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:32.219869 (+   463us) log.cc:844] Serialized 26 byte log entry
0709 12:24:32.575869 (+356000us) spinlock_profiling.cc:243] Waited 10.6 ms on lock 0x7b18003b0440. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b47515e07 00007f7b503709a9 00007f7b4753269b 00007f7b4753093f 00007f7b47532c02 00007f7b4a1ba20b 00007f7b4a1bfd23 00007f7b4a1c00d1 00007f7b4a5dba0f
0709 12:24:33.436619 (+860750us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:24:33.436657 (+    38us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:33.436722 (+    65us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:33.548340 (+111618us) raft_consensus.cc:1685] finished
0709 12:24:33.548362 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:33.548609 (+   247us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":995,"raft.queue_time_us":6579,"raft.run_cpu_time_us":116,"raft.run_wall_time_us":4529,"spinlock_wait_cycles":24307456,"thread_start_us":860558,"threads_started":1,"wal-append.queue_time_us":1177378}
I20260709 12:24:33.633845 26261 consensus_queue.cc:1048] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:24:33.636168 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 25 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:33.637152 26206 leader_election.cc:290] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:33.648785 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:33.650360 17654 leader_election.cc:304] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:33.648789 17682 raft_consensus.cc:614] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:24:33.573004 25866 raft_consensus.cc:2804] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:24:33.684355 17837 raft_consensus.cc:2410] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for term 28 because replica has last-logged OpId of term: 27 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 26 index: 14.
I20260709 12:24:33.561976 26368 consensus_queue.cc:260] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 23.17, Last appended by leader: 17, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:33.681262 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:33.726075 17764 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 22.
I20260709 12:24:33.728518 26340 raft_consensus.cc:493] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:33.729225 26340 raft_consensus.cc:3060] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Advancing to term 20
W20260709 12:24:33.730173 26436 consensus_meta.cc:287] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.906s	user 0.011s	sys 0.001s
I20260709 12:24:33.730634 26436 raft_consensus.cc:515] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:33.731827 17684 raft_consensus.cc:2368] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 22. Current term is 23.
I20260709 12:24:33.732496 26499 leader_election.cc:290] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:33.734649 17759 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38776 (request call id 3053) took 1956 ms. Trace:
I20260709 12:24:33.714993 26479 raft_consensus.cc:2804] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:24:33.735083 17759 rpcz_store.cc:276] 0709 12:24:31.778349 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:32.102164 (+323815us) service_pool.cc:224] Handling call
0709 12:24:32.102713 (+   549us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:32.431463 (+328750us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:32.431494 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:32.433222 (+  1728us) log.cc:844] Serialized 26 byte log entry
0709 12:24:33.355406 (+922184us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:24:33.355474 (+    68us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:33.355581 (+   107us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:33.734414 (+378833us) raft_consensus.cc:1685] finished
0709 12:24:33.734438 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:33.734585 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":13356,"raft.queue_time_us":5567,"raft.run_cpu_time_us":119,"raft.run_wall_time_us":5016,"thread_start_us":1035852,"threads_started":2,"wal-append.queue_time_us":1024645}
I20260709 12:24:33.736292 17578 catalog_manager.cc:5697] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb reported cstate change: term changed from 9 to 16. New cstate: current_term: 16 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } }
I20260709 12:24:33.738523 26354 leader_election.cc:290] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 26 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:24:33.739198 26377 consensus_meta.cc:287] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.626s	user 0.006s	sys 0.004s
I20260709 12:24:33.710459 26420 raft_consensus.cc:2804] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Leader pre-election won for term 30
W20260709 12:24:33.741216 26501 log.cc:927] Time spent T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.322s	user 0.002s	sys 0.000s
I20260709 12:24:33.741564 26420 raft_consensus.cc:493] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:33.741952 26420 raft_consensus.cc:3060] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:24:33.706866 26281 raft_consensus.cc:515] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:33.717589 26438 raft_consensus.cc:3055] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:24:33.744876 26438 raft_consensus.cc:740] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
W20260709 12:24:33.746254 26480 log.cc:927] Time spent T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.467s	user 0.008s	sys 0.003s
I20260709 12:24:33.682744 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:33.747555 18366 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 2500) took 1462 ms. Trace:
I20260709 12:24:33.748064 26368 raft_consensus.cc:3060] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:24:33.748790 26350 consensus_queue.cc:1175] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "334a4e1f59944544b3b3ff36616b57da" responder_term: 23 status { last_received { term: 23 index: 17 } last_committed_idx: 17 last_received_current_leader { term: 23 index: 17 } }
I20260709 12:24:33.682411 26443 raft_consensus.cc:515] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:33.678787 26283 raft_consensus.cc:515] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:33.762045 17831 raft_consensus.cc:2468] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 26.
I20260709 12:24:33.763934 17761 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38786 (request call id 3843) took 1796 ms. Trace:
I20260709 12:24:33.764041 25866 raft_consensus.cc:697] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 27 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:24:33.764420 17761 rpcz_store.cc:276] 0709 12:24:31.967968 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:32.583435 (+615467us) service_pool.cc:224] Handling call
0709 12:24:32.640723 (+ 57288us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:33.606279 (+965556us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:33.763637 (+157358us) spinlock_profiling.cc:243] Waited 965 ms on lock 0x7b54004155d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:24:33.763854 (+   217us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2219276672}
I20260709 12:24:33.765486 26283 leader_election.cc:290] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:33.679178 17760 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38776 (request call id 3069) took 1610 ms. Trace:
I20260709 12:24:33.765944 26441 raft_consensus.cc:515] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:33.656864 17756 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38786 (request call id 3842) took 1808 ms. Trace:
I20260709 12:24:33.748117 18366 rpcz_store.cc:276] 0709 12:24:32.285079 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:32.285221 (+   142us) service_pool.cc:224] Handling call
0709 12:24:32.285667 (+   446us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:32.285937 (+   270us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:32.285976 (+    39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:32.286582 (+   606us) log.cc:844] Serialized 26 byte log entry
0709 12:24:33.140220 (+853638us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:24:33.140273 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:33.140352 (+    79us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:33.745604 (+605252us) spinlock_profiling.cc:243] Waited 2.92 ms on lock 0x7b4400022b4c. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b47459c85 00007f7b4745a6a2 00007f7b4a1cc1db 00007f7b4a1c7aaf 00007f7b4a1bb86e 00007f7b4a1babb6 00007f7b4a1ba7d8 00007f7b4a1c4221 00007f7b4a1c41d9 00007f7b4a1c4169
0709 12:24:33.747164 (+  1560us) raft_consensus.cc:1685] finished
0709 12:24:33.747198 (+    34us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:33.747475 (+   277us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":6724864,"thread_start_us":853254,"threads_started":1,"wal-append.queue_time_us":905156}
I20260709 12:24:33.658193 18362 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59886 (request call id 2278) took 1315 ms. Trace:
I20260709 12:24:33.688733 18374 raft_consensus.cc:1240] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 23. Current term is 24. Ops: []
I20260709 12:24:33.767867 18362 rpcz_store.cc:276] 0709 12:24:32.342549 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:32.342680 (+   131us) service_pool.cc:224] Handling call
0709 12:24:32.342997 (+   317us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:33.564264 (+1221267us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:33.657846 (+ 93582us) spinlock_profiling.cc:243] Waited 1.06 s on lock 0x7b5400570154. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:24:33.658122 (+   276us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2429333376}
W20260709 12:24:33.571342 17801 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:33.769191 17763 raft_consensus.cc:1240] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 26. Current term is 27. Ops: []
W20260709 12:24:33.688545 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 95640.1us
W20260709 12:24:33.611276 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 48217us
I20260709 12:24:33.766273 17760 rpcz_store.cc:276] 0709 12:24:32.069003 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:32.493610 (+424607us) service_pool.cc:224] Handling call
0709 12:24:32.493889 (+   279us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:33.584833 (+1090944us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:33.678887 (+ 94054us) spinlock_profiling.cc:243] Waited 1.09 s on lock 0x7b54003eb054. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:24:33.679114 (+   227us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2497571584}
W20260709 12:24:33.755245 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 104976us
I20260709 12:24:33.706878 17757 raft_consensus.cc:1240] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 27. Current term is 28. Ops: [27.16-27.16]
I20260709 12:24:33.771895 17652 leader_election.cc:304] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:24:33.707623 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
W20260709 12:24:33.708678 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 152783us
I20260709 12:24:33.710282 17680 raft_consensus.cc:1240] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 21. Current term is 22. Ops: [21.18-21.18]
I20260709 12:24:33.766512 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:33.711627 17729 leader_election.cc:304] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:33.712810 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:33.773504 18361 raft_consensus.cc:3060] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:33.773794 18367 raft_consensus.cc:2468] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 20.
I20260709 12:24:33.712757 17755 raft_consensus.cc:2468] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 27.
I20260709 12:24:33.715327 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 25 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:33.710139 17683 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44898 (request call id 1663) took 1384 ms. Trace:
I20260709 12:24:33.774705 18363 raft_consensus.cc:2468] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 24.
I20260709 12:24:33.774631 17683 rpcz_store.cc:276] 0709 12:24:32.325727 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:32.368025 (+ 42298us) service_pool.cc:224] Handling call
0709 12:24:33.709939 (+1341914us) spinlock_profiling.cc:243] Waited 761 ms on lock 0x7b54003467d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:24:33.710084 (+   145us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1750220160}
I20260709 12:24:33.707470 26325 raft_consensus.cc:2804] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:24:33.717024 18333 leader_election.cc:304] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:24:33.771689 17802 leader_election.cc:304] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
W20260709 12:24:33.778554 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 61595.2us
I20260709 12:24:33.758169 17578 catalog_manager.cc:5697] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb reported cstate change: term changed from 13 to 19, leader changed from e063541542a94969898868bf6eb5ce1b (127.16.29.67) to e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66). New cstate: current_term: 19 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: UNKNOWN } } }
I20260709 12:24:33.746637 26281 leader_election.cc:290] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:33.759060 26234 raft_consensus.cc:515] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:33.766758 17756 rpcz_store.cc:276] 0709 12:24:31.848817 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:32.382993 (+534176us) service_pool.cc:224] Handling call
0709 12:24:32.383331 (+   338us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:33.574214 (+1190883us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:33.656482 (+ 82268us) spinlock_profiling.cc:243] Waited 1.18 s on lock 0x7b54004150d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:24:33.656782 (+   300us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2724444544}
I20260709 12:24:33.769819 17801 leader_election.cc:400] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 22 election: Vote denied by peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651) with higher term. Message: Invalid argument: T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 22. Current term is 23.
I20260709 12:24:33.798188 17801 leader_election.cc:403] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 22 election: Cancelling election due to peer responding with higher term
I20260709 12:24:33.697265 26450 consensus_queue.cc:1059] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 21.15, Next index: 16, Last known committed idx: 14, Time since last communication: 0.000s
I20260709 12:24:33.689076 26295 raft_consensus.cc:740] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:24:33.774065 17803 leader_election.cc:304] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:33.720199 18368 raft_consensus.cc:3060] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:33.778319 17758 raft_consensus.cc:2368] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 26. Current term is 27.
W20260709 12:24:33.778889 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 67344us
I20260709 12:24:33.783203 18333 leader_election.cc:304] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:33.758482 26277 raft_consensus.cc:2804] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:24:33.790764 26208 raft_consensus.cc:493] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:33.792223 26302 raft_consensus.cc:2804] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:24:33.825883 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:33.792682 26334 consensus_queue.cc:1059] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 23.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20260709 12:24:33.794716 17652 leader_election.cc:304] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:24:33.797217 18361 raft_consensus.cc:2468] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 18.
I20260709 12:24:33.722337 26164 raft_consensus.cc:493] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:33.828078 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:33.798571 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 28 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:33.784937 26283 raft_consensus.cc:493] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:24:33.799924 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 25 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:33.800190 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:33.800784 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 25 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:33.833509 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 26 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:33.833814 17835 raft_consensus.cc:2468] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 24.
I20260709 12:24:33.834112 17830 raft_consensus.cc:3060] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:33.801718 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:33.834797 17832 raft_consensus.cc:2468] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 20.
I20260709 12:24:33.803947 17801 leader_election.cc:304] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:33.807833 17727 leader_election.cc:400] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 26 election: Vote denied by peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837) with higher term. Message: Invalid argument: T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for earlier term 26. Current term is 27.
I20260709 12:24:33.836589 17727 leader_election.cc:403] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 26 election: Cancelling election due to peer responding with higher term
I20260709 12:24:33.793617 26441 leader_election.cc:290] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 27 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:33.809260 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:33.809669 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:33.797258 26414 raft_consensus.cc:2804] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:24:33.809846 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:33.810314 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 25 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:24:33.839018 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 44412.9us
I20260709 12:24:33.839272 17761 raft_consensus.cc:2468] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 22.
I20260709 12:24:33.811007 26325 raft_consensus.cc:697] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 18 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:24:33.814066 26450 raft_consensus.cc:3055] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:24:33.814764 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 22 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:33.849954 17681 raft_consensus.cc:2410] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 22 because replica has last-logged OpId of term: 20 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 14.
I20260709 12:24:33.752867 26443 leader_election.cc:290] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:33.815419 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:33.739639 26377 raft_consensus.cc:515] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:33.816926 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:33.818259 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 26 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:33.745664 26438 consensus_queue.cc:260] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 22.15, Last appended by leader: 15, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:33.871024 18362 raft_consensus.cc:3060] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Advancing to term 26
I20260709 12:24:33.826241 26419 raft_consensus.cc:2749] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:24:33.736414 26479 raft_consensus.cc:697] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 19 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:24:33.824848 18333 leader_election.cc:304] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:33.806172 26354 consensus_queue.cc:1059] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 26.15, Next index: 16, Last known committed idx: 13, Time since last communication: 0.060s
I20260709 12:24:33.796540 26350 raft_consensus.cc:2804] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:24:33.830264 17836 raft_consensus.cc:2468] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 22.
I20260709 12:24:33.830973 17683 raft_consensus.cc:2410] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 28 because replica has last-logged OpId of term: 26 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 15.
I20260709 12:24:33.797798 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:33.794195 26483 raft_consensus.cc:2804] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:24:33.832422 17762 raft_consensus.cc:1240] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 27. Current term is 28. Ops: []
I20260709 12:24:33.837393 17727 leader_election.cc:304] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:33.797364 26446 raft_consensus.cc:493] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:33.808884 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:33.810555 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:33.849678 26283 raft_consensus.cc:515] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:33.839341 17759 raft_consensus.cc:2468] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 24.
I20260709 12:24:33.843511 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:33.841722 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 22 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:33.848724 17757 raft_consensus.cc:2393] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 23: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:24:33.850576 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:33.913797 17829 raft_consensus.cc:2468] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 21.
I20260709 12:24:33.914042 18366 raft_consensus.cc:2468] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 21.
I20260709 12:24:33.848198 26164 raft_consensus.cc:515] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:33.854923 26436 leader_election.cc:290] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 23 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:33.765125 25866 consensus_queue.cc:237] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 25.15, Last appended by leader: 15, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:33.854624 26295 consensus_queue.cc:260] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 25.15, Last appended by leader: 15, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:24:33.823742 26498 raft_consensus.cc:2749] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Leader election lost for term 22. Reason: Vote denied by peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651) with higher term. Message: Invalid argument: T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 22. Current term is 23.
I20260709 12:24:33.829036 26208 raft_consensus.cc:515] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:33.816294 26277 raft_consensus.cc:493] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:24:33.858130 26234 leader_election.cc:290] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 28 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:33.887490 26302 raft_consensus.cc:697] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 20 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
W20260709 12:24:33.892484 26168 consensus_queue.cc:1175] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "334a4e1f59944544b3b3ff36616b57da" responder_term: 22 status { last_received { term: 22 index: 15 } last_committed_idx: 14 last_received_current_leader { term: 22 index: 15 } }
I20260709 12:24:33.821270 17729 leader_election.cc:304] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:24:33.877249 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 73426.7us
W20260709 12:24:33.932332 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 111196us
I20260709 12:24:33.932845 17801 leader_election.cc:304] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:33.826673 17838 raft_consensus.cc:2468] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 27.
W20260709 12:24:33.888504 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 63732.6us
I20260709 12:24:33.934877 18333 leader_election.cc:304] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:33.936537 18333 leader_election.cc:304] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:33.938088 17831 raft_consensus.cc:2468] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 19.
I20260709 12:24:33.928290 17578 catalog_manager.cc:5697] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb reported cstate change: term changed from 17 to 24, leader changed from 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) to e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66). New cstate: current_term: 24 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:24:33.949982 26208 leader_election.cc:290] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:33.928586 26277 raft_consensus.cc:3060] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:24:33.952055 26234 raft_consensus.cc:2804] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:24:33.952596 26234 raft_consensus.cc:493] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:33.952970 26234 raft_consensus.cc:3060] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:33.956311 26419 raft_consensus.cc:2749] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
W20260709 12:24:33.957141 26271 consensus_queue.cc:1175] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "334a4e1f59944544b3b3ff36616b57da" responder_term: 25 status { last_received { term: 25 index: 15 } last_committed_idx: 14 last_received_current_leader { term: 25 index: 15 } }
I20260709 12:24:33.960338 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 28 candidate_status { last_received { term: 27 index: 21 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:33.961279 26377 leader_election.cc:290] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 26 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:33.962127 26419 raft_consensus.cc:2749] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:24:33.961372 18333 leader_election.cc:304] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:33.962740 26325 consensus_queue.cc:237] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 16.13, Last appended by leader: 13, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:33.963361 26420 raft_consensus.cc:515] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:33.964552 18333 leader_election.cc:304] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:24:33.965337 26419 raft_consensus.cc:2749] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:24:33.965574 26420 leader_election.cc:290] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 30 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:33.966760 18333 leader_election.cc:304] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:33.950600 17578 catalog_manager.cc:5697] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb reported cstate change: term changed from 11 to 20, leader changed from 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) to e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66). New cstate: current_term: 20 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: UNKNOWN } } }
I20260709 12:24:33.848731 26344 raft_consensus.cc:2749] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Leader election lost for term 26. Reason: Vote denied by peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837) with higher term. Message: Invalid argument: T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for earlier term 26. Current term is 27.
W20260709 12:24:33.904819 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 67539.4us
I20260709 12:24:33.826077 26281 consensus_queue.cc:1059] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 20.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.062s
I20260709 12:24:33.909574 18362 raft_consensus.cc:2468] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 26.
I20260709 12:24:33.909699 18368 raft_consensus.cc:2468] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 25.
I20260709 12:24:33.866185 26450 raft_consensus.cc:740] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:24:33.870265 26416 raft_consensus.cc:2804] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Leader election won for term 26
W20260709 12:24:33.842924 17585 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:37172 (request call id 259) took 15858 ms (client timeout 15000 ms). Trace:
I20260709 12:24:33.865722 17833 raft_consensus.cc:2393] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 26: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:24:33.893157 26488 raft_consensus.cc:2749] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:24:33.917421 26446 raft_consensus.cc:515] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:33.919833 26454 raft_consensus.cc:3055] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:24:33.969995 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:34.009649 18374 raft_consensus.cc:3060] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:33.973261 26330 raft_consensus.cc:2749] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:24:33.972329 18333 leader_election.cc:304] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:24:34.057185 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 84912.8us
I20260709 12:24:33.899693 17654 leader_election.cc:304] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b
W20260709 12:24:34.061179 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 161552us
I20260709 12:24:34.064127 17728 leader_election.cc:304] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:33.917264 26483 raft_consensus.cc:697] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 21 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:24:33.975301 26421 raft_consensus.cc:493] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:33.976097 26377 raft_consensus.cc:493] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:33.976202 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
W20260709 12:24:33.980530 26515 log.cc:927] Time spent T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.061s	user 0.000s	sys 0.001s
I20260709 12:24:33.920461 26350 raft_consensus.cc:697] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
W20260709 12:24:33.989791 17729 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:34.088205 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:34.088897 18361 raft_consensus.cc:2368] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 22. Current term is 23.
I20260709 12:24:34.089107 17729 leader_election.cc:304] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:34.089286 17830 raft_consensus.cc:2468] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 26.
I20260709 12:24:34.089759 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:34.090487 18361 raft_consensus.cc:2368] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 25. Current term is 26.
I20260709 12:24:34.090556 18334 leader_election.cc:304] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:34.091198 18364 raft_consensus.cc:1240] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 21. Current term is 22. Ops: []
I20260709 12:24:34.091354 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:34.091990 17756 raft_consensus.cc:3060] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:24:33.823963 25740 raft_consensus.cc:2749] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:24:33.980561 26239 raft_consensus.cc:3055] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:24:34.084375 26514 raft_consensus.cc:2804] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:24:33.973233 26419 raft_consensus.cc:2804] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:24:33.971969 17801 leader_election.cc:304] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:24:34.098637 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 126830us
I20260709 12:24:34.085695 26488 raft_consensus.cc:2804] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Leader pre-election won for term 22
W20260709 12:24:33.998674 17585 rpcz_store.cc:269] 0709 12:24:17.970843 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:17.971440 (+   597us) service_pool.cc:224] Handling call
0709 12:24:33.829359 (+15857919us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:24:28.004967 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:24:28.005636 (+   669us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:24:28.005693 (+    57us) write_op.cc:435] Acquired schema lock
0709 12:24:28.005711 (+    18us) tablet.cc:662] Decoding operations
0709 12:24:28.007313 (+  1602us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:24:28.007430 (+   117us) write_op.cc:641] Partition lock acquired for write op
0709 12:24:28.007458 (+    28us) tablet.cc:685] Acquiring locks for 18 operations
0709 12:24:28.248086 (+240628us) tablet.cc:701] Row locks acquired
0709 12:24:28.248141 (+    55us) write_op.cc:260] PREPARE: finished
0709 12:24:28.248336 (+   195us) write_op.cc:270] Start()
0709 12:24:28.248610 (+   274us) write_op.cc:276] Timestamp: P: 1783599868248264 usec, L: 0
0709 12:24:28.248654 (+    44us) op_driver.cc:348] REPLICATION: starting
0709 12:24:28.249717 (+  1063us) log.cc:844] Serialized 6994 byte log entry
0709 12:24:28.339523 (+ 89806us) op_driver.cc:464] REPLICATION: finished
0709 12:24:28.611493 (+271970us) write_op.cc:301] APPLY: starting
0709 12:24:28.619993 (+  8500us) tablet.cc:1367] starting BulkCheckPresence
0709 12:24:29.465656 (+845663us) tablet.cc:1370] finished BulkCheckPresence
0709 12:24:29.465793 (+   137us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:24:29.694402 (+228609us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:24:29.694820 (+   418us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=36,key_file_lookups=36,delta_file_lookups=72,mrs_lookups=0
0709 12:24:29.694876 (+    56us) write_op.cc:312] APPLY: finished
0709 12:24:29.695384 (+   508us) log.cc:844] Serialized 164 byte log entry
0709 12:24:29.858802 (+163418us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:24:29.909919 (+ 51117us) write_op.cc:454] Released schema lock
0709 12:24:29.910010 (+    91us) write_op.cc:341] FINISH: Updating metrics
0709 12:24:29.910456 (+   446us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:24:29.910491 (+    35us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":3084674,"reported_tablets":44,"child_traces":[["op",{"apply.queue_time_us":240066,"apply.run_cpu_time_us":55938,"apply.run_wall_time_us":1316847,"cfile_cache_hit":74,"cfile_cache_hit_bytes":79439,"num_ops":18,"prepare.queue_time_us":87077,"prepare.run_cpu_time_us":7941,"prepare.run_wall_time_us":245119,"replication_time_us":90631,"spinlock_wait_cycles":1280,"thread_start_us":272457,"threads_started":3,"wal-append.queue_time_us":115411,"wal-append.run_cpu_time_us":1666,"wal-append.run_wall_time_us":1309882}]]}
I20260709 12:24:33.928335 26479 consensus_queue.cc:237] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 18.18, Last appended by leader: 18, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:34.009152 26454 raft_consensus.cc:740] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:24:34.022243 26446 leader_election.cc:290] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:33.994562 26334 consensus_queue.cc:1059] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.031s
I20260709 12:24:33.983203 26441 raft_consensus.cc:2749] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:24:34.024928 26277 raft_consensus.cc:515] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:34.028501 18374 raft_consensus.cc:2468] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 26.
I20260709 12:24:34.030355 26340 raft_consensus.cc:515] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:34.125378 26277 leader_election.cc:290] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 29 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:34.099418 26419 raft_consensus.cc:493] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:34.126783 26419 raft_consensus.cc:3060] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:34.127409 26340 leader_election.cc:290] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 20 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:34.022265 26283 leader_election.cc:290] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:34.035275 26438 raft_consensus.cc:3060] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:34.095510 26239 raft_consensus.cc:740] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 23 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:24:34.132586 26239 consensus_queue.cc:260] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 23.15, Last appended by leader: 15, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:34.133847 26239 raft_consensus.cc:3060] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:34.086987 26421 raft_consensus.cc:515] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:24:34.034255 17578 catalog_manager.cc:5697] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb reported cstate change: term changed from 16 to 19, leader changed from 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) to e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66). New cstate: current_term: 19 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:24:34.038998 17680 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37402 (request call id 3584) took 1525 ms. Trace:
I20260709 12:24:34.137145 17680 rpcz_store.cc:276] 0709 12:24:32.515625 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:32.515766 (+   141us) service_pool.cc:224] Handling call
0709 12:24:33.710061 (+1194295us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:33.773351 (+ 63290us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:34.038811 (+265460us) spinlock_profiling.cc:243] Waited 1.14 s on lock 0x7b54003467d0. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a61f8ad 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:24:34.038938 (+   127us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2612655872}
I20260709 12:24:34.040369 26450 consensus_queue.cc:260] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 21.15, Last appended by leader: 15, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:33.983762 26397 raft_consensus.cc:2749] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 12:24:34.047008 26295 raft_consensus.cc:3060] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:34.048194 26302 consensus_queue.cc:237] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 19.15, Last appended by leader: 15, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:34.010789 26416 raft_consensus.cc:697] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 26 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:24:34.043771 17755 raft_consensus.cc:2468] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 24.
I20260709 12:24:34.051203 26164 leader_election.cc:290] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:34.147670 26309 raft_consensus.cc:493] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:24:34.148633 26309 raft_consensus.cc:515] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:34.140753 26450 raft_consensus.cc:3060] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:34.067027 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:34.067999 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:34.159850 26350 consensus_queue.cc:237] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 21.16, Last appended by leader: 16, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:34.161042 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:34.161912 18362 raft_consensus.cc:2368] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for earlier term 29. Current term is 30.
I20260709 12:24:34.162956 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:34.163317 17654 leader_election.cc:400] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 29 election: Vote denied by peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577) with higher term. Message: Invalid argument: T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for earlier term 29. Current term is 30.
I20260709 12:24:34.163720 17654 leader_election.cc:403] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 29 election: Cancelling election due to peer responding with higher term
I20260709 12:24:34.164608 26322 raft_consensus.cc:3060] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:24:34.166427 26483 consensus_queue.cc:237] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 18.15, Last appended by leader: 15, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:34.071274 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:34.159910 18368 raft_consensus.cc:2468] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 24.
I20260709 12:24:34.170542 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 22 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:34.170976 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:34.171963 17654 leader_election.cc:304] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:34.172494 18368 raft_consensus.cc:2368] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 22. Current term is 23.
I20260709 12:24:34.172775 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 28 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:34.173694 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 22 candidate_status { last_received { term: 21 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:34.083819 26412 raft_consensus.cc:2749] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:24:34.084839 17728 leader_election.cc:400] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 26 election: Vote denied by peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071) with higher term. Message: Invalid argument: T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for earlier term 26. Current term is 27.
W20260709 12:24:34.176823 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 92129.5us
I20260709 12:24:34.177655 17728 leader_election.cc:304] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:34.086982 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:34.087141 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 23 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:34.178797 18363 raft_consensus.cc:2468] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 22.
W20260709 12:24:33.929682 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 76165.1us
I20260709 12:24:34.092890 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:34.092926 18334 leader_election.cc:304] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
W20260709 12:24:34.180156 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 87309.7us
I20260709 12:24:34.181085 17757 raft_consensus.cc:2393] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 27: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:24:34.182015 17652 leader_election.cc:304] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:34.182067 17654 leader_election.cc:304] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:24:34.093338 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 22 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:24:33.980541 26505 log.cc:927] Time spent T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.111s	user 0.001s	sys 0.000s
I20260709 12:24:34.087500 17727 leader_election.cc:304] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:34.096300 18361 raft_consensus.cc:1240] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 24. Current term is 25. Ops: []
I20260709 12:24:34.097218 16501 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260709 12:24:34.184983 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:34.097317 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:34.186290 18365 raft_consensus.cc:2468] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 26.
I20260709 12:24:34.013386 26471 raft_consensus.cc:3055] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:24:34.099802 17801 leader_election.cc:400] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 26 election: Vote denied by peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651) with higher term. Message: Invalid argument: T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 26. Current term is 27.
I20260709 12:24:34.187650 17801 leader_election.cc:403] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 26 election: Cancelling election due to peer responding with higher term
W20260709 12:24:34.188342 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 88681.8us
W20260709 12:24:34.189062 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 101654us
I20260709 12:24:34.189716 17727 leader_election.cc:304] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:34.190923 17727 leader_election.cc:304] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:34.191520 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:34.183854 26340 raft_consensus.cc:2749] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:24:34.192286 26164 raft_consensus.cc:2749] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Leader election lost for term 26. Reason: Vote denied by peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651) with higher term. Message: Invalid argument: T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 26. Current term is 27.
I20260709 12:24:34.193082 26471 raft_consensus.cc:740] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 20 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:24:34.193245 26283 raft_consensus.cc:2749] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260709 12:24:34.193626 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:34.193948 26283 raft_consensus.cc:2804] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:24:34.194540 26283 raft_consensus.cc:493] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:34.194025 26471 consensus_queue.cc:260] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 20.14, Last appended by leader: 14, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:34.194908 26283 raft_consensus.cc:3060] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:24:34.193115 26412 raft_consensus.cc:2804] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:24:34.195902 26412 raft_consensus.cc:493] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:34.196280 26412 raft_consensus.cc:3060] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:34.196064 26516 consensus_queue.cc:1059] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:24:34.197059 26516 raft_consensus.cc:2804] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:24:34.197544 26516 raft_consensus.cc:697] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 23 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:24:34.105314 26454 consensus_queue.cc:260] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 26.15, Last appended by leader: 15, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:34.106601 17684 raft_consensus.cc:1240] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 23. Current term is 24. Ops: [23.17-23.17]
I20260709 12:24:34.198426 26516 consensus_queue.cc:237] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 21.13, Last appended by leader: 13, Current term: 23, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:34.108008 17803 leader_election.cc:400] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 22 pre-election: Vote denied by peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577) with higher term. Message: Invalid argument: T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 22. Current term is 23.
I20260709 12:24:34.108702 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:34.108915 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 25 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:34.201090 17755 raft_consensus.cc:1240] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 23. Current term is 24. Ops: []
I20260709 12:24:34.109262 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:34.113590 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:34.188216 17652 leader_election.cc:304] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:34.098277 26354 raft_consensus.cc:2804] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:24:34.095732 26330 raft_consensus.cc:2749] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:24:34.118685 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:34.124526 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 23 candidate_status { last_received { term: 21 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:34.125217 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:34.100996 26499 raft_consensus.cc:2804] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:24:34.211767 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:34.212188 18371 raft_consensus.cc:2368] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 22. Current term is 23.
I20260709 12:24:34.186317 16501 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:24:34.087224 26377 raft_consensus.cc:515] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:34.101258 26488 raft_consensus.cc:493] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:34.138692 26514 raft_consensus.cc:697] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 22 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:24:34.002727 26443 raft_consensus.cc:2749] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:24:34.073009 17802 leader_election.cc:400] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 22 election: Vote denied by peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071) with higher term. Message: Invalid argument: T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 22. Current term is 23.
I20260709 12:24:34.234505 17802 leader_election.cc:403] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 22 election: Cancelling election due to peer responding with higher term
I20260709 12:24:34.173990 18362 raft_consensus.cc:2410] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 28 because replica has last-logged OpId of term: 26 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 15.
I20260709 12:24:34.175916 18368 raft_consensus.cc:2393] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 22: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:24:34.236667 18368 raft_consensus.cc:1240] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 19. Current term is 20. Ops: []
I20260709 12:24:34.189749 17728 leader_election.cc:304] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:34.201491 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:34.076656 26414 raft_consensus.cc:697] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 27 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:24:34.192093 17727 leader_election.cc:304] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:34.201821 26281 raft_consensus.cc:2749] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260709 12:24:34.201563 17689 raft_consensus.cc:2468] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 24.
I20260709 12:24:34.143119 26446 raft_consensus.cc:3055] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:24:34.202209 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:34.202371 17686 raft_consensus.cc:3060] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:34.210005 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:34.212311 26164 raft_consensus.cc:493] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:34.203850 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:34.204254 26454 raft_consensus.cc:3060] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:34.192116 18365 raft_consensus.cc:3060] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:34.212802 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:34.207559 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:34.269699 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:34.269776 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 25 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:34.270445 17683 raft_consensus.cc:2468] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 24.
I20260709 12:24:34.213572 16501 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:24:34.271458 17680 raft_consensus.cc:1240] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 21. Current term is 22. Ops: []
I20260709 12:24:34.268461 17686 raft_consensus.cc:2468] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 26.
I20260709 12:24:34.272199 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:34.203055 18366 raft_consensus.cc:1275] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 18 index: 12. Preceding OpId from leader: term: 22 index: 14. (index mismatch)
I20260709 12:24:34.208050 18374 raft_consensus.cc:2368] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 20. Current term is 21.
I20260709 12:24:34.215400 17654 leader_election.cc:304] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b
W20260709 12:24:34.275177 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 59911us
I20260709 12:24:34.225024 26271 raft_consensus.cc:2749] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:24:34.276046 26498 consensus_queue.cc:1059] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 23.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20260709 12:24:34.276361 26271 raft_consensus.cc:2804] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:24:34.276784 26271 raft_consensus.cc:493] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:24:34.277158 26271 raft_consensus.cc:3060] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:34.275775 18333 leader_election.cc:304] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:34.151808 26309 leader_election.cc:290] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:34.278187 26283 raft_consensus.cc:515] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:34.231535 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:34.274513 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:34.279429 17838 raft_consensus.cc:2393] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 26: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:24:34.186759 26519 raft_consensus.cc:2749] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:24:34.274518 17756 raft_consensus.cc:2468] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 30.
I20260709 12:24:34.116248 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 28 candidate_status { last_received { term: 27 index: 21 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:34.168848 26416 consensus_queue.cc:237] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 24.17, Last appended by leader: 17, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:34.238709 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:24:34.193783 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 103141us
I20260709 12:24:34.281636 17837 raft_consensus.cc:2393] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 22: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:24:34.281865 18334 leader_election.cc:304] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:34.282524 17729 leader_election.cc:304] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:34.283068 26330 raft_consensus.cc:2749] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260709 12:24:34.240513 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 167705us
I20260709 12:24:34.285213 26377 leader_election.cc:290] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:24:34.239607 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 49922.8us
I20260709 12:24:34.285714 18334 leader_election.cc:304] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:34.286324 17728 leader_election.cc:304] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:34.212446 25967 raft_consensus.cc:3055] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:24:34.291606 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:34.291755 25967 raft_consensus.cc:740] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:24:34.292348 17837 raft_consensus.cc:2468] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 28.
I20260709 12:24:34.292840 25967 consensus_queue.cc:260] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 23.13, Last appended by leader: 13, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:34.212108 26354 raft_consensus.cc:493] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:34.293970 26354 raft_consensus.cc:3060] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:34.294281 25967 raft_consensus.cc:3060] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:34.294440 26368 raft_consensus.cc:2749] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260709 12:24:34.298321 18333 leader_election.cc:304] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
W20260709 12:24:34.245983 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 53958.8us
I20260709 12:24:34.300999 26322 raft_consensus.cc:2749] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Leader election lost for term 29. Reason: Vote denied by peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577) with higher term. Message: Invalid argument: T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for earlier term 29. Current term is 30.
I20260709 12:24:34.246995 17685 raft_consensus.cc:2468] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 27.
I20260709 12:24:34.247332 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:34.204716 26471 raft_consensus.cc:3060] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:34.213644 26513 raft_consensus.cc:2804] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:24:34.304419 26513 raft_consensus.cc:493] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:24:34.304906 26513 raft_consensus.cc:3060] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Advancing to term 25
W20260709 12:24:34.201109 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 93257.5us
I20260709 12:24:34.251328 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 25 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:34.308389 17832 raft_consensus.cc:2468] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 24.
I20260709 12:24:34.252451 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 23 candidate_status { last_received { term: 21 index: 18 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:34.308977 17803 leader_election.cc:400] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 25 election: Vote denied by peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577) with higher term. Message: Invalid argument: T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 25. Current term is 26.
I20260709 12:24:34.256999 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:34.309739 17757 raft_consensus.cc:2368] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for earlier term 29. Current term is 30.
I20260709 12:24:34.250905 26334 raft_consensus.cc:3060] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:34.211215 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:34.314787 17831 raft_consensus.cc:2393] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 23: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:24:34.278630 26340 raft_consensus.cc:2749] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 12:24:34.225426 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:34.230475 18367 raft_consensus.cc:1275] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 23 index: 13. Preceding OpId from leader: term: 24 index: 14. (index mismatch)
I20260709 12:24:34.315420 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:34.283478 17729 leader_election.cc:304] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:24:34.220911 26499 raft_consensus.cc:697] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 27 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:24:34.257679 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:34.330169 17755 raft_consensus.cc:2393] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 28: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
W20260709 12:24:34.330905 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 72445.2us
I20260709 12:24:34.248745 26446 raft_consensus.cc:740] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 24 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:24:34.236078 25866 raft_consensus.cc:493] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:24:34.233250 26488 raft_consensus.cc:3060] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:34.268644 26365 raft_consensus.cc:493] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:34.289628 26206 raft_consensus.cc:2804] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:24:34.290343 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:34.260191 26441 raft_consensus.cc:2804] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:24:34.290869 26399 consensus_queue.cc:1059] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 20.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20260709 12:24:34.310319 17803 leader_election.cc:304] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:34.316690 18333 leader_election.cc:304] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:34.317623 18365 raft_consensus.cc:2468] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 27.
I20260709 12:24:34.346948 17759 raft_consensus.cc:1240] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 27. Current term is 28. Ops: []
I20260709 12:24:34.318323 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:24:34.328833 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 45450.5us
W20260709 12:24:34.351656 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 66302.3us
I20260709 12:24:34.350549 26412 raft_consensus.cc:515] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:34.352358 17729 leader_election.cc:304] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:34.332185 17652 leader_election.cc:304] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:34.277171 26498 raft_consensus.cc:3055] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:24:34.353310 26498 raft_consensus.cc:740] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 23 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:24:34.354233 26498 consensus_queue.cc:260] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 23.16, Last appended by leader: 16, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:34.354689 26412 leader_election.cc:290] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 23 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:34.341416 17830 raft_consensus.cc:2393] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 27: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:24:34.250675 25784 raft_consensus.cc:2749] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260709 12:24:34.234411 26487 consensus_queue.cc:1059] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 26.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.096s
I20260709 12:24:34.330353 26330 raft_consensus.cc:2804] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:24:34.360775 18333 leader_election.cc:304] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:24:34.409775 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 57501.1us
I20260709 12:24:34.357645 18334 leader_election.cc:304] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:24:34.334540 26447 raft_consensus.cc:2749] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
W20260709 12:24:34.414551 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 56985.1us
I20260709 12:24:34.417380 18334 leader_election.cc:304] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:34.334748 26420 raft_consensus.cc:493] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:34.413033 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 60703.3us
I20260709 12:24:34.420025 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:34.355790 26206 raft_consensus.cc:697] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:24:34.259240 26409 consensus_queue.cc:1059] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 26.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.090s
I20260709 12:24:34.423319 18367 raft_consensus.cc:1240] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 20. Current term is 21. Ops: []
I20260709 12:24:34.423938 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:34.438320 18374 raft_consensus.cc:2468] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 22.
I20260709 12:24:34.286249 26422 consensus_queue.cc:1059] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 27.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.076s
I20260709 12:24:34.343312 26441 raft_consensus.cc:493] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:34.310734 26164 raft_consensus.cc:515] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:34.446133 17729 leader_election.cc:304] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:34.410171 26340 raft_consensus.cc:2749] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
W20260709 12:24:34.450793 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 156790us
I20260709 12:24:34.422786 26397 raft_consensus.cc:3055] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:24:34.413589 26330 raft_consensus.cc:493] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:34.338703 26368 raft_consensus.cc:2804] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:24:34.329454 26277 consensus_queue.cc:1059] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 19.13, Next index: 14, Last known committed idx: 12, Time since last communication: 0.111s
I20260709 12:24:34.444520 17681 raft_consensus.cc:2468] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 30.
I20260709 12:24:34.415534 26261 raft_consensus.cc:2804] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:24:34.461922 17728 leader_election.cc:304] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:34.462723 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:34.482280 17759 raft_consensus.cc:1240] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 23. Current term is 24. Ops: []
I20260709 12:24:34.463438 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:34.448026 26447 raft_consensus.cc:2749] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 12:24:34.424129 26208 raft_consensus.cc:2749] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:24:34.458240 26441 raft_consensus.cc:3060] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:34.491374 17728 leader_election.cc:304] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:34.450804 26446 consensus_queue.cc:260] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 24.19, Last appended by leader: 19, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:34.476783 26330 raft_consensus.cc:3060] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:34.428473 26434 raft_consensus.cc:2804] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:24:34.341189 26365 raft_consensus.cc:515] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:34.483104 17833 raft_consensus.cc:2468] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 21.
I20260709 12:24:34.464161 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
W20260709 12:24:34.483855 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 173630us
I20260709 12:24:34.341639 25866 raft_consensus.cc:515] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:34.486145 26421 leader_election.cc:290] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:34.431757 26479 consensus_queue.cc:1059] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.043s
I20260709 12:24:34.460963 26526 raft_consensus.cc:2749] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:24:34.495151 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 27 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:34.510983 17831 raft_consensus.cc:2468] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 26.
I20260709 12:24:34.422842 26450 raft_consensus.cc:2804] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:24:34.471613 26397 raft_consensus.cc:740] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 21 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:24:34.442461 26420 raft_consensus.cc:515] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:34.477990 26454 raft_consensus.cc:2804] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:24:34.504184 17837 raft_consensus.cc:3060] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:34.504726 17803 leader_election.cc:400] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 22 election: Vote denied by peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577) with higher term. Message: Invalid argument: T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 22. Current term is 23.
I20260709 12:24:34.412580 26514 consensus_queue.cc:237] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 20.17, Last appended by leader: 17, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:34.487738 26234 consensus_meta.cc:287] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.534s	user 0.006s	sys 0.006s
I20260709 12:24:34.431221 26414 consensus_queue.cc:237] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 25.15, Last appended by leader: 15, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:34.515527 18334 leader_election.cc:304] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:24:34.517406 17830 raft_consensus.cc:1275] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 24 index: 17. Preceding OpId from leader: term: 25 index: 17. (term mismatch)
I20260709 12:24:34.523388 17830 pending_rounds.cc:85] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b: Aborting all ops after (but not including) 16
I20260709 12:24:34.523691 17830 pending_rounds.cc:107] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b: Aborting uncommitted NO_OP operation due to leader change: 24.17
I20260709 12:24:34.523960 17830 raft_consensus.cc:2889] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:34.503427 26434 raft_consensus.cc:493] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:34.521153 17803 leader_election.cc:304] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
W20260709 12:24:34.524767 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 192931us
I20260709 12:24:34.487371 26419 raft_consensus.cc:515] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:34.506520 26528 raft_consensus.cc:2749] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 12:24:34.527040 17652 leader_election.cc:304] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:34.512850 26421 raft_consensus.cc:2749] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260709 12:24:34.474009 26499 consensus_queue.cc:237] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 25.18, Last appended by leader: 18, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:34.535763 26397 consensus_queue.cc:260] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 21.18, Last appended by leader: 18, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:34.537087 26397 raft_consensus.cc:3060] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:34.540891 26529 raft_consensus.cc:2749] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:24:34.499809 26206 consensus_queue.cc:237] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 24.19, Last appended by leader: 19, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:34.528602 26434 raft_consensus.cc:3060] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:34.542825 26419 leader_election.cc:290] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 19 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:34.543545 18361 raft_consensus.cc:2468] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 19.
I20260709 12:24:34.544277 26498 raft_consensus.cc:3060] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:34.518882 26454 raft_consensus.cc:493] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:34.546300 26454 raft_consensus.cc:3060] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:34.551398 26283 leader_election.cc:290] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 28 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:34.522095 26450 raft_consensus.cc:697] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 27 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:24:34.507112 26416 raft_consensus.cc:493] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:34.412751 17578 catalog_manager.cc:5697] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb reported cstate change: term changed from 16 to 17, leader changed from 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) to e063541542a94969898868bf6eb5ce1b (127.16.29.67). New cstate: current_term: 17 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:24:34.577296 17687 raft_consensus.cc:2410] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for term 26 because replica has last-logged OpId of term: 25 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 14.
W20260709 12:24:34.529980 26524 log.cc:927] Time spent T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.058s	user 0.001s	sys 0.000s
W20260709 12:24:34.581478 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 90204.1us
I20260709 12:24:34.530087 26447 raft_consensus.cc:2804] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:24:34.533843 26294 raft_consensus.cc:2749] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260709 12:24:34.588182 26334 raft_consensus.cc:2749] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Leader election lost for term 22. Reason: Vote denied by peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071) with higher term. Message: Invalid argument: T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 22. Current term is 23.
I20260709 12:24:34.588706 17578 catalog_manager.cc:5697] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb reported cstate change: term changed from 11 to 17, leader changed from e063541542a94969898868bf6eb5ce1b (127.16.29.67) to e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66). New cstate: current_term: 17 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: UNKNOWN } } }
I20260709 12:24:34.589648 25866 leader_election.cc:290] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:34.590966 26434 raft_consensus.cc:515] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:34.583329 26416 raft_consensus.cc:515] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:34.596719 17728 leader_election.cc:304] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:34.527666 26234 raft_consensus.cc:515] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:34.618427 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 85682.5us
W20260709 12:24:34.626003 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 265282us
I20260709 12:24:34.628224 17837 raft_consensus.cc:2468] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 23.
I20260709 12:24:34.630151 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:34.630254 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:34.630970 17837 raft_consensus.cc:2393] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 19: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:24:34.535096 26354 raft_consensus.cc:515] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Starting leader election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:34.630744 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 27 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:34.586803 26261 raft_consensus.cc:697] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 26 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:24:34.636480 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 20 candidate_status { last_received { term: 18 index: 18 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:34.635218 18333 leader_election.cc:304] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:24:34.639086 17681 raft_consensus.cc:2468] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 19.
I20260709 12:24:34.640419 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:34.610468 26447 raft_consensus.cc:697] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 30 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:24:34.644892 18333 leader_election.cc:304] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:34.643406 17836 raft_consensus.cc:2468] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 23.
W20260709 12:24:34.646229 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 125189us
I20260709 12:24:34.647322 17803 leader_election.cc:400] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 22 pre-election: Vote denied by peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577) with higher term. Message: Invalid argument: T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 22. Current term is 23.
I20260709 12:24:34.647789 17803 leader_election.cc:403] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 22 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:24:34.651780 17803 leader_election.cc:304] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:34.625614 26208 raft_consensus.cc:493] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:34.667234 17803 leader_election.cc:400] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 20 election: Vote denied by peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577) with higher term. Message: Invalid argument: T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 20. Current term is 21.
I20260709 12:24:34.633852 26539 raft_consensus.cc:2749] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:24:34.669000 26443 consensus_queue.cc:1048] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:24:34.648936 26450 consensus_queue.cc:237] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 26.16, Last appended by leader: 16, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:34.673710 26330 raft_consensus.cc:515] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Starting leader election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:34.674893 26513 raft_consensus.cc:515] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:34.666603 26281 raft_consensus.cc:493] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:34.677472 26281 raft_consensus.cc:515] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:34.678122 26261 consensus_queue.cc:237] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 24.17, Last appended by leader: 17, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:34.679462 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:34.681092 17759 raft_consensus.cc:1275] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 25 index: 15. Preceding OpId from leader: term: 27 index: 16. (index mismatch)
I20260709 12:24:34.679329 26519 raft_consensus.cc:3055] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:24:34.657177 26239 raft_consensus.cc:2749] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Leader pre-election lost for term 22. Reason: Vote denied by peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577) with higher term. Message: Invalid argument: T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 22. Current term is 23.
I20260709 12:24:34.641834 26368 raft_consensus.cc:697] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
W20260709 12:24:34.689777 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 70266.8us
I20260709 12:24:34.690768 17652 leader_election.cc:304] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:24:34.661540 26420 leader_election.cc:290] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:34.686285 17803 leader_election.cc:304] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:24:34.666476 25866 raft_consensus.cc:2749] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:24:34.685503 26519 raft_consensus.cc:740] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:24:34.723995 17684 raft_consensus.cc:1240] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 24. Current term is 25. Ops: []
W20260709 12:24:34.747196 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 56526.5us
W20260709 12:24:34.752751 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 66531.2us
I20260709 12:24:34.664683 26208 raft_consensus.cc:515] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:34.727406 26490 log.cc:927] Time spent T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.057s	user 0.000s	sys 0.000s
I20260709 12:24:34.729535 26420 raft_consensus.cc:2804] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Leader election won for term 24
W20260709 12:24:34.762733 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 117913us
I20260709 12:24:34.766445 26330 leader_election.cc:290] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 25 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:34.768338 26454 raft_consensus.cc:515] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:34.770644 26454 leader_election.cc:290] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 23 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:34.771327 25866 raft_consensus.cc:493] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:34.772536 26434 leader_election.cc:290] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 22 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:34.774042 26513 leader_election.cc:290] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 25 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:34.687861 26302 consensus_queue.cc:1048] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.068s
I20260709 12:24:34.749640 26294 raft_consensus.cc:2804] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:24:34.773645 25866 raft_consensus.cc:515] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:34.777415 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:34.753486 26365 leader_election.cc:290] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:34.749197 26538 raft_consensus.cc:2749] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:24:34.786592 26208 leader_election.cc:290] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:34.784258 26294 raft_consensus.cc:493] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:24:34.789202 26354 leader_election.cc:290] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 25 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:34.765021 26239 raft_consensus.cc:2804] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:24:34.760011 26537 consensus_queue.cc:1048] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.055s
I20260709 12:24:34.794963 26420 raft_consensus.cc:697] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 24 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:24:34.749935 26447 consensus_queue.cc:237] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 28.19, Last appended by leader: 19, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:34.819732 26420 consensus_queue.cc:237] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 23.17, Last appended by leader: 17, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:34.837654 26535 log.cc:927] Time spent T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.054s	user 0.000s	sys 0.000s
I20260709 12:24:34.829656 26488 raft_consensus.cc:515] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:34.883059 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 25 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:34.885433 17837 raft_consensus.cc:3060] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:34.885838 17830 raft_consensus.cc:1275] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 17 index: 14. Preceding OpId from leader: term: 21 index: 16. (index mismatch)
I20260709 12:24:34.887995 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 20 candidate_status { last_received { term: 18 index: 18 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:34.890993 17755 raft_consensus.cc:2368] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 22. Current term is 23.
I20260709 12:24:34.891613 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:34.893241 17685 raft_consensus.cc:2393] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 22: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:24:34.899037 17683 raft_consensus.cc:1275] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 22 index: 14. Preceding OpId from leader: term: 27 index: 16. (index mismatch)
I20260709 12:24:34.778277 18333 leader_election.cc:304] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:24:34.911689 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 133301us
I20260709 12:24:34.806661 18366 raft_consensus.cc:2468] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 22.
I20260709 12:24:34.920244 18367 raft_consensus.cc:1275] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 18 index: 18. Preceding OpId from leader: term: 19 index: 19. (index mismatch)
I20260709 12:24:34.922046 18368 raft_consensus.cc:3060] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:34.922246 17578 catalog_manager.cc:5697] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb reported cstate change: term changed from 11 to 16. New cstate: current_term: 16 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:24:34.934072 17803 leader_election.cc:304] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:24:34.814939 26239 raft_consensus.cc:493] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:34.938282 26239 raft_consensus.cc:3060] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Advancing to term 20
W20260709 12:24:34.811594 25967 consensus_meta.cc:287] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.517s	user 0.011s	sys 0.002s
W20260709 12:24:34.941591 26271 consensus_meta.cc:287] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.664s	user 0.008s	sys 0.008s
I20260709 12:24:34.942210 26271 raft_consensus.cc:515] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:34.944887 26302 consensus_queue.cc:1048] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:24:34.945907 26208 raft_consensus.cc:2749] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:24:34.946749 25866 leader_election.cc:290] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:34.946954 26208 consensus_queue.cc:1048] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:24:34.813457 26294 raft_consensus.cc:3060] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:24:34.947546 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:34.949091 17759 raft_consensus.cc:2393] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 24: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:24:34.951014 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:34.952348 17689 raft_consensus.cc:1240] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 22. Current term is 23. Ops: []
I20260709 12:24:34.782238 26368 consensus_queue.cc:237] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 22.18, Last appended by leader: 18, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:34.818346 26416 leader_election.cc:290] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:34.983637 26334 raft_consensus.cc:2804] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:24:34.984760 26334 raft_consensus.cc:493] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:24:34.985245 26271 leader_election.cc:290] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 23 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:34.985478 26334 raft_consensus.cc:3060] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:34.991997 17801 leader_election.cc:304] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:34.996333 26302 raft_consensus.cc:3060] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:34.995436 26397 consensus_queue.cc:1048] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:24:34.994953 18368 raft_consensus.cc:2468] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 23.
I20260709 12:24:34.997136 26398 raft_consensus.cc:493] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:24:35.022746 26398 raft_consensus.cc:515] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:35.008658 17682 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37402 (request call id 3595) took 1211 ms. Trace:
W20260709 12:24:35.031615 17802 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:35.053928 26516 raft_consensus.cc:493] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:35.054515 26309 raft_consensus.cc:493] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:35.066634 26309 raft_consensus.cc:515] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:35.069015 26488 leader_election.cc:290] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 22 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:35.070542 26309 leader_election.cc:290] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:35.031427 17682 rpcz_store.cc:276] 0709 12:24:33.797138 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:33.797535 (+   397us) service_pool.cc:224] Handling call
0709 12:24:35.008194 (+1210659us) spinlock_profiling.cc:243] Waited 977 ms on lock 0x7b5400206554. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:24:35.008429 (+   235us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2247968640}
I20260709 12:24:35.075215 26398 leader_election.cc:290] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:35.076738 26516 raft_consensus.cc:515] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:35.081024 26558 log.cc:927] Time spent T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.099s	user 0.001s	sys 0.000s
I20260709 12:24:35.082304 26516 leader_election.cc:290] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:35.082628 26239 raft_consensus.cc:515] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:35.086194 17684 raft_consensus.cc:1275] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 19 index: 15. Preceding OpId from leader: term: 20 index: 16. (index mismatch)
W20260709 12:24:35.086947 26441 consensus_meta.cc:287] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.583s	user 0.010s	sys 0.008s
I20260709 12:24:35.090708 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:35.124778 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:35.125905 26281 leader_election.cc:290] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:35.092134 17682 raft_consensus.cc:1275] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 18 index: 18. Preceding OpId from leader: term: 19 index: 19. (index mismatch)
W20260709 12:24:35.096315 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 74434.7us
I20260709 12:24:35.128443 26519 consensus_queue.cc:260] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 26.16, Last appended by leader: 16, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:35.130143 18361 raft_consensus.cc:1240] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 20. Current term is 21. Ops: []
I20260709 12:24:35.130448 26519 raft_consensus.cc:3060] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:35.132282 26447 raft_consensus.cc:493] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:35.115041 26164 leader_election.cc:290] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:35.132972 26447 raft_consensus.cc:515] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:35.126026 26441 raft_consensus.cc:515] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:35.135964 17764 raft_consensus.cc:2393] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 20: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:24:35.135893 26168 raft_consensus.cc:3055] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:24:35.136475 26168 raft_consensus.cc:740] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:24:35.137003 17831 raft_consensus.cc:2368] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 22. Current term is 23.
I20260709 12:24:35.137243 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:35.137930 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:35.138339 18333 leader_election.cc:400] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 22 election: Vote denied by peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837) with higher term. Message: Invalid argument: T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 22. Current term is 23.
I20260709 12:24:35.138720 18333 leader_election.cc:403] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 22 election: Cancelling election due to peer responding with higher term
I20260709 12:24:35.132099 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:35.138190 18365 raft_consensus.cc:2393] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 22: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:24:35.142946 26557 raft_consensus.cc:2749] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Leader election lost for term 22. Reason: Vote denied by peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837) with higher term. Message: Invalid argument: T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 22. Current term is 23.
I20260709 12:24:35.143441 26537 raft_consensus.cc:493] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:35.007834 26334 raft_consensus.cc:515] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:35.144380 26420 raft_consensus.cc:493] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
W20260709 12:24:35.163832 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 73176.4us
I20260709 12:24:35.124845 26436 raft_consensus.cc:3055] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:24:35.175509 17837 raft_consensus.cc:2468] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 25.
I20260709 12:24:35.183014 26443 consensus_queue.cc:1059] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.008s
I20260709 12:24:35.184300 26443 raft_consensus.cc:3055] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:24:35.184784 26443 raft_consensus.cc:740] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 20 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:24:35.186164 26443 consensus_queue.cc:260] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 20.16, Last appended by leader: 16, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:35.172334 26537 raft_consensus.cc:515] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:35.165364 26164 raft_consensus.cc:493] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:35.173878 26436 raft_consensus.cc:740] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:24:35.193778 26537 leader_election.cc:290] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:35.172340 26397 consensus_queue.cc:1059] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 24.14, Next index: 15, Last known committed idx: 13, Time since last communication: 0.031s
I20260709 12:24:35.187866 26443 raft_consensus.cc:3060] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:35.216516 26447 leader_election.cc:290] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:35.229295 26277 raft_consensus.cc:3055] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:24:35.229818 26277 raft_consensus.cc:740] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:24:35.230430 17757 raft_consensus.cc:1275] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 24 index: 17. Preceding OpId from leader: term: 27 index: 19. (index mismatch)
I20260709 12:24:35.204650 26479 raft_consensus.cc:493] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:35.192332 26420 raft_consensus.cc:515] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:35.231055 26277 consensus_queue.cc:260] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 19.13, Last appended by leader: 13, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:35.216964 26441 leader_election.cc:290] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 20 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:35.257711 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 19 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:35.258141 26334 leader_election.cc:290] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 22 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:35.258632 18361 raft_consensus.cc:2468] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 18.
I20260709 12:24:35.259148 26294 raft_consensus.cc:515] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:35.259558 26531 raft_consensus.cc:493] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:35.260617 26531 raft_consensus.cc:515] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:35.261610 26294 leader_election.cc:290] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 28 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:35.220060 26164 raft_consensus.cc:515] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:35.263020 26531 leader_election.cc:290] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:35.264777 26398 raft_consensus.cc:3055] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:24:35.265173 26398 raft_consensus.cc:740] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 24 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:24:35.266216 26398 consensus_queue.cc:260] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 24.14, Last appended by leader: 14, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:35.258623 18365 raft_consensus.cc:1275] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 25 index: 18. Preceding OpId from leader: term: 27 index: 19. (index mismatch)
I20260709 12:24:35.267750 26398 raft_consensus.cc:3060] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:35.275789 17803 leader_election.cc:304] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:35.280372 17803 leader_election.cc:304] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:24:35.268078 26447 raft_consensus.cc:493] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:35.261582 26239 leader_election.cc:290] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 20 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:35.245466 26537 consensus_queue.cc:1048] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.032s
I20260709 12:24:35.288028 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:35.287227 26334 raft_consensus.cc:2804] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:24:35.284940 25866 raft_consensus.cc:2749] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:24:35.295321 26334 raft_consensus.cc:493] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:35.286556 26277 raft_consensus.cc:3060] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:35.302434 26334 raft_consensus.cc:3060] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:35.305863 26479 raft_consensus.cc:515] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:24:35.316758 26523 log.cc:927] Time spent T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.823s	user 0.005s	sys 0.005s
I20260709 12:24:35.312705 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:35.319289 17756 raft_consensus.cc:2468] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 21.
I20260709 12:24:35.311915 26447 raft_consensus.cc:515] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:35.319478 26479 leader_election.cc:290] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:35.327025 17801 leader_election.cc:304] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:24:35.329753 26563 log.cc:927] Time spent T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.109s	user 0.003s	sys 0.000s
I20260709 12:24:35.331126 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:35.332031 18368 raft_consensus.cc:3060] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:35.332545 26446 raft_consensus.cc:3060] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:35.334736 17687 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37396 (request call id 3184) took 1230 ms. Trace:
I20260709 12:24:35.341478 17687 rpcz_store.cc:276] 0709 12:24:34.108262 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:34.108445 (+   183us) service_pool.cc:224] Handling call
0709 12:24:35.334365 (+1225920us) spinlock_profiling.cc:243] Waited 353 ms on lock 0x7b54003507d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:24:35.334559 (+   194us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":812529408}
I20260709 12:24:35.338385 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:35.342729 26397 raft_consensus.cc:2804] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:24:35.327294 26239 raft_consensus.cc:493] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:35.351982 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:35.315506 26483 raft_consensus.cc:493] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:35.346276 26164 leader_election.cc:290] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:35.361501 18368 raft_consensus.cc:2468] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 20.
I20260709 12:24:35.316357 26168 consensus_queue.cc:260] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 27.21, Last appended by leader: 21, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:35.340078 26479 consensus_queue.cc:1048] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.012s
I20260709 12:24:35.354364 26483 raft_consensus.cc:515] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:35.355014 26397 raft_consensus.cc:493] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
W20260709 12:24:35.375236 17728 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.16.29.65:33071 timed out after 1.675s (SENT) [suppressed 4 similar messages]
I20260709 12:24:35.375926 26168 raft_consensus.cc:3060] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:24:35.376305 26397 raft_consensus.cc:3060] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:35.384065 26239 raft_consensus.cc:515] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:35.387770 26239 leader_election.cc:290] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:35.389608 26234 leader_election.cc:290] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 21 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:35.394886 26354 raft_consensus.cc:493] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:35.395911 26354 raft_consensus.cc:515] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:35.401598 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:35.403208 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:35.404052 17833 raft_consensus.cc:3060] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:35.404186 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:35.404940 17832 raft_consensus.cc:3060] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Advancing to term 23
I20260709 12:24:35.408356 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:35.408622 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 25 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:35.409757 17834 raft_consensus.cc:2393] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 25: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:24:35.410228 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:35.410869 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 25 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:35.411424 17838 raft_consensus.cc:1275] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 24 index: 19. Preceding OpId from leader: term: 26 index: 20. (index mismatch)
I20260709 12:24:35.411784 18362 raft_consensus.cc:2393] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 25: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:24:35.354355 26498 raft_consensus.cc:493] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:35.410267 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:35.413090 18371 raft_consensus.cc:3060] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:35.412770 26498 raft_consensus.cc:515] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:35.411062 18363 raft_consensus.cc:3060] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Advancing to term 21
W20260709 12:24:35.427326 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 136542us
I20260709 12:24:35.430680 26302 raft_consensus.cc:2749] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:24:35.441073 26420 leader_election.cc:290] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:35.441388 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:35.442276 18362 raft_consensus.cc:2393] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 20: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:24:35.442495 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:35.443229 17831 raft_consensus.cc:2393] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 20: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:24:35.356499 26436 consensus_queue.cc:260] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 27.16, Last appended by leader: 16, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:35.415263 26498 leader_election.cc:290] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:35.451751 17801 leader_election.cc:400] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 22 pre-election: Vote denied by peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651) with higher term. Message: Invalid argument: T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 22. Current term is 23.
I20260709 12:24:35.452266 17801 leader_election.cc:403] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 22 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:24:35.457058 18371 raft_consensus.cc:2468] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 23.
I20260709 12:24:35.459460 26421 raft_consensus.cc:493] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:35.456853 26436 raft_consensus.cc:3060] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:24:35.460381 26421 raft_consensus.cc:515] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:35.460390 17727 leader_election.cc:304] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:35.468895 18374 raft_consensus.cc:1275] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 24 index: 17. Preceding OpId from leader: term: 26 index: 18. (index mismatch)
I20260709 12:24:35.435601 26397 raft_consensus.cc:515] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:35.469919 26526 raft_consensus.cc:493] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:35.470530 26526 raft_consensus.cc:515] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:35.471895 26397 leader_election.cc:290] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 21 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:35.472846 26526 leader_election.cc:290] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:24:35.473528 17804 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:35.475011 18364 raft_consensus.cc:1275] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 24 index: 19. Preceding OpId from leader: term: 26 index: 20. (index mismatch)
I20260709 12:24:35.475925 17764 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38786 (request call id 3855) took 1274 ms. Trace:
I20260709 12:24:35.476402 17764 rpcz_store.cc:276] 0709 12:24:34.201101 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:34.201993 (+   892us) service_pool.cc:224] Handling call
0709 12:24:35.475607 (+1273614us) spinlock_profiling.cc:243] Waited 886 ms on lock 0x7b54000fe6d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:24:35.475810 (+   203us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2038671232}
I20260709 12:24:35.478915 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:35.479666 17687 raft_consensus.cc:2468] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 24.
I20260709 12:24:35.479784 18334 leader_election.cc:304] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:24:35.481859 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:35.482191 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 19 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:35.484009 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:35.484843 17689 raft_consensus.cc:2393] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 22: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:24:35.485622 26421 leader_election.cc:290] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:35.485713 26488 raft_consensus.cc:2804] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:24:35.486440 26488 raft_consensus.cc:493] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:24:35.487046 26488 raft_consensus.cc:3060] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:35.488339 26234 raft_consensus.cc:493] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:35.489327 26234 raft_consensus.cc:515] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:35.491420 17727 leader_election.cc:304] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:35.491678 26441 consensus_queue.cc:1048] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:24:35.492645 26368 raft_consensus.cc:2804] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:24:35.493366 26368 raft_consensus.cc:697] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:24:35.491482 26332 raft_consensus.cc:2804] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:24:35.494930 26332 raft_consensus.cc:493] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:35.494766 26368 consensus_queue.cc:237] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 21.16, Last appended by leader: 16, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:35.495466 26332 raft_consensus.cc:3060] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:35.487087 26483 leader_election.cc:290] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:35.504751 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:35.504979 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 22 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:35.505666 17831 raft_consensus.cc:2393] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 27: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:24:35.506676 17830 raft_consensus.cc:1275] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 24 index: 17. Preceding OpId from leader: term: 26 index: 18. (index mismatch)
I20260709 12:24:35.507161 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 28 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:35.507315 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 20 candidate_status { last_received { term: 18 index: 18 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:35.508029 26483 raft_consensus.cc:493] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:35.508311 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:35.457891 26447 leader_election.cc:290] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:24:35.515503 26565 log.cc:927] Time spent T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.139s	user 0.000s	sys 0.000s
I20260709 12:24:35.420372 26537 raft_consensus.cc:493] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:35.517764 26537 raft_consensus.cc:515] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:35.518174 26397 raft_consensus.cc:493] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:35.518867 26397 raft_consensus.cc:515] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:35.520040 26354 leader_election.cc:290] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:35.520682 17757 raft_consensus.cc:2393] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 24: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:24:35.521091 26397 leader_election.cc:290] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:35.521627 26537 leader_election.cc:290] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:35.521800 26397 consensus_queue.cc:1048] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:24:35.524272 17803 leader_election.cc:304] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:35.521128 26234 leader_election.cc:290] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:24:35.509690 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 89841.2us
I20260709 12:24:35.506572 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:35.508579 26483 raft_consensus.cc:515] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:35.529011 26483 leader_election.cc:290] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:35.531849 26516 raft_consensus.cc:493] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:35.532225 26234 consensus_queue.cc:1048] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
I20260709 12:24:35.532568 26516 raft_consensus.cc:515] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:35.535399 26516 leader_election.cc:290] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:35.535580 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 25 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:35.535846 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:35.536690 17764 raft_consensus.cc:2393] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 23: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:24:35.537411 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:35.539057 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:35.540261 17689 raft_consensus.cc:2468] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 24.
I20260709 12:24:35.541283 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:35.541937 17755 raft_consensus.cc:2468] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 24.
I20260709 12:24:35.542378 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:24:35.543586 17654 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:35.544131 17764 raft_consensus.cc:1275] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 23 index: 17. Preceding OpId from leader: term: 24 index: 18. (index mismatch)
I20260709 12:24:35.547345 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:35.547840 26483 raft_consensus.cc:493] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:35.548791 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:35.548636 26483 raft_consensus.cc:515] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:35.548213 18364 raft_consensus.cc:2393] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 21: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:24:35.549592 18371 raft_consensus.cc:1275] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 16 index: 13. Preceding OpId from leader: term: 18 index: 14. (index mismatch)
I20260709 12:24:35.552500 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:35.553300 18364 raft_consensus.cc:1240] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 19. Current term is 20. Ops: []
I20260709 12:24:35.544148 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:35.554137 18366 raft_consensus.cc:2393] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 21: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:24:35.558588 17763 raft_consensus.cc:1240] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 26. Current term is 27. Ops: []
I20260709 12:24:35.559500 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:35.560446 17755 raft_consensus.cc:2468] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 27.
I20260709 12:24:35.562063 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 28 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:35.562013 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:35.536674 17757 raft_consensus.cc:2393] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 25: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:24:35.565268 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:35.566843 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:35.567680 17763 raft_consensus.cc:2468] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 23.
I20260709 12:24:35.568997 18365 raft_consensus.cc:2468] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 22.
I20260709 12:24:35.569157 17729 leader_election.cc:304] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:35.569962 26281 raft_consensus.cc:2749] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 12:24:35.570255 17833 raft_consensus.cc:2468] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 21.
I20260709 12:24:35.577073 26488 raft_consensus.cc:515] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:35.577971 17689 raft_consensus.cc:2468] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 20.
I20260709 12:24:35.581660 17728 leader_election.cc:304] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:35.582938 26281 raft_consensus.cc:2749] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260709 12:24:35.587296 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:35.588114 26498 consensus_queue.cc:1059] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 22.14, Next index: 15, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:24:35.589444 26164 raft_consensus.cc:3055] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:24:35.589783 26164 raft_consensus.cc:740] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 22 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:24:35.589886 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:35.590691 26164 consensus_queue.cc:260] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 22.14, Last appended by leader: 14, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:35.591341 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:35.593724 26399 raft_consensus.cc:493] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
W20260709 12:24:35.592525 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 68339.1us
I20260709 12:24:35.593046 26536 raft_consensus.cc:2804] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:24:35.590157 17755 raft_consensus.cc:1275] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 21 index: 13. Preceding OpId from leader: term: 23 index: 14. (index mismatch)
I20260709 12:24:35.596089 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:35.596529 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:35.594470 18363 raft_consensus.cc:2468] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 21.
I20260709 12:24:35.597349 17831 raft_consensus.cc:1275] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 23 index: 17. Preceding OpId from leader: term: 24 index: 18. (index mismatch)
I20260709 12:24:35.600369 17802 leader_election.cc:304] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:35.596982 18364 raft_consensus.cc:1275] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 24 index: 17. Preceding OpId from leader: term: 26 index: 18. (index mismatch)
I20260709 12:24:35.605504 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 22 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:35.613749 17729 leader_election.cc:304] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:35.620364 17832 raft_consensus.cc:2468] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 23.
I20260709 12:24:35.623423 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:35.628679 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:35.629117 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:35.634639 17802 leader_election.cc:304] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:24:35.627377 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 25 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:35.634497 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:35.594411 26399 raft_consensus.cc:515] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:35.625638 26302 raft_consensus.cc:2804] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:24:35.634095 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 20 candidate_status { last_received { term: 18 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:35.641368 17802 leader_election.cc:304] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:24:35.527573 26334 raft_consensus.cc:515] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:35.605109 26516 raft_consensus.cc:493] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:35.617908 26483 leader_election.cc:290] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:35.650175 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:35.645705 26302 raft_consensus.cc:493] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
W20260709 12:24:35.652103 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:35.652338 26302 raft_consensus.cc:3060] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:35.653599 26399 leader_election.cc:290] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:35.625800 26536 raft_consensus.cc:697] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 20 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
W20260709 12:24:35.655149 17655 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:35.655462 26536 consensus_queue.cc:237] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.13, Last appended by leader: 13, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:35.657255 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:35.658264 17687 raft_consensus.cc:2410] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for term 26 because replica has last-logged OpId of term: 25 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 14.
I20260709 12:24:35.658396 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:35.634335 26421 raft_consensus.cc:2749] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 12:24:35.659845 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 25 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:35.660857 26164 raft_consensus.cc:3060] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:35.661132 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:35.661998 18374 raft_consensus.cc:2393] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 20: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:24:35.662828 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:35.664417 26527 raft_consensus.cc:2749] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Leader pre-election lost for term 22. Reason: Vote denied by peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651) with higher term. Message: Invalid argument: T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 22. Current term is 23.
I20260709 12:24:35.665089 25740 raft_consensus.cc:2804] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:24:35.665534 25740 raft_consensus.cc:493] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:35.665948 25740 raft_consensus.cc:3060] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:35.666448 17688 raft_consensus.cc:3060] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:35.667140 17727 leader_election.cc:304] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:35.655098 17762 raft_consensus.cc:1275] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 23 index: 14. Preceding OpId from leader: term: 27 index: 16. (index mismatch)
W20260709 12:24:35.670373 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:35.670855 17803 leader_election.cc:304] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:35.647141 17729 leader_election.cc:304] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:35.664628 17760 raft_consensus.cc:2393] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 28: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:24:35.659536 17687 raft_consensus.cc:1240] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 21. Current term is 22. Ops: []
I20260709 12:24:35.678194 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 23 candidate_status { last_received { term: 19 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:35.678406 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:35.664752 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:35.682524 17830 raft_consensus.cc:2393] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 23: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:24:35.682228 26354 raft_consensus.cc:2804] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:24:35.683599 26354 raft_consensus.cc:697] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 21 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:24:35.684906 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:35.685750 26302 raft_consensus.cc:515] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:35.685875 17681 raft_consensus.cc:2468] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 24.
I20260709 12:24:35.685472 26354 consensus_queue.cc:237] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 19.14, Last appended by leader: 14, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:35.687496 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:35.688020 26302 leader_election.cc:290] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 24 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:35.688159 17685 raft_consensus.cc:2468] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 23.
I20260709 12:24:35.689518 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:24:35.690222 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 59495.9us
I20260709 12:24:35.679364 25866 raft_consensus.cc:493] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:35.691159 26234 raft_consensus.cc:2804] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:24:35.691761 26234 raft_consensus.cc:697] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:24:35.692847 26234 consensus_queue.cc:237] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 21.15, Last appended by leader: 15, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:35.695238 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:35.695799 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:35.696173 25784 consensus_queue.cc:1048] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:24:35.696398 26516 raft_consensus.cc:515] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:35.697016 26281 consensus_queue.cc:1048] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
I20260709 12:24:35.696566 17683 raft_consensus.cc:2393] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 24: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:24:35.690965 25866 raft_consensus.cc:515] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:35.700373 25866 leader_election.cc:290] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:35.701963 26334 leader_election.cc:290] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 19 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:35.703007 17836 raft_consensus.cc:1240] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 18. Current term is 19. Ops: []
I20260709 12:24:35.704676 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:35.710491 18333 leader_election.cc:304] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:24:35.712713 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 75993.1us
I20260709 12:24:35.711236 17803 leader_election.cc:304] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:24:35.711789 26544 consensus_queue.cc:1048] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20260709 12:24:35.718183 17680 raft_consensus.cc:1275] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 24 index: 17. Preceding OpId from leader: term: 26 index: 18. (index mismatch)
I20260709 12:24:35.718904 26450 raft_consensus.cc:2804] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:24:35.719228 26398 raft_consensus.cc:2749] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:24:35.730680 18333 leader_election.cc:304] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:35.720880 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:24:35.732115 26519 consensus_meta.cc:287] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.601s	user 0.014s	sys 0.001s
I20260709 12:24:35.732923 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:35.734774 17761 raft_consensus.cc:1275] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 25 index: 17. Preceding OpId from leader: term: 30 index: 20. (index mismatch)
I20260709 12:24:35.736609 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:35.737349 17685 raft_consensus.cc:2468] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 28.
I20260709 12:24:35.742334 25740 raft_consensus.cc:515] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:35.727049 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 19 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:35.727972 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:35.728813 18362 raft_consensus.cc:1275] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 21 index: 13. Preceding OpId from leader: term: 23 index: 14. (index mismatch)
I20260709 12:24:35.752465 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:35.720052 17835 raft_consensus.cc:1275] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 21 index: 16. Preceding OpId from leader: term: 23 index: 17. (index mismatch)
I20260709 12:24:35.731168 17803 leader_election.cc:304] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:35.743718 17802 leader_election.cc:304] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:24:35.746030 26447 raft_consensus.cc:2749] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:24:35.726697 18374 raft_consensus.cc:1275] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 21 index: 16. Preceding OpId from leader: term: 23 index: 17. (index mismatch)
I20260709 12:24:35.747009 26421 raft_consensus.cc:493] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:35.748076 26334 raft_consensus.cc:2804] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:24:35.748463 26281 raft_consensus.cc:493] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:24:35.699440 26516 leader_election.cc:290] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:35.647141 26443 raft_consensus.cc:2804] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:24:35.754887 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:35.759604 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:35.760608 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:35.729439 26450 raft_consensus.cc:493] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:35.695225 26332 raft_consensus.cc:515] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Starting leader election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:24:35.752327 26483 consensus_queue.cc:1048] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.019s
I20260709 12:24:35.785256 17837 raft_consensus.cc:2468] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 25.
I20260709 12:24:35.791975 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:35.807057 18365 raft_consensus.cc:2468] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 22.
I20260709 12:24:35.803552 18362 raft_consensus.cc:1240] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 19. Current term is 20. Ops: []
I20260709 12:24:35.803123 17757 raft_consensus.cc:2468] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 21.
I20260709 12:24:35.763485 17688 raft_consensus.cc:2468] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 25.
I20260709 12:24:35.758167 26498 raft_consensus.cc:2749] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:24:35.764158 25866 raft_consensus.cc:2804] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:24:35.814383 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 19 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:35.815299 18361 raft_consensus.cc:3060] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:35.813114 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:35.769811 26450 raft_consensus.cc:3060] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:35.769965 26537 raft_consensus.cc:493] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:35.828779 17683 raft_consensus.cc:1275] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 28 index: 19. Preceding OpId from leader: term: 30 index: 20. (index mismatch)
I20260709 12:24:35.816689 17686 raft_consensus.cc:2468] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 23.
I20260709 12:24:35.821944 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:35.771430 26443 raft_consensus.cc:493] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:35.773339 26334 raft_consensus.cc:697] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 22 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:24:35.832862 25740 leader_election.cc:290] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 20 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:35.832789 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:35.839725 17760 raft_consensus.cc:2393] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 24: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:24:35.840790 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:35.810770 25866 raft_consensus.cc:493] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:35.820715 26368 raft_consensus.cc:493] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:35.846043 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:35.830827 26340 raft_consensus.cc:493] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:35.831290 26302 raft_consensus.cc:493] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:35.836826 26443 raft_consensus.cc:3060] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:35.816114 26281 raft_consensus.cc:515] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:35.848296 25866 raft_consensus.cc:3060] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:24:35.873898 26302 raft_consensus.cc:515] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:35.876387 26340 raft_consensus.cc:515] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:35.877360 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 23 candidate_status { last_received { term: 19 index: 17 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:35.874552 17761 raft_consensus.cc:2468] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 24.
I20260709 12:24:35.843858 25784 raft_consensus.cc:493] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:35.813827 26421 raft_consensus.cc:515] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:35.883519 17802 leader_election.cc:304] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:24:35.890275 25740 raft_consensus.cc:2804] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:24:35.890719 25740 raft_consensus.cc:493] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:35.891072 25740 raft_consensus.cc:3060] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:35.894012 17801 leader_election.cc:304] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:35.906361 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 20 candidate_status { last_received { term: 19 index: 19 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:35.906610 17801 leader_election.cc:304] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:35.901057 17681 raft_consensus.cc:2468] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 24.
I20260709 12:24:35.909875 17685 raft_consensus.cc:2393] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 23: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:24:35.886243 26420 raft_consensus.cc:493] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:35.858121 26537 raft_consensus.cc:515] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:35.928648 26332 leader_election.cc:290] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 27 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:35.937589 18334 leader_election.cc:304] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:35.932370 17802 leader_election.cc:304] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
W20260709 12:24:35.941962 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 60559.5us
I20260709 12:24:35.911120 25784 raft_consensus.cc:515] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:35.955618 17831 raft_consensus.cc:1275] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 24 index: 19. Preceding OpId from leader: term: 25 index: 19. (term mismatch)
I20260709 12:24:35.956179 17831 pending_rounds.cc:85] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b: Aborting all ops after (but not including) 18
I20260709 12:24:35.955986 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 27 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:35.956524 17831 pending_rounds.cc:107] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b: Aborting uncommitted NO_OP operation due to leader change: 24.19
I20260709 12:24:35.956832 17835 raft_consensus.cc:3060] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:35.956918 17831 raft_consensus.cc:2889] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:35.958770 25784 leader_election.cc:290] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:35.960235 25784 consensus_queue.cc:1048] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20260709 12:24:35.960767 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:35.961573 18371 raft_consensus.cc:2393] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 20: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:24:35.933856 26239 raft_consensus.cc:2804] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:24:35.895180 26368 raft_consensus.cc:515] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:35.911042 26538 consensus_queue.cc:1048] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 17, Time since last communication: 0.167s
I20260709 12:24:35.966408 18333 leader_election.cc:304] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:35.989970 18361 raft_consensus.cc:2468] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 19.
I20260709 12:24:35.967727 26479 raft_consensus.cc:2804] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:24:35.999784 17757 raft_consensus.cc:2393] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 28: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:24:35.969029 26398 raft_consensus.cc:493] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:35.937402 26420 raft_consensus.cc:515] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:35.981114 26239 raft_consensus.cc:697] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 21 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:24:35.989104 26294 consensus_queue.cc:1048] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.019s
I20260709 12:24:35.984534 26234 raft_consensus.cc:493] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:36.020779 26569 log.cc:927] Time spent T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.093s	user 0.001s	sys 0.000s
I20260709 12:24:36.022799 17652 leader_election.cc:400] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 29 election: Vote denied by peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651) with higher term. Message: Invalid argument: T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for earlier term 29. Current term is 30.
I20260709 12:24:36.023924 26499 raft_consensus.cc:2749] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:24:35.968595 26397 consensus_queue.cc:1059] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 19.19, Next index: 20, Last known committed idx: 18, Time since last communication: 0.037s
W20260709 12:24:36.025534 26570 log.cc:927] Time spent T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.113s	user 0.000s	sys 0.001s
I20260709 12:24:36.026176 26281 leader_election.cc:290] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:36.026813 26281 consensus_queue.cc:1048] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20260709 12:24:36.029846 26579 consensus_queue.cc:1048] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:24:36.007141 26479 raft_consensus.cc:697] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 24 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:24:36.044620 26479 consensus_queue.cc:237] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 23.13, Last appended by leader: 13, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:35.947311 26334 consensus_queue.cc:237] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 21.15, Last appended by leader: 15, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:36.048532 26340 leader_election.cc:290] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:36.049147 26340 raft_consensus.cc:2749] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 12:24:36.049737 26340 raft_consensus.cc:2749] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
W20260709 12:24:36.023828 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 139075us
I20260709 12:24:36.008416 26398 raft_consensus.cc:515] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:36.057773 26294 consensus_queue.cc:1059] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 27.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20260709 12:24:36.040647 26302 leader_election.cc:290] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:36.042634 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 28 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:36.038882 26239 consensus_queue.cc:237] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 20.14, Last appended by leader: 14, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:36.066054 17838 raft_consensus.cc:2468] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 28.
I20260709 12:24:36.064870 17835 raft_consensus.cc:2468] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 27.
I20260709 12:24:36.062656 17682 raft_consensus.cc:2468] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 20.
I20260709 12:24:36.018102 25740 raft_consensus.cc:515] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:36.058391 26398 leader_election.cc:290] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:36.070124 17802 leader_election.cc:304] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:36.103183 25740 leader_election.cc:290] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 25 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:24:36.105149 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 106609us
W20260709 12:24:36.113961 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:36.071307 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 23 candidate_status { last_received { term: 21 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:36.043385 26443 raft_consensus.cc:515] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:36.124575 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 23 candidate_status { last_received { term: 21 index: 18 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:24:36.128654 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 58620.1us
W20260709 12:24:36.125849 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 156731us
I20260709 12:24:36.127768 18364 raft_consensus.cc:2468] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 22.
I20260709 12:24:36.140271 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:36.140852 17682 raft_consensus.cc:3060] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Advancing to term 25
I20260709 12:24:36.144871 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:36.145599 17838 raft_consensus.cc:2368] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for earlier term 24. Current term is 25.
I20260709 12:24:36.131191 26527 raft_consensus.cc:493] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:36.155570 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 97499us
I20260709 12:24:36.065639 26234 raft_consensus.cc:515] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:36.139698 26446 raft_consensus.cc:2804] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:24:36.158448 17727 leader_election.cc:304] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:36.159710 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 23 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:36.165721 17728 leader_election.cc:304] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
W20260709 12:24:36.169704 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 235616us
I20260709 12:24:36.163740 25784 raft_consensus.cc:2804] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:24:36.170671 18334 leader_election.cc:304] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:36.173300 26538 raft_consensus.cc:2804] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:24:36.173718 26538 raft_consensus.cc:493] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:36.174160 26538 raft_consensus.cc:3060] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:36.174559 26582 consensus_queue.cc:1059] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:24:36.175432 26582 consensus_queue.cc:1243] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer 0da169ff82674dc1947055ce43de68a3 log is divergent from this leader: its last log entry 20.16 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 15
I20260709 12:24:36.176433 18334 leader_election.cc:304] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:36.181160 18334 leader_election.cc:304] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:24:36.182749 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 243368us
W20260709 12:24:36.211608 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 210567us
I20260709 12:24:36.182561 26436 consensus_queue.cc:1048] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20260709 12:24:36.184080 26332 raft_consensus.cc:2749] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:24:36.184464 26447 raft_consensus.cc:3055] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 21 LEADER]: Stepping down as leader of term 21
W20260709 12:24:36.184943 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:36.185243 26544 consensus_queue.cc:1048] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
I20260709 12:24:36.185431 25784 raft_consensus.cc:697] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 27 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:24:36.179466 26446 raft_consensus.cc:697] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 20 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:24:36.220371 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 20 candidate_status { last_received { term: 19 index: 19 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:36.186734 26450 raft_consensus.cc:515] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:36.187803 26234 leader_election.cc:290] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:36.188179 26483 raft_consensus.cc:2749] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:24:36.189692 26368 leader_election.cc:290] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:36.190423 26537 leader_election.cc:290] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:36.191289 17652 leader_election.cc:304] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:36.245748 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 29 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:36.191685 26421 leader_election.cc:290] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:36.193573 26443 leader_election.cc:290] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 25 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:36.158710 26527 raft_consensus.cc:515] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:36.202705 26488 leader_election.cc:290] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 20 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:36.205505 26399 raft_consensus.cc:3055] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:24:36.233002 26398 raft_consensus.cc:493] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:36.217214 17830 raft_consensus.cc:1275] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 21 index: 16. Preceding OpId from leader: term: 23 index: 17. (index mismatch)
W20260709 12:24:36.195578 26164 consensus_meta.cc:287] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.534s	user 0.009s	sys 0.000s
I20260709 12:24:36.222123 17682 raft_consensus.cc:2468] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 25.
I20260709 12:24:36.186825 25866 raft_consensus.cc:515] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:36.237082 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 24 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:36.231563 26587 raft_consensus.cc:2749] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:24:36.240231 17833 raft_consensus.cc:1275] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 24 index: 15. Preceding OpId from leader: term: 27 index: 17. (index mismatch)
I20260709 12:24:36.240124 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 29 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:36.241235 25784 consensus_queue.cc:237] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 24.14, Last appended by leader: 15, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:24:36.245427 26450 leader_election.cc:290] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 24 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:36.246666 17837 raft_consensus.cc:2410] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for term 29 because replica has last-logged OpId of term: 27 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 26 index: 14.
I20260709 12:24:36.215816 26447 raft_consensus.cc:740] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
W20260709 12:24:36.249665 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 58526.2us
W20260709 12:24:36.215085 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 55513.1us
I20260709 12:24:36.252435 26399 raft_consensus.cc:740] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 19 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:24:36.256199 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:36.242616 18363 raft_consensus.cc:2468] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 24.
I20260709 12:24:36.267114 17652 leader_election.cc:304] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:36.262981 17802 leader_election.cc:304] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:24:36.272992 17756 raft_consensus.cc:2393] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 27: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:24:36.273298 17803 leader_election.cc:304] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:24:36.274507 17803 leader_election.cc:304] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:24:36.256243 26420 leader_election.cc:290] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:36.275748 17831 raft_consensus.cc:2410] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for term 29 because replica has last-logged OpId of term: 27 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 26 index: 15.
I20260709 12:24:36.280761 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 28 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:36.281478 17761 raft_consensus.cc:2468] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 27.
I20260709 12:24:36.282315 17834 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 23.
I20260709 12:24:36.286336 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
W20260709 12:24:36.285840 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 142577us
I20260709 12:24:36.294126 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:36.278522 26164 raft_consensus.cc:2804] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:24:36.295780 17801 leader_election.cc:304] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:24:36.297955 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:36.298189 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:36.278849 26516 raft_consensus.cc:2804] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:24:36.298756 17762 raft_consensus.cc:3060] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Advancing to term 25
I20260709 12:24:36.298852 17761 raft_consensus.cc:3060] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:36.283744 26397 raft_consensus.cc:2804] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:24:36.264438 26398 raft_consensus.cc:515] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:36.286773 26450 raft_consensus.cc:493] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:36.324200 26526 raft_consensus.cc:2804] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:24:36.324697 26526 raft_consensus.cc:493] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:36.325048 26526 raft_consensus.cc:3060] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:24:36.300817 26516 raft_consensus.cc:493] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:36.333997 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 26 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:36.335469 17762 raft_consensus.cc:2468] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 25.
I20260709 12:24:36.335570 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 23 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:36.318275 17728 leader_election.cc:304] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
W20260709 12:24:36.336516 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 64159.9us
I20260709 12:24:36.303826 26488 raft_consensus.cc:493] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:36.303143 26397 raft_consensus.cc:493] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:36.349889 26397 raft_consensus.cc:3060] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Advancing to term 23
W20260709 12:24:36.350345 26595 log.cc:927] Time spent T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.053s	user 0.000s	sys 0.001s
I20260709 12:24:36.354434 26399 consensus_queue.cc:260] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 19.19, Last appended by leader: 19, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:36.310137 26398 leader_election.cc:290] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:36.329828 26544 raft_consensus.cc:493] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:24:36.341972 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:36.341992 18365 raft_consensus.cc:2468] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 25.
I20260709 12:24:36.344142 17761 raft_consensus.cc:2468] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 25.
I20260709 12:24:36.328668 26598 consensus_queue.cc:1048] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.008s
I20260709 12:24:36.359229 18362 raft_consensus.cc:1275] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 21 index: 16. Preceding OpId from leader: term: 23 index: 17. (index mismatch)
I20260709 12:24:36.317802 26164 raft_consensus.cc:697] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 25 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:24:36.357371 18367 raft_consensus.cc:2468] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 21.
I20260709 12:24:36.356570 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 26 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:36.339516 26516 raft_consensus.cc:3060] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:36.366916 17803 leader_election.cc:304] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:24:36.368578 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:36.364612 17801 leader_election.cc:304] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:36.369448 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 24 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:36.371687 18365 raft_consensus.cc:2468] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 22.
I20260709 12:24:36.356323 26544 raft_consensus.cc:515] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:36.412122 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:36.413834 18365 raft_consensus.cc:1275] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 22 index: 18. Preceding OpId from leader: term: 25 index: 19. (index mismatch)
I20260709 12:24:36.421994 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 25 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:36.425704 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 25 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:24:36.426693 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 62171us
I20260709 12:24:36.319375 26587 consensus_queue.cc:1048] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.049s
I20260709 12:24:36.438107 25866 leader_election.cc:290] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 28 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:36.339649 26294 consensus_queue.cc:1048] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.033s
I20260709 12:24:36.331054 26450 raft_consensus.cc:515] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:36.430788 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 63961.2us
W20260709 12:24:36.435598 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 117384us
I20260709 12:24:36.444691 17728 leader_election.cc:304] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:36.445958 17728 leader_election.cc:304] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:36.447058 17728 leader_election.cc:304] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:36.444721 17803 leader_election.cc:304] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:36.319144 26446 consensus_queue.cc:237] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 19.13, Last appended by leader: 13, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:36.425979 26604 raft_consensus.cc:493] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:36.332108 26447 consensus_queue.cc:260] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 21.16, Last appended by leader: 16, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:36.359172 26488 raft_consensus.cc:515] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:36.462646 17755 raft_consensus.cc:2468] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 24.
I20260709 12:24:36.438983 26239 raft_consensus.cc:2804] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:24:36.372632 26597 raft_consensus.cc:2804] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:24:36.442000 26587 raft_consensus.cc:493] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:36.472870 26234 raft_consensus.cc:2749] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:24:36.483737 18367 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 23.
I20260709 12:24:36.481640 26580 consensus_queue.cc:1048] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.010s
I20260709 12:24:36.489245 26581 raft_consensus.cc:2749] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
W20260709 12:24:36.493564 26572 log.cc:927] Time spent T 00000000000000000000000000000000 P 2193af31cb894ec9b3d65d939a2a6851: Append to log took a long time: real 0.142s	user 0.006s	sys 0.000s
W20260709 12:24:36.495910 26591 log.cc:927] Time spent T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.178s	user 0.000s	sys 0.001s
W20260709 12:24:36.497254 26590 log.cc:927] Time spent T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.215s	user 0.001s	sys 0.000s
I20260709 12:24:36.497848 26601 raft_consensus.cc:2804] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:24:36.498407 26601 raft_consensus.cc:493] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:36.498807 26601 raft_consensus.cc:3060] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:36.474012 26443 raft_consensus.cc:2749] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:24:36.472035 26599 raft_consensus.cc:2804] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Leader election won for term 19
W20260709 12:24:36.545267 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 72631.5us
I20260709 12:24:36.548424 26580 raft_consensus.cc:493] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:36.487401 26597 raft_consensus.cc:493] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:36.530128 26527 leader_election.cc:290] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:36.553315 26239 raft_consensus.cc:697] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 25 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
W20260709 12:24:36.591809 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 242693us
I20260709 12:24:36.563200 26597 raft_consensus.cc:3060] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:36.485029 26604 raft_consensus.cc:515] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:36.532047 26587 raft_consensus.cc:515] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:36.582438 26594 raft_consensus.cc:2804] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:24:36.608968 18366 raft_consensus.cc:1275] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 18 index: 13. Preceding OpId from leader: term: 20 index: 14. (index mismatch)
I20260709 12:24:36.613538 26612 raft_consensus.cc:493] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:36.614259 26612 raft_consensus.cc:515] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:36.618562 26239 consensus_queue.cc:237] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 23.15, Last appended by leader: 15, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:24:36.622632 26608 log.cc:927] Time spent T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.052s	user 0.000s	sys 0.000s
I20260709 12:24:36.582787 26611 raft_consensus.cc:493] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:24:36.636309 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:36.638469 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:36.639503 17688 raft_consensus.cc:2393] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 24: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:24:36.566246 26580 raft_consensus.cc:515] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:36.560717 26164 consensus_queue.cc:237] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 23.16, Last appended by leader: 16, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:36.641695 18334 leader_election.cc:304] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:36.632782 26599 raft_consensus.cc:697] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 19 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:24:36.642418 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:36.645151 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:36.605145 26294 raft_consensus.cc:493] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:36.648249 17689 raft_consensus.cc:2468] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 27.
I20260709 12:24:36.652765 18334 leader_election.cc:304] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:36.655455 17687 raft_consensus.cc:2393] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 23: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:24:36.659834 17685 raft_consensus.cc:1240] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 21. Current term is 22. Ops: [21.16-21.16]
I20260709 12:24:36.604441 26538 raft_consensus.cc:515] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:36.659639 26527 raft_consensus.cc:493] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:36.665766 26575 log.cc:927] Time spent T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.069s	user 0.001s	sys 0.000s
I20260709 12:24:36.674872 17682 raft_consensus.cc:1275] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 26 index: 16. Preceding OpId from leader: term: 27 index: 17. (index mismatch)
I20260709 12:24:36.679580 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:36.680627 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 29 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:36.681433 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 24 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:36.681795 17683 raft_consensus.cc:2410] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for term 29 because replica has last-logged OpId of term: 27 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 26 index: 15.
I20260709 12:24:36.682126 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 29 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:36.682943 17681 raft_consensus.cc:2410] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for term 29 because replica has last-logged OpId of term: 27 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 26 index: 14.
I20260709 12:24:36.662369 26544 leader_election.cc:290] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:36.661799 26234 raft_consensus.cc:493] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:36.667245 26599 consensus_queue.cc:237] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 16.13, Last appended by leader: 13, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:36.691807 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 27 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:36.693687 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:36.694473 17682 raft_consensus.cc:2393] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 24: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:24:36.702452 26580 leader_election.cc:290] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:36.702692 26618 consensus_queue.cc:1048] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.041s
I20260709 12:24:36.704206 17833 raft_consensus.cc:1275] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 19 index: 14. Preceding OpId from leader: term: 21 index: 15. (index mismatch)
I20260709 12:24:36.705129 26604 leader_election.cc:290] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:36.643049 26611 raft_consensus.cc:515] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:36.661149 26294 raft_consensus.cc:515] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
W20260709 12:24:36.716305 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:36.694208 26488 leader_election.cc:290] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:36.674783 26527 raft_consensus.cc:515] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:36.724941 18334 leader_election.cc:304] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:36.737092 17685 raft_consensus.cc:3060] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:36.729466 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 24 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:36.733458 26587 leader_election.cc:290] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:36.734200 26450 leader_election.cc:290] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:36.715198 26604 raft_consensus.cc:3060] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:36.714884 26234 raft_consensus.cc:515] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:36.741746 18334 leader_election.cc:304] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:36.741746 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:36.750741 26527 leader_election.cc:290] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:36.751217 26625 raft_consensus.cc:2749] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 12:24:36.757629 17683 raft_consensus.cc:2468] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 27.
W20260709 12:24:36.761472 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 87563.1us
I20260709 12:24:36.763381 18333 leader_election.cc:304] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:24:36.713214 26526 raft_consensus.cc:515] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:36.759667 26450 raft_consensus.cc:2749] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:24:36.926517 26627 consensus_queue.cc:1059] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:24:36.927563 26627 consensus_queue.cc:1243] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Peer 0da169ff82674dc1947055ce43de68a3 log is divergent from this leader: its last log entry 20.16 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 15
I20260709 12:24:36.945914 17837 raft_consensus.cc:1240] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 27. Current term is 28. Ops: []
I20260709 12:24:37.020406 17652 leader_election.cc:304] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:37.080399 17685 raft_consensus.cc:2410] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for term 27 because replica has last-logged OpId of term: 25 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 14.
I20260709 12:24:37.080875 17757 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38786 (request call id 3877) took 1268 ms. Trace:
I20260709 12:24:37.084424 26627 raft_consensus.cc:2804] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Leader election won for term 24
W20260709 12:24:37.084450 18371 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.623s	user 0.000s	sys 0.008s
W20260709 12:24:37.085129 18371 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.623s	user 0.000s	sys 0.008s
I20260709 12:24:36.857378 26625 raft_consensus.cc:2804] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:24:37.085765 26625 raft_consensus.cc:493] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:37.086139 26625 raft_consensus.cc:3060] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:24:37.087714 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 324422us
I20260709 12:24:36.857714 26632 raft_consensus.cc:493] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:37.088958 26632 raft_consensus.cc:515] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:37.090803 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 343186us
I20260709 12:24:36.929343 26580 raft_consensus.cc:493] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:24:37.092530 26580 raft_consensus.cc:515] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:37.095494 26580 leader_election.cc:290] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:37.085153 26627 raft_consensus.cc:697] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 24 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:24:37.096879 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:37.097460 26627 consensus_queue.cc:237] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 23.17, Last appended by leader: 17, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:37.086999 18335 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.519s	user 0.004s	sys 0.004s
W20260709 12:24:37.099018 18335 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.533s	user 0.004s	sys 0.004s
I20260709 12:24:37.081254 17757 rpcz_store.cc:276] 0709 12:24:35.812210 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:35.812429 (+   219us) service_pool.cc:224] Handling call
0709 12:24:37.080645 (+1268216us) spinlock_profiling.cc:243] Waited 177 ms on lock 0x7b540019fad4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:24:37.080810 (+   165us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":406037760}
I20260709 12:24:36.856671 26612 leader_election.cc:290] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:36.842285 26597 raft_consensus.cc:515] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:37.106849 17728 leader_election.cc:304] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:37.036628 26446 raft_consensus.cc:493] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:37.091807 26632 leader_election.cc:290] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:36.946862 26516 raft_consensus.cc:515] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:37.131299 26597 leader_election.cc:290] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 27 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:37.121467 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:37.133049 17757 raft_consensus.cc:2468] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 21.
I20260709 12:24:37.121416 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 24 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:37.127632 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:24:37.128541 26397 consensus_meta.cc:287] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.778s	user 0.011s	sys 0.000s
I20260709 12:24:37.043803 26450 consensus_queue.cc:1048] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.079s
I20260709 12:24:37.134227 17758 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 23.
I20260709 12:24:37.128604 26588 raft_consensus.cc:493] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
W20260709 12:24:37.129310 26605 log.cc:927] Time spent T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.184s	user 0.001s	sys 0.000s
I20260709 12:24:37.100044 26368 raft_consensus.cc:2749] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 27 LEADER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:24:37.145955 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:37.149154 17834 raft_consensus.cc:2468] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 23.
I20260709 12:24:37.123235 26519 raft_consensus.cc:2804] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:24:37.155337 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:37.138150 26538 leader_election.cc:290] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 23 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:37.132086 26516 leader_election.cc:290] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 24 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:37.135388 26446 raft_consensus.cc:515] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:37.207002 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 354196us
I20260709 12:24:37.147931 26332 raft_consensus.cc:2804] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:24:37.210345 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:24:37.152825 25866 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.577s	user 0.023s	sys 0.034s
I20260709 12:24:37.207809 17652 leader_election.cc:304] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:37.147421 26397 raft_consensus.cc:515] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:37.226573 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 464395us
I20260709 12:24:37.234508 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:37.207096 26635 raft_consensus.cc:493] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:37.240531 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:37.242429 26611 leader_election.cc:290] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:37.230476 26641 raft_consensus.cc:493] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:37.248497 26637 raft_consensus.cc:2749] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:24:37.211637 26519 raft_consensus.cc:697] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 23 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:24:37.252979 17835 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35042 (request call id 3214) took 1118 ms. Trace:
I20260709 12:24:37.150701 26588 raft_consensus.cc:515] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:37.262825 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 24 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:37.254503 26446 leader_election.cc:290] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:37.210731 26332 raft_consensus.cc:493] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:37.252581 26234 leader_election.cc:290] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:37.260411 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
W20260709 12:24:37.235390 26601 consensus_meta.cc:287] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.736s	user 0.012s	sys 0.000s
I20260709 12:24:37.264681 17830 raft_consensus.cc:1240] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 22. Current term is 23. Ops: []
W20260709 12:24:37.225088 25866 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.654s	user 0.023s	sys 0.034s
I20260709 12:24:37.229063 26538 raft_consensus.cc:493] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:37.270113 17834 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 23.
W20260709 12:24:37.268527 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 60799.8us
I20260709 12:24:37.225660 26294 leader_election.cc:290] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:37.268772 17760 raft_consensus.cc:2393] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 23: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:24:37.275408 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:37.261787 26483 raft_consensus.cc:493] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:37.265498 26332 raft_consensus.cc:3060] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:24:37.281333 17834 raft_consensus.cc:2393] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 25: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:24:37.262744 17835 rpcz_store.cc:276] 0709 12:24:36.134252 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:36.134382 (+   130us) service_pool.cc:224] Handling call
0709 12:24:36.134708 (+   326us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:36.180537 (+ 45829us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:36.180584 (+    47us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:36.215059 (+ 34475us) log.cc:844] Serialized 26 byte log entry
0709 12:24:36.348351 (+133292us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:24:36.357499 (+  9148us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:36.357576 (+    77us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:37.252733 (+895157us) raft_consensus.cc:1685] finished
0709 12:24:37.252755 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:37.252896 (+   141us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":13337,"raft.queue_time_us":21133,"raft.run_cpu_time_us":103,"raft.run_wall_time_us":103,"thread_start_us":79755,"threads_started":1,"wal-append.queue_time_us":186756}
I20260709 12:24:37.269099 26526 leader_election.cc:290] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 28 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:37.288007 17652 leader_election.cc:304] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:37.292488 26519 consensus_queue.cc:237] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 21.18, Last appended by leader: 18, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:37.296023 26294 raft_consensus.cc:2749] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:24:37.264853 26641 raft_consensus.cc:515] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:37.297751 26637 consensus_queue.cc:1048] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.023s
I20260709 12:24:37.291544 26483 raft_consensus.cc:515] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:37.301010 26397 leader_election.cc:290] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 23 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:37.319293 26588 leader_election.cc:290] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:37.269487 26635 raft_consensus.cc:515] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:37.322795 26604 consensus_meta.cc:287] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.563s	user 0.008s	sys 0.000s
I20260709 12:24:37.369576 26604 raft_consensus.cc:2749] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:24:37.274602 26538 raft_consensus.cc:515] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:37.370877 26483 leader_election.cc:290] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:37.371801 26538 leader_election.cc:290] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:37.326395 26399 raft_consensus.cc:3060] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:37.373044 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:37.373919 17682 raft_consensus.cc:2393] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 25: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:24:37.373569 26332 raft_consensus.cc:515] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:37.374863 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:37.334478 26625 raft_consensus.cc:515] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:37.335415 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:37.341085 26447 raft_consensus.cc:3060] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:37.342379 26594 raft_consensus.cc:493] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:37.342911 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 27 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:37.345175 18333 leader_election.cc:304] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:24:37.353173 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:37.353827 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:37.382313 17688 raft_consensus.cc:2468] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 27.
I20260709 12:24:37.298892 17652 leader_election.cc:304] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:37.270020 26601 raft_consensus.cc:515] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:37.357980 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:37.364905 18367 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59886 (request call id 2338) took 1044 ms. Trace:
I20260709 12:24:37.367226 18365 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59886 (request call id 2336) took 1047 ms. Trace:
I20260709 12:24:37.377266 17681 raft_consensus.cc:3060] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:37.385346 17727 leader_election.cc:400] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 24 pre-election: Vote denied by peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837) with higher term. Message: Invalid argument: T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for earlier term 24. Current term is 25.
I20260709 12:24:37.383893 17802 leader_election.cc:304] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:24:37.388430 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:24:37.421931 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 123159us
I20260709 12:24:37.423336 17652 leader_election.cc:304] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:37.423455 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 23 candidate_status { last_received { term: 21 index: 18 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:37.388177 26588 raft_consensus.cc:493] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:37.429809 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:37.431365 17833 raft_consensus.cc:1240] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 25. Current term is 26. Ops: [25.19-25.19]
I20260709 12:24:37.429936 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:37.432775 17688 raft_consensus.cc:2368] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 23. Current term is 24.
I20260709 12:24:37.392450 26627 raft_consensus.cc:2804] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:24:37.388321 26354 raft_consensus.cc:493] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:37.440316 17727 leader_election.cc:403] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 24 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:24:37.443853 17652 leader_election.cc:304] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:37.388023 17680 raft_consensus.cc:2468] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 20.
I20260709 12:24:37.436292 26538 raft_consensus.cc:493] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:37.428208 26619 raft_consensus.cc:2749] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:24:37.443030 18334 leader_election.cc:304] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
W20260709 12:24:37.445428 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 61620.9us
W20260709 12:24:37.451645 17802 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:37.455018 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 73629us
I20260709 12:24:37.438809 18365 rpcz_store.cc:276] 0709 12:24:36.320179 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:36.434695 (+114516us) service_pool.cc:224] Handling call
0709 12:24:36.435071 (+   376us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:36.645482 (+210411us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:36.645518 (+    36us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:36.645567 (+    49us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:24:36.645610 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:37.366998 (+721388us) spinlock_profiling.cc:243] Waited 210 ms on lock 0x7b54002a30d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:24:37.367039 (+    41us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:37.367172 (+   133us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":483496320}
I20260709 12:24:37.439225 18367 rpcz_store.cc:276] 0709 12:24:36.320783 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:36.369218 (+ 48435us) service_pool.cc:224] Handling call
0709 12:24:37.364744 (+995526us) spinlock_profiling.cc:243] Waited 112 ms on lock 0x7b54006c6954. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:24:37.364848 (+   104us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":257032448}
I20260709 12:24:37.431344 26604 raft_consensus.cc:493] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:37.431525 26332 leader_election.cc:290] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 28 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:37.457671 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:37.436929 26488 raft_consensus.cc:493] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:37.458292 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 29 candidate_status { last_received { term: 27 index: 21 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:37.459333 18334 leader_election.cc:400] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 23 election: Vote denied by peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071) with higher term. Message: Invalid argument: T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 23. Current term is 24.
I20260709 12:24:37.462029 18334 leader_election.cc:403] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 23 election: Cancelling election due to peer responding with higher term
I20260709 12:24:37.462044 17682 raft_consensus.cc:2468] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 28.
I20260709 12:24:37.384730 26594 raft_consensus.cc:3060] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:37.462486 17652 leader_election.cc:304] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:37.456533 26587 raft_consensus.cc:3060] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:37.463577 26483 raft_consensus.cc:3060] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:37.458999 26354 raft_consensus.cc:515] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:37.468992 26625 leader_election.cc:290] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 27 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:37.470544 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:37.489231 17759 raft_consensus.cc:2368] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 27. Current term is 28.
W20260709 12:24:37.471118 26577 log.cc:927] Time spent T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 1.419s	user 0.022s	sys 0.001s
I20260709 12:24:37.472997 26601 leader_election.cc:290] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 26 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:37.473415 26604 raft_consensus.cc:515] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:37.439121 26627 raft_consensus.cc:493] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
W20260709 12:24:37.460296 26334 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.406s	user 0.062s	sys 0.026s
W20260709 12:24:37.498473 26334 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.448s	user 0.062s	sys 0.026s
I20260709 12:24:37.500404 26354 leader_election.cc:290] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:37.501130 26354 raft_consensus.cc:493] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:37.501698 26354 raft_consensus.cc:515] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:37.505439 26604 leader_election.cc:290] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:37.506052 26627 raft_consensus.cc:3060] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:37.453052 26538 raft_consensus.cc:515] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:37.510380 26538 leader_election.cc:290] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:37.496098 26271 raft_consensus.cc:2804] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:24:37.474344 17837 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35040 (request call id 3434) took 1111 ms. Trace:
I20260709 12:24:37.475538 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 23 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:24:37.514245 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 54400.7us
I20260709 12:24:37.452880 26640 raft_consensus.cc:493] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:37.451210 26599 raft_consensus.cc:2804] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:24:37.450434 26277 raft_consensus.cc:2749] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:24:37.456597 26622 raft_consensus.cc:2749] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Leader pre-election lost for term 24. Reason: Vote denied by peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837) with higher term. Message: Invalid argument: T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for earlier term 24. Current term is 25.
I20260709 12:24:37.433979 26588 raft_consensus.cc:515] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:37.493199 17652 leader_election.cc:304] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:37.496714 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 28 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:37.491266 26619 consensus_queue.cc:1059] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.010s
I20260709 12:24:37.514210 17837 rpcz_store.cc:276] 0709 12:24:36.362747 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:36.362877 (+   130us) service_pool.cc:224] Handling call
0709 12:24:36.412696 (+ 49819us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:37.085263 (+672567us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:37.474068 (+388805us) spinlock_profiling.cc:243] Waited 454 ms on lock 0x7b54001581d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:24:37.474267 (+   199us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1043291904}
I20260709 12:24:37.531774 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:37.530766 17681 raft_consensus.cc:2468] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 25.
I20260709 12:24:37.531718 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 24 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:37.532552 17683 raft_consensus.cc:3060] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:24:37.481410 26488 raft_consensus.cc:515] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:37.538777 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:37.532111 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:37.541577 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 24 candidate_status { last_received { term: 21 index: 18 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:37.521970 26599 raft_consensus.cc:493] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:37.548095 26599 raft_consensus.cc:3060] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:24:37.549316 26588 leader_election.cc:290] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:37.521555 26640 raft_consensus.cc:515] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:37.550546 17683 raft_consensus.cc:2468] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 28.
I20260709 12:24:37.551744 26640 leader_election.cc:290] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:37.551896 17683 raft_consensus.cc:1275] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 23 index: 13. Preceding OpId from leader: term: 24 index: 14. (index mismatch)
I20260709 12:24:37.527509 26294 raft_consensus.cc:2749] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:24:37.531665 26632 raft_consensus.cc:2804] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:24:37.557253 26632 raft_consensus.cc:697] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 25 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:24:37.558461 26632 consensus_queue.cc:237] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 24.14, Last appended by leader: 14, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:37.544397 26619 raft_consensus.cc:3055] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:24:37.560058 26619 raft_consensus.cc:740] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:24:37.561180 26619 consensus_queue.cc:260] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.14, Last appended by leader: 14, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:37.562623 26619 raft_consensus.cc:3060] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:37.566745 17687 raft_consensus.cc:2468] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 28.
I20260709 12:24:37.570004 26627 raft_consensus.cc:515] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:37.571846 26538 raft_consensus.cc:493] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:37.572474 26538 raft_consensus.cc:515] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:37.574798 26538 leader_election.cc:290] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:24:37.578699 17831 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35042 (request call id 3223) took 1342 ms (client timeout 1544 ms). Trace:
W20260709 12:24:37.579296 17831 rpcz_store.cc:269] 0709 12:24:36.236105 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:36.239843 (+  3738us) service_pool.cc:224] Handling call
0709 12:24:37.578399 (+1338556us) spinlock_profiling.cc:243] Waited 11 ms on lock 0x7b54000abed4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:24:37.578589 (+   190us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":25325952}
I20260709 12:24:37.592824 18365 raft_consensus.cc:1240] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 18. Current term is 19. Ops: [18.14-18.14]
I20260709 12:24:37.594144 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 29 candidate_status { last_received { term: 27 index: 21 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:37.594811 18368 raft_consensus.cc:2468] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 27.
I20260709 12:24:37.520823 26635 leader_election.cc:290] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:37.518694 26271 raft_consensus.cc:697] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 28 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
W20260709 12:24:37.605551 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 67243.8us
I20260709 12:24:37.607625 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:37.610343 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:37.597461 26625 raft_consensus.cc:493] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:37.616374 26625 raft_consensus.cc:515] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:37.597638 26601 raft_consensus.cc:493] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:37.619609 26601 raft_consensus.cc:515] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:37.620764 17829 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35042 (request call id 3210) took 1951 ms. Trace:
I20260709 12:24:37.621984 26537 raft_consensus.cc:493] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:37.621218 17829 rpcz_store.cc:276] 0709 12:24:35.669848 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:35.670057 (+   209us) service_pool.cc:224] Handling call
0709 12:24:35.682963 (+ 12906us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:35.683830 (+   867us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:35.683863 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:35.718553 (+ 34690us) log.cc:844] Serialized 26 byte log entry
0709 12:24:35.808646 (+ 90093us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:24:35.808684 (+    38us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:35.808753 (+    69us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:37.470357 (+1661604us) spinlock_profiling.cc:243] Waited 11.8 ms on lock 0x7b4400022b4c. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b47459c85 00007f7b4745a6a2 00007f7b4a1cc1db 00007f7b4a1c7aaf 00007f7b4a1bb86e 00007f7b4a1babb6 00007f7b4a1ba7d8 00007f7b4a1c4221 00007f7b4a1c41d9 00007f7b4a1c4169
0709 12:24:37.620439 (+150082us) raft_consensus.cc:1685] finished
0709 12:24:37.620465 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:37.620699 (+   234us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":5094,"raft.run_cpu_time_us":112,"raft.run_wall_time_us":112,"spinlock_wait_cycles":27046656,"thread_start_us":89654,"threads_started":1,"wal-append.queue_time_us":262761}
I20260709 12:24:37.622417 26601 leader_election.cc:290] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:37.622565 26537 raft_consensus.cc:515] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:37.624100 18366 raft_consensus.cc:2393] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 21: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:24:37.612083 26450 raft_consensus.cc:493] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:37.625048 26450 raft_consensus.cc:515] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:37.626196 26641 leader_election.cc:290] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:37.626505 26598 raft_consensus.cc:493] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:37.604003 26399 raft_consensus.cc:493] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:37.627020 26598 raft_consensus.cc:515] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:37.610385 26593 raft_consensus.cc:3055] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:24:37.628489 26593 raft_consensus.cc:740] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:24:37.625165 26537 leader_election.cc:290] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:37.629514 26593 consensus_queue.cc:260] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 27.18, Last appended by leader: 18, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:37.630194 26598 leader_election.cc:290] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:37.625237 26587 raft_consensus.cc:2749] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
W20260709 12:24:37.636082 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 67456.9us
I20260709 12:24:37.627676 26399 raft_consensus.cc:515] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:37.638201 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 145103us
I20260709 12:24:37.644686 18368 raft_consensus.cc:1275] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 20 index: 17. Preceding OpId from leader: term: 22 index: 18. (index mismatch)
I20260709 12:24:37.646219 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 24 candidate_status { last_received { term: 21 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:37.647236 18368 raft_consensus.cc:2468] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 23.
I20260709 12:24:37.653169 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 23 candidate_status { last_received { term: 21 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:37.654134 18368 raft_consensus.cc:3060] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:37.623359 26601 consensus_queue.cc:1048] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.014s
I20260709 12:24:37.656100 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 27 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:37.665366 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 28 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:37.666033 17763 raft_consensus.cc:3060] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:24:37.667038 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 24 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:37.686362 17803 leader_election.cc:304] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:24:37.687351 26537 raft_consensus.cc:493] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:37.687783 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:37.688704 17760 raft_consensus.cc:2393] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 28: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:24:37.688030 26537 raft_consensus.cc:515] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:37.689507 26581 raft_consensus.cc:493] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:37.690093 26581 raft_consensus.cc:515] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:37.692960 26537 leader_election.cc:290] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:37.693957 26537 raft_consensus.cc:2804] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:24:37.694511 26537 raft_consensus.cc:493] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:37.695039 26537 raft_consensus.cc:3060] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:37.696587 26640 raft_consensus.cc:493] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:37.697193 26640 raft_consensus.cc:515] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:37.699581 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 26 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:37.700659 18371 raft_consensus.cc:3060] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:37.703801 26582 raft_consensus.cc:493] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:37.704502 26582 raft_consensus.cc:515] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:37.707182 26582 leader_election.cc:290] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:37.708756 26627 leader_election.cc:290] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 23 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:37.712827 26483 raft_consensus.cc:2749] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Leader election lost for term 23. Reason: Vote denied by peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071) with higher term. Message: Invalid argument: T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 23. Current term is 24.
I20260709 12:24:37.713021 17830 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35040 (request call id 3435) took 1349 ms. Trace:
I20260709 12:24:37.682313 26593 raft_consensus.cc:3060] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:24:37.713467 17830 rpcz_store.cc:276] 0709 12:24:36.363461 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:36.363596 (+   135us) service_pool.cc:224] Handling call
0709 12:24:36.363859 (+   263us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:37.271325 (+907466us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:37.712776 (+441451us) spinlock_profiling.cc:243] Waited 900 ms on lock 0x7b54000aafd4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:24:37.712965 (+   189us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2069439744}
I20260709 12:24:37.640112 26641 raft_consensus.cc:493] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:37.724102 17803 leader_election.cc:304] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:37.726150 18371 raft_consensus.cc:2468] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 26.
W20260709 12:24:37.712678 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 105901us
I20260709 12:24:37.726374 18365 raft_consensus.cc:2468] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 24.
I20260709 12:24:37.728379 17727 leader_election.cc:304] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:37.719834 26654 consensus_queue.cc:1048] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:24:37.737128 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 29 candidate_status { last_received { term: 27 index: 21 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:37.738948 17727 leader_election.cc:304] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:37.743530 17727 leader_election.cc:304] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:37.745525 26598 raft_consensus.cc:493] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:24:37.746100 26598 raft_consensus.cc:515] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:37.739936 26653 consensus_queue.cc:1048] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.007s
I20260709 12:24:37.748237 26598 leader_election.cc:290] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:37.748036 26271 consensus_queue.cc:237] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 27.15, Last appended by leader: 15, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:24:37.753221 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 239703us
I20260709 12:24:37.754724 18362 raft_consensus.cc:1275] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 19 index: 13. Preceding OpId from leader: term: 20 index: 14. (index mismatch)
I20260709 12:24:37.741856 17758 raft_consensus.cc:2468] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 28.
I20260709 12:24:37.682482 26368 raft_consensus.cc:493] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:37.765072 18368 raft_consensus.cc:2468] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 23.
I20260709 12:24:37.775184 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:37.789219 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:37.790889 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:37.791343 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:37.791690 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:37.802440 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 27 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:37.805219 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 29 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:37.805925 17830 raft_consensus.cc:2468] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 27.
I20260709 12:24:37.806527 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:24:37.808316 17654 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:37.808188 17727 leader_election.cc:304] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:37.809126 17756 raft_consensus.cc:1240] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 27. Current term is 28. Ops: []
I20260709 12:24:37.756698 26164 raft_consensus.cc:2749] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:24:37.760449 26281 raft_consensus.cc:2749] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:24:37.818579 17763 raft_consensus.cc:2468] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 28.
I20260709 12:24:37.820365 17755 raft_consensus.cc:1240] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 27. Current term is 28. Ops: [25.15-27.16]
I20260709 12:24:37.761845 26332 raft_consensus.cc:2749] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:24:37.820899 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:37.824322 17831 raft_consensus.cc:1275] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 21 index: 15. Preceding OpId from leader: term: 23 index: 16. (index mismatch)
W20260709 12:24:37.832839 26652 log.cc:927] Time spent T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.162s	user 0.000s	sys 0.000s
W20260709 12:24:37.833968 26651 log.cc:927] Time spent T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.097s	user 0.000s	sys 0.000s
I20260709 12:24:37.839984 26483 raft_consensus.cc:493] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:24:37.735522 26641 raft_consensus.cc:515] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:37.827977 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:37.846369 17761 raft_consensus.cc:3060] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:37.848606 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:37.811558 26581 leader_election.cc:290] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:37.821867 17654 leader_election.cc:304] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:24:37.850262 26587 raft_consensus.cc:493] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:37.840606 26483 raft_consensus.cc:515] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:37.853832 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:37.855715 17756 raft_consensus.cc:2468] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 24.
I20260709 12:24:37.814954 26655 raft_consensus.cc:2749] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
W20260709 12:24:37.908792 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 53038.6us
I20260709 12:24:37.910008 17654 leader_election.cc:304] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:24:37.911864 17758 raft_consensus.cc:2468] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 27.
I20260709 12:24:37.851995 26606 raft_consensus.cc:2804] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:24:37.847565 26332 raft_consensus.cc:2804] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:24:37.808149 26368 raft_consensus.cc:515] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:37.846568 26653 consensus_queue.cc:1048] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.068s
I20260709 12:24:37.823060 26599 raft_consensus.cc:515] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:37.913775 26617 raft_consensus.cc:2804] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:24:37.915656 26354 leader_election.cc:290] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:37.960546 17760 raft_consensus.cc:2468] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 22.
W20260709 12:24:37.962680 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 50656.9us
W20260709 12:24:37.961853 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 155380us
I20260709 12:24:37.915239 26581 raft_consensus.cc:493] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
W20260709 12:24:37.965687 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 157587us
I20260709 12:24:37.968814 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:37.947602 26627 raft_consensus.cc:493] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:37.970892 17761 raft_consensus.cc:2468] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 24.
I20260709 12:24:37.971457 17829 raft_consensus.cc:2393] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 28: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:24:37.914186 26606 raft_consensus.cc:493] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:37.964550 17729 leader_election.cc:304] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:24:37.961124 26617 raft_consensus.cc:493] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:24:37.987649 26658 log.cc:927] Time spent T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.083s	user 0.000s	sys 0.000s
W20260709 12:24:37.989593 26660 log.cc:927] Time spent T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.066s	user 0.000s	sys 0.000s
I20260709 12:24:37.905531 26587 raft_consensus.cc:515] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:38.001729 26587 leader_election.cc:290] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:37.915982 26332 raft_consensus.cc:493] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:37.976899 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 22 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:37.966151 26399 leader_election.cc:290] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:38.024567 26483 leader_election.cc:290] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:37.975768 26277 consensus_queue.cc:1048] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.050s
I20260709 12:24:38.028604 26368 leader_election.cc:290] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:38.029068 26354 raft_consensus.cc:2804] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:24:38.029562 26354 raft_consensus.cc:493] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:38.029246 26368 consensus_queue.cc:1048] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:24:38.029992 26354 raft_consensus.cc:3060] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:38.006753 18362 raft_consensus.cc:2468] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 21.
I20260709 12:24:38.038977 26641 leader_election.cc:290] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:38.022011 26332 raft_consensus.cc:3060] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:24:38.040959 26537 raft_consensus.cc:515] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:38.041821 17689 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37396 (request call id 3213) took 1348 ms. Trace:
I20260709 12:24:37.974079 26581 raft_consensus.cc:515] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:38.045960 26581 leader_election.cc:290] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:37.751994 16501 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:24:37.986061 26627 raft_consensus.cc:515] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:38.050058 26627 leader_election.cc:290] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:38.005445 26617 raft_consensus.cc:3060] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Advancing to term 29
W20260709 12:24:38.052295 18363 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59886 (request call id 2327) took 2300 ms (client timeout 1956 ms). Trace:
W20260709 12:24:38.052714 18363 rpcz_store.cc:269] 0709 12:24:35.752010 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:35.752144 (+   134us) service_pool.cc:224] Handling call
0709 12:24:38.052080 (+2299936us) spinlock_profiling.cc:243] Waited 489 ms on lock 0x7b54006aed54. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:24:38.052187 (+   107us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1125570944}
I20260709 12:24:38.053376 26332 raft_consensus.cc:515] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:38.042349 17689 rpcz_store.cc:276] 0709 12:24:36.693811 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:36.693938 (+   127us) service_pool.cc:224] Handling call
0709 12:24:36.703643 (+  9705us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:36.722055 (+ 18412us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:36.722101 (+    46us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:37.576875 (+854774us) spinlock_profiling.cc:243] Waited 5.65 ms on lock 0x7b2c00000c88. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b48f19a11 00007f7b48f29372 00007f7b4a65ba01 00007f7b4a62043a 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2
0709 12:24:37.577450 (+   575us) log.cc:844] Serialized 26 byte log entry
0709 12:24:37.760912 (+183462us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:24:37.801636 (+ 40724us) log.cc:844] Serialized 14 byte log entry
0709 12:24:37.815569 (+ 13933us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:37.815678 (+   109us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:38.041231 (+225553us) raft_consensus.cc:1685] finished
0709 12:24:38.041415 (+   184us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:38.041711 (+   296us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":20025,"prepare.queue_time_us":46204,"prepare.run_cpu_time_us":58,"prepare.run_wall_time_us":11414,"raft.queue_time_us":32764,"raft.run_cpu_time_us":276,"raft.run_wall_time_us":12248,"spinlock_wait_cycles":12985216,"thread_start_us":182993,"threads_started":1,"wal-append.queue_time_us":393081}
W20260709 12:24:38.055212 17802 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:37.981071 26606 raft_consensus.cc:3060] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:38.059482 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 25 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:24:38.060067 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 81343.2us
I20260709 12:24:38.060981 17652 leader_election.cc:304] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:38.055598 26332 leader_election.cc:290] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 29 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:38.061810 17652 leader_election.cc:304] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:38.062647 26294 raft_consensus.cc:2804] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:24:38.063040 26294 raft_consensus.cc:493] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:38.063406 26294 raft_consensus.cc:3060] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:38.067056 26537 leader_election.cc:290] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 24 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:38.067368 26271 raft_consensus.cc:2804] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:24:38.066646 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:38.047508 26581 consensus_queue.cc:1048] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.024s
I20260709 12:24:38.074084 26599 leader_election.cc:290] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 28 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:38.076087 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 29 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:38.077176 17836 raft_consensus.cc:3060] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Advancing to term 29
W20260709 12:24:38.074359 26594 consensus_meta.cc:287] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.580s	user 0.010s	sys 0.000s
I20260709 12:24:38.093513 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 24 candidate_status { last_received { term: 21 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:38.094400 18365 raft_consensus.cc:3060] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:38.074391 26271 raft_consensus.cc:697] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 27 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:24:38.140254 18365 raft_consensus.cc:2468] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 24.
I20260709 12:24:38.137058 17803 leader_election.cc:304] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:24:38.144690 17803 leader_election.cc:304] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:24:38.110440 26627 raft_consensus.cc:493] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:38.149457 17803 leader_election.cc:304] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:24:38.118840 26625 leader_election.cc:290] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:38.160569 26627 raft_consensus.cc:515] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:38.151989 26601 raft_consensus.cc:2804] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:24:38.092785 26594 raft_consensus.cc:515] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:38.174481 18363 raft_consensus.cc:1275] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 16 index: 13. Preceding OpId from leader: term: 19 index: 14. (index mismatch)
I20260709 12:24:38.150503 26537 raft_consensus.cc:2804] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:24:38.176110 26601 raft_consensus.cc:697] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 23 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:24:38.181296 26601 consensus_queue.cc:237] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 21.18, Last appended by leader: 18, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:38.163079 26443 raft_consensus.cc:2804] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:24:38.185159 17803 leader_election.cc:304] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:24:38.150535 26164 consensus_queue.cc:1048] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.027s
I20260709 12:24:38.187211 26627 leader_election.cc:290] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:38.185549 26537 raft_consensus.cc:697] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 26 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:24:38.187275 26302 raft_consensus.cc:2804] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:24:38.189234 26302 raft_consensus.cc:697] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 24 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:24:38.178995 26640 leader_election.cc:290] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:38.185361 26443 raft_consensus.cc:493] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:38.189613 26537 consensus_queue.cc:237] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 22.18, Last appended by leader: 19, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:38.190572 26443 raft_consensus.cc:3060] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:38.190222 26302 consensus_queue.cc:237] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 21.18, Last appended by leader: 18, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:38.199981 17836 raft_consensus.cc:2468] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 29.
I20260709 12:24:38.208623 26617 raft_consensus.cc:515] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:38.213068 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 204097us
I20260709 12:24:38.213666 17727 leader_election.cc:304] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:38.214232 17652 leader_election.cc:304] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:38.218830 17832 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35042 (request call id 3209) took 2543 ms. Trace:
I20260709 12:24:38.229269 26354 raft_consensus.cc:515] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:38.232496 26617 leader_election.cc:290] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 29 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:38.233253 26588 raft_consensus.cc:2804] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:24:38.201653 26271 consensus_queue.cc:237] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 26.15, Last appended by leader: 15, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:38.236498 26483 raft_consensus.cc:493] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:38.238698 26594 leader_election.cc:290] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 25 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:38.237075 26483 raft_consensus.cc:515] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:38.257987 17761 raft_consensus.cc:1275] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 23 index: 15. Preceding OpId from leader: term: 25 index: 16. (index mismatch)
W20260709 12:24:38.258226 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 44062.7us
I20260709 12:24:38.282799 17835 raft_consensus.cc:1275] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 24 index: 14. Preceding OpId from leader: term: 27 index: 15. (index mismatch)
I20260709 12:24:38.282936 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 26 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:38.219432 17832 rpcz_store.cc:276] 0709 12:24:35.669511 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:35.678169 (+  8658us) service_pool.cc:224] Handling call
0709 12:24:35.714306 (+ 36137us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:35.823540 (+109234us) raft_consensus.cc:1534] Early marking committed up to index 16
0709 12:24:35.937765 (+114225us) log.cc:844] Serialized 14 byte log entry
0709 12:24:36.248908 (+311143us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:37.719747 (+1470839us) spinlock_profiling.cc:243] Waited 3.66 ms on lock 0x7b2c000472a8. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b48f19a11 00007f7b48f29372 00007f7b4a65ba01 00007f7b4a62043a 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2
0709 12:24:37.720179 (+   432us) log.cc:844] Serialized 26 byte log entry
0709 12:24:37.858976 (+138797us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:24:38.198438 (+339462us) log.cc:844] Serialized 14 byte log entry
0709 12:24:38.199410 (+   972us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:38.199499 (+    89us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:38.212576 (+ 13077us) raft_consensus.cc:1685] finished
0709 12:24:38.212609 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:38.212802 (+   193us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":22945,"prepare.queue_time_us":173392,"prepare.run_cpu_time_us":71,"prepare.run_wall_time_us":2122,"raft.queue_time_us":33030,"raft.run_cpu_time_us":399,"raft.run_wall_time_us":20348,"spinlock_wait_cycles":8416512,"thread_start_us":789085,"threads_started":5,"wal-append.queue_time_us":590091,"wal-append.run_cpu_time_us":2308,"wal-append.run_wall_time_us":1252603}
I20260709 12:24:38.291055 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 29 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:38.291373 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:38.291810 17684 raft_consensus.cc:2468] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 28.
I20260709 12:24:38.291718 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:38.294094 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 25 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:38.302258 18368 raft_consensus.cc:1240] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 22. Current term is 23. Ops: [22.18-22.18]
I20260709 12:24:38.295994 26354 leader_election.cc:290] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 22 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:38.309382 26483 leader_election.cc:290] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:38.311457 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 25 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:38.313666 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 29 candidate_status { last_received { term: 27 index: 21 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:38.314463 17755 raft_consensus.cc:3060] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:24:38.314994 18368 raft_consensus.cc:1240] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 23. Current term is 24. Ops: []
I20260709 12:24:38.317375 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:38.318161 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:38.257529 26588 raft_consensus.cc:697] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 29 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:24:38.321257 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:38.282797 26593 raft_consensus.cc:493] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:38.231703 26606 raft_consensus.cc:515] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:38.289650 26619 raft_consensus.cc:2804] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:24:38.323930 17762 raft_consensus.cc:1275] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 23 index: 16. Preceding OpId from leader: term: 25 index: 17. (index mismatch)
I20260709 12:24:38.307454 26632 consensus_queue.cc:1059] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.009s
I20260709 12:24:38.324613 26294 raft_consensus.cc:515] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:38.309253 26594 consensus_queue.cc:1048] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.016s
I20260709 12:24:38.323796 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 29 candidate_status { last_received { term: 27 index: 21 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:38.357551 18371 raft_consensus.cc:3060] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Advancing to term 29
W20260709 12:24:38.337212 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 332744us
I20260709 12:24:38.342391 17756 raft_consensus.cc:1275] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 20 index: 14. Preceding OpId from leader: term: 21 index: 15. (index mismatch)
I20260709 12:24:38.351786 17755 raft_consensus.cc:2468] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 29.
I20260709 12:24:38.359306 17729 leader_election.cc:304] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:24:38.381060 17833 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35042 (request call id 3226) took 1665 ms. Trace:
I20260709 12:24:38.381781 17833 rpcz_store.cc:276] 0709 12:24:36.715620 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:36.715798 (+   178us) service_pool.cc:224] Handling call
0709 12:24:36.716258 (+   460us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:37.432204 (+715946us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:38.380583 (+948379us) spinlock_profiling.cc:243] Waited 715 ms on lock 0x7b5400158bd4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:24:38.380909 (+   326us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1644063488}
I20260709 12:24:38.399636 18334 leader_election.cc:304] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b
W20260709 12:24:38.406177 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 347549us
I20260709 12:24:38.409160 17680 raft_consensus.cc:2393] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 25: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:24:38.422992 17801 leader_election.cc:304] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:38.366578 26678 raft_consensus.cc:493] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:38.379020 26340 raft_consensus.cc:493] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:38.431275 18333 leader_election.cc:304] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:24:38.432111 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:38.432865 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:38.405969 26599 raft_consensus.cc:493] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:38.385650 26619 raft_consensus.cc:697] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 24 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:24:38.433185 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:38.453713 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 29 candidate_status { last_received { term: 27 index: 21 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:38.454085 18368 raft_consensus.cc:2468] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 27.
I20260709 12:24:38.406610 26354 raft_consensus.cc:2804] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:24:38.446218 18363 raft_consensus.cc:1275] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 19 index: 14. Preceding OpId from leader: term: 21 index: 15. (index mismatch)
I20260709 12:24:38.450618 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:38.452740 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:38.409289 26483 raft_consensus.cc:493] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:38.455835 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 27 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:38.469733 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:38.385908 26593 raft_consensus.cc:515] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:38.552591 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 292836us
I20260709 12:24:38.438234 18371 raft_consensus.cc:2468] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 29.
I20260709 12:24:38.517478 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:38.668166 18366 raft_consensus.cc:3060] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:24:38.469220 26627 raft_consensus.cc:2804] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Leader pre-election won for term 21
W20260709 12:24:38.584122 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 337791us
I20260709 12:24:38.589181 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 22 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:38.669286 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:24:38.608198 26671 log.cc:927] Time spent T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.169s	user 0.001s	sys 0.000s
I20260709 12:24:38.457028 26678 raft_consensus.cc:515] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:38.608280 26676 log.cc:927] Time spent T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.123s	user 0.000s	sys 0.001s
I20260709 12:24:38.470817 26599 raft_consensus.cc:515] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:38.507947 26588 consensus_queue.cc:237] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 27.18, Last appended by leader: 18, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:38.629426 26681 raft_consensus.cc:2804] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:24:38.632189 26691 raft_consensus.cc:3055] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:24:38.634544 26488 leader_election.cc:290] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:38.635210 26294 leader_election.cc:290] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 25 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:24:38.701066 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 278121us
I20260709 12:24:38.638314 26675 raft_consensus.cc:493] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:24:38.640664 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:38.708511 18365 raft_consensus.cc:2468] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 24.
I20260709 12:24:38.709590 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:38.642549 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:38.448079 26340 raft_consensus.cc:515] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:38.654430 18368 raft_consensus.cc:1240] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 25. Current term is 26. Ops: [25.19-25.19]
I20260709 12:24:38.421973 26334 raft_consensus.cc:493] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:24:38.659621 17762 raft_consensus.cc:2468] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 27.
W20260709 12:24:38.661376 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 230187us
I20260709 12:24:38.717200 18333 leader_election.cc:304] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
W20260709 12:24:38.661970 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 338878us
I20260709 12:24:38.669967 18363 raft_consensus.cc:2468] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 27.
I20260709 12:24:38.671813 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:38.660779 18374 raft_consensus.cc:3060] T a5732e71b9d54f8786b8c511d69943bf P 334a4e1f59944544b3b3ff36616b57da [term 25 LEARNER]: Advancing to term 27
I20260709 12:24:38.680392 26606 leader_election.cc:290] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 23 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:38.689148 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:38.690411 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:38.655328 26641 raft_consensus.cc:2804] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:24:38.657698 26354 raft_consensus.cc:493] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:38.653434 26164 raft_consensus.cc:493] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:24:38.697268 17831 raft_consensus.cc:1275] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 21 index: 13. Preceding OpId from leader: term: 23 index: 14. (index mismatch)
I20260709 12:24:38.699115 17729 leader_election.cc:304] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:38.637332 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:38.709707 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 28 candidate_status { last_received { term: 27 index: 21 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:38.711061 17759 raft_consensus.cc:2468] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 27.
I20260709 12:24:38.740516 17759 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38786 (request call id 3889) took 1080 ms. Trace:
I20260709 12:24:38.740660 17756 raft_consensus.cc:2368] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 28. Current term is 29.
I20260709 12:24:38.708307 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:38.720233 17835 raft_consensus.cc:2368] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for earlier term 27. Current term is 28.
I20260709 12:24:38.723793 26619 consensus_queue.cc:237] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 22.15, Last appended by leader: 15, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:38.709000 26688 consensus_queue.cc:1059] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.030s
W20260709 12:24:38.729234 26679 log.cc:927] Time spent T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.055s	user 0.000s	sys 0.000s
I20260709 12:24:38.729697 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:38.731137 26593 leader_election.cc:290] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:38.716229 26675 raft_consensus.cc:515] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:38.753129 26675 leader_election.cc:290] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:38.753659 17760 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36572 (request call id 1530) took 1022 ms. Trace:
I20260709 12:24:38.754017 17760 rpcz_store.cc:276] 0709 12:24:37.731224 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:37.731820 (+   596us) service_pool.cc:224] Handling call
0709 12:24:38.753482 (+1021662us) spinlock_profiling.cc:243] Waited 183 ms on lock 0x7b54003e53d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:24:38.753601 (+   119us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":420464000}
I20260709 12:24:38.751053 18374 raft_consensus.cc:1275] T a5732e71b9d54f8786b8c511d69943bf P 334a4e1f59944544b3b3ff36616b57da [term 27 LEARNER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 25 index: 15. Preceding OpId from leader: term: 27 index: 15. (term mismatch)
I20260709 12:24:38.755148 18374 pending_rounds.cc:85] T a5732e71b9d54f8786b8c511d69943bf P 334a4e1f59944544b3b3ff36616b57da: Aborting all ops after (but not including) 14
I20260709 12:24:38.755409 18374 pending_rounds.cc:107] T a5732e71b9d54f8786b8c511d69943bf P 334a4e1f59944544b3b3ff36616b57da: Aborting uncommitted NO_OP operation due to leader change: 25.15
I20260709 12:24:38.755647 18374 raft_consensus.cc:2889] T a5732e71b9d54f8786b8c511d69943bf P 334a4e1f59944544b3b3ff36616b57da [term 27 LEARNER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:38.757167 26678 leader_election.cc:290] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:38.732769 26399 raft_consensus.cc:493] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:38.760023 26399 raft_consensus.cc:515] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:38.761304 26599 leader_election.cc:290] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:38.762259 26599 consensus_queue.cc:1048] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20260709 12:24:38.734442 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:38.735852 18333 leader_election.cc:304] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:38.698683 26691 raft_consensus.cc:740] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:24:38.740906 17759 rpcz_store.cc:276] 0709 12:24:37.659992 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:37.664229 (+  4237us) service_pool.cc:224] Handling call
0709 12:24:38.642404 (+978175us) spinlock_profiling.cc:243] Waited 2.12 ms on lock 0x7b5800033398. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b474fbb71 00007f7b474fba22 00007f7b504148d9 00007f7b4f56eee0 00007f7b4f6380ab 00007f7b4f638131 00007f7b4f5e60a0 00007f7b4f5e5984 00007f7b48b606d4 00007f7b48b60662
0709 12:24:38.740454 (+ 98050us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4870016}
W20260709 12:24:38.745038 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 46015.7us
I20260709 12:24:38.676169 26443 raft_consensus.cc:515] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:38.678390 26483 raft_consensus.cc:515] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:38.734357 26641 raft_consensus.cc:493] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:38.748334 17835 raft_consensus.cc:1240] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 24 LEADER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 23. Current term is 24. Ops: []
I20260709 12:24:38.770254 18333 leader_election.cc:400] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 27 election: Vote denied by peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651) with higher term. Message: Invalid argument: T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 27. Current term is 28.
I20260709 12:24:38.770771 18333 leader_election.cc:403] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 27 election: Cancelling election due to peer responding with higher term
I20260709 12:24:38.762552 26399 leader_election.cc:290] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:38.703284 26689 raft_consensus.cc:493] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:38.712930 26681 raft_consensus.cc:697] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 28 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:24:38.769754 26544 raft_consensus.cc:2749] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:24:38.713694 26488 raft_consensus.cc:493] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:38.743206 18366 raft_consensus.cc:2468] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 28.
I20260709 12:24:38.742741 26354 raft_consensus.cc:3060] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:38.740427 26582 raft_consensus.cc:493] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:38.774060 18333 leader_election.cc:304] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:38.708990 26368 consensus_queue.cc:1048] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.057s
I20260709 12:24:38.753666 26675 raft_consensus.cc:2749] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:24:38.814721 17833 raft_consensus.cc:2368] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for earlier term 21. Current term is 22.
I20260709 12:24:38.765694 26635 raft_consensus.cc:2804] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:24:38.817922 17802 leader_election.cc:304] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:38.723858 26334 raft_consensus.cc:515] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:38.768033 26654 raft_consensus.cc:493] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:24:38.772470 26688 consensus_queue.cc:1243] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer e063541542a94969898868bf6eb5ce1b log is divergent from this leader: its last log entry 21.18 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 17
W20260709 12:24:38.839918 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 132639us
I20260709 12:24:38.772800 26641 raft_consensus.cc:3060] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:38.747804 26164 raft_consensus.cc:515] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:38.838727 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 64736.2us
I20260709 12:24:38.879632 18333 leader_election.cc:304] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:38.763110 26601 consensus_queue.cc:1048] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.053s
I20260709 12:24:38.769251 26294 consensus_queue.cc:1059] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 23.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.053s
I20260709 12:24:38.858747 26340 leader_election.cc:290] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:38.859562 26681 consensus_queue.cc:237] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 25.15, Last appended by leader: 15, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:38.847733 26488 raft_consensus.cc:515] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:38.919615 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:38.817835 26705 raft_consensus.cc:2749] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Leader election lost for term 27. Reason: Vote denied by peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651) with higher term. Message: Invalid argument: T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 27. Current term is 28.
I20260709 12:24:38.861526 26544 raft_consensus.cc:2749] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:24:38.837698 26579 raft_consensus.cc:493] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:24:38.865195 26443 leader_election.cc:290] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 22 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:38.849985 26635 raft_consensus.cc:493] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
W20260709 12:24:38.869983 26662 log.cc:927] Time spent T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.838s	user 0.005s	sys 0.004s
I20260709 12:24:38.878029 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:38.883215 17688 raft_consensus.cc:1275] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 23 index: 16. Preceding OpId from leader: term: 25 index: 17. (index mismatch)
I20260709 12:24:38.888649 17729 leader_election.cc:304] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:24:38.890340 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 172088us
I20260709 12:24:38.906665 18333 leader_election.cc:304] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:38.921185 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:38.922036 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 22 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:38.861340 17680 raft_consensus.cc:1275] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 19 index: 13. Preceding OpId from leader: term: 20 index: 14. (index mismatch)
I20260709 12:24:38.919682 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 24 candidate_status { last_received { term: 21 index: 18 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
W20260709 12:24:38.924307 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 252999us
I20260709 12:24:38.861562 26450 leader_election.cc:290] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:38.944571 17686 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44898 (request call id 1699) took 2299 ms. Trace:
I20260709 12:24:38.951260 17654 leader_election.cc:304] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:38.945120 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:38.940366 17681 raft_consensus.cc:2468] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 28.
W20260709 12:24:39.001612 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 95032.4us
I20260709 12:24:38.847361 26582 raft_consensus.cc:515] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:38.952275 17684 raft_consensus.cc:2468] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 20.
I20260709 12:24:39.002410 18333 leader_election.cc:304] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:24:38.879312 26281 consensus_queue.cc:1059] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 22.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.061s
I20260709 12:24:38.812980 26689 raft_consensus.cc:515] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:24:38.923746 26625 raft_consensus.cc:2749] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
W20260709 12:24:39.009395 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 58356.4us
I20260709 12:24:38.879159 26632 consensus_queue.cc:1059] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 20.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.061s
W20260709 12:24:39.014040 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 125509us
I20260709 12:24:38.998970 17686 rpcz_store.cc:276] 0709 12:24:36.640327 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:36.640484 (+   157us) service_pool.cc:224] Handling call
0709 12:24:36.647536 (+  7052us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:36.666356 (+ 18820us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:36.666389 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:38.072301 (+1405912us) spinlock_profiling.cc:243] Waited 2.08 ms on lock 0x7b2c00000c88. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b48f19a11 00007f7b48f29372 00007f7b4a65ba01 00007f7b4a62043a 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2
0709 12:24:38.072808 (+   507us) log.cc:844] Serialized 26 byte log entry
0709 12:24:38.293398 (+220590us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:24:38.293447 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:38.293583 (+   136us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:38.930710 (+637127us) raft_consensus.cc:1685] finished
0709 12:24:38.930731 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:38.939815 (+  9084us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":15128,"raft.queue_time_us":30148,"raft.run_cpu_time_us":162,"raft.run_wall_time_us":10878,"spinlock_wait_cycles":4784384,"thread_start_us":220193,"threads_started":1,"wal-append.queue_time_us":353388}
I20260709 12:24:39.015486 17654 leader_election.cc:304] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:24:39.057197 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:38.941809 26635 raft_consensus.cc:3060] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:24:39.115854 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 239184us
I20260709 12:24:38.944775 26693 raft_consensus.cc:493] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:39.118114 26693 raft_consensus.cc:515] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:39.122437 26693 leader_election.cc:290] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:39.125069 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:38.890097 26654 raft_consensus.cc:515] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:39.128111 17683 raft_consensus.cc:1240] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 22. Current term is 23. Ops: []
I20260709 12:24:38.946792 26700 raft_consensus.cc:2804] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:24:39.132211 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:39.013832 26441 raft_consensus.cc:493] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:39.013417 26399 raft_consensus.cc:493] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:24:39.141410 17829 raft_consensus.cc:2393] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 27: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:24:38.938277 26697 consensus_queue.cc:1048] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.205s
I20260709 12:24:38.944173 26579 raft_consensus.cc:515] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:38.943554 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 266131us
I20260709 12:24:39.128481 26710 raft_consensus.cc:493] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:39.140903 26593 raft_consensus.cc:2804] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Leader election won for term 29
W20260709 12:24:39.164386 26670 log.cc:927] Time spent T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.127s	user 0.001s	sys 0.000s
I20260709 12:24:39.161341 26709 raft_consensus.cc:493] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:39.125100 26713 raft_consensus.cc:2749] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:24:39.132055 26707 raft_consensus.cc:2804] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:24:39.114658 26637 consensus_queue.cc:1059] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.130s
W20260709 12:24:39.238379 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 382549us
W20260709 12:24:39.250203 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 212155us
I20260709 12:24:39.251542 17729 leader_election.cc:304] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:24:39.254535 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:39.257856 17757 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38786 (request call id 3891) took 1592 ms. Trace:
I20260709 12:24:39.258361 17757 rpcz_store.cc:276] 0709 12:24:37.660710 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:37.668796 (+  8086us) service_pool.cc:224] Handling call
0709 12:24:37.684499 (+ 15703us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:37.684738 (+   239us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:37.684767 (+    29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:37.685414 (+   647us) log.cc:844] Serialized 26 byte log entry
0709 12:24:37.859526 (+174112us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:24:37.859575 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:37.859670 (+    95us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:38.645654 (+785984us) spinlock_profiling.cc:243] Waited 32.1 ms on lock 0x7b4400022b4c. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b47459c85 00007f7b4745a6a2 00007f7b4a1cc1db 00007f7b4a1c7aaf 00007f7b4a1bb86e 00007f7b4a1babb6 00007f7b4a1ba7d8 00007f7b4a1c4221 00007f7b4a1c41d9 00007f7b4a1c4169
0709 12:24:38.869606 (+223952us) spinlock_profiling.cc:243] Waited 7.03 ms on lock 0x7b24000f7268. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b473b99c4 00007f7b473b7acb 00007f7b47572dc9 00007f7b47572a55 00007f7b4757209d 00007f7b4a1a410e 00007f7b4a1bb88c 00007f7b4a1babb6 00007f7b4a1ba7d8 00007f7b4a1c4221
0709 12:24:39.252983 (+383377us) raft_consensus.cc:1685] finished
0709 12:24:39.253018 (+    35us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:39.253215 (+   197us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":89983104,"thread_start_us":173646,"threads_started":1,"wal-append.queue_time_us":291669}
I20260709 12:24:39.227198 26700 raft_consensus.cc:697] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 28 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:24:39.274353 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:39.176707 26441 raft_consensus.cc:515] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:39.244990 26695 raft_consensus.cc:493] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:24:39.235342 26443 raft_consensus.cc:493] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:39.227088 26701 raft_consensus.cc:493] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:39.251307 26707 raft_consensus.cc:493] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:24:39.283425 26707 raft_consensus.cc:3060] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:24:39.256148 26675 raft_consensus.cc:2804] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:24:39.184100 26399 raft_consensus.cc:515] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:39.289517 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 21 candidate_status { last_received { term: 19 index: 19 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:39.291606 26579 leader_election.cc:290] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:39.285759 26483 leader_election.cc:290] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:39.246066 26710 raft_consensus.cc:515] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:39.269753 26599 consensus_queue.cc:1048] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.031s
I20260709 12:24:39.316893 17802 leader_election.cc:304] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:39.285370 26675 raft_consensus.cc:493] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:39.325511 17802 leader_election.cc:304] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:24:39.312716 26703 log.cc:927] Time spent T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.374s	user 0.000s	sys 0.001s
I20260709 12:24:39.328127 26582 leader_election.cc:290] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:39.328704 26726 raft_consensus.cc:493] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:39.369318 26717 raft_consensus.cc:2804] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:24:39.369902 26717 raft_consensus.cc:493] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:24:39.370314 26717 raft_consensus.cc:3060] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:24:39.372011 26728 raft_consensus.cc:493] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:39.298815 26281 consensus_queue.cc:1059] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 22.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.030s
I20260709 12:24:39.372825 26728 raft_consensus.cc:515] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:39.292392 26443 raft_consensus.cc:515] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:39.376276 26443 leader_election.cc:290] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:24:39.329845 26718 log.cc:927] Time spent T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.066s	user 0.000s	sys 0.000s
I20260709 12:24:39.377115 26635 raft_consensus.cc:515] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:39.328176 26724 consensus_queue.cc:1048] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.103s
I20260709 12:24:39.324399 26675 raft_consensus.cc:3060] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:39.330335 26700 consensus_queue.cc:237] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 27.16, Last appended by leader: 16, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:24:39.332103 26715 log.cc:927] Time spent T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.328s	user 0.000s	sys 0.001s
I20260709 12:24:39.325124 26334 leader_election.cc:290] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:39.335723 17802 leader_election.cc:304] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
W20260709 12:24:39.399699 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 64072.8us
I20260709 12:24:39.338354 26488 leader_election.cc:290] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:39.289047 26701 raft_consensus.cc:515] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:39.344955 26706 raft_consensus.cc:2749] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:24:39.348647 26689 leader_election.cc:290] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:39.257777 26709 raft_consensus.cc:515] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:39.352988 26441 leader_election.cc:290] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:39.293820 26695 raft_consensus.cc:515] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:39.357177 26399 leader_election.cc:290] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:39.360206 26627 raft_consensus.cc:493] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:24:39.393374 26593 raft_consensus.cc:697] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 29 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:24:39.364084 26710 leader_election.cc:290] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:39.365198 26421 raft_consensus.cc:3055] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:24:39.368903 26726 raft_consensus.cc:515] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:39.325204 26697 raft_consensus.cc:2749] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:24:39.409075 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 29 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:39.451468 17680 raft_consensus.cc:3060] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Advancing to term 29
W20260709 12:24:39.451375 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 199927us
I20260709 12:24:39.416662 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 22 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:39.389164 26635 leader_election.cc:290] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 24 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:39.432626 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:39.436381 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:39.437166 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:39.437907 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:39.440655 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:39.361515 26654 leader_election.cc:290] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:24:39.445376 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 326786us
I20260709 12:24:39.445735 26627 raft_consensus.cc:3060] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:39.473407 17684 raft_consensus.cc:3060] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:39.411820 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:39.435032 26617 raft_consensus.cc:493] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:39.452234 17686 raft_consensus.cc:2468] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 19.
I20260709 12:24:39.453862 17687 raft_consensus.cc:2468] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 20.
I20260709 12:24:39.476970 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 24 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:39.469579 17689 raft_consensus.cc:2393] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 23: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:24:39.402997 26724 raft_consensus.cc:2764] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Leader pre-election decision vote started in defunct term 27: won
I20260709 12:24:39.448031 26421 raft_consensus.cc:740] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
W20260709 12:24:39.446533 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 218119us
I20260709 12:24:39.434866 26334 consensus_queue.cc:1048] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.019s
I20260709 12:24:39.485857 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:39.485891 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:39.472832 17681 raft_consensus.cc:2468] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 26.
I20260709 12:24:39.487665 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:39.445542 26689 consensus_queue.cc:1059] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.030s
I20260709 12:24:39.482616 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 27 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:39.489717 17729 leader_election.cc:304] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:39.497382 17684 raft_consensus.cc:2468] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 25.
I20260709 12:24:39.491369 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:39.491488 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:39.492118 26727 consensus_queue.cc:1048] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:24:39.492519 17762 raft_consensus.cc:1240] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 27. Current term is 28. Ops: [25.15-27.16]
I20260709 12:24:39.494369 17763 raft_consensus.cc:1240] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 21. Current term is 22. Ops: []
I20260709 12:24:39.495121 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:39.490686 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:39.490681 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:39.496214 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 23 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:39.504256 17685 raft_consensus.cc:2468] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 22.
I20260709 12:24:39.504451 17832 raft_consensus.cc:2468] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 22.
I20260709 12:24:39.479262 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 24 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:39.495970 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 20 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:39.480021 18367 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 2555) took 1882 ms. Trace:
I20260709 12:24:39.505538 18367 rpcz_store.cc:276] 0709 12:24:37.592237 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:37.602592 (+ 10355us) service_pool.cc:224] Handling call
0709 12:24:37.604182 (+  1590us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:37.630886 (+ 26704us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:37.630932 (+    46us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:37.630989 (+    57us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:24:37.631048 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:39.474848 (+1843800us) spinlock_profiling.cc:243] Waited 26.4 ms on lock 0x7b5400305554. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:24:39.474900 (+    52us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:39.475030 (+   130us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":60812544}
I20260709 12:24:39.483806 17728 leader_election.cc:304] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:39.498390 17681 raft_consensus.cc:2468] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 22.
I20260709 12:24:39.498675 17757 raft_consensus.cc:3060] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:39.498845 17838 raft_consensus.cc:1275] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 24 index: 14. Preceding OpId from leader: term: 25 index: 15. (index mismatch)
I20260709 12:24:39.470006 26681 raft_consensus.cc:493] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:39.474960 26622 raft_consensus.cc:493] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:39.513327 17684 raft_consensus.cc:1240] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 24. Current term is 25. Ops: []
I20260709 12:24:39.514914 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:39.520879 17838 raft_consensus.cc:2468] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 21.
I20260709 12:24:39.526422 26675 raft_consensus.cc:515] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:39.529044 26709 leader_election.cc:290] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:39.530653 26717 raft_consensus.cc:515] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:39.531688 26736 raft_consensus.cc:3055] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:24:39.532099 26421 consensus_queue.cc:260] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 25.19, Last appended by leader: 19, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:39.532778 26689 raft_consensus.cc:493] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:39.533039 26681 raft_consensus.cc:515] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:39.537636 26441 raft_consensus.cc:2764] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Leader pre-election decision vote started in defunct term 28: won
I20260709 12:24:39.527693 17680 raft_consensus.cc:2468] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 29.
I20260709 12:24:39.547067 17757 raft_consensus.cc:2468] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 25.
W20260709 12:24:39.532791 26641 consensus_meta.cc:287] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.628s	user 0.009s	sys 0.002s
I20260709 12:24:39.481843 26617 raft_consensus.cc:515] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:39.574944 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:39.577705 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:24:39.568060 26354 consensus_meta.cc:287] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.719s	user 0.010s	sys 0.001s
I20260709 12:24:39.579919 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 27 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:39.581379 17838 raft_consensus.cc:2468] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 26.
I20260709 12:24:39.513190 26281 raft_consensus.cc:2804] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:24:39.602895 17760 raft_consensus.cc:1240] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 24. Current term is 25. Ops: []
W20260709 12:24:39.608414 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 99973.7us
I20260709 12:24:39.570246 26736 raft_consensus.cc:740] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 22 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:24:39.611168 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:39.614475 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:39.518679 26593 consensus_queue.cc:237] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 27.21, Last appended by leader: 21, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:39.624105 26334 raft_consensus.cc:493] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:39.625001 26334 raft_consensus.cc:515] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:39.627521 26726 leader_election.cc:290] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:39.630314 26334 leader_election.cc:290] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:39.631722 17728 leader_election.cc:304] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:39.636823 17728 leader_election.cc:304] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:39.640376 26701 leader_election.cc:290] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:39.638967 17728 leader_election.cc:304] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:39.581696 26709 raft_consensus.cc:493] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:39.643249 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:39.613929 17834 raft_consensus.cc:2468] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 20.
I20260709 12:24:39.650285 17831 raft_consensus.cc:2468] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 23.
I20260709 12:24:39.653678 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 20 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:39.657024 17688 raft_consensus.cc:2410] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for term 20 because replica has last-logged OpId of term: 18 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 13.
I20260709 12:24:39.541613 26622 raft_consensus.cc:515] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:39.662835 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:39.661612 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 26 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:39.666186 17684 raft_consensus.cc:2468] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 25.
I20260709 12:24:39.640969 26653 raft_consensus.cc:3060] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:24:39.574471 26699 consensus_queue.cc:1048] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.210s
I20260709 12:24:39.654495 26281 raft_consensus.cc:697] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 28 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:24:39.682901 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:39.683092 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 22 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:39.655037 26701 raft_consensus.cc:493] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:39.655431 26727 raft_consensus.cc:2749] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:24:39.605232 26641 raft_consensus.cc:515] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:39.578291 26689 raft_consensus.cc:515] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:39.660012 26421 raft_consensus.cc:3060] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:39.816349 17652 leader_election.cc:400] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 27 election: Vote denied by peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837) with higher term. Message: Invalid argument: T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for earlier term 27. Current term is 28.
W20260709 12:24:39.836386 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 154316us
I20260709 12:24:39.602769 26354 raft_consensus.cc:515] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:39.865206 26737 raft_consensus.cc:2749] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:24:39.866539 26732 raft_consensus.cc:493] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:24:39.868043 26730 raft_consensus.cc:493] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:39.689553 26728 leader_election.cc:290] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:39.687350 26441 raft_consensus.cc:476] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 27 LEADER]: Not starting pre-election -- already a leader
I20260709 12:24:39.883493 17756 raft_consensus.cc:1240] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 23. Current term is 24. Ops: []
I20260709 12:24:39.662202 26709 raft_consensus.cc:515] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:24:39.891733 26738 log.cc:927] Time spent T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.212s	user 0.000s	sys 0.000s
I20260709 12:24:39.896642 26717 leader_election.cc:290] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 28 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:39.898731 26675 leader_election.cc:290] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 25 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:39.839242 26695 leader_election.cc:290] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:39.846266 26334 raft_consensus.cc:493] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:39.900568 26334 raft_consensus.cc:515] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:39.912563 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 75277.4us
I20260709 12:24:39.961615 17652 leader_election.cc:304] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:39.819494 26701 raft_consensus.cc:515] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:39.868757 26730 raft_consensus.cc:515] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:39.903803 26334 leader_election.cc:290] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:24:39.982249 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 867163us
W20260709 12:24:39.815840 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 299647us
I20260709 12:24:40.066505 17652 leader_election.cc:400] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 21 pre-election: Vote denied by peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837) with higher term. Message: Invalid argument: T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for earlier term 21. Current term is 22.
I20260709 12:24:40.068176 17652 leader_election.cc:403] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 21 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:24:39.867333 26732 raft_consensus.cc:515] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:40.100369 26737 raft_consensus.cc:493] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
W20260709 12:24:40.112653 26748 log.cc:927] Time spent T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.274s	user 0.001s	sys 0.000s
I20260709 12:24:40.112346 26737 raft_consensus.cc:515] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:40.114914 26737 leader_election.cc:290] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:40.115188 26641 leader_election.cc:290] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 22 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:40.119383 26617 leader_election.cc:290] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:39.825112 26736 consensus_queue.cc:260] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 22.16, Last appended by leader: 16, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:40.120173 26681 leader_election.cc:290] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:40.123469 26709 leader_election.cc:290] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:39.945346 26695 consensus_queue.cc:1048] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.143s
I20260709 12:24:40.124725 26689 leader_election.cc:290] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:40.125360 26695 consensus_queue.cc:1059] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.000s
I20260709 12:24:39.978493 26281 consensus_queue.cc:237] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 26.16, Last appended by leader: 16, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:40.126051 26695 consensus_queue.cc:1243] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer e063541542a94969898868bf6eb5ce1b log is divergent from this leader: its last log entry 24.19 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 17
I20260709 12:24:40.127966 26701 leader_election.cc:290] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:40.128192 26617 raft_consensus.cc:3060] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Advancing to term 22
I20260709 12:24:40.129777 26730 leader_election.cc:290] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:40.122659 26354 leader_election.cc:290] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 24 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:39.986070 26622 leader_election.cc:290] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:24:40.099468 26707 consensus_meta.cc:287] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.815s	user 0.013s	sys 0.000s
I20260709 12:24:40.134331 26707 raft_consensus.cc:515] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:40.142462 26709 raft_consensus.cc:2749] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
W20260709 12:24:40.206944 17727 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.526s	user 0.003s	sys 0.007s
W20260709 12:24:40.207561 17727 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.527s	user 0.003s	sys 0.007s
W20260709 12:24:40.207986 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 543134us
W20260709 12:24:40.212491 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 376996us
W20260709 12:24:40.209319 26594 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.522s	user 0.006s	sys 0.003s
W20260709 12:24:40.216146 17685 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.636s	user 0.010s	sys 0.000s
W20260709 12:24:40.216696 17685 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.645s	user 0.010s	sys 0.000s
W20260709 12:24:40.219574 26588 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.549s	user 0.005s	sys 0.013s
W20260709 12:24:40.221817 17831 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.534s	user 0.000s	sys 0.009s
W20260709 12:24:40.222594 17831 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.535s	user 0.000s	sys 0.010s
W20260709 12:24:40.220234 26588 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.550s	user 0.005s	sys 0.014s
W20260709 12:24:40.280848 17801 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:40.284075 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:24:40.322005 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 262715us
W20260709 12:24:40.220420 26594 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.536s	user 0.007s	sys 0.004s
W20260709 12:24:40.322773 18334 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.16.29.65:33071 timed out after 1.568s (SENT) [suppressed 5 similar messages]
I20260709 12:24:40.374821 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 26 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:24:40.376529 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 167332us
I20260709 12:24:40.377998 17684 raft_consensus.cc:2468] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 25.
W20260709 12:24:40.377944 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 55210.7us
I20260709 12:24:40.381101 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 21 candidate_status { last_received { term: 19 index: 19 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:40.388680 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 27 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:40.389752 17680 raft_consensus.cc:2410] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for term 27 because replica has last-logged OpId of term: 25 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 14.
I20260709 12:24:40.390836 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:40.220839 26354 raft_consensus.cc:493] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:24:40.396620 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
W20260709 12:24:40.349395 26654 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.525s	user 0.027s	sys 0.023s
I20260709 12:24:40.404888 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
W20260709 12:24:40.408044 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 341710us
W20260709 12:24:40.406152 26654 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.717s	user 0.027s	sys 0.023s
W20260709 12:24:40.406399 26653 consensus_meta.cc:287] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.724s	user 0.010s	sys 0.000s
I20260709 12:24:40.391378 26732 leader_election.cc:290] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:40.391075 26761 raft_consensus.cc:493] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:40.420128 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:40.422314 17681 raft_consensus.cc:1275] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 27 index: 18. Preceding OpId from leader: term: 29 index: 19. (index mismatch)
I20260709 12:24:40.409996 17652 leader_election.cc:304] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:40.408792 26750 raft_consensus.cc:493] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:40.405709 26354 raft_consensus.cc:515] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:40.438664 26760 raft_consensus.cc:2749] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:24:40.441447 26617 raft_consensus.cc:2749] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Leader pre-election lost for term 21. Reason: Vote denied by peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837) with higher term. Message: Invalid argument: T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for earlier term 21. Current term is 22.
I20260709 12:24:40.426544 26653 raft_consensus.cc:2749] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:24:40.429203 26750 raft_consensus.cc:515] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:40.454165 17684 pending_rounds.cc:85] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3: Aborting all ops after (but not including) 14
I20260709 12:24:40.454584 17684 pending_rounds.cc:107] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3: Aborting uncommitted NO_OP operation due to leader change: 25.15
I20260709 12:24:40.454833 17684 raft_consensus.cc:2889] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:40.426831 26761 raft_consensus.cc:515] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:40.457221 26354 leader_election.cc:290] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:40.457849 26761 leader_election.cc:290] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:40.457871 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:24:40.459003 26627 consensus_meta.cc:287] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.978s	user 0.011s	sys 0.000s
I20260709 12:24:40.459559 26627 raft_consensus.cc:515] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:40.452129 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 23 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:40.427938 26707 leader_election.cc:290] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 28 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:40.463610 17834 raft_consensus.cc:2410] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for term 23 because replica has last-logged OpId of term: 21 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 16.
I20260709 12:24:40.464141 26588 raft_consensus.cc:493] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:40.482092 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:40.482820 17836 raft_consensus.cc:2468] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 20.
I20260709 12:24:40.448946 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:40.484740 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 20 candidate_status { last_received { term: 19 index: 19 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:40.485354 17836 raft_consensus.cc:2468] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 20.
I20260709 12:24:40.486282 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 26 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:40.488199 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:40.452544 26606 consensus_queue.cc:1048] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.040s
I20260709 12:24:40.497967 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:40.491003 26588 raft_consensus.cc:515] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:40.523558 26750 leader_election.cc:290] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:40.525707 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 29 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:40.530486 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 26 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:40.531991 17681 raft_consensus.cc:2468] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 25.
I20260709 12:24:40.529570 26588 leader_election.cc:290] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:40.543823 26711 raft_consensus.cc:2804] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:24:40.559435 26711 raft_consensus.cc:493] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:40.548727 26707 raft_consensus.cc:493] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:40.560672 26707 raft_consensus.cc:515] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:40.563108 26707 leader_election.cc:290] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:40.550401 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:40.552022 26760 raft_consensus.cc:493] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:40.554039 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:40.596381 26760 raft_consensus.cc:515] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:40.597106 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 22 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:40.597420 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:40.597810 17689 raft_consensus.cc:2468] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 22.
I20260709 12:24:40.598754 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:40.599193 26760 leader_election.cc:290] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:40.556247 26587 raft_consensus.cc:493] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:24:40.600291 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:40.600226 26587 raft_consensus.cc:515] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:40.568073 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:40.601548 17689 raft_consensus.cc:2393] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 23: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:24:40.602246 18365 raft_consensus.cc:2393] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 24: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:24:40.572077 26750 raft_consensus.cc:493] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:24:40.573616 26588 raft_consensus.cc:493] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:40.574239 26354 raft_consensus.cc:493] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:24:40.575174 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:40.556356 26635 raft_consensus.cc:493] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
W20260709 12:24:40.579408 17829 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35040 (request call id 3440) took 1889 ms (client timeout 1574 ms). Trace:
I20260709 12:24:40.556342 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 22 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
W20260709 12:24:40.605734 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 123463us
W20260709 12:24:40.583930 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 1.58134e+06us
I20260709 12:24:40.556689 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:40.584435 26627 leader_election.cc:290] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 21 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:40.557821 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:40.558164 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:40.550711 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:40.551620 17684 raft_consensus.cc:3055] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:24:40.617214 18363 raft_consensus.cc:2393] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 28: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:24:40.553754 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:40.617764 17729 leader_election.cc:304] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
W20260709 12:24:40.591666 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 43853.9us
I20260709 12:24:40.612473 18333 leader_election.cc:400] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 28 pre-election: Vote denied by peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651) with higher term. Message: Invalid argument: T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 28. Current term is 29.
I20260709 12:24:40.621181 18333 leader_election.cc:403] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 28 pre-election: Cancelling election due to peer responding with higher term
W20260709 12:24:40.612839 17829 rpcz_store.cc:269] 0709 12:24:38.690346 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:38.690894 (+   548us) service_pool.cc:224] Handling call
0709 12:24:40.579136 (+1888242us) spinlock_profiling.cc:243] Waited 391 ms on lock 0x7b54000aa5d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:24:40.579330 (+   194us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":898262912}
I20260709 12:24:40.617147 17684 raft_consensus.cc:740] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:24:40.553431 18371 raft_consensus.cc:1275] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 21 index: 15. Preceding OpId from leader: term: 23 index: 16. (index mismatch)
I20260709 12:24:40.618330 17652 leader_election.cc:304] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:40.625357 18364 raft_consensus.cc:2393] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 22: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:24:40.627281 17685 raft_consensus.cc:2393] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 22: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:24:40.629841 17755 raft_consensus.cc:1275] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 26 index: 14. Preceding OpId from leader: term: 28 index: 16. (index mismatch)
I20260709 12:24:40.641307 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 26 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:40.622582 17684 consensus_queue.cc:260] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 23.19, Last appended by leader: 19, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:40.559824 26711 raft_consensus.cc:3060] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:40.642040 17652 leader_election.cc:304] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:40.642736 17684 raft_consensus.cc:3060] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:40.646195 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:40.647069 17689 raft_consensus.cc:3060] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:40.649545 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 29 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:40.641964 17757 raft_consensus.cc:2468] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 25.
I20260709 12:24:40.653858 17729 leader_election.cc:304] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:40.625805 26707 raft_consensus.cc:493] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:40.661650 17652 leader_election.cc:304] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:24:40.643080 26625 raft_consensus.cc:2749] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Leader pre-election lost for term 28. Reason: Vote denied by peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651) with higher term. Message: Invalid argument: T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 28. Current term is 29.
I20260709 12:24:40.644882 26593 raft_consensus.cc:2749] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:24:40.644614 26667 raft_consensus.cc:2749] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:24:40.664902 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:40.669971 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:40.671712 17838 raft_consensus.cc:1240] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 23 LEADER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 22. Current term is 23. Ops: [22.18-22.18]
I20260709 12:24:40.670925 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:40.671976 17685 raft_consensus.cc:2468] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 20.
I20260709 12:24:40.672431 17755 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 22.
I20260709 12:24:40.670131 17652 leader_election.cc:304] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:24:40.627699 26635 raft_consensus.cc:515] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:40.677089 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:40.677989 17758 raft_consensus.cc:2393] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 28: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:24:40.678215 26635 leader_election.cc:290] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:40.678925 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:40.679711 26733 raft_consensus.cc:2804] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:24:40.679986 26762 consensus_queue.cc:1059] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 21.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.016s
I20260709 12:24:40.680192 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 26 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:40.681228 17758 raft_consensus.cc:2468] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 25.
I20260709 12:24:40.681272 26762 raft_consensus.cc:3055] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:24:40.681999 26762 raft_consensus.cc:740] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:24:40.682323 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:40.657160 26674 consensus_queue.cc:1048] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.028s
I20260709 12:24:40.682580 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 28 candidate_status { last_received { term: 27 index: 21 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:40.683324 17831 raft_consensus.cc:1240] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 28 LEADER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 27. Current term is 28. Ops: [27.16-27.16]
I20260709 12:24:40.636879 26354 raft_consensus.cc:515] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:40.685801 26164 leader_election.cc:290] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:40.680250 17684 raft_consensus.cc:1275] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 23 index: 19. Preceding OpId from leader: term: 24 index: 19. (term mismatch)
I20260709 12:24:40.682042 18374 raft_consensus.cc:3060] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:40.686347 17684 pending_rounds.cc:85] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3: Aborting all ops after (but not including) 18
I20260709 12:24:40.686743 17684 pending_rounds.cc:107] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3: Aborting uncommitted NO_OP operation due to leader change: 23.19
I20260709 12:24:40.686972 17684 raft_consensus.cc:2889] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:40.681142 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:40.631678 26588 raft_consensus.cc:515] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:40.657704 26758 raft_consensus.cc:2749] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:24:40.689894 26588 leader_election.cc:290] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:40.671175 26606 raft_consensus.cc:2804] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:24:40.693656 26606 raft_consensus.cc:697] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 25 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:24:40.694617 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:40.695185 26640 raft_consensus.cc:493] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:40.681519 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:40.695756 26640 raft_consensus.cc:515] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:40.696933 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:40.697602 17683 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 22.
I20260709 12:24:40.697913 17763 raft_consensus.cc:1240] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 24. Current term is 25. Ops: []
I20260709 12:24:40.698738 26640 leader_election.cc:290] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:40.699085 17763 raft_consensus.cc:1240] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 24. Current term is 25. Ops: []
I20260709 12:24:40.693671 26420 raft_consensus.cc:493] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:40.694661 26606 consensus_queue.cc:237] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 23.13, Last appended by leader: 13, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:40.700392 26420 raft_consensus.cc:515] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:40.699803 18333 connection.cc:256] RPC call timeout handler was delayed by -1.736550368999815s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16710000000000003s for a response
W20260709 12:24:40.701280 18333 connection.cc:256] RPC call timeout handler was delayed by -1.5060953420002079s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15680000000000002s for a response
W20260709 12:24:40.701607 18333 connection.cc:256] RPC call timeout handler was delayed by -1.4643421050000143s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16160000000000002s for a response
W20260709 12:24:40.701911 18333 connection.cc:256] RPC call timeout handler was delayed by -1.3582525580000038s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19530000000000003s for a response
I20260709 12:24:40.670866 26701 raft_consensus.cc:493] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
W20260709 12:24:40.702674 18333 connection.cc:256] RPC call timeout handler was delayed by -1.2586503689999518s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2202s for a response
I20260709 12:24:40.703267 26701 raft_consensus.cc:515] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:40.703548 18333 leader_election.cc:304] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:40.705207 18333 leader_election.cc:304] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:40.705920 26544 raft_consensus.cc:2749] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
W20260709 12:24:40.707139 26619 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 24 current term: 24 is higher.
I20260709 12:24:40.671002 26732 raft_consensus.cc:493] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:40.708695 17760 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38776 (request call id 3126) took 1217 ms. Trace:
I20260709 12:24:40.649309 26519 consensus_queue.cc:1059] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 23.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.060s
I20260709 12:24:40.709195 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:40.709093 17760 rpcz_store.cc:276] 0709 12:24:39.490887 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:39.491574 (+   687us) service_pool.cc:224] Handling call
0709 12:24:39.491863 (+   289us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:39.607820 (+115957us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:40.708477 (+1100657us) spinlock_profiling.cc:243] Waited 86.6 ms on lock 0x7b54001b9254. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:24:40.708639 (+   162us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":199279360}
I20260709 12:24:40.709846 18333 leader_election.cc:304] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:24:40.711665 17688 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44898 (request call id 1722) took 1292 ms. Trace:
I20260709 12:24:40.711864 26354 leader_election.cc:290] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:40.712107 26625 raft_consensus.cc:2804] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:24:40.712404 26678 raft_consensus.cc:3055] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:24:40.712559 26625 raft_consensus.cc:493] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:40.712931 26678 raft_consensus.cc:740] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:24:40.713135 26625 raft_consensus.cc:3060] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:40.713964 26678 consensus_queue.cc:260] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 22.18, Last appended by leader: 18, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:40.715092 26678 raft_consensus.cc:3060] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:40.712117 17688 rpcz_store.cc:276] 0709 12:24:39.419169 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:39.495711 (+ 76542us) service_pool.cc:224] Handling call
0709 12:24:40.711459 (+1215748us) spinlock_profiling.cc:243] Waited 148 ms on lock 0x7b54003471d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:24:40.711604 (+   145us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":340551936}
I20260709 12:24:40.694796 26627 raft_consensus.cc:493] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:40.626919 26750 raft_consensus.cc:515] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:40.680169 26733 raft_consensus.cc:493] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:40.705305 26701 leader_election.cc:290] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:40.717809 18333 leader_election.cc:304] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:40.708428 18371 raft_consensus.cc:1275] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 24 index: 14. Preceding OpId from leader: term: 25 index: 15. (index mismatch)
I20260709 12:24:40.683416 17682 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44898 (request call id 1716) took 1268 ms. Trace:
I20260709 12:24:40.733462 17682 rpcz_store.cc:276] 0709 12:24:39.415466 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:39.415949 (+   483us) service_pool.cc:224] Handling call
0709 12:24:40.682471 (+1266522us) spinlock_profiling.cc:243] Waited 3.14 ms on lock 0x7b5800002a98. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b474fbb71 00007f7b474fbeee 000000000036eb39 00007f7b4f56f1e0 00007f7b4f6380c4 00007f7b4f638131 00007f7b4f5e60a0 00007f7b4f5e5984 00007f7b48b606d4 00007f7b48b60662
0709 12:24:40.683353 (+   882us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":7232512}
I20260709 12:24:40.668766 26653 raft_consensus.cc:3060] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:40.724220 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 28 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:40.726361 17681 raft_consensus.cc:2468] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 22.
W20260709 12:24:40.730628 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 56595.4us
I20260709 12:24:40.734061 26635 raft_consensus.cc:2749] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:24:40.723621 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 28 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:40.740074 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:40.746320 17681 raft_consensus.cc:2468] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 22.
I20260709 12:24:40.741132 18333 leader_election.cc:304] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:24:40.747426 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:40.748354 18333 leader_election.cc:304] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:40.749406 17758 raft_consensus.cc:2468] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 28.
I20260709 12:24:40.702535 26420 leader_election.cc:290] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:40.732035 26733 raft_consensus.cc:3060] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:40.751704 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:40.755314 18333 leader_election.cc:304] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:40.709020 26732 raft_consensus.cc:515] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:40.757917 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:40.758113 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:40.742594 18374 raft_consensus.cc:2468] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 25.
I20260709 12:24:40.735689 26587 leader_election.cc:290] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:40.765337 18333 leader_election.cc:304] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:40.757601 26340 raft_consensus.cc:2749] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:24:40.669054 26707 raft_consensus.cc:515] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:40.774129 18333 leader_election.cc:304] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:40.768363 17763 raft_consensus.cc:2393] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 28: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
W20260709 12:24:40.770555 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 66188.9us
I20260709 12:24:40.756948 26544 raft_consensus.cc:2804] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:24:40.778437 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:40.779170 18371 raft_consensus.cc:2468] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 20.
I20260709 12:24:40.779372 18333 leader_election.cc:304] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:24:40.730326 26627 raft_consensus.cc:515] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:40.781822 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:40.782773 17763 raft_consensus.cc:2393] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 22: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:24:40.783721 26641 raft_consensus.cc:2804] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:24:40.784322 26641 raft_consensus.cc:697] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 23 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:24:40.785090 26732 leader_election.cc:290] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:40.785219 26641 consensus_queue.cc:237] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 22.14, Last appended by leader: 14, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:40.776926 26635 raft_consensus.cc:2749] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:24:40.778693 26544 raft_consensus.cc:493] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:40.788518 26544 raft_consensus.cc:3060] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:40.788511 18333 leader_election.cc:304] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:40.798313 26635 raft_consensus.cc:2749] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:24:40.797767 26691 consensus_queue.cc:260] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 27.16, Last appended by leader: 16, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:40.799790 26691 raft_consensus.cc:3060] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:24:40.779170 26587 raft_consensus.cc:2804] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:24:40.800705 26587 raft_consensus.cc:493] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:40.801131 26587 raft_consensus.cc:3060] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:40.807731 17756 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38776 (request call id 3123) took 1317 ms. Trace:
I20260709 12:24:40.810039 17756 rpcz_store.cc:276] 0709 12:24:39.490035 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:39.490175 (+   140us) service_pool.cc:224] Handling call
0709 12:24:39.495754 (+  5579us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:39.884231 (+388477us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:40.807462 (+923231us) spinlock_profiling.cc:243] Waited 387 ms on lock 0x7b54000fc8d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:24:40.807652 (+   190us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":891179392}
I20260709 12:24:40.811367 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:40.773466 26420 raft_consensus.cc:2804] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:24:40.812309 26420 raft_consensus.cc:493] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:40.812420 26700 raft_consensus.cc:493] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:40.812755 26420 raft_consensus.cc:3060] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:40.813108 26700 raft_consensus.cc:515] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:40.815840 26730 raft_consensus.cc:493] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:40.816605 26730 raft_consensus.cc:515] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:40.819989 17764 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38786 (request call id 3896) took 2459 ms (client timeout 1591 ms). Trace:
W20260709 12:24:40.820478 17764 rpcz_store.cc:269] 0709 12:24:38.316941 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:38.317116 (+   175us) service_pool.cc:224] Handling call
0709 12:24:40.776351 (+2459235us) spinlock_profiling.cc:243] Waited 574 ms on lock 0x7b54003e62d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:24:40.776491 (+   140us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1321047424}
I20260709 12:24:40.821911 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:40.822041 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 27 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:40.823315 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:40.809294 26627 leader_election.cc:290] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:40.824676 26638 raft_consensus.cc:1064] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb: attempting to promote NON_VOTER 334a4e1f59944544b3b3ff36616b57da to VOTER
W20260709 12:24:40.825544 26638 raft_consensus.cc:1068] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb: Unable to promote non-voter 334a4e1f59944544b3b3ff36616b57da: Illegal state: Leader has not yet committed an operation in its own term
I20260709 12:24:40.826010 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:40.828563 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 22 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:40.829277 18366 raft_consensus.cc:3060] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:40.827193 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 21 candidate_status { last_received { term: 19 index: 19 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:40.832954 18364 raft_consensus.cc:2468] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 20.
I20260709 12:24:40.797794 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:40.841683 17728 leader_election.cc:304] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:40.744454 26762 consensus_queue.cc:260] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 21.14, Last appended by leader: 14, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:40.848305 17728 leader_election.cc:304] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:40.851297 17728 leader_election.cc:304] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:40.837942 17802 leader_election.cc:304] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:24:40.846261 18364 raft_consensus.cc:1240] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 21. Current term is 22. Ops: []
I20260709 12:24:40.853828 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 26 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:40.854177 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:40.857336 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 24 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:40.886965 17763 raft_consensus.cc:2393] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 24: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:24:40.894058 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 29 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:40.894980 18364 raft_consensus.cc:2468] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 28.
I20260709 12:24:40.888615 17728 leader_election.cc:304] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:40.899637 17681 raft_consensus.cc:1275] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 22 index: 18. Preceding OpId from leader: term: 23 index: 19. (index mismatch)
I20260709 12:24:40.900709 17763 raft_consensus.cc:1275] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 25 index: 19. Preceding OpId from leader: term: 26 index: 19. (term mismatch)
I20260709 12:24:40.902793 17763 pending_rounds.cc:85] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb: Aborting all ops after (but not including) 18
I20260709 12:24:40.903452 17763 pending_rounds.cc:107] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb: Aborting uncommitted NO_OP operation due to leader change: 25.19
I20260709 12:24:40.871838 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:40.903975 17763 raft_consensus.cc:2889] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:40.905820 17728 leader_election.cc:304] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:40.907162 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:40.909969 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:40.910863 18371 raft_consensus.cc:2468] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 19.
I20260709 12:24:40.911422 17728 leader_election.cc:304] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:40.913677 17728 leader_election.cc:304] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:40.917243 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:40.918568 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:40.870250 18366 raft_consensus.cc:2468] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 22.
I20260709 12:24:40.875331 18334 leader_election.cc:304] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
W20260709 12:24:40.920120 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 44892.8us
I20260709 12:24:40.920372 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:24:40.922408 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 252343us
I20260709 12:24:40.833806 26653 raft_consensus.cc:2749] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:24:40.864584 26752 raft_consensus.cc:2749] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:24:40.924809 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 29 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:40.927151 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:40.874030 26599 raft_consensus.cc:2804] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:24:40.930425 18371 raft_consensus.cc:2468] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 25.
I20260709 12:24:40.878801 26281 raft_consensus.cc:2804] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:24:40.925132 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:40.925688 17652 leader_election.cc:304] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:24:40.930269 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:40.896901 26354 raft_consensus.cc:2749] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:24:40.916394 26742 raft_consensus.cc:2804] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:24:40.859405 26635 consensus_queue.cc:1059] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.068s
I20260709 12:24:40.921434 26667 raft_consensus.cc:2749] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260709 12:24:40.989928 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 27 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:40.927131 26762 raft_consensus.cc:3060] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:40.922354 26627 raft_consensus.cc:2764] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Leader pre-election decision vote started in defunct term 21: won
I20260709 12:24:40.950451 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 26 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:40.873219 26701 consensus_queue.cc:1048] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.064s
I20260709 12:24:40.924449 25784 raft_consensus.cc:2804] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:24:40.995187 17729 leader_election.cc:304] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:40.916246 26717 raft_consensus.cc:2749] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:24:40.991655 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:40.994453 17654 leader_election.cc:304] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:24:41.006060 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 29 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:41.008666 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:41.008616 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:41.004622 25784 raft_consensus.cc:493] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:41.016866 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:24:41.025964 17655 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:40.939828 26678 raft_consensus.cc:2804] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:24:41.024549 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:41.026449 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:41.049309 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 20 candidate_status { last_received { term: 19 index: 19 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:41.053791 17684 raft_consensus.cc:2468] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 26.
I20260709 12:24:41.043406 17829 raft_consensus.cc:2410] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for term 29 because replica has last-logged OpId of term: 27 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 26 index: 14.
I20260709 12:24:41.018623 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:40.935976 26599 raft_consensus.cc:493] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:41.014245 17833 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35042 (request call id 3245) took 1775 ms. Trace:
I20260709 12:24:41.020196 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:41.059718 18363 raft_consensus.cc:2393] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 21: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:24:41.061733 18363 raft_consensus.cc:1240] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 23. Current term is 24. Ops: [21.18-23.19]
I20260709 12:24:40.921207 26755 consensus_queue.cc:1048] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.070s
I20260709 12:24:41.042351 17758 raft_consensus.cc:2468] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 22.
I20260709 12:24:41.010325 17689 raft_consensus.cc:2468] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 21.
W20260709 12:24:41.047312 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 49127.7us
I20260709 12:24:40.958487 26640 raft_consensus.cc:2749] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:24:41.052647 17729 leader_election.cc:304] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
W20260709 12:24:41.043519 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 49158us
I20260709 12:24:41.058126 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 24 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:41.049136 25784 raft_consensus.cc:3060] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:41.059336 17833 rpcz_store.cc:276] 0709 12:24:39.238368 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:39.252048 (+ 13680us) service_pool.cc:224] Handling call
0709 12:24:40.579733 (+1327685us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:40.580305 (+   572us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:40.580348 (+    43us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:40.580973 (+   625us) log.cc:844] Serialized 26 byte log entry
0709 12:24:40.666855 (+ 85882us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:24:40.666892 (+    37us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:40.666963 (+    71us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:40.729399 (+ 62436us) raft_consensus.cc:1685] finished
0709 12:24:40.729419 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:41.013827 (+284408us) spinlock_profiling.cc:243] Waited 1.33 s on lock 0x7b54000aa5d0. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a61f8ad 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:24:41.014151 (+   324us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":164,"raft.run_cpu_time_us":79,"raft.run_wall_time_us":79,"spinlock_wait_cycles":3052462848,"thread_start_us":85462,"threads_started":1,"wal-append.queue_time_us":146491}
I20260709 12:24:41.081506 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 28 candidate_status { last_received { term: 27 index: 21 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:41.082790 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:41.048269 26635 raft_consensus.cc:493] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:24:41.084388 18368 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 2575) took 2024 ms. Trace:
I20260709 12:24:41.084846 18368 rpcz_store.cc:276] 0709 12:24:39.059733 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:39.059871 (+   138us) service_pool.cc:224] Handling call
0709 12:24:39.060195 (+   324us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:40.698714 (+1638519us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:40.698747 (+    33us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:40.698795 (+    48us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:24:40.698828 (+    33us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:41.084166 (+385338us) spinlock_profiling.cc:243] Waited 1.64 s on lock 0x7b5400466ed4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:24:41.084212 (+    46us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:41.084331 (+   119us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3767800064}
W20260709 12:24:41.086127 26720 log.cc:927] Time spent T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 1.770s	user 0.012s	sys 0.012s
I20260709 12:24:41.086582 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:41.048730 26653 raft_consensus.cc:2804] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:24:41.088240 26653 raft_consensus.cc:493] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:24:41.088624 26653 raft_consensus.cc:3060] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:41.089299 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 24 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:41.090281 17680 raft_consensus.cc:2410] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for term 24 because replica has last-logged OpId of term: 22 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 17.
I20260709 12:24:41.091529 17680 raft_consensus.cc:1275] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 23 index: 15. Preceding OpId from leader: term: 25 index: 16. (index mismatch)
I20260709 12:24:41.049022 26701 raft_consensus.cc:2804] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:24:41.092424 26701 raft_consensus.cc:493] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:41.092878 26701 raft_consensus.cc:3060] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:24:41.095603 26653 raft_consensus.cc:515] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:41.097450 17764 raft_consensus.cc:1240] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 20. Current term is 21. Ops: []
I20260709 12:24:41.099283 26653 leader_election.cc:290] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 21 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:41.104321 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 20 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:41.105419 17689 raft_consensus.cc:2410] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for term 20 because replica has last-logged OpId of term: 18 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 13.
I20260709 12:24:41.105302 26625 raft_consensus.cc:515] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:41.107002 17689 raft_consensus.cc:1275] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 18 index: 14. Preceding OpId from leader: term: 19 index: 14. (term mismatch)
I20260709 12:24:41.107344 17689 pending_rounds.cc:85] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3: Aborting all ops after (but not including) 13
I20260709 12:24:41.107575 17689 pending_rounds.cc:107] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3: Aborting uncommitted NO_OP operation due to leader change: 18.14
I20260709 12:24:41.107815 17689 raft_consensus.cc:2889] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:41.109830 26707 leader_election.cc:290] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:41.111749 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:41.116369 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:41.116930 26693 consensus_queue.cc:1048] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.000s
I20260709 12:24:41.117656 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:41.071494 26750 leader_election.cc:290] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:41.118911 17802 leader_election.cc:304] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:24:41.119665 26693 raft_consensus.cc:2804] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:24:41.121083 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:41.121657 17755 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 22.
W20260709 12:24:41.122198 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 69642.3us
I20260709 12:24:41.067765 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 23 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:41.058889 18362 raft_consensus.cc:2393] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 20: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:24:41.047915 26483 raft_consensus.cc:3055] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:24:41.072343 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 24 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:41.141571 17727 leader_election.cc:304] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:41.142694 18363 raft_consensus.cc:1240] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 24. Current term is 25. Ops: []
I20260709 12:24:41.074771 26544 raft_consensus.cc:515] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:41.045774 26700 leader_election.cc:290] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:41.076463 26730 leader_election.cc:290] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:41.076820 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:41.154621 17684 raft_consensus.cc:2468] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 22.
I20260709 12:24:41.076494 18364 raft_consensus.cc:1275] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 20 index: 17. Preceding OpId from leader: term: 23 index: 19. (index mismatch)
I20260709 12:24:41.155751 17727 leader_election.cc:304] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:41.084210 26635 raft_consensus.cc:515] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:41.126047 17688 raft_consensus.cc:2410] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for term 29 because replica has last-logged OpId of term: 27 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 14.
I20260709 12:24:41.128477 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 29 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:41.156401 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:41.132160 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:41.136148 17803 leader_election.cc:304] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:24:41.138741 18333 leader_election.cc:304] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:41.070966 17654 leader_election.cc:304] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:24:41.154052 26667 raft_consensus.cc:2804] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:24:41.121562 26693 raft_consensus.cc:697] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 22 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
W20260709 12:24:41.158638 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 87810.2us
I20260709 12:24:41.159422 17654 leader_election.cc:304] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:41.159267 26693 consensus_queue.cc:237] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 20.16, Last appended by leader: 16, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:41.160566 17654 leader_election.cc:304] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:41.162864 26695 raft_consensus.cc:2804] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:24:41.163268 26695 raft_consensus.cc:493] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:24:41.145452 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 72513us
I20260709 12:24:41.163592 26695 raft_consensus.cc:3060] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:41.025157 26281 raft_consensus.cc:697] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:24:41.165325 17835 raft_consensus.cc:1240] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 27. Current term is 28. Ops: []
I20260709 12:24:41.165719 26281 consensus_queue.cc:237] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 23.17, Last appended by leader: 17, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:41.150593 18371 raft_consensus.cc:2393] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 24: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:24:41.168062 26707 raft_consensus.cc:2749] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:24:41.168705 18371 raft_consensus.cc:1275] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 22 index: 15. Preceding OpId from leader: term: 24 index: 16. (index mismatch)
I20260709 12:24:41.168815 26760 raft_consensus.cc:2749] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:24:41.052134 26678 raft_consensus.cc:493] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:24:41.169561 26760 raft_consensus.cc:2749] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:24:41.169783 26678 raft_consensus.cc:3060] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:41.107651 26625 leader_election.cc:290] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 23 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:41.171150 17681 raft_consensus.cc:1240] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 20. Current term is 21. Ops: [20.14-20.14]
I20260709 12:24:41.151283 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:41.172917 18363 raft_consensus.cc:1240] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 22. Current term is 23. Ops: []
I20260709 12:24:41.063503 26599 raft_consensus.cc:3060] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:41.140259 25784 raft_consensus.cc:515] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:41.176262 26732 raft_consensus.cc:493] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:41.176632 17763 raft_consensus.cc:1240] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 25. Current term is 26. Ops: [25.17-25.17]
I20260709 12:24:41.176777 26732 raft_consensus.cc:515] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:41.177402 25784 leader_election.cc:290] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 26 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:24:41.182474 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 46456.2us
I20260709 12:24:41.185456 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:41.162451 18333 leader_election.cc:304] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:41.186540 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:41.186717 17681 raft_consensus.cc:1275] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 27 index: 16. Preceding OpId from leader: term: 28 index: 17. (index mismatch)
I20260709 12:24:41.187690 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:41.188239 18333 leader_election.cc:304] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:41.188371 17838 raft_consensus.cc:2468] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 23.
I20260709 12:24:41.196209 17682 raft_consensus.cc:1240] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 25 LEADER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 24. Current term is 25. Ops: [24.14-24.14]
I20260709 12:24:41.197269 18366 raft_consensus.cc:2368] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 22. Current term is 23.
I20260709 12:24:41.197542 18333 leader_election.cc:304] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:41.199707 17727 leader_election.cc:304] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:41.141021 26483 raft_consensus.cc:740] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:24:41.206928 17727 leader_election.cc:304] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
W20260709 12:24:41.209743 17652 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:41.210407 18364 raft_consensus.cc:1240] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 27. Current term is 28. Ops: []
W20260709 12:24:41.213063 17652 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:41.154460 26750 raft_consensus.cc:493] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:41.215137 17803 leader_election.cc:304] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:41.215245 18374 raft_consensus.cc:1240] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 20. Current term is 21. Ops: []
I20260709 12:24:41.205251 26340 raft_consensus.cc:2749] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260709 12:24:41.196790 26730 raft_consensus.cc:2804] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:24:41.197363 26625 raft_consensus.cc:2804] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:24:41.186142 26587 raft_consensus.cc:515] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:41.146982 26164 consensus_queue.cc:1059] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.065s
I20260709 12:24:41.222957 26733 raft_consensus.cc:515] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:41.205096 26705 raft_consensus.cc:2749] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:24:41.181843 26675 raft_consensus.cc:2749] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260709 12:24:41.194751 26667 raft_consensus.cc:697] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 25 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:24:41.206918 26755 raft_consensus.cc:2749] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:24:41.211388 26739 raft_consensus.cc:2749] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
W20260709 12:24:41.238885 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 97240.9us
I20260709 12:24:41.195052 26700 consensus_queue.cc:1048] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.036s
I20260709 12:24:41.231294 26441 raft_consensus.cc:2749] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:24:41.264829 17680 raft_consensus.cc:1240] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 20. Current term is 21. Ops: []
I20260709 12:24:41.215016 26627 consensus_queue.cc:1059] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.029s
I20260709 12:24:41.234151 26625 raft_consensus.cc:493] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:41.233098 26730 raft_consensus.cc:493] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:41.263957 26700 consensus_queue.cc:1243] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer 0da169ff82674dc1947055ce43de68a3 log is divergent from this leader: its last log entry 22.18 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 17
I20260709 12:24:41.289304 26667 consensus_queue.cc:237] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 24.18, Last appended by leader: 18, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:41.292575 26544 leader_election.cc:290] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 23 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:41.293221 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:41.294411 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 24 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:41.295415 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:24:41.296454 17686 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37402 (request call id 3656) took 2029 ms (client timeout 2023 ms). Trace:
W20260709 12:24:41.296875 17686 rpcz_store.cc:269] 0709 12:24:39.266929 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:39.267078 (+   149us) service_pool.cc:224] Handling call
0709 12:24:41.296263 (+2029185us) spinlock_profiling.cc:243] Waited 162 ms on lock 0x7b54004254d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:24:41.296371 (+   108us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":372418432}
I20260709 12:24:41.301717 17685 raft_consensus.cc:2468] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 22.
I20260709 12:24:41.306000 18333 leader_election.cc:304] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:41.307616 26627 raft_consensus.cc:2749] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:24:41.317046 26733 leader_election.cc:290] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 21 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:41.230095 26750 raft_consensus.cc:515] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:41.322103 26693 consensus_queue.cc:1048] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.001s
W20260709 12:24:41.323011 26786 log.cc:927] Time spent T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.056s	user 0.000s	sys 0.000s
W20260709 12:24:41.324047 26787 log.cc:927] Time spent T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.059s	user 0.001s	sys 0.000s
I20260709 12:24:41.324482 26701 raft_consensus.cc:515] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:41.326794 17684 raft_consensus.cc:1240] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 28. Current term is 29. Ops: []
I20260709 12:24:41.280936 26691 consensus_queue.cc:1059] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 24.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.047s
I20260709 12:24:41.329300 26750 leader_election.cc:290] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:41.276741 26730 raft_consensus.cc:3060] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:41.332773 26635 leader_election.cc:290] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:41.274052 26625 raft_consensus.cc:3060] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:41.331298 26599 raft_consensus.cc:515] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:24:41.336863 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 107179us
I20260709 12:24:41.337604 17654 leader_election.cc:304] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:41.350111 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:41.285784 26789 raft_consensus.cc:3060] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:41.276980 26598 consensus_queue.cc:1048] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.064s
I20260709 12:24:41.364380 17834 raft_consensus.cc:1240] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 20. Current term is 21. Ops: []
I20260709 12:24:41.326843 26701 leader_election.cc:290] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 29 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:41.372112 26637 consensus_queue.cc:1048] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:24:41.373490 26730 raft_consensus.cc:515] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:41.374387 18334 leader_election.cc:304] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:24:41.375268 17829 raft_consensus.cc:1240] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 25. Current term is 26. Ops: []
I20260709 12:24:41.282891 26420 raft_consensus.cc:515] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:41.390856 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 29 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:41.394464 18363 raft_consensus.cc:3060] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:24:41.392642 18362 raft_consensus.cc:1275] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 27 index: 21. Preceding OpId from leader: term: 29 index: 22. (index mismatch)
I20260709 12:24:41.388938 18334 leader_election.cc:304] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:41.353955 26762 raft_consensus.cc:2749] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:24:41.400929 18334 leader_election.cc:304] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:41.401718 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:41.421684 18334 leader_election.cc:304] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:41.430354 18363 raft_consensus.cc:2468] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 29.
I20260709 12:24:41.437491 26587 leader_election.cc:290] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 21 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:24:41.440402 26711 consensus_meta.cc:287] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.779s	user 0.006s	sys 0.006s
I20260709 12:24:41.441125 26599 leader_election.cc:290] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 21 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:41.441049 26711 raft_consensus.cc:515] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:41.441928 26599 consensus_queue.cc:1048] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:24:41.442430 18368 raft_consensus.cc:1240] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 22. Current term is 23. Ops: []
I20260709 12:24:41.404443 26635 raft_consensus.cc:2749] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:24:41.443831 26705 raft_consensus.cc:2804] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:24:41.444314 26705 raft_consensus.cc:493] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:41.444465 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 25 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:41.445305 17762 raft_consensus.cc:2468] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 24.
I20260709 12:24:41.392798 26691 consensus_queue.cc:1059] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 22.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.071s
I20260709 12:24:41.444746 26705 raft_consensus.cc:3060] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:41.451722 26627 raft_consensus.cc:2804] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:24:41.452253 26627 raft_consensus.cc:493] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:41.452708 26627 raft_consensus.cc:3060] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Advancing to term 23
W20260709 12:24:41.454425 26788 log.cc:927] Time spent T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.059s	user 0.001s	sys 0.000s
I20260709 12:24:41.418598 26695 raft_consensus.cc:515] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:41.399406 26793 raft_consensus.cc:2804] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:24:41.456373 26793 raft_consensus.cc:697] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 21 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:24:41.457561 26695 leader_election.cc:290] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 20 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:41.457348 26793 consensus_queue.cc:237] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.14, Last appended by leader: 14, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:41.457758 26705 raft_consensus.cc:515] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:41.460923 26730 leader_election.cc:290] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 26 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:41.463009 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:41.463873 17689 raft_consensus.cc:2393] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 23: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:24:41.466403 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 25 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:41.468442 26691 raft_consensus.cc:3055] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:24:41.468830 26691 raft_consensus.cc:740] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:24:41.469601 26705 leader_election.cc:290] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 22 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:41.469733 26691 consensus_queue.cc:260] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 22.18, Last appended by leader: 18, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:41.471081 26691 raft_consensus.cc:3060] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:41.483776 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:41.479593 18333 leader_election.cc:304] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:41.497470 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:41.499950 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 27 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:41.439730 26420 leader_election.cc:290] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 27 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:41.443751 26711 leader_election.cc:290] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 23 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:24:41.512032 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 296959us
W20260709 12:24:41.509383 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 252369us
W20260709 12:24:41.513310 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 297394us
I20260709 12:24:41.514052 17803 leader_election.cc:304] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:41.496807 26544 raft_consensus.cc:2804] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:24:41.516047 17684 raft_consensus.cc:2468] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 22.
I20260709 12:24:41.496979 26742 raft_consensus.cc:493] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:24:41.520726 17652 leader_election.cc:304] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:24:41.525357 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:41.526449 17764 raft_consensus.cc:2393] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 22: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:24:41.493108 26701 consensus_queue.cc:1048] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.023s
I20260709 12:24:41.529971 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:41.541535 26798 consensus_queue.cc:1048] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:24:41.546140 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 20 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:41.546730 18364 raft_consensus.cc:1240] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 24. Current term is 25. Ops: []
I20260709 12:24:41.547382 26700 raft_consensus.cc:3055] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:24:41.516901 26544 raft_consensus.cc:493] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:41.521057 26164 consensus_queue.cc:1048] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.056s
I20260709 12:24:41.580008 17654 leader_election.cc:304] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:24:41.515733 26678 raft_consensus.cc:515] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:41.519495 26742 raft_consensus.cc:3060] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:41.602970 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:41.574424 26723 raft_consensus.cc:2749] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:24:41.608462 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 20 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:41.576193 26632 raft_consensus.cc:2804] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:24:41.578696 26544 raft_consensus.cc:3060] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:41.618197 17652 leader_election.cc:304] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:41.628041 17762 raft_consensus.cc:1240] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 27. Current term is 28. Ops: []
I20260709 12:24:41.629570 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:41.631021 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 29 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:41.627869 17652 leader_election.cc:304] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:41.547768 26700 raft_consensus.cc:740] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 21 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:24:41.612705 26709 consensus_queue.cc:1059] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 23.14, Next index: 15, Last known committed idx: 12, Time since last communication: 0.007s
I20260709 12:24:41.622692 26719 consensus_queue.cc:1059] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 0.0, Next index: 16, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:24:41.587486 26750 raft_consensus.cc:493] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:41.528038 26726 consensus_queue.cc:1059] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 19.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.068s
I20260709 12:24:41.636655 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:41.597734 26695 raft_consensus.cc:2804] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:24:41.670300 17764 raft_consensus.cc:2468] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 28.
I20260709 12:24:41.672169 26726 raft_consensus.cc:3055] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:24:41.672590 26726 raft_consensus.cc:740] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 20 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
W20260709 12:24:41.673934 26745 log.cc:927] Time spent T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 1.828s	user 0.002s	sys 0.018s
I20260709 12:24:41.673784 26726 consensus_queue.cc:260] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 20.14, Last appended by leader: 14, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:41.675048 26726 raft_consensus.cc:3060] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:41.675652 17759 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38776 (request call id 3124) took 2185 ms. Trace:
I20260709 12:24:41.676183 17759 rpcz_store.cc:276] 0709 12:24:39.490315 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:39.490752 (+   437us) service_pool.cc:224] Handling call
0709 12:24:39.491078 (+   326us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:39.683921 (+192843us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:39.683961 (+    40us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:39.684022 (+    61us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:24:39.684067 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:41.675372 (+1991305us) spinlock_profiling.cc:243] Waited 193 ms on lock 0x7b5400217854. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:24:41.675420 (+    48us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:41.675553 (+   133us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":442882944}
I20260709 12:24:41.678455 17759 raft_consensus.cc:1275] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 27 index: 21. Preceding OpId from leader: term: 29 index: 22. (index mismatch)
I20260709 12:24:41.605199 26762 consensus_queue.cc:1059] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 24.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.059s
I20260709 12:24:41.624926 26632 raft_consensus.cc:493] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:24:41.687301 17764 raft_consensus.cc:1275] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 20 index: 14. Preceding OpId from leader: term: 24 index: 16. (index mismatch)
I20260709 12:24:41.613392 26637 raft_consensus.cc:493] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:24:41.642864 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:41.595028 26625 raft_consensus.cc:515] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Starting leader election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:41.696956 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 176031us
W20260709 12:24:41.696491 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 57411.4us
I20260709 12:24:41.632978 26749 raft_consensus.cc:2749] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
W20260709 12:24:41.700605 26801 log.cc:927] Time spent T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.086s	user 0.000s	sys 0.001s
I20260709 12:24:41.702006 26678 leader_election.cc:290] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 20 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:41.695992 26695 raft_consensus.cc:697] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 29 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:24:41.708945 26800 raft_consensus.cc:3055] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:24:41.688267 26619 raft_consensus.cc:2804] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:24:41.715094 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 23 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
W20260709 12:24:41.715286 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 201305us
W20260709 12:24:41.716343 17802 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:41.693187 26632 raft_consensus.cc:3060] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:41.696241 26736 raft_consensus.cc:3060] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:41.743794 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 20 candidate_status { last_received { term: 19 index: 19 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:41.761687 18366 raft_consensus.cc:2393] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 20: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:24:41.745906 26758 raft_consensus.cc:493] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:41.718218 26710 consensus_queue.cc:1059] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 23.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.030s
I20260709 12:24:41.763787 26710 consensus_queue.cc:1243] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer 0da169ff82674dc1947055ce43de68a3 log is divergent from this leader: its last log entry 25.14 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 13
I20260709 12:24:41.763705 26758 raft_consensus.cc:515] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:41.764415 26710 raft_consensus.cc:3055] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:24:41.764786 26710 raft_consensus.cc:740] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 24 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:24:41.764950 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 26 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:41.761272 26627 raft_consensus.cc:515] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:41.765599 17762 raft_consensus.cc:3060] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:41.766003 26758 leader_election.cc:290] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:41.722661 26619 raft_consensus.cc:493] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:41.748311 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 26 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:41.691534 26750 raft_consensus.cc:515] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:41.789068 17762 raft_consensus.cc:2468] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 26.
W20260709 12:24:41.790441 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 90879us
I20260709 12:24:41.791381 17803 leader_election.cc:400] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 22 pre-election: Vote denied by peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577) with higher term. Message: Invalid argument: T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 22. Current term is 23.
I20260709 12:24:41.792474 17803 leader_election.cc:403] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 22 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:24:41.793754 17652 leader_election.cc:304] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:41.709019 26796 consensus_queue.cc:1059] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 24.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.065s
I20260709 12:24:41.769984 26619 raft_consensus.cc:3060] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:41.722625 26800 raft_consensus.cc:740] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:24:41.800415 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:41.710985 26637 raft_consensus.cc:515] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:41.784936 26727 raft_consensus.cc:493] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:41.796466 26749 raft_consensus.cc:2749] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 12:24:41.819993 18364 raft_consensus.cc:1275] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 19 index: 17. Preceding OpId from leader: term: 24 index: 19. (index mismatch)
I20260709 12:24:41.801630 26627 leader_election.cc:290] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 23 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:41.804736 26789 raft_consensus.cc:2749] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:24:41.826483 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 21 candidate_status { last_received { term: 19 index: 19 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:41.832093 18366 raft_consensus.cc:2468] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 20.
I20260709 12:24:41.765618 26710 consensus_queue.cc:260] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 24.14, Last appended by leader: 14, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:24:41.829164 26802 log.cc:927] Time spent T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.181s	user 0.001s	sys 0.000s
I20260709 12:24:41.813294 26808 raft_consensus.cc:3060] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Advancing to term 23
I20260709 12:24:41.772691 26700 consensus_queue.cc:260] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 21.15, Last appended by leader: 15, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:41.872505 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:41.885247 26637 leader_election.cc:290] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:41.886751 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:41.892150 26750 leader_election.cc:290] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:41.899748 26483 consensus_queue.cc:260] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 24.18, Last appended by leader: 18, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:41.901633 26483 raft_consensus.cc:3060] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:41.915997 17680 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37396 (request call id 3249) took 1257 ms. Trace:
I20260709 12:24:41.916607 17680 rpcz_store.cc:276] 0709 12:24:40.658688 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:41.146469 (+487781us) service_pool.cc:224] Handling call
0709 12:24:41.146773 (+   304us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:41.265290 (+118517us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:41.915615 (+650325us) spinlock_profiling.cc:243] Waited 118 ms on lock 0x7b540014dcd4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:24:41.915909 (+   294us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":271108480}
I20260709 12:24:41.921581 26732 leader_election.cc:290] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:41.938553 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:41.948004 17758 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36572 (request call id 1546) took 1206 ms. Trace:
I20260709 12:24:41.948189 17755 raft_consensus.cc:1240] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 21. Current term is 22. Ops: [21.15-21.15]
I20260709 12:24:41.948473 17758 rpcz_store.cc:276] 0709 12:24:40.741703 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:40.757494 (+ 15791us) service_pool.cc:224] Handling call
0709 12:24:41.947804 (+1190310us) spinlock_profiling.cc:243] Waited 195 ms on lock 0x7b540019f5d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:24:41.947946 (+   142us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":448132992}
I20260709 12:24:41.950537 18364 raft_consensus.cc:1240] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 28. Current term is 29. Ops: []
I20260709 12:24:41.951162 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:41.951815 17761 raft_consensus.cc:2468] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 28.
I20260709 12:24:41.952303 17761 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38786 (request call id 3907) took 1250 ms. Trace:
W20260709 12:24:41.935534 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 453308us
I20260709 12:24:41.960569 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 26 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:41.961429 17688 raft_consensus.cc:3060] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:41.963587 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:41.939762 17757 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38786 (request call id 3910) took 1237 ms. Trace:
I20260709 12:24:41.964649 17680 raft_consensus.cc:1240] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 24. Current term is 25. Ops: [23.17-24.18]
I20260709 12:24:41.965423 17757 rpcz_store.cc:276] 0709 12:24:40.702424 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:41.938123 (+1235699us) service_pool.cc:224] Handling call
0709 12:24:41.939683 (+  1560us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:24:41.830291 26805 log.cc:927] Time spent T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.059s	user 0.001s	sys 0.000s
I20260709 12:24:41.844630 26544 raft_consensus.cc:515] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:41.824883 26727 raft_consensus.cc:515] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:41.950615 17755 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38786 (request call id 3918) took 1191 ms. Trace:
I20260709 12:24:41.952594 17761 rpcz_store.cc:276] 0709 12:24:40.701490 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:41.950904 (+1249414us) service_pool.cc:224] Handling call
0709 12:24:41.952249 (+  1345us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:41.971832 17755 rpcz_store.cc:276] 0709 12:24:40.759507 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:41.140035 (+380528us) service_pool.cc:224] Handling call
0709 12:24:41.947948 (+807913us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:41.948685 (+   737us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:41.950422 (+  1737us) spinlock_profiling.cc:243] Waited 807 ms on lock 0x7b540019f5d0. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a61f8ad 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:24:41.950547 (+   125us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1857103104}
I20260709 12:24:41.986867 26727 leader_election.cc:290] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:41.987911 26234 consensus_queue.cc:1059] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20260709 12:24:41.988844 26234 raft_consensus.cc:3055] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:24:41.989497 26234 raft_consensus.cc:740] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 28 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:24:41.990656 26234 consensus_queue.cc:260] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 28.17, Last appended by leader: 17, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:41.992614 26234 raft_consensus.cc:3060] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:24:41.992813 18361 raft_consensus.cc:1240] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 24. Current term is 25. Ops: [24.16-24.16]
I20260709 12:24:41.820837 26695 consensus_queue.cc:237] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 26.16, Last appended by leader: 16, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:41.870846 26710 raft_consensus.cc:3060] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Advancing to term 25
W20260709 12:24:42.024303 17729 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:42.026211 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 30 candidate_status { last_received { term: 27 index: 21 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:41.984112 18333 leader_election.cc:304] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:42.038316 18365 raft_consensus.cc:1240] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 23. Current term is 24. Ops: []
I20260709 12:24:42.041011 18334 leader_election.cc:304] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:24:42.046146 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 62138.1us
I20260709 12:24:41.897257 26700 raft_consensus.cc:3060] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:41.776378 26697 consensus_queue.cc:1059] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 27.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.048s
I20260709 12:24:41.978159 26598 raft_consensus.cc:493] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:41.981580 26635 raft_consensus.cc:493] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:41.987789 26544 leader_election.cc:290] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 25 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:24:42.047716 26804 log.cc:927] Time spent T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.353s	user 0.003s	sys 0.001s
I20260709 12:24:42.075770 26641 raft_consensus.cc:2804] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:24:42.099509 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:41.868734 26800 consensus_queue.cc:260] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 27.16, Last appended by leader: 16, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:42.106494 17652 leader_election.cc:304] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:42.107386 17688 raft_consensus.cc:2468] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 26.
I20260709 12:24:42.112833 17685 raft_consensus.cc:2468] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 23.
I20260709 12:24:42.095764 26544 raft_consensus.cc:2749] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:24:42.080488 26519 consensus_queue.cc:1059] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 23.14, Next index: 15, Last known committed idx: 12, Time since last communication: 0.023s
W20260709 12:24:42.118080 17802 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:42.080488 26598 raft_consensus.cc:515] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:42.065922 26732 consensus_queue.cc:1059] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 22.15, Next index: 16, Last known committed idx: 14, Time since last communication: 0.037s
I20260709 12:24:42.127957 17652 leader_election.cc:304] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:24:42.088551 26635 raft_consensus.cc:515] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:42.105690 26758 raft_consensus.cc:493] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:42.105468 26641 raft_consensus.cc:697] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 22 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:24:42.097208 26632 raft_consensus.cc:515] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:42.118752 26277 raft_consensus.cc:2749] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:24:42.143255 17652 leader_election.cc:304] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:42.146508 26598 leader_election.cc:290] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:24:42.147884 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:42.149641 26814 raft_consensus.cc:493] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:42.156004 18371 raft_consensus.cc:1240] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 24. Current term is 25. Ops: []
W20260709 12:24:42.139923 17804 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:42.125824 26697 raft_consensus.cc:493] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:42.150417 26814 raft_consensus.cc:515] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:42.159087 17804 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:42.163794 26271 consensus_queue.cc:1048] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:24:42.143342 26761 raft_consensus.cc:2804] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Leader pre-election won for term 29
W20260709 12:24:42.178647 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 137734us
I20260709 12:24:42.136970 26797 raft_consensus.cc:493] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:24:42.181483 18334 leader_election.cc:304] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:42.125283 26812 consensus_queue.cc:1059] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.056s
I20260709 12:24:42.139395 26693 raft_consensus.cc:493] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:42.188915 18334 leader_election.cc:304] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:42.170930 26760 raft_consensus.cc:2749] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 12:24:42.193975 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 20 candidate_status { last_received { term: 19 index: 19 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:42.201745 17834 raft_consensus.cc:1275] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 27 index: 15. Preceding OpId from leader: term: 28 index: 16. (index mismatch)
I20260709 12:24:42.164831 26519 consensus_queue.cc:1048] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.028s
I20260709 12:24:42.180707 26761 raft_consensus.cc:493] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:42.139657 26758 raft_consensus.cc:515] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:42.204016 18334 leader_election.cc:304] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:42.208214 26544 raft_consensus.cc:493] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:42.222069 26632 leader_election.cc:290] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 26 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:42.227378 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:42.239794 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:24:42.247848 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 43906.8us
I20260709 12:24:42.175621 26697 raft_consensus.cc:515] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:42.259125 17833 raft_consensus.cc:1240] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 23. Current term is 24. Ops: []
I20260709 12:24:42.260339 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 26 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:42.260649 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 25 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:42.262022 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:42.265087 17686 raft_consensus.cc:3060] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:42.183621 26691 consensus_queue.cc:1059] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 24.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.046s
I20260709 12:24:42.270220 26234 raft_consensus.cc:1064] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb: attempting to promote NON_VOTER 334a4e1f59944544b3b3ff36616b57da to VOTER
W20260709 12:24:42.274920 26742 consensus_meta.cc:287] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.650s	user 0.002s	sys 0.007s
I20260709 12:24:42.299160 26816 raft_consensus.cc:493] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:42.299022 26742 raft_consensus.cc:515] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:42.276692 26758 leader_election.cc:290] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:42.277084 17686 raft_consensus.cc:2468] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 23.
I20260709 12:24:42.278034 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 25 candidate_status { last_received { term: 21 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:42.300930 18363 raft_consensus.cc:2468] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 24.
I20260709 12:24:42.229104 26640 raft_consensus.cc:2749] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:24:42.242641 26761 raft_consensus.cc:3060] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:24:42.287773 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 25 candidate_status { last_received { term: 21 index: 18 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:42.261869 26544 raft_consensus.cc:515] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:42.290124 26818 raft_consensus.cc:2804] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:24:42.296355 26635 leader_election.cc:290] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:42.204694 26820 raft_consensus.cc:2804] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:24:42.191203 26693 raft_consensus.cc:515] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:42.287113 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:42.248073 26641 consensus_queue.cc:237] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 20.14, Last appended by leader: 14, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:42.286828 26619 raft_consensus.cc:515] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:42.326417 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 30 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:42.255626 17838 raft_consensus.cc:2468] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 20.
I20260709 12:24:42.327431 17834 raft_consensus.cc:2468] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 29.
I20260709 12:24:42.303102 17654 leader_election.cc:304] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:24:42.331760 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:42.332193 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 26 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:42.333832 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 26 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:42.334934 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 30 candidate_status { last_received { term: 27 index: 21 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:42.304558 26816 raft_consensus.cc:515] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:42.311117 26732 consensus_queue.cc:1059] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true }, Status: INVALID_TERM, Last received: 23.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20260709 12:24:42.309238 26593 consensus_queue.cc:1048] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.003s
I20260709 12:24:42.317039 17763 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38786 (request call id 3914) took 1613 ms. Trace:
W20260709 12:24:42.324133 26736 consensus_meta.cc:287] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.580s	user 0.000s	sys 0.010s
I20260709 12:24:42.321554 26697 leader_election.cc:290] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:42.186969 26797 raft_consensus.cc:515] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:42.338689 26814 leader_election.cc:290] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:42.340569 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:42.340698 26617 raft_consensus.cc:2804] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:24:42.346598 17728 leader_election.cc:304] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
W20260709 12:24:42.422636 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 76116.4us
I20260709 12:24:42.346983 26654 raft_consensus.cc:493] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:42.347465 26667 raft_consensus.cc:493] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:42.355072 18365 raft_consensus.cc:1240] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 26. Current term is 27. Ops: []
I20260709 12:24:42.355929 18363 raft_consensus.cc:1240] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 22. Current term is 23. Ops: []
I20260709 12:24:42.318658 18368 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 2592) took 1325 ms. Trace:
I20260709 12:24:42.425215 18368 rpcz_store.cc:276] 0709 12:24:40.993441 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:41.184855 (+191414us) service_pool.cc:224] Handling call
0709 12:24:41.185244 (+   389us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:41.442855 (+257611us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:42.318173 (+875318us) spinlock_profiling.cc:243] Waited 257 ms on lock 0x7b54006946d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:24:42.318554 (+   381us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":591069056}
I20260709 12:24:42.378820 17833 raft_consensus.cc:1275] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 24 index: 14. Preceding OpId from leader: term: 25 index: 14. (term mismatch)
W20260709 12:24:42.379089 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 179884us
I20260709 12:24:42.426707 17833 pending_rounds.cc:85] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b: Aborting all ops after (but not including) 13
I20260709 12:24:42.426964 17833 pending_rounds.cc:107] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b: Aborting uncommitted NO_OP operation due to leader change: 24.14
I20260709 12:24:42.427189 17833 raft_consensus.cc:2889] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:42.378481 17763 rpcz_store.cc:276] 0709 12:24:40.704021 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:40.943802 (+239781us) service_pool.cc:224] Handling call
0709 12:24:40.944134 (+   332us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:41.177328 (+233194us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:42.316564 (+1139236us) spinlock_profiling.cc:243] Waited 232 ms on lock 0x7b5400216454. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:24:42.316942 (+   378us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":533789440}
I20260709 12:24:42.410935 17801 leader_election.cc:304] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:24:42.445195 17801 leader_election.cc:304] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:42.445901 18363 raft_consensus.cc:1275] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 19 index: 15. Preceding OpId from leader: term: 22 index: 17. (index mismatch)
I20260709 12:24:42.447162 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:42.451829 17801 leader_election.cc:304] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:42.453167 17801 leader_election.cc:304] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:42.464093 26736 raft_consensus.cc:2749] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:24:42.464847 26697 raft_consensus.cc:2749] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:24:42.465523 26697 raft_consensus.cc:2749] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:24:42.466039 17801 leader_election.cc:304] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:42.466372 26693 leader_election.cc:290] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:42.448664 26728 raft_consensus.cc:2804] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:24:42.466980 26728 raft_consensus.cc:493] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:24:42.467307 17801 leader_election.cc:304] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:24:42.467804 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 155630us
I20260709 12:24:42.468439 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:42.469254 18368 raft_consensus.cc:2393] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 21: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:24:42.470037 26816 leader_election.cc:290] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:42.412653 26799 raft_consensus.cc:493] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:42.470512 26697 raft_consensus.cc:2749] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:24:42.470808 26799 raft_consensus.cc:515] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:42.467485 26728 raft_consensus.cc:3060] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:42.472543 26594 consensus_queue.cc:1059] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 22.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20260709 12:24:42.473382 26816 consensus_queue.cc:1048] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20260709 12:24:42.473750 26688 raft_consensus.cc:3055] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:24:42.474185 26688 raft_consensus.cc:740] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:24:42.474550 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:42.466192 17756 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38786 (request call id 3906) took 1765 ms. Trace:
I20260709 12:24:42.475198 17836 raft_consensus.cc:3060] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:42.475014 26688 consensus_queue.cc:260] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 24.16, Last appended by leader: 16, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:42.476723 26764 raft_consensus.cc:3055] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:24:42.477150 26764 raft_consensus.cc:740] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:24:42.477990 26764 consensus_queue.cc:260] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 22.17, Last appended by leader: 17, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:42.475253 17756 rpcz_store.cc:276] 0709 12:24:40.701134 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:40.883473 (+182339us) service_pool.cc:224] Handling call
0709 12:24:40.924985 (+ 41512us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:41.025381 (+100396us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:41.025431 (+    50us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:41.323428 (+297997us) spinlock_profiling.cc:243] Waited 2.33 ms on lock 0x7b2c0000e5c8. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b48f19a11 00007f7b48f29372 00007f7b4a65ba01 00007f7b4a62043a 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2
0709 12:24:41.324051 (+   623us) log.cc:844] Serialized 26 byte log entry
0709 12:24:41.496441 (+172390us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:24:41.496477 (+    36us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:42.465850 (+969373us) spinlock_profiling.cc:243] Waited 50.7 ms on lock 0x7b540019f0d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:24:42.465907 (+    57us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:42.465956 (+    49us) raft_consensus.cc:1685] finished
0709 12:24:42.465975 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:42.466113 (+   138us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":49097,"raft.queue_time_us":49298,"raft.run_cpu_time_us":131,"raft.run_wall_time_us":6406,"spinlock_wait_cycles":121865344,"thread_start_us":171958,"threads_started":1,"wal-append.queue_time_us":258307}
I20260709 12:24:42.480605 26810 raft_consensus.cc:3055] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:24:42.481079 26810 raft_consensus.cc:740] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 27 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:24:42.481938 26810 consensus_queue.cc:260] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 27.19, Last appended by leader: 19, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:42.482904 17685 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44898 (request call id 1744) took 1021 ms. Trace:
I20260709 12:24:42.483320 17685 rpcz_store.cc:276] 0709 12:24:41.461880 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:41.468353 (+  6473us) service_pool.cc:224] Handling call
0709 12:24:41.714940 (+246587us) spinlock_profiling.cc:243] Waited 9.82 ms on lock 0x7b5800002a98. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b474fbb71 00007f7b474fba22 00007f7b504148d9 00007f7b4f56eee0 00007f7b4f6380ab 00007f7b4f638131 00007f7b4f5e60a0 00007f7b4f5e5984 00007f7b48b606d4 00007f7b48b60662
0709 12:24:42.482741 (+767801us) spinlock_profiling.cc:243] Waited 110 ms on lock 0x7b54000c4c54. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:24:42.482848 (+   107us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":274882176}
I20260709 12:24:42.459568 17835 raft_consensus.cc:2468] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 27.
I20260709 12:24:42.469498 17652 leader_election.cc:304] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:42.495221 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:42.498214 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 21 candidate_status { last_received { term: 19 index: 19 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:42.499003 17687 raft_consensus.cc:2468] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 20.
I20260709 12:24:42.497258 17836 raft_consensus.cc:2468] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 24.
I20260709 12:24:42.501916 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:42.503013 17687 raft_consensus.cc:2393] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 21: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:24:42.503783 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:42.445763 26617 raft_consensus.cc:493] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:42.512950 17682 raft_consensus.cc:1240] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 25. Current term is 26. Ops: []
I20260709 12:24:42.514204 17683 raft_consensus.cc:1275] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 19 index: 15. Preceding OpId from leader: term: 22 index: 17. (index mismatch)
I20260709 12:24:42.518018 17801 leader_election.cc:304] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:42.520718 17801 leader_election.cc:304] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:42.457641 26814 raft_consensus.cc:2804] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:24:42.525116 17686 pending_rounds.cc:85] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3: Aborting all ops after (but not including) 17
W20260709 12:24:42.530475 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 61063.5us
I20260709 12:24:42.530905 17681 raft_consensus.cc:1240] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 19. Current term is 20. Ops: [19.14-19.14]
I20260709 12:24:42.528539 17686 pending_rounds.cc:107] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3: Aborting uncommitted NO_OP operation due to leader change: 22.18
I20260709 12:24:42.532680 17686 raft_consensus.cc:2889] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:42.534891 18334 leader_election.cc:304] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
W20260709 12:24:42.536374 26808 consensus_meta.cc:287] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.665s	user 0.009s	sys 0.002s
I20260709 12:24:42.536814 26808 raft_consensus.cc:2749] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Leader pre-election lost for term 22. Reason: Vote denied by peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577) with higher term. Message: Invalid argument: T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 22. Current term is 23.
I20260709 12:24:42.538190 26593 raft_consensus.cc:2804] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:24:42.538733 26593 raft_consensus.cc:697] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 20 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:24:42.539624 26593 consensus_queue.cc:237] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 19.19, Last appended by leader: 19, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:42.451385 26667 raft_consensus.cc:515] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:42.544071 26667 leader_election.cc:290] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:42.545145 26728 raft_consensus.cc:515] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:42.547390 26728 leader_election.cc:290] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 27 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:42.521332 26617 raft_consensus.cc:3060] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:42.451496 26654 raft_consensus.cc:515] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:42.550638 26654 leader_election.cc:290] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:42.522924 26710 raft_consensus.cc:2749] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:24:42.467020 26693 raft_consensus.cc:2749] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:24:42.472458 26619 leader_election.cc:290] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 24 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:24:42.554622 17804 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:42.559147 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 30 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:42.555941 18368 raft_consensus.cc:1275] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 22 index: 18. Preceding OpId from leader: term: 26 index: 19. (index mismatch)
I20260709 12:24:42.564116 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:42.522605 26789 raft_consensus.cc:493] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:42.568396 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 46305.4us
I20260709 12:24:42.526801 26700 raft_consensus.cc:2804] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:24:42.571259 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:42.572333 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 22 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:42.572175 26700 raft_consensus.cc:493] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:24:42.576794 26700 raft_consensus.cc:3060] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:42.578424 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:42.579284 17838 raft_consensus.cc:2410] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for term 22 because replica has last-logged OpId of term: 21 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 14.
I20260709 12:24:42.580210 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:42.580519 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 21 candidate_status { last_received { term: 19 index: 19 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:42.580842 17832 raft_consensus.cc:2468] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 25.
I20260709 12:24:42.581732 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 26 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:42.584223 26733 raft_consensus.cc:3055] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:24:42.584632 26733 raft_consensus.cc:740] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:24:42.585269 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:42.585480 26733 consensus_queue.cc:260] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 24.18, Last appended by leader: 18, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:42.586439 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 24 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:42.586666 26733 raft_consensus.cc:3060] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:42.587226 17835 raft_consensus.cc:2410] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for term 24 because replica has last-logged OpId of term: 23 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 15.
I20260709 12:24:42.576545 26730 consensus_queue.cc:1059] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 20.14, Next index: 15, Last known committed idx: 13, Time since last communication: 0.000s
I20260709 12:24:42.581166 17837 raft_consensus.cc:2468] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 20.
I20260709 12:24:42.591542 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:42.591897 18333 leader_election.cc:304] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:42.589643 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 27 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:42.593429 17835 raft_consensus.cc:3060] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:42.597294 18333 leader_election.cc:304] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:42.592617 17837 raft_consensus.cc:2468] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 23.
I20260709 12:24:42.566934 26814 raft_consensus.cc:697] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 28 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:24:42.569067 26640 raft_consensus.cc:2804] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:24:42.568496 26742 leader_election.cc:290] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 26 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:42.604902 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:42.606078 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
W20260709 12:24:42.610836 17802 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:42.611519 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:42.611649 17802 leader_election.cc:304] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:24:42.612577 17837 raft_consensus.cc:3060] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:42.595785 26619 raft_consensus.cc:493] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:42.602990 26635 raft_consensus.cc:2804] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:24:42.614858 26640 raft_consensus.cc:697] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 23 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:24:42.639495 17838 raft_consensus.cc:2468] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 21.
I20260709 12:24:42.610899 26793 raft_consensus.cc:2749] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
W20260709 12:24:42.652812 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 197046us
I20260709 12:24:42.575472 26789 raft_consensus.cc:515] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:42.658805 17830 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 24.
W20260709 12:24:42.653249 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 229250us
W20260709 12:24:42.661981 17729 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:42.662348 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 27 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:42.662708 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:42.662967 18368 raft_consensus.cc:2468] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 26.
I20260709 12:24:42.608985 26818 raft_consensus.cc:697] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [term 26 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:24:42.657624 18363 raft_consensus.cc:1240] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 20. Current term is 21. Ops: []
I20260709 12:24:42.658147 17837 raft_consensus.cc:2468] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 21.
I20260709 12:24:42.619767 26601 raft_consensus.cc:2804] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:24:42.670341 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:42.670745 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 23 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:42.672273 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 26 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:42.608788 26820 raft_consensus.cc:697] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 22 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:24:42.667975 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:42.625715 26544 leader_election.cc:290] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:42.677700 17728 leader_election.cc:304] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:42.631423 26635 raft_consensus.cc:493] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:42.680444 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:42.681838 17831 raft_consensus.cc:2468] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 25.
W20260709 12:24:42.682497 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 92021us
I20260709 12:24:42.685325 17728 leader_election.cc:304] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:42.686080 17729 leader_election.cc:304] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:42.669703 26601 raft_consensus.cc:493] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:42.640766 26697 consensus_queue.cc:1048] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.041s
I20260709 12:24:42.702836 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 24 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:42.624725 26617 raft_consensus.cc:515] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:42.631085 26619 raft_consensus.cc:515] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:42.719065 26835 raft_consensus.cc:670] T a5732e71b9d54f8786b8c511d69943bf P 334a4e1f59944544b3b3ff36616b57da: failed to trigger leader election: Illegal state: only voting members can start elections: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } [suppressed 3 similar messages]
I20260709 12:24:42.667279 26823 consensus_queue.cc:1059] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 16.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.054s
I20260709 12:24:42.720180 17755 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c"
dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
mode: GRACEFUL
new_leader_uuid: "334a4e1f59944544b3b3ff36616b57da"
 from {username='slave'} at 127.0.0.1:49786
I20260709 12:24:42.720690 26823 raft_consensus.cc:3055] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:24:42.721174 17755 raft_consensus.cc:606] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Received request to transfer leadership to TS 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:42.721212 26823 raft_consensus.cc:740] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 19 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:24:42.721846 17755 raft_consensus.cc:614] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:24:42.723111 26836 raft_consensus.cc:2804] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:24:42.681789 26635 raft_consensus.cc:3060] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:42.641211 26824 consensus_queue.cc:1059] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 25.17, Next index: 18, Last known committed idx: 16, Time since last communication: 0.072s
I20260709 12:24:42.712632 26789 leader_election.cc:290] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:42.713075 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 26 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:42.714393 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 26 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:42.729215 26824 raft_consensus.cc:3055] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:24:42.680208 26638 consensus_queue.cc:1059] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 20.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.036s
I20260709 12:24:42.729609 26824 raft_consensus.cc:740] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 25 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:24:42.730535 26638 raft_consensus.cc:3055] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:24:42.731182 26638 raft_consensus.cc:740] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 20 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:24:42.732100 26638 consensus_queue.cc:260] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.13, Last appended by leader: 13, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:42.733294 26638 raft_consensus.cc:3060] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:42.718231 26617 leader_election.cc:290] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 25 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:42.669422 26654 consensus_queue.cc:1059] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 26.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.035s
I20260709 12:24:42.734699 26654 raft_consensus.cc:3055] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:24:42.735034 26654 raft_consensus.cc:740] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:24:42.710939 26581 raft_consensus.cc:2749] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:24:42.722509 26823 consensus_queue.cc:260] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 19.14, Last appended by leader: 14, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:42.728773 26826 log.cc:927] Time spent T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.088s	user 0.000s	sys 0.000s
I20260709 12:24:42.729635 17683 raft_consensus.cc:2393] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 26: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:24:42.717082 17685 raft_consensus.cc:1240] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 24. Current term is 25. Ops: [24.19-24.19]
I20260709 12:24:42.690733 26829 raft_consensus.cc:493] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:42.705762 26601 raft_consensus.cc:3060] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Advancing to term 21
W20260709 12:24:42.736140 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 328066us
I20260709 12:24:42.743150 17727 leader_election.cc:304] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:42.748396 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:42.748636 26810 raft_consensus.cc:3060] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:24:42.752064 26797 leader_election.cc:290] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:42.738358 26640 consensus_queue.cc:237] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 21.16, Last appended by leader: 16, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:42.738145 17728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:42.743638 17835 raft_consensus.cc:2468] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 27.
I20260709 12:24:42.757340 17727 leader_election.cc:304] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:42.757521 17838 raft_consensus.cc:1275] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 27 index: 18. Preceding OpId from leader: term: 29 index: 19. (index mismatch)
I20260709 12:24:42.735817 26654 consensus_queue.cc:260] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 26.20, Last appended by leader: 20, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:42.760075 26747 raft_consensus.cc:2749] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 12:24:42.763070 26619 leader_election.cc:290] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:24:42.754256 26483 consensus_meta.cc:287] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.850s	user 0.011s	sys 0.002s
I20260709 12:24:42.723686 26836 raft_consensus.cc:493] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:42.736681 26838 raft_consensus.cc:2749] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260709 12:24:42.802155 17728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:42.804917 17727 leader_election.cc:304] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:42.766316 17830 raft_consensus.cc:2468] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 23.
W20260709 12:24:42.807631 17801 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:42.809922 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 125046us
I20260709 12:24:42.697806 26814 consensus_queue.cc:237] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 26.15, Last appended by leader: 15, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:42.810606 18371 raft_consensus.cc:1275] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 24 index: 18. Preceding OpId from leader: term: 25 index: 18. (term mismatch)
I20260709 12:24:42.814925 18371 pending_rounds.cc:85] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da: Aborting all ops after (but not including) 17
I20260709 12:24:42.765503 26823 raft_consensus.cc:3060] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:42.813175 17727 leader_election.cc:304] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:42.815330 18371 pending_rounds.cc:107] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da: Aborting uncommitted NO_OP operation due to leader change: 24.18
I20260709 12:24:42.818563 18371 raft_consensus.cc:2889] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:42.810070 18333 leader_election.cc:304] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:42.730618 26824 consensus_queue.cc:260] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 25.17, Last appended by leader: 17, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:42.823210 18333 leader_election.cc:304] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:24:42.825647 17838 raft_consensus.cc:1240] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 20. Current term is 21. Ops: []
I20260709 12:24:42.806957 26622 raft_consensus.cc:2749] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:24:42.805315 26836 raft_consensus.cc:3060] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:42.824510 26483 raft_consensus.cc:2749] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:24:42.815335 26654 raft_consensus.cc:3060] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:42.828266 18333 leader_election.cc:304] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:42.829806 17755 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278"
dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
mode: GRACEFUL
new_leader_uuid: "334a4e1f59944544b3b3ff36616b57da"
 from {username='slave'} at 127.0.0.1:49786
I20260709 12:24:42.832221 17755 raft_consensus.cc:606] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 LEADER]: Received request to transfer leadership to TS 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:42.814417 26714 raft_consensus.cc:2749] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:24:42.832608 18363 raft_consensus.cc:1275] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 24 index: 18. Preceding OpId from leader: term: 25 index: 19. (index mismatch)
I20260709 12:24:42.835107 18333 leader_election.cc:304] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:42.839388 26835 raft_consensus.cc:2804] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:24:42.842134 26736 raft_consensus.cc:3055] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:24:42.842603 26736 raft_consensus.cc:740] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 20 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
W20260709 12:24:42.844547 26833 log.cc:927] Time spent T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.088s	user 0.000s	sys 0.000s
I20260709 12:24:42.850085 26625 leader_election.cc:290] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 26 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:42.854679 17837 raft_consensus.cc:1240] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 26. Current term is 27. Ops: []
I20260709 12:24:42.857635 18333 leader_election.cc:304] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:42.862532 18333 leader_election.cc:304] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:42.875456 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 27 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:42.877758 17829 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35040 (request call id 3471) took 1692 ms. Trace:
I20260709 12:24:42.878152 17829 rpcz_store.cc:276] 0709 12:24:41.185056 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:41.185172 (+   116us) service_pool.cc:224] Handling call
0709 12:24:41.192632 (+  7460us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:41.392255 (+199623us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:42.877544 (+1485289us) spinlock_profiling.cc:243] Waited 182 ms on lock 0x7b540008cfd4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:24:42.877703 (+   159us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":419607424}
I20260709 12:24:42.878727 17803 leader_election.cc:304] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:42.756799 26829 raft_consensus.cc:515] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:42.831171 26587 raft_consensus.cc:2804] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:24:42.828233 26837 raft_consensus.cc:2749] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:24:42.886811 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 25 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:42.860726 26824 raft_consensus.cc:3060] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:42.760076 26820 consensus_queue.cc:237] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 21.14, Last appended by leader: 14, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:42.887671 17756 raft_consensus.cc:3060] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:42.893702 17803 leader_election.cc:304] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:42.890410 17829 raft_consensus.cc:1240] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 23. Current term is 24. Ops: []
I20260709 12:24:42.885613 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 26 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
W20260709 12:24:42.892805 17728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:42.866886 26483 raft_consensus.cc:2749] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:24:42.866627 26835 raft_consensus.cc:697] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 21 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:24:42.916657 26835 consensus_queue.cc:237] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 20.15, Last appended by leader: 15, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:42.919730 26834 log.cc:927] Time spent T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.111s	user 0.000s	sys 0.000s
I20260709 12:24:42.828902 26742 consensus_queue.cc:1059] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 20.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.057s
I20260709 12:24:42.843499 26736 consensus_queue.cc:260] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 20.14, Last appended by leader: 14, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:42.811106 26818 consensus_queue.cc:237] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 19.12, Last appended by leader: 12, Current term: 26, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:42.877025 26544 raft_consensus.cc:2749] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:24:42.920995 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 27 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:42.882807 26793 raft_consensus.cc:2804] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:24:42.927765 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:24:42.896522 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 69213.5us
I20260709 12:24:42.856869 26714 consensus_queue.cc:1048] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.041s
I20260709 12:24:42.885111 26587 raft_consensus.cc:493] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:24:42.928447 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 22 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:42.926216 17756 raft_consensus.cc:2468] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 25.
I20260709 12:24:42.928447 17761 raft_consensus.cc:2468] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 26.
I20260709 12:24:42.888267 26843 raft_consensus.cc:2749] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:24:42.913981 26789 raft_consensus.cc:2749] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:24:42.873430 26700 raft_consensus.cc:515] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:42.935629 17836 raft_consensus.cc:1275] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 20 index: 14. Preceding OpId from leader: term: 21 index: 15. (index mismatch)
I20260709 12:24:42.936519 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:42.933240 26587 raft_consensus.cc:3060] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:42.939023 26483 consensus_queue.cc:1059] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20260709 12:24:42.940711 17757 raft_consensus.cc:1275] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 20 index: 14. Preceding OpId from leader: term: 22 index: 15. (index mismatch)
I20260709 12:24:42.888062 26723 consensus_queue.cc:1048] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.042s
I20260709 12:24:42.944119 26736 raft_consensus.cc:3060] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:42.939643 26635 raft_consensus.cc:515] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:42.913345 26622 consensus_queue.cc:1048] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.031s
I20260709 12:24:42.935150 26793 raft_consensus.cc:697] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 27 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:24:42.945704 26844 raft_consensus.cc:3055] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:24:42.948663 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:42.975466 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:42.976032 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:42.974808 17837 raft_consensus.cc:2410] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for term 22 because replica has last-logged OpId of term: 21 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 14.
I20260709 12:24:42.898461 26730 consensus_queue.cc:1048] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 14, Time since last communication: 0.037s
W20260709 12:24:42.958500 26761 consensus_meta.cc:287] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.633s	user 0.010s	sys 0.000s
I20260709 12:24:42.976634 17833 raft_consensus.cc:2376] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 21. Re-sending same reply.
W20260709 12:24:42.978498 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 54158.8us
I20260709 12:24:42.979631 17838 raft_consensus.cc:1240] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 22. Current term is 23. Ops: []
I20260709 12:24:42.930581 26710 consensus_queue.cc:1048] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.046s
I20260709 12:24:42.983268 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:42.954007 18333 leader_election.cc:304] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:24:42.992491 17833 raft_consensus.cc:2376] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 24. Re-sending same reply.
I20260709 12:24:42.985277 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 21 candidate_status { last_received { term: 19 index: 19 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:42.984356 17803 leader_election.cc:304] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:42.975134 26844 raft_consensus.cc:740] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:24:42.994774 17803 leader_election.cc:304] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
W20260709 12:24:42.930295 26537 consensus_queue.cc:1175] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "334a4e1f59944544b3b3ff36616b57da" responder_term: 20 status { last_received { term: 20 index: 14 } last_committed_idx: 14 last_received_current_leader { term: 20 index: 14 } }
I20260709 12:24:43.177294 26818 raft_consensus.cc:2804] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:24:43.178615 26368 raft_consensus.cc:493] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:43.179060 26818 raft_consensus.cc:697] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 25 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:24:43.180826 26635 leader_election.cc:290] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 26 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:24:43.081746 26845 log.cc:927] Time spent T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.079s	user 0.000s	sys 0.000s
W20260709 12:24:43.000618 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 46698.3us
I20260709 12:24:43.082198 26793 consensus_queue.cc:237] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 26.20, Last appended by leader: 20, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:43.122304 26599 raft_consensus.cc:493] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:42.982590 26761 raft_consensus.cc:515] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:42.984889 26601 raft_consensus.cc:515] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:43.003352 26843 raft_consensus.cc:3060] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:43.050719 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 27 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:43.186605 18333 leader_election.cc:304] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:43.181701 26836 raft_consensus.cc:515] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:43.190425 26796 raft_consensus.cc:493] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:43.179247 26368 raft_consensus.cc:515] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:43.205317 18333 leader_election.cc:304] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:43.186210 26519 raft_consensus.cc:493] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:43.210593 17689 raft_consensus.cc:1240] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 28. Current term is 29. Ops: [28.17-28.17]
W20260709 12:24:43.221309 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 226686us
W20260709 12:24:43.218567 17801 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:43.208468 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:43.226608 17837 raft_consensus.cc:2393] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 21: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:24:43.236348 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:43.229310 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:43.193588 26700 leader_election.cc:290] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 23 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:43.202102 26635 raft_consensus.cc:2804] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:24:43.245887 17763 raft_consensus.cc:2393] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 27: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:24:43.249312 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 26 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:43.247902 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:43.252764 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 25 candidate_status { last_received { term: 21 index: 18 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:43.237387 26587 raft_consensus.cc:515] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:43.208578 26705 raft_consensus.cc:493] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:43.204327 26627 consensus_queue.cc:1048] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.041s
I20260709 12:24:43.227762 26730 raft_consensus.cc:2749] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:24:43.195797 26599 raft_consensus.cc:515] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:43.233659 26761 leader_election.cc:290] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 29 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:43.248611 26635 raft_consensus.cc:493] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:43.180821 26818 consensus_queue.cc:237] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 22.15, Last appended by leader: 15, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:43.276522 17829 raft_consensus.cc:1240] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 23. Current term is 24. Ops: []
I20260709 12:24:43.257223 26638 raft_consensus.cc:493] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:43.228914 26796 raft_consensus.cc:515] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:43.239454 26519 raft_consensus.cc:515] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:43.208472 26420 consensus_queue.cc:1048] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.056s
I20260709 12:24:43.258921 26742 raft_consensus.cc:493] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
W20260709 12:24:43.291575 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 86331.7us
I20260709 12:24:43.296362 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:43.276446 26635 raft_consensus.cc:3060] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:43.221256 26844 consensus_queue.cc:260] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 20.15, Last appended by leader: 15, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:43.273231 26829 leader_election.cc:290] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:24:43.309913 17652 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:43.278023 26640 raft_consensus.cc:2749] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260709 12:24:43.266150 26705 raft_consensus.cc:515] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:43.283527 26544 raft_consensus.cc:493] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:43.325763 26844 raft_consensus.cc:3060] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:43.329361 26705 leader_election.cc:290] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:43.330634 26764 raft_consensus.cc:3060] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:43.331394 26836 leader_election.cc:290] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 27 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:43.323494 26544 raft_consensus.cc:515] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:43.333950 26544 leader_election.cc:290] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:43.330679 26587 leader_election.cc:290] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 23 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:24:43.342790 26846 log.cc:927] Time spent T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.150s	user 0.000s	sys 0.000s
W20260709 12:24:43.343981 26851 log.cc:927] Time spent T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.154s	user 0.001s	sys 0.000s
W20260709 12:24:43.350572 17728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:43.351456 26719 raft_consensus.cc:3055] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:24:43.352792 17833 raft_consensus.cc:1240] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 20. Current term is 21. Ops: []
I20260709 12:24:43.355232 26368 leader_election.cc:290] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:43.356158 17652 leader_election.cc:304] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:24:43.361256 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 27 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:43.376190 26601 leader_election.cc:290] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 21 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:43.283968 26638 raft_consensus.cc:515] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:43.381716 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 30 candidate_status { last_received { term: 29 index: 22 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:43.382414 17755 raft_consensus.cc:2468] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 29.
I20260709 12:24:43.296769 26742 raft_consensus.cc:515] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:43.373824 17652 leader_election.cc:304] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
W20260709 12:24:43.394753 17728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:43.376124 26849 log.cc:927] Time spent T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.060s	user 0.001s	sys 0.000s
I20260709 12:24:43.396914 26632 raft_consensus.cc:2804] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:24:43.397086 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:43.397498 26632 raft_consensus.cc:697] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 24 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:24:43.397729 17755 raft_consensus.cc:2468] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 21.
I20260709 12:24:43.398439 26632 consensus_queue.cc:237] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 22.14, Last appended by leader: 14, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:43.399566 26796 leader_election.cc:290] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:24:43.402766 17728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:43.403570 17728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:43.403862 17652 leader_election.cc:304] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:43.428979 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:43.429842 17830 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 24.
I20260709 12:24:43.412619 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 21 candidate_status { last_received { term: 19 index: 19 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:43.433820 18361 raft_consensus.cc:3060] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:43.372960 26719 raft_consensus.cc:740] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
W20260709 12:24:43.402912 26810 consensus_meta.cc:287] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.654s	user 0.009s	sys 0.012s
I20260709 12:24:43.386616 26691 raft_consensus.cc:2804] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Leader pre-election won for term 30
W20260709 12:24:43.441875 17804 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:43.441695 26271 consensus_queue.cc:1059] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 25.15, Next index: 16, Last known committed idx: 13, Time since last communication: 0.009s
I20260709 12:24:43.425259 26519 leader_election.cc:290] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:43.427172 26638 leader_election.cc:290] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:43.383395 26816 consensus_queue.cc:1059] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 21.18, Next index: 19, Last known committed idx: 17, Time since last communication: 0.057s
I20260709 12:24:43.424465 26599 leader_election.cc:290] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:43.435079 26761 raft_consensus.cc:2804] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:24:43.464246 17762 raft_consensus.cc:1275] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 26 index: 15. Preceding OpId from leader: term: 28 index: 17. (index mismatch)
I20260709 12:24:43.464126 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 30 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:43.496447 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:43.496654 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:43.496503 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:24:43.450315 26854 log.cc:927] Time spent T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.130s	user 0.000s	sys 0.000s
I20260709 12:24:43.441428 26691 raft_consensus.cc:493] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:43.499035 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:43.500317 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:43.499243 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:43.452113 26420 raft_consensus.cc:493] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:43.504030 17762 raft_consensus.cc:2468] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 28.
I20260709 12:24:43.509572 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 26 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:43.513677 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:43.514552 17762 raft_consensus.cc:2393] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 26: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:24:43.459195 26594 raft_consensus.cc:493] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:43.515338 17832 raft_consensus.cc:2468] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 28.
I20260709 12:24:43.493763 26544 raft_consensus.cc:493] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:43.492093 26747 raft_consensus.cc:493] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:43.479012 18361 raft_consensus.cc:2468] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 21.
I20260709 12:24:43.491859 26727 raft_consensus.cc:493] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:24:43.493528 26705 raft_consensus.cc:493] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:43.457870 26835 raft_consensus.cc:493] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:43.521770 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:43.452726 26796 consensus_queue.cc:1048] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.055s
I20260709 12:24:43.525506 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:43.497609 26761 raft_consensus.cc:493] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:24:43.501282 26691 raft_consensus.cc:3060] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Advancing to term 30
W20260709 12:24:43.535708 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 96912.8us
I20260709 12:24:43.536513 17764 raft_consensus.cc:2468] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 28.
I20260709 12:24:43.513562 26843 raft_consensus.cc:2749] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:24:43.528560 17803 leader_election.cc:304] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:24:43.530007 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 26 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:43.538317 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 22 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:43.541486 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:43.542153 18361 raft_consensus.cc:2468] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 23.
I20260709 12:24:43.542152 17762 raft_consensus.cc:2393] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 26: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:24:43.545982 17837 raft_consensus.cc:2410] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for term 22 because replica has last-logged OpId of term: 20 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 13.
I20260709 12:24:43.543681 17803 leader_election.cc:304] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:24:43.536123 26799 leader_election.cc:290] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:43.540047 26761 raft_consensus.cc:3060] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:24:43.516387 26420 raft_consensus.cc:515] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:43.553236 26736 raft_consensus.cc:2804] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:24:43.555068 26824 raft_consensus.cc:2804] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:24:43.549351 26860 raft_consensus.cc:493] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:43.527419 26594 raft_consensus.cc:515] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:43.538842 26727 raft_consensus.cc:515] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:43.539587 26835 raft_consensus.cc:515] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:43.538975 26705 raft_consensus.cc:515] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:43.581094 26835 leader_election.cc:290] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:24:43.582643 26822 log.cc:927] Time spent T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 1.162s	user 0.000s	sys 0.009s
I20260709 12:24:43.582767 26705 leader_election.cc:290] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:43.529196 26719 consensus_queue.cc:260] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 23.14, Last appended by leader: 14, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:43.583976 26519 consensus_queue.cc:1175] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "e063541542a94969898868bf6eb5ce1b" responder_term: 24 status { last_received { term: 23 index: 14 } last_committed_idx: 14 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 23. Current term is 24. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260709 12:24:43.585361 26420 leader_election.cc:290] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:43.588224 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:43.583982 26719 raft_consensus.cc:3060] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:43.593720 26594 leader_election.cc:290] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:43.593829 17764 raft_consensus.cc:1240] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 27. Current term is 28. Ops: []
I20260709 12:24:43.594266 17832 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37994 (request call id 1650) took 1010 ms. Trace:
I20260709 12:24:43.594666 17832 rpcz_store.cc:276] 0709 12:24:42.584127 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:42.589122 (+  4995us) service_pool.cc:224] Handling call
0709 12:24:43.594085 (+1004963us) spinlock_profiling.cc:243] Waited 838 ms on lock 0x7b54000893d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:24:43.594205 (+   120us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1928050432}
I20260709 12:24:43.595582 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 30 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:43.529963 26544 raft_consensus.cc:515] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:43.599509 17760 raft_consensus.cc:2468] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 21.
I20260709 12:24:43.600543 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:43.601267 17760 raft_consensus.cc:2393] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 26: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:24:43.602896 17689 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37402 (request call id 3685) took 1092 ms. Trace:
I20260709 12:24:43.581101 26727 leader_election.cc:290] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:43.604971 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 26 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:43.606140 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:43.606591 17755 raft_consensus.cc:1275] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 20 index: 15. Preceding OpId from leader: term: 21 index: 16. (index mismatch)
I20260709 12:24:43.607126 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:43.609793 17755 raft_consensus.cc:1275] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 21 index: 14. Preceding OpId from leader: term: 22 index: 15. (index mismatch)
I20260709 12:24:43.609879 26688 raft_consensus.cc:3060] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:43.612200 17758 raft_consensus.cc:1240] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 26. Current term is 27. Ops: [26.19-26.19]
I20260709 12:24:43.537982 26747 raft_consensus.cc:515] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:43.614651 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 29 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:43.635244 17832 raft_consensus.cc:2468] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 28.
I20260709 12:24:43.603296 17689 rpcz_store.cc:276] 0709 12:24:42.510283 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:42.510402 (+   119us) service_pool.cc:224] Handling call
0709 12:24:42.510736 (+   334us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:43.222693 (+711957us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:43.602669 (+379976us) spinlock_profiling.cc:243] Waited 700 ms on lock 0x7b5400426dd4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:24:43.602843 (+   174us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1608971136}
I20260709 12:24:43.622629 26742 leader_election.cc:290] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:43.633678 18333 leader_election.cc:304] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:43.623545 26420 raft_consensus.cc:493] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:43.626509 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 27 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:43.629832 26800 raft_consensus.cc:3060] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:24:43.571524 26733 raft_consensus.cc:493] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:43.638617 17689 raft_consensus.cc:1275] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 27 index: 16. Preceding OpId from leader: term: 28 index: 16. (term mismatch)
I20260709 12:24:43.641286 18333 leader_election.cc:304] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:43.654719 17689 pending_rounds.cc:85] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3: Aborting all ops after (but not including) 15
I20260709 12:24:43.655033 17689 pending_rounds.cc:107] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3: Aborting uncommitted NO_OP operation due to leader change: 27.16
I20260709 12:24:43.655254 17689 raft_consensus.cc:2889] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:43.642064 18365 raft_consensus.cc:1240] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 25. Current term is 26. Ops: []
I20260709 12:24:43.642673 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:43.574995 26824 raft_consensus.cc:493] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:43.644477 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:43.657440 17682 raft_consensus.cc:3060] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:43.645579 26635 raft_consensus.cc:515] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:43.646448 26544 leader_election.cc:290] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:43.653141 26691 raft_consensus.cc:515] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:43.614074 26860 raft_consensus.cc:515] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:43.612967 26865 raft_consensus.cc:493] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:24:43.649003 26606 raft_consensus.cc:3055] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:24:43.596565 26736 raft_consensus.cc:697] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 21 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:24:43.649716 26711 raft_consensus.cc:3055] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:24:43.652585 26747 leader_election.cc:290] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:43.656327 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:43.693853 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:43.694461 17689 raft_consensus.cc:2468] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 16.
I20260709 12:24:43.658775 18364 raft_consensus.cc:2393] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 23: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:24:43.661780 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 29 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:43.667605 17727 leader_election.cc:304] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:43.688597 18371 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59886 (request call id 2392) took 1523 ms. Trace:
I20260709 12:24:43.695297 18362 raft_consensus.cc:2468] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 28.
I20260709 12:24:43.654035 26728 raft_consensus.cc:493] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:43.691850 17682 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 23.
I20260709 12:24:43.695474 18371 rpcz_store.cc:276] 0709 12:24:42.165573 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:42.165743 (+   170us) service_pool.cc:224] Handling call
0709 12:24:42.166019 (+   276us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:42.819093 (+653074us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:43.688333 (+869240us) spinlock_profiling.cc:243] Waited 639 ms on lock 0x7b54006aed54. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:24:43.688535 (+   202us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1470643072}
I20260709 12:24:43.665418 18333 leader_election.cc:304] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:24:43.688629 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 45824.9us
I20260709 12:24:43.697542 17763 raft_consensus.cc:1275] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 28 index: 17. Preceding OpId from leader: term: 29 index: 17. (term mismatch)
I20260709 12:24:43.714818 17763 pending_rounds.cc:85] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb: Aborting all ops after (but not including) 16
W20260709 12:24:43.714447 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 46911.4us
I20260709 12:24:43.715324 17763 pending_rounds.cc:107] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb: Aborting uncommitted NO_OP operation due to leader change: 28.17
I20260709 12:24:43.715803 17763 raft_consensus.cc:2889] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:43.664170 26824 raft_consensus.cc:3060] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:24:43.714215 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 48870.1us
I20260709 12:24:43.717687 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 29 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:43.719069 18333 leader_election.cc:304] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:43.667366 26691 leader_election.cc:290] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 30 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:43.720948 17729 leader_election.cc:304] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:43.667519 26705 raft_consensus.cc:2804] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Leader pre-election won for term 30
W20260709 12:24:43.722324 17653 connection.cc:256] RPC call timeout handler was delayed by -1.8292467729997952s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 1.5s for a response
I20260709 12:24:43.692337 26711 raft_consensus.cc:740] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:24:43.724448 17758 raft_consensus.cc:1240] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 23. Current term is 24. Ops: []
I20260709 12:24:43.649801 26420 raft_consensus.cc:515] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:43.667680 26860 leader_election.cc:290] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:43.664126 26733 raft_consensus.cc:515] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:43.726671 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:43.732621 17832 raft_consensus.cc:2468] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 25.
I20260709 12:24:43.670037 26606 raft_consensus.cc:740] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:24:43.727017 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:43.667613 26587 raft_consensus.cc:2749] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:24:43.734414 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:43.734175 17727 leader_election.cc:304] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:43.729439 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 21 candidate_status { last_received { term: 19 index: 19 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:43.720216 26857 raft_consensus.cc:2804] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:24:43.745769 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:43.720173 26747 raft_consensus.cc:2804] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:24:43.726284 26705 raft_consensus.cc:493] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:43.747591 17802 leader_election.cc:304] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:24:43.688063 26537 raft_consensus.cc:493] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
W20260709 12:24:43.940404 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 192882us
I20260709 12:24:43.940500 26537 raft_consensus.cc:515] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:43.941663 26848 raft_consensus.cc:2804] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:24:43.942229 26848 raft_consensus.cc:493] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:43.942656 26537 leader_election.cc:290] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:43.942699 26848 raft_consensus.cc:3060] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:43.753538 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:43.754099 17763 raft_consensus.cc:1240] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 24. Current term is 25. Ops: [22.15-24.16]
I20260709 12:24:43.944530 26705 raft_consensus.cc:3060] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:24:43.756531 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:24:43.748206 17655 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:43.945348 17759 raft_consensus.cc:2468] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 23.
W20260709 12:24:43.946584 17655 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:43.947063 17655 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:43.758193 17686 raft_consensus.cc:1240] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 24. Current term is 25. Ops: [24.19-24.19]
I20260709 12:24:43.730823 26654 raft_consensus.cc:2749] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 12:24:43.815125 17682 raft_consensus.cc:2468] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 23.
I20260709 12:24:43.718039 26728 raft_consensus.cc:515] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:43.693208 26865 raft_consensus.cc:515] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:43.814882 26544 raft_consensus.cc:3060] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:43.753244 26816 consensus_queue.cc:1059] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 27.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.067s
I20260709 12:24:43.821038 26606 consensus_queue.cc:260] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 25.15, Last appended by leader: 15, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:43.822253 17730 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:43.824049 26711 consensus_queue.cc:260] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 23.15, Last appended by leader: 15, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:43.833343 26839 log.cc:927] Time spent T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.080s	user 0.000s	sys 0.000s
I20260709 12:24:43.973140 26711 raft_consensus.cc:3060] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:43.973682 26816 raft_consensus.cc:3055] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:24:43.974197 26816 raft_consensus.cc:740] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 28 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:24:43.834264 17831 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37994 (request call id 1647) took 1263 ms. Trace:
W20260709 12:24:43.975112 26873 log.cc:927] Time spent T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.055s	user 0.000s	sys 0.000s
I20260709 12:24:43.975353 17831 rpcz_store.cc:276] 0709 12:24:42.570515 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:42.571837 (+  1322us) service_pool.cc:224] Handling call
0709 12:24:43.834089 (+1262252us) spinlock_profiling.cc:243] Waited 99.5 ms on lock 0x7b540008a7d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:24:43.834205 (+   116us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":228891904}
I20260709 12:24:43.757997 26747 raft_consensus.cc:493] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:43.976711 26747 raft_consensus.cc:3060] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:24:43.977109 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:43.978583 26420 leader_election.cc:290] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:43.978691 17685 raft_consensus.cc:1275] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 24 index: 18. Preceding OpId from leader: term: 25 index: 19. (index mismatch)
I20260709 12:24:43.814154 26844 raft_consensus.cc:2804] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:24:43.980114 26844 raft_consensus.cc:493] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:43.980516 26844 raft_consensus.cc:3060] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:43.980592 26728 leader_election.cc:290] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:43.981179 17761 raft_consensus.cc:3060] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:24:43.753499 26799 consensus_queue.cc:1048] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.091s
I20260709 12:24:43.949352 26733 leader_election.cc:290] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:24:43.852187 26864 log.cc:927] Time spent T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.158s	user 0.000s	sys 0.001s
I20260709 12:24:43.861164 26820 raft_consensus.cc:493] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:43.985956 26820 raft_consensus.cc:515] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:43.988131 26820 leader_election.cc:290] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:43.864759 26599 raft_consensus.cc:3055] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:24:43.989042 26599 raft_consensus.cc:740] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 24 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
W20260709 12:24:43.881920 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 162921us
I20260709 12:24:43.989995 26234 consensus_queue.cc:237] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 27.15, Last appended by leader: 15, Current term: 27, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:43.991313 26711 consensus_queue.cc:1048] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20260709 12:24:43.992810 26711 consensus_queue.cc:1048] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20260709 12:24:43.776032 26857 raft_consensus.cc:493] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:43.994916 26711 raft_consensus.cc:493] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:24:43.814636 26736 consensus_queue.cc:237] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 19.19, Last appended by leader: 19, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:43.995504 26711 raft_consensus.cc:515] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:43.997891 26711 leader_election.cc:290] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:43.900863 26593 raft_consensus.cc:493] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
W20260709 12:24:43.998383 17728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:43.998561 26711 raft_consensus.cc:493] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:43.998695 26593 raft_consensus.cc:515] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:24:43.898994 26635 leader_election.cc:290] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 27 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:43.919771 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:43.725211 26632 raft_consensus.cc:493] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:44.001878 26632 raft_consensus.cc:515] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:44.002398 26593 leader_election.cc:290] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:43.999606 26711 raft_consensus.cc:515] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:43.920732 17684 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f"
dest_uuid: "0da169ff82674dc1947055ce43de68a3"
mode: GRACEFUL
new_leader_uuid: "e063541542a94969898868bf6eb5ce1b"
 from {username='slave'} at 127.0.0.1:54540
I20260709 12:24:44.007771 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 29 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:44.008255 17684 raft_consensus.cc:606] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Received request to transfer leadership to TS e063541542a94969898868bf6eb5ce1b
I20260709 12:24:44.008819 17684 raft_consensus.cc:614] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:24:43.921098 18366 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 24.
I20260709 12:24:43.922784 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:44.010162 17761 raft_consensus.cc:2468] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 29.
I20260709 12:24:43.925637 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:44.012421 17687 raft_consensus.cc:2468] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 22.
I20260709 12:24:43.934623 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:43.956090 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:43.918246 26519 raft_consensus.cc:493] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:43.957119 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:43.965368 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:43.999341 17728 leader_election.cc:304] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:44.013511 17680 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 24.
I20260709 12:24:44.015807 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 24 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:24:44.017083 17652 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:44.017612 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:44.017810 17652 leader_election.cc:304] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:24:44.009192 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:44.020875 18364 raft_consensus.cc:2468] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 23.
I20260709 12:24:44.021957 17803 leader_election.cc:304] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3
W20260709 12:24:43.974421 26870 log.cc:927] Time spent T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.057s	user 0.000s	sys 0.000s
I20260709 12:24:44.027307 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:44.034090 17652 leader_election.cc:304] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:44.034160 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:44.038060 17763 raft_consensus.cc:1240] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 26. Current term is 27. Ops: []
I20260709 12:24:44.038951 18333 leader_election.cc:304] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
W20260709 12:24:44.063609 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 338306us
W20260709 12:24:43.999992 26828 log.cc:927] Time spent T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 1.401s	user 0.002s	sys 0.003s
I20260709 12:24:44.067574 17688 raft_consensus.cc:1275] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 21 index: 16. Preceding OpId from leader: term: 25 index: 18. (index mismatch)
I20260709 12:24:44.014345 26857 raft_consensus.cc:3060] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:44.072510 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:44.077349 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:44.078989 17757 raft_consensus.cc:1240] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 25. Current term is 26. Ops: []
I20260709 12:24:44.026587 26838 raft_consensus.cc:2749] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:24:44.085270 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 30 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:44.091684 17763 raft_consensus.cc:2468] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 29.
I20260709 12:24:44.092850 18333 leader_election.cc:304] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:44.093006 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:44.093936 17763 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 24.
I20260709 12:24:44.095381 17763 raft_consensus.cc:1275] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 23 index: 14. Preceding OpId from leader: term: 28 index: 16. (index mismatch)
I20260709 12:24:44.037442 26796 raft_consensus.cc:3060] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:44.069919 26640 raft_consensus.cc:2749] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:24:44.118031 17763 raft_consensus.cc:1275] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 19 index: 12. Preceding OpId from leader: term: 26 index: 13. (index mismatch)
I20260709 12:24:44.020306 26726 consensus_queue.cc:1059] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 21.18, Next index: 19, Last known committed idx: 17, Time since last communication: 0.064s
I20260709 12:24:44.125917 17760 raft_consensus.cc:1275] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 20 index: 12. Preceding OpId from leader: term: 25 index: 14. (index mismatch)
I20260709 12:24:44.025949 26519 raft_consensus.cc:515] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:44.140595 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 29 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:44.129035 17759 raft_consensus.cc:1275] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 26 index: 15. Preceding OpId from leader: term: 28 index: 16. (index mismatch)
I20260709 12:24:44.141613 17687 raft_consensus.cc:1240] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 24 LEADER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 23. Current term is 24. Ops: []
I20260709 12:24:44.141923 17684 raft_consensus.cc:2393] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 29: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:24:44.141069 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 29 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:24:44.142262 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 120381us
W20260709 12:24:44.143311 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 79355.8us
I20260709 12:24:44.142738 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:44.097929 26798 raft_consensus.cc:2804] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:24:44.155908 26798 raft_consensus.cc:493] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:44.156405 26798 raft_consensus.cc:3060] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:44.156237 17728 leader_election.cc:304] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:44.098587 26632 leader_election.cc:290] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:44.144301 17652 leader_election.cc:304] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:44.146864 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:44.147250 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 30 candidate_status { last_received { term: 29 index: 22 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:44.213371 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 23 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:44.214097 17680 raft_consensus.cc:2376] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 23. Re-sending same reply.
I20260709 12:24:44.215306 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:44.148521 26519 leader_election.cc:290] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:44.148892 17685 raft_consensus.cc:1240] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 22. Current term is 23. Ops: []
I20260709 12:24:44.149889 17686 raft_consensus.cc:1275] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 19 index: 13. Preceding OpId from leader: term: 21 index: 15. (index mismatch)
I20260709 12:24:44.150270 26593 consensus_queue.cc:1059] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 22.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20260709 12:24:44.152765 26837 consensus_queue.cc:1048] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20260709 12:24:44.154865 26544 raft_consensus.cc:2749] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:24:44.116806 26625 raft_consensus.cc:2804] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:24:44.230177 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:44.145531 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:44.148561 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 26 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:44.232136 17687 raft_consensus.cc:2468] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 25.
I20260709 12:24:44.070127 26820 consensus_queue.cc:1059] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true }, Status: INVALID_TERM, Last received: 22.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.059s
W20260709 12:24:44.225909 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 81703us
I20260709 12:24:44.140559 26799 raft_consensus.cc:493] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:44.237111 17686 raft_consensus.cc:1275] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 20 index: 16. Preceding OpId from leader: term: 21 index: 16. (term mismatch)
I20260709 12:24:44.242238 17686 pending_rounds.cc:85] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3: Aborting all ops after (but not including) 15
I20260709 12:24:44.242560 17686 pending_rounds.cc:107] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3: Aborting uncommitted NO_OP operation due to leader change: 20.16
I20260709 12:24:44.242859 17686 raft_consensus.cc:2889] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:44.155689 26848 raft_consensus.cc:515] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:24:44.247392 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 95572us
I20260709 12:24:44.248405 17728 leader_election.cc:304] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
W20260709 12:24:44.246403 17654 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:44.251047 18371 raft_consensus.cc:1240] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 26. Current term is 27. Ops: []
I20260709 12:24:44.245115 17652 leader_election.cc:304] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:44.231590 26519 raft_consensus.cc:2749] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:24:44.235913 26625 raft_consensus.cc:493] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:44.262013 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:44.249265 26714 raft_consensus.cc:2749] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
W20260709 12:24:44.267401 17801 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:44.265328 17728 leader_election.cc:304] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:44.269220 18367 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59902 (request call id 2096) took 2122 ms. Trace:
I20260709 12:24:44.274479 26750 consensus_queue.cc:1059] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 25.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.004s
I20260709 12:24:44.279093 26354 consensus_queue.cc:1048] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.018s
I20260709 12:24:44.249624 26688 consensus_queue.cc:1048] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.042s
I20260709 12:24:44.296991 26750 raft_consensus.cc:3055] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:24:44.297499 26750 raft_consensus.cc:740] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 25 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:24:44.297780 17728 leader_election.cc:304] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:44.297991 26354 raft_consensus.cc:2804] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:24:44.298436 26354 raft_consensus.cc:493] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:44.298379 26750 consensus_queue.cc:260] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 25.16, Last appended by leader: 16, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:44.299423 17728 leader_election.cc:304] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:44.281880 18364 raft_consensus.cc:1240] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 24. Current term is 25. Ops: []
W20260709 12:24:44.293931 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 273339us
I20260709 12:24:44.349157 26810 consensus_queue.cc:1048] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.000s
I20260709 12:24:44.255330 26711 leader_election.cc:290] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:44.352659 17835 raft_consensus.cc:1275] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 27 index: 15. Preceding OpId from leader: term: 27 index: 16. (index mismatch)
I20260709 12:24:44.262495 26736 raft_consensus.cc:493] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:44.299149 26354 raft_consensus.cc:3060] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:44.300086 26714 raft_consensus.cc:2749] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:24:44.283787 17755 raft_consensus.cc:1275] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 20 index: 14. Preceding OpId from leader: term: 22 index: 15. (index mismatch)
I20260709 12:24:44.256533 26632 raft_consensus.cc:2804] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:24:44.301204 26750 raft_consensus.cc:3060] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Advancing to term 26
W20260709 12:24:44.294855 26761 consensus_meta.cc:287] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.720s	user 0.011s	sys 0.000s
I20260709 12:24:44.428777 18361 raft_consensus.cc:1240] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 23. Current term is 24. Ops: []
I20260709 12:24:44.429847 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:44.430608 18361 raft_consensus.cc:2393] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 23: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
W20260709 12:24:44.305200 26880 log.cc:927] Time spent T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.076s	user 0.000s	sys 0.000s
I20260709 12:24:44.431524 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 24 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:44.364609 26733 raft_consensus.cc:493] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:44.305835 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:44.436612 18361 raft_consensus.cc:2410] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for term 24 because replica has last-logged OpId of term: 22 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 15.
I20260709 12:24:44.437003 18365 raft_consensus.cc:2410] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for term 24 because replica has last-logged OpId of term: 23 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 14.
W20260709 12:24:44.306969 26886 log.cc:927] Time spent T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.071s	user 0.000s	sys 0.001s
W20260709 12:24:44.310420 18334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:44.313194 26544 raft_consensus.cc:493] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:44.296155 18367 rpcz_store.cc:276] 0709 12:24:42.138366 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:42.138498 (+   132us) service_pool.cc:224] Handling call
0709 12:24:42.138830 (+   332us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:24:42.139189 (+   359us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:42.139218 (+    29us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:24:42.175478 (+ 36260us) log.cc:844] Serialized 52 byte log entry
0709 12:24:42.356418 (+180940us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:24:42.356457 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:42.356515 (+    58us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:43.999346 (+1642831us) spinlock_profiling.cc:243] Waited 33.1 ms on lock 0x7b4400022b4c. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b47459c85 00007f7b4745a6a2 00007f7b4a1cc1db 00007f7b4a1c7aaf 00007f7b4a1bb86e 00007f7b4a1babb6 00007f7b4a1ba7d8 00007f7b4a1c4221 00007f7b4a1c41d9 00007f7b4a1c4169
0709 12:24:44.250107 (+250761us) raft_consensus.cc:1685] finished
0709 12:24:44.250129 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:44.260992 (+ 10863us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":516,"raft.run_cpu_time_us":156,"raft.run_wall_time_us":4600,"spinlock_wait_cycles":76151168,"thread_start_us":158771,"threads_started":1,"wal-append.queue_time_us":379352}
W20260709 12:24:44.345127 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 91497.4us
I20260709 12:24:44.277485 26625 raft_consensus.cc:3060] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Advancing to term 30
W20260709 12:24:44.355031 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 55702.8us
I20260709 12:24:44.459822 17728 leader_election.cc:304] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:44.398778 26796 raft_consensus.cc:2749] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:24:44.468325 17728 leader_election.cc:304] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:44.358345 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:44.358247 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 30 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:44.470616 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:44.360031 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:44.472383 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:44.473014 17682 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 24.
I20260709 12:24:44.364145 18362 raft_consensus.cc:1275] T a5732e71b9d54f8786b8c511d69943bf P 334a4e1f59944544b3b3ff36616b57da [term 27 LEARNER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 27 index: 15. Preceding OpId from leader: term: 27 index: 16. (index mismatch)
I20260709 12:24:44.351683 26726 raft_consensus.cc:493] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:44.363896 26736 raft_consensus.cc:515] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:44.369112 17801 leader_election.cc:304] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:44.369503 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:44.293349 26874 consensus_queue.cc:1059] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.088s
W20260709 12:24:44.353538 26800 consensus_meta.cc:287] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.700s	user 0.009s	sys 0.002s
I20260709 12:24:44.366792 26632 raft_consensus.cc:493] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:44.367250 26592 raft_consensus.cc:3060] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:44.374255 26705 raft_consensus.cc:515] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:44.382567 26354 raft_consensus.cc:515] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:44.374279 26816 consensus_queue.cc:260] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 28.17, Last appended by leader: 17, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:44.387387 26868 raft_consensus.cc:3055] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:24:44.521247 26868 raft_consensus.cc:740] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:24:44.522253 26868 consensus_queue.cc:260] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 22.16, Last appended by leader: 16, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:44.392143 26865 leader_election.cc:290] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:44.320832 26691 raft_consensus.cc:2804] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Leader pre-election won for term 29
W20260709 12:24:44.397325 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:44.404558 17685 raft_consensus.cc:1275] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 19 index: 12. Preceding OpId from leader: term: 26 index: 13. (index mismatch)
I20260709 12:24:44.404955 26888 consensus_queue.cc:1059] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 26.19, Next index: 20, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:24:44.527045 26635 raft_consensus.cc:493] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
W20260709 12:24:44.405632 17655 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:44.487154 26843 raft_consensus.cc:493] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:44.406934 17758 raft_consensus.cc:1275] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 26 index: 20. Preceding OpId from leader: term: 27 index: 21. (index mismatch)
I20260709 12:24:44.410082 26747 raft_consensus.cc:515] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:44.528992 26843 raft_consensus.cc:515] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } }
I20260709 12:24:44.530956 26747 leader_election.cc:290] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 29 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:44.416080 26860 consensus_queue.cc:1048] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:24:44.481268 26581 consensus_queue.cc:1048] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.014s
I20260709 12:24:44.416724 26587 consensus_queue.cc:1048] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 12:24:44.418454 26848 leader_election.cc:290] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 17 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:44.423978 17835 raft_consensus.cc:1240] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 16. Current term is 17. Ops: []
I20260709 12:24:44.533823 26736 leader_election.cc:290] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:44.426267 26599 consensus_queue.cc:260] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 24.19, Last appended by leader: 19, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:44.427855 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 25 candidate_status { last_received { term: 21 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:44.536222 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:44.537078 26798 raft_consensus.cc:515] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:44.537683 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:44.366999 18366 raft_consensus.cc:1240] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 26. Current term is 27. Ops: []
I20260709 12:24:44.539175 26798 leader_election.cc:290] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 24 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:44.520388 26354 leader_election.cc:290] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 25 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:44.247558 26799 raft_consensus.cc:515] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:44.523723 26868 raft_consensus.cc:3060] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:44.458475 26793 consensus_queue.cc:1048] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.051s
I20260709 12:24:44.541994 26799 leader_election.cc:290] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:44.542955 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:44.518044 26705 leader_election.cc:290] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 30 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:44.543586 17833 raft_consensus.cc:3060] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:44.543720 26714 raft_consensus.cc:493] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:44.544500 26714 raft_consensus.cc:515] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:44.459546 26733 raft_consensus.cc:515] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:44.432219 26761 raft_consensus.cc:515] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:44.470999 26544 raft_consensus.cc:515] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:44.472903 26844 raft_consensus.cc:515] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:44.432940 26857 raft_consensus.cc:515] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:44.364964 26810 raft_consensus.cc:493] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:44.455293 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:44.357470 17684 raft_consensus.cc:1275] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 21 index: 14. Preceding OpId from leader: term: 22 index: 15. (index mismatch)
I20260709 12:24:44.467759 17652 leader_election.cc:304] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:44.477921 17680 raft_consensus.cc:1240] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 29. Current term is 30. Ops: []
I20260709 12:24:44.471462 17687 raft_consensus.cc:2393] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 20: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
W20260709 12:24:44.484658 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 115624us
I20260709 12:24:44.486599 17763 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "d48b7de191684318aa08f60a7a395387"
dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
mode: GRACEFUL
new_leader_uuid: "e063541542a94969898868bf6eb5ce1b"
 from {username='slave'} at 127.0.0.1:49786
I20260709 12:24:44.488585 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 21 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:44.487329 26890 raft_consensus.cc:2749] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:24:44.495730 17688 raft_consensus.cc:2410] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for term 29 because replica has last-logged OpId of term: 27 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 26 index: 15.
I20260709 12:24:44.471405 26537 raft_consensus.cc:493] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:44.471930 26742 raft_consensus.cc:2749] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 12:24:44.481256 26420 raft_consensus.cc:493] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:44.520799 26816 raft_consensus.cc:3060] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:24:44.456473 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 22 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:44.585772 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 29 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
W20260709 12:24:44.605140 26880 log.cc:927] Time spent T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.097s	user 0.000s	sys 0.001s
I20260709 12:24:44.647780 17835 raft_consensus.cc:3060] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:24:44.618779 17763 raft_consensus.cc:606] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Received request to transfer leadership to TS e063541542a94969898868bf6eb5ce1b
W20260709 12:24:44.619135 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 151451us
W20260709 12:24:44.631735 26902 log.cc:927] Time spent T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.055s	user 0.000s	sys 0.001s
W20260709 12:24:44.550633 17802 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:44.576275 26796 raft_consensus.cc:3055] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:24:44.622879 18361 raft_consensus.cc:1240] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 29. Current term is 30. Ops: []
I20260709 12:24:44.669397 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 28 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:44.670094 17829 raft_consensus.cc:2468] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 27.
I20260709 12:24:44.670248 17763 raft_consensus.cc:614] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:24:44.652036 18364 raft_consensus.cc:1240] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 21. Current term is 22. Ops: []
I20260709 12:24:44.671473 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 28 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:44.565655 17681 raft_consensus.cc:1275] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 27 index: 15. Preceding OpId from leader: term: 27 index: 16. (index mismatch)
I20260709 12:24:44.487408 26594 raft_consensus.cc:2804] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:24:44.625720 18365 raft_consensus.cc:2410] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for term 21 because replica has last-logged OpId of term: 19 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 13.
I20260709 12:24:44.482723 26837 raft_consensus.cc:2749] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:24:44.689822 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 29 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:44.691061 18361 raft_consensus.cc:3060] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:24:44.637180 17654 leader_election.cc:304] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:44.639755 17833 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 25.
I20260709 12:24:44.547223 26714 leader_election.cc:290] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:24:44.578083 26824 consensus_meta.cc:287] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.853s	user 0.011s	sys 0.000s
I20260709 12:24:44.495009 26632 raft_consensus.cc:3060] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:44.567183 26640 raft_consensus.cc:3055] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:24:44.560043 26606 raft_consensus.cc:3060] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:44.619848 17801 leader_election.cc:304] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:44.611172 26843 leader_election.cc:290] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:24:44.563045 17728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:44.489187 26726 raft_consensus.cc:515] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:44.603550 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:44.737509 17683 raft_consensus.cc:3060] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:44.643990 26537 raft_consensus.cc:515] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:44.643309 26893 log.cc:927] Time spent T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.066s	user 0.002s	sys 0.000s
I20260709 12:24:44.631930 26800 raft_consensus.cc:2749] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
I20260709 12:24:44.558362 17579 catalog_manager.cc:5697] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b reported cstate change: term changed from 12 to 16. New cstate: current_term: 16 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } }
I20260709 12:24:44.527818 26635 raft_consensus.cc:515] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:44.719113 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 48979us
I20260709 12:24:44.683022 26890 raft_consensus.cc:493] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:44.722674 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 102908us
I20260709 12:24:44.681511 26796 raft_consensus.cc:740] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:24:44.642177 26810 raft_consensus.cc:515] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:44.783366 26810 leader_election.cc:290] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:44.728010 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 30 candidate_status { last_received { term: 29 index: 22 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
W20260709 12:24:44.723448 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 86400.1us
I20260709 12:24:44.785131 26537 leader_election.cc:290] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:44.785336 17654 leader_election.cc:304] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:44.730896 17835 raft_consensus.cc:2468] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 29.
I20260709 12:24:44.653429 26420 raft_consensus.cc:515] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:44.786367 26761 leader_election.cc:290] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 28 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:44.787925 26420 leader_election.cc:290] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:24:44.641222 26875 log.cc:927] Time spent T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.413s	user 0.005s	sys 0.000s
I20260709 12:24:44.788635 26592 raft_consensus.cc:2749] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:24:44.703317 26824 raft_consensus.cc:515] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:44.790098 26726 leader_election.cc:290] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:44.791105 26824 leader_election.cc:290] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 24 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:44.720729 26640 raft_consensus.cc:740] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:24:44.762751 26844 leader_election.cc:290] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 26 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:24:44.798568 17803 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:44.767906 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:24:44.768411 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 299581us
I20260709 12:24:44.801867 18334 leader_election.cc:304] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:24:44.773131 18361 raft_consensus.cc:2468] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 29.
I20260709 12:24:44.722105 26891 raft_consensus.cc:2749] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 12:24:44.725224 26888 raft_consensus.cc:2749] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:24:44.780432 17801 leader_election.cc:304] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:44.788100 26733 leader_election.cc:290] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:44.797122 17831 raft_consensus.cc:2393] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 24: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:24:44.793599 17764 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38776 (request call id 3145) took 1302 ms. Trace:
I20260709 12:24:44.766386 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:44.853885 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:44.854449 17833 raft_consensus.cc:3060] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:24:44.804183 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:44.857180 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:44.806915 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:44.807873 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 21 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:44.860605 17829 raft_consensus.cc:2410] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for term 21 because replica has last-logged OpId of term: 19 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 13.
I20260709 12:24:44.808089 17832 raft_consensus.cc:1275] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 22 index: 14. Preceding OpId from leader: term: 24 index: 15. (index mismatch)
I20260709 12:24:44.860893 17654 leader_election.cc:304] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:44.809036 17830 raft_consensus.cc:1240] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 23. Current term is 24. Ops: []
I20260709 12:24:44.809738 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:44.861341 18334 leader_election.cc:304] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:44.808804 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:44.803417 17729 leader_election.cc:304] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:44.827698 17652 leader_election.cc:304] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:24:44.830950 17683 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 25.
W20260709 12:24:44.862454 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 59112.8us
I20260709 12:24:44.824839 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:44.863023 17834 raft_consensus.cc:1240] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 21 LEADER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 20. Current term is 21. Ops: [20.20-20.20]
I20260709 12:24:44.863371 18368 raft_consensus.cc:3060] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:44.863759 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 24 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:44.831063 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:44.828663 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 23 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:44.829839 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:44.864459 17835 raft_consensus.cc:2468] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 22.
I20260709 12:24:44.773967 26714 raft_consensus.cc:493] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:24:44.758463 26820 consensus_queue.cc:1048] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.054s
W20260709 12:24:44.845050 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 64676.2us
I20260709 12:24:44.849905 26796 consensus_queue.cc:260] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 20.20, Last appended by leader: 20, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:44.945809 17801 leader_election.cc:304] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:24:44.946087 17803 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:44.867017 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 24 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:44.853014 17764 rpcz_store.cc:276] 0709 12:24:43.491306 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:43.600649 (+109343us) service_pool.cc:224] Handling call
0709 12:24:43.600887 (+   238us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:24:43.639068 (+ 38181us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:43.639116 (+    48us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:24:43.649955 (+ 10839us) log.cc:844] Serialized 52 byte log entry
0709 12:24:44.015578 (+365623us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:24:44.015617 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:44.015681 (+    64us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:44.555279 (+539598us) spinlock_profiling.cc:243] Waited 5.11 ms on lock 0x7b4400022b4c. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b47459c85 00007f7b4745a6a2 00007f7b4a1cc1db 00007f7b4a1c7aaf 00007f7b4a1bb86e 00007f7b4a1babb6 00007f7b4a1ba7d8 00007f7b4a1c4221 00007f7b4a1c41d9 00007f7b4a1c4169
0709 12:24:44.791629 (+236350us) raft_consensus.cc:1685] finished
0709 12:24:44.791649 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:44.791862 (+   213us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":54719,"raft.queue_time_us":38201,"raft.run_cpu_time_us":111,"raft.run_wall_time_us":7039,"spinlock_wait_cycles":11751680,"thread_start_us":297631,"threads_started":1,"wal-append.queue_time_us":427109}
I20260709 12:24:44.885614 26714 raft_consensus.cc:515] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:44.881958 26761 consensus_queue.cc:1059] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 29.22, Next index: 23, Last known committed idx: 21, Time since last communication: 0.000s
I20260709 12:24:44.875982 26587 consensus_queue.cc:1048] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:24:44.814761 26868 raft_consensus.cc:2749] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:24:44.792579 26640 consensus_queue.cc:260] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 22.12, Last appended by leader: 12, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:44.886432 17833 raft_consensus.cc:2468] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 28.
I20260709 12:24:44.889587 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:44.778302 26837 consensus_queue.cc:1059] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 29.19, Next index: 20, Last known committed idx: 18, Time since last communication: 0.059s
I20260709 12:24:44.851060 26818 raft_consensus.cc:2804] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:24:44.800341 26625 raft_consensus.cc:515] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:44.894064 18368 raft_consensus.cc:2468] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 24.
I20260709 12:24:44.962103 17579 catalog_manager.cc:5697] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b reported cstate change: term changed from 16 to 17, leader changed from e063541542a94969898868bf6eb5ce1b (127.16.29.67) to e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66). New cstate: current_term: 17 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } }
I20260709 12:24:44.847736 26606 consensus_queue.cc:1048] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.048s
I20260709 12:24:44.966061 26868 consensus_queue.cc:1059] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 21.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:24:44.967185 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:44.967209 26606 raft_consensus.cc:3055] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:24:44.968755 26606 raft_consensus.cc:740] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:24:44.969628 26606 consensus_queue.cc:260] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 21.15, Last appended by leader: 15, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:44.970338 26798 raft_consensus.cc:3055] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:24:44.970909 26798 raft_consensus.cc:740] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 26 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:24:44.970969 26606 raft_consensus.cc:3060] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Advancing to term 22
W20260709 12:24:44.918111 26901 log.cc:927] Time spent T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.075s	user 0.000s	sys 0.000s
W20260709 12:24:44.919355 26905 log.cc:927] Time spent T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.052s	user 0.000s	sys 0.000s
W20260709 12:24:44.923228 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 95637.3us
I20260709 12:24:44.924079 26865 raft_consensus.cc:2804] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:24:44.924993 26519 raft_consensus.cc:493] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:44.928550 26728 consensus_queue.cc:1048] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:24:44.923687 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 24 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:44.931658 26635 leader_election.cc:290] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:44.986990 17727 leader_election.cc:304] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:44.896404 26733 raft_consensus.cc:493] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:44.892066 26688 raft_consensus.cc:2749] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:24:44.990688 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:44.862735 26910 raft_consensus.cc:2749] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:24:44.798471 26890 raft_consensus.cc:515] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:44.891613 26592 raft_consensus.cc:493] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:44.893559 26844 raft_consensus.cc:2804] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:24:44.890323 26793 raft_consensus.cc:2749] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:24:44.904819 26891 raft_consensus.cc:493] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:44.867763 18334 leader_election.cc:304] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:44.882843 26898 consensus_queue.cc:1059] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 26.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.058s
I20260709 12:24:44.947933 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:45.003577 17683 raft_consensus.cc:1240] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 25. Current term is 26. Ops: []
I20260709 12:24:45.005127 17764 raft_consensus.cc:1240] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 25. Current term is 26. Ops: []
I20260709 12:24:44.875280 17755 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06"
dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
mode: GRACEFUL
new_leader_uuid: "e063541542a94969898868bf6eb5ce1b"
 from {username='slave'} at 127.0.0.1:49786
I20260709 12:24:44.949035 17801 leader_election.cc:304] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:24:44.949079 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:44.944746 26857 leader_election.cc:290] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 22 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:44.962852 18364 raft_consensus.cc:1240] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 20. Current term is 21. Ops: []
I20260709 12:24:44.970459 18363 raft_consensus.cc:2393] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 30: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:24:44.950122 26888 raft_consensus.cc:2749] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:24:44.951661 26796 raft_consensus.cc:3060] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:44.984488 18368 raft_consensus.cc:1275] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 23 index: 15. Preceding OpId from leader: term: 24 index: 15. (term mismatch)
I20260709 12:24:44.988237 17652 leader_election.cc:304] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:24:44.978109 26818 raft_consensus.cc:697] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 23 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:24:45.001425 26742 raft_consensus.cc:2804] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:24:45.016498 26742 raft_consensus.cc:493] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:45.016892 26742 raft_consensus.cc:3060] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:24:45.016745 26818 consensus_queue.cc:237] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 22.18, Last appended by leader: 18, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:44.990240 26733 raft_consensus.cc:515] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:44.988874 26865 raft_consensus.cc:697] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 29 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:24:44.993469 17727 leader_election.cc:304] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:45.007376 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:24:45.011083 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 62123us
I20260709 12:24:45.024986 17801 leader_election.cc:304] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:24:45.013522 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 145867us
I20260709 12:24:45.013731 18368 pending_rounds.cc:85] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da: Aborting all ops after (but not including) 14
I20260709 12:24:45.025969 18368 pending_rounds.cc:107] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da: Aborting uncommitted NO_OP operation due to leader change: 23.15
I20260709 12:24:45.026240 18368 raft_consensus.cc:2889] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:44.976979 26632 raft_consensus.cc:515] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:44.954979 26878 consensus_queue.cc:1059] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 22.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.050s
W20260709 12:24:45.039232 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 45852.2us
I20260709 12:24:45.040270 18364 raft_consensus.cc:1275] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 19 index: 19. Preceding OpId from leader: term: 21 index: 20. (index mismatch)
W20260709 12:24:45.041472 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 53299.6us
I20260709 12:24:45.043044 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:44.989356 26837 consensus_queue.cc:1059] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 16.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.028s
I20260709 12:24:45.043625 17652 leader_election.cc:304] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:45.042332 17801 leader_election.cc:304] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:45.042361 17727 leader_election.cc:304] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:44.987010 26519 raft_consensus.cc:515] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:44.992259 26868 consensus_queue.cc:1048] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.044s
I20260709 12:24:45.051697 17801 leader_election.cc:304] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:44.978251 26798 consensus_queue.cc:260] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 26.19, Last appended by leader: 19, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:45.053855 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 23 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:45.000241 26271 consensus_queue.cc:1059] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 23.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.042s
I20260709 12:24:45.023679 26910 raft_consensus.cc:2804] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:24:45.015108 26844 raft_consensus.cc:697] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 29 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:24:45.046896 26824 raft_consensus.cc:2749] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:24:45.060691 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 24 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:45.062323 17801 leader_election.cc:304] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:24:45.062590 17688 raft_consensus.cc:2468] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 23.
I20260709 12:24:45.005826 26592 raft_consensus.cc:515] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:45.044770 26635 raft_consensus.cc:2804] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:24:45.047413 26581 raft_consensus.cc:2804] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:24:45.083011 26544 leader_election.cc:290] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:45.052000 26888 raft_consensus.cc:2749] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:24:45.054700 26898 raft_consensus.cc:2749] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:24:45.075263 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:45.047999 26761 raft_consensus.cc:2804] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:24:45.004966 26891 raft_consensus.cc:515] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:45.106784 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:45.107618 17682 raft_consensus.cc:3060] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:45.075966 17801 leader_election.cc:304] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:45.058917 26368 raft_consensus.cc:2749] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:24:45.056041 26843 raft_consensus.cc:2749] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:24:45.059518 26910 raft_consensus.cc:493] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:45.103607 17688 raft_consensus.cc:2468] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 26.
I20260709 12:24:45.105937 17755 raft_consensus.cc:606] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Received request to transfer leadership to TS e063541542a94969898868bf6eb5ce1b
I20260709 12:24:45.147756 17755 raft_consensus.cc:614] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260709 12:24:45.112566 26918 log.cc:927] Time spent T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.128s	user 0.000s	sys 0.001s
I20260709 12:24:45.115083 26719 raft_consensus.cc:3055] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:24:45.115276 26537 consensus_queue.cc:1059] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 26.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:24:45.077991 26635 raft_consensus.cc:493] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:45.077958 26908 consensus_queue.cc:1048] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.039s
I20260709 12:24:45.159512 17686 raft_consensus.cc:1275] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 27 index: 16. Preceding OpId from leader: term: 28 index: 16. (term mismatch)
I20260709 12:24:45.160030 17686 pending_rounds.cc:85] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3: Aborting all ops after (but not including) 15
I20260709 12:24:45.160308 17686 pending_rounds.cc:107] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3: Aborting uncommitted NO_OP operation due to leader change: 27.16
I20260709 12:24:45.160586 17686 raft_consensus.cc:2889] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:45.103094 26761 raft_consensus.cc:493] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:45.126711 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:45.128674 26824 consensus_queue.cc:1059] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 25.16, Next index: 17, Last known committed idx: 15, Time since last communication: 0.000s
I20260709 12:24:45.146880 26798 raft_consensus.cc:3060] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:45.129573 26844 consensus_queue.cc:237] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 27.19, Last appended by leader: 19, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:45.146320 26581 raft_consensus.cc:697] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 25 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:24:45.131299 26865 consensus_queue.cc:237] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 27.16, Last appended by leader: 16, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:45.132441 26810 consensus_queue.cc:1048] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.000s
I20260709 12:24:45.133086 26799 consensus_queue.cc:1048] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20260709 12:24:45.133626 26632 leader_election.cc:290] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 23 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:45.134826 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:45.134812 26742 raft_consensus.cc:515] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:45.140520 26592 leader_election.cc:290] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:45.141144 26816 raft_consensus.cc:2749] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:24:45.142869 26890 leader_election.cc:290] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:45.149479 18366 raft_consensus.cc:1240] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 25 LEADER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 24. Current term is 25. Ops: [24.16-24.16]
W20260709 12:24:45.150158 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 84898.7us
I20260709 12:24:45.119521 26733 leader_election.cc:290] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:45.190105 18334 leader_election.cc:304] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:45.146727 26910 raft_consensus.cc:3060] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:24:45.129220 26593 raft_consensus.cc:3055] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 29 LEADER]: Stepping down as leader of term 29
W20260709 12:24:45.155937 26828 log.cc:927] Time spent T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.071s	user 0.001s	sys 0.000s
I20260709 12:24:45.147998 26750 raft_consensus.cc:3055] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:24:45.157364 26719 raft_consensus.cc:740] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:24:45.197978 26719 consensus_queue.cc:260] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 26.18, Last appended by leader: 18, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:45.199976 26719 raft_consensus.cc:3060] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:45.085431 26726 raft_consensus.cc:2804] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:24:45.204428 26726 raft_consensus.cc:697] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 21 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:24:45.205760 26726 consensus_queue.cc:237] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.13, Last appended by leader: 13, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:45.164111 26635 raft_consensus.cc:3060] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:45.167850 26761 raft_consensus.cc:3060] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:45.166657 26638 raft_consensus.cc:493] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:45.165768 18367 raft_consensus.cc:3060] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:45.216439 26638 raft_consensus.cc:515] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:45.219453 26638 leader_election.cc:290] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:45.224164 26750 raft_consensus.cc:740] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 29 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:24:45.234289 26891 leader_election.cc:290] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:45.139371 17728 leader_election.cc:304] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:45.156181 17757 raft_consensus.cc:2393] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 21: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:24:45.194173 18334 leader_election.cc:304] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:24:45.190402 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 46788us
I20260709 12:24:45.307215 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
W20260709 12:24:45.360265 17655 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:45.320855 17652 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:45.319525 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:45.371565 26799 consensus_queue.cc:1059] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 25.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.012s
I20260709 12:24:45.363106 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:45.390554 26519 leader_election.cc:290] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:24:45.296587 17803 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:45.342343 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:24:45.395637 17804 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:45.397591 26742 leader_election.cc:290] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 28 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:45.194345 26593 raft_consensus.cc:740] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 29 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:24:45.305233 17831 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35040 (request call id 3500) took 1289 ms. Trace:
I20260709 12:24:45.406808 17831 rpcz_store.cc:276] 0709 12:24:44.015409 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:44.015556 (+   147us) service_pool.cc:224] Handling call
0709 12:24:45.304704 (+1289148us) spinlock_profiling.cc:243] Waited 777 ms on lock 0x7b540008b6d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:24:45.304974 (+   270us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1786727808}
I20260709 12:24:45.330317 18363 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59886 (request call id 2419) took 1070 ms. Trace:
I20260709 12:24:45.409282 18363 rpcz_store.cc:276] 0709 12:24:44.259117 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:44.261719 (+  2602us) service_pool.cc:224] Handling call
0709 12:24:45.329185 (+1067466us) spinlock_profiling.cc:243] Waited 699 ms on lock 0x7b54006ae0d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:24:45.329794 (+   609us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1608222080}
I20260709 12:24:45.412994 26640 raft_consensus.cc:3060] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:45.414656 26691 raft_consensus.cc:493] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:24:45.419503 26691 raft_consensus.cc:3060] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:24:45.308519 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:45.428684 26691 raft_consensus.cc:515] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:45.430642 26691 leader_election.cc:290] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 29 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:45.332630 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:45.379614 26594 raft_consensus.cc:493] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:24:45.433122 26594 raft_consensus.cc:3060] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Advancing to term 23
W20260709 12:24:45.317936 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 123693us
W20260709 12:24:45.291148 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 151760us
W20260709 12:24:45.307159 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 171174us
W20260709 12:24:45.295293 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 161401us
I20260709 12:24:45.440189 18374 raft_consensus.cc:1240] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 29. Current term is 30. Ops: []
I20260709 12:24:45.441188 17761 raft_consensus.cc:2468] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 23.
I20260709 12:24:45.440287 17681 raft_consensus.cc:2468] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 21.
W20260709 12:24:45.446533 17803 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:45.448042 26593 consensus_queue.cc:260] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 29.22, Last appended by leader: 22, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:45.456347 26818 consensus_queue.cc:1048] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:24:45.456071 26750 consensus_queue.cc:260] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 29.19, Last appended by leader: 19, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:45.458700 26818 consensus_queue.cc:1048] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:24:45.461395 26800 raft_consensus.cc:3055] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:24:45.461906 26800 raft_consensus.cc:740] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:24:45.463677 26622 raft_consensus.cc:2749] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:24:45.464525 26742 raft_consensus.cc:493] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:45.465415 26742 raft_consensus.cc:515] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:45.466481 18374 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 2617) took 1041 ms. Trace:
I20260709 12:24:45.466918 18374 rpcz_store.cc:276] 0709 12:24:44.425237 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:44.538865 (+113628us) service_pool.cc:224] Handling call
0709 12:24:45.365528 (+826663us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:45.464638 (+ 99110us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:45.466213 (+  1575us) spinlock_profiling.cc:243] Waited 826 ms on lock 0x7b54006ae0d0. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a61f8ad 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:24:45.466388 (+   175us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1899972864}
I20260709 12:24:45.275022 17579 catalog_manager.cc:5697] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b reported cstate change: term changed from 14 to 18. New cstate: current_term: 18 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } }
I20260709 12:24:45.468649 17763 raft_consensus.cc:2468] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 28.
I20260709 12:24:45.472776 26888 raft_consensus.cc:3055] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:24:45.472693 26733 consensus_queue.cc:1048] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:24:45.314057 17802 leader_election.cc:304] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:45.458781 26750 raft_consensus.cc:3060] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Advancing to term 30
W20260709 12:24:45.328914 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 286511us
I20260709 12:24:45.471714 17579 catalog_manager.cc:5697] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b reported cstate change: term changed from 14 to 16, leader changed from e063541542a94969898868bf6eb5ce1b (127.16.29.67) to e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66). New cstate: current_term: 16 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } }
I20260709 12:24:45.483032 26581 consensus_queue.cc:237] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 23.14, Last appended by leader: 14, Current term: 25, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:45.486712 26908 raft_consensus.cc:2749] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 12:24:45.490118 17831 raft_consensus.cc:1240] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 23. Current term is 24. Ops: [23.15-23.15]
I20260709 12:24:45.490229 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:45.491654 17833 raft_consensus.cc:2368] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 23. Current term is 24.
I20260709 12:24:45.492496 17838 raft_consensus.cc:1240] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 27. Current term is 28. Ops: []
I20260709 12:24:45.493076 17834 raft_consensus.cc:1275] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 21 index: 15. Preceding OpId from leader: term: 22 index: 15. (term mismatch)
I20260709 12:24:45.493531 17834 pending_rounds.cc:85] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b: Aborting all ops after (but not including) 14
I20260709 12:24:45.493754 17834 pending_rounds.cc:107] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b: Aborting uncommitted NO_OP operation due to leader change: 21.15
I20260709 12:24:45.493770 17835 raft_consensus.cc:1275] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 21 index: 16. Preceding OpId from leader: term: 23 index: 17. (index mismatch)
I20260709 12:24:45.494243 17834 raft_consensus.cc:2889] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:45.447634 26519 raft_consensus.cc:493] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:45.474464 26593 raft_consensus.cc:3060] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:24:45.473449 26888 raft_consensus.cc:740] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 25 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
W20260709 12:24:45.497675 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 52900.6us
I20260709 12:24:45.478623 26742 leader_election.cc:290] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:45.498698 26691 raft_consensus.cc:493] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:45.509922 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:24:45.514191 26903 log.cc:927] Time spent T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.586s	user 0.000s	sys 0.003s
I20260709 12:24:45.514905 17834 raft_consensus.cc:1275] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 20 index: 15. Preceding OpId from leader: term: 21 index: 16. (index mismatch)
I20260709 12:24:45.240703 26910 raft_consensus.cc:515] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:45.516068 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:45.517195 17837 raft_consensus.cc:1240] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 28. Current term is 29. Ops: []
I20260709 12:24:45.517297 18374 raft_consensus.cc:2393] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 23: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:24:45.518786 26910 leader_election.cc:290] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 29 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:45.495501 26519 raft_consensus.cc:515] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:45.519639 18333 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.16.29.67:36837 timed out after 1.813s (SENT) [suppressed 20 similar messages]
I20260709 12:24:45.521373 17729 leader_election.cc:304] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:45.521852 26519 leader_election.cc:290] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:24:45.522164 26911 log.cc:927] Time spent T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.606s	user 0.000s	sys 0.001s
W20260709 12:24:45.517313 26904 log.cc:927] Time spent T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.593s	user 0.001s	sys 0.001s
I20260709 12:24:45.523646 26354 raft_consensus.cc:3060] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:24:45.525655 17756 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36572 (request call id 1583) took 1455 ms. Trace:
I20260709 12:24:45.526144 17756 rpcz_store.cc:276] 0709 12:24:44.075702 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:44.225734 (+150032us) service_pool.cc:224] Handling call
0709 12:24:44.243865 (+ 18131us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:44.262032 (+ 18167us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:44.262071 (+    39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:44.277028 (+ 14957us) log.cc:844] Serialized 26 byte log entry
0709 12:24:44.661532 (+384504us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:24:44.663205 (+  1673us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:44.663319 (+   114us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:45.513136 (+849817us) spinlock_profiling.cc:243] Waited 7.06 ms on lock 0x7b4400022b4c. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b47459c85 00007f7b4745a6a2 00007f7b4a1cc1db 00007f7b4a1c7aaf 00007f7b4a1bb86e 00007f7b4a1babb6 00007f7b4a1ba7d8 00007f7b4a1c4221 00007f7b4a1c41d9 00007f7b4a1c4169
0709 12:24:45.525125 (+ 11989us) raft_consensus.cc:1685] finished
0709 12:24:45.525148 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:45.525457 (+   309us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":17741,"raft.queue_time_us":17849,"raft.run_cpu_time_us":146,"raft.run_wall_time_us":10401,"spinlock_wait_cycles":16236288,"thread_start_us":340886,"threads_started":1,"wal-append.queue_time_us":498745}
I20260709 12:24:45.528287 26929 raft_consensus.cc:2749] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
W20260709 12:24:45.524108 26913 log.cc:927] Time spent T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.595s	user 0.002s	sys 0.001s
I20260709 12:24:45.531563 26599 raft_consensus.cc:3060] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:45.536213 17579 catalog_manager.cc:5697] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b reported cstate change: term changed from 12 to 13, leader changed from 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) to e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66). New cstate: current_term: 13 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
W20260709 12:24:45.541020 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 228052us
I20260709 12:24:45.567806 17689 raft_consensus.cc:1240] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 29 LEADER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 28. Current term is 29. Ops: [28.17-28.17]
I20260709 12:24:45.581900 26594 raft_consensus.cc:515] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:45.584235 17655 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:45.586366 26843 raft_consensus.cc:3055] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:24:45.586880 26843 raft_consensus.cc:740] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 26 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:24:45.589651 17836 raft_consensus.cc:1275] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 19 index: 12. Preceding OpId from leader: term: 26 index: 13. (index mismatch)
I20260709 12:24:45.562785 26691 raft_consensus.cc:515] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:45.591741 17579 catalog_manager.cc:5697] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b reported cstate change: term changed from 11 to 14. New cstate: current_term: 14 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:24:45.604612 17802 leader_election.cc:304] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:45.596679 16501 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260709 12:24:45.596010 26691 leader_election.cc:290] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:45.615710 26594 leader_election.cc:290] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 23 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:45.596859 26581 raft_consensus.cc:493] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:45.616044 16501 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:24:45.616815 26581 raft_consensus.cc:515] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:45.622013 26581 leader_election.cc:290] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:45.627370 26625 leader_election.cc:290] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 30 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:45.628002 26714 leader_election.cc:290] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:45.628048 16501 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:24:45.632105 17803 leader_election.cc:304] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:24:45.634821 26810 raft_consensus.cc:2804] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:24:45.635612 26810 raft_consensus.cc:697] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 24 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:24:45.636137 26888 consensus_queue.cc:260] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 25.16, Last appended by leader: 16, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:45.637158 26810 consensus_queue.cc:237] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 22.16, Last appended by leader: 16, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:45.637985 26888 raft_consensus.cc:3060] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:45.639246 26929 consensus_queue.cc:1048] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:24:45.642628 26638 consensus_queue.cc:1048] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:24:45.653591 18365 raft_consensus.cc:1240] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 29. Current term is 30. Ops: []
W20260709 12:24:45.654919 17578 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35962 (request call id 360) took 51802 ms (client timeout 15000 ms). Trace:
W20260709 12:24:45.655640 17682 consensus_meta.cc:287] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.547s	user 0.015s	sys 0.009s
I20260709 12:24:45.656188 17682 raft_consensus.cc:2468] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 24.
I20260709 12:24:45.635541 18367 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 25.
I20260709 12:24:45.658227 18367 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59886 (request call id 2430) took 1022 ms. Trace:
I20260709 12:24:45.658638 18367 rpcz_store.cc:276] 0709 12:24:44.635558 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:44.687776 (+ 52218us) service_pool.cc:224] Handling call
0709 12:24:45.126520 (+438744us) spinlock_profiling.cc:243] Waited 1.73 ms on lock 0x7b5800102398. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b474fbb71 00007f7b474fba22 00007f7b504148d9 00007f7b4f56eee0 00007f7b4f6380ab 00007f7b4f638131 00007f7b4f5e60a0 00007f7b4f5e5984 00007f7b48b606d4 00007f7b48b60662
0709 12:24:45.658154 (+531634us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3967744}
I20260709 12:24:45.660164 26924 raft_consensus.cc:2764] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Leader pre-election decision vote started in defunct term 24: won
I20260709 12:24:45.660765 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:45.661528 18363 raft_consensus.cc:2468] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 20.
I20260709 12:24:45.663139 17834 raft_consensus.cc:1240] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 22. Current term is 23. Ops: []
I20260709 12:24:45.664359 17681 raft_consensus.cc:1275] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 24 index: 16. Preceding OpId from leader: term: 25 index: 16. (term mismatch)
I20260709 12:24:45.664826 17681 pending_rounds.cc:85] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3: Aborting all ops after (but not including) 15
I20260709 12:24:45.665078 17681 pending_rounds.cc:107] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3: Aborting uncommitted NO_OP operation due to leader change: 24.16
I20260709 12:24:45.665211 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:45.664283 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:45.665359 17681 raft_consensus.cc:2889] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:45.662360 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:45.678328 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:45.679687 17689 raft_consensus.cc:2468] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 23.
I20260709 12:24:45.680907 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:45.696862 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 30 candidate_status { last_received { term: 29 index: 22 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:45.674407 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:45.594244 17579 catalog_manager.cc:5697] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b reported cstate change: term changed from 12 to 16, leader changed from e063541542a94969898868bf6eb5ce1b (127.16.29.67) to e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66). New cstate: current_term: 16 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } }
I20260709 12:24:45.710507 17579 catalog_manager.cc:5697] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b reported cstate change: term changed from 10 to 13, leader changed from e063541542a94969898868bf6eb5ce1b (127.16.29.67) to 0da169ff82674dc1947055ce43de68a3 (127.16.29.65). New cstate: current_term: 13 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:24:45.712350 17835 raft_consensus.cc:1275] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 26 index: 20. Preceding OpId from leader: term: 27 index: 21. (index mismatch)
W20260709 12:24:45.717762 26761 consensus_meta.cc:287] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.505s	user 0.008s	sys 0.011s
I20260709 12:24:45.718412 26761 raft_consensus.cc:515] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:45.719616 17837 raft_consensus.cc:2393] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 25: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:24:45.720052 17685 raft_consensus.cc:2368] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 23. Current term is 24.
I20260709 12:24:45.720458 17833 raft_consensus.cc:2468] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 20.
I20260709 12:24:45.717931 17838 raft_consensus.cc:2393] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 24: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:24:45.727090 17682 raft_consensus.cc:2468] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 30.
W20260709 12:24:45.741451 26796 consensus_meta.cc:287] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.707s	user 0.012s	sys 0.006s
I20260709 12:24:45.741927 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:24:45.745033 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 66602.1us
I20260709 12:24:45.599264 26800 consensus_queue.cc:260] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 23.17, Last appended by leader: 17, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:45.761425 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:45.763906 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 27 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:45.681732 26843 consensus_queue.cc:260] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 26.15, Last appended by leader: 15, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:45.762667 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 28 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:45.746526 26638 raft_consensus.cc:493] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:24:45.768149 26789 raft_consensus.cc:2804] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:24:45.770264 17830 raft_consensus.cc:2468] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 21.
W20260709 12:24:45.775508 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 171069us
I20260709 12:24:45.776444 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:45.768280 26625 raft_consensus.cc:493] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:45.784703 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:24:45.786773 18333 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:45.787669 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:45.793936 17689 raft_consensus.cc:2410] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for term 27 because replica has last-logged OpId of term: 25 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 16.
I20260709 12:24:45.792210 17682 raft_consensus.cc:2468] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 21.
I20260709 12:24:45.780064 26883 raft_consensus.cc:3055] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:24:45.797869 26587 raft_consensus.cc:493] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:45.655597 17578 rpcz_store.cc:269] 0709 12:23:53.849313 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:53.856736 (+  7423us) service_pool.cc:224] Handling call
0709 12:24:45.615855 (+51759119us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:24:35.278335 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:24:35.295589 (+ 17254us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:24:35.295731 (+   142us) write_op.cc:435] Acquired schema lock
0709 12:24:35.295773 (+    42us) tablet.cc:662] Decoding operations
0709 12:24:35.313087 (+ 17314us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:24:35.313386 (+   299us) write_op.cc:641] Partition lock acquired for write op
0709 12:24:35.313432 (+    46us) tablet.cc:685] Acquiring locks for 9 operations
0709 12:24:35.315422 (+  1990us) tablet.cc:701] Row locks acquired
0709 12:24:35.315482 (+    60us) write_op.cc:260] PREPARE: finished
0709 12:24:35.315786 (+   304us) write_op.cc:270] Start()
0709 12:24:35.341781 (+ 25995us) write_op.cc:276] Timestamp: P: 1783599875315638 usec, L: 0
0709 12:24:35.345409 (+  3628us) op_driver.cc:348] REPLICATION: starting
0709 12:24:35.522980 (+177571us) log.cc:844] Serialized 3599 byte log entry
0709 12:24:36.139568 (+616588us) spinlock_profiling.cc:243] Waited 7.8 ms on lock 0x7b18002ede00. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b475333f3 00007f7b475357c1 00007f7b47535779 00007f7b47535709 00007f7b475356d1 00007f7b475349cd 0000000000388634 0000000000388479 00007f7b4751683a
0709 12:24:36.197446 (+ 57878us) spinlock_profiling.cc:243] Waited 5.28 ms on lock 0x7b4400022b4c. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b47459c85 00007f7b4745a6a2 00007f7b4a1cc1db 00007f7b4a1c7aaf 00007f7b4a1bb86e 00007f7b4a1babb6 00007f7b4a1ba7d8 00007f7b4a1c4221 00007f7b4a1c41d9 00007f7b4a1c4169
0709 12:24:36.628280 (+430834us) op_driver.cc:464] REPLICATION: finished
0709 12:24:37.259648 (+631368us) write_op.cc:301] APPLY: starting
0709 12:24:37.259745 (+    97us) tablet.cc:1367] starting BulkCheckPresence
0709 12:24:37.331798 (+ 72053us) tablet.cc:1370] finished BulkCheckPresence
0709 12:24:37.331841 (+    43us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:24:37.426266 (+ 94425us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:24:37.465524 (+ 39258us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=18,key_file_lookups=18,delta_file_lookups=0,mrs_lookups=0
0709 12:24:37.465607 (+    83us) write_op.cc:312] APPLY: finished
0709 12:24:37.465996 (+   389us) log.cc:844] Serialized 89 byte log entry
0709 12:24:37.466184 (+   188us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:24:37.466330 (+   146us) write_op.cc:454] Released schema lock
0709 12:24:37.466399 (+    69us) write_op.cc:341] FINISH: Updating metrics
0709 12:24:37.466860 (+   461us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:24:37.466903 (+    43us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":371713,"reported_tablets":45,"child_traces":[["op",{"apply.queue_time_us":630971,"apply.run_cpu_time_us":15207,"apply.run_wall_time_us":207194,"cfile_cache_hit":38,"cfile_cache_hit_bytes":41783,"cfile_init":1,"lbm_read_time_us":2754,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":3,"num_ops":9,"prepare.queue_time_us":245291,"prepare.run_cpu_time_us":30158,"prepare.run_wall_time_us":561099,"raft.queue_time_us":101227,"raft.run_cpu_time_us":145428,"raft.run_wall_time_us":507196,"replication_time_us":1281136,"spinlock_wait_cycles":30062720,"thread_start_us":962530,"threads_started":4,"wal-append.queue_time_us":295759,"wal-append.run_cpu_time_us":26594,"wal-append.run_wall_time_us":2518041}]]}
I20260709 12:24:45.800040 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 30 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
W20260709 12:24:45.800282 17578 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35958 (request call id 416) took 17361 ms (client timeout 15000 ms). Trace:
W20260709 12:24:45.800710 17578 rpcz_store.cc:269] 0709 12:24:28.438888 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:45.800041 (+17361153us) service_pool.cc:211] Skipping call since client already timed out
0709 12:24:45.800204 (+   163us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260709 12:24:45.801262 26638 raft_consensus.cc:515] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:45.798317 26625 raft_consensus.cc:515] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:45.798466 26587 raft_consensus.cc:515] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:45.779410 26789 raft_consensus.cc:493] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:45.791656 26354 raft_consensus.cc:2749] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:24:45.811743 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:45.817734 17680 raft_consensus.cc:3060] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:45.818794 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:45.819506 18368 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 23.
I20260709 12:24:45.820381 17683 raft_consensus.cc:1275] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 20 index: 20. Preceding OpId from leader: term: 21 index: 20. (term mismatch)
I20260709 12:24:45.812628 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:45.820768 17683 pending_rounds.cc:85] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3: Aborting all ops after (but not including) 19
I20260709 12:24:45.821077 17683 pending_rounds.cc:107] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3: Aborting uncommitted NO_OP operation due to leader change: 20.20
I20260709 12:24:45.814688 26837 raft_consensus.cc:493] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:45.814777 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:45.822850 18361 raft_consensus.cc:2468] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 25.
I20260709 12:24:45.795775 17652 leader_election.cc:304] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:45.821313 17683 raft_consensus.cc:2889] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:45.824033 26742 raft_consensus.cc:493] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:45.829526 17838 raft_consensus.cc:2393] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 26: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:24:45.821874 26837 raft_consensus.cc:515] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:45.816105 26789 raft_consensus.cc:3060] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:45.833027 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:45.839774 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 30 candidate_status { last_received { term: 29 index: 22 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:45.803869 26883 raft_consensus.cc:740] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:24:45.832809 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 30 candidate_status { last_received { term: 29 index: 22 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:45.837247 26796 raft_consensus.cc:2804] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Leader election won for term 28
W20260709 12:24:45.837816 26635 consensus_meta.cc:287] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.629s	user 0.013s	sys 0.002s
I20260709 12:24:45.834779 17802 leader_election.cc:304] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:24:45.824282 26625 leader_election.cc:290] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:45.844022 17682 raft_consensus.cc:2393] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 30: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:24:45.845872 18362 raft_consensus.cc:2393] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 30: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:24:45.840905 26640 raft_consensus.cc:493] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:45.854586 26796 raft_consensus.cc:697] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 28 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:24:45.886713 26742 raft_consensus.cc:515] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:45.850236 18334 leader_election.cc:304] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:24:45.896569 17803 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:45.855451 26843 raft_consensus.cc:3060] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:45.869935 26638 leader_election.cc:290] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:24:45.905805 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 134120us
I20260709 12:24:45.857806 26761 leader_election.cc:290] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 24 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:45.909432 26741 raft_consensus.cc:493] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:24:45.898516 26592 raft_consensus.cc:493] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:45.857059 26635 raft_consensus.cc:515] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:45.904547 26929 raft_consensus.cc:2804] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Leader pre-election won for term 27
W20260709 12:24:45.920921 17803 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:45.919642 26420 raft_consensus.cc:2804] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:24:45.924906 26420 raft_consensus.cc:493] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:45.924659 26883 consensus_queue.cc:260] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.10, Last appended by leader: 10, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:45.925384 26420 raft_consensus.cc:3060] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:45.925837 26883 raft_consensus.cc:3060] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 16 FOLLOWER]: Advancing to term 17
W20260709 12:24:45.928611 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 133079us
W20260709 12:24:45.929054 26798 consensus_meta.cc:287] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.732s	user 0.009s	sys 0.005s
I20260709 12:24:45.929337 17755 raft_consensus.cc:1240] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 28. Current term is 29. Ops: [27.16-28.17]
I20260709 12:24:45.929477 17729 leader_election.cc:304] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:24:45.913129 26606 consensus_meta.cc:287] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.941s	user 0.013s	sys 0.013s
I20260709 12:24:45.910126 26741 raft_consensus.cc:515] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:45.935523 17680 raft_consensus.cc:2468] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 23.
I20260709 12:24:45.937109 18363 raft_consensus.cc:1240] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 25. Current term is 26. Ops: []
I20260709 12:24:45.938320 17652 leader_election.cc:304] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:45.915737 17579 catalog_manager.cc:5697] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b reported cstate change: term changed from 12 to 15. New cstate: current_term: 15 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:24:45.939545 26761 raft_consensus.cc:2749] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:24:45.906512 26910 consensus_queue.cc:1048] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.015s
I20260709 12:24:45.942610 26761 consensus_queue.cc:1059] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 28.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20260709 12:24:45.914429 26640 raft_consensus.cc:515] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:45.945699 26761 raft_consensus.cc:3055] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:24:45.945952 26592 raft_consensus.cc:515] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:45.947119 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:45.948141 26592 leader_election.cc:290] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:45.948508 26581 raft_consensus.cc:2804] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:24:45.949002 26581 raft_consensus.cc:493] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:45.949435 26581 raft_consensus.cc:3060] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:45.949657 26741 leader_election.cc:290] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:45.952515 26719 consensus_queue.cc:1059] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:24:45.937433 17802 leader_election.cc:304] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:45.953926 17652 leader_election.cc:304] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:45.954385 26865 raft_consensus.cc:3055] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:24:45.932816 26796 consensus_queue.cc:237] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 27.17, Last appended by leader: 17, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:45.954927 26865 raft_consensus.cc:740] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 29 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:24:45.950052 17654 leader_election.cc:304] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:24:45.956130 26865 consensus_queue.cc:260] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 29.17, Last appended by leader: 17, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:45.957991 26865 raft_consensus.cc:3060] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:24:45.959065 26726 raft_consensus.cc:2804] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:24:45.956743 26719 raft_consensus.cc:2804] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:24:45.959513 26726 raft_consensus.cc:493] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:24:45.960304 26726 raft_consensus.cc:3060] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Advancing to term 22
W20260709 12:24:45.960747 17655 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:45.961143 26940 log.cc:927] Time spent T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.372s	user 0.005s	sys 0.000s
I20260709 12:24:45.959719 26719 raft_consensus.cc:493] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:45.961913 26719 raft_consensus.cc:3060] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:45.969749 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:45.971043 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:45.972924 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:45.975107 26587 leader_election.cc:290] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:45.975268 26866 raft_consensus.cc:493] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:24:45.976126 26866 raft_consensus.cc:515] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:45.946111 26761 raft_consensus.cc:740] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
W20260709 12:24:45.989916 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 150866us
I20260709 12:24:45.921589 26929 raft_consensus.cc:493] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:24:45.994364 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 49459.3us
I20260709 12:24:45.993724 17728 leader_election.cc:304] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:45.920312 26837 leader_election.cc:290] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:45.969964 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:46.005116 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 24 candidate_status { last_received { term: 23 index: 19 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:46.005918 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 24 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:46.003173 17729 leader_election.cc:304] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:46.008359 18334 leader_election.cc:304] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:45.947160 26640 leader_election.cc:290] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:24:46.015949 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 62597us
I20260709 12:24:46.016556 17728 leader_election.cc:304] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:46.003340 26929 raft_consensus.cc:3060] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:46.021973 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:46.018489 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:46.022338 17681 raft_consensus.cc:1240] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 22. Current term is 23. Ops: [22.15-22.15]
I20260709 12:24:46.024581 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:46.008899 26910 raft_consensus.cc:493] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:46.027738 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:46.029743 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:46.030570 17833 raft_consensus.cc:2393] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 24: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:24:46.037034 17729 leader_election.cc:304] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:46.008852 26587 consensus_queue.cc:1048] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.020s
W20260709 12:24:46.044317 17803 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:46.036118 26910 raft_consensus.cc:515] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:46.050537 26727 raft_consensus.cc:2804] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:24:46.050956 26727 raft_consensus.cc:493] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:46.051312 26727 raft_consensus.cc:3060] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:46.052416 26910 leader_election.cc:290] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:46.045789 17756 raft_consensus.cc:2468] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 26.
I20260709 12:24:46.029791 26837 raft_consensus.cc:2749] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:24:46.034765 26592 raft_consensus.cc:2804] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:24:46.063928 17830 raft_consensus.cc:2468] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 25.
I20260709 12:24:46.070873 17652 leader_election.cc:304] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:46.072088 26835 raft_consensus.cc:2804] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:24:46.072325 17760 raft_consensus.cc:1240] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 25. Current term is 26. Ops: []
I20260709 12:24:46.072726 26835 raft_consensus.cc:697] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 30 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:24:46.073089 26868 raft_consensus.cc:2749] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:24:46.073832 26835 consensus_queue.cc:237] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 29.22, Last appended by leader: 22, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:46.077931 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 75862.3us
I20260709 12:24:46.080365 26818 consensus_queue.cc:1059] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 21.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
W20260709 12:24:46.091770 26944 log.cc:927] Time spent T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.415s	user 0.003s	sys 0.000s
I20260709 12:24:46.077955 26761 consensus_queue.cc:260] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 28.16, Last appended by leader: 16, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:46.097630 26761 raft_consensus.cc:3060] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:24:46.104099 26742 leader_election.cc:290] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:46.106216 17683 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37402 (request call id 3716) took 1099 ms. Trace:
I20260709 12:24:46.109175 17683 rpcz_store.cc:276] 0709 12:24:45.006606 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:45.006766 (+   160us) service_pool.cc:224] Handling call
0709 12:24:45.007055 (+   289us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:45.842850 (+835795us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:46.105805 (+262955us) spinlock_profiling.cc:243] Waited 750 ms on lock 0x7b54004254d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:24:46.106121 (+   316us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1724620672}
I20260709 12:24:46.122145 26632 raft_consensus.cc:2749] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:24:46.123107 17687 raft_consensus.cc:1240] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 28 LEADER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 27. Current term is 28. Ops: []
I20260709 12:24:46.128119 26924 consensus_queue.cc:1059] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 27.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:24:45.942828 17579 catalog_manager.cc:5697] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b reported cstate change: term changed from 10 to 12, leader changed from 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) to e063541542a94969898868bf6eb5ce1b (127.16.29.67). New cstate: current_term: 12 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: HEALTHY } } }
I20260709 12:24:46.131338 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:46.133175 17579 catalog_manager.cc:5697] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b reported cstate change: term changed from 15 to 17. New cstate: current_term: 17 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } }
W20260709 12:24:46.140667 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 94113us
W20260709 12:24:46.143035 17654 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:46.146142 17684 raft_consensus.cc:2468] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 22.
I20260709 12:24:46.069592 26592 raft_consensus.cc:697] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:24:46.149343 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:46.150020 17685 raft_consensus.cc:2468] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 23.
I20260709 12:24:46.149814 26592 consensus_queue.cc:237] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 22.15, Last appended by leader: 15, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:46.150981 26750 raft_consensus.cc:3060] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:24:46.152199 26635 leader_election.cc:290] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 26 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:46.152915 26635 consensus_queue.cc:1048] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20260709 12:24:46.154346 26635 consensus_queue.cc:1048] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20260709 12:24:46.147331 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:46.186991 17652 leader_election.cc:304] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:46.145948 26866 leader_election.cc:290] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:46.181957 26742 raft_consensus.cc:493] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:46.200541 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
W20260709 12:24:46.206126 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 164971us
I20260709 12:24:46.206936 17801 leader_election.cc:304] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:46.208318 17762 raft_consensus.cc:1240] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 26. Current term is 27. Ops: []
W20260709 12:24:46.200872 26599 consensus_meta.cc:287] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.669s	user 0.012s	sys 0.004s
W20260709 12:24:46.233666 17804 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:46.242110 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 205102us
I20260709 12:24:46.209414 26798 raft_consensus.cc:493] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:46.260888 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:46.212258 26632 raft_consensus.cc:2749] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:24:46.278496 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 30 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:46.279909 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:46.293766 26848 raft_consensus.cc:2749] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:24:46.293288 26594 raft_consensus.cc:2955] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 27 LEADER]: Committing config change with OpId 27.16: config changed from index 12 to 16, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:24:46.296706 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 27 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:46.296975 17684 raft_consensus.cc:1275] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 23 index: 14. Preceding OpId from leader: term: 25 index: 15. (index mismatch)
I20260709 12:24:46.297560 26798 raft_consensus.cc:515] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:46.302263 26800 raft_consensus.cc:3060] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:46.304010 26723 raft_consensus.cc:493] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:46.304589 26723 raft_consensus.cc:515] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:46.308748 26949 log.cc:927] Time spent T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.059s	user 0.000s	sys 0.000s
W20260709 12:24:46.310360 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 123461us
I20260709 12:24:46.313481 26866 consensus_queue.cc:1059] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 25.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:24:46.315057 26866 raft_consensus.cc:493] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:46.315748 26866 raft_consensus.cc:515] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:46.315809 26593 raft_consensus.cc:3055] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:24:46.325327 26593 raft_consensus.cc:740] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:24:46.326498 26593 consensus_queue.cc:260] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 25.14, Last appended by leader: 14, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:46.320413 26866 leader_election.cc:290] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:24:46.328203 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:46.319676 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 112832us
I20260709 12:24:46.331239 17801 leader_election.cc:304] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:24:46.332479 17730 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:46.334810 26810 raft_consensus.cc:2749] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:24:46.335983 26798 leader_election.cc:290] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:46.337831 17801 leader_election.cc:304] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:46.339649 26910 raft_consensus.cc:2749] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:24:46.343240 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:46.344303 26719 raft_consensus.cc:515] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:46.344465 17756 raft_consensus.cc:2468] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 29.
I20260709 12:24:46.345515 17680 raft_consensus.cc:2368] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 26. Current term is 27.
I20260709 12:24:46.346688 17801 leader_election.cc:304] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:46.347534 26593 raft_consensus.cc:3060] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:46.348191 26910 raft_consensus.cc:2804] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:24:46.348760 26910 raft_consensus.cc:493] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:46.349192 26910 raft_consensus.cc:3060] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:24:46.287108 17830 raft_consensus.cc:2468] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 21.
I20260709 12:24:46.312994 17652 leader_election.cc:304] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:46.287947 17683 raft_consensus.cc:2468] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 27.
I20260709 12:24:46.248616 26742 raft_consensus.cc:515] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:46.276882 26640 consensus_queue.cc:1048] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.097s
I20260709 12:24:46.366431 26632 raft_consensus.cc:2804] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:24:46.347764 26789 raft_consensus.cc:515] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:46.378297 17762 raft_consensus.cc:1240] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 27. Current term is 28. Ops: [27.21-27.21]
I20260709 12:24:46.392335 26632 raft_consensus.cc:697] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 23 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:24:46.395977 17760 raft_consensus.cc:3060] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:24:46.372792 26714 raft_consensus.cc:493] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
W20260709 12:24:46.404101 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 90850.2us
I20260709 12:24:46.404876 17652 leader_election.cc:304] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:24:46.393844 26632 consensus_queue.cc:237] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 22.17, Last appended by leader: 17, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:46.418186 26887 raft_consensus.cc:493] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:46.418282 26714 raft_consensus.cc:515] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:46.419299 26887 raft_consensus.cc:515] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:46.419862 26742 leader_election.cc:290] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:46.423259 26741 raft_consensus.cc:493] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:46.423930 26741 raft_consensus.cc:515] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:46.426133 26741 leader_election.cc:290] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:46.436787 26719 leader_election.cc:290] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 27 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:24:46.449654 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 44852.6us
I20260709 12:24:46.453989 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:46.454288 26599 raft_consensus.cc:493] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:46.454649 18365 raft_consensus.cc:3060] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Advancing to term 27
I20260709 12:24:46.454918 26599 raft_consensus.cc:515] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:46.410449 26810 raft_consensus.cc:493] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:46.468405 26810 raft_consensus.cc:515] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:46.471125 26810 leader_election.cc:290] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:46.477445 26789 leader_election.cc:290] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 24 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:46.450723 17652 leader_election.cc:304] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:46.494560 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 28 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:46.495260 17829 raft_consensus.cc:3060] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:24:46.496006 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:46.497552 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 28 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:46.497820 17835 raft_consensus.cc:1240] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 16. Current term is 17. Ops: []
I20260709 12:24:46.498453 17834 raft_consensus.cc:2410] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for term 28 because replica has last-logged OpId of term: 26 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 18.
I20260709 12:24:46.498818 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:46.499586 17835 raft_consensus.cc:2393] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 29: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:24:46.499866 17832 raft_consensus.cc:1275] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 23 index: 14. Preceding OpId from leader: term: 25 index: 15. (index mismatch)
I20260709 12:24:46.501235 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:46.502487 17836 raft_consensus.cc:1275] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 27 index: 19. Preceding OpId from leader: term: 29 index: 20. (index mismatch)
I20260709 12:24:46.502507 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:46.504822 17837 raft_consensus.cc:2468] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 25.
I20260709 12:24:46.506186 26789 raft_consensus.cc:493] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:46.514991 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:46.515146 26599 leader_election.cc:290] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:24:46.516111 26420 consensus_meta.cc:287] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.590s	user 0.011s	sys 0.001s
I20260709 12:24:46.516685 26420 raft_consensus.cc:515] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:46.517686 26714 leader_election.cc:290] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:46.413058 26929 raft_consensus.cc:515] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:46.528730 17762 raft_consensus.cc:2468] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 26.
I20260709 12:24:46.545351 17833 raft_consensus.cc:2393] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 27: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
W20260709 12:24:46.547539 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 425033us
I20260709 12:24:46.551611 17680 raft_consensus.cc:1275] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 20 index: 13. Preceding OpId from leader: term: 21 index: 14. (index mismatch)
W20260709 12:24:46.553154 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:46.518567 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:46.518807 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:46.557205 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:46.558367 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 26 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:46.559768 17683 raft_consensus.cc:1275] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 21 index: 15. Preceding OpId from leader: term: 24 index: 17. (index mismatch)
W20260709 12:24:46.524977 26950 log.cc:927] Time spent T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.162s	user 0.000s	sys 0.001s
W20260709 12:24:46.524557 26952 log.cc:927] Time spent T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.249s	user 0.001s	sys 0.000s
I20260709 12:24:46.555764 17764 raft_consensus.cc:2468] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 23.
I20260709 12:24:46.564069 17686 raft_consensus.cc:2393] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 27: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:24:46.567066 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:46.525914 26929 leader_election.cc:290] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 27 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:46.519650 26420 leader_election.cc:290] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 24 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:46.567497 17689 raft_consensus.cc:1240] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 28. Current term is 29. Ops: []
I20260709 12:24:46.554711 26719 raft_consensus.cc:2749] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:24:46.506752 26789 raft_consensus.cc:515] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:46.581694 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:46.520876 26887 leader_election.cc:290] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:46.564993 17728 leader_election.cc:304] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:24:46.555275 26843 consensus_meta.cc:287] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.650s	user 0.010s	sys 0.001s
I20260709 12:24:46.588402 17684 raft_consensus.cc:2468] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 25.
I20260709 12:24:46.585443 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:24:46.584586 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 133982us
I20260709 12:24:46.585580 18334 leader_election.cc:304] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:24:46.564389 26625 raft_consensus.cc:493] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:24:46.526546 26910 raft_consensus.cc:515] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:46.575824 26866 raft_consensus.cc:2804] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Leader pre-election won for term 29
W20260709 12:24:46.587831 26581 consensus_meta.cc:287] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.638s	user 0.009s	sys 0.004s
I20260709 12:24:46.577826 26723 leader_election.cc:290] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:46.577658 26924 raft_consensus.cc:493] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:24:46.642758 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 28 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:46.653693 18334 leader_election.cc:304] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:46.646248 17802 leader_election.cc:400] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 23 pre-election: Vote denied by peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071) with higher term. Message: Invalid argument: T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 23. Current term is 24.
I20260709 12:24:46.583956 26837 raft_consensus.cc:493] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:24:46.650058 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:24:46.593763 26727 consensus_meta.cc:287] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.542s	user 0.013s	sys 0.003s
I20260709 12:24:46.654418 17802 leader_election.cc:403] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 23 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:24:46.662209 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:46.667243 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:46.667835 17683 raft_consensus.cc:3060] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:46.654742 17755 raft_consensus.cc:2468] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 17.
I20260709 12:24:46.666476 18366 raft_consensus.cc:2468] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 26.
I20260709 12:24:46.671358 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 24 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:46.662601 26625 raft_consensus.cc:515] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:46.694064 26625 leader_election.cc:290] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:46.580384 26742 consensus_queue.cc:1059] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 25.18, Next index: 19, Last known committed idx: 17, Time since last communication: 0.073s
W20260709 12:24:46.646338 26818 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.562s	user 0.019s	sys 0.046s
I20260709 12:24:46.708240 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:46.653250 26750 raft_consensus.cc:2749] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:24:46.715864 17760 raft_consensus.cc:2468] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 29.
I20260709 12:24:46.719079 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:46.654734 26592 raft_consensus.cc:493] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:46.722867 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:24:46.730980 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 157344us
I20260709 12:24:46.735064 17836 raft_consensus.cc:1240] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 23. Current term is 24. Ops: []
I20260709 12:24:46.736078 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:46.737625 18367 raft_consensus.cc:2410] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for term 28 because replica has last-logged OpId of term: 26 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 18.
I20260709 12:24:46.676295 26741 raft_consensus.cc:493] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:46.740571 17689 raft_consensus.cc:1240] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 23. Current term is 24. Ops: []
I20260709 12:24:46.720055 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:46.743083 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 27 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:46.732375 17835 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 23.
W20260709 12:24:46.721418 26865 consensus_meta.cc:287] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.763s	user 0.016s	sys 0.000s
I20260709 12:24:46.753937 18363 raft_consensus.cc:1275] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 27 index: 19. Preceding OpId from leader: term: 29 index: 20. (index mismatch)
W20260709 12:24:46.709839 26818 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.629s	user 0.019s	sys 0.049s
I20260709 12:24:46.713270 26723 raft_consensus.cc:3060] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Advancing to term 24
I20260709 12:24:46.656904 26581 raft_consensus.cc:515] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:46.771539 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:46.772265 26843 consensus_queue.cc:1048] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
W20260709 12:24:46.754252 26951 log.cc:927] Time spent T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.379s	user 0.001s	sys 0.001s
I20260709 12:24:46.670645 26727 raft_consensus.cc:515] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:46.712332 26761 raft_consensus.cc:493] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:46.787384 18366 raft_consensus.cc:2393] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 27: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
W20260709 12:24:46.792064 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 198122us
I20260709 12:24:46.760836 26956 raft_consensus.cc:2804] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Leader pre-election won for term 27
W20260709 12:24:46.807336 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 72891.9us
I20260709 12:24:46.786360 17838 raft_consensus.cc:2468] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 26.
I20260709 12:24:46.761790 26592 raft_consensus.cc:515] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:46.694993 26837 raft_consensus.cc:515] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:46.762765 26883 raft_consensus.cc:493] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:46.782486 26888 raft_consensus.cc:493] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:46.834829 18334 leader_election.cc:400] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 26 pre-election: Vote denied by peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071) with higher term. Message: Invalid argument: T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 26. Current term is 27.
I20260709 12:24:46.837591 17758 raft_consensus.cc:1240] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 LEADER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 22. Current term is 23. Ops: []
I20260709 12:24:46.837689 18334 leader_election.cc:403] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 26 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:24:46.832646 17802 leader_election.cc:304] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:24:46.786830 26750 raft_consensus.cc:493] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:46.700222 26924 raft_consensus.cc:515] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:46.862291 17829 raft_consensus.cc:2468] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 28.
W20260709 12:24:46.844175 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:46.863359 17681 raft_consensus.cc:2468] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 29.
I20260709 12:24:46.864281 18365 raft_consensus.cc:2468] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 27.
I20260709 12:24:46.871484 26800 raft_consensus.cc:493] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:46.789048 26955 consensus_queue.cc:1048] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.291s
I20260709 12:24:46.872143 26800 raft_consensus.cc:515] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:46.873263 26711 raft_consensus.cc:2804] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:24:46.873780 26711 raft_consensus.cc:493] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:46.874593 26599 raft_consensus.cc:2804] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:24:46.811576 26966 raft_consensus.cc:493] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:46.875084 26599 raft_consensus.cc:493] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:46.875696 26599 raft_consensus.cc:3060] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:46.879072 26924 leader_election.cc:290] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:46.879261 26866 raft_consensus.cc:493] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:24:46.880040 18361 raft_consensus.cc:1275] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 23 index: 14. Preceding OpId from leader: term: 25 index: 15. (index mismatch)
I20260709 12:24:46.881160 26970 raft_consensus.cc:493] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:46.881767 26970 raft_consensus.cc:515] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:46.882570 26635 raft_consensus.cc:3055] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:24:46.883040 26635 raft_consensus.cc:740] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:24:46.874435 26800 leader_election.cc:290] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:24:46.877188 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 63741.6us
I20260709 12:24:46.885622 26587 raft_consensus.cc:3055] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:24:46.886054 26587 raft_consensus.cc:740] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:24:46.889875 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 25 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:46.891001 17686 raft_consensus.cc:2393] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 25: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:24:46.874606 26711 raft_consensus.cc:3060] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:24:46.895435 26964 log.cc:927] Time spent T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.225s	user 0.003s	sys 0.000s
I20260709 12:24:46.896205 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 24 candidate_status { last_received { term: 23 index: 19 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:24:46.904567 26958 log.cc:927] Time spent T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.358s	user 0.001s	sys 0.000s
I20260709 12:24:46.874769 26962 raft_consensus.cc:2804] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:24:46.907397 26962 raft_consensus.cc:697] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 22 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:24:46.908455 26962 consensus_queue.cc:237] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 21.15, Last appended by leader: 15, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:46.837457 26844 raft_consensus.cc:493] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:24:46.912199 26844 raft_consensus.cc:515] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:46.838301 26726 consensus_meta.cc:287] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.877s	user 0.017s	sys 0.002s
I20260709 12:24:46.914074 26726 raft_consensus.cc:515] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:46.915624 26711 raft_consensus.cc:515] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:46.916267 26726 leader_election.cc:290] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 22 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:46.917038 26726 raft_consensus.cc:493] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:46.840967 26883 raft_consensus.cc:515] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:24:46.918463 26594 consensus_meta.cc:287] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.623s	user 0.012s	sys 0.001s
I20260709 12:24:46.917887 26726 raft_consensus.cc:515] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:46.920517 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 27 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:46.921525 26726 leader_election.cc:290] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:46.921963 26910 leader_election.cc:290] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 30 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:24:46.877069 26960 log.cc:927] Time spent T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.156s	user 0.000s	sys 0.000s
I20260709 12:24:46.911111 26599 raft_consensus.cc:515] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:46.924353 26789 leader_election.cc:290] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:46.875567 26966 raft_consensus.cc:515] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:46.867391 18333 leader_election.cc:304] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:46.934571 17757 raft_consensus.cc:1240] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 21. Current term is 22. Ops: []
I20260709 12:24:46.937589 26965 consensus_queue.cc:1059] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.003s
I20260709 12:24:46.862679 26963 consensus_queue.cc:1048] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.351s
I20260709 12:24:46.829452 26956 raft_consensus.cc:493] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
W20260709 12:24:46.940160 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 196549us
I20260709 12:24:46.942066 17683 raft_consensus.cc:2468] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 24.
I20260709 12:24:46.941828 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 22 candidate_status { last_received { term: 19 index: 19 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:46.879762 26866 raft_consensus.cc:3060] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:24:46.835223 26887 consensus_queue.cc:1048] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 14, Time since last communication: 0.106s
I20260709 12:24:46.861629 26865 raft_consensus.cc:493] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:24:46.940677 26965 raft_consensus.cc:3055] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:24:46.834300 26810 consensus_queue.cc:1048] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.107s
I20260709 12:24:46.829901 26741 raft_consensus.cc:515] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:46.879773 26924 raft_consensus.cc:2749] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Leader pre-election lost for term 26. Reason: Vote denied by peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071) with higher term. Message: Invalid argument: T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 26. Current term is 27.
I20260709 12:24:46.861891 26888 raft_consensus.cc:515] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:46.828912 26961 consensus_queue.cc:1059] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 27.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.351s
I20260709 12:24:46.949344 18334 leader_election.cc:304] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:24:46.839257 26848 consensus_queue.cc:1059] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 28.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.109s
I20260709 12:24:46.837448 26625 consensus_queue.cc:1048] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.110s
I20260709 12:24:46.877339 26761 raft_consensus.cc:515] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:46.955096 26835 raft_consensus.cc:2749] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:24:46.941337 26956 raft_consensus.cc:3060] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:46.967216 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 24 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:46.967444 17683 raft_consensus.cc:1275] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 21 index: 14. Preceding OpId from leader: term: 23 index: 16. (index mismatch)
W20260709 12:24:46.964946 17730 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:46.940649 26750 raft_consensus.cc:515] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:47.017567 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:47.017863 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:47.018430 17760 raft_consensus.cc:2393] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 22: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:24:46.963807 26965 raft_consensus.cc:740] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:24:46.971371 26957 raft_consensus.cc:493] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:24:46.990368 26640 raft_consensus.cc:493] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:47.048043 26640 raft_consensus.cc:515] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:47.050410 26640 leader_election.cc:290] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:24:47.050732 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 209866us
I20260709 12:24:47.052500 26750 leader_election.cc:290] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:47.052403 26592 leader_election.cc:290] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:47.053028 18333 leader_election.cc:304] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:24:47.053731 17763 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38786 (request call id 3951) took 1037 ms. Trace:
I20260709 12:24:47.054163 17763 rpcz_store.cc:276] 0709 12:24:46.020561 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:46.024114 (+  3553us) service_pool.cc:224] Handling call
0709 12:24:46.040401 (+ 16287us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:46.212102 (+171701us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:46.212132 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:46.243777 (+ 31645us) log.cc:844] Serialized 26 byte log entry
0709 12:24:46.675864 (+432087us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:24:46.675911 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:46.675973 (+    62us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:47.053485 (+377512us) raft_consensus.cc:1685] finished
0709 12:24:47.053508 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:47.053666 (+   158us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":56441,"raft.queue_time_us":3404,"raft.run_cpu_time_us":190,"raft.run_wall_time_us":17557,"thread_start_us":484237,"threads_started":2,"wal-append.queue_time_us":694288}
I20260709 12:24:47.045747 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:47.045933 17764 raft_consensus.cc:1275] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 26 index: 15. Preceding OpId from leader: term: 29 index: 17. (index mismatch)
I20260709 12:24:47.046454 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 24 candidate_status { last_received { term: 23 index: 19 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:47.058269 26883 leader_election.cc:290] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:47.062321 26966 leader_election.cc:290] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:47.063203 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:47.063938 17758 raft_consensus.cc:3060] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:47.064656 26581 leader_election.cc:290] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 22 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:47.009660 26961 consensus_queue.cc:1243] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer 0da169ff82674dc1947055ce43de68a3 log is divergent from this leader: its last log entry 28.18 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 17
I20260709 12:24:47.067584 26965 consensus_queue.cc:260] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 23.19, Last appended by leader: 19, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:47.068831 26837 leader_election.cc:290] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:47.069211 26965 raft_consensus.cc:3060] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:47.069631 26844 leader_election.cc:290] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:47.044965 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:47.079058 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 31 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:47.079898 17760 raft_consensus.cc:2468] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 30.
I20260709 12:24:47.039868 26420 raft_consensus.cc:2804] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:24:46.973802 26865 raft_consensus.cc:515] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:47.087615 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:47.091238 26714 raft_consensus.cc:3055] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:24:47.092489 26723 raft_consensus.cc:2749] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Leader pre-election lost for term 23. Reason: Vote denied by peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071) with higher term. Message: Invalid argument: T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 23. Current term is 24.
I20260709 12:24:47.093000 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:47.103371 17764 raft_consensus.cc:2468] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 26.
I20260709 12:24:47.043152 26955 raft_consensus.cc:493] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:47.094409 26888 leader_election.cc:290] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:47.094902 26635 consensus_queue.cc:260] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 27.17, Last appended by leader: 17, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:47.082947 26957 raft_consensus.cc:515] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:47.103557 17759 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 22.
I20260709 12:24:47.116858 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:24:47.119058 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:47.132695 17758 raft_consensus.cc:2468] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 27.
I20260709 12:24:47.102351 26625 raft_consensus.cc:2804] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:24:47.109508 26714 raft_consensus.cc:740] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:24:47.083391 26800 consensus_queue.cc:1048] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.060s
I20260709 12:24:47.115314 26599 leader_election.cc:290] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 26 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:24:47.157974 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 105482us
I20260709 12:24:47.116926 26420 raft_consensus.cc:697] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 24 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:24:47.144547 18364 raft_consensus.cc:2955] T a5732e71b9d54f8786b8c511d69943bf P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Committing config change with OpId 27.16: config changed from index 12 to 16, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
W20260709 12:24:47.189553 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 240312us
W20260709 12:24:47.186296 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 353954us
I20260709 12:24:47.192587 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:47.194916 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:47.196105 26865 leader_election.cc:290] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:47.194940 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:47.201725 17830 raft_consensus.cc:2468] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 24.
I20260709 12:24:47.201875 18374 raft_consensus.cc:3060] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Advancing to term 22
I20260709 12:24:47.158445 26955 raft_consensus.cc:515] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:47.206100 26955 leader_election.cc:290] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:47.206604 26727 leader_election.cc:290] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 22 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:47.157583 26625 raft_consensus.cc:493] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:24:47.209200 26714 consensus_queue.cc:260] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 25.18, Last appended by leader: 18, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:47.212731 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 31 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:47.213553 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 22 candidate_status { last_received { term: 19 index: 19 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:47.213689 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:47.214514 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:47.215126 17802 leader_election.cc:304] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:24:47.215490 26420 consensus_queue.cc:237] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 22.16, Last appended by leader: 16, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:47.218856 26963 raft_consensus.cc:493] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:24:47.223387 18374 raft_consensus.cc:2468] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 22.
I20260709 12:24:47.224212 26810 raft_consensus.cc:493] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:47.224582 18333 leader_election.cc:304] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:47.224725 26599 raft_consensus.cc:493] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:24:47.230216 17652 leader_election.cc:304] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:24:47.246434 17838 raft_consensus.cc:3060] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:47.245975 17837 raft_consensus.cc:2468] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 26.
W20260709 12:24:47.261263 18334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:47.284420 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 54295us
W20260709 12:24:47.284802 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 69735.4us
I20260709 12:24:47.255872 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:47.284528 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:47.285771 17652 leader_election.cc:304] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:24:47.296242 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 71789.2us
I20260709 12:24:47.227974 26888 raft_consensus.cc:493] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:47.238781 26711 leader_election.cc:290] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 24 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:47.289535 26865 raft_consensus.cc:2749] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260709 12:24:47.239404 26625 raft_consensus.cc:3060] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:47.306238 17761 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38786 (request call id 3952) took 1280 ms. Trace:
I20260709 12:24:47.285248 26961 raft_consensus.cc:2764] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Leader pre-election decision vote started in defunct term 20: won
I20260709 12:24:47.245440 26955 raft_consensus.cc:493] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:24:47.313100 17652 leader_election.cc:304] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:47.308781 18333 leader_election.cc:304] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:47.317085 18374 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 23.
I20260709 12:24:47.289673 26800 raft_consensus.cc:2804] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:24:47.320760 17801 leader_election.cc:304] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
W20260709 12:24:47.248042 17729 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:47.343237 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 306234us
I20260709 12:24:47.306825 26714 raft_consensus.cc:3060] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:47.306491 26635 raft_consensus.cc:3060] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:24:47.311312 26848 raft_consensus.cc:2804] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:24:47.354374 17729 leader_election.cc:304] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:47.374236 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 31 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:47.374974 17683 raft_consensus.cc:2468] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 30.
I20260709 12:24:47.376187 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
W20260709 12:24:47.377885 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 64874.9us
I20260709 12:24:47.339843 26800 raft_consensus.cc:493] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:47.379937 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:47.387338 17686 raft_consensus.cc:2468] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 26.
I20260709 12:24:47.287009 26963 raft_consensus.cc:515] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:47.400622 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:47.401692 17682 raft_consensus.cc:2393] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 22: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:24:47.405380 26866 raft_consensus.cc:515] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:24:47.408027 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 53747.1us
I20260709 12:24:47.308637 26599 raft_consensus.cc:515] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:47.364920 26593 raft_consensus.cc:3060] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:47.345495 26910 raft_consensus.cc:493] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:47.345388 26887 raft_consensus.cc:493] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:47.385596 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:47.386299 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:47.360082 26966 raft_consensus.cc:2749] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:24:47.415109 26985 raft_consensus.cc:493] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:47.344707 26955 raft_consensus.cc:515] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:47.380836 26865 raft_consensus.cc:493] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:47.417289 26581 raft_consensus.cc:2804] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:24:47.412315 17728 leader_election.cc:304] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:47.420091 26723 raft_consensus.cc:2804] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:24:47.420576 26977 consensus_queue.cc:1048] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.001s
I20260709 12:24:47.311137 26810 raft_consensus.cc:515] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:47.365420 26727 consensus_queue.cc:1048] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.066s
I20260709 12:24:47.345685 26888 raft_consensus.cc:515] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:47.132164 26924 raft_consensus.cc:2955] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [term 26 LEADER]: Committing config change with OpId 19.12: config changed from index 5 to 12, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:24:47.313771 17761 rpcz_store.cc:276] 0709 12:24:46.028457 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:46.028599 (+   142us) service_pool.cc:224] Handling call
0709 12:24:46.062054 (+ 33455us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:24:46.201069 (+139015us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:46.201100 (+    31us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:24:46.211777 (+ 10677us) log.cc:844] Serialized 52 byte log entry
0709 12:24:46.567705 (+355928us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:24:46.988157 (+420452us) log.cc:844] Serialized 14 byte log entry
0709 12:24:47.006357 (+ 18200us) log.cc:844] Serialized 14 byte log entry
0709 12:24:47.305892 (+299535us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:47.305979 (+    87us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:47.306025 (+    46us) raft_consensus.cc:1685] finished
0709 12:24:47.306036 (+    11us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:47.306180 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":66881,"prepare.queue_time_us":386135,"prepare.run_cpu_time_us":74,"prepare.run_wall_time_us":75,"raft.queue_time_us":178278,"raft.run_cpu_time_us":292,"raft.run_wall_time_us":1611,"thread_start_us":920813,"threads_started":3,"wal-append.queue_time_us":512636}
W20260709 12:24:47.582073 17730 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:47.582796 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 285609us
W20260709 12:24:47.583865 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 205017us
I20260709 12:24:47.584728 17689 raft_consensus.cc:2393] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 24: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:24:47.391639 26800 raft_consensus.cc:3060] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:47.585080 17652 leader_election.cc:304] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:47.586073 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 26 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:47.430334 17838 raft_consensus.cc:2468] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 27.
I20260709 12:24:47.589807 18365 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 25.
I20260709 12:24:47.591521 17652 leader_election.cc:304] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:47.583825 17654 leader_election.cc:304] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
W20260709 12:24:47.617673 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 205436us
I20260709 12:24:47.615557 17729 leader_election.cc:304] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:47.619154 18333 leader_election.cc:304] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:47.618494 17728 leader_election.cc:304] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:47.439726 26789 raft_consensus.cc:3055] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:24:47.622195 17729 leader_election.cc:304] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:47.433319 26843 raft_consensus.cc:493] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:47.464344 26741 leader_election.cc:290] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:47.615899 26723 raft_consensus.cc:697] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 17 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:24:47.621151 17833 raft_consensus.cc:2468] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 29.
I20260709 12:24:47.438472 26970 leader_election.cc:290] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:47.517235 26726 raft_consensus.cc:493] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:47.548348 26632 raft_consensus.cc:493] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:47.587517 26581 raft_consensus.cc:493] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:47.645087 17728 leader_election.cc:304] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:47.424962 26848 raft_consensus.cc:697] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 29 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:24:47.434970 26963 leader_election.cc:290] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:47.609521 26993 raft_consensus.cc:2749] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:24:47.437773 26599 leader_election.cc:290] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:24:47.623585 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 337864us
I20260709 12:24:47.632478 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:47.634332 18364 raft_consensus.cc:1275] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 27 index: 17. Preceding OpId from leader: term: 28 index: 18. (index mismatch)
I20260709 12:24:47.611188 26761 leader_election.cc:290] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:47.620601 26719 raft_consensus.cc:493] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:47.618221 26990 raft_consensus.cc:493] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:47.459601 26887 raft_consensus.cc:515] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:47.440410 26910 raft_consensus.cc:515] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:47.629240 26888 leader_election.cc:290] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:47.635533 26741 raft_consensus.cc:2804] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:24:47.466528 26956 raft_consensus.cc:515] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:47.677994 26727 raft_consensus.cc:493] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:47.699381 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 391475us
I20260709 12:24:47.678161 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:47.678161 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:47.678669 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:47.700722 17761 raft_consensus.cc:3060] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:24:47.679369 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 30 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:47.605929 26985 raft_consensus.cc:515] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:47.679991 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:47.680466 26837 raft_consensus.cc:3060] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Advancing to term 22
W20260709 12:24:47.681870 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 68906.9us
I20260709 12:24:47.682729 26848 consensus_queue.cc:237] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 28.16, Last appended by leader: 16, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:47.684688 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:47.623979 26980 raft_consensus.cc:2749] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
W20260709 12:24:47.685063 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 101375us
I20260709 12:24:47.686627 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:47.632503 26978 raft_consensus.cc:2804] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:24:47.644086 26844 raft_consensus.cc:2804] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:24:47.630379 26810 leader_election.cc:290] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:47.638155 26789 raft_consensus.cc:740] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 28 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:24:47.627933 26957 leader_election.cc:290] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:47.684258 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:47.650331 26638 raft_consensus.cc:2804] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:24:47.616380 26865 raft_consensus.cc:515] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:47.636529 27001 raft_consensus.cc:2749] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
W20260709 12:24:47.759330 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 144959us
I20260709 12:24:47.707228 18333 leader_election.cc:304] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
W20260709 12:24:47.694017 26965 consensus_meta.cc:287] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.624s	user 0.008s	sys 0.004s
I20260709 12:24:47.637122 26979 consensus_queue.cc:1048] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 12, Time since last communication: 0.071s
I20260709 12:24:47.707451 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:47.513557 26587 consensus_queue.cc:260] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 22.15, Last appended by leader: 15, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:47.636399 26723 consensus_queue.cc:237] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.10, Last appended by leader: 10, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:47.685856 26961 raft_consensus.cc:2749] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:24:47.685549 26866 leader_election.cc:290] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 29 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:47.716262 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:24:47.797175 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 148126us
I20260709 12:24:47.797561 17829 raft_consensus.cc:2468] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 26.
I20260709 12:24:47.718283 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:47.720659 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 26 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:47.720846 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:47.721909 26985 leader_election.cc:290] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:47.717237 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 31 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:47.722172 27004 consensus_queue.cc:1059] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.012s
I20260709 12:24:47.722887 26727 raft_consensus.cc:515] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:47.812520 16501 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:24:47.814565 26727 leader_election.cc:290] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:47.723721 17801 leader_election.cc:304] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:47.723923 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 27 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:47.724287 17830 raft_consensus.cc:2955] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Committing config change with OpId 27.16: config changed from index 12 to 16, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:24:47.816187 26761 raft_consensus.cc:493] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:24:47.816849 26761 raft_consensus.cc:515] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:47.819032 26761 leader_election.cc:290] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:47.819545 26761 raft_consensus.cc:493] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:24:47.635178 26843 raft_consensus.cc:515] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:47.703121 26593 raft_consensus.cc:2749] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:24:47.667918 26726 raft_consensus.cc:515] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:47.757892 17761 raft_consensus.cc:2468] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 30.
I20260709 12:24:47.668448 26581 raft_consensus.cc:3060] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:47.708472 26741 raft_consensus.cc:697] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 27 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:24:47.711827 26978 raft_consensus.cc:493] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:47.854022 26978 raft_consensus.cc:3060] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Advancing to term 31
W20260709 12:24:47.778868 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 75731.8us
I20260709 12:24:47.855871 26843 leader_election.cc:290] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:24:47.781237 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 130734us
I20260709 12:24:47.856724 18333 leader_election.cc:304] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:24:47.857717 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:47.705413 26632 raft_consensus.cc:515] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:47.858350 17833 raft_consensus.cc:2468] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 21.
I20260709 12:24:47.859282 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:47.744508 26844 raft_consensus.cc:697] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:24:47.782208 17727 leader_election.cc:304] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:47.718302 26719 raft_consensus.cc:515] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:47.798787 17834 raft_consensus.cc:2468] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 24.
I20260709 12:24:47.806486 17837 raft_consensus.cc:2393] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 23: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:24:47.797988 17652 leader_election.cc:304] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:24:47.772914 26980 raft_consensus.cc:493] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
W20260709 12:24:47.862143 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 152696us
I20260709 12:24:47.779786 26818 raft_consensus.cc:3060] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:47.810166 26587 raft_consensus.cc:3060] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:47.728717 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:47.825184 27011 raft_consensus.cc:493] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:47.825971 26789 consensus_queue.cc:260] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 28.16, Last appended by leader: 16, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:47.787470 26638 raft_consensus.cc:493] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:47.828166 17684 raft_consensus.cc:1240] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 28. Current term is 29. Ops: []
I20260709 12:24:47.837795 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:24:47.808458 27000 log.cc:927] Time spent T a5732e71b9d54f8786b8c511d69943bf P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.093s	user 0.000s	sys 0.000s
W20260709 12:24:47.849314 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 125680us
I20260709 12:24:47.854085 26985 raft_consensus.cc:3055] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 21 LEADER]: Stepping down as leader of term 21
W20260709 12:24:47.856108 27009 log.cc:927] Time spent T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.078s	user 0.000s	sys 0.001s
I20260709 12:24:47.856873 26888 raft_consensus.cc:2804] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:24:47.779817 17835 raft_consensus.cc:3060] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:47.963672 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 21 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:47.696023 26990 raft_consensus.cc:515] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:47.859820 17833 raft_consensus.cc:2468] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 27.
I20260709 12:24:47.799057 17682 raft_consensus.cc:3060] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:47.861992 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:47.967850 17756 raft_consensus.cc:3060] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:47.856756 17836 raft_consensus.cc:2468] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 26.
W20260709 12:24:47.865041 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 83931.6us
I20260709 12:24:47.860013 26632 leader_election.cc:290] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:47.921156 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:47.925513 17764 raft_consensus.cc:1275] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 29 index: 22. Preceding OpId from leader: term: 30 index: 23. (index mismatch)
I20260709 12:24:47.926086 17680 raft_consensus.cc:3060] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:47.923395 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 30 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:24:47.932945 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 135033us
I20260709 12:24:47.931427 17681 raft_consensus.cc:2468] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 22.
I20260709 12:24:47.938951 17728 leader_election.cc:304] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:47.939553 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:47.937702 26980 raft_consensus.cc:515] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:47.957842 26638 raft_consensus.cc:3060] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:47.959228 27012 consensus_queue.cc:1048] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.065s
I20260709 12:24:47.979542 26742 consensus_queue.cc:1048] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.026s
I20260709 12:24:48.006640 26865 leader_election.cc:290] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:48.007370 26888 raft_consensus.cc:697] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 27 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:24:48.004268 17801 leader_election.cc:304] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:24:48.012164 17652 leader_election.cc:304] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:47.970122 17727 leader_election.cc:304] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
W20260709 12:24:47.981550 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:48.014766 18333 leader_election.cc:304] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:24:48.015208 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 45182.5us
I20260709 12:24:47.961606 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:47.920076 26593 raft_consensus.cc:2804] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Leader pre-election won for term 27
W20260709 12:24:48.022395 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 83533.2us
I20260709 12:24:47.964569 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:47.921120 27005 raft_consensus.cc:493] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:47.845840 26761 raft_consensus.cc:515] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:48.023857 18365 raft_consensus.cc:2468] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 22.
I20260709 12:24:47.932063 26991 raft_consensus.cc:2749] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:24:47.991652 17835 raft_consensus.cc:2468] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 22.
I20260709 12:24:47.940038 26957 raft_consensus.cc:493] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:24:47.992213 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:47.956810 17689 raft_consensus.cc:2393] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 30: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:24:47.961437 26789 raft_consensus.cc:3060] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Advancing to term 29
W20260709 12:24:47.984665 27002 log.cc:927] Time spent T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.117s	user 0.000s	sys 0.000s
I20260709 12:24:48.115907 17835 raft_consensus.cc:1240] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 23. Current term is 24. Ops: [23.17-23.17]
I20260709 12:24:48.079933 17756 raft_consensus.cc:2468] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 24.
I20260709 12:24:48.118044 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:48.027102 17687 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37396 (request call id 3306) took 1084 ms. Trace:
I20260709 12:24:48.026522 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:47.959707 26985 raft_consensus.cc:740] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 21 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
W20260709 12:24:48.110306 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 83860.3us
I20260709 12:24:48.026631 17652 leader_election.cc:304] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:48.119521 17763 raft_consensus.cc:3060] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:24:47.965142 26625 consensus_meta.cc:287] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.612s	user 0.007s	sys 0.006s
I20260709 12:24:48.118453 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 30 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:48.127935 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:48.128120 17835 raft_consensus.cc:2468] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 29.
I20260709 12:24:48.063700 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 24 candidate_status { last_received { term: 23 index: 19 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:48.082429 17654 leader_election.cc:304] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:24:47.979099 26994 raft_consensus.cc:2804] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:24:48.127188 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:48.129796 17760 raft_consensus.cc:2468] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 24.
I20260709 12:24:48.130367 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:47.981163 26971 raft_consensus.cc:493] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:48.130729 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 25 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:48.136704 17727 leader_election.cc:304] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:47.925202 27011 raft_consensus.cc:515] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:48.159520 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 132971us
I20260709 12:24:47.981349 26962 raft_consensus.cc:493] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:24:47.985256 26726 leader_election.cc:290] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:48.004783 26837 raft_consensus.cc:2749] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:24:48.137775 17833 raft_consensus.cc:2368] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 25. Current term is 26.
I20260709 12:24:48.022544 26979 raft_consensus.cc:2749] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:24:48.022377 26593 raft_consensus.cc:493] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:48.022109 26963 raft_consensus.cc:493] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:48.162307 17756 raft_consensus.cc:2468] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 27.
I20260709 12:24:48.129559 26994 raft_consensus.cc:493] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:24:48.162828 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 80503.3us
I20260709 12:24:48.160856 17682 raft_consensus.cc:2468] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 26.
I20260709 12:24:48.161008 17835 raft_consensus.cc:1240] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 27. Current term is 28. Ops: [27.21-27.21]
I20260709 12:24:47.989354 26719 leader_election.cc:290] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:47.991607 26990 leader_election.cc:290] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:48.109068 26865 raft_consensus.cc:493] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:48.275454 17755 raft_consensus.cc:1275] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 21 index: 15. Preceding OpId from leader: term: 22 index: 16. (index mismatch)
I20260709 12:24:48.116930 26982 raft_consensus.cc:493] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:48.188941 17680 raft_consensus.cc:2468] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 27.
I20260709 12:24:48.173381 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 27 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:24:48.213711 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 199025us
I20260709 12:24:48.256760 17757 raft_consensus.cc:1240] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 21. Current term is 22. Ops: [21.14-21.14]
I20260709 12:24:48.287096 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:48.175830 17801 leader_election.cc:304] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:48.118836 17687 rpcz_store.cc:276] 0709 12:24:46.942276 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:46.942397 (+   121us) service_pool.cc:224] Handling call
0709 12:24:46.947519 (+  5122us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:47.313351 (+365832us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:47.313382 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:47.354326 (+ 40944us) log.cc:844] Serialized 26 byte log entry
0709 12:24:47.640809 (+286483us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:24:47.640855 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:48.012686 (+371831us) spinlock_profiling.cc:243] Waited 63.6 ms on lock 0x7b54004259d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:24:48.012729 (+    43us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:48.012788 (+    59us) raft_consensus.cc:1685] finished
0709 12:24:48.012800 (+    12us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:48.027040 (+ 14240us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":28331,"raft.run_cpu_time_us":83,"raft.run_wall_time_us":83,"spinlock_wait_cycles":146267520,"thread_start_us":411550,"threads_started":2,"wal-append.queue_time_us":331600}
I20260709 12:24:48.219161 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:48.036121 26632 raft_consensus.cc:2749] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:24:48.291505 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:48.173398 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:48.125036 26354 raft_consensus.cc:2749] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:24:48.265156 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:48.271659 17728 leader_election.cc:304] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:48.167400 27012 raft_consensus.cc:2804] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:24:48.126108 27005 raft_consensus.cc:515] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:48.287539 17764 raft_consensus.cc:2410] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 27 because replica has last-logged OpId of term: 26 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 12.
W20260709 12:24:48.288189 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 112504us
I20260709 12:24:48.252821 26993 raft_consensus.cc:493] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:48.003041 26741 consensus_queue.cc:237] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 25.15, Last appended by leader: 15, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:48.212631 27014 raft_consensus.cc:2804] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:24:48.027060 26843 raft_consensus.cc:2804] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:24:48.158609 26635 raft_consensus.cc:493] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:48.220216 26593 raft_consensus.cc:3060] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:48.339648 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:48.300690 26726 raft_consensus.cc:2804] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:24:48.167973 26991 raft_consensus.cc:493] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:24:48.251104 26994 raft_consensus.cc:3060] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:48.137446 26966 raft_consensus.cc:2749] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
I20260709 12:24:48.003319 26844 consensus_queue.cc:237] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.14, Last appended by leader: 14, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:48.168274 26977 consensus_queue.cc:1048] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.138s
I20260709 12:24:48.286401 27001 raft_consensus.cc:493] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:48.294734 26837 raft_consensus.cc:2749] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
W20260709 12:24:48.332166 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 60676.6us
I20260709 12:24:48.339398 26979 raft_consensus.cc:2749] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:24:48.286295 26848 raft_consensus.cc:493] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:48.310793 26993 raft_consensus.cc:515] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:48.290234 26719 raft_consensus.cc:493] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:24:48.138849 26957 raft_consensus.cc:515] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:48.402261 26956 leader_election.cc:290] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 27 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:48.403283 26957 leader_election.cc:290] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:48.403976 26728 raft_consensus.cc:493] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:24:48.404564 26728 raft_consensus.cc:515] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:48.406917 26728 leader_election.cc:290] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:48.409174 27017 raft_consensus.cc:493] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:48.409695 27017 raft_consensus.cc:515] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:48.301247 26866 raft_consensus.cc:493] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:48.352133 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:48.191394 26962 raft_consensus.cc:515] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:48.138784 26625 raft_consensus.cc:515] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:48.412848 17687 raft_consensus.cc:2468] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 26.
I20260709 12:24:48.390393 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 26 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:48.327323 27014 raft_consensus.cc:493] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:24:48.229655 26971 raft_consensus.cc:515] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:48.421151 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 284523us
I20260709 12:24:48.301143 26963 raft_consensus.cc:515] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:48.351958 26632 raft_consensus.cc:493] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:48.343741 26726 raft_consensus.cc:493] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:24:48.298241 26865 raft_consensus.cc:515] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:48.437463 18374 raft_consensus.cc:2410] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for term 21 because replica has last-logged OpId of term: 19 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 13.
I20260709 12:24:48.392552 26599 raft_consensus.cc:493] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:48.401005 27013 raft_consensus.cc:493] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:48.315048 26985 consensus_queue.cc:260] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 21.14, Last appended by leader: 14, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:48.357339 26991 raft_consensus.cc:515] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:48.415228 17762 raft_consensus.cc:2468] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 26.
I20260709 12:24:48.442729 17652 leader_election.cc:304] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:48.412360 26866 raft_consensus.cc:515] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:48.299065 26982 raft_consensus.cc:515] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:48.411733 27017 leader_election.cc:290] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:48.436685 26726 raft_consensus.cc:3060] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:48.429250 17684 raft_consensus.cc:2468] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 26.
I20260709 12:24:48.308526 26888 consensus_queue.cc:237] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 25.17, Last appended by leader: 17, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:48.395567 27008 raft_consensus.cc:493] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:48.343307 26635 raft_consensus.cc:515] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:48.480046 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:24:48.439993 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:48.351646 26420 consensus_queue.cc:1059] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 26.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.068s
I20260709 12:24:48.442497 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 27 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:24:48.401399 27022 log.cc:927] Time spent T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.107s	user 0.001s	sys 0.000s
I20260709 12:24:48.421902 27014 raft_consensus.cc:3060] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:48.439865 17727 leader_election.cc:304] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:24:48.494081 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 54314.3us
I20260709 12:24:48.447451 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:48.455276 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:48.455627 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:48.461266 26625 leader_election.cc:290] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 20 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:48.497150 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:48.452930 26599 raft_consensus.cc:515] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:48.461917 26993 leader_election.cc:290] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:48.501120 17757 raft_consensus.cc:2393] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 27: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:24:48.501152 17681 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37396 (request call id 3312) took 1128 ms. Trace:
W20260709 12:24:48.472931 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 147561us
I20260709 12:24:48.475551 17802 leader_election.cc:304] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:24:48.414841 26848 raft_consensus.cc:515] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:48.430876 26719 raft_consensus.cc:515] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:48.494194 17687 raft_consensus.cc:2410] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 27 because replica has last-logged OpId of term: 26 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 12.
W20260709 12:24:48.389667 26800 consensus_meta.cc:287] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.773s	user 0.006s	sys 0.005s
I20260709 12:24:48.495440 18365 raft_consensus.cc:2393] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 26: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:24:48.480664 26761 leader_election.cc:290] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:48.497383 17685 raft_consensus.cc:1240] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 21. Current term is 22. Ops: [21.14-21.14]
I20260709 12:24:48.483527 27017 raft_consensus.cc:2804] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:24:48.396970 27001 raft_consensus.cc:515] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:48.483851 27012 raft_consensus.cc:697] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 24 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:24:48.501770 17681 rpcz_store.cc:276] 0709 12:24:47.366081 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:47.375919 (+  9838us) service_pool.cc:224] Handling call
0709 12:24:48.494532 (+1118613us) spinlock_profiling.cc:243] Waited 490 ms on lock 0x7b540014dcd4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:24:48.494634 (+   102us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1125864704}
I20260709 12:24:48.494803 17727 leader_election.cc:304] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:48.441555 26632 raft_consensus.cc:515] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:48.503700 17802 leader_election.cc:304] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:24:48.522217 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:48.523137 17757 raft_consensus.cc:2393] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 17: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:24:48.524075 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:48.524837 17757 raft_consensus.cc:2393] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 27: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:24:48.525628 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 23 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:48.526403 17757 raft_consensus.cc:2410] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 23 because replica has last-logged OpId of term: 22 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 14.
I20260709 12:24:48.498930 26843 raft_consensus.cc:697] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 30 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:24:48.527487 17757 raft_consensus.cc:1240] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 29. Current term is 30. Ops: [27.16-29.17]
I20260709 12:24:48.528199 26843 consensus_queue.cc:237] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 28.17, Last appended by leader: 17, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:48.529156 17689 raft_consensus.cc:2468] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 27.
I20260709 12:24:48.533876 18333 leader_election.cc:304] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:24:48.538496 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 43774.5us
I20260709 12:24:48.539340 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:48.540095 17681 raft_consensus.cc:2393] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 29: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
W20260709 12:24:48.541285 17802 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:48.543056 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:48.539794 17727 leader_election.cc:304] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:48.546763 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 26 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:48.512167 26985 raft_consensus.cc:3060] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:48.550402 17764 raft_consensus.cc:2393] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 26: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:24:48.510298 26810 raft_consensus.cc:2804] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:24:48.558184 17764 raft_consensus.cc:1275] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 28 index: 16. Preceding OpId from leader: term: 29 index: 17. (index mismatch)
I20260709 12:24:48.451363 27013 raft_consensus.cc:515] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:48.519212 27017 raft_consensus.cc:493] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:48.567574 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:48.568171 17687 raft_consensus.cc:2468] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 26.
W20260709 12:24:48.575080 26818 consensus_meta.cc:287] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.657s	user 0.008s	sys 0.000s
I20260709 12:24:48.549715 26837 raft_consensus.cc:2804] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:24:48.549089 27004 raft_consensus.cc:493] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:48.584952 17686 raft_consensus.cc:1240] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 28. Current term is 29. Ops: []
W20260709 12:24:48.577248 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 121630us
I20260709 12:24:48.552053 26980 leader_election.cc:290] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:48.559607 26924 raft_consensus.cc:2749] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:24:48.592233 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:24:48.575621 27024 log.cc:927] Time spent T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.090s	user 0.000s	sys 0.000s
I20260709 12:24:48.593583 18365 raft_consensus.cc:2393] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 27: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:24:48.594534 17652 leader_election.cc:304] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:48.508085 27008 raft_consensus.cc:515] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:48.567314 27017 raft_consensus.cc:3060] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:24:48.575628 26818 raft_consensus.cc:2749] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:24:48.529984 26800 raft_consensus.cc:515] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:48.579015 27005 leader_election.cc:290] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:48.569129 26810 raft_consensus.cc:697] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 26 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:24:48.578042 26837 raft_consensus.cc:493] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:24:48.604709 17763 raft_consensus.cc:2468] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 24.
I20260709 12:24:48.589241 17686 raft_consensus.cc:2955] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Committing config change with OpId 27.16: config changed from index 12 to 16, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:24:48.606778 17757 raft_consensus.cc:1240] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 23. Current term is 24. Ops: []
I20260709 12:24:48.591459 26728 raft_consensus.cc:493] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:24:48.607591 27019 raft_consensus.cc:493] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
W20260709 12:24:48.609221 27023 log.cc:927] Time spent T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.197s	user 0.001s	sys 0.000s
I20260709 12:24:48.609539 27030 raft_consensus.cc:2804] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:24:48.612140 17688 raft_consensus.cc:2368] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for earlier term 23. Current term is 24.
I20260709 12:24:48.606225 26984 raft_consensus.cc:2804] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:24:48.618695 27019 raft_consensus.cc:515] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:48.648082 26984 raft_consensus.cc:493] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:48.648511 26984 raft_consensus.cc:3060] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:24:48.649691 27019 leader_election.cc:290] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:48.618731 26962 leader_election.cc:290] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:48.620241 26810 consensus_queue.cc:237] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 24.19, Last appended by leader: 19, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:48.621099 18363 raft_consensus.cc:1275] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 22 index: 16. Preceding OpId from leader: term: 24 index: 17. (index mismatch)
I20260709 12:24:48.653110 26991 leader_election.cc:290] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:48.622334 26593 raft_consensus.cc:515] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:48.625453 26632 leader_election.cc:290] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:48.617115 27030 raft_consensus.cc:493] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:48.625890 27008 leader_election.cc:290] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:24:48.626905 27021 log.cc:927] Time spent T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.264s	user 0.002s	sys 0.000s
I20260709 12:24:48.625973 27033 raft_consensus.cc:2749] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:24:48.629359 26971 leader_election.cc:290] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:48.630260 27013 leader_election.cc:290] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:48.630789 27037 raft_consensus.cc:493] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:24:48.631357 27034 raft_consensus.cc:493] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:48.633193 26719 leader_election.cc:290] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:48.599445 26963 leader_election.cc:290] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:24:48.633423 26638 consensus_meta.cc:287] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.628s	user 0.007s	sys 0.003s
I20260709 12:24:48.636668 26599 leader_election.cc:290] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:48.637791 26966 raft_consensus.cc:493] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:48.637874 26726 raft_consensus.cc:515] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:48.616714 26956 raft_consensus.cc:493] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:48.638741 26800 leader_election.cc:290] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 24 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:24:48.639535 27029 log.cc:927] Time spent T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.134s	user 0.000s	sys 0.000s
I20260709 12:24:48.640954 27001 leader_election.cc:290] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:48.642059 26982 leader_election.cc:290] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:48.609915 27038 raft_consensus.cc:493] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:24:48.610062 26993 raft_consensus.cc:493] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:24:48.649834 27044 raft_consensus.cc:493] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:24:48.606992 26837 raft_consensus.cc:3060] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Advancing to term 25
W20260709 12:24:48.660133 26581 consensus_meta.cc:287] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.789s	user 0.012s	sys 0.000s
I20260709 12:24:48.595470 27004 raft_consensus.cc:515] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:48.683951 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 26 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:24:48.693701 26978 consensus_meta.cc:287] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.839s	user 0.008s	sys 0.006s
I20260709 12:24:48.617802 26728 raft_consensus.cc:515] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:48.694695 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:48.689042 26635 leader_election.cc:290] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:48.663707 27030 raft_consensus.cc:3060] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:24:48.707465 18363 raft_consensus.cc:2468] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 24.
I20260709 12:24:48.694505 27040 raft_consensus.cc:493] T a5732e71b9d54f8786b8c511d69943bf P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:48.709451 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:48.716073 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:48.608462 27012 consensus_queue.cc:237] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 22.18, Last appended by leader: 18, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:48.714437 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:48.688999 26638 raft_consensus.cc:515] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:48.696208 26955 leader_election.cc:290] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:48.724511 17837 raft_consensus.cc:1240] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 30 LEADER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 29. Current term is 30. Ops: []
I20260709 12:24:48.727556 26726 leader_election.cc:290] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 25 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:48.727641 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:48.715346 26993 raft_consensus.cc:515] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:48.709261 26581 raft_consensus.cc:515] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:48.732748 18362 raft_consensus.cc:2393] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 27: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:24:48.732615 17835 raft_consensus.cc:1275] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 27 index: 17. Preceding OpId from leader: term: 28 index: 18. (index mismatch)
I20260709 12:24:48.743387 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 26 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:48.693006 26956 raft_consensus.cc:515] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:48.732353 18371 raft_consensus.cc:2468] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 26.
I20260709 12:24:48.741973 17652 leader_election.cc:304] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:48.749230 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:48.684898 27037 raft_consensus.cc:515] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:48.692116 27034 raft_consensus.cc:515] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:48.760686 27034 leader_election.cc:290] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:48.742638 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:48.742035 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:48.693249 26966 raft_consensus.cc:515] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:48.745698 17801 leader_election.cc:304] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:48.748402 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:48.749147 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:48.762945 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:48.750895 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 24 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:48.742339 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 24 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:48.754573 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:48.756398 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:48.756527 26593 leader_election.cc:290] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 27 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:24:48.756886 27025 log.cc:927] Time spent T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.233s	user 0.000s	sys 0.002s
I20260709 12:24:48.767576 17756 raft_consensus.cc:2468] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 26.
I20260709 12:24:48.768395 17836 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35042 (request call id 3334) took 1068 ms. Trace:
I20260709 12:24:48.716732 26978 raft_consensus.cc:515] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:48.745715 17832 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 25.
I20260709 12:24:48.773413 18365 raft_consensus.cc:2468] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 23.
I20260709 12:24:48.801733 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:48.773425 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:24:48.777320 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 45689.5us
I20260709 12:24:48.802754 17686 raft_consensus.cc:2468] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 24.
W20260709 12:24:48.780179 17688 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37396 (request call id 3311) took 1414 ms (client timeout 1755 ms). Trace:
I20260709 12:24:48.781405 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:48.777788 27049 consensus_queue.cc:1059] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 26.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.036s
I20260709 12:24:48.784462 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:48.785328 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:48.735216 27040 raft_consensus.cc:515] T a5732e71b9d54f8786b8c511d69943bf P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:48.789029 27043 log.cc:927] Time spent T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.074s	user 0.000s	sys 0.000s
I20260709 12:24:48.791153 27037 leader_election.cc:290] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:24:48.791779 27041 log.cc:927] Time spent T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.131s	user 0.000s	sys 0.001s
I20260709 12:24:48.706002 27038 raft_consensus.cc:515] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:48.793632 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:48.792356 27039 raft_consensus.cc:2804] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:24:48.823253 17652 leader_election.cc:304] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:48.796206 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:48.705010 27044 raft_consensus.cc:515] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:48.795584 17836 rpcz_store.cc:276] 0709 12:24:47.699838 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:47.706947 (+  7109us) service_pool.cc:224] Handling call
0709 12:24:48.768190 (+1061243us) spinlock_profiling.cc:243] Waited 40.2 ms on lock 0x7b540008a7d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:24:48.768324 (+   134us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":92450688}
I20260709 12:24:48.799408 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 21 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:48.829137 17835 raft_consensus.cc:2410] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for term 21 because replica has last-logged OpId of term: 19 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 13.
I20260709 12:24:48.803519 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:24:48.804201 17688 rpcz_store.cc:269] 0709 12:24:47.365768 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:47.373513 (+  7745us) service_pool.cc:224] Handling call
0709 12:24:48.773934 (+1400421us) spinlock_profiling.cc:243] Waited 1.03 s on lock 0x7b5400345dd4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:24:48.780111 (+  6177us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2366018816}
I20260709 12:24:48.761350 26993 leader_election.cc:290] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:48.815164 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:48.774541 27034 raft_consensus.cc:3060] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:48.831823 17832 raft_consensus.cc:2468] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 24.
I20260709 12:24:48.817903 17683 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37402 (request call id 3746) took 1152 ms. Trace:
I20260709 12:24:48.792209 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:48.835337 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:48.836416 17833 raft_consensus.cc:3060] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:48.809953 18362 raft_consensus.cc:2468] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 30.
I20260709 12:24:48.820294 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:48.825345 17830 raft_consensus.cc:2468] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 25.
I20260709 12:24:48.737628 26924 raft_consensus.cc:493] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:48.848160 26924 raft_consensus.cc:515] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:48.850240 26924 leader_election.cc:290] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:48.825489 17758 raft_consensus.cc:2368] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 30. Current term is 31.
I20260709 12:24:48.833786 17683 rpcz_store.cc:276] 0709 12:24:47.665082 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:47.669841 (+  4759us) service_pool.cc:224] Handling call
0709 12:24:47.670161 (+   320us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:24:47.780435 (+110274us) raft_consensus.cc:1534] Early marking committed up to index 15
0709 12:24:47.799770 (+ 19335us) log.cc:844] Serialized 14 byte log entry
0709 12:24:47.938048 (+138278us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:24:47.955859 (+ 17811us) log.cc:844] Serialized 52 byte log entry
0709 12:24:47.956075 (+   216us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:24:47.956108 (+    33us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:47.956173 (+    65us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:48.817557 (+861384us) raft_consensus.cc:1685] finished
0709 12:24:48.817590 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:48.817813 (+   223us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5201,"prepare.queue_time_us":25745,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":42,"raft.queue_time_us":108196,"raft.run_cpu_time_us":479,"raft.run_wall_time_us":25428,"thread_start_us":188526,"threads_started":2,"wal-append.queue_time_us":299652}
I20260709 12:24:48.826186 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:48.829929 17764 raft_consensus.cc:2468] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 30.
I20260709 12:24:48.831425 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:48.831282 27001 raft_consensus.cc:3055] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 29 LEADER]: Stepping down as leader of term 29
W20260709 12:24:48.840122 26587 consensus_meta.cc:287] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.919s	user 0.010s	sys 0.002s
I20260709 12:24:48.841321 26966 leader_election.cc:290] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:48.843781 27038 leader_election.cc:290] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:48.844100 27044 leader_election.cc:290] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:48.769287 27051 consensus_queue.cc:1048] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 14, Time since last communication: 0.080s
I20260709 12:24:48.845187 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:48.845873 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 26 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:48.865998 18364 raft_consensus.cc:3060] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Advancing to term 26
I20260709 12:24:48.866307 17836 raft_consensus.cc:2393] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 22: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:24:48.847322 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:48.863097 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:48.867522 17757 raft_consensus.cc:3060] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Advancing to term 24
I20260709 12:24:48.868121 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 31 candidate_status { last_received { term: 30 index: 23 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:48.868714 17764 raft_consensus.cc:2468] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 30.
I20260709 12:24:48.869572 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:48.870698 17764 raft_consensus.cc:2393] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 27: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:24:48.862289 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 31 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:48.828717 27039 raft_consensus.cc:493] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:48.873924 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:48.875118 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:48.872162 17837 raft_consensus.cc:2468] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 29.
I20260709 12:24:48.875619 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:48.876062 17652 leader_election.cc:304] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:48.876483 18365 raft_consensus.cc:2393] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 27: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:24:48.876225 17764 raft_consensus.cc:2468] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 27.
I20260709 12:24:48.877339 17652 leader_election.cc:304] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:48.879048 18364 raft_consensus.cc:2468] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 26.
I20260709 12:24:48.881680 17683 raft_consensus.cc:3060] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:48.834890 27037 raft_consensus.cc:2804] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:24:48.885397 18333 leader_election.cc:304] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:24:48.888294 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:48.832288 26956 leader_election.cc:290] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:48.886088 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:48.886303 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:48.892038 17764 raft_consensus.cc:2393] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 26: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:24:48.890432 18364 raft_consensus.cc:2468] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 27.
I20260709 12:24:48.892375 17652 leader_election.cc:304] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:48.896195 18361 raft_consensus.cc:1275] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 22 index: 15. Preceding OpId from leader: term: 23 index: 16. (index mismatch)
I20260709 12:24:48.875447 17803 leader_election.cc:304] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:48.897231 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:48.898099 17830 raft_consensus.cc:3060] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:48.907227 17832 raft_consensus.cc:2393] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 24: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:24:48.887795 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 26 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:48.912190 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 30 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:48.912978 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:48.859349 27001 raft_consensus.cc:740] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 29 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:24:48.909752 18365 raft_consensus.cc:2376] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 26. Re-sending same reply.
I20260709 12:24:48.910125 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:48.918352 17764 raft_consensus.cc:2468] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 25.
I20260709 12:24:48.879621 27039 raft_consensus.cc:3060] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:24:48.918514 17652 leader_election.cc:304] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:48.896121 26800 raft_consensus.cc:2749] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260709 12:24:48.923478 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:48.923902 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 23 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:48.924271 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:48.925653 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 27 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:48.925904 17652 leader_election.cc:304] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:48.887279 27037 raft_consensus.cc:493] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:48.940230 17652 leader_election.cc:304] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:24:48.929527 17764 raft_consensus.cc:2368] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 26. Current term is 27.
I20260709 12:24:48.927415 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:48.920502 26593 raft_consensus.cc:493] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:48.887421 27038 consensus_queue.cc:1048] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.045s
I20260709 12:24:48.934902 26837 raft_consensus.cc:515] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:48.937512 27040 leader_election.cc:290] T a5732e71b9d54f8786b8c511d69943bf P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:48.942443 18374 raft_consensus.cc:2468] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 24.
I20260709 12:24:48.875016 26994 raft_consensus.cc:515] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:48.863116 27049 consensus_queue.cc:1059] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.054s
I20260709 12:24:48.906075 26962 raft_consensus.cc:493] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:48.945921 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:48.874908 26993 consensus_queue.cc:1059] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.053s
I20260709 12:24:48.931123 18333 leader_election.cc:304] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:48.878437 26843 consensus_queue.cc:1048] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.045s
I20260709 12:24:48.949745 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:48.943809 27037 raft_consensus.cc:3060] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:48.951704 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:48.920502 26888 raft_consensus.cc:2749] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260709 12:24:48.954701 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 22 candidate_status { last_received { term: 21 index: 20 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:48.985764 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:48.956410 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:48.958734 27044 raft_consensus.cc:2749] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:24:48.925069 26635 raft_consensus.cc:2804] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:24:48.987691 17683 raft_consensus.cc:2468] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 20.
I20260709 12:24:48.897239 26963 consensus_queue.cc:1059] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 21.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.046s
I20260709 12:24:48.938488 26741 raft_consensus.cc:2804] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:24:48.928113 27012 raft_consensus.cc:2749] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:24:48.950052 17652 leader_election.cc:304] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:48.943461 27033 raft_consensus.cc:2749] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:24:48.985380 17764 raft_consensus.cc:2393] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 25: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:24:48.961779 17830 raft_consensus.cc:2468] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 27.
I20260709 12:24:48.948627 26719 raft_consensus.cc:2804] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:24:48.953044 26632 raft_consensus.cc:493] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:48.996580 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 76700.7us
I20260709 12:24:48.997802 17802 leader_election.cc:304] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:49.026111 18333 leader_election.cc:400] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 25 pre-election: Vote denied by peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837) with higher term. Message: Invalid argument: T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 25. Current term is 26.
I20260709 12:24:49.027207 18333 leader_election.cc:403] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 25 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:24:49.027453 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 26 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:49.028155 17759 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36572 (request call id 1611) took 1167 ms. Trace:
I20260709 12:24:49.027019 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 27 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:49.028600 17759 rpcz_store.cc:276] 0709 12:24:47.861198 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:47.861666 (+   468us) service_pool.cc:224] Handling call
0709 12:24:47.874052 (+ 12386us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:48.138089 (+264037us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:48.138132 (+    43us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:48.138179 (+    47us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:24:48.138211 (+    32us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:49.027926 (+889715us) spinlock_profiling.cc:243] Waited 251 ms on lock 0x7b54003e44d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:24:49.027977 (+    51us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:49.028098 (+   121us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":577884416}
I20260709 12:24:49.036108 18333 leader_election.cc:304] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:24:49.037617 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 87655.8us
I20260709 12:24:48.987427 26728 leader_election.cc:290] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:48.995224 26966 raft_consensus.cc:2749] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:24:48.957974 26984 raft_consensus.cc:515] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:48.948935 26593 raft_consensus.cc:515] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:49.042446 18333 leader_election.cc:304] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:24:49.043776 17757 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 24.
I20260709 12:24:48.994688 26910 leader_election.cc:290] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:49.025027 26714 raft_consensus.cc:493] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:48.994462 26638 leader_election.cc:290] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 27 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:49.041080 17802 leader_election.cc:304] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:24:48.998778 26741 raft_consensus.cc:493] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:49.050918 26994 leader_election.cc:290] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 27 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:24:49.052963 26789 consensus_meta.cc:287] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.919s	user 0.012s	sys 0.000s
W20260709 12:24:49.054188 27014 consensus_meta.cc:287] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.546s	user 0.007s	sys 0.004s
I20260709 12:24:49.054634 26978 leader_election.cc:290] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 31 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:49.034197 26963 raft_consensus.cc:2749] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Leader pre-election lost for term 25. Reason: Vote denied by peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837) with higher term. Message: Invalid argument: T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 25. Current term is 26.
I20260709 12:24:49.055955 26837 leader_election.cc:290] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 25 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:24:49.056504 27061 log.cc:927] Time spent T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.061s	user 0.000s	sys 0.000s
I20260709 12:24:49.056326 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 26 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:49.046995 27008 consensus_queue.cc:1059] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 28.16, Next index: 17, Last known committed idx: 14, Time since last communication: 0.011s
I20260709 12:24:48.966336 26962 raft_consensus.cc:515] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:49.039628 17652 leader_election.cc:304] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:49.033811 26632 raft_consensus.cc:515] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:49.025666 26719 raft_consensus.cc:493] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:49.025278 27040 raft_consensus.cc:3055] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:24:49.128803 27040 raft_consensus.cc:740] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:24:49.129920 27040 consensus_queue.cc:260] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 23.17, Last appended by leader: 17, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:49.130625 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:24:49.132993 18334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:49.062781 17835 raft_consensus.cc:2468] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 20.
I20260709 12:24:49.136858 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:49.143360 26719 raft_consensus.cc:3060] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:49.060321 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:49.145994 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:48.994681 26635 raft_consensus.cc:493] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:49.148173 26635 raft_consensus.cc:3060] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:24:49.111740 27014 raft_consensus.cc:515] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:49.150940 27014 leader_election.cc:290] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 23 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:49.026487 26599 raft_consensus.cc:2749] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:24:49.033855 26993 consensus_queue.cc:1243] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer e063541542a94969898868bf6eb5ce1b log is divergent from this leader: its last log entry 30.18 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 16
I20260709 12:24:49.151963 17682 raft_consensus.cc:2393] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 23: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:24:49.094187 17833 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 24.
I20260709 12:24:49.152940 17684 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37396 (request call id 3321) took 1160 ms. Trace:
W20260709 12:24:49.104153 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 63196.3us
I20260709 12:24:49.106992 17829 raft_consensus.cc:2368] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for earlier term 27. Current term is 28.
I20260709 12:24:49.153496 17684 rpcz_store.cc:276] 0709 12:24:47.992619 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:48.193760 (+201141us) service_pool.cc:224] Handling call
0709 12:24:49.152732 (+958972us) spinlock_profiling.cc:243] Waited 141 ms on lock 0x7b5400348ad4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:24:49.152865 (+   133us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":325264256}
I20260709 12:24:49.160972 27001 consensus_queue.cc:260] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 29.17, Last appended by leader: 17, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:24:49.116909 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 77358.4us
I20260709 12:24:49.113125 17832 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 25.
I20260709 12:24:49.040629 26971 consensus_queue.cc:1059] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 26.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.071s
I20260709 12:24:49.108747 26910 raft_consensus.cc:2804] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:24:49.169322 26910 raft_consensus.cc:493] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:49.169876 26866 leader_election.cc:290] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:49.170254 26910 raft_consensus.cc:3060] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:49.153129 17833 raft_consensus.cc:1240] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 21. Current term is 22. Ops: []
I20260709 12:24:49.173682 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:49.174162 26984 leader_election.cc:290] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 28 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:49.174430 17756 raft_consensus.cc:2468] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 27.
I20260709 12:24:49.105266 26741 raft_consensus.cc:3060] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:49.127691 18361 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59886 (request call id 2462) took 1993 ms. Trace:
I20260709 12:24:49.088222 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 27 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:49.175274 18361 rpcz_store.cc:276] 0709 12:24:47.133778 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:47.134218 (+   440us) service_pool.cc:224] Handling call
0709 12:24:47.160694 (+ 26476us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:48.896577 (+1735883us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:49.127302 (+230725us) spinlock_profiling.cc:243] Waited 1.72 s on lock 0x7b5400499654. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:24:49.127569 (+   267us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3963491584}
I20260709 12:24:49.176191 17757 raft_consensus.cc:2468] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 26.
I20260709 12:24:49.059804 27017 raft_consensus.cc:515] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:49.177243 27008 raft_consensus.cc:3055] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:24:49.177410 17757 raft_consensus.cc:1275] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 22 index: 16. Preceding OpId from leader: term: 24 index: 17. (index mismatch)
I20260709 12:24:49.177663 27008 raft_consensus.cc:740] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 28 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:24:49.178391 27017 leader_election.cc:290] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 31 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:49.169322 17652 leader_election.cc:304] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:49.178467 27008 consensus_queue.cc:260] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 28.16, Last appended by leader: 16, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:49.163754 17831 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37994 (request call id 1698) took 1101 ms. Trace:
W20260709 12:24:49.180372 17572 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet d522f70beb1f417cb98940d90b594823 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 0da169ff82674dc1947055ce43de68a3 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 24 committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:24:49.179770 17831 rpcz_store.cc:276] 0709 12:24:48.061902 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:48.062055 (+   153us) service_pool.cc:224] Handling call
0709 12:24:48.130112 (+ 68057us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:48.347593 (+217481us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:48.347631 (+    38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:48.398140 (+ 50509us) log.cc:844] Serialized 26 byte log entry
0709 12:24:48.569674 (+171534us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:24:48.588406 (+ 18732us) log.cc:844] Serialized 14 byte log entry
0709 12:24:48.689308 (+100902us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:49.162817 (+473509us) spinlock_profiling.cc:243] Waited 212 ms on lock 0x7b5400334ad4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:24:49.162897 (+    80us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:49.163124 (+   227us) raft_consensus.cc:1685] finished
0709 12:24:49.163270 (+   146us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:49.163615 (+   345us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2789,"prepare.queue_time_us":18991,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":39,"raft.queue_time_us":35678,"raft.run_cpu_time_us":285,"raft.run_wall_time_us":73897,"spinlock_wait_cycles":488484096,"thread_start_us":218655,"threads_started":2,"wal-append.queue_time_us":227160}
I20260709 12:24:49.182111 27067 raft_consensus.cc:2804] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:24:49.182591 27067 raft_consensus.cc:493] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:24:49.182982 27067 raft_consensus.cc:3060] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:49.184185 17838 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37994 (request call id 1700) took 1121 ms. Trace:
I20260709 12:24:49.184255 17756 raft_consensus.cc:1240] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 24. Current term is 25. Ops: []
I20260709 12:24:49.104467 26714 raft_consensus.cc:515] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:49.128286 27033 raft_consensus.cc:2804] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:24:49.147595 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 26 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:49.040529 26587 consensus_queue.cc:1048] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.070s
I20260709 12:24:49.131552 27040 raft_consensus.cc:3060] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:49.198167 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 31 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:49.178790 17757 raft_consensus.cc:2955] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Committing config change with OpId 19.12: config changed from index 5 to 12, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:24:49.198890 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 31 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:49.047585 26800 raft_consensus.cc:2804] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:24:49.184782 17838 rpcz_store.cc:276] 0709 12:24:48.062787 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:48.062916 (+   129us) service_pool.cc:224] Handling call
0709 12:24:48.130355 (+ 67439us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:48.169321 (+ 38966us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:48.169368 (+    47us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:48.281622 (+112254us) log.cc:844] Serialized 26 byte log entry
0709 12:24:48.451369 (+169747us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:24:48.494354 (+ 42985us) log.cc:844] Serialized 14 byte log entry
0709 12:24:48.494666 (+   312us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:48.494778 (+   112us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:49.172131 (+677353us) spinlock_profiling.cc:243] Waited 6.21 ms on lock 0x7b1800206aa0. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b475333f3 00007f7b475357c1 00007f7b47535779 00007f7b47535709 00007f7b475356d1 00007f7b475349cd 0000000000388634 0000000000388479 00007f7b4751683a
0709 12:24:49.178616 (+  6485us) spinlock_profiling.cc:243] Waited 2.19 ms on lock 0x7b1800206aa0. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b475333f3 00007f7b475357c1 00007f7b47535779 00007f7b47535709 00007f7b475356d1 00007f7b475349cd 0000000000388634 0000000000388479 00007f7b4751683a
0709 12:24:49.183955 (+  5339us) raft_consensus.cc:1685] finished
0709 12:24:49.183981 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:49.184124 (+   143us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":43630,"prepare.queue_time_us":42250,"prepare.run_cpu_time_us":65,"prepare.run_wall_time_us":67,"raft.queue_time_us":4237,"raft.run_cpu_time_us":232,"raft.run_wall_time_us":46182,"spinlock_wait_cycles":19323520,"thread_start_us":115383,"threads_started":1,"wal-append.queue_time_us":221143}
I20260709 12:24:49.192595 17802 leader_election.cc:304] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
W20260709 12:24:49.192435 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 150043us
I20260709 12:24:49.040480 26818 raft_consensus.cc:2749] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:24:49.196700 17652 leader_election.cc:304] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:49.206529 17833 raft_consensus.cc:1275] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 25 index: 15. Preceding OpId from leader: term: 27 index: 16. (index mismatch)
I20260709 12:24:49.198976 17829 raft_consensus.cc:3060] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Advancing to term 31
I20260709 12:24:49.192305 17755 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36572 (request call id 1617) took 1224 ms. Trace:
I20260709 12:24:49.113868 27051 raft_consensus.cc:493] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:49.204303 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
W20260709 12:24:49.216341 17804 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:49.207563 18333 leader_election.cc:304] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:24:49.190757 27008 raft_consensus.cc:3060] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:24:49.207561 17755 rpcz_store.cc:276] 0709 12:24:47.968448 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:47.968605 (+   157us) service_pool.cc:224] Handling call
0709 12:24:47.980988 (+ 12383us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:48.275929 (+294941us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:49.192068 (+916139us) spinlock_profiling.cc:243] Waited 294 ms on lock 0x7b54003e5dd4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:24:49.192247 (+   179us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":676161408}
I20260709 12:24:49.237718 18333 leader_election.cc:304] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:49.250998 18333 leader_election.cc:304] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:49.198570 27033 raft_consensus.cc:493] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:24:49.201045 26800 raft_consensus.cc:493] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:49.204070 26966 raft_consensus.cc:2804] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:24:49.310745 26966 raft_consensus.cc:697] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 26 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:24:49.311772 26966 consensus_queue.cc:237] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 25.18, Last appended by leader: 18, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:49.312252 26593 leader_election.cc:290] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:49.317087 17680 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44898 (request call id 1800) took 1323 ms. Trace:
I20260709 12:24:49.317498 17680 rpcz_store.cc:276] 0709 12:24:47.993943 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:48.792578 (+798635us) service_pool.cc:224] Handling call
0709 12:24:48.792849 (+   271us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:48.851668 (+ 58819us) raft_consensus.cc:1534] Early marking committed up to index 23
0709 12:24:48.926526 (+ 74858us) log.cc:844] Serialized 14 byte log entry
0709 12:24:49.023271 (+ 96745us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:49.023344 (+    73us) raft_consensus.cc:1645] Marking committed up to 23
0709 12:24:49.023406 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:49.142198 (+118792us) spinlock_profiling.cc:243] Waited 20.7 ms on lock 0x7b180047c080. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b475333f3 00007f7b475357c1 00007f7b47535779 00007f7b47535709 00007f7b475356d1 00007f7b475349cd 0000000000388634 0000000000388479 00007f7b4751683a
0709 12:24:49.316842 (+174644us) spinlock_profiling.cc:243] Waited 58.6 ms on lock 0x7b5400208354. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:24:49.316893 (+    51us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:49.317024 (+   131us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":24008,"prepare.queue_time_us":193557,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":47,"raft.queue_time_us":24363,"raft.run_cpu_time_us":127,"raft.run_wall_time_us":7381,"spinlock_wait_cycles":182237184,"thread_start_us":118872,"threads_started":2,"wal-append.queue_time_us":192258}
I20260709 12:24:49.321173 27011 leader_election.cc:290] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:49.322275 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:49.323415 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:49.325117 26632 leader_election.cc:290] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:49.309500 26962 leader_election.cc:290] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:49.321028 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:49.329707 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:49.333485 17829 raft_consensus.cc:2468] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 31.
I20260709 12:24:49.334582 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:49.335187 17829 raft_consensus.cc:2468] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 23.
I20260709 12:24:49.208583 27027 raft_consensus.cc:493] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:49.338766 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:49.340210 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 26 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:49.262310 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:49.346093 17832 raft_consensus.cc:2468] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 25.
I20260709 12:24:49.254016 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 28 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:49.347287 17838 raft_consensus.cc:2376] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 28. Re-sending same reply.
I20260709 12:24:49.240099 27001 raft_consensus.cc:3060] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Advancing to term 30
W20260709 12:24:49.249588 26985 consensus_meta.cc:287] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.677s	user 0.006s	sys 0.010s
W20260709 12:24:49.237481 27030 consensus_meta.cc:287] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.505s	user 0.011s	sys 0.000s
I20260709 12:24:49.252208 26599 raft_consensus.cc:2804] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:24:49.345541 26800 raft_consensus.cc:3060] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:24:49.230865 27012 raft_consensus.cc:2749] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
W20260709 12:24:49.358965 17802 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:49.252228 27034 raft_consensus.cc:2749] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:24:49.364784 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:49.368366 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:49.382985 17755 raft_consensus.cc:2468] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 25.
I20260709 12:24:49.296617 27033 raft_consensus.cc:3060] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:24:49.383463 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:49.384497 17762 raft_consensus.cc:1275] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 16 index: 10. Preceding OpId from leader: term: 17 index: 11. (index mismatch)
I20260709 12:24:49.287894 26818 raft_consensus.cc:2804] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:24:49.356564 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:49.385654 17689 raft_consensus.cc:3060] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Advancing to term 28
I20260709 12:24:49.255782 26963 raft_consensus.cc:2804] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:24:49.309584 26714 leader_election.cc:290] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:49.251524 26924 raft_consensus.cc:2804] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:24:49.386866 17684 raft_consensus.cc:1275] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 25 index: 17. Preceding OpId from leader: term: 27 index: 18. (index mismatch)
I20260709 12:24:49.249135 27051 raft_consensus.cc:515] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:49.331799 26635 raft_consensus.cc:515] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:49.372991 26599 raft_consensus.cc:493] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:49.395756 17761 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36572 (request call id 1609) took 1535 ms. Trace:
I20260709 12:24:49.399415 17761 rpcz_store.cc:276] 0709 12:24:47.860633 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:47.866323 (+  5690us) service_pool.cc:224] Handling call
0709 12:24:47.866623 (+   300us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:47.932551 (+ 65928us) raft_consensus.cc:1534] Early marking committed up to index 16
0709 12:24:47.959532 (+ 26981us) log.cc:844] Serialized 14 byte log entry
0709 12:24:49.128127 (+1168595us) spinlock_profiling.cc:243] Waited 26.9 ms on lock 0x7b18004a1a00. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b47515e07 00007f7b503709a9 00007f7b4753269b 00007f7b4753093f 00007f7b47532c02 00007f7b4a1ba20b 00007f7b4a1bfd23 00007f7b4a1c023a 00007f7b4a62aa2a
0709 12:24:49.395404 (+267277us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:49.395473 (+    69us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:24:49.395520 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:49.395592 (+    72us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:49.395703 (+   111us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":26836,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":49,"raft.queue_time_us":3128,"spinlock_wait_cycles":61857152,"thread_start_us":267058,"threads_started":1}
I20260709 12:24:49.367465 27030 raft_consensus.cc:515] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:49.389604 26963 raft_consensus.cc:493] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:49.387482 26818 raft_consensus.cc:493] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:49.407325 17689 raft_consensus.cc:2468] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 28.
I20260709 12:24:49.439579 17837 raft_consensus.cc:2468] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 29.
W20260709 12:24:49.441442 17803 leader_election.cc:336] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Timed out: RequestConsensusVote RPC to 127.16.29.68:42577 timed out after 1.583s (SENT)
I20260709 12:24:49.442098 17803 leader_election.cc:304] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:49.352783 27027 raft_consensus.cc:515] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:49.409587 26599 raft_consensus.cc:3060] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:49.399349 26883 raft_consensus.cc:2804] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:24:49.508994 26883 raft_consensus.cc:493] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:24:49.509456 26883 raft_consensus.cc:3060] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:49.472923 27030 leader_election.cc:290] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 28 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:24:49.473670 27039 consensus_meta.cc:287] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.535s	user 0.006s	sys 0.004s
I20260709 12:24:49.512563 26994 raft_consensus.cc:493] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:49.512684 27039 raft_consensus.cc:515] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:49.476872 27019 raft_consensus.cc:3055] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:24:49.513728 27019 raft_consensus.cc:740] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:24:49.513527 26994 raft_consensus.cc:515] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:49.514640 27019 consensus_queue.cc:260] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 23.18, Last appended by leader: 18, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:49.480552 17759 raft_consensus.cc:2393] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 28: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:24:49.400393 26924 raft_consensus.cc:697] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 24 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:24:49.484613 17729 leader_election.cc:304] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:49.485147 26587 raft_consensus.cc:493] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:49.485324 26800 raft_consensus.cc:515] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:49.486673 26887 leader_election.cc:290] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:49.486626 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:49.487516 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:49.489567 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:49.489207 26719 raft_consensus.cc:515] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:49.489797 17760 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38786 (request call id 3970) took 1467 ms. Trace:
I20260709 12:24:49.526855 18367 raft_consensus.cc:2468] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 24.
I20260709 12:24:49.490382 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:49.491099 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:49.491366 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:49.528995 17834 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37994 (request call id 1705) took 1366 ms. Trace:
I20260709 12:24:49.492897 26848 leader_election.cc:290] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:49.491748 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:49.501407 26843 raft_consensus.cc:493] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:49.513247 26635 leader_election.cc:290] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 30 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:49.467131 26789 raft_consensus.cc:2749] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:24:49.410431 26865 leader_election.cc:290] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:49.442672 26963 raft_consensus.cc:3060] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Advancing to term 28
W20260709 12:24:49.522863 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 180063us
I20260709 12:24:49.446966 26818 raft_consensus.cc:3060] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Advancing to term 28
W20260709 12:24:49.516412 27022 log.cc:927] Time spent T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.096s	user 0.000s	sys 0.000s
W20260709 12:24:49.524986 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 274010us
I20260709 12:24:49.525286 17835 raft_consensus.cc:2393] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 28: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:24:49.492203 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:49.517948 27019 raft_consensus.cc:3060] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:49.541450 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:49.529539 17834 rpcz_store.cc:276] 0709 12:24:48.162791 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:48.163397 (+   606us) service_pool.cc:224] Handling call
0709 12:24:48.768326 (+604929us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:48.793020 (+ 24694us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:48.793068 (+    48us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:48.793772 (+   704us) log.cc:844] Serialized 26 byte log entry
0709 12:24:48.832704 (+ 38932us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:24:48.832746 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:48.839232 (+  6486us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:49.121421 (+282189us) raft_consensus.cc:1685] finished
0709 12:24:49.121443 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:49.519470 (+398027us) spinlock_profiling.cc:243] Waited 605 ms on lock 0x7b540008a7d0. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a61f8ad 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:24:49.528915 (+  9445us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":14900,"raft.queue_time_us":16325,"raft.run_cpu_time_us":162,"raft.run_wall_time_us":16691,"spinlock_wait_cycles":1390362880,"thread_start_us":33236,"threads_started":1,"wal-append.queue_time_us":121465}
I20260709 12:24:49.564213 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:49.565871 18362 raft_consensus.cc:2468] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 30.
I20260709 12:24:49.573902 18333 leader_election.cc:304] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:49.526913 17760 rpcz_store.cc:276] 0709 12:24:48.022907 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:48.160045 (+137138us) service_pool.cc:224] Handling call
0709 12:24:48.216158 (+ 56113us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:48.488120 (+271962us) raft_consensus.cc:1534] Early marking committed up to index 16
0709 12:24:48.507518 (+ 19398us) log.cc:844] Serialized 14 byte log entry
0709 12:24:48.659846 (+152328us) spinlock_profiling.cc:243] Waited 5.84 ms on lock 0x7b18001787e0. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b47515e07 00007f7b503709a9 00007f7b4753269b 00007f7b4753093f 00007f7b47532c02 00007f7b4a1ba20b 00007f7b4a1bfd23 00007f7b4a1c023a 00007f7b4a62aa2a
0709 12:24:48.719255 (+ 59409us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:48.719310 (+    55us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:24:48.719352 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:49.444974 (+725622us) spinlock_profiling.cc:243] Waited 2.55 ms on lock 0x7b18001787e0. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b475012a6 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419
0709 12:24:49.489586 (+ 44612us) spinlock_profiling.cc:243] Waited 272 ms on lock 0x7b5400216e54. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:24:49.489631 (+    45us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:49.489747 (+   116us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":18248,"prepare.run_cpu_time_us":48,"prepare.run_wall_time_us":50,"raft.queue_time_us":84,"raft.run_cpu_time_us":123,"raft.run_wall_time_us":14565,"spinlock_wait_cycles":644164992,"thread_start_us":59096,"threads_started":1,"wal-append.queue_time_us":248329}
I20260709 12:24:49.575074 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:49.576076 17689 raft_consensus.cc:2393] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 20: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:24:49.576246 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 31 candidate_status { last_received { term: 30 index: 23 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:49.585141 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:49.587000 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:49.587378 17654 leader_election.cc:304] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:49.532681 26957 raft_consensus.cc:2804] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:24:49.590610 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 27 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:49.592603 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:49.592610 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:49.596943 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 24 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:49.599769 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 25 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:49.601501 18374 raft_consensus.cc:1275] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 25 index: 15. Preceding OpId from leader: term: 27 index: 16. (index mismatch)
I20260709 12:24:49.620493 18374 raft_consensus.cc:1275] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 23 index: 19. Preceding OpId from leader: term: 24 index: 19. (term mismatch)
I20260709 12:24:49.621068 18374 pending_rounds.cc:85] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da: Aborting all ops after (but not including) 18
I20260709 12:24:49.621381 18374 pending_rounds.cc:107] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da: Aborting uncommitted NO_OP operation due to leader change: 23.19
I20260709 12:24:49.621703 18374 raft_consensus.cc:2889] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:49.532370 26587 raft_consensus.cc:515] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:49.623113 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:49.592788 26957 raft_consensus.cc:493] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:49.564486 26843 raft_consensus.cc:515] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:49.628001 17689 raft_consensus.cc:2468] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 27.
I20260709 12:24:49.629161 18368 raft_consensus.cc:1240] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 29. Current term is 30. Ops: []
I20260709 12:24:49.603199 18361 raft_consensus.cc:2410] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for term 24 because replica has last-logged OpId of term: 22 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 14.
I20260709 12:24:49.596614 26866 raft_consensus.cc:2749] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:24:49.580240 26844 consensus_queue.cc:1048] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.045s
I20260709 12:24:49.580775 26966 consensus_queue.cc:1048] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.042s
I20260709 12:24:49.664234 17758 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38786 (request call id 3968) took 1617 ms. Trace:
W20260709 12:24:49.668615 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 94789.6us
I20260709 12:24:49.671051 18365 raft_consensus.cc:3060] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:49.669446 17758 rpcz_store.cc:276] 0709 12:24:48.019612 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:48.019738 (+   126us) service_pool.cc:224] Handling call
0709 12:24:49.636986 (+1617248us) spinlock_profiling.cc:243] Waited 696 ms on lock 0x7b54003e62d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:24:49.637116 (+   130us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1601725312}
I20260709 12:24:49.679207 17803 leader_election.cc:304] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:49.682154 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:49.638564 26957 raft_consensus.cc:3060] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:24:49.681129 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:49.685492 17756 raft_consensus.cc:2393] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 27: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
W20260709 12:24:49.671423 27072 log.cc:927] Time spent T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.052s	user 0.001s	sys 0.000s
I20260709 12:24:49.601684 26910 raft_consensus.cc:515] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:49.665190 26994 leader_election.cc:290] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:24:49.673378 27071 log.cc:927] Time spent T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.077s	user 0.000s	sys 0.001s
I20260709 12:24:49.664696 26962 raft_consensus.cc:2749] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:24:49.691316 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:49.721553 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:24:49.723846 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:49.692742 17803 leader_election.cc:304] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:49.586741 26635 consensus_queue.cc:1059] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 24.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.081s
I20260709 12:24:49.693789 17760 raft_consensus.cc:1240] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 27. Current term is 28. Ops: [27.21-27.21]
I20260709 12:24:49.694810 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 25 candidate_status { last_received { term: 23 index: 19 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:49.635669 26883 raft_consensus.cc:515] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:49.722494 18374 raft_consensus.cc:2468] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 20.
I20260709 12:24:49.688174 27035 raft_consensus.cc:3055] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:24:49.756179 17836 raft_consensus.cc:2468] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 29.
I20260709 12:24:49.760339 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:49.623620 27067 raft_consensus.cc:515] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:49.613018 26924 consensus_queue.cc:237] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 22.17, Last appended by leader: 17, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:49.628728 26741 raft_consensus.cc:515] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:49.757941 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:49.690301 26848 raft_consensus.cc:2749] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
W20260709 12:24:49.786581 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 93920.4us
I20260709 12:24:49.788808 18333 leader_election.cc:304] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:49.790781 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:49.794183 17803 leader_election.cc:304] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:49.796278 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:49.796413 18361 raft_consensus.cc:2410] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 29 because replica has last-logged OpId of term: 28 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 27 index: 17.
I20260709 12:24:49.796973 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:49.684175 26971 consensus_queue.cc:1048] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.079s
I20260709 12:24:49.804316 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:49.804780 18368 raft_consensus.cc:2368] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 26. Current term is 27.
I20260709 12:24:49.809868 18365 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 24.
I20260709 12:24:49.810953 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 22 candidate_status { last_received { term: 21 index: 20 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:49.812053 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 26 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:49.826498 18374 raft_consensus.cc:2376] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 26. Re-sending same reply.
I20260709 12:24:49.758303 26887 raft_consensus.cc:2749] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:24:49.816566 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:49.817840 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 30 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:49.818197 27067 leader_election.cc:290] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 26 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:49.758996 27039 leader_election.cc:290] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 28 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:49.820152 17686 raft_consensus.cc:1240] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 25. Current term is 26. Ops: []
W20260709 12:24:49.838557 27078 log.cc:927] Time spent T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.087s	user 0.000s	sys 0.000s
I20260709 12:24:49.838836 26741 leader_election.cc:290] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 26 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:49.820813 26883 leader_election.cc:290] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 22 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:49.833943 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:49.834754 17683 raft_consensus.cc:2393] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 26: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:24:49.785025 27035 raft_consensus.cc:740] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:24:49.835553 18368 raft_consensus.cc:2393] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 30: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:24:49.838300 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:49.826388 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:49.854934 17689 raft_consensus.cc:2393] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 31: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:24:49.832612 26962 raft_consensus.cc:2749] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:24:49.860570 17728 leader_election.cc:304] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:24:49.842649 26985 raft_consensus.cc:2749] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:24:49.846223 26719 leader_election.cc:290] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 25 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:49.872915 17728 leader_election.cc:304] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:49.857688 26800 leader_election.cc:290] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 31 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:49.883813 17801 leader_election.cc:304] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:49.857051 27027 leader_election.cc:290] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:24:49.913589 17682 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44898 (request call id 1799) took 1920 ms (client timeout 1710 ms). Trace:
I20260709 12:24:49.913758 17681 raft_consensus.cc:1240] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 22. Current term is 23. Ops: []
I20260709 12:24:49.915729 17685 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37402 (request call id 3748) took 2250 ms. Trace:
I20260709 12:24:49.921492 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:49.924232 17685 rpcz_store.cc:276] 0709 12:24:47.665583 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:47.826162 (+160579us) service_pool.cc:224] Handling call
0709 12:24:48.494721 (+668559us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:48.517963 (+ 23242us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:49.915450 (+1397487us) spinlock_profiling.cc:243] Waited 668 ms on lock 0x7b540014dcd0. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a61f8ad 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:24:49.915649 (+   199us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1536695552}
I20260709 12:24:49.925776 17689 raft_consensus.cc:2393] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 29: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
W20260709 12:24:49.927932 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 44219.8us
I20260709 12:24:49.928784 17801 leader_election.cc:400] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 26 pre-election: Vote denied by peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651) with higher term. Message: Invalid argument: T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 26. Current term is 27.
I20260709 12:24:49.929041 17801 leader_election.cc:403] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 26 pre-election: Cancelling election due to peer responding with higher term
W20260709 12:24:49.924026 17682 rpcz_store.cc:269] 0709 12:24:47.993597 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:48.264777 (+271180us) service_pool.cc:224] Handling call
0709 12:24:49.913408 (+1648631us) spinlock_profiling.cc:243] Waited 851 ms on lock 0x7b54000c4754. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:24:49.913529 (+   121us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1956875136}
W20260709 12:24:49.952692 27081 log.cc:927] Time spent T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.123s	user 0.000s	sys 0.000s
I20260709 12:24:49.960340 27035 consensus_queue.cc:260] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 27.21, Last appended by leader: 21, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:49.961541 27035 raft_consensus.cc:3060] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:24:49.952741 26910 leader_election.cc:290] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 25 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:49.964244 26985 raft_consensus.cc:3060] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Advancing to term 27
I20260709 12:24:49.964887 26587 leader_election.cc:290] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:24:49.970163 27037 consensus_meta.cc:287] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.971s	user 0.010s	sys 0.000s
I20260709 12:24:49.970750 27037 raft_consensus.cc:515] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:49.874550 26883 raft_consensus.cc:2804] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:24:49.977451 26883 raft_consensus.cc:697] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 22 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:24:49.978503 26883 consensus_queue.cc:237] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.13, Last appended by leader: 13, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:24:49.912396 27040 consensus_meta.cc:287] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.712s	user 0.009s	sys 0.003s
I20260709 12:24:49.986478 17801 leader_election.cc:304] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
W20260709 12:24:50.000540 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 211855us
I20260709 12:24:49.866956 26963 raft_consensus.cc:515] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:50.001004 18371 raft_consensus.cc:2468] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 24.
I20260709 12:24:50.001730 18333 leader_election.cc:304] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:24:49.921984 26982 raft_consensus.cc:2804] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:24:50.002144 17801 leader_election.cc:400] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 30 pre-election: Vote denied by peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651) with higher term. Message: Invalid argument: T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 30. Current term is 31.
I20260709 12:24:50.006929 17801 leader_election.cc:403] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 30 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:24:50.005266 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:49.950945 26848 raft_consensus.cc:3060] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:50.008876 17682 raft_consensus.cc:1275] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 21 index: 15. Preceding OpId from leader: term: 22 index: 16. (index mismatch)
I20260709 12:24:50.024420 18333 leader_election.cc:304] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:50.002537 26887 raft_consensus.cc:2804] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:24:50.020408 27008 raft_consensus.cc:2749] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Leader pre-election lost for term 30. Reason: Vote denied by peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651) with higher term. Message: Invalid argument: T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 30. Current term is 31.
I20260709 12:24:50.033174 17834 raft_consensus.cc:1240] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 27. Current term is 28. Ops: [27.21-27.21]
I20260709 12:24:50.035873 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:50.014043 27040 raft_consensus.cc:2804] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:24:50.040787 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 25 candidate_status { last_received { term: 23 index: 19 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:50.002791 26726 consensus_queue.cc:1048] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.030s
I20260709 12:24:50.043195 17803 leader_election.cc:304] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:50.052526 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 29 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:50.043619 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:50.053442 17833 raft_consensus.cc:2468] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 27.
I20260709 12:24:50.043233 17831 raft_consensus.cc:2468] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 24.
I20260709 12:24:50.044281 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:50.044931 17837 raft_consensus.cc:1240] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 24. Current term is 25. Ops: []
I20260709 12:24:50.050766 17727 leader_election.cc:304] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:50.057570 17803 leader_election.cc:304] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:50.059010 26910 raft_consensus.cc:2749] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:24:50.059545 17803 leader_election.cc:304] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:50.030169 26887 raft_consensus.cc:493] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:50.031250 26982 raft_consensus.cc:697] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
W20260709 12:24:50.152555 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:50.261487 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 202052us
W20260709 12:24:50.213496 27087 log.cc:927] Time spent T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.177s	user 0.000s	sys 0.001s
W20260709 12:24:50.253355 27001 consensus_meta.cc:287] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.874s	user 0.011s	sys 0.010s
I20260709 12:24:50.253204 18366 raft_consensus.cc:1240] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 24 LEADER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 23. Current term is 24. Ops: []
W20260709 12:24:50.253170 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 202587us
W20260709 12:24:50.254356 27033 consensus_meta.cc:287] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.859s	user 0.013s	sys 0.008s
I20260709 12:24:50.254438 26985 raft_consensus.cc:2749] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Leader pre-election lost for term 26. Reason: Vote denied by peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651) with higher term. Message: Invalid argument: T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 26. Current term is 27.
W20260709 12:24:50.255043 26599 consensus_meta.cc:287] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.747s	user 0.006s	sys 0.010s
I20260709 12:24:50.255231 27037 leader_election.cc:290] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 27 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:50.054755 27040 raft_consensus.cc:697] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 20 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:24:50.301846 17688 raft_consensus.cc:1275] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 22 index: 16. Preceding OpId from leader: term: 24 index: 17. (index mismatch)
W20260709 12:24:50.257941 26957 consensus_meta.cc:287] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.562s	user 0.012s	sys 0.000s
I20260709 12:24:50.258028 27094 raft_consensus.cc:493] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:50.259274 26726 raft_consensus.cc:493] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:50.307495 26726 raft_consensus.cc:515] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:50.302541 26848 raft_consensus.cc:2749] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:24:50.309712 26726 leader_election.cc:290] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:50.312112 27094 raft_consensus.cc:515] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:50.314477 27094 leader_election.cc:290] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:50.304374 27093 raft_consensus.cc:493] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:50.315505 27093 raft_consensus.cc:515] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:50.316422 27030 raft_consensus.cc:493] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:50.316921 27030 raft_consensus.cc:515] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:50.317483 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:50.317636 27093 leader_election.cc:290] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:50.318029 18365 raft_consensus.cc:3060] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:24:50.318830 27030 leader_election.cc:290] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:50.276727 27033 raft_consensus.cc:515] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:50.321511 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:50.322269 17759 raft_consensus.cc:2393] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 24: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:24:50.323309 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:50.324187 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 23 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:50.327795 17830 raft_consensus.cc:2468] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 27.
I20260709 12:24:50.328794 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 31 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:50.329434 17688 raft_consensus.cc:2393] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 31: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:24:50.259620 26963 leader_election.cc:290] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 28 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:50.259755 17760 raft_consensus.cc:1275] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 26 index: 15. Preceding OpId from leader: term: 30 index: 18. (index mismatch)
I20260709 12:24:50.033316 27039 raft_consensus.cc:493] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:50.339435 17836 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35042 (request call id 3355) took 1073 ms. Trace:
I20260709 12:24:50.338953 26789 raft_consensus.cc:493] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:50.340031 17836 rpcz_store.cc:276] 0709 12:24:49.265785 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:49.323190 (+ 57405us) service_pool.cc:224] Handling call
0709 12:24:50.339273 (+1016083us) spinlock_profiling.cc:243] Waited 432 ms on lock 0x7b54003354d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:24:50.339379 (+   106us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":993777536}
I20260709 12:24:50.340530 26789 raft_consensus.cc:515] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:50.341318 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:50.342492 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:50.018055 26587 consensus_queue.cc:1059] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 21.15, Next index: 16, Last known committed idx: 13, Time since last communication: 0.029s
I20260709 12:24:50.260720 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
W20260709 12:24:50.260893 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 333986us
I20260709 12:24:50.344269 17756 raft_consensus.cc:3060] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:24:50.344885 17728 leader_election.cc:400] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 23 pre-election: Vote denied by peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071) with higher term. Message: Invalid argument: T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for earlier term 23. Current term is 24.
I20260709 12:24:50.345227 17728 leader_election.cc:403] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 23 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:24:50.346498 17728 leader_election.cc:304] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:50.041131 27027 raft_consensus.cc:2804] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:24:50.348059 27027 raft_consensus.cc:493] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:50.348485 27027 raft_consensus.cc:3060] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:50.349123 26354 raft_consensus.cc:2749] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Leader pre-election lost for term 23. Reason: Vote denied by peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071) with higher term. Message: Invalid argument: T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for earlier term 23. Current term is 24.
I20260709 12:24:50.350224 26354 raft_consensus.cc:2804] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:24:50.350571 26354 raft_consensus.cc:493] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:50.350903 26354 raft_consensus.cc:3060] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:50.351531 17756 raft_consensus.cc:2468] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 28.
I20260709 12:24:50.262380 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:50.262307 17803 leader_election.cc:304] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:50.304628 26966 raft_consensus.cc:493] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:50.262197 17763 raft_consensus.cc:1275] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 23 index: 16. Preceding OpId from leader: term: 27 index: 18. (index mismatch)
W20260709 12:24:50.353041 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 90812.7us
I20260709 12:24:50.263401 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:50.264513 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
W20260709 12:24:50.265393 26818 consensus_meta.cc:287] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.675s	user 0.014s	sys 0.008s
I20260709 12:24:50.354101 27033 leader_election.cc:290] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 31 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:50.354101 18361 raft_consensus.cc:3060] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:50.060024 26910 raft_consensus.cc:2749] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:24:50.271698 18333 leader_election.cc:304] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:50.281571 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:50.264003 27008 raft_consensus.cc:2749] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:24:50.261641 26887 raft_consensus.cc:3060] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:50.264170 27028 raft_consensus.cc:2749] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:24:50.300302 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:50.357816 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:50.358559 17689 raft_consensus.cc:2393] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 25: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:24:50.359372 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:50.275959 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:50.359921 27008 consensus_queue.cc:1048] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.000s
I20260709 12:24:50.338768 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 27 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:50.360766 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 31 candidate_status { last_received { term: 30 index: 23 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:50.361018 17831 raft_consensus.cc:2468] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 26.
I20260709 12:24:50.361372 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:50.361850 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:50.362361 27051 leader_election.cc:290] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:50.354429 26818 raft_consensus.cc:515] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:50.363037 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 31 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:50.338924 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:50.364184 18362 raft_consensus.cc:2393] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 28: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:24:50.338518 27062 raft_consensus.cc:493] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:50.364861 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
W20260709 12:24:50.365226 27074 log.cc:927] Time spent T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.608s	user 0.003s	sys 0.000s
I20260709 12:24:50.364914 27062 raft_consensus.cc:515] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:50.365751 17833 raft_consensus.cc:2393] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 25: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:24:50.338145 18365 raft_consensus.cc:2468] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 31.
I20260709 12:24:50.352797 26966 raft_consensus.cc:515] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:50.270308 17727 leader_election.cc:304] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:50.366685 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:50.339507 27039 raft_consensus.cc:515] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:50.367168 27062 leader_election.cc:290] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:24:50.367300 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 97090us
I20260709 12:24:50.361882 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:50.368430 18368 raft_consensus.cc:3060] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:50.265357 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 29 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:50.342690 26789 leader_election.cc:290] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:50.271119 18363 raft_consensus.cc:1275] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 24 index: 19. Preceding OpId from leader: term: 26 index: 20. (index mismatch)
I20260709 12:24:50.301466 26599 raft_consensus.cc:515] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:50.353813 17728 leader_election.cc:304] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:50.376024 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:50.355933 26910 raft_consensus.cc:2749] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:24:50.262300 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 23 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:50.369127 17758 raft_consensus.cc:2468] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 28.
I20260709 12:24:50.369390 17829 raft_consensus.cc:2393] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 24: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:24:50.363754 17831 raft_consensus.cc:2393] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 31: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:24:50.369825 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:50.353825 17803 leader_election.cc:400] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 26 pre-election: Vote denied by peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577) with higher term. Message: Invalid argument: T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 26. Current term is 27.
I20260709 12:24:50.392204 17834 raft_consensus.cc:2468] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 22.
I20260709 12:24:50.394673 17803 leader_election.cc:304] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:50.391649 18334 leader_election.cc:304] T a5732e71b9d54f8786b8c511d69943bf P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:50.396929 17803 leader_election.cc:304] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:50.397452 18334 leader_election.cc:304] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:50.397871 27062 raft_consensus.cc:2749] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 12:24:50.369504 17836 raft_consensus.cc:2393] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 27: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:24:50.401934 17836 raft_consensus.cc:1275] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 20 index: 14. Preceding OpId from leader: term: 22 index: 15. (index mismatch)
I20260709 12:24:50.404019 27088 raft_consensus.cc:2749] T a5732e71b9d54f8786b8c511d69943bf P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:24:50.405005 27088 raft_consensus.cc:2749] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:24:50.409373 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:50.372241 27030 raft_consensus.cc:2749] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:24:50.365623 27027 raft_consensus.cc:515] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:50.373454 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:50.416285 27062 raft_consensus.cc:2749] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:24:50.356352 17682 raft_consensus.cc:2393] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 27: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:24:50.422472 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:50.423292 17680 raft_consensus.cc:2393] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 23: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
W20260709 12:24:50.374657 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 103022us
I20260709 12:24:50.371778 18361 raft_consensus.cc:2468] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 25.
I20260709 12:24:50.424167 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:50.424557 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 25 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:50.424845 17680 raft_consensus.cc:2393] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 24: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:24:50.425559 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 25 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:50.427543 26966 leader_election.cc:290] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:50.428325 26818 leader_election.cc:290] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 28 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:50.429271 27040 consensus_queue.cc:237] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 19.14, Last appended by leader: 14, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:50.325258 26957 raft_consensus.cc:515] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:50.433285 17681 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37402 (request call id 3751) took 2762 ms. Trace:
I20260709 12:24:50.433416 26957 leader_election.cc:290] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 31 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:50.433835 18363 raft_consensus.cc:1275] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 20 index: 14. Preceding OpId from leader: term: 22 index: 15. (index mismatch)
I20260709 12:24:50.433790 17681 rpcz_store.cc:276] 0709 12:24:47.670816 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:48.562963 (+892147us) service_pool.cc:224] Handling call
0709 12:24:49.913559 (+1350596us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:49.926346 (+ 12787us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:50.433074 (+506728us) spinlock_profiling.cc:243] Waited 1.35 s on lock 0x7b54000c4750. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a61f8ad 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:24:50.433218 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3104602368}
I20260709 12:24:50.425559 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:50.436622 26599 leader_election.cc:290] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 21 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:50.436779 17680 raft_consensus.cc:2468] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 22.
I20260709 12:24:50.377763 26978 raft_consensus.cc:2749] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:24:50.423056 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:50.437717 17682 raft_consensus.cc:2468] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 26.
I20260709 12:24:50.377792 27035 raft_consensus.cc:2749] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
I20260709 12:24:50.438725 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 25 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:50.378733 17688 raft_consensus.cc:2468] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 29.
I20260709 12:24:50.439605 17803 leader_election.cc:304] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:50.362097 17689 raft_consensus.cc:2393] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 28: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:24:50.415277 17832 raft_consensus.cc:2393] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 28: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:24:50.440538 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:50.441200 17688 raft_consensus.cc:3060] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:24:50.368765 17727 leader_election.cc:304] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:50.393232 26962 raft_consensus.cc:493] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:50.441857 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 29 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:24:50.442473 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 73783.3us
I20260709 12:24:50.390182 27033 raft_consensus.cc:493] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:50.444748 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 25 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:50.445746 17681 raft_consensus.cc:2468] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 24.
I20260709 12:24:50.439695 17682 raft_consensus.cc:2410] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for term 25 because replica has last-logged OpId of term: 23 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 17.
I20260709 12:24:50.448369 17802 leader_election.cc:304] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:50.443691 17727 leader_election.cc:304] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:50.449770 18362 raft_consensus.cc:1240] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 23. Current term is 24. Ops: []
I20260709 12:24:50.457345 17681 raft_consensus.cc:1275] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 21 index: 18. Preceding OpId from leader: term: 26 index: 20. (index mismatch)
I20260709 12:24:50.451056 17803 leader_election.cc:304] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:24:50.464857 17727 leader_election.cc:304] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:24:50.416541 27039 leader_election.cc:290] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:50.422571 26581 leader_election.cc:290] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 27 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:50.473990 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:50.472647 17728 leader_election.cc:304] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:50.522962 17682 raft_consensus.cc:3060] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:50.477331 17727 leader_election.cc:304] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
W20260709 12:24:50.525708 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 53155us
I20260709 12:24:50.527963 18333 leader_election.cc:304] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:24:50.526423 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:50.446462 26978 raft_consensus.cc:2804] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:24:50.529112 17680 raft_consensus.cc:2393] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 31: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:24:50.529466 18333 leader_election.cc:304] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:24:50.530377 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 53156.9us
I20260709 12:24:50.532200 17689 raft_consensus.cc:1275] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 28 index: 16. Preceding OpId from leader: term: 29 index: 17. (index mismatch)
I20260709 12:24:50.530738 18333 leader_election.cc:304] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:50.535094 26883 raft_consensus.cc:2804] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:24:50.535203 26818 consensus_queue.cc:1059] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 24.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
W20260709 12:24:50.537395 27091 log.cc:927] Time spent T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.093s	user 0.000s	sys 0.000s
I20260709 12:24:50.538094 26818 raft_consensus.cc:2749] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:24:50.538756 26818 raft_consensus.cc:493] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:50.535686 26883 raft_consensus.cc:493] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:24:50.536157 27084 raft_consensus.cc:2804] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:24:50.539764 27084 raft_consensus.cc:493] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:50.539438 26818 raft_consensus.cc:515] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:50.540300 27084 raft_consensus.cc:3060] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:50.542941 26818 leader_election.cc:290] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:50.536264 26978 raft_consensus.cc:697] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:24:50.476078 26726 raft_consensus.cc:2804] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:24:50.535528 27088 raft_consensus.cc:2804] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:24:50.544742 27088 raft_consensus.cc:493] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:50.545149 27088 raft_consensus.cc:3060] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:50.544996 26978 consensus_queue.cc:237] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 26.20, Last appended by leader: 20, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:50.548115 26883 raft_consensus.cc:3060] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:24:50.548002 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:50.550976 17728 leader_election.cc:304] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:50.552919 27094 raft_consensus.cc:2804] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:24:50.552675 27088 raft_consensus.cc:515] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:50.553404 27094 raft_consensus.cc:493] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:50.553848 27094 raft_consensus.cc:3060] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:50.378925 26982 consensus_queue.cc:237] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 26.18, Last appended by leader: 18, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:50.541011 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:50.559739 17680 raft_consensus.cc:1275] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 25 index: 18. Preceding OpId from leader: term: 26 index: 19. (index mismatch)
I20260709 12:24:50.562196 17727 leader_election.cc:304] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:50.475713 26888 raft_consensus.cc:2804] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:24:50.566406 17680 raft_consensus.cc:1240] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 30. Current term is 31. Ops: []
I20260709 12:24:50.475768 27030 raft_consensus.cc:2749] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:24:50.574903 17801 leader_election.cc:304] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
W20260709 12:24:50.581277 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 183877us
I20260709 12:24:50.527259 26844 raft_consensus.cc:2804] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:24:50.588152 17801 leader_election.cc:304] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:50.599007 18334 leader_election.cc:304] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:50.604842 17680 raft_consensus.cc:1275] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 29 index: 17. Preceding OpId from leader: term: 30 index: 18. (index mismatch)
I20260709 12:24:50.605851 18334 leader_election.cc:304] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:50.560526 26726 raft_consensus.cc:697] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 28 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:24:50.449285 27033 raft_consensus.cc:515] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:50.629205 17682 raft_consensus.cc:2468] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 21.
I20260709 12:24:50.630224 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 23 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:50.573474 26837 raft_consensus.cc:2749] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 12:24:50.468685 26910 consensus_queue.cc:1048] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 17, Time since last communication: 0.129s
I20260709 12:24:50.449736 26962 raft_consensus.cc:515] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:50.645160 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:50.605751 26985 raft_consensus.cc:2804] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:24:50.626888 26635 raft_consensus.cc:3060] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:50.658139 17687 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44898 (request call id 1798) took 2665 ms. Trace:
I20260709 12:24:50.658547 17687 rpcz_store.cc:276] 0709 12:24:47.993066 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:48.598412 (+605346us) service_pool.cc:224] Handling call
0709 12:24:48.598671 (+   259us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:48.850884 (+252213us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 12:24:48.948435 (+ 97551us) log.cc:844] Serialized 14 byte log entry
0709 12:24:49.201480 (+253045us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:49.201535 (+    55us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:24:49.201582 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:49.860772 (+659190us) spinlock_profiling.cc:243] Waited 3.63 ms on lock 0x7b1800477b20. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b475012a6 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419
0709 12:24:50.657884 (+797112us) spinlock_profiling.cc:243] Waited 252 ms on lock 0x7b5400207954. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:24:50.657941 (+    57us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:50.658060 (+   119us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":6835,"prepare.queue_time_us":142128,"prepare.run_cpu_time_us":50,"prepare.run_wall_time_us":1673,"raft.queue_time_us":135,"raft.run_cpu_time_us":131,"raft.run_wall_time_us":10406,"spinlock_wait_cycles":587890944,"thread_start_us":345933,"threads_started":2,"wal-append.queue_time_us":450970,"wal-append.run_cpu_time_us":1453,"wal-append.run_wall_time_us":1059410}
I20260709 12:24:50.609078 26848 raft_consensus.cc:2804] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:24:50.662945 26848 raft_consensus.cc:493] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:50.663484 26848 raft_consensus.cc:3060] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:24:50.627250 27012 raft_consensus.cc:493] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:50.665169 27012 raft_consensus.cc:515] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:50.600804 26844 raft_consensus.cc:493] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:24:50.605902 26587 consensus_queue.cc:1048] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.053s
I20260709 12:24:50.605329 26789 consensus_queue.cc:1048] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.050s
I20260709 12:24:50.675262 26354 raft_consensus.cc:515] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:50.681340 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:50.626850 26924 consensus_queue.cc:1048] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.048s
I20260709 12:24:50.652987 27028 raft_consensus.cc:2804] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:24:50.687041 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:50.653190 27088 leader_election.cc:290] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 25 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:50.603713 27039 consensus_queue.cc:1048] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 15, Time since last communication: 0.069s
I20260709 12:24:50.688983 26354 leader_election.cc:290] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 27 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:50.676470 26844 raft_consensus.cc:3060] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Advancing to term 29
W20260709 12:24:50.696316 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:50.699252 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 224602us
W20260709 12:24:50.700099 17654 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.16.29.68:42577 timed out after 1.585s (SENT) [suppressed 18 similar messages]
I20260709 12:24:50.680371 26985 raft_consensus.cc:697] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 28 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:24:50.690838 27027 leader_election.cc:290] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 24 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:50.722726 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:50.724310 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:50.726614 17687 raft_consensus.cc:2393] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 27: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:24:50.727741 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:50.733346 17834 raft_consensus.cc:1240] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 21. Current term is 22. Ops: []
I20260709 12:24:50.729533 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 25 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:50.739161 17832 raft_consensus.cc:3060] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:50.746945 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 25 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
W20260709 12:24:50.752856 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 286297us
I20260709 12:24:50.761284 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:50.738596 27035 raft_consensus.cc:493] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:50.745868 27012 leader_election.cc:290] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:50.769829 17802 leader_election.cc:304] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:24:50.763451 26888 raft_consensus.cc:697] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 25 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:24:50.746163 27088 consensus_queue.cc:1048] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.040s
I20260709 12:24:50.695996 26726 consensus_queue.cc:237] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 26.15, Last appended by leader: 15, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:50.804832 18368 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 26.
W20260709 12:24:50.804513 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 102007us
I20260709 12:24:50.807785 17688 raft_consensus.cc:2468] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 28.
I20260709 12:24:50.810525 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:50.812552 18334 leader_election.cc:304] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:50.814215 18363 raft_consensus.cc:1240] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 25. Current term is 26. Ops: []
I20260709 12:24:50.808712 26789 consensus_queue.cc:1059] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 22.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.008s
I20260709 12:24:50.816545 26888 consensus_queue.cc:237] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 23.17, Last appended by leader: 17, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:50.818949 26887 raft_consensus.cc:515] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:50.820317 17833 raft_consensus.cc:2393] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 26: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:24:50.820815 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 23 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:50.821899 26883 raft_consensus.cc:515] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:50.822640 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 29 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:50.823478 18368 raft_consensus.cc:2410] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 29 because replica has last-logged OpId of term: 27 index: 21, which is greater than that of the candidate, which has last-logged OpId of term: 26 index: 20.
I20260709 12:24:50.824281 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:50.824950 18365 raft_consensus.cc:2468] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 23.
I20260709 12:24:50.826108 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:50.827309 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
W20260709 12:24:50.828280 27099 log.cc:927] Time spent T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.134s	user 0.001s	sys 0.000s
I20260709 12:24:50.827337 17803 leader_election.cc:304] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:50.830560 26887 leader_election.cc:290] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 26 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:50.809077 26843 leader_election.cc:290] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:50.832020 26883 leader_election.cc:290] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 29 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:50.833514 27004 leader_election.cc:290] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:50.834658 27033 leader_election.cc:290] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:50.837350 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:50.784756 27035 raft_consensus.cc:515] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:50.790010 27107 consensus_queue.cc:1059] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 30.23, Next index: 24, Last known committed idx: 23, Time since last communication: 0.041s
W20260709 12:24:50.839949 27095 log.cc:927] Time spent T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.183s	user 0.001s	sys 0.000s
I20260709 12:24:50.861212 18365 raft_consensus.cc:2468] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 21.
W20260709 12:24:50.886204 17652 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:50.887521 17652 leader_election.cc:304] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:50.962216 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 28 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:50.972489 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:50.978382 18361 raft_consensus.cc:2393] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 25: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:24:50.845805 26966 raft_consensus.cc:2749] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:24:50.835289 27102 consensus_queue.cc:1059] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 21.15, Next index: 16, Last known committed idx: 13, Time since last communication: 0.152s
I20260709 12:24:50.952060 26635 raft_consensus.cc:2749] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:24:50.808447 26985 consensus_queue.cc:237] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 26.19, Last appended by leader: 19, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:50.952235 27012 raft_consensus.cc:493] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:24:50.952499 27044 raft_consensus.cc:493] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:24:50.969652 27106 raft_consensus.cc:3055] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:24:50.960768 26587 raft_consensus.cc:2804] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Leader election won for term 28
W20260709 12:24:51.005062 27098 log.cc:927] Time spent T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.315s	user 0.002s	sys 0.000s
I20260709 12:24:50.980509 27033 raft_consensus.cc:493] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:24:51.000844 26962 leader_election.cc:290] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:51.033825 27012 raft_consensus.cc:515] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:51.029435 27106 raft_consensus.cc:740] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 30 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:24:51.052073 17686 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44898 (request call id 1805) took 1514 ms. Trace:
I20260709 12:24:51.052497 17686 rpcz_store.cc:276] 0709 12:24:49.537748 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:49.537920 (+   172us) service_pool.cc:224] Handling call
0709 12:24:49.538426 (+   506us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:49.837579 (+299153us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:51.051817 (+1214238us) spinlock_profiling.cc:243] Waited 281 ms on lock 0x7b54003480d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:24:51.052013 (+   196us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":647408512}
W20260709 12:24:51.064728 27108 log.cc:927] Time spent T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.062s	user 0.000s	sys 0.000s
I20260709 12:24:51.058960 27119 raft_consensus.cc:493] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:24:51.067747 27119 raft_consensus.cc:515] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:51.070684 27119 leader_election.cc:290] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:51.071424 27119 raft_consensus.cc:2804] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Leader election won for term 27
W20260709 12:24:51.080437 27112 log.cc:927] Time spent T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.096s	user 0.001s	sys 0.000s
I20260709 12:24:51.081072 17832 raft_consensus.cc:2468] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 25.
I20260709 12:24:51.081729 27119 raft_consensus.cc:697] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 27 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:24:51.089828 27119 consensus_queue.cc:237] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 25.16, Last appended by leader: 16, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:51.087117 27035 leader_election.cc:290] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:24:51.085861 27084 consensus_meta.cc:287] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.545s	user 0.006s	sys 0.006s
I20260709 12:24:51.041455 26632 raft_consensus.cc:493] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:51.099056 27100 log.cc:927] Time spent T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.302s	user 0.001s	sys 0.000s
I20260709 12:24:51.091843 27084 raft_consensus.cc:515] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:51.112356 27101 log.cc:927] Time spent T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.315s	user 0.001s	sys 0.000s
I20260709 12:24:51.149660 18374 raft_consensus.cc:2393] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 23: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:24:51.033007 27044 raft_consensus.cc:515] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:51.146474 27084 leader_election.cc:290] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 23 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:51.155517 27044 leader_election.cc:290] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:24:51.056550 27094 consensus_meta.cc:287] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.502s	user 0.013s	sys 0.000s
W20260709 12:24:51.171429 17835 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37994 (request call id 1712) took 2315 ms (client timeout 3039 ms). Trace:
I20260709 12:24:51.174096 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:51.175244 17762 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36572 (request call id 1630) took 1492 ms. Trace:
W20260709 12:24:51.177340 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:51.172221 17835 rpcz_store.cc:269] 0709 12:24:48.855929 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:49.136484 (+280555us) service_pool.cc:224] Handling call
0709 12:24:51.171139 (+2034655us) spinlock_profiling.cc:243] Waited 387 ms on lock 0x7b5400158bd4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:24:51.171348 (+   209us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":890789632}
W20260709 12:24:51.155648 27110 log.cc:927] Time spent T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.152s	user 0.001s	sys 0.001s
I20260709 12:24:51.175680 17762 rpcz_store.cc:276] 0709 12:24:49.682763 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:49.682904 (+   141us) service_pool.cc:224] Handling call
0709 12:24:51.173877 (+1490973us) spinlock_profiling.cc:243] Waited 1.86 ms on lock 0x7b5800033398. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b474fbb71 00007f7b474fba22 00007f7b504148d9 00007f7b4f56eee0 00007f7b4f6380ab 00007f7b4f638131 00007f7b4f5e60a0 00007f7b4f5e5984 00007f7b48b606d4 00007f7b48b60662
0709 12:24:51.175166 (+  1289us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4272512}
I20260709 12:24:51.045600 27033 raft_consensus.cc:515] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:51.208153 27075 raft_consensus.cc:3055] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:24:51.208729 27075 raft_consensus.cc:740] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:24:51.209870 27075 consensus_queue.cc:260] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 24.18, Last appended by leader: 18, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:51.211516 27075 raft_consensus.cc:3060] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:51.219774 18366 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 2658) took 1731 ms. Trace:
I20260709 12:24:51.220495 18366 rpcz_store.cc:276] 0709 12:24:49.488166 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:49.488279 (+   113us) service_pool.cc:224] Handling call
0709 12:24:49.488557 (+   278us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:50.271699 (+783142us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:51.219436 (+947737us) spinlock_profiling.cc:243] Waited 764 ms on lock 0x7b5400724054. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:24:51.219687 (+   251us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1757993856}
I20260709 12:24:51.228987 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:24:51.230382 17685 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44898 (request call id 1813) took 1662 ms (client timeout 2181 ms). Trace:
W20260709 12:24:51.233116 27083 log.cc:927] Time spent T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 1.416s	user 0.004s	sys 0.007s
I20260709 12:24:51.235060 27127 raft_consensus.cc:493] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:51.238140 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
W20260709 12:24:51.231132 17685 rpcz_store.cc:269] 0709 12:24:49.567847 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:49.926098 (+358251us) service_pool.cc:224] Handling call
0709 12:24:51.228766 (+1302668us) spinlock_profiling.cc:243] Waited 6.33 ms on lock 0x7b5800002a98. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b474fbb71 00007f7b474fba22 00007f7b504148d9 00007f7b4f56eee0 00007f7b4f6380ab 00007f7b4f638131 00007f7b4f5e60a0 00007f7b4f5e5984 00007f7b48b606d4 00007f7b48b60662
0709 12:24:51.230303 (+  1537us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":14563200}
I20260709 12:24:51.159094 26632 raft_consensus.cc:515] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:51.243958 17683 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37402 (request call id 3763) took 2286 ms (client timeout 2608 ms). Trace:
W20260709 12:24:51.252275 17683 rpcz_store.cc:269] 0709 12:24:48.957646 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:49.021507 (+ 63861us) service_pool.cc:224] Handling call
0709 12:24:51.243580 (+2222073us) spinlock_profiling.cc:243] Waited 806 ms on lock 0x7b54004259d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:24:51.243818 (+   238us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1854491392}
I20260709 12:24:51.244834 17689 raft_consensus.cc:1240] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 25. Current term is 26. Ops: []
I20260709 12:24:51.193958 27106 consensus_queue.cc:260] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 30.23, Last appended by leader: 23, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:51.178025 27094 raft_consensus.cc:515] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:51.254987 26866 raft_consensus.cc:493] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:51.288223 27094 leader_election.cc:290] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 27 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:24:51.288323 27125 log.cc:927] Time spent T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.087s	user 0.001s	sys 0.000s
I20260709 12:24:51.290817 27012 leader_election.cc:290] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:51.235965 27127 raft_consensus.cc:515] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:51.278079 27106 raft_consensus.cc:3060] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:24:51.281885 26866 raft_consensus.cc:515] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:24:51.289753 18371 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59918 (request call id 2666) took 1797 ms (client timeout 2219 ms). Trace:
W20260709 12:24:51.294486 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 407106us
I20260709 12:24:51.290048 27033 leader_election.cc:290] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:24:51.320820 18371 rpcz_store.cc:269] 0709 12:24:49.492269 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:49.584841 (+ 92572us) service_pool.cc:224] Handling call
0709 12:24:51.289516 (+1704675us) spinlock_profiling.cc:243] Waited 408 ms on lock 0x7b54006af9d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:24:51.289673 (+   157us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":937736064}
I20260709 12:24:51.321488 17652 leader_election.cc:304] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:51.322644 17652 leader_election.cc:400] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 27 pre-election: Vote denied by peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837) with higher term. Message: Invalid argument: T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for earlier term 27. Current term is 28.
I20260709 12:24:51.322937 17652 leader_election.cc:403] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 27 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:24:51.323558 27012 raft_consensus.cc:2804] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:24:51.324584 17652 leader_election.cc:304] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:24:51.300601 26844 consensus_meta.cc:287] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.584s	user 0.006s	sys 0.005s
I20260709 12:24:51.336889 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:51.339191 26866 leader_election.cc:290] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:51.340472 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
W20260709 12:24:51.329160 26848 consensus_meta.cc:287] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.665s	user 0.007s	sys 0.007s
I20260709 12:24:51.326562 27012 raft_consensus.cc:697] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 24 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
W20260709 12:24:51.348825 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:51.328482 27044 raft_consensus.cc:2749] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Leader pre-election lost for term 27. Reason: Vote denied by peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837) with higher term. Message: Invalid argument: T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for earlier term 27. Current term is 28.
I20260709 12:24:51.357403 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:51.361249 18362 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59886 (request call id 2494) took 1052 ms. Trace:
I20260709 12:24:51.361702 18362 rpcz_store.cc:276] 0709 12:24:50.309204 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:50.429951 (+120747us) service_pool.cc:224] Handling call
0709 12:24:50.430194 (+   243us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:50.456343 (+ 26149us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:51.360975 (+904632us) spinlock_profiling.cc:243] Waited 19.4 ms on lock 0x7b540051aed4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:24:51.361185 (+   210us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":44598016}
I20260709 12:24:51.333670 27033 consensus_queue.cc:1048] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 13, Time since last communication: 0.024s
I20260709 12:24:51.366137 27012 consensus_queue.cc:237] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 22.12, Last appended by leader: 12, Current term: 24, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:51.371444 27127 leader_election.cc:290] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:51.344254 27049 raft_consensus.cc:2804] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:24:51.376240 27035 consensus_queue.cc:1059] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 25.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20260709 12:24:51.377591 27084 raft_consensus.cc:493] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:51.378196 27107 raft_consensus.cc:3055] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:24:51.378139 27084 raft_consensus.cc:515] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:51.378700 27107 raft_consensus.cc:740] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:24:51.380538 27084 leader_election.cc:290] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:24:51.381423 27120 log.cc:927] Time spent T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.315s	user 0.003s	sys 0.000s
W20260709 12:24:51.383571 27121 log.cc:927] Time spent T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.307s	user 0.000s	sys 0.001s
I20260709 12:24:51.388037 17764 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38786 (request call id 3981) took 2050 ms. Trace:
I20260709 12:24:51.388904 17689 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37396 (request call id 3333) took 1010 ms. Trace:
I20260709 12:24:51.389382 17689 rpcz_store.cc:276] 0709 12:24:50.378528 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:50.534030 (+155502us) service_pool.cc:224] Handling call
0709 12:24:51.244569 (+710539us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:51.254158 (+  9589us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:51.388707 (+134549us) spinlock_profiling.cc:243] Waited 710 ms on lock 0x7b54004259d0. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a61f8ad 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:24:51.388846 (+   139us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1633431424}
I20260709 12:24:51.392094 27028 raft_consensus.cc:493] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:24:51.392589 27028 raft_consensus.cc:3060] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:51.396906 27093 consensus_queue.cc:1048] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20260709 12:24:51.399207 26963 raft_consensus.cc:3055] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:24:51.399669 26963 raft_consensus.cc:740] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
W20260709 12:24:51.400050 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 791505us
I20260709 12:24:51.400869 17727 leader_election.cc:304] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 31 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:51.404167 27097 raft_consensus.cc:2749] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Leader election lost for term 31. Reason: could not achieve majority
I20260709 12:24:51.403445 17727 leader_election.cc:304] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
W20260709 12:24:51.406132 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 686067us
I20260709 12:24:51.408947 26883 raft_consensus.cc:2749] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260709 12:24:51.409109 26978 raft_consensus.cc:493] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:51.409245 17728 leader_election.cc:304] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
W20260709 12:24:51.356299 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:51.410135 17755 raft_consensus.cc:1240] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 23. Current term is 24. Ops: [23.18-23.18]
I20260709 12:24:51.356367 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:51.410851 18367 raft_consensus.cc:2468] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 24.
I20260709 12:24:51.412081 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:51.356730 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:51.413960 18365 raft_consensus.cc:2376] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 31. Re-sending same reply.
I20260709 12:24:51.417706 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:51.418223 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 25 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:51.419718 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:51.420792 17756 raft_consensus.cc:2410] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for term 30 because replica has last-logged OpId of term: 29 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 28 index: 16.
I20260709 12:24:51.422082 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 25 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:51.424310 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 27 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:51.425486 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:51.427089 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:51.429482 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 24 candidate_status { last_received { term: 23 index: 19 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:51.430272 18367 raft_consensus.cc:2468] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 23.
I20260709 12:24:51.426237 18363 raft_consensus.cc:2376] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 24. Re-sending same reply.
I20260709 12:24:51.437620 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 22 candidate_status { last_received { term: 21 index: 20 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:51.438503 18363 raft_consensus.cc:2468] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 21.
I20260709 12:24:51.415371 27049 raft_consensus.cc:697] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 31 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:24:51.442416 27049 consensus_queue.cc:237] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 29.19, Last appended by leader: 19, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:51.334345 26844 raft_consensus.cc:515] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:51.415498 27044 raft_consensus.cc:493] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:24:51.445564 17763 raft_consensus.cc:2468] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 29.
I20260709 12:24:51.388517 17764 rpcz_store.cc:276] 0709 12:24:49.337936 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:49.350762 (+ 12826us) service_pool.cc:224] Handling call
0709 12:24:49.351202 (+   440us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:49.351732 (+   530us) raft_consensus.cc:1534] Early marking committed up to index 16
0709 12:24:50.368217 (+1016485us) spinlock_profiling.cc:243] Waited 8.47 ms on lock 0x7b1800428140. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b4a1c84bc 00007f7b4751648b 00007f7b503709a9 00007f7b4753269b 00007f7b4753093f 00007f7b47530455 00007f7b4e0183c7 00007f7b4a62a9a7 00007f7b4a63a0bf
0709 12:24:50.368346 (+   129us) log.cc:844] Serialized 14 byte log entry
0709 12:24:50.602659 (+234313us) log.cc:844] Serialized 14 byte log entry
0709 12:24:51.386026 (+783367us) spinlock_profiling.cc:243] Waited 3.66 ms on lock 0x7b1800428140. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b474adc0a 00007f7b50bed219 00007f7b47535a40 00007f7b475304e4 00007f7b47530455 00007f7b4a618060 00007f7b4a62a716 00007f7b4a63b287 00007f7b4a63b208
0709 12:24:51.386175 (+   149us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:51.386697 (+   522us) log.cc:844] Serialized 26 byte log entry
0709 12:24:51.386847 (+   150us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:24:51.386884 (+    37us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:51.386954 (+    70us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:51.387790 (+   836us) raft_consensus.cc:1685] finished
0709 12:24:51.387809 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:51.387964 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2135,"prepare.queue_time_us":276135,"prepare.run_cpu_time_us":199,"prepare.run_wall_time_us":201,"raft.queue_time_us":797594,"raft.run_cpu_time_us":380,"raft.run_wall_time_us":31276,"spinlock_wait_cycles":27910912,"thread_start_us":516750,"threads_started":2,"wal-append.queue_time_us":364755}
I20260709 12:24:51.448169 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:51.451653 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:51.452689 18374 raft_consensus.cc:2393] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 27: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:24:51.453579 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:51.456368 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:51.450771 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 24 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:51.458207 18371 raft_consensus.cc:2468] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 23.
I20260709 12:24:51.458844 18362 raft_consensus.cc:2410] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for term 24 because replica has last-logged OpId of term: 23 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 14.
I20260709 12:24:51.459424 18367 raft_consensus.cc:2393] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 31: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:24:51.463379 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
W20260709 12:24:51.463599 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 54424.7us
I20260709 12:24:51.464496 17760 raft_consensus.cc:2393] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 31: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:24:51.471715 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:51.482151 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 31 candidate_status { last_received { term: 30 index: 23 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:51.444569 27044 raft_consensus.cc:515] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:51.491714 17757 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36572 (request call id 1623) took 2619 ms. Trace:
I20260709 12:24:51.506894 17729 leader_election.cc:304] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:51.379604 27107 consensus_queue.cc:260] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 25.16, Last appended by leader: 16, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:51.491495 17764 raft_consensus.cc:2393] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 27: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:24:51.355264 26848 raft_consensus.cc:515] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:51.506825 17757 rpcz_store.cc:276] 0709 12:24:48.861063 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:49.178259 (+317196us) service_pool.cc:224] Handling call
0709 12:24:49.178496 (+   237us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:49.178680 (+   184us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 12:24:49.242324 (+ 63644us) log.cc:844] Serialized 14 byte log entry
0709 12:24:50.766703 (+1524379us) spinlock_profiling.cc:243] Waited 3.08 ms on lock 0x7b180016d320. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b47515e07 00007f7b503709a9 00007f7b4753269b 00007f7b4753093f 00007f7b47530455 00007f7b4e0183c7 00007f7b4a62a9a7 00007f7b4a63a0bf 00007f7b4a63a038
0709 12:24:50.847633 (+ 80930us) log.cc:844] Serialized 14 byte log entry
0709 12:24:51.479539 (+631906us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:51.479591 (+    52us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:24:51.479641 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:51.479706 (+    65us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:51.479870 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":459548,"prepare.queue_time_us":147331,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":42,"raft.queue_time_us":489836,"raft.run_cpu_time_us":225,"raft.run_wall_time_us":10466,"spinlock_wait_cycles":7086848,"thread_start_us":448396,"threads_started":3,"wal-append.queue_time_us":569292,"wal-append.run_cpu_time_us":1284,"wal-append.run_wall_time_us":1003721}
I20260709 12:24:51.521164 17729 leader_election.cc:304] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:51.522991 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:51.524055 17757 raft_consensus.cc:2468] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 29.
I20260709 12:24:51.524307 26354 raft_consensus.cc:2749] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:24:51.525058 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:51.435124 26978 raft_consensus.cc:515] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:51.526696 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 31 candidate_status { last_received { term: 30 index: 23 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:51.526858 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:51.527601 17729 leader_election.cc:304] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:51.492080 27096 raft_consensus.cc:2749] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:24:51.469236 27127 consensus_queue.cc:1048] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.055s
I20260709 12:24:51.531514 26354 raft_consensus.cc:2749] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:24:51.467351 26866 consensus_queue.cc:1048] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.058s
I20260709 12:24:51.529028 18333 leader_election.cc:304] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:24:51.509436 26982 raft_consensus.cc:493] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:24:51.520777 26995 raft_consensus.cc:2749] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 12:24:51.540108 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:51.541706 17757 raft_consensus.cc:2468] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 30.
I20260709 12:24:51.527962 27035 raft_consensus.cc:493] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:24:51.534260 27107 raft_consensus.cc:3060] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:51.529549 27129 raft_consensus.cc:493] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:51.518854 17759 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3"
dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
mode: GRACEFUL
new_leader_uuid: "e063541542a94969898868bf6eb5ce1b"
 from {username='slave'} at 127.0.0.1:49786
I20260709 12:24:51.532675 26587 raft_consensus.cc:697] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 28 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:24:51.548983 17759 raft_consensus.cc:606] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Received request to transfer leadership to TS e063541542a94969898868bf6eb5ce1b
I20260709 12:24:51.549432 17759 raft_consensus.cc:614] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260709 12:24:51.548018 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 97640.6us
I20260709 12:24:51.525236 27097 consensus_queue.cc:1059] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 29.20, Next index: 21, Last known committed idx: 19, Time since last communication: 0.019s
I20260709 12:24:51.538113 26844 leader_election.cc:290] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 29 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:51.570314 17654 leader_election.cc:304] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:24:51.541100 27124 raft_consensus.cc:493] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:51.577190 18333 leader_election.cc:304] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:51.546113 26957 consensus_queue.cc:1048] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.033s
I20260709 12:24:51.568387 26995 consensus_queue.cc:1059] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 23.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.016s
I20260709 12:24:51.587307 26956 raft_consensus.cc:3055] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:24:51.587782 26956 raft_consensus.cc:740] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 29 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:24:51.588626 26956 consensus_queue.cc:260] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 29.20, Last appended by leader: 20, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:51.589820 26956 raft_consensus.cc:3060] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:24:51.591351 17829 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37994 (request call id 1726) took 1113 ms. Trace:
I20260709 12:24:51.591758 17829 rpcz_store.cc:276] 0709 12:24:50.478206 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:50.478328 (+   122us) service_pool.cc:224] Handling call
0709 12:24:50.540188 (+ 61860us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:50.540382 (+   194us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 12:24:50.681348 (+140966us) log.cc:844] Serialized 14 byte log entry
0709 12:24:51.590844 (+909496us) spinlock_profiling.cc:243] Waited 5.01 ms on lock 0x7b18004045c0. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b474adc0a 00007f7b50bed219 00007f7b47535a40 00007f7b475304e4 00007f7b47530455 00007f7b4a618060 00007f7b4a62a716 00007f7b4a63b287 00007f7b4a63b208
0709 12:24:51.590978 (+   134us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:51.591027 (+    49us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:24:51.591075 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:51.591162 (+    87us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:51.591293 (+   131us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4347,"prepare.queue_time_us":245012,"prepare.run_cpu_time_us":55,"prepare.run_wall_time_us":1647,"spinlock_wait_cycles":11532544,"thread_start_us":262395,"threads_started":2,"wal-append.queue_time_us":146647}
I20260709 12:24:51.567662 26632 leader_election.cc:290] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:51.544476 26982 raft_consensus.cc:515] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:51.601595 17830 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37994 (request call id 1717) took 1570 ms. Trace:
I20260709 12:24:51.573529 27128 raft_consensus.cc:493] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:51.621675 27128 raft_consensus.cc:515] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:51.579468 27124 raft_consensus.cc:515] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:51.624488 27124 leader_election.cc:290] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:51.607406 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 25 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:51.625370 17764 raft_consensus.cc:2468] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 24.
I20260709 12:24:51.609715 17830 rpcz_store.cc:276] 0709 12:24:50.031247 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:50.031529 (+   282us) service_pool.cc:224] Handling call
0709 12:24:51.601364 (+1569835us) spinlock_profiling.cc:243] Waited 251 ms on lock 0x7b540022b854. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:24:51.601521 (+   157us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":577901952}
I20260709 12:24:51.625962 18365 raft_consensus.cc:2468] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 31.
I20260709 12:24:51.612798 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:51.613576 26957 raft_consensus.cc:3055] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:24:51.615742 27004 raft_consensus.cc:2804] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:24:51.616112 17761 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35"
dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
mode: GRACEFUL
new_leader_uuid: "e063541542a94969898868bf6eb5ce1b"
 from {username='slave'} at 127.0.0.1:49786
I20260709 12:24:51.654301 17761 raft_consensus.cc:606] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Received request to transfer leadership to TS e063541542a94969898868bf6eb5ce1b
I20260709 12:24:51.654690 17761 raft_consensus.cc:614] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:24:51.615013 26887 raft_consensus.cc:2804] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:24:51.568380 27129 raft_consensus.cc:515] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:51.620409 26587 consensus_queue.cc:237] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 26.18, Last appended by leader: 18, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:51.622160 26978 leader_election.cc:290] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:51.669489 17760 raft_consensus.cc:3060] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:24:51.670342 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:51.670971 17834 raft_consensus.cc:3060] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:51.671748 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:51.668205 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:51.596426 17729 leader_election.cc:304] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:51.623736 17727 leader_election.cc:304] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
W20260709 12:24:51.674823 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 78461.8us
W20260709 12:24:51.675141 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 51527.7us
I20260709 12:24:51.675678 17729 leader_election.cc:304] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 31 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:51.578583 27134 raft_consensus.cc:2804] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:24:51.594368 27088 raft_consensus.cc:2749] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:24:51.626677 17652 leader_election.cc:304] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:24:51.681756 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 55138.2us
I20260709 12:24:51.548893 27035 raft_consensus.cc:515] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:51.632253 17830 raft_consensus.cc:1240] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 24. Current term is 25. Ops: []
W20260709 12:24:51.649410 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 822346us
I20260709 12:24:51.683619 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:51.684927 17764 raft_consensus.cc:2393] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 28: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:24:51.642774 17802 leader_election.cc:304] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:51.686857 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 26 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:51.687984 18361 raft_consensus.cc:2393] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 26: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
W20260709 12:24:51.659660 27119 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.554s	user 0.004s	sys 0.009s
I20260709 12:24:51.666457 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 28 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:51.612056 27139 raft_consensus.cc:2804] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:24:51.582016 27096 consensus_queue.cc:1059] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 25.15, Next index: 16, Last known committed idx: 13, Time since last communication: 0.100s
W20260709 12:24:51.689880 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 119646us
I20260709 12:24:51.630019 26957 raft_consensus.cc:740] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:24:51.651680 27062 raft_consensus.cc:493] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:51.693490 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 23 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:51.655778 27044 leader_election.cc:290] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:51.682533 26978 raft_consensus.cc:2749] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Leader election lost for term 31. Reason: could not achieve majority
I20260709 12:24:51.656877 26887 raft_consensus.cc:493] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:51.702996 17835 raft_consensus.cc:2468] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 29.
I20260709 12:24:51.657119 26985 raft_consensus.cc:493] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:24:51.753767 26887 raft_consensus.cc:3060] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:24:51.754139 26985 raft_consensus.cc:515] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:51.705199 17832 raft_consensus.cc:2468] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 27.
I20260709 12:24:51.693344 26866 raft_consensus.cc:2804] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:24:51.694906 17652 leader_election.cc:304] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:51.706918 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:51.704272 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:51.710697 17755 raft_consensus.cc:1240] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 21. Current term is 22. Ops: []
W20260709 12:24:51.707839 27119 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.603s	user 0.004s	sys 0.010s
I20260709 12:24:51.712397 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:51.676988 26995 raft_consensus.cc:2749] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 12:24:51.680193 26956 raft_consensus.cc:3060] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:51.652012 27004 raft_consensus.cc:493] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:51.717589 27028 raft_consensus.cc:515] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:51.717653 26957 consensus_queue.cc:260] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 23.16, Last appended by leader: 16, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:51.718555 27062 raft_consensus.cc:515] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:51.719936 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 28 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:51.720360 27139 raft_consensus.cc:697] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 31 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:24:51.724371 26848 leader_election.cc:290] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 29 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:51.725018 27128 leader_election.cc:290] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:51.730041 17834 raft_consensus.cc:2468] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 27.
I20260709 12:24:51.782030 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:51.783298 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:24:51.788441 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:51.769088 27049 raft_consensus.cc:2804] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:24:51.732033 26982 leader_election.cc:290] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:51.743000 26843 raft_consensus.cc:3055] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:24:51.744365 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:51.825210 17831 raft_consensus.cc:2468] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 24.
I20260709 12:24:51.826211 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 22 candidate_status { last_received { term: 21 index: 20 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:51.827389 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 24 candidate_status { last_received { term: 23 index: 19 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:51.750137 17762 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d"
dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
mode: GRACEFUL
new_leader_uuid: "e063541542a94969898868bf6eb5ce1b"
 from {username='slave'} at 127.0.0.1:49786
I20260709 12:24:51.716223 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:51.828169 17762 raft_consensus.cc:606] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Received request to transfer leadership to TS e063541542a94969898868bf6eb5ce1b
I20260709 12:24:51.828594 17762 raft_consensus.cc:614] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:24:51.830547 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 23 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:51.775382 26888 raft_consensus.cc:2804] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:24:51.682696 27039 raft_consensus.cc:493] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:51.753270 26599 raft_consensus.cc:3060] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:24:51.838567 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:51.839808 17834 raft_consensus.cc:2393] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 27: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:24:51.753506 17838 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35042 (request call id 3356) took 2417 ms. Trace:
I20260709 12:24:51.841436 18365 raft_consensus.cc:1240] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 22. Current term is 23. Ops: [22.15-22.15]
I20260709 12:24:51.682601 17761 raft_consensus.cc:1240] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 26. Current term is 27. Ops: []
I20260709 12:24:51.842347 27039 raft_consensus.cc:515] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:51.844363 17761 raft_consensus.cc:1240] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 24. Current term is 25. Ops: []
I20260709 12:24:51.844683 27039 leader_election.cc:290] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:51.756269 17760 raft_consensus.cc:2468] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 31.
I20260709 12:24:51.841243 17838 rpcz_store.cc:276] 0709 12:24:49.336164 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:49.348128 (+ 11964us) service_pool.cc:224] Handling call
0709 12:24:49.364694 (+ 16566us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:49.365087 (+   393us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:49.365112 (+    25us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:49.368108 (+  2996us) log.cc:844] Serialized 26 byte log entry
0709 12:24:49.667332 (+299224us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:24:51.364724 (+1697392us) spinlock_profiling.cc:243] Waited 41.2 ms on lock 0x7b18004d3e00. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b4a1c84bc 00007f7b4751648b 00007f7b503709a9 00007f7b4753269b 00007f7b4753093f 00007f7b47530455 00007f7b4e0183c7 00007f7b4a62a9a7 00007f7b4a63a0bf
0709 12:24:51.364877 (+   153us) log.cc:844] Serialized 14 byte log entry
0709 12:24:51.748096 (+383219us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:51.748185 (+    89us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:51.748228 (+    43us) raft_consensus.cc:1685] finished
0709 12:24:51.753265 (+  5037us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:51.753415 (+   150us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":172365,"prepare.queue_time_us":15885,"prepare.run_cpu_time_us":59,"prepare.run_wall_time_us":2628,"raft.queue_time_us":183029,"raft.run_cpu_time_us":212,"raft.run_wall_time_us":8416,"spinlock_wait_cycles":94712576,"thread_start_us":722957,"threads_started":4,"wal-append.queue_time_us":687634,"wal-append.run_cpu_time_us":1848,"wal-append.run_wall_time_us":1091378}
I20260709 12:24:51.848285 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 31 candidate_status { last_received { term: 30 index: 23 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:51.848136 27139 consensus_queue.cc:237] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 29.17, Last appended by leader: 17, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:51.848912 18367 raft_consensus.cc:2468] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 31.
W20260709 12:24:51.758311 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 63474.6us
I20260709 12:24:51.850405 17652 leader_election.cc:304] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:51.852381 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:24:51.853711 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 75584.4us
I20260709 12:24:51.766824 17764 raft_consensus.cc:1275] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 22 index: 17. Preceding OpId from leader: term: 24 index: 18. (index mismatch)
I20260709 12:24:51.857334 17764 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "d522f70beb1f417cb98940d90b594823"
dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
mode: GRACEFUL
new_leader_uuid: "e063541542a94969898868bf6eb5ce1b"
 from {username='slave'} at 127.0.0.1:49786
I20260709 12:24:51.858183 17764 raft_consensus.cc:606] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Received request to transfer leadership to TS e063541542a94969898868bf6eb5ce1b
I20260709 12:24:51.858656 17764 raft_consensus.cc:614] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:24:51.705753 27134 raft_consensus.cc:697] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 25 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
W20260709 12:24:51.862824 17652 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:51.774168 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:51.757829 26866 raft_consensus.cc:493] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:51.844695 18333 leader_election.cc:304] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:51.779349 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:51.769954 26789 raft_consensus.cc:493] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:51.770732 27004 raft_consensus.cc:3060] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:24:51.853579 27138 log.cc:927] Time spent T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.280s	user 0.000s	sys 0.004s
I20260709 12:24:51.715981 17654 leader_election.cc:304] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:24:51.931622 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 215738us
I20260709 12:24:51.833055 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:51.836076 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 27 candidate_status { last_received { term: 21 index: 18 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:51.837788 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:51.773485 17832 raft_consensus.cc:1240] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 28. Current term is 29. Ops: []
I20260709 12:24:51.863131 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:51.794498 27049 raft_consensus.cc:493] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:51.878034 17729 leader_election.cc:304] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:51.875407 17652 leader_election.cc:304] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:51.917626 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
W20260709 12:24:51.923650 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 79043.3us
I20260709 12:24:51.936962 18333 leader_election.cc:304] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:24:51.918774 17730 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:51.839797 26888 raft_consensus.cc:493] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:51.840574 26957 raft_consensus.cc:3060] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:51.926676 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:51.824081 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:51.946041 17835 raft_consensus.cc:2468] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 29.
I20260709 12:24:51.947826 26354 raft_consensus.cc:2764] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Leader pre-election decision vote started in defunct term 30: won
W20260709 12:24:51.947803 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 69851.2us
I20260709 12:24:51.951793 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:51.952858 17762 raft_consensus.cc:2393] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 22: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:24:51.864058 27012 raft_consensus.cc:2804] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Leader pre-election won for term 25
W20260709 12:24:51.941732 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 66436.1us
I20260709 12:24:51.929764 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:24:51.955520 17728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:51.916898 26866 raft_consensus.cc:3060] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Advancing to term 30
W20260709 12:24:52.089638 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:51.926735 26985 leader_election.cc:290] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:51.944643 26789 raft_consensus.cc:515] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:51.957110 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 29 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:52.086233 27134 consensus_queue.cc:237] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 22.18, Last appended by leader: 19, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:52.115248 17755 raft_consensus.cc:1275] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 26 index: 15. Preceding OpId from leader: term: 28 index: 16. (index mismatch)
I20260709 12:24:52.103579 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:24:52.125702 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 482950us
W20260709 12:24:52.127220 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 190356us
I20260709 12:24:52.135663 18333 leader_election.cc:304] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:51.824918 26843 raft_consensus.cc:740] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 22 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:24:51.940979 27135 raft_consensus.cc:2749] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260709 12:24:51.958434 26632 raft_consensus.cc:2749] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:24:51.940853 27049 raft_consensus.cc:3060] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:24:51.946975 26888 raft_consensus.cc:3060] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:51.943365 27126 raft_consensus.cc:2749] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:24:52.128973 17763 raft_consensus.cc:1240] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 30. Current term is 31. Ops: []
I20260709 12:24:52.144779 17831 raft_consensus.cc:1240] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 20. Current term is 21. Ops: []
I20260709 12:24:52.161888 17803 leader_election.cc:304] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:52.126891 26818 raft_consensus.cc:493] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:52.106089 27012 raft_consensus.cc:493] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:52.144544 27102 raft_consensus.cc:2749] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:24:51.924518 26982 consensus_queue.cc:1059] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 20.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.201s
I20260709 12:24:51.922781 26887 raft_consensus.cc:515] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:52.121752 17652 leader_election.cc:304] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:52.166302 17803 leader_election.cc:304] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:52.140064 26837 raft_consensus.cc:493] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:52.129514 27040 raft_consensus.cc:493] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:52.130486 26962 raft_consensus.cc:2749] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
W20260709 12:24:52.154462 27098 log.cc:927] Time spent T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.197s	user 0.001s	sys 0.000s
I20260709 12:24:52.178934 17831 raft_consensus.cc:1240] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 24. Current term is 25. Ops: []
W20260709 12:24:52.181151 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 44044.4us
W20260709 12:24:52.205906 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:52.139276 26971 raft_consensus.cc:493] T a5732e71b9d54f8786b8c511d69943bf P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:52.184510 18368 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59902 (request call id 2171) took 1358 ms. Trace:
I20260709 12:24:52.186874 17833 raft_consensus.cc:1240] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 21. Current term is 22. Ops: []
I20260709 12:24:52.189335 17832 raft_consensus.cc:1240] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 25. Current term is 26. Ops: []
I20260709 12:24:52.186100 27012 raft_consensus.cc:3060] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:52.209358 18368 rpcz_store.cc:276] 0709 12:24:50.826374 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:50.827583 (+  1209us) service_pool.cc:224] Handling call
0709 12:24:50.827948 (+   365us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:51.330541 (+502593us) raft_consensus.cc:1534] Early marking committed up to index 17
0709 12:24:51.330572 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:51.335339 (+  4767us) log.cc:844] Serialized 26 byte log entry
0709 12:24:51.527625 (+192286us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:24:51.527682 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:51.527779 (+    97us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:51.830680 (+302901us) spinlock_profiling.cc:243] Waited 40.4 ms on lock 0x7b4400022b4c. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b47459c85 00007f7b4745a6a2 00007f7b4a1cc1db 00007f7b4a1c7aaf 00007f7b4a1bb86e 00007f7b4a1babb6 00007f7b4a1ba7d8 00007f7b4a1c4221 00007f7b4a1c41d9 00007f7b4a1c4169
0709 12:24:52.177505 (+346825us) raft_consensus.cc:1685] finished
0709 12:24:52.177527 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:52.184445 (+  6918us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":409902,"raft.queue_time_us":124630,"raft.run_cpu_time_us":157,"raft.run_wall_time_us":14223,"spinlock_wait_cycles":92997504,"thread_start_us":653240,"threads_started":2,"wal-append.queue_time_us":188273}
W20260709 12:24:52.196235 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 74620.1us
I20260709 12:24:52.219534 17652 leader_election.cc:304] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:52.144400 27124 consensus_queue.cc:1059] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 21.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.048s
I20260709 12:24:52.165040 27129 leader_election.cc:290] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:24:52.228627 27151 log.cc:927] Time spent T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.394s	user 0.001s	sys 0.000s
W20260709 12:24:52.231417 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 65209.1us
I20260709 12:24:52.232231 17803 leader_election.cc:304] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:24:52.233155 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 300501us
I20260709 12:24:52.183743 27149 raft_consensus.cc:2749] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:24:52.181408 26995 raft_consensus.cc:493] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:24:52.221414 17836 raft_consensus.cc:2368] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for earlier term 29. Current term is 30.
I20260709 12:24:52.212446 26978 raft_consensus.cc:493] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:24:52.223981 26971 raft_consensus.cc:515] T a5732e71b9d54f8786b8c511d69943bf P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:52.224717 26635 consensus_queue.cc:1059] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.004s
I20260709 12:24:52.203974 27126 raft_consensus.cc:2749] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:24:52.178014 27051 consensus_queue.cc:1048] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 17, Time since last communication: 0.033s
I20260709 12:24:52.255944 17654 leader_election.cc:304] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:52.265856 17654 leader_election.cc:304] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:52.256875 17757 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f"
dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
mode: GRACEFUL
new_leader_uuid: "334a4e1f59944544b3b3ff36616b57da"
 from {username='slave'} at 127.0.0.1:49786
I20260709 12:24:52.195418 26985 consensus_queue.cc:1048] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.058s
I20260709 12:24:52.186728 26818 raft_consensus.cc:515] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:52.270519 17654 leader_election.cc:304] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:52.257342 17652 leader_election.cc:304] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:52.266686 17757 raft_consensus.cc:606] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 25 LEADER]: Received request to transfer leadership to TS 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:52.255107 27116 raft_consensus.cc:3060] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:52.206584 27040 raft_consensus.cc:515] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:52.296620 17654 leader_election.cc:304] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:24:52.302783 16501 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260709 12:24:52.253484 27111 raft_consensus.cc:3060] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:52.206736 26837 raft_consensus.cc:515] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:52.256701 27119 raft_consensus.cc:2749] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 12:24:52.266672 27028 leader_election.cc:290] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 25 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:52.272620 27044 raft_consensus.cc:2804] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:24:52.254117 27135 consensus_queue.cc:1059] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 24.17, Next index: 18, Last known committed idx: 16, Time since last communication: 0.049s
I20260709 12:24:52.272267 27128 raft_consensus.cc:2749] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:24:52.293186 27124 raft_consensus.cc:493] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:52.211838 26843 consensus_queue.cc:260] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 22.17, Last appended by leader: 17, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:52.265422 26966 consensus_queue.cc:1048] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.052s
I20260709 12:24:52.253463 26978 raft_consensus.cc:515] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:52.298965 27062 leader_election.cc:290] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:52.302765 27129 raft_consensus.cc:3055] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:24:52.302945 27126 raft_consensus.cc:2749] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:24:52.254338 26995 raft_consensus.cc:515] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:52.303792 26635 raft_consensus.cc:493] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:52.473606 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
W20260709 12:24:52.489248 17728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:52.505119 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:52.464006 18363 raft_consensus.cc:1240] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 28 LEADER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 27. Current term is 28. Ops: []
W20260709 12:24:52.348701 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 89220us
I20260709 12:24:52.479518 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 29 candidate_status { last_received { term: 27 index: 21 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:52.471742 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 32 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:52.434577 18374 raft_consensus.cc:1275] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 23 index: 17. Preceding OpId from leader: term: 25 index: 18. (index mismatch)
I20260709 12:24:52.528451 18374 raft_consensus.cc:1240] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 25. Current term is 26. Ops: []
I20260709 12:24:52.527011 17757 raft_consensus.cc:2468] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 31.
I20260709 12:24:52.445231 18333 leader_election.cc:304] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:52.326529 27141 raft_consensus.cc:2804] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:24:52.534406 17829 raft_consensus.cc:2393] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 29: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:24:52.422254 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 25 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:52.420518 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 29 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
W20260709 12:24:52.406810 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 149543us
I20260709 12:24:52.548213 17652 leader_election.cc:304] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:52.425974 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:52.472723 17759 raft_consensus.cc:1275] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 22 index: 18. Preceding OpId from leader: term: 28 index: 20. (index mismatch)
W20260709 12:24:52.509756 17730 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:52.553946 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:52.554226 17801 leader_election.cc:304] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:52.554546 17757 raft_consensus.cc:3060] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:24:52.477289 17758 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38786 (request call id 3991) took 1904 ms. Trace:
I20260709 12:24:52.556639 18371 raft_consensus.cc:1275] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 27 index: 21. Preceding OpId from leader: term: 28 index: 21. (term mismatch)
I20260709 12:24:52.557056 18371 pending_rounds.cc:85] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da: Aborting all ops after (but not including) 20
I20260709 12:24:52.557372 18371 pending_rounds.cc:107] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da: Aborting uncommitted NO_OP operation due to leader change: 27.21
W20260709 12:24:52.534608 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 100591us
W20260709 12:24:52.542474 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 87320.2us
I20260709 12:24:52.557557 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 25 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:52.558197 17763 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 24.
I20260709 12:24:52.328500 27044 raft_consensus.cc:493] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:52.524667 27129 raft_consensus.cc:740] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:24:52.571425 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 29 candidate_status { last_received { term: 27 index: 21 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:52.545342 27141 raft_consensus.cc:493] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:24:52.561828 27134 raft_consensus.cc:3055] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:24:52.562804 27049 raft_consensus.cc:515] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:52.534781 26848 raft_consensus.cc:493] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:52.420269 27135 raft_consensus.cc:493] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:52.466111 27148 raft_consensus.cc:493] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:52.506027 27133 log.cc:927] Time spent T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.938s	user 0.003s	sys 0.000s
I20260709 12:24:52.484970 26962 raft_consensus.cc:3055] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:24:52.485812 27093 raft_consensus.cc:3055] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:24:52.555636 26714 raft_consensus.cc:493] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:52.449282 26924 raft_consensus.cc:493] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:52.533432 27169 raft_consensus.cc:2804] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:24:52.602634 27169 raft_consensus.cc:697] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 27 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:24:52.603649 27169 consensus_queue.cc:237] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 25.16, Last appended by leader: 16, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:52.566491 27012 raft_consensus.cc:515] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:52.608371 26714 raft_consensus.cc:515] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:52.610612 27049 leader_election.cc:290] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 31 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:52.610385 26924 raft_consensus.cc:515] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:52.595391 26962 raft_consensus.cc:740] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:24:52.612284 17681 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37402 (request call id 3780) took 1384 ms. Trace:
I20260709 12:24:52.612130 26962 consensus_queue.cc:260] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 21.16, Last appended by leader: 16, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:52.579962 27134 raft_consensus.cc:740] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:24:52.612730 26924 leader_election.cc:290] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:52.597920 27093 raft_consensus.cc:740] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:24:52.613857 27093 consensus_queue.cc:260] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 22.15, Last appended by leader: 15, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:52.614862 27093 raft_consensus.cc:3060] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:52.593803 26848 raft_consensus.cc:515] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:52.615363 27012 leader_election.cc:290] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 25 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:52.568548 16501 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:24:52.444919 26963 consensus_queue.cc:260] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 21.15, Last appended by leader: 15, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:52.471096 27035 leader_election.cc:290] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:52.478992 26887 leader_election.cc:290] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 30 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:52.480288 26789 leader_election.cc:290] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:52.506989 26837 leader_election.cc:290] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:24:52.569118 27004 consensus_meta.cc:287] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.613s	user 0.005s	sys 0.005s
W20260709 12:24:52.544016 26956 consensus_meta.cc:287] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.759s	user 0.010s	sys 0.000s
I20260709 12:24:52.475239 26985 consensus_queue.cc:1048] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.094s
I20260709 12:24:52.576596 27008 consensus_queue.cc:1048] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 18, Time since last communication: 0.046s
W20260709 12:24:52.478648 26599 consensus_meta.cc:287] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.633s	user 0.008s	sys 0.000s
W20260709 12:24:52.503336 27107 consensus_meta.cc:287] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.953s	user 0.007s	sys 0.004s
W20260709 12:24:52.436223 27161 log.cc:927] Time spent T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.085s	user 0.001s	sys 0.000s
I20260709 12:24:52.542934 17756 raft_consensus.cc:2468] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 24.
W20260709 12:24:52.377321 27105 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.772s	user 0.020s	sys 0.020s
I20260709 12:24:52.557677 18371 raft_consensus.cc:2889] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:52.574625 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 25 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:52.463317 27116 raft_consensus.cc:2749] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:24:52.357777 27139 raft_consensus.cc:2749] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:24:52.441098 26971 leader_election.cc:290] T a5732e71b9d54f8786b8c511d69943bf P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:52.633221 26963 raft_consensus.cc:3060] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:52.442698 27040 leader_election.cc:290] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:52.458969 26587 raft_consensus.cc:493] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:52.501717 26818 leader_election.cc:290] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:52.653692 27012 raft_consensus.cc:493] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:24:52.465899 27168 raft_consensus.cc:493] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:24:52.588744 17755 raft_consensus.cc:1275] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 29 index: 17. Preceding OpId from leader: term: 31 index: 18. (index mismatch)
I20260709 12:24:52.556334 27128 raft_consensus.cc:2749] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 12:24:52.594556 17829 raft_consensus.cc:1275] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 22 index: 12. Preceding OpId from leader: term: 24 index: 13. (index mismatch)
I20260709 12:24:52.596707 17761 raft_consensus.cc:1275] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 29 index: 19. Preceding OpId from leader: term: 31 index: 20. (index mismatch)
I20260709 12:24:52.590091 27094 consensus_queue.cc:1059] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.007s
I20260709 12:24:52.693056 27094 consensus_queue.cc:1243] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer 334a4e1f59944544b3b3ff36616b57da log is divergent from this leader: its last log entry 28.19 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 18
I20260709 12:24:52.693598 27094 raft_consensus.cc:3055] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:24:52.693954 27094 raft_consensus.cc:740] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:24:52.592320 26957 consensus_queue.cc:1059] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 25.15, Next index: 16, Last known committed idx: 13, Time since last communication: 0.005s
I20260709 12:24:52.573065 27062 raft_consensus.cc:2804] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:24:52.573513 27044 raft_consensus.cc:3060] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:52.346104 27154 consensus_queue.cc:1048] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.245s
I20260709 12:24:52.578440 27141 raft_consensus.cc:3060] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:52.581451 27126 consensus_queue.cc:1059] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.019s
I20260709 12:24:52.612646 17681 rpcz_store.cc:276] 0709 12:24:51.241866 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:51.256283 (+ 14417us) service_pool.cc:224] Handling call
0709 12:24:51.256766 (+   483us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:51.263810 (+  7044us) raft_consensus.cc:1534] Early marking committed up to index 17
0709 12:24:51.280801 (+ 16991us) log.cc:844] Serialized 14 byte log entry
0709 12:24:51.418148 (+137347us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:51.469572 (+ 51424us) log.cc:844] Serialized 26 byte log entry
0709 12:24:51.490581 (+ 21009us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:24:51.490639 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:51.490739 (+   100us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:52.505494 (+1014755us) spinlock_profiling.cc:243] Waited 33 ms on lock 0x7b4400022b4c. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b47459c85 00007f7b4745a6a2 00007f7b4a1cc1db 00007f7b4a1c7aaf 00007f7b4a1bb86e 00007f7b4a1babb6 00007f7b4a1ba7d8 00007f7b4a1c4221 00007f7b4a1c41d9 00007f7b4a1c4169
0709 12:24:52.612001 (+106507us) raft_consensus.cc:1685] finished
0709 12:24:52.612087 (+    86us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:52.612226 (+   139us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":16601,"prepare.run_cpu_time_us":85,"prepare.run_wall_time_us":7269,"raft.queue_time_us":246,"raft.run_cpu_time_us":322,"raft.run_wall_time_us":37454,"spinlock_wait_cycles":75966720,"thread_start_us":96510,"threads_started":1,"wal-append.queue_time_us":235139}
I20260709 12:24:52.358393 27124 raft_consensus.cc:515] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:52.545536 26635 raft_consensus.cc:515] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:52.621601 17759 raft_consensus.cc:1275] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 25 index: 16. Preceding OpId from leader: term: 27 index: 17. (index mismatch)
I20260709 12:24:52.614885 26995 leader_election.cc:290] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:52.581266 27119 consensus_queue.cc:1059] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 25.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.048s
I20260709 12:24:52.635170 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:52.732128 17764 raft_consensus.cc:2468] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 25.
I20260709 12:24:52.628361 26966 raft_consensus.cc:493] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:52.664103 17757 raft_consensus.cc:2468] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 30.
I20260709 12:24:52.666491 18364 raft_consensus.cc:2468] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 26.
I20260709 12:24:52.555280 17758 rpcz_store.cc:276] 0709 12:24:50.572709 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:50.572848 (+   139us) service_pool.cc:224] Handling call
0709 12:24:50.604920 (+ 32072us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:51.507922 (+903002us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:51.507969 (+    47us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:52.241959 (+733990us) spinlock_profiling.cc:243] Waited 2.85 ms on lock 0x7b2c0000e5c8. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b48f19a11 00007f7b48f29372 00007f7b4a65ba01 00007f7b4a62043a 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2
0709 12:24:52.242430 (+   471us) log.cc:844] Serialized 26 byte log entry
0709 12:24:52.364595 (+122165us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:24:52.364653 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:52.476969 (+112316us) spinlock_profiling.cc:243] Waited 4.42 ms on lock 0x7b54000fbed4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:24:52.477023 (+    54us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:52.477072 (+    49us) raft_consensus.cc:1685] finished
0709 12:24:52.477085 (+    13us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:52.477230 (+   145us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":881984,"raft.queue_time_us":882203,"raft.run_cpu_time_us":159,"raft.run_wall_time_us":14058,"spinlock_wait_cycles":16708736,"thread_start_us":115052,"threads_started":1,"wal-append.queue_time_us":188498}
I20260709 12:24:52.622107 26848 leader_election.cc:290] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:52.577390 18333 leader_election.cc:304] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:24:52.734933 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 157621us
I20260709 12:24:52.735698 18333 leader_election.cc:304] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:24:52.737074 18333 leader_election.cc:304] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
W20260709 12:24:52.669492 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 372936us
I20260709 12:24:52.738582 17654 leader_election.cc:304] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:52.529691 26888 raft_consensus.cc:515] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:52.596222 27135 raft_consensus.cc:515] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:52.669796 17652 leader_election.cc:304] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:52.740743 27124 leader_election.cc:290] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:52.742388 27135 leader_election.cc:290] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:52.633092 26962 raft_consensus.cc:3060] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:52.744211 17801 leader_election.cc:304] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:52.745242 26635 leader_election.cc:290] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:24:52.665759 27027 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 1.086s	user 0.037s	sys 0.003s
W20260709 12:24:52.747473 27027 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.170s	user 0.038s	sys 0.003s
W20260709 12:24:52.666926 27131 log.cc:927] Time spent T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 1.120s	user 0.000s	sys 0.003s
I20260709 12:24:52.633525 26956 raft_consensus.cc:2749] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:24:52.632553 27102 raft_consensus.cc:493] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:52.750124 17760 raft_consensus.cc:1240] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 25. Current term is 26. Ops: []
I20260709 12:24:52.750240 27102 raft_consensus.cc:515] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:52.752460 17830 raft_consensus.cc:1240] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 22. Current term is 23. Ops: []
I20260709 12:24:52.752758 27102 leader_election.cc:290] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:24:52.664995 27105 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.060s	user 0.028s	sys 0.020s
I20260709 12:24:52.598526 27148 raft_consensus.cc:515] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:52.484364 26844 raft_consensus.cc:3055] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:24:52.753239 26966 raft_consensus.cc:515] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:52.754838 27148 leader_election.cc:290] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:52.755017 27111 raft_consensus.cc:2749] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:24:52.682626 27134 consensus_queue.cc:260] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 24.17, Last appended by leader: 17, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:52.753885 26844 raft_consensus.cc:740] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 25 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:24:52.755416 26966 leader_election.cc:290] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:52.755868 27134 raft_consensus.cc:3060] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:52.756819 27126 raft_consensus.cc:3055] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:24:52.756546 26844 consensus_queue.cc:260] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 25.15, Last appended by leader: 15, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:52.757959 18361 raft_consensus.cc:2410] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 25 because replica has last-logged OpId of term: 23 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 16.
I20260709 12:24:52.758105 26844 raft_consensus.cc:3060] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:52.758003 27040 raft_consensus.cc:2749] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:24:52.758162 26818 raft_consensus.cc:2804] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:24:52.633142 27004 raft_consensus.cc:515] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:52.759470 26818 raft_consensus.cc:493] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:24:52.708686 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 484830us
I20260709 12:24:52.759869 26818 raft_consensus.cc:3060] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:24:52.664855 26599 raft_consensus.cc:2749] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:24:52.760286 17802 leader_election.cc:304] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:52.683405 27049 raft_consensus.cc:493] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:52.761008 27004 leader_election.cc:290] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 24 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:52.761421 17802 leader_election.cc:304] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:52.761742 27004 raft_consensus.cc:2804] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:24:52.761057 27049 raft_consensus.cc:515] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:52.762255 27004 raft_consensus.cc:697] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 21 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:24:52.763754 17654 leader_election.cc:304] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:24:52.765894 27170 raft_consensus.cc:2804] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:24:52.766326 27170 raft_consensus.cc:493] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:24:52.766695 27170 raft_consensus.cc:3060] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:52.763845 27049 leader_election.cc:290] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:52.765750 27139 raft_consensus.cc:2749] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:24:52.764160 27008 raft_consensus.cc:2749] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:24:52.770452 27119 raft_consensus.cc:3055] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:24:52.770870 27119 raft_consensus.cc:740] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:24:52.758414 27107 raft_consensus.cc:2804] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:24:52.772681 27107 raft_consensus.cc:493] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:52.773085 27107 raft_consensus.cc:3060] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:52.773488 26837 raft_consensus.cc:993] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb: : Instructing follower 334a4e1f59944544b3b3ff36616b57da to start an election
I20260709 12:24:52.773984 26837 raft_consensus.cc:1081] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 25 LEADER]: Signalling peer 334a4e1f59944544b3b3ff36616b57da to start an election
I20260709 12:24:52.696168 27168 raft_consensus.cc:515] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:52.765556 17727 leader_election.cc:304] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:52.777339 17727 leader_election.cc:304] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:24:52.777505 27168 leader_election.cc:290] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:52.777868 17756 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36572 (request call id 1645) took 1064 ms. Trace:
I20260709 12:24:52.778374 26978 leader_election.cc:290] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:52.778260 17756 rpcz_store.cc:276] 0709 12:24:51.715628 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:51.715823 (+   195us) service_pool.cc:224] Handling call
0709 12:24:52.777681 (+1061858us) spinlock_profiling.cc:243] Waited 646 ms on lock 0x7b54001b8854. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:24:52.777800 (+   119us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1485163776}
I20260709 12:24:52.779125 26978 raft_consensus.cc:2804] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:24:52.779682 26978 raft_consensus.cc:697] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:24:52.780375 26728 raft_consensus.cc:2804] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:24:52.780591 26956 consensus_queue.cc:1059] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.001s
I20260709 12:24:52.782092 27097 raft_consensus.cc:3055] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:24:52.780968 26728 raft_consensus.cc:697] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:24:52.782562 27097 raft_consensus.cc:740] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:24:52.783419 26728 consensus_queue.cc:237] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 25.14, Last appended by leader: 14, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:52.783432 27097 consensus_queue.cc:260] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 28.21, Last appended by leader: 21, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:52.786311 27156 log.cc:927] Time spent T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.379s	user 0.002s	sys 0.000s
I20260709 12:24:52.786870 27097 raft_consensus.cc:3060] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:24:52.788193 27119 consensus_queue.cc:260] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 25.14, Last appended by leader: 14, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:52.789425 27119 raft_consensus.cc:3060] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:52.791103 18371 raft_consensus.cc:1240] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 22. Current term is 23. Ops: [22.15-22.15]
I20260709 12:24:52.792289 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 27 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:24:52.797242 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 60229.4us
W20260709 12:24:52.797994 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 128301us
I20260709 12:24:52.798265 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 25 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:52.798683 17652 leader_election.cc:304] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:52.799137 17688 raft_consensus.cc:2410] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 25 because replica has last-logged OpId of term: 23 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 16.
I20260709 12:24:52.799754 17681 raft_consensus.cc:1275] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 26 index: 15. Preceding OpId from leader: term: 28 index: 16. (index mismatch)
I20260709 12:24:52.757241 27126 raft_consensus.cc:740] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:24:52.804951 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 25 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:52.805882 17838 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 24.
I20260709 12:24:52.808184 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 27 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:52.631076 27129 consensus_queue.cc:260] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 26.19, Last appended by leader: 19, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:52.810250 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 66120.9us
I20260709 12:24:52.808905 17838 raft_consensus.cc:2468] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 26.
I20260709 12:24:52.811465 17801 leader_election.cc:304] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:52.814569 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:52.815618 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 28 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:52.815941 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:52.816251 17757 raft_consensus.cc:2468] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 27.
I20260709 12:24:52.817610 17727 leader_election.cc:400] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 29 election: Vote denied by peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837) with higher term. Message: Invalid argument: T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for earlier term 29. Current term is 30.
I20260709 12:24:52.774571 27049 raft_consensus.cc:2749] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:24:52.697366 27012 raft_consensus.cc:515] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:52.838042 17763 raft_consensus.cc:1240] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 21. Current term is 22. Ops: []
I20260709 12:24:52.838227 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 27 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:52.803319 17682 raft_consensus.cc:1275] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 23 index: 17. Preceding OpId from leader: term: 25 index: 18. (index mismatch)
I20260709 12:24:52.839251 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:52.796780 17654 leader_election.cc:304] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:52.840389 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 31 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:24:52.840590 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 43887us
I20260709 12:24:52.841225 17757 raft_consensus.cc:2468] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 30.
I20260709 12:24:52.842209 17654 leader_election.cc:304] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:24:52.804010 17829 raft_consensus.cc:1240] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 23. Current term is 24. Ops: []
I20260709 12:24:52.699203 27062 raft_consensus.cc:493] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:52.804514 17652 leader_election.cc:304] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:52.718122 27154 consensus_queue.cc:1243] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer 0da169ff82674dc1947055ce43de68a3 log is divergent from this leader: its last log entry 29.17 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 16
I20260709 12:24:52.780570 26978 consensus_queue.cc:237] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 28.16, Last appended by leader: 16, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:52.848829 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:52.818527 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:52.817444 17801 leader_election.cc:304] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:52.821130 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:52.822158 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 32 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:52.824945 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 25 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:52.889835 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 29 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:52.830186 17727 leader_election.cc:403] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 29 election: Cancelling election due to peer responding with higher term
W20260709 12:24:52.892772 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 75312.1us
I20260709 12:24:52.818940 27128 raft_consensus.cc:2804] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Leader election won for term 31
W20260709 12:24:52.911244 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 93879.3us
I20260709 12:24:52.911788 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:52.912189 17833 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35040 (request call id 3568) took 1145 ms. Trace:
I20260709 12:24:52.843618 17654 leader_election.cc:304] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:24:52.845686 17759 raft_consensus.cc:1240] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 26. Current term is 27. Ops: [26.19-26.19]
I20260709 12:24:52.694940 26587 raft_consensus.cc:515] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:52.890558 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:52.827673 17803 leader_election.cc:304] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:24:52.836298 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:52.853008 17762 raft_consensus.cc:1240] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 24. Current term is 25. Ops: []
I20260709 12:24:52.891701 17688 raft_consensus.cc:3060] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:24:52.878810 18333 leader_election.cc:304] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:52.897245 26956 raft_consensus.cc:2749] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Leader election lost for term 29. Reason: Vote denied by peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837) with higher term. Message: Invalid argument: T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for earlier term 29. Current term is 30.
I20260709 12:24:52.847218 27168 consensus_queue.cc:1048] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.053s
I20260709 12:24:52.904233 27004 consensus_queue.cc:237] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 19.14, Last appended by leader: 14, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:52.891903 27139 raft_consensus.cc:2804] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:24:52.906531 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:52.941707 17760 raft_consensus.cc:2468] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 25.
I20260709 12:24:52.890012 27169 raft_consensus.cc:493] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:52.908033 26714 leader_election.cc:290] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:52.823582 26910 raft_consensus.cc:2804] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:24:52.947288 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 27 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:52.909770 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:52.948189 17682 raft_consensus.cc:3060] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:52.910274 27049 raft_consensus.cc:2749] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:24:52.913571 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:52.914069 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 31 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:52.914845 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 27 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:24:52.920213 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 76684.3us
W20260709 12:24:52.921155 27174 log.cc:927] Time spent T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.094s	user 0.000s	sys 0.000s
I20260709 12:24:52.836246 26599 raft_consensus.cc:2804] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Leader election won for term 30
W20260709 12:24:52.922698 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 95097.5us
W20260709 12:24:52.933969 26866 consensus_meta.cc:287] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.804s	user 0.008s	sys 0.003s
W20260709 12:24:52.927896 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 49229us
I20260709 12:24:52.912688 17833 rpcz_store.cc:276] 0709 12:24:51.766309 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:52.211306 (+444997us) service_pool.cc:224] Handling call
0709 12:24:52.224262 (+ 12956us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:52.534068 (+309806us) raft_consensus.cc:1534] Early marking committed up to index 15
0709 12:24:52.579810 (+ 45742us) log.cc:844] Serialized 14 byte log entry
0709 12:24:52.697622 (+117812us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:52.699313 (+  1691us) log.cc:844] Serialized 26 byte log entry
0709 12:24:52.699453 (+   140us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:24:52.699501 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:52.699565 (+    64us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:52.911975 (+212410us) raft_consensus.cc:1685] finished
0709 12:24:52.911996 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:52.912132 (+   136us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":28407,"prepare.queue_time_us":24247,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":46,"raft.queue_time_us":71730,"raft.run_cpu_time_us":355,"raft.run_wall_time_us":48468,"thread_start_us":350830,"threads_started":2,"wal-append.queue_time_us":236492}
I20260709 12:24:52.847999 26632 raft_consensus.cc:2749] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:24:52.905833 27044 raft_consensus.cc:515] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:52.848766 27062 raft_consensus.cc:3060] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:24:52.826838 18367 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f"
dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
 from {username='slave'} at 127.0.0.1:59886
I20260709 12:24:52.972481 18367 raft_consensus.cc:493] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:24:52.972889 18367 raft_consensus.cc:3060] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:52.934590 27124 raft_consensus.cc:2749] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:24:52.818836 27102 consensus_queue.cc:1059] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.105s
I20260709 12:24:52.944165 17688 raft_consensus.cc:2468] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 30.
I20260709 12:24:52.944640 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:52.932750 27128 raft_consensus.cc:697] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 31 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:24:52.950510 18365 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 24.
I20260709 12:24:52.954506 17654 leader_election.cc:304] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:52.838673 27129 raft_consensus.cc:3060] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:52.955660 17803 leader_election.cc:304] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:52.957199 18333 leader_election.cc:304] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:52.924293 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 32 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:52.954741 17831 raft_consensus.cc:2468] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 30.
I20260709 12:24:52.929589 27111 raft_consensus.cc:2804] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:24:52.932365 27179 raft_consensus.cc:2804] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:24:52.985702 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:52.946841 26910 raft_consensus.cc:493] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:52.909787 27141 raft_consensus.cc:515] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:52.945362 27139 raft_consensus.cc:493] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:24:52.984934 17764 raft_consensus.cc:2410] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for term 32 because replica has last-logged OpId of term: 29 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 26 index: 16.
I20260709 12:24:53.002823 26910 raft_consensus.cc:3060] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:52.933440 26985 raft_consensus.cc:3060] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Advancing to term 22
W20260709 12:24:52.973752 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 169314us
I20260709 12:24:52.987932 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:53.073398 17688 raft_consensus.cc:2393] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 22: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:24:53.074436 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:53.076557 18368 raft_consensus.cc:2468] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 24.
W20260709 12:24:52.989078 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:53.083290 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 25 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:52.989557 17680 raft_consensus.cc:2468] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 26.
I20260709 12:24:52.986292 17831 raft_consensus.cc:2468] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 24.
I20260709 12:24:52.996133 18333 leader_election.cc:304] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:24:53.008170 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:52.994709 26599 raft_consensus.cc:697] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 30 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:24:53.012861 26887 consensus_queue.cc:1048] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:24:53.013525 27149 consensus_queue.cc:1048] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20260709 12:24:53.074229 26587 leader_election.cc:290] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:53.110806 26844 raft_consensus.cc:493] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:53.019030 17830 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35042 (request call id 3383) took 1377 ms. Trace:
I20260709 12:24:53.109589 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:24:53.023129 17803 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:53.115363 17688 raft_consensus.cc:2468] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 27.
I20260709 12:24:53.024060 27116 raft_consensus.cc:2749] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260709 12:24:53.116737 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:53.121294 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:53.049664 27044 leader_election.cc:290] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 22 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:53.059234 18366 raft_consensus.cc:2468] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 22.
I20260709 12:24:53.122463 18362 raft_consensus.cc:1275] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 22 index: 12. Preceding OpId from leader: term: 24 index: 13. (index mismatch)
I20260709 12:24:52.987124 27111 raft_consensus.cc:493] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:53.065899 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:53.114744 17830 rpcz_store.cc:276] 0709 12:24:51.641523 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:51.954151 (+312628us) service_pool.cc:224] Handling call
0709 12:24:52.132185 (+178034us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:52.752875 (+620690us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:53.018722 (+265847us) spinlock_profiling.cc:243] Waited 620 ms on lock 0x7b54002eb654. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:24:53.018953 (+   231us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1426014464}
I20260709 12:24:53.066113 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:52.996114 27049 raft_consensus.cc:2749] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
W20260709 12:24:53.066525 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 48722us
I20260709 12:24:53.125080 18362 raft_consensus.cc:1275] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 26 index: 16. Preceding OpId from leader: term: 31 index: 18. (index mismatch)
I20260709 12:24:53.109697 27097 raft_consensus.cc:493] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:24:53.116616 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:53.127019 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:53.127609 17688 raft_consensus.cc:2468] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 23.
I20260709 12:24:53.128497 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:53.120548 18371 raft_consensus.cc:1240] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 25 LEADER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 24. Current term is 25. Ops: [24.19-24.19]
I20260709 12:24:53.064996 27134 raft_consensus.cc:3060] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Advancing to term 29
W20260709 12:24:53.125159 18334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:53.134169 17684 raft_consensus.cc:1240] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 24. Current term is 25. Ops: []
I20260709 12:24:53.092592 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:53.074738 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:53.120464 18367 raft_consensus.cc:515] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:53.130033 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 25 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:53.118572 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 27 candidate_status { last_received { term: 21 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:52.953250 27169 raft_consensus.cc:515] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:53.131349 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 29 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:53.130504 17682 raft_consensus.cc:2468] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 23.
W20260709 12:24:53.162552 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 90126.1us
I20260709 12:24:53.166208 17654 leader_election.cc:304] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:24:53.182320 18362 raft_consensus.cc:2410] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for term 29 because replica has last-logged OpId of term: 26 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 18.
W20260709 12:24:53.183140 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 187127us
I20260709 12:24:53.068101 27139 raft_consensus.cc:3060] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:53.191303 18367 leader_election.cc:290] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 26 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:24:53.171378 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:53.166177 18361 raft_consensus.cc:1240] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 28 LEADER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 27. Current term is 28. Ops: [27.19-27.19]
I20260709 12:24:53.184963 18334 leader_election.cc:304] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:53.110747 26599 consensus_queue.cc:237] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 28.16, Last appended by leader: 16, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:53.067844 27179 raft_consensus.cc:697] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 23 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:24:52.930189 27126 consensus_queue.cc:260] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 24.19, Last appended by leader: 19, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:53.189787 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:52.974025 26866 raft_consensus.cc:515] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:53.197754 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:53.203711 17688 raft_consensus.cc:2368] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for earlier term 29. Current term is 30.
I20260709 12:24:53.134430 26632 raft_consensus.cc:493] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:24:53.202628 18333 leader_election.cc:304] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:53.126685 27135 raft_consensus.cc:493] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:53.126441 27111 raft_consensus.cc:3060] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:24:53.206748 17729 leader_election.cc:304] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:53.207224 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:53.211912 17682 raft_consensus.cc:3060] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:53.208700 18334 leader_election.cc:304] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:24:53.114289 26844 raft_consensus.cc:515] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:53.213431 18364 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59918 (request call id 2695) took 1796 ms (client timeout 2100 ms). Trace:
I20260709 12:24:53.192721 26963 raft_consensus.cc:2804] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:24:53.216212 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:52.616578 16501 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
W20260709 12:24:53.216954 18364 rpcz_store.cc:269] 0709 12:24:51.417341 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:51.417461 (+   120us) service_pool.cc:224] Handling call
0709 12:24:53.213272 (+1795811us) spinlock_profiling.cc:243] Waited 1.25 s on lock 0x7b5400802d54. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:24:53.213375 (+   103us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2870430464}
I20260709 12:24:53.163829 27097 raft_consensus.cc:515] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:53.206269 26728 raft_consensus.cc:493] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:24:53.218331 26581 raft_consensus.cc:2804] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:24:53.208156 26957 raft_consensus.cc:493] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:24:53.209810 27196 consensus_queue.cc:1048] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.009s
I20260709 12:24:53.210237 26924 raft_consensus.cc:493] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:24:53.215998 26962 raft_consensus.cc:3060] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:53.218757 26966 raft_consensus.cc:2804] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:24:53.260267 27124 raft_consensus.cc:2804] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:24:53.248189 26887 raft_consensus.cc:493] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:53.266230 26963 raft_consensus.cc:697] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 24 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:24:53.265727 26581 raft_consensus.cc:493] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:53.273572 26985 raft_consensus.cc:2749] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:24:53.272979 27149 raft_consensus.cc:493] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:53.217572 27135 raft_consensus.cc:515] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:53.268080 27188 raft_consensus.cc:493] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:53.286298 27181 log.cc:927] Time spent T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.312s	user 0.001s	sys 0.000s
I20260709 12:24:53.214582 26632 raft_consensus.cc:515] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:53.282989 26966 raft_consensus.cc:493] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:53.282904 27168 raft_consensus.cc:493] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
W20260709 12:24:53.298043 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 110148us
I20260709 12:24:53.288151 27124 raft_consensus.cc:493] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:53.287671 27012 leader_election.cc:290] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:53.296154 26581 raft_consensus.cc:3060] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:53.211110 27128 consensus_queue.cc:237] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 30.23, Last appended by leader: 23, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:53.305922 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 88491.5us
I20260709 12:24:53.280874 26714 consensus_queue.cc:1059] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.021s
I20260709 12:24:53.276968 26957 raft_consensus.cc:515] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:53.298882 27197 raft_consensus.cc:2804] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:24:53.308910 26966 raft_consensus.cc:3060] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Advancing to term 31
W20260709 12:24:53.305636 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 59577.8us
I20260709 12:24:53.272315 17682 raft_consensus.cc:2468] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 25.
I20260709 12:24:53.274099 26728 raft_consensus.cc:515] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:53.282759 26924 raft_consensus.cc:515] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:53.316283 17688 raft_consensus.cc:3060] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:53.318259 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:53.312588 27141 leader_election.cc:290] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 25 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:53.294682 26887 raft_consensus.cc:515] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:53.312584 27124 raft_consensus.cc:3060] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Advancing to term 25
W20260709 12:24:53.305821 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 86151.5us
I20260709 12:24:53.336872 17654 leader_election.cc:304] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:53.338160 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:53.321098 18333 leader_election.cc:304] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:53.309800 27049 consensus_queue.cc:1048] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.027s
I20260709 12:24:53.340658 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:53.342285 26888 leader_election.cc:290] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 23 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:24:53.385866 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 49062.3us
I20260709 12:24:53.346097 26728 leader_election.cc:290] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:53.346374 26632 leader_election.cc:290] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:53.346684 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:53.388104 17682 raft_consensus.cc:2368] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for earlier term 29. Current term is 30.
I20260709 12:24:53.389758 17654 leader_election.cc:304] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:53.355460 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:53.390992 17684 raft_consensus.cc:2468] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 26.
I20260709 12:24:53.304553 27188 raft_consensus.cc:515] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:53.355670 27119 consensus_meta.cc:287] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.566s	user 0.004s	sys 0.007s
I20260709 12:24:53.357416 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:53.302884 27149 raft_consensus.cc:515] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:53.336387 26714 consensus_queue.cc:1243] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer 334a4e1f59944544b3b3ff36616b57da log is divergent from this leader: its last log entry 28.19 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 18
I20260709 12:24:53.334395 27197 raft_consensus.cc:493] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
W20260709 12:24:53.392937 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 71881.7us
I20260709 12:24:53.312446 27168 raft_consensus.cc:515] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:53.421643 18367 raft_consensus.cc:1275] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 26 index: 19. Preceding OpId from leader: term: 28 index: 20. (index mismatch)
I20260709 12:24:53.422529 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 28 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:53.423822 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 27 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:53.424252 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 25 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:53.425024 18362 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 24.
I20260709 12:24:53.410612 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:53.426369 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:53.411955 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 27 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:53.362099 27199 consensus_queue.cc:1048] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.057s
I20260709 12:24:53.392055 27202 raft_consensus.cc:493] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:24:53.466028 27202 raft_consensus.cc:515] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:53.471129 26843 raft_consensus.cc:3060] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:53.396102 27141 raft_consensus.cc:2749] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:24:53.472963 27202 leader_election.cc:290] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:53.300864 27179 consensus_queue.cc:237] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 22.17, Last appended by leader: 17, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:53.475805 27051 raft_consensus.cc:493] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:53.476367 17803 leader_election.cc:304] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:53.476433 27051 raft_consensus.cc:515] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:53.477946 17803 leader_election.cc:304] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:24:53.478154 27168 leader_election.cc:290] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:53.478613 27197 raft_consensus.cc:3060] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:53.479193 17803 leader_election.cc:304] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:53.479324 26789 raft_consensus.cc:493] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:53.479933 26789 raft_consensus.cc:515] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:53.478890 27051 leader_election.cc:290] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:53.482465 26789 leader_election.cc:290] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:53.396479 27203 consensus_queue.cc:1048] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.033s
W20260709 12:24:53.396544 27093 consensus_meta.cc:287] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.778s	user 0.012s	sys 0.000s
I20260709 12:24:53.486279 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:53.427064 27062 raft_consensus.cc:515] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:53.490005 17836 raft_consensus.cc:1240] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 32 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 31. Current term is 32. Ops: []
I20260709 12:24:53.491017 27062 leader_election.cc:290] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 32 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:53.494397 26581 raft_consensus.cc:515] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:53.414278 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 27 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:53.496819 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 31 candidate_status { last_received { term: 30 index: 23 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:53.498301 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 32 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:53.499357 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:53.500377 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 25 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:53.501394 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 28 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:53.321308 27012 consensus_queue.cc:1048] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.066s
I20260709 12:24:53.415067 17688 raft_consensus.cc:2410] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for term 27 because replica has last-logged OpId of term: 26 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 18.
I20260709 12:24:53.398273 17654 leader_election.cc:304] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:53.429854 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 25 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:53.522958 17761 raft_consensus.cc:2393] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 25: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:24:53.430949 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: true dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:53.523983 17764 raft_consensus.cc:3055] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 25 LEADER]: Stepping down as leader of term 25
W20260709 12:24:53.531366 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 133522us
W20260709 12:24:53.524344 17655 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:53.524345 17764 raft_consensus.cc:740] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 25 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
W20260709 12:24:53.431102 18333 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:53.431977 27149 leader_election.cc:290] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:53.432592 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:53.433096 18363 raft_consensus.cc:1240] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 22. Current term is 23. Ops: []
I20260709 12:24:53.399827 27201 raft_consensus.cc:2804] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:24:53.407820 26866 leader_election.cc:290] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 30 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:53.439073 27135 leader_election.cc:290] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:53.440214 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:53.442283 26957 leader_election.cc:290] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:53.340706 26963 consensus_queue.cc:237] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 23.16, Last appended by leader: 16, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:53.443607 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:53.443976 26887 leader_election.cc:290] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:53.530696 27039 raft_consensus.cc:2749] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:24:53.454589 26844 leader_election.cc:290] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:53.456714 27188 leader_election.cc:290] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:53.460500 27097 leader_election.cc:290] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:53.385329 27169 leader_election.cc:290] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:53.400694 26632 raft_consensus.cc:2749] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:24:53.516067 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:53.669914 18365 raft_consensus.cc:2468] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 22.
I20260709 12:24:53.538918 17764 consensus_queue.cc:260] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 25.19, Last appended by leader: 19, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:53.671557 17764 raft_consensus.cc:3060] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Advancing to term 26
W20260709 12:24:53.523761 26818 consensus_meta.cc:287] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.763s	user 0.007s	sys 0.003s
I20260709 12:24:53.546746 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:24:53.524284 27129 consensus_meta.cc:287] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.523s	user 0.005s	sys 0.004s
I20260709 12:24:53.539867 26910 raft_consensus.cc:515] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:53.684876 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:53.685750 17837 raft_consensus.cc:2468] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 22.
I20260709 12:24:53.524179 17802 leader_election.cc:304] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:53.554801 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 25 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:24:53.621409 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:53.577821 17682 raft_consensus.cc:1240] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 24. Current term is 25. Ops: []
I20260709 12:24:53.445987 26924 leader_election.cc:290] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:53.530181 26962 raft_consensus.cc:2749] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:24:53.646512 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 24 candidate_status { last_received { term: 23 index: 19 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:53.529671 27134 raft_consensus.cc:2749] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
W20260709 12:24:53.524210 27107 consensus_meta.cc:287] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.751s	user 0.012s	sys 0.000s
I20260709 12:24:53.540442 27044 raft_consensus.cc:2749] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:24:53.543179 27004 raft_consensus.cc:2804] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Leader pre-election won for term 25
W20260709 12:24:53.664402 17685 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.532s	user 0.060s	sys 0.080s
W20260709 12:24:53.703092 17685 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.571s	user 0.061s	sys 0.080s
I20260709 12:24:53.703166 27004 raft_consensus.cc:493] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:53.550246 26581 leader_election.cc:290] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 25 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:24:53.624598 27170 consensus_meta.cc:287] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.857s	user 0.009s	sys 0.002s
I20260709 12:24:53.703397 27107 raft_consensus.cc:515] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:53.705011 27004 raft_consensus.cc:3060] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:53.663650 27093 raft_consensus.cc:493] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:53.711707 27107 leader_election.cc:290] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 25 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:53.551688 27183 raft_consensus.cc:493] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:53.551167 27201 raft_consensus.cc:493] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:24:53.690141 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 121503us
W20260709 12:24:53.691385 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 167297us
I20260709 12:24:53.700139 18367 raft_consensus.cc:1240] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 21. Current term is 22. Ops: []
I20260709 12:24:53.704308 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 24 candidate_status { last_received { term: 23 index: 19 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:53.705246 27170 raft_consensus.cc:515] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:53.677079 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 118831us
I20260709 12:24:53.712478 27093 raft_consensus.cc:515] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:53.540700 27105 raft_consensus.cc:2749] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260709 12:24:53.677848 18371 raft_consensus.cc:2468] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 26.
I20260709 12:24:53.747347 17764 raft_consensus.cc:2468] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 26.
I20260709 12:24:53.717916 17833 raft_consensus.cc:2393] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 26: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:24:53.724355 18333 leader_election.cc:304] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:53.728673 17728 leader_election.cc:304] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:53.683686 26844 raft_consensus.cc:493] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
W20260709 12:24:53.741649 17655 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:53.540773 27154 consensus_queue.cc:1048] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 18, Time since last communication: 0.149s
W20260709 12:24:53.749832 17654 leader_election.cc:336] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Timed out: RequestConsensusVote RPC to 127.16.29.68:42577 timed out after 2.004s (SENT)
I20260709 12:24:53.750283 17728 leader_election.cc:304] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:53.767333 17686 raft_consensus.cc:1240] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 23 LEADER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 22. Current term is 23. Ops: []
I20260709 12:24:53.768085 17728 leader_election.cc:304] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
W20260709 12:24:53.732139 18334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:53.769867 18334 leader_election.cc:304] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:53.771271 17654 leader_election.cc:304] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:53.772709 27044 raft_consensus.cc:3060] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:53.714779 27062 raft_consensus.cc:2804] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:24:53.684527 26818 raft_consensus.cc:515] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:53.799819 17683 raft_consensus.cc:1240] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 23. Current term is 24. Ops: []
I20260709 12:24:53.805207 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 25 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:53.800099 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: true dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:53.806205 17685 raft_consensus.cc:3060] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:53.801816 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 29 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:53.541181 27111 raft_consensus.cc:515] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:53.764575 26581 raft_consensus.cc:2804] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:24:53.717599 27008 raft_consensus.cc:493] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
W20260709 12:24:53.807966 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 83689us
I20260709 12:24:53.745383 27201 raft_consensus.cc:3060] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:53.809798 18334 leader_election.cc:304] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:53.799038 26957 raft_consensus.cc:2749] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:24:53.812626 18333 leader_election.cc:304] T a5732e71b9d54f8786b8c511d69943bf P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:53.815932 17831 raft_consensus.cc:1240] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 21 LEADER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 20. Current term is 21. Ops: [20.15-20.15]
I20260709 12:24:53.832453 26910 leader_election.cc:290] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 26 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:53.835391 27044 raft_consensus.cc:2749] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:24:53.836529 17685 raft_consensus.cc:2468] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 26.
I20260709 12:24:53.840525 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:53.841113 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:53.843358 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 29 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:53.713534 27183 raft_consensus.cc:515] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:53.841219 17686 raft_consensus.cc:2468] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 22.
I20260709 12:24:53.848729 17835 raft_consensus.cc:1240] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 27. Current term is 28. Ops: []
I20260709 12:24:53.847384 18368 raft_consensus.cc:1240] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 27. Current term is 28. Ops: [27.16-27.16]
I20260709 12:24:53.850132 18334 leader_election.cc:304] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:53.810389 27097 raft_consensus.cc:2749] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:24:53.856494 18333 leader_election.cc:304] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:53.765715 27039 consensus_queue.cc:1059] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 24.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.056s
I20260709 12:24:53.811589 27040 raft_consensus.cc:2749] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:24:53.882097 27135 raft_consensus.cc:2749] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:24:53.845701 26581 raft_consensus.cc:697] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 25 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:24:53.884528 26581 consensus_queue.cc:237] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 24.17, Last appended by leader: 17, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:53.886176 27126 raft_consensus.cc:3060] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:53.890329 17756 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38786 (request call id 4014) took 1077 ms. Trace:
I20260709 12:24:53.893255 27094 consensus_queue.cc:260] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 27.19, Last appended by leader: 19, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:53.894340 27094 raft_consensus.cc:3060] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:24:53.844977 27008 raft_consensus.cc:515] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:53.896107 26924 raft_consensus.cc:2749] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:24:53.896425 27008 leader_election.cc:290] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:53.897040 17682 raft_consensus.cc:1240] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 21. Current term is 22. Ops: []
I20260709 12:24:53.893971 27201 raft_consensus.cc:515] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:53.897996 18361 raft_consensus.cc:2393] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 26: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:24:53.898260 17682 raft_consensus.cc:1240] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 20. Current term is 21. Ops: []
I20260709 12:24:53.899485 17682 raft_consensus.cc:1240] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 24. Current term is 25. Ops: []
I20260709 12:24:53.901038 27170 leader_election.cc:290] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 24 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:53.904990 27201 leader_election.cc:290] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 26 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:53.906749 27004 raft_consensus.cc:515] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:53.908031 17829 raft_consensus.cc:2393] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 25: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:24:53.908104 17837 raft_consensus.cc:2468] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 28.
I20260709 12:24:53.909611 27200 raft_consensus.cc:2749] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
W20260709 12:24:53.910394 27139 consensus_meta.cc:287] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.708s	user 0.007s	sys 0.003s
I20260709 12:24:53.910837 27139 raft_consensus.cc:515] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:53.911207 27093 leader_election.cc:290] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:24:53.859462 27187 log.cc:927] Time spent T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.540s	user 0.000s	sys 0.004s
I20260709 12:24:53.797791 26844 raft_consensus.cc:515] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:53.914465 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 25 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:53.814396 27107 raft_consensus.cc:2804] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:24:53.890715 17756 rpcz_store.cc:276] 0709 12:24:52.812948 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:52.813733 (+   785us) service_pool.cc:224] Handling call
0709 12:24:52.814047 (+   314us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:24:52.822649 (+  8602us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:52.822677 (+    28us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:24:52.836145 (+ 13468us) log.cc:844] Serialized 52 byte log entry
0709 12:24:53.465762 (+629617us) spinlock_profiling.cc:243] Waited 8.68 ms on lock 0x7b18003cbf20. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b47515e07 00007f7b503709a9 00007f7b4753269b 00007f7b4753093f 00007f7b47532c02 00007f7b4a1ba20b 00007f7b4a1bfd23 00007f7b4a1c00d1 00007f7b4a5dba0f
0709 12:24:53.644804 (+179042us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:24:53.830298 (+185494us) log.cc:844] Serialized 14 byte log entry
0709 12:24:53.848995 (+ 18697us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:53.849073 (+    78us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:53.890038 (+ 40965us) raft_consensus.cc:1685] finished
0709 12:24:53.890136 (+    98us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:53.890273 (+   137us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":31504,"prepare.queue_time_us":241098,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":42,"raft.queue_time_us":40977,"raft.run_cpu_time_us":260,"raft.run_wall_time_us":52796,"spinlock_wait_cycles":19971840,"thread_start_us":319082,"threads_started":2,"wal-append.queue_time_us":856512}
I20260709 12:24:53.882561 18333 leader_election.cc:304] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:53.814682 27062 raft_consensus.cc:697] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 23 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:24:53.863516 18334 leader_election.cc:304] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
W20260709 12:24:53.921036 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 57609.4us
W20260709 12:24:53.859699 27190 log.cc:927] Time spent T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.552s	user 0.003s	sys 0.000s
I20260709 12:24:53.922590 18334 leader_election.cc:304] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:24:53.860044 26962 raft_consensus.cc:2749] T a5732e71b9d54f8786b8c511d69943bf P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:24:53.890950 26818 leader_election.cc:290] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 28 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:53.945153 26635 consensus_queue.cc:1059] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.000s
I20260709 12:24:53.934018 27040 raft_consensus.cc:2804] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Leader pre-election won for term 23
W20260709 12:24:53.947625 26966 consensus_meta.cc:287] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.587s	user 0.007s	sys 0.001s
I20260709 12:24:53.948112 26966 raft_consensus.cc:515] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:53.950321 26966 leader_election.cc:290] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 31 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:53.927407 27200 raft_consensus.cc:2749] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:24:53.928706 27105 raft_consensus.cc:493] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:53.951290 18366 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59918 (request call id 2694) took 2534 ms (client timeout 2004 ms). Trace:
I20260709 12:24:53.951792 27105 raft_consensus.cc:515] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:24:53.951825 18366 rpcz_store.cc:269] 0709 12:24:51.416999 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:51.426759 (+  9760us) service_pool.cc:224] Handling call
0709 12:24:53.950979 (+2524220us) spinlock_profiling.cc:243] Waited 1.63 s on lock 0x7b54008020d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:24:53.951117 (+   138us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3751774464}
I20260709 12:24:53.926932 27204 raft_consensus.cc:493] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:53.929000 26963 raft_consensus.cc:2804] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:24:53.943584 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:53.944486 27214 consensus_queue.cc:1059] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
W20260709 12:24:53.944990 27192 log.cc:927] Time spent T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.660s	user 0.003s	sys 0.000s
I20260709 12:24:53.940327 27111 leader_election.cc:290] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 28 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:53.954929 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
W20260709 12:24:53.957858 17728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:53.976684 17682 raft_consensus.cc:2368] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for earlier term 29. Current term is 30.
I20260709 12:24:53.977802 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 32 candidate_status { last_received { term: 30 index: 23 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:53.977828 18364 raft_consensus.cc:1240] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 28 LEADER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 27. Current term is 28. Ops: [27.19-27.19]
W20260709 12:24:53.962917 17579 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35966 (request call id 357) took 84602 ms (client timeout 15000 ms). Trace:
I20260709 12:24:53.963151 27035 raft_consensus.cc:3055] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:24:53.963610 27204 raft_consensus.cc:515] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:53.961287 26818 raft_consensus.cc:493] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:24:53.970126 17687 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37402 (request call id 3791) took 1152 ms. Trace:
I20260709 12:24:54.006510 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:54.007438 17682 raft_consensus.cc:2410] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for term 27 because replica has last-logged OpId of term: 26 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 18.
I20260709 12:24:53.974093 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 25 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:54.008231 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:54.008725 18366 raft_consensus.cc:3060] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:53.924553 27107 raft_consensus.cc:493] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:53.978267 17729 leader_election.cc:304] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:53.998075 18371 raft_consensus.cc:2468] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 31.
I20260709 12:24:53.947829 27040 raft_consensus.cc:493] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:53.962251 26789 raft_consensus.cc:493] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:53.977061 27139 leader_election.cc:290] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 22 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:53.964702 27183 leader_election.cc:290] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:53.962070 27154 raft_consensus.cc:493] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:24:53.942086 26844 leader_election.cc:290] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:54.009545 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:53.928097 27201 raft_consensus.cc:493] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:54.076644 26887 raft_consensus.cc:493] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:54.005565 17687 rpcz_store.cc:276] 0709 12:24:52.817959 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:52.819113 (+  1154us) service_pool.cc:224] Handling call
0709 12:24:52.819432 (+   319us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:24:52.819826 (+   394us) raft_consensus.cc:1534] Early marking committed up to index 17
0709 12:24:52.819857 (+    31us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:24:52.834297 (+ 14440us) log.cc:844] Serialized 52 byte log entry
0709 12:24:52.988047 (+153750us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:24:52.988105 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:52.988167 (+    62us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:53.663157 (+674990us) spinlock_profiling.cc:243] Waited 1.98 ms on lock 0x7b4400022b4c. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b47459c85 00007f7b4745a6a2 00007f7b4a1cc1db 00007f7b4a1c7aaf 00007f7b4a1bb86e 00007f7b4a1babb6 00007f7b4a1ba7d8 00007f7b4a1c4221 00007f7b4a1c41d9 00007f7b4a1c4169
0709 12:24:53.969789 (+306632us) raft_consensus.cc:1685] finished
0709 12:24:53.969809 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:53.970040 (+   231us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":97,"raft.run_cpu_time_us":187,"raft.run_wall_time_us":67363,"spinlock_wait_cycles":4561792,"thread_start_us":57852,"threads_started":1,"wal-append.queue_time_us":323499}
I20260709 12:24:54.077488 27201 raft_consensus.cc:515] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:54.078419 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 32 candidate_status { last_received { term: 30 index: 23 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:54.079924 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:54.080590 17687 raft_consensus.cc:2468] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 17.
I20260709 12:24:54.010746 17729 leader_election.cc:304] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:24:54.004648 27124 consensus_meta.cc:287] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.609s	user 0.008s	sys 0.005s
I20260709 12:24:54.019757 17801 leader_election.cc:304] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:24:54.001865 27197 consensus_meta.cc:287] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.523s	user 0.008s	sys 0.010s
W20260709 12:24:54.021811 17730 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:54.020828 26789 raft_consensus.cc:515] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:54.026475 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:54.128087 18361 raft_consensus.cc:2468] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 27.
I20260709 12:24:54.015354 26818 raft_consensus.cc:515] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:54.027483 26843 consensus_meta.cc:287] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.556s	user 0.006s	sys 0.002s
I20260709 12:24:54.002463 27035 raft_consensus.cc:740] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:24:54.027838 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 29 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:54.015002 27097 raft_consensus.cc:2804] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:24:54.031719 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 24 candidate_status { last_received { term: 23 index: 19 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:54.004401 27062 consensus_queue.cc:237] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 21.14, Last appended by leader: 14, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:54.141868 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 25 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
W20260709 12:24:54.000934 17579 rpcz_store.cc:269] 0709 12:23:29.360542 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:29.360910 (+   368us) service_pool.cc:224] Handling call
0709 12:23:30.649878 (+1288968us) spinlock_profiling.cc:243] Waited 81.6 ms on lock 0x7b540010d454. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a58a090 00007f7b4a628f3a 00007f7b503dee9c 00007f7b504dd04e 00007f7b4e99e2a4 00007f7b4e99e232 00007f7b4e99e161 00007f7b4e99e0dc 00007f7b4e99d392 00007f7b4895b48c 00007f7b4895a8c6
0709 12:24:53.962831 (+83312953us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:24:47.416331 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:24:47.417178 (+   847us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:24:47.417234 (+    56us) write_op.cc:435] Acquired schema lock
0709 12:24:47.417262 (+    28us) tablet.cc:662] Decoding operations
0709 12:24:47.418513 (+  1251us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:24:47.418615 (+   102us) write_op.cc:641] Partition lock acquired for write op
0709 12:24:47.418648 (+    33us) tablet.cc:685] Acquiring locks for 11 operations
0709 12:24:47.419805 (+  1157us) tablet.cc:701] Row locks acquired
0709 12:24:47.419858 (+    53us) write_op.cc:260] PREPARE: finished
0709 12:24:47.559746 (+139888us) write_op.cc:270] Start()
0709 12:24:47.829029 (+269283us) spinlock_profiling.cc:243] Waited 8.03 ms on lock 0x7b540010d2a8. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b509dc9e1 00007f7b509db64c 00007f7b4e06b624 00007f7b4e056286 00007f7b4e055d58 00007f7b4e05a9f1 00007f7b4e05a9a9 00007f7b4e05a939 00007f7b4e05a901 00007f7b4e059b2d 0000000000388634
0709 12:24:47.829305 (+   276us) write_op.cc:276] Timestamp: P: 1783599887481798 usec, L: 0
0709 12:24:47.829350 (+    45us) op_driver.cc:348] REPLICATION: starting
0709 12:24:47.830401 (+  1051us) log.cc:844] Serialized 4427 byte log entry
0709 12:24:48.412419 (+582018us) op_driver.cc:464] REPLICATION: finished
0709 12:24:48.683612 (+271193us) write_op.cc:301] APPLY: starting
0709 12:24:48.693242 (+  9630us) tablet.cc:1367] starting BulkCheckPresence
0709 12:24:49.162989 (+469747us) tablet.cc:1370] finished BulkCheckPresence
0709 12:24:49.163124 (+   135us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:24:49.477782 (+314658us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:24:49.478179 (+   397us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=22,key_file_lookups=22,delta_file_lookups=0,mrs_lookups=0
0709 12:24:49.478266 (+    87us) write_op.cc:312] APPLY: finished
0709 12:24:49.478821 (+   555us) log.cc:844] Serialized 105 byte log entry
0709 12:24:49.479141 (+   320us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:24:49.479328 (+   187us) write_op.cc:454] Released schema lock
0709 12:24:49.479414 (+    86us) write_op.cc:341] FINISH: Updating metrics
0709 12:24:49.502239 (+ 22825us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:24:49.502286 (+    47us) write_op.cc:454] Released schema lock
0709 12:24:50.344441 (+842155us) spinlock_profiling.cc:243] Waited 5.72 ms on lock 0x7b18003e4460. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b47533522 00007f7b475357c1 00007f7b47535779 00007f7b47535709 00007f7b475356d1 00007f7b475349cd 0000000000388634 0000000000388479 00007f7b4751683a
Metrics: {"mutex_wait_us":3080255,"reported_tablets":46,"spinlock_wait_cycles":187672320,"child_traces":[["op",{"apply.queue_time_us":198083,"apply.run_cpu_time_us":23021,"apply.run_wall_time_us":835703,"cfile_cache_hit":46,"cfile_cache_hit_bytes":50151,"num_ops":11,"prepare.queue_time_us":359211,"prepare.run_cpu_time_us":70178,"prepare.run_wall_time_us":546575,"raft.queue_time_us":409616,"raft.run_cpu_time_us":5978,"raft.run_wall_time_us":156363,"replication_time_us":582875,"spinlock_wait_cycles":31624064,"thread_start_us":783496,"threads_started":4,"wal-append.queue_time_us":1035154,"wal-append.run_cpu_time_us":29208,"wal-append.run_wall_time_us":2336532}]]}
I20260709 12:24:54.039172 18362 raft_consensus.cc:1240] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 24. Current term is 25. Ops: []
I20260709 12:24:54.040922 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:54.016716 26963 raft_consensus.cc:697] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 26 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:24:54.015101 27040 raft_consensus.cc:3060] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:54.045962 27204 leader_election.cc:290] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:54.046868 27105 leader_election.cc:290] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:54.047266 27154 raft_consensus.cc:515] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:54.048066 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 28 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:54.056522 27126 consensus_queue.cc:1059] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 25.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:24:54.061825 17836 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35040 (request call id 3575) took 1477 ms. Trace:
I20260709 12:24:54.188483 17836 rpcz_store.cc:276] 0709 12:24:52.584576 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:52.584749 (+   173us) service_pool.cc:224] Handling call
0709 12:24:52.585067 (+   318us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:53.490399 (+905332us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:54.061522 (+571123us) spinlock_profiling.cc:243] Waited 905 ms on lock 0x7b5400335ed4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:24:54.061757 (+   235us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2080677760}
I20260709 12:24:54.066076 17727 leader_election.cc:304] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:54.127235 27168 raft_consensus.cc:2749] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:24:54.072299 17689 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37402 (request call id 3784) took 1274 ms. Trace:
I20260709 12:24:54.076555 27004 leader_election.cc:290] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 25 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:24:54.077067 27207 log.cc:927] Time spent T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.230s	user 0.000s	sys 0.002s
I20260709 12:24:54.082044 27124 raft_consensus.cc:515] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:54.090139 17578 catalog_manager.cc:5697] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 21 to 23. New cstate: current_term: 23 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:24:54.013487 27107 raft_consensus.cc:3060] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:54.081558 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:54.097054 17682 raft_consensus.cc:2368] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for earlier term 29. Current term is 30.
I20260709 12:24:54.101274 18366 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 25.
W20260709 12:24:54.081995 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 71349.3us
W20260709 12:24:54.091645 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 165938us
W20260709 12:24:54.088331 27219 log.cc:927] Time spent T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.062s	user 0.001s	sys 0.000s
I20260709 12:24:54.000313 27214 consensus_queue.cc:1243] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer 0da169ff82674dc1947055ce43de68a3 log is divergent from this leader: its last log entry 23.18 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 17
W20260709 12:24:54.124989 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 105269us
I20260709 12:24:54.125355 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:54.125816 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:54.081557 27201 leader_election.cc:290] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:54.127259 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:54.138800 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:54.140672 18363 raft_consensus.cc:3060] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:54.196187 27208 raft_consensus.cc:493] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:54.220928 27208 raft_consensus.cc:515] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:54.225023 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 32 candidate_status { last_received { term: 30 index: 23 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:54.226655 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:54.227389 17685 raft_consensus.cc:2468] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 22.
I20260709 12:24:54.227768 18365 raft_consensus.cc:1240] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 24. Current term is 25. Ops: [24.13-24.13]
I20260709 12:24:54.123912 26978 raft_consensus.cc:2804] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:24:54.229637 26978 raft_consensus.cc:493] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:24:54.230084 26978 raft_consensus.cc:3060] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:54.147395 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:54.232144 17834 raft_consensus.cc:3060] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Advancing to term 30
W20260709 12:24:54.233147 27185 log.cc:927] Time spent T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.957s	user 0.002s	sys 0.000s
W20260709 12:24:54.236315 17755 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.735s	user 0.017s	sys 0.025s
W20260709 12:24:54.236775 17755 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.735s	user 0.017s	sys 0.025s
I20260709 12:24:54.144284 17654 leader_election.cc:304] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:24:54.195451 17689 rpcz_store.cc:276] 0709 12:24:52.797668 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:52.797800 (+   132us) service_pool.cc:224] Handling call
0709 12:24:52.798093 (+   293us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:52.891191 (+ 93098us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:52.891226 (+    35us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:52.891303 (+    77us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:24:52.891352 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:54.072047 (+1180695us) spinlock_profiling.cc:243] Waited 92.9 ms on lock 0x7b54001388d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:24:54.072104 (+    57us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:54.072240 (+   136us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":213652864}
I20260709 12:24:54.244915 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:54.245836 17689 raft_consensus.cc:2393] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 31: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:24:54.209051 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 31 candidate_status { last_received { term: 30 index: 23 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:54.210625 18333 leader_election.cc:304] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:54.154661 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 24 candidate_status { last_received { term: 23 index: 19 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:54.163997 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 22 candidate_status { last_received { term: 21 index: 20 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:54.130625 26957 raft_consensus.cc:493] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:54.186734 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:54.151614 18371 raft_consensus.cc:3060] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:54.141070 27097 raft_consensus.cc:493] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:24:54.190512 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 124551us
I20260709 12:24:54.071240 17681 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37402 (request call id 3790) took 1264 ms. Trace:
I20260709 12:24:54.120144 27012 consensus_queue.cc:1048] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.032s
I20260709 12:24:54.148811 27205 raft_consensus.cc:493] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:54.150450 26910 raft_consensus.cc:493] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:54.213768 17761 raft_consensus.cc:2393] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 30: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:24:54.216404 17683 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37402 (request call id 3788) took 1417 ms. Trace:
I20260709 12:24:54.274474 17683 rpcz_store.cc:276] 0709 12:24:52.798789 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:52.800729 (+  1940us) service_pool.cc:224] Handling call
0709 12:24:52.800985 (+   256us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:53.804065 (+1003080us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:54.216183 (+412118us) spinlock_profiling.cc:243] Waited 999 ms on lock 0x7b54003467d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:24:54.216348 (+   165us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2296599936}
I20260709 12:24:54.217181 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:54.276347 17759 raft_consensus.cc:3060] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:54.257591 17757 raft_consensus.cc:1275] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 27 index: 19. Preceding OpId from leader: term: 28 index: 19. (term mismatch)
I20260709 12:24:54.277451 17757 pending_rounds.cc:85] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb: Aborting all ops after (but not including) 18
I20260709 12:24:54.277704 17757 pending_rounds.cc:107] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb: Aborting uncommitted NO_OP operation due to leader change: 27.19
I20260709 12:24:54.278070 17757 raft_consensus.cc:2889] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:54.202520 17578 catalog_manager.cc:5697] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 24 to 27, leader changed from e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66) to 0da169ff82674dc1947055ce43de68a3 (127.16.29.65). New cstate: current_term: 27 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:24:54.259732 17727 leader_election.cc:304] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:54.141358 27044 raft_consensus.cc:493] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:54.212697 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:54.213364 17756 raft_consensus.cc:3060] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:24:54.218175 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:24:54.262708 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 52205.2us
I20260709 12:24:54.251248 17681 rpcz_store.cc:276] 0709 12:24:52.806787 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:52.807010 (+   223us) service_pool.cc:224] Handling call
0709 12:24:52.807389 (+   379us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:52.807791 (+   402us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:52.807831 (+    40us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:52.808414 (+   583us) log.cc:844] Serialized 26 byte log entry
0709 12:24:52.971215 (+162801us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:24:53.712926 (+741711us) spinlock_profiling.cc:243] Waited 2.14 ms on lock 0x7b180037fe40. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b475012a6 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b47459c85 00007f7b4745a6a2 00007f7b4a1cc1db
0709 12:24:53.713445 (+   519us) spinlock_profiling.cc:243] Waited 17.6 ms on lock 0x7b4400022b4c. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b47459c85 00007f7b4745a6a2 00007f7b4a1cc1db 00007f7b4a1c7aaf 00007f7b4a1bb86e 00007f7b4a1babb6 00007f7b4a1ba7d8 00007f7b4a1c4221 00007f7b4a1c41d9 00007f7b4a1c4169
0709 12:24:54.070348 (+356903us) spinlock_profiling.cc:243] Waited 2.03 ms on lock 0x7b180037fe40. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b4a1c84bc 00007f7b4751648b 00007f7b503709a9 00007f7b4753269b 00007f7b4753093f 00007f7b47530455 00007f7b4e0183c7 00007f7b4a62a9a7 00007f7b4a63a0bf
0709 12:24:54.070548 (+   200us) log.cc:844] Serialized 14 byte log entry
0709 12:24:54.070832 (+   284us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:54.070937 (+   105us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:54.070986 (+    49us) raft_consensus.cc:1685] finished
0709 12:24:54.071004 (+    18us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:54.071167 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":328316,"prepare.run_cpu_time_us":69,"prepare.run_wall_time_us":2407,"raft.queue_time_us":5709,"raft.run_cpu_time_us":209,"raft.run_wall_time_us":12727,"spinlock_wait_cycles":50162176,"thread_start_us":329525,"threads_started":2,"wal-append.queue_time_us":385642}
I20260709 12:24:54.287562 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 32 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:54.253552 17834 raft_consensus.cc:2468] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 30.
I20260709 12:24:54.264160 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 28 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:54.288529 18333 leader_election.cc:304] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:54.077517 26887 raft_consensus.cc:515] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:54.265352 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 29 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:54.272673 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 28 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:54.208209 17687 raft_consensus.cc:2468] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 25.
I20260709 12:24:54.309067 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 26 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:54.280802 17680 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37402 (request call id 3796) took 1334 ms. Trace:
I20260709 12:24:54.310236 17680 rpcz_store.cc:276] 0709 12:24:52.946571 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:52.946970 (+   399us) service_pool.cc:224] Handling call
0709 12:24:54.280564 (+1333594us) spinlock_profiling.cc:243] Waited 38.8 ms on lock 0x7b5400207954. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:24:54.280727 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":89217280}
I20260709 12:24:54.312059 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 28 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:54.313406 17763 raft_consensus.cc:3060] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:24:54.127920 27197 raft_consensus.cc:515] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:54.275656 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 25 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:54.266355 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:54.281332 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:54.291160 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:54.193521 27163 raft_consensus.cc:2749] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:24:54.285629 17578 catalog_manager.cc:5697] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 3 to 26, leader changed from 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) to 0da169ff82674dc1947055ce43de68a3 (127.16.29.65). New cstate: current_term: 26 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:24:54.300388 17686 raft_consensus.cc:2393] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 30: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:24:54.315759 17756 raft_consensus.cc:2410] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for term 31 because replica has last-logged OpId of term: 30 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 29 index: 17.
I20260709 12:24:54.510860 17761 raft_consensus.cc:2393] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 24: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
W20260709 12:24:54.441135 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 152677us
I20260709 12:24:54.514421 17578 catalog_manager.cc:5697] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 17 to 27, leader changed from 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) to 334a4e1f59944544b3b3ff36616b57da (127.16.29.68). New cstate: current_term: 27 leader_uuid: "334a4e1f59944544b3b3ff36616b57da" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 12:24:54.440351 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 120913us
I20260709 12:24:54.329631 17687 raft_consensus.cc:2410] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 26 because replica has last-logged OpId of term: 24 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 18.
I20260709 12:24:54.397374 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 25 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:54.333415 17757 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36572 (request call id 1664) took 1493 ms. Trace:
I20260709 12:24:54.523140 17757 rpcz_store.cc:276] 0709 12:24:52.839565 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:52.843160 (+  3595us) service_pool.cc:224] Handling call
0709 12:24:52.843464 (+   304us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:54.278521 (+1435057us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:54.327783 (+ 49262us) spinlock_profiling.cc:243] Waited 1.4 s on lock 0x7b54002b6454. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:24:54.333337 (+  5554us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3218639616}
I20260709 12:24:54.271317 26632 raft_consensus.cc:493] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:54.411556 17682 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37396 (request call id 3343) took 1267 ms. Trace:
W20260709 12:24:54.443648 26581 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.554s	user 0.041s	sys 0.055s
I20260709 12:24:54.256250 27004 raft_consensus.cc:493] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:54.273330 27097 raft_consensus.cc:3060] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:54.319471 17684 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44898 (request call id 1828) took 1319 ms. Trace:
W20260709 12:24:54.455983 27220 log.cc:927] Time spent T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.433s	user 0.004s	sys 0.000s
I20260709 12:24:54.409648 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 29 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:54.555362 17758 raft_consensus.cc:2468] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 28.
I20260709 12:24:54.431950 17680 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37396 (request call id 3344) took 1287 ms. Trace:
I20260709 12:24:54.532392 18333 leader_election.cc:304] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:24:54.458055 17802 leader_election.cc:304] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:24:54.201771 27124 leader_election.cc:290] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 25 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:54.538516 17682 rpcz_store.cc:276] 0709 12:24:53.143786 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:54.410088 (+1266302us) service_pool.cc:224] Handling call
0709 12:24:54.410453 (+   365us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:54.410831 (+   378us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:54.410895 (+    64us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:54.410986 (+    91us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:24:54.411050 (+    64us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:54.411160 (+   110us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:54.411460 (+   300us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:54.556062 17680 rpcz_store.cc:276] 0709 12:24:53.144130 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:54.430892 (+1286762us) service_pool.cc:224] Handling call
0709 12:24:54.431300 (+   408us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:54.431517 (+   217us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:54.431554 (+    37us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:54.431609 (+    55us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:24:54.431667 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:54.431736 (+    69us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:54.431860 (+   124us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:54.539789 17689 raft_consensus.cc:1240] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 22. Current term is 23. Ops: []
I20260709 12:24:54.558127 17689 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37396 (request call id 3345) took 1413 ms. Trace:
I20260709 12:24:54.558585 17689 rpcz_store.cc:276] 0709 12:24:53.144450 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:54.539044 (+1394594us) service_pool.cc:224] Handling call
0709 12:24:54.539571 (+   527us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:54.557816 (+ 18245us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:54.558056 (+   240us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:54.511569 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:54.525264 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:54.560266 17757 raft_consensus.cc:2393] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 27: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:24:54.549324 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:54.550165 17687 raft_consensus.cc:1240] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 21. Current term is 22. Ops: []
I20260709 12:24:54.573375 17687 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37396 (request call id 3348) took 1428 ms. Trace:
I20260709 12:24:54.574049 17687 rpcz_store.cc:276] 0709 12:24:53.145408 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:54.549567 (+1404159us) service_pool.cc:224] Handling call
0709 12:24:54.549842 (+   275us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:54.573021 (+ 23179us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:54.573303 (+   282us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:54.437752 26924 raft_consensus.cc:493] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:54.211649 27008 raft_consensus.cc:3055] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:24:54.553429 17684 rpcz_store.cc:276] 0709 12:24:53.000253 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:53.536042 (+535789us) service_pool.cc:224] Handling call
0709 12:24:54.281431 (+745389us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:54.281671 (+   240us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:54.281707 (+    36us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:54.281763 (+    56us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:24:54.281813 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:54.281883 (+    70us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:54.319252 (+ 37369us) spinlock_profiling.cc:243] Waited 726 ms on lock 0x7b5400207950. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a61f8ad 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:24:54.319397 (+   145us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1670871424}
W20260709 12:24:54.429282 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 169650us
I20260709 12:24:54.282142 27094 consensus_queue.cc:1059] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 23.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.277s
I20260709 12:24:54.517210 17578 catalog_manager.cc:5697] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 19 to 26, leader changed from 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) to 0da169ff82674dc1947055ce43de68a3 (127.16.29.65). New cstate: current_term: 26 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:24:54.557423 17682 raft_consensus.cc:1240] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 25. Current term is 26. Ops: []
I20260709 12:24:54.548564 26599 raft_consensus.cc:493] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:54.594949 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 450719us
I20260709 12:24:54.442458 27189 raft_consensus.cc:2749] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:24:54.443130 26956 raft_consensus.cc:493] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:54.581319 26962 raft_consensus.cc:2804] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:24:54.589852 27008 raft_consensus.cc:740] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 24 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
W20260709 12:24:54.565474 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 107523us
I20260709 12:24:54.617367 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 22 candidate_status { last_received { term: 21 index: 20 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:54.619789 18367 raft_consensus.cc:1275] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 30 index: 23. Preceding OpId from leader: term: 31 index: 24. (index mismatch)
I20260709 12:24:54.611750 18363 raft_consensus.cc:2468] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 24.
I20260709 12:24:54.455566 27214 raft_consensus.cc:2749] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:24:54.549185 17683 raft_consensus.cc:1240] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 27. Current term is 28. Ops: []
I20260709 12:24:54.393851 27201 raft_consensus.cc:493] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:54.630856 17683 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37396 (request call id 3346) took 1486 ms. Trace:
I20260709 12:24:54.631060 27201 raft_consensus.cc:515] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:54.631264 17683 rpcz_store.cc:276] 0709 12:24:53.144766 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:54.548671 (+1403905us) service_pool.cc:224] Handling call
0709 12:24:54.549018 (+   347us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:54.630595 (+ 81577us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:54.630799 (+   204us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:54.632717 17683 raft_consensus.cc:1240] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 20. Current term is 21. Ops: [19.14-20.15]
I20260709 12:24:54.582068 17689 raft_consensus.cc:1275] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 24 index: 19. Preceding OpId from leader: term: 25 index: 19. (term mismatch)
W20260709 12:24:54.551483 26581 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.661s	user 0.050s	sys 0.055s
I20260709 12:24:54.634419 17689 pending_rounds.cc:85] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3: Aborting all ops after (but not including) 18
I20260709 12:24:54.582638 17761 raft_consensus.cc:1240] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 30. Current term is 31. Ops: []
I20260709 12:24:54.583791 17686 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37396 (request call id 3352) took 1425 ms. Trace:
I20260709 12:24:54.634759 17680 raft_consensus.cc:1275] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 26 index: 18. Preceding OpId from leader: term: 28 index: 19. (index mismatch)
I20260709 12:24:54.588384 17763 raft_consensus.cc:2468] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 28.
I20260709 12:24:54.635033 17686 rpcz_store.cc:276] 0709 12:24:53.158218 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:54.550376 (+1392158us) service_pool.cc:224] Handling call
0709 12:24:54.583319 (+ 32943us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:54.583475 (+   156us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:54.583499 (+    24us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:54.583539 (+    40us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:24:54.583568 (+    29us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:54.583616 (+    48us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:54.583732 (+   116us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":36992}
I20260709 12:24:54.635962 27154 leader_election.cc:290] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:54.636068 27171 consensus_queue.cc:1059] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 23.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
I20260709 12:24:54.637215 27171 raft_consensus.cc:3055] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:24:54.634781 17689 pending_rounds.cc:107] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3: Aborting uncommitted NO_OP operation due to leader change: 24.19
I20260709 12:24:54.638175 17689 raft_consensus.cc:2889] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:54.428552 27035 consensus_queue.cc:260] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 20.15, Last appended by leader: 15, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:54.593645 17682 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37396 (request call id 3353) took 1435 ms. Trace:
I20260709 12:24:54.640295 17682 rpcz_store.cc:276] 0709 12:24:53.158583 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:54.557010 (+1398427us) service_pool.cc:224] Handling call
0709 12:24:54.557298 (+   288us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:54.593374 (+ 36076us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:54.593589 (+   215us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:54.583791 17687 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44898 (request call id 1845) took 1035 ms. Trace:
I20260709 12:24:54.641400 17687 rpcz_store.cc:276] 0709 12:24:53.547975 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:54.582989 (+1035014us) service_pool.cc:224] Handling call
0709 12:24:54.583319 (+   330us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:54.583484 (+   165us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:54.583506 (+    22us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:54.583539 (+    33us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:24:54.583568 (+    29us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:54.583616 (+    48us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:54.583732 (+   116us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":43904}
I20260709 12:24:54.642795 17682 raft_consensus.cc:1240] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 30. Current term is 31. Ops: [28.17-30.18]
I20260709 12:24:54.562444 27004 raft_consensus.cc:515] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:54.286540 27208 leader_election.cc:290] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:54.644268 17687 raft_consensus.cc:1275] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 16 index: 13. Preceding OpId from leader: term: 21 index: 15. (index mismatch)
I20260709 12:24:54.319901 27139 consensus_queue.cc:1059] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.254s
I20260709 12:24:54.290500 26910 raft_consensus.cc:515] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:54.275584 26957 raft_consensus.cc:515] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:54.658139 27208 raft_consensus.cc:493] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:54.658672 27208 raft_consensus.cc:515] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:54.659915 27197 leader_election.cc:290] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 24 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:54.660490 26581 raft_consensus.cc:3055] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:24:54.661106 26581 raft_consensus.cc:740] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:24:54.661895 26581 consensus_queue.cc:260] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 23.17, Last appended by leader: 17, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:54.662649 17760 raft_consensus.cc:1275] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 30 index: 23. Preceding OpId from leader: term: 31 index: 24. (index mismatch)
I20260709 12:24:54.663184 26581 raft_consensus.cc:3060] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:54.606935 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 32 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
W20260709 12:24:54.601557 18374 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.570s	user 0.015s	sys 0.067s
W20260709 12:24:54.664234 18374 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.633s	user 0.015s	sys 0.068s
I20260709 12:24:54.282471 27205 raft_consensus.cc:515] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:54.667505 27205 leader_election.cc:290] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:54.668901 17685 raft_consensus.cc:2393] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 26: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:24:54.291514 27128 raft_consensus.cc:2764] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Leader pre-election decision vote started in defunct term 26: won
I20260709 12:24:54.585255 26843 raft_consensus.cc:2804] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:24:54.670269 26843 raft_consensus.cc:493] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:54.670771 26843 raft_consensus.cc:3060] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:54.393661 27044 raft_consensus.cc:515] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:54.672691 17681 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 26.
I20260709 12:24:54.550447 26632 raft_consensus.cc:515] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:54.674260 17681 raft_consensus.cc:1240] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 24. Current term is 25. Ops: []
I20260709 12:24:54.675444 17681 raft_consensus.cc:1240] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 21. Current term is 22. Ops: []
I20260709 12:24:54.675495 26632 leader_election.cc:290] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:54.600703 26599 raft_consensus.cc:515] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:54.676453 17684 raft_consensus.cc:1275] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 25 index: 14. Preceding OpId from leader: term: 27 index: 15. (index mismatch)
I20260709 12:24:54.640182 26818 leader_election.cc:290] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:54.638607 17761 raft_consensus.cc:1240] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 24. Current term is 25. Ops: []
I20260709 12:24:54.681736 26887 leader_election.cc:290] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:54.639182 17689 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44898 (request call id 1842) took 1096 ms. Trace:
I20260709 12:24:54.644218 17755 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36572 (request call id 1665) took 1804 ms. Trace:
I20260709 12:24:54.682505 17755 rpcz_store.cc:276] 0709 12:24:52.846090 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:52.892975 (+ 46885us) service_pool.cc:224] Handling call
0709 12:24:52.893303 (+   328us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:53.002804 (+109501us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:53.002855 (+    51us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:53.500764 (+497909us) spinlock_profiling.cc:243] Waited 128 ms on lock 0x7b2c0000e5c8. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b48f19a11 00007f7b48f29372 00007f7b4a65ba01 00007f7b4a62043a 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2
0709 12:24:53.501175 (+   411us) log.cc:844] Serialized 26 byte log entry
0709 12:24:54.120694 (+619519us) spinlock_profiling.cc:243] Waited 4.73 ms on lock 0x7b4400022b4c. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b47459c85 00007f7b4745a6a2 00007f7b4a1cc1db 00007f7b4a1c7aaf 00007f7b4a1bb86e 00007f7b4a1babb6 00007f7b4a1ba7d8 00007f7b4a1c4221 00007f7b4a1c41d9 00007f7b4a1c4169
0709 12:24:54.237222 (+116528us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:24:54.237287 (+    65us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:54.237389 (+   102us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:54.644033 (+406644us) raft_consensus.cc:1685] finished
0709 12:24:54.644051 (+    18us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:54.644166 (+   115us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":4494,"raft.run_cpu_time_us":82,"raft.run_wall_time_us":86,"spinlock_wait_cycles":304905216,"thread_start_us":825041,"threads_started":2,"wal-append.queue_time_us":521852}
I20260709 12:24:54.685040 17689 rpcz_store.cc:276] 0709 12:24:53.542708 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:54.581300 (+1038592us) service_pool.cc:224] Handling call
0709 12:24:54.581648 (+   348us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:54.638943 (+ 57295us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:54.639129 (+   186us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:54.686281 26887 raft_consensus.cc:493] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:54.561093 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 25 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:54.687420 26789 leader_election.cc:290] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:54.688014 26887 raft_consensus.cc:515] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:54.689641 27004 leader_election.cc:290] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:54.690824 26887 leader_election.cc:290] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:54.690632 27008 consensus_queue.cc:260] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 24.17, Last appended by leader: 17, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:54.687784 17756 raft_consensus.cc:2393] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 25: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:24:54.691787 27008 raft_consensus.cc:3060] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:54.645105 17682 raft_consensus.cc:1275] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 28 index: 16. Preceding OpId from leader: term: 30 index: 17. (index mismatch)
I20260709 12:24:54.512801 26963 consensus_queue.cc:237] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 25.19, Last appended by leader: 19, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:54.637615 27171 raft_consensus.cc:740] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 23 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
W20260709 12:24:54.601845 26985 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.579s	user 0.042s	sys 0.049s
W20260709 12:24:54.696911 26985 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.674s	user 0.042s	sys 0.049s
I20260709 12:24:54.696769 27171 consensus_queue.cc:260] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 23.19, Last appended by leader: 19, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:54.698093 27171 raft_consensus.cc:3060] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:54.644780 26956 raft_consensus.cc:515] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:54.642603 27221 consensus_queue.cc:1059] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 19.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.062s
I20260709 12:24:54.704524 27221 consensus_queue.cc:1243] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Peer e063541542a94969898868bf6eb5ce1b log is divergent from this leader: its last log entry 21.15 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 14
I20260709 12:24:54.704654 26910 leader_election.cc:290] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:54.705739 26956 leader_election.cc:290] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:54.594941 17578 catalog_manager.cc:5697] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 18 to 21. New cstate: current_term: 21 leader_uuid: "334a4e1f59944544b3b3ff36616b57da" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:24:54.637948 27102 raft_consensus.cc:2749] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:24:54.624740 26962 raft_consensus.cc:493] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:54.594710 27202 consensus_queue.cc:1059] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.045s
I20260709 12:24:54.597986 27124 consensus_queue.cc:1059] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.042s
I20260709 12:24:54.660406 26957 leader_election.cc:290] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:54.546694 18371 raft_consensus.cc:2468] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 25.
I20260709 12:24:54.708501 17578 catalog_manager.cc:5697] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 16 to 22, leader changed from e063541542a94969898868bf6eb5ce1b (127.16.29.67) to 0da169ff82674dc1947055ce43de68a3 (127.16.29.65). New cstate: current_term: 22 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
W20260709 12:24:54.710081 17802 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:54.712157 17759 raft_consensus.cc:2468] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 26.
I20260709 12:24:54.592084 26924 raft_consensus.cc:515] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:54.713125 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 343069us
W20260709 12:24:54.750551 17803 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:54.731711 18334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:54.684333 27215 raft_consensus.cc:3055] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:24:54.644739 27214 raft_consensus.cc:493] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
W20260709 12:24:54.753466 17688 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.571s	user 0.009s	sys 0.022s
I20260709 12:24:54.752749 17578 catalog_manager.cc:5697] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 18 to 24. New cstate: current_term: 24 leader_uuid: "334a4e1f59944544b3b3ff36616b57da" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:24:54.724139 27202 consensus_queue.cc:1243] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer e063541542a94969898868bf6eb5ce1b log is divergent from this leader: its last log entry 21.14 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 13
W20260709 12:24:54.755925 17688 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.574s	user 0.009s	sys 0.023s
I20260709 12:24:54.758880 17688 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44898 (request call id 1830) took 1753 ms. Trace:
I20260709 12:24:54.759598 18366 raft_consensus.cc:1275] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 22 index: 17. Preceding OpId from leader: term: 23 index: 18. (index mismatch)
I20260709 12:24:54.749745 26818 raft_consensus.cc:3055] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:24:54.674263 27044 leader_election.cc:290] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:54.715384 26962 raft_consensus.cc:3060] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Advancing to term 29
W20260709 12:24:54.762552 17762 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.519s	user 0.010s	sys 0.005s
I20260709 12:24:54.753405 27215 raft_consensus.cc:740] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
W20260709 12:24:54.767571 17762 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.524s	user 0.010s	sys 0.005s
I20260709 12:24:54.768253 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:54.769305 17682 raft_consensus.cc:1240] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 24. Current term is 25. Ops: [24.18-24.18]
I20260709 12:24:54.769445 17687 raft_consensus.cc:3060] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:54.755542 27004 raft_consensus.cc:493] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:54.774313 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:54.776875 17685 raft_consensus.cc:2393] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 25: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:24:54.778323 27208 leader_election.cc:290] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:54.778944 27215 consensus_queue.cc:260] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 24.18, Last appended by leader: 18, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:54.780089 27215 raft_consensus.cc:3060] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Advancing to term 25
W20260709 12:24:54.765693 27126 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.567s	user 0.004s	sys 0.029s
I20260709 12:24:54.762055 26818 raft_consensus.cc:740] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
W20260709 12:24:54.794840 17764 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.519s	user 0.004s	sys 0.014s
I20260709 12:24:54.753539 26843 raft_consensus.cc:515] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:54.797860 17764 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.522s	user 0.004s	sys 0.014s
I20260709 12:24:54.757297 17578 catalog_manager.cc:5697] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 15 to 22, leader changed from 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) to e063541542a94969898868bf6eb5ce1b (127.16.29.67). New cstate: current_term: 22 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:24:54.808256 17578 catalog_manager.cc:5697] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 20 to 22. New cstate: current_term: 22 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:24:54.809738 17578 catalog_manager.cc:5697] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 20 to 25, leader changed from 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) to e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66). New cstate: current_term: 25 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:24:54.826534 17801 leader_election.cc:304] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:54.802431 17654 leader_election.cc:304] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:24:54.765357 27201 leader_election.cc:290] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:54.832208 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:54.833197 18363 raft_consensus.cc:2368] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for earlier term 24. Current term is 25.
I20260709 12:24:54.833350 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:54.834174 18367 raft_consensus.cc:2468] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 26.
I20260709 12:24:54.837868 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:54.838593 18367 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 26.
W20260709 12:24:54.793931 27126 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.595s	user 0.005s	sys 0.029s
W20260709 12:24:54.849200 17730 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:54.759752 27214 raft_consensus.cc:515] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:54.850919 26957 raft_consensus.cc:493] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:24:54.851595 26957 raft_consensus.cc:515] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:54.854228 26957 leader_election.cc:290] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:54.856585 27214 leader_election.cc:290] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:54.858863 26957 raft_consensus.cc:493] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:54.859892 27201 consensus_queue.cc:1059] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 16.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20260709 12:24:54.816681 27208 consensus_queue.cc:1059] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 24.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.045s
I20260709 12:24:54.861965 27208 raft_consensus.cc:3055] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:24:54.862388 27208 raft_consensus.cc:740] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:24:54.859637 26957 raft_consensus.cc:515] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:54.847554 26887 raft_consensus.cc:493] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:54.847029 17801 leader_election.cc:304] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:24:54.855649 26978 consensus_meta.cc:287] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.625s	user 0.010s	sys 0.000s
I20260709 12:24:54.867241 17801 leader_election.cc:304] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:54.868443 17687 raft_consensus.cc:2468] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 25.
I20260709 12:24:54.868566 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:54.870558 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 28 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:54.883188 17759 raft_consensus.cc:2468] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 22.
W20260709 12:24:54.883683 17728 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.586s	user 0.034s	sys 0.066s
W20260709 12:24:54.884150 17728 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.586s	user 0.034s	sys 0.066s
I20260709 12:24:54.872542 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:54.874547 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 31 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:54.863348 27208 consensus_queue.cc:260] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 24.15, Last appended by leader: 15, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:54.864765 26887 raft_consensus.cc:515] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:54.889156 17761 raft_consensus.cc:2468] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 30.
I20260709 12:24:54.888275 17755 raft_consensus.cc:2468] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 23.
I20260709 12:24:54.870802 17578 catalog_manager.cc:5697] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 19 to 24, leader changed from e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66) to 0da169ff82674dc1947055ce43de68a3 (127.16.29.65). New cstate: current_term: 24 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:24:54.884909 26818 consensus_queue.cc:260] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 27.17, Last appended by leader: 17, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:54.846382 27044 consensus_queue.cc:1048] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.063s
I20260709 12:24:54.813249 27012 consensus_queue.cc:1059] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 22.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.077s
I20260709 12:24:54.847671 27128 raft_consensus.cc:2804] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:24:54.845978 26910 raft_consensus.cc:493] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:54.759543 17688 rpcz_store.cc:276] 0709 12:24:53.005410 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:53.889163 (+883753us) service_pool.cc:224] Handling call
0709 12:24:53.889443 (+   280us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:53.889629 (+   186us) raft_consensus.cc:1534] Early marking committed up to index 16
0709 12:24:54.181916 (+292287us) log.cc:844] Serialized 14 byte log entry
0709 12:24:54.758422 (+576506us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:54.758492 (+    70us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:24:54.758555 (+    63us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:54.758645 (+    90us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:54.758809 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":748868,"prepare.run_cpu_time_us":63,"prepare.run_wall_time_us":29550,"raft.queue_time_us":104,"thread_start_us":868032,"threads_started":2}
I20260709 12:24:54.848917 26985 raft_consensus.cc:2749] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:24:54.928696 17680 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44898 (request call id 1841) took 1387 ms. Trace:
I20260709 12:24:54.986380 17680 rpcz_store.cc:276] 0709 12:24:53.541814 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:54.580085 (+1038271us) service_pool.cc:224] Handling call
0709 12:24:54.634422 (+ 54337us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:54.635167 (+   745us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:54.928375 (+293208us) spinlock_profiling.cc:243] Waited 53.9 ms on lock 0x7b54001aac50. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a61f8ad 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:24:54.928605 (+   230us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":124025600}
I20260709 12:24:54.930352 17684 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37396 (request call id 3358) took 1014 ms. Trace:
W20260709 12:24:54.933262 17728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:54.890945 17764 raft_consensus.cc:1275] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 28 index: 16. Preceding OpId from leader: term: 30 index: 17. (index mismatch)
W20260709 12:24:54.936964 27040 consensus_meta.cc:287] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.742s	user 0.011s	sys 0.000s
I20260709 12:24:54.786965 27004 raft_consensus.cc:515] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:54.885998 27107 consensus_meta.cc:287] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.682s	user 0.009s	sys 0.001s
W20260709 12:24:54.990646 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 426087us
W20260709 12:24:54.991353 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 472578us
I20260709 12:24:54.886441 27008 consensus_queue.cc:1059] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 30.18, Next index: 19, Last known committed idx: 16, Time since last communication: 0.086s
I20260709 12:24:54.995663 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:54.996272 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 26 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:54.996649 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:54.997253 18363 raft_consensus.cc:2468] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 25.
I20260709 12:24:54.997819 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:54.998132 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:54.998725 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 28 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:55.000762 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
W20260709 12:24:55.001096 27225 log.cc:927] Time spent T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.056s	user 0.000s	sys 0.000s
I20260709 12:24:55.001443 18361 raft_consensus.cc:2393] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 26: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:24:55.002226 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 25 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:55.002871 18361 raft_consensus.cc:2393] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 25: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:24:54.971973 27230 raft_consensus.cc:493] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:54.993249 17729 leader_election.cc:304] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:54.882225 26888 raft_consensus.cc:2804] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:24:54.909665 17801 leader_election.cc:304] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:54.943910 17578 catalog_manager.cc:5697] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 14 to 25, leader changed from 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) to e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66). New cstate: current_term: 25 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:24:54.966918 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:54.910303 27208 raft_consensus.cc:3060] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Advancing to term 25
W20260709 12:24:54.971448 17652 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.723s	user 0.013s	sys 0.025s
W20260709 12:24:55.035133 17652 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.787s	user 0.013s	sys 0.025s
W20260709 12:24:55.035456 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 840843us
I20260709 12:24:54.907270 27154 raft_consensus.cc:2749] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
I20260709 12:24:54.987627 17684 rpcz_store.cc:276] 0709 12:24:53.915549 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:54.590324 (+674775us) service_pool.cc:224] Handling call
0709 12:24:54.606108 (+ 15784us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:54.676954 (+ 70846us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:54.930094 (+253140us) spinlock_profiling.cc:243] Waited 69.9 ms on lock 0x7b5400348ad4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:24:54.930283 (+   189us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":160851712}
I20260709 12:24:54.988482 17728 leader_election.cc:400] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 29 election: Vote denied by peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071) with higher term. Message: Invalid argument: T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for earlier term 29. Current term is 30.
W20260709 12:24:55.067060 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 78762.9us
I20260709 12:24:54.881291 26978 raft_consensus.cc:515] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:54.950995 27126 raft_consensus.cc:493] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:24:54.998618 18364 raft_consensus.cc:2468] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 18.
I20260709 12:24:55.072011 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:55.006114 18374 raft_consensus.cc:1240] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 23. Current term is 24. Ops: []
I20260709 12:24:55.073470 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:55.006889 16501 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:24:54.946597 17687 raft_consensus.cc:1240] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 29. Current term is 30. Ops: []
W20260709 12:24:55.027128 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 423848us
I20260709 12:24:54.903261 27097 raft_consensus.cc:515] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:54.951306 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:55.080410 17685 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 26.
I20260709 12:24:55.082765 18374 raft_consensus.cc:2393] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 28: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:24:55.083696 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 31 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:55.029927 27201 raft_consensus.cc:493] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:55.034139 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:55.084694 17728 leader_election.cc:304] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:55.040287 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 29 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:55.040505 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:55.041524 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:55.042009 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:55.087203 17836 raft_consensus.cc:2393] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 24: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:24:55.043045 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 28 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:55.030887 17578 catalog_manager.cc:5697] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 13 to 24, leader changed from 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) to e063541542a94969898868bf6eb5ce1b (127.16.29.67). New cstate: current_term: 24 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:24:55.088846 17578 catalog_manager.cc:5697] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 14 to 20. New cstate: current_term: 20 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:24:55.090061 17578 catalog_manager.cc:5697] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 17 to 22, leader changed from e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66) to 0da169ff82674dc1947055ce43de68a3 (127.16.29.65). New cstate: current_term: 22 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:24:55.044026 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:55.092159 17829 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 26.
I20260709 12:24:55.045053 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:55.093183 17759 raft_consensus.cc:2468] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 26.
W20260709 12:24:55.048002 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 138407us
I20260709 12:24:55.051124 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:55.095057 17831 raft_consensus.cc:2468] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 26.
I20260709 12:24:55.052981 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 29 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:54.995612 27205 raft_consensus.cc:3055] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:24:54.991102 27128 raft_consensus.cc:697] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 24 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:24:54.994515 26789 raft_consensus.cc:493] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:55.062476 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 28 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:55.062683 17652 leader_election.cc:304] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:55.059939 26985 raft_consensus.cc:2749] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 12:24:55.068485 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 29 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:54.992683 27107 raft_consensus.cc:515] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:54.990706 26910 raft_consensus.cc:515] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:55.034053 27111 raft_consensus.cc:3055] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:24:55.049741 27202 raft_consensus.cc:2804] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:24:55.067094 26888 raft_consensus.cc:697] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 28 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:24:55.074851 26632 raft_consensus.cc:493] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:24:55.094220 17801 leader_election.cc:304] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:55.099994 17728 leader_election.cc:304] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:55.101051 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:55.129907 17829 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 26.
I20260709 12:24:55.130728 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 30 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:55.130824 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:55.131297 17829 raft_consensus.cc:2468] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 29.
I20260709 12:24:55.133011 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:55.133590 17759 raft_consensus.cc:1240] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 27. Current term is 28. Ops: []
I20260709 12:24:55.133055 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 32 candidate_status { last_received { term: 30 index: 23 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:55.101073 17832 raft_consensus.cc:1275] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 25 index: 14. Preceding OpId from leader: term: 27 index: 15. (index mismatch)
I20260709 12:24:55.091280 17578 catalog_manager.cc:5697] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 16 to 21. New cstate: current_term: 21 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:24:55.082950 17834 raft_consensus.cc:1240] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 29. Current term is 30. Ops: []
I20260709 12:24:55.040302 17654 leader_election.cc:400] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 24 election: Vote denied by peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577) with higher term. Message: Invalid argument: T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for earlier term 24. Current term is 25.
I20260709 12:24:55.140208 17654 leader_election.cc:403] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 24 election: Cancelling election due to peer responding with higher term
W20260709 12:24:55.140673 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 108394us
I20260709 12:24:55.119206 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:55.141350 17654 leader_election.cc:304] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:24:55.140430 17578 catalog_manager.cc:5697] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 14 to 21. New cstate: current_term: 21 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:24:55.141672 17831 raft_consensus.cc:2468] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 23.
I20260709 12:24:54.992113 27040 raft_consensus.cc:515] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:55.148954 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:55.107782 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:55.084420 18374 raft_consensus.cc:2393] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 31: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:24:55.062486 27012 raft_consensus.cc:493] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:55.115985 17755 raft_consensus.cc:2468] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 26.
W20260709 12:24:55.116952 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 54369.8us
I20260709 12:24:55.106042 27215 raft_consensus.cc:493] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:55.066521 26957 leader_election.cc:290] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:24:55.161124 17804 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:55.109498 27235 raft_consensus.cc:2749] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 12:24:55.125380 27202 raft_consensus.cc:493] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:55.054867 27093 consensus_queue.cc:1059] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 21.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.071s
I20260709 12:24:55.146943 17758 raft_consensus.cc:2393] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 26: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:24:55.169440 27215 raft_consensus.cc:515] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:55.177788 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:55.150103 18362 raft_consensus.cc:2468] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 23.
I20260709 12:24:55.096855 27205 raft_consensus.cc:740] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:24:55.179438 27215 leader_election.cc:290] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:55.050595 27230 raft_consensus.cc:515] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:55.158435 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 58555.4us
I20260709 12:24:55.185335 17728 leader_election.cc:304] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:24:55.086416 17761 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 26.
I20260709 12:24:55.118597 27044 raft_consensus.cc:2749] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 12:24:55.188853 17728 leader_election.cc:400] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 29 election: Vote denied by peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071) with higher term. Message: Invalid argument: T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for earlier term 29. Current term is 30.
I20260709 12:24:55.190127 17728 leader_election.cc:304] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:55.134526 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 31 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:55.191061 17759 raft_consensus.cc:3060] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:24:55.096866 27126 raft_consensus.cc:515] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:55.170578 17756 raft_consensus.cc:1275] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 23 index: 16. Preceding OpId from leader: term: 24 index: 17. (index mismatch)
I20260709 12:24:55.172896 18365 raft_consensus.cc:2393] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 23: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:24:55.127193 26843 leader_election.cc:290] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 23 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:55.125759 27111 raft_consensus.cc:740] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 30 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:24:55.142364 17578 catalog_manager.cc:5697] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 20 to 22, leader changed from 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) to e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66). New cstate: current_term: 22 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:24:55.176452 27128 consensus_queue.cc:237] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 23.19, Last appended by leader: 19, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:55.176618 26888 consensus_queue.cc:237] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 25.16, Last appended by leader: 16, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:55.177351 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:55.109876 27201 raft_consensus.cc:515] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:55.169530 27234 raft_consensus.cc:493] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:24:55.258646 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:24:55.180449 27232 log.cc:927] Time spent T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.076s	user 0.000s	sys 0.000s
W20260709 12:24:55.152010 17655 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.513s	user 0.005s	sys 0.018s
W20260709 12:24:55.260202 17655 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.621s	user 0.006s	sys 0.019s
I20260709 12:24:55.147871 27139 raft_consensus.cc:2749] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Leader election lost for term 24. Reason: Vote denied by peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577) with higher term. Message: Invalid argument: T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for earlier term 24. Current term is 25.
I20260709 12:24:55.260327 27201 leader_election.cc:290] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:55.156805 17652 leader_election.cc:304] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:55.154750 27105 raft_consensus.cc:2804] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:24:55.157924 26963 raft_consensus.cc:493] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:24:55.261438 26963 raft_consensus.cc:515] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:55.263775 26963 leader_election.cc:290] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:55.161446 27163 raft_consensus.cc:2749] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:24:55.181583 27202 raft_consensus.cc:3060] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:24:55.264875 18366 raft_consensus.cc:1240] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 26. Current term is 27. Ops: []
I20260709 12:24:55.266113 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 30 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:55.266799 18366 raft_consensus.cc:2468] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 29.
I20260709 12:24:55.268226 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:55.268913 18364 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 26.
I20260709 12:24:55.119026 26789 raft_consensus.cc:515] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:55.169431 27244 raft_consensus.cc:2804] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:24:55.181182 27240 raft_consensus.cc:493] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:55.272118 27244 raft_consensus.cc:493] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:24:55.220788 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 126635us
I20260709 12:24:55.272694 27244 raft_consensus.cc:3060] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:55.240861 17759 raft_consensus.cc:2468] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 31.
I20260709 12:24:55.179867 27205 consensus_queue.cc:260] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 24.13, Last appended by leader: 13, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:55.273730 27205 raft_consensus.cc:3060] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:55.274130 26789 leader_election.cc:290] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:55.273932 27202 raft_consensus.cc:515] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 32 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:55.275609 17760 raft_consensus.cc:2393] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 32: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
W20260709 12:24:55.223811 26581 consensus_meta.cc:287] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.560s	user 0.012s	sys 0.000s
I20260709 12:24:55.267673 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:55.147557 26632 raft_consensus.cc:515] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:55.183954 27215 consensus_queue.cc:1048] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.042s
I20260709 12:24:55.276708 27202 leader_election.cc:290] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 32 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:55.276314 26581 raft_consensus.cc:2749] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:24:55.161860 27012 raft_consensus.cc:515] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:55.278239 26632 leader_election.cc:290] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:55.279388 27012 leader_election.cc:290] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:55.251323 27189 raft_consensus.cc:2804] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Leader pre-election won for term 18
W20260709 12:24:55.290771 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 127413us
W20260709 12:24:55.309336 17655 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:55.289239 17801 leader_election.cc:304] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:55.310601 17801 leader_election.cc:304] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:24:55.302323 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:55.311846 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:55.313992 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:55.314682 17688 raft_consensus.cc:3060] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:55.287817 26887 leader_election.cc:290] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:55.287982 17578 catalog_manager.cc:5697] T a5732e71b9d54f8786b8c511d69943bf P 334a4e1f59944544b3b3ff36616b57da reported cstate change: config changed from index -1 to 12, term changed from 19 to 25, leader changed from e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66) to 0da169ff82674dc1947055ce43de68a3 (127.16.29.65), NON_VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) added. New cstate: current_term: 25 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: NON_VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: true } } }
I20260709 12:24:55.319597 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 28 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:55.329349 17685 raft_consensus.cc:2468] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 27.
W20260709 12:24:55.308317 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 151588us
I20260709 12:24:55.303871 17729 leader_election.cc:304] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:55.330864 17652 leader_election.cc:304] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:55.331066 17729 leader_election.cc:304] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:55.312502 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:55.311362 17801 leader_election.cc:304] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:55.333366 17578 catalog_manager.cc:5697] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 25 to 30, leader changed from e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66) to 334a4e1f59944544b3b3ff36616b57da (127.16.29.68). New cstate: current_term: 30 leader_uuid: "334a4e1f59944544b3b3ff36616b57da" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:24:55.303725 27202 consensus_queue.cc:1059] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 26.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.033s
I20260709 12:24:55.337285 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:55.340422 27202 raft_consensus.cc:2804] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:24:55.340838 27202 raft_consensus.cc:493] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:55.341202 27202 raft_consensus.cc:3060] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:55.342466 26818 raft_consensus.cc:3060] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:24:55.337936 17684 raft_consensus.cc:2468] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 26.
I20260709 12:24:55.288693 27105 raft_consensus.cc:697] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 26 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:24:55.307826 27004 leader_election.cc:290] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:24:55.349443 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 192632us
I20260709 12:24:55.350087 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:55.321069 27139 raft_consensus.cc:2749] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:24:55.362329 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:55.363011 17829 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 26.
I20260709 12:24:55.364066 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
W20260709 12:24:55.364226 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 174077us
I20260709 12:24:55.348508 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 32 candidate_status { last_received { term: 30 index: 23 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:55.366040 17728 leader_election.cc:304] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:24:55.350986 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:55.352128 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:55.353241 17652 leader_election.cc:304] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:55.367468 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:55.307152 27189 raft_consensus.cc:493] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:24:55.323490 27008 raft_consensus.cc:2804] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:24:55.329098 26789 raft_consensus.cc:2804] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:24:55.357467 17688 raft_consensus.cc:2468] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 23.
I20260709 12:24:55.358948 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:55.386282 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:55.386830 17831 raft_consensus.cc:2468] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 25.
I20260709 12:24:55.387591 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:55.360339 17834 raft_consensus.cc:1275] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 22 index: 18. Preceding OpId from leader: term: 25 index: 19. (index mismatch)
I20260709 12:24:55.361265 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 31 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
W20260709 12:24:55.366170 17802 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:55.389106 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:55.335246 17578 catalog_manager.cc:5697] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 19 to 23. New cstate: current_term: 23 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:24:55.358446 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 28 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:55.371481 17728 leader_election.cc:304] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:55.390165 17836 raft_consensus.cc:3060] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:24:55.339066 27238 raft_consensus.cc:2804] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:24:55.373543 26843 consensus_queue.cc:1048] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.004s
I20260709 12:24:55.377280 17652 leader_election.cc:304] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 31 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:55.364658 17829 raft_consensus.cc:3060] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:55.345715 26957 raft_consensus.cc:3055] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:24:55.383268 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:55.389664 17834 raft_consensus.cc:2468] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 25.
I20260709 12:24:55.398214 17834 raft_consensus.cc:3060] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:55.347934 26887 raft_consensus.cc:2804] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:24:55.388139 17831 raft_consensus.cc:2468] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 23.
I20260709 12:24:55.390488 17578 catalog_manager.cc:5697] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 13 to 22, leader changed from e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66) to 334a4e1f59944544b3b3ff36616b57da (127.16.29.68). New cstate: current_term: 22 leader_uuid: "334a4e1f59944544b3b3ff36616b57da" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:24:55.346800 26599 leader_election.cc:290] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:55.395236 17652 leader_election.cc:304] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:55.422044 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:55.422673 18366 raft_consensus.cc:3060] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:55.400318 18364 raft_consensus.cc:2468] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 26.
I20260709 12:24:55.402041 27242 raft_consensus.cc:2749] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Leader election lost for term 31. Reason: could not achieve majority
I20260709 12:24:55.402454 26887 raft_consensus.cc:493] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:24:55.402486 27231 log.cc:927] Time spent T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.070s	user 0.000s	sys 0.000s
I20260709 12:24:55.402906 27102 consensus_queue.cc:1048] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20260709 12:24:55.397033 26910 leader_election.cc:290] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:24:55.407413 27254 log.cc:927] Time spent T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.057s	user 0.000s	sys 0.000s
W20260709 12:24:55.408079 27249 log.cc:927] Time spent T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.185s	user 0.001s	sys 0.000s
I20260709 12:24:55.384284 27237 raft_consensus.cc:2804] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:24:55.391817 27107 leader_election.cc:290] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 27 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:55.392176 27238 raft_consensus.cc:493] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:55.409451 17834 raft_consensus.cc:1275] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 23 index: 16. Preceding OpId from leader: term: 24 index: 17. (index mismatch)
W20260709 12:24:55.412504 27248 log.cc:927] Time spent T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.068s	user 0.000s	sys 0.000s
I20260709 12:24:55.413544 27261 raft_consensus.cc:2749] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:24:55.414685 17728 leader_election.cc:304] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:55.383749 27228 raft_consensus.cc:2749] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:24:55.417024 27097 leader_election.cc:290] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 23 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:55.417454 27040 leader_election.cc:290] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 23 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:55.443898 17729 leader_election.cc:304] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:55.376338 27189 raft_consensus.cc:3060] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:55.354952 27245 raft_consensus.cc:2749] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
W20260709 12:24:55.452997 27253 log.cc:927] Time spent T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.143s	user 0.000s	sys 0.001s
I20260709 12:24:55.373087 26978 leader_election.cc:290] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 26 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:55.375798 27008 raft_consensus.cc:493] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:24:55.453248 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 68745.2us
I20260709 12:24:55.511245 18333 leader_election.cc:304] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:55.457188 27139 consensus_queue.cc:1048] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.000s
I20260709 12:24:55.453639 26887 raft_consensus.cc:3060] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:55.518775 27008 raft_consensus.cc:3060] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Advancing to term 31
W20260709 12:24:55.519507 17803 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:55.455003 27205 raft_consensus.cc:2804] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:24:55.520479 27205 raft_consensus.cc:493] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:55.520787 17803 leader_election.cc:304] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:55.520915 27205 raft_consensus.cc:3060] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:24:55.521912 17803 leader_election.cc:304] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:55.413677 27126 leader_election.cc:290] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:55.454316 17652 leader_election.cc:304] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:55.524462 17803 leader_election.cc:304] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:24:55.524495 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 70277.6us
I20260709 12:24:55.525732 26888 raft_consensus.cc:493] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:55.526218 17652 leader_election.cc:304] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
W20260709 12:24:55.464596 26962 consensus_meta.cc:287] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.669s	user 0.009s	sys 0.001s
W20260709 12:24:55.468621 27171 consensus_meta.cc:287] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.770s	user 0.010s	sys 0.002s
I20260709 12:24:55.338882 27111 consensus_queue.cc:260] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 30.18, Last appended by leader: 18, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:55.393157 26789 raft_consensus.cc:493] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:24:55.483817 18366 raft_consensus.cc:2468] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 23.
W20260709 12:24:55.456943 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 64959.6us
I20260709 12:24:55.486944 17762 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38786 (request call id 4019) took 1351 ms. Trace:
I20260709 12:24:55.419994 26957 raft_consensus.cc:740] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 26 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:24:55.489089 17836 raft_consensus.cc:2410] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for term 28 because replica has last-logged OpId of term: 27 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 15.
I20260709 12:24:55.563253 17763 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38776 (request call id 3233) took 1278 ms. Trace:
I20260709 12:24:55.488080 17829 raft_consensus.cc:2468] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 25.
W20260709 12:24:55.495019 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 80424.3us
W20260709 12:24:55.497503 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 53725.6us
I20260709 12:24:55.566366 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:55.497977 27259 raft_consensus.cc:2804] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:24:55.491737 27012 raft_consensus.cc:493] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:55.502571 27163 raft_consensus.cc:3055] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:24:55.504616 27221 raft_consensus.cc:493] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:55.505268 17686 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37402 (request call id 3799) took 1405 ms. Trace:
I20260709 12:24:55.259124 27234 raft_consensus.cc:515] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:55.572124 17686 rpcz_store.cc:276] 0709 12:24:54.100050 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:54.636230 (+536180us) service_pool.cc:224] Handling call
0709 12:24:54.636571 (+   341us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:55.027972 (+391401us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:55.028023 (+    51us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:55.062876 (+ 34853us) log.cc:844] Serialized 26 byte log entry
0709 12:24:55.289717 (+226841us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:24:55.289769 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:55.289865 (+    96us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:55.505042 (+215177us) raft_consensus.cc:1685] finished
0709 12:24:55.505065 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:55.505208 (+   143us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":23941,"raft.run_cpu_time_us":43,"raft.run_wall_time_us":43,"thread_start_us":535358,"threads_started":2,"wal-append.queue_time_us":377803}
I20260709 12:24:55.489982 27237 raft_consensus.cc:493] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:24:55.492053 27238 raft_consensus.cc:3060] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:24:55.515451 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:55.488632 27105 consensus_queue.cc:237] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 24.18, Last appended by leader: 18, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:55.529125 18363 raft_consensus.cc:2410] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 28 because replica has last-logged OpId of term: 27 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 16.
W20260709 12:24:55.578003 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 53617.9us
I20260709 12:24:55.527546 17652 leader_election.cc:304] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:55.272323 27240 raft_consensus.cc:515] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:55.567490 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:55.563730 17763 rpcz_store.cc:276] 0709 12:24:54.284654 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:54.636300 (+351646us) service_pool.cc:224] Handling call
0709 12:24:54.636562 (+   262us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:54.636906 (+   344us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:54.636938 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:54.637470 (+   532us) log.cc:844] Serialized 26 byte log entry
0709 12:24:55.060662 (+423192us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:24:55.060709 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:55.060770 (+    61us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:55.563026 (+502256us) raft_consensus.cc:1685] finished
0709 12:24:55.563054 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:55.563190 (+   136us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":16639,"raft.run_cpu_time_us":63,"raft.run_wall_time_us":64,"thread_start_us":422812,"threads_started":1,"wal-append.queue_time_us":519450}
I20260709 12:24:55.567802 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:55.505599 27235 raft_consensus.cc:2749] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 12:24:55.566836 17729 leader_election.cc:304] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:55.561826 17762 rpcz_store.cc:276] 0709 12:24:54.135448 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:54.135623 (+   175us) service_pool.cc:224] Handling call
0709 12:24:54.136030 (+   407us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:54.136458 (+   428us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:54.136498 (+    40us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:54.137019 (+   521us) log.cc:844] Serialized 26 byte log entry
0709 12:24:54.770342 (+633323us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:24:54.770383 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:54.770443 (+    60us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:55.486720 (+716277us) raft_consensus.cc:1685] finished
0709 12:24:55.486741 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:55.486883 (+   142us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":31312,"raft.queue_time_us":97,"raft.run_cpu_time_us":111,"raft.run_wall_time_us":6860,"thread_start_us":526628,"threads_started":1,"wal-append.queue_time_us":1037607}
I20260709 12:24:55.532581 27245 raft_consensus.cc:2804] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:24:55.528105 27107 raft_consensus.cc:2804] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:24:55.528539 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:55.553929 26599 raft_consensus.cc:2749] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:24:55.584702 17728 leader_election.cc:304] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:55.570603 26789 raft_consensus.cc:3060] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:55.528434 27197 raft_consensus.cc:3055] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 31 LEADER]: Stepping down as leader of term 31
W20260709 12:24:55.564965 27208 consensus_meta.cc:287] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.515s	user 0.010s	sys 0.000s
I20260709 12:24:55.527803 27004 raft_consensus.cc:493] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:55.575174 27163 raft_consensus.cc:740] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:24:55.531083 18333 leader_election.cc:304] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:55.533983 27126 raft_consensus.cc:2749] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:24:55.635542 18374 raft_consensus.cc:2468] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 23.
I20260709 12:24:55.581422 26956 raft_consensus.cc:3055] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:24:55.576453 27237 raft_consensus.cc:3060] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:55.592550 17833 raft_consensus.cc:2393] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 23: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:24:55.578207 27267 raft_consensus.cc:2804] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Leader pre-election won for term 26
W20260709 12:24:55.590129 27251 log.cc:927] Time spent T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.091s	user 0.000s	sys 0.001s
I20260709 12:24:55.571798 27171 raft_consensus.cc:2749] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:24:55.586414 27245 raft_consensus.cc:493] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:55.669916 27245 raft_consensus.cc:3060] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:55.671773 26818 raft_consensus.cc:2804] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:24:55.583067 27111 raft_consensus.cc:3060] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:24:55.582815 27259 raft_consensus.cc:697] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 26 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:24:55.585489 27139 raft_consensus.cc:2804] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:24:55.677191 27139 raft_consensus.cc:697] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 30 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:24:55.591766 27035 raft_consensus.cc:3060] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:55.668067 27008 raft_consensus.cc:515] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:55.576115 27221 raft_consensus.cc:515] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:55.526235 26888 raft_consensus.cc:515] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:55.673623 27264 log.cc:927] Time spent T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.194s	user 0.001s	sys 0.000s
I20260709 12:24:55.635572 27235 raft_consensus.cc:2804] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Leader election won for term 23
W20260709 12:24:55.577422 27062 consensus_queue.cc:1175] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "0da169ff82674dc1947055ce43de68a3" responder_term: 31 status { last_received { term: 29 index: 17 } last_committed_idx: 16 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 30. Current term is 31. Ops: [28.17-30.18]" } } last_received_current_leader { term: 0 index: 0 } }
I20260709 12:24:55.647545 27197 raft_consensus.cc:740] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 31 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:24:55.579610 27012 raft_consensus.cc:515] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:55.580071 27261 consensus_queue.cc:1059] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.089s
I20260709 12:24:55.656260 27107 raft_consensus.cc:697] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 31 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:24:55.672358 27255 consensus_queue.cc:1048] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.022s
I20260709 12:24:55.687467 27221 leader_election.cc:290] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:55.687521 27107 consensus_queue.cc:237] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 29.20, Last appended by leader: 20, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:55.689297 27234 leader_election.cc:290] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:55.689289 27237 raft_consensus.cc:515] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:55.582706 27097 consensus_queue.cc:1059] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 21.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.082s
I20260709 12:24:55.678092 27139 consensus_queue.cc:237] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 28.16, Last appended by leader: 16, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:55.691025 27244 raft_consensus.cc:515] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:55.635819 27256 consensus_queue.cc:1059] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 21.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.086s
I20260709 12:24:55.692529 27256 raft_consensus.cc:3055] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:24:55.692873 27256 raft_consensus.cc:740] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 21 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:24:55.693624 27256 consensus_queue.cc:260] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 21.20, Last appended by leader: 20, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:55.666077 27168 consensus_queue.cc:1048] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.018s
I20260709 12:24:55.694249 27205 raft_consensus.cc:515] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:55.691521 27237 leader_election.cc:290] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 26 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:55.694455 27197 consensus_queue.cc:260] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 31.20, Last appended by leader: 20, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:55.695735 27197 raft_consensus.cc:3060] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:24:55.665587 27267 raft_consensus.cc:493] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:55.697538 27267 raft_consensus.cc:3060] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:55.682487 27235 raft_consensus.cc:697] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:24:55.719740 27267 raft_consensus.cc:515] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:55.726293 27267 leader_election.cc:290] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 26 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:24:55.581739 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 54277.8us
W20260709 12:24:55.760673 17655 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:55.592357 17757 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38776 (request call id 3232) took 1308 ms. Trace:
I20260709 12:24:55.768220 17757 rpcz_store.cc:276] 0709 12:24:54.284413 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:54.584136 (+299723us) service_pool.cc:224] Handling call
0709 12:24:54.594010 (+  9874us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:55.034831 (+440821us) raft_consensus.cc:1534] Early marking committed up to index 16
0709 12:24:55.062142 (+ 27311us) log.cc:844] Serialized 14 byte log entry
0709 12:24:55.288118 (+225976us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:55.299646 (+ 11528us) log.cc:844] Serialized 26 byte log entry
0709 12:24:55.299921 (+   275us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:24:55.299992 (+    71us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:55.300129 (+   137us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:55.592075 (+291946us) raft_consensus.cc:1685] finished
0709 12:24:55.592105 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:55.592281 (+   176us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":46383,"prepare.queue_time_us":188,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":39,"raft.queue_time_us":141632,"raft.run_cpu_time_us":208,"raft.run_wall_time_us":208,"thread_start_us":544540,"threads_started":3,"wal-append.queue_time_us":189244}
W20260709 12:24:55.665093 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 134232us
W20260709 12:24:55.772670 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:55.672258 26818 raft_consensus.cc:493] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:55.774570 26818 raft_consensus.cc:3060] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:24:55.775116 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 208240us
I20260709 12:24:55.772696 18333 leader_election.cc:304] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:55.765213 17683 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37402 (request call id 3798) took 1665 ms. Trace:
W20260709 12:24:55.767154 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 187248us
I20260709 12:24:55.781092 17803 leader_election.cc:304] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:55.762786 17685 raft_consensus.cc:2468] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 25.
I20260709 12:24:55.780103 17683 rpcz_store.cc:276] 0709 12:24:54.099575 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:54.633589 (+534014us) service_pool.cc:224] Handling call
0709 12:24:54.633889 (+   300us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:54.974314 (+340425us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:54.974358 (+    44us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:54.974910 (+   552us) log.cc:844] Serialized 26 byte log entry
0709 12:24:55.144254 (+169344us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:24:55.144296 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:55.144376 (+    80us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:55.764499 (+620123us) raft_consensus.cc:1685] finished
0709 12:24:55.764543 (+    44us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:55.764935 (+   392us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":26400,"raft.run_cpu_time_us":82,"raft.run_wall_time_us":82,"thread_start_us":433120,"threads_started":2,"wal-append.queue_time_us":506347}
W20260709 12:24:55.788434 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 201516us
I20260709 12:24:55.789762 17802 leader_election.cc:304] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:55.793504 17652 leader_election.cc:304] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:24:55.656060 26844 raft_consensus.cc:2749] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:24:55.797268 26844 raft_consensus.cc:493] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:55.798995 26910 raft_consensus.cc:2804] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:24:55.799692 26910 raft_consensus.cc:493] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:55.800338 26910 raft_consensus.cc:3060] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:55.803051 26888 leader_election.cc:290] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:55.807480 26888 raft_consensus.cc:2804] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:24:55.798441 26844 raft_consensus.cc:515] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:55.664084 26956 raft_consensus.cc:740] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 22 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:24:55.844417 27266 raft_consensus.cc:2749] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:24:55.533015 26962 raft_consensus.cc:515] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:55.850337 26962 leader_election.cc:290] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 29 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:24:55.850463 18371 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59918 (request call id 2713) took 1825 ms (client timeout 2155 ms). Trace:
I20260709 12:24:55.845259 26956 consensus_queue.cc:260] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 22.14, Last appended by leader: 14, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:55.853633 26956 raft_consensus.cc:3060] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:55.650856 27004 raft_consensus.cc:515] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:55.720223 27205 leader_election.cc:290] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 29 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:55.715049 27235 consensus_queue.cc:237] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 21.16, Last appended by leader: 16, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:55.857366 27004 leader_election.cc:290] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:55.857991 26844 leader_election.cc:290] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:55.813522 27012 leader_election.cc:290] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:55.813699 27008 leader_election.cc:290] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 31 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:55.821955 27197 raft_consensus.cc:2804] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:24:55.830783 17832 raft_consensus.cc:2393] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 31: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:24:55.836100 27221 raft_consensus.cc:493] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:55.837823 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 515152us
I20260709 12:24:55.842337 27244 leader_election.cc:290] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 27 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:55.853760 18361 raft_consensus.cc:1240] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 24. Current term is 25. Ops: []
I20260709 12:24:55.808725 18333 leader_election.cc:304] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:55.818490 26888 raft_consensus.cc:493] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:55.864655 27197 raft_consensus.cc:493] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:24:55.851263 18371 rpcz_store.cc:269] 0709 12:24:54.025118 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:54.040480 (+ 15362us) service_pool.cc:224] Handling call
0709 12:24:55.850014 (+1809534us) spinlock_profiling.cc:243] Waited 4.8 ms on lock 0x7b540034b554. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:24:55.850247 (+   233us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":11049216}
I20260709 12:24:55.719636 27202 raft_consensus.cc:515] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:55.873773 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 289007us
W20260709 12:24:55.875542 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 67252.3us
I20260709 12:24:55.795665 26789 raft_consensus.cc:515] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:55.879449 26888 raft_consensus.cc:3060] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Advancing to term 28
W20260709 12:24:55.885468 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 50191.3us
I20260709 12:24:55.879443 27197 raft_consensus.cc:3060] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:55.890866 17803 leader_election.cc:304] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:55.892035 27008 raft_consensus.cc:2749] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
W20260709 12:24:55.892084 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 97256.1us
I20260709 12:24:55.897109 27266 consensus_queue.cc:1048] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:24:55.899343 18333 leader_election.cc:304] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:24:55.721796 27259 consensus_queue.cc:237] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 25.16, Last appended by leader: 16, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:55.900614 27107 raft_consensus.cc:2764] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Leader pre-election decision vote started in defunct term 24: won
I20260709 12:24:55.899679 26844 consensus_queue.cc:1048] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.005s
I20260709 12:24:55.902180 18333 leader_election.cc:304] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:55.906348 26962 raft_consensus.cc:2804] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:24:55.907001 26962 raft_consensus.cc:493] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:55.907094 18333 leader_election.cc:304] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:55.907522 26962 raft_consensus.cc:3060] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Advancing to term 26
W20260709 12:24:55.897158 27278 log.cc:927] Time spent T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.169s	user 0.000s	sys 0.000s
I20260709 12:24:55.908761 26924 leader_election.cc:290] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:55.911461 17802 leader_election.cc:304] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:24:55.912101 27012 raft_consensus.cc:493] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:55.912902 27008 raft_consensus.cc:2804] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:24:55.913477 27008 raft_consensus.cc:493] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:55.913031 27012 raft_consensus.cc:515] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:55.913987 27008 raft_consensus.cc:3060] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:55.914381 17802 leader_election.cc:304] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:24:55.916115 27201 raft_consensus.cc:493] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:55.916836 27201 raft_consensus.cc:515] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:55.918489 27205 raft_consensus.cc:493] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
W20260709 12:24:55.920120 17655 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:55.919811 27205 raft_consensus.cc:515] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:55.924358 27221 raft_consensus.cc:515] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:55.928171 17803 leader_election.cc:304] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:55.928566 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:55.929500 17835 raft_consensus.cc:1240] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 22. Current term is 23. Ops: []
W20260709 12:24:55.929931 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:55.931619 26789 leader_election.cc:290] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 23 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:55.933225 26818 raft_consensus.cc:515] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:55.937067 27004 raft_consensus.cc:2804] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:24:55.938032 27004 raft_consensus.cc:697] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 23 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:24:55.940676 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:55.941058 27004 consensus_queue.cc:237] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 21.15, Last appended by leader: 15, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:55.940752 27275 raft_consensus.cc:2749] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:24:55.941747 17683 raft_consensus.cc:2468] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 25.
I20260709 12:24:55.945130 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:24:55.941057 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 45307.3us
I20260709 12:24:55.946779 17803 leader_election.cc:304] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:55.948060 26910 raft_consensus.cc:515] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:55.949156 27107 raft_consensus.cc:2749] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 12:24:55.950099 27107 raft_consensus.cc:2804] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:24:55.950508 27107 raft_consensus.cc:493] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:55.950999 27107 raft_consensus.cc:3060] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:55.950830 27258 raft_consensus.cc:493] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:55.950789 27035 consensus_queue.cc:1048] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.002s
I20260709 12:24:55.952067 27258 raft_consensus.cc:515] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:55.954218 27035 consensus_queue.cc:1048] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20260709 12:24:55.954879 27258 leader_election.cc:290] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:55.957667 18334 leader_election.cc:304] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:24:55.952675 27196 raft_consensus.cc:3055] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:24:55.959360 27196 raft_consensus.cc:740] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:24:55.959617 27035 raft_consensus.cc:2804] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:24:55.960884 27035 raft_consensus.cc:697] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 25 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:24:55.961066 27196 consensus_queue.cc:260] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 27.16, Last appended by leader: 16, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:55.962822 27196 raft_consensus.cc:3060] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:24:55.895686 17652 leader_election.cc:304] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:55.968684 27012 leader_election.cc:290] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:55.969002 27202 leader_election.cc:290] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 27 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:55.923657 27205 leader_election.cc:290] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:55.969841 27245 raft_consensus.cc:515] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:55.968792 27256 raft_consensus.cc:3060] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:55.962891 27035 consensus_queue.cc:237] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 23.17, Last appended by leader: 17, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:55.947165 26957 consensus_queue.cc:260] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 26.18, Last appended by leader: 18, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:55.983539 26957 raft_consensus.cc:3060] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:55.964979 27107 raft_consensus.cc:515] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:55.987136 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 76737.3us
W20260709 12:24:55.989061 17729 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:55.993124 27221 leader_election.cc:290] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:55.999543 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:55.996091 26789 raft_consensus.cc:2749] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:24:56.000828 27245 leader_election.cc:290] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 27 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:56.002076 27261 consensus_queue.cc:1048] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.006s
I20260709 12:24:56.013761 26818 leader_election.cc:290] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 27 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:56.021096 26910 leader_election.cc:290] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 19 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:24:56.021793 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 126498us
I20260709 12:24:55.959329 26887 raft_consensus.cc:515] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:56.010610 27258 raft_consensus.cc:493] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:56.025117 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 22 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:56.025563 27258 raft_consensus.cc:515] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:56.026883 27283 consensus_queue.cc:1059] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
W20260709 12:24:56.024852 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 104741us
I20260709 12:24:56.028055 27283 raft_consensus.cc:3055] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:24:56.028477 27283 raft_consensus.cc:740] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:24:56.028548 17727 leader_election.cc:304] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:56.029687 27283 consensus_queue.cc:260] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 29.17, Last appended by leader: 17, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:56.026129 18371 raft_consensus.cc:2410] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for term 22 because replica has last-logged OpId of term: 20 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 13.
I20260709 12:24:56.032760 27235 raft_consensus.cc:2804] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:24:56.033706 27235 raft_consensus.cc:493] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:56.034567 27235 raft_consensus.cc:3060] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:56.036378 27258 leader_election.cc:290] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:56.037547 27097 consensus_queue.cc:1059] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 0.0, Next index: 17, Last known committed idx: 14, Time since last communication: 0.001s
W20260709 12:24:56.041699 27189 consensus_meta.cc:287] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.530s	user 0.000s	sys 0.019s
I20260709 12:24:56.042891 27201 leader_election.cc:290] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:56.044018 27261 raft_consensus.cc:493] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:56.043306 27189 raft_consensus.cc:515] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:56.044687 27261 raft_consensus.cc:515] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:56.047734 17728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:56.052361 26887 leader_election.cc:290] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 24 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:24:56.052942 17730 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:56.059162 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 63830.3us
I20260709 12:24:56.053941 18333 leader_election.cc:304] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
W20260709 12:24:56.060493 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 113861us
I20260709 12:24:56.065928 27258 consensus_queue.cc:1048] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:24:56.047212 27261 leader_election.cc:290] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:56.087215 17680 raft_consensus.cc:1275] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 20 index: 13. Preceding OpId from leader: term: 23 index: 15. (index mismatch)
I20260709 12:24:56.087710 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:56.053129 27205 raft_consensus.cc:2749] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:24:56.088531 17687 raft_consensus.cc:3060] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:56.088549 27245 raft_consensus.cc:493] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:56.068908 27189 leader_election.cc:290] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 18 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:56.092595 27245 raft_consensus.cc:515] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:56.094888 27245 leader_election.cc:290] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:56.106324 18371 raft_consensus.cc:1240] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 28. Current term is 29. Ops: []
I20260709 12:24:56.061273 27202 raft_consensus.cc:493] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:56.071241 27008 raft_consensus.cc:515] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:56.135841 27107 leader_election.cc:290] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 24 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:56.137332 27163 consensus_queue.cc:260] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 21.15, Last appended by leader: 15, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:56.144663 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:56.148743 17687 raft_consensus.cc:2468] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 27.
I20260709 12:24:56.126914 27258 raft_consensus.cc:2749] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
W20260709 12:24:56.166496 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 47748.1us
W20260709 12:24:56.165858 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 177074us
I20260709 12:24:56.170023 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 29 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:56.171413 17837 raft_consensus.cc:1240] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 23 LEADER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 22. Current term is 23. Ops: [22.14-22.14]
I20260709 12:24:56.172444 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 29 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:56.173945 18371 raft_consensus.cc:1240] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 27. Current term is 28. Ops: [27.16-27.16]
I20260709 12:24:56.179625 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:56.180430 17829 raft_consensus.cc:3060] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:56.181175 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:56.184603 27189 raft_consensus.cc:493] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:24:56.185315 26844 raft_consensus.cc:493] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:56.185477 27189 raft_consensus.cc:515] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:56.186012 26844 raft_consensus.cc:515] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:56.186751 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:56.187532 17832 raft_consensus.cc:3060] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:56.189704 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:56.190827 17760 raft_consensus.cc:2468] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 26.
I20260709 12:24:56.192587 17758 raft_consensus.cc:1240] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 25. Current term is 26. Ops: []
I20260709 12:24:56.193400 27197 raft_consensus.cc:515] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:56.193910 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:56.184808 17836 raft_consensus.cc:1240] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 25. Current term is 26. Ops: []
W20260709 12:24:56.196177 17803 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.16.29.68:42577 timed out after 1.695s (SENT) [suppressed 19 similar messages]
I20260709 12:24:56.196696 17762 raft_consensus.cc:1275] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 22 index: 14. Preceding OpId from leader: term: 23 index: 15. (index mismatch)
I20260709 12:24:56.199839 17686 raft_consensus.cc:1275] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 25 index: 19. Preceding OpId from leader: term: 26 index: 20. (index mismatch)
I20260709 12:24:56.161123 27163 raft_consensus.cc:3060] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Advancing to term 22
W20260709 12:24:56.205716 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:56.206954 17680 raft_consensus.cc:2393] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 23: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:24:56.206542 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:56.209658 27008 leader_election.cc:290] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 27 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:56.212603 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:56.201027 27189 leader_election.cc:290] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:56.201735 27107 raft_consensus.cc:493] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:56.212049 17830 raft_consensus.cc:1240] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 24. Current term is 25. Ops: [24.13-24.13]
W20260709 12:24:56.223145 27291 log.cc:927] Time spent T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.054s	user 0.001s	sys 0.000s
I20260709 12:24:56.241169 18361 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59902 (request call id 2202) took 1209 ms. Trace:
I20260709 12:24:56.241607 18361 rpcz_store.cc:276] 0709 12:24:55.032616 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:55.032870 (+   254us) service_pool.cc:224] Handling call
0709 12:24:55.853029 (+820159us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:55.883508 (+ 30479us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:56.240918 (+357410us) spinlock_profiling.cc:243] Waited 794 ms on lock 0x7b540034b550. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a61f8ad 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:24:56.241064 (+   146us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1825159552}
I20260709 12:24:56.224349 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:56.243530 17680 raft_consensus.cc:2393] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 27: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:24:56.224392 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:56.244727 17686 raft_consensus.cc:2468] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 22.
I20260709 12:24:56.244798 26844 leader_election.cc:290] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:56.244930 17830 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35040 (request call id 3589) took 1206 ms. Trace:
I20260709 12:24:56.224790 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 29 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:56.245430 17830 rpcz_store.cc:276] 0709 12:24:55.038301 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:55.038636 (+   335us) service_pool.cc:224] Handling call
0709 12:24:55.069253 (+ 30617us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:56.231540 (+1162287us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:56.244620 (+ 13080us) spinlock_profiling.cc:243] Waited 1.14 s on lock 0x7b540022ae54. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:24:56.244860 (+   240us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2626494720}
I20260709 12:24:56.224956 26957 raft_consensus.cc:493] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:24:56.221256 26910 consensus_queue.cc:1048] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:24:56.247568 26957 raft_consensus.cc:515] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:56.248103 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:56.249018 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 32 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:56.249843 18368 raft_consensus.cc:1240] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 21. Current term is 22. Ops: []
I20260709 12:24:56.250412 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:56.251204 18366 raft_consensus.cc:2393] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 27: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:24:56.251741 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 29 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:56.252321 27197 leader_election.cc:290] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 27 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:56.253008 18364 raft_consensus.cc:1275] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 24 index: 17. Preceding OpId from leader: term: 25 index: 18. (index mismatch)
I20260709 12:24:56.253427 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 27 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:56.226197 27093 raft_consensus.cc:493] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:56.254422 27093 raft_consensus.cc:515] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:56.256997 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 27 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:56.257422 27283 raft_consensus.cc:3060] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:24:56.227100 27239 consensus_queue.cc:1059] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 25.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.034s
I20260709 12:24:56.261504 27239 raft_consensus.cc:3055] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:24:56.261917 27239 raft_consensus.cc:740] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 25 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
W20260709 12:24:56.262883 27289 log.cc:927] Time spent T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.113s	user 0.000s	sys 0.000s
I20260709 12:24:56.228536 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:56.265072 17761 raft_consensus.cc:2468] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 27.
I20260709 12:24:56.218346 17755 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38776 (request call id 3237) took 1173 ms. Trace:
I20260709 12:24:56.201269 27202 raft_consensus.cc:515] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:56.266312 17755 rpcz_store.cc:276] 0709 12:24:55.039023 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:55.040240 (+  1217us) service_pool.cc:224] Handling call
0709 12:24:56.211785 (+1171545us) spinlock_profiling.cc:243] Waited 29.8 ms on lock 0x7b54003eb554. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:24:56.211981 (+   196us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":68554496}
I20260709 12:24:56.231539 27107 raft_consensus.cc:515] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:56.268287 27202 leader_election.cc:290] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:56.268844 18374 raft_consensus.cc:2468] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 25.
I20260709 12:24:56.270229 27107 leader_election.cc:290] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:56.200680 26888 raft_consensus.cc:515] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:56.275782 17834 raft_consensus.cc:1240] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 27. Current term is 28. Ops: [27.15-27.15]
I20260709 12:24:56.247670 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:56.276553 18371 raft_consensus.cc:3060] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:24:56.278249 17655 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:56.280792 26888 leader_election.cc:290] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 28 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:56.281263 27093 leader_election.cc:290] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:56.282462 27230 leader_election.cc:290] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:56.283386 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:56.284327 17682 raft_consensus.cc:2393] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 27: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:24:56.284840 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 32 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:56.285508 18364 raft_consensus.cc:1240] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 23. Current term is 24. Ops: []
I20260709 12:24:56.292968 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:56.293876 17682 raft_consensus.cc:2468] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 27.
I20260709 12:24:56.296098 17652 leader_election.cc:304] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:56.296799 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:56.236786 17802 leader_election.cc:304] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:24:56.225600 17835 raft_consensus.cc:1240] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 18. Current term is 19. Ops: []
W20260709 12:24:56.234670 17804 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:56.239545 17829 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 27.
I20260709 12:24:56.249869 26957 leader_election.cc:290] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:56.236536 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:24:56.237557 27285 log.cc:927] Time spent T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.106s	user 0.000s	sys 0.000s
W20260709 12:24:56.302386 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 65773.4us
I20260709 12:24:56.316943 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 31 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:56.310631 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 29 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:24:56.325866 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 107839us
I20260709 12:24:56.263231 27239 consensus_queue.cc:260] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 25.17, Last appended by leader: 17, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:56.238644 27240 leader_election.cc:290] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:56.300496 27230 raft_consensus.cc:2749] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:24:56.236001 26599 raft_consensus.cc:493] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:56.295629 26962 raft_consensus.cc:515] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:56.333367 17837 raft_consensus.cc:2468] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 24.
I20260709 12:24:56.344834 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 28 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:56.329368 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:24:56.347697 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 140698us
I20260709 12:24:56.341558 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:56.332914 26957 raft_consensus.cc:493] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:56.359959 17684 raft_consensus.cc:3060] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:24:56.351972 17686 raft_consensus.cc:1240] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 28. Current term is 29. Ops: [28.19-28.19]
I20260709 12:24:56.355615 17729 leader_election.cc:304] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:56.364015 17727 leader_election.cc:304] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:56.348542 26910 raft_consensus.cc:2804] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:24:56.369119 18374 raft_consensus.cc:1240] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 25. Current term is 26. Ops: []
I20260709 12:24:56.365620 27261 raft_consensus.cc:2749] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
W20260709 12:24:56.376880 27238 consensus_meta.cc:287] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.723s	user 0.014s	sys 0.007s
I20260709 12:24:56.378613 27238 raft_consensus.cc:515] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:56.379980 26599 raft_consensus.cc:515] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:56.381390 27238 leader_election.cc:290] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 30 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:56.381572 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:56.380021 27259 raft_consensus.cc:2804] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:24:56.383327 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 30 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:56.371421 26910 raft_consensus.cc:493] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:56.384080 17833 raft_consensus.cc:3060] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:24:56.384318 26910 raft_consensus.cc:3060] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:56.384920 18368 raft_consensus.cc:2468] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 23.
I20260709 12:24:56.383525 27259 raft_consensus.cc:493] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:56.387020 27259 raft_consensus.cc:3060] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:56.396756 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:24:56.397957 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 49912.4us
I20260709 12:24:56.399633 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:56.400506 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:56.400727 27238 raft_consensus.cc:493] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:24:56.401486 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 22 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:56.401858 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:56.401532 27238 raft_consensus.cc:515] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:24:56.404878 17727 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:56.400622 17838 raft_consensus.cc:1240] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 32 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 31. Current term is 32. Ops: [31.20-31.20]
I20260709 12:24:56.400660 17836 raft_consensus.cc:2393] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 27: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:24:56.408854 17833 raft_consensus.cc:2468] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 30.
I20260709 12:24:56.353602 27093 consensus_queue.cc:1059] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 25.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.021s
I20260709 12:24:56.365743 27239 raft_consensus.cc:3060] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:56.354583 27197 consensus_queue.cc:1048] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.055s
I20260709 12:24:56.349598 27189 consensus_queue.cc:1048] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.057s
I20260709 12:24:56.362249 27202 consensus_queue.cc:1059] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 18.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.049s
I20260709 12:24:56.417553 17729 leader_election.cc:304] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:56.418066 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:56.423936 17836 raft_consensus.cc:2393] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 27: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:24:56.425400 17829 raft_consensus.cc:1275] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 22 index: 17. Preceding OpId from leader: term: 23 index: 18. (index mismatch)
I20260709 12:24:56.431221 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:56.436868 26962 leader_election.cc:290] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 26 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:56.437325 27202 raft_consensus.cc:2804] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:24:56.417073 26957 raft_consensus.cc:515] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:56.437876 27202 raft_consensus.cc:493] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:24:56.438336 27202 raft_consensus.cc:3060] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:56.442669 17654 leader_election.cc:304] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:24:56.444186 27197 raft_consensus.cc:2804] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:24:56.444782 27197 raft_consensus.cc:697] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 25 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:24:56.437232 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:56.445667 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 26 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:56.446414 17835 raft_consensus.cc:2468] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 25.
I20260709 12:24:56.450178 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 27 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:24:56.457103 17655 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:56.461851 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 45495us
W20260709 12:24:56.462034 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 56482.5us
I20260709 12:24:56.467468 18371 raft_consensus.cc:2468] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 27.
I20260709 12:24:56.468789 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:56.470980 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:56.472478 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:56.415292 27261 consensus_queue.cc:1059] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 23.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.037s
I20260709 12:24:56.476322 26888 consensus_queue.cc:1059] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 25.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:24:56.476609 17801 leader_election.cc:304] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:24:56.482295 17655 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:56.468739 26957 leader_election.cc:290] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:56.512316 26887 consensus_queue.cc:1048] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.001s
W20260709 12:24:56.514158 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 50767us
I20260709 12:24:56.514362 17832 raft_consensus.cc:2468] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 23.
I20260709 12:24:56.517760 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:56.518456 17756 raft_consensus.cc:3060] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:56.521163 17834 raft_consensus.cc:1275] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 21 index: 16. Preceding OpId from leader: term: 23 index: 17. (index mismatch)
I20260709 12:24:56.513279 26887 consensus_queue.cc:1243] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer e2909cff1da44ad387f96a8ae3aa5afb log is divergent from this leader: its last log entry 22.14 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 13
W20260709 12:24:56.528772 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 52381.3us
I20260709 12:24:56.529717 17801 leader_election.cc:304] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:56.523077 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 31 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:56.530637 27111 raft_consensus.cc:2749] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:24:56.530896 17755 raft_consensus.cc:2393] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 31: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:24:56.532401 17755 raft_consensus.cc:1275] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 24 index: 18. Preceding OpId from leader: term: 26 index: 19. (index mismatch)
W20260709 12:24:56.534035 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 337778us
I20260709 12:24:56.537037 17838 raft_consensus.cc:1275] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 28 index: 16. Preceding OpId from leader: term: 30 index: 17. (index mismatch)
W20260709 12:24:56.538813 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 61269.1us
I20260709 12:24:56.416010 27035 consensus_queue.cc:1059] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 22.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.109s
I20260709 12:24:56.556839 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:56.558063 17684 raft_consensus.cc:2468] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 28.
I20260709 12:24:56.559114 17757 raft_consensus.cc:1275] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 25 index: 16. Preceding OpId from leader: term: 28 index: 17. (index mismatch)
I20260709 12:24:56.556658 18367 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59902 (request call id 2198) took 1543 ms. Trace:
I20260709 12:24:56.567070 17760 raft_consensus.cc:1240] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 17. Current term is 18. Ops: []
I20260709 12:24:56.514027 26910 raft_consensus.cc:515] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:56.585623 17802 leader_election.cc:304] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
W20260709 12:24:56.591701 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 55145us
I20260709 12:24:56.598182 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:56.601874 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:24:56.602015 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 168328us
I20260709 12:24:56.521350 27235 raft_consensus.cc:515] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:56.606848 17832 raft_consensus.cc:2468] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 23.
I20260709 12:24:56.608098 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 29 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:56.604796 17654 leader_election.cc:304] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:24:56.605343 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:56.565032 18367 rpcz_store.cc:276] 0709 12:24:54.997573 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:54.999075 (+  1502us) service_pool.cc:224] Handling call
0709 12:24:55.049016 (+ 49941us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:55.060981 (+ 11965us) raft_consensus.cc:1534] Early marking committed up to index 19
0709 12:24:55.877202 (+816221us) spinlock_profiling.cc:243] Waited 15.1 ms on lock 0x7b1800245080. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b47515e07 00007f7b503709a9 00007f7b4753269b 00007f7b4753093f 00007f7b47530455 00007f7b4e0183c7 00007f7b4a62a9a7 00007f7b4a63a0bf 00007f7b4a63a038
0709 12:24:56.069325 (+192123us) log.cc:844] Serialized 14 byte log entry
0709 12:24:56.356071 (+286746us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:56.476156 (+120085us) log.cc:844] Serialized 26 byte log entry
0709 12:24:56.476341 (+   185us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:24:56.476388 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:56.476549 (+   161us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:56.513630 (+ 37081us) raft_consensus.cc:1685] finished
0709 12:24:56.524848 (+ 11218us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:56.541143 (+ 16295us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":1083902,"prepare.run_cpu_time_us":60,"prepare.run_wall_time_us":3314,"raft.queue_time_us":19323,"raft.run_cpu_time_us":306,"raft.run_wall_time_us":21878,"spinlock_wait_cycles":34798464,"thread_start_us":348265,"threads_started":2,"wal-append.queue_time_us":284165}
I20260709 12:24:56.564525 27275 raft_consensus.cc:2749] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:24:56.629509 17835 raft_consensus.cc:3060] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:24:56.630986 17756 raft_consensus.cc:2468] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 27.
I20260709 12:24:56.591020 26962 raft_consensus.cc:493] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:24:56.632705 17838 raft_consensus.cc:2393] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 24: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:24:56.606343 26789 raft_consensus.cc:2804] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:24:56.636056 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:56.601755 27189 raft_consensus.cc:493] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:56.641407 17764 raft_consensus.cc:2393] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 24: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:24:56.564280 27245 consensus_queue.cc:1059] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 28.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.062s
I20260709 12:24:56.646862 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 30 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:56.647475 17760 raft_consensus.cc:2393] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 27: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:24:56.650499 26962 raft_consensus.cc:515] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:56.650866 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:56.651845 17837 raft_consensus.cc:2393] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 24: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:24:56.652933 26962 leader_election.cc:290] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:56.661031 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 27 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:56.661914 18374 raft_consensus.cc:2468] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 26.
I20260709 12:24:56.664062 27205 raft_consensus.cc:2804] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:24:56.668516 26910 leader_election.cc:290] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 25 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:56.677094 26789 raft_consensus.cc:697] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 28 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:24:56.678579 26789 consensus_queue.cc:237] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 25.14, Last appended by leader: 14, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:56.626482 27197 consensus_queue.cc:237] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 24.17, Last appended by leader: 17, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:56.683756 27237 raft_consensus.cc:493] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:24:56.684640 27237 raft_consensus.cc:515] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:56.687340 27237 leader_election.cc:290] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:56.704010 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 28 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
W20260709 12:24:56.712271 17803 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:56.755306 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:56.698947 27062 raft_consensus.cc:3055] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:24:56.629047 27008 raft_consensus.cc:493] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:56.711365 27189 raft_consensus.cc:515] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:56.789858 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:56.803839 17837 raft_consensus.cc:2393] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 27: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:24:56.804425 18367 raft_consensus.cc:3060] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:24:56.806195 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:56.808480 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:56.818560 17835 raft_consensus.cc:2468] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 29.
I20260709 12:24:56.790730 27205 raft_consensus.cc:697] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 27 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:24:56.825860 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:56.836707 27128 raft_consensus.cc:493] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:56.837318 27128 raft_consensus.cc:515] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:56.838337 27093 raft_consensus.cc:3055] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:24:56.796126 27062 raft_consensus.cc:740] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 25 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:24:56.838794 27093 raft_consensus.cc:740] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:24:56.816077 26957 raft_consensus.cc:493] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:56.827067 26962 raft_consensus.cc:493] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:24:56.853641 26962 raft_consensus.cc:515] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:56.854926 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 29 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:56.856557 27189 leader_election.cc:290] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:56.849258 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:56.869897 27303 raft_consensus.cc:493] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:24:56.870198 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 24 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:56.870962 17833 raft_consensus.cc:2393] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 24: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:24:56.870682 27303 raft_consensus.cc:515] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:56.873209 27093 consensus_queue.cc:260] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 23.17, Last appended by leader: 17, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:56.874116 27303 leader_election.cc:290] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:56.874557 27205 consensus_queue.cc:237] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 25.16, Last appended by leader: 16, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:56.875553 27062 consensus_queue.cc:260] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 25.15, Last appended by leader: 15, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:56.876734 27062 raft_consensus.cc:3060] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:56.881829 18333 leader_election.cc:304] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:56.816727 27008 raft_consensus.cc:515] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:56.811591 27259 raft_consensus.cc:515] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:56.889739 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 284951us
W20260709 12:24:56.862439 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 249632us
I20260709 12:24:56.902624 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:56.903753 17833 raft_consensus.cc:2393] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 23: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:24:56.909898 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:56.912489 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:56.913058 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 26 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:56.913885 17652 leader_election.cc:304] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:56.915695 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 29 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:56.916846 17761 raft_consensus.cc:2393] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 25: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:24:56.916545 17757 raft_consensus.cc:3060] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:24:56.920987 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:56.896677 26910 raft_consensus.cc:493] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:56.924224 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:56.915729 18334 leader_election.cc:304] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:56.906831 26818 raft_consensus.cc:3060] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:24:56.929512 18364 raft_consensus.cc:2393] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 27: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:24:56.926141 17834 raft_consensus.cc:2393] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 26: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:24:56.937497 27111 consensus_queue.cc:1048] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:24:56.937863 26887 consensus_queue.cc:1048] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20260709 12:24:56.938150 26888 consensus_queue.cc:1059] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 17.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 12:24:56.939121 27111 raft_consensus.cc:3055] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:24:56.939499 27111 raft_consensus.cc:740] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 17 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:24:56.940521 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 27 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:56.940362 27111 consensus_queue.cc:260] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.11, Last appended by leader: 11, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:56.941061 17764 raft_consensus.cc:2468] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 26.
I20260709 12:24:56.941704 27111 raft_consensus.cc:3060] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:56.941886 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:56.916620 26962 leader_election.cc:290] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:56.938275 17801 leader_election.cc:304] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:24:56.891999 26957 raft_consensus.cc:515] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:56.923471 27197 raft_consensus.cc:2749] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260709 12:24:56.951290 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 69585.9us
I20260709 12:24:56.949450 26910 raft_consensus.cc:515] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:56.953718 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 43996.9us
I20260709 12:24:56.957512 27238 leader_election.cc:290] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:56.970767 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:56.971519 26887 raft_consensus.cc:2804] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:24:56.972340 18333 leader_election.cc:304] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:56.946520 27107 raft_consensus.cc:2804] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:24:56.979477 18334 leader_election.cc:304] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:24:56.948855 27202 consensus_meta.cc:287] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.510s	user 0.012s	sys 0.001s
I20260709 12:24:56.982728 17687 raft_consensus.cc:1240] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 21. Current term is 22. Ops: []
I20260709 12:24:56.990139 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:56.990648 27202 raft_consensus.cc:515] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:56.991420 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:56.992192 17686 raft_consensus.cc:2393] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 25: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:24:56.992926 27202 leader_election.cc:290] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 25 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:56.993568 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 32 candidate_status { last_received { term: 30 index: 23 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:56.994211 17680 raft_consensus.cc:1275] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 24 index: 17. Preceding OpId from leader: term: 25 index: 18. (index mismatch)
I20260709 12:24:56.994457 26887 raft_consensus.cc:697] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 27 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:24:56.995455 26887 consensus_queue.cc:237] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 25.14, Last appended by leader: 14, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:56.992877 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:56.999194 17682 raft_consensus.cc:3060] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:56.995385 18366 raft_consensus.cc:1275] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 23 index: 19. Preceding OpId from leader: term: 24 index: 20. (index mismatch)
I20260709 12:24:57.003013 17652 leader_election.cc:304] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:24:57.007910 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 32 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:57.007799 27275 raft_consensus.cc:493] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:57.013064 18363 raft_consensus.cc:2468] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 23.
I20260709 12:24:57.023989 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:57.023402 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 27 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:57.026244 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:57.029960 17687 raft_consensus.cc:2393] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 27: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:24:57.032370 18364 raft_consensus.cc:1275] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 20 index: 15. Preceding OpId from leader: term: 21 index: 15. (term mismatch)
I20260709 12:24:57.032874 18364 pending_rounds.cc:85] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da: Aborting all ops after (but not including) 14
I20260709 12:24:57.033843 17680 raft_consensus.cc:1240] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 27. Current term is 28. Ops: [27.15-27.15]
I20260709 12:24:57.035357 17684 raft_consensus.cc:2468] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 26.
I20260709 12:24:57.037719 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:57.044456 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:57.046093 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:57.046664 17681 raft_consensus.cc:2468] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 27.
I20260709 12:24:57.047134 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:57.047564 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:57.048640 27259 leader_election.cc:290] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 25 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:57.049257 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 27 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:57.049431 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 29 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:57.050156 17686 raft_consensus.cc:2393] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 29: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:24:57.050455 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 22 candidate_status { last_received { term: 21 index: 20 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:57.050963 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 28 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:57.047838 17684 raft_consensus.cc:2393] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 25: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
W20260709 12:24:57.052904 27300 log.cc:927] Time spent T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.073s	user 0.000s	sys 0.000s
I20260709 12:24:57.053684 17684 raft_consensus.cc:1275] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 23 index: 17. Preceding OpId from leader: term: 25 index: 18. (index mismatch)
I20260709 12:24:57.051206 17681 raft_consensus.cc:2393] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 22: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:24:57.054834 27008 leader_election.cc:290] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:24:57.056224 27283 consensus_meta.cc:287] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.798s	user 0.015s	sys 0.003s
I20260709 12:24:57.051771 17686 raft_consensus.cc:2410] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for term 28 because replica has last-logged OpId of term: 26 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 18.
I20260709 12:24:57.060271 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:57.061154 18363 raft_consensus.cc:2393] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 26: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:24:57.035254 26924 raft_consensus.cc:2804] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:24:57.062235 26924 raft_consensus.cc:697] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 27 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:24:57.062927 26818 raft_consensus.cc:2749] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 32 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:24:57.063190 26924 consensus_queue.cc:237] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 26.18, Last appended by leader: 18, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:57.044471 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 31 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:24:57.065268 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 85876.4us
I20260709 12:24:57.066344 18334 leader_election.cc:304] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:57.067165 26818 raft_consensus.cc:2749] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:24:57.028523 26962 raft_consensus.cc:2749] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:24:57.008448 27245 consensus_queue.cc:1048] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.034s
I20260709 12:24:57.035586 27197 raft_consensus.cc:2804] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:24:57.074002 27197 raft_consensus.cc:493] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:24:57.074406 27197 raft_consensus.cc:3060] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:57.011037 27154 raft_consensus.cc:493] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:57.063772 18365 raft_consensus.cc:1240] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 32 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 31. Current term is 32. Ops: []
I20260709 12:24:57.038674 26888 raft_consensus.cc:493] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:24:57.033172 18364 pending_rounds.cc:107] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da: Aborting uncommitted NO_OP operation due to leader change: 20.15
I20260709 12:24:57.082109 18364 raft_consensus.cc:2889] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:57.035936 27275 raft_consensus.cc:515] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:57.083810 18334 leader_election.cc:304] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:57.086123 17757 raft_consensus.cc:2468] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 29.
I20260709 12:24:56.984975 27107 raft_consensus.cc:493] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:57.004371 26957 leader_election.cc:290] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:57.037264 27128 leader_election.cc:290] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:57.085583 27154 raft_consensus.cc:515] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:57.090517 17728 leader_election.cc:304] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:57.090512 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:57.090989 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:57.092301 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:57.116292 18371 raft_consensus.cc:2393] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 26: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:24:57.088717 18366 raft_consensus.cc:2393] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 23: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:24:57.096731 27266 consensus_queue.cc:1048] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.009s
I20260709 12:24:57.098523 17682 raft_consensus.cc:2468] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 26.
I20260709 12:24:57.099733 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:57.122803 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:57.121717 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:57.127712 18374 raft_consensus.cc:2393] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 25: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:24:57.126142 17763 raft_consensus.cc:2468] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 22.
I20260709 12:24:57.128679 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:57.124719 17757 raft_consensus.cc:1240] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 32 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 31. Current term is 32. Ops: [31.24-31.24]
I20260709 12:24:57.096521 27107 raft_consensus.cc:3060] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Advancing to term 26
W20260709 12:24:57.113271 27239 consensus_meta.cc:287] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.695s	user 0.010s	sys 0.002s
I20260709 12:24:57.132854 18367 raft_consensus.cc:2468] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 30.
I20260709 12:24:57.138921 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 27 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:57.141284 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:57.141625 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:57.142647 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:57.144129 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 31 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:57.144820 18374 raft_consensus.cc:2393] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 25: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:24:57.146236 17832 raft_consensus.cc:1240] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 22. Current term is 23. Ops: []
I20260709 12:24:57.127161 27035 raft_consensus.cc:2749] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:24:57.140641 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:57.150800 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:57.149622 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 22 candidate_status { last_received { term: 21 index: 20 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:57.158510 17833 raft_consensus.cc:2468] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 22.
I20260709 12:24:57.151564 18368 raft_consensus.cc:2468] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 23.
W20260709 12:24:57.159320 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 75571.6us
I20260709 12:24:57.160391 18334 leader_election.cc:304] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:24:57.096869 26888 raft_consensus.cc:515] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:57.174007 27293 log.cc:927] Time spent T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.856s	user 0.002s	sys 0.004s
I20260709 12:24:57.174839 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:57.175670 18374 raft_consensus.cc:3060] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:57.176079 17688 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37402 (request call id 3828) took 1108 ms. Trace:
I20260709 12:24:57.176458 17688 rpcz_store.cc:276] 0709 12:24:56.067597 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:56.075389 (+  7792us) service_pool.cc:224] Handling call
0709 12:24:56.075836 (+   447us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:24:56.076412 (+   576us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:56.076459 (+    47us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:24:56.077189 (+   730us) log.cc:844] Serialized 52 byte log entry
0709 12:24:56.126450 (+ 49261us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:24:56.151294 (+ 24844us) log.cc:844] Serialized 14 byte log entry
0709 12:24:56.155681 (+  4387us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:56.155793 (+   112us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:56.225655 (+ 69862us) spinlock_profiling.cc:243] Waited 1.29 ms on lock 0x7b180026e5a0. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b47533522 00007f7b475357c1 00007f7b47535779 00007f7b47535709 00007f7b475356d1 00007f7b475349cd 0000000000388634 0000000000388479 00007f7b4751683a
0709 12:24:57.173450 (+947795us) spinlock_profiling.cc:243] Waited 10.3 ms on lock 0x7b4400022b4c. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b47459c85 00007f7b4745a6a2 00007f7b4a1cc1db 00007f7b4a1c7aaf 00007f7b4a1bb86e 00007f7b4a1babb6 00007f7b4a1ba7d8 00007f7b4a1c4221 00007f7b4a1c41d9 00007f7b4a1c4169
0709 12:24:57.175733 (+  2283us) raft_consensus.cc:1685] finished
0709 12:24:57.175754 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:57.175912 (+   158us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":34397,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":2648,"raft.queue_time_us":4188,"raft.run_cpu_time_us":253,"raft.run_wall_time_us":5072,"spinlock_wait_cycles":26568192,"thread_start_us":48766,"threads_started":1,"wal-append.queue_time_us":121319}
I20260709 12:24:57.140538 26818 raft_consensus.cc:493] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:24:57.178819 26818 raft_consensus.cc:515] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:57.148667 27245 raft_consensus.cc:493] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:57.179862 27202 raft_consensus.cc:493] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:57.180266 27245 raft_consensus.cc:515] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:57.180488 27202 raft_consensus.cc:515] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:57.180923 26818 leader_election.cc:290] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:57.182330 27245 leader_election.cc:290] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:57.182631 27202 leader_election.cc:290] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:57.183559 17834 raft_consensus.cc:1275] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 21 index: 16. Preceding OpId from leader: term: 25 index: 18. (index mismatch)
I20260709 12:24:57.185578 18367 raft_consensus.cc:2393] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 23: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
W20260709 12:24:57.188083 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 97648.4us
I20260709 12:24:57.190069 17758 raft_consensus.cc:2468] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 26.
I20260709 12:24:57.157704 26957 raft_consensus.cc:2749] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
I20260709 12:24:57.196228 18363 raft_consensus.cc:2393] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 23: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:24:57.196408 18368 raft_consensus.cc:1275] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 24 index: 18. Preceding OpId from leader: term: 26 index: 19. (index mismatch)
I20260709 12:24:57.200920 17764 raft_consensus.cc:1240] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 25. Current term is 26. Ops: []
I20260709 12:24:57.199208 17728 leader_election.cc:304] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:57.202607 17762 pending_rounds.cc:85] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb: Aborting all ops after (but not including) 13
I20260709 12:24:57.203009 17762 pending_rounds.cc:107] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb: Aborting uncommitted NO_OP operation due to leader change: 22.14
I20260709 12:24:57.203291 17762 raft_consensus.cc:2889] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:24:57.200927 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 197985us
I20260709 12:24:57.194689 27266 raft_consensus.cc:2804] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:24:57.232463 17728 leader_election.cc:304] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:57.189702 27259 raft_consensus.cc:493] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:24:57.190582 27062 raft_consensus.cc:493] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:57.197309 27283 consensus_queue.cc:1048] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.038s
I20260709 12:24:57.237562 18367 raft_consensus.cc:1275] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 27 index: 17. Preceding OpId from leader: term: 28 index: 17. (term mismatch)
I20260709 12:24:57.238773 17728 leader_election.cc:304] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:57.238725 18333 leader_election.cc:304] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:24:57.243628 17728 leader_election.cc:304] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
W20260709 12:24:57.244032 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 81897.4us
I20260709 12:24:57.238003 18367 pending_rounds.cc:85] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da: Aborting all ops after (but not including) 16
I20260709 12:24:57.247368 18367 pending_rounds.cc:107] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da: Aborting uncommitted NO_OP operation due to leader change: 27.17
W20260709 12:24:57.247414 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 87142.4us
I20260709 12:24:57.247910 18367 raft_consensus.cc:2889] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:57.251927 18374 raft_consensus.cc:2468] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 24.
I20260709 12:24:57.249109 18334 leader_election.cc:304] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:24:57.260277 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:57.260890 18333 leader_election.cc:304] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:57.261149 17833 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 25.
I20260709 12:24:57.233800 27266 raft_consensus.cc:493] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:57.265219 17683 raft_consensus.cc:1240] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 27 LEADER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 26. Current term is 27. Ops: []
I20260709 12:24:57.264945 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 31 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:57.270608 17764 raft_consensus.cc:2410] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 31 because replica has last-logged OpId of term: 29 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 28 index: 16.
I20260709 12:24:57.271477 17728 leader_election.cc:304] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:57.243584 27237 raft_consensus.cc:2749] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:24:57.243139 26957 raft_consensus.cc:2749] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 12:24:57.252398 27238 raft_consensus.cc:2804] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:24:57.235631 27245 consensus_queue.cc:1048] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.033s
I20260709 12:24:57.243757 27275 leader_election.cc:290] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:57.263402 27035 raft_consensus.cc:2804] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:24:57.269102 27266 raft_consensus.cc:3060] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:57.252139 27303 consensus_queue.cc:1059] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 25.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.032s
I20260709 12:24:57.282877 27202 raft_consensus.cc:2749] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 12:24:57.262877 27062 raft_consensus.cc:515] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:57.335795 17804 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:57.284062 26818 raft_consensus.cc:2804] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:24:57.332546 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:57.283222 27238 raft_consensus.cc:493] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:57.268813 27259 raft_consensus.cc:515] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:57.351660 27240 raft_consensus.cc:3055] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:24:57.357517 27240 raft_consensus.cc:740] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:24:57.293253 27035 raft_consensus.cc:493] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:57.295567 26910 leader_election.cc:290] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:24:57.364861 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 107417us
I20260709 12:24:57.359385 27238 raft_consensus.cc:3060] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:24:57.377194 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 116354us
I20260709 12:24:57.381095 17764 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 25.
I20260709 12:24:57.356572 26818 raft_consensus.cc:493] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:57.381559 18333 leader_election.cc:304] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:24:57.362702 27328 raft_consensus.cc:2749] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:24:57.378240 27205 raft_consensus.cc:493] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:57.361738 27035 raft_consensus.cc:3060] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:24:57.410055 27315 log.cc:927] Time spent T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.119s	user 0.000s	sys 0.000s
I20260709 12:24:57.367508 27154 leader_election.cc:290] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:24:57.413187 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 126603us
I20260709 12:24:57.394601 26818 raft_consensus.cc:3060] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:24:57.360749 27327 consensus_queue.cc:1059] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 26.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.046s
I20260709 12:24:57.420202 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:57.419672 17652 leader_election.cc:304] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:24:57.438442 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 57558.9us
I20260709 12:24:57.410516 27128 raft_consensus.cc:493] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:57.397778 27240 consensus_queue.cc:260] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 22.15, Last appended by leader: 15, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:57.452622 27240 raft_consensus.cc:3060] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:57.456023 27128 raft_consensus.cc:515] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:57.459520 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 26 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:57.460577 17757 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 25.
I20260709 12:24:57.460163 27325 consensus_queue.cc:1048] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.036s
W20260709 12:24:57.465909 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 194496us
I20260709 12:24:57.463256 17764 raft_consensus.cc:2468] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 31.
W20260709 12:24:57.472085 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 90606.6us
I20260709 12:24:57.466914 17728 leader_election.cc:304] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:57.473242 27328 raft_consensus.cc:2804] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:24:57.473730 27328 raft_consensus.cc:493] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:24:57.474722 17728 leader_election.cc:304] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:57.443464 27245 raft_consensus.cc:2749] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:24:57.474560 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 26 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:57.477770 27202 raft_consensus.cc:2749] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:24:57.478011 17763 raft_consensus.cc:2410] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for term 26 because replica has last-logged OpId of term: 24 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 18.
I20260709 12:24:57.479717 26599 leader_election.cc:290] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:24:57.480082 27313 log.cc:927] Time spent T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.232s	user 0.000s	sys 0.001s
I20260709 12:24:57.474129 27328 raft_consensus.cc:3060] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:24:57.480563 27324 raft_consensus.cc:2749] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:24:57.481870 27324 raft_consensus.cc:2804] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:24:57.482318 27324 raft_consensus.cc:697] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 29 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
W20260709 12:24:57.484549 27318 log.cc:927] Time spent T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.122s	user 0.000s	sys 0.001s
W20260709 12:24:57.488526 27314 log.cc:927] Time spent T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.128s	user 0.000s	sys 0.000s
I20260709 12:24:57.414594 27205 raft_consensus.cc:515] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:57.459528 27128 leader_election.cc:290] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:57.473032 18333 leader_election.cc:304] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
W20260709 12:24:57.490576 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 71025.9us
I20260709 12:24:57.492792 17652 leader_election.cc:304] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:57.499248 17764 raft_consensus.cc:1275] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 29 index: 17. Preceding OpId from leader: term: 30 index: 17. (term mismatch)
W20260709 12:24:57.502949 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 105577us
I20260709 12:24:57.483179 27324 consensus_queue.cc:237] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 28.18, Last appended by leader: 18, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:57.509214 17652 leader_election.cc:304] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:57.509331 18333 leader_election.cc:304] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:57.510351 17728 leader_election.cc:304] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:57.499670 17764 pending_rounds.cc:85] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb: Aborting all ops after (but not including) 16
I20260709 12:24:57.511821 27328 raft_consensus.cc:515] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:57.515234 27205 leader_election.cc:290] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:57.522228 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 32 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:57.539477 27328 leader_election.cc:290] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 28 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:57.539808 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:57.517349 27325 raft_consensus.cc:2749] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 12:24:57.522948 17764 pending_rounds.cc:107] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb: Aborting uncommitted NO_OP operation due to leader change: 29.17
I20260709 12:24:57.540870 17764 raft_consensus.cc:2889] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:24:57.526032 27317 log.cc:927] Time spent T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.235s	user 0.001s	sys 0.000s
I20260709 12:24:57.542162 27325 raft_consensus.cc:2749] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:24:57.543301 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:57.543980 17764 raft_consensus.cc:2468] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 27.
I20260709 12:24:57.544827 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:57.545929 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:57.546561 17764 raft_consensus.cc:2468] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 22.
I20260709 12:24:57.516237 26844 raft_consensus.cc:3055] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:24:57.548334 26844 raft_consensus.cc:740] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:24:57.549180 26844 consensus_queue.cc:260] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:57.550414 26844 raft_consensus.cc:3060] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Advancing to term 19
W20260709 12:24:57.526536 27319 log.cc:927] Time spent T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.229s	user 0.000s	sys 0.001s
I20260709 12:24:57.523998 27128 raft_consensus.cc:2749] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
W20260709 12:24:57.553977 17730 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:57.510633 17801 leader_election.cc:304] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:24:57.554013 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 44871.8us
I20260709 12:24:57.555084 18333 leader_election.cc:304] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3
W20260709 12:24:57.557175 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 46949.7us
I20260709 12:24:57.559782 17652 leader_election.cc:304] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:57.561599 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:57.562285 17757 raft_consensus.cc:2468] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 25.
W20260709 12:24:57.571231 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 60653.3us
I20260709 12:24:57.574075 17728 leader_election.cc:304] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:24:57.576915 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
W20260709 12:24:57.521474 27111 consensus_meta.cc:287] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.579s	user 0.006s	sys 0.006s
I20260709 12:24:57.585057 17838 raft_consensus.cc:1275] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 29 index: 20. Preceding OpId from leader: term: 31 index: 21. (index mismatch)
I20260709 12:24:57.587491 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:57.588716 18364 raft_consensus.cc:2393] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 25: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:24:57.561471 27328 raft_consensus.cc:2749] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:24:57.593394 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:57.552284 27259 leader_election.cc:290] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:57.591149 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 29 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:57.561393 27245 raft_consensus.cc:2749] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:24:57.621804 17760 raft_consensus.cc:2393] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 27: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
W20260709 12:24:57.622081 17758 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38786 (request call id 4051) took 1253 ms (client timeout 1512 ms). Trace:
W20260709 12:24:57.622512 17758 rpcz_store.cc:269] 0709 12:24:56.368848 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:56.369047 (+   199us) service_pool.cc:224] Handling call
0709 12:24:57.621914 (+1252867us) spinlock_profiling.cc:243] Waited 755 ms on lock 0x7b54004150d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:24:57.622023 (+   109us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1736191232}
I20260709 12:24:57.623215 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 27 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:57.624228 26818 raft_consensus.cc:515] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:57.626315 27196 raft_consensus.cc:2749] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:24:57.590392 27336 raft_consensus.cc:2804] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:24:57.627704 27336 raft_consensus.cc:697] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:24:57.604807 27171 consensus_queue.cc:1048] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.007s
I20260709 12:24:57.628736 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:57.628685 27336 consensus_queue.cc:237] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 25.15, Last appended by leader: 15, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:57.629376 27339 raft_consensus.cc:493] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:57.629959 27339 raft_consensus.cc:515] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:57.630844 26818 leader_election.cc:290] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 28 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:57.631439 26818 raft_consensus.cc:493] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:57.606243 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 25 candidate_status { last_received { term: 23 index: 19 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:57.633288 18367 raft_consensus.cc:2468] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 24.
I20260709 12:24:57.633773 27111 raft_consensus.cc:493] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:57.634325 27111 raft_consensus.cc:515] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:57.635068 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:57.598042 18333 leader_election.cc:304] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:57.635939 27341 raft_consensus.cc:2749] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:24:57.636415 27111 leader_election.cc:290] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:57.637902 18333 leader_election.cc:304] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:57.638922 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:57.639139 27238 raft_consensus.cc:515] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:57.639755 17654 leader_election.cc:304] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:57.641012 27316 raft_consensus.cc:2749] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 12:24:57.605688 27335 raft_consensus.cc:2804] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:24:57.616438 26956 raft_consensus.cc:3055] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:24:57.629599 17756 raft_consensus.cc:2393] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 27: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:24:57.578473 27205 raft_consensus.cc:493] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:24:57.591629 17729 leader_election.cc:304] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:24:57.654433 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 62890.3us
I20260709 12:24:57.640993 17654 leader_election.cc:304] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:57.624678 27329 raft_consensus.cc:2804] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:24:57.634967 27339 leader_election.cc:290] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:57.604189 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 26 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:57.612491 27235 leader_election.cc:290] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 24 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:57.614241 26888 leader_election.cc:290] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:57.653246 17756 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38776 (request call id 3252) took 1068 ms. Trace:
I20260709 12:24:57.689188 17763 raft_consensus.cc:1240] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 26. Current term is 27. Ops: []
I20260709 12:24:57.657512 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 25 candidate_status { last_received { term: 23 index: 19 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:57.662521 18368 raft_consensus.cc:1275] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 21 index: 16. Preceding OpId from leader: term: 23 index: 17. (index mismatch)
W20260709 12:24:57.690449 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 130786us
I20260709 12:24:57.691169 17652 leader_election.cc:304] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:57.662092 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:57.663410 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:57.692492 18374 raft_consensus.cc:2393] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 24: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:24:57.641618 27196 consensus_queue.cc:1048] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.011s
I20260709 12:24:57.576205 27325 consensus_queue.cc:1048] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.077s
I20260709 12:24:57.693699 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:57.562848 27197 raft_consensus.cc:515] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:57.664031 27321 raft_consensus.cc:3060] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:57.644937 26818 raft_consensus.cc:515] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:57.654983 27107 consensus_meta.cc:287] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.514s	user 0.011s	sys 0.000s
I20260709 12:24:57.706552 17729 leader_election.cc:304] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:57.662073 27329 raft_consensus.cc:493] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:24:57.732721 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:57.652609 27335 raft_consensus.cc:493] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:57.661010 26957 raft_consensus.cc:2804] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:24:57.676944 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 28 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:57.677619 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 25 candidate_status { last_received { term: 23 index: 19 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:57.678556 17684 raft_consensus.cc:1240] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 28. Current term is 29. Ops: [28.19-28.19]
I20260709 12:24:57.680119 18367 raft_consensus.cc:1240] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 26. Current term is 27. Ops: []
I20260709 12:24:57.653867 26956 raft_consensus.cc:740] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:24:57.681164 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:57.681437 18333 leader_election.cc:304] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:24:57.681679 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 27 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:57.682000 27341 consensus_queue.cc:1048] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.000s
I20260709 12:24:57.685209 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:57.687333 17803 leader_election.cc:304] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:57.689441 17756 rpcz_store.cc:276] 0709 12:24:56.584463 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:56.763760 (+179297us) service_pool.cc:224] Handling call
0709 12:24:57.628589 (+864829us) spinlock_profiling.cc:243] Waited 4.34 ms on lock 0x7b5800033398. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b474fbb71 00007f7b474fba22 00007f7b504148d9 00007f7b4f56eee0 00007f7b4f6380ab 00007f7b4f638131 00007f7b4f5e60a0 00007f7b4f5e5984 00007f7b48b606d4 00007f7b48b60662
0709 12:24:57.653190 (+ 24601us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":9983232}
I20260709 12:24:57.718861 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 29 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:57.696238 18361 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 25.
W20260709 12:24:57.693142 27320 log.cc:927] Time spent T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.252s	user 0.000s	sys 0.003s
I20260709 12:24:57.729426 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:57.664247 27245 raft_consensus.cc:2804] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:24:57.733949 17652 leader_election.cc:304] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:57.693595 26962 raft_consensus.cc:493] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:57.795249 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 88816.8us
I20260709 12:24:57.798465 17764 raft_consensus.cc:2393] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 26: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:24:57.790140 17689 raft_consensus.cc:3060] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:24:57.790326 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 29 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:57.791610 17682 raft_consensus.cc:3060] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Advancing to term 28
W20260709 12:24:57.744848 17728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:57.794939 18364 raft_consensus.cc:2468] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 28.
I20260709 12:24:57.791968 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 28 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:57.793237 17687 raft_consensus.cc:1240] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 25. Current term is 26. Ops: []
W20260709 12:24:57.793344 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 111998us
I20260709 12:24:57.716367 26844 raft_consensus.cc:493] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:24:57.804792 18333 leader_election.cc:304] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:57.806689 18361 raft_consensus.cc:1240] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 23. Current term is 24. Ops: []
I20260709 12:24:57.807447 26962 raft_consensus.cc:515] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:57.814860 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:57.815718 17756 raft_consensus.cc:3060] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:24:57.817786 18362 raft_consensus.cc:2393] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 28: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:24:57.745460 27329 raft_consensus.cc:3060] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:57.786998 27128 raft_consensus.cc:2804] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:24:57.787436 27328 raft_consensus.cc:493] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:24:57.834213 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:57.835803 18374 raft_consensus.cc:2468] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 25.
I20260709 12:24:57.794790 27245 raft_consensus.cc:493] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:57.809819 27235 raft_consensus.cc:2804] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:24:57.794567 26957 raft_consensus.cc:493] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:24:57.691828 27205 raft_consensus.cc:515] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:57.833156 17756 raft_consensus.cc:2468] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 28.
W20260709 12:24:57.848294 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 43605.6us
I20260709 12:24:57.860651 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 33 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:57.821866 27171 raft_consensus.cc:2749] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:24:57.844201 27328 raft_consensus.cc:515] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:57.876732 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 24 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:24:57.852695 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 165408us
I20260709 12:24:57.877995 17764 raft_consensus.cc:2393] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 24: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:24:57.744625 27107 raft_consensus.cc:515] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:57.906245 27107 leader_election.cc:290] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 26 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:57.906872 18334 leader_election.cc:304] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:57.907084 27107 consensus_queue.cc:1059] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 26.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:24:57.908663 27238 leader_election.cc:290] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 27 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:57.846287 18368 raft_consensus.cc:2468] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 23.
I20260709 12:24:57.862629 27235 raft_consensus.cc:697] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 30 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:24:57.870180 17756 raft_consensus.cc:2410] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 33 because replica has last-logged OpId of term: 31 index: 20, which is greater than that of the candidate, which has last-logged OpId of term: 29 index: 19.
W20260709 12:24:57.871317 17789 heartbeater.cc:646] Failed to heartbeat to 127.16.29.126:39149 (7 consecutive failures): Timed out: Failed to send heartbeat to master: TSHeartbeat RPC to 127.16.29.126:39149 timed out after 15.000s (SENT)
I20260709 12:24:57.836547 27128 raft_consensus.cc:493] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:24:57.786633 27035 raft_consensus.cc:515] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:57.875047 17682 raft_consensus.cc:2468] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 28.
I20260709 12:24:57.746482 27335 raft_consensus.cc:3060] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:24:57.878139 17803 leader_election.cc:304] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:57.847904 27245 raft_consensus.cc:3060] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:57.881356 27338 raft_consensus.cc:2804] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:24:57.881872 27328 leader_election.cc:290] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:57.882953 26924 raft_consensus.cc:3055] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:24:57.883996 27307 raft_consensus.cc:493] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:57.884905 17755 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d"
dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
mode: GRACEFUL
new_leader_uuid: "e063541542a94969898868bf6eb5ce1b"
 from {username='slave'} at 127.0.0.1:49786
I20260709 12:24:57.834309 27239 consensus_queue.cc:1059] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.038s
I20260709 12:24:57.888547 26956 consensus_queue.cc:260] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 27.15, Last appended by leader: 15, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:57.890355 26818 leader_election.cc:290] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:57.898406 18366 raft_consensus.cc:1275] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 21 index: 15. Preceding OpId from leader: term: 23 index: 16. (index mismatch)
I20260709 12:24:57.902063 27093 raft_consensus.cc:3060] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:57.909230 27343 raft_consensus.cc:2749] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:24:57.838914 26844 raft_consensus.cc:515] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:57.859043 27240 raft_consensus.cc:2749] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:24:57.911214 17760 raft_consensus.cc:1240] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 28. Current term is 29. Ops: [28.19-28.19]
W20260709 12:24:57.855392 17802 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:57.972751 17684 raft_consensus.cc:1275] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 26 index: 18. Preceding OpId from leader: term: 27 index: 19. (index mismatch)
I20260709 12:24:57.918663 17689 raft_consensus.cc:2468] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 28.
I20260709 12:24:57.862241 18333 leader_election.cc:304] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:57.928141 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 27 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:57.928949 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:57.955107 26924 raft_consensus.cc:740] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:24:57.861155 26957 raft_consensus.cc:3060] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:57.859005 27196 raft_consensus.cc:2804] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Leader election won for term 29
W20260709 12:24:57.939965 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 61901.3us
I20260709 12:24:57.940966 17728 leader_election.cc:304] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:24:58.003458 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 62589.1us
I20260709 12:24:58.003782 17803 leader_election.cc:304] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 31 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:57.938942 27128 raft_consensus.cc:3060] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:57.942946 27235 consensus_queue.cc:237] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 28.21, Last appended by leader: 21, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:57.943934 27035 leader_election.cc:290] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 24 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:57.863471 27339 raft_consensus.cc:493] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:58.009335 27339 raft_consensus.cc:515] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:58.010567 27335 raft_consensus.cc:515] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:58.011731 27339 leader_election.cc:290] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:57.944693 27353 raft_consensus.cc:493] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:58.012543 27353 raft_consensus.cc:515] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:58.012854 27359 consensus_queue.cc:1048] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:24:57.946858 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:58.013927 17688 raft_consensus.cc:2468] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 27.
I20260709 12:24:57.937361 27325 consensus_queue.cc:1059] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 25.19, Next index: 20, Last known committed idx: 18, Time since last communication: 0.017s
I20260709 12:24:57.948031 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 27 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:57.948678 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:57.955578 17755 raft_consensus.cc:606] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Received request to transfer leadership to TS e063541542a94969898868bf6eb5ce1b
I20260709 12:24:58.016158 17755 raft_consensus.cc:614] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:24:57.948026 27171 raft_consensus.cc:2749] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
W20260709 12:24:57.938666 27266 consensus_meta.cc:287] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.604s	user 0.006s	sys 0.006s
I20260709 12:24:57.958808 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 24 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:57.961964 17756 raft_consensus.cc:1275] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 29 index: 20. Preceding OpId from leader: term: 31 index: 21. (index mismatch)
I20260709 12:24:57.964043 27239 consensus_queue.cc:1243] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer 0da169ff82674dc1947055ce43de68a3 log is divergent from this leader: its last log entry 27.17 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 16
I20260709 12:24:57.964823 26956 raft_consensus.cc:3060] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Advancing to term 28
W20260709 12:24:57.968636 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 234754us
I20260709 12:24:58.032891 17803 leader_election.cc:304] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:24:57.969278 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 27 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:57.928141 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 32 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:58.033394 17652 leader_election.cc:304] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:58.001940 17764 raft_consensus.cc:1240] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 27. Current term is 28. Ops: []
I20260709 12:24:57.999145 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 26 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:58.035156 18367 raft_consensus.cc:2410] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for term 26 because replica has last-logged OpId of term: 24 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 18.
I20260709 12:24:58.013171 27335 leader_election.cc:290] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 32 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:57.946318 27205 leader_election.cc:290] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:58.015451 18362 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 25.
I20260709 12:24:58.038839 17803 leader_election.cc:304] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:57.972427 27307 raft_consensus.cc:515] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:58.003620 27008 raft_consensus.cc:493] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:58.044981 17761 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38786 (request call id 4052) took 1632 ms (client timeout 2050 ms). Trace:
W20260709 12:24:58.052217 17761 rpcz_store.cc:269] 0709 12:24:56.412835 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:56.418276 (+  5441us) service_pool.cc:224] Handling call
0709 12:24:58.044770 (+1626494us) spinlock_profiling.cc:243] Waited 485 ms on lock 0x7b540042c054. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:24:58.044909 (+   139us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1114970880}
I20260709 12:24:58.053177 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 24 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:58.045836 17756 raft_consensus.cc:1275] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 25 index: 14. Preceding OpId from leader: term: 27 index: 15. (index mismatch)
I20260709 12:24:58.053679 17803 leader_election.cc:304] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:58.054056 18374 raft_consensus.cc:2468] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 23.
W20260709 12:24:58.043099 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 180959us
I20260709 12:24:58.032666 27196 raft_consensus.cc:697] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 29 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:24:58.041911 27189 raft_consensus.cc:2804] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:24:58.061580 18367 raft_consensus.cc:1275] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 25 index: 16. Preceding OpId from leader: term: 27 index: 17. (index mismatch)
I20260709 12:24:58.062484 18366 raft_consensus.cc:1240] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 23. Current term is 24. Ops: []
I20260709 12:24:58.058816 18333 leader_election.cc:304] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:58.049571 27343 raft_consensus.cc:2749] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:24:58.057363 27316 raft_consensus.cc:2804] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:24:58.079589 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 24 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:58.045229 27354 raft_consensus.cc:493] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:24:58.084296 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:58.084920 17838 raft_consensus.cc:2468] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 25.
I20260709 12:24:58.050971 27353 leader_election.cc:290] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:58.050693 27256 raft_consensus.cc:2804] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:24:58.081882 17803 leader_election.cc:304] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:58.049511 26789 raft_consensus.cc:2749] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Leader election lost for term 31. Reason: could not achieve majority
I20260709 12:24:58.075929 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 29 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:58.056263 26599 raft_consensus.cc:2749] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:24:58.050731 27197 leader_election.cc:290] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 26 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:58.086294 17761 raft_consensus.cc:1275] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 26 index: 18. Preceding OpId from leader: term: 27 index: 19. (index mismatch)
I20260709 12:24:58.036646 27266 raft_consensus.cc:515] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:58.090574 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 25 candidate_status { last_received { term: 23 index: 19 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:24:58.094099 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 60801.1us
I20260709 12:24:58.079725 27245 raft_consensus.cc:515] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:58.097424 17687 raft_consensus.cc:1240] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 25. Current term is 26. Ops: []
I20260709 12:24:58.099972 17684 raft_consensus.cc:1275] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 21 index: 15. Preceding OpId from leader: term: 23 index: 16. (index mismatch)
W20260709 12:24:58.104601 27324 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.596s	user 0.031s	sys 0.039s
I20260709 12:24:58.107963 18333 leader_election.cc:304] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:24:58.072774 27240 raft_consensus.cc:2804] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:24:58.117410 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:58.118548 17682 raft_consensus.cc:1240] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 27. Current term is 28. Ops: []
I20260709 12:24:58.073980 27205 raft_consensus.cc:493] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:24:58.082376 27189 raft_consensus.cc:697] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:24:58.118067 17684 raft_consensus.cc:2468] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 31.
I20260709 12:24:58.087553 27256 raft_consensus.cc:493] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:58.084106 27316 raft_consensus.cc:493] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:24:58.127673 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
W20260709 12:24:58.115743 27324 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.607s	user 0.032s	sys 0.040s
I20260709 12:24:58.105857 27154 raft_consensus.cc:2749] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:24:58.140285 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 27 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:58.141664 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 31 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:58.113932 26910 consensus_queue.cc:1048] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.031s
I20260709 12:24:58.145411 17761 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665"
dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
mode: GRACEFUL
new_leader_uuid: "334a4e1f59944544b3b3ff36616b57da"
 from {username='slave'} at 127.0.0.1:49786
I20260709 12:24:58.146112 17761 raft_consensus.cc:606] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Received request to transfer leadership to TS 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:58.146471 17761 raft_consensus.cc:614] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:24:58.121794 27359 raft_consensus.cc:2804] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:24:58.149117 18365 raft_consensus.cc:1240] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 29 LEADER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 28. Current term is 29. Ops: []
I20260709 12:24:58.081607 27008 raft_consensus.cc:515] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:58.132552 27256 raft_consensus.cc:3060] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:58.141208 27316 raft_consensus.cc:3060] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:58.129559 27240 raft_consensus.cc:697] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 28 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:24:58.153026 27359 raft_consensus.cc:493] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:58.154726 26887 raft_consensus.cc:2804] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:24:58.114854 27354 raft_consensus.cc:515] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:58.125450 27205 raft_consensus.cc:515] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:58.151803 27270 raft_consensus.cc:493] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:58.147238 27197 consensus_queue.cc:1059] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 30.23, Next index: 24, Last known committed idx: 23, Time since last communication: 0.020s
I20260709 12:24:58.189750 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 26 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:58.190716 17682 raft_consensus.cc:2410] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 26 because replica has last-logged OpId of term: 24 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 12.
I20260709 12:24:58.193058 27205 leader_election.cc:290] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:58.193711 27205 raft_consensus.cc:493] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:24:58.194293 27205 raft_consensus.cc:515] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:58.171988 27359 raft_consensus.cc:3060] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:58.176331 26844 leader_election.cc:290] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:58.212656 27363 consensus_queue.cc:1048] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.055s
I20260709 12:24:58.197068 26887 raft_consensus.cc:697] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 24 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:24:58.213675 27245 leader_election.cc:290] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 24 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:58.215149 27008 leader_election.cc:290] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:58.217267 17761 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "d48b7de191684318aa08f60a7a395387"
dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
mode: GRACEFUL
new_leader_uuid: "e063541542a94969898868bf6eb5ce1b"
 from {username='slave'} at 127.0.0.1:49786
I20260709 12:24:58.237286 17761 raft_consensus.cc:606] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Received request to transfer leadership to TS e063541542a94969898868bf6eb5ce1b
I20260709 12:24:58.237655 17761 raft_consensus.cc:614] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:24:58.127473 26924 consensus_queue.cc:260] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 28.19, Last appended by leader: 19, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:58.223554 17687 raft_consensus.cc:2468] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 25.
I20260709 12:24:58.239470 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 33 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:58.240607 17682 raft_consensus.cc:2410] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 33 because replica has last-logged OpId of term: 31 index: 20, which is greater than that of the candidate, which has last-logged OpId of term: 29 index: 19.
I20260709 12:24:58.242313 17687 raft_consensus.cc:1275] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 25 index: 14. Preceding OpId from leader: term: 28 index: 15. (index mismatch)
I20260709 12:24:58.245874 17838 raft_consensus.cc:1275] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 24 index: 17. Preceding OpId from leader: term: 25 index: 18. (index mismatch)
I20260709 12:24:58.246968 17801 leader_election.cc:304] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:24:58.247444 17730 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:58.250876 17730 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:58.251366 17730 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:58.257368 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 24 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:58.256132 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 33 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:58.260113 17836 raft_consensus.cc:2468] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 32.
I20260709 12:24:58.263839 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 33 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:58.264524 17761 raft_consensus.cc:2468] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 31.
I20260709 12:24:58.265452 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 32 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:58.266295 17764 raft_consensus.cc:2410] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for term 32 because replica has last-logged OpId of term: 30 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 29 index: 17.
I20260709 12:24:58.267197 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:58.268271 17764 raft_consensus.cc:2393] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 25: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
W20260709 12:24:58.268249 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 145314us
I20260709 12:24:58.279098 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:58.279920 17687 raft_consensus.cc:1275] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 25 index: 14. Preceding OpId from leader: term: 27 index: 15. (index mismatch)
I20260709 12:24:58.189458 27270 raft_consensus.cc:515] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:58.288717 17832 raft_consensus.cc:1240] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 28. Current term is 29. Ops: []
I20260709 12:24:58.289628 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:58.291772 26599 consensus_queue.cc:1059] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 27.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:24:58.293318 26599 raft_consensus.cc:3055] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:24:58.293815 26599 raft_consensus.cc:740] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 27 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:24:58.287379 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:58.179387 27196 consensus_queue.cc:237] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 26.18, Last appended by leader: 18, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:58.302598 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:58.294981 26599 consensus_queue.cc:260] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 27.18, Last appended by leader: 18, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:24:58.305807 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 57894.4us
I20260709 12:24:58.302850 17803 leader_election.cc:304] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:24:58.308990 17803 leader_election.cc:304] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:58.307328 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 23 candidate_status { last_received { term: 21 index: 20 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:58.309772 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:58.310011 17836 raft_consensus.cc:2468] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 22.
I20260709 12:24:58.283257 27008 raft_consensus.cc:2749] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:24:58.289044 26844 raft_consensus.cc:493] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:58.311942 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:58.313602 17763 raft_consensus.cc:3060] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:24:58.313812 17652 leader_election.cc:304] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:58.315326 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:58.315950 17682 raft_consensus.cc:3060] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:24:58.316762 17764 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "d522f70beb1f417cb98940d90b594823"
dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
mode: GRACEFUL
new_leader_uuid: "e063541542a94969898868bf6eb5ce1b"
 from {username='slave'} at 127.0.0.1:49786
I20260709 12:24:58.317397 17764 raft_consensus.cc:606] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Received request to transfer leadership to TS e063541542a94969898868bf6eb5ce1b
I20260709 12:24:58.317752 17764 raft_consensus.cc:614] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:24:58.319013 27245 raft_consensus.cc:2804] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:24:58.318960 27267 consensus_queue.cc:1048] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.034s
I20260709 12:24:58.319633 27245 raft_consensus.cc:493] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:24:58.320963 27357 log.cc:927] Time spent T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.123s	user 0.001s	sys 0.000s
W20260709 12:24:58.321581 27356 log.cc:927] Time spent T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.147s	user 0.001s	sys 0.000s
W20260709 12:24:58.323769 27367 log.cc:927] Time spent T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.065s	user 0.000s	sys 0.000s
I20260709 12:24:58.324163 27266 leader_election.cc:290] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 23 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:58.327466 26962 leader_election.cc:290] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:58.330765 27369 raft_consensus.cc:2804] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:24:58.285343 27240 consensus_queue.cc:237] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 26.19, Last appended by leader: 19, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:58.313313 27154 raft_consensus.cc:2804] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:24:58.349378 27245 raft_consensus.cc:3060] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:58.312026 27205 leader_election.cc:290] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:24:58.335191 27321 consensus_meta.cc:287] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.608s	user 0.009s	sys 0.000s
I20260709 12:24:58.238545 27189 consensus_queue.cc:237] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 27.18, Last appended by leader: 18, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:24:58.380000 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:58.380990 27359 raft_consensus.cc:515] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:58.339690 26599 raft_consensus.cc:3060] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:24:58.318732 26844 raft_consensus.cc:515] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:58.349429 17728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:58.341317 27328 raft_consensus.cc:3055] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:24:58.331297 26957 raft_consensus.cc:515] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:58.352780 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 24 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:58.390643 17836 raft_consensus.cc:2468] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 23.
I20260709 12:24:58.356467 17764 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad"
dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
mode: GRACEFUL
new_leader_uuid: "e063541542a94969898868bf6eb5ce1b"
 from {username='slave'} at 127.0.0.1:49786
I20260709 12:24:58.391655 17764 raft_consensus.cc:606] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Received request to transfer leadership to TS e063541542a94969898868bf6eb5ce1b
I20260709 12:24:58.363929 27369 raft_consensus.cc:493] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:58.392089 17764 raft_consensus.cc:614] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:24:58.358434 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:58.334615 27371 consensus_queue.cc:1048] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.019s
W20260709 12:24:58.313133 27355 log.cc:927] Time spent T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.222s	user 0.001s	sys 0.000s
I20260709 12:24:58.366622 27354 leader_election.cc:290] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:58.425513 27354 consensus_queue.cc:1059] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 31.24, Next index: 25, Last known committed idx: 23, Time since last communication: 0.000s
I20260709 12:24:58.354168 27008 consensus_queue.cc:1048] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.013s
I20260709 12:24:58.364399 27154 raft_consensus.cc:493] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:58.430747 27154 raft_consensus.cc:3060] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:58.365211 17652 leader_election.cc:304] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:24:58.376096 17682 raft_consensus.cc:2468] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 32.
I20260709 12:24:58.369837 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 27 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:58.393076 18374 raft_consensus.cc:2468] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 25.
I20260709 12:24:58.397511 18334 leader_election.cc:304] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:58.398232 17728 leader_election.cc:304] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:58.407478 27259 raft_consensus.cc:493] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:58.409123 17836 raft_consensus.cc:1240] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 28 LEADER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 27. Current term is 28. Ops: [27.15-27.15]
I20260709 12:24:58.409206 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:58.410022 17830 raft_consensus.cc:1240] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 18. Current term is 19. Ops: []
I20260709 12:24:58.410241 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 32 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:58.437829 17759 raft_consensus.cc:1240] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 26. Current term is 27. Ops: []
I20260709 12:24:58.414634 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:24:58.431778 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 66626.7us
I20260709 12:24:58.392115 18333 leader_election.cc:304] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:24:58.344507 26887 consensus_queue.cc:237] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 22.15, Last appended by leader: 15, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:58.426882 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:58.429188 26844 leader_election.cc:290] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:58.436054 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:58.438624 17689 raft_consensus.cc:2468] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 25.
W20260709 12:24:58.449579 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 51433.9us
I20260709 12:24:58.450416 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:58.420928 27238 consensus_queue.cc:1048] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
W20260709 12:24:58.439514 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 47532.8us
I20260709 12:24:58.441963 18361 raft_consensus.cc:2468] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 28.
I20260709 12:24:58.448931 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 27 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:58.455955 17803 leader_election.cc:304] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:58.390204 27328 raft_consensus.cc:740] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 26 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:24:58.391765 27321 raft_consensus.cc:2749] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:24:58.437444 26818 raft_consensus.cc:2749] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260709 12:24:58.457056 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:58.469499 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:58.451133 17682 raft_consensus.cc:3060] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:58.432950 27197 raft_consensus.cc:493] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:58.464526 17755 raft_consensus.cc:1275] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 27 index: 15. Preceding OpId from leader: term: 28 index: 15. (term mismatch)
I20260709 12:24:58.433367 27205 raft_consensus.cc:493] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:58.457023 17803 leader_election.cc:304] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:24:58.469025 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:58.385919 27353 consensus_queue.cc:1048] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.069s
I20260709 12:24:58.393198 27369 raft_consensus.cc:3060] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:58.473489 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:58.474303 17755 pending_rounds.cc:85] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb: Aborting all ops after (but not including) 14
I20260709 12:24:58.448594 26962 raft_consensus.cc:2804] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:24:58.510138 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:58.449527 26957 leader_election.cc:290] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 27 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:58.467557 17802 leader_election.cc:304] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:58.511495 17755 pending_rounds.cc:107] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb: Aborting uncommitted NO_OP operation due to leader change: 27.15
I20260709 12:24:58.463588 26910 raft_consensus.cc:2749] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:24:58.431965 27128 raft_consensus.cc:515] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:58.532382 17757 raft_consensus.cc:2468] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 24.
W20260709 12:24:58.521035 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 53559.3us
W20260709 12:24:58.532974 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 93279.4us
W20260709 12:24:58.510083 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 53122us
I20260709 12:24:58.512575 27363 raft_consensus.cc:2804] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:24:58.521653 17755 raft_consensus.cc:2889] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:58.469094 27343 raft_consensus.cc:493] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:58.531550 18371 raft_consensus.cc:3060] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:58.512245 16501 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260709 12:24:58.531673 17682 raft_consensus.cc:2468] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 23.
I20260709 12:24:58.512177 26888 raft_consensus.cc:2804] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:24:58.457041 27093 raft_consensus.cc:2804] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:24:58.453466 27371 consensus_queue.cc:1048] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.080s
I20260709 12:24:58.448617 27259 raft_consensus.cc:515] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:58.520915 26962 raft_consensus.cc:493] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:24:58.583150 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 169649us
I20260709 12:24:58.534348 27111 raft_consensus.cc:2749] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:24:58.510710 27197 raft_consensus.cc:515] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:58.577514 27363 raft_consensus.cc:493] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:58.583724 27093 raft_consensus.cc:493] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
W20260709 12:24:58.608397 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 69904.8us
I20260709 12:24:58.520773 27205 raft_consensus.cc:515] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:58.582522 26888 raft_consensus.cc:493] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:24:58.701320 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 129663us
I20260709 12:24:58.707371 17802 leader_election.cc:304] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:58.725731 17652 leader_election.cc:304] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:24:58.609436 17803 leader_election.cc:304] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
W20260709 12:24:58.743453 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 134124us
I20260709 12:24:58.745663 17802 leader_election.cc:304] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:58.586830 27343 raft_consensus.cc:515] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:58.576853 27328 consensus_queue.cc:260] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 26.17, Last appended by leader: 17, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:58.764214 18371 raft_consensus.cc:2468] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 25.
I20260709 12:24:58.588264 26962 raft_consensus.cc:3060] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:58.629383 27128 leader_election.cc:290] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 27 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:58.772455 27196 raft_consensus.cc:2804] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:24:58.612043 27363 raft_consensus.cc:3060] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:24:58.772919 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:58.774206 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:58.774901 17758 raft_consensus.cc:2393] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 18: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:24:58.783291 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:58.789177 17763 raft_consensus.cc:2468] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 32.
I20260709 12:24:58.680719 27230 raft_consensus.cc:3055] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 28 LEADER]: Stepping down as leader of term 28
W20260709 12:24:58.780072 17727 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:58.708415 27093 raft_consensus.cc:3060] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:24:58.784492 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 31 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:58.792120 17689 raft_consensus.cc:2468] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 28.
I20260709 12:24:58.757113 27008 raft_consensus.cc:493] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:58.759955 26910 raft_consensus.cc:2804] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Leader pre-election won for term 29
W20260709 12:24:58.617581 27375 log.cc:927] Time spent T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.228s	user 0.000s	sys 0.001s
I20260709 12:24:58.629197 27359 leader_election.cc:290] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 26 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:58.775717 27371 raft_consensus.cc:2749] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:24:58.739244 26888 raft_consensus.cc:3060] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:58.619841 27259 leader_election.cc:290] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:58.790813 27385 consensus_queue.cc:1059] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 28.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.043s
I20260709 12:24:58.757725 27111 raft_consensus.cc:2804] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:24:58.772955 27196 raft_consensus.cc:493] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:24:58.804527 27256 consensus_meta.cc:287] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.632s	user 0.006s	sys 0.005s
I20260709 12:24:58.801678 27230 raft_consensus.cc:740] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:24:58.811249 26910 raft_consensus.cc:493] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:58.789964 27267 consensus_queue.cc:1048] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.036s
W20260709 12:24:58.849174 17728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:58.806954 27328 raft_consensus.cc:3060] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:58.854411 27111 raft_consensus.cc:697] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 25 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:24:58.829952 27307 leader_election.cc:290] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:58.857370 26962 raft_consensus.cc:515] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:58.859958 26910 raft_consensus.cc:3060] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:24:58.860935 17833 raft_consensus.cc:2468] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 28.
I20260709 12:24:58.832365 27196 raft_consensus.cc:3060] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Advancing to term 23
W20260709 12:24:58.860508 27383 log.cc:927] Time spent T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.090s	user 0.000s	sys 0.000s
I20260709 12:24:58.813719 27008 raft_consensus.cc:515] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:24:58.877817 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 132229us
I20260709 12:24:58.921833 26962 leader_election.cc:290] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 24 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:24:58.924507 27350 log.cc:927] Time spent T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.980s	user 0.000s	sys 0.005s
I20260709 12:24:58.852846 27256 raft_consensus.cc:515] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:58.928660 27387 log.cc:927] Time spent T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.107s	user 0.000s	sys 0.000s
I20260709 12:24:58.928428 26910 raft_consensus.cc:515] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:58.929365 17832 raft_consensus.cc:2393] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 25: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
W20260709 12:24:58.930181 27377 log.cc:927] Time spent T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.206s	user 0.000s	sys 0.001s
I20260709 12:24:58.930712 26910 leader_election.cc:290] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 29 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:58.931128 17683 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44898 (request call id 1881) took 1263 ms. Trace:
I20260709 12:24:58.931598 17683 rpcz_store.cc:276] 0709 12:24:57.668101 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:57.689918 (+ 21817us) service_pool.cc:224] Handling call
0709 12:24:57.695084 (+  5166us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:57.797212 (+102128us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:57.797267 (+    55us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:57.811073 (+ 13806us) log.cc:844] Serialized 26 byte log entry
0709 12:24:58.348266 (+537193us) spinlock_profiling.cc:243] Waited 5.78 ms on lock 0x7b18004a91a0. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b47515e07 00007f7b503709a9 00007f7b4753269b 00007f7b4753093f 00007f7b47532c02 00007f7b4a1ba20b 00007f7b4a1bfd23 00007f7b4a1c00d1 00007f7b4a5dba0f
0709 12:24:58.785111 (+436845us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:24:58.785152 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:58.785216 (+    64us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:58.930837 (+145621us) raft_consensus.cc:1685] finished
0709 12:24:58.930860 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:58.931073 (+   213us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":16969,"raft.queue_time_us":130489,"raft.run_cpu_time_us":171,"raft.run_wall_time_us":23638,"spinlock_wait_cycles":13285248,"thread_start_us":436662,"threads_started":1,"wal-append.queue_time_us":973558}
I20260709 12:24:58.934846 27230 consensus_queue.cc:260] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 28.18, Last appended by leader: 18, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:58.936299 27338 raft_consensus.cc:493] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:58.936846 27338 raft_consensus.cc:3060] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:58.937175 17762 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38786 (request call id 4065) took 1746 ms. Trace:
I20260709 12:24:58.938274 17654 leader_election.cc:304] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:58.937564 17762 rpcz_store.cc:276] 0709 12:24:57.191131 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:57.191295 (+   164us) service_pool.cc:224] Handling call
0709 12:24:57.202373 (+ 11078us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:24:57.640984 (+438611us) spinlock_profiling.cc:243] Waited 2.3 ms on lock 0x7b18003cbb00. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b4a1c84bc 00007f7b4751648b 00007f7b503709a9 00007f7b4753269b 00007f7b4753093f 00007f7b47530455 00007f7b4a618060 00007f7b4a618bfb 00007f7b4a62364b
0709 12:24:57.641120 (+   136us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:57.641147 (+    27us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:24:57.641705 (+   558us) log.cc:844] Serialized 52 byte log entry
0709 12:24:57.825788 (+184083us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:24:57.825821 (+    33us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:57.825883 (+    62us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:58.923758 (+1097875us) spinlock_profiling.cc:243] Waited 15.3 ms on lock 0x7b4400022b4c. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b47459c85 00007f7b4745a6a2 00007f7b4a1cc1db 00007f7b4a1c7aaf 00007f7b4a1bb86e 00007f7b4a1babb6 00007f7b4a1ba7d8 00007f7b4a1c4221 00007f7b4a1c41d9 00007f7b4a1c4169
0709 12:24:58.936914 (+ 13156us) raft_consensus.cc:1685] finished
0709 12:24:58.936936 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:58.937085 (+   149us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5391,"raft.queue_time_us":20648,"raft.run_cpu_time_us":121,"raft.run_wall_time_us":4391,"spinlock_wait_cycles":40562176,"thread_start_us":204967,"threads_started":2,"wal-append.queue_time_us":302516}
I20260709 12:24:58.938848 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 27 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:58.939596 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:58.939826 27373 raft_consensus.cc:2749] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:24:58.939605 17759 raft_consensus.cc:2393] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 27: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
W20260709 12:24:58.941254 18368 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59886 (request call id 2588) took 1289 ms (client timeout 1711 ms). Trace:
W20260709 12:24:58.941735 18368 rpcz_store.cc:269] 0709 12:24:57.652063 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:57.690589 (+ 38526us) service_pool.cc:224] Handling call
0709 12:24:58.941128 (+1250539us) spinlock_profiling.cc:243] Waited 152 ms on lock 0x7b5400570154. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:24:58.941208 (+    80us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":348882176}
I20260709 12:24:58.943926 27230 raft_consensus.cc:3060] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Advancing to term 29
W20260709 12:24:58.945429 27380 log.cc:927] Time spent T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.350s	user 0.001s	sys 0.000s
I20260709 12:24:58.940049 18364 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59886 (request call id 2587) took 1288 ms. Trace:
W20260709 12:24:58.956427 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:58.949222 18364 rpcz_store.cc:276] 0709 12:24:57.651755 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:57.692741 (+ 40986us) service_pool.cc:224] Handling call
0709 12:24:58.939861 (+1247120us) spinlock_profiling.cc:243] Waited 4.24 ms on lock 0x7b540047ca54. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:24:58.939994 (+   133us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":9757056}
I20260709 12:24:58.976567 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:58.977304 17833 raft_consensus.cc:2468] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 18.
I20260709 12:24:58.978425 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 27 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:58.980140 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:58.988943 17688 raft_consensus.cc:1240] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 27 LEADER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 26. Current term is 27. Ops: [26.17-26.17]
W20260709 12:24:59.003186 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 52065.1us
W20260709 12:24:59.001866 27329 consensus_meta.cc:287] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 1.156s	user 0.006s	sys 0.007s
I20260709 12:24:58.982427 26910 consensus_queue.cc:1048] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.039s
W20260709 12:24:59.040187 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 297298us
W20260709 12:24:59.048218 17652 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:59.021580 27329 raft_consensus.cc:515] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:59.054649 17652 leader_election.cc:304] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:24:59.044229 27189 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.662s	user 0.004s	sys 0.013s
I20260709 12:24:59.015403 27111 consensus_queue.cc:237] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 24.15, Last appended by leader: 15, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:59.064970 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 32 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:59.069427 17833 raft_consensus.cc:2393] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 32: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:24:59.071408 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:59.076390 17838 raft_consensus.cc:2393] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 29: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:24:59.052107 27370 raft_consensus.cc:493] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:59.059396 27259 raft_consensus.cc:493] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:59.082700 16501 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
W20260709 12:24:59.068235 27189 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.686s	user 0.005s	sys 0.013s
I20260709 12:24:59.062894 27307 raft_consensus.cc:2749] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
W20260709 12:24:59.112882 27245 consensus_meta.cc:287] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.729s	user 0.009s	sys 0.001s
W20260709 12:24:59.115926 27369 consensus_meta.cc:287] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.570s	user 0.009s	sys 0.002s
I20260709 12:24:59.116624 27369 raft_consensus.cc:515] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:59.119052 27259 raft_consensus.cc:515] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:59.121628 27259 leader_election.cc:290] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:59.122750 27370 raft_consensus.cc:515] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:59.125061 27370 leader_election.cc:290] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:59.062886 27256 leader_election.cc:290] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 25 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:59.130313 17755 raft_consensus.cc:1275] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 26 index: 18. Preceding OpId from leader: term: 29 index: 19. (index mismatch)
I20260709 12:24:59.093477 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:24:59.098436 27393 log.cc:927] Time spent T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.056s	user 0.000s	sys 0.000s
I20260709 12:24:59.100049 17764 raft_consensus.cc:1240] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 32 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 31. Current term is 32. Ops: [31.24-31.24]
I20260709 12:24:59.092876 17652 leader_election.cc:304] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
W20260709 12:24:59.129019 27341 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.186s	user 0.023s	sys 0.028s
I20260709 12:24:59.131630 17759 raft_consensus.cc:2468] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 27.
I20260709 12:24:59.113437 27245 raft_consensus.cc:515] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:59.129900 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 125074us
I20260709 12:24:59.137328 17652 leader_election.cc:304] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:24:59.139019 27341 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.196s	user 0.023s	sys 0.028s
I20260709 12:24:59.154034 17680 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37396 (request call id 3397) took 1480 ms. Trace:
I20260709 12:24:59.137522 27307 raft_consensus.cc:2804] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:24:59.158115 27369 leader_election.cc:290] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 26 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:59.181384 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 26 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:59.181584 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:59.182034 18364 raft_consensus.cc:3060] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:59.182186 18374 raft_consensus.cc:3060] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:24:59.158622 27382 raft_consensus.cc:2749] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 32 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:24:59.158737 27035 raft_consensus.cc:3055] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:24:59.152985 27008 leader_election.cc:290] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:59.184966 27035 raft_consensus.cc:740] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:24:59.185539 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:59.185885 27035 consensus_queue.cc:260] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 25.19, Last appended by leader: 19, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:59.160095 26962 raft_consensus.cc:493] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:59.187247 27035 raft_consensus.cc:3060] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:59.187325 26962 raft_consensus.cc:515] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:59.190100 26962 leader_election.cc:290] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:24:59.162814 27365 log.cc:927] Time spent T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.887s	user 0.001s	sys 0.000s
I20260709 12:24:59.163795 17834 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37994 (request call id 1793) took 1086 ms. Trace:
I20260709 12:24:59.165952 27196 raft_consensus.cc:515] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:59.166985 27154 consensus_meta.cc:287] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.736s	user 0.012s	sys 0.000s
I20260709 12:24:59.154695 17680 rpcz_store.cc:276] 0709 12:24:57.673332 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:57.687562 (+ 14230us) service_pool.cc:224] Handling call
0709 12:24:57.687821 (+   259us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:57.791642 (+103821us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:57.791683 (+    41us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:57.791726 (+    43us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:24:57.791758 (+    32us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:59.153710 (+1361952us) spinlock_profiling.cc:243] Waited 104 ms on lock 0x7b54003462d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:24:59.153768 (+    58us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:59.153935 (+   167us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":238343040}
I20260709 12:24:59.145016 17802 leader_election.cc:304] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
W20260709 12:24:59.178638 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:59.207456 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:59.207562 17727 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:59.209729 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
W20260709 12:24:59.143545 27360 log.cc:927] Time spent T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.931s	user 0.003s	sys 0.000s
I20260709 12:24:59.162367 18334 leader_election.cc:304] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:24:59.210882 18364 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 26.
I20260709 12:24:59.212286 17758 raft_consensus.cc:1275] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 26 index: 17. Preceding OpId from leader: term: 27 index: 17. (term mismatch)
I20260709 12:24:59.212708 17758 pending_rounds.cc:85] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb: Aborting all ops after (but not including) 16
I20260709 12:24:59.212917 17758 pending_rounds.cc:107] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb: Aborting uncommitted NO_OP operation due to leader change: 26.17
W20260709 12:24:59.162235 27316 consensus_meta.cc:287] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.992s	user 0.012s	sys 0.000s
I20260709 12:24:59.211103 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 27 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:59.213130 17758 raft_consensus.cc:2889] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:24:59.214298 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 69404.3us
I20260709 12:24:59.204211 17834 rpcz_store.cc:276] 0709 12:24:58.077135 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:58.077277 (+   142us) service_pool.cc:224] Handling call
0709 12:24:58.077613 (+   336us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:58.159423 (+ 81810us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:58.159455 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:58.172466 (+ 13011us) log.cc:844] Serialized 26 byte log entry
0709 12:24:58.311037 (+138571us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:24:58.311085 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:59.163366 (+852281us) spinlock_profiling.cc:243] Waited 53.8 ms on lock 0x7b54000893d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:24:59.163418 (+    52us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:59.163473 (+    55us) raft_consensus.cc:1685] finished
0709 12:24:59.163487 (+    14us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:59.163732 (+   245us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":27384,"raft.queue_time_us":31687,"raft.run_cpu_time_us":140,"raft.run_wall_time_us":3896,"spinlock_wait_cycles":123760512,"thread_start_us":80619,"threads_started":1,"wal-append.queue_time_us":144384}
I20260709 12:24:59.215600 18368 raft_consensus.cc:1275] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 24 index: 17. Preceding OpId from leader: term: 25 index: 18. (index mismatch)
I20260709 12:24:59.175904 27307 raft_consensus.cc:493] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:24:59.223199 27307 raft_consensus.cc:3060] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:59.212208 27154 raft_consensus.cc:515] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:59.228755 17832 raft_consensus.cc:1240] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 24. Current term is 25. Ops: []
I20260709 12:24:59.232928 27373 raft_consensus.cc:493] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:59.228048 27154 leader_election.cc:290] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 25 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:59.214860 27316 raft_consensus.cc:515] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:59.233487 27373 raft_consensus.cc:515] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:59.234460 17684 raft_consensus.cc:1240] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 22. Current term is 23. Ops: []
I20260709 12:24:59.236258 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 23 candidate_status { last_received { term: 21 index: 20 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:59.236917 18367 raft_consensus.cc:2468] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 22.
I20260709 12:24:59.237306 27339 raft_consensus.cc:2804] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:24:59.236215 27316 leader_election.cc:290] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 23 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:59.237900 27339 raft_consensus.cc:697] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 23 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:24:59.215062 26962 raft_consensus.cc:493] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:59.238649 26962 raft_consensus.cc:515] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:59.240034 26887 raft_consensus.cc:493] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:59.240840 26962 leader_election.cc:290] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:59.240604 26887 raft_consensus.cc:515] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:59.218508 27329 leader_election.cc:290] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 22 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:59.218901 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 27 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:59.201187 18374 raft_consensus.cc:2468] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 29.
I20260709 12:24:59.245081 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 31 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:59.247872 26887 leader_election.cc:290] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:59.255558 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:59.256392 17763 raft_consensus.cc:2393] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 27: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:24:59.257319 27245 leader_election.cc:290] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 26 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:59.259938 27339 consensus_queue.cc:237] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 21.15, Last appended by leader: 15, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:59.208385 27093 raft_consensus.cc:515] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:59.259325 26888 raft_consensus.cc:515] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:59.264745 17831 raft_consensus.cc:1240] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 25. Current term is 26. Ops: [25.18-25.18]
I20260709 12:24:59.266335 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:59.266511 17760 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36572 (request call id 1706) took 1717 ms. Trace:
I20260709 12:24:59.266620 27093 leader_election.cc:290] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 28 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:59.267021 17760 rpcz_store.cc:276] 0709 12:24:57.549626 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:57.972891 (+423265us) service_pool.cc:224] Handling call
0709 12:24:58.002552 (+ 29661us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:58.037051 (+ 34499us) raft_consensus.cc:1534] Early marking committed up to index 17
0709 12:24:58.145116 (+108065us) log.cc:844] Serialized 14 byte log entry
0709 12:24:58.303432 (+158316us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:58.303509 (+    77us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:24:58.303573 (+    64us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:59.266302 (+962729us) spinlock_profiling.cc:243] Waited 2.26 ms on lock 0x7b54001b8854. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:24:59.266346 (+    44us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:59.266455 (+   109us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":31492,"prepare.queue_time_us":126339,"prepare.run_cpu_time_us":62,"prepare.run_wall_time_us":3435,"raft.queue_time_us":28164,"raft.run_cpu_time_us":119,"raft.run_wall_time_us":4089,"spinlock_wait_cycles":5191936,"thread_start_us":170404,"threads_started":2,"wal-append.queue_time_us":156816}
I20260709 12:24:59.248332 27370 raft_consensus.cc:2804] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 32 FOLLOWER]: Leader election won for term 32
I20260709 12:24:59.266623 26888 leader_election.cc:290] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 26 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:59.184128 27363 raft_consensus.cc:515] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:59.276497 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:24:59.279868 17728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:59.279479 18363 raft_consensus.cc:2468] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 26.
W20260709 12:24:59.284864 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 122595us
I20260709 12:24:59.278664 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:59.287248 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:59.287683 17838 raft_consensus.cc:2393] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 26: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:24:59.287987 17836 raft_consensus.cc:2468] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 26.
I20260709 12:24:59.292172 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 23 candidate_status { last_received { term: 21 index: 20 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:59.290161 17803 leader_election.cc:304] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:24:59.292815 17829 raft_consensus.cc:3060] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:59.295434 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:59.274147 27370 raft_consensus.cc:697] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 32 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:24:59.296051 17763 raft_consensus.cc:3060] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:59.296491 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 25 candidate_status { last_received { term: 24 index: 20 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
W20260709 12:24:59.289218 27390 log.cc:927] Time spent T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.341s	user 0.004s	sys 0.000s
I20260709 12:24:59.304569 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:59.307943 27008 raft_consensus.cc:2804] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:24:59.308911 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:59.312633 17760 raft_consensus.cc:3060] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:59.313859 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:59.314668 18363 raft_consensus.cc:3060] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:59.316032 17763 raft_consensus.cc:2468] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 23.
I20260709 12:24:59.316967 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 28 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:59.294869 27196 leader_election.cc:290] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 23 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:24:59.325405 17757 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36572 (request call id 1701) took 1792 ms (client timeout 1763 ms). Trace:
W20260709 12:24:59.325872 17757 rpcz_store.cc:269] 0709 12:24:57.547972 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:57.576648 (+ 28676us) service_pool.cc:224] Handling call
0709 12:24:59.320232 (+1743584us) spinlock_profiling.cc:243] Waited 813 ms on lock 0x7b54003e58d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:24:59.320350 (+   118us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1869037952}
I20260709 12:24:59.328860 18363 raft_consensus.cc:2468] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 26.
I20260709 12:24:59.317459 27316 raft_consensus.cc:493] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:59.332894 17829 raft_consensus.cc:2468] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 23.
I20260709 12:24:59.334326 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 28 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:59.318437 27008 raft_consensus.cc:697] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 29 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:24:59.333827 27373 leader_election.cc:290] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:59.337280 27062 leader_election.cc:290] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:59.335731 17652 leader_election.cc:304] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:24:59.360203 27358 raft_consensus.cc:3055] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 31 LEADER]: Stepping down as leader of term 31
I20260709 12:24:59.365756 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:59.377597 17756 raft_consensus.cc:2393] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 25: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:24:59.366473 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:24:59.378697 17730 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:59.367947 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 26 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:59.370824 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:59.374169 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:59.375108 17755 raft_consensus.cc:1275] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 22 index: 15. Preceding OpId from leader: term: 24 index: 16. (index mismatch)
I20260709 12:24:59.376209 27343 leader_election.cc:290] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:59.379169 17652 leader_election.cc:304] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:59.387387 27008 consensus_queue.cc:237] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 27.16, Last appended by leader: 16, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:59.390118 17728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:59.396246 17652 leader_election.cc:304] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:59.399132 27363 leader_election.cc:290] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 33 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:59.400156 27397 raft_consensus.cc:2804] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:24:59.400588 27397 raft_consensus.cc:493] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:24:59.400946 27397 raft_consensus.cc:3060] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:24:59.405591 27196 consensus_queue.cc:1059] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 30.23, Next index: 24, Last known committed idx: 23, Time since last communication: 0.000s
I20260709 12:24:59.398283 27245 consensus_queue.cc:1048] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.009s
I20260709 12:24:59.377172 27358 raft_consensus.cc:740] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 31 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:24:59.409262 27358 consensus_queue.cc:260] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 31.24, Last appended by leader: 24, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:59.410657 27358 raft_consensus.cc:3060] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:24:59.383289 27373 raft_consensus.cc:2804] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:24:59.412220 27373 raft_consensus.cc:697] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 23 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:24:59.413089 27373 consensus_queue.cc:237] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 21.20, Last appended by leader: 20, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:59.343622 27370 consensus_queue.cc:237] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 30.18, Last appended by leader: 18, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:59.416599 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:59.417817 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:59.362632 27316 raft_consensus.cc:515] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:59.418550 18365 raft_consensus.cc:2393] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 29: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:24:59.419361 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:59.419992 27316 leader_election.cc:290] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:59.420696 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:59.420904 27415 consensus_queue.cc:1059] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 22.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20260709 12:24:59.421679 18368 raft_consensus.cc:2393] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 26: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
W20260709 12:24:59.421829 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:59.422277 27413 raft_consensus.cc:3055] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:24:59.422740 27413 raft_consensus.cc:740] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:24:59.422619 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 23 candidate_status { last_received { term: 21 index: 20 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:59.423781 27413 consensus_queue.cc:260] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 22.16, Last appended by leader: 16, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:59.424705 18363 raft_consensus.cc:3060] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:59.425089 27413 raft_consensus.cc:3060] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:59.426872 18366 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59902 (request call id 2239) took 1067 ms. Trace:
I20260709 12:24:59.396510 27392 raft_consensus.cc:2749] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:24:59.427248 18366 rpcz_store.cc:276] 0709 12:24:58.359118 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:58.453692 (+ 94574us) service_pool.cc:224] Handling call
0709 12:24:58.947005 (+493313us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:58.969304 (+ 22299us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:58.969338 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:58.969945 (+   607us) log.cc:844] Serialized 26 byte log entry
0709 12:24:59.146137 (+176192us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:24:59.223617 (+ 77480us) log.cc:844] Serialized 14 byte log entry
0709 12:24:59.317490 (+ 93873us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:59.317569 (+    79us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:59.317610 (+    41us) raft_consensus.cc:1685] finished
0709 12:24:59.317621 (+    11us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:59.426702 (+109081us) spinlock_profiling.cc:243] Waited 474 ms on lock 0x7b540047ca50. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a61f8ad 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:24:59.426817 (+   115us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":27644,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":41,"raft.queue_time_us":7312,"raft.run_cpu_time_us":231,"raft.run_wall_time_us":18907,"spinlock_wait_cycles":1089739136,"thread_start_us":275233,"threads_started":3,"wal-append.queue_time_us":166400}
I20260709 12:24:59.432605 27270 leader_election.cc:290] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:59.437209 17727 leader_election.cc:304] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:59.437173 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:59.438087 17684 raft_consensus.cc:3060] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:24:59.437170 27369 consensus_queue.cc:1059] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 26.19, Next index: 20, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:24:59.399461 17652 leader_election.cc:304] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:59.445514 17654 leader_election.cc:304] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:24:59.447414 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:59.441318 27189 raft_consensus.cc:2749] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:24:59.441115 27328 raft_consensus.cc:493] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:59.451161 27328 raft_consensus.cc:515] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:59.453672 27328 leader_election.cc:290] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:59.457223 27154 raft_consensus.cc:3055] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:24:59.457617 27154 raft_consensus.cc:740] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 26 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:24:59.458485 27154 consensus_queue.cc:260] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 26.19, Last appended by leader: 19, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:59.458499 17684 raft_consensus.cc:2468] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 29.
I20260709 12:24:59.460935 27205 leader_election.cc:290] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:59.461555 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:59.421854 27338 raft_consensus.cc:515] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:59.463451 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 26 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:59.464447 17683 raft_consensus.cc:2393] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 26: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:24:59.464665 27338 leader_election.cc:290] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 26 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:59.465039 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:59.466248 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:59.466871 17687 raft_consensus.cc:2468] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 26.
I20260709 12:24:59.469600 27093 raft_consensus.cc:493] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:59.469992 27189 raft_consensus.cc:493] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:59.470121 27093 raft_consensus.cc:515] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:59.470484 27189 raft_consensus.cc:515] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:24:59.473930 17730 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:59.474259 27189 leader_election.cc:290] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:59.158046 16501 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
W20260709 12:24:59.477340 27379 log.cc:927] Time spent T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.891s	user 0.004s	sys 0.000s
I20260709 12:24:59.462973 18363 raft_consensus.cc:2468] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 23.
I20260709 12:24:59.478752 17835 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35040 (request call id 3613) took 1233 ms. Trace:
I20260709 12:24:59.479156 17835 rpcz_store.cc:276] 0709 12:24:58.245399 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:58.247915 (+  2516us) service_pool.cc:224] Handling call
0709 12:24:58.274210 (+ 26295us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:58.284589 (+ 10379us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:58.284635 (+    46us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:58.301733 (+ 17098us) log.cc:844] Serialized 26 byte log entry
0709 12:24:58.383955 (+ 82222us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:24:58.384303 (+   348us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:58.384377 (+    74us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:59.476755 (+1092378us) spinlock_profiling.cc:243] Waited 6.89 ms on lock 0x7b4400022b4c. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b47459c85 00007f7b4745a6a2 00007f7b4a1cc1db 00007f7b4a1c7aaf 00007f7b4a1bb86e 00007f7b4a1babb6 00007f7b4a1ba7d8 00007f7b4a1c4221 00007f7b4a1c41d9 00007f7b4a1c4169
0709 12:24:59.478437 (+  1682us) raft_consensus.cc:1685] finished
0709 12:24:59.478459 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:59.478693 (+   234us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":9477,"raft.queue_time_us":126,"raft.run_cpu_time_us":178,"raft.run_wall_time_us":10961,"spinlock_wait_cycles":15844096,"thread_start_us":65146,"threads_started":1,"wal-append.queue_time_us":155170}
I20260709 12:24:59.481671 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:59.482229 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:24:59.483220 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 44275.9us
I20260709 12:24:59.483256 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:24:59.485842 17727 leader_election.cc:304] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:59.486735 27189 raft_consensus.cc:2749] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:24:59.493811 26924 raft_consensus.cc:3060] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Advancing to term 29
W20260709 12:24:59.495258 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 49807.6us
I20260709 12:24:59.463910 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 25 candidate_status { last_received { term: 24 index: 20 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:59.487694 27197 leader_election.cc:290] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:59.488341 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:59.506618 18374 raft_consensus.cc:2468] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 26.
I20260709 12:24:59.489355 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 32 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:24:59.491113 27093 leader_election.cc:290] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:59.487272 17727 leader_election.cc:304] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:24:59.508148 17802 leader_election.cc:304] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:24:59.497385 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:24:59.504004 17760 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 26.
I20260709 12:24:59.459762 27154 raft_consensus.cc:3060] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:24:59.537282 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 139713us
W20260709 12:24:59.537810 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 138415us
W20260709 12:24:59.548875 17730 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:59.539445 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:24:59.556921 18363 raft_consensus.cc:1240] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 25. Current term is 26. Ops: []
W20260709 12:24:59.567844 18371 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59886 (request call id 2591) took 1776 ms (client timeout 2071 ms). Trace:
W20260709 12:24:59.568444 18371 rpcz_store.cc:269] 0709 12:24:57.791553 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:57.794563 (+  3010us) service_pool.cc:224] Handling call
0709 12:24:59.567640 (+1773077us) spinlock_profiling.cc:243] Waited 687 ms on lock 0x7b5400467b54. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:24:59.567770 (+   130us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1580162304}
I20260709 12:24:59.534327 27189 raft_consensus.cc:2804] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:24:59.533910 27256 raft_consensus.cc:493] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:59.560834 17654 leader_election.cc:304] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:59.536975 26599 raft_consensus.cc:2804] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:24:59.585541 17801 leader_election.cc:304] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:59.552816 27316 raft_consensus.cc:2804] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:24:59.536878 27093 raft_consensus.cc:493] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:59.594367 17652 leader_election.cc:304] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:59.596973 17654 leader_election.cc:304] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:59.591935 17803 leader_election.cc:304] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:24:59.549124 27341 raft_consensus.cc:493] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:59.591006 27189 raft_consensus.cc:493] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:59.583830 27382 raft_consensus.cc:493] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:24:59.595314 26599 raft_consensus.cc:493] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:59.570626 27196 consensus_queue.cc:1059] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 23.18, Next index: 19, Last known committed idx: 17, Time since last communication: 0.048s
I20260709 12:24:59.540580 27205 consensus_queue.cc:1048] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.062s
I20260709 12:24:59.610674 17802 leader_election.cc:304] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:24:59.561551 26887 consensus_queue.cc:1048] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.046s
I20260709 12:24:59.603804 27359 raft_consensus.cc:493] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:59.618119 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 33 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:59.603571 27363 raft_consensus.cc:2749] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 12:24:59.601382 27327 raft_consensus.cc:493] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:59.613004 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:59.622524 26599 raft_consensus.cc:3060] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:59.590014 27406 consensus_queue.cc:1048] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 13, Time since last communication: 0.030s
I20260709 12:24:59.622850 27062 raft_consensus.cc:2804] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:24:59.636152 17835 raft_consensus.cc:3060] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:24:59.612795 27189 raft_consensus.cc:3060] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:59.612733 27385 raft_consensus.cc:2804] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Leader pre-election won for term 24
W20260709 12:24:59.644340 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 47446.3us
I20260709 12:24:59.601824 27256 raft_consensus.cc:515] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:59.611562 27395 raft_consensus.cc:2749] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:24:59.602317 27245 consensus_queue.cc:1059] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 24.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.029s
I20260709 12:24:59.639565 27321 raft_consensus.cc:493] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:24:59.649282 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:59.625198 26789 raft_consensus.cc:2749] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:24:59.646163 27267 raft_consensus.cc:2749] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 12:24:59.606397 27093 raft_consensus.cc:515] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:59.680893 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 33 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:59.634874 27338 raft_consensus.cc:493] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:59.674957 17654 leader_election.cc:304] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:59.703843 17654 leader_election.cc:304] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:24:59.701233 17758 raft_consensus.cc:3060] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Advancing to term 33
I20260709 12:24:59.622263 27392 consensus_queue.cc:1059] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 24.20, Next index: 21, Last known committed idx: 19, Time since last communication: 0.048s
I20260709 12:24:59.633394 27266 raft_consensus.cc:493] T a5732e71b9d54f8786b8c511d69943bf P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:24:59.608623 27341 raft_consensus.cc:515] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:59.634151 18333 leader_election.cc:304] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:59.715552 17763 raft_consensus.cc:1240] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 22. Current term is 23. Ops: []
I20260709 12:24:59.679073 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:24:59.666240 27197 raft_consensus.cc:2749] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260709 12:24:59.669493 27259 raft_consensus.cc:493] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:59.683485 27062 raft_consensus.cc:697] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 26 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:24:59.632117 27382 raft_consensus.cc:515] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:59.643133 27359 raft_consensus.cc:515] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:59.711081 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:59.729059 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 24 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:59.716979 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:59.730378 17763 raft_consensus.cc:3060] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Advancing to term 23
W20260709 12:24:59.713194 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 127733us
I20260709 12:24:59.665474 27307 raft_consensus.cc:515] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:59.722689 17835 raft_consensus.cc:2468] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 33.
I20260709 12:24:59.725459 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 29 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:59.725459 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:24:59.727535 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 93468us
I20260709 12:24:59.716506 17831 raft_consensus.cc:2468] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 18.
I20260709 12:24:59.732652 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:59.732965 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 24 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:59.668486 27327 raft_consensus.cc:515] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:59.765182 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:59.769994 17829 raft_consensus.cc:2468] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 19.
I20260709 12:24:59.771317 17801 leader_election.cc:304] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:59.776122 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:24:59.725803 27351 log.cc:927] Time spent T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 1.783s	user 0.000s	sys 0.005s
I20260709 12:24:59.706079 27316 raft_consensus.cc:697] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 23 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:24:59.769280 17755 raft_consensus.cc:2376] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 24. Re-sending same reply.
W20260709 12:24:59.780555 27418 log.cc:927] Time spent T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.074s	user 0.000s	sys 0.000s
W20260709 12:24:59.781786 27420 log.cc:927] Time spent T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.086s	user 0.000s	sys 0.000s
I20260709 12:24:59.781920 17755 raft_consensus.cc:1275] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 25 index: 18. Preceding OpId from leader: term: 28 index: 20. (index mismatch)
W20260709 12:24:59.783443 27421 log.cc:927] Time spent T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.054s	user 0.000s	sys 0.000s
I20260709 12:24:59.784324 27062 consensus_queue.cc:237] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 24.17, Last appended by leader: 17, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:59.780460 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 33 candidate_status { last_received { term: 31 index: 24 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:59.799240 17764 raft_consensus.cc:2468] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 32.
I20260709 12:24:59.800385 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 28 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:59.801055 17761 raft_consensus.cc:2468] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 27.
I20260709 12:24:59.805785 17763 raft_consensus.cc:2468] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 23.
I20260709 12:24:59.731330 27240 raft_consensus.cc:2749] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:24:59.805845 17758 raft_consensus.cc:2468] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 33.
I20260709 12:24:59.700642 27321 raft_consensus.cc:515] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:59.722147 27266 raft_consensus.cc:515] T a5732e71b9d54f8786b8c511d69943bf P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:24:59.817611 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 46727.1us
I20260709 12:24:59.711040 27338 raft_consensus.cc:515] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:59.777442 27197 raft_consensus.cc:2749] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:24:59.815958 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:24:59.814673 17801 leader_election.cc:304] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:24:59.821456 17761 raft_consensus.cc:3060] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:59.737005 27245 consensus_queue.cc:1048] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.080s
I20260709 12:24:59.729579 27259 raft_consensus.cc:515] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:59.862519 27093 leader_election.cc:290] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:59.862794 17681 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37396 (request call id 3394) took 2190 ms. Trace:
I20260709 12:24:59.825104 27431 raft_consensus.cc:493] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:24:59.798094 27406 raft_consensus.cc:493] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:59.867690 18364 raft_consensus.cc:1275] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 28 index: 19. Preceding OpId from leader: term: 29 index: 19. (term mismatch)
W20260709 12:24:59.834442 27426 log.cc:927] Time spent T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.066s	user 0.000s	sys 0.000s
I20260709 12:24:59.887733 18364 pending_rounds.cc:85] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da: Aborting all ops after (but not including) 18
I20260709 12:24:59.813920 27111 raft_consensus.cc:2749] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:24:59.814908 27196 raft_consensus.cc:2804] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:24:59.889534 18364 pending_rounds.cc:107] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da: Aborting uncommitted NO_OP operation due to leader change: 28.19
I20260709 12:24:59.890343 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:24:59.886366 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 72679.1us
W20260709 12:24:59.889858 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 186094us
I20260709 12:24:59.886997 17761 raft_consensus.cc:2468] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 26.
I20260709 12:24:59.866466 26599 raft_consensus.cc:515] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:24:59.926112 17801 leader_election.cc:304] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:24:59.832026 17728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:59.929855 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:24:59.885689 17730 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:59.930935 18364 raft_consensus.cc:2889] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:59.832798 27429 raft_consensus.cc:493] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:59.951516 17654 leader_election.cc:304] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:59.866073 27256 leader_election.cc:290] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:24:59.880069 26888 raft_consensus.cc:2749] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:24:59.953217 17834 raft_consensus.cc:2393] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 27: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:24:59.883818 17681 rpcz_store.cc:276] 0709 12:24:57.672336 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:57.672491 (+   155us) service_pool.cc:224] Handling call
0709 12:24:57.680053 (+  7562us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:57.680467 (+   414us) raft_consensus.cc:1534] Early marking committed up to index 16
0709 12:24:57.680496 (+    29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:57.680980 (+   484us) log.cc:844] Serialized 26 byte log entry
0709 12:24:57.792095 (+111115us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:24:57.792136 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:57.792203 (+    67us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:59.566844 (+1774641us) spinlock_profiling.cc:243] Waited 7.02 ms on lock 0x7b4400022b4c. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b47459c85 00007f7b4745a6a2 00007f7b4a1cc1db 00007f7b4a1c7aaf 00007f7b4a1bb86e 00007f7b4a1babb6 00007f7b4a1ba7d8 00007f7b4a1c4221 00007f7b4a1c41d9 00007f7b4a1c4169
0709 12:24:59.862445 (+295601us) raft_consensus.cc:1685] finished
0709 12:24:59.862469 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:59.862710 (+   241us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":21660,"raft.queue_time_us":3343,"raft.run_cpu_time_us":141,"raft.run_wall_time_us":9863,"spinlock_wait_cycles":16152960,"thread_start_us":110723,"threads_started":1,"wal-append.queue_time_us":195213}
I20260709 12:24:59.953647 18367 raft_consensus.cc:2393] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 24: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:24:59.876370 27316 consensus_queue.cc:237] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 22.16, Last appended by leader: 16, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:24:59.864804 27338 leader_election.cc:290] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:59.832523 27423 raft_consensus.cc:493] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:24:59.881000 27354 raft_consensus.cc:493] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:24:59.877923 27359 leader_election.cc:290] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:59.959614 17654 leader_election.cc:304] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:24:59.990761 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:59.888934 27107 consensus_queue.cc:1059] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.040s
I20260709 12:24:59.867913 27327 leader_election.cc:290] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:59.880574 27259 leader_election.cc:290] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:24:59.863842 27197 consensus_queue.cc:1048] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.092s
I20260709 12:24:59.882761 27413 consensus_queue.cc:1048] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.096s
I20260709 12:24:59.889360 27324 consensus_queue.cc:1048] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.104s
I20260709 12:24:59.954959 27196 raft_consensus.cc:697] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 26 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:25:00.000125 26599 leader_election.cc:290] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 27 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:25:00.003173 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:25:00.021023 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:00.006526 17688 raft_consensus.cc:1275] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 26 index: 19. Preceding OpId from leader: term: 28 index: 20. (index mismatch)
W20260709 12:24:59.979043 27397 consensus_meta.cc:287] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.578s	user 0.012s	sys 0.000s
I20260709 12:24:59.930040 27369 raft_consensus.cc:2749] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 12:24:59.929946 27437 raft_consensus.cc:493] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:00.017910 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 24 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:24:59.980368 27415 raft_consensus.cc:493] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:00.021281 17686 raft_consensus.cc:3060] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:00.003938 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 28 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:25:00.010704 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 396676us
I20260709 12:24:59.892325 27406 raft_consensus.cc:515] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:00.022406 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 33 candidate_status { last_received { term: 31 index: 24 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:25:00.023898 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 235557us
I20260709 12:25:00.025695 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 22 candidate_status { last_received { term: 21 index: 20 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:25:00.026666 17681 raft_consensus.cc:1275] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 28 index: 18. Preceding OpId from leader: term: 29 index: 19. (index mismatch)
W20260709 12:25:00.012125 27435 log.cc:927] Time spent T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.073s	user 0.000s	sys 0.000s
I20260709 12:25:00.038582 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:00.041214 17682 raft_consensus.cc:1240] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 26. Current term is 27. Ops: []
I20260709 12:25:00.042080 17652 leader_election.cc:304] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:25:00.042769 17830 raft_consensus.cc:2393] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 19: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:24:59.893085 27431 raft_consensus.cc:515] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:00.021446 27259 raft_consensus.cc:493] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:25:00.046558 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:25:00.048403 17689 raft_consensus.cc:3060] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Advancing to term 26
W20260709 12:25:00.039913 27373 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.541s	user 0.008s	sys 0.028s
I20260709 12:25:00.030865 27230 raft_consensus.cc:2749] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:25:00.058048 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 22 candidate_status { last_received { term: 21 index: 20 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:00.072376 17835 raft_consensus.cc:2368] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 22. Current term is 23.
I20260709 12:25:00.060878 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:00.060328 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 28 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:24:59.987919 27429 raft_consensus.cc:515] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:00.077282 17760 raft_consensus.cc:1240] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 27. Current term is 28. Ops: []
I20260709 12:25:00.058382 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:00.079207 17728 leader_election.cc:304] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:00.075408 17764 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 27.
I20260709 12:25:00.081472 18363 raft_consensus.cc:1240] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 25. Current term is 26. Ops: [25.18-25.18]
I20260709 12:25:00.082383 17682 raft_consensus.cc:1275] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 21 index: 15. Preceding OpId from leader: term: 23 index: 16. (index mismatch)
I20260709 12:25:00.078094 17830 raft_consensus.cc:2468] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 23.
I20260709 12:25:00.084250 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:00.086463 17760 raft_consensus.cc:1275] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 32 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 30 index: 18. Preceding OpId from leader: term: 32 index: 19. (index mismatch)
I20260709 12:25:00.041054 27369 raft_consensus.cc:493] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:25:00.046028 27341 leader_election.cc:290] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:24:59.999199 27423 raft_consensus.cc:515] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:00.016429 27354 raft_consensus.cc:515] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:00.089699 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:25:00.069721 27373 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.628s	user 0.009s	sys 0.028s
I20260709 12:25:00.101552 17681 raft_consensus.cc:2393] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 24: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:25:00.057767 27436 raft_consensus.cc:493] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:25:00.076632 27382 leader_election.cc:290] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:00.039240 27415 raft_consensus.cc:515] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:00.037348 27437 raft_consensus.cc:515] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:00.091667 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 289361us
I20260709 12:25:00.095495 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 32 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:00.068480 27259 raft_consensus.cc:515] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:00.027566 27397 raft_consensus.cc:515] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:00.129613 17689 raft_consensus.cc:2468] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 26.
I20260709 12:25:00.116907 27369 raft_consensus.cc:515] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:00.117973 27436 raft_consensus.cc:515] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:00.131376 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 29 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:25:00.133948 27154 consensus_meta.cc:287] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.594s	user 0.012s	sys 0.000s
I20260709 12:25:00.135915 27429 leader_election.cc:290] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:00.136246 27430 raft_consensus.cc:493] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:25:00.137251 27423 leader_election.cc:290] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:25:00.138195 27414 log.cc:927] Time spent T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.754s	user 0.004s	sys 0.000s
I20260709 12:25:00.138670 27431 leader_election.cc:290] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:00.139802 27443 consensus_queue.cc:1059] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 24.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:25:00.251986 17687 raft_consensus.cc:1240] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 27. Current term is 28. Ops: []
I20260709 12:25:00.141203 27446 raft_consensus.cc:2749] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260709 12:25:00.142771 17686 raft_consensus.cc:2468] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 26.
I20260709 12:25:00.153163 17803 leader_election.cc:304] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:25:00.124333 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 30 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:00.193005 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 32 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:00.118754 27433 raft_consensus.cc:2749] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:25:00.121869 27382 raft_consensus.cc:2804] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 33 FOLLOWER]: Leader election won for term 33
W20260709 12:25:00.255930 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 102858us
W20260709 12:25:00.196709 27358 consensus_meta.cc:287] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.782s	user 0.012s	sys 0.003s
I20260709 12:25:00.135952 27354 leader_election.cc:290] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:00.274688 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:00.278330 17680 raft_consensus.cc:2468] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 18.
I20260709 12:25:00.273757 17803 leader_election.cc:304] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:25:00.286697 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:00.276002 17685 raft_consensus.cc:2468] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 32.
I20260709 12:25:00.290537 17686 raft_consensus.cc:1275] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 25 index: 15. Preceding OpId from leader: term: 26 index: 16. (index mismatch)
W20260709 12:25:00.260413 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 133192us
I20260709 12:25:00.293922 17729 leader_election.cc:304] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:25:00.198583 27266 leader_election.cc:290] T a5732e71b9d54f8786b8c511d69943bf P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:00.115345 27196 consensus_queue.cc:237] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 25.19, Last appended by leader: 19, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:00.333701 17729 leader_election.cc:304] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:00.261421 27431 raft_consensus.cc:2749] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:25:00.284132 27442 raft_consensus.cc:493] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:00.332058 27382 raft_consensus.cc:697] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 33 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:25:00.294924 27154 raft_consensus.cc:3060] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:00.362496 17729 leader_election.cc:304] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:25:00.261384 27429 consensus_queue.cc:1048] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.103s
W20260709 12:25:00.365526 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 323413us
W20260709 12:25:00.368155 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 263872us
I20260709 12:25:00.370890 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 27 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:00.373446 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 27 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:00.373970 27354 raft_consensus.cc:493] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:25:00.374473 27354 raft_consensus.cc:515] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:00.374904 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:00.375665 17755 raft_consensus.cc:2393] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 18: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:25:00.376609 27354 leader_election.cc:290] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:00.251580 27406 leader_election.cc:290] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:00.221524 27430 raft_consensus.cc:515] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:00.384485 17729 leader_election.cc:304] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:25:00.337711 27451 raft_consensus.cc:2749] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:25:00.399449 18374 raft_consensus.cc:1275] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 21 index: 20. Preceding OpId from leader: term: 23 index: 21. (index mismatch)
I20260709 12:25:00.413676 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:00.384102 27448 raft_consensus.cc:2749] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:25:00.412521 27462 raft_consensus.cc:493] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:00.436254 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 355753us
I20260709 12:25:00.366536 27442 raft_consensus.cc:515] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:00.446393 17728 leader_election.cc:304] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:00.460704 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 33 candidate_status { last_received { term: 30 index: 23 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:00.443218 27455 raft_consensus.cc:493] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:00.473891 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 89512.3us
I20260709 12:25:00.476919 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:00.473932 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:00.478631 17755 raft_consensus.cc:3060] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:00.467823 27446 raft_consensus.cc:2749] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260709 12:25:00.446257 27451 raft_consensus.cc:2764] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Leader election decision vote started in defunct term 25: won
I20260709 12:25:00.482167 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:00.489219 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:00.493321 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 24 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:00.494256 17680 raft_consensus.cc:2468] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 23.
I20260709 12:25:00.490168 17681 raft_consensus.cc:2468] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 29.
I20260709 12:25:00.483994 27328 raft_consensus.cc:2804] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:25:00.498862 27328 raft_consensus.cc:493] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:00.499275 27328 raft_consensus.cc:3060] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:25:00.508332 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:00.510325 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:00.512224 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 25 candidate_status { last_received { term: 24 index: 20 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:25:00.515933 27453 log.cc:927] Time spent T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.073s	user 0.001s	sys 0.000s
W20260709 12:25:00.516968 27459 log.cc:927] Time spent T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.106s	user 0.000s	sys 0.001s
W20260709 12:25:00.519559 27460 log.cc:927] Time spent T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.089s	user 0.000s	sys 0.000s
I20260709 12:25:00.477700 27455 raft_consensus.cc:515] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:00.520331 27427 raft_consensus.cc:3055] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:25:00.520756 27427 raft_consensus.cc:740] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:25:00.488245 27369 leader_election.cc:290] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:25:00.525676 27189 consensus_meta.cc:287] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.842s	user 0.011s	sys 0.000s
I20260709 12:25:00.526137 27189 raft_consensus.cc:515] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:00.527521 27432 raft_consensus.cc:3055] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:25:00.527946 27432 raft_consensus.cc:740] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:25:00.489524 27448 consensus_queue.cc:1048] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.039s
I20260709 12:25:00.528885 27432 consensus_queue.cc:260] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 24.20, Last appended by leader: 20, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:00.448027 27462 raft_consensus.cc:515] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:00.533494 27462 leader_election.cc:290] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:00.523713 27359 raft_consensus.cc:3055] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:25:00.534539 27359 raft_consensus.cc:740] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:25:00.537273 27452 raft_consensus.cc:493] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:00.537873 27452 raft_consensus.cc:515] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:00.539223 27205 raft_consensus.cc:3055] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:25:00.539680 27205 raft_consensus.cc:740] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:25:00.540104 26924 raft_consensus.cc:493] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:25:00.540756 27266 raft_consensus.cc:493] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:00.540853 27205 consensus_queue.cc:260] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 25.18, Last appended by leader: 18, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:00.541378 27463 consensus_queue.cc:1048] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:25:00.445077 27382 consensus_queue.cc:237] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 31.20, Last appended by leader: 20, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:00.542207 27205 raft_consensus.cc:3060] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:00.490167 27437 leader_election.cc:290] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:00.540810 26924 raft_consensus.cc:515] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:00.557087 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:00.558269 17761 raft_consensus.cc:2468] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 29.
I20260709 12:25:00.541730 27397 leader_election.cc:290] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 28 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:00.541464 27266 raft_consensus.cc:515] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:00.563512 18364 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 2761) took 1351 ms. Trace:
I20260709 12:25:00.564136 18364 rpcz_store.cc:276] 0709 12:24:59.211747 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:59.212800 (+  1053us) service_pool.cc:224] Handling call
0709 12:24:59.243168 (+ 30368us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:59.979729 (+736561us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:00.563233 (+583504us) spinlock_profiling.cc:243] Waited 624 ms on lock 0x7b5400466ed4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:25:00.563445 (+   212us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1434139392}
I20260709 12:25:00.567063 17836 raft_consensus.cc:1240] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 26. Current term is 27. Ops: []
W20260709 12:25:00.569777 17654 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:00.587754 17652 leader_election.cc:304] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:00.588994 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:00.592357 17764 raft_consensus.cc:2468] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 22.
I20260709 12:25:00.598788 18334 leader_election.cc:304] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:25:00.592798 27107 consensus_queue.cc:1048] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.011s
I20260709 12:25:00.600895 18334 leader_election.cc:304] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
W20260709 12:25:00.607292 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 120395us
I20260709 12:25:00.607430 17652 leader_election.cc:304] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:00.604749 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:00.576164 27430 leader_election.cc:290] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:00.604166 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 29 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:00.590471 27370 raft_consensus.cc:493] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:25:00.614778 26599 consensus_queue.cc:1059] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 27.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20260709 12:25:00.590582 26924 leader_election.cc:290] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:00.591486 27436 leader_election.cc:290] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:00.535542 27359 consensus_queue.cc:260] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 26.20, Last appended by leader: 20, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:00.620577 17755 raft_consensus.cc:2468] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 28.
I20260709 12:25:00.610917 27343 raft_consensus.cc:2804] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:25:00.606451 27341 consensus_queue.cc:1048] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.017s
I20260709 12:25:00.616802 27358 raft_consensus.cc:2804] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:25:00.621603 27369 consensus_queue.cc:1048] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 16, Time since last communication: 0.007s
I20260709 12:25:00.628554 27154 raft_consensus.cc:2749] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:25:00.625506 27370 raft_consensus.cc:515] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:00.630355 27062 raft_consensus.cc:3055] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:25:00.632215 27442 leader_election.cc:290] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:00.610673 27354 raft_consensus.cc:2749] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:25:00.651420 17652 leader_election.cc:304] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:25:00.636795 17729 leader_election.cc:304] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:25:00.627368 27452 leader_election.cc:290] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:25:00.656435 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 463811us
I20260709 12:25:00.657658 17764 raft_consensus.cc:1275] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 22 index: 16. Preceding OpId from leader: term: 23 index: 17. (index mismatch)
I20260709 12:25:00.635607 26924 raft_consensus.cc:2804] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:25:00.650080 17801 leader_election.cc:304] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:00.661293 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 29 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:00.662411 17759 raft_consensus.cc:2468] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 28.
I20260709 12:25:00.642683 27324 raft_consensus.cc:493] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
W20260709 12:25:00.677171 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:00.671085 17652 leader_election.cc:304] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:00.645955 27358 raft_consensus.cc:493] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:25:00.664381 17729 leader_election.cc:304] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:00.647188 27343 raft_consensus.cc:697] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 26 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:25:00.650487 27062 raft_consensus.cc:740] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 27 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:25:00.690230 27442 raft_consensus.cc:2804] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:25:00.661026 27128 raft_consensus.cc:2804] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:25:00.649585 27328 raft_consensus.cc:515] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:25:00.689064 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 88287.2us
I20260709 12:25:00.664590 27359 raft_consensus.cc:3060] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:00.698568 18333 leader_election.cc:400] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 22 election: Vote denied by peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837) with higher term. Message: Invalid argument: T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 22. Current term is 23.
I20260709 12:25:00.694511 17729 leader_election.cc:304] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:25:00.720095 18333 leader_election.cc:403] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 22 election: Cancelling election due to peer responding with higher term
I20260709 12:25:00.688560 26887 raft_consensus.cc:2804] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:25:00.684511 27266 leader_election.cc:290] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:00.691689 27358 raft_consensus.cc:3060] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:00.689301 26924 raft_consensus.cc:697] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 26 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:25:00.726240 17688 raft_consensus.cc:1240] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 23. Current term is 24. Ops: []
I20260709 12:25:00.696394 27436 raft_consensus.cc:2749] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:25:00.731954 17729 leader_election.cc:304] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:25:00.700168 27442 raft_consensus.cc:493] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:00.601699 27395 consensus_queue.cc:1048] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.124s
I20260709 12:25:00.741925 17729 leader_election.cc:304] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:25:00.627619 27456 consensus_queue.cc:1048] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.122s
I20260709 12:25:00.744326 17729 leader_election.cc:304] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:00.725888 26887 raft_consensus.cc:493] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:00.690752 27324 raft_consensus.cc:515] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:00.724278 27128 raft_consensus.cc:697] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 28 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:25:00.696569 27354 consensus_queue.cc:1048] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.036s
W20260709 12:25:00.770470 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:00.790697 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 47268us
I20260709 12:25:00.795277 17729 leader_election.cc:304] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:00.758073 27107 raft_consensus.cc:2749] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Leader election lost for term 22. Reason: Vote denied by peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837) with higher term. Message: Invalid argument: T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 22. Current term is 23.
I20260709 12:25:00.736967 27321 leader_election.cc:290] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:00.758491 27462 raft_consensus.cc:2749] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:25:00.736531 27341 raft_consensus.cc:493] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:00.758440 27452 raft_consensus.cc:493] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:25:00.789683 26887 raft_consensus.cc:3060] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:00.743695 27442 raft_consensus.cc:3060] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:00.805258 27341 raft_consensus.cc:515] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:00.805624 17729 leader_election.cc:304] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:25:00.758531 27423 raft_consensus.cc:2804] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:25:00.811822 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:00.831012 17763 raft_consensus.cc:2468] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 31.
I20260709 12:25:00.832653 17801 leader_election.cc:304] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:00.833617 27267 raft_consensus.cc:2804] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:25:00.833995 27267 raft_consensus.cc:493] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:25:00.834678 27267 raft_consensus.cc:3060] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:25:00.837237 27324 leader_election.cc:290] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:25:00.838907 27478 log.cc:927] Time spent T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.098s	user 0.000s	sys 0.001s
I20260709 12:25:00.840139 27462 raft_consensus.cc:2749] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:25:00.830606 27423 raft_consensus.cc:493] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:25:00.840879 27423 raft_consensus.cc:3060] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:00.841828 17729 leader_election.cc:304] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:25:00.844099 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:00.844043 27473 consensus_queue.cc:1059] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 25.18, Next index: 19, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:25:00.812276 27370 leader_election.cc:290] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:00.844859 18364 raft_consensus.cc:2468] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 18.
I20260709 12:25:00.845059 27473 raft_consensus.cc:2804] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:25:00.845173 27476 raft_consensus.cc:3055] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 25 LEADER]: Stepping down as leader of term 25
W20260709 12:25:00.812507 27468 log.cc:927] Time spent T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.052s	user 0.000s	sys 0.000s
I20260709 12:25:00.845687 27473 raft_consensus.cc:697] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:25:00.815133 27197 raft_consensus.cc:493] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:00.818593 27477 raft_consensus.cc:2749] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 12:25:00.819056 27475 raft_consensus.cc:2804] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:25:00.846905 27475 raft_consensus.cc:493] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:25:00.846480 27197 raft_consensus.cc:515] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:00.846695 27473 consensus_queue.cc:237] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 23.17, Last appended by leader: 17, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:00.847303 27475 raft_consensus.cc:3060] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:00.812067 27472 raft_consensus.cc:2749] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:25:00.845705 27476 raft_consensus.cc:740] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 25 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:25:00.849381 27415 leader_election.cc:290] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:25:00.849658 27476 consensus_queue.cc:260] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 25.18, Last appended by leader: 18, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:00.850920 27476 raft_consensus.cc:3060] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:00.854413 27427 consensus_queue.cc:260] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 23.18, Last appended by leader: 18, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:00.646724 27413 consensus_queue.cc:1175] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "0da169ff82674dc1947055ce43de68a3" responder_term: 27 status { last_received { term: 26 index: 20 } last_committed_idx: 19 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 26. Current term is 27. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260709 12:25:00.860313 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 33 candidate_status { last_received { term: 30 index: 23 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:00.792560 27343 consensus_queue.cc:237] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 24.18, Last appended by leader: 18, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:00.861300 18363 raft_consensus.cc:2410] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for term 33 because replica has last-logged OpId of term: 31 index: 24, which is greater than that of the candidate, which has last-logged OpId of term: 30 index: 23.
I20260709 12:25:00.862360 18374 raft_consensus.cc:1240] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 23. Current term is 24. Ops: [23.17-23.17]
I20260709 12:25:00.863606 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 24 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:00.810063 27436 raft_consensus.cc:2749] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
W20260709 12:25:00.869114 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 179362us
I20260709 12:25:00.861115 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:00.874490 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:00.875393 18366 raft_consensus.cc:2468] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 24.
I20260709 12:25:00.871767 18333 leader_election.cc:304] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:25:00.877684 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:00.878896 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:00.879479 18363 raft_consensus.cc:2468] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 23.
I20260709 12:25:00.880985 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 30 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:00.881841 18364 raft_consensus.cc:2468] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 29.
I20260709 12:25:00.878322 17762 raft_consensus.cc:2468] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 19.
I20260709 12:25:00.884490 27128 consensus_queue.cc:237] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 27.19, Last appended by leader: 19, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:00.896667 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:00.899493 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:00.863540 27189 leader_election.cc:290] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 27 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:00.900519 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:00.902287 18374 raft_consensus.cc:2393] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 26: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:25:00.901324 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:00.902500 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 28 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:00.903129 18371 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 27.
I20260709 12:25:00.903501 18368 raft_consensus.cc:3060] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:00.809774 27452 raft_consensus.cc:515] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:00.904925 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:00.920083 18367 raft_consensus.cc:1275] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 24 index: 17. Preceding OpId from leader: term: 26 index: 18. (index mismatch)
I20260709 12:25:00.920722 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 28 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:00.921339 17760 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 27.
I20260709 12:25:00.921450 27452 leader_election.cc:290] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:00.921947 18367 raft_consensus.cc:1275] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 20 index: 14. Preceding OpId from leader: term: 24 index: 16. (index mismatch)
I20260709 12:25:00.804117 27062 consensus_queue.cc:260] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 27.16, Last appended by leader: 16, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:00.923272 27062 raft_consensus.cc:3060] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:00.924993 27423 raft_consensus.cc:515] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:00.931208 17755 raft_consensus.cc:1275] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 21 index: 15. Preceding OpId from leader: term: 23 index: 16. (index mismatch)
I20260709 12:25:00.934679 17764 raft_consensus.cc:1240] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 27. Current term is 28. Ops: [27.19-27.19]
I20260709 12:25:00.936535 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:25:00.937827 17682 raft_consensus.cc:3060] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:25:00.892117 27324 raft_consensus.cc:2804] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:25:00.942087 27324 raft_consensus.cc:493] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:00.942492 27324 raft_consensus.cc:3060] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:00.942087 27434 raft_consensus.cc:493] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:25:00.943152 27434 raft_consensus.cc:515] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:25:00.943966 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 227302us
I20260709 12:25:00.945371 27434 leader_election.cc:290] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:25:00.945892 27423 leader_election.cc:290] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 29 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:00.913161 27341 leader_election.cc:290] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:25:00.914109 27328 leader_election.cc:290] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 19 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:00.909632 27197 leader_election.cc:290] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:00.915284 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:00.829895 26924 consensus_queue.cc:237] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 24.15, Last appended by leader: 15, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:00.917795 27475 raft_consensus.cc:515] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:00.972445 17728 leader_election.cc:304] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:00.919445 27385 raft_consensus.cc:493] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
W20260709 12:25:00.922032 17655 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:00.922207 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 32 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:00.887349 26887 raft_consensus.cc:515] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:00.946583 17652 leader_election.cc:304] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:00.947345 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 27 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:00.981025 17762 raft_consensus.cc:2468] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 23.
I20260709 12:25:00.947630 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 25 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:00.948477 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:00.952575 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:00.954865 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:00.954777 17729 leader_election.cc:304] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:00.986900 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 30 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:00.956070 27341 raft_consensus.cc:493] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:00.956288 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 27 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:00.957065 17831 raft_consensus.cc:1240] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 26 LEADER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 25. Current term is 26. Ops: [25.18-25.18]
I20260709 12:25:00.961999 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:00.965090 27370 raft_consensus.cc:493] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:25:00.966250 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:00.967279 27423 raft_consensus.cc:493] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:25:00.914487 18368 raft_consensus.cc:2468] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 27.
I20260709 12:25:00.989946 18364 raft_consensus.cc:1275] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 22 index: 16. Preceding OpId from leader: term: 23 index: 17. (index mismatch)
I20260709 12:25:00.992518 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 28 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:01.003115 18361 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 27.
I20260709 12:25:00.973843 17728 leader_election.cc:304] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:01.008786 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 28 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:01.009115 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 30 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:25:01.010882 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 64388.3us
I20260709 12:25:01.009624 17687 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 27.
I20260709 12:25:01.014130 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:01.032604 17803 leader_election.cc:304] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:25:01.032445 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 77792.2us
W20260709 12:25:01.034513 17652 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:01.024227 17682 raft_consensus.cc:2468] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 19.
I20260709 12:25:01.045753 17729 leader_election.cc:304] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:25:01.046013 17652 leader_election.cc:304] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:01.032065 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 29 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:00.969779 27475 leader_election.cc:290] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 23 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:25:01.012676 27338 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.955s	user 0.010s	sys 0.011s
I20260709 12:25:01.048895 18371 raft_consensus.cc:1240] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 25. Current term is 26. Ops: [25.18-25.18]
I20260709 12:25:01.050442 17654 leader_election.cc:304] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:25:01.052188 18333 leader_election.cc:336] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 24 election: RPC error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Timed out: RequestConsensusVote RPC to 127.16.29.66:39651 timed out after 1.763s (SENT)
I20260709 12:25:01.056995 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:25:01.056028 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 30 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:25:01.102655 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:01.103112 17802 leader_election.cc:304] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:25:01.102805 18333 leader_election.cc:304] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:25:01.107663 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 62047.3us
W20260709 12:25:01.108335 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 57929.1us
I20260709 12:25:01.008776 27385 raft_consensus.cc:3060] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:25:01.113198 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 58198.6us
I20260709 12:25:01.149577 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:01.151633 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:01.126924 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:01.016366 27189 raft_consensus.cc:2804] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:25:01.113523 26599 raft_consensus.cc:3060] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Advancing to term 26
W20260709 12:25:01.154265 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 45629.2us
W20260709 12:25:01.109598 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 57464.6us
I20260709 12:25:01.156009 17729 leader_election.cc:304] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:01.015657 27197 raft_consensus.cc:493] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:25:01.008606 27328 raft_consensus.cc:2804] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:25:01.154313 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:01.111972 27406 raft_consensus.cc:2804] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:25:01.099968 27448 raft_consensus.cc:2804] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:25:01.049113 27196 raft_consensus.cc:493] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:25:01.013250 27354 raft_consensus.cc:493] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:25:01.163932 27196 raft_consensus.cc:515] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:01.164062 27354 raft_consensus.cc:515] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:01.108027 27394 raft_consensus.cc:493] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:01.110885 27338 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.067s	user 0.017s	sys 0.011s
I20260709 12:25:01.059732 27433 raft_consensus.cc:2749] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:25:01.048830 27205 raft_consensus.cc:493] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:25:01.055965 27245 raft_consensus.cc:493] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:25:01.151672 27472 raft_consensus.cc:2749] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
W20260709 12:25:01.395001 27128 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.507s	user 0.011s	sys 0.023s
W20260709 12:25:01.414546 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:01.149536 27359 raft_consensus.cc:2749] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
W20260709 12:25:01.430938 17652 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:01.495468 17652 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:01.151465 27461 raft_consensus.cc:2804] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:25:01.032042 27423 raft_consensus.cc:515] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:01.512447 17763 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.548s	user 0.026s	sys 0.061s
I20260709 12:25:01.049437 27452 consensus_queue.cc:1059] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 27.19, Next index: 20, Last known committed idx: 18, Time since last communication: 0.184s
I20260709 12:25:01.013028 27341 raft_consensus.cc:515] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:01.406700 17683 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.604s	user 0.022s	sys 0.031s
I20260709 12:25:01.157639 27189 raft_consensus.cc:493] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:25:01.163228 27328 raft_consensus.cc:493] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:01.013227 27370 raft_consensus.cc:515] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:01.153851 27446 consensus_queue.cc:1059] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 23.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.359s
I20260709 12:25:01.385673 27343 raft_consensus.cc:493] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:01.441304 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:01.291636 27406 raft_consensus.cc:493] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:25:01.554014 17763 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.590s	user 0.026s	sys 0.068s
I20260709 12:25:01.108423 27441 consensus_queue.cc:1048] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.361s
I20260709 12:25:01.398646 27462 raft_consensus.cc:2804] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Leader pre-election won for term 25
W20260709 12:25:01.569406 17683 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.769s	user 0.024s	sys 0.032s
I20260709 12:25:01.388288 27448 raft_consensus.cc:493] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:25:01.579286 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:01.058681 27267 raft_consensus.cc:515] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 32 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:01.509558 26599 raft_consensus.cc:2749] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:25:01.512274 27461 raft_consensus.cc:493] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:01.432240 27431 raft_consensus.cc:493] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:01.600201 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 490737us
W20260709 12:25:01.610947 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 455199us
W20260709 12:25:01.635561 17655 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.517s	user 0.036s	sys 0.053s
W20260709 12:25:01.680356 17655 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.562s	user 0.036s	sys 0.053s
I20260709 12:25:01.397066 27358 raft_consensus.cc:515] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:25:01.685925 17653 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.536s	user 0.027s	sys 0.076s
I20260709 12:25:01.681917 17728 leader_election.cc:304] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:01.684762 17802 leader_election.cc:304] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:01.684509 26599 raft_consensus.cc:493] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:25:01.567260 27189 raft_consensus.cc:3060] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Advancing to term 30
W20260709 12:25:01.687733 17653 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.562s	user 0.027s	sys 0.076s
I20260709 12:25:01.694867 17837 raft_consensus.cc:2410] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for term 30 because replica has last-logged OpId of term: 29 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 28 index: 18.
I20260709 12:25:01.588397 27328 raft_consensus.cc:3060] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:01.492098 27205 raft_consensus.cc:515] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:01.610219 27473 raft_consensus.cc:493] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:25:01.575730 27433 raft_consensus.cc:493] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:25:01.706472 26599 raft_consensus.cc:515] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:01.601590 27449 raft_consensus.cc:493] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:25:01.707857 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:01.655534 27461 raft_consensus.cc:3060] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:01.711349 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:01.492660 27270 consensus_queue.cc:1048] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.131s
I20260709 12:25:01.510413 27245 raft_consensus.cc:515] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:01.511811 27197 raft_consensus.cc:515] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:01.623327 27462 raft_consensus.cc:493] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:01.668515 27448 raft_consensus.cc:3060] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:01.683976 27363 raft_consensus.cc:3060] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:01.632483 27406 raft_consensus.cc:3060] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:01.770931 17759 raft_consensus.cc:2393] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 28: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:25:01.753660 26599 leader_election.cc:290] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:01.684051 26789 raft_consensus.cc:493] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:25:01.714622 27373 raft_consensus.cc:493] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:01.816071 27245 leader_election.cc:290] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:01.819901 27373 raft_consensus.cc:515] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:01.820250 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:01.820950 27245 raft_consensus.cc:493] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:01.821458 27245 raft_consensus.cc:515] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:01.822005 27373 leader_election.cc:290] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:01.817932 27270 raft_consensus.cc:493] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:01.824431 27270 raft_consensus.cc:515] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:01.819029 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:01.708770 27128 raft_consensus.cc:493] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:25:01.813330 26599 consensus_queue.cc:1048] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.016s
W20260709 12:25:01.863955 18364 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.755s	user 0.026s	sys 0.067s
W20260709 12:25:01.867046 18364 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.758s	user 0.026s	sys 0.067s
I20260709 12:25:01.868292 26599 raft_consensus.cc:2749] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 12:25:01.876072 26599 raft_consensus.cc:493] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:25:01.876737 26599 raft_consensus.cc:515] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:01.877226 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:01.752955 27267 leader_election.cc:290] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 32 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:01.889020 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 29 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:01.636454 27343 raft_consensus.cc:515] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:01.803413 27446 raft_consensus.cc:2804] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Leader election won for term 19
W20260709 12:25:01.929899 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:01.945454 26599 leader_election.cc:290] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:25:01.760290 27062 consensus_meta.cc:287] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.825s	user 0.012s	sys 0.000s
W20260709 12:25:01.751036 27464 log.cc:927] Time spent T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.100s	user 0.001s	sys 0.000s
I20260709 12:25:01.713718 27395 raft_consensus.cc:493] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:25:01.710960 27111 raft_consensus.cc:493] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:01.823724 27245 leader_election.cc:290] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:25:01.942210 17687 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.777s	user 0.027s	sys 0.070s
I20260709 12:25:01.945945 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 28 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:01.875670 26789 raft_consensus.cc:515] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:01.946271 27196 leader_election.cc:290] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:01.946588 27467 consensus_queue.cc:1059] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 21.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.807s
I20260709 12:25:01.946712 27270 leader_election.cc:290] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:01.952854 27446 raft_consensus.cc:697] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:25:01.884052 27128 raft_consensus.cc:515] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:01.955026 17687 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.790s	user 0.027s	sys 0.071s
I20260709 12:25:01.969077 27467 raft_consensus.cc:3055] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:25:01.957654 17755 raft_consensus.cc:2468] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 27.
I20260709 12:25:01.862237 27462 raft_consensus.cc:3060] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:01.702307 27431 raft_consensus.cc:515] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:01.430106 27394 raft_consensus.cc:515] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:01.802979 27449 raft_consensus.cc:515] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:01.953265 27107 raft_consensus.cc:493] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:01.953739 27324 consensus_meta.cc:287] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 1.011s	user 0.009s	sys 0.000s
W20260709 12:25:01.965530 27483 log.cc:927] Time spent T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 1.081s	user 0.003s	sys 0.003s
I20260709 12:25:01.812804 27433 raft_consensus.cc:515] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:01.800624 27473 raft_consensus.cc:515] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:01.967303 27341 leader_election.cc:290] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:25:02.045116 17688 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.932s	user 0.034s	sys 0.085s
I20260709 12:25:01.957226 26789 leader_election.cc:290] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:01.965201 27358 leader_election.cc:290] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 30 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:01.952628 27343 leader_election.cc:290] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:01.962570 27441 raft_consensus.cc:493] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:25:02.071327 17832 raft_consensus.cc:3060] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:25:01.987133 27334 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.821s	user 0.009s	sys 0.090s
W20260709 12:25:02.087357 27334 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.932s	user 0.010s	sys 0.092s
I20260709 12:25:01.966049 27196 raft_consensus.cc:493] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:02.077841 17688 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.965s	user 0.034s	sys 0.085s
I20260709 12:25:02.039952 27472 raft_consensus.cc:493] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:01.971076 27446 consensus_queue.cc:237] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.11, Last appended by leader: 11, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:01.957873 27111 raft_consensus.cc:515] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:25:02.083367 27496 log.cc:927] Time spent T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.109s	user 0.001s	sys 0.001s
I20260709 12:25:02.084707 27205 leader_election.cc:290] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:02.085457 27423 leader_election.cc:290] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:02.086030 17683 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37396 (request call id 3420) took 1367 ms. Trace:
I20260709 12:25:02.101008 17683 rpcz_store.cc:276] 0709 12:25:00.718307 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:00.724757 (+  6450us) service_pool.cc:224] Handling call
0709 12:25:00.725210 (+   453us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:00.732608 (+  7398us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:00.732645 (+    37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:00.758675 (+ 26030us) log.cc:844] Serialized 26 byte log entry
0709 12:25:01.623740 (+865065us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:25:01.623787 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:01.623868 (+    81us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:02.085824 (+461956us) raft_consensus.cc:1685] finished
0709 12:25:02.085843 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:02.085973 (+   130us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":297,"raft.run_cpu_time_us":146,"raft.run_wall_time_us":8191,"thread_start_us":823033,"threads_started":1,"wal-append.queue_time_us":1291356}
I20260709 12:25:02.102181 17836 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35042 (request call id 3449) took 2457 ms. Trace:
I20260709 12:25:02.102551 17836 rpcz_store.cc:276] 0709 12:24:59.645191 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:59.645336 (+   145us) service_pool.cc:224] Handling call
0709 12:25:00.566852 (+921516us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:00.567576 (+   724us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:02.101953 (+1534377us) spinlock_profiling.cc:243] Waited 921 ms on lock 0x7b54003cc8d0. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a61f8ad 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:25:02.102097 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2118364544}
I20260709 12:25:02.086565 27328 raft_consensus.cc:515] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:02.104063 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 27 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:02.104816 27328 leader_election.cc:290] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 24 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:02.104975 17829 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37994 (request call id 1812) took 1072 ms. Trace:
I20260709 12:25:01.985288 27467 raft_consensus.cc:740] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:25:02.105587 17829 rpcz_store.cc:276] 0709 12:25:01.032942 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:02.103740 (+1070798us) service_pool.cc:224] Handling call
0709 12:25:02.104912 (+  1172us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:02.106855 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:02.106495 27467 consensus_queue.cc:260] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 23.17, Last appended by leader: 17, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:01.954432 27395 raft_consensus.cc:515] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:02.109508 18368 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 2783) took 1117 ms. Trace:
W20260709 12:25:02.112255 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:02.103822 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:02.095206 17832 raft_consensus.cc:2468] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 27.
I20260709 12:25:02.112193 18368 rpcz_store.cc:276] 0709 12:25:00.992446 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:01.002083 (+  9637us) service_pool.cc:224] Handling call
0709 12:25:01.002409 (+   326us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:01.026395 (+ 23986us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:01.026442 (+    47us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:01.027130 (+   688us) log.cc:844] Serialized 26 byte log entry
0709 12:25:01.609475 (+582345us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:25:01.609566 (+    91us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:01.609750 (+   184us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:02.109209 (+499459us) raft_consensus.cc:1685] finished
0709 12:25:02.109231 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:02.109395 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":24765,"raft.queue_time_us":23708,"raft.run_cpu_time_us":131,"raft.run_wall_time_us":22058,"thread_start_us":455784,"threads_started":1,"wal-append.queue_time_us":964546}
I20260709 12:25:02.107717 17829 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35040 (request call id 3641) took 1139 ms. Trace:
I20260709 12:25:02.107918 27467 raft_consensus.cc:3060] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:25:02.092975 27476 consensus_meta.cc:287] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 1.209s	user 0.011s	sys 0.000s
I20260709 12:25:02.093605 27128 leader_election.cc:290] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:25:02.115164 27436 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.955s	user 0.014s	sys 0.033s
W20260709 12:25:02.131820 18362 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.620s	user 0.005s	sys 0.007s
W20260709 12:25:02.142594 18362 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.631s	user 0.006s	sys 0.007s
W20260709 12:25:02.132505 17804 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.734s	user 0.005s	sys 0.008s
W20260709 12:25:02.143319 17804 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.745s	user 0.005s	sys 0.008s
I20260709 12:25:02.135324 17836 raft_consensus.cc:3060] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:25:02.117115 17829 rpcz_store.cc:276] 0709 12:25:00.967900 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:02.106549 (+1138649us) service_pool.cc:224] Handling call
0709 12:25:02.107650 (+  1101us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:02.067390 27107 raft_consensus.cc:515] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:02.138705 17756 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38776 (request call id 3276) took 2470 ms. Trace:
W20260709 12:25:02.136025 27338 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.562s	user 0.004s	sys 0.008s
I20260709 12:25:02.070724 27324 raft_consensus.cc:515] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:02.132802 17682 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.702s	user 0.006s	sys 0.006s
W20260709 12:25:02.151461 17682 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.750s	user 0.007s	sys 0.006s
I20260709 12:25:02.148667 17756 rpcz_store.cc:276] 0709 12:24:59.648690 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:59.648821 (+   131us) service_pool.cc:224] Handling call
0709 12:24:59.679166 (+ 30345us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:00.026629 (+347463us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:00.026666 (+    37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:00.041659 (+ 14993us) log.cc:844] Serialized 26 byte log entry
0709 12:25:00.385549 (+343890us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:25:00.385692 (+   143us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:02.110305 (+1724613us) spinlock_profiling.cc:243] Waited 46.1 ms on lock 0x7b54002b6454. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:25:02.110374 (+    69us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:02.110436 (+    62us) raft_consensus.cc:1685] finished
0709 12:25:02.110454 (+    18us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:02.118768 (+  8314us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":203540,"raft.run_cpu_time_us":178,"raft.run_wall_time_us":27679,"spinlock_wait_cycles":106113280,"thread_start_us":581105,"threads_started":2,"wal-append.queue_time_us":303973,"wal-append.run_cpu_time_us":3293,"wal-append.run_wall_time_us":1155915}
W20260709 12:25:02.144572 27436 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.983s	user 0.014s	sys 0.034s
I20260709 12:25:02.158607 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 28 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:02.096266 27441 raft_consensus.cc:515] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:02.098732 27196 raft_consensus.cc:515] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:02.152088 27452 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.546s	user 0.004s	sys 0.008s
I20260709 12:25:02.212121 17836 raft_consensus.cc:2468] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 19.
W20260709 12:25:02.157805 27338 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.609s	user 0.004s	sys 0.008s
I20260709 12:25:02.108469 27472 raft_consensus.cc:515] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:25:02.235608 17837 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35040 (request call id 3643) took 1262 ms (client timeout 1531 ms). Trace:
I20260709 12:25:02.161175 27509 raft_consensus.cc:493] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:25:02.218607 27448 raft_consensus.cc:515] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:02.168058 27510 raft_consensus.cc:493] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:02.204449 27324 leader_election.cc:290] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 24 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:02.177176 27259 leader_election.cc:290] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:02.156247 27370 leader_election.cc:290] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:02.202615 27107 leader_election.cc:290] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:02.200784 27111 leader_election.cc:290] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:25:02.166561 27442 consensus_meta.cc:287] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 1.338s	user 0.007s	sys 0.004s
W20260709 12:25:02.168941 27502 log.cc:927] Time spent T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.070s	user 0.000s	sys 0.000s
W20260709 12:25:02.183483 27499 log.cc:927] Time spent T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.211s	user 0.000s	sys 0.001s
W20260709 12:25:02.212656 17655 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.530s	user 0.015s	sys 0.024s
I20260709 12:25:02.240521 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 31 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:02.261060 17835 raft_consensus.cc:2468] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 30.
I20260709 12:25:02.262545 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 29 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:02.167894 27450 raft_consensus.cc:493] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:02.264246 27450 raft_consensus.cc:515] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:02.162330 27270 raft_consensus.cc:493] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:25:02.266455 27270 raft_consensus.cc:515] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:02.236486 17837 rpcz_store.cc:269] 0709 12:25:00.968574 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:01.008859 (+ 40285us) service_pool.cc:224] Handling call
0709 12:25:02.230308 (+1221449us) spinlock_profiling.cc:243] Waited 684 ms on lock 0x7b54002eac54. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:25:02.230495 (+   187us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1574237568}
I20260709 12:25:02.270301 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 25 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:02.271070 17837 raft_consensus.cc:2468] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 23.
I20260709 12:25:02.165086 27062 raft_consensus.cc:493] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:25:02.272511 27062 raft_consensus.cc:515] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:02.275094 27062 leader_election.cc:290] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:25:02.201660 27427 raft_consensus.cc:3060] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:02.193022 27473 leader_election.cc:290] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:02.194311 27395 leader_election.cc:290] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:25:02.221052 27189 consensus_meta.cc:287] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.505s	user 0.009s	sys 0.000s
I20260709 12:25:02.271672 17837 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37994 (request call id 1813) took 1114 ms. Trace:
I20260709 12:25:02.276652 27189 raft_consensus.cc:515] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:02.277192 17837 rpcz_store.cc:276] 0709 12:25:01.157617 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:02.269951 (+1112334us) service_pool.cc:224] Handling call
0709 12:25:02.271601 (+  1650us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:02.278579 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:02.280133 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 25 candidate_status { last_received { term: 24 index: 20 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:25:02.236845 27385 consensus_meta.cc:287] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 1.071s	user 0.011s	sys 0.000s
I20260709 12:25:02.281387 27385 raft_consensus.cc:515] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:02.238809 27197 leader_election.cc:290] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:02.284657 17680 raft_consensus.cc:2393] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 24: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:25:02.285782 17835 raft_consensus.cc:2393] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 25: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
W20260709 12:25:02.216775 27452 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.626s	user 0.004s	sys 0.009s
W20260709 12:25:02.290880 17655 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.608s	user 0.016s	sys 0.024s
I20260709 12:25:02.292595 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 29 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:02.206395 27406 raft_consensus.cc:515] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:02.296738 17801 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.543s	user 0.023s	sys 0.007s
W20260709 12:25:02.297214 17801 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.544s	user 0.023s	sys 0.007s
W20260709 12:25:02.250726 26924 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.639s	user 0.018s	sys 0.021s
W20260709 12:25:02.298753 17761 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.139s	user 0.004s	sys 0.026s
W20260709 12:25:02.299093 17761 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.140s	user 0.004s	sys 0.026s
W20260709 12:25:02.297981 17653 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.603s	user 0.013s	sys 0.017s
I20260709 12:25:02.290678 17835 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35040 (request call id 3640) took 1332 ms. Trace:
I20260709 12:25:02.301678 18364 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 2784) took 1309 ms. Trace:
I20260709 12:25:02.302034 17837 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35042 (request call id 3460) took 1335 ms. Trace:
I20260709 12:25:02.301097 17835 rpcz_store.cc:276] 0709 12:25:00.957896 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:02.287478 (+1329582us) service_pool.cc:224] Handling call
0709 12:25:02.289856 (+  2378us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:02.290112 (+   256us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:02.290155 (+    43us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:02.290223 (+    68us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:25:02.290304 (+    81us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:02.290385 (+    81us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:02.290552 (+   167us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:25:02.299571 17653 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.605s	user 0.013s	sys 0.017s
W20260709 12:25:02.304535 17729 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.145s	user 0.002s	sys 0.029s
W20260709 12:25:02.305228 17729 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.146s	user 0.002s	sys 0.029s
W20260709 12:25:02.305661 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 1.14965e+06us
W20260709 12:25:02.305706 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:02.306326 18333 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 1.150s	user 0.005s	sys 0.033s
I20260709 12:25:02.295298 27431 leader_election.cc:290] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:02.306396 17729 leader_election.cc:304] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:25:02.315327 17835 raft_consensus.cc:1275] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 21 index: 20. Preceding OpId from leader: term: 23 index: 21. (index mismatch)
I20260709 12:25:02.318643 17829 raft_consensus.cc:2393] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 28: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:25:02.331856 17835 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35040 (request call id 3642) took 1363 ms. Trace:
I20260709 12:25:02.332362 17835 rpcz_store.cc:276] 0709 12:25:00.968266 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:02.304139 (+1335873us) service_pool.cc:224] Handling call
0709 12:25:02.304429 (+   290us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:02.331544 (+ 27115us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:02.331795 (+   251us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:02.334062 17835 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35040 (request call id 3645) took 1364 ms. Trace:
I20260709 12:25:02.334555 17835 rpcz_store.cc:276] 0709 12:25:00.969171 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:02.333256 (+1364085us) service_pool.cc:224] Handling call
0709 12:25:02.333565 (+   309us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:02.333780 (+   215us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:02.333803 (+    23us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:02.333838 (+    35us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:25:02.333866 (+    28us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:02.333913 (+    47us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:02.334013 (+   100us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:02.305178 17837 rpcz_store.cc:276] 0709 12:25:00.966790 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:02.299659 (+1332869us) service_pool.cc:224] Handling call
0709 12:25:02.300052 (+   393us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:02.300269 (+   217us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:02.300305 (+    36us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:02.301748 (+  1443us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:25:02.301794 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:02.301863 (+    69us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:02.301984 (+   121us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:25:02.298997 26924 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.710s	user 0.019s	sys 0.021s
I20260709 12:25:02.297668 27395 raft_consensus.cc:493] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:25:02.260751 27442 raft_consensus.cc:515] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:02.361117 17838 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.205s	user 0.024s	sys 0.021s
W20260709 12:25:02.361521 17838 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.205s	user 0.024s	sys 0.021s
I20260709 12:25:02.341404 17686 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37402 (request call id 3873) took 1363 ms. Trace:
I20260709 12:25:02.340994 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 31 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:25:02.357511 17802 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.648s	user 0.028s	sys 0.014s
W20260709 12:25:02.362773 17802 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.653s	user 0.028s	sys 0.015s
I20260709 12:25:02.246142 27510 raft_consensus.cc:515] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:02.302179 18364 rpcz_store.cc:276] 0709 12:25:00.997083 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:01.002219 (+  5136us) service_pool.cc:224] Handling call
0709 12:25:01.002465 (+   246us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:01.002657 (+   192us) raft_consensus.cc:1534] Early marking committed up to index 20
0709 12:25:01.868348 (+865691us) log.cc:844] Serialized 14 byte log entry
0709 12:25:02.301286 (+432938us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:02.301337 (+    51us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:25:02.301382 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:02.301468 (+    86us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:02.301608 (+   140us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":22607,"prepare.queue_time_us":870016,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":41,"raft.queue_time_us":2643,"raft.run_cpu_time_us":210,"raft.run_wall_time_us":20226,"thread_start_us":1191933,"threads_started":2}
I20260709 12:25:02.366226 27510 leader_election.cc:290] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:02.298627 27354 leader_election.cc:290] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:02.385905 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:02.245115 27461 raft_consensus.cc:515] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:02.244550 27509 raft_consensus.cc:515] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:02.394855 17763 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38776 (request call id 3290) took 1524 ms. Trace:
I20260709 12:25:02.300530 27438 raft_consensus.cc:493] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:02.336344 27430 raft_consensus.cc:493] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:25:02.362166 17838 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35040 (request call id 3636) took 1409 ms. Trace:
I20260709 12:25:02.362407 17686 rpcz_store.cc:276] 0709 12:25:00.978385 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:02.340499 (+1362114us) service_pool.cc:224] Handling call
0709 12:25:02.340812 (+   313us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:02.341013 (+   201us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:02.341044 (+    31us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:02.341088 (+    44us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:25:02.341119 (+    31us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:02.341173 (+    54us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:02.341327 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:02.368539 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:02.369306 18366 raft_consensus.cc:1240] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 26 LEADER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 25. Current term is 26. Ops: []
W20260709 12:25:02.369378 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 684695us
I20260709 12:25:02.359556 17837 raft_consensus.cc:1240] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 32 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 31. Current term is 32. Ops: []
I20260709 12:25:02.386534 27395 raft_consensus.cc:515] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:25:02.353813 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 47497.1us
I20260709 12:25:02.473393 17729 leader_election.cc:304] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
W20260709 12:25:02.330458 18333 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 1.174s	user 0.009s	sys 0.033s
W20260709 12:25:02.474979 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 1.31872e+06us
W20260709 12:25:02.475544 18333 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.16.29.66:39651 timed out after 2.177s (SENT) [suppressed 9 similar messages]
I20260709 12:25:02.386596 17760 raft_consensus.cc:3060] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:25:02.388664 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:02.390218 17759 raft_consensus.cc:1240] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 28. Current term is 29. Ops: []
I20260709 12:25:02.306102 27111 raft_consensus.cc:493] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:25:02.435489 17831 raft_consensus.cc:1275] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 25 index: 19. Preceding OpId from leader: term: 26 index: 20. (index mismatch)
I20260709 12:25:02.432912 27432 raft_consensus.cc:3060] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:02.437407 26887 leader_election.cc:290] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 24 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:02.435657 18362 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 2781) took 1449 ms. Trace:
I20260709 12:25:02.484867 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:02.387310 27433 leader_election.cc:290] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:02.432751 27448 leader_election.cc:290] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 20 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:02.370326 27517 raft_consensus.cc:2804] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:25:02.440577 27472 leader_election.cc:290] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:25:02.493880 27518 log.cc:927] Time spent T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.081s	user 0.000s	sys 0.000s
I20260709 12:25:02.484740 18362 rpcz_store.cc:276] 0709 12:25:00.986455 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:00.990751 (+  4296us) service_pool.cc:224] Handling call
0709 12:25:00.991178 (+   427us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:01.031538 (+ 40360us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:01.031577 (+    39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:01.162229 (+130652us) log.cc:844] Serialized 26 byte log entry
0709 12:25:02.143009 (+980780us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:25:02.143055 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:02.143127 (+    72us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:02.435414 (+292287us) raft_consensus.cc:1685] finished
0709 12:25:02.435439 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:02.435588 (+   149us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":9054,"raft.run_cpu_time_us":192,"raft.run_wall_time_us":49533,"thread_start_us":631286,"threads_started":1,"wal-append.queue_time_us":902455}
I20260709 12:25:02.496870 27406 leader_election.cc:290] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 28 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:25:02.498761 27363 consensus_meta.cc:287] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.638s	user 0.008s	sys 0.004s
I20260709 12:25:02.499490 27363 raft_consensus.cc:2749] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:25:02.466102 27462 raft_consensus.cc:515] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:02.506547 27462 leader_election.cc:290] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 25 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:02.512363 27266 raft_consensus.cc:493] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:25:02.512704 18333 leader_election.cc:304] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:02.513206 27266 raft_consensus.cc:515] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:02.472425 27307 leader_election.cc:290] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 18 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:02.414619 27438 raft_consensus.cc:515] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:02.424860 27500 log.cc:927] Time spent T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.061s	user 0.000s	sys 0.000s
W20260709 12:25:02.424302 27513 log.cc:927] Time spent T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.125s	user 0.001s	sys 0.000s
I20260709 12:25:02.433468 27441 leader_election.cc:290] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:02.463406 17683 raft_consensus.cc:1240] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 28. Current term is 29. Ops: [28.20-28.20]
I20260709 12:25:02.444898 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:02.395445 17763 rpcz_store.cc:276] 0709 12:25:00.870635 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:00.870777 (+   142us) service_pool.cc:224] Handling call
0709 12:25:00.871128 (+   351us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:00.883161 (+ 12033us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:00.883190 (+    29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:00.883634 (+   444us) log.cc:844] Serialized 26 byte log entry
0709 12:25:01.620583 (+736949us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:25:01.620642 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:01.620743 (+   101us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:02.394506 (+773763us) raft_consensus.cc:1685] finished
0709 12:25:02.394542 (+    36us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:02.394768 (+   226us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":11664,"raft.queue_time_us":16901,"raft.run_cpu_time_us":138,"raft.run_wall_time_us":12984,"thread_start_us":656211,"threads_started":1,"wal-append.queue_time_us":1137578}
I20260709 12:25:02.444485 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:25:02.453392 17654 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:02.480535 27436 raft_consensus.cc:2749] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:25:02.404120 17838 rpcz_store.cc:276] 0709 12:25:00.952486 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:00.952741 (+   255us) service_pool.cc:224] Handling call
0709 12:25:00.953097 (+   356us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:00.953332 (+   235us) raft_consensus.cc:1534] Early marking committed up to index 17
0709 12:25:00.953663 (+   331us) log.cc:844] Serialized 14 byte log entry
0709 12:25:02.361892 (+1408229us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:02.361938 (+    46us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:25:02.361978 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:02.362029 (+    51us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:02.362124 (+    95us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":168,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":42,"raft.queue_time_us":392225,"raft.run_cpu_time_us":134,"raft.run_wall_time_us":21653,"spinlock_wait_cycles":93084800,"thread_start_us":1407612,"threads_started":2,"wal-append.queue_time_us":596247}
I20260709 12:25:02.415493 17686 raft_consensus.cc:1240] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 29. Current term is 30. Ops: []
I20260709 12:25:02.532622 17686 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37402 (request call id 3874) took 1553 ms. Trace:
I20260709 12:25:02.533138 17686 rpcz_store.cc:276] 0709 12:25:00.978761 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:02.414714 (+1435953us) service_pool.cc:224] Handling call
0709 12:25:02.415294 (+   580us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:02.532255 (+116961us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:02.532544 (+   289us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:02.413993 17756 raft_consensus.cc:2468] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 24.
I20260709 12:25:02.304381 27062 raft_consensus.cc:493] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:02.469058 17802 leader_election.cc:304] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:25:02.469373 17685 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44898 (request call id 1902) took 2458 ms. Trace:
I20260709 12:25:02.479355 17729 leader_election.cc:304] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:02.480655 17688 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37396 (request call id 3421) took 1759 ms. Trace:
I20260709 12:25:02.488373 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:02.486047 17764 raft_consensus.cc:2468] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 27.
W20260709 12:25:02.466744 27451 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.584s	user 0.000s	sys 0.039s
I20260709 12:25:02.486505 27196 leader_election.cc:290] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:02.490980 27517 raft_consensus.cc:493] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:25:02.419914 27430 raft_consensus.cc:515] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:02.518625 27475 raft_consensus.cc:3055] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:25:02.520943 18333 leader_election.cc:304] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:02.530400 17757 raft_consensus.cc:2393] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 27: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:25:02.405082 27455 leader_election.cc:290] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:02.541097 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 28 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:02.521896 27266 leader_election.cc:290] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:02.555207 17763 raft_consensus.cc:3060] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:02.535444 17685 rpcz_store.cc:276] 0709 12:25:00.006377 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:00.006526 (+   149us) service_pool.cc:224] Handling call
0709 12:25:02.456776 (+2450250us) spinlock_profiling.cc:243] Waited 241 ms on lock 0x7b5400208354. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:25:02.464873 (+  8097us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":555346816}
I20260709 12:25:02.536375 17688 rpcz_store.cc:276] 0709 12:25:00.720802 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:00.749307 (+ 28505us) service_pool.cc:224] Handling call
0709 12:25:00.749778 (+   471us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:00.750046 (+   268us) raft_consensus.cc:1534] Early marking committed up to index 18
0709 12:25:00.894592 (+144546us) log.cc:844] Serialized 14 byte log entry
0709 12:25:02.471794 (+1577202us) log.cc:844] Serialized 14 byte log entry
0709 12:25:02.480159 (+  8365us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:02.480321 (+   162us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:25:02.480379 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:02.480450 (+    71us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:02.480580 (+   130us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":8627,"prepare.queue_time_us":352049,"prepare.run_cpu_time_us":50,"prepare.run_wall_time_us":4801,"raft.queue_time_us":410468,"raft.run_cpu_time_us":334,"raft.run_wall_time_us":30238,"thread_start_us":1679108,"threads_started":4,"wal-append.queue_time_us":916702}
I20260709 12:25:02.545101 17759 raft_consensus.cc:2468] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 27.
I20260709 12:25:02.557312 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:02.559252 18333 leader_election.cc:304] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:02.560262 18333 leader_election.cc:304] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
W20260709 12:25:02.554185 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 74937.6us
W20260709 12:25:02.545195 27496 log.cc:927] Time spent T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.057s	user 0.000s	sys 0.000s
I20260709 12:25:02.549528 17686 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37402 (request call id 3875) took 1570 ms. Trace:
I20260709 12:25:02.527727 27452 raft_consensus.cc:2749] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260709 12:25:02.546828 27451 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.714s	user 0.000s	sys 0.040s
I20260709 12:25:02.567744 17686 rpcz_store.cc:276] 0709 12:25:00.979075 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:02.548255 (+1569180us) service_pool.cc:224] Handling call
0709 12:25:02.548774 (+   519us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:02.549024 (+   250us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:02.549070 (+    46us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:02.549136 (+    66us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:25:02.549187 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:02.549267 (+    80us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:02.549417 (+   150us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:02.571025 27266 raft_consensus.cc:2804] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:25:02.551744 27062 raft_consensus.cc:515] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:02.572340 27527 raft_consensus.cc:2749] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
W20260709 12:25:02.572358 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 103437us
I20260709 12:25:02.571600 27266 raft_consensus.cc:493] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:25:02.573235 27062 leader_election.cc:290] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:02.573662 17802 leader_election.cc:304] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:02.574786 27406 raft_consensus.cc:2749] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:25:02.576337 27524 consensus_queue.cc:1059] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 29.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20260709 12:25:02.545091 27307 raft_consensus.cc:493] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:02.560324 17760 raft_consensus.cc:2468] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 32.
I20260709 12:25:02.579361 17763 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 28.
I20260709 12:25:02.488399 27111 raft_consensus.cc:515] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:02.545420 27443 raft_consensus.cc:493] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:25:02.553002 27196 raft_consensus.cc:493] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:02.554109 27475 raft_consensus.cc:740] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:25:02.601645 27475 consensus_queue.cc:260] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 23.16, Last appended by leader: 16, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:02.602759 27475 raft_consensus.cc:3060] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:25:02.604604 17836 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35042 (request call id 3461) took 1637 ms (client timeout 1884 ms). Trace:
W20260709 12:25:02.605055 17836 rpcz_store.cc:269] 0709 12:25:00.972241 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:02.103541 (+1131300us) service_pool.cc:224] Handling call
0709 12:25:02.604382 (+500841us) spinlock_profiling.cc:243] Waited 21.7 ms on lock 0x7b54000898d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:25:02.604529 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":50016000}
I20260709 12:25:02.553151 27517 raft_consensus.cc:3060] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:02.581281 27510 raft_consensus.cc:3055] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:25:02.607014 27510 raft_consensus.cc:740] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 29 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:25:02.607901 27510 consensus_queue.cc:260] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 29.17, Last appended by leader: 17, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:02.609102 27510 raft_consensus.cc:3060] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:02.610369 27395 leader_election.cc:290] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:02.580368 27451 raft_consensus.cc:493] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:25:02.570484 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 24 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:25:02.583493 27307 raft_consensus.cc:515] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:02.611747 27451 raft_consensus.cc:515] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:02.612785 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 24 candidate_status { last_received { term: 21 index: 20 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:02.613920 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 28 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:25:02.614037 27307 leader_election.cc:290] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:02.614581 17686 raft_consensus.cc:2393] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 28: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:25:02.614943 17761 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38786 (request call id 4112) took 1866 ms. Trace:
I20260709 12:25:02.564070 27338 raft_consensus.cc:2749] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260709 12:25:02.595188 17682 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44898 (request call id 1917) took 1645 ms. Trace:
I20260709 12:25:02.616473 17682 rpcz_store.cc:276] 0709 12:25:00.949398 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:01.164101 (+214703us) service_pool.cc:224] Handling call
0709 12:25:01.164473 (+   372us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:02.152002 (+987529us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:02.152035 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:02.161988 (+  9953us) log.cc:844] Serialized 26 byte log entry
0709 12:25:02.473529 (+311541us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:25:02.473592 (+    63us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:02.473687 (+    95us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:02.594964 (+121277us) raft_consensus.cc:1685] finished
0709 12:25:02.594987 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:02.595128 (+   141us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":115864,"raft.queue_time_us":460670,"raft.run_cpu_time_us":142,"raft.run_wall_time_us":59949,"spinlock_wait_cycles":14592,"thread_start_us":975589,"threads_started":2,"wal-append.queue_time_us":337789}
I20260709 12:25:02.618386 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:02.618772 27509 leader_election.cc:290] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:02.619035 17682 raft_consensus.cc:2393] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 19: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:25:02.619447 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:02.596401 27443 raft_consensus.cc:515] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:02.552523 27519 consensus_queue.cc:1048] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 18, Time since last communication: 0.052s
I20260709 12:25:02.568547 27448 raft_consensus.cc:2804] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:25:02.615402 17761 rpcz_store.cc:276] 0709 12:25:00.748151 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:00.748312 (+   161us) service_pool.cc:224] Handling call
0709 12:25:00.790044 (+ 41732us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:00.792788 (+  2744us) raft_consensus.cc:1534] Early marking committed up to index 18
0709 12:25:00.932780 (+139992us) log.cc:844] Serialized 14 byte log entry
0709 12:25:01.150843 (+218063us) log.cc:844] Serialized 14 byte log entry
0709 12:25:02.299427 (+1148584us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:02.313122 (+ 13695us) log.cc:844] Serialized 26 byte log entry
0709 12:25:02.315218 (+  2096us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:25:02.315282 (+    64us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:02.315370 (+    88us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:02.614727 (+299357us) raft_consensus.cc:1685] finished
0709 12:25:02.614749 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:02.614889 (+   140us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":148426,"prepare.queue_time_us":140903,"prepare.run_cpu_time_us":99,"prepare.run_wall_time_us":5237,"raft.queue_time_us":315714,"raft.run_cpu_time_us":549,"raft.run_wall_time_us":94016,"thread_start_us":1356446,"threads_started":2,"wal-append.queue_time_us":1154771}
I20260709 12:25:02.600128 27196 raft_consensus.cc:515] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:02.625578 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 27 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:02.602078 27525 raft_consensus.cc:493] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:25:02.570340 18333 leader_election.cc:304] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:25:02.595012 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 24 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:02.620138 17757 raft_consensus.cc:2468] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 24.
I20260709 12:25:02.573593 27266 raft_consensus.cc:3060] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:02.550541 27370 consensus_queue.cc:1059] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 28.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.032s
I20260709 12:25:02.630273 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 29 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:02.622756 17835 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37994 (request call id 1811) took 1599 ms. Trace:
I20260709 12:25:02.631183 17759 raft_consensus.cc:2393] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 29: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:25:02.619858 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:02.564877 27324 raft_consensus.cc:2804] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:25:02.621409 27461 leader_election.cc:290] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 28 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:02.583307 27450 leader_election.cc:290] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:02.615442 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:25:02.631412 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 61145us
I20260709 12:25:02.633392 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 29 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:02.587443 17760 raft_consensus.cc:1240] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 32 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 31. Current term is 32. Ops: [31.21-31.21]
I20260709 12:25:02.639273 17652 leader_election.cc:304] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:02.643785 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:02.644470 17836 raft_consensus.cc:2468] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 28.
I20260709 12:25:02.646649 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:02.633462 17763 raft_consensus.cc:2468] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 26.
I20260709 12:25:02.649646 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:02.650632 18333 leader_election.cc:304] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:02.652649 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 31 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:02.631521 17835 rpcz_store.cc:276] 0709 12:25:00.984518 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:02.336161 (+1351643us) service_pool.cc:224] Handling call
0709 12:25:02.336464 (+   303us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:02.336672 (+   208us) raft_consensus.cc:1534] Early marking committed up to index 19
0709 12:25:02.367431 (+ 30759us) log.cc:844] Serialized 14 byte log entry
0709 12:25:02.583358 (+215927us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:02.583430 (+    72us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:25:02.583495 (+    65us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:02.583585 (+    90us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:02.583790 (+   205us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":17683,"prepare.queue_time_us":48173,"prepare.run_cpu_time_us":1966,"prepare.run_wall_time_us":58971,"raft.queue_time_us":18365,"raft.run_cpu_time_us":116,"raft.run_wall_time_us":1941,"spinlock_wait_cycles":57726720,"thread_start_us":197743,"threads_started":1,"wal-append.queue_time_us":229815}
I20260709 12:25:02.603579 27430 leader_election.cc:290] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:02.656931 18333 leader_election.cc:304] T a5732e71b9d54f8786b8c511d69943bf P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:02.658177 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:02.664000 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:02.668977 17836 raft_consensus.cc:2468] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 28.
I20260709 12:25:02.620880 27438 leader_election.cc:290] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:02.664886 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:02.665542 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 27 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:25:02.666132 17652 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:02.668541 27461 raft_consensus.cc:493] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:02.666193 18333 leader_election.cc:304] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:02.676801 27461 raft_consensus.cc:515] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:02.678474 27531 log.cc:927] Time spent T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.083s	user 0.001s	sys 0.000s
I20260709 12:25:02.679041 27461 leader_election.cc:290] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:02.679610 27489 raft_consensus.cc:2749] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:25:02.680977 17833 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35042 (request call id 3450) took 3035 ms. Trace:
I20260709 12:25:02.667742 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:02.664799 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:02.670554 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 30 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:02.670526 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:02.614966 27451 leader_election.cc:290] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:02.673890 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:02.629619 27448 raft_consensus.cc:697] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 27 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:25:02.677085 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:02.652010 27338 raft_consensus.cc:2804] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:25:02.625131 27111 leader_election.cc:290] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:25:02.682852 17652 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:02.683868 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:02.684048 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 27 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:02.682402 17687 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44898 (request call id 1916) took 1733 ms. Trace:
I20260709 12:25:02.681385 17833 rpcz_store.cc:276] 0709 12:24:59.645481 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:59.646024 (+   543us) service_pool.cc:224] Handling call
0709 12:24:59.646312 (+   288us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:00.844079 (+1197767us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:00.844110 (+    31us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:00.844161 (+    51us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:25:00.844191 (+    30us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:02.680762 (+1836571us) spinlock_profiling.cc:243] Waited 1.2 s on lock 0x7b54000aa5d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:25:02.680807 (+    45us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:02.680922 (+   115us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2754178816}
I20260709 12:25:02.685169 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 27 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:02.638715 27324 raft_consensus.cc:493] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:02.685613 17756 raft_consensus.cc:2393] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 30: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:25:02.674405 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:02.681126 27196 leader_election.cc:290] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:02.682989 17834 raft_consensus.cc:2468] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 24.
I20260709 12:25:02.688575 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 29 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:02.656054 27452 raft_consensus.cc:3060] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:02.656646 27509 raft_consensus.cc:3055] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:25:02.691658 17687 rpcz_store.cc:276] 0709 12:25:00.948718 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:01.164212 (+215494us) service_pool.cc:224] Handling call
0709 12:25:01.164484 (+   272us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:01.972779 (+808295us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:01.972824 (+    45us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:01.990818 (+ 17994us) log.cc:844] Serialized 26 byte log entry
0709 12:25:02.463231 (+472413us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:25:02.644288 (+181057us) log.cc:844] Serialized 14 byte log entry
0709 12:25:02.655916 (+ 11628us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:02.656022 (+   106us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:02.682211 (+ 26189us) raft_consensus.cc:1685] finished
0709 12:25:02.682230 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:02.682348 (+   118us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":16799,"prepare.queue_time_us":162588,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":41,"raft.queue_time_us":451140,"raft.run_cpu_time_us":278,"raft.run_wall_time_us":27123,"spinlock_wait_cycles":16768,"thread_start_us":1334566,"threads_started":3,"wal-append.queue_time_us":511680}
I20260709 12:25:02.694904 17759 raft_consensus.cc:3060] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:02.695484 17837 raft_consensus.cc:2368] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for earlier term 18. Current term is 19.
I20260709 12:25:02.690088 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:02.691591 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 27 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:02.696384 17830 raft_consensus.cc:2468] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 26.
I20260709 12:25:02.671228 27442 leader_election.cc:290] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 29 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:02.698632 17652 leader_election.cc:304] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:02.692508 17682 raft_consensus.cc:2468] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 26.
I20260709 12:25:02.655499 27505 raft_consensus.cc:2804] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:25:02.669246 27413 raft_consensus.cc:2749] T a5732e71b9d54f8786b8c511d69943bf P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:25:02.695372 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 26 candidate_status { last_received { term: 24 index: 20 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:02.700989 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:02.652288 27517 raft_consensus.cc:515] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:02.696349 17761 raft_consensus.cc:2468] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 30.
I20260709 12:25:02.707084 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 27 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:25:02.708930 17680 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44898 (request call id 1921) took 1746 ms (client timeout 2005 ms). Trace:
W20260709 12:25:02.709358 17680 rpcz_store.cc:269] 0709 12:25:00.962395 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:00.965960 (+  3565us) service_pool.cc:224] Handling call
0709 12:25:02.708699 (+1742739us) spinlock_profiling.cc:243] Waited 1.28 s on lock 0x7b5400207e54. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:25:02.708835 (+   136us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2950946176}
I20260709 12:25:02.706247 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 30 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:02.669376 27107 consensus_queue.cc:1048] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.028s
I20260709 12:25:02.638331 27525 raft_consensus.cc:515] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:02.669878 26924 consensus_queue.cc:1059] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 26.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.032s
I20260709 12:25:02.712925 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 28 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:02.712168 17756 raft_consensus.cc:2468] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 26.
I20260709 12:25:02.710897 17680 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37402 (request call id 3878) took 1731 ms. Trace:
I20260709 12:25:02.694116 27324 raft_consensus.cc:3060] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:25:02.689574 27338 raft_consensus.cc:493] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:25:02.713958 17680 rpcz_store.cc:276] 0709 12:25:00.993358 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:02.710155 (+1716797us) service_pool.cc:224] Handling call
0709 12:25:02.710475 (+   320us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:02.710633 (+   158us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:02.710656 (+    23us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:02.710696 (+    40us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:25:02.710724 (+    28us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:02.710772 (+    48us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:02.710851 (+    79us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:02.715293 17652 leader_election.cc:304] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:02.716480 17652 leader_election.cc:304] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:02.702713 27505 raft_consensus.cc:493] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:02.721318 27373 raft_consensus.cc:2749] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:25:02.697320 27438 raft_consensus.cc:493] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:02.697779 27509 raft_consensus.cc:740] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 28 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:25:02.730563 17764 raft_consensus.cc:2393] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 30: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:25:02.710886 27455 raft_consensus.cc:493] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:02.714460 27196 raft_consensus.cc:493] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:25:02.729934 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:02.722739 27506 raft_consensus.cc:2749] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:25:02.723623 27338 raft_consensus.cc:3060] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:25:02.721405 27505 raft_consensus.cc:3060] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:02.728112 27430 raft_consensus.cc:2749] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:25:02.735447 17680 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37402 (request call id 3879) took 1741 ms. Trace:
I20260709 12:25:02.738330 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:02.742657 17760 raft_consensus.cc:2468] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 27.
I20260709 12:25:02.743178 17652 leader_election.cc:304] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:02.744922 17680 rpcz_store.cc:276] 0709 12:25:00.993778 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:02.715048 (+1721270us) service_pool.cc:224] Handling call
0709 12:25:02.731836 (+ 16788us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:02.732118 (+   282us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:02.732160 (+    42us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:02.734267 (+  2107us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:25:02.735126 (+   859us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:02.735227 (+   101us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:02.735372 (+   145us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:02.754459 17682 raft_consensus.cc:1240] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 25. Current term is 26. Ops: []
I20260709 12:25:02.755666 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 27 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:02.756443 17756 raft_consensus.cc:2410] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for term 27 because replica has last-logged OpId of term: 24 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 18.
I20260709 12:25:02.755085 17652 leader_election.cc:304] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:25:02.730306 27442 consensus_queue.cc:1048] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.013s
I20260709 12:25:02.760294 17652 leader_election.cc:400] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 18 election: Vote denied by peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837) with higher term. Message: Invalid argument: T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for earlier term 18. Current term is 19.
I20260709 12:25:02.760614 17652 leader_election.cc:403] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 18 election: Cancelling election due to peer responding with higher term
I20260709 12:25:02.757162 27395 raft_consensus.cc:2804] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:25:02.762403 27395 raft_consensus.cc:493] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:25:02.757664 17684 raft_consensus.cc:1240] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 23. Current term is 24. Ops: []
I20260709 12:25:02.755386 17681 raft_consensus.cc:2468] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 24.
I20260709 12:25:02.763108 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 28 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:02.738418 27443 leader_election.cc:290] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:02.763347 17684 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37402 (request call id 3884) took 1756 ms. Trace:
I20260709 12:25:02.764178 17760 raft_consensus.cc:2393] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 28: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:25:02.764751 17680 raft_consensus.cc:1275] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 32 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 29 index: 17. Preceding OpId from leader: term: 32 index: 19. (index mismatch)
I20260709 12:25:02.762820 27395 raft_consensus.cc:3060] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:02.767345 17684 rpcz_store.cc:276] 0709 12:25:01.007468 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:02.757061 (+1749593us) service_pool.cc:224] Handling call
0709 12:25:02.757494 (+   433us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:02.763095 (+  5601us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:02.763295 (+   200us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:02.769380 17680 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37402 (request call id 3883) took 1774 ms. Trace:
I20260709 12:25:02.769749 17680 rpcz_store.cc:276] 0709 12:25:01.006941 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:02.745902 (+1738961us) service_pool.cc:224] Handling call
0709 12:25:02.764491 (+ 18589us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:02.768122 (+  3631us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:02.769338 (+  1216us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:02.761468 17682 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37402 (request call id 3880) took 1767 ms. Trace:
I20260709 12:25:02.770469 17682 rpcz_store.cc:276] 0709 12:25:00.994126 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:02.742100 (+1747974us) service_pool.cc:224] Handling call
0709 12:25:02.754278 (+ 12178us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:02.760913 (+  6635us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:02.761393 (+   480us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:02.732667 27438 raft_consensus.cc:515] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:02.736919 27455 raft_consensus.cc:515] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:02.743901 27397 consensus_queue.cc:1048] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.025s
I20260709 12:25:02.779376 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 33 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:02.784718 17682 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37402 (request call id 3887) took 1762 ms. Trace:
I20260709 12:25:02.788170 17837 raft_consensus.cc:1275] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 33 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 29 index: 19. Preceding OpId from leader: term: 31 index: 20. (index mismatch)
I20260709 12:25:02.765388 27427 raft_consensus.cc:2804] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:25:02.741998 27196 raft_consensus.cc:515] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:02.764452 27522 raft_consensus.cc:493] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:02.791724 17682 rpcz_store.cc:276] 0709 12:25:01.047926 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:02.771147 (+1723221us) service_pool.cc:224] Handling call
0709 12:25:02.784182 (+ 13035us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:02.784376 (+   194us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:02.784404 (+    28us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:02.784459 (+    55us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:25:02.784494 (+    35us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:02.784551 (+    57us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:02.784661 (+   110us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:02.792292 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:25:02.799993 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 122471us
I20260709 12:25:02.738220 27448 consensus_queue.cc:237] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 25.17, Last appended by leader: 17, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:02.775816 27373 raft_consensus.cc:2749] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Leader election lost for term 18. Reason: Vote denied by peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837) with higher term. Message: Invalid argument: T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for earlier term 18. Current term is 19.
I20260709 12:25:02.803825 17831 raft_consensus.cc:2468] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 27.
I20260709 12:25:02.804828 18333 leader_election.cc:304] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:02.812855 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 33 candidate_status { last_received { term: 31 index: 24 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:02.813684 17756 raft_consensus.cc:2468] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 32.
I20260709 12:25:02.816028 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 33 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:02.816100 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:02.816833 17761 raft_consensus.cc:2410] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for term 33 because replica has last-logged OpId of term: 32 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 29 index: 17.
I20260709 12:25:02.817152 27455 leader_election.cc:290] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:02.818383 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:02.820822 27449 leader_election.cc:290] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:25:02.824813 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 1.85976e+06us
I20260709 12:25:02.826618 17759 raft_consensus.cc:2468] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 29.
I20260709 12:25:02.826711 17727 leader_election.cc:304] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:02.829002 27428 raft_consensus.cc:2749] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:25:02.832676 27452 raft_consensus.cc:2749] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:25:02.832962 17727 leader_election.cc:304] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:02.843617 27436 raft_consensus.cc:2749] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:25:02.844120 27451 consensus_queue.cc:1059] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:25:02.788966 27509 consensus_queue.cc:260] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 28.17, Last appended by leader: 17, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:02.808738 27522 raft_consensus.cc:515] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:02.846513 27509 raft_consensus.cc:3060] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:02.797655 27427 raft_consensus.cc:493] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:25:02.848865 27427 raft_consensus.cc:3060] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:02.852315 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 24 candidate_status { last_received { term: 23 index: 21 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:02.853930 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:02.855863 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 27 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:02.858243 17686 raft_consensus.cc:1275] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 25 index: 18. Preceding OpId from leader: term: 26 index: 18. (term mismatch)
I20260709 12:25:02.858642 17686 pending_rounds.cc:85] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3: Aborting all ops after (but not including) 17
I20260709 12:25:02.858907 17686 pending_rounds.cc:107] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3: Aborting uncommitted NO_OP operation due to leader change: 25.18
I20260709 12:25:02.859200 17686 raft_consensus.cc:2889] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:02.869014 27525 leader_election.cc:290] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:02.872376 27196 leader_election.cc:290] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:02.875160 27154 consensus_queue.cc:1059] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 23.16, Next index: 17, Last known committed idx: 14, Time since last communication: 0.000s
I20260709 12:25:02.875394 27369 consensus_queue.cc:1048] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 16, Time since last communication: 0.000s
I20260709 12:25:02.875615 27406 consensus_queue.cc:1059] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:25:02.876380 27008 raft_consensus.cc:3055] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:25:02.876616 27406 raft_consensus.cc:3055] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:25:02.876786 27008 raft_consensus.cc:740] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 23 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:25:02.877164 27406 raft_consensus.cc:740] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 25 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:25:02.875973 27369 consensus_queue.cc:1243] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer 0da169ff82674dc1947055ce43de68a3 log is divergent from this leader: its last log entry 29.17 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 16
I20260709 12:25:02.878077 27008 consensus_queue.cc:260] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 23.16, Last appended by leader: 16, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:02.882441 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:02.836961 27385 leader_election.cc:290] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 24 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:02.884740 27517 leader_election.cc:290] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 30 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:02.799472 27443 consensus_queue.cc:1048] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 19, Time since last communication: 0.088s
W20260709 12:25:02.887506 17727 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:02.892537 27438 leader_election.cc:290] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:02.892994 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
W20260709 12:25:02.944098 17681 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44898 (request call id 1915) took 1997 ms (client timeout 2551 ms). Trace:
I20260709 12:25:02.944548 17830 raft_consensus.cc:2368] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for earlier term 23. Current term is 24.
W20260709 12:25:02.944491 17681 rpcz_store.cc:269] 0709 12:25:00.947163 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:00.947323 (+   160us) service_pool.cc:224] Handling call
0709 12:25:02.943912 (+1996589us) spinlock_profiling.cc:243] Waited 1.77 s on lock 0x7b54000c4754. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:25:02.944040 (+   128us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4078546688}
I20260709 12:25:02.946063 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:02.948015 27406 consensus_queue.cc:260] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 25.16, Last appended by leader: 16, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:02.948676 17682 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37402 (request call id 3889) took 1893 ms. Trace:
I20260709 12:25:02.949110 17727 leader_election.cc:400] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 23 election: Vote denied by peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837) with higher term. Message: Invalid argument: T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for earlier term 23. Current term is 24.
I20260709 12:25:02.949043 17682 rpcz_store.cc:276] 0709 12:25:01.055559 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:02.804796 (+1749237us) service_pool.cc:224] Handling call
0709 12:25:02.948185 (+143389us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:02.948361 (+   176us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:02.948386 (+    25us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:02.948427 (+    41us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:25:02.948458 (+    31us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:02.948522 (+    64us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:02.948620 (+    98us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:02.949812 17727 leader_election.cc:403] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 23 election: Cancelling election due to peer responding with higher term
I20260709 12:25:02.955811 27475 raft_consensus.cc:3060] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:02.962869 27505 raft_consensus.cc:515] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:02.965250 17652 leader_election.cc:304] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:25:02.966626 17652 leader_election.cc:304] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:02.970923 18334 leader_election.cc:304] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:25:02.971248 27197 raft_consensus.cc:2804] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:25:02.971674 27197 raft_consensus.cc:493] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:25:02.971863 27489 raft_consensus.cc:2804] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:25:02.972128 27197 raft_consensus.cc:3060] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:02.972281 27489 raft_consensus.cc:493] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:02.972690 27489 raft_consensus.cc:3060] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:02.973855 27443 raft_consensus.cc:2804] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:25:02.974373 27443 raft_consensus.cc:493] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:25:02.974826 27443 raft_consensus.cc:3060] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:02.893296 27359 raft_consensus.cc:3055] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:25:02.895135 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:02.979029 27359 raft_consensus.cc:740] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:25:02.979243 17831 raft_consensus.cc:2468] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 30.
I20260709 12:25:02.879611 27008 raft_consensus.cc:3060] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:02.979981 27359 consensus_queue.cc:260] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 27.19, Last appended by leader: 19, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:02.981283 27359 raft_consensus.cc:3060] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:02.982702 17727 leader_election.cc:304] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:25:02.983927 27428 raft_consensus.cc:2804] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:25:02.984288 27428 raft_consensus.cc:493] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:25:02.984359 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:02.984717 27428 raft_consensus.cc:3060] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:02.984951 17831 raft_consensus.cc:3060] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:02.949280 27406 raft_consensus.cc:3060] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Advancing to term 26
W20260709 12:25:02.896075 27537 log.cc:927] Time spent T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.063s	user 0.000s	sys 0.000s
I20260709 12:25:02.896152 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:25:02.988361 17730 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:02.897719 17836 raft_consensus.cc:1275] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 26 index: 20. Preceding OpId from leader: term: 30 index: 22. (index mismatch)
I20260709 12:25:02.896488 17683 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44898 (request call id 1918) took 1945 ms. Trace:
I20260709 12:25:02.988965 17683 rpcz_store.cc:276] 0709 12:25:00.950924 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:02.102749 (+1151825us) service_pool.cc:224] Handling call
0709 12:25:02.103082 (+   333us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:02.530109 (+427027us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:02.896220 (+366111us) spinlock_profiling.cc:243] Waited 301 ms on lock 0x7b54000cf154. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:25:02.896431 (+   211us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":693118336}
I20260709 12:25:02.989614 17838 raft_consensus.cc:2468] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 30.
I20260709 12:25:02.989897 17836 raft_consensus.cc:1275] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 23 index: 17. Preceding OpId from leader: term: 25 index: 18. (index mismatch)
W20260709 12:25:02.898706 17804 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:02.990739 27451 consensus_queue.cc:1048] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 20, Time since last communication: 0.000s
I20260709 12:25:02.898314 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 27 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:02.992195 17758 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 26.
I20260709 12:25:02.992588 17838 raft_consensus.cc:1275] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 17 index: 11. Preceding OpId from leader: term: 19 index: 12. (index mismatch)
I20260709 12:25:02.899442 27270 leader_election.cc:290] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:02.899334 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:02.994086 27270 raft_consensus.cc:493] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:02.900468 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:02.994949 27522 leader_election.cc:290] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:25:02.903249 17832 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35042 (request call id 3459) took 1936 ms (client timeout 1743 ms). Trace:
I20260709 12:25:02.902443 18366 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59902 (request call id 2256) took 2025 ms. Trace:
I20260709 12:25:02.904094 17686 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37402 (request call id 3890) took 1803 ms. Trace:
I20260709 12:25:02.947656 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:02.905211 27517 raft_consensus.cc:493] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:02.996076 17686 rpcz_store.cc:276] 0709 12:25:01.100967 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:02.857384 (+1756417us) service_pool.cc:224] Handling call
0709 12:25:02.857796 (+   412us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:02.859714 (+  1918us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:02.904040 (+ 44326us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:02.997112 27517 raft_consensus.cc:515] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:02.906499 27525 raft_consensus.cc:493] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:25:02.893304 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 29 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:02.998133 27525 raft_consensus.cc:515] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:02.998934 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 27 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:02.999651 17837 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 26.
I20260709 12:25:02.995975 18366 rpcz_store.cc:276] 0709 12:25:00.877019 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:00.877665 (+   646us) service_pool.cc:224] Handling call
0709 12:25:00.877947 (+   282us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:02.472348 (+1594401us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:02.902216 (+429868us) spinlock_profiling.cc:243] Waited 1.49 s on lock 0x7b540047de54. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:25:02.902387 (+   171us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3429405568}
I20260709 12:25:03.000380 17830 raft_consensus.cc:2393] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 30: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:25:03.000557 27525 leader_election.cc:290] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:25:02.905901 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 101143us
W20260709 12:25:02.910025 17829 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35040 (request call id 3646) took 1940 ms (client timeout 2054 ms). Trace:
W20260709 12:25:02.912333 27540 log.cc:927] Time spent T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.074s	user 0.001s	sys 0.000s
W20260709 12:25:03.001571 17829 rpcz_store.cc:269] 0709 12:25:00.969485 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:02.144368 (+1174883us) service_pool.cc:224] Handling call
0709 12:25:02.909850 (+765482us) spinlock_profiling.cc:243] Waited 146 ms on lock 0x7b5400266454. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:25:02.909974 (+   124us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":336058624}
I20260709 12:25:03.002313 18333 leader_election.cc:304] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:03.003657 26818 consensus_queue.cc:1059] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 29.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.000s
I20260709 12:25:02.918860 27395 raft_consensus.cc:515] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:02.920665 17685 raft_consensus.cc:1240] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 23. Current term is 24. Ops: []
I20260709 12:25:03.004498 27463 raft_consensus.cc:2749] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:25:03.005172 27415 raft_consensus.cc:3055] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 31 LEADER]: Stepping down as leader of term 31
I20260709 12:25:03.005677 27415 raft_consensus.cc:740] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 31 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:25:02.924451 17801 leader_election.cc:304] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:25:02.909116 17834 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35042 (request call id 3457) took 1957 ms (client timeout 1581 ms). Trace:
I20260709 12:25:03.006795 17685 raft_consensus.cc:1275] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 23 index: 18. Preceding OpId from leader: term: 26 index: 19. (index mismatch)
W20260709 12:25:03.007153 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 82772.8us
W20260709 12:25:03.007007 17834 rpcz_store.cc:269] 0709 12:25:00.951782 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:00.951926 (+   144us) service_pool.cc:224] Handling call
0709 12:25:02.908941 (+1957015us) spinlock_profiling.cc:243] Waited 1.68 s on lock 0x7b54003359d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:25:02.909049 (+   108us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3868571392}
W20260709 12:25:02.927614 27495 log.cc:927] Time spent T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.884s	user 0.000s	sys 0.002s
I20260709 12:25:02.937891 27394 leader_election.cc:290] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:03.010244 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:02.943463 27245 raft_consensus.cc:493] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:02.988294 17759 raft_consensus.cc:2468] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 26.
I20260709 12:25:02.900962 26924 raft_consensus.cc:493] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:25:03.022976 26924 raft_consensus.cc:515] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:03.024677 17764 raft_consensus.cc:1275] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 26 index: 18. Preceding OpId from leader: term: 28 index: 20. (index mismatch)
I20260709 12:25:03.025172 26924 leader_election.cc:290] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:03.017158 27413 raft_consensus.cc:493] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:03.017884 27510 raft_consensus.cc:2804] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:25:03.026257 27510 raft_consensus.cc:493] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
W20260709 12:25:02.901592 17762 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36572 (request call id 1724) took 2013 ms (client timeout 1515 ms). Trace:
W20260709 12:25:03.026787 17762 rpcz_store.cc:269] 0709 12:25:00.888687 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:00.899240 (+ 10553us) service_pool.cc:224] Handling call
0709 12:25:02.901441 (+2002201us) spinlock_profiling.cc:243] Waited 80.7 ms on lock 0x7b540019f5d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:25:02.901531 (+    90us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":185676672}
I20260709 12:25:03.006520 27415 consensus_queue.cc:260] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 31.21, Last appended by leader: 21, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:02.995117 17835 raft_consensus.cc:2468] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 28.
I20260709 12:25:03.027928 27415 raft_consensus.cc:3060] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:25:03.028347 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:02.904533 17689 raft_consensus.cc:1275] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 27 index: 16. Preceding OpId from leader: term: 29 index: 17. (index mismatch)
I20260709 12:25:02.994611 27270 raft_consensus.cc:515] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:02.905021 27519 raft_consensus.cc:3060] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:03.007351 18333 leader_election.cc:304] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:03.007934 17801 leader_election.cc:304] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:02.934695 27189 leader_election.cc:290] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 30 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:03.010308 17652 leader_election.cc:304] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:25:03.006803 17831 raft_consensus.cc:2468] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 25.
I20260709 12:25:03.011925 17758 raft_consensus.cc:1240] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 29. Current term is 30. Ops: []
W20260709 12:25:03.069667 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 61829.1us
I20260709 12:25:02.999538 27517 leader_election.cc:290] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:02.942533 27338 raft_consensus.cc:515] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:03.025938 27413 raft_consensus.cc:515] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:02.995534 17832 rpcz_store.cc:269] 0709 12:25:00.966506 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:01.818661 (+852155us) service_pool.cc:224] Handling call
0709 12:25:02.903092 (+1084431us) spinlock_profiling.cc:243] Waited 244 ms on lock 0x7b54001586d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:25:02.903184 (+    92us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":560772992}
I20260709 12:25:03.082039 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:03.034307 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:03.035435 18362 raft_consensus.cc:1240] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 23. Current term is 24. Ops: []
I20260709 12:25:03.037045 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:03.082847 18361 raft_consensus.cc:2468] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 26.
I20260709 12:25:03.083482 18364 raft_consensus.cc:2368] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for earlier term 23. Current term is 24.
I20260709 12:25:03.038578 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 29 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:03.038743 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 24 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
W20260709 12:25:03.065445 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 58156.2us
I20260709 12:25:03.085347 18363 raft_consensus.cc:2468] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 28.
I20260709 12:25:03.086458 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 26 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:03.036221 27510 raft_consensus.cc:3060] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:03.089905 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 25 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:03.090344 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:03.090797 17758 raft_consensus.cc:2410] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for term 25 because replica has last-logged OpId of term: 24 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 14.
I20260709 12:25:03.091499 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:03.065515 17685 raft_consensus.cc:1275] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 24 index: 15. Preceding OpId from leader: term: 26 index: 16. (index mismatch)
I20260709 12:25:03.068296 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:03.070355 17801 leader_election.cc:304] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:25:03.071240 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 61006.5us
I20260709 12:25:03.097796 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 24 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:03.098755 17762 raft_consensus.cc:2410] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for term 24 because replica has last-logged OpId of term: 23 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 16.
I20260709 12:25:03.031262 17727 leader_election.cc:304] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:03.075248 17689 pending_rounds.cc:85] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3: Aborting all ops after (but not including) 16
I20260709 12:25:03.102950 17689 pending_rounds.cc:107] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3: Aborting uncommitted NO_OP operation due to leader change: 29.17
I20260709 12:25:03.103219 17689 raft_consensus.cc:2889] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 32 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:03.076315 18374 raft_consensus.cc:1275] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 26 index: 20. Preceding OpId from leader: term: 30 index: 22. (index mismatch)
I20260709 12:25:03.079619 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:03.083982 27522 raft_consensus.cc:2804] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:25:03.093943 27413 leader_election.cc:290] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:25:03.118795 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:25:03.119695 17685 raft_consensus.cc:2393] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 30: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:25:03.094094 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:03.100138 17652 leader_election.cc:304] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:03.102337 17759 raft_consensus.cc:1240] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 28. Current term is 29. Ops: [26.19-28.20]
I20260709 12:25:03.104153 18365 raft_consensus.cc:3060] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:03.105125 17833 raft_consensus.cc:3060] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:03.080569 27111 raft_consensus.cc:2804] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:25:03.105557 18371 raft_consensus.cc:1275] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 23 index: 17. Preceding OpId from leader: term: 25 index: 18. (index mismatch)
I20260709 12:25:03.073544 17684 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37402 (request call id 3885) took 2065 ms. Trace:
I20260709 12:25:03.107723 17801 leader_election.cc:304] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:25:03.097769 17682 raft_consensus.cc:2468] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 25.
I20260709 12:25:03.111603 27450 raft_consensus.cc:2804] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:25:03.111726 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:03.127168 17684 rpcz_store.cc:276] 0709 12:25:01.007795 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:02.768715 (+1760920us) service_pool.cc:224] Handling call
0709 12:25:02.769015 (+   300us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:02.769195 (+   180us) raft_consensus.cc:1534] Early marking committed up to index 15
0709 12:25:02.774576 (+  5381us) log.cc:844] Serialized 14 byte log entry
0709 12:25:03.073105 (+298529us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:03.073172 (+    67us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:25:03.073244 (+    72us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:03.073327 (+    83us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:03.073476 (+   149us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":11156,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":36,"raft.queue_time_us":823,"raft.run_cpu_time_us":126,"raft.run_wall_time_us":29711,"thread_start_us":297088,"threads_started":1}
I20260709 12:25:03.112164 17763 raft_consensus.cc:1275] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 24 index: 18. Preceding OpId from leader: term: 26 index: 19. (index mismatch)
I20260709 12:25:03.080569 27542 raft_consensus.cc:493] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:03.133160 17833 raft_consensus.cc:2468] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 20.
I20260709 12:25:03.055321 27446 raft_consensus.cc:493] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:03.030943 27436 consensus_queue.cc:1048] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.080s
I20260709 12:25:03.077648 27463 raft_consensus.cc:3060] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Advancing to term 30
W20260709 12:25:03.105110 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 73930us
I20260709 12:25:03.031394 27451 consensus_queue.cc:1048] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.076s
I20260709 12:25:03.125624 18367 raft_consensus.cc:3060] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:03.036427 27266 raft_consensus.cc:515] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:03.109762 17762 raft_consensus.cc:1240] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 23. Current term is 24. Ops: [23.16-23.16]
I20260709 12:25:03.138585 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:03.072188 27524 consensus_queue.cc:1048] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.058s
I20260709 12:25:03.129284 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:03.143916 17684 raft_consensus.cc:2468] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 28.
I20260709 12:25:03.114238 27370 raft_consensus.cc:2749] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:25:03.137760 17801 leader_election.cc:304] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:03.036228 27245 raft_consensus.cc:515] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:03.054383 27324 raft_consensus.cc:515] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 33 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:03.135677 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:25:03.146629 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:03.128720 17652 leader_election.cc:304] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:25:03.132040 18333 connection.cc:256] RPC call timeout handler was delayed by -1.8297308930000327s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1515s for a response
W20260709 12:25:03.149863 18333 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:03.149014 17801 leader_election.cc:304] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:03.080412 27428 raft_consensus.cc:515] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:03.152905 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:03.153838 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:03.163785 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:03.175431 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 31 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:03.176162 18368 raft_consensus.cc:2468] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 30.
I20260709 12:25:03.171952 26924 raft_consensus.cc:493] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:25:03.206277 18374 raft_consensus.cc:2393] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 28: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:25:03.172061 27431 raft_consensus.cc:2804] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:25:03.172178 17801 leader_election.cc:304] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:25:03.303601 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 131500us
I20260709 12:25:03.303819 17728 leader_election.cc:304] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:03.126178 27111 raft_consensus.cc:493] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:03.126565 27505 leader_election.cc:290] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 28 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:03.150290 17652 leader_election.cc:304] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:03.305198 27111 raft_consensus.cc:3060] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:03.304905 26818 consensus_queue.cc:1059] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 26.19, Next index: 20, Last known committed idx: 17, Time since last communication: 0.001s
W20260709 12:25:03.306165 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 155953us
I20260709 12:25:03.306985 27154 raft_consensus.cc:2804] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:25:03.307229 17652 leader_election.cc:304] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:03.307499 27154 raft_consensus.cc:493] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:25:03.179642 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 24 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:03.308041 27154 raft_consensus.cc:3060] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:03.183049 18367 raft_consensus.cc:2468] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 20.
I20260709 12:25:03.308723 27441 raft_consensus.cc:2804] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:25:03.148443 27394 raft_consensus.cc:2804] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:25:03.132419 27450 raft_consensus.cc:493] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:03.205159 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:03.132170 27432 raft_consensus.cc:2749] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260709 12:25:03.205813 17680 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37402 (request call id 3886) took 2194 ms. Trace:
I20260709 12:25:03.244881 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 27 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:03.142895 27413 raft_consensus.cc:493] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:25:03.298870 27395 leader_election.cc:290] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 28 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:03.140476 27542 raft_consensus.cc:515] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:03.173032 27196 raft_consensus.cc:2804] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:25:03.243841 27369 consensus_queue.cc:1048] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.165s
I20260709 12:25:03.316291 27394 raft_consensus.cc:493] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:25:03.396842 27441 raft_consensus.cc:697] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 29 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:25:03.332163 27413 raft_consensus.cc:515] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:03.416497 27441 consensus_queue.cc:237] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 26.19, Last appended by leader: 19, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:03.418141 27413 leader_election.cc:290] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:03.339394 18365 raft_consensus.cc:2468] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 25.
I20260709 12:25:03.420396 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 27 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:03.421873 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:03.422749 18365 raft_consensus.cc:2393] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 24: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:25:03.340080 27450 raft_consensus.cc:3060] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:03.423573 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 28 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:03.344605 18333 leader_election.cc:304] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
W20260709 12:25:03.431370 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 125159us
I20260709 12:25:03.432770 18333 leader_election.cc:304] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:03.327145 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:03.433911 18333 leader_election.cc:304] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:03.340684 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 28 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:03.434762 17682 raft_consensus.cc:2393] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 28: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
W20260709 12:25:03.347682 17803 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:03.433748 18362 raft_consensus.cc:2468] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 24.
I20260709 12:25:03.437202 18333 leader_election.cc:304] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:03.400930 17764 raft_consensus.cc:2468] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 26.
I20260709 12:25:03.301916 17687 raft_consensus.cc:2468] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 25.
I20260709 12:25:03.309792 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 24 candidate_status { last_received { term: 21 index: 20 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:03.441807 18367 raft_consensus.cc:2410] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 24 because replica has last-logged OpId of term: 23 index: 21, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 20.
I20260709 12:25:03.184069 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:03.310614 17652 leader_election.cc:304] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:25:03.442816 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 132273us
I20260709 12:25:03.444005 17652 leader_election.cc:304] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:03.311275 17728 leader_election.cc:304] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:03.311578 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 27 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:03.445859 17684 raft_consensus.cc:2468] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 26.
W20260709 12:25:03.446069 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 141083us
I20260709 12:25:03.313938 17686 raft_consensus.cc:1240] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 26. Current term is 27. Ops: []
I20260709 12:25:03.176375 27062 raft_consensus.cc:2804] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 32 FOLLOWER]: Leader election won for term 32
I20260709 12:25:03.316912 17680 rpcz_store.cc:276] 0709 12:25:01.011873 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:02.770311 (+1758438us) service_pool.cc:224] Handling call
0709 12:25:02.782786 (+ 12475us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:02.783022 (+   236us) raft_consensus.cc:1534] Early marking committed up to index 15
0709 12:25:02.783603 (+   581us) log.cc:844] Serialized 14 byte log entry
0709 12:25:03.205461 (+421858us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:03.205516 (+    55us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:25:03.205563 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:03.205624 (+    61us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:03.205755 (+   131us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":10660,"prepare.queue_time_us":146,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":33,"raft.queue_time_us":823,"raft.run_cpu_time_us":165,"raft.run_wall_time_us":67529,"thread_start_us":421463,"threads_started":1}
I20260709 12:25:03.318053 26924 raft_consensus.cc:515] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:03.145363 27446 raft_consensus.cc:515] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:03.438369 17682 raft_consensus.cc:1240] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 28 LEADER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 27. Current term is 28. Ops: []
I20260709 12:25:03.447358 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:03.448706 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 25 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:03.302793 27544 raft_consensus.cc:493] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:03.319192 27525 raft_consensus.cc:2749] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
W20260709 12:25:03.322821 17578 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:37172 (request call id 260) took 26983 ms (client timeout 15000 ms). Trace:
I20260709 12:25:03.446768 17728 leader_election.cc:304] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:03.504702 18367 raft_consensus.cc:2468] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 24.
I20260709 12:25:03.410894 27394 raft_consensus.cc:3060] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:03.370874 27428 leader_election.cc:290] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 31 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:25:03.439105 27452 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.533s	user 0.010s	sys 0.018s
W20260709 12:25:03.469485 17804 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:03.508982 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 62289.4us
I20260709 12:25:03.322975 17585 catalog_manager.cc:5697] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b reported cstate change: term changed from 20 to 23, leader changed from e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66) to e063541542a94969898868bf6eb5ce1b (127.16.29.67). New cstate: current_term: 23 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: HEALTHY } } }
I20260709 12:25:03.453610 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:03.509352 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 27 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:03.528959 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 27 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:03.453588 27128 raft_consensus.cc:2749] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:25:03.517807 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:03.408324 27196 raft_consensus.cc:493] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:03.594903 27196 raft_consensus.cc:3060] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:25:03.358734 27338 leader_election.cc:290] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 19 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:03.596786 27338 raft_consensus.cc:2804] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:25:03.597311 27338 raft_consensus.cc:493] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:03.597764 27338 raft_consensus.cc:3060] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Advancing to term 31
W20260709 12:25:03.603135 17829 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.593s	user 0.018s	sys 0.026s
W20260709 12:25:03.603819 17829 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.594s	user 0.018s	sys 0.026s
I20260709 12:25:03.312951 27431 raft_consensus.cc:493] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
W20260709 12:25:03.406334 27427 consensus_meta.cc:287] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.557s	user 0.009s	sys 0.000s
I20260709 12:25:03.378573 27542 leader_election.cc:290] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:03.389385 27266 leader_election.cc:290] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 28 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:25:03.393656 27324 leader_election.cc:290] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 33 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:25:03.365278 27270 leader_election.cc:290] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:03.408124 27522 raft_consensus.cc:493] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
W20260709 12:25:03.525650 17836 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.534s	user 0.017s	sys 0.004s
I20260709 12:25:03.529042 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:03.401460 27359 raft_consensus.cc:3055] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:25:03.429210 27413 raft_consensus.cc:2749] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:25:03.530611 17728 leader_election.cc:304] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:25:03.648272 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 117744us
W20260709 12:25:03.530714 27452 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.621s	user 0.012s	sys 0.023s
I20260709 12:25:03.575316 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:03.664671 18362 raft_consensus.cc:2468] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 21.
I20260709 12:25:03.658366 17802 leader_election.cc:304] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:25:03.581034 26924 leader_election.cc:290] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:03.585750 27446 leader_election.cc:290] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:03.620143 27431 raft_consensus.cc:3060] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:03.458889 27517 raft_consensus.cc:2749] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
I20260709 12:25:03.301280 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:03.459218 27438 raft_consensus.cc:2804] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:25:03.356976 27197 raft_consensus.cc:515] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:03.642244 27522 raft_consensus.cc:3060] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:03.394802 27489 raft_consensus.cc:515] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:03.414847 27510 raft_consensus.cc:515] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:03.613373 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 28 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:03.746405 17762 raft_consensus.cc:3060] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:03.614073 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:03.614617 17763 raft_consensus.cc:1240] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 30. Current term is 31. Ops: []
I20260709 12:25:03.616526 17585 catalog_manager.cc:5697] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b reported cstate change: term changed from 16 to 23, leader changed from 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) to e063541542a94969898868bf6eb5ce1b (127.16.29.67). New cstate: current_term: 23 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: HEALTHY } } }
I20260709 12:25:03.367933 27154 raft_consensus.cc:515] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Starting leader election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:03.754966 17755 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.645s	user 0.040s	sys 0.087s
I20260709 12:25:03.756403 17585 catalog_manager.cc:5697] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b reported cstate change: term changed from 14 to 20, leader changed from 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) to e063541542a94969898868bf6eb5ce1b (127.16.29.67). New cstate: current_term: 20 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20260709 12:25:03.757625 17755 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.648s	user 0.041s	sys 0.087s
I20260709 12:25:03.622974 27427 raft_consensus.cc:515] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:03.638630 18367 raft_consensus.cc:2468] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 26.
W20260709 12:25:03.639354 17836 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.648s	user 0.018s	sys 0.004s
I20260709 12:25:03.519004 27062 raft_consensus.cc:697] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 32 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:25:03.496716 27442 consensus_queue.cc:1059] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 29.19, Next index: 20, Last known committed idx: 18, Time since last communication: 0.111s
W20260709 12:25:03.530229 27307 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.567s	user 0.012s	sys 0.006s
I20260709 12:25:03.604400 27428 raft_consensus.cc:2804] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:25:03.704504 27436 raft_consensus.cc:2804] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:25:03.696156 27324 raft_consensus.cc:3060] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:03.499173 27432 consensus_queue.cc:1048] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 18, Time since last communication: 0.124s
I20260709 12:25:03.721638 27270 raft_consensus.cc:2804] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:25:03.724395 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 33 candidate_status { last_received { term: 31 index: 24 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:03.790333 18362 raft_consensus.cc:2393] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 33: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:25:03.784391 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:03.791322 18367 raft_consensus.cc:2468] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 25.
W20260709 12:25:03.644685 27443 consensus_meta.cc:287] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.669s	user 0.013s	sys 0.000s
I20260709 12:25:03.792603 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:03.645606 27542 raft_consensus.cc:2749] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:25:03.640573 27359 raft_consensus.cc:740] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:25:03.723591 27449 raft_consensus.cc:493] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:03.760565 17757 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.690s	user 0.033s	sys 0.026s
W20260709 12:25:03.738360 27475 consensus_meta.cc:287] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.782s	user 0.009s	sys 0.003s
I20260709 12:25:03.763203 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 29 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:03.764166 18368 raft_consensus.cc:2368] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 24. Current term is 25.
W20260709 12:25:03.778648 17837 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.772s	user 0.017s	sys 0.045s
I20260709 12:25:03.634904 17832 raft_consensus.cc:2468] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 28.
W20260709 12:25:03.752386 27461 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.834s	user 0.030s	sys 0.097s
I20260709 12:25:03.736505 27438 raft_consensus.cc:493] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:25:03.794019 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:03.524169 27544 raft_consensus.cc:515] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:03.786588 17838 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.650s	user 0.015s	sys 0.004s
I20260709 12:25:03.827548 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:25:03.780467 17585 catalog_manager.cc:5697] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b reported cstate change: term changed from 17 to 24, leader changed from e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66) to e063541542a94969898868bf6eb5ce1b (127.16.29.67). New cstate: current_term: 24 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: HEALTHY } } }
I20260709 12:25:03.949047 17686 raft_consensus.cc:1275] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 17 index: 11. Preceding OpId from leader: term: 19 index: 12. (index mismatch)
W20260709 12:25:03.952288 17838 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.816s	user 0.020s	sys 0.004s
I20260709 12:25:03.836892 27270 raft_consensus.cc:493] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:25:03.956070 18334 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.770s	user 0.045s	sys 0.084s
W20260709 12:25:03.956532 18334 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.771s	user 0.045s	sys 0.084s
I20260709 12:25:03.903038 17688 raft_consensus.cc:2393] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 29: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
W20260709 12:25:03.785113 27415 consensus_meta.cc:287] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.752s	user 0.008s	sys 0.003s
W20260709 12:25:03.785617 27307 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.827s	user 0.013s	sys 0.006s
I20260709 12:25:03.787446 27428 raft_consensus.cc:493] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:25:03.783551 27448 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.954s	user 0.005s	sys 0.059s
I20260709 12:25:03.989784 18365 raft_consensus.cc:3060] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Advancing to term 28
W20260709 12:25:03.788868 27107 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.884s	user 0.017s	sys 0.046s
I20260709 12:25:04.004004 18365 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 28.
I20260709 12:25:04.004478 27154 leader_election.cc:290] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 27 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:25:04.019917 27107 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.094s	user 0.017s	sys 0.055s
I20260709 12:25:04.025002 27510 leader_election.cc:290] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 25 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:04.027362 27197 leader_election.cc:290] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 31 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:04.030577 17729 leader_election.cc:304] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:25:04.032207 27559 raft_consensus.cc:2749] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260709 12:25:04.033102 27427 leader_election.cc:290] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 29 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:04.034184 17729 leader_election.cc:304] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:25:04.035656 27559 raft_consensus.cc:2804] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Leader pre-election won for term 29
W20260709 12:25:03.508639 17578 rpcz_store.cc:269] 0709 12:24:36.224472 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:45.831847 (+9607375us) service_pool.cc:224] Handling call
0709 12:25:03.162704 (+17330857us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:24:55.677063 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:24:55.677607 (+   544us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:24:55.677655 (+    48us) write_op.cc:435] Acquired schema lock
0709 12:24:55.677674 (+    19us) tablet.cc:662] Decoding operations
0709 12:24:55.724490 (+ 46816us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:24:55.724572 (+    82us) write_op.cc:641] Partition lock acquired for write op
0709 12:24:55.724611 (+    39us) tablet.cc:685] Acquiring locks for 23 operations
0709 12:24:55.726462 (+  1851us) tablet.cc:701] Row locks acquired
0709 12:24:55.726483 (+    21us) write_op.cc:260] PREPARE: finished
0709 12:24:55.726631 (+   148us) write_op.cc:270] Start()
0709 12:24:55.726738 (+   107us) write_op.cc:276] Timestamp: P: 1783599895726581 usec, L: 0
0709 12:24:55.726758 (+    20us) op_driver.cc:348] REPLICATION: starting
0709 12:24:55.727552 (+   794us) log.cc:844] Serialized 9018 byte log entry
0709 12:24:55.873842 (+146290us) op_driver.cc:464] REPLICATION: finished
0709 12:24:56.052060 (+178218us) write_op.cc:301] APPLY: starting
0709 12:24:56.052209 (+   149us) tablet.cc:1367] starting BulkCheckPresence
0709 12:24:56.088224 (+ 36015us) spinlock_profiling.cc:243] Waited 91.7 ms on lock 0x7b540010d454. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a61d32a 00007f7b4a5a2f14 00007f7b4a5a2ec3 00007f7b4a5a2e38 00007f7b4a5a2df3 00007f7b4a5a20dc 00007f7b4a5cbe16 00007f7b4a5b3cbb 00007f7b4a59e39d 00007f7b4a5a1d75
0709 12:24:56.379674 (+291450us) tablet.cc:1370] finished BulkCheckPresence
0709 12:24:56.379727 (+    53us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:24:56.617771 (+238044us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:24:56.618166 (+   395us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=46,key_file_lookups=46,delta_file_lookups=0,mrs_lookups=0
0709 12:24:56.618226 (+    60us) write_op.cc:312] APPLY: finished
0709 12:24:56.618800 (+   574us) log.cc:844] Serialized 204 byte log entry
0709 12:24:56.619002 (+   202us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:24:56.619518 (+   516us) write_op.cc:454] Released schema lock
0709 12:24:56.619588 (+    70us) write_op.cc:341] FINISH: Updating metrics
0709 12:24:56.774777 (+155189us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:24:56.779359 (+  4582us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":4152,"reported_tablets":45,"child_traces":[["op",{"apply.queue_time_us":120778,"apply.run_cpu_time_us":40305,"apply.run_wall_time_us":586972,"cfile_cache_hit":94,"cfile_cache_hit_bytes":100359,"num_ops":23,"prepare.queue_time_us":210413,"prepare.run_cpu_time_us":7932,"prepare.run_wall_time_us":144834,"raft.queue_time_us":788,"raft.run_cpu_time_us":16465,"raft.run_wall_time_us":356075,"replication_time_us":146565,"spinlock_wait_cycles":210850688,"thread_start_us":413220,"threads_started":4,"wal-append.queue_time_us":1984,"wal-append.run_cpu_time_us":4256,"wal-append.run_wall_time_us":1949663}]]}
I20260709 12:25:04.036164 27559 raft_consensus.cc:493] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:25:04.036712 27559 raft_consensus.cc:3060] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Advancing to term 29
W20260709 12:25:04.037760 17578 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35966 (request call id 360) took 19021 ms (client timeout 15000 ms). Trace:
I20260709 12:25:03.969534 17585 catalog_manager.cc:5697] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b reported cstate change: term changed from 22 to 24. New cstate: current_term: 24 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
W20260709 12:25:03.893595 17757 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.823s	user 0.037s	sys 0.032s
I20260709 12:25:03.958334 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:03.959959 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:25:03.956876 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 771090us
W20260709 12:25:03.877668 17803 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:03.903605 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:25:03.972199 17727 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.729s	user 0.004s	sys 0.029s
W20260709 12:25:03.893254 27008 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.666s	user 0.034s	sys 0.035s
I20260709 12:25:03.981566 18364 raft_consensus.cc:2468] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 32.
W20260709 12:25:03.896068 27461 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.978s	user 0.031s	sys 0.098s
I20260709 12:25:03.893297 27442 raft_consensus.cc:493] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
W20260709 12:25:03.894667 17837 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.888s	user 0.018s	sys 0.055s
I20260709 12:25:03.992472 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 29 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:03.836144 27436 raft_consensus.cc:493] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:03.900797 27369 raft_consensus.cc:493] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:03.899638 27438 raft_consensus.cc:3060] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Advancing to term 26
W20260709 12:25:03.977099 27363 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.007s	user 0.041s	sys 0.105s
I20260709 12:25:04.035667 27489 leader_election.cc:290] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 25 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:04.019183 27544 leader_election.cc:290] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:25:04.040763 17727 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.797s	user 0.005s	sys 0.029s
W20260709 12:25:04.090018 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 946630us
W20260709 12:25:04.038319 17578 rpcz_store.cc:269] 0709 12:24:45.016583 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:04.037475 (+19020892us) service_pool.cc:211] Skipping call since client already timed out
0709 12:25:04.037658 (+   183us) inbound_call.cc:177] Queueing failure response
Metrics: {}
W20260709 12:25:04.042737 18334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:03.960909 27270 raft_consensus.cc:3060] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:04.063623 17729 leader_election.cc:400] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 23 election: Vote denied by peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577) with higher term. Message: Invalid argument: T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for earlier term 23. Current term is 24.
I20260709 12:25:03.957823 27475 raft_consensus.cc:2749] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Leader election lost for term 23. Reason: Vote denied by peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837) with higher term. Message: Invalid argument: T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for earlier term 23. Current term is 24.
W20260709 12:25:04.075137 27406 consensus_meta.cc:287] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 1.088s	user 0.009s	sys 0.003s
W20260709 12:25:04.085903 27363 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.131s	user 0.041s	sys 0.106s
I20260709 12:25:04.064373 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 25 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:04.041113 17757 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38776 (request call id 3318) took 1027 ms. Trace:
I20260709 12:25:04.118563 18368 raft_consensus.cc:2393] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 25: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:25:04.046090 18367 raft_consensus.cc:2393] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 26: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:25:04.045580 18374 raft_consensus.cc:2393] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 24: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
W20260709 12:25:04.065343 17801 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.761s	user 0.015s	sys 0.016s
I20260709 12:25:03.991343 27428 raft_consensus.cc:3060] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:25:04.045076 27008 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.873s	user 0.035s	sys 0.036s
I20260709 12:25:04.078740 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:03.839756 27525 consensus_queue.cc:1059] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 26.16, Next index: 17, Last known committed idx: 15, Time since last communication: 0.201s
I20260709 12:25:03.959425 27449 raft_consensus.cc:515] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:04.132974 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 28 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:04.135823 18367 raft_consensus.cc:3060] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:04.085951 17585 catalog_manager.cc:5697] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b reported cstate change: term changed from 13 to 24. New cstate: current_term: 24 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: HEALTHY } } }
I20260709 12:25:04.101402 17729 leader_election.cc:304] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:25:03.959583 27443 raft_consensus.cc:515] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Starting leader election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:04.119447 17801 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.815s	user 0.015s	sys 0.016s
W20260709 12:25:04.143155 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 838751us
W20260709 12:25:03.996333 27448 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.167s	user 0.005s	sys 0.060s
I20260709 12:25:04.147351 27107 consensus_queue.cc:1048] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:25:04.079393 27436 raft_consensus.cc:3060] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:04.148234 27107 consensus_queue.cc:1243] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer 0da169ff82674dc1947055ce43de68a3 log is divergent from this leader: its last log entry 23.18 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 17
I20260709 12:25:04.133846 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:04.151737 17838 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35042 (request call id 3475) took 1157 ms. Trace:
I20260709 12:25:04.152454 18374 raft_consensus.cc:2393] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 27: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:25:04.152860 27107 consensus_queue.cc:1048] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20260709 12:25:04.153337 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 27 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:04.153965 27443 leader_election.cc:290] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 27 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:04.153977 18374 raft_consensus.cc:2468] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 26.
I20260709 12:25:04.154819 17683 raft_consensus.cc:1240] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 26. Current term is 27. Ops: []
I20260709 12:25:04.155624 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 26 candidate_status { last_received { term: 24 index: 20 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:04.156522 18374 raft_consensus.cc:2468] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 24.
I20260709 12:25:04.157739 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:04.118608 17757 rpcz_store.cc:276] 0709 12:25:03.012852 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:03.012986 (+   134us) service_pool.cc:224] Handling call
0709 12:25:03.013298 (+   312us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:03.013532 (+   234us) raft_consensus.cc:1534] Early marking committed up to index 17
0709 12:25:03.070714 (+ 57182us) log.cc:844] Serialized 14 byte log entry
0709 12:25:04.039569 (+968855us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:04.039630 (+    61us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:25:04.039676 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:04.039736 (+    60us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:04.039846 (+   110us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2353,"prepare.queue_time_us":95875,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":45,"raft.queue_time_us":80,"raft.run_cpu_time_us":183,"raft.run_wall_time_us":24313,"thread_start_us":968424,"threads_started":1}
I20260709 12:25:04.152184 17838 rpcz_store.cc:276] 0709 12:25:02.993966 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:03.088009 (+ 94043us) service_pool.cc:224] Handling call
0709 12:25:03.088358 (+   349us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:25:03.096051 (+  7693us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:03.096083 (+    32us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:25:03.096750 (+   667us) log.cc:844] Serialized 52 byte log entry
0709 12:25:03.953530 (+856780us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:25:03.953574 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:03.953652 (+    78us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:04.151470 (+197818us) raft_consensus.cc:1685] finished
0709 12:25:04.151494 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:04.151655 (+   161us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":7270,"raft.queue_time_us":18107,"raft.run_cpu_time_us":125,"raft.run_wall_time_us":3475,"thread_start_us":816663,"threads_started":1,"wal-append.queue_time_us":1032177}
W20260709 12:25:04.124344 17835 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.094s	user 0.000s	sys 0.043s
I20260709 12:25:04.120764 17727 leader_election.cc:304] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:04.125947 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 30 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:25:04.161970 17652 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.711s	user 0.014s	sys 0.029s
W20260709 12:25:04.162278 17652 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.711s	user 0.014s	sys 0.029s
I20260709 12:25:04.078969 27442 raft_consensus.cc:515] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:04.127661 18361 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.749s	user 0.022s	sys 0.030s
I20260709 12:25:04.162194 18368 raft_consensus.cc:2468] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 29.
W20260709 12:25:04.161736 17835 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.131s	user 0.000s	sys 0.044s
I20260709 12:25:04.163467 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 29 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:25:04.164332 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 713682us
I20260709 12:25:04.164475 18368 raft_consensus.cc:2410] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for term 29 because replica has last-logged OpId of term: 27 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 26 index: 18.
W20260709 12:25:04.165926 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 64654.1us
I20260709 12:25:04.166518 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 27 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:04.166847 17729 leader_election.cc:304] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:04.167497 18368 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 26.
I20260709 12:25:04.158442 18374 raft_consensus.cc:2468] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 26.
I20260709 12:25:04.168562 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 28 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:04.169198 17688 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37402 (request call id 3902) took 1355 ms. Trace:
I20260709 12:25:04.169601 17688 rpcz_store.cc:276] 0709 12:25:02.814289 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:02.853620 (+ 39331us) service_pool.cc:224] Handling call
0709 12:25:04.168988 (+1315368us) spinlock_profiling.cc:243] Waited 983 ms on lock 0x7b5400206f54. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:25:04.169144 (+   156us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2261667072}
W20260709 12:25:04.171005 18361 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.793s	user 0.022s	sys 0.031s
I20260709 12:25:04.171353 17758 raft_consensus.cc:2468] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 24.
I20260709 12:25:04.172257 17681 raft_consensus.cc:1240] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 28. Current term is 29. Ops: []
I20260709 12:25:04.173561 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 28 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
W20260709 12:25:04.176918 27555 log.cc:927] Time spent T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.052s	user 0.000s	sys 0.000s
I20260709 12:25:04.177335 27442 leader_election.cc:290] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:04.178236 17829 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35040 (request call id 3669) took 1185 ms. Trace:
I20260709 12:25:04.178696 17829 rpcz_store.cc:276] 0709 12:25:02.992778 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:03.002491 (+  9713us) service_pool.cc:224] Handling call
0709 12:25:03.002856 (+   365us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:03.003258 (+   402us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:03.003292 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:03.003823 (+   531us) log.cc:844] Serialized 26 byte log entry
0709 12:25:03.604282 (+600459us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:25:03.604325 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:03.604400 (+    75us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:04.177981 (+573581us) raft_consensus.cc:1685] finished
0709 12:25:04.178008 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:04.178180 (+   172us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":100,"raft.run_cpu_time_us":90,"raft.run_wall_time_us":92,"thread_start_us":594485,"threads_started":1,"wal-append.queue_time_us":1062063}
W20260709 12:25:04.184062 27111 consensus_meta.cc:287] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.878s	user 0.008s	sys 0.004s
I20260709 12:25:04.190105 27111 raft_consensus.cc:515] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:04.193068 27111 leader_election.cc:290] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 28 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:04.121105 27517 consensus_queue.cc:1048] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 20, Time since last communication: 0.073s
I20260709 12:25:04.194504 27107 raft_consensus.cc:493] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:04.188383 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 27 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:04.196616 18374 raft_consensus.cc:1275] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 25 index: 15. Preceding OpId from leader: term: 29 index: 17. (index mismatch)
I20260709 12:25:04.183583 18367 raft_consensus.cc:2468] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 28.
I20260709 12:25:04.197422 18374 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59902 (request call id 2274) took 1023 ms. Trace:
I20260709 12:25:04.199182 18374 rpcz_store.cc:276] 0709 12:25:03.174176 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:04.195817 (+1021641us) service_pool.cc:224] Handling call
0709 12:25:04.196225 (+   408us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:04.197145 (+   920us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:04.197357 (+   212us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:04.196295 17837 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35040 (request call id 3670) took 1203 ms. Trace:
I20260709 12:25:04.087201 27369 raft_consensus.cc:515] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:04.211478 16501 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
W20260709 12:25:04.211402 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 90727.8us
I20260709 12:25:04.208374 17837 rpcz_store.cc:276] 0709 12:25:02.993007 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:03.004303 (+ 11296us) service_pool.cc:224] Handling call
0709 12:25:03.005425 (+  1122us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:25:03.005928 (+   503us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:03.005954 (+    26us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:25:03.006510 (+   556us) log.cc:844] Serialized 52 byte log entry
0709 12:25:04.072330 (+1065820us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:25:04.072386 (+    56us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:04.072481 (+    95us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:04.196024 (+123543us) raft_consensus.cc:1685] finished
0709 12:25:04.196046 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:04.196221 (+   175us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":95,"raft.run_cpu_time_us":71,"raft.run_wall_time_us":71,"thread_start_us":1065402,"threads_started":1,"wal-append.queue_time_us":1092936}
I20260709 12:25:04.195061 27107 raft_consensus.cc:515] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:04.254086 18365 raft_consensus.cc:1275] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 25 index: 19. Preceding OpId from leader: term: 26 index: 20. (index mismatch)
I20260709 12:25:04.255528 18365 raft_consensus.cc:1275] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 27 index: 19. Preceding OpId from leader: term: 28 index: 20. (index mismatch)
I20260709 12:25:04.255927 27107 leader_election.cc:290] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:25:04.215993 27363 raft_consensus.cc:3055] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:25:04.256681 27363 raft_consensus.cc:740] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 29 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:25:04.258278 27451 raft_consensus.cc:2804] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:25:04.257728 27363 consensus_queue.cc:260] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 29.19, Last appended by leader: 19, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:04.258809 27451 raft_consensus.cc:697] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:25:04.259047 27363 raft_consensus.cc:3060] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Advancing to term 30
W20260709 12:25:04.216475 27463 consensus_meta.cc:287] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 1.072s	user 0.012s	sys 0.000s
I20260709 12:25:04.073535 27359 consensus_queue.cc:260] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 28.20, Last appended by leader: 20, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:04.119379 27446 consensus_queue.cc:1048] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.092s
W20260709 12:25:04.220558 27550 log.cc:927] Time spent T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.102s	user 0.000s	sys 0.000s
W20260709 12:25:04.227788 17684 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.772s	user 0.028s	sys 0.022s
I20260709 12:25:04.126699 27475 raft_consensus.cc:493] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 32 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
W20260709 12:25:04.229849 17685 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.769s	user 0.030s	sys 0.046s
W20260709 12:25:04.277906 17685 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.822s	user 0.031s	sys 0.046s
I20260709 12:25:04.074288 27062 consensus_queue.cc:237] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 31.21, Last appended by leader: 21, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:04.252238 17585 catalog_manager.cc:5697] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b reported cstate change: term changed from 17 to 20. New cstate: current_term: 20 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } }
I20260709 12:25:04.243567 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:04.209908 27525 raft_consensus.cc:3060] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:04.246711 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 27 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:04.250324 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 27 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:25:04.250634 17802 leader_election.cc:304] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:04.265682 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
W20260709 12:25:04.284641 17762 consensus_meta.cc:287] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.538s	user 0.010s	sys 0.000s
I20260709 12:25:04.285085 17762 raft_consensus.cc:2468] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 28.
W20260709 12:25:04.265303 27462 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.967s	user 0.036s	sys 0.037s
I20260709 12:25:04.286283 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 27 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:04.288151 17832 raft_consensus.cc:1240] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 24. Current term is 25. Ops: []
I20260709 12:25:04.268787 17652 leader_election.cc:304] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:04.228650 27517 raft_consensus.cc:2804] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:25:04.294584 27395 raft_consensus.cc:2804] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:25:04.295128 27395 raft_consensus.cc:697] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 30 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:25:04.296295 27111 raft_consensus.cc:3060] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:04.296008 27395 consensus_queue.cc:237] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 29.17, Last appended by leader: 17, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:04.272225 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 25 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:04.229163 27324 raft_consensus.cc:2749] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:25:04.232344 27524 raft_consensus.cc:493] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:25:04.299782 27524 raft_consensus.cc:515] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:04.302582 27524 leader_election.cc:290] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:25:04.304431 27462 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.007s	user 0.036s	sys 0.038s
W20260709 12:25:04.275074 17684 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.820s	user 0.029s	sys 0.022s
W20260709 12:25:04.308679 27522 consensus_meta.cc:287] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.554s	user 0.010s	sys 0.000s
I20260709 12:25:04.309566 17681 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37396 (request call id 3427) took 1767 ms. Trace:
I20260709 12:25:04.309715 18364 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59902 (request call id 2268) took 1157 ms. Trace:
I20260709 12:25:04.310151 18364 rpcz_store.cc:276] 0709 12:25:03.152474 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:03.152614 (+   140us) service_pool.cc:224] Handling call
0709 12:25:04.309542 (+1156928us) spinlock_profiling.cc:243] Waited 808 ms on lock 0x7b5400305f54. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:25:04.309660 (+   118us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1858853248}
I20260709 12:25:04.310047 17681 rpcz_store.cc:276] 0709 12:25:02.541605 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:02.945583 (+403978us) service_pool.cc:224] Handling call
0709 12:25:04.169142 (+1223559us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:04.172670 (+  3528us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:04.309286 (+136616us) spinlock_profiling.cc:243] Waited 1.22 s on lock 0x7b5400206f50. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a61f8ad 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:25:04.309490 (+   204us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2813100800}
I20260709 12:25:04.276813 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:04.271350 27463 raft_consensus.cc:2749] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
W20260709 12:25:04.211372 27519 consensus_meta.cc:287] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 1.145s	user 0.008s	sys 0.004s
W20260709 12:25:04.313946 17655 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:04.291702 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 64036.3us
I20260709 12:25:04.293534 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 24 candidate_status { last_received { term: 23 index: 21 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:04.309293 27522 raft_consensus.cc:515] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:04.312860 17836 raft_consensus.cc:1240] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 26. Current term is 27. Ops: [26.16-26.16]
W20260709 12:25:04.328594 17654 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:04.281742 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 26 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:04.312804 17833 raft_consensus.cc:2393] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 24: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:25:04.330416 17654 leader_election.cc:304] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:04.333500 18371 raft_consensus.cc:2468] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 30.
I20260709 12:25:04.319377 17652 leader_election.cc:304] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:25:04.320708 27552 log.cc:927] Time spent T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.245s	user 0.000s	sys 0.000s
I20260709 12:25:04.321372 27245 leader_election.cc:290] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:04.290088 27475 raft_consensus.cc:515] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:25:04.324421 27556 log.cc:927] Time spent T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.092s	user 0.000s	sys 0.000s
W20260709 12:25:04.279719 27431 consensus_meta.cc:287] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.537s	user 0.002s	sys 0.008s
I20260709 12:25:04.341261 27431 raft_consensus.cc:515] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:04.343394 27431 leader_election.cc:290] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 24 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:04.344820 27446 raft_consensus.cc:493] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:25:04.345392 27446 raft_consensus.cc:515] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:04.313539 27519 raft_consensus.cc:2749] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:25:04.337505 17833 raft_consensus.cc:1275] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 25 index: 16. Preceding OpId from leader: term: 26 index: 16. (term mismatch)
I20260709 12:25:04.352558 17833 pending_rounds.cc:85] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b: Aborting all ops after (but not including) 15
I20260709 12:25:04.352939 17833 pending_rounds.cc:107] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b: Aborting uncommitted NO_OP operation due to leader change: 25.16
I20260709 12:25:04.352301 17585 catalog_manager.cc:5697] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b reported cstate change: term changed from 17 to 19, leader changed from e063541542a94969898868bf6eb5ce1b (127.16.29.67) to e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66). New cstate: current_term: 19 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:25:04.353286 17833 raft_consensus.cc:2889] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:04.340564 27443 raft_consensus.cc:2749] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:25:04.340714 27461 raft_consensus.cc:2804] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:25:04.359383 17688 raft_consensus.cc:1240] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 27. Current term is 28. Ops: []
I20260709 12:25:04.346761 18362 raft_consensus.cc:2468] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 24.
I20260709 12:25:04.347030 17654 leader_election.cc:304] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b
W20260709 12:25:04.336046 17689 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.575s	user 0.026s	sys 0.011s
W20260709 12:25:04.360504 17689 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.604s	user 0.026s	sys 0.011s
W20260709 12:25:04.360742 17755 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.598s	user 0.040s	sys 0.059s
I20260709 12:25:04.259684 27451 consensus_queue.cc:237] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 23.17, Last appended by leader: 17, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:04.362418 17682 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44898 (request call id 1929) took 1220 ms. Trace:
I20260709 12:25:04.363456 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:04.367682 17682 rpcz_store.cc:276] 0709 12:25:03.142058 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:03.437885 (+295827us) service_pool.cc:224] Handling call
0709 12:25:03.438197 (+   312us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:03.508794 (+ 70597us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:04.362169 (+853375us) spinlock_profiling.cc:243] Waited 443 ms on lock 0x7b540030b720. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a597dc0 00007f7b4a623a02 00007f7b4a61fdd6 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2
0709 12:25:04.362362 (+   193us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1019204864}
I20260709 12:25:04.362757 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
W20260709 12:25:04.362429 17755 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.602s	user 0.040s	sys 0.059s
W20260709 12:25:04.364094 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:04.355279 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:25:04.346345 27450 consensus_meta.cc:287] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.922s	user 0.006s	sys 0.006s
I20260709 12:25:04.372879 17683 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44898 (request call id 1923) took 1542 ms. Trace:
I20260709 12:25:04.378551 17683 rpcz_store.cc:276] 0709 12:25:02.830631 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:02.990673 (+160042us) service_pool.cc:224] Handling call
0709 12:25:02.990918 (+   245us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:04.155277 (+1164359us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:04.372669 (+217392us) spinlock_profiling.cc:243] Waited 1.16 s on lock 0x7b5400207954. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:25:04.372830 (+   161us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2676315904}
I20260709 12:25:04.378959 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:04.379572 17760 raft_consensus.cc:3060] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:04.345402 27475 leader_election.cc:290] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:04.373984 17585 catalog_manager.cc:5697] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b reported cstate change: term changed from 23 to 24, leader changed from e063541542a94969898868bf6eb5ce1b (127.16.29.67) to 0da169ff82674dc1947055ce43de68a3 (127.16.29.65). New cstate: current_term: 24 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:25:04.377678 27197 raft_consensus.cc:493] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:04.361104 17654 leader_election.cc:304] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:25:04.359056 27461 raft_consensus.cc:493] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:25:04.389331 17804 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.554s	user 0.015s	sys 0.025s
W20260709 12:25:04.368024 17764 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.622s	user 0.010s	sys 0.028s
W20260709 12:25:04.397689 17804 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.562s	user 0.015s	sys 0.025s
W20260709 12:25:04.397881 17764 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.652s	user 0.011s	sys 0.029s
I20260709 12:25:04.390801 17654 leader_election.cc:304] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:25:04.396131 18333 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.958s	user 0.034s	sys 0.036s
W20260709 12:25:04.400899 18333 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.963s	user 0.034s	sys 0.036s
I20260709 12:25:04.401584 17654 leader_election.cc:304] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:04.402925 17654 leader_election.cc:304] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:04.404968 18367 raft_consensus.cc:1240] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 23. Current term is 24. Ops: []
I20260709 12:25:04.406221 18367 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59902 (request call id 2290) took 1101 ms. Trace:
I20260709 12:25:04.406571 18367 rpcz_store.cc:276] 0709 12:25:03.304688 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:04.202479 (+897791us) service_pool.cc:224] Handling call
0709 12:25:04.404764 (+202285us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:04.405379 (+   615us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:04.405953 (+   574us) spinlock_profiling.cc:243] Waited 202 ms on lock 0x7b54008020d0. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a61f8ad 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:25:04.406167 (+   214us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":464529152}
I20260709 12:25:04.371176 27427 raft_consensus.cc:2804] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Leader pre-election won for term 22
W20260709 12:25:04.382128 27196 consensus_meta.cc:287] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.787s	user 0.011s	sys 0.000s
W20260709 12:25:04.401614 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 964412us
I20260709 12:25:04.415016 18333 leader_election.cc:304] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:04.407683 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:04.425783 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 27 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
W20260709 12:25:04.410550 27394 consensus_meta.cc:287] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.821s	user 0.003s	sys 0.007s
I20260709 12:25:04.427925 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 26 candidate_status { last_received { term: 24 index: 20 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:25:04.429239 17804 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:04.429481 18333 leader_election.cc:304] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:04.408430 27443 raft_consensus.cc:2804] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:25:04.396271 27432 raft_consensus.cc:493] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:25:04.412281 27461 raft_consensus.cc:3060] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:04.408599 27128 raft_consensus.cc:2749] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:25:04.441428 27369 leader_election.cc:290] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:25:04.441793 18371 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59886 (request call id 2626) took 1298 ms. Trace:
I20260709 12:25:04.458084 18371 rpcz_store.cc:276] 0709 12:25:03.143254 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:03.143388 (+   134us) service_pool.cc:224] Handling call
0709 12:25:04.441597 (+1298209us) spinlock_profiling.cc:243] Waited 996 ms on lock 0x7b5400466ed4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:25:04.441732 (+   135us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2291924224}
I20260709 12:25:04.442921 18365 raft_consensus.cc:1240] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 29. Current term is 30. Ops: []
I20260709 12:25:04.421172 27245 raft_consensus.cc:2749] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:25:04.381680 27428 raft_consensus.cc:515] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:04.416498 27427 raft_consensus.cc:493] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:25:04.383589 27450 raft_consensus.cc:515] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:04.456244 18333 leader_election.cc:304] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:04.426234 27442 raft_consensus.cc:3060] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:04.460481 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 26 candidate_status { last_received { term: 24 index: 20 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:04.461238 18371 raft_consensus.cc:1240] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 27. Current term is 28. Ops: []
I20260709 12:25:04.433254 17585 catalog_manager.cc:5697] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b reported cstate change: config changed from index 5 to 12, term changed from 18 to 22, leader changed from e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66) to <none>, 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER, no longer has a pending config: VOTER 0da169ff82674dc1947055ce43de68a3 (127.16.29.65), VOTER e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66), VOTER e063541542a94969898868bf6eb5ce1b (127.16.29.67), VOTER 334a4e1f59944544b3b3ff36616b57da (127.16.29.68). New cstate: current_term: 22 committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:25:04.424997 27561 raft_consensus.cc:3060] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:04.498824 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:04.498991 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 27 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:04.408241 27197 raft_consensus.cc:515] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:04.510435 17585 catalog_manager.cc:5697] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b reported cstate change: term changed from 13 to 20. New cstate: current_term: 20 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:25:04.504207 17760 raft_consensus.cc:2468] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 28.
I20260709 12:25:04.489702 27427 raft_consensus.cc:3060] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:04.504853 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:04.506292 27450 leader_election.cc:290] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 29 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:04.507758 17759 raft_consensus.cc:2368] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for earlier term 28. Current term is 29.
I20260709 12:25:04.420126 27196 raft_consensus.cc:515] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 33 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:04.509156 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:04.436985 27443 raft_consensus.cc:493] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:25:04.521837 17585 catalog_manager.cc:5697] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b reported cstate change: term changed from 15 to 19, leader changed from 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) to e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66). New cstate: current_term: 19 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:25:04.460992 27406 raft_consensus.cc:493] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:04.507968 17838 raft_consensus.cc:2393] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 27: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:25:04.446825 27548 raft_consensus.cc:2749] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:25:04.504364 27544 consensus_queue.cc:1048] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.017s
I20260709 12:25:04.434027 27307 consensus_queue.cc:1059] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 30.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.063s
I20260709 12:25:04.520982 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:04.514137 17801 leader_election.cc:304] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:04.544060 18333 leader_election.cc:304] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:04.525177 17585 catalog_manager.cc:5697] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b reported cstate change: term changed from 16 to 18, leader changed from 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) to e063541542a94969898868bf6eb5ce1b (127.16.29.67). New cstate: current_term: 18 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: HEALTHY } } }
I20260709 12:25:04.436719 27394 raft_consensus.cc:515] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:04.507283 27559 raft_consensus.cc:515] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:04.510095 27128 raft_consensus.cc:493] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
W20260709 12:25:04.521226 27338 consensus_meta.cc:287] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.923s	user 0.013s	sys 0.000s
I20260709 12:25:04.451010 27432 raft_consensus.cc:515] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:25:04.530233 27452 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.823s	user 0.022s	sys 0.026s
I20260709 12:25:04.552924 17762 raft_consensus.cc:2393] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 24: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:25:04.573634 18333 leader_election.cc:304] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:25:04.617231 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 43723us
I20260709 12:25:04.525497 27443 raft_consensus.cc:3060] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:04.490038 27542 consensus_queue.cc:1059] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 24.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.078s
I20260709 12:25:04.493352 27270 raft_consensus.cc:515] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:04.620687 18333 leader_election.cc:304] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:04.523588 27449 leader_election.cc:290] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:04.622378 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:04.546078 27369 raft_consensus.cc:493] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:25:04.607038 27154 raft_consensus.cc:493] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
W20260709 12:25:04.629805 27569 log.cc:927] Time spent T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.057s	user 0.000s	sys 0.000s
I20260709 12:25:04.637552 27394 leader_election.cc:290] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 28 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:25:04.635567 27525 raft_consensus.cc:2749] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:25:04.639209 27446 leader_election.cc:290] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:04.641824 17757 raft_consensus.cc:1240] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 28. Current term is 29. Ops: []
I20260709 12:25:04.651244 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:04.603199 27509 raft_consensus.cc:2804] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:25:04.607200 27415 raft_consensus.cc:2749] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260709 12:25:04.618606 27452 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.905s	user 0.022s	sys 0.026s
I20260709 12:25:04.637931 27197 leader_election.cc:290] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:25:04.641278 27438 consensus_meta.cc:287] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.542s	user 0.006s	sys 0.004s
W20260709 12:25:04.672835 17758 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36572 (request call id 1740) took 1579 ms (client timeout 2047 ms). Trace:
W20260709 12:25:04.710592 17758 rpcz_store.cc:269] 0709 12:25:03.093317 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:03.093826 (+   509us) service_pool.cc:224] Handling call
0709 12:25:04.672663 (+1578837us) spinlock_profiling.cc:243] Waited 998 ms on lock 0x7b540019f5d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:25:04.672779 (+   116us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2295983360}
I20260709 12:25:04.715132 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 28 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:04.716854 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 27 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:04.717638 17758 raft_consensus.cc:3060] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:04.612784 27338 raft_consensus.cc:515] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:04.696206 17762 raft_consensus.cc:3060] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:04.695670 17585 catalog_manager.cc:5697] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b reported cstate change: term changed from 15 to 23. New cstate: current_term: 23 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } }
I20260709 12:25:04.734083 17585 catalog_manager.cc:5697] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b reported cstate change: term changed from 12 to 16, leader changed from e063541542a94969898868bf6eb5ce1b (127.16.29.67) to e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66). New cstate: current_term: 16 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } }
I20260709 12:25:04.674383 17833 raft_consensus.cc:2393] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 26: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:25:04.699219 18366 raft_consensus.cc:2468] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 29.
W20260709 12:25:04.700320 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 446869us
I20260709 12:25:04.640187 27450 raft_consensus.cc:493] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:04.694571 27559 leader_election.cc:290] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 29 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:04.550665 27563 consensus_queue.cc:1048] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.124s
I20260709 12:25:04.660969 27517 raft_consensus.cc:697] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:25:04.723119 27394 raft_consensus.cc:493] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:04.730091 17684 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37402 (request call id 3908) took 1317 ms. Trace:
I20260709 12:25:04.729933 27197 raft_consensus.cc:3055] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:25:04.551479 27406 raft_consensus.cc:515] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:04.672787 27522 leader_election.cc:290] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 28 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:25:04.790959 17729 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.623s	user 0.009s	sys 0.022s
W20260709 12:25:04.817505 17729 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.650s	user 0.010s	sys 0.023s
W20260709 12:25:04.818383 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 651560us
I20260709 12:25:04.820165 17729 leader_election.cc:304] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
W20260709 12:25:04.797626 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 392115us
I20260709 12:25:04.709017 27509 raft_consensus.cc:697] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 28 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:25:04.821907 27509 consensus_queue.cc:237] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 26.19, Last appended by leader: 19, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:04.709913 27525 raft_consensus.cc:493] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:04.802711 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:04.801407 17684 rpcz_store.cc:276] 0709 12:25:03.412622 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:03.447433 (+ 34811us) service_pool.cc:224] Handling call
0709 12:25:03.447729 (+   296us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:03.454600 (+  6871us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:03.454645 (+    45us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:03.455255 (+   610us) log.cc:844] Serialized 26 byte log entry
0709 12:25:04.308788 (+853533us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:25:04.308834 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:04.308903 (+    69us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:04.729828 (+420925us) raft_consensus.cc:1685] finished
0709 12:25:04.729856 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:04.730021 (+   165us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5278,"raft.queue_time_us":6494,"raft.run_cpu_time_us":142,"raft.run_wall_time_us":50165,"thread_start_us":853185,"threads_started":1,"wal-append.queue_time_us":1034664}
I20260709 12:25:04.613121 27128 raft_consensus.cc:515] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:04.827185 17652 leader_election.cc:304] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:04.791330 27452 raft_consensus.cc:493] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:04.830487 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 427602us
I20260709 12:25:04.620102 27548 raft_consensus.cc:2749] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
W20260709 12:25:04.805868 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 478316us
I20260709 12:25:04.705890 27542 consensus_queue.cc:1048] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.089s
I20260709 12:25:04.735491 17585 catalog_manager.cc:5697] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b reported cstate change: term changed from 13 to 17, leader changed from 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) to e063541542a94969898868bf6eb5ce1b (127.16.29.67). New cstate: current_term: 17 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20260709 12:25:04.711546 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 467303us
W20260709 12:25:04.797626 27576 log.cc:927] Time spent T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.087s	user 0.000s	sys 0.000s
I20260709 12:25:04.806161 27197 raft_consensus.cc:740] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 30 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:25:04.825820 27525 raft_consensus.cc:515] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:04.660523 27369 raft_consensus.cc:515] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:04.829493 17760 raft_consensus.cc:2468] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 24.
I20260709 12:25:04.847024 17585 catalog_manager.cc:5697] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b reported cstate change: term changed from 11 to 17, leader changed from e063541542a94969898868bf6eb5ce1b (127.16.29.67) to 0da169ff82674dc1947055ce43de68a3 (127.16.29.65). New cstate: current_term: 17 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:25:04.669178 27154 raft_consensus.cc:515] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:04.834170 27324 raft_consensus.cc:2749] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:25:04.730368 27438 raft_consensus.cc:515] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:04.816632 17689 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37402 (request call id 3903) took 1922 ms. Trace:
I20260709 12:25:04.817032 27559 raft_consensus.cc:3055] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:25:04.808599 27432 leader_election.cc:290] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:04.791729 27450 raft_consensus.cc:515] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:04.897701 27572 log.cc:927] Time spent T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.189s	user 0.000s	sys 0.001s
I20260709 12:25:04.898605 27585 raft_consensus.cc:2804] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:25:04.899353 27585 raft_consensus.cc:493] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:25:04.900236 17764 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38776 (request call id 3321) took 1287 ms. Trace:
W20260709 12:25:04.901496 27574 log.cc:927] Time spent T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.194s	user 0.000s	sys 0.000s
I20260709 12:25:04.900660 17764 rpcz_store.cc:276] 0709 12:25:03.612498 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:03.612654 (+   156us) service_pool.cc:224] Handling call
0709 12:25:03.646956 (+ 34302us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:03.647277 (+   321us) raft_consensus.cc:1534] Early marking committed up to index 18
0709 12:25:03.647315 (+    38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:03.709087 (+ 61772us) log.cc:844] Serialized 26 byte log entry
0709 12:25:04.398443 (+689356us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:25:04.398501 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:04.398592 (+    91us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:04.899994 (+501402us) raft_consensus.cc:1685] finished
0709 12:25:04.900014 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:04.900172 (+   158us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":652758,"threads_started":1,"wal-append.queue_time_us":910505}
I20260709 12:25:04.828424 27505 raft_consensus.cc:2749] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:25:04.847705 27462 raft_consensus.cc:493] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:04.896726 27577 consensus_queue.cc:1059] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 25.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.025s
W20260709 12:25:04.953810 27571 log.cc:927] Time spent T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.124s	user 0.000s	sys 0.001s
I20260709 12:25:04.863819 27559 raft_consensus.cc:740] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:25:04.959863 17685 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37402 (request call id 3906) took 1555 ms. Trace:
I20260709 12:25:04.960034 27270 leader_election.cc:290] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 26 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:04.960520 17685 rpcz_store.cc:276] 0709 12:25:03.404317 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:03.445019 (+ 40702us) service_pool.cc:224] Handling call
0709 12:25:03.445355 (+   336us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:04.278457 (+833102us) raft_consensus.cc:1534] Early marking committed up to index 15
0709 12:25:04.278504 (+    47us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:04.384077 (+105573us) spinlock_profiling.cc:243] Waited 7.7 ms on lock 0x7b2c00000c88. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b48f19a11 00007f7b48f29372 00007f7b4a65ba01 00007f7b4a62043a 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2
0709 12:25:04.384544 (+   467us) log.cc:844] Serialized 26 byte log entry
0709 12:25:04.841970 (+457426us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:25:04.842022 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:04.842098 (+    76us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:04.959440 (+117342us) raft_consensus.cc:1685] finished
0709 12:25:04.959474 (+    34us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:04.959779 (+   305us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":13281,"raft.queue_time_us":78382,"raft.run_cpu_time_us":135,"raft.run_wall_time_us":60381,"spinlock_wait_cycles":17699200,"thread_start_us":1234154,"threads_started":2,"wal-append.queue_time_us":546735}
I20260709 12:25:04.961661 27197 consensus_queue.cc:260] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 30.17, Last appended by leader: 17, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:04.874372 27196 leader_election.cc:290] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 33 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:04.838119 27452 raft_consensus.cc:515] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:04.968480 27452 leader_election.cc:290] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:04.973194 18374 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 2788) took 1647 ms. Trace:
I20260709 12:25:04.973664 18374 rpcz_store.cc:276] 0709 12:25:03.326215 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:04.220124 (+893909us) service_pool.cc:224] Handling call
0709 12:25:04.220469 (+   345us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:04.229249 (+  8780us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:04.229279 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:04.229759 (+   480us) log.cc:844] Serialized 26 byte log entry
0709 12:25:04.307648 (+ 77889us) spinlock_profiling.cc:243] Waited 16.4 ms on lock 0x7b18003e7e80. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b47515e07 00007f7b503709a9 00007f7b4753269b 00007f7b4753093f 00007f7b47532c02 00007f7b4a1ba20b 00007f7b4a1bfd23 00007f7b4a1c00d1 00007f7b4a5dba0f
0709 12:25:04.704474 (+396826us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:25:04.704536 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:04.704639 (+   103us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:04.972854 (+268215us) raft_consensus.cc:1685] finished
0709 12:25:04.972876 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:04.973130 (+   254us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1553,"raft.queue_time_us":1784,"raft.run_cpu_time_us":131,"raft.run_wall_time_us":4271,"spinlock_wait_cycles":37741824,"thread_start_us":396559,"threads_started":1,"wal-append.queue_time_us":723729}
I20260709 12:25:04.843767 27517 consensus_queue.cc:237] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:04.994237 27462 raft_consensus.cc:515] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:05.035050 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:05.035928 17760 raft_consensus.cc:3060] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:04.831092 17755 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38776 (request call id 3320) took 1816 ms. Trace:
I20260709 12:25:04.852933 17801 leader_election.cc:304] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:25:04.856549 17655 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:04.858564 17728 leader_election.cc:304] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:25:04.834708 17758 raft_consensus.cc:2468] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 27.
W20260709 12:25:04.860065 17804 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:04.837883 27428 leader_election.cc:290] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 27 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:04.858433 17689 rpcz_store.cc:276] 0709 12:25:02.894659 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:03.031292 (+136633us) service_pool.cc:224] Handling call
0709 12:25:03.074949 (+ 43657us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 12:25:03.112850 (+ 37901us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:03.112881 (+    31us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 12:25:03.313359 (+200478us) spinlock_profiling.cc:243] Waited 2.79 ms on lock 0x7b180042a5a0. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b475012a6 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b48f19a11 00007f7b48f29372 00007f7b4a65ba01 00007f7b4a62043a
0709 12:25:03.651854 (+338495us) spinlock_profiling.cc:243] Waited 1.51 ms on lock 0x7b2c00000c88. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b48f19a11 00007f7b48f29372 00007f7b4a65ba01 00007f7b4a62043a 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2
0709 12:25:03.710495 (+ 58641us) log.cc:844] Serialized 78 byte log entry
0709 12:25:04.361077 (+650582us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:25:04.794886 (+433809us) log.cc:844] Serialized 14 byte log entry
0709 12:25:04.795277 (+   391us) log.cc:844] Serialized 14 byte log entry
0709 12:25:04.816204 (+ 20927us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:04.816289 (+    85us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:04.816339 (+    50us) raft_consensus.cc:1685] finished
0709 12:25:04.816358 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:04.816552 (+   194us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4677,"raft.queue_time_us":43846,"raft.run_cpu_time_us":295,"raft.run_wall_time_us":15661,"spinlock_wait_cycles":12566016,"thread_start_us":1013448,"threads_started":2,"wal-append.queue_time_us":814024}
W20260709 12:25:05.091640 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 233158us
I20260709 12:25:04.862461 17762 raft_consensus.cc:2468] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 28.
W20260709 12:25:05.093623 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 240776us
I20260709 12:25:04.808670 27394 raft_consensus.cc:515] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:04.851559 27111 consensus_meta.cc:287] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.555s	user 0.005s	sys 0.013s
I20260709 12:25:04.897646 27578 raft_consensus.cc:493] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
W20260709 12:25:04.844152 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 184619us
I20260709 12:25:05.107327 18333 leader_election.cc:304] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:25:04.845731 27522 raft_consensus.cc:2804] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:25:04.912815 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 26 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:04.902913 27585 raft_consensus.cc:3060] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:04.922479 27438 leader_election.cc:290] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 26 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:05.117532 17801 leader_election.cc:304] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:05.118587 18368 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59902 (request call id 2273) took 1944 ms. Trace:
I20260709 12:25:05.079469 17755 rpcz_store.cc:276] 0709 12:25:03.014431 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:03.014557 (+   126us) service_pool.cc:224] Handling call
0709 12:25:03.014888 (+   331us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:03.015111 (+   223us) raft_consensus.cc:1534] Early marking committed up to index 17
0709 12:25:03.759680 (+744569us) log.cc:844] Serialized 14 byte log entry
0709 12:25:04.830760 (+1071080us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:04.830818 (+    58us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:25:04.830861 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:04.830925 (+    64us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:04.831038 (+   113us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":55402,"prepare.queue_time_us":136213,"prepare.run_cpu_time_us":8447,"prepare.run_wall_time_us":58182,"raft.queue_time_us":330615,"raft.run_cpu_time_us":134,"raft.run_wall_time_us":8078,"thread_start_us":1720576,"threads_started":3,"wal-append.queue_time_us":888338}
I20260709 12:25:05.119216 17837 raft_consensus.cc:1240] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 26. Current term is 27. Ops: []
I20260709 12:25:05.118959 18368 rpcz_store.cc:276] 0709 12:25:03.173921 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:04.175078 (+1001157us) service_pool.cc:224] Handling call
0709 12:25:04.176416 (+  1338us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:25:04.177234 (+   818us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:04.177352 (+   118us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:25:04.177939 (+   587us) log.cc:844] Serialized 52 byte log entry
0709 12:25:04.656208 (+478269us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:25:04.674502 (+ 18294us) log.cc:844] Serialized 14 byte log entry
0709 12:25:05.104681 (+430179us) log.cc:844] Serialized 14 byte log entry
0709 12:25:05.118138 (+ 13457us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:05.118250 (+   112us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:05.118315 (+    65us) raft_consensus.cc:1685] finished
0709 12:25:05.118334 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:05.118509 (+   175us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3180,"prepare.queue_time_us":39909,"prepare.run_cpu_time_us":51,"prepare.run_wall_time_us":51,"raft.queue_time_us":49195,"raft.run_cpu_time_us":244,"raft.run_wall_time_us":46714,"thread_start_us":906439,"threads_started":2,"wal-append.queue_time_us":465318}
W20260709 12:25:05.115691 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 278689us
I20260709 12:25:05.125279 17654 leader_election.cc:304] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:25:04.850904 17729 leader_election.cc:304] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:25:05.131233 17755 raft_consensus.cc:1275] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 25 index: 18. Preceding OpId from leader: term: 26 index: 19. (index mismatch)
I20260709 12:25:05.140597 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:05.158682 17762 raft_consensus.cc:1240] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 29. Current term is 30. Ops: []
I20260709 12:25:05.161736 18371 raft_consensus.cc:3060] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Advancing to term 28
W20260709 12:25:05.137992 17832 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.810s	user 0.002s	sys 0.010s
W20260709 12:25:05.162909 17832 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.835s	user 0.003s	sys 0.012s
I20260709 12:25:05.164573 17764 raft_consensus.cc:1240] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 27. Current term is 28. Ops: []
I20260709 12:25:05.165158 17654 leader_election.cc:304] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:25:05.133657 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 27 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:05.133807 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
W20260709 12:25:05.168788 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 341679us
I20260709 12:25:05.106071 27008 raft_consensus.cc:2804] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:25:05.173454 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:05.174322 17831 raft_consensus.cc:2368] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 19. Current term is 20.
I20260709 12:25:05.169982 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
W20260709 12:25:05.130357 17761 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.784s	user 0.012s	sys 0.047s
I20260709 12:25:05.142114 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:05.175127 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 27 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:25:05.175458 17761 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.829s	user 0.012s	sys 0.049s
I20260709 12:25:05.177591 17654 leader_election.cc:304] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:05.173422 27590 raft_consensus.cc:2804] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:25:05.180859 27590 raft_consensus.cc:493] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:25:05.181311 27590 raft_consensus.cc:3060] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:05.173262 27438 raft_consensus.cc:2804] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:25:05.179980 27008 raft_consensus.cc:697] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 24 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:25:05.119328 27489 raft_consensus.cc:2749] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:25:05.114617 27111 raft_consensus.cc:2749] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:25:05.184506 18368 raft_consensus.cc:2393] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 20: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:25:05.185678 17760 raft_consensus.cc:2468] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 31.
I20260709 12:25:05.193574 17762 raft_consensus.cc:1240] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 30. Current term is 31. Ops: []
I20260709 12:25:05.186950 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:05.196784 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 26 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:05.184173 27438 raft_consensus.cc:493] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:25:05.157703 27415 raft_consensus.cc:2804] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:25:05.204603 27438 raft_consensus.cc:3060] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:05.205017 27415 raft_consensus.cc:697] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 26 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:25:05.206029 27415 consensus_queue.cc:237] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 25.18, Last appended by leader: 18, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:05.204618 27154 leader_election.cc:290] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:25:05.129410 26818 raft_consensus.cc:493] T a5732e71b9d54f8786b8c511d69943bf P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:25:05.157492 27548 raft_consensus.cc:493] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:25:05.166555 27406 leader_election.cc:290] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:05.157184 27522 raft_consensus.cc:697] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 28 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:25:05.159464 27449 raft_consensus.cc:2804] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:25:05.188196 18364 raft_consensus.cc:2393] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 24: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
W20260709 12:25:05.136931 27461 consensus_meta.cc:287] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.676s	user 0.014s	sys 0.004s
I20260709 12:25:05.115140 27582 consensus_queue.cc:1059] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 28.19, Next index: 20, Last known committed idx: 18, Time since last communication: 0.073s
I20260709 12:25:05.231268 18371 raft_consensus.cc:2468] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 28.
I20260709 12:25:05.191944 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:05.232888 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:05.233516 17837 raft_consensus.cc:2468] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 26.
I20260709 12:25:05.234314 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 25 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:05.234823 17837 raft_consensus.cc:3060] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Advancing to term 25
I20260709 12:25:05.120850 27452 consensus_queue.cc:1059] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 21.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.072s
I20260709 12:25:05.120167 27578 raft_consensus.cc:515] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:05.238517 27578 leader_election.cc:290] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:05.238857 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 28 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:05.239921 17836 raft_consensus.cc:2393] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 28: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:25:05.239614 27548 raft_consensus.cc:515] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:05.241902 27548 leader_election.cc:290] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:25:05.209578 27576 log.cc:927] Time spent T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.050s	user 0.000s	sys 0.000s
I20260709 12:25:05.231369 27582 raft_consensus.cc:2804] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:25:05.245168 27582 raft_consensus.cc:697] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 30 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:25:05.239073 27578 raft_consensus.cc:3055] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:25:05.172681 27338 leader_election.cc:290] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 31 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:05.198223 18374 raft_consensus.cc:2468] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 25.
I20260709 12:25:05.246691 27582 consensus_queue.cc:237] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 29.19, Last appended by leader: 19, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:05.252629 27449 raft_consensus.cc:697] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 28 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:25:05.274928 27449 consensus_queue.cc:237] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 27.17, Last appended by leader: 17, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:05.248770 26818 raft_consensus.cc:515] T a5732e71b9d54f8786b8c511d69943bf P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:05.281358 26818 leader_election.cc:290] T a5732e71b9d54f8786b8c511d69943bf P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:05.248282 27008 consensus_queue.cc:237] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 23.17, Last appended by leader: 17, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:05.210309 27369 leader_election.cc:290] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:25:05.294305 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 102958us
W20260709 12:25:05.294441 18334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:05.228194 27598 raft_consensus.cc:2749] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:25:05.219199 27428 raft_consensus.cc:493] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
W20260709 12:25:05.300530 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 449689us
W20260709 12:25:05.293953 27436 consensus_meta.cc:287] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 1.145s	user 0.027s	sys 0.012s
I20260709 12:25:05.250636 27578 raft_consensus.cc:740] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:25:05.315521 17654 leader_election.cc:304] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:25:05.308193 27595 raft_consensus.cc:3055] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:25:05.307049 27525 leader_election.cc:290] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:05.338119 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 29 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
W20260709 12:25:05.381078 18366 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59886 (request call id 2614) took 2347 ms (client timeout 1694 ms). Trace:
W20260709 12:25:05.381510 18366 rpcz_store.cc:269] 0709 12:25:03.034001 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:03.034101 (+   100us) service_pool.cc:224] Handling call
0709 12:25:05.380860 (+2346759us) spinlock_profiling.cc:243] Waited 1.29 s on lock 0x7b540047ca54. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:25:05.380971 (+   111us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2964123008}
I20260709 12:25:05.381964 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 33 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:05.339296 17654 leader_election.cc:304] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:05.382710 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
W20260709 12:25:05.383247 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 44037.6us
I20260709 12:25:05.383956 17758 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38776 (request call id 3326) took 1025 ms. Trace:
I20260709 12:25:05.384300 18363 raft_consensus.cc:1240] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 28. Current term is 29. Ops: []
I20260709 12:25:05.385121 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:25:05.384295 17758 rpcz_store.cc:276] 0709 12:25:04.358655 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:05.076144 (+717489us) service_pool.cc:224] Handling call
0709 12:25:05.076510 (+   366us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:25:05.076964 (+   454us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:05.077015 (+    51us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:25:05.095038 (+ 18023us) log.cc:844] Serialized 52 byte log entry
0709 12:25:05.309839 (+214801us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:25:05.309880 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:05.309972 (+    92us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:05.383730 (+ 73758us) raft_consensus.cc:1685] finished
0709 12:25:05.383750 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:05.383895 (+   145us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":7026,"raft.queue_time_us":1990,"raft.run_cpu_time_us":261,"raft.run_wall_time_us":11403,"thread_start_us":139964,"threads_started":1,"wal-append.queue_time_us":241903}
I20260709 12:25:05.386722 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 33 candidate_status { last_received { term: 31 index: 24 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:25:05.340790 27595 raft_consensus.cc:740] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:25:05.337862 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 32 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:05.357574 27359 raft_consensus.cc:3060] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:05.359048 18371 raft_consensus.cc:1240] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 30. Current term is 31. Ops: [28.21-30.22]
I20260709 12:25:05.248265 27461 raft_consensus.cc:515] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:05.361886 27600 log.cc:927] Time spent T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.052s	user 0.000s	sys 0.000s
I20260709 12:25:05.363375 18365 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 2808) took 1406 ms. Trace:
I20260709 12:25:05.368283 27432 raft_consensus.cc:2804] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:25:05.440379 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 27 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:05.441345 17689 raft_consensus.cc:2410] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for term 27 because replica has last-logged OpId of term: 26 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 17.
I20260709 12:25:05.442833 17680 pending_rounds.cc:85] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3: Aborting all ops after (but not including) 17
I20260709 12:25:05.443226 17680 pending_rounds.cc:107] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3: Aborting uncommitted NO_OP operation due to leader change: 23.18
I20260709 12:25:05.443545 17680 raft_consensus.cc:2889] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:25:05.368978 27442 consensus_meta.cc:287] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.861s	user 0.012s	sys 0.017s
W20260709 12:25:05.374817 27566 log.cc:927] Time spent T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.858s	user 0.009s	sys 0.004s
I20260709 12:25:05.375447 17756 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38786 (request call id 4147) took 1018 ms. Trace:
I20260709 12:25:05.376878 27578 consensus_queue.cc:260] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 28.19, Last appended by leader: 19, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:05.293227 27395 consensus_queue.cc:1048] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 18, Time since last communication: 0.045s
I20260709 12:25:05.378207 27436 raft_consensus.cc:515] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:05.379483 27561 consensus_meta.cc:287] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.858s	user 0.018s	sys 0.013s
I20260709 12:25:05.310182 27448 raft_consensus.cc:2749] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
W20260709 12:25:05.380250 17759 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38776 (request call id 3324) took 1766 ms (client timeout 1578 ms). Trace:
I20260709 12:25:05.379930 27522 consensus_queue.cc:237] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 26.20, Last appended by leader: 20, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:05.388089 18366 raft_consensus.cc:1275] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 23 index: 17. Preceding OpId from leader: term: 25 index: 18. (index mismatch)
I20260709 12:25:05.486479 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 33 candidate_status { last_received { term: 31 index: 24 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:05.487496 18366 raft_consensus.cc:2393] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 33: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:25:05.387863 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 28 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:05.389248 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 29 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:05.386840 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:05.397143 17684 raft_consensus.cc:2393] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 33: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:25:05.400168 18368 raft_consensus.cc:2410] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for term 33 because replica has last-logged OpId of term: 31 index: 21, which is greater than that of the candidate, which has last-logged OpId of term: 29 index: 20.
I20260709 12:25:05.387742 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 25 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:25:05.360422 27450 leader_election.cc:290] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:05.402051 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:05.494333 18364 raft_consensus.cc:2468] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 24.
I20260709 12:25:05.405606 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:05.495515 17685 raft_consensus.cc:2468] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 27.
W20260709 12:25:05.406719 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 215737us
I20260709 12:25:05.417963 18365 rpcz_store.cc:276] 0709 12:25:03.957170 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:04.305189 (+348019us) service_pool.cc:224] Handling call
0709 12:25:04.442708 (+137519us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:04.459300 (+ 16592us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:05.363088 (+903788us) spinlock_profiling.cc:243] Waited 137 ms on lock 0x7b5400466ed0. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a61f8ad 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:25:05.363295 (+   207us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":315502848}
I20260709 12:25:05.447971 27604 raft_consensus.cc:493] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:25:05.451246 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 27 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:25:05.502076 27443 consensus_meta.cc:287] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.845s	user 0.006s	sys 0.015s
I20260709 12:25:05.451298 17756 rpcz_store.cc:276] 0709 12:25:04.349300 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:05.033570 (+684270us) service_pool.cc:224] Handling call
0709 12:25:05.034005 (+   435us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:05.241363 (+207358us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:05.241413 (+    50us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:05.241468 (+    55us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:25:05.241511 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:05.366945 (+125434us) spinlock_profiling.cc:243] Waited 207 ms on lock 0x7b5400415ad4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:25:05.367011 (+    66us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:05.367197 (+   186us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":476361984}
I20260709 12:25:05.502446 27604 raft_consensus.cc:515] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:05.502583 27443 raft_consensus.cc:515] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:05.335677 27307 consensus_queue.cc:1048] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.118s
I20260709 12:25:05.383172 27607 consensus_queue.cc:1048] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.139s
I20260709 12:25:05.418004 27463 consensus_queue.cc:1059] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 26.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.066s
I20260709 12:25:05.491293 17837 raft_consensus.cc:2468] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 25.
I20260709 12:25:05.493139 17683 raft_consensus.cc:2393] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 28: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:25:05.453588 27442 raft_consensus.cc:2749] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:25:05.499756 27559 consensus_queue.cc:260] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 26.16, Last appended by leader: 16, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:05.500913 27436 leader_election.cc:290] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 27 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:05.520668 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:05.501914 18363 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59902 (request call id 2269) took 2349 ms. Trace:
I20260709 12:25:05.334981 27428 raft_consensus.cc:515] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:05.507694 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:05.508818 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 29 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:05.508970 17834 raft_consensus.cc:2468] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 29.
I20260709 12:25:05.509218 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 29 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:05.488938 27561 raft_consensus.cc:2749] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:25:05.402060 27438 raft_consensus.cc:515] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:05.512844 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 28 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:05.524228 17832 raft_consensus.cc:1240] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 28 LEADER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 27. Current term is 28. Ops: []
I20260709 12:25:05.524587 17838 raft_consensus.cc:2393] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 28: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:25:05.513559 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:05.513808 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:25:05.481920 17759 rpcz_store.cc:269] 0709 12:25:03.613605 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:03.613753 (+   148us) service_pool.cc:224] Handling call
0709 12:25:05.380055 (+1766302us) spinlock_profiling.cc:243] Waited 760 ms on lock 0x7b54000ff0d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:25:05.380187 (+   132us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1748159360}
I20260709 12:25:05.521522 18363 rpcz_store.cc:276] 0709 12:25:03.152782 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:03.252724 (+ 99942us) service_pool.cc:224] Handling call
0709 12:25:05.384023 (+2131299us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:05.385127 (+  1104us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:05.501634 (+116507us) spinlock_profiling.cc:243] Waited 2.13 s on lock 0x7b540047ca50. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a61f8ad 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:25:05.501808 (+   174us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4900447616}
I20260709 12:25:05.512867 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:05.515781 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:05.527431 17837 raft_consensus.cc:2468] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 29.
W20260709 12:25:05.491633 27363 consensus_meta.cc:287] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 1.232s	user 0.015s	sys 0.007s
W20260709 12:25:05.493657 27427 consensus_meta.cc:287] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.969s	user 0.017s	sys 0.001s
I20260709 12:25:05.527748 18361 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59902 (request call id 2272) took 2373 ms. Trace:
W20260709 12:25:05.518142 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 196458us
I20260709 12:25:05.480648 27446 raft_consensus.cc:493] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:25:05.523636 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:05.551291 17755 raft_consensus.cc:2368] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for earlier term 26. Current term is 27.
I20260709 12:25:05.509955 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 33 candidate_status { last_received { term: 31 index: 24 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:05.552155 17760 raft_consensus.cc:3060] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:25:05.525801 17833 raft_consensus.cc:2468] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 27.
I20260709 12:25:05.518426 27394 leader_election.cc:290] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:05.543718 18361 rpcz_store.cc:276] 0709 12:25:03.154001 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:03.154140 (+   139us) service_pool.cc:224] Handling call
0709 12:25:03.154533 (+   393us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:03.377251 (+222718us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:03.377298 (+    47us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:03.377931 (+   633us) log.cc:844] Serialized 26 byte log entry
0709 12:25:04.171568 (+793637us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:25:04.171628 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:04.361843 (+190215us) spinlock_profiling.cc:243] Waited 222 ms on lock 0x7b540034b554. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:25:04.361978 (+   135us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:05.374114 (+1012136us) spinlock_profiling.cc:243] Waited 1.77 ms on lock 0x7b4400022b4c. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b47459c85 00007f7b4745a6a2 00007f7b4a1cc1db 00007f7b4a1c7aaf 00007f7b4a1bb86e 00007f7b4a1babb6 00007f7b4a1ba7d8 00007f7b4a1c4221 00007f7b4a1c41d9 00007f7b4a1c4169
0709 12:25:05.527540 (+153426us) raft_consensus.cc:1685] finished
0709 12:25:05.527563 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:05.527692 (+   129us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":41770,"raft.run_cpu_time_us":126,"raft.run_wall_time_us":124,"spinlock_wait_cycles":515191936,"thread_start_us":793140,"threads_started":1,"wal-append.queue_time_us":1083074}
I20260709 12:25:05.574790 18362 raft_consensus.cc:2468] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 25.
I20260709 12:25:05.576756 17682 raft_consensus.cc:2468] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 25.
W20260709 12:25:05.582864 17655 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:05.585964 27597 log.cc:927] Time spent T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.405s	user 0.002s	sys 0.000s
I20260709 12:25:05.520721 27245 raft_consensus.cc:493] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:25:05.601999 27245 raft_consensus.cc:515] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:05.604317 27462 leader_election.cc:290] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:05.596228 17757 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38786 (request call id 4140) took 1362 ms. Trace:
I20260709 12:25:05.605693 17757 rpcz_store.cc:276] 0709 12:25:04.234126 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:04.235008 (+   882us) service_pool.cc:224] Handling call
0709 12:25:04.235304 (+   296us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:04.703409 (+468105us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:05.595806 (+892397us) spinlock_profiling.cc:243] Waited 406 ms on lock 0x7b54004155d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:25:05.596129 (+   323us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":934391680}
I20260709 12:25:05.598209 17761 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38786 (request call id 4142) took 1363 ms. Trace:
I20260709 12:25:05.606737 17761 rpcz_store.cc:276] 0709 12:25:04.234742 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:04.236561 (+  1819us) service_pool.cc:224] Handling call
0709 12:25:04.236807 (+   246us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:04.345617 (+108810us) raft_consensus.cc:1534] Early marking committed up to index 19
0709 12:25:04.345842 (+   225us) log.cc:844] Serialized 14 byte log entry
0709 12:25:05.175970 (+830128us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:05.176020 (+    50us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:25:05.176065 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:05.597126 (+421061us) spinlock_profiling.cc:243] Waited 9.24 ms on lock 0x7b180040ef40. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b475012a6 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419
0709 12:25:05.597876 (+   750us) spinlock_profiling.cc:243] Waited 109 ms on lock 0x7b540019fad4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:25:05.597929 (+    53us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:05.598124 (+   195us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":6164,"prepare.run_cpu_time_us":66,"prepare.run_wall_time_us":10281,"raft.queue_time_us":99,"raft.run_cpu_time_us":142,"raft.run_wall_time_us":6758,"spinlock_wait_cycles":270989440,"thread_start_us":829795,"threads_started":1,"wal-append.queue_time_us":886247}
I20260709 12:25:05.615114 27415 raft_consensus.cc:3055] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:25:05.644407 27415 raft_consensus.cc:740] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:25:05.646515 27415 consensus_queue.cc:260] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 26.13, Last appended by leader: 13, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:05.621184 17763 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38786 (request call id 4143) took 1379 ms. Trace:
I20260709 12:25:05.661669 17763 rpcz_store.cc:276] 0709 12:25:04.241942 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:04.243312 (+  1370us) service_pool.cc:224] Handling call
0709 12:25:05.620834 (+1377522us) spinlock_profiling.cc:243] Waited 1.18 s on lock 0x7b54003e4ed4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:25:05.621061 (+   227us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2707128704}
W20260709 12:25:05.623337 27565 log.cc:927] Time spent T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 1.112s	user 0.010s	sys 0.000s
I20260709 12:25:05.630219 17802 leader_election.cc:304] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:05.641966 18362 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59918 (request call id 2815) took 1310 ms. Trace:
I20260709 12:25:05.650290 17834 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35040 (request call id 3678) took 1191 ms. Trace:
I20260709 12:25:05.664906 17837 raft_consensus.cc:1275] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 29 index: 17. Preceding OpId from leader: term: 30 index: 18. (index mismatch)
I20260709 12:25:05.665251 17834 rpcz_store.cc:276] 0709 12:25:04.459196 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:04.459305 (+   109us) service_pool.cc:224] Handling call
0709 12:25:05.650132 (+1190827us) spinlock_profiling.cc:243] Waited 750 ms on lock 0x7b54003345d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:25:05.650232 (+   100us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1723954432}
I20260709 12:25:05.651041 27567 raft_consensus.cc:3055] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:25:05.591696 27441 raft_consensus.cc:493] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:05.618620 27363 raft_consensus.cc:493] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:25:05.604451 27245 leader_election.cc:290] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:05.664407 18362 rpcz_store.cc:276] 0709 12:25:04.331682 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:04.404647 (+ 72965us) service_pool.cc:224] Handling call
0709 12:25:05.641710 (+1237063us) spinlock_profiling.cc:243] Waited 1.13 s on lock 0x7b5400552654. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:25:05.641876 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2606941440}
I20260709 12:25:05.576897 17760 raft_consensus.cc:2468] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 33.
I20260709 12:25:05.574153 27436 raft_consensus.cc:493] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:05.566650 27559 raft_consensus.cc:3060] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:25:05.697844 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 251641us
I20260709 12:25:05.520179 27608 consensus_queue.cc:1059] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 27.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.163s
I20260709 12:25:05.691911 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 26 candidate_status { last_received { term: 24 index: 20 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:25:05.664393 27462 raft_consensus.cc:493] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:05.692652 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:25:05.693316 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 34 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:05.699196 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 27 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:05.690591 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 28 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:05.718569 17682 raft_consensus.cc:2410] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 28 because replica has last-logged OpId of term: 26 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 15.
W20260709 12:25:05.699762 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 853871us
I20260709 12:25:05.701958 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:05.703548 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 33 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:05.704308 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:05.720176 17831 raft_consensus.cc:3060] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Advancing to term 29
W20260709 12:25:05.720582 17803 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:05.706142 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 29 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:05.706887 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 33 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:05.709602 17687 raft_consensus.cc:3060] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Advancing to term 26
W20260709 12:25:05.676596 17804 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:05.653695 27432 raft_consensus.cc:697] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 24 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:25:05.728370 27432 consensus_queue.cc:237] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 23.16, Last appended by leader: 16, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:05.712682 27245 raft_consensus.cc:493] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:05.731515 27245 raft_consensus.cc:515] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:05.734265 27245 leader_election.cc:290] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:05.707547 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:05.680769 27196 raft_consensus.cc:3060] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:05.562652 27427 raft_consensus.cc:515] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:05.736510 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:05.677346 27567 raft_consensus.cc:740] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:25:05.679826 27062 raft_consensus.cc:493] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:25:05.713299 17652 leader_election.cc:304] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:25:05.743713 17831 raft_consensus.cc:2468] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 29.
I20260709 12:25:05.565801 27446 raft_consensus.cc:515] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:05.688704 27394 raft_consensus.cc:2749] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 12:25:05.509733 27595 consensus_queue.cc:260] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 23.16, Last appended by leader: 16, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:05.745106 27062 raft_consensus.cc:515] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:05.757875 27609 log.cc:927] Time spent T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.070s	user 0.000s	sys 0.000s
I20260709 12:25:05.748484 27567 consensus_queue.cc:260] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 24.17, Last appended by leader: 17, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:05.755923 27443 leader_election.cc:290] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 26 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:05.758615 17683 raft_consensus.cc:1240] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 21. Current term is 22. Ops: []
W20260709 12:25:05.748183 27614 log.cc:927] Time spent T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.071s	user 0.001s	sys 0.000s
I20260709 12:25:05.710206 27509 raft_consensus.cc:493] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:05.761415 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:25:05.738622 27572 log.cc:927] Time spent T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.060s	user 0.000s	sys 0.001s
I20260709 12:25:05.700486 27564 consensus_queue.cc:1059] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 27.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.075s
W20260709 12:25:05.767427 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 54504.5us
I20260709 12:25:05.710486 27111 raft_consensus.cc:493] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:05.769214 17830 raft_consensus.cc:1275] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 23 index: 17. Preceding OpId from leader: term: 25 index: 18. (index mismatch)
W20260709 12:25:05.770156 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:05.761032 27463 raft_consensus.cc:493] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:25:05.711902 27436 raft_consensus.cc:515] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:05.773592 17835 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35040 (request call id 3673) took 2779 ms. Trace:
I20260709 12:25:05.774312 17756 raft_consensus.cc:3060] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:05.696103 27363 raft_consensus.cc:515] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:05.805917 17835 rpcz_store.cc:276] 0709 12:25:02.993752 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:03.029293 (+ 35541us) service_pool.cc:224] Handling call
0709 12:25:03.029599 (+   306us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:03.029925 (+   326us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:03.029968 (+    43us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:03.030452 (+   484us) log.cc:844] Serialized 26 byte log entry
0709 12:25:04.163070 (+1132618us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:25:04.163110 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:04.163181 (+    71us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:05.622555 (+1459374us) spinlock_profiling.cc:243] Waited 5.78 ms on lock 0x7b4400022b4c. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b47459c85 00007f7b4745a6a2 00007f7b4a1cc1db 00007f7b4a1c7aaf 00007f7b4a1bb86e 00007f7b4a1babb6 00007f7b4a1ba7d8 00007f7b4a1c4221 00007f7b4a1c41d9 00007f7b4a1c4169
0709 12:25:05.768844 (+146289us) raft_consensus.cc:1685] finished
0709 12:25:05.768866 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:05.773467 (+  4601us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":413,"raft.queue_time_us":6171,"raft.run_cpu_time_us":158,"raft.run_wall_time_us":6949,"spinlock_wait_cycles":13285760,"thread_start_us":1132215,"threads_started":1,"wal-append.queue_time_us":1395131}
I20260709 12:25:05.805971 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 33 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:05.827351 17689 raft_consensus.cc:2468] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 32.
W20260709 12:25:05.781541 27590 consensus_meta.cc:287] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.600s	user 0.011s	sys 0.000s
I20260709 12:25:05.766834 17687 raft_consensus.cc:2468] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 26.
W20260709 12:25:05.805155 18334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:05.694237 27441 raft_consensus.cc:515] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:05.837677 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:25:05.838891 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:05.829998 17829 raft_consensus.cc:1240] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 26. Current term is 27. Ops: [26.16-26.16]
I20260709 12:25:05.839540 17685 raft_consensus.cc:2468] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 24.
I20260709 12:25:05.840348 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:05.833396 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 32 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:05.743261 27462 raft_consensus.cc:515] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:25:05.787674 27611 log.cc:927] Time spent T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.071s	user 0.001s	sys 0.000s
I20260709 12:25:05.782960 27567 raft_consensus.cc:3060] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:05.784229 27595 raft_consensus.cc:3060] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:05.820935 17652 leader_election.cc:304] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:05.796650 27438 leader_election.cc:290] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 27 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:05.783293 27604 leader_election.cc:290] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:05.858500 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:25:05.802289 27428 leader_election.cc:290] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:05.862979 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 27 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:25:05.789255 27062 leader_election.cc:290] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:05.801750 27427 leader_election.cc:290] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 22 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:05.801222 27446 leader_election.cc:290] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:05.866145 17687 raft_consensus.cc:3060] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:05.867378 18334 leader_election.cc:304] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:25:05.871950 17685 raft_consensus.cc:1240] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 30. Current term is 31. Ops: []
I20260709 12:25:05.875437 17756 raft_consensus.cc:2468] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 28.
I20260709 12:25:05.876710 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:05.804173 27617 raft_consensus.cc:2804] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:25:05.880430 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:05.904532 17835 raft_consensus.cc:1240] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 28. Current term is 29. Ops: []
I20260709 12:25:05.928009 17685 raft_consensus.cc:1240] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 25. Current term is 26. Ops: []
W20260709 12:25:05.905838 18333 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:05.929744 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:05.930521 18365 raft_consensus.cc:3060] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:05.907697 27462 leader_election.cc:290] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:05.932926 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 27 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:05.933584 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:05.933993 18364 raft_consensus.cc:3060] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:05.934363 18368 raft_consensus.cc:3060] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:05.910794 17728 leader_election.cc:304] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:25:05.912465 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 224079us
W20260709 12:25:05.938588 17727 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:05.913160 27585 consensus_meta.cc:287] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.748s	user 0.008s	sys 0.003s
I20260709 12:25:05.946945 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 33 candidate_status { last_received { term: 31 index: 24 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:05.947928 17763 raft_consensus.cc:2393] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 33: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:25:05.913383 27197 raft_consensus.cc:3060] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Advancing to term 31
W20260709 12:25:05.913987 27615 log.cc:927] Time spent T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.129s	user 0.000s	sys 0.000s
I20260709 12:25:05.841427 27463 raft_consensus.cc:515] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:05.968896 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:05.970098 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 31 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:05.918304 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:05.919162 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 28 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:05.847699 27441 leader_election.cc:290] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:05.922035 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 31 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
W20260709 12:25:05.922827 17654 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:05.922976 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 29 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:05.924186 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 28 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:05.996450 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:05.950268 17803 leader_election.cc:304] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:05.830158 27111 raft_consensus.cc:515] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:05.865120 27564 raft_consensus.cc:493] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:05.954262 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 133496us
I20260709 12:25:05.958788 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 25 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:05.917418 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
W20260709 12:25:05.910840 27625 log.cc:927] Time spent T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.108s	user 0.000s	sys 0.001s
I20260709 12:25:05.975237 17687 raft_consensus.cc:2468] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 27.
I20260709 12:25:05.843142 27590 raft_consensus.cc:515] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:05.970674 18371 raft_consensus.cc:2468] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 30.
I20260709 12:25:06.013697 18364 raft_consensus.cc:2468] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 27.
I20260709 12:25:06.018265 17763 raft_consensus.cc:2410] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for term 25 because replica has last-logged OpId of term: 23 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 16.
I20260709 12:25:05.978787 18365 raft_consensus.cc:2468] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 26.
I20260709 12:25:05.985800 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:06.019142 18366 raft_consensus.cc:2468] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 24.
I20260709 12:25:06.020566 17687 raft_consensus.cc:1240] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 26. Current term is 27. Ops: []
W20260709 12:25:06.013901 27008 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.632s	user 0.022s	sys 0.037s
W20260709 12:25:05.968065 27606 log.cc:927] Time spent T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.424s	user 0.000s	sys 0.001s
I20260709 12:25:05.901806 27370 raft_consensus.cc:493] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:05.905093 27338 raft_consensus.cc:493] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:05.996608 17759 raft_consensus.cc:2393] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 28: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:25:05.779932 27509 raft_consensus.cc:515] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:06.000200 17761 raft_consensus.cc:2368] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 27. Current term is 28.
I20260709 12:25:06.000959 17757 raft_consensus.cc:2393] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 28: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
W20260709 12:25:06.003360 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 97657.7us
I20260709 12:25:06.023180 27111 leader_election.cc:290] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:25:06.024674 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 74547.7us
W20260709 12:25:06.025344 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 114678us
I20260709 12:25:06.040572 17728 leader_election.cc:304] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:05.991317 27607 raft_consensus.cc:2749] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
W20260709 12:25:06.022121 27008 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.640s	user 0.023s	sys 0.037s
I20260709 12:25:05.947424 27577 raft_consensus.cc:493] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:25:05.947121 27245 raft_consensus.cc:493] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:25:06.023664 27582 raft_consensus.cc:493] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:05.962965 27617 raft_consensus.cc:697] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 28 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:25:06.029690 27062 raft_consensus.cc:2749] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:25:05.988678 18368 raft_consensus.cc:2468] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 22.
I20260709 12:25:06.042668 17803 leader_election.cc:304] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:25:06.027267 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:06.053385 17757 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3"
dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
mode: GRACEFUL
new_leader_uuid: "e063541542a94969898868bf6eb5ce1b"
 from {username='slave'} at 127.0.0.1:49786
I20260709 12:25:06.064313 17757 raft_consensus.cc:606] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Received request to transfer leadership to TS e063541542a94969898868bf6eb5ce1b
I20260709 12:25:05.962430 27585 raft_consensus.cc:515] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:06.042418 27451 raft_consensus.cc:493] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:06.030580 27324 raft_consensus.cc:2804] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:25:06.037190 17802 leader_election.cc:304] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:06.060206 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:06.040509 27461 leader_election.cc:290] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 27 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:06.040084 27525 raft_consensus.cc:493] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:06.026548 27564 raft_consensus.cc:515] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:06.039716 27370 raft_consensus.cc:515] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:06.253446 17804 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:06.171713 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:06.292567 18366 raft_consensus.cc:2468] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 24.
I20260709 12:25:06.064587 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:06.162002 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:06.044071 27475 raft_consensus.cc:493] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:06.045821 27428 raft_consensus.cc:2804] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:25:06.050824 27577 raft_consensus.cc:515] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:06.045634 27338 raft_consensus.cc:515] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:06.005146 17652 leader_election.cc:304] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:25:06.036154 17727 leader_election.cc:304] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 31 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
W20260709 12:25:06.596958 17803 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.524s	user 0.018s	sys 0.022s
I20260709 12:25:06.485286 17680 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44898 (request call id 1939) took 1096 ms. Trace:
W20260709 12:25:06.486559 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:06.294395 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 257282us
I20260709 12:25:06.597971 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:06.600906 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:25:06.483563 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 486536us
I20260709 12:25:06.601977 17831 raft_consensus.cc:2468] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 27.
I20260709 12:25:06.601853 17830 raft_consensus.cc:2468] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 21.
W20260709 12:25:06.597971 17803 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.525s	user 0.019s	sys 0.022s
W20260709 12:25:06.603861 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 531225us
I20260709 12:25:06.603942 17654 leader_election.cc:304] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:06.604280 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 31 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:06.606779 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 25 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:06.559970 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 34 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:25:06.609468 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 302541us
I20260709 12:25:06.610911 18333 leader_election.cc:304] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 33 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:25:06.491719 17655 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:06.586618 17763 raft_consensus.cc:1240] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 24. Current term is 25. Ops: []
W20260709 12:25:06.612154 18334 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.665s	user 0.001s	sys 0.009s
W20260709 12:25:06.612633 18334 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.666s	user 0.001s	sys 0.009s
I20260709 12:25:06.612870 17838 raft_consensus.cc:1240] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 24. Current term is 25. Ops: []
W20260709 12:25:06.613130 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 745789us
W20260709 12:25:06.614272 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 609044us
W20260709 12:25:06.615278 18334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:06.615127 17803 leader_election.cc:304] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:06.572543 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:06.618664 17802 leader_election.cc:304] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:25:06.620922 17803 leader_election.cc:304] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:25:06.530416 17756 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38776 (request call id 3334) took 1023 ms. Trace:
I20260709 12:25:06.625041 18333 leader_election.cc:400] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 19 election: Vote denied by peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837) with higher term. Message: Invalid argument: T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 19. Current term is 20.
I20260709 12:25:06.638056 18333 leader_election.cc:403] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 19 election: Cancelling election due to peer responding with higher term
W20260709 12:25:06.589679 17760 consensus_meta.cc:287] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.569s	user 0.010s	sys 0.011s
I20260709 12:25:06.626441 17756 rpcz_store.cc:276] 0709 12:25:05.506851 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:05.508197 (+  1346us) service_pool.cc:224] Handling call
0709 12:25:06.529913 (+1021716us) spinlock_profiling.cc:243] Waited 253 ms on lock 0x7b5400310e54. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:25:06.530173 (+   260us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":581658368}
I20260709 12:25:06.064679 17757 raft_consensus.cc:614] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:25:06.270673 27196 raft_consensus.cc:2749] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
W20260709 12:25:06.088402 27619 log.cc:927] Time spent T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.141s	user 0.001s	sys 0.000s
W20260709 12:25:06.285439 27359 consensus_meta.cc:287] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.882s	user 0.014s	sys 0.000s
I20260709 12:25:06.598017 17680 rpcz_store.cc:276] 0709 12:25:05.388251 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:05.388383 (+   132us) service_pool.cc:224] Handling call
0709 12:25:05.442565 (+ 54182us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:25:05.503994 (+ 61429us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:05.504029 (+    35us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:25:05.504603 (+   574us) log.cc:844] Serialized 52 byte log entry
0709 12:25:05.683657 (+179054us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:25:05.683729 (+    72us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:05.683804 (+    75us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:05.910103 (+226299us) spinlock_profiling.cc:243] Waited 1.47 ms on lock 0x7b4400022b4c. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b47459c85 00007f7b4745a6a2 00007f7b4a1cc1db 00007f7b4a1c7aaf 00007f7b4a1bb86e 00007f7b4a1babb6 00007f7b4a1ba7d8 00007f7b4a1c4221 00007f7b4a1c41d9 00007f7b4a1c4169
0709 12:25:06.484012 (+573909us) raft_consensus.cc:1685] finished
0709 12:25:06.484101 (+    89us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:06.484840 (+   739us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":13650,"raft.queue_time_us":123747,"raft.run_cpu_time_us":128,"raft.run_wall_time_us":9471,"spinlock_wait_cycles":3389184,"thread_start_us":178555,"threads_started":1,"wal-append.queue_time_us":274290}
W20260709 12:25:06.338651 27618 log.cc:927] Time spent T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.232s	user 0.000s	sys 0.001s
I20260709 12:25:06.665447 27629 raft_consensus.cc:2749] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 12:25:06.666329 27642 raft_consensus.cc:2804] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Leader election won for term 26
W20260709 12:25:06.348734 27634 log.cc:927] Time spent T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.255s	user 0.002s	sys 0.000s
I20260709 12:25:06.650731 17760 raft_consensus.cc:2468] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 28.
W20260709 12:25:06.401798 27567 consensus_meta.cc:287] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.542s	user 0.010s	sys 0.001s
I20260709 12:25:06.169337 27431 raft_consensus.cc:3055] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 27 LEADER]: Stepping down as leader of term 27
W20260709 12:25:06.351183 27631 log.cc:927] Time spent T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.262s	user 0.001s	sys 0.000s
I20260709 12:25:06.073680 27449 raft_consensus.cc:2804] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:25:06.672317 27428 raft_consensus.cc:493] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 32 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:25:06.700649 27428 raft_consensus.cc:3060] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:25:06.056133 27245 raft_consensus.cc:515] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:06.704511 27008 consensus_queue.cc:1059] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 25.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:25:06.139029 27363 leader_election.cc:290] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:06.706282 27245 leader_election.cc:290] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:06.706602 27369 raft_consensus.cc:2749] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:25:06.151029 27617 consensus_queue.cc:237] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 27.16, Last appended by leader: 16, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:06.707598 27369 raft_consensus.cc:3055] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:25:06.708098 27369 raft_consensus.cc:740] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 25 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:25:06.709441 27369 consensus_queue.cc:260] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 25.18, Last appended by leader: 18, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:06.675935 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 27 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:06.159497 17758 raft_consensus.cc:2468] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 27.
I20260709 12:25:06.154482 27415 raft_consensus.cc:3060] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:06.283344 27561 raft_consensus.cc:493] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:25:06.153415 27590 leader_election.cc:290] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 30 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:06.677556 27443 raft_consensus.cc:2804] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:25:06.104247 27632 consensus_queue.cc:1048] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.594s
I20260709 12:25:06.676062 27639 raft_consensus.cc:2749] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 33 FOLLOWER]: Leader election lost for term 33. Reason: could not achieve majority
I20260709 12:25:06.667834 27577 leader_election.cc:290] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:06.152542 27509 leader_election.cc:290] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:06.349217 27395 raft_consensus.cc:3055] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:25:06.159631 27436 leader_election.cc:290] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:06.681218 27338 leader_election.cc:290] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:06.151597 27578 raft_consensus.cc:3060] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:06.682121 18368 raft_consensus.cc:2376] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 29. Re-sending same reply.
I20260709 12:25:06.684053 27638 raft_consensus.cc:493] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:25:06.062889 27582 raft_consensus.cc:515] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:06.685706 17759 raft_consensus.cc:2393] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 31: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
W20260709 12:25:06.418550 27595 consensus_meta.cc:287] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.550s	user 0.000s	sys 0.011s
I20260709 12:25:06.662979 17654 leader_election.cc:304] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:06.164517 27463 leader_election.cc:290] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:25:06.747989 27449 raft_consensus.cc:493] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:25:06.748602 27449 raft_consensus.cc:3060] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:06.322695 27324 raft_consensus.cc:697] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 25 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:25:06.751204 27324 consensus_queue.cc:237] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 23.18, Last appended by leader: 18, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:06.752003 27395 raft_consensus.cc:740] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:25:06.753633 17758 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38776 (request call id 3335) took 1246 ms. Trace:
I20260709 12:25:06.753422 27395 consensus_queue.cc:260] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 27.17, Last appended by leader: 17, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:06.754014 17758 rpcz_store.cc:276] 0709 12:25:05.507275 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:05.507433 (+   158us) service_pool.cc:224] Handling call
0709 12:25:06.753311 (+1245878us) spinlock_profiling.cc:243] Waited 637 ms on lock 0x7b54003bd3d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:25:06.753463 (+   152us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1464382720}
I20260709 12:25:06.063944 27448 raft_consensus.cc:2804] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:25:06.755322 27448 raft_consensus.cc:493] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:06.755813 27448 raft_consensus.cc:3060] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:06.698751 18362 raft_consensus.cc:2468] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 23.
I20260709 12:25:06.758689 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 24 candidate_status { last_received { term: 21 index: 20 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:06.759346 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 27 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:06.759809 18362 raft_consensus.cc:2410] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 24 because replica has last-logged OpId of term: 23 index: 21, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 20.
I20260709 12:25:06.760989 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 27 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:06.761142 27475 raft_consensus.cc:515] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:06.762436 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:06.763397 18362 raft_consensus.cc:2468] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 24.
I20260709 12:25:06.676424 27442 raft_consensus.cc:2804] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:25:06.763782 18374 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59886 (request call id 2632) took 1428 ms. Trace:
I20260709 12:25:06.764050 27442 raft_consensus.cc:493] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:06.764479 27442 raft_consensus.cc:3060] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:06.764219 18374 rpcz_store.cc:276] 0709 12:25:05.334966 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:05.336932 (+  1966us) service_pool.cc:224] Handling call
0709 12:25:05.337335 (+   403us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:05.341077 (+  3742us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:05.341110 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:05.341555 (+   445us) log.cc:844] Serialized 26 byte log entry
0709 12:25:05.578315 (+236760us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:25:05.578358 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:05.670627 (+ 92269us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:06.763256 (+1092629us) raft_consensus.cc:1685] finished
0709 12:25:06.763324 (+    68us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:06.763551 (+   227us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3291,"raft.queue_time_us":3447,"raft.run_cpu_time_us":131,"raft.run_wall_time_us":132,"thread_start_us":125679,"threads_started":1,"wal-append.queue_time_us":430478}
I20260709 12:25:06.743072 27406 consensus_queue.cc:1059] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 30.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.024s
I20260709 12:25:06.767306 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 26 candidate_status { last_received { term: 24 index: 20 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:06.768433 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 31 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:06.769467 18366 raft_consensus.cc:2393] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 31: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
W20260709 12:25:06.772055 17729 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.624s	user 0.006s	sys 0.010s
W20260709 12:25:06.772547 17729 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.736s	user 0.007s	sys 0.010s
I20260709 12:25:06.303742 27451 raft_consensus.cc:515] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:06.774605 27640 raft_consensus.cc:2749] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Leader election lost for term 31. Reason: could not achieve majority
W20260709 12:25:06.775409 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 739111us
I20260709 12:25:06.775952 18368 raft_consensus.cc:1275] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 26 index: 19. Preceding OpId from leader: term: 28 index: 20. (index mismatch)
I20260709 12:25:06.310168 27525 raft_consensus.cc:515] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:06.777864 17686 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44898 (request call id 1940) took 1389 ms. Trace:
I20260709 12:25:06.780157 27643 raft_consensus.cc:493] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:25:06.780939 27643 raft_consensus.cc:515] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:06.781968 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:06.782856 17756 raft_consensus.cc:2468] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 30.
I20260709 12:25:06.784013 27638 raft_consensus.cc:515] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:06.785053 17680 raft_consensus.cc:1240] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 26. Current term is 27. Ops: []
I20260709 12:25:06.790366 17689 raft_consensus.cc:1240] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 29. Current term is 30. Ops: []
I20260709 12:25:06.725015 27443 raft_consensus.cc:493] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:25:06.304561 27630 raft_consensus.cc:2749] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:25:06.723129 27369 raft_consensus.cc:3060] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Advancing to term 26
W20260709 12:25:06.679795 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 643607us
I20260709 12:25:06.729368 27431 raft_consensus.cc:740] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 27 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:25:06.344610 27585 leader_election.cc:290] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 25 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:06.348009 27564 leader_election.cc:290] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:06.349941 27370 leader_election.cc:290] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:06.742422 27338 raft_consensus.cc:493] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:06.742211 27561 raft_consensus.cc:515] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:06.743419 27359 raft_consensus.cc:2749] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Leader election lost for term 19. Reason: Vote denied by peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837) with higher term. Message: Invalid argument: T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 19. Current term is 20.
W20260709 12:25:06.719925 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 51658.2us
I20260709 12:25:06.756599 18361 raft_consensus.cc:1275] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 28 index: 20. Preceding OpId from leader: term: 29 index: 20. (term mismatch)
I20260709 12:25:06.807525 27642 raft_consensus.cc:697] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 26 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:25:06.824743 27561 leader_election.cc:290] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:06.826436 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 27 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:06.825906 27642 consensus_queue.cc:237] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 24.20, Last appended by leader: 20, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:06.820181 27245 raft_consensus.cc:2749] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:25:06.828928 27338 raft_consensus.cc:515] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:06.829504 17804 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:06.830039 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:06.830803 17764 raft_consensus.cc:2468] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 29.
I20260709 12:25:06.831341 27338 leader_election.cc:290] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:06.832589 17760 raft_consensus.cc:1240] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 27. Current term is 28. Ops: []
I20260709 12:25:06.835402 18366 raft_consensus.cc:1275] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 18 index: 14. Preceding OpId from leader: term: 20 index: 15. (index mismatch)
I20260709 12:25:06.805836 27443 raft_consensus.cc:3060] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:06.823001 27638 leader_election.cc:290] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:06.778293 17686 rpcz_store.cc:276] 0709 12:25:05.388537 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:05.388833 (+   296us) service_pool.cc:224] Handling call
0709 12:25:05.397802 (+  8969us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:05.680616 (+282814us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:05.680652 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:05.693459 (+ 12807us) log.cc:844] Serialized 26 byte log entry
0709 12:25:06.028115 (+334656us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:25:06.028163 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:06.028228 (+    65us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:06.348020 (+319792us) spinlock_profiling.cc:243] Waited 997 us on lock 0x7b4400022b4c. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b47459c85 00007f7b4745a6a2 00007f7b4a1cc1db 00007f7b4a1c7aaf 00007f7b4a1bb86e 00007f7b4a1babb6 00007f7b4a1ba7d8 00007f7b4a1c4221 00007f7b4a1c41d9 00007f7b4a1c4169
0709 12:25:06.777327 (+429307us) raft_consensus.cc:1685] finished
0709 12:25:06.777379 (+    52us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:06.777705 (+   326us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":8436,"raft.queue_time_us":147513,"raft.run_cpu_time_us":209,"raft.run_wall_time_us":25869,"spinlock_wait_cycles":2292480,"thread_start_us":539887,"threads_started":2,"wal-append.queue_time_us":352449}
I20260709 12:25:06.821372 18361 pending_rounds.cc:85] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da: Aborting all ops after (but not including) 19
I20260709 12:25:06.823096 27509 raft_consensus.cc:3055] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:25:06.844087 17727 leader_election.cc:304] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:06.844466 27509 raft_consensus.cc:740] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 30 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:25:06.845232 27585 raft_consensus.cc:2749] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260709 12:25:06.845570 27509 consensus_queue.cc:260] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 30.17, Last appended by leader: 17, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:06.843966 18361 pending_rounds.cc:107] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da: Aborting uncommitted NO_OP operation due to leader change: 28.20
I20260709 12:25:06.847234 17727 leader_election.cc:304] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:06.847430 18361 raft_consensus.cc:2889] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:06.848026 27509 raft_consensus.cc:3060] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:06.848528 27544 raft_consensus.cc:493] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:06.848737 27582 leader_election.cc:290] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:06.849133 27544 raft_consensus.cc:515] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:06.850732 27582 raft_consensus.cc:2804] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:25:06.851279 27582 raft_consensus.cc:697] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
W20260709 12:25:06.854013 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 238605us
I20260709 12:25:06.854050 27582 consensus_queue.cc:237] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 27.16, Last appended by leader: 16, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:06.817271 27431 consensus_queue.cc:260] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 27.15, Last appended by leader: 15, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:06.851631 27525 leader_election.cc:290] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:06.888614 18374 raft_consensus.cc:3060] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Advancing to term 26
W20260709 12:25:06.904501 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 296661us
I20260709 12:25:06.906008 17654 leader_election.cc:304] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:25:06.907136 27617 raft_consensus.cc:2764] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Leader pre-election decision vote started in defunct term 25: won
I20260709 12:25:06.910681 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 27 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:06.912107 17760 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 26.
W20260709 12:25:06.922336 18334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:06.936210 27451 leader_election.cc:290] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:25:06.937449 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 312704us
I20260709 12:25:06.938606 18333 leader_election.cc:304] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:06.939459 17729 leader_election.cc:304] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:25:06.941975 17729 leader_election.cc:304] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:06.936702 17652 leader_election.cc:304] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:25:06.934302 17654 leader_election.cc:304] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:25:06.948227 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 26 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:06.951179 27580 raft_consensus.cc:2749] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:25:06.954749 27660 raft_consensus.cc:493] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:06.957489 18367 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 2822) took 1033 ms. Trace:
I20260709 12:25:06.958487 17654 leader_election.cc:304] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:25:06.959093 27431 raft_consensus.cc:3060] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:06.959717 17654 leader_election.cc:304] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
W20260709 12:25:06.960037 27197 consensus_meta.cc:287] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.959s	user 0.002s	sys 0.022s
I20260709 12:25:06.960868 27197 raft_consensus.cc:2804] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:25:06.961438 27570 raft_consensus.cc:2804] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:25:06.962427 27570 raft_consensus.cc:697] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 26 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:25:06.964207 27617 raft_consensus.cc:2804] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:25:06.964649 27617 raft_consensus.cc:493] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:06.965010 27617 raft_consensus.cc:3060] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Advancing to term 31
W20260709 12:25:06.966526 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 44826.4us
I20260709 12:25:06.967316 27441 raft_consensus.cc:2804] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:25:06.967922 27441 raft_consensus.cc:493] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:06.968161 27008 consensus_queue.cc:1059] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 28.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.001s
I20260709 12:25:06.969085 27627 consensus_queue.cc:1048] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:25:06.969935 27475 leader_election.cc:290] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:06.970629 27062 raft_consensus.cc:3055] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:25:06.950044 27643 leader_election.cc:290] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:25:06.968463 27441 raft_consensus.cc:3060] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:06.961539 27197 raft_consensus.cc:697] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 27 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:25:06.971060 27062 raft_consensus.cc:740] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 28 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
W20260709 12:25:06.985314 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 46930.3us
I20260709 12:25:06.986059 17729 leader_election.cc:304] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:06.910573 27585 consensus_queue.cc:1048] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.039s
I20260709 12:25:06.920603 27245 consensus_queue.cc:1059] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 27.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.030s
I20260709 12:25:06.972924 17652 leader_election.cc:304] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:06.995534 27544 leader_election.cc:290] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:25:06.954926 27559 consensus_queue.cc:1059] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 26.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.044s
I20260709 12:25:07.056481 27559 raft_consensus.cc:493] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:25:07.057158 27559 raft_consensus.cc:515] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:07.060029 27604 raft_consensus.cc:3055] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:25:07.060565 27604 raft_consensus.cc:740] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 30 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:25:07.061295 18333 leader_election.cc:304] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:25:07.006091 27570 consensus_queue.cc:237] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 25.18, Last appended by leader: 18, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:07.063257 27436 consensus_queue.cc:1048] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:25:07.064261 27664 raft_consensus.cc:493] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:25:07.064942 27664 raft_consensus.cc:515] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:07.008419 27062 consensus_queue.cc:260] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 28.20, Last appended by leader: 20, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:07.067517 27062 raft_consensus.cc:3060] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:06.985350 17684 raft_consensus.cc:2468] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 31.
I20260709 12:25:06.956684 27128 leader_election.cc:290] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:06.992131 27475 raft_consensus.cc:2804] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:25:07.016736 27666 consensus_queue.cc:1059] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 26.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.001s
I20260709 12:25:07.014487 27443 raft_consensus.cc:515] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:06.955003 27270 consensus_queue.cc:1059] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 21.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.053s
I20260709 12:25:06.957922 18367 rpcz_store.cc:276] 0709 12:25:05.875722 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:05.877323 (+  1601us) service_pool.cc:224] Handling call
0709 12:25:05.940708 (+ 63385us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:06.303131 (+362423us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:06.303181 (+    50us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:06.303892 (+   711us) log.cc:844] Serialized 26 byte log entry
0709 12:25:06.814620 (+510728us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:25:06.836441 (+ 21821us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:06.859727 (+ 23286us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:06.859783 (+    56us) raft_consensus.cc:1685] finished
0709 12:25:06.859800 (+    17us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:06.909082 (+ 49282us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":23705,"raft.queue_time_us":196084,"raft.run_cpu_time_us":144,"raft.run_wall_time_us":2435,"thread_start_us":729722,"threads_started":2,"wal-append.queue_time_us":471597}
I20260709 12:25:07.082458 27627 raft_consensus.cc:2804] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:25:07.083072 27627 raft_consensus.cc:697] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 29 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:25:07.083559 27438 raft_consensus.cc:493] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:07.084204 27438 raft_consensus.cc:515] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:07.084776 27640 raft_consensus.cc:493] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:25:07.084501 27627 consensus_queue.cc:237] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 28.19, Last appended by leader: 19, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:25:07.032318 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:07.040359 27431 consensus_queue.cc:1059] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 29.17, Next index: 18, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:25:07.041229 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:06.948424 27525 raft_consensus.cc:493] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:07.089183 27525 raft_consensus.cc:515] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:07.090162 17684 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37402 (request call id 3925) took 1372 ms (client timeout 1606 ms). Trace:
W20260709 12:25:07.090584 17684 rpcz_store.cc:269] 0709 12:25:05.717853 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:05.736282 (+ 18429us) service_pool.cc:224] Handling call
0709 12:25:07.089977 (+1353695us) spinlock_profiling.cc:243] Waited 1.23 s on lock 0x7b54001aa754. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:25:07.090109 (+   132us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2831832320}
I20260709 12:25:06.988528 27451 raft_consensus.cc:2749] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:25:06.957511 27660 raft_consensus.cc:515] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:07.043522 17682 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37396 (request call id 3441) took 1301 ms. Trace:
I20260709 12:25:07.046021 27395 raft_consensus.cc:3060] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:06.986092 27661 raft_consensus.cc:2749] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:25:07.050570 18366 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59886 (request call id 2644) took 1062 ms. Trace:
I20260709 12:25:07.051424 27245 consensus_queue.cc:1243] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer e063541542a94969898868bf6eb5ce1b log is divergent from this leader: its last log entry 28.18 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 17
I20260709 12:25:07.052042 27363 raft_consensus.cc:493] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:25:07.052977 27580 raft_consensus.cc:493] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:07.053107 27582 raft_consensus.cc:493] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:25:07.054574 27585 raft_consensus.cc:493] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:07.113019 27585 raft_consensus.cc:515] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:07.114192 27660 leader_election.cc:290] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:07.115311 27585 leader_election.cc:290] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:07.113025 27582 raft_consensus.cc:515] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:07.118748 27582 leader_election.cc:290] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:07.119148 27443 leader_election.cc:290] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 30 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:07.061893 27604 consensus_queue.cc:260] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 30.22, Last appended by leader: 22, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:07.006903 27197 consensus_queue.cc:237] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 26.20, Last appended by leader: 20, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:07.119971 27363 raft_consensus.cc:515] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:07.122298 27363 leader_election.cc:290] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:07.122648 27604 raft_consensus.cc:3060] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:07.123250 17756 raft_consensus.cc:1275] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 32 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 29 index: 20. Preceding OpId from leader: term: 32 index: 22. (index mismatch)
I20260709 12:25:07.000682 18361 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 2828) took 1041 ms. Trace:
I20260709 12:25:07.067423 27664 leader_election.cc:290] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:07.001557 17801 leader_election.cc:304] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:07.124126 18361 rpcz_store.cc:276] 0709 12:25:05.959851 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:05.960074 (+   223us) service_pool.cc:224] Handling call
0709 12:25:05.960450 (+   376us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:06.848137 (+887687us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:07.000346 (+152209us) spinlock_profiling.cc:243] Waited 783 ms on lock 0x7b54006aed54. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:25:07.000603 (+   257us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1801552640}
I20260709 12:25:07.060389 17654 leader_election.cc:304] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:25:07.055691 27441 raft_consensus.cc:515] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:25:07.126511 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 77636.6us
I20260709 12:25:07.127848 27441 leader_election.cc:290] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 28 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:07.128242 27443 raft_consensus.cc:2804] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:25:07.128623 17760 raft_consensus.cc:1240] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 26. Current term is 27. Ops: [26.19-26.19]
I20260709 12:25:07.128878 27443 raft_consensus.cc:697] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 22 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:25:07.077450 27448 raft_consensus.cc:515] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:07.130110 27443 consensus_queue.cc:237] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 21.15, Last appended by leader: 15, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:07.131096 27448 leader_election.cc:290] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 25 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:07.076453 27475 raft_consensus.cc:493] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:25:07.081777 27462 raft_consensus.cc:493] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:07.098531 17682 rpcz_store.cc:276] 0709 12:25:05.742165 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:05.837428 (+ 95263us) service_pool.cc:224] Handling call
0709 12:25:07.043140 (+1205712us) spinlock_profiling.cc:243] Waited 1.06 s on lock 0x7b54001388d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:25:07.043304 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2432501120}
I20260709 12:25:07.108929 18374 raft_consensus.cc:2468] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 26.
I20260709 12:25:07.131940 27475 raft_consensus.cc:3060] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:07.132045 27462 raft_consensus.cc:515] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:07.133627 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 25 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:07.134500 18364 raft_consensus.cc:2410] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for term 25 because replica has last-logged OpId of term: 23 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 15.
I20260709 12:25:07.135001 27462 leader_election.cc:290] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:07.137358 18365 raft_consensus.cc:1275] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 29 index: 19. Preceding OpId from leader: term: 30 index: 20. (index mismatch)
I20260709 12:25:07.104317 18366 rpcz_store.cc:276] 0709 12:25:05.989258 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:06.770050 (+780792us) service_pool.cc:224] Handling call
0709 12:25:06.770479 (+   429us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:06.835917 (+ 65438us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:07.050297 (+214380us) spinlock_profiling.cc:243] Waited 53 ms on lock 0x7b5400031bd4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:25:07.050482 (+   185us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":121822464}
I20260709 12:25:07.138873 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 29 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:07.139583 18361 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 28.
I20260709 12:25:07.110088 27544 raft_consensus.cc:2804] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Leader pre-election won for term 27
W20260709 12:25:07.140228 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 79186.5us
I20260709 12:25:07.140625 27544 raft_consensus.cc:493] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:25:07.141043 27544 raft_consensus.cc:3060] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:07.141132 18333 leader_election.cc:304] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:07.141948 27664 raft_consensus.cc:2749] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:25:07.143625 27475 raft_consensus.cc:515] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:07.144348 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 176574us
I20260709 12:25:07.103190 27640 raft_consensus.cc:515] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:07.147099 27559 leader_election.cc:290] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:07.148536 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:07.149329 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 30 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:07.147017 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 29 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:07.155462 18367 raft_consensus.cc:2410] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for term 30 because replica has last-logged OpId of term: 26 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 18.
I20260709 12:25:07.156708 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:07.171090 17764 raft_consensus.cc:2376] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 24. Re-sending same reply.
I20260709 12:25:07.152493 17652 leader_election.cc:304] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:25:07.153949 27245 consensus_queue.cc:1059] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 27.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.019s
I20260709 12:25:07.173386 27245 raft_consensus.cc:2804] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:25:07.173491 27664 consensus_queue.cc:1059] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20260709 12:25:07.174018 27245 raft_consensus.cc:697] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 31 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:25:07.174460 27664 raft_consensus.cc:3055] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:25:07.174853 27664 raft_consensus.cc:740] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:25:07.157891 27580 raft_consensus.cc:515] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:07.154791 18366 raft_consensus.cc:2468] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 25.
I20260709 12:25:07.178544 27580 leader_election.cc:290] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:07.156239 18371 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 28.
W20260709 12:25:07.164384 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 208624us
I20260709 12:25:07.171119 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 32 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:07.175140 27245 consensus_queue.cc:237] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 30.17, Last appended by leader: 17, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:07.174029 17652 leader_election.cc:304] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:07.262084 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 27 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:07.266404 17760 raft_consensus.cc:2468] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 31.
I20260709 12:25:07.266659 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:07.228515 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 25 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:07.151224 27196 raft_consensus.cc:3055] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:25:07.164955 27394 raft_consensus.cc:2749] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:25:07.157368 27369 raft_consensus.cc:3055] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:25:07.134158 27442 raft_consensus.cc:515] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:07.261490 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 27 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:07.151726 27525 leader_election.cc:290] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:07.285270 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 33 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:07.164343 27431 raft_consensus.cc:3055] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:25:07.154223 27640 leader_election.cc:290] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:07.292191 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 27 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:07.293838 18361 raft_consensus.cc:3060] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:07.157193 27475 leader_election.cc:290] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 25 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:07.288463 18367 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 26.
W20260709 12:25:07.278124 27428 consensus_meta.cc:287] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.571s	user 0.006s	sys 0.011s
I20260709 12:25:07.307896 17756 raft_consensus.cc:2468] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 32.
I20260709 12:25:07.227830 27509 raft_consensus.cc:493] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
W20260709 12:25:07.307413 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 686902us
I20260709 12:25:07.279083 27196 raft_consensus.cc:740] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:25:07.312253 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 28 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:07.286471 27369 raft_consensus.cc:740] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 21 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:25:07.308864 18364 raft_consensus.cc:2468] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 24.
I20260709 12:25:07.311578 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 29 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:07.315232 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:07.318028 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:07.320358 18374 raft_consensus.cc:3060] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:07.270991 27270 consensus_queue.cc:1059] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 24.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.039s
I20260709 12:25:07.318113 17801 leader_election.cc:304] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:07.162868 27617 raft_consensus.cc:515] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:07.300962 27431 raft_consensus.cc:740] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 29 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:25:07.328413 17764 raft_consensus.cc:1240] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 26. Current term is 27. Ops: []
I20260709 12:25:07.329066 17682 raft_consensus.cc:2393] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 27: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:25:07.312848 27661 raft_consensus.cc:493] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:07.181136 27664 consensus_queue.cc:260] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 26.19, Last appended by leader: 19, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:07.332377 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 26 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:07.338639 17801 leader_election.cc:304] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:07.315430 27438 leader_election.cc:290] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:07.338165 17838 raft_consensus.cc:2468] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 30.
I20260709 12:25:07.344312 27505 raft_consensus.cc:2749] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:25:07.344482 27671 raft_consensus.cc:493] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:07.364635 17803 leader_election.cc:304] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:25:07.365231 17804 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:07.311002 27428 raft_consensus.cc:515] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:07.356897 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:07.391441 17760 raft_consensus.cc:2393] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 29: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:25:07.395159 18367 raft_consensus.cc:1275] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 23 index: 16. Preceding OpId from leader: term: 24 index: 17. (index mismatch)
I20260709 12:25:07.400741 27671 raft_consensus.cc:515] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:07.401230 27196 consensus_queue.cc:260] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 27.17, Last appended by leader: 17, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:07.402570 27196 raft_consensus.cc:3060] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:07.403010 27671 leader_election.cc:290] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:07.405498 27657 raft_consensus.cc:3055] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:25:07.405929 27657 raft_consensus.cc:740] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 26 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:25:07.406783 27657 consensus_queue.cc:260] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 26.20, Last appended by leader: 20, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:07.361563 27666 raft_consensus.cc:2804] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:25:07.409502 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:07.396049 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:07.417404 18361 raft_consensus.cc:2468] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 25.
W20260709 12:25:07.417368 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 78802.8us
I20260709 12:25:07.410163 17829 raft_consensus.cc:2468] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 28.
W20260709 12:25:07.429698 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 255464us
I20260709 12:25:07.340891 27509 raft_consensus.cc:515] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:07.420640 27639 raft_consensus.cc:493] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:07.434859 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 145518us
I20260709 12:25:07.435528 27639 raft_consensus.cc:515] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:07.358045 27661 raft_consensus.cc:515] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:07.440025 17837 raft_consensus.cc:1275] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 18 index: 14. Preceding OpId from leader: term: 20 index: 15. (index mismatch)
I20260709 12:25:07.441047 27661 leader_election.cc:290] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:25:07.441265 27657 raft_consensus.cc:3060] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:25:07.443024 27668 log.cc:927] Time spent T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.133s	user 0.000s	sys 0.001s
I20260709 12:25:07.450971 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 30 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:07.443362 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:07.456321 18364 raft_consensus.cc:1240] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 27. Current term is 28. Ops: []
I20260709 12:25:07.458478 27632 raft_consensus.cc:2749] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:25:07.446488 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:07.474324 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:07.460713 27428 leader_election.cc:290] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 33 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:07.443771 27394 raft_consensus.cc:2804] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Leader pre-election won for term 30
W20260709 12:25:07.469972 27415 consensus_meta.cc:287] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.731s	user 0.014s	sys 0.000s
I20260709 12:25:07.470641 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 27 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:07.471042 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 25 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:07.471760 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 28 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:07.470147 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 29 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:25:07.447238 27449 consensus_meta.cc:287] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.698s	user 0.014s	sys 0.000s
I20260709 12:25:07.475816 17758 raft_consensus.cc:2468] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 23.
I20260709 12:25:07.478467 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:07.344482 27369 consensus_queue.cc:260] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 21.15, Last appended by leader: 15, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:07.481295 17652 leader_election.cc:400] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 28 election: Vote denied by peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651) with higher term. Message: Invalid argument: T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for earlier term 28. Current term is 29.
I20260709 12:25:07.482959 17836 raft_consensus.cc:2393] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 25: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:25:07.484838 27442 leader_election.cc:290] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 25 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:07.484899 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:07.482733 27475 raft_consensus.cc:493] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:07.485666 18371 raft_consensus.cc:3060] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:07.485711 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:07.486044 17838 raft_consensus.cc:2468] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 24.
I20260709 12:25:07.482402 17830 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 26.
I20260709 12:25:07.486965 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:07.487555 17834 raft_consensus.cc:2468] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 27.
I20260709 12:25:07.489930 17760 raft_consensus.cc:2468] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 29.
I20260709 12:25:07.490718 17833 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 28.
W20260709 12:25:07.497849 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 157652us
I20260709 12:25:07.482205 27394 raft_consensus.cc:493] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:07.506512 18374 raft_consensus.cc:2468] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 30.
W20260709 12:25:07.502384 27672 log.cc:927] Time spent T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.166s	user 0.000s	sys 0.001s
W20260709 12:25:07.511866 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 190495us
I20260709 12:25:07.515683 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:07.502636 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:07.518162 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:07.524276 27197 raft_consensus.cc:493] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:25:07.524010 27475 raft_consensus.cc:515] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:07.525223 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:07.525733 17689 raft_consensus.cc:1240] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 30. Current term is 31. Ops: []
I20260709 12:25:07.526510 27475 leader_election.cc:290] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:07.527529 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:07.528510 17756 raft_consensus.cc:1275] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 26 index: 19. Preceding OpId from leader: term: 28 index: 20. (index mismatch)
I20260709 12:25:07.509024 27394 raft_consensus.cc:3060] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Advancing to term 30
W20260709 12:25:07.550359 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:07.551790 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 29 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:07.560268 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 25 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:07.548499 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 28 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:07.561273 18364 raft_consensus.cc:2468] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 27.
I20260709 12:25:07.561410 17689 raft_consensus.cc:2410] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for term 25 because replica has last-logged OpId of term: 23 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 15.
I20260709 12:25:07.491034 27449 raft_consensus.cc:515] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:07.578068 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 29 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:07.579453 17764 raft_consensus.cc:1240] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 33 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 32. Current term is 33. Ops: []
I20260709 12:25:07.567830 17836 raft_consensus.cc:2468] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 29.
W20260709 12:25:07.585100 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 122306us
I20260709 12:25:07.586489 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 33 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:07.527278 27197 raft_consensus.cc:515] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:07.587289 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 26 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:07.589742 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 29 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:07.489588 27431 consensus_queue.cc:260] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 29.17, Last appended by leader: 17, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:07.589221 17759 raft_consensus.cc:1240] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 24. Current term is 25. Ops: [24.18-24.18]
I20260709 12:25:07.590372 17829 raft_consensus.cc:2468] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 29.
I20260709 12:25:07.594949 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 29 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:07.596575 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 24 candidate_status { last_received { term: 23 index: 21 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:07.597383 17829 raft_consensus.cc:2468] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 23.
I20260709 12:25:07.591571 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 28 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:07.598734 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 30 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:07.574065 27369 raft_consensus.cc:3060] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:07.598901 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 29 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:07.605870 17684 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 28.
W20260709 12:25:07.606767 27676 log.cc:927] Time spent T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.151s	user 0.001s	sys 0.000s
I20260709 12:25:07.612452 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 29 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:07.613231 18374 raft_consensus.cc:2468] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 28.
I20260709 12:25:07.492975 17682 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f"
dest_uuid: "0da169ff82674dc1947055ce43de68a3"
mode: GRACEFUL
new_leader_uuid: "e063541542a94969898868bf6eb5ce1b"
 from {username='slave'} at 127.0.0.1:54540
I20260709 12:25:07.613956 27449 leader_election.cc:290] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 27 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:07.614305 17682 raft_consensus.cc:606] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Received request to transfer leadership to TS e063541542a94969898868bf6eb5ce1b
I20260709 12:25:07.614699 17682 raft_consensus.cc:614] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:25:07.616863 27197 leader_election.cc:290] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:07.617654 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 30 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:25:07.624192 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 94981.8us
I20260709 12:25:07.625214 17803 leader_election.cc:304] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:07.626801 27666 raft_consensus.cc:493] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:07.627343 27666 raft_consensus.cc:3060] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:07.632875 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:07.634277 27639 leader_election.cc:290] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:07.602808 17729 leader_election.cc:304] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:25:07.641733 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:07.551951 27062 consensus_queue.cc:1048] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 20, Time since last communication: 0.085s
W20260709 12:25:07.643594 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 50700.3us
I20260709 12:25:07.644399 17686 raft_consensus.cc:2393] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 28: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:25:07.648415 17652 leader_election.cc:400] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 26 election: Vote denied by peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651) with higher term. Message: Invalid argument: T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for earlier term 26. Current term is 27.
W20260709 12:25:07.605904 27673 log.cc:927] Time spent T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.103s	user 0.000s	sys 0.000s
I20260709 12:25:07.646003 27446 raft_consensus.cc:2804] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:25:07.591545 27642 raft_consensus.cc:3055] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:25:07.647012 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
W20260709 12:25:07.646793 18334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:07.648729 17652 leader_election.cc:403] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 26 election: Cancelling election due to peer responding with higher term
I20260709 12:25:07.650583 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 27 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:25:07.639103 17728 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.16.29.65:33071 timed out after 1.514s (SENT) [suppressed 23 similar messages]
I20260709 12:25:07.652333 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 29 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:07.654521 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 33 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:07.638670 27617 leader_election.cc:290] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 31 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:07.642935 27431 raft_consensus.cc:3060] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:07.657563 17685 raft_consensus.cc:2393] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 27: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:25:07.658447 27564 consensus_queue.cc:1059] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 23.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.007s
I20260709 12:25:07.659435 17652 leader_election.cc:304] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:25:07.659843 18333 leader_election.cc:304] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:07.661537 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:07.673273 17838 raft_consensus.cc:2468] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 31.
W20260709 12:25:07.663362 18334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:07.675674 18365 raft_consensus.cc:2410] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for term 29 because replica has last-logged OpId of term: 26 index: 20, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 19.
W20260709 12:25:07.651072 27544 consensus_meta.cc:287] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.509s	user 0.008s	sys 0.004s
I20260709 12:25:07.658464 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 29 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:07.656183 17729 leader_election.cc:304] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:25:07.662794 17834 raft_consensus.cc:1275] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 27 index: 16. Preceding OpId from leader: term: 29 index: 17. (index mismatch)
I20260709 12:25:07.701301 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 28 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:07.654215 27446 raft_consensus.cc:493] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:25:07.659327 27590 raft_consensus.cc:2749] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Leader election lost for term 26. Reason: Vote denied by peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651) with higher term. Message: Invalid argument: T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for earlier term 26. Current term is 27.
I20260709 12:25:07.654027 27642 raft_consensus.cc:740] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 24 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:25:07.702605 18364 raft_consensus.cc:3060] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:07.641697 27632 consensus_queue.cc:1048] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 18, Time since last communication: 0.028s
I20260709 12:25:07.710140 17684 raft_consensus.cc:2468] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 27.
I20260709 12:25:07.712399 17803 leader_election.cc:304] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:25:07.709170 17727 leader_election.cc:304] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:07.713554 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:07.716643 17680 raft_consensus.cc:1240] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 27 LEADER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 26. Current term is 27. Ops: []
I20260709 12:25:07.652321 27629 consensus_queue.cc:1059] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 32.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.059s
W20260709 12:25:07.721789 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 56264.6us
I20260709 12:25:07.721316 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:07.722896 17685 raft_consensus.cc:1240] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 29. Current term is 30. Ops: []
W20260709 12:25:07.721228 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 61950.7us
I20260709 12:25:07.710305 27270 raft_consensus.cc:3055] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:25:07.707834 27509 leader_election.cc:290] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:25:07.710150 27446 raft_consensus.cc:3060] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:07.728026 17682 raft_consensus.cc:2468] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 28.
W20260709 12:25:07.729540 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 79366.1us
I20260709 12:25:07.729465 17652 leader_election.cc:304] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
W20260709 12:25:07.730120 17804 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:07.733163 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 80745.5us
I20260709 12:25:07.717455 27639 raft_consensus.cc:2749] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:25:07.727411 17684 raft_consensus.cc:2468] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 19.
I20260709 12:25:07.730242 17802 leader_election.cc:304] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:07.708730 27544 raft_consensus.cc:515] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:07.739923 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:07.744117 17727 leader_election.cc:304] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:07.744308 27008 raft_consensus.cc:2804] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:25:07.744817 27008 raft_consensus.cc:493] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:25:07.745280 27008 raft_consensus.cc:3060] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:07.745523 17727 leader_election.cc:304] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:07.746526 27580 raft_consensus.cc:2764] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Leader pre-election decision vote started in defunct term 27: won
W20260709 12:25:07.746904 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 44370.1us
I20260709 12:25:07.747786 27580 raft_consensus.cc:2804] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:25:07.748222 27580 raft_consensus.cc:493] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:07.748560 27580 raft_consensus.cc:3060] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:07.749630 27642 consensus_queue.cc:260] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 24.16, Last appended by leader: 16, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:07.750975 27642 raft_consensus.cc:3060] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:07.726858 27640 raft_consensus.cc:2804] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:25:07.747794 17803 leader_election.cc:304] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:25:07.731112 27270 raft_consensus.cc:740] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 24 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:25:07.730695 27671 raft_consensus.cc:2749] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 12:25:07.757154 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:25:07.739020 27617 raft_consensus.cc:2804] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 33 FOLLOWER]: Leader election won for term 33
I20260709 12:25:07.741906 27590 raft_consensus.cc:2804] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:25:07.768536 17684 raft_consensus.cc:2393] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 25: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:25:07.755393 27640 raft_consensus.cc:493] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:07.778219 18366 raft_consensus.cc:1240] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 26. Current term is 27. Ops: []
W20260709 12:25:07.780203 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 50852.5us
I20260709 12:25:07.772694 27629 raft_consensus.cc:2749] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:25:07.786864 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:07.788170 18333 leader_election.cc:304] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:07.799369 17729 leader_election.cc:304] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:25:07.757088 27577 consensus_queue.cc:1048] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.027s
I20260709 12:25:07.803069 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 29 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:07.804450 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 29 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:07.805581 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:07.806241 17684 raft_consensus.cc:2468] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 23.
I20260709 12:25:07.807024 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:07.806890 27580 raft_consensus.cc:515] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:07.807636 17684 raft_consensus.cc:2468] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 30.
I20260709 12:25:07.808553 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:07.809665 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 33 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:25:07.809805 27580 leader_election.cc:290] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 25 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:07.810853 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 30 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:07.812438 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:07.814147 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 32 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:07.817332 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 24 candidate_status { last_received { term: 23 index: 21 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:25:07.818035 27395 consensus_meta.cc:287] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.713s	user 0.009s	sys 0.000s
I20260709 12:25:07.819336 17684 raft_consensus.cc:3055] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:25:07.819610 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:07.820745 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:25:07.823449 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:25:07.824239 27270 consensus_queue.cc:260] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 24.18, Last appended by leader: 18, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:07.783807 27590 raft_consensus.cc:697] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 28 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:25:07.827946 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:25:07.828406 27370 raft_consensus.cc:2749] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
W20260709 12:25:07.828816 27678 log.cc:927] Time spent T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.056s	user 0.000s	sys 0.001s
I20260709 12:25:07.782662 27617 raft_consensus.cc:697] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 33 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
W20260709 12:25:07.829893 27645 log.cc:927] Time spent T 00000000000000000000000000000000 P 2193af31cb894ec9b3d65d939a2a6851: Append to log took a long time: real 0.103s	user 0.000s	sys 0.000s
I20260709 12:25:07.831408 18364 raft_consensus.cc:2468] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 25.
I20260709 12:25:07.832401 27008 raft_consensus.cc:515] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:07.832984 27544 leader_election.cc:290] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 27 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:07.842854 27664 raft_consensus.cc:3060] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:07.843288 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:07.843415 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:07.780046 27640 raft_consensus.cc:3060] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:07.851104 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:07.855530 18371 raft_consensus.cc:2468] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 25.
W20260709 12:25:07.868510 17801 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:07.877663 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 31 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:07.971647 18368 raft_consensus.cc:3060] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:07.892879 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 30 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:07.893260 27564 raft_consensus.cc:3055] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 32 LEADER]: Stepping down as leader of term 32
I20260709 12:25:07.896437 17680 raft_consensus.cc:1240] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 29. Current term is 30. Ops: []
W20260709 12:25:07.897375 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 112287us
I20260709 12:25:07.904373 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:25:07.905210 17654 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:07.906060 27580 raft_consensus.cc:2804] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:25:07.908131 27680 raft_consensus.cc:493] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
W20260709 12:25:07.914915 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 169462us
I20260709 12:25:07.916623 17684 raft_consensus.cc:740] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:25:07.904305 27639 raft_consensus.cc:2749] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:25:07.920897 17687 raft_consensus.cc:2376] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 27. Re-sending same reply.
I20260709 12:25:07.975756 17685 raft_consensus.cc:2468] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 24.
W20260709 12:25:07.983502 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 76701.2us
I20260709 12:25:07.980106 17681 raft_consensus.cc:1275] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 27 index: 17. Preceding OpId from leader: term: 28 index: 18. (index mismatch)
W20260709 12:25:07.981418 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 201631us
I20260709 12:25:07.986660 17727 leader_election.cc:304] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:07.970913 27449 raft_consensus.cc:2804] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:25:08.001937 17682 raft_consensus.cc:2393] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 30: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:25:07.975104 27564 raft_consensus.cc:740] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 32 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:25:08.002800 17802 leader_election.cc:304] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:25:08.010099 17729 leader_election.cc:304] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:25:07.964984 27666 raft_consensus.cc:515] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:07.991597 27580 raft_consensus.cc:697] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 29 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:25:08.021185 17727 leader_election.cc:304] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:08.008574 17684 consensus_queue.cc:260] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 28.17, Last appended by leader: 17, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:08.026240 17684 raft_consensus.cc:3060] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:08.037335 18374 raft_consensus.cc:1275] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 26 index: 20. Preceding OpId from leader: term: 28 index: 21. (index mismatch)
I20260709 12:25:08.013700 27428 raft_consensus.cc:2804] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:25:08.041923 17835 raft_consensus.cc:1240] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 30. Current term is 31. Ops: []
I20260709 12:25:08.040251 17686 raft_consensus.cc:1275] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 24 index: 20. Preceding OpId from leader: term: 26 index: 21. (index mismatch)
I20260709 12:25:08.016558 27629 raft_consensus.cc:2804] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Leader pre-election won for term 20
W20260709 12:25:08.052697 18334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:08.061852 27570 consensus_queue.cc:1048] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 17, Time since last communication: 0.001s
W20260709 12:25:08.066233 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 318554us
I20260709 12:25:08.067225 17803 leader_election.cc:304] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:25:08.068238 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:08.059551 27590 consensus_queue.cc:237] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 27.15, Last appended by leader: 15, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:08.072495 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 27 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:08.073541 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 27 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:08.074134 17763 raft_consensus.cc:3060] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:08.076337 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:08.077981 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 31 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:08.081082 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:08.083155 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:08.088228 27564 consensus_queue.cc:260] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 32.19, Last appended by leader: 19, Current term: 32, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:08.016091 27671 consensus_queue.cc:1048] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 18, Time since last communication: 0.040s
I20260709 12:25:08.090819 27671 raft_consensus.cc:2804] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:25:08.091224 27671 raft_consensus.cc:493] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:25:08.091589 27671 raft_consensus.cc:3060] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:08.093895 17803 leader_election.cc:304] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:25:08.094836 27475 raft_consensus.cc:2804] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:25:08.095300 27475 raft_consensus.cc:493] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:08.087724 27632 raft_consensus.cc:3060] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:08.100750 27449 raft_consensus.cc:493] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:25:08.101526 27449 raft_consensus.cc:3060] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:08.104144 27692 consensus_queue.cc:1059] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.001s
I20260709 12:25:08.106359 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:08.111291 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 29 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:08.112344 27577 raft_consensus.cc:3055] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:25:08.112824 27577 raft_consensus.cc:740] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 30 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:25:08.083384 17763 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 27.
I20260709 12:25:08.017542 27062 consensus_queue.cc:1048] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.033s
I20260709 12:25:08.107769 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:08.004601 27680 raft_consensus.cc:515] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:08.116336 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 54999.3us
I20260709 12:25:08.046453 27428 raft_consensus.cc:493] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:08.049588 27629 raft_consensus.cc:493] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
W20260709 12:25:08.120097 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 127926us
I20260709 12:25:08.078032 17758 raft_consensus.cc:1275] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 25 index: 19. Preceding OpId from leader: term: 28 index: 21. (index mismatch)
I20260709 12:25:08.113855 27577 consensus_queue.cc:260] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 30.20, Last appended by leader: 20, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:08.116951 17836 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 28.
I20260709 12:25:08.095655 27475 raft_consensus.cc:3060] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:25:08.129855 17684 raft_consensus.cc:1275] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 28 index: 17. Preceding OpId from leader: term: 29 index: 17. (term mismatch)
I20260709 12:25:08.132638 17684 pending_rounds.cc:85] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3: Aborting all ops after (but not including) 16
I20260709 12:25:08.133702 17684 pending_rounds.cc:107] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3: Aborting uncommitted NO_OP operation due to leader change: 28.17
I20260709 12:25:08.134195 17684 raft_consensus.cc:2889] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:08.051290 27394 raft_consensus.cc:515] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:08.141093 17832 raft_consensus.cc:2468] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 27.
I20260709 12:25:08.023061 27580 consensus_queue.cc:237] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 28.17, Last appended by leader: 17, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:08.127317 27428 raft_consensus.cc:3060] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:08.153117 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:08.153865 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:25:08.158231 18334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:08.142285 17831 raft_consensus.cc:1275] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 23 index: 16. Preceding OpId from leader: term: 24 index: 17. (index mismatch)
I20260709 12:25:08.144771 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:08.145598 17763 raft_consensus.cc:1240] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 27. Current term is 28. Ops: []
W20260709 12:25:08.207307 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:08.132651 27629 raft_consensus.cc:3060] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:08.063091 27525 consensus_queue.cc:1048] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.079s
I20260709 12:25:08.220647 17762 raft_consensus.cc:2393] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 27: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
W20260709 12:25:08.228461 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 90968.1us
W20260709 12:25:08.226145 17655 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:08.231490 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 83861us
I20260709 12:25:08.230151 17654 leader_election.cc:304] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:25:08.248660 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:08.050926 27617 consensus_queue.cc:237] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 31.24, Last appended by leader: 24, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:08.249747 17763 raft_consensus.cc:2468] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 31.
I20260709 12:25:08.266172 17757 raft_consensus.cc:3060] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:08.222411 27369 raft_consensus.cc:2804] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:25:08.143293 27395 consensus_queue.cc:1048] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.113s
I20260709 12:25:08.271180 18333 leader_election.cc:304] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:25:08.305452 17762 raft_consensus.cc:1240] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 LEADER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 28. Current term is 29. Ops: []
I20260709 12:25:08.309965 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:08.311404 17763 raft_consensus.cc:2393] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 25: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:25:08.148150 27694 consensus_queue.cc:1048] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.150s
I20260709 12:25:08.253842 27245 raft_consensus.cc:493] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:25:08.272109 27638 raft_consensus.cc:493] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:08.317523 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 296411us
I20260709 12:25:08.316322 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:25:08.312455 27688 log.cc:927] Time spent T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.173s	user 0.000s	sys 0.002s
I20260709 12:25:08.325731 17837 raft_consensus.cc:2468] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 25.
W20260709 12:25:08.341866 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 50331.1us
I20260709 12:25:08.268950 27505 raft_consensus.cc:2804] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:25:08.261824 27008 leader_election.cc:290] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 31 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:25:08.345187 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 31 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:08.317485 27639 raft_consensus.cc:2804] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:25:08.344210 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:25:08.331076 27431 consensus_meta.cc:287] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.655s	user 0.010s	sys 0.002s
W20260709 12:25:08.341833 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 340211us
W20260709 12:25:08.365032 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 106666us
I20260709 12:25:08.366407 17652 leader_election.cc:304] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:08.369196 17833 raft_consensus.cc:2410] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for term 30 because replica has last-logged OpId of term: 29 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 15.
I20260709 12:25:08.314574 27197 consensus_queue.cc:1059] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 26.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.048s
I20260709 12:25:08.321187 27196 consensus_queue.cc:1048] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.046s
I20260709 12:25:08.369675 17728 leader_election.cc:304] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:25:08.377080 17652 leader_election.cc:304] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:08.348119 27666 leader_election.cc:290] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 31 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:25:08.380187 17802 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:08.442015 17802 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:08.442946 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 349175us
W20260709 12:25:08.443157 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 433102us
I20260709 12:25:08.445485 17729 leader_election.cc:304] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:25:08.447726 27446 consensus_meta.cc:287] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.691s	user 0.009s	sys 0.003s
I20260709 12:25:08.363826 27505 raft_consensus.cc:697] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 30 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:25:08.342154 27638 raft_consensus.cc:515] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:08.463236 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:25:08.338860 27245 raft_consensus.cc:515] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:08.479205 18334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:08.386162 27696 log.cc:927] Time spent T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.150s	user 0.000s	sys 0.001s
W20260709 12:25:08.387132 27698 log.cc:927] Time spent T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.061s	user 0.000s	sys 0.000s
W20260709 12:25:08.484221 18368 consensus_meta.cc:287] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.508s	user 0.014s	sys 0.000s
I20260709 12:25:08.511924 18368 raft_consensus.cc:2468] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 31.
I20260709 12:25:08.388691 27394 leader_election.cc:290] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 30 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:08.378429 27639 raft_consensus.cc:697] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 28 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:25:08.508361 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
W20260709 12:25:08.509775 17804 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:08.404660 27680 leader_election.cc:290] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:25:08.532868 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 225494us
I20260709 12:25:08.473598 27666 raft_consensus.cc:493] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:08.535804 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 531194us
W20260709 12:25:08.546686 17801 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:08.571408 17803 leader_election.cc:304] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:25:08.453822 27446 raft_consensus.cc:515] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:08.528007 27338 raft_consensus.cc:493] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:08.481541 27443 consensus_queue.cc:1048] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 19, Time since last communication: 0.091s
I20260709 12:25:08.578267 18333 leader_election.cc:304] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
W20260709 12:25:08.570617 27685 log.cc:927] Time spent T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.814s	user 0.005s	sys 0.000s
I20260709 12:25:08.574529 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:08.594014 17831 raft_consensus.cc:3060] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:08.573860 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:08.595233 17836 raft_consensus.cc:3060] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:08.591293 18374 raft_consensus.cc:1275] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 25 index: 18. Preceding OpId from leader: term: 26 index: 19. (index mismatch)
I20260709 12:25:08.538153 27197 raft_consensus.cc:3055] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:25:08.596573 17837 raft_consensus.cc:1240] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 30. Current term is 31. Ops: [30.18-30.18]
I20260709 12:25:08.523229 27664 consensus_queue.cc:1059] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 24.18, Next index: 19, Last known committed idx: 17, Time since last communication: 0.056s
I20260709 12:25:08.602187 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:08.593852 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 29 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:08.602898 17837 raft_consensus.cc:2468] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 31.
I20260709 12:25:08.533411 27525 consensus_queue.cc:1048] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.057s
I20260709 12:25:08.532509 27694 consensus_queue.cc:1048] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.058s
I20260709 12:25:08.605283 18368 raft_consensus.cc:1275] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 21 index: 15. Preceding OpId from leader: term: 22 index: 16. (index mismatch)
W20260709 12:25:08.592764 27681 log.cc:927] Time spent T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.850s	user 0.010s	sys 0.002s
W20260709 12:25:08.594249 27695 log.cc:927] Time spent T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.059s	user 0.000s	sys 0.000s
I20260709 12:25:08.597643 27338 raft_consensus.cc:515] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:08.616149 27245 leader_election.cc:290] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:08.617681 27369 raft_consensus.cc:493] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:08.618852 18362 raft_consensus.cc:1275] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 26 index: 20. Preceding OpId from leader: term: 27 index: 21. (index mismatch)
W20260709 12:25:08.622555 17655 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:08.625536 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 95514.2us
I20260709 12:25:08.644222 17837 raft_consensus.cc:1240] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 28 LEADER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 27. Current term is 28. Ops: []
I20260709 12:25:08.622871 17761 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 28.
I20260709 12:25:08.572433 27666 raft_consensus.cc:515] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:08.633118 17830 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.517s	user 0.000s	sys 0.012s
W20260709 12:25:08.633459 17804 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:08.607215 27697 log.cc:927] Time spent T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.335s	user 0.000s	sys 0.002s
I20260709 12:25:08.601154 27197 raft_consensus.cc:740] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
W20260709 12:25:08.646831 17830 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.531s	user 0.000s	sys 0.017s
W20260709 12:25:08.698765 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 351729us
I20260709 12:25:08.628754 27394 raft_consensus.cc:493] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:08.620678 27705 raft_consensus.cc:493] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:08.674770 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 297772us
I20260709 12:25:08.705163 17652 leader_election.cc:304] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:08.706831 27525 raft_consensus.cc:2749] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:25:08.646374 27369 raft_consensus.cc:3060] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Advancing to term 25
W20260709 12:25:08.718787 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 273391us
I20260709 12:25:08.723416 17652 leader_election.cc:304] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:08.725028 17837 raft_consensus.cc:1240] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 26. Current term is 27. Ops: []
I20260709 12:25:08.739354 17729 leader_election.cc:304] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:08.741353 17729 leader_election.cc:304] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
W20260709 12:25:08.725075 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 134064us
I20260709 12:25:08.744366 18334 leader_election.cc:304] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:25:08.744789 27270 raft_consensus.cc:3060] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Advancing to term 25
W20260709 12:25:08.730101 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 369875us
I20260709 12:25:08.747391 17728 leader_election.cc:304] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
W20260709 12:25:08.749454 17730 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:08.752705 27638 leader_election.cc:290] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:08.753506 27638 raft_consensus.cc:2804] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:25:08.754094 27638 raft_consensus.cc:697] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 24 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
W20260709 12:25:08.756551 27690 log.cc:927] Time spent T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.785s	user 0.003s	sys 0.003s
I20260709 12:25:08.758297 18367 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59886 (request call id 2656) took 1313 ms. Trace:
I20260709 12:25:08.758773 18367 rpcz_store.cc:276] 0709 12:25:07.444847 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:07.462830 (+ 17983us) service_pool.cc:224] Handling call
0709 12:25:07.483278 (+ 20448us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:07.489293 (+  6015us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:07.489323 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:07.519267 (+ 29944us) log.cc:844] Serialized 26 byte log entry
0709 12:25:07.756969 (+237702us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:25:07.757010 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:07.757080 (+    70us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:08.630826 (+873746us) spinlock_profiling.cc:243] Waited 2.46 ms on lock 0x7b4400022b4c. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b47459c85 00007f7b4745a6a2 00007f7b4a1cc1db 00007f7b4a1c7aaf 00007f7b4a1bb86e 00007f7b4a1babb6 00007f7b4a1ba7d8 00007f7b4a1c4221 00007f7b4a1c41d9 00007f7b4a1c4169
0709 12:25:08.757938 (+127112us) raft_consensus.cc:1685] finished
0709 12:25:08.757969 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:08.758224 (+   255us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":80,"raft.run_cpu_time_us":223,"raft.run_wall_time_us":28811,"spinlock_wait_cycles":5665152,"thread_start_us":175260,"threads_started":1,"wal-append.queue_time_us":350351}
W20260709 12:25:08.761874 27706 log.cc:927] Time spent T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.125s	user 0.000s	sys 0.000s
I20260709 12:25:08.700207 27544 consensus_queue.cc:1059] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 29.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.028s
I20260709 12:25:08.768952 27544 raft_consensus.cc:2804] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:25:08.769436 27544 raft_consensus.cc:493] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:08.769853 27544 raft_consensus.cc:3060] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Advancing to term 25
W20260709 12:25:08.774067 27712 log.cc:927] Time spent T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.065s	user 0.000s	sys 0.000s
I20260709 12:25:08.775079 17837 raft_consensus.cc:1275] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 25 index: 18. Preceding OpId from leader: term: 26 index: 19. (index mismatch)
I20260709 12:25:08.785543 17837 raft_consensus.cc:1275] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 28 index: 19. Preceding OpId from leader: term: 29 index: 20. (index mismatch)
I20260709 12:25:08.753520 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:08.788532 17761 raft_consensus.cc:2393] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 31: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:25:08.755479 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:08.796883 27338 leader_election.cc:290] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:08.799607 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:08.802266 17755 raft_consensus.cc:1240] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 23. Current term is 24. Ops: []
I20260709 12:25:08.810410 17689 raft_consensus.cc:2468] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 31.
W20260709 12:25:08.820791 17757 consensus_meta.cc:287] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.547s	user 0.013s	sys 0.000s
I20260709 12:25:08.821431 17757 raft_consensus.cc:2468] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 28.
I20260709 12:25:08.823832 17837 raft_consensus.cc:1240] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 28. Current term is 29. Ops: []
W20260709 12:25:08.728660 27640 consensus_meta.cc:287] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.733s	user 0.012s	sys 0.001s
I20260709 12:25:08.799489 27338 raft_consensus.cc:3055] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:25:08.714044 27525 raft_consensus.cc:2804] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Leader pre-election won for term 29
W20260709 12:25:08.831580 17834 consensus_meta.cc:287] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.700s	user 0.006s	sys 0.007s
I20260709 12:25:08.839916 17834 raft_consensus.cc:2468] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 22.
I20260709 12:25:08.841375 17837 raft_consensus.cc:1275] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 30 index: 17. Preceding OpId from leader: term: 31 index: 18. (index mismatch)
W20260709 12:25:08.829291 27712 log.cc:927] Time spent T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.054s	user 0.003s	sys 0.000s
I20260709 12:25:08.836309 17836 raft_consensus.cc:2468] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 28.
I20260709 12:25:08.714535 27394 raft_consensus.cc:515] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:08.826838 27245 raft_consensus.cc:2804] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:25:08.712302 27705 raft_consensus.cc:515] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:08.700165 27639 consensus_queue.cc:237] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 27.17, Last appended by leader: 17, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:08.839562 27525 raft_consensus.cc:493] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:25:08.874891 18334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:08.642639 27505 consensus_queue.cc:237] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 29.17, Last appended by leader: 17, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:08.845203 27632 consensus_meta.cc:287] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.745s	user 0.009s	sys 0.016s
I20260709 12:25:08.894523 27713 raft_consensus.cc:493] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:08.897573 17689 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9"
dest_uuid: "0da169ff82674dc1947055ce43de68a3"
mode: GRACEFUL
new_leader_uuid: "e063541542a94969898868bf6eb5ce1b"
 from {username='slave'} at 127.0.0.1:54540
I20260709 12:25:08.898907 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:25:08.834692 27338 raft_consensus.cc:740] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 29 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:25:08.755263 27638 consensus_queue.cc:237] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 22.15, Last appended by leader: 15, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:08.876075 27617 raft_consensus.cc:2749] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:25:08.875113 27714 raft_consensus.cc:2749] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 12:25:08.917742 17683 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44898 (request call id 1957) took 1076 ms. Trace:
I20260709 12:25:08.905797 27525 raft_consensus.cc:3060] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:08.911109 17654 leader_election.cc:304] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
W20260709 12:25:08.944638 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 242715us
W20260709 12:25:08.950474 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:08.840632 27640 raft_consensus.cc:515] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:25:08.957683 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 210358us
I20260709 12:25:08.949834 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:25:08.958745 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 215936us
I20260709 12:25:08.962114 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:08.962949 17729 leader_election.cc:304] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:08.961045 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 27 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:08.905378 27723 raft_consensus.cc:2764] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Leader pre-election decision vote started in defunct term 26: won
I20260709 12:25:08.971995 17729 leader_election.cc:304] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:25:08.909886 27632 raft_consensus.cc:2749] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:25:08.976522 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 27 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:08.978668 17838 raft_consensus.cc:3060] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:08.946815 27395 raft_consensus.cc:3060] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:08.947531 27720 raft_consensus.cc:493] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:08.949956 17683 rpcz_store.cc:276] 0709 12:25:07.841756 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:07.927285 (+ 85529us) service_pool.cc:224] Handling call
0709 12:25:07.977259 (+ 49974us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:07.983026 (+  5767us) raft_consensus.cc:1534] Early marking committed up to index 19
0709 12:25:08.010356 (+ 27330us) log.cc:844] Serialized 14 byte log entry
0709 12:25:08.906159 (+895803us) log.cc:844] Serialized 14 byte log entry
0709 12:25:08.917384 (+ 11225us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:08.917437 (+    53us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:25:08.917487 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:08.917556 (+    69us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:08.917686 (+   130us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":22188,"prepare.queue_time_us":15151,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":44,"raft.queue_time_us":29270,"raft.run_cpu_time_us":550,"raft.run_wall_time_us":17924,"thread_start_us":861828,"threads_started":3,"wal-append.queue_time_us":443354}
I20260709 12:25:08.844179 27197 consensus_queue.cc:260] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 26.18, Last appended by leader: 18, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:09.002686 17764 raft_consensus.cc:1240] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 30. Current term is 31. Ops: []
W20260709 12:25:09.004629 18334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:08.954949 27446 leader_election.cc:290] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 24 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:08.910617 27713 raft_consensus.cc:515] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:09.041600 17759 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38786 (request call id 4172) took 1168 ms. Trace:
W20260709 12:25:09.040972 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 72016.8us
W20260709 12:25:09.047937 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 71355.3us
I20260709 12:25:09.093755 17729 leader_election.cc:304] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
W20260709 12:25:09.095232 17727 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:08.988504 27570 raft_consensus.cc:2804] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Leader election won for term 31
W20260709 12:25:09.100803 27701 log.cc:927] Time spent T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.426s	user 0.002s	sys 0.000s
I20260709 12:25:09.101521 27728 raft_consensus.cc:493] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
W20260709 12:25:09.101977 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 99146us
I20260709 12:25:09.103392 17729 leader_election.cc:304] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:25:09.003854 27708 log.cc:927] Time spent T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.379s	user 0.000s	sys 0.001s
W20260709 12:25:09.102159 27718 log.cc:927] Time spent T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.086s	user 0.000s	sys 0.000s
I20260709 12:25:09.089915 17759 rpcz_store.cc:276] 0709 12:25:07.842992 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:07.844017 (+  1025us) service_pool.cc:224] Handling call
0709 12:25:07.844347 (+   330us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:07.844757 (+   410us) raft_consensus.cc:1534] Early marking committed up to index 19
0709 12:25:08.027648 (+182891us) log.cc:844] Serialized 14 byte log entry
0709 12:25:08.700781 (+673133us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:08.714390 (+ 13609us) log.cc:844] Serialized 26 byte log entry
0709 12:25:08.714622 (+   232us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:25:08.714696 (+    74us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:08.714820 (+   124us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:09.010684 (+295864us) raft_consensus.cc:1685] finished
0709 12:25:09.010731 (+    47us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:09.011100 (+   369us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":15141,"prepare.queue_time_us":403689,"prepare.run_cpu_time_us":68,"prepare.run_wall_time_us":3384,"raft.queue_time_us":264869,"raft.run_cpu_time_us":169,"raft.run_wall_time_us":170,"thread_start_us":792127,"threads_started":3,"wal-append.queue_time_us":552162}
I20260709 12:25:09.123783 17689 raft_consensus.cc:606] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Received request to transfer leadership to TS e063541542a94969898868bf6eb5ce1b
I20260709 12:25:09.124266 17689 raft_consensus.cc:614] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260709 12:25:09.130221 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 406909us
I20260709 12:25:09.130874 27713 leader_election.cc:290] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:09.131136 17652 leader_election.cc:304] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:25:09.133010 17728 leader_election.cc:304] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
W20260709 12:25:09.135924 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 150596us
W20260709 12:25:09.138878 17654 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:09.093194 27446 raft_consensus.cc:2749] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:25:09.092073 27666 leader_election.cc:290] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:09.145545 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 25 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:09.111605 27739 raft_consensus.cc:2804] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Leader pre-election won for term 28
W20260709 12:25:09.149577 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 571361us
W20260709 12:25:09.153204 17833 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37994 (request call id 1848) took 1575 ms (client timeout 2027 ms). Trace:
I20260709 12:25:09.153694 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:09.154780 18371 raft_consensus.cc:2468] T a5732e71b9d54f8786b8c511d69943bf P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 27.
I20260709 12:25:09.157809 17683 raft_consensus.cc:1240] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 29. Current term is 30. Ops: []
I20260709 12:25:09.157826 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 27 candidate_status { last_received { term: 24 index: 20 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:09.011390 27714 consensus_queue.cc:1048] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.136s
I20260709 12:25:09.160252 18365 raft_consensus.cc:2468] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 26.
W20260709 12:25:09.153633 17833 rpcz_store.cc:269] 0709 12:25:07.572107 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:07.598410 (+ 26303us) service_pool.cc:224] Handling call
0709 12:25:09.144073 (+1545663us) spinlock_profiling.cc:243] Waited 765 ms on lock 0x7b540008cfd4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:25:09.147187 (+  3114us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1759595392}
I20260709 12:25:09.165536 18362 raft_consensus.cc:1240] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 30. Current term is 31. Ops: [30.20-30.20]
I20260709 12:25:09.131523 27713 consensus_queue.cc:1048] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.017s
I20260709 12:25:09.156144 27739 raft_consensus.cc:493] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:09.179517 17652 leader_election.cc:304] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:09.182821 27737 raft_consensus.cc:2804] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:25:09.183917 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:09.184698 18371 raft_consensus.cc:3060] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:09.188267 27617 raft_consensus.cc:2804] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:25:09.188731 27617 raft_consensus.cc:493] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:09.189091 27617 raft_consensus.cc:3060] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Advancing to term 29
W20260709 12:25:09.170017 27671 consensus_meta.cc:287] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 1.025s	user 0.016s	sys 0.001s
I20260709 12:25:09.202090 18333 leader_election.cc:304] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:25:09.202270 17760 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38786 (request call id 4171) took 1359 ms. Trace:
W20260709 12:25:09.205138 17831 consensus_meta.cc:287] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.600s	user 0.008s	sys 0.005s
I20260709 12:25:09.206517 17831 raft_consensus.cc:2468] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 30.
I20260709 12:25:09.204025 17760 rpcz_store.cc:276] 0709 12:25:07.842653 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:07.843755 (+  1102us) service_pool.cc:224] Handling call
0709 12:25:07.844162 (+   407us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:25:07.844567 (+   405us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:07.844596 (+    29us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:25:07.845318 (+   722us) log.cc:844] Serialized 52 byte log entry
0709 12:25:08.146613 (+301295us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:25:08.146663 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:08.146749 (+    86us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:09.201935 (+1055186us) raft_consensus.cc:1685] finished
0709 12:25:09.201969 (+    34us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:09.202183 (+   214us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":892,"raft.run_cpu_time_us":101,"raft.run_wall_time_us":100,"thread_start_us":300899,"threads_started":1,"wal-append.queue_time_us":525632}
W20260709 12:25:09.217200 17730 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:09.222337 18371 raft_consensus.cc:2468] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 24.
I20260709 12:25:09.112416 27338 consensus_queue.cc:260] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 29.19, Last appended by leader: 19, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:09.147372 27728 raft_consensus.cc:515] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:09.012725 27720 raft_consensus.cc:515] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:09.258289 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 79152.8us
I20260709 12:25:09.291200 18333 leader_election.cc:304] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:25:09.224293 27721 log.cc:927] Time spent T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.432s	user 0.001s	sys 0.002s
W20260709 12:25:09.262094 27428 consensus_meta.cc:287] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 1.017s	user 0.012s	sys 0.000s
I20260709 12:25:09.223636 27570 raft_consensus.cc:697] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 31 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:25:09.198441 27739 raft_consensus.cc:3060] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Advancing to term 28
W20260709 12:25:09.257362 27730 log.cc:927] Time spent T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.104s	user 0.001s	sys 0.000s
I20260709 12:25:09.224181 27725 consensus_queue.cc:1048] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.198s
W20260709 12:25:09.224493 27731 log.cc:927] Time spent T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.052s	user 0.000s	sys 0.000s
I20260709 12:25:09.314152 17755 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38776 (request call id 3346) took 1240 ms. Trace:
I20260709 12:25:09.216214 27446 raft_consensus.cc:2804] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:25:09.291850 27737 raft_consensus.cc:697] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:25:09.224277 27722 raft_consensus.cc:2804] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Leader pre-election won for term 29
W20260709 12:25:09.259068 27726 log.cc:927] Time spent T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.068s	user 0.000s	sys 0.000s
I20260709 12:25:09.217289 27671 raft_consensus.cc:515] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:09.258648 27732 log.cc:927] Time spent T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.075s	user 0.000s	sys 0.000s
I20260709 12:25:09.215562 27394 leader_election.cc:290] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:09.358095 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:09.257663 27727 raft_consensus.cc:493] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:09.320385 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 108746us
I20260709 12:25:09.224594 27724 raft_consensus.cc:2804] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:25:09.364514 17652 leader_election.cc:304] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da
W20260709 12:25:09.319098 27709 log.cc:927] Time spent T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.656s	user 0.002s	sys 0.001s
I20260709 12:25:09.263036 27705 leader_election.cc:290] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:09.324393 27704 raft_consensus.cc:493] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:09.336058 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 172551us
I20260709 12:25:09.329958 17755 rpcz_store.cc:276] 0709 12:25:08.073821 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:08.080027 (+  6206us) service_pool.cc:224] Handling call
0709 12:25:08.080483 (+   456us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:08.828892 (+748409us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:09.307707 (+478815us) spinlock_profiling.cc:243] Waited 525 ms on lock 0x7b54003e5dd4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:25:09.314078 (+  6371us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1208039168}
I20260709 12:25:09.320040 27338 raft_consensus.cc:3060] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Advancing to term 30
W20260709 12:25:09.348807 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 192303us
W20260709 12:25:09.333272 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 200330us
I20260709 12:25:09.328018 27640 leader_election.cc:290] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 31 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:09.343580 27722 raft_consensus.cc:493] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:25:09.381337 17835 raft_consensus.cc:2468] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 26.
W20260709 12:25:09.412046 17835 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35040 (request call id 3691) took 1343 ms (client timeout 1712 ms). Trace:
I20260709 12:25:09.381119 18333 leader_election.cc:304] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:25:09.321353 27428 raft_consensus.cc:515] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:09.412525 17654 leader_election.cc:304] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
W20260709 12:25:09.412698 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 48280.4us
I20260709 12:25:09.414083 27714 raft_consensus.cc:2749] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:25:09.414273 27428 leader_election.cc:290] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 30 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:25:09.261611 27745 log.cc:927] Time spent T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.060s	user 0.000s	sys 0.000s
I20260709 12:25:09.402509 17829 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35040 (request call id 3697) took 1331 ms. Trace:
I20260709 12:25:09.414790 27714 consensus_queue.cc:1048] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20260709 12:25:09.416205 27714 raft_consensus.cc:2804] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:25:09.415045 17829 rpcz_store.cc:276] 0709 12:25:08.071091 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:08.076855 (+  5764us) service_pool.cc:224] Handling call
0709 12:25:08.233387 (+156532us) spinlock_profiling.cc:243] Waited 12.2 ms on lock 0x7b580013fe98. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b474fbb71 00007f7b474fba22 00007f7b504148d9 00007f7b4f56eee0 00007f7b4f6380ab 00007f7b4f638131 00007f7b4f5f9fc0 00007f7b4f5e4929 00007f7b4f5e43d2 00007f7b48b5bbd4
0709 12:25:08.233632 (+   245us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:08.247870 (+ 14238us) raft_consensus.cc:1534] Early marking committed up to index 21
0709 12:25:08.321227 (+ 73357us) log.cc:844] Serialized 14 byte log entry
0709 12:25:08.644336 (+323109us) log.cc:844] Serialized 14 byte log entry
0709 12:25:09.401916 (+757580us) spinlock_profiling.cc:243] Waited 7.42 ms on lock 0x7b1800436ae0. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b474adc0a 00007f7b50bed219 00007f7b47535a40 00007f7b475304e4 00007f7b47530455 00007f7b4a618060 00007f7b4a62a716 00007f7b4a63b287 00007f7b4a63b208
0709 12:25:09.402066 (+   150us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:09.402123 (+    57us) raft_consensus.cc:1645] Marking committed up to 21
0709 12:25:09.402169 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:09.402235 (+    66us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:09.402397 (+   162us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3446,"prepare.queue_time_us":49141,"prepare.run_cpu_time_us":301,"prepare.run_wall_time_us":4685,"raft.queue_time_us":2053,"raft.run_cpu_time_us":315,"raft.run_wall_time_us":45476,"spinlock_wait_cycles":45203584,"thread_start_us":206199,"threads_started":1,"wal-append.queue_time_us":546729}
I20260709 12:25:09.416714 27714 raft_consensus.cc:493] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:25:09.347666 27692 raft_consensus.cc:493] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:09.417325 27714 raft_consensus.cc:3060] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:09.417516 27692 raft_consensus.cc:515] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:09.419492 27746 raft_consensus.cc:493] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 32 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:25:09.420886 27746 raft_consensus.cc:515] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:09.422895 27746 leader_election.cc:290] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:09.423372 27746 raft_consensus.cc:493] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:25:09.420372 27759 raft_consensus.cc:2804] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:25:09.424149 27759 raft_consensus.cc:493] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:09.424480 27759 raft_consensus.cc:3060] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:25:09.412398 17835 rpcz_store.cc:269] 0709 12:25:08.069098 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:08.082550 (+ 13452us) service_pool.cc:224] Handling call
0709 12:25:08.961852 (+879302us) spinlock_profiling.cc:243] Waited 7.29 ms on lock 0x7b580013fe98. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b474fbb71 00007f7b474fba22 00007f7b504148d9 00007f7b4f56eee0 00007f7b4f6380ab 00007f7b4f638131 00007f7b4f5e60a0 00007f7b4f5e5984 00007f7b48b606d4 00007f7b48b60662
0709 12:25:09.411985 (+450133us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":16773504}
I20260709 12:25:09.423825 27746 raft_consensus.cc:515] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:09.430552 27740 raft_consensus.cc:493] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:09.383739 27727 raft_consensus.cc:515] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:09.431634 27577 raft_consensus.cc:3060] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:09.434659 27724 raft_consensus.cc:697] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:25:09.411638 17728 leader_election.cc:304] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:09.371812 27446 raft_consensus.cc:697] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:25:09.442103 17830 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35040 (request call id 3696) took 1371 ms. Trace:
W20260709 12:25:09.445490 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 671988us
W20260709 12:25:09.446275 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 650043us
I20260709 12:25:09.448552 17802 leader_election.cc:304] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:25:09.398803 27734 raft_consensus.cc:493] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:09.437849 27727 leader_election.cc:290] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:25:09.453310 27711 raft_consensus.cc:2749] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:25:09.453931 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 34 candidate_status { last_received { term: 31 index: 24 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:09.436780 27638 raft_consensus.cc:2804] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:25:09.457413 18364 raft_consensus.cc:1275] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 33 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 31 index: 24. Preceding OpId from leader: term: 33 index: 25. (index mismatch)
I20260709 12:25:09.436376 27722 raft_consensus.cc:3060] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Advancing to term 29
W20260709 12:25:09.446761 27732 log.cc:927] Time spent T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.064s	user 0.000s	sys 0.000s
I20260709 12:25:09.397080 27744 raft_consensus.cc:2749] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
W20260709 12:25:09.461012 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 79960us
W20260709 12:25:09.464424 17804 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:09.464453 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:25:09.456527 27629 consensus_meta.cc:287] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 1.208s	user 0.007s	sys 0.007s
I20260709 12:25:09.471323 18333 leader_election.cc:304] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
W20260709 12:25:09.473060 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 61547.5us
I20260709 12:25:09.476797 17728 leader_election.cc:304] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:25:09.461338 27638 raft_consensus.cc:493] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:25:09.472851 27449 consensus_meta.cc:287] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 1.371s	user 0.015s	sys 0.002s
I20260709 12:25:09.486697 27758 raft_consensus.cc:2764] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Leader pre-election decision vote started in defunct term 23: won
I20260709 12:25:09.435700 27704 raft_consensus.cc:515] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:09.489087 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 30 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:09.478240 27723 raft_consensus.cc:2749] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:25:09.494851 17728 leader_election.cc:304] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:25:09.496095 17838 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 27.
W20260709 12:25:09.497298 18334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:09.474519 27509 raft_consensus.cc:2804] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:25:09.431754 27740 raft_consensus.cc:515] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:09.492666 17801 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:09.508767 17838 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35040 (request call id 3695) took 1438 ms. Trace:
I20260709 12:25:09.499711 18364 raft_consensus.cc:2468] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 28.
I20260709 12:25:09.500434 17756 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38776 (request call id 3342) took 1427 ms. Trace:
I20260709 12:25:09.462397 27734 raft_consensus.cc:515] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:09.501312 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:25:09.488719 27525 consensus_meta.cc:287] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.502s	user 0.004s	sys 0.009s
I20260709 12:25:09.509671 17756 rpcz_store.cc:276] 0709 12:25:08.072535 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:08.078906 (+  6371us) service_pool.cc:224] Handling call
0709 12:25:08.079245 (+   339us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:25:08.079640 (+   395us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:08.079664 (+    24us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:25:08.080398 (+   734us) log.cc:844] Serialized 52 byte log entry
0709 12:25:08.320847 (+240449us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:25:08.320888 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:08.320952 (+    64us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:09.189770 (+868818us) spinlock_profiling.cc:243] Waited 23.8 ms on lock 0x7b4400022b4c. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b47459c85 00007f7b4745a6a2 00007f7b4a1cc1db 00007f7b4a1c7aaf 00007f7b4a1bb86e 00007f7b4a1babb6 00007f7b4a1ba7d8 00007f7b4a1c4221 00007f7b4a1c41d9 00007f7b4a1c4169
0709 12:25:09.491947 (+302177us) raft_consensus.cc:1685] finished
0709 12:25:09.491970 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:09.500368 (+  8398us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":25009,"raft.run_cpu_time_us":105,"raft.run_wall_time_us":104,"spinlock_wait_cycles":54819584,"thread_start_us":240058,"threads_started":1,"wal-append.queue_time_us":367736}
I20260709 12:25:09.509140 17838 rpcz_store.cc:276] 0709 12:25:08.070411 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:08.076248 (+  5837us) service_pool.cc:224] Handling call
0709 12:25:08.976360 (+900112us) spinlock_profiling.cc:243] Waited 12.3 ms on lock 0x7b580013fe98. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b474fbb71 00007f7b474fba22 00007f7b504148d9 00007f7b4f56eee0 00007f7b4f6380ab 00007f7b4f638131 00007f7b4f5e60a0 00007f7b4f5e5984 00007f7b48b606d4 00007f7b48b60662
0709 12:25:09.508702 (+532342us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":28382976}
W20260709 12:25:09.516247 17802 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:09.516800 17802 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:09.517134 17802 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:09.518999 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 95858.8us
I20260709 12:25:09.519099 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:09.517355 17801 leader_election.cc:304] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 31 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:09.521497 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:09.518496 17755 raft_consensus.cc:2468] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 24.
I20260709 12:25:09.491648 27638 raft_consensus.cc:3060] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:25:09.533435 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:09.531355 17652 leader_election.cc:304] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:25:09.523811 27580 raft_consensus.cc:2749] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:25:09.500007 27394 raft_consensus.cc:493] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:25:09.534114 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:09.483807 27713 consensus_queue.cc:1048] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.041s
I20260709 12:25:09.535306 17728 leader_election.cc:304] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:25:09.461871 27570 consensus_queue.cc:237] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 29.19, Last appended by leader: 19, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:09.530382 27111 raft_consensus.cc:493] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:09.557072 17728 leader_election.cc:304] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:25:09.549259 27505 raft_consensus.cc:2804] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:25:09.561116 27505 raft_consensus.cc:697] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 28 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:25:09.561998 27505 consensus_queue.cc:237] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 26.16, Last appended by leader: 16, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:09.539271 17652 leader_election.cc:304] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:25:09.543743 27762 consensus_queue.cc:1048] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.064s
I20260709 12:25:09.567548 17652 leader_election.cc:304] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:25:09.546550 18366 raft_consensus.cc:1240] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 23. Current term is 24. Ops: []
I20260709 12:25:09.547996 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 34 candidate_status { last_received { term: 31 index: 24 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:09.480269 27629 raft_consensus.cc:515] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:09.569098 17758 raft_consensus.cc:2468] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 33.
I20260709 12:25:09.570129 27762 raft_consensus.cc:2804] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:25:09.571082 27629 leader_election.cc:290] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 20 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:25:09.478860 27757 consensus_queue.cc:1059] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 26.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.059s
I20260709 12:25:09.574055 27757 raft_consensus.cc:2804] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:25:09.574652 27757 raft_consensus.cc:697] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 27 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:25:09.513919 27720 leader_election.cc:290] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:25:09.539320 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 58168.2us
I20260709 12:25:09.552129 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 21 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:09.499766 27449 raft_consensus.cc:515] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:09.590082 17728 leader_election.cc:304] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 33 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:09.580865 27062 raft_consensus.cc:493] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:09.576721 27640 raft_consensus.cc:493] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:25:09.552390 27431 raft_consensus.cc:493] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:09.556401 27723 raft_consensus.cc:2804] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:25:09.527146 27525 raft_consensus.cc:515] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:09.552045 27738 raft_consensus.cc:2804] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Leader pre-election won for term 32
W20260709 12:25:09.608222 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 90926.3us
I20260709 12:25:09.583254 27632 raft_consensus.cc:2749] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Leader election lost for term 31. Reason: could not achieve majority
W20260709 12:25:09.606858 27475 consensus_meta.cc:287] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 1.451s	user 0.014s	sys 0.001s
I20260709 12:25:09.587020 27642 raft_consensus.cc:493] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:09.532984 27446 consensus_queue.cc:237] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 23.16, Last appended by leader: 16, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:09.596168 27509 raft_consensus.cc:697] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 31 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:25:09.611857 27703 raft_consensus.cc:2749] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:25:09.496274 27737 consensus_queue.cc:237] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 24.16, Last appended by leader: 16, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:09.609411 27723 raft_consensus.cc:493] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:25:09.584400 27111 raft_consensus.cc:515] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:09.610855 27738 raft_consensus.cc:493] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:25:09.579277 27394 raft_consensus.cc:515] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:09.650141 17728 leader_election.cc:304] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:25:09.601742 27743 consensus_queue.cc:1059] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 29.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.184s
I20260709 12:25:09.651996 17689 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed"
dest_uuid: "0da169ff82674dc1947055ce43de68a3"
mode: GRACEFUL
new_leader_uuid: "334a4e1f59944544b3b3ff36616b57da"
 from {username='slave'} at 127.0.0.1:54540
I20260709 12:25:09.659464 27755 raft_consensus.cc:2749] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 33 FOLLOWER]: Leader election lost for term 33. Reason: could not achieve majority
I20260709 12:25:09.649590 27761 raft_consensus.cc:2804] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:25:09.612670 27640 raft_consensus.cc:515] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:09.665207 27449 leader_election.cc:290] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 26 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:25:09.703387 27770 log.cc:927] Time spent T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.053s	user 0.000s	sys 0.000s
I20260709 12:25:09.666870 27617 raft_consensus.cc:515] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:09.704514 27740 leader_election.cc:290] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:09.652220 27727 raft_consensus.cc:493] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:09.670197 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:09.706296 17835 raft_consensus.cc:2393] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 27: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
W20260709 12:25:09.682227 17727 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:09.672072 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:09.610211 27062 raft_consensus.cc:515] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:09.612911 27431 raft_consensus.cc:515] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:09.549882 27724 consensus_queue.cc:237] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 23.17, Last appended by leader: 17, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:09.671998 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 27 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:09.690280 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:09.670037 27752 raft_consensus.cc:2804] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Leader pre-election won for term 31
W20260709 12:25:09.691439 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 123958us
I20260709 12:25:09.691991 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 21 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:09.764325 17829 raft_consensus.cc:2468] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 26.
W20260709 12:25:09.670815 18333 connection.cc:256] RPC call timeout handler was delayed by -1.7703892399999859s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2892s for a response
W20260709 12:25:09.695847 27768 log.cc:927] Time spent T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.060s	user 0.000s	sys 0.000s
I20260709 12:25:09.769703 27245 raft_consensus.cc:493] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:09.770255 27245 raft_consensus.cc:3060] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:25:09.770481 17764 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38776 (request call id 3340) took 1751 ms. Trace:
I20260709 12:25:09.770907 17764 rpcz_store.cc:276] 0709 12:25:08.018741 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:08.018906 (+   165us) service_pool.cc:224] Handling call
0709 12:25:08.035290 (+ 16384us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:09.005109 (+969819us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:09.770226 (+765117us) spinlock_profiling.cc:243] Waited 964 ms on lock 0x7b54002977d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:25:09.770417 (+   191us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2216604544}
W20260709 12:25:09.774904 27369 consensus_meta.cc:287] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 1.042s	user 0.011s	sys 0.000s
I20260709 12:25:09.775868 27752 raft_consensus.cc:493] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:25:09.776361 27752 raft_consensus.cc:3060] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:09.778976 27692 leader_election.cc:290] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:09.655628 27762 raft_consensus.cc:697] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 30 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
W20260709 12:25:09.697983 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 47931.4us
I20260709 12:25:09.447544 17830 rpcz_store.cc:276] 0709 12:25:08.070749 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:08.076551 (+  5802us) service_pool.cc:224] Handling call
0709 12:25:08.113363 (+ 36812us) spinlock_profiling.cc:243] Waited 12.2 ms on lock 0x7b580013fe98. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b474fbb71 00007f7b474fba22 00007f7b504148d9 00007f7b4f56eee0 00007f7b4f6380ab 00007f7b4f638131 00007f7b4f5f9fc0 00007f7b4f5e4929 00007f7b4f5e43d2 00007f7b48b5bbd4
0709 12:25:08.113786 (+   423us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:08.114381 (+   595us) raft_consensus.cc:1534] Early marking committed up to index 20
0709 12:25:08.115236 (+   855us) log.cc:844] Serialized 14 byte log entry
0709 12:25:09.435589 (+1320353us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:09.435650 (+    61us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:25:09.441855 (+  6205us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:09.441927 (+    72us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:09.442036 (+   109us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":705130,"prepare.queue_time_us":1385,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":38,"raft.queue_time_us":742904,"spinlock_wait_cycles":27987328,"thread_start_us":582913,"threads_started":1,"wal-append.queue_time_us":619887}
I20260709 12:25:09.699126 17689 raft_consensus.cc:606] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 31 LEADER]: Received request to transfer leadership to TS 334a4e1f59944544b3b3ff36616b57da
I20260709 12:25:09.784008 27640 leader_election.cc:290] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:09.575847 27757 consensus_queue.cc:237] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 24.13, Last appended by leader: 13, Current term: 27, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:09.785091 27746 leader_election.cc:290] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:09.684540 27753 raft_consensus.cc:493] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:09.785920 27753 raft_consensus.cc:515] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:09.787977 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 24 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:09.789034 27394 leader_election.cc:290] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:09.783588 17728 leader_election.cc:304] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:25:09.791355 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:09.793001 27564 raft_consensus.cc:3060] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 32 FOLLOWER]: Advancing to term 33
W20260709 12:25:09.793193 27270 consensus_meta.cc:287] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 1.048s	user 0.010s	sys 0.002s
I20260709 12:25:09.793749 27666 raft_consensus.cc:493] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:09.794214 27666 raft_consensus.cc:515] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:09.794854 27755 raft_consensus.cc:2804] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:25:09.795406 27755 raft_consensus.cc:697] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 27 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:25:09.796945 27062 leader_election.cc:290] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:25:09.798246 27740 consensus_queue.cc:1048] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20260709 12:25:09.796334 27755 consensus_queue.cc:237] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 25.18, Last appended by leader: 18, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:09.801195 27544 consensus_meta.cc:287] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 1.031s	user 0.013s	sys 0.000s
I20260709 12:25:09.801613 27544 raft_consensus.cc:515] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:09.804549 27762 consensus_queue.cc:237] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 29.19, Last appended by leader: 19, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:09.808519 27753 leader_election.cc:290] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:09.809520 27775 raft_consensus.cc:493] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:09.810029 27775 raft_consensus.cc:515] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:09.811100 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:25:09.811220 27666 leader_election.cc:290] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:09.811997 27617 leader_election.cc:290] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 29 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:09.812362 17764 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c"
dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
mode: GRACEFUL
new_leader_uuid: "e063541542a94969898868bf6eb5ce1b"
 from {username='slave'} at 127.0.0.1:49786
I20260709 12:25:09.812944 17764 raft_consensus.cc:606] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 29 LEADER]: Received request to transfer leadership to TS e063541542a94969898868bf6eb5ce1b
W20260709 12:25:09.814075 27395 consensus_meta.cc:287] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.815s	user 0.011s	sys 0.001s
I20260709 12:25:09.814554 27395 raft_consensus.cc:2749] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:25:09.814780 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:09.815421 17687 raft_consensus.cc:2468] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 28.
I20260709 12:25:09.816103 16501 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 5
W20260709 12:25:09.811805 17834 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35040 (request call id 3693) took 1742 ms (client timeout 1920 ms). Trace:
I20260709 12:25:09.707484 17831 raft_consensus.cc:2468] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 26.
I20260709 12:25:09.690960 27723 raft_consensus.cc:3060] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:09.713310 27680 raft_consensus.cc:3055] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:25:09.651216 27475 raft_consensus.cc:515] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:09.803625 27544 leader_election.cc:290] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 25 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:25:09.731851 27727 raft_consensus.cc:515] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:09.670683 27749 consensus_queue.cc:1048] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.162s
I20260709 12:25:09.742048 17832 raft_consensus.cc:2468] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 25.
I20260709 12:25:09.851215 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 29 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:25:09.852111 17689 raft_consensus.cc:2393] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 29: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:25:09.713238 27111 leader_election.cc:290] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:09.744550 27525 leader_election.cc:290] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 29 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:09.856997 27734 leader_election.cc:290] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:09.856812 27714 raft_consensus.cc:515] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:09.745647 27704 leader_election.cc:290] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:09.749361 27638 raft_consensus.cc:515] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 33 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:09.860216 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:09.862154 27638 leader_election.cc:290] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 33 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:09.756214 27743 raft_consensus.cc:493] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:09.862771 27714 leader_election.cc:290] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 26 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:09.862944 27743 raft_consensus.cc:515] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:09.753481 27509 consensus_queue.cc:237] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 30.17, Last appended by leader: 17, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:09.865376 27743 leader_election.cc:290] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:09.696581 27738 raft_consensus.cc:3060] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:25:09.763998 17763 raft_consensus.cc:2410] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for term 31 because replica has last-logged OpId of term: 30 index: 20, which is greater than that of the candidate, which has last-logged OpId of term: 29 index: 19.
I20260709 12:25:09.700922 27761 raft_consensus.cc:493] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:25:09.767107 17833 raft_consensus.cc:2468] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 20.
I20260709 12:25:09.692474 17838 raft_consensus.cc:1275] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 23 index: 18. Preceding OpId from leader: term: 25 index: 19. (index mismatch)
I20260709 12:25:09.867054 18374 raft_consensus.cc:2368] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 32. Current term is 33.
I20260709 12:25:09.820983 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 28 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:09.868350 18368 raft_consensus.cc:2376] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 28. Re-sending same reply.
I20260709 12:25:09.869410 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:09.870069 18368 raft_consensus.cc:3060] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:09.652971 27642 raft_consensus.cc:515] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:25:09.818171 17834 rpcz_store.cc:269] 0709 12:25:08.069779 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:08.075712 (+  5933us) service_pool.cc:224] Handling call
0709 12:25:09.811661 (+1735949us) spinlock_profiling.cc:243] Waited 35.8 ms on lock 0x7b54000ab9d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:25:09.811754 (+    93us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":82239360}
I20260709 12:25:09.825767 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 33 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:09.829407 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:25:09.830281 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:09.830350 17729 leader_election.cc:304] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:09.832520 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 34 candidate_status { last_received { term: 30 index: 23 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:09.832716 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 27 candidate_status { last_received { term: 24 index: 20 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:09.833209 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:25:09.775415 27369 raft_consensus.cc:515] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:09.835160 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 32 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:09.835757 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:09.836153 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:09.836939 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:09.823470 27680 raft_consensus.cc:740] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:25:09.853415 17829 raft_consensus.cc:1275] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 21 index: 15. Preceding OpId from leader: term: 22 index: 16. (index mismatch)
I20260709 12:25:09.837970 27725 consensus_queue.cc:1048] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 23, Time since last communication: 0.013s
I20260709 12:25:09.874342 17837 raft_consensus.cc:2468] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 33.
I20260709 12:25:09.860353 27727 leader_election.cc:290] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:09.701673 27703 raft_consensus.cc:2804] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:25:09.872057 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:09.881757 17831 raft_consensus.cc:2468] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 26.
I20260709 12:25:09.837450 27722 raft_consensus.cc:515] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:09.875392 17686 raft_consensus.cc:1240] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 25. Current term is 26. Ops: []
W20260709 12:25:09.875129 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 44863.5us
I20260709 12:25:09.894558 17729 leader_election.cc:304] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:09.895788 18363 raft_consensus.cc:2368] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 24. Current term is 25.
I20260709 12:25:09.887400 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:09.896795 18365 raft_consensus.cc:2393] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 27: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:25:09.881052 17830 raft_consensus.cc:2468] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 27.
I20260709 12:25:09.856550 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 29 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:09.897400 17832 raft_consensus.cc:3060] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:09.890007 27270 raft_consensus.cc:493] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:25:09.904707 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 25 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:09.905373 17833 raft_consensus.cc:3060] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:09.905615 18368 raft_consensus.cc:2468] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 30.
W20260709 12:25:09.875708 27739 consensus_meta.cc:287] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.527s	user 0.002s	sys 0.009s
I20260709 12:25:09.910485 17836 raft_consensus.cc:1240] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 24. Current term is 25. Ops: [24.17-24.17]
I20260709 12:25:09.909968 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:09.914752 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 27 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:09.915115 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:09.873221 27761 raft_consensus.cc:3060] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:09.885545 27704 raft_consensus.cc:2804] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:25:09.917599 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:25:09.918298 17685 raft_consensus.cc:3060] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:09.916496 18368 raft_consensus.cc:2468] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 25.
I20260709 12:25:09.918977 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:09.915836 18365 raft_consensus.cc:2468] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 27.
I20260709 12:25:09.921141 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:09.921746 18365 raft_consensus.cc:2468] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 21.
I20260709 12:25:09.890136 27703 raft_consensus.cc:493] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:09.919624 18363 raft_consensus.cc:2468] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 30.
I20260709 12:25:09.936627 17756 raft_consensus.cc:2368] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 27. Current term is 29.
I20260709 12:25:09.929862 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:25:09.938465 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 53582.6us
I20260709 12:25:09.892513 27564 raft_consensus.cc:493] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:25:09.942377 27394 raft_consensus.cc:493] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:09.940093 27703 raft_consensus.cc:3060] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:09.923857 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:09.964543 17835 raft_consensus.cc:3060] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:09.923161 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:09.948421 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:09.945585 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:09.957984 17832 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 29.
I20260709 12:25:09.952294 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:09.946208 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:09.943624 18368 raft_consensus.cc:2376] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 28. Re-sending same reply.
I20260709 12:25:10.004237 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:09.912914 27525 raft_consensus.cc:2804] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:25:10.003250 18365 raft_consensus.cc:2468] T a5732e71b9d54f8786b8c511d69943bf P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 27.
I20260709 12:25:10.005750 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 26 candidate_status { last_received { term: 24 index: 20 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:10.007601 18368 raft_consensus.cc:2376] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 26. Re-sending same reply.
I20260709 12:25:10.006950 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:10.009785 18365 raft_consensus.cc:2468] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 25.
I20260709 12:25:09.943024 27704 raft_consensus.cc:697] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:25:10.016548 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 26 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:10.017340 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 27 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:09.942497 27782 raft_consensus.cc:493] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:09.945338 27642 leader_election.cc:290] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:25:10.018111 18365 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 26.
I20260709 12:25:09.914021 27270 raft_consensus.cc:515] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:10.025873 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 29 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:10.093683 18363 raft_consensus.cc:3060] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:10.026638 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:10.029188 18368 raft_consensus.cc:1240] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 29. Current term is 30. Ops: [27.16-29.17]
I20260709 12:25:10.094702 17832 raft_consensus.cc:2393] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 31: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:25:10.031774 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 30 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:10.046448 17833 raft_consensus.cc:2468] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 25.
I20260709 12:25:10.095522 17834 raft_consensus.cc:2468] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 29.
I20260709 12:25:10.096086 17835 raft_consensus.cc:2468] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 30.
I20260709 12:25:10.008567 27525 raft_consensus.cc:493] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:10.099484 18368 raft_consensus.cc:1240] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 21. Current term is 22. Ops: []
I20260709 12:25:10.096735 17727 leader_election.cc:304] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
W20260709 12:25:10.028074 27759 consensus_meta.cc:287] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.603s	user 0.000s	sys 0.012s
I20260709 12:25:10.107795 18365 raft_consensus.cc:1240] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 24. Current term is 25. Ops: []
I20260709 12:25:10.111372 18368 raft_consensus.cc:1275] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 24 index: 20. Preceding OpId from leader: term: 26 index: 21. (index mismatch)
I20260709 12:25:10.116490 17833 raft_consensus.cc:1240] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 29. Current term is 30. Ops: [29.20-29.20]
I20260709 12:25:10.093140 27111 raft_consensus.cc:493] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:25:10.116040 27720 raft_consensus.cc:3055] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:25:10.119836 27720 raft_consensus.cc:740] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 30 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:25:10.120669 27720 consensus_queue.cc:260] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 30.18, Last appended by leader: 18, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:10.121701 27720 raft_consensus.cc:3060] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:10.107016 27724 raft_consensus.cc:2804] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:25:10.123611 27724 raft_consensus.cc:697] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 25 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:25:10.124603 27724 consensus_queue.cc:237] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 22.15, Last appended by leader: 15, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:10.136173 27431 leader_election.cc:290] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:10.141196 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:10.142017 27775 leader_election.cc:290] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:25:10.143994 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:25:10.148161 17829 raft_consensus.cc:1240] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 25. Current term is 26. Ops: []
I20260709 12:25:10.149967 27723 raft_consensus.cc:515] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:10.151436 27722 leader_election.cc:290] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 29 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:10.161671 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 34 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:25:10.162633 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:10.104209 27525 raft_consensus.cc:3060] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:10.167079 17762 raft_consensus.cc:1275] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 33 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 30 index: 23. Preceding OpId from leader: term: 33 index: 25. (index mismatch)
I20260709 12:25:10.162317 17681 raft_consensus.cc:2468] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 32.
W20260709 12:25:10.165566 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 242393us
I20260709 12:25:10.169435 17802 leader_election.cc:304] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:25:10.171741 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:10.173418 17764 raft_consensus.cc:1240] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 32 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 31. Current term is 32. Ops: []
I20260709 12:25:10.175168 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 29 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:10.176174 17757 raft_consensus.cc:2393] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 29: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:25:10.180119 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 33 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:10.172526 17755 raft_consensus.cc:2468] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 28.
I20260709 12:25:10.186141 17761 raft_consensus.cc:1275] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 27 index: 15. Preceding OpId from leader: term: 28 index: 16. (index mismatch)
I20260709 12:25:10.187142 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:10.196575 17758 raft_consensus.cc:2393] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 33: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:25:10.187799 17762 raft_consensus.cc:2468] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 25.
I20260709 12:25:10.197692 18361 raft_consensus.cc:1240] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 33 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 32. Current term is 33. Ops: []
I20260709 12:25:10.030282 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:10.048312 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:25:10.045044 17801 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:10.027307 27062 raft_consensus.cc:493] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:10.001367 27394 raft_consensus.cc:515] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:10.033123 27369 leader_election.cc:290] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 25 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:25:10.041467 27338 consensus_meta.cc:287] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.605s	user 0.011s	sys 0.000s
I20260709 12:25:10.090572 17685 raft_consensus.cc:2468] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 20.
I20260709 12:25:09.918503 27739 raft_consensus.cc:515] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Starting leader election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:10.091960 17763 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38776 (request call id 3357) took 1800 ms (client timeout 2124 ms). Trace:
I20260709 12:25:10.101698 17727 leader_election.cc:304] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:10.101718 18333 leader_election.cc:304] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:09.945971 27680 consensus_queue.cc:260] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 26.19, Last appended by leader: 19, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:10.167737 27111 raft_consensus.cc:515] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:10.150341 27671 leader_election.cc:290] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 25 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:09.964872 27564 raft_consensus.cc:515] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:10.199753 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:10.199774 18367 raft_consensus.cc:2468] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 25.
I20260709 12:25:10.085291 27782 raft_consensus.cc:515] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:10.161353 27475 leader_election.cc:290] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 32 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:10.202821 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:25:10.244691 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 143030us
I20260709 12:25:10.246703 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
W20260709 12:25:10.253798 17802 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:10.254761 17802 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:10.244210 17763 rpcz_store.cc:269] 0709 12:25:08.291610 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:08.326820 (+ 35210us) service_pool.cc:224] Handling call
0709 12:25:10.091732 (+1764912us) spinlock_profiling.cc:243] Waited 1.41 s on lock 0x7b54002b6454. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:25:10.091899 (+   167us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3249111424}
I20260709 12:25:10.296564 17681 raft_consensus.cc:1275] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 23 index: 18. Preceding OpId from leader: term: 25 index: 19. (index mismatch)
I20260709 12:25:10.284968 18362 raft_consensus.cc:2468] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 30.
I20260709 12:25:10.283756 17837 raft_consensus.cc:2393] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 25: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:25:10.313665 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 29 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:10.314431 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 32 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:10.315213 17836 raft_consensus.cc:2468] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 31.
I20260709 12:25:10.317030 17683 raft_consensus.cc:1275] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 27 index: 17. Preceding OpId from leader: term: 28 index: 18. (index mismatch)
I20260709 12:25:10.314738 17831 raft_consensus.cc:2393] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 29: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:25:10.320104 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:25:10.320873 17681 raft_consensus.cc:3060] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:10.323014 17830 raft_consensus.cc:1240] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 19. Current term is 20. Ops: []
I20260709 12:25:10.179133 27245 raft_consensus.cc:515] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:10.205081 27666 raft_consensus.cc:2804] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:25:10.203223 27723 leader_election.cc:290] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 24 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:25:10.321892 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 230677us
I20260709 12:25:10.328547 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:10.345160 17763 raft_consensus.cc:2468] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 28.
I20260709 12:25:10.299819 27716 raft_consensus.cc:2804] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:25:10.160123 27759 raft_consensus.cc:515] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:10.337415 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 28 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:10.159938 27704 consensus_queue.cc:237] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 23.17, Last appended by leader: 17, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:10.352432 17687 raft_consensus.cc:2468] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 27.
I20260709 12:25:10.338976 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:25:10.330365 17838 raft_consensus.cc:1240] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 29. Current term is 30. Ops: []
I20260709 12:25:10.293728 27722 raft_consensus.cc:493] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:25:10.298173 27751 raft_consensus.cc:2749] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:25:10.297595 27640 raft_consensus.cc:493] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:10.326089 27737 raft_consensus.cc:493] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:10.329634 27680 raft_consensus.cc:3060] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:25:10.335125 27786 log.cc:927] Time spent T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.195s	user 0.000s	sys 0.001s
I20260709 12:25:10.358546 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:10.367452 17836 raft_consensus.cc:3060] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:25:10.328652 27728 leader_election.cc:290] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:10.369138 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:10.336622 27666 raft_consensus.cc:493] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 33 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:10.364316 16501 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:25:10.387305 17682 raft_consensus.cc:1240] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 26. Current term is 27. Ops: [26.21-26.21]
W20260709 12:25:10.410249 18334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:10.419610 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:25:10.443390 27707 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.861s	user 0.006s	sys 0.015s
I20260709 12:25:10.224557 27062 raft_consensus.cc:515] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:10.369621 27752 consensus_meta.cc:287] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.593s	user 0.008s	sys 0.002s
I20260709 12:25:10.336082 27338 consensus_queue.cc:1048] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.044s
I20260709 12:25:10.333652 27740 consensus_queue.cc:1059] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 25.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.036s
I20260709 12:25:10.495137 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 29 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:10.352582 27638 consensus_queue.cc:1048] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.092s
I20260709 12:25:10.367970 27664 raft_consensus.cc:493] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:25:10.368734 27716 raft_consensus.cc:697] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 30 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:25:10.392853 27666 raft_consensus.cc:3060] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 33 FOLLOWER]: Advancing to term 34
W20260709 12:25:10.487988 27707 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.908s	user 0.006s	sys 0.015s
W20260709 12:25:10.550051 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 51331.6us
I20260709 12:25:10.552619 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:10.555131 17686 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44898 (request call id 1962) took 1141 ms. Trace:
I20260709 12:25:10.368582 27640 raft_consensus.cc:515] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:10.555997 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:10.562049 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 30 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:10.380956 27722 raft_consensus.cc:515] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:10.539755 27270 leader_election.cc:290] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:25:10.464108 27737 raft_consensus.cc:515] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:10.513549 27752 raft_consensus.cc:515] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:10.712134 17801 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:10.616762 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 48978us
W20260709 12:25:10.671175 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 398275us
I20260709 12:25:10.676108 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:10.825273 17836 raft_consensus.cc:2468] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 32.
I20260709 12:25:10.753170 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:10.572584 17686 rpcz_store.cc:276] 0709 12:25:09.413504 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:09.418454 (+  4950us) service_pool.cc:224] Handling call
0709 12:25:09.451115 (+ 32661us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:09.899712 (+448597us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:10.554903 (+655191us) spinlock_profiling.cc:243] Waited 416 ms on lock 0x7b54003485d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:25:10.555070 (+   167us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":957012736}
I20260709 12:25:10.740144 17763 raft_consensus.cc:1240] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 28. Current term is 29. Ops: []
W20260709 12:25:10.606521 18363 consensus_meta.cc:287] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.502s	user 0.007s	sys 0.003s
I20260709 12:25:10.927587 18363 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 29.
I20260709 12:25:10.928090 18363 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59918 (request call id 2864) took 1092 ms. Trace:
W20260709 12:25:10.626719 27795 log.cc:927] Time spent T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.237s	user 0.001s	sys 0.000s
I20260709 12:25:10.928342 18363 rpcz_store.cc:276] 0709 12:25:09.835402 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:10.021385 (+185983us) service_pool.cc:224] Handling call
0709 12:25:10.928042 (+906657us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:25:10.613601 27738 consensus_meta.cc:287] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.748s	user 0.005s	sys 0.005s
I20260709 12:25:10.823190 27369 raft_consensus.cc:493] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:10.561460 27664 raft_consensus.cc:515] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:10.925704 27761 consensus_meta.cc:287] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.923s	user 0.016s	sys 0.003s
I20260709 12:25:10.587423 27564 leader_election.cc:290] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:10.585929 27062 leader_election.cc:290] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:10.918092 27394 leader_election.cc:290] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:25:11.043871 17803 leader_election.cc:336] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Timed out: RequestConsensusVote RPC to 127.16.29.68:42577 timed out after 1.578s (SENT)
I20260709 12:25:10.626976 27725 consensus_queue.cc:1059] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 23.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.416s
I20260709 12:25:11.052273 17834 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35042 (request call id 3519) took 1186 ms. Trace:
I20260709 12:25:11.012859 27431 raft_consensus.cc:493] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:25:11.035715 18333 leader_election.cc:304] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:11.057173 17834 rpcz_store.cc:276] 0709 12:25:09.865407 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:10.101111 (+235704us) service_pool.cc:224] Handling call
0709 12:25:10.115229 (+ 14118us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:10.118242 (+  3013us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:10.118275 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:10.168203 (+ 49928us) log.cc:844] Serialized 26 byte log entry
0709 12:25:10.371905 (+203702us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:25:10.371952 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:10.372042 (+    90us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:11.052051 (+680009us) raft_consensus.cc:1685] finished
0709 12:25:11.052074 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:11.052213 (+   139us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1276,"raft.run_cpu_time_us":104,"raft.run_wall_time_us":35952,"thread_start_us":159309,"threads_started":1,"wal-append.queue_time_us":654471}
W20260709 12:25:11.062950 17835 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.566s	user 0.006s	sys 0.019s
W20260709 12:25:11.063458 17835 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.567s	user 0.006s	sys 0.019s
I20260709 12:25:11.066825 18333 leader_election.cc:304] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:25:11.058262 17803 leader_election.cc:304] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:25:11.044648 27245 leader_election.cc:290] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 32 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:11.085415 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 34 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:11.086335 17764 raft_consensus.cc:2468] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 33.
W20260709 12:25:11.066537 27505 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.525s	user 0.014s	sys 0.012s
I20260709 12:25:11.070375 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 34 candidate_status { last_received { term: 30 index: 23 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:25:11.091632 18371 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.510s	user 0.010s	sys 0.021s
I20260709 12:25:11.097855 18333 leader_election.cc:304] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:11.036535 27759 leader_election.cc:290] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 27 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:25:11.060432 27703 consensus_meta.cc:287] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 1.058s	user 0.010s	sys 0.001s
W20260709 12:25:11.065232 27680 consensus_meta.cc:287] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.684s	user 0.005s	sys 0.006s
I20260709 12:25:11.010895 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 34 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:11.046239 27761 raft_consensus.cc:515] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:11.043569 27369 raft_consensus.cc:515] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:11.190325 18371 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.609s	user 0.010s	sys 0.023s
W20260709 12:25:11.193879 17681 consensus_meta.cc:287] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.862s	user 0.010s	sys 0.000s
I20260709 12:25:11.075984 27638 raft_consensus.cc:2804] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Leader pre-election won for term 29
W20260709 12:25:11.095167 27792 log.cc:927] Time spent T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.534s	user 0.000s	sys 0.001s
I20260709 12:25:11.196851 17758 raft_consensus.cc:2468] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 33.
I20260709 12:25:11.199162 17681 raft_consensus.cc:2468] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 25.
I20260709 12:25:11.090550 27338 raft_consensus.cc:2804] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Leader pre-election won for term 24
W20260709 12:25:11.131925 27505 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.613s	user 0.014s	sys 0.015s
I20260709 12:25:11.113163 27739 leader_election.cc:290] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 28 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:11.247301 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 32 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:11.248440 18367 raft_consensus.cc:2468] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 31.
I20260709 12:25:11.189055 27723 raft_consensus.cc:493] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:11.117350 27725 raft_consensus.cc:3055] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 23 LEADER]: Stepping down as leader of term 23
W20260709 12:25:11.249797 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:11.195652 27394 raft_consensus.cc:2749] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
W20260709 12:25:11.249331 18365 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.921s	user 0.020s	sys 0.061s
W20260709 12:25:11.254292 18365 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.926s	user 0.020s	sys 0.061s
I20260709 12:25:11.249449 27338 raft_consensus.cc:493] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:25:11.258827 27338 raft_consensus.cc:3060] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:25:11.259136 27799 log.cc:927] Time spent T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.320s	user 0.000s	sys 0.001s
W20260709 12:25:11.261433 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 233129us
W20260709 12:25:11.262995 17803 leader_election.cc:336] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 32 election: RPC error from VoteRequest() call to peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Timed out: RequestConsensusVote RPC to 127.16.29.68:42577 timed out after 1.664s (SENT)
W20260709 12:25:11.263958 27801 log.cc:927] Time spent T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.603s	user 0.000s	sys 0.001s
I20260709 12:25:11.267824 27338 raft_consensus.cc:515] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:11.270949 17803 leader_election.cc:336] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Timed out: RequestConsensusVote RPC to 127.16.29.68:42577 timed out after 1.702s (SENT)
W20260709 12:25:11.251107 27787 log.cc:927] Time spent T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 1.119s	user 0.007s	sys 0.001s
I20260709 12:25:11.271515 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:11.205958 27638 raft_consensus.cc:493] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:25:11.231562 27680 raft_consensus.cc:2749] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:25:11.057734 27738 raft_consensus.cc:515] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 32 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:11.252996 27796 log.cc:927] Time spent T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.702s	user 0.001s	sys 0.000s
W20260709 12:25:11.276414 17803 leader_election.cc:336] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 28 election: RPC error from VoteRequest() call to peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Timed out: RequestConsensusVote RPC to 127.16.29.68:42577 timed out after 1.707s (SENT)
I20260709 12:25:11.280133 17803 leader_election.cc:304] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:25:11.195732 27431 raft_consensus.cc:515] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:11.202661 27703 raft_consensus.cc:515] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:11.257797 27725 raft_consensus.cc:740] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:25:11.281888 27338 leader_election.cc:290] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 24 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:11.278088 27638 raft_consensus.cc:3060] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:11.320083 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:11.323611 27338 raft_consensus.cc:493] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:25:11.326100 27338 raft_consensus.cc:515] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:11.328305 27338 leader_election.cc:290] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:25:11.330317 27720 consensus_meta.cc:287] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 1.208s	user 0.007s	sys 0.004s
W20260709 12:25:11.332403 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 56020us
W20260709 12:25:11.333673 27707 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.516s	user 0.002s	sys 0.009s
W20260709 12:25:11.334024 27707 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.584s	user 0.002s	sys 0.009s
W20260709 12:25:11.334240 17803 leader_election.cc:336] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Timed out: RequestConsensusVote RPC to 127.16.29.68:42577 timed out after 1.870s (SENT)
W20260709 12:25:11.335294 17803 leader_election.cc:336] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Timed out: RequestConsensusVote RPC to 127.16.29.68:42577 timed out after 1.875s (SENT)
W20260709 12:25:11.339332 27812 log.cc:927] Time spent T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.051s	user 0.000s	sys 0.000s
I20260709 12:25:11.341960 27431 leader_election.cc:290] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:11.271041 27723 raft_consensus.cc:515] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:25:11.344203 17801 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:11.345402 27723 leader_election.cc:290] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:11.284448 27394 raft_consensus.cc:2749] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260709 12:25:11.335785 17803 leader_election.cc:304] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:25:11.353883 17755 raft_consensus.cc:1240] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 28. Current term is 29. Ops: []
I20260709 12:25:11.354007 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:11.354992 17763 raft_consensus.cc:2393] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 32: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:25:11.355120 27738 leader_election.cc:290] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 32 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:25:11.367070 18367 raft_consensus.cc:2468] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 25.
I20260709 12:25:11.353339 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 28 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:11.372164 17758 raft_consensus.cc:2393] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 28: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:25:11.382611 27761 leader_election.cc:290] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 30 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:25:11.387343 17655 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:11.387732 27638 raft_consensus.cc:515] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:11.371253 27431 raft_consensus.cc:2749] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:25:11.374015 27737 leader_election.cc:290] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:11.346294 27725 consensus_queue.cc:260] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 23.17, Last appended by leader: 17, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:11.376538 27640 leader_election.cc:290] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:25:11.402377 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 67144.2us
I20260709 12:25:11.409140 18364 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 2863) took 1596 ms. Trace:
W20260709 12:25:11.415055 17803 leader_election.cc:336] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Timed out: RequestConsensusVote RPC to 127.16.29.68:42577 timed out after 1.906s (SENT)
I20260709 12:25:11.392802 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 30 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:25:11.392069 18374 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.521s	user 0.013s	sys 0.004s
W20260709 12:25:11.415833 18374 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.549s	user 0.013s	sys 0.004s
W20260709 12:25:11.395344 27666 consensus_meta.cc:287] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.781s	user 0.001s	sys 0.010s
I20260709 12:25:11.390259 27664 leader_election.cc:290] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:11.413386 18364 rpcz_store.cc:276] 0709 12:25:09.812959 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:09.816115 (+  3156us) service_pool.cc:224] Handling call
0709 12:25:09.816423 (+   308us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:09.816757 (+   334us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:09.816783 (+    26us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:09.817289 (+   506us) log.cc:844] Serialized 26 byte log entry
0709 12:25:09.917011 (+ 99722us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:25:09.917056 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:09.917124 (+    68us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:10.934607 (+1017483us) spinlock_profiling.cc:243] Waited 6.42 ms on lock 0x7b4400022b4c. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b47459c85 00007f7b4745a6a2 00007f7b4a1cc1db 00007f7b4a1c7aaf 00007f7b4a1bb86e 00007f7b4a1babb6 00007f7b4a1ba7d8 00007f7b4a1c4221 00007f7b4a1c41d9 00007f7b4a1c4169
0709 12:25:11.394511 (+459904us) raft_consensus.cc:1685] finished
0709 12:25:11.394534 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:11.409040 (+ 14506us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":5238,"raft.run_cpu_time_us":163,"raft.run_wall_time_us":8394,"spinlock_wait_cycles":14755712,"thread_start_us":99245,"threads_started":1,"wal-append.queue_time_us":283815}
W20260709 12:25:11.420992 17654 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:11.423743 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 24 candidate_status { last_received { term: 23 index: 21 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
W20260709 12:25:11.416944 17833 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.500s	user 0.015s	sys 0.036s
W20260709 12:25:11.427242 17833 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.678s	user 0.015s	sys 0.036s
I20260709 12:25:11.431540 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 32 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:11.433097 18367 raft_consensus.cc:2393] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 32: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:25:11.421746 17729 leader_election.cc:304] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:10.944626 27716 consensus_queue.cc:237] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 28.19, Last appended by leader: 19, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:25:11.436789 17803 leader_election.cc:336] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Timed out: RequestConsensusVote RPC to 127.16.29.68:42577 timed out after 2.020s (SENT)
I20260709 12:25:11.391584 27722 leader_election.cc:290] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:11.439960 17758 raft_consensus.cc:1275] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 30 index: 17. Preceding OpId from leader: term: 31 index: 18. (index mismatch)
I20260709 12:25:11.441140 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 29 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:11.442276 27431 raft_consensus.cc:493] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:11.443032 27720 raft_consensus.cc:493] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:25:11.445592 18365 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59902 (request call id 2329) took 1580 ms. Trace:
W20260709 12:25:11.465021 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:11.471536 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 373731us
I20260709 12:25:11.440735 27725 raft_consensus.cc:3060] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:11.501856 18333 leader_election.cc:304] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:11.503067 18364 raft_consensus.cc:1240] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 29. Current term is 30. Ops: []
I20260709 12:25:11.528139 18333 leader_election.cc:304] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:11.419533 27666 raft_consensus.cc:515] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 34 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:11.529919 18333 leader_election.cc:304] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:25:11.459476 18365 rpcz_store.cc:276] 0709 12:25:09.865496 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:10.115334 (+249838us) service_pool.cc:224] Handling call
0709 12:25:10.115607 (+   273us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:10.160046 (+ 44439us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:10.160074 (+    28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:10.204864 (+ 44790us) log.cc:844] Serialized 26 byte log entry
0709 12:25:11.254767 (+1049903us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:25:11.254819 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:11.254908 (+    89us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:11.445390 (+190482us) raft_consensus.cc:1685] finished
0709 12:25:11.445412 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:11.445539 (+   127us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":41816,"raft.run_cpu_time_us":146,"raft.run_wall_time_us":15979,"thread_start_us":926815,"threads_started":1,"wal-append.queue_time_us":1123768}
I20260709 12:25:11.557734 18333 leader_election.cc:304] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:25:11.618777 17652 leader_election.cc:304] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:11.538658 18365 raft_consensus.cc:1240] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 32 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 31. Current term is 32. Ops: []
I20260709 12:25:11.597297 18361 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59902 (request call id 2309) took 1806 ms. Trace:
I20260709 12:25:11.527701 27617 raft_consensus.cc:493] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:25:11.470875 27720 raft_consensus.cc:515] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:11.540030 27738 raft_consensus.cc:2749] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:25:11.638850 17755 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38786 (request call id 4180) took 1468 ms. Trace:
I20260709 12:25:11.677591 17755 rpcz_store.cc:276] 0709 12:25:10.170227 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:10.182711 (+ 12484us) service_pool.cc:224] Handling call
0709 12:25:10.203247 (+ 20536us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:11.354388 (+1151141us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:11.638638 (+284250us) spinlock_profiling.cc:243] Waited 1.15 s on lock 0x7b540019f0d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:25:11.638785 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2645775744}
I20260709 12:25:11.676487 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 29 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:11.680482 18367 raft_consensus.cc:2393] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 29: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:25:11.562727 18364 raft_consensus.cc:1240] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 23. Current term is 24. Ops: []
I20260709 12:25:11.638850 18365 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 2871) took 1067 ms. Trace:
I20260709 12:25:11.683434 18365 rpcz_store.cc:276] 0709 12:25:10.571312 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:11.536841 (+965529us) service_pool.cc:224] Handling call
0709 12:25:11.537117 (+   276us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:11.638563 (+101446us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:11.638785 (+   222us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:11.639022 18361 rpcz_store.cc:276] 0709 12:25:09.790553 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:09.796025 (+  5472us) service_pool.cc:224] Handling call
0709 12:25:10.197510 (+401485us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:10.198176 (+   666us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:11.597012 (+1398836us) spinlock_profiling.cc:243] Waited 401 ms on lock 0x7b5400305f50. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a61f8ad 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:25:11.597224 (+   212us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":922534272}
W20260709 12:25:11.711405 18371 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.513s	user 0.015s	sys 0.093s
W20260709 12:25:11.711817 18371 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.513s	user 0.015s	sys 0.093s
I20260709 12:25:11.712409 18364 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 2870) took 1141 ms. Trace:
W20260709 12:25:11.712563 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 93904.4us
I20260709 12:25:11.713241 18367 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59886 (request call id 2674) took 1141 ms. Trace:
I20260709 12:25:11.712795 18364 rpcz_store.cc:276] 0709 12:25:10.571140 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:11.534664 (+963524us) service_pool.cc:224] Handling call
0709 12:25:11.562492 (+ 27828us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:11.681337 (+118845us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:11.712346 (+ 31009us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:11.712795 18371 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 2868) took 1143 ms. Trace:
I20260709 12:25:11.637441 27617 raft_consensus.cc:515] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:25:11.715692 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 278918us
I20260709 12:25:11.716216 18365 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59886 (request call id 2677) took 1066 ms. Trace:
I20260709 12:25:11.713801 18367 rpcz_store.cc:276] 0709 12:25:10.571809 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:11.462749 (+890940us) service_pool.cc:224] Handling call
0709 12:25:11.713181 (+250432us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:25:11.717470 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 159847us
I20260709 12:25:11.462977 27722 raft_consensus.cc:2804] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Leader pre-election won for term 32
W20260709 12:25:11.595155 27791 log.cc:927] Time spent T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 1.443s	user 0.000s	sys 0.017s
W20260709 12:25:11.610412 27784 log.cc:927] Time spent T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 1.478s	user 0.006s	sys 0.000s
I20260709 12:25:11.718416 18333 leader_election.cc:304] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:25:11.721843 27720 leader_election.cc:290] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:11.595582 27703 leader_election.cc:290] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 29 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:11.620831 27823 raft_consensus.cc:493] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:11.612682 27666 leader_election.cc:290] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 34 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:25:11.588063 27664 raft_consensus.cc:2804] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Leader pre-election won for term 27
W20260709 12:25:11.728610 27821 log.cc:927] Time spent T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.082s	user 0.000s	sys 0.000s
I20260709 12:25:11.716604 18365 rpcz_store.cc:276] 0709 12:25:10.650259 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:11.715411 (+1065152us) service_pool.cc:224] Handling call
0709 12:25:11.715749 (+   338us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:11.715936 (+   187us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:11.715960 (+    24us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:11.715999 (+    39us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:25:11.716026 (+    27us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:11.716072 (+    46us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:11.716171 (+    99us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:25:11.732777 27525 consensus_meta.cc:287] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 1.545s	user 0.009s	sys 0.005s
I20260709 12:25:11.714775 18371 rpcz_store.cc:276] 0709 12:25:10.570648 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:10.570784 (+   136us) service_pool.cc:224] Handling call
0709 12:25:10.580470 (+  9686us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:10.580818 (+   348us) raft_consensus.cc:1534] Early marking committed up to index 19
0709 12:25:11.198512 (+617694us) log.cc:844] Serialized 14 byte log entry
0709 12:25:11.712461 (+513949us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:11.712530 (+    69us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:25:11.712574 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:11.712640 (+    66us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:11.712747 (+   107us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":467719,"prepare.run_cpu_time_us":58,"prepare.run_wall_time_us":60,"raft.queue_time_us":3171,"raft.run_cpu_time_us":102,"raft.run_wall_time_us":3238,"thread_start_us":1130627,"threads_started":2}
I20260709 12:25:11.736302 27751 raft_consensus.cc:2804] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:25:11.719497 27762 raft_consensus.cc:2749] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
W20260709 12:25:11.563606 27818 log.cc:927] Time spent T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.059s	user 0.000s	sys 0.000s
I20260709 12:25:11.743269 17688 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37396 (request call id 3483) took 1909 ms. Trace:
I20260709 12:25:11.743646 18333 leader_election.cc:304] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
W20260709 12:25:11.762239 17803 leader_election.cc:336] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Timed out: RequestConsensusVote RPC to 127.16.29.68:42577 timed out after 2.040s (SENT)
I20260709 12:25:11.622699 27767 raft_consensus.cc:493] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:25:11.535686 27761 raft_consensus.cc:2749] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:25:11.463537 27431 raft_consensus.cc:515] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:11.594524 27638 leader_election.cc:290] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 29 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:25:11.779289 18336 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.525s	user 0.013s	sys 0.060s
W20260709 12:25:11.794160 18336 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.540s	user 0.013s	sys 0.060s
I20260709 12:25:11.763406 27680 raft_consensus.cc:2804] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:25:11.774045 27751 raft_consensus.cc:493] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:25:11.824332 27751 raft_consensus.cc:3060] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:11.797704 27767 raft_consensus.cc:515] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:11.764662 27664 raft_consensus.cc:493] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:25:11.830217 27664 raft_consensus.cc:3060] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:11.830691 27767 leader_election.cc:290] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:11.764605 27823 raft_consensus.cc:515] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:11.833534 27782 leader_election.cc:290] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:25:11.833712 27823 leader_election.cc:290] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:11.791296 27338 consensus_queue.cc:1048] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.044s
W20260709 12:25:11.840104 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 47528.6us
W20260709 12:25:11.792141 17655 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:11.796859 27680 raft_consensus.cc:493] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:11.841693 27680 raft_consensus.cc:3060] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:11.802429 27762 consensus_queue.cc:1059] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 29.20, Next index: 21, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:25:11.805337 27431 leader_election.cc:290] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:25:11.808089 17835 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35042 (request call id 3521) took 1942 ms. Trace:
I20260709 12:25:11.861768 17835 rpcz_store.cc:276] 0709 12:25:09.868683 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:10.100696 (+232013us) service_pool.cc:224] Handling call
0709 12:25:10.113985 (+ 13289us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:10.344330 (+230345us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:10.344376 (+    46us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:10.356197 (+ 11821us) log.cc:844] Serialized 26 byte log entry
0709 12:25:11.069030 (+712833us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:25:11.069073 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:11.807614 (+738541us) spinlock_profiling.cc:243] Waited 206 ms on lock 0x7b54002ebb54. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:25:11.807667 (+    53us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:11.807725 (+    58us) raft_consensus.cc:1685] finished
0709 12:25:11.807743 (+    18us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:11.808024 (+   281us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":15792,"raft.run_cpu_time_us":115,"raft.run_wall_time_us":1794,"spinlock_wait_cycles":474445184,"thread_start_us":566879,"threads_started":1,"wal-append.queue_time_us":898308}
I20260709 12:25:11.811147 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:11.740336 27722 raft_consensus.cc:493] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:25:11.812685 27788 log.cc:927] Time spent T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 1.698s	user 0.011s	sys 0.023s
W20260709 12:25:11.801489 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:11.813450 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 96923.4us
W20260709 12:25:11.814179 27807 log.cc:927] Time spent T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.491s	user 0.010s	sys 0.003s
I20260709 12:25:11.815776 27369 leader_election.cc:290] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:11.817823 27197 raft_consensus.cc:3060] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:11.777643 17688 rpcz_store.cc:276] 0709 12:25:09.834270 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:09.853508 (+ 19238us) service_pool.cc:224] Handling call
0709 12:25:09.856604 (+  3096us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:09.917403 (+ 60799us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:09.917432 (+    29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:09.937086 (+ 19654us) log.cc:844] Serialized 26 byte log entry
0709 12:25:10.008170 (+ 71084us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:25:10.008208 (+    38us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:10.884540 (+876332us) spinlock_profiling.cc:243] Waited 14.9 ms on lock 0x7b54001388d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:25:10.922418 (+ 37878us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:11.594394 (+671976us) spinlock_profiling.cc:243] Waited 9.37 ms on lock 0x7b4400022b4c. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b47459c85 00007f7b4745a6a2 00007f7b4a1cc1db 00007f7b4a1c7aaf 00007f7b4a1bb86e 00007f7b4a1babb6 00007f7b4a1ba7d8 00007f7b4a1c4221 00007f7b4a1c41d9 00007f7b4a1c4169
0709 12:25:11.742938 (+148544us) raft_consensus.cc:1685] finished
0709 12:25:11.742960 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:11.743212 (+   252us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":95535,"raft.queue_time_us":46236,"raft.run_cpu_time_us":178,"raft.run_wall_time_us":33892,"spinlock_wait_cycles":55757952,"thread_start_us":61579,"threads_started":1,"wal-append.queue_time_us":176945}
I20260709 12:25:11.776963 27525 raft_consensus.cc:515] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:11.063050 16501 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:25:11.852733 17652 leader_election.cc:304] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:25:11.873562 18333 leader_election.cc:304] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
W20260709 12:25:11.866966 17759 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38786 (request call id 4182) took 1686 ms (client timeout 2204 ms). Trace:
W20260709 12:25:11.878046 17759 rpcz_store.cc:269] 0709 12:25:10.180494 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:10.182055 (+  1561us) service_pool.cc:224] Handling call
0709 12:25:11.811012 (+1628957us) spinlock_profiling.cc:243] Waited 1.44 ms on lock 0x7b5800033398. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b474fbb71 00007f7b474fbeee 000000000036eb39 00007f7b4f56f1e0 00007f7b4f6380c4 00007f7b4f638131 00007f7b4f5e60a0 00007f7b4f5e5984 00007f7b48b606d4 00007f7b48b60662
0709 12:25:11.866877 (+ 55865us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3319680}
I20260709 12:25:11.873767 18365 raft_consensus.cc:1275] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 21 index: 16. Preceding OpId from leader: term: 25 index: 18. (index mismatch)
W20260709 12:25:11.873528 17803 leader_election.cc:336] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Timed out: RequestConsensusVote RPC to 127.16.29.68:42577 timed out after 2.171s (SENT)
I20260709 12:25:11.879356 17803 leader_election.cc:304] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:11.893210 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 28 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:11.894016 18367 raft_consensus.cc:3060] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:11.887674 18333 leader_election.cc:400] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 27 pre-election: Vote denied by peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651) with higher term. Message: Invalid argument: T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 27. Current term is 29.
I20260709 12:25:11.896442 27590 raft_consensus.cc:2804] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:25:11.896713 18333 leader_election.cc:403] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 27 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:25:11.896919 27590 raft_consensus.cc:493] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:25:11.897344 27590 raft_consensus.cc:3060] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:11.897668 17760 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38776 (request call id 3361) took 1757 ms. Trace:
I20260709 12:25:11.898049 17760 rpcz_store.cc:276] 0709 12:25:10.154679 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:10.159076 (+  4397us) service_pool.cc:224] Handling call
0709 12:25:10.159445 (+   369us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:25:10.159836 (+   391us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:10.159869 (+    33us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:25:10.176584 (+ 16715us) log.cc:844] Serialized 52 byte log entry
0709 12:25:10.660208 (+483624us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:25:10.660258 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:10.660344 (+    86us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:11.813789 (+1153445us) spinlock_profiling.cc:243] Waited 45.4 ms on lock 0x7b24000f7418. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b473b99c4 00007f7b473b7acb 00007f7b47572dc9 00007f7b47572a55 00007f7b4757209d 00007f7b4a1a410e 00007f7b4a1bb88c 00007f7b4a1babb6 00007f7b4a1ba7d8 00007f7b4a1c4221
0709 12:25:11.897341 (+ 83552us) raft_consensus.cc:1685] finished
0709 12:25:11.897372 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:11.897601 (+   229us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":731,"raft.run_cpu_time_us":141,"raft.run_wall_time_us":9575,"spinlock_wait_cycles":104510720,"thread_start_us":435182,"threads_started":1,"wal-append.queue_time_us":1052329}
I20260709 12:25:11.867781 27722 raft_consensus.cc:3060] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Advancing to term 32
W20260709 12:25:11.902653 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 2.06744e+06us
I20260709 12:25:11.863031 27638 raft_consensus.cc:493] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:11.904974 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 30 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:11.872291 27826 raft_consensus.cc:493] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:11.905066 27828 raft_consensus.cc:2804] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:25:11.911075 17829 raft_consensus.cc:1275] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 29 index: 17. Preceding OpId from leader: term: 30 index: 18. (index mismatch)
I20260709 12:25:11.911506 27823 consensus_queue.cc:1048] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20260709 12:25:11.912314 17834 raft_consensus.cc:1275] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 26 index: 20. Preceding OpId from leader: term: 27 index: 21. (index mismatch)
W20260709 12:25:11.912441 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 44987us
I20260709 12:25:11.905079 27369 raft_consensus.cc:2749] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:25:11.905784 27720 consensus_queue.cc:1059] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.010s
I20260709 12:25:11.915205 17836 raft_consensus.cc:1240] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 30. Current term is 31. Ops: [30.18-30.18]
I20260709 12:25:11.916484 17832 raft_consensus.cc:1275] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 25 index: 15. Preceding OpId from leader: term: 28 index: 17. (index mismatch)
I20260709 12:25:11.940398 18361 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59886 (request call id 2680) took 1122 ms. Trace:
I20260709 12:25:11.924239 18367 raft_consensus.cc:2468] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 28.
I20260709 12:25:11.924996 17762 raft_consensus.cc:2393] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 30: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:25:11.926414 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 27 candidate_status { last_received { term: 24 index: 20 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:11.926956 17837 raft_consensus.cc:2410] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for term 30 because replica has last-logged OpId of term: 29 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 27 index: 16.
W20260709 12:25:11.941128 17803 leader_election.cc:336] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 24 election: RPC error from VoteRequest() call to peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Timed out: RequestConsensusVote RPC to 127.16.29.68:42577 timed out after 2.222s (SENT)
I20260709 12:25:11.862569 27767 consensus_queue.cc:1048] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 23, Time since last communication: 0.042s
I20260709 12:25:11.932163 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:11.937821 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:11.945663 17830 raft_consensus.cc:3060] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:11.945722 18371 raft_consensus.cc:2376] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 30. Re-sending same reply.
I20260709 12:25:11.928546 17835 raft_consensus.cc:1240] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 32 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 31. Current term is 32. Ops: [31.18-31.18]
I20260709 12:25:11.943106 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 32 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:11.905903 27753 raft_consensus.cc:3060] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:11.946846 18367 raft_consensus.cc:2468] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 31.
W20260709 12:25:11.944020 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 146578us
I20260709 12:25:11.942368 17803 leader_election.cc:304] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:11.951705 18363 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 2872) took 1380 ms. Trace:
I20260709 12:25:11.952451 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:11.932986 27828 raft_consensus.cc:493] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 33 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:25:11.957917 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:11.959475 17835 raft_consensus.cc:1275] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 22 index: 12. Preceding OpId from leader: term: 27 index: 14. (index mismatch)
I20260709 12:25:11.943804 27738 raft_consensus.cc:2804] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:25:11.961297 18367 raft_consensus.cc:2468] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 27.
I20260709 12:25:11.961690 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:25:11.956696 18363 rpcz_store.cc:276] 0709 12:25:10.571468 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:11.715720 (+1144252us) service_pool.cc:224] Handling call
0709 12:25:11.715968 (+   248us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:11.716135 (+   167us) raft_consensus.cc:1534] Early marking committed up to index 16
0709 12:25:11.720510 (+  4375us) log.cc:844] Serialized 14 byte log entry
0709 12:25:11.951383 (+230873us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:11.951435 (+    52us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:25:11.951476 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:11.951536 (+    60us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:11.951652 (+   116us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":6448,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":33,"raft.queue_time_us":86,"thread_start_us":230495,"threads_started":1}
I20260709 12:25:11.962798 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:11.961395 27828 raft_consensus.cc:3060] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:25:11.965579 27111 leader_election.cc:290] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:11.933423 27638 raft_consensus.cc:515] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:11.963562 27338 raft_consensus.cc:493] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:11.970417 27338 raft_consensus.cc:515] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:11.973517 18374 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 2867) took 1404 ms. Trace:
I20260709 12:25:11.973927 18374 rpcz_store.cc:276] 0709 12:25:10.569313 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:10.569878 (+   565us) service_pool.cc:224] Handling call
0709 12:25:10.570247 (+   369us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:10.570573 (+   326us) raft_consensus.cc:1534] Early marking committed up to index 24
0709 12:25:11.416262 (+845689us) log.cc:844] Serialized 14 byte log entry
0709 12:25:11.797334 (+381072us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:11.848474 (+ 51140us) log.cc:844] Serialized 26 byte log entry
0709 12:25:11.863187 (+ 14713us) raft_consensus.cc:1645] Marking committed up to 24
0709 12:25:11.863231 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:11.863295 (+    64us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:11.973289 (+109994us) raft_consensus.cc:1685] finished
0709 12:25:11.973311 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:11.973453 (+   142us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":45206,"prepare.queue_time_us":475544,"prepare.run_cpu_time_us":70,"prepare.run_wall_time_us":5100,"raft.queue_time_us":26648,"raft.run_cpu_time_us":249,"raft.run_wall_time_us":2647,"thread_start_us":876363,"threads_started":2,"wal-append.queue_time_us":390352}
I20260709 12:25:11.976835 27525 leader_election.cc:290] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 30 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:11.958555 27369 raft_consensus.cc:2749] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260709 12:25:11.977267 27590 raft_consensus.cc:515] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:11.979349 27590 leader_election.cc:290] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 29 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:11.988066 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 30 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:11.989019 18374 raft_consensus.cc:2468] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 29.
I20260709 12:25:11.996661 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 34 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:11.997848 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 28 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:25:11.998633 17683 raft_consensus.cc:3060] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:11.977396 17834 raft_consensus.cc:1275] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 29 index: 19. Preceding OpId from leader: term: 30 index: 20. (index mismatch)
I20260709 12:25:12.001417 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 28 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:25:12.002487 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 68406.7us
I20260709 12:25:12.002502 17685 raft_consensus.cc:2410] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for term 28 because replica has last-logged OpId of term: 26 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 18.
I20260709 12:25:12.005370 17689 raft_consensus.cc:1275] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 23 index: 16. Preceding OpId from leader: term: 25 index: 17. (index mismatch)
I20260709 12:25:12.006883 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:25:12.008050 17686 raft_consensus.cc:1240] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 19. Current term is 20. Ops: []
I20260709 12:25:12.009430 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:25:12.010205 17688 raft_consensus.cc:3060] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:12.007633 17689 raft_consensus.cc:3060] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:12.014375 27761 raft_consensus.cc:493] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:12.014750 27197 consensus_queue.cc:1059] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 31.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:25:12.014984 27761 raft_consensus.cc:515] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:12.016477 27767 raft_consensus.cc:3055] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 31 LEADER]: Stepping down as leader of term 31
I20260709 12:25:12.015858 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:12.017062 27767 raft_consensus.cc:740] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 31 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:25:12.018011 27767 consensus_queue.cc:260] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 31.18, Last appended by leader: 18, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:12.020217 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:12.020814 27767 raft_consensus.cc:3060] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:25:12.020254 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 32 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:11.958760 27720 raft_consensus.cc:3055] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 31 LEADER]: Stepping down as leader of term 31
I20260709 12:25:11.983244 27751 raft_consensus.cc:515] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:12.028160 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 29 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:12.033389 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:25:12.034030 17687 raft_consensus.cc:3060] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:11.936297 27826 raft_consensus.cc:515] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:12.036856 17803 leader_election.cc:304] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:25:12.040251 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 24 candidate_status { last_received { term: 23 index: 21 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:12.045001 18367 raft_consensus.cc:2393] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 24: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:25:12.046265 17834 raft_consensus.cc:1275] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 27 index: 15. Preceding OpId from leader: term: 28 index: 16. (index mismatch)
I20260709 12:25:12.048117 17684 raft_consensus.cc:2468] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 27.
W20260709 12:25:12.046046 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 123924us
I20260709 12:25:12.026015 27720 raft_consensus.cc:740] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 31 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:25:12.055480 17689 raft_consensus.cc:2410] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for term 29 because replica has last-logged OpId of term: 28 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 27 index: 17.
I20260709 12:25:12.053607 17803 leader_election.cc:304] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:25:12.040512 27369 raft_consensus.cc:493] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:12.050417 27111 raft_consensus.cc:2804] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:25:12.048079 27617 leader_election.cc:290] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:12.080715 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:12.081646 18367 raft_consensus.cc:2393] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 27: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:25:12.092188 17683 raft_consensus.cc:2468] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 28.
I20260709 12:25:12.098475 17688 raft_consensus.cc:2468] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 24.
I20260709 12:25:12.108611 17687 raft_consensus.cc:2468] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 31.
I20260709 12:25:12.111841 27826 leader_election.cc:290] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:25:12.114178 27720 consensus_queue.cc:260] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 31.18, Last appended by leader: 18, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:25:12.116981 27785 log.cc:927] Time spent T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 2.106s	user 0.007s	sys 0.009s
I20260709 12:25:12.119489 18368 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59902 (request call id 2327) took 2254 ms. Trace:
I20260709 12:25:12.031065 27664 raft_consensus.cc:515] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:12.082571 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:12.141256 18367 raft_consensus.cc:2376] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 30. Re-sending same reply.
I20260709 12:25:12.142175 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:12.142781 18367 raft_consensus.cc:2468] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 26.
I20260709 12:25:12.143373 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 27 candidate_status { last_received { term: 24 index: 20 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:12.143726 18367 raft_consensus.cc:3060] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:12.048961 27749 raft_consensus.cc:493] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:12.139149 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 87713.9us
W20260709 12:25:12.140043 18362 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59902 (request call id 2314) took 2299 ms (client timeout 1875 ms). Trace:
I20260709 12:25:12.136812 18368 rpcz_store.cc:276] 0709 12:25:09.864934 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:10.112532 (+247598us) service_pool.cc:224] Handling call
0709 12:25:10.112928 (+   396us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:10.116891 (+  3963us) raft_consensus.cc:1534] Early marking committed up to index 18
0709 12:25:10.500230 (+383339us) log.cc:844] Serialized 14 byte log entry
0709 12:25:12.118928 (+1618698us) spinlock_profiling.cc:243] Waited 5.58 ms on lock 0x7b1800378160. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b4a1c84bc 00007f7b4751648b 00007f7b503709a9 00007f7b4753269b 00007f7b4753093f 00007f7b47532c02 00007f7b4a1ba20b 00007f7b4a1bfd23 00007f7b4a1c023a
0709 12:25:12.119137 (+   209us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:12.119187 (+    50us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:25:12.119233 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:12.119300 (+    67us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:12.119430 (+   130us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":765595,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":41,"raft.queue_time_us":815,"raft.run_cpu_time_us":129,"raft.run_wall_time_us":129,"spinlock_wait_cycles":12830080,"thread_start_us":806790,"threads_started":2,"wal-append.queue_time_us":720815}
I20260709 12:25:12.076457 27111 raft_consensus.cc:493] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:12.146803 27111 raft_consensus.cc:3060] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:12.147504 27749 raft_consensus.cc:515] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:12.148031 27671 raft_consensus.cc:3055] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 LEADER]: Stepping down as leader of term 25
W20260709 12:25:12.145695 18362 rpcz_store.cc:269] 0709 12:25:09.832625 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:09.839053 (+  6428us) service_pool.cc:224] Handling call
0709 12:25:12.132073 (+2293020us) spinlock_profiling.cc:243] Waited 424 ms on lock 0x7b540034c1d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:25:12.132172 (+    99us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":975038208}
I20260709 12:25:12.148932 27840 consensus_queue.cc:1048] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20260709 12:25:12.149358 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 29 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:11.940986 18361 rpcz_store.cc:276] 0709 12:25:10.817965 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:11.690401 (+872436us) service_pool.cc:224] Handling call
0709 12:25:11.739565 (+ 49164us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:11.739841 (+   276us) raft_consensus.cc:1534] Early marking committed up to index 15
0709 12:25:11.768061 (+ 28220us) log.cc:844] Serialized 14 byte log entry
0709 12:25:11.939620 (+171559us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:11.939675 (+    55us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:25:11.939719 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:11.940261 (+   542us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:11.940350 (+    89us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":25914,"prepare.run_cpu_time_us":53,"prepare.run_wall_time_us":1627,"raft.queue_time_us":4755,"thread_start_us":170602,"threads_started":1}
I20260709 12:25:12.150843 17652 leader_election.cc:304] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:12.151288 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 28 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:12.151628 27840 raft_consensus.cc:2804] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:25:12.152029 27840 raft_consensus.cc:493] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:25:12.152385 27840 raft_consensus.cc:3060] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:12.153399 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 28 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:12.153615 27664 leader_election.cc:290] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 27 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:12.077157 27395 raft_consensus.cc:493] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:12.154202 18371 raft_consensus.cc:2410] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for term 28 because replica has last-logged OpId of term: 26 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 18.
I20260709 12:25:12.148412 27671 raft_consensus.cc:740] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:25:12.154409 27395 raft_consensus.cc:515] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:12.154995 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 28 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:12.150496 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:12.155649 27671 consensus_queue.cc:260] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 25.18, Last appended by leader: 18, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:12.156569 18362 raft_consensus.cc:1240] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 29. Current term is 30. Ops: []
I20260709 12:25:12.155781 18371 raft_consensus.cc:2393] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 28: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:25:12.160475 18361 raft_consensus.cc:2410] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for term 28 because replica has last-logged OpId of term: 26 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 18.
I20260709 12:25:12.163071 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:25:12.171914 17802 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:12.172400 17802 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:12.172596 27449 raft_consensus.cc:493] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:12.173061 27449 raft_consensus.cc:515] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:12.173383 17802 leader_election.cc:304] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:25:12.174113 27782 raft_consensus.cc:2804] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:25:12.174593 27782 raft_consensus.cc:697] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 20 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:25:12.175490 27782 consensus_queue.cc:237] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 19.12, Last appended by leader: 12, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:12.177584 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 28 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:12.178416 17834 raft_consensus.cc:3060] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:12.177752 18334 leader_election.cc:304] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:12.180193 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 29 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:12.181051 17838 raft_consensus.cc:2468] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 28.
I20260709 12:25:12.182659 18368 raft_consensus.cc:1240] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 26. Current term is 27. Ops: []
I20260709 12:25:12.189935 17830 raft_consensus.cc:2468] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 26.
I20260709 12:25:12.138675 27617 raft_consensus.cc:493] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:12.191388 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 34 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:12.135147 27711 raft_consensus.cc:2764] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Leader pre-election decision vote started in defunct term 26: won
W20260709 12:25:12.200012 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 49252.8us
I20260709 12:25:12.200946 17652 leader_election.cc:304] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:25:12.201656 27590 raft_consensus.cc:2804] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:25:12.202114 27590 raft_consensus.cc:493] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:25:12.061544 27505 consensus_queue.cc:1059] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 23.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.103s
I20260709 12:25:12.133594 27369 raft_consensus.cc:515] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:12.218555 27828 raft_consensus.cc:515] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 34 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:12.221369 17762 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38786 (request call id 4187) took 1664 ms (client timeout 1976 ms). Trace:
I20260709 12:25:12.221952 27722 raft_consensus.cc:515] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 32 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:12.223870 17833 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35042 (request call id 3523) took 2354 ms. Trace:
I20260709 12:25:12.157883 27751 leader_election.cc:290] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 27 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:12.205399 18367 raft_consensus.cc:2468] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 27.
I20260709 12:25:12.156670 27395 leader_election.cc:290] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:25:12.221994 17762 rpcz_store.cc:269] 0709 12:25:10.557440 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:10.557648 (+   208us) service_pool.cc:224] Handling call
0709 12:25:12.221084 (+1663436us) spinlock_profiling.cc:243] Waited 998 ms on lock 0x7b54004155d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:25:12.221284 (+   200us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2294673280}
W20260709 12:25:12.204872 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 151427us
I20260709 12:25:12.199249 27831 raft_consensus.cc:2749] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 12:25:12.233345 17834 raft_consensus.cc:2468] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 28.
I20260709 12:25:12.251462 17682 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37402 (request call id 3951) took 2732 ms. Trace:
I20260709 12:25:12.241057 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:12.252022 17682 rpcz_store.cc:276] 0709 12:25:09.493863 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:09.494035 (+   172us) service_pool.cc:224] Handling call
0709 12:25:09.501531 (+  7496us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:10.389494 (+887963us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:12.225528 (+1836034us) spinlock_profiling.cc:243] Waited 885 ms on lock 0x7b54003467d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:25:12.225807 (+   279us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2036525952}
I20260709 12:25:12.252494 18367 raft_consensus.cc:2468] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 27.
I20260709 12:25:12.206777 18368 raft_consensus.cc:1275] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 25 index: 15. Preceding OpId from leader: term: 30 index: 18. (index mismatch)
I20260709 12:25:12.224426 17833 rpcz_store.cc:276] 0709 12:25:09.869387 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:10.119857 (+250470us) service_pool.cc:224] Handling call
0709 12:25:10.120170 (+   313us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:10.313373 (+193203us) raft_consensus.cc:1534] Early marking committed up to index 18
0709 12:25:10.748868 (+435495us) log.cc:844] Serialized 14 byte log entry
0709 12:25:11.441296 (+692428us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:11.441368 (+    72us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:25:11.441431 (+    63us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:12.222345 (+780914us) spinlock_profiling.cc:243] Waited 48.6 ms on lock 0x7b1800493000. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b475012a6 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419
0709 12:25:12.223495 (+  1150us) spinlock_profiling.cc:243] Waited 193 ms on lock 0x7b54001586d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:25:12.223558 (+    63us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:12.223758 (+   200us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":13480,"prepare.queue_time_us":731736,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":34,"raft.queue_time_us":13595,"raft.run_cpu_time_us":141,"raft.run_wall_time_us":14701,"spinlock_wait_cycles":555658368,"thread_start_us":1084402,"threads_started":2,"wal-append.queue_time_us":843081}
I20260709 12:25:12.202055 17652 leader_election.cc:304] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:12.193217 27720 raft_consensus.cc:3060] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:25:12.193684 27664 raft_consensus.cc:493] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:12.225200 27590 raft_consensus.cc:3060] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:12.271715 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 24 candidate_status { last_received { term: 23 index: 21 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:12.273316 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 27 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:12.281849 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:12.288331 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:12.289180 17838 raft_consensus.cc:2393] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 30: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:25:12.290414 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 28 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:12.300555 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 27 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:25:12.282557 17835 raft_consensus.cc:2468] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 21.
I20260709 12:25:12.302798 17837 raft_consensus.cc:2393] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 28: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
W20260709 12:25:12.303459 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 101470us
I20260709 12:25:12.308331 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:12.308221 17652 leader_election.cc:304] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:12.304797 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 33 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:12.311213 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:12.311908 17830 raft_consensus.cc:2468] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 25.
I20260709 12:25:12.311354 27111 raft_consensus.cc:515] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:12.315343 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 120208us
I20260709 12:25:12.315958 17838 raft_consensus.cc:1240] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 24. Current term is 25. Ops: []
I20260709 12:25:12.317203 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:12.204535 27617 raft_consensus.cc:515] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:12.320399 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:12.328938 17687 raft_consensus.cc:2468] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 27.
I20260709 12:25:12.321557 27197 consensus_queue.cc:1059] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 28.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.001s
I20260709 12:25:12.336660 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:25:12.322757 27766 consensus_queue.cc:1059] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 23.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20260709 12:25:12.325372 27823 consensus_queue.cc:1048] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.000s
I20260709 12:25:12.326974 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 31 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:12.346118 17833 raft_consensus.cc:2410] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for term 31 because replica has last-logged OpId of term: 30 index: 22, which is greater than that of the candidate, which has last-logged OpId of term: 26 index: 20.
I20260709 12:25:12.268934 27722 leader_election.cc:290] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 32 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:12.333747 17803 leader_election.cc:304] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:25:12.313333 27826 raft_consensus.cc:493] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:12.269969 27664 raft_consensus.cc:515] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:12.313702 27725 raft_consensus.cc:2804] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:25:12.338552 18334 leader_election.cc:304] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
W20260709 12:25:12.349248 27846 log.cc:927] Time spent T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.089s	user 0.000s	sys 0.000s
I20260709 12:25:12.302549 27824 raft_consensus.cc:493] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:12.317852 27505 raft_consensus.cc:493] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:12.328200 27570 raft_consensus.cc:2804] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:25:12.344878 17838 raft_consensus.cc:1240] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 25. Current term is 26. Ops: []
I20260709 12:25:12.353715 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:12.360618 27008 raft_consensus.cc:2804] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:25:12.331802 27666 consensus_queue.cc:1048] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.030s
I20260709 12:25:12.380177 27008 raft_consensus.cc:493] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:12.380645 27008 raft_consensus.cc:3060] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:12.383116 27761 leader_election.cc:290] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:12.362004 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 24 candidate_status { last_received { term: 23 index: 21 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:12.384167 17831 raft_consensus.cc:3060] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:25:12.385216 27850 log.cc:927] Time spent T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.055s	user 0.000s	sys 0.001s
W20260709 12:25:12.365633 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 43576.5us
I20260709 12:25:12.376951 27369 leader_election.cc:290] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:25:12.378372 27617 leader_election.cc:290] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:12.393227 27449 leader_election.cc:290] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:12.399123 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:12.357741 27725 raft_consensus.cc:493] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 32 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:25:12.400086 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 29 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:12.402245 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:25:12.383137 27845 log.cc:927] Time spent T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.070s	user 0.000s	sys 0.000s
I20260709 12:25:12.344000 27751 consensus_queue.cc:1059] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 25.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.026s
I20260709 12:25:12.403959 17834 raft_consensus.cc:2393] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 25: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
W20260709 12:25:12.395850 27680 consensus_meta.cc:287] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.554s	user 0.011s	sys 0.000s
I20260709 12:25:12.422343 17831 raft_consensus.cc:2468] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 24.
I20260709 12:25:12.426234 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:12.427227 17687 raft_consensus.cc:2468] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 27.
I20260709 12:25:12.364825 27853 raft_consensus.cc:2804] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:25:12.396066 27749 leader_election.cc:290] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:12.435580 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:25:12.397243 27570 raft_consensus.cc:697] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 29 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:25:12.359431 27826 raft_consensus.cc:515] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:12.439484 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 79433us
I20260709 12:25:12.406970 27725 raft_consensus.cc:3060] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:25:12.437739 18334 leader_election.cc:304] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:25:12.438982 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:12.444574 17654 leader_election.cc:304] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:25:12.443182 18365 raft_consensus.cc:2468] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 27.
I20260709 12:25:12.406636 27828 leader_election.cc:290] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 34 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:25:12.388859 27782 consensus_queue.cc:1059] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.067s
I20260709 12:25:12.389000 27505 raft_consensus.cc:515] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:12.441512 17803 leader_election.cc:304] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:12.448625 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 32 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:25:12.455562 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:12.456398 17831 raft_consensus.cc:3060] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:12.368172 27824 raft_consensus.cc:515] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:12.463531 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 27 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:12.464241 17834 raft_consensus.cc:3060] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:12.463070 17803 leader_election.cc:304] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:25:12.470875 17838 raft_consensus.cc:2468] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 29.
I20260709 12:25:12.470476 17803 leader_election.cc:304] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:25:12.473021 17803 leader_election.cc:304] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:25:12.431985 27680 raft_consensus.cc:515] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:12.479375 17803 leader_election.cc:304] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:25:12.485150 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 27 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:12.486881 17838 raft_consensus.cc:2393] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 27: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:25:12.490412 17831 raft_consensus.cc:2468] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 27.
I20260709 12:25:12.469485 27782 raft_consensus.cc:2804] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:25:12.441978 27757 consensus_queue.cc:1048] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.037s
W20260709 12:25:12.493352 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 48893.6us
I20260709 12:25:12.440858 27857 consensus_queue.cc:1048] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.045s
I20260709 12:25:12.492662 17834 raft_consensus.cc:2468] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 27.
I20260709 12:25:12.499289 17652 leader_election.cc:304] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:25:12.478024 27369 raft_consensus.cc:2804] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Leader pre-election won for term 28
W20260709 12:25:12.485109 27753 consensus_meta.cc:287] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.535s	user 0.007s	sys 0.004s
I20260709 12:25:12.479868 27197 raft_consensus.cc:2804] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:25:12.481668 27858 consensus_queue.cc:1048] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.024s
I20260709 12:25:12.480235 27666 raft_consensus.cc:2804] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:25:12.532728 27666 raft_consensus.cc:493] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:12.533281 27666 raft_consensus.cc:3060] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:12.463253 27766 consensus_queue.cc:1048] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.034s
I20260709 12:25:12.504698 18364 raft_consensus.cc:1240] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 25. Current term is 26. Ops: []
I20260709 12:25:12.455495 27856 consensus_queue.cc:1048] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.041s
W20260709 12:25:12.508738 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 64247.3us
I20260709 12:25:12.504040 27859 raft_consensus.cc:2749] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 12:25:12.484817 27431 raft_consensus.cc:2749] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 12:25:12.547029 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:12.551374 17838 raft_consensus.cc:2468] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 31.
W20260709 12:25:12.553187 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 115459us
I20260709 12:25:12.516076 27852 raft_consensus.cc:2804] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:25:12.569355 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:12.571532 17836 raft_consensus.cc:1240] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 26. Current term is 27. Ops: []
I20260709 12:25:12.573117 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:25:12.573783 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 70213.1us
I20260709 12:25:12.514576 27111 leader_election.cc:290] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 31 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:12.515718 27369 raft_consensus.cc:493] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:25:12.576249 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:12.581216 18365 raft_consensus.cc:2468] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 25.
I20260709 12:25:12.576337 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:25:12.582156 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 101722us
I20260709 12:25:12.582612 17830 raft_consensus.cc:2468] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 27.
I20260709 12:25:12.576776 17654 leader_election.cc:304] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:25:12.577152 18334 leader_election.cc:304] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:25:12.583820 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:12.585160 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:12.537163 27197 raft_consensus.cc:493] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:25:12.538992 27753 raft_consensus.cc:2749] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Leader pre-election lost for term 27. Reason: Vote denied by peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651) with higher term. Message: Invalid argument: T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 27. Current term is 29.
W20260709 12:25:12.598649 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:12.652074 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 28 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:12.592634 18371 raft_consensus.cc:2393] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 27: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:25:12.653473 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:12.600594 18365 raft_consensus.cc:1240] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 26. Current term is 27. Ops: [26.21-26.21]
I20260709 12:25:12.654359 18367 raft_consensus.cc:3060] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:12.567822 27858 raft_consensus.cc:2804] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:25:12.659198 17803 leader_election.cc:304] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:25:12.594193 27369 raft_consensus.cc:3060] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:12.666461 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:12.667097 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 26 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:12.568740 27853 raft_consensus.cc:697] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 25 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:25:12.656770 27197 raft_consensus.cc:3060] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:25:12.719295 17684 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44898 (request call id 1958) took 3814 ms (client timeout 1770 ms). Trace:
I20260709 12:25:12.685144 18367 raft_consensus.cc:2468] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 31.
I20260709 12:25:12.580192 27860 raft_consensus.cc:493] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:25:12.700552 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 29 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:12.702117 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:12.703245 27449 raft_consensus.cc:3055] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 32 LEADER]: Stepping down as leader of term 32
I20260709 12:25:12.703313 27839 raft_consensus.cc:3055] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:25:12.703940 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 28 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:12.704160 27738 raft_consensus.cc:493] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:25:12.576506 27852 raft_consensus.cc:493] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:12.748733 27744 raft_consensus.cc:493] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:25:12.756064 27852 raft_consensus.cc:3060] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:12.756425 27744 raft_consensus.cc:515] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:12.758536 27744 leader_election.cc:290] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:12.720417 27858 raft_consensus.cc:697] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 26 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:25:12.761338 27824 leader_election.cc:290] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:12.762037 27860 raft_consensus.cc:515] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:12.762538 27739 raft_consensus.cc:493] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:25:12.763120 27739 raft_consensus.cc:515] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:12.764784 27860 leader_election.cc:290] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:12.765862 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 30 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:12.749634 27839 raft_consensus.cc:740] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:25:12.715809 27865 raft_consensus.cc:493] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:12.767405 27839 consensus_queue.cc:260] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 24.17, Last appended by leader: 17, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:12.768515 27839 raft_consensus.cc:3060] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:12.768311 27865 raft_consensus.cc:515] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:12.770423 27865 leader_election.cc:290] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:12.770640 17728 leader_election.cc:304] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:25:12.771207 27738 raft_consensus.cc:3060] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:12.707427 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 30 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:12.773072 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 26 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:12.773775 17681 raft_consensus.cc:2468] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 25.
I20260709 12:25:12.774206 27525 raft_consensus.cc:493] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:25:12.774711 27525 raft_consensus.cc:515] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:12.775508 17681 raft_consensus.cc:1240] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 32 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 31. Current term is 32. Ops: []
I20260709 12:25:12.715905 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 29 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:12.776571 27640 raft_consensus.cc:2804] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:25:12.776988 27640 raft_consensus.cc:493] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:12.765215 27739 leader_election.cc:290] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:12.696576 27762 raft_consensus.cc:2804] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Leader election won for term 27
W20260709 12:25:12.719763 17684 rpcz_store.cc:269] 0709 12:25:08.898359 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:08.898640 (+   281us) service_pool.cc:224] Handling call
0709 12:25:12.711761 (+3813121us) spinlock_profiling.cc:243] Waited 3.1 s on lock 0x7b54002526d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:25:12.711901 (+   140us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":7120339328}
I20260709 12:25:12.777230 27858 consensus_queue.cc:237] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 25.18, Last appended by leader: 18, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:12.795634 17835 raft_consensus.cc:1275] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 24 index: 18. Preceding OpId from leader: term: 25 index: 18. (term mismatch)
I20260709 12:25:12.796093 17835 pending_rounds.cc:85] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b: Aborting all ops after (but not including) 17
I20260709 12:25:12.796361 17835 pending_rounds.cc:107] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b: Aborting uncommitted NO_OP operation due to leader change: 24.18
I20260709 12:25:12.796705 17835 raft_consensus.cc:2889] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:25:12.722617 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 63524us
I20260709 12:25:12.723568 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 30 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:12.724061 27395 raft_consensus.cc:3055] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:25:12.725239 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 32 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:12.804888 18361 raft_consensus.cc:3060] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:25:12.726293 27853 consensus_queue.cc:237] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 23.16, Last appended by leader: 16, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:12.726974 27752 leader_election.cc:290] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 31 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:12.728258 18366 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 2865) took 2159 ms. Trace:
I20260709 12:25:12.734740 27590 raft_consensus.cc:515] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:12.723277 17729 leader_election.cc:304] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:12.735807 27008 raft_consensus.cc:515] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:12.650965 27761 raft_consensus.cc:2804] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:25:12.738585 18363 raft_consensus.cc:1275] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 23 index: 16. Preceding OpId from leader: term: 25 index: 17. (index mismatch)
I20260709 12:25:12.738631 18368 raft_consensus.cc:1240] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 28. Current term is 29. Ops: []
I20260709 12:25:12.744961 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 32 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:12.593952 27782 raft_consensus.cc:697] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 30 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:25:12.751469 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:12.752086 17686 raft_consensus.cc:2393] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 28: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:25:12.754038 27716 raft_consensus.cc:493] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:25:12.749480 27826 leader_election.cc:290] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:12.708885 17764 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38786 (request call id 4193) took 1520 ms. Trace:
I20260709 12:25:12.712958 27823 raft_consensus.cc:3055] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:25:12.779196 17728 leader_election.cc:304] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 31 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:12.706956 17688 raft_consensus.cc:1240] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 23. Current term is 24. Ops: []
I20260709 12:25:12.861260 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 30 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:12.778252 27338 leader_election.cc:290] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:25:12.782649 17727 leader_election.cc:304] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
W20260709 12:25:12.788970 17655 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:12.789623 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 34 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:12.862828 17761 raft_consensus.cc:1275] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 26 index: 16. Preceding OpId from leader: term: 28 index: 17. (index mismatch)
I20260709 12:25:12.895289 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 28 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:12.897297 17684 raft_consensus.cc:1275] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 19 index: 12. Preceding OpId from leader: term: 20 index: 13. (index mismatch)
I20260709 12:25:12.790405 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:12.719317 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 34 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:25:12.898581 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 29 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:12.899510 17830 raft_consensus.cc:1240] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 24. Current term is 25. Ops: [24.17-24.17]
I20260709 12:25:12.799963 18371 raft_consensus.cc:3060] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:12.800988 18364 raft_consensus.cc:3055] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:25:12.900588 18364 raft_consensus.cc:740] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
W20260709 12:25:12.891203 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 112066us
I20260709 12:25:12.901409 17652 leader_election.cc:304] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
W20260709 12:25:12.778416 27767 consensus_meta.cc:287] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.741s	user 0.007s	sys 0.003s
I20260709 12:25:12.650678 27749 consensus_queue.cc:1059] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 26.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.158s
I20260709 12:25:12.777328 27640 raft_consensus.cc:3060] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:12.804199 17802 leader_election.cc:304] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:25:12.746780 27449 raft_consensus.cc:740] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 32 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:25:12.811892 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:12.854961 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 34 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:12.909507 17835 raft_consensus.cc:1240] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 30. Current term is 31. Ops: [30.18-30.18]
I20260709 12:25:12.570658 27570 consensus_queue.cc:237] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 25.15, Last appended by leader: 15, Current term: 29, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:12.656975 27840 raft_consensus.cc:515] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Starting leader election with config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:12.856092 17763 raft_consensus.cc:1240] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 30 LEADER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 29. Current term is 30. Ops: []
I20260709 12:25:12.850611 17755 raft_consensus.cc:1240] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 23. Current term is 24. Ops: []
I20260709 12:25:12.852586 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:12.840860 17803 leader_election.cc:304] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
W20260709 12:25:12.920225 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 79468.3us
I20260709 12:25:12.852648 27761 raft_consensus.cc:493] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:12.850893 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:12.928774 17757 raft_consensus.cc:2393] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 24: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:25:12.852888 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 29 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:12.805121 17834 raft_consensus.cc:1275] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 22 index: 15. Preceding OpId from leader: term: 25 index: 16. (index mismatch)
I20260709 12:25:12.901706 17728 leader_election.cc:304] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:25:12.902711 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 32 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:12.902901 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:25:12.903313 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 120755us
I20260709 12:25:12.931588 18361 raft_consensus.cc:2468] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 32.
I20260709 12:25:12.833295 27431 raft_consensus.cc:2804] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:25:12.861033 27823 raft_consensus.cc:740] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 29 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
W20260709 12:25:12.906562 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 329850us
I20260709 12:25:12.940407 17654 leader_election.cc:304] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:25:12.854717 17764 rpcz_store.cc:276] 0709 12:25:11.188378 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:11.190047 (+  1669us) service_pool.cc:224] Handling call
0709 12:25:11.228440 (+ 38393us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:11.228707 (+   267us) raft_consensus.cc:1534] Early marking committed up to index 22
0709 12:25:11.257414 (+ 28707us) log.cc:844] Serialized 14 byte log entry
0709 12:25:11.744516 (+487102us) log.cc:844] Serialized 14 byte log entry
0709 12:25:12.538869 (+794353us) spinlock_profiling.cc:243] Waited 2.26 ms on lock 0x7b18003e6380. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b475012a6 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b475333f3 00007f7b475357c1
0709 12:25:12.701058 (+162189us) spinlock_profiling.cc:243] Waited 3.83 ms on lock 0x7b18003e6380. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b475333f3 00007f7b475357c1 00007f7b47535779 00007f7b47535709 00007f7b475356d1 00007f7b475349cd 0000000000388634 0000000000388479 00007f7b4751683a
0709 12:25:12.708399 (+  7341us) spinlock_profiling.cc:243] Waited 1.89 ms on lock 0x7b18003e6380. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b474adc0a 00007f7b50bed219 00007f7b47535a40 00007f7b475304e4 00007f7b47530455 00007f7b4a618060 00007f7b4a62a716 00007f7b4a63b287 00007f7b4a63b208
0709 12:25:12.708531 (+   132us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:12.708576 (+    45us) raft_consensus.cc:1645] Marking committed up to 22
0709 12:25:12.708633 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:12.708708 (+    75us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:12.708829 (+   121us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":987756,"prepare.queue_time_us":40236,"prepare.run_cpu_time_us":85,"prepare.run_wall_time_us":87,"raft.queue_time_us":981756,"raft.run_cpu_time_us":251,"raft.run_wall_time_us":29039,"spinlock_wait_cycles":18356864,"thread_start_us":486463,"threads_started":1,"wal-append.queue_time_us":682522}
I20260709 12:25:12.839352 27859 consensus_queue.cc:1059] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 24.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.081s
I20260709 12:25:12.909669 17760 raft_consensus.cc:3060] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:25:12.843071 17838 raft_consensus.cc:3060] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Advancing to term 32
W20260709 12:25:12.913134 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 109017us
I20260709 12:25:12.942728 17652 leader_election.cc:304] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:25:12.907955 27446 raft_consensus.cc:2749] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260709 12:25:12.839834 27875 raft_consensus.cc:493] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:12.977474 27875 raft_consensus.cc:515] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:12.978085 27525 leader_election.cc:290] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:12.937659 27431 raft_consensus.cc:493] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:12.978724 27431 raft_consensus.cc:3060] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:12.979023 17687 raft_consensus.cc:2468] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 32.
I20260709 12:25:12.916339 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:12.803205 27723 consensus_queue.cc:1059] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 19.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.100s
I20260709 12:25:12.855230 27744 raft_consensus.cc:2804] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:25:12.838348 18366 rpcz_store.cc:276] 0709 12:25:10.568643 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:10.568824 (+   181us) service_pool.cc:224] Handling call
0709 12:25:10.573275 (+  4451us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:11.061876 (+488601us) raft_consensus.cc:1534] Early marking committed up to index 17
0709 12:25:11.075280 (+ 13404us) log.cc:844] Serialized 14 byte log entry
0709 12:25:11.498813 (+423533us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:11.498873 (+    60us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:25:11.498917 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:11.960300 (+461383us) spinlock_profiling.cc:243] Waited 2.08 ms on lock 0x7b18003bb7e0. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b475012a6 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419
0709 12:25:12.727995 (+767695us) spinlock_profiling.cc:243] Waited 488 ms on lock 0x7b54008020d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:25:12.728055 (+    60us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:12.728179 (+   124us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":186770,"prepare.run_cpu_time_us":30,"prepare.run_wall_time_us":31,"raft.queue_time_us":222,"raft.run_cpu_time_us":149,"raft.run_wall_time_us":35142,"spinlock_wait_cycles":1127772928,"thread_start_us":423082,"threads_started":1,"wal-append.queue_time_us":701098,"wal-append.run_cpu_time_us":1496,"wal-append.run_wall_time_us":1053952}
I20260709 12:25:12.921061 17803 leader_election.cc:304] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:25:12.842568 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 119371us
I20260709 12:25:12.780133 27873 consensus_queue.cc:1059] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 28.20, Next index: 21, Last known committed idx: 19, Time since last communication: 0.111s
I20260709 12:25:12.834478 27703 raft_consensus.cc:493] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:12.839617 27762 raft_consensus.cc:697] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 27 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:25:12.936358 17759 raft_consensus.cc:2410] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 29 because replica has last-logged OpId of term: 26 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 15.
I20260709 12:25:12.938576 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 29 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:12.933089 17727 leader_election.cc:304] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:12.908149 27739 raft_consensus.cc:2749] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Leader election lost for term 31. Reason: could not achieve majority
I20260709 12:25:12.901355 18364 consensus_queue.cc:260] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 24.16, Last appended by leader: 16, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:12.803557 27395 raft_consensus.cc:740] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:25:12.914458 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:25:12.891606 27716 raft_consensus.cc:515] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:12.922534 27505 leader_election.cc:290] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:12.976023 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 28 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:12.911410 27864 raft_consensus.cc:2804] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Leader pre-election won for term 32
W20260709 12:25:12.920580 27720 consensus_meta.cc:287] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.657s	user 0.007s	sys 0.004s
I20260709 12:25:12.971273 17802 leader_election.cc:304] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:25:12.933820 27680 leader_election.cc:290] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 26 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:13.060704 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:13.078925 18364 raft_consensus.cc:3060] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:13.060462 18371 raft_consensus.cc:2468] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 30.
I20260709 12:25:13.079258 17686 raft_consensus.cc:2410] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 32 because replica has last-logged OpId of term: 30 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 29 index: 17.
I20260709 12:25:12.935515 27761 raft_consensus.cc:3060] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:13.083225 17729 leader_election.cc:304] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
W20260709 12:25:13.098739 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 124764us
W20260709 12:25:13.088356 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:13.088913 17764 raft_consensus.cc:2468] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 27.
W20260709 12:25:13.090679 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 157644us
I20260709 12:25:13.106281 17680 raft_consensus.cc:2410] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 32 because replica has last-logged OpId of term: 31 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 30 index: 17.
I20260709 12:25:12.994814 27878 raft_consensus.cc:2804] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:25:13.111249 17727 leader_election.cc:304] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:12.992141 27871 raft_consensus.cc:2749] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:25:12.975132 27865 raft_consensus.cc:2804] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Leader pre-election won for term 28
W20260709 12:25:13.155299 17801 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.572s	user 0.008s	sys 0.015s
I20260709 12:25:13.107239 27704 raft_consensus.cc:2804] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 32 FOLLOWER]: Leader election won for term 32
I20260709 12:25:13.128026 17758 raft_consensus.cc:2468] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 28.
I20260709 12:25:13.145418 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:13.163556 17764 raft_consensus.cc:2468] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 31.
I20260709 12:25:12.979820 27875 leader_election.cc:290] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:25:13.158121 17831 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.612s	user 0.009s	sys 0.015s
W20260709 12:25:13.158732 17801 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.575s	user 0.009s	sys 0.015s
W20260709 12:25:13.164783 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 170804us
W20260709 12:25:13.165022 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 581881us
W20260709 12:25:13.164755 17831 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.619s	user 0.009s	sys 0.015s
I20260709 12:25:13.166445 18364 raft_consensus.cc:1275] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 24 index: 16. Preceding OpId from leader: term: 25 index: 16. (term mismatch)
I20260709 12:25:13.142305 17838 raft_consensus.cc:2468] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 32.
I20260709 12:25:13.167073 18364 pending_rounds.cc:85] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da: Aborting all ops after (but not including) 15
I20260709 12:25:13.167824 18364 pending_rounds.cc:107] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da: Aborting uncommitted NO_OP operation due to leader change: 24.16
I20260709 12:25:13.168243 18364 raft_consensus.cc:2889] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:13.014956 27752 raft_consensus.cc:2749] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:25:13.101193 27864 raft_consensus.cc:493] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:13.110493 27744 raft_consensus.cc:697] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:25:13.144109 27878 raft_consensus.cc:493] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:12.971273 27832 consensus_queue.cc:1059] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.215s
I20260709 12:25:12.986334 27738 raft_consensus.cc:515] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:13.144883 27577 raft_consensus.cc:3060] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:25:13.198146 17755 raft_consensus.cc:1275] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 22 index: 12. Preceding OpId from leader: term: 27 index: 14. (index mismatch)
I20260709 12:25:13.150934 27196 consensus_queue.cc:1059] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 26.19, Next index: 20, Last known committed idx: 18, Time since last communication: 0.042s
I20260709 12:25:13.153919 27865 raft_consensus.cc:493] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:13.146476 27857 raft_consensus.cc:493] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:25:13.102129 27703 raft_consensus.cc:515] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:13.158516 27809 raft_consensus.cc:2804] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:25:13.217589 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 30 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:13.218410 17755 raft_consensus.cc:2468] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 29.
I20260709 12:25:13.195011 17801 leader_election.cc:304] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:13.185765 27877 raft_consensus.cc:2749] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:25:13.229310 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:13.230322 17688 raft_consensus.cc:2468] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 24.
I20260709 12:25:13.187973 27864 raft_consensus.cc:3060] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:25:13.224346 17801 leader_election.cc:304] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:25:12.984865 27823 consensus_queue.cc:260] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 29.20, Last appended by leader: 20, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:13.234769 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 31 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:13.235204 17760 raft_consensus.cc:2468] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 34.
I20260709 12:25:13.151316 27617 consensus_queue.cc:1059] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 29.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.063s
I20260709 12:25:13.242304 17801 leader_election.cc:304] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:13.231767 17681 raft_consensus.cc:1240] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 28. Current term is 29. Ops: []
I20260709 12:25:13.250469 17681 raft_consensus.cc:1240] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 30. Current term is 31. Ops: []
I20260709 12:25:13.197803 27704 raft_consensus.cc:697] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 32 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:25:13.251271 17756 raft_consensus.cc:3060] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:25:13.250536 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 26 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:13.198060 27878 raft_consensus.cc:3060] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:13.246804 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 29 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:13.107549 27449 consensus_queue.cc:260] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 32.22, Last appended by leader: 22, Current term: 32, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:13.216032 27865 raft_consensus.cc:3060] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:13.115260 27782 consensus_queue.cc:237] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 28.17, Last appended by leader: 17, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:13.254417 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 34 candidate_status { last_received { term: 33 index: 25 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:25:13.265002 17684 raft_consensus.cc:1240] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 26. Current term is 27. Ops: []
I20260709 12:25:13.221935 27809 raft_consensus.cc:493] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:13.351326 27809 raft_consensus.cc:3060] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:13.234866 27857 raft_consensus.cc:515] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:13.353595 27857 leader_election.cc:290] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:13.269173 27884 raft_consensus.cc:2749] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:25:13.358172 27884 raft_consensus.cc:2749] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:25:13.358783 27884 raft_consensus.cc:2804] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:25:13.359215 27884 raft_consensus.cc:493] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 33 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:25:13.359583 27884 raft_consensus.cc:3060] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 33 FOLLOWER]: Advancing to term 34
W20260709 12:25:13.360311 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 439294us
I20260709 12:25:13.188843 27395 consensus_queue.cc:260] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 23.21, Last appended by leader: 21, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:13.245340 27800 raft_consensus.cc:3055] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:25:13.271900 18334 leader_election.cc:304] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:25:13.249361 27752 raft_consensus.cc:3055] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 29 LEADER]: Stepping down as leader of term 29
W20260709 12:25:13.346026 18333 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.16.29.66:39651 timed out after 1.530s (SENT) [suppressed 19 similar messages]
I20260709 12:25:13.219821 27680 raft_consensus.cc:493] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:25:13.266461 27757 raft_consensus.cc:3055] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:25:13.256485 27008 leader_election.cc:290] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 26 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:25:13.361963 17803 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:13.362268 17756 raft_consensus.cc:2468] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 32.
I20260709 12:25:13.266053 17755 raft_consensus.cc:2468] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 28.
W20260709 12:25:13.367094 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 395885us
W20260709 12:25:13.379098 17655 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:13.380580 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 108781us
I20260709 12:25:13.345746 27823 raft_consensus.cc:3060] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:13.409065 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:13.344133 27590 leader_election.cc:290] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 27 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:25:13.378760 27876 log.cc:927] Time spent T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.111s	user 0.000s	sys 0.002s
I20260709 12:25:13.381037 27752 raft_consensus.cc:740] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 29 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:25:13.408723 27338 raft_consensus.cc:493] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:25:13.374096 27800 raft_consensus.cc:740] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 29 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:25:13.409425 27859 raft_consensus.cc:2749] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:25:13.244747 27762 consensus_queue.cc:237] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 24.20, Last appended by leader: 20, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:13.347821 27720 consensus_queue.cc:1048] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 15, Time since last communication: 0.076s
W20260709 12:25:13.429009 27666 consensus_meta.cc:287] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.895s	user 0.006s	sys 0.006s
I20260709 12:25:13.412918 27753 raft_consensus.cc:2804] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:25:13.416787 17802 leader_election.cc:304] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:13.421562 27449 raft_consensus.cc:3060] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:25:13.479084 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 30 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:13.480268 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
W20260709 12:25:13.482275 17680 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37402 (request call id 3971) took 1317 ms (client timeout 1618 ms). Trace:
W20260709 12:25:13.482577 27852 consensus_meta.cc:287] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.726s	user 0.009s	sys 0.000s
W20260709 12:25:13.482684 17680 rpcz_store.cc:269] 0709 12:25:12.164804 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:12.164973 (+   169us) service_pool.cc:224] Handling call
0709 12:25:12.701930 (+536957us) spinlock_profiling.cc:243] Waited 8.88 ms on lock 0x7b5800002a98. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b474fbb71 00007f7b474fba22 00007f7b504148d9 00007f7b4f56eee0 00007f7b4f6380ab 00007f7b4f638131 00007f7b4f5e60a0 00007f7b4f5e5984 00007f7b48b606d4 00007f7b48b60662
0709 12:25:13.482082 (+780152us) spinlock_profiling.cc:243] Waited 357 ms on lock 0x7b5400206554. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:25:13.482221 (+   139us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":841816192}
I20260709 12:25:13.444106 27338 raft_consensus.cc:515] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:13.486214 27338 leader_election.cc:290] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:13.486307 27525 raft_consensus.cc:3055] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:25:13.456477 27840 leader_election.cc:290] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 28 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:13.486785 27525 raft_consensus.cc:740] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:25:13.458564 27703 leader_election.cc:290] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:13.422907 18334 leader_election.cc:304] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:25:13.418313 27680 raft_consensus.cc:515] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:13.488240 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 65402.3us
I20260709 12:25:13.466372 27716 leader_election.cc:290] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:25:13.451680 27885 log.cc:927] Time spent T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.092s	user 0.000s	sys 0.000s
I20260709 12:25:13.490885 17687 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44898 (request call id 1978) took 1305 ms. Trace:
W20260709 12:25:13.491029 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:13.491532 17652 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:13.491274 17687 rpcz_store.cc:276] 0709 12:25:12.185857 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:12.448382 (+262525us) service_pool.cc:224] Handling call
0709 12:25:13.490721 (+1042339us) spinlock_profiling.cc:243] Waited 460 ms on lock 0x7b5400425ed4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:25:13.490829 (+   108us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1057199872}
I20260709 12:25:13.492748 27738 leader_election.cc:290] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 21 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:13.493335 27738 raft_consensus.cc:2804] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:25:13.493764 27738 raft_consensus.cc:493] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:25:13.372648 27744 consensus_queue.cc:237] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 24.17, Last appended by leader: 17, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:13.494117 27738 raft_consensus.cc:3060] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:13.495978 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:13.496479 18364 raft_consensus.cc:1240] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 23. Current term is 24. Ops: []
I20260709 12:25:13.476784 27800 consensus_queue.cc:260] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 29.20, Last appended by leader: 20, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:13.477674 27666 raft_consensus.cc:515] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:13.483016 27852 raft_consensus.cc:515] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:13.409499 27872 consensus_queue.cc:1059] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 24.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.055s
W20260709 12:25:13.516871 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 100247us
I20260709 12:25:13.521366 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:25:13.521914 17834 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.528s	user 0.003s	sys 0.009s
W20260709 12:25:13.463770 27369 consensus_meta.cc:287] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.682s	user 0.006s	sys 0.000s
I20260709 12:25:13.497289 17652 leader_election.cc:304] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:13.470687 27752 consensus_queue.cc:260] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 29.18, Last appended by leader: 18, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:13.457664 27577 raft_consensus.cc:2749] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 32 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:25:13.498721 18364 raft_consensus.cc:1240] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 34 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 33. Current term is 34. Ops: []
I20260709 12:25:13.499706 17838 raft_consensus.cc:1240] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 26. Current term is 27. Ops: []
I20260709 12:25:13.501403 18363 raft_consensus.cc:1240] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 26. Current term is 27. Ops: []
I20260709 12:25:13.536732 17652 leader_election.cc:304] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:13.502709 18365 raft_consensus.cc:1240] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 29. Current term is 30. Ops: []
I20260709 12:25:13.505290 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 26 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:13.505016 27338 consensus_queue.cc:1059] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 28.18, Next index: 19, Last known committed idx: 17, Time since last communication: 0.003s
I20260709 12:25:13.508064 18366 raft_consensus.cc:3060] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:25:13.524180 27431 consensus_meta.cc:287] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.545s	user 0.005s	sys 0.005s
I20260709 12:25:13.444574 27877 consensus_queue.cc:1059] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.071s
I20260709 12:25:13.515509 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
W20260709 12:25:13.515700 27197 consensus_meta.cc:287] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.788s	user 0.009s	sys 0.001s
I20260709 12:25:13.489652 27680 leader_election.cc:290] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:25:13.522239 17834 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.529s	user 0.003s	sys 0.009s
I20260709 12:25:13.433389 27008 consensus_queue.cc:1048] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.070s
I20260709 12:25:13.518543 17802 leader_election.cc:304] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:25:13.493196 27864 raft_consensus.cc:515] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:13.498450 27753 raft_consensus.cc:493] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:13.537173 17833 raft_consensus.cc:2393] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 30: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:25:13.523332 27859 raft_consensus.cc:2749] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:25:13.413386 27757 raft_consensus.cc:740] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:25:13.507823 27800 raft_consensus.cc:3060] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:13.541676 17652 leader_election.cc:304] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:25:13.616884 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 75311.9us
W20260709 12:25:13.537969 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 192741us
I20260709 12:25:13.547489 18374 raft_consensus.cc:1240] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 27. Current term is 28. Ops: []
I20260709 12:25:13.547973 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:13.621186 18364 raft_consensus.cc:3060] T a5732e71b9d54f8786b8c511d69943bf P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Advancing to term 28
W20260709 12:25:13.622082 17730 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:13.623411 27769 consensus_queue.cc:1048] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 12:25:13.572129 27752 raft_consensus.cc:3060] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:13.623899 17652 leader_election.cc:304] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:25:13.625340 17652 leader_election.cc:304] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:25:13.625568 27769 raft_consensus.cc:493] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:13.626075 27769 raft_consensus.cc:515] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:13.626549 17652 leader_election.cc:304] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:13.627553 27880 raft_consensus.cc:2804] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:25:13.628014 27880 raft_consensus.cc:493] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:13.628116 27766 raft_consensus.cc:2804] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:25:13.628468 27880 raft_consensus.cc:3060] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:13.628695 27766 raft_consensus.cc:493] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:25:13.629256 27766 raft_consensus.cc:3060] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:13.629390 17830 raft_consensus.cc:1240] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 30. Current term is 31. Ops: [30.18-30.18]
I20260709 12:25:13.572474 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:13.632102 27759 raft_consensus.cc:2749] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:25:13.633379 18363 raft_consensus.cc:1240] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 26. Current term is 27. Ops: []
I20260709 12:25:13.635039 27769 leader_election.cc:290] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:13.572788 18365 raft_consensus.cc:1275] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 29 index: 19. Preceding OpId from leader: term: 30 index: 20. (index mismatch)
I20260709 12:25:13.574086 27767 raft_consensus.cc:3055] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:25:13.641872 27767 raft_consensus.cc:740] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:25:13.643289 27767 consensus_queue.cc:260] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 26.20, Last appended by leader: 20, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:13.574940 17829 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35040 (request call id 3745) took 1028 ms. Trace:
I20260709 12:25:13.588081 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:13.487676 27525 consensus_queue.cc:260] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 19.12, Last appended by leader: 12, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:13.644821 17829 rpcz_store.cc:276] 0709 12:25:12.546493 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:12.546606 (+   113us) service_pool.cc:224] Handling call
0709 12:25:12.570874 (+ 24268us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:12.792996 (+222122us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:12.793040 (+    44us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:12.850369 (+ 57329us) log.cc:844] Serialized 26 byte log entry
0709 12:25:13.378284 (+527915us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:25:13.378327 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:13.378392 (+    65us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:13.574711 (+196319us) raft_consensus.cc:1685] finished
0709 12:25:13.574731 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:13.574883 (+   152us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":165552,"spinlock_wait_cycles":27006336,"thread_start_us":666196,"threads_started":2,"wal-append.queue_time_us":617800}
I20260709 12:25:13.646096 27525 raft_consensus.cc:3060] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:13.575085 27794 consensus_queue.cc:1059] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 26.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
I20260709 12:25:13.532878 27638 leader_election.cc:290] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:25:13.540421 27749 consensus_queue.cc:1048] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.056s
I20260709 12:25:13.532109 27369 raft_consensus.cc:515] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:13.497417 27704 consensus_queue.cc:237] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 30.17, Last appended by leader: 17, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:25:13.597695 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 79255.2us
W20260709 12:25:13.628923 17652 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:13.536921 27590 raft_consensus.cc:2749] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
W20260709 12:25:13.590613 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 507497us
I20260709 12:25:13.652374 17729 leader_election.cc:304] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:13.547479 27196 raft_consensus.cc:2749] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:25:13.596325 27753 raft_consensus.cc:3060] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:13.670459 17801 leader_election.cc:304] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:13.620913 27895 raft_consensus.cc:3055] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:25:13.595571 27680 raft_consensus.cc:3055] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:25:13.662721 17652 leader_election.cc:304] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:13.547433 27857 consensus_queue.cc:1059] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 33.25, Next index: 26, Last known committed idx: 24, Time since last communication: 0.076s
I20260709 12:25:13.587253 27431 raft_consensus.cc:515] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:13.663277 17729 leader_election.cc:304] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:13.590831 27338 consensus_queue.cc:1059] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 30.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.070s
I20260709 12:25:13.669982 17802 leader_election.cc:304] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:25:13.674685 27769 consensus_queue.cc:1059] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 29.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.018s
I20260709 12:25:13.689833 27446 consensus_queue.cc:1048] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.001s
W20260709 12:25:13.691177 27890 log.cc:927] Time spent T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.193s	user 0.000s	sys 0.001s
W20260709 12:25:13.695842 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 754999us
I20260709 12:25:13.698207 18368 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59902 (request call id 2352) took 1112 ms. Trace:
W20260709 12:25:13.700172 27854 log.cc:927] Time spent T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 1.306s	user 0.005s	sys 0.000s
I20260709 12:25:13.703754 27666 leader_election.cc:290] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 30 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:13.706631 27753 raft_consensus.cc:515] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:13.709108 27809 raft_consensus.cc:515] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:13.710376 27894 raft_consensus.cc:2749] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:25:13.591990 27197 raft_consensus.cc:515] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:13.615135 27877 raft_consensus.cc:2749] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:25:13.680195 17801 leader_election.cc:304] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:13.614861 27892 consensus_queue.cc:1059] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.077s
I20260709 12:25:13.761989 17833 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35042 (request call id 3545) took 1017 ms. Trace:
I20260709 12:25:13.733373 18364 raft_consensus.cc:2468] T a5732e71b9d54f8786b8c511d69943bf P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 28.
I20260709 12:25:13.762733 18374 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59886 (request call id 2689) took 1041 ms. Trace:
I20260709 12:25:13.620784 27617 raft_consensus.cc:2749] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
W20260709 12:25:13.749574 27761 consensus_meta.cc:287] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.622s	user 0.008s	sys 0.001s
W20260709 12:25:13.841023 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 171137us
I20260709 12:25:13.839803 18368 rpcz_store.cc:276] 0709 12:25:12.585540 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:12.585675 (+   135us) service_pool.cc:224] Handling call
0709 12:25:12.586021 (+   346us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:12.852178 (+266157us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:13.697905 (+845727us) spinlock_profiling.cc:243] Waited 137 ms on lock 0x7b54005460d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:25:13.698136 (+   231us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":316096256}
I20260709 12:25:13.907472 17685 raft_consensus.cc:1240] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 26. Current term is 27. Ops: []
I20260709 12:25:13.846720 17728 leader_election.cc:304] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:25:13.879746 18366 raft_consensus.cc:1275] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 22 index: 12. Preceding OpId from leader: term: 27 index: 14. (index mismatch)
I20260709 12:25:13.664667 27755 raft_consensus.cc:2749] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260709 12:25:13.746542 27857 raft_consensus.cc:2749] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260709 12:25:13.746436 27196 raft_consensus.cc:3055] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 33 LEADER]: Stepping down as leader of term 33
I20260709 12:25:13.757135 27369 leader_election.cc:290] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 28 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:13.889716 18374 rpcz_store.cc:276] 0709 12:25:12.721332 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:12.722901 (+  1569us) service_pool.cc:224] Handling call
0709 12:25:12.723169 (+   268us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:13.620940 (+897771us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:13.762518 (+141578us) spinlock_profiling.cc:243] Waited 814 ms on lock 0x7b54006946d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:25:13.762677 (+   159us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1873160448}
W20260709 12:25:13.746476 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 83827us
W20260709 12:25:13.887100 17688 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.531s	user 0.004s	sys 0.036s
W20260709 12:25:13.930850 17688 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.575s	user 0.004s	sys 0.037s
W20260709 12:25:13.888389 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 225181us
I20260709 12:25:13.724972 27895 raft_consensus.cc:740] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 28 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
W20260709 12:25:13.888046 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 207956us
W20260709 12:25:13.757916 27889 log.cc:927] Time spent T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.289s	user 0.001s	sys 0.001s
W20260709 12:25:13.779299 27878 consensus_meta.cc:287] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.515s	user 0.005s	sys 0.002s
I20260709 12:25:13.714001 27680 raft_consensus.cc:740] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
W20260709 12:25:13.729066 27725 consensus_meta.cc:287] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 1.275s	user 0.011s	sys 0.000s
I20260709 12:25:13.903709 18367 raft_consensus.cc:2393] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 26: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
W20260709 12:25:13.917384 17654 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:13.915194 27761 raft_consensus.cc:515] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:13.773777 27431 leader_election.cc:290] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 26 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:13.865880 27111 raft_consensus.cc:2749] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
W20260709 12:25:13.931950 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 442527us
I20260709 12:25:13.770922 27664 leader_election.cc:290] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:25:13.937698 17831 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35040 (request call id 3739) took 1393 ms. Trace:
I20260709 12:25:13.933369 17729 leader_election.cc:304] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:25:13.758244 27852 leader_election.cc:290] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 28 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:25:13.664944 27891 consensus_queue.cc:1059] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 23.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.255s
I20260709 12:25:13.899684 27873 raft_consensus.cc:2804] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 34 FOLLOWER]: Leader election won for term 34
I20260709 12:25:13.960668 27725 raft_consensus.cc:515] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:13.963967 27873 raft_consensus.cc:697] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 34 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:25:13.963701 27895 consensus_queue.cc:260] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 28.18, Last appended by leader: 18, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:13.965075 17689 raft_consensus.cc:3060] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:25:13.965013 27873 consensus_queue.cc:237] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 32.19, Last appended by leader: 19, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:13.966284 27725 leader_election.cc:290] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 33 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:13.891263 17833 rpcz_store.cc:276] 0709 12:25:12.744353 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:12.744461 (+   108us) service_pool.cc:224] Handling call
0709 12:25:13.761811 (+1017350us) spinlock_profiling.cc:243] Waited 768 ms on lock 0x7b5400158bd4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:25:13.761932 (+   121us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1767526144}
I20260709 12:25:13.665254 27759 consensus_queue.cc:1059] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 26.19, Next index: 20, Last known committed idx: 18, Time since last communication: 0.238s
I20260709 12:25:13.840332 27865 raft_consensus.cc:515] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:13.904801 27752 raft_consensus.cc:2804] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Leader election won for term 28
W20260709 12:25:13.962677 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 116049us
I20260709 12:25:13.900266 27716 consensus_queue.cc:1048] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.062s
I20260709 12:25:13.939956 27883 raft_consensus.cc:2749] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:25:13.962399 17831 rpcz_store.cc:276] 0709 12:25:12.544635 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:12.544798 (+   163us) service_pool.cc:224] Handling call
0709 12:25:12.545349 (+   551us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:13.166254 (+620905us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:13.166285 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:13.166989 (+   704us) log.cc:844] Serialized 26 byte log entry
0709 12:25:13.526528 (+359539us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:25:13.936997 (+410469us) log.cc:844] Serialized 14 byte log entry
0709 12:25:13.937306 (+   309us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:13.937424 (+   118us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:13.937490 (+    66us) raft_consensus.cc:1685] finished
0709 12:25:13.937509 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:13.937644 (+   135us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":430282,"raft.queue_time_us":180664,"raft.run_cpu_time_us":75,"raft.run_wall_time_us":76,"spinlock_wait_cycles":32284032,"thread_start_us":1233181,"threads_started":3,"wal-append.queue_time_us":512791}
I20260709 12:25:13.970791 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 33 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:13.941876 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 33 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:13.993589 17834 raft_consensus.cc:2468] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 32.
I20260709 12:25:13.974771 17729 leader_election.cc:304] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:25:13.934885 27196 raft_consensus.cc:740] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 33 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:25:13.764245 27880 raft_consensus.cc:515] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:13.957039 27901 raft_consensus.cc:3055] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 31 LEADER]: Stepping down as leader of term 31
I20260709 12:25:13.950379 27666 consensus_queue.cc:1048] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.032s
I20260709 12:25:13.981065 17728 leader_election.cc:304] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:13.962734 18334 leader_election.cc:304] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:25:13.935190 27738 raft_consensus.cc:515] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:13.983227 17689 raft_consensus.cc:2468] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 34.
I20260709 12:25:14.001610 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:14.003139 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:13.976292 27755 raft_consensus.cc:2804] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Leader election won for term 24
W20260709 12:25:13.989902 17763 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.500s	user 0.017s	sys 0.027s
I20260709 12:25:14.016081 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:13.956449 27878 raft_consensus.cc:515] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:14.032455 27111 raft_consensus.cc:493] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:14.033572 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 31 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:14.033474 27111 raft_consensus.cc:515] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:14.036482 27111 leader_election.cc:290] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:13.988732 27752 raft_consensus.cc:697] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:25:14.043401 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 26 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:13.997114 27901 raft_consensus.cc:740] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 31 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:25:14.007494 27570 raft_consensus.cc:493] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:14.016808 27446 raft_consensus.cc:2749] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:25:14.007454 27703 raft_consensus.cc:2804] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:25:14.048823 17763 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38776 (request call id 3374) took 1230 ms. Trace:
I20260709 12:25:14.070554 17763 rpcz_store.cc:276] 0709 12:25:12.818309 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:12.937474 (+119165us) service_pool.cc:224] Handling call
0709 12:25:12.993973 (+ 56499us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:12.994309 (+   336us) raft_consensus.cc:1534] Early marking committed up to index 19
0709 12:25:13.118030 (+123721us) log.cc:844] Serialized 14 byte log entry
0709 12:25:13.489756 (+371726us) spinlock_profiling.cc:243] Waited 22.6 ms on lock 0x7b1800498640. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b47515e07 00007f7b503709a9 00007f7b4753269b 00007f7b4753093f 00007f7b47532c02 00007f7b4a1ba20b 00007f7b4a1bfd23 00007f7b4a1c023a 00007f7b4a62aa2a
0709 12:25:14.048353 (+558597us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:14.048433 (+    80us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:25:14.048497 (+    64us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:14.048597 (+   100us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:14.048755 (+   158us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":50005,"prepare.queue_time_us":113975,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":47,"raft.queue_time_us":43802,"raft.run_cpu_time_us":276,"raft.run_wall_time_us":44651,"spinlock_wait_cycles":51891456,"thread_start_us":516411,"threads_started":1}
I20260709 12:25:14.019460 27852 raft_consensus.cc:2804] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 32 FOLLOWER]: Leader election won for term 32
I20260709 12:25:14.078032 27852 raft_consensus.cc:697] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 32 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:25:14.078999 27878 leader_election.cc:290] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 28 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:13.975590 27828 consensus_queue.cc:1059] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 26.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.049s
I20260709 12:25:14.079219 27852 consensus_queue.cc:237] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 31.18, Last appended by leader: 18, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:14.080857 27828 raft_consensus.cc:3055] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:25:14.081307 27828 raft_consensus.cc:740] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:25:14.082332 27828 consensus_queue.cc:260] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 26.16, Last appended by leader: 16, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:14.083739 27828 raft_consensus.cc:3060] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:14.088532 27901 consensus_queue.cc:260] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 31.18, Last appended by leader: 18, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:14.091296 27880 leader_election.cc:290] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 30 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:14.093657 27738 leader_election.cc:290] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 25 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:25:14.094686 27865 leader_election.cc:290] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 28 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:25:14.095477 27884 consensus_meta.cc:287] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.714s	user 0.007s	sys 0.002s
I20260709 12:25:14.054157 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
W20260709 12:25:14.098146 17764 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.601s	user 0.015s	sys 0.007s
W20260709 12:25:14.098563 17764 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.602s	user 0.015s	sys 0.007s
I20260709 12:25:14.064507 17801 leader_election.cc:304] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:14.048468 17832 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35040 (request call id 3747) took 1156 ms. Trace:
W20260709 12:25:14.054014 27839 consensus_meta.cc:287] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 1.285s	user 0.008s	sys 0.004s
W20260709 12:25:14.057331 27640 consensus_meta.cc:287] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 1.119s	user 0.008s	sys 0.004s
I20260709 12:25:14.009512 27859 consensus_queue.cc:1059] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 23.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.048s
I20260709 12:25:14.057934 27902 raft_consensus.cc:2804] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:25:14.103497 17801 leader_election.cc:304] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:14.103084 17832 rpcz_store.cc:276] 0709 12:25:12.892206 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:12.893765 (+  1559us) service_pool.cc:224] Handling call
0709 12:25:12.919173 (+ 25408us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:25:12.919507 (+   334us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:12.919536 (+    29us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:25:12.935137 (+ 15601us) log.cc:844] Serialized 52 byte log entry
0709 12:25:13.215660 (+280523us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:25:13.215700 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:13.215763 (+    63us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:13.757110 (+541347us) spinlock_profiling.cc:243] Waited 15.8 ms on lock 0x7b4400022b4c. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b47459c85 00007f7b4745a6a2 00007f7b4a1cc1db 00007f7b4a1c7aaf 00007f7b4a1bb86e 00007f7b4a1babb6 00007f7b4a1ba7d8 00007f7b4a1c4221 00007f7b4a1c41d9 00007f7b4a1c4169
0709 12:25:14.048069 (+290959us) raft_consensus.cc:1685] finished
0709 12:25:14.048100 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:14.048394 (+   294us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":180516,"raft.run_cpu_time_us":193,"raft.run_wall_time_us":38010,"spinlock_wait_cycles":36227968,"thread_start_us":236882,"threads_started":1,"wal-append.queue_time_us":496617}
W20260709 12:25:14.110435 17760 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.537s	user 0.002s	sys 0.028s
W20260709 12:25:14.122272 17760 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.549s	user 0.003s	sys 0.028s
W20260709 12:25:14.112169 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 171349us
I20260709 12:25:14.060346 27755 raft_consensus.cc:697] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 24 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:25:14.003717 27680 consensus_queue.cc:260] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 28.18, Last appended by leader: 18, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:14.128031 27680 raft_consensus.cc:3060] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:14.128381 27755 consensus_queue.cc:237] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 23.15, Last appended by leader: 15, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:14.047338 27525 consensus_queue.cc:1059] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.054s
W20260709 12:25:14.123440 17801 leader_election.cc:336] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 26 election: RPC error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Timed out: RequestConsensusVote RPC to 127.16.29.66:39651 timed out after 1.674s (SENT)
I20260709 12:25:14.175385 17801 leader_election.cc:304] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:14.098917 27703 raft_consensus.cc:697] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 31 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:25:14.060067 27577 consensus_queue.cc:1048] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.058s
W20260709 12:25:14.132105 17652 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:14.108803 27111 raft_consensus.cc:493] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:14.162366 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 245724us
I20260709 12:25:14.069049 27570 raft_consensus.cc:515] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:14.108678 27782 raft_consensus.cc:2749] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
W20260709 12:25:14.178771 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 55360.8us
I20260709 12:25:14.183094 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 32 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:14.185690 17652 leader_election.cc:304] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:25:14.185781 17802 leader_election.cc:304] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:14.185863 17763 raft_consensus.cc:2468] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 31.
I20260709 12:25:14.173462 27891 raft_consensus.cc:2749] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:25:14.122941 27197 leader_election.cc:290] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 27 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:14.119469 27640 raft_consensus.cc:515] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:14.194730 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:25:14.195729 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 169080us
I20260709 12:25:14.124745 27753 leader_election.cc:290] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 26 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:25:14.099630 27884 raft_consensus.cc:515] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 34 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:14.069519 27196 consensus_queue.cc:260] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 33.25, Last appended by leader: 25, Current term: 33, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:14.177297 27877 raft_consensus.cc:493] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:25:14.177047 27809 leader_election.cc:290] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 29 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:14.198465 17729 leader_election.cc:304] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:14.182837 27369 raft_consensus.cc:2749] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:25:14.182651 27865 raft_consensus.cc:493] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:14.201565 27446 consensus_queue.cc:1048] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.005s
I20260709 12:25:14.204078 17727 leader_election.cc:304] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:14.210005 17652 leader_election.cc:304] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:14.228850 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:14.228901 17761 raft_consensus.cc:2393] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 29: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:25:14.226007 27877 raft_consensus.cc:515] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:14.232409 27570 leader_election.cc:290] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:14.233944 27877 leader_election.cc:290] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:14.238210 17835 raft_consensus.cc:1240] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 34 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 33. Current term is 34. Ops: []
I20260709 12:25:14.239921 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 30 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:14.239699 27443 raft_consensus.cc:2804] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:25:14.241686 17763 raft_consensus.cc:2393] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 30: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:25:14.242839 17727 leader_election.cc:304] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:14.197739 27922 raft_consensus.cc:2749] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 32 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:25:14.195818 27916 raft_consensus.cc:2804] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:25:14.187166 27111 raft_consensus.cc:515] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:14.298877 27918 log.cc:927] Time spent T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.089s	user 0.000s	sys 0.001s
W20260709 12:25:14.298494 27925 log.cc:927] Time spent T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.061s	user 0.000s	sys 0.000s
W20260709 12:25:14.298444 27910 log.cc:927] Time spent T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.088s	user 0.000s	sys 0.000s
I20260709 12:25:14.319929 27111 leader_election.cc:290] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:14.237102 27865 raft_consensus.cc:515] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:14.322484 27800 consensus_meta.cc:287] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.674s	user 0.010s	sys 0.000s
I20260709 12:25:14.322796 27865 leader_election.cc:290] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:14.202816 27722 raft_consensus.cc:3055] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:25:14.262635 17763 raft_consensus.cc:1240] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 28. Current term is 29. Ops: [28.17-28.17]
I20260709 12:25:14.301635 17755 raft_consensus.cc:1275] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 19 index: 12. Preceding OpId from leader: term: 20 index: 13. (index mismatch)
I20260709 12:25:14.242324 27196 raft_consensus.cc:3060] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 33 FOLLOWER]: Advancing to term 34
W20260709 12:25:14.247251 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 48841.6us
I20260709 12:25:14.324481 27863 raft_consensus.cc:3055] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:25:14.338665 27863 raft_consensus.cc:740] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:25:14.339742 27863 consensus_queue.cc:260] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 25.18, Last appended by leader: 18, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:14.340898 27863 raft_consensus.cc:3060] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:14.230866 27761 leader_election.cc:290] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 28 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:25:14.323778 27766 consensus_meta.cc:287] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.694s	user 0.008s	sys 0.000s
I20260709 12:25:14.248370 27753 raft_consensus.cc:493] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:25:14.200172 27752 consensus_queue.cc:237] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 26.13, Last appended by leader: 13, Current term: 28, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:14.337965 17729 leader_election.cc:304] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:25:14.325016 27640 leader_election.cc:290] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 29 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:14.296509 27926 raft_consensus.cc:2749] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260709 12:25:14.241873 27443 raft_consensus.cc:493] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:14.248986 27446 raft_consensus.cc:2749] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:25:14.282856 17760 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38776 (request call id 3385) took 1128 ms. Trace:
I20260709 12:25:14.334954 27722 raft_consensus.cc:740] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:25:14.353340 17755 raft_consensus.cc:1240] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 28. Current term is 29. Ops: []
I20260709 12:25:14.274235 27809 raft_consensus.cc:2804] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 32 FOLLOWER]: Leader election won for term 32
I20260709 12:25:14.273211 27703 consensus_queue.cc:237] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 30.20, Last appended by leader: 20, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:14.366549 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:14.366724 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 29 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:14.366915 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 30 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:14.316829 27916 raft_consensus.cc:493] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:25:14.290673 27692 consensus_queue.cc:1048] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.062s
I20260709 12:25:14.374034 17763 raft_consensus.cc:1240] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 28. Current term is 29. Ops: [28.18-28.18]
W20260709 12:25:14.313620 17652 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:14.364002 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:25:14.379987 18334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:14.372347 17836 raft_consensus.cc:1275] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 28 index: 19. Preceding OpId from leader: term: 30 index: 20. (index mismatch)
W20260709 12:25:14.376672 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:14.379678 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 33 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:14.385782 17755 raft_consensus.cc:3060] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:25:14.389241 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 26 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:14.390060 17683 raft_consensus.cc:2410] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 26 because replica has last-logged OpId of term: 24 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 16.
I20260709 12:25:14.362236 27666 raft_consensus.cc:493] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:25:14.395462 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 34 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:25:14.372347 27879 raft_consensus.cc:493] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:25:14.351756 27766 raft_consensus.cc:515] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:14.362452 27929 raft_consensus.cc:2804] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:25:14.362375 27443 raft_consensus.cc:3060] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:25:14.380668 27916 raft_consensus.cc:3060] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:14.381177 27809 raft_consensus.cc:697] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 32 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:25:14.403223 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 29 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:14.413926 17680 raft_consensus.cc:1240] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 28. Current term is 29. Ops: []
I20260709 12:25:14.401088 27446 raft_consensus.cc:493] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:14.416304 27446 raft_consensus.cc:515] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:14.416602 27809 consensus_queue.cc:237] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 30.22, Last appended by leader: 22, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:14.419143 27446 leader_election.cc:290] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:14.401885 27666 raft_consensus.cc:515] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:14.355350 27753 raft_consensus.cc:515] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:14.404171 27879 raft_consensus.cc:515] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:14.427392 27888 log.cc:927] Time spent T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.975s	user 0.001s	sys 0.002s
I20260709 12:25:14.429172 17832 raft_consensus.cc:1275] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 24 index: 16. Preceding OpId from leader: term: 25 index: 17. (index mismatch)
I20260709 12:25:14.389598 27744 consensus_queue.cc:1048] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.040s
I20260709 12:25:14.424644 27929 raft_consensus.cc:697] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 30 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:25:14.430938 17832 raft_consensus.cc:1240] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 25. Current term is 26. Ops: []
I20260709 12:25:14.429260 27879 leader_election.cc:290] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:14.431366 27929 consensus_queue.cc:237] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 28.20, Last appended by leader: 20, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:14.434181 27395 raft_consensus.cc:3060] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:14.435223 17832 raft_consensus.cc:1240] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 25. Current term is 26. Ops: [25.18-25.18]
I20260709 12:25:14.398265 27884 leader_election.cc:290] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 34 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:25:14.439419 17832 raft_consensus.cc:1240] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 25. Current term is 26. Ops: []
I20260709 12:25:14.388989 27722 consensus_queue.cc:260] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 25.17, Last appended by leader: 17, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:14.448679 27722 raft_consensus.cc:3060] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Advancing to term 26
W20260709 12:25:14.456254 18334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:14.459183 17757 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38776 (request call id 3377) took 1640 ms. Trace:
I20260709 12:25:14.459410 27753 leader_election.cc:290] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:14.459692 17757 rpcz_store.cc:276] 0709 12:25:12.819138 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:12.940367 (+121229us) service_pool.cc:224] Handling call
0709 12:25:12.974912 (+ 34545us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:12.975219 (+   307us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:12.975247 (+    28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:12.977860 (+  2613us) log.cc:844] Serialized 26 byte log entry
0709 12:25:13.209600 (+231740us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:25:13.209643 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:13.209712 (+    69us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:14.426780 (+1217068us) spinlock_profiling.cc:243] Waited 31 ms on lock 0x7b4400022b4c. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b47459c85 00007f7b4745a6a2 00007f7b4a1cc1db 00007f7b4a1c7aaf 00007f7b4a1bb86e 00007f7b4a1babb6 00007f7b4a1ba7d8 00007f7b4a1c4221 00007f7b4a1c41d9 00007f7b4a1c4169
0709 12:25:14.458867 (+ 32087us) raft_consensus.cc:1685] finished
0709 12:25:14.458890 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:14.459099 (+   209us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":3455,"raft.run_cpu_time_us":78,"raft.run_wall_time_us":79,"spinlock_wait_cycles":71375616,"thread_start_us":231314,"threads_started":1,"wal-append.queue_time_us":456380}
W20260709 12:25:14.461256 27935 log.cc:927] Time spent T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.061s	user 0.000s	sys 0.001s
I20260709 12:25:14.461849 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:25:14.466018 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:14.467362 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 27 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:14.471963 27666 leader_election.cc:290] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:14.473131 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 34 candidate_status { last_received { term: 33 index: 25 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:14.480386 17755 raft_consensus.cc:2410] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for term 33 because replica has last-logged OpId of term: 32 index: 22, which is greater than that of the candidate, which has last-logged OpId of term: 31 index: 21.
I20260709 12:25:14.485198 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 29 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:14.486743 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:14.488057 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:14.484608 27724 raft_consensus.cc:493] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:14.489503 27724 raft_consensus.cc:515] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:14.492309 17832 raft_consensus.cc:1240] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 27. Current term is 28. Ops: []
I20260709 12:25:14.492923 27902 raft_consensus.cc:493] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:25:14.493407 27902 raft_consensus.cc:3060] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Advancing to term 32
W20260709 12:25:14.493678 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:14.495416 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:14.495761 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:14.496578 27832 consensus_queue.cc:1059] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:25:14.497380 27832 consensus_queue.cc:1243] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer 334a4e1f59944544b3b3ff36616b57da log is divergent from this leader: its last log entry 26.19 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 17
I20260709 12:25:14.498471 27832 raft_consensus.cc:3055] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:25:14.499392 27832 raft_consensus.cc:740] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 27 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:25:14.500269 27832 consensus_queue.cc:260] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 27.19, Last appended by leader: 19, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:14.501140 17686 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37402 (request call id 3982) took 1608 ms (client timeout 1762 ms). Trace:
W20260709 12:25:14.507293 17686 rpcz_store.cc:269] 0709 12:25:12.892941 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:12.896592 (+  3651us) service_pool.cc:224] Handling call
0709 12:25:14.500937 (+1604345us) spinlock_profiling.cc:243] Waited 148 ms on lock 0x7b54000c5654. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:25:14.501082 (+   145us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":339961600}
I20260709 12:25:14.508526 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:14.509124 17757 raft_consensus.cc:2468] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 27.
I20260709 12:25:14.509903 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 32 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:14.513224 27832 raft_consensus.cc:3060] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:14.518913 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 29 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:25:14.519989 17728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:14.521445 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:14.524065 18364 raft_consensus.cc:1240] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 24. Current term is 25. Ops: [24.18-24.18]
I20260709 12:25:14.525162 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:14.525961 18364 raft_consensus.cc:2410] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for term 29 because replica has last-logged OpId of term: 28 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 27 index: 17.
I20260709 12:25:14.526189 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 25 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:14.520972 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:14.528419 17686 raft_consensus.cc:1240] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 27 LEADER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 26. Current term is 27. Ops: [26.21-26.21]
I20260709 12:25:14.506466 27902 raft_consensus.cc:515] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 32 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:14.531806 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 32 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:14.533785 17681 raft_consensus.cc:2393] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 32: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:25:14.534080 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:14.534940 18366 raft_consensus.cc:2393] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 30: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:25:14.532315 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 28 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:25:14.539505 17654 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:14.542629 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 34 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:25:14.520733 27666 raft_consensus.cc:493] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:25:14.552141 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:14.548811 18366 raft_consensus.cc:1240] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 20. Current term is 21. Ops: []
I20260709 12:25:14.552970 17654 leader_election.cc:304] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:14.536815 27443 raft_consensus.cc:515] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:14.555506 27443 leader_election.cc:290] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 33 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:14.556281 17757 raft_consensus.cc:2468] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 32.
I20260709 12:25:14.557497 17654 leader_election.cc:304] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:14.557945 17681 raft_consensus.cc:2393] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 27: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:25:14.559204 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:14.559792 17755 raft_consensus.cc:3060] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:14.556026 27934 raft_consensus.cc:493] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:14.562369 27934 raft_consensus.cc:515] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:14.563055 27857 raft_consensus.cc:2749] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:25:14.564318 27934 leader_election.cc:290] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:14.545557 27884 consensus_queue.cc:1059] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 24.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.022s
I20260709 12:25:14.568277 27880 consensus_queue.cc:1048] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
I20260709 12:25:14.558598 27666 raft_consensus.cc:515] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:14.568981 17761 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38776 (request call id 3378) took 1717 ms (client timeout 2047 ms). Trace:
W20260709 12:25:14.569370 17761 rpcz_store.cc:269] 0709 12:25:12.852456 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:12.913638 (+ 61182us) service_pool.cc:224] Handling call
0709 12:25:14.568787 (+1655149us) spinlock_profiling.cc:243] Waited 1.14 s on lock 0x7b54003eb554. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:25:14.568924 (+   137us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2632234240}
I20260709 12:25:14.558710 27759 raft_consensus.cc:2749] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 12:25:14.536695 27577 consensus_queue.cc:1059] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.024s
I20260709 12:25:14.570623 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 26 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:14.571013 27577 raft_consensus.cc:3055] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:25:14.571375 27577 raft_consensus.cc:740] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:25:14.556284 27878 raft_consensus.cc:493] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:14.572898 27878 raft_consensus.cc:515] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:14.571839 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:14.574142 17761 raft_consensus.cc:2410] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for term 30 because replica has last-logged OpId of term: 29 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 28 index: 17.
I20260709 12:25:14.572296 27577 consensus_queue.cc:260] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 25.18, Last appended by leader: 18, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:14.574883 27577 raft_consensus.cc:3060] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Advancing to term 26
W20260709 12:25:14.577142 17727 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:14.546096 27713 raft_consensus.cc:493] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:14.536278 27722 consensus_queue.cc:1048] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.025s
I20260709 12:25:14.584756 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 28 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:14.582083 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 29 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:14.579830 27713 raft_consensus.cc:515] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:14.584424 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:14.538916 27446 consensus_queue.cc:1059] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 23.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.037s
I20260709 12:25:14.587613 18366 raft_consensus.cc:1240] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 30. Current term is 31. Ops: [27.16-30.18]
I20260709 12:25:14.589470 27713 leader_election.cc:290] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:14.594391 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:25:14.590480 17680 raft_consensus.cc:1240] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 26 LEADER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 25. Current term is 26. Ops: []
W20260709 12:25:14.595196 18333 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 1.214s	user 0.012s	sys 0.044s
I20260709 12:25:14.595124 17682 raft_consensus.cc:3060] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:14.574887 27878 leader_election.cc:290] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:14.551209 27755 consensus_queue.cc:1059] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 27.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.036s
I20260709 12:25:14.585212 27884 consensus_queue.cc:1243] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer 0da169ff82674dc1947055ce43de68a3 log is divergent from this leader: its last log entry 27.21 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 20
I20260709 12:25:14.605792 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:14.606022 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:25:14.606796 17680 raft_consensus.cc:3060] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:14.608350 17838 raft_consensus.cc:1240] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 23. Current term is 24. Ops: []
I20260709 12:25:14.610185 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:14.610647 17838 raft_consensus.cc:3060] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:14.611608 27666 leader_election.cc:290] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:14.611675 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 33 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:14.591687 27196 consensus_queue.cc:1048] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.010s
I20260709 12:25:14.614331 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:14.614840 18366 raft_consensus.cc:3060] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:14.616783 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:14.617604 18374 raft_consensus.cc:2393] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 28: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:25:14.618842 17761 raft_consensus.cc:1240] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 31 LEADER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 30. Current term is 31. Ops: []
I20260709 12:25:14.619761 17755 raft_consensus.cc:2468] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 25.
I20260709 12:25:14.612308 27875 raft_consensus.cc:3055] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:25:14.624670 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 30 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:14.625532 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:14.627435 18374 raft_consensus.cc:3060] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:25:14.628504 18333 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 1.247s	user 0.012s	sys 0.045s
W20260709 12:25:14.628880 27942 log.cc:927] Time spent T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.067s	user 0.000s	sys 0.000s
W20260709 12:25:14.629025 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 1.28294e+06us
I20260709 12:25:14.632673 18366 raft_consensus.cc:2468] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 28.
I20260709 12:25:14.632776 27864 leader_election.cc:290] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 32 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:14.634152 17835 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35040 (request call id 3757) took 1133 ms. Trace:
I20260709 12:25:14.634433 17835 rpcz_store.cc:276] 0709 12:25:13.500374 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:13.500605 (+   231us) service_pool.cc:224] Handling call
0709 12:25:13.508536 (+  7931us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:14.238637 (+730101us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:14.633977 (+395340us) spinlock_profiling.cc:243] Waited 729 ms on lock 0x7b5400335ed4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:25:14.634108 (+   131us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1677638400}
I20260709 12:25:14.604542 27916 raft_consensus.cc:515] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:25:14.626324 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:14.615715 27929 raft_consensus.cc:3055] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:25:14.649609 27929 raft_consensus.cc:740] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:25:14.650297 27902 leader_election.cc:290] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 32 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:14.649855 27757 consensus_queue.cc:260] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 26.19, Last appended by leader: 19, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:14.651046 27757 raft_consensus.cc:3060] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:14.650545 27929 consensus_queue.cc:260] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 25.18, Last appended by leader: 18, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:14.651973 27929 raft_consensus.cc:3060] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:14.639711 17832 raft_consensus.cc:1275] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 25 index: 18. Preceding OpId from leader: term: 26 index: 19. (index mismatch)
I20260709 12:25:14.644454 17801 leader_election.cc:304] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:25:14.654477 27873 raft_consensus.cc:2804] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:25:14.654909 27873 raft_consensus.cc:493] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:14.655591 27873 raft_consensus.cc:3060] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:25:14.656378 27881 raft_consensus.cc:3055] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:25:14.656893 27881 raft_consensus.cc:740] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 30 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:25:14.657724 27881 consensus_queue.cc:260] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 30.20, Last appended by leader: 20, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:14.658677 27881 raft_consensus.cc:3060] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:14.645920 17680 raft_consensus.cc:2468] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 25.
I20260709 12:25:14.661145 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 29 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:25:14.662209 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 29 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:14.663130 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:14.625898 17838 raft_consensus.cc:2468] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 28.
I20260709 12:25:14.665403 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 29 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:14.666208 17683 raft_consensus.cc:2393] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 29: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:25:14.666972 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:14.668162 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:25:14.668291 27874 consensus_queue.cc:1048] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:25:14.669085 17683 raft_consensus.cc:2393] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 28: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:25:14.669723 17759 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38776 (request call id 3379) took 1776 ms. Trace:
I20260709 12:25:14.669859 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 26 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:25:14.670399 17683 raft_consensus.cc:3060] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:14.665402 17680 raft_consensus.cc:2393] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 32: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
W20260709 12:25:14.671790 17730 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:14.675129 17832 raft_consensus.cc:1275] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 32 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 30 index: 17. Preceding OpId from leader: term: 32 index: 18. (index mismatch)
I20260709 12:25:14.670083 17759 rpcz_store.cc:276] 0709 12:25:12.893003 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:13.104033 (+211030us) service_pool.cc:224] Handling call
0709 12:25:13.140727 (+ 36694us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:25:13.522588 (+381861us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:13.522619 (+    31us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:25:13.532568 (+  9949us) log.cc:844] Serialized 52 byte log entry
0709 12:25:14.021178 (+488610us) raft_consensus.cc:1645] Marking committed up to 24
0709 12:25:14.057266 (+ 36088us) log.cc:844] Serialized 14 byte log entry
0709 12:25:14.177801 (+120535us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:14.669348 (+491547us) spinlock_profiling.cc:243] Waited 66.2 ms on lock 0x7b54003f2854. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:25:14.669397 (+    49us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:14.669442 (+    45us) raft_consensus.cc:1685] finished
0709 12:25:14.669543 (+   101us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:14.669672 (+   129us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":18761,"prepare.queue_time_us":32372,"prepare.run_cpu_time_us":71,"prepare.run_wall_time_us":5522,"raft.queue_time_us":81461,"raft.run_cpu_time_us":238,"raft.run_wall_time_us":14300,"spinlock_wait_cycles":152173440,"thread_start_us":843167,"threads_started":3,"wal-append.queue_time_us":629056}
I20260709 12:25:14.685263 18366 raft_consensus.cc:1275] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 24 index: 20. Preceding OpId from leader: term: 27 index: 21. (index mismatch)
I20260709 12:25:14.687211 17761 raft_consensus.cc:1240] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 28. Current term is 29. Ops: []
I20260709 12:25:14.687425 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:14.688050 17680 raft_consensus.cc:2468] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 30.
I20260709 12:25:14.688733 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:25:14.689239 17680 raft_consensus.cc:2393] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 29: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:25:14.690728 17829 raft_consensus.cc:1275] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 25 index: 15. Preceding OpId from leader: term: 29 index: 16. (index mismatch)
I20260709 12:25:14.692654 17652 leader_election.cc:304] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:25:14.696007 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 25 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:14.626673 27875 raft_consensus.cc:740] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 28 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:25:14.701653 17680 raft_consensus.cc:2393] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 25: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:25:14.702596 17683 raft_consensus.cc:2468] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 26.
I20260709 12:25:14.704461 17684 raft_consensus.cc:1240] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 28. Current term is 29. Ops: []
I20260709 12:25:14.706820 17759 raft_consensus.cc:1240] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 26. Current term is 27. Ops: [26.19-26.19]
I20260709 12:25:14.708117 17683 raft_consensus.cc:1240] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 28. Current term is 29. Ops: []
I20260709 12:25:14.708581 17680 raft_consensus.cc:1240] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 30. Current term is 31. Ops: []
I20260709 12:25:14.709901 17680 raft_consensus.cc:1240] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 24. Current term is 25. Ops: []
I20260709 12:25:14.708602 18374 raft_consensus.cc:2468] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 27.
I20260709 12:25:14.713430 17761 raft_consensus.cc:1240] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 25 LEADER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 24. Current term is 25. Ops: []
I20260709 12:25:14.718884 17683 raft_consensus.cc:1275] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 23 index: 16. Preceding OpId from leader: term: 25 index: 17. (index mismatch)
I20260709 12:25:14.668556 27891 consensus_queue.cc:1059] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 28.20, Next index: 21, Last known committed idx: 19, Time since last communication: 0.040s
I20260709 12:25:14.727998 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 30 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
W20260709 12:25:14.725162 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 95420.8us
I20260709 12:25:14.725284 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 32 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:14.701313 27929 consensus_queue.cc:1048] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.025s
I20260709 12:25:14.705497 27570 raft_consensus.cc:2804] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:25:14.731297 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:14.731807 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:14.734903 17829 raft_consensus.cc:3060] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:14.739248 18374 raft_consensus.cc:1275] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 25 index: 15. Preceding OpId from leader: term: 29 index: 16. (index mismatch)
I20260709 12:25:14.695587 27823 consensus_queue.cc:1059] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 33.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.033s
I20260709 12:25:14.701222 27864 consensus_queue.cc:1048] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.032s
I20260709 12:25:14.745858 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:14.741802 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:14.738431 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 33 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:14.748034 17761 raft_consensus.cc:2368] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 34 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for earlier term 33. Current term is 34.
I20260709 12:25:14.744218 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:14.752261 17728 leader_election.cc:304] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:14.753665 17728 leader_election.cc:304] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:14.754967 17728 leader_election.cc:304] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:14.758009 17834 raft_consensus.cc:2393] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 26: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:25:14.720868 27769 consensus_queue.cc:1059] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 26.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.025s
I20260709 12:25:14.764655 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 26 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:14.761068 18334 leader_election.cc:304] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:25:14.766242 18334 leader_election.cc:304] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:25:14.767613 17728 leader_election.cc:304] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:14.776453 27338 raft_consensus.cc:2804] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:25:14.777356 27875 consensus_queue.cc:260] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 28.20, Last appended by leader: 20, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:14.778712 27875 raft_consensus.cc:3060] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:14.781741 27809 raft_consensus.cc:2804] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:25:14.782253 27809 raft_consensus.cc:493] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:25:14.782711 27809 raft_consensus.cc:3060] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:14.769482 27703 raft_consensus.cc:2749] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:25:14.743184 27570 raft_consensus.cc:697] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 28 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:25:14.795148 27853 consensus_queue.cc:1048] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:25:14.764194 27716 raft_consensus.cc:2749] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260709 12:25:14.770282 27525 raft_consensus.cc:3060] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:14.797250 17688 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44898 (request call id 1991) took 1568 ms. Trace:
I20260709 12:25:14.797605 27916 leader_election.cc:290] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 29 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:14.797668 17688 rpcz_store.cc:276] 0709 12:25:13.228610 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:13.247349 (+ 18739us) service_pool.cc:224] Handling call
0709 12:25:13.247714 (+   365us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:13.355853 (+108139us) raft_consensus.cc:1534] Early marking committed up to index 19
0709 12:25:13.932215 (+576362us) log.cc:844] Serialized 14 byte log entry
0709 12:25:14.110811 (+178596us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:14.110879 (+    68us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:25:14.110939 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:14.796913 (+685974us) spinlock_profiling.cc:243] Waited 108 ms on lock 0x7b54000c4754. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:25:14.796958 (+    45us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:14.797191 (+   233us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":26579,"prepare.queue_time_us":628080,"prepare.run_cpu_time_us":66,"prepare.run_wall_time_us":6578,"raft.queue_time_us":4812,"raft.run_cpu_time_us":138,"raft.run_wall_time_us":4898,"spinlock_wait_cycles":248130944,"thread_start_us":753046,"threads_started":2,"wal-append.queue_time_us":338219}
I20260709 12:25:14.360370 17760 rpcz_store.cc:276] 0709 12:25:13.154845 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:13.252952 (+ 98107us) service_pool.cc:224] Handling call
0709 12:25:13.253307 (+   355us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:13.253539 (+   232us) raft_consensus.cc:1534] Early marking committed up to index 20
0709 12:25:13.344871 (+ 91332us) log.cc:844] Serialized 14 byte log entry
0709 12:25:13.573315 (+228444us) spinlock_profiling.cc:243] Waited 3.16 ms on lock 0x7b1800156a00. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b47515e07 00007f7b503709a9 00007f7b4753269b 00007f7b4753093f 00007f7b47532c02 00007f7b4a1ba20b 00007f7b4a1bfd23 00007f7b4a1c023a 00007f7b4a62aa2a
0709 12:25:14.269741 (+696426us) log.cc:844] Serialized 14 byte log entry
0709 12:25:14.282431 (+ 12690us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:14.282504 (+    73us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:25:14.282565 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:14.282648 (+    83us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:14.282787 (+   139us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":64933,"prepare.queue_time_us":220573,"prepare.run_cpu_time_us":76,"prepare.run_wall_time_us":8589,"raft.queue_time_us":74149,"raft.run_cpu_time_us":328,"raft.run_wall_time_us":9189,"spinlock_wait_cycles":7277312,"thread_start_us":618817,"threads_started":2,"wal-append.queue_time_us":979566}
W20260709 12:25:14.804760 17729 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:14.808063 27338 raft_consensus.cc:697] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 25 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:25:14.809089 27338 consensus_queue.cc:237] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 23.17, Last appended by leader: 17, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:14.810169 27570 consensus_queue.cc:237] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 26.18, Last appended by leader: 18, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:14.814285 17803 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:14.817881 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 32 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:25:14.821425 17803 leader_election.cc:304] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:25:14.823522 17682 raft_consensus.cc:2468] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 28.
I20260709 12:25:14.826829 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:14.828202 17685 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37402 (request call id 3973) took 2661 ms. Trace:
W20260709 12:25:14.830219 17728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:14.830786 17689 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37402 (request call id 3978) took 2654 ms (client timeout 2211 ms). Trace:
I20260709 12:25:14.832415 17680 raft_consensus.cc:1275] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 34 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 32 index: 19. Preceding OpId from leader: term: 34 index: 20. (index mismatch)
I20260709 12:25:14.834434 17832 raft_consensus.cc:2393] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 28: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
W20260709 12:25:14.834710 17756 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38776 (request call id 3372) took 2008 ms (client timeout 1915 ms). Trace:
I20260709 12:25:14.828603 17685 rpcz_store.cc:276] 0709 12:25:12.167233 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:12.167929 (+   696us) service_pool.cc:224] Handling call
0709 12:25:12.671426 (+503497us) spinlock_profiling.cc:243] Waited 8.94 ms on lock 0x7b5800002a98. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b474fbb71 00007f7b474fba22 00007f7b504148d9 00007f7b4f56eee0 00007f7b4f6380ab 00007f7b4f638131 00007f7b4f5f9fc0 00007f7b4f5e4929 00007f7b4f5e43d2 00007f7b48b5bbd4
0709 12:25:12.703024 (+ 31598us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:13.908120 (+1205096us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:14.827977 (+919857us) spinlock_profiling.cc:243] Waited 1.2 s on lock 0x7b5400347bd4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:25:14.828141 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2789920000}
W20260709 12:25:14.835111 17756 rpcz_store.cc:269] 0709 12:25:12.816858 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:12.817621 (+   763us) service_pool.cc:224] Handling call
0709 12:25:14.824743 (+2007122us) spinlock_profiling.cc:243] Waited 332 ms on lock 0x7b5400216e54. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:25:14.824849 (+   106us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":762557312}
I20260709 12:25:14.836484 17682 raft_consensus.cc:1240] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 33 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 32. Current term is 33. Ops: []
I20260709 12:25:14.837678 17685 raft_consensus.cc:1240] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 27. Current term is 28. Ops: []
W20260709 12:25:14.831351 17689 rpcz_store.cc:269] 0709 12:25:12.176723 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:12.719061 (+542338us) service_pool.cc:224] Handling call
0709 12:25:14.830501 (+2111440us) spinlock_profiling.cc:243] Waited 1.07 s on lock 0x7b5400426dd4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:25:14.830669 (+   168us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2453500288}
W20260709 12:25:14.841336 18333 connection.cc:256] RPC call timeout handler was delayed by -1.3116509330002373s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1821s for a response
I20260709 12:25:14.783812 27863 raft_consensus.cc:2804] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:25:14.844539 18374 raft_consensus.cc:1240] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 26. Current term is 27. Ops: []
I20260709 12:25:14.844805 17685 raft_consensus.cc:1240] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 23. Current term is 24. Ops: []
I20260709 12:25:14.845693 17688 raft_consensus.cc:1275] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 29 index: 20. Preceding OpId from leader: term: 30 index: 20. (term mismatch)
I20260709 12:25:14.847921 17688 pending_rounds.cc:85] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3: Aborting all ops after (but not including) 19
I20260709 12:25:14.848248 17688 pending_rounds.cc:107] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3: Aborting uncommitted NO_OP operation due to leader change: 29.20
I20260709 12:25:14.848563 17688 raft_consensus.cc:2889] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:14.852417 27863 raft_consensus.cc:697] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 26 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:25:14.853531 27863 consensus_queue.cc:237] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 23.16, Last appended by leader: 16, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:14.839852 27883 raft_consensus.cc:2804] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:25:14.856493 27883 raft_consensus.cc:697] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 31 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:25:14.857657 27883 consensus_queue.cc:237] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 29.17, Last appended by leader: 17, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:14.860653 17585 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35966 (request call id 359) took 51693 ms (client timeout 15000 ms). Trace:
I20260709 12:25:14.865202 18366 raft_consensus.cc:2468] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 31.
W20260709 12:25:14.845175 18333 connection.cc:256] RPC call timeout handler was delayed by -1.3116509330002373s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1821s for a response
W20260709 12:25:14.870533 18333 connection.cc:256] RPC call timeout handler was delayed by -1.2513509330001398s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1888s for a response
W20260709 12:25:14.871059 18333 connection.cc:256] RPC call timeout handler was delayed by -1.2324509330001092s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19090000000000001s for a response
I20260709 12:25:14.847376 17682 raft_consensus.cc:1240] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 25. Current term is 26. Ops: [25.17-25.17]
I20260709 12:25:14.811803 27716 consensus_queue.cc:1059] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.034s
W20260709 12:25:14.871500 18333 connection.cc:256] RPC call timeout handler was delayed by -1.2207509330000903s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1922s for a response
W20260709 12:25:14.876065 18333 connection.cc:256] RPC call timeout handler was delayed by -1.1631509329999972s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1986s for a response
W20260709 12:25:14.877122 18333 connection.cc:256] RPC call timeout handler was delayed by -1.1631509329999972s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1986s for a response
I20260709 12:25:14.878515 27853 raft_consensus.cc:493] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:14.879163 27853 raft_consensus.cc:515] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:14.881381 27853 leader_election.cc:290] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:14.883730 27873 raft_consensus.cc:515] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:25:14.876770 27944 log.cc:927] Time spent T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.322s	user 0.002s	sys 0.002s
I20260709 12:25:14.886897 27873 leader_election.cc:290] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 32 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:25:14.888481 17682 raft_consensus.cc:1240] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 28. Current term is 30. Ops: []
I20260709 12:25:14.888571 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 32 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:25:14.889578 17688 raft_consensus.cc:2393] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 32: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
W20260709 12:25:14.877691 18333 connection.cc:256] RPC call timeout handler was delayed by -1.1181509330003792s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2036s for a response
W20260709 12:25:14.889967 18333 connection.cc:256] RPC call timeout handler was delayed by -1.10015093300035s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2056s for a response
W20260709 12:25:14.890378 18333 connection.cc:256] RPC call timeout handler was delayed by -1.0776509330003137s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2081s for a response
I20260709 12:25:14.890671 18374 raft_consensus.cc:1275] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 25 index: 18. Preceding OpId from leader: term: 26 index: 19. (index mismatch)
W20260709 12:25:14.890995 18333 connection.cc:256] RPC call timeout handler was delayed by -1.0038509330001943s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2163s for a response
I20260709 12:25:14.897464 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 33 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:14.917356 17755 raft_consensus.cc:2410] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for term 33 because replica has last-logged OpId of term: 32 index: 22, which is greater than that of the candidate, which has last-logged OpId of term: 31 index: 21.
I20260709 12:25:14.906610 27767 raft_consensus.cc:3060] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:14.912653 17681 raft_consensus.cc:1240] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 27 LEADER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 26. Current term is 27. Ops: [26.21-26.21]
I20260709 12:25:14.914054 27873 consensus_queue.cc:1059] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 28.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:25:14.915145 27891 consensus_queue.cc:1059] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 26.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
W20260709 12:25:14.889739 27946 log.cc:927] Time spent T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.159s	user 0.000s	sys 0.000s
I20260709 12:25:14.924561 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:14.945567 27711 consensus_queue.cc:1059] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 28.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.003s
I20260709 12:25:14.915632 27865 raft_consensus.cc:3055] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:25:14.947896 17729 leader_election.cc:304] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:25:14.875295 27881 raft_consensus.cc:493] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:14.881619 27943 log.cc:927] Time spent T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.329s	user 0.001s	sys 0.000s
I20260709 12:25:14.928128 27640 raft_consensus.cc:3055] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:25:14.967643 27640 raft_consensus.cc:740] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:25:14.968046 27891 raft_consensus.cc:3055] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:25:14.968465 27891 raft_consensus.cc:740] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 26 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:25:14.968611 27640 consensus_queue.cc:260] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 24.18, Last appended by leader: 18, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:14.969415 27891 consensus_queue.cc:260] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 26.18, Last appended by leader: 18, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:14.969985 27640 raft_consensus.cc:3060] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:14.947640 27666 consensus_queue.cc:1059] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 23.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.042s
I20260709 12:25:14.970777 27891 raft_consensus.cc:3060] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:14.961835 27865 raft_consensus.cc:740] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:25:14.954615 18364 raft_consensus.cc:1275] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 26 index: 13. Preceding OpId from leader: term: 28 index: 14. (index mismatch)
W20260709 12:25:14.954115 18334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:14.979104 27809 raft_consensus.cc:515] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:14.954943 18333 leader_election.cc:304] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:14.958930 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 32 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:14.962566 18361 raft_consensus.cc:1240] T a5732e71b9d54f8786b8c511d69943bf P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 27. Current term is 28. Ops: []
I20260709 12:25:14.951611 27525 raft_consensus.cc:2749] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:25:14.966194 27881 raft_consensus.cc:515] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:14.861162 17585 rpcz_store.cc:269] 0709 12:24:23.167168 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:34.135994 (+10968826us) service_pool.cc:224] Handling call
0709 12:24:34.218601 (+ 82607us) spinlock_profiling.cc:243] Waited 5.47 ms on lock 0x7b540010d454. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a58a090 00007f7b4a628f3a 00007f7b503dee9c 00007f7b504dd04e 00007f7b4e99e2a4 00007f7b4e99e232 00007f7b4e99e161 00007f7b4e99e0dc 00007f7b4e99d392 00007f7b4895b48c 00007f7b4895a8c6
0709 12:25:14.860426 (+40641825us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:25:05.289953 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:25:05.290608 (+   655us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:25:05.290669 (+    61us) write_op.cc:435] Acquired schema lock
0709 12:25:05.290688 (+    19us) tablet.cc:662] Decoding operations
0709 12:25:05.461856 (+171168us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:25:05.517186 (+ 55330us) write_op.cc:641] Partition lock acquired for write op
0709 12:25:05.519230 (+  2044us) tablet.cc:685] Acquiring locks for 17 operations
0709 12:25:05.751307 (+232077us) tablet.cc:701] Row locks acquired
0709 12:25:05.751362 (+    55us) write_op.cc:260] PREPARE: finished
0709 12:25:05.751544 (+   182us) write_op.cc:270] Start()
0709 12:25:05.751712 (+   168us) write_op.cc:276] Timestamp: P: 1783599905751479 usec, L: 0
0709 12:25:05.751739 (+    27us) op_driver.cc:348] REPLICATION: starting
0709 12:25:05.961959 (+210220us) log.cc:844] Serialized 6682 byte log entry
0709 12:25:06.149084 (+187125us) spinlock_profiling.cc:243] Waited 6.13 ms on lock 0x7b4400022b4c. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b47459c85 00007f7b4745a6a2 00007f7b4a1cc1db 00007f7b4a1c7aaf 00007f7b475160ac 00007f7b503709a9 00007f7b4753269b 00007f7b4753093f 00007f7b47532c02 00007f7b4a1ba20b
0709 12:25:06.429862 (+280778us) op_driver.cc:464] REPLICATION: finished
0709 12:25:06.682493 (+252631us) write_op.cc:301] APPLY: starting
0709 12:25:06.682779 (+   286us) tablet.cc:1367] starting BulkCheckPresence
0709 12:25:06.899976 (+217197us) tablet.cc:1370] finished BulkCheckPresence
0709 12:25:06.900054 (+    78us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:25:07.482463 (+582409us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:25:07.636303 (+153840us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=34,key_file_lookups=34,delta_file_lookups=0,mrs_lookups=0
0709 12:25:07.646678 (+ 10375us) write_op.cc:312] APPLY: finished
0709 12:25:07.658999 (+ 12321us) log.cc:844] Serialized 156 byte log entry
0709 12:25:07.659257 (+   258us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:25:07.659680 (+   423us) write_op.cc:454] Released schema lock
0709 12:25:07.659810 (+   130us) write_op.cc:341] FINISH: Updating metrics
0709 12:25:07.660382 (+   572us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:25:07.660418 (+    36us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":4548,"reported_tablets":46,"spinlock_wait_cycles":12585728,"child_traces":[["op",{"apply.queue_time_us":1394,"apply.run_cpu_time_us":31261,"apply.run_wall_time_us":1228713,"cfile_cache_hit":70,"cfile_cache_hit_bytes":75255,"num_ops":17,"prepare.queue_time_us":341389,"prepare.run_cpu_time_us":20272,"prepare.run_wall_time_us":1035056,"raft.queue_time_us":24527,"raft.run_cpu_time_us":1157,"raft.run_wall_time_us":2250,"replication_time_us":677910,"spinlock_wait_cycles":14101760,"thread_start_us":855085,"threads_started":4,"wal-append.queue_time_us":413657,"wal-append.run_cpu_time_us":6840,"wal-append.run_wall_time_us":2600126}]]}
I20260709 12:25:14.971251 27832 raft_consensus.cc:2749] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:25:14.972189 27865 consensus_queue.cc:260] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 28.17, Last appended by leader: 17, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:14.954181 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:15.002539 18374 raft_consensus.cc:2468] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 24.
I20260709 12:25:14.986393 27766 leader_election.cc:290] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 29 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:14.989058 27901 raft_consensus.cc:3060] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:25:14.991451 17729 leader_election.cc:304] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:25:15.009503 17729 leader_election.cc:304] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:25:14.993299 17829 raft_consensus.cc:2468] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 26.
I20260709 12:25:14.957782 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
W20260709 12:25:15.001091 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 46271.5us
I20260709 12:25:15.012511 18333 leader_election.cc:304] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 34 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:15.011504 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
W20260709 12:25:15.004179 17728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:15.017951 18333 leader_election.cc:304] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:15.018910 27753 raft_consensus.cc:2749] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 34 FOLLOWER]: Leader election lost for term 34. Reason: could not achieve majority
I20260709 12:25:15.024863 27871 raft_consensus.cc:3055] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:25:15.025331 27871 raft_consensus.cc:740] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 26 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:25:15.026197 27871 consensus_queue.cc:260] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 26.21, Last appended by leader: 21, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:15.027535 27871 raft_consensus.cc:3060] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:15.030301 27881 leader_election.cc:290] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:15.019174 27525 raft_consensus.cc:2749] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260709 12:25:15.031428 27865 raft_consensus.cc:3060] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:15.033092 27740 raft_consensus.cc:2749] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:25:15.042661 27809 leader_election.cc:290] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 30 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:15.043133 27755 consensus_queue.cc:1059] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 20.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:25:15.044978 27525 raft_consensus.cc:3055] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:25:15.045593 27525 raft_consensus.cc:740] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:25:15.046725 27525 consensus_queue.cc:260] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 20.15, Last appended by leader: 15, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:15.049316 27809 consensus_queue.cc:1048] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:25:15.052013 27832 consensus_queue.cc:1059] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 27.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20260709 12:25:15.053433 27577 raft_consensus.cc:3055] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:25:15.054003 27577 raft_consensus.cc:740] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 27 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:25:15.054596 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:15.055251 27577 consensus_queue.cc:260] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 27.16, Last appended by leader: 16, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:15.057722 27577 raft_consensus.cc:3060] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Advancing to term 28
W20260709 12:25:15.058969 17652 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:15.061873 27891 raft_consensus.cc:493] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:15.062423 27891 raft_consensus.cc:515] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:15.064299 18367 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59918 (request call id 2888) took 1566 ms. Trace:
I20260709 12:25:15.064455 18371 raft_consensus.cc:1240] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 26 LEADER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 25. Current term is 26. Ops: [25.17-25.17]
I20260709 12:25:15.064797 27891 leader_election.cc:290] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:15.064703 18367 rpcz_store.cc:276] 0709 12:25:13.498042 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:13.505043 (+  7001us) service_pool.cc:224] Handling call
0709 12:25:15.064055 (+1559012us) spinlock_profiling.cc:243] Waited 357 ms on lock 0x7b5400499654. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:25:15.064220 (+   165us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":820286720}
I20260709 12:25:15.072257 17830 raft_consensus.cc:2410] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for term 32 because replica has last-logged OpId of term: 31 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 30 index: 17.
W20260709 12:25:15.072366 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 54532us
I20260709 12:25:15.072695 27856 raft_consensus.cc:3055] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:25:15.073256 27856 raft_consensus.cc:740] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 30 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
W20260709 12:25:15.073974 18334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:15.074563 27881 raft_consensus.cc:2749] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
I20260709 12:25:15.074378 27856 consensus_queue.cc:260] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 30.18, Last appended by leader: 18, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:15.075224 17727 leader_election.cc:304] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:15.076795 27640 raft_consensus.cc:2749] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 32 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:25:15.078915 27713 raft_consensus.cc:3055] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 33 LEADER]: Stepping down as leader of term 33
I20260709 12:25:15.079475 27713 raft_consensus.cc:740] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 33 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:25:15.085642 17763 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38786 (request call id 4213) took 1049 ms. Trace:
I20260709 12:25:15.086229 17763 rpcz_store.cc:276] 0709 12:25:14.036556 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:14.336191 (+299635us) service_pool.cc:224] Handling call
0709 12:25:14.337897 (+  1706us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:14.382423 (+ 44526us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:15.085269 (+702846us) spinlock_profiling.cc:243] Waited 28.2 ms on lock 0x7b540019f0d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:25:15.085555 (+   286us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":64940416}
I20260709 12:25:15.088182 17755 raft_consensus.cc:2410] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for term 25 because replica has last-logged OpId of term: 24 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 17.
I20260709 12:25:15.088698 27446 raft_consensus.cc:3055] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:25:15.089260 27446 raft_consensus.cc:740] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 27 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:25:15.090039 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 34 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:15.090766 17755 raft_consensus.cc:3060] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:25:15.090483 27446 consensus_queue.cc:260] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 27.18, Last appended by leader: 18, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:15.093614 27713 consensus_queue.cc:260] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 33.21, Last appended by leader: 21, Current term: 33, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:15.095438 27713 raft_consensus.cc:3060] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:25:15.099395 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:15.100307 17763 raft_consensus.cc:2376] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 32 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 32. Re-sending same reply.
I20260709 12:25:15.104446 18333 leader_election.cc:304] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:15.110251 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 31 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:15.114630 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:15.115362 17763 raft_consensus.cc:3060] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Advancing to term 28
W20260709 12:25:15.119421 17730 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:15.122553 17755 raft_consensus.cc:2468] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 34.
I20260709 12:25:15.124321 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 29 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:15.129364 18333 leader_election.cc:304] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:15.109680 27856 raft_consensus.cc:3060] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:15.136394 17755 raft_consensus.cc:2468] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 29.
I20260709 12:25:15.125849 27853 raft_consensus.cc:493] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:25:15.111795 27881 raft_consensus.cc:2749] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:25:15.138191 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:15.133029 27903 raft_consensus.cc:493] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:15.141417 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:15.149986 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:15.150882 17755 raft_consensus.cc:2410] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for term 25 because replica has last-logged OpId of term: 24 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 17.
I20260709 12:25:15.151676 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:15.141346 18333 leader_election.cc:304] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:15.152680 27724 leader_election.cc:290] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:15.153064 18333 leader_election.cc:304] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:15.133170 27863 raft_consensus.cc:2749] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:25:15.137843 27753 raft_consensus.cc:2804] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:25:15.134829 27446 raft_consensus.cc:3060] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:15.153941 17755 raft_consensus.cc:3060] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:25:15.157732 18333 leader_election.cc:304] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:15.146764 17763 raft_consensus.cc:2468] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 28.
I20260709 12:25:15.164485 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:15.165597 27724 raft_consensus.cc:493] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:25:15.168224 27724 raft_consensus.cc:515] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:15.171257 27724 leader_election.cc:290] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:15.172626 17763 raft_consensus.cc:2393] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 28: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:25:15.172995 27671 raft_consensus.cc:3060] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:15.173533 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 29 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:15.174243 17763 raft_consensus.cc:2393] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 29: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:25:15.174966 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 33 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:15.178357 27431 raft_consensus.cc:3055] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:25:15.179042 27431 raft_consensus.cc:740] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 23 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
W20260709 12:25:15.179471 17652 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:15.179127 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 26 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:15.180176 27431 consensus_queue.cc:260] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 23.15, Last appended by leader: 15, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:15.180783 27872 raft_consensus.cc:3055] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:25:15.181239 27872 raft_consensus.cc:740] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 28 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:25:15.181461 27431 raft_consensus.cc:3060] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:15.181625 17755 raft_consensus.cc:2468] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 32.
I20260709 12:25:15.182066 27872 consensus_queue.cc:260] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 28.16, Last appended by leader: 16, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:15.183399 27872 raft_consensus.cc:3060] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:15.182516 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:15.185128 17755 raft_consensus.cc:2410] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for term 26 because replica has last-logged OpId of term: 25 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 16.
I20260709 12:25:15.181490 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 26 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:15.186165 17763 raft_consensus.cc:2468] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 25.
I20260709 12:25:15.156283 27753 raft_consensus.cc:493] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:25:15.156199 27338 raft_consensus.cc:2749] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:25:15.197542 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 32 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:25:15.198927 17728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:15.199136 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:15.200147 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 30 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:15.188021 27692 consensus_queue.cc:1059] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 23.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.013s
I20260709 12:25:15.200784 17763 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 29.
I20260709 12:25:15.201517 27895 raft_consensus.cc:3060] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Advancing to term 29
W20260709 12:25:15.201840 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:15.203059 17763 raft_consensus.cc:1275] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 25 index: 15. Preceding OpId from leader: term: 29 index: 16. (index mismatch)
I20260709 12:25:15.200863 27692 consensus_queue.cc:1243] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer 334a4e1f59944544b3b3ff36616b57da log is divergent from this leader: its last log entry 26.17 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 16
I20260709 12:25:15.204555 27724 raft_consensus.cc:3055] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:25:15.204972 27724 raft_consensus.cc:740] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:25:15.205832 27724 consensus_queue.cc:260] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 25.17, Last appended by leader: 17, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:15.207181 27724 raft_consensus.cc:3060] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:15.208355 27338 raft_consensus.cc:2749] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:25:15.197175 27753 raft_consensus.cc:3060] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:15.159309 27881 raft_consensus.cc:2749] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:25:15.210285 17763 raft_consensus.cc:1240] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 34 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 33. Current term is 34. Ops: []
W20260709 12:25:15.212324 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 200446us
I20260709 12:25:15.141337 27853 raft_consensus.cc:515] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:15.219121 17763 raft_consensus.cc:1240] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 28. Current term is 29. Ops: []
I20260709 12:25:15.217756 17686 raft_consensus.cc:2468] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 27.
I20260709 12:25:15.155175 27903 raft_consensus.cc:515] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:25:15.224611 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 66951.2us
I20260709 12:25:15.225426 18333 leader_election.cc:304] T a5732e71b9d54f8786b8c511d69943bf P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:25:15.237923 27945 log.cc:927] Time spent T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.688s	user 0.003s	sys 0.000s
I20260709 12:25:15.243216 27884 raft_consensus.cc:3055] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:25:15.245273 17829 raft_consensus.cc:1240] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 27. Current term is 28. Ops: []
I20260709 12:25:15.248706 17837 raft_consensus.cc:1275] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 32 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 30 index: 22. Preceding OpId from leader: term: 32 index: 23. (index mismatch)
I20260709 12:25:15.249316 17829 raft_consensus.cc:1275] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 28 index: 20. Preceding OpId from leader: term: 30 index: 21. (index mismatch)
I20260709 12:25:15.243767 27884 raft_consensus.cc:740] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 28 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:25:15.251757 27884 consensus_queue.cc:260] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 28.15, Last appended by leader: 15, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:15.253288 27884 raft_consensus.cc:3060] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:15.256400 17728 leader_election.cc:304] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:25:15.259774 18334 leader_election.cc:304] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:25:15.261507 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 34 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:15.266276 17837 raft_consensus.cc:2468] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 33.
I20260709 12:25:15.268966 17762 raft_consensus.cc:2468] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 26.
I20260709 12:25:15.263608 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:15.270124 17838 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 27.
W20260709 12:25:15.272831 17654 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:15.273463 27898 raft_consensus.cc:2749] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
I20260709 12:25:15.294682 27671 consensus_queue.cc:1048] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 18, Time since last communication: 0.000s
I20260709 12:25:15.295773 17801 leader_election.cc:304] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
W20260709 12:25:15.280318 17729 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:15.271610 27927 raft_consensus.cc:493] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:15.284586 17654 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:15.295025 17652 leader_election.cc:400] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 33 election: Vote denied by peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651) with higher term. Message: Invalid argument: T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 34 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for earlier term 33. Current term is 34.
I20260709 12:25:15.281275 27692 consensus_queue.cc:1059] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20260709 12:25:15.305522 17755 raft_consensus.cc:2468] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 28.
I20260709 12:25:15.305826 27692 consensus_queue.cc:1243] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer 0da169ff82674dc1947055ce43de68a3 log is divergent from this leader: its last log entry 31.18 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 17
I20260709 12:25:15.306602 27692 raft_consensus.cc:3055] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 32 LEADER]: Stepping down as leader of term 32
I20260709 12:25:15.307016 27692 raft_consensus.cc:740] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 32 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:25:15.309106 27853 leader_election.cc:290] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:25:15.265411 27929 raft_consensus.cc:493] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:25:15.313000 27929 raft_consensus.cc:515] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:15.315594 27929 leader_election.cc:290] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:15.299759 17652 leader_election.cc:403] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 33 election: Cancelling election due to peer responding with higher term
I20260709 12:25:15.318558 27903 leader_election.cc:290] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:15.319795 17652 leader_election.cc:304] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:25:15.320044 27927 raft_consensus.cc:515] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:15.321035 27525 raft_consensus.cc:3060] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:15.321532 17652 leader_election.cc:304] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:15.322925 27839 raft_consensus.cc:2749] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 34 FOLLOWER]: Leader election lost for term 33. Reason: Vote denied by peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651) with higher term. Message: Invalid argument: T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 34 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for earlier term 33. Current term is 34.
I20260709 12:25:15.255308 27881 raft_consensus.cc:2749] T a5732e71b9d54f8786b8c511d69943bf P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:25:15.323771 27839 raft_consensus.cc:2804] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 32 FOLLOWER]: Leader election won for term 32
I20260709 12:25:15.324383 27839 raft_consensus.cc:697] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 32 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:25:15.324741 17761 raft_consensus.cc:1275] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 29 index: 18. Preceding OpId from leader: term: 30 index: 18. (term mismatch)
I20260709 12:25:15.325153 17761 pending_rounds.cc:85] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb: Aborting all ops after (but not including) 17
I20260709 12:25:15.325372 17761 pending_rounds.cc:107] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb: Aborting uncommitted NO_OP operation due to leader change: 29.18
I20260709 12:25:15.325605 17761 raft_consensus.cc:2889] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:15.325528 27839 consensus_queue.cc:237] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 30.17, Last appended by leader: 17, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:15.328045 27570 raft_consensus.cc:2804] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:25:15.328553 27570 raft_consensus.cc:697] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 29 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:25:15.329459 27570 consensus_queue.cc:237] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 28.15, Last appended by leader: 15, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:15.332190 17652 leader_election.cc:304] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
W20260709 12:25:15.301226 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:15.334872 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:15.303448 27338 raft_consensus.cc:2804] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:25:15.268366 27840 raft_consensus.cc:493] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:15.344825 27898 consensus_queue.cc:1048] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.001s
I20260709 12:25:15.344882 27840 raft_consensus.cc:515] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:15.317036 27449 raft_consensus.cc:2804] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 34 FOLLOWER]: Leader election won for term 34
I20260709 12:25:15.346781 27449 raft_consensus.cc:697] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 34 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:25:15.348148 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 33 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:15.300729 17758 raft_consensus.cc:1240] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 24 LEADER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 23. Current term is 24. Ops: []
I20260709 12:25:15.297212 17801 leader_election.cc:304] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:15.325676 17687 raft_consensus.cc:1240] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 27. Current term is 28. Ops: []
W20260709 12:25:15.338282 17802 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:15.341993 17654 leader_election.cc:304] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:25:15.343276 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:15.343214 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 31 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:15.342761 17727 leader_election.cc:304] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:15.365958 17758 raft_consensus.cc:1275] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 32 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 31 index: 18. Preceding OpId from leader: term: 32 index: 19. (index mismatch)
W20260709 12:25:15.351078 17728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:15.367462 17758 raft_consensus.cc:1240] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 32 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 31. Current term is 32. Ops: [31.20-31.20]
I20260709 12:25:15.351264 17836 raft_consensus.cc:1240] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 31 LEADER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 30. Current term is 31. Ops: [30.18-30.18]
W20260709 12:25:15.369372 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 103769us
W20260709 12:25:15.371033 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 73923.2us
I20260709 12:25:15.371726 17652 leader_election.cc:304] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:15.371861 17837 raft_consensus.cc:1240] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 26. Current term is 27. Ops: []
I20260709 12:25:15.374087 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:15.375839 17832 raft_consensus.cc:2393] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 32: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:25:15.376338 17652 leader_election.cc:304] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:15.384543 17759 raft_consensus.cc:2410] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for term 28 because replica has last-logged OpId of term: 27 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 26 index: 19.
I20260709 12:25:15.341369 27871 raft_consensus.cc:493] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:15.374900 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 149536us
I20260709 12:25:15.387603 18333 leader_election.cc:304] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 31 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:15.344470 27338 raft_consensus.cc:697] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 28 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:25:15.388959 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:25:15.390095 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 48201.2us
I20260709 12:25:15.389906 17652 leader_election.cc:304] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:15.349156 27449 consensus_queue.cc:237] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 33.21, Last appended by leader: 21, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:25:15.448340 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 58508us
I20260709 12:25:15.448449 27951 raft_consensus.cc:2804] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:25:15.430070 17755 raft_consensus.cc:1240] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 28. Current term is 29. Ops: [28.17-28.17]
I20260709 12:25:15.449137 27951 raft_consensus.cc:493] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:25:15.449563 27951 raft_consensus.cc:3060] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:15.452718 27865 consensus_queue.cc:1048] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 13, Time since last communication: 0.000s
I20260709 12:25:15.404052 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:15.455451 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:15.457532 27927 leader_election.cc:290] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:15.459328 27891 consensus_queue.cc:1048] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
W20260709 12:25:15.460088 17803 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:15.369903 27874 consensus_queue.cc:1048] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.091s
I20260709 12:25:15.461154 17832 raft_consensus.cc:2468] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 26.
I20260709 12:25:15.465502 17755 raft_consensus.cc:1240] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 32 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 31. Current term is 32. Ops: [31.18-31.18]
W20260709 12:25:15.464393 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 55991.4us
I20260709 12:25:15.466578 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 32 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:15.428821 17831 raft_consensus.cc:3060] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:15.467288 17755 raft_consensus.cc:2393] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 32: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
W20260709 12:25:15.438264 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 50727.5us
I20260709 12:25:15.466140 17654 leader_election.cc:304] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:25:15.469717 18333 leader_election.cc:304] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:25:15.470712 17832 raft_consensus.cc:1240] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 26. Current term is 27. Ops: []
I20260709 12:25:15.470781 17836 raft_consensus.cc:2468] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 25.
I20260709 12:25:15.473183 17834 raft_consensus.cc:1275] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 23 index: 15. Preceding OpId from leader: term: 24 index: 16. (index mismatch)
W20260709 12:25:15.456692 27949 log.cc:927] Time spent T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.083s	user 0.000s	sys 0.000s
I20260709 12:25:15.395725 17835 raft_consensus.cc:1275] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 26 index: 21. Preceding OpId from leader: term: 27 index: 21. (term mismatch)
I20260709 12:25:15.476059 17835 pending_rounds.cc:85] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b: Aborting all ops after (but not including) 20
I20260709 12:25:15.476364 17835 pending_rounds.cc:107] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b: Aborting uncommitted NO_OP operation due to leader change: 26.21
I20260709 12:25:15.476681 17835 raft_consensus.cc:2889] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:15.472213 17654 leader_election.cc:304] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:25:15.486631 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 34 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:15.486737 17654 leader_election.cc:304] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:15.487742 17801 leader_election.cc:304] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:15.490384 17727 leader_election.cc:304] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:15.344544 27824 consensus_queue.cc:1048] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.120s
I20260709 12:25:15.377315 27713 raft_consensus.cc:2804] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:25:15.493969 18333 leader_election.cc:304] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:15.395454 27759 raft_consensus.cc:2749] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260709 12:25:15.498128 27800 raft_consensus.cc:2749] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:25:15.333264 27692 consensus_queue.cc:260] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 32.18, Last appended by leader: 18, Current term: 32, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:15.463287 27767 raft_consensus.cc:2804] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:25:15.463490 27778 raft_consensus.cc:2804] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 32 FOLLOWER]: Leader election won for term 32
I20260709 12:25:15.494753 27871 raft_consensus.cc:515] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:15.498509 27874 raft_consensus.cc:2749] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
I20260709 12:25:15.375756 27922 raft_consensus.cc:2749] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:25:15.505841 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 29 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:15.440761 27965 raft_consensus.cc:2749] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 32 FOLLOWER]: Leader election lost for term 31. Reason: could not achieve majority
I20260709 12:25:15.495745 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:15.350189 27446 consensus_queue.cc:1048] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.149s
I20260709 12:25:15.494029 27766 raft_consensus.cc:2804] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:25:15.544348 27724 raft_consensus.cc:493] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:15.521930 18333 leader_election.cc:304] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:15.496241 27725 raft_consensus.cc:2804] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:25:15.498077 27903 raft_consensus.cc:2804] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:25:15.370899 27929 raft_consensus.cc:2804] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:25:15.458911 27951 raft_consensus.cc:515] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:15.496542 27744 raft_consensus.cc:493] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:25:15.533847 17764 raft_consensus.cc:1275] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 34 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 32 index: 19. Preceding OpId from leader: term: 34 index: 20. (index mismatch)
I20260709 12:25:15.472061 27865 consensus_queue.cc:1048] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.045s
I20260709 12:25:15.531672 27008 raft_consensus.cc:493] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:25:15.347784 27840 leader_election.cc:290] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:15.534502 27881 raft_consensus.cc:3060] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:15.519590 27968 raft_consensus.cc:2804] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:25:15.549296 27692 raft_consensus.cc:3060] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:25:15.545056 27767 raft_consensus.cc:697] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 27 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:25:15.576829 27871 leader_election.cc:290] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:15.566155 27766 raft_consensus.cc:493] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:15.572224 27724 raft_consensus.cc:515] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:15.585206 17655 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:15.671943 17655 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:15.672398 17655 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:15.572409 27824 raft_consensus.cc:493] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:15.589469 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:15.589990 17829 raft_consensus.cc:1275] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 26 index: 13. Preceding OpId from leader: term: 28 index: 14. (index mismatch)
I20260709 12:25:15.590847 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 34 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:15.591066 18374 raft_consensus.cc:1275] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 32 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 26 index: 20. Preceding OpId from leader: term: 32 index: 23. (index mismatch)
I20260709 12:25:15.696028 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:15.696581 18374 raft_consensus.cc:2468] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 25.
I20260709 12:25:15.591938 18371 raft_consensus.cc:1275] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 28 index: 20. Preceding OpId from leader: term: 30 index: 21. (index mismatch)
I20260709 12:25:15.592804 18367 raft_consensus.cc:1275] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 30 index: 20. Preceding OpId from leader: term: 31 index: 21. (index mismatch)
I20260709 12:25:15.593348 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:15.594029 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 29 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:15.700984 18364 raft_consensus.cc:2468] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 28.
W20260709 12:25:15.608094 17728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:15.546916 27778 raft_consensus.cc:697] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 32 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
W20260709 12:25:15.613723 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 91870.4us
I20260709 12:25:15.704993 18363 raft_consensus.cc:2393] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 30: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:25:15.614259 27753 raft_consensus.cc:515] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:15.607653 17832 raft_consensus.cc:2468] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 29.
I20260709 12:25:15.613638 27903 raft_consensus.cc:493] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:15.713204 27927 consensus_queue.cc:1048] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.001s
I20260709 12:25:15.713892 27903 raft_consensus.cc:3060] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:15.714303 27892 raft_consensus.cc:493] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:15.714775 27892 raft_consensus.cc:515] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:15.714936 27898 consensus_queue.cc:1048] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.000s
I20260709 12:25:15.612334 27725 raft_consensus.cc:493] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:15.717310 27725 raft_consensus.cc:3060] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:15.634645 17831 raft_consensus.cc:2468] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 30.
I20260709 12:25:15.613298 27929 raft_consensus.cc:493] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 33 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:15.720402 27929 raft_consensus.cc:3060] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:25:15.547325 27338 consensus_queue.cc:237] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 26.19, Last appended by leader: 19, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:15.673563 17802 leader_election.cc:304] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:25:15.725960 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 25 candidate_status { last_received { term: 23 index: 21 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:25:15.726631 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 53198.1us
I20260709 12:25:15.726809 17683 raft_consensus.cc:2468] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 24.
I20260709 12:25:15.695039 18368 raft_consensus.cc:2468] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 33.
I20260709 12:25:15.728816 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:15.729270 27902 consensus_queue.cc:1048] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:25:15.731769 27874 raft_consensus.cc:2804] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:25:15.706171 18333 leader_election.cc:304] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:25:15.624950 27008 raft_consensus.cc:515] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:15.709599 17729 leader_election.cc:304] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:15.710194 17687 raft_consensus.cc:1275] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 30 index: 18. Preceding OpId from leader: term: 31 index: 18. (term mismatch)
I20260709 12:25:15.734056 17687 pending_rounds.cc:85] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3: Aborting all ops after (but not including) 17
I20260709 12:25:15.734299 17687 pending_rounds.cc:107] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3: Aborting uncommitted NO_OP operation due to leader change: 30.18
I20260709 12:25:15.734527 17687 raft_consensus.cc:2889] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:15.706351 27824 raft_consensus.cc:515] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:15.732332 27874 raft_consensus.cc:493] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:15.748409 27874 raft_consensus.cc:3060] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:15.749459 27824 leader_election.cc:290] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:25:15.753024 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 43531.8us
I20260709 12:25:15.691993 27766 raft_consensus.cc:3060] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:25:15.762199 27008 leader_election.cc:290] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:15.763660 27008 raft_consensus.cc:493] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:15.763834 27853 raft_consensus.cc:493] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:15.764204 27008 raft_consensus.cc:515] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:15.764619 27892 leader_election.cc:290] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:15.619576 27744 raft_consensus.cc:515] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:15.765197 17756 raft_consensus.cc:3060] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:15.766186 27724 leader_election.cc:290] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:15.766731 27744 leader_election.cc:290] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:15.767565 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 33 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:15.768165 18361 raft_consensus.cc:1240] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 21. Current term is 22. Ops: []
I20260709 12:25:15.768220 17760 raft_consensus.cc:2468] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 32.
I20260709 12:25:15.768709 27753 leader_election.cc:290] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 22 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:15.769534 27753 raft_consensus.cc:2804] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:25:15.770584 27008 leader_election.cc:290] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:25:15.764400 27853 raft_consensus.cc:515] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:15.670279 27968 raft_consensus.cc:697] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 24 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:25:15.757810 17802 leader_election.cc:304] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:25:15.774107 17756 raft_consensus.cc:2468] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 29.
I20260709 12:25:15.698946 27965 raft_consensus.cc:2764] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Leader election decision vote started in defunct term 29: won
I20260709 12:25:15.770015 17801 leader_election.cc:304] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:15.713975 27446 consensus_queue.cc:1048] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 20, Time since last communication: 0.064s
I20260709 12:25:15.780202 27008 raft_consensus.cc:2804] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:25:15.782707 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:15.789239 27008 raft_consensus.cc:493] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:15.789762 27008 raft_consensus.cc:3060] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:15.771371 27755 raft_consensus.cc:2804] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:25:15.708814 27856 raft_consensus.cc:493] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:15.792346 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 31 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:15.788549 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 33 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:15.769968 27753 raft_consensus.cc:697] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 27 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:25:15.808591 27753 consensus_queue.cc:237] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 26.19, Last appended by leader: 19, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:15.813086 27853 leader_election.cc:290] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:15.813830 27853 raft_consensus.cc:493] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:15.769984 27902 consensus_queue.cc:1048] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.033s
I20260709 12:25:15.814342 27853 raft_consensus.cc:515] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:15.710847 27871 consensus_queue.cc:1048] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.082s
I20260709 12:25:15.820698 27872 raft_consensus.cc:2804] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:25:15.821206 27872 raft_consensus.cc:493] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:15.821578 27872 raft_consensus.cc:3060] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 32 FOLLOWER]: Advancing to term 33
W20260709 12:25:15.822829 27884 consensus_meta.cc:287] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.525s	user 0.008s	sys 0.001s
I20260709 12:25:15.798127 27755 raft_consensus.cc:493] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:25:15.824007 27755 raft_consensus.cc:3060] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:15.827672 27951 leader_election.cc:290] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 26 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:15.828658 27853 leader_election.cc:290] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:15.828361 27856 raft_consensus.cc:515] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:15.830753 27856 leader_election.cc:290] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:15.831389 17683 raft_consensus.cc:2468] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 27.
I20260709 12:25:15.794509 27922 consensus_queue.cc:1048] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 15, Time since last communication: 0.038s
I20260709 12:25:15.764760 27778 consensus_queue.cc:237] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 30.18, Last appended by leader: 18, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:15.806885 27968 consensus_queue.cc:237] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 23.21, Last appended by leader: 21, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:15.839829 17801 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:15.860302 17801 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:15.873040 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 34 candidate_status { last_received { term: 33 index: 25 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:15.873937 18364 raft_consensus.cc:2393] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 34: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
W20260709 12:25:15.881630 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 389077us
I20260709 12:25:15.885253 17689 raft_consensus.cc:1275] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 26 index: 13. Preceding OpId from leader: term: 28 index: 14. (index mismatch)
I20260709 12:25:15.885509 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 29 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:15.886549 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 31 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:15.886541 17688 raft_consensus.cc:1275] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 23 index: 15. Preceding OpId from leader: term: 24 index: 16. (index mismatch)
I20260709 12:25:15.888257 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:15.884572 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:25:15.887245 18362 raft_consensus.cc:2468] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 30.
I20260709 12:25:15.835052 27725 raft_consensus.cc:515] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:15.893579 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:15.894379 27767 consensus_queue.cc:237] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 26.18, Last appended by leader: 18, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:15.897349 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:15.902174 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:15.901552 17729 leader_election.cc:304] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:25:15.822183 27903 raft_consensus.cc:515] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:15.906507 17728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:15.879765 17835 raft_consensus.cc:2468] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 29.
I20260709 12:25:15.921818 27929 raft_consensus.cc:515] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 34 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:15.922621 17761 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38786 (request call id 4223) took 1179 ms. Trace:
I20260709 12:25:15.923266 17755 raft_consensus.cc:2393] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 34: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:25:15.923174 17761 rpcz_store.cc:276] 0709 12:25:14.743488 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:15.326722 (+583234us) service_pool.cc:224] Handling call
0709 12:25:15.327058 (+   336us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:15.327417 (+   359us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:15.327457 (+    40us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:15.338056 (+ 10599us) log.cc:844] Serialized 26 byte log entry
0709 12:25:15.674300 (+336244us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:25:15.699297 (+ 24997us) log.cc:844] Serialized 14 byte log entry
0709 12:25:15.699699 (+   402us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:15.699798 (+    99us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:15.922378 (+222580us) raft_consensus.cc:1685] finished
0709 12:25:15.922403 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:15.922561 (+   158us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":17981,"prepare.run_wall_time_us":44,"raft.queue_time_us":5610,"raft.run_cpu_time_us":184,"raft.run_wall_time_us":5109,"spinlock_wait_cycles":43963648,"thread_start_us":335920,"threads_started":1,"wal-append.queue_time_us":582291}
I20260709 12:25:15.925500 27929 leader_election.cc:290] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 34 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:15.927929 27725 leader_election.cc:290] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 26 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:15.930291 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 34 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:15.930966 18363 raft_consensus.cc:3060] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:25:15.881608 27902 consensus_queue.cc:1059] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 21.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.042s
I20260709 12:25:15.934978 17685 raft_consensus.cc:1275] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 32 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 31 index: 18. Preceding OpId from leader: term: 32 index: 19. (index mismatch)
I20260709 12:25:15.936354 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 29 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:15.936934 17685 raft_consensus.cc:1275] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 23 index: 16. Preceding OpId from leader: term: 26 index: 17. (index mismatch)
I20260709 12:25:15.937788 17682 raft_consensus.cc:1275] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 23 index: 17. Preceding OpId from leader: term: 25 index: 18. (index mismatch)
I20260709 12:25:15.938258 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 32 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:15.939079 17684 raft_consensus.cc:2410] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for term 32 because replica has last-logged OpId of term: 29 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 15.
I20260709 12:25:15.953491 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 34 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:15.954557 17834 raft_consensus.cc:3060] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:25:15.956318 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:15.958226 17680 raft_consensus.cc:2468] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 30.
I20260709 12:25:15.957013 17682 raft_consensus.cc:2468] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 27.
I20260709 12:25:15.973682 27008 raft_consensus.cc:515] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:15.981422 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 34 candidate_status { last_received { term: 33 index: 25 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:15.982278 17688 raft_consensus.cc:2468] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 34.
I20260709 12:25:16.001044 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 25 candidate_status { last_received { term: 23 index: 21 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:16.007454 17684 raft_consensus.cc:1240] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 25. Current term is 26. Ops: []
I20260709 12:25:16.018239 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:16.030393 27008 leader_election.cc:290] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 25 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:16.031085 17834 raft_consensus.cc:2468] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 34.
I20260709 12:25:16.033109 17727 leader_election.cc:304] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:15.980060 27853 raft_consensus.cc:493] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:15.983450 27881 raft_consensus.cc:2749] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
W20260709 12:25:16.054790 18366 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.622s	user 0.009s	sys 0.020s
W20260709 12:25:16.055332 18366 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.623s	user 0.009s	sys 0.020s
I20260709 12:25:16.017381 27902 raft_consensus.cc:493] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:16.056021 17838 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.518s	user 0.017s	sys 0.012s
W20260709 12:25:16.056317 17762 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38786 (request call id 4199) took 3242 ms (client timeout 1674 ms). Trace:
I20260709 12:25:16.019641 27446 raft_consensus.cc:3055] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:25:16.057621 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 35 candidate_status { last_received { term: 33 index: 25 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:25:16.039444 17833 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.577s	user 0.021s	sys 0.035s
I20260709 12:25:16.055016 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
W20260709 12:25:16.071064 17833 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.609s	user 0.021s	sys 0.036s
W20260709 12:25:16.059561 17762 rpcz_store.cc:269] 0709 12:25:12.810709 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:12.811555 (+   846us) service_pool.cc:224] Handling call
0709 12:25:16.052755 (+3241200us) spinlock_profiling.cc:243] Waited 2.3 s on lock 0x7b540042c054. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:25:16.052878 (+   123us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5295320960}
I20260709 12:25:16.071143 18364 raft_consensus.cc:2468] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 34.
W20260709 12:25:16.060818 17838 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.524s	user 0.017s	sys 0.012s
I20260709 12:25:16.049889 18363 raft_consensus.cc:2468] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 34.
I20260709 12:25:16.072108 27446 raft_consensus.cc:740] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 21 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:25:16.084802 27446 consensus_queue.cc:260] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 21.15, Last appended by leader: 15, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:16.087455 27927 raft_consensus.cc:493] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:16.088191 27927 raft_consensus.cc:515] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:16.088680 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:16.090620 27927 leader_election.cc:290] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:16.087409 27446 raft_consensus.cc:3060] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:16.091861 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:16.092765 17834 raft_consensus.cc:2468] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 26.
I20260709 12:25:16.100648 18366 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 2912) took 1176 ms. Trace:
I20260709 12:25:16.108282 27577 consensus_queue.cc:1048] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20260709 12:25:16.113627 27725 raft_consensus.cc:493] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:16.114380 27725 raft_consensus.cc:515] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:16.107667 17728 leader_election.cc:304] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:16.120890 17728 leader_election.cc:304] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:25:16.122730 27898 raft_consensus.cc:2749] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:25:16.124519 17728 leader_election.cc:304] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:25:16.101432 18366 rpcz_store.cc:276] 0709 12:25:14.924041 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:14.952165 (+ 28124us) service_pool.cc:224] Handling call
0709 12:25:14.952602 (+   437us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:14.952982 (+   380us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:14.953010 (+    28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:14.953446 (+   436us) log.cc:844] Serialized 26 byte log entry
0709 12:25:15.134703 (+181257us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:25:15.188903 (+ 54200us) log.cc:844] Serialized 14 byte log entry
0709 12:25:15.432508 (+243605us) spinlock_profiling.cc:243] Waited 2.7 ms on lock 0x7b1800408ee0. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b47515e07 00007f7b503709a9 00007f7b4753269b 00007f7b4753093f 00007f7b47530455 00007f7b4a618060 00007f7b4a62a716 00007f7b4a63b287 00007f7b4a63b208
0709 12:25:16.055754 (+623246us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:16.055850 (+    96us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:16.055923 (+    73us) raft_consensus.cc:1685] finished
0709 12:25:16.094796 (+ 38873us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:16.100514 (+  5718us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":53951,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":42,"raft.queue_time_us":27108,"raft.run_cpu_time_us":214,"raft.run_wall_time_us":12243,"spinlock_wait_cycles":6208256,"thread_start_us":803887,"threads_started":2,"wal-append.queue_time_us":257864}
I20260709 12:25:16.134493 27577 raft_consensus.cc:2804] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:25:16.135237 27577 raft_consensus.cc:697] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 30 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:25:16.136363 27577 consensus_queue.cc:237] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 29.20, Last appended by leader: 20, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:16.142663 27927 raft_consensus.cc:493] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:16.143296 27927 raft_consensus.cc:515] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:16.146607 27927 leader_election.cc:290] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:16.148772 17578 catalog_manager.cc:5697] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 reported cstate change: term changed from 16 to 25, leader changed from e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66) to e063541542a94969898868bf6eb5ce1b (127.16.29.67). New cstate: current_term: 25 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } }
I20260709 12:25:16.151818 18334 leader_election.cc:304] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:25:16.151654 17578 catalog_manager.cc:5697] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 reported cstate change: term changed from 23 to 27. New cstate: current_term: 27 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } }
I20260709 12:25:16.153038 17578 catalog_manager.cc:5697] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 reported cstate change: term changed from 23 to 27, leader changed from e063541542a94969898868bf6eb5ce1b (127.16.29.67) to 0da169ff82674dc1947055ce43de68a3 (127.16.29.65). New cstate: current_term: 27 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: HEALTHY } } }
I20260709 12:25:16.142580 27725 leader_election.cc:290] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:16.157552 17728 leader_election.cc:304] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:25:16.159592 17756 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38776 (request call id 3411) took 1119 ms. Trace:
I20260709 12:25:16.174614 17756 rpcz_store.cc:276] 0709 12:25:15.039842 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:15.054317 (+ 14475us) service_pool.cc:224] Handling call
0709 12:25:16.159380 (+1105063us) spinlock_profiling.cc:243] Waited 710 ms on lock 0x7b5400415fd4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:25:16.159515 (+   135us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1632840064}
I20260709 12:25:16.018625 27929 consensus_queue.cc:1048] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.052s
I20260709 12:25:16.165194 17763 raft_consensus.cc:1240] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 28. Current term is 29. Ops: []
W20260709 12:25:16.169648 17831 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35040 (request call id 3766) took 1552 ms (client timeout 1577 ms). Trace:
W20260709 12:25:16.176397 17831 rpcz_store.cc:269] 0709 12:25:14.616857 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:14.618132 (+  1275us) service_pool.cc:224] Handling call
0709 12:25:16.169345 (+1551213us) spinlock_profiling.cc:243] Waited 802 ms on lock 0x7b54003340d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:25:16.169558 (+   213us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1843685632}
I20260709 12:25:16.034720 27744 raft_consensus.cc:493] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:16.171240 27008 raft_consensus.cc:493] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:16.186980 27008 raft_consensus.cc:515] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:16.187233 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 29 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:25:16.205570 17655 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:16.212810 17655 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:16.219492 17687 raft_consensus.cc:2468] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 28.
I20260709 12:25:16.224936 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 25 candidate_status { last_received { term: 23 index: 21 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:25:16.073616 27724 raft_consensus.cc:2804] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 34 FOLLOWER]: Leader election won for term 34
I20260709 12:25:16.225589 17688 raft_consensus.cc:3060] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:16.228516 27881 consensus_queue.cc:1048] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20260709 12:25:16.229904 27881 raft_consensus.cc:2804] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:25:16.230257 27881 raft_consensus.cc:493] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:16.230571 27881 raft_consensus.cc:3060] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Advancing to term 28
W20260709 12:25:16.241022 17655 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:16.242234 17688 raft_consensus.cc:2468] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 25.
I20260709 12:25:16.054816 27853 raft_consensus.cc:515] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:16.070134 27766 raft_consensus.cc:515] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 32 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:16.189126 27927 raft_consensus.cc:2804] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:25:16.292402 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 29 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:16.288100 17654 leader_election.cc:304] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
W20260709 12:25:16.252627 18365 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.641s	user 0.009s	sys 0.014s
W20260709 12:25:16.294422 18365 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.683s	user 0.010s	sys 0.015s
I20260709 12:25:16.250800 27008 leader_election.cc:290] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:16.254468 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:16.073616 27902 raft_consensus.cc:515] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:16.257580 17727 leader_election.cc:304] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:25:16.260123 17801 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:16.259858 17802 leader_election.cc:304] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
W20260709 12:25:16.305748 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 45997.8us
I20260709 12:25:16.306583 27008 raft_consensus.cc:2804] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:25:16.281292 17763 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38786 (request call id 4218) took 1555 ms. Trace:
W20260709 12:25:16.289821 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:16.306916 17802 leader_election.cc:304] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
W20260709 12:25:16.240507 27839 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.630s	user 0.005s	sys 0.018s
W20260709 12:25:16.256699 27713 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.739s	user 0.014s	sys 0.011s
I20260709 12:25:16.295540 18363 raft_consensus.cc:2468] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 26.
I20260709 12:25:16.115294 27965 consensus_queue.cc:1048] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.140s
I20260709 12:25:16.324843 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 29 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:16.251837 27724 raft_consensus.cc:697] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 34 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:25:16.200790 27744 raft_consensus.cc:515] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:16.314952 18374 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.617s	user 0.022s	sys 0.021s
I20260709 12:25:16.308391 17578 catalog_manager.cc:5697] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 reported cstate change: term changed from 27 to 31. New cstate: current_term: 31 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20260709 12:25:16.186269 27863 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.644s	user 0.011s	sys 0.033s
I20260709 12:25:16.307219 27008 raft_consensus.cc:493] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:16.347280 27008 raft_consensus.cc:3060] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:16.348014 17681 raft_consensus.cc:1240] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 32 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 31. Current term is 32. Ops: []
W20260709 12:25:16.348187 18374 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.650s	user 0.022s	sys 0.022s
W20260709 12:25:16.314986 27713 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.801s	user 0.015s	sys 0.012s
I20260709 12:25:16.301357 27927 raft_consensus.cc:493] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 33 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
W20260709 12:25:16.317654 27839 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.743s	user 0.006s	sys 0.018s
W20260709 12:25:16.361981 17829 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.646s	user 0.014s	sys 0.026s
W20260709 12:25:16.362358 17829 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.647s	user 0.015s	sys 0.026s
I20260709 12:25:16.248541 27972 consensus_queue.cc:1048] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.112s
W20260709 12:25:16.349713 17759 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.838s	user 0.011s	sys 0.038s
W20260709 12:25:16.366932 17759 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.864s	user 0.011s	sys 0.038s
I20260709 12:25:16.248682 27874 raft_consensus.cc:515] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:16.357124 27863 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.818s	user 0.012s	sys 0.033s
I20260709 12:25:16.308257 17763 rpcz_store.cc:276] 0709 12:25:14.725404 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:15.220943 (+495539us) service_pool.cc:224] Handling call
0709 12:25:16.164961 (+944018us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:16.185240 (+ 20279us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:16.280932 (+ 95692us) spinlock_profiling.cc:243] Waited 944 ms on lock 0x7b5400415fd0. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a61f8ad 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:25:16.281203 (+   271us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2170237312}
W20260709 12:25:16.309746 27570 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.767s	user 0.008s	sys 0.044s
I20260709 12:25:16.332051 27860 raft_consensus.cc:2804] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:25:16.359424 17578 catalog_manager.cc:5697] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 reported cstate change: term changed from 23 to 25, leader changed from 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) to e063541542a94969898868bf6eb5ce1b (127.16.29.67). New cstate: current_term: 25 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:25:16.345109 27892 raft_consensus.cc:493] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:16.344146 27871 raft_consensus.cc:2804] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:25:16.376277 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:16.379989 18363 raft_consensus.cc:3060] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Advancing to term 26
W20260709 12:25:16.385886 17654 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:16.409056 27724 consensus_queue.cc:237] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 32.22, Last appended by leader: 22, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:16.360657 27713 raft_consensus.cc:697] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 28 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:25:16.412814 27713 consensus_queue.cc:237] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 26.19, Last appended by leader: 19, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:16.369037 27692 consensus_meta.cc:287] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.726s	user 0.009s	sys 0.001s
I20260709 12:25:16.416771 27766 leader_election.cc:290] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 32 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:16.419101 27874 leader_election.cc:290] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 28 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:16.419982 27871 raft_consensus.cc:697] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 25 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:25:16.420874 27871 consensus_queue.cc:237] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 23.21, Last appended by leader: 21, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:16.424821 27902 leader_election.cc:290] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:25:16.368958 27872 consensus_meta.cc:287] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.547s	user 0.010s	sys 0.000s
I20260709 12:25:16.425549 27872 raft_consensus.cc:515] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:16.427433 27872 leader_election.cc:290] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 33 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:25:16.437141 27853 leader_election.cc:290] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:25:16.377236 27570 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.834s	user 0.008s	sys 0.044s
I20260709 12:25:16.377447 27860 raft_consensus.cc:493] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 34 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:16.446532 27860 raft_consensus.cc:3060] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:25:16.364477 27927 raft_consensus.cc:3060] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:25:16.367136 27703 raft_consensus.cc:2804] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:25:16.456038 27703 raft_consensus.cc:493] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:16.456580 27703 raft_consensus.cc:3060] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:25:16.435633 27755 consensus_meta.cc:287] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.611s	user 0.003s	sys 0.006s
I20260709 12:25:16.481141 17578 catalog_manager.cc:5697] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 reported cstate change: term changed from 14 to 29, leader changed from e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66) to e063541542a94969898868bf6eb5ce1b (127.16.29.67). New cstate: current_term: 29 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } }
W20260709 12:25:16.471287 27449 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.811s	user 0.019s	sys 0.027s
I20260709 12:25:16.480265 27431 raft_consensus.cc:493] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:16.484802 27449 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.824s	user 0.019s	sys 0.028s
I20260709 12:25:16.489390 27008 raft_consensus.cc:515] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:16.493252 17801 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:16.380374 27671 raft_consensus.cc:493] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:16.461989 27338 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.736s	user 0.013s	sys 0.032s
W20260709 12:25:16.499409 17836 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.998s	user 0.015s	sys 0.036s
W20260709 12:25:16.499963 17836 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.999s	user 0.015s	sys 0.036s
I20260709 12:25:16.500762 17578 catalog_manager.cc:5697] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 reported cstate change: term changed from 16 to 27, leader changed from e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66) to e063541542a94969898868bf6eb5ce1b (127.16.29.67). New cstate: current_term: 27 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } }
W20260709 12:25:16.503531 17652 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.006s	user 0.011s	sys 0.039s
W20260709 12:25:16.503923 17652 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.006s	user 0.011s	sys 0.039s
W20260709 12:25:16.504386 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 1.02097e+06us
W20260709 12:25:16.509074 17686 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.783s	user 0.018s	sys 0.030s
W20260709 12:25:16.509452 17686 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.784s	user 0.018s	sys 0.030s
I20260709 12:25:16.517103 18363 raft_consensus.cc:2468] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 26.
I20260709 12:25:16.525882 27008 leader_election.cc:290] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 28 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:25:16.504662 27338 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.778s	user 0.013s	sys 0.033s
I20260709 12:25:16.462312 27863 consensus_queue.cc:1059] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 25.17, Next index: 18, Last known committed idx: 16, Time since last communication: 0.042s
W20260709 12:25:16.524241 17830 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.762s	user 0.015s	sys 0.032s
I20260709 12:25:16.523042 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:16.461892 27892 raft_consensus.cc:515] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:16.536164 17830 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.774s	user 0.016s	sys 0.033s
I20260709 12:25:16.533543 17578 catalog_manager.cc:5697] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 reported cstate change: term changed from 22 to 26, leader changed from <none> to 334a4e1f59944544b3b3ff36616b57da (127.16.29.68). New cstate: current_term: 26 leader_uuid: "334a4e1f59944544b3b3ff36616b57da" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:25:16.536553 18363 raft_consensus.cc:2468] T a5732e71b9d54f8786b8c511d69943bf P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 28.
I20260709 12:25:16.545401 18363 raft_consensus.cc:1275] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 26 index: 18. Preceding OpId from leader: term: 28 index: 19. (index mismatch)
I20260709 12:25:16.564567 17833 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35042 (request call id 3578) took 1160 ms. Trace:
I20260709 12:25:16.566363 27853 raft_consensus.cc:3055] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:25:16.565025 17833 rpcz_store.cc:276] 0709 12:25:15.404568 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:15.404754 (+   186us) service_pool.cc:224] Handling call
0709 12:25:15.425296 (+ 20542us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:15.461420 (+ 36124us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:15.461452 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:15.461891 (+   439us) log.cc:844] Serialized 26 byte log entry
0709 12:25:16.071509 (+609618us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:25:16.071550 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:16.071618 (+    68us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:16.564321 (+492703us) raft_consensus.cc:1685] finished
0709 12:25:16.564344 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:16.564505 (+   161us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":54314,"raft.queue_time_us":36567,"raft.run_cpu_time_us":83,"raft.run_wall_time_us":82,"spinlock_wait_cycles":14592,"thread_start_us":609211,"threads_started":1,"wal-append.queue_time_us":1099519}
I20260709 12:25:16.566802 27853 raft_consensus.cc:740] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:25:16.523563 27976 consensus_queue.cc:1048] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.045s
I20260709 12:25:16.563515 27744 leader_election.cc:290] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:16.571115 27903 leader_election.cc:290] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 27 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:16.484998 27431 raft_consensus.cc:515] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:16.573572 27431 leader_election.cc:290] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:16.523494 27570 raft_consensus.cc:493] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:16.574317 27570 raft_consensus.cc:515] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:16.576577 27570 leader_election.cc:290] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:16.547191 27978 raft_consensus.cc:493] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:16.577666 27978 raft_consensus.cc:515] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:16.578846 17838 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35040 (request call id 3780) took 1182 ms. Trace:
I20260709 12:25:16.579221 17838 rpcz_store.cc:276] 0709 12:25:15.396234 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:15.396383 (+   149us) service_pool.cc:224] Handling call
0709 12:25:15.396725 (+   342us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:15.397222 (+   497us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:15.397264 (+    42us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:15.494590 (+ 97326us) log.cc:844] Serialized 26 byte log entry
0709 12:25:16.072157 (+577567us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:25:16.072208 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:16.072298 (+    90us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:16.578642 (+506344us) raft_consensus.cc:1685] finished
0709 12:25:16.578663 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:16.578790 (+   127us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":11236,"raft.queue_time_us":112,"raft.run_cpu_time_us":92,"raft.run_wall_time_us":92,"thread_start_us":535256,"threads_started":1,"wal-append.queue_time_us":1018633}
I20260709 12:25:16.487295 27755 raft_consensus.cc:515] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:16.548520 17652 leader_election.cc:304] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:16.556063 27892 leader_election.cc:290] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:16.560431 27703 raft_consensus.cc:515] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:16.561060 27980 log.cc:927] Time spent T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.067s	user 0.000s	sys 0.001s
I20260709 12:25:16.504004 27671 raft_consensus.cc:515] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:25:16.609856 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 61444us
I20260709 12:25:16.612186 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 25 candidate_status { last_received { term: 23 index: 21 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:16.615792 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:16.616860 17834 raft_consensus.cc:2393] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 28: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:25:16.591001 17578 catalog_manager.cc:5697] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 reported cstate change: term changed from 23 to 26, leader changed from e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66) to 334a4e1f59944544b3b3ff36616b57da (127.16.29.68). New cstate: current_term: 26 leader_uuid: "334a4e1f59944544b3b3ff36616b57da" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
W20260709 12:25:16.594229 17837 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.796s	user 0.025s	sys 0.027s
W20260709 12:25:16.622733 17837 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.833s	user 0.025s	sys 0.027s
I20260709 12:25:16.626554 17833 raft_consensus.cc:1275] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 26 index: 18. Preceding OpId from leader: term: 28 index: 19. (index mismatch)
I20260709 12:25:16.529407 27766 consensus_queue.cc:1059] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 29.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.062s
I20260709 12:25:16.630406 17837 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35040 (request call id 3779) took 1254 ms. Trace:
W20260709 12:25:16.631307 17832 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.755s	user 0.026s	sys 0.028s
W20260709 12:25:16.632305 17832 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.757s	user 0.026s	sys 0.028s
W20260709 12:25:16.637739 17757 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.867s	user 0.012s	sys 0.033s
W20260709 12:25:16.638302 17757 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.868s	user 0.012s	sys 0.033s
I20260709 12:25:16.641413 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:16.642169 17831 raft_consensus.cc:3060] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Advancing to term 26
W20260709 12:25:16.656028 18361 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.821s	user 0.011s	sys 0.037s
W20260709 12:25:16.657665 18361 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.822s	user 0.012s	sys 0.037s
I20260709 12:25:16.658501 17578 catalog_manager.cc:5697] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 reported cstate change: term changed from 26 to 31. New cstate: current_term: 31 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20260709 12:25:16.663990 18368 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.736s	user 0.005s	sys 0.025s
W20260709 12:25:16.668761 18368 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.741s	user 0.005s	sys 0.025s
I20260709 12:25:16.586336 27881 raft_consensus.cc:515] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:16.628002 27570 raft_consensus.cc:2804] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:25:16.700160 17578 catalog_manager.cc:5697] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 reported cstate change: term changed from 27 to 28, leader changed from 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) to 0da169ff82674dc1947055ce43de68a3 (127.16.29.65). New cstate: current_term: 28 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 12:25:16.639184 27778 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.711s	user 0.005s	sys 0.018s
I20260709 12:25:16.686661 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:16.686160 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:16.656291 27766 consensus_queue.cc:1243] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer e2909cff1da44ad387f96a8ae3aa5afb log is divergent from this leader: its last log entry 30.20 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 19
W20260709 12:25:16.705441 17758 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.838s	user 0.019s	sys 0.012s
W20260709 12:25:16.657058 27753 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.845s	user 0.012s	sys 0.051s
I20260709 12:25:16.656617 27800 raft_consensus.cc:493] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:16.657156 27525 raft_consensus.cc:2804] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Leader pre-election won for term 31
W20260709 12:25:16.704365 18371 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.787s	user 0.013s	sys 0.017s
I20260709 12:25:16.704680 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 29 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:25:16.789928 18371 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.873s	user 0.014s	sys 0.017s
I20260709 12:25:16.567749 27853 consensus_queue.cc:260] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 25.17, Last appended by leader: 17, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:16.802906 17578 catalog_manager.cc:5697] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 reported cstate change: term changed from 21 to 23. New cstate: current_term: 23 leader_uuid: "334a4e1f59944544b3b3ff36616b57da" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
W20260709 12:25:16.705070 17760 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.885s	user 0.019s	sys 0.018s
I20260709 12:25:16.691831 17831 raft_consensus.cc:2468] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 26.
W20260709 12:25:16.806015 17760 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.985s	user 0.019s	sys 0.018s
W20260709 12:25:16.706246 17758 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.867s	user 0.020s	sys 0.013s
W20260709 12:25:16.699043 18333 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.965s	user 0.011s	sys 0.053s
W20260709 12:25:16.695633 27980 log.cc:927] Time spent T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.059s	user 0.000s	sys 0.000s
I20260709 12:25:16.631655 17837 rpcz_store.cc:276] 0709 12:25:15.376445 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:15.376572 (+   127us) service_pool.cc:224] Handling call
0709 12:25:15.376835 (+   263us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:15.377024 (+   189us) raft_consensus.cc:1534] Early marking committed up to index 17
0709 12:25:15.789686 (+412662us) log.cc:844] Serialized 14 byte log entry
0709 12:25:16.629709 (+840023us) log.cc:844] Serialized 14 byte log entry
0709 12:25:16.629990 (+   281us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:16.630051 (+    61us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:25:16.630129 (+    78us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:16.630211 (+    82us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:16.630343 (+   132us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":196523,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":38,"raft.queue_time_us":361,"raft.run_cpu_time_us":124,"raft.run_wall_time_us":6805,"thread_start_us":1152112,"threads_started":2}
I20260709 12:25:16.699276 27978 leader_election.cc:290] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:25:16.709698 27753 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.898s	user 0.012s	sys 0.051s
W20260709 12:25:16.709393 27778 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.781s	user 0.005s	sys 0.019s
W20260709 12:25:16.784233 17652 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:16.808729 18333 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.075s	user 0.011s	sys 0.053s
W20260709 12:25:16.824123 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 1.11794e+06us
I20260709 12:25:16.823436 17578 catalog_manager.cc:5697] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 reported cstate change: term changed from 24 to 25, leader changed from 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) to e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66). New cstate: current_term: 25 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:25:16.825249 18333 leader_election.cc:304] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3
W20260709 12:25:16.823545 17652 connection.cc:256] RPC call timeout handler was delayed by -1.001823431000048s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1577s for a response
I20260709 12:25:16.829145 27766 raft_consensus.cc:493] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:25:16.829674 27766 raft_consensus.cc:515] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:16.829947 27839 consensus_queue.cc:1048] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20260709 12:25:16.831797 27766 leader_election.cc:290] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:16.833694 17652 leader_election.cc:304] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 33 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:16.834810 27839 raft_consensus.cc:2749] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 33 FOLLOWER]: Leader election lost for term 33. Reason: could not achieve majority
I20260709 12:25:16.837399 17652 leader_election.cc:304] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:25:16.742136 17801 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:16.838874 27431 raft_consensus.cc:493] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 32 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:25:16.839480 27431 raft_consensus.cc:515] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:25:16.840646 27983 log.cc:927] Time spent T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.052s	user 0.000s	sys 0.000s
I20260709 12:25:16.841602 27431 leader_election.cc:290] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:25:16.728161 17833 raft_consensus.cc:1240] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 33 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 32. Current term is 33. Ops: []
W20260709 12:25:16.742448 17655 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:16.812340 17758 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38776 (request call id 3410) took 1773 ms. Trace:
I20260709 12:25:16.844218 17758 rpcz_store.cc:276] 0709 12:25:15.039486 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:15.386511 (+347025us) service_pool.cc:224] Handling call
0709 12:25:15.386787 (+   276us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:15.386988 (+   201us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 12:25:15.472803 (+ 85815us) log.cc:844] Serialized 14 byte log entry
0709 12:25:15.839594 (+366791us) spinlock_profiling.cc:243] Waited 13.3 ms on lock 0x7b18000e7760. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b47515e07 00007f7b503709a9 00007f7b4753269b 00007f7b4753093f 00007f7b47532c02 00007f7b4a1ba20b 00007f7b4a1bfd23 00007f7b4a1c023a 00007f7b4a62aa2a
0709 12:25:16.808189 (+968595us) log.cc:844] Serialized 14 byte log entry
0709 12:25:16.811800 (+  3611us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:16.811873 (+    73us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:25:16.811968 (+    95us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:16.812090 (+   122us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:16.812262 (+   172us) inbound_call.cc:177] Queueing success response
0709 12:25:16.826638 (+ 14376us) spinlock_profiling.cc:243] Waited 3.57 ms on lock 0x7b18000e7760. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b475333f3 00007f7b475357c1 00007f7b47535779 00007f7b47535709 00007f7b475356d1 00007f7b475349cd 0000000000388634 0000000000388479 00007f7b4751683a
Metrics: {"mutex_wait_us":3329,"prepare.queue_time_us":85827,"prepare.run_cpu_time_us":53,"prepare.run_wall_time_us":2072,"raft.queue_time_us":6013,"raft.run_cpu_time_us":83,"raft.run_wall_time_us":83,"spinlock_wait_cycles":48849152,"thread_start_us":967771,"threads_started":1}
I20260709 12:25:16.784447 27570 raft_consensus.cc:697] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 29 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
W20260709 12:25:16.898918 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:16.874253 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 49075.3us
W20260709 12:25:16.882362 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 45051.8us
I20260709 12:25:16.916997 27755 leader_election.cc:290] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 29 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:25:16.896423 17801 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:16.814277 27898 raft_consensus.cc:2749] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
W20260709 12:25:16.871402 17803 leader_election.cc:336] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 26 election: RPC error from VoteRequest() call to peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Timed out: RequestConsensusVote RPC to 127.16.29.68:42577 timed out after 1.648s (SENT)
I20260709 12:25:16.704169 27927 raft_consensus.cc:515] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 34 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:16.783617 27800 raft_consensus.cc:515] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:16.832465 27985 log.cc:927] Time spent T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.134s	user 0.000s	sys 0.000s
I20260709 12:25:16.945811 18333 leader_election.cc:304] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:16.942960 17803 leader_election.cc:304] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:25:16.949141 17680 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.792s	user 0.008s	sys 0.021s
W20260709 12:25:16.954805 17680 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.798s	user 0.009s	sys 0.021s
I20260709 12:25:16.956643 18333 leader_election.cc:304] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
W20260709 12:25:16.953749 17728 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.733s	user 0.019s	sys 0.040s
W20260709 12:25:16.957895 17728 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.737s	user 0.019s	sys 0.040s
I20260709 12:25:16.958642 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:16.959615 18363 raft_consensus.cc:2393] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 26: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:25:16.962244 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 30 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:25:16.909788 27577 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.769s	user 0.003s	sys 0.024s
I20260709 12:25:16.913928 27853 raft_consensus.cc:3060] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:16.915068 27431 raft_consensus.cc:493] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:25:16.964766 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 29 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:25:16.966646 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 26 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:16.968146 17682 raft_consensus.cc:2393] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 30: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:25:17.007676 18363 raft_consensus.cc:3060] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Advancing to term 26
W20260709 12:25:16.968676 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 97329.5us
I20260709 12:25:16.913523 27989 raft_consensus.cc:2804] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:25:16.941215 17578 catalog_manager.cc:5697] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 reported cstate change: config changed from index 12 to 16, term changed from 25 to 27, leader changed from 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) to e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) changed from NON_VOTER to VOTER. New cstate: current_term: 27 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:25:17.012863 17578 catalog_manager.cc:5697] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 reported cstate change: term changed from 19 to 24, leader changed from 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) to e063541542a94969898868bf6eb5ce1b (127.16.29.67). New cstate: current_term: 24 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:25:17.018509 17728 leader_election.cc:304] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:25:16.965273 27755 raft_consensus.cc:493] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:16.965590 27973 raft_consensus.cc:2749] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:25:17.035792 18365 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59886 (request call id 2719) took 1425 ms. Trace:
I20260709 12:25:17.036293 18365 rpcz_store.cc:276] 0709 12:25:15.610016 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:15.610166 (+   150us) service_pool.cc:224] Handling call
0709 12:25:15.610638 (+   472us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:15.611029 (+   391us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:15.611070 (+    41us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:15.611636 (+   566us) log.cc:844] Serialized 26 byte log entry
0709 12:25:16.294864 (+683228us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:25:16.294901 (+    37us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:16.294960 (+    59us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:17.035537 (+740577us) raft_consensus.cc:1685] finished
0709 12:25:17.035566 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:17.035725 (+   159us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1148,"raft.run_cpu_time_us":112,"raft.run_wall_time_us":1850,"thread_start_us":682932,"threads_started":1,"wal-append.queue_time_us":1016380}
I20260709 12:25:16.965392 27903 raft_consensus.cc:493] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:17.023331 17803 leader_election.cc:336] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 26 election: RPC error from VoteRequest() call to peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Timed out: RequestConsensusVote RPC to 127.16.29.68:42577 timed out after 1.544s (SENT)
I20260709 12:25:17.404887 17803 leader_election.cc:304] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:17.405762 17832 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35042 (request call id 3588) took 1532 ms. Trace:
W20260709 12:25:17.380957 17683 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.024s	user 0.000s	sys 0.014s
W20260709 12:25:17.381733 17684 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.051s	user 0.012s	sys 0.017s
W20260709 12:25:17.408239 17683 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.052s	user 0.000s	sys 0.015s
W20260709 12:25:17.408347 17684 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.078s	user 0.012s	sys 0.018s
I20260709 12:25:17.382867 17578 catalog_manager.cc:5697] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 reported cstate change: term changed from 21 to 23, leader changed from 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) to e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66). New cstate: current_term: 23 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:25:17.410074 17684 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44898 (request call id 2010) took 1170 ms. Trace:
I20260709 12:25:17.410228 17578 catalog_manager.cc:5697] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 reported cstate change: term changed from 17 to 21, leader changed from 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) to 334a4e1f59944544b3b3ff36616b57da (127.16.29.68). New cstate: current_term: 21 leader_uuid: "334a4e1f59944544b3b3ff36616b57da" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:25:17.410533 17684 rpcz_store.cc:276] 0709 12:25:16.239805 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:16.239955 (+   150us) service_pool.cc:224] Handling call
0709 12:25:16.240265 (+   310us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:16.250772 (+ 10507us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:16.250821 (+    49us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:16.251472 (+   651us) log.cc:844] Serialized 26 byte log entry
0709 12:25:17.409296 (+1157824us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:25:17.409418 (+   122us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:17.409601 (+   183us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:17.409680 (+    79us) raft_consensus.cc:1685] finished
0709 12:25:17.409701 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:17.409995 (+   294us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":6562,"raft.queue_time_us":95,"raft.run_cpu_time_us":143,"raft.run_wall_time_us":36474,"thread_start_us":1078057,"threads_started":1,"wal-append.queue_time_us":1001278}
W20260709 12:25:17.147683 17689 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.773s	user 0.015s	sys 0.021s
W20260709 12:25:17.238260 17730 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.971s	user 0.014s	sys 0.015s
W20260709 12:25:17.241659 17759 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.863s	user 0.011s	sys 0.011s
W20260709 12:25:17.296708 17801 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:17.072930 17727 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.769s	user 0.013s	sys 0.029s
W20260709 12:25:17.130848 17802 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.806s	user 0.013s	sys 0.021s
I20260709 12:25:17.406399 17832 rpcz_store.cc:276] 0709 12:25:15.873088 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:15.874028 (+   940us) service_pool.cc:224] Handling call
0709 12:25:15.874428 (+   400us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:15.874868 (+   440us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:15.874907 (+    39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:15.875429 (+   522us) log.cc:844] Serialized 26 byte log entry
0709 12:25:16.633164 (+757735us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:25:16.633226 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:16.633323 (+    97us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:17.404918 (+771595us) raft_consensus.cc:1685] finished
0709 12:25:17.404971 (+    53us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:17.405406 (+   435us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1315,"raft.run_cpu_time_us":105,"raft.run_wall_time_us":106,"thread_start_us":757248,"threads_started":1,"wal-append.queue_time_us":1447690}
I20260709 12:25:17.383697 17680 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37396 (request call id 3521) took 1228 ms. Trace:
W20260709 12:25:17.412186 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 388881us
W20260709 12:25:17.238961 18335 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.879s	user 0.007s	sys 0.023s
W20260709 12:25:17.388795 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:17.099606 18364 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.771s	user 0.012s	sys 0.019s
W20260709 12:25:17.335577 18336 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.875s	user 0.010s	sys 0.031s
W20260709 12:25:17.144022 18362 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.646s	user 0.005s	sys 0.025s
I20260709 12:25:17.064200 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 26 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
W20260709 12:25:17.238669 18366 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.880s	user 0.007s	sys 0.023s
W20260709 12:25:17.145583 18374 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.649s	user 0.010s	sys 0.020s
I20260709 12:25:17.031265 18363 raft_consensus.cc:2410] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 26 because replica has last-logged OpId of term: 25 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 17.
W20260709 12:25:16.975291 27577 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.835s	user 0.003s	sys 0.024s
W20260709 12:25:17.246362 17836 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.737s	user 0.019s	sys 0.019s
W20260709 12:25:17.250722 27809 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.145s	user 0.006s	sys 0.032s
I20260709 12:25:17.295145 27881 leader_election.cc:290] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 28 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:16.973049 27874 raft_consensus.cc:2749] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:25:17.308198 27898 raft_consensus.cc:2804] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:25:17.240962 27903 raft_consensus.cc:515] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:17.337682 27927 leader_election.cc:290] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 34 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:17.024477 27431 raft_consensus.cc:515] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:17.015439 27892 raft_consensus.cc:493] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:25:17.276192 27525 raft_consensus.cc:493] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:25:17.070807 27570 consensus_queue.cc:237] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 28.16, Last appended by leader: 16, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:17.041571 27671 leader_election.cc:290] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:25:17.136415 18367 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.768s	user 0.031s	sys 0.026s
W20260709 12:25:17.478899 18367 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.116s	user 0.033s	sys 0.044s
I20260709 12:25:17.479230 16501 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:25:17.420694 17680 rpcz_store.cc:276] 0709 12:25:16.155546 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:16.155693 (+   147us) service_pool.cc:224] Handling call
0709 12:25:16.156021 (+   328us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:16.156386 (+   365us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:16.156413 (+    27us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:16.156859 (+   446us) log.cc:844] Serialized 26 byte log entry
0709 12:25:16.955291 (+798432us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:25:16.955345 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:16.955435 (+    90us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:17.382882 (+427447us) raft_consensus.cc:1685] finished
0709 12:25:17.382964 (+    82us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:17.383406 (+   442us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1087,"raft.run_cpu_time_us":78,"raft.run_wall_time_us":80,"spinlock_wait_cycles":36608,"thread_start_us":798001,"threads_started":1,"wal-append.queue_time_us":1181289}
I20260709 12:25:17.480571 27431 leader_election.cc:290] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:17.481839 27525 raft_consensus.cc:3060] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:17.482426 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:17.065533 27703 leader_election.cc:290] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 27 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:25:17.212663 27713 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.661s	user 0.015s	sys 0.054s
W20260709 12:25:17.488780 27713 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.954s	user 0.016s	sys 0.055s
W20260709 12:25:17.245858 27724 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.666s	user 0.001s	sys 0.006s
I20260709 12:25:17.500495 17683 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37402 (request call id 4011) took 1329 ms. Trace:
I20260709 12:25:17.032213 27725 consensus_queue.cc:1048] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 19, Time since last communication: 0.047s
I20260709 12:25:17.040544 27755 raft_consensus.cc:515] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:17.505286 17763 raft_consensus.cc:1275] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 21 index: 15. Preceding OpId from leader: term: 25 index: 17. (index mismatch)
I20260709 12:25:17.505930 17755 raft_consensus.cc:1275] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 25 index: 17. Preceding OpId from leader: term: 26 index: 17. (term mismatch)
I20260709 12:25:17.506358 27755 leader_election.cc:290] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:25:17.241508 27767 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.883s	user 0.009s	sys 0.013s
I20260709 12:25:17.506376 17755 pending_rounds.cc:85] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb: Aborting all ops after (but not including) 16
W20260709 12:25:17.507457 27767 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.157s	user 0.009s	sys 0.014s
I20260709 12:25:17.507475 17755 pending_rounds.cc:107] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb: Aborting uncommitted NO_OP operation due to leader change: 25.17
W20260709 12:25:17.501416 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 482879us
I20260709 12:25:17.508098 17755 raft_consensus.cc:2889] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:17.509845 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 30 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:17.511150 17756 raft_consensus.cc:2468] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 29.
I20260709 12:25:17.330474 27800 leader_election.cc:290] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:17.240307 27986 raft_consensus.cc:3055] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 31 LEADER]: Stepping down as leader of term 31
I20260709 12:25:17.513001 27986 raft_consensus.cc:740] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 31 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:25:17.287442 28010 raft_consensus.cc:493] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:17.516731 28010 raft_consensus.cc:515] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:17.250574 27860 consensus_meta.cc:287] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.803s	user 0.013s	sys 0.005s
I20260709 12:25:17.482510 27431 raft_consensus.cc:493] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:17.521001 27860 raft_consensus.cc:515] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 35 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:17.116361 27994 log.cc:927] Time spent T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.076s	user 0.000s	sys 0.001s
W20260709 12:25:17.072559 27991 log.cc:927] Time spent T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.123s	user 0.000s	sys 0.000s
I20260709 12:25:17.526081 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:17.526672 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 30 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:25:17.068370 27990 log.cc:927] Time spent T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.148s	user 0.000s	sys 0.001s
I20260709 12:25:17.528644 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 29 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:17.529816 17755 raft_consensus.cc:2393] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 29: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
W20260709 12:25:17.126080 27984 log.cc:927] Time spent T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.305s	user 0.000s	sys 0.001s
I20260709 12:25:17.532761 17829 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35042 (request call id 3582) took 1913 ms. Trace:
I20260709 12:25:17.533214 17829 rpcz_store.cc:276] 0709 12:25:15.619373 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:15.714227 (+ 94854us) service_pool.cc:224] Handling call
0709 12:25:15.714580 (+   353us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:15.714886 (+   306us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:15.714913 (+    27us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:15.715344 (+   431us) log.cc:844] Serialized 26 byte log entry
0709 12:25:16.362727 (+647383us) raft_consensus.cc:1645] Marking committed up to 22
0709 12:25:16.362766 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:16.362836 (+    70us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:17.532255 (+1169419us) raft_consensus.cc:1685] finished
0709 12:25:17.532295 (+    40us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:17.532562 (+   267us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1536,"raft.run_cpu_time_us":79,"raft.run_wall_time_us":79,"thread_start_us":647052,"threads_started":1,"wal-append.queue_time_us":990429}
I20260709 12:25:17.343075 28013 raft_consensus.cc:493] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:25:17.535584 28013 raft_consensus.cc:515] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:17.538838 28013 leader_election.cc:290] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:25:17.246217 27968 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.060s	user 0.010s	sys 0.028s
W20260709 12:25:17.540401 27968 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.375s	user 0.011s	sys 0.029s
I20260709 12:25:16.970461 27965 raft_consensus.cc:2804] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:25:17.542474 27965 raft_consensus.cc:697] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 26 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:25:17.331737 18368 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59886 (request call id 2723) took 1496 ms. Trace:
I20260709 12:25:17.294929 27929 raft_consensus.cc:493] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:17.544250 18368 rpcz_store.cc:276] 0709 12:25:15.841659 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:15.842093 (+   434us) service_pool.cc:224] Handling call
0709 12:25:15.842411 (+   318us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:15.842715 (+   304us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:15.842743 (+    28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:15.881551 (+ 38808us) log.cc:844] Serialized 26 byte log entry
0709 12:25:16.669312 (+787761us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:25:16.669367 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:16.669457 (+    90us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:17.331490 (+662033us) raft_consensus.cc:1685] finished
0709 12:25:17.331510 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:17.331676 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":25431,"raft.queue_time_us":238162,"raft.run_cpu_time_us":154,"raft.run_wall_time_us":104350,"thread_start_us":741136,"threads_started":1,"wal-append.queue_time_us":1419814}
W20260709 12:25:17.146494 27871 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.723s	user 0.033s	sys 0.023s
W20260709 12:25:17.343956 27995 log.cc:927] Time spent T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.094s	user 0.002s	sys 0.000s
I20260709 12:25:17.554570 17686 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37402 (request call id 4005) took 1847 ms. Trace:
I20260709 12:25:17.413605 17578 catalog_manager.cc:5697] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 reported cstate change: term changed from 24 to 26. New cstate: current_term: 26 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:25:17.555096 17686 rpcz_store.cc:276] 0709 12:25:15.706758 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:15.707660 (+   902us) service_pool.cc:224] Handling call
0709 12:25:15.724332 (+ 16672us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:15.724707 (+   375us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:15.724748 (+    41us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:15.725292 (+   544us) log.cc:844] Serialized 26 byte log entry
0709 12:25:16.509849 (+784557us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:25:16.509896 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:16.509967 (+    71us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:17.554063 (+1044096us) raft_consensus.cc:1685] finished
0709 12:25:17.554129 (+    66us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:17.554399 (+   270us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":25374,"raft.run_cpu_time_us":89,"raft.run_wall_time_us":90,"thread_start_us":784224,"threads_started":1,"wal-append.queue_time_us":1095354}
I20260709 12:25:17.556504 17757 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38786 (request call id 4228) took 2089 ms. Trace:
W20260709 12:25:17.419031 17801 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:17.420257 17802 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.096s	user 0.014s	sys 0.021s
I20260709 12:25:17.557180 17578 catalog_manager.cc:5697] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 reported cstate change: term changed from 17 to 21. New cstate: current_term: 21 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:25:17.556926 17757 rpcz_store.cc:276] 0709 12:25:15.466905 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:15.582825 (+115920us) service_pool.cc:224] Handling call
0709 12:25:15.583221 (+   396us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:15.754197 (+170976us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:15.754244 (+    47us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:15.770230 (+ 15986us) log.cc:844] Serialized 26 byte log entry
0709 12:25:16.638791 (+868561us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:25:16.638846 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:16.815474 (+176628us) spinlock_profiling.cc:243] Waited 171 ms on lock 0x7b54002b5a54. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:25:16.815542 (+    68us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:17.337956 (+522414us) spinlock_profiling.cc:243] Waited 6.11 ms on lock 0x7b4400022b4c. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b47459c85 00007f7b4745a6a2 00007f7b4a1cc1db 00007f7b4a1c7aaf 00007f7b4a1bb86e 00007f7b4a1babb6 00007f7b4a1ba7d8 00007f7b4a1c4221 00007f7b4a1c41d9 00007f7b4a1c4169
0709 12:25:17.556015 (+218059us) raft_consensus.cc:1685] finished
0709 12:25:17.556059 (+    44us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:17.556316 (+   257us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1571,"raft.run_cpu_time_us":77,"raft.run_wall_time_us":77,"spinlock_wait_cycles":406216320,"thread_start_us":868215,"threads_started":1,"wal-append.queue_time_us":1297095}
W20260709 12:25:17.417213 17730 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.150s	user 0.014s	sys 0.016s
I20260709 12:25:17.561419 27800 raft_consensus.cc:493] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:17.432468 18366 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.074s	user 0.008s	sys 0.023s
I20260709 12:25:17.562261 27800 raft_consensus.cc:515] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:17.431314 18336 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.971s	user 0.011s	sys 0.032s
I20260709 12:25:17.564508 27703 consensus_queue.cc:1059] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.001s
I20260709 12:25:17.566360 27965 consensus_queue.cc:237] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 24.16, Last appended by leader: 16, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:17.419641 17727 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.115s	user 0.013s	sys 0.029s
W20260709 12:25:17.417713 17759 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.048s	user 0.012s	sys 0.011s
W20260709 12:25:17.416996 17689 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.048s	user 0.017s	sys 0.021s
I20260709 12:25:17.433128 18361 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59886 (request call id 2722) took 1601 ms. Trace:
I20260709 12:25:17.501144 17683 rpcz_store.cc:276] 0709 12:25:16.171585 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:16.172094 (+   509us) service_pool.cc:224] Handling call
0709 12:25:16.224225 (+ 52131us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:16.224575 (+   350us) raft_consensus.cc:1534] Early marking committed up to index 17
0709 12:25:16.250234 (+ 25659us) log.cc:844] Serialized 14 byte log entry
0709 12:25:16.356236 (+106002us) spinlock_profiling.cc:243] Waited 5.58 ms on lock 0x7b18002507e0. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b47515e07 00007f7b503709a9 00007f7b4753269b 00007f7b4753093f 00007f7b47532c02 00007f7b4a1ba20b 00007f7b4a1bfd23 00007f7b4a1c023a 00007f7b4a62aa2a
0709 12:25:16.558494 (+202258us) spinlock_profiling.cc:243] Waited 2.18 ms on lock 0x7b18002507e0. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b475333f3 00007f7b475357c1 00007f7b47535779 00007f7b47535709 00007f7b475356d1 00007f7b475349cd 0000000000388634 0000000000388479 00007f7b4751683a
0709 12:25:17.410008 (+851514us) log.cc:844] Serialized 14 byte log entry
0709 12:25:17.410388 (+   380us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:17.411374 (+   986us) log.cc:844] Serialized 26 byte log entry
0709 12:25:17.411567 (+   193us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:25:17.411608 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:17.411742 (+   134us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:17.499975 (+ 88233us) raft_consensus.cc:1685] finished
0709 12:25:17.500001 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:17.500383 (+   382us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":63253,"prepare.run_cpu_time_us":110,"prepare.run_wall_time_us":112,"raft.queue_time_us":55771,"raft.run_cpu_time_us":654,"raft.run_wall_time_us":19887,"spinlock_wait_cycles":17843712,"thread_start_us":1052375,"threads_started":1,"wal-append.queue_time_us":1090201}
I20260709 12:25:17.568732 18361 rpcz_store.cc:276] 0709 12:25:15.831309 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:15.833685 (+  2376us) service_pool.cc:224] Handling call
0709 12:25:15.834110 (+   425us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:15.834515 (+   405us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:15.834559 (+    44us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:15.835075 (+   516us) log.cc:844] Serialized 26 byte log entry
0709 12:25:16.667354 (+832279us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:25:16.667412 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:16.667528 (+   116us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:17.432274 (+764746us) raft_consensus.cc:1685] finished
0709 12:25:17.432315 (+    41us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:17.432839 (+   524us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":65576,"raft.run_cpu_time_us":171,"raft.run_wall_time_us":26927,"thread_start_us":831874,"threads_started":1,"wal-append.queue_time_us":1506515}
I20260709 12:25:17.539624 28013 raft_consensus.cc:493] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
W20260709 12:25:17.433324 18374 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.942s	user 0.011s	sys 0.020s
W20260709 12:25:17.429908 18364 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.101s	user 0.012s	sys 0.025s
W20260709 12:25:17.430998 18362 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.936s	user 0.008s	sys 0.025s
I20260709 12:25:17.435068 17760 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38786 (request call id 4230) took 1616 ms. Trace:
W20260709 12:25:17.435578 17836 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.935s	user 0.020s	sys 0.020s
W20260709 12:25:17.435913 27809 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.330s	user 0.007s	sys 0.033s
I20260709 12:25:17.441224 27903 leader_election.cc:290] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:17.437727 27874 raft_consensus.cc:2749] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:25:17.442273 27927 raft_consensus.cc:493] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:17.435081 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:17.583509 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:17.591049 27809 raft_consensus.cc:493] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:17.591423 18374 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59886 (request call id 2721) took 1962 ms. Trace:
I20260709 12:25:17.592340 27809 raft_consensus.cc:515] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:17.592612 18374 rpcz_store.cc:276] 0709 12:25:15.629377 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:15.697219 (+ 67842us) service_pool.cc:224] Handling call
0709 12:25:15.697483 (+   264us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:16.348881 (+651398us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:16.348923 (+    42us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:16.367972 (+ 19049us) log.cc:844] Serialized 26 byte log entry
0709 12:25:17.586254 (+1218282us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:25:17.586365 (+   111us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:17.590875 (+  4510us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:17.590929 (+    54us) raft_consensus.cc:1685] finished
0709 12:25:17.590942 (+    13us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:17.591344 (+   402us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":71542,"raft.run_cpu_time_us":77,"raft.run_wall_time_us":77,"thread_start_us":1741596,"threads_started":2,"wal-append.queue_time_us":941411}
I20260709 12:25:17.594825 27881 raft_consensus.cc:493] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:17.595556 28028 raft_consensus.cc:493] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:25:17.595870 27881 raft_consensus.cc:515] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:17.596398 28028 raft_consensus.cc:515] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:17.599335 28028 leader_election.cc:290] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:25:17.599419 27881 leader_election.cc:290] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:25:17.437423 27898 raft_consensus.cc:493] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:17.435107 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:17.475258 27856 raft_consensus.cc:493] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:17.483770 17758 raft_consensus.cc:2410] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for term 26 because replica has last-logged OpId of term: 25 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 17.
I20260709 12:25:17.496196 27671 raft_consensus.cc:493] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:17.478940 27892 raft_consensus.cc:515] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:25:17.500671 27724 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.920s	user 0.002s	sys 0.006s
I20260709 12:25:17.526448 17763 raft_consensus.cc:1275] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 24 index: 18. Preceding OpId from leader: term: 25 index: 18. (term mismatch)
I20260709 12:25:17.622988 17763 pending_rounds.cc:85] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb: Aborting all ops after (but not including) 17
I20260709 12:25:17.623308 17763 pending_rounds.cc:107] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb: Aborting uncommitted NO_OP operation due to leader change: 24.18
W20260709 12:25:17.066999 27987 log.cc:927] Time spent T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.214s	user 0.000s	sys 0.000s
I20260709 12:25:17.630231 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 35 candidate_status { last_received { term: 33 index: 25 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:17.630664 18362 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 2927) took 1260 ms. Trace:
I20260709 12:25:17.631467 17756 raft_consensus.cc:2468] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 34.
I20260709 12:25:17.527736 17761 raft_consensus.cc:2468] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 29.
I20260709 12:25:17.520879 27431 raft_consensus.cc:515] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:17.528463 17830 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35042 (request call id 3585) took 1771 ms. Trace:
I20260709 12:25:17.034832 27989 raft_consensus.cc:697] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 27 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:25:17.633572 17836 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35040 (request call id 3784) took 2133 ms. Trace:
I20260709 12:25:17.633738 27008 raft_consensus.cc:493] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:25:17.545198 27929 raft_consensus.cc:515] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:17.635031 27431 leader_election.cc:290] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:17.634763 27008 raft_consensus.cc:515] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:25:17.109730 18334 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.760s	user 0.010s	sys 0.022s
W20260709 12:25:17.636368 18334 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.294s	user 0.010s	sys 0.022s
I20260709 12:25:17.636540 18365 raft_consensus.cc:2393] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 21: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:25:17.637804 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 25 candidate_status { last_received { term: 23 index: 21 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:17.547586 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 34 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:17.635074 27989 consensus_queue.cc:237] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 26.20, Last appended by leader: 20, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:17.640372 18368 raft_consensus.cc:2393] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 34: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:25:17.640440 18363 raft_consensus.cc:3060] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:17.640905 27809 leader_election.cc:290] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:17.641477 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:17.642261 18368 raft_consensus.cc:2393] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 21: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:25:17.643333 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 29 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:25:17.554272 27871 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.125s	user 0.034s	sys 0.023s
I20260709 12:25:17.639237 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:17.644620 27671 raft_consensus.cc:515] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:17.430908 18335 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.072s	user 0.007s	sys 0.024s
I20260709 12:25:17.515053 27986 consensus_queue.cc:260] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 31.20, Last appended by leader: 20, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:17.559036 17578 catalog_manager.cc:5697] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 reported cstate change: term changed from 16 to 17, leader changed from e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66) to e063541542a94969898868bf6eb5ce1b (127.16.29.67). New cstate: current_term: 17 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } }
I20260709 12:25:17.647518 27671 leader_election.cc:290] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:17.600634 27903 raft_consensus.cc:493] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:17.582099 17760 rpcz_store.cc:276] 0709 12:25:15.818760 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:15.818925 (+   165us) service_pool.cc:224] Handling call
0709 12:25:15.819289 (+   364us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:15.819657 (+   368us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:15.819689 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:15.820213 (+   524us) log.cc:844] Serialized 26 byte log entry
0709 12:25:16.806793 (+986580us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:25:16.806862 (+    69us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:16.806971 (+   109us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:17.434219 (+627248us) raft_consensus.cc:1685] finished
0709 12:25:17.434290 (+    71us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:17.434712 (+   422us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":7158,"raft.run_cpu_time_us":88,"raft.run_wall_time_us":88,"thread_start_us":986061,"threads_started":1,"wal-append.queue_time_us":1523774}
I20260709 12:25:17.415563 17652 leader_election.cc:304] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:17.611234 17654 leader_election.cc:304] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:25:17.617669 28011 raft_consensus.cc:3055] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 31 LEADER]: Stepping down as leader of term 31
W20260709 12:25:17.617995 17803 leader_election.cc:336] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Timed out: RequestConsensusVote RPC to 127.16.29.68:42577 timed out after 1.852s (SENT)
I20260709 12:25:17.618999 27927 raft_consensus.cc:515] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:17.660107 18368 raft_consensus.cc:2468] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 28.
I20260709 12:25:17.660297 27766 raft_consensus.cc:2749] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:25:17.660679 17803 leader_election.cc:304] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:17.661409 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 30 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:17.662004 27927 leader_election.cc:290] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:17.662743 27766 raft_consensus.cc:2804] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:25:17.662859 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 28 candidate_status { last_received { term: 24 index: 20 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:17.663554 28017 raft_consensus.cc:493] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:17.660447 27903 raft_consensus.cc:515] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:17.664283 28017 raft_consensus.cc:515] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:17.666038 27903 leader_election.cc:290] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:17.666946 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:17.669409 17758 raft_consensus.cc:3060] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:17.623561 17763 raft_consensus.cc:2889] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:17.630076 28035 raft_consensus.cc:493] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:25:17.639549 27856 raft_consensus.cc:515] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:17.647420 27986 raft_consensus.cc:3060] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Advancing to term 32
W20260709 12:25:17.432026 18371 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.611s	user 0.002s	sys 0.030s
I20260709 12:25:17.600257 27874 raft_consensus.cc:493] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:17.606734 27898 raft_consensus.cc:3060] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:17.648300 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:17.639237 27008 leader_election.cc:290] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:25:17.633129 17830 rpcz_store.cc:276] 0709 12:25:15.757041 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:15.760496 (+  3455us) service_pool.cc:224] Handling call
0709 12:25:15.760815 (+   319us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:15.761143 (+   328us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:15.761166 (+    23us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:15.761652 (+   486us) log.cc:844] Serialized 26 byte log entry
0709 12:25:16.543303 (+781651us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:25:16.543365 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:16.543471 (+   106us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:17.528046 (+984575us) raft_consensus.cc:1685] finished
0709 12:25:17.528092 (+    46us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:17.528299 (+   207us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1078,"raft.run_cpu_time_us":76,"raft.run_wall_time_us":78,"thread_start_us":781215,"threads_started":1,"wal-append.queue_time_us":1187127}
I20260709 12:25:17.631672 18362 rpcz_store.cc:276] 0709 12:25:16.370119 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:16.375830 (+  5711us) service_pool.cc:224] Handling call
0709 12:25:16.472190 (+ 96360us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:16.472558 (+   368us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 12:25:17.591397 (+1118839us) log.cc:844] Serialized 14 byte log entry
0709 12:25:17.615562 (+ 24165us) log.cc:844] Serialized 14 byte log entry
0709 12:25:17.620919 (+  5357us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:17.621593 (+   674us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:25:17.623039 (+  1446us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:17.623605 (+   566us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:17.630428 (+  6823us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":851318,"prepare.run_cpu_time_us":99,"prepare.run_wall_time_us":108,"raft.queue_time_us":786,"raft.run_cpu_time_us":238,"raft.run_wall_time_us":240,"thread_start_us":1106769,"threads_started":2}
W20260709 12:25:17.657905 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 1.40032e+06us
I20260709 12:25:17.658732 27929 leader_election.cc:290] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:17.659385 17689 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37396 (request call id 3517) took 1771 ms. Trace:
I20260709 12:25:17.621279 27767 raft_consensus.cc:493] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:17.635659 27431 raft_consensus.cc:493] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:25:17.666859 28017 leader_election.cc:290] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:17.700024 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:17.700901 18374 raft_consensus.cc:2368] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 27. Current term is 28.
I20260709 12:25:17.701608 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 32 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:17.701790 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:17.702521 18374 raft_consensus.cc:2368] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 27. Current term is 28.
I20260709 12:25:17.703173 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:17.653630 28011 raft_consensus.cc:740] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 31 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:25:17.704777 17832 raft_consensus.cc:1240] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 27. Current term is 28. Ops: []
I20260709 12:25:17.704862 28011 consensus_queue.cc:260] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 31.21, Last appended by leader: 21, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:17.706305 28011 raft_consensus.cc:3060] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:25:17.708491 28041 raft_consensus.cc:2749] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:25:17.708643 27874 raft_consensus.cc:515] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:17.709744 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:17.710630 17829 raft_consensus.cc:2393] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 26: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:25:17.711030 27874 leader_election.cc:290] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:25:17.711942 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:17.712821 17833 raft_consensus.cc:3060] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:17.716485 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:17.717463 17837 raft_consensus.cc:2468] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 26.
I20260709 12:25:17.717628 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 33 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:25:17.719728 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:25:17.720687 17680 raft_consensus.cc:3060] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Advancing to term 28
W20260709 12:25:17.686635 18371 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.866s	user 0.002s	sys 0.031s
I20260709 12:25:17.688638 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 34 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:25:17.685758 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 74905.7us
I20260709 12:25:17.688889 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:17.689450 18365 raft_consensus.cc:3060] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Advancing to term 28
W20260709 12:25:17.689666 27853 consensus_meta.cc:287] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.674s	user 0.007s	sys 0.013s
I20260709 12:25:17.690797 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:17.691665 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 28 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:25:17.692122 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:17.692207 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 32 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:17.694742 27431 raft_consensus.cc:515] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:17.696028 27766 raft_consensus.cc:697] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 30 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:25:17.696717 27725 raft_consensus.cc:2804] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:25:17.688912 27577 raft_consensus.cc:493] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:17.698424 17763 raft_consensus.cc:1275] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 32 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 29 index: 17. Preceding OpId from leader: term: 30 index: 18. (index mismatch)
I20260709 12:25:17.703392 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 30 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:17.663045 27892 leader_election.cc:290] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:25:17.753348 17683 raft_consensus.cc:3060] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:17.634052 17836 rpcz_store.cc:276] 0709 12:25:15.499834 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:15.500030 (+   196us) service_pool.cc:224] Handling call
0709 12:25:15.500769 (+   739us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:15.500974 (+   205us) raft_consensus.cc:1534] Early marking committed up to index 16
0709 12:25:16.500450 (+999476us) log.cc:844] Serialized 14 byte log entry
0709 12:25:17.632999 (+1132549us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:17.633096 (+    97us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:25:17.633242 (+   146us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:17.633357 (+   115us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:17.633506 (+   149us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":83031,"prepare.run_cpu_time_us":27,"prepare.run_wall_time_us":27,"raft.queue_time_us":63620,"raft.run_cpu_time_us":210,"raft.run_wall_time_us":209,"thread_start_us":2121575,"threads_started":3,"wal-append.queue_time_us":820052}
I20260709 12:25:17.753768 17762 raft_consensus.cc:2468] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 22.
I20260709 12:25:17.753371 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:17.750088 18362 raft_consensus.cc:2393] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 34: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:25:17.750241 18363 raft_consensus.cc:2468] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 26.
I20260709 12:25:17.703433 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 30 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:17.676657 27860 leader_election.cc:290] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 35 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:17.676384 27881 raft_consensus.cc:493] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:25:17.756796 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:17.757061 27725 raft_consensus.cc:697] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 26 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
W20260709 12:25:17.757756 28037 log.cc:927] Time spent T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.085s	user 0.000s	sys 0.000s
I20260709 12:25:17.757264 27881 raft_consensus.cc:515] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:17.758890 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 33 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:17.758554 27725 consensus_queue.cc:237] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 24.17, Last appended by leader: 17, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:17.759765 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 344908us
I20260709 12:25:17.760412 28010 leader_election.cc:290] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:25:17.759105 18333 connection.cc:256] RPC call timeout handler was delayed by -1.3426037089998317s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1549s for a response
W20260709 12:25:17.761047 18333 connection.cc:256] RPC call timeout handler was delayed by -1.2517037089996847s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.165s for a response
W20260709 12:25:17.761446 18333 connection.cc:256] RPC call timeout handler was delayed by -1.2472037089996775s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1655s for a response
I20260709 12:25:17.761355 17652 leader_election.cc:304] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
W20260709 12:25:17.761811 18333 connection.cc:256] RPC call timeout handler was delayed by -1.23640370899966s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16670000000000001s for a response
I20260709 12:25:17.762797 18333 leader_election.cc:304] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:17.766492 27978 consensus_queue.cc:1048] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:25:17.751700 17761 raft_consensus.cc:2468] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 28.
I20260709 12:25:17.763692 17652 leader_election.cc:304] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:17.759790 17830 raft_consensus.cc:2368] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 34 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 33. Current term is 34.
I20260709 12:25:17.758580 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 25 candidate_status { last_received { term: 23 index: 21 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:17.751220 17764 raft_consensus.cc:2468] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 29.
I20260709 12:25:17.756789 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 28 candidate_status { last_received { term: 24 index: 20 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:17.775017 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 30 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:17.782233 17760 raft_consensus.cc:2468] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 29.
I20260709 12:25:17.768260 27767 raft_consensus.cc:515] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:17.789829 17831 raft_consensus.cc:2468] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 29.
I20260709 12:25:17.790210 17759 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38776 (request call id 3412) took 2750 ms. Trace:
I20260709 12:25:17.790804 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 33 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:17.791564 17760 raft_consensus.cc:2468] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 33.
I20260709 12:25:17.792173 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:17.792881 17838 raft_consensus.cc:3060] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Advancing to term 28
I20260709 12:25:17.796255 27744 raft_consensus.cc:493] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:25:17.796726 27431 leader_election.cc:290] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:25:17.796988 27744 raft_consensus.cc:515] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:17.797653 27881 leader_election.cc:290] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:25:17.790215 18333 leader_election.cc:304] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:17.798949 27766 consensus_queue.cc:237] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 28.18, Last appended by leader: 18, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:17.801180 18333 leader_election.cc:304] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:17.807335 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 33 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:17.686502 28013 raft_consensus.cc:515] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:17.802704 27856 leader_election.cc:290] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:17.790660 17759 rpcz_store.cc:276] 0709 12:25:15.040149 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:15.400658 (+360509us) service_pool.cc:224] Handling call
0709 12:25:15.488845 (+ 88187us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:15.489119 (+   274us) raft_consensus.cc:1534] Early marking committed up to index 16
0709 12:25:16.369245 (+880126us) log.cc:844] Serialized 14 byte log entry
0709 12:25:17.781859 (+1412614us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:17.789698 (+  7839us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:25:17.789795 (+    97us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:17.789931 (+   136us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:17.790136 (+   205us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":42713,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":37,"raft.queue_time_us":14048,"raft.run_cpu_time_us":110,"raft.run_wall_time_us":110,"thread_start_us":2227800,"threads_started":3,"wal-append.queue_time_us":961182}
I20260709 12:25:17.808528 28028 raft_consensus.cc:2749] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 12:25:17.812089 27881 raft_consensus.cc:3060] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:17.817298 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 28 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:17.818137 18374 raft_consensus.cc:2468] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 29.
I20260709 12:25:17.818922 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:17.819160 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:17.808667 27978 raft_consensus.cc:2749] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:25:17.820231 28043 raft_consensus.cc:493] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:25:17.821049 28043 raft_consensus.cc:515] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:17.823756 18333 leader_election.cc:304] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:17.824555 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:17.825485 18368 raft_consensus.cc:2468] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 30.
I20260709 12:25:17.826866 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 32 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:17.827270 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 33 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:17.828091 27853 raft_consensus.cc:2749] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260709 12:25:17.828238 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 28 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:17.828912 27853 consensus_queue.cc:1048] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20260709 12:25:17.829447 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 29 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:17.830757 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 30 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:17.831377 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 26 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:17.779325 27577 raft_consensus.cc:515] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:17.825536 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 62067.9us
I20260709 12:25:17.782416 28010 raft_consensus.cc:493] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:25:17.836339 28010 raft_consensus.cc:515] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:17.838256 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 29 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:17.838696 28010 leader_election.cc:290] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:17.839119 18333 leader_election.cc:304] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
W20260709 12:25:17.806939 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 51945us
I20260709 12:25:17.851788 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:17.780256 28017 raft_consensus.cc:2749] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:25:17.849996 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 30 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:17.855662 28036 raft_consensus.cc:493] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:17.856091 27993 raft_consensus.cc:493] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:25:17.851277 18333 leader_election.cc:304] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:17.681308 17689 rpcz_store.cc:276] 0709 12:25:15.887798 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:15.887919 (+   121us) service_pool.cc:224] Handling call
0709 12:25:15.888219 (+   300us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:15.888563 (+   344us) raft_consensus.cc:1534] Early marking committed up to index 19
0709 12:25:15.890839 (+  2276us) log.cc:844] Serialized 14 byte log entry
0709 12:25:16.368106 (+477267us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:16.368832 (+   726us) log.cc:844] Serialized 26 byte log entry
0709 12:25:16.368970 (+   138us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:25:17.629253 (+1260283us) log.cc:844] Serialized 14 byte log entry
0709 12:25:17.658835 (+ 29582us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:17.658964 (+   129us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:17.659052 (+    88us) raft_consensus.cc:1685] finished
0709 12:25:17.659074 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:17.659311 (+   237us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":50912,"prepare.queue_time_us":197050,"prepare.run_cpu_time_us":61,"prepare.run_wall_time_us":62,"raft.queue_time_us":47767,"raft.run_cpu_time_us":409,"raft.run_wall_time_us":6430,"thread_start_us":1632590,"threads_started":2,"wal-append.queue_time_us":803264}
I20260709 12:25:17.885759 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:17.888731 17833 raft_consensus.cc:2468] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 22.
I20260709 12:25:17.852435 17759 raft_consensus.cc:2468] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 28.
I20260709 12:25:17.887686 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 33 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:17.864140 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:17.847730 28043 leader_election.cc:290] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:17.848732 27577 leader_election.cc:290] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:25:17.905208 17801 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:17.906013 17758 raft_consensus.cc:2468] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 26.
I20260709 12:25:17.905416 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:17.888022 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 28 candidate_status { last_received { term: 27 index: 21 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:17.919737 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
W20260709 12:25:17.923942 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 43870.8us
I20260709 12:25:17.925896 18374 raft_consensus.cc:2393] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 26: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:25:17.923195 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:25:17.926481 17654 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:17.859259 27874 raft_consensus.cc:493] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:17.782130 28035 raft_consensus.cc:515] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:17.934264 27860 raft_consensus.cc:493] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 34 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:25:17.943954 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 30 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:17.943604 28017 raft_consensus.cc:3060] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Advancing to term 29
W20260709 12:25:17.945547 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:17.944164 27860 raft_consensus.cc:515] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:17.932752 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:17.944653 17689 raft_consensus.cc:2468] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 29.
I20260709 12:25:17.947194 17684 raft_consensus.cc:2393] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 33: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:25:17.947700 17761 raft_consensus.cc:2393] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 27: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:25:17.849953 27892 raft_consensus.cc:493] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:25:17.905616 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:17.888525 27856 consensus_queue.cc:1048] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.052s
I20260709 12:25:17.928476 27713 raft_consensus.cc:493] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:25:17.955298 17760 raft_consensus.cc:2393] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 29: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:25:17.957702 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:17.970503 27881 raft_consensus.cc:2749] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:25:17.974232 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:17.974927 18371 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59886 (request call id 2727) took 2128 ms. Trace:
I20260709 12:25:17.975651 18374 raft_consensus.cc:1240] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 26. Current term is 27. Ops: []
I20260709 12:25:17.976014 28016 raft_consensus.cc:493] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 32 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:25:17.931775 27993 raft_consensus.cc:515] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:17.976483 18374 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59902 (request call id 2365) took 1013 ms. Trace:
I20260709 12:25:17.976694 28016 raft_consensus.cc:515] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:17.976969 18374 rpcz_store.cc:276] 0709 12:25:16.962611 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:17.939817 (+977206us) service_pool.cc:224] Handling call
0709 12:25:17.960366 (+ 20549us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:17.976200 (+ 15834us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:17.976399 (+   199us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:17.978259 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 30 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:17.979100 27993 leader_election.cc:290] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:17.979807 27713 raft_consensus.cc:515] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:17.981552 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 363628us
I20260709 12:25:17.922297 27898 raft_consensus.cc:515] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:17.978889 18374 raft_consensus.cc:2468] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 29.
W20260709 12:25:18.009841 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 44303us
I20260709 12:25:17.966221 28040 raft_consensus.cc:493] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:25:18.020504 18363 raft_consensus.cc:3060] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:18.020746 17838 raft_consensus.cc:2468] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 28.
I20260709 12:25:18.024897 17683 raft_consensus.cc:2468] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 28.
I20260709 12:25:18.025581 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:18.026244 17833 raft_consensus.cc:3060] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:17.980400 27860 leader_election.cc:290] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:18.031965 17755 raft_consensus.cc:1240] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 28. Current term is 29. Ops: []
I20260709 12:25:17.975391 18371 rpcz_store.cc:276] 0709 12:25:15.846984 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:15.849930 (+  2946us) service_pool.cc:224] Handling call
0709 12:25:15.850440 (+   510us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 12:25:15.850883 (+   443us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:15.850921 (+    38us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 12:25:15.872684 (+ 21763us) log.cc:844] Serialized 78 byte log entry
0709 12:25:16.790541 (+917857us) raft_consensus.cc:1645] Marking committed up to 22
0709 12:25:17.774668 (+984127us) log.cc:844] Serialized 14 byte log entry
0709 12:25:17.857060 (+ 82392us) log.cc:844] Serialized 14 byte log entry
0709 12:25:17.947717 (+ 90657us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:17.966455 (+ 18738us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:17.974647 (+  8192us) raft_consensus.cc:1685] finished
0709 12:25:17.974663 (+    16us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:17.974852 (+   189us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":29241,"prepare.queue_time_us":532790,"prepare.run_cpu_time_us":91,"prepare.run_wall_time_us":86,"raft.queue_time_us":268516,"raft.run_cpu_time_us":352,"raft.run_wall_time_us":5904,"thread_start_us":1775030,"threads_started":2,"wal-append.queue_time_us":1429429}
W20260709 12:25:18.033610 28038 log.cc:927] Time spent T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.109s	user 0.001s	sys 0.000s
I20260709 12:25:18.033705 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:18.034492 18371 raft_consensus.cc:2468] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 26.
W20260709 12:25:18.035666 28048 log.cc:927] Time spent T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.088s	user 0.000s	sys 0.000s
W20260709 12:25:18.036229 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 189978us
I20260709 12:25:18.037117 18333 leader_election.cc:304] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:25:18.037506 28039 log.cc:927] Time spent T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.230s	user 0.000s	sys 0.000s
I20260709 12:25:18.038188 18333 leader_election.cc:304] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:17.905755 27853 consensus_queue.cc:1048] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.161s
I20260709 12:25:17.951560 27874 raft_consensus.cc:515] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:18.030467 17764 raft_consensus.cc:2468] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 29.
I20260709 12:25:18.053367 18363 raft_consensus.cc:2468] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 27.
I20260709 12:25:18.056345 18367 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 2924) took 2038 ms. Trace:
W20260709 12:25:18.058959 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 106047us
I20260709 12:25:17.975910 28046 raft_consensus.cc:493] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:18.063302 18333 leader_election.cc:304] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:18.069027 18361 raft_consensus.cc:1275] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 25 index: 15. Preceding OpId from leader: term: 31 index: 18. (index mismatch)
I20260709 12:25:18.061173 18362 raft_consensus.cc:1275] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 28 index: 17. Preceding OpId from leader: term: 30 index: 18. (index mismatch)
W20260709 12:25:18.071966 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 99050.4us
I20260709 12:25:17.947413 28036 raft_consensus.cc:515] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:18.072806 17684 raft_consensus.cc:1275] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 25 index: 18. Preceding OpId from leader: term: 26 index: 19. (index mismatch)
I20260709 12:25:18.077459 18361 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59902 (request call id 2374) took 1077 ms. Trace:
I20260709 12:25:18.078159 18361 rpcz_store.cc:276] 0709 12:25:16.996148 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:17.956325 (+960177us) service_pool.cc:224] Handling call
0709 12:25:18.024275 (+ 67950us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:18.069481 (+ 45206us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:18.073337 (+  3856us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:18.075050 18362 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59902 (request call id 2364) took 1109 ms. Trace:
I20260709 12:25:18.081066 18362 rpcz_store.cc:276] 0709 12:25:16.962332 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:17.907294 (+944962us) service_pool.cc:224] Handling call
0709 12:25:18.018186 (+110892us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:18.071210 (+ 53024us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:18.071393 (+   183us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:17.974876 27570 raft_consensus.cc:493] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:25:18.082825 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:25:18.085953 17687 raft_consensus.cc:3060] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:18.062831 27881 raft_consensus.cc:2804] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:25:18.020962 27892 raft_consensus.cc:515] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:18.093501 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:18.093765 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 30 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:18.094416 17831 raft_consensus.cc:2468] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 29.
I20260709 12:25:18.077246 27993 raft_consensus.cc:2804] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:25:18.056913 18367 rpcz_store.cc:276] 0709 12:25:16.018067 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:16.343907 (+325840us) service_pool.cc:224] Handling call
0709 12:25:16.362327 (+ 18420us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:17.479783 (+1117456us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:17.479846 (+    63us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:17.481002 (+  1156us) log.cc:844] Serialized 26 byte log entry
0709 12:25:17.612778 (+131776us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:25:17.612831 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:17.612973 (+   142us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:18.046068 (+433095us) raft_consensus.cc:1685] finished
0709 12:25:18.046088 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:18.056284 (+ 10196us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":183974,"raft.run_cpu_time_us":165,"raft.run_wall_time_us":11976,"thread_start_us":1247900,"threads_started":2,"wal-append.queue_time_us":293332}
I20260709 12:25:18.094372 17830 raft_consensus.cc:2468] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 26.
I20260709 12:25:18.100554 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 29 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:18.105343 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 31 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:18.096885 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 29 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:18.092105 17801 leader_election.cc:304] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:18.069172 27903 raft_consensus.cc:493] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:25:18.091526 27881 raft_consensus.cc:493] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:18.109097 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 31 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:18.055964 27989 consensus_queue.cc:1048] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.045s
I20260709 12:25:18.092896 27577 raft_consensus.cc:493] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:18.119099 27874 leader_election.cc:290] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:25:18.123426 27892 leader_election.cc:290] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:18.124024 27570 raft_consensus.cc:515] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:18.124833 28033 raft_consensus.cc:493] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:18.126024 28003 log.cc:927] Time spent T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.166s	user 0.001s	sys 0.000s
I20260709 12:25:18.104941 27993 raft_consensus.cc:493] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:18.149300 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 29 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:18.116930 27881 raft_consensus.cc:3060] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:18.108680 27978 consensus_queue.cc:1059] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.044s
W20260709 12:25:18.158968 17655 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:18.137286 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 51855.7us
I20260709 12:25:18.145709 27993 raft_consensus.cc:3060] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:18.145226 28013 leader_election.cc:290] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:18.164006 27753 raft_consensus.cc:2804] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:25:18.216131 27978 consensus_queue.cc:1243] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer e063541542a94969898868bf6eb5ce1b log is divergent from this leader: its last log entry 31.18 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 17
I20260709 12:25:18.216389 28035 leader_election.cc:290] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:25:18.217474 28047 log.cc:927] Time spent T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.169s	user 0.001s	sys 0.000s
I20260709 12:25:18.205428 17833 raft_consensus.cc:2468] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 27.
I20260709 12:25:18.229494 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 29 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:18.230288 17833 raft_consensus.cc:2468] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 28.
I20260709 12:25:18.218576 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:18.231156 27753 raft_consensus.cc:697] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 22 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:25:18.232831 27753 consensus_queue.cc:237] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 21.15, Last appended by leader: 15, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:18.237571 28036 leader_election.cc:290] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:18.240473 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:18.243842 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:18.244386 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 30 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:18.244705 17685 raft_consensus.cc:2468] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 30.
I20260709 12:25:18.245213 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:18.245639 17830 raft_consensus.cc:2410] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for term 30 because replica has last-logged OpId of term: 28 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 26 index: 16.
I20260709 12:25:18.245908 17832 raft_consensus.cc:2468] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 20.
W20260709 12:25:18.225816 28049 log.cc:927] Time spent T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.291s	user 0.001s	sys 0.000s
I20260709 12:25:18.159164 27713 leader_election.cc:290] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:18.246840 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:18.148432 28016 leader_election.cc:290] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:18.247077 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:18.231276 17838 raft_consensus.cc:3060] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:18.247773 17834 raft_consensus.cc:2468] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 27.
W20260709 12:25:18.159147 28051 log.cc:927] Time spent T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.094s	user 0.001s	sys 0.000s
W20260709 12:25:18.257686 17728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:18.111711 28046 raft_consensus.cc:515] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:18.262823 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:18.264894 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:18.265321 17838 raft_consensus.cc:2468] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 27.
W20260709 12:25:18.266275 17680 consensus_meta.cc:287] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.545s	user 0.008s	sys 0.003s
I20260709 12:25:18.266646 17680 raft_consensus.cc:2468] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 28.
W20260709 12:25:18.154064 17803 leader_election.cc:336] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 28 election: RPC error from VoteRequest() call to peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Timed out: RequestConsensusVote RPC to 127.16.29.68:42577 timed out after 1.944s (SENT)
I20260709 12:25:18.101951 28040 raft_consensus.cc:515] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:18.263943 17801 leader_election.cc:304] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:25:18.268836 18365 consensus_meta.cc:287] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.515s	user 0.012s	sys 0.000s
I20260709 12:25:18.164186 27744 leader_election.cc:290] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:18.277163 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:18.278540 17757 raft_consensus.cc:2468] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 26.
I20260709 12:25:18.270607 18365 raft_consensus.cc:2468] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 28.
I20260709 12:25:18.272047 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:18.273878 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 30 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:18.270821 17803 leader_election.cc:304] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:18.188907 28045 raft_consensus.cc:493] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:18.146634 28043 raft_consensus.cc:2804] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:25:18.281050 17688 raft_consensus.cc:1240] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 24. Current term is 25. Ops: []
I20260709 12:25:18.280931 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:18.283773 27839 raft_consensus.cc:2749] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:25:18.285660 28040 leader_election.cc:290] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:18.285902 17835 raft_consensus.cc:2468] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 26.
I20260709 12:25:18.282675 17836 raft_consensus.cc:2393] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 30: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
W20260709 12:25:18.288951 28055 log.cc:927] Time spent T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.131s	user 0.000s	sys 0.001s
W20260709 12:25:18.270182 28003 log.cc:927] Time spent T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.051s	user 0.000s	sys 0.000s
I20260709 12:25:18.290906 18365 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59902 (request call id 2359) took 1318 ms. Trace:
I20260709 12:25:18.169867 27903 raft_consensus.cc:515] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:18.295157 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 79394.8us
I20260709 12:25:18.299017 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 34 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:18.161334 27570 leader_election.cc:290] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:18.300942 18365 rpcz_store.cc:276] 0709 12:25:16.960917 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:17.645578 (+684661us) service_pool.cc:224] Handling call
0709 12:25:18.279471 (+633893us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:18.303762 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 31 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:18.221426 27577 raft_consensus.cc:515] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:18.303433 17686 raft_consensus.cc:2468] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 29.
I20260709 12:25:18.320369 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:25:18.321400 28042 log.cc:927] Time spent T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.261s	user 0.002s	sys 0.000s
I20260709 12:25:18.281539 28028 raft_consensus.cc:2764] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Leader pre-election decision vote started in defunct term 27: won
I20260709 12:25:18.322721 27767 leader_election.cc:290] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:18.325037 17761 raft_consensus.cc:1240] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 28. Current term is 29. Ops: []
I20260709 12:25:18.321406 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:18.321518 28040 raft_consensus.cc:2804] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:25:18.334967 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 30 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:18.337466 17687 raft_consensus.cc:2468] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 28.
I20260709 12:25:18.345045 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 29 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:18.296170 28043 raft_consensus.cc:493] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:18.159610 28033 raft_consensus.cc:515] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:18.347734 17764 raft_consensus.cc:2468] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 28.
I20260709 12:25:18.350999 17801 leader_election.cc:304] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 33 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:18.353257 17688 raft_consensus.cc:2393] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 28: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:25:18.354187 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:18.321529 27898 leader_election.cc:290] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 29 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:18.357338 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:18.365260 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:18.369053 17682 raft_consensus.cc:2468] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 32.
I20260709 12:25:18.357080 28043 raft_consensus.cc:3060] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:18.370581 17758 raft_consensus.cc:1240] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 32 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 31. Current term is 32. Ops: [31.20-31.20]
I20260709 12:25:18.372732 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:18.363175 27871 raft_consensus.cc:2749] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
W20260709 12:25:18.381960 27525 consensus_meta.cc:287] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.899s	user 0.015s	sys 0.001s
I20260709 12:25:18.382403 27525 raft_consensus.cc:515] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:18.347654 28040 raft_consensus.cc:493] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 32 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:25:18.395448 27800 leader_election.cc:290] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:25:18.412593 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 260988us
W20260709 12:25:18.432986 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 82100.5us
I20260709 12:25:18.433713 28040 raft_consensus.cc:3060] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:25:18.340790 28045 raft_consensus.cc:515] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:18.436250 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 29 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:18.437928 18366 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 2921) took 2436 ms. Trace:
W20260709 12:25:18.463595 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 92113.5us
I20260709 12:25:18.438736 28046 leader_election.cc:290] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:18.385383 18333 leader_election.cc:304] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:18.441172 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 33 candidate_status { last_received { term: 32 index: 23 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:18.465902 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 35 candidate_status { last_received { term: 33 index: 25 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:18.467499 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:18.467806 17757 raft_consensus.cc:3060] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:25:18.466092 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:18.464722 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 33 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:18.470170 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 33 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:18.470486 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 34 candidate_status { last_received { term: 33 index: 25 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:25:18.471017 17836 raft_consensus.cc:2393] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 28: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:25:18.471390 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 34 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:18.470350 17801 leader_election.cc:304] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:18.472312 17685 raft_consensus.cc:2468] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 33.
I20260709 12:25:18.476485 17686 raft_consensus.cc:2368] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 35 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for earlier term 34. Current term is 35.
W20260709 12:25:18.466239 28050 log.cc:927] Time spent T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.322s	user 0.001s	sys 0.001s
W20260709 12:25:18.467016 28011 consensus_meta.cc:287] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.760s	user 0.009s	sys 0.006s
I20260709 12:25:18.490131 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 26 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:18.487676 17802 leader_election.cc:304] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:25:18.460126 27856 raft_consensus.cc:3055] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 32 LEADER]: Stepping down as leader of term 32
I20260709 12:25:18.492415 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:18.457405 28057 raft_consensus.cc:2749] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 33 FOLLOWER]: Leader election lost for term 33. Reason: could not achieve majority
I20260709 12:25:18.463136 18366 rpcz_store.cc:276] 0709 12:25:16.001253 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:16.289253 (+288000us) service_pool.cc:224] Handling call
0709 12:25:16.289733 (+   480us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:17.562938 (+1273205us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:17.563003 (+    65us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:17.564112 (+  1109us) log.cc:844] Serialized 26 byte log entry
0709 12:25:17.670467 (+106355us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:25:17.670512 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:17.688061 (+ 17549us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:18.437624 (+749563us) raft_consensus.cc:1685] finished
0709 12:25:18.437655 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:18.437851 (+   196us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":153101,"raft.run_cpu_time_us":129,"raft.run_wall_time_us":9646,"thread_start_us":1309836,"threads_started":2,"wal-append.queue_time_us":285721}
I20260709 12:25:18.497783 17685 raft_consensus.cc:2468] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 28.
I20260709 12:25:18.496704 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 30 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:18.498903 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 30 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:18.499132 17680 raft_consensus.cc:2410] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for term 30 because replica has last-logged OpId of term: 28 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 26 index: 16.
I20260709 12:25:18.500643 17756 raft_consensus.cc:2410] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for term 32 because replica has last-logged OpId of term: 31 index: 21, which is greater than that of the candidate, which has last-logged OpId of term: 30 index: 20.
W20260709 12:25:18.494861 17803 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.16.29.68:42577 timed out after 2.936s (SENT) [suppressed 19 similar messages]
I20260709 12:25:18.505524 17802 leader_election.cc:304] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:25:18.478772 27989 raft_consensus.cc:2749] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:25:18.507203 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 29 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:25:18.508425 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 35 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:18.562583 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:25:18.567601 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 72757.5us
I20260709 12:25:18.490072 28041 raft_consensus.cc:2804] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:25:18.574613 17683 raft_consensus.cc:3060] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:18.489840 28036 raft_consensus.cc:493] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:18.576095 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 75894.5us
I20260709 12:25:18.504595 27753 raft_consensus.cc:493] T a5732e71b9d54f8786b8c511d69943bf P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:18.576182 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 70731.2us
I20260709 12:25:18.576861 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:18.494776 27525 leader_election.cc:290] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 31 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:18.566636 17727 leader_election.cc:304] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:18.581758 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 30 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:18.497498 27856 raft_consensus.cc:740] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 32 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:25:18.582731 17762 raft_consensus.cc:2468] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 29.
I20260709 12:25:18.580154 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 34 candidate_status { last_received { term: 33 index: 25 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:18.583590 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 28 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:18.588336 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 32 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:18.592602 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 35 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:25:18.594623 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 227609us
I20260709 12:25:18.595451 18333 leader_election.cc:304] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:18.596872 17757 raft_consensus.cc:2468] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 35.
I20260709 12:25:18.597200 27965 raft_consensus.cc:2749] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:25:18.597338 18333 leader_election.cc:304] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:18.598049 27965 consensus_queue.cc:1059] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 0.0, Next index: 20, Last known committed idx: 18, Time since last communication: 0.000s
I20260709 12:25:18.599002 27965 raft_consensus.cc:3055] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:25:18.598305 28057 raft_consensus.cc:2804] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:25:18.599380 28033 leader_election.cc:290] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:18.602101 27965 raft_consensus.cc:740] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
W20260709 12:25:18.586215 28050 log.cc:927] Time spent T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.084s	user 0.004s	sys 0.004s
I20260709 12:25:18.603739 27965 consensus_queue.cc:260] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 27.20, Last appended by leader: 20, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:18.604560 18364 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 2920) took 2603 ms. Trace:
I20260709 12:25:18.605108 18364 rpcz_store.cc:276] 0709 12:25:16.001030 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:16.238953 (+237923us) service_pool.cc:224] Handling call
0709 12:25:16.293050 (+ 54097us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:16.293412 (+   362us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 12:25:17.601686 (+1308274us) log.cc:844] Serialized 14 byte log entry
0709 12:25:17.956323 (+354637us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:18.039783 (+ 83460us) log.cc:844] Serialized 26 byte log entry
0709 12:25:18.039975 (+   192us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:25:18.040011 (+    36us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:18.040099 (+    88us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:18.604331 (+564232us) raft_consensus.cc:1685] finished
0709 12:25:18.604352 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:18.604487 (+   135us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":1028724,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":49,"raft.queue_time_us":181956,"raft.run_cpu_time_us":382,"raft.run_wall_time_us":22369,"thread_start_us":1538743,"threads_started":2,"wal-append.queue_time_us":417466}
I20260709 12:25:18.605664 27965 raft_consensus.cc:3060] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:18.608497 18333 leader_election.cc:400] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 33 election: Vote denied by peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837) with higher term. Message: Invalid argument: T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 34 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 33. Current term is 34.
I20260709 12:25:18.608929 18333 leader_election.cc:403] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 33 election: Cancelling election due to peer responding with higher term
I20260709 12:25:18.613205 17683 raft_consensus.cc:2468] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 29.
I20260709 12:25:18.617324 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:18.618017 17764 raft_consensus.cc:2468] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 25.
I20260709 12:25:18.625485 17755 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36572 (request call id 1814) took 1087 ms. Trace:
I20260709 12:25:18.501595 27767 raft_consensus.cc:493] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:25:18.581215 17801 leader_election.cc:304] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:25:18.648964 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:18.649914 18363 raft_consensus.cc:2468] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 21.
I20260709 12:25:18.626044 17755 rpcz_store.cc:276] 0709 12:25:17.537937 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:17.538110 (+   173us) service_pool.cc:224] Handling call
0709 12:25:17.539152 (+  1042us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:18.050215 (+511063us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:18.625199 (+574984us) spinlock_profiling.cc:243] Waited 493 ms on lock 0x7b54003e53d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:25:18.625412 (+   213us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1132834560}
W20260709 12:25:18.630854 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 140420us
I20260709 12:25:18.661837 18366 raft_consensus.cc:2393] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 34 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 34: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
W20260709 12:25:18.614216 28068 log.cc:927] Time spent T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.239s	user 0.002s	sys 0.000s
I20260709 12:25:18.573848 28041 raft_consensus.cc:493] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
W20260709 12:25:18.681286 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 99350.6us
W20260709 12:25:18.681965 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 100816us
I20260709 12:25:18.686452 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 30 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:18.662601 28033 raft_consensus.cc:2804] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:25:18.666749 27989 raft_consensus.cc:2749] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:25:18.666589 27892 raft_consensus.cc:2804] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:25:18.694873 17727 leader_election.cc:304] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:18.701944 17802 leader_election.cc:304] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:25:18.699986 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:18.701313 27892 raft_consensus.cc:493] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:25:18.675892 28016 raft_consensus.cc:2749] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 34 FOLLOWER]: Leader election lost for term 33. Reason: Vote denied by peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837) with higher term. Message: Invalid argument: T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 34 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 33. Current term is 34.
I20260709 12:25:18.687686 28041 raft_consensus.cc:3060] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:18.703769 18366 raft_consensus.cc:2468] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 26.
I20260709 12:25:18.705637 27892 raft_consensus.cc:3060] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:18.710266 17802 leader_election.cc:304] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:25:18.711349 17801 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:18.720364 27577 leader_election.cc:290] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:25:18.720665 28066 log.cc:927] Time spent T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.147s	user 0.000s	sys 0.000s
I20260709 12:25:18.705849 28054 raft_consensus.cc:2804] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:25:18.721285 28054 raft_consensus.cc:493] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:18.721642 28054 raft_consensus.cc:3060] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Advancing to term 30
W20260709 12:25:18.723487 28065 log.cc:927] Time spent T 00000000000000000000000000000000 P 2193af31cb894ec9b3d65d939a2a6851: Append to log took a long time: real 0.347s	user 0.003s	sys 0.000s
I20260709 12:25:18.705662 27856 consensus_queue.cc:260] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 32.18, Last appended by leader: 18, Current term: 32, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:18.724813 27856 raft_consensus.cc:3060] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:25:18.725193 17728 leader_election.cc:304] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:18.726450 17681 raft_consensus.cc:1240] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 27. Current term is 28. Ops: []
I20260709 12:25:18.696751 28033 raft_consensus.cc:493] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:18.727991 28033 raft_consensus.cc:3060] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:18.656939 28036 raft_consensus.cc:515] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:25:18.731092 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 45433.5us
I20260709 12:25:18.698467 27525 raft_consensus.cc:493] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:18.732949 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 46411.9us
I20260709 12:25:18.740662 17757 raft_consensus.cc:1240] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 30. Current term is 31. Ops: []
I20260709 12:25:18.740733 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 31 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:18.742194 27008 raft_consensus.cc:2804] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:25:18.742586 27008 raft_consensus.cc:493] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:25:18.742971 27008 raft_consensus.cc:3060] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:18.739434 27577 raft_consensus.cc:2804] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:25:18.693445 27767 raft_consensus.cc:515] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:18.744866 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:18.745537 17728 leader_election.cc:400] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 34 election: Vote denied by peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071) with higher term. Message: Invalid argument: T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 35 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for earlier term 34. Current term is 35.
I20260709 12:25:18.747536 17728 leader_election.cc:403] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 34 election: Cancelling election due to peer responding with higher term
I20260709 12:25:18.759593 17762 raft_consensus.cc:2468] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 29.
I20260709 12:25:18.762455 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:18.763234 17757 raft_consensus.cc:2468] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 28.
I20260709 12:25:18.702172 27753 raft_consensus.cc:515] T a5732e71b9d54f8786b8c511d69943bf P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:18.748898 28017 consensus_meta.cc:287] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.804s	user 0.013s	sys 0.004s
I20260709 12:25:18.774021 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 33 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:18.773919 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 27 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:18.775336 17681 raft_consensus.cc:2410] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for term 33 because replica has last-logged OpId of term: 31 index: 20, which is greater than that of the candidate, which has last-logged OpId of term: 30 index: 20.
I20260709 12:25:18.741834 27431 raft_consensus.cc:2804] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:25:18.744138 27898 raft_consensus.cc:2804] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:25:18.764418 18334 leader_election.cc:304] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:25:18.763909 17727 leader_election.cc:304] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:18.746182 27577 raft_consensus.cc:493] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:18.779089 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:18.789561 18363 raft_consensus.cc:2468] T a5732e71b9d54f8786b8c511d69943bf P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 28.
W20260709 12:25:18.790532 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:18.766299 27767 leader_election.cc:290] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:18.781841 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:18.741362 27525 raft_consensus.cc:515] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:18.749393 28016 consensus_queue.cc:1059] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.040s
W20260709 12:25:18.797317 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 51425.1us
I20260709 12:25:18.798810 17728 leader_election.cc:304] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:18.770002 27744 raft_consensus.cc:2749] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 35 FOLLOWER]: Leader election lost for term 34. Reason: Vote denied by peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071) with higher term. Message: Invalid argument: T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 35 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for earlier term 34. Current term is 35.
W20260709 12:25:18.803469 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 48195.2us
I20260709 12:25:18.758917 27989 consensus_queue.cc:1048] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 18, Time since last communication: 0.034s
I20260709 12:25:18.800911 17727 leader_election.cc:304] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:18.807237 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:18.788246 27898 raft_consensus.cc:493] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 33 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:18.811993 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 33 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:18.791997 28017 raft_consensus.cc:2749] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:25:18.794698 27577 raft_consensus.cc:3060] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:18.791639 27431 raft_consensus.cc:697] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 28 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:25:18.797883 28045 leader_election.cc:290] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:25:18.810371 27881 consensus_meta.cc:287] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.548s	user 0.010s	sys 0.007s
W20260709 12:25:18.842801 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 44474.9us
I20260709 12:25:18.803238 28010 raft_consensus.cc:2804] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:25:18.860656 18334 leader_election.cc:304] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:25:18.865065 17727 leader_election.cc:304] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:18.877220 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 29 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:18.878043 18363 raft_consensus.cc:2468] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 28.
I20260709 12:25:18.836683 27898 raft_consensus.cc:3060] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:25:18.882015 18374 raft_consensus.cc:1240] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 28. Current term is 29. Ops: []
I20260709 12:25:18.882988 28012 raft_consensus.cc:2804] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:25:18.883458 28012 raft_consensus.cc:493] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:18.883611 27753 leader_election.cc:290] T a5732e71b9d54f8786b8c511d69943bf P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:18.884032 17755 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 27.
I20260709 12:25:18.884354 27753 raft_consensus.cc:2804] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:25:18.884788 27753 raft_consensus.cc:493] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:25:18.885159 27753 raft_consensus.cc:3060] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Advancing to term 32
W20260709 12:25:18.886725 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 137697us
I20260709 12:25:18.888420 17652 leader_election.cc:304] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:25:18.889218 27766 consensus_queue.cc:1048] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.000s
I20260709 12:25:18.889798 17652 leader_election.cc:304] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:25:18.890802 27766 raft_consensus.cc:2804] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:25:18.891186 27766 raft_consensus.cc:493] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:18.891400 17652 leader_election.cc:304] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:18.892076 27976 raft_consensus.cc:2804] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:25:18.892469 27976 raft_consensus.cc:493] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:18.892843 27976 raft_consensus.cc:3060] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:18.896364 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 26 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:18.896505 27431 consensus_queue.cc:237] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 26.19, Last appended by leader: 19, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:18.898362 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 33 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:18.899683 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:18.900342 18371 raft_consensus.cc:3060] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:18.839797 27965 raft_consensus.cc:2804] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:25:18.891562 27766 raft_consensus.cc:3060] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:18.903261 27008 raft_consensus.cc:515] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:18.864291 27744 raft_consensus.cc:2804] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Leader election won for term 29
W20260709 12:25:18.911481 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 46575.4us
I20260709 12:25:18.913928 17727 leader_election.cc:304] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:18.942930 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 27 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:18.944625 27008 leader_election.cc:290] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 29 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:25:18.942433 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 29 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:18.947360 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 30 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:25:18.949519 17728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:18.903807 28012 raft_consensus.cc:3060] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Advancing to term 29
W20260709 12:25:18.914705 27993 consensus_meta.cc:287] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.648s	user 0.010s	sys 0.002s
I20260709 12:25:18.953527 18368 raft_consensus.cc:1275] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 34 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 31 index: 21. Preceding OpId from leader: term: 34 index: 23. (index mismatch)
I20260709 12:25:18.916386 27839 raft_consensus.cc:2749] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:25:18.957046 27671 raft_consensus.cc:2804] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:25:18.941062 28046 raft_consensus.cc:2749] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:25:18.904783 28010 raft_consensus.cc:697] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:25:18.880311 27881 raft_consensus.cc:515] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:18.961374 17652 leader_election.cc:304] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:25:18.953891 27744 raft_consensus.cc:697] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 29 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:25:18.954903 28057 raft_consensus.cc:697] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 28 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:25:18.953382 28013 raft_consensus.cc:493] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 32 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:25:18.915817 27965 raft_consensus.cc:697] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 28 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
W20260709 12:25:18.975474 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 367180us
W20260709 12:25:18.972249 28074 log.cc:927] Time spent T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.177s	user 0.000s	sys 0.001s
I20260709 12:25:18.902863 28043 raft_consensus.cc:515] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:18.962707 27671 raft_consensus.cc:493] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:18.991160 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 35 candidate_status { last_received { term: 33 index: 25 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:18.991752 18368 raft_consensus.cc:3060] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:25:18.959909 27993 raft_consensus.cc:515] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:18.989200 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 32 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:18.998749 18365 raft_consensus.cc:2468] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 31.
I20260709 12:25:18.986879 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:18.999162 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 28 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:18.996613 27856 raft_consensus.cc:2804] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Leader election won for term 26
W20260709 12:25:19.081144 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 280727us
I20260709 12:25:19.042658 27671 raft_consensus.cc:3060] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:18.976154 28041 raft_consensus.cc:515] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:25:19.083665 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 207655us
I20260709 12:25:19.092059 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 30 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:19.098133 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:19.104221 18364 raft_consensus.cc:2393] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 30: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:25:19.061609 28036 leader_election.cc:290] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:25:19.077021 27856 raft_consensus.cc:697] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 26 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
W20260709 12:25:19.107402 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 154985us
W20260709 12:25:19.112999 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 252562us
I20260709 12:25:19.044607 28013 raft_consensus.cc:515] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:19.119942 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 203212us
I20260709 12:25:19.123261 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 25 candidate_status { last_received { term: 23 index: 21 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:19.122685 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:19.116745 17803 leader_election.cc:304] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:19.142235 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:19.142922 17762 raft_consensus.cc:2468] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 28.
I20260709 12:25:19.139075 17727 leader_election.cc:304] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:19.144204 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:19.115900 27525 leader_election.cc:290] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:19.174727 27993 leader_election.cc:290] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 30 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:25:19.186822 17654 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:19.200381 27856 consensus_queue.cc:237] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 25.17, Last appended by leader: 17, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:19.204212 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:25:19.204457 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 65543.9us
I20260709 12:25:19.205173 18364 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 27.
I20260709 12:25:19.205470 17727 leader_election.cc:304] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:19.206717 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 28 candidate_status { last_received { term: 27 index: 21 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:19.207114 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:19.207976 18365 raft_consensus.cc:2393] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 27: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:25:19.208144 28099 raft_consensus.cc:2804] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:25:19.208666 28099 raft_consensus.cc:493] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:25:19.208897 28078 log.cc:927] Time spent T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.089s	user 0.001s	sys 0.000s
I20260709 12:25:19.204264 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 29 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:19.208841 28011 raft_consensus.cc:2804] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:25:19.206717 17727 leader_election.cc:304] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:19.200459 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 33 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:19.144062 28016 raft_consensus.cc:493] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:19.227831 17652 leader_election.cc:304] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:19.229543 17652 leader_election.cc:304] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:19.194868 28080 raft_consensus.cc:493] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:19.194904 28087 raft_consensus.cc:3060] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:19.209141 28099 raft_consensus.cc:3060] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:19.236212 27978 raft_consensus.cc:2804] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:25:19.117503 28010 consensus_queue.cc:237] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 25.18, Last appended by leader: 18, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:19.145133 28057 consensus_queue.cc:237] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 26.20, Last appended by leader: 20, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:19.233271 27881 leader_election.cc:290] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 30 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:25:19.258857 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 146878us
I20260709 12:25:19.138574 27965 consensus_queue.cc:237] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 26.16, Last appended by leader: 16, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:19.124886 27744 consensus_queue.cc:237] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 27.19, Last appended by leader: 19, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:19.258008 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 63615.4us
I20260709 12:25:19.280048 27978 raft_consensus.cc:493] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:19.286871 27978 raft_consensus.cc:3060] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:19.287282 17803 leader_election.cc:304] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:25:19.288174 28085 raft_consensus.cc:2804] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:25:19.288674 28085 raft_consensus.cc:493] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:25:19.289085 28085 raft_consensus.cc:3060] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:19.292811 17803 leader_election.cc:304] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:25:19.236943 28016 raft_consensus.cc:515] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:19.294100 28013 leader_election.cc:290] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:19.224737 27800 consensus_queue.cc:1059] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 28.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.052s
I20260709 12:25:19.295256 27800 raft_consensus.cc:2804] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:25:19.296085 27431 raft_consensus.cc:2749] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:25:19.297804 17803 leader_election.cc:400] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 27 pre-election: Vote denied by peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577) with higher term. Message: Invalid argument: T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 27. Current term is 28.
W20260709 12:25:19.300581 28090 log.cc:927] Time spent T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.087s	user 0.000s	sys 0.000s
I20260709 12:25:19.301609 28085 raft_consensus.cc:515] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:19.307950 17803 leader_election.cc:403] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 27 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:25:19.309971 18368 raft_consensus.cc:2468] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 35.
I20260709 12:25:19.295585 28016 leader_election.cc:290] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:25:19.345082 17758 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38776 (request call id 3425) took 1090 ms. Trace:
I20260709 12:25:19.347522 17829 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35040 (request call id 3798) took 1182 ms. Trace:
W20260709 12:25:19.354828 17756 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38776 (request call id 3420) took 1665 ms (client timeout 1907 ms). Trace:
I20260709 12:25:19.350898 17758 rpcz_store.cc:276] 0709 12:25:18.254857 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:18.255309 (+   452us) service_pool.cc:224] Handling call
0709 12:25:18.265043 (+  9734us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:18.371007 (+105964us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:19.344703 (+973696us) spinlock_profiling.cc:243] Waited 88.7 ms on lock 0x7b54002b6454. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:25:19.344997 (+   294us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":204079104}
W20260709 12:25:19.355268 17756 rpcz_store.cc:269] 0709 12:25:17.689410 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:17.691960 (+  2550us) service_pool.cc:224] Handling call
0709 12:25:19.349301 (+1657341us) spinlock_profiling.cc:243] Waited 747 ms on lock 0x7b54003e44d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:25:19.354734 (+  5433us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1719215872}
I20260709 12:25:19.295651 27800 raft_consensus.cc:493] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
W20260709 12:25:19.312487 27892 consensus_meta.cc:287] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.603s	user 0.010s	sys 0.002s
I20260709 12:25:19.338596 28045 raft_consensus.cc:2804] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:25:19.351913 17829 rpcz_store.cc:276] 0709 12:25:18.165105 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:18.205919 (+ 40814us) service_pool.cc:224] Handling call
0709 12:25:18.248288 (+ 42369us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:18.272764 (+ 24476us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 12:25:18.326771 (+ 54007us) log.cc:844] Serialized 14 byte log entry
0709 12:25:18.682595 (+355824us) log.cc:844] Serialized 14 byte log entry
0709 12:25:18.981884 (+299289us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:18.981952 (+    68us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:25:18.982002 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:19.347233 (+365231us) spinlock_profiling.cc:243] Waited 24.1 ms on lock 0x7b540022ae54. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:25:19.347293 (+    60us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:19.347450 (+   157us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":110206,"prepare.queue_time_us":78246,"prepare.run_cpu_time_us":370,"prepare.run_wall_time_us":369,"raft.queue_time_us":103645,"raft.run_cpu_time_us":226,"raft.run_wall_time_us":10527,"spinlock_wait_cycles":55495936,"thread_start_us":465920,"threads_started":2,"wal-append.queue_time_us":371480}
W20260709 12:25:19.377089 28054 consensus_meta.cc:287] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.655s	user 0.020s	sys 0.000s
I20260709 12:25:19.377228 27892 raft_consensus.cc:515] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:19.304018 27898 raft_consensus.cc:515] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 34 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:25:19.389740 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 105138us
W20260709 12:25:19.391417 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 161995us
W20260709 12:25:19.389833 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 183207us
I20260709 12:25:19.392225 17652 leader_election.cc:304] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 35 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:25:19.407414 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 187883us
I20260709 12:25:19.408705 17654 leader_election.cc:304] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:19.257368 27993 raft_consensus.cc:493] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:25:19.313975 27570 raft_consensus.cc:3055] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:25:19.410288 17727 leader_election.cc:304] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:19.370496 28085 leader_election.cc:290] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 29 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:19.313575 28043 leader_election.cc:290] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 30 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:25:19.388790 28040 consensus_meta.cc:287] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.899s	user 0.003s	sys 0.009s
W20260709 12:25:19.420426 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 225883us
I20260709 12:25:19.377314 27431 raft_consensus.cc:2749] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Leader pre-election lost for term 27. Reason: Vote denied by peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577) with higher term. Message: Invalid argument: T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 27. Current term is 28.
I20260709 12:25:19.384825 27800 raft_consensus.cc:3060] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:19.386408 28045 raft_consensus.cc:493] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:19.421378 27570 raft_consensus.cc:740] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:25:19.391355 27525 raft_consensus.cc:493] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 34 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:25:19.406806 28041 leader_election.cc:290] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 29 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:25:19.256752 28080 raft_consensus.cc:515] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:19.391149 28054 raft_consensus.cc:515] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:19.459518 27892 leader_election.cc:290] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 31 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:25:19.459208 27978 raft_consensus.cc:515] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:19.452270 27929 raft_consensus.cc:3060] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:19.461654 27978 leader_election.cc:290] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 30 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:19.464598 27570 consensus_queue.cc:260] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 28.21, Last appended by leader: 21, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:19.465368 28012 consensus_meta.cc:287] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.502s	user 0.005s	sys 0.004s
I20260709 12:25:19.466141 27570 raft_consensus.cc:3060] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:19.466010 28012 raft_consensus.cc:515] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:19.468024 27978 raft_consensus.cc:2749] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
W20260709 12:25:19.478837 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 181189us
I20260709 12:25:19.445015 28045 raft_consensus.cc:3060] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:25:19.481371 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 88156.9us
I20260709 12:25:19.485226 17727 leader_election.cc:304] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:19.486779 17727 leader_election.cc:304] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:19.489293 18364 raft_consensus.cc:1275] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 26 index: 18. Preceding OpId from leader: term: 27 index: 19. (index mismatch)
W20260709 12:25:19.492720 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 100569us
I20260709 12:25:19.438251 28011 raft_consensus.cc:697] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
W20260709 12:25:19.480202 27577 consensus_meta.cc:287] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.599s	user 0.010s	sys 0.001s
W20260709 12:25:19.482705 28096 log.cc:927] Time spent T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.099s	user 0.000s	sys 0.000s
W20260709 12:25:19.482509 28076 log.cc:927] Time spent T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.331s	user 0.000s	sys 0.001s
I20260709 12:25:19.439651 28040 raft_consensus.cc:515] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:25:19.503371 18371 consensus_meta.cc:287] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.602s	user 0.010s	sys 0.000s
I20260709 12:25:19.509835 18371 raft_consensus.cc:2468] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 31.
I20260709 12:25:19.445878 27993 raft_consensus.cc:515] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:19.504590 17803 leader_election.cc:304] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:25:19.496434 28103 log.cc:927] Time spent T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.094s	user 0.000s	sys 0.000s
W20260709 12:25:19.505607 27753 consensus_meta.cc:287] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.615s	user 0.008s	sys 0.004s
W20260709 12:25:19.495597 28081 log.cc:927] Time spent T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.232s	user 0.000s	sys 0.001s
I20260709 12:25:19.495741 28115 raft_consensus.cc:493] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:25:19.484050 27525 raft_consensus.cc:515] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:19.495617 28113 raft_consensus.cc:2804] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 35 FOLLOWER]: Leader election won for term 35
I20260709 12:25:19.533883 17838 raft_consensus.cc:2468] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 33.
W20260709 12:25:19.523602 27976 consensus_meta.cc:287] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.630s	user 0.007s	sys 0.006s
W20260709 12:25:19.532949 28077 log.cc:927] Time spent T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.311s	user 0.000s	sys 0.001s
I20260709 12:25:19.523815 27892 raft_consensus.cc:2749] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 34 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:25:19.508391 27577 raft_consensus.cc:515] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:19.559232 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 150657us
I20260709 12:25:19.526175 28110 raft_consensus.cc:493] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:19.559438 17760 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36572 (request call id 1821) took 1215 ms. Trace:
I20260709 12:25:19.521921 27753 raft_consensus.cc:515] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 32 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:19.514413 28099 raft_consensus.cc:515] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Starting leader election with config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:19.575922 28091 log.cc:927] Time spent T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.251s	user 0.000s	sys 0.001s
W20260709 12:25:19.576702 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 80672.7us
I20260709 12:25:19.577226 28099 leader_election.cc:290] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 29 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:19.561084 17760 rpcz_store.cc:276] 0709 12:25:18.348699 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:18.351872 (+  3173us) service_pool.cc:224] Handling call
0709 12:25:18.389444 (+ 37572us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:18.462934 (+ 73490us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:18.462965 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:18.470951 (+  7986us) log.cc:844] Serialized 26 byte log entry
0709 12:25:18.809479 (+338528us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:25:18.809536 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:19.348461 (+538925us) spinlock_profiling.cc:243] Waited 25.4 ms on lock 0x7b5400194e54. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:25:19.348523 (+    62us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:19.548074 (+199551us) raft_consensus.cc:1685] finished
0709 12:25:19.548104 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:19.559369 (+ 11265us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":36319,"raft.queue_time_us":22857,"raft.run_cpu_time_us":109,"raft.run_wall_time_us":7695,"spinlock_wait_cycles":58414464,"thread_start_us":300292,"threads_started":1,"wal-append.queue_time_us":495681}
W20260709 12:25:19.578383 17728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:19.541159 28113 raft_consensus.cc:697] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 35 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
W20260709 12:25:19.567050 28106 log.cc:927] Time spent T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.061s	user 0.001s	sys 0.000s
W20260709 12:25:19.567618 28101 log.cc:927] Time spent T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.087s	user 0.000s	sys 0.000s
I20260709 12:25:19.568043 28110 raft_consensus.cc:515] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:19.579928 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 43725.4us
I20260709 12:25:19.564222 27744 raft_consensus.cc:2749] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:25:19.563118 28012 leader_election.cc:290] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 29 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:25:19.599277 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 112607us
I20260709 12:25:19.556777 27976 raft_consensus.cc:515] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:19.536512 28115 raft_consensus.cc:515] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:19.582789 28099 raft_consensus.cc:2749] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:25:19.611310 27976 leader_election.cc:290] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 30 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:19.611699 17803 leader_election.cc:304] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:25:19.584710 28011 consensus_queue.cc:237] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 25.17, Last appended by leader: 17, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:19.617434 18333 leader_election.cc:304] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:25:19.629166 17655 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:19.626559 27898 leader_election.cc:290] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 34 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:19.628582 17763 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38776 (request call id 3424) took 1464 ms. Trace:
I20260709 12:25:19.630915 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:25:19.631704 17684 raft_consensus.cc:2368] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 29. Current term is 30.
I20260709 12:25:19.631593 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:25:19.630753 17763 rpcz_store.cc:276] 0709 12:25:18.163978 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:18.165295 (+  1317us) service_pool.cc:224] Handling call
0709 12:25:18.628989 (+463694us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:18.664967 (+ 35978us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:18.665010 (+    43us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:18.680616 (+ 15606us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:25:18.680675 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:18.680900 (+   225us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:19.628388 (+947488us) spinlock_profiling.cc:243] Waited 463 ms on lock 0x7b5400400bd0. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a61f8ad 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:25:19.628521 (+   133us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1065482624}
I20260709 12:25:19.616346 17729 leader_election.cc:304] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:25:19.636276 28087 raft_consensus.cc:2749] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:25:19.640944 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:25:19.652354 27929 raft_consensus.cc:2749] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:25:19.655552 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:25:19.656526 17684 raft_consensus.cc:3060] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Advancing to term 31
W20260709 12:25:19.662066 28116 log.cc:927] Time spent T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.055s	user 0.001s	sys 0.014s
I20260709 12:25:19.663368 28080 leader_election.cc:290] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:19.695597 27577 leader_election.cc:290] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 27 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:19.696274 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 30 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:19.696449 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:19.696861 17758 raft_consensus.cc:3060] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:19.697283 28040 leader_election.cc:290] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 33 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:25:19.698948 28115 leader_election.cc:290] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:19.700224 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 30 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:25:19.700440 17683 raft_consensus.cc:1240] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 28. Current term is 29. Ops: [26.19-28.20]
I20260709 12:25:19.700948 17682 raft_consensus.cc:2393] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 30: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:25:19.648581 27892 raft_consensus.cc:2749] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:25:19.701597 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:19.701737 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
W20260709 12:25:19.649127 28033 consensus_meta.cc:287] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.921s	user 0.013s	sys 0.000s
I20260709 12:25:19.702849 28033 raft_consensus.cc:515] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:19.703485 28036 raft_consensus.cc:493] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:25:19.703547 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 29 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:19.704396 17837 raft_consensus.cc:2468] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 28.
I20260709 12:25:19.704198 28036 raft_consensus.cc:515] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:19.704981 28033 leader_election.cc:290] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 30 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:19.705404 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 33 candidate_status { last_received { term: 32 index: 23 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:19.706095 17830 raft_consensus.cc:2468] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 32.
I20260709 12:25:19.664054 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 33 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:19.706308 28036 leader_election.cc:290] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:25:19.673300 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 57243.7us
I20260709 12:25:19.707413 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:19.708117 17829 raft_consensus.cc:2468] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 26.
I20260709 12:25:19.708387 17729 leader_election.cc:304] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:25:19.664479 17681 raft_consensus.cc:1275] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 26 index: 19. Preceding OpId from leader: term: 28 index: 20. (index mismatch)
W20260709 12:25:19.668291 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 50952.2us
W20260709 12:25:19.663538 17838 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35042 (request call id 3605) took 1382 ms (client timeout 1641 ms). Trace:
I20260709 12:25:19.669863 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:19.671307 17763 raft_consensus.cc:1275] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 28 index: 15. Preceding OpId from leader: term: 29 index: 16. (index mismatch)
I20260709 12:25:19.668838 27989 raft_consensus.cc:2749] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:25:19.709717 18333 leader_election.cc:304] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:25:19.705067 27671 consensus_meta.cc:287] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.590s	user 0.010s	sys 0.000s
I20260709 12:25:19.712790 18333 leader_election.cc:304] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:19.720639 18333 leader_election.cc:304] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:25:19.720841 17801 leader_election.cc:304] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:25:19.654764 28028 raft_consensus.cc:3055] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:25:19.723662 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:19.723995 27993 leader_election.cc:290] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:25:19.710683 17838 rpcz_store.cc:269] 0709 12:25:18.284338 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:18.289834 (+  5496us) service_pool.cc:224] Handling call
0709 12:25:19.663332 (+1373498us) spinlock_profiling.cc:243] Waited 1.22 s on lock 0x7b54001581d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:25:19.663448 (+   116us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2795657472}
I20260709 12:25:19.661015 28054 leader_election.cc:290] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 30 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:25:19.741139 17836 raft_consensus.cc:1240] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 25 LEADER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 24. Current term is 25. Ops: []
I20260709 12:25:19.724965 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 34 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:25:19.726643 28080 raft_consensus.cc:2804] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Leader pre-election won for term 27
W20260709 12:25:19.727890 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 56669.5us
I20260709 12:25:19.728242 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 33 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:19.746778 17758 raft_consensus.cc:2468] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 30.
W20260709 12:25:19.730212 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 118615us
W20260709 12:25:19.690888 28072 log.cc:927] Time spent T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 1.094s	user 0.012s	sys 0.000s
I20260709 12:25:19.732257 17689 raft_consensus.cc:1240] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 27. Current term is 28. Ops: []
I20260709 12:25:19.732479 17755 raft_consensus.cc:1275] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 24 index: 18. Preceding OpId from leader: term: 28 index: 20. (index mismatch)
I20260709 12:25:19.710052 28117 raft_consensus.cc:2749] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:25:19.733129 17729 leader_election.cc:304] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
W20260709 12:25:19.724740 28122 log.cc:927] Time spent T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.076s	user 0.000s	sys 0.000s
I20260709 12:25:19.740305 17801 leader_election.cc:304] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:19.743736 17756 raft_consensus.cc:2393] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 29: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:25:19.756053 17684 raft_consensus.cc:2468] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 31.
I20260709 12:25:19.745183 17680 raft_consensus.cc:3060] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:25:19.729691 28017 raft_consensus.cc:2804] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:25:19.749989 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 33 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:19.745890 28080 raft_consensus.cc:493] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:19.727986 27881 raft_consensus.cc:2749] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:25:19.754827 17729 leader_election.cc:304] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:25:19.740942 28099 raft_consensus.cc:2749] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:25:19.760589 17803 leader_election.cc:304] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:25:19.758177 17762 raft_consensus.cc:2368] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 34 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 33. Current term is 34.
I20260709 12:25:19.733085 28028 raft_consensus.cc:740] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:25:19.748134 28040 raft_consensus.cc:2804] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:25:19.762689 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:19.769368 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:19.774303 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 35 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:19.777423 17761 raft_consensus.cc:3060] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:19.777794 17683 raft_consensus.cc:2410] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for term 35 because replica has last-logged OpId of term: 34 index: 22, which is greater than that of the candidate, which has last-logged OpId of term: 33 index: 21.
I20260709 12:25:19.773733 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 30 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:19.773837 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 29 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:19.779845 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 33 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:19.779922 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 33 candidate_status { last_received { term: 32 index: 23 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:19.781401 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:19.742063 28087 consensus_queue.cc:1059] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 30.18, Next index: 19, Last known committed idx: 17, Time since last communication: 0.026s
I20260709 12:25:19.741936 28112 consensus_queue.cc:1059] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 0.0, Next index: 20, Last known committed idx: 18, Time since last communication: 0.026s
I20260709 12:25:19.788885 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 30 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:19.803570 17836 raft_consensus.cc:2393] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 30: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:25:19.741269 27671 raft_consensus.cc:515] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:19.761250 28080 raft_consensus.cc:3060] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:19.797271 17760 raft_consensus.cc:2393] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 30: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:25:19.768757 28111 raft_consensus.cc:2749] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:25:19.773425 28033 raft_consensus.cc:2749] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:25:19.785837 27577 consensus_queue.cc:1048] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 21, Time since last communication: 0.027s
I20260709 12:25:19.806980 28036 raft_consensus.cc:2804] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:25:19.810097 27965 raft_consensus.cc:493] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:25:19.810468 28110 leader_election.cc:290] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:19.811323 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 30 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:19.812237 17759 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36572 (request call id 1819) took 1536 ms. Trace:
I20260709 12:25:19.889942 17756 raft_consensus.cc:2393] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 30: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:25:19.889720 17759 rpcz_store.cc:276] 0709 12:25:18.275604 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:18.275753 (+   149us) service_pool.cc:224] Handling call
0709 12:25:18.288057 (+ 12304us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:18.288416 (+   359us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:18.288448 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:18.288964 (+   516us) log.cc:844] Serialized 26 byte log entry
0709 12:25:18.462940 (+173976us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:25:18.462977 (+    37us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:18.463039 (+    62us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:19.616704 (+1153665us) spinlock_profiling.cc:243] Waited 64.6 ms on lock 0x7b24000f7388. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b473b99c4 00007f7b473b7acb 00007f7b47572dc9 00007f7b47572a55 00007f7b4757209d 00007f7b4a1a410e 00007f7b4a1bb88c 00007f7b4a1babb6 00007f7b4a1ba7d8 00007f7b4a1c4221
0709 12:25:19.811949 (+195245us) raft_consensus.cc:1685] finished
0709 12:25:19.811972 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:19.812180 (+   208us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1458,"raft.run_cpu_time_us":207,"raft.run_wall_time_us":19935,"spinlock_wait_cycles":148593024,"thread_start_us":173462,"threads_started":1,"wal-append.queue_time_us":306939}
I20260709 12:25:19.813140 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 30 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:19.814039 17761 raft_consensus.cc:2468] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 30.
I20260709 12:25:19.814275 17755 raft_consensus.cc:1240] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 LEADER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 26. Current term is 27. Ops: []
I20260709 12:25:19.814843 27881 raft_consensus.cc:493] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:25:19.817665 18333 leader_election.cc:304] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:19.819165 28028 consensus_queue.cc:260] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 24.22, Last appended by leader: 22, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:19.723491 28113 consensus_queue.cc:237] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 33.25, Last appended by leader: 25, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:19.828066 28119 raft_consensus.cc:493] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 34 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:25:19.770030 28054 raft_consensus.cc:2749] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:25:19.758731 28017 raft_consensus.cc:493] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:25:19.834867 28112 consensus_queue.cc:1243] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer e2909cff1da44ad387f96a8ae3aa5afb log is divergent from this leader: its last log entry 27.19 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 18
I20260709 12:25:19.840962 27993 raft_consensus.cc:493] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:19.843302 17682 raft_consensus.cc:1240] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 24. Current term is 25. Ops: [24.22-24.22]
I20260709 12:25:19.843636 27903 leader_election.cc:290] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:19.843758 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:19.843344 27525 leader_election.cc:290] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:19.848495 28087 raft_consensus.cc:3055] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:25:19.849494 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:19.921864 17758 raft_consensus.cc:2468] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 26.
I20260709 12:25:19.851192 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 31 candidate_status { last_received { term: 30 index: 21 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:19.804178 17681 raft_consensus.cc:2468] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 32.
I20260709 12:25:19.887719 17802 leader_election.cc:400] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 29 election: Vote denied by peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071) with higher term. Message: Invalid argument: T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 29. Current term is 30.
I20260709 12:25:19.906309 28017 raft_consensus.cc:3060] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:19.866994 18374 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 2932) took 2271 ms. Trace:
I20260709 12:25:19.892472 17762 raft_consensus.cc:2393] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 30: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:25:19.941226 18374 rpcz_store.cc:276] 0709 12:25:17.595526 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:18.033623 (+438097us) service_pool.cc:224] Handling call
0709 12:25:18.033969 (+   346us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:18.902896 (+868927us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:19.866670 (+963774us) spinlock_profiling.cc:243] Waited 848 ms on lock 0x7b54006fe354. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:25:19.866916 (+   246us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1949797120}
I20260709 12:25:19.776594 28040 raft_consensus.cc:493] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:25:19.897312 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 79724.2us
I20260709 12:25:19.902433 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 33 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:25:19.893738 27965 raft_consensus.cc:515] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:19.898602 27881 raft_consensus.cc:515] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:19.907092 17685 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37396 (request call id 3535) took 1472 ms. Trace:
I20260709 12:25:19.892062 28035 raft_consensus.cc:493] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:19.885919 28036 raft_consensus.cc:493] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:25:19.918538 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
W20260709 12:25:19.919211 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 165486us
I20260709 12:25:19.920920 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:19.908540 27989 raft_consensus.cc:2804] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:25:19.983307 18333 leader_election.cc:304] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:19.940207 17802 leader_election.cc:403] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 29 election: Cancelling election due to peer responding with higher term
I20260709 12:25:19.940214 27525 raft_consensus.cc:493] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:19.945941 27766 consensus_meta.cc:287] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 1.005s	user 0.009s	sys 0.003s
I20260709 12:25:19.946520 28112 consensus_queue.cc:1059] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.001s
I20260709 12:25:19.950491 17680 raft_consensus.cc:2468] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 34.
I20260709 12:25:19.921612 28087 raft_consensus.cc:740] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 30 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:25:19.954150 17836 raft_consensus.cc:1275] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 26 index: 18. Preceding OpId from leader: term: 27 index: 19. (index mismatch)
I20260709 12:25:19.955437 17838 raft_consensus.cc:1275] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 32 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 29 index: 17. Preceding OpId from leader: term: 32 index: 19. (index mismatch)
I20260709 12:25:19.956167 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:19.905609 28028 raft_consensus.cc:3060] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:19.920859 28105 consensus_queue.cc:1059] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 25.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.072s
I20260709 12:25:19.982769 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 31 candidate_status { last_received { term: 30 index: 21 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:25:19.984283 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 107323us
I20260709 12:25:19.873625 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:19.996606 17801 leader_election.cc:400] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 33 election: Vote denied by peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651) with higher term. Message: Invalid argument: T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 34 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 33. Current term is 34.
I20260709 12:25:19.996929 17801 leader_election.cc:403] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 33 election: Cancelling election due to peer responding with higher term
I20260709 12:25:19.963554 28040 raft_consensus.cc:3060] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Advancing to term 31
W20260709 12:25:19.984340 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 316990us
I20260709 12:25:19.991573 17832 raft_consensus.cc:1240] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 29. Current term is 30. Ops: []
I20260709 12:25:19.962334 27856 raft_consensus.cc:493] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:25:19.917980 27993 raft_consensus.cc:515] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:19.992041 17829 raft_consensus.cc:2468] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 29.
I20260709 12:25:19.908298 28119 raft_consensus.cc:515] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:19.967058 17685 rpcz_store.cc:276] 0709 12:25:18.435041 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:18.499867 (+ 64826us) service_pool.cc:224] Handling call
0709 12:25:18.574009 (+ 74142us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:18.773581 (+199572us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:18.773629 (+    48us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:18.785368 (+ 11739us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:25:18.785424 (+    56us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:19.906798 (+1121374us) spinlock_profiling.cc:243] Waited 188 ms on lock 0x7b54001388d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:25:19.906861 (+    63us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:19.907022 (+   161us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":432847616}
I20260709 12:25:19.994441 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:20.024467 17837 raft_consensus.cc:2468] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 27.
I20260709 12:25:19.988350 27431 raft_consensus.cc:3060] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:20.013736 18364 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 2949) took 1025 ms. Trace:
I20260709 12:25:20.027998 18364 rpcz_store.cc:276] 0709 12:25:18.988025 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:19.206021 (+217996us) service_pool.cc:224] Handling call
0709 12:25:19.206365 (+   344us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:19.489873 (+283508us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:20.001538 (+511665us) spinlock_profiling.cc:243] Waited 277 ms on lock 0x7b540034b554. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:25:20.013643 (+ 12105us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":637403008}
I20260709 12:25:20.005532 17801 leader_election.cc:304] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:25:20.022882 17728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:20.032414 17801 leader_election.cc:304] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:20.026743 17729 leader_election.cc:304] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:20.012835 17802 leader_election.cc:304] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:19.983853 27989 raft_consensus.cc:493] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 32 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:25:19.985003 28036 raft_consensus.cc:3060] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:19.971076 17762 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3"
dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
mode: GRACEFUL
new_leader_uuid: "e063541542a94969898868bf6eb5ce1b"
 from {username='slave'} at 127.0.0.1:49786
I20260709 12:25:20.038465 17762 raft_consensus.cc:606] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 32 FOLLOWER]: Received request to transfer leadership to TS e063541542a94969898868bf6eb5ce1b
I20260709 12:25:20.038833 17762 raft_consensus.cc:614] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 32 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:25:20.039160 27881 leader_election.cc:290] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:25:20.039812 27881 raft_consensus.cc:493] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:25:20.035655 27856 raft_consensus.cc:515] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:20.040411 27881 raft_consensus.cc:515] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:20.042832 27881 leader_election.cc:290] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:20.043466 27881 raft_consensus.cc:2749] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
I20260709 12:25:20.031033 17652 leader_election.cc:304] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:20.036825 17833 raft_consensus.cc:2468] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 30.
I20260709 12:25:20.027154 17803 leader_election.cc:304] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:20.020735 28041 raft_consensus.cc:3060] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:25:19.985574 28035 raft_consensus.cc:515] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:20.068444 28087 consensus_queue.cc:260] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 30.18, Last appended by leader: 18, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:20.069865 28087 raft_consensus.cc:3060] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:20.072232 17762 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690"
dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
mode: GRACEFUL
new_leader_uuid: "e063541542a94969898868bf6eb5ce1b"
 from {username='slave'} at 127.0.0.1:49786
I20260709 12:25:20.072953 17762 raft_consensus.cc:606] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 34 FOLLOWER]: Received request to transfer leadership to TS e063541542a94969898868bf6eb5ce1b
I20260709 12:25:20.073339 17762 raft_consensus.cc:614] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 34 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:25:20.072444 28036 raft_consensus.cc:515] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:20.076292 28036 leader_election.cc:290] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 31 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:20.086117 27671 leader_election.cc:290] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 22 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:25:20.088141 28130 log.cc:927] Time spent T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.118s	user 0.001s	sys 0.000s
I20260709 12:25:20.090374 28119 leader_election.cc:290] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:20.044186 27892 raft_consensus.cc:2749] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260709 12:25:19.992839 27525 raft_consensus.cc:515] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:20.095082 27525 leader_election.cc:290] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:20.044500 27989 raft_consensus.cc:3060] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 32 FOLLOWER]: Advancing to term 33
W20260709 12:25:20.107229 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 253016us
W20260709 12:25:20.088169 27800 consensus_meta.cc:287] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.642s	user 0.007s	sys 0.003s
I20260709 12:25:20.016027 27766 raft_consensus.cc:515] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Starting leader election with config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:20.052487 28121 raft_consensus.cc:2749] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
W20260709 12:25:20.141350 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 74984.3us
I20260709 12:25:20.055562 28010 raft_consensus.cc:2804] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Leader election won for term 31
W20260709 12:25:20.142160 17654 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:20.145262 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:20.147504 17838 raft_consensus.cc:1275] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 21 index: 15. Preceding OpId from leader: term: 22 index: 16. (index mismatch)
I20260709 12:25:20.147403 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 32 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:20.140544 17728 leader_election.cc:304] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:20.103041 28043 raft_consensus.cc:2804] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Leader election won for term 30
W20260709 12:25:20.148532 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 116194us
W20260709 12:25:20.150137 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 166951us
I20260709 12:25:20.151338 17728 leader_election.cc:304] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:20.133206 27892 raft_consensus.cc:2749] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
I20260709 12:25:20.141258 28036 raft_consensus.cc:2749] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:25:20.168519 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 29 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:20.172027 17688 raft_consensus.cc:2393] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 29: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:25:20.172943 18364 raft_consensus.cc:1275] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 24 index: 17. Preceding OpId from leader: term: 26 index: 18. (index mismatch)
I20260709 12:25:20.174458 18367 raft_consensus.cc:2410] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for term 29 because replica has last-logged OpId of term: 28 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 26 index: 13.
W20260709 12:25:20.175200 17801 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:20.183591 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 29 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:20.184314 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:20.186527 17757 raft_consensus.cc:2468] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 28.
I20260709 12:25:20.184886 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 30 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:20.188454 18366 raft_consensus.cc:2393] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 30: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
W20260709 12:25:20.190341 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 177736us
I20260709 12:25:20.192799 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 26 candidate_status { last_received { term: 23 index: 21 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:20.193418 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 29 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:20.200637 17802 leader_election.cc:304] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:20.202490 17802 leader_election.cc:304] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:20.153026 28010 raft_consensus.cc:697] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 31 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:25:20.169011 27978 raft_consensus.cc:493] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:20.175673 27577 raft_consensus.cc:2804] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 34 FOLLOWER]: Leader election won for term 34
I20260709 12:25:20.144956 27800 raft_consensus.cc:515] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:20.204772 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:20.214131 17833 raft_consensus.cc:2468] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 25.
I20260709 12:25:20.208738 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 29 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:20.210215 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:20.216256 18363 raft_consensus.cc:2468] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 26.
I20260709 12:25:20.210769 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:20.217126 18365 raft_consensus.cc:2376] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 27. Re-sending same reply.
I20260709 12:25:20.169023 28043 raft_consensus.cc:697] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 30 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:25:20.209501 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 29 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:20.219046 18361 raft_consensus.cc:3060] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:20.193027 28033 raft_consensus.cc:2804] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:25:20.194796 28121 raft_consensus.cc:493] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:20.223033 18368 raft_consensus.cc:2468] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 29.
W20260709 12:25:20.225405 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 76369.4us
W20260709 12:25:20.226658 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 61353.8us
W20260709 12:25:20.215349 28128 log.cc:927] Time spent T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.251s	user 0.001s	sys 0.001s
I20260709 12:25:20.234030 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:20.234663 18365 raft_consensus.cc:3060] T a5732e71b9d54f8786b8c511d69943bf P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:20.237246 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:20.237867 18363 raft_consensus.cc:3060] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:25:20.240242 17835 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35040 (request call id 3805) took 1731 ms (client timeout 1865 ms). Trace:
W20260709 12:25:20.240633 17835 rpcz_store.cc:269] 0709 12:25:18.508770 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:18.575556 (+ 66786us) service_pool.cc:224] Handling call
0709 12:25:20.240051 (+1664495us) spinlock_profiling.cc:243] Waited 1.6 s on lock 0x7b540022b354. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:25:20.240190 (+   139us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3680176896}
W20260709 12:25:20.241110 17801 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:20.242548 28095 raft_consensus.cc:3055] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:25:20.242936 28095 raft_consensus.cc:740] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:25:20.221585 28033 raft_consensus.cc:493] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:20.243822 18365 raft_consensus.cc:2468] T a5732e71b9d54f8786b8c511d69943bf P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 29.
I20260709 12:25:20.245057 17834 raft_consensus.cc:1240] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 28 LEADER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 27. Current term is 28. Ops: []
I20260709 12:25:20.256332 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 30 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:20.258569 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 35 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:20.218327 27577 raft_consensus.cc:697] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 34 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:25:20.261699 18366 raft_consensus.cc:2393] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 30: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:25:20.243793 28095 consensus_queue.cc:260] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 28.20, Last appended by leader: 20, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:20.216624 27978 raft_consensus.cc:515] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:20.262121 17762 raft_consensus.cc:1275] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 27 index: 17. Preceding OpId from leader: term: 30 index: 19. (index mismatch)
I20260709 12:25:20.272156 17757 raft_consensus.cc:1240] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 29. Current term is 30. Ops: []
I20260709 12:25:20.261435 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 33 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:25:20.276665 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 74240.5us
I20260709 12:25:20.244439 28033 raft_consensus.cc:3060] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:25:20.279912 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:20.282150 17763 raft_consensus.cc:3060] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:20.265023 28149 consensus_queue.cc:1048] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.014s
I20260709 12:25:20.265106 27892 raft_consensus.cc:2804] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:25:20.284410 17829 raft_consensus.cc:1275] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 34 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 32 index: 22. Preceding OpId from leader: term: 34 index: 23. (index mismatch)
I20260709 12:25:20.285063 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 29 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:20.290549 18363 raft_consensus.cc:2468] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 27.
I20260709 12:25:20.273097 28057 raft_consensus.cc:2749] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:25:20.297598 17654 leader_election.cc:304] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:25:20.236475 28121 raft_consensus.cc:515] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:20.302104 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 30 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:20.304772 17829 raft_consensus.cc:2410] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for term 30 because replica has last-logged OpId of term: 29 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 28 index: 15.
I20260709 12:25:20.305691 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:20.305856 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:20.306463 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:20.306658 17833 raft_consensus.cc:2393] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 29: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:25:20.306856 18363 raft_consensus.cc:3060] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 21 FOLLOWER]: Advancing to term 22
W20260709 12:25:20.307372 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 63132.3us
I20260709 12:25:20.282375 28146 raft_consensus.cc:493] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:25:20.316980 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 33 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:20.329032 17829 raft_consensus.cc:2468] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 32.
I20260709 12:25:20.336935 17654 leader_election.cc:304] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:20.304132 27993 leader_election.cc:290] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:20.349452 17729 leader_election.cc:304] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:25:20.338967 17652 leader_election.cc:304] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:20.245606 28010 consensus_queue.cc:237] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 30.18, Last appended by leader: 18, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:20.303188 27892 raft_consensus.cc:697] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 31 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
W20260709 12:25:20.338940 28131 log.cc:927] Time spent T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.238s	user 0.001s	sys 0.001s
I20260709 12:25:20.317939 28095 raft_consensus.cc:3060] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Advancing to term 29
W20260709 12:25:20.384264 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 45372.7us
W20260709 12:25:20.357694 28045 consensus_meta.cc:287] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.850s	user 0.013s	sys 0.000s
W20260709 12:25:20.385313 28138 log.cc:927] Time spent T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.071s	user 0.000s	sys 0.000s
I20260709 12:25:20.355526 17729 leader_election.cc:304] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:25:20.412415 18363 raft_consensus.cc:2468] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 22.
I20260709 12:25:20.304405 28043 consensus_queue.cc:237] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 28.16, Last appended by leader: 16, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:20.357071 28113 raft_consensus.cc:2749] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:25:20.391327 28046 raft_consensus.cc:2749] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:25:20.391378 28157 raft_consensus.cc:493] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:20.389621 28035 leader_election.cc:290] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:25:20.425410 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 82334.2us
I20260709 12:25:20.352080 28119 consensus_queue.cc:1048] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 17, Time since last communication: 0.068s
I20260709 12:25:20.339315 28146 raft_consensus.cc:515] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:20.427520 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:25:20.437077 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 81655.2us
I20260709 12:25:20.439201 17729 leader_election.cc:304] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:20.441661 17729 leader_election.cc:304] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:20.355095 27577 consensus_queue.cc:237] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 32.18, Last appended by leader: 18, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:20.430130 27800 leader_election.cc:290] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 26 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:20.416021 28045 raft_consensus.cc:515] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:20.473613 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 72969.1us
I20260709 12:25:20.458563 28099 raft_consensus.cc:2804] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:25:20.448491 27525 raft_consensus.cc:2804] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:25:20.491000 28112 raft_consensus.cc:493] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:20.491745 28112 raft_consensus.cc:515] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:20.492647 28153 consensus_queue.cc:1048] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:25:20.494299 28112 leader_election.cc:290] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:25:20.482403 28142 log.cc:927] Time spent T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.256s	user 0.000s	sys 0.001s
I20260709 12:25:20.477934 27767 raft_consensus.cc:493] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:20.435308 28157 raft_consensus.cc:515] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:20.538967 28144 raft_consensus.cc:493] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:20.542311 28099 raft_consensus.cc:697] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:25:20.551751 28133 raft_consensus.cc:2804] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Leader pre-election won for term 32
W20260709 12:25:20.564760 28143 log.cc:927] Time spent T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.232s	user 0.000s	sys 0.001s
I20260709 12:25:20.546895 27525 raft_consensus.cc:697] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:25:20.480420 28113 consensus_queue.cc:1048] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.079s
I20260709 12:25:20.563807 28112 raft_consensus.cc:493] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
W20260709 12:25:20.587801 28145 log.cc:927] Time spent T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.268s	user 0.001s	sys 0.000s
W20260709 12:25:20.568542 28040 consensus_meta.cc:287] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.541s	user 0.006s	sys 0.004s
W20260709 12:25:20.585095 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 562406us
W20260709 12:25:20.587491 28161 log.cc:927] Time spent T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.107s	user 0.000s	sys 0.001s
I20260709 12:25:20.567184 28149 raft_consensus.cc:493] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:20.588654 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 34 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:20.590642 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 34 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:20.497107 27892 consensus_queue.cc:237] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 30.20, Last appended by leader: 20, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:20.572674 28133 raft_consensus.cc:493] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
W20260709 12:25:20.588068 28139 log.cc:927] Time spent T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.221s	user 0.000s	sys 0.001s
I20260709 12:25:20.574555 28159 raft_consensus.cc:2804] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:25:20.599720 18361 raft_consensus.cc:2468] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 29.
W20260709 12:25:20.607184 28163 log.cc:927] Time spent T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.122s	user 0.000s	sys 0.001s
W20260709 12:25:20.607566 28135 log.cc:927] Time spent T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.280s	user 0.000s	sys 0.001s
I20260709 12:25:20.613526 17763 raft_consensus.cc:2468] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 30.
I20260709 12:25:20.620775 28040 raft_consensus.cc:515] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:25:20.666842 27989 consensus_meta.cc:287] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.524s	user 0.008s	sys 0.004s
I20260709 12:25:20.669032 28157 leader_election.cc:290] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:20.670267 28045 leader_election.cc:290] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 27 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:20.558543 27767 raft_consensus.cc:515] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:20.670667 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:20.680140 28054 consensus_queue.cc:1048] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 15, Time since last communication: 0.001s
W20260709 12:25:20.680925 28080 consensus_meta.cc:287] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.825s	user 0.006s	sys 0.006s
I20260709 12:25:20.623929 28112 raft_consensus.cc:515] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:20.682176 28017 consensus_meta.cc:287] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.716s	user 0.008s	sys 0.002s
I20260709 12:25:20.684435 28166 raft_consensus.cc:493] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:20.685310 18333 leader_election.cc:304] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:25:20.686646 28168 consensus_queue.cc:1059] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:25:20.687839 17727 leader_election.cc:304] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:20.688185 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:25:20.687724 28110 consensus_queue.cc:1048] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.001s
I20260709 12:25:20.592531 28126 raft_consensus.cc:2804] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:25:20.569460 28144 raft_consensus.cc:515] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:20.608268 28121 leader_election.cc:290] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:20.653471 28171 raft_consensus.cc:493] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:25:20.693661 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
W20260709 12:25:20.759222 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 73983.7us
I20260709 12:25:20.699555 17803 leader_election.cc:304] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:25:20.711292 28167 consensus_queue.cc:1048] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.036s
I20260709 12:25:20.706202 17760 raft_consensus.cc:1275] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 35 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 33 index: 25. Preceding OpId from leader: term: 35 index: 26. (index mismatch)
I20260709 12:25:20.710727 17755 raft_consensus.cc:1275] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 21 index: 15. Preceding OpId from leader: term: 26 index: 18. (index mismatch)
W20260709 12:25:20.663712 28087 consensus_meta.cc:287] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.593s	user 0.008s	sys 0.000s
I20260709 12:25:20.716156 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:20.718546 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:20.768193 17761 raft_consensus.cc:2468] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 22.
I20260709 12:25:20.715662 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 34 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:20.729125 17652 leader_election.cc:304] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:20.769910 17801 leader_election.cc:304] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:20.771132 17758 raft_consensus.cc:2393] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 29: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:25:20.726001 28149 raft_consensus.cc:515] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:25:20.716648 28041 consensus_meta.cc:287] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.651s	user 0.011s	sys 0.000s
W20260709 12:25:20.717126 28028 consensus_meta.cc:287] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.712s	user 0.010s	sys 0.002s
I20260709 12:25:20.713527 28035 raft_consensus.cc:493] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:20.740290 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 270285us
I20260709 12:25:20.710952 28133 raft_consensus.cc:3060] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:25:20.768424 28168 consensus_queue.cc:1243] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer e2909cff1da44ad387f96a8ae3aa5afb log is divergent from this leader: its last log entry 27.18 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 16
I20260709 12:25:20.748710 17830 raft_consensus.cc:1240] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 30. Current term is 31. Ops: []
I20260709 12:25:20.749533 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:20.788225 17831 raft_consensus.cc:2468] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 27.
I20260709 12:25:20.788807 17831 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37994 (request call id 1894) took 1088 ms. Trace:
I20260709 12:25:20.789129 17831 rpcz_store.cc:276] 0709 12:25:19.700628 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:19.707172 (+  6544us) service_pool.cc:224] Handling call
0709 12:25:20.749377 (+1042205us) spinlock_profiling.cc:243] Waited 1.7 ms on lock 0x7b580013fe98. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b474fbb71 00007f7b474fba22 00007f7b504148d9 00007f7b4f56eee0 00007f7b4f6380ab 00007f7b4f638131 00007f7b4f5e60a0 00007f7b4f5e5984 00007f7b48b606d4 00007f7b48b60662
0709 12:25:20.788721 (+ 39344us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3901184}
I20260709 12:25:20.790385 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:20.734308 28146 leader_election.cc:290] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:20.737143 27525 consensus_queue.cc:237] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 25.18, Last appended by leader: 18, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:20.790854 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 29 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:20.791414 17836 raft_consensus.cc:3060] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:20.746902 28080 raft_consensus.cc:515] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:20.760547 18333 leader_election.cc:304] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:20.793275 28080 leader_election.cc:290] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 27 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:20.761245 27767 leader_election.cc:290] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:20.762012 27753 leader_election.cc:290] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 32 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:25:20.762717 28144 leader_election.cc:290] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:20.749115 28166 raft_consensus.cc:515] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:20.762836 17834 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37994 (request call id 1893) took 1062 ms. Trace:
I20260709 12:25:20.796047 17834 rpcz_store.cc:276] 0709 12:25:19.700320 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:19.706745 (+  6425us) service_pool.cc:224] Handling call
0709 12:25:20.244879 (+538134us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:20.255614 (+ 10735us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:20.762648 (+507034us) spinlock_profiling.cc:243] Waited 538 ms on lock 0x7b540022b350. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a61f8ad 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:25:20.762779 (+   131us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1236928384}
I20260709 12:25:20.796730 28166 leader_election.cc:290] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:20.763254 28045 raft_consensus.cc:2749] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:25:20.797745 28057 raft_consensus.cc:2804] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:25:20.664149 28099 consensus_queue.cc:237] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 20.15, Last appended by leader: 15, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:20.790889 17831 raft_consensus.cc:3060] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:20.852903 17688 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44898 (request call id 2024) took 1182 ms. Trace:
I20260709 12:25:20.853372 17688 rpcz_store.cc:276] 0709 12:25:19.670194 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:19.696206 (+ 26012us) service_pool.cc:224] Handling call
0709 12:25:20.852659 (+1156453us) spinlock_profiling.cc:243] Waited 470 ms on lock 0x7b54003507d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:25:20.852807 (+   148us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1081962752}
W20260709 12:25:20.776072 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 76663us
I20260709 12:25:20.854492 17803 leader_election.cc:304] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:20.856626 27881 raft_consensus.cc:3055] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:25:20.857242 17803 leader_election.cc:304] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:25:20.784137 17654 leader_election.cc:304] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:20.783285 28041 raft_consensus.cc:2749] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 34 FOLLOWER]: Leader election lost for term 33. Reason: Vote denied by peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651) with higher term. Message: Invalid argument: T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 34 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 33. Current term is 34.
I20260709 12:25:20.768337 27927 raft_consensus.cc:493] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:20.761288 28126 raft_consensus.cc:697] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 29 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:25:20.780838 28087 raft_consensus.cc:2749] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:25:20.785076 28035 raft_consensus.cc:515] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:20.744405 27431 consensus_meta.cc:287] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.713s	user 0.007s	sys 0.003s
I20260709 12:25:20.798190 17832 raft_consensus.cc:2468] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 31.
I20260709 12:25:20.798921 18371 raft_consensus.cc:2410] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for term 26 because replica has last-logged OpId of term: 24 index: 22, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 21.
I20260709 12:25:20.801405 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:20.769235 17764 raft_consensus.cc:2468] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 32.
I20260709 12:25:20.865582 17829 raft_consensus.cc:2468] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 34.
I20260709 12:25:20.769116 28157 raft_consensus.cc:2804] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Leader pre-election won for term 27
W20260709 12:25:20.831358 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 70867.4us
I20260709 12:25:20.865516 17834 raft_consensus.cc:3060] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Advancing to term 29
I20260709 12:25:20.746595 28017 raft_consensus.cc:515] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:20.802686 28057 raft_consensus.cc:493] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:20.825806 28175 raft_consensus.cc:2804] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:25:20.844580 28184 consensus_queue.cc:1048] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 24, Time since last communication: 0.001s
I20260709 12:25:20.768748 28121 consensus_queue.cc:1059] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.061s
W20260709 12:25:20.885362 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 115538us
I20260709 12:25:20.739008 27989 raft_consensus.cc:515] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 33 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:20.835033 28080 raft_consensus.cc:2749] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
I20260709 12:25:20.888181 18333 leader_election.cc:304] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:25:20.862583 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 174811us
I20260709 12:25:20.863142 17803 leader_election.cc:304] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:20.766526 28171 raft_consensus.cc:515] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:20.857088 27881 raft_consensus.cc:740] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:25:20.889747 17833 raft_consensus.cc:3060] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:20.890040 18374 raft_consensus.cc:1275] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 28 index: 20. Preceding OpId from leader: term: 29 index: 20. (term mismatch)
I20260709 12:25:20.897332 17727 leader_election.cc:304] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:20.903653 18374 pending_rounds.cc:85] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da: Aborting all ops after (but not including) 19
I20260709 12:25:20.904120 18374 pending_rounds.cc:107] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da: Aborting uncommitted NO_OP operation due to leader change: 28.20
I20260709 12:25:20.904430 18374 raft_consensus.cc:2889] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:20.904667 17727 leader_election.cc:304] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:25:20.865011 28159 raft_consensus.cc:697] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 30 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:25:20.864841 28054 raft_consensus.cc:2749] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:25:20.867020 28115 raft_consensus.cc:493] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:25:20.898891 18366 raft_consensus.cc:2393] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 29: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:25:20.897935 17831 raft_consensus.cc:2468] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 22.
I20260709 12:25:20.870982 28036 raft_consensus.cc:2749] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:25:20.876399 28057 raft_consensus.cc:3060] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:20.870918 27431 raft_consensus.cc:2749] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Leader election lost for term 29. Reason: Vote denied by peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071) with higher term. Message: Invalid argument: T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 29. Current term is 30.
W20260709 12:25:20.908103 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 124046us
I20260709 12:25:20.922473 17654 leader_election.cc:304] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:20.894671 28189 raft_consensus.cc:2749] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:25:20.912694 18365 raft_consensus.cc:1240] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 33 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 32. Current term is 33. Ops: []
I20260709 12:25:20.912644 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 28 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:20.865209 27927 raft_consensus.cc:515] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:20.904366 28033 consensus_meta.cc:287] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.608s	user 0.005s	sys 0.007s
I20260709 12:25:20.925289 28033 raft_consensus.cc:515] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 33 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:20.869180 28157 raft_consensus.cc:493] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:20.927496 28157 raft_consensus.cc:3060] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:25:20.927706 28180 log.cc:927] Time spent T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.059s	user 0.000s	sys 0.000s
I20260709 12:25:20.895314 28175 raft_consensus.cc:697] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 30 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:25:20.933043 28035 leader_election.cc:290] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:20.932729 28175 consensus_queue.cc:237] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 28.17, Last appended by leader: 17, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:20.912003 28188 raft_consensus.cc:3060] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:20.933837 28126 consensus_queue.cc:237] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 28.21, Last appended by leader: 21, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:20.907964 28095 raft_consensus.cc:2804] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:25:20.936169 28095 raft_consensus.cc:493] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:25:20.936389 18367 raft_consensus.cc:3055] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:25:20.936625 28095 raft_consensus.cc:3060] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:20.936825 18367 raft_consensus.cc:740] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:25:20.937458 27989 leader_election.cc:290] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 33 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:20.911760 28087 raft_consensus.cc:2749] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:25:20.937664 18367 consensus_queue.cc:260] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 26.17, Last appended by leader: 17, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:20.932837 27881 consensus_queue.cc:260] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 27.21, Last appended by leader: 21, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:20.938787 18367 raft_consensus.cc:3060] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:20.933706 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:20.939359 17831 raft_consensus.cc:2468] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 27.
I20260709 12:25:20.915329 28010 raft_consensus.cc:2804] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:25:20.955070 28149 leader_election.cc:290] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:25:20.955418 28040 leader_election.cc:290] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 31 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:25:20.919966 27753 raft_consensus.cc:493] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:25:20.924015 28080 raft_consensus.cc:493] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:25:20.994155 17835 raft_consensus.cc:1240] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 31 LEADER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 30. Current term is 31. Ops: []
I20260709 12:25:20.944819 28177 raft_consensus.cc:3055] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:25:20.940469 28043 raft_consensus.cc:493] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
W20260709 12:25:20.998358 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 75978.3us
I20260709 12:25:20.991109 28054 raft_consensus.cc:493] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:25:20.999326 28112 leader_election.cc:290] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:25:21.016587 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 31 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:21.017274 17758 raft_consensus.cc:3060] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:21.015202 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 34 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:20.922534 28115 raft_consensus.cc:515] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:20.945945 28010 raft_consensus.cc:493] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 32 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:25:20.966080 27881 raft_consensus.cc:3060] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:21.015515 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 27 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:21.022200 17764 raft_consensus.cc:2468] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 32.
I20260709 12:25:21.037608 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 33 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:21.038326 17760 raft_consensus.cc:2468] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 32.
I20260709 12:25:21.014832 28188 raft_consensus.cc:2749] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:25:21.000169 28177 raft_consensus.cc:740] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 29 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:25:21.074256 28177 consensus_queue.cc:260] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 29.17, Last appended by leader: 17, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:21.075462 28177 raft_consensus.cc:3060] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:21.078311 17762 raft_consensus.cc:1275] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 21 index: 15. Preceding OpId from leader: term: 22 index: 16. (index mismatch)
I20260709 12:25:21.079931 17830 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35040 (request call id 3812) took 1118 ms. Trace:
I20260709 12:25:21.018436 28159 consensus_queue.cc:237] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 28.18, Last appended by leader: 18, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:21.080971 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:21.081712 17763 raft_consensus.cc:2393] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 27: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:25:21.080498 17830 rpcz_store.cc:276] 0709 12:25:19.961813 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:19.962026 (+   213us) service_pool.cc:224] Handling call
0709 12:25:19.962401 (+   375us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:20.787751 (+825350us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:21.079685 (+291934us) spinlock_profiling.cc:243] Waited 784 ms on lock 0x7b54003340d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:25:21.079850 (+   165us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1803760384}
I20260709 12:25:21.082993 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 29 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:21.003790 28080 raft_consensus.cc:515] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:21.085024 28080 leader_election.cc:290] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:20.996656 27753 raft_consensus.cc:515] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:21.086264 28201 log.cc:927] Time spent T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.066s	user 0.003s	sys 0.000s
I20260709 12:25:21.086992 27753 leader_election.cc:290] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:21.003440 17833 raft_consensus.cc:2468] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 27.
I20260709 12:25:21.088913 18333 leader_election.cc:304] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:25:21.089526 28115 leader_election.cc:290] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:21.008169 28194 raft_consensus.cc:2749] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
I20260709 12:25:21.089763 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:21.089886 18333 leader_election.cc:304] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:21.090500 17833 raft_consensus.cc:2393] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 31: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:25:21.090792 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:21.090972 18333 leader_election.cc:304] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:21.091509 17757 raft_consensus.cc:2393] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 32: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:25:21.091713 28112 raft_consensus.cc:493] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:25:21.092267 28112 raft_consensus.cc:515] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:21.094280 28112 leader_election.cc:290] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:21.094767 28112 raft_consensus.cc:2804] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:25:21.095108 28112 raft_consensus.cc:493] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:21.095438 28112 raft_consensus.cc:3060] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:21.103046 17686 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37402 (request call id 4048) took 1255 ms. Trace:
I20260709 12:25:21.103704 17686 rpcz_store.cc:276] 0709 12:25:19.847914 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:19.978914 (+131000us) service_pool.cc:224] Handling call
0709 12:25:19.990302 (+ 11388us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:20.052171 (+ 61869us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:20.052216 (+    45us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:20.052274 (+    58us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:25:20.052313 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:21.064285 (+1011972us) spinlock_profiling.cc:243] Waited 61.6 ms on lock 0x7b5400426dd4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:25:21.102805 (+ 38520us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:21.102968 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":141703424}
W20260709 12:25:21.105391 28151 log.cc:927] Time spent T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.756s	user 0.001s	sys 0.002s
I20260709 12:25:21.085733 17729 leader_election.cc:304] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:21.108795 27671 raft_consensus.cc:2804] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:25:21.110090 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:21.110898 18364 raft_consensus.cc:2393] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 27: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:25:21.111488 27671 raft_consensus.cc:697] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 29 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:25:21.112402 27766 leader_election.cc:290] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 29 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:21.112517 27671 consensus_queue.cc:237] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 27.18, Last appended by leader: 18, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:21.113894 17760 raft_consensus.cc:1240] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 28. Current term is 29. Ops: []
I20260709 12:25:20.966392 28189 consensus_queue.cc:1048] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 15, Time since last communication: 0.064s
W20260709 12:25:21.114888 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 107136us
I20260709 12:25:21.079993 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 35 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:25:21.051047 28140 log.cc:927] Time spent T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.835s	user 0.002s	sys 0.000s
I20260709 12:25:21.056624 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 30 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:21.012863 28054 raft_consensus.cc:515] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:21.011853 28017 leader_election.cc:290] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 29 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:25:21.139915 28111 consensus_queue.cc:1059] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 30.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.001s
I20260709 12:25:21.140868 28054 leader_election.cc:290] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:21.008225 28043 raft_consensus.cc:515] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:21.141752 28111 consensus_queue.cc:1243] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer e063541542a94969898868bf6eb5ce1b log is divergent from this leader: its last log entry 31.21 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 20
I20260709 12:25:21.141507 28157 raft_consensus.cc:515] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:21.142517 28127 raft_consensus.cc:3055] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:25:21.142964 28127 raft_consensus.cc:740] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 30 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:25:21.143327 28043 leader_election.cc:290] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:25:21.035372 28010 raft_consensus.cc:3060] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:25:21.144564 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:21.131559 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:21.145856 17835 raft_consensus.cc:3060] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:25:21.146204 28204 log.cc:927] Time spent T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.138s	user 0.000s	sys 0.001s
I20260709 12:25:21.073307 28171 leader_election.cc:290] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:25:21.071864 28192 log.cc:927] Time spent T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.056s	user 0.001s	sys 0.000s
I20260709 12:25:21.150439 28105 raft_consensus.cc:2804] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:25:21.150925 28105 raft_consensus.cc:493] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:25:21.151340 28105 raft_consensus.cc:3060] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:21.153720 18333 leader_election.cc:304] T a5732e71b9d54f8786b8c511d69943bf P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:21.154098 17652 leader_election.cc:304] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:21.155098 28171 consensus_queue.cc:1048] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20260709 12:25:21.155915 28104 raft_consensus.cc:2764] T a5732e71b9d54f8786b8c511d69943bf P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Leader pre-election decision vote started in defunct term 28: won
I20260709 12:25:21.156644 28171 raft_consensus.cc:2804] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:25:21.157094 28171 raft_consensus.cc:493] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:25:21.156960 28104 consensus_queue.cc:1048] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:25:21.157732 28171 raft_consensus.cc:3060] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:21.157413 28095 raft_consensus.cc:515] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:21.135900 27767 consensus_queue.cc:1048] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:25:21.135391 17834 raft_consensus.cc:2468] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 29.
I20260709 12:25:21.138450 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 28 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:21.137557 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 30 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:21.161592 27978 leader_election.cc:290] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:21.083650 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 35 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:21.162309 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:21.161844 17837 raft_consensus.cc:2393] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 30: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:25:21.144061 28127 consensus_queue.cc:260] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 30.20, Last appended by leader: 20, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:21.163929 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 26 candidate_status { last_received { term: 23 index: 21 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:21.164073 28127 raft_consensus.cc:3060] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:21.010052 28184 consensus_queue.cc:1048] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.062s
I20260709 12:25:21.165176 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 27 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:21.162628 17831 raft_consensus.cc:2410] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for term 35 because replica has last-logged OpId of term: 32 index: 22, which is greater than that of the candidate, which has last-logged OpId of term: 31 index: 21.
I20260709 12:25:21.091306 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 35 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:21.167191 17727 leader_election.cc:304] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:25:21.155285 18333 leader_election.cc:304] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:25:21.131592 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:21.174319 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 35 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:21.177583 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:25:21.149953 27753 raft_consensus.cc:2804] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:25:21.176283 17727 leader_election.cc:304] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:21.180174 17836 raft_consensus.cc:2468] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 29.
I20260709 12:25:21.173074 17758 raft_consensus.cc:2468] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 31.
I20260709 12:25:21.185042 27577 raft_consensus.cc:2804] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:25:21.185745 27577 raft_consensus.cc:697] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 29 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:25:21.185894 17686 raft_consensus.cc:1240] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 30. Current term is 31. Ops: []
I20260709 12:25:21.189546 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:21.190228 17831 raft_consensus.cc:2468] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 25.
I20260709 12:25:21.192162 17727 leader_election.cc:304] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:21.196950 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:21.197764 17836 raft_consensus.cc:2468] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 22.
I20260709 12:25:21.187000 27577 consensus_queue.cc:237] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 27.16, Last appended by leader: 16, Current term: 29, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:21.204133 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 28 candidate_status { last_received { term: 27 index: 21 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:21.208058 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 29 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:21.207316 17836 raft_consensus.cc:1275] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 25 index: 18. Preceding OpId from leader: term: 27 index: 19. (index mismatch)
I20260709 12:25:21.209964 17837 raft_consensus.cc:2468] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 26.
I20260709 12:25:21.212849 18363 raft_consensus.cc:1240] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 28. Current term is 29. Ops: []
I20260709 12:25:21.214403 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 35 candidate_status { last_received { term: 33 index: 25 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
W20260709 12:25:21.215030 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:21.215000 18361 raft_consensus.cc:2376] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 35 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 35. Re-sending same reply.
I20260709 12:25:21.215623 17834 raft_consensus.cc:2468] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 30.
I20260709 12:25:21.216039 17685 raft_consensus.cc:1275] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 25 index: 18. Preceding OpId from leader: term: 29 index: 20. (index mismatch)
I20260709 12:25:21.216573 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 33 candidate_status { last_received { term: 32 index: 23 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:21.220021 28100 consensus_queue.cc:1059] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 32.23, Next index: 24, Last known committed idx: 22, Time since last communication: 0.000s
I20260709 12:25:21.223063 18333 leader_election.cc:304] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:25:21.159787 28095 leader_election.cc:290] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 29 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:25:21.224149 18333 leader_election.cc:304] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:25:21.177940 28157 leader_election.cc:290] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 27 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:25:21.224001 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 28 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:21.179375 28054 raft_consensus.cc:493] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:25:21.217505 17729 leader_election.cc:304] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:21.208263 17831 raft_consensus.cc:2468] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 27.
I20260709 12:25:21.201149 28148 raft_consensus.cc:2804] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:25:21.252725 17834 raft_consensus.cc:1275] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 24 index: 16. Preceding OpId from leader: term: 27 index: 18. (index mismatch)
I20260709 12:25:21.338708 17686 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce"
dest_uuid: "0da169ff82674dc1947055ce43de68a3"
mode: GRACEFUL
new_leader_uuid: "334a4e1f59944544b3b3ff36616b57da"
 from {username='slave'} at 127.0.0.1:54540
I20260709 12:25:21.339345 17686 raft_consensus.cc:606] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Received request to transfer leadership to TS 334a4e1f59944544b3b3ff36616b57da
I20260709 12:25:21.339709 17686 raft_consensus.cc:614] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:25:21.339608 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:21.340380 17834 raft_consensus.cc:2393] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 29: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:25:21.202487 17758 raft_consensus.cc:1240] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 30. Current term is 31. Ops: [28.18-30.19]
I20260709 12:25:21.341593 17836 raft_consensus.cc:1240] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 26. Current term is 27. Ops: []
I20260709 12:25:21.188352 27753 raft_consensus.cc:493] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
W20260709 12:25:21.275252 18333 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:21.346221 18366 raft_consensus.cc:1240] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 26. Current term is 27. Ops: []
I20260709 12:25:21.205071 28045 raft_consensus.cc:2749] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
I20260709 12:25:21.354583 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 31 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:25:21.366555 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 30 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:21.226343 28104 raft_consensus.cc:493] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:21.370323 17680 raft_consensus.cc:1275] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 26 index: 20. Preceding OpId from leader: term: 28 index: 21. (index mismatch)
I20260709 12:25:21.385179 18363 raft_consensus.cc:2393] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 30: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:25:21.365392 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 34 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:21.386905 17682 raft_consensus.cc:2410] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 34 because replica has last-logged OpId of term: 32 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 31 index: 18.
I20260709 12:25:21.254123 28080 raft_consensus.cc:2804] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:25:21.388782 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:21.338454 27800 raft_consensus.cc:493] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:25:21.390909 27800 raft_consensus.cc:515] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:21.392338 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:21.392994 17758 raft_consensus.cc:2468] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 20.
I20260709 12:25:21.348564 28148 raft_consensus.cc:493] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:21.393832 28148 raft_consensus.cc:3060] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:21.400653 17835 raft_consensus.cc:2468] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 27.
I20260709 12:25:21.402932 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 29 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:21.406602 28190 raft_consensus.cc:2749] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:25:21.406567 17835 raft_consensus.cc:1275] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 26 index: 16. Preceding OpId from leader: term: 28 index: 17. (index mismatch)
I20260709 12:25:21.348227 28017 raft_consensus.cc:2804] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:25:21.407723 28017 raft_consensus.cc:493] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:25:21.408259 28017 raft_consensus.cc:3060] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:21.349439 27753 raft_consensus.cc:3060] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:21.349597 28054 raft_consensus.cc:515] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:21.413122 28054 leader_election.cc:290] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:25:21.415390 28202 log.cc:927] Time spent T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.067s	user 0.001s	sys 0.000s
I20260709 12:25:21.416195 28104 raft_consensus.cc:515] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:21.418450 28104 leader_election.cc:290] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:25:21.418896 28133 consensus_meta.cc:287] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.633s	user 0.007s	sys 0.002s
I20260709 12:25:21.420506 28112 raft_consensus.cc:515] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:21.422660 28112 leader_election.cc:290] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 28 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:21.422943 28010 raft_consensus.cc:515] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:21.425025 28010 leader_election.cc:290] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 33 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:21.427168 28057 raft_consensus.cc:515] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:21.430217 28057 leader_election.cc:290] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 23 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:21.431238 17838 raft_consensus.cc:2393] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 23: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:25:21.435520 27800 leader_election.cc:290] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:21.436962 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 29 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:21.427107 28017 raft_consensus.cc:515] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:21.442735 17756 raft_consensus.cc:1240] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 33 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 32. Current term is 33. Ops: []
I20260709 12:25:21.433544 17652 leader_election.cc:304] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:21.389972 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:21.456537 17689 raft_consensus.cc:2468] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 32.
I20260709 12:25:21.461928 17803 leader_election.cc:304] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:21.463097 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:21.464665 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:21.419318 28133 raft_consensus.cc:515] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 32 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:21.466536 18367 consensus_meta.cc:287] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.520s	user 0.009s	sys 0.000s
I20260709 12:25:21.473030 17652 leader_election.cc:304] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:21.368064 27927 leader_election.cc:290] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:21.507287 18367 raft_consensus.cc:1275] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 26 index: 17. Preceding OpId from leader: term: 27 index: 18. (index mismatch)
I20260709 12:25:21.524263 18367 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59886 (request call id 2771) took 1225 ms. Trace:
I20260709 12:25:21.524768 18367 rpcz_store.cc:276] 0709 12:25:20.298567 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:20.738364 (+439797us) service_pool.cc:224] Handling call
0709 12:25:20.738711 (+   347us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:20.936298 (+197587us) spinlock_profiling.cc:243] Waited 48.5 ms on lock 0x7b54006c64a0. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a597dc0 00007f7b4a6221c0 00007f7b4a623231 00007f7b4a61fd8f 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c
0709 12:25:21.523732 (+587434us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:21.524190 (+   458us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":111448960}
I20260709 12:25:21.526466 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 33 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:21.527283 18367 raft_consensus.cc:2468] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 32.
I20260709 12:25:21.518913 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:21.519327 17835 raft_consensus.cc:1240] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 28. Current term is 29. Ops: []
W20260709 12:25:21.477438 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 260035us
I20260709 12:25:21.529215 17729 leader_election.cc:304] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:25:21.529703 18364 raft_consensus.cc:2410] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for term 28 because replica has last-logged OpId of term: 27 index: 21, which is greater than that of the candidate, which has last-logged OpId of term: 26 index: 20.
I20260709 12:25:21.548931 17803 leader_election.cc:304] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:21.554543 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 28 candidate_status { last_received { term: 27 index: 21 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:25:21.555271 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 63053.2us
I20260709 12:25:21.442198 28080 raft_consensus.cc:493] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:25:21.568975 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:21.570035 18363 raft_consensus.cc:2393] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 28: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:25:21.568903 17689 raft_consensus.cc:1240] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 32 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 31. Current term is 32. Ops: []
I20260709 12:25:21.571857 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:21.586162 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 33 candidate_status { last_received { term: 32 index: 23 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:25:21.586252 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 57133.2us
I20260709 12:25:21.587067 18366 raft_consensus.cc:2393] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 33: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:25:21.589737 18333 leader_election.cc:304] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:21.479468 28057 raft_consensus.cc:2749] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
I20260709 12:25:21.521126 28184 raft_consensus.cc:2804] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:25:21.591493 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 28 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:21.648579 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:21.648883 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 33 candidate_status { last_received { term: 32 index: 23 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:21.519747 27767 raft_consensus.cc:493] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:21.652987 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:21.656252 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 29 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
W20260709 12:25:21.656728 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 67084.4us
I20260709 12:25:21.472694 27431 consensus_queue.cc:1048] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.130s
I20260709 12:25:21.664222 17803 leader_election.cc:304] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:25:21.668454 17829 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35042 (request call id 3626) took 1091 ms. Trace:
I20260709 12:25:21.671793 17829 rpcz_store.cc:276] 0709 12:25:20.576723 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:20.585283 (+  8560us) service_pool.cc:224] Handling call
0709 12:25:21.668283 (+1083000us) spinlock_profiling.cc:243] Waited 210 ms on lock 0x7b54001581d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:25:21.668394 (+   111us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":482472832}
I20260709 12:25:21.674470 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:21.675427 17761 raft_consensus.cc:2368] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 32. Current term is 33.
I20260709 12:25:21.572026 28010 raft_consensus.cc:2749] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 12:25:21.570982 28080 raft_consensus.cc:3060] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Advancing to term 26
W20260709 12:25:21.721542 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 57422.7us
I20260709 12:25:21.715224 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 29 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:21.716251 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 34 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:21.723414 17835 raft_consensus.cc:2468] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 33.
I20260709 12:25:21.585487 28040 raft_consensus.cc:2749] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:25:21.726864 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 36 candidate_status { last_received { term: 33 index: 25 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:21.728410 18333 leader_election.cc:400] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 32 pre-election: Vote denied by peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651) with higher term. Message: Invalid argument: T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 32. Current term is 33.
I20260709 12:25:21.729576 18333 leader_election.cc:403] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 32 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:25:21.519702 28105 raft_consensus.cc:515] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:21.736166 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 29 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:21.736873 18363 raft_consensus.cc:2468] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 28.
W20260709 12:25:21.739838 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 49801.6us
I20260709 12:25:21.659125 28184 raft_consensus.cc:493] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 33 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:21.669355 28104 raft_consensus.cc:2749] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:25:21.671190 27744 raft_consensus.cc:3060] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:21.572554 28111 consensus_queue.cc:1048] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 18, Time since last communication: 0.147s
I20260709 12:25:21.778177 17683 raft_consensus.cc:1240] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 25. Current term is 26. Ops: []
I20260709 12:25:21.646817 27881 consensus_queue.cc:1048] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.103s
I20260709 12:25:21.763801 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 28 candidate_status { last_received { term: 27 index: 21 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:25:21.759852 18334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:21.747160 28016 raft_consensus.cc:3060] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Advancing to term 33
I20260709 12:25:21.756842 28057 raft_consensus.cc:3060] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:21.754752 28184 raft_consensus.cc:3060] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:25:21.813774 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 29 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:21.670645 27767 raft_consensus.cc:515] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:21.818893 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 29 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:21.792382 28157 raft_consensus.cc:493] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:21.842506 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:21.847072 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:21.849293 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 29 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:25:21.861030 17831 raft_consensus.cc:1275] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 23 index: 16. Preceding OpId from leader: term: 26 index: 18. (index mismatch)
I20260709 12:25:21.871470 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:25:21.872861 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 29 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:25:21.873883 28183 raft_consensus.cc:493] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:21.874598 28183 raft_consensus.cc:515] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:21.876739 28183 leader_election.cc:290] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:21.876886 27767 leader_election.cc:290] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:25:21.878046 28209 log.cc:927] Time spent T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.098s	user 0.000s	sys 0.000s
W20260709 12:25:21.880089 28154 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.621s	user 0.014s	sys 0.046s
W20260709 12:25:21.880527 28154 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.714s	user 0.014s	sys 0.046s
I20260709 12:25:21.864856 28207 raft_consensus.cc:2804] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:25:21.884375 28207 raft_consensus.cc:493] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:21.884835 28207 raft_consensus.cc:3060] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 32 FOLLOWER]: Advancing to term 33
W20260709 12:25:21.884863 18371 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59918 (request call id 2960) took 1692 ms (client timeout 1852 ms). Trace:
W20260709 12:25:21.885546 18371 rpcz_store.cc:269] 0709 12:25:20.192435 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:20.192558 (+   123us) service_pool.cc:224] Handling call
0709 12:25:21.884596 (+1692038us) spinlock_profiling.cc:243] Waited 598 ms on lock 0x7b5400727254. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:25:21.884776 (+   180us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1376102784}
I20260709 12:25:21.873947 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:21.890156 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:25:21.895751 18333 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:21.770747 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 567466us
W20260709 12:25:21.862166 28127 consensus_meta.cc:287] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.697s	user 0.010s	sys 0.002s
I20260709 12:25:21.854828 28017 leader_election.cc:290] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 23 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:21.848249 28104 raft_consensus.cc:493] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:25:21.913196 17682 raft_consensus.cc:2393] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 32: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:25:21.923169 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 36 candidate_status { last_received { term: 33 index: 25 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:25:21.745707 27525 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.634s	user 0.012s	sys 0.016s
I20260709 12:25:21.928059 17803 leader_election.cc:304] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:21.933305 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:21.934116 18371 raft_consensus.cc:3060] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:21.937433 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:21.938951 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
W20260709 12:25:21.944725 28205 log.cc:927] Time spent T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.284s	user 0.001s	sys 0.000s
I20260709 12:25:21.854233 28157 raft_consensus.cc:515] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:21.945680 27892 raft_consensus.cc:3060] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:25:21.944504 27671 raft_consensus.cc:493] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:21.995995 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:21.998414 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:25:21.978581 28011 raft_consensus.cc:493] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 34 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:25:21.826220 28148 raft_consensus.cc:515] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:22.028726 18371 raft_consensus.cc:2468] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 31.
I20260709 12:25:21.972074 28104 raft_consensus.cc:515] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:22.056483 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 27 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:22.059456 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:22.063990 18365 raft_consensus.cc:2468] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 26.
W20260709 12:25:22.065137 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 498116us
W20260709 12:25:22.070051 17681 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.628s	user 0.007s	sys 0.006s
W20260709 12:25:22.070685 17681 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.682s	user 0.007s	sys 0.006s
I20260709 12:25:21.963246 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 27 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:22.073227 18363 raft_consensus.cc:2393] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 27: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:25:22.068712 17652 leader_election.cc:304] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:22.016731 28133 leader_election.cc:290] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 32 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:22.106948 18366 raft_consensus.cc:3060] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Advancing to term 29
W20260709 12:25:21.938527 27525 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.827s	user 0.013s	sys 0.016s
W20260709 12:25:22.114418 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 45190.3us
I20260709 12:25:22.116670 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:22.119303 18365 raft_consensus.cc:2368] T a5732e71b9d54f8786b8c511d69943bf P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 28. Current term is 29.
I20260709 12:25:22.137403 17683 raft_consensus.cc:1240] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 27. Current term is 28. Ops: [27.21-27.21]
W20260709 12:25:22.096109 28099 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.655s	user 0.000s	sys 0.015s
I20260709 12:25:22.021997 28011 raft_consensus.cc:515] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:22.002547 27671 raft_consensus.cc:515] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:22.125697 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:22.089905 28017 raft_consensus.cc:493] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
W20260709 12:25:22.145541 28099 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.729s	user 0.000s	sys 0.016s
I20260709 12:25:22.122676 27767 raft_consensus.cc:2804] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:25:22.155216 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 31 candidate_status { last_received { term: 30 index: 21 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:22.136034 18334 leader_election.cc:304] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:22.166566 18367 raft_consensus.cc:2393] T a5732e71b9d54f8786b8c511d69943bf P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 29: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:25:22.172941 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 30 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:22.098495 28153 consensus_queue.cc:1059] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 27.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.053s
I20260709 12:25:22.175853 18334 leader_election.cc:304] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:22.178273 17683 raft_consensus.cc:1275] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 26 index: 16. Preceding OpId from leader: term: 28 index: 17. (index mismatch)
I20260709 12:25:22.209415 18367 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59918 (request call id 2967) took 1016 ms. Trace:
I20260709 12:25:22.211097 18367 rpcz_store.cc:276] 0709 12:25:21.192721 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:22.070080 (+877359us) service_pool.cc:224] Handling call
0709 12:25:22.209141 (+139061us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:25:22.214347 17759 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.558s	user 0.010s	sys 0.032s
W20260709 12:25:22.214725 17759 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.559s	user 0.010s	sys 0.032s
I20260709 12:25:22.175837 27881 raft_consensus.cc:2804] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 32 FOLLOWER]: Leader election won for term 32
I20260709 12:25:22.171900 27767 raft_consensus.cc:493] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:25:22.225555 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:22.230121 18367 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 27.
I20260709 12:25:22.230685 18367 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59902 (request call id 2400) took 1011 ms. Trace:
I20260709 12:25:22.230875 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 28 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:22.231088 18367 rpcz_store.cc:276] 0709 12:25:21.219176 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:22.222373 (+1003197us) service_pool.cc:224] Handling call
0709 12:25:22.230624 (+  8251us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:22.242381 18363 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59902 (request call id 2398) took 1023 ms. Trace:
I20260709 12:25:22.168064 28017 raft_consensus.cc:515] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:22.232398 28210 log.cc:927] Time spent T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.296s	user 0.000s	sys 0.001s
I20260709 12:25:22.243083 18363 rpcz_store.cc:276] 0709 12:25:21.218669 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:22.149435 (+930766us) service_pool.cc:224] Handling call
0709 12:25:22.242213 (+ 92778us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:25:22.236886 28177 consensus_meta.cc:287] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 1.159s	user 0.007s	sys 0.006s
I20260709 12:25:22.228905 27767 raft_consensus.cc:3060] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:22.266989 17680 raft_consensus.cc:1275] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 29 index: 17. Preceding OpId from leader: term: 30 index: 17. (term mismatch)
I20260709 12:25:22.267514 17680 pending_rounds.cc:85] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3: Aborting all ops after (but not including) 16
I20260709 12:25:22.267795 17680 pending_rounds.cc:107] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3: Aborting uncommitted NO_OP operation due to leader change: 29.17
I20260709 12:25:22.268105 17680 raft_consensus.cc:2889] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:25:22.247335 28215 log.cc:927] Time spent T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.246s	user 0.001s	sys 0.000s
I20260709 12:25:22.235775 27881 raft_consensus.cc:697] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 32 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:25:22.234494 28159 raft_consensus.cc:493] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:25:22.243563 28133 raft_consensus.cc:3055] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:25:22.284186 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 26 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:22.290935 18367 raft_consensus.cc:2393] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 26: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:25:22.291471 18367 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59902 (request call id 2403) took 1071 ms. Trace:
I20260709 12:25:22.291852 18367 rpcz_store.cc:276] 0709 12:25:21.219950 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:22.252066 (+1032116us) service_pool.cc:224] Handling call
0709 12:25:22.291397 (+ 39331us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:22.285439 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 30 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:22.293422 18363 raft_consensus.cc:2410] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 30 because replica has last-logged OpId of term: 29 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 15.
I20260709 12:25:22.223950 28183 consensus_queue.cc:1059] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 25.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.026s
I20260709 12:25:22.250272 28154 raft_consensus.cc:493] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:22.297106 18363 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59902 (request call id 2402) took 1077 ms. Trace:
I20260709 12:25:22.297904 18363 rpcz_store.cc:276] 0709 12:25:21.219685 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:22.250979 (+1031294us) service_pool.cc:224] Handling call
0709 12:25:22.297024 (+ 46045us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:25:22.303033 17728 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.843s	user 0.013s	sys 0.062s
W20260709 12:25:22.309732 17728 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.850s	user 0.013s	sys 0.062s
W20260709 12:25:22.310070 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 962616us
I20260709 12:25:22.271723 28157 leader_election.cc:290] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:25:22.322630 17759 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639"
dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
mode: GRACEFUL
new_leader_uuid: "334a4e1f59944544b3b3ff36616b57da"
 from {username='slave'} at 127.0.0.1:49786
I20260709 12:25:22.323360 17759 raft_consensus.cc:606] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Received request to transfer leadership to TS 334a4e1f59944544b3b3ff36616b57da
I20260709 12:25:22.323930 17759 raft_consensus.cc:614] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:25:22.327520 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 29 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:22.332746 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 28 candidate_status { last_received { term: 27 index: 21 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:22.333527 18363 raft_consensus.cc:2468] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 27.
I20260709 12:25:22.296729 28133 raft_consensus.cc:740] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 30 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:25:22.322134 28104 leader_election.cc:290] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:22.298180 28232 raft_consensus.cc:2749] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:25:22.348345 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 30 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:22.324636 28177 raft_consensus.cc:493] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:22.352442 17833 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35042 (request call id 3616) took 2069 ms. Trace:
W20260709 12:25:22.330243 28211 log.cc:927] Time spent T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.277s	user 0.000s	sys 0.001s
I20260709 12:25:22.356189 17831 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35040 (request call id 3821) took 1167 ms. Trace:
W20260709 12:25:22.356622 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 180909us
I20260709 12:25:22.353530 18363 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59918 (request call id 2972) took 1159 ms. Trace:
I20260709 12:25:22.359025 18363 rpcz_store.cc:276] 0709 12:25:21.193881 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:22.336657 (+1142776us) service_pool.cc:224] Handling call
0709 12:25:22.353458 (+ 16801us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:22.359107 18334 leader_election.cc:304] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:22.362068 18334 leader_election.cc:304] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:22.307897 28159 raft_consensus.cc:515] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:22.358506 17831 rpcz_store.cc:276] 0709 12:25:21.189154 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:21.867578 (+678424us) service_pool.cc:224] Handling call
0709 12:25:21.867967 (+   389us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:21.868245 (+   278us) raft_consensus.cc:1534] Early marking committed up to index 16
0709 12:25:21.924129 (+ 55884us) log.cc:844] Serialized 14 byte log entry
0709 12:25:22.355801 (+431672us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:22.355854 (+    53us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:25:22.355913 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:22.355974 (+    61us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:22.356121 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":6095,"prepare.queue_time_us":135305,"prepare.run_cpu_time_us":58,"prepare.run_wall_time_us":60,"thread_start_us":430803,"threads_started":2,"wal-append.queue_time_us":397542}
I20260709 12:25:22.310878 28154 raft_consensus.cc:515] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:22.352872 17833 rpcz_store.cc:276] 0709 12:25:20.285312 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:21.179536 (+894224us) service_pool.cc:224] Handling call
0709 12:25:21.179872 (+   336us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:21.180082 (+   210us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 12:25:21.566798 (+386716us) log.cc:844] Serialized 14 byte log entry
0709 12:25:22.352053 (+785255us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:22.352120 (+    67us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:25:22.352170 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:22.352243 (+    73us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:22.352379 (+   136us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":106961,"prepare.queue_time_us":676265,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":45,"thread_start_us":992198,"threads_started":3,"wal-append.queue_time_us":559117}
I20260709 12:25:22.370685 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 32 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:22.372179 18334 leader_election.cc:304] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:22.374740 17803 leader_election.cc:304] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:25:22.375124 17755 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38776 (request call id 3450) took 1350 ms. Trace:
I20260709 12:25:22.349838 28148 leader_election.cc:290] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 28 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:22.380755 18367 raft_consensus.cc:2410] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for term 29 because replica has last-logged OpId of term: 27 index: 20, which is greater than that of the candidate, which has last-logged OpId of term: 26 index: 19.
I20260709 12:25:22.376353 17652 leader_election.cc:304] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:22.383170 17833 raft_consensus.cc:1240] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 28. Current term is 29. Ops: []
I20260709 12:25:22.390070 17833 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35040 (request call id 3822) took 1183 ms. Trace:
I20260709 12:25:22.390663 17833 rpcz_store.cc:276] 0709 12:25:21.206757 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:22.381957 (+1175200us) service_pool.cc:224] Handling call
0709 12:25:22.382935 (+   978us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:22.389718 (+  6783us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:22.389995 (+   277us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:25:22.373426 28219 log.cc:927] Time spent T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.197s	user 0.000s	sys 0.001s
I20260709 12:25:22.375568 17755 rpcz_store.cc:276] 0709 12:25:21.025249 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:21.025449 (+   200us) service_pool.cc:224] Handling call
0709 12:25:21.031574 (+  6125us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:21.043544 (+ 11970us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:21.043578 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:21.087675 (+ 44097us) log.cc:844] Serialized 26 byte log entry
0709 12:25:21.470757 (+383082us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:25:21.528957 (+ 58200us) log.cc:844] Serialized 14 byte log entry
0709 12:25:21.573628 (+ 44671us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:21.573728 (+   100us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:22.374918 (+801190us) raft_consensus.cc:1685] finished
0709 12:25:22.374940 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:22.375065 (+   125us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":47828,"prepare.queue_time_us":85233,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":40,"raft.queue_time_us":129157,"raft.run_cpu_time_us":119,"raft.run_wall_time_us":13507,"spinlock_wait_cycles":9268480,"thread_start_us":382724,"threads_started":1,"wal-append.queue_time_us":748974}
I20260709 12:25:22.364938 28104 raft_consensus.cc:2749] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:25:22.369269 28011 leader_election.cc:290] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:22.399974 18374 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59886 (request call id 2767) took 2121 ms. Trace:
I20260709 12:25:22.358613 28177 raft_consensus.cc:515] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:22.408205 18374 rpcz_store.cc:276] 0709 12:25:20.278992 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:20.658740 (+379748us) service_pool.cc:224] Handling call
0709 12:25:20.659082 (+   342us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:20.905197 (+246115us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:22.399634 (+1494437us) spinlock_profiling.cc:243] Waited 219 ms on lock 0x7b54006946d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:25:22.399893 (+   259us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":503395072}
I20260709 12:25:22.402508 17829 raft_consensus.cc:1240] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 30. Current term is 31. Ops: []
I20260709 12:25:22.415086 18374 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 2968) took 1222 ms. Trace:
I20260709 12:25:22.415704 18374 rpcz_store.cc:276] 0709 12:25:21.192932 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:22.414056 (+1221124us) service_pool.cc:224] Handling call
0709 12:25:22.414450 (+   394us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:22.414666 (+   216us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:22.414712 (+    46us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:22.414766 (+    54us) raft_consensus.cc:1645] Marking committed up to 21
0709 12:25:22.414811 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:22.414880 (+    69us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:22.415015 (+   135us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:22.392977 18363 raft_consensus.cc:1275] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 35 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 33 index: 25. Preceding OpId from leader: term: 35 index: 26. (index mismatch)
W20260709 12:25:22.393599 28217 log.cc:927] Time spent T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.210s	user 0.001s	sys 0.000s
I20260709 12:25:22.387338 18366 raft_consensus.cc:2468] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 29.
I20260709 12:25:22.418246 18363 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 2966) took 1225 ms. Trace:
W20260709 12:25:22.402729 28231 log.cc:927] Time spent T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.065s	user 0.000s	sys 0.000s
I20260709 12:25:22.428635 18363 rpcz_store.cc:276] 0709 12:25:21.192430 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:22.360226 (+1167796us) service_pool.cc:224] Handling call
0709 12:25:22.380615 (+ 20389us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:22.417955 (+ 37340us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:22.418183 (+   228us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:25:22.413287 28227 log.cc:927] Time spent T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.193s	user 0.000s	sys 0.001s
I20260709 12:25:22.394388 28232 raft_consensus.cc:2749] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
W20260709 12:25:22.428972 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 54340.2us
I20260709 12:25:22.401420 28148 raft_consensus.cc:2749] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260709 12:25:22.402906 28017 leader_election.cc:290] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:22.484755 18374 raft_consensus.cc:1240] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 30. Current term is 31. Ops: []
W20260709 12:25:22.426440 28236 log.cc:927] Time spent T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.072s	user 0.000s	sys 0.001s
I20260709 12:25:22.479717 17803 leader_election.cc:304] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3
W20260709 12:25:22.426327 28016 consensus_meta.cc:287] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.609s	user 0.010s	sys 0.003s
W20260709 12:25:22.479166 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 102735us
I20260709 12:25:22.388399 27881 consensus_queue.cc:237] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 31.20, Last appended by leader: 20, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:22.427932 28183 raft_consensus.cc:493] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:22.492996 17652 leader_election.cc:304] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:25:22.484246 28230 log.cc:927] Time spent T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.068s	user 0.000s	sys 0.000s
I20260709 12:25:22.516721 18363 raft_consensus.cc:1240] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 22. Current term is 23. Ops: []
I20260709 12:25:22.519335 17681 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37402 (request call id 4053) took 1243 ms. Trace:
I20260709 12:25:22.393405 28133 consensus_queue.cc:260] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 30.19, Last appended by leader: 19, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:22.520608 18363 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 2973) took 1326 ms. Trace:
I20260709 12:25:22.487615 28036 raft_consensus.cc:493] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:25:22.522737 18363 rpcz_store.cc:276] 0709 12:25:21.194123 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:22.430506 (+1236383us) service_pool.cc:224] Handling call
0709 12:25:22.516428 (+ 85922us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:22.520280 (+  3852us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:22.520536 (+   256us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:22.483608 28228 raft_consensus.cc:2749] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:25:22.522212 28250 raft_consensus.cc:2804] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:25:22.524715 28251 raft_consensus.cc:2749] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
W20260709 12:25:22.525633 28238 log.cc:927] Time spent T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.096s	user 0.000s	sys 0.000s
W20260709 12:25:22.527251 28221 log.cc:927] Time spent T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.236s	user 0.000s	sys 0.000s
I20260709 12:25:22.521196 17681 rpcz_store.cc:276] 0709 12:25:21.282464 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:21.283578 (+  1114us) service_pool.cc:224] Handling call
0709 12:25:21.339717 (+ 56139us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:21.339970 (+   253us) raft_consensus.cc:1534] Early marking committed up to index 22
0709 12:25:22.071352 (+731382us) log.cc:844] Serialized 14 byte log entry
0709 12:25:22.517947 (+446595us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:22.518531 (+   584us) raft_consensus.cc:1645] Marking committed up to 22
0709 12:25:22.518673 (+   142us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:22.518865 (+   192us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:22.519161 (+   296us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":858,"prepare.queue_time_us":79403,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":39,"raft.queue_time_us":98481,"raft.run_cpu_time_us":108,"raft.run_wall_time_us":110,"thread_start_us":1127740,"threads_started":3,"wal-append.queue_time_us":440815}
I20260709 12:25:22.531337 27856 leader_election.cc:290] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:22.533208 28105 leader_election.cc:290] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 31 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:22.538070 17835 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35040 (request call id 3820) took 1362 ms. Trace:
I20260709 12:25:22.515295 28016 raft_consensus.cc:2749] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 33 FOLLOWER]: Leader pre-election lost for term 32. Reason: Vote denied by peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651) with higher term. Message: Invalid argument: T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 32. Current term is 33.
W20260709 12:25:22.624269 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 203559us
W20260709 12:25:22.630005 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 116226us
W20260709 12:25:22.624616 27744 consensus_meta.cc:287] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.816s	user 0.012s	sys 0.002s
I20260709 12:25:22.519007 28183 raft_consensus.cc:515] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:22.613960 28250 raft_consensus.cc:493] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:25:22.630704 28112 raft_consensus.cc:493] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
W20260709 12:25:22.644491 17832 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35042 (request call id 3612) took 2362 ms (client timeout 2117 ms). Trace:
W20260709 12:25:22.649842 17832 rpcz_store.cc:269] 0709 12:25:20.281894 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:20.282008 (+   114us) service_pool.cc:224] Handling call
0709 12:25:22.644273 (+2362265us) spinlock_profiling.cc:243] Waited 461 ms on lock 0x7b540022b854. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:25:22.644400 (+   127us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1061053312}
I20260709 12:25:22.645686 17837 raft_consensus.cc:1275] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 30 index: 18. Preceding OpId from leader: term: 31 index: 19. (index mismatch)
I20260709 12:25:22.624969 28011 raft_consensus.cc:493] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:22.644038 28233 log.cc:927] Time spent T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.230s	user 0.001s	sys 0.000s
I20260709 12:25:22.632679 28148 raft_consensus.cc:493] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:22.651618 17652 leader_election.cc:304] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:22.543840 28036 raft_consensus.cc:515] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:25:22.654788 28171 consensus_meta.cc:287] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 1.496s	user 0.013s	sys 0.003s
I20260709 12:25:22.635735 28247 raft_consensus.cc:2804] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:25:22.646498 27744 raft_consensus.cc:2749] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:25:22.649999 28250 raft_consensus.cc:3060] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:25:22.661108 28234 log.cc:927] Time spent T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.290s	user 0.001s	sys 0.000s
I20260709 12:25:22.657377 28183 leader_election.cc:290] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:25:22.661552 28159 leader_election.cc:290] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:22.676930 28154 leader_election.cc:290] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:22.653653 28112 raft_consensus.cc:515] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:22.723651 17762 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38776 (request call id 3462) took 1319 ms. Trace:
W20260709 12:25:22.656870 27753 consensus_meta.cc:287] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 1.248s	user 0.009s	sys 0.005s
I20260709 12:25:22.659067 28011 raft_consensus.cc:515] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:22.676879 28120 raft_consensus.cc:2749] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:25:22.653857 27976 consensus_queue.cc:1059] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 28.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.032s
I20260709 12:25:22.682968 28253 raft_consensus.cc:493] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:25:22.699800 28247 raft_consensus.cc:697] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 31 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:25:22.832391 17755 raft_consensus.cc:1275] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 27 index: 17. Preceding OpId from leader: term: 30 index: 19. (index mismatch)
I20260709 12:25:22.677022 28148 raft_consensus.cc:515] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:22.839910 17763 raft_consensus.cc:2468] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 29.
I20260709 12:25:22.730459 17762 rpcz_store.cc:276] 0709 12:25:21.404581 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:21.406739 (+  2158us) service_pool.cc:224] Handling call
0709 12:25:21.407066 (+   327us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 12:25:21.572868 (+165802us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:21.572931 (+    63us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 12:25:21.663149 (+ 90218us) log.cc:844] Serialized 78 byte log entry
0709 12:25:22.119666 (+456517us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:25:22.149511 (+ 29845us) log.cc:844] Serialized 14 byte log entry
0709 12:25:22.217695 (+ 68184us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:22.217780 (+    85us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:22.723296 (+505516us) raft_consensus.cc:1685] finished
0709 12:25:22.723328 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:22.723565 (+   237us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":181458,"prepare.queue_time_us":32172,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":36,"raft.queue_time_us":251911,"raft.run_cpu_time_us":136,"raft.run_wall_time_us":9284,"spinlock_wait_cycles":9481984,"thread_start_us":321103,"threads_started":1,"wal-append.queue_time_us":512446}
I20260709 12:25:22.679476 28171 raft_consensus.cc:515] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:22.817667 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:22.692584 27856 consensus_queue.cc:1048] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.133s
I20260709 12:25:22.851604 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 35 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:22.736027 28177 leader_election.cc:290] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:25:22.839247 28172 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 27 current term: 27 is higher.
I20260709 12:25:22.863111 17757 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38776 (request call id 3458) took 1650 ms. Trace:
I20260709 12:25:22.863564 18368 raft_consensus.cc:1240] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 27. Current term is 28. Ops: [27.19-27.19]
I20260709 12:25:22.864420 18368 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 2976) took 1653 ms. Trace:
I20260709 12:25:22.864845 18368 rpcz_store.cc:276] 0709 12:25:21.210880 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:22.862950 (+1652070us) service_pool.cc:224] Handling call
0709 12:25:22.863339 (+   389us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:22.864169 (+   830us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:22.864366 (+   197us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:22.824515 28159 consensus_queue.cc:1059] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 28.19, Next index: 20, Last known committed idx: 18, Time since last communication: 0.019s
I20260709 12:25:22.865988 28159 raft_consensus.cc:3055] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 28 LEADER]: Stepping down as leader of term 28
W20260709 12:25:22.623551 28229 consensus_queue.cc:1175] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "e063541542a94969898868bf6eb5ce1b" responder_term: 31 status { last_received { term: 28 index: 18 } last_committed_idx: 18 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 30. Current term is 31. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260709 12:25:22.866385 28159 raft_consensus.cc:740] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:25:22.867812 28159 consensus_queue.cc:260] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 28.19, Last appended by leader: 19, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:22.869216 28159 raft_consensus.cc:3060] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:22.863533 17757 rpcz_store.cc:276] 0709 12:25:21.212289 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:21.212535 (+   246us) service_pool.cc:224] Handling call
0709 12:25:21.338241 (+125706us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:21.771316 (+433075us) raft_consensus.cc:1534] Early marking committed up to index 25
0709 12:25:21.837007 (+ 65691us) log.cc:844] Serialized 14 byte log entry
0709 12:25:22.248342 (+411335us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:22.259721 (+ 11379us) log.cc:844] Serialized 26 byte log entry
0709 12:25:22.259922 (+   201us) raft_consensus.cc:1645] Marking committed up to 25
0709 12:25:22.259960 (+    38us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:22.260032 (+    72us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:22.862875 (+602843us) raft_consensus.cc:1685] finished
0709 12:25:22.862898 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:22.863051 (+   153us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":139026,"prepare.queue_time_us":63289,"prepare.run_cpu_time_us":64,"prepare.run_wall_time_us":5914,"raft.queue_time_us":183816,"raft.run_cpu_time_us":335,"raft.run_wall_time_us":55513,"thread_start_us":692638,"threads_started":2,"wal-append.queue_time_us":526868}
I20260709 12:25:22.839216 28254 consensus_queue.cc:1059] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 30.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.045s
I20260709 12:25:22.874002 28254 raft_consensus.cc:3055] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:25:22.874402 28254 raft_consensus.cc:740] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 30 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:25:22.876355 28256 raft_consensus.cc:493] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 34 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:25:22.876966 28256 raft_consensus.cc:515] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:22.879585 28256 leader_election.cc:290] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:22.868362 28171 leader_election.cc:290] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 30 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:22.826707 28154 raft_consensus.cc:493] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:25:22.887169 28011 leader_election.cc:290] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:25:22.871759 28184 consensus_meta.cc:287] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 1.035s	user 0.008s	sys 0.002s
I20260709 12:25:22.825541 27753 raft_consensus.cc:515] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:22.903141 27856 raft_consensus.cc:493] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:22.901347 28154 raft_consensus.cc:515] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:22.904560 28148 leader_election.cc:290] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:22.905405 18374 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 2970) took 1712 ms. Trace:
W20260709 12:25:22.891741 28207 consensus_meta.cc:287] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 1.006s	user 0.014s	sys 0.000s
I20260709 12:25:22.895704 28011 raft_consensus.cc:493] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:22.875358 28254 consensus_queue.cc:260] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 30.20, Last appended by leader: 20, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:22.858814 28253 raft_consensus.cc:515] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:22.910387 17760 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38776 (request call id 3453) took 1869 ms. Trace:
I20260709 12:25:22.913550 28157 raft_consensus.cc:3055] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:25:22.913633 28184 raft_consensus.cc:515] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 34 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:22.916746 28036 leader_election.cc:290] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:25:22.919958 18366 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59902 (request call id 2406) took 1699 ms (client timeout 1981 ms). Trace:
I20260709 12:25:22.920028 27744 raft_consensus.cc:493] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:22.920852 18363 raft_consensus.cc:1240] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 28. Current term is 29. Ops: []
I20260709 12:25:22.924835 28258 raft_consensus.cc:493] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:22.926864 17755 raft_consensus.cc:1275] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 28 index: 16. Preceding OpId from leader: term: 30 index: 17. (index mismatch)
I20260709 12:25:22.927145 17837 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35042 (request call id 3634) took 1796 ms. Trace:
I20260709 12:25:22.927563 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:22.928912 28183 raft_consensus.cc:493] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:22.931313 27431 raft_consensus.cc:493] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:25:22.931654 17801 leader_election.cc:304] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:25:22.932461 28016 raft_consensus.cc:493] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:22.936506 28237 log.cc:927] Time spent T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.588s	user 0.006s	sys 0.006s
I20260709 12:25:22.938232 17763 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36572 (request call id 1841) took 1290 ms. Trace:
I20260709 12:25:22.941075 17758 raft_consensus.cc:1275] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 30 index: 20. Preceding OpId from leader: term: 31 index: 21. (index mismatch)
I20260709 12:25:22.947670 17764 raft_consensus.cc:1240] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 28. Current term is 29. Ops: []
W20260709 12:25:22.948721 28156 log.cc:927] Time spent T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 2.399s	user 0.000s	sys 0.017s
I20260709 12:25:22.949703 17689 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37402 (request call id 4061) took 1458 ms. Trace:
I20260709 12:25:22.628273 17835 rpcz_store.cc:276] 0709 12:25:21.175310 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:21.861013 (+685703us) service_pool.cc:224] Handling call
0709 12:25:21.869806 (+  8793us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:21.870049 (+   243us) raft_consensus.cc:1534] Early marking committed up to index 16
0709 12:25:22.250639 (+380590us) log.cc:844] Serialized 14 byte log entry
0709 12:25:22.537691 (+287052us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:22.537742 (+    51us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:25:22.537789 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:22.537873 (+    84us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:22.538006 (+   133us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":58368,"prepare.queue_time_us":197561,"prepare.run_cpu_time_us":74,"prepare.run_wall_time_us":7403,"raft.queue_time_us":132292,"raft.run_cpu_time_us":153,"raft.run_wall_time_us":15368,"thread_start_us":546287,"threads_started":3}
I20260709 12:25:22.949914 28247 consensus_queue.cc:237] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 30.20, Last appended by leader: 20, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:22.992712 17760 rpcz_store.cc:276] 0709 12:25:21.041312 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:21.041589 (+   277us) service_pool.cc:224] Handling call
0709 12:25:21.041985 (+   396us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:21.129555 (+ 87570us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:22.910144 (+1780589us) spinlock_profiling.cc:243] Waited 71.7 ms on lock 0x7b5400310e54. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:25:22.910326 (+   182us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":164866304}
I20260709 12:25:22.992831 28157 raft_consensus.cc:740] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:25:23.004150 28157 consensus_queue.cc:260] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 25.18, Last appended by leader: 18, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:22.956264 28033 leader_election.cc:290] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 33 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:25:22.964461 18333 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:22.994632 18366 rpcz_store.cc:269] 0709 12:25:21.220882 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:21.897272 (+676390us) service_pool.cc:224] Handling call
0709 12:25:22.919751 (+1022479us) spinlock_profiling.cc:243] Waited 167 ms on lock 0x7b54006144d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:25:22.919894 (+   143us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":384960256}
I20260709 12:25:23.007225 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 30 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:22.983500 18374 rpcz_store.cc:276] 0709 12:25:21.193400 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:22.417062 (+1223662us) service_pool.cc:224] Handling call
0709 12:25:22.417427 (+   365us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:22.489464 (+ 72037us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:22.905158 (+415694us) spinlock_profiling.cc:243] Waited 65.1 ms on lock 0x7b540034c1d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:25:22.905342 (+   184us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":149698816}
I20260709 12:25:23.008569 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 36 candidate_status { last_received { term: 33 index: 25 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:23.009302 18374 raft_consensus.cc:2468] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 35.
I20260709 12:25:22.985128 27856 raft_consensus.cc:515] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:22.956307 28112 leader_election.cc:290] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:25:22.998179 17761 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38786 (request call id 4266) took 1053 ms. Trace:
W20260709 12:25:22.998481 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 66927.8us
I20260709 12:25:23.011518 18366 raft_consensus.cc:2468] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 29.
I20260709 12:25:23.000200 17689 rpcz_store.cc:276] 0709 12:25:21.490850 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:21.529361 (+ 38511us) service_pool.cc:224] Handling call
0709 12:25:21.530036 (+   675us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:21.571375 (+ 41339us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:22.949439 (+1378064us) spinlock_profiling.cc:243] Waited 38.6 ms on lock 0x7b54004277d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:25:22.949644 (+   205us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":88686464}
I20260709 12:25:23.060186 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 31 candidate_status { last_received { term: 30 index: 21 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:23.060494 17801 leader_election.cc:304] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:25:23.060876 18374 raft_consensus.cc:2468] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 30.
I20260709 12:25:23.059988 18333 leader_election.cc:304] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
W20260709 12:25:22.985818 27892 consensus_meta.cc:287] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.987s	user 0.010s	sys 0.003s
I20260709 12:25:22.999171 17763 rpcz_store.cc:276] 0709 12:25:21.647869 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:21.649801 (+  1932us) service_pool.cc:224] Handling call
0709 12:25:22.938021 (+1288220us) spinlock_profiling.cc:243] Waited 970 ms on lock 0x7b5400400bd4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:25:22.938165 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2230662528}
I20260709 12:25:23.061419 17761 rpcz_store.cc:276] 0709 12:25:21.945241 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:21.945406 (+   165us) service_pool.cc:224] Handling call
0709 12:25:21.985969 (+ 40563us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:22.177775 (+191806us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:22.177826 (+    51us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:22.177890 (+    64us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:25:22.177941 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:22.997890 (+819949us) spinlock_profiling.cc:243] Waited 192 ms on lock 0x7b540019fad4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:25:22.997955 (+    65us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:22.998106 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":440465664}
I20260709 12:25:22.996569 17837 rpcz_store.cc:276] 0709 12:25:21.131053 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:21.408223 (+277170us) service_pool.cc:224] Handling call
0709 12:25:22.645355 (+1237132us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:22.652271 (+  6916us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:22.926914 (+274643us) spinlock_profiling.cc:243] Waited 1.24 s on lock 0x7b540022b850. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a61f8ad 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:25:22.927072 (+   158us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2844513536}
I20260709 12:25:23.069098 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 26 candidate_status { last_received { term: 24 index: 22 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:23.069190 17801 leader_election.cc:304] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:23.070793 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 28 candidate_status { last_received { term: 27 index: 21 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:23.064392 18333 leader_election.cc:304] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:25:23.072615 28241 raft_consensus.cc:2804] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:25:23.072947 28174 consensus_queue.cc:1048] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
W20260709 12:25:23.073385 18364 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59918 (request call id 2962) took 1882 ms (client timeout 1865 ms). Trace:
I20260709 12:25:23.078126 18374 raft_consensus.cc:1275] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 24 index: 22. Preceding OpId from leader: term: 25 index: 22. (term mismatch)
I20260709 12:25:23.083422 17761 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995"
dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
mode: GRACEFUL
new_leader_uuid: "e063541542a94969898868bf6eb5ce1b"
 from {username='slave'} at 127.0.0.1:49786
I20260709 12:25:22.994271 28184 leader_election.cc:290] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 34 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:22.999315 28154 leader_election.cc:290] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:25:23.064437 28080 consensus_meta.cc:287] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 1.307s	user 0.013s	sys 0.000s
I20260709 12:25:23.125135 28105 raft_consensus.cc:2804] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:25:23.125198 28080 raft_consensus.cc:515] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:23.125730 28105 raft_consensus.cc:493] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:25:23.126118 28105 raft_consensus.cc:3060] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:23.127408 28080 leader_election.cc:290] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 26 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:23.129468 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 35 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:23.130059 17760 raft_consensus.cc:2468] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 34.
I20260709 12:25:23.131711 17758 raft_consensus.cc:1275] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 26 index: 16. Preceding OpId from leader: term: 30 index: 18. (index mismatch)
W20260709 12:25:23.124250 18364 rpcz_store.cc:269] 0709 12:25:21.191077 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:21.191201 (+   124us) service_pool.cc:224] Handling call
0709 12:25:23.073181 (+1881980us) spinlock_profiling.cc:243] Waited 275 ms on lock 0x7b54007e7cd4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:25:23.073316 (+   135us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":631416704}
I20260709 12:25:23.133154 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 34 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:23.133936 17757 raft_consensus.cc:3060] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 32 FOLLOWER]: Advancing to term 34
I20260709 12:25:23.133865 28251 consensus_queue.cc:1048] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:25:22.995793 28258 raft_consensus.cc:515] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:23.137795 28258 leader_election.cc:290] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:23.138353 28258 raft_consensus.cc:2749] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:25:23.122808 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:25:23.123510 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 59193.2us
I20260709 12:25:22.995078 27965 leader_election.cc:290] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:23.086802 28249 raft_consensus.cc:493] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:22.994930 28253 leader_election.cc:290] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:23.000488 28250 raft_consensus.cc:515] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:23.076198 28112 raft_consensus.cc:493] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:23.141633 18333 leader_election.cc:304] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 33 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:23.124097 18374 pending_rounds.cc:85] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da: Aborting all ops after (but not including) 21
I20260709 12:25:23.146992 18374 pending_rounds.cc:107] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da: Aborting uncommitted NO_OP operation due to leader change: 24.22
I20260709 12:25:23.147351 18374 raft_consensus.cc:2889] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:23.000375 28036 raft_consensus.cc:2804] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:25:23.141315 17835 raft_consensus.cc:2468] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 24.
I20260709 12:25:22.996342 27671 leader_election.cc:290] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:23.077080 28172 raft_consensus.cc:493] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:23.147262 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:23.123775 17761 raft_consensus.cc:606] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Received request to transfer leadership to TS e063541542a94969898868bf6eb5ce1b
I20260709 12:25:22.997694 28183 raft_consensus.cc:515] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:22.998747 28016 raft_consensus.cc:515] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:23.151337 18333 leader_election.cc:304] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:23.083422 28157 raft_consensus.cc:3060] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:23.082980 27892 raft_consensus.cc:2749] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 33 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:25:22.990588 28207 raft_consensus.cc:515] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 33 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:23.084631 27766 raft_consensus.cc:493] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:23.155841 17761 raft_consensus.cc:614] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:25:22.997928 27431 raft_consensus.cc:515] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:22.994881 27744 raft_consensus.cc:515] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:23.150858 28241 raft_consensus.cc:697] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 31 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:25:23.195737 27744 leader_election.cc:290] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:22.990720 28011 raft_consensus.cc:515] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:23.196743 28241 consensus_queue.cc:237] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 28.18, Last appended by leader: 18, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:23.159065 27965 raft_consensus.cc:2749] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 33 FOLLOWER]: Leader election lost for term 33. Reason: could not achieve majority
I20260709 12:25:23.199225 27965 consensus_queue.cc:1048] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20260709 12:25:23.160957 28172 raft_consensus.cc:515] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:23.201915 27767 consensus_meta.cc:287] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.930s	user 0.011s	sys 0.000s
I20260709 12:25:23.202499 28172 leader_election.cc:290] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:23.203802 27753 leader_election.cc:290] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 26 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:23.206003 28207 leader_election.cc:290] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 33 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:23.208455 28016 leader_election.cc:290] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:23.209175 27766 raft_consensus.cc:515] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:23.210441 28183 leader_election.cc:290] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:25:23.211272 27766 leader_election.cc:290] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:23.212245 18366 raft_consensus.cc:1275] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 27 index: 21. Preceding OpId from leader: term: 28 index: 21. (term mismatch)
I20260709 12:25:23.212615 18366 pending_rounds.cc:85] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da: Aborting all ops after (but not including) 20
I20260709 12:25:23.212838 18366 pending_rounds.cc:107] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da: Aborting uncommitted NO_OP operation due to leader change: 27.21
I20260709 12:25:23.213068 18366 raft_consensus.cc:2889] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:23.157620 18374 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59902 (request call id 2401) took 1938 ms. Trace:
I20260709 12:25:23.189438 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:23.214351 18374 rpcz_store.cc:276] 0709 12:25:21.219429 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:23.077372 (+1857943us) service_pool.cc:224] Handling call
0709 12:25:23.077636 (+   264us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:23.157416 (+ 79780us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:23.157574 (+   158us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:23.152667 28036 raft_consensus.cc:493] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 33 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:23.198304 28011 leader_election.cc:290] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:23.157605 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 35 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:23.214637 17835 raft_consensus.cc:3060] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:23.190234 17757 raft_consensus.cc:2468] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 34.
I20260709 12:25:23.224963 18374 raft_consensus.cc:1240] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 27. Current term is 28. Ops: []
W20260709 12:25:23.262987 18334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:23.172329 28080 raft_consensus.cc:2804] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:25:23.274338 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:23.283031 17837 raft_consensus.cc:3060] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:23.279706 16501 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260709 12:25:23.148324 28249 raft_consensus.cc:515] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:23.153878 28112 raft_consensus.cc:515] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:23.289772 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:23.292341 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 403749us
I20260709 12:25:23.225814 28148 raft_consensus.cc:2804] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Leader pre-election won for term 27
W20260709 12:25:23.281602 28057 consensus_meta.cc:287] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 1.448s	user 0.009s	sys 0.005s
I20260709 12:25:23.234598 28036 raft_consensus.cc:3060] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:25:23.202433 27767 raft_consensus.cc:515] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:23.300319 17803 leader_election.cc:304] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:23.325944 17803 leader_election.cc:400] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 28 pre-election: Vote denied by peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577) with higher term. Message: Invalid argument: T a5732e71b9d54f8786b8c511d69943bf P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 28. Current term is 29.
I20260709 12:25:23.331450 17803 leader_election.cc:403] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 28 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:25:23.295789 28080 raft_consensus.cc:697] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 27 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:25:23.342376 17803 leader_election.cc:304] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:25:23.299696 27892 raft_consensus.cc:493] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:23.299340 28174 raft_consensus.cc:493] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:25:23.302961 28245 raft_consensus.cc:493] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:23.356925 17655 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:23.305245 28057 raft_consensus.cc:2749] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:25:23.359859 17833 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35040 (request call id 3824) took 2002 ms. Trace:
I20260709 12:25:23.336607 28244 raft_consensus.cc:2749] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:25:23.307966 28256 consensus_queue.cc:1059] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 28.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.055s
I20260709 12:25:23.364627 17833 rpcz_store.cc:276] 0709 12:25:21.357173 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:22.395901 (+1038728us) service_pool.cc:224] Handling call
0709 12:25:22.396197 (+   296us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:22.396378 (+   181us) raft_consensus.cc:1534] Early marking committed up to index 21
0709 12:25:22.396688 (+   310us) log.cc:844] Serialized 14 byte log entry
0709 12:25:22.907309 (+510621us) spinlock_profiling.cc:243] Waited 6.42 ms on lock 0x7b18002dece0. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b47515e07 00007f7b503709a9 00007f7b4753269b 00007f7b4753093f 00007f7b47532c02 00007f7b4a1ba20b 00007f7b4a1bfd23 00007f7b4a1c023a 00007f7b4a62aa2a
0709 12:25:23.359450 (+452141us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:23.359504 (+    54us) raft_consensus.cc:1645] Marking committed up to 21
0709 12:25:23.359551 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:23.359625 (+    74us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:23.359786 (+   161us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":103,"prepare.run_cpu_time_us":53,"prepare.run_wall_time_us":2020,"raft.queue_time_us":114,"spinlock_wait_cycles":14762368,"thread_start_us":451906,"threads_started":1}
I20260709 12:25:23.346335 28054 raft_consensus.cc:2749] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Leader pre-election lost for term 28. Reason: Vote denied by peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577) with higher term. Message: Invalid argument: T a5732e71b9d54f8786b8c511d69943bf P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 28. Current term is 29.
I20260709 12:25:23.399914 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 34 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:23.405800 28011 raft_consensus.cc:493] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:25:23.407238 18368 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 2978) took 2192 ms. Trace:
I20260709 12:25:23.358951 28149 raft_consensus.cc:2804] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:25:23.407658 18368 rpcz_store.cc:276] 0709 12:25:21.214763 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:22.865683 (+1650920us) service_pool.cc:224] Handling call
0709 12:25:22.866037 (+   354us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:22.873000 (+  6963us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:22.873034 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:22.873618 (+   584us) log.cc:844] Serialized 26 byte log entry
0709 12:25:23.216580 (+342962us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:25:23.216629 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:23.216749 (+   120us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:23.406995 (+190246us) raft_consensus.cc:1685] finished
0709 12:25:23.407025 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:23.407178 (+   153us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":621,"raft.run_cpu_time_us":89,"raft.run_wall_time_us":89,"thread_start_us":313828,"threads_started":1,"wal-append.queue_time_us":523896}
I20260709 12:25:23.324666 28105 raft_consensus.cc:515] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:23.414356 28216 log.cc:927] Time spent T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 1.087s	user 0.000s	sys 0.005s
I20260709 12:25:23.415417 18368 raft_consensus.cc:1240] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 28. Current term is 29. Ops: []
W20260709 12:25:23.416314 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:23.416842 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 35 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:23.434675 17830 raft_consensus.cc:2410] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for term 35 because replica has last-logged OpId of term: 32 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 31 index: 18.
I20260709 12:25:23.424202 27856 leader_election.cc:290] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:23.441817 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:23.444082 27744 raft_consensus.cc:493] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:25:23.444864 27744 raft_consensus.cc:515] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:23.368331 17829 raft_consensus.cc:1240] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 27. Current term is 28. Ops: [27.19-27.19]
I20260709 12:25:23.363240 17835 raft_consensus.cc:2468] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 23.
W20260709 12:25:23.474931 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 405809us
W20260709 12:25:23.476819 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 348575us
I20260709 12:25:23.504166 18368 raft_consensus.cc:1275] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 28 index: 18. Preceding OpId from leader: term: 30 index: 19. (index mismatch)
I20260709 12:25:23.406558 28011 raft_consensus.cc:515] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:23.511435 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 145858us
I20260709 12:25:23.412483 28149 raft_consensus.cc:493] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
W20260709 12:25:23.510002 17652 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:23.359382 27892 raft_consensus.cc:515] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:23.362519 28174 raft_consensus.cc:515] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:23.530980 17803 leader_election.cc:304] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:25:23.529058 17652 leader_election.cc:304] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:25:23.538687 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 377025us
I20260709 12:25:23.537570 17652 leader_election.cc:304] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:23.504317 28119 raft_consensus.cc:493] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
W20260709 12:25:23.566840 17832 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.566s	user 0.008s	sys 0.031s
I20260709 12:25:23.561158 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 29 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:23.561507 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 29 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:25:23.554646 27881 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.600s	user 0.029s	sys 0.045s
I20260709 12:25:23.510211 28250 leader_election.cc:290] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 27 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:23.535646 28149 raft_consensus.cc:3060] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:23.569002 28119 raft_consensus.cc:515] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:23.570248 27881 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.615s	user 0.030s	sys 0.046s
I20260709 12:25:23.575385 28112 leader_election.cc:290] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:25:23.569317 17832 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.569s	user 0.008s	sys 0.031s
I20260709 12:25:23.551148 28251 raft_consensus.cc:2804] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:25:23.576526 28251 raft_consensus.cc:493] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 34 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:25:23.576970 28251 raft_consensus.cc:3060] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:25:23.536211 27766 raft_consensus.cc:493] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:23.577495 28119 leader_election.cc:290] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:23.577903 27766 raft_consensus.cc:515] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:23.579913 28174 leader_election.cc:290] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:23.580785 27766 leader_election.cc:290] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:23.582211 28011 leader_election.cc:290] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:23.467015 28245 raft_consensus.cc:515] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:23.586045 28245 leader_election.cc:290] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:23.593115 18366 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59902 (request call id 2409) took 2371 ms. Trace:
I20260709 12:25:23.593704 18366 rpcz_store.cc:276] 0709 12:25:21.221683 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:23.088787 (+1867104us) service_pool.cc:224] Handling call
0709 12:25:23.089181 (+   394us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:23.213528 (+124347us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:23.592765 (+379237us) spinlock_profiling.cc:243] Waited 123 ms on lock 0x7b54007e7cd4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:25:23.593026 (+   261us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":282190208}
I20260709 12:25:23.594403 27892 leader_election.cc:290] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:23.596844 18363 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 2975) took 2396 ms. Trace:
I20260709 12:25:23.597121 27744 leader_election.cc:290] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:23.563293 27431 leader_election.cc:290] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:23.597388 18363 rpcz_store.cc:276] 0709 12:25:21.200115 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:22.543700 (+1343585us) service_pool.cc:224] Handling call
0709 12:25:22.920639 (+376939us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:22.995098 (+ 74459us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:23.596574 (+601476us) spinlock_profiling.cc:243] Waited 286 ms on lock 0x7b54006144d0. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a61f8ad 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:25:23.596764 (+   190us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":658177408}
I20260709 12:25:23.599200 17680 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37402 (request call id 4057) took 2227 ms. Trace:
I20260709 12:25:23.599752 17680 rpcz_store.cc:276] 0709 12:25:21.371676 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:21.371914 (+   238us) service_pool.cc:224] Handling call
0709 12:25:21.395697 (+ 23783us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:22.268662 (+872965us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:23.598848 (+1330186us) spinlock_profiling.cc:243] Waited 871 ms on lock 0x7b5400338954. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:25:23.599085 (+   237us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2002715008}
W20260709 12:25:23.601598 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 89170.5us
I20260709 12:25:23.595242 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 28 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:23.602499 17803 leader_election.cc:304] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:25:23.603616 18366 raft_consensus.cc:2410] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 28 because replica has last-logged OpId of term: 26 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 16.
I20260709 12:25:23.607333 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:23.608157 17830 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 29.
W20260709 12:25:23.506222 17834 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.512s	user 0.026s	sys 0.023s
I20260709 12:25:23.553671 28016 raft_consensus.cc:493] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:23.612528 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 31 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:23.612650 28016 raft_consensus.cc:515] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:23.613607 17829 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35040 (request call id 3833) took 1959 ms. Trace:
I20260709 12:25:23.613845 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:23.613986 17829 rpcz_store.cc:276] 0709 12:25:21.654292 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:22.484612 (+830320us) service_pool.cc:224] Handling call
0709 12:25:22.489212 (+  4600us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:23.474180 (+984968us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:23.613354 (+139174us) spinlock_profiling.cc:243] Waited 879 ms on lock 0x7b540008b6d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:25:23.613552 (+   198us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2021073280}
I20260709 12:25:23.614814 28016 leader_election.cc:290] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:23.615100 17830 raft_consensus.cc:2393] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 29: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:25:23.615244 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 30 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:23.532948 28253 raft_consensus.cc:493] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:25:23.510049 28244 consensus_queue.cc:1048] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.057s
I20260709 12:25:23.615851 17829 raft_consensus.cc:2468] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 29.
I20260709 12:25:23.619181 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 28 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:23.619762 17829 raft_consensus.cc:3060] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:23.621176 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 32 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
W20260709 12:25:23.616997 17834 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.623s	user 0.026s	sys 0.023s
I20260709 12:25:23.615970 28253 raft_consensus.cc:515] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:23.616288 17836 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37994 (request call id 1900) took 2715 ms. Trace:
I20260709 12:25:23.633983 17836 rpcz_store.cc:276] 0709 12:25:20.900798 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:21.246096 (+345298us) service_pool.cc:224] Handling call
0709 12:25:21.341364 (+ 95268us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:21.342050 (+   686us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:23.616123 (+2274073us) spinlock_profiling.cc:243] Waited 94.8 ms on lock 0x7b540008cad0. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a61f8ad 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:25:23.616238 (+   115us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":218052992}
I20260709 12:25:23.635394 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:23.616452 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 31 candidate_status { last_received { term: 30 index: 21 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:23.629477 28126 raft_consensus.cc:3055] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 32 LEADER]: Stepping down as leader of term 32
I20260709 12:25:23.636494 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 26 candidate_status { last_received { term: 23 index: 21 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:23.637111 18371 raft_consensus.cc:2468] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 25.
I20260709 12:25:23.636667 17836 raft_consensus.cc:3060] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Advancing to term 26
W20260709 12:25:23.638947 18333 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.16.29.66:39651 timed out after 1.747s (SENT) [suppressed 18 similar messages]
I20260709 12:25:23.640745 18333 leader_election.cc:304] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:23.642508 18333 leader_election.cc:304] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:25:23.562989 28095 raft_consensus.cc:493] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:23.651955 18333 leader_election.cc:304] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:25:23.654517 17832 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35040 (request call id 3834) took 1999 ms. Trace:
I20260709 12:25:23.654865 17837 raft_consensus.cc:2468] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 30.
I20260709 12:25:23.654961 17832 rpcz_store.cc:276] 0709 12:25:21.654644 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:22.650902 (+996258us) service_pool.cc:224] Handling call
0709 12:25:22.651247 (+   345us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:25:22.651601 (+   354us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:22.651628 (+    27us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:25:22.654225 (+  2597us) log.cc:844] Serialized 52 byte log entry
0709 12:25:22.974589 (+320364us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:25:23.581510 (+606921us) log.cc:844] Serialized 14 byte log entry
0709 12:25:23.654022 (+ 72512us) log.cc:844] Serialized 14 byte log entry
0709 12:25:23.654224 (+   202us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:23.654298 (+    74us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:23.654333 (+    35us) raft_consensus.cc:1685] finished
0709 12:25:23.654347 (+    14us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:23.654465 (+   118us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":66029,"raft.queue_time_us":74970,"raft.run_cpu_time_us":93,"raft.run_wall_time_us":1309,"thread_start_us":821488,"threads_started":2,"wal-append.queue_time_us":530740}
I20260709 12:25:23.667315 17830 raft_consensus.cc:3060] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Advancing to term 31
W20260709 12:25:23.669293 17654 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:23.657267 28016 raft_consensus.cc:2804] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:25:23.648427 28126 raft_consensus.cc:740] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 32 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:25:23.648844 28146 raft_consensus.cc:2749] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:25:23.654430 27766 raft_consensus.cc:2804] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:25:23.552266 28080 consensus_queue.cc:237] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 25.18, Last appended by leader: 18, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:23.651707 28119 consensus_queue.cc:1048] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.034s
I20260709 12:25:23.689111 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 29 candidate_status { last_received { term: 27 index: 21 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:23.674348 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 29 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:23.688802 17830 raft_consensus.cc:2468] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 31.
I20260709 12:25:23.674379 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 36 candidate_status { last_received { term: 33 index: 25 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:23.671682 17836 raft_consensus.cc:2468] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 26.
I20260709 12:25:23.685935 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 33 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:25:23.699257 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 32 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:23.699582 17680 raft_consensus.cc:3060] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:25:23.676303 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 33 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:23.700237 17832 raft_consensus.cc:2410] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for term 32 because replica has last-logged OpId of term: 30 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 28 index: 17.
W20260709 12:25:23.703122 17727 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.745s	user 0.059s	sys 0.141s
I20260709 12:25:23.699864 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 29 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:25:23.703521 17727 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.746s	user 0.059s	sys 0.141s
I20260709 12:25:23.702638 18333 leader_election.cc:304] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:25:23.705555 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:23.706209 17832 raft_consensus.cc:2468] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 27.
I20260709 12:25:23.707233 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 23 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:23.708729 17836 raft_consensus.cc:2468] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 28.
I20260709 12:25:23.711387 18333 leader_election.cc:304] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:25:23.712558 17832 raft_consensus.cc:2468] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 22.
I20260709 12:25:23.656920 28095 raft_consensus.cc:515] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:23.717835 28095 leader_election.cc:290] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:23.716636 17832 raft_consensus.cc:1240] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 26. Current term is 27. Ops: []
I20260709 12:25:23.733726 28149 raft_consensus.cc:515] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:23.735504 17832 raft_consensus.cc:1240] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 27. Current term is 28. Ops: []
W20260709 12:25:23.739284 28265 log.cc:927] Time spent T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.173s	user 0.000s	sys 0.001s
I20260709 12:25:23.741092 28045 consensus_queue.cc:1048] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20260709 12:25:23.742964 28045 consensus_queue.cc:1048] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.000s
I20260709 12:25:23.743652 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 28 candidate_status { last_received { term: 27 index: 21 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:23.744155 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 33 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:23.744513 17836 raft_consensus.cc:3060] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:23.745364 18333 leader_election.cc:304] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
W20260709 12:25:23.745227 17727 leader_election.cc:336] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 34 pre-election: RPC error from VoteRequest() call to peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837): Timed out: RequestConsensusVote RPC to 127.16.29.67:36837 timed out after 1.641s (SENT)
I20260709 12:25:23.756292 17836 raft_consensus.cc:2468] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 28.
I20260709 12:25:23.757426 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 33 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:23.757620 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 28 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:23.758535 18371 raft_consensus.cc:2410] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for term 33 because replica has last-logged OpId of term: 31 index: 21, which is greater than that of the candidate, which has last-logged OpId of term: 30 index: 20.
I20260709 12:25:23.760053 17836 raft_consensus.cc:2468] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 27.
I20260709 12:25:23.672264 28244 raft_consensus.cc:2749] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:25:23.757721 18333 leader_election.cc:304] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:25:23.697507 28105 leader_election.cc:290] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 28 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:23.767746 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 33 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:25:23.762558 18362 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 2957) took 3578 ms. Trace:
I20260709 12:25:23.685163 27881 raft_consensus.cc:2804] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:25:23.687672 27766 raft_consensus.cc:493] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:25:23.765825 17689 raft_consensus.cc:1240] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 29. Current term is 30. Ops: []
I20260709 12:25:23.690768 28016 raft_consensus.cc:697] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 23 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:25:23.683302 28184 raft_consensus.cc:493] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:25:23.705410 28262 raft_consensus.cc:493] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:23.677016 27767 leader_election.cc:290] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 21 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:23.765045 18333 leader_election.cc:304] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:25:23.748172 27753 raft_consensus.cc:2804] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Leader election won for term 26
W20260709 12:25:23.780817 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 178413us
I20260709 12:25:23.768705 18362 rpcz_store.cc:276] 0709 12:25:20.184113 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:20.194430 (+ 10317us) service_pool.cc:224] Handling call
0709 12:25:22.402702 (+2208272us) spinlock_profiling.cc:243] Waited 1.09 ms on lock 0x7b5800102398. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b474fbb71 00007f7b474fba22 00007f7b504148d9 00007f7b4f56eee0 00007f7b4f6380ab 00007f7b4f638131 00007f7b4f5f9fc0 00007f7b4f5e4929 00007f7b4f5e43d2 00007f7b48b5bbd4
0709 12:25:22.402910 (+   208us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:22.403115 (+   205us) raft_consensus.cc:1534] Early marking committed up to index 19
0709 12:25:22.411651 (+  8536us) log.cc:844] Serialized 14 byte log entry
0709 12:25:23.751586 (+1339935us) spinlock_profiling.cc:243] Waited 5.7 ms on lock 0x7b18000b12e0. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b4a1c84bc 00007f7b4751648b 00007f7b503709a9 00007f7b4753269b 00007f7b4753093f 00007f7b47532c02 00007f7b4a1ba20b 00007f7b4a1bfd23 00007f7b4a1c023a
0709 12:25:23.762197 (+ 10611us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:23.762259 (+    62us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:25:23.762308 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:23.762371 (+    63us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:23.762502 (+   131us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3668,"prepare.queue_time_us":10497,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":39,"raft.queue_time_us":5391,"spinlock_wait_cycles":15630976,"thread_start_us":66338,"threads_started":1,"wal-append.queue_time_us":114551,"wal-append.run_cpu_time_us":1067,"wal-append.run_wall_time_us":1006210}
I20260709 12:25:23.754892 28011 raft_consensus.cc:493] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:23.759738 28095 raft_consensus.cc:2804] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:25:23.783552 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 35 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:23.675215 28112 consensus_queue.cc:1048] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.091s
I20260709 12:25:23.749989 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:23.777648 17680 raft_consensus.cc:2468] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 33.
I20260709 12:25:23.763381 28146 raft_consensus.cc:2804] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:25:23.788518 17803 leader_election.cc:304] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:25:23.794204 17837 raft_consensus.cc:3060] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:23.778496 27881 raft_consensus.cc:493] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:23.789510 17682 raft_consensus.cc:1275] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 28 index: 21. Preceding OpId from leader: term: 29 index: 22. (index mismatch)
I20260709 12:25:23.731067 28251 raft_consensus.cc:515] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 35 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:23.782511 27766 raft_consensus.cc:3060] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Advancing to term 25
W20260709 12:25:23.799347 18334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:23.794104 17688 raft_consensus.cc:2468] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 34.
W20260709 12:25:23.802762 17655 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:23.800951 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 32 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:23.814654 17689 raft_consensus.cc:1275] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 34 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 31 index: 18. Preceding OpId from leader: term: 34 index: 19. (index mismatch)
I20260709 12:25:23.815071 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:23.816301 17829 raft_consensus.cc:2468] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 28.
I20260709 12:25:23.814553 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 31 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:23.788807 28276 raft_consensus.cc:2804] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:25:23.795512 28146 raft_consensus.cc:493] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:25:23.813022 28157 consensus_meta.cc:287] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.641s	user 0.009s	sys 0.002s
I20260709 12:25:23.790949 28274 raft_consensus.cc:2804] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:25:23.793675 28095 raft_consensus.cc:493] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:23.789779 28262 raft_consensus.cc:515] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:25:23.832355 28268 log.cc:927] Time spent T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.124s	user 0.000s	sys 0.000s
I20260709 12:25:23.834100 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:23.840270 17681 raft_consensus.cc:2468] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 26.
I20260709 12:25:23.806548 27753 raft_consensus.cc:697] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 26 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:25:23.834180 17654 leader_election.cc:304] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:23.809368 28174 raft_consensus.cc:2804] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:25:23.809934 27881 raft_consensus.cc:3060] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:23.836134 17837 raft_consensus.cc:2468] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 26.
I20260709 12:25:23.837013 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:23.787891 28184 raft_consensus.cc:515] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:23.759430 28126 consensus_queue.cc:260] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 32.23, Last appended by leader: 23, Current term: 32, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:23.814129 27892 consensus_queue.cc:1048] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.028s
I20260709 12:25:23.857693 27753 consensus_queue.cc:237] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 25.18, Last appended by leader: 18, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:23.864499 17834 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35042 (request call id 3633) took 2946 ms. Trace:
I20260709 12:25:23.874025 17834 rpcz_store.cc:276] 0709 12:25:20.918396 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:21.358988 (+440592us) service_pool.cc:224] Handling call
0709 12:25:21.359326 (+   338us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:21.359759 (+   433us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:21.359802 (+    43us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:21.360374 (+   572us) log.cc:844] Serialized 26 byte log entry
0709 12:25:22.994106 (+1633732us) spinlock_profiling.cc:243] Waited 4.46 ms on lock 0x7b18002c9da0. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b47515e07 00007f7b503709a9 00007f7b4753269b 00007f7b4753093f 00007f7b47532c02 00007f7b4a1ba20b 00007f7b4a1bfd23 00007f7b4a1c00d1 00007f7b4a5dba0f
0709 12:25:23.623133 (+629027us) raft_consensus.cc:1645] Marking committed up to 22
0709 12:25:23.623178 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:23.623249 (+    71us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:23.864162 (+240913us) raft_consensus.cc:1685] finished
0709 12:25:23.864187 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:23.864433 (+   246us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":98,"raft.run_cpu_time_us":155,"raft.run_wall_time_us":22943,"spinlock_wait_cycles":10248064,"thread_start_us":628828,"threads_started":1,"wal-append.queue_time_us":2268998}
W20260709 12:25:23.864850 28203 log.cc:927] Time spent T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 2.017s	user 0.006s	sys 0.006s
I20260709 12:25:23.839252 27767 consensus_queue.cc:1059] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.027s
I20260709 12:25:23.839248 28095 raft_consensus.cc:3060] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:23.834910 28146 raft_consensus.cc:3060] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:23.795261 28011 raft_consensus.cc:515] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:23.868664 17654 leader_election.cc:304] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:23.986389 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:23.837750 28159 consensus_queue.cc:1059] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 28.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.037s
I20260709 12:25:23.850957 17682 raft_consensus.cc:2393] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 26: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:25:23.835618 28274 raft_consensus.cc:493] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:23.880856 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 28 candidate_status { last_received { term: 27 index: 21 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:25:23.867542 28036 consensus_meta.cc:287] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.544s	user 0.010s	sys 0.000s
I20260709 12:25:23.848361 28174 raft_consensus.cc:493] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:25:23.993104 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:23.993923 17683 raft_consensus.cc:2393] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 30: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:25:23.994139 17756 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38776 (request call id 3459) took 2746 ms. Trace:
I20260709 12:25:23.876041 28253 leader_election.cc:290] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:23.850498 28149 leader_election.cc:290] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 28 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:23.985179 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 29 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:25:24.000381 17686 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44898 (request call id 2044) took 2177 ms (client timeout 2069 ms). Trace:
I20260709 12:25:24.001392 17684 raft_consensus.cc:2468] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 28.
W20260709 12:25:24.001320 17686 rpcz_store.cc:269] 0709 12:25:21.817366 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:21.836300 (+ 18934us) service_pool.cc:224] Handling call
0709 12:25:23.880686 (+2044386us) spinlock_profiling.cc:243] Waited 2.3 ms on lock 0x7b5800002a98. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b474fbb71 00007f7b474fbeee 000000000036eb39 00007f7b4f56f1e0 00007f7b4f6380c4 00007f7b4f638131 00007f7b4f5e60a0 00007f7b4f5e5984 00007f7b48b606d4 00007f7b48b60662
0709 12:25:23.994335 (+113649us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5316224}
I20260709 12:25:23.870339 17680 raft_consensus.cc:2468] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 31.
I20260709 12:25:23.989737 17802 leader_election.cc:304] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:23.999832 17682 raft_consensus.cc:1240] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 20. Current term is 21. Ops: []
I20260709 12:25:23.994493 17756 rpcz_store.cc:276] 0709 12:25:21.247763 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:21.249130 (+  1367us) service_pool.cc:224] Handling call
0709 12:25:21.352231 (+103101us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:21.449246 (+ 97015us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:23.993920 (+2544674us) spinlock_profiling.cc:243] Waited 90.3 ms on lock 0x7b54002977d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:25:23.994084 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":207645440}
I20260709 12:25:23.995064 18374 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59886 (request call id 2777) took 1060 ms. Trace:
I20260709 12:25:23.948184 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 29 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:23.883529 28126 raft_consensus.cc:3060] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:25:23.936782 27993 raft_consensus.cc:493] T a5732e71b9d54f8786b8c511d69943bf P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:24.002460 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 215552us
I20260709 12:25:23.978542 28244 raft_consensus.cc:493] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:23.891366 28188 consensus_queue.cc:1175] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "e063541542a94969898868bf6eb5ce1b" responder_term: 32 status { last_received { term: 32 index: 23 } last_committed_idx: 23 last_received_current_leader { term: 32 index: 23 } }
I20260709 12:25:23.938808 28087 raft_consensus.cc:493] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 34 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:25:24.047932 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 28 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:24.048388 28087 raft_consensus.cc:515] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:24.049845 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:24.050408 27965 raft_consensus.cc:493] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:24.050899 27965 raft_consensus.cc:515] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:24.007016 17689 raft_consensus.cc:2468] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 28.
I20260709 12:25:23.986801 27431 raft_consensus.cc:493] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:23.848706 28016 consensus_queue.cc:237] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 22.16, Last appended by leader: 16, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:24.010524 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
W20260709 12:25:24.062448 17652 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.502s	user 0.025s	sys 0.027s
W20260709 12:25:24.063009 17652 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.503s	user 0.026s	sys 0.027s
I20260709 12:25:23.813807 28241 consensus_queue.cc:1048] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.184s
I20260709 12:25:24.013528 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:24.079458 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 28 candidate_status { last_received { term: 27 index: 21 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:24.000557 28054 raft_consensus.cc:493] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:24.019966 27993 raft_consensus.cc:515] T a5732e71b9d54f8786b8c511d69943bf P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:24.087291 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 28 candidate_status { last_received { term: 27 index: 21 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:24.021057 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:24.008306 28253 raft_consensus.cc:2749] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:25:24.025255 28148 raft_consensus.cc:493] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:25:24.026379 28011 leader_election.cc:290] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:23.997520 28174 raft_consensus.cc:3060] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:24.027107 18333 leader_election.cc:304] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:25:23.994278 28119 raft_consensus.cc:2749] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:25:24.093271 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:24.027304 28146 raft_consensus.cc:515] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:24.028471 28254 raft_consensus.cc:3060] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:24.029475 28184 leader_election.cc:290] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:24.030167 27767 raft_consensus.cc:2749] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:25:23.996785 28036 raft_consensus.cc:515] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 34 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:24.121871 27993 leader_election.cc:290] T a5732e71b9d54f8786b8c511d69943bf P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:24.122510 28036 leader_election.cc:290] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 34 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:25:24.031599 28251 leader_election.cc:290] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 35 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:23.994494 28274 raft_consensus.cc:3060] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:24.036563 28262 leader_election.cc:290] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:24.037091 17764 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38776 (request call id 3465) took 1913 ms. Trace:
I20260709 12:25:24.038091 27892 raft_consensus.cc:493] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:24.039521 17838 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35042 (request call id 3630) took 3365 ms (client timeout 1899 ms). Trace:
W20260709 12:25:24.142994 17838 rpcz_store.cc:269] 0709 12:25:20.674309 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:20.933441 (+259132us) service_pool.cc:224] Handling call
0709 12:25:24.039329 (+3105888us) spinlock_profiling.cc:243] Waited 489 ms on lock 0x7b5400159ad4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:25:24.039449 (+   120us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1123738880}
I20260709 12:25:24.041517 17831 raft_consensus.cc:1240] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 22. Current term is 23. Ops: [22.16-22.16]
I20260709 12:25:24.041893 28112 raft_consensus.cc:493] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:25:24.044219 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 29 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:25:24.146358 17655 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:24.089818 28054 raft_consensus.cc:515] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:25:24.046223 28266 log.cc:927] Time spent T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.394s	user 0.000s	sys 0.000s
I20260709 12:25:24.046906 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:24.003374 18374 rpcz_store.cc:276] 0709 12:25:22.934794 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:23.229629 (+294835us) service_pool.cc:224] Handling call
0709 12:25:23.229911 (+   282us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:23.230146 (+   235us) raft_consensus.cc:1534] Early marking committed up to index 23
0709 12:25:23.694143 (+463997us) log.cc:844] Serialized 14 byte log entry
0709 12:25:23.994713 (+300570us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:23.994769 (+    56us) raft_consensus.cc:1645] Marking committed up to 23
0709 12:25:23.994815 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:23.994894 (+    79us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:23.995007 (+   113us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":560944,"prepare.run_cpu_time_us":48,"prepare.run_wall_time_us":2762,"thread_start_us":763799,"threads_started":2}
I20260709 12:25:24.052968 28087 leader_election.cc:290] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:24.053467 28244 raft_consensus.cc:515] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:25:24.063719 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 526212us
I20260709 12:25:24.062120 18362 raft_consensus.cc:2376] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 29. Re-sending same reply.
I20260709 12:25:24.083086 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 28 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:24.080873 18364 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59902 (request call id 2411) took 2858 ms. Trace:
I20260709 12:25:24.082717 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:25:24.095841 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 30 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:25:24.096410 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 69412.2us
W20260709 12:25:24.099601 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 346394us
I20260709 12:25:24.109409 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:25:24.112937 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:24.090592 28126 raft_consensus.cc:493] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:24.118595 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 112460us
I20260709 12:25:24.091558 28148 raft_consensus.cc:3060] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:24.129340 17833 raft_consensus.cc:2393] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 34 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 34: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:25:24.075522 28095 raft_consensus.cc:515] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:24.079398 27431 raft_consensus.cc:515] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:24.177369 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:24.178203 18362 raft_consensus.cc:3060] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:24.162802 18364 rpcz_store.cc:276] 0709 12:25:21.222176 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:23.133921 (+1911745us) service_pool.cc:224] Handling call
0709 12:25:23.134235 (+   314us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 12:25:23.649835 (+515600us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:23.649880 (+    45us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 12:25:23.650637 (+   757us) log.cc:844] Serialized 78 byte log entry
0709 12:25:23.785796 (+135159us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:25:24.063344 (+277548us) log.cc:844] Serialized 14 byte log entry
0709 12:25:24.080004 (+ 16660us) log.cc:844] Serialized 14 byte log entry
0709 12:25:24.080324 (+   320us) log.cc:844] Serialized 14 byte log entry
0709 12:25:24.080505 (+   181us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:24.080598 (+    93us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:24.080638 (+    40us) raft_consensus.cc:1685] finished
0709 12:25:24.080648 (+    10us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:24.080804 (+   156us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":15917,"prepare.queue_time_us":78838,"prepare.run_cpu_time_us":121,"prepare.run_wall_time_us":872,"raft.queue_time_us":51758,"raft.run_cpu_time_us":366,"raft.run_wall_time_us":62071,"spinlock_wait_cycles":12073984,"thread_start_us":878115,"threads_started":3,"wal-append.queue_time_us":208183}
I20260709 12:25:24.138087 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 35 candidate_status { last_received { term: 34 index: 23 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:24.141599 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 31 candidate_status { last_received { term: 30 index: 21 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:24.136583 17764 rpcz_store.cc:276] 0709 12:25:22.123493 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:22.184791 (+ 61298us) service_pool.cc:224] Handling call
0709 12:25:22.947446 (+762655us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:22.999866 (+ 52420us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:24.036908 (+1037042us) spinlock_profiling.cc:243] Waited 762 ms on lock 0x7b5400400bd0. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a61f8ad 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:25:24.037035 (+   127us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1752658816}
I20260709 12:25:24.045537 17681 raft_consensus.cc:1275] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 30 index: 20. Preceding OpId from leader: term: 31 index: 21. (index mismatch)
I20260709 12:25:24.122560 27993 raft_consensus.cc:2804] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Leader election won for term 28
W20260709 12:25:24.148313 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 279760us
I20260709 12:25:24.163316 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 31 candidate_status { last_received { term: 30 index: 21 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:24.161940 18333 leader_election.cc:304] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:25:24.184531 18374 raft_consensus.cc:2393] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 31: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:25:24.152467 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 34 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:24.172129 17652 leader_election.cc:304] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:24.186909 17761 raft_consensus.cc:2393] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 34 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 34: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:25:24.184794 18333 leader_election.cc:304] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:25:24.195221 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 30 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:24.196115 17834 raft_consensus.cc:2468] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 29.
I20260709 12:25:24.196635 17802 leader_election.cc:304] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:25:24.197234 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 28 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:24.198592 17652 leader_election.cc:304] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:25:24.179723 28087 consensus_queue.cc:1059] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 31.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.024s
I20260709 12:25:24.197391 17803 leader_election.cc:304] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:25:24.205726 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 33 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:24.206339 17834 raft_consensus.cc:3060] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:25:24.207168 17829 raft_consensus.cc:1275] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 24 index: 17. Preceding OpId from leader: term: 25 index: 18. (index mismatch)
I20260709 12:25:24.207898 17837 raft_consensus.cc:1240] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 30. Current term is 31. Ops: []
I20260709 12:25:24.208523 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 35 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:24.217389 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 30 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:24.222491 17830 raft_consensus.cc:1275] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 34 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 32 index: 18. Preceding OpId from leader: term: 34 index: 19. (index mismatch)
I20260709 12:25:24.222726 17836 raft_consensus.cc:1275] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 27 index: 16. Preceding OpId from leader: term: 29 index: 17. (index mismatch)
I20260709 12:25:24.223821 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 31 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:24.222558 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:24.224736 17829 raft_consensus.cc:2468] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 29.
I20260709 12:25:24.225203 17652 leader_election.cc:304] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:25:24.225801 17829 raft_consensus.cc:1240] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 29. Current term is 30. Ops: []
I20260709 12:25:24.225196 17836 raft_consensus.cc:1240] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 25. Current term is 26. Ops: []
I20260709 12:25:24.227754 17836 raft_consensus.cc:1240] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 27. Current term is 28. Ops: [27.19-27.19]
I20260709 12:25:24.227376 17829 raft_consensus.cc:1275] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 20 index: 15. Preceding OpId from leader: term: 22 index: 16. (index mismatch)
I20260709 12:25:24.223938 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 32 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:24.241156 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 31 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:24.189917 17654 leader_election.cc:304] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:24.245738 17652 leader_election.cc:304] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:24.148149 27892 raft_consensus.cc:515] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:24.258486 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 53235.4us
I20260709 12:25:24.199472 28157 raft_consensus.cc:2804] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:25:24.306924 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 31 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:24.271003 28256 raft_consensus.cc:2804] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:25:24.273238 28289 raft_consensus.cc:2804] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:25:24.259444 17834 raft_consensus.cc:2468] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 33.
I20260709 12:25:24.290608 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 33 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:24.309899 17763 raft_consensus.cc:3060] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:25:24.318384 28249 leader_election.cc:290] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:24.149396 28112 raft_consensus.cc:515] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:24.270938 28045 consensus_queue.cc:1048] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.009s
I20260709 12:25:24.204077 28262 raft_consensus.cc:2749] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:25:24.255746 17829 raft_consensus.cc:1240] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 25. Current term is 26. Ops: []
I20260709 12:25:24.353215 17763 raft_consensus.cc:2468] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 33.
I20260709 12:25:24.193216 28036 consensus_queue.cc:1048] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.079s
I20260709 12:25:24.147387 28276 raft_consensus.cc:697] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 28 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
W20260709 12:25:24.423247 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 177562us
I20260709 12:25:24.240762 28229 raft_consensus.cc:2804] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:25:24.180053 28126 raft_consensus.cc:515] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:24.181643 27766 raft_consensus.cc:515] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:25:24.427762 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 160791us
I20260709 12:25:24.429495 17830 raft_consensus.cc:2468] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 30.
W20260709 12:25:24.307320 27978 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.568s	user 0.023s	sys 0.015s
I20260709 12:25:24.215476 28110 consensus_queue.cc:1048] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.204s
I20260709 12:25:24.203915 28247 consensus_queue.cc:1059] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 20.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.214s
I20260709 12:25:24.223501 27767 consensus_queue.cc:1059] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 26.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.199s
I20260709 12:25:24.469139 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 35 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:24.322165 28256 raft_consensus.cc:493] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:24.485613 18371 raft_consensus.cc:3060] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:24.466720 27965 leader_election.cc:290] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:24.471189 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:24.418532 28270 raft_consensus.cc:2804] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:25:24.553273 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 35 candidate_status { last_received { term: 34 index: 23 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:24.425770 27993 raft_consensus.cc:697] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 28 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:25:24.468117 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 33 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:25:24.495410 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 305568us
I20260709 12:25:24.472532 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 33 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:24.518335 17759 raft_consensus.cc:2410] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 28 because replica has last-logged OpId of term: 25 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 17.
I20260709 12:25:24.437371 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 31 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:24.519251 17652 leader_election.cc:304] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:24.485272 28189 raft_consensus.cc:2804] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 34 FOLLOWER]: Leader election won for term 34
I20260709 12:25:24.548329 28315 consensus_queue.cc:1048] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:25:24.549052 28313 raft_consensus.cc:493] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:25:24.518813 28159 raft_consensus.cc:2749] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:25:24.307370 27671 consensus_queue.cc:1059] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 30.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.215s
W20260709 12:25:24.504180 27978 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.765s	user 0.023s	sys 0.018s
I20260709 12:25:24.555002 17685 raft_consensus.cc:1240] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 24. Current term is 25. Ops: []
I20260709 12:25:24.472100 17801 leader_election.cc:304] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:24.475339 28253 raft_consensus.cc:2804] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:25:24.422731 28157 raft_consensus.cc:697] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 26 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:25:24.517189 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 32 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:24.564648 17764 raft_consensus.cc:2393] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 33: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:25:24.562137 17762 raft_consensus.cc:2468] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 32.
I20260709 12:25:24.472955 28289 raft_consensus.cc:697] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 31 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:25:24.550240 28251 raft_consensus.cc:3055] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:25:24.619422 28146 leader_election.cc:290] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 23 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:24.558121 28262 consensus_queue.cc:1059] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 28.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.094s
W20260709 12:25:24.669543 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 197553us
I20260709 12:25:24.670365 17801 leader_election.cc:304] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:24.577576 18371 raft_consensus.cc:2468] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 27.
I20260709 12:25:24.514372 28256 raft_consensus.cc:3060] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:24.421947 28175 consensus_queue.cc:1059] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 26.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.141s
I20260709 12:25:24.549741 28152 raft_consensus.cc:493] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:24.679397 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 120516us
I20260709 12:25:24.569234 28242 raft_consensus.cc:2749] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:25:24.642247 28282 consensus_queue.cc:1059] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 32.19, Next index: 20, Last known committed idx: 17, Time since last communication: 0.046s
I20260709 12:25:24.671862 17801 leader_election.cc:304] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:24.665971 28054 leader_election.cc:290] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:25:24.662833 28251 raft_consensus.cc:740] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:25:24.677102 28247 raft_consensus.cc:2749] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:25:24.679021 28271 raft_consensus.cc:3055] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:25:24.708997 17801 leader_election.cc:304] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:24.676093 28253 raft_consensus.cc:697] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 29 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:25:24.680363 28305 raft_consensus.cc:3055] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:25:24.676869 28229 raft_consensus.cc:697] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 30 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:25:24.644827 28313 raft_consensus.cc:515] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:24.688213 28303 raft_consensus.cc:2749] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:25:24.699721 28189 raft_consensus.cc:697] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 34 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:25:24.704262 28315 raft_consensus.cc:3055] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:25:24.688549 28110 consensus_queue.cc:1048] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.039s
I20260709 12:25:24.704509 28095 leader_election.cc:290] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 28 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:25:24.730847 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 50221.8us
I20260709 12:25:24.660090 28276 consensus_queue.cc:237] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 27.21, Last appended by leader: 21, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:24.692970 27767 consensus_queue.cc:1048] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 24, Time since last communication: 0.046s
I20260709 12:25:24.723975 27978 raft_consensus.cc:2749] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:25:24.711486 28271 raft_consensus.cc:740] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:25:24.723767 28305 raft_consensus.cc:740] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 20 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:25:24.688908 28152 raft_consensus.cc:515] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:24.702564 28159 consensus_queue.cc:1059] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 22.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.042s
I20260709 12:25:24.730692 28244 leader_election.cc:290] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:25:24.762624 28324 consensus_queue.cc:1059] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 26.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
W20260709 12:25:24.758140 28291 log.cc:927] Time spent T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.200s	user 0.001s	sys 0.000s
I20260709 12:25:24.742159 28292 raft_consensus.cc:2749] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
W20260709 12:25:24.757957 28279 log.cc:927] Time spent T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.329s	user 0.000s	sys 0.001s
I20260709 12:25:24.746898 28315 raft_consensus.cc:740] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 30 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:25:24.787811 28112 leader_election.cc:290] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:24.882781 28253 consensus_queue.cc:237] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 28.19, Last appended by leader: 19, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:24.752897 28175 raft_consensus.cc:3055] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 32 LEADER]: Stepping down as leader of term 32
I20260709 12:25:24.758056 28303 raft_consensus.cc:2749] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
W20260709 12:25:24.951819 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 242923us
I20260709 12:25:24.982728 17801 leader_election.cc:304] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:24.729861 28157 consensus_queue.cc:237] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 25.16, Last appended by leader: 16, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:24.883733 28229 consensus_queue.cc:237] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 29.17, Last appended by leader: 17, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:24.693373 27993 consensus_queue.cc:237] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 25.18, Last appended by leader: 18, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:24.771673 28189 consensus_queue.cc:237] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 32.18, Last appended by leader: 18, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:25:24.957523 18364 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.684s	user 0.002s	sys 0.006s
W20260709 12:25:24.958609 18362 consensus_meta.cc:287] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.780s	user 0.006s	sys 0.005s
I20260709 12:25:24.962752 17654 leader_election.cc:304] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:25:24.951849 28298 log.cc:927] Time spent T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.292s	user 0.005s	sys 0.012s
W20260709 12:25:24.976429 17729 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.556s	user 0.002s	sys 0.010s
W20260709 12:25:25.001000 17729 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.581s	user 0.003s	sys 0.010s
I20260709 12:25:25.000443 18362 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 28.
W20260709 12:25:24.980053 17756 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.509s	user 0.008s	sys 0.005s
W20260709 12:25:24.976468 28310 log.cc:927] Time spent T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.304s	user 0.003s	sys 0.014s
W20260709 12:25:24.954902 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 444809us
I20260709 12:25:24.753095 28251 consensus_queue.cc:260] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 26.18, Last appended by leader: 18, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:24.959659 28183 raft_consensus.cc:493] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 34 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:25:24.959491 28095 raft_consensus.cc:493] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
W20260709 12:25:25.000361 18364 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.749s	user 0.002s	sys 0.007s
I20260709 12:25:24.981974 28175 raft_consensus.cc:740] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 32 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:25:25.003389 18362 raft_consensus.cc:1275] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 28 index: 17. Preceding OpId from leader: term: 31 index: 19. (index mismatch)
I20260709 12:25:25.004271 17654 leader_election.cc:304] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:25:25.005551 17756 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.535s	user 0.008s	sys 0.005s
I20260709 12:25:24.956723 28126 leader_election.cc:290] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:25.005300 17801 leader_election.cc:304] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:25:25.010294 17729 connection.cc:256] RPC call timeout handler was delayed by -1.5408841970001959s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2082s for a response
I20260709 12:25:24.754901 28289 consensus_queue.cc:237] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 30.21, Last appended by leader: 21, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:25.010474 17727 leader_election.cc:304] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:25:25.002755 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 483566us
I20260709 12:25:25.013856 18364 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59902 (request call id 2425) took 1002 ms. Trace:
I20260709 12:25:25.017671 28095 raft_consensus.cc:515] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:25.033289 17688 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37402 (request call id 4064) took 1151 ms. Trace:
I20260709 12:25:25.037078 18364 rpcz_store.cc:276] 0709 12:25:24.025648 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:24.215349 (+189701us) service_pool.cc:224] Handling call
0709 12:25:24.215704 (+   355us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:25.013470 (+797766us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:25.013502 (+    32us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:25.013549 (+    47us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:25:25.013585 (+    36us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:25.013666 (+    81us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:25.013796 (+   130us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":80108,"raft.queue_time_us":88959,"raft.run_cpu_time_us":211,"raft.run_wall_time_us":112104,"thread_start_us":762540,"threads_started":1}
I20260709 12:25:25.039043 17654 leader_election.cc:304] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:25.010074 28159 raft_consensus.cc:2804] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:25:25.041553 17801 leader_election.cc:304] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:25.007409 28292 raft_consensus.cc:2749] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
W20260709 12:25:25.044449 28296 log.cc:927] Time spent T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.228s	user 0.000s	sys 0.000s
W20260709 12:25:25.045006 27881 consensus_meta.cc:287] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 1.173s	user 0.006s	sys 0.005s
I20260709 12:25:25.045447 27881 raft_consensus.cc:515] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:25.046871 17835 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35040 (request call id 3838) took 1918 ms. Trace:
I20260709 12:25:25.032437 28251 raft_consensus.cc:3060] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:25.048321 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 33 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:25.040191 17652 leader_election.cc:304] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:25.057006 17758 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38786 (request call id 4277) took 1007 ms. Trace:
I20260709 12:25:25.057472 17758 rpcz_store.cc:276] 0709 12:25:24.049511 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:24.056538 (+  7027us) service_pool.cc:224] Handling call
0709 12:25:24.056826 (+   288us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:24.076850 (+ 20024us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:24.076899 (+    49us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:24.077523 (+   624us) log.cc:844] Serialized 26 byte log entry
0709 12:25:24.180193 (+102670us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:25:24.180258 (+    65us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:24.180362 (+   104us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:25.056786 (+876424us) raft_consensus.cc:1685] finished
0709 12:25:25.056807 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:25.056952 (+   145us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":123,"raft.run_cpu_time_us":159,"raft.run_wall_time_us":17797,"thread_start_us":84926,"threads_started":1,"wal-append.queue_time_us":422006}
I20260709 12:25:25.050314 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 31 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:25.059667 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 23 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:25.060199 18364 raft_consensus.cc:2410] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for term 31 because replica has last-logged OpId of term: 28 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 27 index: 17.
I20260709 12:25:25.060286 17829 raft_consensus.cc:3060] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:25.049401 18374 raft_consensus.cc:2393] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 30: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:25:25.066458 28095 leader_election.cc:290] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:25.037284 17688 rpcz_store.cc:276] 0709 12:25:23.882038 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:23.882220 (+   182us) service_pool.cc:224] Handling call
0709 12:25:23.882655 (+   435us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:23.883087 (+   432us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:23.883128 (+    41us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:23.937193 (+ 54065us) log.cc:844] Serialized 26 byte log entry
0709 12:25:24.165716 (+228523us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:25:24.165781 (+    65us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:24.165882 (+   101us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:25.033050 (+867168us) raft_consensus.cc:1685] finished
0709 12:25:25.033073 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:25.033227 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4029,"raft.queue_time_us":111006,"raft.run_cpu_time_us":159,"raft.run_wall_time_us":10594,"thread_start_us":172220,"threads_started":1,"wal-append.queue_time_us":329664}
I20260709 12:25:25.013733 28249 raft_consensus.cc:493] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:25:25.047277 17835 rpcz_store.cc:276] 0709 12:25:23.128250 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:23.474765 (+346515us) service_pool.cc:224] Handling call
0709 12:25:23.475100 (+   335us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:25:23.513283 (+ 38183us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:23.513318 (+    35us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:25:23.518478 (+  5160us) log.cc:844] Serialized 52 byte log entry
0709 12:25:24.033677 (+515199us) spinlock_profiling.cc:243] Waited 6.81 ms on lock 0x7b18004e0b20. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b47515e07 00007f7b503709a9 00007f7b4753269b 00007f7b4753093f 00007f7b47532c02 00007f7b4a1ba20b 00007f7b4a1bfd23 00007f7b4a1c00d1 00007f7b4a5dba0f
0709 12:25:24.241538 (+207861us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:25:24.418969 (+177431us) log.cc:844] Serialized 14 byte log entry
0709 12:25:24.565218 (+146249us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:24.565312 (+    94us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:25.046546 (+481234us) raft_consensus.cc:1685] finished
0709 12:25:25.046650 (+   104us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:25.046808 (+   158us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":10547,"prepare.queue_time_us":78791,"prepare.run_cpu_time_us":57,"prepare.run_wall_time_us":9053,"raft.queue_time_us":243941,"raft.run_cpu_time_us":261,"raft.run_wall_time_us":9110,"spinlock_wait_cycles":15656832,"thread_start_us":353546,"threads_started":2,"wal-append.queue_time_us":1206247}
I20260709 12:25:25.051812 17654 leader_election.cc:304] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:25.076557 17652 leader_election.cc:304] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:25.074390 17829 raft_consensus.cc:2468] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 23.
I20260709 12:25:25.048975 28159 raft_consensus.cc:493] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:25:25.078739 27881 leader_election.cc:290] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 30 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:25.079059 28159 raft_consensus.cc:3060] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:25.079452 28120 raft_consensus.cc:2804] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 33 FOLLOWER]: Leader election won for term 33
I20260709 12:25:25.082310 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 29 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:25.084713 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 31 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:25.085448 18364 raft_consensus.cc:2468] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 30.
W20260709 12:25:25.086615 28174 consensus_meta.cc:287] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.978s	user 0.014s	sys 0.002s
I20260709 12:25:25.087067 28174 raft_consensus.cc:515] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:25.076440 28153 consensus_queue.cc:1048] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.012s
I20260709 12:25:25.056697 28244 raft_consensus.cc:493] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:25.088554 28152 leader_election.cc:290] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:25.015652 28305 consensus_queue.cc:260] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.13, Last appended by leader: 13, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:25.088932 28244 raft_consensus.cc:515] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:25.089469 28256 raft_consensus.cc:515] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:25.090266 28305 raft_consensus.cc:3060] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:25.068636 28126 raft_consensus.cc:2804] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:25:25.091440 28126 raft_consensus.cc:493] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:25.091905 28126 raft_consensus.cc:3060] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:25.068004 27856 consensus_queue.cc:1059] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 28.20, Next index: 21, Last known committed idx: 19, Time since last communication: 0.024s
I20260709 12:25:25.092231 28320 consensus_queue.cc:1059] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 25.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20260709 12:25:25.092473 28175 consensus_queue.cc:260] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 32.19, Last appended by leader: 19, Current term: 32, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:25.093312 28320 consensus_queue.cc:1059] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 25.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20260709 12:25:25.093638 28175 raft_consensus.cc:3060] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:25:25.094267 28320 consensus_queue.cc:1048] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20260709 12:25:25.095955 28326 consensus_queue.cc:1048] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.027s
I20260709 12:25:25.096751 28326 consensus_queue.cc:1243] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer 334a4e1f59944544b3b3ff36616b57da log is divergent from this leader: its last log entry 26.17 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 16
I20260709 12:25:25.096871 28269 raft_consensus.cc:3055] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:25:25.097273 28269 raft_consensus.cc:740] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 25 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:25:25.097924 28326 raft_consensus.cc:3055] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:25:25.098400 28326 raft_consensus.cc:740] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:25:25.098024 28269 consensus_queue.cc:260] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 25.16, Last appended by leader: 16, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:25.099335 28269 raft_consensus.cc:3060] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:25.096065 28011 consensus_queue.cc:1059] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 25.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
W20260709 12:25:25.100296 28297 log.cc:927] Time spent T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.230s	user 0.001s	sys 0.006s
W20260709 12:25:25.100853 28316 log.cc:927] Time spent T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.330s	user 0.003s	sys 0.005s
I20260709 12:25:25.048666 28303 raft_consensus.cc:2749] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:25:25.032802 28315 consensus_queue.cc:260] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 30.21, Last appended by leader: 21, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:25.104257 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 49072.4us
I20260709 12:25:25.105177 17729 leader_election.cc:304] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:25.109723 18362 raft_consensus.cc:2468] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 33.
I20260709 12:25:25.107540 18333 leader_election.cc:304] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
W20260709 12:25:25.132357 17727 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:25.091616 28256 leader_election.cc:290] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 29 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:25.041069 28228 raft_consensus.cc:2804] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:25:25.135514 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:25:25.137916 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 60064.9us
I20260709 12:25:25.069264 28054 raft_consensus.cc:2749] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:25:25.056034 27767 raft_consensus.cc:2749] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:25:25.149073 28112 raft_consensus.cc:2804] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:25:25.103722 28174 leader_election.cc:290] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 28 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:25.099156 28326 consensus_queue.cc:260] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 27.19, Last appended by leader: 19, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:25.022333 28183 raft_consensus.cc:515] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:25.106671 27671 consensus_queue.cc:1059] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 29.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.060s
I20260709 12:25:25.133236 28249 raft_consensus.cc:515] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:25.171149 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:25.292052 18362 raft_consensus.cc:2468] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 26.
W20260709 12:25:25.292302 17652 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:25.285776 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 28 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:25.281849 17832 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35040 (request call id 3845) took 1799 ms. Trace:
W20260709 12:25:25.290803 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 239063us
I20260709 12:25:25.294471 17832 rpcz_store.cc:276] 0709 12:25:23.482021 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:23.718896 (+236875us) service_pool.cc:224] Handling call
0709 12:25:23.719144 (+   248us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:23.736042 (+ 16898us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:25.281549 (+1545507us) spinlock_profiling.cc:243] Waited 16.1 ms on lock 0x7b540022ae54. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:25:25.281776 (+   227us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":37067648}
I20260709 12:25:25.295629 17654 leader_election.cc:304] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:25:25.289254 17830 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35042 (request call id 3645) took 1081 ms. Trace:
I20260709 12:25:25.296372 17830 rpcz_store.cc:276] 0709 12:25:24.208283 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:24.240912 (+ 32629us) service_pool.cc:224] Handling call
0709 12:25:25.289083 (+1048171us) spinlock_profiling.cc:243] Waited 188 ms on lock 0x7b540008bbd4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:25:25.289194 (+   111us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":431681920}
W20260709 12:25:25.284790 17833 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35040 (request call id 3842) took 2155 ms (client timeout 2066 ms). Trace:
I20260709 12:25:25.297361 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
W20260709 12:25:25.297957 17833 rpcz_store.cc:269] 0709 12:25:23.129251 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:23.399551 (+270300us) service_pool.cc:224] Handling call
0709 12:25:25.284531 (+1884980us) spinlock_profiling.cc:243] Waited 719 ms on lock 0x7b54003354d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:25:25.284710 (+   179us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1653397376}
I20260709 12:25:25.298344 17830 raft_consensus.cc:1240] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 28. Current term is 29. Ops: []
I20260709 12:25:25.148047 28228 raft_consensus.cc:493] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:25.290596 17687 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37396 (request call id 3567) took 1479 ms. Trace:
I20260709 12:25:25.300863 18365 raft_consensus.cc:1275] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 27 index: 20. Preceding OpId from leader: term: 28 index: 20. (term mismatch)
I20260709 12:25:25.301313 18365 pending_rounds.cc:85] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da: Aborting all ops after (but not including) 19
I20260709 12:25:25.301604 18365 pending_rounds.cc:107] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da: Aborting uncommitted NO_OP operation due to leader change: 27.20
I20260709 12:25:25.301833 18365 raft_consensus.cc:2889] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:25.302445 17832 raft_consensus.cc:1240] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 27. Current term is 28. Ops: []
I20260709 12:25:25.304032 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:25.304636 17654 leader_election.cc:304] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:25.304888 17836 raft_consensus.cc:2468] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 25.
W20260709 12:25:25.306265 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:25.312039 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 35 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:25.300089 17687 rpcz_store.cc:276] 0709 12:25:23.810936 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:23.813073 (+  2137us) service_pool.cc:224] Handling call
0709 12:25:23.824359 (+ 11286us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:25:23.835776 (+ 11417us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:23.835841 (+    65us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:25:23.839038 (+  3197us) log.cc:844] Serialized 52 byte log entry
0709 12:25:24.147778 (+308740us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:25:24.147818 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:24.147899 (+    81us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:25.156461 (+1008562us) raft_consensus.cc:1685] finished
0709 12:25:25.156485 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:25.290530 (+134045us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1971,"raft.run_cpu_time_us":119,"raft.run_wall_time_us":3451,"thread_start_us":290423,"threads_started":1,"wal-append.queue_time_us":1297333}
I20260709 12:25:25.317867 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 32 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:25.165102 28153 raft_consensus.cc:2804] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 32 FOLLOWER]: Leader pre-election won for term 33
W20260709 12:25:25.319542 18367 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59918 (request call id 2969) took 4115 ms (client timeout 3042 ms). Trace:
W20260709 12:25:25.329331 18367 rpcz_store.cc:269] 0709 12:25:21.193161 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:22.308331 (+1115170us) service_pool.cc:224] Handling call
0709 12:25:25.299981 (+2991650us) spinlock_profiling.cc:243] Waited 47.8 ms on lock 0x7b540047bdd4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:25:25.309017 (+  9036us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":109843200}
W20260709 12:25:25.260854 28274 consensus_meta.cc:287] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 1.114s	user 0.015s	sys 0.023s
I20260709 12:25:25.291189 28326 raft_consensus.cc:3060] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:25.280625 27892 leader_election.cc:290] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:25.323539 17836 raft_consensus.cc:1275] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 28 index: 17. Preceding OpId from leader: term: 30 index: 18. (index mismatch)
I20260709 12:25:25.156061 28330 consensus_queue.cc:1048] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 17, Time since last communication: 0.080s
I20260709 12:25:25.328817 17835 raft_consensus.cc:1240] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 22. Current term is 23. Ops: []
I20260709 12:25:25.168707 28112 raft_consensus.cc:697] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 23 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:25:25.199805 27978 raft_consensus.cc:3055] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:25:25.338229 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:25.171694 28045 raft_consensus.cc:493] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:25:25.314275 28256 raft_consensus.cc:2804] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:25:25.309407 28054 raft_consensus.cc:493] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:25.193686 28171 raft_consensus.cc:2749] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
W20260709 12:25:25.320573 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 214177us
I20260709 12:25:25.290573 28313 leader_election.cc:290] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:25.309372 28228 raft_consensus.cc:3060] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:25.355676 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 29 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
W20260709 12:25:25.337409 28254 consensus_meta.cc:287] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 1.209s	user 0.012s	sys 0.000s
I20260709 12:25:25.377839 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 26 candidate_status { last_received { term: 24 index: 22 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:25.381664 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 30 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:25.382091 18362 raft_consensus.cc:1240] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 31 LEADER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 30. Current term is 31. Ops: []
I20260709 12:25:25.382287 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 31 candidate_status { last_received { term: 30 index: 21 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:25.329350 28153 raft_consensus.cc:493] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 32 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:25:25.387537 28153 raft_consensus.cc:3060] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:25:25.379266 28045 raft_consensus.cc:515] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:25.387763 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:25:25.388057 28337 log.cc:927] Time spent T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.072s	user 0.000s	sys 0.001s
I20260709 12:25:25.388474 28311 raft_consensus.cc:2804] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:25:25.388919 28311 raft_consensus.cc:493] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 35 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:25:25.389474 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:25:25.384698 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 28 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:25.352819 28229 raft_consensus.cc:2749] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 33 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
W20260709 12:25:25.198822 28328 consensus_queue.cc:1175] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "334a4e1f59944544b3b3ff36616b57da" responder_term: 28 status { last_received { term: 25 index: 18 } last_committed_idx: 18 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 27. Current term is 28. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260709 12:25:25.394582 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 30 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:25.396970 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 31 candidate_status { last_received { term: 30 index: 21 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:25.376101 27978 raft_consensus.cc:740] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:25:25.375806 28011 raft_consensus.cc:493] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:25.380568 28120 raft_consensus.cc:697] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 33 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:25:25.378397 28256 raft_consensus.cc:493] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:25:25.382895 28330 consensus_queue.cc:1243] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer 334a4e1f59944544b3b3ff36616b57da log is divergent from this leader: its last log entry 26.19 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 17
I20260709 12:25:25.391033 28313 raft_consensus.cc:493] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:25.402040 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 31 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:25.338842 28274 raft_consensus.cc:515] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:25.406275 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 30 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:25.402212 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:25.392624 27800 raft_consensus.cc:493] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:25.410243 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:25.394533 28311 raft_consensus.cc:3060] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:25:25.412781 17681 raft_consensus.cc:2468] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 29.
I20260709 12:25:25.480234 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:25:25.602752 28251 consensus_meta.cc:287] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.539s	user 0.004s	sys 0.006s
I20260709 12:25:25.530365 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 36 candidate_status { last_received { term: 33 index: 25 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:25.481369 28183 leader_election.cc:290] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:25.664086 28045 leader_election.cc:290] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:25.665720 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 26 candidate_status { last_received { term: 24 index: 22 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:25.667388 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:25.669166 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 29 candidate_status { last_received { term: 27 index: 21 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:25.439352 27800 raft_consensus.cc:515] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:25.548743 18363 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59902 (request call id 2422) took 1538 ms (client timeout 1635 ms). Trace:
I20260709 12:25:25.466723 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:25.673269 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 30 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:25.673851 17682 raft_consensus.cc:2468] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 29.
I20260709 12:25:25.674233 28133 raft_consensus.cc:3060] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:25.674635 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 30 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:25.676095 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 31 candidate_status { last_received { term: 30 index: 21 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:25:25.676788 18363 rpcz_store.cc:269] 0709 12:25:24.009939 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:24.079134 (+ 69195us) service_pool.cc:224] Handling call
0709 12:25:25.548520 (+1469386us) spinlock_profiling.cc:243] Waited 1.25 s on lock 0x7b54001efd54. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:25:25.548681 (+   161us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2877576064}
I20260709 12:25:25.677371 18367 raft_consensus.cc:1240] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 26 LEADER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 25. Current term is 26. Ops: []
I20260709 12:25:25.678102 18363 raft_consensus.cc:1240] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 33 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 32. Current term is 33. Ops: []
I20260709 12:25:25.678664 18367 raft_consensus.cc:1275] T a5732e71b9d54f8786b8c511d69943bf P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 27 index: 16. Preceding OpId from leader: term: 29 index: 17. (index mismatch)
I20260709 12:25:25.681759 17831 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37994 (request call id 1913) took 4488 ms. Trace:
I20260709 12:25:25.682153 17831 rpcz_store.cc:276] 0709 12:25:21.193126 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:22.381052 (+1187926us) service_pool.cc:224] Handling call
0709 12:25:24.040281 (+1659229us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:24.145669 (+105388us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:25.681557 (+1535888us) spinlock_profiling.cc:243] Waited 1.66 s on lock 0x7b5400159ad0. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a61f8ad 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:25:25.681698 (+   141us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3814924160}
I20260709 12:25:25.684465 17685 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37396 (request call id 3559) took 2013 ms. Trace:
I20260709 12:25:25.684855 17685 rpcz_store.cc:276] 0709 12:25:23.671071 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:23.671222 (+   151us) service_pool.cc:224] Handling call
0709 12:25:23.674829 (+  3607us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:24.650720 (+975891us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:25.684217 (+1033497us) spinlock_profiling.cc:243] Waited 880 ms on lock 0x7b540014dcd4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:25:25.684407 (+   190us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2023652096}
I20260709 12:25:25.686014 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
W20260709 12:25:25.689985 28175 consensus_meta.cc:287] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.596s	user 0.009s	sys 0.004s
W20260709 12:25:25.691570 28295 log.cc:927] Time spent T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.309s	user 0.001s	sys 0.000s
W20260709 12:25:25.498397 17729 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:25.383117 28054 raft_consensus.cc:515] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:25.490188 28153 raft_consensus.cc:515] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 33 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:25.440289 18333 leader_election.cc:336] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 29 election: RPC error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Timed out: RequestConsensusVote RPC to 127.16.29.66:39651 timed out after 1.747s (SENT)
I20260709 12:25:25.704268 18333 leader_election.cc:304] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:25:25.496380 28338 log.cc:927] Time spent T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.206s	user 0.000s	sys 0.000s
W20260709 12:25:25.609426 28148 consensus_meta.cc:287] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 1.410s	user 0.007s	sys 0.007s
I20260709 12:25:25.533654 28249 leader_election.cc:290] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:25.463073 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 23 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:25.532369 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 28 candidate_status { last_received { term: 27 index: 21 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:25.549633 28177 raft_consensus.cc:3055] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:25:25.482543 18365 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59902 (request call id 2407) took 4261 ms. Trace:
I20260709 12:25:25.713768 18365 rpcz_store.cc:276] 0709 12:25:21.221185 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:23.074823 (+1853638us) service_pool.cc:224] Handling call
0709 12:25:25.300546 (+2225723us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:25.314631 (+ 14085us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:25.482312 (+167681us) spinlock_profiling.cc:243] Waited 2.23 s on lock 0x7b540047bdd0. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a61f8ad 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:25:25.482465 (+   153us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5117950208}
W20260709 12:25:25.468031 17652 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:25.446684 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 32 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:25.464092 28112 consensus_queue.cc:237] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 20.15, Last appended by leader: 15, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:25.534545 28159 raft_consensus.cc:515] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:25.408490 28256 raft_consensus.cc:3060] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Advancing to term 26
W20260709 12:25:25.489876 28329 log.cc:927] Time spent T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.111s	user 0.000s	sys 0.000s
I20260709 12:25:25.415946 28313 raft_consensus.cc:515] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:25:25.502599 28334 log.cc:927] Time spent T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.182s	user 0.001s	sys 0.000s
I20260709 12:25:25.486675 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 31 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:25.524073 27978 consensus_queue.cc:260] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 28.20, Last appended by leader: 20, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:25.408859 28011 raft_consensus.cc:515] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:25.488732 28274 leader_election.cc:290] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 28 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:25.649333 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 29 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:25.753845 17755 raft_consensus.cc:2376] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 29. Re-sending same reply.
I20260709 12:25:25.754088 28054 leader_election.cc:290] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:25:25.754905 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 28 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:25:25.650460 28269 consensus_meta.cc:287] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.551s	user 0.012s	sys 0.000s
I20260709 12:25:25.757609 28313 leader_election.cc:290] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:25.757675 28228 raft_consensus.cc:515] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:25.759910 28011 leader_election.cc:290] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:25.654412 18361 raft_consensus.cc:1275] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 28 index: 21. Preceding OpId from leader: term: 29 index: 22. (index mismatch)
I20260709 12:25:25.734567 28284 raft_consensus.cc:3055] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:25:25.760946 28284 raft_consensus.cc:740] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:25:25.761363 17687 raft_consensus.cc:2393] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 30: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:25:25.661475 28244 leader_election.cc:290] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:25:25.644657 17727 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:25.761818 28284 consensus_queue.cc:260] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 28.20, Last appended by leader: 20, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:25.763110 17727 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:25.763343 28284 raft_consensus.cc:3060] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:25.644454 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:25.671958 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 28 candidate_status { last_received { term: 27 index: 21 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:25:25.456102 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 30 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:25:25.614176 28126 consensus_meta.cc:287] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.522s	user 0.007s	sys 0.005s
I20260709 12:25:25.698426 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:25.767261 17681 raft_consensus.cc:2368] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 32. Current term is 33.
I20260709 12:25:25.471079 18364 raft_consensus.cc:1240] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 27. Current term is 28. Ops: []
W20260709 12:25:25.701234 17654 connection.cc:256] RPC call timeout handler was delayed by -1.7234970469999098s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3042s for a response
I20260709 12:25:25.768807 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 31 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:25.542557 17759 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38786 (request call id 4279) took 1460 ms. Trace:
W20260709 12:25:25.705312 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 265057us
I20260709 12:25:25.771205 17686 raft_consensus.cc:1240] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 25. Current term is 26. Ops: []
I20260709 12:25:25.772398 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 23 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:25.703059 17729 leader_election.cc:304] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:25.709899 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 27 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:25.769587 17683 raft_consensus.cc:2468] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 30.
I20260709 12:25:25.719854 17831 raft_consensus.cc:1240] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 27. Current term is 28. Ops: [27.19-27.19]
I20260709 12:25:25.703567 28045 consensus_queue.cc:1048] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.005s
I20260709 12:25:25.706939 28183 raft_consensus.cc:2749] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260709 12:25:25.713675 28177 raft_consensus.cc:740] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:25:25.740271 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 30 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:25.745128 28153 leader_election.cc:290] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 33 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:25.746861 28159 leader_election.cc:290] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 31 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:25.748239 17682 raft_consensus.cc:2393] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 31: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:25:25.711441 28148 raft_consensus.cc:515] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:25.752686 28315 raft_consensus.cc:3060] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:25.801452 28148 leader_election.cc:290] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 27 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:25.746785 17763 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38776 (request call id 3485) took 1278 ms. Trace:
I20260709 12:25:25.802038 17763 rpcz_store.cc:276] 0709 12:25:24.468177 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:24.521581 (+ 53404us) service_pool.cc:224] Handling call
0709 12:25:24.521962 (+   381us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:25:24.549311 (+ 27349us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:24.549353 (+    42us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:25:24.550032 (+   679us) log.cc:844] Serialized 52 byte log entry
0709 12:25:25.007012 (+456980us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:25:25.169386 (+162374us) log.cc:844] Serialized 14 byte log entry
0709 12:25:25.169611 (+   225us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:25.169705 (+    94us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:25.746570 (+576865us) raft_consensus.cc:1685] finished
0709 12:25:25.746590 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:25.746725 (+   135us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":33926,"prepare.queue_time_us":49109,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":41,"raft.queue_time_us":350384,"raft.run_cpu_time_us":273,"raft.run_wall_time_us":31172,"thread_start_us":581716,"threads_started":2,"wal-append.queue_time_us":694171}
W20260709 12:25:25.774199 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 71216.5us
I20260709 12:25:25.771210 17759 rpcz_store.cc:276] 0709 12:25:24.082581 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:24.082773 (+   192us) service_pool.cc:224] Handling call
0709 12:25:25.542288 (+1459515us) spinlock_profiling.cc:243] Waited 356 ms on lock 0x7b54000fe1d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:25:25.542476 (+   188us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":818272512}
I20260709 12:25:25.725472 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 31 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:25.804216 17729 leader_election.cc:304] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:25.805629 18374 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59902 (request call id 2424) took 1794 ms. Trace:
I20260709 12:25:25.805841 28311 raft_consensus.cc:515] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 36 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:25.806020 18374 rpcz_store.cc:276] 0709 12:25:24.011571 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:24.217047 (+205476us) service_pool.cc:224] Handling call
0709 12:25:25.805435 (+1588388us) spinlock_profiling.cc:243] Waited 831 ms on lock 0x7b54002a30d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:25:25.805567 (+   132us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1911197440}
I20260709 12:25:25.807103 28184 raft_consensus.cc:493] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:25.808347 18333 leader_election.cc:304] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:25.809340 18362 raft_consensus.cc:1240] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 29. Current term is 30. Ops: []
I20260709 12:25:25.776001 17756 raft_consensus.cc:2468] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 26.
I20260709 12:25:25.776664 17762 raft_consensus.cc:1275] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 25 index: 18. Preceding OpId from leader: term: 27 index: 19. (index mismatch)
I20260709 12:25:25.817847 28244 raft_consensus.cc:493] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:25.777382 17757 raft_consensus.cc:1240] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 33 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 32. Current term is 33. Ops: []
I20260709 12:25:25.818372 28244 raft_consensus.cc:515] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:25.770298 28305 consensus_meta.cc:287] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.680s	user 0.014s	sys 0.000s
I20260709 12:25:25.778283 28330 raft_consensus.cc:2804] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:25:25.824117 28330 raft_consensus.cc:493] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:25.824537 28330 raft_consensus.cc:3060] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:25.787002 28274 consensus_queue.cc:1059] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 21.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20260709 12:25:25.727269 28120 consensus_queue.cc:237] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 31.20, Last appended by leader: 20, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:25.766423 28126 raft_consensus.cc:515] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:25.788116 17683 raft_consensus.cc:1240] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 28. Current term is 29. Ops: [28.17-28.17]
I20260709 12:25:25.774574 28011 raft_consensus.cc:493] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:25.768390 28269 raft_consensus.cc:493] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:25:25.790622 17758 raft_consensus.cc:2410] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for term 28 because replica has last-logged OpId of term: 27 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 18.
I20260709 12:25:25.781454 28228 leader_election.cc:290] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 30 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:25.832052 17829 raft_consensus.cc:2468] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 21.
I20260709 12:25:25.807641 28184 raft_consensus.cc:515] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:25.830595 28326 raft_consensus.cc:2804] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 33 FOLLOWER]: Leader election won for term 33
I20260709 12:25:25.828797 28183 raft_consensus.cc:2749] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
W20260709 12:25:25.843693 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 127293us
I20260709 12:25:25.830475 28311 leader_election.cc:290] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 36 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:25.853266 17680 raft_consensus.cc:1275] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 30 index: 19. Preceding OpId from leader: term: 31 index: 19. (term mismatch)
I20260709 12:25:25.853854 17680 pending_rounds.cc:85] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3: Aborting all ops after (but not including) 18
I20260709 12:25:25.854137 17680 pending_rounds.cc:107] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3: Aborting uncommitted NO_OP operation due to leader change: 30.19
I20260709 12:25:25.854203 28126 leader_election.cc:290] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 30 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:25.854521 17680 raft_consensus.cc:2889] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:25.862231 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:25.863646 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:25.865550 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 28 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:25.866387 17837 raft_consensus.cc:2468] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 27.
I20260709 12:25:25.868112 17837 raft_consensus.cc:1275] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 27 index: 18. Preceding OpId from leader: term: 29 index: 19. (index mismatch)
I20260709 12:25:25.868351 17831 raft_consensus.cc:2393] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 23: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:25:25.872072 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 35 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:25.873127 17831 raft_consensus.cc:2410] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for term 35 because replica has last-logged OpId of term: 31 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 30 index: 17.
I20260709 12:25:25.878353 17761 raft_consensus.cc:2468] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 31.
I20260709 12:25:25.879990 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:25.835553 28287 consensus_queue.cc:1048] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.029s
I20260709 12:25:25.898310 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:25.837538 28011 raft_consensus.cc:515] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:25.900883 18368 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59902 (request call id 2428) took 1622 ms. Trace:
I20260709 12:25:25.901244 28011 leader_election.cc:290] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:25.902937 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 32 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:25.901400 18368 rpcz_store.cc:276] 0709 12:25:24.279688 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:24.557451 (+277763us) service_pool.cc:224] Handling call
0709 12:25:24.557785 (+   334us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:24.645796 (+ 88011us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:24.645838 (+    42us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:25.686747 (+1040909us) spinlock_profiling.cc:243] Waited 1.2 ms on lock 0x7b2c000123a8. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b48f19a11 00007f7b48f29372 00007f7b4a65ba01 00007f7b4a62043a 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2
0709 12:25:25.687158 (+   411us) log.cc:844] Serialized 26 byte log entry
0709 12:25:25.808168 (+121010us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:25:25.808207 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:25.808298 (+    91us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:25.900436 (+ 92138us) raft_consensus.cc:1685] finished
0709 12:25:25.900460 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:25.900816 (+   356us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":87594,"raft.queue_time_us":87718,"raft.run_cpu_time_us":107,"raft.run_wall_time_us":6020,"spinlock_wait_cycles":3603072,"thread_start_us":120704,"threads_started":1,"wal-append.queue_time_us":149233}
I20260709 12:25:25.880831 17837 raft_consensus.cc:2468] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 30.
W20260709 12:25:25.904767 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:25.882030 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 33 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:25.856247 28207 raft_consensus.cc:493] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 34 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:25:25.836899 28269 raft_consensus.cc:515] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:25.841725 28177 consensus_queue.cc:260] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 28.16, Last appended by leader: 16, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:25.870893 28159 consensus_queue.cc:1059] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true }, Status: INVALID_TERM, Last received: 27.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.018s
I20260709 12:25:25.893625 17831 raft_consensus.cc:1240] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 22. Current term is 23. Ops: [22.16-22.16]
I20260709 12:25:25.894357 28126 raft_consensus.cc:493] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:25.895583 17759 raft_consensus.cc:1240] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 28. Current term is 30. Ops: []
I20260709 12:25:25.811924 28271 consensus_queue.cc:260] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 27.19, Last appended by leader: 19, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:25.901122 17801 leader_election.cc:304] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:25.906621 17834 raft_consensus.cc:2376] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 33 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 33. Re-sending same reply.
I20260709 12:25:25.841333 28274 consensus_queue.cc:1059] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 0.0, Next index: 21, Last known committed idx: 19, Time since last communication: 0.027s
I20260709 12:25:25.904431 17652 leader_election.cc:304] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:25:25.912650 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:25.926231 28305 raft_consensus.cc:2749] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:25:25.926856 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 35 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:25.912425 28207 raft_consensus.cc:515] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:25.913266 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:25.930410 17837 raft_consensus.cc:3060] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:25.866362 28326 raft_consensus.cc:697] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 33 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:25:25.915462 17727 leader_election.cc:304] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:25.917946 17831 raft_consensus.cc:1240] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 25. Current term is 26. Ops: []
I20260709 12:25:25.919952 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 31 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:25.920928 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 24 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:25.937248 17759 raft_consensus.cc:2410] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 24 because replica has last-logged OpId of term: 22 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 15.
I20260709 12:25:25.923668 17763 raft_consensus.cc:1275] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 27 index: 17. Preceding OpId from leader: term: 31 index: 19. (index mismatch)
I20260709 12:25:25.938721 17652 leader_election.cc:304] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 35 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:25.933382 18363 raft_consensus.cc:1240] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 23 LEADER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 22. Current term is 23. Ops: []
I20260709 12:25:25.943423 17727 leader_election.cc:304] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:25.915894 28244 leader_election.cc:290] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:25.949177 17837 raft_consensus.cc:2468] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 30.
I20260709 12:25:25.951159 17652 leader_election.cc:304] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:25.951977 17727 leader_election.cc:304] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:25.915174 28153 consensus_queue.cc:1059] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 28.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.035s
I20260709 12:25:25.911152 28228 consensus_queue.cc:1059] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 27.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.034s
I20260709 12:25:25.958627 28119 raft_consensus.cc:2749] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:25:25.961675 28348 raft_consensus.cc:2804] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:25:25.940563 27976 raft_consensus.cc:2804] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:25:25.940986 28274 consensus_queue.cc:1243] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer e2909cff1da44ad387f96a8ae3aa5afb log is divergent from this leader: its last log entry 30.20 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 19
I20260709 12:25:25.971670 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:25.923228 28126 raft_consensus.cc:515] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:25.950052 28271 raft_consensus.cc:3060] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:25.971975 28348 raft_consensus.cc:493] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:25:25.958770 28311 raft_consensus.cc:3060] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:25.957921 28320 consensus_queue.cc:1048] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.064s
I20260709 12:25:26.008152 28326 consensus_queue.cc:237] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 31.21, Last appended by leader: 21, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:25.948046 28011 raft_consensus.cc:2804] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:25:26.013718 17755 raft_consensus.cc:2393] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 30: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:25:25.960167 28240 raft_consensus.cc:2749] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 35 FOLLOWER]: Leader election lost for term 35. Reason: could not achieve majority
I20260709 12:25:25.969950 28229 consensus_queue.cc:1059] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 22.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.038s
I20260709 12:25:25.963387 28336 consensus_queue.cc:1048] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.045s
I20260709 12:25:26.005831 28207 leader_election.cc:290] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:26.013252 27976 raft_consensus.cc:493] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:25:26.038143 28045 raft_consensus.cc:493] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:25:26.035676 28330 raft_consensus.cc:515] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:26.051055 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 31 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:25.969065 28188 consensus_queue.cc:1059] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 20.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.053s
I20260709 12:25:26.020813 28348 raft_consensus.cc:3060] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:26.025696 28011 raft_consensus.cc:493] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:26.055605 27976 raft_consensus.cc:3060] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:26.071527 17760 raft_consensus.cc:2393] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 31: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:25:26.085474 17837 raft_consensus.cc:1275] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 25 index: 18. Preceding OpId from leader: term: 26 index: 19. (index mismatch)
I20260709 12:25:26.088411 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:26.056609 28045 raft_consensus.cc:515] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:26.092381 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:25:26.074261 28358 log.cc:927] Time spent T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.051s	user 0.000s	sys 0.000s
I20260709 12:25:26.070681 27431 leader_election.cc:290] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:26.094094 17837 raft_consensus.cc:2393] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 27: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:25:26.073751 28011 raft_consensus.cc:3060] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:26.094645 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 29 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:26.096413 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 30 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:26.098237 17831 raft_consensus.cc:3060] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:26.073194 28126 leader_election.cc:290] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:26.101086 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 32 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:26.102330 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 28 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:26.106019 28293 raft_consensus.cc:3055] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:25:26.106564 28293 raft_consensus.cc:740] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:25:26.108497 28148 raft_consensus.cc:3055] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:25:26.108996 28148 raft_consensus.cc:740] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
W20260709 12:25:26.110677 28359 log.cc:927] Time spent T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.081s	user 0.000s	sys 0.000s
W20260709 12:25:26.112215 28361 log.cc:927] Time spent T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.051s	user 0.001s	sys 0.000s
I20260709 12:25:26.116572 28330 leader_election.cc:290] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 31 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:26.118813 28045 leader_election.cc:290] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:26.095237 28269 leader_election.cc:290] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:26.120146 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 35 candidate_status { last_received { term: 34 index: 23 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:26.124444 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:26.125101 17836 raft_consensus.cc:3060] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:26.092782 28256 raft_consensus.cc:515] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:26.126765 28274 consensus_queue.cc:1048] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20260709 12:25:26.128322 28184 leader_election.cc:290] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:26.128975 28256 leader_election.cc:290] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 26 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:26.129707 16501 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:25:26.130774 16501 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:25:26.131652 17688 raft_consensus.cc:2393] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 26: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:25:26.160737 17831 raft_consensus.cc:2468] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 30.
I20260709 12:25:26.162506 18333 leader_election.cc:304] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:25:26.164728 28274 raft_consensus.cc:2804] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:25:26.165290 28274 raft_consensus.cc:697] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 30 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:25:26.166235 28274 consensus_queue.cc:237] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 27.18, Last appended by leader: 18, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:26.170290 28112 raft_consensus.cc:493] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:25:26.170853 28112 raft_consensus.cc:515] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:26.172729 28311 raft_consensus.cc:2749] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
W20260709 12:25:26.180812 17729 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:26.181196 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:26.181900 17764 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 29.
I20260709 12:25:26.182894 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:26.184898 28011 raft_consensus.cc:515] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:26.185287 28260 raft_consensus.cc:3055] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:25:26.185779 28260 raft_consensus.cc:740] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:25:26.188297 28011 leader_election.cc:290] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 28 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:26.107502 28293 consensus_queue.cc:260] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 26.19, Last appended by leader: 19, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:26.188966 27766 leader_election.cc:290] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 25 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:26.189209 28293 raft_consensus.cc:3060] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:26.194691 28319 raft_consensus.cc:3055] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:25:26.195190 28319 raft_consensus.cc:740] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:25:26.196161 28319 consensus_queue.cc:260] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 22.16, Last appended by leader: 16, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:26.198383 28112 leader_election.cc:290] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:26.161511 28348 raft_consensus.cc:515] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:26.201773 28348 leader_election.cc:290] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 31 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:26.202291 17680 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37402 (request call id 4074) took 2067 ms. Trace:
I20260709 12:25:26.202723 17680 rpcz_store.cc:276] 0709 12:25:24.134458 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:24.136121 (+  1663us) service_pool.cc:224] Handling call
0709 12:25:24.149162 (+ 13041us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:25.855166 (+1706004us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:26.202051 (+346885us) spinlock_profiling.cc:243] Waited 1.7 s on lock 0x7b5400054bd4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:25:26.202232 (+   181us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3918012288}
I20260709 12:25:26.206334 28177 raft_consensus.cc:3060] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:26.209331 28287 raft_consensus.cc:3055] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 32 LEADER]: Stepping down as leader of term 32
I20260709 12:25:26.209780 28287 raft_consensus.cc:740] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 32 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:25:26.210685 28287 consensus_queue.cc:260] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 32.21, Last appended by leader: 21, Current term: 32, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:26.212721 28159 raft_consensus.cc:3055] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:25:26.213155 28159 raft_consensus.cc:740] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:25:26.213963 28159 consensus_queue.cc:260] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 27.21, Last appended by leader: 21, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:26.215402 28159 raft_consensus.cc:3060] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:26.216650 17830 raft_consensus.cc:1275] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 23 index: 17. Preceding OpId from leader: term: 27 index: 19. (index mismatch)
I20260709 12:25:26.197502 28319 raft_consensus.cc:3060] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:26.224458 18333 leader_election.cc:304] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:26.226361 28105 raft_consensus.cc:493] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:26.226578 28276 raft_consensus.cc:2804] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:25:26.227018 28276 raft_consensus.cc:493] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:25:26.226934 28105 raft_consensus.cc:515] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:26.227505 28276 raft_consensus.cc:3060] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:26.228175 17729 leader_election.cc:304] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:26.229362 28105 leader_election.cc:290] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:25:26.230362 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 31 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:26.231047 18368 raft_consensus.cc:2468] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 30.
I20260709 12:25:26.231242 28011 raft_consensus.cc:2804] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:25:26.231698 28011 raft_consensus.cc:493] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:25:26.232141 28011 raft_consensus.cc:3060] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:26.233244 28328 consensus_queue.cc:1059] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 25.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20260709 12:25:26.234011 28328 consensus_queue.cc:1243] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer 334a4e1f59944544b3b3ff36616b57da log is divergent from this leader: its last log entry 26.19 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 18
I20260709 12:25:26.234333 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:26.234997 18368 raft_consensus.cc:2468] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 31.
I20260709 12:25:26.235152 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 28 candidate_status { last_received { term: 27 index: 21 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:26.235723 18361 raft_consensus.cc:3060] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:26.199574 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 30 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:26.237779 17836 raft_consensus.cc:2468] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 27.
I20260709 12:25:26.239670 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 29 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:26.241730 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 24 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:26.244415 18361 raft_consensus.cc:2468] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 28.
I20260709 12:25:26.242605 18333 leader_election.cc:304] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:25:26.247287 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 29 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:26.248746 18365 raft_consensus.cc:2468] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 28.
I20260709 12:25:26.250020 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 29 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:26.252597 18333 leader_election.cc:304] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:26.254935 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 31 candidate_status { last_received { term: 30 index: 21 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:26.255489 18361 raft_consensus.cc:3060] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:26.255640 17681 raft_consensus.cc:1240] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 33 LEADER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 32. Current term is 33. Ops: []
I20260709 12:25:26.243078 28133 raft_consensus.cc:493] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:26.253463 18371 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59902 (request call id 2420) took 2244 ms (client timeout 1605 ms). Trace:
W20260709 12:25:26.256349 18371 rpcz_store.cc:269] 0709 12:25:24.009226 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:24.009415 (+   189us) service_pool.cc:224] Handling call
0709 12:25:26.253293 (+2243878us) spinlock_profiling.cc:243] Waited 397 ms on lock 0x7b54006af9d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:25:26.253400 (+   107us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":912750848}
I20260709 12:25:26.258380 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 28 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:26.259238 18367 raft_consensus.cc:1240] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 26. Current term is 27. Ops: [26.18-26.18]
I20260709 12:25:26.253602 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:26.260161 18365 raft_consensus.cc:2393] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 29: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:25:26.234460 28328 consensus_queue.cc:1048] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.009s
I20260709 12:25:26.261454 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:26.262395 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:26.263398 18371 raft_consensus.cc:2468] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 29.
I20260709 12:25:26.266247 18361 raft_consensus.cc:2468] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 31.
I20260709 12:25:26.267280 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:26.267689 18333 leader_election.cc:304] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:26.267908 18361 raft_consensus.cc:1275] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 28 index: 19. Preceding OpId from leader: term: 29 index: 20. (index mismatch)
I20260709 12:25:26.269078 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:26.269416 18361 raft_consensus.cc:1240] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 27. Current term is 28. Ops: []
I20260709 12:25:26.245126 28287 raft_consensus.cc:3060] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:25:26.272868 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 36 candidate_status { last_received { term: 35 index: 26 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:26.273474 18368 raft_consensus.cc:3060] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:25:26.254601 28112 raft_consensus.cc:2804] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:25:26.282617 18368 raft_consensus.cc:2468] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 36.
I20260709 12:25:26.258608 28105 raft_consensus.cc:2749] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:25:26.285705 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 29 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:26.287843 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 33 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:26.244181 28269 consensus_queue.cc:1048] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.035s
I20260709 12:25:26.286902 18361 raft_consensus.cc:1240] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 36 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 35. Current term is 36. Ops: [35.26-35.26]
I20260709 12:25:26.265144 28133 raft_consensus.cc:515] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:26.272028 28314 raft_consensus.cc:2749] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260709 12:25:26.286500 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:26.299856 18368 raft_consensus.cc:2393] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 27: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:25:26.298024 17729 leader_election.cc:304] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 36 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:26.186739 28260 consensus_queue.cc:260] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 27.14, Last appended by leader: 14, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:26.301391 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 28 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:26.302013 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 34 candidate_status { last_received { term: 32 index: 23 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:26.306458 18371 raft_consensus.cc:2468] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 33.
I20260709 12:25:26.307802 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 31 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:26.308547 18371 raft_consensus.cc:3060] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:26.310602 17729 leader_election.cc:304] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:26.312925 18368 raft_consensus.cc:1240] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 27. Current term is 28. Ops: [27.19-27.19]
I20260709 12:25:26.315244 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 35 candidate_status { last_received { term: 34 index: 23 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:26.270581 27976 raft_consensus.cc:515] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:26.318650 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 30 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:26.293749 28112 raft_consensus.cc:697] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 27 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:25:26.321771 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 30 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:26.321970 28112 consensus_queue.cc:237] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 26.18, Last appended by leader: 18, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:26.323050 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 30 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:26.320184 18367 raft_consensus.cc:2468] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 29.
I20260709 12:25:26.311502 28348 raft_consensus.cc:2804] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 36 FOLLOWER]: Leader election won for term 36
I20260709 12:25:26.328719 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 32 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:25:26.330070 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:26.332512 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 31 candidate_status { last_received { term: 30 index: 21 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:26.337589 28133 leader_election.cc:290] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:25:26.343238 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:26.344090 28348 raft_consensus.cc:697] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 36 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:25:26.345017 28348 consensus_queue.cc:237] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 35.26, Last appended by leader: 26, Current term: 36, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:26.316089 28326 raft_consensus.cc:2804] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:25:26.348114 28326 raft_consensus.cc:493] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 33 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:26.348460 28326 raft_consensus.cc:3060] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:25:26.357098 17729 leader_election.cc:304] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:26.361908 18371 raft_consensus.cc:2468] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 31.
I20260709 12:25:26.362020 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 31 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:26.365021 18366 raft_consensus.cc:1240] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 26 LEADER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 25. Current term is 26. Ops: []
I20260709 12:25:26.357496 28260 raft_consensus.cc:3060] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:26.371726 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 31 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:26.372548 17764 raft_consensus.cc:2410] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for term 31 because replica has last-logged OpId of term: 30 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 28 index: 18.
I20260709 12:25:26.378188 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 28 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:26.379029 17755 raft_consensus.cc:2410] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for term 28 because replica has last-logged OpId of term: 26 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 17.
I20260709 12:25:26.360041 28045 raft_consensus.cc:2804] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Leader pre-election won for term 30
W20260709 12:25:26.384258 17727 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:26.379789 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 28 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:26.363152 28148 consensus_queue.cc:260] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 22.16, Last appended by leader: 16, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:26.394222 28148 raft_consensus.cc:3060] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 22 FOLLOWER]: Advancing to term 23
W20260709 12:25:26.396396 28367 log.cc:927] Time spent T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.070s	user 0.000s	sys 0.000s
I20260709 12:25:26.385905 28045 raft_consensus.cc:493] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:26.416695 17727 leader_election.cc:304] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:26.414366 17829 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35040 (request call id 3848) took 1117 ms. Trace:
I20260709 12:25:26.420096 17829 rpcz_store.cc:276] 0709 12:25:25.296905 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:25.297050 (+   145us) service_pool.cc:224] Handling call
0709 12:25:26.414204 (+1117154us) spinlock_profiling.cc:243] Waited 522 ms on lock 0x7b54000898d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:25:26.414308 (+   104us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1200680320}
I20260709 12:25:26.421552 18367 raft_consensus.cc:1240] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 27 LEADER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 26. Current term is 27. Ops: []
I20260709 12:25:26.394255 27976 leader_election.cc:290] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 26 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:26.424041 17755 raft_consensus.cc:1240] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 22. Current term is 23. Ops: []
I20260709 12:25:26.428642 17763 raft_consensus.cc:1240] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 27. Current term is 28. Ops: [27.19-27.19]
I20260709 12:25:26.427205 17727 leader_election.cc:304] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:26.435411 17689 raft_consensus.cc:1275] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 30 index: 20. Preceding OpId from leader: term: 31 index: 21. (index mismatch)
I20260709 12:25:26.437323 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 28 candidate_status { last_received { term: 27 index: 21 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
W20260709 12:25:26.439059 28366 log.cc:927] Time spent T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.065s	user 0.000s	sys 0.000s
I20260709 12:25:26.442847 28011 raft_consensus.cc:515] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:26.447016 28276 raft_consensus.cc:515] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:26.448190 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:26.448876 17680 raft_consensus.cc:2468] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 29.
I20260709 12:25:26.431094 28269 raft_consensus.cc:2804] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 33 FOLLOWER]: Leader election won for term 33
I20260709 12:25:26.426216 28245 raft_consensus.cc:3055] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 31 LEADER]: Stepping down as leader of term 31
I20260709 12:25:26.451061 28245 raft_consensus.cc:740] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 31 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:25:26.452101 28245 consensus_queue.cc:260] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 31.18, Last appended by leader: 18, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:26.453655 28245 raft_consensus.cc:3060] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Advancing to term 32
W20260709 12:25:26.453912 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:26.454267 28269 raft_consensus.cc:697] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 33 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:25:26.454828 18363 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59886 (request call id 2793) took 1367 ms. Trace:
I20260709 12:25:26.455140 28269 consensus_queue.cc:237] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 32.19, Last appended by leader: 19, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:26.459911 28330 consensus_queue.cc:1059] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 25.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.029s
I20260709 12:25:26.461374 17685 raft_consensus.cc:2410] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for term 28 because replica has last-logged OpId of term: 27 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 26 index: 18.
I20260709 12:25:26.455214 18363 rpcz_store.cc:276] 0709 12:25:25.087447 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:25.678772 (+591325us) service_pool.cc:224] Handling call
0709 12:25:25.679003 (+   231us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:25.948356 (+269353us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:26.454602 (+506246us) spinlock_profiling.cc:243] Waited 254 ms on lock 0x7b5400031bd4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:25:26.454773 (+   171us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":584433024}
I20260709 12:25:26.462574 17681 raft_consensus.cc:1240] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 35 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 34. Current term is 35. Ops: []
I20260709 12:25:26.421973 28045 raft_consensus.cc:3060] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:26.433961 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 31 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:25:26.480995 17654 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:26.477980 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 30 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:26.482303 18363 raft_consensus.cc:3060] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Advancing to term 30
W20260709 12:25:26.478883 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:26.478036 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 26 candidate_status { last_received { term: 25 index: 22 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:26.423008 28328 raft_consensus.cc:493] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:25:26.490388 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 34 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
W20260709 12:25:26.494477 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:26.495360 18371 raft_consensus.cc:1240] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 29. Current term is 30. Ops: []
I20260709 12:25:26.484772 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 35 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:26.496789 17681 raft_consensus.cc:2468] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 34.
I20260709 12:25:26.418686 28314 consensus_queue.cc:1048] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.065s
I20260709 12:25:26.468494 28348 raft_consensus.cc:2804] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:25:26.500618 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 29 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:26.489041 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:26.502889 17761 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36572 (request call id 1849) took 1122 ms. Trace:
I20260709 12:25:26.503263 17761 rpcz_store.cc:276] 0709 12:25:25.380544 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:25.381412 (+   868us) service_pool.cc:224] Handling call
0709 12:25:26.502738 (+1121326us) spinlock_profiling.cc:243] Waited 457 ms on lock 0x7b54004155d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:25:26.502834 (+    96us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1052772224}
I20260709 12:25:26.505213 18363 raft_consensus.cc:2468] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 30.
W20260709 12:25:26.506507 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 93389.9us
I20260709 12:25:26.507071 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:26.507786 18363 raft_consensus.cc:2468] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 26.
I20260709 12:25:26.509070 18363 raft_consensus.cc:1240] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 27. Current term is 28. Ops: [25.17-27.18]
I20260709 12:25:26.510315 18363 raft_consensus.cc:1217] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Deduplicated request from leader. Original: 24.17->[25.18-29.20]   Dedup: 25.18->[27.19-29.20]
I20260709 12:25:26.510839 18363 pending_rounds.cc:85] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da: Aborting all ops after (but not including) 18
I20260709 12:25:26.511142 18363 pending_rounds.cc:107] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da: Aborting uncommitted NO_OP operation due to leader change: 26.19
I20260709 12:25:26.511590 18363 raft_consensus.cc:2889] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:25:26.513139 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 56332.8us
I20260709 12:25:26.515549 17654 leader_election.cc:304] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b
W20260709 12:25:26.522876 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:26.518698 17654 leader_election.cc:304] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
W20260709 12:25:26.524449 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:26.527433 17829 raft_consensus.cc:1275] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 27 index: 21. Preceding OpId from leader: term: 28 index: 22. (index mismatch)
I20260709 12:25:26.535637 17654 leader_election.cc:304] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:26.513506 28348 raft_consensus.cc:697] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 30 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:25:26.522764 28159 raft_consensus.cc:2804] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:25:26.543547 28330 raft_consensus.cc:493] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:25:26.544201 28330 raft_consensus.cc:515] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:26.546537 28330 leader_election.cc:290] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:26.539423 28293 raft_consensus.cc:2749] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260709 12:25:26.551909 17829 raft_consensus.cc:1275] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 30 index: 21. Preceding OpId from leader: term: 31 index: 22. (index mismatch)
I20260709 12:25:26.554713 28314 consensus_queue.cc:1048] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.000s
I20260709 12:25:26.569557 28276 leader_election.cc:290] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 30 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:26.572009 28011 leader_election.cc:290] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 26 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:26.498925 28328 raft_consensus.cc:515] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:26.577543 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:26.578192 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:26.579909 17831 raft_consensus.cc:1275] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 22 index: 16. Preceding OpId from leader: term: 23 index: 17. (index mismatch)
I20260709 12:25:26.578334 17761 raft_consensus.cc:3060] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:26.538915 28282 raft_consensus.cc:2804] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:25:26.596987 17761 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 30.
W20260709 12:25:26.597405 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:26.542088 28159 raft_consensus.cc:493] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:25:26.621299 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:26.629424 18334 connection.cc:256] RPC call timeout handler was delayed by -2.4368363720000161s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1694s for a response
W20260709 12:25:26.629777 18334 connection.cc:256] RPC call timeout handler was delayed by -2.4273630129996491s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19790000000000002s for a response
W20260709 12:25:26.630015 18334 connection.cc:256] RPC call timeout handler was delayed by -2.4183630129996345s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19890000000000002s for a response
W20260709 12:25:26.630544 18334 connection.cc:256] RPC call timeout handler was delayed by -2.09933637200038s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2069s for a response
W20260709 12:25:26.630798 18334 connection.cc:256] RPC call timeout handler was delayed by -2.0741363720003392s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2097s for a response
I20260709 12:25:26.631716 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 29 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:26.632287 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 29 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:26.633092 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:25:26.633232 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 32 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:26.633930 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 32 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:26.634902 17684 raft_consensus.cc:1240] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 25. Current term is 26. Ops: []
I20260709 12:25:26.635809 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:25:26.637462 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:26.638224 17687 raft_consensus.cc:2468] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 27.
I20260709 12:25:26.639398 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 28 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:26.640971 17688 raft_consensus.cc:1275] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 27 index: 16. Preceding OpId from leader: term: 29 index: 17. (index mismatch)
I20260709 12:25:26.641317 18362 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59886 (request call id 2782) took 1635 ms. Trace:
I20260709 12:25:26.642001 17758 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36572 (request call id 1852) took 1257 ms. Trace:
I20260709 12:25:26.642530 17758 rpcz_store.cc:276] 0709 12:25:25.384293 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:25.384437 (+   144us) service_pool.cc:224] Handling call
0709 12:25:26.641825 (+1257388us) spinlock_profiling.cc:243] Waited 400 ms on lock 0x7b540042c054. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:25:26.641946 (+   121us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":919933312}
I20260709 12:25:26.643280 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 29 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:26.643940 18367 raft_consensus.cc:2468] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 28.
I20260709 12:25:26.644675 17685 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44898 (request call id 2063) took 1031 ms. Trace:
I20260709 12:25:26.645016 18367 raft_consensus.cc:1240] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 27. Current term is 28. Ops: []
I20260709 12:25:26.645061 17685 rpcz_store.cc:276] 0709 12:25:25.613150 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:25.685725 (+ 72575us) service_pool.cc:224] Handling call
0709 12:25:26.644428 (+958703us) spinlock_profiling.cc:243] Waited 768 ms on lock 0x7b5400347bd4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:25:26.644600 (+   172us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1767496064}
I20260709 12:25:26.646409 18367 raft_consensus.cc:1240] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 26. Current term is 27. Ops: []
I20260709 12:25:26.649106 27800 leader_election.cc:290] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:26.650630 17687 raft_consensus.cc:2410] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for term 28 because replica has last-logged OpId of term: 26 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 17.
I20260709 12:25:26.654394 18333 leader_election.cc:304] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:26.536916 28274 consensus_queue.cc:1048] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.078s
I20260709 12:25:26.656256 18366 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59886 (request call id 2785) took 1644 ms. Trace:
I20260709 12:25:26.656642 18366 rpcz_store.cc:276] 0709 12:25:25.012323 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:25.554372 (+542049us) service_pool.cc:224] Handling call
0709 12:25:25.554771 (+   399us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:26.365699 (+810928us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:26.656034 (+290335us) spinlock_profiling.cc:243] Waited 810 ms on lock 0x7b5400570154. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:25:26.656196 (+   162us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1862827264}
I20260709 12:25:26.658133 18366 raft_consensus.cc:1275] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 33 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 32 index: 21. Preceding OpId from leader: term: 33 index: 21. (term mismatch)
I20260709 12:25:26.658435 18366 pending_rounds.cc:85] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da: Aborting all ops after (but not including) 20
I20260709 12:25:26.658641 18366 pending_rounds.cc:107] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da: Aborting uncommitted NO_OP operation due to leader change: 32.21
I20260709 12:25:26.658856 18366 raft_consensus.cc:2889] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 33 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:26.663157 18371 raft_consensus.cc:1240] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 29. Current term is 30. Ops: [29.22-29.22]
I20260709 12:25:26.664215 28348 consensus_queue.cc:237] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 28.16, Last appended by leader: 16, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:26.669006 28328 leader_election.cc:290] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:26.669301 18364 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59886 (request call id 2780) took 1664 ms. Trace:
I20260709 12:25:26.669648 18364 rpcz_store.cc:276] 0709 12:25:25.005232 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:25.086088 (+ 80856us) service_pool.cc:224] Handling call
0709 12:25:25.086377 (+   289us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:25.767732 (+681355us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:26.669093 (+901361us) spinlock_profiling.cc:243] Waited 380 ms on lock 0x7b540051aed4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:25:26.669247 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":873412992}
I20260709 12:25:26.671696 17683 raft_consensus.cc:2393] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 32: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:25:26.633903 17689 raft_consensus.cc:2468] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 31.
I20260709 12:25:26.641709 18362 rpcz_store.cc:276] 0709 12:25:25.005798 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:25.390236 (+384438us) service_pool.cc:224] Handling call
0709 12:25:25.809111 (+418875us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:25.809824 (+   713us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:26.641124 (+831300us) spinlock_profiling.cc:243] Waited 404 ms on lock 0x7b54002a30d0. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a61f8ad 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:25:26.641258 (+   134us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":928436608}
I20260709 12:25:26.663259 28045 raft_consensus.cc:515] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:26.678573 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 29 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:26.680004 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:26.680238 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 35 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:26.681350 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 32 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:26.682124 17684 raft_consensus.cc:2410] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for term 32 because replica has last-logged OpId of term: 28 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 27 index: 17.
I20260709 12:25:26.682497 17680 raft_consensus.cc:1240] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 33 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 32. Current term is 33. Ops: []
I20260709 12:25:26.683094 18333 leader_election.cc:304] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:26.683459 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 36 candidate_status { last_received { term: 35 index: 26 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:25:26.614383 28312 raft_consensus.cc:3055] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:25:26.684180 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:26.686758 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 32 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:26.687417 18365 raft_consensus.cc:2468] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 31.
W20260709 12:25:26.701990 28380 log.cc:927] Time spent T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.164s	user 0.003s	sys 0.000s
I20260709 12:25:26.702706 28326 raft_consensus.cc:515] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 34 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:26.706362 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:25:26.706995 17685 raft_consensus.cc:3060] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:26.628114 28159 raft_consensus.cc:3060] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:26.626163 28282 raft_consensus.cc:697] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 31 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:25:26.711133 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:26.712700 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:26.728477 17688 raft_consensus.cc:2468] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 27.
I20260709 12:25:26.717249 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 31 candidate_status { last_received { term: 30 index: 21 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:26.730037 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 32 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:26.730641 17758 raft_consensus.cc:2468] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 31.
I20260709 12:25:26.731506 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:26.736299 28287 raft_consensus.cc:2749] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:25:26.738245 17758 raft_consensus.cc:1275] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 34 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 30 index: 17. Preceding OpId from leader: term: 34 index: 19. (index mismatch)
I20260709 12:25:26.739037 28045 leader_election.cc:290] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 30 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:26.730647 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:26.741370 17680 raft_consensus.cc:2393] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 21: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:25:26.721232 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 33 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:26.721768 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 30 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:26.743283 18362 raft_consensus.cc:2468] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 26.
I20260709 12:25:26.722373 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 35 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:26.732692 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 30 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:26.734918 28148 raft_consensus.cc:493] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:26.614616 28314 consensus_queue.cc:1048] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.089s
I20260709 12:25:26.735780 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:25:26.750186 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:26.740867 18365 raft_consensus.cc:2393] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 28: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:25:26.720461 18333 leader_election.cc:304] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:26.753185 18371 raft_consensus.cc:1240] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 26 LEADER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 25. Current term is 26. Ops: []
I20260709 12:25:26.754546 28282 consensus_queue.cc:237] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 30.21, Last appended by leader: 21, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:26.755167 17763 raft_consensus.cc:1240] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 36 LEADER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 35. Current term is 36. Ops: []
I20260709 12:25:26.756381 28112 raft_consensus.cc:2749] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:25:26.757129 28112 consensus_queue.cc:1059] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 25.18, Next index: 19, Last known committed idx: 17, Time since last communication: 0.000s
I20260709 12:25:26.744153 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 34 candidate_status { last_received { term: 32 index: 23 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:26.758968 18362 raft_consensus.cc:3060] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 33 FOLLOWER]: Advancing to term 34
W20260709 12:25:26.759074 17728 connection.cc:256] RPC call timeout handler was delayed by -2.0771356340001148s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1769s for a response
W20260709 12:25:26.759523 17728 connection.cc:256] RPC call timeout handler was delayed by -2.0672356340000988s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17800000000000002s for a response
I20260709 12:25:26.751632 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:25:26.744350 17686 raft_consensus.cc:2468] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 29.
W20260709 12:25:26.746944 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:26.685683 28105 raft_consensus.cc:2749] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260709 12:25:26.768009 17688 raft_consensus.cc:2393] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 26: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:25:26.686017 28312 raft_consensus.cc:740] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:25:26.763648 17755 raft_consensus.cc:3055] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:25:26.770995 17755 raft_consensus.cc:740] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:25:26.764827 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 30 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:26.708387 28326 leader_election.cc:290] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 34 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:25:26.768879 17652 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:26.772689 18371 raft_consensus.cc:3060] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Advancing to term 30
W20260709 12:25:26.772994 17652 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:26.769881 17728 connection.cc:256] RPC call timeout handler was delayed by -1.9844356339999649s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18720000000000003s for a response
W20260709 12:25:26.774765 17728 connection.cc:256] RPC call timeout handler was delayed by -1.7954356340001141s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2082s for a response
W20260709 12:25:26.775084 17728 connection.cc:256] RPC call timeout handler was delayed by -1.6955356339999526s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21930000000000002s for a response
W20260709 12:25:26.775454 17728 connection.cc:256] RPC call timeout handler was delayed by -1.4977748679998513s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1673s for a response
W20260709 12:25:26.775820 17728 connection.cc:256] RPC call timeout handler was delayed by -1.3333227289995193s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2318s for a response
W20260709 12:25:26.776108 17728 connection.cc:256] RPC call timeout handler was delayed by -1.1393063450000227s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2117s for a response
W20260709 12:25:26.776368 17728 connection.cc:256] RPC call timeout handler was delayed by -1.1101894139997057s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16570000000000001s for a response
I20260709 12:25:26.791052 18371 raft_consensus.cc:2468] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 30.
I20260709 12:25:26.768271 28287 raft_consensus.cc:3055] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:25:26.795436 28045 raft_consensus.cc:493] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:26.747682 28148 raft_consensus.cc:515] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:26.798484 18367 raft_consensus.cc:1240] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 23 LEADER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 22. Current term is 23. Ops: []
I20260709 12:25:26.798640 28159 raft_consensus.cc:515] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:26.799110 17756 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36572 (request call id 1857) took 1397 ms (client timeout 1600 ms). Trace:
I20260709 12:25:26.799384 28188 raft_consensus.cc:493] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 34 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:25:26.802515 18365 raft_consensus.cc:1240] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 30. Current term is 31. Ops: []
I20260709 12:25:26.771780 17755 consensus_queue.cc:260] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 22.16, Last appended by leader: 16, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:26.880235 17756 rpcz_store.cc:269] 0709 12:25:25.401849 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:25.401976 (+   127us) service_pool.cc:224] Handling call
0709 12:25:26.798953 (+1396977us) spinlock_profiling.cc:243] Waited 348 ms on lock 0x7b540019f5d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:25:26.799053 (+   100us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":799268224}
I20260709 12:25:26.820068 17763 raft_consensus.cc:1240] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 30. Current term is 31. Ops: []
I20260709 12:25:26.818228 28153 raft_consensus.cc:493] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:25:26.819831 28293 raft_consensus.cc:493] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:26.821892 17760 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36572 (request call id 1863) took 1360 ms. Trace:
I20260709 12:25:26.892915 17760 rpcz_store.cc:276] 0709 12:25:25.461154 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:25.725148 (+263994us) service_pool.cc:224] Handling call
0709 12:25:26.821715 (+1096567us) spinlock_profiling.cc:243] Waited 257 ms on lock 0x7b54002b5f54. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:25:26.821837 (+   122us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":590458112}
I20260709 12:25:26.823546 17759 raft_consensus.cc:1240] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 29. Current term is 30. Ops: []
I20260709 12:25:26.826040 27978 raft_consensus.cc:3060] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:26.826320 17729 leader_election.cc:304] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:25:26.894325 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 68081.2us
I20260709 12:25:26.826045 17654 leader_election.cc:304] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:25:26.830960 18362 raft_consensus.cc:2468] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 34.
W20260709 12:25:26.895902 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 69919.4us
I20260709 12:25:26.896694 17654 leader_election.cc:304] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:25:26.902851 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 34 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:26.904284 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 35 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:26.837141 17685 raft_consensus.cc:2468] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 30.
I20260709 12:25:26.905225 17831 raft_consensus.cc:2393] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 34 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 34: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
W20260709 12:25:26.850926 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 66931.9us
I20260709 12:25:26.817919 17652 leader_election.cc:304] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:26.908720 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 32 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:26.911763 17762 raft_consensus.cc:3055] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:25:26.912147 17762 raft_consensus.cc:740] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:25:26.913089 17654 leader_election.cc:304] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:25:26.908120 27800 raft_consensus.cc:2804] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:25:26.915958 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 26 candidate_status { last_received { term: 25 index: 22 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:26.916558 28303 raft_consensus.cc:493] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:26.918188 17838 raft_consensus.cc:1275] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 29 index: 17. Preceding OpId from leader: term: 30 index: 18. (index mismatch)
I20260709 12:25:26.919638 28305 raft_consensus.cc:493] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:26.920338 28305 raft_consensus.cc:515] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:26.921068 28311 consensus_queue.cc:1059] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.000s
I20260709 12:25:26.921967 28311 consensus_queue.cc:1059] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20260709 12:25:26.828377 28287 raft_consensus.cc:740] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
W20260709 12:25:26.923781 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 105955us
W20260709 12:25:26.923949 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:26.924100 17835 raft_consensus.cc:1240] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 26. Current term is 27. Ops: []
I20260709 12:25:26.947800 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:26.959125 17834 raft_consensus.cc:1240] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 26. Current term is 27. Ops: []
I20260709 12:25:26.961644 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:26.907742 17728 leader_election.cc:304] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
W20260709 12:25:26.965530 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 52539.9us
I20260709 12:25:26.923182 28313 raft_consensus.cc:3055] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 29 LEADER]: Stepping down as leader of term 29
W20260709 12:25:26.983093 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:26.923106 28305 leader_election.cc:290] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:25:26.986908 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 79265.9us
I20260709 12:25:26.924561 28318 raft_consensus.cc:493] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:26.910720 28011 raft_consensus.cc:2804] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:25:26.893533 28126 consensus_queue.cc:1059] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 29.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.092s
I20260709 12:25:26.923803 28270 raft_consensus.cc:493] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:25:26.878829 28045 raft_consensus.cc:515] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:26.976716 27856 raft_consensus.cc:2804] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:25:27.010870 27856 raft_consensus.cc:493] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:27.011305 27856 raft_consensus.cc:3060] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:25:27.000615 27976 raft_consensus.cc:2804] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:25:27.014057 27976 raft_consensus.cc:493] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:25:27.014499 27976 raft_consensus.cc:3060] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:26.946872 28054 raft_consensus.cc:493] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:25:26.946475 28250 raft_consensus.cc:493] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:25:26.947059 28260 raft_consensus.cc:2804] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:25:26.924284 28326 consensus_queue.cc:1048] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.081s
I20260709 12:25:27.007949 28270 raft_consensus.cc:3060] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Advancing to term 29
W20260709 12:25:27.010437 28383 log.cc:927] Time spent T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.062s	user 0.000s	sys 0.000s
I20260709 12:25:26.903928 28153 raft_consensus.cc:515] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:26.976859 28157 raft_consensus.cc:493] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:26.898661 28188 raft_consensus.cc:515] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:26.995994 28247 raft_consensus.cc:493] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:26.994091 28313 raft_consensus.cc:740] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 29 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
W20260709 12:25:27.022616 18334 connection.cc:256] RPC call timeout handler was delayed by -1.9680423760000849s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1617s for a response
I20260709 12:25:26.996732 28148 leader_election.cc:290] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:25:26.999872 28245 raft_consensus.cc:493] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:26.910197 28293 raft_consensus.cc:515] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:26.964356 28254 consensus_queue.cc:1059] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 26.18, Next index: 19, Last known committed idx: 17, Time since last communication: 0.057s
I20260709 12:25:26.924912 28269 consensus_queue.cc:1059] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.083s
I20260709 12:25:27.029469 17834 raft_consensus.cc:1240] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 27. Current term is 28. Ops: []
W20260709 12:25:27.031607 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:27.076886 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 32 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:27.031939 17757 raft_consensus.cc:1275] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 33 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 30 index: 20. Preceding OpId from leader: term: 33 index: 21. (index mismatch)
W20260709 12:25:27.049387 18334 connection.cc:256] RPC call timeout handler was delayed by -1.6891966049997791s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2202s for a response
W20260709 12:25:27.077883 18334 connection.cc:256] RPC call timeout handler was delayed by -1.5982524479995845s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.202s for a response
W20260709 12:25:27.078289 18334 connection.cc:256] RPC call timeout handler was delayed by -1.5755363719999878s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2651s for a response
W20260709 12:25:27.078634 18334 connection.cc:256] RPC call timeout handler was delayed by -1.3936714500000562s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1811s for a response
I20260709 12:25:27.022908 28260 raft_consensus.cc:493] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:25:27.078914 18334 connection.cc:256] RPC call timeout handler was delayed by -1.3428630129997146s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3184s for a response
I20260709 12:25:27.026439 28054 raft_consensus.cc:515] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:27.081590 17764 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38786 (request call id 4296) took 1156 ms. Trace:
I20260709 12:25:27.080302 28260 raft_consensus.cc:3060] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:25:27.081194 18334 connection.cc:256] RPC call timeout handler was delayed by -1.1782914619998337s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15660000000000002s for a response
I20260709 12:25:27.093278 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
W20260709 12:25:27.095942 18334 connection.cc:256] RPC call timeout handler was delayed by -1.1719914619998235s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1573s for a response
W20260709 12:25:27.100502 18334 connection.cc:256] RPC call timeout handler was delayed by -1.0586645149996912s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2868s for a response
I20260709 12:25:27.071594 17803 leader_election.cc:304] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:25:27.096218 17838 raft_consensus.cc:2376] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 21. Re-sending same reply.
I20260709 12:25:27.026716 28330 consensus_queue.cc:1059] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 28.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.055s
I20260709 12:25:27.092906 17764 rpcz_store.cc:276] 0709 12:25:25.925630 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:26.397605 (+471975us) service_pool.cc:224] Handling call
0709 12:25:26.397882 (+   277us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:26.398065 (+   183us) raft_consensus.cc:1534] Early marking committed up to index 17
0709 12:25:26.422433 (+ 24368us) log.cc:844] Serialized 14 byte log entry
0709 12:25:26.817295 (+394862us) spinlock_profiling.cc:243] Waited 6.82 ms on lock 0x7b180031bf40. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b47515e07 00007f7b503709a9 00007f7b4753269b 00007f7b4753093f 00007f7b47532c02 00007f7b4a1ba20b 00007f7b4a1bfd23 00007f7b4a1c023a 00007f7b4a62aa2a
0709 12:25:27.081248 (+263953us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:27.081305 (+    57us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:25:27.081351 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:27.081419 (+    68us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:27.081533 (+   114us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1082,"prepare.queue_time_us":24022,"prepare.run_cpu_time_us":51,"prepare.run_wall_time_us":3152,"spinlock_wait_cycles":15680384,"thread_start_us":262608,"threads_started":1}
I20260709 12:25:27.026635 28250 raft_consensus.cc:515] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:27.118673 17803 leader_election.cc:304] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:27.081280 28269 consensus_queue.cc:1243] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer 334a4e1f59944544b3b3ff36616b57da log is divergent from this leader: its last log entry 26.17 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 16
I20260709 12:25:27.051932 28247 raft_consensus.cc:515] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:27.027395 28287 consensus_queue.cc:260] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 27.19, Last appended by leader: 19, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:27.008211 28318 raft_consensus.cc:515] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:26.917071 28303 raft_consensus.cc:515] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:27.130180 17838 raft_consensus.cc:1275] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 34 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 31 index: 18. Preceding OpId from leader: term: 34 index: 19. (index mismatch)
I20260709 12:25:27.110396 27431 raft_consensus.cc:2749] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:25:27.050487 28157 raft_consensus.cc:515] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:27.074887 28245 raft_consensus.cc:515] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:27.102540 18334 leader_election.cc:304] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:25:27.121253 28348 raft_consensus.cc:2804] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 34 FOLLOWER]: Leader pre-election won for term 35
W20260709 12:25:27.174060 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 71636.9us
I20260709 12:25:27.174998 18334 leader_election.cc:304] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:25:27.162819 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 169386us
I20260709 12:25:27.134754 17762 consensus_queue.cc:260] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 25.18, Last appended by leader: 18, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:27.148658 17803 leader_election.cc:304] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:25:27.089856 28282 consensus_queue.cc:1059] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 27.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.042s
I20260709 12:25:27.182816 18334 leader_election.cc:304] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:27.185256 17762 raft_consensus.cc:3060] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:27.177342 17652 leader_election.cc:304] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 33 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:25:27.162122 28177 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.588s	user 0.011s	sys 0.021s
I20260709 12:25:27.160789 28149 raft_consensus.cc:2749] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:25:27.125765 28305 consensus_queue.cc:1048] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.056s
I20260709 12:25:27.163599 28159 leader_election.cc:290] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 30 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:27.172281 28269 raft_consensus.cc:3055] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:25:27.184412 28105 raft_consensus.cc:2764] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Leader pre-election decision vote started in defunct term 28: won
I20260709 12:25:27.194859 18334 leader_election.cc:304] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:27.199355 17838 raft_consensus.cc:1240] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 29. Current term is 30. Ops: [29.19-29.19]
W20260709 12:25:27.200743 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:27.193415 17833 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35042 (request call id 3659) took 1328 ms. Trace:
I20260709 12:25:27.173286 28348 raft_consensus.cc:493] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 34 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
W20260709 12:25:27.197791 28177 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.623s	user 0.012s	sys 0.021s
I20260709 12:25:27.242704 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 31 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:27.247051 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:25:27.193619 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 227110us
W20260709 12:25:27.235409 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 54779.3us
I20260709 12:25:27.116063 28313 consensus_queue.cc:260] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 29.19, Last appended by leader: 19, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:27.237401 28159 consensus_queue.cc:1048] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.000s
I20260709 12:25:27.252667 28045 leader_election.cc:290] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:27.189601 28388 raft_consensus.cc:2749] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 36 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:25:27.233229 17833 rpcz_store.cc:276] 0709 12:25:25.864751 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:25.864864 (+   113us) service_pool.cc:224] Handling call
0709 12:25:25.865121 (+   257us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:26.137963 (+272842us) raft_consensus.cc:1534] Early marking committed up to index 23
0709 12:25:26.138253 (+   290us) log.cc:844] Serialized 14 byte log entry
0709 12:25:26.923730 (+785477us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:26.923781 (+    51us) raft_consensus.cc:1645] Marking committed up to 23
0709 12:25:26.923834 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:27.193136 (+269302us) spinlock_profiling.cc:243] Waited 273 ms on lock 0x7b54000893d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:25:27.193193 (+    57us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:27.193346 (+   153us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":691286,"prepare.queue_time_us":513,"prepare.run_cpu_time_us":28,"prepare.run_wall_time_us":30,"raft.queue_time_us":742590,"raft.run_cpu_time_us":104,"raft.run_wall_time_us":4403,"spinlock_wait_cycles":626888960,"thread_start_us":110146,"threads_started":1,"wal-append.queue_time_us":152089}
I20260709 12:25:27.225600 28269 raft_consensus.cc:740] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 29 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:25:27.248921 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 28 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
W20260709 12:25:27.250464 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:27.238278 28183 raft_consensus.cc:2749] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:25:27.250728 17835 raft_consensus.cc:2393] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 21: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:25:27.201423 28153 leader_election.cc:290] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:27.247720 28348 raft_consensus.cc:3060] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 34 FOLLOWER]: Advancing to term 35
W20260709 12:25:27.256006 28382 log.cc:927] Time spent T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.088s	user 0.001s	sys 0.000s
I20260709 12:25:27.256220 28312 consensus_queue.cc:260] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 29.17, Last appended by leader: 17, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:27.258775 28157 leader_election.cc:290] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:27.259060 28126 raft_consensus.cc:3055] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:25:27.266889 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 34 candidate_status { last_received { term: 32 index: 23 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:25:27.240054 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 91490.2us
I20260709 12:25:27.191620 28274 raft_consensus.cc:2749] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:25:27.297081 17834 raft_consensus.cc:2468] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 33.
I20260709 12:25:27.297367 17803 leader_election.cc:304] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:27.250063 28011 raft_consensus.cc:697] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 31 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:25:27.299201 17803 leader_election.cc:304] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:25:27.277680 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 281899us
I20260709 12:25:27.203233 28398 consensus_queue.cc:1048] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.079s
I20260709 12:25:27.284332 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:27.286429 28308 raft_consensus.cc:493] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:25:27.340790 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:27.341437 17803 leader_election.cc:304] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:25:27.342540 17803 leader_election.cc:304] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
W20260709 12:25:27.285221 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 372813us
I20260709 12:25:27.265254 28309 raft_consensus.cc:2804] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:25:27.247457 28303 leader_election.cc:290] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:25:27.290202 17762 raft_consensus.cc:1275] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 25 index: 18. Preceding OpId from leader: term: 26 index: 19. (index mismatch)
I20260709 12:25:27.277067 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:27.343914 17729 leader_election.cc:304] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:27.348244 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 34 candidate_status { last_received { term: 32 index: 23 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:27.289906 28188 leader_election.cc:290] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:27.295130 28293 leader_election.cc:290] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:27.375600 17834 raft_consensus.cc:2393] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 28: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:25:27.376708 17762 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36572 (request call id 1889) took 1023 ms. Trace:
I20260709 12:25:27.296588 28126 raft_consensus.cc:740] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:25:27.381880 17762 rpcz_store.cc:276] 0709 12:25:26.352933 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:26.459567 (+106634us) service_pool.cc:224] Handling call
0709 12:25:26.459948 (+   381us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:26.911668 (+451720us) spinlock_profiling.cc:243] Waited 39.2 ms on lock 0x7b54003c6f20. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a597dc0 00007f7b4a6221c0 00007f7b4a623231 00007f7b4a61fd8f 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c
0709 12:25:27.134398 (+222730us) spinlock_profiling.cc:243] Waited 15.5 ms on lock 0x7b4c00414340. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b488eca20 00007f7b488fda29 00007f7b488c3644 00007f7b488eaf2d 00007f7b488eab93 00007f7b4a61b18b 00007f7b4a61bcb5 00007f7b4a616f30 00007f7b4a6187de 00007f7b4a622b49 00007f7b4a623416
0709 12:25:27.376445 (+242047us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:27.376652 (+   207us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":125725184}
I20260709 12:25:27.376950 17835 raft_consensus.cc:3060] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:25:27.386551 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:27.387375 17681 raft_consensus.cc:2468] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 23.
I20260709 12:25:27.389055 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:25:27.413090 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 34 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:27.415678 17689 raft_consensus.cc:2468] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 33.
W20260709 12:25:27.417565 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 75124.7us
I20260709 12:25:27.418350 17803 leader_election.cc:304] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
W20260709 12:25:27.419540 17730 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:27.420346 17687 raft_consensus.cc:1240] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 29. Current term is 30. Ops: []
I20260709 12:25:27.341277 28403 raft_consensus.cc:2749] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:25:27.373659 28312 raft_consensus.cc:3060] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:27.374344 28402 raft_consensus.cc:2804] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:25:27.380626 28254 consensus_queue.cc:1243] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer 334a4e1f59944544b3b3ff36616b57da log is divergent from this leader: its last log entry 27.19 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 17
I20260709 12:25:27.447742 28254 consensus_queue.cc:1059] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 29.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:25:27.449371 28407 raft_consensus.cc:493] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:25:27.449968 28407 raft_consensus.cc:515] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:27.424422 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:25:27.375104 28309 raft_consensus.cc:493] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:27.453703 28309 raft_consensus.cc:3060] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:27.437237 28400 consensus_queue.cc:1048] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 18, Time since last communication: 0.023s
W20260709 12:25:27.456459 28395 log.cc:927] Time spent T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.159s	user 0.000s	sys 0.000s
W20260709 12:25:27.405489 28341 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 26 current term: 26 is higher.
I20260709 12:25:27.426432 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 34 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:27.427140 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 34 candidate_status { last_received { term: 32 index: 23 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:27.458432 17764 raft_consensus.cc:2393] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 34: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:25:27.432139 28303 raft_consensus.cc:2804] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:25:27.459368 28303 raft_consensus.cc:493] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:27.459769 28303 raft_consensus.cc:3060] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:27.449402 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:27.460892 28247 leader_election.cc:290] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:27.464015 28407 leader_election.cc:290] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:27.464851 28407 raft_consensus.cc:3055] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:25:27.465240 28407 raft_consensus.cc:740] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 29 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:25:27.466204 28407 consensus_queue.cc:260] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 29.16, Last appended by leader: 16, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:27.468029 28054 leader_election.cc:290] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:27.468214 17837 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37994 (request call id 1948) took 1338 ms. Trace:
I20260709 12:25:27.468605 17837 rpcz_store.cc:276] 0709 12:25:26.129789 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:26.132523 (+  2734us) service_pool.cc:224] Handling call
0709 12:25:26.132895 (+   372us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:26.909524 (+776629us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:26.909568 (+    44us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:26.910071 (+   503us) log.cc:844] Serialized 26 byte log entry
0709 12:25:27.285818 (+375747us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:25:27.285862 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:27.285937 (+    75us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:27.467961 (+182024us) raft_consensus.cc:1685] finished
0709 12:25:27.467986 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:27.468149 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":787326,"raft.queue_time_us":825661,"raft.run_cpu_time_us":183,"raft.run_wall_time_us":26596,"thread_start_us":286335,"threads_started":1,"wal-append.queue_time_us":449368}
I20260709 12:25:27.423306 28011 consensus_queue.cc:237] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 28.20, Last appended by leader: 20, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:27.365935 28308 raft_consensus.cc:515] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:27.434818 18374 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59886 (request call id 2799) took 1163 ms. Trace:
I20260709 12:25:27.474944 18374 rpcz_store.cc:276] 0709 12:25:26.271073 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:26.520078 (+249005us) service_pool.cc:224] Handling call
0709 12:25:26.520431 (+   353us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:25:26.520842 (+   411us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:26.520873 (+    31us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:25:26.521475 (+   602us) log.cc:844] Serialized 52 byte log entry
0709 12:25:26.744933 (+223458us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:25:26.744969 (+    36us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:26.745029 (+    60us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:27.434513 (+689484us) raft_consensus.cc:1685] finished
0709 12:25:27.434542 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:27.434745 (+   203us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":100,"raft.run_cpu_time_us":135,"raft.run_wall_time_us":3346,"thread_start_us":205121,"threads_started":1,"wal-append.queue_time_us":557227}
I20260709 12:25:27.424180 17685 raft_consensus.cc:1240] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 25. Current term is 26. Ops: []
I20260709 12:25:27.476907 28126 consensus_queue.cc:260] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 27.18, Last appended by leader: 18, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:27.478721 28126 raft_consensus.cc:3060] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Advancing to term 28
W20260709 12:25:27.481000 28387 log.cc:927] Time spent T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.313s	user 0.000s	sys 0.001s
I20260709 12:25:27.482625 28318 leader_election.cc:290] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:27.486397 27856 raft_consensus.cc:515] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:27.488776 27856 leader_election.cc:290] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 32 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:27.489359 28242 raft_consensus.cc:493] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:25:27.489885 28153 raft_consensus.cc:3055] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:25:27.490351 28153 raft_consensus.cc:740] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:25:27.494030 17682 raft_consensus.cc:2468] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 27.
I20260709 12:25:27.430536 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:27.435986 28396 raft_consensus.cc:2804] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:25:27.436615 28159 raft_consensus.cc:2749] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 33 FOLLOWER]: Leader election lost for term 33. Reason: could not achieve majority
I20260709 12:25:27.433434 28241 raft_consensus.cc:2804] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:25:27.496503 28313 raft_consensus.cc:3060] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:27.496874 28241 raft_consensus.cc:697] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 27 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:25:27.498234 28241 consensus_queue.cc:237] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 26.19, Last appended by leader: 19, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:27.437211 28250 leader_election.cc:290] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:25:27.413863 28397 log.cc:927] Time spent T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.162s	user 0.002s	sys 0.000s
I20260709 12:25:27.424994 28399 consensus_queue.cc:1059] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 35.26, Next index: 27, Last known committed idx: 26, Time since last communication: 0.026s
I20260709 12:25:27.500900 28399 consensus_queue.cc:1243] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer e2909cff1da44ad387f96a8ae3aa5afb log is divergent from this leader: its last log entry 36.27 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 26
I20260709 12:25:27.501526 28399 raft_consensus.cc:3055] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 35 LEADER]: Stepping down as leader of term 35
I20260709 12:25:27.501910 28399 raft_consensus.cc:740] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 35 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:25:27.501965 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 29 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:27.502799 17832 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 28.
I20260709 12:25:27.438622 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:25:27.418864 28385 log.cc:927] Time spent T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.293s	user 0.001s	sys 0.000s
I20260709 12:25:27.506654 17829 raft_consensus.cc:1275] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 27 index: 18. Preceding OpId from leader: term: 30 index: 19. (index mismatch)
I20260709 12:25:27.506912 18363 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59886 (request call id 2797) took 1252 ms. Trace:
I20260709 12:25:27.439386 28409 consensus_queue.cc:1048] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.000s
I20260709 12:25:27.441222 28403 consensus_queue.cc:1048] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.000s
W20260709 12:25:27.445945 27978 consensus_meta.cc:287] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.534s	user 0.009s	sys 0.000s
I20260709 12:25:27.446911 28402 raft_consensus.cc:697] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 28 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:25:27.379274 28269 consensus_queue.cc:260] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 29.22, Last appended by leader: 22, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:27.489377 28404 raft_consensus.cc:493] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:27.507264 18363 rpcz_store.cc:276] 0709 12:25:26.254926 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:26.509838 (+254912us) service_pool.cc:224] Handling call
0709 12:25:26.510133 (+   295us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 12:25:26.522389 (+ 12256us) raft_consensus.cc:1534] Early marking committed up to index 18
0709 12:25:26.537302 (+ 14913us) log.cc:844] Serialized 14 byte log entry
0709 12:25:26.906400 (+369098us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:25:26.906944 (+   544us) log.cc:844] Serialized 52 byte log entry
0709 12:25:26.907086 (+   142us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:25:26.907116 (+    30us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:26.907191 (+    75us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:27.506714 (+599523us) raft_consensus.cc:1685] finished
0709 12:25:27.506734 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:27.506861 (+   127us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":14253,"prepare.run_cpu_time_us":51,"prepare.run_wall_time_us":35,"raft.queue_time_us":214,"raft.run_cpu_time_us":235,"raft.run_wall_time_us":50156,"thread_start_us":351988,"threads_started":1,"wal-append.queue_time_us":538299}
W20260709 12:25:27.507758 17730 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:27.399307 28406 consensus_queue.cc:1048] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.009s
W20260709 12:25:27.510221 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 332948us
I20260709 12:25:27.510411 17832 raft_consensus.cc:1275] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 25 index: 18. Preceding OpId from leader: term: 28 index: 19. (index mismatch)
I20260709 12:25:27.591490 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 29 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:27.502962 28399 consensus_queue.cc:260] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 35.26, Last appended by leader: 26, Current term: 35, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:27.546530 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 29 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:27.554381 17835 raft_consensus.cc:2468] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 34.
I20260709 12:25:27.495657 28287 raft_consensus.cc:3060] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:27.592952 17764 raft_consensus.cc:2410] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 29 because replica has last-logged OpId of term: 27 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 18.
I20260709 12:25:27.516306 17681 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 28.
I20260709 12:25:27.374274 27800 raft_consensus.cc:697] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 30 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:25:27.593977 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:27.594497 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:27.495700 28396 raft_consensus.cc:697] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:25:27.592618 17683 raft_consensus.cc:2468] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 29.
I20260709 12:25:27.594633 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:27.605096 18367 raft_consensus.cc:2393] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 30: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:25:27.599630 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:27.595908 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:27.607339 17760 raft_consensus.cc:2393] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 30: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:25:27.573956 28112 raft_consensus.cc:493] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:27.596976 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 34 candidate_status { last_received { term: 32 index: 23 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:27.596704 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 29 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:27.605835 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 32 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:27.603555 17762 raft_consensus.cc:2468] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 26.
I20260709 12:25:27.610360 17764 raft_consensus.cc:3055] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:25:27.611004 17764 raft_consensus.cc:740] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 30 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
W20260709 12:25:27.612255 17727 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:27.611629 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 272508us
W20260709 12:25:27.820487 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 476524us
W20260709 12:25:27.667186 17730 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:27.814145 17758 raft_consensus.cc:1275] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 29 index: 17. Preceding OpId from leader: term: 30 index: 18. (index mismatch)
I20260709 12:25:27.619114 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:27.829067 17652 leader_election.cc:304] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:27.829147 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:27.830463 17760 raft_consensus.cc:2468] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 26.
I20260709 12:25:27.618618 17764 consensus_queue.cc:260] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 30.21, Last appended by leader: 21, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:27.832686 17764 raft_consensus.cc:3060] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:27.633108 17755 raft_consensus.cc:3060] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:27.652362 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 29 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:25:27.613286 18364 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.599s	user 0.007s	sys 0.023s
W20260709 12:25:27.838958 18364 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.824s	user 0.007s	sys 0.023s
W20260709 12:25:27.620602 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 425773us
I20260709 12:25:27.667003 17801 leader_election.cc:304] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:25:27.841563 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 174502us
I20260709 12:25:27.841949 18334 leader_election.cc:304] T a5732e71b9d54f8786b8c511d69943bf P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:25:27.684130 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:27.616055 18366 raft_consensus.cc:2393] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 34: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:25:27.844796 27431 raft_consensus.cc:2804] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:25:27.845489 27431 raft_consensus.cc:493] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:27.846035 27431 raft_consensus.cc:3060] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:27.850342 18334 leader_election.cc:304] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:25:27.654019 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 58737.8us
W20260709 12:25:27.854480 17727 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:27.634325 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:27.811724 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:27.856499 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:27.724709 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 299678us
W20260709 12:25:27.861830 18333 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:27.863158 17803 leader_election.cc:304] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:27.685329 27800 consensus_queue.cc:237] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 28.21, Last appended by leader: 21, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:27.603025 27856 consensus_queue.cc:1059] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 29.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.060s
I20260709 12:25:27.942497 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:25:27.744911 27976 consensus_meta.cc:287] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.719s	user 0.001s	sys 0.010s
I20260709 12:25:27.943477 28414 consensus_queue.cc:1048] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:25:27.944818 27976 raft_consensus.cc:515] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:27.628053 28112 raft_consensus.cc:515] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:27.948791 28426 raft_consensus.cc:2749] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
W20260709 12:25:27.949496 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 86714.2us
I20260709 12:25:27.950865 17803 leader_election.cc:304] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:27.489878 28242 raft_consensus.cc:515] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:27.634732 28245 leader_election.cc:290] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:25:27.692097 28270 consensus_meta.cc:287] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.643s	user 0.012s	sys 0.000s
W20260709 12:25:27.758569 28260 consensus_meta.cc:287] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.631s	user 0.011s	sys 0.001s
I20260709 12:25:27.969772 28426 raft_consensus.cc:3060] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:27.980715 28270 raft_consensus.cc:515] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:27.665683 28390 consensus_queue.cc:1048] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.012s
I20260709 12:25:27.866214 27978 raft_consensus.cc:493] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:27.605813 28293 raft_consensus.cc:493] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:27.691401 28315 raft_consensus.cc:3055] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:25:27.993937 27978 raft_consensus.cc:515] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:27.994220 28293 raft_consensus.cc:515] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:27.997211 27978 leader_election.cc:290] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:27.588959 28159 raft_consensus.cc:493] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:27.998486 28159 raft_consensus.cc:515] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:27.999696 27431 raft_consensus.cc:515] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:27.877240 28250 consensus_queue.cc:1048] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
W20260709 12:25:28.013242 17654 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.720s	user 0.002s	sys 0.015s
I20260709 12:25:27.878243 28398 consensus_queue.cc:1048] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.001s
I20260709 12:25:27.649546 28399 raft_consensus.cc:3060] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:25:27.879330 28274 raft_consensus.cc:2804] T a5732e71b9d54f8786b8c511d69943bf P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:25:28.024140 28274 raft_consensus.cc:493] T a5732e71b9d54f8786b8c511d69943bf P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:25:28.024770 28274 raft_consensus.cc:3060] T a5732e71b9d54f8786b8c511d69943bf P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:28.026854 17757 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38776 (request call id 3503) took 1207 ms. Trace:
I20260709 12:25:27.590400 28276 consensus_queue.cc:1048] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.012s
W20260709 12:25:28.034320 17683 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37396 (request call id 3576) took 1356 ms (client timeout 1598 ms). Trace:
I20260709 12:25:27.598111 28405 consensus_queue.cc:1059] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 30.18, Next index: 19, Last known committed idx: 17, Time since last communication: 0.060s
W20260709 12:25:27.748097 28410 log.cc:927] Time spent T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.098s	user 0.001s	sys 0.000s
I20260709 12:25:28.042347 27431 leader_election.cc:290] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 27 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:27.880405 28282 raft_consensus.cc:493] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:25:27.630609 28303 raft_consensus.cc:515] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:28.062383 28303 leader_election.cc:290] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 29 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:27.661978 28308 leader_election.cc:290] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:27.746460 28309 raft_consensus.cc:515] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:28.067164 17838 raft_consensus.cc:2393] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 31: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:25:28.068347 28309 leader_election.cc:290] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 31 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:27.639376 28318 raft_consensus.cc:493] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 34 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:25:27.880744 28424 raft_consensus.cc:493] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:25:28.069747 28318 raft_consensus.cc:515] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:28.069832 28424 raft_consensus.cc:515] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:27.694489 28396 consensus_queue.cc:237] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 25.18, Last appended by leader: 18, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:28.072609 17835 raft_consensus.cc:2393] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 29: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
W20260709 12:25:28.074916 28393 log.cc:927] Time spent T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.702s	user 0.002s	sys 0.001s
I20260709 12:25:27.682668 28402 consensus_queue.cc:237] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 27.21, Last appended by leader: 21, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:27.594563 28404 raft_consensus.cc:515] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:25:28.077364 28392 log.cc:927] Time spent T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.821s	user 0.008s	sys 0.002s
I20260709 12:25:28.078732 28404 leader_election.cc:290] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:25:27.594383 28406 consensus_queue.cc:1243] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer 0da169ff82674dc1947055ce43de68a3 log is divergent from this leader: its last log entry 31.18 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 17
W20260709 12:25:27.886400 28420 log.cc:927] Time spent T a5732e71b9d54f8786b8c511d69943bf P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.136s	user 0.002s	sys 0.000s
I20260709 12:25:27.888036 17830 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37994 (request call id 1953) took 1336 ms. Trace:
I20260709 12:25:27.888964 17802 leader_election.cc:304] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:27.894695 28416 raft_consensus.cc:2749] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:25:27.891541 28409 raft_consensus.cc:2749] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
W20260709 12:25:27.917886 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 86100.2us
I20260709 12:25:27.951519 28112 leader_election.cc:290] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:27.944836 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:28.085353 17830 rpcz_store.cc:276] 0709 12:25:26.551106 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:26.552768 (+  1662us) service_pool.cc:224] Handling call
0709 12:25:26.553079 (+   311us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:25:27.165245 (+612166us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:27.165276 (+    31us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:25:27.165857 (+   581us) log.cc:844] Serialized 52 byte log entry
0709 12:25:27.563304 (+397447us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:25:27.563376 (+    72us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:27.563524 (+   148us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:27.887131 (+323607us) raft_consensus.cc:1685] finished
0709 12:25:27.887195 (+    64us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:27.887673 (+   478us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":444551,"raft.queue_time_us":452550,"raft.run_cpu_time_us":136,"raft.run_wall_time_us":12793,"thread_start_us":509111,"threads_started":2,"wal-append.queue_time_us":498773}
W20260709 12:25:27.959270 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:27.972618 28242 leader_election.cc:290] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:28.085702 18364 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59886 (request call id 2806) took 1414 ms. Trace:
I20260709 12:25:28.101096 18364 rpcz_store.cc:276] 0709 12:25:26.671497 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:26.672424 (+   927us) service_pool.cc:224] Handling call
0709 12:25:26.672712 (+   288us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:27.007620 (+334908us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:27.007653 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:27.014354 (+  6701us) log.cc:844] Serialized 26 byte log entry
0709 12:25:27.839956 (+825602us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:25:27.840081 (+   125us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:27.840260 (+   179us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:28.085410 (+245150us) raft_consensus.cc:1685] finished
0709 12:25:28.085432 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:28.085626 (+   194us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":15866,"raft.queue_time_us":43179,"raft.run_cpu_time_us":105,"raft.run_wall_time_us":17563,"thread_start_us":1144834,"threads_started":2,"wal-append.queue_time_us":679555}
I20260709 12:25:28.102818 28269 raft_consensus.cc:3060] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:27.976472 17801 leader_election.cc:304] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:28.104048 28409 raft_consensus.cc:493] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
W20260709 12:25:28.104337 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 165100us
I20260709 12:25:28.104903 28409 raft_consensus.cc:515] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:27.980755 28260 raft_consensus.cc:515] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:27.603468 28153 consensus_queue.cc:260] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 26.18, Last appended by leader: 18, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:28.109586 28153 raft_consensus.cc:3060] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:27.994210 28315 raft_consensus.cc:740] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:25:28.113701 28315 consensus_queue.cc:260] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 27.19, Last appended by leader: 19, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:28.001008 17803 leader_election.cc:304] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:25:28.116849 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 147702us
I20260709 12:25:28.001219 28159 leader_election.cc:290] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:28.001837 17728 leader_election.cc:304] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:25:28.003619 28305 raft_consensus.cc:2804] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:25:28.005112 17831 raft_consensus.cc:1275] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 26 index: 18. Preceding OpId from leader: term: 27 index: 19. (index mismatch)
I20260709 12:25:28.008620 27976 leader_election.cc:290] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 29 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:27.967243 17729 leader_election.cc:304] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:27.876981 18334 leader_election.cc:304] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:25:28.018323 17654 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.768s	user 0.002s	sys 0.015s
W20260709 12:25:28.035003 17683 rpcz_store.cc:269] 0709 12:25:26.678098 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:26.678277 (+   179us) service_pool.cc:224] Handling call
0709 12:25:28.033779 (+1355502us) spinlock_profiling.cc:243] Waited 913 ms on lock 0x7b5400380254. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:25:28.033986 (+   207us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2100422016}
I20260709 12:25:28.040671 17832 raft_consensus.cc:1240] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 26. Current term is 27. Ops: []
I20260709 12:25:28.040974 28448 raft_consensus.cc:3055] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 30 LEADER]: Stepping down as leader of term 30
W20260709 12:25:28.042402 17730 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:28.043385 17836 raft_consensus.cc:1240] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 28. Current term is 29. Ops: []
I20260709 12:25:28.027334 17757 rpcz_store.cc:276] 0709 12:25:26.819068 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:26.819187 (+   119us) service_pool.cc:224] Handling call
0709 12:25:26.819484 (+   297us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:27.077606 (+258122us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:28.026215 (+948609us) spinlock_profiling.cc:243] Waited 208 ms on lock 0x7b54002b6454. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:25:28.026610 (+   395us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":478497536}
I20260709 12:25:27.979768 17727 leader_election.cc:304] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:28.053216 17837 raft_consensus.cc:1240] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 25. Current term is 26. Ops: []
I20260709 12:25:28.051851 17755 raft_consensus.cc:1275] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 22 index: 16. Preceding OpId from leader: term: 23 index: 16. (term mismatch)
I20260709 12:25:28.063279 28282 raft_consensus.cc:515] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:27.771689 28348 raft_consensus.cc:515] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 35 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:25:28.128822 17730 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:28.129252 28260 leader_election.cc:290] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 27 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:28.129348 28282 leader_election.cc:290] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:28.088562 28054 raft_consensus.cc:2764] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Leader pre-election decision vote started in defunct term 29: won
I20260709 12:25:28.129899 28348 leader_election.cc:290] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 35 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:28.126158 28448 raft_consensus.cc:740] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 30 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
W20260709 12:25:28.130997 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 213546us
I20260709 12:25:28.132139 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:28.132787 17729 leader_election.cc:304] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:25:28.133381 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 313637us
W20260709 12:25:28.134531 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 156361us
I20260709 12:25:28.134588 17728 leader_election.cc:304] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:25:28.135290 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 31 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:28.087935 28416 raft_consensus.cc:493] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:28.135807 28407 raft_consensus.cc:3060] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:28.136229 28416 raft_consensus.cc:515] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:28.138336 17728 leader_election.cc:304] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:25:28.139025 28416 leader_election.cc:290] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:25:28.139950 28330 raft_consensus.cc:2749] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:25:28.141063 28330 raft_consensus.cc:2804] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 34 FOLLOWER]: Leader election won for term 34
I20260709 12:25:28.141772 28330 raft_consensus.cc:697] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 34 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:25:28.142081 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:28.142563 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 35 candidate_status { last_received { term: 34 index: 23 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:28.133167 17836 raft_consensus.cc:2468] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 23.
I20260709 12:25:28.143613 28341 raft_consensus.cc:2749] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260709 12:25:28.134779 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:28.144757 17763 raft_consensus.cc:3060] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:28.144882 28341 raft_consensus.cc:2804] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:25:28.145599 28341 raft_consensus.cc:493] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:25:28.146059 28341 raft_consensus.cc:3060] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:28.149755 28425 raft_consensus.cc:2804] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:25:28.149955 28293 leader_election.cc:290] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:28.150306 28425 raft_consensus.cc:493] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:25:28.150781 28425 raft_consensus.cc:3060] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:25:28.077384 17833 raft_consensus.cc:1240] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 29. Current term is 30. Ops: []
I20260709 12:25:28.126056 17755 pending_rounds.cc:85] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb: Aborting all ops after (but not including) 15
W20260709 12:25:28.127341 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 877467us
I20260709 12:25:28.088058 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 32 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:28.140949 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:25:28.086823 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 220627us
I20260709 12:25:28.093513 17652 leader_election.cc:304] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:28.094775 28424 leader_election.cc:290] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:28.161091 28404 raft_consensus.cc:2749] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 34 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:25:28.171696 17654 leader_election.cc:304] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:25:28.145064 28315 raft_consensus.cc:3060] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:28.083824 28270 leader_election.cc:290] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 29 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:28.133241 28406 raft_consensus.cc:2804] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:25:28.090101 28318 leader_election.cc:290] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:28.180099 28305 raft_consensus.cc:493] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:25:28.132179 28448 consensus_queue.cc:260] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 30.18, Last appended by leader: 18, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:28.182933 28308 consensus_queue.cc:1059] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 34.22, Next index: 23, Last known committed idx: 22, Time since last communication: 0.001s
W20260709 12:25:28.183620 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 306992us
I20260709 12:25:28.152509 17755 pending_rounds.cc:107] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb: Aborting uncommitted NO_OP operation due to leader change: 22.16
I20260709 12:25:28.210189 17755 raft_consensus.cc:2889] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:28.175406 17802 leader_election.cc:304] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:28.143105 28330 consensus_queue.cc:237] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 32.23, Last appended by leader: 23, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:28.121757 28409 leader_election.cc:290] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:28.139721 28416 raft_consensus.cc:2749] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:25:28.160580 28242 raft_consensus.cc:2804] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:25:28.093115 28426 raft_consensus.cc:2749] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:25:28.171245 28274 raft_consensus.cc:515] T a5732e71b9d54f8786b8c511d69943bf P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Starting leader election with config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:28.222324 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 50925.3us
I20260709 12:25:28.211057 28305 raft_consensus.cc:3060] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:25:28.223471 17758 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38776 (request call id 3500) took 1499 ms. Trace:
I20260709 12:25:28.225476 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 34 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:28.224555 17764 raft_consensus.cc:1275] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 30 index: 21. Preceding OpId from leader: term: 31 index: 22. (index mismatch)
I20260709 12:25:28.226557 17758 rpcz_store.cc:276] 0709 12:25:26.724261 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:26.739591 (+ 15330us) service_pool.cc:224] Handling call
0709 12:25:26.739842 (+   251us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:27.824496 (+1084654us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:28.221087 (+396591us) spinlock_profiling.cc:243] Waited 1.07 s on lock 0x7b54000ff0d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:25:28.223380 (+  2293us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2467575680}
I20260709 12:25:28.232569 18364 raft_consensus.cc:2410] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for term 34 because replica has last-logged OpId of term: 31 index: 20, which is greater than that of the candidate, which has last-logged OpId of term: 30 index: 20.
I20260709 12:25:28.217816 28406 raft_consensus.cc:697] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 30 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
W20260709 12:25:28.234768 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 60628.7us
W20260709 12:25:28.237239 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 145487us
W20260709 12:25:28.237382 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 62166.1us
I20260709 12:25:28.238581 17802 leader_election.cc:304] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:28.240361 18365 raft_consensus.cc:1275] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 33 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 31 index: 21. Preceding OpId from leader: term: 33 index: 22. (index mismatch)
I20260709 12:25:28.240690 17755 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36572 (request call id 1892) took 1478 ms. Trace:
I20260709 12:25:28.241039 17755 rpcz_store.cc:276] 0709 12:25:26.762818 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:26.763088 (+   270us) service_pool.cc:224] Handling call
0709 12:25:26.763491 (+   403us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:27.632601 (+869110us) spinlock_profiling.cc:243] Waited 540 ms on lock 0x7b5800a0a0d0. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a57c0aa 00007f7b4a5fe447 00007f7b4a616fe7 00007f7b4a6187de 00007f7b4a622b49 00007f7b4a623416 00007f7b4a61fd8f 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4
0709 12:25:28.240358 (+607757us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:28.240634 (+   276us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1241729792}
W20260709 12:25:28.241521 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 103435us
W20260709 12:25:28.229986 28126 consensus_meta.cc:287] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.751s	user 0.011s	sys 0.016s
I20260709 12:25:28.242323 18366 raft_consensus.cc:1240] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 27. Current term is 28. Ops: [25.18-27.19]
I20260709 12:25:28.242739 17759 raft_consensus.cc:2468] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 28.
W20260709 12:25:28.244167 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 125616us
I20260709 12:25:28.245641 28271 raft_consensus.cc:3055] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:25:28.246059 28271 raft_consensus.cc:740] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:25:28.248246 28309 raft_consensus.cc:2804] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:25:28.248723 28448 raft_consensus.cc:3060] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:28.248822 28309 raft_consensus.cc:493] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 34 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:25:28.249413 28309 raft_consensus.cc:3060] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 34 FOLLOWER]: Advancing to term 35
W20260709 12:25:28.254099 28320 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 23 current term: 23 is higher.
I20260709 12:25:28.256862 28404 raft_consensus.cc:3055] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 34 LEADER]: Stepping down as leader of term 34
I20260709 12:25:28.257427 28404 raft_consensus.cc:740] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 34 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:25:28.257493 28054 raft_consensus.cc:2749] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:25:28.258550 28404 consensus_queue.cc:260] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 34.22, Last appended by leader: 22, Current term: 34, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:28.261898 17728 leader_election.cc:304] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:25:28.239635 17652 leader_election.cc:304] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:28.247174 28271 consensus_queue.cc:260] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 27.21, Last appended by leader: 21, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:28.223351 28159 raft_consensus.cc:2749] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:25:28.263957 28401 consensus_queue.cc:1059] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 24.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:25:28.289479 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:28.291774 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:28.293192 17801 leader_election.cc:304] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:28.277805 28126 raft_consensus.cc:2804] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:25:28.287408 28409 raft_consensus.cc:2804] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:25:28.301563 28157 consensus_queue.cc:1048] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.079s
I20260709 12:25:28.302093 28409 raft_consensus.cc:493] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:28.302521 28409 raft_consensus.cc:3060] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:25:28.302958 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 29 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:28.303814 18368 raft_consensus.cc:3060] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:28.306959 18362 raft_consensus.cc:1275] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 36 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 33 index: 25. Preceding OpId from leader: term: 36 index: 27. (index mismatch)
I20260709 12:25:28.307248 18366 raft_consensus.cc:2393] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 28: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
W20260709 12:25:28.308455 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 79879.2us
I20260709 12:25:28.309458 28404 raft_consensus.cc:3060] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:25:28.310444 18334 leader_election.cc:400] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 32 pre-election: Vote denied by peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071) with higher term. Message: Invalid argument: T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 32. Current term is 33.
I20260709 12:25:28.312106 27800 raft_consensus.cc:2749] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
W20260709 12:25:28.312285 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 72827.6us
I20260709 12:25:28.313278 17652 leader_election.cc:304] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:28.314862 28159 raft_consensus.cc:2749] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:25:28.315114 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:28.299831 28424 raft_consensus.cc:493] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:28.317013 28424 raft_consensus.cc:515] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:28.320328 28450 log.cc:927] Time spent T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.056s	user 0.001s	sys 0.001s
I20260709 12:25:28.320566 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 33 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:28.321218 18361 raft_consensus.cc:2468] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 31.
I20260709 12:25:28.308535 28271 raft_consensus.cc:3060] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:28.331414 28401 consensus_queue.cc:1048] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:25:28.237450 28406 consensus_queue.cc:237] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 28.20, Last appended by leader: 20, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:28.298272 28274 leader_election.cc:290] T a5732e71b9d54f8786b8c511d69943bf P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 30 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:28.333268 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 29 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:28.334977 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:28.340185 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 29 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:25:28.340416 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 56595.3us
I20260709 12:25:28.342782 28242 raft_consensus.cc:493] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:25:28.343364 28242 raft_consensus.cc:3060] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:25:28.344391 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 51310.2us
I20260709 12:25:28.345201 28308 raft_consensus.cc:2804] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:25:28.345642 28308 raft_consensus.cc:493] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:25:28.346036 28308 raft_consensus.cc:3060] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:28.353912 17802 leader_election.cc:304] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:25:28.356441 28054 raft_consensus.cc:2804] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:25:28.356902 28054 raft_consensus.cc:493] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:25:28.357354 28054 raft_consensus.cc:3060] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:28.361085 17802 leader_election.cc:304] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:28.362622 28348 raft_consensus.cc:3055] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:25:28.362700 28426 raft_consensus.cc:2804] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:25:28.363251 28348 raft_consensus.cc:740] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 24 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:25:28.363375 28426 raft_consensus.cc:493] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 33 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:28.363921 28426 raft_consensus.cc:3060] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:25:28.364746 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:28.365903 28188 raft_consensus.cc:493] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:28.366489 28188 raft_consensus.cc:515] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:28.367924 28313 raft_consensus.cc:493] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:28.368709 28320 raft_consensus.cc:493] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:25:28.368574 28313 raft_consensus.cc:515] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:28.369442 28320 raft_consensus.cc:515] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:28.370051 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 29 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:28.372022 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
W20260709 12:25:28.372661 17730 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:28.388672 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 35 candidate_status { last_received { term: 34 index: 23 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:28.389442 18364 raft_consensus.cc:2468] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 34.
I20260709 12:25:28.390060 17758 raft_consensus.cc:1275] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 29 index: 19. Preceding OpId from leader: term: 30 index: 19. (term mismatch)
I20260709 12:25:28.390451 17758 pending_rounds.cc:85] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb: Aborting all ops after (but not including) 18
I20260709 12:25:28.390735 17758 pending_rounds.cc:107] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb: Aborting uncommitted NO_OP operation due to leader change: 29.19
I20260709 12:25:28.391007 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:28.391319 17758 raft_consensus.cc:2889] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:28.391952 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 29 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:25:28.391144 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 78036.8us
I20260709 12:25:28.396517 17652 leader_election.cc:304] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:28.373943 28348 consensus_queue.cc:260] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 24.16, Last appended by leader: 16, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:28.301333 28126 raft_consensus.cc:493] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:25:28.379047 18365 raft_consensus.cc:1240] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 27. Current term is 28. Ops: []
W20260709 12:25:28.380599 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:28.399472 28126 raft_consensus.cc:3060] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:28.381801 28403 consensus_queue.cc:1059] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 25.22, Next index: 23, Last known committed idx: 22, Time since last communication: 0.001s
I20260709 12:25:28.381824 18367 raft_consensus.cc:1275] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 26 index: 19. Preceding OpId from leader: term: 27 index: 20. (index mismatch)
I20260709 12:25:28.400468 27431 raft_consensus.cc:3055] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:25:28.382685 17728 leader_election.cc:304] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:28.400951 27431 raft_consensus.cc:740] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 25 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:25:28.376025 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 31 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:28.402439 18361 raft_consensus.cc:2393] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 31: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:25:28.410796 28416 raft_consensus.cc:493] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 34 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:25:28.411399 28416 raft_consensus.cc:515] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:28.413523 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 89916.2us
I20260709 12:25:28.387141 17689 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 30.
I20260709 12:25:28.386703 28409 raft_consensus.cc:515] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 32 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:28.390686 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 29 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:28.415870 17761 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 27.
I20260709 12:25:28.419199 28409 leader_election.cc:290] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 32 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:28.391820 18364 raft_consensus.cc:3060] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:28.392136 17757 raft_consensus.cc:1275] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 27 index: 19. Preceding OpId from leader: term: 28 index: 19. (term mismatch)
I20260709 12:25:28.420258 17757 pending_rounds.cc:85] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb: Aborting all ops after (but not including) 18
I20260709 12:25:28.420562 17757 pending_rounds.cc:107] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb: Aborting uncommitted NO_OP operation due to leader change: 27.19
I20260709 12:25:28.420871 17757 raft_consensus.cc:2889] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:28.380353 28270 consensus_queue.cc:1059] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.001s
I20260709 12:25:28.423101 28270 raft_consensus.cc:3055] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:25:28.423477 28270 raft_consensus.cc:740] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 29 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:25:28.424678 28270 consensus_queue.cc:260] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 29.20, Last appended by leader: 20, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:28.426143 28270 raft_consensus.cc:3060] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:28.382459 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:28.413525 28242 raft_consensus.cc:515] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:28.402177 27431 consensus_queue.cc:260] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 25.22, Last appended by leader: 22, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:28.402469 17728 leader_election.cc:304] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:28.384377 28341 raft_consensus.cc:515] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:28.319824 28424 leader_election.cc:290] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:25:28.407020 17655 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:28.407526 28149 raft_consensus.cc:493] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:28.377633 28188 leader_election.cc:290] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:28.411556 28054 raft_consensus.cc:515] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:28.399338 28348 raft_consensus.cc:3060] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:28.410578 28320 leader_election.cc:290] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:28.413652 28416 leader_election.cc:290] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:28.385010 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 30 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:28.431056 28183 consensus_queue.cc:1059] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 26.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:25:28.432410 28391 raft_consensus.cc:493] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:25:28.432232 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:25:28.433102 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 74573.4us
I20260709 12:25:28.434108 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:28.384315 17802 leader_election.cc:304] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:28.435439 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 34 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:28.387001 17763 raft_consensus.cc:2468] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 27.
I20260709 12:25:28.430192 18334 leader_election.cc:304] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:28.333811 16501 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
W20260709 12:25:28.448544 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 50217us
I20260709 12:25:28.371582 28313 leader_election.cc:290] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:28.468884 18368 raft_consensus.cc:2468] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 29.
W20260709 12:25:28.477384 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:28.477636 18366 raft_consensus.cc:1275] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 28 index: 20. Preceding OpId from leader: term: 31 index: 21. (index mismatch)
I20260709 12:25:28.478567 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 31 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:28.481585 17760 raft_consensus.cc:1240] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 26. Current term is 27. Ops: []
I20260709 12:25:28.489912 18363 raft_consensus.cc:1240] T a5732e71b9d54f8786b8c511d69943bf P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 29. Current term is 30. Ops: []
I20260709 12:25:28.482106 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 33 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:28.494099 17762 raft_consensus.cc:1240] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 26. Current term is 27. Ops: []
I20260709 12:25:28.495379 17801 leader_election.cc:304] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:25:28.490024 18361 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 30.
I20260709 12:25:28.499455 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 31 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:25:28.504716 17801 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:28.486797 27800 raft_consensus.cc:2804] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:25:28.518335 17755 raft_consensus.cc:2468] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 28.
I20260709 12:25:28.506295 28401 raft_consensus.cc:2749] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:25:28.522403 17757 raft_consensus.cc:2468] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 33.
I20260709 12:25:28.473362 28149 raft_consensus.cc:515] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:25:28.527971 28454 log.cc:927] Time spent T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.086s	user 0.005s	sys 0.003s
I20260709 12:25:28.528285 28402 raft_consensus.cc:493] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:25:28.528134 28276 consensus_queue.cc:1048] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.001s
W20260709 12:25:28.529248 28434 log.cc:927] Time spent T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.117s	user 0.001s	sys 0.004s
I20260709 12:25:28.528970 28112 consensus_queue.cc:1059] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 28.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
W20260709 12:25:28.529556 28444 log.cc:927] Time spent T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.108s	user 0.007s	sys 0.000s
W20260709 12:25:28.528997 28458 log.cc:927] Time spent T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.110s	user 0.000s	sys 0.004s
W20260709 12:25:28.557341 17730 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:28.531564 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 30 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:28.524473 18333 leader_election.cc:304] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:25:28.531960 28011 raft_consensus.cc:2749] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
W20260709 12:25:28.532378 28456 log.cc:927] Time spent T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.119s	user 0.000s	sys 0.010s
I20260709 12:25:28.516494 28391 raft_consensus.cc:515] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:28.488590 28407 consensus_queue.cc:1048] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.044s
I20260709 12:25:28.562589 28406 consensus_queue.cc:1048] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 24, Time since last communication: 0.001s
I20260709 12:25:28.562980 28407 consensus_queue.cc:1243] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer e063541542a94969898868bf6eb5ce1b log is divergent from this leader: its last log entry 31.18 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 17
I20260709 12:25:28.565181 28391 leader_election.cc:290] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:25:28.534126 28455 log.cc:927] Time spent T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.117s	user 0.000s	sys 0.010s
I20260709 12:25:28.534689 28390 raft_consensus.cc:2749] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
W20260709 12:25:28.567070 17730 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:28.570557 28406 consensus_queue.cc:1059] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 29.17, Next index: 18, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:25:28.555934 28425 raft_consensus.cc:515] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:28.571894 28406 raft_consensus.cc:3055] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:25:28.572419 28406 raft_consensus.cc:740] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 29 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:25:28.574015 17757 raft_consensus.cc:1240] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 32 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 31. Current term is 32. Ops: []
I20260709 12:25:28.574254 18374 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59902 (request call id 2440) took 1031 ms. Trace:
I20260709 12:25:28.575012 28402 raft_consensus.cc:515] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:28.574643 18374 rpcz_store.cc:276] 0709 12:25:27.542545 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:27.542726 (+   181us) service_pool.cc:224] Handling call
0709 12:25:27.543157 (+   431us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:27.543602 (+   445us) raft_consensus.cc:1534] Early marking committed up to index 18
0709 12:25:27.543640 (+    38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:27.544253 (+   613us) log.cc:844] Serialized 26 byte log entry
0709 12:25:27.645676 (+101423us) spinlock_profiling.cc:243] Waited 5.62 ms on lock 0x7b4800220a58. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4f549aa1 00007f7b4f540b37 00007f7b4f5409ca 00007f7b4f63826f 00007f7b4f637f37 00007f7b4f64c55c 00007f7b4f64c508 00007f7b4f64c488 00007f7b4f64c443 00007f7b4f64b6ec 00007f7b4e575a96
0709 12:25:28.008150 (+362474us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:25:28.008236 (+    86us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:28.008342 (+   106us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:28.341704 (+333362us) spinlock_profiling.cc:243] Waited 10.3 ms on lock 0x7b180048e140. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b475333f3 00007f7b475357c1 00007f7b47535779 00007f7b47535709 00007f7b475356d1 00007f7b475349cd 0000000000388634 0000000000388479 00007f7b4751683a
0709 12:25:28.573754 (+232050us) raft_consensus.cc:1685] finished
0709 12:25:28.573788 (+    34us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:28.574025 (+   237us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":11822,"raft.queue_time_us":30749,"raft.run_cpu_time_us":1144,"raft.run_wall_time_us":60424,"spinlock_wait_cycles":36639744,"thread_start_us":419934,"threads_started":1,"wal-append.queue_time_us":435498}
I20260709 12:25:28.576742 28309 raft_consensus.cc:515] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 35 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:28.577785 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:28.578470 17835 raft_consensus.cc:3060] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:28.578666 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:28.581265 28309 leader_election.cc:290] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 35 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:25:28.536414 28157 raft_consensus.cc:493] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:28.522255 28409 raft_consensus.cc:493] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:28.584708 28157 raft_consensus.cc:515] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:28.584867 28409 raft_consensus.cc:515] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:28.587013 28157 leader_election.cc:290] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:28.587170 28409 leader_election.cc:290] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:28.537519 28247 raft_consensus.cc:2804] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:25:28.587954 28309 raft_consensus.cc:493] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:28.588052 28247 raft_consensus.cc:697] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 27 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:25:28.588512 28309 raft_consensus.cc:515] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:28.534797 17578 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35958 (request call id 417) took 32119 ms (client timeout 15000 ms). Trace:
I20260709 12:25:28.537778 28242 leader_election.cc:290] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 27 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:25:28.538704 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 108633us
W20260709 12:25:28.528863 28457 log.cc:927] Time spent T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.128s	user 0.002s	sys 0.002s
I20260709 12:25:28.526150 17756 raft_consensus.cc:2468] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 23.
I20260709 12:25:28.592516 18371 raft_consensus.cc:2468] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 29.
I20260709 12:25:28.540298 28308 raft_consensus.cc:515] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:28.541827 28305 raft_consensus.cc:515] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 32 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:28.542622 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 158482us
I20260709 12:25:28.517287 27800 raft_consensus.cc:697] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 21 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:25:28.548254 17585 catalog_manager.cc:5697] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 28 to 29, leader changed from 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) to 334a4e1f59944544b3b3ff36616b57da (127.16.29.68). New cstate: current_term: 29 leader_uuid: "334a4e1f59944544b3b3ff36616b57da" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 12:25:28.549654 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 147329us
I20260709 12:25:28.517133 28320 consensus_queue.cc:1048] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.035s
I20260709 12:25:28.557367 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:28.604125 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 32 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
W20260709 12:25:28.538136 28460 log.cc:927] Time spent T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.112s	user 0.000s	sys 0.010s
I20260709 12:25:28.540535 28341 leader_election.cc:290] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 27 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:25:28.597169 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 74504.3us
I20260709 12:25:28.589435 28247 consensus_queue.cc:237] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 25.16, Last appended by leader: 16, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:28.583721 17654 leader_election.cc:304] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:25:28.518505 28398 raft_consensus.cc:2749] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:25:28.606304 17652 leader_election.cc:304] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:28.606988 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 35 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:28.608695 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 36 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:28.613143 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 30 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:28.613360 27800 consensus_queue.cc:237] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.13, Last appended by leader: 13, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:28.616009 28157 raft_consensus.cc:2804] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:25:28.616469 28157 raft_consensus.cc:493] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:28.616518 28309 leader_election.cc:290] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:28.616955 28157 raft_consensus.cc:3060] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:28.617800 28416 raft_consensus.cc:3055] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:25:28.618300 28416 raft_consensus.cc:740] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:25:28.620265 17802 leader_election.cc:304] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:28.621330 28309 raft_consensus.cc:2804] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:25:28.622010 28309 raft_consensus.cc:697] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 28 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:25:28.623579 28309 consensus_queue.cc:237] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 27.14, Last appended by leader: 14, Current term: 28, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:28.624794 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:28.625396 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 29 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:28.626600 18334 leader_election.cc:304] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:28.626921 28305 leader_election.cc:290] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 32 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:28.627532 28390 raft_consensus.cc:2749] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:25:28.628645 18334 leader_election.cc:304] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:28.629441 28390 raft_consensus.cc:2749] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:25:28.630385 28390 consensus_queue.cc:1059] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 25.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.001s
I20260709 12:25:28.631736 28390 raft_consensus.cc:3055] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:25:28.632165 17728 leader_election.cc:304] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:25:28.630416 18334 leader_election.cc:304] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:25:28.633155 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 36 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:28.633486 28314 raft_consensus.cc:2764] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Leader pre-election decision vote started in defunct term 26: won
I20260709 12:25:28.633950 17728 leader_election.cc:304] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:28.634593 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 29 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:28.634963 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 33 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:28.635763 17837 raft_consensus.cc:2468] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 32.
I20260709 12:25:28.636740 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:28.638043 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 30 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:28.638226 18334 leader_election.cc:304] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:28.638828 17838 raft_consensus.cc:2410] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for term 30 because replica has last-logged OpId of term: 28 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 26 index: 16.
I20260709 12:25:28.639397 28112 raft_consensus.cc:2749] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260709 12:25:28.639739 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 32 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:28.641549 18334 leader_election.cc:304] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:28.642614 28112 raft_consensus.cc:2749] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:25:28.642688 28426 raft_consensus.cc:515] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 34 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:25:28.646185 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 62559.8us
I20260709 12:25:28.651970 17654 leader_election.cc:304] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:28.596410 28402 leader_election.cc:290] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:28.648747 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 29 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:28.660269 18334 leader_election.cc:304] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:25:28.661298 28399 raft_consensus.cc:2804] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:25:28.661984 28399 raft_consensus.cc:493] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:28.662388 28399 raft_consensus.cc:3060] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 32 FOLLOWER]: Advancing to term 33
W20260709 12:25:28.663424 17727 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.16.29.67:36837 timed out after 1.910s (SENT) [suppressed 23 similar messages]
I20260709 12:25:28.663667 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:28.664028 28341 raft_consensus.cc:2804] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:25:28.664662 28341 raft_consensus.cc:493] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:25:28.665490 28341 raft_consensus.cc:3060] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:28.665829 28406 consensus_queue.cc:260] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 29.17, Last appended by leader: 17, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:28.667199 28406 raft_consensus.cc:3060] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:28.674388 17835 raft_consensus.cc:2468] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 30.
I20260709 12:25:28.679171 28271 raft_consensus.cc:2749] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:25:28.679701 18334 leader_election.cc:304] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:28.677948 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:28.684757 28315 raft_consensus.cc:493] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:28.685442 28315 raft_consensus.cc:515] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:25:28.688004 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 54176.7us
I20260709 12:25:28.688131 28328 consensus_queue.cc:1048] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 19, Time since last communication: 0.001s
I20260709 12:25:28.688918 28149 leader_election.cc:290] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:25:28.689606 17728 leader_election.cc:304] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
W20260709 12:25:28.679741 28463 log.cc:927] Time spent T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.120s	user 0.000s	sys 0.000s
I20260709 12:25:28.690927 28320 raft_consensus.cc:3060] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:28.691454 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 34 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:25:28.689631 28054 leader_election.cc:290] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 24 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:28.680500 28314 raft_consensus.cc:2804] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:25:28.694676 18334 leader_election.cc:304] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:28.696123 28274 raft_consensus.cc:2749] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
I20260709 12:25:28.697001 28274 raft_consensus.cc:3060] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:25:28.619365 28416 consensus_queue.cc:260] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 28.17, Last appended by leader: 17, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:28.699783 17834 pending_rounds.cc:85] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b: Aborting all ops after (but not including) 17
I20260709 12:25:28.700116 17834 pending_rounds.cc:107] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b: Aborting uncommitted NO_OP operation due to leader change: 31.18
I20260709 12:25:28.700356 17834 raft_consensus.cc:2889] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 34 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:28.654477 28401 raft_consensus.cc:2749] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:25:28.683799 18364 raft_consensus.cc:2468] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 27.
I20260709 12:25:28.632210 28390 raft_consensus.cc:740] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:25:28.674854 28391 consensus_queue.cc:1059] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.031s
I20260709 12:25:28.693058 28314 raft_consensus.cc:493] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:25:28.706403 28314 raft_consensus.cc:3060] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Advancing to term 32
W20260709 12:25:28.714076 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 111692us
I20260709 12:25:28.710644 28315 leader_election.cc:290] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:28.710638 28276 consensus_queue.cc:1059] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 32.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.006s
I20260709 12:25:28.706425 28391 raft_consensus.cc:3055] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:25:28.728289 28391 raft_consensus.cc:740] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:25:28.716998 28308 leader_election.cc:290] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 28 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:28.729336 28391 consensus_queue.cc:260] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 26.17, Last appended by leader: 17, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:28.729918 17728 leader_election.cc:304] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:25:28.731051 28391 raft_consensus.cc:3060] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:28.732080 27431 raft_consensus.cc:3060] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:28.732614 28401 raft_consensus.cc:2804] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:25:28.733143 28401 raft_consensus.cc:493] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:25:28.733589 28401 raft_consensus.cc:3060] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:28.717154 28426 leader_election.cc:290] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 34 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:25:28.737654 28465 log.cc:927] Time spent T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.169s	user 0.003s	sys 0.000s
W20260709 12:25:28.741184 28464 log.cc:927] Time spent T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.062s	user 0.001s	sys 0.000s
I20260709 12:25:28.741809 17683 raft_consensus.cc:1240] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 32 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 31. Current term is 32. Ops: [31.21-31.21]
I20260709 12:25:28.742506 28157 raft_consensus.cc:515] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:28.744149 17682 raft_consensus.cc:1240] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 32 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 31. Current term is 32. Ops: [31.19-31.19]
I20260709 12:25:28.745635 28157 leader_election.cc:290] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 24 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:28.750368 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:25:28.751291 17684 raft_consensus.cc:2393] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 29: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
W20260709 12:25:28.763607 28462 log.cc:927] Time spent T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.202s	user 0.000s	sys 0.001s
I20260709 12:25:28.765245 28328 raft_consensus.cc:493] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:28.765985 28328 raft_consensus.cc:515] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:28.765292 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 32 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:28.770151 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:28.770637 28314 raft_consensus.cc:515] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 32 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:28.772848 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 34 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:28.774011 17757 raft_consensus.cc:1275] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 29 index: 22. Preceding OpId from leader: term: 30 index: 22. (term mismatch)
I20260709 12:25:28.774437 17757 pending_rounds.cc:85] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb: Aborting all ops after (but not including) 21
I20260709 12:25:28.774691 17757 pending_rounds.cc:107] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb: Aborting uncommitted NO_OP operation due to leader change: 29.22
I20260709 12:25:28.774940 17757 raft_consensus.cc:2889] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:28.776361 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 29 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:25:28.777141 17682 raft_consensus.cc:3060] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Advancing to term 29
W20260709 12:25:28.780251 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 128422us
I20260709 12:25:28.780668 17755 raft_consensus.cc:1240] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 35 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 34. Current term is 35. Ops: []
I20260709 12:25:28.782722 28328 leader_election.cc:290] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:28.783496 28416 raft_consensus.cc:3060] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:28.785903 17654 leader_election.cc:304] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:28.787259 17654 leader_election.cc:304] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:28.784788 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 31 candidate_status { last_received { term: 29 index: 22 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:28.760464 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 30 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:28.788415 18361 raft_consensus.cc:2468] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 30.
I20260709 12:25:28.790318 17729 leader_election.cc:304] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:28.792171 17685 raft_consensus.cc:1275] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 27 index: 21. Preceding OpId from leader: term: 28 index: 22. (index mismatch)
I20260709 12:25:28.793965 28314 leader_election.cc:290] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 32 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:25:28.801462 17730 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:28.800479 17761 raft_consensus.cc:2468] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 28.
I20260709 12:25:28.803161 17652 leader_election.cc:304] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:28.780912 17683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:28.806504 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:25:28.817617 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 96808.4us
I20260709 12:25:28.807050 17829 raft_consensus.cc:2468] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 35.
I20260709 12:25:28.818279 17835 raft_consensus.cc:1240] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 29. Current term is 30. Ops: []
I20260709 12:25:28.813210 17585 catalog_manager.cc:5697] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 23 to 25. New cstate: current_term: 25 leader_uuid: "334a4e1f59944544b3b3ff36616b57da" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:25:28.815447 17801 leader_election.cc:304] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:28.821990 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 35 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:28.828836 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:28.804548 28328 raft_consensus.cc:2804] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Leader pre-election won for term 31
W20260709 12:25:28.831382 17727 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:28.832954 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 45773.4us
I20260709 12:25:28.831264 17685 raft_consensus.cc:2468] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 29.
I20260709 12:25:28.833922 17654 leader_election.cc:304] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:25:28.838766 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 35 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:25:28.839516 17687 raft_consensus.cc:3060] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:25:28.843246 17682 raft_consensus.cc:2410] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 29 because replica has last-logged OpId of term: 27 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 26 index: 18.
I20260709 12:25:28.802733 28126 raft_consensus.cc:515] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:25:28.843992 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:28.844647 17759 raft_consensus.cc:1240] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 30. Current term is 31. Ops: []
I20260709 12:25:28.832387 28315 raft_consensus.cc:493] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:28.845834 28126 leader_election.cc:290] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 31 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:28.846601 28330 raft_consensus.cc:493] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:28.846856 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:25:28.847178 28330 raft_consensus.cc:515] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:28.847681 17688 raft_consensus.cc:3060] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:28.845830 28315 raft_consensus.cc:515] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:28.848872 17585 catalog_manager.cc:5697] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 21 to 23, leader changed from 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) to e063541542a94969898868bf6eb5ce1b (127.16.29.67). New cstate: current_term: 23 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:25:28.849581 28330 leader_election.cc:290] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:28.850602 17585 catalog_manager.cc:5697] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 20 to 24. New cstate: current_term: 24 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:25:28.851214 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:28.851814 17680 raft_consensus.cc:2468] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 27.
I20260709 12:25:28.824916 28303 raft_consensus.cc:2804] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:25:28.852562 28390 consensus_queue.cc:260] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 25.19, Last appended by leader: 19, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:28.853379 28303 raft_consensus.cc:697] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 28 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:25:28.854399 28390 raft_consensus.cc:3060] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:28.854640 28303 consensus_queue.cc:237] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 27.18, Last appended by leader: 18, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:28.851998 17585 catalog_manager.cc:5697] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 24 to 26, leader changed from 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) to e063541542a94969898868bf6eb5ce1b (127.16.29.67). New cstate: current_term: 26 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:25:28.824769 28159 raft_consensus.cc:2749] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
I20260709 12:25:28.856906 28254 raft_consensus.cc:2749] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:25:28.822068 28271 raft_consensus.cc:2749] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
I20260709 12:25:28.859277 17835 raft_consensus.cc:1240] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 34 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 33. Current term is 34. Ops: []
I20260709 12:25:28.872937 17687 raft_consensus.cc:2468] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 35.
I20260709 12:25:28.851570 28315 leader_election.cc:290] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:28.805053 28308 consensus_queue.cc:1059] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 31.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.027s
I20260709 12:25:28.865504 17688 raft_consensus.cc:2468] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 24.
W20260709 12:25:28.911363 17801 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:28.919397 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 32 candidate_status { last_received { term: 30 index: 21 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:28.833253 28328 raft_consensus.cc:493] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:28.823081 28404 consensus_queue.cc:1059] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 34.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.051s
W20260709 12:25:28.928539 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 125487us
I20260709 12:25:28.931452 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 35 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:28.931723 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 29 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:28.910542 17654 leader_election.cc:304] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
W20260709 12:25:28.942782 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 61472.3us
I20260709 12:25:28.860229 28320 raft_consensus.cc:2749] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
W20260709 12:25:28.961637 17730 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:28.968648 18368 raft_consensus.cc:1275] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 34 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 32 index: 23. Preceding OpId from leader: term: 34 index: 24. (index mismatch)
I20260709 12:25:28.911412 28409 raft_consensus.cc:2804] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:25:28.974376 17835 raft_consensus.cc:1275] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 28 index: 20. Preceding OpId from leader: term: 31 index: 21. (index mismatch)
I20260709 12:25:28.977695 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 31 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:28.933734 28328 raft_consensus.cc:3060] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:28.981628 17829 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 30.
I20260709 12:25:28.933218 28159 raft_consensus.cc:493] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:28.977154 28407 raft_consensus.cc:2804] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:25:28.990684 17762 raft_consensus.cc:2468] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 21.
W20260709 12:25:29.002470 17758 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.566s	user 0.006s	sys 0.050s
I20260709 12:25:28.979616 28409 raft_consensus.cc:493] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:25:29.002982 17758 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.567s	user 0.006s	sys 0.050s
I20260709 12:25:29.009860 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 31 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:29.005505 17585 catalog_manager.cc:5697] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 19 to 21. New cstate: current_term: 21 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:25:28.985351 28054 raft_consensus.cc:493] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:25:29.020080 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
W20260709 12:25:28.590444 17578 rpcz_store.cc:269] 0709 12:24:56.415265 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:04.157450 (+7742185us) service_pool.cc:224] Handling call
0709 12:25:28.534533 (+24377083us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:25:18.159306 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:25:18.160208 (+   902us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:25:18.160274 (+    66us) write_op.cc:435] Acquired schema lock
0709 12:25:18.160309 (+    35us) tablet.cc:662] Decoding operations
0709 12:25:18.163032 (+  2723us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:25:18.163148 (+   116us) write_op.cc:641] Partition lock acquired for write op
0709 12:25:18.163195 (+    47us) tablet.cc:685] Acquiring locks for 20 operations
0709 12:25:18.226503 (+ 63308us) tablet.cc:701] Row locks acquired
0709 12:25:18.226588 (+    85us) write_op.cc:260] PREPARE: finished
0709 12:25:18.226926 (+   338us) write_op.cc:270] Start()
0709 12:25:18.227302 (+   376us) write_op.cc:276] Timestamp: P: 1783599918226851 usec, L: 0
0709 12:25:18.227348 (+    46us) op_driver.cc:348] REPLICATION: starting
0709 12:25:18.232447 (+  5099us) log.cc:844] Serialized 7904 byte log entry
0709 12:25:18.680288 (+447841us) spinlock_profiling.cc:243] Waited 93.7 ms on lock 0x7b4400022b4c. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b47459c85 00007f7b4745a6a2 00007f7b4a1cc1db 00007f7b4a1c7aaf 00007f7b4a1bb86e 00007f7b4a1babb6 00007f7b4a1ba7d8 00007f7b4a1c4221 00007f7b4a1c41d9 00007f7b4a1c4169
0709 12:25:19.413781 (+733493us) op_driver.cc:464] REPLICATION: finished
0709 12:25:19.628155 (+214374us) write_op.cc:301] APPLY: starting
0709 12:25:19.628268 (+   113us) tablet.cc:1367] starting BulkCheckPresence
0709 12:25:20.675174 (+1046906us) tablet.cc:1370] finished BulkCheckPresence
0709 12:25:20.675228 (+    54us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:25:20.793254 (+118026us) spinlock_profiling.cc:243] Waited 29.6 ms on lock 0x7b1800465580. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b47533522 00007f7b475357c1 00007f7b47535779 00007f7b47535709 00007f7b475356d1 00007f7b475349cd 0000000000388634 0000000000388479 00007f7b4751683a
0709 12:25:21.074624 (+281370us) spinlock_profiling.cc:243] Waited 6.13 ms on lock 0x7b1800465580. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b47533557 00007f7b475357c1 00007f7b47535779 00007f7b47535709 00007f7b475356d1 00007f7b475349cd 0000000000388634 0000000000388479 00007f7b4751683a
0709 12:25:22.184327 (+1109703us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:25:22.312081 (+127754us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=40,key_file_lookups=40,delta_file_lookups=0,mrs_lookups=0
0709 12:25:22.329142 (+ 17061us) write_op.cc:312] APPLY: finished
0709 12:25:22.479378 (+150236us) log.cc:844] Serialized 180 byte log entry
0709 12:25:22.666807 (+187429us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:25:22.695685 (+ 28878us) write_op.cc:454] Released schema lock
0709 12:25:22.749106 (+ 53421us) write_op.cc:341] FINISH: Updating metrics
0709 12:25:22.892063 (+142957us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:25:22.892100 (+    37us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":383,"reported_tablets":45,"child_traces":[["op",{"apply.queue_time_us":175676,"apply.run_cpu_time_us":68814,"apply.run_wall_time_us":3215623,"cfile_cache_hit":82,"cfile_cache_hit_bytes":87807,"num_ops":20,"prepare.queue_time_us":163010,"prepare.run_cpu_time_us":20027,"prepare.run_wall_time_us":136793,"raft.queue_time_us":643004,"raft.run_cpu_time_us":12444,"raft.run_wall_time_us":198729,"replication_time_us":1143116,"spinlock_wait_cycles":297585920,"thread_start_us":592897,"threads_started":5,"wal-append.queue_time_us":496817,"wal-append.run_cpu_time_us":37295,"wal-append.run_wall_time_us":2704572}]]}
I20260709 12:25:29.005378 28407 raft_consensus.cc:697] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 27 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:25:29.013471 28409 raft_consensus.cc:3060] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:29.035292 17585 catalog_manager.cc:5697] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 14 to 21, leader changed from 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) to e063541542a94969898868bf6eb5ce1b (127.16.29.67). New cstate: current_term: 21 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:25:29.004520 28404 consensus_queue.cc:1059] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 30.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.032s
I20260709 12:25:29.050287 17831 raft_consensus.cc:3060] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Advancing to term 31
W20260709 12:25:29.052918 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 370700us
I20260709 12:25:29.056834 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:29.064670 17585 catalog_manager.cc:5697] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 25 to 28, leader changed from e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66) to e063541542a94969898868bf6eb5ce1b (127.16.29.67). New cstate: current_term: 28 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:25:29.075486 17688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:29.076493 17688 raft_consensus.cc:2393] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 28: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:25:29.076249 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 34 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:29.077530 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 30 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:25:29.078110 17687 raft_consensus.cc:3060] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:29.079015 17686 raft_consensus.cc:1275] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 33 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 32 index: 19. Preceding OpId from leader: term: 33 index: 20. (index mismatch)
I20260709 12:25:29.080008 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 33 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:29.080346 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 35 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:29.080806 17686 raft_consensus.cc:2393] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 33: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:25:29.081328 17685 raft_consensus.cc:2393] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 35: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:25:29.081663 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:25:29.082242 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 35 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:25:29.082664 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:29.082981 17682 raft_consensus.cc:1240] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 35 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 34. Current term is 35. Ops: [32.18-34.19]
I20260709 12:25:29.083345 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 31 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:29.084832 17683 raft_consensus.cc:1275] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 36 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 35 index: 26. Preceding OpId from leader: term: 36 index: 27. (index mismatch)
I20260709 12:25:29.088793 17688 raft_consensus.cc:1240] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 30 LEADER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 29. Current term is 30. Ops: [29.22-29.22]
I20260709 12:25:29.096208 17685 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 30.
I20260709 12:25:29.082259 17686 raft_consensus.cc:3060] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:25:29.099718 17730 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:29.110858 17728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:29.129154 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:25:29.030570 28159 raft_consensus.cc:515] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:29.148888 17727 leader_election.cc:304] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:29.149209 28425 leader_election.cc:290] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 32 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:29.149686 28320 consensus_queue.cc:1059] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
W20260709 12:25:29.151211 17578 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35962 (request call id 363) took 23241 ms (client timeout 15000 ms). Trace:
W20260709 12:25:29.151635 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 94317.6us
I20260709 12:25:29.153236 28274 raft_consensus.cc:2749] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 32 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:25:29.158258 28315 consensus_queue.cc:1048] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.001s
I20260709 12:25:29.161537 28159 leader_election.cc:290] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:29.163671 17680 raft_consensus.cc:1275] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 28 index: 16. Preceding OpId from leader: term: 30 index: 17. (index mismatch)
I20260709 12:25:29.132738 28309 raft_consensus.cc:3055] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:25:29.056576 28399 raft_consensus.cc:515] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 33 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:29.165512 28404 raft_consensus.cc:493] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:25:29.039502 28054 raft_consensus.cc:515] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:29.167795 17831 raft_consensus.cc:2468] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 31.
W20260709 12:25:29.228894 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 80076.8us
I20260709 12:25:29.227805 28309 raft_consensus.cc:740] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 30 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
W20260709 12:25:29.231072 17578 rpcz_store.cc:269] 0709 12:25:05.910180 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:29.150937 (+23240757us) service_pool.cc:211] Skipping call since client already timed out
0709 12:25:29.151117 (+   180us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260709 12:25:29.231721 17802 leader_election.cc:304] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 34 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:29.231832 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 34 candidate_status { last_received { term: 32 index: 23 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:29.173727 17758 raft_consensus.cc:1240] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 32 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 31. Current term is 32. Ops: []
I20260709 12:25:29.173014 17759 raft_consensus.cc:3060] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Advancing to term 28
W20260709 12:25:29.181522 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:29.235109 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:29.236872 28399 leader_election.cc:290] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 33 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:29.172251 28183 raft_consensus.cc:3055] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 32 LEADER]: Stepping down as leader of term 32
I20260709 12:25:29.229912 17727 leader_election.cc:304] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:29.237558 28308 raft_consensus.cc:2749] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 34 FOLLOWER]: Leader election lost for term 34. Reason: could not achieve majority
I20260709 12:25:29.241673 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:29.187906 17801 leader_election.cc:304] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:29.242977 17835 raft_consensus.cc:2376] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 34 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 34. Re-sending same reply.
I20260709 12:25:29.178964 17585 catalog_manager.cc:5697] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 25 to 26, leader changed from e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66) to 334a4e1f59944544b3b3ff36616b57da (127.16.29.68). New cstate: current_term: 26 leader_uuid: "334a4e1f59944544b3b3ff36616b57da" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 12:25:29.170992 28470 log.cc:927] Time spent T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.143s	user 0.000s	sys 0.000s
I20260709 12:25:29.253772 27431 raft_consensus.cc:3055] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 34 LEADER]: Stepping down as leader of term 34
I20260709 12:25:29.261650 27431 raft_consensus.cc:740] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 34 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:25:29.256311 17687 raft_consensus.cc:2468] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 30.
I20260709 12:25:29.248349 17803 leader_election.cc:304] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:25:29.268005 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 266333us
W20260709 12:25:29.291797 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:29.263772 28313 raft_consensus.cc:2804] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:25:29.301429 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:29.308941 18371 raft_consensus.cc:1240] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 26. Current term is 27. Ops: []
I20260709 12:25:29.300541 17585 catalog_manager.cc:5697] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 20 to 26. New cstate: current_term: 26 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:25:29.241290 28183 raft_consensus.cc:740] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 32 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:25:29.180007 28424 raft_consensus.cc:493] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:29.237573 28404 raft_consensus.cc:515] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:29.307530 18363 raft_consensus.cc:1275] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 30 index: 21. Preceding OpId from leader: term: 31 index: 22. (index mismatch)
I20260709 12:25:29.319069 17687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 34 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:25:29.308991 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 30 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:25:29.314667 17578 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35966 (request call id 361) took 19697 ms (client timeout 15000 ms). Trace:
I20260709 12:25:29.229388 28126 consensus_queue.cc:1048] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 22, Time since last communication: 0.069s
W20260709 12:25:29.320560 17578 rpcz_store.cc:269] 0709 12:25:09.658114 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:29.303552 (+19645438us) service_pool.cc:211] Skipping call since client already timed out
0709 12:25:29.309620 (+  6068us) inbound_call.cc:177] Queueing failure response
Metrics: {}
W20260709 12:25:29.323036 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 97760.2us
I20260709 12:25:29.324723 17802 leader_election.cc:304] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:25:29.293887 28409 raft_consensus.cc:515] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:29.335111 17687 raft_consensus.cc:3060] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:25:29.308624 28313 raft_consensus.cc:493] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:29.314708 28308 raft_consensus.cc:2804] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:25:29.310029 28305 raft_consensus.cc:3055] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 31 LEADER]: Stepping down as leader of term 31
I20260709 12:25:29.309835 28480 raft_consensus.cc:493] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:25:29.367198 18365 raft_consensus.cc:2393] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 30: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:25:29.372922 17652 leader_election.cc:304] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:29.374155 17652 leader_election.cc:304] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:29.365746 28308 raft_consensus.cc:493] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:29.170006 28407 consensus_queue.cc:237] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 25.18, Last appended by leader: 18, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:29.328415 28426 raft_consensus.cc:2804] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:25:29.368350 27431 consensus_queue.cc:260] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 34.20, Last appended by leader: 20, Current term: 34, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:29.379258 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 31 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:29.400553 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 31 candidate_status { last_received { term: 29 index: 22 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:29.379493 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:25:29.345650 28293 raft_consensus.cc:2749] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:25:29.337145 27800 consensus_queue.cc:1048] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 19, Time since last communication: 0.046s
I20260709 12:25:29.386548 17652 leader_election.cc:304] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:29.347664 28313 raft_consensus.cc:3060] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:29.388559 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 31 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:29.389137 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 33 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
W20260709 12:25:29.476926 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 90481us
I20260709 12:25:29.477010 17835 raft_consensus.cc:3060] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:25:29.389720 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 30 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:29.392258 17728 leader_election.cc:304] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:25:29.392011 28480 raft_consensus.cc:515] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:29.392812 17686 raft_consensus.cc:2468] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 27.
I20260709 12:25:29.394706 18374 raft_consensus.cc:1275] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 28 index: 21. Preceding OpId from leader: term: 30 index: 22. (index mismatch)
I20260709 12:25:29.366513 28305 raft_consensus.cc:740] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 31 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:25:29.475580 17687 raft_consensus.cc:2468] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 34.
W20260709 12:25:29.395380 28471 log.cc:927] Time spent T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.057s	user 0.000s	sys 0.000s
I20260709 12:25:29.396126 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 33 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:29.489449 18371 raft_consensus.cc:3060] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:25:29.376294 28308 raft_consensus.cc:3060] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:29.396677 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:29.456812 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:29.461071 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:29.418813 17585 catalog_manager.cc:5697] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 22 to 28, leader changed from e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66) to e063541542a94969898868bf6eb5ce1b (127.16.29.67). New cstate: current_term: 28 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:25:29.476264 27800 consensus_queue.cc:1243] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer e2909cff1da44ad387f96a8ae3aa5afb log is divergent from this leader: its last log entry 30.20 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 19
I20260709 12:25:29.486399 28153 raft_consensus.cc:2749] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:25:29.479504 17652 leader_election.cc:304] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:29.398612 28426 raft_consensus.cc:493] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 34 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:25:29.479765 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 32 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:29.494520 17829 raft_consensus.cc:2468] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 30.
I20260709 12:25:29.400039 28254 consensus_queue.cc:1048] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.080s
I20260709 12:25:29.483198 27431 raft_consensus.cc:3060] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:25:29.504024 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 29 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:29.336045 28424 raft_consensus.cc:515] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:29.474792 28401 consensus_meta.cc:287] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.741s	user 0.009s	sys 0.005s
I20260709 12:25:29.491732 18368 raft_consensus.cc:2393] T a5732e71b9d54f8786b8c511d69943bf P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 30: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:25:29.380236 28159 raft_consensus.cc:2749] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:25:29.493917 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 34 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:29.369931 28399 consensus_queue.cc:1048] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.117s
I20260709 12:25:29.499990 17685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 29 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:29.511210 17680 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 28.
I20260709 12:25:29.506354 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:29.376989 28274 raft_consensus.cc:493] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:29.336321 28309 consensus_queue.cc:260] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 30.17, Last appended by leader: 17, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:25:29.499433 28391 consensus_meta.cc:287] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.768s	user 0.013s	sys 0.002s
I20260709 12:25:29.513041 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:25:29.518715 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:29.519711 18368 raft_consensus.cc:2393] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 28: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
W20260709 12:25:29.503338 28416 consensus_meta.cc:287] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.718s	user 0.010s	sys 0.002s
I20260709 12:25:29.505687 28426 raft_consensus.cc:3060] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:25:29.525396 17585 catalog_manager.cc:5697] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 19 to 24, leader changed from e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66) to 334a4e1f59944544b3b3ff36616b57da (127.16.29.68). New cstate: current_term: 24 leader_uuid: "334a4e1f59944544b3b3ff36616b57da" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:25:29.381774 28247 consensus_queue.cc:1059] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 30.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.134s
I20260709 12:25:29.525681 17685 raft_consensus.cc:2468] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 28.
I20260709 12:25:29.529677 27431 consensus_queue.cc:1059] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 28.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20260709 12:25:29.547784 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 30 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:29.550757 27431 raft_consensus.cc:2749] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260709 12:25:29.552925 27431 raft_consensus.cc:3055] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 31 LEADER]: Stepping down as leader of term 31
I20260709 12:25:29.553355 27431 raft_consensus.cc:740] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 31 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:25:29.554190 17680 raft_consensus.cc:2393] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 30: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:25:29.558355 17689 raft_consensus.cc:1240] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 28. Current term is 29. Ops: []
I20260709 12:25:29.570601 18368 raft_consensus.cc:1275] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 27 index: 21. Preceding OpId from leader: term: 28 index: 22. (index mismatch)
I20260709 12:25:29.577747 17686 raft_consensus.cc:1275] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 33 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 30 index: 20. Preceding OpId from leader: term: 33 index: 22. (index mismatch)
I20260709 12:25:29.532533 28486 raft_consensus.cc:493] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:29.580760 27431 consensus_queue.cc:260] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 31.19, Last appended by leader: 19, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:29.550964 28159 raft_consensus.cc:493] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:29.588102 17686 raft_consensus.cc:1240] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 23. Current term is 24. Ops: []
I20260709 12:25:29.557525 28054 leader_election.cc:290] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:25:29.572597 28390 consensus_meta.cc:287] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.718s	user 0.009s	sys 0.004s
I20260709 12:25:29.573180 28247 raft_consensus.cc:3055] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 31 LEADER]: Stepping down as leader of term 31
I20260709 12:25:29.513943 28401 raft_consensus.cc:515] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:29.574590 28391 raft_consensus.cc:493] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:29.488042 28183 consensus_queue.cc:260] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 32.19, Last appended by leader: 19, Current term: 32, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:29.595129 27431 raft_consensus.cc:3060] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:25:29.577845 28309 raft_consensus.cc:3060] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:29.592994 17835 raft_consensus.cc:2468] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 33.
I20260709 12:25:29.605526 18333 leader_election.cc:304] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:29.606405 28159 raft_consensus.cc:515] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:29.535354 28274 raft_consensus.cc:515] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:29.598874 28247 raft_consensus.cc:740] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 31 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
W20260709 12:25:29.619808 28482 log.cc:927] Time spent T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.221s	user 0.000s	sys 0.000s
I20260709 12:25:29.621698 17689 raft_consensus.cc:1275] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 28 index: 20. Preceding OpId from leader: term: 30 index: 21. (index mismatch)
W20260709 12:25:29.639487 17684 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.844s	user 0.011s	sys 0.026s
I20260709 12:25:29.551122 28489 consensus_queue.cc:1048] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.069s
I20260709 12:25:29.641177 17760 raft_consensus.cc:2468] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 29.
I20260709 12:25:29.637748 17585 catalog_manager.cc:5697] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 22 to 31, leader changed from e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66) to 334a4e1f59944544b3b3ff36616b57da (127.16.29.68). New cstate: current_term: 31 leader_uuid: "334a4e1f59944544b3b3ff36616b57da" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:25:29.626878 28404 leader_election.cc:290] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:25:29.645241 17684 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.850s	user 0.011s	sys 0.026s
I20260709 12:25:29.580617 28485 consensus_queue.cc:1048] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.066s
I20260709 12:25:29.630527 28448 raft_consensus.cc:2749] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:25:29.622931 28112 consensus_queue.cc:1048] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.067s
I20260709 12:25:29.709904 28487 raft_consensus.cc:2749] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:25:29.626937 28314 consensus_queue.cc:1048] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.065s
W20260709 12:25:29.716938 28406 consensus_meta.cc:287] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 1.049s	user 0.013s	sys 0.001s
I20260709 12:25:29.718008 28406 raft_consensus.cc:2749] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:25:29.718887 28406 raft_consensus.cc:493] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:29.719604 28406 raft_consensus.cc:515] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:29.699522 17686 raft_consensus.cc:1275] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 27 index: 21. Preceding OpId from leader: term: 28 index: 22. (index mismatch)
I20260709 12:25:29.726347 28406 leader_election.cc:290] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:29.704860 28401 leader_election.cc:290] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 29 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:29.637374 28409 leader_election.cc:290] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 30 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:25:29.687250 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 81872.3us
I20260709 12:25:29.574987 28305 consensus_queue.cc:260] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 31.21, Last appended by leader: 21, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:29.723490 18371 raft_consensus.cc:2468] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 33.
I20260709 12:25:29.728704 28480 leader_election.cc:290] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:25:29.705746 28477 log.cc:927] Time spent T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.181s	user 0.000s	sys 0.000s
I20260709 12:25:29.704629 28424 leader_election.cc:290] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:25:29.768966 17759 consensus_meta.cc:287] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.530s	user 0.011s	sys 0.000s
I20260709 12:25:29.594040 28486 raft_consensus.cc:515] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:29.766628 17585 catalog_manager.cc:5697] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 21 to 23, leader changed from e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66) to 334a4e1f59944544b3b3ff36616b57da (127.16.29.68). New cstate: current_term: 23 leader_uuid: "334a4e1f59944544b3b3ff36616b57da" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:25:29.696666 28390 raft_consensus.cc:2804] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:25:29.631771 28391 raft_consensus.cc:515] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:29.780377 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:29.769548 28271 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.589s	user 0.012s	sys 0.077s
I20260709 12:25:29.785061 28390 raft_consensus.cc:493] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:29.806715 28390 raft_consensus.cc:3060] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:29.783456 28247 consensus_queue.cc:260] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 31.21, Last appended by leader: 21, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:29.810328 28247 raft_consensus.cc:3060] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:25:29.781795 28305 raft_consensus.cc:3060] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:25:29.785198 18366 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59886 (request call id 2828) took 1083 ms. Trace:
I20260709 12:25:29.815380 18366 rpcz_store.cc:276] 0709 12:25:28.702016 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:28.706491 (+  4475us) service_pool.cc:224] Handling call
0709 12:25:28.706847 (+   356us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:28.707274 (+   427us) raft_consensus.cc:1534] Early marking committed up to index 19
0709 12:25:28.707324 (+    50us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:28.707847 (+   523us) log.cc:844] Serialized 26 byte log entry
0709 12:25:29.128804 (+420957us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:25:29.129767 (+   963us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:29.131111 (+  1344us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:29.784894 (+653783us) raft_consensus.cc:1685] finished
0709 12:25:29.784927 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:29.785117 (+   190us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":3939,"raft.run_cpu_time_us":104,"raft.run_wall_time_us":7740,"thread_start_us":417037,"threads_started":1,"wal-append.queue_time_us":765851}
I20260709 12:25:29.788482 17759 raft_consensus.cc:1275] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 27 index: 14. Preceding OpId from leader: term: 28 index: 15. (index mismatch)
W20260709 12:25:29.816864 28328 consensus_meta.cc:287] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.817s	user 0.008s	sys 0.004s
I20260709 12:25:29.821012 28328 raft_consensus.cc:515] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:29.823171 28328 leader_election.cc:290] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 31 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:25:29.824014 28482 log.cc:927] Time spent T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.058s	user 0.000s	sys 0.000s
I20260709 12:25:29.763556 28314 raft_consensus.cc:493] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:25:29.845316 17585 catalog_manager.cc:5697] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 22 to 27. New cstate: current_term: 27 leader_uuid: "334a4e1f59944544b3b3ff36616b57da" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:25:29.836071 28112 raft_consensus.cc:493] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:29.830698 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:29.874075 17803 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.559s	user 0.019s	sys 0.039s
W20260709 12:25:29.881070 17803 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.567s	user 0.019s	sys 0.040s
W20260709 12:25:29.881705 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 633390us
I20260709 12:25:29.881644 18333 leader_election.cc:304] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:25:29.839290 28271 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.659s	user 0.013s	sys 0.077s
W20260709 12:25:29.848512 17801 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.551s	user 0.024s	sys 0.036s
W20260709 12:25:29.895895 17801 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.602s	user 0.026s	sys 0.037s
W20260709 12:25:29.896387 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 708577us
W20260709 12:25:29.898260 17801 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:29.898738 17801 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:29.832907 28401 raft_consensus.cc:2749] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:25:29.836252 28416 raft_consensus.cc:493] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:29.850116 28183 raft_consensus.cc:3060] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:25:29.916078 27431 raft_consensus.cc:493] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:25:29.918581 27431 raft_consensus.cc:515] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:29.870481 28495 raft_consensus.cc:493] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:29.920961 18333 leader_election.cc:304] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:29.920917 27431 leader_election.cc:290] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:29.788815 28480 consensus_queue.cc:1048] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.110s
W20260709 12:25:29.922226 17803 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:29.829561 28486 leader_election.cc:290] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:29.926792 27431 raft_consensus.cc:493] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:25:29.873761 28112 raft_consensus.cc:515] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:29.921928 28320 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.617s	user 0.003s	sys 0.032s
I20260709 12:25:29.925184 28054 raft_consensus.cc:493] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:25:29.839162 28390 raft_consensus.cc:515] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:29.838191 28313 raft_consensus.cc:515] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:29.936442 17803 leader_election.cc:304] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 31 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:29.928862 17585 catalog_manager.cc:5697] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 26 to 28. New cstate: current_term: 28 leader_uuid: "334a4e1f59944544b3b3ff36616b57da" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 12:25:29.918735 28303 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.531s	user 0.021s	sys 0.015s
W20260709 12:25:29.931051 17802 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.586s	user 0.003s	sys 0.034s
W20260709 12:25:29.980559 17802 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.636s	user 0.003s	sys 0.034s
W20260709 12:25:29.981420 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 656763us
W20260709 12:25:29.932423 17762 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.549s	user 0.019s	sys 0.018s
I20260709 12:25:29.871601 28314 raft_consensus.cc:515] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:29.939647 27431 raft_consensus.cc:515] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:29.959872 28341 consensus_meta.cc:287] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 1.289s	user 0.012s	sys 0.003s
W20260709 12:25:29.950006 28425 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.554s	user 0.003s	sys 0.028s
W20260709 12:25:29.969822 17729 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.784s	user 0.020s	sys 0.031s
W20260709 12:25:29.974891 17654 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.587s	user 0.021s	sys 0.033s
W20260709 12:25:29.993677 17729 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.808s	user 0.020s	sys 0.034s
W20260709 12:25:29.993839 17654 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.606s	user 0.024s	sys 0.035s
I20260709 12:25:29.993979 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 29 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:25:29.994032 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 808473us
W20260709 12:25:29.958765 17727 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.680s	user 0.017s	sys 0.030s
W20260709 12:25:29.948462 28320 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.687s	user 0.004s	sys 0.033s
W20260709 12:25:29.982484 17762 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.599s	user 0.019s	sys 0.018s
I20260709 12:25:29.929591 28416 raft_consensus.cc:515] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:29.927953 28157 raft_consensus.cc:2804] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:25:29.989358 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 35 candidate_status { last_received { term: 34 index: 23 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:25:29.991971 17730 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.569s	user 0.016s	sys 0.010s
W20260709 12:25:29.994354 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 606586us
W20260709 12:25:30.008775 17730 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.585s	user 0.016s	sys 0.010s
I20260709 12:25:30.010115 17802 leader_election.cc:304] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:25:29.994982 17727 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.716s	user 0.017s	sys 0.031s
I20260709 12:25:29.996707 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 29 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:25:30.010888 17730 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:30.013613 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 31 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
W20260709 12:25:30.013972 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 784160us
I20260709 12:25:30.014262 18367 raft_consensus.cc:3060] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:30.014911 17836 raft_consensus.cc:1275] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 20 index: 15. Preceding OpId from leader: term: 23 index: 16. (index mismatch)
W20260709 12:25:30.000859 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 64397.9us
I20260709 12:25:30.001624 17759 raft_consensus.cc:1275] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 30 index: 21. Preceding OpId from leader: term: 31 index: 22. (index mismatch)
W20260709 12:25:29.985603 28303 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.598s	user 0.021s	sys 0.015s
I20260709 12:25:30.002996 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 29 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:30.005097 17835 raft_consensus.cc:2468] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 27.
I20260709 12:25:30.012588 27431 leader_election.cc:290] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:30.007586 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 29 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:30.016103 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 35 candidate_status { last_received { term: 34 index: 23 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:30.017130 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 29 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:29.938141 28495 raft_consensus.cc:515] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:30.028591 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:25:30.029081 17686 raft_consensus.cc:1240] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 26. Current term is 27. Ops: []
I20260709 12:25:30.029335 17682 raft_consensus.cc:2393] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 32: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:25:30.030081 28495 leader_election.cc:290] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:25:29.993468 28425 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.597s	user 0.006s	sys 0.031s
I20260709 12:25:30.020514 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 29 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:30.031199 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:30.024300 27431 raft_consensus.cc:493] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:30.024436 18368 raft_consensus.cc:2468] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 28.
I20260709 12:25:30.027259 28490 raft_consensus.cc:493] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:30.032174 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:30.032945 17757 raft_consensus.cc:2468] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 25.
I20260709 12:25:30.032584 28490 raft_consensus.cc:515] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:30.034279 28498 log.cc:927] Time spent T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.083s	user 0.000s	sys 0.000s
I20260709 12:25:30.027868 17681 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37402 (request call id 4101) took 1171 ms. Trace:
I20260709 12:25:30.032197 27431 raft_consensus.cc:515] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:30.034875 17681 rpcz_store.cc:276] 0709 12:25:28.749765 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:28.752135 (+  2370us) service_pool.cc:224] Handling call
0709 12:25:28.752580 (+   445us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:28.754272 (+  1692us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:28.754307 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:28.754918 (+   611us) log.cc:844] Serialized 26 byte log entry
0709 12:25:29.037952 (+283034us) raft_consensus.cc:1645] Marking committed up to 21
0709 12:25:29.048330 (+ 10378us) log.cc:844] Serialized 14 byte log entry
0709 12:25:29.482143 (+433813us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:29.482222 (+    79us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:29.920495 (+438273us) raft_consensus.cc:1685] finished
0709 12:25:29.920528 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:29.920705 (+   177us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":33003,"prepare.queue_time_us":10231,"prepare.run_cpu_time_us":53,"prepare.run_wall_time_us":61974,"raft.queue_time_us":116781,"raft.run_cpu_time_us":275,"raft.run_wall_time_us":8462,"thread_start_us":649898,"threads_started":2,"wal-append.queue_time_us":556498}
I20260709 12:25:30.038228 28425 raft_consensus.cc:493] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:30.037948 27431 leader_election.cc:290] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:30.039007 27431 raft_consensus.cc:2749] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Leader election lost for term 31. Reason: could not achieve majority
I20260709 12:25:30.039915 27431 consensus_queue.cc:1048] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.000s
I20260709 12:25:30.040167 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:30.040795 17832 raft_consensus.cc:3060] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:30.041138 17830 raft_consensus.cc:1240] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 23. Current term is 24. Ops: []
I20260709 12:25:30.041245 27431 raft_consensus.cc:2749] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
W20260709 12:25:30.027495 28497 log.cc:927] Time spent T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.052s	user 0.000s	sys 0.000s
I20260709 12:25:30.033851 28406 consensus_queue.cc:1059] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:25:30.038720 28425 raft_consensus.cc:515] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:30.043478 17801 leader_election.cc:304] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:30.044878 28425 leader_election.cc:290] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:30.045552 28425 raft_consensus.cc:493] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:25:30.046139 28425 raft_consensus.cc:515] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:30.048607 28496 log.cc:927] Time spent T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.069s	user 0.000s	sys 0.000s
I20260709 12:25:30.049352 17681 raft_consensus.cc:1275] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 26 index: 18. Preceding OpId from leader: term: 27 index: 19. (index mismatch)
I20260709 12:25:30.049403 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 26 candidate_status { last_received { term: 25 index: 22 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:30.035175 28490 leader_election.cc:290] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:30.051329 17682 raft_consensus.cc:2393] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 26: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:25:30.050889 17654 leader_election.cc:304] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:25:30.050310 28491 consensus_queue.cc:1048] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.003s
I20260709 12:25:30.053821 28490 raft_consensus.cc:2804] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 33 FOLLOWER]: Leader election won for term 33
I20260709 12:25:30.054643 28490 raft_consensus.cc:697] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 33 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:25:30.055723 28490 consensus_queue.cc:237] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 31.18, Last appended by leader: 18, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:30.057943 17756 raft_consensus.cc:1275] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 25 index: 16. Preceding OpId from leader: term: 27 index: 17. (index mismatch)
I20260709 12:25:30.028044 17761 raft_consensus.cc:2468] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 28.
I20260709 12:25:30.035501 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:30.063169 17755 raft_consensus.cc:2468] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 28.
I20260709 12:25:30.080916 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 32 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:30.081545 28500 raft_consensus.cc:3055] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:25:30.082043 28500 raft_consensus.cc:740] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:25:30.082521 28313 leader_election.cc:290] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 31 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:30.081676 17756 raft_consensus.cc:2468] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 31.
I20260709 12:25:30.082906 28500 consensus_queue.cc:260] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 26.19, Last appended by leader: 19, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:30.084033 28303 consensus_queue.cc:1048] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
W20260709 12:25:30.085157 28499 log.cc:927] Time spent T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.074s	user 0.000s	sys 0.000s
I20260709 12:25:30.085628 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:30.101732 17763 raft_consensus.cc:3060] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:30.027259 28159 leader_election.cc:290] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:30.007498 28157 raft_consensus.cc:493] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:29.998162 28308 raft_consensus.cc:515] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:30.002629 28112 leader_election.cc:290] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:30.092284 28305 raft_consensus.cc:2804] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:25:30.093199 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 31 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:30.094405 28416 leader_election.cc:290] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:30.094564 17801 leader_election.cc:304] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:30.008874 28404 consensus_queue.cc:1048] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.093s
I20260709 12:25:29.969099 28054 raft_consensus.cc:515] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:30.096751 17832 raft_consensus.cc:2468] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 24.
I20260709 12:25:30.098193 18367 raft_consensus.cc:2468] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 31.
I20260709 12:25:29.999066 28341 raft_consensus.cc:515] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:30.066829 28320 raft_consensus.cc:493] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:30.123100 17728 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.625s	user 0.014s	sys 0.027s
W20260709 12:25:30.115545 17652 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.613s	user 0.032s	sys 0.020s
I20260709 12:25:30.119099 28390 leader_election.cc:290] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 29 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:30.116925 17757 raft_consensus.cc:2468] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 30.
W20260709 12:25:30.127297 28148 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.559s	user 0.005s	sys 0.018s
I20260709 12:25:30.115224 28157 raft_consensus.cc:3060] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:30.119762 28500 raft_consensus.cc:3060] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:25:30.129323 17652 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.627s	user 0.032s	sys 0.020s
I20260709 12:25:30.104465 28480 raft_consensus.cc:493] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
W20260709 12:25:30.141149 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 661698us
I20260709 12:25:30.141053 28480 raft_consensus.cc:515] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:30.141965 17652 leader_election.cc:304] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:30.142783 28515 raft_consensus.cc:2749] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
W20260709 12:25:30.128904 17728 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.631s	user 0.014s	sys 0.027s
I20260709 12:25:30.116532 28305 raft_consensus.cc:493] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:25:30.144457 17652 leader_election.cc:304] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:30.140058 28054 leader_election.cc:290] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:25:30.140098 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 123540us
I20260709 12:25:30.149572 18334 leader_election.cc:304] T a5732e71b9d54f8786b8c511d69943bf P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:30.150345 28390 raft_consensus.cc:2749] T a5732e71b9d54f8786b8c511d69943bf P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
I20260709 12:25:30.151542 28390 consensus_queue.cc:1059] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:25:30.124704 28391 leader_election.cc:290] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:30.125183 28416 raft_consensus.cc:493] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:25:30.148254 17763 raft_consensus.cc:2468] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 24.
I20260709 12:25:30.129807 28501 raft_consensus.cc:2804] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Leader pre-election won for term 28
W20260709 12:25:30.129995 28148 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.731s	user 0.005s	sys 0.019s
I20260709 12:25:30.143215 28480 leader_election.cc:290] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:30.154906 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:30.164350 17755 raft_consensus.cc:2468] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 28.
I20260709 12:25:30.152030 28112 consensus_queue.cc:1048] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.005s
W20260709 12:25:30.149565 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 757362us
I20260709 12:25:30.155441 17652 leader_election.cc:304] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:30.148052 28305 raft_consensus.cc:3060] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:30.156658 28390 raft_consensus.cc:3055] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:25:30.156891 28159 raft_consensus.cc:2749] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
W20260709 12:25:30.168152 17727 leader_election.cc:336] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 31 election: RPC error from VoteRequest() call to peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837): Timed out: RequestConsensusVote RPC to 127.16.29.67:36837 timed out after 1.521s (SENT)
I20260709 12:25:30.185878 17727 leader_election.cc:304] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 31 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:30.182816 17728 leader_election.cc:304] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
W20260709 12:25:30.216488 17683 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.612s	user 0.019s	sys 0.032s
W20260709 12:25:30.217978 17683 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.628s	user 0.019s	sys 0.032s
I20260709 12:25:30.129703 28502 consensus_queue.cc:1048] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.048s
I20260709 12:25:30.240172 17681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 29 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:25:30.240908 17681 raft_consensus.cc:3060] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:30.165910 28501 raft_consensus.cc:493] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:25:30.244011 17686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 29 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:30.136759 28320 raft_consensus.cc:515] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:30.247514 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 29 candidate_status { last_received { term: 27 index: 21 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
W20260709 12:25:30.247761 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 65024.3us
I20260709 12:25:30.251138 17682 raft_consensus.cc:2468] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 28.
I20260709 12:25:30.182197 28390 raft_consensus.cc:740] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:25:30.257174 17728 leader_election.cc:304] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:25:30.257436 17686 raft_consensus.cc:2468] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 28.
I20260709 12:25:30.211074 28515 raft_consensus.cc:2804] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:25:30.166934 28391 consensus_queue.cc:1059] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 26.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.081s
I20260709 12:25:30.262099 18334 leader_election.cc:304] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:25:30.163355 28416 raft_consensus.cc:515] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:30.267585 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 99466.1us
I20260709 12:25:30.271584 17683 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37396 (request call id 3599) took 1192 ms. Trace:
I20260709 12:25:30.259678 28515 raft_consensus.cc:493] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 35 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:30.248178 28404 raft_consensus.cc:493] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:25:30.211532 28480 consensus_queue.cc:1048] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.049s
W20260709 12:25:30.273154 28309 consensus_meta.cc:287] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.629s	user 0.008s	sys 0.003s
I20260709 12:25:30.279851 18334 leader_election.cc:304] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:25:30.280869 28126 raft_consensus.cc:493] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:25:30.290601 28401 consensus_queue.cc:1048] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.001s
W20260709 12:25:30.291390 17804 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.747s	user 0.008s	sys 0.024s
W20260709 12:25:30.312568 17804 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.913s	user 0.009s	sys 0.024s
W20260709 12:25:30.292384 28508 log.cc:927] Time spent T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.126s	user 0.000s	sys 0.000s
I20260709 12:25:30.294471 28406 raft_consensus.cc:493] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:25:30.297186 28159 raft_consensus.cc:493] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:30.298040 28112 raft_consensus.cc:493] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:30.300417 28308 leader_election.cc:290] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 22 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:25:30.284081 28503 raft_consensus.cc:493] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:30.331787 17682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 31 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:30.332543 17682 raft_consensus.cc:2468] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 30.
I20260709 12:25:30.302183 28330 raft_consensus.cc:3060] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:25:30.267904 28506 raft_consensus.cc:2749] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:25:30.247264 28501 raft_consensus.cc:3060] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:30.285418 28514 raft_consensus.cc:2749] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Leader election lost for term 31. Reason: could not achieve majority
I20260709 12:25:30.275513 17683 rpcz_store.cc:276] 0709 12:25:29.078783 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:29.085765 (+  6982us) service_pool.cc:224] Handling call
0709 12:25:29.086127 (+   362us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:29.086478 (+   351us) raft_consensus.cc:1534] Early marking committed up to index 20
0709 12:25:29.532583 (+446105us) log.cc:844] Serialized 14 byte log entry
0709 12:25:30.270346 (+737763us) log.cc:844] Serialized 14 byte log entry
0709 12:25:30.270768 (+   422us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:30.270898 (+   130us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:25:30.271024 (+   126us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:30.271221 (+   197us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:30.271494 (+   273us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":39476,"prepare.queue_time_us":83314,"prepare.run_cpu_time_us":30,"prepare.run_wall_time_us":30,"raft.queue_time_us":35831,"raft.run_cpu_time_us":118,"raft.run_wall_time_us":117,"thread_start_us":1074398,"threads_started":2}
W20260709 12:25:30.321105 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 311054us
W20260709 12:25:30.320420 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 409747us
I20260709 12:25:30.283790 28515 raft_consensus.cc:3060] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:25:30.301548 28274 leader_election.cc:290] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:25:30.315580 28404 raft_consensus.cc:515] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:30.263630 28314 leader_election.cc:290] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:30.353657 18334 leader_election.cc:304] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:25:30.354806 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 30 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:30.362840 17802 leader_election.cc:304] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:25:30.330626 28391 raft_consensus.cc:2804] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:25:30.367659 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:30.388960 18368 raft_consensus.cc:2468] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 27.
I20260709 12:25:30.390539 17802 leader_election.cc:304] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 35 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:30.391642 17802 leader_election.cc:304] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:25:30.412608 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 37 candidate_status { last_received { term: 35 index: 26 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:30.413949 18367 raft_consensus.cc:2410] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 36 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for term 37 because replica has last-logged OpId of term: 36 index: 27, which is greater than that of the candidate, which has last-logged OpId of term: 35 index: 26.
I20260709 12:25:30.324432 28126 raft_consensus.cc:515] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:30.329377 28406 raft_consensus.cc:515] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:30.417830 17802 leader_election.cc:304] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:30.416981 18363 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 3032) took 1023 ms. Trace:
I20260709 12:25:30.331856 28159 raft_consensus.cc:515] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:30.425437 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:30.426118 18368 raft_consensus.cc:2468] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 27.
I20260709 12:25:30.426227 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 29 candidate_status { last_received { term: 27 index: 21 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:30.390544 28148 raft_consensus.cc:2804] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:25:30.428248 28148 raft_consensus.cc:493] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:25:30.343995 28112 raft_consensus.cc:515] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:30.388195 28509 raft_consensus.cc:493] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:30.391340 28341 leader_election.cc:290] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 27 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:25:30.432130 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 63304.7us
I20260709 12:25:30.433970 28148 raft_consensus.cc:3060] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:30.421175 18363 rpcz_store.cc:276] 0709 12:25:29.393429 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:29.395133 (+  1704us) service_pool.cc:224] Handling call
0709 12:25:29.398569 (+  3436us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:29.398827 (+   258us) raft_consensus.cc:1534] Early marking committed up to index 21
0709 12:25:29.976145 (+577318us) log.cc:844] Serialized 14 byte log entry
0709 12:25:30.415363 (+439218us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:30.415424 (+    61us) raft_consensus.cc:1645] Marking committed up to 21
0709 12:25:30.415482 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:30.415572 (+    90us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:30.416917 (+  1345us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":54113,"prepare.queue_time_us":83293,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":38,"raft.queue_time_us":17722,"thread_start_us":902547,"threads_started":2}
I20260709 12:25:30.433039 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 29 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:30.415115 28314 raft_consensus.cc:2804] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:25:30.477309 28308 raft_consensus.cc:2804] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Leader pre-election won for term 28
W20260709 12:25:30.435688 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 416857us
I20260709 12:25:30.431749 28054 raft_consensus.cc:2804] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 34 FOLLOWER]: Leader election won for term 34
I20260709 12:25:30.441530 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:30.441609 17689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 31 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:30.420787 28495 raft_consensus.cc:2804] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:25:30.416981 28309 raft_consensus.cc:2804] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 35 FOLLOWER]: Leader election won for term 35
I20260709 12:25:30.480537 17683 raft_consensus.cc:1275] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 26 index: 19. Preceding OpId from leader: term: 27 index: 20. (index mismatch)
I20260709 12:25:30.348959 28503 raft_consensus.cc:515] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:30.485404 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 32 candidate_status { last_received { term: 30 index: 21 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:30.352859 28390 consensus_queue.cc:260] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 23.17, Last appended by leader: 17, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:30.425344 28247 raft_consensus.cc:2804] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:25:30.481261 17684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3" is_pre_election: true
I20260709 12:25:30.484798 17680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "0da169ff82674dc1947055ce43de68a3"
I20260709 12:25:30.517608 18333 leader_election.cc:304] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:30.519017 18333 leader_election.cc:304] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:30.431963 28303 consensus_queue.cc:1059] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 28.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.064s
I20260709 12:25:30.490432 28271 raft_consensus.cc:493] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 34 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:25:30.492187 17834 raft_consensus.cc:1275] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 34 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 32 index: 23. Preceding OpId from leader: term: 34 index: 24. (index mismatch)
I20260709 12:25:30.493213 17837 raft_consensus.cc:1275] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 28 index: 16. Preceding OpId from leader: term: 30 index: 17. (index mismatch)
W20260709 12:25:30.493255 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 236178us
I20260709 12:25:30.494432 17832 raft_consensus.cc:2410] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for term 29 because replica has last-logged OpId of term: 28 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 18.
I20260709 12:25:30.432237 18367 raft_consensus.cc:1240] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 30. Current term is 31. Ops: [30.22-30.22]
I20260709 12:25:30.389844 28391 raft_consensus.cc:493] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:30.498874 17803 leader_election.cc:304] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:25:30.479693 28274 raft_consensus.cc:493] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:30.488510 28409 raft_consensus.cc:493] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:30.498579 28426 consensus_meta.cc:287] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.941s	user 0.009s	sys 0.004s
I20260709 12:25:30.508466 17729 leader_election.cc:304] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:30.510927 28425 leader_election.cc:290] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:30.512398 17689 raft_consensus.cc:2468] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 30.
I20260709 12:25:30.539125 17729 leader_election.cc:304] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:30.417351 28486 raft_consensus.cc:493] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:30.523825 17728 leader_election.cc:304] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:30.497825 28314 raft_consensus.cc:493] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:30.525286 28391 raft_consensus.cc:3060] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:30.500367 28522 raft_consensus.cc:493] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:30.539381 17681 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 29.
I20260709 12:25:30.552767 17682 raft_consensus.cc:1240] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 28. Current term is 29. Ops: []
I20260709 12:25:30.546427 17836 raft_consensus.cc:2468] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 31.
I20260709 12:25:30.505394 28416 leader_election.cc:290] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:25:30.544132 28274 raft_consensus.cc:515] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:30.548695 28426 raft_consensus.cc:515] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 35 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:30.561560 28159 leader_election.cc:290] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:30.561908 28274 leader_election.cc:290] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:25:30.562270 28406 leader_election.cc:290] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:30.538575 28271 raft_consensus.cc:515] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:30.562515 28274 raft_consensus.cc:2804] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:25:30.563040 28274 raft_consensus.cc:697] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 24 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:25:30.564213 28271 leader_election.cc:290] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:30.565405 28112 leader_election.cc:290] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:30.565970 28112 raft_consensus.cc:493] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:25:30.566188 28157 raft_consensus.cc:515] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:30.566476 28112 raft_consensus.cc:515] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:30.556665 28314 raft_consensus.cc:3060] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:30.567998 28522 raft_consensus.cc:515] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:30.568897 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 29 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:30.571389 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 35 candidate_status { last_received { term: 32 index: 23 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:30.572294 18371 raft_consensus.cc:2368] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for earlier term 29. Current term is 30.
I20260709 12:25:30.578809 17829 raft_consensus.cc:2393] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 35: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
W20260709 12:25:30.548465 28407 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.760s	user 0.042s	sys 0.029s
I20260709 12:25:30.535843 28390 raft_consensus.cc:3060] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:25:30.605088 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:30.611825 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 31 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:30.612558 17763 raft_consensus.cc:2468] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 30.
I20260709 12:25:30.613969 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 29 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:30.520090 28519 consensus_queue.cc:1059] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 26.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.032s
I20260709 12:25:30.614935 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 29 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:30.544564 28516 raft_consensus.cc:493] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:25:30.496043 28509 raft_consensus.cc:515] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:30.618371 17764 raft_consensus.cc:2410] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for term 29 because replica has last-logged OpId of term: 27 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 26 index: 17.
I20260709 12:25:30.619032 17652 leader_election.cc:304] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:30.548192 28341 raft_consensus.cc:2804] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Leader pre-election won for term 29
W20260709 12:25:30.582669 17730 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:30.627912 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 32 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:30.631121 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 31 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:30.635063 17835 raft_consensus.cc:2468] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 32.
W20260709 12:25:30.609047 28407 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.825s	user 0.043s	sys 0.030s
I20260709 12:25:30.686832 17758 raft_consensus.cc:2410] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for term 29 because replica has last-logged OpId of term: 27 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 17.
I20260709 12:25:30.551450 28526 raft_consensus.cc:2804] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:25:30.557255 28425 raft_consensus.cc:2804] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:25:30.543919 28409 raft_consensus.cc:515] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:30.703584 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:30.617254 28491 raft_consensus.cc:493] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:30.708359 17688 raft_consensus.cc:1240] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 27 LEADER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 26. Current term is 27. Ops: []
I20260709 12:25:30.702847 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 30 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:30.712148 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 31 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:30.614534 28159 raft_consensus.cc:493] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:25:30.715709 17830 raft_consensus.cc:3060] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:30.715097 17759 raft_consensus.cc:3060] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:30.621734 28523 raft_consensus.cc:2804] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:25:30.557585 28486 raft_consensus.cc:515] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:30.685652 28416 raft_consensus.cc:493] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:30.725625 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:30.685222 28487 raft_consensus.cc:2749] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:25:30.689137 28526 raft_consensus.cc:493] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:30.726291 17764 raft_consensus.cc:2468] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 27.
I20260709 12:25:30.607543 28521 consensus_queue.cc:1048] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.169s
W20260709 12:25:30.728649 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 204888us
I20260709 12:25:30.753436 17728 leader_election.cc:304] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 35 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:30.731383 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 37 candidate_status { last_received { term: 35 index: 26 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:30.691730 28341 raft_consensus.cc:493] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:30.740113 28416 raft_consensus.cc:515] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:30.742002 28526 raft_consensus.cc:3060] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:30.733477 28112 leader_election.cc:290] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:25:30.733815 28183 raft_consensus.cc:3055] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:25:30.685494 28502 consensus_queue.cc:1048] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.042s
W20260709 12:25:30.758973 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 260185us
I20260709 12:25:30.711854 28530 raft_consensus.cc:3060] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:25:30.749415 17684 raft_consensus.cc:2393] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 25: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:25:30.716367 28519 raft_consensus.cc:493] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:30.686313 28516 raft_consensus.cc:515] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:30.726801 28157 leader_election.cc:290] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 30 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:25:30.723062 28425 raft_consensus.cc:697] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 31 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
W20260709 12:25:30.850637 28500 consensus_meta.cc:287] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.710s	user 0.003s	sys 0.008s
I20260709 12:25:30.785933 17759 raft_consensus.cc:2468] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 30.
I20260709 12:25:30.820214 17764 pending_rounds.cc:85] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb: Aborting all ops after (but not including) 19
I20260709 12:25:30.851869 17764 pending_rounds.cc:107] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb: Aborting uncommitted NO_OP operation due to leader change: 30.20
I20260709 12:25:30.852276 17764 raft_consensus.cc:2889] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 33 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:30.750840 28523 raft_consensus.cc:697] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 29 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:25:30.628703 28274 consensus_queue.cc:237] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 22.16, Last appended by leader: 16, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:30.780055 28514 raft_consensus.cc:2749] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 35 FOLLOWER]: Leader election lost for term 35. Reason: could not achieve majority
I20260709 12:25:30.771507 28524 raft_consensus.cc:493] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:30.786618 28183 raft_consensus.cc:740] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:25:30.848275 28487 raft_consensus.cc:493] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:30.850049 28425 consensus_queue.cc:237] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 28.21, Last appended by leader: 22, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:30.770380 28309 raft_consensus.cc:697] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 35 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:25:30.759034 28341 raft_consensus.cc:3060] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:30.852360 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:30.854090 17728 leader_election.cc:304] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:30.783167 28112 raft_consensus.cc:493] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:30.780398 28054 raft_consensus.cc:697] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 34 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:25:30.819113 28536 raft_consensus.cc:493] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:30.819077 28247 raft_consensus.cc:697] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 24 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:25:30.759358 28491 raft_consensus.cc:515] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:30.895998 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 35 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:30.914973 17759 raft_consensus.cc:2393] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 35: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
W20260709 12:25:30.901386 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 66831.4us
I20260709 12:25:30.916124 17838 raft_consensus.cc:2393] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 27: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:25:30.827098 28313 consensus_queue.cc:1048] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.053s
I20260709 12:25:30.783178 28159 raft_consensus.cc:515] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:25:30.910198 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 56215.5us
W20260709 12:25:30.910761 18334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:30.872044 28522 leader_election.cc:290] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:30.912310 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 34 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:25:30.889324 28520 log.cc:927] Time spent T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.131s	user 0.000s	sys 0.000s
I20260709 12:25:30.915827 17803 leader_election.cc:304] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:30.834100 28529 consensus_queue.cc:1059] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.087s
I20260709 12:25:30.929382 17803 leader_election.cc:304] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:30.930482 17728 leader_election.cc:304] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:25:30.883756 17652 leader_election.cc:304] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:30.835242 28519 raft_consensus.cc:515] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:30.939416 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 55813.7us
I20260709 12:25:30.949338 17803 leader_election.cc:304] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:30.921795 28500 raft_consensus.cc:2749] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
I20260709 12:25:30.944382 28401 raft_consensus.cc:2804] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:25:30.958313 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 35 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:30.958971 17838 raft_consensus.cc:2468] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 35.
I20260709 12:25:30.929033 28407 raft_consensus.cc:493] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:25:30.969769 28159 leader_election.cc:290] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:30.970404 28159 raft_consensus.cc:493] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:30.970903 28159 raft_consensus.cc:515] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:30.971163 28553 raft_consensus.cc:2804] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:25:30.907831 28487 raft_consensus.cc:515] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:30.891223 28524 raft_consensus.cc:515] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:30.913303 28112 raft_consensus.cc:515] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:30.944460 28486 leader_election.cc:290] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:30.990127 17830 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 31.
I20260709 12:25:30.976274 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 29 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:30.973992 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:30.977348 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:31.027267 17758 raft_consensus.cc:2468] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 25.
I20260709 12:25:31.006497 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:30.975447 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:25:30.978900 17727 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:30.984934 28515 consensus_meta.cc:287] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.598s	user 0.010s	sys 0.000s
W20260709 12:25:30.983613 28528 log.cc:927] Time spent T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.215s	user 0.000s	sys 0.000s
W20260709 12:25:30.986189 28330 consensus_meta.cc:287] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.633s	user 0.008s	sys 0.003s
I20260709 12:25:31.036546 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:30.957456 28542 raft_consensus.cc:2749] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
I20260709 12:25:30.949383 28551 raft_consensus.cc:2749] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 12:25:30.996905 28416 leader_election.cc:290] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:30.992455 28550 raft_consensus.cc:493] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
W20260709 12:25:31.004246 28305 consensus_meta.cc:287] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.763s	user 0.012s	sys 0.000s
I20260709 12:25:30.971275 28308 raft_consensus.cc:493] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:25:30.971997 28553 raft_consensus.cc:493] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:30.970641 28522 raft_consensus.cc:2749] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:25:31.028836 28509 leader_election.cc:290] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
/apache/dev/git/kudu/src/kudu/master/auto_leader_rebalancer-test.cc:461: Failure
Failed
Bad status: Illegal state: leader_transfer_task size should be 0, but 5
W20260709 12:25:31.046998 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 106377us
I20260709 12:25:30.950912 28183 consensus_queue.cc:260] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 26.17, Last appended by leader: 17, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:31.046016 28409 leader_election.cc:290] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:31.045270 17652 leader_election.cc:304] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:30.997042 28159 leader_election.cc:290] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:30.921377 28536 raft_consensus.cc:515] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:30.977885 28491 leader_election.cc:290] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:30.945678 28504 consensus_queue.cc:1048] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 20, Time since last communication: 0.083s
W20260709 12:25:30.993417 28534 log.cc:927] Time spent T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.093s	user 0.000s	sys 0.000s
I20260709 12:25:31.038815 28391 raft_consensus.cc:515] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:31.057401 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:31.097664 28308 raft_consensus.cc:3060] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:30.984624 28126 leader_election.cc:290] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:30.971712 28315 raft_consensus.cc:3055] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 33 LEADER]: Stepping down as leader of term 33
I20260709 12:25:31.106678 28553 raft_consensus.cc:3060] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:31.076161 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:25:31.060639 17727 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:31.090410 17757 raft_consensus.cc:2468] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 31.
I20260709 12:25:31.021004 28538 raft_consensus.cc:493] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 34 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:25:31.093890 28487 leader_election.cc:290] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:31.058386 28406 consensus_queue.cc:1048] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.038s
I20260709 12:25:31.058386 28247 consensus_queue.cc:237] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 23.17, Last appended by leader: 17, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:31.041661 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 29 candidate_status { last_received { term: 27 index: 21 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:31.206323 17830 raft_consensus.cc:2410] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for term 29 because replica has last-logged OpId of term: 28 index: 22, which is greater than that of the candidate, which has last-logged OpId of term: 27 index: 21.
I20260709 12:25:30.973011 28303 consensus_queue.cc:1048] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.086s
I20260709 12:25:31.100176 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 31 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:31.109748 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 31 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:31.207835 18361 raft_consensus.cc:3060] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:31.054112 28330 raft_consensus.cc:2749] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 33 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:25:31.054787 28401 raft_consensus.cc:697] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:25:31.114048 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:30.996181 28523 consensus_queue.cc:237] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 26.18, Last appended by leader: 18, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:31.122473 28527 log.cc:927] Time spent T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.256s	user 0.000s	sys 0.001s
I20260709 12:25:31.123471 28524 leader_election.cc:290] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:25:31.124563 17831 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37994 (request call id 1978) took 1146 ms. Trace:
W20260709 12:25:31.125604 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 75078.1us
W20260709 12:25:31.126065 17802 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.591s	user 0.009s	sys 0.021s
I20260709 12:25:31.127970 28536 leader_election.cc:290] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:31.128468 27431 raft_consensus.cc:3055] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:25:31.239264 17831 rpcz_store.cc:276] 0709 12:25:29.977794 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:29.977919 (+   125us) service_pool.cc:224] Handling call
0709 12:25:29.978333 (+   414us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:30.004042 (+ 25709us) raft_consensus.cc:1534] Early marking committed up to index 17
0709 12:25:30.004095 (+    53us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:30.019138 (+ 15043us) log.cc:844] Serialized 26 byte log entry
0709 12:25:30.435086 (+415948us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:25:30.435127 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:30.437055 (+  1928us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:30.982978 (+545923us) spinlock_profiling.cc:243] Waited 4.17 ms on lock 0x7b4400022b4c. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b47459c85 00007f7b4745a6a2 00007f7b4a1cc1db 00007f7b4a1c7aaf 00007f7b4a1bb86e 00007f7b4a1babb6 00007f7b4a1ba7d8 00007f7b4a1c4221 00007f7b4a1c41d9 00007f7b4a1c4169
0709 12:25:31.124270 (+141292us) raft_consensus.cc:1685] finished
0709 12:25:31.124290 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:31.124503 (+   213us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":17927,"raft.queue_time_us":53670,"raft.run_cpu_time_us":111,"raft.run_wall_time_us":111,"spinlock_wait_cycles":9583488,"thread_start_us":340149,"threads_started":1,"wal-append.queue_time_us":548598}
I20260709 12:25:31.247865 27431 raft_consensus.cc:740] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 28 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:25:31.248873 27431 consensus_queue.cc:260] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 28.20, Last appended by leader: 20, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:31.128791 16501 tablet_server.cc:179] TabletServer@127.16.29.65:0 shutting down...
W20260709 12:25:31.129376 28546 log.cc:927] Time spent T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.076s	user 0.000s	sys 0.001s
I20260709 12:25:31.130288 28341 raft_consensus.cc:515] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:31.121707 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 76637us
I20260709 12:25:30.986085 28426 leader_election.cc:290] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 35 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:30.938585 28514 consensus_queue.cc:1048] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.097s
I20260709 12:25:30.988581 28320 leader_election.cc:290] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:31.107282 28500 raft_consensus.cc:493] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:31.261236 28500 raft_consensus.cc:515] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:31.264101 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 115912us
I20260709 12:25:31.264580 28561 raft_consensus.cc:493] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:31.264869 17727 leader_election.cc:304] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:31.107717 28550 raft_consensus.cc:515] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:31.027590 28407 raft_consensus.cc:515] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:31.133745 28404 leader_election.cc:290] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:25:31.027827 28054 consensus_queue.cc:237] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 32.19, Last appended by leader: 19, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:31.097548 28305 raft_consensus.cc:515] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:31.232338 18362 raft_consensus.cc:2393] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 27: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
W20260709 12:25:31.150230 28314 consensus_meta.cc:287] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.582s	user 0.008s	sys 0.004s
W20260709 12:25:31.240458 17802 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.705s	user 0.009s	sys 0.021s
W20260709 12:25:31.365814 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 839604us
W20260709 12:25:31.238272 18333 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.625s	user 0.003s	sys 0.015s
I20260709 12:25:31.074580 28309 consensus_queue.cc:237] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 34.19, Last appended by leader: 19, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:31.247257 17654 leader_election.cc:304] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:25:31.141978 28315 raft_consensus.cc:740] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 33 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
W20260709 12:25:31.385640 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 138449us
I20260709 12:25:31.254338 17652 leader_election.cc:304] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:31.254755 17833 raft_consensus.cc:1240] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 35 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 34. Current term is 35. Ops: []
I20260709 12:25:31.255716 17838 raft_consensus.cc:1240] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 32 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 31. Current term is 32. Ops: []
I20260709 12:25:31.054584 28515 raft_consensus.cc:515] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 36 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:25:31.263589 28531 log.cc:927] Time spent T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.337s	user 0.001s	sys 0.001s
I20260709 12:25:31.266882 17829 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37994 (request call id 1975) took 1300 ms. Trace:
I20260709 12:25:31.268973 27431 raft_consensus.cc:3060] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Advancing to term 29
W20260709 12:25:31.272063 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:31.288273 17763 raft_consensus.cc:1240] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 31 LEADER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 30. Current term is 31. Ops: [30.22-30.22]
I20260709 12:25:31.230712 28557 raft_consensus.cc:493] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 36 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:25:31.203826 28554 raft_consensus.cc:2804] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Leader pre-election won for term 28
W20260709 12:25:31.356997 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 92201.6us
W20260709 12:25:31.370985 18333 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.758s	user 0.004s	sys 0.015s
I20260709 12:25:31.253568 28524 raft_consensus.cc:493] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:25:31.263820 28500 leader_election.cc:290] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:25:31.384976 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 247052us
W20260709 12:25:31.393937 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 139688us
I20260709 12:25:31.203718 28538 raft_consensus.cc:515] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:25:31.448427 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 835413us
I20260709 12:25:31.393638 28487 raft_consensus.cc:2804] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:25:31.358441 28529 consensus_queue.cc:1243] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer 0da169ff82674dc1947055ce43de68a3 log is divergent from this leader: its last log entry 31.18 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 17
I20260709 12:25:31.453994 17652 leader_election.cc:304] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:25:31.446377 17835 raft_consensus.cc:2468] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 25.
W20260709 12:25:31.448524 28546 log.cc:927] Time spent T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.079s	user 0.000s	sys 0.000s
I20260709 12:25:31.450618 17727 leader_election.cc:304] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:31.461735 17838 raft_consensus.cc:1240] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 21. Current term is 22. Ops: []
I20260709 12:25:31.458930 18333 leader_election.cc:304] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:31.460635 17829 rpcz_store.cc:276] 0709 12:25:29.953961 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:29.972200 (+ 18239us) service_pool.cc:224] Handling call
0709 12:25:31.254290 (+1282090us) spinlock_profiling.cc:243] Waited 564 ms on lock 0x7b54000893d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:25:31.254453 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1296251264}
I20260709 12:25:31.386574 28516 leader_election.cc:290] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:25:31.511366 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 60846.7us
I20260709 12:25:31.517565 17727 leader_election.cc:304] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:31.494074 17652 leader_election.cc:304] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:25:31.525563 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 67092.9us
I20260709 12:25:31.528728 18367 raft_consensus.cc:1240] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 28. Current term is 29. Ops: []
I20260709 12:25:31.390985 28404 raft_consensus.cc:493] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:25:31.454178 28509 raft_consensus.cc:2749] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:25:31.539856 17727 leader_election.cc:304] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:25:31.385574 28490 raft_consensus.cc:493] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:31.460683 28491 raft_consensus.cc:493] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:31.543800 18367 raft_consensus.cc:1275] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 26 index: 17. Preceding OpId from leader: term: 27 index: 17. (term mismatch)
I20260709 12:25:31.456482 28504 raft_consensus.cc:493] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:31.551277 18367 pending_rounds.cc:85] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da: Aborting all ops after (but not including) 16
I20260709 12:25:31.459841 28159 raft_consensus.cc:493] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:25:31.469764 28487 raft_consensus.cc:493] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:31.561003 18367 pending_rounds.cc:107] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da: Aborting uncommitted NO_OP operation due to leader change: 26.17
I20260709 12:25:31.554968 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:31.568385 18367 raft_consensus.cc:2889] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:31.568724 18362 raft_consensus.cc:2393] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 28: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:25:31.563555 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 29 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:31.565584 17761 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36572 (request call id 1909) took 1015 ms. Trace:
I20260709 12:25:31.469965 28523 raft_consensus.cc:493] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:31.482360 28320 raft_consensus.cc:2749] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:25:31.481801 28409 raft_consensus.cc:493] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:31.294309 28561 raft_consensus.cc:515] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:31.569519 17761 rpcz_store.cc:276] 0709 12:25:30.549733 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:30.549954 (+   221us) service_pool.cc:224] Handling call
0709 12:25:30.550389 (+   435us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:30.552152 (+  1763us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:30.552182 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:30.552643 (+   461us) log.cc:844] Serialized 26 byte log entry
0709 12:25:30.819649 (+267006us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:25:30.965300 (+145651us) log.cc:844] Serialized 14 byte log entry
0709 12:25:31.107613 (+142313us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:31.107713 (+   100us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:31.565295 (+457582us) raft_consensus.cc:1685] finished
0709 12:25:31.565325 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:31.565507 (+   182us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":162434,"prepare.queue_time_us":117995,"prepare.run_cpu_time_us":58,"prepare.run_wall_time_us":7496,"raft.queue_time_us":172154,"raft.run_cpu_time_us":299,"raft.run_wall_time_us":43351,"thread_start_us":260393,"threads_started":2,"wal-append.queue_time_us":438849}
I20260709 12:25:31.492544 28486 raft_consensus.cc:493] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:25:31.594413 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:31.550376 28529 raft_consensus.cc:2749] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:25:31.604811 17829 raft_consensus.cc:2468] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 30.
I20260709 12:25:31.458707 28524 raft_consensus.cc:515] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:31.456135 28559 consensus_queue.cc:1059] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 28.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.141s
I20260709 12:25:31.568959 28126 raft_consensus.cc:2804] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:25:31.576443 28487 raft_consensus.cc:3060] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:31.564378 28568 raft_consensus.cc:2749] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:25:31.576802 28569 raft_consensus.cc:2749] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:25:31.677299 28426 raft_consensus.cc:493] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:31.678017 28426 raft_consensus.cc:515] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:31.469437 28557 raft_consensus.cc:515] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:31.575649 28536 raft_consensus.cc:2804] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:25:31.387308 28314 raft_consensus.cc:515] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:31.637539 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 83267.5us
W20260709 12:25:31.642513 28549 log.cc:927] Time spent T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.076s	user 0.000s	sys 0.000s
I20260709 12:25:31.643059 28566 raft_consensus.cc:493] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:31.644590 28559 raft_consensus.cc:3055] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:25:31.645428 17686 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44898 (request call id 2097) took 1328 ms. Trace:
W20260709 12:25:31.646029 28501 consensus_meta.cc:287] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 1.256s	user 0.010s	sys 0.001s
W20260709 12:25:31.646806 28556 log.cc:927] Time spent T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.265s	user 0.000s	sys 0.001s
I20260709 12:25:31.650440 28515 leader_election.cc:290] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 36 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:31.651963 28407 leader_election.cc:290] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:31.654203 28561 leader_election.cc:290] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:31.594156 28523 raft_consensus.cc:515] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:31.655848 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 31 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:31.668371 28579 consensus_queue.cc:1048] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:25:31.669433 28564 consensus_queue.cc:1048] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:25:31.672693 17834 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35042 (request call id 3707) took 1238 ms. Trace:
I20260709 12:25:31.579144 28490 raft_consensus.cc:515] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:31.454646 28554 raft_consensus.cc:493] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:25:31.703645 28552 log.cc:927] Time spent T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.520s	user 0.000s	sys 0.001s
W20260709 12:25:31.705027 28562 log.cc:927] Time spent T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.085s	user 0.000s	sys 0.000s
I20260709 12:25:31.708732 28305 leader_election.cc:290] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 25 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:25:31.709254 28507 raft_consensus.cc:3055] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 34 LEADER]: Stepping down as leader of term 34
I20260709 12:25:31.597936 28159 raft_consensus.cc:515] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:31.691344 28391 leader_election.cc:290] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 29 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:31.711292 28538 leader_election.cc:290] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:25:31.810909 28183 raft_consensus.cc:3060] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:31.713258 28524 leader_election.cc:290] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:31.606832 28409 raft_consensus.cc:515] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:31.575814 28491 raft_consensus.cc:515] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:31.975212 28409 leader_election.cc:290] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:31.976184 28491 leader_election.cc:290] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:31.978143 17756 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36572 (request call id 1912) took 1420 ms. Trace:
I20260709 12:25:31.956207 18366 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59886 (request call id 2837) took 1960 ms. Trace:
I20260709 12:25:31.980088 28507 raft_consensus.cc:740] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 34 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:25:31.804946 28426 leader_election.cc:290] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:25:31.777915 28308 consensus_meta.cc:287] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.624s	user 0.009s	sys 0.000s
I20260709 12:25:31.981524 28308 raft_consensus.cc:515] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:31.982291 28554 raft_consensus.cc:3060] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:31.982467 28159 leader_election.cc:290] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:31.983397 17764 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38776 (request call id 3522) took 1357 ms. Trace:
I20260709 12:25:31.983786 17764 rpcz_store.cc:276] 0709 12:25:30.625491 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:30.757532 (+132041us) service_pool.cc:224] Handling call
0709 12:25:30.784410 (+ 26878us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:25:30.883358 (+ 98948us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:30.883393 (+    35us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:25:30.884083 (+   690us) log.cc:844] Serialized 52 byte log entry
0709 12:25:31.051729 (+167646us) raft_consensus.cc:1645] Marking committed up to 21
0709 12:25:31.108988 (+ 57259us) log.cc:844] Serialized 14 byte log entry
0709 12:25:31.231049 (+122061us) log.cc:844] Serialized 14 byte log entry
0709 12:25:31.231541 (+   492us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:31.232762 (+  1221us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:31.794652 (+561890us) spinlock_profiling.cc:243] Waited 6.07 ms on lock 0x7b18002732e0. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b47533557 00007f7b475357c1 00007f7b47535779 00007f7b47535709 00007f7b475356d1 00007f7b475349cd 0000000000388634 0000000000388479 00007f7b4751683a
0709 12:25:31.983169 (+188517us) raft_consensus.cc:1685] finished
0709 12:25:31.983197 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:31.983326 (+   129us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":38417,"prepare.queue_time_us":628972,"prepare.run_cpu_time_us":83,"prepare.run_wall_time_us":84,"raft.queue_time_us":66262,"raft.run_cpu_time_us":394,"raft.run_wall_time_us":65688,"spinlock_wait_cycles":15420032,"thread_start_us":231558,"threads_started":2,"wal-append.queue_time_us":635111}
I20260709 12:25:31.986042 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 36 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:31.978504 17756 rpcz_store.cc:276] 0709 12:25:30.558100 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:30.558201 (+   101us) service_pool.cc:224] Handling call
0709 12:25:30.568443 (+ 10242us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:30.574504 (+  6061us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:30.574548 (+    44us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:30.620848 (+ 46300us) log.cc:844] Serialized 26 byte log entry
0709 12:25:30.851908 (+231060us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:25:31.155021 (+303113us) log.cc:844] Serialized 14 byte log entry
0709 12:25:31.155257 (+   236us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:31.155341 (+    84us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:31.977931 (+822590us) raft_consensus.cc:1685] finished
0709 12:25:31.977950 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:31.978078 (+   128us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":59511,"prepare.queue_time_us":157392,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":49,"raft.queue_time_us":143049,"raft.run_cpu_time_us":317,"raft.run_wall_time_us":71007,"thread_start_us":330087,"threads_started":2,"wal-append.queue_time_us":315972}
I20260709 12:25:31.808476 28112 leader_election.cc:290] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:31.718528 28550 leader_election.cc:290] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:25:31.715924 28148 consensus_meta.cc:287] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 1.213s	user 0.010s	sys 0.004s
I20260709 12:25:31.543982 28315 consensus_queue.cc:260] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 33.20, Last appended by leader: 20, Current term: 33, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:31.947618 28416 raft_consensus.cc:493] T a5732e71b9d54f8786b8c511d69943bf P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:31.988833 28315 raft_consensus.cc:3060] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:25:31.719942 28536 raft_consensus.cc:697] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 30 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:25:31.989163 28416 raft_consensus.cc:515] T a5732e71b9d54f8786b8c511d69943bf P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:31.990192 28536 consensus_queue.cc:237] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 28.20, Last appended by leader: 20, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:31.560585 28404 raft_consensus.cc:515] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:31.696975 28509 raft_consensus.cc:493] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:31.724646 17682 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37402 (request call id 4113) took 1294 ms. Trace:
I20260709 12:25:31.734571 17688 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37396 (request call id 3604) took 2607 ms. Trace:
I20260709 12:25:31.992272 28501 raft_consensus.cc:515] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:31.992467 17682 rpcz_store.cc:276] 0709 12:25:30.430315 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:30.431063 (+   748us) service_pool.cc:224] Handling call
0709 12:25:30.552606 (+121543us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:30.553159 (+   553us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:31.724369 (+1171210us) spinlock_profiling.cc:243] Waited 70.5 ms on lock 0x7b5400345dd0. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a61f8ad 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:25:31.724572 (+   203us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":162230528}
I20260709 12:25:31.993463 17836 raft_consensus.cc:2468] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 28.
I20260709 12:25:31.613968 28565 raft_consensus.cc:2804] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:25:31.994535 28565 raft_consensus.cc:493] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:31.994736 17836 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35042 (request call id 3708) took 1560 ms. Trace:
I20260709 12:25:31.994975 28565 raft_consensus.cc:3060] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:31.995189 17836 rpcz_store.cc:276] 0709 12:25:30.434752 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:30.976051 (+541299us) service_pool.cc:224] Handling call
0709 12:25:31.994554 (+1018503us) spinlock_profiling.cc:243] Waited 967 ms on lock 0x7b54003d4fd4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:25:31.994678 (+   124us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2223505792}
I20260709 12:25:31.543185 28401 consensus_queue.cc:237] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 26.17, Last appended by leader: 17, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:31.999226 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:25:31.731464 18371 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59886 (request call id 2838) took 1734 ms (client timeout 1508 ms). Trace:
I20260709 12:25:31.578347 28504 raft_consensus.cc:515] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:32.000459 18371 rpcz_store.cc:269] 0709 12:25:29.997464 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:29.997676 (+   212us) service_pool.cc:224] Handling call
0709 12:25:31.731178 (+1733502us) spinlock_profiling.cc:243] Waited 548 ms on lock 0x7b5400724054. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:25:31.731340 (+   162us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1259414784}
I20260709 12:25:32.002516 28504 leader_election.cc:290] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:31.999954 17836 raft_consensus.cc:2468] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 32.
I20260709 12:25:31.813651 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 25 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:31.806437 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 35 candidate_status { last_received { term: 34 index: 23 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:32.005671 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 33 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:25:31.797736 17803 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:31.762565 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:31.789973 17833 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35042 (request call id 3706) took 1356 ms. Trace:
I20260709 12:25:31.801486 17835 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35042 (request call id 3711) took 1322 ms. Trace:
I20260709 12:25:31.863456 17763 raft_consensus.cc:1275] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 27 index: 18. Preceding OpId from leader: term: 28 index: 19. (index mismatch)
I20260709 12:25:31.926355 17801 leader_election.cc:304] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:31.891682 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:25:32.013613 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 87393us
I20260709 12:25:32.014038 17757 raft_consensus.cc:2468] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 29.
W20260709 12:25:31.737124 28390 consensus_meta.cc:287] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 1.118s	user 0.009s	sys 0.003s
I20260709 12:25:31.987603 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:32.011199 17835 rpcz_store.cc:276] 0709 12:25:30.478614 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:30.975222 (+496608us) service_pool.cc:224] Handling call
0709 12:25:31.801241 (+826019us) spinlock_profiling.cc:243] Waited 417 ms on lock 0x7b54002eb654. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:25:31.801395 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":959954304}
I20260709 12:25:32.011601 17686 rpcz_store.cc:276] 0709 12:25:30.317381 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:30.318076 (+   695us) service_pool.cc:224] Handling call
0709 12:25:30.340412 (+ 22336us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:30.386095 (+ 45683us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:30.386132 (+    37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:30.417764 (+ 31632us) log.cc:844] Serialized 26 byte log entry
0709 12:25:30.785701 (+367937us) raft_consensus.cc:1645] Marking committed up to 22
0709 12:25:30.913386 (+127685us) log.cc:844] Serialized 14 byte log entry
0709 12:25:31.160290 (+246904us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:31.160398 (+   108us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:31.645117 (+484719us) raft_consensus.cc:1685] finished
0709 12:25:31.645149 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:31.645349 (+   200us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":33405,"prepare.queue_time_us":328189,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":37,"raft.queue_time_us":94537,"raft.run_cpu_time_us":242,"raft.run_wall_time_us":7227,"thread_start_us":612002,"threads_started":3,"wal-append.queue_time_us":1033268}
I20260709 12:25:32.010574 17833 rpcz_store.cc:276] 0709 12:25:30.434045 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:30.434229 (+   184us) service_pool.cc:224] Handling call
0709 12:25:31.254554 (+820325us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:31.387699 (+133145us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:31.789697 (+401998us) spinlock_profiling.cc:243] Waited 770 ms on lock 0x7b54000893d0. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a61f8ad 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:25:31.789879 (+   182us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1770525568}
I20260709 12:25:31.988488 28148 raft_consensus.cc:515] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:31.622905 28486 raft_consensus.cc:515] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:31.722255 17758 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38776 (request call id 3525) took 1096 ms. Trace:
I20260709 12:25:31.992581 17688 rpcz_store.cc:276] 0709 12:25:29.127461 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:29.127766 (+   305us) service_pool.cc:224] Handling call
0709 12:25:29.128392 (+   626us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:30.714918 (+1586526us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:31.734261 (+1019343us) spinlock_profiling.cc:243] Waited 1.58 s on lock 0x7b54003002d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:25:31.734476 (+   215us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3633063168}
I20260709 12:25:32.014520 17801 leader_election.cc:304] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:31.771570 28519 leader_election.cc:290] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:32.055339 28308 leader_election.cc:290] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 28 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:31.980412 18366 rpcz_store.cc:276] 0709 12:25:29.995462 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:29.995702 (+   240us) service_pool.cc:224] Handling call
0709 12:25:29.996161 (+   459us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:30.385976 (+389815us) raft_consensus.cc:1534] Early marking committed up to index 23
0709 12:25:30.423565 (+ 37589us) log.cc:844] Serialized 14 byte log entry
0709 12:25:31.264814 (+841249us) spinlock_profiling.cc:243] Waited 1.7 ms on lock 0x7b1800171f40. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b47515e07 00007f7b503709a9 00007f7b4753269b 00007f7b4753093f 00007f7b47532c02 00007f7b4a1ba20b 00007f7b4a1bfd23 00007f7b4a1c023a 00007f7b4a62aa2a
0709 12:25:31.591644 (+326830us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:31.606974 (+ 15330us) log.cc:844] Serialized 26 byte log entry
0709 12:25:31.607132 (+   158us) raft_consensus.cc:1645] Marking committed up to 23
0709 12:25:31.607176 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:31.607239 (+    63us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:31.955756 (+348517us) raft_consensus.cc:1685] finished
0709 12:25:31.955924 (+   168us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:31.956127 (+   203us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":39367,"prepare.run_cpu_time_us":63,"prepare.run_wall_time_us":4012,"raft.queue_time_us":182639,"raft.run_cpu_time_us":2585,"raft.run_wall_time_us":30827,"spinlock_wait_cycles":3910528,"thread_start_us":691839,"threads_started":2,"wal-append.queue_time_us":1523174}
I20260709 12:25:31.957653 18374 raft_consensus.cc:1240] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 30. Current term is 31. Ops: []
W20260709 12:25:32.017220 18361 consensus_meta.cc:287] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.785s	user 0.010s	sys 0.000s
I20260709 12:25:31.855057 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 36 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:32.048306 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 32 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:31.830076 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 24 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:32.058607 18361 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 31.
I20260709 12:25:31.746253 17652 leader_election.cc:304] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:32.009701 17838 raft_consensus.cc:2468] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 31.
I20260709 12:25:31.995625 28404 leader_election.cc:290] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:32.094319 28491 raft_consensus.cc:2804] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:25:32.094820 28491 raft_consensus.cc:493] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 34 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:25:32.095263 28491 raft_consensus.cc:3060] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 34 FOLLOWER]: Advancing to term 35
W20260709 12:25:32.095521 18336 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.893s	user 0.018s	sys 0.009s
W20260709 12:25:32.095865 18336 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.894s	user 0.018s	sys 0.009s
I20260709 12:25:32.048329 17756 raft_consensus.cc:2393] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 32: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
W20260709 12:25:32.047967 17728 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.541s	user 0.025s	sys 0.019s
W20260709 12:25:32.098009 17728 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.591s	user 0.026s	sys 0.019s
I20260709 12:25:32.050304 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:32.050920 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:25:31.770485 28530 consensus_meta.cc:287] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.885s	user 0.008s	sys 0.004s
I20260709 12:25:32.101434 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 27 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:32.010594 17834 rpcz_store.cc:276] 0709 12:25:30.434509 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:31.040915 (+606406us) service_pool.cc:224] Handling call
0709 12:25:31.089906 (+ 48991us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:31.101001 (+ 11095us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:31.101381 (+   380us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:31.110724 (+  9343us) log.cc:844] Serialized 26 byte log entry
0709 12:25:31.461408 (+350684us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:25:31.461535 (+   127us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:31.461895 (+   360us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:31.672412 (+210517us) raft_consensus.cc:1685] finished
0709 12:25:31.672441 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:31.672609 (+   168us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":17126,"raft.queue_time_us":8728,"raft.run_cpu_time_us":133,"raft.run_wall_time_us":9861,"thread_start_us":276966,"threads_started":1,"wal-append.queue_time_us":533878}
I20260709 12:25:32.101859 28530 raft_consensus.cc:2749] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 35 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:25:32.102061 18364 raft_consensus.cc:2468] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 26.
I20260709 12:25:32.103246 17764 raft_consensus.cc:3060] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 35 FOLLOWER]: Advancing to term 36
W20260709 12:25:32.085937 17831 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.634s	user 0.102s	sys 0.174s
W20260709 12:25:31.775111 28511 log.cc:927] Time spent T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 1.361s	user 0.000s	sys 0.003s
I20260709 12:25:32.056522 17758 rpcz_store.cc:276] 0709 12:25:30.626254 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:31.052778 (+426524us) service_pool.cc:224] Handling call
0709 12:25:31.053121 (+   343us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:31.096995 (+ 43874us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:31.100161 (+  3166us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:31.110382 (+ 10221us) log.cc:844] Serialized 26 byte log entry
0709 12:25:31.471606 (+361224us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:25:31.471646 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:31.471712 (+    66us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:31.722030 (+250318us) raft_consensus.cc:1685] finished
0709 12:25:31.722051 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:31.722194 (+   143us) inbound_call.cc:177] Queueing success response
0709 12:25:31.752333 (+ 30139us) spinlock_profiling.cc:243] Waited 10.7 ms on lock 0x7b48002214d8. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4f549aa1 00007f7b4f540b37 00007f7b4f5409ca 00007f7b4f63826f 00007f7b4f637f37 00007f7b4f64c55c 00007f7b4f64c508 00007f7b4f64c488 00007f7b4f64c443 00007f7b4f64b6ec 00007f7b4e575a96
Metrics: {"mutex_wait_us":37090,"raft.queue_time_us":69639,"raft.run_cpu_time_us":3634,"raft.run_wall_time_us":629407,"spinlock_wait_cycles":24702720,"thread_start_us":340292,"threads_started":1,"wal-append.queue_time_us":549284}
W20260709 12:25:31.767359 28532 log.cc:927] Time spent T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.986s	user 0.000s	sys 0.003s
W20260709 12:25:32.092905 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 346782us
I20260709 12:25:31.751658 28303 raft_consensus.cc:493] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:32.150362 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 35 candidate_status { last_received { term: 34 index: 23 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:32.150921 18363 raft_consensus.cc:3060] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:25:32.100674 28515 raft_consensus.cc:493] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:32.101286 17835 raft_consensus.cc:2468] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 29.
I20260709 12:25:32.053524 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 27 candidate_status { last_received { term: 25 index: 22 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:32.152011 17836 raft_consensus.cc:2468] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 26.
W20260709 12:25:31.950946 28553 consensus_meta.cc:287] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.713s	user 0.008s	sys 0.001s
W20260709 12:25:32.098521 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 594637us
I20260709 12:25:32.086375 17837 raft_consensus.cc:1240] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 35 LEADER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 34. Current term is 35. Ops: []
W20260709 12:25:32.052995 27431 consensus_meta.cc:287] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.601s	user 0.007s	sys 0.003s
W20260709 12:25:31.791345 28526 consensus_meta.cc:287] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 1.008s	user 0.006s	sys 0.005s
I20260709 12:25:31.986768 28490 leader_election.cc:290] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:31.926707 28502 raft_consensus.cc:493] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:32.103837 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 27 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:31.963824 28314 leader_election.cc:290] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 31 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:25:32.117167 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:32.104221 17831 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.652s	user 0.102s	sys 0.174s
I20260709 12:25:31.882690 28516 raft_consensus.cc:493] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:32.101830 17759 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.558s	user 0.002s	sys 0.006s
W20260709 12:25:32.179229 17759 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.635s	user 0.002s	sys 0.006s
I20260709 12:25:31.699288 28274 raft_consensus.cc:493] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:32.115072 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 35 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:32.103909 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 31 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:32.119000 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:32.119817 17727 leader_election.cc:304] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:32.120319 17829 raft_consensus.cc:3060] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:25:31.927150 28543 raft_consensus.cc:493] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:31.814070 28320 raft_consensus.cc:493] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:25:31.809688 28503 leader_election.cc:290] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:31.992312 28509 raft_consensus.cc:515] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:31.701174 28126 raft_consensus.cc:697] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 31 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:25:32.155397 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 34 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:31.760926 28495 raft_consensus.cc:493] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 33 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
W20260709 12:25:32.220791 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 100995us
I20260709 12:25:31.765904 28341 leader_election.cc:290] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 29 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:31.817544 28540 raft_consensus.cc:493] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:32.015038 28559 raft_consensus.cc:740] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 30 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:25:31.750048 28545 raft_consensus.cc:493] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:25:32.175071 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:32.172601 28538 raft_consensus.cc:2804] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:25:32.176939 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 27 candidate_status { last_received { term: 25 index: 22 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:32.177608 18364 raft_consensus.cc:1275] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 26 index: 17. Preceding OpId from leader: term: 28 index: 19. (index mismatch)
I20260709 12:25:31.943655 28557 leader_election.cc:290] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:25:32.182440 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:32.198264 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 33 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:32.204512 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 34 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:32.114473 28305 raft_consensus.cc:2804] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Leader pre-election won for term 32
W20260709 12:25:32.205660 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 54351.8us
I20260709 12:25:32.204639 18371 raft_consensus.cc:2468] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 29.
I20260709 12:25:32.104379 28426 raft_consensus.cc:493] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:32.148765 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 24 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:32.252080 17652 leader_election.cc:304] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:32.252753 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 34 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:32.253471 18361 raft_consensus.cc:2468] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 33.
I20260709 12:25:31.914024 28487 raft_consensus.cc:515] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:32.276248 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 27 candidate_status { last_received { term: 25 index: 22 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:32.277647 18368 raft_consensus.cc:1240] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 28. Current term is 29. Ops: []
I20260709 12:25:32.278968 18368 raft_consensus.cc:1275] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 33 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 31 index: 18. Preceding OpId from leader: term: 33 index: 19. (index mismatch)
I20260709 12:25:32.119603 28404 raft_consensus.cc:493] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:25:32.280066 28404 raft_consensus.cc:515] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:32.282684 28404 leader_election.cc:290] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:32.284394 28559 consensus_queue.cc:260] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 30.22, Last appended by leader: 22, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:32.181193 28502 raft_consensus.cc:515] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:32.287262 28502 leader_election.cc:290] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:25:32.209666 17762 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36572 (request call id 1910) took 1656 ms. Trace:
I20260709 12:25:32.287869 17762 rpcz_store.cc:276] 0709 12:25:30.552998 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:30.553361 (+   363us) service_pool.cc:224] Handling call
0709 12:25:30.570615 (+ 17254us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:30.606101 (+ 35486us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:30.606136 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:30.618474 (+ 12338us) log.cc:844] Serialized 26 byte log entry
0709 12:25:30.898022 (+279548us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:25:30.898067 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:30.898139 (+    72us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:32.209411 (+1311272us) raft_consensus.cc:1685] finished
0709 12:25:32.209437 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:32.209597 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":70306,"raft.queue_time_us":46364,"raft.run_cpu_time_us":204,"raft.run_wall_time_us":68664,"thread_start_us":168323,"threads_started":1,"wal-append.queue_time_us":736345}
I20260709 12:25:32.288735 28554 raft_consensus.cc:515] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:32.181339 28556 log.cc:927] Time spent T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.093s	user 0.000s	sys 0.000s
I20260709 12:25:32.221174 18366 raft_consensus.cc:2410] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for term 34 because replica has last-logged OpId of term: 33 index: 22, which is greater than that of the candidate, which has last-logged OpId of term: 30 index: 20.
I20260709 12:25:32.173898 28425 raft_consensus.cc:493] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:25:32.247244 18365 raft_consensus.cc:2468] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 25.
I20260709 12:25:32.247047 28538 raft_consensus.cc:493] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:25:32.290814 28425 raft_consensus.cc:515] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:32.291206 28538 raft_consensus.cc:3060] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:32.215400 28320 raft_consensus.cc:515] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:32.178721 28303 raft_consensus.cc:515] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:32.179826 28515 raft_consensus.cc:515] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:32.294378 28303 leader_election.cc:290] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:32.295145 28426 raft_consensus.cc:515] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:32.296034 28515 leader_election.cc:290] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:32.296260 17758 raft_consensus.cc:2410] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for term 33 because replica has last-logged OpId of term: 31 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 28 index: 18.
I20260709 12:25:32.296991 17836 raft_consensus.cc:1275] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 33 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 31 index: 18. Preceding OpId from leader: term: 33 index: 19. (index mismatch)
I20260709 12:25:32.297238 28426 leader_election.cc:290] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:32.297629 28425 leader_election.cc:290] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:32.173023 28526 raft_consensus.cc:515] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:32.298776 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:32.299371 18365 raft_consensus.cc:2468] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 29.
I20260709 12:25:32.299628 28501 leader_election.cc:290] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 28 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:32.300338 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 30 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:32.300246 28545 raft_consensus.cc:515] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:32.300360 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 31 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:32.301524 18365 raft_consensus.cc:2393] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 31: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:25:32.302459 28545 leader_election.cc:290] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:32.303308 17832 raft_consensus.cc:2410] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for term 33 because replica has last-logged OpId of term: 31 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 28 index: 18.
I20260709 12:25:32.198760 28516 raft_consensus.cc:515] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:32.305481 28516 leader_election.cc:290] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:25:32.255121 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 29 candidate_status { last_received { term: 28 index: 22 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:32.255795 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 25 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:32.306499 18364 raft_consensus.cc:2410] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for term 25 because replica has last-logged OpId of term: 23 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 16.
I20260709 12:25:32.306704 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 29 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:32.307235 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:32.255618 28507 consensus_queue.cc:260] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 34.19, Last appended by leader: 19, Current term: 34, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:32.255962 28305 raft_consensus.cc:493] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:25:32.308321 28507 raft_consensus.cc:3060] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:25:32.308691 28305 raft_consensus.cc:3060] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:25:32.210217 28416 leader_election.cc:290] T a5732e71b9d54f8786b8c511d69943bf P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:32.249464 17756 raft_consensus.cc:2468] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 33.
I20260709 12:25:32.270299 28564 consensus_queue.cc:1059] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 34.23, Next index: 24, Last known committed idx: 23, Time since last communication: 0.001s
I20260709 12:25:32.173020 28504 consensus_queue.cc:1059] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 31.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.054s
I20260709 12:25:32.264797 28486 leader_election.cc:290] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:32.285578 28559 raft_consensus.cc:3060] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:32.306623 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 32 candidate_status { last_received { term: 30 index: 21 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:32.312899 28504 raft_consensus.cc:3055] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 34 LEADER]: Stepping down as leader of term 34
I20260709 12:25:32.313321 28504 raft_consensus.cc:740] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 34 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:25:32.271992 28540 raft_consensus.cc:515] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:32.314199 28504 consensus_queue.cc:260] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 34.23, Last appended by leader: 23, Current term: 34, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:32.103390 28578 consensus_queue.cc:1048] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.161s
I20260709 12:25:32.316818 28578 raft_consensus.cc:3055] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 31 LEADER]: Stepping down as leader of term 31
I20260709 12:25:32.317193 28578 raft_consensus.cc:740] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 31 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:25:32.318005 28578 consensus_queue.cc:260] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 31.19, Last appended by leader: 19, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:32.174677 28553 raft_consensus.cc:515] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:32.048080 28566 raft_consensus.cc:515] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:32.332916 18361 raft_consensus.cc:1240] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 28. Current term is 29. Ops: [28.22-28.22]
W20260709 12:25:32.333299 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 81315.2us
I20260709 12:25:32.307374 18371 raft_consensus.cc:2468] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 28.
I20260709 12:25:32.303653 17833 raft_consensus.cc:2468] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 25.
I20260709 12:25:32.182251 17835 raft_consensus.cc:1240] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 34 LEADER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 33. Current term is 34. Ops: [33.20-33.20]
I20260709 12:25:32.255524 28490 raft_consensus.cc:493] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:25:32.343228 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:32.344309 18368 raft_consensus.cc:2393] T a5732e71b9d54f8786b8c511d69943bf P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 30: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:25:32.305181 28538 raft_consensus.cc:515] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:32.227276 28495 raft_consensus.cc:3060] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:25:32.374001 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:32.374305 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 30 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
W20260709 12:25:32.378295 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 363826us
I20260709 12:25:32.368894 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 34 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:32.380594 17757 raft_consensus.cc:2468] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 33.
I20260709 12:25:32.369622 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 29 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:32.383002 18361 raft_consensus.cc:2468] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 28.
I20260709 12:25:32.384248 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:32.385259 18361 raft_consensus.cc:2393] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 27: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:25:32.370841 18363 raft_consensus.cc:2468] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 35.
I20260709 12:25:32.366626 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 31 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:32.352746 17652 leader_election.cc:304] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:25:32.391110 17801 leader_election.cc:304] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:32.395148 17764 raft_consensus.cc:2468] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 36.
I20260709 12:25:32.404912 17756 raft_consensus.cc:2393] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 30: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
W20260709 12:25:32.367132 18334 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.825s	user 0.014s	sys 0.021s
W20260709 12:25:32.405661 18334 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.864s	user 0.014s	sys 0.022s
W20260709 12:25:32.405962 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 979756us
I20260709 12:25:32.410864 18362 raft_consensus.cc:1275] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 25 index: 18. Preceding OpId from leader: term: 27 index: 19. (index mismatch)
I20260709 12:25:32.411056 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:32.411746 18371 raft_consensus.cc:2468] T a5732e71b9d54f8786b8c511d69943bf P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 30.
I20260709 12:25:32.412518 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 35 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:32.335397 28486 raft_consensus.cc:2749] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:25:32.205443 28274 raft_consensus.cc:515] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:32.413676 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:25:32.414258 18334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:32.417526 17757 raft_consensus.cc:2468] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 30.
I20260709 12:25:32.407864 17834 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35040 (request call id 3897) took 1461 ms. Trace:
I20260709 12:25:32.418126 17834 rpcz_store.cc:276] 0709 12:25:30.946699 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:32.102442 (+1155743us) service_pool.cc:224] Handling call
0709 12:25:32.152306 (+ 49864us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:32.152530 (+   224us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:32.152566 (+    36us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:32.152620 (+    54us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:25:32.152664 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:32.152732 (+    68us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:32.407726 (+254994us) spinlock_profiling.cc:243] Waited 49.4 ms on lock 0x7b54004416d0. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a61f8ad 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:25:32.407812 (+    86us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":113514240}
I20260709 12:25:32.418956 17760 raft_consensus.cc:2410] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for term 35 because replica has last-logged OpId of term: 33 index: 20, which is greater than that of the candidate, which has last-logged OpId of term: 32 index: 19.
I20260709 12:25:32.415226 17802 leader_election.cc:304] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:32.416076 17762 raft_consensus.cc:2393] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 31: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:25:32.419710 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 31 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:32.419777 17652 leader_election.cc:304] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:32.366654 28416 raft_consensus.cc:493] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:32.425470 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 31 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:32.375396 28578 raft_consensus.cc:3060] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:25:32.437791 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 29 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:32.438230 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:32.438593 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 35 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:32.439487 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 28 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:32.439991 18371 raft_consensus.cc:2468] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 27.
I20260709 12:25:32.429498 18361 raft_consensus.cc:2376] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 31. Re-sending same reply.
I20260709 12:25:32.431595 17801 leader_election.cc:304] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:32.433328 17802 leader_election.cc:304] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:25:32.441634 28247 raft_consensus.cc:2804] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:25:32.441897 28309 raft_consensus.cc:2804] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:25:32.442152 28247 raft_consensus.cc:493] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:25:32.442546 28309 raft_consensus.cc:493] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:32.442818 28247 raft_consensus.cc:3060] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:32.406788 28315 raft_consensus.cc:493] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 34 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:25:32.411561 28407 raft_consensus.cc:2804] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:25:32.438666 18368 raft_consensus.cc:2368] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for earlier term 29. Current term is 30.
I20260709 12:25:32.448542 18366 raft_consensus.cc:2468] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 29.
W20260709 12:25:32.414496 28485 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.659s	user 0.017s	sys 0.052s
I20260709 12:25:32.445595 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:32.451613 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:25:32.430755 18333 connection.cc:256] RPC call timeout handler was delayed by -1.5692746569998235s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1654s for a response
I20260709 12:25:32.459008 17763 raft_consensus.cc:2468] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 25.
I20260709 12:25:32.417950 28341 raft_consensus.cc:493] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:32.462003 17762 raft_consensus.cc:1240] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 LEADER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 26. Current term is 27. Ops: []
W20260709 12:25:32.453930 18333 connection.cc:256] RPC call timeout handler was delayed by -1.3271746569998868s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19230000000000003s for a response
I20260709 12:25:32.466293 17834 raft_consensus.cc:2393] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 30: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:25:32.406213 28515 consensus_queue.cc:1059] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.071s
I20260709 12:25:32.422015 28519 raft_consensus.cc:493] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 33 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:25:32.422171 28409 raft_consensus.cc:493] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:25:32.424885 28303 raft_consensus.cc:493] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:32.426123 28516 raft_consensus.cc:2804] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:25:32.466022 18333 leader_election.cc:304] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:32.449987 28486 raft_consensus.cc:2804] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:25:32.478744 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 29 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:32.466017 28530 raft_consensus.cc:2804] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:25:32.480010 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:25:32.457266 28485 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.702s	user 0.017s	sys 0.052s
I20260709 12:25:32.509728 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 36 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:32.444639 28320 leader_election.cc:290] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:32.447566 28309 raft_consensus.cc:3060] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:32.498765 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:32.404538 28490 raft_consensus.cc:515] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:32.438323 28501 raft_consensus.cc:2749] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:25:32.446350 28416 raft_consensus.cc:515] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:32.460708 28407 raft_consensus.cc:493] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:25:32.507619 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:32.495394 28516 raft_consensus.cc:493] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:25:32.445236 28545 consensus_queue.cc:1048] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.052s
I20260709 12:25:32.524655 17837 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35042 (request call id 3709) took 2082 ms. Trace:
I20260709 12:25:32.549064 17762 raft_consensus.cc:3060] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:32.504053 28515 raft_consensus.cc:3055] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:25:32.503366 28314 raft_consensus.cc:2804] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:25:32.500005 28486 raft_consensus.cc:493] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:25:32.554384 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:32.551545 28516 raft_consensus.cc:3060] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Advancing to term 29
W20260709 12:25:32.566519 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 344258us
I20260709 12:25:32.556095 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 26 candidate_status { last_received { term: 25 index: 22 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:32.555495 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 31 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:32.548259 17837 rpcz_store.cc:276] 0709 12:25:30.442564 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:31.046100 (+603536us) service_pool.cc:224] Handling call
0709 12:25:31.053044 (+  6944us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:32.156860 (+1103816us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:32.524408 (+367548us) spinlock_profiling.cc:243] Waited 1.03 s on lock 0x7b54001581d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:25:32.524597 (+   189us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2375945984}
I20260709 12:25:32.468909 28341 raft_consensus.cc:515] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:32.580343 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 29 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:32.580945 18371 raft_consensus.cc:3060] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:32.464960 28315 raft_consensus.cc:515] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:32.563151 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:32.563556 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 29 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:32.583422 17838 raft_consensus.cc:2410] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for term 29 because replica has last-logged OpId of term: 27 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 17.
I20260709 12:25:32.481284 28126 consensus_queue.cc:237] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 30.17, Last appended by leader: 17, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:32.498454 28409 raft_consensus.cc:515] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:32.567171 28314 raft_consensus.cc:493] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:25:32.577231 17831 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35040 (request call id 3894) took 1953 ms. Trace:
I20260709 12:25:32.557785 17729 leader_election.cc:304] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:25:32.526692 28540 leader_election.cc:290] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:25:32.590164 17727 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:32.578364 17834 raft_consensus.cc:2468] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 26.
I20260709 12:25:32.550650 28148 leader_election.cc:290] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 29 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:32.547394 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 37 candidate_status { last_received { term: 36 index: 27 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:32.583083 17760 raft_consensus.cc:2468] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 30.
I20260709 12:25:32.586956 17729 leader_election.cc:304] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:32.483063 28308 consensus_queue.cc:1059] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.097s
I20260709 12:25:32.602095 17727 leader_election.cc:304] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:32.579543 28486 raft_consensus.cc:3060] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:25:32.619004 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 32 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:32.561369 28515 raft_consensus.cc:740] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:25:32.566452 28526 leader_election.cc:290] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 29 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:25:32.613152 28565 consensus_meta.cc:287] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.611s	user 0.005s	sys 0.009s
I20260709 12:25:32.596293 17729 leader_election.cc:304] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:32.596315 28425 raft_consensus.cc:2804] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:25:32.501965 28523 leader_election.cc:290] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:25:32.614682 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 29 candidate_status { last_received { term: 27 index: 21 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:32.592181 28314 raft_consensus.cc:3060] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:32.503151 28303 raft_consensus.cc:515] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:32.581612 28407 raft_consensus.cc:3060] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:25:32.619844 17838 raft_consensus.cc:2393] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 32: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:25:32.614265 28148 raft_consensus.cc:493] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:25:32.495267 28519 raft_consensus.cc:515] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:32.530435 28426 raft_consensus.cc:493] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:32.604888 28320 raft_consensus.cc:2804] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:25:32.620199 28545 raft_consensus.cc:493] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:25:32.602310 28305 raft_consensus.cc:515] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 32 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:32.651480 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:32.636399 28425 raft_consensus.cc:493] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:32.327162 28543 raft_consensus.cc:515] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:32.655661 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 29 candidate_status { last_received { term: 27 index: 21 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:32.638258 28553 leader_election.cc:290] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 31 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:32.648336 28538 leader_election.cc:290] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 31 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:32.735499 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 28 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:25:32.720609 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 300804us
I20260709 12:25:32.636478 28565 raft_consensus.cc:515] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:32.727085 18371 raft_consensus.cc:2468] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 29.
W20260709 12:25:32.730425 17829 consensus_meta.cc:287] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.506s	user 0.010s	sys 0.001s
I20260709 12:25:32.742342 17829 raft_consensus.cc:2468] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 36.
I20260709 12:25:32.748422 28313 raft_consensus.cc:2804] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:25:32.749758 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 29 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:32.750674 18371 raft_consensus.cc:1240] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 35 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 34. Current term is 35. Ops: []
I20260709 12:25:32.746361 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 29 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:32.761031 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 29 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:32.734489 28425 raft_consensus.cc:3060] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:32.733144 28320 raft_consensus.cc:493] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:25:32.750857 28183 consensus_meta.cc:287] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.775s	user 0.009s	sys 0.002s
I20260709 12:25:32.784431 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 31 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:32.779950 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:32.790602 17837 raft_consensus.cc:2468] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 30.
I20260709 12:25:32.792191 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 29 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:32.792963 17837 raft_consensus.cc:3060] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:32.798301 17833 raft_consensus.cc:2393] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 31: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:25:32.653226 28515 consensus_queue.cc:260] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 21.14, Last appended by leader: 14, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:32.751832 28566 leader_election.cc:290] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:25:32.728906 28148 raft_consensus.cc:515] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:32.748884 28313 raft_consensus.cc:493] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:25:32.803798 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 33 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:32.808385 17757 raft_consensus.cc:2468] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 32.
I20260709 12:25:32.786674 28320 raft_consensus.cc:3060] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:32.739236 28545 raft_consensus.cc:515] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:32.736418 28426 raft_consensus.cc:515] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:32.815085 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 29 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:32.801252 28589 raft_consensus.cc:493] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:32.829399 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:32.830592 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 32 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:32.810067 28416 leader_election.cc:290] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:32.804929 28603 raft_consensus.cc:2804] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:25:32.809541 28313 raft_consensus.cc:3060] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Advancing to term 30
W20260709 12:25:32.832705 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 230711us
I20260709 12:25:32.833830 17838 raft_consensus.cc:2468] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 29.
I20260709 12:25:32.839895 17757 raft_consensus.cc:2393] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 28: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:25:32.819514 28515 raft_consensus.cc:3060] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:32.817404 28341 leader_election.cc:290] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:25:32.932330 17803 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.722s	user 0.015s	sys 0.035s
I20260709 12:25:32.935679 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 33 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:32.817751 28315 leader_election.cc:290] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:32.963999 17760 raft_consensus.cc:2468] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 31.
W20260709 12:25:32.964473 17803 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.754s	user 0.015s	sys 0.035s
W20260709 12:25:32.980836 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 802582us
I20260709 12:25:32.974753 28565 leader_election.cc:290] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 26 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:32.829582 28409 leader_election.cc:290] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:32.832787 28566 raft_consensus.cc:493] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:32.981382 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 464353us
I20260709 12:25:32.992199 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 35 candidate_status { last_received { term: 32 index: 23 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:32.985602 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 29 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:32.994303 17762 raft_consensus.cc:2468] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 28.
I20260709 12:25:32.842593 28603 raft_consensus.cc:493] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:25:33.000856 18333 leader_election.cc:304] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:25:33.009263 18362 raft_consensus.cc:2393] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 29: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:25:33.028323 28503 consensus_queue.cc:1059] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20260709 12:25:33.030287 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 29 candidate_status { last_received { term: 28 index: 22 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:33.031318 18363 raft_consensus.cc:2393] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 29: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:25:33.032464 17836 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35040 (request call id 3904) took 1302 ms. Trace:
I20260709 12:25:33.032951 17836 rpcz_store.cc:276] 0709 12:25:31.730280 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:32.152754 (+422474us) service_pool.cc:224] Handling call
0709 12:25:32.153088 (+   334us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:32.297374 (+144286us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:33.032174 (+734800us) spinlock_profiling.cc:243] Waited 144 ms on lock 0x7b540008cfd4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:25:33.032393 (+   219us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":330229504}
I20260709 12:25:33.034356 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:32.982535 28502 raft_consensus.cc:493] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:33.035933 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
W20260709 12:25:33.036616 28592 log.cc:927] Time spent T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.934s	user 0.003s	sys 0.000s
I20260709 12:25:33.036954 28305 leader_election.cc:290] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 32 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:25:33.038390 28590 log.cc:927] Time spent T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.943s	user 0.000s	sys 0.003s
I20260709 12:25:33.039244 17764 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38776 (request call id 3539) took 1139 ms. Trace:
I20260709 12:25:33.036283 28502 raft_consensus.cc:515] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:33.041575 17830 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35040 (request call id 3893) took 2418 ms. Trace:
I20260709 12:25:33.042588 28502 leader_election.cc:290] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:25:33.043205 28507 consensus_meta.cc:287] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.734s	user 0.007s	sys 0.000s
I20260709 12:25:32.828953 28589 raft_consensus.cc:515] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:33.044098 17763 raft_consensus.cc:2410] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for term 36 because replica has last-logged OpId of term: 34 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 31 index: 18.
I20260709 12:25:33.044027 28487 leader_election.cc:290] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 28 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:33.025121 28566 raft_consensus.cc:515] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:33.045048 28502 raft_consensus.cc:493] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:33.045661 28502 raft_consensus.cc:515] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:33.046435 17835 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35042 (request call id 3717) took 1577 ms. Trace:
I20260709 12:25:33.047003 28566 leader_election.cc:290] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:33.019112 28603 raft_consensus.cc:3060] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Advancing to term 28
W20260709 12:25:33.052868 28593 log.cc:927] Time spent T 00000000000000000000000000000000 P 2193af31cb894ec9b3d65d939a2a6851: Append to log took a long time: real 0.967s	user 0.001s	sys 0.002s
I20260709 12:25:33.046800 17835 rpcz_store.cc:276] 0709 12:25:31.469036 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:32.152186 (+683150us) service_pool.cc:224] Handling call
0709 12:25:32.157260 (+  5074us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:32.334709 (+177449us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:33.046194 (+711485us) spinlock_profiling.cc:243] Waited 24.7 ms on lock 0x7b5400267354. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:25:33.046381 (+   187us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":56864128}
I20260709 12:25:33.054803 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:33.055320 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:33.039597 17764 rpcz_store.cc:276] 0709 12:25:31.899970 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:31.985767 (+ 85797us) service_pool.cc:224] Handling call
0709 12:25:33.039057 (+1053290us) spinlock_profiling.cc:243] Waited 117 ms on lock 0x7b54003e62d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:25:33.039170 (+   113us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":268243712}
I20260709 12:25:33.057051 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 31 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:33.057121 28426 leader_election.cc:290] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:33.057663 17764 raft_consensus.cc:3060] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Advancing to term 31
W20260709 12:25:33.058282 28602 log.cc:927] Time spent T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.240s	user 0.000s	sys 0.000s
I20260709 12:25:33.059546 28504 raft_consensus.cc:3060] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:25:33.041929 17830 rpcz_store.cc:276] 0709 12:25:30.623296 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:31.207067 (+583771us) service_pool.cc:224] Handling call
0709 12:25:31.252355 (+ 45288us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:31.510113 (+257758us) raft_consensus.cc:1534] Early marking committed up to index 19
0709 12:25:32.029586 (+519473us) log.cc:844] Serialized 14 byte log entry
0709 12:25:32.364744 (+335158us) log.cc:844] Serialized 14 byte log entry
0709 12:25:32.409896 (+ 45152us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:32.409945 (+    49us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:25:32.409992 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:32.557951 (+147959us) spinlock_profiling.cc:243] Waited 4.79 ms on lock 0x7b18002bf000. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b475333f3 00007f7b475357c1 00007f7b47535779 00007f7b47535709 00007f7b475356d1 00007f7b475349cd 0000000000388634 0000000000388479 00007f7b4751683a
0709 12:25:33.040776 (+482825us) spinlock_profiling.cc:243] Waited 10.1 ms on lock 0x7b18002bf000. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b475012a6 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419
0709 12:25:33.041378 (+   602us) spinlock_profiling.cc:243] Waited 257 ms on lock 0x7b54003354d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:25:33.041420 (+    42us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:33.041523 (+   103us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":577925,"prepare.run_cpu_time_us":60,"prepare.run_wall_time_us":62,"raft.queue_time_us":45432,"raft.run_cpu_time_us":130,"raft.run_wall_time_us":40247,"spinlock_wait_cycles":694529792,"thread_start_us":785954,"threads_started":2,"wal-append.queue_time_us":386476}
I20260709 12:25:33.061730 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 27 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:33.062414 17830 raft_consensus.cc:2468] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 25.
I20260709 12:25:33.062975 28274 leader_election.cc:290] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:33.063143 17835 raft_consensus.cc:2468] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 29.
I20260709 12:25:33.036706 18363 raft_consensus.cc:3060] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:33.064177 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:33.065586 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 33 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:32.587447 17831 rpcz_store.cc:276] 0709 12:25:30.623601 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:31.291071 (+667470us) service_pool.cc:224] Handling call
0709 12:25:31.386154 (+ 95083us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:31.386365 (+   211us) raft_consensus.cc:1534] Early marking committed up to index 18
0709 12:25:32.173826 (+787461us) log.cc:844] Serialized 14 byte log entry
0709 12:25:32.576796 (+402970us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:32.576850 (+    54us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:25:32.576901 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:32.576994 (+    93us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:32.577156 (+   162us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":699708,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":45,"raft.queue_time_us":656,"raft.run_cpu_time_us":121,"raft.run_wall_time_us":5785,"thread_start_us":1123644,"threads_started":2}
I20260709 12:25:33.066496 18362 raft_consensus.cc:2410] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for term 33 because replica has last-logged OpId of term: 31 index: 21, which is greater than that of the candidate, which has last-logged OpId of term: 30 index: 20.
I20260709 12:25:33.066771 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:33.068354 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 31 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:33.069473 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 32 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:33.069967 28509 leader_election.cc:290] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:33.024814 28523 raft_consensus.cc:493] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:25:33.070501 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 33 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:33.071187 28523 raft_consensus.cc:515] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:25:33.072068 17829 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35040 (request call id 3906) took 1248 ms (client timeout 1567 ms). Trace:
W20260709 12:25:33.072451 17829 rpcz_store.cc:269] 0709 12:25:31.823293 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:31.854643 (+ 31350us) service_pool.cc:224] Handling call
0709 12:25:33.071910 (+1217267us) spinlock_profiling.cc:243] Waited 60.9 ms on lock 0x7b5400335ed4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:25:33.072017 (+   107us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":140132736}
I20260709 12:25:33.073503 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:33.074168 17829 raft_consensus.cc:2393] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 31: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:25:33.074963 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 29 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:33.075138 17764 raft_consensus.cc:2468] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 31.
W20260709 12:25:33.076336 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 75563.3us
I20260709 12:25:33.077076 18374 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59886 (request call id 2846) took 1977 ms. Trace:
I20260709 12:25:33.077332 28610 raft_consensus.cc:2804] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 32 FOLLOWER]: Leader election won for term 32
I20260709 12:25:33.077493 18333 leader_election.cc:304] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:33.078040 28610 raft_consensus.cc:697] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 32 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
W20260709 12:25:33.078840 28591 log.cc:927] Time spent T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.978s	user 0.002s	sys 0.000s
I20260709 12:25:33.081555 28613 raft_consensus.cc:2749] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 35 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:25:33.067734 18362 raft_consensus.cc:1240] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 30. Current term is 31. Ops: [30.22-30.22]
I20260709 12:25:33.072103 17757 raft_consensus.cc:1275] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 34 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 33 index: 20. Preceding OpId from leader: term: 34 index: 20. (term mismatch)
I20260709 12:25:33.084282 17757 pending_rounds.cc:85] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb: Aborting all ops after (but not including) 19
I20260709 12:25:33.084550 17757 pending_rounds.cc:107] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb: Aborting uncommitted NO_OP operation due to leader change: 33.20
I20260709 12:25:33.084820 17757 raft_consensus.cc:2889] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 34 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:33.077498 18374 rpcz_store.cc:276] 0709 12:25:31.101415 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:31.108829 (+  7414us) service_pool.cc:224] Handling call
0709 12:25:31.131771 (+ 22942us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:32.060502 (+928731us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:33.076859 (+1016357us) spinlock_profiling.cc:243] Waited 826 ms on lock 0x7b54007602d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:25:33.077021 (+   162us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1898695040}
I20260709 12:25:33.055410 18361 raft_consensus.cc:2468] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 27.
W20260709 12:25:33.049340 28605 log.cc:927] Time spent T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.070s	user 0.000s	sys 0.000s
I20260709 12:25:33.078850 18333 leader_election.cc:304] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:33.089200 18367 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59902 (request call id 2470) took 1027 ms. Trace:
W20260709 12:25:33.070389 28491 consensus_meta.cc:287] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.975s	user 0.008s	sys 0.009s
W20260709 12:25:33.092300 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:33.093233 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:33.089612 18367 rpcz_store.cc:276] 0709 12:25:32.061632 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:32.061794 (+   162us) service_pool.cc:224] Handling call
0709 12:25:32.062240 (+   446us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:32.104749 (+ 42509us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:32.104796 (+    47us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:32.148871 (+ 44075us) log.cc:844] Serialized 26 byte log entry
0709 12:25:32.612243 (+463372us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:25:32.612304 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:32.612398 (+    94us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:33.088973 (+476575us) raft_consensus.cc:1685] finished
0709 12:25:33.088994 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:33.089135 (+   141us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":47614,"raft.queue_time_us":42570,"raft.run_cpu_time_us":123,"raft.run_wall_time_us":39968,"thread_start_us":385701,"threads_started":1,"wal-append.queue_time_us":609674}
I20260709 12:25:33.100034 18333 leader_election.cc:304] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:25:33.101575 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 505355us
I20260709 12:25:33.077521 18363 raft_consensus.cc:2468] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 28.
I20260709 12:25:33.105216 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:33.106247 17761 raft_consensus.cc:2368] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 28. Current term is 29.
I20260709 12:25:33.114048 18333 leader_election.cc:304] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:25:33.117987 17801 leader_election.cc:304] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:33.119498 17801 leader_election.cc:304] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:33.120184 17837 raft_consensus.cc:2468] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 29.
I20260709 12:25:33.079108 28610 consensus_queue.cc:237] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 31.21, Last appended by leader: 21, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:33.122649 18363 raft_consensus.cc:1275] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 26 index: 17. Preceding OpId from leader: term: 27 index: 18. (index mismatch)
I20260709 12:25:33.122730 18333 leader_election.cc:304] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:33.123860 28426 raft_consensus.cc:493] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:33.133930 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 29 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:33.138262 17761 raft_consensus.cc:3060] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:33.139962 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 29 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:33.141300 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 29 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:33.142006 17837 raft_consensus.cc:3060] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Advancing to term 29
I20260709 12:25:33.047802 28502 leader_election.cc:290] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:25:33.048522 28543 leader_election.cc:290] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:33.110605 28526 raft_consensus.cc:2749] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260709 12:25:33.147531 17801 leader_election.cc:304] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:33.126430 28538 raft_consensus.cc:2804] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:25:33.161432 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 36 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:33.162432 17762 raft_consensus.cc:2393] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 36 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 36: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:25:33.126307 28305 raft_consensus.cc:2804] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:25:33.121896 28564 raft_consensus.cc:493] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:25:33.124850 28509 raft_consensus.cc:493] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:33.148249 28309 consensus_meta.cc:287] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.597s	user 0.011s	sys 0.000s
I20260709 12:25:33.136552 28183 raft_consensus.cc:2804] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:25:33.129302 28607 raft_consensus.cc:493] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:33.128626 28566 raft_consensus.cc:2749] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:25:33.126605 28490 leader_election.cc:290] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:33.145052 28609 raft_consensus.cc:3060] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Advancing to term 32
W20260709 12:25:33.168192 28559 consensus_meta.cc:287] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.855s	user 0.012s	sys 0.000s
I20260709 12:25:33.184026 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 34 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:25:33.184489 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:33.110956 28491 raft_consensus.cc:515] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 35 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:33.205467 28491 leader_election.cc:290] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 35 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:33.172647 28315 raft_consensus.cc:493] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:25:33.206146 28315 raft_consensus.cc:515] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:33.208295 28597 log.cc:927] Time spent T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 1.123s	user 0.000s	sys 0.003s
I20260709 12:25:33.192734 28303 leader_election.cc:290] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:25:33.192551 28309 raft_consensus.cc:515] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:33.212123 17836 raft_consensus.cc:2368] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 30. Current term is 31.
I20260709 12:25:33.213348 17727 leader_election.cc:304] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:33.214131 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 36 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:33.214265 28614 raft_consensus.cc:2804] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:25:33.214838 28614 raft_consensus.cc:697] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:25:33.215234 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 31 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:33.215402 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 30 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:33.215782 28614 consensus_queue.cc:237] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 27.19, Last appended by leader: 19, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:33.216473 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 35 candidate_status { last_received { term: 32 index: 23 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:33.217193 17833 raft_consensus.cc:2468] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 34.
I20260709 12:25:33.221390 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 31 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:33.222500 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:33.222636 18364 raft_consensus.cc:1275] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 26 index: 18. Preceding OpId from leader: term: 29 index: 19. (index mismatch)
I20260709 12:25:33.223028 18366 raft_consensus.cc:2468] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 26.
W20260709 12:25:33.224802 28486 consensus_meta.cc:287] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.598s	user 0.008s	sys 0.000s
I20260709 12:25:33.225250 28486 raft_consensus.cc:515] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:25:33.227421 28314 consensus_meta.cc:287] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.572s	user 0.008s	sys 0.001s
I20260709 12:25:33.194996 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:33.195668 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:33.195381 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 35 candidate_status { last_received { term: 32 index: 23 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:33.202979 18361 raft_consensus.cc:2468] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 33.
I20260709 12:25:33.212013 28309 leader_election.cc:290] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 31 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:33.192013 28183 raft_consensus.cc:493] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:25:33.176311 28538 raft_consensus.cc:697] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 25 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:25:33.163450 28519 leader_election.cc:290] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:25:33.254884 18363 raft_consensus.cc:2368] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 28. Current term is 29.
I20260709 12:25:33.211529 28313 raft_consensus.cc:515] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:33.190007 17761 raft_consensus.cc:2468] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 29.
I20260709 12:25:33.240082 17836 raft_consensus.cc:2393] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 31: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:25:33.230643 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:33.232817 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 36 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:33.287902 17831 raft_consensus.cc:2468] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 35.
I20260709 12:25:33.236114 28607 raft_consensus.cc:515] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:33.199213 17801 leader_election.cc:304] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:33.184650 18333 leader_election.cc:304] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:33.170859 28485 raft_consensus.cc:493] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:25:33.174286 28305 raft_consensus.cc:493] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:33.231519 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 35 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:33.296180 17829 raft_consensus.cc:2468] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 34.
I20260709 12:25:33.146927 28426 raft_consensus.cc:515] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:33.179968 28502 raft_consensus.cc:2804] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 32 FOLLOWER]: Leader pre-election won for term 33
W20260709 12:25:33.216379 28608 log.cc:927] Time spent T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.124s	user 0.000s	sys 0.000s
W20260709 12:25:33.240377 28612 log.cc:927] Time spent T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.069s	user 0.000s	sys 0.000s
I20260709 12:25:33.269754 18374 raft_consensus.cc:2368] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 28. Current term is 29.
I20260709 12:25:33.271746 17837 raft_consensus.cc:2468] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 29.
W20260709 12:25:33.277156 17652 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.536s	user 0.010s	sys 0.026s
W20260709 12:25:33.308322 17652 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.567s	user 0.010s	sys 0.026s
W20260709 12:25:33.308775 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 568050us
W20260709 12:25:33.276594 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 255231us
I20260709 12:25:33.227849 28314 raft_consensus.cc:515] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:33.254575 28530 raft_consensus.cc:493] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:25:33.258031 17654 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:33.317201 17652 leader_election.cc:304] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:33.290424 28545 leader_election.cc:290] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:33.302973 28519 raft_consensus.cc:2804] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:25:33.301431 28305 raft_consensus.cc:3060] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:25:33.289737 17761 raft_consensus.cc:1240] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 30. Current term is 31. Ops: []
W20260709 12:25:33.292410 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 107829us
I20260709 12:25:33.322870 17836 raft_consensus.cc:1240] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 26. Current term is 27. Ops: []
I20260709 12:25:33.288296 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 28 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:33.192416 28509 raft_consensus.cc:515] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:33.282047 28183 raft_consensus.cc:3060] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:33.305266 28309 raft_consensus.cc:2804] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:25:33.185935 28564 raft_consensus.cc:515] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:33.292063 28599 raft_consensus.cc:493] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:33.318625 28519 raft_consensus.cc:493] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 33 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:33.301821 28502 raft_consensus.cc:493] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:33.325882 18333 leader_election.cc:304] T a5732e71b9d54f8786b8c511d69943bf P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:25:33.298467 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 99428.6us
I20260709 12:25:33.319586 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 36 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:33.271624 28148 leader_election.cc:290] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:33.323607 17652 leader_election.cc:304] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:33.320132 28530 raft_consensus.cc:3060] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:33.331386 18371 raft_consensus.cc:1240] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 30. Current term is 31. Ops: []
I20260709 12:25:33.322211 28559 raft_consensus.cc:2804] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:25:33.319437 28486 leader_election.cc:290] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 32 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:33.302606 28485 raft_consensus.cc:515] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:33.371918 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 48333.4us
I20260709 12:25:33.372427 17762 raft_consensus.cc:2410] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for term 28 because replica has last-logged OpId of term: 26 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 18.
I20260709 12:25:33.342410 28519 raft_consensus.cc:3060] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:25:33.360124 17756 raft_consensus.cc:2468] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 35.
I20260709 12:25:33.331553 28545 consensus_queue.cc:1048] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 16, Time since last communication: 0.007s
I20260709 12:25:33.372833 17652 leader_election.cc:304] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 36 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:33.317735 28543 consensus_queue.cc:1048] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.041s
I20260709 12:25:33.353606 28502 raft_consensus.cc:3060] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:25:33.381753 17801 leader_election.cc:400] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 28 election: Vote denied by peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651) with higher term. Message: Invalid argument: T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 28. Current term is 29.
I20260709 12:25:33.386206 17652 leader_election.cc:304] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:25:33.341262 28309 raft_consensus.cc:697] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 31 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:25:33.392091 17801 leader_election.cc:403] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 28 election: Cancelling election due to peer responding with higher term
I20260709 12:25:33.352988 28303 consensus_queue.cc:1048] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 17, Time since last communication: 0.027s
I20260709 12:25:33.400318 18333 leader_election.cc:304] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:25:33.370802 28566 raft_consensus.cc:2804] T a5732e71b9d54f8786b8c511d69943bf P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:25:33.393981 17652 leader_election.cc:304] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:33.373121 28559 raft_consensus.cc:493] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 33 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:33.404615 28487 raft_consensus.cc:2804] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:25:33.353682 28538 consensus_queue.cc:237] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 24.16, Last appended by leader: 16, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:33.378664 28159 raft_consensus.cc:2804] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:25:33.411418 17832 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35040 (request call id 3907) took 1587 ms. Trace:
I20260709 12:25:33.414909 18333 leader_election.cc:304] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:33.354990 28599 raft_consensus.cc:515] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:33.419117 17832 rpcz_store.cc:276] 0709 12:25:31.823733 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:32.005281 (+181548us) service_pool.cc:224] Handling call
0709 12:25:33.411222 (+1405941us) spinlock_profiling.cc:243] Waited 297 ms on lock 0x7b54003340d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:25:33.411365 (+   143us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":681959168}
I20260709 12:25:33.388460 28545 consensus_queue.cc:1243] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer 334a4e1f59944544b3b3ff36616b57da log is divergent from this leader: its last log entry 26.17 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 16
I20260709 12:25:33.424304 18333 leader_election.cc:304] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:33.426074 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:33.433631 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 35 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:33.434046 17837 raft_consensus.cc:3060] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:25:33.420502 17652 leader_election.cc:304] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:25:33.436477 17652 leader_election.cc:304] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:33.392098 28618 raft_consensus.cc:2804] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 36 FOLLOWER]: Leader election won for term 36
I20260709 12:25:33.444792 18333 leader_election.cc:304] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
W20260709 12:25:33.445744 18334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:33.446557 18333 leader_election.cc:304] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:25:33.448272 18333 leader_election.cc:400] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 30 pre-election: Vote denied by peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837) with higher term. Message: Invalid argument: T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 30. Current term is 31.
I20260709 12:25:33.450133 18333 leader_election.cc:403] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 30 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:25:33.411125 28564 leader_election.cc:290] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:33.413025 28566 raft_consensus.cc:493] T a5732e71b9d54f8786b8c511d69943bf P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:33.427389 28559 raft_consensus.cc:3060] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:25:33.453052 18333 leader_election.cc:304] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:33.451333 28565 raft_consensus.cc:3060] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:33.457389 28561 consensus_queue.cc:1048] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.008s
I20260709 12:25:33.461848 28390 raft_consensus.cc:3060] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Advancing to term 31
I20260709 12:25:33.425253 28487 raft_consensus.cc:493] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:33.411415 28633 raft_consensus.cc:2749] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Leader election lost for term 28. Reason: Vote denied by peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651) with higher term. Message: Invalid argument: T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 28. Current term is 29.
I20260709 12:25:33.432109 28557 raft_consensus.cc:2804] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:25:33.442426 28159 raft_consensus.cc:697] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 28 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:25:33.428031 28490 raft_consensus.cc:2749] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:25:33.428354 28112 raft_consensus.cc:2749] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:25:33.480785 17832 raft_consensus.cc:2468] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 35.
W20260709 12:25:33.441584 28615 log.cc:927] Time spent T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.264s	user 0.001s	sys 0.000s
I20260709 12:25:33.454579 28526 raft_consensus.cc:2749] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:25:33.455454 28536 raft_consensus.cc:2804] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:25:33.458319 28416 raft_consensus.cc:2804] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:25:33.503258 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 31 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
W20260709 12:25:33.504140 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 188157us
I20260709 12:25:33.462018 28618 raft_consensus.cc:697] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 36 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:25:33.467140 28566 raft_consensus.cc:3060] T a5732e71b9d54f8786b8c511d69943bf P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:33.461500 28274 raft_consensus.cc:2804] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:25:33.513854 17803 leader_election.cc:304] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:33.513973 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 29 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:33.534647 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 29 candidate_status { last_received { term: 28 index: 22 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:33.493389 28557 raft_consensus.cc:493] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:25:33.538916 17803 leader_election.cc:304] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 35 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:33.493181 28487 raft_consensus.cc:3060] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:33.519563 17837 raft_consensus.cc:2468] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 32.
I20260709 12:25:33.546860 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 33 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:33.513891 28543 raft_consensus.cc:493] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:33.550297 28543 raft_consensus.cc:515] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:33.552618 28543 leader_election.cc:290] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:33.557264 17654 leader_election.cc:304] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:33.560408 17832 raft_consensus.cc:2410] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for term 33 because replica has last-logged OpId of term: 31 index: 21, which is greater than that of the candidate, which has last-logged OpId of term: 30 index: 20.
W20260709 12:25:33.549777 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 96929.6us
I20260709 12:25:33.562193 17727 leader_election.cc:304] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:33.562369 18333 leader_election.cc:304] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:33.516433 28416 raft_consensus.cc:493] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:33.565302 17727 leader_election.cc:304] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:33.515514 28536 raft_consensus.cc:493] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:25:33.554414 28425 consensus_meta.cc:287] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.758s	user 0.008s	sys 0.001s
I20260709 12:25:33.522962 28313 leader_election.cc:290] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 30 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:33.581691 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:33.532187 28274 raft_consensus.cc:493] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:33.583544 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:33.579919 18333 leader_election.cc:304] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:25:33.548731 28557 raft_consensus.cc:3060] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:33.475988 28603 raft_consensus.cc:515] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:33.610213 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
W20260709 12:25:33.608843 17802 leader_election.cc:336] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:33.610383 17759 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38776 (request call id 3535) took 2656 ms. Trace:
I20260709 12:25:33.566771 28536 raft_consensus.cc:3060] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:33.576821 28416 raft_consensus.cc:3060] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:33.613821 17759 rpcz_store.cc:276] 0709 12:25:30.946670 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:31.265592 (+318922us) service_pool.cc:224] Handling call
0709 12:25:31.358672 (+ 93080us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:31.358918 (+   246us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:31.358954 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:31.399233 (+ 40279us) log.cc:844] Serialized 26 byte log entry
0709 12:25:33.078259 (+1679026us) spinlock_profiling.cc:243] Waited 13.2 ms on lock 0x7b4400022b4c. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b47459c85 00007f7b4745a6a2 00007f7b4a1cc1db 00007f7b4a1c7aaf 00007f7b4a1bb86e 00007f7b4a1babb6 00007f7b4a1ba7d8 00007f7b4a1c4221 00007f7b4a1c41d9 00007f7b4a1c4169
0709 12:25:33.602827 (+524568us) spinlock_profiling.cc:243] Waited 3.15 ms on lock 0x7b18004a2360. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b4a1c84bc 00007f7b4751648b 00007f7b503709a9 00007f7b4753269b 00007f7b4753093f 00007f7b47532c02 00007f7b4a1ba20b 00007f7b4a1bfd23 00007f7b4a1c00d1
0709 12:25:33.603004 (+   177us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:25:33.603046 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:33.603137 (+    91us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:33.603183 (+    46us) raft_consensus.cc:1685] finished
0709 12:25:33.603198 (+    15us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:33.603373 (+   175us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":37510784,"thread_start_us":635298,"threads_started":1,"wal-append.queue_time_us":618445}
I20260709 12:25:33.613307 17802 leader_election.cc:304] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
W20260709 12:25:33.623116 28515 consensus_meta.cc:287] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.643s	user 0.008s	sys 0.003s
I20260709 12:25:33.602684 28308 raft_consensus.cc:2804] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 35 FOLLOWER]: Leader election won for term 35
I20260709 12:25:33.623695 28637 consensus_queue.cc:1048] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:25:33.580760 28543 consensus_queue.cc:1059] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 27.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.044s
W20260709 12:25:33.625208 28624 log.cc:927] Time spent T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.219s	user 0.000s	sys 0.001s
I20260709 12:25:33.526818 28309 consensus_queue.cc:237] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 30.19, Last appended by leader: 19, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:33.581847 28506 raft_consensus.cc:2804] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:25:33.560248 28303 raft_consensus.cc:2749] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:25:33.583711 28126 raft_consensus.cc:2804] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:25:33.577026 28426 leader_election.cc:290] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:33.637048 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 34 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:33.656183 17837 raft_consensus.cc:2468] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 33.
I20260709 12:25:33.657428 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:33.658159 17837 raft_consensus.cc:2468] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 31.
I20260709 12:25:33.658849 28589 leader_election.cc:290] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:25:33.659118 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 30 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:33.581512 28112 raft_consensus.cc:2749] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:25:33.664489 28600 raft_consensus.cc:3055] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:25:33.664988 28600 raft_consensus.cc:740] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 28 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:25:33.665920 28600 consensus_queue.cc:260] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 28.22, Last appended by leader: 22, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:33.606703 28274 raft_consensus.cc:3060] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:33.667443 28600 raft_consensus.cc:3060] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Advancing to term 29
W20260709 12:25:33.668761 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 59973.4us
I20260709 12:25:33.669401 18333 leader_election.cc:304] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
W20260709 12:25:33.669497 17802 leader_election.cc:336] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:33.670811 18333 leader_election.cc:304] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:25:33.632411 28515 raft_consensus.cc:493] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:33.672148 28515 raft_consensus.cc:515] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:33.674265 28515 leader_election.cc:290] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:25:33.677052 28621 log.cc:927] Time spent T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.219s	user 0.000s	sys 0.001s
W20260709 12:25:33.678191 28631 log.cc:927] Time spent T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.114s	user 0.000s	sys 0.000s
I20260709 12:25:33.641191 28509 leader_election.cc:290] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:33.645958 28618 consensus_queue.cc:237] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 34.22, Last appended by leader: 22, Current term: 36, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:33.645783 28485 leader_election.cc:290] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:33.638335 28506 raft_consensus.cc:493] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 35 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:33.650056 28526 raft_consensus.cc:2804] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:25:33.605901 28507 raft_consensus.cc:493] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
W20260709 12:25:33.650800 28516 consensus_meta.cc:287] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 1.062s	user 0.005s	sys 0.005s
W20260709 12:25:33.651702 28320 consensus_meta.cc:287] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.831s	user 0.010s	sys 0.000s
I20260709 12:25:33.607540 28314 leader_election.cc:290] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 31 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:25:33.714654 17728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:33.644450 28126 raft_consensus.cc:493] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 34 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
W20260709 12:25:33.632395 28247 consensus_meta.cc:287] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 1.189s	user 0.012s	sys 0.004s
W20260709 12:25:33.654829 28495 consensus_meta.cc:287] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 1.239s	user 0.012s	sys 0.000s
I20260709 12:25:33.609783 28425 raft_consensus.cc:515] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:33.547297 28633 consensus_queue.cc:1059] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 28.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.108s
I20260709 12:25:33.669759 28636 raft_consensus.cc:2749] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:25:33.648233 28159 consensus_queue.cc:237] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 26.18, Last appended by leader: 18, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:33.612046 28561 raft_consensus.cc:2749] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:25:33.702293 17837 raft_consensus.cc:1240] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 32 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 31. Current term is 32. Ops: []
I20260709 12:25:33.659613 28589 raft_consensus.cc:493] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:25:33.652602 28409 raft_consensus.cc:493] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:33.641870 28308 raft_consensus.cc:697] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 35 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
W20260709 12:25:33.725118 17728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:33.726835 28126 raft_consensus.cc:3060] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 34 FOLLOWER]: Advancing to term 35
W20260709 12:25:33.710412 28540 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.693s	user 0.018s	sys 0.018s
I20260709 12:25:33.627789 28502 raft_consensus.cc:515] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:33.790299 17764 raft_consensus.cc:1240] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 28. Current term is 29. Ops: [28.19-28.19]
I20260709 12:25:33.725124 28313 raft_consensus.cc:493] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:33.709874 28506 raft_consensus.cc:3060] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:25:33.715061 28504 raft_consensus.cc:493] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:25:33.788300 17837 raft_consensus.cc:1275] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 21 index: 15. Preceding OpId from leader: term: 24 index: 17. (index mismatch)
I20260709 12:25:33.724846 28564 raft_consensus.cc:493] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:25:33.725819 28526 raft_consensus.cc:697] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 29 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:25:33.722716 28516 raft_consensus.cc:515] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:33.714962 28507 raft_consensus.cc:515] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:33.707741 28566 raft_consensus.cc:515] T a5732e71b9d54f8786b8c511d69943bf P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Starting leader election with config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:33.726667 28638 raft_consensus.cc:2804] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:25:33.775156 28247 raft_consensus.cc:515] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:33.777464 28607 leader_election.cc:290] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:25:33.789750 28540 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.790s	user 0.022s	sys 0.024s
W20260709 12:25:33.799086 28635 log.cc:927] Time spent T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.148s	user 0.001s	sys 0.000s
I20260709 12:25:33.791016 28409 raft_consensus.cc:515] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:33.707702 28112 raft_consensus.cc:2804] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:25:33.808879 17833 raft_consensus.cc:1240] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 28. Current term is 29. Ops: []
W20260709 12:25:33.812103 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 266240us
I20260709 12:25:33.780680 28495 raft_consensus.cc:515] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 34 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:33.808852 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 358989us
I20260709 12:25:33.742093 28320 raft_consensus.cc:515] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Starting leader election with config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:33.791525 28314 consensus_queue.cc:1059] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 31.22, Next index: 23, Last known committed idx: 22, Time since last communication: 0.019s
I20260709 12:25:33.799847 28638 raft_consensus.cc:493] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:25:33.789361 28589 raft_consensus.cc:515] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:33.814762 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 255668us
I20260709 12:25:33.829373 17652 leader_election.cc:304] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:33.831662 17803 leader_election.cc:304] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:25:33.833385 17803 leader_election.cc:304] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:33.811810 28112 raft_consensus.cc:493] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 33 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:33.796797 28313 raft_consensus.cc:515] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:33.799499 28564 raft_consensus.cc:515] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:33.794708 28305 raft_consensus.cc:515] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:33.845163 17803 leader_election.cc:304] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:25:33.845830 17654 leader_election.cc:304] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:33.849573 17803 leader_election.cc:400] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 28 pre-election: Vote denied by peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577) with higher term. Message: Invalid argument: T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 28. Current term is 29.
I20260709 12:25:33.855952 17803 leader_election.cc:403] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 28 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:25:33.846864 18365 raft_consensus.cc:1240] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 34 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 33. Current term is 34. Ops: []
I20260709 12:25:33.800959 28504 raft_consensus.cc:515] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:33.858212 17803 leader_election.cc:400] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 28 election: Vote denied by peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577) with higher term. Message: Invalid argument: T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 28. Current term is 29.
W20260709 12:25:33.858855 17802 consensus_peers.cc:597] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:25:33.859519 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 188291us
I20260709 12:25:33.828123 28638 raft_consensus.cc:3060] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:25:33.878167 17654 leader_election.cc:304] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:33.847949 28112 raft_consensus.cc:3060] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:25:33.881039 17654 leader_election.cc:304] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:33.882145 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 32 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:33.884989 28409 leader_election.cc:290] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:33.885557 28526 consensus_queue.cc:237] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 27.19, Last appended by leader: 19, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:33.887383 28407 consensus_meta.cc:287] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 1.167s	user 0.011s	sys 0.000s
I20260709 12:25:33.887997 28607 raft_consensus.cc:2749] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:25:33.888499 28643 raft_consensus.cc:493] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:25:33.888762 28642 raft_consensus.cc:3060] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:33.888656 28112 raft_consensus.cc:515] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 34 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:33.889112 28643 raft_consensus.cc:515] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:33.890542 28561 raft_consensus.cc:2804] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:25:33.891029 28561 raft_consensus.cc:493] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:25:33.891462 28561 raft_consensus.cc:3060] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:33.891706 28112 leader_election.cc:290] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 34 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:25:33.861361 17802 leader_election.cc:336] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:33.871490 17652 leader_election.cc:304] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:33.861199 28538 raft_consensus.cc:2749] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Leader pre-election lost for term 28. Reason: Vote denied by peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577) with higher term. Message: Invalid argument: T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 28. Current term is 29.
I20260709 12:25:33.892195 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 33 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:33.892982 17654 leader_election.cc:304] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:33.865628 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:33.847975 28545 raft_consensus.cc:2749] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:25:33.897023 17652 leader_election.cc:304] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 35 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:25:33.894094 17802 leader_election.cc:304] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:25:33.850281 28613 raft_consensus.cc:3055] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 31 LEADER]: Stepping down as leader of term 31
I20260709 12:25:33.849365 28636 raft_consensus.cc:2804] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:25:33.907753 17654 leader_election.cc:304] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:33.909147 17652 leader_election.cc:304] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:25:33.924906 17652 leader_election.cc:304] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:33.925272 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 30 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:33.926218 18374 raft_consensus.cc:1275] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 23 index: 17. Preceding OpId from leader: term: 24 index: 18. (index mismatch)
W20260709 12:25:33.893648 28519 consensus_meta.cc:287] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.520s	user 0.010s	sys 0.000s
W20260709 12:25:33.915445 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:33.917122 17831 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35042 (request call id 3721) took 1331 ms. Trace:
I20260709 12:25:33.930781 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:33.896281 28607 raft_consensus.cc:2749] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:25:33.913494 17830 raft_consensus.cc:1240] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 27. Current term is 28. Ops: []
I20260709 12:25:33.855654 28309 raft_consensus.cc:2804] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:25:33.929041 17654 leader_election.cc:304] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:33.952481 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 31 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:33.931278 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:33.955253 17831 rpcz_store.cc:276] 0709 12:25:32.585307 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:33.301852 (+716545us) service_pool.cc:224] Handling call
0709 12:25:33.313432 (+ 11580us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:33.321035 (+  7603us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:33.321096 (+    61us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:33.332153 (+ 11057us) log.cc:844] Serialized 26 byte log entry
0709 12:25:33.438876 (+106723us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:25:33.438939 (+    63us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:33.439032 (+    93us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:33.916921 (+477889us) raft_consensus.cc:1685] finished
0709 12:25:33.916941 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:33.917068 (+   127us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":9013,"raft.run_cpu_time_us":172,"raft.run_wall_time_us":12457,"thread_start_us":44926,"threads_started":1,"wal-append.queue_time_us":173729}
I20260709 12:25:33.958874 18368 raft_consensus.cc:2368] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 28. Current term is 29.
W20260709 12:25:33.929301 18334 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.16.29.65:33071 timed out after 1.519s (SENT) [suppressed 52 similar messages]
I20260709 12:25:33.917536 17803 leader_election.cc:304] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:33.848104 28610 consensus_queue.cc:1048] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.069s
I20260709 12:25:33.892318 28540 raft_consensus.cc:2749] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
W20260709 12:25:33.961691 17801 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:33.904919 28603 leader_election.cc:290] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 28 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:25:33.963326 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 101993us
I20260709 12:25:33.894160 28504 leader_election.cc:290] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:25:33.973029 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 55953.5us
I20260709 12:25:33.919390 28636 raft_consensus.cc:493] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:25:33.996435 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 34 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:33.891846 28643 leader_election.cc:290] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:33.916249 28613 raft_consensus.cc:740] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 31 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:25:33.997174 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 29 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:33.912735 28485 raft_consensus.cc:3060] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:33.931227 28491 raft_consensus.cc:2804] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 32 FOLLOWER]: Leader election won for term 32
I20260709 12:25:33.839650 28308 consensus_queue.cc:237] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 34.23, Last appended by leader: 23, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:33.931371 28509 raft_consensus.cc:2749] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260709 12:25:33.998167 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 33 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:34.002076 28607 raft_consensus.cc:493] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:33.953116 28490 raft_consensus.cc:3060] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:33.887792 28407 raft_consensus.cc:515] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:25:33.962529 28578 consensus_meta.cc:287] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 1.517s	user 0.014s	sys 0.000s
W20260709 12:25:33.970748 17802 consensus_peers.cc:597] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:25:34.024585 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 53919.7us
I20260709 12:25:33.924427 28409 raft_consensus.cc:2804] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 35 FOLLOWER]: Leader election won for term 35
I20260709 12:25:33.919130 28515 consensus_queue.cc:1048] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.078s
I20260709 12:25:34.012617 17830 raft_consensus.cc:2468] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 28.
I20260709 12:25:33.955788 28519 raft_consensus.cc:515] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 34 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:33.931545 28148 raft_consensus.cc:2804] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:25:34.007436 28538 raft_consensus.cc:2749] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
I20260709 12:25:33.968775 28309 raft_consensus.cc:493] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 33 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:25:34.025317 17802 leader_election.cc:336] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 25 election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:34.033905 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:25:34.034420 17802 leader_election.cc:336] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 35 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:34.027349 17760 raft_consensus.cc:2468] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 32.
I20260709 12:25:34.009696 28636 raft_consensus.cc:3060] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:34.034788 17802 leader_election.cc:304] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:25:34.034163 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 105229us
W20260709 12:25:34.041517 17802 leader_election.cc:336] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 33 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:34.042495 17654 leader_election.cc:304] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:25:34.043041 17802 leader_election.cc:336] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 28 election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:34.046586 17830 raft_consensus.cc:1240] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 31 LEADER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 30. Current term is 31. Ops: []
I20260709 12:25:34.047156 17802 leader_election.cc:304] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:25:34.009788 28633 consensus_queue.cc:1048] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.024s
I20260709 12:25:34.022891 28502 leader_election.cc:290] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 33 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:25:34.025895 28247 leader_election.cc:290] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 30 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:25:34.050335 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 96982.8us
W20260709 12:25:34.051578 17802 leader_election.cc:336] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:34.058341 17802 leader_election.cc:304] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:25:34.060401 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 131134us
W20260709 12:25:34.060981 17728 leader_election.cc:336] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 29 election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Timed out: RequestConsensusVote RPC to 127.16.29.65:33071 timed out after 1.508s (SENT)
W20260709 12:25:34.061736 17728 leader_election.cc:336] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 27 election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Timed out: RequestConsensusVote RPC to 127.16.29.65:33071 timed out after 1.511s (SENT)
I20260709 12:25:34.062340 17728 leader_election.cc:304] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
W20260709 12:25:34.064129 17728 leader_election.cc:336] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 31 election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Timed out: RequestConsensusVote RPC to 127.16.29.65:33071 timed out after 1.522s (SENT)
W20260709 12:25:34.066030 17728 leader_election.cc:336] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 31 election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Timed out: RequestConsensusVote RPC to 127.16.29.65:33071 timed out after 1.531s (SENT)
W20260709 12:25:34.066129 18334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:34.038658 28309 raft_consensus.cc:3060] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:25:34.033130 28491 raft_consensus.cc:697] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 32 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:25:34.068606 17654 leader_election.cc:304] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:25:34.037053 28645 raft_consensus.cc:2749] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:25:34.027235 28607 raft_consensus.cc:515] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:34.018271 28641 consensus_queue.cc:1059] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 27.19, Next index: 20, Last known committed idx: 17, Time since last communication: 0.047s
I20260709 12:25:34.044204 28054 raft_consensus.cc:2749] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
W20260709 12:25:34.077643 17802 consensus_peers.cc:597] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:25:34.055140 28538 raft_consensus.cc:2749] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260709 12:25:34.057255 28516 leader_election.cc:290] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 29 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:34.048668 28409 raft_consensus.cc:697] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 35 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:25:34.089635 28651 raft_consensus.cc:2749] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:25:34.085388 28645 raft_consensus.cc:2749] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
W20260709 12:25:34.090708 28648 log.cc:927] Time spent T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.065s	user 0.000s	sys 0.000s
W20260709 12:25:34.090914 28604 log.cc:927] Time spent T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 1.063s	user 0.005s	sys 0.000s
I20260709 12:25:34.091091 28607 leader_election.cc:290] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:25:34.091607 28647 log.cc:927] Time spent T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.085s	user 0.000s	sys 0.000s
W20260709 12:25:34.084710 17802 leader_election.cc:336] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 33 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:34.059970 28495 leader_election.cc:290] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 34 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:25:34.068076 28578 raft_consensus.cc:2749] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
W20260709 12:25:34.091729 28646 log.cc:927] Time spent T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.068s	user 0.000s	sys 0.000s
I20260709 12:25:34.074011 28519 leader_election.cc:290] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 34 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:25:34.124508 28554 leader_election.cc:290] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 28 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:34.125417 28409 consensus_queue.cc:237] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 34.20, Last appended by leader: 20, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:25:34.127768 28594 log.cc:927] Time spent T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 2.043s	user 0.004s	sys 0.000s
I20260709 12:25:34.177570 28501 raft_consensus.cc:3055] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:25:34.170428 17729 leader_election.cc:400] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 29 election: Vote denied by peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577) with higher term. Message: Invalid argument: T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for earlier term 29. Current term is 30.
I20260709 12:25:34.195448 28501 raft_consensus.cc:740] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 28 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:25:34.169549 17833 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37994 (request call id 1985) took 1728 ms. Trace:
I20260709 12:25:34.196183 17833 rpcz_store.cc:276] 0709 12:25:32.441389 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:33.217717 (+776328us) service_pool.cc:224] Handling call
0709 12:25:33.217989 (+   272us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:33.815713 (+597724us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:34.169285 (+353572us) spinlock_profiling.cc:243] Waited 591 ms on lock 0x7b540028d054. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:25:34.169469 (+   184us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1358304128}
I20260709 12:25:34.196683 28501 consensus_queue.cc:260] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 28.15, Last appended by leader: 15, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:34.128509 28523 leader_election.cc:290] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:25:34.195492 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 110835us
I20260709 12:25:34.080768 28509 raft_consensus.cc:2804] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Leader pre-election won for term 28
W20260709 12:25:34.198568 17802 leader_election.cc:336] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 34 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:34.198872 28501 raft_consensus.cc:3060] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:34.067641 28613 consensus_queue.cc:260] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 31.22, Last appended by leader: 22, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:34.199715 17802 leader_election.cc:336] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:34.199999 28613 raft_consensus.cc:3060] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Advancing to term 32
W20260709 12:25:34.200691 17802 leader_election.cc:336] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:34.201171 17802 leader_election.cc:304] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:34.130352 18365 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 3054) took 1830 ms. Trace:
I20260709 12:25:34.186419 17758 raft_consensus.cc:1275] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 35 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 34 index: 19. Preceding OpId from leader: term: 35 index: 20. (index mismatch)
I20260709 12:25:34.086848 28313 leader_election.cc:290] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:34.201792 18365 rpcz_store.cc:276] 0709 12:25:32.299563 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:32.302046 (+  2483us) service_pool.cc:224] Handling call
0709 12:25:32.302318 (+   272us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:33.857571 (+1555253us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:34.130092 (+272521us) spinlock_profiling.cc:243] Waited 1.54 s on lock 0x7b5400466ed4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:25:34.130287 (+   195us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3543420800}
I20260709 12:25:34.152153 28603 raft_consensus.cc:493] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:34.158831 28566 leader_election.cc:290] T a5732e71b9d54f8786b8c511d69943bf P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 31 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:34.088503 28644 raft_consensus.cc:2804] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:25:34.162218 28407 leader_election.cc:290] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 32 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:34.197391 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 30 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:34.128093 28650 raft_consensus.cc:493] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:25:34.179387 17763 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38776 (request call id 3548) took 1679 ms. Trace:
W20260709 12:25:34.128497 28565 consensus_meta.cc:287] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.654s	user 0.008s	sys 0.004s
I20260709 12:25:34.073896 28491 consensus_queue.cc:237] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 30.18, Last appended by leader: 18, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:34.239754 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:34.240962 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 32 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:34.241537 18365 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 31.
I20260709 12:25:34.242354 28309 raft_consensus.cc:515] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 34 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:34.086385 28305 leader_election.cc:290] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 33 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:25:34.220507 28390 consensus_meta.cc:287] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.728s	user 0.007s	sys 0.006s
I20260709 12:25:34.195574 17729 leader_election.cc:403] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 29 election: Cancelling election due to peer responding with higher term
I20260709 12:25:34.096024 28589 leader_election.cc:290] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:25:34.190985 28487 consensus_meta.cc:287] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.584s	user 0.009s	sys 0.000s
I20260709 12:25:34.243669 28390 raft_consensus.cc:2749] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Leader pre-election lost for term 30. Reason: Vote denied by peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837) with higher term. Message: Invalid argument: T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 30. Current term is 31.
I20260709 12:25:34.244206 28487 raft_consensus.cc:515] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:34.246757 28487 leader_election.cc:290] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 27 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:34.182423 28507 leader_election.cc:290] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:34.175715 28425 leader_election.cc:290] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 29 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:34.248006 28313 raft_consensus.cc:2749] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Leader election lost for term 29. Reason: Vote denied by peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577) with higher term. Message: Invalid argument: T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for earlier term 29. Current term is 30.
W20260709 12:25:34.249106 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 78845.3us
W20260709 12:25:34.159631 28530 consensus_meta.cc:287] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.784s	user 0.003s	sys 0.007s
I20260709 12:25:34.230155 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 30 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:25:34.210291 28609 consensus_meta.cc:287] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.944s	user 0.013s	sys 0.001s
I20260709 12:25:34.229055 17763 rpcz_store.cc:276] 0709 12:25:32.499513 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:32.509462 (+  9949us) service_pool.cc:224] Handling call
0709 12:25:34.179084 (+1669622us) spinlock_profiling.cc:243] Waited 533 ms on lock 0x7b54003e3fd4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:25:34.179306 (+   222us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1226764160}
I20260709 12:25:34.233464 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 29 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:34.191992 28303 raft_consensus.cc:3055] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:25:34.242830 18365 pending_rounds.cc:85] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da: Aborting all ops after (but not including) 16
I20260709 12:25:34.198602 28509 raft_consensus.cc:493] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:25:34.249147 17728 leader_election.cc:336] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Timed out: RequestConsensusVote RPC to 127.16.29.65:33071 timed out after 1.854s (SENT)
I20260709 12:25:34.250097 17835 raft_consensus.cc:2468] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 29.
W20260709 12:25:34.226970 17802 leader_election.cc:336] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 36 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:25:34.219529 28183 consensus_meta.cc:287] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.875s	user 0.011s	sys 0.000s
I20260709 12:25:34.172519 28320 leader_election.cc:290] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 31 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:34.250931 17729 leader_election.cc:304] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:25:34.218637 28495 consensus_queue.cc:1059] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 30.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.078s
I20260709 12:25:34.165063 28599 leader_election.cc:290] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:34.185494 28630 raft_consensus.cc:493] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:34.307829 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 29 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:25:34.336352 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:34.284327 18371 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 3055) took 1348 ms. Trace:
I20260709 12:25:34.307062 17755 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38786 (request call id 4337) took 2501 ms. Trace:
I20260709 12:25:34.299193 17831 raft_consensus.cc:2393] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 34 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 34: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:25:34.307161 17758 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38786 (request call id 4353) took 1261 ms. Trace:
I20260709 12:25:34.316951 28313 raft_consensus.cc:2804] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Leader election won for term 31
W20260709 12:25:34.316774 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 65927.6us
I20260709 12:25:34.342646 17833 raft_consensus.cc:2410] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for term 30 because replica has last-logged OpId of term: 28 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 27 index: 14.
I20260709 12:25:34.342730 17729 leader_election.cc:304] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
W20260709 12:25:34.317796 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 90891.2us
I20260709 12:25:34.338913 18371 rpcz_store.cc:276] 0709 12:25:32.936327 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:32.936478 (+   151us) service_pool.cc:224] Handling call
0709 12:25:32.962888 (+ 26410us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:33.364254 (+401366us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:34.284031 (+919777us) spinlock_profiling.cc:243] Waited 368 ms on lock 0x7b5400802d54. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:25:34.284252 (+   221us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":846983552}
I20260709 12:25:34.318545 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 27 candidate_status { last_received { term: 25 index: 22 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:34.341423 18366 raft_consensus.cc:2468] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 31.
I20260709 12:25:34.344187 17837 raft_consensus.cc:3060] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:25:34.318959 28557 consensus_meta.cc:287] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.676s	user 0.012s	sys 0.000s
I20260709 12:25:34.319144 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 31 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:34.351384 17838 raft_consensus.cc:2393] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 31: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:25:34.319744 17652 leader_election.cc:304] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:25:34.320953 17835 raft_consensus.cc:1240] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 29. Current term is 30. Ops: []
I20260709 12:25:34.321141 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:34.321762 28183 raft_consensus.cc:515] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:34.252556 28506 raft_consensus.cc:515] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 36 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:25:34.309057 28606 log.cc:927] Time spent T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 1.302s	user 0.001s	sys 0.003s
I20260709 12:25:34.324923 28509 raft_consensus.cc:3060] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Advancing to term 28
W20260709 12:25:34.325672 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 76563.4us
I20260709 12:25:34.326021 28303 raft_consensus.cc:740] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 28 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:25:34.316620 18365 pending_rounds.cc:107] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da: Aborting uncommitted NO_OP operation due to leader change: 26.17
I20260709 12:25:34.219260 28519 consensus_queue.cc:1059] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 28.19, Next index: 20, Last known committed idx: 18, Time since last communication: 0.071s
I20260709 12:25:34.225246 28644 raft_consensus.cc:493] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:34.218068 28503 raft_consensus.cc:2749] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
W20260709 12:25:34.332131 18334 consensus_peers.cc:597] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:25:34.238840 28536 consensus_meta.cc:287] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.579s	user 0.007s	sys 0.004s
I20260709 12:25:34.345101 28495 consensus_queue.cc:1243] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer e063541542a94969898868bf6eb5ce1b log is divergent from this leader: its last log entry 31.20 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 19
I20260709 12:25:34.339278 17758 rpcz_store.cc:276] 0709 12:25:33.045683 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:33.072185 (+ 26502us) service_pool.cc:224] Handling call
0709 12:25:34.185814 (+1113629us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:34.201961 (+ 16147us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:34.306955 (+104994us) spinlock_profiling.cc:243] Waited 1.11 s on lock 0x7b54003e3fd0. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a61f8ad 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:25:34.307096 (+   141us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2560314240}
W20260709 12:25:34.373353 18334 leader_election.cc:336] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:25:34.373478 17728 leader_election.cc:336] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Timed out: RequestConsensusVote RPC to 127.16.29.65:33071 timed out after 2.054s (SENT)
W20260709 12:25:34.374260 18334 leader_election.cc:336] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:34.374655 18334 leader_election.cc:304] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:34.338951 17755 rpcz_store.cc:276] 0709 12:25:31.805225 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:31.805425 (+   200us) service_pool.cc:224] Handling call
0709 12:25:31.805894 (+   469us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:32.004130 (+198236us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:32.004176 (+    46us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:32.048850 (+ 44674us) log.cc:844] Serialized 26 byte log entry
0709 12:25:32.605223 (+556373us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:25:32.605263 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:33.068012 (+462749us) spinlock_profiling.cc:243] Waited 198 ms on lock 0x7b540019f5d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:25:33.068057 (+    45us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:33.952212 (+884155us) spinlock_profiling.cc:243] Waited 8.67 ms on lock 0x7b4400022b4c. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b47459c85 00007f7b4745a6a2 00007f7b4a1cc1db 00007f7b4a1c7aaf 00007f7b4a1bb86e 00007f7b4a1babb6 00007f7b4a1ba7d8 00007f7b4a1c4221 00007f7b4a1c41d9 00007f7b4a1c4169
0709 12:25:34.306742 (+354530us) raft_consensus.cc:1685] finished
0709 12:25:34.306776 (+    34us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:34.306974 (+   198us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":15695,"raft.queue_time_us":574,"raft.run_cpu_time_us":143,"raft.run_wall_time_us":36456,"spinlock_wait_cycles":474813184,"thread_start_us":426458,"threads_started":1,"wal-append.queue_time_us":690071}
I20260709 12:25:34.336341 18367 raft_consensus.cc:2468] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 28.
I20260709 12:25:34.257248 28530 raft_consensus.cc:515] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:25:34.255021 28274 consensus_meta.cc:287] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.584s	user 0.010s	sys 0.000s
I20260709 12:25:34.253965 28565 raft_consensus.cc:2749] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:25:34.356564 17834 raft_consensus.cc:2368] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for earlier term 32. Current term is 33.
I20260709 12:25:34.284174 28609 raft_consensus.cc:2749] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 32 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
W20260709 12:25:34.342777 28416 consensus_meta.cc:287] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.653s	user 0.008s	sys 0.004s
I20260709 12:25:34.360574 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 30 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:34.366508 18365 raft_consensus.cc:2889] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:34.234321 28603 raft_consensus.cc:515] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:34.370558 28644 raft_consensus.cc:3060] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Advancing to term 30
W20260709 12:25:34.380506 18334 leader_election.cc:336] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:34.250036 28650 raft_consensus.cc:515] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:34.382639 17837 raft_consensus.cc:2468] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 27.
I20260709 12:25:34.359297 28664 raft_consensus.cc:2804] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:25:34.400926 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 30 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:34.401532 28664 raft_consensus.cc:493] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:34.401991 28664 raft_consensus.cc:3060] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:34.402242 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 37 candidate_status { last_received { term: 35 index: 26 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:34.403314 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 29 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:34.404119 28303 consensus_queue.cc:260] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 28.21, Last appended by leader: 21, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:34.405185 28303 raft_consensus.cc:3060] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:34.406697 28506 leader_election.cc:290] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 36 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:34.407619 28650 leader_election.cc:290] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:25:34.408250 17802 consensus_peers.cc:597] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:25:34.410763 17802 consensus_peers.cc:597] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:25:34.412266 17802 leader_election.cc:336] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 36 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:34.384495 18371 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 29.
I20260709 12:25:34.413010 28309 leader_election.cc:290] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 34 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:34.413460 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 31 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:34.381259 28390 raft_consensus.cc:2749] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:25:34.383378 17729 leader_election.cc:304] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:34.370391 28519 raft_consensus.cc:3055] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:25:34.415205 28519 raft_consensus.cc:740] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
W20260709 12:25:34.365187 28126 consensus_meta.cc:287] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.571s	user 0.001s	sys 0.011s
W20260709 12:25:34.388916 18334 leader_election.cc:336] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 30 election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:34.416317 18363 raft_consensus.cc:2468] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 32.
I20260709 12:25:34.416529 18334 leader_election.cc:304] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:34.417398 28589 raft_consensus.cc:2749] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
W20260709 12:25:34.418929 17802 leader_election.cc:336] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:34.359843 28557 raft_consensus.cc:515] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:34.424649 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 32 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:34.363265 28313 raft_consensus.cc:697] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 31 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
W20260709 12:25:34.426087 18334 leader_election.cc:336] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:34.426256 18374 raft_consensus.cc:2393] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 29: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:25:34.428399 18367 raft_consensus.cc:2468] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 31.
I20260709 12:25:34.370882 28578 raft_consensus.cc:3060] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:25:34.431593 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 48317.7us
W20260709 12:25:34.383248 17728 leader_election.cc:336] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:25:34.423978 17802 leader_election.cc:336] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:34.428120 28665 raft_consensus.cc:493] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:34.442873 28504 consensus_queue.cc:1059] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.000s
I20260709 12:25:34.443176 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 34 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:34.443737 28148 raft_consensus.cc:493] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 34 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:25:34.443981 18367 raft_consensus.cc:2393] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 34 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 34: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:25:34.444254 28148 raft_consensus.cc:3060] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:25:34.388320 28416 raft_consensus.cc:515] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:34.450071 17832 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37994 (request call id 2003) took 1729 ms. Trace:
I20260709 12:25:34.450956 28519 consensus_queue.cc:260] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 28.19, Last appended by leader: 19, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:34.452283 28519 raft_consensus.cc:3060] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:34.454030 18362 raft_consensus.cc:2468] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 29.
I20260709 12:25:34.457337 17836 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37994 (request call id 1987) took 2015 ms. Trace:
W20260709 12:25:34.457701 28638 consensus_meta.cc:287] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.568s	user 0.006s	sys 0.003s
I20260709 12:25:34.458173 18364 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59902 (request call id 2480) took 2152 ms. Trace:
I20260709 12:25:34.458117 28638 raft_consensus.cc:515] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 32 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:34.458595 18364 rpcz_store.cc:276] 0709 12:25:32.305335 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:32.308207 (+  2872us) service_pool.cc:224] Handling call
0709 12:25:33.222345 (+914138us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:33.222952 (+   607us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:34.457959 (+1235007us) spinlock_profiling.cc:243] Waited 914 ms on lock 0x7b540034b550. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a61f8ad 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:25:34.458094 (+   135us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2101735808}
I20260709 12:25:34.442026 18371 raft_consensus.cc:1240] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 28. Current term is 29. Ops: []
I20260709 12:25:34.442298 28425 raft_consensus.cc:3060] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Advancing to term 29
W20260709 12:25:34.461323 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 81832.8us
W20260709 12:25:34.426578 28559 consensus_meta.cc:287] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.895s	user 0.013s	sys 0.000s
I20260709 12:25:34.461050 17803 leader_election.cc:400] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 28 pre-election: Vote denied by peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577) with higher term. Message: Invalid argument: T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 28. Current term is 29.
I20260709 12:25:34.345280 28630 raft_consensus.cc:515] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:34.376471 28536 raft_consensus.cc:515] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:34.457705 17836 rpcz_store.cc:276] 0709 12:25:32.442067 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:33.292658 (+850591us) service_pool.cc:224] Handling call
0709 12:25:33.292935 (+   277us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:33.327209 (+ 34274us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:34.457133 (+1129924us) spinlock_profiling.cc:243] Waited 29.7 ms on lock 0x7b54003359d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:25:34.457284 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":68412160}
W20260709 12:25:34.466028 17728 leader_election.cc:336] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:34.430060 18334 leader_election.cc:304] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:34.614112 17728 leader_election.cc:304] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
W20260709 12:25:34.615115 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 154169us
I20260709 12:25:34.618000 17803 leader_election.cc:304] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3
W20260709 12:25:34.614408 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 190451us
I20260709 12:25:34.450467 17832 rpcz_store.cc:276] 0709 12:25:32.720780 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:33.673773 (+952993us) service_pool.cc:224] Handling call
0709 12:25:33.674219 (+   446us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:33.789223 (+115004us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:33.789265 (+    42us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:33.793008 (+  3743us) log.cc:844] Serialized 26 byte log entry
0709 12:25:34.019631 (+226623us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:25:34.019675 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:34.449682 (+430007us) spinlock_profiling.cc:243] Waited 105 ms on lock 0x7b5400334fd4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:25:34.449726 (+    44us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:34.449769 (+    43us) raft_consensus.cc:1685] finished
0709 12:25:34.449783 (+    14us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:34.450013 (+   230us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5199,"raft.queue_time_us":5394,"raft.run_cpu_time_us":160,"raft.run_wall_time_us":4260,"spinlock_wait_cycles":242328704,"thread_start_us":222190,"threads_started":1,"wal-append.queue_time_us":333309}
W20260709 12:25:34.615811 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 149804us
I20260709 12:25:34.625074 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:34.644186 17836 raft_consensus.cc:2393] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 22: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:25:34.427678 28485 raft_consensus.cc:2749] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
W20260709 12:25:34.649024 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 222949us
W20260709 12:25:34.636955 17728 consensus_peers.cc:597] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:25:34.652503 18334 leader_election.cc:336] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:25:34.645049 17802 leader_election.cc:336] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:34.651121 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 29 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
W20260709 12:25:34.653435 18334 leader_election.cc:336] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 34 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:34.651232 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:34.442325 28313 consensus_queue.cc:237] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 29.16, Last appended by leader: 16, Current term: 31, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:34.644397 28559 raft_consensus.cc:515] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 34 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:34.659914 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:34.660751 17832 raft_consensus.cc:3060] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Advancing to term 31
W20260709 12:25:34.652608 17728 consensus_peers.cc:597] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:25:34.665072 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 47163.8us
W20260709 12:25:34.657861 17802 leader_election.cc:336] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:25:34.666920 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 233999us
I20260709 12:25:34.671651 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 25 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:25:34.676365 18334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:34.689817 18334 leader_election.cc:336] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 37 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:34.691320 18334 leader_election.cc:304] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:34.488976 28315 leader_election.cc:290] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:25:34.670784 17728 consensus_peers.cc:597] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:25:34.698277 17836 raft_consensus.cc:2468] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 24.
I20260709 12:25:34.617014 28487 raft_consensus.cc:493] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:34.620918 28506 raft_consensus.cc:2749] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:25:34.471891 28650 raft_consensus.cc:3055] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 30 LEADER]: Stepping down as leader of term 30
W20260709 12:25:34.696148 17802 leader_election.cc:336] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 31 election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:34.431480 28565 consensus_queue.cc:1059] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 33.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.238s
I20260709 12:25:34.705962 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:25:34.708881 18334 leader_election.cc:336] T a5732e71b9d54f8786b8c511d69943bf P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:34.730432 17829 raft_consensus.cc:2468] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 30.
W20260709 12:25:34.732733 18334 leader_election.cc:336] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:25:34.734086 18334 leader_election.cc:336] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 29 election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:34.429942 28126 raft_consensus.cc:515] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 35 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:34.652330 28515 raft_consensus.cc:493] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
W20260709 12:25:34.722870 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 51695.6us
I20260709 12:25:34.443861 28504 consensus_queue.cc:1048] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.220s
I20260709 12:25:34.662358 28495 raft_consensus.cc:2749] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:25:34.703900 28530 leader_election.cc:290] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 26 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:34.755546 28416 leader_election.cc:290] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 27 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:34.697163 28503 raft_consensus.cc:493] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:34.431612 28407 consensus_queue.cc:1059] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 30.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.273s
I20260709 12:25:34.386883 28659 consensus_queue.cc:1059] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 29.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.321s
I20260709 12:25:34.383627 28274 raft_consensus.cc:515] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:34.669981 28638 leader_election.cc:290] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 32 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:34.673002 28183 leader_election.cc:290] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 26 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:34.489933 28664 raft_consensus.cc:515] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:34.696883 28501 raft_consensus.cc:2804] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:25:34.656970 28390 consensus_queue.cc:1059] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 26.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.085s
I20260709 12:25:34.747871 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:34.734649 18334 leader_election.cc:304] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:34.720814 28610 raft_consensus.cc:2749] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:25:34.730788 28487 raft_consensus.cc:515] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:25:34.759765 28490 consensus_meta.cc:287] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.733s	user 0.010s	sys 0.000s
I20260709 12:25:34.719394 17832 raft_consensus.cc:2468] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 31.
W20260709 12:25:34.760236 28636 consensus_meta.cc:287] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.699s	user 0.010s	sys 0.000s
I20260709 12:25:34.697008 28309 consensus_queue.cc:1048] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.050s
I20260709 12:25:34.730942 28650 raft_consensus.cc:740] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 30 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:25:34.796710 28650 consensus_queue.cc:260] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 30.21, Last appended by leader: 21, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:34.763216 28613 consensus_meta.cc:287] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.560s	user 0.007s	sys 0.003s
W20260709 12:25:34.782758 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 48717.5us
W20260709 12:25:34.798460 18334 leader_election.cc:336] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 31 election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:34.798861 18334 leader_election.cc:304] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 31 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:34.442902 28665 raft_consensus.cc:515] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:34.791350 18361 raft_consensus.cc:2393] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 30: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:25:34.788770 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 27 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
W20260709 12:25:34.808548 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:34.813534 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 145481us
I20260709 12:25:34.778909 28501 raft_consensus.cc:493] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:34.754067 28515 raft_consensus.cc:515] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:34.807626 18334 leader_election.cc:336] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:34.821417 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:34.786005 28416 raft_consensus.cc:2749] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260709 12:25:34.825906 28536 leader_election.cc:290] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 27 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:25:34.814036 28491 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.569s	user 0.014s	sys 0.019s
W20260709 12:25:34.826649 28491 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.585s	user 0.014s	sys 0.019s
I20260709 12:25:34.827406 28564 leader_election.cc:290] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:34.828447 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 26 candidate_status { last_received { term: 25 index: 22 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
W20260709 12:25:34.821458 18334 leader_election.cc:336] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:34.829340 17829 raft_consensus.cc:2368] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 26. Current term is 27.
I20260709 12:25:34.770594 28503 raft_consensus.cc:515] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:34.828508 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
W20260709 12:25:34.830224 18334 leader_election.cc:336] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 29 election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:25:34.833384 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:34.835146 18334 leader_election.cc:304] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:34.836171 17838 raft_consensus.cc:2393] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 31: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:25:34.805450 28490 raft_consensus.cc:2749] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:25:34.807595 28559 leader_election.cc:290] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 34 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:34.814517 28649 raft_consensus.cc:2749] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Leader election lost for term 31. Reason: could not achieve majority
W20260709 12:25:34.842432 18334 leader_election.cc:336] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:34.820533 28501 raft_consensus.cc:3060] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:34.845290 18333 leader_election.cc:304] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:25:34.843267 18334 leader_election.cc:336] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:34.847252 18334 leader_election.cc:304] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:34.850181 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 27 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:34.803715 28636 raft_consensus.cc:515] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:34.865108 17763 raft_consensus.cc:1275] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 22 index: 16. Preceding OpId from leader: term: 24 index: 17. (index mismatch)
W20260709 12:25:34.845275 28561 consensus_meta.cc:287] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.953s	user 0.010s	sys 0.000s
W20260709 12:25:34.867038 18334 leader_election.cc:336] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:34.867184 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 31 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:34.878858 17838 raft_consensus.cc:1275] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 36 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 34 index: 22. Preceding OpId from leader: term: 36 index: 23. (index mismatch)
I20260709 12:25:34.879158 17763 raft_consensus.cc:3060] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:34.878270 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 34 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:34.843966 28613 raft_consensus.cc:2749] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260709 12:25:34.880470 18334 leader_election.cc:304] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
W20260709 12:25:34.888242 18334 leader_election.cc:336] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 34 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:34.889587 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:34.889678 17832 raft_consensus.cc:3060] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:34.892499 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 34 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:34.893365 17757 raft_consensus.cc:3060] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:25:34.894665 18367 raft_consensus.cc:1240] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 34 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 33. Current term is 34. Ops: [33.19-33.19]
I20260709 12:25:34.865114 28609 raft_consensus.cc:2749] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:25:34.898782 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:34.902068 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:34.865614 28274 leader_election.cc:290] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 30 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:34.902848 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 32 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:34.904107 17838 raft_consensus.cc:2393] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 32: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:25:34.876279 28610 raft_consensus.cc:2749] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:25:34.906795 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:34.909497 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:34.910562 28672 raft_consensus.cc:493] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:34.905083 17652 leader_election.cc:304] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:34.909813 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 29 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:34.852424 28416 consensus_queue.cc:1059] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 30.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.043s
I20260709 12:25:34.921659 28564 raft_consensus.cc:493] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:34.922353 28564 raft_consensus.cc:515] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:34.911242 28672 raft_consensus.cc:515] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:34.925891 28672 leader_election.cc:290] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:34.926432 28530 raft_consensus.cc:493] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:34.926949 28530 raft_consensus.cc:515] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:34.929096 28530 leader_election.cc:290] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:34.929427 28564 leader_election.cc:290] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:34.929797 17831 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37994 (request call id 2015) took 1030 ms. Trace:
I20260709 12:25:34.930657 17760 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38786 (request call id 4358) took 1883 ms. Trace:
I20260709 12:25:34.931110 17760 rpcz_store.cc:276] 0709 12:25:33.047244 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:33.070262 (+ 23018us) service_pool.cc:224] Handling call
0709 12:25:34.930512 (+1860250us) spinlock_profiling.cc:243] Waited 956 ms on lock 0x7b54002b5a54. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:25:34.930605 (+    93us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2198527232}
I20260709 12:25:34.882596 28561 raft_consensus.cc:515] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:34.934906 17830 raft_consensus.cc:1240] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 34 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 33. Current term is 34. Ops: [33.19-33.19]
I20260709 12:25:34.930528 17831 rpcz_store.cc:276] 0709 12:25:33.899147 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:33.959197 (+ 60050us) service_pool.cc:224] Handling call
0709 12:25:34.929511 (+970314us) spinlock_profiling.cc:243] Waited 291 ms on lock 0x7b540008cfd4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:25:34.929713 (+   202us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":670248192}
I20260709 12:25:34.938159 28664 leader_election.cc:290] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 30 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:34.940927 28630 leader_election.cc:290] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:34.942935 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:34.965349 28126 leader_election.cc:290] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 35 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:34.952178 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:35.050633 17762 raft_consensus.cc:2468] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 30.
I20260709 12:25:34.919667 28407 raft_consensus.cc:2749] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:25:34.932618 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 36 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:34.960414 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:34.949434 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 34 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:34.933836 17652 leader_election.cc:400] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 32 election: Vote denied by peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837) with higher term. Message: Invalid argument: T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for earlier term 32. Current term is 33.
W20260709 12:25:34.915009 28642 consensus_meta.cc:287] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 1.026s	user 0.013s	sys 0.000s
I20260709 12:25:34.897962 28638 raft_consensus.cc:2749] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:25:35.054855 17835 raft_consensus.cc:3060] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:25:35.055325 17652 leader_election.cc:403] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 32 election: Cancelling election due to peer responding with higher term
I20260709 12:25:34.991952 18361 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59902 (request call id 2489) took 1127 ms. Trace:
W20260709 12:25:35.067184 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 133500us
I20260709 12:25:35.056452 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 30 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:35.068063 17652 leader_election.cc:304] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
W20260709 12:25:35.047449 28644 consensus_meta.cc:287] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.627s	user 0.008s	sys 0.002s
W20260709 12:25:35.029264 18334 leader_election.cc:336] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 34 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:35.057396 17756 raft_consensus.cc:2393] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 31: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:25:35.072562 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:35.085762 17762 raft_consensus.cc:3060] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:25:35.085388 17831 raft_consensus.cc:1275] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 32 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 31 index: 21. Preceding OpId from leader: term: 32 index: 22. (index mismatch)
W20260709 12:25:35.009902 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 44141.3us
I20260709 12:25:34.975768 28578 raft_consensus.cc:2749] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:25:34.991066 28112 raft_consensus.cc:3055] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:25:34.980690 28487 leader_election.cc:290] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:34.981738 28515 leader_election.cc:290] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:35.091570 18333 leader_election.cc:304] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:25:35.090229 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 59573.1us
W20260709 12:25:34.990067 28662 log.cc:927] Time spent T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.668s	user 0.002s	sys 0.005s
I20260709 12:25:35.067046 18361 rpcz_store.cc:276] 0709 12:25:33.865079 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:33.865277 (+   198us) service_pool.cc:224] Handling call
0709 12:25:34.991675 (+1126398us) spinlock_profiling.cc:243] Waited 889 ms on lock 0x7b540047de54. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:25:34.991866 (+   191us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2043574656}
W20260709 12:25:35.064787 28148 consensus_meta.cc:287] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.620s	user 0.011s	sys 0.000s
I20260709 12:25:35.109792 18365 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59902 (request call id 2498) took 1194 ms. Trace:
I20260709 12:25:35.110519 17803 leader_election.cc:304] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:25:34.992053 28675 raft_consensus.cc:493] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:35.081486 28559 raft_consensus.cc:2804] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:25:35.073315 28490 raft_consensus.cc:3060] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 32 FOLLOWER]: Advancing to term 33
W20260709 12:25:35.119132 17804 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:35.028271 17832 raft_consensus.cc:2468] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 27.
I20260709 12:25:35.122102 17835 raft_consensus.cc:2468] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 36.
I20260709 12:25:35.117022 17764 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38786 (request call id 4354) took 2071 ms. Trace:
I20260709 12:25:35.122843 17764 rpcz_store.cc:276] 0709 12:25:33.046046 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:33.075777 (+ 29731us) service_pool.cc:224] Handling call
0709 12:25:33.082423 (+  6646us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:33.790715 (+708292us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:35.116743 (+1326028us) spinlock_profiling.cc:243] Waited 708 ms on lock 0x7b54000fe6d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:25:35.116960 (+   217us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1627445504}
I20260709 12:25:35.124454 18333 leader_election.cc:400] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 26 election: Vote denied by peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837) with higher term. Message: Invalid argument: T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 26. Current term is 27.
I20260709 12:25:35.124790 18333 leader_election.cc:403] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 26 election: Cancelling election due to peer responding with higher term
I20260709 12:25:35.069046 28642 raft_consensus.cc:2749] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:25:35.124938 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:35.035657 28390 consensus_queue.cc:1048] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.069s
I20260709 12:25:35.131356 17835 raft_consensus.cc:2468] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 22.
I20260709 12:25:35.113962 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 30 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
W20260709 12:25:35.135174 17801 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:35.136305 17801 leader_election.cc:304] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:25:35.138176 17763 raft_consensus.cc:2468] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 31.
I20260709 12:25:35.141485 17762 raft_consensus.cc:2468] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 32.
I20260709 12:25:35.141906 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 25 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:35.142107 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 33 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:35.142867 17756 raft_consensus.cc:3060] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Advancing to term 33
I20260709 12:25:35.084690 28664 consensus_queue.cc:1048] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.037s
I20260709 12:25:35.120839 18366 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59902 (request call id 2493) took 1209 ms. Trace:
I20260709 12:25:35.128700 18333 leader_election.cc:304] T a5732e71b9d54f8786b8c511d69943bf P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 31 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:25:35.113325 28509 consensus_meta.cc:287] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.742s	user 0.007s	sys 0.004s
I20260709 12:25:35.034675 17757 raft_consensus.cc:2468] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 34.
I20260709 12:25:35.104763 28112 raft_consensus.cc:740] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:25:35.088533 28644 raft_consensus.cc:515] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:35.165602 17801 leader_election.cc:304] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:35.200588 18363 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59902 (request call id 2488) took 1338 ms. Trace:
I20260709 12:25:35.201676 18368 raft_consensus.cc:1275] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 28 index: 20. Preceding OpId from leader: term: 30 index: 21. (index mismatch)
I20260709 12:25:35.208096 18366 rpcz_store.cc:276] 0709 12:25:33.911323 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:33.911473 (+   150us) service_pool.cc:224] Handling call
0709 12:25:35.120638 (+1209165us) spinlock_profiling.cc:243] Waited 289 ms on lock 0x7b54006144d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:25:35.120772 (+   134us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":665739008}
I20260709 12:25:35.208124 18363 rpcz_store.cc:276] 0709 12:25:33.861961 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:33.862174 (+   213us) service_pool.cc:224] Handling call
0709 12:25:35.200324 (+1338150us) spinlock_profiling.cc:243] Waited 516 ms on lock 0x7b540047ca54. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:25:35.200505 (+   181us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1186020736}
I20260709 12:25:35.189858 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 29 candidate_status { last_received { term: 28 index: 22 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:35.211966 17835 raft_consensus.cc:2468] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 28.
I20260709 12:25:35.139580 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
W20260709 12:25:35.140455 18334 consensus_peers.cc:597] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:25:35.213651 17756 raft_consensus.cc:2468] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 33.
W20260709 12:25:35.254918 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 126287us
I20260709 12:25:35.175359 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 35 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
W20260709 12:25:35.256086 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 115707us
I20260709 12:25:35.215940 18366 raft_consensus.cc:1275] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 28 index: 21. Preceding OpId from leader: term: 31 index: 22. (index mismatch)
I20260709 12:25:35.256498 17764 raft_consensus.cc:2368] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 36 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for earlier term 35. Current term is 36.
I20260709 12:25:35.208477 17801 leader_election.cc:304] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:35.109448 28609 raft_consensus.cc:2749] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:25:35.240065 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 31 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:35.224960 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 32 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:35.259008 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 31 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
W20260709 12:25:35.207042 28671 log.cc:927] Time spent T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.053s	user 0.000s	sys 0.000s
I20260709 12:25:35.265523 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:35.266387 17757 raft_consensus.cc:2393] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 32: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:25:35.270220 17835 raft_consensus.cc:1240] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 27. Current term is 28. Ops: []
I20260709 12:25:35.134953 28610 raft_consensus.cc:3060] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:25:35.179728 28425 consensus_meta.cc:287] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.697s	user 0.010s	sys 0.002s
I20260709 12:25:35.278825 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:35.141297 28620 raft_consensus.cc:3060] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:35.286623 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:35.168406 28599 raft_consensus.cc:2764] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 36 FOLLOWER]: Leader pre-election decision vote started in defunct term 35: won
I20260709 12:25:35.286697 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 29 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:35.289651 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 29 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:35.289827 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 37 candidate_status { last_received { term: 35 index: 26 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:35.284457 18368 raft_consensus.cc:2468] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 29.
I20260709 12:25:35.168087 28485 raft_consensus.cc:3055] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 33 LEADER]: Stepping down as leader of term 33
I20260709 12:25:35.179211 28677 raft_consensus.cc:493] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:35.225515 28630 raft_consensus.cc:2749] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:25:35.145026 28559 raft_consensus.cc:697] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 27 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:25:35.174036 28308 consensus_queue.cc:1048] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.121s
I20260709 12:25:35.294345 17835 raft_consensus.cc:1275] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 27 index: 19. Preceding OpId from leader: term: 29 index: 20. (index mismatch)
I20260709 12:25:35.307319 17835 raft_consensus.cc:1240] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 31 LEADER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 30. Current term is 31. Ops: []
W20260709 12:25:35.294274 28676 log.cc:927] Time spent T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.130s	user 0.001s	sys 0.000s
I20260709 12:25:35.269395 28649 raft_consensus.cc:2749] T a5732e71b9d54f8786b8c511d69943bf P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Leader election lost for term 31. Reason: could not achieve majority
I20260709 12:25:35.123071 28148 raft_consensus.cc:515] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 35 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:35.296046 28485 raft_consensus.cc:740] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 33 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:25:35.259683 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 25 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:35.300854 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:35.303251 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 36 candidate_status { last_received { term: 34 index: 23 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:35.315434 18366 raft_consensus.cc:2468] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 35.
I20260709 12:25:35.292435 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:35.315361 17756 raft_consensus.cc:2410] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 25 because replica has last-logged OpId of term: 22 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 15.
I20260709 12:25:35.316433 17757 raft_consensus.cc:3060] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:35.304626 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 32 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:35.185981 28501 raft_consensus.cc:515] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:35.294929 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:35.286289 28425 raft_consensus.cc:2749] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:25:35.286100 28642 raft_consensus.cc:493] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:35.224843 28509 raft_consensus.cc:515] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:35.133345 28675 raft_consensus.cc:515] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:35.287809 28416 consensus_queue.cc:1048] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.028s
I20260709 12:25:35.345613 17757 raft_consensus.cc:2468] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 29.
I20260709 12:25:35.346661 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 33 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:35.347234 17757 raft_consensus.cc:3060] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:25:35.348311 17760 raft_consensus.cc:2410] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for term 30 because replica has last-logged OpId of term: 28 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 27 index: 14.
I20260709 12:25:35.324792 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 34 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:35.326996 17764 raft_consensus.cc:2393] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 30: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:25:35.314402 28630 raft_consensus.cc:2749] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:25:35.353229 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 32 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:25:35.340292 28681 log.cc:927] Time spent T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.167s	user 0.001s	sys 0.000s
I20260709 12:25:35.340847 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 25 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:35.341679 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 31 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:35.362464 18374 raft_consensus.cc:2393] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 31: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:25:35.363212 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 34 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:35.363484 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 27 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
W20260709 12:25:35.364079 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 107670us
I20260709 12:25:35.364622 18374 raft_consensus.cc:2393] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 27: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:25:35.342334 17831 raft_consensus.cc:2393] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 30: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:25:35.342834 28501 leader_election.cc:290] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 30 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:35.349855 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:35.367630 18366 raft_consensus.cc:2393] T a5732e71b9d54f8786b8c511d69943bf P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 31: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:25:35.353888 18368 raft_consensus.cc:2468] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 32.
I20260709 12:25:35.367430 18361 raft_consensus.cc:2393] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 34 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 34: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:25:35.353924 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:35.369904 17803 leader_election.cc:304] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:25:35.370039 17764 raft_consensus.cc:2468] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 21.
I20260709 12:25:35.365837 17759 raft_consensus.cc:3060] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:25:35.300750 28677 raft_consensus.cc:515] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:35.372027 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 163702us
I20260709 12:25:35.365995 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 27 candidate_status { last_received { term: 25 index: 22 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:35.310217 28112 consensus_queue.cc:260] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 27.19, Last appended by leader: 19, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:35.353734 28561 leader_election.cc:290] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 30 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:25:35.371191 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 300494us
I20260709 12:25:35.371354 18366 raft_consensus.cc:1240] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 35 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 34. Current term is 35. Ops: []
I20260709 12:25:35.379187 17801 leader_election.cc:304] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:35.373513 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:35.388388 17801 leader_election.cc:304] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:25:35.388947 17756 raft_consensus.cc:2393] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 26: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:25:35.388332 18366 raft_consensus.cc:1275] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 35 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 34 index: 23. Preceding OpId from leader: term: 35 index: 24. (index mismatch)
I20260709 12:25:35.386209 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 32 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:35.384207 18368 raft_consensus.cc:1240] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 30. Current term is 31. Ops: [30.21-30.21]
I20260709 12:25:35.391613 18364 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 3067) took 1053 ms. Trace:
I20260709 12:25:35.396183 17729 leader_election.cc:304] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:25:35.358858 17757 raft_consensus.cc:2468] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 33.
I20260709 12:25:35.387812 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 30 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:35.399226 18361 raft_consensus.cc:3060] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:35.407853 17764 raft_consensus.cc:1275] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 30 index: 19. Preceding OpId from leader: term: 31 index: 20. (index mismatch)
I20260709 12:25:35.409441 17757 raft_consensus.cc:1275] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 35 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 34 index: 23. Preceding OpId from leader: term: 35 index: 24. (index mismatch)
I20260709 12:25:35.410244 28495 raft_consensus.cc:3055] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:25:35.411592 28677 leader_election.cc:290] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:25:35.413012 17728 consensus_peers.cc:597] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:25:35.422022 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 708376us
I20260709 12:25:35.352326 28642 raft_consensus.cc:515] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:35.415863 17801 leader_election.cc:304] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:35.398638 28308 raft_consensus.cc:2804] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:25:35.418358 17759 raft_consensus.cc:2468] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 34.
I20260709 12:25:35.441047 18361 raft_consensus.cc:2468] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 30.
I20260709 12:25:35.415670 28112 raft_consensus.cc:3060] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:35.439658 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 29 candidate_status { last_received { term: 28 index: 22 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:25:35.437718 17728 consensus_peers.cc:597] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:25:35.416759 17729 leader_election.cc:304] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:35.443076 17757 pending_rounds.cc:85] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb: Aborting all ops after (but not including) 18
I20260709 12:25:35.447203 17757 pending_rounds.cc:107] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb: Aborting uncommitted NO_OP operation due to leader change: 26.19
I20260709 12:25:35.447468 17757 raft_consensus.cc:2889] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:35.448503 17729 leader_election.cc:304] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:35.417520 28614 raft_consensus.cc:2764] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Leader pre-election decision vote started in defunct term 28: won
I20260709 12:25:35.460433 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:35.418198 28495 raft_consensus.cc:740] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:25:35.420740 28501 raft_consensus.cc:2804] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:25:35.433641 28630 raft_consensus.cc:2804] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:25:35.376533 28559 consensus_queue.cc:237] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 25.22, Last appended by leader: 22, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:35.417516 28425 consensus_queue.cc:1048] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.047s
I20260709 12:25:35.457828 28126 raft_consensus.cc:2804] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:25:35.485014 17758 raft_consensus.cc:1240] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 35 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 34. Current term is 35. Ops: []
I20260709 12:25:35.406458 28485 consensus_queue.cc:260] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 33.21, Last appended by leader: 21, Current term: 33, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:35.458781 28693 raft_consensus.cc:2749] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:25:35.468025 28501 raft_consensus.cc:493] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:35.466885 28507 raft_consensus.cc:2804] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:25:35.467092 28308 raft_consensus.cc:697] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 31 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:25:35.469666 28630 raft_consensus.cc:493] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
W20260709 12:25:35.521196 28688 log.cc:927] Time spent T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.056s	user 0.000s	sys 0.000s
I20260709 12:25:35.546603 28557 leader_election.cc:290] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 29 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:25:35.549816 28691 log.cc:927] Time spent T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.077s	user 0.000s	sys 0.000s
I20260709 12:25:35.553211 17758 raft_consensus.cc:1240] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 30. Current term is 31. Ops: []
I20260709 12:25:35.461123 28677 consensus_queue.cc:1059] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 27.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.030s
I20260709 12:25:35.557745 17761 raft_consensus.cc:1240] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 29. Current term is 30. Ops: []
I20260709 12:25:35.487782 28126 raft_consensus.cc:493] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 35 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:35.494585 28682 raft_consensus.cc:493] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:35.471527 28672 consensus_queue.cc:1059] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 34.24, Next index: 25, Last known committed idx: 24, Time since last communication: 0.033s
I20260709 12:25:35.489343 28644 leader_election.cc:290] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 30 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:35.563383 17761 raft_consensus.cc:1275] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 36 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 34 index: 22. Preceding OpId from leader: term: 36 index: 23. (index mismatch)
I20260709 12:25:35.504922 28630 raft_consensus.cc:3060] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:35.578476 17758 raft_consensus.cc:1240] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 34 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 33. Current term is 34. Ops: []
I20260709 12:25:35.581295 17761 raft_consensus.cc:1275] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 25 index: 18. Preceding OpId from leader: term: 29 index: 20. (index mismatch)
I20260709 12:25:35.582923 28675 leader_election.cc:290] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:35.584544 28603 leader_election.cc:290] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:35.505978 28501 raft_consensus.cc:3060] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:25:35.587654 17761 raft_consensus.cc:1240] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 25. Current term is 26. Ops: []
I20260709 12:25:35.584906 28642 leader_election.cc:290] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:35.505714 28507 raft_consensus.cc:493] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:25:35.563802 28485 raft_consensus.cc:3060] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:25:35.604023 28665 leader_election.cc:290] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:35.560793 28148 leader_election.cc:290] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 35 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:35.563122 28126 raft_consensus.cc:3060] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:25:35.601539 28308 consensus_queue.cc:237] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 29.20, Last appended by leader: 20, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:35.598713 28507 raft_consensus.cc:3060] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:25:35.622659 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:35.623406 18366 raft_consensus.cc:3060] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:35.614028 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 30 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:35.615389 18368 raft_consensus.cc:1275] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 25 index: 17. Preceding OpId from leader: term: 28 index: 19. (index mismatch)
W20260709 12:25:35.626825 28683 log.cc:927] Time spent T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.188s	user 0.000s	sys 0.000s
W20260709 12:25:35.631860 17804 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:35.634238 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
W20260709 12:25:35.642270 17804 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:35.649361 17833 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35040 (request call id 3925) took 1456 ms (client timeout 1512 ms). Trace:
W20260709 12:25:35.649901 17833 rpcz_store.cc:269] 0709 12:25:34.193023 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:34.197122 (+  4099us) service_pool.cc:224] Handling call
0709 12:25:35.635477 (+1438355us) spinlock_profiling.cc:243] Waited 118 ms on lock 0x7b540022ae54. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:25:35.649273 (+ 13796us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":271276416}
I20260709 12:25:35.594980 28693 raft_consensus.cc:3055] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:25:35.640815 17652 leader_election.cc:304] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:25:35.653796 17652 leader_election.cc:304] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:25:35.664894 17758 raft_consensus.cc:1240] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 30. Current term is 31. Ops: []
I20260709 12:25:35.585494 28682 raft_consensus.cc:515] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:35.648058 28490 raft_consensus.cc:2749] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 33 FOLLOWER]: Leader election lost for term 32. Reason: Vote denied by peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837) with higher term. Message: Invalid argument: T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for earlier term 32. Current term is 33.
I20260709 12:25:35.557901 28495 consensus_queue.cc:260] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 26.19, Last appended by leader: 19, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:35.657241 28677 raft_consensus.cc:493] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:25:35.657646 28610 raft_consensus.cc:2749] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Leader election lost for term 26. Reason: Vote denied by peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837) with higher term. Message: Invalid argument: T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 26. Current term is 27.
I20260709 12:25:35.657423 28693 raft_consensus.cc:740] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 28 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:25:35.660035 28665 raft_consensus.cc:2804] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:25:35.661033 28642 raft_consensus.cc:2804] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:25:35.701735 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 30 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
W20260709 12:25:35.702595 18334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:35.708606 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 241283us
W20260709 12:25:35.705356 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 258406us
I20260709 12:25:35.715413 18366 raft_consensus.cc:2468] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 30.
I20260709 12:25:35.707625 28495 raft_consensus.cc:3060] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:35.709183 28665 raft_consensus.cc:493] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:35.708536 28642 raft_consensus.cc:493] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:25:35.736047 28706 raft_consensus.cc:493] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:35.736217 28636 leader_election.cc:290] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 26 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:35.736690 28706 raft_consensus.cc:515] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:35.738981 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 36 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:35.740407 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 36 candidate_status { last_received { term: 34 index: 23 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:35.741475 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 35 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:35.741714 28565 raft_consensus.cc:3055] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:25:35.742192 28565 raft_consensus.cc:740] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 30 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:25:35.742566 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 35 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:35.743047 28565 consensus_queue.cc:260] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 30.19, Last appended by leader: 19, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:35.743614 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:35.743973 28693 consensus_queue.cc:260] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 28.19, Last appended by leader: 19, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:35.745776 17832 raft_consensus.cc:1240] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 29. Current term is 30. Ops: []
I20260709 12:25:35.750070 18374 raft_consensus.cc:2468] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 27.
I20260709 12:25:35.751582 17830 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35040 (request call id 3924) took 1566 ms. Trace:
W20260709 12:25:35.751971 17728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:35.752641 28503 leader_election.cc:290] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:25:35.753703 18371 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59902 (request call id 2500) took 1557 ms. Trace:
I20260709 12:25:35.754081 18371 rpcz_store.cc:276] 0709 12:25:34.196541 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:34.414405 (+217864us) service_pool.cc:224] Handling call
0709 12:25:34.441808 (+ 27403us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:34.461172 (+ 19364us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:35.753526 (+1292354us) spinlock_profiling.cc:243] Waited 27.1 ms on lock 0x7b54007e7cd0. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a61f8ad 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:25:35.753648 (+   122us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":62289280}
I20260709 12:25:35.754874 28320 raft_consensus.cc:493] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:25:35.755383 28320 raft_consensus.cc:515] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:35.753532 17830 rpcz_store.cc:276] 0709 12:25:34.185611 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:34.192026 (+  6415us) service_pool.cc:224] Handling call
0709 12:25:34.934151 (+742125us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:35.048493 (+114342us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:35.751369 (+702876us) spinlock_profiling.cc:243] Waited 742 ms on lock 0x7b540008cfd0. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a61f8ad 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:25:35.751527 (+   158us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1705524096}
I20260709 12:25:35.758906 17830 raft_consensus.cc:1275] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 30 index: 17. Preceding OpId from leader: term: 31 index: 18. (index mismatch)
I20260709 12:25:35.706440 28677 raft_consensus.cc:515] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:35.761809 17755 raft_consensus.cc:1240] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 26. Current term is 27. Ops: []
I20260709 12:25:35.767359 28618 consensus_queue.cc:1059] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 31.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20260709 12:25:35.768468 28618 raft_consensus.cc:3055] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 33 LEADER]: Stepping down as leader of term 33
I20260709 12:25:35.768826 28618 raft_consensus.cc:740] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 33 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:25:35.769665 28618 consensus_queue.cc:260] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 33.19, Last appended by leader: 19, Current term: 33, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:35.770962 28618 raft_consensus.cc:3060] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:25:35.730355 28665 raft_consensus.cc:3060] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:35.733202 28642 raft_consensus.cc:3060] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:35.779335 18367 raft_consensus.cc:3055] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:25:35.780714 18367 raft_consensus.cc:740] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 30 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:25:35.787022 28565 raft_consensus.cc:3060] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:35.781661 18367 consensus_queue.cc:260] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 30.19, Last appended by leader: 19, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:35.806803 18367 raft_consensus.cc:3060] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:35.775964 28620 raft_consensus.cc:2749] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
W20260709 12:25:35.813078 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:35.820878 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:35.813407 17801 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:35.826467 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 34 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:35.827246 18366 raft_consensus.cc:2393] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 34: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:25:35.824496 17801 leader_election.cc:304] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:35.831748 17801 leader_election.cc:304] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:35.823603 17652 leader_election.cc:304] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:35.112422 18365 rpcz_store.cc:276] 0709 12:25:33.915592 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:34.242151 (+326559us) service_pool.cc:224] Handling call
0709 12:25:34.242559 (+   408us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 12:25:34.386905 (+144346us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:34.386957 (+    52us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 12:25:34.426035 (+ 39078us) log.cc:844] Serialized 78 byte log entry
0709 12:25:34.765512 (+339477us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:25:35.097078 (+331566us) log.cc:844] Serialized 14 byte log entry
0709 12:25:35.097832 (+   754us) log.cc:844] Serialized 14 byte log entry
0709 12:25:35.109381 (+ 11549us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:35.109488 (+   107us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:35.109543 (+    55us) raft_consensus.cc:1685] finished
0709 12:25:35.109561 (+    18us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:35.109738 (+   177us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":10781,"prepare.queue_time_us":153765,"prepare.run_cpu_time_us":49,"prepare.run_wall_time_us":50,"raft.queue_time_us":11863,"raft.run_cpu_time_us":244,"raft.run_wall_time_us":46600,"spinlock_wait_cycles":19173248,"thread_start_us":614513,"threads_started":2,"wal-append.queue_time_us":443097}
I20260709 12:25:35.836405 17833 raft_consensus.cc:1240] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 26. Current term is 27. Ops: []
I20260709 12:25:35.774313 28644 consensus_queue.cc:1059] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 29.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.038s
I20260709 12:25:35.837877 18371 raft_consensus.cc:1275] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 24 index: 17. Preceding OpId from leader: term: 29 index: 20. (index mismatch)
I20260709 12:25:35.848850 17837 raft_consensus.cc:2468] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 30.
I20260709 12:25:35.849413 17801 leader_election.cc:304] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:25:35.849949 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 23 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:35.850884 17762 raft_consensus.cc:2410] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 23 because replica has last-logged OpId of term: 21 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 13.
I20260709 12:25:35.855944 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 36 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:35.857923 18366 raft_consensus.cc:1240] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 29. Current term is 30. Ops: []
I20260709 12:25:35.863341 18367 raft_consensus.cc:1275] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 30 index: 19. Preceding OpId from leader: term: 31 index: 20. (index mismatch)
I20260709 12:25:35.864081 18367 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59902 (request call id 2505) took 1102 ms. Trace:
I20260709 12:25:35.864389 18367 rpcz_store.cc:276] 0709 12:25:34.761219 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:34.909559 (+148340us) service_pool.cc:224] Handling call
0709 12:25:34.909862 (+   303us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:35.779208 (+869346us) spinlock_profiling.cc:243] Waited 72.4 ms on lock 0x7b54002a38a0. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a597dc0 00007f7b4a6221c0 00007f7b4a623231 00007f7b4a61fd8f 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c
0709 12:25:35.863809 (+ 84601us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:35.864028 (+   219us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":166422784}
I20260709 12:25:35.868817 17756 raft_consensus.cc:2393] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 36 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 36: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:25:35.822245 28485 raft_consensus.cc:493] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:35.831089 28636 raft_consensus.cc:493] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:35.835851 28699 raft_consensus.cc:493] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:35.840955 28503 raft_consensus.cc:2804] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 33 FOLLOWER]: Leader election won for term 33
I20260709 12:25:35.875625 17801 leader_election.cc:304] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:35.843052 28637 raft_consensus.cc:2749] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:25:35.886694 28703 raft_consensus.cc:2804] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:25:35.891708 28703 raft_consensus.cc:697] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 29 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:25:35.891974 28692 consensus_queue.cc:1048] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
I20260709 12:25:35.893005 28703 consensus_queue.cc:237] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 28.19, Last appended by leader: 19, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:35.894590 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:35.895334 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:35.895839 28561 consensus_queue.cc:1059] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 29.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:25:35.897513 28578 consensus_queue.cc:1059] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 29.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.000s
I20260709 12:25:35.898617 28603 raft_consensus.cc:3055] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:25:35.899055 28603 raft_consensus.cc:740] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 29 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
W20260709 12:25:35.900503 28705 log.cc:927] Time spent T 00000000000000000000000000000000 P 2193af31cb894ec9b3d65d939a2a6851: Append to log took a long time: real 0.114s	user 0.000s	sys 0.000s
I20260709 12:25:35.901818 28308 raft_consensus.cc:2804] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 33 FOLLOWER]: Leader election won for term 33
I20260709 12:25:35.902438 28308 raft_consensus.cc:697] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 33 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:25:35.904274 28308 consensus_queue.cc:237] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 31.21, Last appended by leader: 21, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:25:35.906098 28661 log.cc:927] Time spent T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 1.585s	user 0.001s	sys 0.007s
I20260709 12:25:35.392030 18364 rpcz_store.cc:276] 0709 12:25:34.338273 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:34.459522 (+121249us) service_pool.cc:224] Handling call
0709 12:25:35.118943 (+659421us) spinlock_profiling.cc:243] Waited 4.71 ms on lock 0x7b5800102398. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b474fbb71 00007f7b474fba22 00007f7b504148d9 00007f7b4f56eee0 00007f7b4f6380ab 00007f7b4f638131 00007f7b4f5f9fc0 00007f7b4f5e4929 00007f7b4f5e43d2 00007f7b48b5bbd4
0709 12:25:35.119157 (+   214us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:35.119374 (+   217us) raft_consensus.cc:1534] Early marking committed up to index 22
0709 12:25:35.325405 (+206031us) log.cc:844] Serialized 14 byte log entry
0709 12:25:35.391329 (+ 65924us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:35.391377 (+    48us) raft_consensus.cc:1645] Marking committed up to 22
0709 12:25:35.391415 (+    38us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:35.391470 (+    55us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:35.391564 (+    94us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":15266,"prepare.queue_time_us":215599,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":43,"raft.queue_time_us":95,"spinlock_wait_cycles":10829184,"thread_start_us":221831,"threads_started":2}
I20260709 12:25:35.909478 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 35 candidate_status { last_received { term: 34 index: 24 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:35.916455 28540 raft_consensus.cc:3055] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:25:35.916884 28540 raft_consensus.cc:740] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 29 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:25:35.917697 28540 consensus_queue.cc:260] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 29.16, Last appended by leader: 16, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:35.918943 28540 raft_consensus.cc:3060] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:35.920205 28706 leader_election.cc:290] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:35.920383 17759 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36572 (request call id 1945) took 1008 ms. Trace:
I20260709 12:25:35.875401 28651 consensus_queue.cc:1059] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 34.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.046s
I20260709 12:25:35.920724 17759 rpcz_store.cc:276] 0709 12:25:34.911762 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:34.913180 (+  1418us) service_pool.cc:224] Handling call
0709 12:25:35.920216 (+1007036us) spinlock_profiling.cc:243] Waited 310 ms on lock 0x7b54000fcdd4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:25:35.920325 (+   109us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":713953536}
W20260709 12:25:35.925756 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 538756us
W20260709 12:25:35.928227 18333 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:35.912525 18367 raft_consensus.cc:1275] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 29 index: 16. Preceding OpId from leader: term: 31 index: 17. (index mismatch)
W20260709 12:25:35.912021 28698 log.cc:927] Time spent T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.081s	user 0.001s	sys 0.000s
W20260709 12:25:35.876025 28526 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 31 current term: 31 is higher.
I20260709 12:25:35.930979 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 33 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:35.935418 18371 raft_consensus.cc:2468] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 32.
W20260709 12:25:35.880410 28712 log.cc:927] Time spent T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.132s	user 0.000s	sys 0.001s
I20260709 12:25:35.868000 28620 raft_consensus.cc:2749] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:25:35.933780 17654 leader_election.cc:304] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:35.941005 17654 leader_election.cc:304] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
W20260709 12:25:35.942018 17802 leader_election.cc:336] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 33 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:35.942626 17802 leader_election.cc:304] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:25:35.935523 18333 leader_election.cc:304] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:35.949929 17654 leader_election.cc:304] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:35.948385 18333 leader_election.cc:304] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:35.955168 18333 leader_election.cc:304] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:25:35.961040 17802 consensus_peers.cc:597] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:25:35.868315 28425 consensus_queue.cc:1048] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 17, Time since last communication: 0.078s
I20260709 12:25:35.934772 28649 raft_consensus.cc:493] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:35.979650 18333 leader_election.cc:304] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:35.944254 28708 raft_consensus.cc:493] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:35.957471 28638 raft_consensus.cc:2804] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:25:35.966704 28495 raft_consensus.cc:2749] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
I20260709 12:25:35.883921 28485 raft_consensus.cc:515] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:35.933002 28699 raft_consensus.cc:515] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:35.888648 28636 raft_consensus.cc:515] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:35.967000 28390 raft_consensus.cc:2804] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 34 FOLLOWER]: Leader election won for term 34
I20260709 12:25:35.951861 28682 leader_election.cc:290] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:35.981624 28620 raft_consensus.cc:2749] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:25:35.956578 28507 raft_consensus.cc:515] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 32 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:36.022964 18333 leader_election.cc:304] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:25:35.987262 28148 raft_consensus.cc:2804] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 34 FOLLOWER]: Leader election won for term 34
I20260709 12:25:36.029532 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 30 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:35.981567 28112 consensus_queue.cc:1048] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 17, Time since last communication: 0.040s
I20260709 12:25:36.031559 18365 raft_consensus.cc:2393] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 30: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:25:36.040968 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
W20260709 12:25:35.972360 17802 leader_election.cc:336] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 30 election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:36.046070 28507 leader_election.cc:290] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 32 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:36.031677 28303 raft_consensus.cc:493] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:36.046851 28303 raft_consensus.cc:515] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:36.049104 28303 leader_election.cc:290] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:25:36.055918 17730 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:36.071555 17804 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:36.001345 28708 raft_consensus.cc:515] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:36.025952 28682 raft_consensus.cc:2749] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:25:36.001058 28501 raft_consensus.cc:515] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 33 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:36.060753 28675 raft_consensus.cc:3055] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 34 LEADER]: Stepping down as leader of term 34
I20260709 12:25:36.033632 28610 raft_consensus.cc:2804] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Leader election won for term 27
W20260709 12:25:36.061342 28713 log.cc:927] Time spent T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.178s	user 0.001s	sys 0.000s
I20260709 12:25:36.105412 17755 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36572 (request call id 1941) took 1224 ms. Trace:
I20260709 12:25:36.105940 17755 rpcz_store.cc:276] 0709 12:25:34.880693 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:34.889709 (+  9016us) service_pool.cc:224] Handling call
0709 12:25:34.915762 (+ 26053us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:35.774563 (+858801us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:36.105069 (+330506us) spinlock_profiling.cc:243] Waited 846 ms on lock 0x7b54003e58d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:25:36.105329 (+   260us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1945284992}
I20260709 12:25:36.036127 28148 raft_consensus.cc:697] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 34 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:25:36.025712 28126 raft_consensus.cc:515] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 36 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:36.079697 28526 raft_consensus.cc:493] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:25:36.082983 28636 leader_election.cc:290] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:35.985296 28649 raft_consensus.cc:515] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:36.111371 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 219643us
I20260709 12:25:36.039458 28557 consensus_queue.cc:1048] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 17, Time since last communication: 0.065s
I20260709 12:25:36.091207 28675 raft_consensus.cc:740] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 34 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:25:36.101214 28677 leader_election.cc:290] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:25:36.113067 28710 log.cc:927] Time spent T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.252s	user 0.000s	sys 0.001s
W20260709 12:25:36.130970 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 249175us
W20260709 12:25:36.133355 18334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:36.114570 28682 raft_consensus.cc:3060] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Advancing to term 31
W20260709 12:25:36.122309 17804 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:36.111348 28610 raft_consensus.cc:697] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 27 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
W20260709 12:25:36.122215 28711 log.cc:927] Time spent T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.231s	user 0.001s	sys 0.000s
I20260709 12:25:36.123145 17802 leader_election.cc:304] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:36.115557 28526 raft_consensus.cc:515] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:36.172226 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 199886us
I20260709 12:25:36.171780 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:25:36.173280 17802 leader_election.cc:336] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 33 election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:25:36.174767 17802 leader_election.cc:336] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 33 election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:25:36.175937 17802 leader_election.cc:336] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:25:36.184674 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 234850us
W20260709 12:25:36.171584 28409 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.544s	user 0.059s	sys 0.068s
I20260709 12:25:36.185321 17802 leader_election.cc:304] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:36.141202 28636 consensus_queue.cc:1048] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.042s
I20260709 12:25:36.187114 17654 leader_election.cc:304] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:36.177184 28148 consensus_queue.cc:237] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 32.19, Last appended by leader: 19, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:36.218896 28699 leader_election.cc:290] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:25:36.220217 28630 consensus_meta.cc:287] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.625s	user 0.000s	sys 0.011s
W20260709 12:25:36.220436 28696 log.cc:927] Time spent T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.719s	user 0.003s	sys 0.000s
I20260709 12:25:36.221822 28610 consensus_queue.cc:237] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 25.19, Last appended by leader: 19, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:36.224016 28526 leader_election.cc:290] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:36.227308 28721 consensus_queue.cc:1048] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.000s
I20260709 12:25:36.228188 28651 raft_consensus.cc:493] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:36.230140 28649 leader_election.cc:290] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:36.232599 28126 leader_election.cc:290] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 36 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:36.186648 28485 leader_election.cc:290] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:36.234100 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:36.234532 28509 leader_election.cc:290] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 28 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:36.235396 18362 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59902 (request call id 2506) took 1433 ms. Trace:
I20260709 12:25:36.240258 28320 leader_election.cc:290] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:36.249917 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 36 candidate_status { last_received { term: 34 index: 23 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:36.250177 17755 raft_consensus.cc:2393] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 29: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
W20260709 12:25:36.189671 28409 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.562s	user 0.059s	sys 0.068s
W20260709 12:25:36.242841 17804 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:36.245138 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 30 candidate_status { last_received { term: 28 index: 22 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:25:36.189234 28695 log.cc:927] Time spent T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.129s	user 0.004s	sys 0.000s
I20260709 12:25:36.251843 18364 raft_consensus.cc:2468] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 29.
I20260709 12:25:36.252820 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:36.253610 17836 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37994 (request call id 2018) took 1791 ms. Trace:
I20260709 12:25:36.253973 17836 rpcz_store.cc:276] 0709 12:25:34.462308 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:35.202360 (+740052us) service_pool.cc:224] Handling call
0709 12:25:35.202704 (+   344us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:25:35.224595 (+ 21891us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:35.224647 (+    52us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:25:35.280111 (+ 55464us) log.cc:844] Serialized 52 byte log entry
0709 12:25:35.438583 (+158472us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:25:35.453398 (+ 14815us) log.cc:844] Serialized 14 byte log entry
0709 12:25:35.629224 (+175826us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:35.629320 (+    96us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:36.057284 (+427964us) spinlock_profiling.cc:243] Waited 3.15 ms on lock 0x7b1800281b60. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b475333f3 00007f7b475357c1 00007f7b47535779 00007f7b47535709 00007f7b475356d1 00007f7b475349cd 0000000000388634 0000000000388479 00007f7b4751683a
0709 12:25:36.172181 (+114897us) spinlock_profiling.cc:243] Waited 7.63 ms on lock 0x7b1800281b60. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b47533522 00007f7b475357c1 00007f7b47535779 00007f7b47535709 00007f7b475356d1 00007f7b475349cd 0000000000388634 0000000000388479 00007f7b4751683a
0709 12:25:36.253406 (+ 81225us) raft_consensus.cc:1685] finished
0709 12:25:36.253426 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:36.253560 (+   134us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":15058,"prepare.run_cpu_time_us":55,"prepare.run_wall_time_us":55,"raft.queue_time_us":14074,"raft.run_cpu_time_us":298,"raft.run_wall_time_us":23423,"spinlock_wait_cycles":24791680,"thread_start_us":309445,"threads_started":2,"wal-append.queue_time_us":187826}
I20260709 12:25:36.180716 28309 consensus_queue.cc:1059] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.064s
I20260709 12:25:36.183960 28308 raft_consensus.cc:493] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:36.170207 28503 raft_consensus.cc:697] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 33 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
W20260709 12:25:36.247288 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 71401.6us
I20260709 12:25:36.265009 17761 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36572 (request call id 1948) took 1007 ms. Trace:
W20260709 12:25:36.265626 17802 leader_election.cc:336] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 34 election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:36.250998 18374 raft_consensus.cc:3060] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:25:36.202404 28708 leader_election.cc:290] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:36.265373 17761 rpcz_store.cc:276] 0709 12:25:35.257796 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:35.594124 (+336328us) service_pool.cc:224] Handling call
0709 12:25:35.594539 (+   415us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:35.599741 (+  5202us) raft_consensus.cc:1534] Early marking committed up to index 16
0709 12:25:35.614111 (+ 14370us) log.cc:844] Serialized 14 byte log entry
0709 12:25:35.724988 (+110877us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:35.734727 (+  9739us) log.cc:844] Serialized 26 byte log entry
0709 12:25:35.734923 (+   196us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:25:35.734956 (+    33us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:35.735418 (+   462us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:36.264819 (+529401us) raft_consensus.cc:1685] finished
0709 12:25:36.264840 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:36.264952 (+   112us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4586,"prepare.queue_time_us":709,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":43,"raft.queue_time_us":7261,"raft.run_cpu_time_us":272,"raft.run_wall_time_us":9485,"thread_start_us":107673,"threads_started":1,"wal-append.queue_time_us":189816}
I20260709 12:25:36.177063 28675 consensus_queue.cc:260] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 34.24, Last appended by leader: 24, Current term: 34, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:36.249620 18362 rpcz_store.cc:276] 0709 12:25:34.802195 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:34.939868 (+137673us) service_pool.cc:224] Handling call
0709 12:25:34.940232 (+   364us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:35.006816 (+ 66584us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:35.006865 (+    49us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:35.011222 (+  4357us) log.cc:844] Serialized 26 byte log entry
0709 12:25:35.924925 (+913703us) spinlock_profiling.cc:243] Waited 2.42 ms on lock 0x7b180031bf40. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b47515e07 00007f7b503709a9 00007f7b4753269b 00007f7b4753093f 00007f7b47532c02 00007f7b4a1ba20b 00007f7b4a1bfd23 00007f7b4a1c00d1 00007f7b4a5dba0f
0709 12:25:36.167370 (+242445us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:25:36.167411 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:36.167488 (+    77us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:36.235008 (+ 67520us) raft_consensus.cc:1685] finished
0709 12:25:36.235037 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:36.235324 (+   287us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":66110,"raft.queue_time_us":69074,"raft.run_cpu_time_us":198,"raft.run_wall_time_us":33207,"spinlock_wait_cycles":5566848,"thread_start_us":242261,"threads_started":1,"wal-append.queue_time_us":1212921}
I20260709 12:25:36.203229 28717 raft_consensus.cc:2749] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
I20260709 12:25:36.260924 28509 raft_consensus.cc:3060] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:36.248569 28630 raft_consensus.cc:515] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:36.257510 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 32 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:36.297132 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 35 candidate_status { last_received { term: 34 index: 24 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:36.286686 28645 raft_consensus.cc:3055] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:25:36.302215 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 32 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:36.247126 28559 raft_consensus.cc:2804] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:25:36.297106 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:36.266105 17802 leader_election.cc:304] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 34 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:25:36.299947 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 29 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:36.304546 17829 raft_consensus.cc:2368] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 29. Current term is 30.
I20260709 12:25:36.271921 28501 leader_election.cc:290] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 33 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:36.283969 28699 raft_consensus.cc:2749] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:25:36.275782 18374 raft_consensus.cc:2468] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 36.
I20260709 12:25:36.256436 28651 raft_consensus.cc:515] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:36.267629 28409 consensus_queue.cc:1048] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 19, Time since last communication: 0.034s
I20260709 12:25:36.306880 17832 raft_consensus.cc:1240] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 30. Current term is 31. Ops: []
I20260709 12:25:36.269870 28308 raft_consensus.cc:515] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:36.302057 28645 raft_consensus.cc:740] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 25 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:25:36.341290 17833 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 31.
I20260709 12:25:36.326687 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 32 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:36.326706 28725 raft_consensus.cc:493] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:36.333676 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 30 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:36.334553 17755 raft_consensus.cc:1240] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 27. Current term is 28. Ops: []
I20260709 12:25:36.334815 17759 raft_consensus.cc:1240] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 36 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 35. Current term is 36. Ops: [35.24-35.24]
I20260709 12:25:36.300833 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 33 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:36.349180 17756 raft_consensus.cc:3060] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:25:36.303419 28559 raft_consensus.cc:493] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:25:36.344424 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 178664us
W20260709 12:25:36.342456 17802 consensus_peers.cc:597] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:25:36.347432 17761 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 29.
W20260709 12:25:36.360253 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 65918.5us
I20260709 12:25:36.360409 17801 leader_election.cc:304] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:36.323971 28717 raft_consensus.cc:2749] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 34 FOLLOWER]: Leader election lost for term 34. Reason: could not achieve majority
I20260709 12:25:36.345221 28390 raft_consensus.cc:697] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 34 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:25:36.353571 28559 raft_consensus.cc:3060] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Advancing to term 32
W20260709 12:25:36.362048 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 154699us
I20260709 12:25:36.401537 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 35 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:36.403622 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 35 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:36.410750 17764 raft_consensus.cc:2368] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 36 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for earlier term 35. Current term is 36.
I20260709 12:25:36.409477 17654 leader_election.cc:304] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
W20260709 12:25:36.381389 28642 consensus_meta.cc:287] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.576s	user 0.011s	sys 0.000s
I20260709 12:25:36.413782 17654 leader_election.cc:304] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 34 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:36.415020 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 30 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:36.416325 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:36.416679 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:36.419628 17764 raft_consensus.cc:2410] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for term 30 because replica has last-logged OpId of term: 28 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 27 index: 14.
I20260709 12:25:36.419907 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 31 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:36.423249 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 29 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:25:36.423638 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 60793.4us
I20260709 12:25:36.424398 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 30 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:36.424885 17756 raft_consensus.cc:2468] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 33.
I20260709 12:25:36.425381 17831 raft_consensus.cc:2410] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for term 30 because replica has last-logged OpId of term: 28 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 27 index: 14.
I20260709 12:25:36.375835 18371 raft_consensus.cc:2468] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 32.
I20260709 12:25:36.427565 17837 raft_consensus.cc:1240] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 36 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 35. Current term is 36. Ops: []
I20260709 12:25:36.381183 28638 raft_consensus.cc:697] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 31 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:25:36.429627 17803 leader_election.cc:304] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:25:36.433207 17830 raft_consensus.cc:1275] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 25 index: 22. Preceding OpId from leader: term: 27 index: 23. (index mismatch)
W20260709 12:25:36.434127 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 73781.8us
I20260709 12:25:36.429947 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:36.431674 17654 leader_election.cc:304] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:25:36.442476 17838 raft_consensus.cc:2393] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 28: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:25:36.443423 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 29 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:36.444408 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 35 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:36.396729 28305 consensus_queue.cc:1048] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.040s
I20260709 12:25:36.344810 28503 consensus_queue.cc:237] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 31.19, Last appended by leader: 19, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:36.453828 28734 raft_consensus.cc:493] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:36.455418 28728 raft_consensus.cc:2804] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:25:36.433434 28126 raft_consensus.cc:493] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:36.436707 28732 raft_consensus.cc:2749] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 12:25:36.456785 28308 leader_election.cc:290] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:36.460737 28651 leader_election.cc:290] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:36.462081 28630 leader_election.cc:290] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 31 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:25:36.462639 28721 raft_consensus.cc:2804] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:25:36.464437 28638 consensus_queue.cc:237] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 30.19, Last appended by leader: 19, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:36.414072 28645 consensus_queue.cc:260] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 25.17, Last appended by leader: 17, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:36.451366 28731 raft_consensus.cc:2749] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 34 FOLLOWER]: Leader election lost for term 34. Reason: could not achieve majority
W20260709 12:25:36.470866 28618 consensus_meta.cc:287] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.699s	user 0.010s	sys 0.002s
I20260709 12:25:36.471146 28699 consensus_queue.cc:1059] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 27.17, Next index: 18, Last known committed idx: 16, Time since last communication: 0.000s
I20260709 12:25:36.471941 28682 raft_consensus.cc:2749] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:25:36.472559 28390 consensus_queue.cc:237] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 33.21, Last appended by leader: 21, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:36.354131 28725 raft_consensus.cc:515] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:36.474963 28665 consensus_meta.cc:287] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.688s	user 0.010s	sys 0.000s
I20260709 12:25:36.441018 28501 raft_consensus.cc:2749] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 36 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:25:36.479914 28733 raft_consensus.cc:493] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:36.512178 17727 leader_election.cc:304] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:36.448859 28650 raft_consensus.cc:3060] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Advancing to term 31
W20260709 12:25:36.518241 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 86178.9us
I20260709 12:25:36.424085 28642 raft_consensus.cc:515] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:36.521648 28642 leader_election.cc:290] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 29 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:36.481809 28728 raft_consensus.cc:493] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:36.525453 28728 raft_consensus.cc:3060] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Advancing to term 30
W20260709 12:25:36.528733 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 91373.5us
W20260709 12:25:36.506094 17802 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:36.438230 28717 consensus_queue.cc:1048] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.051s
I20260709 12:25:36.513417 17727 leader_election.cc:304] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
W20260709 12:25:36.505795 28724 log.cc:927] Time spent T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.208s	user 0.001s	sys 0.001s
W20260709 12:25:36.524580 28727 log.cc:927] Time spent T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.162s	user 0.001s	sys 0.000s
I20260709 12:25:36.536042 17727 leader_election.cc:304] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:25:36.549091 17802 leader_election.cc:336] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 26 election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:36.556339 18371 raft_consensus.cc:1240] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 32 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 31. Current term is 32. Ops: []
W20260709 12:25:36.559985 18334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:36.557878 17802 leader_election.cc:336] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:36.485208 28126 raft_consensus.cc:515] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:36.586454 17802 leader_election.cc:304] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:25:36.588107 17652 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:36.528021 28320 raft_consensus.cc:2749] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260709 12:25:36.483937 28734 raft_consensus.cc:515] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:36.483060 28730 consensus_queue.cc:1059] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 34.23, Next index: 24, Last known committed idx: 23, Time since last communication: 0.074s
W20260709 12:25:36.593242 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 163680us
I20260709 12:25:36.529130 28645 raft_consensus.cc:3060] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Advancing to term 26
W20260709 12:25:36.596851 17802 leader_election.cc:336] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 30 election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:36.597646 17652 leader_election.cc:400] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 35 election: Vote denied by peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651) with higher term. Message: Invalid argument: T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 36 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for earlier term 35. Current term is 36.
I20260709 12:25:36.560987 28425 raft_consensus.cc:2804] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:25:36.600621 28425 raft_consensus.cc:493] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:25:36.600773 17802 leader_election.cc:304] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:25:36.603135 17762 raft_consensus.cc:1275] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 26 index: 18. Preceding OpId from leader: term: 28 index: 19. (index mismatch)
I20260709 12:25:36.559108 28614 raft_consensus.cc:2749] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
W20260709 12:25:36.608068 17802 leader_election.cc:336] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:36.586354 28721 raft_consensus.cc:697] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 30 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:25:36.602319 28425 raft_consensus.cc:3060] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:36.624854 18363 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59886 (request call id 2878) took 1407 ms. Trace:
W20260709 12:25:36.624759 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 69357.7us
I20260709 12:25:36.625494 18363 rpcz_store.cc:276] 0709 12:25:35.217932 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:35.368819 (+150887us) service_pool.cc:224] Handling call
0709 12:25:35.369086 (+   267us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:35.375148 (+  6062us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:35.375179 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:35.379294 (+  4115us) log.cc:844] Serialized 26 byte log entry
0709 12:25:36.137091 (+757797us) spinlock_profiling.cc:243] Waited 5.26 ms on lock 0x7b1800488860. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b503fbab1 00007f7b50408281 00007f7b47515e07 00007f7b503709a9 00007f7b4753269b 00007f7b4753093f 00007f7b47532c02 00007f7b4a1ba20b 00007f7b4a1bfd23 00007f7b4a1c00d1 00007f7b4a5dba0f
0709 12:25:36.192965 (+ 55874us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:25:36.193006 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:36.193065 (+    59us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:36.614262 (+421197us) raft_consensus.cc:1685] finished
0709 12:25:36.614291 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:36.624791 (+ 10500us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":7924,"raft.queue_time_us":7909,"raft.run_cpu_time_us":130,"raft.run_wall_time_us":8481,"spinlock_wait_cycles":12098432,"thread_start_us":55688,"threads_started":1,"wal-append.queue_time_us":927148}
I20260709 12:25:36.600103 28699 raft_consensus.cc:2749] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:25:36.512769 28665 raft_consensus.cc:515] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:36.603868 28744 raft_consensus.cc:493] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:25:36.586468 28731 consensus_queue.cc:1059] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.039s
W20260709 12:25:36.658447 17802 leader_election.cc:336] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:36.595858 28682 consensus_queue.cc:1048] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.039s
I20260709 12:25:36.738262 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
W20260709 12:25:36.739073 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 80685.7us
I20260709 12:25:36.558795 28733 raft_consensus.cc:515] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:36.750027 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 33 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:36.754586 17759 raft_consensus.cc:2468] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 32.
I20260709 12:25:36.633248 28630 raft_consensus.cc:2749] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
I20260709 12:25:36.697896 28740 raft_consensus.cc:493] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:25:36.798465 28702 raft_consensus.cc:493] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 34 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:25:36.801245 17759 raft_consensus.cc:1240] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 35 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 34. Current term is 35. Ops: []
I20260709 12:25:36.746219 28610 raft_consensus.cc:493] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:36.811048 28610 raft_consensus.cc:515] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:36.811743 28733 leader_election.cc:290] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:36.812341 28733 raft_consensus.cc:493] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:25:36.812853 28733 raft_consensus.cc:515] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:36.813236 28610 leader_election.cc:290] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:36.813773 28610 raft_consensus.cc:493] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:36.814168 28610 raft_consensus.cc:515] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:36.810067 28721 consensus_queue.cc:237] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 28.20, Last appended by leader: 20, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:36.817632 17764 raft_consensus.cc:1275] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 33 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 31 index: 21. Preceding OpId from leader: term: 33 index: 22. (index mismatch)
I20260709 12:25:36.745946 28744 raft_consensus.cc:515] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:36.816877 28610 leader_election.cc:290] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:25:36.759774 28509 raft_consensus.cc:2749] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:25:36.796541 28308 raft_consensus.cc:493] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
W20260709 12:25:36.835119 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 403495us
I20260709 12:25:36.838016 17654 leader_election.cc:304] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
W20260709 12:25:36.834259 28703 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.927s	user 0.007s	sys 0.009s
W20260709 12:25:36.837978 28745 log.cc:927] Time spent T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.085s	user 0.001s	sys 0.000s
W20260709 12:25:36.850441 18334 leader_election.cc:336] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 34 election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Timed out: RequestConsensusVote RPC to 127.16.29.65:33071 timed out after 1.545s (SENT)
I20260709 12:25:36.848870 17654 leader_election.cc:304] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:25:36.848369 28703 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.945s	user 0.007s	sys 0.010s
I20260709 12:25:36.748478 28603 consensus_queue.cc:260] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 29.20, Last appended by leader: 20, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:36.868402 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 241076us
W20260709 12:25:36.876813 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 110108us
I20260709 12:25:36.809311 28740 raft_consensus.cc:515] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:36.829871 28702 raft_consensus.cc:515] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:36.849282 28682 raft_consensus.cc:2749] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
W20260709 12:25:36.856721 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 245356us
I20260709 12:25:36.888383 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:36.888451 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 29 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:36.852871 28665 leader_election.cc:290] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 23 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:36.891163 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 25 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:36.898031 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 29 candidate_status { last_received { term: 28 index: 22 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:36.877300 28725 leader_election.cc:290] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:36.869465 28526 raft_consensus.cc:493] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 36 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:25:36.875833 28741 raft_consensus.cc:493] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:36.888429 28731 raft_consensus.cc:2804] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:25:36.901829 28757 raft_consensus.cc:493] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:36.901712 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:36.934037 18333 leader_election.cc:304] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 35 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:25:36.902135 28750 log.cc:927] Time spent T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.065s	user 0.000s	sys 0.001s
I20260709 12:25:36.902302 28509 raft_consensus.cc:493] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:36.903095 18363 raft_consensus.cc:1275] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 25 index: 22. Preceding OpId from leader: term: 27 index: 23. (index mismatch)
I20260709 12:25:36.904388 28740 leader_election.cc:290] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:25:36.841349 28308 raft_consensus.cc:515] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:36.925402 18371 raft_consensus.cc:2410] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for term 25 because replica has last-logged OpId of term: 24 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 16.
I20260709 12:25:36.852097 28644 consensus_queue.cc:1048] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.047s
I20260709 12:25:36.890617 17727 leader_election.cc:304] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 36 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:36.889613 28610 raft_consensus.cc:493] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:36.929697 18367 raft_consensus.cc:2410] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for term 28 because replica has last-logged OpId of term: 27 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 26 index: 18.
W20260709 12:25:36.932744 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 48397.9us
I20260709 12:25:36.934507 18368 raft_consensus.cc:3060] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Advancing to term 28
W20260709 12:25:36.953089 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 62593.3us
I20260709 12:25:36.953747 17727 leader_election.cc:304] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:25:36.954486 28754 raft_consensus.cc:2804] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 36 FOLLOWER]: Leader election won for term 36
I20260709 12:25:36.955039 28754 raft_consensus.cc:697] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 36 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:25:36.934828 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 33 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:36.936918 17763 raft_consensus.cc:2410] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for term 31 because replica has last-logged OpId of term: 30 index: 21, which is greater than that of the candidate, which has last-logged OpId of term: 28 index: 20.
I20260709 12:25:36.938400 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 29 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:36.957325 18365 raft_consensus.cc:2468] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 28.
I20260709 12:25:36.957135 28754 consensus_queue.cc:237] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 34.20, Last appended by leader: 20, Current term: 36, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:25:36.959017 18334 leader_election.cc:336] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 31 election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:36.926738 28561 raft_consensus.cc:493] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:36.967142 18334 leader_election.cc:336] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:36.967676 18368 raft_consensus.cc:2410] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for term 28 because replica has last-logged OpId of term: 27 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 26 index: 17.
I20260709 12:25:36.969745 18334 leader_election.cc:304] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:36.941066 28725 raft_consensus.cc:493] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:36.938585 28126 leader_election.cc:290] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:25:36.974378 18334 leader_election.cc:336] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 34 election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:36.944082 28603 raft_consensus.cc:3060] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:36.976895 18334 leader_election.cc:304] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 34 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:36.937943 28692 raft_consensus.cc:493] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:36.942893 28731 raft_consensus.cc:697] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 30 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:25:36.959287 28708 raft_consensus.cc:493] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:36.991087 18334 leader_election.cc:336] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:36.934795 28526 raft_consensus.cc:515] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:36.994527 18334 leader_election.cc:304] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:36.997701 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:36.994551 28692 raft_consensus.cc:515] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:37.009509 17759 raft_consensus.cc:3060] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 21 FOLLOWER]: Advancing to term 23
I20260709 12:25:37.011622 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 31 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:36.950081 28509 raft_consensus.cc:515] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:37.020799 18334 consensus_peers.cc:597] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:25:36.968694 28610 raft_consensus.cc:515] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:36.995657 28733 leader_election.cc:290] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:36.993916 28638 raft_consensus.cc:2749] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 35 FOLLOWER]: Leader election lost for term 35. Reason: could not achieve majority
I20260709 12:25:36.935281 28741 raft_consensus.cc:515] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:25:37.025599 18334 leader_election.cc:336] T a5732e71b9d54f8786b8c511d69943bf P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 31 election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:37.006880 28751 raft_consensus.cc:2804] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:25:37.046856 28744 leader_election.cc:290] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:37.049458 28485 raft_consensus.cc:493] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:36.974969 28561 raft_consensus.cc:515] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:37.053941 18334 leader_election.cc:336] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:37.059258 18334 leader_election.cc:304] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:36.987233 28725 raft_consensus.cc:515] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:25:37.062885 18334 leader_election.cc:336] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 34 election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:37.011510 28708 raft_consensus.cc:515] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:36.946480 28757 raft_consensus.cc:515] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:37.064458 18334 leader_election.cc:336] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 26 election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:25:37.064791 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 109756us
I20260709 12:25:37.059990 17759 raft_consensus.cc:2468] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 23.
I20260709 12:25:37.072180 28725 leader_election.cc:290] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:37.056667 28753 raft_consensus.cc:2749] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 34 FOLLOWER]: Leader election lost for term 34. Reason: could not achieve majority
I20260709 12:25:37.055193 28733 raft_consensus.cc:2749] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
W20260709 12:25:37.079829 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 99645.9us
W20260709 12:25:37.080582 17728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:37.091275 17804 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:37.063776 28760 raft_consensus.cc:2749] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:25:37.073132 28638 raft_consensus.cc:2749] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:25:37.084920 28731 consensus_queue.cc:237] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 29.20, Last appended by leader: 20, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:37.104660 28752 log.cc:927] Time spent T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.059s	user 0.001s	sys 0.000s
I20260709 12:25:37.105780 28757 leader_election.cc:290] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:37.087405 28769 consensus_queue.cc:1048] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.028s
I20260709 12:25:37.110913 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 29 candidate_status { last_received { term: 28 index: 22 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:37.063503 28485 raft_consensus.cc:515] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:37.114777 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 180840us
I20260709 12:25:37.115748 18333 leader_election.cc:400] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 29 election: Vote denied by peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837) with higher term. Message: Invalid argument: T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 29. Current term is 30.
I20260709 12:25:37.116128 18333 leader_election.cc:403] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 29 election: Cancelling election due to peer responding with higher term
I20260709 12:25:37.117535 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 33 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:37.114135 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 25 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:37.118799 28485 leader_election.cc:290] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:25:37.119810 28761 log.cc:927] Time spent T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.059s	user 0.000s	sys 0.001s
I20260709 12:25:37.126400 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:37.123143 18333 leader_election.cc:304] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:25:37.127239 17838 raft_consensus.cc:3060] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:37.111308 28744 raft_consensus.cc:493] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:25:37.139940 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 30 candidate_status { last_received { term: 28 index: 22 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:37.140303 17803 leader_election.cc:304] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:25:37.150882 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 33 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:37.138430 28638 raft_consensus.cc:2804] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:25:37.159395 28638 raft_consensus.cc:697] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 30 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:25:37.160331 28638 consensus_queue.cc:237] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 28.17, Last appended by leader: 17, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:37.160534 28425 raft_consensus.cc:515] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:37.161679 17759 raft_consensus.cc:1240] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 30. Current term is 31. Ops: []
W20260709 12:25:37.162423 28728 consensus_meta.cc:287] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.636s	user 0.007s	sys 0.003s
I20260709 12:25:37.162742 28561 leader_election.cc:290] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:37.165359 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 32 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:37.166513 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:37.136686 28753 raft_consensus.cc:2749] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Leader election lost for term 29. Reason: Vote denied by peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837) with higher term. Message: Invalid argument: T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 29. Current term is 30.
I20260709 12:25:37.169044 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 35 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:37.169625 17755 raft_consensus.cc:2468] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 34.
I20260709 12:25:37.173709 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:37.151115 28742 raft_consensus.cc:2749] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:25:37.151892 28702 leader_election.cc:290] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:37.151381 28757 raft_consensus.cc:493] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:37.177655 28741 leader_election.cc:290] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:25:37.178860 17755 raft_consensus.cc:2393] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 31: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:25:37.178735 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 35 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:37.205152 17831 raft_consensus.cc:2468] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 34.
I20260709 12:25:37.205310 28757 raft_consensus.cc:515] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:37.206102 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:37.206898 17831 raft_consensus.cc:2393] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 31: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:25:37.207538 28757 leader_election.cc:290] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:37.162832 28728 raft_consensus.cc:515] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:37.213167 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 30 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:37.213376 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 30 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:37.214329 17760 raft_consensus.cc:2410] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for term 30 because replica has last-logged OpId of term: 28 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 27 index: 14.
I20260709 12:25:37.214226 18366 raft_consensus.cc:2468] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 29.
I20260709 12:25:37.152462 28744 raft_consensus.cc:515] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:25:37.215512 17728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:37.216365 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 33 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:37.216261 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 31 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:37.219725 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 29 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:37.219630 17838 raft_consensus.cc:2468] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 23.
I20260709 12:25:37.220561 18366 raft_consensus.cc:2468] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 28.
I20260709 12:25:37.222591 17760 raft_consensus.cc:2393] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 33: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:25:37.223847 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:37.223991 17830 raft_consensus.cc:2393] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 30: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
W20260709 12:25:37.224110 17728 leader_election.cc:336] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 30 election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:37.224764 17728 leader_election.cc:304] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
W20260709 12:25:37.222273 28559 consensus_meta.cc:287] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.803s	user 0.012s	sys 0.001s
I20260709 12:25:37.211688 28692 leader_election.cc:290] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:37.246830 28651 raft_consensus.cc:3055] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 35 LEADER]: Stepping down as leader of term 35
I20260709 12:25:37.247476 28651 raft_consensus.cc:740] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 35 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:25:37.250751 28610 leader_election.cc:290] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:37.251112 28734 leader_election.cc:290] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:37.253278 17832 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35040 (request call id 3941) took 1586 ms. Trace:
I20260709 12:25:37.253664 17832 rpcz_store.cc:276] 0709 12:25:35.666605 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:35.747518 (+ 80913us) service_pool.cc:224] Handling call
0709 12:25:35.747752 (+   234us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:36.332678 (+584926us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:37.253062 (+920384us) spinlock_profiling.cc:243] Waited 559 ms on lock 0x7b54002ebb54. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:25:37.253222 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1285330176}
W20260709 12:25:37.256723 17763 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38776 (request call id 3569) took 2317 ms (client timeout 1792 ms). Trace:
W20260709 12:25:37.257112 17763 rpcz_store.cc:269] 0709 12:25:34.939169 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:35.239653 (+300484us) service_pool.cc:224] Handling call
0709 12:25:37.256522 (+2016869us) spinlock_profiling.cc:243] Waited 1.67 s on lock 0x7b5400400bd4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:25:37.256654 (+   132us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3845069184}
I20260709 12:25:37.259737 28425 leader_election.cc:290] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 31 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:37.215611 28753 raft_consensus.cc:493] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:25:37.261282 28565 raft_consensus.cc:3055] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 34 LEADER]: Stepping down as leader of term 34
I20260709 12:25:37.261211 28753 raft_consensus.cc:515] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:37.261700 28565 raft_consensus.cc:740] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 34 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:25:37.263005 28565 consensus_queue.cc:260] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 34.19, Last appended by leader: 19, Current term: 34, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:37.263492 28753 leader_election.cc:290] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:37.264593 28559 raft_consensus.cc:515] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 32 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:37.266805 28559 leader_election.cc:290] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 32 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:25:37.268098 28715 log.cc:927] Time spent T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.927s	user 0.001s	sys 0.002s
I20260709 12:25:37.268759 17652 leader_election.cc:304] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:25:37.269116 28645 consensus_meta.cc:287] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.663s	user 0.011s	sys 0.000s
W20260709 12:25:37.265084 28737 log.cc:927] Time spent T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.670s	user 0.002s	sys 0.000s
I20260709 12:25:37.270108 17652 leader_election.cc:304] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 31 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:37.271251 17652 leader_election.cc:304] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:37.271974 28721 raft_consensus.cc:2749] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:25:37.272172 28757 raft_consensus.cc:2749] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Leader election lost for term 31. Reason: could not achieve majority
I20260709 12:25:37.272706 28721 raft_consensus.cc:2749] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:25:37.275789 17654 leader_election.cc:304] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:37.277225 17654 leader_election.cc:304] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:37.279202 28726 raft_consensus.cc:2749] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260709 12:25:37.279852 28726 raft_consensus.cc:2749] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
W20260709 12:25:37.287600 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 63506.3us
I20260709 12:25:37.288312 17654 leader_election.cc:304] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
W20260709 12:25:37.288777 17728 leader_election.cc:336] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:25:37.289994 17728 leader_election.cc:336] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:37.290505 17728 leader_election.cc:304] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:37.290678 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 32 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:37.292035 17652 leader_election.cc:304] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:37.280467 28726 consensus_queue.cc:1048] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.013s
I20260709 12:25:37.295663 17654 leader_election.cc:304] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:37.295967 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:37.296684 17832 raft_consensus.cc:2468] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 23.
W20260709 12:25:37.296921 17728 leader_election.cc:336] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:37.297345 17728 leader_election.cc:304] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:25:37.297366 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 33 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:37.296845 17654 leader_election.cc:304] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:37.299661 28561 raft_consensus.cc:2749] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:25:37.300269 28425 raft_consensus.cc:2749] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 33 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:25:37.301602 28749 raft_consensus.cc:2749] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
I20260709 12:25:37.302533 28744 leader_election.cc:290] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:25:37.303138 28501 raft_consensus.cc:3055] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 35 LEADER]: Stepping down as leader of term 35
I20260709 12:25:37.303526 28501 raft_consensus.cc:740] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 35 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:25:37.304049 17836 raft_consensus.cc:2393] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 33: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
W20260709 12:25:37.305392 28756 log.cc:927] Time spent T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.371s	user 0.002s	sys 0.000s
I20260709 12:25:37.309013 17654 leader_election.cc:304] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:25:37.309396 28735 log.cc:927] Time spent T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.781s	user 0.001s	sys 0.001s
I20260709 12:25:37.323035 18364 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59886 (request call id 2891) took 1237 ms. Trace:
I20260709 12:25:37.323413 18364 rpcz_store.cc:276] 0709 12:25:36.086170 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:36.254539 (+168369us) service_pool.cc:224] Handling call
0709 12:25:36.254841 (+   302us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 12:25:36.257953 (+  3112us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:36.258000 (+    47us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 12:25:36.258860 (+   860us) log.cc:844] Serialized 78 byte log entry
0709 12:25:36.419802 (+160942us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:25:36.601019 (+181217us) log.cc:844] Serialized 14 byte log entry
0709 12:25:36.601435 (+   416us) log.cc:844] Serialized 14 byte log entry
0709 12:25:36.604204 (+  2769us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:36.604306 (+   102us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:37.308826 (+704520us) spinlock_profiling.cc:243] Waited 50 ms on lock 0x7b4400022b4c. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b47459c85 00007f7b4745a6a2 00007f7b4a1cc1db 00007f7b4a1c7aaf 00007f7b4a1bb86e 00007f7b4a1babb6 00007f7b4a1ba7d8 00007f7b4a1c4221 00007f7b4a1c41d9 00007f7b4a1c4169
0709 12:25:37.322784 (+ 13958us) raft_consensus.cc:1685] finished
0709 12:25:37.322809 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:37.322974 (+   165us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":6350,"prepare.queue_time_us":545483,"prepare.run_cpu_time_us":80,"prepare.run_wall_time_us":84,"raft.queue_time_us":5425,"raft.run_cpu_time_us":379,"raft.run_wall_time_us":14756,"spinlock_wait_cycles":114917248,"thread_start_us":270663,"threads_started":2,"wal-append.queue_time_us":202963}
I20260709 12:25:37.309715 28726 raft_consensus.cc:2749] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:25:37.307300 28501 consensus_queue.cc:260] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 35.24, Last appended by leader: 24, Current term: 35, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:37.326817 17728 consensus_peers.cc:597] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:25:37.313068 28728 leader_election.cc:290] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 30 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:37.313457 28559 raft_consensus.cc:3060] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:37.295858 17729 leader_election.cc:304] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:37.327955 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 30 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:37.248553 28651 consensus_queue.cc:260] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 35.21, Last appended by leader: 21, Current term: 35, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:37.329124 17729 leader_election.cc:304] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 36 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:37.319999 28603 consensus_queue.cc:1048] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:25:37.329619 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 35 candidate_status { last_received { term: 34 index: 24 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:37.335177 17801 leader_election.cc:304] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:37.335479 17729 leader_election.cc:304] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:25:37.338142 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:37.332409 17763 raft_consensus.cc:1240] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 27. Current term is 28. Ops: []
I20260709 12:25:37.332849 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 30 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:37.333673 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:25:37.335866 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 43909.9us
I20260709 12:25:37.331252 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 37 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:37.340205 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 32 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:37.341195 17836 raft_consensus.cc:2393] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 32: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:25:37.336568 28507 raft_consensus.cc:2749] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:25:37.339037 18364 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 29.
I20260709 12:25:37.344225 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 37 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:37.346979 17836 raft_consensus.cc:2410] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 36 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for term 37 because replica has last-logged OpId of term: 36 index: 23, which is greater than that of the candidate, which has last-logged OpId of term: 34 index: 22.
I20260709 12:25:37.348191 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 36 candidate_status { last_received { term: 34 index: 23 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:37.349845 17838 raft_consensus.cc:1275] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 29 index: 16. Preceding OpId from leader: term: 31 index: 17. (index mismatch)
I20260709 12:25:37.350273 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 30 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:37.350871 17758 raft_consensus.cc:2468] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 29.
I20260709 12:25:37.347528 17652 leader_election.cc:304] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:25:37.344321 17728 consensus_peers.cc:597] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:25:37.338379 17801 leader_election.cc:304] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:37.355320 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 33 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:37.356160 18366 raft_consensus.cc:2468] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 32.
I20260709 12:25:37.384902 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:37.385702 17838 raft_consensus.cc:2393] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 31: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:25:37.333679 28561 raft_consensus.cc:2749] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260709 12:25:37.342308 28425 raft_consensus.cc:2804] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 36 FOLLOWER]: Leader election won for term 36
I20260709 12:25:37.356513 18364 raft_consensus.cc:1275] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 29 index: 20. Preceding OpId from leader: term: 31 index: 21. (index mismatch)
I20260709 12:25:37.387635 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 33 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:37.334378 28749 raft_consensus.cc:2764] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 33 FOLLOWER]: Leader pre-election decision vote started in defunct term 32: won
I20260709 12:25:37.393951 18361 raft_consensus.cc:3060] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:25:37.386904 28782 raft_consensus.cc:2804] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 32 FOLLOWER]: Leader election won for term 32
I20260709 12:25:37.393404 17727 leader_election.cc:304] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:25:37.386672 28778 consensus_queue.cc:1059] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 31.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.035s
I20260709 12:25:37.337163 28744 raft_consensus.cc:493] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:37.345443 28309 raft_consensus.cc:2804] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 33 FOLLOWER]: Leader election won for term 33
I20260709 12:25:37.342429 28637 raft_consensus.cc:2749] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:25:37.351804 28501 raft_consensus.cc:3060] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:25:37.386395 28682 raft_consensus.cc:3060] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:37.386502 28728 raft_consensus.cc:2804] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:25:37.419802 28728 raft_consensus.cc:493] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:37.420321 28728 raft_consensus.cc:3060] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:25:37.402529 28425 raft_consensus.cc:697] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 36 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:25:37.389636 28620 consensus_queue.cc:1048] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.035s
I20260709 12:25:37.423851 18361 raft_consensus.cc:2468] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 33.
W20260709 12:25:37.406863 17802 leader_election.cc:336] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:37.411247 17801 leader_election.cc:304] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:37.397676 28782 raft_consensus.cc:697] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 32 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:25:37.345635 28665 consensus_queue.cc:1059] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 34.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.053s
I20260709 12:25:37.418314 28726 raft_consensus.cc:493] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:37.418877 28309 raft_consensus.cc:697] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 33 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:25:37.459708 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 30 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:37.387742 28651 raft_consensus.cc:3060] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 35 FOLLOWER]: Advancing to term 36
W20260709 12:25:37.448309 17802 leader_election.cc:336] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:25:37.449221 17728 leader_election.cc:336] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 36 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:25:37.462607 17728 leader_election.cc:336] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 35 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:25:37.463718 17728 consensus_peers.cc:597] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:25:37.466212 17728 leader_election.cc:336] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:25:37.458461 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 47272.9us
W20260709 12:25:37.467681 17728 leader_election.cc:336] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:25:37.461454 17802 leader_election.cc:336] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 31 election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:37.425036 28425 consensus_queue.cc:237] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 34.23, Last appended by leader: 23, Current term: 36, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:37.411562 28744 raft_consensus.cc:515] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:37.424243 28507 raft_consensus.cc:2804] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:25:37.427646 28751 raft_consensus.cc:493] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:25:37.468243 17728 leader_election.cc:304] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:25:37.425242 28702 consensus_queue.cc:1059] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 27.17, Next index: 18, Last known committed idx: 16, Time since last communication: 0.335s
I20260709 12:25:37.472267 28726 raft_consensus.cc:515] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:37.509063 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 30 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:25:37.761899 28762 log.cc:927] Time spent T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.726s	user 0.000s	sys 0.002s
I20260709 12:25:37.761993 17837 raft_consensus.cc:2468] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 29.
I20260709 12:25:37.762502 28726 leader_election.cc:290] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:37.530709 28665 raft_consensus.cc:493] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:37.524185 28309 consensus_queue.cc:237] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 31.19, Last appended by leader: 19, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:37.763460 28665 raft_consensus.cc:515] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:37.764659 28746 log.cc:927] Time spent T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.913s	user 0.003s	sys 0.000s
I20260709 12:25:37.765928 28665 leader_election.cc:290] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:25:37.694737 28485 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.516s	user 0.013s	sys 0.030s
W20260709 12:25:37.766719 28485 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.588s	user 0.014s	sys 0.035s
I20260709 12:25:37.518338 18365 raft_consensus.cc:1240] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 36 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 35. Current term is 36. Ops: [35.24-35.24]
I20260709 12:25:37.520959 28782 consensus_queue.cc:237] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 31.22, Last appended by leader: 22, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:37.518792 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 30 candidate_status { last_received { term: 28 index: 22 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:37.768844 18374 raft_consensus.cc:3060] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Advancing to term 30
W20260709 12:25:37.543392 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 183474us
W20260709 12:25:37.536154 17804 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:37.620152 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 257314us
I20260709 12:25:37.517704 17803 leader_election.cc:304] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:25:37.475950 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 30 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:37.777743 17838 raft_consensus.cc:2410] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for term 30 because replica has last-logged OpId of term: 28 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 27 index: 14.
W20260709 12:25:37.616739 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 223420us
W20260709 12:25:37.751055 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 44856.7us
I20260709 12:25:37.759537 17802 leader_election.cc:304] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 31 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:37.773810 17729 leader_election.cc:304] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:37.782394 18333 leader_election.cc:304] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:25:37.482611 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 30 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:37.789320 18374 raft_consensus.cc:2468] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 30.
I20260709 12:25:37.572168 17755 raft_consensus.cc:1275] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 33 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 31 index: 19. Preceding OpId from leader: term: 33 index: 20. (index mismatch)
I20260709 12:25:37.519279 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:25:37.639837 17730 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:37.482189 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 35 candidate_status { last_received { term: 34 index: 24 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:37.792284 18368 raft_consensus.cc:2393] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 35: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
W20260709 12:25:37.793757 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 326086us
W20260709 12:25:37.794693 17728 leader_election.cc:336] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 31 election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:37.795205 17728 leader_election.cc:304] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 31 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
W20260709 12:25:37.796655 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 279042us
W20260709 12:25:37.798233 17804 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:37.694548 28723 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 1.179s	user 0.022s	sys 0.017s
I20260709 12:25:37.804360 18333 leader_election.cc:304] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:37.772034 17652 leader_election.cc:304] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:37.802233 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:37.688392 28651 raft_consensus.cc:493] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:37.808531 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 32 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:37.738301 28741 raft_consensus.cc:493] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:37.725243 28559 raft_consensus.cc:2749] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:25:37.735680 28778 raft_consensus.cc:493] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:37.809510 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 348070us
I20260709 12:25:37.806660 17652 leader_election.cc:304] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:37.537147 28675 raft_consensus.cc:3060] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:25:37.468629 28636 raft_consensus.cc:2804] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:25:37.811781 28665 raft_consensus.cc:2804] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:25:37.823568 17831 raft_consensus.cc:1275] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 25 index: 19. Preceding OpId from leader: term: 27 index: 20. (index mismatch)
I20260709 12:25:37.809257 28772 raft_consensus.cc:2804] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:25:37.534000 28708 leader_election.cc:290] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:25:37.809980 28723 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.290s	user 0.023s	sys 0.018s
I20260709 12:25:37.785369 28495 raft_consensus.cc:493] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:37.836880 28559 consensus_queue.cc:1048] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20260709 12:25:37.770244 28751 raft_consensus.cc:3060] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Advancing to term 25
W20260709 12:25:37.836031 17802 consensus_peers.cc:597] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:25:37.839807 28786 log.cc:927] Time spent T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.092s	user 0.000s	sys 0.000s
I20260709 12:25:37.835304 28753 raft_consensus.cc:2749] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
W20260709 12:25:37.843571 28739 log.cc:927] Time spent T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 1.338s	user 0.003s	sys 0.000s
I20260709 12:25:37.835944 28702 raft_consensus.cc:2749] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Leader election lost for term 31. Reason: could not achieve majority
I20260709 12:25:37.817775 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:37.815030 28790 raft_consensus.cc:2749] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:25:37.511336 28509 leader_election.cc:290] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:25:37.831854 17652 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:37.748073 28728 raft_consensus.cc:515] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 33 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:37.835728 17728 leader_election.cc:336] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:37.844022 28729 raft_consensus.cc:493] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:37.816984 17803 leader_election.cc:304] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:25:37.818576 28651 raft_consensus.cc:515] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:37.784672 28507 raft_consensus.cc:493] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:25:37.814239 28677 raft_consensus.cc:2804] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:25:37.876307 28677 raft_consensus.cc:493] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:37.876981 28677 raft_consensus.cc:3060] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:37.813959 28308 leader_election.cc:290] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:37.851552 17728 leader_election.cc:304] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:37.827692 28726 raft_consensus.cc:2804] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:25:37.880363 28726 raft_consensus.cc:493] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 34 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:25:37.837786 28783 log.cc:927] Time spent T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.109s	user 0.001s	sys 0.000s
W20260709 12:25:37.880851 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 45159.7us
I20260709 12:25:37.880967 28726 raft_consensus.cc:3060] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 34 FOLLOWER]: Advancing to term 35
W20260709 12:25:37.888000 17804 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:37.862931 17832 raft_consensus.cc:2410] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 36 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for term 36 because replica has last-logged OpId of term: 35 index: 24, which is greater than that of the candidate, which has last-logged OpId of term: 34 index: 23.
I20260709 12:25:37.828581 28706 raft_consensus.cc:2804] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:25:37.844492 28665 raft_consensus.cc:697] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 23 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:25:37.828307 28772 raft_consensus.cc:493] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:37.892999 28772 raft_consensus.cc:3060] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:37.893615 28753 raft_consensus.cc:493] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:37.894104 28753 raft_consensus.cc:515] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:37.891736 28790 raft_consensus.cc:2749] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:25:37.828758 28778 raft_consensus.cc:515] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:37.889462 28706 raft_consensus.cc:493] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:37.896275 28706 raft_consensus.cc:3060] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:37.897650 28778 leader_election.cc:290] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:37.829329 28636 raft_consensus.cc:493] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:37.899982 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 30 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:37.900316 28651 leader_election.cc:290] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:37.876389 28507 raft_consensus.cc:3060] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:37.901290 28148 raft_consensus.cc:493] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 33 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:25:37.901582 28665 consensus_queue.cc:237] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 21.14, Last appended by leader: 14, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:37.903157 28693 raft_consensus.cc:3060] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:37.837430 28495 raft_consensus.cc:515] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:37.889665 28501 raft_consensus.cc:2804] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:25:37.904170 28501 raft_consensus.cc:493] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:37.904549 28501 raft_consensus.cc:3060] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:37.863752 28769 raft_consensus.cc:2749] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Leader election lost for term 31. Reason: could not achieve majority
I20260709 12:25:37.892931 28303 raft_consensus.cc:3055] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:25:37.901836 28148 raft_consensus.cc:515] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:37.900964 17830 raft_consensus.cc:2393] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 30: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:25:37.864095 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 34 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:25:37.865756 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 48863.8us
I20260709 12:25:37.915235 17803 leader_election.cc:304] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:25:37.864262 17652 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:37.900128 28636 raft_consensus.cc:3060] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:37.920852 17652 leader_election.cc:304] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:37.819453 28741 raft_consensus.cc:515] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:37.922104 17803 leader_election.cc:304] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:25:37.912763 28303 raft_consensus.cc:740] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 27 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:25:37.866510 28729 raft_consensus.cc:515] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:25:37.947073 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 142824us
I20260709 12:25:37.927693 17652 leader_election.cc:304] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:25:37.948474 17652 leader_election.cc:304] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:25:37.958231 17802 consensus_peers.cc:597] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:25:37.921763 28599 raft_consensus.cc:2749] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
W20260709 12:25:37.966917 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 193210us
I20260709 12:25:37.928333 28753 leader_election.cc:290] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:25:37.980245 17802 leader_election.cc:336] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 37 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:37.983155 28769 consensus_queue.cc:1048] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:25:37.984237 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 34 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:37.947639 28637 raft_consensus.cc:3060] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:25:37.986286 28495 leader_election.cc:290] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:38.000237 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 32 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:38.000667 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 29 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:38.000988 17759 raft_consensus.cc:2468] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 31.
I20260709 12:25:37.988883 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 32 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:37.968990 28540 consensus_queue.cc:1059] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 30.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.021s
I20260709 12:25:37.962386 28559 raft_consensus.cc:2804] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:25:37.998437 17802 leader_election.cc:304] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:37.964973 28651 raft_consensus.cc:3060] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Advancing to term 31
W20260709 12:25:38.016820 17802 leader_election.cc:336] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:38.017520 17802 leader_election.cc:304] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:25:38.018913 17802 leader_election.cc:336] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 30 election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:25:38.038332 17802 leader_election.cc:336] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:38.006855 28706 raft_consensus.cc:515] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:38.044087 17762 raft_consensus.cc:1240] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 28. Current term is 29. Ops: [28.19-28.19]
I20260709 12:25:38.047091 28706 leader_election.cc:290] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 29 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:25:38.058689 17802 leader_election.cc:336] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:38.065778 28148 leader_election.cc:290] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:25:38.066171 28741 leader_election.cc:290] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:25:38.069118 28729 leader_election.cc:290] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:38.015993 28308 raft_consensus.cc:2804] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:25:38.012984 28789 raft_consensus.cc:493] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:38.015224 28559 raft_consensus.cc:493] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 34 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:25:38.061923 17802 consensus_peers.cc:597] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:25:37.997350 28677 raft_consensus.cc:515] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:38.096496 17764 raft_consensus.cc:1240] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 36 LEADER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 35. Current term is 36. Ops: []
W20260709 12:25:38.103125 28796 log.cc:927] Time spent T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.189s	user 0.000s	sys 0.001s
I20260709 12:25:38.105479 28565 raft_consensus.cc:3060] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:25:38.074790 28802 raft_consensus.cc:493] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:38.106735 28802 raft_consensus.cc:515] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:38.079377 28764 log.cc:927] Time spent T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.050s	user 0.001s	sys 0.000s
W20260709 12:25:38.110780 28682 consensus_meta.cc:287] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.679s	user 0.004s	sys 0.006s
I20260709 12:25:38.111382 28682 raft_consensus.cc:2749] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:25:38.126070 28526 leader_election.cc:290] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:25:38.136543 28728 leader_election.cc:290] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 33 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:25:38.144613 28800 log.cc:927] Time spent T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.061s	user 0.001s	sys 0.000s
I20260709 12:25:38.156087 28677 leader_election.cc:290] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 30 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:38.156720 28749 raft_consensus.cc:3055] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 31 LEADER]: Stepping down as leader of term 31
I20260709 12:25:38.157209 28749 raft_consensus.cc:740] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 31 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:25:38.157478 17835 raft_consensus.cc:2468] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 29.
I20260709 12:25:38.158041 28749 consensus_queue.cc:260] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 31.21, Last appended by leader: 21, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:38.158983 17835 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37994 (request call id 2034) took 1870 ms (client timeout 1929 ms). Trace:
I20260709 12:25:38.159329 28749 raft_consensus.cc:3060] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:25:38.159430 28503 raft_consensus.cc:493] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:38.160112 28503 raft_consensus.cc:515] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:38.159353 17835 rpcz_store.cc:269] 0709 12:25:36.296517 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:36.296710 (+   193us) service_pool.cc:224] Handling call
0709 12:25:38.158816 (+1862106us) spinlock_profiling.cc:243] Waited 1.85 s on lock 0x7b540008cad4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:25:38.158927 (+   111us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4262793856}
W20260709 12:25:38.161953 28747 log.cc:927] Time spent T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 1.410s	user 0.003s	sys 0.000s
I20260709 12:25:38.162744 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 32 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:38.163332 17835 raft_consensus.cc:2468] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 31.
I20260709 12:25:38.075312 28723 raft_consensus.cc:2749] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:25:38.079108 28308 raft_consensus.cc:493] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:38.172730 28308 raft_consensus.cc:3060] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:38.088492 28559 raft_consensus.cc:3060] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 34 FOLLOWER]: Advancing to term 35
W20260709 12:25:38.180142 28791 log.cc:927] Time spent T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.339s	user 0.002s	sys 0.000s
I20260709 12:25:38.088094 28789 raft_consensus.cc:515] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:38.182860 28789 leader_election.cc:290] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:38.028726 28309 raft_consensus.cc:2749] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:25:38.204409 17761 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38776 (request call id 3580) took 1794 ms. Trace:
I20260709 12:25:38.206995 17764 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38776 (request call id 3587) took 1312 ms. Trace:
W20260709 12:25:38.208581 28768 log.cc:927] Time spent T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 1.168s	user 0.002s	sys 0.000s
I20260709 12:25:38.215490 17761 rpcz_store.cc:276] 0709 12:25:36.410153 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:36.520804 (+110651us) service_pool.cc:224] Handling call
0709 12:25:36.521157 (+   353us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:36.521402 (+   245us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:36.521429 (+    27us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:36.522102 (+   673us) log.cc:844] Serialized 26 byte log entry
0709 12:25:36.570050 (+ 47948us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:25:36.570104 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:36.570204 (+   100us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:38.161413 (+1591209us) spinlock_profiling.cc:243] Waited 1.38 ms on lock 0x7b4400022b4c. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b47459c85 00007f7b4745a6a2 00007f7b4a1cc1db 00007f7b4a1c7aaf 00007f7b4a1bb86e 00007f7b4a1babb6 00007f7b4a1ba7d8 00007f7b4a1c4221 00007f7b4a1c41d9 00007f7b4a1c4169
0709 12:25:38.185325 (+ 23912us) raft_consensus.cc:1685] finished
0709 12:25:38.185355 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:38.204341 (+ 18986us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3176320,"thread_start_us":47455,"threads_started":1,"wal-append.queue_time_us":88238}
I20260709 12:25:38.210337 28751 raft_consensus.cc:515] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:38.212139 28503 leader_election.cc:290] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:38.215746 17764 rpcz_store.cc:276] 0709 12:25:36.894732 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:37.032865 (+138133us) service_pool.cc:224] Handling call
0709 12:25:37.033121 (+   256us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:38.116162 (+1083041us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:38.206727 (+ 90565us) spinlock_profiling.cc:243] Waited 1.05 s on lock 0x7b540019fad4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:25:38.206933 (+   206us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2416718080}
I20260709 12:25:38.221518 17757 raft_consensus.cc:2468] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 35.
W20260709 12:25:38.222541 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 234994us
W20260709 12:25:38.227515 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 183234us
W20260709 12:25:38.234508 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 312492us
I20260709 12:25:38.249090 17803 leader_election.cc:304] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:38.247107 17652 leader_election.cc:304] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:38.252955 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:38.253897 17764 raft_consensus.cc:2393] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 28: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:25:38.258457 18374 raft_consensus.cc:1240] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 32 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 31. Current term is 32. Ops: []
I20260709 12:25:38.261971 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 29 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:38.262992 18364 raft_consensus.cc:3060] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:38.272616 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:38.131286 28303 consensus_queue.cc:260] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 27.17, Last appended by leader: 17, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:38.279593 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 31 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:38.293663 18371 raft_consensus.cc:2468] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 30.
I20260709 12:25:38.280545 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 32 candidate_status { last_received { term: 30 index: 21 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:38.294718 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 30 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:38.297443 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 33 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:38.251699 17803 leader_election.cc:304] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:25:38.290906 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 32 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:38.301904 17756 raft_consensus.cc:2468] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 31.
I20260709 12:25:38.306036 28728 raft_consensus.cc:493] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:38.306861 28728 raft_consensus.cc:515] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:38.309681 28728 leader_election.cc:290] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:25:38.224277 28790 raft_consensus.cc:493] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:38.310596 28790 raft_consensus.cc:515] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:38.293071 28802 leader_election.cc:290] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:38.318747 17838 raft_consensus.cc:1275] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 28 index: 20. Preceding OpId from leader: term: 30 index: 21. (index mismatch)
I20260709 12:25:38.295089 18362 raft_consensus.cc:2468] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 31.
I20260709 12:25:38.323494 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 34 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:38.309986 28751 leader_election.cc:290] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 25 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:38.326915 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 33 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:38.225941 28819 raft_consensus.cc:493] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:38.332077 18368 raft_consensus.cc:1240] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 27. Current term is 28. Ops: []
I20260709 12:25:38.227128 28126 raft_consensus.cc:493] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:38.335803 17804 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:38.314725 28744 leader_election.cc:290] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:25:38.342504 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 90913.9us
I20260709 12:25:38.315268 28559 raft_consensus.cc:515] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 35 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:38.276939 28721 raft_consensus.cc:2804] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:25:38.358362 28721 raft_consensus.cc:493] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:38.359126 28721 raft_consensus.cc:3060] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:25:38.319329 28650 raft_consensus.cc:493] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:38.360980 28650 raft_consensus.cc:515] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:38.363430 28650 leader_election.cc:290] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:38.273383 28495 raft_consensus.cc:493] T a5732e71b9d54f8786b8c511d69943bf P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:38.342104 28819 raft_consensus.cc:515] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:38.364630 28495 raft_consensus.cc:515] T a5732e71b9d54f8786b8c511d69943bf P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:38.366108 28819 leader_election.cc:290] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:38.274729 28636 raft_consensus.cc:515] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:38.277585 28753 raft_consensus.cc:493] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:25:38.213243 28772 raft_consensus.cc:515] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:38.357703 28559 leader_election.cc:290] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 35 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:38.390317 17830 raft_consensus.cc:1240] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 29. Current term is 30. Ops: [29.20-29.20]
I20260709 12:25:38.338755 28651 raft_consensus.cc:2749] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:25:38.279366 28702 consensus_queue.cc:1048] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 18, Time since last communication: 0.093s
I20260709 12:25:38.371992 28503 raft_consensus.cc:2804] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:25:38.368583 28303 raft_consensus.cc:3060] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:38.368374 28636 leader_election.cc:290] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 30 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:38.400440 28557 raft_consensus.cc:493] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:38.401787 28814 raft_consensus.cc:493] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:25:38.447455 18364 raft_consensus.cc:2468] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 29.
I20260709 12:25:38.341958 28126 raft_consensus.cc:515] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:38.367547 28507 raft_consensus.cc:515] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:38.368243 28753 raft_consensus.cc:515] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:38.450949 28814 raft_consensus.cc:515] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:38.486858 28792 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.646s	user 0.014s	sys 0.042s
W20260709 12:25:38.492508 28792 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.652s	user 0.014s	sys 0.042s
I20260709 12:25:38.494875 28814 leader_election.cc:290] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:25:38.497174 28790 leader_election.cc:290] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:25:38.545001 17727 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:38.552731 17756 raft_consensus.cc:1240] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 27. Current term is 28. Ops: []
I20260709 12:25:38.557782 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:25:38.558519 17728 leader_election.cc:336] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 36 election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:38.559365 17756 raft_consensus.cc:2468] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 26.
I20260709 12:25:38.561462 17729 leader_election.cc:304] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:25:38.563472 17652 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:38.565287 17758 raft_consensus.cc:1275] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 34 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 32 index: 19. Preceding OpId from leader: term: 34 index: 20. (index mismatch)
I20260709 12:25:38.565806 17652 leader_election.cc:304] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:38.566640 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:38.563406 17757 raft_consensus.cc:1240] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 24. Current term is 25. Ops: []
I20260709 12:25:38.569092 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 27 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:38.569774 17757 raft_consensus.cc:2468] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 26.
I20260709 12:25:38.567701 17758 raft_consensus.cc:2468] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 29.
W20260709 12:25:38.572304 17728 leader_election.cc:336] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 26 election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:38.573305 17728 leader_election.cc:304] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:38.561961 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 31 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:25:38.574865 17760 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.575s	user 0.001s	sys 0.017s
I20260709 12:25:38.575343 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 37 candidate_status { last_received { term: 36 index: 27 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:25:38.575500 17760 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.576s	user 0.002s	sys 0.017s
W20260709 12:25:38.576030 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 205816us
W20260709 12:25:38.580400 17728 leader_election.cc:336] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 35 election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:25:38.576227 17763 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.501s	user 0.010s	sys 0.008s
I20260709 12:25:38.581764 17801 leader_election.cc:304] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:25:38.582688 17763 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.511s	user 0.010s	sys 0.008s
I20260709 12:25:38.575798 17756 raft_consensus.cc:2468] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 30.
I20260709 12:25:38.586046 17801 leader_election.cc:304] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:25:38.590220 17652 leader_election.cc:304] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:38.581279 17728 leader_election.cc:304] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 35 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:38.563874 17729 leader_election.cc:304] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:25:38.598338 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 321608us
W20260709 12:25:38.602814 17728 leader_election.cc:336] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 35 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:38.614339 17728 leader_election.cc:304] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:38.597466 17652 leader_election.cc:304] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:25:38.614694 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 358946us
I20260709 12:25:38.624068 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:38.633236 18374 raft_consensus.cc:1240] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 28. Current term is 29. Ops: [26.18-28.19]
I20260709 12:25:38.633656 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 35 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:38.634506 18361 raft_consensus.cc:3060] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:25:38.634742 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 30 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:38.624114 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:38.637918 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 32 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:38.639463 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 32 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:38.635831 18374 raft_consensus.cc:2393] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 30: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:25:38.640610 17729 leader_election.cc:304] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:38.641943 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:38.642843 18374 raft_consensus.cc:2468] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 28.
I20260709 12:25:38.642911 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 32 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:38.643136 17756 raft_consensus.cc:1275] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 34 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 33 index: 21. Preceding OpId from leader: term: 34 index: 22. (index mismatch)
I20260709 12:25:38.643179 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:38.643981 17759 raft_consensus.cc:2468] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 31.
I20260709 12:25:38.644300 17763 raft_consensus.cc:2468] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 27.
W20260709 12:25:38.619788 18334 leader_election.cc:336] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 27 election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:38.625221 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 30 candidate_status { last_received { term: 28 index: 22 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:38.629937 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 30 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:38.645838 18363 raft_consensus.cc:2393] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 30: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:25:38.646067 18368 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 29.
I20260709 12:25:38.646858 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 29 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:38.644789 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 32 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:38.647163 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 33 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:38.648121 18368 raft_consensus.cc:3060] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:25:38.648314 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 30 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:38.626041 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 35 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:38.649170 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:38.649749 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 32 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:38.650146 18363 raft_consensus.cc:2393] T a5732e71b9d54f8786b8c511d69943bf P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 31: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:25:38.650868 18364 raft_consensus.cc:2410] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for term 32 because replica has last-logged OpId of term: 29 index: 20, which is greater than that of the candidate, which has last-logged OpId of term: 28 index: 19.
I20260709 12:25:38.629112 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 36 candidate_status { last_received { term: 34 index: 24 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:38.652667 18365 raft_consensus.cc:2468] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 35.
I20260709 12:25:38.643975 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 35 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:38.592362 17801 leader_election.cc:304] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:25:38.624354 17728 leader_election.cc:336] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 36 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:25:38.646792 18334 leader_election.cc:336] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 30 election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:38.445758 28503 raft_consensus.cc:697] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 30 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:25:38.710007 28725 raft_consensus.cc:2749] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:25:38.712000 17728 leader_election.cc:304] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:38.710940 28757 raft_consensus.cc:2804] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:25:38.710845 18333 leader_election.cc:304] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:25:38.707423 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 54107us
I20260709 12:25:38.726274 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 30 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:38.713826 28757 raft_consensus.cc:493] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:38.734724 28754 raft_consensus.cc:2804] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:25:38.766775 28754 raft_consensus.cc:493] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:25:38.767237 28754 raft_consensus.cc:3060] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:38.734804 18361 raft_consensus.cc:2468] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 35.
I20260709 12:25:38.770805 28753 leader_election.cc:290] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:25:38.772172 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 61605.6us
I20260709 12:25:38.772095 28638 consensus_queue.cc:1059] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 24.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.040s
I20260709 12:25:38.774281 28814 raft_consensus.cc:3055] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:25:38.774750 28814 raft_consensus.cc:740] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
W20260709 12:25:38.774852 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 128173us
W20260709 12:25:38.776223 18334 leader_election.cc:336] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 29 election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:25:38.778244 18334 leader_election.cc:336] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:38.778141 28148 consensus_queue.cc:1048] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.001s
I20260709 12:25:38.736096 28320 raft_consensus.cc:2749] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:25:38.739230 28790 raft_consensus.cc:2749] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 35 FOLLOWER]: Leader election lost for term 35. Reason: could not achieve majority
I20260709 12:25:38.739740 28821 raft_consensus.cc:2804] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:25:38.448770 28557 raft_consensus.cc:515] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:38.740094 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 147994us
I20260709 12:25:38.740342 28561 raft_consensus.cc:493] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:38.740947 28794 raft_consensus.cc:2804] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:25:38.741050 28526 consensus_queue.cc:1048] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.024s
I20260709 12:25:38.742283 28495 leader_election.cc:290] T a5732e71b9d54f8786b8c511d69943bf P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:38.743098 28636 raft_consensus.cc:2749] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:25:38.819335 28636 raft_consensus.cc:493] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:25:38.819016 28503 consensus_queue.cc:237] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 28.22, Last appended by leader: 22, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:38.820144 28636 raft_consensus.cc:515] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:38.822697 28636 leader_election.cc:290] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:25:38.744611 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 120352us
I20260709 12:25:38.776261 28814 consensus_queue.cc:260] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 24.17, Last appended by leader: 17, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:38.824863 17728 leader_election.cc:336] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 34 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:38.825208 17801 leader_election.cc:304] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:38.825549 17728 leader_election.cc:304] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
W20260709 12:25:38.827440 17728 leader_election.cc:336] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:38.828292 28561 raft_consensus.cc:515] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:38.831063 28561 leader_election.cc:290] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:38.831450 28751 raft_consensus.cc:2749] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 34 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:25:38.832357 17728 leader_election.cc:304] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
W20260709 12:25:38.833619 28726 consensus_meta.cc:287] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.952s	user 0.009s	sys 0.004s
I20260709 12:25:38.745108 28507 leader_election.cc:290] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 24 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:38.703413 28644 raft_consensus.cc:493] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:38.834908 28751 raft_consensus.cc:2749] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 32 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:25:38.835601 28644 raft_consensus.cc:515] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:38.836530 28636 raft_consensus.cc:2804] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:25:38.837071 28636 raft_consensus.cc:493] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:38.837538 28636 raft_consensus.cc:3060] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:38.837340 28309 consensus_queue.cc:1048] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.106s
I20260709 12:25:38.838229 28644 leader_election.cc:290] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:38.734499 17761 raft_consensus.cc:1240] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 23. Current term is 24. Ops: []
I20260709 12:25:38.754024 28677 raft_consensus.cc:493] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:38.839684 28677 raft_consensus.cc:515] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:38.842829 28677 leader_election.cc:290] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:38.754841 18365 raft_consensus.cc:1275] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 33 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 31 index: 19. Preceding OpId from leader: term: 33 index: 20. (index mismatch)
I20260709 12:25:38.755478 28731 raft_consensus.cc:493] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:38.757974 17764 raft_consensus.cc:2468] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 27.
I20260709 12:25:38.760020 28618 consensus_queue.cc:1059] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 26.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.032s
I20260709 12:25:38.760933 28702 raft_consensus.cc:2804] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Leader pre-election won for term 27
W20260709 12:25:38.765550 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 48029.8us
I20260709 12:25:38.770670 28425 raft_consensus.cc:2749] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:25:38.737012 28778 consensus_queue.cc:1059] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 27.18, Next index: 19, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:25:38.792186 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 32 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:38.779501 18334 leader_election.cc:304] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:38.798961 28741 raft_consensus.cc:493] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:25:38.799178 28794 raft_consensus.cc:697] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:25:38.775399 18333 leader_election.cc:304] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:38.799811 28728 raft_consensus.cc:2804] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:25:38.801030 28772 leader_election.cc:290] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 30 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:38.801146 28822 consensus_queue.cc:1048] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.001s
I20260709 12:25:38.737021 28769 raft_consensus.cc:2804] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:25:38.803442 28792 raft_consensus.cc:2749] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
W20260709 12:25:38.803853 28501 consensus_meta.cc:287] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.899s	user 0.011s	sys 0.007s
I20260709 12:25:38.813705 28717 raft_consensus.cc:2749] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:25:38.797335 28821 raft_consensus.cc:493] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:38.834141 28726 raft_consensus.cc:515] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 35 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:38.850363 28702 raft_consensus.cc:493] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:38.781472 28757 raft_consensus.cc:3060] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:38.824863 28557 leader_election.cc:290] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:38.896942 17832 raft_consensus.cc:1275] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 28 index: 17. Preceding OpId from leader: term: 30 index: 18. (index mismatch)
I20260709 12:25:38.890981 18374 raft_consensus.cc:2468] T a5732e71b9d54f8786b8c511d69943bf P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 31.
I20260709 12:25:38.896031 18368 raft_consensus.cc:2468] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 33.
I20260709 12:25:38.878866 28753 raft_consensus.cc:2749] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:25:38.905043 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 33 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:38.825867 28814 raft_consensus.cc:3060] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Advancing to term 25
W20260709 12:25:38.907905 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 63182.7us
I20260709 12:25:38.905524 28526 raft_consensus.cc:493] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:38.879835 28728 raft_consensus.cc:493] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 34 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:25:38.923249 28811 raft_consensus.cc:493] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:38.923511 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 30 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:38.924095 28811 raft_consensus.cc:515] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:38.925670 17758 raft_consensus.cc:1275] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 33 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 31 index: 19. Preceding OpId from leader: term: 33 index: 20. (index mismatch)
I20260709 12:25:38.902897 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 31 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:38.897540 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:38.890393 28769 raft_consensus.cc:493] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:38.919044 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 36 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:38.921227 28495 raft_consensus.cc:2749] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:25:38.935137 17757 raft_consensus.cc:2468] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 30.
I20260709 12:25:38.934948 28526 raft_consensus.cc:515] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:38.938961 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 160816us
I20260709 12:25:38.903105 28702 raft_consensus.cc:3060] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:38.903549 28821 raft_consensus.cc:3060] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:38.928032 28126 leader_election.cc:290] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:38.962478 28509 consensus_queue.cc:1059] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 30.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.119s
I20260709 12:25:38.963317 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:38.963960 28789 consensus_queue.cc:1059] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 25.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.032s
I20260709 12:25:38.965282 28726 leader_election.cc:290] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 35 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:38.905751 28501 raft_consensus.cc:515] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:38.927742 28731 raft_consensus.cc:515] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:38.923580 28728 raft_consensus.cc:3060] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:25:38.940100 28638 raft_consensus.cc:3060] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:38.970623 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 31 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:38.980525 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:38.969841 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 27 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:38.933353 28811 leader_election.cc:290] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:38.942124 28769 raft_consensus.cc:3060] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:25:38.938071 28526 leader_election.cc:290] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:38.988266 17729 leader_election.cc:304] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:39.002272 17832 raft_consensus.cc:2468] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 30.
W20260709 12:25:39.002982 18334 leader_election.cc:336] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:39.017997 28731 leader_election.cc:290] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:39.023979 28495 raft_consensus.cc:493] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
W20260709 12:25:39.029377 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 182127us
I20260709 12:25:39.038285 28320 raft_consensus.cc:3055] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:25:39.002857 17831 raft_consensus.cc:2468] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 31.
I20260709 12:25:39.053500 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 35 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:39.054139 17756 raft_consensus.cc:3060] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:25:39.062994 17758 raft_consensus.cc:2393] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 25: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
W20260709 12:25:39.066974 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 145980us
I20260709 12:25:39.068487 17837 raft_consensus.cc:2468] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 27.
I20260709 12:25:39.052115 28320 raft_consensus.cc:740] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:25:39.065078 28822 raft_consensus.cc:493] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:39.073206 28495 raft_consensus.cc:515] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:39.078109 28822 raft_consensus.cc:515] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:39.079979 28637 consensus_meta.cc:287] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 1.027s	user 0.010s	sys 0.006s
I20260709 12:25:39.080101 28126 raft_consensus.cc:2804] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:25:39.080816 28126 raft_consensus.cc:493] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 35 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:39.081329 28126 raft_consensus.cc:3060] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 35 FOLLOWER]: Advancing to term 36
W20260709 12:25:39.083611 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 308397us
I20260709 12:25:39.084460 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 35 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:39.085443 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 32 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:39.087926 28805 raft_consensus.cc:493] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:39.088636 28839 raft_consensus.cc:493] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:39.089174 28839 raft_consensus.cc:515] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:39.080603 28637 raft_consensus.cc:2749] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 33 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:25:39.104763 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 36 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:39.111517 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:38.942219 28741 raft_consensus.cc:515] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:39.105325 28794 consensus_queue.cc:237] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 26.18, Last appended by leader: 18, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:39.140645 28693 raft_consensus.cc:493] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:25:39.073798 17654 leader_election.cc:304] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:25:39.131973 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 51925.6us
I20260709 12:25:39.143057 28126 raft_consensus.cc:515] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 36 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:39.167232 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:39.168175 17836 raft_consensus.cc:3060] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:25:39.172955 28805 raft_consensus.cc:515] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:39.174080 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:39.174347 28126 leader_election.cc:290] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 36 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:39.174825 18365 raft_consensus.cc:2468] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 27.
I20260709 12:25:39.175395 28805 leader_election.cc:290] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:39.175694 28693 raft_consensus.cc:515] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:25:39.176178 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 43717us
I20260709 12:25:39.178730 28772 consensus_queue.cc:1059] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 27.19, Next index: 20, Last known committed idx: 17, Time since last communication: 0.000s
I20260709 12:25:39.178943 28693 leader_election.cc:290] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:25:39.181396 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 132765us
I20260709 12:25:39.182410 17654 leader_election.cc:304] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:39.190371 28754 raft_consensus.cc:515] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:39.191403 28308 consensus_meta.cc:287] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 1.018s	user 0.001s	sys 0.017s
I20260709 12:25:39.192610 28320 consensus_queue.cc:260] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 27.18, Last appended by leader: 18, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:39.194248 28320 raft_consensus.cc:3060] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:39.196236 28741 leader_election.cc:290] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:25:39.198274 28731 raft_consensus.cc:2749] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
I20260709 12:25:39.198637 18374 raft_consensus.cc:1275] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 36 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 34 index: 23. Preceding OpId from leader: term: 36 index: 24. (index mismatch)
I20260709 12:25:39.199139 28731 raft_consensus.cc:3060] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:39.200322 28637 raft_consensus.cc:493] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:39.200774 28822 leader_election.cc:290] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:39.201154 28637 raft_consensus.cc:515] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:39.201751 28822 raft_consensus.cc:493] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:25:39.202576 28822 raft_consensus.cc:515] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:39.132534 28148 raft_consensus.cc:493] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:25:39.207248 28495 leader_election.cc:290] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:39.207429 28148 raft_consensus.cc:515] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:39.208730 28501 leader_election.cc:290] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 29 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:39.208938 28636 raft_consensus.cc:515] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:39.209806 28148 leader_election.cc:290] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:39.210589 17835 raft_consensus.cc:2368] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 29. Current term is 30.
W20260709 12:25:39.175704 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 172831us
I20260709 12:25:39.212122 17654 leader_election.cc:304] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:39.215535 17756 raft_consensus.cc:2468] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 35.
I20260709 12:25:39.211301 28636 leader_election.cc:290] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 30 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:39.179782 28772 raft_consensus.cc:3055] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:25:39.161033 28526 raft_consensus.cc:493] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
W20260709 12:25:39.229367 18334 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 117 similar messages]
W20260709 12:25:39.252828 28721 consensus_meta.cc:287] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.893s	user 0.009s	sys 0.010s
I20260709 12:25:39.253415 28721 raft_consensus.cc:515] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 32 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:39.254046 28702 raft_consensus.cc:515] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:25:39.256546 18334 leader_election.cc:336] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:39.257486 28561 raft_consensus.cc:493] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:39.257969 18334 leader_election.cc:336] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:39.258116 28561 raft_consensus.cc:515] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:39.258426 18334 leader_election.cc:304] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:39.260062 28725 raft_consensus.cc:2749] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:25:39.260150 28526 raft_consensus.cc:515] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:39.261940 28744 raft_consensus.cc:493] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:39.262471 28744 raft_consensus.cc:515] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:39.263127 28526 leader_election.cc:290] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:39.263831 28812 raft_consensus.cc:493] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:25:39.264614 28744 leader_election.cc:290] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:39.264509 28812 raft_consensus.cc:515] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:39.281665 17830 raft_consensus.cc:2393] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 27: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:25:39.291695 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:39.298496 28702 leader_election.cc:290] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 27 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:25:39.306293 28725 raft_consensus.cc:493] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:39.307219 28725 raft_consensus.cc:515] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:39.310209 28725 leader_election.cc:290] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:39.323814 28638 raft_consensus.cc:2749] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
W20260709 12:25:39.347893 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:39.404697 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 131818us
I20260709 12:25:39.218719 28308 raft_consensus.cc:515] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:39.409037 18333 leader_election.cc:304] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:25:39.409291 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 151396us
I20260709 12:25:39.410491 18333 leader_election.cc:304] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:39.411441 28526 consensus_queue.cc:1048] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.001s
I20260709 12:25:39.412398 28561 leader_election.cc:290] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:39.413233 28526 raft_consensus.cc:2804] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:25:39.413308 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 30 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:39.413794 28526 raft_consensus.cc:493] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:25:39.413756 28649 consensus_queue.cc:1059] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 23.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:25:39.414377 28526 raft_consensus.cc:3060] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:39.414577 28682 raft_consensus.cc:3055] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:25:39.414975 28649 raft_consensus.cc:2804] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:25:39.431572 28649 raft_consensus.cc:493] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:39.284718 28721 leader_election.cc:290] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 32 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:39.432106 28649 raft_consensus.cc:3060] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:39.415753 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 36 candidate_status { last_received { term: 34 index: 24 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:39.433347 18374 raft_consensus.cc:3060] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:25:39.433920 17832 raft_consensus.cc:2393] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 30: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:25:39.434870 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:39.435490 17832 raft_consensus.cc:3060] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:39.416510 17829 raft_consensus.cc:2393] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 30: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:25:39.437458 28812 leader_election.cc:290] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:39.290139 28805 raft_consensus.cc:2749] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:25:39.438397 17829 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35042 (request call id 3761) took 1207 ms. Trace:
I20260709 12:25:39.438827 17829 rpcz_store.cc:276] 0709 12:25:38.230746 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:38.230941 (+   195us) service_pool.cc:224] Handling call
0709 12:25:39.438150 (+1207209us) spinlock_profiling.cc:243] Waited 1.12 s on lock 0x7b540022ae54. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:25:39.438315 (+   165us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2576913792}
I20260709 12:25:39.420333 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 35 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:25:39.440450 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 605790us
I20260709 12:25:39.440732 17760 raft_consensus.cc:2468] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 34.
I20260709 12:25:39.440765 28637 leader_election.cc:290] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:25:39.441471 28303 consensus_meta.cc:287] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.996s	user 0.010s	sys 0.002s
I20260709 12:25:39.426580 18367 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59886 (request call id 2905) took 1170 ms. Trace:
I20260709 12:25:39.415050 28682 raft_consensus.cc:740] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 30 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:25:39.429977 17759 raft_consensus.cc:2468] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 35.
I20260709 12:25:39.443521 28682 consensus_queue.cc:260] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 30.18, Last appended by leader: 18, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:39.444733 28682 raft_consensus.cc:3060] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:39.442315 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 37 candidate_status { last_received { term: 35 index: 24 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:39.442087 17831 raft_consensus.cc:2468] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 28.
I20260709 12:25:39.242353 28772 raft_consensus.cc:740] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 29 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:25:39.464887 18374 raft_consensus.cc:2468] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 36.
I20260709 12:25:39.417490 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 37 candidate_status { last_received { term: 34 index: 23 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:39.411576 28308 leader_election.cc:290] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 30 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:39.243387 28503 raft_consensus.cc:493] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
W20260709 12:25:39.472604 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 67708.6us
I20260709 12:25:39.474869 17833 raft_consensus.cc:2410] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 36 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for term 37 because replica has last-logged OpId of term: 35 index: 24, which is greater than that of the candidate, which has last-logged OpId of term: 34 index: 23.
W20260709 12:25:39.479065 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:39.478878 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 36 candidate_status { last_received { term: 34 index: 24 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:39.442201 18367 rpcz_store.cc:276] 0709 12:25:38.255775 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:38.276111 (+ 20336us) service_pool.cc:224] Handling call
0709 12:25:38.278948 (+  2837us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:38.625682 (+346734us) raft_consensus.cc:1534] Early marking committed up to index 22
0709 12:25:38.724180 (+ 98498us) log.cc:844] Serialized 14 byte log entry
0709 12:25:38.929928 (+205748us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:38.930057 (+   129us) raft_consensus.cc:1645] Marking committed up to 22
0709 12:25:38.930181 (+   124us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:39.426026 (+495845us) spinlock_profiling.cc:243] Waited 346 ms on lock 0x7b54006fe354. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:25:39.426079 (+    53us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:39.426385 (+   306us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":6014,"prepare.queue_time_us":359744,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":45,"raft.queue_time_us":37479,"raft.run_cpu_time_us":129,"raft.run_wall_time_us":129,"spinlock_wait_cycles":795591936,"thread_start_us":200614,"threads_started":2}
I20260709 12:25:39.488613 28644 raft_consensus.cc:3055] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:25:39.500903 17836 raft_consensus.cc:2468] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 35.
I20260709 12:25:39.495118 17579 catalog_manager.cc:5697] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb reported cstate change: term changed from 18 to 20, leader changed from e063541542a94969898868bf6eb5ce1b (127.16.29.67) to e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66). New cstate: current_term: 20 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: HEALTHY } } }
I20260709 12:25:39.289826 28148 raft_consensus.cc:493] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:39.416395 28638 raft_consensus.cc:3055] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:25:39.506042 17729 leader_election.cc:304] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:39.514425 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 29 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:39.416996 28754 leader_election.cc:290] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 31 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:39.522651 17831 raft_consensus.cc:2393] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 29: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:25:39.530887 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 29 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:39.532011 17727 leader_election.cc:304] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:39.488693 28721 raft_consensus.cc:493] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:39.502374 28644 raft_consensus.cc:740] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
W20260709 12:25:39.567183 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 60553.7us
I20260709 12:25:39.578657 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 30 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:39.584045 28645 raft_consensus.cc:493] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:39.584306 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 47666.2us
I20260709 12:25:39.584851 28645 raft_consensus.cc:515] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:39.587417 17803 leader_election.cc:304] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:39.516968 28638 raft_consensus.cc:740] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:25:39.456269 28821 raft_consensus.cc:515] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:25:39.595500 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 411645us
I20260709 12:25:39.603049 17833 raft_consensus.cc:2393] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 30: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
W20260709 12:25:39.603861 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 71959.4us
I20260709 12:25:39.624322 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 32 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:39.629503 17831 raft_consensus.cc:2468] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 31.
I20260709 12:25:39.630693 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 33 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:39.621384 17652 leader_election.cc:304] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 35 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:25:39.590238 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 31 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:39.645659 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:39.646530 17760 raft_consensus.cc:3060] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:39.638342 17801 leader_election.cc:304] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:39.651605 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 32 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:39.647554 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 33 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:39.663479 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 34 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:39.587832 28792 raft_consensus.cc:2749] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 12:25:39.595348 28782 raft_consensus.cc:2804] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Leader pre-election won for term 28
W20260709 12:25:39.644253 28814 consensus_meta.cc:287] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.714s	user 0.015s	sys 0.000s
W20260709 12:25:39.678879 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 91553.2us
I20260709 12:25:39.640970 28699 consensus_queue.cc:1048] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.045s
I20260709 12:25:39.648876 28682 raft_consensus.cc:493] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:39.678902 18363 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59918 (request call id 3098) took 1058 ms. Trace:
W20260709 12:25:39.692272 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 117966us
I20260709 12:25:39.686017 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 32 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:39.658555 28818 raft_consensus.cc:493] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:39.687868 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 67927.4us
I20260709 12:25:39.641391 28320 raft_consensus.cc:493] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:25:39.587803 28645 leader_election.cc:290] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:25:39.607070 28503 raft_consensus.cc:515] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:39.665761 28728 consensus_meta.cc:287] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.612s	user 0.007s	sys 0.003s
I20260709 12:25:39.711650 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 30 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:39.685786 28706 consensus_queue.cc:1048] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.044s
W20260709 12:25:39.687261 28769 consensus_meta.cc:287] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.681s	user 0.012s	sys 0.000s
I20260709 12:25:39.677495 28812 raft_consensus.cc:2749] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
W20260709 12:25:39.721613 17652 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:39.580681 28148 raft_consensus.cc:515] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:39.730517 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 37 candidate_status { last_received { term: 34 index: 23 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:39.686931 28782 raft_consensus.cc:493] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:39.731053 17759 raft_consensus.cc:2468] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 32.
I20260709 12:25:39.754274 17652 leader_election.cc:304] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 35 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:39.762753 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 31 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:39.699767 18363 rpcz_store.cc:276] 0709 12:25:38.620762 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:38.650816 (+ 30054us) service_pool.cc:224] Handling call
0709 12:25:38.651234 (+   418us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:38.651951 (+   717us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:38.651984 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:38.653072 (+  1088us) log.cc:844] Serialized 26 byte log entry
0709 12:25:38.902279 (+249207us) raft_consensus.cc:1645] Marking committed up to 22
0709 12:25:38.902332 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:38.902405 (+    73us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:39.678641 (+776236us) raft_consensus.cc:1685] finished
0709 12:25:39.678667 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:39.678826 (+   159us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":53757,"raft.run_cpu_time_us":374,"raft.run_wall_time_us":40732,"thread_start_us":246765,"threads_started":1,"wal-append.queue_time_us":578874}
I20260709 12:25:39.774910 17835 raft_consensus.cc:1275] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 36 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 34 index: 20. Preceding OpId from leader: term: 36 index: 21. (index mismatch)
I20260709 12:25:39.775036 17803 leader_election.cc:304] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:25:39.760537 28769 raft_consensus.cc:515] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 32 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:39.779076 17838 raft_consensus.cc:1240] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 33 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 32. Current term is 33. Ops: []
I20260709 12:25:39.779527 28769 leader_election.cc:290] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 32 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:39.792589 17832 raft_consensus.cc:2468] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 24.
I20260709 12:25:39.794911 17652 leader_election.cc:304] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:39.799276 28706 raft_consensus.cc:493] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:39.803933 28749 consensus_meta.cc:287] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 1.421s	user 0.014s	sys 0.009s
I20260709 12:25:39.805542 28148 leader_election.cc:290] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:25:39.806151 28853 log.cc:927] Time spent T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.101s	user 0.000s	sys 0.000s
I20260709 12:25:39.751812 28509 raft_consensus.cc:2804] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 35 FOLLOWER]: Leader election won for term 35
I20260709 12:25:39.806910 28509 raft_consensus.cc:697] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 35 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:25:39.807495 28723 consensus_queue.cc:1048] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.038s
I20260709 12:25:39.808545 28509 consensus_queue.cc:237] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 33.19, Last appended by leader: 19, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:39.810714 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 184231us
I20260709 12:25:39.811803 17652 leader_election.cc:304] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:25:39.692906 28721 raft_consensus.cc:515] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:25:39.807137 28856 log.cc:927] Time spent T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.176s	user 0.001s	sys 0.000s
I20260709 12:25:39.813653 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 29 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:39.817828 17756 raft_consensus.cc:2410] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for term 29 because replica has last-logged OpId of term: 27 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 26 index: 17.
I20260709 12:25:39.762262 28308 raft_consensus.cc:2804] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:25:39.879626 28308 raft_consensus.cc:493] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:39.880297 28308 raft_consensus.cc:3060] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Advancing to term 31
W20260709 12:25:39.784588 28757 consensus_meta.cc:287] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.863s	user 0.011s	sys 0.003s
I20260709 12:25:39.816593 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 32 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
W20260709 12:25:39.824497 28838 log.cc:927] Time spent T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.165s	user 0.000s	sys 0.000s
I20260709 12:25:39.824584 28637 raft_consensus.cc:2804] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:25:39.825723 17755 raft_consensus.cc:2468] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 32.
I20260709 12:25:39.771423 28782 raft_consensus.cc:3060] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:39.799482 28789 raft_consensus.cc:2804] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 35 FOLLOWER]: Leader election won for term 35
I20260709 12:25:39.826774 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 34 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:39.828274 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 32 candidate_status { last_received { term: 30 index: 21 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
W20260709 12:25:39.832157 16501 thread.cc:527] Waited for 1000ms trying to join with heartbeat (tid 17714)
I20260709 12:25:39.834542 28651 raft_consensus.cc:2804] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:25:39.841156 28821 leader_election.cc:290] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 31 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:39.848165 17830 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35040 (request call id 3965) took 1056 ms. Trace:
W20260709 12:25:39.901371 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 453000us
I20260709 12:25:39.815094 28644 consensus_queue.cc:260] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 28.19, Last appended by leader: 19, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:39.794791 28682 raft_consensus.cc:515] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:39.901703 17830 rpcz_store.cc:276] 0709 12:25:38.848668 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:38.886798 (+ 38130us) service_pool.cc:224] Handling call
0709 12:25:39.847902 (+961104us) spinlock_profiling.cc:243] Waited 284 ms on lock 0x7b54002eb654. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:25:39.848087 (+   185us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":652809600}
W20260709 12:25:39.854458 28859 log.cc:927] Time spent T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.079s	user 0.000s	sys 0.003s
I20260709 12:25:39.678121 28638 consensus_queue.cc:260] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 23.16, Last appended by leader: 16, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:39.909159 18364 raft_consensus.cc:1275] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 25 index: 17. Preceding OpId from leader: term: 26 index: 18. (index mismatch)
I20260709 12:25:39.864260 17758 raft_consensus.cc:2410] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for term 30 because replica has last-logged OpId of term: 28 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 27 index: 14.
I20260709 12:25:39.742363 28728 raft_consensus.cc:515] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 35 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:39.770313 28320 raft_consensus.cc:515] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:39.866787 17579 catalog_manager.cc:5697] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb reported cstate change: term changed from 15 to 23, leader changed from 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) to e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66). New cstate: current_term: 23 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 12:25:39.871433 17585 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:37172 (request call id 261) took 35246 ms (client timeout 15000 ms). Trace:
I20260709 12:25:39.873250 28649 raft_consensus.cc:515] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:39.733441 28772 consensus_queue.cc:260] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 29.20, Last appended by leader: 20, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:39.890861 18367 raft_consensus.cc:3060] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:39.889287 17729 leader_election.cc:304] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 36 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:39.894110 17759 raft_consensus.cc:3060] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:25:39.803438 28706 raft_consensus.cc:515] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:39.903677 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 493288us
I20260709 12:25:39.896709 28651 raft_consensus.cc:493] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 35 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:39.890955 28637 raft_consensus.cc:493] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 34 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:39.917007 17838 raft_consensus.cc:1240] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 30. Current term is 31. Ops: []
I20260709 12:25:39.918293 17756 raft_consensus.cc:1240] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 30. Current term is 31. Ops: [30.21-30.21]
I20260709 12:25:39.920310 18368 raft_consensus.cc:3060] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:25:39.848202 28822 leader_election.cc:290] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:39.897351 28503 leader_election.cc:290] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:39.847456 28526 raft_consensus.cc:515] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:39.797298 28818 raft_consensus.cc:515] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:39.908553 28789 raft_consensus.cc:697] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 35 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:25:39.944458 17829 raft_consensus.cc:2468] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 35.
I20260709 12:25:39.893621 28757 raft_consensus.cc:515] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:39.938283 28812 raft_consensus.cc:2804] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:25:39.965629 17761 raft_consensus.cc:2468] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 33.
W20260709 12:25:39.970796 18336 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:39.943866 28651 raft_consensus.cc:3060] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:25:39.976727 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:39.947383 28637 raft_consensus.cc:3060] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:25:39.977327 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 31 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:39.973431 17579 catalog_manager.cc:5697] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb reported cstate change: term changed from 24 to 26. New cstate: current_term: 26 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } }
W20260709 12:25:39.985010 18334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:39.972295 17728 consensus_peers.cc:597] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:25:39.982650 18364 raft_consensus.cc:2468] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 25.
I20260709 12:25:39.974424 28812 raft_consensus.cc:493] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:40.002655 28812 raft_consensus.cc:3060] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:25:40.002933 28789 consensus_queue.cc:237] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 33.21, Last appended by leader: 21, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:39.973443 28721 leader_election.cc:290] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:40.004221 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 32 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:40.006527 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 31 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:40.007790 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 37 candidate_status { last_received { term: 35 index: 24 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:40.008031 17762 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36572 (request call id 1959) took 1731 ms. Trace:
I20260709 12:25:40.008870 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:40.009519 18364 raft_consensus.cc:3060] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Advancing to term 30
I20260709 12:25:40.009769 28839 leader_election.cc:290] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:40.010075 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 29 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:25:40.010706 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 241322us
W20260709 12:25:40.011505 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 200088us
I20260709 12:25:40.012367 17654 leader_election.cc:304] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:40.013258 28721 raft_consensus.cc:2749] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:25:40.013733 17654 leader_election.cc:304] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:40.014914 28721 raft_consensus.cc:3060] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:40.014662 28637 raft_consensus.cc:515] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 35 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:40.015980 28675 raft_consensus.cc:2804] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 36 FOLLOWER]: Leader election won for term 36
W20260709 12:25:40.016635 18371 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.048s	user 0.002s	sys 0.013s
W20260709 12:25:40.017038 18371 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.170s	user 0.003s	sys 0.013s
I20260709 12:25:40.017707 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 31 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:40.017767 18371 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59886 (request call id 2909) took 1401 ms. Trace:
I20260709 12:25:40.019348 28682 leader_election.cc:290] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:40.020040 28320 leader_election.cc:290] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:25:40.020219 17728 leader_election.cc:336] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 37 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:40.020892 17728 leader_election.cc:304] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
W20260709 12:25:40.020337 17755 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36572 (request call id 1963) took 1726 ms (client timeout 1691 ms). Trace:
I20260709 12:25:39.995992 28645 raft_consensus.cc:493] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
W20260709 12:25:40.025271 17755 rpcz_store.cc:269] 0709 12:25:38.294133 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:38.297054 (+  2921us) service_pool.cc:224] Handling call
0709 12:25:40.020066 (+1723012us) spinlock_profiling.cc:243] Waited 1.51 s on lock 0x7b54002b5f54. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:25:40.020244 (+   178us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3483300608}
I20260709 12:25:40.024358 28847 raft_consensus.cc:493] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:40.026067 28645 raft_consensus.cc:515] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:40.026942 28847 raft_consensus.cc:515] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:40.028141 28749 raft_consensus.cc:2749] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:25:40.028275 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:25:40.029860 17728 leader_election.cc:336] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:40.032083 18374 raft_consensus.cc:2468] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 30.
I20260709 12:25:40.018400 18371 rpcz_store.cc:276] 0709 12:25:38.616183 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:38.629365 (+ 13182us) service_pool.cc:224] Handling call
0709 12:25:38.630060 (+   695us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:38.630700 (+   640us) raft_consensus.cc:1534] Early marking committed up to index 20
0709 12:25:38.727801 (+ 97101us) log.cc:844] Serialized 14 byte log entry
0709 12:25:40.017458 (+1289657us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:40.017521 (+    63us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:25:40.017560 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:40.017617 (+    57us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:40.017719 (+   102us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":359626,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":38,"raft.queue_time_us":1476,"raft.run_cpu_time_us":80,"raft.run_wall_time_us":80,"spinlock_wait_cycles":11520,"thread_start_us":1314939,"threads_started":2}
I20260709 12:25:40.033469 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 29 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:40.033651 28728 leader_election.cc:290] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 35 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:40.025298 17579 catalog_manager.cc:5697] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb reported cstate change: term changed from 21 to 23. New cstate: current_term: 23 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:25:40.011804 28644 raft_consensus.cc:3060] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:40.016570 28675 raft_consensus.cc:697] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 36 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:25:40.013823 28818 leader_election.cc:290] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:40.034041 18371 raft_consensus.cc:3060] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:40.036147 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 29 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:40.048065 18364 raft_consensus.cc:2468] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 30.
I20260709 12:25:40.011546 17803 leader_election.cc:304] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:40.051045 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:40.036293 18362 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59902 (request call id 2541) took 1326 ms. Trace:
I20260709 12:25:40.051082 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:40.038825 17579 catalog_manager.cc:5697] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb reported cstate change: term changed from 24 to 26. New cstate: current_term: 26 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:25:40.083431 17759 raft_consensus.cc:2468] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 32.
I20260709 12:25:40.082919 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 36 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:40.085521 17760 raft_consensus.cc:2468] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 30.
I20260709 12:25:40.090983 28603 raft_consensus.cc:493] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:40.091651 28603 raft_consensus.cc:515] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:40.091403 17579 catalog_manager.cc:5697] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb reported cstate change: term changed from 18 to 20, leader changed from e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66) to e063541542a94969898868bf6eb5ce1b (127.16.29.67). New cstate: current_term: 20 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:25:40.093456 28740 raft_consensus.cc:493] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:25:40.094149 28740 raft_consensus.cc:515] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:40.039387 28706 leader_election.cc:290] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:40.096666 28637 leader_election.cc:290] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 35 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:40.098881 28637 raft_consensus.cc:493] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:40.099151 28740 leader_election.cc:290] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:40.099489 28637 raft_consensus.cc:515] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:40.101713 28637 leader_election.cc:290] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:40.101365 17579 catalog_manager.cc:5697] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb reported cstate change: term changed from 24 to 25, leader changed from e063541542a94969898868bf6eb5ce1b (127.16.29.67) to 0da169ff82674dc1947055ce43de68a3 (127.16.29.65). New cstate: current_term: 25 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } }
I20260709 12:25:40.054116 18362 rpcz_store.cc:276] 0709 12:25:38.710227 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:38.852973 (+142746us) service_pool.cc:224] Handling call
0709 12:25:38.884620 (+ 31647us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:38.971333 (+ 86713us) raft_consensus.cc:1534] Early marking committed up to index 18
0709 12:25:38.971368 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:38.971897 (+   529us) log.cc:844] Serialized 26 byte log entry
0709 12:25:39.047576 (+ 75679us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:25:39.047627 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:39.047721 (+    94us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:40.035957 (+988236us) raft_consensus.cc:1685] finished
0709 12:25:40.035979 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:40.036237 (+   258us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":971,"raft.queue_time_us":27614,"raft.run_cpu_time_us":179,"raft.run_wall_time_us":179,"thread_start_us":75095,"threads_started":1,"wal-append.queue_time_us":1050036}
I20260709 12:25:40.107853 17833 raft_consensus.cc:1240] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 23. Current term is 24. Ops: []
W20260709 12:25:40.108806 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 93107us
I20260709 12:25:39.978251 28790 consensus_queue.cc:1048] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.105s
I20260709 12:25:40.008955 28649 leader_election.cc:290] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 28 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:40.037417 28526 leader_election.cc:290] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 28 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:25:40.114807 17834 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35040 (request call id 3972) took 1140 ms. Trace:
I20260709 12:25:40.035748 28757 leader_election.cc:290] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 27 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:40.124029 28728 raft_consensus.cc:2804] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 33 FOLLOWER]: Leader election won for term 33
I20260709 12:25:40.124838 28675 consensus_queue.cc:237] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 34.24, Last appended by leader: 24, Current term: 36, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:40.128028 28638 raft_consensus.cc:3060] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:25:40.128873 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 117410us
I20260709 12:25:40.129460 28651 raft_consensus.cc:515] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 36 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:40.131925 28772 raft_consensus.cc:3060] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:40.008445 17762 rpcz_store.cc:276] 0709 12:25:38.251210 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:38.252982 (+  1772us) service_pool.cc:224] Handling call
0709 12:25:38.253405 (+   423us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:25:38.319391 (+ 65986us) raft_consensus.cc:1534] Early marking committed up to index 17
0709 12:25:38.319422 (+    31us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:25:38.578515 (+259093us) spinlock_profiling.cc:243] Waited 20.5 ms on lock 0x7b2c0000e5c8. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b48f19a11 00007f7b48f29372 00007f7b4a65ba01 00007f7b4a62043a 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2
0709 12:25:38.584138 (+  5623us) log.cc:844] Serialized 52 byte log entry
0709 12:25:38.716626 (+132488us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:25:38.716667 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:38.723136 (+  6469us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:39.965619 (+1242483us) raft_consensus.cc:1685] finished
0709 12:25:39.965638 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:39.982087 (+ 16449us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":65566,"raft.queue_time_us":77867,"raft.run_cpu_time_us":141,"raft.run_wall_time_us":8027,"spinlock_wait_cycles":47193984,"thread_start_us":126942,"threads_started":1,"wal-append.queue_time_us":928703}
I20260709 12:25:40.146369 17834 rpcz_store.cc:276] 0709 12:25:38.974127 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:39.086689 (+112562us) service_pool.cc:224] Handling call
0709 12:25:39.087151 (+   462us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:39.237571 (+150420us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:39.237622 (+    51us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:39.252770 (+ 15148us) log.cc:844] Serialized 26 byte log entry
0709 12:25:39.498245 (+245475us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:25:39.498625 (+   380us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:39.498830 (+   205us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:39.853654 (+354824us) spinlock_profiling.cc:243] Waited 11.9 ms on lock 0x7b4400022b4c. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b47459c85 00007f7b4745a6a2 00007f7b4a1cc1db 00007f7b4a1c7aaf 00007f7b4a1bb86e 00007f7b4a1babb6 00007f7b4a1ba7d8 00007f7b4a1c4221 00007f7b4a1c41d9 00007f7b4a1c4169
0709 12:25:40.104233 (+250579us) raft_consensus.cc:1685] finished
0709 12:25:40.104259 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:40.114731 (+ 10472us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":42644,"raft.queue_time_us":33430,"raft.run_cpu_time_us":116,"raft.run_wall_time_us":4903,"spinlock_wait_cycles":29312256,"thread_start_us":211122,"threads_started":1,"wal-append.queue_time_us":454004}
I20260709 12:25:40.173022 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 27 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
W20260709 12:25:40.181424 17728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:40.139868 28812 raft_consensus.cc:515] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 32 FOLLOWER]: Starting leader election with config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:40.149478 18368 raft_consensus.cc:2468] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 32.
I20260709 12:25:40.154892 18367 raft_consensus.cc:2468] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 31.
W20260709 12:25:40.166200 17802 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.530s	user 0.027s	sys 0.017s
I20260709 12:25:40.188972 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 35 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:25:40.189334 17802 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.553s	user 0.027s	sys 0.017s
W20260709 12:25:40.189630 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 567738us
I20260709 12:25:40.038921 28847 leader_election.cc:290] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:25:40.173933 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 30 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:25:40.163558 28794 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.895s	user 0.103s	sys 0.281s
I20260709 12:25:40.196683 18362 raft_consensus.cc:1240] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 30. Current term is 31. Ops: []
I20260709 12:25:40.198051 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 29 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:40.190387 17838 raft_consensus.cc:2393] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 27: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:25:40.193769 17836 raft_consensus.cc:2393] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 30: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:25:40.194305 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 32 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:40.194391 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 35 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
W20260709 12:25:40.199787 17802 leader_election.cc:336] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:40.205646 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 29 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:40.206288 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 31 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:40.206759 18368 raft_consensus.cc:2410] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for term 29 because replica has last-logged OpId of term: 27 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 26 index: 17.
I20260709 12:25:40.208124 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 33 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:40.209077 17760 raft_consensus.cc:3060] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:40.180382 28728 raft_consensus.cc:697] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 33 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
W20260709 12:25:40.200367 28794 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.926s	user 0.104s	sys 0.281s
I20260709 12:25:40.213039 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 31 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:40.191684 28303 raft_consensus.cc:493] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:40.226320 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 30 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:40.228895 18367 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 29.
I20260709 12:25:40.203727 28603 leader_election.cc:290] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:25:40.230437 17802 leader_election.cc:336] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 30 election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:40.202394 28721 raft_consensus.cc:2749] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:25:40.230942 17802 leader_election.cc:304] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
W20260709 12:25:40.233552 28848 log.cc:927] Time spent T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.252s	user 0.001s	sys 0.001s
I20260709 12:25:40.234637 28651 leader_election.cc:290] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 36 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:40.237694 28699 raft_consensus.cc:2749] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
I20260709 12:25:40.238481 28812 leader_election.cc:290] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 32 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:40.238857 28308 raft_consensus.cc:515] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:40.241305 28308 leader_election.cc:290] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 31 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:25:40.242216 17837 raft_consensus.cc:2393] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 31: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:25:40.242687 28303 raft_consensus.cc:515] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:40.243237 17727 leader_election.cc:304] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:25:40.245114 28303 leader_election.cc:290] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:25:40.245997 28731 consensus_meta.cc:287] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 1.046s	user 0.009s	sys 0.006s
I20260709 12:25:40.246435 28731 raft_consensus.cc:2749] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:25:40.248364 28645 leader_election.cc:290] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:25:39.914484 17585 rpcz_store.cc:269] 0709 12:25:04.625091 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:15.024557 (+10399466us) service_pool.cc:224] Handling call
0709 12:25:15.185521 (+160964us) spinlock_profiling.cc:243] Waited 45 ms on lock 0x7b540010d454. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a58a090 00007f7b4a628f3a 00007f7b503dee9c 00007f7b504dd04e 00007f7b4e99e2a4 00007f7b4e99e232 00007f7b4e99e161 00007f7b4e99e0dc 00007f7b4e99d392 00007f7b4895b48c 00007f7b4895a8c6
0709 12:25:39.871136 (+24685615us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:25:30.744249 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:25:31.059229 (+314980us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:25:31.095485 (+ 36256us) write_op.cc:435] Acquired schema lock
0709 12:25:31.095512 (+    27us) tablet.cc:662] Decoding operations
0709 12:25:31.668745 (+573233us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:25:31.668850 (+   105us) write_op.cc:641] Partition lock acquired for write op
0709 12:25:31.668879 (+    29us) tablet.cc:685] Acquiring locks for 16 operations
0709 12:25:31.670224 (+  1345us) tablet.cc:701] Row locks acquired
0709 12:25:31.670273 (+    49us) write_op.cc:260] PREPARE: finished
0709 12:25:31.670472 (+   199us) write_op.cc:270] Start()
0709 12:25:31.670646 (+   174us) write_op.cc:276] Timestamp: P: 1783599931670401 usec, L: 0
0709 12:25:31.670685 (+    39us) op_driver.cc:348] REPLICATION: starting
0709 12:25:31.671696 (+  1011us) log.cc:844] Serialized 6234 byte log entry
0709 12:25:33.047543 (+1375847us) spinlock_profiling.cc:243] Waited 29.8 ms on lock 0x7b4400022b4c. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b47459c85 00007f7b4745a6a2 00007f7b4a1cc1db 00007f7b4a1c7aaf 00007f7b4a1bb86e 00007f7b4a1babb6 00007f7b4a1ba7d8 00007f7b4a1c4221 00007f7b4a1c41d9 00007f7b4a1c4169
0709 12:25:33.392371 (+344828us) op_driver.cc:464] REPLICATION: finished
0709 12:25:33.879272 (+486901us) write_op.cc:301] APPLY: starting
0709 12:25:33.879398 (+   126us) tablet.cc:1367] starting BulkCheckPresence
0709 12:25:34.236635 (+357237us) tablet.cc:1370] finished BulkCheckPresence
0709 12:25:34.266911 (+ 30276us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:25:35.427072 (+1160161us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:25:35.475152 (+ 48080us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=32,key_file_lookups=32,delta_file_lookups=0,mrs_lookups=0
0709 12:25:35.489265 (+ 14113us) write_op.cc:312] APPLY: finished
0709 12:25:35.540737 (+ 51472us) log.cc:844] Serialized 148 byte log entry
0709 12:25:35.749333 (+208596us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:25:35.777675 (+ 28342us) write_op.cc:454] Released schema lock
0709 12:25:35.795951 (+ 18276us) write_op.cc:341] FINISH: Updating metrics
0709 12:25:35.841306 (+ 45355us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:25:35.844968 (+  3662us) write_op.cc:454] Released schema lock
Metrics: {"dns_us":265,"mutex_wait_us":19647,"reported_tablets":45,"spinlock_wait_cycles":103393664,"child_traces":[["op",{"apply.queue_time_us":444632,"apply.run_cpu_time_us":40936,"apply.run_wall_time_us":1946705,"cfile_cache_hit":66,"cfile_cache_hit_bytes":71071,"num_ops":16,"prepare.queue_time_us":436205,"prepare.run_cpu_time_us":37220,"prepare.run_wall_time_us":1410947,"raft.queue_time_us":265760,"raft.run_cpu_time_us":35949,"raft.run_wall_time_us":393295,"replication_time_us":1670790,"spinlock_wait_cycles":68549376,"thread_start_us":1246870,"threads_started":5,"wal-append.queue_time_us":608089,"wal-append.run_cpu_time_us":49383,"wal-append.run_wall_time_us":3356836}]]}
I20260709 12:25:40.253023 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:40.258169 17760 raft_consensus.cc:2468] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 31.
I20260709 12:25:40.262610 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 34 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:40.266777 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 35 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:40.266798 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 29 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:40.267478 18367 raft_consensus.cc:2468] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 28.
I20260709 12:25:40.253715 17834 raft_consensus.cc:2468] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 27.
I20260709 12:25:40.272897 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 32 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:40.275452 17829 raft_consensus.cc:1275] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 36 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 35 index: 24. Preceding OpId from leader: term: 36 index: 24. (term mismatch)
I20260709 12:25:40.239477 28782 raft_consensus.cc:515] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:40.280267 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:40.280334 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 36 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:40.276032 17829 pending_rounds.cc:85] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b: Aborting all ops after (but not including) 23
I20260709 12:25:40.264915 28320 raft_consensus.cc:2804] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:25:40.288185 17832 raft_consensus.cc:3060] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 34 FOLLOWER]: Advancing to term 36
I20260709 12:25:40.281569 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 36 candidate_status { last_received { term: 34 index: 24 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:40.289096 17833 raft_consensus.cc:3060] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:25:40.292186 28819 raft_consensus.cc:493] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:40.292526 28561 raft_consensus.cc:493] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:40.292814 28819 raft_consensus.cc:515] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:40.293104 28561 raft_consensus.cc:515] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:40.294890 28819 leader_election.cc:290] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:40.266228 28749 raft_consensus.cc:493] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:25:40.288545 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 36 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:40.294948 28561 leader_election.cc:290] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:25:40.295595 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 52507.9us
I20260709 12:25:40.286283 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 30 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:40.297843 17832 raft_consensus.cc:2468] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 36.
I20260709 12:25:40.281742 17829 pending_rounds.cc:107] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b: Aborting uncommitted NO_OP operation due to leader change: 35.24
I20260709 12:25:40.301468 17829 raft_consensus.cc:2889] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 36 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:40.302842 17756 raft_consensus.cc:2410] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for term 36 because replica has last-logged OpId of term: 35 index: 20, which is greater than that of the candidate, which has last-logged OpId of term: 31 index: 18.
I20260709 12:25:40.303848 18362 raft_consensus.cc:2393] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 30: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:25:40.311332 17652 leader_election.cc:304] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 36 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:25:40.288755 28603 raft_consensus.cc:493] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:25:40.288513 28320 raft_consensus.cc:493] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:40.319025 17727 leader_election.cc:304] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:25:40.321617 17830 raft_consensus.cc:1275] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 20 index: 13. Preceding OpId from leader: term: 23 index: 15. (index mismatch)
I20260709 12:25:40.303318 28821 raft_consensus.cc:493] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:40.324362 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:40.324915 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 27 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:40.326676 17762 raft_consensus.cc:3060] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:40.248380 28728 consensus_queue.cc:237] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 31.22, Last appended by leader: 22, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:40.306569 28819 raft_consensus.cc:493] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:40.311733 28726 raft_consensus.cc:493] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:40.321494 28794 consensus_queue.cc:1059] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.007s
I20260709 12:25:40.334262 28665 raft_consensus.cc:493] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:40.333889 28819 raft_consensus.cc:515] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:40.334802 28665 raft_consensus.cc:515] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:40.332499 28682 consensus_queue.cc:1048] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.004s
I20260709 12:25:40.336939 28665 leader_election.cc:290] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:40.338608 28782 leader_election.cc:290] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 28 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:40.321892 28320 raft_consensus.cc:3060] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:40.341933 18371 raft_consensus.cc:2468] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 29.
I20260709 12:25:40.319429 28651 raft_consensus.cc:2804] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 36 FOLLOWER]: Leader election won for term 36
I20260709 12:25:40.342957 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 35 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:40.336875 28819 leader_election.cc:290] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:40.329273 28769 raft_consensus.cc:2804] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:25:40.310106 28749 raft_consensus.cc:515] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:40.349817 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 28 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:40.347589 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 25 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:40.317256 28706 consensus_queue.cc:1048] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.023s
W20260709 12:25:40.360697 17802 leader_election.cc:336] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:25:40.349361 28864 log.cc:927] Time spent T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.077s	user 0.001s	sys 0.001s
I20260709 12:25:40.325636 28603 raft_consensus.cc:515] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:40.365398 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 407441us
I20260709 12:25:40.352533 28651 raft_consensus.cc:697] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 36 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:25:40.368796 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 36 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:40.374768 17756 raft_consensus.cc:3060] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 35 FOLLOWER]: Advancing to term 36
W20260709 12:25:40.375921 28846 log.cc:927] Time spent T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.326s	user 0.003s	sys 0.000s
W20260709 12:25:40.377889 28854 log.cc:927] Time spent T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.109s	user 0.000s	sys 0.001s
I20260709 12:25:40.378015 18361 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59902 (request call id 2540) took 1668 ms. Trace:
I20260709 12:25:40.379062 17764 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38786 (request call id 4412) took 1434 ms. Trace:
I20260709 12:25:40.380160 28861 raft_consensus.cc:493] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:40.379453 17764 rpcz_store.cc:276] 0709 12:25:38.945110 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:38.998350 (+ 53240us) service_pool.cc:224] Handling call
0709 12:25:38.998790 (+   440us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:38.999534 (+   744us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:38.999570 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:39.000301 (+   731us) log.cc:844] Serialized 26 byte log entry
0709 12:25:39.166057 (+165756us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:25:39.166100 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:39.166172 (+    72us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:40.378873 (+1212701us) raft_consensus.cc:1685] finished
0709 12:25:40.378894 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:40.379007 (+   113us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":2015,"raft.run_cpu_time_us":179,"raft.run_wall_time_us":181,"thread_start_us":165261,"threads_started":1,"wal-append.queue_time_us":1193439}
I20260709 12:25:40.378479 18361 rpcz_store.cc:276] 0709 12:25:38.709837 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:38.768398 (+ 58561us) service_pool.cc:224] Handling call
0709 12:25:38.768793 (+   395us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:38.769155 (+   362us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:38.769181 (+    26us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:38.769721 (+   540us) log.cc:844] Serialized 26 byte log entry
0709 12:25:38.869932 (+100211us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:25:38.869997 (+    65us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:38.870097 (+   100us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:40.179339 (+1309242us) spinlock_profiling.cc:243] Waited 16.8 ms on lock 0x7b4400022b4c. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b47459c85 00007f7b4745a6a2 00007f7b4a1cc1db 00007f7b4a1c7aaf 00007f7b4a1bb86e 00007f7b4a1babb6 00007f7b4a1ba7d8 00007f7b4a1c4221 00007f7b4a1c41d9 00007f7b4a1c4169
0709 12:25:40.377692 (+198353us) raft_consensus.cc:1685] finished
0709 12:25:40.377716 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:40.377952 (+   236us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":38593408,"thread_start_us":99644,"threads_started":1,"wal-append.queue_time_us":1170923}
I20260709 12:25:40.365741 28782 raft_consensus.cc:3055] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 32 LEADER]: Stepping down as leader of term 32
I20260709 12:25:40.387063 28782 raft_consensus.cc:740] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 32 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:25:40.380823 28861 raft_consensus.cc:515] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:40.388046 28782 consensus_queue.cc:260] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 32.23, Last appended by leader: 23, Current term: 32, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:40.389271 28782 raft_consensus.cc:3060] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:25:40.390208 28861 leader_election.cc:290] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:40.394357 28749 leader_election.cc:290] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:40.346109 28821 raft_consensus.cc:515] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:40.397615 28821 leader_election.cc:290] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:40.398952 17729 leader_election.cc:304] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:40.399950 28790 raft_consensus.cc:2804] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 32 FOLLOWER]: Leader election won for term 32
I20260709 12:25:40.400511 28790 raft_consensus.cc:697] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 32 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:25:40.401527 28790 consensus_queue.cc:237] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 30.21, Last appended by leader: 21, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:40.402233 17802 leader_election.cc:336] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:25:40.402608 17728 leader_election.cc:336] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 31 election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:40.403084 17728 leader_election.cc:304] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 31 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:40.410895 17833 raft_consensus.cc:2468] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 36.
W20260709 12:25:40.403595 17802 leader_election.cc:336] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:40.416574 17762 raft_consensus.cc:2468] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 27.
I20260709 12:25:40.417455 17756 raft_consensus.cc:2468] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 36.
I20260709 12:25:40.412590 17729 leader_election.cc:304] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:40.353322 28726 raft_consensus.cc:515] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:25:40.417296 17802 leader_election.cc:336] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 33 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:40.404178 28794 raft_consensus.cc:2749] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Leader election lost for term 31. Reason: could not achieve majority
I20260709 12:25:40.423020 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 30 candidate_status { last_received { term: 28 index: 22 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:40.427407 18371 raft_consensus.cc:1240] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 27. Current term is 28. Ops: []
I20260709 12:25:40.428346 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 37 candidate_status { last_received { term: 35 index: 26 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:25:40.429548 28870 log.cc:927] Time spent T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.072s	user 0.000s	sys 0.000s
W20260709 12:25:40.430560 28868 log.cc:927] Time spent T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.100s	user 0.000s	sys 0.001s
W20260709 12:25:40.438966 17802 leader_election.cc:336] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 29 election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:40.441673 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 37 candidate_status { last_received { term: 35 index: 26 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:40.441896 17801 leader_election.cc:304] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:40.440600 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:40.445236 17801 leader_election.cc:304] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:25:40.446374 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
W20260709 12:25:40.446400 18333 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:40.447155 17760 raft_consensus.cc:2393] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 27: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:25:40.452875 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 30 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:40.453500 17759 raft_consensus.cc:3060] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Advancing to term 30
W20260709 12:25:40.456554 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 44027.9us
W20260709 12:25:40.457357 17802 leader_election.cc:336] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:40.455267 17801 leader_election.cc:304] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:40.464059 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:25:40.462960 17802 leader_election.cc:336] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 27 election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:40.460510 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 24 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:40.467526 17801 leader_election.cc:304] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:25:40.465160 17802 leader_election.cc:336] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 30 election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:40.465179 17652 leader_election.cc:304] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:40.471079 17757 raft_consensus.cc:2393] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 28: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:25:40.473441 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 35 candidate_status { last_received { term: 34 index: 24 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:40.473567 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 25 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:40.473946 17654 leader_election.cc:304] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:40.475024 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 28 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:40.452623 28839 raft_consensus.cc:2749] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:25:40.481376 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 34 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:40.481981 18374 raft_consensus.cc:2468] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 33.
I20260709 12:25:40.456362 28645 raft_consensus.cc:2804] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:25:40.483860 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 33 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:40.488438 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:40.487779 18364 raft_consensus.cc:1275] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 28 index: 20. Preceding OpId from leader: term: 30 index: 21. (index mismatch)
I20260709 12:25:40.485718 17652 leader_election.cc:304] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:25:40.486864 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 30 candidate_status { last_received { term: 28 index: 22 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:40.487004 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 32 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:40.489460 17764 raft_consensus.cc:3060] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:25:40.487438 18365 raft_consensus.cc:1275] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 35 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 31 index: 18. Preceding OpId from leader: term: 35 index: 20. (index mismatch)
I20260709 12:25:40.464983 28308 raft_consensus.cc:2804] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:25:40.492710 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 37 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:40.493866 17654 leader_election.cc:304] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:40.466566 28675 raft_consensus.cc:2804] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:25:40.498016 18362 raft_consensus.cc:1240] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 33 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 32. Current term is 33. Ops: []
I20260709 12:25:40.494822 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 31 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:40.498711 17757 raft_consensus.cc:2393] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 31: Already voted for candidate 0da169ff82674dc1947055ce43de68a3 in this term.
I20260709 12:25:40.502604 28637 raft_consensus.cc:2749] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:25:40.503763 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:40.504448 17755 raft_consensus.cc:2468] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 31.
I20260709 12:25:40.506141 28726 leader_election.cc:290] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:40.484108 28644 raft_consensus.cc:3060] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:40.480664 28847 raft_consensus.cc:2749] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 12:25:40.511359 28651 consensus_queue.cc:237] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 34.19, Last appended by leader: 19, Current term: 36, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:40.514307 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 32 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:40.480916 28822 raft_consensus.cc:493] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:40.516644 17652 leader_election.cc:304] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:40.519268 17652 leader_election.cc:304] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:25:40.516096 17801 leader_election.cc:304] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:40.487934 28645 raft_consensus.cc:493] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 32 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:25:40.523238 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:40.532027 18364 raft_consensus.cc:2393] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 28: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
W20260709 12:25:40.529336 17728 leader_election.cc:336] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:40.516115 28821 raft_consensus.cc:2749] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:25:40.545744 17652 leader_election.cc:304] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:40.522272 28682 raft_consensus.cc:2749] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:25:40.560287 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:40.537191 28561 raft_consensus.cc:2804] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Leader election won for term 31
W20260709 12:25:40.529531 28872 log.cc:927] Time spent T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.188s	user 0.000s	sys 0.001s
I20260709 12:25:40.491721 28665 consensus_queue.cc:1059] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 29.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.039s
W20260709 12:25:40.563947 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 70129.1us
I20260709 12:25:40.567255 17759 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 30.
I20260709 12:25:40.571800 17756 raft_consensus.cc:1240] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 35 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 34. Current term is 35. Ops: [34.22-34.22]
W20260709 12:25:40.573035 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 103240us
I20260709 12:25:40.522074 28731 raft_consensus.cc:2804] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 32 FOLLOWER]: Leader election won for term 32
W20260709 12:25:40.577862 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 253282us
I20260709 12:25:40.535564 28645 raft_consensus.cc:3060] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:25:40.586979 17652 leader_election.cc:304] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:40.533579 28675 raft_consensus.cc:697] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 31 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
W20260709 12:25:40.589777 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 60480.7us
I20260709 12:25:40.592100 17654 leader_election.cc:304] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:40.558367 28717 raft_consensus.cc:2804] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:25:40.598889 17729 leader_election.cc:304] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:25:40.593981 17802 leader_election.cc:336] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:40.608309 17654 leader_election.cc:304] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
W20260709 12:25:40.609201 17802 leader_election.cc:336] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:40.609643 17802 leader_election.cc:304] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:40.606354 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:40.522406 28822 raft_consensus.cc:515] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:40.617733 17652 leader_election.cc:304] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:25:40.614902 17758 raft_consensus.cc:1240] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 32 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 31. Current term is 32. Ops: []
I20260709 12:25:40.618474 18363 raft_consensus.cc:2393] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 28: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
W20260709 12:25:40.619745 17802 leader_election.cc:336] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:40.594250 28509 raft_consensus.cc:2749] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:25:40.587811 28840 raft_consensus.cc:493] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:25:40.593369 28561 raft_consensus.cc:697] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 31 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:25:40.604477 28603 leader_election.cc:290] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:40.607190 28717 raft_consensus.cc:493] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:40.595139 28772 consensus_queue.cc:1048] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.032s
W20260709 12:25:40.627418 17728 leader_election.cc:336] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:25:40.627300 17802 consensus_peers.cc:597] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:25:40.622763 28726 raft_consensus.cc:2749] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
W20260709 12:25:40.637966 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 44019.7us
I20260709 12:25:40.623754 28731 raft_consensus.cc:697] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 32 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:25:40.628010 28821 raft_consensus.cc:2749] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:25:40.631726 28717 raft_consensus.cc:3060] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Advancing to term 32
W20260709 12:25:40.645793 17728 consensus_peers.cc:597] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:25:40.646333 17728 leader_election.cc:336] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 29 election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:40.647156 28559 raft_consensus.cc:2749] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
W20260709 12:25:40.650640 28876 log.cc:927] Time spent T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.116s	user 0.000s	sys 0.001s
I20260709 12:25:40.651954 17831 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37994 (request call id 2058) took 1246 ms. Trace:
I20260709 12:25:40.633062 28840 raft_consensus.cc:515] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:40.652343 17831 rpcz_store.cc:276] 0709 12:25:39.405579 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:39.693079 (+287500us) service_pool.cc:224] Handling call
0709 12:25:39.771015 (+ 77936us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:39.947345 (+176330us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:39.947383 (+    38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:39.956926 (+  9543us) log.cc:844] Serialized 26 byte log entry
0709 12:25:40.365609 (+408683us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:25:40.366036 (+   427us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:40.366108 (+    72us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:40.651699 (+285591us) raft_consensus.cc:1685] finished
0709 12:25:40.651722 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:40.651871 (+   149us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":120169,"raft.queue_time_us":102673,"raft.run_cpu_time_us":180,"raft.run_wall_time_us":25063,"thread_start_us":330886,"threads_started":1,"wal-append.queue_time_us":480124}
W20260709 12:25:40.658294 17802 leader_election.cc:336] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:25:40.665124 17802 leader_election.cc:336] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:40.640143 28509 raft_consensus.cc:2804] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Leader election won for term 27
W20260709 12:25:40.666971 17728 leader_election.cc:336] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:40.665576 17802 leader_election.cc:304] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:40.667497 17728 leader_election.cc:304] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:40.633958 28637 consensus_queue.cc:1059] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.022s
I20260709 12:25:40.636035 28603 consensus_queue.cc:1048] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.023s
I20260709 12:25:40.668164 18333 leader_election.cc:304] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
W20260709 12:25:40.668947 17802 leader_election.cc:336] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 32 election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:40.669582 28699 raft_consensus.cc:2749] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:25:40.670153 28644 raft_consensus.cc:2749] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:25:40.672155 28509 raft_consensus.cc:697] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 27 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:25:40.696687 18363 raft_consensus.cc:1275] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 36 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 34 index: 24. Preceding OpId from leader: term: 36 index: 25. (index mismatch)
I20260709 12:25:40.673662 28706 raft_consensus.cc:2749] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:25:40.704947 28873 raft_consensus.cc:2804] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:25:40.705755 28769 raft_consensus.cc:697] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:25:40.707707 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 24 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:25:40.707701 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 256992us
W20260709 12:25:40.710146 18334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:40.702227 28503 raft_consensus.cc:2749] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:25:40.700335 28822 leader_election.cc:290] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:40.714778 18333 leader_election.cc:400] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 29 pre-election: Vote denied by peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837) with higher term. Message: Invalid argument: T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 29. Current term is 30.
I20260709 12:25:40.715265 18333 leader_election.cc:403] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 29 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:25:40.701059 28559 raft_consensus.cc:493] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:40.724572 17802 leader_election.cc:336] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 31 election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:40.724722 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:40.725358 17831 raft_consensus.cc:3060] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:40.707180 28772 raft_consensus.cc:2804] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:25:40.725169 17802 leader_election.cc:304] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 31 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:40.737155 28861 raft_consensus.cc:493] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:40.737716 28861 raft_consensus.cc:515] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:40.673231 28561 consensus_queue.cc:237] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 29.16, Last appended by leader: 16, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:25:40.743321 17802 leader_election.cc:336] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 37 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:40.711730 28308 raft_consensus.cc:697] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 30 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:25:40.744117 17802 leader_election.cc:304] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:40.736236 28840 leader_election.cc:290] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:40.673475 28675 consensus_queue.cc:237] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 29.20, Last appended by leader: 20, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:40.747282 28900 raft_consensus.cc:2749] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Leader pre-election lost for term 29. Reason: Vote denied by peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837) with higher term. Message: Invalid argument: T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 29. Current term is 30.
W20260709 12:25:40.739609 28879 log.cc:927] Time spent T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.135s	user 0.001s	sys 0.000s
I20260709 12:25:40.732218 28559 raft_consensus.cc:515] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:40.733606 28772 raft_consensus.cc:493] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:25:40.761559 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 142061us
I20260709 12:25:40.763456 17654 leader_election.cc:304] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 35 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:40.749264 28303 raft_consensus.cc:2749] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Leader election lost for term 31. Reason: could not achieve majority
I20260709 12:25:40.774912 17654 leader_election.cc:304] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:40.760978 28772 raft_consensus.cc:3060] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 32 FOLLOWER]: Advancing to term 33
W20260709 12:25:40.776100 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 158427us
I20260709 12:25:40.709514 28731 consensus_queue.cc:237] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 30.19, Last appended by leader: 19, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:40.778991 17652 leader_election.cc:304] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:40.780642 28821 raft_consensus.cc:3060] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:40.786053 17831 raft_consensus.cc:2468] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 28.
I20260709 12:25:40.786978 17652 leader_election.cc:304] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:40.787358 17727 leader_election.cc:304] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:40.788568 17764 raft_consensus.cc:2468] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 32.
I20260709 12:25:40.777624 28644 raft_consensus.cc:2749] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 35 FOLLOWER]: Leader election lost for term 35. Reason: could not achieve majority
W20260709 12:25:40.790372 28882 log.cc:927] Time spent T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.090s	user 0.000s	sys 0.001s
W20260709 12:25:40.793551 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 90578.9us
I20260709 12:25:40.784760 28651 raft_consensus.cc:493] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:40.791307 28875 log.cc:927] Time spent T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.309s	user 0.003s	sys 0.004s
I20260709 12:25:40.799939 18333 leader_election.cc:304] T a5732e71b9d54f8786b8c511d69943bf P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:40.806934 17759 raft_consensus.cc:1275] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 35 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 33 index: 21. Preceding OpId from leader: term: 35 index: 22. (index mismatch)
I20260709 12:25:40.758649 28769 consensus_queue.cc:237] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 23.16, Last appended by leader: 16, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:40.794068 28782 raft_consensus.cc:2804] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:25:40.808480 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 35 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:40.791783 28757 raft_consensus.cc:2749] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:25:40.809332 17729 leader_election.cc:304] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:40.812531 17830 raft_consensus.cc:1275] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 35 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 31 index: 18. Preceding OpId from leader: term: 35 index: 20. (index mismatch)
I20260709 12:25:40.811625 28861 leader_election.cc:290] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:40.811196 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 37 candidate_status { last_received { term: 35 index: 21 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:40.819576 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 37 candidate_status { last_received { term: 35 index: 21 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:40.820662 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 35 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:40.820955 17654 leader_election.cc:304] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:40.823522 17654 leader_election.cc:304] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:40.770814 28509 consensus_queue.cc:237] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 24.16, Last appended by leader: 16, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:40.792773 28721 consensus_queue.cc:1059] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 27.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.017s
I20260709 12:25:40.805099 28644 raft_consensus.cc:2749] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:25:40.807603 28725 raft_consensus.cc:3055] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:25:40.807718 28308 consensus_queue.cc:237] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 28.19, Last appended by leader: 19, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:40.835548 18333 leader_election.cc:304] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:40.824347 28754 raft_consensus.cc:2749] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:25:40.828464 28900 raft_consensus.cc:2804] T a5732e71b9d54f8786b8c511d69943bf P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Leader pre-election won for term 32
W20260709 12:25:40.853896 28886 log.cc:927] Time spent T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.091s	user 0.000s	sys 0.001s
I20260709 12:25:40.861152 17654 leader_election.cc:304] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:40.854204 28559 leader_election.cc:290] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:40.858690 18333 leader_election.cc:304] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:25:40.862493 28726 consensus_queue.cc:1048] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 18, Time since last communication: 0.000s
W20260709 12:25:40.863574 28888 log.cc:927] Time spent T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.070s	user 0.001s	sys 0.000s
I20260709 12:25:40.839685 28725 raft_consensus.cc:740] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 30 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:25:40.840549 28561 raft_consensus.cc:2749] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
W20260709 12:25:40.893932 16501 thread.cc:527] Waited for 2000ms trying to join with heartbeat (tid 17714)
I20260709 12:25:40.849726 28893 raft_consensus.cc:2749] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:25:40.859423 28900 raft_consensus.cc:493] T a5732e71b9d54f8786b8c511d69943bf P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:25:40.893855 18334 leader_election.cc:336] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:40.836740 28794 consensus_queue.cc:1048] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.051s
I20260709 12:25:40.842511 28757 consensus_queue.cc:1048] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.057s
I20260709 12:25:40.897200 28782 raft_consensus.cc:697] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:25:40.818802 28651 raft_consensus.cc:515] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:40.900547 28840 raft_consensus.cc:2749] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:25:40.900267 28861 raft_consensus.cc:2749] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
W20260709 12:25:40.913380 28889 log.cc:927] Time spent T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.072s	user 0.000s	sys 0.000s
I20260709 12:25:40.920048 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 29 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:40.925015 18363 raft_consensus.cc:2393] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 29: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:25:40.903728 28896 raft_consensus.cc:2749] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:25:40.933209 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 29 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:25:40.918699 28638 consensus_meta.cc:287] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.743s	user 0.008s	sys 0.004s
I20260709 12:25:40.900985 28644 consensus_queue.cc:1048] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 18, Time since last communication: 0.019s
I20260709 12:25:40.905889 28900 raft_consensus.cc:3060] T a5732e71b9d54f8786b8c511d69943bf P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:25:40.940557 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 29 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:40.954082 17831 raft_consensus.cc:2393] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 29: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:25:40.940562 17837 raft_consensus.cc:1275] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 36 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 34 index: 24. Preceding OpId from leader: term: 36 index: 25. (index mismatch)
I20260709 12:25:40.907248 28725 consensus_queue.cc:260] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 30.21, Last appended by leader: 21, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:40.914507 28794 raft_consensus.cc:493] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:40.906332 28906 raft_consensus.cc:2804] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:25:40.940971 18334 leader_election.cc:304] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
W20260709 12:25:40.959378 28899 log.cc:927] Time spent T 00000000000000000000000000000000 P 2193af31cb894ec9b3d65d939a2a6851: Append to log took a long time: real 0.053s	user 0.002s	sys 0.000s
I20260709 12:25:40.909265 28782 consensus_queue.cc:237] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 26.19, Last appended by leader: 19, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:40.978578 17652 leader_election.cc:304] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:40.980118 17652 leader_election.cc:304] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:40.964721 28861 raft_consensus.cc:2749] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:25:40.958195 28906 raft_consensus.cc:493] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 33 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:25:40.972116 28881 log.cc:927] Time spent T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.277s	user 0.003s	sys 0.000s
I20260709 12:25:40.967760 28561 raft_consensus.cc:493] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:40.985476 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 126897us
I20260709 12:25:40.987941 28819 raft_consensus.cc:2749] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:25:40.978765 28725 raft_consensus.cc:3060] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:40.964524 28794 raft_consensus.cc:515] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:41.009375 18361 raft_consensus.cc:3055] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 34 LEADER]: Stepping down as leader of term 34
I20260709 12:25:41.009799 18361 raft_consensus.cc:740] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 34 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:25:41.010512 18361 consensus_queue.cc:260] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 34.22, Last appended by leader: 22, Current term: 34, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:41.011477 18361 raft_consensus.cc:3060] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 34 FOLLOWER]: Advancing to term 35
W20260709 12:25:41.000694 28905 log.cc:927] Time spent T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.064s	user 0.000s	sys 0.001s
I20260709 12:25:40.985471 28906 raft_consensus.cc:3060] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:25:40.989854 28509 raft_consensus.cc:2749] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:25:41.024896 17756 raft_consensus.cc:2468] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 28.
I20260709 12:25:40.982869 28896 consensus_queue.cc:1048] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.034s
W20260709 12:25:41.020649 28897 log.cc:927] Time spent T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.170s	user 0.001s	sys 0.000s
I20260709 12:25:41.033612 17652 leader_election.cc:304] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:41.047233 28638 raft_consensus.cc:493] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:41.056320 18333 leader_election.cc:304] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:41.001960 28561 raft_consensus.cc:515] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:41.054616 28638 raft_consensus.cc:515] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:41.048292 28320 consensus_meta.cc:287] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.694s	user 0.007s	sys 0.006s
I20260709 12:25:41.071162 18362 raft_consensus.cc:1240] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 27. Current term is 28. Ops: []
I20260709 12:25:41.087121 18361 raft_consensus.cc:1275] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 35 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 34 index: 22. Preceding OpId from leader: term: 35 index: 22. (term mismatch)
I20260709 12:25:41.098002 18361 pending_rounds.cc:85] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da: Aborting all ops after (but not including) 21
I20260709 12:25:41.098327 18361 pending_rounds.cc:107] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da: Aborting uncommitted NO_OP operation due to leader change: 34.22
I20260709 12:25:41.098696 18361 raft_consensus.cc:2889] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 35 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:41.117973 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 32 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:25:41.118204 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 51158.1us
I20260709 12:25:41.118997 18361 raft_consensus.cc:2393] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 32: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:25:41.081712 28917 raft_consensus.cc:3060] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:41.124830 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:41.124538 28772 raft_consensus.cc:515] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 33 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:41.125541 28915 raft_consensus.cc:2749] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:25:41.126153 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 29 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:41.126267 28915 consensus_queue.cc:1059] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
I20260709 12:25:41.126820 28772 leader_election.cc:290] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 33 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:25:41.127244 28908 log.cc:927] Time spent T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.101s	user 0.001s	sys 0.000s
I20260709 12:25:41.128230 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 28 candidate_status { last_received { term: 27 index: 23 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:41.128804 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 29 candidate_status { last_received { term: 28 index: 22 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:25:41.129395 28912 log.cc:927] Time spent T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.142s	user 0.001s	sys 0.000s
I20260709 12:25:41.130311 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:41.131093 18367 raft_consensus.cc:2393] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 28: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:25:41.132565 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 35 candidate_status { last_received { term: 34 index: 24 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
W20260709 12:25:41.133036 28901 log.cc:927] Time spent T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.193s	user 0.001s	sys 0.000s
I20260709 12:25:41.134035 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 32 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:41.135241 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 32 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:41.135900 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:41.136476 18364 raft_consensus.cc:2468] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 25.
I20260709 12:25:41.136737 18374 raft_consensus.cc:1275] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 27 index: 21. Preceding OpId from leader: term: 30 index: 23. (index mismatch)
I20260709 12:25:41.137861 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 33 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:41.116006 28651 leader_election.cc:290] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:41.141222 18364 raft_consensus.cc:1275] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 33 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 31 index: 22. Preceding OpId from leader: term: 33 index: 23. (index mismatch)
I20260709 12:25:41.145879 28900 raft_consensus.cc:515] T a5732e71b9d54f8786b8c511d69943bf P 334a4e1f59944544b3b3ff36616b57da [term 32 FOLLOWER]: Starting leader election with config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:41.154007 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:41.154843 17763 raft_consensus.cc:3060] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:41.080618 28320 raft_consensus.cc:515] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:41.166293 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 29 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:41.167182 18364 raft_consensus.cc:2468] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 28.
I20260709 12:25:41.174710 17758 raft_consensus.cc:1240] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 31 LEADER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 30. Current term is 31. Ops: [30.21-30.21]
I20260709 12:25:41.177280 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 30 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:41.177891 18361 raft_consensus.cc:3060] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Advancing to term 30
W20260709 12:25:41.185927 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 43549us
I20260709 12:25:41.187723 18371 raft_consensus.cc:1275] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 32 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 28 index: 20. Preceding OpId from leader: term: 32 index: 22. (index mismatch)
W20260709 12:25:41.167243 28645 consensus_meta.cc:287] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.556s	user 0.007s	sys 0.005s
I20260709 12:25:41.152469 28900 leader_election.cc:290] T a5732e71b9d54f8786b8c511d69943bf P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 32 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:25:41.177794 28717 consensus_meta.cc:287] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.521s	user 0.004s	sys 0.009s
I20260709 12:25:41.191432 17803 leader_election.cc:304] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3
W20260709 12:25:41.199976 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 306129us
W20260709 12:25:41.200453 28913 log.cc:927] Time spent T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.081s	user 0.001s	sys 0.000s
W20260709 12:25:41.203900 18334 leader_election.cc:336] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:41.176783 28651 raft_consensus.cc:493] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:41.206881 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 33 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:41.207636 17759 raft_consensus.cc:3060] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:25:41.207540 28651 raft_consensus.cc:515] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:41.209964 28651 leader_election.cc:290] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:41.214833 28794 leader_election.cc:290] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:41.201529 28645 raft_consensus.cc:515] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:25:41.217729 28920 log.cc:927] Time spent T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.093s	user 0.000s	sys 0.001s
I20260709 12:25:41.218051 17833 raft_consensus.cc:1240] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 33 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 32. Current term is 33. Ops: []
I20260709 12:25:41.205224 28717 raft_consensus.cc:515] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:41.220639 28638 leader_election.cc:290] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:41.220970 28717 leader_election.cc:290] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 32 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:25:41.206171 28927 raft_consensus.cc:493] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:41.206768 28320 leader_election.cc:290] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 29 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:41.227540 17759 raft_consensus.cc:2468] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 33.
I20260709 12:25:41.228228 18361 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 30.
I20260709 12:25:41.230376 18333 leader_election.cc:304] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:41.231817 28308 raft_consensus.cc:2804] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:25:41.205124 18334 leader_election.cc:304] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:41.239058 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 32 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:41.240000 17759 raft_consensus.cc:2393] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 32: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:25:41.240444 17803 leader_election.cc:304] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:25:41.241237 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:41.228361 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:41.228169 28821 raft_consensus.cc:2749] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:25:41.217753 28645 leader_election.cc:290] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 33 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:25:41.233011 28927 raft_consensus.cc:515] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:41.246708 28927 leader_election.cc:290] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:41.248442 28561 leader_election.cc:290] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:41.251911 17763 raft_consensus.cc:2468] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 28.
I20260709 12:25:41.254778 28926 raft_consensus.cc:2804] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 33 FOLLOWER]: Leader election won for term 33
I20260709 12:25:41.255334 28308 raft_consensus.cc:697] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 29 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:25:41.257332 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 34 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:41.258636 17759 raft_consensus.cc:2410] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for term 34 because replica has last-logged OpId of term: 32 index: 23, which is greater than that of the candidate, which has last-logged OpId of term: 31 index: 22.
I20260709 12:25:41.259248 17803 leader_election.cc:304] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:25:41.259598 18333 leader_election.cc:304] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:41.264549 28898 raft_consensus.cc:2749] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:25:41.265115 17803 leader_election.cc:304] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:41.265314 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:41.267045 18374 raft_consensus.cc:3060] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:41.269129 28812 raft_consensus.cc:2749] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:25:41.271859 17803 leader_election.cc:304] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 35 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:41.274341 28923 raft_consensus.cc:2804] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:25:41.274956 28923 raft_consensus.cc:697] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 28 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:25:41.275328 28891 raft_consensus.cc:2749] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:25:41.276091 28873 raft_consensus.cc:493] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 33 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:25:41.275915 28923 consensus_queue.cc:237] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 27.19, Last appended by leader: 19, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:41.277463 18374 raft_consensus.cc:2468] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 29.
I20260709 12:25:41.276567 28873 raft_consensus.cc:3060] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:25:41.278411 28891 raft_consensus.cc:2749] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 36 FOLLOWER]: Leader election lost for term 35. Reason: could not achieve majority
I20260709 12:25:41.280977 28644 consensus_queue.cc:1059] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 32.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
I20260709 12:25:41.282150 28644 raft_consensus.cc:3055] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 32 LEADER]: Stepping down as leader of term 32
I20260709 12:25:41.282521 28644 raft_consensus.cc:740] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 32 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:25:41.271999 28926 raft_consensus.cc:697] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 33 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:25:41.283320 28644 consensus_queue.cc:260] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 32.19, Last appended by leader: 19, Current term: 32, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:41.290150 28644 raft_consensus.cc:3060] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:25:41.257315 28308 consensus_queue.cc:237] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 27.19, Last appended by leader: 19, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:41.288853 28926 consensus_queue.cc:237] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 31.21, Last appended by leader: 21, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:41.268391 28924 raft_consensus.cc:2749] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:25:41.294440 28559 raft_consensus.cc:493] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:41.295779 18367 raft_consensus.cc:2393] T a5732e71b9d54f8786b8c511d69943bf P 334a4e1f59944544b3b3ff36616b57da [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 32: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:25:41.295671 17803 leader_election.cc:304] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:41.298183 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 34 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:25:41.305213 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 101359us
I20260709 12:25:41.307802 17803 leader_election.cc:304] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
W20260709 12:25:41.307145 18334 leader_election.cc:336] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:41.301242 28822 raft_consensus.cc:493] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:41.310186 18334 leader_election.cc:304] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:41.303582 28847 raft_consensus.cc:2804] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:25:41.287503 28873 raft_consensus.cc:515] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 34 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:41.295130 28559 raft_consensus.cc:515] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:41.308930 28812 consensus_queue.cc:1048] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 21, Time since last communication: 0.013s
I20260709 12:25:41.325896 28917 raft_consensus.cc:2749] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:25:41.326604 28917 raft_consensus.cc:493] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:41.332268 17729 leader_election.cc:304] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:41.314492 28840 raft_consensus.cc:2804] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:25:41.315009 28847 raft_consensus.cc:493] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:25:41.338920 28884 raft_consensus.cc:2804] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:25:41.339417 28884 raft_consensus.cc:493] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:41.340018 28884 raft_consensus.cc:3060] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:41.314335 28822 raft_consensus.cc:515] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:41.345882 17832 raft_consensus.cc:1275] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 25 index: 17. Preceding OpId from leader: term: 27 index: 17. (term mismatch)
I20260709 12:25:41.346289 17832 pending_rounds.cc:85] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b: Aborting all ops after (but not including) 16
I20260709 12:25:41.346375 28822 leader_election.cc:290] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:41.346547 17832 pending_rounds.cc:107] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b: Aborting uncommitted NO_OP operation due to leader change: 25.17
I20260709 12:25:41.347149 17832 raft_consensus.cc:2889] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:41.348163 17834 raft_consensus.cc:1275] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 36 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 34 index: 19. Preceding OpId from leader: term: 36 index: 20. (index mismatch)
I20260709 12:25:41.335840 28637 consensus_queue.cc:1059] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 27.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.016s
I20260709 12:25:41.352499 28757 consensus_queue.cc:1048] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.016s
I20260709 12:25:41.327020 28917 raft_consensus.cc:515] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:41.357837 17763 raft_consensus.cc:1240] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 27. Current term is 28. Ops: []
I20260709 12:25:41.358201 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:41.358419 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 34 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:41.360291 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 34 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:41.355986 17729 leader_election.cc:304] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:41.309840 28898 consensus_queue.cc:1048] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.023s
W20260709 12:25:41.369511 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 62395.5us
I20260709 12:25:41.371531 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 32 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:41.368846 28840 raft_consensus.cc:697] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 30 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:25:41.332464 28929 raft_consensus.cc:2749] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:25:41.376240 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 34 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:41.382282 17760 raft_consensus.cc:1275] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 24 index: 16. Preceding OpId from leader: term: 27 index: 17. (index mismatch)
I20260709 12:25:41.354436 28847 raft_consensus.cc:3060] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:41.376495 28894 raft_consensus.cc:493] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:41.380131 28893 consensus_queue.cc:1048] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.015s
I20260709 12:25:41.378206 28840 consensus_queue.cc:237] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 28.15, Last appended by leader: 15, Current term: 30, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:41.395007 28894 raft_consensus.cc:515] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:41.404314 28769 raft_consensus.cc:2804] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:25:41.398828 17652 leader_election.cc:304] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:41.422962 17760 raft_consensus.cc:1275] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 32 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 30 index: 19. Preceding OpId from leader: term: 32 index: 20. (index mismatch)
I20260709 12:25:41.425654 17652 leader_election.cc:304] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:25:41.427421 28928 log.cc:927] Time spent T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.179s	user 0.000s	sys 0.001s
I20260709 12:25:41.428848 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 32 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:41.433744 17652 leader_election.cc:304] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:41.404969 28893 consensus_queue.cc:1059] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 27.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.020s
I20260709 12:25:41.380219 28726 consensus_queue.cc:1048] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.051s
W20260709 12:25:41.436517 17802 leader_election.cc:336] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 33 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:41.376127 28754 consensus_queue.cc:1048] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 20, Time since last communication: 0.055s
I20260709 12:25:41.435416 28896 raft_consensus.cc:2749] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:25:41.421648 28891 raft_consensus.cc:493] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:25:41.375787 28906 raft_consensus.cc:515] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 34 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:41.466401 28651 raft_consensus.cc:493] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:41.467000 28651 raft_consensus.cc:515] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:41.439926 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 365070us
I20260709 12:25:41.467844 28906 leader_election.cc:290] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 34 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:25:41.468315 17728 leader_election.cc:336] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 31 election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:25:41.469475 17728 leader_election.cc:336] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 32 election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:25:41.470403 17728 consensus_peers.cc:597] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:25:41.471359 17728 leader_election.cc:336] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 33 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:41.473702 28894 leader_election.cc:290] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:41.436194 28822 raft_consensus.cc:2749] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:25:41.446600 17802 leader_election.cc:304] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
W20260709 12:25:41.455339 28930 log.cc:927] Time spent T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.140s	user 0.000s	sys 0.001s
I20260709 12:25:41.463472 28819 raft_consensus.cc:3060] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:25:41.464138 28559 leader_election.cc:290] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:41.441762 28769 raft_consensus.cc:697] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 29 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
W20260709 12:25:41.478962 17802 consensus_peers.cc:597] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:25:41.471807 17728 leader_election.cc:304] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
W20260709 12:25:41.597761 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 119980us
W20260709 12:25:41.599260 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 120351us
I20260709 12:25:41.468104 18368 raft_consensus.cc:1240] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 34 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 33. Current term is 34. Ops: [33.20-33.20]
I20260709 12:25:41.473985 28651 leader_election.cc:290] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:25:41.605830 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 134490us
I20260709 12:25:41.607179 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 33 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:41.598580 28645 raft_consensus.cc:2749] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
W20260709 12:25:41.605749 17802 consensus_peers.cc:597] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:25:41.615393 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 34 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:41.624975 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 32 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:41.629117 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:41.602689 28894 consensus_queue.cc:1048] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.014s
I20260709 12:25:41.598340 28891 raft_consensus.cc:515] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:41.600673 28665 consensus_queue.cc:1048] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.014s
I20260709 12:25:41.631430 17758 raft_consensus.cc:2393] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 32: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:25:41.611629 28749 raft_consensus.cc:2749] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:25:41.631465 17759 raft_consensus.cc:2468] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 31.
I20260709 12:25:41.613896 28873 leader_election.cc:290] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 34 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:25:41.661037 17802 leader_election.cc:336] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 33 election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:41.646486 28769 consensus_queue.cc:237] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 27.17, Last appended by leader: 17, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:41.670800 17802 leader_election.cc:336] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 32 election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:25:41.672422 28937 log.cc:927] Time spent T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.066s	user 0.000s	sys 0.001s
I20260709 12:25:41.683270 28891 leader_election.cc:290] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:41.632014 28884 raft_consensus.cc:515] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:41.687587 28884 leader_election.cc:290] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 29 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:41.708629 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:41.709339 17833 raft_consensus.cc:3060] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:41.712162 28789 raft_consensus.cc:3055] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:25:41.712287 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 34 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:41.712746 28789 raft_consensus.cc:740] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:25:41.713742 28789 consensus_queue.cc:260] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 27.19, Last appended by leader: 19, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:41.715035 28789 raft_consensus.cc:3060] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:41.718118 18374 raft_consensus.cc:1275] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 29 index: 20. Preceding OpId from leader: term: 31 index: 21. (index mismatch)
I20260709 12:25:41.725158 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 29 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:41.725512 17832 raft_consensus.cc:1275] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 23 index: 16. Preceding OpId from leader: term: 24 index: 17. (index mismatch)
I20260709 12:25:41.725766 18365 raft_consensus.cc:3060] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:41.734371 17833 raft_consensus.cc:2468] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 29.
I20260709 12:25:41.752312 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:41.753257 17761 raft_consensus.cc:3060] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:25:41.758612 17762 raft_consensus.cc:1275] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 33 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 32 index: 23. Preceding OpId from leader: term: 33 index: 23. (term mismatch)
I20260709 12:25:41.759099 17762 pending_rounds.cc:85] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb: Aborting all ops after (but not including) 22
I20260709 12:25:41.760237 17762 pending_rounds.cc:107] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb: Aborting uncommitted NO_OP operation due to leader change: 32.23
I20260709 12:25:41.761461 17762 raft_consensus.cc:2889] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 33 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:41.723628 28603 raft_consensus.cc:493] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:25:41.770620 28603 raft_consensus.cc:515] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:41.772843 28603 leader_election.cc:290] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:41.719730 28782 raft_consensus.cc:493] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:25:41.776286 28782 raft_consensus.cc:515] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:41.778740 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 33 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:41.788200 28308 raft_consensus.cc:493] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:41.788785 28308 raft_consensus.cc:515] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:41.790824 28308 leader_election.cc:290] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:41.778762 28782 leader_election.cc:290] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:41.795564 28782 consensus_queue.cc:1048] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.001s
I20260709 12:25:41.727981 28884 consensus_queue.cc:1048] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.069s
I20260709 12:25:41.801082 28917 leader_election.cc:290] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:41.843437 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 33 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:41.844967 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 32 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:41.846921 18365 raft_consensus.cc:2468] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 29.
I20260709 12:25:41.849293 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 32 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:41.850628 18362 raft_consensus.cc:2393] T a5732e71b9d54f8786b8c511d69943bf P 334a4e1f59944544b3b3ff36616b57da [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 32: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:25:41.849329 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 32 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:41.852898 17761 raft_consensus.cc:2468] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 32.
W20260709 12:25:41.839700 28915 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.688s	user 0.032s	sys 0.052s
I20260709 12:25:41.867753 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 32 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:25:41.871341 17728 leader_election.cc:336] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:41.877753 17728 leader_election.cc:304] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:41.876226 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:41.817581 28898 raft_consensus.cc:493] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:25:41.879334 17762 raft_consensus.cc:2368] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 32. Current term is 33.
I20260709 12:25:41.839949 28906 raft_consensus.cc:493] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:41.881953 17837 raft_consensus.cc:2393] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 32: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
W20260709 12:25:41.877133 28915 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.750s	user 0.032s	sys 0.053s
I20260709 12:25:41.887456 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 31 candidate_status { last_received { term: 30 index: 21 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:25:41.900223 16501 thread.cc:527] Waited for 3000ms trying to join with heartbeat (tid 17714)
I20260709 12:25:41.917169 18374 raft_consensus.cc:1275] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 26 index: 19. Preceding OpId from leader: term: 28 index: 20. (index mismatch)
I20260709 12:25:41.919775 17729 leader_election.cc:304] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:41.904418 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:41.922106 28706 raft_consensus.cc:2804] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:25:41.924994 18364 raft_consensus.cc:2468] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 30.
I20260709 12:25:41.926659 28794 raft_consensus.cc:2749] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:25:41.885890 28906 raft_consensus.cc:515] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:41.871899 28847 raft_consensus.cc:515] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:41.910050 28884 raft_consensus.cc:493] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:41.951735 28819 raft_consensus.cc:2749] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 32 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:25:41.910187 28769 raft_consensus.cc:493] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:41.960747 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 236057us
I20260709 12:25:41.963653 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 31 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:41.964876 17832 raft_consensus.cc:2410] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for term 31 because replica has last-logged OpId of term: 30 index: 21, which is greater than that of the candidate, which has last-logged OpId of term: 28 index: 20.
I20260709 12:25:41.977077 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 29 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:25:41.977818 18334 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.607s	user 0.027s	sys 0.065s
W20260709 12:25:42.051563 18334 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.681s	user 0.027s	sys 0.066s
W20260709 12:25:42.051926 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 681412us
I20260709 12:25:41.976224 28717 raft_consensus.cc:493] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:42.052788 28717 raft_consensus.cc:515] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:42.053557 18334 leader_election.cc:336] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 37 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:42.054057 18334 leader_election.cc:304] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:42.056360 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 36 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:42.058200 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 33 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:42.058817 17835 raft_consensus.cc:3060] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Advancing to term 33
I20260709 12:25:42.058907 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:42.059659 17830 raft_consensus.cc:2468] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 24.
I20260709 12:25:42.060258 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 32 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:42.060520 28717 leader_election.cc:290] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:42.061044 17831 raft_consensus.cc:2393] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 32: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:25:41.886586 28898 raft_consensus.cc:515] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:42.062021 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 34 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:25:42.063441 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 86352.3us
I20260709 12:25:42.059968 28769 raft_consensus.cc:515] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:42.050658 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 31 candidate_status { last_received { term: 27 index: 21 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:42.071727 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:42.072520 17837 raft_consensus.cc:2393] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 28: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:25:42.064211 17727 leader_election.cc:304] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 34 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:41.958802 28706 raft_consensus.cc:697] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:25:42.077210 17801 leader_election.cc:304] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 33 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:25:42.084339 18334 consensus_peers.cc:597] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:25:42.063192 28898 leader_election.cc:290] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:42.085872 28603 raft_consensus.cc:2749] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 34 FOLLOWER]: Leader election lost for term 34. Reason: could not achieve majority
W20260709 12:25:42.095916 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 224570us
W20260709 12:25:42.093389 18334 consensus_peers.cc:597] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:25:42.077798 28638 raft_consensus.cc:2749] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:25:42.077553 28915 raft_consensus.cc:493] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:25:41.961123 28884 raft_consensus.cc:515] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:42.095458 28675 consensus_queue.cc:1048] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.040s
W20260709 12:25:42.139910 18334 consensus_peers.cc:597] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:25:42.134675 28308 raft_consensus.cc:2749] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 33 FOLLOWER]: Leader election lost for term 33. Reason: could not achieve majority
I20260709 12:25:42.136914 28915 raft_consensus.cc:515] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:42.145299 17728 leader_election.cc:336] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:42.146353 17728 leader_election.cc:304] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
W20260709 12:25:42.143361 18334 leader_election.cc:336] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 34 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:42.203521 18334 leader_election.cc:304] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:25:42.209813 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 64643us
I20260709 12:25:42.253656 17837 raft_consensus.cc:1275] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 26 index: 19. Preceding OpId from leader: term: 28 index: 20. (index mismatch)
W20260709 12:25:42.257026 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 113720us
W20260709 12:25:42.257948 18334 leader_election.cc:336] T a5732e71b9d54f8786b8c511d69943bf P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:42.260708 17801 leader_election.cc:304] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:25:42.268733 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:42.259562 28561 raft_consensus.cc:493] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:42.270454 17762 raft_consensus.cc:1275] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 27 index: 18. Preceding OpId from leader: term: 30 index: 20. (index mismatch)
I20260709 12:25:42.270299 28561 raft_consensus.cc:515] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:42.149242 28839 consensus_queue.cc:1048] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.065s
I20260709 12:25:42.268927 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:42.255975 16501 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:25:42.208241 28308 raft_consensus.cc:493] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:42.284519 17801 leader_election.cc:400] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 32 pre-election: Vote denied by peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651) with higher term. Message: Invalid argument: T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 32. Current term is 33.
I20260709 12:25:42.285015 17801 leader_election.cc:403] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 32 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:25:42.208614 28953 raft_consensus.cc:2749] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:25:42.140887 28706 consensus_queue.cc:237] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 27.18, Last appended by leader: 18, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:42.293713 17728 consensus_peers.cc:597] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:25:42.259404 28638 raft_consensus.cc:2749] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:25:42.300501 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 34 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:42.264514 28682 raft_consensus.cc:493] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:42.295760 17728 leader_election.cc:336] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:25:42.306140 17728 consensus_peers.cc:597] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:25:42.302923 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 45016.3us
W20260709 12:25:42.307140 18334 consensus_peers.cc:597] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:25:42.307703 18334 leader_election.cc:336] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:42.279784 28891 raft_consensus.cc:2804] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 32 FOLLOWER]: Leader election won for term 32
I20260709 12:25:42.289399 28884 leader_election.cc:290] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:42.360091 28717 raft_consensus.cc:493] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:42.360188 28951 raft_consensus.cc:493] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:42.361063 28951 raft_consensus.cc:515] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:42.337673 28891 raft_consensus.cc:697] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 32 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:25:42.363399 28951 leader_election.cc:290] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:42.362895 28891 consensus_queue.cc:237] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 30.18, Last appended by leader: 18, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:25:42.384279 28949 log.cc:927] Time spent T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.259s	user 0.001s	sys 0.000s
I20260709 12:25:42.385344 17762 raft_consensus.cc:1275] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 36 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 34 index: 19. Preceding OpId from leader: term: 36 index: 20. (index mismatch)
I20260709 12:25:42.379639 17652 leader_election.cc:304] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:42.387336 28637 raft_consensus.cc:2749] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:25:42.391341 28561 leader_election.cc:290] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:42.394132 17652 leader_election.cc:304] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:42.394942 28637 raft_consensus.cc:2749] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:25:42.396755 17652 leader_election.cc:304] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:42.397496 28637 raft_consensus.cc:3060] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:25:42.402743 17652 leader_election.cc:304] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:42.360934 28717 raft_consensus.cc:515] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:42.433063 17835 raft_consensus.cc:2468] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 33.
I20260709 12:25:42.296523 28308 raft_consensus.cc:515] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:42.431531 28603 raft_consensus.cc:493] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:42.435611 28898 raft_consensus.cc:2749] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 33 FOLLOWER]: Leader pre-election lost for term 32. Reason: Vote denied by peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651) with higher term. Message: Invalid argument: T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 32. Current term is 33.
I20260709 12:25:42.439214 28847 leader_election.cc:290] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 25 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:42.449302 28915 leader_election.cc:290] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:42.440868 28894 raft_consensus.cc:2749] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:25:42.451979 28603 raft_consensus.cc:515] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:42.455475 28603 leader_election.cc:290] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:42.320035 28682 raft_consensus.cc:515] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:42.460155 17836 raft_consensus.cc:2393] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 30: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:25:42.464360 18367 raft_consensus.cc:1275] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 23 index: 16. Preceding OpId from leader: term: 24 index: 17. (index mismatch)
I20260709 12:25:42.469645 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 34 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:42.470331 18361 raft_consensus.cc:2468] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 33.
I20260709 12:25:42.465726 17652 leader_election.cc:304] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:42.437198 28906 leader_election.cc:290] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:25:42.466023 28769 leader_election.cc:290] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:42.476447 28717 leader_election.cc:290] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:25:42.478480 28894 raft_consensus.cc:2804] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:25:42.508020 18333 leader_election.cc:304] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:25:42.508210 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 226114us
I20260709 12:25:42.492144 28956 raft_consensus.cc:493] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
W20260709 12:25:42.503595 17573 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet d48b7de191684318aa08f60a7a395387 with cas_config_opid_index 12: ChangeConfig:REMOVE_PEER failed with leader 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 334a4e1f59944544b3b3ff36616b57da is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 28 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:25:42.504989 28894 raft_consensus.cc:493] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:25:42.518090 28950 log.cc:927] Time spent T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.291s	user 0.000s	sys 0.001s
I20260709 12:25:42.527091 17759 raft_consensus.cc:1275] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 28 index: 15. Preceding OpId from leader: term: 30 index: 16. (index mismatch)
I20260709 12:25:42.522521 28894 raft_consensus.cc:3060] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:25:42.536746 18368 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59902 (request call id 2554) took 1409 ms. Trace:
I20260709 12:25:42.537967 18365 raft_consensus.cc:3055] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 31 LEADER]: Stepping down as leader of term 31
I20260709 12:25:42.538313 18365 raft_consensus.cc:740] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 31 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:25:42.518986 28638 raft_consensus.cc:2749] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:25:42.532044 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:42.537119 18368 rpcz_store.cc:276] 0709 12:25:41.127587 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:41.127712 (+   125us) service_pool.cc:224] Handling call
0709 12:25:41.128066 (+   354us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:41.601637 (+473571us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:42.536521 (+934884us) spinlock_profiling.cc:243] Waited 340 ms on lock 0x7b540047ca54. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:25:42.536688 (+   167us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":781505280}
I20260709 12:25:42.543401 17763 raft_consensus.cc:2393] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 25: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:25:42.533003 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:42.513067 28917 consensus_queue.cc:1048] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.019s
I20260709 12:25:42.512683 28717 consensus_queue.cc:1048] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 18, Time since last communication: 0.019s
I20260709 12:25:42.522301 28956 raft_consensus.cc:515] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:42.551594 18365 consensus_queue.cc:260] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 31.20, Last appended by leader: 20, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:42.560685 17729 leader_election.cc:304] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:42.584995 28682 leader_election.cc:290] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:42.585677 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 34 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:42.588049 16501 tablet_replica.cc:333] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3: stopping tablet replica
I20260709 12:25:42.589799 16501 raft_consensus.cc:2243] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 35 LEADER]: Raft consensus shutting down.
I20260709 12:25:42.597208 18365 raft_consensus.cc:3060] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:25:42.557061 28790 raft_consensus.cc:493] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:42.612282 16501 raft_consensus.cc:2272] T 23c0cde5227f470388b8ed1fc0786119 P 0da169ff82674dc1947055ce43de68a3 [term 35 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:42.624344 28308 leader_election.cc:290] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:42.595541 28749 raft_consensus.cc:493] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 34 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:25:42.623121 28894 raft_consensus.cc:515] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:42.646188 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:42.654294 17835 raft_consensus.cc:2468] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 25.
I20260709 12:25:42.591063 28782 consensus_queue.cc:1048] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.036s
I20260709 12:25:42.610945 28757 raft_consensus.cc:493] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:42.648849 17836 raft_consensus.cc:1275] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 29 index: 16. Preceding OpId from leader: term: 31 index: 17. (index mismatch)
I20260709 12:25:42.649004 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 34 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:42.627187 28790 raft_consensus.cc:515] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:42.660681 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:42.664651 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 32 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:42.665545 17835 raft_consensus.cc:2468] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 31.
I20260709 12:25:42.649727 17834 raft_consensus.cc:1275] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 27 index: 17. Preceding OpId from leader: term: 29 index: 18. (index mismatch)
I20260709 12:25:42.649958 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 37 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:42.605202 28873 raft_consensus.cc:2804] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Leader pre-election won for term 30
W20260709 12:25:42.645320 18334 leader_election.cc:336] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 30 election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:42.666739 17764 raft_consensus.cc:2468] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 36.
I20260709 12:25:42.667136 18334 leader_election.cc:304] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:42.679065 17755 raft_consensus.cc:2393] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 34 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 34: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:25:42.682713 18365 raft_consensus.cc:1275] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 32 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 31 index: 20. Preceding OpId from leader: term: 32 index: 20. (term mismatch)
I20260709 12:25:42.683311 18365 pending_rounds.cc:85] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da: Aborting all ops after (but not including) 19
I20260709 12:25:42.683606 18365 pending_rounds.cc:107] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da: Aborting uncommitted NO_OP operation due to leader change: 31.20
I20260709 12:25:42.684006 18365 raft_consensus.cc:2889] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 32 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:42.691002 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 32 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:25:42.691943 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 51966us
I20260709 12:25:42.692894 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:25:42.696542 18334 leader_election.cc:336] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:42.697034 18334 leader_election.cc:304] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:42.697180 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 31 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:42.743419 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 34 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:42.697752 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 31 candidate_status { last_received { term: 28 index: 22 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:25:42.748715 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 54440.8us
W20260709 12:25:42.749446 18334 leader_election.cc:336] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:42.749821 18334 leader_election.cc:304] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
W20260709 12:25:42.751593 18334 leader_election.cc:336] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:42.785609 18334 leader_election.cc:304] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:42.679000 28873 raft_consensus.cc:493] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
W20260709 12:25:42.790583 18334 leader_election.cc:336] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:42.791270 18334 leader_election.cc:304] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
W20260709 12:25:42.793159 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 275460us
I20260709 12:25:42.794154 18333 leader_election.cc:304] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:42.799585 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 31 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:42.649727 28749 raft_consensus.cc:515] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:42.809201 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 34 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:25:42.784911 28900 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 32 current term: 32 is higher.
I20260709 12:25:42.808579 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:42.803007 17801 leader_election.cc:304] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:42.748399 28906 raft_consensus.cc:2749] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:25:42.784005 28638 raft_consensus.cc:2749] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
I20260709 12:25:42.784163 28915 raft_consensus.cc:2749] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:25:42.687835 28757 raft_consensus.cc:515] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:42.690517 28953 consensus_queue.cc:1048] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.123s
I20260709 12:25:42.696705 28847 consensus_queue.cc:1048] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.114s
I20260709 12:25:42.800580 28873 raft_consensus.cc:3060] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:42.802081 28945 raft_consensus.cc:2749] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:25:42.834578 17834 raft_consensus.cc:1275] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 33 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 31 index: 21. Preceding OpId from leader: term: 33 index: 22. (index mismatch)
I20260709 12:25:42.842406 17764 raft_consensus.cc:1217] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 32 FOLLOWER]: Deduplicated request from leader. Original: 30.19->[31.20-31.20]   Dedup: 31.20->[]
I20260709 12:25:42.843130 17764 raft_consensus.cc:1240] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 32 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 31. Current term is 32. Ops: [31.20-31.20]
I20260709 12:25:42.828152 28965 raft_consensus.cc:2804] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 36 FOLLOWER]: Leader pre-election won for term 37
I20260709 12:25:42.847535 28965 raft_consensus.cc:493] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 36 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:25:42.848048 28965 raft_consensus.cc:3060] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:25:42.809857 28917 consensus_queue.cc:1048] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.029s
I20260709 12:25:42.828557 28967 raft_consensus.cc:493] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:42.887847 28749 leader_election.cc:290] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:25:42.888091 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:42.814891 28952 consensus_queue.cc:1048] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.039s
W20260709 12:25:42.898387 28637 consensus_meta.cc:287] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.500s	user 0.011s	sys 0.000s
I20260709 12:25:42.861145 28915 raft_consensus.cc:2804] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:25:42.904738 28682 raft_consensus.cc:493] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:42.924037 28945 raft_consensus.cc:2749] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:25:42.946852 28637 raft_consensus.cc:2749] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 32 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:25:42.963084 28682 raft_consensus.cc:515] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:42.949774 28665 raft_consensus.cc:493] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:42.949102 28915 raft_consensus.cc:493] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:25:42.961222 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 153763us
W20260709 12:25:42.962061 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 168027us
I20260709 12:25:42.984205 18333 leader_election.cc:304] T a5732e71b9d54f8786b8c511d69943bf P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:42.986706 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 31 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:42.987414 18365 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 30.
I20260709 12:25:42.987545 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:42.988128 18364 raft_consensus.cc:2468] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 25.
W20260709 12:25:42.983297 17802 consensus_peers.cc:597] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:25:42.913410 28967 raft_consensus.cc:515] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:42.995069 28906 raft_consensus.cc:2749] T a5732e71b9d54f8786b8c511d69943bf P 334a4e1f59944544b3b3ff36616b57da [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
I20260709 12:25:42.970692 16501 tablet_replica.cc:333] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3: stopping tablet replica
W20260709 12:25:43.001896 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 211330us
W20260709 12:25:43.002516 18334 leader_election.cc:336] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:25:43.003029 18334 leader_election.cc:336] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:43.003340 18334 leader_election.cc:304] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:42.984131 28915 raft_consensus.cc:3060] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:42.992419 18333 leader_election.cc:304] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:43.000648 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 31 candidate_status { last_received { term: 28 index: 22 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:42.989881 28945 raft_consensus.cc:493] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:25:43.020653 18365 raft_consensus.cc:2468] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 30.
I20260709 12:25:42.974633 28637 consensus_queue.cc:1048] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.030s
I20260709 12:25:42.993693 28970 raft_consensus.cc:493] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
W20260709 12:25:43.021740 18334 leader_election.cc:336] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 28 election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:25:43.023382 17802 consensus_peers.cc:597] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:25:43.028798 16501 raft_consensus.cc:2243] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 35 LEADER]: Raft consensus shutting down.
W20260709 12:25:43.029269 17803 leader_election.cc:336] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Timed out: RequestConsensusVote RPC to 127.16.29.68:42577 timed out after 1.640s (SENT)
I20260709 12:25:43.030591 17803 leader_election.cc:304] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:25:43.033198 16501 raft_consensus.cc:2272] T 9439565c5dbb4f4ba3375aaef2c9add5 P 0da169ff82674dc1947055ce43de68a3 [term 35 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:42.990041 28665 raft_consensus.cc:515] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:43.034705 18334 leader_election.cc:336] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:43.036557 28682 leader_election.cc:290] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:43.023751 28906 raft_consensus.cc:2749] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
W20260709 12:25:43.039534 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 230992us
W20260709 12:25:43.043378 18334 leader_election.cc:336] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:25:43.043597 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 51267.3us
I20260709 12:25:43.045965 18333 leader_election.cc:304] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:43.050762 17729 leader_election.cc:304] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:25:43.058746 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:43.084424 18333 leader_election.cc:304] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:43.093746 17762 raft_consensus.cc:2393] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 32: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:25:43.034863 28308 raft_consensus.cc:2749] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:25:43.056188 28638 raft_consensus.cc:2749] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
W20260709 12:25:43.141597 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 57267.6us
I20260709 12:25:43.145529 18333 leader_election.cc:304] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:25:43.031339 28945 raft_consensus.cc:515] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:43.033486 28970 raft_consensus.cc:515] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:43.141289 28906 raft_consensus.cc:2749] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
W20260709 12:25:43.172514 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 179598us
W20260709 12:25:43.180487 28979 log.cc:927] Time spent T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.174s	user 0.000s	sys 0.001s
I20260709 12:25:43.146296 28757 leader_election.cc:290] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:25:43.153010 28974 log.cc:927] Time spent T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.069s	user 0.000s	sys 0.000s
I20260709 12:25:43.216688 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 37 candidate_status { last_received { term: 35 index: 21 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:25:43.193753 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 161056us
W20260709 12:25:43.192277 28971 log.cc:927] Time spent T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.207s	user 0.000s	sys 0.001s
W20260709 12:25:43.222529 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:43.214612 28503 raft_consensus.cc:493] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:25:43.185978 28980 raft_consensus.cc:493] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
W20260709 12:25:43.289183 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 61172.7us
I20260709 12:25:43.292851 28992 raft_consensus.cc:493] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:43.321187 17831 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35042 (request call id 3797) took 1067 ms. Trace:
I20260709 12:25:43.372313 17831 rpcz_store.cc:276] 0709 12:25:42.253917 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:42.254085 (+   168us) service_pool.cc:224] Handling call
0709 12:25:42.269537 (+ 15452us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:42.468899 (+199362us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:42.468929 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:42.471817 (+  2888us) log.cc:844] Serialized 26 byte log entry
0709 12:25:42.805538 (+333721us) raft_consensus.cc:1645] Marking committed up to 25
0709 12:25:42.836732 (+ 31194us) log.cc:844] Serialized 14 byte log entry
0709 12:25:42.837016 (+   284us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:42.837102 (+    86us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:43.320870 (+483768us) raft_consensus.cc:1685] finished
0709 12:25:43.320891 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:43.321071 (+   180us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":57785,"prepare.queue_time_us":22836,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":41,"raft.queue_time_us":52839,"raft.run_cpu_time_us":303,"raft.run_wall_time_us":44052,"thread_start_us":472541,"threads_started":2,"wal-append.queue_time_us":413144}
W20260709 12:25:43.328814 28991 log.cc:927] Time spent T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.176s	user 0.000s	sys 0.000s
I20260709 12:25:43.342356 28665 leader_election.cc:290] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:43.345160 28970 leader_election.cc:290] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:25:43.357034 28990 log.cc:927] Time spent T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.137s	user 0.000s	sys 0.000s
I20260709 12:25:43.219949 28638 raft_consensus.cc:2764] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Leader pre-election decision vote started in defunct term 28: won
W20260709 12:25:43.361411 28988 log.cc:927] Time spent T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.222s	user 0.001s	sys 0.000s
I20260709 12:25:43.191430 28983 raft_consensus.cc:2749] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
W20260709 12:25:43.360503 28982 log.cc:927] Time spent T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.186s	user 0.000s	sys 0.000s
I20260709 12:25:43.221282 28906 raft_consensus.cc:493] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 32 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:25:43.418403 28992 raft_consensus.cc:515] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:43.426498 17654 leader_election.cc:304] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
W20260709 12:25:43.430716 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 285248us
I20260709 12:25:43.292306 28884 raft_consensus.cc:493] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:43.433987 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 37 candidate_status { last_received { term: 35 index: 21 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:43.434829 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:43.437265 28995 raft_consensus.cc:493] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:43.438001 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 31 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:43.437814 28995 raft_consensus.cc:515] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:43.421587 28981 raft_consensus.cc:493] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:43.424631 28665 raft_consensus.cc:493] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:43.443744 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 31 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:43.445206 18363 raft_consensus.cc:2410] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for term 31 because replica has last-logged OpId of term: 29 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 27 index: 19.
W20260709 12:25:43.446715 17655 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:43.448544 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 37 candidate_status { last_received { term: 35 index: 24 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:25:43.450702 17803 leader_election.cc:336] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Timed out: RequestConsensusVote RPC to 127.16.29.68:42577 timed out after 2.062s (SENT)
I20260709 12:25:43.451264 17803 leader_election.cc:304] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:25:43.427548 28970 raft_consensus.cc:493] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:25:43.454659 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 31 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:43.456768 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 31 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:43.481289 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:43.484359 28997 raft_consensus.cc:2804] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:25:43.422343 28503 raft_consensus.cc:515] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:43.465756 28665 raft_consensus.cc:515] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:43.484895 28997 raft_consensus.cc:493] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:43.422243 28980 raft_consensus.cc:515] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:43.439775 28906 raft_consensus.cc:515] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:43.492681 28945 leader_election.cc:290] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:43.503326 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:43.501881 28997 raft_consensus.cc:3060] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:43.447381 28981 raft_consensus.cc:515] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:43.506013 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 448640us
W20260709 12:25:43.528560 18335 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:43.444499 28884 raft_consensus.cc:515] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:43.498987 28967 leader_election.cc:290] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:43.500605 28955 raft_consensus.cc:2749] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:25:43.505534 28985 raft_consensus.cc:493] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:43.538743 17729 leader_election.cc:304] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:43.543612 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 31 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:25:43.544072 28999 log.cc:927] Time spent T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.077s	user 0.001s	sys 0.000s
W20260709 12:25:43.547722 28873 consensus_meta.cc:287] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.702s	user 0.008s	sys 0.004s
I20260709 12:25:43.552240 18363 raft_consensus.cc:1275] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 27 index: 19. Preceding OpId from leader: term: 29 index: 20. (index mismatch)
W20260709 12:25:43.561676 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 110984us
I20260709 12:25:43.487056 28970 raft_consensus.cc:515] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:43.549005 28967 raft_consensus.cc:493] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:43.565124 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 138685us
W20260709 12:25:43.563071 17803 leader_election.cc:336] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 37 pre-election: RPC error from VoteRequest() call to peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577): Timed out: RequestConsensusVote RPC to 127.16.29.68:42577 timed out after 2.064s (SENT)
I20260709 12:25:43.573336 17803 leader_election.cc:304] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:25:43.581015 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 56587.8us
W20260709 12:25:43.586300 18334 leader_election.cc:336] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 36 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:43.588733 17652 leader_election.cc:304] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:43.589283 18334 leader_election.cc:304] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
W20260709 12:25:43.591332 17761 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.510s	user 0.000s	sys 0.003s
W20260709 12:25:43.591755 17761 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.510s	user 0.000s	sys 0.003s
I20260709 12:25:43.565086 29003 raft_consensus.cc:2804] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:25:43.552562 28985 raft_consensus.cc:515] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:43.562906 28873 raft_consensus.cc:515] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:43.585780 28995 leader_election.cc:290] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:43.612591 28955 raft_consensus.cc:2749] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:25:43.578055 28967 raft_consensus.cc:515] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:43.617188 29003 raft_consensus.cc:493] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:43.635020 17755 raft_consensus.cc:2410] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 31 because replica has last-logged OpId of term: 29 index: 20, which is greater than that of the candidate, which has last-logged OpId of term: 27 index: 19.
I20260709 12:25:43.635640 18368 raft_consensus.cc:1275] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 28 index: 19. Preceding OpId from leader: term: 30 index: 20. (index mismatch)
I20260709 12:25:43.636806 28789 raft_consensus.cc:493] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:43.640538 28789 raft_consensus.cc:515] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:43.641575 28985 leader_election.cc:290] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:43.642385 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 31 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:43.642792 28789 leader_election.cc:290] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:43.643116 18366 raft_consensus.cc:2468] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 30.
I20260709 12:25:43.643528 28789 raft_consensus.cc:2749] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:25:43.647821 28997 raft_consensus.cc:515] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:43.649217 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 32 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:43.650108 18366 raft_consensus.cc:2393] T a5732e71b9d54f8786b8c511d69943bf P 334a4e1f59944544b3b3ff36616b57da [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 32: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:25:43.647913 28967 leader_election.cc:290] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:43.650906 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:43.636909 29003 raft_consensus.cc:3060] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:43.661855 28884 leader_election.cc:290] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:43.662912 28790 leader_election.cc:290] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:43.665323 28503 leader_election.cc:290] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:43.666289 28906 leader_election.cc:290] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:25:43.667835 28665 leader_election.cc:290] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:43.668637 28970 leader_election.cc:290] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:43.668931 28981 leader_election.cc:290] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:43.634123 29008 raft_consensus.cc:493] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:25:43.669808 29008 raft_consensus.cc:515] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:43.671633 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:43.672821 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 32 candidate_status { last_received { term: 30 index: 21 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:43.673621 18362 raft_consensus.cc:2410] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for term 32 because replica has last-logged OpId of term: 31 index: 21, which is greater than that of the candidate, which has last-logged OpId of term: 30 index: 21.
I20260709 12:25:43.677254 28873 leader_election.cc:290] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 30 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:25:43.681114 28965 consensus_meta.cc:287] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.832s	user 0.012s	sys 0.000s
I20260709 12:25:43.681528 28965 raft_consensus.cc:515] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 37 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:43.684103 16501 tablet_replica.cc:333] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3: stopping tablet replica
I20260709 12:25:43.685302 16501 raft_consensus.cc:2243] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:43.686038 16501 raft_consensus.cc:2272] T ab84345f9aa742a7b85389e9b6f2ce90 P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Raft consensus is shut down!
W20260709 12:25:43.686996 28915 consensus_meta.cc:287] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da: Time spent flushing consensus metadata: real 0.665s	user 0.002s	sys 0.009s
I20260709 12:25:43.687484 28915 raft_consensus.cc:515] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:43.687995 29008 leader_election.cc:290] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:43.696519 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 31 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:43.697811 18362 raft_consensus.cc:2410] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for term 31 because replica has last-logged OpId of term: 29 index: 20, which is greater than that of the candidate, which has last-logged OpId of term: 27 index: 19.
I20260709 12:25:43.683609 28965 leader_election.cc:290] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 37 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:25:43.700338 18374 raft_consensus.cc:2393] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 25: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:25:43.705809 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 31 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:43.707401 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 36 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:43.710188 18367 raft_consensus.cc:1275] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 27 index: 18. Preceding OpId from leader: term: 29 index: 19. (index mismatch)
I20260709 12:25:43.667956 28894 leader_election.cc:290] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 32 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:43.716992 18367 raft_consensus.cc:1275] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 27 index: 14. Preceding OpId from leader: term: 30 index: 16. (index mismatch)
I20260709 12:25:43.717548 28915 leader_election.cc:290] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 25 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:43.706807 29005 raft_consensus.cc:493] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:43.720008 29005 raft_consensus.cc:515] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:43.722384 29005 leader_election.cc:290] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:43.724529 29003 raft_consensus.cc:515] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:43.726984 29003 leader_election.cc:290] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 26 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:43.728461 28956 leader_election.cc:290] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:43.715317 16501 tablet_replica.cc:333] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3: stopping tablet replica
I20260709 12:25:43.730423 16501 raft_consensus.cc:2243] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 31 LEADER]: Raft consensus shutting down.
I20260709 12:25:43.728478 28997 leader_election.cc:290] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 31 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:25:43.738406 17573 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet d48b7de191684318aa08f60a7a395387 with cas_config_opid_index 12: ChangeConfig:REMOVE_PEER failed with leader 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 334a4e1f59944544b3b3ff36616b57da is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 28 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:25:43.744174 29009 consensus_queue.cc:1059] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 31.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.001s
I20260709 12:25:43.745110 28645 raft_consensus.cc:493] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:25:43.748104 28965 raft_consensus.cc:493] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:43.753015 29008 raft_consensus.cc:3055] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 33 LEADER]: Stepping down as leader of term 33
I20260709 12:25:43.753517 29008 raft_consensus.cc:740] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 33 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:25:43.753270 28965 raft_consensus.cc:515] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:43.754393 29008 consensus_queue.cc:260] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 33.20, Last appended by leader: 20, Current term: 33, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:43.756439 28675 raft_consensus.cc:493] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:43.757015 28675 raft_consensus.cc:515] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:43.761500 28965 leader_election.cc:290] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:25:43.763792 28965 raft_consensus.cc:493] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 36 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:25:43.764894 28645 raft_consensus.cc:515] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:43.765328 28965 raft_consensus.cc:515] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:43.766764 17761 raft_consensus.cc:1275] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 27 index: 19. Preceding OpId from leader: term: 28 index: 20. (index mismatch)
I20260709 12:25:43.768234 28965 leader_election.cc:290] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:25:43.769960 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 31 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:25:43.770629 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 184370us
W20260709 12:25:43.771539 18334 leader_election.cc:336] T a5732e71b9d54f8786b8c511d69943bf P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 32 election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:43.772037 29000 raft_consensus.cc:2749] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:25:43.772769 29000 raft_consensus.cc:493] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:43.773233 29000 raft_consensus.cc:515] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:43.773952 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 32 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:43.775270 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 33 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:43.777571 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 31 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:43.777827 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:43.776820 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:43.779078 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
W20260709 12:25:43.773092 18334 consensus_peers.cc:597] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:25:43.759819 28675 leader_election.cc:290] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:43.772076 17761 raft_consensus.cc:1275] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 33 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 31 index: 21. Preceding OpId from leader: term: 33 index: 22. (index mismatch)
I20260709 12:25:43.781687 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 34 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:25:43.782953 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 122498us
I20260709 12:25:43.783201 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:43.780267 28953 consensus_queue.cc:1048] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.002s
I20260709 12:25:43.786152 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:25:43.781422 18334 leader_election.cc:336] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:43.786980 18334 leader_election.cc:304] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:43.787392 29000 leader_election.cc:290] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:25:43.775177 28645 leader_election.cc:290] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:25:43.799826 28973 log.cc:927] Time spent T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.764s	user 0.005s	sys 0.000s
I20260709 12:25:43.801644 17760 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38786 (request call id 4453) took 1239 ms. Trace:
I20260709 12:25:43.802031 17760 rpcz_store.cc:276] 0709 12:25:42.562789 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:42.562999 (+   210us) service_pool.cc:224] Handling call
0709 12:25:42.597340 (+ 34341us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:42.828608 (+231268us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:42.828665 (+    57us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:42.828712 (+    47us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:25:42.828740 (+    28us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:43.801404 (+972664us) spinlock_profiling.cc:243] Waited 231 ms on lock 0x7b54002977d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:25:43.801454 (+    50us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:43.801585 (+   131us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":531395456}
I20260709 12:25:43.807627 17764 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36572 (request call id 1988) took 1162 ms. Trace:
I20260709 12:25:43.808099 17764 rpcz_store.cc:276] 0709 12:25:42.645682 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:42.696768 (+ 51086us) service_pool.cc:224] Handling call
0709 12:25:42.842287 (+145519us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:42.843493 (+  1206us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:43.807412 (+963919us) spinlock_profiling.cc:243] Waited 145 ms on lock 0x7b54003eb550. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a61f8ad 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:25:43.807562 (+   150us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":333890304}
I20260709 12:25:43.820931 18368 raft_consensus.cc:1275] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 27 index: 17. Preceding OpId from leader: term: 29 index: 18. (index mismatch)
I20260709 12:25:43.825796 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 33 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:43.826828 18368 raft_consensus.cc:2468] T a5732e71b9d54f8786b8c511d69943bf P 334a4e1f59944544b3b3ff36616b57da [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 32.
W20260709 12:25:43.830343 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 1.13119e+06us
I20260709 12:25:43.830708 28900 consensus_queue.cc:1048] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.001s
I20260709 12:25:43.832099 28898 raft_consensus.cc:493] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:25:43.832696 28898 raft_consensus.cc:515] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:43.835053 28898 leader_election.cc:290] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:25:43.836076 18363 raft_consensus.cc:1240] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 24. Current term is 25. Ops: [24.17-24.17]
I20260709 12:25:43.840847 16501 pending_rounds.cc:70] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3: Trying to abort 1 pending ops.
I20260709 12:25:43.841241 16501 pending_rounds.cc:77] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3: Aborting op as it isn't in flight: id { term: 31 index: 17 } timestamp: 7305625357477961728 op_type: NO_OP noop_request { }
I20260709 12:25:43.841717 16501 raft_consensus.cc:2889] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 31 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260709 12:25:43.842125 16501 raft_consensus.cc:2272] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Raft consensus is shut down!
W20260709 12:25:43.877714 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 289060us
I20260709 12:25:43.880324 17652 leader_election.cc:304] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:43.889717 18365 raft_consensus.cc:2468] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 30.
W20260709 12:25:43.890600 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 45147us
I20260709 12:25:43.896335 18366 raft_consensus.cc:2468] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 25.
I20260709 12:25:43.904175 17729 leader_election.cc:304] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:43.905815 17652 leader_election.cc:304] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:43.907999 28987 raft_consensus.cc:2804] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:25:43.920444 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 31 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:43.917100 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 33 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:43.922446 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 33 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:43.923832 17837 raft_consensus.cc:2410] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for term 33 because replica has last-logged OpId of term: 32 index: 22, which is greater than that of the candidate, which has last-logged OpId of term: 30 index: 20.
I20260709 12:25:43.909157 28987 raft_consensus.cc:697] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 30 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
W20260709 12:25:43.926597 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 74983.9us
I20260709 12:25:43.934252 29025 raft_consensus.cc:2749] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
W20260709 12:25:43.927279 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 65150.7us
I20260709 12:25:43.937693 17803 leader_election.cc:304] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:25:43.940162 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 30 candidate_status { last_received { term: 28 index: 22 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:43.904942 28960 raft_consensus.cc:2749] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
I20260709 12:25:43.922488 29000 raft_consensus.cc:2749] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:25:43.922441 28794 raft_consensus.cc:493] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 33 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
W20260709 12:25:43.924822 28891 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.180s	user 0.032s	sys 0.023s
W20260709 12:25:43.969049 28891 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.277s	user 0.032s	sys 0.023s
I20260709 12:25:43.970683 28792 raft_consensus.cc:2804] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:25:43.973167 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 31 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:43.975344 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 31 candidate_status { last_received { term: 28 index: 22 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:43.975268 17803 leader_election.cc:304] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:25:43.980288 17654 leader_election.cc:304] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:43.942814 28726 raft_consensus.cc:493] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:43.900604 28984 consensus_queue.cc:1048] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.072s
I20260709 12:25:43.948194 28645 raft_consensus.cc:493] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:25:43.942631 29023 raft_consensus.cc:493] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:25:43.992686 17756 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38786 (request call id 4452) took 1437 ms. Trace:
I20260709 12:25:43.973677 28965 raft_consensus.cc:2804] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:25:43.925725 28675 consensus_queue.cc:1048] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.055s
I20260709 12:25:43.984790 17836 raft_consensus.cc:1240] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 37 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 36. Current term is 37. Ops: []
I20260709 12:25:43.993830 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 31 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:43.971771 29001 raft_consensus.cc:493] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 36 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:25:43.993382 17756 rpcz_store.cc:276] 0709 12:25:42.555305 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:42.555491 (+   186us) service_pool.cc:224] Handling call
0709 12:25:42.555941 (+   450us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:42.593814 (+ 37873us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:42.593863 (+    49us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:43.886157 (+1292294us) spinlock_profiling.cc:243] Waited 4.83 ms on lock 0x7b2c0000e5c8. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b48f19a11 00007f7b48f29372 00007f7b4a65ba01 00007f7b4a62043a 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2
0709 12:25:43.886590 (+   433us) log.cc:844] Serialized 26 byte log entry
0709 12:25:43.970274 (+ 83684us) raft_consensus.cc:1645] Marking committed up to 22
0709 12:25:43.970313 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:43.970394 (+    81us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:43.985766 (+ 15372us) raft_consensus.cc:1685] finished
0709 12:25:43.985787 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:43.992612 (+  6825us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":54116,"raft.queue_time_us":38628,"raft.run_cpu_time_us":148,"raft.run_wall_time_us":3049,"spinlock_wait_cycles":11102080,"thread_start_us":48245,"threads_started":1,"wal-append.queue_time_us":48599}
I20260709 12:25:43.966581 28900 raft_consensus.cc:493] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 36 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:25:44.000000 17727 leader_election.cc:304] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:25:43.994935 17728 leader_election.cc:336] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 29 election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:43.966885 28955 consensus_queue.cc:1048] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.032s
W20260709 12:25:44.028661 17728 leader_election.cc:336] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:43.982442 28964 raft_consensus.cc:2749] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
I20260709 12:25:43.989704 28789 raft_consensus.cc:2749] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:25:44.141172 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 34 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:44.085789 17652 leader_election.cc:304] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:44.120613 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 34 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:44.038151 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 35 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:44.080116 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 33 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:44.128994 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 31 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:44.142264 17833 raft_consensus.cc:2468] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 33.
I20260709 12:25:44.142570 18366 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 30.
I20260709 12:25:44.090281 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 34 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:44.144363 17836 raft_consensus.cc:2468] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 33.
I20260709 12:25:44.128042 17728 leader_election.cc:304] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
W20260709 12:25:44.101233 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 58745.9us
I20260709 12:25:44.130285 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 30 candidate_status { last_received { term: 28 index: 22 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:43.977262 28794 raft_consensus.cc:515] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:44.133553 17756 raft_consensus.cc:1240] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 34 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 33. Current term is 34. Ops: []
I20260709 12:25:44.193091 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 33 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:44.193464 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 29 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:44.193714 17833 raft_consensus.cc:2468] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 32.
I20260709 12:25:44.194933 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 34 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:44.134217 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 37 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:44.196059 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 29 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:44.196681 29008 raft_consensus.cc:3060] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:25:44.196777 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 29 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:44.197077 28957 raft_consensus.cc:493] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:25:44.197629 28957 raft_consensus.cc:515] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:44.198181 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 32 candidate_status { last_received { term: 30 index: 21 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:44.200429 29018 raft_consensus.cc:493] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:25:44.199591 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab84345f9aa742a7b85389e9b6f2ce90" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 32 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:44.200531 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 31 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:44.202804 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 31 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:44.203132 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 36 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:44.064788 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:25:44.137415 17804 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:44.006625 28952 raft_consensus.cc:493] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 33 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:25:44.138844 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:44.203383 18374 raft_consensus.cc:2468] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 30.
I20260709 12:25:44.030256 28997 raft_consensus.cc:493] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:43.999565 28965 raft_consensus.cc:493] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:44.072305 28992 leader_election.cc:290] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:44.151556 28956 raft_consensus.cc:493] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:25:44.153718 28603 raft_consensus.cc:493] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 32 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:25:44.154369 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 32 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:44.154450 17727 leader_election.cc:304] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:43.999034 29023 raft_consensus.cc:515] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:44.064988 28706 consensus_queue.cc:1048] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.098s
I20260709 12:25:43.993992 28726 raft_consensus.cc:515] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:43.987481 29033 consensus_queue.cc:1048] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.146s
I20260709 12:25:44.166375 17755 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38786 (request call id 4455) took 1487 ms. Trace:
I20260709 12:25:44.262722 17755 rpcz_store.cc:276] 0709 12:25:42.679067 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:42.696982 (+ 17915us) service_pool.cc:224] Handling call
0709 12:25:44.166148 (+1469166us) spinlock_profiling.cc:243] Waited 932 ms on lock 0x7b54000fe1d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:25:44.166315 (+   167us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2143653248}
I20260709 12:25:44.119560 28898 consensus_queue.cc:1048] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 18, Time since last communication: 0.051s
W20260709 12:25:44.174697 17802 leader_election.cc:336] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:44.176467 29024 raft_consensus.cc:2749] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
W20260709 12:25:44.176607 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 90943.4us
I20260709 12:25:43.996160 29002 consensus_queue.cc:1048] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.187s
I20260709 12:25:44.030845 28900 raft_consensus.cc:515] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:44.183494 28891 raft_consensus.cc:493] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:25:43.998795 28987 consensus_queue.cc:237] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 29.20, Last appended by leader: 20, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:44.184589 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 37 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:44.030647 29001 raft_consensus.cc:515] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:44.189230 28980 leader_election.cc:290] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:44.189577 17654 leader_election.cc:304] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:44.190379 28792 raft_consensus.cc:493] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 33 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:25:44.190953 28903 consensus_queue.cc:1059] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 23.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:25:44.191937 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 31 candidate_status { last_received { term: 28 index: 22 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
W20260709 12:25:44.241787 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 213166us
I20260709 12:25:44.242214 17831 raft_consensus.cc:2393] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 30: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
W20260709 12:25:44.229429 18334 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 139 similar messages]
W20260709 12:25:44.251555 17579 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35962 (request call id 362) took 61911 ms (client timeout 15000 ms). Trace:
I20260709 12:25:44.029826 28953 raft_consensus.cc:2804] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:25:44.030503 29005 raft_consensus.cc:493] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:44.265962 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 111625us
I20260709 12:25:44.282627 28873 raft_consensus.cc:2749] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
W20260709 12:25:44.267108 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 92443.3us
I20260709 12:25:44.267823 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:44.350982 17836 raft_consensus.cc:3060] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:44.201171 29018 raft_consensus.cc:515] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:44.277310 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 33 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:44.357317 17727 leader_election.cc:304] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:25:44.193954 28794 leader_election.cc:290] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:44.199769 28957 leader_election.cc:290] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:44.281526 17652 leader_election.cc:304] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
W20260709 12:25:44.285384 17801 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:44.287986 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 29 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:25:44.369542 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 88125.6us
I20260709 12:25:44.296490 17759 raft_consensus.cc:2468] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 32.
I20260709 12:25:44.261477 28965 raft_consensus.cc:3060] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:44.254860 28952 raft_consensus.cc:515] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:44.261654 29019 raft_consensus.cc:2749] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
W20260709 12:25:44.320600 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 91211.6us
W20260709 12:25:44.327005 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 137481us
W20260709 12:25:44.269025 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 282966us
I20260709 12:25:44.269047 28603 raft_consensus.cc:515] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:25:44.320473 17579 rpcz_store.cc:269] 0709 12:24:42.340450 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:54.441845 (+12101395us) service_pool.cc:224] Handling call
0709 12:24:55.466424 (+1024579us) spinlock_profiling.cc:243] Waited 25.4 ms on lock 0x7b5400002210. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b474fbb71 00007f7b474fc0b2 00007f7b50475759 00007f7b5040da76 00007f7b503ded58 00007f7b504dd04e 00007f7b4e99e2a4 00007f7b4e99e232 00007f7b4e99e161 00007f7b4e99e0dc
0709 12:25:44.251405 (+48784981us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:25:40.476937 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:25:40.503070 (+ 26133us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:25:40.503121 (+    51us) write_op.cc:435] Acquired schema lock
0709 12:25:40.503140 (+    19us) tablet.cc:662] Decoding operations
0709 12:25:40.618212 (+115072us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:25:40.629956 (+ 11744us) write_op.cc:641] Partition lock acquired for write op
0709 12:25:40.634139 (+  4183us) tablet.cc:685] Acquiring locks for 7 operations
0709 12:25:40.652530 (+ 18391us) tablet.cc:701] Row locks acquired
0709 12:25:40.652580 (+    50us) write_op.cc:260] PREPARE: finished
0709 12:25:40.652772 (+   192us) write_op.cc:270] Start()
0709 12:25:40.652917 (+   145us) write_op.cc:276] Timestamp: P: 1783599940652705 usec, L: 0
0709 12:25:40.652954 (+    37us) op_driver.cc:348] REPLICATION: starting
0709 12:25:40.686816 (+ 33862us) log.cc:844] Serialized 2825 byte log entry
0709 12:25:41.122424 (+435608us) op_driver.cc:464] REPLICATION: finished
0709 12:25:41.249317 (+126893us) spinlock_profiling.cc:243] Waited 9.68 ms on lock 0x7b4400022b4c. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b47459c85 00007f7b4745a6a2 00007f7b4a1cc1db 00007f7b4a1c7aaf 00007f7b475160ac 00007f7b503709a9 00007f7b4753269b 00007f7b4753093f 00007f7b47532c02 00007f7b4e056ef0
0709 12:25:41.398980 (+149663us) write_op.cc:301] APPLY: starting
0709 12:25:41.431936 (+ 32956us) tablet.cc:1367] starting BulkCheckPresence
0709 12:25:41.642596 (+210660us) tablet.cc:1370] finished BulkCheckPresence
0709 12:25:41.642656 (+    60us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:25:41.945257 (+302601us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:25:41.945580 (+   323us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=14,key_file_lookups=14,delta_file_lookups=0,mrs_lookups=0
0709 12:25:41.945650 (+    70us) write_op.cc:312] APPLY: finished
0709 12:25:41.946007 (+   357us) log.cc:844] Serialized 73 byte log entry
0709 12:25:41.957552 (+ 11545us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:25:41.975305 (+ 17753us) write_op.cc:454] Released schema lock
0709 12:25:42.055974 (+ 80669us) write_op.cc:341] FINISH: Updating metrics
0709 12:25:42.056400 (+   426us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:25:42.056436 (+    36us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":1212662,"reported_tablets":45,"spinlock_wait_cycles":58439936,"child_traces":[["op",{"apply.queue_time_us":202422,"apply.run_cpu_time_us":14672,"apply.run_wall_time_us":680932,"cfile_cache_hit":30,"cfile_cache_hit_bytes":33415,"num_ops":7,"prepare.queue_time_us":323188,"prepare.run_cpu_time_us":14182,"prepare.run_wall_time_us":327483,"raft.queue_time_us":101925,"raft.run_cpu_time_us":9577,"raft.run_wall_time_us":242650,"replication_time_us":433989,"spinlock_wait_cycles":22298112,"thread_start_us":570962,"threads_started":4,"wal-append.queue_time_us":136718,"wal-append.run_cpu_time_us":31025,"wal-append.run_wall_time_us":2125795}]]}
I20260709 12:25:44.361371 29003 raft_consensus.cc:2804] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:25:44.361541 17727 leader_election.cc:304] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:44.260496 28997 raft_consensus.cc:515] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:44.276031 28956 raft_consensus.cc:515] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:44.329959 28980 raft_consensus.cc:2749] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:25:44.365262 29023 leader_election.cc:290] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:44.366917 29001 leader_election.cc:290] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:44.394176 17652 leader_election.cc:304] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:44.329530 28953 raft_consensus.cc:493] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:44.370046 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 29 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:43.997212 28645 raft_consensus.cc:515] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:44.321486 28792 raft_consensus.cc:3060] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:25:44.379208 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:44.378463 17803 leader_election.cc:304] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:44.298200 28992 consensus_queue.cc:1059] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 36.20, Next index: 21, Last known committed idx: 19, Time since last communication: 0.073s
I20260709 12:25:44.384153 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:44.385267 17654 leader_election.cc:304] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: 
I20260709 12:25:44.306864 28891 raft_consensus.cc:515] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:44.396661 17727 leader_election.cc:304] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:44.376080 29024 raft_consensus.cc:2804] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:25:44.414816 17652 leader_election.cc:304] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:44.384064 29009 raft_consensus.cc:493] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:25:44.385038 28898 raft_consensus.cc:493] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:44.433231 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 34 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:44.434310 17835 raft_consensus.cc:2410] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 34 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for term 34 because replica has last-logged OpId of term: 33 index: 20, which is greater than that of the candidate, which has last-logged OpId of term: 31 index: 19.
I20260709 12:25:44.384616 29003 raft_consensus.cc:493] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 33 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:25:44.436951 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 51752.6us
W20260709 12:25:44.428325 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 49942.8us
I20260709 12:25:44.438297 17803 leader_election.cc:304] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:44.383317 29042 raft_consensus.cc:493] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:25:44.356683 29005 raft_consensus.cc:515] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:44.443744 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:44.445116 17652 leader_election.cc:304] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:44.414595 29023 raft_consensus.cc:2804] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:25:44.441215 17837 raft_consensus.cc:2468] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 28.
I20260709 12:25:44.415688 28953 raft_consensus.cc:3060] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:25:44.442212 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 32 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:44.431660 29024 raft_consensus.cc:493] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 33 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:44.436704 29019 raft_consensus.cc:493] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:44.429842 28980 raft_consensus.cc:2749] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:25:44.446780 17654 leader_election.cc:304] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 31 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:44.473603 17836 raft_consensus.cc:2468] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 26.
I20260709 12:25:44.468901 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:44.441901 29003 raft_consensus.cc:3060] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:25:44.441948 28769 raft_consensus.cc:2749] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:25:44.447628 28917 raft_consensus.cc:2749] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
W20260709 12:25:44.514182 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 67483.4us
I20260709 12:25:44.472709 29020 raft_consensus.cc:493] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:44.446673 28992 raft_consensus.cc:2804] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:25:44.513520 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 37 candidate_status { last_received { term: 36 index: 24 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:44.513159 16501 tablet_replica.cc:333] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3: stopping tablet replica
I20260709 12:25:44.533819 17834 raft_consensus.cc:2468] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 36.
I20260709 12:25:44.519794 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 31 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:44.470842 29024 raft_consensus.cc:3060] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 33 FOLLOWER]: Advancing to term 34
W20260709 12:25:44.520710 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 222157us
W20260709 12:25:44.519665 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 46038.1us
I20260709 12:25:44.448740 29023 raft_consensus.cc:493] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:44.535351 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 29 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:44.537880 17652 leader_election.cc:304] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:44.444893 28898 raft_consensus.cc:515] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:44.539464 18334 leader_election.cc:336] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:44.446318 29009 raft_consensus.cc:515] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:44.478188 28645 leader_election.cc:290] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:25:44.546139 18334 leader_election.cc:336] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:25:44.546393 17652 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:44.446990 29042 raft_consensus.cc:515] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:44.547871 18334 leader_election.cc:304] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:44.473219 28819 raft_consensus.cc:2749] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:25:44.530536 29025 raft_consensus.cc:2749] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Leader election lost for term 31. Reason: could not achieve majority
I20260709 12:25:44.532817 28992 raft_consensus.cc:493] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:44.539193 28980 raft_consensus.cc:493] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:44.580453 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:44.475626 29019 raft_consensus.cc:515] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:44.542439 29023 raft_consensus.cc:3060] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:25:44.521013 29041 raft_consensus.cc:3060] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:44.554663 28682 raft_consensus.cc:2749] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
W20260709 12:25:44.585764 29034 log.cc:927] Time spent T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.139s	user 0.000s	sys 0.000s
I20260709 12:25:44.586582 29005 leader_election.cc:290] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:44.587361 28997 leader_election.cc:290] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:44.588378 28956 leader_election.cc:290] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:25:44.581398 29031 log.cc:927] Time spent T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3: Append to log took a long time: real 0.264s	user 0.000s	sys 0.002s
I20260709 12:25:44.590837 29009 leader_election.cc:290] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:25:44.592782 29040 log.cc:927] Time spent T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.082s	user 0.001s	sys 0.000s
I20260709 12:25:44.554184 29020 raft_consensus.cc:515] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:44.582815 28789 raft_consensus.cc:493] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:44.612133 17802 leader_election.cc:336] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:44.674857 17802 leader_election.cc:304] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
W20260709 12:25:44.678681 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 66615.4us
W20260709 12:25:44.679821 17802 consensus_peers.cc:597] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
W20260709 12:25:44.680785 17802 leader_election.cc:336] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 37 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:25:44.681559 17802 consensus_peers.cc:597] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:25:44.617254 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 34 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:25:44.618770 29043 log.cc:927] Time spent T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.105s	user 0.000s	sys 0.001s
I20260709 12:25:44.620015 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 33 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:44.620491 29018 leader_election.cc:290] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:44.623157 29042 leader_election.cc:290] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:25:44.596215 28726 leader_election.cc:290] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:44.626260 28891 leader_election.cc:290] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:25:44.580314 28638 raft_consensus.cc:2749] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
W20260709 12:25:44.595830 29028 log.cc:927] Time spent T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.660s	user 0.004s	sys 0.000s
I20260709 12:25:44.628793 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 34 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:25:44.615742 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 296024us
W20260709 12:25:44.606949 29036 log.cc:927] Time spent T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.069s	user 0.000s	sys 0.000s
I20260709 12:25:44.634788 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:25:44.557117 18334 leader_election.cc:336] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 33 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:44.601652 28503 raft_consensus.cc:2764] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Leader election decision vote started in defunct term 25: won
I20260709 12:25:44.693152 28898 leader_election.cc:290] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:25:44.695663 17728 leader_election.cc:336] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:44.690991 18334 leader_election.cc:304] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:25:44.644753 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 207953us
I20260709 12:25:44.608053 28952 leader_election.cc:290] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:25:44.651270 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 213044us
I20260709 12:25:44.658527 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 37 candidate_status { last_received { term: 36 index: 24 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:44.642004 28992 raft_consensus.cc:3060] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:44.666069 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 36 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:44.606339 28965 raft_consensus.cc:515] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:44.668854 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 37 candidate_status { last_received { term: 35 index: 26 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:44.669169 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:44.704821 18365 raft_consensus.cc:2468] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 35.
I20260709 12:25:44.644196 28900 leader_election.cc:290] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:44.671010 17652 leader_election.cc:304] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:44.674384 28956 raft_consensus.cc:3055] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:25:44.649938 28980 raft_consensus.cc:515] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:44.682085 17802 consensus_peers.cc:597] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:25:44.688330 17834 raft_consensus.cc:2393] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 33: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:25:44.679824 28645 raft_consensus.cc:2749] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:25:44.716881 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 32 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:44.718107 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac984176358d453a8800cc9e0e304061" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 37 candidate_status { last_received { term: 35 index: 26 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:44.719393 28900 raft_consensus.cc:2749] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:25:44.683666 28789 raft_consensus.cc:515] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:44.721416 28789 leader_election.cc:290] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:44.721576 29020 leader_election.cc:290] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:25:44.711512 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 154421us
I20260709 12:25:44.722285 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 32 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:25:44.722425 18334 leader_election.cc:336] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 33 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:44.722816 18334 leader_election.cc:304] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:44.723050 17835 raft_consensus.cc:2410] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for term 32 because replica has last-logged OpId of term: 30 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 28 index: 16.
I20260709 12:25:44.714694 29005 raft_consensus.cc:2749] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
W20260709 12:25:44.714174 17802 consensus_peers.cc:597] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:25:44.721956 16501 raft_consensus.cc:2243] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:44.726001 17729 leader_election.cc:304] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:25:44.728009 17802 leader_election.cc:336] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:44.729604 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 32 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:44.731451 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:44.732939 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 29 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:44.734264 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 29 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:44.734445 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 32 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:44.695461 28792 raft_consensus.cc:515] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 34 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:44.695511 28603 leader_election.cc:290] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:44.736166 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 33 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:44.741146 17829 raft_consensus.cc:2393] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 31: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:25:44.739305 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 29 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:25:44.728698 17802 leader_election.cc:336] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 33 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:44.740135 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 32 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:44.751417 17727 leader_election.cc:304] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:25:44.707599 28956 raft_consensus.cc:740] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:25:44.755080 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 37 candidate_status { last_received { term: 36 index: 25 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:44.760383 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 29 candidate_status { last_received { term: 28 index: 22 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:44.761317 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:44.736097 28900 raft_consensus.cc:2749] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:25:44.762059 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 30 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:44.764559 17801 leader_election.cc:304] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:25:44.766327 17573 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet d48b7de191684318aa08f60a7a395387 with cas_config_opid_index 12: ChangeConfig:REMOVE_PEER failed with leader 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 334a4e1f59944544b3b3ff36616b57da is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 28 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:25:44.767783 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 33 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:44.719084 29024 raft_consensus.cc:515] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 34 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:44.744591 28320 raft_consensus.cc:2804] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 35 FOLLOWER]: Leader pre-election won for term 36
W20260709 12:25:44.772500 17802 leader_election.cc:336] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 37 election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:44.754973 28997 raft_consensus.cc:493] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:44.782977 18333 leader_election.cc:304] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:44.779568 17802 leader_election.cc:304] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 37 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:44.790251 28559 raft_consensus.cc:3055] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 36 LEADER]: Stepping down as leader of term 36
I20260709 12:25:44.790423 28965 leader_election.cc:290] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 31 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:44.792250 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a698a75c7954dd7b24e7889cb19ae35" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:44.793339 29019 leader_election.cc:290] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:44.793731 28997 raft_consensus.cc:515] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:44.794071 28953 raft_consensus.cc:515] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:44.795279 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:44.796679 29050 raft_consensus.cc:2749] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:25:44.777712 28320 raft_consensus.cc:493] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 35 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:25:44.797434 17758 raft_consensus.cc:1240] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 34 FOLLOWER]: Rejecting Update request from peer e063541542a94969898868bf6eb5ce1b for earlier term 33. Current term is 34. Ops: []
I20260709 12:25:44.827131 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 34 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:44.798009 28980 leader_election.cc:290] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:44.773499 28952 raft_consensus.cc:2804] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:25:44.798185 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 30 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:44.767189 28638 raft_consensus.cc:493] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:44.800158 28956 consensus_queue.cc:260] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 24.17, Last appended by leader: 17, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:44.802913 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:44.806066 18333 leader_election.cc:304] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 34 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:44.807106 17803 leader_election.cc:304] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:25:44.813531 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 35 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:44.819700 17654 leader_election.cc:304] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:44.823760 29037 raft_consensus.cc:2749] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 37 FOLLOWER]: Leader election lost for term 37. Reason: could not achieve majority
I20260709 12:25:44.832314 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 37 candidate_status { last_received { term: 36 index: 25 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:44.837239 18368 raft_consensus.cc:2468] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 34.
I20260709 12:25:44.839409 17803 leader_election.cc:304] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:25:44.839242 17654 leader_election.cc:304] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:44.841660 17803 leader_election.cc:304] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:44.842839 18333 leader_election.cc:304] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:44.803504 28503 raft_consensus.cc:493] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 36 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:25:44.847391 17763 raft_consensus.cc:1240] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 32 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 31. Current term is 32. Ops: []
I20260709 12:25:44.815546 28900 raft_consensus.cc:2749] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:25:44.810856 28559 raft_consensus.cc:740] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 36 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:25:44.833555 28320 raft_consensus.cc:3060] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:25:44.855329 29061 raft_consensus.cc:2749] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
W20260709 12:25:44.856274 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 104902us
W20260709 12:25:44.860980 28847 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.585s	user 0.005s	sys 0.014s
I20260709 12:25:44.850639 18333 leader_election.cc:304] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:25:44.863900 17728 leader_election.cc:336] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:44.828153 29051 raft_consensus.cc:493] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:25:44.869233 17728 leader_election.cc:304] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:44.866411 29020 raft_consensus.cc:2749] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:25:44.878468 29051 raft_consensus.cc:515] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:44.881544 29051 leader_election.cc:290] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:44.881744 28559 consensus_queue.cc:260] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 36.20, Last appended by leader: 20, Current term: 36, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:44.883057 28559 raft_consensus.cc:3060] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:25:44.838824 28638 raft_consensus.cc:515] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:44.885316 18362 raft_consensus.cc:2468] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 29.
I20260709 12:25:44.886721 28638 leader_election.cc:290] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:25:44.877118 28847 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.608s	user 0.005s	sys 0.014s
W20260709 12:25:44.912036 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 48164.2us
I20260709 12:25:44.856081 28891 raft_consensus.cc:2749] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:25:44.857177 29052 raft_consensus.cc:2706] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Received pre-election callback for term 29 while not running: Illegal state: RaftConsensus is not running: State = Stopping
I20260709 12:25:44.846967 29042 raft_consensus.cc:2749] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:25:44.848033 28984 raft_consensus.cc:2749] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 34 FOLLOWER]: Leader election lost for term 34. Reason: could not achieve majority
I20260709 12:25:44.974454 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 35 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:44.975304 17830 raft_consensus.cc:2468] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 34.
I20260709 12:25:44.960570 29023 raft_consensus.cc:515] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 33 FOLLOWER]: Starting leader election with config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:44.939409 17728 leader_election.cc:336] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 35 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:44.982297 17728 leader_election.cc:304] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:44.849329 28952 raft_consensus.cc:697] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 26 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
W20260709 12:25:44.964715 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 113248us
I20260709 12:25:44.962446 28873 raft_consensus.cc:2749] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
W20260709 12:25:44.972626 29034 log.cc:927] Time spent T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.106s	user 0.001s	sys 0.000s
W20260709 12:25:44.990020 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 67225us
I20260709 12:25:44.862458 28819 raft_consensus.cc:493] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:44.966109 29028 log.cc:927] Time spent T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.107s	user 0.001s	sys 0.000s
I20260709 12:25:44.981243 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 34 candidate_status { last_received { term: 33 index: 23 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:44.938771 28980 raft_consensus.cc:493] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:44.863483 28503 raft_consensus.cc:515] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:44.862116 29001 raft_consensus.cc:2749] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
W20260709 12:25:44.988725 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 138169us
I20260709 12:25:45.000782 18333 leader_election.cc:304] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:25:44.995905 17728 leader_election.cc:336] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 33 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:25:45.003916 17728 consensus_peers.cc:597] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:25:45.040453 29024 leader_election.cc:290] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 34 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:45.043301 28847 raft_consensus.cc:493] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:25:45.043431 28952 consensus_queue.cc:237] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 24.17, Last appended by leader: 17, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:45.044014 28847 raft_consensus.cc:515] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:45.048990 28847 leader_election.cc:290] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:25:45.050936 28992 raft_consensus.cc:515] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:44.985651 28951 raft_consensus.cc:493] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:25:44.995352 28794 raft_consensus.cc:2749] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
W20260709 12:25:45.054692 17728 leader_election.cc:336] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 34 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:25:45.058830 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:45.005360 28980 raft_consensus.cc:515] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:44.988857 28984 raft_consensus.cc:2749] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:25:45.071307 17834 raft_consensus.cc:2468] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 36.
I20260709 12:25:45.075587 17652 leader_election.cc:304] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:45.077075 28898 consensus_queue.cc:1059] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 33.23, Next index: 24, Last known committed idx: 22, Time since last communication: 0.001s
I20260709 12:25:45.082592 17728 leader_election.cc:304] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:45.052920 28997 leader_election.cc:290] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:25:45.086072 17802 leader_election.cc:336] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 34 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:45.001034 28819 raft_consensus.cc:515] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:45.070909 29001 raft_consensus.cc:493] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:25:45.091915 17652 leader_election.cc:304] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:45.089330 17801 leader_election.cc:304] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:45.096529 17837 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35040 (request call id 4033) took 1159 ms. Trace:
I20260709 12:25:45.005542 29005 raft_consensus.cc:493] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:45.112087 17652 leader_election.cc:304] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:45.112541 17727 leader_election.cc:304] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:25:45.101939 17802 leader_election.cc:336] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 37 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:45.108489 17837 rpcz_store.cc:276] 0709 12:25:43.937298 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:43.939711 (+  2413us) service_pool.cc:224] Handling call
0709 12:25:45.096284 (+1156573us) spinlock_profiling.cc:243] Waited 489 ms on lock 0x7b54000ab4d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:25:45.096427 (+   143us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1125773056}
W20260709 12:25:45.104692 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 50032.9us
W20260709 12:25:45.109481 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 108798us
W20260709 12:25:45.114338 17728 leader_election.cc:336] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 33 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:45.113178 17802 leader_election.cc:304] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:45.116308 17728 leader_election.cc:304] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:25:45.093842 28945 raft_consensus.cc:2749] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 12:25:45.095619 28789 raft_consensus.cc:2749] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:25:45.127966 18333 leader_election.cc:304] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:45.097918 28603 raft_consensus.cc:493] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:25:45.110347 29049 raft_consensus.cc:3055] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 33 LEADER]: Stepping down as leader of term 33
I20260709 12:25:45.130070 29049 raft_consensus.cc:740] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 33 LEADER]: Becoming Follower/Learner. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:25:45.129787 28603 raft_consensus.cc:515] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:45.130867 29049 consensus_queue.cc:260] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 33.23, Last appended by leader: 23, Current term: 33, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:45.132114 29049 raft_consensus.cc:3060] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:25:45.109464 28782 raft_consensus.cc:2749] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:25:45.090121 28951 raft_consensus.cc:515] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:45.117841 28898 raft_consensus.cc:2749] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:25:45.118142 29044 raft_consensus.cc:2804] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:25:45.144500 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 33 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:45.145098 17837 raft_consensus.cc:3060] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:25:45.136606 18333 leader_election.cc:304] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:45.161369 28320 raft_consensus.cc:515] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 36 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:45.147691 28945 raft_consensus.cc:2749] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
W20260709 12:25:45.165372 29003 consensus_meta.cc:287] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb: Time spent flushing consensus metadata: real 0.650s	user 0.009s	sys 0.002s
I20260709 12:25:45.173082 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 31 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:45.353276 18367 raft_consensus.cc:3060] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:45.119731 28917 raft_consensus.cc:2749] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
W20260709 12:25:45.137984 17652 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:45.173375 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 30 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:45.118517 28997 raft_consensus.cc:2749] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:25:45.122537 29064 raft_consensus.cc:2749] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:25:45.115293 29001 raft_consensus.cc:515] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:45.141108 28987 raft_consensus.cc:2749] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:25:45.117897 29005 raft_consensus.cc:515] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:45.164011 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 33 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:45.363320 18365 raft_consensus.cc:3060] T a5732e71b9d54f8786b8c511d69943bf P 334a4e1f59944544b3b3ff36616b57da [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:25:45.145483 29066 raft_consensus.cc:2804] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 36 FOLLOWER]: Leader pre-election won for term 37
I20260709 12:25:45.146914 29044 raft_consensus.cc:493] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 34 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:45.356791 17652 leader_election.cc:304] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:25:45.371258 29063 raft_consensus.cc:493] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:25:45.368093 29044 raft_consensus.cc:3060] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:25:45.392104 28320 leader_election.cc:290] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 36 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:25:45.425822 29062 log.cc:927] Time spent T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.281s	user 0.000s	sys 0.000s
I20260709 12:25:45.426692 28819 leader_election.cc:290] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:45.428392 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 36 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:45.430523 28503 leader_election.cc:290] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:45.367633 29066 raft_consensus.cc:493] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 36 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:25:45.432816 29066 raft_consensus.cc:3060] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:25:45.436081 18364 raft_consensus.cc:2468] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 35.
I20260709 12:25:45.437637 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 29 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:25:45.440598 29067 log.cc:927] Time spent T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.139s	user 0.000s	sys 0.000s
I20260709 12:25:45.444842 29001 leader_election.cc:290] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:45.445876 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 31 candidate_status { last_received { term: 30 index: 23 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:45.446842 29005 leader_election.cc:290] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:25:45.448035 29069 log.cc:927] Time spent T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.081s	user 0.001s	sys 0.000s
I20260709 12:25:45.452656 28980 leader_election.cc:290] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:45.457309 28951 leader_election.cc:290] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:45.458786 17759 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38786 (request call id 4466) took 1318 ms. Trace:
W20260709 12:25:45.452694 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 338364us
I20260709 12:25:45.377712 28997 raft_consensus.cc:2804] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:25:45.473510 18367 raft_consensus.cc:2468] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 31.
W20260709 12:25:45.461220 17728 leader_election.cc:336] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 33 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:45.464119 17837 raft_consensus.cc:2468] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 33.
I20260709 12:25:45.474797 17728 leader_election.cc:304] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:45.459263 17759 rpcz_store.cc:276] 0709 12:25:44.140849 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:44.140962 (+   113us) service_pool.cc:224] Handling call
0709 12:25:45.458613 (+1317651us) spinlock_profiling.cc:243] Waited 155 ms on lock 0x7b54003bd3d4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:25:45.458725 (+   112us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":355423104}
W20260709 12:25:45.476192 17728 leader_election.cc:336] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 36 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:25:45.477205 17728 leader_election.cc:336] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:45.477653 17728 leader_election.cc:304] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:45.467108 18365 raft_consensus.cc:2468] T a5732e71b9d54f8786b8c511d69943bf P 334a4e1f59944544b3b3ff36616b57da [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 33.
I20260709 12:25:45.362448 29003 raft_consensus.cc:515] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 34 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:45.485446 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44ad5363c5954da4a1bff5857780fd5c" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 34 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:25:45.487921 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 311493us
I20260709 12:25:45.473627 28997 raft_consensus.cc:493] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:45.381623 29063 raft_consensus.cc:515] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:45.485081 28794 raft_consensus.cc:3060] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 32 FOLLOWER]: Advancing to term 33
W20260709 12:25:45.488637 17804 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:45.520896 17653 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:45.520987 17803 leader_election.cc:304] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:25:45.494272 18364 raft_consensus.cc:2393] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 29: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:25:45.504585 17764 raft_consensus.cc:1240] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 34 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 33. Current term is 34. Ops: [33.22-33.22]
I20260709 12:25:45.504758 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 37 candidate_status { last_received { term: 36 index: 25 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:45.523736 17803 leader_election.cc:304] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
W20260709 12:25:45.491051 17728 leader_election.cc:336] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 34 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:45.529389 18367 raft_consensus.cc:1240] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 24. Current term is 25. Ops: [24.17-24.17]
I20260709 12:25:45.534135 29072 raft_consensus.cc:2804] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:25:45.496098 29068 raft_consensus.cc:493] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:45.492409 28997 raft_consensus.cc:3060] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Advancing to term 30
W20260709 12:25:45.493342 28559 consensus_meta.cc:287] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.610s	user 0.010s	sys 0.002s
I20260709 12:25:45.500928 29003 leader_election.cc:290] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 34 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:45.495291 29063 leader_election.cc:290] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:25:45.538398 18374 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59886 (request call id 2993) took 1300 ms. Trace:
I20260709 12:25:45.541611 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 30 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:45.553088 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:45.542914 17728 leader_election.cc:304] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:45.510718 28675 raft_consensus.cc:2749] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:25:45.550300 18374 rpcz_store.cc:276] 0709 12:25:44.238549 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:44.238790 (+   241us) service_pool.cc:224] Handling call
0709 12:25:44.239063 (+   273us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:44.358349 (+119286us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:44.358392 (+    43us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:44.358448 (+    56us) raft_consensus.cc:1645] Marking committed up to 24
0709 12:25:44.358484 (+    36us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:45.538172 (+1179688us) spinlock_profiling.cc:243] Waited 119 ms on lock 0x7b5400305f54. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:25:45.538229 (+    57us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:45.538345 (+   116us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":273734400}
I20260709 12:25:45.557080 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9439565c5dbb4f4ba3375aaef2c9add5" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 36 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:45.557580 18374 raft_consensus.cc:3060] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:25:45.448068 29066 raft_consensus.cc:515] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 37 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:45.552188 29072 raft_consensus.cc:697] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 31 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
W20260709 12:25:45.581713 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 90687.1us
W20260709 12:25:45.582696 17728 leader_election.cc:336] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 33 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:45.583119 17728 leader_election.cc:304] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:45.562557 29001 raft_consensus.cc:493] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:45.584827 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 34 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:45.566609 28559 raft_consensus.cc:493] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:45.590102 18374 raft_consensus.cc:2468] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 36.
I20260709 12:25:45.592109 29003 raft_consensus.cc:2749] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
W20260709 12:25:45.591960 29041 consensus_meta.cc:287] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.923s	user 0.012s	sys 0.000s
I20260709 12:25:45.549547 29068 raft_consensus.cc:515] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:45.594501 28559 raft_consensus.cc:515] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:45.607455 29066 leader_election.cc:290] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 37 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:45.609365 29077 raft_consensus.cc:2804] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:25:45.610888 28956 raft_consensus.cc:3060] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:45.586160 28675 raft_consensus.cc:2749] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:25:45.619380 29041 raft_consensus.cc:2749] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:25:45.634402 17838 raft_consensus.cc:1240] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 24. Current term is 25. Ops: []
I20260709 12:25:45.591773 29001 raft_consensus.cc:515] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:45.644049 29072 consensus_queue.cc:237] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 28.21, Last appended by leader: 21, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:45.649320 29057 log.cc:927] Time spent T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.691s	user 0.005s	sys 0.006s
I20260709 12:25:45.650153 28794 raft_consensus.cc:2749] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 33 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:25:45.629177 29077 raft_consensus.cc:493] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 35 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
W20260709 12:25:45.664342 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 70958.7us
I20260709 12:25:45.649354 29068 leader_election.cc:290] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:25:45.688810 17728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:45.664662 29077 raft_consensus.cc:3060] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:25:45.688246 28559 leader_election.cc:290] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:25:45.696404 17728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:45.719923 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 34 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:45.720814 17837 raft_consensus.cc:3060] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:25:45.722227 18333 leader_election.cc:304] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:45.727918 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 31 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:45.740167 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 33 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:45.705966 29041 raft_consensus.cc:493] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:25:45.748016 17830 raft_consensus.cc:2393] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 33: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:25:45.749332 17837 raft_consensus.cc:2468] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 34.
I20260709 12:25:45.729454 29002 raft_consensus.cc:2749] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:25:45.751385 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 35 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:45.709249 29044 raft_consensus.cc:515] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 35 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:45.757974 17652 leader_election.cc:304] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:25:45.736682 29081 raft_consensus.cc:493] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:25:45.762519 28960 raft_consensus.cc:2804] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 34 FOLLOWER]: Leader election won for term 34
I20260709 12:25:45.763101 28960 raft_consensus.cc:697] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 34 LEADER]: Becoming Leader. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:25:45.763782 29044 leader_election.cc:290] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 35 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:45.764046 28960 consensus_queue.cc:237] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 32.19, Last appended by leader: 19, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:45.750957 29041 raft_consensus.cc:515] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:45.769604 29041 leader_election.cc:290] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:45.770777 29001 leader_election.cc:290] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:45.779363 18362 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59902 (request call id 2599) took 1017 ms. Trace:
I20260709 12:25:45.779906 18362 rpcz_store.cc:276] 0709 12:25:44.761752 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:44.761856 (+   104us) service_pool.cc:224] Handling call
0709 12:25:45.779064 (+1017208us) spinlock_profiling.cc:243] Waited 120 ms on lock 0x7b5400570154. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:25:45.779247 (+   183us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":275066240}
W20260709 12:25:45.767369 29038 log.cc:927] Time spent T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 1.255s	user 0.006s	sys 0.013s
I20260709 12:25:45.791435 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 32 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:45.794215 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 35 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:45.797821 18371 raft_consensus.cc:1275] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 29 index: 20. Preceding OpId from leader: term: 30 index: 21. (index mismatch)
I20260709 12:25:45.798736 17758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 33 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:45.799082 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e417cb980e44a77962d2e3dfd58aad0" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 29 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:45.800010 17759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 34 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:45.822676 17759 raft_consensus.cc:3060] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:25:45.800958 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 33 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:45.801815 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55c231d5fd5471285b2d7879a38f38b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 35 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:45.824339 17755 raft_consensus.cc:2410] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 for term 35 because replica has last-logged OpId of term: 34 index: 20, which is greater than that of the candidate, which has last-logged OpId of term: 32 index: 19.
I20260709 12:25:45.825352 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 32 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:45.826746 17755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 29 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:45.804711 17760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 37 candidate_status { last_received { term: 36 index: 25 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:45.792232 29081 raft_consensus.cc:515] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:45.808311 29023 leader_election.cc:290] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 33 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:45.809888 17757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 32 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:25:45.783913 29049 consensus_meta.cc:287] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b: Time spent flushing consensus metadata: real 0.651s	user 0.011s	sys 0.000s
I20260709 12:25:45.816000 29044 raft_consensus.cc:493] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:25:45.828907 18362 raft_consensus.cc:3060] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:25:45.829676 17652 leader_election.cc:304] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:45.833084 17756 raft_consensus.cc:2393] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 33: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:25:45.841102 17762 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38786 (request call id 4461) took 1709 ms. Trace:
I20260709 12:25:45.841854 29081 leader_election.cc:290] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:45.845606 29077 raft_consensus.cc:515] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 36 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:45.832559 28637 raft_consensus.cc:493] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:45.850281 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 32 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:25:45.857781 29047 log.cc:927] Time spent T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 1.205s	user 0.000s	sys 0.012s
I20260709 12:25:45.859202 28992 leader_election.cc:290] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 31 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:45.859571 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 34 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:25:45.874091 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 44535.7us
I20260709 12:25:45.864239 17762 rpcz_store.cc:276] 0709 12:25:44.132108 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:44.132225 (+   117us) service_pool.cc:224] Handling call
0709 12:25:44.152311 (+ 20086us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:44.152698 (+   387us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 12:25:44.152728 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:44.153210 (+   482us) log.cc:844] Serialized 26 byte log entry
0709 12:25:44.265853 (+112643us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:25:44.265896 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:44.265959 (+    63us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:45.766791 (+1500832us) spinlock_profiling.cc:243] Waited 6.6 ms on lock 0x7b4400022b4c. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b47459c85 00007f7b4745a6a2 00007f7b4a1cc1db 00007f7b4a1c7aaf 00007f7b4a1bb86e 00007f7b4a1babb6 00007f7b4a1ba7d8 00007f7b4a1c4221 00007f7b4a1c41d9 00007f7b4a1c4169
0709 12:25:45.840841 (+ 74050us) raft_consensus.cc:1685] finished
0709 12:25:45.840863 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:45.841043 (+   180us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4913,"raft.queue_time_us":391,"raft.run_cpu_time_us":103,"raft.run_wall_time_us":89,"spinlock_wait_cycles":15180032,"thread_start_us":112261,"threads_started":1,"wal-append.queue_time_us":283929}
I20260709 12:25:45.880450 17759 raft_consensus.cc:2468] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 34.
I20260709 12:25:45.867479 17801 leader_election.cc:304] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:45.890751 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 29 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:45.892771 17652 leader_election.cc:304] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:45.891389 17801 leader_election.cc:304] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:45.882293 28917 raft_consensus.cc:2749] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:25:45.897447 28847 raft_consensus.cc:2749] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 34 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:25:45.899567 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 32 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:45.889014 29049 raft_consensus.cc:2749] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:25:45.901063 17763 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38776 (request call id 3674) took 1731 ms. Trace:
I20260709 12:25:45.901433 17763 rpcz_store.cc:276] 0709 12:25:44.169460 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:44.240260 (+ 70800us) service_pool.cc:224] Handling call
0709 12:25:44.240518 (+   258us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:44.860164 (+619646us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:45.900848 (+1040684us) spinlock_profiling.cc:243] Waited 604 ms on lock 0x7b5400217854. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:25:45.901009 (+   161us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1388891392}
I20260709 12:25:45.882992 29044 raft_consensus.cc:515] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:45.897037 28992 raft_consensus.cc:3060] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:25:45.886066 28637 raft_consensus.cc:515] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:45.915234 17652 leader_election.cc:304] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:45.930867 28917 raft_consensus.cc:2749] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:25:45.941864 29044 leader_election.cc:290] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:45.943787 29077 leader_election.cc:290] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 36 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:45.948462 28997 raft_consensus.cc:515] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:45.951215 28997 leader_election.cc:290] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 30 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:45.958088 28637 leader_election.cc:290] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:45.958230 28917 raft_consensus.cc:493] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:45.958952 28917 raft_consensus.cc:515] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:45.959761 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 36 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:45.960510 18371 raft_consensus.cc:3060] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:25:45.967916 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 28 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:45.977082 28792 leader_election.cc:290] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 34 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:45.977890 28792 raft_consensus.cc:493] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:25:45.978489 28792 raft_consensus.cc:515] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:45.996533 17729 leader_election.cc:304] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:45.997375 18371 raft_consensus.cc:2468] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 36.
I20260709 12:25:45.998239 29081 raft_consensus.cc:2804] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 33 FOLLOWER]: Leader election won for term 33
I20260709 12:25:45.998898 29081 raft_consensus.cc:697] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 33 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:25:46.001137 18362 raft_consensus.cc:2468] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 35.
I20260709 12:25:46.002556 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 30 candidate_status { last_received { term: 28 index: 22 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:45.948482 17832 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37994 (request call id 2079) took 1572 ms. Trace:
I20260709 12:25:46.022624 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 31 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:46.023516 18362 raft_consensus.cc:2393] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 31: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:25:46.025085 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 33 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:46.025806 18371 raft_consensus.cc:2393] T a5732e71b9d54f8786b8c511d69943bf P 334a4e1f59944544b3b3ff36616b57da [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0da169ff82674dc1947055ce43de68a3 in current term 33: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:25:46.026541 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7da467817a4ba88aa5797c3f8f493f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 33 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:46.028434 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 32 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:46.029448 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 30 candidate_status { last_received { term: 28 index: 22 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:46.018781 17832 rpcz_store.cc:276] 0709 12:25:44.375644 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:44.375986 (+   342us) service_pool.cc:224] Handling call
0709 12:25:45.100735 (+724749us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:45.100902 (+   167us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:45.100925 (+    23us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:45.100968 (+    43us) raft_consensus.cc:1645] Marking committed up to 22
0709 12:25:45.100996 (+    28us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:45.101047 (+    51us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:45.948253 (+847206us) spinlock_profiling.cc:243] Waited 724 ms on lock 0x7b54000ab4d0. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a61f8ad 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:25:45.948406 (+   153us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1665872128}
I20260709 12:25:46.035861 17654 leader_election.cc:304] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:46.043648 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 34 candidate_status { last_received { term: 33 index: 23 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:46.045542 17729 leader_election.cc:304] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 36 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:46.049387 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 28 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:46.078104 18366 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59886 (request call id 3001) took 1412 ms. Trace:
I20260709 12:25:46.043220 28997 raft_consensus.cc:493] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:46.077411 28956 raft_consensus.cc:2804] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 36 FOLLOWER]: Leader election won for term 36
I20260709 12:25:46.106072 18366 rpcz_store.cc:276] 0709 12:25:44.665530 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:44.665670 (+   140us) service_pool.cc:224] Handling call
0709 12:25:44.665975 (+   305us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:44.696792 (+ 30817us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:44.696823 (+    31us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:44.696864 (+    41us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:25:44.696894 (+    30us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:46.077731 (+1380837us) spinlock_profiling.cc:243] Waited 30.6 ms on lock 0x7b54006c6954. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:25:46.077780 (+    49us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:46.077940 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":70378240}
W20260709 12:25:46.118571 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 82798.4us
I20260709 12:25:46.119380 17654 leader_election.cc:304] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:25:46.116647 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 108361us
I20260709 12:25:46.138491 17834 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37994 (request call id 2091) took 1326 ms. Trace:
W20260709 12:25:46.138463 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 103071us
I20260709 12:25:46.139183 17834 rpcz_store.cc:276] 0709 12:25:44.811940 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:44.812076 (+   136us) service_pool.cc:224] Handling call
0709 12:25:46.138335 (+1326259us) spinlock_profiling.cc:243] Waited 234 ms on lock 0x7b5400089dd4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:25:46.138436 (+   101us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":537214208}
I20260709 12:25:46.114418 28997 raft_consensus.cc:515] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:46.051813 29081 consensus_queue.cc:237] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 29.17, Last appended by leader: 17, Current term: 33, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:46.133906 28992 raft_consensus.cc:2749] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 33 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:25:46.146711 17829 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35040 (request call id 4034) took 2194 ms. Trace:
I20260709 12:25:46.148535 17829 rpcz_store.cc:276] 0709 12:25:43.949395 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:43.949594 (+   199us) service_pool.cc:224] Handling call
0709 12:25:46.144019 (+2194425us) spinlock_profiling.cc:243] Waited 765 ms on lock 0x7b540022b354. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a6242fa 00007f7b4f5e5b17 00007f7b48b606d4 00007f7b48b60662 00007f7b48b60591 00007f7b48b6050c 00007f7b48b5f7c2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:25:46.144153 (+   134us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1758916352}
I20260709 12:25:46.151242 28917 leader_election.cc:290] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:46.152051 28917 raft_consensus.cc:3060] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:46.160132 28792 leader_election.cc:290] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:46.142436 28721 raft_consensus.cc:2749] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
W20260709 12:25:46.168869 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 43556.7us
I20260709 12:25:46.169680 28927 raft_consensus.cc:493] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:25:46.170499 17654 leader_election.cc:304] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:46.174015 28927 raft_consensus.cc:515] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:46.175503 29100 raft_consensus.cc:493] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:46.176424 28927 leader_election.cc:290] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:46.176090 29100 raft_consensus.cc:515] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:46.181350 29100 leader_election.cc:290] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:25:46.197233 29098 log.cc:927] Time spent T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.052s	user 0.000s	sys 0.000s
I20260709 12:25:46.197837 28603 leader_election.cc:290] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:46.197901 28638 raft_consensus.cc:493] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:25:46.198609 28638 raft_consensus.cc:515] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:46.202389 17838 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35042 (request call id 3819) took 1592 ms. Trace:
I20260709 12:25:46.202840 17838 rpcz_store.cc:276] 0709 12:25:44.609836 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:44.610146 (+   310us) service_pool.cc:224] Handling call
0709 12:25:44.626948 (+ 16802us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:45.634866 (+1007918us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:46.202146 (+567280us) spinlock_profiling.cc:243] Waited 1.01 s on lock 0x7b5400334fd4. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:25:46.202325 (+   179us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2316444416}
I20260709 12:25:46.206001 17654 leader_election.cc:304] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 31 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:46.207466 28638 leader_election.cc:290] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:46.208345 28997 leader_election.cc:290] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:46.218829 28917 raft_consensus.cc:2749] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:25:46.220587 17764 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36572 (request call id 2002) took 1603 ms. Trace:
I20260709 12:25:46.225998 17764 rpcz_store.cc:276] 0709 12:25:44.617539 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:44.686145 (+ 68606us) service_pool.cc:224] Handling call
0709 12:25:44.686375 (+   230us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:45.524470 (+838095us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:46.220272 (+695802us) spinlock_profiling.cc:243] Waited 818 ms on lock 0x7b54002b5f54. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a620e01 00007f7b4a61f83e 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6
0709 12:25:46.220517 (+   245us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1881233280}
I20260709 12:25:46.221436 28953 leader_election.cc:290] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 32 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:46.244127 17803 leader_election.cc:304] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:46.248792 29060 raft_consensus.cc:2804] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:25:46.249298 29060 raft_consensus.cc:493] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:25:46.249732 29060 raft_consensus.cc:3060] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Advancing to term 30
W20260709 12:25:46.254096 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 204995us
I20260709 12:25:46.256543 29023 consensus_queue.cc:1048] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 17, Time since last communication: 0.000s
I20260709 12:25:46.258747 28956 raft_consensus.cc:697] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 36 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:25:46.260010 18365 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59886 (request call id 3006) took 1279 ms. Trace:
I20260709 12:25:46.259671 28956 consensus_queue.cc:237] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 35.22, Last appended by leader: 22, Current term: 36, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:46.260465 18365 rpcz_store.cc:276] 0709 12:25:44.980426 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:45.485919 (+505493us) service_pool.cc:224] Handling call
0709 12:25:45.990728 (+504809us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:45.991006 (+   278us) raft_consensus.cc:1534] Early marking committed up to index 19
0709 12:25:45.991346 (+   340us) log.cc:844] Serialized 14 byte log entry
0709 12:25:46.032110 (+ 40764us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:46.032159 (+    49us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:25:46.032205 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:46.032271 (+    66us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:46.259799 (+227528us) spinlock_profiling.cc:243] Waited 504 ms on lock 0x7b5400499650. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a61f8ad 00007f7b4f5e4419 00007f7b48b5bbd4 00007f7b48b5bb62 00007f7b48b5ba91 00007f7b48b5ba0c 00007f7b48b5acc2 00007f7b4895b48c 00007f7b4895a8c6 00007f7b4895a26f
0709 12:25:46.259943 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":1988,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":47,"raft.queue_time_us":109,"raft.run_cpu_time_us":90,"raft.run_wall_time_us":90,"spinlock_wait_cycles":1160099456,"thread_start_us":40327,"threads_started":1,"wal-append.queue_time_us":40712}
I20260709 12:25:46.266011 17729 leader_election.cc:304] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 35 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:46.266398 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 31 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:46.266907 17729 leader_election.cc:304] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 34 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:46.267199 17829 raft_consensus.cc:2468] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 30.
I20260709 12:25:46.268494 28957 raft_consensus.cc:2804] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 35 FOLLOWER]: Leader election won for term 35
I20260709 12:25:46.269265 28957 raft_consensus.cc:697] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 35 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:25:46.269768 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:46.270486 29079 raft_consensus.cc:493] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 33 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:25:46.270500 17832 raft_consensus.cc:2393] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 26: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:25:46.271049 29079 raft_consensus.cc:515] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:46.271651 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 37 candidate_status { last_received { term: 36 index: 24 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:46.272222 17838 raft_consensus.cc:3060] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:25:46.273118 29079 leader_election.cc:290] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:46.268519 29078 raft_consensus.cc:2749] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 34 FOLLOWER]: Leader election lost for term 34. Reason: could not achieve majority
I20260709 12:25:46.276980 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:46.279067 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 31 candidate_status { last_received { term: 30 index: 21 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:46.270174 28957 consensus_queue.cc:237] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 33.20, Last appended by leader: 20, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:46.281121 18362 raft_consensus.cc:3060] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:46.284308 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 31 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:46.286693 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 37 candidate_status { last_received { term: 36 index: 24 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
W20260709 12:25:46.291067 17728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:46.296345 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 31 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:46.296937 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 33 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:46.298121 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 33 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:46.304373 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 25 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:25:46.308422 17573 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet d48b7de191684318aa08f60a7a395387 with cas_config_opid_index 12: ChangeConfig:REMOVE_PEER failed with leader 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 334a4e1f59944544b3b3ff36616b57da is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 28 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:25:46.310379 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 32 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:46.312517 17803 leader_election.cc:304] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:46.322535 18363 raft_consensus.cc:2468] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 30.
W20260709 12:25:46.326683 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 56379us
I20260709 12:25:46.335433 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 37 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:46.326079 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 34 candidate_status { last_received { term: 33 index: 23 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:46.338189 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 34 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:46.338512 17756 raft_consensus.cc:2393] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 34: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:25:46.341228 16501 raft_consensus.cc:2272] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:46.344102 28997 raft_consensus.cc:2749] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Leader election lost for term 31. Reason: could not achieve majority
I20260709 12:25:46.344990 16501 tablet_replica.cc:333] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3: stopping tablet replica
I20260709 12:25:46.346463 16501 raft_consensus.cc:2243] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 31 LEADER]: Raft consensus shutting down.
I20260709 12:25:46.347955 16501 raft_consensus.cc:2272] T 56b500bd1b914ebfb3f19afc8a330665 P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Raft consensus is shut down!
W20260709 12:25:46.348908 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 143006us
I20260709 12:25:46.328094 18333 leader_election.cc:304] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:46.352238 17838 raft_consensus.cc:2468] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 37 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 37.
I20260709 12:25:46.357513 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 32 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:46.357810 17837 raft_consensus.cc:1240] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 37 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 36. Current term is 37. Ops: []
I20260709 12:25:46.358148 17831 raft_consensus.cc:3060] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:25:46.360440 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 37 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:46.351378 18362 raft_consensus.cc:1275] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 24 index: 17. Preceding OpId from leader: term: 26 index: 18. (index mismatch)
I20260709 12:25:46.362802 17838 raft_consensus.cc:1275] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 27 index: 19. Preceding OpId from leader: term: 30 index: 21. (index mismatch)
I20260709 12:25:46.354322 18333 leader_election.cc:304] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:25:46.356599 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 44239.4us
I20260709 12:25:46.365561 17834 raft_consensus.cc:2393] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 37 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 37: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:25:46.367604 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 34 candidate_status { last_received { term: 33 index: 23 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:46.325548 28637 raft_consensus.cc:2749] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:25:46.367761 17729 leader_election.cc:304] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:46.369599 17835 raft_consensus.cc:2468] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 34.
I20260709 12:25:46.367192 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 32 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:46.377825 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 31 candidate_status { last_received { term: 30 index: 21 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:46.380913 29041 raft_consensus.cc:493] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:46.385744 29041 raft_consensus.cc:515] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:46.390396 17803 leader_election.cc:304] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 36 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:25:46.392189 28645 raft_consensus.cc:2804] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 36 FOLLOWER]: Leader election won for term 36
I20260709 12:25:46.392746 28645 raft_consensus.cc:697] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 36 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:25:46.393702 28645 consensus_queue.cc:237] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 35.20, Last appended by leader: 20, Current term: 36, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:46.333204 29066 consensus_queue.cc:1059] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 31.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.042s
I20260709 12:25:46.397373 29079 raft_consensus.cc:3055] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 33 LEADER]: Stepping down as leader of term 33
I20260709 12:25:46.397791 29079 raft_consensus.cc:740] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 33 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:25:46.398856 29079 consensus_queue.cc:260] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 33.22, Last appended by leader: 22, Current term: 33, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:46.364888 28900 raft_consensus.cc:2804] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 36 FOLLOWER]: Leader pre-election won for term 37
I20260709 12:25:46.402519 29079 raft_consensus.cc:3060] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:25:46.402958 28900 raft_consensus.cc:493] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 36 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:25:46.403501 28900 raft_consensus.cc:3060] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:25:46.373116 29072 raft_consensus.cc:2749] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:25:46.381732 28792 raft_consensus.cc:2804] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:25:46.414572 28792 raft_consensus.cc:493] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:25:46.415046 28792 raft_consensus.cc:3060] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:46.416538 29104 raft_consensus.cc:493] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:46.417352 29104 raft_consensus.cc:515] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:46.436717 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 57445.5us
I20260709 12:25:46.438407 29078 consensus_queue.cc:1048] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20260709 12:25:46.440799 18362 raft_consensus.cc:1240] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 37 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 36. Current term is 37. Ops: []
I20260709 12:25:46.440954 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 35 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:46.441538 17832 raft_consensus.cc:3060] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:25:46.442124 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:25:46.443378 17802 leader_election.cc:336] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:46.443759 17833 raft_consensus.cc:1275] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 24 index: 17. Preceding OpId from leader: term: 26 index: 18. (index mismatch)
I20260709 12:25:46.397384 29060 raft_consensus.cc:515] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:46.378506 28956 raft_consensus.cc:493] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
W20260709 12:25:46.444402 17802 leader_election.cc:336] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:46.444506 28956 raft_consensus.cc:515] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:46.447022 28956 leader_election.cc:290] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:46.428149 17829 raft_consensus.cc:1275] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 34 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 32 index: 19. Preceding OpId from leader: term: 34 index: 20. (index mismatch)
I20260709 12:25:46.376165 29001 raft_consensus.cc:493] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:25:46.448892 29001 raft_consensus.cc:515] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:46.450783 17832 raft_consensus.cc:2468] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 35.
I20260709 12:25:46.462409 28956 consensus_queue.cc:1059] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 36.24, Next index: 25, Last known committed idx: 24, Time since last communication: 0.000s
I20260709 12:25:46.463799 28706 raft_consensus.cc:3055] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 36 LEADER]: Stepping down as leader of term 36
I20260709 12:25:46.464212 28706 raft_consensus.cc:740] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 36 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:25:46.465117 28706 consensus_queue.cc:260] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 36.24, Last appended by leader: 24, Current term: 36, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:46.466179 28706 raft_consensus.cc:3060] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:25:46.479782 29072 raft_consensus.cc:493] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:25:46.480491 29072 raft_consensus.cc:515] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:46.431746 17652 leader_election.cc:304] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:46.445006 17802 leader_election.cc:304] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:46.451449 29001 leader_election.cc:290] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:25:46.429823 28960 raft_consensus.cc:493] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:46.429234 28980 raft_consensus.cc:493] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:46.452833 17837 raft_consensus.cc:1240] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 29. Current term is 30. Ops: []
I20260709 12:25:46.457129 16501 tablet_replica.cc:333] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3: stopping tablet replica
I20260709 12:25:46.484687 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 29 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:46.438097 29104 leader_election.cc:290] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:25:46.489848 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 58196.3us
I20260709 12:25:46.486120 17831 raft_consensus.cc:2410] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb for term 32 because replica has last-logged OpId of term: 31 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 29 index: 16.
I20260709 12:25:46.508764 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 32 candidate_status { last_received { term: 28 index: 22 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:46.493222 28891 raft_consensus.cc:2749] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:25:46.497013 28992 raft_consensus.cc:2749] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
W20260709 12:25:46.526711 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 172451us
I20260709 12:25:46.461258 28900 raft_consensus.cc:515] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 37 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:46.504251 28980 raft_consensus.cc:515] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:46.534282 17764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 33 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:46.535298 28980 leader_election.cc:290] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:46.503996 28960 raft_consensus.cc:515] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:46.542536 28960 leader_election.cc:290] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:46.543623 29041 leader_election.cc:290] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:46.544839 29072 leader_election.cc:290] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:46.546603 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 31 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:46.551479 29060 leader_election.cc:290] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 30 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:46.553194 28900 leader_election.cc:290] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 37 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:46.555303 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 25 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:46.559966 28792 raft_consensus.cc:515] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:46.561570 17756 raft_consensus.cc:1240] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Rejecting Update request from peer 0da169ff82674dc1947055ce43de68a3 for earlier term 30. Current term is 31. Ops: []
I20260709 12:25:46.562882 16501 raft_consensus.cc:2243] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:46.563966 16501 raft_consensus.cc:2272] T 12b1fcd35e12402e8d6df7504c578e01 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:46.572161 28706 raft_consensus.cc:493] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:46.572803 28706 raft_consensus.cc:515] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:46.572325 16501 tablet_replica.cc:333] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3: stopping tablet replica
I20260709 12:25:46.576148 16501 raft_consensus.cc:2243] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 32 LEADER]: Raft consensus shutting down.
I20260709 12:25:46.576993 28706 leader_election.cc:290] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:46.578029 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:46.578701 18364 raft_consensus.cc:2468] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 25.
I20260709 12:25:46.579900 17729 leader_election.cc:304] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:46.581151 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 32 candidate_status { last_received { term: 30 index: 21 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:46.581591 28706 raft_consensus.cc:2804] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:25:46.582067 28706 raft_consensus.cc:493] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:46.582405 28706 raft_consensus.cc:3060] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:46.584004 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 30 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:46.578058 16501 raft_consensus.cc:2272] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 [term 32 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:46.584585 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 32 candidate_status { last_received { term: 28 index: 22 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:46.585213 28960 consensus_queue.cc:1059] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 30.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.001s
I20260709 12:25:46.522358 29104 consensus_queue.cc:1048] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.064s
I20260709 12:25:46.595504 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 27 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:46.597853 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 32 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:46.601678 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69716f6d5e7549418fdd8d54d7b722ce" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 32 candidate_status { last_received { term: 30 index: 21 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:46.603062 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
W20260709 12:25:46.603659 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 117761us
I20260709 12:25:46.612310 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 27 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:46.595521 28638 raft_consensus.cc:2804] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:25:46.618968 18368 raft_consensus.cc:1275] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 28 index: 21. Preceding OpId from leader: term: 31 index: 22. (index mismatch)
I20260709 12:25:46.621480 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5dabc1347544bd8b4e81fa5903d4a8b" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 32 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:46.621582 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:46.625728 17654 leader_election.cc:304] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:46.623811 18333 leader_election.cc:304] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:25:46.626963 17802 leader_election.cc:336] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:46.628994 17802 leader_election.cc:304] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
W20260709 12:25:46.630185 17802 leader_election.cc:336] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:46.630609 17802 leader_election.cc:304] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:25:46.632814 17802 leader_election.cc:336] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:46.633252 17802 leader_election.cc:304] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:46.634536 28955 raft_consensus.cc:2749] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
W20260709 12:25:46.625495 18334 consensus_peers.cc:597] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:25:46.622591 28960 raft_consensus.cc:493] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:46.638581 18333 leader_election.cc:304] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 37 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:46.733896 28927 raft_consensus.cc:2804] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 37 FOLLOWER]: Leader election won for term 37
I20260709 12:25:46.734489 28927 raft_consensus.cc:697] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 37 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:25:46.735471 28927 consensus_queue.cc:237] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 36.24, Last appended by leader: 24, Current term: 37, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:46.666640 17761 raft_consensus.cc:1275] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 34 FOLLOWER]: Refusing update from remote peer 0da169ff82674dc1947055ce43de68a3: Log matching property violated. Preceding OpId in replica: term: 32 index: 19. Preceding OpId from leader: term: 34 index: 20. (index mismatch)
I20260709 12:25:46.672734 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 31 candidate_status { last_received { term: 30 index: 21 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
W20260709 12:25:46.732019 17802 leader_election.cc:336] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 31 election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:46.681104 18365 raft_consensus.cc:1240] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 37 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 36. Current term is 37. Ops: []
W20260709 12:25:46.743168 17802 consensus_peers.cc:597] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
W20260709 12:25:46.742399 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 103905us
I20260709 12:25:46.633535 28645 raft_consensus.cc:2749] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:25:46.711733 28997 raft_consensus.cc:3055] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:25:46.756759 28997 raft_consensus.cc:740] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 30 LEADER]: Becoming Follower/Learner. State: Replica: 0da169ff82674dc1947055ce43de68a3, State: Running, Role: LEADER
I20260709 12:25:46.639285 29106 raft_consensus.cc:2749] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
I20260709 12:25:46.744189 29103 consensus_queue.cc:1048] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.001s
I20260709 12:25:46.758026 28997 consensus_queue.cc:260] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 30.21, Last appended by leader: 21, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:46.639449 29072 raft_consensus.cc:2749] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
W20260709 12:25:46.717368 29108 log.cc:927] Time spent T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.081s	user 0.000s	sys 0.003s
I20260709 12:25:46.667661 28792 leader_election.cc:290] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 31 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:46.659242 28638 raft_consensus.cc:493] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:25:46.760187 28638 raft_consensus.cc:3060] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:46.628796 29079 raft_consensus.cc:2749] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:25:46.657374 16501 tablet_replica.cc:333] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3: stopping tablet replica
W20260709 12:25:46.716866 29115 log.cc:927] Time spent T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.085s	user 0.000s	sys 0.001s
I20260709 12:25:46.651816 17654 leader_election.cc:304] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:46.748219 18333 leader_election.cc:304] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:46.667037 28706 raft_consensus.cc:515] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:46.780381 28900 raft_consensus.cc:2804] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:25:46.780869 16501 raft_consensus.cc:2243] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Raft consensus shutting down.
W20260709 12:25:46.747184 17802 leader_election.cc:336] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 36 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:25:46.781697 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 129982us
W20260709 12:25:46.782521 17802 leader_election.cc:336] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:46.783010 17654 leader_election.cc:304] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:46.783042 17802 leader_election.cc:304] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:46.780529 18333 leader_election.cc:304] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:46.785331 16501 raft_consensus.cc:2272] T 1a698a75c7954dd7b24e7889cb19ae35 P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:46.787138 17654 leader_election.cc:304] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:46.759725 28997 raft_consensus.cc:3060] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:46.788623 17654 leader_election.cc:304] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:46.789960 29060 raft_consensus.cc:2749] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 32 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:25:46.797675 28637 raft_consensus.cc:2749] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:25:46.801838 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 37 candidate_status { last_received { term: 36 index: 25 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:46.800905 17727 leader_election.cc:304] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 34 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:46.811499 17654 leader_election.cc:304] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:46.812805 17803 leader_election.cc:304] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:46.817570 28706 leader_election.cc:290] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 26 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:46.818665 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:46.818759 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:46.819307 18361 raft_consensus.cc:3060] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:46.819658 17834 raft_consensus.cc:2468] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 25.
I20260709 12:25:46.802521 29000 raft_consensus.cc:2749] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:25:46.826638 17834 raft_consensus.cc:1275] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 33 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 29 index: 17. Preceding OpId from leader: term: 33 index: 18. (index mismatch)
I20260709 12:25:46.826640 16501 tablet_replica.cc:333] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3: stopping tablet replica
I20260709 12:25:46.833278 18361 raft_consensus.cc:2468] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 26.
I20260709 12:25:46.833281 17834 raft_consensus.cc:1275] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 35 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 33 index: 20. Preceding OpId from leader: term: 35 index: 21. (index mismatch)
I20260709 12:25:46.785744 29106 raft_consensus.cc:2749] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:25:46.752343 28960 raft_consensus.cc:515] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:46.839991 16501 raft_consensus.cc:2243] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 37 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:46.841500 28960 leader_election.cc:290] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:46.780871 28900 raft_consensus.cc:493] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 33 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:25:46.842470 28900 raft_consensus.cc:3060] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:25:46.842972 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 38 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:46.843839 17837 raft_consensus.cc:2468] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 37.
I20260709 12:25:46.761088 29019 consensus_queue.cc:1059] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 36.25, Next index: 26, Last known committed idx: 25, Time since last communication: 0.050s
I20260709 12:25:46.849884 28903 raft_consensus.cc:2749] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 34 FOLLOWER]: Leader election lost for term 34. Reason: could not achieve majority
I20260709 12:25:46.838935 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:46.852717 17834 raft_consensus.cc:3060] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:46.857270 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 37 candidate_status { last_received { term: 36 index: 25 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:46.789709 28721 raft_consensus.cc:2749] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:25:46.872272 29081 consensus_queue.cc:1048] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:25:46.875168 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 38 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:46.875833 17756 raft_consensus.cc:2468] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da169ff82674dc1947055ce43de68a3 in term 36.
I20260709 12:25:46.852202 28992 raft_consensus.cc:2749] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:25:46.854691 28965 raft_consensus.cc:2749] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:25:46.882414 17652 leader_election.cc:304] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:25:46.881385 18365 raft_consensus.cc:1275] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 36 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 35 index: 22. Preceding OpId from leader: term: 36 index: 23. (index mismatch)
W20260709 12:25:46.880854 17654 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 69436.9us
I20260709 12:25:46.883671 18361 raft_consensus.cc:1275] T a5732e71b9d54f8786b8c511d69943bf P 334a4e1f59944544b3b3ff36616b57da [term 33 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 29 index: 17. Preceding OpId from leader: term: 33 index: 18. (index mismatch)
I20260709 12:25:46.878938 18333 leader_election.cc:304] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 37 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:46.889091 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 30 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:46.896467 16501 raft_consensus.cc:2272] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 37 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:46.901072 17729 leader_election.cc:304] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:46.901436 29103 raft_consensus.cc:493] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:46.859161 29041 consensus_queue.cc:1048] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.042s
I20260709 12:25:46.884526 28960 raft_consensus.cc:2749] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:25:46.884910 28721 raft_consensus.cc:493] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:46.905032 28927 raft_consensus.cc:2749] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 37 FOLLOWER]: Leader election lost for term 37. Reason: could not achieve majority
I20260709 12:25:46.904910 29106 raft_consensus.cc:2706] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 [term 37 FOLLOWER]: Received pre-election callback for term 38 while not running: Illegal state: RaftConsensus is not running: State = Stopped
I20260709 12:25:46.906287 28637 raft_consensus.cc:493] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:25:46.910439 29003 raft_consensus.cc:2804] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:25:46.933040 17578 catalog_manager.cc:5697] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 reported cstate change: term changed from 25 to 30, leader changed from e063541542a94969898868bf6eb5ce1b (127.16.29.67) to 0da169ff82674dc1947055ce43de68a3 (127.16.29.65). New cstate: current_term: 30 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: UNKNOWN } } }
W20260709 12:25:46.935972 17580 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35958 (request call id 414) took 127993 ms (client timeout 15000 ms). Trace:
W20260709 12:25:46.936488 17580 rpcz_store.cc:269] 0709 12:23:38.889344 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:38.889574 (+   230us) service_pool.cc:224] Handling call
0709 12:25:46.866746 (+127977172us) inbound_call.cc:177] Queueing success response
Metrics: {"reported_tablets":45}
I20260709 12:25:46.935781 17578 catalog_manager.cc:5697] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 reported cstate change: term changed from 12 to 29. New cstate: current_term: 29 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:25:46.915228 29078 consensus_queue.cc:1048] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.032s
I20260709 12:25:46.909512 29103 raft_consensus.cc:515] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:46.985821 17834 raft_consensus.cc:2468] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 26.
I20260709 12:25:46.945815 29003 raft_consensus.cc:697] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:25:46.928541 28721 raft_consensus.cc:515] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:46.992156 29044 consensus_queue.cc:1048] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.012s
I20260709 12:25:46.994454 17578 catalog_manager.cc:5697] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 reported cstate change: term changed from 31 to 35. New cstate: current_term: 35 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20260709 12:25:47.020268 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 205970us
W20260709 12:25:47.020268 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 153226us
I20260709 12:25:46.942811 28637 raft_consensus.cc:515] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:47.025206 17803 leader_election.cc:304] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:47.032191 17803 leader_election.cc:304] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:47.049000 28637 leader_election.cc:290] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:47.037770 29009 raft_consensus.cc:2749] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
I20260709 12:25:47.038247 28965 raft_consensus.cc:2749] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:25:47.039690 17578 catalog_manager.cc:5697] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 reported cstate change: term changed from 25 to 28. New cstate: current_term: 28 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:25:47.067045 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 29 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:47.074039 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 29 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:47.040158 28638 raft_consensus.cc:515] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:47.076850 17578 catalog_manager.cc:5697] T 13a48228b0d44d3ea21f474203dd02e3 P 0da169ff82674dc1947055ce43de68a3 reported cstate change: term changed from 20 to 34, leader changed from e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66) to 0da169ff82674dc1947055ce43de68a3 (127.16.29.65). New cstate: current_term: 34 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: UNKNOWN } } }
I20260709 12:25:47.081362 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 33 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:47.062616 29019 consensus_queue.cc:1059] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: INVALID_TERM, Last received: 29.18, Next index: 19, Last known committed idx: 16, Time since last communication: 0.026s
I20260709 12:25:47.089010 18365 raft_consensus.cc:1275] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 35 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 31 index: 19. Preceding OpId from leader: term: 35 index: 21. (index mismatch)
W20260709 12:25:47.089686 18334 leader_election.cc:336] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:47.090200 18334 leader_election.cc:304] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:47.080623 28903 consensus_queue.cc:1048] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.011s
I20260709 12:25:47.092237 29000 raft_consensus.cc:2749] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
W20260709 12:25:47.092329 18334 leader_election.cc:336] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:47.092864 18334 leader_election.cc:304] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:47.080901 28957 consensus_queue.cc:1048] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.013s
I20260709 12:25:47.094005 29000 raft_consensus.cc:2749] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
W20260709 12:25:47.097298 18334 leader_election.cc:336] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:47.097770 18334 leader_election.cc:304] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
W20260709 12:25:47.100214 18334 leader_election.cc:336] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:47.102770 29000 raft_consensus.cc:2749] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
W20260709 12:25:47.103484 17728 leader_election.cc:336] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 33 election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:47.104763 29103 leader_election.cc:290] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:25:47.105815 29120 log.cc:927] Time spent T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.141s	user 0.002s	sys 0.000s
W20260709 12:25:47.104799 17728 leader_election.cc:336] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 32 election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:47.106621 17728 leader_election.cc:304] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:47.055872 29123 consensus_queue.cc:1048] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.044s
W20260709 12:25:47.107769 18334 leader_election.cc:336] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 33 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:47.108266 18334 leader_election.cc:304] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:47.056099 29003 consensus_queue.cc:237] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 24.17, Last appended by leader: 17, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:47.112604 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "154e6b769a8d476ba26f7fe3a521fc8d" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 33 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:47.114563 28927 raft_consensus.cc:493] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:25:47.115103 28927 raft_consensus.cc:515] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:47.116186 29000 raft_consensus.cc:2749] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:25:47.117780 28927 leader_election.cc:290] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:47.118490 17654 leader_election.cc:304] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:47.119305 29103 raft_consensus.cc:2749] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:25:47.119920 29081 raft_consensus.cc:3055] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 36 LEADER]: Stepping down as leader of term 36
I20260709 12:25:47.120317 29081 raft_consensus.cc:740] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 36 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:25:47.121148 29081 consensus_queue.cc:260] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 36.25, Last appended by leader: 25, Current term: 36, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:47.122298 29081 raft_consensus.cc:3060] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:25:47.130704 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 29 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:47.132361 17654 leader_election.cc:304] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:47.132423 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:25:47.103566 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 239901us
I20260709 12:25:47.138574 18363 raft_consensus.cc:1240] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 30. Current term is 31. Ops: []
W20260709 12:25:47.144520 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 52630.5us
I20260709 12:25:47.158057 17652 leader_election.cc:304] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:25:47.135703 17802 consensus_peers.cc:597] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
W20260709 12:25:47.170675 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 65907.8us
W20260709 12:25:47.171330 17728 leader_election.cc:336] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 37 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:47.083669 28721 leader_election.cc:290] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:47.171684 17728 leader_election.cc:304] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
W20260709 12:25:47.171636 17802 leader_election.cc:336] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 36 election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:25:47.173346 17802 consensus_peers.cc:597] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:25:47.177695 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 65973.1us
I20260709 12:25:47.134034 29103 raft_consensus.cc:2749] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:25:47.184769 29104 raft_consensus.cc:2749] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
W20260709 12:25:47.187088 17728 leader_election.cc:336] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 36 election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:47.187718 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 37 candidate_status { last_received { term: 35 index: 21 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:47.188321 28956 raft_consensus.cc:3060] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:25:47.189146 28638 leader_election.cc:290] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 31 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:47.197345 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 31 candidate_status { last_received { term: 30 index: 21 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:47.198220 17834 raft_consensus.cc:3060] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:47.201561 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3418610c3e1a4e28824db2b86acff690" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 37 candidate_status { last_received { term: 35 index: 21 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:25:47.202767 17728 leader_election.cc:336] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 37 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:47.110589 17578 catalog_manager.cc:5697] T 22fa38faae58422e9cecc99973cab688 P 0da169ff82674dc1947055ce43de68a3 reported cstate change: term changed from 25 to 29. New cstate: current_term: 29 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: HEALTHY } } }
I20260709 12:25:47.210074 17728 leader_election.cc:304] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:47.182691 29123 raft_consensus.cc:2749] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
I20260709 12:25:47.185266 28721 raft_consensus.cc:493] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:47.217743 17578 catalog_manager.cc:5697] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 reported cstate change: term changed from 22 to 28, leader changed from 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) to e063541542a94969898868bf6eb5ce1b (127.16.29.67). New cstate: current_term: 28 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:25:47.219823 17834 raft_consensus.cc:2468] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 31.
I20260709 12:25:47.219286 17578 catalog_manager.cc:5697] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 reported cstate change: term changed from 27 to 30, leader changed from e063541542a94969898868bf6eb5ce1b (127.16.29.67) to 0da169ff82674dc1947055ce43de68a3 (127.16.29.65). New cstate: current_term: 30 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: UNKNOWN } } }
I20260709 12:25:47.214920 16501 tablet_replica.cc:333] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3: stopping tablet replica
I20260709 12:25:47.165369 28900 raft_consensus.cc:515] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 34 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:47.226132 18334 leader_election.cc:336] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:47.220990 28792 consensus_queue.cc:1059] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: INVALID_TERM, Last received: 30.21, Next index: 22, Last known committed idx: 17, Time since last communication: 0.015s
I20260709 12:25:47.232949 28706 raft_consensus.cc:2749] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:25:47.208776 28957 consensus_queue.cc:1048] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 19, Time since last communication: 0.026s
I20260709 12:25:47.234798 16501 raft_consensus.cc:2243] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 34 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:47.239475 16501 raft_consensus.cc:2272] T e55c231d5fd5471285b2d7879a38f38b P 0da169ff82674dc1947055ce43de68a3 [term 34 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:47.235720 18334 leader_election.cc:304] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:47.242081 28927 raft_consensus.cc:2749] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
W20260709 12:25:47.242904 18334 consensus_peers.cc:597] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:25:47.244203 17831 raft_consensus.cc:1240] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 30. Current term is 31. Ops: [29.20-30.21]
I20260709 12:25:47.245010 16501 tablet_replica.cc:333] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3: stopping tablet replica
I20260709 12:25:47.245633 28956 raft_consensus.cc:2749] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 37 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:25:47.246553 17578 catalog_manager.cc:5697] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 reported cstate change: term changed from 22 to 29, leader changed from 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) to e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66). New cstate: current_term: 29 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:25:47.249221 17578 catalog_manager.cc:5697] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 reported cstate change: term changed from 24 to 27. New cstate: current_term: 27 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:25:47.252264 29123 raft_consensus.cc:3055] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:25:47.252736 29123 raft_consensus.cc:740] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 30 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:25:47.253604 29123 consensus_queue.cc:260] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 30.21, Last appended by leader: 21, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:47.254966 29123 raft_consensus.cc:3060] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Advancing to term 31
W20260709 12:25:47.255712 17802 leader_election.cc:336] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:47.256142 28308 raft_consensus.cc:493] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:47.256683 28308 raft_consensus.cc:515] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:47.253413 17578 catalog_manager.cc:5697] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 reported cstate change: term changed from 26 to 28, leader changed from 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) to e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66). New cstate: current_term: 28 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:25:47.260097 28308 leader_election.cc:290] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:47.259824 17578 catalog_manager.cc:5697] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 reported cstate change: term changed from 17 to 34, leader changed from e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66) to e063541542a94969898868bf6eb5ce1b (127.16.29.67). New cstate: current_term: 34 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } }
I20260709 12:25:47.267311 17578 catalog_manager.cc:5697] T 10f96c39fba3496696f64385105bfeaa P 0da169ff82674dc1947055ce43de68a3 reported cstate change: term changed from 22 to 30. New cstate: current_term: 30 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20260709 12:25:47.275811 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 63427.2us
W20260709 12:25:47.291548 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 94838.8us
I20260709 12:25:47.265617 28900 leader_election.cc:290] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 34 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:47.221692 28721 raft_consensus.cc:515] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:47.307898 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 31 candidate_status { last_received { term: 30 index: 21 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:47.302300 17578 catalog_manager.cc:5697] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 reported cstate change: term changed from 31 to 32, leader changed from 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) to 334a4e1f59944544b3b3ff36616b57da (127.16.29.68). New cstate: current_term: 32 leader_uuid: "334a4e1f59944544b3b3ff36616b57da" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:25:47.311221 17578 catalog_manager.cc:5697] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 reported cstate change: term changed from 18 to 25. New cstate: current_term: 25 leader_uuid: "334a4e1f59944544b3b3ff36616b57da" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:25:47.330739 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 34 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:25:47.330446 17802 consensus_peers.cc:597] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:25:47.333411 17578 catalog_manager.cc:5697] T 8a208e793881438b975b83755dc05cfa P 0da169ff82674dc1947055ce43de68a3 reported cstate change: term changed from 25 to 26, leader changed from 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) to 0da169ff82674dc1947055ce43de68a3 (127.16.29.65). New cstate: current_term: 26 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 12:25:47.342494 17728 leader_election.cc:336] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:47.343026 17728 leader_election.cc:304] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
W20260709 12:25:47.344201 17728 consensus_peers.cc:597] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
W20260709 12:25:47.344689 17728 leader_election.cc:336] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:47.345052 17728 leader_election.cc:304] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:47.347095 16501 raft_consensus.cc:2243] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 27 LEADER]: Raft consensus shutting down.
W20260709 12:25:47.353499 17728 leader_election.cc:336] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:25:47.355744 17728 leader_election.cc:336] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:47.357376 17728 leader_election.cc:304] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:25:47.359691 17578 catalog_manager.cc:5697] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 reported cstate change: term changed from 18 to 34, leader changed from e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66) to e063541542a94969898868bf6eb5ce1b (127.16.29.67). New cstate: current_term: 34 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } }
I20260709 12:25:47.381902 28706 raft_consensus.cc:2749] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:25:47.346405 28721 leader_election.cc:290] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:25:47.386610 17652 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 85656.7us
I20260709 12:25:47.388769 16501 raft_consensus.cc:2272] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:47.372325 28792 raft_consensus.cc:2749] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
W20260709 12:25:47.387094 28997 consensus_meta.cc:287] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3: Time spent flushing consensus metadata: real 0.575s	user 0.008s	sys 0.003s
I20260709 12:25:47.392735 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 36 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:47.393620 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 36 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:47.374080 29130 raft_consensus.cc:2749] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:25:47.391290 17578 catalog_manager.cc:5697] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 reported cstate change: term changed from 24 to 27, leader changed from e063541542a94969898868bf6eb5ce1b (127.16.29.67) to 0da169ff82674dc1947055ce43de68a3 (127.16.29.65). New cstate: current_term: 27 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:25:47.400580 17578 catalog_manager.cc:5697] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 reported cstate change: term changed from 23 to 26, leader changed from e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66) to 334a4e1f59944544b3b3ff36616b57da (127.16.29.68). New cstate: current_term: 26 leader_uuid: "334a4e1f59944544b3b3ff36616b57da" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:25:47.404464 17578 catalog_manager.cc:5697] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 reported cstate change: term changed from 23 to 26, leader changed from e063541542a94969898868bf6eb5ce1b (127.16.29.67) to 0da169ff82674dc1947055ce43de68a3 (127.16.29.65). New cstate: current_term: 26 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:25:47.406064 17578 catalog_manager.cc:5697] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 reported cstate change: term changed from 26 to 27, leader changed from e063541542a94969898868bf6eb5ce1b (127.16.29.67) to 0da169ff82674dc1947055ce43de68a3 (127.16.29.65). New cstate: current_term: 27 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:25:47.407721 17578 catalog_manager.cc:5697] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 reported cstate change: term changed from 26 to 28. New cstate: current_term: 28 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:25:47.419456 17652 leader_election.cc:304] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:47.420503 17578 catalog_manager.cc:5697] T 7cb3768d215e409fb9f5baf18d124442 P 0da169ff82674dc1947055ce43de68a3 reported cstate change: term changed from 21 to 25. New cstate: current_term: 25 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:25:47.421047 29136 raft_consensus.cc:2749] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:25:47.421864 17578 catalog_manager.cc:5697] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 reported cstate change: term changed from 17 to 20. New cstate: current_term: 20 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } }
I20260709 12:25:47.423238 17578 catalog_manager.cc:5697] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 reported cstate change: term changed from 22 to 28, leader changed from e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66) to 0da169ff82674dc1947055ce43de68a3 (127.16.29.65). New cstate: current_term: 28 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:25:47.425395 17578 catalog_manager.cc:5697] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 reported cstate change: term changed from 21 to 22, leader changed from e063541542a94969898868bf6eb5ce1b (127.16.29.67) to 0da169ff82674dc1947055ce43de68a3 (127.16.29.65). New cstate: current_term: 22 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:25:47.427480 17652 leader_election.cc:304] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:47.430435 29136 raft_consensus.cc:2749] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
W20260709 12:25:47.455745 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 246575us
I20260709 12:25:47.461369 17835 raft_consensus.cc:1275] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 37 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 36 index: 24. Preceding OpId from leader: term: 37 index: 25. (index mismatch)
I20260709 12:25:47.468763 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 34 candidate_status { last_received { term: 33 index: 23 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:47.470935 17761 raft_consensus.cc:2393] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 34 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 34: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:25:47.465394 16501 tablet_replica.cc:333] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3: stopping tablet replica
I20260709 12:25:47.475579 28903 raft_consensus.cc:3055] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:25:47.486971 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 34 candidate_status { last_received { term: 33 index: 23 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:47.492652 17763 raft_consensus.cc:1275] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 37 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 36 index: 24. Preceding OpId from leader: term: 37 index: 25. (index mismatch)
I20260709 12:25:47.491814 28903 raft_consensus.cc:740] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 29 LEADER]: Becoming Follower/Learner. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
W20260709 12:25:47.520569 17802 consensus_peers.cc:597] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:25:47.522878 29001 raft_consensus.cc:493] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:25:47.523805 29001 raft_consensus.cc:515] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:47.521327 17802 leader_election.cc:336] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 34 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:47.525938 17802 leader_election.cc:304] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:47.535105 17756 raft_consensus.cc:2468] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 31.
I20260709 12:25:47.546767 17834 raft_consensus.cc:1240] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Rejecting Update request from peer e2909cff1da44ad387f96a8ae3aa5afb for earlier term 29. Current term is 30. Ops: []
I20260709 12:25:47.546761 28965 raft_consensus.cc:2749] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:25:47.550056 28903 consensus_queue.cc:260] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 29.18, Last appended by leader: 18, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:47.551604 17728 consensus_peers.cc:597] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:25:47.552266 17728 consensus_peers.cc:597] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:25:47.559680 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 34 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:47.551635 28903 raft_consensus.cc:3060] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:47.561163 18365 raft_consensus.cc:1275] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 36 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 35 index: 20. Preceding OpId from leader: term: 36 index: 21. (index mismatch)
I20260709 12:25:47.566104 29001 leader_election.cc:290] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:25:47.559603 17837 raft_consensus.cc:2468] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 34.
I20260709 12:25:47.599678 18333 leader_election.cc:304] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:25:47.605052 16501 raft_consensus.cc:2243] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 34 LEADER]: Raft consensus shutting down.
W20260709 12:25:47.604319 18334 leader_election.cc:336] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:47.604578 18333 leader_election.cc:304] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:25:47.611960 17573 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet d48b7de191684318aa08f60a7a395387 with cas_config_opid_index 12: ChangeConfig:REMOVE_PEER failed with leader 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 334a4e1f59944544b3b3ff36616b57da is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 28 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:25:47.613575 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 32 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:47.614382 17756 raft_consensus.cc:2468] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 31.
W20260709 12:25:47.620675 17728 leader_election.cc:336] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 31 election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:47.625581 17728 leader_election.cc:304] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 31 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:25:47.621411 28638 raft_consensus.cc:2804] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 34 FOLLOWER]: Leader election won for term 34
I20260709 12:25:47.628306 29127 raft_consensus.cc:493] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:47.628921 28638 raft_consensus.cc:697] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 34 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:25:47.629122 29127 raft_consensus.cc:515] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:47.610028 29079 consensus_queue.cc:1048] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.021s
W20260709 12:25:47.634526 17802 consensus_peers.cc:597] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:25:47.608707 29001 raft_consensus.cc:2804] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:25:47.640851 29076 consensus_queue.cc:1048] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.001s
I20260709 12:25:47.646968 29001 raft_consensus.cc:697] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 31 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:25:47.648851 29127 leader_election.cc:290] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:25:47.635696 17728 consensus_peers.cc:597] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:25:47.654341 17728 consensus_peers.cc:597] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
W20260709 12:25:47.654909 17728 consensus_peers.cc:597] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:25:47.629884 28638 consensus_queue.cc:237] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 33.23, Last appended by leader: 23, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:47.661515 16501 raft_consensus.cc:2272] T 00a02ab39fcd4078ad49697693cf3be8 P 0da169ff82674dc1947055ce43de68a3 [term 34 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:47.658792 29001 consensus_queue.cc:237] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 30.21, Last appended by leader: 21, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:47.662777 18333 leader_election.cc:304] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:25:47.640511 29003 raft_consensus.cc:2749] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Leader election lost for term 31. Reason: could not achieve majority
I20260709 12:25:47.662180 29079 consensus_queue.cc:1048] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.000s
I20260709 12:25:47.668547 29134 raft_consensus.cc:2804] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:25:47.673846 29134 raft_consensus.cc:493] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:25:47.675328 29134 raft_consensus.cc:3060] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Advancing to term 32
W20260709 12:25:47.677083 29149 log.cc:927] Time spent T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.052s	user 0.002s	sys 0.000s
I20260709 12:25:47.684576 16501 tablet_replica.cc:333] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3: stopping tablet replica
I20260709 12:25:47.688341 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 35 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:47.689056 17763 raft_consensus.cc:2468] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 34.
I20260709 12:25:47.689392 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 35 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:47.688271 16501 raft_consensus.cc:2243] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Raft consensus shutting down.
W20260709 12:25:47.693778 17728 consensus_peers.cc:597] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:25:47.695845 16501 raft_consensus.cc:2272] T 6fcb5bdaa4be4d619bb945ffd44090ba P 0da169ff82674dc1947055ce43de68a3 [term 24 FOLLOWER]: Raft consensus is shut down!
W20260709 12:25:47.708699 18334 consensus_peers.cc:597] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
W20260709 12:25:47.700134 17728 consensus_peers.cc:597] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:25:47.726672 16501 tablet_replica.cc:333] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3: stopping tablet replica
I20260709 12:25:47.727727 17834 raft_consensus.cc:1275] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 24 index: 17. Preceding OpId from leader: term: 26 index: 18. (index mismatch)
I20260709 12:25:47.727701 16501 raft_consensus.cc:2243] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Raft consensus shutting down.
W20260709 12:25:47.732178 17728 consensus_peers.cc:597] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
W20260709 12:25:47.731822 17802 consensus_peers.cc:597] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260709 12:25:47.734982 16501 pending_rounds.cc:70] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3: Trying to abort 1 pending ops.
I20260709 12:25:47.735280 16501 pending_rounds.cc:77] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3: Aborting op as it isn't in flight: id { term: 30 index: 21 } timestamp: 7305625345868558336 op_type: NO_OP noop_request { }
I20260709 12:25:47.735718 16501 raft_consensus.cc:2889] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260709 12:25:47.757922 16501 raft_consensus.cc:2272] T 69716f6d5e7549418fdd8d54d7b722ce P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:47.761035 16501 tablet_replica.cc:333] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3: stopping tablet replica
I20260709 12:25:47.762097 16501 raft_consensus.cc:2243] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 35 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:47.762660 16501 raft_consensus.cc:2272] T d11ed0a8f1094765abe028dc60aa8bed P 0da169ff82674dc1947055ce43de68a3 [term 35 FOLLOWER]: Raft consensus is shut down!
W20260709 12:25:47.765492 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 66232.9us
I20260709 12:25:47.766409 18333 leader_election.cc:304] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:25:47.739730 28903 consensus_queue.cc:1048] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:25:47.727471 29134 raft_consensus.cc:515] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 32 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:47.772177 29134 leader_election.cc:290] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 32 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:25:47.777417 29060 raft_consensus.cc:493] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:25:47.778149 29060 raft_consensus.cc:515] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:47.784850 18374 raft_consensus.cc:1275] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 23 index: 16. Preceding OpId from leader: term: 26 index: 18. (index mismatch)
I20260709 12:25:47.787305 28957 consensus_queue.cc:1048] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:25:47.788378 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 34 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:47.794211 28927 raft_consensus.cc:2804] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:25:47.794649 28927 raft_consensus.cc:493] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 34 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:47.794972 28927 raft_consensus.cc:3060] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:25:47.793653 29060 leader_election.cc:290] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:47.801048 17756 raft_consensus.cc:2393] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 34: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:25:47.807006 28997 raft_consensus.cc:493] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:47.807554 28997 raft_consensus.cc:515] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:47.801064 28903 raft_consensus.cc:493] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:25:47.808423 28903 raft_consensus.cc:515] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:47.809741 28997 leader_election.cc:290] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:47.811774 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 34 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:25:47.812855 18334 leader_election.cc:336] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 32 election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:47.811798 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 34 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:47.825382 17652 leader_election.cc:304] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:25:47.826081 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 59795.6us
I20260709 12:25:47.826546 28997 raft_consensus.cc:2749] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:25:47.833582 16501 tablet_replica.cc:333] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3: stopping tablet replica
I20260709 12:25:47.835187 28903 leader_election.cc:290] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:47.839959 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b500bd1b914ebfb3f19afc8a330665" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 32 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:47.840844 28927 raft_consensus.cc:515] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 35 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:47.847478 17728 leader_election.cc:336] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 33 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:25:47.858798 18334 consensus_peers.cc:597] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:25:47.860741 16501 raft_consensus.cc:2243] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:47.868603 16501 pending_rounds.cc:70] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3: Trying to abort 1 pending ops.
I20260709 12:25:47.868885 16501 pending_rounds.cc:77] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3: Aborting op as it isn't in flight: id { term: 28 index: 22 } timestamp: 7305625305410908160 op_type: NO_OP noop_request { }
I20260709 12:25:47.861374 17756 raft_consensus.cc:3060] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:25:47.879588 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 33 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:47.895118 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 34 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:47.895994 18362 raft_consensus.cc:2368] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 34. Current term is 35.
W20260709 12:25:47.901185 18334 consensus_peers.cc:597] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:25:47.903599 17729 leader_election.cc:304] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:25:47.869264 16501 raft_consensus.cc:2889] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260709 12:25:47.936614 16501 raft_consensus.cc:2272] T bd3425997bcf4fe7abc0399fd06e54d9 P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:47.863994 28927 leader_election.cc:290] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 35 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:47.951913 29078 raft_consensus.cc:2749] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
W20260709 12:25:47.951678 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 48177.5us
I20260709 12:25:47.965206 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 35 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:47.965818 17763 raft_consensus.cc:3060] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:25:47.939806 16501 tablet_replica.cc:333] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3: stopping tablet replica
I20260709 12:25:47.967255 16501 raft_consensus.cc:2243] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:47.967922 16501 raft_consensus.cc:2272] T d48b7de191684318aa08f60a7a395387 P 0da169ff82674dc1947055ce43de68a3 [term 28 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:47.976347 17803 leader_election.cc:400] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 34 pre-election: Vote denied by peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577) with higher term. Message: Invalid argument: T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 34. Current term is 35.
I20260709 12:25:47.976822 17803 leader_election.cc:403] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 34 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:25:47.976948 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 35 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:47.977536 17837 raft_consensus.cc:3060] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 33 FOLLOWER]: Advancing to term 35
I20260709 12:25:47.979394 17756 raft_consensus.cc:2468] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 32.
I20260709 12:25:47.986361 17763 raft_consensus.cc:2468] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 35.
I20260709 12:25:47.976446 28891 raft_consensus.cc:493] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 36 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:25:47.988471 28891 raft_consensus.cc:515] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:47.999035 18333 leader_election.cc:304] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:25:48.000272 29134 raft_consensus.cc:2804] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 32 FOLLOWER]: Leader election won for term 32
I20260709 12:25:48.000815 29134 raft_consensus.cc:697] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 32 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:25:48.001754 29134 consensus_queue.cc:237] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 31.22, Last appended by leader: 22, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:48.007040 28891 leader_election.cc:290] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:48.007668 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 37 candidate_status { last_received { term: 36 index: 25 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:48.008610 17761 raft_consensus.cc:2468] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 37.
I20260709 12:25:48.010234 17837 raft_consensus.cc:2468] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 35.
I20260709 12:25:48.014285 29122 raft_consensus.cc:2749] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 35 FOLLOWER]: Leader pre-election lost for term 34. Reason: Vote denied by peer 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577) with higher term. Message: Invalid argument: T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 34. Current term is 35.
I20260709 12:25:48.010353 17801 leader_election.cc:304] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:48.016036 29076 raft_consensus.cc:2804] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 36 FOLLOWER]: Leader pre-election won for term 37
I20260709 12:25:48.016569 29076 raft_consensus.cc:493] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 36 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:25:48.016821 16501 tablet_replica.cc:333] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3: stopping tablet replica
I20260709 12:25:48.017019 29076 raft_consensus.cc:3060] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:25:48.017755 16501 raft_consensus.cc:2243] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:48.018376 16501 raft_consensus.cc:2272] T b5462347eb74482b8cd11beb1d39b8a9 P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:48.008611 18333 leader_election.cc:304] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 35 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:48.013131 29078 raft_consensus.cc:493] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:48.025518 29001 raft_consensus.cc:2804] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 35 FOLLOWER]: Leader election won for term 35
I20260709 12:25:48.025640 29078 raft_consensus.cc:515] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:48.028805 29001 raft_consensus.cc:697] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 35 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:25:48.030021 29078 leader_election.cc:290] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:48.029809 29001 consensus_queue.cc:237] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 33.22, Last appended by leader: 22, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:48.030999 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 35 candidate_status { last_received { term: 33 index: 23 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:48.033717 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 35 candidate_status { last_received { term: 33 index: 23 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:25:48.033726 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 56038.5us
I20260709 12:25:48.037720 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 37 candidate_status { last_received { term: 36 index: 25 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:48.038470 18362 raft_consensus.cc:2393] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 37 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 37: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:25:48.034475 17831 raft_consensus.cc:2468] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 34.
I20260709 12:25:48.041685 17727 leader_election.cc:304] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:25:48.043763 28927 raft_consensus.cc:493] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:25:48.051668 28927 raft_consensus.cc:515] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:48.053787 28927 leader_election.cc:290] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:25:48.059552 18334 consensus_peers.cc:597] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:25:48.056120 29078 raft_consensus.cc:2804] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:25:48.062338 29078 raft_consensus.cc:493] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 34 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:48.062738 29078 raft_consensus.cc:3060] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 34 FOLLOWER]: Advancing to term 35
W20260709 12:25:48.071655 18334 leader_election.cc:336] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:48.076030 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 31 candidate_status { last_received { term: 30 index: 21 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:48.076857 17763 raft_consensus.cc:2393] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 31: Already voted for candidate e2909cff1da44ad387f96a8ae3aa5afb in this term.
I20260709 12:25:48.043918 28637 raft_consensus.cc:493] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:48.078945 17837 raft_consensus.cc:1275] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 34 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 33 index: 23. Preceding OpId from leader: term: 34 index: 24. (index mismatch)
I20260709 12:25:48.079115 18333 leader_election.cc:304] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:48.083789 29079 raft_consensus.cc:3060] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:48.086347 16501 tablet_replica.cc:333] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3: stopping tablet replica
I20260709 12:25:48.094651 16501 raft_consensus.cc:2243] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 32 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:48.095366 16501 raft_consensus.cc:2272] T 19a1253e500942cd8a1a0f568159cfa3 P 0da169ff82674dc1947055ce43de68a3 [term 32 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:48.087267 28638 consensus_queue.cc:1048] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.013s
I20260709 12:25:48.079723 28637 raft_consensus.cc:515] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:48.088127 29076 raft_consensus.cc:515] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 37 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:48.130264 28637 leader_election.cc:290] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:48.139185 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 34 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:48.145471 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 34 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:48.146857 17652 leader_election.cc:304] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:48.125737 16501 tablet_replica.cc:333] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3: stopping tablet replica
I20260709 12:25:48.150732 28997 raft_consensus.cc:2749] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
W20260709 12:25:48.168220 17802 consensus_peers.cc:597] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260709 12:25:48.172302 29079 raft_consensus.cc:2749] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:25:48.177834 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 37 candidate_status { last_received { term: 36 index: 25 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:48.177436 29076 leader_election.cc:290] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 37 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:48.179735 16501 raft_consensus.cc:2243] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 33 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:48.184091 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 37 candidate_status { last_received { term: 36 index: 25 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:48.189468 18362 raft_consensus.cc:2393] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 37 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 37: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:25:48.208477 17761 raft_consensus.cc:2468] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 37 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 37.
I20260709 12:25:48.209806 17801 leader_election.cc:304] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 37 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:25:48.223402 29078 raft_consensus.cc:515] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 35 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:48.225150 17763 raft_consensus.cc:1240] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 35 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 34. Current term is 35. Ops: []
I20260709 12:25:48.214217 29076 raft_consensus.cc:2804] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 37 FOLLOWER]: Leader election won for term 37
I20260709 12:25:48.232479 29076 raft_consensus.cc:697] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 37 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:25:48.235110 16501 raft_consensus.cc:2272] T 44ad5363c5954da4a1bff5857780fd5c P 0da169ff82674dc1947055ce43de68a3 [term 33 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:48.239434 17763 raft_consensus.cc:1275] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 30 index: 21. Preceding OpId from leader: term: 31 index: 22. (index mismatch)
I20260709 12:25:48.243356 29127 consensus_queue.cc:1059] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 0.0, Next index: 24, Last known committed idx: 22, Time since last communication: 0.001s
I20260709 12:25:48.233520 29076 consensus_queue.cc:237] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 36.25, Last appended by leader: 25, Current term: 37, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:48.248248 29078 leader_election.cc:290] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 35 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:48.252369 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 35 candidate_status { last_received { term: 33 index: 23 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:48.258072 28927 raft_consensus.cc:3055] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 34 LEADER]: Stepping down as leader of term 34
I20260709 12:25:48.258555 28927 raft_consensus.cc:740] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 34 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:25:48.251654 29079 consensus_queue.cc:1048] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.009s
I20260709 12:25:48.260558 28997 raft_consensus.cc:493] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:48.259416 28927 consensus_queue.cc:260] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 34.24, Last appended by leader: 24, Current term: 34, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:48.261160 28997 raft_consensus.cc:515] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:48.261672 28927 raft_consensus.cc:3060] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:25:48.263851 28997 leader_election.cc:290] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:48.266825 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:48.273717 16501 tablet_replica.cc:333] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3: stopping tablet replica
I20260709 12:25:48.274865 16501 raft_consensus.cc:2243] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:48.275563 16501 raft_consensus.cc:2272] T ab4d9071484d4dc0998db462cb01a10f P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:48.277302 17763 raft_consensus.cc:1240] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 35 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 34. Current term is 35. Ops: [34.24-34.24]
I20260709 12:25:48.283326 17837 raft_consensus.cc:1275] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 27 index: 19. Preceding OpId from leader: term: 31 index: 22. (index mismatch)
I20260709 12:25:48.292336 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50f9ff8f24a14a079db34b1ba88fa278" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:48.298738 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 35 candidate_status { last_received { term: 33 index: 23 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:48.307474 17652 leader_election.cc:304] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:48.309324 28997 raft_consensus.cc:2749] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
W20260709 12:25:48.310774 17728 consensus_peers.cc:597] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:25:48.312073 17727 leader_election.cc:304] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 35 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:48.311357 29001 consensus_queue.cc:1048] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 17, Time since last communication: 0.004s
I20260709 12:25:48.317678 16501 tablet_replica.cc:333] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3: stopping tablet replica
I20260709 12:25:48.320134 17580 catalog_manager.cc:5697] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 35 to 36, leader changed from 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) to e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66). New cstate: current_term: 36 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:25:48.321452 29130 raft_consensus.cc:2749] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 35 FOLLOWER]: Leader election lost for term 35. Reason: could not achieve majority
I20260709 12:25:48.325441 17580 catalog_manager.cc:5697] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 29 to 32. New cstate: current_term: 32 leader_uuid: "334a4e1f59944544b3b3ff36616b57da" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:25:48.344173 17580 catalog_manager.cc:5697] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 26 to 27, leader changed from 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) to e063541542a94969898868bf6eb5ce1b (127.16.29.67). New cstate: current_term: 27 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:25:48.347646 16501 raft_consensus.cc:2243] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:48.346892 17580 catalog_manager.cc:5697] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 26 to 27, leader changed from 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) to 334a4e1f59944544b3b3ff36616b57da (127.16.29.68). New cstate: current_term: 27 leader_uuid: "334a4e1f59944544b3b3ff36616b57da" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:25:48.351334 17580 catalog_manager.cc:5697] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 26 to 31, leader changed from e063541542a94969898868bf6eb5ce1b (127.16.29.67) to e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66). New cstate: current_term: 31 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
W20260709 12:25:48.358660 17802 consensus_peers.cc:597] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:25:48.352926 17580 catalog_manager.cc:5697] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 23 to 26, leader changed from e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66) to 334a4e1f59944544b3b3ff36616b57da (127.16.29.68). New cstate: current_term: 26 leader_uuid: "334a4e1f59944544b3b3ff36616b57da" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 12:25:48.361310 17572 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 66608.2us
I20260709 12:25:48.362875 17580 catalog_manager.cc:5697] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 21 to 24, leader changed from e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66) to 334a4e1f59944544b3b3ff36616b57da (127.16.29.68). New cstate: current_term: 24 leader_uuid: "334a4e1f59944544b3b3ff36616b57da" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:25:48.365602 17580 catalog_manager.cc:5697] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 18 to 23, leader changed from e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66) to 334a4e1f59944544b3b3ff36616b57da (127.16.29.68). New cstate: current_term: 23 leader_uuid: "334a4e1f59944544b3b3ff36616b57da" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:25:48.368489 17580 catalog_manager.cc:5697] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 23 to 28, leader changed from e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66) to 334a4e1f59944544b3b3ff36616b57da (127.16.29.68). New cstate: current_term: 28 leader_uuid: "334a4e1f59944544b3b3ff36616b57da" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:25:48.370427 17580 catalog_manager.cc:5697] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 28 to 31, leader changed from e063541542a94969898868bf6eb5ce1b (127.16.29.67) to e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66). New cstate: current_term: 31 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:25:48.372232 17580 catalog_manager.cc:5697] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 25 to 30, leader changed from e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66) to 334a4e1f59944544b3b3ff36616b57da (127.16.29.68). New cstate: current_term: 30 leader_uuid: "334a4e1f59944544b3b3ff36616b57da" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:25:48.374727 16501 raft_consensus.cc:2272] T 8e417cb980e44a77962d2e3dfd58aad0 P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:48.374269 17580 catalog_manager.cc:5697] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 27 to 28, leader changed from 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) to 334a4e1f59944544b3b3ff36616b57da (127.16.29.68). New cstate: current_term: 28 leader_uuid: "334a4e1f59944544b3b3ff36616b57da" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:25:48.376657 17580 catalog_manager.cc:5697] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 28 to 31, leader changed from e063541542a94969898868bf6eb5ce1b (127.16.29.67) to 334a4e1f59944544b3b3ff36616b57da (127.16.29.68). New cstate: current_term: 31 leader_uuid: "334a4e1f59944544b3b3ff36616b57da" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:25:48.382845 16501 tablet_replica.cc:333] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3: stopping tablet replica
I20260709 12:25:48.383782 17580 catalog_manager.cc:5697] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 28 to 29, leader changed from 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) to e063541542a94969898868bf6eb5ce1b (127.16.29.67). New cstate: current_term: 29 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
W20260709 12:25:48.386189 17578 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35958 (request call id 418) took 21371 ms (client timeout 15000 ms). Trace:
I20260709 12:25:48.386094 17580 catalog_manager.cc:5697] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 31 to 34, leader changed from 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) to e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66). New cstate: current_term: 34 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:25:48.388974 17580 catalog_manager.cc:5697] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 30 to 33, leader changed from 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) to e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66). New cstate: current_term: 33 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:25:48.401687 16501 raft_consensus.cc:2243] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 32 FOLLOWER]: Raft consensus shutting down.
W20260709 12:25:48.389212 17572 catalog_manager.cc:4755] TS 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): ChangeConfig:REMOVE_PEER RPC failed for tablet e5dabc1347544bd8b4e81fa5903d4a8b: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:48.406817 16501 pending_rounds.cc:70] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3: Trying to abort 1 pending ops.
I20260709 12:25:48.407701 16501 pending_rounds.cc:77] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3: Aborting op as it isn't in flight: id { term: 30 index: 22 } timestamp: 7305625305041154048 op_type: NO_OP noop_request { }
I20260709 12:25:48.408046 16501 raft_consensus.cc:2889] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 32 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260709 12:25:48.408385 16501 raft_consensus.cc:2272] T ea7da467817a4ba88aa5797c3f8f493f P 0da169ff82674dc1947055ce43de68a3 [term 32 FOLLOWER]: Raft consensus is shut down!
W20260709 12:25:48.386619 17578 rpcz_store.cc:269] 0709 12:25:27.079826 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:29.372855 (+2293029us) service_pool.cc:224] Handling call
0709 12:25:48.386118 (+19013263us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:25:47.653275 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:25:47.653833 (+   558us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:25:47.653867 (+    34us) write_op.cc:435] Acquired schema lock
0709 12:25:47.653885 (+    18us) tablet.cc:662] Decoding operations
0709 12:25:47.680816 (+ 26931us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:25:47.680922 (+   106us) write_op.cc:641] Partition lock acquired for write op
0709 12:25:47.680957 (+    35us) tablet.cc:685] Acquiring locks for 26 operations
0709 12:25:47.753169 (+ 72212us) tablet.cc:701] Row locks acquired
0709 12:25:47.753192 (+    23us) write_op.cc:260] PREPARE: finished
0709 12:25:47.753345 (+   153us) write_op.cc:270] Start()
0709 12:25:47.753465 (+   120us) write_op.cc:276] Timestamp: P: 1783599947753295 usec, L: 0
0709 12:25:47.753500 (+    35us) op_driver.cc:348] REPLICATION: starting
0709 12:25:47.754238 (+   738us) log.cc:844] Serialized 10189 byte log entry
0709 12:25:47.907126 (+152888us) op_driver.cc:464] REPLICATION: finished
0709 12:25:47.920720 (+ 13594us) write_op.cc:301] APPLY: starting
0709 12:25:47.920806 (+    86us) tablet.cc:1367] starting BulkCheckPresence
0709 12:25:48.002831 (+ 82025us) tablet.cc:1370] finished BulkCheckPresence
0709 12:25:48.002993 (+   162us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:25:48.188290 (+185297us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:25:48.192589 (+  4299us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=52,key_file_lookups=52,delta_file_lookups=0,mrs_lookups=0
0709 12:25:48.192643 (+    54us) write_op.cc:312] APPLY: finished
0709 12:25:48.193148 (+   505us) log.cc:844] Serialized 228 byte log entry
0709 12:25:48.193349 (+   201us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:25:48.199726 (+  6377us) write_op.cc:454] Released schema lock
0709 12:25:48.199802 (+    76us) write_op.cc:341] FINISH: Updating metrics
0709 12:25:48.217098 (+ 17296us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:25:48.217133 (+    35us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":45,"child_traces":[["op",{"apply.queue_time_us":13307,"apply.run_cpu_time_us":38408,"apply.run_wall_time_us":291936,"cfile_cache_hit":106,"cfile_cache_hit_bytes":112911,"num_ops":26,"prepare.queue_time_us":10134,"prepare.run_cpu_time_us":12662,"prepare.run_wall_time_us":206373,"raft.queue_time_us":45251,"raft.run_cpu_time_us":986,"raft.run_wall_time_us":44257,"replication_time_us":153457,"thread_start_us":188385,"threads_started":4,"wal-append.queue_time_us":67313}]]}
I20260709 12:25:48.428557 16501 tablet_replica.cc:333] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3: stopping tablet replica
I20260709 12:25:48.429454 16501 raft_consensus.cc:2243] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:48.429981 16501 raft_consensus.cc:2272] T 84421f094faa40f39b30b4c3d0737639 P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:48.436061 17761 raft_consensus.cc:1275] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 32 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 31 index: 22. Preceding OpId from leader: term: 32 index: 23. (index mismatch)
I20260709 12:25:48.438900 29134 consensus_queue.cc:1048] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.000s
I20260709 12:25:48.439808 28997 raft_consensus.cc:493] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:48.440397 28997 raft_consensus.cc:515] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:48.445549 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 31 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:48.450445 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 31 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:48.454030 28997 leader_election.cc:290] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:48.447650 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 31 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:48.457635 17652 leader_election.cc:304] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:25:48.469115 17728 consensus_peers.cc:597] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:25:48.469131 29076 raft_consensus.cc:493] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:48.471606 29076 raft_consensus.cc:515] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:25:48.477386 17802 leader_election.cc:336] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 33 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:48.478120 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 33 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:48.478849 17756 raft_consensus.cc:2468] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 32.
I20260709 12:25:48.480098 17801 leader_election.cc:304] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:25:48.480885 29009 raft_consensus.cc:2804] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:25:48.481256 29009 raft_consensus.cc:493] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:48.481385 28997 raft_consensus.cc:2749] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:25:48.481633 29009 raft_consensus.cc:3060] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:25:48.484612 29076 leader_election.cc:290] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:25:48.490658 29079 raft_consensus.cc:493] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:25:48.491207 29079 raft_consensus.cc:515] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:48.493513 29079 leader_election.cc:290] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:48.494195 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:48.494957 17756 raft_consensus.cc:2468] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 30.
I20260709 12:25:48.498791 18333 leader_election.cc:304] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:48.499558 29079 raft_consensus.cc:2804] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:25:48.499948 29079 raft_consensus.cc:493] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:25:48.500341 29079 raft_consensus.cc:3060] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:48.495218 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:48.519729 17831 raft_consensus.cc:2393] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 30: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:25:48.538322 16501 tablet_replica.cc:333] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3: stopping tablet replica
I20260709 12:25:48.539330 16501 raft_consensus.cc:2243] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 32 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:48.537087 29009 raft_consensus.cc:515] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:25:48.548089 17802 consensus_peers.cc:597] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:25:48.549268 16501 raft_consensus.cc:2272] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 0da169ff82674dc1947055ce43de68a3 [term 32 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:48.556627 17756 raft_consensus.cc:1275] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 35 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 31 index: 21. Preceding OpId from leader: term: 35 index: 23. (index mismatch)
I20260709 12:25:48.559279 29127 consensus_queue.cc:1048] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 21, Time since last communication: 0.000s
W20260709 12:25:48.565191 17802 consensus_peers.cc:597] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260709 12:25:48.566067 16501 tablet_replica.cc:333] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3: stopping tablet replica
W20260709 12:25:48.568432 17728 consensus_peers.cc:597] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:25:48.570199 28955 raft_consensus.cc:493] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:48.573066 28955 raft_consensus.cc:515] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:48.575182 28955 leader_election.cc:290] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:25:48.575271 18334 consensus_peers.cc:597] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:25:48.577250 16501 raft_consensus.cc:2243] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:48.578545 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 31 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:48.579133 17756 raft_consensus.cc:2468] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 30.
I20260709 12:25:48.581821 17801 leader_election.cc:304] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:48.582734 29076 raft_consensus.cc:2804] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:25:48.582953 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 31 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:25:48.585419 17802 leader_election.cc:336] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 33 election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:48.585593 16501 raft_consensus.cc:2272] T e5dabc1347544bd8b4e81fa5903d4a8b P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:48.583138 29076 raft_consensus.cc:493] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:48.588254 28955 raft_consensus.cc:493] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:25:48.599114 28955 raft_consensus.cc:515] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:48.588661 29076 raft_consensus.cc:3060] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:48.606470 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa38faae58422e9cecc99973cab688" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 33 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:48.607219 17761 raft_consensus.cc:3060] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:25:48.611773 28955 leader_election.cc:290] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:48.614418 17761 raft_consensus.cc:2468] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 33.
I20260709 12:25:48.606746 29009 leader_election.cc:290] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 33 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:25:48.615685 17801 leader_election.cc:304] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:25:48.625950 17831 raft_consensus.cc:1275] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 35 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 33 index: 22. Preceding OpId from leader: term: 35 index: 23. (index mismatch)
I20260709 12:25:48.627698 28927 consensus_queue.cc:1048] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.002s
I20260709 12:25:48.629294 29079 raft_consensus.cc:515] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:48.631095 16501 tablet_replica.cc:333] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3: stopping tablet replica
I20260709 12:25:48.631995 18362 raft_consensus.cc:2468] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 30.
I20260709 12:25:48.634948 29009 raft_consensus.cc:2804] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 33 FOLLOWER]: Leader election won for term 33
I20260709 12:25:48.635694 29009 raft_consensus.cc:697] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 33 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
W20260709 12:25:48.635952 17802 leader_election.cc:336] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:48.638542 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:48.636934 29009 consensus_queue.cc:237] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 31.17, Last appended by leader: 17, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:48.645952 17761 raft_consensus.cc:2468] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 30.
I20260709 12:25:48.656723 16501 raft_consensus.cc:2243] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 33 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:48.659643 16501 raft_consensus.cc:2272] T a5732e71b9d54f8786b8c511d69943bf P 0da169ff82674dc1947055ce43de68a3 [term 33 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:48.663517 18333 leader_election.cc:304] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:48.666800 28927 raft_consensus.cc:2804] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:25:48.667322 28927 raft_consensus.cc:697] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 30 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:25:48.668210 28927 consensus_queue.cc:237] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 29.18, Last appended by leader: 18, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:48.675520 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 85147us
I20260709 12:25:48.680037 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:48.683459 16501 tablet_replica.cc:333] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3: stopping tablet replica
I20260709 12:25:48.683223 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 28 candidate_status { last_received { term: 27 index: 23 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:48.689185 17803 leader_election.cc:304] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:25:48.690909 29165 raft_consensus.cc:2749] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:25:48.693418 16501 raft_consensus.cc:2243] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:48.694084 16501 raft_consensus.cc:2272] T 45397041e3ac43c8901c5cb03f27c7b9 P 0da169ff82674dc1947055ce43de68a3 [term 31 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:48.678821 29079 leader_election.cc:290] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 30 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:48.727522 17761 raft_consensus.cc:1275] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 37 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 36 index: 25. Preceding OpId from leader: term: 37 index: 26. (index mismatch)
I20260709 12:25:48.729063 28955 consensus_queue.cc:1048] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
W20260709 12:25:48.730906 17802 consensus_peers.cc:597] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
W20260709 12:25:48.740821 17802 consensus_peers.cc:597] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:25:48.744057 16501 tablet_replica.cc:333] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3: stopping tablet replica
I20260709 12:25:48.706959 29076 raft_consensus.cc:515] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:48.778812 17579 catalog_manager.cc:5697] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b reported cstate change: term changed from 30 to 31, leader changed from 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) to e063541542a94969898868bf6eb5ce1b (127.16.29.67). New cstate: current_term: 31 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:25:48.785879 17579 catalog_manager.cc:5697] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b reported cstate change: term changed from 29 to 34. New cstate: current_term: 34 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: HEALTHY } } }
I20260709 12:25:48.788160 17579 catalog_manager.cc:5697] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b reported cstate change: term changed from 26 to 34. New cstate: current_term: 34 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:25:48.789951 17837 raft_consensus.cc:2368] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 30. Current term is 31.
W20260709 12:25:48.790657 17580 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:37172 (request call id 262) took 13490 ms (client timeout 15000 ms). Trace:
I20260709 12:25:48.790217 17579 catalog_manager.cc:5697] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b reported cstate change: term changed from 11 to 27, leader changed from e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66) to e063541542a94969898868bf6eb5ce1b (127.16.29.67). New cstate: current_term: 27 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:25:48.791750 17579 catalog_manager.cc:5697] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b reported cstate change: term changed from 27 to 29, leader changed from 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) to e063541542a94969898868bf6eb5ce1b (127.16.29.67). New cstate: current_term: 29 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:25:48.794528 17579 catalog_manager.cc:5697] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b reported cstate change: term changed from 22 to 28, leader changed from 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) to e063541542a94969898868bf6eb5ce1b (127.16.29.67). New cstate: current_term: 28 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:25:48.797313 17579 catalog_manager.cc:5697] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b reported cstate change: term changed from 20 to 27, leader changed from e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66) to 334a4e1f59944544b3b3ff36616b57da (127.16.29.68). New cstate: current_term: 27 leader_uuid: "334a4e1f59944544b3b3ff36616b57da" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:25:48.799463 17579 catalog_manager.cc:5697] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b reported cstate change: term changed from 27 to 28, leader changed from 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) to e063541542a94969898868bf6eb5ce1b (127.16.29.67). New cstate: current_term: 28 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:25:48.801061 17579 catalog_manager.cc:5697] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b reported cstate change: term changed from 20 to 24. New cstate: current_term: 24 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } }
I20260709 12:25:48.807624 18333 leader_election.cc:400] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 30 election: Vote denied by peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837) with higher term. Message: Invalid argument: T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da for earlier term 30. Current term is 31.
I20260709 12:25:48.815694 29076 leader_election.cc:290] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 31 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:48.816915 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 31 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
W20260709 12:25:48.791041 17580 rpcz_store.cc:269] 0709 12:25:35.304210 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:46.937742 (+11633532us) service_pool.cc:224] Handling call
0709 12:25:48.788750 (+1851008us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:25:48.490183 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:25:48.490714 (+   531us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:25:48.490747 (+    33us) write_op.cc:435] Acquired schema lock
0709 12:25:48.490765 (+    18us) tablet.cc:662] Decoding operations
0709 12:25:48.491821 (+  1056us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:25:48.491919 (+    98us) write_op.cc:641] Partition lock acquired for write op
0709 12:25:48.491951 (+    32us) tablet.cc:685] Acquiring locks for 16 operations
0709 12:25:48.493328 (+  1377us) tablet.cc:701] Row locks acquired
0709 12:25:48.493355 (+    27us) write_op.cc:260] PREPARE: finished
0709 12:25:48.493497 (+   142us) write_op.cc:270] Start()
0709 12:25:48.493612 (+   115us) write_op.cc:276] Timestamp: P: 1783599948493442 usec, L: 0
0709 12:25:48.493635 (+    23us) op_driver.cc:348] REPLICATION: starting
0709 12:25:48.494268 (+   633us) log.cc:844] Serialized 6230 byte log entry
0709 12:25:48.584440 (+ 90172us) op_driver.cc:464] REPLICATION: finished
0709 12:25:48.588592 (+  4152us) write_op.cc:301] APPLY: starting
0709 12:25:48.590671 (+  2079us) tablet.cc:1367] starting BulkCheckPresence
0709 12:25:48.617831 (+ 27160us) tablet.cc:1370] finished BulkCheckPresence
0709 12:25:48.617917 (+    86us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:25:48.691816 (+ 73899us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:25:48.725279 (+ 33463us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=32,key_file_lookups=32,delta_file_lookups=0,mrs_lookups=0
0709 12:25:48.725328 (+    49us) write_op.cc:312] APPLY: finished
0709 12:25:48.725679 (+   351us) log.cc:844] Serialized 148 byte log entry
0709 12:25:48.726508 (+   829us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:25:48.726706 (+   198us) write_op.cc:454] Released schema lock
0709 12:25:48.726759 (+    53us) write_op.cc:341] FINISH: Updating metrics
0709 12:25:48.727113 (+   354us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:25:48.727139 (+    26us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":33,"reported_tablets":45,"spinlock_wait_cycles":34688,"child_traces":[["op",{"apply.queue_time_us":465,"apply.run_cpu_time_us":20541,"apply.run_wall_time_us":138524,"cfile_cache_hit":64,"cfile_cache_hit_bytes":66944,"num_ops":16,"prepare.queue_time_us":81649,"prepare.run_cpu_time_us":4370,"prepare.run_wall_time_us":4371,"replication_time_us":90649,"thread_start_us":81317,"threads_started":1}]]}
W20260709 12:25:48.876253 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 52309.6us
W20260709 12:25:48.879743 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 53336.8us
I20260709 12:25:48.795023 28927 raft_consensus.cc:493] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
W20260709 12:25:48.881891 18334 consensus_peers.cc:597] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260709 12:25:48.878974 28637 raft_consensus.cc:493] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:48.902181 28637 raft_consensus.cc:515] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:48.913105 28637 leader_election.cc:290] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:48.885869 28927 raft_consensus.cc:515] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:48.915334 28927 leader_election.cc:290] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:48.945029 16501 raft_consensus.cc:2243] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:48.950282 16501 raft_consensus.cc:2272] T 3219dcbfd86340ecbde507517060ff81 P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:48.955601 16501 tablet_replica.cc:333] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3: stopping tablet replica
I20260709 12:25:48.957043 16501 raft_consensus.cc:2243] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:48.958088 16501 raft_consensus.cc:2272] T 5ad2828ec58f490a9ea564d9805b56b7 P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:48.962157 16501 tablet_replica.cc:333] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3: stopping tablet replica
I20260709 12:25:48.977896 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 28 candidate_status { last_received { term: 27 index: 23 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:48.979455 17831 raft_consensus.cc:2468] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 27.
I20260709 12:25:48.963634 16501 raft_consensus.cc:2243] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 32 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:48.985383 16501 raft_consensus.cc:2272] T 154e6b769a8d476ba26f7fe3a521fc8d P 0da169ff82674dc1947055ce43de68a3 [term 32 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:49.013018 29183 raft_consensus.cc:493] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 34 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:25:49.018193 16501 tablet_replica.cc:333] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3: stopping tablet replica
I20260709 12:25:49.027837 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 31 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:49.036239 18374 raft_consensus.cc:1275] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 37 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 36 index: 25. Preceding OpId from leader: term: 37 index: 26. (index mismatch)
I20260709 12:25:49.014248 29183 raft_consensus.cc:515] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:49.058569 29183 leader_election.cc:290] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:25:49.039506 17763 raft_consensus.cc:3060] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:49.050853 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 28 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:49.065261 29079 raft_consensus.cc:493] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:49.066053 29079 raft_consensus.cc:515] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:49.069656 29079 leader_election.cc:290] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:49.072115 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "0da169ff82674dc1947055ce43de68a3" candidate_term: 28 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:49.083472 16501 raft_consensus.cc:2243] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 33 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:49.085294 16501 raft_consensus.cc:2272] T 3de52d85935746e9a1e770fef3529988 P 0da169ff82674dc1947055ce43de68a3 [term 33 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:49.096400 18333 leader_election.cc:304] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:25:49.119117 16501 tablet_replica.cc:333] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3: stopping tablet replica
I20260709 12:25:49.107020 28927 raft_consensus.cc:2804] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:25:49.121245 28927 raft_consensus.cc:493] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:25:49.121713 28927 raft_consensus.cc:3060] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:49.144646 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 35 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:49.145560 17762 raft_consensus.cc:2468] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 34.
I20260709 12:25:49.149667 29130 raft_consensus.cc:493] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 34 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:25:49.150244 29130 raft_consensus.cc:515] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:49.152779 29130 leader_election.cc:290] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:49.156414 17801 leader_election.cc:304] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:25:49.157246 17728 consensus_peers.cc:597] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:25:49.157399 28891 raft_consensus.cc:2804] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:25:49.157876 28891 raft_consensus.cc:493] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 34 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:25:49.158324 28891 raft_consensus.cc:3060] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:25:49.158715 17763 raft_consensus.cc:2468] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 31.
I20260709 12:25:49.160028 16501 raft_consensus.cc:2243] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 35 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:49.161648 16501 pending_rounds.cc:70] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3: Trying to abort 1 pending ops.
I20260709 12:25:49.168033 16501 pending_rounds.cc:77] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3: Aborting op as it isn't in flight: id { term: 31 index: 18 } timestamp: 7305625221443633152 op_type: NO_OP noop_request { }
W20260709 12:25:49.169998 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 49946us
I20260709 12:25:49.172431 16501 raft_consensus.cc:2889] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 35 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260709 12:25:49.177796 17654 leader_election.cc:304] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da169ff82674dc1947055ce43de68a3; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:49.178077 16501 raft_consensus.cc:2272] T 5f52bc21096e4519acb00ed1ea68c343 P 0da169ff82674dc1947055ce43de68a3 [term 35 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:49.181835 16501 tablet_replica.cc:333] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3: stopping tablet replica
I20260709 12:25:49.182036 28637 raft_consensus.cc:2749] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:25:49.226397 16501 raft_consensus.cc:2243] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 33 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:49.227382 16501 raft_consensus.cc:2272] T 3ed3f6bd4d904d42a3a48e346ed9227d P 0da169ff82674dc1947055ce43de68a3 [term 33 FOLLOWER]: Raft consensus is shut down!
W20260709 12:25:49.183362 18334 leader_election.cc:336] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:49.187176 17801 leader_election.cc:304] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:25:49.228168 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 38 candidate_status { last_received { term: 36 index: 25 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
W20260709 12:25:49.228642 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 45427.3us
W20260709 12:25:49.229573 18334 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 228 similar messages]
I20260709 12:25:49.230464 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 38 candidate_status { last_received { term: 36 index: 25 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:49.192806 28927 raft_consensus.cc:515] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:49.231951 29188 raft_consensus.cc:2804] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:25:49.232640 29188 raft_consensus.cc:697] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 31 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:25:49.233232 28927 leader_election.cc:290] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 28 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:49.233986 29188 consensus_queue.cc:237] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 29.18, Last appended by leader: 18, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:49.233986 28927 raft_consensus.cc:493] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:25:49.192987 29076 consensus_queue.cc:1048] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.001s
W20260709 12:25:49.195506 17802 consensus_peers.cc:597] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260709 12:25:49.206070 28891 raft_consensus.cc:515] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 35 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:49.238673 28927 raft_consensus.cc:515] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:49.283767 17728 leader_election.cc:336] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 35 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:49.284075 28927 leader_election.cc:290] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:25:49.284914 17728 consensus_peers.cc:597] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:25:49.294260 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 28 candidate_status { last_received { term: 27 index: 23 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:49.294857 17831 raft_consensus.cc:3060] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:49.315217 17831 raft_consensus.cc:2468] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 28.
I20260709 12:25:49.320277 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 33 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:49.321058 17761 raft_consensus.cc:2468] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 32.
W20260709 12:25:49.321117 29159 log.cc:927] Time spent T 00000000000000000000000000000000 P 2193af31cb894ec9b3d65d939a2a6851: Append to log took a long time: real 0.112s	user 0.003s	sys 0.010s
W20260709 12:25:49.318244 18334 leader_election.cc:336] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 28 election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:25:49.324303 18334 leader_election.cc:336] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 33 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:49.328321 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 35 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:49.329137 17837 raft_consensus.cc:2393] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 35: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:25:49.329804 18333 leader_election.cc:304] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:25:49.330582 29079 raft_consensus.cc:2804] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:25:49.331065 29079 raft_consensus.cc:697] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 28 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:25:49.333638 17761 raft_consensus.cc:1240] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 30. Current term is 31. Ops: []
I20260709 12:25:49.334167 29079 consensus_queue.cc:237] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 27.23, Last appended by leader: 23, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:49.339259 18333 leader_election.cc:304] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:49.340273 28638 raft_consensus.cc:2749] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:25:49.341581 18333 leader_election.cc:304] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:25:49.352246 17727 leader_election.cc:304] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:49.353924 28638 raft_consensus.cc:2804] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:25:49.354348 28638 raft_consensus.cc:493] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 32 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:25:49.354688 28638 raft_consensus.cc:3060] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:25:49.356678 29130 raft_consensus.cc:3060] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:25:49.377139 28891 leader_election.cc:290] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 35 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:25:49.378139 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a02ab39fcd4078ad49697693cf3be8" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 35 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
W20260709 12:25:49.384925 17572 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 61194.7us
I20260709 12:25:49.384178 16501 tablet_replica.cc:333] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3: stopping tablet replica
I20260709 12:25:49.395174 29190 raft_consensus.cc:493] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:49.395872 29190 raft_consensus.cc:515] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:49.407717 29190 leader_election.cc:290] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:25:49.409633 17728 leader_election.cc:336] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 38 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:25:49.409833 17802 leader_election.cc:336] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 35 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:25:49.410638 17802 consensus_peers.cc:597] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:25:49.418006 17838 raft_consensus.cc:1240] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 31 LEADER]: Rejecting Update request from peer 334a4e1f59944544b3b3ff36616b57da for earlier term 30. Current term is 31. Ops: []
W20260709 12:25:49.415622 17573 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet d522f70beb1f417cb98940d90b594823 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 334a4e1f59944544b3b3ff36616b57da is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 30 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:25:49.422789 29134 consensus_queue.cc:1059] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: INVALID_TERM, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:25:49.423712 28927 raft_consensus.cc:3055] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:25:49.424098 28927 raft_consensus.cc:740] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 30 LEADER]: Becoming Follower/Learner. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:25:49.425056 28927 consensus_queue.cc:260] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 30.19, Last appended by leader: 19, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:49.426272 28927 raft_consensus.cc:3060] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:49.432724 17763 raft_consensus.cc:1275] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 33 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 29 index: 16. Preceding OpId from leader: term: 33 index: 18. (index mismatch)
I20260709 12:25:49.434767 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 38 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:49.437616 17831 raft_consensus.cc:2468] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 37.
I20260709 12:25:49.440236 16501 raft_consensus.cc:2243] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 27 LEADER]: Raft consensus shutting down.
W20260709 12:25:49.450341 17572 catalog_manager.cc:4755] TS 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): ChangeConfig:REMOVE_PEER RPC failed for tablet e5dabc1347544bd8b4e81fa5903d4a8b: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:49.447055 29130 raft_consensus.cc:2749] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 35 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:25:49.441074 28638 raft_consensus.cc:515] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 33 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:49.468693 29127 raft_consensus.cc:493] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:49.443951 29188 consensus_queue.cc:1048] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.018s
W20260709 12:25:49.461030 17802 consensus_peers.cc:597] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260709 12:25:49.476033 29127 raft_consensus.cc:515] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:49.478619 29127 leader_election.cc:290] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:49.484838 29185 raft_consensus.cc:493] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:49.485406 29185 raft_consensus.cc:515] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:49.489344 29185 leader_election.cc:290] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:49.490103 28638 leader_election.cc:290] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 33 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:25:49.491202 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 32 candidate_status { last_received { term: 30 index: 21 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:49.491926 18362 raft_consensus.cc:2468] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 31.
I20260709 12:25:49.493268 17729 leader_election.cc:304] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:49.494159 29190 raft_consensus.cc:2804] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:25:49.494606 29190 raft_consensus.cc:493] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:49.494989 29190 raft_consensus.cc:3060] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Advancing to term 32
W20260709 12:25:49.501247 17728 leader_election.cc:336] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:49.504392 16501 raft_consensus.cc:2272] T 53e726e4ebbb4ca58b6a6102c9b0850f P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:49.504773 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 36 candidate_status { last_received { term: 34 index: 24 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:49.508843 17756 raft_consensus.cc:2468] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 35.
I20260709 12:25:49.509091 16501 tablet_replica.cc:333] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3: stopping tablet replica
I20260709 12:25:49.510215 16501 raft_consensus.cc:2243] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:49.510998 16501 pending_rounds.cc:70] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3: Trying to abort 1 pending ops.
I20260709 12:25:49.511557 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 36 candidate_status { last_received { term: 34 index: 24 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:49.511722 18333 leader_election.cc:304] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:49.511989 17756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f96c39fba3496696f64385105bfeaa" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 33 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:49.512542 17756 raft_consensus.cc:3060] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 32 FOLLOWER]: Advancing to term 33
W20260709 12:25:49.515825 18334 leader_election.cc:336] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 33 election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:49.518770 29134 raft_consensus.cc:2804] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:25:49.519279 29134 raft_consensus.cc:493] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 35 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:49.519616 29134 raft_consensus.cc:3060] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:25:49.531844 16501 pending_rounds.cc:77] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3: Aborting op as it isn't in flight: id { term: 27 index: 19 } timestamp: 7305625256865935360 op_type: NO_OP noop_request { }
I20260709 12:25:49.532445 16501 raft_consensus.cc:2889] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
W20260709 12:25:49.503189 17802 consensus_peers.cc:597] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
W20260709 12:25:49.536100 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 62063.7us
I20260709 12:25:49.536360 17761 raft_consensus.cc:2468] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 35.
I20260709 12:25:49.565245 17801 leader_election.cc:304] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 35 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:49.566843 29188 raft_consensus.cc:2804] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 35 FOLLOWER]: Leader election won for term 35
I20260709 12:25:49.567549 29188 raft_consensus.cc:697] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 35 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:25:49.568759 29188 consensus_queue.cc:237] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 34.20, Last appended by leader: 20, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:49.569108 17727 leader_election.cc:304] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:25:49.570565 29185 raft_consensus.cc:2804] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 37 FOLLOWER]: Leader pre-election won for term 38
I20260709 12:25:49.570928 29185 raft_consensus.cc:493] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 37 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:49.571257 29185 raft_consensus.cc:3060] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 37 FOLLOWER]: Advancing to term 38
W20260709 12:25:49.576104 17579 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35966 (request call id 362) took 14659 ms (client timeout 15000 ms). Trace:
I20260709 12:25:49.589341 16501 raft_consensus.cc:2272] T ea2f2b5792f84bbba2738f1ce1e90122 P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Raft consensus is shut down!
W20260709 12:25:49.576617 17579 rpcz_store.cc:269] 0709 12:25:34.917007 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:44.414836 (+9497829us) service_pool.cc:224] Handling call
0709 12:25:49.575873 (+5161037us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:25:49.006850 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:25:49.007896 (+  1046us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:25:49.007968 (+    72us) write_op.cc:435] Acquired schema lock
0709 12:25:49.008018 (+    50us) tablet.cc:662] Decoding operations
0709 12:25:49.009749 (+  1731us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:25:49.009854 (+   105us) write_op.cc:641] Partition lock acquired for write op
0709 12:25:49.009911 (+    57us) tablet.cc:685] Acquiring locks for 9 operations
0709 12:25:49.015244 (+  5333us) tablet.cc:701] Row locks acquired
0709 12:25:49.015289 (+    45us) write_op.cc:260] PREPARE: finished
0709 12:25:49.027154 (+ 11865us) write_op.cc:270] Start()
0709 12:25:49.068005 (+ 40851us) write_op.cc:276] Timestamp: P: 1783599949015406 usec, L: 0
0709 12:25:49.068053 (+    48us) op_driver.cc:348] REPLICATION: starting
0709 12:25:49.069327 (+  1274us) log.cc:844] Serialized 3646 byte log entry
0709 12:25:49.073111 (+  3784us) op_driver.cc:464] REPLICATION: finished
0709 12:25:49.073904 (+   793us) write_op.cc:301] APPLY: starting
0709 12:25:49.074015 (+   111us) tablet.cc:1367] starting BulkCheckPresence
0709 12:25:49.125462 (+ 51447us) tablet.cc:1370] finished BulkCheckPresence
0709 12:25:49.125604 (+   142us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:25:49.155712 (+ 30108us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:25:49.208058 (+ 52346us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=18,key_file_lookups=18,delta_file_lookups=0,mrs_lookups=0
0709 12:25:49.208143 (+    85us) write_op.cc:312] APPLY: finished
0709 12:25:49.208633 (+   490us) log.cc:844] Serialized 89 byte log entry
0709 12:25:49.220861 (+ 12228us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:25:49.220982 (+   121us) write_op.cc:454] Released schema lock
0709 12:25:49.221054 (+    72us) write_op.cc:341] FINISH: Updating metrics
0709 12:25:49.224529 (+  3475us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:25:49.224575 (+    46us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":30278,"reported_tablets":46,"child_traces":[["op",{"apply.queue_time_us":460,"apply.run_cpu_time_us":15417,"apply.run_wall_time_us":150581,"cfile_cache_hit":36,"cfile_cache_hit_bytes":37656,"num_ops":9,"prepare.queue_time_us":50907,"prepare.run_cpu_time_us":6758,"prepare.run_wall_time_us":73325,"replication_time_us":4846}]]}
W20260709 12:25:49.599084 17802 leader_election.cc:336] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 35 election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:49.616837 17756 raft_consensus.cc:2468] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 33.
I20260709 12:25:49.661533 16501 tablet_replica.cc:333] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3: stopping tablet replica
I20260709 12:25:49.662552 16501 raft_consensus.cc:2243] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:49.663275 16501 raft_consensus.cc:2272] T d522f70beb1f417cb98940d90b594823 P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:49.666888 29190 raft_consensus.cc:515] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 32 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:49.644327 29134 raft_consensus.cc:515] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 36 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:49.673473 29134 leader_election.cc:290] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 36 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:49.677132 29190 leader_election.cc:290] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 32 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:25:49.682308 17582 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35966 (request call id 358) took 115422 ms (client timeout 15000 ms). Trace:
W20260709 12:25:49.682947 17582 rpcz_store.cc:269] 0709 12:23:54.265857 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:54.271843 (+  5986us) service_pool.cc:224] Handling call
0709 12:25:49.682078 (+115410235us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":109879,"reported_tablets":46}
W20260709 12:25:49.689635 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 55997.9us
I20260709 12:25:49.699374 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 36 candidate_status { last_received { term: 34 index: 24 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:49.701503 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 36 candidate_status { last_received { term: 34 index: 24 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:49.702307 17761 raft_consensus.cc:3060] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:25:49.702759 18333 leader_election.cc:304] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:25:49.706234 29201 raft_consensus.cc:2804] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 33 FOLLOWER]: Leader election won for term 33
I20260709 12:25:49.706372 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84421f094faa40f39b30b4c3d0737639" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 32 candidate_status { last_received { term: 30 index: 21 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:25:49.706907 29201 raft_consensus.cc:697] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 33 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:25:49.708092 29201 consensus_queue.cc:237] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 32.20, Last appended by leader: 20, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:49.707041 18363 raft_consensus.cc:3060] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:25:49.713721 16501 tablet_replica.cc:333] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3: stopping tablet replica
W20260709 12:25:49.720196 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 81267.9us
I20260709 12:25:49.740347 29185 raft_consensus.cc:515] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 38 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:49.751469 18363 raft_consensus.cc:2468] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 32.
I20260709 12:25:49.761885 16501 raft_consensus.cc:2243] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:49.770020 17729 leader_election.cc:304] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:25:49.774369 17802 consensus_peers.cc:597] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
W20260709 12:25:49.780550 18334 consensus_peers.cc:597] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:25:49.780789 29185 leader_election.cc:290] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 38 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:49.783159 16501 raft_consensus.cc:2272] T 044ba5aee1d6499a975c6b29d932fd95 P 0da169ff82674dc1947055ce43de68a3 [term 27 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:49.786526 16501 tablet_replica.cc:333] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3: stopping tablet replica
I20260709 12:25:49.787340 16501 raft_consensus.cc:2243] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 36 LEADER]: Raft consensus shutting down.
I20260709 12:25:49.789422 28891 raft_consensus.cc:493] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 34 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:25:49.790064 28891 raft_consensus.cc:515] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:49.792708 17837 raft_consensus.cc:1275] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 27 index: 23. Preceding OpId from leader: term: 28 index: 24. (index mismatch)
I20260709 12:25:49.795565 28927 consensus_queue.cc:1048] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.000s
W20260709 12:25:49.797659 17802 consensus_peers.cc:597] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
W20260709 12:25:49.797972 17655 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:49.798902 17761 raft_consensus.cc:2468] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 36.
I20260709 12:25:49.812067 18333 leader_election.cc:304] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 36 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:25:49.854233 16501 raft_consensus.cc:2272] T e2bcfe1b94464b6d9620cc3b8a17a2ad P 0da169ff82674dc1947055ce43de68a3 [term 36 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:49.840785 28927 raft_consensus.cc:2804] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 36 FOLLOWER]: Leader election won for term 36
I20260709 12:25:49.884445 28927 raft_consensus.cc:697] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 36 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:25:49.885478 28927 consensus_queue.cc:237] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 34.24, Last appended by leader: 24, Current term: 36, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:49.843170 29202 raft_consensus.cc:2804] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 32 FOLLOWER]: Leader election won for term 32
I20260709 12:25:49.891043 29202 raft_consensus.cc:697] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 32 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:25:49.892249 29202 consensus_queue.cc:237] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 30.21, Last appended by leader: 21, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:49.862892 28891 leader_election.cc:290] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:49.899397 17763 raft_consensus.cc:1275] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 29 index: 18. Preceding OpId from leader: term: 31 index: 19. (index mismatch)
I20260709 12:25:49.865046 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 35 candidate_status { last_received { term: 34 index: 24 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:49.879469 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13a48228b0d44d3ea21f474203dd02e3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 38 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:49.901124 17838 raft_consensus.cc:3060] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:25:49.925058 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 35 candidate_status { last_received { term: 34 index: 24 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:49.929339 17763 raft_consensus.cc:2368] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 36 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 35. Current term is 36.
W20260709 12:25:50.025750 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 76353.5us
W20260709 12:25:50.034142 29208 log.cc:927] Time spent T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da: Append to log took a long time: real 0.050s	user 0.001s	sys 0.000s
W20260709 12:25:50.036847 29212 log.cc:927] Time spent T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.069s	user 0.000s	sys 0.000s
W20260709 12:25:50.037629 29197 log.cc:927] Time spent T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.073s	user 0.000s	sys 0.000s
I20260709 12:25:50.044924 17801 leader_election.cc:400] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 35 pre-election: Vote denied by peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651) with higher term. Message: Invalid argument: T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 36 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 35. Current term is 36.
I20260709 12:25:50.045301 17801 leader_election.cc:403] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 35 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:25:50.050508 29076 raft_consensus.cc:3060] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 34 FOLLOWER]: Advancing to term 36
I20260709 12:25:50.065954 16501 tablet_replica.cc:333] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3: stopping tablet replica
I20260709 12:25:50.041954 29183 consensus_queue.cc:1048] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.038s
W20260709 12:25:50.126207 17572 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 93097.8us
W20260709 12:25:50.131771 17728 leader_election.cc:336] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 32 election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:50.123769 16501 raft_consensus.cc:2243] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 36 FOLLOWER]: Raft consensus shutting down.
W20260709 12:25:50.141184 29214 log.cc:927] Time spent T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b: Append to log took a long time: real 0.082s	user 0.001s	sys 0.000s
W20260709 12:25:50.143958 17583 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35958 (request call id 413) took 157984 ms (client timeout 15000 ms). Trace:
W20260709 12:25:50.144447 17583 rpcz_store.cc:269] 0709 12:23:12.047744 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:12.052017 (+  4273us) service_pool.cc:224] Handling call
0709 12:25:50.028365 (+157976348us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":46340,"reported_tablets":45}
I20260709 12:25:50.152007 17838 raft_consensus.cc:2468] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 38 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 38.
W20260709 12:25:50.137693 17728 consensus_peers.cc:597] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
W20260709 12:25:50.155077 18334 consensus_peers.cc:597] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260709 12:25:50.178161 29076 raft_consensus.cc:2749] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 36 FOLLOWER]: Leader pre-election lost for term 35. Reason: Vote denied by peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651) with higher term. Message: Invalid argument: T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 36 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b for earlier term 35. Current term is 36.
I20260709 12:25:50.194972 16501 pending_rounds.cc:70] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3: Trying to abort 1 pending ops.
I20260709 12:25:50.195374 16501 pending_rounds.cc:77] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3: Aborting op as it isn't in flight: id { term: 35 index: 21 } timestamp: 7305625336327012352 op_type: NO_OP noop_request { }
W20260709 12:25:50.209901 29215 log.cc:927] Time spent T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb: Append to log took a long time: real 0.102s	user 0.001s	sys 0.001s
W20260709 12:25:50.214690 18334 consensus_peers.cc:597] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:25:50.220657 18365 raft_consensus.cc:1275] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 30 index: 19. Preceding OpId from leader: term: 31 index: 19. (term mismatch)
I20260709 12:25:50.221099 18365 pending_rounds.cc:85] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da: Aborting all ops after (but not including) 18
I20260709 12:25:50.221324 18365 pending_rounds.cc:107] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da: Aborting uncommitted NO_OP operation due to leader change: 30.19
I20260709 12:25:50.221590 18365 raft_consensus.cc:2889] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:50.195775 16501 raft_consensus.cc:2889] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 36 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260709 12:25:50.235850 16501 raft_consensus.cc:2272] T 3418610c3e1a4e28824db2b86acff690 P 0da169ff82674dc1947055ce43de68a3 [term 36 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:50.239110 16501 tablet_replica.cc:333] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3: stopping tablet replica
I20260709 12:25:50.239945 16501 raft_consensus.cc:2243] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 23 LEADER]: Raft consensus shutting down.
I20260709 12:25:50.242924 16501 raft_consensus.cc:2272] T a9aeb14eca4a43dcbeaccffdf7e28995 P 0da169ff82674dc1947055ce43de68a3 [term 23 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:50.245921 16501 tablet_replica.cc:333] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3: stopping tablet replica
W20260709 12:25:50.175590 17728 consensus_peers.cc:597] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
W20260709 12:25:50.247004 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 81009.2us
W20260709 12:25:50.248335 17728 leader_election.cc:336] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 38 election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:50.254050 29184 consensus_queue.cc:1048] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:25:50.278860 17761 raft_consensus.cc:1275] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 35 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 34 index: 20. Preceding OpId from leader: term: 35 index: 21. (index mismatch)
I20260709 12:25:50.279150 17727 leader_election.cc:304] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 38 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:25:50.280376 29211 raft_consensus.cc:2804] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 38 FOLLOWER]: Leader election won for term 38
I20260709 12:25:50.281107 29211 raft_consensus.cc:697] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 38 LEADER]: Becoming Leader. State: Replica: e2909cff1da44ad387f96a8ae3aa5afb, State: Running, Role: LEADER
I20260709 12:25:50.282186 29211 consensus_queue.cc:237] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 36.20, Last appended by leader: 20, Current term: 38, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:50.289157 28891 consensus_queue.cc:1048] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.001s
W20260709 12:25:50.296067 17802 consensus_peers.cc:597] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
W20260709 12:25:50.308458 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 58557.7us
W20260709 12:25:50.334228 18334 consensus_peers.cc:597] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260709 12:25:50.347669 16501 raft_consensus.cc:2243] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:50.348560 16501 pending_rounds.cc:70] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3: Trying to abort 1 pending ops.
I20260709 12:25:50.364101 16501 pending_rounds.cc:77] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3: Aborting op as it isn't in flight: id { term: 28 index: 19 } timestamp: 7305625329810866176 op_type: NO_OP noop_request { }
I20260709 12:25:50.364471 16501 raft_consensus.cc:2889] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260709 12:25:50.364827 16501 raft_consensus.cc:2272] T 50f9ff8f24a14a079db34b1ba88fa278 P 0da169ff82674dc1947055ce43de68a3 [term 29 FOLLOWER]: Raft consensus is shut down!
W20260709 12:25:50.379251 17584 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35962 (request call id 359) took 143462 ms (client timeout 15000 ms). Trace:
I20260709 12:25:50.387416 18365 raft_consensus.cc:1275] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 32 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 30 index: 21. Preceding OpId from leader: term: 32 index: 22. (index mismatch)
W20260709 12:25:50.379686 17584 rpcz_store.cc:269] 0709 12:23:26.916991 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:26.917261 (+   270us) service_pool.cc:224] Handling call
0709 12:25:50.379056 (+143461795us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":316,"reported_tablets":45}
I20260709 12:25:50.397821 16501 tablet_replica.cc:333] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3: stopping tablet replica
I20260709 12:25:50.398547 17763 consensus_queue.cc:237] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 28.14, Last appended by leader: 13, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:50.398715 16501 raft_consensus.cc:2243] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 36 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:50.416697 16501 raft_consensus.cc:2272] T ac984176358d453a8800cc9e0e304061 P 0da169ff82674dc1947055ce43de68a3 [term 36 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:50.420986 17579 catalog_manager.cc:5697] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 25 to 27. New cstate: current_term: 27 leader_uuid: "334a4e1f59944544b3b3ff36616b57da" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:25:50.422741 17579 catalog_manager.cc:5697] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 28 to 30, leader changed from e063541542a94969898868bf6eb5ce1b (127.16.29.67) to 0da169ff82674dc1947055ce43de68a3 (127.16.29.65). New cstate: current_term: 30 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:25:50.427160 17834 raft_consensus.cc:1275] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 36 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 34 index: 24. Preceding OpId from leader: term: 36 index: 25. (index mismatch)
I20260709 12:25:50.453174 18365 raft_consensus.cc:1275] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 28 index: 14. Preceding OpId from leader: term: 28 index: 15. (index mismatch)
I20260709 12:25:50.438822 17763 raft_consensus.cc:1275] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 33 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 32 index: 20. Preceding OpId from leader: term: 33 index: 21. (index mismatch)
I20260709 12:25:50.440141 17761 raft_consensus.cc:1275] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 36 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 33 index: 23. Preceding OpId from leader: term: 36 index: 25. (index mismatch)
W20260709 12:25:50.445832 17728 consensus_peers.cc:597] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260709 12:25:50.452656 17579 catalog_manager.cc:5697] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 32 to 34. New cstate: current_term: 34 leader_uuid: "334a4e1f59944544b3b3ff36616b57da" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:25:50.436179 16501 tablet_replica.cc:333] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3: stopping tablet replica
I20260709 12:25:50.457047 16501 raft_consensus.cc:2243] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 30 LEADER]: Raft consensus shutting down.
I20260709 12:25:50.457247 17579 catalog_manager.cc:5697] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 26 to 36, leader changed from 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) to e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66). New cstate: current_term: 36 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:25:50.458295 16501 raft_consensus.cc:2272] T c64345b0a2614ba4a382b83feb235cad P 0da169ff82674dc1947055ce43de68a3 [term 30 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:50.472325 29204 consensus_queue.cc:1048] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.000s
W20260709 12:25:50.462832 18334 consensus_peers.cc:597] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260709 12:25:50.458848 17579 catalog_manager.cc:5697] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 28 to 31. New cstate: current_term: 31 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:25:50.479499 29219 consensus_queue.cc:1048] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.001s
I20260709 12:25:50.480921 17579 catalog_manager.cc:5697] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 26 to 29, leader changed from 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) to e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66). New cstate: current_term: 29 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:25:50.482576 17579 catalog_manager.cc:5697] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 20 to 30. New cstate: current_term: 30 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:25:50.488204 29186 consensus_queue.cc:1048] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
W20260709 12:25:50.492066 17728 consensus_peers.cc:597] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
W20260709 12:25:50.492818 17728 consensus_peers.cc:597] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
W20260709 12:25:50.494640 17728 consensus_peers.cc:597] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260709 12:25:50.497622 17579 catalog_manager.cc:5697] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 27 to 29, leader changed from 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) to e063541542a94969898868bf6eb5ce1b (127.16.29.67). New cstate: current_term: 29 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
W20260709 12:25:50.505251 17728 consensus_peers.cc:597] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
W20260709 12:25:50.506368 17728 consensus_peers.cc:597] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
W20260709 12:25:50.496904 17802 consensus_peers.cc:597] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:25:50.509294 17802 consensus_peers.cc:597] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260709 12:25:50.511847 29204 consensus_queue.cc:1048] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.001s
I20260709 12:25:50.512888 28927 consensus_queue.cc:1048] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 22, Time since last communication: 0.000s
W20260709 12:25:50.527084 17728 consensus_peers.cc:597] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:25:50.522832 17579 catalog_manager.cc:5697] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 28 to 30. New cstate: current_term: 30 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
W20260709 12:25:50.537286 18334 consensus_peers.cc:597] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260709 12:25:50.543359 17579 catalog_manager.cc:5697] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 31 to 35, leader changed from e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66) to 334a4e1f59944544b3b3ff36616b57da (127.16.29.68). New cstate: current_term: 35 leader_uuid: "334a4e1f59944544b3b3ff36616b57da" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 12:25:50.544188 17728 consensus_peers.cc:597] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260709 12:25:50.556653 17579 catalog_manager.cc:5697] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 26 to 28, leader changed from 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) to e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66). New cstate: current_term: 28 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
W20260709 12:25:50.570869 17728 consensus_peers.cc:597] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:25:50.573966 17579 catalog_manager.cc:5697] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 22 to 30, leader changed from e063541542a94969898868bf6eb5ce1b (127.16.29.67) to 334a4e1f59944544b3b3ff36616b57da (127.16.29.68). New cstate: current_term: 30 leader_uuid: "334a4e1f59944544b3b3ff36616b57da" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:25:50.592949 29216 raft_consensus.cc:2955] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 LEADER]: Committing config change with OpId 28.15: config changed from index 12 to 15, VOTER e063541542a94969898868bf6eb5ce1b (127.16.29.67) evicted. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:25:50.608107 18363 raft_consensus.cc:2955] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Committing config change with OpId 28.15: config changed from index 12 to 15, VOTER e063541542a94969898868bf6eb5ce1b (127.16.29.67) evicted. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:25:50.599685 17579 catalog_manager.cc:5697] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 24 to 26, leader changed from 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) to e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66). New cstate: current_term: 26 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:25:50.634315 17579 catalog_manager.cc:5697] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 22 to 24, leader changed from 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) to e063541542a94969898868bf6eb5ce1b (127.16.29.67). New cstate: current_term: 24 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:25:50.641489 17579 catalog_manager.cc:5697] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 23 to 26, leader changed from 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) to e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66). New cstate: current_term: 26 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:25:50.658811 29203 raft_consensus.cc:493] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:25:50.659480 29203 raft_consensus.cc:515] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:50.680768 17579 catalog_manager.cc:5697] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 28 to 29, leader changed from 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) to e063541542a94969898868bf6eb5ce1b (127.16.29.67). New cstate: current_term: 29 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:25:50.686272 17579 catalog_manager.cc:5697] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 29 to 31. New cstate: current_term: 31 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:25:50.690563 29225 raft_consensus.cc:493] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:25:50.691321 29225 raft_consensus.cc:515] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:50.693722 29225 leader_election.cc:290] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:25:50.697492 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:50.698199 17762 raft_consensus.cc:2468] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 27.
I20260709 12:25:50.699429 17801 leader_election.cc:304] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:50.702111 29223 raft_consensus.cc:2804] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:25:50.702499 29223 raft_consensus.cc:493] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:25:50.702854 29203 leader_election.cc:290] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:50.702876 29223 raft_consensus.cc:3060] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:50.723027 17579 catalog_manager.cc:5697] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 31 to 35. New cstate: current_term: 35 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:25:50.724865 17579 catalog_manager.cc:5697] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 34 to 37, leader changed from e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66) to e063541542a94969898868bf6eb5ce1b (127.16.29.67). New cstate: current_term: 37 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
W20260709 12:25:50.724123 17728 leader_election.cc:336] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:50.733110 17579 catalog_manager.cc:5697] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 26 to 31, leader changed from 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) to e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66). New cstate: current_term: 31 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:25:50.735200 17579 catalog_manager.cc:5697] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 29 to 31, leader changed from 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) to e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66). New cstate: current_term: 31 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:25:50.736750 17579 catalog_manager.cc:5697] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 24 to 36. New cstate: current_term: 36 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
W20260709 12:25:50.744728 18334 consensus_peers.cc:597] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260709 12:25:50.747289 17579 catalog_manager.cc:5697] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 29 to 31. New cstate: current_term: 31 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
W20260709 12:25:50.744827 17802 leader_election.cc:336] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:50.730487 29223 raft_consensus.cc:515] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:50.752489 17579 catalog_manager.cc:5697] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 24 to 29, leader changed from 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) to e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66). New cstate: current_term: 29 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:25:50.757135 17579 catalog_manager.cc:5697] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 27 to 31. New cstate: current_term: 31 leader_uuid: "334a4e1f59944544b3b3ff36616b57da" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:25:50.778740 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:50.779914 17830 raft_consensus.cc:2393] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2909cff1da44ad387f96a8ae3aa5afb in current term 28: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:25:50.781876 17727 leader_election.cc:304] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:50.785845 29218 raft_consensus.cc:3060] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:50.790403 17571 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet d48b7de191684318aa08f60a7a395387 with cas_config_opid_index 12: ChangeConfig:REMOVE_PEER succeeded (attempt 9)
I20260709 12:25:50.800163 29218 raft_consensus.cc:2749] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:25:50.811352 29223 leader_election.cc:290] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 28 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:25:50.813189 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
W20260709 12:25:50.814816 17802 leader_election.cc:336] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 28 election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:50.836171 17830 raft_consensus.cc:1275] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 38 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 36 index: 20. Preceding OpId from leader: term: 38 index: 21. (index mismatch)
I20260709 12:25:50.837497 29203 consensus_queue.cc:1048] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 19, Time since last communication: 0.000s
I20260709 12:25:50.814404 17579 catalog_manager.cc:5697] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 21 to 28, leader changed from 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) to 334a4e1f59944544b3b3ff36616b57da (127.16.29.68). New cstate: current_term: 28 leader_uuid: "334a4e1f59944544b3b3ff36616b57da" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:25:50.842552 17579 catalog_manager.cc:5697] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 34 to 37, leader changed from e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66) to 334a4e1f59944544b3b3ff36616b57da (127.16.29.68). New cstate: current_term: 37 leader_uuid: "334a4e1f59944544b3b3ff36616b57da" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:25:50.843390 17762 raft_consensus.cc:2468] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 28.
I20260709 12:25:50.844580 17579 catalog_manager.cc:5697] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 23 to 29, leader changed from 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) to e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66). New cstate: current_term: 29 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
W20260709 12:25:50.851810 17728 consensus_peers.cc:597] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:25:50.852265 17801 leader_election.cc:304] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:25:50.853521 29223 raft_consensus.cc:2804] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:25:50.854300 29223 raft_consensus.cc:697] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 28 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:25:50.855311 29223 consensus_queue.cc:237] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 27.17, Last appended by leader: 17, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:50.847358 17579 catalog_manager.cc:5697] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 27 to 32, leader changed from 334a4e1f59944544b3b3ff36616b57da (127.16.29.68) to e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66). New cstate: current_term: 32 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:25:50.864609 17579 catalog_manager.cc:5697] T a5732e71b9d54f8786b8c511d69943bf P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 27 to 33. New cstate: current_term: 33 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:25:50.867188 17579 catalog_manager.cc:5697] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 28 to 32. New cstate: current_term: 32 leader_uuid: "334a4e1f59944544b3b3ff36616b57da" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:25:50.871343 17579 catalog_manager.cc:5697] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 28 to 30. New cstate: current_term: 30 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:25:50.874300 17579 catalog_manager.cc:5697] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 25 to 28, leader changed from e063541542a94969898868bf6eb5ce1b (127.16.29.67) to 334a4e1f59944544b3b3ff36616b57da (127.16.29.68). New cstate: current_term: 28 leader_uuid: "334a4e1f59944544b3b3ff36616b57da" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:25:50.899253 29216 raft_consensus.cc:493] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:25:50.899917 29216 raft_consensus.cc:515] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:50.904569 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 37 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:25:50.911023 17728 leader_election.cc:336] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 37 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:50.911530 17728 leader_election.cc:304] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:50.925547 29220 raft_consensus.cc:2749] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:25:50.922864 29216 leader_election.cc:290] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:25:50.953224 17802 consensus_peers.cc:597] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 16: this message will repeat every 5th retry.
W20260709 12:25:50.976312 17802 consensus_peers.cc:597] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 16: this message will repeat every 5th retry.
W20260709 12:25:50.991608 17728 consensus_peers.cc:597] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
W20260709 12:25:51.015213 17728 consensus_peers.cc:597] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
W20260709 12:25:51.020443 17802 consensus_peers.cc:597] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 16: this message will repeat every 5th retry.
W20260709 12:25:51.191921 17802 consensus_peers.cc:597] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
W20260709 12:25:51.227845 17728 consensus_peers.cc:597] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
W20260709 12:25:51.272338 17572 catalog_manager.cc:4755] TS 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): ChangeConfig:REMOVE_PEER RPC failed for tablet e5dabc1347544bd8b4e81fa5903d4a8b: Network error: recv got EOF from unknown peer (error 108)
I20260709 12:25:51.310410 17761 raft_consensus.cc:1275] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 27 index: 17. Preceding OpId from leader: term: 28 index: 18. (index mismatch)
I20260709 12:25:51.317142 29223 consensus_queue.cc:1048] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
W20260709 12:25:51.421950 17728 consensus_peers.cc:597] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 12:25:51.436631 29176 raft_consensus.cc:493] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
W20260709 12:25:51.438839 18334 consensus_peers.cc:597] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:25:51.437292 29176 raft_consensus.cc:515] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:51.449361 29176 leader_election.cc:290] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:25:51.450897 17761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 37 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:51.451648 17761 raft_consensus.cc:2468] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 36.
I20260709 12:25:51.469561 17801 leader_election.cc:304] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:51.472847 29227 raft_consensus.cc:2804] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 36 FOLLOWER]: Leader pre-election won for term 37
I20260709 12:25:51.473343 29227 raft_consensus.cc:493] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 36 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:25:51.473807 29227 raft_consensus.cc:3060] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:25:51.529567 29227 raft_consensus.cc:515] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 37 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:51.551522 17762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 37 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:51.552181 17762 raft_consensus.cc:3060] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:25:51.568094 29227 leader_election.cc:290] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 37 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:25:51.571019 17802 consensus_peers.cc:597] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:25:51.577211 18334 consensus_peers.cc:597] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 12:25:51.579965 17762 raft_consensus.cc:2468] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 37 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 37.
I20260709 12:25:51.586202 17801 leader_election.cc:304] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 37 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:51.587153 29227 raft_consensus.cc:2804] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 37 FOLLOWER]: Leader election won for term 37
I20260709 12:25:51.587845 29227 raft_consensus.cc:697] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 37 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:25:51.588730 29227 consensus_queue.cc:237] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 36.23, Last appended by leader: 23, Current term: 37, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:51.591535 29218 raft_consensus.cc:493] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:25:51.592165 29218 raft_consensus.cc:515] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:51.594377 29218 leader_election.cc:290] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:25:51.575666 17802 consensus_peers.cc:597] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:25:51.595247 17802 consensus_peers.cc:597] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:25:51.596621 17802 consensus_peers.cc:597] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:25:51.597213 17802 leader_election.cc:336] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 37 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111)
W20260709 12:25:51.598191 17802 consensus_peers.cc:597] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:25:51.602631 17728 leader_election.cc:336] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111)
I20260709 12:25:51.637777 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:25:51.656236 17802 leader_election.cc:336] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 37 election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111)
I20260709 12:25:51.673681 17727 leader_election.cc:304] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:25:51.678637 29253 raft_consensus.cc:2749] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:25:51.692356 17579 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:37172 (request call id 263) took 1824 ms. Trace:
I20260709 12:25:51.692847 17579 rpcz_store.cc:276] 0709 12:25:49.868557 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:49.868959 (+   402us) service_pool.cc:224] Handling call
0709 12:25:51.692260 (+1823301us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:25:50.971585 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:25:50.972182 (+   597us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:25:50.972227 (+    45us) write_op.cc:435] Acquired schema lock
0709 12:25:50.972246 (+    19us) tablet.cc:662] Decoding operations
0709 12:25:50.974624 (+  2378us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:25:50.974703 (+    79us) write_op.cc:641] Partition lock acquired for write op
0709 12:25:50.974730 (+    27us) tablet.cc:685] Acquiring locks for 33 operations
0709 12:25:51.030327 (+ 55597us) tablet.cc:701] Row locks acquired
0709 12:25:51.030349 (+    22us) write_op.cc:260] PREPARE: finished
0709 12:25:51.030503 (+   154us) write_op.cc:270] Start()
0709 12:25:51.030620 (+   117us) write_op.cc:276] Timestamp: P: 1783599951030453 usec, L: 0
0709 12:25:51.030642 (+    22us) op_driver.cc:348] REPLICATION: starting
0709 12:25:51.031345 (+   703us) log.cc:844] Serialized 12876 byte log entry
0709 12:25:51.137146 (+105801us) op_driver.cc:464] REPLICATION: finished
0709 12:25:51.159240 (+ 22094us) write_op.cc:301] APPLY: starting
0709 12:25:51.159328 (+    88us) tablet.cc:1367] starting BulkCheckPresence
0709 12:25:51.375610 (+216282us) tablet.cc:1370] finished BulkCheckPresence
0709 12:25:51.375742 (+   132us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:25:51.638276 (+262534us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:25:51.638696 (+   420us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=66,key_file_lookups=66,delta_file_lookups=0,mrs_lookups=0
0709 12:25:51.638740 (+    44us) write_op.cc:312] APPLY: finished
0709 12:25:51.639310 (+   570us) log.cc:844] Serialized 284 byte log entry
0709 12:25:51.639489 (+   179us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:25:51.639750 (+   261us) write_op.cc:454] Released schema lock
0709 12:25:51.639827 (+    77us) write_op.cc:341] FINISH: Updating metrics
0709 12:25:51.640326 (+   499us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:25:51.640355 (+    29us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":2130,"reported_tablets":44,"child_traces":[["op",{"apply.queue_time_us":21826,"apply.run_cpu_time_us":49835,"apply.run_wall_time_us":481038,"cfile_cache_hit":134,"cfile_cache_hit_bytes":142199,"num_ops":33,"prepare.queue_time_us":19440,"prepare.run_cpu_time_us":15911,"prepare.run_wall_time_us":108031,"raft.queue_time_us":22324,"raft.run_cpu_time_us":3925,"raft.run_wall_time_us":35107,"replication_time_us":106324,"thread_start_us":103742,"threads_started":4,"wal-append.queue_time_us":45786}]]}
I20260709 12:25:51.729853 17585 catalog_manager.cc:5697] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb reported cstate change: term changed from 23 to 34. New cstate: current_term: 34 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: UNKNOWN } } }
I20260709 12:25:51.743469 17585 catalog_manager.cc:5697] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb reported cstate change: term changed from 26 to 30. New cstate: current_term: 30 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } }
W20260709 12:25:51.766196 17802 consensus_peers.cc:597] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:25:51.779687 17802 consensus_peers.cc:597] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 12:25:51.870191 29227 raft_consensus.cc:493] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:25:51.870808 29227 raft_consensus.cc:515] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:51.876657 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 29 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:25:51.883142 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d48b7de191684318aa08f60a7a395387" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 29 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:51.887840 17801 leader_election.cc:304] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:25:51.888924 29176 raft_consensus.cc:2749] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:25:51.897486 29188 raft_consensus.cc:493] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:25:51.898345 29188 raft_consensus.cc:515] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:25:51.909224 17802 leader_election.cc:336] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111)
I20260709 12:25:51.909852 29227 leader_election.cc:290] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:51.912003 29205 raft_consensus.cc:493] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:25:51.918987 29205 raft_consensus.cc:515] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:51.921555 29205 leader_election.cc:290] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:51.922569 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 31 candidate_status { last_received { term: 30 index: 21 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:25:51.923237 17830 raft_consensus.cc:2468] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 30.
I20260709 12:25:51.925246 18333 leader_election.cc:304] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:25:51.926196 29205 raft_consensus.cc:2804] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Leader pre-election won for term 31
W20260709 12:25:51.926605 18334 leader_election.cc:336] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111)
I20260709 12:25:51.933038 29188 leader_election.cc:290] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:25:51.927309 29205 raft_consensus.cc:493] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:25:51.934805 29205 raft_consensus.cc:3060] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Advancing to term 31
W20260709 12:25:51.935055 17585 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35962 (request call id 364) took 23196 ms (client timeout 15000 ms). Trace:
W20260709 12:25:51.935504 17585 rpcz_store.cc:269] 0709 12:25:28.738632 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:40.249988 (+11511356us) service_pool.cc:224] Handling call
0709 12:25:51.934940 (+11684952us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:25:51.813053 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:25:51.813701 (+   648us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:25:51.813736 (+    35us) write_op.cc:435] Acquired schema lock
0709 12:25:51.813756 (+    20us) tablet.cc:662] Decoding operations
0709 12:25:51.814062 (+   306us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:25:51.814122 (+    60us) write_op.cc:641] Partition lock acquired for write op
0709 12:25:51.814140 (+    18us) tablet.cc:685] Acquiring locks for 2 operations
0709 12:25:51.814374 (+   234us) tablet.cc:701] Row locks acquired
0709 12:25:51.814386 (+    12us) write_op.cc:260] PREPARE: finished
0709 12:25:51.814506 (+   120us) write_op.cc:270] Start()
0709 12:25:51.814597 (+    91us) write_op.cc:276] Timestamp: P: 1783599951814454 usec, L: 0
0709 12:25:51.814613 (+    16us) op_driver.cc:348] REPLICATION: starting
0709 12:25:51.815184 (+   571us) log.cc:844] Serialized 923 byte log entry
0709 12:25:51.837182 (+ 21998us) op_driver.cc:464] REPLICATION: finished
0709 12:25:51.844140 (+  6958us) write_op.cc:301] APPLY: starting
0709 12:25:51.844269 (+   129us) tablet.cc:1367] starting BulkCheckPresence
0709 12:25:51.845749 (+  1480us) tablet.cc:1370] finished BulkCheckPresence
0709 12:25:51.845851 (+   102us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:25:51.847189 (+  1338us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:25:51.847364 (+   175us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=4,key_file_lookups=4,delta_file_lookups=0,mrs_lookups=0
0709 12:25:51.847417 (+    53us) write_op.cc:312] APPLY: finished
0709 12:25:51.847647 (+   230us) log.cc:844] Serialized 33 byte log entry
0709 12:25:51.847824 (+   177us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:25:51.848288 (+   464us) write_op.cc:454] Released schema lock
0709 12:25:51.848331 (+    43us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"mutex_wait_us":1855,"reported_tablets":45,"child_traces":[["op",{"apply.queue_time_us":6687,"apply.run_cpu_time_us":4181,"apply.run_wall_time_us":4517,"cfile_cache_hit":8,"cfile_cache_hit_bytes":8368,"num_ops":2,"prepare.queue_time_us":58435,"prepare.run_cpu_time_us":2538,"prepare.run_wall_time_us":7256,"replication_time_us":22400,"thread_start_us":49498,"threads_started":1}]]}
I20260709 12:25:51.940274 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" is_pre_election: true
I20260709 12:25:51.941005 17763 raft_consensus.cc:2468] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 23.
I20260709 12:25:51.949868 17801 leader_election.cc:304] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:25:51.950994 29227 raft_consensus.cc:2804] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:25:51.951514 29227 raft_consensus.cc:493] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Starting leader election (detected failure of leader 0da169ff82674dc1947055ce43de68a3)
I20260709 12:25:51.951931 29227 raft_consensus.cc:3060] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:25:51.960453 17728 consensus_peers.cc:597] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 12:25:51.971202 29205 raft_consensus.cc:515] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:51.994105 18334 leader_election.cc:336] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 31 election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111)
I20260709 12:25:51.995327 29205 leader_election.cc:290] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 31 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:25:51.995775 17571 catalog_manager.cc:5220] ChangeConfig:REMOVE_PEER RPC for tablet d48b7de191684318aa08f60a7a395387 with cas_config_opid_index 12: ChangeConfig:REMOVE_PEER failed with leader e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651) due to CAS failure; no further retry: Illegal state: Request specified cas_config_opid_index of 12 but the committed config has opid_index of 15
W20260709 12:25:52.002612 17586 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35962 (request call id 357) took 193373 ms (client timeout 15000 ms). Trace:
I20260709 12:25:52.015954 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 31 candidate_status { last_received { term: 30 index: 21 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:25:52.024225 17830 raft_consensus.cc:3060] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:52.016183 17579 catalog_manager.cc:5697] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 31 to 36. New cstate: current_term: 36 leader_uuid: "334a4e1f59944544b3b3ff36616b57da" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:25:52.032150 17830 raft_consensus.cc:2468] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 31.
W20260709 12:25:52.032577 17586 rpcz_store.cc:269] 0709 12:22:38.629699 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:38.681295 (+ 51596us) service_pool.cc:224] Handling call
0709 12:22:40.039089 (+1357794us) spinlock_profiling.cc:243] Waited 251 ms on lock 0x7b540010d454. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a58a090 00007f7b4a628f3a 00007f7b503dee9c 00007f7b504dd04e 00007f7b4e99e2a4 00007f7b4e99e232 00007f7b4e99e161 00007f7b4e99e0dc 00007f7b4e99d392 00007f7b4895b48c 00007f7b4895a8c6
0709 12:25:52.002373 (+191963284us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":17534,"reported_tablets":45,"spinlock_wait_cycles":577761664}
W20260709 12:25:52.034106 17802 leader_election.cc:336] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111)
W20260709 12:25:52.035458 17802 consensus_peers.cc:597] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 12:25:52.036263 17579 catalog_manager.cc:5697] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da reported cstate change: config changed from index 12 to 15, VOTER e063541542a94969898868bf6eb5ce1b (127.16.29.67) evicted. New cstate: current_term: 28 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:25:52.036861 29227 raft_consensus.cc:515] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:52.037871 17579 catalog_manager.cc:5697] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 27 to 31. New cstate: current_term: 31 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:25:52.038959 29227 leader_election.cc:290] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 24 election: Requested vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:25:52.039258 17579 catalog_manager.cc:5697] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 30 to 33, leader changed from 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) to 334a4e1f59944544b3b3ff36616b57da (127.16.29.68). New cstate: current_term: 33 leader_uuid: "334a4e1f59944544b3b3ff36616b57da" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:25:52.040877 17579 catalog_manager.cc:5697] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 28 to 32, leader changed from 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) to e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66). New cstate: current_term: 32 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:25:52.052505 18333 leader_election.cc:304] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:25:52.053439 17763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "e2909cff1da44ad387f96a8ae3aa5afb"
I20260709 12:25:52.054106 17763 raft_consensus.cc:3060] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:52.056331 29257 raft_consensus.cc:2804] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Leader election won for term 31
W20260709 12:25:52.065851 17802 consensus_peers.cc:597] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:25:52.068724 17802 leader_election.cc:336] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 24 election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111)
I20260709 12:25:52.073529 17763 raft_consensus.cc:2468] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 24.
W20260709 12:25:52.061483 17587 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35958 (request call id 415) took 103829 ms (client timeout 15000 ms). Trace:
W20260709 12:25:52.080271 17587 rpcz_store.cc:269] 0709 12:24:08.240273 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:08.271462 (+ 31189us) service_pool.cc:224] Handling call
0709 12:25:52.061293 (+103789831us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":14702,"reported_tablets":45}
W20260709 12:25:52.081285 17571 connection.cc:466] server connection from 127.0.0.1:35958 torn down before Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35958 (request call id 415) could send its response
I20260709 12:25:52.085341 17801 leader_election.cc:304] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3
I20260709 12:25:52.086187 29227 raft_consensus.cc:2804] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:25:52.086710 29227 raft_consensus.cc:697] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 24 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:25:52.087638 29227 consensus_queue.cc:237] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 23.15, Last appended by leader: 15, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:52.175194 17821 tablet_service.cc:1558] Processing DeleteTablet for tablet d48b7de191684318aa08f60a7a395387 with delete_type TABLET_DATA_TOMBSTONED (TS e063541542a94969898868bf6eb5ce1b not found in new config with opid_index 15) from {username='slave'} at 127.0.0.1:45728
I20260709 12:25:52.141103 29257 raft_consensus.cc:697] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 31 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:25:52.181226 29257 consensus_queue.cc:237] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 30.21, Last appended by leader: 21, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:52.201715 17761 raft_consensus.cc:1275] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 37 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 36 index: 23. Preceding OpId from leader: term: 37 index: 24. (index mismatch)
W20260709 12:25:52.201809 17728 consensus_peers.cc:597] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 12:25:52.203389 29176 consensus_queue.cc:1048] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.000s
I20260709 12:25:52.227731 29264 tablet_replica.cc:333] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b: stopping tablet replica
W20260709 12:25:52.227955 17802 consensus_peers.cc:597] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:25:52.229018 29264 raft_consensus.cc:2243] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:52.241477 29264 raft_consensus.cc:2272] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:52.248116 29264 ts_tablet_manager.cc:1916] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20260709 12:25:52.285391 17728 consensus_peers.cc:597] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:25:52.291654 16501 thread.cc:527] Waited for 1000ms trying to join with diag-logger (tid 17710)
I20260709 12:25:52.304761 17578 catalog_manager.cc:5697] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb reported cstate change: term changed from 30 to 32, leader changed from 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) to e063541542a94969898868bf6eb5ce1b (127.16.29.67). New cstate: current_term: 32 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } }
I20260709 12:25:52.307251 17578 catalog_manager.cc:5697] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb reported cstate change: term changed from 27 to 33, leader changed from 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) to e063541542a94969898868bf6eb5ce1b (127.16.29.67). New cstate: current_term: 33 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } }
I20260709 12:25:52.309923 17579 catalog_manager.cc:5697] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 30 to 31, leader changed from 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) to 334a4e1f59944544b3b3ff36616b57da (127.16.29.68). New cstate: current_term: 31 leader_uuid: "334a4e1f59944544b3b3ff36616b57da" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:25:52.329331 17578 catalog_manager.cc:5697] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb reported cstate change: term changed from 24 to 27, leader changed from e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66) to 0da169ff82674dc1947055ce43de68a3 (127.16.29.65). New cstate: current_term: 27 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } }
I20260709 12:25:52.365969 17578 catalog_manager.cc:5697] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb reported cstate change: term changed from 34 to 35, leader changed from e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66) to e063541542a94969898868bf6eb5ce1b (127.16.29.67). New cstate: current_term: 35 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } }
I20260709 12:25:52.375114 17578 catalog_manager.cc:5697] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb reported cstate change: term changed from 30 to 31, leader changed from e063541542a94969898868bf6eb5ce1b (127.16.29.67) to e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66). New cstate: current_term: 31 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: HEALTHY } } }
I20260709 12:25:52.393090 29264 ts_tablet_manager.cc:1929] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 28.14
W20260709 12:25:52.403963 18334 consensus_peers.cc:597] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:25:52.406612 17578 catalog_manager.cc:5697] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb reported cstate change: term changed from 34 to 36. New cstate: current_term: 36 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } }
I20260709 12:25:52.413182 17578 catalog_manager.cc:5697] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb reported cstate change: term changed from 34 to 36, leader changed from e063541542a94969898868bf6eb5ce1b (127.16.29.67) to e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66). New cstate: current_term: 36 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: HEALTHY } } }
W20260709 12:25:52.409524 17581 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35962 (request call id 361) took 93429 ms (client timeout 15000 ms). Trace:
W20260709 12:25:52.424046 17581 rpcz_store.cc:269] 0709 12:24:18.980204 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:27.750577 (+8770373us) service_pool.cc:224] Handling call
0709 12:25:52.409221 (+84658644us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2316,"reported_tablets":45}
I20260709 12:25:52.423440 17578 catalog_manager.cc:5697] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb reported cstate change: term changed from 27 to 33. New cstate: current_term: 33 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } }
I20260709 12:25:52.415808 29264 log.cc:1199] T d48b7de191684318aa08f60a7a395387 P e063541542a94969898868bf6eb5ce1b: Deleting WAL directory at /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599551912677-16501-0/minicluster-data/ts-2-root/wals/d48b7de191684318aa08f60a7a395387
I20260709 12:25:52.436789 17571 catalog_manager.cc:5028] TS e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837): tablet d48b7de191684318aa08f60a7a395387 (table test-workload [id=ddc5b74250df4c65909b73b3742d615f]) successfully deleted
I20260709 12:25:52.429788 17578 catalog_manager.cc:5697] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb reported cstate change: term changed from 34 to 36, leader changed from e063541542a94969898868bf6eb5ce1b (127.16.29.67) to 0da169ff82674dc1947055ce43de68a3 (127.16.29.65). New cstate: current_term: 36 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } }
I20260709 12:25:52.448094 17578 catalog_manager.cc:5697] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb reported cstate change: term changed from 20 to 30. New cstate: current_term: 30 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: HEALTHY } } }
I20260709 12:25:52.451020 17578 catalog_manager.cc:5697] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb reported cstate change: term changed from 20 to 23, leader changed from e063541542a94969898868bf6eb5ce1b (127.16.29.67) to 0da169ff82674dc1947055ce43de68a3 (127.16.29.65). New cstate: current_term: 23 leader_uuid: "0da169ff82674dc1947055ce43de68a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } }
W20260709 12:25:52.494779 18334 consensus_peers.cc:597] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:25:52.507774 17802 consensus_peers.cc:597] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:25:52.527078 18334 consensus_peers.cc:597] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:25:52.541049 17802 consensus_peers.cc:597] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:25:52.565857 17762 raft_consensus.cc:1275] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 23 index: 15. Preceding OpId from leader: term: 24 index: 16. (index mismatch)
I20260709 12:25:52.567777 29268 consensus_queue.cc:1048] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
W20260709 12:25:52.600291 17728 consensus_peers.cc:597] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 12:25:52.609467 17830 raft_consensus.cc:1275] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 30 index: 21. Preceding OpId from leader: term: 31 index: 22. (index mismatch)
I20260709 12:25:52.611306 29257 consensus_queue.cc:1048] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.000s
W20260709 12:25:52.614734 17802 consensus_peers.cc:597] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:25:52.649590 18334 consensus_peers.cc:597] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:25:52.657231 17728 consensus_peers.cc:597] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:25:52.728783 17802 consensus_peers.cc:597] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:25:52.766948 17728 consensus_peers.cc:597] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 12:25:52.773000 17578 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35962 (request call id 365) took 5713 ms. Trace:
I20260709 12:25:52.779124 17584 catalog_manager.cc:5697] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b reported cstate change: term changed from 29 to 33, leader changed from 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) to e063541542a94969898868bf6eb5ce1b (127.16.29.67). New cstate: current_term: 33 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: HEALTHY } } }
W20260709 12:25:52.781680 17802 consensus_peers.cc:597] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 12:25:52.782637 17584 catalog_manager.cc:5697] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b reported cstate change: term changed from 30 to 35, leader changed from 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) to e063541542a94969898868bf6eb5ce1b (127.16.29.67). New cstate: current_term: 35 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: HEALTHY } } }
I20260709 12:25:52.773463 17578 rpcz_store.cc:276] 0709 12:25:47.059999 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:48.415203 (+1355204us) service_pool.cc:224] Handling call
0709 12:25:52.772917 (+4357714us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:25:52.461542 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:25:52.461957 (+   415us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:25:52.461979 (+    22us) write_op.cc:435] Acquired schema lock
0709 12:25:52.461992 (+    13us) tablet.cc:662] Decoding operations
0709 12:25:52.462608 (+   616us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:25:52.462658 (+    50us) write_op.cc:641] Partition lock acquired for write op
0709 12:25:52.462674 (+    16us) tablet.cc:685] Acquiring locks for 11 operations
0709 12:25:52.463261 (+   587us) tablet.cc:701] Row locks acquired
0709 12:25:52.463272 (+    11us) write_op.cc:260] PREPARE: finished
0709 12:25:52.463366 (+    94us) write_op.cc:270] Start()
0709 12:25:52.463441 (+    75us) write_op.cc:276] Timestamp: P: 1783599952463336 usec, L: 0
0709 12:25:52.463454 (+    13us) op_driver.cc:348] REPLICATION: starting
0709 12:25:52.469693 (+  6239us) log.cc:844] Serialized 4385 byte log entry
0709 12:25:52.472003 (+  2310us) op_driver.cc:464] REPLICATION: finished
0709 12:25:52.472466 (+   463us) write_op.cc:301] APPLY: starting
0709 12:25:52.472508 (+    42us) spinlock_profiling.cc:243] Waited 5.61 ms on lock 0x7b540010d454. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b4a589571 00007f7b4a61be28 00007f7b4e056581 00007f7b4e055d58 00007f7b4e05a9f1 00007f7b4e05a9a9 00007f7b4e05a939 00007f7b4e05a901 00007f7b4e059b2d 0000000000388634 0000000000388479
0709 12:25:52.472549 (+    41us) tablet.cc:1367] starting BulkCheckPresence
0709 12:25:52.530928 (+ 58379us) tablet.cc:1370] finished BulkCheckPresence
0709 12:25:52.530961 (+    33us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:25:52.573084 (+ 42123us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:25:52.573346 (+   262us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=22,key_file_lookups=22,delta_file_lookups=0,mrs_lookups=0
0709 12:25:52.573407 (+    61us) write_op.cc:312] APPLY: finished
0709 12:25:52.573789 (+   382us) log.cc:844] Serialized 105 byte log entry
0709 12:25:52.577322 (+  3533us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:25:52.577475 (+   153us) write_op.cc:454] Released schema lock
0709 12:25:52.577534 (+    59us) write_op.cc:341] FINISH: Updating metrics
0709 12:25:52.674572 (+ 97038us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:25:52.674611 (+    39us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":30876,"reported_tablets":45,"child_traces":[["op",{"apply.queue_time_us":220,"apply.run_cpu_time_us":14965,"apply.run_wall_time_us":202048,"cfile_cache_hit":44,"cfile_cache_hit_bytes":46024,"num_ops":11,"prepare.queue_time_us":156,"prepare.run_cpu_time_us":5589,"prepare.run_wall_time_us":11122,"replication_time_us":8410,"spinlock_wait_cycles":12892544}]]}
W20260709 12:25:52.828330 17728 consensus_peers.cc:597] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:25:52.854803 17728 consensus_peers.cc:597] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:25:52.911201 17581 catalog_manager.cc:5697] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb reported cstate change: term changed from 27 to 28, leader changed from 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) to e063541542a94969898868bf6eb5ce1b (127.16.29.67). New cstate: current_term: 28 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } }
I20260709 12:25:52.926205 17581 catalog_manager.cc:5697] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb reported cstate change: term changed from 36 to 38, leader changed from 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) to e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66). New cstate: current_term: 38 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: HEALTHY } } }
W20260709 12:25:52.948782 18334 consensus_peers.cc:597] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 12:25:52.957114 17584 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35966 (request call id 363) took 2421 ms. Trace:
I20260709 12:25:52.959362 17584 rpcz_store.cc:276] 0709 12:25:50.535906 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:50.544999 (+  9093us) service_pool.cc:224] Handling call
0709 12:25:52.957004 (+2412005us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:25:52.794728 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:25:52.795305 (+   577us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:25:52.795342 (+    37us) write_op.cc:435] Acquired schema lock
0709 12:25:52.795362 (+    20us) tablet.cc:662] Decoding operations
0709 12:25:52.795787 (+   425us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:25:52.795844 (+    57us) write_op.cc:641] Partition lock acquired for write op
0709 12:25:52.795872 (+    28us) tablet.cc:685] Acquiring locks for 2 operations
0709 12:25:52.796241 (+   369us) tablet.cc:701] Row locks acquired
0709 12:25:52.796260 (+    19us) write_op.cc:260] PREPARE: finished
0709 12:25:52.796408 (+   148us) write_op.cc:270] Start()
0709 12:25:52.796534 (+   126us) write_op.cc:276] Timestamp: P: 1783599952796360 usec, L: 0
0709 12:25:52.796557 (+    23us) op_driver.cc:348] REPLICATION: starting
0709 12:25:52.797155 (+   598us) log.cc:844] Serialized 923 byte log entry
0709 12:25:52.799431 (+  2276us) op_driver.cc:464] REPLICATION: finished
0709 12:25:52.799908 (+   477us) write_op.cc:301] APPLY: starting
0709 12:25:52.799996 (+    88us) tablet.cc:1367] starting BulkCheckPresence
0709 12:25:52.801327 (+  1331us) tablet.cc:1370] finished BulkCheckPresence
0709 12:25:52.801426 (+    99us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:25:52.802809 (+  1383us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:25:52.802976 (+   167us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=4,key_file_lookups=4,delta_file_lookups=0,mrs_lookups=0
0709 12:25:52.803031 (+    55us) write_op.cc:312] APPLY: finished
0709 12:25:52.803245 (+   214us) log.cc:844] Serialized 33 byte log entry
0709 12:25:52.803404 (+   159us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:25:52.803505 (+   101us) write_op.cc:454] Released schema lock
0709 12:25:52.803556 (+    51us) write_op.cc:341] FINISH: Updating metrics
0709 12:25:52.804558 (+  1002us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:25:52.804585 (+    27us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":11991,"reported_tablets":44,"child_traces":[["op",{"apply.queue_time_us":252,"apply.run_cpu_time_us":3968,"apply.run_wall_time_us":4634,"cfile_cache_hit":8,"cfile_cache_hit_bytes":8368,"num_ops":2,"prepare.queue_time_us":390,"prepare.run_cpu_time_us":2688,"prepare.run_wall_time_us":2703,"replication_time_us":2747}]]}
I20260709 12:25:52.957957 17581 catalog_manager.cc:5697] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb reported cstate change: term changed from 36 to 37, leader changed from 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) to e063541542a94969898868bf6eb5ce1b (127.16.29.67). New cstate: current_term: 37 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } }
I20260709 12:25:52.969027 17581 catalog_manager.cc:5697] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb reported cstate change: term changed from 23 to 24, leader changed from 0da169ff82674dc1947055ce43de68a3 (127.16.29.65) to e063541542a94969898868bf6eb5ce1b (127.16.29.67). New cstate: current_term: 24 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } }
W20260709 12:25:53.025884 18334 consensus_peers.cc:597] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:25:53.071341 17728 consensus_peers.cc:597] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:25:53.126880 18334 consensus_peers.cc:597] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:25:53.176692 17728 consensus_peers.cc:597] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:25:53.217669 17728 consensus_peers.cc:597] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:25:53.274461 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 56843us
W20260709 12:25:53.292207 16501 thread.cc:527] Waited for 2000ms trying to join with diag-logger (tid 17710)
W20260709 12:25:53.377467 18334 consensus_peers.cc:597] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 12:25:53.405037 17762 consensus_queue.cc:237] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 31.17, Last appended by leader: 16, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:53.408705 17802 consensus_peers.cc:597] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:25:53.416685 17728 consensus_peers.cc:597] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:25:53.430637 17728 consensus_peers.cc:597] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:25:53.432372 17728 consensus_peers.cc:597] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:25:53.513213 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 60230.6us
I20260709 12:25:53.549508 18363 raft_consensus.cc:1275] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Refusing update from remote peer e2909cff1da44ad387f96a8ae3aa5afb: Log matching property violated. Preceding OpId in replica: term: 31 index: 17. Preceding OpId from leader: term: 31 index: 18. (index mismatch)
W20260709 12:25:53.548319 17728 consensus_peers.cc:597] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:25:53.557704 17728 consensus_peers.cc:597] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:25:53.570209 29280 consensus_queue.cc:1048] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20260709 12:25:53.611843 29280 raft_consensus.cc:2955] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 31 LEADER]: Committing config change with OpId 31.18: config changed from index 5 to 18, VOTER e063541542a94969898868bf6eb5ce1b (127.16.29.67) evicted. New config: { opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:25:53.624644 18362 raft_consensus.cc:2955] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Committing config change with OpId 31.18: config changed from index 5 to 18, VOTER e063541542a94969898868bf6eb5ce1b (127.16.29.67) evicted. New config: { opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:25:53.644388 17571 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet e5dabc1347544bd8b4e81fa5903d4a8b with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 9)
I20260709 12:25:53.649008 17584 catalog_manager.cc:5697] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da reported cstate change: config changed from index 5 to 18, VOTER e063541542a94969898868bf6eb5ce1b (127.16.29.67) evicted. New cstate: current_term: 31 leader_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" committed_config { opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } } }
I20260709 12:25:53.715615 17821 tablet_service.cc:1558] Processing DeleteTablet for tablet e5dabc1347544bd8b4e81fa5903d4a8b with delete_type TABLET_DATA_TOMBSTONED (TS e063541542a94969898868bf6eb5ce1b not found in new config with opid_index 18) from {username='slave'} at 127.0.0.1:45728
W20260709 12:25:53.756210 17802 consensus_peers.cc:597] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:25:53.761353 18334 consensus_peers.cc:597] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20260709 12:25:53.764861 29290 tablet_replica.cc:333] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b: stopping tablet replica
W20260709 12:25:53.769068 17728 consensus_peers.cc:597] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 12:25:53.789974 29290 raft_consensus.cc:2243] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:53.790689 29290 raft_consensus.cc:2272] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:53.796358 29290 ts_tablet_manager.cc:1916] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20260709 12:25:53.798285 17802 consensus_peers.cc:597] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20260709 12:25:53.828961 29290 ts_tablet_manager.cc:1929] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 31.17
I20260709 12:25:53.829530 29290 log.cc:1199] T e5dabc1347544bd8b4e81fa5903d4a8b P e063541542a94969898868bf6eb5ce1b: Deleting WAL directory at /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599551912677-16501-0/minicluster-data/ts-2-root/wals/e5dabc1347544bd8b4e81fa5903d4a8b
I20260709 12:25:53.831314 17571 catalog_manager.cc:5028] TS e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837): tablet e5dabc1347544bd8b4e81fa5903d4a8b (table test-workload [id=ddc5b74250df4c65909b73b3742d615f]) successfully deleted
W20260709 12:25:53.851385 17802 consensus_peers.cc:597] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:25:54.009289 17802 consensus_peers.cc:597] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:25:54.028188 17802 consensus_peers.cc:597] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:25:54.086160 18334 consensus_peers.cc:597] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:25:54.112178 17802 consensus_peers.cc:597] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:25:54.115237 17802 consensus_peers.cc:597] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:25:54.235146 17728 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111) [suppressed 433 similar messages]
W20260709 12:25:54.254812 17802 consensus_peers.cc:597] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:25:54.263897 17802 consensus_peers.cc:597] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:25:54.285576 17802 consensus_peers.cc:597] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:25:54.292717 16501 thread.cc:527] Waited for 3000ms trying to join with diag-logger (tid 17710)
W20260709 12:25:54.396018 17802 consensus_peers.cc:597] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:25:54.479525 17728 consensus_peers.cc:597] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:25:54.579360 17802 consensus_peers.cc:597] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:25:54.590428 17728 consensus_peers.cc:597] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:25:54.706259 17802 consensus_peers.cc:597] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:25:54.801941 18334 consensus_peers.cc:597] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:25:54.820420 17728 consensus_peers.cc:597] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:25:54.877142 17802 consensus_peers.cc:597] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:25:54.919615 17802 consensus_peers.cc:597] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:25:54.984197 17728 consensus_peers.cc:597] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:25:55.013372 17728 consensus_peers.cc:597] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:25:55.132259 18334 consensus_peers.cc:597] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:25:55.204514 17728 consensus_peers.cc:597] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:25:55.256699 17728 consensus_peers.cc:597] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:25:55.296104 16501 thread.cc:527] Waited for 4000ms trying to join with diag-logger (tid 17710)
W20260709 12:25:55.343123 18334 consensus_peers.cc:597] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:25:55.432880 17728 consensus_peers.cc:597] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:25:55.547921 17728 consensus_peers.cc:597] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:25:55.554394 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 63084.5us
W20260709 12:25:55.589527 17728 consensus_peers.cc:597] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:25:55.634161 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 44674.3us
W20260709 12:25:55.601977 18334 consensus_peers.cc:597] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:25:55.635922 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 68902.9us
W20260709 12:25:55.636521 18334 consensus_peers.cc:597] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:25:55.637043 18334 consensus_peers.cc:597] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:25:55.670801 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 133148us
W20260709 12:25:55.704183 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 45280.9us
W20260709 12:25:55.668510 17728 consensus_peers.cc:597] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:25:55.717037 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 48533us
W20260709 12:25:55.727573 18334 consensus_peers.cc:597] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:25:55.842432 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 49620.2us
W20260709 12:25:55.907920 17728 consensus_peers.cc:597] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:25:55.960228 17728 consensus_peers.cc:597] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:25:55.968128 18334 consensus_peers.cc:597] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:25:56.168058 17728 consensus_peers.cc:597] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:25:56.251008 18334 consensus_peers.cc:597] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:25:56.296752 17728 consensus_peers.cc:597] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:25:56.370154 17728 consensus_peers.cc:597] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:25:56.373167 17802 consensus_peers.cc:597] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:25:56.399864 17802 consensus_peers.cc:597] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:25:56.400897 17802 consensus_peers.cc:597] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:25:56.632951 17802 consensus_peers.cc:597] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:25:56.675302 17802 consensus_peers.cc:597] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:25:56.799647 18334 consensus_peers.cc:597] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:25:56.904994 17802 consensus_peers.cc:597] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:25:56.906543 17802 consensus_peers.cc:597] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:25:56.987978 17728 consensus_peers.cc:597] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:25:57.049064 17802 consensus_peers.cc:597] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:25:57.149204 17728 consensus_peers.cc:597] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:25:57.224370 17802 consensus_peers.cc:597] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:25:57.279670 17802 consensus_peers.cc:597] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:25:57.353498 18334 consensus_peers.cc:597] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:25:57.386355 17802 consensus_peers.cc:597] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:25:57.440011 17802 consensus_peers.cc:597] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:25:57.461879 17728 consensus_peers.cc:597] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:25:57.585078 17728 consensus_peers.cc:597] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:25:57.746170 17728 consensus_peers.cc:597] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:25:57.759534 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 58217.7us
W20260709 12:25:57.873052 18334 consensus_peers.cc:597] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:25:57.877015 17802 consensus_peers.cc:597] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:25:57.892323 17802 consensus_peers.cc:597] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:25:57.900666 17802 consensus_peers.cc:597] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20260709 12:25:57.903059 16501 tablet_server.cc:196] TabletServer@127.16.29.65:0 shutdown complete.
W20260709 12:25:57.939379 18334 consensus_peers.cc:597] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:25:57.953658 17728 consensus_peers.cc:597] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:25:57.957362 17728 consensus_peers.cc:597] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:25:57.978636 17728 consensus_peers.cc:597] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:25:58.014981 17802 consensus_peers.cc:597] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:25:58.029625 17802 consensus_peers.cc:597] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:25:58.083524 17728 consensus_peers.cc:597] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:25:58.087466 17728 consensus_peers.cc:597] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:25:58.136277 18334 consensus_peers.cc:597] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:25:58.217509 17728 consensus_peers.cc:597] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:25:58.233855 17728 consensus_peers.cc:597] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:25:58.257520 18334 consensus_peers.cc:597] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20260709 12:25:58.293663 16501 tablet_server.cc:179] TabletServer@127.16.29.66:0 shutting down...
W20260709 12:25:58.295400 18334 consensus_peers.cc:597] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:25:58.299685 17801 consensus_peers.cc:597] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b -> Peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Couldn't send request to peer e2909cff1da44ad387f96a8ae3aa5afb. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:25:58.301460 18334 consensus_peers.cc:597] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:25:58.339336 17801 consensus_peers.cc:597] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b -> Peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Couldn't send request to peer e2909cff1da44ad387f96a8ae3aa5afb. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:25:58.353147 17801 consensus_peers.cc:597] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b -> Peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Couldn't send request to peer e2909cff1da44ad387f96a8ae3aa5afb. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:25:58.378958 17801 consensus_peers.cc:597] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b -> Peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Couldn't send request to peer e2909cff1da44ad387f96a8ae3aa5afb. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:25:58.404255 17801 consensus_peers.cc:597] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b -> Peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Couldn't send request to peer e2909cff1da44ad387f96a8ae3aa5afb. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:25:58.416998 18334 consensus_peers.cc:597] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:25:58.431983 17801 consensus_peers.cc:597] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b -> Peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Couldn't send request to peer e2909cff1da44ad387f96a8ae3aa5afb. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:25:58.442075 17802 consensus_peers.cc:597] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:25:58.443845 17801 consensus_peers.cc:597] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b -> Peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Couldn't send request to peer e2909cff1da44ad387f96a8ae3aa5afb. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:25:58.472046 17801 consensus_peers.cc:597] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b -> Peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Couldn't send request to peer e2909cff1da44ad387f96a8ae3aa5afb. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:25:58.526106 18333 consensus_peers.cc:597] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da -> Peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Couldn't send request to peer e2909cff1da44ad387f96a8ae3aa5afb. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:25:58.527508 18333 consensus_peers.cc:597] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da -> Peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Couldn't send request to peer e2909cff1da44ad387f96a8ae3aa5afb. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:25:58.528195 18333 consensus_peers.cc:597] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da -> Peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Couldn't send request to peer e2909cff1da44ad387f96a8ae3aa5afb. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:25:58.532775 18333 consensus_peers.cc:597] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da -> Peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Couldn't send request to peer e2909cff1da44ad387f96a8ae3aa5afb. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:25:58.543624 16501 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:25:58.545522 16501 tablet_replica.cc:333] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb: stopping tablet replica
W20260709 12:25:58.547622 18333 consensus_peers.cc:597] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da -> Peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Couldn't send request to peer e2909cff1da44ad387f96a8ae3aa5afb. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:25:58.551609 16501 raft_consensus.cc:2243] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 36 LEADER]: Raft consensus shutting down.
I20260709 12:25:58.555837 16501 raft_consensus.cc:2272] T 9439565c5dbb4f4ba3375aaef2c9add5 P e2909cff1da44ad387f96a8ae3aa5afb [term 36 FOLLOWER]: Raft consensus is shut down!
W20260709 12:25:58.556210 18333 consensus_peers.cc:597] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da -> Peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Couldn't send request to peer e2909cff1da44ad387f96a8ae3aa5afb. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:25:58.560333 16501 tablet_replica.cc:333] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb: stopping tablet replica
I20260709 12:25:58.561270 16501 raft_consensus.cc:2243] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 33 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:58.561992 16501 raft_consensus.cc:2272] T 22fa38faae58422e9cecc99973cab688 P e2909cff1da44ad387f96a8ae3aa5afb [term 33 FOLLOWER]: Raft consensus is shut down!
W20260709 12:25:58.571973 17801 consensus_peers.cc:597] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b -> Peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Couldn't send request to peer e2909cff1da44ad387f96a8ae3aa5afb. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:25:58.581229 16501 tablet_replica.cc:333] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb: stopping tablet replica
I20260709 12:25:58.582526 16501 raft_consensus.cc:2243] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 31 LEADER]: Raft consensus shutting down.
I20260709 12:25:58.586867 16501 raft_consensus.cc:2272] T ab84345f9aa742a7b85389e9b6f2ce90 P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:58.598767 16501 tablet_replica.cc:333] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb: stopping tablet replica
I20260709 12:25:58.599793 16501 raft_consensus.cc:2243] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 32 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:58.600611 16501 raft_consensus.cc:2272] T 56b500bd1b914ebfb3f19afc8a330665 P e2909cff1da44ad387f96a8ae3aa5afb [term 32 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:58.603960 16501 tablet_replica.cc:333] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb: stopping tablet replica
I20260709 12:25:58.604918 16501 raft_consensus.cc:2243] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 35 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:58.605603 16501 raft_consensus.cc:2272] T 9160d9f98b8c406db2771d9ea2e0995f P e2909cff1da44ad387f96a8ae3aa5afb [term 35 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:58.609176 16501 tablet_replica.cc:333] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb: stopping tablet replica
I20260709 12:25:58.610261 16501 raft_consensus.cc:2243] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 33 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:58.611236 16501 raft_consensus.cc:2272] T 10f96c39fba3496696f64385105bfeaa P e2909cff1da44ad387f96a8ae3aa5afb [term 33 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:58.614220 16501 tablet_replica.cc:333] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb: stopping tablet replica
I20260709 12:25:58.615165 16501 raft_consensus.cc:2243] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 30 LEADER]: Raft consensus shutting down.
I20260709 12:25:58.616482 16501 raft_consensus.cc:2272] T 1a698a75c7954dd7b24e7889cb19ae35 P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:58.621001 16501 tablet_replica.cc:333] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb: stopping tablet replica
I20260709 12:25:58.621958 16501 raft_consensus.cc:2243] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 35 LEADER]: Raft consensus shutting down.
I20260709 12:25:58.623196 16501 raft_consensus.cc:2272] T f5f0dd0890f44e35bbb838da838faa9c P e2909cff1da44ad387f96a8ae3aa5afb [term 35 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:58.626296 16501 tablet_replica.cc:333] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb: stopping tablet replica
I20260709 12:25:58.627280 16501 raft_consensus.cc:2243] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 34 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:58.628144 16501 raft_consensus.cc:2272] T e55c231d5fd5471285b2d7879a38f38b P e2909cff1da44ad387f96a8ae3aa5afb [term 34 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:58.630978 16501 tablet_replica.cc:333] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb: stopping tablet replica
I20260709 12:25:58.631825 16501 raft_consensus.cc:2243] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 38 LEADER]: Raft consensus shutting down.
W20260709 12:25:58.640571 17801 consensus_peers.cc:597] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b -> Peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Couldn't send request to peer e2909cff1da44ad387f96a8ae3aa5afb. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:25:58.643051 16501 raft_consensus.cc:2272] T 13a48228b0d44d3ea21f474203dd02e3 P e2909cff1da44ad387f96a8ae3aa5afb [term 38 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:58.646279 16501 tablet_replica.cc:333] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb: stopping tablet replica
I20260709 12:25:58.647174 16501 raft_consensus.cc:2243] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 LEADER]: Raft consensus shutting down.
I20260709 12:25:58.648326 16501 raft_consensus.cc:2272] T 8cab6f01c6a2485cb5024322a1bd5462 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Raft consensus is shut down!
W20260709 12:25:58.699843 18334 consensus_peers.cc:597] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:25:58.923281 18333 consensus_peers.cc:597] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da -> Peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Couldn't send request to peer e2909cff1da44ad387f96a8ae3aa5afb. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:25:59.130802 17728 consensus_peers.cc:597] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:25:59.131409 17728 consensus_peers.cc:597] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:25:59.132175 17728 consensus_peers.cc:597] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20260709 12:25:59.715895 16501 tablet_replica.cc:333] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb: stopping tablet replica
I20260709 12:25:59.720700 16501 raft_consensus.cc:2243] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 35 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:59.722636 16501 raft_consensus.cc:2272] T 00a02ab39fcd4078ad49697693cf3be8 P e2909cff1da44ad387f96a8ae3aa5afb [term 35 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:59.728091 16501 tablet_replica.cc:333] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb: stopping tablet replica
I20260709 12:25:59.733942 16501 raft_consensus.cc:2243] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 31 LEADER]: Raft consensus shutting down.
W20260709 12:25:59.734016 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 1.10498e+06us
I20260709 12:25:59.735545 29306 raft_consensus.cc:493] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:25:59.737092 16501 raft_consensus.cc:2272] T 69716f6d5e7549418fdd8d54d7b722ce P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:59.745518 29306 raft_consensus.cc:515] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:59.752183 16501 tablet_replica.cc:333] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb: stopping tablet replica
I20260709 12:25:59.752854 29306 leader_election.cc:290] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:59.753600 16501 raft_consensus.cc:2243] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:59.755573 16501 raft_consensus.cc:2272] T f8f6bfcb37be41c6917282a17239ad06 P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:59.785915 16501 tablet_replica.cc:333] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb: stopping tablet replica
I20260709 12:25:59.788071 16501 raft_consensus.cc:2243] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 LEADER]: Raft consensus shutting down.
I20260709 12:25:59.798086 16501 raft_consensus.cc:2272] T d48b7de191684318aa08f60a7a395387 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:59.822685 29308 raft_consensus.cc:493] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 34 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:25:59.823331 29306 raft_consensus.cc:493] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 33 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:25:59.830832 29307 raft_consensus.cc:493] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 37 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:25:59.830626 29306 raft_consensus.cc:515] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:59.823623 29308 raft_consensus.cc:515] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:59.824625 16501 tablet_replica.cc:333] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb: stopping tablet replica
I20260709 12:25:59.836066 29306 leader_election.cc:290] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:59.837030 29306 raft_consensus.cc:493] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 35 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:25:59.838713 29306 raft_consensus.cc:515] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:59.832284 29307 raft_consensus.cc:515] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:59.848773 16501 raft_consensus.cc:2243] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 34 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:59.850581 29307 leader_election.cc:290] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:25:59.852566 17729 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:59.852936 29308 leader_election.cc:290] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:59.853413 29306 leader_election.cc:290] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:59.876405 29313 raft_consensus.cc:493] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 35 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:25:59.894560 29280 raft_consensus.cc:493] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 33 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:25:59.895560 29280 raft_consensus.cc:515] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:59.898964 29280 leader_election.cc:290] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:59.885804 29313 raft_consensus.cc:515] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:59.913496 29317 raft_consensus.cc:493] T a5732e71b9d54f8786b8c511d69943bf P 334a4e1f59944544b3b3ff36616b57da [term 33 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
W20260709 12:25:59.917284 17801 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 393 similar messages]
I20260709 12:25:59.914661 29317 raft_consensus.cc:515] T a5732e71b9d54f8786b8c511d69943bf P 334a4e1f59944544b3b3ff36616b57da [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:59.942196 29313 leader_election.cc:290] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:59.960585 29314 raft_consensus.cc:493] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 37 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:25:59.961764 29314 raft_consensus.cc:515] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:25:59.972185 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 55111.3us
I20260709 12:25:59.972664 29326 raft_consensus.cc:493] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:25:59.973783 29326 raft_consensus.cc:515] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:59.976311 29320 raft_consensus.cc:493] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 35 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:25:59.976464 29317 leader_election.cc:290] T a5732e71b9d54f8786b8c511d69943bf P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:59.977283 29320 raft_consensus.cc:515] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:59.978860 29321 raft_consensus.cc:493] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:25:59.979702 29321 raft_consensus.cc:515] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:59.982600 29321 leader_election.cc:290] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:59.979717 29320 leader_election.cc:290] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:59.984124 29293 raft_consensus.cc:493] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 37 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:25:59.985306 29293 raft_consensus.cc:515] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:25:59.988786 29293 leader_election.cc:290] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:59.989408 29293 raft_consensus.cc:493] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:25:59.990147 29293 raft_consensus.cc:515] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:25:59.993377 29293 leader_election.cc:290] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:25:59.995440 29321 raft_consensus.cc:493] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:25:59.997327 29326 leader_election.cc:290] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:25:59.997279 29321 raft_consensus.cc:515] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:26:00.003734 29307 raft_consensus.cc:493] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:26:00.004575 29307 raft_consensus.cc:515] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:26:00.006619 17728 leader_election.cc:336] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 34 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111)
W20260709 12:26:00.007398 17728 leader_election.cc:336] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 36 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111)
W20260709 12:26:00.008611 17728 leader_election.cc:336] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 35 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111)
I20260709 12:26:00.007349 29307 leader_election.cc:290] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:26:00.011214 17728 leader_election.cc:336] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 34 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111)
I20260709 12:26:00.014225 29310 raft_consensus.cc:493] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 37 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:26:00.013801 29307 raft_consensus.cc:493] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:26:00.015189 29310 raft_consensus.cc:515] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:26:00.015852 29307 raft_consensus.cc:515] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:26:00.009403 29321 leader_election.cc:290] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:26:00.013564 17728 consensus_peers.cc:597] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20260709 12:26:00.038498 29319 raft_consensus.cc:493] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:26:00.039638 29319 raft_consensus.cc:515] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:26:00.042955 29319 leader_election.cc:290] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:26:00.047160 29310 leader_election.cc:290] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:26:00.015969 29320 raft_consensus.cc:493] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:26:00.071712 29314 leader_election.cc:290] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:26:00.072418 16501 raft_consensus.cc:2272] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [term 34 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:00.073009 29314 raft_consensus.cc:493] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:26:00.073737 29281 raft_consensus.cc:493] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:26:00.073258 29320 raft_consensus.cc:515] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:26:00.074107 29314 raft_consensus.cc:515] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:26:00.078998 29314 leader_election.cc:290] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:26:00.074776 29281 raft_consensus.cc:515] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:26:00.092567 17728 leader_election.cc:336] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111)
W20260709 12:26:00.093530 17728 leader_election.cc:336] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 38 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111)
W20260709 12:26:00.094010 17728 leader_election.cc:336] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111)
I20260709 12:26:00.099102 29281 leader_election.cc:290] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:26:00.105288 29277 raft_consensus.cc:493] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:26:00.106081 29277 raft_consensus.cc:515] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:26:00.095932 29280 raft_consensus.cc:493] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 32 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:26:00.108469 29280 raft_consensus.cc:515] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:26:00.110538 29280 leader_election.cc:290] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:26:00.111030 29280 raft_consensus.cc:493] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:26:00.111579 29280 raft_consensus.cc:515] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:26:00.115043 17728 leader_election.cc:336] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111)
I20260709 12:26:00.114441 29321 raft_consensus.cc:493] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:26:00.123404 29307 leader_election.cc:290] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:26:00.131526 29321 raft_consensus.cc:515] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:26:00.097265 29306 raft_consensus.cc:493] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 37 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
W20260709 12:26:00.124527 17728 leader_election.cc:336] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 33 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111)
I20260709 12:26:00.133471 29306 raft_consensus.cc:515] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:26:00.127228 29329 raft_consensus.cc:493] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:26:00.135459 29321 leader_election.cc:290] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:26:00.136685 29329 raft_consensus.cc:515] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:26:00.127753 29277 leader_election.cc:290] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:26:00.135689 29306 leader_election.cc:290] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:26:00.157163 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 59016.6us
I20260709 12:26:00.158700 29325 raft_consensus.cc:493] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:26:00.159583 29325 raft_consensus.cc:515] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:26:00.130489 29320 leader_election.cc:290] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:26:00.172232 29317 raft_consensus.cc:493] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:26:00.173260 29317 raft_consensus.cc:515] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:26:00.176703 29312 raft_consensus.cc:493] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:26:00.177678 29320 raft_consensus.cc:493] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:26:00.177599 29312 raft_consensus.cc:515] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:26:00.178632 29320 raft_consensus.cc:515] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:26:00.180819 29325 leader_election.cc:290] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:26:00.180961 29320 leader_election.cc:290] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:26:00.181912 29317 leader_election.cc:290] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:26:00.184089 16501 tablet_replica.cc:333] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb: stopping tablet replica
I20260709 12:26:00.185535 29317 raft_consensus.cc:493] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:26:00.186394 29317 raft_consensus.cc:515] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:26:00.185703 29330 raft_consensus.cc:493] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:26:00.188261 29324 raft_consensus.cc:493] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 36 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:26:00.188364 29330 raft_consensus.cc:515] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:26:00.189162 29324 raft_consensus.cc:515] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:26:00.189692 29315 raft_consensus.cc:493] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 33 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:26:00.190547 29315 raft_consensus.cc:515] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:26:00.191504 29330 leader_election.cc:290] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:26:00.192081 29314 raft_consensus.cc:493] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 36 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:26:00.192291 29337 raft_consensus.cc:493] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:26:00.191732 29324 leader_election.cc:290] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:26:00.193040 29323 raft_consensus.cc:493] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:26:00.193639 29311 raft_consensus.cc:493] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 38 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:26:00.185567 16501 raft_consensus.cc:2243] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:00.194226 29317 leader_election.cc:290] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:26:00.222434 29328 raft_consensus.cc:493] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:26:00.223559 29328 raft_consensus.cc:515] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:26:00.227046 29328 leader_election.cc:290] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:26:00.228061 29333 raft_consensus.cc:493] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:26:00.228919 29333 raft_consensus.cc:515] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:26:00.229733 29312 leader_election.cc:290] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:26:00.222502 29317 raft_consensus.cc:493] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 32 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:26:00.171288 29329 leader_election.cc:290] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:26:00.233994 29334 raft_consensus.cc:493] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:26:00.228084 29331 raft_consensus.cc:493] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:26:00.236799 29331 raft_consensus.cc:515] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:26:00.220275 29314 raft_consensus.cc:515] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:26:00.239296 29331 leader_election.cc:290] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:26:00.219077 29311 raft_consensus.cc:515] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:26:00.240767 16501 raft_consensus.cc:2272] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:00.241842 29280 leader_election.cc:290] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:26:00.201877 29310 raft_consensus.cc:493] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:26:00.242337 29330 raft_consensus.cc:493] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:26:00.244972 29319 raft_consensus.cc:493] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:26:00.247747 29320 raft_consensus.cc:493] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:26:00.247951 29330 raft_consensus.cc:515] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:26:00.248093 29319 raft_consensus.cc:515] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:26:00.248636 29320 raft_consensus.cc:515] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:26:00.250416 29330 leader_election.cc:290] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:26:00.250767 29320 leader_election.cc:290] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:26:00.251153 17728 leader_election.cc:336] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111)
I20260709 12:26:00.254235 29326 raft_consensus.cc:493] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:26:00.254576 29304 raft_consensus.cc:493] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 33 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:26:00.255113 29326 raft_consensus.cc:515] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:26:00.257572 17729 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:00.258976 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:26:00.255321 29304 raft_consensus.cc:515] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:26:00.260696 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f52bc21096e4519acb00ed1ea68c343" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 36 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:26:00.261242 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 38 candidate_status { last_received { term: 37 index: 25 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:26:00.259150 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed3f6bd4d904d42a3a48e346ed9227d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 34 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:26:00.261850 29304 leader_election.cc:290] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:26:00.261940 17838 raft_consensus.cc:2468] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 37.
I20260709 12:26:00.260892 29326 leader_election.cc:290] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:26:00.257011 29281 raft_consensus.cc:493] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:26:00.256333 29338 raft_consensus.cc:493] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:26:00.243062 29317 raft_consensus.cc:515] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:26:00.263098 29281 raft_consensus.cc:515] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:26:00.263424 29338 raft_consensus.cc:515] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:26:00.265573 29317 leader_election.cc:290] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:26:00.265690 29281 leader_election.cc:290] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:26:00.263909 29311 leader_election.cc:290] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:26:00.266134 29338 leader_election.cc:290] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:26:00.265945 29323 raft_consensus.cc:515] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:26:00.239756 29315 leader_election.cc:290] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:26:00.267869 29315 raft_consensus.cc:493] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:26:00.268540 29315 raft_consensus.cc:515] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:26:00.268972 29323 leader_election.cc:290] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:26:00.239727 29314 leader_election.cc:290] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:26:00.270622 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3de52d85935746e9a1e770fef3529988" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 34 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:26:00.271669 18334 leader_election.cc:336] T a5732e71b9d54f8786b8c511d69943bf P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 34 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111)
I20260709 12:26:00.242193 29332 raft_consensus.cc:493] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:26:00.273085 29332 raft_consensus.cc:515] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:26:00.277621 29333 leader_election.cc:290] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:26:00.235229 29334 raft_consensus.cc:515] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:26:00.250795 29319 leader_election.cc:290] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:26:00.281065 29334 leader_election.cc:290] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:26:00.233135 29337 raft_consensus.cc:515] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:26:00.287463 29337 leader_election.cc:290] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:26:00.349583 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 38 candidate_status { last_received { term: 37 index: 26 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:26:00.349957 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 32 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:26:00.350869 17836 raft_consensus.cc:2468] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 31.
W20260709 12:26:00.272393 18334 leader_election.cc:336] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111)
W20260709 12:26:00.351851 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 79481.7us
W20260709 12:26:00.352522 18334 consensus_peers.cc:597] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:26:00.352955 18334 leader_election.cc:336] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111)
W20260709 12:26:00.353410 18334 leader_election.cc:336] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111)
I20260709 12:26:00.270754 29315 leader_election.cc:290] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:26:00.322460 29281 raft_consensus.cc:493] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:26:00.355778 29281 raft_consensus.cc:515] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:26:00.322261 17801 consensus_peers.cc:597] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b -> Peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Couldn't send request to peer e2909cff1da44ad387f96a8ae3aa5afb. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:26:00.357975 29281 leader_election.cc:290] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:26:00.309932 29314 raft_consensus.cc:493] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 36 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
W20260709 12:26:00.358840 17859 debug-util.cc:398] Leaking SignalData structure 0x7b08001e28c0 after lost signal to thread 16504
W20260709 12:26:00.360040 17859 debug-util.cc:398] Leaking SignalData structure 0x7b080027ac20 after lost signal to thread 17621
W20260709 12:26:00.360656 17859 debug-util.cc:398] Leaking SignalData structure 0x7b08003a2220 after lost signal to thread 17788
W20260709 12:26:00.361878 17859 debug-util.cc:398] Leaking SignalData structure 0x7b0800408ea0 after lost signal to thread 17862
W20260709 12:26:00.363122 17859 debug-util.cc:398] Leaking SignalData structure 0x7b0800330260 after lost signal to thread 18409
I20260709 12:26:00.293282 17837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9aeb14eca4a43dcbeaccffdf7e28995" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:26:00.300323 29324 raft_consensus.cc:493] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:26:00.296993 29339 raft_consensus.cc:493] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:26:00.365190 29324 raft_consensus.cc:515] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:26:00.365377 29339 raft_consensus.cc:515] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:26:00.367565 29324 leader_election.cc:290] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:26:00.301039 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:26:00.324170 29334 raft_consensus.cc:493] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:26:00.326287 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2bcfe1b94464b6d9620cc3b8a17a2ad" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 38 candidate_status { last_received { term: 37 index: 24 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:26:00.315666 16501 tablet_replica.cc:333] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb: stopping tablet replica
I20260709 12:26:00.325445 29332 leader_election.cc:290] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:26:00.334254 17835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 32 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:26:00.336436 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 37 candidate_status { last_received { term: 36 index: 25 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:26:00.329901 17728 consensus_peers.cc:597] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:26:00.377553 17728 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 47759.8us
I20260709 12:26:00.344388 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19a1253e500942cd8a1a0f568159cfa3" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 33 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:26:00.345484 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11ed0a8f1094765abe028dc60aa8bed" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 35 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:26:00.379096 16501 raft_consensus.cc:2243] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 32 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:00.347601 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e726e4ebbb4ca58b6a6102c9b0850f" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 29 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:26:00.380012 16501 raft_consensus.cc:2272] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [term 32 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:00.316447 29317 raft_consensus.cc:493] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:26:00.380810 29317 raft_consensus.cc:515] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:26:00.349160 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 31 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:26:00.384199 29317 leader_election.cc:290] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:26:00.384979 29334 raft_consensus.cc:515] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:26:00.386381 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12b1fcd35e12402e8d6df7504c578e01" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 29 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:26:00.387370 18334 leader_election.cc:336] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111)
I20260709 12:26:00.318405 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:26:00.389122 18364 raft_consensus.cc:2468] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 31.
I20260709 12:26:00.368620 29339 leader_election.cc:290] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:26:00.296139 29310 raft_consensus.cc:515] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:26:00.387579 29334 leader_election.cc:290] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:26:00.390596 16501 tablet_replica.cc:333] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb: stopping tablet replica
I20260709 12:26:00.391638 29310 leader_election.cc:290] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:26:00.391736 16501 raft_consensus.cc:2243] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 LEADER]: Raft consensus shutting down.
I20260709 12:26:00.371929 29314 raft_consensus.cc:515] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:26:00.372831 29326 raft_consensus.cc:493] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:26:00.392949 16501 raft_consensus.cc:2272] T 19dad6ab92894d708ec8eb7e8ac28185 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:00.393460 29326 raft_consensus.cc:515] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:26:00.374348 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:26:00.396528 18374 raft_consensus.cc:2468] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 29.
I20260709 12:26:00.377558 17830 raft_consensus.cc:2468] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 36.
I20260709 12:26:00.344092 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 38 candidate_status { last_received { term: 37 index: 25 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:26:00.398520 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 32 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:26:00.399168 18374 raft_consensus.cc:2468] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 31.
I20260709 12:26:00.400070 29326 leader_election.cc:290] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:26:00.406526 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 37 candidate_status { last_received { term: 36 index: 25 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:26:00.388479 18334 leader_election.cc:336] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 37 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111)
W20260709 12:26:00.407430 18334 leader_election.cc:336] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111)
W20260709 12:26:00.408447 18334 leader_election.cc:336] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 37 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111)
I20260709 12:26:00.408645 16501 tablet_replica.cc:333] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb: stopping tablet replica
I20260709 12:26:00.410195 16501 raft_consensus.cc:2243] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 33 LEADER]: Raft consensus shutting down.
I20260709 12:26:00.412796 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 34 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:26:00.414209 17838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 36 candidate_status { last_received { term: 35 index: 21 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:26:00.428943 17729 leader_election.cc:304] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:26:00.429554 17838 raft_consensus.cc:2468] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 35.
W20260709 12:26:00.429751 17729 raft_consensus.cc:2686] T d11ed0a8f1094765abe028dc60aa8bed P e2909cff1da44ad387f96a8ae3aa5afb: unable to run election callback: Service unavailable: Thread pool token was shut down
I20260709 12:26:00.414920 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 38 candidate_status { last_received { term: 37 index: 26 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:26:00.415104 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 31 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:26:00.415781 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea2f2b5792f84bbba2738f1ce1e90122" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:26:00.431646 17727 leader_election.cc:304] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:26:00.432940 29335 raft_consensus.cc:2749] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:26:00.417254 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d522f70beb1f417cb98940d90b594823" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 31 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:26:00.418587 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 38 candidate_status { last_received { term: 37 index: 26 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:26:00.434224 18361 raft_consensus.cc:2468] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 30.
I20260709 12:26:00.434530 18364 raft_consensus.cc:2468] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2909cff1da44ad387f96a8ae3aa5afb in term 37.
I20260709 12:26:00.418846 29314 leader_election.cc:290] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:26:00.419513 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 32 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:26:00.424281 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 32 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:26:00.417066 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f6bfcb37be41c6917282a17239ad06" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:26:00.435803 17727 leader_election.cc:304] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:26:00.437430 17729 leader_election.cc:304] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:26:00.438279 17727 leader_election.cc:304] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:26:00.447968 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 29 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:26:00.448652 17829 raft_consensus.cc:2468] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 28.
I20260709 12:26:00.449286 17727 leader_election.cc:304] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:26:00.449643 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3425997bcf4fe7abc0399fd06e54d9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 31 candidate_status { last_received { term: 30 index: 23 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:26:00.438973 29280 raft_consensus.cc:2749] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:26:00.451077 29306 raft_consensus.cc:2804] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 37 FOLLOWER]: Leader pre-election won for term 38
I20260709 12:26:00.451558 29306 raft_consensus.cc:493] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 37 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:26:00.451084 29277 raft_consensus.cc:2749] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:26:00.452041 29306 raft_consensus.cc:3060] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 37 FOLLOWER]: Advancing to term 38
W20260709 12:26:00.455307 17802 consensus_peers.cc:597] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:26:00.441244 18334 leader_election.cc:336] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111)
I20260709 12:26:00.461858 17727 leader_election.cc:304] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
W20260709 12:26:00.462198 18334 leader_election.cc:336] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 34 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111)
I20260709 12:26:00.444357 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23c0cde5227f470388b8ed1fc0786119" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 37 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:26:00.465559 18334 leader_election.cc:336] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 33 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111)
I20260709 12:26:00.445613 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:26:00.446439 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45397041e3ac43c8901c5cb03f27c7b9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 32 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
W20260709 12:26:00.466180 18334 leader_election.cc:336] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111)
I20260709 12:26:00.447175 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:26:00.434223 16501 raft_consensus.cc:2272] T 44ad5363c5954da4a1bff5857780fd5c P e2909cff1da44ad387f96a8ae3aa5afb [term 33 FOLLOWER]: Raft consensus is shut down!
W20260709 12:26:00.462795 17727 raft_consensus.cc:2686] T 0e02889970bc40dc8ca2e81483f211e0 P e2909cff1da44ad387f96a8ae3aa5afb: unable to run election callback: Service unavailable: Thread pool token was shut down
I20260709 12:26:00.467398 17727 leader_election.cc:304] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:26:00.468266 29335 raft_consensus.cc:2749] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:26:00.470077 29280 raft_consensus.cc:2749] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:26:00.471896 17727 leader_election.cc:304] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:26:00.474449 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 36 candidate_status { last_received { term: 35 index: 21 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:26:00.475299 18374 raft_consensus.cc:2468] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 35.
I20260709 12:26:00.467525 17729 leader_election.cc:304] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:26:00.475584 16501 tablet_replica.cc:333] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb: stopping tablet replica
W20260709 12:26:00.476991 18334 leader_election.cc:336] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111)
I20260709 12:26:00.480895 29280 raft_consensus.cc:2804] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Leader pre-election won for term 32
W20260709 12:26:00.481277 18334 leader_election.cc:336] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111)
I20260709 12:26:00.480572 29335 raft_consensus.cc:2749] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:26:00.481336 29280 raft_consensus.cc:493] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:26:00.482369 29280 raft_consensus.cc:3060] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:26:00.479326 16501 raft_consensus.cc:2243] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 31 LEADER]: Raft consensus shutting down.
I20260709 12:26:00.498457 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9160d9f98b8c406db2771d9ea2e0995f" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 36 candidate_status { last_received { term: 35 index: 23 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:26:00.499840 16501 raft_consensus.cc:2272] T ea7da467817a4ba88aa5797c3f8f493f P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:00.507575 16501 tablet_replica.cc:333] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb: stopping tablet replica
W20260709 12:26:00.496081 17802 consensus_peers.cc:597] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:26:00.513676 17802 consensus_peers.cc:597] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 12:26:00.487483 29306 raft_consensus.cc:515] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 38 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:26:00.514415 17802 consensus_peers.cc:597] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:26:00.514868 17802 leader_election.cc:336] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111)
I20260709 12:26:00.496074 17727 leader_election.cc:304] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:26:00.495002 17729 leader_election.cc:304] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 
I20260709 12:26:00.517549 29335 raft_consensus.cc:2804] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:26:00.517961 29335 raft_consensus.cc:493] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:26:00.518378 29335 raft_consensus.cc:3060] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Advancing to term 30
W20260709 12:26:00.518985 17727 raft_consensus.cc:2686] T 19a1253e500942cd8a1a0f568159cfa3 P e2909cff1da44ad387f96a8ae3aa5afb: unable to run election callback: Service unavailable: Thread pool token was shut down
I20260709 12:26:00.522984 17727 leader_election.cc:304] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
W20260709 12:26:00.512439 17801 consensus_peers.cc:597] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b -> Peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Couldn't send request to peer e2909cff1da44ad387f96a8ae3aa5afb. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:26:00.515806 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 38 candidate_status { last_received { term: 37 index: 25 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:26:00.518224 17729 leader_election.cc:304] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
I20260709 12:26:00.515981 29306 leader_election.cc:290] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 38 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:26:00.529270 17801 consensus_peers.cc:597] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b -> Peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Couldn't send request to peer e2909cff1da44ad387f96a8ae3aa5afb. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:26:00.529336 29321 raft_consensus.cc:2804] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:26:00.529894 29321 raft_consensus.cc:493] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
W20260709 12:26:00.530273 17801 leader_election.cc:336] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 36 pre-election: RPC error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:00.523821 29277 raft_consensus.cc:2749] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
W20260709 12:26:00.531553 17801 consensus_peers.cc:597] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b -> Peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Couldn't send request to peer e2909cff1da44ad387f96a8ae3aa5afb. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:26:00.532276 17801 consensus_peers.cc:597] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b -> Peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Couldn't send request to peer e2909cff1da44ad387f96a8ae3aa5afb. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:26:00.534183 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:26:00.534539 17801 leader_election.cc:336] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 36 pre-election: RPC error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:00.530328 29321 raft_consensus.cc:3060] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Advancing to term 32
W20260709 12:26:00.535315 17801 leader_election.cc:336] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 38 pre-election: RPC error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:00.513780 16501 raft_consensus.cc:2243] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 LEADER]: Raft consensus shutting down.
I20260709 12:26:00.537326 16501 raft_consensus.cc:2272] T 8e417cb980e44a77962d2e3dfd58aad0 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Raft consensus is shut down!
W20260709 12:26:00.515705 17802 consensus_peers.cc:597] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20260709 12:26:00.549516 29321 raft_consensus.cc:515] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 32 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:26:00.553270 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb3768d215e409fb9f5baf18d124442" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 29 candidate_status { last_received { term: 28 index: 24 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:26:00.524227 17727 leader_election.cc:304] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:26:00.568909 16501 tablet_replica.cc:333] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb: stopping tablet replica
I20260709 12:26:00.559276 29335 raft_consensus.cc:515] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:26:00.608817 16501 raft_consensus.cc:2243] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 33 LEADER]: Raft consensus shutting down.
I20260709 12:26:00.609421 17803 leader_election.cc:304] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:26:00.609850 29335 leader_election.cc:290] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 30 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:26:00.552412 18334 leader_election.cc:336] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111)
I20260709 12:26:00.563716 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 37 candidate_status { last_received { term: 36 index: 25 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:26:00.610846 18334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 58482.4us
W20260709 12:26:00.611361 18334 leader_election.cc:336] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111)
I20260709 12:26:00.611588 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 29 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
W20260709 12:26:00.611966 18334 leader_election.cc:336] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111)
W20260709 12:26:00.565778 17801 consensus_peers.cc:597] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b -> Peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Couldn't send request to peer e2909cff1da44ad387f96a8ae3aa5afb. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:26:00.612294 18374 raft_consensus.cc:2468] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 28.
W20260709 12:26:00.612828 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 47118.3us
W20260709 12:26:00.614049 17801 leader_election.cc:336] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:26:00.614892 17801 leader_election.cc:336] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 37 pre-election: RPC error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:00.521678 29280 raft_consensus.cc:515] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 32 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:26:00.617620 29280 leader_election.cc:290] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 32 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:26:00.570132 17729 leader_election.cc:304] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb; no voters: e063541542a94969898868bf6eb5ce1b
W20260709 12:26:00.618621 17729 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 48596.4us
I20260709 12:26:00.612319 16501 raft_consensus.cc:2272] T a5732e71b9d54f8786b8c511d69943bf P e2909cff1da44ad387f96a8ae3aa5afb [term 33 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:00.621538 29303 raft_consensus.cc:2804] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 37 FOLLOWER]: Leader pre-election won for term 38
I20260709 12:26:00.621994 29303 raft_consensus.cc:493] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 37 FOLLOWER]: Starting leader election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:26:00.622391 29303 raft_consensus.cc:3060] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:26:00.570930 29277 raft_consensus.cc:2749] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:26:00.623724 16501 tablet_replica.cc:333] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb: stopping tablet replica
I20260709 12:26:00.624861 16501 raft_consensus.cc:2243] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 31 LEADER]: Raft consensus shutting down.
I20260709 12:26:00.626592 16501 raft_consensus.cc:2272] T e5dabc1347544bd8b4e81fa5903d4a8b P e2909cff1da44ad387f96a8ae3aa5afb [term 31 FOLLOWER]: Raft consensus is shut down!
W20260709 12:26:00.571221 17727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:36837, user_credentials={real_user=slave}} blocked reactor thread for 47069.8us
I20260709 12:26:00.628935 17727 leader_election.cc:304] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da
I20260709 12:26:00.629423 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:26:00.630647 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
W20260709 12:26:00.572837 17802 leader_election.cc:336] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111)
I20260709 12:26:00.631637 29306 raft_consensus.cc:2804] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 36 FOLLOWER]: Leader pre-election won for term 37
W20260709 12:26:00.632097 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 59297.1us
I20260709 12:26:00.632437 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 38 candidate_status { last_received { term: 37 index: 25 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
W20260709 12:26:00.632583 17802 leader_election.cc:336] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 34 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111)
W20260709 12:26:00.633088 17802 leader_election.cc:336] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111)
W20260709 12:26:00.633538 17802 leader_election.cc:336] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111)
W20260709 12:26:00.633975 17802 leader_election.cc:336] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111)
W20260709 12:26:00.634433 17802 leader_election.cc:336] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 39 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111)
W20260709 12:26:00.634847 17802 leader_election.cc:336] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 33 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111)
W20260709 12:26:00.634976 17801 leader_election.cc:336] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:26:00.635289 17802 leader_election.cc:336] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111)
I20260709 12:26:00.573356 29321 leader_election.cc:290] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 32 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:26:00.574216 18361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3219dcbfd86340ecbde507517060ff81" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:26:00.585987 18364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:26:00.592340 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 34 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:26:00.593742 18362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044ba5aee1d6499a975c6b29d932fd95" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:26:00.595186 18367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4dcc70202ad4cd6880749fbb3a6d0c6" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 33 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:26:00.595925 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e02889970bc40dc8ca2e81483f211e0" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 32 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:26:00.596946 18363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ad2828ec58f490a9ea564d9805b56b7" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 29 candidate_status { last_received { term: 28 index: 22 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:26:00.604513 18366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64345b0a2614ba4a382b83feb235cad" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 32 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:26:00.607363 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:26:00.610455 29310 raft_consensus.cc:2804] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 35 FOLLOWER]: Leader pre-election won for term 36
W20260709 12:26:00.635735 17801 leader_election.cc:336] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 34 pre-election: RPC error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:26:00.648681 18334 leader_election.cc:336] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 37 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111)
I20260709 12:26:00.633531 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 32 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:26:00.675900 29310 raft_consensus.cc:493] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 35 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:26:00.632118 29306 raft_consensus.cc:493] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 36 FOLLOWER]: Starting leader election (detected failure of leader 334a4e1f59944544b3b3ff36616b57da)
I20260709 12:26:00.682739 29310 raft_consensus.cc:3060] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:26:00.683266 17801 leader_election.cc:304] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:26:00.683028 29306 raft_consensus.cc:3060] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:26:00.674351 16501 tablet_replica.cc:333] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb: stopping tablet replica
W20260709 12:26:00.635767 17802 leader_election.cc:336] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111)
I20260709 12:26:00.695139 29315 raft_consensus.cc:2749] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
W20260709 12:26:00.695284 17802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:33071, user_credentials={real_user=slave}} blocked reactor thread for 59259.7us
W20260709 12:26:00.696228 17801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 60520.1us
W20260709 12:26:00.697808 17801 leader_election.cc:336] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:00.698624 17801 leader_election.cc:304] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:26:00.700893 29315 raft_consensus.cc:2749] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
W20260709 12:26:00.702400 17801 leader_election.cc:336] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:00.704335 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b56346a04a14c5d9f73c5205b74cf2d" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 38 candidate_status { last_received { term: 37 index: 25 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:26:00.704890 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "217734cc2eb24a7a8d92bfef30ee5c52" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:26:00.705150 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11896555f5d7458dbb02a3203b27f1cf" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 32 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:26:00.705681 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4092aabb101d411798b0354f88a5dd2b" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:26:00.719403 17801 leader_election.cc:304] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:26:00.719666 17803 leader_election.cc:304] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:26:00.721125 29303 raft_consensus.cc:515] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 38 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:26:00.721983 29319 raft_consensus.cc:2749] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:26:00.722752 29315 raft_consensus.cc:2749] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:26:00.720492 29306 raft_consensus.cc:515] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 37 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:26:00.724764 16501 raft_consensus.cc:2243] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 37 FOLLOWER]: Raft consensus shutting down.
W20260709 12:26:00.727197 17801 leader_election.cc:336] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 39 pre-election: RPC error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:00.727800 16501 raft_consensus.cc:2272] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [term 37 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:00.727738 17801 leader_election.cc:304] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:26:00.728817 17803 leader_election.cc:304] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:26:00.729503 29315 raft_consensus.cc:2749] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 38 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
I20260709 12:26:00.729849 29319 raft_consensus.cc:2749] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:26:00.730468 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 37 candidate_status { last_received { term: 36 index: 25 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
W20260709 12:26:00.730895 17801 leader_election.cc:336] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:00.732046 17803 leader_election.cc:304] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:26:00.732911 29319 raft_consensus.cc:2749] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:26:00.728994 29306 leader_election.cc:290] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 37 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:26:00.737496 16501 tablet_replica.cc:333] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb: stopping tablet replica
I20260709 12:26:00.738659 16501 raft_consensus.cc:2243] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 38 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:00.739396 16501 raft_consensus.cc:2272] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [term 38 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:00.714442 29310 raft_consensus.cc:515] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 36 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:26:00.745270 29310 leader_election.cc:290] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 36 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:26:00.754349 17727 leader_election.cc:304] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 38 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:26:00.757838 17727 leader_election.cc:304] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:26:00.759958 17727 leader_election.cc:304] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:26:00.757390 29280 raft_consensus.cc:2749] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 38 FOLLOWER]: Leader election lost for term 38. Reason: could not achieve majority
I20260709 12:26:00.759688 17803 leader_election.cc:304] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:26:00.762143 29303 leader_election.cc:290] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 38 election: Requested vote from peers e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:26:00.767853 17802 consensus_peers.cc:597] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:26:00.769366 17802 consensus_peers.cc:597] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20260709 12:26:00.771919 17727 leader_election.cc:304] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
I20260709 12:26:00.772213 17834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 38 candidate_status { last_received { term: 37 index: 26 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:26:00.773145 17836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4d9071484d4dc0998db462cb01a10f" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:26:00.773001 17832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a208e793881438b975b83755dc05cfa" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:26:00.776376 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28062a50142448aa97994888b4765920" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 38 candidate_status { last_received { term: 37 index: 26 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:26:00.777256 17833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5462347eb74482b8cd11beb1d39b8a9" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:26:00.777554 17830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fcb5bdaa4be4d619bb945ffd44090ba" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:26:00.777276 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:26:00.760423 29335 raft_consensus.cc:2749] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
I20260709 12:26:00.780789 17829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f940f29438f475294357324452abe76" candidate_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" candidate_term: 37 candidate_status { last_received { term: 36 index: 25 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
W20260709 12:26:00.777827 17801 leader_election.cc:336] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:00.770824 29311 raft_consensus.cc:2749] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:26:00.775769 29303 raft_consensus.cc:2749] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
I20260709 12:26:00.778362 29306 raft_consensus.cc:2749] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
W20260709 12:26:00.787925 17801 leader_election.cc:336] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:00.820392 16501 tablet_replica.cc:333] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb: stopping tablet replica
I20260709 12:26:00.821683 16501 raft_consensus.cc:2243] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 32 LEADER]: Raft consensus shutting down.
I20260709 12:26:00.823138 16501 raft_consensus.cc:2272] T 84421f094faa40f39b30b4c3d0737639 P e2909cff1da44ad387f96a8ae3aa5afb [term 32 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:00.826527 17727 leader_election.cc:304] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 38 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
W20260709 12:26:00.827746 17727 raft_consensus.cc:2686] T 28062a50142448aa97994888b4765920 P e2909cff1da44ad387f96a8ae3aa5afb: unable to run election callback: Service unavailable: Thread pool token was shut down
W20260709 12:26:00.828824 18333 consensus_peers.cc:597] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da -> Peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Couldn't send request to peer e2909cff1da44ad387f96a8ae3aa5afb. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:26:00.828866 17727 leader_election.cc:304] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb [CANDIDATE]: Term 37 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2909cff1da44ad387f96a8ae3aa5afb; no voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b
W20260709 12:26:00.829749 17727 raft_consensus.cc:2686] T 0f940f29438f475294357324452abe76 P e2909cff1da44ad387f96a8ae3aa5afb: unable to run election callback: Service unavailable: Thread pool token was shut down
I20260709 12:26:00.837136 16501 tablet_replica.cc:333] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb: stopping tablet replica
W20260709 12:26:00.829499 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 75661.1us
I20260709 12:26:00.838421 16501 raft_consensus.cc:2243] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 32 FOLLOWER]: Raft consensus shutting down.
W20260709 12:26:00.864197 17802 consensus_peers.cc:597] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:26:00.853089 17801 leader_election.cc:336] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 36 election: RPC error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:00.884835 18333 leader_election.cc:304] T a5732e71b9d54f8786b8c511d69943bf P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
W20260709 12:26:00.886963 17802 consensus_peers.cc:597] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 12:26:00.839320 16501 raft_consensus.cc:2272] T 4092aabb101d411798b0354f88a5dd2b P e2909cff1da44ad387f96a8ae3aa5afb [term 32 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:00.892107 29314 raft_consensus.cc:2749] T a5732e71b9d54f8786b8c511d69943bf P 334a4e1f59944544b3b3ff36616b57da [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:26:00.893334 18333 leader_election.cc:304] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:26:00.895196 16501 tablet_replica.cc:333] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb: stopping tablet replica
I20260709 12:26:00.904182 29314 raft_consensus.cc:2804] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:26:00.904774 29314 raft_consensus.cc:493] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 35 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:26:00.905236 29314 raft_consensus.cc:3060] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:26:00.905012 16501 raft_consensus.cc:2243] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 32 LEADER]: Raft consensus shutting down.
I20260709 12:26:00.907465 16501 raft_consensus.cc:2272] T 154e6b769a8d476ba26f7fe3a521fc8d P e2909cff1da44ad387f96a8ae3aa5afb [term 32 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:00.912717 18333 leader_election.cc:304] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:26:00.916880 29326 raft_consensus.cc:2749] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:26:00.918118 18333 leader_election.cc:304] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:26:00.920102 29326 raft_consensus.cc:2749] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:26:00.920369 16501 tablet_replica.cc:333] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb: stopping tablet replica
I20260709 12:26:00.922422 18333 leader_election.cc:304] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:26:00.928574 29326 raft_consensus.cc:2804] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:26:00.928990 29326 raft_consensus.cc:493] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:26:00.929414 29326 raft_consensus.cc:3060] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:26:00.929750 16501 raft_consensus.cc:2243] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 33 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:00.933314 16501 raft_consensus.cc:2272] T 3de52d85935746e9a1e770fef3529988 P e2909cff1da44ad387f96a8ae3aa5afb [term 33 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:00.935739 18333 leader_election.cc:304] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:26:00.942327 29334 raft_consensus.cc:2749] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:26:00.944135 18333 leader_election.cc:304] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:26:00.945127 29334 raft_consensus.cc:2749] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:26:00.945416 16501 tablet_replica.cc:333] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb: stopping tablet replica
I20260709 12:26:00.946543 16501 raft_consensus.cc:2243] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 35 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:00.947218 18333 leader_election.cc:304] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:26:00.952646 29334 raft_consensus.cc:2749] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:26:00.955101 18333 leader_election.cc:304] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:26:00.926877 29314 raft_consensus.cc:515] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 36 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:26:00.959113 29314 leader_election.cc:290] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 36 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:26:00.940708 29326 raft_consensus.cc:515] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:26:00.960464 29334 raft_consensus.cc:2749] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:26:00.964493 29326 leader_election.cc:290] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 29 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:26:00.961660 18333 leader_election.cc:304] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:26:00.966392 18333 leader_election.cc:304] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:26:00.947722 16501 raft_consensus.cc:2272] T 5f52bc21096e4519acb00ed1ea68c343 P e2909cff1da44ad387f96a8ae3aa5afb [term 35 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:00.974941 29326 raft_consensus.cc:2749] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:26:00.981753 18333 leader_election.cc:304] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:26:00.982577 29334 raft_consensus.cc:2749] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:26:00.982722 29326 raft_consensus.cc:2749] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:26:00.982571 16501 tablet_replica.cc:333] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb: stopping tablet replica
I20260709 12:26:00.998805 16501 raft_consensus.cc:2243] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 33 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:01.000275 17803 leader_election.cc:304] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:26:01.000917 16501 raft_consensus.cc:2272] T 3ed3f6bd4d904d42a3a48e346ed9227d P e2909cff1da44ad387f96a8ae3aa5afb [term 33 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:01.001570 29332 raft_consensus.cc:2749] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:26:01.002487 17803 leader_election.cc:304] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:26:01.010228 17803 leader_election.cc:304] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:26:01.015568 29310 raft_consensus.cc:2804] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:26:01.016081 29310 raft_consensus.cc:493] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:26:01.016465 29310 raft_consensus.cc:3060] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:26:01.017768 17803 leader_election.cc:304] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:26:01.019340 17803 leader_election.cc:304] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:26:01.020089 29315 raft_consensus.cc:2749] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:26:01.020941 29315 raft_consensus.cc:2749] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:26:01.023442 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 36 candidate_status { last_received { term: 35 index: 21 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:26:01.024318 17831 raft_consensus.cc:2393] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 36 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 36: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
I20260709 12:26:01.024173 29341 raft_consensus.cc:2749] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:26:01.025800 17803 leader_election.cc:304] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:26:01.027297 17803 leader_election.cc:304] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:26:01.027977 29341 raft_consensus.cc:2749] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:26:01.032760 17803 leader_election.cc:304] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:26:01.035784 29341 raft_consensus.cc:2749] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:26:01.032343 29332 raft_consensus.cc:2749] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:26:01.037606 17803 leader_election.cc:304] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, 334a4e1f59944544b3b3ff36616b57da
I20260709 12:26:01.029971 29310 raft_consensus.cc:515] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:26:01.038937 29332 raft_consensus.cc:2749] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:26:01.041083 29310 leader_election.cc:290] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 29 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:26:01.041268 17803 leader_election.cc:304] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:26:01.042513 29310 raft_consensus.cc:2749] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:26:01.049861 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 29 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:26:01.050704 17831 raft_consensus.cc:2393] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate 334a4e1f59944544b3b3ff36616b57da in current term 29: Already voted for candidate e063541542a94969898868bf6eb5ce1b in this term.
W20260709 12:26:01.053009 17801 leader_election.cc:336] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 29 election: RPC error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:01.105207 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 36 candidate_status { last_received { term: 35 index: 21 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:26:01.106554 18368 raft_consensus.cc:2393] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 36 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 36: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:26:01.109740 16501 tablet_replica.cc:333] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb: stopping tablet replica
I20260709 12:26:01.120326 16501 raft_consensus.cc:2243] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Raft consensus shutting down.
W20260709 12:26:01.131686 17802 consensus_peers.cc:597] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:26:01.134137 18334 consensus_peers.cc:597] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20260709 12:26:01.138770 16501 raft_consensus.cc:2272] T 12b1fcd35e12402e8d6df7504c578e01 P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:01.142976 16501 tablet_replica.cc:333] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb: stopping tablet replica
I20260709 12:26:01.144030 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 29 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:26:01.145617 18371 raft_consensus.cc:2393] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate e063541542a94969898868bf6eb5ce1b in current term 29: Already voted for candidate 334a4e1f59944544b3b3ff36616b57da in this term.
I20260709 12:26:01.147219 16501 raft_consensus.cc:2243] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 LEADER]: Raft consensus shutting down.
I20260709 12:26:01.148120 17803 leader_election.cc:304] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 36 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:26:01.152536 29341 raft_consensus.cc:2749] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 36 FOLLOWER]: Leader election lost for term 36. Reason: could not achieve majority
I20260709 12:26:01.157313 16501 raft_consensus.cc:2272] T 50f9ff8f24a14a079db34b1ba88fa278 P e2909cff1da44ad387f96a8ae3aa5afb [term 29 FOLLOWER]: Raft consensus is shut down!
W20260709 12:26:01.150470 18333 leader_election.cc:336] T a5732e71b9d54f8786b8c511d69943bf P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 34 pre-election: RPC error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:01.163920 16501 tablet_replica.cc:333] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb: stopping tablet replica
I20260709 12:26:01.165822 16501 raft_consensus.cc:2243] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 38 FOLLOWER]: Raft consensus shutting down.
W20260709 12:26:01.164631 18333 leader_election.cc:336] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 36 pre-election: RPC error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:26:01.167171 18333 leader_election.cc:336] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:26:01.168243 18333 leader_election.cc:336] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 38 pre-election: RPC error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:01.168640 18333 leader_election.cc:304] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:26:01.169381 29314 raft_consensus.cc:2749] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
W20260709 12:26:01.171155 18333 leader_election.cc:336] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:01.171597 18333 leader_election.cc:304] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:26:01.172609 16501 raft_consensus.cc:2272] T 8b56346a04a14c5d9f73c5205b74cf2d P e2909cff1da44ad387f96a8ae3aa5afb [term 38 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:01.174654 29314 raft_consensus.cc:2749] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
W20260709 12:26:01.176079 18333 leader_election.cc:336] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:01.176527 18333 leader_election.cc:304] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:26:01.179206 16501 tablet_replica.cc:333] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb: stopping tablet replica
I20260709 12:26:01.180229 16501 raft_consensus.cc:2243] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 36 LEADER]: Raft consensus shutting down.
I20260709 12:26:01.181967 16501 raft_consensus.cc:2272] T ac984176358d453a8800cc9e0e304061 P e2909cff1da44ad387f96a8ae3aa5afb [term 36 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:01.185848 16501 tablet_replica.cc:333] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb: stopping tablet replica
I20260709 12:26:01.186785 29314 raft_consensus.cc:2749] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
W20260709 12:26:01.188546 18333 leader_election.cc:336] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:01.189005 18333 leader_election.cc:304] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:26:01.195073 29314 raft_consensus.cc:2749] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:26:01.204087 16501 raft_consensus.cc:2243] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 LEADER]: Raft consensus shutting down.
I20260709 12:26:01.205336 16501 raft_consensus.cc:2272] T 1a4040191930435786e2eeea338bb3d3 P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Raft consensus is shut down!
W20260709 12:26:01.195505 18333 leader_election.cc:336] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:01.207307 18333 leader_election.cc:304] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:26:01.208089 29314 raft_consensus.cc:2749] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
W20260709 12:26:01.209281 18333 leader_election.cc:336] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 37 pre-election: RPC error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:01.209699 18333 leader_election.cc:304] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:26:01.210462 29314 raft_consensus.cc:2749] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
W20260709 12:26:01.211722 18333 leader_election.cc:336] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:01.212174 18333 leader_election.cc:304] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:26:01.212874 29314 raft_consensus.cc:2749] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
W20260709 12:26:01.214241 18333 leader_election.cc:336] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:01.214658 18333 leader_election.cc:304] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:26:01.215741 29314 raft_consensus.cc:2749] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
W20260709 12:26:01.216187 17802 consensus_peers.cc:597] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20260709 12:26:01.226338 16501 tablet_replica.cc:333] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb: stopping tablet replica
I20260709 12:26:01.227418 16501 raft_consensus.cc:2243] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Raft consensus shutting down.
W20260709 12:26:01.237879 18333 leader_election.cc:336] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 34 pre-election: RPC error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:01.238376 18333 leader_election.cc:304] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:26:01.239071 29314 raft_consensus.cc:2749] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
W20260709 12:26:01.240211 18333 leader_election.cc:336] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:01.240621 18333 leader_election.cc:304] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:26:01.241317 29314 raft_consensus.cc:2749] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
W20260709 12:26:01.242462 18333 leader_election.cc:336] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 33 pre-election: RPC error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:01.242828 18333 leader_election.cc:304] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:26:01.243474 29314 raft_consensus.cc:2749] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:26:01.244221 16501 raft_consensus.cc:2272] T 53e726e4ebbb4ca58b6a6102c9b0850f P e2909cff1da44ad387f96a8ae3aa5afb [term 28 FOLLOWER]: Raft consensus is shut down!
W20260709 12:26:01.250138 18334 consensus_peers.cc:597] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:26:01.251083 18333 leader_election.cc:336] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:26:01.252314 18333 leader_election.cc:336] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:01.252832 18333 leader_election.cc:304] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:26:01.253666 29314 raft_consensus.cc:2749] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
W20260709 12:26:01.254837 18333 leader_election.cc:336] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:01.255229 18333 leader_election.cc:304] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:26:01.255927 29314 raft_consensus.cc:2749] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
W20260709 12:26:01.257457 18333 leader_election.cc:336] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 37 pre-election: RPC error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:01.263219 16501 tablet_replica.cc:333] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb: stopping tablet replica
I20260709 12:26:01.258139 18333 leader_election.cc:304] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:26:01.265447 29326 raft_consensus.cc:2749] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:26:01.267658 16501 raft_consensus.cc:2243] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 37 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:01.272940 16501 raft_consensus.cc:2272] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e2909cff1da44ad387f96a8ae3aa5afb [term 37 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:01.278471 17803 leader_election.cc:304] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 334a4e1f59944544b3b3ff36616b57da, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:26:01.279654 29315 raft_consensus.cc:2749] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260709 12:26:01.282119 16501 tablet_replica.cc:333] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb: stopping tablet replica
I20260709 12:26:01.282982 16501 raft_consensus.cc:2243] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 26 LEADER]: Raft consensus shutting down.
I20260709 12:26:01.289544 16501 raft_consensus.cc:2272] T 0540d79e2ec2406588b295145aa6d54d P e2909cff1da44ad387f96a8ae3aa5afb [term 26 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:01.302506 16501 tablet_replica.cc:333] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb: stopping tablet replica
I20260709 12:26:01.303452 16501 raft_consensus.cc:2243] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:01.304253 16501 raft_consensus.cc:2272] T 217734cc2eb24a7a8d92bfef30ee5c52 P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:01.307770 16501 tablet_replica.cc:333] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb: stopping tablet replica
I20260709 12:26:01.308596 16501 raft_consensus.cc:2243] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 32 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:01.309758 16501 raft_consensus.cc:2272] T 11896555f5d7458dbb02a3203b27f1cf P e2909cff1da44ad387f96a8ae3aa5afb [term 32 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:01.312934 16501 tablet_replica.cc:333] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb: stopping tablet replica
W20260709 12:26:01.320140 17728 consensus_peers.cc:597] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20260709 12:26:01.317899 16501 raft_consensus.cc:2243] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:01.321736 16501 raft_consensus.cc:2272] T d522f70beb1f417cb98940d90b594823 P e2909cff1da44ad387f96a8ae3aa5afb [term 30 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:01.325086 16501 tablet_replica.cc:333] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb: stopping tablet replica
I20260709 12:26:01.325809 16501 raft_consensus.cc:2243] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 36 LEADER]: Raft consensus shutting down.
I20260709 12:26:01.327441 16501 raft_consensus.cc:2272] T 3418610c3e1a4e28824db2b86acff690 P e2909cff1da44ad387f96a8ae3aa5afb [term 36 FOLLOWER]: Raft consensus is shut down!
W20260709 12:26:01.331715 18333 leader_election.cc:336] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 36 election: RPC error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:01.332170 18333 leader_election.cc:304] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 36 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:26:01.332851 29314 raft_consensus.cc:2749] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 36 FOLLOWER]: Leader election lost for term 36. Reason: could not achieve majority
W20260709 12:26:01.334017 18333 leader_election.cc:336] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 29 election: RPC error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:01.334416 18333 leader_election.cc:304] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:26:01.335122 29314 raft_consensus.cc:2749] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260709 12:26:01.343595 16501 tablet_replica.cc:333] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb: stopping tablet replica
I20260709 12:26:01.344586 16501 raft_consensus.cc:2243] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:01.345345 16501 raft_consensus.cc:2272] T a9aeb14eca4a43dcbeaccffdf7e28995 P e2909cff1da44ad387f96a8ae3aa5afb [term 24 FOLLOWER]: Raft consensus is shut down!
W20260709 12:26:01.366478 17802 consensus_peers.cc:597] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:26:01.606173 18334 consensus_peers.cc:597] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:26:01.631556 17802 consensus_peers.cc:597] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:26:01.642781 18334 consensus_peers.cc:597] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:26:01.664551 17802 consensus_peers.cc:597] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:26:01.680994 17802 consensus_peers.cc:597] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:26:01.773156 17730 connection.cc:466] server connection from 127.0.0.1:38786 torn down before Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38786 (request call id 5005) could send its response
W20260709 12:26:01.796762 17802 consensus_peers.cc:597] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20260709 12:26:01.814723 29326 raft_consensus.cc:493] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:26:01.815431 29326 raft_consensus.cc:515] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:26:01.817973 29326 leader_election.cc:290] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:26:01.819499 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:26:01.820235 17831 raft_consensus.cc:2468] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 29.
I20260709 12:26:01.821578 18333 leader_election.cc:304] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:26:01.822402 29326 raft_consensus.cc:2804] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Leader pre-election won for term 30
W20260709 12:26:01.822463 18333 leader_election.cc:336] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Network error: Client connection negotiation failed: client connection to 127.16.29.66:39651: connect: Connection refused (error 111)
I20260709 12:26:01.822964 29326 raft_consensus.cc:493] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:26:01.823547 29326 raft_consensus.cc:3060] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:26:01.839641 29326 raft_consensus.cc:515] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:26:01.840507 29314 raft_consensus.cc:493] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:26:01.841349 29314 raft_consensus.cc:515] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:26:01.845594 29326 leader_election.cc:290] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 30 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:26:01.851172 18334 leader_election.cc:336] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111)
I20260709 12:26:01.852356 29314 leader_election.cc:290] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:26:01.860267 17802 consensus_peers.cc:597] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20260709 12:26:01.875687 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19dad6ab92894d708ec8eb7e8ac28185" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b"
I20260709 12:26:01.876451 17831 raft_consensus.cc:3060] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Advancing to term 30
W20260709 12:26:01.881919 18333 leader_election.cc:336] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 30 election: RPC error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Network error: Client connection negotiation failed: client connection to 127.16.29.66:39651: connect: Connection refused (error 111)
W20260709 12:26:01.885916 17802 consensus_peers.cc:597] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:26:01.888427 18333 leader_election.cc:336] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Network error: Client connection negotiation failed: client connection to 127.16.29.66:39651: connect: Connection refused (error 111)
I20260709 12:26:01.889052 18333 leader_election.cc:304] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:26:01.889840 29314 raft_consensus.cc:2749] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:26:01.897891 17831 raft_consensus.cc:2468] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 30.
W20260709 12:26:01.906415 18334 consensus_peers.cc:597] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20260709 12:26:01.913198 18333 leader_election.cc:304] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:26:01.914292 29314 raft_consensus.cc:2804] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:26:01.914901 29314 raft_consensus.cc:697] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 30 LEADER]: Becoming Leader. State: Replica: 334a4e1f59944544b3b3ff36616b57da, State: Running, Role: LEADER
I20260709 12:26:01.916049 29314 consensus_queue.cc:237] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 29.20, Last appended by leader: 20, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:26:01.927554 18334 consensus_peers.cc:597] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20260709 12:26:01.975775 17584 catalog_manager.cc:5697] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da reported cstate change: term changed from 29 to 30, leader changed from e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66) to 334a4e1f59944544b3b3ff36616b57da (127.16.29.68). New cstate: current_term: 30 leader_uuid: "334a4e1f59944544b3b3ff36616b57da" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 12:26:02.142143 18334 consensus_peers.cc:597] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:26:02.153700 17801 consensus_peers.cc:597] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b -> Peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Couldn't send request to peer e2909cff1da44ad387f96a8ae3aa5afb. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.66:39651: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:26:02.174222 17801 consensus_peers.cc:597] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b -> Peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Couldn't send request to peer e2909cff1da44ad387f96a8ae3aa5afb. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.66:39651: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:26:02.194353 17801 consensus_peers.cc:597] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b -> Peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Couldn't send request to peer e2909cff1da44ad387f96a8ae3aa5afb. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.66:39651: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:26:02.234331 18334 consensus_peers.cc:597] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:26:02.248171 17802 consensus_peers.cc:597] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20260709 12:26:02.272145 29314 raft_consensus.cc:493] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 32 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:26:02.272686 29314 raft_consensus.cc:515] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:26:02.281800 18334 leader_election.cc:336] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 33 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111)
I20260709 12:26:02.283499 29314 leader_election.cc:290] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:26:02.283532 29342 raft_consensus.cc:493] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 38 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:26:02.284606 29342 raft_consensus.cc:515] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:26:02.286890 29342 leader_election.cc:290] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:26:02.294694 18333 leader_election.cc:336] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 33 pre-election: RPC error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Network error: Client connection negotiation failed: client connection to 127.16.29.66:39651: connect: Connection refused (error 111)
I20260709 12:26:02.295235 18333 leader_election.cc:304] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:26:02.296289 29314 raft_consensus.cc:2749] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
W20260709 12:26:02.297030 17801 leader_election.cc:336] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 39 pre-election: RPC error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Network error: Client connection negotiation failed: client connection to 127.16.29.66:39651: connect: Connection refused (error 111)
W20260709 12:26:02.297971 17802 leader_election.cc:336] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 39 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111)
I20260709 12:26:02.298429 17802 leader_election.cc:304] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:26:02.299650 29342 raft_consensus.cc:2749] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 38 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
W20260709 12:26:02.368274 18333 consensus_peers.cc:597] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da -> Peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Couldn't send request to peer e2909cff1da44ad387f96a8ae3aa5afb. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.66:39651: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:26:02.371333 17801 consensus_peers.cc:597] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b -> Peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Couldn't send request to peer e2909cff1da44ad387f96a8ae3aa5afb. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.66:39651: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:26:02.398109 29342 raft_consensus.cc:493] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 36 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:26:02.398738 29342 raft_consensus.cc:515] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:26:02.400957 29342 leader_election.cc:290] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:26:02.404513 17801 consensus_peers.cc:597] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b -> Peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Couldn't send request to peer e2909cff1da44ad387f96a8ae3aa5afb. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.66:39651: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:26:02.410477 17801 leader_election.cc:336] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 37 pre-election: RPC error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Network error: Client connection negotiation failed: client connection to 127.16.29.66:39651: connect: Connection refused (error 111)
W20260709 12:26:02.412340 17802 leader_election.cc:336] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 37 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111)
I20260709 12:26:02.412755 17802 leader_election.cc:304] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:26:02.413868 29342 raft_consensus.cc:2749] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
W20260709 12:26:02.433279 18334 consensus_peers.cc:597] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20260709 12:26:02.446682 29334 raft_consensus.cc:493] T a5732e71b9d54f8786b8c511d69943bf P 334a4e1f59944544b3b3ff36616b57da [term 33 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:26:02.447381 29334 raft_consensus.cc:515] T a5732e71b9d54f8786b8c511d69943bf P 334a4e1f59944544b3b3ff36616b57da [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:26:02.456619 18333 leader_election.cc:336] T a5732e71b9d54f8786b8c511d69943bf P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 34 pre-election: RPC error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Network error: Client connection negotiation failed: client connection to 127.16.29.66:39651: connect: Connection refused (error 111)
I20260709 12:26:02.458556 29334 leader_election.cc:290] T a5732e71b9d54f8786b8c511d69943bf P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:26:02.460732 17831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "334a4e1f59944544b3b3ff36616b57da" candidate_term: 34 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "e063541542a94969898868bf6eb5ce1b" is_pre_election: true
I20260709 12:26:02.463091 17831 raft_consensus.cc:2468] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 334a4e1f59944544b3b3ff36616b57da in term 33.
W20260709 12:26:02.462157 18334 leader_election.cc:336] T a5732e71b9d54f8786b8c511d69943bf P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 34 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111)
I20260709 12:26:02.464061 18334 leader_election.cc:304] T a5732e71b9d54f8786b8c511d69943bf P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:26:02.465529 29334 raft_consensus.cc:2749] T a5732e71b9d54f8786b8c511d69943bf P 334a4e1f59944544b3b3ff36616b57da [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
W20260709 12:26:02.469583 17801 consensus_peers.cc:597] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b -> Peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Couldn't send request to peer e2909cff1da44ad387f96a8ae3aa5afb. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.66:39651: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:26:02.485288 29342 raft_consensus.cc:493] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:26:02.485877 29342 raft_consensus.cc:515] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:26:02.488065 29342 leader_election.cc:290] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:26:02.496222 17802 leader_election.cc:336] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111)
I20260709 12:26:02.500511 29334 raft_consensus.cc:493] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:26:02.501121 29334 raft_consensus.cc:515] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:26:02.502827 17801 leader_election.cc:336] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Network error: Client connection negotiation failed: client connection to 127.16.29.66:39651: connect: Connection refused (error 111)
I20260709 12:26:02.503288 17801 leader_election.cc:304] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:26:02.505486 29342 raft_consensus.cc:2749] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
W20260709 12:26:02.513319 18334 leader_election.cc:336] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111)
W20260709 12:26:02.516252 18333 leader_election.cc:336] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Network error: Client connection negotiation failed: client connection to 127.16.29.66:39651: connect: Connection refused (error 111)
I20260709 12:26:02.516717 18333 leader_election.cc:304] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:26:02.517906 29314 raft_consensus.cc:2749] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:26:02.518787 29334 leader_election.cc:290] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:26:02.522253 17831 raft_consensus.cc:1275] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Refusing update from remote peer 334a4e1f59944544b3b3ff36616b57da: Log matching property violated. Preceding OpId in replica: term: 29 index: 20. Preceding OpId from leader: term: 30 index: 21. (index mismatch)
I20260709 12:26:02.525380 29342 raft_consensus.cc:493] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:26:02.525906 29342 raft_consensus.cc:515] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:26:02.528016 29342 leader_election.cc:290] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:26:02.529718 29334 consensus_queue.cc:1048] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.001s
W20260709 12:26:02.534797 17802 leader_election.cc:336] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111)
W20260709 12:26:02.539201 17801 leader_election.cc:336] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Network error: Client connection negotiation failed: client connection to 127.16.29.66:39651: connect: Connection refused (error 111)
I20260709 12:26:02.544101 17801 leader_election.cc:304] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:26:02.546515 29342 raft_consensus.cc:2749] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
W20260709 12:26:02.594245 17802 consensus_peers.cc:597] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20260709 12:26:02.609284 29315 raft_consensus.cc:493] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:26:02.609920 29315 raft_consensus.cc:515] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:26:02.621564 29315 leader_election.cc:290] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:26:02.622325 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:26:02.622946 18368 raft_consensus.cc:2468] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 26.
I20260709 12:26:02.624137 17803 leader_election.cc:304] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:26:02.624917 29315 raft_consensus.cc:2804] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:26:02.625278 29315 raft_consensus.cc:493] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:26:02.625615 29315 raft_consensus.cc:3060] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:26:02.640702 17801 consensus_peers.cc:597] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b -> Peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Couldn't send request to peer e2909cff1da44ad387f96a8ae3aa5afb. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.66:39651: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:26:02.641276 17801 consensus_peers.cc:597] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b -> Peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Couldn't send request to peer e2909cff1da44ad387f96a8ae3aa5afb. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.66:39651: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:26:02.657914 18333 consensus_peers.cc:597] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da -> Peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Couldn't send request to peer e2909cff1da44ad387f96a8ae3aa5afb. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.66:39651: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:26:02.661630 18333 consensus_peers.cc:597] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da -> Peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Couldn't send request to peer e2909cff1da44ad387f96a8ae3aa5afb. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.66:39651: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:26:02.666563 18334 consensus_peers.cc:597] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:26:02.671077 17801 consensus_peers.cc:597] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b -> Peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Couldn't send request to peer e2909cff1da44ad387f96a8ae3aa5afb. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.66:39651: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:26:02.671552 17801 leader_election.cc:336] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Network error: Client connection negotiation failed: client connection to 127.16.29.66:39651: connect: Connection refused (error 111)
I20260709 12:26:02.670396 29315 raft_consensus.cc:515] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:26:02.672227 17801 consensus_peers.cc:597] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b -> Peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Couldn't send request to peer e2909cff1da44ad387f96a8ae3aa5afb. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.66:39651: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:26:02.680858 18333 consensus_peers.cc:597] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da -> Peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Couldn't send request to peer e2909cff1da44ad387f96a8ae3aa5afb. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.66:39651: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:26:02.686046 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a4040191930435786e2eeea338bb3d3" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:26:02.686712 18368 raft_consensus.cc:3060] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:26:02.687479 17801 consensus_peers.cc:597] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b -> Peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Couldn't send request to peer e2909cff1da44ad387f96a8ae3aa5afb. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.66:39651: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:26:02.689767 17801 leader_election.cc:336] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 27 election: RPC error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Network error: Client connection negotiation failed: client connection to 127.16.29.66:39651: connect: Connection refused (error 111)
I20260709 12:26:02.692248 29315 leader_election.cc:290] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 27 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:26:02.695178 18368 raft_consensus.cc:2468] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 27.
I20260709 12:26:02.704963 17803 leader_election.cc:304] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:26:02.706223 29315 raft_consensus.cc:2804] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:26:02.706794 29315 raft_consensus.cc:697] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 27 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:26:02.707777 29315 consensus_queue.cc:237] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 26.18, Last appended by leader: 18, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:26:02.710525 17802 consensus_peers.cc:597] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:26:02.718389 17801 consensus_peers.cc:597] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b -> Peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Couldn't send request to peer e2909cff1da44ad387f96a8ae3aa5afb. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.66:39651: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:26:02.727120 29354 raft_consensus.cc:493] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:26:02.727728 29354 raft_consensus.cc:515] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:26:02.729887 29354 leader_election.cc:290] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:26:02.735895 18333 leader_election.cc:336] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Network error: Client connection negotiation failed: client connection to 127.16.29.66:39651: connect: Connection refused (error 111)
W20260709 12:26:02.737146 18334 leader_election.cc:336] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111)
I20260709 12:26:02.737560 18334 leader_election.cc:304] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:26:02.738299 29354 raft_consensus.cc:2749] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:26:02.753015 17581 catalog_manager.cc:5697] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b reported cstate change: term changed from 26 to 27, leader changed from e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66) to e063541542a94969898868bf6eb5ce1b (127.16.29.67). New cstate: current_term: 27 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20260709 12:26:02.761262 17801 consensus_peers.cc:597] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b -> Peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Couldn't send request to peer e2909cff1da44ad387f96a8ae3aa5afb. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.66:39651: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:26:02.795538 18333 consensus_peers.cc:597] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da -> Peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Couldn't send request to peer e2909cff1da44ad387f96a8ae3aa5afb. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.66:39651: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:26:02.796464 17801 consensus_peers.cc:597] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b -> Peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Couldn't send request to peer e2909cff1da44ad387f96a8ae3aa5afb. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.66:39651: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:26:02.797761 16501 thread.cc:527] Waited for 1000ms trying to join with diag-logger (tid 17785)
W20260709 12:26:02.806061 17802 consensus_peers.cc:597] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:26:02.812150 18333 consensus_peers.cc:597] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da -> Peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Couldn't send request to peer e2909cff1da44ad387f96a8ae3aa5afb. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.66:39651: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:26:02.837996 18333 consensus_peers.cc:597] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da -> Peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Couldn't send request to peer e2909cff1da44ad387f96a8ae3aa5afb. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.66:39651: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:26:02.895284 17802 consensus_peers.cc:597] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:26:02.905138 17801 consensus_peers.cc:597] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b -> Peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Couldn't send request to peer e2909cff1da44ad387f96a8ae3aa5afb. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.66:39651: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:26:02.924119 29354 raft_consensus.cc:493] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:26:02.924662 29354 raft_consensus.cc:515] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:26:02.926609 29354 leader_election.cc:290] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:26:02.931137 18333 leader_election.cc:336] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Network error: Client connection negotiation failed: client connection to 127.16.29.66:39651: connect: Connection refused (error 111)
W20260709 12:26:02.932838 18334 leader_election.cc:336] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111)
I20260709 12:26:02.933234 18334 leader_election.cc:304] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:26:02.934373 29354 raft_consensus.cc:2749] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
W20260709 12:26:02.938721 17802 consensus_peers.cc:597] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:26:02.946224 17801 consensus_peers.cc:597] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b -> Peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Couldn't send request to peer e2909cff1da44ad387f96a8ae3aa5afb. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.66:39651: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:26:02.972563 17802 consensus_peers.cc:597] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20260709 12:26:02.985842 29315 raft_consensus.cc:493] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:26:02.986498 29315 raft_consensus.cc:515] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:26:02.990682 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:26:02.991300 18368 raft_consensus.cc:2468] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 26.
I20260709 12:26:02.993438 17803 leader_election.cc:304] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 
I20260709 12:26:02.994220 29310 raft_consensus.cc:2804] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:26:02.994659 29310 raft_consensus.cc:493] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Starting leader election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:26:02.995045 29310 raft_consensus.cc:3060] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:26:02.996116 17801 leader_election.cc:336] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Network error: Client connection negotiation failed: client connection to 127.16.29.66:39651: connect: Connection refused (error 111)
I20260709 12:26:02.996788 29315 leader_election.cc:290] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:26:03.025669 29310 raft_consensus.cc:515] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:26:03.042121 29310 leader_election.cc:290] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 27 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:26:03.042934 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0540d79e2ec2406588b295145aa6d54d" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:26:03.043663 18368 raft_consensus.cc:3060] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:26:03.048579 17801 leader_election.cc:336] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 27 election: RPC error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Network error: Client connection negotiation failed: client connection to 127.16.29.66:39651: connect: Connection refused (error 111)
I20260709 12:26:03.054850 18368 raft_consensus.cc:2468] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 27.
I20260709 12:26:03.056286 17803 leader_election.cc:304] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:26:03.057476 29310 raft_consensus.cc:2804] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:26:03.058873 29310 raft_consensus.cc:697] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 27 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:26:03.061652 29310 consensus_queue.cc:237] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 26.18, Last appended by leader: 18, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:26:03.105931 18333 consensus_peers.cc:597] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da -> Peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Couldn't send request to peer e2909cff1da44ad387f96a8ae3aa5afb. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.66:39651: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:26:03.105535 17581 catalog_manager.cc:5697] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b reported cstate change: term changed from 26 to 27, leader changed from e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66) to e063541542a94969898868bf6eb5ce1b (127.16.29.67). New cstate: current_term: 27 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20260709 12:26:03.128831 18333 consensus_peers.cc:597] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da -> Peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Couldn't send request to peer e2909cff1da44ad387f96a8ae3aa5afb. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.66:39651: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:26:03.135444 29354 raft_consensus.cc:493] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 36 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:26:03.136191 29354 raft_consensus.cc:515] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:26:03.138540 29354 leader_election.cc:290] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:26:03.144049 18334 leader_election.cc:336] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 37 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111)
I20260709 12:26:03.147050 29314 raft_consensus.cc:493] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 32 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:26:03.147682 29314 raft_consensus.cc:515] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:26:03.150058 29314 leader_election.cc:290] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:26:03.151044 29334 raft_consensus.cc:493] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 36 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:26:03.151573 29334 raft_consensus.cc:515] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:26:03.154034 29334 leader_election.cc:290] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:26:03.164891 18334 leader_election.cc:336] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 33 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111)
W20260709 12:26:03.166244 18333 leader_election.cc:336] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 37 pre-election: RPC error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Network error: Client connection negotiation failed: client connection to 127.16.29.66:39651: connect: Connection refused (error 111)
I20260709 12:26:03.166770 18333 leader_election.cc:304] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:26:03.170338 29314 raft_consensus.cc:2749] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
W20260709 12:26:03.171866 18334 leader_election.cc:336] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 37 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111)
I20260709 12:26:03.192551 29314 raft_consensus.cc:493] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 33 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:26:03.193251 29314 raft_consensus.cc:515] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:26:03.195904 29314 leader_election.cc:290] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:26:03.192606 17802 consensus_peers.cc:597] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:26:03.204546 18333 leader_election.cc:336] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 33 pre-election: RPC error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Network error: Client connection negotiation failed: client connection to 127.16.29.66:39651: connect: Connection refused (error 111)
W20260709 12:26:03.205506 18334 leader_election.cc:336] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 34 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111)
I20260709 12:26:03.206023 18333 leader_election.cc:304] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:26:03.207507 29334 raft_consensus.cc:2749] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
W20260709 12:26:03.207744 18333 leader_election.cc:336] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 37 pre-election: RPC error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Network error: Client connection negotiation failed: client connection to 127.16.29.66:39651: connect: Connection refused (error 111)
I20260709 12:26:03.208597 18333 leader_election.cc:304] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:26:03.209317 29334 raft_consensus.cc:2749] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:26:03.217571 29342 raft_consensus.cc:493] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:03.218138 29342 raft_consensus.cc:515] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:26:03.227651 29342 leader_election.cc:290] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:26:03.228012 18368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 37 candidate_status { last_received { term: 35 index: 21 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:26:03.229828 18368 raft_consensus.cc:2468] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 36.
W20260709 12:26:03.232617 17801 consensus_peers.cc:597] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b -> Peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Couldn't send request to peer e2909cff1da44ad387f96a8ae3aa5afb. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.66:39651: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:26:03.235993 17801 leader_election.cc:336] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 37 pre-election: RPC error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Network error: Client connection negotiation failed: client connection to 127.16.29.66:39651: connect: Connection refused (error 111)
W20260709 12:26:03.239153 18333 leader_election.cc:336] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 34 pre-election: RPC error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Network error: Client connection negotiation failed: client connection to 127.16.29.66:39651: connect: Connection refused (error 111)
I20260709 12:26:03.239632 18333 leader_election.cc:304] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:26:03.240334 29334 raft_consensus.cc:2749] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:26:03.242447 17803 leader_election.cc:304] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:26:03.245057 29342 raft_consensus.cc:2804] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 36 FOLLOWER]: Leader pre-election won for term 37
I20260709 12:26:03.246407 29342 raft_consensus.cc:493] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 36 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:03.246801 29342 raft_consensus.cc:3060] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:26:03.272785 29314 raft_consensus.cc:493] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:26:03.273407 29314 raft_consensus.cc:515] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:26:03.280954 18334 leader_election.cc:336] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111)
I20260709 12:26:03.275830 29314 leader_election.cc:290] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:26:03.288586 18368 raft_consensus.cc:1275] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 26 index: 18. Preceding OpId from leader: term: 27 index: 19. (index mismatch)
I20260709 12:26:03.291816 29310 consensus_queue.cc:1048] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
W20260709 12:26:03.296676 18333 leader_election.cc:336] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Network error: Client connection negotiation failed: client connection to 127.16.29.66:39651: connect: Connection refused (error 111)
I20260709 12:26:03.296994 29310 raft_consensus.cc:493] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:03.297267 18333 leader_election.cc:304] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:26:03.271824 29342 raft_consensus.cc:515] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 37 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:26:03.298022 29310 raft_consensus.cc:515] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:26:03.301426 29310 leader_election.cc:290] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:26:03.307642 17801 leader_election.cc:336] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 37 election: RPC error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Network error: Client connection negotiation failed: client connection to 127.16.29.66:39651: connect: Connection refused (error 111)
W20260709 12:26:03.308408 17801 leader_election.cc:336] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Network error: Client connection negotiation failed: client connection to 127.16.29.66:39651: connect: Connection refused (error 111)
I20260709 12:26:03.309352 29342 leader_election.cc:290] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 37 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:26:03.310451 29334 raft_consensus.cc:2749] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:26:03.316329 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5f0dd0890f44e35bbb838da838faa9c" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 37 candidate_status { last_received { term: 35 index: 21 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:26:03.317011 18374 raft_consensus.cc:3060] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:26:03.320277 18371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 30 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:26:03.321059 18371 raft_consensus.cc:2468] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 29.
I20260709 12:26:03.322876 17803 leader_election.cc:304] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:26:03.325165 29342 raft_consensus.cc:2804] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:26:03.325600 29342 raft_consensus.cc:493] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:03.325984 29342 raft_consensus.cc:3060] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Advancing to term 30
W20260709 12:26:03.333434 17802 consensus_peers.cc:597] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:26:03.343101 17802 consensus_peers.cc:597] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20260709 12:26:03.356184 18374 raft_consensus.cc:2468] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 37 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 37.
W20260709 12:26:03.372656 18334 consensus_peers.cc:597] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20260709 12:26:03.387379 17803 leader_election.cc:304] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 37 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:26:03.394134 29342 raft_consensus.cc:515] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:26:03.397397 29342 leader_election.cc:290] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 30 election: Requested vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:26:03.400226 17801 leader_election.cc:336] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 30 election: RPC error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Network error: Client connection negotiation failed: client connection to 127.16.29.66:39651: connect: Connection refused (error 111)
I20260709 12:26:03.401098 29342 raft_consensus.cc:2804] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 37 FOLLOWER]: Leader election won for term 37
I20260709 12:26:03.401693 29342 raft_consensus.cc:697] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 37 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:26:03.402688 29342 consensus_queue.cc:237] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 35.21, Last appended by leader: 21, Current term: 37, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:26:03.438601 17803 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:42577, user_credentials={real_user=slave}} blocked reactor thread for 51353.1us
I20260709 12:26:03.440567 18374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cab6f01c6a2485cb5024322a1bd5462" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 30 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da"
I20260709 12:26:03.441195 18374 raft_consensus.cc:3060] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:26:03.454634 18374 raft_consensus.cc:2468] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 30.
I20260709 12:26:03.456809 17803 leader_election.cc:304] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:26:03.466365 29334 raft_consensus.cc:493] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:26:03.470252 29334 raft_consensus.cc:515] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:26:03.472425 29334 leader_election.cc:290] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:26:03.464455 29366 raft_consensus.cc:2804] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:26:03.478374 29366 raft_consensus.cc:697] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 30 LEADER]: Becoming Leader. State: Replica: e063541542a94969898868bf6eb5ce1b, State: Running, Role: LEADER
I20260709 12:26:03.479589 29366 consensus_queue.cc:237] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 28.20, Last appended by leader: 20, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:26:03.484900 18333 leader_election.cc:336] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Network error: Client connection negotiation failed: client connection to 127.16.29.66:39651: connect: Connection refused (error 111)
I20260709 12:26:03.516980 17581 catalog_manager.cc:5697] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b reported cstate change: term changed from 35 to 37, leader changed from e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66) to e063541542a94969898868bf6eb5ce1b (127.16.29.67). New cstate: current_term: 37 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20260709 12:26:03.523694 18334 leader_election.cc:336] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111)
I20260709 12:26:03.526018 18334 leader_election.cc:304] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:26:03.526818 29334 raft_consensus.cc:2749] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:26:03.546728 17581 catalog_manager.cc:5697] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b reported cstate change: term changed from 28 to 30, leader changed from e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66) to e063541542a94969898868bf6eb5ce1b (127.16.29.67). New cstate: current_term: 30 leader_uuid: "e063541542a94969898868bf6eb5ce1b" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20260709 12:26:03.549739 17802 consensus_peers.cc:597] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20260709 12:26:03.585796 18371 raft_consensus.cc:1275] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 26 index: 18. Preceding OpId from leader: term: 27 index: 19. (index mismatch)
I20260709 12:26:03.587405 29342 consensus_queue.cc:1048] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20260709 12:26:03.605866 29334 raft_consensus.cc:493] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:26:03.606482 29334 raft_consensus.cc:515] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:26:03.609593 29334 leader_election.cc:290] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:26:03.613245 18334 leader_election.cc:336] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111)
W20260709 12:26:03.621821 18333 leader_election.cc:336] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Network error: Client connection negotiation failed: client connection to 127.16.29.66:39651: connect: Connection refused (error 111)
I20260709 12:26:03.622225 18333 leader_election.cc:304] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:26:03.622838 29314 raft_consensus.cc:2749] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
W20260709 12:26:03.627444 18334 consensus_peers.cc:597] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:26:03.637199 17801 consensus_peers.cc:597] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b -> Peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Couldn't send request to peer e2909cff1da44ad387f96a8ae3aa5afb. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.66:39651: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:26:03.798254 16501 thread.cc:527] Waited for 2000ms trying to join with diag-logger (tid 17785)
W20260709 12:26:03.825946 17802 consensus_peers.cc:597] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20260709 12:26:03.897341 18371 raft_consensus.cc:1275] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 37 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 35 index: 21. Preceding OpId from leader: term: 37 index: 22. (index mismatch)
I20260709 12:26:03.898981 29342 consensus_queue.cc:1048] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.000s
W20260709 12:26:03.906407 17802 consensus_peers.cc:597] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:26:03.920609 17801 consensus_peers.cc:597] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b -> Peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Couldn't send request to peer e2909cff1da44ad387f96a8ae3aa5afb. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.66:39651: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:26:03.956287 17801 consensus_peers.cc:597] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b -> Peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Couldn't send request to peer e2909cff1da44ad387f96a8ae3aa5afb. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.66:39651: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:26:03.971334 18371 raft_consensus.cc:1275] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Refusing update from remote peer e063541542a94969898868bf6eb5ce1b: Log matching property violated. Preceding OpId in replica: term: 28 index: 20. Preceding OpId from leader: term: 30 index: 21. (index mismatch)
I20260709 12:26:03.974076 29366 consensus_queue.cc:1048] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.001s
I20260709 12:26:04.064101 29314 raft_consensus.cc:493] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 32 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:26:04.064687 29314 raft_consensus.cc:515] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:26:04.066977 29314 leader_election.cc:290] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:26:04.073762 18333 leader_election.cc:336] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 33 pre-election: RPC error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Network error: Client connection negotiation failed: client connection to 127.16.29.66:39651: connect: Connection refused (error 111)
W20260709 12:26:04.077116 18334 leader_election.cc:336] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 33 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111)
I20260709 12:26:04.079433 18334 leader_election.cc:304] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:26:04.080246 29314 raft_consensus.cc:2749] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
W20260709 12:26:04.093986 17802 consensus_peers.cc:597] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:26:04.104408 17802 consensus_peers.cc:597] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20260709 12:26:04.198652 29365 raft_consensus.cc:493] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 36 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:26:04.199263 29365 raft_consensus.cc:515] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:26:04.202997 29365 leader_election.cc:290] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:26:04.207387 17802 leader_election.cc:336] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 37 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111)
W20260709 12:26:04.212181 17801 leader_election.cc:336] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 37 pre-election: RPC error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Network error: Client connection negotiation failed: client connection to 127.16.29.66:39651: connect: Connection refused (error 111)
I20260709 12:26:04.215279 17801 leader_election.cc:304] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:26:04.216365 29365 raft_consensus.cc:2749] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
W20260709 12:26:04.256085 18334 consensus_peers.cc:597] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:26:04.281534 18334 consensus_peers.cc:597] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:26:04.331246 18334 consensus_peers.cc:597] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20260709 12:26:04.415196 29342 raft_consensus.cc:493] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 38 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:26:04.415856 29342 raft_consensus.cc:515] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
W20260709 12:26:04.423002 17802 leader_election.cc:336] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 39 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111)
I20260709 12:26:04.431782 29342 leader_election.cc:290] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:26:04.443485 17801 consensus_peers.cc:597] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b -> Peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Couldn't send request to peer e2909cff1da44ad387f96a8ae3aa5afb. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.66:39651: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:26:04.449620 17801 leader_election.cc:336] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 39 pre-election: RPC error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Network error: Client connection negotiation failed: client connection to 127.16.29.66:39651: connect: Connection refused (error 111)
I20260709 12:26:04.459849 17801 leader_election.cc:304] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:26:04.462352 29365 raft_consensus.cc:2749] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 38 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
W20260709 12:26:04.516673 17802 consensus_peers.cc:597] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:26:04.561394 17801 consensus_peers.cc:597] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b -> Peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Couldn't send request to peer e2909cff1da44ad387f96a8ae3aa5afb. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.66:39651: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:26:04.571378 17801 consensus_peers.cc:597] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b -> Peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Couldn't send request to peer e2909cff1da44ad387f96a8ae3aa5afb. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.66:39651: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:26:04.585377 18334 consensus_peers.cc:597] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20260709 12:26:04.602440 29375 raft_consensus.cc:493] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:26:04.603189 29375 raft_consensus.cc:515] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:26:04.612051 18334 leader_election.cc:336] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111)
I20260709 12:26:04.624807 29375 leader_election.cc:290] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:26:04.655756 18334 consensus_peers.cc:597] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:26:04.660233 17802 consensus_peers.cc:597] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:26:04.668735 17802 consensus_peers.cc:597] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:26:04.671989 18333 leader_election.cc:336] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Network error: Client connection negotiation failed: client connection to 127.16.29.66:39651: connect: Connection refused (error 111)
I20260709 12:26:04.682674 18333 leader_election.cc:304] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:26:04.684118 29314 raft_consensus.cc:2749] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
W20260709 12:26:04.722800 17802 consensus_peers.cc:597] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:26:04.755280 17801 consensus_peers.cc:597] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b -> Peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Couldn't send request to peer e2909cff1da44ad387f96a8ae3aa5afb. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.66:39651: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:26:04.764071 17801 consensus_peers.cc:597] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b -> Peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Couldn't send request to peer e2909cff1da44ad387f96a8ae3aa5afb. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.66:39651: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 12:26:04.777170 29365 raft_consensus.cc:493] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:26:04.777837 29365 raft_consensus.cc:515] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:26:04.780661 29365 leader_election.cc:290] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:26:04.798744 16501 thread.cc:527] Waited for 3000ms trying to join with diag-logger (tid 17785)
W20260709 12:26:04.813777 18334 consensus_peers.cc:597] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:26:04.815704 17802 consensus_peers.cc:597] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:26:04.823448 17801 consensus_peers.cc:597] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b -> Peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Couldn't send request to peer e2909cff1da44ad387f96a8ae3aa5afb. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.66:39651: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:26:04.826287 17801 leader_election.cc:336] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Network error: Client connection negotiation failed: client connection to 127.16.29.66:39651: connect: Connection refused (error 111)
W20260709 12:26:04.832037 17802 leader_election.cc:336] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111)
I20260709 12:26:04.838131 17802 leader_election.cc:304] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:26:04.842646 29365 raft_consensus.cc:2749] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
W20260709 12:26:04.900337 18334 consensus_peers.cc:597] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:26:04.921860 18333 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.29.66:39651: connect: Connection refused (error 111) [suppressed 669 similar messages]
W20260709 12:26:04.953249 18333 consensus_peers.cc:597] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da -> Peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Couldn't send request to peer e2909cff1da44ad387f96a8ae3aa5afb. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.66:39651: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:26:04.991590 17801 consensus_peers.cc:597] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b -> Peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Couldn't send request to peer e2909cff1da44ad387f96a8ae3aa5afb. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.66:39651: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:26:04.993392 18333 consensus_peers.cc:597] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da -> Peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Couldn't send request to peer e2909cff1da44ad387f96a8ae3aa5afb. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.66:39651: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:26:04.999966 17802 consensus_peers.cc:597] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:26:05.024533 17801 consensus_peers.cc:597] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b -> Peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Couldn't send request to peer e2909cff1da44ad387f96a8ae3aa5afb. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.66:39651: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 12:26:05.046234 29365 raft_consensus.cc:493] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 33 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:26:05.046877 29365 raft_consensus.cc:515] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:26:05.061479 17801 consensus_peers.cc:597] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b -> Peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Couldn't send request to peer e2909cff1da44ad387f96a8ae3aa5afb. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.66:39651: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 12:26:05.066260 29376 raft_consensus.cc:493] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:26:05.066144 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 34 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:26:05.067023 18365 raft_consensus.cc:2468] T a5732e71b9d54f8786b8c511d69943bf P 334a4e1f59944544b3b3ff36616b57da [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 33.
I20260709 12:26:05.066874 29376 raft_consensus.cc:515] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:26:05.069410 29365 leader_election.cc:290] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
I20260709 12:26:05.069691 29376 leader_election.cc:290] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:26:05.071285 17802 leader_election.cc:336] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 34 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111)
W20260709 12:26:05.079351 17802 leader_election.cc:336] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111)
W20260709 12:26:05.082535 18334 consensus_peers.cc:597] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:26:05.084340 17801 leader_election.cc:336] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 34 pre-election: RPC error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Network error: Client connection negotiation failed: client connection to 127.16.29.66:39651: connect: Connection refused (error 111)
I20260709 12:26:05.084787 17801 leader_election.cc:304] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:26:05.086055 29376 raft_consensus.cc:2749] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
W20260709 12:26:05.090415 18333 consensus_peers.cc:597] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da -> Peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Couldn't send request to peer e2909cff1da44ad387f96a8ae3aa5afb. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.66:39651: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:26:05.090706 17801 leader_election.cc:336] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Network error: Client connection negotiation failed: client connection to 127.16.29.66:39651: connect: Connection refused (error 111)
I20260709 12:26:05.105448 17801 leader_election.cc:304] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:26:05.120579 17802 consensus_peers.cc:597] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20260709 12:26:05.120504 29376 raft_consensus.cc:2749] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
W20260709 12:26:05.148028 17801 consensus_peers.cc:597] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b -> Peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Couldn't send request to peer e2909cff1da44ad387f96a8ae3aa5afb. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.66:39651: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:26:05.152163 17801 consensus_peers.cc:597] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b -> Peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Couldn't send request to peer e2909cff1da44ad387f96a8ae3aa5afb. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.66:39651: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:26:05.169032 18333 consensus_peers.cc:597] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da -> Peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Couldn't send request to peer e2909cff1da44ad387f96a8ae3aa5afb. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.66:39651: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:26:05.185230 17801 consensus_peers.cc:597] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b -> Peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Couldn't send request to peer e2909cff1da44ad387f96a8ae3aa5afb. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.66:39651: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:26:05.226619 17802 consensus_peers.cc:597] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:26:05.230999 18333 consensus_peers.cc:597] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da -> Peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Couldn't send request to peer e2909cff1da44ad387f96a8ae3aa5afb. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.66:39651: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 12:26:05.252888 29375 raft_consensus.cc:493] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 32 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:26:05.253554 29375 raft_consensus.cc:515] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:26:05.266609 18334 leader_election.cc:336] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 33 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111)
W20260709 12:26:05.267567 18333 leader_election.cc:336] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 33 pre-election: RPC error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Network error: Client connection negotiation failed: client connection to 127.16.29.66:39651: connect: Connection refused (error 111)
I20260709 12:26:05.268045 18333 leader_election.cc:304] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:26:05.269145 29314 raft_consensus.cc:2749] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:26:05.271250 29375 leader_election.cc:290] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:26:05.275209 29314 raft_consensus.cc:493] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:26:05.275799 29314 raft_consensus.cc:515] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:26:05.277977 29314 leader_election.cc:290] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:26:05.291028 18334 leader_election.cc:336] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111)
W20260709 12:26:05.301190 18333 leader_election.cc:336] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Network error: Client connection negotiation failed: client connection to 127.16.29.66:39651: connect: Connection refused (error 111)
I20260709 12:26:05.304759 18333 leader_election.cc:304] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:26:05.305783 29314 raft_consensus.cc:2749] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
W20260709 12:26:05.319530 17801 consensus_peers.cc:597] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b -> Peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Couldn't send request to peer e2909cff1da44ad387f96a8ae3aa5afb. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.66:39651: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:26:05.319578 18333 consensus_peers.cc:597] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da -> Peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Couldn't send request to peer e2909cff1da44ad387f96a8ae3aa5afb. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.66:39651: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 12:26:05.327008 29314 raft_consensus.cc:493] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 36 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:26:05.327698 29314 raft_consensus.cc:515] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:26:05.329679 29314 leader_election.cc:290] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:26:05.337121 18333 leader_election.cc:336] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 37 pre-election: RPC error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Network error: Client connection negotiation failed: client connection to 127.16.29.66:39651: connect: Connection refused (error 111)
W20260709 12:26:05.350188 18334 leader_election.cc:336] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 37 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111)
I20260709 12:26:05.350688 18334 leader_election.cc:304] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:26:05.355119 29375 raft_consensus.cc:2749] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
W20260709 12:26:05.378942 18333 consensus_peers.cc:597] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da -> Peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Couldn't send request to peer e2909cff1da44ad387f96a8ae3aa5afb. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.66:39651: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 12:26:05.380576 29375 raft_consensus.cc:493] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:26:05.381220 29375 raft_consensus.cc:515] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:26:05.384130 29375 leader_election.cc:290] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:26:05.389235 18333 leader_election.cc:336] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Network error: Client connection negotiation failed: client connection to 127.16.29.66:39651: connect: Connection refused (error 111)
W20260709 12:26:05.389775 18334 leader_election.cc:336] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111)
I20260709 12:26:05.390241 18334 leader_election.cc:304] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:26:05.391367 29375 raft_consensus.cc:2749] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
W20260709 12:26:05.392261 17802 consensus_peers.cc:597] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20260709 12:26:05.407416 29375 raft_consensus.cc:493] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:26:05.408124 29375 raft_consensus.cc:515] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:26:05.411062 29375 leader_election.cc:290] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:26:05.417423 17801 consensus_peers.cc:597] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b -> Peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Couldn't send request to peer e2909cff1da44ad387f96a8ae3aa5afb. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.66:39651: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:26:05.419409 18333 leader_election.cc:336] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Network error: Client connection negotiation failed: client connection to 127.16.29.66:39651: connect: Connection refused (error 111)
W20260709 12:26:05.419766 17802 consensus_peers.cc:597] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:26:05.430066 18334 leader_election.cc:336] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111)
I20260709 12:26:05.430615 18334 leader_election.cc:304] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:26:05.431335 29375 raft_consensus.cc:2749] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
W20260709 12:26:05.469601 17801 consensus_peers.cc:597] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b -> Peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Couldn't send request to peer e2909cff1da44ad387f96a8ae3aa5afb. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.66:39651: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:26:05.481503 17801 consensus_peers.cc:597] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b -> Peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Couldn't send request to peer e2909cff1da44ad387f96a8ae3aa5afb. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.66:39651: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:26:05.493364 18333 consensus_peers.cc:597] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da -> Peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Couldn't send request to peer e2909cff1da44ad387f96a8ae3aa5afb. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.66:39651: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:26:05.546489 17802 consensus_peers.cc:597] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:26:05.580127 17801 consensus_peers.cc:597] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b -> Peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Couldn't send request to peer e2909cff1da44ad387f96a8ae3aa5afb. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.66:39651: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:26:05.784399 18333 consensus_peers.cc:597] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da -> Peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Couldn't send request to peer e2909cff1da44ad387f96a8ae3aa5afb. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.66:39651: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 12:26:05.806600 29314 raft_consensus.cc:493] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 36 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:26:05.807569 29314 raft_consensus.cc:515] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:26:05.819957 18334 leader_election.cc:336] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 37 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111)
I20260709 12:26:05.823408 29314 leader_election.cc:290] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:26:05.839525 18333 leader_election.cc:336] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 37 pre-election: RPC error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Network error: Client connection negotiation failed: client connection to 127.16.29.66:39651: connect: Connection refused (error 111)
I20260709 12:26:05.840102 18333 leader_election.cc:304] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:26:05.846371 29314 raft_consensus.cc:2749] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:26:05.925664 29375 raft_consensus.cc:493] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:26:05.926340 29375 raft_consensus.cc:515] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:26:05.941591 29375 leader_election.cc:290] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:26:05.948822 17802 consensus_peers.cc:597] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:26:05.950346 18334 leader_election.cc:336] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111)
W20260709 12:26:05.953744 18333 leader_election.cc:336] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Network error: Client connection negotiation failed: client connection to 127.16.29.66:39651: connect: Connection refused (error 111)
I20260709 12:26:05.954319 18333 leader_election.cc:304] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:26:05.955737 29375 raft_consensus.cc:2749] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
W20260709 12:26:06.024606 18334 consensus_peers.cc:597] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20260709 12:26:06.035392 29375 raft_consensus.cc:493] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:26:06.036001 29375 raft_consensus.cc:515] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:26:06.038126 29375 leader_election.cc:290] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:26:06.072283 18334 leader_election.cc:336] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111)
W20260709 12:26:06.077726 18333 leader_election.cc:336] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Network error: Client connection negotiation failed: client connection to 127.16.29.66:39651: connect: Connection refused (error 111)
I20260709 12:26:06.078347 18333 leader_election.cc:304] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:26:06.081179 29314 raft_consensus.cc:2749] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
W20260709 12:26:06.087821 17802 consensus_peers.cc:597] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:26:06.091099 18334 consensus_peers.cc:597] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:26:06.091682 17802 consensus_peers.cc:597] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:26:06.184839 17802 consensus_peers.cc:597] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:26:06.228392 17801 consensus_peers.cc:597] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b -> Peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Couldn't send request to peer e2909cff1da44ad387f96a8ae3aa5afb. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.66:39651: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:26:06.292420 17801 consensus_peers.cc:597] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b -> Peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Couldn't send request to peer e2909cff1da44ad387f96a8ae3aa5afb. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.66:39651: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:26:06.430042 17802 consensus_peers.cc:597] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20260709 12:26:06.463239 29375 raft_consensus.cc:493] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 33 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:26:06.463843 29375 raft_consensus.cc:515] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:26:06.466224 29375 leader_election.cc:290] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:26:06.479399 18334 leader_election.cc:336] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 34 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111)
W20260709 12:26:06.480322 18333 leader_election.cc:336] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 34 pre-election: RPC error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Network error: Client connection negotiation failed: client connection to 127.16.29.66:39651: connect: Connection refused (error 111)
I20260709 12:26:06.480746 18333 leader_election.cc:304] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:26:06.482573 29375 raft_consensus.cc:2749] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
W20260709 12:26:06.577345 17802 consensus_peers.cc:597] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20260709 12:26:06.613909 29365 raft_consensus.cc:493] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 38 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:26:06.614522 29365 raft_consensus.cc:515] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:26:06.616928 29365 leader_election.cc:290] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:26:06.634934 17802 leader_election.cc:336] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 39 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111)
W20260709 12:26:06.637005 17801 leader_election.cc:336] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 39 pre-election: RPC error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Network error: Client connection negotiation failed: client connection to 127.16.29.66:39651: connect: Connection refused (error 111)
I20260709 12:26:06.663727 17801 leader_election.cc:304] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:26:06.667030 29365 raft_consensus.cc:2749] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 38 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
W20260709 12:26:06.693027 18334 consensus_peers.cc:597] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:26:06.703904 17802 consensus_peers.cc:597] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:26:06.792687 17801 consensus_peers.cc:597] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b -> Peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Couldn't send request to peer e2909cff1da44ad387f96a8ae3aa5afb. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.66:39651: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:26:06.812413 17802 consensus_peers.cc:597] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20260709 12:26:06.836185 16501 tablet_server.cc:196] TabletServer@127.16.29.66:0 shutdown complete.
W20260709 12:26:06.837049 18334 consensus_peers.cc:597] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:26:06.885002 17801 consensus_peers.cc:597] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b -> Peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Couldn't send request to peer e2909cff1da44ad387f96a8ae3aa5afb. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.66:39651: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:26:06.906256 18334 consensus_peers.cc:597] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:26:06.919371 18334 consensus_peers.cc:597] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:26:06.941221 17802 consensus_peers.cc:597] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20260709 12:26:06.974252 29375 raft_consensus.cc:493] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 32 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:26:06.974903 29375 raft_consensus.cc:515] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:26:06.996479 17801 consensus_peers.cc:597] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b -> Peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Couldn't send request to peer e2909cff1da44ad387f96a8ae3aa5afb. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.66:39651: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:26:06.997117 18334 leader_election.cc:336] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 33 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111)
W20260709 12:26:06.999955 18333 leader_election.cc:336] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 33 pre-election: RPC error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Network error: Client connection negotiation failed: client connection to 127.16.29.66:39651: connect: Connection refused (error 111)
I20260709 12:26:07.000365 18333 leader_election.cc:304] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:26:07.001089 29314 raft_consensus.cc:2749] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:26:07.005497 29366 raft_consensus.cc:493] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 33 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:26:07.006182 29366 raft_consensus.cc:515] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:26:07.011009 17801 consensus_peers.cc:597] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b -> Peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Couldn't send request to peer e2909cff1da44ad387f96a8ae3aa5afb. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.66:39651: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 12:26:07.016026 29375 leader_election.cc:290] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:26:07.045811 18334 consensus_peers.cc:597] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20260709 12:26:07.047669 18365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5732e71b9d54f8786b8c511d69943bf" candidate_uuid: "e063541542a94969898868bf6eb5ce1b" candidate_term: 34 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "334a4e1f59944544b3b3ff36616b57da" is_pre_election: true
I20260709 12:26:07.048334 18365 raft_consensus.cc:2468] T a5732e71b9d54f8786b8c511d69943bf P 334a4e1f59944544b3b3ff36616b57da [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e063541542a94969898868bf6eb5ce1b in term 33.
I20260709 12:26:07.049700 29366 leader_election.cc:290] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), 334a4e1f59944544b3b3ff36616b57da (127.16.29.68:42577)
W20260709 12:26:07.055269 17801 leader_election.cc:336] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 34 pre-election: RPC error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Network error: Client connection negotiation failed: client connection to 127.16.29.66:39651: connect: Connection refused (error 111)
W20260709 12:26:07.074072 17802 leader_election.cc:336] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 34 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111)
I20260709 12:26:07.074698 17802 leader_election.cc:304] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da, e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:26:07.079834 29380 raft_consensus.cc:2749] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
W20260709 12:26:07.116190 17801 consensus_peers.cc:597] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b -> Peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Couldn't send request to peer e2909cff1da44ad387f96a8ae3aa5afb. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.66:39651: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 12:26:07.132895 16501 tablet_server.cc:179] TabletServer@127.16.29.67:0 shutting down...
W20260709 12:26:07.169196 18333 consensus_peers.cc:597] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da -> Peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837): Couldn't send request to peer e063541542a94969898868bf6eb5ce1b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:26:07.217744 29366 raft_consensus.cc:493] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 36 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:26:07.218298 29366 raft_consensus.cc:515] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } }
I20260709 12:26:07.224092 29366 leader_election.cc:290] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:26:07.232963 17802 leader_election.cc:336] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 37 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111)
W20260709 12:26:07.240406 18333 consensus_peers.cc:597] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da -> Peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Couldn't send request to peer e2909cff1da44ad387f96a8ae3aa5afb. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.66:39651: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:26:07.245003 17801 leader_election.cc:336] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 37 pre-election: RPC error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Network error: Client connection negotiation failed: client connection to 127.16.29.66:39651: connect: Connection refused (error 111)
I20260709 12:26:07.245875 17801 leader_election.cc:304] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e063541542a94969898868bf6eb5ce1b; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:26:07.248427 29366 raft_consensus.cc:2749] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
W20260709 12:26:07.249353 17801 consensus_peers.cc:597] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b -> Peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Couldn't send request to peer e2909cff1da44ad387f96a8ae3aa5afb. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.66:39651: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:26:07.261972 18333 consensus_peers.cc:597] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da -> Peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837): Couldn't send request to peer e063541542a94969898868bf6eb5ce1b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:26:07.262727 18333 consensus_peers.cc:597] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da -> Peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837): Couldn't send request to peer e063541542a94969898868bf6eb5ce1b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:26:07.318112 18333 consensus_peers.cc:597] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da -> Peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837): Couldn't send request to peer e063541542a94969898868bf6eb5ce1b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:26:07.328449 18333 consensus_peers.cc:597] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da -> Peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837): Couldn't send request to peer e063541542a94969898868bf6eb5ce1b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:26:07.331346 17801 consensus_peers.cc:597] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b -> Peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Couldn't send request to peer e2909cff1da44ad387f96a8ae3aa5afb. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.66:39651: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:26:07.339039 17802 consensus_peers.cc:597] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:26:07.356148 18334 consensus_peers.cc:597] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:26:07.398945 17802 consensus_peers.cc:597] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:26:07.414273 18333 consensus_peers.cc:597] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da -> Peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837): Couldn't send request to peer e063541542a94969898868bf6eb5ce1b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:26:07.422485 16501 ts_tablet_manager.cc:1507] Shutting down tablet manager...
W20260709 12:26:07.424583 17801 consensus_peers.cc:597] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b -> Peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Couldn't send request to peer e2909cff1da44ad387f96a8ae3aa5afb. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.66:39651: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 12:26:07.426019 16501 tablet_replica.cc:333] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b: stopping tablet replica
I20260709 12:26:07.427016 16501 raft_consensus.cc:2243] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 33 LEADER]: Raft consensus shutting down.
I20260709 12:26:07.428303 16501 raft_consensus.cc:2272] T 22fa38faae58422e9cecc99973cab688 P e063541542a94969898868bf6eb5ce1b [term 33 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:07.438194 16501 tablet_replica.cc:333] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b: stopping tablet replica
I20260709 12:26:07.439122 16501 raft_consensus.cc:2243] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 36 LEADER]: Raft consensus shutting down.
W20260709 12:26:07.440059 18333 consensus_peers.cc:597] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da -> Peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Couldn't send request to peer e2909cff1da44ad387f96a8ae3aa5afb. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.66:39651: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:26:07.447417 18333 consensus_peers.cc:597] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da -> Peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Couldn't send request to peer e2909cff1da44ad387f96a8ae3aa5afb. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.66:39651: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:26:07.451064 17801 consensus_peers.cc:597] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b -> Peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Couldn't send request to peer e2909cff1da44ad387f96a8ae3aa5afb. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.66:39651: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 12:26:07.458606 16501 raft_consensus.cc:2272] T 23c0cde5227f470388b8ed1fc0786119 P e063541542a94969898868bf6eb5ce1b [term 36 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:07.461710 16501 tablet_replica.cc:333] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b: stopping tablet replica
I20260709 12:26:07.462644 16501 raft_consensus.cc:2243] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 27 LEADER]: Raft consensus shutting down.
I20260709 12:26:07.464111 16501 raft_consensus.cc:2272] T 8a208e793881438b975b83755dc05cfa P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:07.466703 16501 tablet_replica.cc:333] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b: stopping tablet replica
I20260709 12:26:07.467672 16501 raft_consensus.cc:2243] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Raft consensus shutting down.
W20260709 12:26:07.474709 18333 consensus_peers.cc:597] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da -> Peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837): Couldn't send request to peer e063541542a94969898868bf6eb5ce1b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:26:07.503579 17802 consensus_peers.cc:597] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:26:07.504751 18333 consensus_peers.cc:597] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da -> Peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837): Couldn't send request to peer e063541542a94969898868bf6eb5ce1b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:26:07.508910 16501 raft_consensus.cc:2272] T ab84345f9aa742a7b85389e9b6f2ce90 P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:07.513811 16501 tablet_replica.cc:333] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b: stopping tablet replica
I20260709 12:26:07.514742 16501 raft_consensus.cc:2243] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:07.515439 16501 raft_consensus.cc:2272] T 1a698a75c7954dd7b24e7889cb19ae35 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:07.518421 16501 tablet_replica.cc:333] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b: stopping tablet replica
I20260709 12:26:07.519367 16501 raft_consensus.cc:2243] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 37 LEADER]: Raft consensus shutting down.
I20260709 12:26:07.520758 16501 raft_consensus.cc:2272] T f5f0dd0890f44e35bbb838da838faa9c P e063541542a94969898868bf6eb5ce1b [term 37 FOLLOWER]: Raft consensus is shut down!
W20260709 12:26:07.536880 18333 consensus_peers.cc:597] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da -> Peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837): Couldn't send request to peer e063541542a94969898868bf6eb5ce1b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:26:07.540017 17802 consensus_peers.cc:597] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20260709 12:26:07.546734 29314 raft_consensus.cc:493] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:26:07.547295 29314 raft_consensus.cc:515] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:26:07.549171 29314 leader_election.cc:290] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:26:07.553246 18334 leader_election.cc:336] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111)
W20260709 12:26:07.558034 18333 consensus_peers.cc:597] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da -> Peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837): Couldn't send request to peer e063541542a94969898868bf6eb5ce1b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:26:07.563524 17802 consensus_peers.cc:597] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:26:07.564059 18333 leader_election.cc:336] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Network error: Client connection negotiation failed: client connection to 127.16.29.66:39651: connect: Connection refused (error 111)
I20260709 12:26:07.564918 18333 leader_election.cc:304] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:26:07.568002 29314 raft_consensus.cc:2749] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:26:07.571421 16501 tablet_replica.cc:333] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b: stopping tablet replica
I20260709 12:26:07.572395 16501 raft_consensus.cc:2243] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 34 LEADER]: Raft consensus shutting down.
I20260709 12:26:07.573594 16501 raft_consensus.cc:2272] T e55c231d5fd5471285b2d7879a38f38b P e063541542a94969898868bf6eb5ce1b [term 34 FOLLOWER]: Raft consensus is shut down!
W20260709 12:26:07.575223 18334 consensus_peers.cc:597] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20260709 12:26:07.576393 16501 tablet_replica.cc:333] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b: stopping tablet replica
I20260709 12:26:07.577239 16501 raft_consensus.cc:2243] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 30 LEADER]: Raft consensus shutting down.
I20260709 12:26:07.578437 16501 raft_consensus.cc:2272] T 8cab6f01c6a2485cb5024322a1bd5462 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:07.581938 16501 tablet_replica.cc:333] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b: stopping tablet replica
I20260709 12:26:07.582782 16501 raft_consensus.cc:2243] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:07.583765 16501 raft_consensus.cc:2272] T 7cb3768d215e409fb9f5baf18d124442 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:07.586869 16501 tablet_replica.cc:333] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b: stopping tablet replica
I20260709 12:26:07.587769 16501 raft_consensus.cc:2243] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 35 LEADER]: Raft consensus shutting down.
I20260709 12:26:07.589251 16501 raft_consensus.cc:2272] T 00a02ab39fcd4078ad49697693cf3be8 P e063541542a94969898868bf6eb5ce1b [term 35 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:07.592072 16501 tablet_replica.cc:333] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b: stopping tablet replica
I20260709 12:26:07.592973 16501 raft_consensus.cc:2243] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 38 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:07.593781 16501 raft_consensus.cc:2272] T 13a48228b0d44d3ea21f474203dd02e3 P e063541542a94969898868bf6eb5ce1b [term 38 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:07.597000 16501 tablet_replica.cc:333] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b: stopping tablet replica
I20260709 12:26:07.597846 16501 raft_consensus.cc:2243] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 24 LEADER]: Raft consensus shutting down.
W20260709 12:26:07.606894 18333 consensus_peers.cc:597] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da -> Peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837): Couldn't send request to peer e063541542a94969898868bf6eb5ce1b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:26:07.610260 16501 raft_consensus.cc:2272] T 6fcb5bdaa4be4d619bb945ffd44090ba P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:07.614151 16501 tablet_replica.cc:333] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b: stopping tablet replica
I20260709 12:26:07.615245 16501 raft_consensus.cc:2243] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 35 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:07.616043 16501 raft_consensus.cc:2272] T 9160d9f98b8c406db2771d9ea2e0995f P e063541542a94969898868bf6eb5ce1b [term 35 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:07.618851 16501 tablet_replica.cc:333] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b: stopping tablet replica
I20260709 12:26:07.620048 16501 raft_consensus.cc:2243] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 30 LEADER]: Raft consensus shutting down.
I20260709 12:26:07.626150 29375 raft_consensus.cc:493] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 32 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:26:07.626737 29375 raft_consensus.cc:515] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:26:07.628937 29375 leader_election.cc:290] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:26:07.634397 18334 leader_election.cc:336] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 33 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111)
W20260709 12:26:07.647114 17801 consensus_peers.cc:597] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b -> Peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Couldn't send request to peer e2909cff1da44ad387f96a8ae3aa5afb. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.66:39651: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:26:07.648639 18333 leader_election.cc:336] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 33 pre-election: RPC error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Network error: Client connection negotiation failed: client connection to 127.16.29.66:39651: connect: Connection refused (error 111)
I20260709 12:26:07.649111 18333 leader_election.cc:304] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:26:07.649787 29375 raft_consensus.cc:2749] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:26:07.655753 16501 raft_consensus.cc:2272] T f8f6bfcb37be41c6917282a17239ad06 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:07.658599 16501 tablet_replica.cc:333] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b: stopping tablet replica
W20260709 12:26:07.660392 18333 consensus_peers.cc:597] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da -> Peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Couldn't send request to peer e2909cff1da44ad387f96a8ae3aa5afb. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.66:39651: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 12:26:07.661839 16501 raft_consensus.cc:2243] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 30 LEADER]: Raft consensus shutting down.
I20260709 12:26:07.663116 16501 raft_consensus.cc:2272] T bd3425997bcf4fe7abc0399fd06e54d9 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:07.666254 16501 tablet_replica.cc:333] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b: stopping tablet replica
I20260709 12:26:07.667059 16501 raft_consensus.cc:2243] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 27 LEADER]: Raft consensus shutting down.
I20260709 12:26:07.668308 16501 raft_consensus.cc:2272] T 1a4040191930435786e2eeea338bb3d3 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:07.670344 16501 tablet_replica.cc:333] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b: stopping tablet replica
I20260709 12:26:07.670938 16501 raft_consensus.cc:2243] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 29 LEADER]: Raft consensus shutting down.
I20260709 12:26:07.672009 16501 raft_consensus.cc:2272] T ea2f2b5792f84bbba2738f1ce1e90122 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:07.674444 16501 tablet_replica.cc:333] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b: stopping tablet replica
I20260709 12:26:07.675314 16501 raft_consensus.cc:2243] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 28 LEADER]: Raft consensus shutting down.
I20260709 12:26:07.677084 16501 raft_consensus.cc:2272] T 53e726e4ebbb4ca58b6a6102c9b0850f P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Raft consensus is shut down!
W20260709 12:26:07.679013 17801 consensus_peers.cc:597] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b -> Peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Couldn't send request to peer e2909cff1da44ad387f96a8ae3aa5afb. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.66:39651: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 12:26:07.680588 16501 tablet_replica.cc:333] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b: stopping tablet replica
I20260709 12:26:07.686611 16501 raft_consensus.cc:2243] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 37 LEADER]: Raft consensus shutting down.
I20260709 12:26:07.704042 16501 raft_consensus.cc:2272] T e2bcfe1b94464b6d9620cc3b8a17a2ad P e063541542a94969898868bf6eb5ce1b [term 37 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:07.707103 16501 tablet_replica.cc:333] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b: stopping tablet replica
I20260709 12:26:07.708048 16501 raft_consensus.cc:2243] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 27 LEADER]: Raft consensus shutting down.
I20260709 12:26:07.709195 16501 raft_consensus.cc:2272] T 0540d79e2ec2406588b295145aa6d54d P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:07.711943 16501 tablet_replica.cc:333] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b: stopping tablet replica
I20260709 12:26:07.712831 16501 raft_consensus.cc:2243] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:07.713621 16501 raft_consensus.cc:2272] T 5ad2828ec58f490a9ea564d9805b56b7 P e063541542a94969898868bf6eb5ce1b [term 28 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:07.716475 16501 tablet_replica.cc:333] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b: stopping tablet replica
I20260709 12:26:07.717366 16501 raft_consensus.cc:2243] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:07.724510 16501 raft_consensus.cc:2272] T 3219dcbfd86340ecbde507517060ff81 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:07.727200 16501 tablet_replica.cc:333] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b: stopping tablet replica
I20260709 12:26:07.728039 16501 raft_consensus.cc:2243] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 31 LEADER]: Raft consensus shutting down.
I20260709 12:26:07.729218 16501 raft_consensus.cc:2272] T b5462347eb74482b8cd11beb1d39b8a9 P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:07.732043 16501 tablet_replica.cc:333] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b: stopping tablet replica
I20260709 12:26:07.732851 16501 raft_consensus.cc:2243] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 29 LEADER]: Raft consensus shutting down.
I20260709 12:26:07.734254 16501 raft_consensus.cc:2272] T ab4d9071484d4dc0998db462cb01a10f P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:07.742786 16501 tablet_replica.cc:333] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b: stopping tablet replica
I20260709 12:26:07.750499 16501 raft_consensus.cc:2243] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:07.751674 16501 raft_consensus.cc:2272] T 0e02889970bc40dc8ca2e81483f211e0 P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Raft consensus is shut down!
W20260709 12:26:07.754304 18333 consensus_peers.cc:597] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da -> Peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Couldn't send request to peer e2909cff1da44ad387f96a8ae3aa5afb. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.66:39651: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 12:26:07.758890 16501 tablet_replica.cc:333] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b: stopping tablet replica
I20260709 12:26:07.759840 16501 raft_consensus.cc:2243] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 32 LEADER]: Raft consensus shutting down.
I20260709 12:26:07.761040 16501 raft_consensus.cc:2272] T 19a1253e500942cd8a1a0f568159cfa3 P e063541542a94969898868bf6eb5ce1b [term 32 FOLLOWER]: Raft consensus is shut down!
W20260709 12:26:07.770617 17801 consensus_peers.cc:597] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b -> Peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Couldn't send request to peer e2909cff1da44ad387f96a8ae3aa5afb. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.66:39651: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 12:26:07.772168 16501 tablet_replica.cc:333] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b: stopping tablet replica
I20260709 12:26:07.773087 16501 raft_consensus.cc:2243] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:07.773936 16501 raft_consensus.cc:2272] T 19dad6ab92894d708ec8eb7e8ac28185 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:07.776721 16501 tablet_replica.cc:333] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b: stopping tablet replica
I20260709 12:26:07.779747 16501 raft_consensus.cc:2243] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 37 LEADER]: Raft consensus shutting down.
I20260709 12:26:07.781006 16501 raft_consensus.cc:2272] T 28062a50142448aa97994888b4765920 P e063541542a94969898868bf6eb5ce1b [term 37 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:07.783723 16501 tablet_replica.cc:333] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b: stopping tablet replica
I20260709 12:26:07.784535 16501 raft_consensus.cc:2243] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 33 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:07.785590 16501 raft_consensus.cc:2272] T a5732e71b9d54f8786b8c511d69943bf P e063541542a94969898868bf6eb5ce1b [term 33 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:07.790503 16501 tablet_replica.cc:333] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b: stopping tablet replica
I20260709 12:26:07.791311 16501 raft_consensus.cc:2243] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 31 LEADER]: Raft consensus shutting down.
I20260709 12:26:07.792546 16501 raft_consensus.cc:2272] T 45397041e3ac43c8901c5cb03f27c7b9 P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Raft consensus is shut down!
W20260709 12:26:07.794704 18334 consensus_peers.cc:597] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20260709 12:26:07.798333 16501 tablet_replica.cc:333] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b: stopping tablet replica
I20260709 12:26:07.799266 16501 raft_consensus.cc:2243] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 32 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:07.800149 16501 raft_consensus.cc:2272] T e4dcc70202ad4cd6880749fbb3a6d0c6 P e063541542a94969898868bf6eb5ce1b [term 32 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:07.802802 16501 tablet_replica.cc:333] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b: stopping tablet replica
I20260709 12:26:07.803570 16501 raft_consensus.cc:2243] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 36 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:07.804330 16501 raft_consensus.cc:2272] T 0f940f29438f475294357324452abe76 P e063541542a94969898868bf6eb5ce1b [term 36 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:07.807157 16501 tablet_replica.cc:333] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b: stopping tablet replica
I20260709 12:26:07.807960 16501 raft_consensus.cc:2243] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 33 LEADER]: Raft consensus shutting down.
I20260709 12:26:07.809105 16501 raft_consensus.cc:2272] T 3de52d85935746e9a1e770fef3529988 P e063541542a94969898868bf6eb5ce1b [term 33 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:07.811712 16501 tablet_replica.cc:333] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b: stopping tablet replica
I20260709 12:26:07.812408 16501 raft_consensus.cc:2243] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 31 LEADER]: Raft consensus shutting down.
I20260709 12:26:07.813499 16501 raft_consensus.cc:2272] T 4092aabb101d411798b0354f88a5dd2b P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:07.819010 29375 raft_consensus.cc:493] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:26:07.819594 29375 raft_consensus.cc:515] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:26:07.820204 16501 tablet_replica.cc:333] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b: stopping tablet replica
I20260709 12:26:07.821139 16501 raft_consensus.cc:2243] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 31 LEADER]: Raft consensus shutting down.
I20260709 12:26:07.821832 29375 leader_election.cc:290] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:26:07.826406 18333 leader_election.cc:336] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Network error: Client connection negotiation failed: client connection to 127.16.29.66:39651: connect: Connection refused (error 111)
W20260709 12:26:07.828133 18334 leader_election.cc:336] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111)
I20260709 12:26:07.828524 18334 leader_election.cc:304] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:26:07.823074 16501 raft_consensus.cc:2272] T 11896555f5d7458dbb02a3203b27f1cf P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:07.833266 16501 tablet_replica.cc:333] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b: stopping tablet replica
I20260709 12:26:07.834836 16501 raft_consensus.cc:2243] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 30 LEADER]: Raft consensus shutting down.
I20260709 12:26:07.836758 16501 raft_consensus.cc:2272] T d522f70beb1f417cb98940d90b594823 P e063541542a94969898868bf6eb5ce1b [term 30 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:07.833560 29314 raft_consensus.cc:2749] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:26:07.840373 16501 tablet_replica.cc:333] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b: stopping tablet replica
I20260709 12:26:07.848587 16501 raft_consensus.cc:2243] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 33 LEADER]: Raft consensus shutting down.
I20260709 12:26:07.873808 16501 raft_consensus.cc:2272] T 3ed3f6bd4d904d42a3a48e346ed9227d P e063541542a94969898868bf6eb5ce1b [term 33 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:07.877641 16501 tablet_replica.cc:333] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b: stopping tablet replica
I20260709 12:26:07.879570 16501 raft_consensus.cc:2243] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 35 LEADER]: Raft consensus shutting down.
I20260709 12:26:07.882570 16501 raft_consensus.cc:2272] T 5f52bc21096e4519acb00ed1ea68c343 P e063541542a94969898868bf6eb5ce1b [term 35 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:07.889452 16501 tablet_replica.cc:333] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b: stopping tablet replica
I20260709 12:26:07.891124 16501 raft_consensus.cc:2243] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 37 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:07.894943 16501 raft_consensus.cc:2272] T 8b56346a04a14c5d9f73c5205b74cf2d P e063541542a94969898868bf6eb5ce1b [term 37 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:07.900621 16501 tablet_replica.cc:333] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b: stopping tablet replica
I20260709 12:26:07.902362 16501 raft_consensus.cc:2243] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:07.904022 16501 raft_consensus.cc:2272] T c64345b0a2614ba4a382b83feb235cad P e063541542a94969898868bf6eb5ce1b [term 31 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:07.907934 16501 tablet_replica.cc:333] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b: stopping tablet replica
I20260709 12:26:07.908679 16501 raft_consensus.cc:2243] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:07.909336 16501 raft_consensus.cc:2272] T 044ba5aee1d6499a975c6b29d932fd95 P e063541542a94969898868bf6eb5ce1b [term 27 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:07.911660 16501 tablet_replica.cc:333] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b: stopping tablet replica
I20260709 12:26:07.912338 16501 raft_consensus.cc:2243] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 29 LEADER]: Raft consensus shutting down.
I20260709 12:26:07.913580 16501 raft_consensus.cc:2272] T 217734cc2eb24a7a8d92bfef30ee5c52 P e063541542a94969898868bf6eb5ce1b [term 29 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:07.916165 16501 tablet_replica.cc:333] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b: stopping tablet replica
I20260709 12:26:07.916854 16501 raft_consensus.cc:2243] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 36 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:07.917454 16501 raft_consensus.cc:2272] T 3418610c3e1a4e28824db2b86acff690 P e063541542a94969898868bf6eb5ce1b [term 36 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:07.938705 16501 tablet_replica.cc:333] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b: stopping tablet replica
I20260709 12:26:07.939445 16501 raft_consensus.cc:2243] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 24 LEADER]: Raft consensus shutting down.
I20260709 12:26:07.941088 16501 raft_consensus.cc:2272] T a9aeb14eca4a43dcbeaccffdf7e28995 P e063541542a94969898868bf6eb5ce1b [term 24 FOLLOWER]: Raft consensus is shut down!
W20260709 12:26:07.944921 18333 consensus_peers.cc:597] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da -> Peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Couldn't send request to peer e2909cff1da44ad387f96a8ae3aa5afb. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.66:39651: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:26:07.962109 18333 consensus_peers.cc:597] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da -> Peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Couldn't send request to peer e2909cff1da44ad387f96a8ae3aa5afb. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.66:39651: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:26:08.070290 18333 consensus_peers.cc:597] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da -> Peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Couldn't send request to peer e2909cff1da44ad387f96a8ae3aa5afb. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.66:39651: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 12:26:08.246419 29382 raft_consensus.cc:493] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 36 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:26:08.247016 29382 raft_consensus.cc:515] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:26:08.246024 18333 consensus_peers.cc:597] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da -> Peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Couldn't send request to peer e2909cff1da44ad387f96a8ae3aa5afb. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.66:39651: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 12:26:08.254101 29382 leader_election.cc:290] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:26:08.258615 18334 leader_election.cc:336] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 37 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111)
W20260709 12:26:08.264529 18333 leader_election.cc:336] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 37 pre-election: RPC error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Network error: Client connection negotiation failed: client connection to 127.16.29.66:39651: connect: Connection refused (error 111)
I20260709 12:26:08.264989 18333 leader_election.cc:304] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:26:08.271477 29382 raft_consensus.cc:2749] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
W20260709 12:26:08.460028 18334 consensus_peers.cc:597] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20260709 12:26:08.604446 29375 raft_consensus.cc:493] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:26:08.605093 29375 raft_consensus.cc:515] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:26:08.610216 29375 leader_election.cc:290] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:26:08.615571 18334 leader_election.cc:336] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111)
W20260709 12:26:08.616336 18334 consensus_peers.cc:597] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 12:26:08.626472 18333 leader_election.cc:336] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Network error: Client connection negotiation failed: client connection to 127.16.29.66:39651: connect: Connection refused (error 111)
I20260709 12:26:08.627292 18333 leader_election.cc:304] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:26:08.628093 29375 raft_consensus.cc:2749] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:26:08.645407 29375 raft_consensus.cc:493] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:26:08.645995 29375 raft_consensus.cc:515] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:26:08.649245 29375 leader_election.cc:290] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:26:08.653584 18334 leader_election.cc:336] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111)
I20260709 12:26:08.656858 29381 raft_consensus.cc:493] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 33 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:26:08.657425 29381 raft_consensus.cc:515] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:26:08.659858 29381 leader_election.cc:290] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:26:08.665861 18334 leader_election.cc:336] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 34 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111)
I20260709 12:26:08.672585 29375 raft_consensus.cc:493] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:26:08.673311 29375 raft_consensus.cc:515] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:26:08.675623 29375 leader_election.cc:290] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:26:08.681478 18334 leader_election.cc:336] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111)
W20260709 12:26:08.682545 18333 leader_election.cc:336] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Network error: Client connection negotiation failed: client connection to 127.16.29.66:39651: connect: Connection refused (error 111)
I20260709 12:26:08.683053 18333 leader_election.cc:304] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:26:08.684052 29375 raft_consensus.cc:2749] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
W20260709 12:26:08.693138 18333 leader_election.cc:336] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 34 pre-election: RPC error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Network error: Client connection negotiation failed: client connection to 127.16.29.66:39651: connect: Connection refused (error 111)
I20260709 12:26:08.693900 18333 leader_election.cc:304] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:26:08.694643 29375 raft_consensus.cc:2749] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:26:08.695483 29375 raft_consensus.cc:493] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 36 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:26:08.695966 29375 raft_consensus.cc:515] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:26:08.698856 29375 leader_election.cc:290] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:26:08.703747 18334 leader_election.cc:336] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 37 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111)
W20260709 12:26:08.714303 18333 leader_election.cc:336] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837): Network error: Client connection negotiation failed: client connection to 127.16.29.67:36837: connect: Connection refused (error 111)
I20260709 12:26:08.714847 18333 leader_election.cc:304] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:26:08.715715 29375 raft_consensus.cc:2749] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
W20260709 12:26:08.719143 18333 leader_election.cc:336] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 37 pre-election: RPC error from VoteRequest() call to peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837): Network error: Client connection negotiation failed: client connection to 127.16.29.67:36837: connect: Connection refused (error 111)
I20260709 12:26:08.719599 18333 leader_election.cc:304] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:26:08.720422 29375 raft_consensus.cc:2749] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:26:08.735919 29375 raft_consensus.cc:493] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:26:08.736402 29375 raft_consensus.cc:515] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:26:08.738293 29375 leader_election.cc:290] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:26:08.743115 29375 raft_consensus.cc:493] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:26:08.743646 29375 raft_consensus.cc:515] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:26:08.747989 29375 leader_election.cc:290] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:26:08.753686 18334 leader_election.cc:336] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111)
W20260709 12:26:08.782706 18333 leader_election.cc:336] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837): Network error: Client connection negotiation failed: client connection to 127.16.29.67:36837: connect: Connection refused (error 111)
I20260709 12:26:08.783272 18333 leader_election.cc:304] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:26:08.784080 29375 raft_consensus.cc:2749] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
W20260709 12:26:08.786268 18333 leader_election.cc:336] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837): Network error: Client connection negotiation failed: client connection to 127.16.29.67:36837: connect: Connection refused (error 111)
W20260709 12:26:08.787145 18333 leader_election.cc:336] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Network error: Client connection negotiation failed: client connection to 127.16.29.66:39651: connect: Connection refused (error 111)
I20260709 12:26:08.787549 18333 leader_election.cc:304] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:26:08.788491 29375 raft_consensus.cc:2749] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:26:08.793763 29375 raft_consensus.cc:493] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:26:08.794292 29375 raft_consensus.cc:515] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:26:08.796567 29375 leader_election.cc:290] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:26:08.813094 18333 leader_election.cc:336] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Network error: Client connection negotiation failed: client connection to 127.16.29.66:39651: connect: Connection refused (error 111)
W20260709 12:26:08.813858 18333 leader_election.cc:336] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837): Network error: Client connection negotiation failed: client connection to 127.16.29.67:36837: connect: Connection refused (error 111)
I20260709 12:26:08.814280 18333 leader_election.cc:304] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:26:08.815021 29375 raft_consensus.cc:2749] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:26:08.843407 29375 raft_consensus.cc:493] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:26:08.844343 29381 raft_consensus.cc:493] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 32 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:26:08.844884 29381 raft_consensus.cc:515] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:26:08.847834 29381 leader_election.cc:290] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:26:08.850834 29375 raft_consensus.cc:515] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:26:08.854544 18334 leader_election.cc:336] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 33 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111)
I20260709 12:26:08.868804 29375 leader_election.cc:290] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:26:08.869349 29314 raft_consensus.cc:493] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:26:08.873246 29314 raft_consensus.cc:515] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:26:08.875655 29314 leader_election.cc:290] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:26:08.889758 29314 raft_consensus.cc:493] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 37 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:26:08.890372 29314 raft_consensus.cc:515] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:26:08.893288 29314 leader_election.cc:290] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:26:08.901404 18334 leader_election.cc:336] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111)
I20260709 12:26:08.902463 29381 raft_consensus.cc:493] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:26:08.903028 29381 raft_consensus.cc:515] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:26:08.905583 29381 leader_election.cc:290] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:26:08.907925 29381 raft_consensus.cc:493] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:26:08.908732 29381 raft_consensus.cc:515] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:26:08.910823 29381 leader_election.cc:290] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:26:08.906668 29375 raft_consensus.cc:493] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:26:08.911456 29375 raft_consensus.cc:515] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:26:08.914829 29375 leader_election.cc:290] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:26:08.915385 29384 raft_consensus.cc:493] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:26:08.917141 29384 raft_consensus.cc:515] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
W20260709 12:26:08.926344 18333 leader_election.cc:336] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837): Network error: Client connection negotiation failed: client connection to 127.16.29.67:36837: connect: Connection refused (error 111)
W20260709 12:26:08.926936 18333 leader_election.cc:336] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837): Network error: Client connection negotiation failed: client connection to 127.16.29.67:36837: connect: Connection refused (error 111)
I20260709 12:26:08.927300 18333 leader_election.cc:304] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:26:08.928165 29381 raft_consensus.cc:2749] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:26:08.919502 29384 leader_election.cc:290] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:26:08.938426 18334 leader_election.cc:336] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111)
W20260709 12:26:08.928192 18333 leader_election.cc:336] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 38 pre-election: RPC error from VoteRequest() call to peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837): Network error: Client connection negotiation failed: client connection to 127.16.29.67:36837: connect: Connection refused (error 111)
W20260709 12:26:08.941275 18333 leader_election.cc:336] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 33 pre-election: RPC error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Network error: Client connection negotiation failed: client connection to 127.16.29.66:39651: connect: Connection refused (error 111)
I20260709 12:26:08.941671 18333 leader_election.cc:304] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:26:08.942559 18333 leader_election.cc:336] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Network error: Client connection negotiation failed: client connection to 127.16.29.66:39651: connect: Connection refused (error 111)
I20260709 12:26:08.942960 29384 raft_consensus.cc:2749] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:26:08.942948 18333 leader_election.cc:304] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:26:08.943941 29381 raft_consensus.cc:2749] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
W20260709 12:26:08.944324 18333 leader_election.cc:336] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 38 pre-election: RPC error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Network error: Client connection negotiation failed: client connection to 127.16.29.66:39651: connect: Connection refused (error 111)
I20260709 12:26:08.944762 18333 leader_election.cc:304] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:26:08.945420 29381 raft_consensus.cc:2749] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
W20260709 12:26:08.949595 18333 leader_election.cc:336] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837): Network error: Client connection negotiation failed: client connection to 127.16.29.67:36837: connect: Connection refused (error 111)
W20260709 12:26:08.950140 18333 leader_election.cc:336] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837): Network error: Client connection negotiation failed: client connection to 127.16.29.67:36837: connect: Connection refused (error 111)
W20260709 12:26:08.950958 18333 leader_election.cc:336] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837): Network error: Client connection negotiation failed: client connection to 127.16.29.67:36837: connect: Connection refused (error 111)
I20260709 12:26:08.951323 18333 leader_election.cc:304] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
W20260709 12:26:08.952339 18333 leader_election.cc:336] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837): Network error: Client connection negotiation failed: client connection to 127.16.29.67:36837: connect: Connection refused (error 111)
I20260709 12:26:08.952831 29381 raft_consensus.cc:2749] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
W20260709 12:26:08.953125 18333 leader_election.cc:336] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Network error: Client connection negotiation failed: client connection to 127.16.29.66:39651: connect: Connection refused (error 111)
I20260709 12:26:08.953548 18333 leader_election.cc:304] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:26:08.954409 18333 leader_election.cc:336] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Network error: Client connection negotiation failed: client connection to 127.16.29.66:39651: connect: Connection refused (error 111)
I20260709 12:26:08.954807 18333 leader_election.cc:304] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:26:08.954882 29381 raft_consensus.cc:2749] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:26:08.955508 29384 raft_consensus.cc:2749] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
W20260709 12:26:08.955703 18333 leader_election.cc:336] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Network error: Client connection negotiation failed: client connection to 127.16.29.66:39651: connect: Connection refused (error 111)
I20260709 12:26:08.956135 18333 leader_election.cc:304] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:26:08.957294 29381 raft_consensus.cc:2749] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:26:08.963958 29384 raft_consensus.cc:493] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 36 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:26:08.964504 29384 raft_consensus.cc:515] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:26:08.966540 29384 leader_election.cc:290] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:26:08.976477 18334 leader_election.cc:336] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 37 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111)
W20260709 12:26:08.988632 18333 leader_election.cc:336] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 37 pre-election: RPC error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Network error: Client connection negotiation failed: client connection to 127.16.29.66:39651: connect: Connection refused (error 111)
I20260709 12:26:08.989241 18333 leader_election.cc:304] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:26:08.990103 29381 raft_consensus.cc:2749] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:26:09.010061 29381 raft_consensus.cc:493] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:26:09.010643 29381 raft_consensus.cc:515] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:26:09.013252 29381 leader_election.cc:290] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:26:09.020182 18334 leader_election.cc:336] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111)
W20260709 12:26:09.027351 18333 leader_election.cc:336] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837): Network error: Client connection negotiation failed: client connection to 127.16.29.67:36837: connect: Connection refused (error 111)
I20260709 12:26:09.028013 18333 leader_election.cc:304] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:26:09.028851 29381 raft_consensus.cc:2749] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:26:09.046164 29381 raft_consensus.cc:493] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:26:09.046780 29381 raft_consensus.cc:515] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:26:09.053977 29381 leader_election.cc:290] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:26:09.054332 18334 leader_election.cc:336] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111)
W20260709 12:26:09.064679 18333 leader_election.cc:336] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Network error: Client connection negotiation failed: client connection to 127.16.29.66:39651: connect: Connection refused (error 111)
I20260709 12:26:09.065235 18333 leader_election.cc:304] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:26:09.066159 29384 raft_consensus.cc:2749] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
W20260709 12:26:09.067170 18333 leader_election.cc:336] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837): Network error: Client connection negotiation failed: client connection to 127.16.29.67:36837: connect: Connection refused (error 111)
I20260709 12:26:09.137419 29381 raft_consensus.cc:493] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:26:09.138033 29381 raft_consensus.cc:515] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:26:09.140367 29314 raft_consensus.cc:493] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:26:09.140935 29314 raft_consensus.cc:515] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:26:09.142901 29314 leader_election.cc:290] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
I20260709 12:26:09.144790 29381 leader_election.cc:290] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:26:09.146111 18334 consensus_peers.cc:597] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:26:09.150182 18334 leader_election.cc:336] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111)
I20260709 12:26:09.155623 29381 raft_consensus.cc:493] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:26:09.156316 29381 raft_consensus.cc:515] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:26:09.164860 29381 leader_election.cc:290] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:26:09.177711 18334 leader_election.cc:336] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111)
W20260709 12:26:09.189929 18333 leader_election.cc:336] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837): Network error: Client connection negotiation failed: client connection to 127.16.29.67:36837: connect: Connection refused (error 111)
W20260709 12:26:09.190586 18333 leader_election.cc:336] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837): Network error: Client connection negotiation failed: client connection to 127.16.29.67:36837: connect: Connection refused (error 111)
I20260709 12:26:09.190991 18333 leader_election.cc:304] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:26:09.191751 29381 raft_consensus.cc:2749] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
W20260709 12:26:09.192981 18333 leader_election.cc:336] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837): Network error: Client connection negotiation failed: client connection to 127.16.29.67:36837: connect: Connection refused (error 111)
I20260709 12:26:09.193432 18333 leader_election.cc:304] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e063541542a94969898868bf6eb5ce1b
I20260709 12:26:09.194160 29381 raft_consensus.cc:2749] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
W20260709 12:26:09.197449 18333 leader_election.cc:336] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Network error: Client connection negotiation failed: client connection to 127.16.29.66:39651: connect: Connection refused (error 111)
I20260709 12:26:09.197970 18333 leader_election.cc:304] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:26:09.198786 29381 raft_consensus.cc:2749] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:26:09.258005 29381 raft_consensus.cc:493] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 37 FOLLOWER]: Starting pre-election (detected failure of leader e063541542a94969898868bf6eb5ce1b)
I20260709 12:26:09.258591 29381 raft_consensus.cc:515] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:26:09.261122 29381 leader_election.cc:290] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:26:09.271461 18333 leader_election.cc:336] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 38 pre-election: RPC error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Network error: Client connection negotiation failed: client connection to 127.16.29.66:39651: connect: Connection refused (error 111)
W20260709 12:26:09.272295 18333 leader_election.cc:336] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 38 pre-election: RPC error from VoteRequest() call to peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837): Network error: Client connection negotiation failed: client connection to 127.16.29.67:36837: connect: Connection refused (error 111)
I20260709 12:26:09.272771 18333 leader_election.cc:304] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: e063541542a94969898868bf6eb5ce1b, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:26:09.273648 29381 raft_consensus.cc:2749] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
W20260709 12:26:09.344508 18406 debug-util.cc:398] Leaking SignalData structure 0x7b08000c48c0 after lost signal to thread 16504
W20260709 12:26:09.345615 18406 debug-util.cc:398] Leaking SignalData structure 0x7b080038e560 after lost signal to thread 17621
W20260709 12:26:09.346933 18406 debug-util.cc:398] Leaking SignalData structure 0x7b08003af5a0 after lost signal to thread 18409
W20260709 12:26:09.382959 18334 consensus_peers.cc:597] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:26:09.404274 18334 consensus_peers.cc:597] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 12:26:09.483076 18334 consensus_peers.cc:597] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260709 12:26:09.543200 18333 consensus_peers.cc:597] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da -> Peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837): Couldn't send request to peer e063541542a94969898868bf6eb5ce1b. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.67:36837: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:26:09.594338 18334 consensus_peers.cc:597] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 12:26:09.640815 18333 consensus_peers.cc:597] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da -> Peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837): Couldn't send request to peer e063541542a94969898868bf6eb5ce1b. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.67:36837: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:26:09.654368 18333 consensus_peers.cc:597] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da -> Peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837): Couldn't send request to peer e063541542a94969898868bf6eb5ce1b. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.67:36837: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:26:09.697645 16501 tablet_server.cc:196] TabletServer@127.16.29.67:0 shutdown complete.
W20260709 12:26:09.762861 18333 consensus_peers.cc:597] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da -> Peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837): Couldn't send request to peer e063541542a94969898868bf6eb5ce1b. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.67:36837: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:26:09.777413 18333 consensus_peers.cc:597] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da -> Peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Couldn't send request to peer e2909cff1da44ad387f96a8ae3aa5afb. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.66:39651: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:26:09.823670 18333 consensus_peers.cc:597] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da -> Peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Couldn't send request to peer e2909cff1da44ad387f96a8ae3aa5afb. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.66:39651: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:26:09.826788 18333 consensus_peers.cc:597] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da -> Peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Couldn't send request to peer e2909cff1da44ad387f96a8ae3aa5afb. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.66:39651: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20260709 12:26:09.915277 29314 raft_consensus.cc:493] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:26:09.917232 29314 raft_consensus.cc:515] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:26:09.921284 29314 leader_election.cc:290] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
I20260709 12:26:09.921965 29314 raft_consensus.cc:493] T a5732e71b9d54f8786b8c511d69943bf P 334a4e1f59944544b3b3ff36616b57da [term 33 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
W20260709 12:26:09.922075 18334 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111) [suppressed 547 similar messages]
I20260709 12:26:09.924145 29314 raft_consensus.cc:515] T a5732e71b9d54f8786b8c511d69943bf P 334a4e1f59944544b3b3ff36616b57da [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "e063541542a94969898868bf6eb5ce1b" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 36837 } attrs { replace: true } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:26:09.930547 29314 leader_election.cc:290] T a5732e71b9d54f8786b8c511d69943bf P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651), e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837)
W20260709 12:26:09.930660 18333 consensus_peers.cc:597] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da -> Peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837): Couldn't send request to peer e063541542a94969898868bf6eb5ce1b. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.67:36837: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:26:09.937597 18333 leader_election.cc:336] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Network error: Client connection negotiation failed: client connection to 127.16.29.66:39651: connect: Connection refused (error 111)
W20260709 12:26:09.938381 18334 leader_election.cc:336] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111)
I20260709 12:26:09.939019 18334 leader_election.cc:304] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:26:09.940222 18334 leader_election.cc:336] T a5732e71b9d54f8786b8c511d69943bf P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 34 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111)
I20260709 12:26:09.940888 29314 raft_consensus.cc:2749] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
W20260709 12:26:09.944619 18333 leader_election.cc:336] T a5732e71b9d54f8786b8c511d69943bf P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 34 pre-election: RPC error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Network error: Client connection negotiation failed: client connection to 127.16.29.66:39651: connect: Connection refused (error 111)
I20260709 12:26:10.023169 16501 tablet_server.cc:179] TabletServer@127.16.29.68:0 shutting down...
I20260709 12:26:10.039620 18333 leader_election.cc:304] T a5732e71b9d54f8786b8c511d69943bf P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
W20260709 12:26:10.052224 18333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:39651, user_credentials={real_user=slave}} blocked reactor thread for 108559us
W20260709 12:26:10.052961 18333 leader_election.cc:336] T a5732e71b9d54f8786b8c511d69943bf P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 34 pre-election: RPC error from VoteRequest() call to peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837): Network error: Client connection negotiation failed: client connection to 127.16.29.67:36837: connect: Connection refused (error 111)
I20260709 12:26:10.055045 29381 raft_consensus.cc:2749] T a5732e71b9d54f8786b8c511d69943bf P 334a4e1f59944544b3b3ff36616b57da [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:26:10.061599 29314 raft_consensus.cc:493] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Starting pre-election (detected failure of leader e2909cff1da44ad387f96a8ae3aa5afb)
I20260709 12:26:10.062548 29314 raft_consensus.cc:515] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0da169ff82674dc1947055ce43de68a3" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 33071 } } peers { permanent_uuid: "e2909cff1da44ad387f96a8ae3aa5afb" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 39651 } } peers { permanent_uuid: "334a4e1f59944544b3b3ff36616b57da" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 42577 } attrs { promote: false } }
I20260709 12:26:10.065310 29314 leader_election.cc:290] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071), e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651)
W20260709 12:26:10.076524 18334 leader_election.cc:336] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111)
W20260709 12:26:10.109037 18333 consensus_peers.cc:597] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da -> Peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837): Couldn't send request to peer e063541542a94969898868bf6eb5ce1b. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.67:36837: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:26:10.111248 18333 consensus_peers.cc:597] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da -> Peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837): Couldn't send request to peer e063541542a94969898868bf6eb5ce1b. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.67:36837: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:26:10.111837 18333 consensus_peers.cc:597] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da -> Peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837): Couldn't send request to peer e063541542a94969898868bf6eb5ce1b. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.67:36837: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:26:10.114389 18333 consensus_peers.cc:597] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da -> Peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Couldn't send request to peer e2909cff1da44ad387f96a8ae3aa5afb. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.66:39651: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:26:10.114931 18333 leader_election.cc:336] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Network error: Client connection negotiation failed: client connection to 127.16.29.66:39651: connect: Connection refused (error 111)
I20260709 12:26:10.115329 18333 leader_election.cc:304] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 334a4e1f59944544b3b3ff36616b57da; no voters: 0da169ff82674dc1947055ce43de68a3, e2909cff1da44ad387f96a8ae3aa5afb
I20260709 12:26:10.116156 29314 raft_consensus.cc:2749] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
W20260709 12:26:10.129420 18333 consensus_peers.cc:597] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da -> Peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837): Couldn't send request to peer e063541542a94969898868bf6eb5ce1b. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.67:36837: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:26:10.130352 18333 consensus_peers.cc:597] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da -> Peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837): Couldn't send request to peer e063541542a94969898868bf6eb5ce1b. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.67:36837: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:26:10.130925 18333 consensus_peers.cc:597] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da -> Peer e2909cff1da44ad387f96a8ae3aa5afb (127.16.29.66:39651): Couldn't send request to peer e2909cff1da44ad387f96a8ae3aa5afb. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.66:39651: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:26:10.135660 18333 consensus_peers.cc:597] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da -> Peer e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837): Couldn't send request to peer e063541542a94969898868bf6eb5ce1b. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.67:36837: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:26:10.140924 18334 consensus_peers.cc:597] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20260709 12:26:10.185041 16501 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:26:10.186365 16501 tablet_replica.cc:333] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da: stopping tablet replica
I20260709 12:26:10.188246 16501 raft_consensus.cc:2243] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:10.189733 16501 raft_consensus.cc:2272] T d522f70beb1f417cb98940d90b594823 P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:10.196141 16501 tablet_replica.cc:333] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da: stopping tablet replica
I20260709 12:26:10.197496 16501 raft_consensus.cc:2243] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 28 LEADER]: Raft consensus shutting down.
I20260709 12:26:10.199870 16501 raft_consensus.cc:2272] T 7cb3768d215e409fb9f5baf18d124442 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:10.204027 16501 tablet_replica.cc:333] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da: stopping tablet replica
I20260709 12:26:10.205021 16501 raft_consensus.cc:2243] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:10.206562 16501 raft_consensus.cc:2272] T 6fcb5bdaa4be4d619bb945ffd44090ba P 334a4e1f59944544b3b3ff36616b57da [term 24 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:10.211283 16501 tablet_replica.cc:333] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da: stopping tablet replica
I20260709 12:26:10.212464 16501 raft_consensus.cc:2243] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:10.213617 16501 raft_consensus.cc:2272] T 8cab6f01c6a2485cb5024322a1bd5462 P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:10.217859 16501 tablet_replica.cc:333] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da: stopping tablet replica
I20260709 12:26:10.219036 16501 raft_consensus.cc:2243] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:10.220268 16501 raft_consensus.cc:2272] T 45397041e3ac43c8901c5cb03f27c7b9 P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:10.224608 16501 tablet_replica.cc:333] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da: stopping tablet replica
I20260709 12:26:10.225559 16501 raft_consensus.cc:2243] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:10.228164 16501 raft_consensus.cc:2272] T ea7da467817a4ba88aa5797c3f8f493f P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:10.231961 16501 tablet_replica.cc:333] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da: stopping tablet replica
I20260709 12:26:10.232975 16501 raft_consensus.cc:2243] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 37 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:10.234107 16501 raft_consensus.cc:2272] T 28062a50142448aa97994888b4765920 P 334a4e1f59944544b3b3ff36616b57da [term 37 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:10.238112 16501 tablet_replica.cc:333] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da: stopping tablet replica
I20260709 12:26:10.239179 16501 raft_consensus.cc:2243] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 33 LEADER]: Raft consensus shutting down.
I20260709 12:26:10.241302 16501 raft_consensus.cc:2272] T 10f96c39fba3496696f64385105bfeaa P 334a4e1f59944544b3b3ff36616b57da [term 33 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:10.244840 16501 tablet_replica.cc:333] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da: stopping tablet replica
I20260709 12:26:10.245702 16501 raft_consensus.cc:2243] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 32 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:10.247596 16501 raft_consensus.cc:2272] T 84421f094faa40f39b30b4c3d0737639 P 334a4e1f59944544b3b3ff36616b57da [term 32 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:10.251950 16501 tablet_replica.cc:333] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da: stopping tablet replica
I20260709 12:26:10.253124 16501 raft_consensus.cc:2243] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 32 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:10.256214 16501 raft_consensus.cc:2272] T 154e6b769a8d476ba26f7fe3a521fc8d P 334a4e1f59944544b3b3ff36616b57da [term 32 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:10.259918 16501 tablet_replica.cc:333] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da: stopping tablet replica
I20260709 12:26:10.260875 16501 raft_consensus.cc:2243] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 28 LEADER]: Raft consensus shutting down.
I20260709 12:26:10.262737 16501 raft_consensus.cc:2272] T 5ad2828ec58f490a9ea564d9805b56b7 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:10.266603 16501 tablet_replica.cc:333] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da: stopping tablet replica
I20260709 12:26:10.267714 16501 raft_consensus.cc:2243] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:10.268941 16501 raft_consensus.cc:2272] T 4092aabb101d411798b0354f88a5dd2b P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:10.273176 16501 tablet_replica.cc:333] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da: stopping tablet replica
I20260709 12:26:10.274225 16501 raft_consensus.cc:2243] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 27 LEADER]: Raft consensus shutting down.
I20260709 12:26:10.276463 16501 raft_consensus.cc:2272] T 044ba5aee1d6499a975c6b29d932fd95 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:10.280473 16501 tablet_replica.cc:333] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da: stopping tablet replica
I20260709 12:26:10.281287 16501 raft_consensus.cc:2243] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:10.282912 16501 raft_consensus.cc:2272] T 50f9ff8f24a14a079db34b1ba88fa278 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:10.286509 16501 tablet_replica.cc:333] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da: stopping tablet replica
I20260709 12:26:10.287274 16501 raft_consensus.cc:2243] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 37 LEADER]: Raft consensus shutting down.
I20260709 12:26:10.288764 16501 raft_consensus.cc:2272] T 8b56346a04a14c5d9f73c5205b74cf2d P 334a4e1f59944544b3b3ff36616b57da [term 37 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:10.291967 16501 tablet_replica.cc:333] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da: stopping tablet replica
I20260709 12:26:10.292770 16501 raft_consensus.cc:2243] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 28 LEADER]: Raft consensus shutting down.
I20260709 12:26:10.294281 16501 raft_consensus.cc:2272] T 12b1fcd35e12402e8d6df7504c578e01 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:10.298017 16501 tablet_replica.cc:333] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da: stopping tablet replica
I20260709 12:26:10.298928 16501 raft_consensus.cc:2243] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 31 LEADER]: Raft consensus shutting down.
I20260709 12:26:10.300520 16501 raft_consensus.cc:2272] T c64345b0a2614ba4a382b83feb235cad P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:10.303835 16501 tablet_replica.cc:333] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da: stopping tablet replica
I20260709 12:26:10.304704 16501 raft_consensus.cc:2243] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 36 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:10.305752 16501 raft_consensus.cc:2272] T ac984176358d453a8800cc9e0e304061 P 334a4e1f59944544b3b3ff36616b57da [term 36 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:10.309036 16501 tablet_replica.cc:333] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da: stopping tablet replica
I20260709 12:26:10.309785 16501 raft_consensus.cc:2243] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 30 LEADER]: Raft consensus shutting down.
I20260709 12:26:10.311117 16501 raft_consensus.cc:2272] T 3219dcbfd86340ecbde507517060ff81 P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:10.314841 16501 tablet_replica.cc:333] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da: stopping tablet replica
I20260709 12:26:10.315747 16501 raft_consensus.cc:2243] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:10.316669 16501 raft_consensus.cc:2272] T 0540d79e2ec2406588b295145aa6d54d P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:10.320153 16501 tablet_replica.cc:333] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da: stopping tablet replica
I20260709 12:26:10.321048 16501 raft_consensus.cc:2243] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 31 LEADER]: Raft consensus shutting down.
I20260709 12:26:10.322505 16501 raft_consensus.cc:2272] T 0e02889970bc40dc8ca2e81483f211e0 P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:10.325732 16501 tablet_replica.cc:333] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da: stopping tablet replica
I20260709 12:26:10.326555 16501 raft_consensus.cc:2243] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 30 LEADER]: Raft consensus shutting down.
I20260709 12:26:10.328190 16501 raft_consensus.cc:2272] T 19dad6ab92894d708ec8eb7e8ac28185 P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:10.331616 16501 tablet_replica.cc:333] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da: stopping tablet replica
I20260709 12:26:10.332468 16501 raft_consensus.cc:2243] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 37 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:10.333794 16501 raft_consensus.cc:2272] T f5f0dd0890f44e35bbb838da838faa9c P 334a4e1f59944544b3b3ff36616b57da [term 37 FOLLOWER]: Raft consensus is shut down!
W20260709 12:26:10.336582 18334 consensus_peers.cc:597] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da -> Peer 0da169ff82674dc1947055ce43de68a3 (127.16.29.65:33071): Couldn't send request to peer 0da169ff82674dc1947055ce43de68a3. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:33071: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20260709 12:26:10.338145 16501 tablet_replica.cc:333] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da: stopping tablet replica
I20260709 12:26:10.339720 16501 raft_consensus.cc:2243] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:10.340900 16501 raft_consensus.cc:2272] T 69716f6d5e7549418fdd8d54d7b722ce P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:10.344233 16501 tablet_replica.cc:333] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da: stopping tablet replica
I20260709 12:26:10.345137 16501 raft_consensus.cc:2243] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:10.346191 16501 raft_consensus.cc:2272] T b5462347eb74482b8cd11beb1d39b8a9 P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:10.349323 16501 tablet_replica.cc:333] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da: stopping tablet replica
I20260709 12:26:10.350144 16501 raft_consensus.cc:2243] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 36 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:10.352382 16501 raft_consensus.cc:2272] T 9439565c5dbb4f4ba3375aaef2c9add5 P 334a4e1f59944544b3b3ff36616b57da [term 36 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:10.355419 16501 tablet_replica.cc:333] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da: stopping tablet replica
I20260709 12:26:10.356200 16501 raft_consensus.cc:2243] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:10.356971 16501 raft_consensus.cc:2272] T 8a208e793881438b975b83755dc05cfa P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:10.359918 16501 tablet_replica.cc:333] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da: stopping tablet replica
I20260709 12:26:10.360746 16501 raft_consensus.cc:2243] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 34 LEADER]: Raft consensus shutting down.
I20260709 12:26:10.362438 16501 raft_consensus.cc:2272] T d11ed0a8f1094765abe028dc60aa8bed P 334a4e1f59944544b3b3ff36616b57da [term 34 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:10.365907 16501 tablet_replica.cc:333] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da: stopping tablet replica
I20260709 12:26:10.366864 16501 raft_consensus.cc:2243] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:10.369503 16501 raft_consensus.cc:2272] T e5dabc1347544bd8b4e81fa5903d4a8b P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:10.373238 16501 tablet_replica.cc:333] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da: stopping tablet replica
I20260709 12:26:10.374081 16501 raft_consensus.cc:2243] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 36 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:10.375120 16501 raft_consensus.cc:2272] T 23c0cde5227f470388b8ed1fc0786119 P 334a4e1f59944544b3b3ff36616b57da [term 36 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:10.378032 16501 tablet_replica.cc:333] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da: stopping tablet replica
I20260709 12:26:10.378840 16501 raft_consensus.cc:2243] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 36 LEADER]: Raft consensus shutting down.
I20260709 12:26:10.380290 16501 raft_consensus.cc:2272] T 0f940f29438f475294357324452abe76 P 334a4e1f59944544b3b3ff36616b57da [term 36 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:10.383073 16501 tablet_replica.cc:333] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da: stopping tablet replica
I20260709 12:26:10.383860 16501 raft_consensus.cc:2243] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 33 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:10.384647 16501 raft_consensus.cc:2272] T 44ad5363c5954da4a1bff5857780fd5c P 334a4e1f59944544b3b3ff36616b57da [term 33 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:10.387840 16501 tablet_replica.cc:333] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da: stopping tablet replica
I20260709 12:26:10.388597 16501 raft_consensus.cc:2243] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 32 LEADER]: Raft consensus shutting down.
I20260709 12:26:10.390887 16501 raft_consensus.cc:2272] T 56b500bd1b914ebfb3f19afc8a330665 P 334a4e1f59944544b3b3ff36616b57da [term 32 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:10.393819 16501 tablet_replica.cc:333] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da: stopping tablet replica
I20260709 12:26:10.394595 16501 raft_consensus.cc:2243] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 32 LEADER]: Raft consensus shutting down.
I20260709 12:26:10.396123 16501 raft_consensus.cc:2272] T e4dcc70202ad4cd6880749fbb3a6d0c6 P 334a4e1f59944544b3b3ff36616b57da [term 32 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:10.398308 16501 tablet_replica.cc:333] T a5732e71b9d54f8786b8c511d69943bf P 334a4e1f59944544b3b3ff36616b57da: stopping tablet replica
I20260709 12:26:10.398981 16501 raft_consensus.cc:2243] T a5732e71b9d54f8786b8c511d69943bf P 334a4e1f59944544b3b3ff36616b57da [term 33 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:10.400194 16501 raft_consensus.cc:2272] T a5732e71b9d54f8786b8c511d69943bf P 334a4e1f59944544b3b3ff36616b57da [term 33 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:10.403090 16501 tablet_replica.cc:333] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da: stopping tablet replica
I20260709 12:26:10.403627 16501 raft_consensus.cc:2243] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:10.404453 16501 raft_consensus.cc:2272] T ea2f2b5792f84bbba2738f1ce1e90122 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:10.406466 16501 tablet_replica.cc:333] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da: stopping tablet replica
I20260709 12:26:10.406996 16501 raft_consensus.cc:2243] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:10.407794 16501 raft_consensus.cc:2272] T 11896555f5d7458dbb02a3203b27f1cf P 334a4e1f59944544b3b3ff36616b57da [term 31 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:10.409883 16501 tablet_replica.cc:333] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da: stopping tablet replica
I20260709 12:26:10.410415 16501 raft_consensus.cc:2243] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:10.410979 16501 raft_consensus.cc:2272] T 217734cc2eb24a7a8d92bfef30ee5c52 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:10.413424 16501 tablet_replica.cc:333] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da: stopping tablet replica
I20260709 12:26:10.414281 16501 raft_consensus.cc:2243] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:10.415027 16501 raft_consensus.cc:2272] T 1a4040191930435786e2eeea338bb3d3 P 334a4e1f59944544b3b3ff36616b57da [term 27 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:10.417721 16501 tablet_replica.cc:333] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da: stopping tablet replica
I20260709 12:26:10.418310 16501 raft_consensus.cc:2243] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:10.419237 16501 raft_consensus.cc:2272] T d48b7de191684318aa08f60a7a395387 P 334a4e1f59944544b3b3ff36616b57da [term 28 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:10.421890 16501 tablet_replica.cc:333] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da: stopping tablet replica
I20260709 12:26:10.422451 16501 raft_consensus.cc:2243] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 35 LEADER]: Raft consensus shutting down.
I20260709 12:26:10.423514 16501 raft_consensus.cc:2272] T 9160d9f98b8c406db2771d9ea2e0995f P 334a4e1f59944544b3b3ff36616b57da [term 35 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:10.425792 16501 tablet_replica.cc:333] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da: stopping tablet replica
I20260709 12:26:10.426301 16501 raft_consensus.cc:2243] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:10.426961 16501 raft_consensus.cc:2272] T bd3425997bcf4fe7abc0399fd06e54d9 P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:10.429255 16501 tablet_replica.cc:333] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da: stopping tablet replica
I20260709 12:26:10.429834 16501 raft_consensus.cc:2243] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:10.430701 16501 raft_consensus.cc:2272] T f8f6bfcb37be41c6917282a17239ad06 P 334a4e1f59944544b3b3ff36616b57da [term 30 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:10.433226 16501 tablet_replica.cc:333] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da: stopping tablet replica
I20260709 12:26:10.433789 16501 raft_consensus.cc:2243] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:10.434469 16501 raft_consensus.cc:2272] T ab4d9071484d4dc0998db462cb01a10f P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:10.437326 16501 tablet_replica.cc:333] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da: stopping tablet replica
I20260709 12:26:10.437911 16501 raft_consensus.cc:2243] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:10.438493 16501 raft_consensus.cc:2272] T 8e417cb980e44a77962d2e3dfd58aad0 P 334a4e1f59944544b3b3ff36616b57da [term 29 FOLLOWER]: Raft consensus is shut down!
W20260709 12:26:10.954314 17618 debug-util.cc:398] Leaking SignalData structure 0x7b080013a4e0 after lost signal to thread 16504
W20260709 12:26:10.955600 17618 debug-util.cc:398] Leaking SignalData structure 0x7b080036ef60 after lost signal to thread 17621
W20260709 12:26:10.956955 17618 debug-util.cc:398] Leaking SignalData structure 0x7b080040a420 after lost signal to thread 18409
I20260709 12:26:11.724380 16501 tablet_server.cc:196] TabletServer@127.16.29.68:0 shutdown complete.
I20260709 12:26:11.872926 16501 master.cc:562] Master@127.16.29.126:39149 shutting down...
W20260709 12:26:11.941133 17635 auto_rebalancer.cc:873] Removing replace marker failed: Catalog manager is not initialized. State: Closing
W20260709 12:26:11.941514 17635 auto_rebalancer.cc:901] Could not move replica: Service unavailable: Catalog manager is not initialized. State: Closing
W20260709 12:26:11.941821 17635 auto_rebalancer.cc:330] scheduled replica move failed to complete: Service unavailable: Catalog manager is not initialized. State: Closing
W20260709 12:26:49.465653 17571 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.29.67:36837: connect: Connection refused (error 111) [suppressed 33 similar messages]
W20260709 12:26:49.469901 17571 catalog_manager.cc:4755] TS e063541542a94969898868bf6eb5ce1b (127.16.29.67:36837): ChangeConfig:REMOVE_PEER RPC failed for tablet d522f70beb1f417cb98940d90b594823: Network error: Client connection negotiation failed: client connection to 127.16.29.67:36837: connect: Connection refused (error 111)
I20260709 12:26:53.407313 16501 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 2193af31cb894ec9b3d65d939a2a6851 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:26:53.408684 16501 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 2193af31cb894ec9b3d65d939a2a6851 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:53.409195 16501 tablet_replica.cc:333] T 00000000000000000000000000000000 P 2193af31cb894ec9b3d65d939a2a6851: stopping tablet replica
I20260709 12:26:53.643116 16501 master.cc:584] Master@127.16.29.126:39149 shutdown complete.
I20260709 12:26:53.700887 16501 test_util.cc:182] -----------------------------------------------
I20260709 12:26:53.701120 16501 test_util.cc:183] Had failures, leaving test files at /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599551912677-16501-0
[  FAILED  ] LeaderRebalancerTest.AddTserver (399362 ms)
[ RUN      ] LeaderRebalancerTest.RestartTserver
I20260709 12:26:53.704814 16501 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.16.29.126:41639
I20260709 12:26:53.706081 16501 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:26:53.712621 29395 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:26:53.712970 29396 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:26:53.713037 29398 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:26:53.714473 16501 server_base.cc:1061] running on GCE node
I20260709 12:26:53.720218 16501 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:26:53.720453 16501 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:26:53.720661 16501 hybrid_clock.cc:648] HybridClock initialized: now 1783600013720626 us; error 0 us; skew 500 ppm
I20260709 12:26:53.721212 16501 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:26:53.724047 16501 webserver.cc:533] Webserver started at http://127.16.29.126:33957/ using document root <none> and password file <none>
I20260709 12:26:53.724567 16501 fs_manager.cc:362] Metadata directory not provided
I20260709 12:26:53.724766 16501 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:26:53.725046 16501 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:26:53.726357 16501 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599551912677-16501-0/minicluster-data/master-0-root/instance:
uuid: "8cb294248c0f442d9c16f6b66cd0648e"
format_stamp: "Formatted at 2026-07-09 12:26:53 on dist-test-slave-5n9v"
I20260709 12:26:53.731086 16501 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.006s	sys 0.000s
I20260709 12:26:53.734800 29403 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:26:53.735670 16501 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.000s
I20260709 12:26:53.736022 16501 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599551912677-16501-0/minicluster-data/master-0-root
uuid: "8cb294248c0f442d9c16f6b66cd0648e"
format_stamp: "Formatted at 2026-07-09 12:26:53 on dist-test-slave-5n9v"
I20260709 12:26:53.736341 16501 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599551912677-16501-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599551912677-16501-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599551912677-16501-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:26:53.758028 16501 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:26:53.759313 16501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:26:53.802724 16501 rpc_server.cc:307] RPC server started. Bound to: 127.16.29.126:41639
I20260709 12:26:53.802842 29454 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.29.126:41639 every 8 connection(s)
I20260709 12:26:53.807610 29455 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:53.824503 29455 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8cb294248c0f442d9c16f6b66cd0648e: Bootstrap starting.
I20260709 12:26:53.830302 29455 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8cb294248c0f442d9c16f6b66cd0648e: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:53.835080 29455 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8cb294248c0f442d9c16f6b66cd0648e: No bootstrap required, opened a new log
I20260709 12:26:53.837396 29455 raft_consensus.cc:359] T 00000000000000000000000000000000 P 8cb294248c0f442d9c16f6b66cd0648e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cb294248c0f442d9c16f6b66cd0648e" member_type: VOTER }
I20260709 12:26:53.837863 29455 raft_consensus.cc:385] T 00000000000000000000000000000000 P 8cb294248c0f442d9c16f6b66cd0648e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:53.838281 29455 raft_consensus.cc:740] T 00000000000000000000000000000000 P 8cb294248c0f442d9c16f6b66cd0648e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8cb294248c0f442d9c16f6b66cd0648e, State: Initialized, Role: FOLLOWER
I20260709 12:26:53.838902 29455 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8cb294248c0f442d9c16f6b66cd0648e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cb294248c0f442d9c16f6b66cd0648e" member_type: VOTER }
I20260709 12:26:53.839444 29455 raft_consensus.cc:399] T 00000000000000000000000000000000 P 8cb294248c0f442d9c16f6b66cd0648e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260709 12:26:53.839676 29455 raft_consensus.cc:493] T 00000000000000000000000000000000 P 8cb294248c0f442d9c16f6b66cd0648e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260709 12:26:53.840050 29455 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 8cb294248c0f442d9c16f6b66cd0648e [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:53.845109 29455 raft_consensus.cc:515] T 00000000000000000000000000000000 P 8cb294248c0f442d9c16f6b66cd0648e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cb294248c0f442d9c16f6b66cd0648e" member_type: VOTER }
I20260709 12:26:53.845626 29455 leader_election.cc:304] T 00000000000000000000000000000000 P 8cb294248c0f442d9c16f6b66cd0648e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8cb294248c0f442d9c16f6b66cd0648e; no voters: 
I20260709 12:26:53.846956 29455 leader_election.cc:290] T 00000000000000000000000000000000 P 8cb294248c0f442d9c16f6b66cd0648e [CANDIDATE]: Term 1 election: Requested vote from peers 
I20260709 12:26:53.847520 29458 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 8cb294248c0f442d9c16f6b66cd0648e [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:26:53.849072 29458 raft_consensus.cc:697] T 00000000000000000000000000000000 P 8cb294248c0f442d9c16f6b66cd0648e [term 1 LEADER]: Becoming Leader. State: Replica: 8cb294248c0f442d9c16f6b66cd0648e, State: Running, Role: LEADER
I20260709 12:26:53.850135 29455 sys_catalog.cc:565] T 00000000000000000000000000000000 P 8cb294248c0f442d9c16f6b66cd0648e [sys.catalog]: configured and running, proceeding with master startup.
I20260709 12:26:53.849822 29458 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8cb294248c0f442d9c16f6b66cd0648e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cb294248c0f442d9c16f6b66cd0648e" member_type: VOTER }
I20260709 12:26:53.858771 29459 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8cb294248c0f442d9c16f6b66cd0648e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8cb294248c0f442d9c16f6b66cd0648e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cb294248c0f442d9c16f6b66cd0648e" member_type: VOTER } }
I20260709 12:26:53.858961 29460 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8cb294248c0f442d9c16f6b66cd0648e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8cb294248c0f442d9c16f6b66cd0648e. Latest consensus state: current_term: 1 leader_uuid: "8cb294248c0f442d9c16f6b66cd0648e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cb294248c0f442d9c16f6b66cd0648e" member_type: VOTER } }
I20260709 12:26:53.859542 29459 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8cb294248c0f442d9c16f6b66cd0648e [sys.catalog]: This master's current role is: LEADER
I20260709 12:26:53.859683 29460 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8cb294248c0f442d9c16f6b66cd0648e [sys.catalog]: This master's current role is: LEADER
I20260709 12:26:53.861394 29468 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260709 12:26:53.866550 29468 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260709 12:26:53.867956 16501 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260709 12:26:53.876969 29468 catalog_manager.cc:1383] Generated new cluster ID: 6a4ede5409e34d328dcd6a70e1af5de4
I20260709 12:26:53.877322 29468 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260709 12:26:53.903869 29468 catalog_manager.cc:1406] Generated new certificate authority record
I20260709 12:26:53.905373 29468 catalog_manager.cc:1540] Loading token signing keys...
I20260709 12:26:53.922312 29468 catalog_manager.cc:6070] T 00000000000000000000000000000000 P 8cb294248c0f442d9c16f6b66cd0648e: Generated new TSK 0
I20260709 12:26:53.923020 29468 catalog_manager.cc:1550] Initializing in-progress tserver states...
I20260709 12:26:53.935259 16501 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:26:53.941934 29476 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:26:53.943284 29477 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:26:53.944939 29479 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:26:53.945986 16501 server_base.cc:1061] running on GCE node
I20260709 12:26:53.949079 16501 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:26:53.949296 16501 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:26:53.949432 16501 hybrid_clock.cc:648] HybridClock initialized: now 1783600013949420 us; error 0 us; skew 500 ppm
I20260709 12:26:53.950030 16501 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:26:53.952977 16501 webserver.cc:533] Webserver started at http://127.16.29.65:33857/ using document root <none> and password file <none>
I20260709 12:26:53.953452 16501 fs_manager.cc:362] Metadata directory not provided
I20260709 12:26:53.953612 16501 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:26:53.953822 16501 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:26:53.955067 16501 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599551912677-16501-0/minicluster-data/ts-0-root/instance:
uuid: "584dac86183a41d9b3bea1c2085e62fa"
format_stamp: "Formatted at 2026-07-09 12:26:53 on dist-test-slave-5n9v"
I20260709 12:26:53.959945 16501 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.002s	sys 0.004s
I20260709 12:26:53.963603 29484 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:26:53.964511 16501 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.003s	sys 0.000s
I20260709 12:26:53.964864 16501 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599551912677-16501-0/minicluster-data/ts-0-root
uuid: "584dac86183a41d9b3bea1c2085e62fa"
format_stamp: "Formatted at 2026-07-09 12:26:53 on dist-test-slave-5n9v"
I20260709 12:26:53.965226 16501 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599551912677-16501-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599551912677-16501-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599551912677-16501-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:26:53.994881 16501 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:26:53.996274 16501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:26:53.997864 16501 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:26:54.000308 16501 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:26:54.000533 16501 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:26:54.000771 16501 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:26:54.000960 16501 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:26:54.044796 16501 rpc_server.cc:307] RPC server started. Bound to: 127.16.29.65:35033
I20260709 12:26:54.044919 29546 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.29.65:35033 every 8 connection(s)
I20260709 12:26:54.050199 16501 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:26:54.059468 29551 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:26:54.060917 29554 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:26:54.060876 29552 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:26:54.066777 16501 server_base.cc:1061] running on GCE node
I20260709 12:26:54.071132 16501 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:26:54.071381 16501 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:26:54.071574 16501 hybrid_clock.cc:648] HybridClock initialized: now 1783600014071555 us; error 0 us; skew 500 ppm
I20260709 12:26:54.072175 16501 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:26:54.072621 29547 heartbeater.cc:344] Connected to a master server at 127.16.29.126:41639
I20260709 12:26:54.073045 29547 heartbeater.cc:461] Registering TS with master...
I20260709 12:26:54.073889 29547 heartbeater.cc:507] Master 127.16.29.126:41639 requested a full tablet report, sending...
I20260709 12:26:54.075160 16501 webserver.cc:533] Webserver started at http://127.16.29.66:34701/ using document root <none> and password file <none>
I20260709 12:26:54.075680 16501 fs_manager.cc:362] Metadata directory not provided
I20260709 12:26:54.075935 16501 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:26:54.076068 29420 ts_manager.cc:194] Registered new tserver with Master: 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033)
I20260709 12:26:54.076233 16501 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:26:54.077553 16501 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599551912677-16501-0/minicluster-data/ts-1-root/instance:
uuid: "d73d92fb35234d16824c55689b5fce28"
format_stamp: "Formatted at 2026-07-09 12:26:54 on dist-test-slave-5n9v"
I20260709 12:26:54.078081 29420 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:53434
I20260709 12:26:54.082360 16501 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.006s	sys 0.000s
I20260709 12:26:54.086108 29559 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:26:54.086925 16501 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.000s
I20260709 12:26:54.087253 16501 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599551912677-16501-0/minicluster-data/ts-1-root
uuid: "d73d92fb35234d16824c55689b5fce28"
format_stamp: "Formatted at 2026-07-09 12:26:54 on dist-test-slave-5n9v"
I20260709 12:26:54.087570 16501 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599551912677-16501-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599551912677-16501-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599551912677-16501-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:26:54.106088 16501 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:26:54.107419 16501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:26:54.109064 16501 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:26:54.111801 16501 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:26:54.112105 16501 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.001s	sys 0.000s
I20260709 12:26:54.112463 16501 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:26:54.112672 16501 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:26:54.156594 16501 rpc_server.cc:307] RPC server started. Bound to: 127.16.29.66:36825
I20260709 12:26:54.156708 29621 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.29.66:36825 every 8 connection(s)
I20260709 12:26:54.161649 16501 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:26:54.169637 29625 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:26:54.171700 29626 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:26:54.173756 29628 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:26:54.174731 29622 heartbeater.cc:344] Connected to a master server at 127.16.29.126:41639
I20260709 12:26:54.174856 16501 server_base.cc:1061] running on GCE node
I20260709 12:26:54.175315 29622 heartbeater.cc:461] Registering TS with master...
I20260709 12:26:54.176213 16501 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20260709 12:26:54.176339 29622 heartbeater.cc:507] Master 127.16.29.126:41639 requested a full tablet report, sending...
W20260709 12:26:54.176534 16501 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:26:54.176851 16501 hybrid_clock.cc:648] HybridClock initialized: now 1783600014176832 us; error 0 us; skew 500 ppm
I20260709 12:26:54.177565 16501 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:26:54.178853 29420 ts_manager.cc:194] Registered new tserver with Master: d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825)
I20260709 12:26:54.180498 16501 webserver.cc:533] Webserver started at http://127.16.29.67:33695/ using document root <none> and password file <none>
I20260709 12:26:54.180629 29420 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:53442
I20260709 12:26:54.181140 16501 fs_manager.cc:362] Metadata directory not provided
I20260709 12:26:54.181401 16501 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:26:54.181777 16501 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:26:54.183094 16501 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599551912677-16501-0/minicluster-data/ts-2-root/instance:
uuid: "d13c58ad8ac441c4a9b396eb04b1a476"
format_stamp: "Formatted at 2026-07-09 12:26:54 on dist-test-slave-5n9v"
I20260709 12:26:54.188014 16501 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.005s	sys 0.002s
I20260709 12:26:54.192003 29633 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:26:54.192865 16501 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20260709 12:26:54.193197 16501 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599551912677-16501-0/minicluster-data/ts-2-root
uuid: "d13c58ad8ac441c4a9b396eb04b1a476"
format_stamp: "Formatted at 2026-07-09 12:26:54 on dist-test-slave-5n9v"
I20260709 12:26:54.193535 16501 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599551912677-16501-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599551912677-16501-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599551912677-16501-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:26:54.215655 16501 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:26:54.216909 16501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:26:54.218387 16501 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:26:54.220739 16501 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:26:54.220973 16501 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:26:54.221215 16501 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:26:54.221387 16501 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:26:54.266075 16501 rpc_server.cc:307] RPC server started. Bound to: 127.16.29.67:38209
I20260709 12:26:54.266177 29695 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.29.67:38209 every 8 connection(s)
I20260709 12:26:54.271946 16501 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:26:54.280156 29699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:26:54.280640 29700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:26:54.283488 29696 heartbeater.cc:344] Connected to a master server at 127.16.29.126:41639
I20260709 12:26:54.283953 29696 heartbeater.cc:461] Registering TS with master...
W20260709 12:26:54.284775 29703 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:26:54.284973 29696 heartbeater.cc:507] Master 127.16.29.126:41639 requested a full tablet report, sending...
I20260709 12:26:54.286180 16501 server_base.cc:1061] running on GCE node
I20260709 12:26:54.287351 16501 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:26:54.287549 16501 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:26:54.287595 29420 ts_manager.cc:194] Registered new tserver with Master: d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209)
I20260709 12:26:54.287825 16501 hybrid_clock.cc:648] HybridClock initialized: now 1783600014287805 us; error 0 us; skew 500 ppm
I20260709 12:26:54.288676 16501 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:26:54.289731 29420 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:53446
I20260709 12:26:54.291427 16501 webserver.cc:533] Webserver started at http://127.16.29.68:33009/ using document root <none> and password file <none>
I20260709 12:26:54.292032 16501 fs_manager.cc:362] Metadata directory not provided
I20260709 12:26:54.292244 16501 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:26:54.292516 16501 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:26:54.293666 16501 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599551912677-16501-0/minicluster-data/ts-3-root/instance:
uuid: "c6610c00ffb948ce9c34f41dec3d49e8"
format_stamp: "Formatted at 2026-07-09 12:26:54 on dist-test-slave-5n9v"
I20260709 12:26:54.298367 16501 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.006s	sys 0.000s
I20260709 12:26:54.302263 29707 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:26:54.303282 16501 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.000s	sys 0.002s
I20260709 12:26:54.303618 16501 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599551912677-16501-0/minicluster-data/ts-3-root
uuid: "c6610c00ffb948ce9c34f41dec3d49e8"
format_stamp: "Formatted at 2026-07-09 12:26:54 on dist-test-slave-5n9v"
I20260709 12:26:54.303978 16501 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599551912677-16501-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599551912677-16501-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599551912677-16501-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:26:54.329352 16501 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:26:54.331079 16501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:26:54.333122 16501 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:26:54.335374 16501 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:26:54.335642 16501 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:26:54.335916 16501 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:26:54.336102 16501 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:26:54.381567 16501 rpc_server.cc:307] RPC server started. Bound to: 127.16.29.68:43895
I20260709 12:26:54.381711 29769 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.29.68:43895 every 8 connection(s)
I20260709 12:26:54.396507 29770 heartbeater.cc:344] Connected to a master server at 127.16.29.126:41639
I20260709 12:26:54.396978 29770 heartbeater.cc:461] Registering TS with master...
I20260709 12:26:54.397780 29770 heartbeater.cc:507] Master 127.16.29.126:41639 requested a full tablet report, sending...
I20260709 12:26:54.399675 29420 ts_manager.cc:194] Registered new tserver with Master: c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895)
I20260709 12:26:54.400173 16501 internal_mini_cluster.cc:371] 4 TS(s) registered with all masters after 0.01518925s
I20260709 12:26:54.401149 29420 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:53460
I20260709 12:26:54.431075 29420 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53474:
name: "test-workload"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
  rows: "\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}""\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}"
  indirect_data: """"
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I20260709 12:26:54.685127 29735 tablet_service.cc:1511] Processing CreateTablet for tablet 55061e62184b45f0bb1726adc1a4af81 (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260709 12:26:54.686939 29735 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 55061e62184b45f0bb1726adc1a4af81. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:54.689109 29734 tablet_service.cc:1511] Processing CreateTablet for tablet 592801934cad4789b9ce8dc09f1470bd (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260709 12:26:54.690701 29734 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 592801934cad4789b9ce8dc09f1470bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:54.695300 29729 tablet_service.cc:1511] Processing CreateTablet for tablet 4685c2e824c14fc1bcd45c16169798cc (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260709 12:26:54.696086 29733 tablet_service.cc:1511] Processing CreateTablet for tablet 742beffede4f407d8a72909d2b147e6d (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260709 12:26:54.696863 29729 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4685c2e824c14fc1bcd45c16169798cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:54.698441 29732 tablet_service.cc:1511] Processing CreateTablet for tablet fb6e78ccc01c4707af34cddf81619eab (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20260709 12:26:54.701220 29731 tablet_service.cc:1511] Processing CreateTablet for tablet 3e9451c3797c4496a66dfdb3e9fb5e38 (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260709 12:26:54.703709 29730 tablet_service.cc:1511] Processing CreateTablet for tablet 43c7bfb33af74d4f912e5153e8a54b3e (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20260709 12:26:54.708171 29732 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb6e78ccc01c4707af34cddf81619eab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:54.709080 29727 tablet_service.cc:1511] Processing CreateTablet for tablet cdf706ffabc34902ad3d1bf2cee7889e (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20260709 12:26:54.710570 29727 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cdf706ffabc34902ad3d1bf2cee7889e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:54.711441 29731 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e9451c3797c4496a66dfdb3e9fb5e38. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:54.711920 29728 tablet_service.cc:1511] Processing CreateTablet for tablet 8627b58b4f42491597125eb45f8d3b61 (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260709 12:26:54.713346 29728 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8627b58b4f42491597125eb45f8d3b61. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:54.713999 29733 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 742beffede4f407d8a72909d2b147e6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:54.717832 29726 tablet_service.cc:1511] Processing CreateTablet for tablet 62f8c9472b214576b9002e02536a3ace (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20260709 12:26:54.719588 29726 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62f8c9472b214576b9002e02536a3ace. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:54.720322 29730 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 43c7bfb33af74d4f912e5153e8a54b3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:54.725868 29587 tablet_service.cc:1511] Processing CreateTablet for tablet 55061e62184b45f0bb1726adc1a4af81 (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260709 12:26:54.727521 29587 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 55061e62184b45f0bb1726adc1a4af81. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:54.729523 29586 tablet_service.cc:1511] Processing CreateTablet for tablet d21ab1641501475b829863275dc8182a (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260709 12:26:54.731771 29585 tablet_service.cc:1511] Processing CreateTablet for tablet 592801934cad4789b9ce8dc09f1470bd (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260709 12:26:54.732903 29586 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d21ab1641501475b829863275dc8182a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:54.733937 29585 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 592801934cad4789b9ce8dc09f1470bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:54.739992 29584 tablet_service.cc:1511] Processing CreateTablet for tablet 7ae3b542012d4f35a9c363eee22976ec (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260709 12:26:54.743075 29583 tablet_service.cc:1511] Processing CreateTablet for tablet fb6e78ccc01c4707af34cddf81619eab (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20260709 12:26:54.743934 29578 tablet_service.cc:1511] Processing CreateTablet for tablet d42259e436d241b29fb4ded16d9b6455 (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20260709 12:26:54.745570 29582 tablet_service.cc:1511] Processing CreateTablet for tablet 43c7bfb33af74d4f912e5153e8a54b3e (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20260709 12:26:54.746315 29511 tablet_service.cc:1511] Processing CreateTablet for tablet 592801934cad4789b9ce8dc09f1470bd (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260709 12:26:54.747078 29508 tablet_service.cc:1511] Processing CreateTablet for tablet fb6e78ccc01c4707af34cddf81619eab (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20260709 12:26:54.748096 29581 tablet_service.cc:1511] Processing CreateTablet for tablet fcb6a7f720b14dc98e821f663357a0bb (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260709 12:26:54.749121 29510 tablet_service.cc:1511] Processing CreateTablet for tablet 742beffede4f407d8a72909d2b147e6d (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260709 12:26:54.750535 29580 tablet_service.cc:1511] Processing CreateTablet for tablet 4685c2e824c14fc1bcd45c16169798cc (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260709 12:26:54.751565 29509 tablet_service.cc:1511] Processing CreateTablet for tablet 7ae3b542012d4f35a9c363eee22976ec (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260709 12:26:54.753018 29579 tablet_service.cc:1511] Processing CreateTablet for tablet cdf706ffabc34902ad3d1bf2cee7889e (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20260709 12:26:54.754009 29507 tablet_service.cc:1511] Processing CreateTablet for tablet 3e9451c3797c4496a66dfdb3e9fb5e38 (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260709 12:26:54.755518 29512 tablet_service.cc:1511] Processing CreateTablet for tablet d21ab1641501475b829863275dc8182a (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260709 12:26:54.741706 29584 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ae3b542012d4f35a9c363eee22976ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:54.757093 29505 tablet_service.cc:1511] Processing CreateTablet for tablet 4685c2e824c14fc1bcd45c16169798cc (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260709 12:26:54.759465 29504 tablet_service.cc:1511] Processing CreateTablet for tablet 8627b58b4f42491597125eb45f8d3b61 (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260709 12:26:54.760493 29582 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 43c7bfb33af74d4f912e5153e8a54b3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:54.762779 29503 tablet_service.cc:1511] Processing CreateTablet for tablet d42259e436d241b29fb4ded16d9b6455 (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20260709 12:26:54.764420 29580 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4685c2e824c14fc1bcd45c16169798cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:54.764750 29506 tablet_service.cc:1511] Processing CreateTablet for tablet fcb6a7f720b14dc98e821f663357a0bb (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260709 12:26:54.766335 29578 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d42259e436d241b29fb4ded16d9b6455. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:54.772327 29579 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cdf706ffabc34902ad3d1bf2cee7889e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:54.747758 29511 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 592801934cad4789b9ce8dc09f1470bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:54.776306 29505 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4685c2e824c14fc1bcd45c16169798cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:54.777073 29509 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ae3b542012d4f35a9c363eee22976ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:54.777624 29581 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fcb6a7f720b14dc98e821f663357a0bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:54.778393 29507 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e9451c3797c4496a66dfdb3e9fb5e38. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:54.778923 29583 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb6e78ccc01c4707af34cddf81619eab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:54.779685 29510 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 742beffede4f407d8a72909d2b147e6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:54.780645 29512 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d21ab1641501475b829863275dc8182a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:54.781384 29506 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fcb6a7f720b14dc98e821f663357a0bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:54.782085 29508 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb6e78ccc01c4707af34cddf81619eab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:54.782814 29504 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8627b58b4f42491597125eb45f8d3b61. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:54.783695 29503 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d42259e436d241b29fb4ded16d9b6455. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:54.777107 29735 tablet_service.cc:1511] Processing CreateTablet for tablet 9633e3b1d78f403097e4a00b7d4b869b (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20260709 12:26:54.833494 29735 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9633e3b1d78f403097e4a00b7d4b869b. 1 dirs total, 0 dirs full, 0 dirs failed
W20260709 12:26:54.862526 29467 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:26:54.895928 29790 tablet_bootstrap.cc:492] T 592801934cad4789b9ce8dc09f1470bd P c6610c00ffb948ce9c34f41dec3d49e8: Bootstrap starting.
I20260709 12:26:54.899989 29661 tablet_service.cc:1511] Processing CreateTablet for tablet 55061e62184b45f0bb1726adc1a4af81 (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260709 12:26:54.901417 29661 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 55061e62184b45f0bb1726adc1a4af81. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:54.902501 29655 tablet_service.cc:1511] Processing CreateTablet for tablet fcb6a7f720b14dc98e821f663357a0bb (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260709 12:26:54.903281 29660 tablet_service.cc:1511] Processing CreateTablet for tablet d21ab1641501475b829863275dc8182a (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260709 12:26:54.904021 29655 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fcb6a7f720b14dc98e821f663357a0bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:54.904711 29660 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d21ab1641501475b829863275dc8182a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:54.906311 29654 tablet_service.cc:1511] Processing CreateTablet for tablet 8627b58b4f42491597125eb45f8d3b61 (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260709 12:26:54.906875 29658 tablet_service.cc:1511] Processing CreateTablet for tablet 7ae3b542012d4f35a9c363eee22976ec (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260709 12:26:54.907796 29654 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8627b58b4f42491597125eb45f8d3b61. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:54.909226 29657 tablet_service.cc:1511] Processing CreateTablet for tablet 3e9451c3797c4496a66dfdb3e9fb5e38 (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260709 12:26:54.910970 29657 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e9451c3797c4496a66dfdb3e9fb5e38. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:54.915488 29652 tablet_service.cc:1511] Processing CreateTablet for tablet d42259e436d241b29fb4ded16d9b6455 (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20260709 12:26:54.917114 29652 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d42259e436d241b29fb4ded16d9b6455. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:54.917423 29734 tablet_service.cc:1511] Processing CreateTablet for tablet ed057e06dec148a0b9d7062beeb9d005 (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20260709 12:26:54.919284 29659 tablet_service.cc:1511] Processing CreateTablet for tablet 742beffede4f407d8a72909d2b147e6d (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260709 12:26:54.922804 29653 tablet_service.cc:1511] Processing CreateTablet for tablet cdf706ffabc34902ad3d1bf2cee7889e (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20260709 12:26:54.924137 29659 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 742beffede4f407d8a72909d2b147e6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:54.913431 29656 tablet_service.cc:1511] Processing CreateTablet for tablet 43c7bfb33af74d4f912e5153e8a54b3e (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20260709 12:26:54.926409 29658 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ae3b542012d4f35a9c363eee22976ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:54.928036 29653 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cdf706ffabc34902ad3d1bf2cee7889e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:54.959359 29734 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ed057e06dec148a0b9d7062beeb9d005. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:54.969489 29656 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 43c7bfb33af74d4f912e5153e8a54b3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:54.983201 29790 tablet_bootstrap.cc:654] T 592801934cad4789b9ce8dc09f1470bd P c6610c00ffb948ce9c34f41dec3d49e8: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:55.003080 29792 tablet_bootstrap.cc:492] T fb6e78ccc01c4707af34cddf81619eab P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap starting.
I20260709 12:26:55.005620 29508 tablet_service.cc:1511] Processing CreateTablet for tablet 62f8c9472b214576b9002e02536a3ace (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20260709 12:26:55.061738 29508 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62f8c9472b214576b9002e02536a3ace. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:55.016005 29654 tablet_service.cc:1511] Processing CreateTablet for tablet ed057e06dec148a0b9d7062beeb9d005 (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20260709 12:26:55.017606 29794 tablet_bootstrap.cc:492] T 8627b58b4f42491597125eb45f8d3b61 P d13c58ad8ac441c4a9b396eb04b1a476: Bootstrap starting.
I20260709 12:26:55.065833 29654 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ed057e06dec148a0b9d7062beeb9d005. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:55.015393 29652 tablet_service.cc:1511] Processing CreateTablet for tablet 9633e3b1d78f403097e4a00b7d4b869b (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20260709 12:26:55.072770 29652 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9633e3b1d78f403097e4a00b7d4b869b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:55.064570 29579 tablet_service.cc:1511] Processing CreateTablet for tablet 76140ad6258543118a9bfeee7cf0d5fc (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20260709 12:26:55.081030 29547 heartbeater.cc:499] Master 127.16.29.126:41639 was elected leader, sending a full tablet report...
I20260709 12:26:55.081445 29579 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 76140ad6258543118a9bfeee7cf0d5fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:55.085512 29794 tablet_bootstrap.cc:654] T 8627b58b4f42491597125eb45f8d3b61 P d13c58ad8ac441c4a9b396eb04b1a476: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:55.020184 29512 tablet_service.cc:1511] Processing CreateTablet for tablet 9633e3b1d78f403097e4a00b7d4b869b (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20260709 12:26:55.030403 29731 tablet_service.cc:1511] Processing CreateTablet for tablet b3c02c910d2b43cc9265bb1d98491cd3 (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20260709 12:26:55.037325 29728 tablet_service.cc:1511] Processing CreateTablet for tablet f4516786e5564e9eac08ca472f79a9f0 (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20260709 12:26:55.092247 29728 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f4516786e5564e9eac08ca472f79a9f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:55.098807 29657 tablet_service.cc:1511] Processing CreateTablet for tablet 2707271fd46f456aa765026642a3efa9 (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20260709 12:26:55.031508 29506 tablet_service.cc:1511] Processing CreateTablet for tablet 2707271fd46f456aa765026642a3efa9 (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20260709 12:26:55.044639 29509 tablet_service.cc:1511] Processing CreateTablet for tablet b3c02c910d2b43cc9265bb1d98491cd3 (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20260709 12:26:55.047274 29584 tablet_service.cc:1511] Processing CreateTablet for tablet ed057e06dec148a0b9d7062beeb9d005 (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20260709 12:26:55.048296 29580 tablet_service.cc:1511] Processing CreateTablet for tablet 62f8c9472b214576b9002e02536a3ace (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20260709 12:26:55.048908 29729 tablet_service.cc:1511] Processing CreateTablet for tablet 8017a44219404faa8db8ff049c65205e (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260709 12:26:55.052026 29581 tablet_service.cc:1511] Processing CreateTablet for tablet 2707271fd46f456aa765026642a3efa9 (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20260709 12:26:55.052419 29732 tablet_service.cc:1511] Processing CreateTablet for tablet 76140ad6258543118a9bfeee7cf0d5fc (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20260709 12:26:55.099426 29507 tablet_service.cc:1511] Processing CreateTablet for tablet 8017a44219404faa8db8ff049c65205e (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260709 12:26:55.054610 29511 tablet_service.cc:1511] Processing CreateTablet for tablet 71e2de235adf46d9a2407659080d42ef (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20260709 12:26:55.059154 29727 tablet_service.cc:1511] Processing CreateTablet for tablet a46289e8fc824dd5a818e30df6e7ec3e (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20260709 12:26:55.054610 29510 tablet_service.cc:1511] Processing CreateTablet for tablet f4516786e5564e9eac08ca472f79a9f0 (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20260709 12:26:55.128505 29506 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2707271fd46f456aa765026642a3efa9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:55.132696 29658 tablet_service.cc:1511] Processing CreateTablet for tablet f4516786e5564e9eac08ca472f79a9f0 (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20260709 12:26:55.133227 29509 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b3c02c910d2b43cc9265bb1d98491cd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:55.083761 29733 tablet_service.cc:1511] Processing CreateTablet for tablet 00c6cf271c91400aa834467def9af9b1 (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20260709 12:26:55.132722 29503 tablet_service.cc:1511] Processing CreateTablet for tablet a46289e8fc824dd5a818e30df6e7ec3e (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20260709 12:26:55.120294 29586 tablet_service.cc:1511] Processing CreateTablet for tablet b3c02c910d2b43cc9265bb1d98491cd3 (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20260709 12:26:55.139448 29657 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2707271fd46f456aa765026642a3efa9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:55.148892 29584 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ed057e06dec148a0b9d7062beeb9d005. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:55.156057 29731 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b3c02c910d2b43cc9265bb1d98491cd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:55.161646 29583 tablet_service.cc:1511] Processing CreateTablet for tablet 71e2de235adf46d9a2407659080d42ef (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20260709 12:26:55.162781 29507 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8017a44219404faa8db8ff049c65205e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:55.162889 29586 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b3c02c910d2b43cc9265bb1d98491cd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:55.157555 29658 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f4516786e5564e9eac08ca472f79a9f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:55.169551 29510 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f4516786e5564e9eac08ca472f79a9f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:55.169682 29732 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 76140ad6258543118a9bfeee7cf0d5fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:55.172258 29729 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8017a44219404faa8db8ff049c65205e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:55.172382 29503 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a46289e8fc824dd5a818e30df6e7ec3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:55.172823 29726 tablet_service.cc:1511] Processing CreateTablet for tablet 3f41ce441fb047c8bbb741b1b9cbc0ec (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20260709 12:26:55.174091 29726 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3f41ce441fb047c8bbb741b1b9cbc0ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:55.175709 29733 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00c6cf271c91400aa834467def9af9b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:55.176646 29727 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a46289e8fc824dd5a818e30df6e7ec3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:55.176335 29655 tablet_service.cc:1511] Processing CreateTablet for tablet 76140ad6258543118a9bfeee7cf0d5fc (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20260709 12:26:55.178471 29730 tablet_service.cc:1511] Processing CreateTablet for tablet 58154d49c1a344f2b51366ede3c87de5 (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20260709 12:26:55.182533 29583 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 71e2de235adf46d9a2407659080d42ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:55.183490 29622 heartbeater.cc:499] Master 127.16.29.126:41639 was elected leader, sending a full tablet report...
I20260709 12:26:55.182130 29730 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 58154d49c1a344f2b51366ede3c87de5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:55.180485 29585 tablet_service.cc:1511] Processing CreateTablet for tablet 00c6cf271c91400aa834467def9af9b1 (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20260709 12:26:55.197831 29581 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2707271fd46f456aa765026642a3efa9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:55.202049 29511 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 71e2de235adf46d9a2407659080d42ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:55.212844 29735 tablet_service.cc:1511] Processing CreateTablet for tablet 66f652ba2c224885a2aea974bfe8f88a (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20260709 12:26:55.216452 29587 tablet_service.cc:1511] Processing CreateTablet for tablet 3615115233c34461b85763151751e475 (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20260709 12:26:55.217319 29735 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 66f652ba2c224885a2aea974bfe8f88a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:55.217705 29587 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3615115233c34461b85763151751e475. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:55.222303 29578 tablet_service.cc:1511] Processing CreateTablet for tablet 66f652ba2c224885a2aea974bfe8f88a (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20260709 12:26:55.223711 29578 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 66f652ba2c224885a2aea974bfe8f88a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:55.225622 29512 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9633e3b1d78f403097e4a00b7d4b869b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:55.207310 29661 tablet_service.cc:1511] Processing CreateTablet for tablet 00c6cf271c91400aa834467def9af9b1 (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20260709 12:26:55.206164 29655 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 76140ad6258543118a9bfeee7cf0d5fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:55.238113 29660 tablet_service.cc:1511] Processing CreateTablet for tablet 3615115233c34461b85763151751e475 (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20260709 12:26:55.253067 29505 tablet_service.cc:1511] Processing CreateTablet for tablet 3615115233c34461b85763151751e475 (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20260709 12:26:55.257406 29661 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00c6cf271c91400aa834467def9af9b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:55.242050 29792 tablet_bootstrap.cc:654] T fb6e78ccc01c4707af34cddf81619eab P 584dac86183a41d9b3bea1c2085e62fa: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:55.281867 29653 tablet_service.cc:1511] Processing CreateTablet for tablet 71e2de235adf46d9a2407659080d42ef (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20260709 12:26:55.284339 29653 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 71e2de235adf46d9a2407659080d42ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:55.289603 29660 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3615115233c34461b85763151751e475. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:55.292898 29696 heartbeater.cc:499] Master 127.16.29.126:41639 was elected leader, sending a full tablet report...
I20260709 12:26:55.292979 29580 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62f8c9472b214576b9002e02536a3ace. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:55.295863 29585 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00c6cf271c91400aa834467def9af9b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:55.298758 29790 tablet_bootstrap.cc:492] T 592801934cad4789b9ce8dc09f1470bd P c6610c00ffb948ce9c34f41dec3d49e8: No bootstrap required, opened a new log
I20260709 12:26:55.303931 29790 ts_tablet_manager.cc:1403] T 592801934cad4789b9ce8dc09f1470bd P c6610c00ffb948ce9c34f41dec3d49e8: Time spent bootstrapping tablet: real 0.409s	user 0.018s	sys 0.017s
I20260709 12:26:55.301056 29659 tablet_service.cc:1511] Processing CreateTablet for tablet a46289e8fc824dd5a818e30df6e7ec3e (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20260709 12:26:55.316574 29659 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a46289e8fc824dd5a818e30df6e7ec3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:55.302659 29793 tablet_bootstrap.cc:492] T 43c7bfb33af74d4f912e5153e8a54b3e P d73d92fb35234d16824c55689b5fce28: Bootstrap starting.
I20260709 12:26:55.303215 29505 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3615115233c34461b85763151751e475. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:55.282409 29504 tablet_service.cc:1511] Processing CreateTablet for tablet 3f41ce441fb047c8bbb741b1b9cbc0ec (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20260709 12:26:55.307091 29582 tablet_service.cc:1511] Processing CreateTablet for tablet 8017a44219404faa8db8ff049c65205e (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260709 12:26:55.322490 29504 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3f41ce441fb047c8bbb741b1b9cbc0ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:55.322986 29582 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8017a44219404faa8db8ff049c65205e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:55.300172 29508 tablet_service.cc:1511] Processing CreateTablet for tablet 66f652ba2c224885a2aea974bfe8f88a (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20260709 12:26:55.328169 29508 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 66f652ba2c224885a2aea974bfe8f88a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:55.361769 29794 tablet_bootstrap.cc:492] T 8627b58b4f42491597125eb45f8d3b61 P d13c58ad8ac441c4a9b396eb04b1a476: No bootstrap required, opened a new log
I20260709 12:26:55.362493 29794 ts_tablet_manager.cc:1403] T 8627b58b4f42491597125eb45f8d3b61 P d13c58ad8ac441c4a9b396eb04b1a476: Time spent bootstrapping tablet: real 0.345s	user 0.041s	sys 0.093s
I20260709 12:26:55.301538 29734 tablet_service.cc:1511] Processing CreateTablet for tablet 6a30685caefb4d0e80fa4ea87d70760c (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20260709 12:26:55.365170 29734 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a30685caefb4d0e80fa4ea87d70760c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:55.365061 29790 raft_consensus.cc:359] T 592801934cad4789b9ce8dc09f1470bd P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:26:55.365638 29794 raft_consensus.cc:359] T 8627b58b4f42491597125eb45f8d3b61 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:26:55.366061 29790 raft_consensus.cc:385] T 592801934cad4789b9ce8dc09f1470bd P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:55.366417 29794 raft_consensus.cc:385] T 8627b58b4f42491597125eb45f8d3b61 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:55.366638 29790 raft_consensus.cc:740] T 592801934cad4789b9ce8dc09f1470bd P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6610c00ffb948ce9c34f41dec3d49e8, State: Initialized, Role: FOLLOWER
I20260709 12:26:55.366842 29794 raft_consensus.cc:740] T 8627b58b4f42491597125eb45f8d3b61 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d13c58ad8ac441c4a9b396eb04b1a476, State: Initialized, Role: FOLLOWER
I20260709 12:26:55.371505 29794 consensus_queue.cc:260] T 8627b58b4f42491597125eb45f8d3b61 P d13c58ad8ac441c4a9b396eb04b1a476 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:26:55.377094 29792 tablet_bootstrap.cc:492] T fb6e78ccc01c4707af34cddf81619eab P 584dac86183a41d9b3bea1c2085e62fa: No bootstrap required, opened a new log
I20260709 12:26:55.377651 29792 ts_tablet_manager.cc:1403] T fb6e78ccc01c4707af34cddf81619eab P 584dac86183a41d9b3bea1c2085e62fa: Time spent bootstrapping tablet: real 0.375s	user 0.013s	sys 0.024s
W20260709 12:26:55.380064 29405 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.16.29.68:43895, user_credentials={real_user=slave}} blocked reactor thread for 61661.7us
I20260709 12:26:55.403676 29770 heartbeater.cc:499] Master 127.16.29.126:41639 was elected leader, sending a full tablet report...
I20260709 12:26:55.410637 29661 tablet_service.cc:1511] Processing CreateTablet for tablet 58154d49c1a344f2b51366ede3c87de5 (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20260709 12:26:55.412081 29661 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 58154d49c1a344f2b51366ede3c87de5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:55.427686 29509 tablet_service.cc:1511] Processing CreateTablet for tablet c6e621aa77d5474580dfc5949801ccc9 (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20260709 12:26:55.429248 29509 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c6e621aa77d5474580dfc5949801ccc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:55.440613 29579 tablet_service.cc:1511] Processing CreateTablet for tablet 58154d49c1a344f2b51366ede3c87de5 (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20260709 12:26:55.442163 29579 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 58154d49c1a344f2b51366ede3c87de5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:55.423671 29654 tablet_service.cc:1511] Processing CreateTablet for tablet c6e621aa77d5474580dfc5949801ccc9 (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20260709 12:26:55.453224 29654 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c6e621aa77d5474580dfc5949801ccc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:55.460912 29656 tablet_service.cc:1511] Processing CreateTablet for tablet 3f41ce441fb047c8bbb741b1b9cbc0ec (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20260709 12:26:55.462509 29656 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3f41ce441fb047c8bbb741b1b9cbc0ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:55.463912 29512 tablet_service.cc:1511] Processing CreateTablet for tablet 6a30685caefb4d0e80fa4ea87d70760c (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20260709 12:26:55.475608 29797 raft_consensus.cc:493] T 8627b58b4f42491597125eb45f8d3b61 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:55.478235 29586 tablet_service.cc:1511] Processing CreateTablet for tablet 07a2b94c0823467695508292a9ccdcc5 (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20260709 12:26:55.479619 29586 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07a2b94c0823467695508292a9ccdcc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:55.498843 29657 tablet_service.cc:1511] Processing CreateTablet for tablet 31a190e9f5ff407fab90502fb3197b70 (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20260709 12:26:55.500798 29726 tablet_service.cc:1511] Processing CreateTablet for tablet 31a190e9f5ff407fab90502fb3197b70 (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20260709 12:26:55.506743 29726 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 31a190e9f5ff407fab90502fb3197b70. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:55.512341 29790 consensus_queue.cc:260] T 592801934cad4789b9ce8dc09f1470bd P c6610c00ffb948ce9c34f41dec3d49e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:26:55.476552 29797 raft_consensus.cc:515] T 8627b58b4f42491597125eb45f8d3b61 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:26:55.524482 29512 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a30685caefb4d0e80fa4ea87d70760c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:55.526373 29797 leader_election.cc:290] T 8627b58b4f42491597125eb45f8d3b61 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895), 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033)
I20260709 12:26:55.539388 29730 tablet_service.cc:1511] Processing CreateTablet for tablet 07a2b94c0823467695508292a9ccdcc5 (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20260709 12:26:55.541353 29652 tablet_service.cc:1511] Processing CreateTablet for tablet 07a2b94c0823467695508292a9ccdcc5 (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20260709 12:26:55.553258 29507 tablet_service.cc:1511] Processing CreateTablet for tablet 1d1c1f58c4d340ac951dfa403d34138a (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20260709 12:26:55.575031 29507 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d1c1f58c4d340ac951dfa403d34138a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:55.556802 29580 tablet_service.cc:1511] Processing CreateTablet for tablet 6a30685caefb4d0e80fa4ea87d70760c (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20260709 12:26:55.580036 29730 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07a2b94c0823467695508292a9ccdcc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:55.580539 29580 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a30685caefb4d0e80fa4ea87d70760c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:55.581880 29511 tablet_service.cc:1511] Processing CreateTablet for tablet 425777eed34441cd93e0902bdc126516 (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20260709 12:26:55.592255 29733 tablet_service.cc:1511] Processing CreateTablet for tablet 425777eed34441cd93e0902bdc126516 (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20260709 12:26:55.592785 29511 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 425777eed34441cd93e0902bdc126516. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:55.593523 29733 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 425777eed34441cd93e0902bdc126516. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:55.584205 29578 tablet_service.cc:1511] Processing CreateTablet for tablet 16649fd17d434e3c95e9d585d2404659 (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20260709 12:26:55.550856 29510 tablet_service.cc:1511] Processing CreateTablet for tablet 16649fd17d434e3c95e9d585d2404659 (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20260709 12:26:55.549744 29503 tablet_service.cc:1511] Processing CreateTablet for tablet 31a190e9f5ff407fab90502fb3197b70 (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20260709 12:26:55.592891 29657 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 31a190e9f5ff407fab90502fb3197b70. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:55.577692 29581 tablet_service.cc:1511] Processing CreateTablet for tablet 1d1c1f58c4d340ac951dfa403d34138a (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20260709 12:26:55.599839 29793 tablet_bootstrap.cc:700] T 43c7bfb33af74d4f912e5153e8a54b3e P d73d92fb35234d16824c55689b5fce28: Time spent opening tablet: real 0.215s	user 0.004s	sys 0.001s
I20260709 12:26:55.607628 29510 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 16649fd17d434e3c95e9d585d2404659. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:55.638063 29503 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 31a190e9f5ff407fab90502fb3197b70. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:55.639664 29658 tablet_service.cc:1511] Processing CreateTablet for tablet 959940284eb24fdf83d78438d710318a (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20260709 12:26:55.593539 29727 tablet_service.cc:1511] Processing CreateTablet for tablet 1d1c1f58c4d340ac951dfa403d34138a (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20260709 12:26:55.613236 29792 raft_consensus.cc:359] T fb6e78ccc01c4707af34cddf81619eab P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:26:55.614004 29654 tablet_service.cc:1511] Processing CreateTablet for tablet d1918297b9eb41f5847a211e668ae748 (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20260709 12:26:55.616783 29585 tablet_service.cc:1511] Processing CreateTablet for tablet c6e621aa77d5474580dfc5949801ccc9 (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20260709 12:26:55.620929 29506 tablet_service.cc:1511] Processing CreateTablet for tablet d1918297b9eb41f5847a211e668ae748 (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20260709 12:26:55.656912 29506 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d1918297b9eb41f5847a211e668ae748. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:55.650600 29792 raft_consensus.cc:385] T fb6e78ccc01c4707af34cddf81619eab P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:55.659860 29792 raft_consensus.cc:740] T fb6e78ccc01c4707af34cddf81619eab P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 584dac86183a41d9b3bea1c2085e62fa, State: Initialized, Role: FOLLOWER
I20260709 12:26:55.661131 29792 consensus_queue.cc:260] T fb6e78ccc01c4707af34cddf81619eab P 584dac86183a41d9b3bea1c2085e62fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:26:55.621130 29578 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 16649fd17d434e3c95e9d585d2404659. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:55.664863 29732 tablet_service.cc:1511] Processing CreateTablet for tablet c1c8411289a74b5685700229d245ff75 (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20260709 12:26:55.650704 29655 tablet_service.cc:1511] Processing CreateTablet for tablet 16649fd17d434e3c95e9d585d2404659 (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20260709 12:26:55.670383 29504 tablet_service.cc:1511] Processing CreateTablet for tablet e67a2affef634fd8bc96a64fa83cffdb (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20260709 12:26:55.624145 29735 tablet_service.cc:1511] Processing CreateTablet for tablet 959940284eb24fdf83d78438d710318a (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20260709 12:26:55.629145 29794 ts_tablet_manager.cc:1434] T 8627b58b4f42491597125eb45f8d3b61 P d13c58ad8ac441c4a9b396eb04b1a476: Time spent starting tablet: real 0.266s	user 0.039s	sys 0.042s
I20260709 12:26:55.631497 29653 tablet_service.cc:1511] Processing CreateTablet for tablet 425777eed34441cd93e0902bdc126516 (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20260709 12:26:55.591476 29731 tablet_service.cc:1511] Processing CreateTablet for tablet 5be58d64d45e40d7981018b84a01b1ed (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20260709 12:26:55.645715 29727 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d1c1f58c4d340ac951dfa403d34138a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:55.648128 29652 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07a2b94c0823467695508292a9ccdcc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:55.677989 29508 tablet_service.cc:1511] Processing CreateTablet for tablet cbf03426b6df43e5b2b9c56e2eb7687c (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20260709 12:26:55.678844 29793 tablet_bootstrap.cc:654] T 43c7bfb33af74d4f912e5153e8a54b3e P d73d92fb35234d16824c55689b5fce28: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:55.679904 29504 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e67a2affef634fd8bc96a64fa83cffdb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:55.686147 29654 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d1918297b9eb41f5847a211e668ae748. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:55.686820 29794 tablet_bootstrap.cc:492] T d42259e436d241b29fb4ded16d9b6455 P d13c58ad8ac441c4a9b396eb04b1a476: Bootstrap starting.
I20260709 12:26:55.690621 29734 tablet_service.cc:1511] Processing CreateTablet for tablet 1d4e0ad0462e4f74a44155aa3ffaf87f (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20260709 12:26:55.692077 29790 ts_tablet_manager.cc:1434] T 592801934cad4789b9ce8dc09f1470bd P c6610c00ffb948ce9c34f41dec3d49e8: Time spent starting tablet: real 0.379s	user 0.008s	sys 0.022s
I20260709 12:26:55.696477 29655 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 16649fd17d434e3c95e9d585d2404659. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:55.701131 29583 tablet_service.cc:1511] Processing CreateTablet for tablet 5be58d64d45e40d7981018b84a01b1ed (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20260709 12:26:55.702557 29583 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5be58d64d45e40d7981018b84a01b1ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:55.702785 29653 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 425777eed34441cd93e0902bdc126516. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:55.704104 29734 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d4e0ad0462e4f74a44155aa3ffaf87f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:55.705796 29656 tablet_service.cc:1511] Processing CreateTablet for tablet c1c8411289a74b5685700229d245ff75 (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20260709 12:26:55.708720 29656 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c1c8411289a74b5685700229d245ff75. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:55.712344 29658 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 959940284eb24fdf83d78438d710318a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:55.714146 29584 tablet_service.cc:1511] Processing CreateTablet for tablet c1c8411289a74b5685700229d245ff75 (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20260709 12:26:55.715504 29584 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c1c8411289a74b5685700229d245ff75. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:55.716303 29579 tablet_service.cc:1511] Processing CreateTablet for tablet d1918297b9eb41f5847a211e668ae748 (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20260709 12:26:55.717574 29579 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d1918297b9eb41f5847a211e668ae748. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:55.717209 29661 tablet_service.cc:1511] Processing CreateTablet for tablet cbf03426b6df43e5b2b9c56e2eb7687c (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20260709 12:26:55.718611 29661 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cbf03426b6df43e5b2b9c56e2eb7687c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:55.720963 29728 tablet_service.cc:1511] Processing CreateTablet for tablet 6e37d72807e547898f4fce9e8b82b2b0 (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20260709 12:26:55.649386 29660 tablet_service.cc:1511] Processing CreateTablet for tablet d299fb38094e47e8aa3890ca74f6b366 (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20260709 12:26:55.650259 29659 tablet_service.cc:1511] Processing CreateTablet for tablet e67a2affef634fd8bc96a64fa83cffdb (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20260709 12:26:55.724718 29659 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e67a2affef634fd8bc96a64fa83cffdb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:55.725159 29790 tablet_bootstrap.cc:492] T 55061e62184b45f0bb1726adc1a4af81 P c6610c00ffb948ce9c34f41dec3d49e8: Bootstrap starting.
I20260709 12:26:55.725831 29510 tablet_service.cc:1511] Processing CreateTablet for tablet 1d4e0ad0462e4f74a44155aa3ffaf87f (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20260709 12:26:55.654027 29729 tablet_service.cc:1511] Processing CreateTablet for tablet e67a2affef634fd8bc96a64fa83cffdb (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20260709 12:26:55.671562 29587 tablet_service.cc:1511] Processing CreateTablet for tablet 959940284eb24fdf83d78438d710318a (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20260709 12:26:55.726287 29581 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d1c1f58c4d340ac951dfa403d34138a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:55.733850 29509 tablet_service.cc:1511] Processing CreateTablet for tablet 01788883c8ea42989f246a3486111a7e (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20260709 12:26:55.735311 29509 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 01788883c8ea42989f246a3486111a7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:55.653846 29505 tablet_service.cc:1511] Processing CreateTablet for tablet 5be58d64d45e40d7981018b84a01b1ed (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20260709 12:26:55.736510 29587 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 959940284eb24fdf83d78438d710318a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:55.737877 29585 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c6e621aa77d5474580dfc5949801ccc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:55.736632 29505 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5be58d64d45e40d7981018b84a01b1ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:55.744871 29508 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cbf03426b6df43e5b2b9c56e2eb7687c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:55.746071 29510 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d4e0ad0462e4f74a44155aa3ffaf87f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:55.756101 29507 tablet_service.cc:1511] Processing CreateTablet for tablet 6e37d72807e547898f4fce9e8b82b2b0 (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20260709 12:26:55.757746 29507 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e37d72807e547898f4fce9e8b82b2b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:55.758057 29726 tablet_service.cc:1511] Processing CreateTablet for tablet d299fb38094e47e8aa3890ca74f6b366 (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20260709 12:26:55.759488 29728 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e37d72807e547898f4fce9e8b82b2b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:55.760232 29729 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e67a2affef634fd8bc96a64fa83cffdb. 1 dirs total, 0 dirs full, 0 dirs failed
W20260709 12:26:55.772157 29697 tablet.cc:2406] T 8627b58b4f42491597125eb45f8d3b61 P d13c58ad8ac441c4a9b396eb04b1a476: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260709 12:26:55.772231 29731 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5be58d64d45e40d7981018b84a01b1ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:55.776775 29660 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d299fb38094e47e8aa3890ca74f6b366. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:55.776775 29735 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 959940284eb24fdf83d78438d710318a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:55.779796 29732 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c1c8411289a74b5685700229d245ff75. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:55.781054 29580 tablet_service.cc:1511] Processing CreateTablet for tablet cbf03426b6df43e5b2b9c56e2eb7687c (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20260709 12:26:55.782392 29580 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cbf03426b6df43e5b2b9c56e2eb7687c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:55.806200 29726 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d299fb38094e47e8aa3890ca74f6b366. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:55.811098 29522 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8627b58b4f42491597125eb45f8d3b61" candidate_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "584dac86183a41d9b3bea1c2085e62fa" is_pre_election: true
W20260709 12:26:55.813154 29634 leader_election.cc:343] T 8627b58b4f42491597125eb45f8d3b61 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:55.819416 29790 tablet_bootstrap.cc:654] T 55061e62184b45f0bb1726adc1a4af81 P c6610c00ffb948ce9c34f41dec3d49e8: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:55.829253 29654 tablet_service.cc:1511] Processing CreateTablet for tablet 6e37d72807e547898f4fce9e8b82b2b0 (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20260709 12:26:55.830891 29654 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e37d72807e547898f4fce9e8b82b2b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:55.788030 29512 tablet_service.cc:1511] Processing CreateTablet for tablet 5ee924f859f54f64908499bfac48ee62 (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20260709 12:26:55.843534 29794 tablet_bootstrap.cc:654] T d42259e436d241b29fb4ded16d9b6455 P d13c58ad8ac441c4a9b396eb04b1a476: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:55.848402 29792 ts_tablet_manager.cc:1434] T fb6e78ccc01c4707af34cddf81619eab P 584dac86183a41d9b3bea1c2085e62fa: Time spent starting tablet: real 0.470s	user 0.006s	sys 0.008s
I20260709 12:26:55.850209 29745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8627b58b4f42491597125eb45f8d3b61" candidate_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" is_pre_election: true
W20260709 12:26:55.852126 29635 leader_election.cc:343] T 8627b58b4f42491597125eb45f8d3b61 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:55.852594 29635 leader_election.cc:304] T 8627b58b4f42491597125eb45f8d3b61 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d13c58ad8ac441c4a9b396eb04b1a476; no voters: 584dac86183a41d9b3bea1c2085e62fa, c6610c00ffb948ce9c34f41dec3d49e8
I20260709 12:26:55.853883 29798 raft_consensus.cc:2749] T 8627b58b4f42491597125eb45f8d3b61 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:26:55.857383 29511 tablet_service.cc:1511] Processing CreateTablet for tablet fb7d78e534634e0f992fc39b578ebd8d (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20260709 12:26:55.857654 29506 tablet_service.cc:1511] Processing CreateTablet for tablet 6c8b2fa9f2c044ec9eec97c72b561042 (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20260709 12:26:55.862670 29512 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ee924f859f54f64908499bfac48ee62. 1 dirs total, 0 dirs full, 0 dirs failed
W20260709 12:26:55.865474 29467 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:26:55.835350 29582 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:50710 (request call id 16) took 1066 ms. Trace:
I20260709 12:26:55.866564 29652 tablet_service.cc:1511] Processing CreateTablet for tablet 5ee924f859f54f64908499bfac48ee62 (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20260709 12:26:55.886761 29582 rpcz_store.cc:276] 0709 12:26:54.767989 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:55.164100 (+396111us) service_pool.cc:224] Handling call
0709 12:26:55.322553 (+158453us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 12:26:55.322645 (+    92us) ts_tablet_manager.cc:682] Creating new metadata...
0709 12:26:55.490881 (+168236us) tablet_metadata.cc:665] Metadata flushed
0709 12:26:55.694868 (+203987us) tablet_replica.cc:206] Creating consensus instance
0709 12:26:55.833908 (+139040us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:26:55.888659 29652 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ee924f859f54f64908499bfac48ee62. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:55.890237 29582 tablet_service.cc:1511] Processing CreateTablet for tablet 5ee924f859f54f64908499bfac48ee62 (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20260709 12:26:55.891616 29582 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ee924f859f54f64908499bfac48ee62. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:55.906685 29579 tablet_service.cc:1511] Processing CreateTablet for tablet 3617af2dee3f48cf85973b1c94cdc33a (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20260709 12:26:55.908393 29579 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3617af2dee3f48cf85973b1c94cdc33a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:55.910286 29585 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:50710 (request call id 21) took 1025 ms. Trace:
I20260709 12:26:55.910804 29585 rpcz_store.cc:276] 0709 12:26:54.884621 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:55.474266 (+589645us) service_pool.cc:224] Handling call
0709 12:26:55.655832 (+181566us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 12:26:55.655981 (+   149us) ts_tablet_manager.cc:682] Creating new metadata...
0709 12:26:55.824335 (+168354us) tablet_metadata.cc:665] Metadata flushed
0709 12:26:55.909528 (+ 85193us) tablet_replica.cc:206] Creating consensus instance
0709 12:26:55.910218 (+   690us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:26:55.928105 29792 tablet_bootstrap.cc:492] T d21ab1641501475b829863275dc8182a P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap starting.
I20260709 12:26:55.885179 29587 tablet_service.cc:1511] Processing CreateTablet for tablet d299fb38094e47e8aa3890ca74f6b366 (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20260709 12:26:55.870993 29586 tablet_service.cc:1511] Processing CreateTablet for tablet 1d4e0ad0462e4f74a44155aa3ffaf87f (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20260709 12:26:55.891813 29729 tablet_service.cc:1511] Processing CreateTablet for tablet 6c8b2fa9f2c044ec9eec97c72b561042 (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20260709 12:26:55.929344 29585 tablet_service.cc:1511] Processing CreateTablet for tablet f3e17f0a8fc14095850e8138fd79b86e (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20260709 12:26:55.929610 29586 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d4e0ad0462e4f74a44155aa3ffaf87f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:55.896119 29511 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb7d78e534634e0f992fc39b578ebd8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:55.930662 29585 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3e17f0a8fc14095850e8138fd79b86e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:55.892524 29578 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:50710 (request call id 24) took 1002 ms. Trace:
I20260709 12:26:55.931119 29578 rpcz_store.cc:276] 0709 12:26:54.889663 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:55.583216 (+693553us) service_pool.cc:224] Handling call
0709 12:26:55.603690 (+ 20474us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 12:26:55.620228 (+ 16538us) ts_tablet_manager.cc:682] Creating new metadata...
0709 12:26:55.816174 (+195946us) tablet_metadata.cc:665] Metadata flushed
0709 12:26:55.891944 (+ 75770us) tablet_replica.cc:206] Creating consensus instance
0709 12:26:55.892460 (+   516us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:26:55.901933 29733 tablet_service.cc:1511] Processing CreateTablet for tablet 01788883c8ea42989f246a3486111a7e (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20260709 12:26:55.902750 29584 tablet_service.cc:1511] Processing CreateTablet for tablet 6c8b2fa9f2c044ec9eec97c72b561042 (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20260709 12:26:55.877975 29657 tablet_service.cc:1511] Processing CreateTablet for tablet fb7d78e534634e0f992fc39b578ebd8d (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20260709 12:26:55.939546 29728 tablet_service.cc:1511] Processing CreateTablet for tablet 394378d078194257b9e04fc659c1f5d4 (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20260709 12:26:55.918617 29793 tablet_bootstrap.cc:492] T 43c7bfb33af74d4f912e5153e8a54b3e P d73d92fb35234d16824c55689b5fce28: No bootstrap required, opened a new log
I20260709 12:26:55.919234 29509 tablet_service.cc:1511] Processing CreateTablet for tablet abec627108a84083884e64e1eb9dcda6 (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20260709 12:26:55.951437 29793 ts_tablet_manager.cc:1403] T 43c7bfb33af74d4f912e5153e8a54b3e P d73d92fb35234d16824c55689b5fce28: Time spent bootstrapping tablet: real 0.787s	user 0.006s	sys 0.017s
I20260709 12:26:55.920303 29503 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:48124 (request call id 23) took 1033 ms. Trace:
I20260709 12:26:55.952287 29503 rpcz_store.cc:276] 0709 12:26:54.887225 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:55.540646 (+653421us) service_pool.cc:224] Handling call
0709 12:26:55.606122 (+ 65476us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 12:26:55.607693 (+  1571us) ts_tablet_manager.cc:682] Creating new metadata...
0709 12:26:55.718704 (+111011us) tablet_metadata.cc:665] Metadata flushed
0709 12:26:55.872675 (+153971us) tablet_replica.cc:206] Creating consensus instance
0709 12:26:55.920241 (+ 47566us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":53303}
I20260709 12:26:55.954577 29503 tablet_service.cc:1511] Processing CreateTablet for tablet d5fcd1f030614bbe82cf1dc261dfc1c3 (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20260709 12:26:55.917553 29510 tablet_service.cc:1511] Processing CreateTablet for tablet ef20d7124e7a44aa97a219ee194f4b8b (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20260709 12:26:55.929709 29729 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c8b2fa9f2c044ec9eec97c72b561042. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:55.914757 29734 tablet_service.cc:1511] Processing CreateTablet for tablet ef20d7124e7a44aa97a219ee194f4b8b (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20260709 12:26:55.897809 29732 tablet_service.cc:1511] Processing CreateTablet for tablet 3617af2dee3f48cf85973b1c94cdc33a (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20260709 12:26:55.936295 29506 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c8b2fa9f2c044ec9eec97c72b561042. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:55.896335 29731 tablet_service.cc:1511] Processing CreateTablet for tablet fb7d78e534634e0f992fc39b578ebd8d (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20260709 12:26:55.940054 29657 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb7d78e534634e0f992fc39b578ebd8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:55.941900 29578 tablet_service.cc:1511] Processing CreateTablet for tablet 01788883c8ea42989f246a3486111a7e (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20260709 12:26:55.942471 29730 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33132 (request call id 23) took 1046 ms. Trace:
I20260709 12:26:55.944088 29579 tablet_service.cc:1511] Processing CreateTablet for tablet 394378d078194257b9e04fc659c1f5d4 (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20260709 12:26:55.944619 29505 tablet_service.cc:1511] Processing CreateTablet for tablet f76101eb30f14b98a9aa8c41e0d6f831 (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20260709 12:26:55.946265 29581 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:50710 (request call id 25) took 1055 ms. Trace:
I20260709 12:26:55.941857 29507 tablet_service.cc:1511] Processing CreateTablet for tablet f8c01e05981747a395e5483b551297dd (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20260709 12:26:55.987843 29581 rpcz_store.cc:276] 0709 12:26:54.891265 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:55.576026 (+684761us) service_pool.cc:224] Handling call
0709 12:26:55.621972 (+ 45946us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 12:26:55.622202 (+   230us) ts_tablet_manager.cc:682] Creating new metadata...
0709 12:26:55.743352 (+121150us) tablet_metadata.cc:665] Metadata flushed
0709 12:26:55.945703 (+202351us) tablet_replica.cc:206] Creating consensus instance
0709 12:26:55.946204 (+   501us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:26:55.917502 29504 tablet_service.cc:1511] Processing CreateTablet for tablet f3e17f0a8fc14095850e8138fd79b86e (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20260709 12:26:55.990219 29581 tablet_service.cc:1511] Processing CreateTablet for tablet f8c01e05981747a395e5483b551297dd (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20260709 12:26:55.955446 29793 raft_consensus.cc:359] T 43c7bfb33af74d4f912e5153e8a54b3e P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:26:55.992787 29793 raft_consensus.cc:385] T 43c7bfb33af74d4f912e5153e8a54b3e P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:55.993155 29793 raft_consensus.cc:740] T 43c7bfb33af74d4f912e5153e8a54b3e P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d73d92fb35234d16824c55689b5fce28, State: Initialized, Role: FOLLOWER
I20260709 12:26:55.966615 29799 raft_consensus.cc:493] T 592801934cad4789b9ce8dc09f1470bd P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:55.994102 29799 raft_consensus.cc:515] T 592801934cad4789b9ce8dc09f1470bd P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:26:55.994588 29659 tablet_service.cc:1511] Processing CreateTablet for tablet abec627108a84083884e64e1eb9dcda6 (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20260709 12:26:55.966909 29509 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet abec627108a84083884e64e1eb9dcda6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:55.996102 29659 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet abec627108a84083884e64e1eb9dcda6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:55.996541 29504 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3e17f0a8fc14095850e8138fd79b86e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:55.966687 29584 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c8b2fa9f2c044ec9eec97c72b561042. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:55.967242 29733 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 01788883c8ea42989f246a3486111a7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:55.998813 29587 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d299fb38094e47e8aa3890ca74f6b366. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:55.999477 29505 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f76101eb30f14b98a9aa8c41e0d6f831. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:55.970831 29790 tablet_bootstrap.cc:492] T 55061e62184b45f0bb1726adc1a4af81 P c6610c00ffb948ce9c34f41dec3d49e8: No bootstrap required, opened a new log
I20260709 12:26:56.001081 29790 ts_tablet_manager.cc:1403] T 55061e62184b45f0bb1726adc1a4af81 P c6610c00ffb948ce9c34f41dec3d49e8: Time spent bootstrapping tablet: real 0.306s	user 0.014s	sys 0.008s
I20260709 12:26:56.000154 29793 consensus_queue.cc:260] T 43c7bfb33af74d4f912e5153e8a54b3e P d73d92fb35234d16824c55689b5fce28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:26:56.002151 29512 tablet_service.cc:1511] Processing CreateTablet for tablet 0c428ff656ae4b9d99d5b5073f98725d (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20260709 12:26:55.956454 29658 tablet_service.cc:1511] Processing CreateTablet for tablet 3617af2dee3f48cf85973b1c94cdc33a (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20260709 12:26:55.973114 29508 tablet_service.cc:1511] Processing CreateTablet for tablet a3dcbb9c64bf4b5bb6680e8cb81af93e (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20260709 12:26:55.927448 29655 tablet_service.cc:1511] Processing CreateTablet for tablet f3e17f0a8fc14095850e8138fd79b86e (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20260709 12:26:55.977301 29586 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:50710 (request call id 31) took 1008 ms. Trace:
I20260709 12:26:55.982936 29656 tablet_service.cc:1511] Processing CreateTablet for tablet 394378d078194257b9e04fc659c1f5d4 (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20260709 12:26:55.991394 29794 tablet_bootstrap.cc:492] T d42259e436d241b29fb4ded16d9b6455 P d13c58ad8ac441c4a9b396eb04b1a476: No bootstrap required, opened a new log
I20260709 12:26:56.011150 29507 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f8c01e05981747a395e5483b551297dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:56.011837 29510 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef20d7124e7a44aa97a219ee194f4b8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:56.011438 29794 ts_tablet_manager.cc:1403] T d42259e436d241b29fb4ded16d9b6455 P d13c58ad8ac441c4a9b396eb04b1a476: Time spent bootstrapping tablet: real 0.325s	user 0.016s	sys 0.001s
I20260709 12:26:55.892941 29653 tablet_service.cc:1511] Processing CreateTablet for tablet ef20d7124e7a44aa97a219ee194f4b8b (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20260709 12:26:56.005273 29799 leader_election.cc:290] T 592801934cad4789b9ce8dc09f1470bd P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033), d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825)
I20260709 12:26:56.005683 29790 raft_consensus.cc:359] T 55061e62184b45f0bb1726adc1a4af81 P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:26:56.016316 29790 raft_consensus.cc:385] T 55061e62184b45f0bb1726adc1a4af81 P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:56.016664 29790 raft_consensus.cc:740] T 55061e62184b45f0bb1726adc1a4af81 P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6610c00ffb948ce9c34f41dec3d49e8, State: Initialized, Role: FOLLOWER
I20260709 12:26:56.016780 29653 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef20d7124e7a44aa97a219ee194f4b8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:56.017701 29790 consensus_queue.cc:260] T 55061e62184b45f0bb1726adc1a4af81 P c6610c00ffb948ce9c34f41dec3d49e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:26:56.023766 29790 ts_tablet_manager.cc:1434] T 55061e62184b45f0bb1726adc1a4af81 P c6610c00ffb948ce9c34f41dec3d49e8: Time spent starting tablet: real 0.022s	user 0.003s	sys 0.005s
I20260709 12:26:56.020462 29794 raft_consensus.cc:359] T d42259e436d241b29fb4ded16d9b6455 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:26:56.025395 29794 raft_consensus.cc:385] T d42259e436d241b29fb4ded16d9b6455 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:56.025971 29794 raft_consensus.cc:740] T d42259e436d241b29fb4ded16d9b6455 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d13c58ad8ac441c4a9b396eb04b1a476, State: Initialized, Role: FOLLOWER
I20260709 12:26:56.026106 29503 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d5fcd1f030614bbe82cf1dc261dfc1c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:56.025050 29656 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 394378d078194257b9e04fc659c1f5d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:56.030224 29581 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f8c01e05981747a395e5483b551297dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:56.036227 29578 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 01788883c8ea42989f246a3486111a7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:56.036751 29727 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33132 (request call id 24) took 1119 ms. Trace:
I20260709 12:26:56.037524 29727 rpcz_store.cc:276] 0709 12:26:54.917298 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:55.562827 (+645529us) service_pool.cc:224] Handling call
0709 12:26:55.645082 (+ 82255us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 12:26:55.645207 (+   125us) ts_tablet_manager.cc:682] Creating new metadata...
0709 12:26:55.728934 (+ 83727us) tablet_metadata.cc:665] Metadata flushed
0709 12:26:55.961953 (+233019us) tablet_replica.cc:206] Creating consensus instance
0709 12:26:56.036682 (+ 74729us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:26:56.044739 29790 tablet_bootstrap.cc:492] T 3e9451c3797c4496a66dfdb3e9fb5e38 P c6610c00ffb948ce9c34f41dec3d49e8: Bootstrap starting.
I20260709 12:26:56.044817 29661 tablet_service.cc:1511] Processing CreateTablet for tablet f76101eb30f14b98a9aa8c41e0d6f831 (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20260709 12:26:55.988281 29580 tablet_service.cc:1511] Processing CreateTablet for tablet abec627108a84083884e64e1eb9dcda6 (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20260709 12:26:56.047422 29580 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet abec627108a84083884e64e1eb9dcda6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:56.045347 29794 consensus_queue.cc:260] T d42259e436d241b29fb4ded16d9b6455 P d13c58ad8ac441c4a9b396eb04b1a476 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:26:56.052135 29794 ts_tablet_manager.cc:1434] T d42259e436d241b29fb4ded16d9b6455 P d13c58ad8ac441c4a9b396eb04b1a476: Time spent starting tablet: real 0.039s	user 0.004s	sys 0.004s
I20260709 12:26:56.053174 29794 tablet_bootstrap.cc:492] T 3e9451c3797c4496a66dfdb3e9fb5e38 P d13c58ad8ac441c4a9b396eb04b1a476: Bootstrap starting.
I20260709 12:26:55.965096 29730 rpcz_store.cc:276] 0709 12:26:54.895714 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:55.538320 (+642606us) service_pool.cc:224] Handling call
0709 12:26:55.540301 (+  1981us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 12:26:55.540446 (+   145us) ts_tablet_manager.cc:682] Creating new metadata...
0709 12:26:55.733409 (+192963us) tablet_metadata.cc:665] Metadata flushed
0709 12:26:55.941961 (+208552us) tablet_replica.cc:206] Creating consensus instance
0709 12:26:55.942411 (+   450us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:26:56.033138 29728 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 394378d078194257b9e04fc659c1f5d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:56.062187 29730 tablet_service.cc:1511] Processing CreateTablet for tablet f76101eb30f14b98a9aa8c41e0d6f831 (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20260709 12:26:56.027508 29583 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:50710 (request call id 28) took 1050 ms. Trace:
I20260709 12:26:56.063580 29583 rpcz_store.cc:276] 0709 12:26:54.960521 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:55.700078 (+739557us) service_pool.cc:224] Handling call
0709 12:26:55.702048 (+  1970us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 12:26:55.702157 (+   109us) ts_tablet_manager.cc:682] Creating new metadata...
0709 12:26:55.965707 (+263550us) tablet_metadata.cc:665] Metadata flushed
0709 12:26:55.994548 (+ 28841us) tablet_replica.cc:206] Creating consensus instance
0709 12:26:56.010875 (+ 16327us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:26:56.067307 29583 tablet_service.cc:1511] Processing CreateTablet for tablet 137e03144e754f6e9da4f0544513d27c (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20260709 12:26:56.060295 29579 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 394378d078194257b9e04fc659c1f5d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:56.010648 29586 rpcz_store.cc:276] 0709 12:26:54.968873 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:55.870004 (+901131us) service_pool.cc:224] Handling call
0709 12:26:55.929003 (+ 58999us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 12:26:55.929125 (+   122us) ts_tablet_manager.cc:682] Creating new metadata...
0709 12:26:55.965880 (+ 36755us) tablet_metadata.cc:665] Metadata flushed
0709 12:26:55.976714 (+ 10834us) tablet_replica.cc:206] Creating consensus instance
0709 12:26:55.977235 (+   521us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:26:56.067924 29730 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f76101eb30f14b98a9aa8c41e0d6f831. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:56.073302 29508 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3dcbb9c64bf4b5bb6680e8cb81af93e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:56.077171 29734 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef20d7124e7a44aa97a219ee194f4b8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:56.080272 29731 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb7d78e534634e0f992fc39b578ebd8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:56.081115 29511 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:48124 (request call id 35) took 1087 ms. Trace:
I20260709 12:26:56.060050 29727 tablet_service.cc:1511] Processing CreateTablet for tablet f8c01e05981747a395e5483b551297dd (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20260709 12:26:56.048727 29735 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33132 (request call id 26) took 1130 ms. Trace:
I20260709 12:26:56.081244 29726 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33132 (request call id 32) took 1115 ms. Trace:
I20260709 12:26:56.082609 29804 raft_consensus.cc:493] T fb6e78ccc01c4707af34cddf81619eab P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:56.081687 29511 rpcz_store.cc:276] 0709 12:26:54.993504 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:55.856382 (+862878us) service_pool.cc:224] Handling call
0709 12:26:55.858334 (+  1952us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 12:26:55.858432 (+    98us) ts_tablet_manager.cc:682] Creating new metadata...
0709 12:26:55.969412 (+110980us) tablet_metadata.cc:665] Metadata flushed
0709 12:26:56.029368 (+ 59956us) tablet_replica.cc:206] Creating consensus instance
0709 12:26:56.081051 (+ 51683us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:26:56.084784 29582 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:50710 (request call id 33) took 1093 ms. Trace:
I20260709 12:26:56.095410 29661 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f76101eb30f14b98a9aa8c41e0d6f831. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:56.092263 29804 raft_consensus.cc:515] T fb6e78ccc01c4707af34cddf81619eab P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:26:56.097982 29804 leader_election.cc:290] T fb6e78ccc01c4707af34cddf81619eab P 584dac86183a41d9b3bea1c2085e62fa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895), d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825)
I20260709 12:26:56.098949 29512 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c428ff656ae4b9d99d5b5073f98725d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:56.099653 29732 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3617af2dee3f48cf85973b1c94cdc33a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:56.089838 29726 rpcz_store.cc:276] 0709 12:26:54.965979 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:55.757087 (+791108us) service_pool.cc:224] Handling call
0709 12:26:55.758980 (+  1893us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 12:26:55.759082 (+   102us) ts_tablet_manager.cc:682] Creating new metadata...
0709 12:26:55.895772 (+136690us) tablet_metadata.cc:665] Metadata flushed
0709 12:26:56.080751 (+184979us) tablet_replica.cc:206] Creating consensus instance
0709 12:26:56.081187 (+   436us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:26:56.121263 29583 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 137e03144e754f6e9da4f0544513d27c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:56.089064 29735 rpcz_store.cc:276] 0709 12:26:54.918521 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:55.579258 (+660737us) service_pool.cc:224] Handling call
0709 12:26:55.671713 (+ 92455us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 12:26:55.671808 (+    95us) ts_tablet_manager.cc:682] Creating new metadata...
0709 12:26:55.858224 (+186416us) tablet_metadata.cc:665] Metadata flushed
0709 12:26:55.961160 (+102936us) tablet_replica.cc:206] Creating consensus instance
0709 12:26:56.035062 (+ 73902us) spinlock_profiling.cc:243] Waited 1.17 ms on lock 0x7b58007e7598. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b474fbb71 00007f7b474fc30e 00007f7b5043d8c9 00007f7b4f56ee51 00007f7b4f635b1b 00007f7b4f63170f 00007f7b4f631ecd 00007f7b4f5e0400 00007f7b48a65af4 00007f7b48a65a82
0709 12:26:56.048567 (+ 13505us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2700160}
I20260709 12:26:56.092540 29582 rpcz_store.cc:276] 0709 12:26:54.991621 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:55.889262 (+897641us) service_pool.cc:224] Handling call
0709 12:26:55.891142 (+  1880us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 12:26:55.891241 (+    99us) ts_tablet_manager.cc:682] Creating new metadata...
0709 12:26:55.973267 (+ 82026us) tablet_metadata.cc:665] Metadata flushed
0709 12:26:56.084276 (+111009us) tablet_replica.cc:206] Creating consensus instance
0709 12:26:56.084726 (+   450us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:26:56.121029 29655 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3e17f0a8fc14095850e8138fd79b86e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:56.161624 29582 tablet_service.cc:1511] Processing CreateTablet for tablet a3dcbb9c64bf4b5bb6680e8cb81af93e (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20260709 12:26:56.162638 29658 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3617af2dee3f48cf85973b1c94cdc33a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:56.163486 29582 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3dcbb9c64bf4b5bb6680e8cb81af93e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:56.164444 29793 ts_tablet_manager.cc:1434] T 43c7bfb33af74d4f912e5153e8a54b3e P d73d92fb35234d16824c55689b5fce28: Time spent starting tablet: real 0.212s	user 0.011s	sys 0.006s
I20260709 12:26:56.161631 29735 tablet_service.cc:1511] Processing CreateTablet for tablet a3dcbb9c64bf4b5bb6680e8cb81af93e (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20260709 12:26:56.073356 29586 tablet_service.cc:1511] Processing CreateTablet for tablet d5fcd1f030614bbe82cf1dc261dfc1c3 (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20260709 12:26:56.126579 29660 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51980 (request call id 32) took 1051 ms. Trace:
I20260709 12:26:56.174829 29660 rpcz_store.cc:276] 0709 12:26:55.075034 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:55.648356 (+573322us) service_pool.cc:224] Handling call
0709 12:26:55.776104 (+127748us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 12:26:55.776229 (+   125us) ts_tablet_manager.cc:682] Creating new metadata...
0709 12:26:55.961758 (+185529us) tablet_metadata.cc:665] Metadata flushed
0709 12:26:56.125954 (+164196us) tablet_replica.cc:206] Creating consensus instance
0709 12:26:56.126506 (+   552us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:26:56.177759 29660 tablet_service.cc:1511] Processing CreateTablet for tablet d5fcd1f030614bbe82cf1dc261dfc1c3 (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20260709 12:26:56.178764 29731 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33132 (request call id 34) took 1212 ms. Trace:
I20260709 12:26:56.179078 29522 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592801934cad4789b9ce8dc09f1470bd" candidate_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "584dac86183a41d9b3bea1c2085e62fa" is_pre_election: true
I20260709 12:26:56.193589 29731 rpcz_store.cc:276] 0709 12:26:54.966604 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:55.895400 (+928796us) service_pool.cc:224] Handling call
0709 12:26:55.985398 (+ 89998us) spinlock_profiling.cc:243] Waited 17.4 ms on lock 0x7b58007e7598. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b474fbe9c 00007f7b474fba5b 00007f7b504148d9 00007f7b4f56eee0 00007f7b4f654831 00007f7b4f631ac3 00007f7b4f5e0400 00007f7b48a65af4 00007f7b48a65a82 00007f7b48a659b1
0709 12:26:56.035316 (+ 49918us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 12:26:56.035415 (+    99us) ts_tablet_manager.cc:682] Creating new metadata...
0709 12:26:56.132682 (+ 97267us) tablet_metadata.cc:665] Metadata flushed
0709 12:26:56.177764 (+ 45082us) tablet_replica.cc:206] Creating consensus instance
0709 12:26:56.178707 (+   943us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":41790592}
I20260709 12:26:56.197362 29792 tablet_bootstrap.cc:654] T d21ab1641501475b829863275dc8182a P 584dac86183a41d9b3bea1c2085e62fa: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:56.198189 29731 tablet_service.cc:1511] Processing CreateTablet for tablet 0c428ff656ae4b9d99d5b5073f98725d (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
W20260709 12:26:56.198567 29708 leader_election.cc:343] T 592801934cad4789b9ce8dc09f1470bd P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:56.197420 29794 tablet_bootstrap.cc:654] T 3e9451c3797c4496a66dfdb3e9fb5e38 P d13c58ad8ac441c4a9b396eb04b1a476: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:56.135181 29657 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51980 (request call id 34) took 1059 ms. Trace:
I20260709 12:26:56.214455 29657 rpcz_store.cc:276] 0709 12:26:55.075719 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:55.877020 (+801301us) service_pool.cc:224] Handling call
0709 12:26:55.939550 (+ 62530us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 12:26:55.939651 (+   101us) ts_tablet_manager.cc:682] Creating new metadata...
0709 12:26:56.039356 (+ 99705us) tablet_metadata.cc:665] Metadata flushed
0709 12:26:56.134621 (+ 95265us) tablet_replica.cc:206] Creating consensus instance
0709 12:26:56.135120 (+   499us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:26:56.158964 29726 tablet_service.cc:1511] Processing CreateTablet for tablet 137e03144e754f6e9da4f0544513d27c (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20260709 12:26:56.217588 29799 raft_consensus.cc:493] T 55061e62184b45f0bb1726adc1a4af81 P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:56.217967 29510 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:48124 (request call id 38) took 1146 ms. Trace:
I20260709 12:26:56.218106 29799 raft_consensus.cc:515] T 55061e62184b45f0bb1726adc1a4af81 P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:26:56.218364 29510 rpcz_store.cc:276] 0709 12:26:55.071434 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:55.916732 (+845298us) service_pool.cc:224] Handling call
0709 12:26:55.964917 (+ 48185us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 12:26:55.967317 (+  2400us) ts_tablet_manager.cc:682] Creating new metadata...
0709 12:26:56.080372 (+113055us) tablet_metadata.cc:665] Metadata flushed
0709 12:26:56.217449 (+137077us) tablet_replica.cc:206] Creating consensus instance
0709 12:26:56.217896 (+   447us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:26:56.220798 29799 leader_election.cc:290] T 55061e62184b45f0bb1726adc1a4af81 P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825), d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209)
I20260709 12:26:56.221333 29504 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:48124 (request call id 36) took 1227 ms. Trace:
I20260709 12:26:56.184870 29793 tablet_bootstrap.cc:492] T 7ae3b542012d4f35a9c363eee22976ec P d73d92fb35234d16824c55689b5fce28: Bootstrap starting.
I20260709 12:26:56.120432 29654 tablet_service.cc:1511] Processing CreateTablet for tablet 137e03144e754f6e9da4f0544513d27c (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20260709 12:26:56.212246 29660 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d5fcd1f030614bbe82cf1dc261dfc1c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:56.122828 29652 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51980 (request call id 33) took 1047 ms. Trace:
I20260709 12:26:56.135046 29584 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:50710 (request call id 34) took 1143 ms. Trace:
I20260709 12:26:56.212248 29727 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f8c01e05981747a395e5483b551297dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:56.228665 29654 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 137e03144e754f6e9da4f0544513d27c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:56.227083 29652 rpcz_store.cc:276] 0709 12:26:55.075362 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:55.862940 (+787578us) service_pool.cc:224] Handling call
0709 12:26:55.888112 (+ 25172us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 12:26:55.888217 (+   105us) ts_tablet_manager.cc:682] Creating new metadata...
0709 12:26:55.954635 (+ 66418us) tablet_metadata.cc:665] Metadata flushed
0709 12:26:56.122285 (+167650us) tablet_replica.cc:206] Creating consensus instance
0709 12:26:56.122766 (+   481us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:26:56.236670 29586 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d5fcd1f030614bbe82cf1dc261dfc1c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:56.236328 29657 tablet_service.cc:1511] Processing CreateTablet for tablet 0c428ff656ae4b9d99d5b5073f98725d (DEFAULT_TABLE table=test-workload [id=f7780cb756d84faa8b3aabae44e6820b]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20260709 12:26:56.227818 29584 rpcz_store.cc:276] 0709 12:26:54.992049 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:55.901819 (+909770us) service_pool.cc:224] Handling call
0709 12:26:55.938935 (+ 37116us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 12:26:55.939043 (+   108us) ts_tablet_manager.cc:682] Creating new metadata...
0709 12:26:56.048769 (+109726us) tablet_metadata.cc:665] Metadata flushed
0709 12:26:56.134510 (+ 85741us) tablet_replica.cc:206] Creating consensus instance
0709 12:26:56.134979 (+   469us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:26:56.243124 29655 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51980 (request call id 36) took 1114 ms. Trace:
I20260709 12:26:56.243433 29793 tablet_bootstrap.cc:654] T 7ae3b542012d4f35a9c363eee22976ec P d73d92fb35234d16824c55689b5fce28: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:56.244172 29587 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:50710 (request call id 32) took 1253 ms. Trace:
I20260709 12:26:56.259953 29657 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c428ff656ae4b9d99d5b5073f98725d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:56.260015 29587 rpcz_store.cc:276] 0709 12:26:54.991215 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:55.884066 (+892851us) service_pool.cc:224] Handling call
0709 12:26:55.932980 (+ 48914us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 12:26:55.933078 (+    98us) ts_tablet_manager.cc:682] Creating new metadata...
0709 12:26:56.123850 (+190772us) tablet_metadata.cc:665] Metadata flushed
0709 12:26:56.218514 (+ 94664us) tablet_replica.cc:206] Creating consensus instance
0709 12:26:56.244113 (+ 25599us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":685}
I20260709 12:26:56.244653 29731 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c428ff656ae4b9d99d5b5073f98725d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:56.232107 29578 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:50710 (request call id 37) took 1237 ms. Trace:
I20260709 12:26:56.223531 29504 rpcz_store.cc:276] 0709 12:26:54.993847 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:55.916497 (+922650us) service_pool.cc:224] Handling call
0709 12:26:55.989385 (+ 72888us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 12:26:55.989522 (+   137us) ts_tablet_manager.cc:682] Creating new metadata...
0709 12:26:56.055358 (+ 65836us) tablet_metadata.cc:665] Metadata flushed
0709 12:26:56.220749 (+165391us) tablet_replica.cc:206] Creating consensus instance
0709 12:26:56.221269 (+   520us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:26:56.277251 29579 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:50710 (request call id 38) took 1205 ms. Trace:
I20260709 12:26:56.277731 29579 rpcz_store.cc:276] 0709 12:26:55.071949 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:55.942308 (+870359us) service_pool.cc:224] Handling call
0709 12:26:56.009871 (+ 67563us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 12:26:56.020426 (+ 10555us) ts_tablet_manager.cc:682] Creating new metadata...
0709 12:26:56.216076 (+195650us) tablet_metadata.cc:665] Metadata flushed
0709 12:26:56.267585 (+ 51509us) tablet_replica.cc:206] Creating consensus instance
0709 12:26:56.277190 (+  9605us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:26:56.270207 29578 rpcz_store.cc:276] 0709 12:26:54.995188 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:55.940957 (+945769us) service_pool.cc:224] Handling call
0709 12:26:55.969930 (+ 28973us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 12:26:55.970054 (+   124us) ts_tablet_manager.cc:682] Creating new metadata...
0709 12:26:56.085744 (+115690us) tablet_metadata.cc:665] Metadata flushed
0709 12:26:56.188734 (+102990us) tablet_replica.cc:206] Creating consensus instance
0709 12:26:56.228085 (+ 39351us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:26:56.255244 29658 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51980 (request call id 35) took 1146 ms. Trace:
I20260709 12:26:56.280275 29658 rpcz_store.cc:276] 0709 12:26:55.127972 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:55.902925 (+774953us) service_pool.cc:224] Handling call
0709 12:26:56.029047 (+126122us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 12:26:56.032593 (+  3546us) ts_tablet_manager.cc:682] Creating new metadata...
0709 12:26:56.213653 (+181060us) tablet_metadata.cc:665] Metadata flushed
0709 12:26:56.254619 (+ 40966us) tablet_replica.cc:206] Creating consensus instance
0709 12:26:56.255180 (+   561us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:26:56.243523 29655 rpcz_store.cc:276] 0709 12:26:55.128393 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:55.926436 (+798043us) service_pool.cc:224] Handling call
0709 12:26:56.033980 (+107544us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 12:26:56.034089 (+   109us) ts_tablet_manager.cc:682] Creating new metadata...
0709 12:26:56.221359 (+187270us) tablet_metadata.cc:665] Metadata flushed
0709 12:26:56.242590 (+ 21231us) tablet_replica.cc:206] Creating consensus instance
0709 12:26:56.243061 (+   471us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:26:56.282917 29506 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:48124 (request call id 34) took 1289 ms. Trace:
I20260709 12:26:56.283375 29506 rpcz_store.cc:276] 0709 12:26:54.993185 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:55.856051 (+862866us) service_pool.cc:224] Handling call
0709 12:26:55.896452 (+ 40401us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 12:26:55.896551 (+    99us) ts_tablet_manager.cc:682] Creating new metadata...
0709 12:26:56.121581 (+225030us) tablet_metadata.cc:665] Metadata flushed
0709 12:26:56.282389 (+160808us) tablet_replica.cc:206] Creating consensus instance
0709 12:26:56.282854 (+   465us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:26:56.286733 29745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb6e78ccc01c4707af34cddf81619eab" candidate_uuid: "584dac86183a41d9b3bea1c2085e62fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" is_pre_election: true
I20260709 12:26:56.289011 29793 tablet_bootstrap.cc:492] T 7ae3b542012d4f35a9c363eee22976ec P d73d92fb35234d16824c55689b5fce28: No bootstrap required, opened a new log
W20260709 12:26:56.289353 29486 leader_election.cc:343] T fb6e78ccc01c4707af34cddf81619eab P 584dac86183a41d9b3bea1c2085e62fa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:56.289686 29793 ts_tablet_manager.cc:1403] T 7ae3b542012d4f35a9c363eee22976ec P d73d92fb35234d16824c55689b5fce28: Time spent bootstrapping tablet: real 0.105s	user 0.014s	sys 0.002s
I20260709 12:26:56.290706 29653 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51980 (request call id 37) took 1162 ms. Trace:
I20260709 12:26:56.293376 29735 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3dcbb9c64bf4b5bb6680e8cb81af93e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:56.270366 29580 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:50710 (request call id 39) took 1101 ms. Trace:
I20260709 12:26:56.305629 29580 rpcz_store.cc:276] 0709 12:26:55.168180 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:55.955313 (+787133us) service_pool.cc:224] Handling call
0709 12:26:56.046943 (+ 91630us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 12:26:56.047038 (+    95us) ts_tablet_manager.cc:682] Creating new metadata...
0709 12:26:56.190081 (+143043us) tablet_metadata.cc:665] Metadata flushed
0709 12:26:56.214582 (+ 24501us) tablet_replica.cc:206] Creating consensus instance
0709 12:26:56.267987 (+ 53405us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:26:56.295928 29793 raft_consensus.cc:359] T 7ae3b542012d4f35a9c363eee22976ec P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:26:56.308121 29793 raft_consensus.cc:385] T 7ae3b542012d4f35a9c363eee22976ec P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:56.308768 29793 raft_consensus.cc:740] T 7ae3b542012d4f35a9c363eee22976ec P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d73d92fb35234d16824c55689b5fce28, State: Initialized, Role: FOLLOWER
I20260709 12:26:56.309137 29597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592801934cad4789b9ce8dc09f1470bd" candidate_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d73d92fb35234d16824c55689b5fce28" is_pre_election: true
I20260709 12:26:56.309633 29793 consensus_queue.cc:260] T 7ae3b542012d4f35a9c363eee22976ec P d73d92fb35234d16824c55689b5fce28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:26:56.311183 29596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55061e62184b45f0bb1726adc1a4af81" candidate_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d73d92fb35234d16824c55689b5fce28" is_pre_election: true
I20260709 12:26:56.312661 29793 ts_tablet_manager.cc:1434] T 7ae3b542012d4f35a9c363eee22976ec P d73d92fb35234d16824c55689b5fce28: Time spent starting tablet: real 0.023s	user 0.007s	sys 0.000s
W20260709 12:26:56.314937 29708 leader_election.cc:343] T 592801934cad4789b9ce8dc09f1470bd P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:56.315618 29708 leader_election.cc:304] T 592801934cad4789b9ce8dc09f1470bd P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c6610c00ffb948ce9c34f41dec3d49e8; no voters: 584dac86183a41d9b3bea1c2085e62fa, d73d92fb35234d16824c55689b5fce28
I20260709 12:26:56.316865 29799 raft_consensus.cc:2749] T 592801934cad4789b9ce8dc09f1470bd P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260709 12:26:56.318310 29708 leader_election.cc:343] T 55061e62184b45f0bb1726adc1a4af81 P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:56.297112 29730 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33132 (request call id 40) took 1018 ms. Trace:
I20260709 12:26:56.298138 29728 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33132 (request call id 38) took 1150 ms. Trace:
I20260709 12:26:56.320190 29728 rpcz_store.cc:276] 0709 12:26:55.147380 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:55.938469 (+791089us) service_pool.cc:224] Handling call
0709 12:26:55.956434 (+ 17965us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 12:26:55.965309 (+  8875us) ts_tablet_manager.cc:682] Creating new metadata...
0709 12:26:56.213715 (+248406us) tablet_metadata.cc:665] Metadata flushed
0709 12:26:56.278621 (+ 64906us) tablet_replica.cc:206] Creating consensus instance
0709 12:26:56.298057 (+ 19436us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":18782}
I20260709 12:26:56.298203 29733 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33132 (request call id 36) took 1331 ms. Trace:
I20260709 12:26:56.321915 29733 rpcz_store.cc:276] 0709 12:26:54.967398 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:55.900913 (+933515us) service_pool.cc:224] Handling call
0709 12:26:55.940533 (+ 39620us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 12:26:55.940622 (+    89us) ts_tablet_manager.cc:682] Creating new metadata...
0709 12:26:56.036796 (+ 96174us) tablet_metadata.cc:665] Metadata flushed
0709 12:26:56.261759 (+224963us) tablet_replica.cc:206] Creating consensus instance
0709 12:26:56.298142 (+ 36383us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4716}
I20260709 12:26:56.291807 29653 rpcz_store.cc:276] 0709 12:26:55.128736 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:55.892026 (+763290us) service_pool.cc:224] Handling call
0709 12:26:56.016021 (+123995us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 12:26:56.016197 (+   176us) ts_tablet_manager.cc:682] Creating new metadata...
0709 12:26:56.043486 (+ 27289us) tablet_metadata.cc:665] Metadata flushed
0709 12:26:56.210773 (+167287us) tablet_replica.cc:206] Creating consensus instance
0709 12:26:56.290234 (+ 79461us) spinlock_profiling.cc:243] Waited 11 ms on lock 0x7b58005bc598. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b474fbb71 00007f7b474fc1a2 00007f7b5043d889 00007f7b4f56ed88 00007f7b4f635973 00007f7b4f63170f 00007f7b4f631ecd 00007f7b4f5e0400 00007f7b48a65af4 00007f7b48a65a82
0709 12:26:56.290632 (+   398us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":25249536}
I20260709 12:26:56.329052 29793 tablet_bootstrap.cc:492] T 4685c2e824c14fc1bcd45c16169798cc P d73d92fb35234d16824c55689b5fce28: Bootstrap starting.
I20260709 12:26:56.297838 29661 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51980 (request call id 40) took 1020 ms. Trace:
I20260709 12:26:56.330289 29790 tablet_bootstrap.cc:654] T 3e9451c3797c4496a66dfdb3e9fb5e38 P c6610c00ffb948ce9c34f41dec3d49e8: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:56.319326 29730 rpcz_store.cc:276] 0709 12:26:55.278608 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:56.061136 (+782528us) service_pool.cc:224] Handling call
0709 12:26:56.063094 (+  1958us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 12:26:56.067155 (+  4061us) ts_tablet_manager.cc:682] Creating new metadata...
0709 12:26:56.200772 (+133617us) tablet_metadata.cc:665] Metadata flushed
0709 12:26:56.296494 (+ 95722us) tablet_replica.cc:206] Creating consensus instance
0709 12:26:56.297031 (+   537us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":6577}
I20260709 12:26:56.333453 29729 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33132 (request call id 33) took 1367 ms. Trace:
I20260709 12:26:56.329943 29661 rpcz_store.cc:276] 0709 12:26:55.277756 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:56.027326 (+749570us) service_pool.cc:224] Handling call
0709 12:26:56.046563 (+ 19237us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 12:26:56.046655 (+    92us) ts_tablet_manager.cc:682] Creating new metadata...
0709 12:26:56.191556 (+144901us) tablet_metadata.cc:665] Metadata flushed
0709 12:26:56.297270 (+105714us) tablet_replica.cc:206] Creating consensus instance
0709 12:26:56.297775 (+   505us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:26:56.333966 29729 rpcz_store.cc:276] 0709 12:26:54.966291 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:55.890816 (+924525us) service_pool.cc:224] Handling call
0709 12:26:55.929224 (+ 38408us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 12:26:55.929321 (+    97us) ts_tablet_manager.cc:682] Creating new metadata...
0709 12:26:56.129560 (+200239us) tablet_metadata.cc:665] Metadata flushed
0709 12:26:56.326719 (+197159us) tablet_replica.cc:206] Creating consensus instance
0709 12:26:56.333383 (+  6664us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":8014}
I20260709 12:26:56.336833 29659 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51980 (request call id 39) took 1060 ms. Trace:
I20260709 12:26:56.318598 29726 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 137e03144e754f6e9da4f0544513d27c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:56.259335 29585 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:50710 (request call id 36) took 1264 ms. Trace:
I20260709 12:26:56.337355 29659 rpcz_store.cc:276] 0709 12:26:55.276606 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:55.993614 (+717008us) service_pool.cc:224] Handling call
0709 12:26:55.995468 (+  1854us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 12:26:55.995569 (+   101us) ts_tablet_manager.cc:682] Creating new metadata...
0709 12:26:56.179982 (+184413us) tablet_metadata.cc:665] Metadata flushed
0709 12:26:56.336136 (+156154us) tablet_replica.cc:206] Creating consensus instance
0709 12:26:56.336751 (+   615us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:26:56.352181 29808 raft_consensus.cc:493] T 43c7bfb33af74d4f912e5153e8a54b3e P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:56.352778 29808 raft_consensus.cc:515] T 43c7bfb33af74d4f912e5153e8a54b3e P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:26:56.356132 29808 leader_election.cc:290] T 43c7bfb33af74d4f912e5153e8a54b3e P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209), c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895)
I20260709 12:26:56.362159 29581 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:50710 (request call id 40) took 1109 ms. Trace:
I20260709 12:26:56.363754 29581 rpcz_store.cc:276] 0709 12:26:55.252941 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:55.989144 (+736203us) service_pool.cc:224] Handling call
0709 12:26:55.992301 (+  3157us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 12:26:55.992410 (+   109us) ts_tablet_manager.cc:682] Creating new metadata...
0709 12:26:56.167758 (+175348us) tablet_metadata.cc:665] Metadata flushed
0709 12:26:56.347069 (+179311us) tablet_replica.cc:206] Creating consensus instance
0709 12:26:56.362087 (+ 15018us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:26:56.342394 29585 rpcz_store.cc:276] 0709 12:26:54.994753 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:55.928320 (+933567us) service_pool.cc:224] Handling call
0709 12:26:55.930220 (+  1900us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 12:26:55.930313 (+    93us) ts_tablet_manager.cc:682] Creating new metadata...
0709 12:26:56.098729 (+168416us) tablet_metadata.cc:665] Metadata flushed
0709 12:26:56.258588 (+159859us) tablet_replica.cc:206] Creating consensus instance
0709 12:26:56.259248 (+   660us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:26:56.368935 29792 tablet_bootstrap.cc:492] T d21ab1641501475b829863275dc8182a P 584dac86183a41d9b3bea1c2085e62fa: No bootstrap required, opened a new log
I20260709 12:26:56.369659 29792 ts_tablet_manager.cc:1403] T d21ab1641501475b829863275dc8182a P 584dac86183a41d9b3bea1c2085e62fa: Time spent bootstrapping tablet: real 0.496s	user 0.030s	sys 0.036s
I20260709 12:26:56.373553 29727 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33132 (request call id 39) took 1096 ms. Trace:
I20260709 12:26:56.373997 29727 rpcz_store.cc:276] 0709 12:26:55.277713 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:56.042156 (+764443us) service_pool.cc:224] Handling call
0709 12:26:56.088186 (+ 46030us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 12:26:56.089137 (+   951us) ts_tablet_manager.cc:682] Creating new metadata...
0709 12:26:56.330947 (+241810us) tablet_metadata.cc:665] Metadata flushed
0709 12:26:56.372913 (+ 41966us) tablet_replica.cc:206] Creating consensus instance
0709 12:26:56.373481 (+   568us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:26:56.386013 29505 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:48124 (request call id 41) took 1123 ms. Trace:
I20260709 12:26:56.395411 29732 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33132 (request call id 35) took 1428 ms. Trace:
I20260709 12:26:56.395869 29732 rpcz_store.cc:276] 0709 12:26:54.966925 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:55.896885 (+929960us) service_pool.cc:224] Handling call
0709 12:26:55.986084 (+ 89199us) spinlock_profiling.cc:243] Waited 14.9 ms on lock 0x7b58007e7598. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b474fbb71 00007f7b474fc1a2 00007f7b5043d889 00007f7b4f56ed88 00007f7b4f631b80 00007f7b4f5e0400 00007f7b48a65af4 00007f7b48a65a82 00007f7b48a659b1 00007f7b48a6592c
0709 12:26:56.036052 (+ 49968us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 12:26:56.036247 (+   195us) ts_tablet_manager.cc:682] Creating new metadata...
0709 12:26:56.301767 (+265520us) tablet_metadata.cc:665] Metadata flushed
0709 12:26:56.394864 (+ 93097us) tablet_replica.cc:206] Creating consensus instance
0709 12:26:56.395350 (+   486us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":34263296}
I20260709 12:26:56.401525 29792 raft_consensus.cc:359] T d21ab1641501475b829863275dc8182a P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:26:56.402680 29792 raft_consensus.cc:385] T d21ab1641501475b829863275dc8182a P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:56.403071 29792 raft_consensus.cc:740] T d21ab1641501475b829863275dc8182a P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 584dac86183a41d9b3bea1c2085e62fa, State: Initialized, Role: FOLLOWER
I20260709 12:26:56.405750 29792 consensus_queue.cc:260] T d21ab1641501475b829863275dc8182a P 584dac86183a41d9b3bea1c2085e62fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:26:56.409185 29792 ts_tablet_manager.cc:1434] T d21ab1641501475b829863275dc8182a P 584dac86183a41d9b3bea1c2085e62fa: Time spent starting tablet: real 0.039s	user 0.011s	sys 0.000s
I20260709 12:26:56.410357 29792 tablet_bootstrap.cc:492] T fcb6a7f720b14dc98e821f663357a0bb P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap starting.
I20260709 12:26:56.410104 29671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55061e62184b45f0bb1726adc1a4af81" candidate_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" is_pre_election: true
W20260709 12:26:56.412096 29709 leader_election.cc:343] T 55061e62184b45f0bb1726adc1a4af81 P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:56.412618 29709 leader_election.cc:304] T 55061e62184b45f0bb1726adc1a4af81 P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c6610c00ffb948ce9c34f41dec3d49e8; no voters: d13c58ad8ac441c4a9b396eb04b1a476, d73d92fb35234d16824c55689b5fce28
I20260709 12:26:56.413496 29799 raft_consensus.cc:2749] T 55061e62184b45f0bb1726adc1a4af81 P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:26:56.386698 29505 rpcz_store.cc:276] 0709 12:26:55.262886 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:55.943303 (+680417us) service_pool.cc:224] Handling call
0709 12:26:55.998895 (+ 55592us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 12:26:55.999119 (+   224us) ts_tablet_manager.cc:682] Creating new metadata...
0709 12:26:56.177130 (+178011us) tablet_metadata.cc:665] Metadata flushed
0709 12:26:56.385307 (+208177us) tablet_replica.cc:206] Creating consensus instance
0709 12:26:56.385929 (+   622us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:26:56.415999 29734 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33132 (request call id 37) took 1349 ms. Trace:
I20260709 12:26:56.417616 29798 raft_consensus.cc:493] T d42259e436d241b29fb4ded16d9b6455 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:56.419745 29798 raft_consensus.cc:515] T d42259e436d241b29fb4ded16d9b6455 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:26:56.418679 29734 rpcz_store.cc:276] 0709 12:26:55.066350 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:55.913750 (+847400us) service_pool.cc:224] Handling call
0709 12:26:56.029378 (+115628us) spinlock_profiling.cc:243] Waited 1.06 ms on lock 0x7b58007e7598. stack: 00007f7b47501343 00007f7b47501240 00007f7b46e901d4 00007f7b50bce304 00007f7b50bce299 00007f7b473bbdb1 00007f7b474fbe9c 00007f7b474fba5b 00007f7b504148d9 00007f7b4f56eee0 00007f7b4f654831 00007f7b4f631ac3 00007f7b4f5e0400 00007f7b48a65af4 00007f7b48a65a82 00007f7b48a659b1
0709 12:26:56.045248 (+ 15870us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 12:26:56.045365 (+   117us) ts_tablet_manager.cc:682] Creating new metadata...
0709 12:26:56.186260 (+140895us) tablet_metadata.cc:665] Metadata flushed
0709 12:26:56.415428 (+229168us) tablet_replica.cc:206] Creating consensus instance
0709 12:26:56.415934 (+   506us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":10539,"spinlock_wait_cycles":2450176}
I20260709 12:26:56.425308 29794 tablet_bootstrap.cc:492] T 3e9451c3797c4496a66dfdb3e9fb5e38 P d13c58ad8ac441c4a9b396eb04b1a476: No bootstrap required, opened a new log
I20260709 12:26:56.426035 29794 ts_tablet_manager.cc:1403] T 3e9451c3797c4496a66dfdb3e9fb5e38 P d13c58ad8ac441c4a9b396eb04b1a476: Time spent bootstrapping tablet: real 0.373s	user 0.008s	sys 0.018s
I20260709 12:26:56.429538 29792 tablet_bootstrap.cc:654] T fcb6a7f720b14dc98e821f663357a0bb P 584dac86183a41d9b3bea1c2085e62fa: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:56.431862 29522 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d42259e436d241b29fb4ded16d9b6455" candidate_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "584dac86183a41d9b3bea1c2085e62fa" is_pre_election: true
I20260709 12:26:56.433612 29793 tablet_bootstrap.cc:654] T 4685c2e824c14fc1bcd45c16169798cc P d73d92fb35234d16824c55689b5fce28: Neither blocks nor log segments found. Creating new log.
W20260709 12:26:56.433535 29634 leader_election.cc:343] T d42259e436d241b29fb4ded16d9b6455 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:56.435603 29798 leader_election.cc:290] T d42259e436d241b29fb4ded16d9b6455 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033), d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825)
I20260709 12:26:56.437911 29794 raft_consensus.cc:359] T 3e9451c3797c4496a66dfdb3e9fb5e38 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:26:56.443033 29794 raft_consensus.cc:385] T 3e9451c3797c4496a66dfdb3e9fb5e38 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:56.443451 29794 raft_consensus.cc:740] T 3e9451c3797c4496a66dfdb3e9fb5e38 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d13c58ad8ac441c4a9b396eb04b1a476, State: Initialized, Role: FOLLOWER
I20260709 12:26:56.444528 29794 consensus_queue.cc:260] T 3e9451c3797c4496a66dfdb3e9fb5e38 P d13c58ad8ac441c4a9b396eb04b1a476 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:26:56.447487 29794 ts_tablet_manager.cc:1434] T 3e9451c3797c4496a66dfdb3e9fb5e38 P d13c58ad8ac441c4a9b396eb04b1a476: Time spent starting tablet: real 0.021s	user 0.006s	sys 0.000s
I20260709 12:26:56.449762 29794 tablet_bootstrap.cc:492] T fcb6a7f720b14dc98e821f663357a0bb P d13c58ad8ac441c4a9b396eb04b1a476: Bootstrap starting.
I20260709 12:26:56.460485 29726 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33132 (request call id 41) took 1029 ms. Trace:
I20260709 12:26:56.461472 29726 rpcz_store.cc:276] 0709 12:26:55.431197 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:56.157937 (+726740us) service_pool.cc:224] Handling call
0709 12:26:56.216389 (+ 58452us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 12:26:56.216493 (+   104us) ts_tablet_manager.cc:682] Creating new metadata...
0709 12:26:56.385732 (+169239us) tablet_metadata.cc:665] Metadata flushed
0709 12:26:56.459750 (+ 74018us) tablet_replica.cc:206] Creating consensus instance
0709 12:26:56.460400 (+   650us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:26:56.440919 29656 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51980 (request call id 38) took 1279 ms. Trace:
I20260709 12:26:56.468663 29794 tablet_bootstrap.cc:654] T fcb6a7f720b14dc98e821f663357a0bb P d13c58ad8ac441c4a9b396eb04b1a476: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:56.464300 29656 rpcz_store.cc:276] 0709 12:26:55.161395 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:55.981940 (+820545us) service_pool.cc:224] Handling call
0709 12:26:56.024529 (+ 42589us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 12:26:56.024637 (+   108us) ts_tablet_manager.cc:682] Creating new metadata...
0709 12:26:56.272406 (+247769us) tablet_metadata.cc:665] Metadata flushed
0709 12:26:56.440254 (+167848us) tablet_replica.cc:206] Creating consensus instance
0709 12:26:56.440838 (+   584us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:26:56.476660 29790 tablet_bootstrap.cc:492] T 3e9451c3797c4496a66dfdb3e9fb5e38 P c6610c00ffb948ce9c34f41dec3d49e8: No bootstrap required, opened a new log
I20260709 12:26:56.477209 29790 ts_tablet_manager.cc:1403] T 3e9451c3797c4496a66dfdb3e9fb5e38 P c6610c00ffb948ce9c34f41dec3d49e8: Time spent bootstrapping tablet: real 0.433s	user 0.023s	sys 0.008s
I20260709 12:26:56.480647 29792 tablet_bootstrap.cc:492] T fcb6a7f720b14dc98e821f663357a0bb P 584dac86183a41d9b3bea1c2085e62fa: No bootstrap required, opened a new log
I20260709 12:26:56.483162 29792 ts_tablet_manager.cc:1403] T fcb6a7f720b14dc98e821f663357a0bb P 584dac86183a41d9b3bea1c2085e62fa: Time spent bootstrapping tablet: real 0.073s	user 0.021s	sys 0.022s
I20260709 12:26:56.483003 29596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb6e78ccc01c4707af34cddf81619eab" candidate_uuid: "584dac86183a41d9b3bea1c2085e62fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d73d92fb35234d16824c55689b5fce28" is_pre_election: true
I20260709 12:26:56.483573 29745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43c7bfb33af74d4f912e5153e8a54b3e" candidate_uuid: "d73d92fb35234d16824c55689b5fce28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" is_pre_election: true
I20260709 12:26:56.484755 29671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43c7bfb33af74d4f912e5153e8a54b3e" candidate_uuid: "d73d92fb35234d16824c55689b5fce28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" is_pre_election: true
W20260709 12:26:56.485200 29561 leader_election.cc:343] T 43c7bfb33af74d4f912e5153e8a54b3e P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:56.486543 29793 tablet_bootstrap.cc:492] T 4685c2e824c14fc1bcd45c16169798cc P d73d92fb35234d16824c55689b5fce28: No bootstrap required, opened a new log
W20260709 12:26:56.486735 29485 leader_election.cc:343] T fb6e78ccc01c4707af34cddf81619eab P 584dac86183a41d9b3bea1c2085e62fa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:56.486510 29792 raft_consensus.cc:359] T fcb6a7f720b14dc98e821f663357a0bb P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:26:56.487349 29793 ts_tablet_manager.cc:1403] T 4685c2e824c14fc1bcd45c16169798cc P d73d92fb35234d16824c55689b5fce28: Time spent bootstrapping tablet: real 0.159s	user 0.013s	sys 0.004s
I20260709 12:26:56.487534 29792 raft_consensus.cc:385] T fcb6a7f720b14dc98e821f663357a0bb P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:56.487462 29485 leader_election.cc:304] T fb6e78ccc01c4707af34cddf81619eab P 584dac86183a41d9b3bea1c2085e62fa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 584dac86183a41d9b3bea1c2085e62fa; no voters: c6610c00ffb948ce9c34f41dec3d49e8, d73d92fb35234d16824c55689b5fce28
I20260709 12:26:56.488046 29792 raft_consensus.cc:740] T fcb6a7f720b14dc98e821f663357a0bb P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 584dac86183a41d9b3bea1c2085e62fa, State: Initialized, Role: FOLLOWER
I20260709 12:26:56.488482 29804 raft_consensus.cc:2749] T fb6e78ccc01c4707af34cddf81619eab P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:26:56.490340 29794 tablet_bootstrap.cc:492] T fcb6a7f720b14dc98e821f663357a0bb P d13c58ad8ac441c4a9b396eb04b1a476: No bootstrap required, opened a new log
W20260709 12:26:56.490501 29561 leader_election.cc:343] T 43c7bfb33af74d4f912e5153e8a54b3e P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:56.489089 29792 consensus_queue.cc:260] T fcb6a7f720b14dc98e821f663357a0bb P 584dac86183a41d9b3bea1c2085e62fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:26:56.490947 29794 ts_tablet_manager.cc:1403] T fcb6a7f720b14dc98e821f663357a0bb P d13c58ad8ac441c4a9b396eb04b1a476: Time spent bootstrapping tablet: real 0.041s	user 0.011s	sys 0.003s
I20260709 12:26:56.492897 29792 ts_tablet_manager.cc:1434] T fcb6a7f720b14dc98e821f663357a0bb P 584dac86183a41d9b3bea1c2085e62fa: Time spent starting tablet: real 0.009s	user 0.004s	sys 0.004s
I20260709 12:26:56.494331 29792 tablet_bootstrap.cc:492] T 7ae3b542012d4f35a9c363eee22976ec P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap starting.
I20260709 12:26:56.494484 29561 leader_election.cc:304] T 43c7bfb33af74d4f912e5153e8a54b3e P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d73d92fb35234d16824c55689b5fce28; no voters: c6610c00ffb948ce9c34f41dec3d49e8, d13c58ad8ac441c4a9b396eb04b1a476
I20260709 12:26:56.496233 29790 raft_consensus.cc:359] T 3e9451c3797c4496a66dfdb3e9fb5e38 P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:26:56.497123 29790 raft_consensus.cc:385] T 3e9451c3797c4496a66dfdb3e9fb5e38 P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:56.497470 29790 raft_consensus.cc:740] T 3e9451c3797c4496a66dfdb3e9fb5e38 P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6610c00ffb948ce9c34f41dec3d49e8, State: Initialized, Role: FOLLOWER
I20260709 12:26:56.497663 29808 raft_consensus.cc:2749] T 43c7bfb33af74d4f912e5153e8a54b3e P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:26:56.494628 29794 raft_consensus.cc:359] T fcb6a7f720b14dc98e821f663357a0bb P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:26:56.498669 29794 raft_consensus.cc:385] T fcb6a7f720b14dc98e821f663357a0bb P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:56.498970 29794 raft_consensus.cc:740] T fcb6a7f720b14dc98e821f663357a0bb P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d13c58ad8ac441c4a9b396eb04b1a476, State: Initialized, Role: FOLLOWER
I20260709 12:26:56.499917 29794 consensus_queue.cc:260] T fcb6a7f720b14dc98e821f663357a0bb P d13c58ad8ac441c4a9b396eb04b1a476 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:26:56.501264 29596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d42259e436d241b29fb4ded16d9b6455" candidate_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d73d92fb35234d16824c55689b5fce28" is_pre_election: true
I20260709 12:26:56.501785 29792 tablet_bootstrap.cc:654] T 7ae3b542012d4f35a9c363eee22976ec P 584dac86183a41d9b3bea1c2085e62fa: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:56.502192 29790 consensus_queue.cc:260] T 3e9451c3797c4496a66dfdb3e9fb5e38 P c6610c00ffb948ce9c34f41dec3d49e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
W20260709 12:26:56.503036 29634 leader_election.cc:343] T d42259e436d241b29fb4ded16d9b6455 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:56.503571 29634 leader_election.cc:304] T d42259e436d241b29fb4ded16d9b6455 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d13c58ad8ac441c4a9b396eb04b1a476; no voters: 584dac86183a41d9b3bea1c2085e62fa, d73d92fb35234d16824c55689b5fce28
I20260709 12:26:56.504415 29798 raft_consensus.cc:2749] T d42259e436d241b29fb4ded16d9b6455 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:26:56.499868 29793 raft_consensus.cc:359] T 4685c2e824c14fc1bcd45c16169798cc P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:26:56.507773 29793 raft_consensus.cc:385] T 4685c2e824c14fc1bcd45c16169798cc P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:56.508128 29793 raft_consensus.cc:740] T 4685c2e824c14fc1bcd45c16169798cc P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d73d92fb35234d16824c55689b5fce28, State: Initialized, Role: FOLLOWER
I20260709 12:26:56.509572 29794 ts_tablet_manager.cc:1434] T fcb6a7f720b14dc98e821f663357a0bb P d13c58ad8ac441c4a9b396eb04b1a476: Time spent starting tablet: real 0.018s	user 0.008s	sys 0.000s
I20260709 12:26:56.509162 29793 consensus_queue.cc:260] T 4685c2e824c14fc1bcd45c16169798cc P d73d92fb35234d16824c55689b5fce28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:26:56.511094 29794 tablet_bootstrap.cc:492] T 7ae3b542012d4f35a9c363eee22976ec P d13c58ad8ac441c4a9b396eb04b1a476: Bootstrap starting.
I20260709 12:26:56.516184 29793 ts_tablet_manager.cc:1434] T 4685c2e824c14fc1bcd45c16169798cc P d73d92fb35234d16824c55689b5fce28: Time spent starting tablet: real 0.028s	user 0.008s	sys 0.000s
I20260709 12:26:56.517381 29793 tablet_bootstrap.cc:492] T fcb6a7f720b14dc98e821f663357a0bb P d73d92fb35234d16824c55689b5fce28: Bootstrap starting.
I20260709 12:26:56.521907 29790 ts_tablet_manager.cc:1434] T 3e9451c3797c4496a66dfdb3e9fb5e38 P c6610c00ffb948ce9c34f41dec3d49e8: Time spent starting tablet: real 0.044s	user 0.008s	sys 0.000s
I20260709 12:26:56.529482 29793 tablet_bootstrap.cc:654] T fcb6a7f720b14dc98e821f663357a0bb P d73d92fb35234d16824c55689b5fce28: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:56.532127 29794 tablet_bootstrap.cc:654] T 7ae3b542012d4f35a9c363eee22976ec P d13c58ad8ac441c4a9b396eb04b1a476: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:56.533034 29790 tablet_bootstrap.cc:492] T 8627b58b4f42491597125eb45f8d3b61 P c6610c00ffb948ce9c34f41dec3d49e8: Bootstrap starting.
I20260709 12:26:56.541793 29793 tablet_bootstrap.cc:492] T fcb6a7f720b14dc98e821f663357a0bb P d73d92fb35234d16824c55689b5fce28: No bootstrap required, opened a new log
I20260709 12:26:56.542366 29793 ts_tablet_manager.cc:1403] T fcb6a7f720b14dc98e821f663357a0bb P d73d92fb35234d16824c55689b5fce28: Time spent bootstrapping tablet: real 0.025s	user 0.005s	sys 0.012s
I20260709 12:26:56.542933 29790 tablet_bootstrap.cc:654] T 8627b58b4f42491597125eb45f8d3b61 P c6610c00ffb948ce9c34f41dec3d49e8: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:56.544914 29792 tablet_bootstrap.cc:492] T 7ae3b542012d4f35a9c363eee22976ec P 584dac86183a41d9b3bea1c2085e62fa: No bootstrap required, opened a new log
I20260709 12:26:56.544929 29794 tablet_bootstrap.cc:492] T 7ae3b542012d4f35a9c363eee22976ec P d13c58ad8ac441c4a9b396eb04b1a476: No bootstrap required, opened a new log
I20260709 12:26:56.545581 29792 ts_tablet_manager.cc:1403] T 7ae3b542012d4f35a9c363eee22976ec P 584dac86183a41d9b3bea1c2085e62fa: Time spent bootstrapping tablet: real 0.052s	user 0.019s	sys 0.027s
I20260709 12:26:56.545606 29794 ts_tablet_manager.cc:1403] T 7ae3b542012d4f35a9c363eee22976ec P d13c58ad8ac441c4a9b396eb04b1a476: Time spent bootstrapping tablet: real 0.035s	user 0.011s	sys 0.012s
I20260709 12:26:56.545709 29793 raft_consensus.cc:359] T fcb6a7f720b14dc98e821f663357a0bb P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:26:56.546882 29793 raft_consensus.cc:385] T fcb6a7f720b14dc98e821f663357a0bb P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:56.547263 29793 raft_consensus.cc:740] T fcb6a7f720b14dc98e821f663357a0bb P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d73d92fb35234d16824c55689b5fce28, State: Initialized, Role: FOLLOWER
I20260709 12:26:56.549008 29792 raft_consensus.cc:359] T 7ae3b542012d4f35a9c363eee22976ec P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:26:56.548292 29793 consensus_queue.cc:260] T fcb6a7f720b14dc98e821f663357a0bb P d73d92fb35234d16824c55689b5fce28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:26:56.549193 29794 raft_consensus.cc:359] T 7ae3b542012d4f35a9c363eee22976ec P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:26:56.549813 29792 raft_consensus.cc:385] T 7ae3b542012d4f35a9c363eee22976ec P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:56.550050 29794 raft_consensus.cc:385] T 7ae3b542012d4f35a9c363eee22976ec P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:56.550222 29792 raft_consensus.cc:740] T 7ae3b542012d4f35a9c363eee22976ec P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 584dac86183a41d9b3bea1c2085e62fa, State: Initialized, Role: FOLLOWER
I20260709 12:26:56.550416 29794 raft_consensus.cc:740] T 7ae3b542012d4f35a9c363eee22976ec P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d13c58ad8ac441c4a9b396eb04b1a476, State: Initialized, Role: FOLLOWER
I20260709 12:26:56.551153 29792 consensus_queue.cc:260] T 7ae3b542012d4f35a9c363eee22976ec P 584dac86183a41d9b3bea1c2085e62fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:26:56.551240 29794 consensus_queue.cc:260] T 7ae3b542012d4f35a9c363eee22976ec P d13c58ad8ac441c4a9b396eb04b1a476 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:26:56.556044 29792 ts_tablet_manager.cc:1434] T 7ae3b542012d4f35a9c363eee22976ec P 584dac86183a41d9b3bea1c2085e62fa: Time spent starting tablet: real 0.010s	user 0.007s	sys 0.000s
I20260709 12:26:56.556478 29793 ts_tablet_manager.cc:1434] T fcb6a7f720b14dc98e821f663357a0bb P d73d92fb35234d16824c55689b5fce28: Time spent starting tablet: real 0.014s	user 0.006s	sys 0.005s
I20260709 12:26:56.557113 29792 tablet_bootstrap.cc:492] T 742beffede4f407d8a72909d2b147e6d P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap starting.
I20260709 12:26:56.557590 29793 tablet_bootstrap.cc:492] T d21ab1641501475b829863275dc8182a P d73d92fb35234d16824c55689b5fce28: Bootstrap starting.
I20260709 12:26:56.560829 29794 ts_tablet_manager.cc:1434] T 7ae3b542012d4f35a9c363eee22976ec P d13c58ad8ac441c4a9b396eb04b1a476: Time spent starting tablet: real 0.015s	user 0.007s	sys 0.000s
I20260709 12:26:56.560999 29808 raft_consensus.cc:493] T 4685c2e824c14fc1bcd45c16169798cc P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:56.561604 29808 raft_consensus.cc:515] T 4685c2e824c14fc1bcd45c16169798cc P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:26:56.564895 29745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4685c2e824c14fc1bcd45c16169798cc" candidate_uuid: "d73d92fb35234d16824c55689b5fce28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" is_pre_election: true
I20260709 12:26:56.565305 29794 tablet_bootstrap.cc:492] T 742beffede4f407d8a72909d2b147e6d P d13c58ad8ac441c4a9b396eb04b1a476: Bootstrap starting.
W20260709 12:26:56.566569 29561 leader_election.cc:343] T 4685c2e824c14fc1bcd45c16169798cc P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:56.567629 29792 tablet_bootstrap.cc:654] T 742beffede4f407d8a72909d2b147e6d P 584dac86183a41d9b3bea1c2085e62fa: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:56.567689 29808 leader_election.cc:290] T 4685c2e824c14fc1bcd45c16169798cc P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895), 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033)
I20260709 12:26:56.574659 29793 tablet_bootstrap.cc:654] T d21ab1641501475b829863275dc8182a P d73d92fb35234d16824c55689b5fce28: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:56.574651 29794 tablet_bootstrap.cc:654] T 742beffede4f407d8a72909d2b147e6d P d13c58ad8ac441c4a9b396eb04b1a476: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:56.580770 29522 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4685c2e824c14fc1bcd45c16169798cc" candidate_uuid: "d73d92fb35234d16824c55689b5fce28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "584dac86183a41d9b3bea1c2085e62fa" is_pre_election: true
W20260709 12:26:56.583055 29560 leader_election.cc:343] T 4685c2e824c14fc1bcd45c16169798cc P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:56.583537 29560 leader_election.cc:304] T 4685c2e824c14fc1bcd45c16169798cc P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d73d92fb35234d16824c55689b5fce28; no voters: 584dac86183a41d9b3bea1c2085e62fa, c6610c00ffb948ce9c34f41dec3d49e8
I20260709 12:26:56.584976 29808 raft_consensus.cc:2749] T 4685c2e824c14fc1bcd45c16169798cc P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:26:56.587689 29790 tablet_bootstrap.cc:492] T 8627b58b4f42491597125eb45f8d3b61 P c6610c00ffb948ce9c34f41dec3d49e8: No bootstrap required, opened a new log
I20260709 12:26:56.588804 29790 ts_tablet_manager.cc:1403] T 8627b58b4f42491597125eb45f8d3b61 P c6610c00ffb948ce9c34f41dec3d49e8: Time spent bootstrapping tablet: real 0.056s	user 0.015s	sys 0.007s
I20260709 12:26:56.588825 29793 tablet_bootstrap.cc:492] T d21ab1641501475b829863275dc8182a P d73d92fb35234d16824c55689b5fce28: No bootstrap required, opened a new log
I20260709 12:26:56.589550 29794 tablet_bootstrap.cc:492] T 742beffede4f407d8a72909d2b147e6d P d13c58ad8ac441c4a9b396eb04b1a476: No bootstrap required, opened a new log
I20260709 12:26:56.589926 29793 ts_tablet_manager.cc:1403] T d21ab1641501475b829863275dc8182a P d73d92fb35234d16824c55689b5fce28: Time spent bootstrapping tablet: real 0.033s	user 0.011s	sys 0.010s
I20260709 12:26:56.591933 29794 ts_tablet_manager.cc:1403] T 742beffede4f407d8a72909d2b147e6d P d13c58ad8ac441c4a9b396eb04b1a476: Time spent bootstrapping tablet: real 0.027s	user 0.020s	sys 0.000s
I20260709 12:26:56.592932 29804 raft_consensus.cc:493] T 7ae3b542012d4f35a9c363eee22976ec P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:56.593160 29793 raft_consensus.cc:359] T d21ab1641501475b829863275dc8182a P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:26:56.593050 29790 raft_consensus.cc:359] T 8627b58b4f42491597125eb45f8d3b61 P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:26:56.593672 29804 raft_consensus.cc:515] T 7ae3b542012d4f35a9c363eee22976ec P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:26:56.594115 29793 raft_consensus.cc:385] T d21ab1641501475b829863275dc8182a P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:56.594210 29790 raft_consensus.cc:385] T 8627b58b4f42491597125eb45f8d3b61 P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:56.594542 29793 raft_consensus.cc:740] T d21ab1641501475b829863275dc8182a P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d73d92fb35234d16824c55689b5fce28, State: Initialized, Role: FOLLOWER
I20260709 12:26:56.594776 29790 raft_consensus.cc:740] T 8627b58b4f42491597125eb45f8d3b61 P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6610c00ffb948ce9c34f41dec3d49e8, State: Initialized, Role: FOLLOWER
I20260709 12:26:56.595672 29794 raft_consensus.cc:359] T 742beffede4f407d8a72909d2b147e6d P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:26:56.596570 29794 raft_consensus.cc:385] T 742beffede4f407d8a72909d2b147e6d P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:56.595772 29790 consensus_queue.cc:260] T 8627b58b4f42491597125eb45f8d3b61 P c6610c00ffb948ce9c34f41dec3d49e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:26:56.597066 29596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ae3b542012d4f35a9c363eee22976ec" candidate_uuid: "584dac86183a41d9b3bea1c2085e62fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d73d92fb35234d16824c55689b5fce28" is_pre_election: true
I20260709 12:26:56.597615 29804 leader_election.cc:290] T 7ae3b542012d4f35a9c363eee22976ec P 584dac86183a41d9b3bea1c2085e62fa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825), d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209)
I20260709 12:26:56.597713 29596 raft_consensus.cc:2468] T 7ae3b542012d4f35a9c363eee22976ec P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 584dac86183a41d9b3bea1c2085e62fa in term 0.
I20260709 12:26:56.599237 29485 leader_election.cc:304] T 7ae3b542012d4f35a9c363eee22976ec P 584dac86183a41d9b3bea1c2085e62fa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 584dac86183a41d9b3bea1c2085e62fa, d73d92fb35234d16824c55689b5fce28; no voters: 
I20260709 12:26:56.600200 29804 raft_consensus.cc:2804] T 7ae3b542012d4f35a9c363eee22976ec P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:26:56.600582 29804 raft_consensus.cc:493] T 7ae3b542012d4f35a9c363eee22976ec P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:56.600914 29804 raft_consensus.cc:3060] T 7ae3b542012d4f35a9c363eee22976ec P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:56.606877 29790 ts_tablet_manager.cc:1434] T 8627b58b4f42491597125eb45f8d3b61 P c6610c00ffb948ce9c34f41dec3d49e8: Time spent starting tablet: real 0.017s	user 0.006s	sys 0.001s
I20260709 12:26:56.595624 29793 consensus_queue.cc:260] T d21ab1641501475b829863275dc8182a P d73d92fb35234d16824c55689b5fce28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:26:56.609817 29804 raft_consensus.cc:515] T 7ae3b542012d4f35a9c363eee22976ec P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:26:56.613725 29671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ae3b542012d4f35a9c363eee22976ec" candidate_uuid: "584dac86183a41d9b3bea1c2085e62fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" is_pre_election: true
I20260709 12:26:56.614703 29671 raft_consensus.cc:2468] T 7ae3b542012d4f35a9c363eee22976ec P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 584dac86183a41d9b3bea1c2085e62fa in term 0.
I20260709 12:26:56.615199 29596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ae3b542012d4f35a9c363eee22976ec" candidate_uuid: "584dac86183a41d9b3bea1c2085e62fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d73d92fb35234d16824c55689b5fce28"
I20260709 12:26:56.615787 29596 raft_consensus.cc:3060] T 7ae3b542012d4f35a9c363eee22976ec P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:56.618014 29794 raft_consensus.cc:740] T 742beffede4f407d8a72909d2b147e6d P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d13c58ad8ac441c4a9b396eb04b1a476, State: Initialized, Role: FOLLOWER
I20260709 12:26:56.619676 29794 consensus_queue.cc:260] T 742beffede4f407d8a72909d2b147e6d P d13c58ad8ac441c4a9b396eb04b1a476 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:26:56.623155 29790 tablet_bootstrap.cc:492] T fb6e78ccc01c4707af34cddf81619eab P c6610c00ffb948ce9c34f41dec3d49e8: Bootstrap starting.
I20260709 12:26:56.624384 29596 raft_consensus.cc:2468] T 7ae3b542012d4f35a9c363eee22976ec P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 584dac86183a41d9b3bea1c2085e62fa in term 1.
I20260709 12:26:56.625769 29485 leader_election.cc:304] T 7ae3b542012d4f35a9c363eee22976ec P 584dac86183a41d9b3bea1c2085e62fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 584dac86183a41d9b3bea1c2085e62fa, d73d92fb35234d16824c55689b5fce28; no voters: 
I20260709 12:26:56.627724 29794 ts_tablet_manager.cc:1434] T 742beffede4f407d8a72909d2b147e6d P d13c58ad8ac441c4a9b396eb04b1a476: Time spent starting tablet: real 0.035s	user 0.006s	sys 0.001s
I20260709 12:26:56.629202 29794 tablet_bootstrap.cc:492] T cdf706ffabc34902ad3d1bf2cee7889e P d13c58ad8ac441c4a9b396eb04b1a476: Bootstrap starting.
I20260709 12:26:56.612684 29793 ts_tablet_manager.cc:1434] T d21ab1641501475b829863275dc8182a P d73d92fb35234d16824c55689b5fce28: Time spent starting tablet: real 0.022s	user 0.008s	sys 0.000s
I20260709 12:26:56.630861 29793 tablet_bootstrap.cc:492] T cdf706ffabc34902ad3d1bf2cee7889e P d73d92fb35234d16824c55689b5fce28: Bootstrap starting.
I20260709 12:26:56.636853 29792 tablet_bootstrap.cc:492] T 742beffede4f407d8a72909d2b147e6d P 584dac86183a41d9b3bea1c2085e62fa: No bootstrap required, opened a new log
I20260709 12:26:56.637468 29792 ts_tablet_manager.cc:1403] T 742beffede4f407d8a72909d2b147e6d P 584dac86183a41d9b3bea1c2085e62fa: Time spent bootstrapping tablet: real 0.081s	user 0.021s	sys 0.031s
I20260709 12:26:56.639410 29804 leader_election.cc:290] T 7ae3b542012d4f35a9c363eee22976ec P 584dac86183a41d9b3bea1c2085e62fa [CANDIDATE]: Term 1 election: Requested vote from peers d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825), d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209)
I20260709 12:26:56.639317 29671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ae3b542012d4f35a9c363eee22976ec" candidate_uuid: "584dac86183a41d9b3bea1c2085e62fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d13c58ad8ac441c4a9b396eb04b1a476"
I20260709 12:26:56.640655 29804 raft_consensus.cc:2804] T 7ae3b542012d4f35a9c363eee22976ec P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:26:56.640678 29671 raft_consensus.cc:3060] T 7ae3b542012d4f35a9c363eee22976ec P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:56.641250 29804 raft_consensus.cc:697] T 7ae3b542012d4f35a9c363eee22976ec P 584dac86183a41d9b3bea1c2085e62fa [term 1 LEADER]: Becoming Leader. State: Replica: 584dac86183a41d9b3bea1c2085e62fa, State: Running, Role: LEADER
I20260709 12:26:56.640807 29792 raft_consensus.cc:359] T 742beffede4f407d8a72909d2b147e6d P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:26:56.641775 29792 raft_consensus.cc:385] T 742beffede4f407d8a72909d2b147e6d P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:56.642132 29792 raft_consensus.cc:740] T 742beffede4f407d8a72909d2b147e6d P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 584dac86183a41d9b3bea1c2085e62fa, State: Initialized, Role: FOLLOWER
I20260709 12:26:56.642169 29804 consensus_queue.cc:237] T 7ae3b542012d4f35a9c363eee22976ec P 584dac86183a41d9b3bea1c2085e62fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:26:56.643190 29792 consensus_queue.cc:260] T 742beffede4f407d8a72909d2b147e6d P 584dac86183a41d9b3bea1c2085e62fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:26:56.648903 29790 tablet_bootstrap.cc:654] T fb6e78ccc01c4707af34cddf81619eab P c6610c00ffb948ce9c34f41dec3d49e8: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:56.651731 29671 raft_consensus.cc:2468] T 7ae3b542012d4f35a9c363eee22976ec P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 584dac86183a41d9b3bea1c2085e62fa in term 1.
I20260709 12:26:56.651450 29793 tablet_bootstrap.cc:654] T cdf706ffabc34902ad3d1bf2cee7889e P d73d92fb35234d16824c55689b5fce28: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:56.653788 29792 ts_tablet_manager.cc:1434] T 742beffede4f407d8a72909d2b147e6d P 584dac86183a41d9b3bea1c2085e62fa: Time spent starting tablet: real 0.016s	user 0.010s	sys 0.007s
I20260709 12:26:56.653915 29794 tablet_bootstrap.cc:654] T cdf706ffabc34902ad3d1bf2cee7889e P d13c58ad8ac441c4a9b396eb04b1a476: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:56.656545 29792 tablet_bootstrap.cc:492] T 592801934cad4789b9ce8dc09f1470bd P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap starting.
I20260709 12:26:56.665073 29792 tablet_bootstrap.cc:654] T 592801934cad4789b9ce8dc09f1470bd P 584dac86183a41d9b3bea1c2085e62fa: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:56.666812 29804 raft_consensus.cc:493] T fcb6a7f720b14dc98e821f663357a0bb P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:56.667352 29804 raft_consensus.cc:515] T fcb6a7f720b14dc98e821f663357a0bb P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:26:56.668361 29420 catalog_manager.cc:5697] T 7ae3b542012d4f35a9c363eee22976ec P 584dac86183a41d9b3bea1c2085e62fa reported cstate change: term changed from 0 to 1, leader changed from <none> to 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65). New cstate: current_term: 1 leader_uuid: "584dac86183a41d9b3bea1c2085e62fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:56.670653 29596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fcb6a7f720b14dc98e821f663357a0bb" candidate_uuid: "584dac86183a41d9b3bea1c2085e62fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d73d92fb35234d16824c55689b5fce28" is_pre_election: true
I20260709 12:26:56.671330 29596 raft_consensus.cc:2468] T fcb6a7f720b14dc98e821f663357a0bb P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 584dac86183a41d9b3bea1c2085e62fa in term 0.
I20260709 12:26:56.673017 29485 leader_election.cc:304] T fcb6a7f720b14dc98e821f663357a0bb P 584dac86183a41d9b3bea1c2085e62fa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 584dac86183a41d9b3bea1c2085e62fa, d73d92fb35234d16824c55689b5fce28; no voters: 
I20260709 12:26:56.674052 29835 raft_consensus.cc:2804] T fcb6a7f720b14dc98e821f663357a0bb P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:26:56.674539 29835 raft_consensus.cc:493] T fcb6a7f720b14dc98e821f663357a0bb P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:56.674772 29790 tablet_bootstrap.cc:492] T fb6e78ccc01c4707af34cddf81619eab P c6610c00ffb948ce9c34f41dec3d49e8: No bootstrap required, opened a new log
I20260709 12:26:56.674933 29804 leader_election.cc:290] T fcb6a7f720b14dc98e821f663357a0bb P 584dac86183a41d9b3bea1c2085e62fa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825), d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209)
I20260709 12:26:56.675840 29671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fcb6a7f720b14dc98e821f663357a0bb" candidate_uuid: "584dac86183a41d9b3bea1c2085e62fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" is_pre_election: true
I20260709 12:26:56.676524 29790 ts_tablet_manager.cc:1403] T fb6e78ccc01c4707af34cddf81619eab P c6610c00ffb948ce9c34f41dec3d49e8: Time spent bootstrapping tablet: real 0.054s	user 0.016s	sys 0.012s
I20260709 12:26:56.676222 29794 tablet_bootstrap.cc:492] T cdf706ffabc34902ad3d1bf2cee7889e P d13c58ad8ac441c4a9b396eb04b1a476: No bootstrap required, opened a new log
I20260709 12:26:56.676748 29671 raft_consensus.cc:2468] T fcb6a7f720b14dc98e821f663357a0bb P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 584dac86183a41d9b3bea1c2085e62fa in term 0.
I20260709 12:26:56.677336 29794 ts_tablet_manager.cc:1403] T cdf706ffabc34902ad3d1bf2cee7889e P d13c58ad8ac441c4a9b396eb04b1a476: Time spent bootstrapping tablet: real 0.048s	user 0.013s	sys 0.007s
I20260709 12:26:56.678673 29793 tablet_bootstrap.cc:492] T cdf706ffabc34902ad3d1bf2cee7889e P d73d92fb35234d16824c55689b5fce28: No bootstrap required, opened a new log
I20260709 12:26:56.674966 29835 raft_consensus.cc:3060] T fcb6a7f720b14dc98e821f663357a0bb P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:56.681174 29794 raft_consensus.cc:359] T cdf706ffabc34902ad3d1bf2cee7889e P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:26:56.682065 29794 raft_consensus.cc:385] T cdf706ffabc34902ad3d1bf2cee7889e P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:56.682397 29794 raft_consensus.cc:740] T cdf706ffabc34902ad3d1bf2cee7889e P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d13c58ad8ac441c4a9b396eb04b1a476, State: Initialized, Role: FOLLOWER
I20260709 12:26:56.683455 29794 consensus_queue.cc:260] T cdf706ffabc34902ad3d1bf2cee7889e P d13c58ad8ac441c4a9b396eb04b1a476 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:26:56.684125 29792 tablet_bootstrap.cc:492] T 592801934cad4789b9ce8dc09f1470bd P 584dac86183a41d9b3bea1c2085e62fa: No bootstrap required, opened a new log
I20260709 12:26:56.684041 29790 raft_consensus.cc:359] T fb6e78ccc01c4707af34cddf81619eab P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:26:56.684845 29792 ts_tablet_manager.cc:1403] T 592801934cad4789b9ce8dc09f1470bd P 584dac86183a41d9b3bea1c2085e62fa: Time spent bootstrapping tablet: real 0.029s	user 0.015s	sys 0.006s
I20260709 12:26:56.687141 29790 raft_consensus.cc:385] T fb6e78ccc01c4707af34cddf81619eab P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:56.687493 29790 raft_consensus.cc:740] T fb6e78ccc01c4707af34cddf81619eab P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6610c00ffb948ce9c34f41dec3d49e8, State: Initialized, Role: FOLLOWER
I20260709 12:26:56.687736 29793 ts_tablet_manager.cc:1403] T cdf706ffabc34902ad3d1bf2cee7889e P d73d92fb35234d16824c55689b5fce28: Time spent bootstrapping tablet: real 0.057s	user 0.010s	sys 0.004s
I20260709 12:26:56.688614 29792 raft_consensus.cc:359] T 592801934cad4789b9ce8dc09f1470bd P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:26:56.689443 29792 raft_consensus.cc:385] T 592801934cad4789b9ce8dc09f1470bd P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:56.688412 29790 consensus_queue.cc:260] T fb6e78ccc01c4707af34cddf81619eab P c6610c00ffb948ce9c34f41dec3d49e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:26:56.689884 29792 raft_consensus.cc:740] T 592801934cad4789b9ce8dc09f1470bd P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 584dac86183a41d9b3bea1c2085e62fa, State: Initialized, Role: FOLLOWER
I20260709 12:26:56.690243 29794 ts_tablet_manager.cc:1434] T cdf706ffabc34902ad3d1bf2cee7889e P d13c58ad8ac441c4a9b396eb04b1a476: Time spent starting tablet: real 0.013s	user 0.007s	sys 0.000s
I20260709 12:26:56.690845 29792 consensus_queue.cc:260] T 592801934cad4789b9ce8dc09f1470bd P 584dac86183a41d9b3bea1c2085e62fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:26:56.691637 29794 tablet_bootstrap.cc:492] T d21ab1641501475b829863275dc8182a P d13c58ad8ac441c4a9b396eb04b1a476: Bootstrap starting.
I20260709 12:26:56.693513 29792 ts_tablet_manager.cc:1434] T 592801934cad4789b9ce8dc09f1470bd P 584dac86183a41d9b3bea1c2085e62fa: Time spent starting tablet: real 0.008s	user 0.008s	sys 0.000s
I20260709 12:26:56.693418 29793 raft_consensus.cc:359] T cdf706ffabc34902ad3d1bf2cee7889e P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:26:56.694554 29793 raft_consensus.cc:385] T cdf706ffabc34902ad3d1bf2cee7889e P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:56.694826 29792 tablet_bootstrap.cc:492] T d42259e436d241b29fb4ded16d9b6455 P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap starting.
I20260709 12:26:56.695858 29793 raft_consensus.cc:740] T cdf706ffabc34902ad3d1bf2cee7889e P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d73d92fb35234d16824c55689b5fce28, State: Initialized, Role: FOLLOWER
I20260709 12:26:56.690169 29835 raft_consensus.cc:515] T fcb6a7f720b14dc98e821f663357a0bb P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:26:56.697599 29790 ts_tablet_manager.cc:1434] T fb6e78ccc01c4707af34cddf81619eab P c6610c00ffb948ce9c34f41dec3d49e8: Time spent starting tablet: real 0.021s	user 0.008s	sys 0.000s
I20260709 12:26:56.699095 29596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fcb6a7f720b14dc98e821f663357a0bb" candidate_uuid: "584dac86183a41d9b3bea1c2085e62fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d73d92fb35234d16824c55689b5fce28"
I20260709 12:26:56.699601 29790 tablet_bootstrap.cc:492] T 4685c2e824c14fc1bcd45c16169798cc P c6610c00ffb948ce9c34f41dec3d49e8: Bootstrap starting.
I20260709 12:26:56.699843 29794 tablet_bootstrap.cc:654] T d21ab1641501475b829863275dc8182a P d13c58ad8ac441c4a9b396eb04b1a476: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:56.699900 29596 raft_consensus.cc:3060] T fcb6a7f720b14dc98e821f663357a0bb P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:56.704301 29792 tablet_bootstrap.cc:654] T d42259e436d241b29fb4ded16d9b6455 P 584dac86183a41d9b3bea1c2085e62fa: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:56.709920 29835 leader_election.cc:290] T fcb6a7f720b14dc98e821f663357a0bb P 584dac86183a41d9b3bea1c2085e62fa [CANDIDATE]: Term 1 election: Requested vote from peers d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825), d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209)
I20260709 12:26:56.710649 29671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fcb6a7f720b14dc98e821f663357a0bb" candidate_uuid: "584dac86183a41d9b3bea1c2085e62fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d13c58ad8ac441c4a9b396eb04b1a476"
I20260709 12:26:56.711344 29671 raft_consensus.cc:3060] T fcb6a7f720b14dc98e821f663357a0bb P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:56.710995 29793 consensus_queue.cc:260] T cdf706ffabc34902ad3d1bf2cee7889e P d73d92fb35234d16824c55689b5fce28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:26:56.713829 29793 ts_tablet_manager.cc:1434] T cdf706ffabc34902ad3d1bf2cee7889e P d73d92fb35234d16824c55689b5fce28: Time spent starting tablet: real 0.026s	user 0.006s	sys 0.000s
I20260709 12:26:56.714516 29596 raft_consensus.cc:2468] T fcb6a7f720b14dc98e821f663357a0bb P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 584dac86183a41d9b3bea1c2085e62fa in term 1.
I20260709 12:26:56.715993 29485 leader_election.cc:304] T fcb6a7f720b14dc98e821f663357a0bb P 584dac86183a41d9b3bea1c2085e62fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 584dac86183a41d9b3bea1c2085e62fa, d73d92fb35234d16824c55689b5fce28; no voters: 
I20260709 12:26:56.717788 29835 raft_consensus.cc:2804] T fcb6a7f720b14dc98e821f663357a0bb P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:26:56.718575 29835 raft_consensus.cc:697] T fcb6a7f720b14dc98e821f663357a0bb P 584dac86183a41d9b3bea1c2085e62fa [term 1 LEADER]: Becoming Leader. State: Replica: 584dac86183a41d9b3bea1c2085e62fa, State: Running, Role: LEADER
I20260709 12:26:56.720078 29835 consensus_queue.cc:237] T fcb6a7f720b14dc98e821f663357a0bb P 584dac86183a41d9b3bea1c2085e62fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:26:56.724189 29793 tablet_bootstrap.cc:492] T fb6e78ccc01c4707af34cddf81619eab P d73d92fb35234d16824c55689b5fce28: Bootstrap starting.
I20260709 12:26:56.723014 29808 raft_consensus.cc:493] T d21ab1641501475b829863275dc8182a P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:56.730973 29808 raft_consensus.cc:515] T d21ab1641501475b829863275dc8182a P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:26:56.735320 29670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21ab1641501475b829863275dc8182a" candidate_uuid: "d73d92fb35234d16824c55689b5fce28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" is_pre_election: true
I20260709 12:26:56.737185 29522 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21ab1641501475b829863275dc8182a" candidate_uuid: "d73d92fb35234d16824c55689b5fce28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "584dac86183a41d9b3bea1c2085e62fa" is_pre_election: true
I20260709 12:26:56.738153 29522 raft_consensus.cc:2468] T d21ab1641501475b829863275dc8182a P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d73d92fb35234d16824c55689b5fce28 in term 0.
I20260709 12:26:56.738623 29793 tablet_bootstrap.cc:654] T fb6e78ccc01c4707af34cddf81619eab P d73d92fb35234d16824c55689b5fce28: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:56.725773 29671 raft_consensus.cc:2468] T fcb6a7f720b14dc98e821f663357a0bb P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 584dac86183a41d9b3bea1c2085e62fa in term 1.
W20260709 12:26:56.740263 29561 leader_election.cc:343] T d21ab1641501475b829863275dc8182a P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:56.727432 29794 tablet_bootstrap.cc:492] T d21ab1641501475b829863275dc8182a P d13c58ad8ac441c4a9b396eb04b1a476: No bootstrap required, opened a new log
I20260709 12:26:56.741243 29794 ts_tablet_manager.cc:1403] T d21ab1641501475b829863275dc8182a P d13c58ad8ac441c4a9b396eb04b1a476: Time spent bootstrapping tablet: real 0.050s	user 0.009s	sys 0.017s
I20260709 12:26:56.742251 29808 leader_election.cc:290] T d21ab1641501475b829863275dc8182a P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033), d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209)
I20260709 12:26:56.721046 29790 tablet_bootstrap.cc:654] T 4685c2e824c14fc1bcd45c16169798cc P c6610c00ffb948ce9c34f41dec3d49e8: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:56.744352 29794 raft_consensus.cc:359] T d21ab1641501475b829863275dc8182a P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:26:56.745077 29794 raft_consensus.cc:385] T d21ab1641501475b829863275dc8182a P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:56.745352 29794 raft_consensus.cc:740] T d21ab1641501475b829863275dc8182a P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d13c58ad8ac441c4a9b396eb04b1a476, State: Initialized, Role: FOLLOWER
I20260709 12:26:56.742924 29560 leader_election.cc:304] T d21ab1641501475b829863275dc8182a P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 584dac86183a41d9b3bea1c2085e62fa, d73d92fb35234d16824c55689b5fce28; no voters: d13c58ad8ac441c4a9b396eb04b1a476
I20260709 12:26:56.746207 29808 raft_consensus.cc:2804] T d21ab1641501475b829863275dc8182a P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:26:56.746095 29794 consensus_queue.cc:260] T d21ab1641501475b829863275dc8182a P d13c58ad8ac441c4a9b396eb04b1a476 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:26:56.746851 29808 raft_consensus.cc:493] T d21ab1641501475b829863275dc8182a P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:56.747555 29808 raft_consensus.cc:3060] T d21ab1641501475b829863275dc8182a P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:56.748811 29794 ts_tablet_manager.cc:1434] T d21ab1641501475b829863275dc8182a P d13c58ad8ac441c4a9b396eb04b1a476: Time spent starting tablet: real 0.007s	user 0.006s	sys 0.001s
I20260709 12:26:56.753602 29794 tablet_bootstrap.cc:492] T 55061e62184b45f0bb1726adc1a4af81 P d13c58ad8ac441c4a9b396eb04b1a476: Bootstrap starting.
I20260709 12:26:56.756222 29792 tablet_bootstrap.cc:492] T d42259e436d241b29fb4ded16d9b6455 P 584dac86183a41d9b3bea1c2085e62fa: No bootstrap required, opened a new log
I20260709 12:26:56.756709 29792 ts_tablet_manager.cc:1403] T d42259e436d241b29fb4ded16d9b6455 P 584dac86183a41d9b3bea1c2085e62fa: Time spent bootstrapping tablet: real 0.062s	user 0.020s	sys 0.011s
I20260709 12:26:56.759541 29794 tablet_bootstrap.cc:654] T 55061e62184b45f0bb1726adc1a4af81 P d13c58ad8ac441c4a9b396eb04b1a476: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:56.759994 29792 raft_consensus.cc:359] T d42259e436d241b29fb4ded16d9b6455 P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:26:56.760816 29792 raft_consensus.cc:385] T d42259e436d241b29fb4ded16d9b6455 P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:56.762151 29792 raft_consensus.cc:740] T d42259e436d241b29fb4ded16d9b6455 P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 584dac86183a41d9b3bea1c2085e62fa, State: Initialized, Role: FOLLOWER
I20260709 12:26:56.763005 29792 consensus_queue.cc:260] T d42259e436d241b29fb4ded16d9b6455 P 584dac86183a41d9b3bea1c2085e62fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:26:56.765924 29792 ts_tablet_manager.cc:1434] T d42259e436d241b29fb4ded16d9b6455 P 584dac86183a41d9b3bea1c2085e62fa: Time spent starting tablet: real 0.009s	user 0.007s	sys 0.000s
I20260709 12:26:56.760790 29808 raft_consensus.cc:515] T d21ab1641501475b829863275dc8182a P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:26:56.768079 29792 tablet_bootstrap.cc:492] T 3e9451c3797c4496a66dfdb3e9fb5e38 P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap starting.
I20260709 12:26:56.770010 29835 raft_consensus.cc:493] T 742beffede4f407d8a72909d2b147e6d P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:56.770673 29522 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21ab1641501475b829863275dc8182a" candidate_uuid: "d73d92fb35234d16824c55689b5fce28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "584dac86183a41d9b3bea1c2085e62fa"
I20260709 12:26:56.770561 29835 raft_consensus.cc:515] T 742beffede4f407d8a72909d2b147e6d P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:26:56.771062 29671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d21ab1641501475b829863275dc8182a" candidate_uuid: "d73d92fb35234d16824c55689b5fce28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d13c58ad8ac441c4a9b396eb04b1a476"
I20260709 12:26:56.771382 29522 raft_consensus.cc:3060] T d21ab1641501475b829863275dc8182a P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:56.771714 29671 raft_consensus.cc:3060] T d21ab1641501475b829863275dc8182a P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:56.772840 29835 leader_election.cc:290] T 742beffede4f407d8a72909d2b147e6d P 584dac86183a41d9b3bea1c2085e62fa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209), c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895)
I20260709 12:26:56.775636 29670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "742beffede4f407d8a72909d2b147e6d" candidate_uuid: "584dac86183a41d9b3bea1c2085e62fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" is_pre_election: true
I20260709 12:26:56.775944 29745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "742beffede4f407d8a72909d2b147e6d" candidate_uuid: "584dac86183a41d9b3bea1c2085e62fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" is_pre_election: true
I20260709 12:26:56.776388 29670 raft_consensus.cc:2468] T 742beffede4f407d8a72909d2b147e6d P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 584dac86183a41d9b3bea1c2085e62fa in term 0.
W20260709 12:26:56.777346 29486 leader_election.cc:343] T 742beffede4f407d8a72909d2b147e6d P 584dac86183a41d9b3bea1c2085e62fa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:56.778113 29486 leader_election.cc:304] T 742beffede4f407d8a72909d2b147e6d P 584dac86183a41d9b3bea1c2085e62fa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 584dac86183a41d9b3bea1c2085e62fa, d13c58ad8ac441c4a9b396eb04b1a476; no voters: c6610c00ffb948ce9c34f41dec3d49e8
I20260709 12:26:56.781338 29835 raft_consensus.cc:2804] T 742beffede4f407d8a72909d2b147e6d P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:26:56.781728 29835 raft_consensus.cc:493] T 742beffede4f407d8a72909d2b147e6d P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:56.782135 29835 raft_consensus.cc:3060] T 742beffede4f407d8a72909d2b147e6d P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:56.782544 29671 raft_consensus.cc:2468] T d21ab1641501475b829863275dc8182a P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d73d92fb35234d16824c55689b5fce28 in term 1.
I20260709 12:26:56.782806 29793 tablet_bootstrap.cc:492] T fb6e78ccc01c4707af34cddf81619eab P d73d92fb35234d16824c55689b5fce28: No bootstrap required, opened a new log
I20260709 12:26:56.783420 29793 ts_tablet_manager.cc:1403] T fb6e78ccc01c4707af34cddf81619eab P d73d92fb35234d16824c55689b5fce28: Time spent bootstrapping tablet: real 0.064s	user 0.007s	sys 0.007s
I20260709 12:26:56.784072 29561 leader_election.cc:304] T d21ab1641501475b829863275dc8182a P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d13c58ad8ac441c4a9b396eb04b1a476, d73d92fb35234d16824c55689b5fce28; no voters: 
I20260709 12:26:56.784435 29522 raft_consensus.cc:2468] T d21ab1641501475b829863275dc8182a P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d73d92fb35234d16824c55689b5fce28 in term 1.
I20260709 12:26:56.786528 29793 raft_consensus.cc:359] T fb6e78ccc01c4707af34cddf81619eab P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:26:56.787262 29793 raft_consensus.cc:385] T fb6e78ccc01c4707af34cddf81619eab P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:56.787568 29793 raft_consensus.cc:740] T fb6e78ccc01c4707af34cddf81619eab P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d73d92fb35234d16824c55689b5fce28, State: Initialized, Role: FOLLOWER
I20260709 12:26:56.788360 29793 consensus_queue.cc:260] T fb6e78ccc01c4707af34cddf81619eab P d73d92fb35234d16824c55689b5fce28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:26:56.790269 29835 raft_consensus.cc:515] T 742beffede4f407d8a72909d2b147e6d P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:26:56.794020 29792 tablet_bootstrap.cc:654] T 3e9451c3797c4496a66dfdb3e9fb5e38 P 584dac86183a41d9b3bea1c2085e62fa: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:56.795397 29808 leader_election.cc:290] T d21ab1641501475b829863275dc8182a P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 1 election: Requested vote from peers 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033), d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209)
I20260709 12:26:56.794405 29835 leader_election.cc:290] T 742beffede4f407d8a72909d2b147e6d P 584dac86183a41d9b3bea1c2085e62fa [CANDIDATE]: Term 1 election: Requested vote from peers d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209), c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895)
I20260709 12:26:56.795637 29745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "742beffede4f407d8a72909d2b147e6d" candidate_uuid: "584dac86183a41d9b3bea1c2085e62fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6610c00ffb948ce9c34f41dec3d49e8"
W20260709 12:26:56.798691 29486 leader_election.cc:343] T 742beffede4f407d8a72909d2b147e6d P 584dac86183a41d9b3bea1c2085e62fa [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:56.799676 29851 raft_consensus.cc:2804] T d21ab1641501475b829863275dc8182a P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:26:56.800765 29851 raft_consensus.cc:697] T d21ab1641501475b829863275dc8182a P d73d92fb35234d16824c55689b5fce28 [term 1 LEADER]: Becoming Leader. State: Replica: d73d92fb35234d16824c55689b5fce28, State: Running, Role: LEADER
I20260709 12:26:56.801841 29851 consensus_queue.cc:237] T d21ab1641501475b829863275dc8182a P d73d92fb35234d16824c55689b5fce28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:26:56.803848 29793 ts_tablet_manager.cc:1434] T fb6e78ccc01c4707af34cddf81619eab P d73d92fb35234d16824c55689b5fce28: Time spent starting tablet: real 0.020s	user 0.007s	sys 0.003s
I20260709 12:26:56.804062 29798 raft_consensus.cc:493] T 3e9451c3797c4496a66dfdb3e9fb5e38 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:56.805150 29798 raft_consensus.cc:515] T 3e9451c3797c4496a66dfdb3e9fb5e38 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:26:56.806277 29793 tablet_bootstrap.cc:492] T 592801934cad4789b9ce8dc09f1470bd P d73d92fb35234d16824c55689b5fce28: Bootstrap starting.
I20260709 12:26:56.807337 29798 leader_election.cc:290] T 3e9451c3797c4496a66dfdb3e9fb5e38 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895), 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033)
I20260709 12:26:56.808614 29522 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e9451c3797c4496a66dfdb3e9fb5e38" candidate_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "584dac86183a41d9b3bea1c2085e62fa" is_pre_election: true
W20260709 12:26:56.810776 29634 leader_election.cc:343] T 3e9451c3797c4496a66dfdb3e9fb5e38 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:56.811264 29852 raft_consensus.cc:493] T fb6e78ccc01c4707af34cddf81619eab P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:56.811748 29852 raft_consensus.cc:515] T fb6e78ccc01c4707af34cddf81619eab P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:26:56.809108 29745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e9451c3797c4496a66dfdb3e9fb5e38" candidate_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" is_pre_election: true
I20260709 12:26:56.804476 29671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "742beffede4f407d8a72909d2b147e6d" candidate_uuid: "584dac86183a41d9b3bea1c2085e62fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d13c58ad8ac441c4a9b396eb04b1a476"
I20260709 12:26:56.816538 29671 raft_consensus.cc:3060] T 742beffede4f407d8a72909d2b147e6d P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:56.821795 29794 tablet_bootstrap.cc:492] T 55061e62184b45f0bb1726adc1a4af81 P d13c58ad8ac441c4a9b396eb04b1a476: No bootstrap required, opened a new log
I20260709 12:26:56.822353 29794 ts_tablet_manager.cc:1403] T 55061e62184b45f0bb1726adc1a4af81 P d13c58ad8ac441c4a9b396eb04b1a476: Time spent bootstrapping tablet: real 0.069s	user 0.018s	sys 0.008s
I20260709 12:26:56.815526 29745 raft_consensus.cc:2468] T 3e9451c3797c4496a66dfdb3e9fb5e38 P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d13c58ad8ac441c4a9b396eb04b1a476 in term 0.
I20260709 12:26:56.825240 29794 raft_consensus.cc:359] T 55061e62184b45f0bb1726adc1a4af81 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:26:56.825237 29420 catalog_manager.cc:5697] T fcb6a7f720b14dc98e821f663357a0bb P 584dac86183a41d9b3bea1c2085e62fa reported cstate change: term changed from 0 to 1, leader changed from <none> to 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65). New cstate: current_term: 1 leader_uuid: "584dac86183a41d9b3bea1c2085e62fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:56.826141 29794 raft_consensus.cc:385] T 55061e62184b45f0bb1726adc1a4af81 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:56.825812 29635 leader_election.cc:304] T 3e9451c3797c4496a66dfdb3e9fb5e38 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c6610c00ffb948ce9c34f41dec3d49e8, d13c58ad8ac441c4a9b396eb04b1a476; no voters: 584dac86183a41d9b3bea1c2085e62fa
I20260709 12:26:56.826463 29794 raft_consensus.cc:740] T 55061e62184b45f0bb1726adc1a4af81 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d13c58ad8ac441c4a9b396eb04b1a476, State: Initialized, Role: FOLLOWER
I20260709 12:26:56.827297 29794 consensus_queue.cc:260] T 55061e62184b45f0bb1726adc1a4af81 P d13c58ad8ac441c4a9b396eb04b1a476 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:26:56.827688 29798 raft_consensus.cc:2804] T 3e9451c3797c4496a66dfdb3e9fb5e38 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:26:56.828835 29798 raft_consensus.cc:493] T 3e9451c3797c4496a66dfdb3e9fb5e38 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:56.829638 29798 raft_consensus.cc:3060] T 3e9451c3797c4496a66dfdb3e9fb5e38 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:56.832875 29799 raft_consensus.cc:493] T fb6e78ccc01c4707af34cddf81619eab P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:56.833487 29799 raft_consensus.cc:515] T fb6e78ccc01c4707af34cddf81619eab P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:26:56.835908 29799 leader_election.cc:290] T fb6e78ccc01c4707af34cddf81619eab P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033), d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825)
I20260709 12:26:56.838496 29596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb6e78ccc01c4707af34cddf81619eab" candidate_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d73d92fb35234d16824c55689b5fce28" is_pre_election: true
I20260709 12:26:56.839677 29596 raft_consensus.cc:2468] T fb6e78ccc01c4707af34cddf81619eab P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6610c00ffb948ce9c34f41dec3d49e8 in term 0.
I20260709 12:26:56.841884 29522 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb6e78ccc01c4707af34cddf81619eab" candidate_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "584dac86183a41d9b3bea1c2085e62fa" is_pre_election: true
I20260709 12:26:56.842881 29522 raft_consensus.cc:2468] T fb6e78ccc01c4707af34cddf81619eab P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6610c00ffb948ce9c34f41dec3d49e8 in term 0.
I20260709 12:26:56.843237 29835 raft_consensus.cc:493] T d42259e436d241b29fb4ded16d9b6455 P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:56.843780 29835 raft_consensus.cc:515] T d42259e436d241b29fb4ded16d9b6455 P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:26:56.844750 29708 leader_election.cc:304] T fb6e78ccc01c4707af34cddf81619eab P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 584dac86183a41d9b3bea1c2085e62fa, c6610c00ffb948ce9c34f41dec3d49e8; no voters: 
I20260709 12:26:56.845914 29835 leader_election.cc:290] T d42259e436d241b29fb4ded16d9b6455 P 584dac86183a41d9b3bea1c2085e62fa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825), d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209)
I20260709 12:26:56.846160 29799 raft_consensus.cc:2804] T fb6e78ccc01c4707af34cddf81619eab P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:26:56.846566 29799 raft_consensus.cc:493] T fb6e78ccc01c4707af34cddf81619eab P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:56.846926 29799 raft_consensus.cc:3060] T fb6e78ccc01c4707af34cddf81619eab P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:56.847663 29670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d42259e436d241b29fb4ded16d9b6455" candidate_uuid: "584dac86183a41d9b3bea1c2085e62fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" is_pre_election: true
I20260709 12:26:56.848361 29670 raft_consensus.cc:2468] T d42259e436d241b29fb4ded16d9b6455 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 584dac86183a41d9b3bea1c2085e62fa in term 0.
I20260709 12:26:56.849689 29486 leader_election.cc:304] T d42259e436d241b29fb4ded16d9b6455 P 584dac86183a41d9b3bea1c2085e62fa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 584dac86183a41d9b3bea1c2085e62fa, d13c58ad8ac441c4a9b396eb04b1a476; no voters: 
I20260709 12:26:56.850654 29835 raft_consensus.cc:2804] T d42259e436d241b29fb4ded16d9b6455 P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:26:56.851025 29835 raft_consensus.cc:493] T d42259e436d241b29fb4ded16d9b6455 P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:56.851342 29835 raft_consensus.cc:3060] T d42259e436d241b29fb4ded16d9b6455 P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:56.853591 29790 tablet_bootstrap.cc:492] T 4685c2e824c14fc1bcd45c16169798cc P c6610c00ffb948ce9c34f41dec3d49e8: No bootstrap required, opened a new log
I20260709 12:26:56.853932 29596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d42259e436d241b29fb4ded16d9b6455" candidate_uuid: "584dac86183a41d9b3bea1c2085e62fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d73d92fb35234d16824c55689b5fce28" is_pre_election: true
I20260709 12:26:56.854435 29790 ts_tablet_manager.cc:1403] T 4685c2e824c14fc1bcd45c16169798cc P c6610c00ffb948ce9c34f41dec3d49e8: Time spent bootstrapping tablet: real 0.156s	user 0.009s	sys 0.009s
W20260709 12:26:56.856698 29485 leader_election.cc:343] T d42259e436d241b29fb4ded16d9b6455 P 584dac86183a41d9b3bea1c2085e62fa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:56.863538 29745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb6e78ccc01c4707af34cddf81619eab" candidate_uuid: "d73d92fb35234d16824c55689b5fce28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" is_pre_election: true
I20260709 12:26:56.866350 29804 raft_consensus.cc:493] T 592801934cad4789b9ce8dc09f1470bd P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:56.867538 29792 tablet_bootstrap.cc:492] T 3e9451c3797c4496a66dfdb3e9fb5e38 P 584dac86183a41d9b3bea1c2085e62fa: No bootstrap required, opened a new log
I20260709 12:26:56.867173 29804 raft_consensus.cc:515] T 592801934cad4789b9ce8dc09f1470bd P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:26:56.868116 29792 ts_tablet_manager.cc:1403] T 3e9451c3797c4496a66dfdb3e9fb5e38 P 584dac86183a41d9b3bea1c2085e62fa: Time spent bootstrapping tablet: real 0.100s	user 0.022s	sys 0.004s
I20260709 12:26:56.869390 29799 raft_consensus.cc:515] T fb6e78ccc01c4707af34cddf81619eab P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:26:56.871070 29792 raft_consensus.cc:359] T 3e9451c3797c4496a66dfdb3e9fb5e38 P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:26:56.871335 29804 leader_election.cc:290] T 592801934cad4789b9ce8dc09f1470bd P 584dac86183a41d9b3bea1c2085e62fa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895), d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825)
I20260709 12:26:56.871397 29744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592801934cad4789b9ce8dc09f1470bd" candidate_uuid: "584dac86183a41d9b3bea1c2085e62fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" is_pre_election: true
I20260709 12:26:56.871834 29792 raft_consensus.cc:385] T 3e9451c3797c4496a66dfdb3e9fb5e38 P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:56.871469 29799 leader_election.cc:290] T fb6e78ccc01c4707af34cddf81619eab P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 1 election: Requested vote from peers 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033), d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825)
I20260709 12:26:56.874029 29745 raft_consensus.cc:2393] T fb6e78ccc01c4707af34cddf81619eab P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d73d92fb35234d16824c55689b5fce28 in current term 1: Already voted for candidate c6610c00ffb948ce9c34f41dec3d49e8 in this term.
I20260709 12:26:56.878036 29792 raft_consensus.cc:740] T 3e9451c3797c4496a66dfdb3e9fb5e38 P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 584dac86183a41d9b3bea1c2085e62fa, State: Initialized, Role: FOLLOWER
I20260709 12:26:56.878149 29744 raft_consensus.cc:2468] T 592801934cad4789b9ce8dc09f1470bd P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 584dac86183a41d9b3bea1c2085e62fa in term 0.
I20260709 12:26:56.879072 29792 consensus_queue.cc:260] T 3e9451c3797c4496a66dfdb3e9fb5e38 P 584dac86183a41d9b3bea1c2085e62fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:26:56.879992 29486 leader_election.cc:304] T 592801934cad4789b9ce8dc09f1470bd P 584dac86183a41d9b3bea1c2085e62fa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 584dac86183a41d9b3bea1c2085e62fa, c6610c00ffb948ce9c34f41dec3d49e8; no voters: 
I20260709 12:26:56.880807 29798 raft_consensus.cc:515] T 3e9451c3797c4496a66dfdb3e9fb5e38 P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:26:56.881647 29834 raft_consensus.cc:2804] T 592801934cad4789b9ce8dc09f1470bd P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:26:56.881911 29792 ts_tablet_manager.cc:1434] T 3e9451c3797c4496a66dfdb3e9fb5e38 P 584dac86183a41d9b3bea1c2085e62fa: Time spent starting tablet: real 0.013s	user 0.006s	sys 0.000s
I20260709 12:26:56.882146 29834 raft_consensus.cc:493] T 592801934cad4789b9ce8dc09f1470bd P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:56.882593 29834 raft_consensus.cc:3060] T 592801934cad4789b9ce8dc09f1470bd P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:56.882988 29792 tablet_bootstrap.cc:492] T 4685c2e824c14fc1bcd45c16169798cc P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap starting.
I20260709 12:26:56.885355 29521 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e9451c3797c4496a66dfdb3e9fb5e38" candidate_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "584dac86183a41d9b3bea1c2085e62fa"
I20260709 12:26:56.885598 29835 raft_consensus.cc:515] T d42259e436d241b29fb4ded16d9b6455 P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:26:56.886509 29521 raft_consensus.cc:3060] T 3e9451c3797c4496a66dfdb3e9fb5e38 P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:56.887920 29835 leader_election.cc:290] T d42259e436d241b29fb4ded16d9b6455 P 584dac86183a41d9b3bea1c2085e62fa [CANDIDATE]: Term 1 election: Requested vote from peers d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825), d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209)
I20260709 12:26:56.888513 29798 leader_election.cc:290] T 3e9451c3797c4496a66dfdb3e9fb5e38 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 1 election: Requested vote from peers c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895), 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033)
I20260709 12:26:56.890012 29792 tablet_bootstrap.cc:654] T 4685c2e824c14fc1bcd45c16169798cc P 584dac86183a41d9b3bea1c2085e62fa: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:56.893489 29794 ts_tablet_manager.cc:1434] T 55061e62184b45f0bb1726adc1a4af81 P d13c58ad8ac441c4a9b396eb04b1a476: Time spent starting tablet: real 0.071s	user 0.024s	sys 0.023s
I20260709 12:26:56.894521 29794 tablet_bootstrap.cc:492] T 43c7bfb33af74d4f912e5153e8a54b3e P d13c58ad8ac441c4a9b396eb04b1a476: Bootstrap starting.
I20260709 12:26:56.875584 29522 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb6e78ccc01c4707af34cddf81619eab" candidate_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "584dac86183a41d9b3bea1c2085e62fa"
W20260709 12:26:56.876504 29467 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:26:56.895071 29522 raft_consensus.cc:3060] T fb6e78ccc01c4707af34cddf81619eab P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:56.876888 29597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592801934cad4789b9ce8dc09f1470bd" candidate_uuid: "584dac86183a41d9b3bea1c2085e62fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d73d92fb35234d16824c55689b5fce28" is_pre_election: true
I20260709 12:26:56.898144 29799 raft_consensus.cc:493] T 8627b58b4f42491597125eb45f8d3b61 P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:56.876770 29790 raft_consensus.cc:359] T 4685c2e824c14fc1bcd45c16169798cc P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:26:56.898777 29799 raft_consensus.cc:515] T 8627b58b4f42491597125eb45f8d3b61 P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:26:56.899137 29790 raft_consensus.cc:385] T 4685c2e824c14fc1bcd45c16169798cc P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:56.899626 29790 raft_consensus.cc:740] T 4685c2e824c14fc1bcd45c16169798cc P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6610c00ffb948ce9c34f41dec3d49e8, State: Initialized, Role: FOLLOWER
I20260709 12:26:56.900905 29794 tablet_bootstrap.cc:654] T 43c7bfb33af74d4f912e5153e8a54b3e P d13c58ad8ac441c4a9b396eb04b1a476: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:56.900568 29790 consensus_queue.cc:260] T 4685c2e824c14fc1bcd45c16169798cc P c6610c00ffb948ce9c34f41dec3d49e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:26:56.904232 29596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb6e78ccc01c4707af34cddf81619eab" candidate_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d73d92fb35234d16824c55689b5fce28"
I20260709 12:26:56.905192 29745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e9451c3797c4496a66dfdb3e9fb5e38" candidate_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6610c00ffb948ce9c34f41dec3d49e8"
I20260709 12:26:56.907953 29745 raft_consensus.cc:3060] T 3e9451c3797c4496a66dfdb3e9fb5e38 P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:56.900996 29799 leader_election.cc:290] T 8627b58b4f42491597125eb45f8d3b61 P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209), 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033)
I20260709 12:26:56.906853 29670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8627b58b4f42491597125eb45f8d3b61" candidate_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" is_pre_election: true
I20260709 12:26:56.909381 29670 raft_consensus.cc:2468] T 8627b58b4f42491597125eb45f8d3b61 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6610c00ffb948ce9c34f41dec3d49e8 in term 0.
I20260709 12:26:56.909257 29669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d42259e436d241b29fb4ded16d9b6455" candidate_uuid: "584dac86183a41d9b3bea1c2085e62fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d13c58ad8ac441c4a9b396eb04b1a476"
I20260709 12:26:56.910466 29669 raft_consensus.cc:3060] T d42259e436d241b29fb4ded16d9b6455 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:56.911048 29709 leader_election.cc:304] T 8627b58b4f42491597125eb45f8d3b61 P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c6610c00ffb948ce9c34f41dec3d49e8, d13c58ad8ac441c4a9b396eb04b1a476; no voters: 
I20260709 12:26:56.912127 29799 raft_consensus.cc:2804] T 8627b58b4f42491597125eb45f8d3b61 P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:26:56.912484 29799 raft_consensus.cc:493] T 8627b58b4f42491597125eb45f8d3b61 P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:56.912814 29799 raft_consensus.cc:3060] T 8627b58b4f42491597125eb45f8d3b61 P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:56.907207 29597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d42259e436d241b29fb4ded16d9b6455" candidate_uuid: "584dac86183a41d9b3bea1c2085e62fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d73d92fb35234d16824c55689b5fce28"
I20260709 12:26:56.917507 29596 raft_consensus.cc:3060] T fb6e78ccc01c4707af34cddf81619eab P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:26:56.905563 29485 leader_election.cc:343] T 592801934cad4789b9ce8dc09f1470bd P 584dac86183a41d9b3bea1c2085e62fa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:56.925578 29799 raft_consensus.cc:515] T 8627b58b4f42491597125eb45f8d3b61 P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:26:56.927035 29520 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8627b58b4f42491597125eb45f8d3b61" candidate_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "584dac86183a41d9b3bea1c2085e62fa" is_pre_election: true
W20260709 12:26:56.928344 29485 leader_election.cc:343] T d42259e436d241b29fb4ded16d9b6455 P 584dac86183a41d9b3bea1c2085e62fa [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:56.928694 29670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8627b58b4f42491597125eb45f8d3b61" candidate_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d13c58ad8ac441c4a9b396eb04b1a476"
I20260709 12:26:56.929502 29670 raft_consensus.cc:3060] T 8627b58b4f42491597125eb45f8d3b61 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:26:56.929853 29708 leader_election.cc:343] T 8627b58b4f42491597125eb45f8d3b61 P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:56.934849 29834 raft_consensus.cc:515] T 592801934cad4789b9ce8dc09f1470bd P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:26:56.937723 29670 raft_consensus.cc:2468] T 8627b58b4f42491597125eb45f8d3b61 P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6610c00ffb948ce9c34f41dec3d49e8 in term 1.
I20260709 12:26:56.939337 29709 leader_election.cc:304] T 8627b58b4f42491597125eb45f8d3b61 P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c6610c00ffb948ce9c34f41dec3d49e8, d13c58ad8ac441c4a9b396eb04b1a476; no voters: 
I20260709 12:26:56.944761 29794 tablet_bootstrap.cc:492] T 43c7bfb33af74d4f912e5153e8a54b3e P d13c58ad8ac441c4a9b396eb04b1a476: No bootstrap required, opened a new log
I20260709 12:26:56.945295 29671 raft_consensus.cc:2468] T 742beffede4f407d8a72909d2b147e6d P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 584dac86183a41d9b3bea1c2085e62fa in term 1.
I20260709 12:26:56.944890 29799 leader_election.cc:290] T 8627b58b4f42491597125eb45f8d3b61 P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 1 election: Requested vote from peers d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209), 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033)
I20260709 12:26:56.945640 29794 ts_tablet_manager.cc:1403] T 43c7bfb33af74d4f912e5153e8a54b3e P d13c58ad8ac441c4a9b396eb04b1a476: Time spent bootstrapping tablet: real 0.051s	user 0.007s	sys 0.007s
I20260709 12:26:56.968096 29799 raft_consensus.cc:2804] T 8627b58b4f42491597125eb45f8d3b61 P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:26:56.968546 29799 raft_consensus.cc:697] T 8627b58b4f42491597125eb45f8d3b61 P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 LEADER]: Becoming Leader. State: Replica: c6610c00ffb948ce9c34f41dec3d49e8, State: Running, Role: LEADER
I20260709 12:26:56.969686 29799 consensus_queue.cc:237] T 8627b58b4f42491597125eb45f8d3b61 P c6610c00ffb948ce9c34f41dec3d49e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:26:56.973119 29486 leader_election.cc:304] T 742beffede4f407d8a72909d2b147e6d P 584dac86183a41d9b3bea1c2085e62fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 584dac86183a41d9b3bea1c2085e62fa, d13c58ad8ac441c4a9b396eb04b1a476; no voters: c6610c00ffb948ce9c34f41dec3d49e8
I20260709 12:26:56.973578 29794 raft_consensus.cc:359] T 43c7bfb33af74d4f912e5153e8a54b3e P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:26:56.974550 29794 raft_consensus.cc:385] T 43c7bfb33af74d4f912e5153e8a54b3e P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:56.974905 29794 raft_consensus.cc:740] T 43c7bfb33af74d4f912e5153e8a54b3e P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d13c58ad8ac441c4a9b396eb04b1a476, State: Initialized, Role: FOLLOWER
I20260709 12:26:56.974938 29804 raft_consensus.cc:2804] T 742beffede4f407d8a72909d2b147e6d P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:26:56.946451 29669 raft_consensus.cc:2468] T d42259e436d241b29fb4ded16d9b6455 P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 584dac86183a41d9b3bea1c2085e62fa in term 1.
I20260709 12:26:56.975845 29804 raft_consensus.cc:697] T 742beffede4f407d8a72909d2b147e6d P 584dac86183a41d9b3bea1c2085e62fa [term 1 LEADER]: Becoming Leader. State: Replica: 584dac86183a41d9b3bea1c2085e62fa, State: Running, Role: LEADER
I20260709 12:26:56.949095 29520 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb6e78ccc01c4707af34cddf81619eab" candidate_uuid: "d73d92fb35234d16824c55689b5fce28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "584dac86183a41d9b3bea1c2085e62fa" is_pre_election: true
I20260709 12:26:56.977281 29860 raft_consensus.cc:493] T 4685c2e824c14fc1bcd45c16169798cc P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:56.978366 29560 leader_election.cc:304] T fb6e78ccc01c4707af34cddf81619eab P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d73d92fb35234d16824c55689b5fce28; no voters: 584dac86183a41d9b3bea1c2085e62fa, c6610c00ffb948ce9c34f41dec3d49e8
I20260709 12:26:56.978634 29794 consensus_queue.cc:260] T 43c7bfb33af74d4f912e5153e8a54b3e P d13c58ad8ac441c4a9b396eb04b1a476 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:26:56.979266 29744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592801934cad4789b9ce8dc09f1470bd" candidate_uuid: "584dac86183a41d9b3bea1c2085e62fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6610c00ffb948ce9c34f41dec3d49e8"
I20260709 12:26:56.977550 29804 consensus_queue.cc:237] T 742beffede4f407d8a72909d2b147e6d P 584dac86183a41d9b3bea1c2085e62fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:26:56.981171 29794 ts_tablet_manager.cc:1434] T 43c7bfb33af74d4f912e5153e8a54b3e P d13c58ad8ac441c4a9b396eb04b1a476: Time spent starting tablet: real 0.013s	user 0.007s	sys 0.002s
I20260709 12:26:56.978367 29486 leader_election.cc:304] T d42259e436d241b29fb4ded16d9b6455 P 584dac86183a41d9b3bea1c2085e62fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 584dac86183a41d9b3bea1c2085e62fa, d13c58ad8ac441c4a9b396eb04b1a476; no voters: d73d92fb35234d16824c55689b5fce28
I20260709 12:26:56.982734 29794 tablet_bootstrap.cc:492] T 00c6cf271c91400aa834467def9af9b1 P d13c58ad8ac441c4a9b396eb04b1a476: Bootstrap starting.
I20260709 12:26:56.982813 29835 raft_consensus.cc:2804] T d42259e436d241b29fb4ded16d9b6455 P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:26:56.983572 29835 raft_consensus.cc:697] T d42259e436d241b29fb4ded16d9b6455 P 584dac86183a41d9b3bea1c2085e62fa [term 1 LEADER]: Becoming Leader. State: Replica: 584dac86183a41d9b3bea1c2085e62fa, State: Running, Role: LEADER
I20260709 12:26:56.985001 29835 consensus_queue.cc:237] T d42259e436d241b29fb4ded16d9b6455 P 584dac86183a41d9b3bea1c2085e62fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:26:56.979866 29744 raft_consensus.cc:3060] T 592801934cad4789b9ce8dc09f1470bd P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:56.950024 29852 leader_election.cc:290] T fb6e78ccc01c4707af34cddf81619eab P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895), 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033)
I20260709 12:26:56.950688 29855 raft_consensus.cc:493] T 55061e62184b45f0bb1726adc1a4af81 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:56.992597 29855 raft_consensus.cc:515] T 55061e62184b45f0bb1726adc1a4af81 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:26:56.994364 29794 tablet_bootstrap.cc:654] T 00c6cf271c91400aa834467def9af9b1 P d13c58ad8ac441c4a9b396eb04b1a476: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:56.999100 29743 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55061e62184b45f0bb1726adc1a4af81" candidate_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" is_pre_election: true
I20260709 12:26:56.999907 29743 raft_consensus.cc:2468] T 55061e62184b45f0bb1726adc1a4af81 P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d13c58ad8ac441c4a9b396eb04b1a476 in term 0.
I20260709 12:26:57.001395 29635 leader_election.cc:304] T 55061e62184b45f0bb1726adc1a4af81 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c6610c00ffb948ce9c34f41dec3d49e8, d13c58ad8ac441c4a9b396eb04b1a476; no voters: 
I20260709 12:26:57.002292 29798 raft_consensus.cc:2804] T 55061e62184b45f0bb1726adc1a4af81 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:26:57.002683 29798 raft_consensus.cc:493] T 55061e62184b45f0bb1726adc1a4af81 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:57.003016 29798 raft_consensus.cc:3060] T 55061e62184b45f0bb1726adc1a4af81 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:57.004292 29418 catalog_manager.cc:5697] T 8627b58b4f42491597125eb45f8d3b61 P c6610c00ffb948ce9c34f41dec3d49e8 reported cstate change: term changed from 0 to 1, leader changed from <none> to c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68). New cstate: current_term: 1 leader_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:56.995623 29855 leader_election.cc:290] T 55061e62184b45f0bb1726adc1a4af81 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825), c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895)
I20260709 12:26:56.952194 29745 raft_consensus.cc:2468] T 3e9451c3797c4496a66dfdb3e9fb5e38 P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d13c58ad8ac441c4a9b396eb04b1a476 in term 1.
I20260709 12:26:56.954564 29519 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8627b58b4f42491597125eb45f8d3b61" candidate_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "584dac86183a41d9b3bea1c2085e62fa"
I20260709 12:26:57.013487 29635 leader_election.cc:304] T 3e9451c3797c4496a66dfdb3e9fb5e38 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c6610c00ffb948ce9c34f41dec3d49e8, d13c58ad8ac441c4a9b396eb04b1a476; no voters: 
I20260709 12:26:57.017990 29596 raft_consensus.cc:2468] T fb6e78ccc01c4707af34cddf81619eab P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6610c00ffb948ce9c34f41dec3d49e8 in term 1.
I20260709 12:26:57.018194 29595 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55061e62184b45f0bb1726adc1a4af81" candidate_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d73d92fb35234d16824c55689b5fce28" is_pre_election: true
I20260709 12:26:57.018646 29851 raft_consensus.cc:2749] T fb6e78ccc01c4707af34cddf81619eab P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260709 12:26:57.019838 29634 leader_election.cc:343] T 55061e62184b45f0bb1726adc1a4af81 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:57.020895 29855 raft_consensus.cc:2804] T 3e9451c3797c4496a66dfdb3e9fb5e38 P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Leader election won for term 1
W20260709 12:26:57.024326 29708 leader_election.cc:343] T 8627b58b4f42491597125eb45f8d3b61 P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:57.025863 29522 raft_consensus.cc:2468] T fb6e78ccc01c4707af34cddf81619eab P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6610c00ffb948ce9c34f41dec3d49e8 in term 1.
I20260709 12:26:57.029806 29708 leader_election.cc:304] T fb6e78ccc01c4707af34cddf81619eab P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 584dac86183a41d9b3bea1c2085e62fa, c6610c00ffb948ce9c34f41dec3d49e8; no voters: 
I20260709 12:26:57.032527 29790 ts_tablet_manager.cc:1434] T 4685c2e824c14fc1bcd45c16169798cc P c6610c00ffb948ce9c34f41dec3d49e8: Time spent starting tablet: real 0.178s	user 0.033s	sys 0.034s
I20260709 12:26:57.032598 29852 raft_consensus.cc:493] T cdf706ffabc34902ad3d1bf2cee7889e P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:57.033778 29852 raft_consensus.cc:515] T cdf706ffabc34902ad3d1bf2cee7889e P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:26:57.037155 29792 tablet_bootstrap.cc:492] T 4685c2e824c14fc1bcd45c16169798cc P 584dac86183a41d9b3bea1c2085e62fa: No bootstrap required, opened a new log
I20260709 12:26:57.037694 29792 ts_tablet_manager.cc:1403] T 4685c2e824c14fc1bcd45c16169798cc P 584dac86183a41d9b3bea1c2085e62fa: Time spent bootstrapping tablet: real 0.155s	user 0.045s	sys 0.014s
I20260709 12:26:57.037859 29671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cdf706ffabc34902ad3d1bf2cee7889e" candidate_uuid: "d73d92fb35234d16824c55689b5fce28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" is_pre_election: true
I20260709 12:26:57.038800 29671 raft_consensus.cc:2468] T cdf706ffabc34902ad3d1bf2cee7889e P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d73d92fb35234d16824c55689b5fce28 in term 0.
I20260709 12:26:56.977840 29860 raft_consensus.cc:515] T 4685c2e824c14fc1bcd45c16169798cc P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:26:56.966020 29834 leader_election.cc:290] T 592801934cad4789b9ce8dc09f1470bd P 584dac86183a41d9b3bea1c2085e62fa [CANDIDATE]: Term 1 election: Requested vote from peers c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895), d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825)
I20260709 12:26:56.967237 29597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592801934cad4789b9ce8dc09f1470bd" candidate_uuid: "584dac86183a41d9b3bea1c2085e62fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d73d92fb35234d16824c55689b5fce28"
I20260709 12:26:57.041409 29561 leader_election.cc:304] T cdf706ffabc34902ad3d1bf2cee7889e P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d13c58ad8ac441c4a9b396eb04b1a476, d73d92fb35234d16824c55689b5fce28; no voters: 
I20260709 12:26:57.042973 29855 raft_consensus.cc:697] T 3e9451c3797c4496a66dfdb3e9fb5e38 P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 LEADER]: Becoming Leader. State: Replica: d13c58ad8ac441c4a9b396eb04b1a476, State: Running, Role: LEADER
I20260709 12:26:57.044057 29597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4685c2e824c14fc1bcd45c16169798cc" candidate_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d73d92fb35234d16824c55689b5fce28" is_pre_election: true
I20260709 12:26:57.044757 29597 raft_consensus.cc:2468] T 4685c2e824c14fc1bcd45c16169798cc P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6610c00ffb948ce9c34f41dec3d49e8 in term 0.
I20260709 12:26:57.045001 29858 raft_consensus.cc:2804] T fb6e78ccc01c4707af34cddf81619eab P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:26:57.045778 29858 raft_consensus.cc:697] T fb6e78ccc01c4707af34cddf81619eab P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 LEADER]: Becoming Leader. State: Replica: c6610c00ffb948ce9c34f41dec3d49e8, State: Running, Role: LEADER
I20260709 12:26:57.046063 29792 raft_consensus.cc:359] T 4685c2e824c14fc1bcd45c16169798cc P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:26:57.044067 29855 consensus_queue.cc:237] T 3e9451c3797c4496a66dfdb3e9fb5e38 P d13c58ad8ac441c4a9b396eb04b1a476 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:26:57.046838 29792 raft_consensus.cc:385] T 4685c2e824c14fc1bcd45c16169798cc P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:57.046680 29858 consensus_queue.cc:237] T fb6e78ccc01c4707af34cddf81619eab P c6610c00ffb948ce9c34f41dec3d49e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:26:56.962324 29521 raft_consensus.cc:2468] T 3e9451c3797c4496a66dfdb3e9fb5e38 P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d13c58ad8ac441c4a9b396eb04b1a476 in term 1.
I20260709 12:26:57.035472 29744 raft_consensus.cc:2468] T 592801934cad4789b9ce8dc09f1470bd P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 584dac86183a41d9b3bea1c2085e62fa in term 1.
I20260709 12:26:57.048753 29486 leader_election.cc:304] T 592801934cad4789b9ce8dc09f1470bd P 584dac86183a41d9b3bea1c2085e62fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 584dac86183a41d9b3bea1c2085e62fa, c6610c00ffb948ce9c34f41dec3d49e8; no voters: 
I20260709 12:26:57.049607 29834 raft_consensus.cc:2804] T 592801934cad4789b9ce8dc09f1470bd P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:26:57.050117 29834 raft_consensus.cc:697] T 592801934cad4789b9ce8dc09f1470bd P 584dac86183a41d9b3bea1c2085e62fa [term 1 LEADER]: Becoming Leader. State: Replica: 584dac86183a41d9b3bea1c2085e62fa, State: Running, Role: LEADER
I20260709 12:26:57.049427 29418 catalog_manager.cc:5697] T d21ab1641501475b829863275dc8182a P d73d92fb35234d16824c55689b5fce28 reported cstate change: term changed from 0 to 1, leader changed from <none> to d73d92fb35234d16824c55689b5fce28 (127.16.29.66). New cstate: current_term: 1 leader_uuid: "d73d92fb35234d16824c55689b5fce28" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:57.050930 29834 consensus_queue.cc:237] T 592801934cad4789b9ce8dc09f1470bd P 584dac86183a41d9b3bea1c2085e62fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:26:57.047227 29792 raft_consensus.cc:740] T 4685c2e824c14fc1bcd45c16169798cc P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 584dac86183a41d9b3bea1c2085e62fa, State: Initialized, Role: FOLLOWER
I20260709 12:26:57.052316 29521 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4685c2e824c14fc1bcd45c16169798cc" candidate_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "584dac86183a41d9b3bea1c2085e62fa" is_pre_election: true
I20260709 12:26:57.052702 29792 consensus_queue.cc:260] T 4685c2e824c14fc1bcd45c16169798cc P 584dac86183a41d9b3bea1c2085e62fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:26:57.053684 29521 raft_consensus.cc:2468] T 4685c2e824c14fc1bcd45c16169798cc P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6610c00ffb948ce9c34f41dec3d49e8 in term 0.
I20260709 12:26:57.054920 29792 ts_tablet_manager.cc:1434] T 4685c2e824c14fc1bcd45c16169798cc P 584dac86183a41d9b3bea1c2085e62fa: Time spent starting tablet: real 0.017s	user 0.001s	sys 0.003s
I20260709 12:26:57.055387 29708 leader_election.cc:304] T 4685c2e824c14fc1bcd45c16169798cc P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c6610c00ffb948ce9c34f41dec3d49e8, d73d92fb35234d16824c55689b5fce28; no voters: 
W20260709 12:26:57.054979 29485 leader_election.cc:343] T 592801934cad4789b9ce8dc09f1470bd P 584dac86183a41d9b3bea1c2085e62fa [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:57.057119 29799 raft_consensus.cc:2804] T 4685c2e824c14fc1bcd45c16169798cc P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:26:57.057176 29744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cdf706ffabc34902ad3d1bf2cee7889e" candidate_uuid: "d73d92fb35234d16824c55689b5fce28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" is_pre_election: true
I20260709 12:26:57.057797 29799 raft_consensus.cc:493] T 4685c2e824c14fc1bcd45c16169798cc P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:57.058238 29799 raft_consensus.cc:3060] T 4685c2e824c14fc1bcd45c16169798cc P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:26:57.063071 29561 leader_election.cc:343] T cdf706ffabc34902ad3d1bf2cee7889e P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:57.055693 29852 leader_election.cc:290] T cdf706ffabc34902ad3d1bf2cee7889e P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209), c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895)
I20260709 12:26:57.065086 29852 raft_consensus.cc:2804] T cdf706ffabc34902ad3d1bf2cee7889e P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:26:57.065606 29852 raft_consensus.cc:493] T cdf706ffabc34902ad3d1bf2cee7889e P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:57.066048 29852 raft_consensus.cc:3060] T cdf706ffabc34902ad3d1bf2cee7889e P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:56.964053 29793 tablet_bootstrap.cc:654] T 592801934cad4789b9ce8dc09f1470bd P d73d92fb35234d16824c55689b5fce28: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:57.069049 29790 tablet_bootstrap.cc:492] T cdf706ffabc34902ad3d1bf2cee7889e P c6610c00ffb948ce9c34f41dec3d49e8: Bootstrap starting.
I20260709 12:26:57.038468 29798 raft_consensus.cc:515] T 55061e62184b45f0bb1726adc1a4af81 P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:26:57.077147 29793 tablet_bootstrap.cc:492] T 592801934cad4789b9ce8dc09f1470bd P d73d92fb35234d16824c55689b5fce28: No bootstrap required, opened a new log
I20260709 12:26:57.077477 29597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55061e62184b45f0bb1726adc1a4af81" candidate_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d73d92fb35234d16824c55689b5fce28"
I20260709 12:26:57.078048 29793 ts_tablet_manager.cc:1403] T 592801934cad4789b9ce8dc09f1470bd P d73d92fb35234d16824c55689b5fce28: Time spent bootstrapping tablet: real 0.272s	user 0.007s	sys 0.005s
W20260709 12:26:57.080499 29634 leader_election.cc:343] T 55061e62184b45f0bb1726adc1a4af81 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:57.082528 29793 raft_consensus.cc:359] T 592801934cad4789b9ce8dc09f1470bd P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:26:57.083544 29793 raft_consensus.cc:385] T 592801934cad4789b9ce8dc09f1470bd P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:57.084095 29798 leader_election.cc:290] T 55061e62184b45f0bb1726adc1a4af81 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 1 election: Requested vote from peers d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825), c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895)
I20260709 12:26:57.041431 29860 leader_election.cc:290] T 4685c2e824c14fc1bcd45c16169798cc P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033), d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825)
I20260709 12:26:57.070883 29799 raft_consensus.cc:515] T 4685c2e824c14fc1bcd45c16169798cc P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:26:57.084200 29744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55061e62184b45f0bb1726adc1a4af81" candidate_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6610c00ffb948ce9c34f41dec3d49e8"
I20260709 12:26:57.088241 29793 raft_consensus.cc:740] T 592801934cad4789b9ce8dc09f1470bd P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d73d92fb35234d16824c55689b5fce28, State: Initialized, Role: FOLLOWER
I20260709 12:26:57.071678 29792 tablet_bootstrap.cc:492] T 8627b58b4f42491597125eb45f8d3b61 P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap starting.
I20260709 12:26:57.096000 29744 raft_consensus.cc:3060] T 55061e62184b45f0bb1726adc1a4af81 P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:57.102674 29793 consensus_queue.cc:260] T 592801934cad4789b9ce8dc09f1470bd P d73d92fb35234d16824c55689b5fce28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:26:57.111327 29799 leader_election.cc:290] T 4685c2e824c14fc1bcd45c16169798cc P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 1 election: Requested vote from peers 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033), d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825)
I20260709 12:26:57.112869 29597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4685c2e824c14fc1bcd45c16169798cc" candidate_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d73d92fb35234d16824c55689b5fce28"
I20260709 12:26:57.113543 29597 raft_consensus.cc:3060] T 4685c2e824c14fc1bcd45c16169798cc P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:57.113677 29521 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4685c2e824c14fc1bcd45c16169798cc" candidate_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "584dac86183a41d9b3bea1c2085e62fa"
I20260709 12:26:57.114782 29521 raft_consensus.cc:3060] T 4685c2e824c14fc1bcd45c16169798cc P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:57.123865 29793 ts_tablet_manager.cc:1434] T 592801934cad4789b9ce8dc09f1470bd P d73d92fb35234d16824c55689b5fce28: Time spent starting tablet: real 0.045s	user 0.011s	sys 0.000s
I20260709 12:26:57.126173 29794 tablet_bootstrap.cc:492] T 00c6cf271c91400aa834467def9af9b1 P d13c58ad8ac441c4a9b396eb04b1a476: No bootstrap required, opened a new log
I20260709 12:26:57.125792 29418 catalog_manager.cc:5697] T fb6e78ccc01c4707af34cddf81619eab P c6610c00ffb948ce9c34f41dec3d49e8 reported cstate change: term changed from 0 to 1, leader changed from <none> to c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68). New cstate: current_term: 1 leader_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:57.132016 29794 ts_tablet_manager.cc:1403] T 00c6cf271c91400aa834467def9af9b1 P d13c58ad8ac441c4a9b396eb04b1a476: Time spent bootstrapping tablet: real 0.149s	user 0.030s	sys 0.018s
I20260709 12:26:57.130935 29521 raft_consensus.cc:2468] T 4685c2e824c14fc1bcd45c16169798cc P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6610c00ffb948ce9c34f41dec3d49e8 in term 1.
I20260709 12:26:57.133306 29708 leader_election.cc:304] T 4685c2e824c14fc1bcd45c16169798cc P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 584dac86183a41d9b3bea1c2085e62fa, c6610c00ffb948ce9c34f41dec3d49e8; no voters: 
I20260709 12:26:57.134164 29799 raft_consensus.cc:2804] T 4685c2e824c14fc1bcd45c16169798cc P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:26:57.134615 29799 raft_consensus.cc:697] T 4685c2e824c14fc1bcd45c16169798cc P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 LEADER]: Becoming Leader. State: Replica: c6610c00ffb948ce9c34f41dec3d49e8, State: Running, Role: LEADER
I20260709 12:26:57.127926 29793 tablet_bootstrap.cc:492] T 55061e62184b45f0bb1726adc1a4af81 P d73d92fb35234d16824c55689b5fce28: Bootstrap starting.
I20260709 12:26:57.120857 29798 raft_consensus.cc:493] T 43c7bfb33af74d4f912e5153e8a54b3e P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:57.117460 29417 catalog_manager.cc:5697] T 742beffede4f407d8a72909d2b147e6d P 584dac86183a41d9b3bea1c2085e62fa reported cstate change: term changed from 0 to 1, leader changed from <none> to 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65). New cstate: current_term: 1 leader_uuid: "584dac86183a41d9b3bea1c2085e62fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:57.135437 29799 consensus_queue.cc:237] T 4685c2e824c14fc1bcd45c16169798cc P c6610c00ffb948ce9c34f41dec3d49e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:26:57.138460 29794 raft_consensus.cc:359] T 00c6cf271c91400aa834467def9af9b1 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:26:57.139423 29794 raft_consensus.cc:385] T 00c6cf271c91400aa834467def9af9b1 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:57.139779 29794 raft_consensus.cc:740] T 00c6cf271c91400aa834467def9af9b1 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d13c58ad8ac441c4a9b396eb04b1a476, State: Initialized, Role: FOLLOWER
I20260709 12:26:57.140544 29852 raft_consensus.cc:515] T cdf706ffabc34902ad3d1bf2cee7889e P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:26:57.140697 29794 consensus_queue.cc:260] T 00c6cf271c91400aa834467def9af9b1 P d13c58ad8ac441c4a9b396eb04b1a476 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:26:57.142683 29852 leader_election.cc:290] T cdf706ffabc34902ad3d1bf2cee7889e P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 1 election: Requested vote from peers d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209), c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895)
I20260709 12:26:57.138430 29417 catalog_manager.cc:5697] T d42259e436d241b29fb4ded16d9b6455 P 584dac86183a41d9b3bea1c2085e62fa reported cstate change: term changed from 0 to 1, leader changed from <none> to 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65). New cstate: current_term: 1 leader_uuid: "584dac86183a41d9b3bea1c2085e62fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:57.137414 29798 raft_consensus.cc:515] T 43c7bfb33af74d4f912e5153e8a54b3e P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:26:57.144052 29417 catalog_manager.cc:5697] T 592801934cad4789b9ce8dc09f1470bd P 584dac86183a41d9b3bea1c2085e62fa reported cstate change: term changed from 0 to 1, leader changed from <none> to 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65). New cstate: current_term: 1 leader_uuid: "584dac86183a41d9b3bea1c2085e62fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:57.145479 29798 leader_election.cc:290] T 43c7bfb33af74d4f912e5153e8a54b3e P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825), c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895)
I20260709 12:26:57.145776 29671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cdf706ffabc34902ad3d1bf2cee7889e" candidate_uuid: "d73d92fb35234d16824c55689b5fce28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d13c58ad8ac441c4a9b396eb04b1a476"
I20260709 12:26:57.145992 29745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cdf706ffabc34902ad3d1bf2cee7889e" candidate_uuid: "d73d92fb35234d16824c55689b5fce28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6610c00ffb948ce9c34f41dec3d49e8"
I20260709 12:26:57.146406 29671 raft_consensus.cc:3060] T cdf706ffabc34902ad3d1bf2cee7889e P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:57.149159 29745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43c7bfb33af74d4f912e5153e8a54b3e" candidate_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" is_pre_election: true
I20260709 12:26:57.149784 29792 tablet_bootstrap.cc:654] T 8627b58b4f42491597125eb45f8d3b61 P 584dac86183a41d9b3bea1c2085e62fa: Neither blocks nor log segments found. Creating new log.
W20260709 12:26:57.150578 29635 leader_election.cc:343] T 43c7bfb33af74d4f912e5153e8a54b3e P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:57.148146 29419 catalog_manager.cc:5697] T 3e9451c3797c4496a66dfdb3e9fb5e38 P d13c58ad8ac441c4a9b396eb04b1a476 reported cstate change: term changed from 0 to 1, leader changed from <none> to d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67). New cstate: current_term: 1 leader_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } health_report { overall_health: UNKNOWN } } }
W20260709 12:26:57.152323 29561 leader_election.cc:343] T cdf706ffabc34902ad3d1bf2cee7889e P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:57.153548 29596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43c7bfb33af74d4f912e5153e8a54b3e" candidate_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d73d92fb35234d16824c55689b5fce28" is_pre_election: true
I20260709 12:26:57.153867 29744 raft_consensus.cc:2468] T 55061e62184b45f0bb1726adc1a4af81 P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d13c58ad8ac441c4a9b396eb04b1a476 in term 1.
I20260709 12:26:57.154214 29596 raft_consensus.cc:2468] T 43c7bfb33af74d4f912e5153e8a54b3e P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d13c58ad8ac441c4a9b396eb04b1a476 in term 0.
I20260709 12:26:57.155197 29635 leader_election.cc:304] T 55061e62184b45f0bb1726adc1a4af81 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c6610c00ffb948ce9c34f41dec3d49e8, d13c58ad8ac441c4a9b396eb04b1a476; no voters: d73d92fb35234d16824c55689b5fce28
I20260709 12:26:57.155568 29634 leader_election.cc:304] T 43c7bfb33af74d4f912e5153e8a54b3e P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d13c58ad8ac441c4a9b396eb04b1a476, d73d92fb35234d16824c55689b5fce28; no voters: c6610c00ffb948ce9c34f41dec3d49e8
I20260709 12:26:57.156424 29798 raft_consensus.cc:2804] T 55061e62184b45f0bb1726adc1a4af81 P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:26:57.157052 29597 raft_consensus.cc:2468] T 4685c2e824c14fc1bcd45c16169798cc P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6610c00ffb948ce9c34f41dec3d49e8 in term 1.
I20260709 12:26:57.157236 29865 raft_consensus.cc:2804] T 43c7bfb33af74d4f912e5153e8a54b3e P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:26:57.157687 29865 raft_consensus.cc:493] T 43c7bfb33af74d4f912e5153e8a54b3e P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:57.158025 29865 raft_consensus.cc:3060] T 43c7bfb33af74d4f912e5153e8a54b3e P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:57.161966 29790 tablet_bootstrap.cc:654] T cdf706ffabc34902ad3d1bf2cee7889e P c6610c00ffb948ce9c34f41dec3d49e8: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:57.162537 29671 raft_consensus.cc:2468] T cdf706ffabc34902ad3d1bf2cee7889e P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d73d92fb35234d16824c55689b5fce28 in term 1.
I20260709 12:26:57.164292 29834 consensus_queue.cc:1048] T 7ae3b542012d4f35a9c363eee22976ec P 584dac86183a41d9b3bea1c2085e62fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:26:57.165555 29561 leader_election.cc:304] T cdf706ffabc34902ad3d1bf2cee7889e P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d13c58ad8ac441c4a9b396eb04b1a476, d73d92fb35234d16824c55689b5fce28; no voters: c6610c00ffb948ce9c34f41dec3d49e8
I20260709 12:26:57.167490 29793 tablet_bootstrap.cc:654] T 55061e62184b45f0bb1726adc1a4af81 P d73d92fb35234d16824c55689b5fce28: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:57.173661 29865 raft_consensus.cc:515] T 43c7bfb33af74d4f912e5153e8a54b3e P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:26:57.174451 29798 raft_consensus.cc:697] T 55061e62184b45f0bb1726adc1a4af81 P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 LEADER]: Becoming Leader. State: Replica: d13c58ad8ac441c4a9b396eb04b1a476, State: Running, Role: LEADER
I20260709 12:26:57.175395 29794 ts_tablet_manager.cc:1434] T 00c6cf271c91400aa834467def9af9b1 P d13c58ad8ac441c4a9b396eb04b1a476: Time spent starting tablet: real 0.043s	user 0.006s	sys 0.001s
I20260709 12:26:57.178287 29794 tablet_bootstrap.cc:492] T ed057e06dec148a0b9d7062beeb9d005 P d13c58ad8ac441c4a9b396eb04b1a476: Bootstrap starting.
I20260709 12:26:57.179963 29852 raft_consensus.cc:2804] T cdf706ffabc34902ad3d1bf2cee7889e P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:26:57.186198 29852 raft_consensus.cc:697] T cdf706ffabc34902ad3d1bf2cee7889e P d73d92fb35234d16824c55689b5fce28 [term 1 LEADER]: Becoming Leader. State: Replica: d73d92fb35234d16824c55689b5fce28, State: Running, Role: LEADER
I20260709 12:26:57.187106 29852 consensus_queue.cc:237] T cdf706ffabc34902ad3d1bf2cee7889e P d73d92fb35234d16824c55689b5fce28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:26:57.189975 29865 leader_election.cc:290] T 43c7bfb33af74d4f912e5153e8a54b3e P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 1 election: Requested vote from peers d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825), c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895)
I20260709 12:26:57.176614 29798 consensus_queue.cc:237] T 55061e62184b45f0bb1726adc1a4af81 P d13c58ad8ac441c4a9b396eb04b1a476 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:26:57.193989 29744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43c7bfb33af74d4f912e5153e8a54b3e" candidate_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6610c00ffb948ce9c34f41dec3d49e8"
W20260709 12:26:57.196198 29635 leader_election.cc:343] T 43c7bfb33af74d4f912e5153e8a54b3e P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:57.198189 29834 consensus_queue.cc:1048] T 7ae3b542012d4f35a9c363eee22976ec P 584dac86183a41d9b3bea1c2085e62fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:26:57.198964 29794 tablet_bootstrap.cc:654] T ed057e06dec148a0b9d7062beeb9d005 P d13c58ad8ac441c4a9b396eb04b1a476: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:57.200070 29418 catalog_manager.cc:5697] T 4685c2e824c14fc1bcd45c16169798cc P c6610c00ffb948ce9c34f41dec3d49e8 reported cstate change: term changed from 0 to 1, leader changed from <none> to c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68). New cstate: current_term: 1 leader_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:57.204068 29597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43c7bfb33af74d4f912e5153e8a54b3e" candidate_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d73d92fb35234d16824c55689b5fce28"
I20260709 12:26:57.204821 29597 raft_consensus.cc:3060] T 43c7bfb33af74d4f912e5153e8a54b3e P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:57.213029 29597 raft_consensus.cc:2468] T 43c7bfb33af74d4f912e5153e8a54b3e P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d13c58ad8ac441c4a9b396eb04b1a476 in term 1.
I20260709 12:26:57.216835 29790 tablet_bootstrap.cc:492] T cdf706ffabc34902ad3d1bf2cee7889e P c6610c00ffb948ce9c34f41dec3d49e8: No bootstrap required, opened a new log
I20260709 12:26:57.218353 29792 tablet_bootstrap.cc:492] T 8627b58b4f42491597125eb45f8d3b61 P 584dac86183a41d9b3bea1c2085e62fa: No bootstrap required, opened a new log
I20260709 12:26:57.220036 29808 consensus_queue.cc:1048] T d21ab1641501475b829863275dc8182a P d73d92fb35234d16824c55689b5fce28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:26:57.225739 29420 catalog_manager.cc:5697] T 55061e62184b45f0bb1726adc1a4af81 P d13c58ad8ac441c4a9b396eb04b1a476 reported cstate change: term changed from 0 to 1, leader changed from <none> to d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67). New cstate: current_term: 1 leader_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } health_report { overall_health: HEALTHY } } }
I20260709 12:26:57.227715 29792 ts_tablet_manager.cc:1403] T 8627b58b4f42491597125eb45f8d3b61 P 584dac86183a41d9b3bea1c2085e62fa: Time spent bootstrapping tablet: real 0.172s	user 0.021s	sys 0.000s
I20260709 12:26:57.240826 29793 tablet_bootstrap.cc:492] T 55061e62184b45f0bb1726adc1a4af81 P d73d92fb35234d16824c55689b5fce28: No bootstrap required, opened a new log
I20260709 12:26:57.241555 29793 ts_tablet_manager.cc:1403] T 55061e62184b45f0bb1726adc1a4af81 P d73d92fb35234d16824c55689b5fce28: Time spent bootstrapping tablet: real 0.114s	user 0.025s	sys 0.019s
I20260709 12:26:57.240639 29834 consensus_queue.cc:1048] T fcb6a7f720b14dc98e821f663357a0bb P 584dac86183a41d9b3bea1c2085e62fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:26:57.234068 29790 ts_tablet_manager.cc:1403] T cdf706ffabc34902ad3d1bf2cee7889e P c6610c00ffb948ce9c34f41dec3d49e8: Time spent bootstrapping tablet: real 0.165s	user 0.007s	sys 0.013s
I20260709 12:26:57.247675 29793 raft_consensus.cc:359] T 55061e62184b45f0bb1726adc1a4af81 P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:26:57.248525 29793 raft_consensus.cc:385] T 55061e62184b45f0bb1726adc1a4af81 P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:57.248893 29793 raft_consensus.cc:740] T 55061e62184b45f0bb1726adc1a4af81 P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d73d92fb35234d16824c55689b5fce28, State: Initialized, Role: FOLLOWER
I20260709 12:26:57.252318 29793 consensus_queue.cc:260] T 55061e62184b45f0bb1726adc1a4af81 P d73d92fb35234d16824c55689b5fce28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:26:57.258064 29793 ts_tablet_manager.cc:1434] T 55061e62184b45f0bb1726adc1a4af81 P d73d92fb35234d16824c55689b5fce28: Time spent starting tablet: real 0.016s	user 0.006s	sys 0.001s
I20260709 12:26:57.257679 29792 raft_consensus.cc:359] T 8627b58b4f42491597125eb45f8d3b61 P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:26:57.258628 29792 raft_consensus.cc:385] T 8627b58b4f42491597125eb45f8d3b61 P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:57.258950 29792 raft_consensus.cc:740] T 8627b58b4f42491597125eb45f8d3b61 P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 584dac86183a41d9b3bea1c2085e62fa, State: Initialized, Role: FOLLOWER
I20260709 12:26:57.261642 29792 consensus_queue.cc:260] T 8627b58b4f42491597125eb45f8d3b61 P 584dac86183a41d9b3bea1c2085e62fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:26:57.266851 29790 raft_consensus.cc:359] T cdf706ffabc34902ad3d1bf2cee7889e P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:26:57.267737 29790 raft_consensus.cc:385] T cdf706ffabc34902ad3d1bf2cee7889e P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:57.268081 29790 raft_consensus.cc:740] T cdf706ffabc34902ad3d1bf2cee7889e P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6610c00ffb948ce9c34f41dec3d49e8, State: Initialized, Role: FOLLOWER
I20260709 12:26:57.268970 29790 consensus_queue.cc:260] T cdf706ffabc34902ad3d1bf2cee7889e P c6610c00ffb948ce9c34f41dec3d49e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:26:57.271342 29793 tablet_bootstrap.cc:492] T d42259e436d241b29fb4ded16d9b6455 P d73d92fb35234d16824c55689b5fce28: Bootstrap starting.
I20260709 12:26:57.271656 29790 ts_tablet_manager.cc:1434] T cdf706ffabc34902ad3d1bf2cee7889e P c6610c00ffb948ce9c34f41dec3d49e8: Time spent starting tablet: real 0.026s	user 0.006s	sys 0.000s
I20260709 12:26:57.272717 29790 tablet_bootstrap.cc:492] T 742beffede4f407d8a72909d2b147e6d P c6610c00ffb948ce9c34f41dec3d49e8: Bootstrap starting.
I20260709 12:26:57.272545 29634 leader_election.cc:304] T 43c7bfb33af74d4f912e5153e8a54b3e P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d13c58ad8ac441c4a9b396eb04b1a476, d73d92fb35234d16824c55689b5fce28; no voters: c6610c00ffb948ce9c34f41dec3d49e8
I20260709 12:26:57.276532 29798 raft_consensus.cc:2804] T 43c7bfb33af74d4f912e5153e8a54b3e P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:26:57.277056 29798 raft_consensus.cc:697] T 43c7bfb33af74d4f912e5153e8a54b3e P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 LEADER]: Becoming Leader. State: Replica: d13c58ad8ac441c4a9b396eb04b1a476, State: Running, Role: LEADER
I20260709 12:26:57.277473 29792 ts_tablet_manager.cc:1434] T 8627b58b4f42491597125eb45f8d3b61 P 584dac86183a41d9b3bea1c2085e62fa: Time spent starting tablet: real 0.048s	user 0.006s	sys 0.000s
I20260709 12:26:57.277904 29798 consensus_queue.cc:237] T 43c7bfb33af74d4f912e5153e8a54b3e P d13c58ad8ac441c4a9b396eb04b1a476 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:26:57.278582 29792 tablet_bootstrap.cc:492] T 62f8c9472b214576b9002e02536a3ace P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap starting.
I20260709 12:26:57.280930 29790 tablet_bootstrap.cc:654] T 742beffede4f407d8a72909d2b147e6d P c6610c00ffb948ce9c34f41dec3d49e8: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:57.275314 29417 catalog_manager.cc:5697] T cdf706ffabc34902ad3d1bf2cee7889e P d73d92fb35234d16824c55689b5fce28 reported cstate change: term changed from 0 to 1, leader changed from <none> to d73d92fb35234d16824c55689b5fce28 (127.16.29.66). New cstate: current_term: 1 leader_uuid: "d73d92fb35234d16824c55689b5fce28" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:57.292721 29855 raft_consensus.cc:493] T 00c6cf271c91400aa834467def9af9b1 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:57.293289 29855 raft_consensus.cc:515] T 00c6cf271c91400aa834467def9af9b1 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:26:57.296303 29792 tablet_bootstrap.cc:654] T 62f8c9472b214576b9002e02536a3ace P 584dac86183a41d9b3bea1c2085e62fa: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:57.295866 29794 tablet_bootstrap.cc:492] T ed057e06dec148a0b9d7062beeb9d005 P d13c58ad8ac441c4a9b396eb04b1a476: No bootstrap required, opened a new log
I20260709 12:26:57.297101 29794 ts_tablet_manager.cc:1403] T ed057e06dec148a0b9d7062beeb9d005 P d13c58ad8ac441c4a9b396eb04b1a476: Time spent bootstrapping tablet: real 0.119s	user 0.010s	sys 0.037s
I20260709 12:26:57.314278 29855 leader_election.cc:290] T 00c6cf271c91400aa834467def9af9b1 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825), c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895)
I20260709 12:26:57.318894 29744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00c6cf271c91400aa834467def9af9b1" candidate_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" is_pre_election: true
I20260709 12:26:57.320163 29852 consensus_queue.cc:1048] T d21ab1641501475b829863275dc8182a P d73d92fb35234d16824c55689b5fce28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260709 12:26:57.320674 29635 leader_election.cc:343] T 00c6cf271c91400aa834467def9af9b1 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:57.330201 29794 raft_consensus.cc:359] T ed057e06dec148a0b9d7062beeb9d005 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:26:57.334512 29794 raft_consensus.cc:385] T ed057e06dec148a0b9d7062beeb9d005 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:57.330822 29835 consensus_queue.cc:1048] T fcb6a7f720b14dc98e821f663357a0bb P 584dac86183a41d9b3bea1c2085e62fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.011s
W20260709 12:26:57.348603 29878 log.cc:927] Time spent T cdf706ffabc34902ad3d1bf2cee7889e P d73d92fb35234d16824c55689b5fce28: Append to log took a long time: real 0.055s	user 0.001s	sys 0.000s
I20260709 12:26:57.334923 29852 raft_consensus.cc:493] T 592801934cad4789b9ce8dc09f1470bd P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:57.351276 29852 raft_consensus.cc:515] T 592801934cad4789b9ce8dc09f1470bd P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:26:57.355559 29522 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592801934cad4789b9ce8dc09f1470bd" candidate_uuid: "d73d92fb35234d16824c55689b5fce28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "584dac86183a41d9b3bea1c2085e62fa" is_pre_election: true
I20260709 12:26:57.356441 29793 tablet_bootstrap.cc:654] T d42259e436d241b29fb4ded16d9b6455 P d73d92fb35234d16824c55689b5fce28: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:57.335438 29794 raft_consensus.cc:740] T ed057e06dec148a0b9d7062beeb9d005 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d13c58ad8ac441c4a9b396eb04b1a476, State: Initialized, Role: FOLLOWER
I20260709 12:26:57.368333 29744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592801934cad4789b9ce8dc09f1470bd" candidate_uuid: "d73d92fb35234d16824c55689b5fce28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" is_pre_election: true
I20260709 12:26:57.369556 29744 raft_consensus.cc:2393] T 592801934cad4789b9ce8dc09f1470bd P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d73d92fb35234d16824c55689b5fce28 in current term 1: Already voted for candidate 584dac86183a41d9b3bea1c2085e62fa in this term.
I20260709 12:26:57.331569 29596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00c6cf271c91400aa834467def9af9b1" candidate_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d73d92fb35234d16824c55689b5fce28" is_pre_election: true
I20260709 12:26:57.364138 29794 consensus_queue.cc:260] T ed057e06dec148a0b9d7062beeb9d005 P d13c58ad8ac441c4a9b396eb04b1a476 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:26:57.372246 29420 catalog_manager.cc:5697] T 43c7bfb33af74d4f912e5153e8a54b3e P d13c58ad8ac441c4a9b396eb04b1a476 reported cstate change: term changed from 0 to 1, leader changed from <none> to d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67). New cstate: current_term: 1 leader_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:57.380080 29522 raft_consensus.cc:3060] T 8627b58b4f42491597125eb45f8d3b61 P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:57.380020 29561 leader_election.cc:304] T 592801934cad4789b9ce8dc09f1470bd P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d73d92fb35234d16824c55689b5fce28; no voters: 584dac86183a41d9b3bea1c2085e62fa, c6610c00ffb948ce9c34f41dec3d49e8
I20260709 12:26:57.389954 29808 raft_consensus.cc:3060] T 592801934cad4789b9ce8dc09f1470bd P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:26:57.390051 29634 leader_election.cc:343] T 00c6cf271c91400aa834467def9af9b1 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:57.390852 29634 leader_election.cc:304] T 00c6cf271c91400aa834467def9af9b1 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d13c58ad8ac441c4a9b396eb04b1a476; no voters: c6610c00ffb948ce9c34f41dec3d49e8, d73d92fb35234d16824c55689b5fce28
I20260709 12:26:57.391844 29855 raft_consensus.cc:2749] T 00c6cf271c91400aa834467def9af9b1 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:26:57.371130 29852 leader_election.cc:290] T 592801934cad4789b9ce8dc09f1470bd P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033), c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895)
I20260709 12:26:57.409183 29856 raft_consensus.cc:493] T 8627b58b4f42491597125eb45f8d3b61 P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:57.409766 29856 raft_consensus.cc:515] T 8627b58b4f42491597125eb45f8d3b61 P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
W20260709 12:26:57.418805 29486 consensus_peers.cc:597] T 742beffede4f407d8a72909d2b147e6d P 584dac86183a41d9b3bea1c2085e62fa -> Peer c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895): Couldn't send request to peer c6610c00ffb948ce9c34f41dec3d49e8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:26:57.418715 29794 ts_tablet_manager.cc:1434] T ed057e06dec148a0b9d7062beeb9d005 P d13c58ad8ac441c4a9b396eb04b1a476: Time spent starting tablet: real 0.121s	user 0.011s	sys 0.000s
I20260709 12:26:57.420725 29794 tablet_bootstrap.cc:492] T 2707271fd46f456aa765026642a3efa9 P d13c58ad8ac441c4a9b396eb04b1a476: Bootstrap starting.
I20260709 12:26:57.422297 29790 tablet_bootstrap.cc:492] T 742beffede4f407d8a72909d2b147e6d P c6610c00ffb948ce9c34f41dec3d49e8: No bootstrap required, opened a new log
I20260709 12:26:57.422888 29790 ts_tablet_manager.cc:1403] T 742beffede4f407d8a72909d2b147e6d P c6610c00ffb948ce9c34f41dec3d49e8: Time spent bootstrapping tablet: real 0.150s	user 0.007s	sys 0.009s
I20260709 12:26:57.426630 29790 raft_consensus.cc:359] T 742beffede4f407d8a72909d2b147e6d P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:26:57.427145 29794 tablet_bootstrap.cc:654] T 2707271fd46f456aa765026642a3efa9 P d13c58ad8ac441c4a9b396eb04b1a476: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:57.427428 29790 raft_consensus.cc:385] T 742beffede4f407d8a72909d2b147e6d P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:57.429404 29744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8627b58b4f42491597125eb45f8d3b61" candidate_uuid: "584dac86183a41d9b3bea1c2085e62fa" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" is_pre_election: true
I20260709 12:26:57.429724 29790 raft_consensus.cc:740] T 742beffede4f407d8a72909d2b147e6d P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6610c00ffb948ce9c34f41dec3d49e8, State: Initialized, Role: FOLLOWER
I20260709 12:26:57.430816 29856 leader_election.cc:290] T 8627b58b4f42491597125eb45f8d3b61 P 584dac86183a41d9b3bea1c2085e62fa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209), c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895)
I20260709 12:26:57.430619 29790 consensus_queue.cc:260] T 742beffede4f407d8a72909d2b147e6d P c6610c00ffb948ce9c34f41dec3d49e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:26:57.433470 29808 raft_consensus.cc:2749] T 592801934cad4789b9ce8dc09f1470bd P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:26:57.429123 29670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8627b58b4f42491597125eb45f8d3b61" candidate_uuid: "584dac86183a41d9b3bea1c2085e62fa" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" is_pre_election: true
I20260709 12:26:57.438617 29670 raft_consensus.cc:2468] T 8627b58b4f42491597125eb45f8d3b61 P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 584dac86183a41d9b3bea1c2085e62fa in term 1.
I20260709 12:26:57.439801 29486 leader_election.cc:304] T 8627b58b4f42491597125eb45f8d3b61 P 584dac86183a41d9b3bea1c2085e62fa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 584dac86183a41d9b3bea1c2085e62fa, d13c58ad8ac441c4a9b396eb04b1a476; no voters: c6610c00ffb948ce9c34f41dec3d49e8
I20260709 12:26:57.441207 29856 raft_consensus.cc:2804] T 8627b58b4f42491597125eb45f8d3b61 P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:26:57.441596 29856 raft_consensus.cc:493] T 8627b58b4f42491597125eb45f8d3b61 P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:57.441915 29856 raft_consensus.cc:3060] T 8627b58b4f42491597125eb45f8d3b61 P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:26:57.445837 29790 ts_tablet_manager.cc:1434] T 742beffede4f407d8a72909d2b147e6d P c6610c00ffb948ce9c34f41dec3d49e8: Time spent starting tablet: real 0.023s	user 0.006s	sys 0.002s
I20260709 12:26:57.446702 29790 tablet_bootstrap.cc:492] T 9633e3b1d78f403097e4a00b7d4b869b P c6610c00ffb948ce9c34f41dec3d49e8: Bootstrap starting.
I20260709 12:26:57.444021 29799 consensus_queue.cc:1048] T 8627b58b4f42491597125eb45f8d3b61 P c6610c00ffb948ce9c34f41dec3d49e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
I20260709 12:26:57.453444 29792 tablet_bootstrap.cc:492] T 62f8c9472b214576b9002e02536a3ace P 584dac86183a41d9b3bea1c2085e62fa: No bootstrap required, opened a new log
I20260709 12:26:57.454037 29792 ts_tablet_manager.cc:1403] T 62f8c9472b214576b9002e02536a3ace P 584dac86183a41d9b3bea1c2085e62fa: Time spent bootstrapping tablet: real 0.176s	user 0.056s	sys 0.046s
I20260709 12:26:57.458124 29792 raft_consensus.cc:359] T 62f8c9472b214576b9002e02536a3ace P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:26:57.458891 29792 raft_consensus.cc:385] T 62f8c9472b214576b9002e02536a3ace P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:57.459242 29792 raft_consensus.cc:740] T 62f8c9472b214576b9002e02536a3ace P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 584dac86183a41d9b3bea1c2085e62fa, State: Initialized, Role: FOLLOWER
I20260709 12:26:57.460156 29792 consensus_queue.cc:260] T 62f8c9472b214576b9002e02536a3ace P 584dac86183a41d9b3bea1c2085e62fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:26:57.463829 29799 consensus_queue.cc:1048] T 8627b58b4f42491597125eb45f8d3b61 P c6610c00ffb948ce9c34f41dec3d49e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:26:57.465950 29792 ts_tablet_manager.cc:1434] T 62f8c9472b214576b9002e02536a3ace P 584dac86183a41d9b3bea1c2085e62fa: Time spent starting tablet: real 0.012s	user 0.006s	sys 0.000s
I20260709 12:26:57.468443 29793 tablet_bootstrap.cc:492] T d42259e436d241b29fb4ded16d9b6455 P d73d92fb35234d16824c55689b5fce28: No bootstrap required, opened a new log
I20260709 12:26:57.473621 29793 ts_tablet_manager.cc:1403] T d42259e436d241b29fb4ded16d9b6455 P d73d92fb35234d16824c55689b5fce28: Time spent bootstrapping tablet: real 0.202s	user 0.019s	sys 0.001s
I20260709 12:26:57.474107 29856 raft_consensus.cc:515] T 8627b58b4f42491597125eb45f8d3b61 P 584dac86183a41d9b3bea1c2085e62fa [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:26:57.474375 29799 consensus_queue.cc:1048] T fb6e78ccc01c4707af34cddf81619eab P c6610c00ffb948ce9c34f41dec3d49e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:26:57.475473 29521 raft_consensus.cc:1240] T 8627b58b4f42491597125eb45f8d3b61 P 584dac86183a41d9b3bea1c2085e62fa [term 2 FOLLOWER]: Rejecting Update request from peer c6610c00ffb948ce9c34f41dec3d49e8 for earlier term 1. Current term is 2. Ops: [1.1-1.1]
I20260709 12:26:57.475824 29792 tablet_bootstrap.cc:492] T b3c02c910d2b43cc9265bb1d98491cd3 P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap starting.
I20260709 12:26:57.478390 29860 consensus_queue.cc:1059] T 8627b58b4f42491597125eb45f8d3b61 P c6610c00ffb948ce9c34f41dec3d49e8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:26:57.479240 29860 raft_consensus.cc:3055] T 8627b58b4f42491597125eb45f8d3b61 P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:26:57.479562 29860 raft_consensus.cc:740] T 8627b58b4f42491597125eb45f8d3b61 P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c6610c00ffb948ce9c34f41dec3d49e8, State: Running, Role: LEADER
I20260709 12:26:57.479588 29793 raft_consensus.cc:359] T d42259e436d241b29fb4ded16d9b6455 P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:26:57.480854 29793 raft_consensus.cc:385] T d42259e436d241b29fb4ded16d9b6455 P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:57.482132 29793 raft_consensus.cc:740] T d42259e436d241b29fb4ded16d9b6455 P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d73d92fb35234d16824c55689b5fce28, State: Initialized, Role: FOLLOWER
I20260709 12:26:57.483981 29860 consensus_queue.cc:260] T 8627b58b4f42491597125eb45f8d3b61 P c6610c00ffb948ce9c34f41dec3d49e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:26:57.485664 29860 raft_consensus.cc:3060] T 8627b58b4f42491597125eb45f8d3b61 P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:26:57.486344 29669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8627b58b4f42491597125eb45f8d3b61" candidate_uuid: "584dac86183a41d9b3bea1c2085e62fa" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d13c58ad8ac441c4a9b396eb04b1a476"
I20260709 12:26:57.488029 29790 tablet_bootstrap.cc:654] T 9633e3b1d78f403097e4a00b7d4b869b P c6610c00ffb948ce9c34f41dec3d49e8: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:57.488417 29793 consensus_queue.cc:260] T d42259e436d241b29fb4ded16d9b6455 P d73d92fb35234d16824c55689b5fce28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:26:57.498145 29793 ts_tablet_manager.cc:1434] T d42259e436d241b29fb4ded16d9b6455 P d73d92fb35234d16824c55689b5fce28: Time spent starting tablet: real 0.024s	user 0.004s	sys 0.001s
I20260709 12:26:57.492449 29835 consensus_queue.cc:1048] T 742beffede4f407d8a72909d2b147e6d P 584dac86183a41d9b3bea1c2085e62fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.006s
I20260709 12:26:57.498935 29744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8627b58b4f42491597125eb45f8d3b61" candidate_uuid: "584dac86183a41d9b3bea1c2085e62fa" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6610c00ffb948ce9c34f41dec3d49e8"
I20260709 12:26:57.499329 29793 tablet_bootstrap.cc:492] T 62f8c9472b214576b9002e02536a3ace P d73d92fb35234d16824c55689b5fce28: Bootstrap starting.
I20260709 12:26:57.500280 29856 leader_election.cc:290] T 8627b58b4f42491597125eb45f8d3b61 P 584dac86183a41d9b3bea1c2085e62fa [CANDIDATE]: Term 2 election: Requested vote from peers d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209), c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895)
I20260709 12:26:57.507530 29792 tablet_bootstrap.cc:654] T b3c02c910d2b43cc9265bb1d98491cd3 P 584dac86183a41d9b3bea1c2085e62fa: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:57.513586 29744 raft_consensus.cc:2410] T 8627b58b4f42491597125eb45f8d3b61 P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 584dac86183a41d9b3bea1c2085e62fa for term 2 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20260709 12:26:57.516556 29486 leader_election.cc:304] T 8627b58b4f42491597125eb45f8d3b61 P 584dac86183a41d9b3bea1c2085e62fa [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 584dac86183a41d9b3bea1c2085e62fa; no voters: c6610c00ffb948ce9c34f41dec3d49e8, d13c58ad8ac441c4a9b396eb04b1a476
I20260709 12:26:57.517436 29856 raft_consensus.cc:2749] T 8627b58b4f42491597125eb45f8d3b61 P 584dac86183a41d9b3bea1c2085e62fa [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:26:57.518560 29804 consensus_queue.cc:1048] T d42259e436d241b29fb4ded16d9b6455 P 584dac86183a41d9b3bea1c2085e62fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:26:57.519295 29793 tablet_bootstrap.cc:654] T 62f8c9472b214576b9002e02536a3ace P d73d92fb35234d16824c55689b5fce28: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:57.527408 29794 tablet_bootstrap.cc:492] T 2707271fd46f456aa765026642a3efa9 P d13c58ad8ac441c4a9b396eb04b1a476: No bootstrap required, opened a new log
I20260709 12:26:57.528191 29794 ts_tablet_manager.cc:1403] T 2707271fd46f456aa765026642a3efa9 P d13c58ad8ac441c4a9b396eb04b1a476: Time spent bootstrapping tablet: real 0.108s	user 0.039s	sys 0.007s
I20260709 12:26:57.523259 29856 consensus_queue.cc:1048] T 592801934cad4789b9ce8dc09f1470bd P 584dac86183a41d9b3bea1c2085e62fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.015s
I20260709 12:26:57.555156 29855 consensus_queue.cc:1048] T 3e9451c3797c4496a66dfdb3e9fb5e38 P d13c58ad8ac441c4a9b396eb04b1a476 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:26:57.564576 29597 raft_consensus.cc:3060] T d42259e436d241b29fb4ded16d9b6455 P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:57.572263 29855 consensus_queue.cc:1048] T 3e9451c3797c4496a66dfdb3e9fb5e38 P d13c58ad8ac441c4a9b396eb04b1a476 [LEADER]: Connected to new peer: Peer: permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260709 12:26:57.578876 29794 raft_consensus.cc:359] T 2707271fd46f456aa765026642a3efa9 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:26:57.579479 29794 raft_consensus.cc:385] T 2707271fd46f456aa765026642a3efa9 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:57.579761 29794 raft_consensus.cc:740] T 2707271fd46f456aa765026642a3efa9 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d13c58ad8ac441c4a9b396eb04b1a476, State: Initialized, Role: FOLLOWER
I20260709 12:26:57.581802 29794 consensus_queue.cc:260] T 2707271fd46f456aa765026642a3efa9 P d13c58ad8ac441c4a9b396eb04b1a476 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:26:57.593739 29794 ts_tablet_manager.cc:1434] T 2707271fd46f456aa765026642a3efa9 P d13c58ad8ac441c4a9b396eb04b1a476: Time spent starting tablet: real 0.065s	user 0.005s	sys 0.003s
I20260709 12:26:57.599624 29804 consensus_queue.cc:1048] T d42259e436d241b29fb4ded16d9b6455 P 584dac86183a41d9b3bea1c2085e62fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:26:57.600646 29856 consensus_queue.cc:1048] T 592801934cad4789b9ce8dc09f1470bd P 584dac86183a41d9b3bea1c2085e62fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:26:57.584429 29860 consensus_queue.cc:1048] T fb6e78ccc01c4707af34cddf81619eab P c6610c00ffb948ce9c34f41dec3d49e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.019s
I20260709 12:26:57.608753 29793 tablet_bootstrap.cc:492] T 62f8c9472b214576b9002e02536a3ace P d73d92fb35234d16824c55689b5fce28: No bootstrap required, opened a new log
I20260709 12:26:57.609685 29793 ts_tablet_manager.cc:1403] T 62f8c9472b214576b9002e02536a3ace P d73d92fb35234d16824c55689b5fce28: Time spent bootstrapping tablet: real 0.111s	user 0.014s	sys 0.002s
I20260709 12:26:57.612715 29794 tablet_bootstrap.cc:492] T 9633e3b1d78f403097e4a00b7d4b869b P d13c58ad8ac441c4a9b396eb04b1a476: Bootstrap starting.
I20260709 12:26:57.614589 29790 tablet_bootstrap.cc:492] T 9633e3b1d78f403097e4a00b7d4b869b P c6610c00ffb948ce9c34f41dec3d49e8: No bootstrap required, opened a new log
I20260709 12:26:57.615342 29790 ts_tablet_manager.cc:1403] T 9633e3b1d78f403097e4a00b7d4b869b P c6610c00ffb948ce9c34f41dec3d49e8: Time spent bootstrapping tablet: real 0.169s	user 0.016s	sys 0.011s
I20260709 12:26:57.627561 29595 raft_consensus.cc:3060] T 55061e62184b45f0bb1726adc1a4af81 P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:57.632396 29792 tablet_bootstrap.cc:492] T b3c02c910d2b43cc9265bb1d98491cd3 P 584dac86183a41d9b3bea1c2085e62fa: No bootstrap required, opened a new log
I20260709 12:26:57.633039 29792 ts_tablet_manager.cc:1403] T b3c02c910d2b43cc9265bb1d98491cd3 P 584dac86183a41d9b3bea1c2085e62fa: Time spent bootstrapping tablet: real 0.158s	user 0.015s	sys 0.003s
I20260709 12:26:57.634195 29904 raft_consensus.cc:493] T cdf706ffabc34902ad3d1bf2cee7889e P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:57.634716 29904 raft_consensus.cc:515] T cdf706ffabc34902ad3d1bf2cee7889e P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:26:57.644675 29792 raft_consensus.cc:359] T b3c02c910d2b43cc9265bb1d98491cd3 P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:26:57.645525 29792 raft_consensus.cc:385] T b3c02c910d2b43cc9265bb1d98491cd3 P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:57.645877 29792 raft_consensus.cc:740] T b3c02c910d2b43cc9265bb1d98491cd3 P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 584dac86183a41d9b3bea1c2085e62fa, State: Initialized, Role: FOLLOWER
I20260709 12:26:57.646425 29793 raft_consensus.cc:359] T 62f8c9472b214576b9002e02536a3ace P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:26:57.647234 29793 raft_consensus.cc:385] T 62f8c9472b214576b9002e02536a3ace P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:57.647533 29793 raft_consensus.cc:740] T 62f8c9472b214576b9002e02536a3ace P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d73d92fb35234d16824c55689b5fce28, State: Initialized, Role: FOLLOWER
I20260709 12:26:57.648990 29594 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cdf706ffabc34902ad3d1bf2cee7889e" candidate_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d73d92fb35234d16824c55689b5fce28" is_pre_election: true
I20260709 12:26:57.649745 29790 raft_consensus.cc:359] T 9633e3b1d78f403097e4a00b7d4b869b P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:26:57.650542 29790 raft_consensus.cc:385] T 9633e3b1d78f403097e4a00b7d4b869b P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:57.647500 29904 leader_election.cc:290] T cdf706ffabc34902ad3d1bf2cee7889e P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825), d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209)
I20260709 12:26:57.648983 29669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cdf706ffabc34902ad3d1bf2cee7889e" candidate_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" is_pre_election: true
I20260709 12:26:57.658954 29669 raft_consensus.cc:2393] T cdf706ffabc34902ad3d1bf2cee7889e P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c6610c00ffb948ce9c34f41dec3d49e8 in current term 1: Already voted for candidate d73d92fb35234d16824c55689b5fce28 in this term.
I20260709 12:26:57.660658 29855 raft_consensus.cc:493] T ed057e06dec148a0b9d7062beeb9d005 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:57.663190 29855 raft_consensus.cc:515] T ed057e06dec148a0b9d7062beeb9d005 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:26:57.624418 29799 consensus_queue.cc:1048] T 4685c2e824c14fc1bcd45c16169798cc P c6610c00ffb948ce9c34f41dec3d49e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.038s
I20260709 12:26:57.655421 29793 consensus_queue.cc:260] T 62f8c9472b214576b9002e02536a3ace P d73d92fb35234d16824c55689b5fce28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:26:57.650823 29790 raft_consensus.cc:740] T 9633e3b1d78f403097e4a00b7d4b869b P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6610c00ffb948ce9c34f41dec3d49e8, State: Initialized, Role: FOLLOWER
I20260709 12:26:57.666674 29794 tablet_bootstrap.cc:654] T 9633e3b1d78f403097e4a00b7d4b869b P d13c58ad8ac441c4a9b396eb04b1a476: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:57.646786 29792 consensus_queue.cc:260] T b3c02c910d2b43cc9265bb1d98491cd3 P 584dac86183a41d9b3bea1c2085e62fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:26:57.670817 29743 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed057e06dec148a0b9d7062beeb9d005" candidate_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" is_pre_election: true
I20260709 12:26:57.671144 29792 ts_tablet_manager.cc:1434] T b3c02c910d2b43cc9265bb1d98491cd3 P 584dac86183a41d9b3bea1c2085e62fa: Time spent starting tablet: real 0.038s	user 0.008s	sys 0.002s
I20260709 12:26:57.672706 29792 tablet_bootstrap.cc:492] T 9633e3b1d78f403097e4a00b7d4b869b P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap starting.
W20260709 12:26:57.672907 29635 leader_election.cc:343] T ed057e06dec148a0b9d7062beeb9d005 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:57.674014 29855 leader_election.cc:290] T ed057e06dec148a0b9d7062beeb9d005 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825), c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895)
I20260709 12:26:57.676914 29793 ts_tablet_manager.cc:1434] T 62f8c9472b214576b9002e02536a3ace P d73d92fb35234d16824c55689b5fce28: Time spent starting tablet: real 0.067s	user 0.008s	sys 0.000s
I20260709 12:26:57.678444 29793 tablet_bootstrap.cc:492] T 00c6cf271c91400aa834467def9af9b1 P d73d92fb35234d16824c55689b5fce28: Bootstrap starting.
I20260709 12:26:57.681226 29790 consensus_queue.cc:260] T 9633e3b1d78f403097e4a00b7d4b869b P c6610c00ffb948ce9c34f41dec3d49e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:26:57.683017 29594 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed057e06dec148a0b9d7062beeb9d005" candidate_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d73d92fb35234d16824c55689b5fce28" is_pre_election: true
W20260709 12:26:57.684830 29634 leader_election.cc:343] T ed057e06dec148a0b9d7062beeb9d005 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:57.685384 29634 leader_election.cc:304] T ed057e06dec148a0b9d7062beeb9d005 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d13c58ad8ac441c4a9b396eb04b1a476; no voters: c6610c00ffb948ce9c34f41dec3d49e8, d73d92fb35234d16824c55689b5fce28
I20260709 12:26:57.686278 29855 raft_consensus.cc:2749] T ed057e06dec148a0b9d7062beeb9d005 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:26:57.689348 29904 consensus_queue.cc:1048] T 4685c2e824c14fc1bcd45c16169798cc P c6610c00ffb948ce9c34f41dec3d49e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:26:57.689857 29709 leader_election.cc:304] T cdf706ffabc34902ad3d1bf2cee7889e P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c6610c00ffb948ce9c34f41dec3d49e8; no voters: d13c58ad8ac441c4a9b396eb04b1a476, d73d92fb35234d16824c55689b5fce28
I20260709 12:26:57.691205 29904 raft_consensus.cc:3060] T cdf706ffabc34902ad3d1bf2cee7889e P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:57.695175 29856 raft_consensus.cc:493] T 62f8c9472b214576b9002e02536a3ace P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:57.695734 29856 raft_consensus.cc:515] T 62f8c9472b214576b9002e02536a3ace P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:26:57.698735 29790 ts_tablet_manager.cc:1434] T 9633e3b1d78f403097e4a00b7d4b869b P c6610c00ffb948ce9c34f41dec3d49e8: Time spent starting tablet: real 0.083s	user 0.006s	sys 0.001s
I20260709 12:26:57.700079 29790 tablet_bootstrap.cc:492] T 43c7bfb33af74d4f912e5153e8a54b3e P c6610c00ffb948ce9c34f41dec3d49e8: Bootstrap starting.
I20260709 12:26:57.701897 29743 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f8c9472b214576b9002e02536a3ace" candidate_uuid: "584dac86183a41d9b3bea1c2085e62fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" is_pre_election: true
I20260709 12:26:57.705307 29808 raft_consensus.cc:493] T 55061e62184b45f0bb1726adc1a4af81 P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:57.706049 29808 raft_consensus.cc:515] T 55061e62184b45f0bb1726adc1a4af81 P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:26:57.707650 29865 raft_consensus.cc:493] T 2707271fd46f456aa765026642a3efa9 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:57.708212 29865 raft_consensus.cc:515] T 2707271fd46f456aa765026642a3efa9 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:26:57.709831 29808 leader_election.cc:290] T 55061e62184b45f0bb1726adc1a4af81 P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895), d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209)
I20260709 12:26:57.710315 29865 leader_election.cc:290] T 2707271fd46f456aa765026642a3efa9 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033), d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825)
I20260709 12:26:57.717229 29798 consensus_queue.cc:1048] T 55061e62184b45f0bb1726adc1a4af81 P d13c58ad8ac441c4a9b396eb04b1a476 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260709 12:26:57.703697 29486 leader_election.cc:343] T 62f8c9472b214576b9002e02536a3ace P 584dac86183a41d9b3bea1c2085e62fa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:57.720142 29743 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55061e62184b45f0bb1726adc1a4af81" candidate_uuid: "d73d92fb35234d16824c55689b5fce28" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" is_pre_election: true
I20260709 12:26:57.720846 29743 raft_consensus.cc:2468] T 55061e62184b45f0bb1726adc1a4af81 P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d73d92fb35234d16824c55689b5fce28 in term 1.
I20260709 12:26:57.725198 29904 raft_consensus.cc:2749] T cdf706ffabc34902ad3d1bf2cee7889e P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:26:57.726562 29792 tablet_bootstrap.cc:654] T 9633e3b1d78f403097e4a00b7d4b869b P 584dac86183a41d9b3bea1c2085e62fa: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:57.727362 29521 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2707271fd46f456aa765026642a3efa9" candidate_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "584dac86183a41d9b3bea1c2085e62fa" is_pre_election: true
I20260709 12:26:57.720752 29669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55061e62184b45f0bb1726adc1a4af81" candidate_uuid: "d73d92fb35234d16824c55689b5fce28" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" is_pre_election: true
I20260709 12:26:57.729290 29561 leader_election.cc:304] T 55061e62184b45f0bb1726adc1a4af81 P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c6610c00ffb948ce9c34f41dec3d49e8, d73d92fb35234d16824c55689b5fce28; no voters: 
I20260709 12:26:57.740435 29856 leader_election.cc:290] T 62f8c9472b214576b9002e02536a3ace P 584dac86183a41d9b3bea1c2085e62fa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895), d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825)
W20260709 12:26:57.742527 29634 leader_election.cc:343] T 2707271fd46f456aa765026642a3efa9 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:57.747177 29808 raft_consensus.cc:2804] T 55061e62184b45f0bb1726adc1a4af81 P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:26:57.747670 29808 raft_consensus.cc:493] T 55061e62184b45f0bb1726adc1a4af81 P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:57.748036 29808 raft_consensus.cc:3060] T 55061e62184b45f0bb1726adc1a4af81 P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Advancing to term 2
W20260709 12:26:57.749500 29635 consensus_peers.cc:597] T 43c7bfb33af74d4f912e5153e8a54b3e P d13c58ad8ac441c4a9b396eb04b1a476 -> Peer c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895): Couldn't send request to peer c6610c00ffb948ce9c34f41dec3d49e8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:26:57.753201 29597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2707271fd46f456aa765026642a3efa9" candidate_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d73d92fb35234d16824c55689b5fce28" is_pre_election: true
I20260709 12:26:57.754614 29597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f8c9472b214576b9002e02536a3ace" candidate_uuid: "584dac86183a41d9b3bea1c2085e62fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d73d92fb35234d16824c55689b5fce28" is_pre_election: true
I20260709 12:26:57.755259 29597 raft_consensus.cc:2468] T 62f8c9472b214576b9002e02536a3ace P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 584dac86183a41d9b3bea1c2085e62fa in term 0.
W20260709 12:26:57.755508 29634 leader_election.cc:343] T 2707271fd46f456aa765026642a3efa9 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:57.756269 29634 leader_election.cc:304] T 2707271fd46f456aa765026642a3efa9 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d13c58ad8ac441c4a9b396eb04b1a476; no voters: 584dac86183a41d9b3bea1c2085e62fa, d73d92fb35234d16824c55689b5fce28
I20260709 12:26:57.756433 29485 leader_election.cc:304] T 62f8c9472b214576b9002e02536a3ace P 584dac86183a41d9b3bea1c2085e62fa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 584dac86183a41d9b3bea1c2085e62fa, d73d92fb35234d16824c55689b5fce28; no voters: c6610c00ffb948ce9c34f41dec3d49e8
I20260709 12:26:57.757818 29865 raft_consensus.cc:2749] T 2707271fd46f456aa765026642a3efa9 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:26:57.758460 29834 raft_consensus.cc:2804] T 62f8c9472b214576b9002e02536a3ace P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:26:57.758867 29834 raft_consensus.cc:493] T 62f8c9472b214576b9002e02536a3ace P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:57.759196 29834 raft_consensus.cc:3060] T 62f8c9472b214576b9002e02536a3ace P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:57.760345 29808 raft_consensus.cc:515] T 55061e62184b45f0bb1726adc1a4af81 P d73d92fb35234d16824c55689b5fce28 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:26:57.761526 29595 raft_consensus.cc:1240] T 55061e62184b45f0bb1726adc1a4af81 P d73d92fb35234d16824c55689b5fce28 [term 2 FOLLOWER]: Rejecting Update request from peer d13c58ad8ac441c4a9b396eb04b1a476 for earlier term 1. Current term is 2. Ops: [1.1-1.1]
I20260709 12:26:57.765209 29808 leader_election.cc:290] T 55061e62184b45f0bb1726adc1a4af81 P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 2 election: Requested vote from peers c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895), d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209)
I20260709 12:26:57.766129 29798 consensus_queue.cc:1059] T 55061e62184b45f0bb1726adc1a4af81 P d13c58ad8ac441c4a9b396eb04b1a476 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:26:57.767064 29798 raft_consensus.cc:3055] T 55061e62184b45f0bb1726adc1a4af81 P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:26:57.767409 29745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55061e62184b45f0bb1726adc1a4af81" candidate_uuid: "d73d92fb35234d16824c55689b5fce28" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6610c00ffb948ce9c34f41dec3d49e8"
I20260709 12:26:57.767442 29798 raft_consensus.cc:740] T 55061e62184b45f0bb1726adc1a4af81 P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d13c58ad8ac441c4a9b396eb04b1a476, State: Running, Role: LEADER
I20260709 12:26:57.768052 29745 raft_consensus.cc:3060] T 55061e62184b45f0bb1726adc1a4af81 P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:26:57.768646 29798 consensus_queue.cc:260] T 55061e62184b45f0bb1726adc1a4af81 P d13c58ad8ac441c4a9b396eb04b1a476 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:26:57.769187 29669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55061e62184b45f0bb1726adc1a4af81" candidate_uuid: "d73d92fb35234d16824c55689b5fce28" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d13c58ad8ac441c4a9b396eb04b1a476"
I20260709 12:26:57.770071 29855 consensus_queue.cc:1048] T 55061e62184b45f0bb1726adc1a4af81 P d13c58ad8ac441c4a9b396eb04b1a476 [NON_LEADER]: Connected to new peer: Peer: permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:26:57.776470 29793 tablet_bootstrap.cc:654] T 00c6cf271c91400aa834467def9af9b1 P d73d92fb35234d16824c55689b5fce28: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:57.779130 29745 raft_consensus.cc:2468] T 55061e62184b45f0bb1726adc1a4af81 P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d73d92fb35234d16824c55689b5fce28 in term 2.
I20260709 12:26:57.780440 29561 leader_election.cc:304] T 55061e62184b45f0bb1726adc1a4af81 P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c6610c00ffb948ce9c34f41dec3d49e8, d73d92fb35234d16824c55689b5fce28; no voters: 
I20260709 12:26:57.781548 29892 raft_consensus.cc:2804] T 55061e62184b45f0bb1726adc1a4af81 P d73d92fb35234d16824c55689b5fce28 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:26:57.782210 29892 raft_consensus.cc:697] T 55061e62184b45f0bb1726adc1a4af81 P d73d92fb35234d16824c55689b5fce28 [term 2 LEADER]: Becoming Leader. State: Replica: d73d92fb35234d16824c55689b5fce28, State: Running, Role: LEADER
I20260709 12:26:57.783964 29834 raft_consensus.cc:515] T 62f8c9472b214576b9002e02536a3ace P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:26:57.785040 29904 raft_consensus.cc:493] T 9633e3b1d78f403097e4a00b7d4b869b P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:57.785744 29794 tablet_bootstrap.cc:492] T 9633e3b1d78f403097e4a00b7d4b869b P d13c58ad8ac441c4a9b396eb04b1a476: No bootstrap required, opened a new log
I20260709 12:26:57.785573 29904 raft_consensus.cc:515] T 9633e3b1d78f403097e4a00b7d4b869b P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:26:57.786252 29794 ts_tablet_manager.cc:1403] T 9633e3b1d78f403097e4a00b7d4b869b P d13c58ad8ac441c4a9b396eb04b1a476: Time spent bootstrapping tablet: real 0.174s	user 0.022s	sys 0.022s
I20260709 12:26:57.783152 29892 consensus_queue.cc:237] T 55061e62184b45f0bb1726adc1a4af81 P d73d92fb35234d16824c55689b5fce28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:26:57.789673 29794 raft_consensus.cc:359] T 9633e3b1d78f403097e4a00b7d4b869b P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:26:57.790210 29671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9633e3b1d78f403097e4a00b7d4b869b" candidate_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" is_pre_election: true
I20260709 12:26:57.790503 29794 raft_consensus.cc:385] T 9633e3b1d78f403097e4a00b7d4b869b P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:57.791007 29794 raft_consensus.cc:740] T 9633e3b1d78f403097e4a00b7d4b869b P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d13c58ad8ac441c4a9b396eb04b1a476, State: Initialized, Role: FOLLOWER
I20260709 12:26:57.792009 29794 consensus_queue.cc:260] T 9633e3b1d78f403097e4a00b7d4b869b P d13c58ad8ac441c4a9b396eb04b1a476 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:26:57.792531 29904 leader_election.cc:290] T 9633e3b1d78f403097e4a00b7d4b869b P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209), 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033)
I20260709 12:26:57.793231 29671 raft_consensus.cc:2468] T 9633e3b1d78f403097e4a00b7d4b869b P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6610c00ffb948ce9c34f41dec3d49e8 in term 0.
I20260709 12:26:57.794317 29794 ts_tablet_manager.cc:1434] T 9633e3b1d78f403097e4a00b7d4b869b P d13c58ad8ac441c4a9b396eb04b1a476: Time spent starting tablet: real 0.008s	user 0.008s	sys 0.000s
I20260709 12:26:57.795499 29794 tablet_bootstrap.cc:492] T 76140ad6258543118a9bfeee7cf0d5fc P d13c58ad8ac441c4a9b396eb04b1a476: Bootstrap starting.
I20260709 12:26:57.797780 29519 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9633e3b1d78f403097e4a00b7d4b869b" candidate_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "584dac86183a41d9b3bea1c2085e62fa" is_pre_election: true
I20260709 12:26:57.800283 29709 leader_election.cc:304] T 9633e3b1d78f403097e4a00b7d4b869b P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c6610c00ffb948ce9c34f41dec3d49e8, d13c58ad8ac441c4a9b396eb04b1a476; no voters: 
I20260709 12:26:57.802629 29860 raft_consensus.cc:2804] T 9633e3b1d78f403097e4a00b7d4b869b P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:26:57.803052 29860 raft_consensus.cc:493] T 9633e3b1d78f403097e4a00b7d4b869b P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:57.803416 29860 raft_consensus.cc:3060] T 9633e3b1d78f403097e4a00b7d4b869b P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:26:57.804906 29708 leader_election.cc:343] T 9633e3b1d78f403097e4a00b7d4b869b P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:57.805805 29834 leader_election.cc:290] T 62f8c9472b214576b9002e02536a3ace P 584dac86183a41d9b3bea1c2085e62fa [CANDIDATE]: Term 1 election: Requested vote from peers c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895), d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825)
I20260709 12:26:57.807616 29852 consensus_queue.cc:1048] T cdf706ffabc34902ad3d1bf2cee7889e P d73d92fb35234d16824c55689b5fce28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:26:57.809939 29595 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f8c9472b214576b9002e02536a3ace" candidate_uuid: "584dac86183a41d9b3bea1c2085e62fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d73d92fb35234d16824c55689b5fce28"
I20260709 12:26:57.810657 29595 raft_consensus.cc:3060] T 62f8c9472b214576b9002e02536a3ace P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:57.813961 29860 raft_consensus.cc:515] T 9633e3b1d78f403097e4a00b7d4b869b P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:26:57.830174 29792 tablet_bootstrap.cc:492] T 9633e3b1d78f403097e4a00b7d4b869b P 584dac86183a41d9b3bea1c2085e62fa: No bootstrap required, opened a new log
I20260709 12:26:57.830765 29792 ts_tablet_manager.cc:1403] T 9633e3b1d78f403097e4a00b7d4b869b P 584dac86183a41d9b3bea1c2085e62fa: Time spent bootstrapping tablet: real 0.158s	user 0.012s	sys 0.004s
I20260709 12:26:57.833770 29792 raft_consensus.cc:359] T 9633e3b1d78f403097e4a00b7d4b869b P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:26:57.834571 29792 raft_consensus.cc:385] T 9633e3b1d78f403097e4a00b7d4b869b P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:57.834836 29792 raft_consensus.cc:740] T 9633e3b1d78f403097e4a00b7d4b869b P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 584dac86183a41d9b3bea1c2085e62fa, State: Initialized, Role: FOLLOWER
I20260709 12:26:57.834147 29798 raft_consensus.cc:3060] T 55061e62184b45f0bb1726adc1a4af81 P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:26:57.835733 29792 consensus_queue.cc:260] T 9633e3b1d78f403097e4a00b7d4b869b P 584dac86183a41d9b3bea1c2085e62fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:26:57.838177 29794 tablet_bootstrap.cc:654] T 76140ad6258543118a9bfeee7cf0d5fc P d13c58ad8ac441c4a9b396eb04b1a476: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:57.846611 29790 tablet_bootstrap.cc:654] T 43c7bfb33af74d4f912e5153e8a54b3e P c6610c00ffb948ce9c34f41dec3d49e8: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:57.859421 29865 consensus_queue.cc:1048] T 43c7bfb33af74d4f912e5153e8a54b3e P d13c58ad8ac441c4a9b396eb04b1a476 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:26:57.864962 29856 raft_consensus.cc:493] T 9633e3b1d78f403097e4a00b7d4b869b P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:57.865788 29856 raft_consensus.cc:515] T 9633e3b1d78f403097e4a00b7d4b869b P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:26:57.838660 29745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f8c9472b214576b9002e02536a3ace" candidate_uuid: "584dac86183a41d9b3bea1c2085e62fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6610c00ffb948ce9c34f41dec3d49e8"
I20260709 12:26:57.868510 29860 leader_election.cc:290] T 9633e3b1d78f403097e4a00b7d4b869b P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 1 election: Requested vote from peers d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209), 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033)
I20260709 12:26:57.868798 29792 ts_tablet_manager.cc:1434] T 9633e3b1d78f403097e4a00b7d4b869b P 584dac86183a41d9b3bea1c2085e62fa: Time spent starting tablet: real 0.038s	user 0.011s	sys 0.000s
I20260709 12:26:57.872417 29792 tablet_bootstrap.cc:492] T 2707271fd46f456aa765026642a3efa9 P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap starting.
I20260709 12:26:57.835676 29852 consensus_queue.cc:1048] T cdf706ffabc34902ad3d1bf2cee7889e P d73d92fb35234d16824c55689b5fce28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.037s
W20260709 12:26:57.878356 29708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:35033, user_credentials={real_user=slave}} blocked reactor thread for 78019.2us
I20260709 12:26:57.882427 29670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9633e3b1d78f403097e4a00b7d4b869b" candidate_uuid: "584dac86183a41d9b3bea1c2085e62fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" is_pre_election: true
I20260709 12:26:57.883350 29670 raft_consensus.cc:2468] T 9633e3b1d78f403097e4a00b7d4b869b P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 584dac86183a41d9b3bea1c2085e62fa in term 0.
I20260709 12:26:57.882558 29595 raft_consensus.cc:2468] T 62f8c9472b214576b9002e02536a3ace P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 584dac86183a41d9b3bea1c2085e62fa in term 1.
I20260709 12:26:57.882361 29671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9633e3b1d78f403097e4a00b7d4b869b" candidate_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d13c58ad8ac441c4a9b396eb04b1a476"
I20260709 12:26:57.884829 29671 raft_consensus.cc:3060] T 9633e3b1d78f403097e4a00b7d4b869b P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:26:57.888377 29486 leader_election.cc:343] T 62f8c9472b214576b9002e02536a3ace P 584dac86183a41d9b3bea1c2085e62fa [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:57.883183 29856 leader_election.cc:290] T 9633e3b1d78f403097e4a00b7d4b869b P 584dac86183a41d9b3bea1c2085e62fa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209), c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895)
I20260709 12:26:57.889802 29486 leader_election.cc:304] T 9633e3b1d78f403097e4a00b7d4b869b P 584dac86183a41d9b3bea1c2085e62fa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 584dac86183a41d9b3bea1c2085e62fa, d13c58ad8ac441c4a9b396eb04b1a476; no voters: 
I20260709 12:26:57.892277 29745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9633e3b1d78f403097e4a00b7d4b869b" candidate_uuid: "584dac86183a41d9b3bea1c2085e62fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" is_pre_election: true
I20260709 12:26:57.893174 29745 raft_consensus.cc:2393] T 9633e3b1d78f403097e4a00b7d4b869b P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 584dac86183a41d9b3bea1c2085e62fa in current term 1: Already voted for candidate c6610c00ffb948ce9c34f41dec3d49e8 in this term.
I20260709 12:26:57.893108 29485 leader_election.cc:304] T 62f8c9472b214576b9002e02536a3ace P 584dac86183a41d9b3bea1c2085e62fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 584dac86183a41d9b3bea1c2085e62fa, d73d92fb35234d16824c55689b5fce28; no voters: c6610c00ffb948ce9c34f41dec3d49e8
I20260709 12:26:57.893944 29669 raft_consensus.cc:2410] T 55061e62184b45f0bb1726adc1a4af81 P d13c58ad8ac441c4a9b396eb04b1a476 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate d73d92fb35234d16824c55689b5fce28 for term 2 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20260709 12:26:57.894505 29834 raft_consensus.cc:2804] T 9633e3b1d78f403097e4a00b7d4b869b P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:26:57.894904 29834 raft_consensus.cc:493] T 9633e3b1d78f403097e4a00b7d4b869b P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:57.895253 29834 raft_consensus.cc:3060] T 9633e3b1d78f403097e4a00b7d4b869b P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:57.883237 29522 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9633e3b1d78f403097e4a00b7d4b869b" candidate_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "584dac86183a41d9b3bea1c2085e62fa"
I20260709 12:26:57.900943 29793 tablet_bootstrap.cc:492] T 00c6cf271c91400aa834467def9af9b1 P d73d92fb35234d16824c55689b5fce28: No bootstrap required, opened a new log
I20260709 12:26:57.901474 29793 ts_tablet_manager.cc:1403] T 00c6cf271c91400aa834467def9af9b1 P d73d92fb35234d16824c55689b5fce28: Time spent bootstrapping tablet: real 0.223s	user 0.023s	sys 0.030s
I20260709 12:26:57.904174 29860 raft_consensus.cc:493] T 742beffede4f407d8a72909d2b147e6d P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:57.906224 29835 raft_consensus.cc:493] T b3c02c910d2b43cc9265bb1d98491cd3 P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:57.906769 29835 raft_consensus.cc:515] T b3c02c910d2b43cc9265bb1d98491cd3 P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:26:57.908067 29792 tablet_bootstrap.cc:654] T 2707271fd46f456aa765026642a3efa9 P 584dac86183a41d9b3bea1c2085e62fa: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:57.909065 29835 leader_election.cc:290] T b3c02c910d2b43cc9265bb1d98491cd3 P 584dac86183a41d9b3bea1c2085e62fa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895), d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825)
I20260709 12:26:57.899487 29856 raft_consensus.cc:2804] T 62f8c9472b214576b9002e02536a3ace P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:26:57.924126 29856 raft_consensus.cc:697] T 62f8c9472b214576b9002e02536a3ace P 584dac86183a41d9b3bea1c2085e62fa [term 1 LEADER]: Becoming Leader. State: Replica: 584dac86183a41d9b3bea1c2085e62fa, State: Running, Role: LEADER
I20260709 12:26:57.925137 29856 consensus_queue.cc:237] T 62f8c9472b214576b9002e02536a3ace P 584dac86183a41d9b3bea1c2085e62fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:26:57.911654 29793 raft_consensus.cc:359] T 00c6cf271c91400aa834467def9af9b1 P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:26:57.904722 29860 raft_consensus.cc:515] T 742beffede4f407d8a72909d2b147e6d P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:26:57.913638 29595 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3c02c910d2b43cc9265bb1d98491cd3" candidate_uuid: "584dac86183a41d9b3bea1c2085e62fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d73d92fb35234d16824c55689b5fce28" is_pre_election: true
I20260709 12:26:57.914830 29745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3c02c910d2b43cc9265bb1d98491cd3" candidate_uuid: "584dac86183a41d9b3bea1c2085e62fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" is_pre_election: true
I20260709 12:26:57.926819 29793 raft_consensus.cc:385] T 00c6cf271c91400aa834467def9af9b1 P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260709 12:26:57.929363 29486 leader_election.cc:343] T b3c02c910d2b43cc9265bb1d98491cd3 P 584dac86183a41d9b3bea1c2085e62fa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:57.930160 29860 leader_election.cc:290] T 742beffede4f407d8a72909d2b147e6d P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033), d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209)
I20260709 12:26:57.927626 29793 raft_consensus.cc:740] T 00c6cf271c91400aa834467def9af9b1 P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d73d92fb35234d16824c55689b5fce28, State: Initialized, Role: FOLLOWER
I20260709 12:26:57.932665 29669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "742beffede4f407d8a72909d2b147e6d" candidate_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" is_pre_election: true
I20260709 12:26:57.931912 29793 consensus_queue.cc:260] T 00c6cf271c91400aa834467def9af9b1 P d73d92fb35234d16824c55689b5fce28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:26:57.932709 29519 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "742beffede4f407d8a72909d2b147e6d" candidate_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "584dac86183a41d9b3bea1c2085e62fa" is_pre_election: true
I20260709 12:26:57.936592 29793 ts_tablet_manager.cc:1434] T 00c6cf271c91400aa834467def9af9b1 P d73d92fb35234d16824c55689b5fce28: Time spent starting tablet: real 0.035s	user 0.008s	sys 0.000s
W20260709 12:26:57.937088 29485 leader_election.cc:343] T b3c02c910d2b43cc9265bb1d98491cd3 P 584dac86183a41d9b3bea1c2085e62fa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:57.937601 29485 leader_election.cc:304] T b3c02c910d2b43cc9265bb1d98491cd3 P 584dac86183a41d9b3bea1c2085e62fa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 584dac86183a41d9b3bea1c2085e62fa; no voters: c6610c00ffb948ce9c34f41dec3d49e8, d73d92fb35234d16824c55689b5fce28
I20260709 12:26:57.937898 29793 tablet_bootstrap.cc:492] T 76140ad6258543118a9bfeee7cf0d5fc P d73d92fb35234d16824c55689b5fce28: Bootstrap starting.
I20260709 12:26:57.942075 29835 raft_consensus.cc:2749] T b3c02c910d2b43cc9265bb1d98491cd3 P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:26:57.942214 29708 leader_election.cc:304] T 742beffede4f407d8a72909d2b147e6d P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c6610c00ffb948ce9c34f41dec3d49e8; no voters: 584dac86183a41d9b3bea1c2085e62fa, d13c58ad8ac441c4a9b396eb04b1a476
I20260709 12:26:57.943195 29860 raft_consensus.cc:2749] T 742beffede4f407d8a72909d2b147e6d P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:26:57.948002 29793 tablet_bootstrap.cc:654] T 76140ad6258543118a9bfeee7cf0d5fc P d73d92fb35234d16824c55689b5fce28: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:57.947767 29671 raft_consensus.cc:2468] T 9633e3b1d78f403097e4a00b7d4b869b P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6610c00ffb948ce9c34f41dec3d49e8 in term 1.
I20260709 12:26:57.957515 29709 leader_election.cc:304] T 9633e3b1d78f403097e4a00b7d4b869b P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c6610c00ffb948ce9c34f41dec3d49e8, d13c58ad8ac441c4a9b396eb04b1a476; no voters: 
I20260709 12:26:57.958555 29860 raft_consensus.cc:2804] T 9633e3b1d78f403097e4a00b7d4b869b P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:26:57.959072 29860 raft_consensus.cc:697] T 9633e3b1d78f403097e4a00b7d4b869b P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 LEADER]: Becoming Leader. State: Replica: c6610c00ffb948ce9c34f41dec3d49e8, State: Running, Role: LEADER
I20260709 12:26:57.930893 29834 raft_consensus.cc:515] T 9633e3b1d78f403097e4a00b7d4b869b P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:26:57.965445 29794 tablet_bootstrap.cc:492] T 76140ad6258543118a9bfeee7cf0d5fc P d13c58ad8ac441c4a9b396eb04b1a476: No bootstrap required, opened a new log
I20260709 12:26:57.966182 29794 ts_tablet_manager.cc:1403] T 76140ad6258543118a9bfeee7cf0d5fc P d13c58ad8ac441c4a9b396eb04b1a476: Time spent bootstrapping tablet: real 0.171s	user 0.022s	sys 0.010s
I20260709 12:26:57.968500 29522 raft_consensus.cc:2393] T 9633e3b1d78f403097e4a00b7d4b869b P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate c6610c00ffb948ce9c34f41dec3d49e8 in current term 1: Already voted for candidate 584dac86183a41d9b3bea1c2085e62fa in this term.
I20260709 12:26:57.971853 29794 raft_consensus.cc:359] T 76140ad6258543118a9bfeee7cf0d5fc P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:26:57.972092 29860 consensus_queue.cc:237] T 9633e3b1d78f403097e4a00b7d4b869b P c6610c00ffb948ce9c34f41dec3d49e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:26:57.973174 29794 raft_consensus.cc:385] T 76140ad6258543118a9bfeee7cf0d5fc P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:57.973773 29794 raft_consensus.cc:740] T 76140ad6258543118a9bfeee7cf0d5fc P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d13c58ad8ac441c4a9b396eb04b1a476, State: Initialized, Role: FOLLOWER
I20260709 12:26:57.976725 29834 leader_election.cc:290] T 9633e3b1d78f403097e4a00b7d4b869b P 584dac86183a41d9b3bea1c2085e62fa [CANDIDATE]: Term 1 election: Requested vote from peers d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209), c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895)
I20260709 12:26:57.978349 29794 consensus_queue.cc:260] T 76140ad6258543118a9bfeee7cf0d5fc P d13c58ad8ac441c4a9b396eb04b1a476 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:26:57.981905 29744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9633e3b1d78f403097e4a00b7d4b869b" candidate_uuid: "584dac86183a41d9b3bea1c2085e62fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6610c00ffb948ce9c34f41dec3d49e8"
I20260709 12:26:57.982645 29790 tablet_bootstrap.cc:492] T 43c7bfb33af74d4f912e5153e8a54b3e P c6610c00ffb948ce9c34f41dec3d49e8: No bootstrap required, opened a new log
I20260709 12:26:57.983162 29790 ts_tablet_manager.cc:1403] T 43c7bfb33af74d4f912e5153e8a54b3e P c6610c00ffb948ce9c34f41dec3d49e8: Time spent bootstrapping tablet: real 0.283s	user 0.012s	sys 0.003s
I20260709 12:26:57.986364 29671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9633e3b1d78f403097e4a00b7d4b869b" candidate_uuid: "584dac86183a41d9b3bea1c2085e62fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d13c58ad8ac441c4a9b396eb04b1a476"
I20260709 12:26:57.987300 29671 raft_consensus.cc:2393] T 9633e3b1d78f403097e4a00b7d4b869b P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 584dac86183a41d9b3bea1c2085e62fa in current term 1: Already voted for candidate c6610c00ffb948ce9c34f41dec3d49e8 in this term.
W20260709 12:26:57.988394 29467 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:26:57.989511 29486 leader_election.cc:304] T 9633e3b1d78f403097e4a00b7d4b869b P 584dac86183a41d9b3bea1c2085e62fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 584dac86183a41d9b3bea1c2085e62fa; no voters: c6610c00ffb948ce9c34f41dec3d49e8, d13c58ad8ac441c4a9b396eb04b1a476
I20260709 12:26:57.991225 29835 raft_consensus.cc:2749] T 9633e3b1d78f403097e4a00b7d4b869b P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 12:26:57.990943 29790 raft_consensus.cc:359] T 43c7bfb33af74d4f912e5153e8a54b3e P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:26:57.996079 29790 raft_consensus.cc:385] T 43c7bfb33af74d4f912e5153e8a54b3e P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:57.996279 29794 ts_tablet_manager.cc:1434] T 76140ad6258543118a9bfeee7cf0d5fc P d13c58ad8ac441c4a9b396eb04b1a476: Time spent starting tablet: real 0.030s	user 0.010s	sys 0.000s
I20260709 12:26:57.997876 29794 tablet_bootstrap.cc:492] T c6e621aa77d5474580dfc5949801ccc9 P d13c58ad8ac441c4a9b396eb04b1a476: Bootstrap starting.
I20260709 12:26:57.996415 29790 raft_consensus.cc:740] T 43c7bfb33af74d4f912e5153e8a54b3e P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6610c00ffb948ce9c34f41dec3d49e8, State: Initialized, Role: FOLLOWER
I20260709 12:26:57.998744 29790 consensus_queue.cc:260] T 43c7bfb33af74d4f912e5153e8a54b3e P c6610c00ffb948ce9c34f41dec3d49e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:26:58.001197 29790 ts_tablet_manager.cc:1434] T 43c7bfb33af74d4f912e5153e8a54b3e P c6610c00ffb948ce9c34f41dec3d49e8: Time spent starting tablet: real 0.018s	user 0.005s	sys 0.002s
I20260709 12:26:58.002303 29790 tablet_bootstrap.cc:492] T 62f8c9472b214576b9002e02536a3ace P c6610c00ffb948ce9c34f41dec3d49e8: Bootstrap starting.
I20260709 12:26:58.009052 29793 tablet_bootstrap.cc:492] T 76140ad6258543118a9bfeee7cf0d5fc P d73d92fb35234d16824c55689b5fce28: No bootstrap required, opened a new log
I20260709 12:26:58.011838 29744 raft_consensus.cc:3060] T 742beffede4f407d8a72909d2b147e6d P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:58.014230 29793 ts_tablet_manager.cc:1403] T 76140ad6258543118a9bfeee7cf0d5fc P d73d92fb35234d16824c55689b5fce28: Time spent bootstrapping tablet: real 0.077s	user 0.023s	sys 0.009s
I20260709 12:26:58.016713 29793 raft_consensus.cc:359] T 76140ad6258543118a9bfeee7cf0d5fc P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:26:58.013207 29794 tablet_bootstrap.cc:654] T c6e621aa77d5474580dfc5949801ccc9 P d13c58ad8ac441c4a9b396eb04b1a476: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:58.013337 29792 tablet_bootstrap.cc:492] T 2707271fd46f456aa765026642a3efa9 P 584dac86183a41d9b3bea1c2085e62fa: No bootstrap required, opened a new log
I20260709 12:26:58.017880 29792 ts_tablet_manager.cc:1403] T 2707271fd46f456aa765026642a3efa9 P 584dac86183a41d9b3bea1c2085e62fa: Time spent bootstrapping tablet: real 0.148s	user 0.014s	sys 0.012s
I20260709 12:26:58.019048 29798 raft_consensus.cc:493] T 76140ad6258543118a9bfeee7cf0d5fc P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:58.019694 29798 raft_consensus.cc:515] T 76140ad6258543118a9bfeee7cf0d5fc P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:26:58.021006 29792 raft_consensus.cc:359] T 2707271fd46f456aa765026642a3efa9 P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:26:58.019302 29790 tablet_bootstrap.cc:654] T 62f8c9472b214576b9002e02536a3ace P c6610c00ffb948ce9c34f41dec3d49e8: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:58.021813 29792 raft_consensus.cc:385] T 2707271fd46f456aa765026642a3efa9 P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:58.017458 29793 raft_consensus.cc:385] T 76140ad6258543118a9bfeee7cf0d5fc P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:58.022140 29798 leader_election.cc:290] T 76140ad6258543118a9bfeee7cf0d5fc P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825), c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895)
I20260709 12:26:58.022719 29793 raft_consensus.cc:740] T 76140ad6258543118a9bfeee7cf0d5fc P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d73d92fb35234d16824c55689b5fce28, State: Initialized, Role: FOLLOWER
I20260709 12:26:58.023747 29793 consensus_queue.cc:260] T 76140ad6258543118a9bfeee7cf0d5fc P d73d92fb35234d16824c55689b5fce28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:26:58.026356 29793 ts_tablet_manager.cc:1434] T 76140ad6258543118a9bfeee7cf0d5fc P d73d92fb35234d16824c55689b5fce28: Time spent starting tablet: real 0.012s	user 0.005s	sys 0.000s
I20260709 12:26:58.025724 29417 catalog_manager.cc:5697] T 9633e3b1d78f403097e4a00b7d4b869b P c6610c00ffb948ce9c34f41dec3d49e8 reported cstate change: term changed from 0 to 1, leader changed from <none> to c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68). New cstate: current_term: 1 leader_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:58.027530 29793 tablet_bootstrap.cc:492] T b3c02c910d2b43cc9265bb1d98491cd3 P d73d92fb35234d16824c55689b5fce28: Bootstrap starting.
I20260709 12:26:58.028306 29745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76140ad6258543118a9bfeee7cf0d5fc" candidate_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" is_pre_election: true
I20260709 12:26:58.022169 29792 raft_consensus.cc:740] T 2707271fd46f456aa765026642a3efa9 P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 584dac86183a41d9b3bea1c2085e62fa, State: Initialized, Role: FOLLOWER
W20260709 12:26:58.030205 29635 leader_election.cc:343] T 76140ad6258543118a9bfeee7cf0d5fc P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:58.030107 29792 consensus_queue.cc:260] T 2707271fd46f456aa765026642a3efa9 P 584dac86183a41d9b3bea1c2085e62fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:26:58.031711 29595 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76140ad6258543118a9bfeee7cf0d5fc" candidate_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d73d92fb35234d16824c55689b5fce28" is_pre_election: true
I20260709 12:26:58.032464 29595 raft_consensus.cc:2468] T 76140ad6258543118a9bfeee7cf0d5fc P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d13c58ad8ac441c4a9b396eb04b1a476 in term 0.
I20260709 12:26:58.032734 29792 ts_tablet_manager.cc:1434] T 2707271fd46f456aa765026642a3efa9 P 584dac86183a41d9b3bea1c2085e62fa: Time spent starting tablet: real 0.015s	user 0.006s	sys 0.001s
I20260709 12:26:58.033816 29792 tablet_bootstrap.cc:492] T a46289e8fc824dd5a818e30df6e7ec3e P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap starting.
I20260709 12:26:58.034055 29634 leader_election.cc:304] T 76140ad6258543118a9bfeee7cf0d5fc P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d13c58ad8ac441c4a9b396eb04b1a476, d73d92fb35234d16824c55689b5fce28; no voters: c6610c00ffb948ce9c34f41dec3d49e8
I20260709 12:26:58.035753 29798 raft_consensus.cc:2804] T 76140ad6258543118a9bfeee7cf0d5fc P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:26:58.036221 29798 raft_consensus.cc:493] T 76140ad6258543118a9bfeee7cf0d5fc P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:58.036377 29793 tablet_bootstrap.cc:654] T b3c02c910d2b43cc9265bb1d98491cd3 P d73d92fb35234d16824c55689b5fce28: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:58.036669 29798 raft_consensus.cc:3060] T 76140ad6258543118a9bfeee7cf0d5fc P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:58.038152 29790 tablet_bootstrap.cc:492] T 62f8c9472b214576b9002e02536a3ace P c6610c00ffb948ce9c34f41dec3d49e8: No bootstrap required, opened a new log
I20260709 12:26:58.039254 29790 ts_tablet_manager.cc:1403] T 62f8c9472b214576b9002e02536a3ace P c6610c00ffb948ce9c34f41dec3d49e8: Time spent bootstrapping tablet: real 0.037s	user 0.009s	sys 0.006s
I20260709 12:26:58.042791 29790 raft_consensus.cc:359] T 62f8c9472b214576b9002e02536a3ace P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:26:58.043678 29790 raft_consensus.cc:385] T 62f8c9472b214576b9002e02536a3ace P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:58.049073 29790 raft_consensus.cc:740] T 62f8c9472b214576b9002e02536a3ace P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6610c00ffb948ce9c34f41dec3d49e8, State: Initialized, Role: FOLLOWER
I20260709 12:26:58.050173 29792 tablet_bootstrap.cc:654] T a46289e8fc824dd5a818e30df6e7ec3e P 584dac86183a41d9b3bea1c2085e62fa: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:58.051273 29790 consensus_queue.cc:260] T 62f8c9472b214576b9002e02536a3ace P c6610c00ffb948ce9c34f41dec3d49e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:26:58.057358 29793 tablet_bootstrap.cc:492] T b3c02c910d2b43cc9265bb1d98491cd3 P d73d92fb35234d16824c55689b5fce28: No bootstrap required, opened a new log
I20260709 12:26:58.057711 29790 ts_tablet_manager.cc:1434] T 62f8c9472b214576b9002e02536a3ace P c6610c00ffb948ce9c34f41dec3d49e8: Time spent starting tablet: real 0.018s	user 0.007s	sys 0.000s
I20260709 12:26:58.057894 29793 ts_tablet_manager.cc:1403] T b3c02c910d2b43cc9265bb1d98491cd3 P d73d92fb35234d16824c55689b5fce28: Time spent bootstrapping tablet: real 0.031s	user 0.019s	sys 0.007s
I20260709 12:26:58.058717 29790 tablet_bootstrap.cc:492] T ed057e06dec148a0b9d7062beeb9d005 P c6610c00ffb948ce9c34f41dec3d49e8: Bootstrap starting.
I20260709 12:26:58.060248 29798 raft_consensus.cc:515] T 76140ad6258543118a9bfeee7cf0d5fc P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:26:58.061429 29793 raft_consensus.cc:359] T b3c02c910d2b43cc9265bb1d98491cd3 P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:26:58.064688 29793 raft_consensus.cc:385] T b3c02c910d2b43cc9265bb1d98491cd3 P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:58.064977 29793 raft_consensus.cc:740] T b3c02c910d2b43cc9265bb1d98491cd3 P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d73d92fb35234d16824c55689b5fce28, State: Initialized, Role: FOLLOWER
I20260709 12:26:58.065706 29793 consensus_queue.cc:260] T b3c02c910d2b43cc9265bb1d98491cd3 P d73d92fb35234d16824c55689b5fce28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:26:58.069639 29790 tablet_bootstrap.cc:654] T ed057e06dec148a0b9d7062beeb9d005 P c6610c00ffb948ce9c34f41dec3d49e8: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:58.070753 29793 ts_tablet_manager.cc:1434] T b3c02c910d2b43cc9265bb1d98491cd3 P d73d92fb35234d16824c55689b5fce28: Time spent starting tablet: real 0.013s	user 0.008s	sys 0.001s
I20260709 12:26:58.070832 29852 raft_consensus.cc:493] T 00c6cf271c91400aa834467def9af9b1 P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:58.072208 29793 tablet_bootstrap.cc:492] T 66f652ba2c224885a2aea974bfe8f88a P d73d92fb35234d16824c55689b5fce28: Bootstrap starting.
I20260709 12:26:58.071645 29852 raft_consensus.cc:515] T 00c6cf271c91400aa834467def9af9b1 P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:26:58.075451 29671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00c6cf271c91400aa834467def9af9b1" candidate_uuid: "d73d92fb35234d16824c55689b5fce28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" is_pre_election: true
I20260709 12:26:58.076182 29671 raft_consensus.cc:2468] T 00c6cf271c91400aa834467def9af9b1 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d73d92fb35234d16824c55689b5fce28 in term 0.
I20260709 12:26:58.076284 29852 leader_election.cc:290] T 00c6cf271c91400aa834467def9af9b1 P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209), c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895)
I20260709 12:26:58.077438 29561 leader_election.cc:304] T 00c6cf271c91400aa834467def9af9b1 P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d13c58ad8ac441c4a9b396eb04b1a476, d73d92fb35234d16824c55689b5fce28; no voters: 
I20260709 12:26:58.077765 29744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00c6cf271c91400aa834467def9af9b1" candidate_uuid: "d73d92fb35234d16824c55689b5fce28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" is_pre_election: true
W20260709 12:26:58.082612 29561 leader_election.cc:343] T 00c6cf271c91400aa834467def9af9b1 P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:58.083617 29852 raft_consensus.cc:2804] T 00c6cf271c91400aa834467def9af9b1 P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:26:58.084252 29852 raft_consensus.cc:493] T 00c6cf271c91400aa834467def9af9b1 P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:58.084619 29852 raft_consensus.cc:3060] T 00c6cf271c91400aa834467def9af9b1 P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:58.085011 29595 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76140ad6258543118a9bfeee7cf0d5fc" candidate_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d73d92fb35234d16824c55689b5fce28"
I20260709 12:26:58.085798 29595 raft_consensus.cc:3060] T 76140ad6258543118a9bfeee7cf0d5fc P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:58.093816 29852 raft_consensus.cc:515] T 00c6cf271c91400aa834467def9af9b1 P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:26:58.097805 29671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00c6cf271c91400aa834467def9af9b1" candidate_uuid: "d73d92fb35234d16824c55689b5fce28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d13c58ad8ac441c4a9b396eb04b1a476"
I20260709 12:26:58.097958 29852 leader_election.cc:290] T 00c6cf271c91400aa834467def9af9b1 P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 1 election: Requested vote from peers d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209), c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895)
I20260709 12:26:58.098471 29671 raft_consensus.cc:3060] T 00c6cf271c91400aa834467def9af9b1 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:58.098557 29744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00c6cf271c91400aa834467def9af9b1" candidate_uuid: "d73d92fb35234d16824c55689b5fce28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6610c00ffb948ce9c34f41dec3d49e8"
W20260709 12:26:58.100005 29561 leader_election.cc:343] T 00c6cf271c91400aa834467def9af9b1 P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:58.102248 29794 tablet_bootstrap.cc:492] T c6e621aa77d5474580dfc5949801ccc9 P d13c58ad8ac441c4a9b396eb04b1a476: No bootstrap required, opened a new log
I20260709 12:26:58.102396 29792 tablet_bootstrap.cc:492] T a46289e8fc824dd5a818e30df6e7ec3e P 584dac86183a41d9b3bea1c2085e62fa: No bootstrap required, opened a new log
I20260709 12:26:58.102476 29744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76140ad6258543118a9bfeee7cf0d5fc" candidate_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6610c00ffb948ce9c34f41dec3d49e8"
I20260709 12:26:58.102873 29595 raft_consensus.cc:2468] T 76140ad6258543118a9bfeee7cf0d5fc P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d13c58ad8ac441c4a9b396eb04b1a476 in term 1.
I20260709 12:26:58.100984 29798 leader_election.cc:290] T 76140ad6258543118a9bfeee7cf0d5fc P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 1 election: Requested vote from peers d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825), c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895)
I20260709 12:26:58.104278 29634 leader_election.cc:304] T 76140ad6258543118a9bfeee7cf0d5fc P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d13c58ad8ac441c4a9b396eb04b1a476, d73d92fb35234d16824c55689b5fce28; no voters: 
W20260709 12:26:58.105031 29635 leader_election.cc:343] T 76140ad6258543118a9bfeee7cf0d5fc P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:58.105672 29793 tablet_bootstrap.cc:654] T 66f652ba2c224885a2aea974bfe8f88a P d73d92fb35234d16824c55689b5fce28: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:58.105732 29798 raft_consensus.cc:2804] T 76140ad6258543118a9bfeee7cf0d5fc P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:26:58.106956 29798 raft_consensus.cc:697] T 76140ad6258543118a9bfeee7cf0d5fc P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 LEADER]: Becoming Leader. State: Replica: d13c58ad8ac441c4a9b396eb04b1a476, State: Running, Role: LEADER
I20260709 12:26:58.107852 29798 consensus_queue.cc:237] T 76140ad6258543118a9bfeee7cf0d5fc P d13c58ad8ac441c4a9b396eb04b1a476 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:26:58.111352 29792 ts_tablet_manager.cc:1403] T a46289e8fc824dd5a818e30df6e7ec3e P 584dac86183a41d9b3bea1c2085e62fa: Time spent bootstrapping tablet: real 0.078s	user 0.011s	sys 0.011s
I20260709 12:26:58.114652 29792 raft_consensus.cc:359] T a46289e8fc824dd5a818e30df6e7ec3e P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:26:58.115396 29792 raft_consensus.cc:385] T a46289e8fc824dd5a818e30df6e7ec3e P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:58.115672 29792 raft_consensus.cc:740] T a46289e8fc824dd5a818e30df6e7ec3e P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 584dac86183a41d9b3bea1c2085e62fa, State: Initialized, Role: FOLLOWER
I20260709 12:26:58.111199 29418 catalog_manager.cc:5697] T 62f8c9472b214576b9002e02536a3ace P 584dac86183a41d9b3bea1c2085e62fa reported cstate change: term changed from 0 to 1, leader changed from <none> to 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65). New cstate: current_term: 1 leader_uuid: "584dac86183a41d9b3bea1c2085e62fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:58.120177 29792 consensus_queue.cc:260] T a46289e8fc824dd5a818e30df6e7ec3e P 584dac86183a41d9b3bea1c2085e62fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:26:58.128168 29671 raft_consensus.cc:2468] T 00c6cf271c91400aa834467def9af9b1 P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d73d92fb35234d16824c55689b5fce28 in term 1.
I20260709 12:26:58.128710 29792 ts_tablet_manager.cc:1434] T a46289e8fc824dd5a818e30df6e7ec3e P 584dac86183a41d9b3bea1c2085e62fa: Time spent starting tablet: real 0.017s	user 0.007s	sys 0.000s
I20260709 12:26:58.102809 29794 ts_tablet_manager.cc:1403] T c6e621aa77d5474580dfc5949801ccc9 P d13c58ad8ac441c4a9b396eb04b1a476: Time spent bootstrapping tablet: real 0.105s	user 0.029s	sys 0.016s
I20260709 12:26:58.130182 29561 leader_election.cc:304] T 00c6cf271c91400aa834467def9af9b1 P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d13c58ad8ac441c4a9b396eb04b1a476, d73d92fb35234d16824c55689b5fce28; no voters: c6610c00ffb948ce9c34f41dec3d49e8
I20260709 12:26:58.132087 29851 raft_consensus.cc:2804] T 00c6cf271c91400aa834467def9af9b1 P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:26:58.132859 29851 raft_consensus.cc:697] T 00c6cf271c91400aa834467def9af9b1 P d73d92fb35234d16824c55689b5fce28 [term 1 LEADER]: Becoming Leader. State: Replica: d73d92fb35234d16824c55689b5fce28, State: Running, Role: LEADER
I20260709 12:26:58.132587 29794 raft_consensus.cc:359] T c6e621aa77d5474580dfc5949801ccc9 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:26:58.133627 29792 tablet_bootstrap.cc:492] T 8017a44219404faa8db8ff049c65205e P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap starting.
I20260709 12:26:58.133832 29794 raft_consensus.cc:385] T c6e621aa77d5474580dfc5949801ccc9 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:58.134133 29794 raft_consensus.cc:740] T c6e621aa77d5474580dfc5949801ccc9 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d13c58ad8ac441c4a9b396eb04b1a476, State: Initialized, Role: FOLLOWER
I20260709 12:26:58.134910 29794 consensus_queue.cc:260] T c6e621aa77d5474580dfc5949801ccc9 P d13c58ad8ac441c4a9b396eb04b1a476 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:26:58.134281 29851 consensus_queue.cc:237] T 00c6cf271c91400aa834467def9af9b1 P d73d92fb35234d16824c55689b5fce28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:26:58.144521 29794 ts_tablet_manager.cc:1434] T c6e621aa77d5474580dfc5949801ccc9 P d13c58ad8ac441c4a9b396eb04b1a476: Time spent starting tablet: real 0.015s	user 0.006s	sys 0.001s
I20260709 12:26:58.145584 29794 tablet_bootstrap.cc:492] T 71e2de235adf46d9a2407659080d42ef P d13c58ad8ac441c4a9b396eb04b1a476: Bootstrap starting.
I20260709 12:26:58.149701 29790 tablet_bootstrap.cc:492] T ed057e06dec148a0b9d7062beeb9d005 P c6610c00ffb948ce9c34f41dec3d49e8: No bootstrap required, opened a new log
I20260709 12:26:58.149153 29416 catalog_manager.cc:5697] T 76140ad6258543118a9bfeee7cf0d5fc P d13c58ad8ac441c4a9b396eb04b1a476 reported cstate change: term changed from 0 to 1, leader changed from <none> to d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67). New cstate: current_term: 1 leader_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:58.150223 29790 ts_tablet_manager.cc:1403] T ed057e06dec148a0b9d7062beeb9d005 P c6610c00ffb948ce9c34f41dec3d49e8: Time spent bootstrapping tablet: real 0.092s	user 0.015s	sys 0.000s
I20260709 12:26:58.153182 29793 tablet_bootstrap.cc:492] T 66f652ba2c224885a2aea974bfe8f88a P d73d92fb35234d16824c55689b5fce28: No bootstrap required, opened a new log
I20260709 12:26:58.153694 29793 ts_tablet_manager.cc:1403] T 66f652ba2c224885a2aea974bfe8f88a P d73d92fb35234d16824c55689b5fce28: Time spent bootstrapping tablet: real 0.082s	user 0.025s	sys 0.020s
I20260709 12:26:58.153456 29790 raft_consensus.cc:359] T ed057e06dec148a0b9d7062beeb9d005 P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:26:58.154361 29790 raft_consensus.cc:385] T ed057e06dec148a0b9d7062beeb9d005 P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:58.154633 29790 raft_consensus.cc:740] T ed057e06dec148a0b9d7062beeb9d005 P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6610c00ffb948ce9c34f41dec3d49e8, State: Initialized, Role: FOLLOWER
I20260709 12:26:58.155403 29790 consensus_queue.cc:260] T ed057e06dec148a0b9d7062beeb9d005 P c6610c00ffb948ce9c34f41dec3d49e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:26:58.156843 29793 raft_consensus.cc:359] T 66f652ba2c224885a2aea974bfe8f88a P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:26:58.157608 29793 raft_consensus.cc:385] T 66f652ba2c224885a2aea974bfe8f88a P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:58.157874 29793 raft_consensus.cc:740] T 66f652ba2c224885a2aea974bfe8f88a P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d73d92fb35234d16824c55689b5fce28, State: Initialized, Role: FOLLOWER
I20260709 12:26:58.158581 29793 consensus_queue.cc:260] T 66f652ba2c224885a2aea974bfe8f88a P d73d92fb35234d16824c55689b5fce28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:26:58.159572 29834 raft_consensus.cc:493] T a46289e8fc824dd5a818e30df6e7ec3e P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:58.160205 29834 raft_consensus.cc:515] T a46289e8fc824dd5a818e30df6e7ec3e P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:26:58.161003 29792 tablet_bootstrap.cc:654] T 8017a44219404faa8db8ff049c65205e P 584dac86183a41d9b3bea1c2085e62fa: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:58.161947 29793 ts_tablet_manager.cc:1434] T 66f652ba2c224885a2aea974bfe8f88a P d73d92fb35234d16824c55689b5fce28: Time spent starting tablet: real 0.008s	user 0.004s	sys 0.004s
I20260709 12:26:58.163422 29793 tablet_bootstrap.cc:492] T 2707271fd46f456aa765026642a3efa9 P d73d92fb35234d16824c55689b5fce28: Bootstrap starting.
I20260709 12:26:58.163219 29671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a46289e8fc824dd5a818e30df6e7ec3e" candidate_uuid: "584dac86183a41d9b3bea1c2085e62fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" is_pre_election: true
I20260709 12:26:58.164847 29744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a46289e8fc824dd5a818e30df6e7ec3e" candidate_uuid: "584dac86183a41d9b3bea1c2085e62fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" is_pre_election: true
W20260709 12:26:58.166181 29486 leader_election.cc:343] T a46289e8fc824dd5a818e30df6e7ec3e P 584dac86183a41d9b3bea1c2085e62fa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:58.166555 29834 leader_election.cc:290] T a46289e8fc824dd5a818e30df6e7ec3e P 584dac86183a41d9b3bea1c2085e62fa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209), c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895)
I20260709 12:26:58.159824 29794 tablet_bootstrap.cc:654] T 71e2de235adf46d9a2407659080d42ef P d13c58ad8ac441c4a9b396eb04b1a476: Neither blocks nor log segments found. Creating new log.
W20260709 12:26:58.167651 29486 leader_election.cc:343] T a46289e8fc824dd5a818e30df6e7ec3e P 584dac86183a41d9b3bea1c2085e62fa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:58.168207 29486 leader_election.cc:304] T a46289e8fc824dd5a818e30df6e7ec3e P 584dac86183a41d9b3bea1c2085e62fa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 584dac86183a41d9b3bea1c2085e62fa; no voters: c6610c00ffb948ce9c34f41dec3d49e8, d13c58ad8ac441c4a9b396eb04b1a476
I20260709 12:26:58.169996 29420 catalog_manager.cc:5697] T 55061e62184b45f0bb1726adc1a4af81 P d73d92fb35234d16824c55689b5fce28 reported cstate change: term changed from 1 to 2, leader changed from d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67) to d73d92fb35234d16824c55689b5fce28 (127.16.29.66). New cstate: current_term: 2 leader_uuid: "d73d92fb35234d16824c55689b5fce28" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:58.171501 29744 raft_consensus.cc:1275] T 55061e62184b45f0bb1726adc1a4af81 P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Refusing update from remote peer d73d92fb35234d16824c55689b5fce28: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260709 12:26:58.172338 29834 raft_consensus.cc:2749] T a46289e8fc824dd5a818e30df6e7ec3e P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:26:58.173295 29892 consensus_queue.cc:1048] T 55061e62184b45f0bb1726adc1a4af81 P d73d92fb35234d16824c55689b5fce28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:26:58.173285 29793 tablet_bootstrap.cc:654] T 2707271fd46f456aa765026642a3efa9 P d73d92fb35234d16824c55689b5fce28: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:58.177443 29745 raft_consensus.cc:3060] T 43c7bfb33af74d4f912e5153e8a54b3e P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:58.171446 29790 ts_tablet_manager.cc:1434] T ed057e06dec148a0b9d7062beeb9d005 P c6610c00ffb948ce9c34f41dec3d49e8: Time spent starting tablet: real 0.021s	user 0.007s	sys 0.000s
I20260709 12:26:58.179788 29790 tablet_bootstrap.cc:492] T 3f41ce441fb047c8bbb741b1b9cbc0ec P c6610c00ffb948ce9c34f41dec3d49e8: Bootstrap starting.
I20260709 12:26:58.194942 29792 tablet_bootstrap.cc:492] T 8017a44219404faa8db8ff049c65205e P 584dac86183a41d9b3bea1c2085e62fa: No bootstrap required, opened a new log
I20260709 12:26:58.195744 29792 ts_tablet_manager.cc:1403] T 8017a44219404faa8db8ff049c65205e P 584dac86183a41d9b3bea1c2085e62fa: Time spent bootstrapping tablet: real 0.066s	user 0.022s	sys 0.007s
I20260709 12:26:58.196569 29790 tablet_bootstrap.cc:654] T 3f41ce441fb047c8bbb741b1b9cbc0ec P c6610c00ffb948ce9c34f41dec3d49e8: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:58.198747 29792 raft_consensus.cc:359] T 8017a44219404faa8db8ff049c65205e P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:26:58.199572 29792 raft_consensus.cc:385] T 8017a44219404faa8db8ff049c65205e P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:58.199975 29792 raft_consensus.cc:740] T 8017a44219404faa8db8ff049c65205e P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 584dac86183a41d9b3bea1c2085e62fa, State: Initialized, Role: FOLLOWER
I20260709 12:26:58.200999 29792 consensus_queue.cc:260] T 8017a44219404faa8db8ff049c65205e P 584dac86183a41d9b3bea1c2085e62fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:26:58.214941 29792 ts_tablet_manager.cc:1434] T 8017a44219404faa8db8ff049c65205e P 584dac86183a41d9b3bea1c2085e62fa: Time spent starting tablet: real 0.019s	user 0.008s	sys 0.000s
I20260709 12:26:58.216295 29792 tablet_bootstrap.cc:492] T f4516786e5564e9eac08ca472f79a9f0 P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap starting.
I20260709 12:26:58.220068 29671 raft_consensus.cc:1275] T 55061e62184b45f0bb1726adc1a4af81 P d13c58ad8ac441c4a9b396eb04b1a476 [term 2 FOLLOWER]: Refusing update from remote peer d73d92fb35234d16824c55689b5fce28: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 1. (term mismatch)
I20260709 12:26:58.220635 29671 pending_rounds.cc:85] T 55061e62184b45f0bb1726adc1a4af81 P d13c58ad8ac441c4a9b396eb04b1a476: Aborting all ops after (but not including) 0
I20260709 12:26:58.220887 29671 pending_rounds.cc:107] T 55061e62184b45f0bb1726adc1a4af81 P d13c58ad8ac441c4a9b396eb04b1a476: Aborting uncommitted NO_OP operation due to leader change: 1.1
I20260709 12:26:58.221226 29671 raft_consensus.cc:2889] T 55061e62184b45f0bb1726adc1a4af81 P d13c58ad8ac441c4a9b396eb04b1a476 [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:26:58.230120 29852 consensus_queue.cc:1048] T 55061e62184b45f0bb1726adc1a4af81 P d73d92fb35234d16824c55689b5fce28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:26:58.236454 29790 tablet_bootstrap.cc:492] T 3f41ce441fb047c8bbb741b1b9cbc0ec P c6610c00ffb948ce9c34f41dec3d49e8: No bootstrap required, opened a new log
I20260709 12:26:58.236982 29790 ts_tablet_manager.cc:1403] T 3f41ce441fb047c8bbb741b1b9cbc0ec P c6610c00ffb948ce9c34f41dec3d49e8: Time spent bootstrapping tablet: real 0.058s	user 0.019s	sys 0.007s
I20260709 12:26:58.269356 29793 tablet_bootstrap.cc:492] T 2707271fd46f456aa765026642a3efa9 P d73d92fb35234d16824c55689b5fce28: No bootstrap required, opened a new log
I20260709 12:26:58.269909 29794 tablet_bootstrap.cc:492] T 71e2de235adf46d9a2407659080d42ef P d13c58ad8ac441c4a9b396eb04b1a476: No bootstrap required, opened a new log
I20260709 12:26:58.270059 29793 ts_tablet_manager.cc:1403] T 2707271fd46f456aa765026642a3efa9 P d73d92fb35234d16824c55689b5fce28: Time spent bootstrapping tablet: real 0.107s	user 0.033s	sys 0.012s
I20260709 12:26:58.270427 29794 ts_tablet_manager.cc:1403] T 71e2de235adf46d9a2407659080d42ef P d13c58ad8ac441c4a9b396eb04b1a476: Time spent bootstrapping tablet: real 0.125s	user 0.011s	sys 0.019s
I20260709 12:26:58.273769 29793 raft_consensus.cc:359] T 2707271fd46f456aa765026642a3efa9 P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:26:58.274394 29793 raft_consensus.cc:385] T 2707271fd46f456aa765026642a3efa9 P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:58.274642 29793 raft_consensus.cc:740] T 2707271fd46f456aa765026642a3efa9 P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d73d92fb35234d16824c55689b5fce28, State: Initialized, Role: FOLLOWER
I20260709 12:26:58.275252 29793 consensus_queue.cc:260] T 2707271fd46f456aa765026642a3efa9 P d73d92fb35234d16824c55689b5fce28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:26:58.275553 29858 raft_consensus.cc:493] T ed057e06dec148a0b9d7062beeb9d005 P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:58.276124 29858 raft_consensus.cc:515] T ed057e06dec148a0b9d7062beeb9d005 P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:26:58.277745 29793 ts_tablet_manager.cc:1434] T 2707271fd46f456aa765026642a3efa9 P d73d92fb35234d16824c55689b5fce28: Time spent starting tablet: real 0.007s	user 0.008s	sys 0.000s
I20260709 12:26:58.278842 29793 tablet_bootstrap.cc:492] T 3615115233c34461b85763151751e475 P d73d92fb35234d16824c55689b5fce28: Bootstrap starting.
I20260709 12:26:58.289701 29858 leader_election.cc:290] T ed057e06dec148a0b9d7062beeb9d005 P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825), d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209)
I20260709 12:26:58.292752 29670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed057e06dec148a0b9d7062beeb9d005" candidate_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" is_pre_election: true
I20260709 12:26:58.292896 29595 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed057e06dec148a0b9d7062beeb9d005" candidate_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d73d92fb35234d16824c55689b5fce28" is_pre_election: true
I20260709 12:26:58.293558 29670 raft_consensus.cc:2468] T ed057e06dec148a0b9d7062beeb9d005 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6610c00ffb948ce9c34f41dec3d49e8 in term 0.
W20260709 12:26:58.294939 29708 leader_election.cc:343] T ed057e06dec148a0b9d7062beeb9d005 P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:58.295809 29709 leader_election.cc:304] T ed057e06dec148a0b9d7062beeb9d005 P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c6610c00ffb948ce9c34f41dec3d49e8, d13c58ad8ac441c4a9b396eb04b1a476; no voters: d73d92fb35234d16824c55689b5fce28
I20260709 12:26:58.300817 29792 tablet_bootstrap.cc:654] T f4516786e5564e9eac08ca472f79a9f0 P 584dac86183a41d9b3bea1c2085e62fa: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:58.300853 29794 raft_consensus.cc:359] T 71e2de235adf46d9a2407659080d42ef P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:26:58.303478 29794 raft_consensus.cc:385] T 71e2de235adf46d9a2407659080d42ef P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:58.303818 29794 raft_consensus.cc:740] T 71e2de235adf46d9a2407659080d42ef P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d13c58ad8ac441c4a9b396eb04b1a476, State: Initialized, Role: FOLLOWER
I20260709 12:26:58.304693 29794 consensus_queue.cc:260] T 71e2de235adf46d9a2407659080d42ef P d13c58ad8ac441c4a9b396eb04b1a476 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:26:58.306342 29792 tablet_bootstrap.cc:492] T f4516786e5564e9eac08ca472f79a9f0 P 584dac86183a41d9b3bea1c2085e62fa: No bootstrap required, opened a new log
I20260709 12:26:58.306933 29792 ts_tablet_manager.cc:1403] T f4516786e5564e9eac08ca472f79a9f0 P 584dac86183a41d9b3bea1c2085e62fa: Time spent bootstrapping tablet: real 0.091s	user 0.014s	sys 0.000s
I20260709 12:26:58.307443 29851 raft_consensus.cc:493] T b3c02c910d2b43cc9265bb1d98491cd3 P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:58.308000 29851 raft_consensus.cc:515] T b3c02c910d2b43cc9265bb1d98491cd3 P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:26:58.308683 29420 catalog_manager.cc:5697] T 00c6cf271c91400aa834467def9af9b1 P d73d92fb35234d16824c55689b5fce28 reported cstate change: term changed from 0 to 1, leader changed from <none> to d73d92fb35234d16824c55689b5fce28 (127.16.29.66). New cstate: current_term: 1 leader_uuid: "d73d92fb35234d16824c55689b5fce28" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:58.309653 29858 raft_consensus.cc:2804] T ed057e06dec148a0b9d7062beeb9d005 P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:26:58.313671 29858 raft_consensus.cc:493] T ed057e06dec148a0b9d7062beeb9d005 P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:58.313776 29522 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3c02c910d2b43cc9265bb1d98491cd3" candidate_uuid: "d73d92fb35234d16824c55689b5fce28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "584dac86183a41d9b3bea1c2085e62fa" is_pre_election: true
I20260709 12:26:58.314105 29858 raft_consensus.cc:3060] T ed057e06dec148a0b9d7062beeb9d005 P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:58.314411 29522 raft_consensus.cc:2468] T b3c02c910d2b43cc9265bb1d98491cd3 P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d73d92fb35234d16824c55689b5fce28 in term 0.
I20260709 12:26:58.315320 29560 leader_election.cc:304] T b3c02c910d2b43cc9265bb1d98491cd3 P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 584dac86183a41d9b3bea1c2085e62fa, d73d92fb35234d16824c55689b5fce28; no voters: 
I20260709 12:26:58.315984 29852 raft_consensus.cc:2804] T b3c02c910d2b43cc9265bb1d98491cd3 P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:26:58.307184 29790 raft_consensus.cc:359] T 3f41ce441fb047c8bbb741b1b9cbc0ec P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:26:58.310267 29851 leader_election.cc:290] T b3c02c910d2b43cc9265bb1d98491cd3 P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895), 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033)
I20260709 12:26:58.316280 29852 raft_consensus.cc:493] T b3c02c910d2b43cc9265bb1d98491cd3 P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:58.316658 29790 raft_consensus.cc:385] T 3f41ce441fb047c8bbb741b1b9cbc0ec P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:58.311872 29793 tablet_bootstrap.cc:654] T 3615115233c34461b85763151751e475 P d73d92fb35234d16824c55689b5fce28: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:58.311825 29745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3c02c910d2b43cc9265bb1d98491cd3" candidate_uuid: "d73d92fb35234d16824c55689b5fce28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" is_pre_election: true
I20260709 12:26:58.317216 29790 raft_consensus.cc:740] T 3f41ce441fb047c8bbb741b1b9cbc0ec P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6610c00ffb948ce9c34f41dec3d49e8, State: Initialized, Role: FOLLOWER
I20260709 12:26:58.318055 29790 consensus_queue.cc:260] T 3f41ce441fb047c8bbb741b1b9cbc0ec P c6610c00ffb948ce9c34f41dec3d49e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:26:58.316654 29852 raft_consensus.cc:3060] T b3c02c910d2b43cc9265bb1d98491cd3 P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:26:58.318364 29561 leader_election.cc:343] T b3c02c910d2b43cc9265bb1d98491cd3 P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:58.310243 29792 raft_consensus.cc:359] T f4516786e5564e9eac08ca472f79a9f0 P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:26:58.313236 29794 ts_tablet_manager.cc:1434] T 71e2de235adf46d9a2407659080d42ef P d13c58ad8ac441c4a9b396eb04b1a476: Time spent starting tablet: real 0.043s	user 0.008s	sys 0.000s
I20260709 12:26:58.353348 29792 raft_consensus.cc:385] T f4516786e5564e9eac08ca472f79a9f0 P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:58.353780 29792 raft_consensus.cc:740] T f4516786e5564e9eac08ca472f79a9f0 P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 584dac86183a41d9b3bea1c2085e62fa, State: Initialized, Role: FOLLOWER
I20260709 12:26:58.354537 29792 consensus_queue.cc:260] T f4516786e5564e9eac08ca472f79a9f0 P 584dac86183a41d9b3bea1c2085e62fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:26:58.356679 29792 ts_tablet_manager.cc:1434] T f4516786e5564e9eac08ca472f79a9f0 P 584dac86183a41d9b3bea1c2085e62fa: Time spent starting tablet: real 0.049s	user 0.012s	sys 0.022s
I20260709 12:26:58.361650 29792 tablet_bootstrap.cc:492] T 71e2de235adf46d9a2407659080d42ef P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap starting.
I20260709 12:26:58.364861 29794 tablet_bootstrap.cc:492] T f4516786e5564e9eac08ca472f79a9f0 P d13c58ad8ac441c4a9b396eb04b1a476: Bootstrap starting.
W20260709 12:26:58.367177 29561 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:43895, user_credentials={real_user=slave}} blocked reactor thread for 48850.7us
I20260709 12:26:58.367797 29792 tablet_bootstrap.cc:654] T 71e2de235adf46d9a2407659080d42ef P 584dac86183a41d9b3bea1c2085e62fa: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:58.335417 29834 raft_consensus.cc:493] T 2707271fd46f456aa765026642a3efa9 P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:58.368983 29834 raft_consensus.cc:515] T 2707271fd46f456aa765026642a3efa9 P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:26:58.358642 29790 ts_tablet_manager.cc:1434] T 3f41ce441fb047c8bbb741b1b9cbc0ec P c6610c00ffb948ce9c34f41dec3d49e8: Time spent starting tablet: real 0.121s	user 0.016s	sys 0.025s
I20260709 12:26:58.373787 29671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2707271fd46f456aa765026642a3efa9" candidate_uuid: "584dac86183a41d9b3bea1c2085e62fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" is_pre_election: true
I20260709 12:26:58.374519 29671 raft_consensus.cc:2468] T 2707271fd46f456aa765026642a3efa9 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 584dac86183a41d9b3bea1c2085e62fa in term 0.
I20260709 12:26:58.376178 29486 leader_election.cc:304] T 2707271fd46f456aa765026642a3efa9 P 584dac86183a41d9b3bea1c2085e62fa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 584dac86183a41d9b3bea1c2085e62fa, d13c58ad8ac441c4a9b396eb04b1a476; no voters: 
I20260709 12:26:58.377851 29856 raft_consensus.cc:2804] T 2707271fd46f456aa765026642a3efa9 P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:26:58.323607 29858 raft_consensus.cc:515] T ed057e06dec148a0b9d7062beeb9d005 P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:26:58.378438 29856 raft_consensus.cc:493] T 2707271fd46f456aa765026642a3efa9 P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:58.378969 29856 raft_consensus.cc:3060] T 2707271fd46f456aa765026642a3efa9 P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:58.381389 29790 tablet_bootstrap.cc:492] T 58154d49c1a344f2b51366ede3c87de5 P c6610c00ffb948ce9c34f41dec3d49e8: Bootstrap starting.
I20260709 12:26:58.381152 29671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed057e06dec148a0b9d7062beeb9d005" candidate_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d13c58ad8ac441c4a9b396eb04b1a476"
I20260709 12:26:58.381857 29671 raft_consensus.cc:3060] T ed057e06dec148a0b9d7062beeb9d005 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:58.384279 29858 leader_election.cc:290] T ed057e06dec148a0b9d7062beeb9d005 P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 1 election: Requested vote from peers d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825), d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209)
I20260709 12:26:58.393352 29671 raft_consensus.cc:2468] T ed057e06dec148a0b9d7062beeb9d005 P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6610c00ffb948ce9c34f41dec3d49e8 in term 1.
I20260709 12:26:58.394810 29709 leader_election.cc:304] T ed057e06dec148a0b9d7062beeb9d005 P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c6610c00ffb948ce9c34f41dec3d49e8, d13c58ad8ac441c4a9b396eb04b1a476; no voters: 
I20260709 12:26:58.395941 29858 raft_consensus.cc:2804] T ed057e06dec148a0b9d7062beeb9d005 P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:26:58.396445 29790 tablet_bootstrap.cc:654] T 58154d49c1a344f2b51366ede3c87de5 P c6610c00ffb948ce9c34f41dec3d49e8: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:58.396435 29858 raft_consensus.cc:697] T ed057e06dec148a0b9d7062beeb9d005 P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 LEADER]: Becoming Leader. State: Replica: c6610c00ffb948ce9c34f41dec3d49e8, State: Running, Role: LEADER
I20260709 12:26:58.397471 29858 consensus_queue.cc:237] T ed057e06dec148a0b9d7062beeb9d005 P c6610c00ffb948ce9c34f41dec3d49e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:26:58.399010 29595 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2707271fd46f456aa765026642a3efa9" candidate_uuid: "584dac86183a41d9b3bea1c2085e62fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d73d92fb35234d16824c55689b5fce28" is_pre_election: true
I20260709 12:26:58.400053 29595 raft_consensus.cc:2468] T 2707271fd46f456aa765026642a3efa9 P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 584dac86183a41d9b3bea1c2085e62fa in term 0.
I20260709 12:26:58.408836 29790 tablet_bootstrap.cc:492] T 58154d49c1a344f2b51366ede3c87de5 P c6610c00ffb948ce9c34f41dec3d49e8: No bootstrap required, opened a new log
I20260709 12:26:58.409466 29790 ts_tablet_manager.cc:1403] T 58154d49c1a344f2b51366ede3c87de5 P c6610c00ffb948ce9c34f41dec3d49e8: Time spent bootstrapping tablet: real 0.028s	user 0.019s	sys 0.000s
I20260709 12:26:58.412883 29790 raft_consensus.cc:359] T 58154d49c1a344f2b51366ede3c87de5 P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:26:58.413585 29790 raft_consensus.cc:385] T 58154d49c1a344f2b51366ede3c87de5 P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:58.413847 29790 raft_consensus.cc:740] T 58154d49c1a344f2b51366ede3c87de5 P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6610c00ffb948ce9c34f41dec3d49e8, State: Initialized, Role: FOLLOWER
I20260709 12:26:58.414582 29790 consensus_queue.cc:260] T 58154d49c1a344f2b51366ede3c87de5 P c6610c00ffb948ce9c34f41dec3d49e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:26:58.417032 29790 ts_tablet_manager.cc:1434] T 58154d49c1a344f2b51366ede3c87de5 P c6610c00ffb948ce9c34f41dec3d49e8: Time spent starting tablet: real 0.007s	user 0.008s	sys 0.000s
I20260709 12:26:58.418087 29790 tablet_bootstrap.cc:492] T 00c6cf271c91400aa834467def9af9b1 P c6610c00ffb948ce9c34f41dec3d49e8: Bootstrap starting.
I20260709 12:26:58.375633 29834 leader_election.cc:290] T 2707271fd46f456aa765026642a3efa9 P 584dac86183a41d9b3bea1c2085e62fa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825), d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209)
I20260709 12:26:58.418999 29416 catalog_manager.cc:5697] T ed057e06dec148a0b9d7062beeb9d005 P c6610c00ffb948ce9c34f41dec3d49e8 reported cstate change: term changed from 0 to 1, leader changed from <none> to c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68). New cstate: current_term: 1 leader_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } health_report { overall_health: HEALTHY } } }
I20260709 12:26:58.424254 29790 tablet_bootstrap.cc:654] T 00c6cf271c91400aa834467def9af9b1 P c6610c00ffb948ce9c34f41dec3d49e8: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:58.429502 29798 raft_consensus.cc:493] T 71e2de235adf46d9a2407659080d42ef P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:58.430265 29798 raft_consensus.cc:515] T 71e2de235adf46d9a2407659080d42ef P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:26:58.433830 29522 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "71e2de235adf46d9a2407659080d42ef" candidate_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "584dac86183a41d9b3bea1c2085e62fa" is_pre_election: true
I20260709 12:26:58.434068 29798 leader_election.cc:290] T 71e2de235adf46d9a2407659080d42ef P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033), d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825)
W20260709 12:26:58.435194 29634 leader_election.cc:343] T 71e2de235adf46d9a2407659080d42ef P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:58.437556 29798 raft_consensus.cc:493] T c6e621aa77d5474580dfc5949801ccc9 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:58.438226 29798 raft_consensus.cc:515] T c6e621aa77d5474580dfc5949801ccc9 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:26:58.442904 29522 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6e621aa77d5474580dfc5949801ccc9" candidate_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "584dac86183a41d9b3bea1c2085e62fa" is_pre_election: true
W20260709 12:26:58.444268 29634 leader_election.cc:343] T c6e621aa77d5474580dfc5949801ccc9 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:58.444991 29798 leader_election.cc:290] T c6e621aa77d5474580dfc5949801ccc9 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033), d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825)
I20260709 12:26:58.451686 29790 tablet_bootstrap.cc:492] T 00c6cf271c91400aa834467def9af9b1 P c6610c00ffb948ce9c34f41dec3d49e8: No bootstrap required, opened a new log
I20260709 12:26:58.452303 29790 ts_tablet_manager.cc:1403] T 00c6cf271c91400aa834467def9af9b1 P c6610c00ffb948ce9c34f41dec3d49e8: Time spent bootstrapping tablet: real 0.034s	user 0.014s	sys 0.018s
I20260709 12:26:58.453004 29858 raft_consensus.cc:493] T 3f41ce441fb047c8bbb741b1b9cbc0ec P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:58.453506 29858 raft_consensus.cc:515] T 3f41ce441fb047c8bbb741b1b9cbc0ec P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:26:58.455204 29790 raft_consensus.cc:359] T 00c6cf271c91400aa834467def9af9b1 P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:26:58.455683 29858 leader_election.cc:290] T 3f41ce441fb047c8bbb741b1b9cbc0ec P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209), 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033)
I20260709 12:26:58.455962 29790 raft_consensus.cc:385] T 00c6cf271c91400aa834467def9af9b1 P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:58.456264 29790 raft_consensus.cc:740] T 00c6cf271c91400aa834467def9af9b1 P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6610c00ffb948ce9c34f41dec3d49e8, State: Initialized, Role: FOLLOWER
I20260709 12:26:58.456935 29790 consensus_queue.cc:260] T 00c6cf271c91400aa834467def9af9b1 P c6610c00ffb948ce9c34f41dec3d49e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:26:58.457928 29671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3f41ce441fb047c8bbb741b1b9cbc0ec" candidate_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" is_pre_election: true
I20260709 12:26:58.458958 29790 ts_tablet_manager.cc:1434] T 00c6cf271c91400aa834467def9af9b1 P c6610c00ffb948ce9c34f41dec3d49e8: Time spent starting tablet: real 0.006s	user 0.004s	sys 0.000s
W20260709 12:26:58.459540 29709 leader_election.cc:343] T 3f41ce441fb047c8bbb741b1b9cbc0ec P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:58.462116 29790 tablet_bootstrap.cc:492] T a46289e8fc824dd5a818e30df6e7ec3e P c6610c00ffb948ce9c34f41dec3d49e8: Bootstrap starting.
I20260709 12:26:58.471293 29952 raft_consensus.cc:493] T 8017a44219404faa8db8ff049c65205e P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:58.471872 29952 raft_consensus.cc:515] T 8017a44219404faa8db8ff049c65205e P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:26:58.473022 29744 raft_consensus.cc:3060] T 62f8c9472b214576b9002e02536a3ace P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:58.474053 29595 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed057e06dec148a0b9d7062beeb9d005" candidate_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d73d92fb35234d16824c55689b5fce28"
I20260709 12:26:58.481806 29522 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3f41ce441fb047c8bbb741b1b9cbc0ec" candidate_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "584dac86183a41d9b3bea1c2085e62fa" is_pre_election: true
W20260709 12:26:58.483546 29708 leader_election.cc:343] T 3f41ce441fb047c8bbb741b1b9cbc0ec P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:58.484441 29708 leader_election.cc:304] T 3f41ce441fb047c8bbb741b1b9cbc0ec P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c6610c00ffb948ce9c34f41dec3d49e8; no voters: 584dac86183a41d9b3bea1c2085e62fa, d13c58ad8ac441c4a9b396eb04b1a476
I20260709 12:26:58.485414 29790 tablet_bootstrap.cc:654] T a46289e8fc824dd5a818e30df6e7ec3e P c6610c00ffb948ce9c34f41dec3d49e8: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:58.485469 29858 raft_consensus.cc:2749] T 3f41ce441fb047c8bbb741b1b9cbc0ec P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:26:58.487022 29953 consensus_queue.cc:1048] T 62f8c9472b214576b9002e02536a3ace P 584dac86183a41d9b3bea1c2085e62fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:26:58.483578 29952 leader_election.cc:290] T 8017a44219404faa8db8ff049c65205e P 584dac86183a41d9b3bea1c2085e62fa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895), d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825)
W20260709 12:26:58.490427 29708 leader_election.cc:343] T ed057e06dec148a0b9d7062beeb9d005 P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:58.490615 29790 tablet_bootstrap.cc:492] T a46289e8fc824dd5a818e30df6e7ec3e P c6610c00ffb948ce9c34f41dec3d49e8: No bootstrap required, opened a new log
I20260709 12:26:58.491185 29790 ts_tablet_manager.cc:1403] T a46289e8fc824dd5a818e30df6e7ec3e P c6610c00ffb948ce9c34f41dec3d49e8: Time spent bootstrapping tablet: real 0.029s	user 0.011s	sys 0.000s
I20260709 12:26:58.491406 29595 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8017a44219404faa8db8ff049c65205e" candidate_uuid: "584dac86183a41d9b3bea1c2085e62fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d73d92fb35234d16824c55689b5fce28" is_pre_election: true
I20260709 12:26:58.493991 29790 raft_consensus.cc:359] T a46289e8fc824dd5a818e30df6e7ec3e P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:26:58.494709 29790 raft_consensus.cc:385] T a46289e8fc824dd5a818e30df6e7ec3e P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:58.494994 29790 raft_consensus.cc:740] T a46289e8fc824dd5a818e30df6e7ec3e P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6610c00ffb948ce9c34f41dec3d49e8, State: Initialized, Role: FOLLOWER
I20260709 12:26:58.496379 29790 consensus_queue.cc:260] T a46289e8fc824dd5a818e30df6e7ec3e P c6610c00ffb948ce9c34f41dec3d49e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:26:58.495391 29793 tablet_bootstrap.cc:492] T 3615115233c34461b85763151751e475 P d73d92fb35234d16824c55689b5fce28: No bootstrap required, opened a new log
I20260709 12:26:58.497632 29793 ts_tablet_manager.cc:1403] T 3615115233c34461b85763151751e475 P d73d92fb35234d16824c55689b5fce28: Time spent bootstrapping tablet: real 0.219s	user 0.004s	sys 0.011s
I20260709 12:26:58.467113 29856 raft_consensus.cc:515] T 2707271fd46f456aa765026642a3efa9 P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:26:58.499444 29593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "71e2de235adf46d9a2407659080d42ef" candidate_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d73d92fb35234d16824c55689b5fce28" is_pre_election: true
I20260709 12:26:58.500792 29794 tablet_bootstrap.cc:654] T f4516786e5564e9eac08ca472f79a9f0 P d13c58ad8ac441c4a9b396eb04b1a476: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:58.501013 29593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6e621aa77d5474580dfc5949801ccc9" candidate_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d73d92fb35234d16824c55689b5fce28" is_pre_election: true
I20260709 12:26:58.501266 29744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8017a44219404faa8db8ff049c65205e" candidate_uuid: "584dac86183a41d9b3bea1c2085e62fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" is_pre_election: true
I20260709 12:26:58.505234 29799 raft_consensus.cc:493] T a46289e8fc824dd5a818e30df6e7ec3e P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:58.505844 29799 raft_consensus.cc:515] T a46289e8fc824dd5a818e30df6e7ec3e P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:26:58.506489 29856 leader_election.cc:290] T 2707271fd46f456aa765026642a3efa9 P 584dac86183a41d9b3bea1c2085e62fa [CANDIDATE]: Term 1 election: Requested vote from peers d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825), d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209)
W20260709 12:26:58.510589 29486 leader_election.cc:343] T 8017a44219404faa8db8ff049c65205e P 584dac86183a41d9b3bea1c2085e62fa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:58.515442 29792 tablet_bootstrap.cc:492] T 71e2de235adf46d9a2407659080d42ef P 584dac86183a41d9b3bea1c2085e62fa: No bootstrap required, opened a new log
I20260709 12:26:58.516027 29792 ts_tablet_manager.cc:1403] T 71e2de235adf46d9a2407659080d42ef P 584dac86183a41d9b3bea1c2085e62fa: Time spent bootstrapping tablet: real 0.155s	user 0.054s	sys 0.050s
I20260709 12:26:58.516373 29670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2707271fd46f456aa765026642a3efa9" candidate_uuid: "584dac86183a41d9b3bea1c2085e62fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d13c58ad8ac441c4a9b396eb04b1a476"
I20260709 12:26:58.517017 29670 raft_consensus.cc:3060] T 2707271fd46f456aa765026642a3efa9 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:58.519032 29792 raft_consensus.cc:359] T 71e2de235adf46d9a2407659080d42ef P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:26:58.520777 29851 raft_consensus.cc:493] T 66f652ba2c224885a2aea974bfe8f88a P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:58.520851 29860 consensus_queue.cc:1048] T 9633e3b1d78f403097e4a00b7d4b869b P c6610c00ffb948ce9c34f41dec3d49e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:26:58.521392 29851 raft_consensus.cc:515] T 66f652ba2c224885a2aea974bfe8f88a P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:26:58.522708 29790 ts_tablet_manager.cc:1434] T a46289e8fc824dd5a818e30df6e7ec3e P c6610c00ffb948ce9c34f41dec3d49e8: Time spent starting tablet: real 0.031s	user 0.006s	sys 0.000s
I20260709 12:26:58.523624 29851 leader_election.cc:290] T 66f652ba2c224885a2aea974bfe8f88a P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895), 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033)
I20260709 12:26:58.523805 29790 tablet_bootstrap.cc:492] T 66f652ba2c224885a2aea974bfe8f88a P c6610c00ffb948ce9c34f41dec3d49e8: Bootstrap starting.
I20260709 12:26:58.525236 29793 raft_consensus.cc:359] T 3615115233c34461b85763151751e475 P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:26:58.519776 29792 raft_consensus.cc:385] T 71e2de235adf46d9a2407659080d42ef P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:58.525930 29793 raft_consensus.cc:385] T 3615115233c34461b85763151751e475 P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:58.526181 29792 raft_consensus.cc:740] T 71e2de235adf46d9a2407659080d42ef P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 584dac86183a41d9b3bea1c2085e62fa, State: Initialized, Role: FOLLOWER
I20260709 12:26:58.526355 29522 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66f652ba2c224885a2aea974bfe8f88a" candidate_uuid: "d73d92fb35234d16824c55689b5fce28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "584dac86183a41d9b3bea1c2085e62fa" is_pre_election: true
I20260709 12:26:58.526309 29793 raft_consensus.cc:740] T 3615115233c34461b85763151751e475 P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d73d92fb35234d16824c55689b5fce28, State: Initialized, Role: FOLLOWER
I20260709 12:26:58.515518 29671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a46289e8fc824dd5a818e30df6e7ec3e" candidate_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" is_pre_election: true
I20260709 12:26:58.527839 29793 consensus_queue.cc:260] T 3615115233c34461b85763151751e475 P d73d92fb35234d16824c55689b5fce28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:26:58.528172 29792 consensus_queue.cc:260] T 71e2de235adf46d9a2407659080d42ef P 584dac86183a41d9b3bea1c2085e62fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
W20260709 12:26:58.528947 29709 leader_election.cc:343] T a46289e8fc824dd5a818e30df6e7ec3e P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:58.509804 29799 leader_election.cc:290] T a46289e8fc824dd5a818e30df6e7ec3e P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209), 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033)
I20260709 12:26:58.532410 29790 tablet_bootstrap.cc:654] T 66f652ba2c224885a2aea974bfe8f88a P c6610c00ffb948ce9c34f41dec3d49e8: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:58.536612 29792 ts_tablet_manager.cc:1434] T 71e2de235adf46d9a2407659080d42ef P 584dac86183a41d9b3bea1c2085e62fa: Time spent starting tablet: real 0.020s	user 0.007s	sys 0.000s
I20260709 12:26:58.538201 29792 tablet_bootstrap.cc:492] T 3615115233c34461b85763151751e475 P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap starting.
I20260709 12:26:58.538978 29670 raft_consensus.cc:2468] T 2707271fd46f456aa765026642a3efa9 P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 584dac86183a41d9b3bea1c2085e62fa in term 1.
I20260709 12:26:58.540679 29486 leader_election.cc:304] T 2707271fd46f456aa765026642a3efa9 P 584dac86183a41d9b3bea1c2085e62fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 584dac86183a41d9b3bea1c2085e62fa, d13c58ad8ac441c4a9b396eb04b1a476; no voters: 
W20260709 12:26:58.540700 29560 leader_election.cc:343] T 66f652ba2c224885a2aea974bfe8f88a P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:58.541687 29856 raft_consensus.cc:2804] T 2707271fd46f456aa765026642a3efa9 P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:26:58.542339 29856 raft_consensus.cc:697] T 2707271fd46f456aa765026642a3efa9 P 584dac86183a41d9b3bea1c2085e62fa [term 1 LEADER]: Becoming Leader. State: Replica: 584dac86183a41d9b3bea1c2085e62fa, State: Running, Role: LEADER
I20260709 12:26:58.543375 29856 consensus_queue.cc:237] T 2707271fd46f456aa765026642a3efa9 P 584dac86183a41d9b3bea1c2085e62fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
W20260709 12:26:58.544929 29634 leader_election.cc:343] T 71e2de235adf46d9a2407659080d42ef P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:58.545428 29634 leader_election.cc:304] T 71e2de235adf46d9a2407659080d42ef P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d13c58ad8ac441c4a9b396eb04b1a476; no voters: 584dac86183a41d9b3bea1c2085e62fa, d73d92fb35234d16824c55689b5fce28
I20260709 12:26:58.546607 29798 raft_consensus.cc:2749] T 71e2de235adf46d9a2407659080d42ef P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:26:58.501775 29852 raft_consensus.cc:515] T b3c02c910d2b43cc9265bb1d98491cd3 P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:26:58.550477 29852 leader_election.cc:290] T b3c02c910d2b43cc9265bb1d98491cd3 P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 1 election: Requested vote from peers c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895), 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033)
I20260709 12:26:58.551930 29593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2707271fd46f456aa765026642a3efa9" candidate_uuid: "584dac86183a41d9b3bea1c2085e62fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d73d92fb35234d16824c55689b5fce28"
I20260709 12:26:58.551993 29519 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3c02c910d2b43cc9265bb1d98491cd3" candidate_uuid: "d73d92fb35234d16824c55689b5fce28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "584dac86183a41d9b3bea1c2085e62fa"
I20260709 12:26:58.552722 29593 raft_consensus.cc:3060] T 2707271fd46f456aa765026642a3efa9 P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:58.552757 29519 raft_consensus.cc:3060] T b3c02c910d2b43cc9265bb1d98491cd3 P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:58.554037 29793 ts_tablet_manager.cc:1434] T 3615115233c34461b85763151751e475 P d73d92fb35234d16824c55689b5fce28: Time spent starting tablet: real 0.056s	user 0.007s	sys 0.001s
I20260709 12:26:58.554764 29794 tablet_bootstrap.cc:492] T f4516786e5564e9eac08ca472f79a9f0 P d13c58ad8ac441c4a9b396eb04b1a476: No bootstrap required, opened a new log
I20260709 12:26:58.555234 29794 ts_tablet_manager.cc:1403] T f4516786e5564e9eac08ca472f79a9f0 P d13c58ad8ac441c4a9b396eb04b1a476: Time spent bootstrapping tablet: real 0.191s	user 0.012s	sys 0.010s
I20260709 12:26:58.558904 29794 raft_consensus.cc:359] T f4516786e5564e9eac08ca472f79a9f0 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:26:58.559593 29794 raft_consensus.cc:385] T f4516786e5564e9eac08ca472f79a9f0 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:58.559863 29794 raft_consensus.cc:740] T f4516786e5564e9eac08ca472f79a9f0 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d13c58ad8ac441c4a9b396eb04b1a476, State: Initialized, Role: FOLLOWER
I20260709 12:26:58.560724 29794 consensus_queue.cc:260] T f4516786e5564e9eac08ca472f79a9f0 P d13c58ad8ac441c4a9b396eb04b1a476 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:26:58.560781 29522 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a46289e8fc824dd5a818e30df6e7ec3e" candidate_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "584dac86183a41d9b3bea1c2085e62fa" is_pre_election: true
I20260709 12:26:58.560920 29519 raft_consensus.cc:2468] T b3c02c910d2b43cc9265bb1d98491cd3 P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d73d92fb35234d16824c55689b5fce28 in term 1.
I20260709 12:26:58.561767 29522 raft_consensus.cc:2468] T a46289e8fc824dd5a818e30df6e7ec3e P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6610c00ffb948ce9c34f41dec3d49e8 in term 0.
I20260709 12:26:58.562826 29794 ts_tablet_manager.cc:1434] T f4516786e5564e9eac08ca472f79a9f0 P d13c58ad8ac441c4a9b396eb04b1a476: Time spent starting tablet: real 0.007s	user 0.000s	sys 0.006s
I20260709 12:26:58.563843 29794 tablet_bootstrap.cc:492] T a46289e8fc824dd5a818e30df6e7ec3e P d13c58ad8ac441c4a9b396eb04b1a476: Bootstrap starting.
I20260709 12:26:58.564640 29560 leader_election.cc:304] T b3c02c910d2b43cc9265bb1d98491cd3 P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 584dac86183a41d9b3bea1c2085e62fa, d73d92fb35234d16824c55689b5fce28; no voters: 
I20260709 12:26:58.560843 29793 tablet_bootstrap.cc:492] T 71e2de235adf46d9a2407659080d42ef P d73d92fb35234d16824c55689b5fce28: Bootstrap starting.
W20260709 12:26:58.565933 29634 leader_election.cc:343] T c6e621aa77d5474580dfc5949801ccc9 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:58.566541 29634 leader_election.cc:304] T c6e621aa77d5474580dfc5949801ccc9 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d13c58ad8ac441c4a9b396eb04b1a476; no voters: 584dac86183a41d9b3bea1c2085e62fa, d73d92fb35234d16824c55689b5fce28
I20260709 12:26:58.567901 29798 raft_consensus.cc:2749] T c6e621aa77d5474580dfc5949801ccc9 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:26:58.569905 29794 tablet_bootstrap.cc:654] T a46289e8fc824dd5a818e30df6e7ec3e P d13c58ad8ac441c4a9b396eb04b1a476: Neither blocks nor log segments found. Creating new log.
W20260709 12:26:58.570194 29485 leader_election.cc:343] T 8017a44219404faa8db8ff049c65205e P 584dac86183a41d9b3bea1c2085e62fa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:58.570710 29485 leader_election.cc:304] T 8017a44219404faa8db8ff049c65205e P 584dac86183a41d9b3bea1c2085e62fa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 584dac86183a41d9b3bea1c2085e62fa; no voters: c6610c00ffb948ce9c34f41dec3d49e8, d73d92fb35234d16824c55689b5fce28
I20260709 12:26:58.563865 29593 raft_consensus.cc:2468] T 2707271fd46f456aa765026642a3efa9 P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 584dac86183a41d9b3bea1c2085e62fa in term 1.
I20260709 12:26:58.574898 29744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66f652ba2c224885a2aea974bfe8f88a" candidate_uuid: "d73d92fb35234d16824c55689b5fce28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" is_pre_election: true
I20260709 12:26:58.575191 29793 tablet_bootstrap.cc:654] T 71e2de235adf46d9a2407659080d42ef P d73d92fb35234d16824c55689b5fce28: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:58.575332 29792 tablet_bootstrap.cc:654] T 3615115233c34461b85763151751e475 P 584dac86183a41d9b3bea1c2085e62fa: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:58.577641 29856 raft_consensus.cc:2749] T 8017a44219404faa8db8ff049c65205e P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:26:58.581447 29744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3c02c910d2b43cc9265bb1d98491cd3" candidate_uuid: "d73d92fb35234d16824c55689b5fce28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6610c00ffb948ce9c34f41dec3d49e8"
I20260709 12:26:58.582515 29858 raft_consensus.cc:493] T 62f8c9472b214576b9002e02536a3ace P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 584dac86183a41d9b3bea1c2085e62fa)
I20260709 12:26:58.583206 29858 raft_consensus.cc:515] T 62f8c9472b214576b9002e02536a3ace P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:26:58.584038 29708 leader_election.cc:304] T a46289e8fc824dd5a818e30df6e7ec3e P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 584dac86183a41d9b3bea1c2085e62fa, c6610c00ffb948ce9c34f41dec3d49e8; no voters: d13c58ad8ac441c4a9b396eb04b1a476
I20260709 12:26:58.584373 29799 raft_consensus.cc:493] T 00c6cf271c91400aa834467def9af9b1 P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:58.584985 29799 raft_consensus.cc:515] T 00c6cf271c91400aa834467def9af9b1 P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:26:58.589344 29799 leader_election.cc:290] T 00c6cf271c91400aa834467def9af9b1 P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825), d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209)
I20260709 12:26:58.590083 29799 raft_consensus.cc:2804] T a46289e8fc824dd5a818e30df6e7ec3e P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:26:58.590446 29799 raft_consensus.cc:493] T a46289e8fc824dd5a818e30df6e7ec3e P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:58.590889 29799 raft_consensus.cc:3060] T a46289e8fc824dd5a818e30df6e7ec3e P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:58.591058 29668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00c6cf271c91400aa834467def9af9b1" candidate_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" is_pre_election: true
I20260709 12:26:58.592443 29668 raft_consensus.cc:2393] T 00c6cf271c91400aa834467def9af9b1 P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c6610c00ffb948ce9c34f41dec3d49e8 in current term 1: Already voted for candidate d73d92fb35234d16824c55689b5fce28 in this term.
I20260709 12:26:58.593120 29858 leader_election.cc:290] T 62f8c9472b214576b9002e02536a3ace P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033), d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825)
I20260709 12:26:58.577931 29852 raft_consensus.cc:2804] T b3c02c910d2b43cc9265bb1d98491cd3 P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:26:58.604429 29852 raft_consensus.cc:697] T b3c02c910d2b43cc9265bb1d98491cd3 P d73d92fb35234d16824c55689b5fce28 [term 1 LEADER]: Becoming Leader. State: Replica: d73d92fb35234d16824c55689b5fce28, State: Running, Role: LEADER
I20260709 12:26:58.605432 29852 consensus_queue.cc:237] T b3c02c910d2b43cc9265bb1d98491cd3 P d73d92fb35234d16824c55689b5fce28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:26:58.586782 29953 consensus_queue.cc:1048] T 62f8c9472b214576b9002e02536a3ace P 584dac86183a41d9b3bea1c2085e62fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.021s
I20260709 12:26:58.612888 29790 tablet_bootstrap.cc:492] T 66f652ba2c224885a2aea974bfe8f88a P c6610c00ffb948ce9c34f41dec3d49e8: No bootstrap required, opened a new log
I20260709 12:26:58.613448 29790 ts_tablet_manager.cc:1403] T 66f652ba2c224885a2aea974bfe8f88a P c6610c00ffb948ce9c34f41dec3d49e8: Time spent bootstrapping tablet: real 0.090s	user 0.020s	sys 0.012s
I20260709 12:26:58.624095 29799 raft_consensus.cc:515] T a46289e8fc824dd5a818e30df6e7ec3e P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
W20260709 12:26:58.625069 29561 leader_election.cc:343] T 66f652ba2c224885a2aea974bfe8f88a P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:58.615056 29417 catalog_manager.cc:5697] T 2707271fd46f456aa765026642a3efa9 P 584dac86183a41d9b3bea1c2085e62fa reported cstate change: term changed from 0 to 1, leader changed from <none> to 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65). New cstate: current_term: 1 leader_uuid: "584dac86183a41d9b3bea1c2085e62fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:58.631073 29522 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f8c9472b214576b9002e02536a3ace" candidate_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "584dac86183a41d9b3bea1c2085e62fa" is_pre_election: true
I20260709 12:26:58.631608 29790 raft_consensus.cc:359] T 66f652ba2c224885a2aea974bfe8f88a P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:26:58.634142 29790 raft_consensus.cc:385] T 66f652ba2c224885a2aea974bfe8f88a P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:58.634454 29790 raft_consensus.cc:740] T 66f652ba2c224885a2aea974bfe8f88a P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6610c00ffb948ce9c34f41dec3d49e8, State: Initialized, Role: FOLLOWER
I20260709 12:26:58.630055 29962 consensus_queue.cc:1048] T 9633e3b1d78f403097e4a00b7d4b869b P c6610c00ffb948ce9c34f41dec3d49e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20260709 12:26:58.637307 29790 consensus_queue.cc:260] T 66f652ba2c224885a2aea974bfe8f88a P c6610c00ffb948ce9c34f41dec3d49e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:26:58.638849 29561 leader_election.cc:304] T 66f652ba2c224885a2aea974bfe8f88a P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d73d92fb35234d16824c55689b5fce28; no voters: 584dac86183a41d9b3bea1c2085e62fa, c6610c00ffb948ce9c34f41dec3d49e8
I20260709 12:26:58.648931 29790 ts_tablet_manager.cc:1434] T 66f652ba2c224885a2aea974bfe8f88a P c6610c00ffb948ce9c34f41dec3d49e8: Time spent starting tablet: real 0.035s	user 0.001s	sys 0.007s
I20260709 12:26:58.650022 29790 tablet_bootstrap.cc:492] T b3c02c910d2b43cc9265bb1d98491cd3 P c6610c00ffb948ce9c34f41dec3d49e8: Bootstrap starting.
I20260709 12:26:58.651283 29793 tablet_bootstrap.cc:492] T 71e2de235adf46d9a2407659080d42ef P d73d92fb35234d16824c55689b5fce28: No bootstrap required, opened a new log
I20260709 12:26:58.651820 29793 ts_tablet_manager.cc:1403] T 71e2de235adf46d9a2407659080d42ef P d73d92fb35234d16824c55689b5fce28: Time spent bootstrapping tablet: real 0.091s	user 0.016s	sys 0.009s
I20260709 12:26:58.649784 29851 raft_consensus.cc:2749] T 66f652ba2c224885a2aea974bfe8f88a P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:26:58.654119 29858 raft_consensus.cc:493] T 58154d49c1a344f2b51366ede3c87de5 P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:58.661244 29794 tablet_bootstrap.cc:492] T a46289e8fc824dd5a818e30df6e7ec3e P d13c58ad8ac441c4a9b396eb04b1a476: No bootstrap required, opened a new log
I20260709 12:26:58.661741 29794 ts_tablet_manager.cc:1403] T a46289e8fc824dd5a818e30df6e7ec3e P d13c58ad8ac441c4a9b396eb04b1a476: Time spent bootstrapping tablet: real 0.098s	user 0.019s	sys 0.008s
W20260709 12:26:58.658661 29561 leader_election.cc:343] T b3c02c910d2b43cc9265bb1d98491cd3 P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:58.655483 29858 raft_consensus.cc:515] T 58154d49c1a344f2b51366ede3c87de5 P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
W20260709 12:26:58.681756 29635 consensus_peers.cc:597] T 76140ad6258543118a9bfeee7cf0d5fc P d13c58ad8ac441c4a9b396eb04b1a476 -> Peer c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895): Couldn't send request to peer c6610c00ffb948ce9c34f41dec3d49e8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:26:58.681991 29790 tablet_bootstrap.cc:654] T b3c02c910d2b43cc9265bb1d98491cd3 P c6610c00ffb948ce9c34f41dec3d49e8: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:58.683908 29669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a46289e8fc824dd5a818e30df6e7ec3e" candidate_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d13c58ad8ac441c4a9b396eb04b1a476"
I20260709 12:26:58.684901 29858 leader_election.cc:290] T 58154d49c1a344f2b51366ede3c87de5 P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825), d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209)
I20260709 12:26:58.671839 29962 raft_consensus.cc:493] T 8627b58b4f42491597125eb45f8d3b61 P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:58.692911 29794 raft_consensus.cc:359] T a46289e8fc824dd5a818e30df6e7ec3e P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:26:58.702518 29794 raft_consensus.cc:385] T a46289e8fc824dd5a818e30df6e7ec3e P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:58.702948 29794 raft_consensus.cc:740] T a46289e8fc824dd5a818e30df6e7ec3e P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d13c58ad8ac441c4a9b396eb04b1a476, State: Initialized, Role: FOLLOWER
W20260709 12:26:58.707692 29709 leader_election.cc:343] T a46289e8fc824dd5a818e30df6e7ec3e P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:58.707965 29669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58154d49c1a344f2b51366ede3c87de5" candidate_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" is_pre_election: true
I20260709 12:26:58.700433 29962 raft_consensus.cc:515] T 8627b58b4f42491597125eb45f8d3b61 P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:26:58.712488 29962 leader_election.cc:290] T 8627b58b4f42491597125eb45f8d3b61 P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209), 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033)
I20260709 12:26:58.712765 29793 raft_consensus.cc:359] T 71e2de235adf46d9a2407659080d42ef P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:26:58.713285 29745 raft_consensus.cc:3060] T 00c6cf271c91400aa834467def9af9b1 P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:58.714936 29799 leader_election.cc:290] T a46289e8fc824dd5a818e30df6e7ec3e P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 1 election: Requested vote from peers d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209), 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033)
I20260709 12:26:58.716902 29670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8627b58b4f42491597125eb45f8d3b61" candidate_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" is_pre_election: true
I20260709 12:26:58.718611 29792 tablet_bootstrap.cc:492] T 3615115233c34461b85763151751e475 P 584dac86183a41d9b3bea1c2085e62fa: No bootstrap required, opened a new log
I20260709 12:26:58.719156 29792 ts_tablet_manager.cc:1403] T 3615115233c34461b85763151751e475 P 584dac86183a41d9b3bea1c2085e62fa: Time spent bootstrapping tablet: real 0.181s	user 0.014s	sys 0.024s
I20260709 12:26:58.719502 29521 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a46289e8fc824dd5a818e30df6e7ec3e" candidate_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "584dac86183a41d9b3bea1c2085e62fa"
I20260709 12:26:58.721170 29519 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8627b58b4f42491597125eb45f8d3b61" candidate_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "584dac86183a41d9b3bea1c2085e62fa" is_pre_election: true
I20260709 12:26:58.718987 29419 catalog_manager.cc:5697] T b3c02c910d2b43cc9265bb1d98491cd3 P d73d92fb35234d16824c55689b5fce28 reported cstate change: term changed from 0 to 1, leader changed from <none> to d73d92fb35234d16824c55689b5fce28 (127.16.29.66). New cstate: current_term: 1 leader_uuid: "d73d92fb35234d16824c55689b5fce28" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } health_report { overall_health: HEALTHY } } }
I20260709 12:26:58.721789 29519 raft_consensus.cc:2468] T 8627b58b4f42491597125eb45f8d3b61 P 584dac86183a41d9b3bea1c2085e62fa [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6610c00ffb948ce9c34f41dec3d49e8 in term 2.
I20260709 12:26:58.722364 29792 raft_consensus.cc:359] T 3615115233c34461b85763151751e475 P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:26:58.720142 29521 raft_consensus.cc:3060] T a46289e8fc824dd5a818e30df6e7ec3e P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:58.723147 29792 raft_consensus.cc:385] T 3615115233c34461b85763151751e475 P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:58.723510 29792 raft_consensus.cc:740] T 3615115233c34461b85763151751e475 P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 584dac86183a41d9b3bea1c2085e62fa, State: Initialized, Role: FOLLOWER
W20260709 12:26:58.722755 29709 leader_election.cc:343] T 58154d49c1a344f2b51366ede3c87de5 P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:58.724287 29793 raft_consensus.cc:385] T 71e2de235adf46d9a2407659080d42ef P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:58.725443 29793 raft_consensus.cc:740] T 71e2de235adf46d9a2407659080d42ef P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d73d92fb35234d16824c55689b5fce28, State: Initialized, Role: FOLLOWER
I20260709 12:26:58.726383 29793 consensus_queue.cc:260] T 71e2de235adf46d9a2407659080d42ef P d73d92fb35234d16824c55689b5fce28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:26:58.728250 29794 consensus_queue.cc:260] T a46289e8fc824dd5a818e30df6e7ec3e P d13c58ad8ac441c4a9b396eb04b1a476 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:26:58.733114 29793 ts_tablet_manager.cc:1434] T 71e2de235adf46d9a2407659080d42ef P d73d92fb35234d16824c55689b5fce28: Time spent starting tablet: real 0.081s	user 0.006s	sys 0.002s
W20260709 12:26:58.732362 29634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:35033, user_credentials={real_user=slave}} blocked reactor thread for 47633.8us
I20260709 12:26:58.734288 29793 tablet_bootstrap.cc:492] T ed057e06dec148a0b9d7062beeb9d005 P d73d92fb35234d16824c55689b5fce28: Bootstrap starting.
I20260709 12:26:58.731433 29708 leader_election.cc:304] T 8627b58b4f42491597125eb45f8d3b61 P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 584dac86183a41d9b3bea1c2085e62fa, c6610c00ffb948ce9c34f41dec3d49e8; no voters: d13c58ad8ac441c4a9b396eb04b1a476
I20260709 12:26:58.747714 29790 tablet_bootstrap.cc:492] T b3c02c910d2b43cc9265bb1d98491cd3 P c6610c00ffb948ce9c34f41dec3d49e8: No bootstrap required, opened a new log
I20260709 12:26:58.748373 29790 ts_tablet_manager.cc:1403] T b3c02c910d2b43cc9265bb1d98491cd3 P c6610c00ffb948ce9c34f41dec3d49e8: Time spent bootstrapping tablet: real 0.099s	user 0.022s	sys 0.010s
I20260709 12:26:58.749660 29799 raft_consensus.cc:2804] T 8627b58b4f42491597125eb45f8d3b61 P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:26:58.754194 29799 raft_consensus.cc:493] T 8627b58b4f42491597125eb45f8d3b61 P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:58.754570 29799 raft_consensus.cc:3060] T 8627b58b4f42491597125eb45f8d3b61 P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:26:58.757431 29792 consensus_queue.cc:260] T 3615115233c34461b85763151751e475 P 584dac86183a41d9b3bea1c2085e62fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:26:58.763099 29794 ts_tablet_manager.cc:1434] T a46289e8fc824dd5a818e30df6e7ec3e P d13c58ad8ac441c4a9b396eb04b1a476: Time spent starting tablet: real 0.101s	user 0.018s	sys 0.007s
I20260709 12:26:58.764281 29794 tablet_bootstrap.cc:492] T 3615115233c34461b85763151751e475 P d13c58ad8ac441c4a9b396eb04b1a476: Bootstrap starting.
I20260709 12:26:58.769094 29792 ts_tablet_manager.cc:1434] T 3615115233c34461b85763151751e475 P 584dac86183a41d9b3bea1c2085e62fa: Time spent starting tablet: real 0.050s	user 0.013s	sys 0.000s
I20260709 12:26:58.771018 29792 tablet_bootstrap.cc:492] T 3f41ce441fb047c8bbb741b1b9cbc0ec P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap starting.
I20260709 12:26:58.777119 29953 raft_consensus.cc:493] T 71e2de235adf46d9a2407659080d42ef P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:58.777736 29953 raft_consensus.cc:515] T 71e2de235adf46d9a2407659080d42ef P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:26:58.779954 29953 leader_election.cc:290] T 71e2de235adf46d9a2407659080d42ef P 584dac86183a41d9b3bea1c2085e62fa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825), d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209)
I20260709 12:26:58.782236 29670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "71e2de235adf46d9a2407659080d42ef" candidate_uuid: "584dac86183a41d9b3bea1c2085e62fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" is_pre_election: true
I20260709 12:26:58.783025 29670 raft_consensus.cc:2468] T 71e2de235adf46d9a2407659080d42ef P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 584dac86183a41d9b3bea1c2085e62fa in term 0.
I20260709 12:26:58.784312 29486 leader_election.cc:304] T 71e2de235adf46d9a2407659080d42ef P 584dac86183a41d9b3bea1c2085e62fa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 584dac86183a41d9b3bea1c2085e62fa, d13c58ad8ac441c4a9b396eb04b1a476; no voters: 
I20260709 12:26:58.785203 29953 raft_consensus.cc:2804] T 71e2de235adf46d9a2407659080d42ef P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:26:58.785607 29953 raft_consensus.cc:493] T 71e2de235adf46d9a2407659080d42ef P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:58.785960 29953 raft_consensus.cc:3060] T 71e2de235adf46d9a2407659080d42ef P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:58.787482 29790 raft_consensus.cc:359] T b3c02c910d2b43cc9265bb1d98491cd3 P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:26:58.793124 29790 raft_consensus.cc:385] T b3c02c910d2b43cc9265bb1d98491cd3 P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:58.793562 29790 raft_consensus.cc:740] T b3c02c910d2b43cc9265bb1d98491cd3 P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6610c00ffb948ce9c34f41dec3d49e8, State: Initialized, Role: FOLLOWER
I20260709 12:26:58.797224 29971 raft_consensus.cc:493] T a46289e8fc824dd5a818e30df6e7ec3e P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:58.797950 29971 raft_consensus.cc:515] T a46289e8fc824dd5a818e30df6e7ec3e P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:26:58.800771 29521 raft_consensus.cc:2468] T a46289e8fc824dd5a818e30df6e7ec3e P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6610c00ffb948ce9c34f41dec3d49e8 in term 1.
I20260709 12:26:58.801245 29744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a46289e8fc824dd5a818e30df6e7ec3e" candidate_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" is_pre_election: true
I20260709 12:26:58.802104 29744 raft_consensus.cc:2393] T a46289e8fc824dd5a818e30df6e7ec3e P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d13c58ad8ac441c4a9b396eb04b1a476 in current term 1: Already voted for candidate c6610c00ffb948ce9c34f41dec3d49e8 in this term.
I20260709 12:26:58.802562 29595 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00c6cf271c91400aa834467def9af9b1" candidate_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d73d92fb35234d16824c55689b5fce28" is_pre_election: true
I20260709 12:26:58.803032 29596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f8c9472b214576b9002e02536a3ace" candidate_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d73d92fb35234d16824c55689b5fce28" is_pre_election: true
I20260709 12:26:58.804327 29793 tablet_bootstrap.cc:654] T ed057e06dec148a0b9d7062beeb9d005 P d73d92fb35234d16824c55689b5fce28: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:58.806452 29594 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58154d49c1a344f2b51366ede3c87de5" candidate_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d73d92fb35234d16824c55689b5fce28" is_pre_election: true
I20260709 12:26:58.806692 29799 raft_consensus.cc:515] T 8627b58b4f42491597125eb45f8d3b61 P c6610c00ffb948ce9c34f41dec3d49e8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:26:58.807616 29708 leader_election.cc:304] T a46289e8fc824dd5a818e30df6e7ec3e P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 584dac86183a41d9b3bea1c2085e62fa, c6610c00ffb948ce9c34f41dec3d49e8; no voters: d13c58ad8ac441c4a9b396eb04b1a476
I20260709 12:26:58.808799 29962 raft_consensus.cc:2804] T a46289e8fc824dd5a818e30df6e7ec3e P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:26:58.809422 29962 raft_consensus.cc:697] T a46289e8fc824dd5a818e30df6e7ec3e P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 LEADER]: Becoming Leader. State: Replica: c6610c00ffb948ce9c34f41dec3d49e8, State: Running, Role: LEADER
I20260709 12:26:58.810869 29794 tablet_bootstrap.cc:654] T 3615115233c34461b85763151751e475 P d13c58ad8ac441c4a9b396eb04b1a476: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:58.810508 29962 consensus_queue.cc:237] T a46289e8fc824dd5a818e30df6e7ec3e P c6610c00ffb948ce9c34f41dec3d49e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:26:58.812223 29790 consensus_queue.cc:260] T b3c02c910d2b43cc9265bb1d98491cd3 P c6610c00ffb948ce9c34f41dec3d49e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:26:58.814077 29968 consensus_queue.cc:1048] T 76140ad6258543118a9bfeee7cf0d5fc P d13c58ad8ac441c4a9b396eb04b1a476 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:26:58.818444 29670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8627b58b4f42491597125eb45f8d3b61" candidate_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d13c58ad8ac441c4a9b396eb04b1a476"
I20260709 12:26:58.819053 29708 leader_election.cc:304] T 00c6cf271c91400aa834467def9af9b1 P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c6610c00ffb948ce9c34f41dec3d49e8; no voters: d13c58ad8ac441c4a9b396eb04b1a476, d73d92fb35234d16824c55689b5fce28
I20260709 12:26:58.824945 29860 raft_consensus.cc:2749] T 00c6cf271c91400aa834467def9af9b1 P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:26:58.825413 29708 leader_election.cc:304] T 62f8c9472b214576b9002e02536a3ace P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c6610c00ffb948ce9c34f41dec3d49e8; no voters: 584dac86183a41d9b3bea1c2085e62fa, d73d92fb35234d16824c55689b5fce28
I20260709 12:26:58.826552 29971 leader_election.cc:290] T a46289e8fc824dd5a818e30df6e7ec3e P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895), 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033)
I20260709 12:26:58.826503 29860 raft_consensus.cc:2749] T 62f8c9472b214576b9002e02536a3ace P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:26:58.828804 29852 raft_consensus.cc:493] T 3615115233c34461b85763151751e475 P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:58.829384 29852 raft_consensus.cc:515] T 3615115233c34461b85763151751e475 P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:26:58.830605 29521 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a46289e8fc824dd5a818e30df6e7ec3e" candidate_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "584dac86183a41d9b3bea1c2085e62fa" is_pre_election: true
I20260709 12:26:58.831511 29521 raft_consensus.cc:2393] T a46289e8fc824dd5a818e30df6e7ec3e P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d13c58ad8ac441c4a9b396eb04b1a476 in current term 1: Already voted for candidate c6610c00ffb948ce9c34f41dec3d49e8 in this term.
I20260709 12:26:58.831789 29852 leader_election.cc:290] T 3615115233c34461b85763151751e475 P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033), d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209)
I20260709 12:26:58.832897 29852 consensus_queue.cc:1048] T 00c6cf271c91400aa834467def9af9b1 P d73d92fb35234d16824c55689b5fce28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:26:58.833178 29634 leader_election.cc:304] T a46289e8fc824dd5a818e30df6e7ec3e P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d13c58ad8ac441c4a9b396eb04b1a476; no voters: 584dac86183a41d9b3bea1c2085e62fa, c6610c00ffb948ce9c34f41dec3d49e8
I20260709 12:26:58.834859 29851 consensus_queue.cc:1048] T 00c6cf271c91400aa834467def9af9b1 P d73d92fb35234d16824c55689b5fce28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:26:58.836964 29594 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "71e2de235adf46d9a2407659080d42ef" candidate_uuid: "584dac86183a41d9b3bea1c2085e62fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d73d92fb35234d16824c55689b5fce28" is_pre_election: true
I20260709 12:26:58.837649 29594 raft_consensus.cc:2468] T 71e2de235adf46d9a2407659080d42ef P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 584dac86183a41d9b3bea1c2085e62fa in term 0.
I20260709 12:26:58.837714 29971 raft_consensus.cc:3060] T a46289e8fc824dd5a818e30df6e7ec3e P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:58.838670 29856 raft_consensus.cc:493] T f4516786e5564e9eac08ca472f79a9f0 P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:58.839126 29856 raft_consensus.cc:515] T f4516786e5564e9eac08ca472f79a9f0 P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:26:58.842183 29790 ts_tablet_manager.cc:1434] T b3c02c910d2b43cc9265bb1d98491cd3 P c6610c00ffb948ce9c34f41dec3d49e8: Time spent starting tablet: real 0.094s	user 0.005s	sys 0.003s
I20260709 12:26:58.839619 29953 raft_consensus.cc:515] T 71e2de235adf46d9a2407659080d42ef P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:26:58.848559 29856 leader_election.cc:290] T f4516786e5564e9eac08ca472f79a9f0 P 584dac86183a41d9b3bea1c2085e62fa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209), c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895)
I20260709 12:26:58.821542 29792 tablet_bootstrap.cc:654] T 3f41ce441fb047c8bbb741b1b9cbc0ec P 584dac86183a41d9b3bea1c2085e62fa: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:58.849692 29790 tablet_bootstrap.cc:492] T 8017a44219404faa8db8ff049c65205e P c6610c00ffb948ce9c34f41dec3d49e8: Bootstrap starting.
W20260709 12:26:58.827030 29708 leader_election.cc:343] T 58154d49c1a344f2b51366ede3c87de5 P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:58.851573 29708 leader_election.cc:304] T 58154d49c1a344f2b51366ede3c87de5 P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c6610c00ffb948ce9c34f41dec3d49e8; no voters: d13c58ad8ac441c4a9b396eb04b1a476, d73d92fb35234d16824c55689b5fce28
I20260709 12:26:58.853206 29671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4516786e5564e9eac08ca472f79a9f0" candidate_uuid: "584dac86183a41d9b3bea1c2085e62fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" is_pre_election: true
I20260709 12:26:58.853396 29971 raft_consensus.cc:2749] T a46289e8fc824dd5a818e30df6e7ec3e P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:26:58.853895 29671 raft_consensus.cc:2468] T f4516786e5564e9eac08ca472f79a9f0 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 584dac86183a41d9b3bea1c2085e62fa in term 0.
I20260709 12:26:58.856142 29860 raft_consensus.cc:2749] T 58154d49c1a344f2b51366ede3c87de5 P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:26:58.856702 29744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4516786e5564e9eac08ca472f79a9f0" candidate_uuid: "584dac86183a41d9b3bea1c2085e62fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" is_pre_election: true
I20260709 12:26:58.856082 29486 leader_election.cc:304] T f4516786e5564e9eac08ca472f79a9f0 P 584dac86183a41d9b3bea1c2085e62fa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 584dac86183a41d9b3bea1c2085e62fa, d13c58ad8ac441c4a9b396eb04b1a476; no voters: 
I20260709 12:26:58.858031 29799 leader_election.cc:290] T 8627b58b4f42491597125eb45f8d3b61 P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 3 election: Requested vote from peers d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209), 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033)
I20260709 12:26:58.870406 29793 tablet_bootstrap.cc:492] T ed057e06dec148a0b9d7062beeb9d005 P d73d92fb35234d16824c55689b5fce28: No bootstrap required, opened a new log
W20260709 12:26:58.863704 29486 leader_election.cc:343] T f4516786e5564e9eac08ca472f79a9f0 P 584dac86183a41d9b3bea1c2085e62fa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:58.872190 29951 raft_consensus.cc:493] T 3615115233c34461b85763151751e475 P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:58.875505 29951 raft_consensus.cc:515] T 3615115233c34461b85763151751e475 P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:26:58.875830 29521 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3615115233c34461b85763151751e475" candidate_uuid: "d73d92fb35234d16824c55689b5fce28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "584dac86183a41d9b3bea1c2085e62fa" is_pre_election: true
I20260709 12:26:58.876514 29521 raft_consensus.cc:2468] T 3615115233c34461b85763151751e475 P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d73d92fb35234d16824c55689b5fce28 in term 0.
I20260709 12:26:58.877755 29951 leader_election.cc:290] T 3615115233c34461b85763151751e475 P 584dac86183a41d9b3bea1c2085e62fa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825), d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209)
I20260709 12:26:58.872839 29793 ts_tablet_manager.cc:1403] T ed057e06dec148a0b9d7062beeb9d005 P d73d92fb35234d16824c55689b5fce28: Time spent bootstrapping tablet: real 0.139s	user 0.015s	sys 0.007s
I20260709 12:26:58.864140 29670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3615115233c34461b85763151751e475" candidate_uuid: "d73d92fb35234d16824c55689b5fce28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" is_pre_election: true
I20260709 12:26:58.877437 29419 catalog_manager.cc:5697] T a46289e8fc824dd5a818e30df6e7ec3e P c6610c00ffb948ce9c34f41dec3d49e8 reported cstate change: term changed from 0 to 1, leader changed from <none> to c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68). New cstate: current_term: 1 leader_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:58.874691 29962 raft_consensus.cc:493] T b3c02c910d2b43cc9265bb1d98491cd3 P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:58.879917 29962 raft_consensus.cc:515] T b3c02c910d2b43cc9265bb1d98491cd3 P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:26:58.882718 29962 leader_election.cc:290] T b3c02c910d2b43cc9265bb1d98491cd3 P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033), d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825)
I20260709 12:26:58.879614 29522 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8627b58b4f42491597125eb45f8d3b61" candidate_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "584dac86183a41d9b3bea1c2085e62fa"
I20260709 12:26:58.885360 29670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3615115233c34461b85763151751e475" candidate_uuid: "584dac86183a41d9b3bea1c2085e62fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" is_pre_election: true
I20260709 12:26:58.885612 29522 raft_consensus.cc:3060] T 8627b58b4f42491597125eb45f8d3b61 P 584dac86183a41d9b3bea1c2085e62fa [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:26:58.887207 29486 leader_election.cc:343] T 3615115233c34461b85763151751e475 P 584dac86183a41d9b3bea1c2085e62fa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:58.888183 29560 leader_election.cc:304] T 3615115233c34461b85763151751e475 P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 584dac86183a41d9b3bea1c2085e62fa, d73d92fb35234d16824c55689b5fce28; no voters: 
I20260709 12:26:58.889128 29793 raft_consensus.cc:359] T ed057e06dec148a0b9d7062beeb9d005 P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:26:58.889837 29793 raft_consensus.cc:385] T ed057e06dec148a0b9d7062beeb9d005 P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:58.890105 29793 raft_consensus.cc:740] T ed057e06dec148a0b9d7062beeb9d005 P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d73d92fb35234d16824c55689b5fce28, State: Initialized, Role: FOLLOWER
I20260709 12:26:58.890326 29852 raft_consensus.cc:2804] T 3615115233c34461b85763151751e475 P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:26:58.890929 29852 raft_consensus.cc:493] T 3615115233c34461b85763151751e475 P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:58.891479 29852 raft_consensus.cc:3060] T 3615115233c34461b85763151751e475 P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:58.866931 29856 raft_consensus.cc:2804] T f4516786e5564e9eac08ca472f79a9f0 P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:26:58.896370 29856 raft_consensus.cc:493] T f4516786e5564e9eac08ca472f79a9f0 P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:58.896745 29856 raft_consensus.cc:3060] T f4516786e5564e9eac08ca472f79a9f0 P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:58.912381 29953 leader_election.cc:290] T 71e2de235adf46d9a2407659080d42ef P 584dac86183a41d9b3bea1c2085e62fa [CANDIDATE]: Term 1 election: Requested vote from peers d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825), d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209)
I20260709 12:26:58.913913 29670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "71e2de235adf46d9a2407659080d42ef" candidate_uuid: "584dac86183a41d9b3bea1c2085e62fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d13c58ad8ac441c4a9b396eb04b1a476"
I20260709 12:26:58.914815 29670 raft_consensus.cc:3060] T 71e2de235adf46d9a2407659080d42ef P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:58.906844 29793 consensus_queue.cc:260] T ed057e06dec148a0b9d7062beeb9d005 P d73d92fb35234d16824c55689b5fce28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:26:58.925642 29521 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3c02c910d2b43cc9265bb1d98491cd3" candidate_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "584dac86183a41d9b3bea1c2085e62fa" is_pre_election: true
I20260709 12:26:58.926569 29521 raft_consensus.cc:2393] T b3c02c910d2b43cc9265bb1d98491cd3 P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c6610c00ffb948ce9c34f41dec3d49e8 in current term 1: Already voted for candidate d73d92fb35234d16824c55689b5fce28 in this term.
I20260709 12:26:58.928153 29790 tablet_bootstrap.cc:654] T 8017a44219404faa8db8ff049c65205e P c6610c00ffb948ce9c34f41dec3d49e8: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:58.929098 29522 raft_consensus.cc:2468] T 8627b58b4f42491597125eb45f8d3b61 P 584dac86183a41d9b3bea1c2085e62fa [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6610c00ffb948ce9c34f41dec3d49e8 in term 3.
I20260709 12:26:58.930399 29593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "71e2de235adf46d9a2407659080d42ef" candidate_uuid: "584dac86183a41d9b3bea1c2085e62fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d73d92fb35234d16824c55689b5fce28"
I20260709 12:26:58.931085 29593 raft_consensus.cc:3060] T 71e2de235adf46d9a2407659080d42ef P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:58.933429 29793 ts_tablet_manager.cc:1434] T ed057e06dec148a0b9d7062beeb9d005 P d73d92fb35234d16824c55689b5fce28: Time spent starting tablet: real 0.050s	user 0.007s	sys 0.000s
I20260709 12:26:58.934356 29793 tablet_bootstrap.cc:492] T 58154d49c1a344f2b51366ede3c87de5 P d73d92fb35234d16824c55689b5fce28: Bootstrap starting.
I20260709 12:26:58.941972 29794 tablet_bootstrap.cc:492] T 3615115233c34461b85763151751e475 P d13c58ad8ac441c4a9b396eb04b1a476: No bootstrap required, opened a new log
I20260709 12:26:58.942546 29794 ts_tablet_manager.cc:1403] T 3615115233c34461b85763151751e475 P d13c58ad8ac441c4a9b396eb04b1a476: Time spent bootstrapping tablet: real 0.178s	user 0.018s	sys 0.013s
I20260709 12:26:58.947650 29799 consensus_queue.cc:1048] T ed057e06dec148a0b9d7062beeb9d005 P c6610c00ffb948ce9c34f41dec3d49e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:26:58.965397 29594 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3615115233c34461b85763151751e475" candidate_uuid: "584dac86183a41d9b3bea1c2085e62fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d73d92fb35234d16824c55689b5fce28" is_pre_election: true
I20260709 12:26:58.964529 29794 raft_consensus.cc:359] T 3615115233c34461b85763151751e475 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:26:58.967445 29794 raft_consensus.cc:385] T 3615115233c34461b85763151751e475 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:58.967825 29794 raft_consensus.cc:740] T 3615115233c34461b85763151751e475 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d13c58ad8ac441c4a9b396eb04b1a476, State: Initialized, Role: FOLLOWER
I20260709 12:26:58.972786 29968 raft_consensus.cc:493] T 8627b58b4f42491597125eb45f8d3b61 P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c6610c00ffb948ce9c34f41dec3d49e8)
I20260709 12:26:58.976821 29968 raft_consensus.cc:515] T 8627b58b4f42491597125eb45f8d3b61 P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:26:58.977144 29856 raft_consensus.cc:515] T f4516786e5564e9eac08ca472f79a9f0 P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:26:58.980530 29668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4516786e5564e9eac08ca472f79a9f0" candidate_uuid: "584dac86183a41d9b3bea1c2085e62fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d13c58ad8ac441c4a9b396eb04b1a476"
I20260709 12:26:58.981772 29668 raft_consensus.cc:3060] T f4516786e5564e9eac08ca472f79a9f0 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:58.986939 29794 consensus_queue.cc:260] T 3615115233c34461b85763151751e475 P d13c58ad8ac441c4a9b396eb04b1a476 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:26:58.983415 29593 raft_consensus.cc:2468] T 71e2de235adf46d9a2407659080d42ef P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 584dac86183a41d9b3bea1c2085e62fa in term 1.
W20260709 12:26:58.991161 29561 leader_election.cc:343] T 3615115233c34461b85763151751e475 P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:58.999220 29794 ts_tablet_manager.cc:1434] T 3615115233c34461b85763151751e475 P d13c58ad8ac441c4a9b396eb04b1a476: Time spent starting tablet: real 0.056s	user 0.009s	sys 0.000s
I20260709 12:26:59.003108 29794 tablet_bootstrap.cc:492] T 3f41ce441fb047c8bbb741b1b9cbc0ec P d13c58ad8ac441c4a9b396eb04b1a476: Bootstrap starting.
I20260709 12:26:58.979635 29670 raft_consensus.cc:2468] T 71e2de235adf46d9a2407659080d42ef P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 584dac86183a41d9b3bea1c2085e62fa in term 1.
I20260709 12:26:59.015844 29792 tablet_bootstrap.cc:492] T 3f41ce441fb047c8bbb741b1b9cbc0ec P 584dac86183a41d9b3bea1c2085e62fa: No bootstrap required, opened a new log
I20260709 12:26:59.016706 29792 ts_tablet_manager.cc:1403] T 3f41ce441fb047c8bbb741b1b9cbc0ec P 584dac86183a41d9b3bea1c2085e62fa: Time spent bootstrapping tablet: real 0.247s	user 0.014s	sys 0.023s
I20260709 12:26:59.018985 29794 tablet_bootstrap.cc:654] T 3f41ce441fb047c8bbb741b1b9cbc0ec P d13c58ad8ac441c4a9b396eb04b1a476: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:59.031392 29745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4516786e5564e9eac08ca472f79a9f0" candidate_uuid: "584dac86183a41d9b3bea1c2085e62fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6610c00ffb948ce9c34f41dec3d49e8"
I20260709 12:26:59.024609 29962 raft_consensus.cc:493] T 66f652ba2c224885a2aea974bfe8f88a P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:59.033457 29962 raft_consensus.cc:515] T 66f652ba2c224885a2aea974bfe8f88a P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:26:59.041457 29792 raft_consensus.cc:359] T 3f41ce441fb047c8bbb741b1b9cbc0ec P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:26:59.043980 29792 raft_consensus.cc:385] T 3f41ce441fb047c8bbb741b1b9cbc0ec P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:59.044427 29792 raft_consensus.cc:740] T 3f41ce441fb047c8bbb741b1b9cbc0ec P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 584dac86183a41d9b3bea1c2085e62fa, State: Initialized, Role: FOLLOWER
I20260709 12:26:59.046682 29852 raft_consensus.cc:515] T 3615115233c34461b85763151751e475 P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:26:59.050706 29793 tablet_bootstrap.cc:654] T 58154d49c1a344f2b51366ede3c87de5 P d73d92fb35234d16824c55689b5fce28: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:59.051108 29594 raft_consensus.cc:2393] T 3615115233c34461b85763151751e475 P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 584dac86183a41d9b3bea1c2085e62fa in current term 1: Already voted for candidate d73d92fb35234d16824c55689b5fce28 in this term.
I20260709 12:26:59.054989 29668 raft_consensus.cc:2468] T f4516786e5564e9eac08ca472f79a9f0 P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 584dac86183a41d9b3bea1c2085e62fa in term 1.
I20260709 12:26:59.056169 29962 leader_election.cc:290] T 66f652ba2c224885a2aea974bfe8f88a P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033), d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825)
I20260709 12:26:59.057078 29852 leader_election.cc:290] T 3615115233c34461b85763151751e475 P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 1 election: Requested vote from peers 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033), d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209)
W20260709 12:26:59.062250 29486 leader_election.cc:343] T f4516786e5564e9eac08ca472f79a9f0 P 584dac86183a41d9b3bea1c2085e62fa [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:59.064417 29522 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3615115233c34461b85763151751e475" candidate_uuid: "d73d92fb35234d16824c55689b5fce28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "584dac86183a41d9b3bea1c2085e62fa"
I20260709 12:26:59.065102 29522 raft_consensus.cc:3060] T 3615115233c34461b85763151751e475 P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:59.008549 29708 leader_election.cc:304] T 8627b58b4f42491597125eb45f8d3b61 P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 584dac86183a41d9b3bea1c2085e62fa, c6610c00ffb948ce9c34f41dec3d49e8; no voters: d13c58ad8ac441c4a9b396eb04b1a476
I20260709 12:26:59.074337 29799 raft_consensus.cc:2804] T 8627b58b4f42491597125eb45f8d3b61 P c6610c00ffb948ce9c34f41dec3d49e8 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:26:59.074883 29799 raft_consensus.cc:697] T 8627b58b4f42491597125eb45f8d3b61 P c6610c00ffb948ce9c34f41dec3d49e8 [term 3 LEADER]: Becoming Leader. State: Replica: c6610c00ffb948ce9c34f41dec3d49e8, State: Running, Role: LEADER
I20260709 12:26:59.075740 29799 consensus_queue.cc:237] T 8627b58b4f42491597125eb45f8d3b61 P c6610c00ffb948ce9c34f41dec3d49e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
W20260709 12:26:59.079551 29708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:35033, user_credentials={real_user=slave}} blocked reactor thread for 74386.5us
I20260709 12:26:59.084550 29968 leader_election.cc:290] T 8627b58b4f42491597125eb45f8d3b61 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895), 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033)
I20260709 12:26:59.084040 29745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8627b58b4f42491597125eb45f8d3b61" candidate_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" is_pre_election: true
I20260709 12:26:59.085459 29596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3c02c910d2b43cc9265bb1d98491cd3" candidate_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d73d92fb35234d16824c55689b5fce28" is_pre_election: true
I20260709 12:26:59.094103 29486 leader_election.cc:304] T 71e2de235adf46d9a2407659080d42ef P 584dac86183a41d9b3bea1c2085e62fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 584dac86183a41d9b3bea1c2085e62fa, d13c58ad8ac441c4a9b396eb04b1a476; no voters: 
I20260709 12:26:59.095445 29486 leader_election.cc:304] T f4516786e5564e9eac08ca472f79a9f0 P 584dac86183a41d9b3bea1c2085e62fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 584dac86183a41d9b3bea1c2085e62fa, d13c58ad8ac441c4a9b396eb04b1a476; no voters: c6610c00ffb948ce9c34f41dec3d49e8
I20260709 12:26:59.096483 29953 raft_consensus.cc:2804] T 71e2de235adf46d9a2407659080d42ef P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:26:59.099305 29521 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8627b58b4f42491597125eb45f8d3b61" candidate_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "584dac86183a41d9b3bea1c2085e62fa" is_pre_election: true
I20260709 12:26:59.102718 29954 raft_consensus.cc:2804] T f4516786e5564e9eac08ca472f79a9f0 P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:26:59.092612 29593 raft_consensus.cc:3060] T ed057e06dec148a0b9d7062beeb9d005 P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:59.027134 29856 leader_election.cc:290] T f4516786e5564e9eac08ca472f79a9f0 P 584dac86183a41d9b3bea1c2085e62fa [CANDIDATE]: Term 1 election: Requested vote from peers d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209), c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895)
I20260709 12:26:59.103482 29521 raft_consensus.cc:2368] T 8627b58b4f42491597125eb45f8d3b61 P 584dac86183a41d9b3bea1c2085e62fa [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d13c58ad8ac441c4a9b396eb04b1a476 for earlier term 2. Current term is 3.
I20260709 12:26:59.124377 29792 consensus_queue.cc:260] T 3f41ce441fb047c8bbb741b1b9cbc0ec P 584dac86183a41d9b3bea1c2085e62fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:26:59.134657 29954 raft_consensus.cc:697] T f4516786e5564e9eac08ca472f79a9f0 P 584dac86183a41d9b3bea1c2085e62fa [term 1 LEADER]: Becoming Leader. State: Replica: 584dac86183a41d9b3bea1c2085e62fa, State: Running, Role: LEADER
I20260709 12:26:59.140086 29793 tablet_bootstrap.cc:492] T 58154d49c1a344f2b51366ede3c87de5 P d73d92fb35234d16824c55689b5fce28: No bootstrap required, opened a new log
W20260709 12:26:59.141366 29467 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:26:59.141362 29522 raft_consensus.cc:2468] T 3615115233c34461b85763151751e475 P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d73d92fb35234d16824c55689b5fce28 in term 1.
I20260709 12:26:59.148180 29790 tablet_bootstrap.cc:492] T 8017a44219404faa8db8ff049c65205e P c6610c00ffb948ce9c34f41dec3d49e8: No bootstrap required, opened a new log
I20260709 12:26:59.134274 29953 raft_consensus.cc:697] T 71e2de235adf46d9a2407659080d42ef P 584dac86183a41d9b3bea1c2085e62fa [term 1 LEADER]: Becoming Leader. State: Replica: 584dac86183a41d9b3bea1c2085e62fa, State: Running, Role: LEADER
I20260709 12:26:59.150593 29790 ts_tablet_manager.cc:1403] T 8017a44219404faa8db8ff049c65205e P c6610c00ffb948ce9c34f41dec3d49e8: Time spent bootstrapping tablet: real 0.307s	user 0.026s	sys 0.061s
I20260709 12:26:59.151157 29953 consensus_queue.cc:237] T 71e2de235adf46d9a2407659080d42ef P 584dac86183a41d9b3bea1c2085e62fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:26:59.152966 29793 ts_tablet_manager.cc:1403] T 58154d49c1a344f2b51366ede3c87de5 P d73d92fb35234d16824c55689b5fce28: Time spent bootstrapping tablet: real 0.216s	user 0.019s	sys 0.008s
I20260709 12:26:59.161775 29634 leader_election.cc:400] T 8627b58b4f42491597125eb45f8d3b61 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033) with higher term. Message: Invalid argument: T 8627b58b4f42491597125eb45f8d3b61 P 584dac86183a41d9b3bea1c2085e62fa [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d13c58ad8ac441c4a9b396eb04b1a476 for earlier term 2. Current term is 3.
I20260709 12:26:59.162220 29634 leader_election.cc:403] T 8627b58b4f42491597125eb45f8d3b61 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:26:59.163630 29793 raft_consensus.cc:359] T 58154d49c1a344f2b51366ede3c87de5 P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:26:59.164496 29793 raft_consensus.cc:385] T 58154d49c1a344f2b51366ede3c87de5 P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:59.164794 29793 raft_consensus.cc:740] T 58154d49c1a344f2b51366ede3c87de5 P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d73d92fb35234d16824c55689b5fce28, State: Initialized, Role: FOLLOWER
I20260709 12:26:59.166312 29793 consensus_queue.cc:260] T 58154d49c1a344f2b51366ede3c87de5 P d73d92fb35234d16824c55689b5fce28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:26:59.167903 29596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66f652ba2c224885a2aea974bfe8f88a" candidate_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d73d92fb35234d16824c55689b5fce28" is_pre_election: true
I20260709 12:26:59.171772 29596 raft_consensus.cc:2468] T 66f652ba2c224885a2aea974bfe8f88a P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6610c00ffb948ce9c34f41dec3d49e8 in term 0.
I20260709 12:26:59.174576 29521 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66f652ba2c224885a2aea974bfe8f88a" candidate_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "584dac86183a41d9b3bea1c2085e62fa" is_pre_election: true
I20260709 12:26:59.176096 29667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3615115233c34461b85763151751e475" candidate_uuid: "d73d92fb35234d16824c55689b5fce28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d13c58ad8ac441c4a9b396eb04b1a476"
I20260709 12:26:59.177407 29667 raft_consensus.cc:3060] T 3615115233c34461b85763151751e475 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:59.178666 29951 consensus_queue.cc:1048] T 2707271fd46f456aa765026642a3efa9 P 584dac86183a41d9b3bea1c2085e62fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:26:59.180195 29485 leader_election.cc:304] T 3615115233c34461b85763151751e475 P 584dac86183a41d9b3bea1c2085e62fa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 584dac86183a41d9b3bea1c2085e62fa; no voters: d13c58ad8ac441c4a9b396eb04b1a476, d73d92fb35234d16824c55689b5fce28
I20260709 12:26:59.182255 29793 ts_tablet_manager.cc:1434] T 58154d49c1a344f2b51366ede3c87de5 P d73d92fb35234d16824c55689b5fce28: Time spent starting tablet: real 0.029s	user 0.006s	sys 0.000s
I20260709 12:26:59.182580 29560 leader_election.cc:304] T 3615115233c34461b85763151751e475 P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 584dac86183a41d9b3bea1c2085e62fa, d73d92fb35234d16824c55689b5fce28; no voters: 
I20260709 12:26:59.183341 29793 tablet_bootstrap.cc:492] T 6a30685caefb4d0e80fa4ea87d70760c P d73d92fb35234d16824c55689b5fce28: Bootstrap starting.
I20260709 12:26:59.183539 29851 raft_consensus.cc:2804] T 3615115233c34461b85763151751e475 P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:26:59.184166 29851 raft_consensus.cc:697] T 3615115233c34461b85763151751e475 P d73d92fb35234d16824c55689b5fce28 [term 1 LEADER]: Becoming Leader. State: Replica: d73d92fb35234d16824c55689b5fce28, State: Running, Role: LEADER
I20260709 12:26:59.185077 29851 consensus_queue.cc:237] T 3615115233c34461b85763151751e475 P d73d92fb35234d16824c55689b5fce28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:26:59.185662 29667 raft_consensus.cc:2468] T 3615115233c34461b85763151751e475 P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d73d92fb35234d16824c55689b5fce28 in term 1.
I20260709 12:26:59.186975 29856 raft_consensus.cc:2749] T 3615115233c34461b85763151751e475 P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:26:59.171190 29420 catalog_manager.cc:5697] T 8627b58b4f42491597125eb45f8d3b61 P c6610c00ffb948ce9c34f41dec3d49e8 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:59.190042 29790 raft_consensus.cc:359] T 8017a44219404faa8db8ff049c65205e P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:26:59.172688 29794 tablet_bootstrap.cc:492] T 3f41ce441fb047c8bbb741b1b9cbc0ec P d13c58ad8ac441c4a9b396eb04b1a476: No bootstrap required, opened a new log
I20260709 12:26:59.191177 29793 tablet_bootstrap.cc:654] T 6a30685caefb4d0e80fa4ea87d70760c P d73d92fb35234d16824c55689b5fce28: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:59.192072 29708 leader_election.cc:304] T b3c02c910d2b43cc9265bb1d98491cd3 P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c6610c00ffb948ce9c34f41dec3d49e8; no voters: 584dac86183a41d9b3bea1c2085e62fa, d73d92fb35234d16824c55689b5fce28
I20260709 12:26:59.193089 29799 raft_consensus.cc:3060] T b3c02c910d2b43cc9265bb1d98491cd3 P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:59.190788 29790 raft_consensus.cc:385] T 8017a44219404faa8db8ff049c65205e P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:59.191193 29794 ts_tablet_manager.cc:1403] T 3f41ce441fb047c8bbb741b1b9cbc0ec P d13c58ad8ac441c4a9b396eb04b1a476: Time spent bootstrapping tablet: real 0.191s	user 0.011s	sys 0.003s
I20260709 12:26:59.198657 29790 raft_consensus.cc:740] T 8017a44219404faa8db8ff049c65205e P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6610c00ffb948ce9c34f41dec3d49e8, State: Initialized, Role: FOLLOWER
I20260709 12:26:59.168707 29798 raft_consensus.cc:3060] T 8627b58b4f42491597125eb45f8d3b61 P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Advancing to term 3
W20260709 12:26:59.200776 29708 leader_election.cc:343] T 66f652ba2c224885a2aea974bfe8f88a P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:59.202045 29799 raft_consensus.cc:2749] T b3c02c910d2b43cc9265bb1d98491cd3 P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:26:59.202618 29892 consensus_queue.cc:1048] T b3c02c910d2b43cc9265bb1d98491cd3 P d73d92fb35234d16824c55689b5fce28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:26:59.203032 29954 consensus_queue.cc:237] T f4516786e5564e9eac08ca472f79a9f0 P 584dac86183a41d9b3bea1c2085e62fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:26:59.207264 29794 raft_consensus.cc:359] T 3f41ce441fb047c8bbb741b1b9cbc0ec P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:26:59.208495 29792 ts_tablet_manager.cc:1434] T 3f41ce441fb047c8bbb741b1b9cbc0ec P 584dac86183a41d9b3bea1c2085e62fa: Time spent starting tablet: real 0.191s	user 0.013s	sys 0.003s
I20260709 12:26:59.215739 29790 consensus_queue.cc:260] T 8017a44219404faa8db8ff049c65205e P c6610c00ffb948ce9c34f41dec3d49e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:26:59.209112 29794 raft_consensus.cc:385] T 3f41ce441fb047c8bbb741b1b9cbc0ec P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:59.221477 29794 raft_consensus.cc:740] T 3f41ce441fb047c8bbb741b1b9cbc0ec P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d13c58ad8ac441c4a9b396eb04b1a476, State: Initialized, Role: FOLLOWER
I20260709 12:26:59.217952 29954 consensus_queue.cc:1048] T 2707271fd46f456aa765026642a3efa9 P 584dac86183a41d9b3bea1c2085e62fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
I20260709 12:26:59.222460 29794 consensus_queue.cc:260] T 3f41ce441fb047c8bbb741b1b9cbc0ec P d13c58ad8ac441c4a9b396eb04b1a476 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:26:59.209057 29798 raft_consensus.cc:2749] T 8627b58b4f42491597125eb45f8d3b61 P d13c58ad8ac441c4a9b396eb04b1a476 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033) with higher term. Message: Invalid argument: T 8627b58b4f42491597125eb45f8d3b61 P 584dac86183a41d9b3bea1c2085e62fa [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d13c58ad8ac441c4a9b396eb04b1a476 for earlier term 2. Current term is 3.
I20260709 12:26:59.227377 29790 ts_tablet_manager.cc:1434] T 8017a44219404faa8db8ff049c65205e P c6610c00ffb948ce9c34f41dec3d49e8: Time spent starting tablet: real 0.076s	user 0.008s	sys 0.000s
I20260709 12:26:59.228907 29790 tablet_bootstrap.cc:492] T f4516786e5564e9eac08ca472f79a9f0 P c6610c00ffb948ce9c34f41dec3d49e8: Bootstrap starting.
I20260709 12:26:59.236433 29794 ts_tablet_manager.cc:1434] T 3f41ce441fb047c8bbb741b1b9cbc0ec P d13c58ad8ac441c4a9b396eb04b1a476: Time spent starting tablet: real 0.038s	user 0.006s	sys 0.001s
I20260709 12:26:59.237497 29794 tablet_bootstrap.cc:492] T 58154d49c1a344f2b51366ede3c87de5 P d13c58ad8ac441c4a9b396eb04b1a476: Bootstrap starting.
I20260709 12:26:59.242735 29987 raft_consensus.cc:493] T 3f41ce441fb047c8bbb741b1b9cbc0ec P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:59.243500 29987 raft_consensus.cc:515] T 3f41ce441fb047c8bbb741b1b9cbc0ec P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:26:59.248797 29892 consensus_queue.cc:1048] T b3c02c910d2b43cc9265bb1d98491cd3 P d73d92fb35234d16824c55689b5fce28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260709 12:26:59.253893 29985 log.cc:927] Time spent T 8627b58b4f42491597125eb45f8d3b61 P c6610c00ffb948ce9c34f41dec3d49e8: Append to log took a long time: real 0.115s	user 0.001s	sys 0.000s
I20260709 12:26:59.217300 29708 leader_election.cc:304] T 66f652ba2c224885a2aea974bfe8f88a P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c6610c00ffb948ce9c34f41dec3d49e8, d73d92fb35234d16824c55689b5fce28; no voters: 584dac86183a41d9b3bea1c2085e62fa
I20260709 12:26:59.256758 29799 raft_consensus.cc:2804] T 66f652ba2c224885a2aea974bfe8f88a P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:26:59.257256 29799 raft_consensus.cc:493] T 66f652ba2c224885a2aea974bfe8f88a P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:59.257807 29799 raft_consensus.cc:3060] T 66f652ba2c224885a2aea974bfe8f88a P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:59.261024 29858 consensus_queue.cc:1048] T ed057e06dec148a0b9d7062beeb9d005 P c6610c00ffb948ce9c34f41dec3d49e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:26:59.262872 29858 consensus_queue.cc:1048] T a46289e8fc824dd5a818e30df6e7ec3e P c6610c00ffb948ce9c34f41dec3d49e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:26:59.274257 29790 tablet_bootstrap.cc:654] T f4516786e5564e9eac08ca472f79a9f0 P c6610c00ffb948ce9c34f41dec3d49e8: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:59.305720 29792 tablet_bootstrap.cc:492] T 66f652ba2c224885a2aea974bfe8f88a P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap starting.
I20260709 12:26:59.301148 29798 raft_consensus.cc:493] T 3f41ce441fb047c8bbb741b1b9cbc0ec P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:59.314412 29798 raft_consensus.cc:515] T 3f41ce441fb047c8bbb741b1b9cbc0ec P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:26:59.325948 29793 tablet_bootstrap.cc:492] T 6a30685caefb4d0e80fa4ea87d70760c P d73d92fb35234d16824c55689b5fce28: No bootstrap required, opened a new log
I20260709 12:26:59.328655 29667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3f41ce441fb047c8bbb741b1b9cbc0ec" candidate_uuid: "584dac86183a41d9b3bea1c2085e62fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" is_pre_election: true
I20260709 12:26:59.332222 29792 tablet_bootstrap.cc:654] T 66f652ba2c224885a2aea974bfe8f88a P 584dac86183a41d9b3bea1c2085e62fa: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:59.332373 29987 leader_election.cc:290] T 3f41ce441fb047c8bbb741b1b9cbc0ec P 584dac86183a41d9b3bea1c2085e62fa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209), c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895)
I20260709 12:26:59.333425 29744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3f41ce441fb047c8bbb741b1b9cbc0ec" candidate_uuid: "584dac86183a41d9b3bea1c2085e62fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" is_pre_election: true
I20260709 12:26:59.343386 29522 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3f41ce441fb047c8bbb741b1b9cbc0ec" candidate_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "584dac86183a41d9b3bea1c2085e62fa" is_pre_election: true
I20260709 12:26:59.344095 29522 raft_consensus.cc:2468] T 3f41ce441fb047c8bbb741b1b9cbc0ec P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d13c58ad8ac441c4a9b396eb04b1a476 in term 0.
I20260709 12:26:59.338690 29420 catalog_manager.cc:5697] T 3615115233c34461b85763151751e475 P d73d92fb35234d16824c55689b5fce28 reported cstate change: term changed from 0 to 1, leader changed from <none> to d73d92fb35234d16824c55689b5fce28 (127.16.29.66). New cstate: current_term: 1 leader_uuid: "d73d92fb35234d16824c55689b5fce28" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:59.339282 29798 leader_election.cc:290] T 3f41ce441fb047c8bbb741b1b9cbc0ec P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895), 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033)
I20260709 12:26:59.330207 29799 raft_consensus.cc:515] T 66f652ba2c224885a2aea974bfe8f88a P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:26:59.353430 29799 leader_election.cc:290] T 66f652ba2c224885a2aea974bfe8f88a P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 1 election: Requested vote from peers 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033), d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825)
I20260709 12:26:59.339108 29667 raft_consensus.cc:2468] T 3f41ce441fb047c8bbb741b1b9cbc0ec P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 584dac86183a41d9b3bea1c2085e62fa in term 0.
I20260709 12:26:59.340302 29793 ts_tablet_manager.cc:1403] T 6a30685caefb4d0e80fa4ea87d70760c P d73d92fb35234d16824c55689b5fce28: Time spent bootstrapping tablet: real 0.157s	user 0.012s	sys 0.023s
I20260709 12:26:59.340342 29794 tablet_bootstrap.cc:654] T 58154d49c1a344f2b51366ede3c87de5 P d13c58ad8ac441c4a9b396eb04b1a476: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:59.350811 29744 raft_consensus.cc:2468] T 3f41ce441fb047c8bbb741b1b9cbc0ec P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 584dac86183a41d9b3bea1c2085e62fa in term 0.
I20260709 12:26:59.355283 29634 leader_election.cc:304] T 3f41ce441fb047c8bbb741b1b9cbc0ec P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 584dac86183a41d9b3bea1c2085e62fa, d13c58ad8ac441c4a9b396eb04b1a476; no voters: 
I20260709 12:26:59.356289 29743 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3f41ce441fb047c8bbb741b1b9cbc0ec" candidate_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" is_pre_election: true
I20260709 12:26:59.359354 29593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66f652ba2c224885a2aea974bfe8f88a" candidate_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d73d92fb35234d16824c55689b5fce28"
I20260709 12:26:59.362152 29520 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66f652ba2c224885a2aea974bfe8f88a" candidate_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "584dac86183a41d9b3bea1c2085e62fa"
W20260709 12:26:59.373023 29708 leader_election.cc:343] T 66f652ba2c224885a2aea974bfe8f88a P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:59.373675 29593 raft_consensus.cc:3060] T 66f652ba2c224885a2aea974bfe8f88a P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:59.380228 29486 leader_election.cc:304] T 3f41ce441fb047c8bbb741b1b9cbc0ec P 584dac86183a41d9b3bea1c2085e62fa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 584dac86183a41d9b3bea1c2085e62fa, d13c58ad8ac441c4a9b396eb04b1a476; no voters: 
I20260709 12:26:59.381232 29954 raft_consensus.cc:2804] T 3f41ce441fb047c8bbb741b1b9cbc0ec P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:26:59.381847 29954 raft_consensus.cc:493] T 3f41ce441fb047c8bbb741b1b9cbc0ec P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:59.382315 29954 raft_consensus.cc:3060] T 3f41ce441fb047c8bbb741b1b9cbc0ec P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:59.373808 29743 raft_consensus.cc:2468] T 3f41ce441fb047c8bbb741b1b9cbc0ec P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d13c58ad8ac441c4a9b396eb04b1a476 in term 0.
I20260709 12:26:59.374445 29798 raft_consensus.cc:2804] T 3f41ce441fb047c8bbb741b1b9cbc0ec P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:26:59.387529 29798 raft_consensus.cc:493] T 3f41ce441fb047c8bbb741b1b9cbc0ec P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:59.392726 29798 raft_consensus.cc:3060] T 3f41ce441fb047c8bbb741b1b9cbc0ec P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:59.401814 29793 raft_consensus.cc:359] T 6a30685caefb4d0e80fa4ea87d70760c P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:26:59.403870 29793 raft_consensus.cc:385] T 6a30685caefb4d0e80fa4ea87d70760c P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:59.404496 29793 raft_consensus.cc:740] T 6a30685caefb4d0e80fa4ea87d70760c P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d73d92fb35234d16824c55689b5fce28, State: Initialized, Role: FOLLOWER
I20260709 12:26:59.407589 29954 raft_consensus.cc:515] T 3f41ce441fb047c8bbb741b1b9cbc0ec P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:26:59.411420 29954 leader_election.cc:290] T 3f41ce441fb047c8bbb741b1b9cbc0ec P 584dac86183a41d9b3bea1c2085e62fa [CANDIDATE]: Term 1 election: Requested vote from peers d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209), c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895)
I20260709 12:26:59.411643 29743 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3f41ce441fb047c8bbb741b1b9cbc0ec" candidate_uuid: "584dac86183a41d9b3bea1c2085e62fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6610c00ffb948ce9c34f41dec3d49e8"
I20260709 12:26:59.412302 29743 raft_consensus.cc:3060] T 3f41ce441fb047c8bbb741b1b9cbc0ec P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:59.412145 29799 consensus_queue.cc:1048] T a46289e8fc824dd5a818e30df6e7ec3e P c6610c00ffb948ce9c34f41dec3d49e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:26:59.405372 29793 consensus_queue.cc:260] T 6a30685caefb4d0e80fa4ea87d70760c P d73d92fb35234d16824c55689b5fce28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:26:59.419996 29667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3f41ce441fb047c8bbb741b1b9cbc0ec" candidate_uuid: "584dac86183a41d9b3bea1c2085e62fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d13c58ad8ac441c4a9b396eb04b1a476"
I20260709 12:26:59.425387 29892 raft_consensus.cc:493] T ed057e06dec148a0b9d7062beeb9d005 P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c6610c00ffb948ce9c34f41dec3d49e8)
I20260709 12:26:59.426628 29892 raft_consensus.cc:515] T ed057e06dec148a0b9d7062beeb9d005 P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:26:59.432825 29790 tablet_bootstrap.cc:492] T f4516786e5564e9eac08ca472f79a9f0 P c6610c00ffb948ce9c34f41dec3d49e8: No bootstrap required, opened a new log
I20260709 12:26:59.433358 29790 ts_tablet_manager.cc:1403] T f4516786e5564e9eac08ca472f79a9f0 P c6610c00ffb948ce9c34f41dec3d49e8: Time spent bootstrapping tablet: real 0.205s	user 0.011s	sys 0.029s
I20260709 12:26:59.436985 29790 raft_consensus.cc:359] T f4516786e5564e9eac08ca472f79a9f0 P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:26:59.438102 29790 raft_consensus.cc:385] T f4516786e5564e9eac08ca472f79a9f0 P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:59.438509 29790 raft_consensus.cc:740] T f4516786e5564e9eac08ca472f79a9f0 P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6610c00ffb948ce9c34f41dec3d49e8, State: Initialized, Role: FOLLOWER
I20260709 12:26:59.440896 29420 catalog_manager.cc:5697] T 71e2de235adf46d9a2407659080d42ef P 584dac86183a41d9b3bea1c2085e62fa reported cstate change: term changed from 0 to 1, leader changed from <none> to 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65). New cstate: current_term: 1 leader_uuid: "584dac86183a41d9b3bea1c2085e62fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:59.447026 29962 raft_consensus.cc:493] T 8017a44219404faa8db8ff049c65205e P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:59.447701 29744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed057e06dec148a0b9d7062beeb9d005" candidate_uuid: "d73d92fb35234d16824c55689b5fce28" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" is_pre_election: true
I20260709 12:26:59.447543 29962 raft_consensus.cc:515] T 8017a44219404faa8db8ff049c65205e P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:26:59.449012 29668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed057e06dec148a0b9d7062beeb9d005" candidate_uuid: "d73d92fb35234d16824c55689b5fce28" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" is_pre_election: true
I20260709 12:26:59.449762 29962 leader_election.cc:290] T 8017a44219404faa8db8ff049c65205e P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033), d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825)
I20260709 12:26:59.451853 29594 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8017a44219404faa8db8ff049c65205e" candidate_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d73d92fb35234d16824c55689b5fce28" is_pre_election: true
I20260709 12:26:59.452279 29521 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8017a44219404faa8db8ff049c65205e" candidate_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "584dac86183a41d9b3bea1c2085e62fa" is_pre_election: true
I20260709 12:26:59.452965 29521 raft_consensus.cc:2468] T 8017a44219404faa8db8ff049c65205e P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6610c00ffb948ce9c34f41dec3d49e8 in term 0.
I20260709 12:26:59.451820 29561 leader_election.cc:304] T ed057e06dec148a0b9d7062beeb9d005 P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d73d92fb35234d16824c55689b5fce28; no voters: c6610c00ffb948ce9c34f41dec3d49e8, d13c58ad8ac441c4a9b396eb04b1a476
I20260709 12:26:59.454802 30003 raft_consensus.cc:2749] T ed057e06dec148a0b9d7062beeb9d005 P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:26:59.446645 29892 leader_election.cc:290] T ed057e06dec148a0b9d7062beeb9d005 P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209), c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895)
I20260709 12:26:59.446642 29798 raft_consensus.cc:515] T 3f41ce441fb047c8bbb741b1b9cbc0ec P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:26:59.457939 29667 raft_consensus.cc:2393] T 3f41ce441fb047c8bbb741b1b9cbc0ec P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 584dac86183a41d9b3bea1c2085e62fa in current term 1: Already voted for candidate d13c58ad8ac441c4a9b396eb04b1a476 in this term.
W20260709 12:26:59.458045 29708 leader_election.cc:343] T 8017a44219404faa8db8ff049c65205e P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:59.458220 29593 raft_consensus.cc:2468] T 66f652ba2c224885a2aea974bfe8f88a P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6610c00ffb948ce9c34f41dec3d49e8 in term 1.
I20260709 12:26:59.459213 29708 leader_election.cc:304] T 8017a44219404faa8db8ff049c65205e P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 584dac86183a41d9b3bea1c2085e62fa, c6610c00ffb948ce9c34f41dec3d49e8; no voters: d73d92fb35234d16824c55689b5fce28
I20260709 12:26:59.460531 29860 raft_consensus.cc:2804] T 8017a44219404faa8db8ff049c65205e P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:26:59.460551 29708 leader_election.cc:304] T 66f652ba2c224885a2aea974bfe8f88a P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c6610c00ffb948ce9c34f41dec3d49e8, d73d92fb35234d16824c55689b5fce28; no voters: 584dac86183a41d9b3bea1c2085e62fa
I20260709 12:26:59.460994 29860 raft_consensus.cc:493] T 8017a44219404faa8db8ff049c65205e P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:59.461359 29860 raft_consensus.cc:3060] T 8017a44219404faa8db8ff049c65205e P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:59.461817 29962 raft_consensus.cc:2804] T 66f652ba2c224885a2aea974bfe8f88a P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:26:59.462525 29962 raft_consensus.cc:697] T 66f652ba2c224885a2aea974bfe8f88a P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 LEADER]: Becoming Leader. State: Replica: c6610c00ffb948ce9c34f41dec3d49e8, State: Running, Role: LEADER
I20260709 12:26:59.463346 29962 consensus_queue.cc:237] T 66f652ba2c224885a2aea974bfe8f88a P c6610c00ffb948ce9c34f41dec3d49e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:26:59.464872 29790 consensus_queue.cc:260] T f4516786e5564e9eac08ca472f79a9f0 P c6610c00ffb948ce9c34f41dec3d49e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:26:59.469326 29790 ts_tablet_manager.cc:1434] T f4516786e5564e9eac08ca472f79a9f0 P c6610c00ffb948ce9c34f41dec3d49e8: Time spent starting tablet: real 0.036s	user 0.008s	sys 0.000s
I20260709 12:26:59.470600 29790 tablet_bootstrap.cc:492] T 76140ad6258543118a9bfeee7cf0d5fc P c6610c00ffb948ce9c34f41dec3d49e8: Bootstrap starting.
I20260709 12:26:59.475364 29794 tablet_bootstrap.cc:492] T 58154d49c1a344f2b51366ede3c87de5 P d13c58ad8ac441c4a9b396eb04b1a476: No bootstrap required, opened a new log
I20260709 12:26:59.475407 29744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3f41ce441fb047c8bbb741b1b9cbc0ec" candidate_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6610c00ffb948ce9c34f41dec3d49e8"
I20260709 12:26:59.476384 29794 ts_tablet_manager.cc:1403] T 58154d49c1a344f2b51366ede3c87de5 P d13c58ad8ac441c4a9b396eb04b1a476: Time spent bootstrapping tablet: real 0.239s	user 0.014s	sys 0.014s
I20260709 12:26:59.480516 29798 leader_election.cc:290] T 3f41ce441fb047c8bbb741b1b9cbc0ec P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 1 election: Requested vote from peers c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895), 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033)
I20260709 12:26:59.482065 29521 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3f41ce441fb047c8bbb741b1b9cbc0ec" candidate_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "584dac86183a41d9b3bea1c2085e62fa"
I20260709 12:26:59.483939 29521 raft_consensus.cc:2393] T 3f41ce441fb047c8bbb741b1b9cbc0ec P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate d13c58ad8ac441c4a9b396eb04b1a476 in current term 1: Already voted for candidate 584dac86183a41d9b3bea1c2085e62fa in this term.
I20260709 12:26:59.485245 29634 leader_election.cc:304] T 3f41ce441fb047c8bbb741b1b9cbc0ec P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d13c58ad8ac441c4a9b396eb04b1a476; no voters: 584dac86183a41d9b3bea1c2085e62fa, c6610c00ffb948ce9c34f41dec3d49e8
I20260709 12:26:59.486564 29798 raft_consensus.cc:2749] T 3f41ce441fb047c8bbb741b1b9cbc0ec P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 12:26:59.487190 29793 ts_tablet_manager.cc:1434] T 6a30685caefb4d0e80fa4ea87d70760c P d73d92fb35234d16824c55689b5fce28: Time spent starting tablet: real 0.123s	user 0.032s	sys 0.014s
I20260709 12:26:59.477914 29420 catalog_manager.cc:5697] T f4516786e5564e9eac08ca472f79a9f0 P 584dac86183a41d9b3bea1c2085e62fa reported cstate change: term changed from 0 to 1, leader changed from <none> to 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65). New cstate: current_term: 1 leader_uuid: "584dac86183a41d9b3bea1c2085e62fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } health_report { overall_health: HEALTHY } } }
I20260709 12:26:59.488365 29793 tablet_bootstrap.cc:492] T 8017a44219404faa8db8ff049c65205e P d73d92fb35234d16824c55689b5fce28: Bootstrap starting.
I20260709 12:26:59.489442 29860 raft_consensus.cc:515] T 8017a44219404faa8db8ff049c65205e P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:26:59.491581 29860 leader_election.cc:290] T 8017a44219404faa8db8ff049c65205e P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 1 election: Requested vote from peers 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033), d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825)
I20260709 12:26:59.492422 29743 raft_consensus.cc:2468] T 3f41ce441fb047c8bbb741b1b9cbc0ec P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 584dac86183a41d9b3bea1c2085e62fa in term 1.
I20260709 12:26:59.493541 29486 leader_election.cc:304] T 3f41ce441fb047c8bbb741b1b9cbc0ec P 584dac86183a41d9b3bea1c2085e62fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 584dac86183a41d9b3bea1c2085e62fa, c6610c00ffb948ce9c34f41dec3d49e8; no voters: d13c58ad8ac441c4a9b396eb04b1a476
I20260709 12:26:59.494151 29794 raft_consensus.cc:359] T 58154d49c1a344f2b51366ede3c87de5 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:26:59.494892 29794 raft_consensus.cc:385] T 58154d49c1a344f2b51366ede3c87de5 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:59.494740 29954 raft_consensus.cc:2804] T 3f41ce441fb047c8bbb741b1b9cbc0ec P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:26:59.495501 29793 tablet_bootstrap.cc:654] T 8017a44219404faa8db8ff049c65205e P d73d92fb35234d16824c55689b5fce28: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:59.496301 29954 raft_consensus.cc:697] T 3f41ce441fb047c8bbb741b1b9cbc0ec P 584dac86183a41d9b3bea1c2085e62fa [term 1 LEADER]: Becoming Leader. State: Replica: 584dac86183a41d9b3bea1c2085e62fa, State: Running, Role: LEADER
I20260709 12:26:59.497205 29954 consensus_queue.cc:237] T 3f41ce441fb047c8bbb741b1b9cbc0ec P 584dac86183a41d9b3bea1c2085e62fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:26:59.494736 29596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8017a44219404faa8db8ff049c65205e" candidate_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d73d92fb35234d16824c55689b5fce28"
I20260709 12:26:59.495249 29794 raft_consensus.cc:740] T 58154d49c1a344f2b51366ede3c87de5 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d13c58ad8ac441c4a9b396eb04b1a476, State: Initialized, Role: FOLLOWER
I20260709 12:26:59.500826 29794 consensus_queue.cc:260] T 58154d49c1a344f2b51366ede3c87de5 P d13c58ad8ac441c4a9b396eb04b1a476 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
W20260709 12:26:59.501854 29708 leader_election.cc:343] T 8017a44219404faa8db8ff049c65205e P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:59.503062 29792 tablet_bootstrap.cc:492] T 66f652ba2c224885a2aea974bfe8f88a P 584dac86183a41d9b3bea1c2085e62fa: No bootstrap required, opened a new log
I20260709 12:26:59.505837 29792 ts_tablet_manager.cc:1403] T 66f652ba2c224885a2aea974bfe8f88a P 584dac86183a41d9b3bea1c2085e62fa: Time spent bootstrapping tablet: real 0.200s	user 0.014s	sys 0.003s
I20260709 12:26:59.506364 29521 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8017a44219404faa8db8ff049c65205e" candidate_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "584dac86183a41d9b3bea1c2085e62fa"
I20260709 12:26:59.507002 29521 raft_consensus.cc:3060] T 8017a44219404faa8db8ff049c65205e P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:59.509454 29792 raft_consensus.cc:359] T 66f652ba2c224885a2aea974bfe8f88a P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:26:59.510488 29792 raft_consensus.cc:385] T 66f652ba2c224885a2aea974bfe8f88a P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:59.510833 29792 raft_consensus.cc:740] T 66f652ba2c224885a2aea974bfe8f88a P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 584dac86183a41d9b3bea1c2085e62fa, State: Initialized, Role: FOLLOWER
I20260709 12:26:59.524499 29792 consensus_queue.cc:260] T 66f652ba2c224885a2aea974bfe8f88a P 584dac86183a41d9b3bea1c2085e62fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:26:59.514521 29794 ts_tablet_manager.cc:1434] T 58154d49c1a344f2b51366ede3c87de5 P d13c58ad8ac441c4a9b396eb04b1a476: Time spent starting tablet: real 0.038s	user 0.006s	sys 0.001s
I20260709 12:26:59.526744 29794 tablet_bootstrap.cc:492] T d1918297b9eb41f5847a211e668ae748 P d13c58ad8ac441c4a9b396eb04b1a476: Bootstrap starting.
I20260709 12:26:59.536417 29792 ts_tablet_manager.cc:1434] T 66f652ba2c224885a2aea974bfe8f88a P 584dac86183a41d9b3bea1c2085e62fa: Time spent starting tablet: real 0.030s	user 0.009s	sys 0.000s
I20260709 12:26:59.523564 29790 tablet_bootstrap.cc:654] T 76140ad6258543118a9bfeee7cf0d5fc P c6610c00ffb948ce9c34f41dec3d49e8: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:59.541316 29792 tablet_bootstrap.cc:492] T 16649fd17d434e3c95e9d585d2404659 P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap starting.
I20260709 12:26:59.546352 29794 tablet_bootstrap.cc:654] T d1918297b9eb41f5847a211e668ae748 P d13c58ad8ac441c4a9b396eb04b1a476: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:59.548904 29521 raft_consensus.cc:2468] T 8017a44219404faa8db8ff049c65205e P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6610c00ffb948ce9c34f41dec3d49e8 in term 1.
I20260709 12:26:59.550123 29793 tablet_bootstrap.cc:492] T 8017a44219404faa8db8ff049c65205e P d73d92fb35234d16824c55689b5fce28: No bootstrap required, opened a new log
I20260709 12:26:59.550607 29793 ts_tablet_manager.cc:1403] T 8017a44219404faa8db8ff049c65205e P d73d92fb35234d16824c55689b5fce28: Time spent bootstrapping tablet: real 0.063s	user 0.016s	sys 0.035s
I20260709 12:26:59.550815 29792 tablet_bootstrap.cc:654] T 16649fd17d434e3c95e9d585d2404659 P 584dac86183a41d9b3bea1c2085e62fa: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:59.550961 29708 leader_election.cc:304] T 8017a44219404faa8db8ff049c65205e P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 584dac86183a41d9b3bea1c2085e62fa, c6610c00ffb948ce9c34f41dec3d49e8; no voters: d73d92fb35234d16824c55689b5fce28
I20260709 12:26:59.551843 29860 raft_consensus.cc:2804] T 8017a44219404faa8db8ff049c65205e P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:26:59.552448 30006 raft_consensus.cc:493] T 58154d49c1a344f2b51366ede3c87de5 P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:59.552989 30006 raft_consensus.cc:515] T 58154d49c1a344f2b51366ede3c87de5 P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:26:59.554883 29860 raft_consensus.cc:697] T 8017a44219404faa8db8ff049c65205e P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 LEADER]: Becoming Leader. State: Replica: c6610c00ffb948ce9c34f41dec3d49e8, State: Running, Role: LEADER
I20260709 12:26:59.555716 30006 leader_election.cc:290] T 58154d49c1a344f2b51366ede3c87de5 P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209), c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895)
I20260709 12:26:59.555824 29860 consensus_queue.cc:237] T 8017a44219404faa8db8ff049c65205e P c6610c00ffb948ce9c34f41dec3d49e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:26:59.556380 29793 raft_consensus.cc:359] T 8017a44219404faa8db8ff049c65205e P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:26:59.557096 29793 raft_consensus.cc:385] T 8017a44219404faa8db8ff049c65205e P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:59.557344 29793 raft_consensus.cc:740] T 8017a44219404faa8db8ff049c65205e P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d73d92fb35234d16824c55689b5fce28, State: Initialized, Role: FOLLOWER
I20260709 12:26:59.559058 29669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58154d49c1a344f2b51366ede3c87de5" candidate_uuid: "d73d92fb35234d16824c55689b5fce28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" is_pre_election: true
I20260709 12:26:59.559669 29743 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58154d49c1a344f2b51366ede3c87de5" candidate_uuid: "d73d92fb35234d16824c55689b5fce28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" is_pre_election: true
I20260709 12:26:59.560387 29743 raft_consensus.cc:2468] T 58154d49c1a344f2b51366ede3c87de5 P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d73d92fb35234d16824c55689b5fce28 in term 0.
I20260709 12:26:59.560575 29790 tablet_bootstrap.cc:492] T 76140ad6258543118a9bfeee7cf0d5fc P c6610c00ffb948ce9c34f41dec3d49e8: No bootstrap required, opened a new log
I20260709 12:26:59.561369 29790 ts_tablet_manager.cc:1403] T 76140ad6258543118a9bfeee7cf0d5fc P c6610c00ffb948ce9c34f41dec3d49e8: Time spent bootstrapping tablet: real 0.091s	user 0.014s	sys 0.001s
I20260709 12:26:59.561973 29561 leader_election.cc:304] T 58154d49c1a344f2b51366ede3c87de5 P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c6610c00ffb948ce9c34f41dec3d49e8, d73d92fb35234d16824c55689b5fce28; no voters: 
I20260709 12:26:59.562922 30006 raft_consensus.cc:2804] T 58154d49c1a344f2b51366ede3c87de5 P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:26:59.563114 29669 raft_consensus.cc:2468] T 58154d49c1a344f2b51366ede3c87de5 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d73d92fb35234d16824c55689b5fce28 in term 0.
I20260709 12:26:59.563308 30006 raft_consensus.cc:493] T 58154d49c1a344f2b51366ede3c87de5 P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:59.563689 30006 raft_consensus.cc:3060] T 58154d49c1a344f2b51366ede3c87de5 P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:59.555119 29419 catalog_manager.cc:5697] T 66f652ba2c224885a2aea974bfe8f88a P c6610c00ffb948ce9c34f41dec3d49e8 reported cstate change: term changed from 0 to 1, leader changed from <none> to c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68). New cstate: current_term: 1 leader_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:59.565197 29790 raft_consensus.cc:359] T 76140ad6258543118a9bfeee7cf0d5fc P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:26:59.566193 29790 raft_consensus.cc:385] T 76140ad6258543118a9bfeee7cf0d5fc P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:59.566534 29790 raft_consensus.cc:740] T 76140ad6258543118a9bfeee7cf0d5fc P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6610c00ffb948ce9c34f41dec3d49e8, State: Initialized, Role: FOLLOWER
I20260709 12:26:59.567806 29793 consensus_queue.cc:260] T 8017a44219404faa8db8ff049c65205e P d73d92fb35234d16824c55689b5fce28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:26:59.570905 29793 ts_tablet_manager.cc:1434] T 8017a44219404faa8db8ff049c65205e P d73d92fb35234d16824c55689b5fce28: Time spent starting tablet: real 0.020s	user 0.005s	sys 0.002s
I20260709 12:26:59.574321 29793 tablet_bootstrap.cc:492] T 07a2b94c0823467695508292a9ccdcc5 P d73d92fb35234d16824c55689b5fce28: Bootstrap starting.
I20260709 12:26:59.579849 29962 raft_consensus.cc:493] T f4516786e5564e9eac08ca472f79a9f0 P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:59.580479 29962 raft_consensus.cc:515] T f4516786e5564e9eac08ca472f79a9f0 P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:26:59.580402 29790 consensus_queue.cc:260] T 76140ad6258543118a9bfeee7cf0d5fc P c6610c00ffb948ce9c34f41dec3d49e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:26:59.582885 29962 leader_election.cc:290] T f4516786e5564e9eac08ca472f79a9f0 P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209), 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033)
I20260709 12:26:59.585417 29790 ts_tablet_manager.cc:1434] T 76140ad6258543118a9bfeee7cf0d5fc P c6610c00ffb948ce9c34f41dec3d49e8: Time spent starting tablet: real 0.024s	user 0.007s	sys 0.001s
I20260709 12:26:59.590906 29790 tablet_bootstrap.cc:492] T 6a30685caefb4d0e80fa4ea87d70760c P c6610c00ffb948ce9c34f41dec3d49e8: Bootstrap starting.
I20260709 12:26:59.590134 29418 catalog_manager.cc:5697] T 3f41ce441fb047c8bbb741b1b9cbc0ec P 584dac86183a41d9b3bea1c2085e62fa reported cstate change: term changed from 0 to 1, leader changed from <none> to 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65). New cstate: current_term: 1 leader_uuid: "584dac86183a41d9b3bea1c2085e62fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } health_report { overall_health: HEALTHY } } }
I20260709 12:26:59.580850 29953 consensus_queue.cc:1048] T 71e2de235adf46d9a2407659080d42ef P 584dac86183a41d9b3bea1c2085e62fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:26:59.599961 29793 tablet_bootstrap.cc:654] T 07a2b94c0823467695508292a9ccdcc5 P d73d92fb35234d16824c55689b5fce28: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:59.608747 29669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4516786e5564e9eac08ca472f79a9f0" candidate_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" is_pre_election: true
I20260709 12:26:59.609673 29669 raft_consensus.cc:2393] T f4516786e5564e9eac08ca472f79a9f0 P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c6610c00ffb948ce9c34f41dec3d49e8 in current term 1: Already voted for candidate 584dac86183a41d9b3bea1c2085e62fa in this term.
I20260709 12:26:59.616626 29520 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4516786e5564e9eac08ca472f79a9f0" candidate_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "584dac86183a41d9b3bea1c2085e62fa" is_pre_election: true
I20260709 12:26:59.620030 29520 raft_consensus.cc:1275] T 8627b58b4f42491597125eb45f8d3b61 P 584dac86183a41d9b3bea1c2085e62fa [term 3 FOLLOWER]: Refusing update from remote peer c6610c00ffb948ce9c34f41dec3d49e8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 12:26:59.620268 29790 tablet_bootstrap.cc:654] T 6a30685caefb4d0e80fa4ea87d70760c P c6610c00ffb948ce9c34f41dec3d49e8: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:59.622164 29708 leader_election.cc:304] T f4516786e5564e9eac08ca472f79a9f0 P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c6610c00ffb948ce9c34f41dec3d49e8; no voters: 584dac86183a41d9b3bea1c2085e62fa, d13c58ad8ac441c4a9b396eb04b1a476
I20260709 12:26:59.623226 29962 raft_consensus.cc:3060] T f4516786e5564e9eac08ca472f79a9f0 P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:59.627132 29793 tablet_bootstrap.cc:492] T 07a2b94c0823467695508292a9ccdcc5 P d73d92fb35234d16824c55689b5fce28: No bootstrap required, opened a new log
I20260709 12:26:59.627679 29793 ts_tablet_manager.cc:1403] T 07a2b94c0823467695508292a9ccdcc5 P d73d92fb35234d16824c55689b5fce28: Time spent bootstrapping tablet: real 0.054s	user 0.019s	sys 0.002s
I20260709 12:26:59.636737 29793 raft_consensus.cc:359] T 07a2b94c0823467695508292a9ccdcc5 P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:26:59.645396 29793 raft_consensus.cc:385] T 07a2b94c0823467695508292a9ccdcc5 P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:59.645882 29793 raft_consensus.cc:740] T 07a2b94c0823467695508292a9ccdcc5 P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d73d92fb35234d16824c55689b5fce28, State: Initialized, Role: FOLLOWER
I20260709 12:26:59.648793 29858 consensus_queue.cc:1048] T 8627b58b4f42491597125eb45f8d3b61 P c6610c00ffb948ce9c34f41dec3d49e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:26:59.651852 29989 consensus_queue.cc:1048] T 71e2de235adf46d9a2407659080d42ef P 584dac86183a41d9b3bea1c2085e62fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:26:59.651063 29793 consensus_queue.cc:260] T 07a2b94c0823467695508292a9ccdcc5 P d73d92fb35234d16824c55689b5fce28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:26:59.657368 29743 raft_consensus.cc:3060] T 76140ad6258543118a9bfeee7cf0d5fc P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:59.659651 29793 ts_tablet_manager.cc:1434] T 07a2b94c0823467695508292a9ccdcc5 P d73d92fb35234d16824c55689b5fce28: Time spent starting tablet: real 0.032s	user 0.008s	sys 0.000s
I20260709 12:26:59.664100 29793 tablet_bootstrap.cc:492] T 959940284eb24fdf83d78438d710318a P d73d92fb35234d16824c55689b5fce28: Bootstrap starting.
I20260709 12:26:59.679929 29989 consensus_queue.cc:1048] T f4516786e5564e9eac08ca472f79a9f0 P 584dac86183a41d9b3bea1c2085e62fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:26:59.685213 29667 raft_consensus.cc:1275] T 8627b58b4f42491597125eb45f8d3b61 P d13c58ad8ac441c4a9b396eb04b1a476 [term 3 FOLLOWER]: Refusing update from remote peer c6610c00ffb948ce9c34f41dec3d49e8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 12:26:59.687454 29858 consensus_queue.cc:1048] T 8627b58b4f42491597125eb45f8d3b61 P c6610c00ffb948ce9c34f41dec3d49e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:26:59.686992 29793 tablet_bootstrap.cc:654] T 959940284eb24fdf83d78438d710318a P d73d92fb35234d16824c55689b5fce28: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:59.629168 30006 raft_consensus.cc:515] T 58154d49c1a344f2b51366ede3c87de5 P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:26:59.736359 29792 tablet_bootstrap.cc:492] T 16649fd17d434e3c95e9d585d2404659 P 584dac86183a41d9b3bea1c2085e62fa: No bootstrap required, opened a new log
I20260709 12:26:59.736956 29792 ts_tablet_manager.cc:1403] T 16649fd17d434e3c95e9d585d2404659 P 584dac86183a41d9b3bea1c2085e62fa: Time spent bootstrapping tablet: real 0.196s	user 0.068s	sys 0.098s
I20260709 12:26:59.699330 29962 raft_consensus.cc:2749] T f4516786e5564e9eac08ca472f79a9f0 P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:26:59.755180 29794 tablet_bootstrap.cc:492] T d1918297b9eb41f5847a211e668ae748 P d13c58ad8ac441c4a9b396eb04b1a476: No bootstrap required, opened a new log
I20260709 12:26:59.755741 29794 ts_tablet_manager.cc:1403] T d1918297b9eb41f5847a211e668ae748 P d13c58ad8ac441c4a9b396eb04b1a476: Time spent bootstrapping tablet: real 0.229s	user 0.032s	sys 0.032s
I20260709 12:26:59.765137 29851 raft_consensus.cc:493] T 6a30685caefb4d0e80fa4ea87d70760c P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:59.765923 29851 raft_consensus.cc:515] T 6a30685caefb4d0e80fa4ea87d70760c P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:26:59.769550 30006 leader_election.cc:290] T 58154d49c1a344f2b51366ede3c87de5 P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 1 election: Requested vote from peers d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209), c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895)
I20260709 12:26:59.780725 29794 raft_consensus.cc:359] T d1918297b9eb41f5847a211e668ae748 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:26:59.782239 29794 raft_consensus.cc:385] T d1918297b9eb41f5847a211e668ae748 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:59.784802 29670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58154d49c1a344f2b51366ede3c87de5" candidate_uuid: "d73d92fb35234d16824c55689b5fce28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d13c58ad8ac441c4a9b396eb04b1a476"
I20260709 12:26:59.785517 29670 raft_consensus.cc:3060] T 58154d49c1a344f2b51366ede3c87de5 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:59.787806 29744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58154d49c1a344f2b51366ede3c87de5" candidate_uuid: "d73d92fb35234d16824c55689b5fce28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6610c00ffb948ce9c34f41dec3d49e8"
I20260709 12:26:59.782562 29794 raft_consensus.cc:740] T d1918297b9eb41f5847a211e668ae748 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d13c58ad8ac441c4a9b396eb04b1a476, State: Initialized, Role: FOLLOWER
I20260709 12:26:59.789244 29794 consensus_queue.cc:260] T d1918297b9eb41f5847a211e668ae748 P d13c58ad8ac441c4a9b396eb04b1a476 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:26:59.788477 29744 raft_consensus.cc:3060] T 58154d49c1a344f2b51366ede3c87de5 P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:59.799738 29794 ts_tablet_manager.cc:1434] T d1918297b9eb41f5847a211e668ae748 P d13c58ad8ac441c4a9b396eb04b1a476: Time spent starting tablet: real 0.044s	user 0.006s	sys 0.000s
I20260709 12:26:59.801014 29794 tablet_bootstrap.cc:492] T 07a2b94c0823467695508292a9ccdcc5 P d13c58ad8ac441c4a9b396eb04b1a476: Bootstrap starting.
I20260709 12:26:59.788600 29745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6a30685caefb4d0e80fa4ea87d70760c" candidate_uuid: "d73d92fb35234d16824c55689b5fce28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" is_pre_election: true
W20260709 12:26:59.803234 30018 log.cc:927] Time spent T 8627b58b4f42491597125eb45f8d3b61 P 584dac86183a41d9b3bea1c2085e62fa: Append to log took a long time: real 0.071s	user 0.001s	sys 0.001s
I20260709 12:26:59.810827 29794 tablet_bootstrap.cc:654] T 07a2b94c0823467695508292a9ccdcc5 P d13c58ad8ac441c4a9b396eb04b1a476: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:59.813427 29521 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6a30685caefb4d0e80fa4ea87d70760c" candidate_uuid: "d73d92fb35234d16824c55689b5fce28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "584dac86183a41d9b3bea1c2085e62fa" is_pre_election: true
W20260709 12:26:59.820874 29560 leader_election.cc:343] T 6a30685caefb4d0e80fa4ea87d70760c P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:26:59.828856 29561 leader_election.cc:343] T 6a30685caefb4d0e80fa4ea87d70760c P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:59.829351 29561 leader_election.cc:304] T 6a30685caefb4d0e80fa4ea87d70760c P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d73d92fb35234d16824c55689b5fce28; no voters: 584dac86183a41d9b3bea1c2085e62fa, c6610c00ffb948ce9c34f41dec3d49e8
I20260709 12:26:59.832068 30006 raft_consensus.cc:2749] T 6a30685caefb4d0e80fa4ea87d70760c P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:26:59.832628 29851 leader_election.cc:290] T 6a30685caefb4d0e80fa4ea87d70760c P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895), 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033)
I20260709 12:26:59.832966 29989 consensus_queue.cc:1048] T f4516786e5564e9eac08ca472f79a9f0 P 584dac86183a41d9b3bea1c2085e62fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:26:59.840798 29793 tablet_bootstrap.cc:492] T 959940284eb24fdf83d78438d710318a P d73d92fb35234d16824c55689b5fce28: No bootstrap required, opened a new log
I20260709 12:26:59.841316 29793 ts_tablet_manager.cc:1403] T 959940284eb24fdf83d78438d710318a P d73d92fb35234d16824c55689b5fce28: Time spent bootstrapping tablet: real 0.178s	user 0.020s	sys 0.000s
I20260709 12:26:59.855156 29852 consensus_queue.cc:1048] T 3615115233c34461b85763151751e475 P d73d92fb35234d16824c55689b5fce28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:26:59.847293 29792 raft_consensus.cc:359] T 16649fd17d434e3c95e9d585d2404659 P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:26:59.858006 29792 raft_consensus.cc:385] T 16649fd17d434e3c95e9d585d2404659 P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:59.858366 29792 raft_consensus.cc:740] T 16649fd17d434e3c95e9d585d2404659 P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 584dac86183a41d9b3bea1c2085e62fa, State: Initialized, Role: FOLLOWER
I20260709 12:26:59.861146 29670 raft_consensus.cc:2468] T 58154d49c1a344f2b51366ede3c87de5 P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d73d92fb35234d16824c55689b5fce28 in term 1.
I20260709 12:26:59.862830 29561 leader_election.cc:304] T 58154d49c1a344f2b51366ede3c87de5 P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d13c58ad8ac441c4a9b396eb04b1a476, d73d92fb35234d16824c55689b5fce28; no voters: 
I20260709 12:26:59.863286 29792 consensus_queue.cc:260] T 16649fd17d434e3c95e9d585d2404659 P 584dac86183a41d9b3bea1c2085e62fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:26:59.865250 30006 raft_consensus.cc:2804] T 58154d49c1a344f2b51366ede3c87de5 P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:26:59.865880 30006 raft_consensus.cc:697] T 58154d49c1a344f2b51366ede3c87de5 P d73d92fb35234d16824c55689b5fce28 [term 1 LEADER]: Becoming Leader. State: Replica: d73d92fb35234d16824c55689b5fce28, State: Running, Role: LEADER
I20260709 12:26:59.866423 29792 ts_tablet_manager.cc:1434] T 16649fd17d434e3c95e9d585d2404659 P 584dac86183a41d9b3bea1c2085e62fa: Time spent starting tablet: real 0.129s	user 0.008s	sys 0.000s
I20260709 12:26:59.866976 30006 consensus_queue.cc:237] T 58154d49c1a344f2b51366ede3c87de5 P d73d92fb35234d16824c55689b5fce28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:26:59.867653 29792 tablet_bootstrap.cc:492] T c6e621aa77d5474580dfc5949801ccc9 P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap starting.
I20260709 12:26:59.875550 29989 raft_consensus.cc:493] T 66f652ba2c224885a2aea974bfe8f88a P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:59.876128 29989 raft_consensus.cc:515] T 66f652ba2c224885a2aea974bfe8f88a P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:26:59.878134 29989 leader_election.cc:290] T 66f652ba2c224885a2aea974bfe8f88a P 584dac86183a41d9b3bea1c2085e62fa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895), d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825)
I20260709 12:26:59.878268 29521 raft_consensus.cc:3060] T 66f652ba2c224885a2aea974bfe8f88a P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:59.856863 29852 consensus_queue.cc:1048] T 3615115233c34461b85763151751e475 P d73d92fb35234d16824c55689b5fce28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:26:59.882426 29744 raft_consensus.cc:2468] T 58154d49c1a344f2b51366ede3c87de5 P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d73d92fb35234d16824c55689b5fce28 in term 1.
I20260709 12:26:59.885669 29594 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66f652ba2c224885a2aea974bfe8f88a" candidate_uuid: "584dac86183a41d9b3bea1c2085e62fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d73d92fb35234d16824c55689b5fce28" is_pre_election: true
I20260709 12:26:59.886482 29594 raft_consensus.cc:2393] T 66f652ba2c224885a2aea974bfe8f88a P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 584dac86183a41d9b3bea1c2085e62fa in current term 1: Already voted for candidate c6610c00ffb948ce9c34f41dec3d49e8 in this term.
W20260709 12:26:59.889674 30023 log.cc:927] Time spent T 76140ad6258543118a9bfeee7cf0d5fc P c6610c00ffb948ce9c34f41dec3d49e8: Append to log took a long time: real 0.054s	user 0.001s	sys 0.000s
I20260709 12:26:59.892587 29793 raft_consensus.cc:359] T 959940284eb24fdf83d78438d710318a P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:26:59.893587 29793 raft_consensus.cc:385] T 959940284eb24fdf83d78438d710318a P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:59.894065 29793 raft_consensus.cc:740] T 959940284eb24fdf83d78438d710318a P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d73d92fb35234d16824c55689b5fce28, State: Initialized, Role: FOLLOWER
I20260709 12:26:59.894948 29793 consensus_queue.cc:260] T 959940284eb24fdf83d78438d710318a P d73d92fb35234d16824c55689b5fce28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:26:59.893604 29792 tablet_bootstrap.cc:654] T c6e621aa77d5474580dfc5949801ccc9 P 584dac86183a41d9b3bea1c2085e62fa: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:59.904156 29793 ts_tablet_manager.cc:1434] T 959940284eb24fdf83d78438d710318a P d73d92fb35234d16824c55689b5fce28: Time spent starting tablet: real 0.063s	user 0.009s	sys 0.000s
I20260709 12:26:59.900543 29851 raft_consensus.cc:493] T 07a2b94c0823467695508292a9ccdcc5 P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:59.907155 29851 raft_consensus.cc:515] T 07a2b94c0823467695508292a9ccdcc5 P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:26:59.907915 29790 tablet_bootstrap.cc:492] T 6a30685caefb4d0e80fa4ea87d70760c P c6610c00ffb948ce9c34f41dec3d49e8: No bootstrap required, opened a new log
I20260709 12:26:59.908457 29790 ts_tablet_manager.cc:1403] T 6a30685caefb4d0e80fa4ea87d70760c P c6610c00ffb948ce9c34f41dec3d49e8: Time spent bootstrapping tablet: real 0.319s	user 0.009s	sys 0.016s
I20260709 12:26:59.910001 29851 leader_election.cc:290] T 07a2b94c0823467695508292a9ccdcc5 P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209), c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895)
I20260709 12:26:59.911628 29743 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66f652ba2c224885a2aea974bfe8f88a" candidate_uuid: "584dac86183a41d9b3bea1c2085e62fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" is_pre_election: true
I20260709 12:26:59.917066 29743 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07a2b94c0823467695508292a9ccdcc5" candidate_uuid: "d73d92fb35234d16824c55689b5fce28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" is_pre_election: true
I20260709 12:26:59.918514 29669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07a2b94c0823467695508292a9ccdcc5" candidate_uuid: "d73d92fb35234d16824c55689b5fce28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" is_pre_election: true
I20260709 12:26:59.917199 29790 raft_consensus.cc:359] T 6a30685caefb4d0e80fa4ea87d70760c P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:26:59.924350 29790 raft_consensus.cc:385] T 6a30685caefb4d0e80fa4ea87d70760c P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:59.924867 29790 raft_consensus.cc:740] T 6a30685caefb4d0e80fa4ea87d70760c P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6610c00ffb948ce9c34f41dec3d49e8, State: Initialized, Role: FOLLOWER
I20260709 12:26:59.924274 29417 catalog_manager.cc:5697] T 8017a44219404faa8db8ff049c65205e P c6610c00ffb948ce9c34f41dec3d49e8 reported cstate change: term changed from 0 to 1, leader changed from <none> to c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68). New cstate: current_term: 1 leader_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:59.925221 29794 tablet_bootstrap.cc:492] T 07a2b94c0823467695508292a9ccdcc5 P d13c58ad8ac441c4a9b396eb04b1a476: No bootstrap required, opened a new log
I20260709 12:26:59.925962 29794 ts_tablet_manager.cc:1403] T 07a2b94c0823467695508292a9ccdcc5 P d13c58ad8ac441c4a9b396eb04b1a476: Time spent bootstrapping tablet: real 0.125s	user 0.021s	sys 0.021s
W20260709 12:26:59.923846 29561 leader_election.cc:343] T 07a2b94c0823467695508292a9ccdcc5 P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:59.925840 29790 consensus_queue.cc:260] T 6a30685caefb4d0e80fa4ea87d70760c P c6610c00ffb948ce9c34f41dec3d49e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:26:59.902532 29858 consensus_queue.cc:1048] T 66f652ba2c224885a2aea974bfe8f88a P c6610c00ffb948ce9c34f41dec3d49e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.027s
I20260709 12:26:59.938555 29793 tablet_bootstrap.cc:492] T 16649fd17d434e3c95e9d585d2404659 P d73d92fb35234d16824c55689b5fce28: Bootstrap starting.
W20260709 12:26:59.928457 29561 leader_election.cc:343] T 07a2b94c0823467695508292a9ccdcc5 P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:59.941640 29561 leader_election.cc:304] T 07a2b94c0823467695508292a9ccdcc5 P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d73d92fb35234d16824c55689b5fce28; no voters: c6610c00ffb948ce9c34f41dec3d49e8, d13c58ad8ac441c4a9b396eb04b1a476
I20260709 12:26:59.944063 29892 raft_consensus.cc:2749] T 07a2b94c0823467695508292a9ccdcc5 P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:26:59.945982 29794 raft_consensus.cc:359] T 07a2b94c0823467695508292a9ccdcc5 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:26:59.946748 29794 raft_consensus.cc:385] T 07a2b94c0823467695508292a9ccdcc5 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:59.947000 29794 raft_consensus.cc:740] T 07a2b94c0823467695508292a9ccdcc5 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d13c58ad8ac441c4a9b396eb04b1a476, State: Initialized, Role: FOLLOWER
I20260709 12:26:59.948843 29486 leader_election.cc:304] T 66f652ba2c224885a2aea974bfe8f88a P 584dac86183a41d9b3bea1c2085e62fa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 584dac86183a41d9b3bea1c2085e62fa; no voters: c6610c00ffb948ce9c34f41dec3d49e8, d73d92fb35234d16824c55689b5fce28
I20260709 12:26:59.952348 29790 ts_tablet_manager.cc:1434] T 6a30685caefb4d0e80fa4ea87d70760c P c6610c00ffb948ce9c34f41dec3d49e8: Time spent starting tablet: real 0.044s	user 0.008s	sys 0.000s
I20260709 12:26:59.947652 29794 consensus_queue.cc:260] T 07a2b94c0823467695508292a9ccdcc5 P d13c58ad8ac441c4a9b396eb04b1a476 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:26:59.954651 29790 tablet_bootstrap.cc:492] T 31a190e9f5ff407fab90502fb3197b70 P c6610c00ffb948ce9c34f41dec3d49e8: Bootstrap starting.
I20260709 12:26:59.971832 29794 ts_tablet_manager.cc:1434] T 07a2b94c0823467695508292a9ccdcc5 P d13c58ad8ac441c4a9b396eb04b1a476: Time spent starting tablet: real 0.045s	user 0.007s	sys 0.000s
I20260709 12:26:59.971097 29418 catalog_manager.cc:5697] T 58154d49c1a344f2b51366ede3c87de5 P d73d92fb35234d16824c55689b5fce28 reported cstate change: term changed from 0 to 1, leader changed from <none> to d73d92fb35234d16824c55689b5fce28 (127.16.29.66). New cstate: current_term: 1 leader_uuid: "d73d92fb35234d16824c55689b5fce28" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:59.972951 29794 tablet_bootstrap.cc:492] T 425777eed34441cd93e0902bdc126516 P d13c58ad8ac441c4a9b396eb04b1a476: Bootstrap starting.
I20260709 12:26:59.979198 29792 tablet_bootstrap.cc:492] T c6e621aa77d5474580dfc5949801ccc9 P 584dac86183a41d9b3bea1c2085e62fa: No bootstrap required, opened a new log
I20260709 12:26:59.979758 29792 ts_tablet_manager.cc:1403] T c6e621aa77d5474580dfc5949801ccc9 P 584dac86183a41d9b3bea1c2085e62fa: Time spent bootstrapping tablet: real 0.112s	user 0.019s	sys 0.007s
I20260709 12:26:59.985615 30006 raft_consensus.cc:493] T 8017a44219404faa8db8ff049c65205e P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:59.985921 29892 raft_consensus.cc:493] T 959940284eb24fdf83d78438d710318a P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:59.986488 29892 raft_consensus.cc:515] T 959940284eb24fdf83d78438d710318a P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:26:59.989635 29858 raft_consensus.cc:493] T 6a30685caefb4d0e80fa4ea87d70760c P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:59.990176 29858 raft_consensus.cc:515] T 6a30685caefb4d0e80fa4ea87d70760c P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:26:59.992425 29971 raft_consensus.cc:493] T d1918297b9eb41f5847a211e668ae748 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:59.992657 29743 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "959940284eb24fdf83d78438d710318a" candidate_uuid: "d73d92fb35234d16824c55689b5fce28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" is_pre_election: true
I20260709 12:26:59.992969 29971 raft_consensus.cc:515] T d1918297b9eb41f5847a211e668ae748 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:26:59.993861 29519 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6a30685caefb4d0e80fa4ea87d70760c" candidate_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "584dac86183a41d9b3bea1c2085e62fa" is_pre_election: true
I20260709 12:26:59.994195 29790 tablet_bootstrap.cc:654] T 31a190e9f5ff407fab90502fb3197b70 P c6610c00ffb948ce9c34f41dec3d49e8: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:59.995002 29971 leader_election.cc:290] T d1918297b9eb41f5847a211e668ae748 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033), d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825)
I20260709 12:26:59.995534 29858 leader_election.cc:290] T 6a30685caefb4d0e80fa4ea87d70760c P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033), d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825)
W20260709 12:26:59.997277 29708 leader_election.cc:343] T 6a30685caefb4d0e80fa4ea87d70760c P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:59.997527 29522 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1918297b9eb41f5847a211e668ae748" candidate_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "584dac86183a41d9b3bea1c2085e62fa" is_pre_election: true
W20260709 12:26:59.998780 29561 leader_election.cc:343] T 959940284eb24fdf83d78438d710318a P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:59.988826 29892 leader_election.cc:290] T 959940284eb24fdf83d78438d710318a P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209), c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895)
I20260709 12:26:59.999100 29594 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1918297b9eb41f5847a211e668ae748" candidate_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d73d92fb35234d16824c55689b5fce28" is_pre_election: true
I20260709 12:26:59.999861 29596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6a30685caefb4d0e80fa4ea87d70760c" candidate_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d73d92fb35234d16824c55689b5fce28" is_pre_election: true
I20260709 12:27:00.003296 29794 tablet_bootstrap.cc:654] T 425777eed34441cd93e0902bdc126516 P d13c58ad8ac441c4a9b396eb04b1a476: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:00.003526 29596 raft_consensus.cc:2468] T 6a30685caefb4d0e80fa4ea87d70760c P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6610c00ffb948ce9c34f41dec3d49e8 in term 0.
I20260709 12:27:00.000515 29988 raft_consensus.cc:2749] T 66f652ba2c224885a2aea974bfe8f88a P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:00.006196 29708 leader_election.cc:304] T 6a30685caefb4d0e80fa4ea87d70760c P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c6610c00ffb948ce9c34f41dec3d49e8, d73d92fb35234d16824c55689b5fce28; no voters: 584dac86183a41d9b3bea1c2085e62fa
I20260709 12:27:00.006971 29858 raft_consensus.cc:2804] T 6a30685caefb4d0e80fa4ea87d70760c P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:26:59.986425 30006 raft_consensus.cc:515] T 8017a44219404faa8db8ff049c65205e P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:00.008281 29793 tablet_bootstrap.cc:654] T 16649fd17d434e3c95e9d585d2404659 P d73d92fb35234d16824c55689b5fce28: Neither blocks nor log segments found. Creating new log.
W20260709 12:27:00.001761 29634 leader_election.cc:343] T d1918297b9eb41f5847a211e668ae748 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:00.008735 29858 raft_consensus.cc:493] T 6a30685caefb4d0e80fa4ea87d70760c P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:59.983829 29792 raft_consensus.cc:359] T c6e621aa77d5474580dfc5949801ccc9 P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:27:00.011708 29792 raft_consensus.cc:385] T c6e621aa77d5474580dfc5949801ccc9 P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:00.012058 29792 raft_consensus.cc:740] T c6e621aa77d5474580dfc5949801ccc9 P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 584dac86183a41d9b3bea1c2085e62fa, State: Initialized, Role: FOLLOWER
I20260709 12:27:00.012962 29792 consensus_queue.cc:260] T c6e621aa77d5474580dfc5949801ccc9 P 584dac86183a41d9b3bea1c2085e62fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:27:00.016157 30006 leader_election.cc:290] T 8017a44219404faa8db8ff049c65205e P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895), 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033)
I20260709 12:27:00.017035 29792 ts_tablet_manager.cc:1434] T c6e621aa77d5474580dfc5949801ccc9 P 584dac86183a41d9b3bea1c2085e62fa: Time spent starting tablet: real 0.037s	user 0.005s	sys 0.002s
I20260709 12:27:00.018877 29792 tablet_bootstrap.cc:492] T 1d1c1f58c4d340ac951dfa403d34138a P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap starting.
I20260709 12:27:00.011986 29858 raft_consensus.cc:3060] T 6a30685caefb4d0e80fa4ea87d70760c P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:00.029582 29520 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8017a44219404faa8db8ff049c65205e" candidate_uuid: "d73d92fb35234d16824c55689b5fce28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "584dac86183a41d9b3bea1c2085e62fa" is_pre_election: true
I20260709 12:27:00.010663 29670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "959940284eb24fdf83d78438d710318a" candidate_uuid: "d73d92fb35234d16824c55689b5fce28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" is_pre_election: true
W20260709 12:27:00.032223 29561 leader_election.cc:343] T 959940284eb24fdf83d78438d710318a P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:27:00.019037 29634 leader_election.cc:343] T d1918297b9eb41f5847a211e668ae748 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:00.032688 29561 leader_election.cc:304] T 959940284eb24fdf83d78438d710318a P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d73d92fb35234d16824c55689b5fce28; no voters: c6610c00ffb948ce9c34f41dec3d49e8, d13c58ad8ac441c4a9b396eb04b1a476
I20260709 12:27:00.032771 29634 leader_election.cc:304] T d1918297b9eb41f5847a211e668ae748 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d13c58ad8ac441c4a9b396eb04b1a476; no voters: 584dac86183a41d9b3bea1c2085e62fa, d73d92fb35234d16824c55689b5fce28
I20260709 12:27:00.035038 29962 consensus_queue.cc:1048] T 66f652ba2c224885a2aea974bfe8f88a P c6610c00ffb948ce9c34f41dec3d49e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:27:00.019577 29745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8017a44219404faa8db8ff049c65205e" candidate_uuid: "d73d92fb35234d16824c55689b5fce28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" is_pre_election: true
I20260709 12:27:00.030480 29520 raft_consensus.cc:2393] T 8017a44219404faa8db8ff049c65205e P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d73d92fb35234d16824c55689b5fce28 in current term 1: Already voted for candidate c6610c00ffb948ce9c34f41dec3d49e8 in this term.
I20260709 12:27:00.041415 29892 raft_consensus.cc:2749] T 959940284eb24fdf83d78438d710318a P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:00.048796 29798 raft_consensus.cc:493] T 07a2b94c0823467695508292a9ccdcc5 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:00.043730 29971 raft_consensus.cc:2749] T d1918297b9eb41f5847a211e668ae748 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:00.061157 29953 raft_consensus.cc:493] T c6e621aa77d5474580dfc5949801ccc9 P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:00.064527 29953 raft_consensus.cc:515] T c6e621aa77d5474580dfc5949801ccc9 P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:27:00.067564 29953 leader_election.cc:290] T c6e621aa77d5474580dfc5949801ccc9 P 584dac86183a41d9b3bea1c2085e62fa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825), d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209)
I20260709 12:27:00.052146 29798 raft_consensus.cc:515] T 07a2b94c0823467695508292a9ccdcc5 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:27:00.070204 29670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6e621aa77d5474580dfc5949801ccc9" candidate_uuid: "584dac86183a41d9b3bea1c2085e62fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" is_pre_election: true
I20260709 12:27:00.073019 29792 tablet_bootstrap.cc:654] T 1d1c1f58c4d340ac951dfa403d34138a P 584dac86183a41d9b3bea1c2085e62fa: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:00.074424 29593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6e621aa77d5474580dfc5949801ccc9" candidate_uuid: "584dac86183a41d9b3bea1c2085e62fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d73d92fb35234d16824c55689b5fce28" is_pre_election: true
I20260709 12:27:00.081571 29745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07a2b94c0823467695508292a9ccdcc5" candidate_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" is_pre_election: true
I20260709 12:27:00.075827 29560 leader_election.cc:304] T 8017a44219404faa8db8ff049c65205e P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d73d92fb35234d16824c55689b5fce28; no voters: 584dac86183a41d9b3bea1c2085e62fa, c6610c00ffb948ce9c34f41dec3d49e8
I20260709 12:27:00.076059 29962 consensus_queue.cc:1048] T 8017a44219404faa8db8ff049c65205e P c6610c00ffb948ce9c34f41dec3d49e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:27:00.083847 29892 raft_consensus.cc:3060] T 8017a44219404faa8db8ff049c65205e P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:00.081605 29670 raft_consensus.cc:2468] T c6e621aa77d5474580dfc5949801ccc9 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 584dac86183a41d9b3bea1c2085e62fa in term 0.
I20260709 12:27:00.084247 29593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07a2b94c0823467695508292a9ccdcc5" candidate_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d73d92fb35234d16824c55689b5fce28" is_pre_election: true
I20260709 12:27:00.079797 29858 raft_consensus.cc:515] T 6a30685caefb4d0e80fa4ea87d70760c P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:00.080359 29798 leader_election.cc:290] T 07a2b94c0823467695508292a9ccdcc5 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825), c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895)
I20260709 12:27:00.085143 29593 raft_consensus.cc:2468] T 07a2b94c0823467695508292a9ccdcc5 P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d13c58ad8ac441c4a9b396eb04b1a476 in term 0.
I20260709 12:27:00.088356 29858 leader_election.cc:290] T 6a30685caefb4d0e80fa4ea87d70760c P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 1 election: Requested vote from peers 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033), d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825)
W20260709 12:27:00.089432 29635 leader_election.cc:343] T 07a2b94c0823467695508292a9ccdcc5 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:00.092262 29634 leader_election.cc:304] T 07a2b94c0823467695508292a9ccdcc5 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d13c58ad8ac441c4a9b396eb04b1a476, d73d92fb35234d16824c55689b5fce28; no voters: c6610c00ffb948ce9c34f41dec3d49e8
I20260709 12:27:00.093585 29971 raft_consensus.cc:2804] T 07a2b94c0823467695508292a9ccdcc5 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:00.094095 29971 raft_consensus.cc:493] T 07a2b94c0823467695508292a9ccdcc5 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:00.094564 29971 raft_consensus.cc:3060] T 07a2b94c0823467695508292a9ccdcc5 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:00.095965 29593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6a30685caefb4d0e80fa4ea87d70760c" candidate_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d73d92fb35234d16824c55689b5fce28"
I20260709 12:27:00.096905 29593 raft_consensus.cc:3060] T 6a30685caefb4d0e80fa4ea87d70760c P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:00.107673 29790 tablet_bootstrap.cc:492] T 31a190e9f5ff407fab90502fb3197b70 P c6610c00ffb948ce9c34f41dec3d49e8: No bootstrap required, opened a new log
I20260709 12:27:00.108371 29790 ts_tablet_manager.cc:1403] T 31a190e9f5ff407fab90502fb3197b70 P c6610c00ffb948ce9c34f41dec3d49e8: Time spent bootstrapping tablet: real 0.154s	user 0.034s	sys 0.032s
I20260709 12:27:00.111709 29790 raft_consensus.cc:359] T 31a190e9f5ff407fab90502fb3197b70 P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:27:00.112525 29790 raft_consensus.cc:385] T 31a190e9f5ff407fab90502fb3197b70 P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:00.112870 29790 raft_consensus.cc:740] T 31a190e9f5ff407fab90502fb3197b70 P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6610c00ffb948ce9c34f41dec3d49e8, State: Initialized, Role: FOLLOWER
I20260709 12:27:00.113600 29790 consensus_queue.cc:260] T 31a190e9f5ff407fab90502fb3197b70 P c6610c00ffb948ce9c34f41dec3d49e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:27:00.117981 29793 tablet_bootstrap.cc:492] T 16649fd17d434e3c95e9d585d2404659 P d73d92fb35234d16824c55689b5fce28: No bootstrap required, opened a new log
I20260709 12:27:00.118517 29793 ts_tablet_manager.cc:1403] T 16649fd17d434e3c95e9d585d2404659 P d73d92fb35234d16824c55689b5fce28: Time spent bootstrapping tablet: real 0.181s	user 0.009s	sys 0.011s
I20260709 12:27:00.119976 29892 raft_consensus.cc:2749] T 8017a44219404faa8db8ff049c65205e P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:00.121379 29793 raft_consensus.cc:359] T 16649fd17d434e3c95e9d585d2404659 P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:27:00.122179 29793 raft_consensus.cc:385] T 16649fd17d434e3c95e9d585d2404659 P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:00.122520 29793 raft_consensus.cc:740] T 16649fd17d434e3c95e9d585d2404659 P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d73d92fb35234d16824c55689b5fce28, State: Initialized, Role: FOLLOWER
I20260709 12:27:00.124912 29793 consensus_queue.cc:260] T 16649fd17d434e3c95e9d585d2404659 P d73d92fb35234d16824c55689b5fce28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:27:00.125736 29790 ts_tablet_manager.cc:1434] T 31a190e9f5ff407fab90502fb3197b70 P c6610c00ffb948ce9c34f41dec3d49e8: Time spent starting tablet: real 0.017s	user 0.001s	sys 0.006s
I20260709 12:27:00.127316 29790 tablet_bootstrap.cc:492] T c1c8411289a74b5685700229d245ff75 P c6610c00ffb948ce9c34f41dec3d49e8: Bootstrap starting.
I20260709 12:27:00.130930 29971 raft_consensus.cc:515] T 07a2b94c0823467695508292a9ccdcc5 P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:27:00.131501 29794 tablet_bootstrap.cc:492] T 425777eed34441cd93e0902bdc126516 P d13c58ad8ac441c4a9b396eb04b1a476: No bootstrap required, opened a new log
I20260709 12:27:00.132206 29794 ts_tablet_manager.cc:1403] T 425777eed34441cd93e0902bdc126516 P d13c58ad8ac441c4a9b396eb04b1a476: Time spent bootstrapping tablet: real 0.159s	user 0.024s	sys 0.016s
I20260709 12:27:00.133368 29971 leader_election.cc:290] T 07a2b94c0823467695508292a9ccdcc5 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 1 election: Requested vote from peers d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825), c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895)
I20260709 12:27:00.134733 29793 ts_tablet_manager.cc:1434] T 16649fd17d434e3c95e9d585d2404659 P d73d92fb35234d16824c55689b5fce28: Time spent starting tablet: real 0.016s	user 0.007s	sys 0.002s
I20260709 12:27:00.134951 29794 raft_consensus.cc:359] T 425777eed34441cd93e0902bdc126516 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:27:00.135548 29794 raft_consensus.cc:385] T 425777eed34441cd93e0902bdc126516 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:00.135617 29793 tablet_bootstrap.cc:492] T c1c8411289a74b5685700229d245ff75 P d73d92fb35234d16824c55689b5fce28: Bootstrap starting.
I20260709 12:27:00.187170 29522 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6a30685caefb4d0e80fa4ea87d70760c" candidate_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "584dac86183a41d9b3bea1c2085e62fa"
I20260709 12:27:00.193428 29596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07a2b94c0823467695508292a9ccdcc5" candidate_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d73d92fb35234d16824c55689b5fce28"
W20260709 12:27:00.193106 29485 leader_election.cc:343] T c6e621aa77d5474580dfc5949801ccc9 P 584dac86183a41d9b3bea1c2085e62fa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:00.194471 29596 raft_consensus.cc:3060] T 07a2b94c0823467695508292a9ccdcc5 P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:00.194705 29593 raft_consensus.cc:2468] T 6a30685caefb4d0e80fa4ea87d70760c P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6610c00ffb948ce9c34f41dec3d49e8 in term 1.
W20260709 12:27:00.202521 29708 leader_election.cc:343] T 6a30685caefb4d0e80fa4ea87d70760c P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:00.203384 29792 tablet_bootstrap.cc:492] T 1d1c1f58c4d340ac951dfa403d34138a P 584dac86183a41d9b3bea1c2085e62fa: No bootstrap required, opened a new log
I20260709 12:27:00.204140 29790 tablet_bootstrap.cc:654] T c1c8411289a74b5685700229d245ff75 P c6610c00ffb948ce9c34f41dec3d49e8: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:00.224043 29953 raft_consensus.cc:493] T 16649fd17d434e3c95e9d585d2404659 P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:00.135838 29794 raft_consensus.cc:740] T 425777eed34441cd93e0902bdc126516 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d13c58ad8ac441c4a9b396eb04b1a476, State: Initialized, Role: FOLLOWER
I20260709 12:27:00.224139 29988 consensus_queue.cc:1048] T 3f41ce441fb047c8bbb741b1b9cbc0ec P 584dac86183a41d9b3bea1c2085e62fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260709 12:27:00.225055 29953 raft_consensus.cc:515] T 16649fd17d434e3c95e9d585d2404659 P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:27:00.243665 29792 ts_tablet_manager.cc:1403] T 1d1c1f58c4d340ac951dfa403d34138a P 584dac86183a41d9b3bea1c2085e62fa: Time spent bootstrapping tablet: real 0.226s	user 0.017s	sys 0.009s
I20260709 12:27:00.236194 29794 consensus_queue.cc:260] T 425777eed34441cd93e0902bdc126516 P d13c58ad8ac441c4a9b396eb04b1a476 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:27:00.264947 29953 leader_election.cc:290] T 16649fd17d434e3c95e9d585d2404659 P 584dac86183a41d9b3bea1c2085e62fa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825), d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209)
I20260709 12:27:00.275271 29486 leader_election.cc:304] T c6e621aa77d5474580dfc5949801ccc9 P 584dac86183a41d9b3bea1c2085e62fa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 584dac86183a41d9b3bea1c2085e62fa, d13c58ad8ac441c4a9b396eb04b1a476; no voters: d73d92fb35234d16824c55689b5fce28
I20260709 12:27:00.276239 29745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07a2b94c0823467695508292a9ccdcc5" candidate_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6610c00ffb948ce9c34f41dec3d49e8"
I20260709 12:27:00.277591 29858 consensus_queue.cc:1048] T 8017a44219404faa8db8ff049c65205e P c6610c00ffb948ce9c34f41dec3d49e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:00.282981 29596 raft_consensus.cc:2468] T 07a2b94c0823467695508292a9ccdcc5 P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d13c58ad8ac441c4a9b396eb04b1a476 in term 1.
I20260709 12:27:00.287528 29792 raft_consensus.cc:359] T 1d1c1f58c4d340ac951dfa403d34138a P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:00.343205 29792 raft_consensus.cc:385] T 1d1c1f58c4d340ac951dfa403d34138a P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:00.343570 29792 raft_consensus.cc:740] T 1d1c1f58c4d340ac951dfa403d34138a P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 584dac86183a41d9b3bea1c2085e62fa, State: Initialized, Role: FOLLOWER
I20260709 12:27:00.344376 29792 consensus_queue.cc:260] T 1d1c1f58c4d340ac951dfa403d34138a P 584dac86183a41d9b3bea1c2085e62fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:00.344617 29988 raft_consensus.cc:2804] T c6e621aa77d5474580dfc5949801ccc9 P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:00.345363 29988 raft_consensus.cc:493] T c6e621aa77d5474580dfc5949801ccc9 P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:00.345909 29988 raft_consensus.cc:3060] T c6e621aa77d5474580dfc5949801ccc9 P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:00.346318 29792 ts_tablet_manager.cc:1434] T 1d1c1f58c4d340ac951dfa403d34138a P 584dac86183a41d9b3bea1c2085e62fa: Time spent starting tablet: real 0.102s	user 0.007s	sys 0.001s
I20260709 12:27:00.335649 29597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "16649fd17d434e3c95e9d585d2404659" candidate_uuid: "584dac86183a41d9b3bea1c2085e62fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d73d92fb35234d16824c55689b5fce28" is_pre_election: true
I20260709 12:27:00.337046 29708 leader_election.cc:304] T 6a30685caefb4d0e80fa4ea87d70760c P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c6610c00ffb948ce9c34f41dec3d49e8, d73d92fb35234d16824c55689b5fce28; no voters: 584dac86183a41d9b3bea1c2085e62fa
I20260709 12:27:00.350896 29792 tablet_bootstrap.cc:492] T 6a30685caefb4d0e80fa4ea87d70760c P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap starting.
W20260709 12:27:00.352741 29635 leader_election.cc:343] T 07a2b94c0823467695508292a9ccdcc5 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:00.331904 29794 ts_tablet_manager.cc:1434] T 425777eed34441cd93e0902bdc126516 P d13c58ad8ac441c4a9b396eb04b1a476: Time spent starting tablet: real 0.199s	user 0.017s	sys 0.018s
I20260709 12:27:00.348282 29597 raft_consensus.cc:2468] T 16649fd17d434e3c95e9d585d2404659 P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 584dac86183a41d9b3bea1c2085e62fa in term 0.
I20260709 12:27:00.358654 29634 leader_election.cc:304] T 07a2b94c0823467695508292a9ccdcc5 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d13c58ad8ac441c4a9b396eb04b1a476, d73d92fb35234d16824c55689b5fce28; no voters: c6610c00ffb948ce9c34f41dec3d49e8
I20260709 12:27:00.361920 29858 raft_consensus.cc:2804] T 6a30685caefb4d0e80fa4ea87d70760c P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:00.364027 30041 raft_consensus.cc:2804] T 07a2b94c0823467695508292a9ccdcc5 P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:00.364349 29858 raft_consensus.cc:697] T 6a30685caefb4d0e80fa4ea87d70760c P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 LEADER]: Becoming Leader. State: Replica: c6610c00ffb948ce9c34f41dec3d49e8, State: Running, Role: LEADER
I20260709 12:27:00.364691 30041 raft_consensus.cc:697] T 07a2b94c0823467695508292a9ccdcc5 P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 LEADER]: Becoming Leader. State: Replica: d13c58ad8ac441c4a9b396eb04b1a476, State: Running, Role: LEADER
I20260709 12:27:00.366322 29858 consensus_queue.cc:237] T 6a30685caefb4d0e80fa4ea87d70760c P c6610c00ffb948ce9c34f41dec3d49e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:00.366314 30041 consensus_queue.cc:237] T 07a2b94c0823467695508292a9ccdcc5 P d13c58ad8ac441c4a9b396eb04b1a476 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
W20260709 12:27:00.373518 29486 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:38209, user_credentials={real_user=slave}} blocked reactor thread for 194612us
I20260709 12:27:00.399710 29794 tablet_bootstrap.cc:492] T 31a190e9f5ff407fab90502fb3197b70 P d13c58ad8ac441c4a9b396eb04b1a476: Bootstrap starting.
I20260709 12:27:00.418141 29792 tablet_bootstrap.cc:654] T 6a30685caefb4d0e80fa4ea87d70760c P 584dac86183a41d9b3bea1c2085e62fa: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:00.436676 29988 raft_consensus.cc:515] T c6e621aa77d5474580dfc5949801ccc9 P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:27:00.439468 29988 leader_election.cc:290] T c6e621aa77d5474580dfc5949801ccc9 P 584dac86183a41d9b3bea1c2085e62fa [CANDIDATE]: Term 1 election: Requested vote from peers d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825), d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209)
I20260709 12:27:00.462981 29485 leader_election.cc:304] T 16649fd17d434e3c95e9d585d2404659 P 584dac86183a41d9b3bea1c2085e62fa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 584dac86183a41d9b3bea1c2085e62fa, d73d92fb35234d16824c55689b5fce28; no voters: 
I20260709 12:27:00.464624 29954 raft_consensus.cc:2804] T 16649fd17d434e3c95e9d585d2404659 P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:00.465219 29954 raft_consensus.cc:493] T 16649fd17d434e3c95e9d585d2404659 P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:00.465677 29954 raft_consensus.cc:3060] T 16649fd17d434e3c95e9d585d2404659 P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:00.466152 29790 tablet_bootstrap.cc:492] T c1c8411289a74b5685700229d245ff75 P c6610c00ffb948ce9c34f41dec3d49e8: No bootstrap required, opened a new log
I20260709 12:27:00.466995 29790 ts_tablet_manager.cc:1403] T c1c8411289a74b5685700229d245ff75 P c6610c00ffb948ce9c34f41dec3d49e8: Time spent bootstrapping tablet: real 0.340s	user 0.027s	sys 0.015s
I20260709 12:27:00.489589 29593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6e621aa77d5474580dfc5949801ccc9" candidate_uuid: "584dac86183a41d9b3bea1c2085e62fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d73d92fb35234d16824c55689b5fce28"
I20260709 12:27:00.491102 29793 tablet_bootstrap.cc:654] T c1c8411289a74b5685700229d245ff75 P d73d92fb35234d16824c55689b5fce28: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:00.495894 29794 tablet_bootstrap.cc:654] T 31a190e9f5ff407fab90502fb3197b70 P d13c58ad8ac441c4a9b396eb04b1a476: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:00.500233 29851 consensus_queue.cc:1048] T 58154d49c1a344f2b51366ede3c87de5 P d73d92fb35234d16824c55689b5fce28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:00.503315 29851 consensus_queue.cc:1048] T 58154d49c1a344f2b51366ede3c87de5 P d73d92fb35234d16824c55689b5fce28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260709 12:27:00.502410 29485 leader_election.cc:343] T c6e621aa77d5474580dfc5949801ccc9 P 584dac86183a41d9b3bea1c2085e62fa [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:00.495249 29419 catalog_manager.cc:5697] T 6a30685caefb4d0e80fa4ea87d70760c P c6610c00ffb948ce9c34f41dec3d49e8 reported cstate change: term changed from 0 to 1, leader changed from <none> to c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68). New cstate: current_term: 1 leader_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:00.531165 29793 tablet_bootstrap.cc:492] T c1c8411289a74b5685700229d245ff75 P d73d92fb35234d16824c55689b5fce28: No bootstrap required, opened a new log
I20260709 12:27:00.531967 29793 ts_tablet_manager.cc:1403] T c1c8411289a74b5685700229d245ff75 P d73d92fb35234d16824c55689b5fce28: Time spent bootstrapping tablet: real 0.397s	user 0.019s	sys 0.004s
I20260709 12:27:00.532559 29794 tablet_bootstrap.cc:492] T 31a190e9f5ff407fab90502fb3197b70 P d13c58ad8ac441c4a9b396eb04b1a476: No bootstrap required, opened a new log
I20260709 12:27:00.533289 29794 ts_tablet_manager.cc:1403] T 31a190e9f5ff407fab90502fb3197b70 P d13c58ad8ac441c4a9b396eb04b1a476: Time spent bootstrapping tablet: real 0.134s	user 0.018s	sys 0.000s
I20260709 12:27:00.537245 29667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "16649fd17d434e3c95e9d585d2404659" candidate_uuid: "584dac86183a41d9b3bea1c2085e62fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" is_pre_election: true
W20260709 12:27:00.540380 29467 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:27:00.540107 29790 raft_consensus.cc:359] T c1c8411289a74b5685700229d245ff75 P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:27:00.540942 29790 raft_consensus.cc:385] T c1c8411289a74b5685700229d245ff75 P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:00.541217 29790 raft_consensus.cc:740] T c1c8411289a74b5685700229d245ff75 P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6610c00ffb948ce9c34f41dec3d49e8, State: Initialized, Role: FOLLOWER
I20260709 12:27:00.542047 29790 consensus_queue.cc:260] T c1c8411289a74b5685700229d245ff75 P c6610c00ffb948ce9c34f41dec3d49e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:27:00.546741 29790 ts_tablet_manager.cc:1434] T c1c8411289a74b5685700229d245ff75 P c6610c00ffb948ce9c34f41dec3d49e8: Time spent starting tablet: real 0.079s	user 0.008s	sys 0.000s
I20260709 12:27:00.548115 29790 tablet_bootstrap.cc:492] T e67a2affef634fd8bc96a64fa83cffdb P c6610c00ffb948ce9c34f41dec3d49e8: Bootstrap starting.
I20260709 12:27:00.550508 29954 raft_consensus.cc:515] T 16649fd17d434e3c95e9d585d2404659 P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:27:00.561976 29792 tablet_bootstrap.cc:492] T 6a30685caefb4d0e80fa4ea87d70760c P 584dac86183a41d9b3bea1c2085e62fa: No bootstrap required, opened a new log
I20260709 12:27:00.562714 29792 ts_tablet_manager.cc:1403] T 6a30685caefb4d0e80fa4ea87d70760c P 584dac86183a41d9b3bea1c2085e62fa: Time spent bootstrapping tablet: real 0.215s	user 0.027s	sys 0.019s
I20260709 12:27:00.566955 30038 consensus_queue.cc:1048] T 3f41ce441fb047c8bbb741b1b9cbc0ec P 584dac86183a41d9b3bea1c2085e62fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:00.570060 29860 raft_consensus.cc:493] T 31a190e9f5ff407fab90502fb3197b70 P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:00.570786 29860 raft_consensus.cc:515] T 31a190e9f5ff407fab90502fb3197b70 P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:27:00.579198 29670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31a190e9f5ff407fab90502fb3197b70" candidate_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" is_pre_election: true
I20260709 12:27:00.581727 30041 raft_consensus.cc:493] T 425777eed34441cd93e0902bdc126516 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:27:00.582585 29709 leader_election.cc:343] T 31a190e9f5ff407fab90502fb3197b70 P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:00.582438 30041 raft_consensus.cc:515] T 425777eed34441cd93e0902bdc126516 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:27:00.587134 30041 leader_election.cc:290] T 425777eed34441cd93e0902bdc126516 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895), 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033)
I20260709 12:27:00.588940 29860 leader_election.cc:290] T 31a190e9f5ff407fab90502fb3197b70 P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209), 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033)
I20260709 12:27:00.589476 29743 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "425777eed34441cd93e0902bdc126516" candidate_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" is_pre_election: true
I20260709 12:27:00.590153 29597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "16649fd17d434e3c95e9d585d2404659" candidate_uuid: "584dac86183a41d9b3bea1c2085e62fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d73d92fb35234d16824c55689b5fce28"
I20260709 12:27:00.596141 29597 raft_consensus.cc:3060] T 16649fd17d434e3c95e9d585d2404659 P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:00.593174 29521 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31a190e9f5ff407fab90502fb3197b70" candidate_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "584dac86183a41d9b3bea1c2085e62fa" is_pre_election: true
I20260709 12:27:00.568282 29954 leader_election.cc:290] T 16649fd17d434e3c95e9d585d2404659 P 584dac86183a41d9b3bea1c2085e62fa [CANDIDATE]: Term 1 election: Requested vote from peers d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825), d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209)
I20260709 12:27:00.592087 29792 raft_consensus.cc:359] T 6a30685caefb4d0e80fa4ea87d70760c P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:00.598500 29792 raft_consensus.cc:385] T 6a30685caefb4d0e80fa4ea87d70760c P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:00.598856 29792 raft_consensus.cc:740] T 6a30685caefb4d0e80fa4ea87d70760c P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 584dac86183a41d9b3bea1c2085e62fa, State: Initialized, Role: FOLLOWER
I20260709 12:27:00.579852 29793 raft_consensus.cc:359] T c1c8411289a74b5685700229d245ff75 P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:27:00.600778 29793 raft_consensus.cc:385] T c1c8411289a74b5685700229d245ff75 P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:00.599701 29792 consensus_queue.cc:260] T 6a30685caefb4d0e80fa4ea87d70760c P 584dac86183a41d9b3bea1c2085e62fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:00.601219 29793 raft_consensus.cc:740] T c1c8411289a74b5685700229d245ff75 P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d73d92fb35234d16824c55689b5fce28, State: Initialized, Role: FOLLOWER
I20260709 12:27:00.602195 29793 consensus_queue.cc:260] T c1c8411289a74b5685700229d245ff75 P d73d92fb35234d16824c55689b5fce28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:27:00.605077 29793 ts_tablet_manager.cc:1434] T c1c8411289a74b5685700229d245ff75 P d73d92fb35234d16824c55689b5fce28: Time spent starting tablet: real 0.073s	user 0.008s	sys 0.001s
I20260709 12:27:00.605310 29794 raft_consensus.cc:359] T 31a190e9f5ff407fab90502fb3197b70 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:27:00.606215 29794 raft_consensus.cc:385] T 31a190e9f5ff407fab90502fb3197b70 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:00.606379 29793 tablet_bootstrap.cc:492] T d1918297b9eb41f5847a211e668ae748 P d73d92fb35234d16824c55689b5fce28: Bootstrap starting.
I20260709 12:27:00.606532 29794 raft_consensus.cc:740] T 31a190e9f5ff407fab90502fb3197b70 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d13c58ad8ac441c4a9b396eb04b1a476, State: Initialized, Role: FOLLOWER
I20260709 12:27:00.603742 29790 tablet_bootstrap.cc:654] T e67a2affef634fd8bc96a64fa83cffdb P c6610c00ffb948ce9c34f41dec3d49e8: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:00.607337 29794 consensus_queue.cc:260] T 31a190e9f5ff407fab90502fb3197b70 P d13c58ad8ac441c4a9b396eb04b1a476 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:27:00.608794 29670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6e621aa77d5474580dfc5949801ccc9" candidate_uuid: "584dac86183a41d9b3bea1c2085e62fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d13c58ad8ac441c4a9b396eb04b1a476"
W20260709 12:27:00.610497 29708 leader_election.cc:343] T 31a190e9f5ff407fab90502fb3197b70 P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:00.611065 29708 leader_election.cc:304] T 31a190e9f5ff407fab90502fb3197b70 P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c6610c00ffb948ce9c34f41dec3d49e8; no voters: 584dac86183a41d9b3bea1c2085e62fa, d13c58ad8ac441c4a9b396eb04b1a476
W20260709 12:27:00.613111 29635 leader_election.cc:343] T 425777eed34441cd93e0902bdc126516 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:00.613716 29858 raft_consensus.cc:2749] T 31a190e9f5ff407fab90502fb3197b70 P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:00.615795 29794 ts_tablet_manager.cc:1434] T 31a190e9f5ff407fab90502fb3197b70 P d13c58ad8ac441c4a9b396eb04b1a476: Time spent starting tablet: real 0.082s	user 0.006s	sys 0.002s
I20260709 12:27:00.616961 29794 tablet_bootstrap.cc:492] T 959940284eb24fdf83d78438d710318a P d13c58ad8ac441c4a9b396eb04b1a476: Bootstrap starting.
I20260709 12:27:00.610982 29670 raft_consensus.cc:3060] T c6e621aa77d5474580dfc5949801ccc9 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:00.629374 29521 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "425777eed34441cd93e0902bdc126516" candidate_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "584dac86183a41d9b3bea1c2085e62fa" is_pre_election: true
I20260709 12:27:00.636035 29671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "16649fd17d434e3c95e9d585d2404659" candidate_uuid: "584dac86183a41d9b3bea1c2085e62fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d13c58ad8ac441c4a9b396eb04b1a476"
W20260709 12:27:00.637974 29634 leader_election.cc:343] T 425777eed34441cd93e0902bdc126516 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:00.638578 29634 leader_election.cc:304] T 425777eed34441cd93e0902bdc126516 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d13c58ad8ac441c4a9b396eb04b1a476; no voters: 584dac86183a41d9b3bea1c2085e62fa, c6610c00ffb948ce9c34f41dec3d49e8
I20260709 12:27:00.635062 29420 catalog_manager.cc:5697] T 07a2b94c0823467695508292a9ccdcc5 P d13c58ad8ac441c4a9b396eb04b1a476 reported cstate change: term changed from 0 to 1, leader changed from <none> to d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67). New cstate: current_term: 1 leader_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:00.642758 29792 ts_tablet_manager.cc:1434] T 6a30685caefb4d0e80fa4ea87d70760c P 584dac86183a41d9b3bea1c2085e62fa: Time spent starting tablet: real 0.078s	user 0.012s	sys 0.000s
I20260709 12:27:00.649077 30043 raft_consensus.cc:2749] T 425777eed34441cd93e0902bdc126516 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260709 12:27:00.652801 29486 leader_election.cc:343] T 16649fd17d434e3c95e9d585d2404659 P 584dac86183a41d9b3bea1c2085e62fa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:00.654371 29597 raft_consensus.cc:2468] T 16649fd17d434e3c95e9d585d2404659 P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 584dac86183a41d9b3bea1c2085e62fa in term 1.
W20260709 12:27:00.655284 29486 leader_election.cc:343] T 16649fd17d434e3c95e9d585d2404659 P 584dac86183a41d9b3bea1c2085e62fa [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:00.659775 29485 leader_election.cc:304] T 16649fd17d434e3c95e9d585d2404659 P 584dac86183a41d9b3bea1c2085e62fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 584dac86183a41d9b3bea1c2085e62fa, d73d92fb35234d16824c55689b5fce28; no voters: d13c58ad8ac441c4a9b396eb04b1a476
I20260709 12:27:00.661541 29954 raft_consensus.cc:2804] T 16649fd17d434e3c95e9d585d2404659 P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:00.662616 29954 raft_consensus.cc:697] T 16649fd17d434e3c95e9d585d2404659 P 584dac86183a41d9b3bea1c2085e62fa [term 1 LEADER]: Becoming Leader. State: Replica: 584dac86183a41d9b3bea1c2085e62fa, State: Running, Role: LEADER
I20260709 12:27:00.664032 29954 consensus_queue.cc:237] T 16649fd17d434e3c95e9d585d2404659 P 584dac86183a41d9b3bea1c2085e62fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:27:00.672516 29794 tablet_bootstrap.cc:654] T 959940284eb24fdf83d78438d710318a P d13c58ad8ac441c4a9b396eb04b1a476: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:00.682210 29792 tablet_bootstrap.cc:492] T d1918297b9eb41f5847a211e668ae748 P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap starting.
I20260709 12:27:00.696025 29670 raft_consensus.cc:2468] T c6e621aa77d5474580dfc5949801ccc9 P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 584dac86183a41d9b3bea1c2085e62fa in term 1.
I20260709 12:27:00.694597 29793 tablet_bootstrap.cc:654] T d1918297b9eb41f5847a211e668ae748 P d73d92fb35234d16824c55689b5fce28: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:00.704795 29790 tablet_bootstrap.cc:492] T e67a2affef634fd8bc96a64fa83cffdb P c6610c00ffb948ce9c34f41dec3d49e8: No bootstrap required, opened a new log
I20260709 12:27:00.705610 29790 ts_tablet_manager.cc:1403] T e67a2affef634fd8bc96a64fa83cffdb P c6610c00ffb948ce9c34f41dec3d49e8: Time spent bootstrapping tablet: real 0.158s	user 0.012s	sys 0.003s
I20260709 12:27:00.717334 29790 raft_consensus.cc:359] T e67a2affef634fd8bc96a64fa83cffdb P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:27:00.718071 29790 raft_consensus.cc:385] T e67a2affef634fd8bc96a64fa83cffdb P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:00.718786 29790 raft_consensus.cc:740] T e67a2affef634fd8bc96a64fa83cffdb P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6610c00ffb948ce9c34f41dec3d49e8, State: Initialized, Role: FOLLOWER
I20260709 12:27:00.722666 29790 consensus_queue.cc:260] T e67a2affef634fd8bc96a64fa83cffdb P c6610c00ffb948ce9c34f41dec3d49e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:27:00.730031 29790 ts_tablet_manager.cc:1434] T e67a2affef634fd8bc96a64fa83cffdb P c6610c00ffb948ce9c34f41dec3d49e8: Time spent starting tablet: real 0.024s	user 0.006s	sys 0.001s
I20260709 12:27:00.731510 29790 tablet_bootstrap.cc:492] T 5be58d64d45e40d7981018b84a01b1ed P c6610c00ffb948ce9c34f41dec3d49e8: Bootstrap starting.
I20260709 12:27:00.734020 29988 raft_consensus.cc:493] T 6a30685caefb4d0e80fa4ea87d70760c P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:00.734580 29988 raft_consensus.cc:515] T 6a30685caefb4d0e80fa4ea87d70760c P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:00.738123 29988 leader_election.cc:290] T 6a30685caefb4d0e80fa4ea87d70760c P 584dac86183a41d9b3bea1c2085e62fa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895), d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825)
I20260709 12:27:00.740285 29486 leader_election.cc:304] T c6e621aa77d5474580dfc5949801ccc9 P 584dac86183a41d9b3bea1c2085e62fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 584dac86183a41d9b3bea1c2085e62fa, d13c58ad8ac441c4a9b396eb04b1a476; no voters: d73d92fb35234d16824c55689b5fce28
I20260709 12:27:00.741138 30038 raft_consensus.cc:2804] T c6e621aa77d5474580dfc5949801ccc9 P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:00.742651 29597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6a30685caefb4d0e80fa4ea87d70760c" candidate_uuid: "584dac86183a41d9b3bea1c2085e62fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d73d92fb35234d16824c55689b5fce28" is_pre_election: true
I20260709 12:27:00.744308 30038 raft_consensus.cc:697] T c6e621aa77d5474580dfc5949801ccc9 P 584dac86183a41d9b3bea1c2085e62fa [term 1 LEADER]: Becoming Leader. State: Replica: 584dac86183a41d9b3bea1c2085e62fa, State: Running, Role: LEADER
I20260709 12:27:00.745410 29597 raft_consensus.cc:2393] T 6a30685caefb4d0e80fa4ea87d70760c P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 584dac86183a41d9b3bea1c2085e62fa in current term 1: Already voted for candidate c6610c00ffb948ce9c34f41dec3d49e8 in this term.
I20260709 12:27:00.745810 30038 consensus_queue.cc:237] T c6e621aa77d5474580dfc5949801ccc9 P 584dac86183a41d9b3bea1c2085e62fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:27:00.764964 29745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6a30685caefb4d0e80fa4ea87d70760c" candidate_uuid: "584dac86183a41d9b3bea1c2085e62fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" is_pre_election: true
I20260709 12:27:00.766901 29953 raft_consensus.cc:493] T 1d1c1f58c4d340ac951dfa403d34138a P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:00.767452 29953 raft_consensus.cc:515] T 1d1c1f58c4d340ac951dfa403d34138a P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:00.769285 29792 tablet_bootstrap.cc:654] T d1918297b9eb41f5847a211e668ae748 P 584dac86183a41d9b3bea1c2085e62fa: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:00.769882 29953 leader_election.cc:290] T 1d1c1f58c4d340ac951dfa403d34138a P 584dac86183a41d9b3bea1c2085e62fa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895), d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825)
I20260709 12:27:00.772718 29486 leader_election.cc:304] T 6a30685caefb4d0e80fa4ea87d70760c P 584dac86183a41d9b3bea1c2085e62fa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 584dac86183a41d9b3bea1c2085e62fa; no voters: c6610c00ffb948ce9c34f41dec3d49e8, d73d92fb35234d16824c55689b5fce28
I20260709 12:27:00.775312 29745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d1c1f58c4d340ac951dfa403d34138a" candidate_uuid: "584dac86183a41d9b3bea1c2085e62fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" is_pre_election: true
I20260709 12:27:00.776962 29794 tablet_bootstrap.cc:492] T 959940284eb24fdf83d78438d710318a P d13c58ad8ac441c4a9b396eb04b1a476: No bootstrap required, opened a new log
I20260709 12:27:00.782763 29596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d1c1f58c4d340ac951dfa403d34138a" candidate_uuid: "584dac86183a41d9b3bea1c2085e62fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d73d92fb35234d16824c55689b5fce28" is_pre_election: true
I20260709 12:27:00.790113 29793 tablet_bootstrap.cc:492] T d1918297b9eb41f5847a211e668ae748 P d73d92fb35234d16824c55689b5fce28: No bootstrap required, opened a new log
I20260709 12:27:00.792485 29793 ts_tablet_manager.cc:1403] T d1918297b9eb41f5847a211e668ae748 P d73d92fb35234d16824c55689b5fce28: Time spent bootstrapping tablet: real 0.186s	user 0.020s	sys 0.011s
I20260709 12:27:00.793313 29794 ts_tablet_manager.cc:1403] T 959940284eb24fdf83d78438d710318a P d13c58ad8ac441c4a9b396eb04b1a476: Time spent bootstrapping tablet: real 0.177s	user 0.011s	sys 0.042s
W20260709 12:27:00.786633 29485 leader_election.cc:343] T 1d1c1f58c4d340ac951dfa403d34138a P 584dac86183a41d9b3bea1c2085e62fa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:00.783921 29953 raft_consensus.cc:3060] T 6a30685caefb4d0e80fa4ea87d70760c P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:00.802582 29790 tablet_bootstrap.cc:654] T 5be58d64d45e40d7981018b84a01b1ed P c6610c00ffb948ce9c34f41dec3d49e8: Neither blocks nor log segments found. Creating new log.
W20260709 12:27:00.815412 29486 leader_election.cc:343] T 1d1c1f58c4d340ac951dfa403d34138a P 584dac86183a41d9b3bea1c2085e62fa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:00.815938 29486 leader_election.cc:304] T 1d1c1f58c4d340ac951dfa403d34138a P 584dac86183a41d9b3bea1c2085e62fa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 584dac86183a41d9b3bea1c2085e62fa; no voters: c6610c00ffb948ce9c34f41dec3d49e8, d73d92fb35234d16824c55689b5fce28
W20260709 12:27:00.814555 29635 consensus_peers.cc:597] T 07a2b94c0823467695508292a9ccdcc5 P d13c58ad8ac441c4a9b396eb04b1a476 -> Peer c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895): Couldn't send request to peer c6610c00ffb948ce9c34f41dec3d49e8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:00.817713 30040 raft_consensus.cc:2749] T 1d1c1f58c4d340ac951dfa403d34138a P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:00.822005 29792 tablet_bootstrap.cc:492] T d1918297b9eb41f5847a211e668ae748 P 584dac86183a41d9b3bea1c2085e62fa: No bootstrap required, opened a new log
I20260709 12:27:00.822587 29792 ts_tablet_manager.cc:1403] T d1918297b9eb41f5847a211e668ae748 P 584dac86183a41d9b3bea1c2085e62fa: Time spent bootstrapping tablet: real 0.141s	user 0.013s	sys 0.001s
I20260709 12:27:00.825635 29792 raft_consensus.cc:359] T d1918297b9eb41f5847a211e668ae748 P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:27:00.814402 29794 raft_consensus.cc:359] T 959940284eb24fdf83d78438d710318a P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:27:00.826347 29792 raft_consensus.cc:385] T d1918297b9eb41f5847a211e668ae748 P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:00.826674 29794 raft_consensus.cc:385] T 959940284eb24fdf83d78438d710318a P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:00.827107 29794 raft_consensus.cc:740] T 959940284eb24fdf83d78438d710318a P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d13c58ad8ac441c4a9b396eb04b1a476, State: Initialized, Role: FOLLOWER
I20260709 12:27:00.826659 29792 raft_consensus.cc:740] T d1918297b9eb41f5847a211e668ae748 P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 584dac86183a41d9b3bea1c2085e62fa, State: Initialized, Role: FOLLOWER
I20260709 12:27:00.832472 29793 raft_consensus.cc:359] T d1918297b9eb41f5847a211e668ae748 P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:27:00.833320 29793 raft_consensus.cc:385] T d1918297b9eb41f5847a211e668ae748 P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:00.833617 29793 raft_consensus.cc:740] T d1918297b9eb41f5847a211e668ae748 P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d73d92fb35234d16824c55689b5fce28, State: Initialized, Role: FOLLOWER
I20260709 12:27:00.834451 29793 consensus_queue.cc:260] T d1918297b9eb41f5847a211e668ae748 P d73d92fb35234d16824c55689b5fce28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:27:00.835265 29794 consensus_queue.cc:260] T 959940284eb24fdf83d78438d710318a P d13c58ad8ac441c4a9b396eb04b1a476 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:27:00.843609 29860 raft_consensus.cc:493] T c1c8411289a74b5685700229d245ff75 P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:00.852792 30041 raft_consensus.cc:493] T 959940284eb24fdf83d78438d710318a P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:00.853410 30041 raft_consensus.cc:515] T 959940284eb24fdf83d78438d710318a P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:27:00.851117 29860 raft_consensus.cc:515] T c1c8411289a74b5685700229d245ff75 P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:27:00.854766 29792 consensus_queue.cc:260] T d1918297b9eb41f5847a211e668ae748 P 584dac86183a41d9b3bea1c2085e62fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:27:00.864168 29792 ts_tablet_manager.cc:1434] T d1918297b9eb41f5847a211e668ae748 P 584dac86183a41d9b3bea1c2085e62fa: Time spent starting tablet: real 0.041s	user 0.011s	sys 0.000s
I20260709 12:27:00.866293 29792 tablet_bootstrap.cc:492] T 425777eed34441cd93e0902bdc126516 P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap starting.
I20260709 12:27:00.869503 29793 ts_tablet_manager.cc:1434] T d1918297b9eb41f5847a211e668ae748 P d73d92fb35234d16824c55689b5fce28: Time spent starting tablet: real 0.077s	user 0.008s	sys 0.000s
I20260709 12:27:00.872452 30041 leader_election.cc:290] T 959940284eb24fdf83d78438d710318a P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825), c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895)
I20260709 12:27:00.868574 29953 raft_consensus.cc:2749] T 6a30685caefb4d0e80fa4ea87d70760c P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:00.883607 29597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "959940284eb24fdf83d78438d710318a" candidate_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d73d92fb35234d16824c55689b5fce28" is_pre_election: true
I20260709 12:27:00.884505 29597 raft_consensus.cc:2468] T 959940284eb24fdf83d78438d710318a P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d13c58ad8ac441c4a9b396eb04b1a476 in term 0.
I20260709 12:27:00.885579 29745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "959940284eb24fdf83d78438d710318a" candidate_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" is_pre_election: true
I20260709 12:27:00.887719 29634 leader_election.cc:304] T 959940284eb24fdf83d78438d710318a P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d13c58ad8ac441c4a9b396eb04b1a476, d73d92fb35234d16824c55689b5fce28; no voters: 
I20260709 12:27:00.888900 30043 raft_consensus.cc:2804] T 959940284eb24fdf83d78438d710318a P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:00.889350 30043 raft_consensus.cc:493] T 959940284eb24fdf83d78438d710318a P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:00.889664 30043 raft_consensus.cc:3060] T 959940284eb24fdf83d78438d710318a P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:00.890398 29860 leader_election.cc:290] T c1c8411289a74b5685700229d245ff75 P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825), d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209)
I20260709 12:27:00.891722 29670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c1c8411289a74b5685700229d245ff75" candidate_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" is_pre_election: true
I20260709 12:27:00.893625 29794 ts_tablet_manager.cc:1434] T 959940284eb24fdf83d78438d710318a P d13c58ad8ac441c4a9b396eb04b1a476: Time spent starting tablet: real 0.100s	user 0.009s	sys 0.004s
W20260709 12:27:00.896829 29709 leader_election.cc:343] T c1c8411289a74b5685700229d245ff75 P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:00.897101 29597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c1c8411289a74b5685700229d245ff75" candidate_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d73d92fb35234d16824c55689b5fce28" is_pre_election: true
I20260709 12:27:00.897764 29597 raft_consensus.cc:2468] T c1c8411289a74b5685700229d245ff75 P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6610c00ffb948ce9c34f41dec3d49e8 in term 0.
W20260709 12:27:00.893669 29635 leader_election.cc:343] T 959940284eb24fdf83d78438d710318a P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:00.899431 29708 leader_election.cc:304] T c1c8411289a74b5685700229d245ff75 P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c6610c00ffb948ce9c34f41dec3d49e8, d73d92fb35234d16824c55689b5fce28; no voters: d13c58ad8ac441c4a9b396eb04b1a476
I20260709 12:27:00.900892 29860 raft_consensus.cc:2804] T c1c8411289a74b5685700229d245ff75 P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:00.901934 29860 raft_consensus.cc:493] T c1c8411289a74b5685700229d245ff75 P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:00.902367 29860 raft_consensus.cc:3060] T c1c8411289a74b5685700229d245ff75 P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:00.907588 30041 consensus_queue.cc:1048] T 07a2b94c0823467695508292a9ccdcc5 P d13c58ad8ac441c4a9b396eb04b1a476 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.009s
I20260709 12:27:00.920897 29792 tablet_bootstrap.cc:654] T 425777eed34441cd93e0902bdc126516 P 584dac86183a41d9b3bea1c2085e62fa: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:00.922086 29794 tablet_bootstrap.cc:492] T 16649fd17d434e3c95e9d585d2404659 P d13c58ad8ac441c4a9b396eb04b1a476: Bootstrap starting.
I20260709 12:27:00.921322 29858 consensus_queue.cc:1048] T 6a30685caefb4d0e80fa4ea87d70760c P c6610c00ffb948ce9c34f41dec3d49e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20260709 12:27:00.939677 29793 tablet_bootstrap.cc:492] T c6e621aa77d5474580dfc5949801ccc9 P d73d92fb35234d16824c55689b5fce28: Bootstrap starting.
I20260709 12:27:00.934031 30043 raft_consensus.cc:515] T 959940284eb24fdf83d78438d710318a P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:27:00.942821 30043 leader_election.cc:290] T 959940284eb24fdf83d78438d710318a P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 1 election: Requested vote from peers d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825), c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895)
I20260709 12:27:00.944195 29743 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "959940284eb24fdf83d78438d710318a" candidate_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6610c00ffb948ce9c34f41dec3d49e8"
I20260709 12:27:00.945403 29597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "959940284eb24fdf83d78438d710318a" candidate_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d73d92fb35234d16824c55689b5fce28"
W20260709 12:27:00.945827 29635 leader_election.cc:343] T 959940284eb24fdf83d78438d710318a P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:00.946074 29597 raft_consensus.cc:3060] T 959940284eb24fdf83d78438d710318a P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:00.973781 29858 consensus_queue.cc:1048] T 6a30685caefb4d0e80fa4ea87d70760c P c6610c00ffb948ce9c34f41dec3d49e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:00.977849 29793 tablet_bootstrap.cc:654] T c6e621aa77d5474580dfc5949801ccc9 P d73d92fb35234d16824c55689b5fce28: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:00.977636 29860 raft_consensus.cc:515] T c1c8411289a74b5685700229d245ff75 P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:27:00.986824 29794 tablet_bootstrap.cc:654] T 16649fd17d434e3c95e9d585d2404659 P d13c58ad8ac441c4a9b396eb04b1a476: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:00.989508 29860 leader_election.cc:290] T c1c8411289a74b5685700229d245ff75 P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 1 election: Requested vote from peers d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825), d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209)
I20260709 12:27:00.990501 29595 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c1c8411289a74b5685700229d245ff75" candidate_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d73d92fb35234d16824c55689b5fce28"
I20260709 12:27:00.991516 29595 raft_consensus.cc:3060] T c1c8411289a74b5685700229d245ff75 P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:00.973865 29597 raft_consensus.cc:2468] T 959940284eb24fdf83d78438d710318a P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d13c58ad8ac441c4a9b396eb04b1a476 in term 1.
I20260709 12:27:00.980409 29790 tablet_bootstrap.cc:492] T 5be58d64d45e40d7981018b84a01b1ed P c6610c00ffb948ce9c34f41dec3d49e8: No bootstrap required, opened a new log
I20260709 12:27:00.980718 29418 catalog_manager.cc:5697] T 16649fd17d434e3c95e9d585d2404659 P 584dac86183a41d9b3bea1c2085e62fa reported cstate change: term changed from 0 to 1, leader changed from <none> to 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65). New cstate: current_term: 1 leader_uuid: "584dac86183a41d9b3bea1c2085e62fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:01.007323 29790 ts_tablet_manager.cc:1403] T 5be58d64d45e40d7981018b84a01b1ed P c6610c00ffb948ce9c34f41dec3d49e8: Time spent bootstrapping tablet: real 0.276s	user 0.032s	sys 0.006s
I20260709 12:27:01.007937 29418 catalog_manager.cc:5697] T c6e621aa77d5474580dfc5949801ccc9 P 584dac86183a41d9b3bea1c2085e62fa reported cstate change: term changed from 0 to 1, leader changed from <none> to 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65). New cstate: current_term: 1 leader_uuid: "584dac86183a41d9b3bea1c2085e62fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:01.010782 29792 tablet_bootstrap.cc:492] T 425777eed34441cd93e0902bdc126516 P 584dac86183a41d9b3bea1c2085e62fa: No bootstrap required, opened a new log
I20260709 12:27:00.992731 29670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c1c8411289a74b5685700229d245ff75" candidate_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d13c58ad8ac441c4a9b396eb04b1a476"
I20260709 12:27:01.011068 29790 raft_consensus.cc:359] T 5be58d64d45e40d7981018b84a01b1ed P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:01.011960 29790 raft_consensus.cc:385] T 5be58d64d45e40d7981018b84a01b1ed P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:01.012318 29790 raft_consensus.cc:740] T 5be58d64d45e40d7981018b84a01b1ed P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6610c00ffb948ce9c34f41dec3d49e8, State: Initialized, Role: FOLLOWER
I20260709 12:27:01.011305 29792 ts_tablet_manager.cc:1403] T 425777eed34441cd93e0902bdc126516 P 584dac86183a41d9b3bea1c2085e62fa: Time spent bootstrapping tablet: real 0.145s	user 0.013s	sys 0.008s
W20260709 12:27:01.012822 29709 leader_election.cc:343] T c1c8411289a74b5685700229d245ff75 P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:01.013202 29790 consensus_queue.cc:260] T 5be58d64d45e40d7981018b84a01b1ed P c6610c00ffb948ce9c34f41dec3d49e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:01.014735 29634 leader_election.cc:304] T 959940284eb24fdf83d78438d710318a P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d13c58ad8ac441c4a9b396eb04b1a476, d73d92fb35234d16824c55689b5fce28; no voters: c6610c00ffb948ce9c34f41dec3d49e8
I20260709 12:27:01.015443 29790 ts_tablet_manager.cc:1434] T 5be58d64d45e40d7981018b84a01b1ed P c6610c00ffb948ce9c34f41dec3d49e8: Time spent starting tablet: real 0.008s	user 0.001s	sys 0.003s
I20260709 12:27:01.016420 30072 raft_consensus.cc:2804] T 959940284eb24fdf83d78438d710318a P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:01.016955 29790 tablet_bootstrap.cc:492] T 425777eed34441cd93e0902bdc126516 P c6610c00ffb948ce9c34f41dec3d49e8: Bootstrap starting.
I20260709 12:27:01.017246 30072 raft_consensus.cc:697] T 959940284eb24fdf83d78438d710318a P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 LEADER]: Becoming Leader. State: Replica: d13c58ad8ac441c4a9b396eb04b1a476, State: Running, Role: LEADER
I20260709 12:27:01.018251 30072 consensus_queue.cc:237] T 959940284eb24fdf83d78438d710318a P d13c58ad8ac441c4a9b396eb04b1a476 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:27:01.022228 30043 raft_consensus.cc:493] T 31a190e9f5ff407fab90502fb3197b70 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:01.022828 30043 raft_consensus.cc:515] T 31a190e9f5ff407fab90502fb3197b70 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:27:01.024873 29790 tablet_bootstrap.cc:654] T 425777eed34441cd93e0902bdc126516 P c6610c00ffb948ce9c34f41dec3d49e8: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:01.027974 29743 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31a190e9f5ff407fab90502fb3197b70" candidate_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" is_pre_election: true
I20260709 12:27:01.025099 30043 leader_election.cc:290] T 31a190e9f5ff407fab90502fb3197b70 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895), 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033)
I20260709 12:27:01.029366 29743 raft_consensus.cc:2468] T 31a190e9f5ff407fab90502fb3197b70 P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d13c58ad8ac441c4a9b396eb04b1a476 in term 0.
I20260709 12:27:01.028661 29595 raft_consensus.cc:2468] T c1c8411289a74b5685700229d245ff75 P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6610c00ffb948ce9c34f41dec3d49e8 in term 1.
I20260709 12:27:01.042407 29792 raft_consensus.cc:359] T 425777eed34441cd93e0902bdc126516 P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:27:01.043704 29792 raft_consensus.cc:385] T 425777eed34441cd93e0902bdc126516 P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:01.044081 29792 raft_consensus.cc:740] T 425777eed34441cd93e0902bdc126516 P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 584dac86183a41d9b3bea1c2085e62fa, State: Initialized, Role: FOLLOWER
I20260709 12:27:01.043910 29635 leader_election.cc:304] T 31a190e9f5ff407fab90502fb3197b70 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c6610c00ffb948ce9c34f41dec3d49e8, d13c58ad8ac441c4a9b396eb04b1a476; no voters: 
I20260709 12:27:01.040846 29521 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31a190e9f5ff407fab90502fb3197b70" candidate_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "584dac86183a41d9b3bea1c2085e62fa" is_pre_election: true
I20260709 12:27:01.046787 30043 raft_consensus.cc:2804] T 31a190e9f5ff407fab90502fb3197b70 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:01.048319 29708 leader_election.cc:304] T c1c8411289a74b5685700229d245ff75 P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c6610c00ffb948ce9c34f41dec3d49e8, d73d92fb35234d16824c55689b5fce28; no voters: d13c58ad8ac441c4a9b396eb04b1a476
W20260709 12:27:01.056767 29634 leader_election.cc:343] T 31a190e9f5ff407fab90502fb3197b70 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:01.057126 29858 raft_consensus.cc:2804] T c1c8411289a74b5685700229d245ff75 P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:01.057710 29858 raft_consensus.cc:697] T c1c8411289a74b5685700229d245ff75 P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 LEADER]: Becoming Leader. State: Replica: c6610c00ffb948ce9c34f41dec3d49e8, State: Running, Role: LEADER
I20260709 12:27:01.047200 30043 raft_consensus.cc:493] T 31a190e9f5ff407fab90502fb3197b70 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:01.058615 29858 consensus_queue.cc:237] T c1c8411289a74b5685700229d245ff75 P c6610c00ffb948ce9c34f41dec3d49e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:27:01.059268 30043 raft_consensus.cc:3060] T 31a190e9f5ff407fab90502fb3197b70 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:01.065492 29792 consensus_queue.cc:260] T 425777eed34441cd93e0902bdc126516 P 584dac86183a41d9b3bea1c2085e62fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:27:01.069350 29792 ts_tablet_manager.cc:1434] T 425777eed34441cd93e0902bdc126516 P 584dac86183a41d9b3bea1c2085e62fa: Time spent starting tablet: real 0.056s	user 0.006s	sys 0.002s
I20260709 12:27:01.074536 29794 tablet_bootstrap.cc:492] T 16649fd17d434e3c95e9d585d2404659 P d13c58ad8ac441c4a9b396eb04b1a476: No bootstrap required, opened a new log
I20260709 12:27:01.075109 29794 ts_tablet_manager.cc:1403] T 16649fd17d434e3c95e9d585d2404659 P d13c58ad8ac441c4a9b396eb04b1a476: Time spent bootstrapping tablet: real 0.155s	user 0.024s	sys 0.002s
I20260709 12:27:01.076341 29417 catalog_manager.cc:5697] T 959940284eb24fdf83d78438d710318a P d13c58ad8ac441c4a9b396eb04b1a476 reported cstate change: term changed from 0 to 1, leader changed from <none> to d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67). New cstate: current_term: 1 leader_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:01.105819 29790 tablet_bootstrap.cc:492] T 425777eed34441cd93e0902bdc126516 P c6610c00ffb948ce9c34f41dec3d49e8: No bootstrap required, opened a new log
I20260709 12:27:01.106395 29790 ts_tablet_manager.cc:1403] T 425777eed34441cd93e0902bdc126516 P c6610c00ffb948ce9c34f41dec3d49e8: Time spent bootstrapping tablet: real 0.090s	user 0.011s	sys 0.002s
I20260709 12:27:01.109498 29790 raft_consensus.cc:359] T 425777eed34441cd93e0902bdc126516 P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:27:01.111209 29790 raft_consensus.cc:385] T 425777eed34441cd93e0902bdc126516 P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:01.111552 29790 raft_consensus.cc:740] T 425777eed34441cd93e0902bdc126516 P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6610c00ffb948ce9c34f41dec3d49e8, State: Initialized, Role: FOLLOWER
I20260709 12:27:01.112907 29790 consensus_queue.cc:260] T 425777eed34441cd93e0902bdc126516 P c6610c00ffb948ce9c34f41dec3d49e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:27:01.117516 29793 tablet_bootstrap.cc:492] T c6e621aa77d5474580dfc5949801ccc9 P d73d92fb35234d16824c55689b5fce28: No bootstrap required, opened a new log
I20260709 12:27:01.118079 29793 ts_tablet_manager.cc:1403] T c6e621aa77d5474580dfc5949801ccc9 P d73d92fb35234d16824c55689b5fce28: Time spent bootstrapping tablet: real 0.198s	user 0.017s	sys 0.005s
I20260709 12:27:01.122028 29793 raft_consensus.cc:359] T c6e621aa77d5474580dfc5949801ccc9 P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:27:01.122779 29793 raft_consensus.cc:385] T c6e621aa77d5474580dfc5949801ccc9 P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:01.123272 29794 raft_consensus.cc:359] T 16649fd17d434e3c95e9d585d2404659 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:27:01.124027 29794 raft_consensus.cc:385] T 16649fd17d434e3c95e9d585d2404659 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:01.124281 29794 raft_consensus.cc:740] T 16649fd17d434e3c95e9d585d2404659 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d13c58ad8ac441c4a9b396eb04b1a476, State: Initialized, Role: FOLLOWER
I20260709 12:27:01.123051 29793 raft_consensus.cc:740] T c6e621aa77d5474580dfc5949801ccc9 P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d73d92fb35234d16824c55689b5fce28, State: Initialized, Role: FOLLOWER
W20260709 12:27:01.126819 29486 consensus_peers.cc:597] T 16649fd17d434e3c95e9d585d2404659 P 584dac86183a41d9b3bea1c2085e62fa -> Peer d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209): Couldn't send request to peer d13c58ad8ac441c4a9b396eb04b1a476. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:01.128750 29793 consensus_queue.cc:260] T c6e621aa77d5474580dfc5949801ccc9 P d73d92fb35234d16824c55689b5fce28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:27:01.131546 29790 ts_tablet_manager.cc:1434] T 425777eed34441cd93e0902bdc126516 P c6610c00ffb948ce9c34f41dec3d49e8: Time spent starting tablet: real 0.025s	user 0.010s	sys 0.000s
I20260709 12:27:01.142683 29417 catalog_manager.cc:5697] T c1c8411289a74b5685700229d245ff75 P c6610c00ffb948ce9c34f41dec3d49e8 reported cstate change: term changed from 0 to 1, leader changed from <none> to c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68). New cstate: current_term: 1 leader_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } health_report { overall_health: HEALTHY } } }
W20260709 12:27:01.134045 29635 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.68:43895, user_credentials={real_user=slave}} blocked reactor thread for 90260.5us
I20260709 12:27:01.139513 30043 raft_consensus.cc:515] T 31a190e9f5ff407fab90502fb3197b70 P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:27:01.158187 30043 leader_election.cc:290] T 31a190e9f5ff407fab90502fb3197b70 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 1 election: Requested vote from peers c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895), 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033)
I20260709 12:27:01.144680 29792 tablet_bootstrap.cc:492] T e67a2affef634fd8bc96a64fa83cffdb P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap starting.
I20260709 12:27:01.161653 29743 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31a190e9f5ff407fab90502fb3197b70" candidate_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6610c00ffb948ce9c34f41dec3d49e8"
I20260709 12:27:01.162376 29743 raft_consensus.cc:3060] T 31a190e9f5ff407fab90502fb3197b70 P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:01.145345 29790 tablet_bootstrap.cc:492] T 1d4e0ad0462e4f74a44155aa3ffaf87f P c6610c00ffb948ce9c34f41dec3d49e8: Bootstrap starting.
I20260709 12:27:01.145869 29793 ts_tablet_manager.cc:1434] T c6e621aa77d5474580dfc5949801ccc9 P d73d92fb35234d16824c55689b5fce28: Time spent starting tablet: real 0.027s	user 0.005s	sys 0.002s
I20260709 12:27:01.159344 29521 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31a190e9f5ff407fab90502fb3197b70" candidate_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "584dac86183a41d9b3bea1c2085e62fa"
I20260709 12:27:01.164959 29793 tablet_bootstrap.cc:492] T 3617af2dee3f48cf85973b1c94cdc33a P d73d92fb35234d16824c55689b5fce28: Bootstrap starting.
I20260709 12:27:01.165810 29860 raft_consensus.cc:493] T e67a2affef634fd8bc96a64fa83cffdb P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:01.166494 29860 raft_consensus.cc:515] T e67a2affef634fd8bc96a64fa83cffdb P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
W20260709 12:27:01.168514 29634 leader_election.cc:343] T 31a190e9f5ff407fab90502fb3197b70 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:01.168992 29860 leader_election.cc:290] T e67a2affef634fd8bc96a64fa83cffdb P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209), 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033)
I20260709 12:27:01.135022 29794 consensus_queue.cc:260] T 16649fd17d434e3c95e9d585d2404659 P d13c58ad8ac441c4a9b396eb04b1a476 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:27:01.171150 29521 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e67a2affef634fd8bc96a64fa83cffdb" candidate_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "584dac86183a41d9b3bea1c2085e62fa" is_pre_election: true
I20260709 12:27:01.173367 29670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e67a2affef634fd8bc96a64fa83cffdb" candidate_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" is_pre_election: true
I20260709 12:27:01.176492 29794 ts_tablet_manager.cc:1434] T 16649fd17d434e3c95e9d585d2404659 P d13c58ad8ac441c4a9b396eb04b1a476: Time spent starting tablet: real 0.101s	user 0.008s	sys 0.000s
I20260709 12:27:01.171650 29793 tablet_bootstrap.cc:654] T 3617af2dee3f48cf85973b1c94cdc33a P d73d92fb35234d16824c55689b5fce28: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:01.171737 29790 tablet_bootstrap.cc:654] T 1d4e0ad0462e4f74a44155aa3ffaf87f P c6610c00ffb948ce9c34f41dec3d49e8: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:01.180972 29792 tablet_bootstrap.cc:654] T e67a2affef634fd8bc96a64fa83cffdb P 584dac86183a41d9b3bea1c2085e62fa: Neither blocks nor log segments found. Creating new log.
W20260709 12:27:01.181640 29708 leader_election.cc:343] T e67a2affef634fd8bc96a64fa83cffdb P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:27:01.190188 29709 leader_election.cc:343] T e67a2affef634fd8bc96a64fa83cffdb P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:01.190804 29709 leader_election.cc:304] T e67a2affef634fd8bc96a64fa83cffdb P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c6610c00ffb948ce9c34f41dec3d49e8; no voters: 584dac86183a41d9b3bea1c2085e62fa, d13c58ad8ac441c4a9b396eb04b1a476
I20260709 12:27:01.190258 29794 tablet_bootstrap.cc:492] T c1c8411289a74b5685700229d245ff75 P d13c58ad8ac441c4a9b396eb04b1a476: Bootstrap starting.
I20260709 12:27:01.192843 29860 raft_consensus.cc:2749] T e67a2affef634fd8bc96a64fa83cffdb P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:01.189008 29743 raft_consensus.cc:2468] T 31a190e9f5ff407fab90502fb3197b70 P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d13c58ad8ac441c4a9b396eb04b1a476 in term 1.
I20260709 12:27:01.200906 30038 consensus_queue.cc:1048] T 16649fd17d434e3c95e9d585d2404659 P 584dac86183a41d9b3bea1c2085e62fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:27:01.203229 29635 leader_election.cc:304] T 31a190e9f5ff407fab90502fb3197b70 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c6610c00ffb948ce9c34f41dec3d49e8, d13c58ad8ac441c4a9b396eb04b1a476; no voters: 584dac86183a41d9b3bea1c2085e62fa
I20260709 12:27:01.210515 29790 tablet_bootstrap.cc:492] T 1d4e0ad0462e4f74a44155aa3ffaf87f P c6610c00ffb948ce9c34f41dec3d49e8: No bootstrap required, opened a new log
I20260709 12:27:01.211700 30081 raft_consensus.cc:2804] T 31a190e9f5ff407fab90502fb3197b70 P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:01.212427 30081 raft_consensus.cc:697] T 31a190e9f5ff407fab90502fb3197b70 P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 LEADER]: Becoming Leader. State: Replica: d13c58ad8ac441c4a9b396eb04b1a476, State: Running, Role: LEADER
I20260709 12:27:01.212724 29790 ts_tablet_manager.cc:1403] T 1d4e0ad0462e4f74a44155aa3ffaf87f P c6610c00ffb948ce9c34f41dec3d49e8: Time spent bootstrapping tablet: real 0.067s	user 0.008s	sys 0.007s
I20260709 12:27:01.214160 29794 tablet_bootstrap.cc:654] T c1c8411289a74b5685700229d245ff75 P d13c58ad8ac441c4a9b396eb04b1a476: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:01.214900 29793 tablet_bootstrap.cc:492] T 3617af2dee3f48cf85973b1c94cdc33a P d73d92fb35234d16824c55689b5fce28: No bootstrap required, opened a new log
I20260709 12:27:01.221306 29793 ts_tablet_manager.cc:1403] T 3617af2dee3f48cf85973b1c94cdc33a P d73d92fb35234d16824c55689b5fce28: Time spent bootstrapping tablet: real 0.057s	user 0.010s	sys 0.003s
I20260709 12:27:01.222457 29790 raft_consensus.cc:359] T 1d4e0ad0462e4f74a44155aa3ffaf87f P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:01.223177 29790 raft_consensus.cc:385] T 1d4e0ad0462e4f74a44155aa3ffaf87f P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:01.223503 29790 raft_consensus.cc:740] T 1d4e0ad0462e4f74a44155aa3ffaf87f P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6610c00ffb948ce9c34f41dec3d49e8, State: Initialized, Role: FOLLOWER
I20260709 12:27:01.224325 29790 consensus_queue.cc:260] T 1d4e0ad0462e4f74a44155aa3ffaf87f P c6610c00ffb948ce9c34f41dec3d49e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:01.224906 29793 raft_consensus.cc:359] T 3617af2dee3f48cf85973b1c94cdc33a P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:27:01.225659 29793 raft_consensus.cc:385] T 3617af2dee3f48cf85973b1c94cdc33a P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:01.225965 29793 raft_consensus.cc:740] T 3617af2dee3f48cf85973b1c94cdc33a P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d73d92fb35234d16824c55689b5fce28, State: Initialized, Role: FOLLOWER
I20260709 12:27:01.226722 29793 consensus_queue.cc:260] T 3617af2dee3f48cf85973b1c94cdc33a P d73d92fb35234d16824c55689b5fce28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:27:01.229477 29793 ts_tablet_manager.cc:1434] T 3617af2dee3f48cf85973b1c94cdc33a P d73d92fb35234d16824c55689b5fce28: Time spent starting tablet: real 0.008s	user 0.005s	sys 0.000s
I20260709 12:27:01.231499 29793 tablet_bootstrap.cc:492] T 1d1c1f58c4d340ac951dfa403d34138a P d73d92fb35234d16824c55689b5fce28: Bootstrap starting.
I20260709 12:27:01.233014 30006 raft_consensus.cc:493] T c6e621aa77d5474580dfc5949801ccc9 P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:01.233580 30006 raft_consensus.cc:515] T c6e621aa77d5474580dfc5949801ccc9 P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:27:01.238865 29521 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6e621aa77d5474580dfc5949801ccc9" candidate_uuid: "d73d92fb35234d16824c55689b5fce28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "584dac86183a41d9b3bea1c2085e62fa" is_pre_election: true
I20260709 12:27:01.240106 29794 tablet_bootstrap.cc:492] T c1c8411289a74b5685700229d245ff75 P d13c58ad8ac441c4a9b396eb04b1a476: No bootstrap required, opened a new log
I20260709 12:27:01.240466 29790 ts_tablet_manager.cc:1434] T 1d4e0ad0462e4f74a44155aa3ffaf87f P c6610c00ffb948ce9c34f41dec3d49e8: Time spent starting tablet: real 0.022s	user 0.005s	sys 0.002s
I20260709 12:27:01.240634 29794 ts_tablet_manager.cc:1403] T c1c8411289a74b5685700229d245ff75 P d13c58ad8ac441c4a9b396eb04b1a476: Time spent bootstrapping tablet: real 0.051s	user 0.011s	sys 0.002s
I20260709 12:27:01.241699 29790 tablet_bootstrap.cc:492] T 6e37d72807e547898f4fce9e8b82b2b0 P c6610c00ffb948ce9c34f41dec3d49e8: Bootstrap starting.
I20260709 12:27:01.241621 30038 raft_consensus.cc:493] T d1918297b9eb41f5847a211e668ae748 P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:01.233014 30081 consensus_queue.cc:237] T 31a190e9f5ff407fab90502fb3197b70 P d13c58ad8ac441c4a9b396eb04b1a476 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:27:01.242659 30006 leader_election.cc:290] T c6e621aa77d5474580dfc5949801ccc9 P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033), d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209)
I20260709 12:27:01.244401 29794 raft_consensus.cc:359] T c1c8411289a74b5685700229d245ff75 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:27:01.245316 29794 raft_consensus.cc:385] T c1c8411289a74b5685700229d245ff75 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:01.245748 29794 raft_consensus.cc:740] T c1c8411289a74b5685700229d245ff75 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d13c58ad8ac441c4a9b396eb04b1a476, State: Initialized, Role: FOLLOWER
I20260709 12:27:01.246619 29794 consensus_queue.cc:260] T c1c8411289a74b5685700229d245ff75 P d13c58ad8ac441c4a9b396eb04b1a476 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:27:01.249176 29793 tablet_bootstrap.cc:654] T 1d1c1f58c4d340ac951dfa403d34138a P d73d92fb35234d16824c55689b5fce28: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:01.242385 30038 raft_consensus.cc:515] T d1918297b9eb41f5847a211e668ae748 P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:27:01.251691 30038 leader_election.cc:290] T d1918297b9eb41f5847a211e668ae748 P 584dac86183a41d9b3bea1c2085e62fa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825), d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209)
I20260709 12:27:01.252895 29794 ts_tablet_manager.cc:1434] T c1c8411289a74b5685700229d245ff75 P d13c58ad8ac441c4a9b396eb04b1a476: Time spent starting tablet: real 0.012s	user 0.007s	sys 0.000s
I20260709 12:27:01.254040 29667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6e621aa77d5474580dfc5949801ccc9" candidate_uuid: "d73d92fb35234d16824c55689b5fce28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" is_pre_election: true
I20260709 12:27:01.255338 29667 raft_consensus.cc:2393] T c6e621aa77d5474580dfc5949801ccc9 P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d73d92fb35234d16824c55689b5fce28 in current term 1: Already voted for candidate 584dac86183a41d9b3bea1c2085e62fa in this term.
I20260709 12:27:01.256367 29790 tablet_bootstrap.cc:654] T 6e37d72807e547898f4fce9e8b82b2b0 P c6610c00ffb948ce9c34f41dec3d49e8: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:01.256704 30006 raft_consensus.cc:493] T d1918297b9eb41f5847a211e668ae748 P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:01.256790 29561 leader_election.cc:304] T c6e621aa77d5474580dfc5949801ccc9 P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d73d92fb35234d16824c55689b5fce28; no voters: 584dac86183a41d9b3bea1c2085e62fa, d13c58ad8ac441c4a9b396eb04b1a476
I20260709 12:27:01.257375 30006 raft_consensus.cc:515] T d1918297b9eb41f5847a211e668ae748 P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:27:01.258463 29794 tablet_bootstrap.cc:492] T e67a2affef634fd8bc96a64fa83cffdb P d13c58ad8ac441c4a9b396eb04b1a476: Bootstrap starting.
I20260709 12:27:01.263262 30003 raft_consensus.cc:3060] T c6e621aa77d5474580dfc5949801ccc9 P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:01.259684 30006 leader_election.cc:290] T d1918297b9eb41f5847a211e668ae748 P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033), d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209)
I20260709 12:27:01.267659 29521 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1918297b9eb41f5847a211e668ae748" candidate_uuid: "d73d92fb35234d16824c55689b5fce28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "584dac86183a41d9b3bea1c2085e62fa" is_pre_election: true
I20260709 12:27:01.268399 29521 raft_consensus.cc:2468] T d1918297b9eb41f5847a211e668ae748 P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d73d92fb35234d16824c55689b5fce28 in term 0.
I20260709 12:27:01.270721 29560 leader_election.cc:304] T d1918297b9eb41f5847a211e668ae748 P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 584dac86183a41d9b3bea1c2085e62fa, d73d92fb35234d16824c55689b5fce28; no voters: 
I20260709 12:27:01.271705 30006 raft_consensus.cc:2804] T d1918297b9eb41f5847a211e668ae748 P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:01.272162 30006 raft_consensus.cc:493] T d1918297b9eb41f5847a211e668ae748 P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:01.272415 29790 tablet_bootstrap.cc:492] T 6e37d72807e547898f4fce9e8b82b2b0 P c6610c00ffb948ce9c34f41dec3d49e8: No bootstrap required, opened a new log
I20260709 12:27:01.272545 30006 raft_consensus.cc:3060] T d1918297b9eb41f5847a211e668ae748 P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:01.273026 29790 ts_tablet_manager.cc:1403] T 6e37d72807e547898f4fce9e8b82b2b0 P c6610c00ffb948ce9c34f41dec3d49e8: Time spent bootstrapping tablet: real 0.032s	user 0.011s	sys 0.001s
I20260709 12:27:01.276202 29790 raft_consensus.cc:359] T 6e37d72807e547898f4fce9e8b82b2b0 P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:27:01.276979 29790 raft_consensus.cc:385] T 6e37d72807e547898f4fce9e8b82b2b0 P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:01.277343 29790 raft_consensus.cc:740] T 6e37d72807e547898f4fce9e8b82b2b0 P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6610c00ffb948ce9c34f41dec3d49e8, State: Initialized, Role: FOLLOWER
I20260709 12:27:01.278187 29790 consensus_queue.cc:260] T 6e37d72807e547898f4fce9e8b82b2b0 P c6610c00ffb948ce9c34f41dec3d49e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:27:01.281626 29790 ts_tablet_manager.cc:1434] T 6e37d72807e547898f4fce9e8b82b2b0 P c6610c00ffb948ce9c34f41dec3d49e8: Time spent starting tablet: real 0.008s	user 0.007s	sys 0.000s
I20260709 12:27:01.282740 29790 tablet_bootstrap.cc:492] T 07a2b94c0823467695508292a9ccdcc5 P c6610c00ffb948ce9c34f41dec3d49e8: Bootstrap starting.
I20260709 12:27:01.285327 30043 raft_consensus.cc:493] T c1c8411289a74b5685700229d245ff75 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:01.286085 29860 raft_consensus.cc:493] T 6e37d72807e547898f4fce9e8b82b2b0 P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:01.285893 30043 raft_consensus.cc:515] T c1c8411289a74b5685700229d245ff75 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:27:01.286559 29860 raft_consensus.cc:515] T 6e37d72807e547898f4fce9e8b82b2b0 P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:27:01.288129 30043 leader_election.cc:290] T c1c8411289a74b5685700229d245ff75 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825), c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895)
I20260709 12:27:01.288691 29860 leader_election.cc:290] T 6e37d72807e547898f4fce9e8b82b2b0 P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209), 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033)
I20260709 12:27:01.290565 29743 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c1c8411289a74b5685700229d245ff75" candidate_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" is_pre_election: true
I20260709 12:27:01.295797 29667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e37d72807e547898f4fce9e8b82b2b0" candidate_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" is_pre_election: true
W20260709 12:27:01.297274 29709 leader_election.cc:343] T 6e37d72807e547898f4fce9e8b82b2b0 P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:01.301774 29521 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e37d72807e547898f4fce9e8b82b2b0" candidate_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "584dac86183a41d9b3bea1c2085e62fa" is_pre_election: true
W20260709 12:27:01.303444 29708 leader_election.cc:343] T 6e37d72807e547898f4fce9e8b82b2b0 P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:01.303934 29708 leader_election.cc:304] T 6e37d72807e547898f4fce9e8b82b2b0 P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c6610c00ffb948ce9c34f41dec3d49e8; no voters: 584dac86183a41d9b3bea1c2085e62fa, d13c58ad8ac441c4a9b396eb04b1a476
I20260709 12:27:01.305136 29860 raft_consensus.cc:2749] T 6e37d72807e547898f4fce9e8b82b2b0 P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:01.307276 29790 tablet_bootstrap.cc:654] T 07a2b94c0823467695508292a9ccdcc5 P c6610c00ffb948ce9c34f41dec3d49e8: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:01.295311 30006 raft_consensus.cc:515] T d1918297b9eb41f5847a211e668ae748 P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
W20260709 12:27:01.312707 29561 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:38209, user_credentials={real_user=slave}} blocked reactor thread for 56192.1us
I20260709 12:27:01.314175 30006 leader_election.cc:290] T d1918297b9eb41f5847a211e668ae748 P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 1 election: Requested vote from peers 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033), d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209)
I20260709 12:27:01.317591 29667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1918297b9eb41f5847a211e668ae748" candidate_uuid: "d73d92fb35234d16824c55689b5fce28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" is_pre_election: true
I20260709 12:27:01.318331 29667 raft_consensus.cc:2468] T d1918297b9eb41f5847a211e668ae748 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d73d92fb35234d16824c55689b5fce28 in term 0.
I20260709 12:27:01.319968 29521 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1918297b9eb41f5847a211e668ae748" candidate_uuid: "d73d92fb35234d16824c55689b5fce28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "584dac86183a41d9b3bea1c2085e62fa"
I20260709 12:27:01.320765 29521 raft_consensus.cc:3060] T d1918297b9eb41f5847a211e668ae748 P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:01.320312 29418 catalog_manager.cc:5697] T 31a190e9f5ff407fab90502fb3197b70 P d13c58ad8ac441c4a9b396eb04b1a476 reported cstate change: term changed from 0 to 1, leader changed from <none> to d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67). New cstate: current_term: 1 leader_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:01.335265 29670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1918297b9eb41f5847a211e668ae748" candidate_uuid: "584dac86183a41d9b3bea1c2085e62fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" is_pre_election: true
I20260709 12:27:01.336131 29670 raft_consensus.cc:2468] T d1918297b9eb41f5847a211e668ae748 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 584dac86183a41d9b3bea1c2085e62fa in term 0.
I20260709 12:27:01.343605 29521 raft_consensus.cc:2468] T d1918297b9eb41f5847a211e668ae748 P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d73d92fb35234d16824c55689b5fce28 in term 1.
I20260709 12:27:01.343627 29486 leader_election.cc:304] T d1918297b9eb41f5847a211e668ae748 P 584dac86183a41d9b3bea1c2085e62fa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 584dac86183a41d9b3bea1c2085e62fa, d13c58ad8ac441c4a9b396eb04b1a476; no voters: 
I20260709 12:27:01.347162 29954 raft_consensus.cc:2764] T d1918297b9eb41f5847a211e668ae748 P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260709 12:27:01.354683 29594 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c1c8411289a74b5685700229d245ff75" candidate_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d73d92fb35234d16824c55689b5fce28" is_pre_election: true
I20260709 12:27:01.355605 29594 raft_consensus.cc:2393] T c1c8411289a74b5685700229d245ff75 P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d13c58ad8ac441c4a9b396eb04b1a476 in current term 1: Already voted for candidate c6610c00ffb948ce9c34f41dec3d49e8 in this term.
I20260709 12:27:01.357460 29790 tablet_bootstrap.cc:492] T 07a2b94c0823467695508292a9ccdcc5 P c6610c00ffb948ce9c34f41dec3d49e8: No bootstrap required, opened a new log
I20260709 12:27:01.357947 29790 ts_tablet_manager.cc:1403] T 07a2b94c0823467695508292a9ccdcc5 P c6610c00ffb948ce9c34f41dec3d49e8: Time spent bootstrapping tablet: real 0.075s	user 0.016s	sys 0.010s
I20260709 12:27:01.358043 29594 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1918297b9eb41f5847a211e668ae748" candidate_uuid: "584dac86183a41d9b3bea1c2085e62fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d73d92fb35234d16824c55689b5fce28" is_pre_election: true
I20260709 12:27:01.360231 29594 raft_consensus.cc:2393] T d1918297b9eb41f5847a211e668ae748 P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 584dac86183a41d9b3bea1c2085e62fa in current term 1: Already voted for candidate d73d92fb35234d16824c55689b5fce28 in this term.
I20260709 12:27:01.362596 29793 tablet_bootstrap.cc:492] T 1d1c1f58c4d340ac951dfa403d34138a P d73d92fb35234d16824c55689b5fce28: No bootstrap required, opened a new log
I20260709 12:27:01.363116 29793 ts_tablet_manager.cc:1403] T 1d1c1f58c4d340ac951dfa403d34138a P d73d92fb35234d16824c55689b5fce28: Time spent bootstrapping tablet: real 0.133s	user 0.017s	sys 0.027s
I20260709 12:27:01.364130 29560 leader_election.cc:304] T d1918297b9eb41f5847a211e668ae748 P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 584dac86183a41d9b3bea1c2085e62fa, d73d92fb35234d16824c55689b5fce28; no voters: 
I20260709 12:27:01.365075 30006 raft_consensus.cc:2804] T d1918297b9eb41f5847a211e668ae748 P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:01.365538 30006 raft_consensus.cc:697] T d1918297b9eb41f5847a211e668ae748 P d73d92fb35234d16824c55689b5fce28 [term 1 LEADER]: Becoming Leader. State: Replica: d73d92fb35234d16824c55689b5fce28, State: Running, Role: LEADER
I20260709 12:27:01.368834 30006 consensus_queue.cc:237] T d1918297b9eb41f5847a211e668ae748 P d73d92fb35234d16824c55689b5fce28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:27:01.370563 29793 raft_consensus.cc:359] T 1d1c1f58c4d340ac951dfa403d34138a P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:01.371387 29793 raft_consensus.cc:385] T 1d1c1f58c4d340ac951dfa403d34138a P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:01.371709 29793 raft_consensus.cc:740] T 1d1c1f58c4d340ac951dfa403d34138a P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d73d92fb35234d16824c55689b5fce28, State: Initialized, Role: FOLLOWER
I20260709 12:27:01.372400 29793 consensus_queue.cc:260] T 1d1c1f58c4d340ac951dfa403d34138a P d73d92fb35234d16824c55689b5fce28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:01.376088 30038 consensus_queue.cc:1048] T c6e621aa77d5474580dfc5949801ccc9 P 584dac86183a41d9b3bea1c2085e62fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:27:01.378427 29793 ts_tablet_manager.cc:1434] T 1d1c1f58c4d340ac951dfa403d34138a P d73d92fb35234d16824c55689b5fce28: Time spent starting tablet: real 0.015s	user 0.007s	sys 0.000s
I20260709 12:27:01.385262 29793 tablet_bootstrap.cc:492] T cbf03426b6df43e5b2b9c56e2eb7687c P d73d92fb35234d16824c55689b5fce28: Bootstrap starting.
I20260709 12:27:01.389873 29794 tablet_bootstrap.cc:654] T e67a2affef634fd8bc96a64fa83cffdb P d13c58ad8ac441c4a9b396eb04b1a476: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:01.393962 29792 tablet_bootstrap.cc:492] T e67a2affef634fd8bc96a64fa83cffdb P 584dac86183a41d9b3bea1c2085e62fa: No bootstrap required, opened a new log
I20260709 12:27:01.394690 29792 ts_tablet_manager.cc:1403] T e67a2affef634fd8bc96a64fa83cffdb P 584dac86183a41d9b3bea1c2085e62fa: Time spent bootstrapping tablet: real 0.250s	user 0.021s	sys 0.023s
I20260709 12:27:01.400496 29790 raft_consensus.cc:359] T 07a2b94c0823467695508292a9ccdcc5 P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:27:01.401692 29790 raft_consensus.cc:385] T 07a2b94c0823467695508292a9ccdcc5 P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:01.402050 29790 raft_consensus.cc:740] T 07a2b94c0823467695508292a9ccdcc5 P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6610c00ffb948ce9c34f41dec3d49e8, State: Initialized, Role: FOLLOWER
I20260709 12:27:01.373052 30003 raft_consensus.cc:2749] T c6e621aa77d5474580dfc5949801ccc9 P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:01.409816 29670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1918297b9eb41f5847a211e668ae748" candidate_uuid: "d73d92fb35234d16824c55689b5fce28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d13c58ad8ac441c4a9b396eb04b1a476"
I20260709 12:27:01.410790 29634 leader_election.cc:304] T c1c8411289a74b5685700229d245ff75 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d13c58ad8ac441c4a9b396eb04b1a476; no voters: c6610c00ffb948ce9c34f41dec3d49e8, d73d92fb35234d16824c55689b5fce28
I20260709 12:27:01.415022 29790 consensus_queue.cc:260] T 07a2b94c0823467695508292a9ccdcc5 P c6610c00ffb948ce9c34f41dec3d49e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:27:01.406314 29792 raft_consensus.cc:359] T e67a2affef634fd8bc96a64fa83cffdb P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:27:01.417673 29790 ts_tablet_manager.cc:1434] T 07a2b94c0823467695508292a9ccdcc5 P c6610c00ffb948ce9c34f41dec3d49e8: Time spent starting tablet: real 0.059s	user 0.007s	sys 0.002s
I20260709 12:27:01.417857 29792 raft_consensus.cc:385] T e67a2affef634fd8bc96a64fa83cffdb P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:01.418323 29792 raft_consensus.cc:740] T e67a2affef634fd8bc96a64fa83cffdb P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 584dac86183a41d9b3bea1c2085e62fa, State: Initialized, Role: FOLLOWER
I20260709 12:27:01.418891 30081 raft_consensus.cc:3060] T c1c8411289a74b5685700229d245ff75 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:01.419260 29792 consensus_queue.cc:260] T e67a2affef634fd8bc96a64fa83cffdb P 584dac86183a41d9b3bea1c2085e62fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:27:01.415288 29793 tablet_bootstrap.cc:654] T cbf03426b6df43e5b2b9c56e2eb7687c P d73d92fb35234d16824c55689b5fce28: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:01.429989 29792 ts_tablet_manager.cc:1434] T e67a2affef634fd8bc96a64fa83cffdb P 584dac86183a41d9b3bea1c2085e62fa: Time spent starting tablet: real 0.035s	user 0.010s	sys 0.000s
I20260709 12:27:01.436946 29792 tablet_bootstrap.cc:492] T 1d4e0ad0462e4f74a44155aa3ffaf87f P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap starting.
I20260709 12:27:01.436165 30046 raft_consensus.cc:493] T 5be58d64d45e40d7981018b84a01b1ed P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:01.437722 30046 raft_consensus.cc:515] T 5be58d64d45e40d7981018b84a01b1ed P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:01.442008 30046 leader_election.cc:290] T 5be58d64d45e40d7981018b84a01b1ed P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033), d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825)
I20260709 12:27:01.444137 29594 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5be58d64d45e40d7981018b84a01b1ed" candidate_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d73d92fb35234d16824c55689b5fce28" is_pre_election: true
W20260709 12:27:01.446450 29708 leader_election.cc:343] T 5be58d64d45e40d7981018b84a01b1ed P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:01.444918 29521 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5be58d64d45e40d7981018b84a01b1ed" candidate_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "584dac86183a41d9b3bea1c2085e62fa" is_pre_election: true
W20260709 12:27:01.449908 29708 leader_election.cc:343] T 5be58d64d45e40d7981018b84a01b1ed P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:01.450351 29708 leader_election.cc:304] T 5be58d64d45e40d7981018b84a01b1ed P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c6610c00ffb948ce9c34f41dec3d49e8; no voters: 584dac86183a41d9b3bea1c2085e62fa, d73d92fb35234d16824c55689b5fce28
I20260709 12:27:01.454866 29792 tablet_bootstrap.cc:654] T 1d4e0ad0462e4f74a44155aa3ffaf87f P 584dac86183a41d9b3bea1c2085e62fa: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:01.443632 30038 consensus_queue.cc:1048] T c6e621aa77d5474580dfc5949801ccc9 P 584dac86183a41d9b3bea1c2085e62fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.013s
I20260709 12:27:01.451673 30046 raft_consensus.cc:2749] T 5be58d64d45e40d7981018b84a01b1ed P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:01.459761 29790 tablet_bootstrap.cc:492] T 1d1c1f58c4d340ac951dfa403d34138a P c6610c00ffb948ce9c34f41dec3d49e8: Bootstrap starting.
I20260709 12:27:01.442880 29670 raft_consensus.cc:3060] T d1918297b9eb41f5847a211e668ae748 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:01.468644 29418 catalog_manager.cc:5697] T d1918297b9eb41f5847a211e668ae748 P d73d92fb35234d16824c55689b5fce28 reported cstate change: term changed from 0 to 1, leader changed from <none> to d73d92fb35234d16824c55689b5fce28 (127.16.29.66). New cstate: current_term: 1 leader_uuid: "d73d92fb35234d16824c55689b5fce28" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:01.484231 30071 raft_consensus.cc:493] T e67a2affef634fd8bc96a64fa83cffdb P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:01.484977 30071 raft_consensus.cc:515] T e67a2affef634fd8bc96a64fa83cffdb P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:27:01.487371 30071 leader_election.cc:290] T e67a2affef634fd8bc96a64fa83cffdb P 584dac86183a41d9b3bea1c2085e62fa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209), c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895)
I20260709 12:27:01.492687 29743 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e67a2affef634fd8bc96a64fa83cffdb" candidate_uuid: "584dac86183a41d9b3bea1c2085e62fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" is_pre_election: true
I20260709 12:27:01.493431 29743 raft_consensus.cc:2468] T e67a2affef634fd8bc96a64fa83cffdb P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 584dac86183a41d9b3bea1c2085e62fa in term 0.
I20260709 12:27:01.495003 29486 leader_election.cc:304] T e67a2affef634fd8bc96a64fa83cffdb P 584dac86183a41d9b3bea1c2085e62fa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 584dac86183a41d9b3bea1c2085e62fa, c6610c00ffb948ce9c34f41dec3d49e8; no voters: 
I20260709 12:27:01.497848 30068 raft_consensus.cc:2804] T e67a2affef634fd8bc96a64fa83cffdb P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:01.498260 30068 raft_consensus.cc:493] T e67a2affef634fd8bc96a64fa83cffdb P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:01.498594 30068 raft_consensus.cc:3060] T e67a2affef634fd8bc96a64fa83cffdb P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:01.505323 29667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e67a2affef634fd8bc96a64fa83cffdb" candidate_uuid: "584dac86183a41d9b3bea1c2085e62fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" is_pre_election: true
I20260709 12:27:01.506428 30043 raft_consensus.cc:493] T 16649fd17d434e3c95e9d585d2404659 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:01.506942 30043 raft_consensus.cc:515] T 16649fd17d434e3c95e9d585d2404659 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
W20260709 12:27:01.507601 29635 consensus_peers.cc:597] T 959940284eb24fdf83d78438d710318a P d13c58ad8ac441c4a9b396eb04b1a476 -> Peer c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895): Couldn't send request to peer c6610c00ffb948ce9c34f41dec3d49e8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:01.511310 30072 consensus_queue.cc:1048] T 959940284eb24fdf83d78438d710318a P d13c58ad8ac441c4a9b396eb04b1a476 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:27:01.512684 29793 tablet_bootstrap.cc:492] T cbf03426b6df43e5b2b9c56e2eb7687c P d73d92fb35234d16824c55689b5fce28: No bootstrap required, opened a new log
I20260709 12:27:01.513226 29793 ts_tablet_manager.cc:1403] T cbf03426b6df43e5b2b9c56e2eb7687c P d73d92fb35234d16824c55689b5fce28: Time spent bootstrapping tablet: real 0.128s	user 0.017s	sys 0.020s
I20260709 12:27:01.513079 29521 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "16649fd17d434e3c95e9d585d2404659" candidate_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "584dac86183a41d9b3bea1c2085e62fa" is_pre_election: true
I20260709 12:27:01.508996 30043 leader_election.cc:290] T 16649fd17d434e3c95e9d585d2404659 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033), d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825)
I20260709 12:27:01.521553 29793 raft_consensus.cc:359] T cbf03426b6df43e5b2b9c56e2eb7687c P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:27:01.522219 29793 raft_consensus.cc:385] T cbf03426b6df43e5b2b9c56e2eb7687c P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260709 12:27:01.516278 29486 leader_election.cc:343] T e67a2affef634fd8bc96a64fa83cffdb P 584dac86183a41d9b3bea1c2085e62fa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:01.522518 29793 raft_consensus.cc:740] T cbf03426b6df43e5b2b9c56e2eb7687c P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d73d92fb35234d16824c55689b5fce28, State: Initialized, Role: FOLLOWER
I20260709 12:27:01.523355 29793 consensus_queue.cc:260] T cbf03426b6df43e5b2b9c56e2eb7687c P d73d92fb35234d16824c55689b5fce28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:27:01.523924 30038 raft_consensus.cc:493] T 425777eed34441cd93e0902bdc126516 P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:01.500190 30081 raft_consensus.cc:2749] T c1c8411289a74b5685700229d245ff75 P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:01.524457 30038 raft_consensus.cc:515] T 425777eed34441cd93e0902bdc126516 P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:27:01.525914 29794 tablet_bootstrap.cc:492] T e67a2affef634fd8bc96a64fa83cffdb P d13c58ad8ac441c4a9b396eb04b1a476: No bootstrap required, opened a new log
I20260709 12:27:01.526376 29794 ts_tablet_manager.cc:1403] T e67a2affef634fd8bc96a64fa83cffdb P d13c58ad8ac441c4a9b396eb04b1a476: Time spent bootstrapping tablet: real 0.268s	user 0.012s	sys 0.028s
I20260709 12:27:01.527952 29793 ts_tablet_manager.cc:1434] T cbf03426b6df43e5b2b9c56e2eb7687c P d73d92fb35234d16824c55689b5fce28: Time spent starting tablet: real 0.014s	user 0.007s	sys 0.002s
I20260709 12:27:01.529110 29793 tablet_bootstrap.cc:492] T 1d4e0ad0462e4f74a44155aa3ffaf87f P d73d92fb35234d16824c55689b5fce28: Bootstrap starting.
I20260709 12:27:01.528513 29594 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "16649fd17d434e3c95e9d585d2404659" candidate_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d73d92fb35234d16824c55689b5fce28" is_pre_election: true
I20260709 12:27:01.530501 30038 leader_election.cc:290] T 425777eed34441cd93e0902bdc126516 P 584dac86183a41d9b3bea1c2085e62fa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209), c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895)
I20260709 12:27:01.531491 29794 raft_consensus.cc:359] T e67a2affef634fd8bc96a64fa83cffdb P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:27:01.532269 29794 raft_consensus.cc:385] T e67a2affef634fd8bc96a64fa83cffdb P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:01.532608 29794 raft_consensus.cc:740] T e67a2affef634fd8bc96a64fa83cffdb P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d13c58ad8ac441c4a9b396eb04b1a476, State: Initialized, Role: FOLLOWER
I20260709 12:27:01.533460 29794 consensus_queue.cc:260] T e67a2affef634fd8bc96a64fa83cffdb P d13c58ad8ac441c4a9b396eb04b1a476 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:27:01.535501 29790 tablet_bootstrap.cc:654] T 1d1c1f58c4d340ac951dfa403d34138a P c6610c00ffb948ce9c34f41dec3d49e8: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:01.535697 29793 tablet_bootstrap.cc:654] T 1d4e0ad0462e4f74a44155aa3ffaf87f P d73d92fb35234d16824c55689b5fce28: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:01.536213 30068 raft_consensus.cc:515] T e67a2affef634fd8bc96a64fa83cffdb P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:27:01.536044 29794 ts_tablet_manager.cc:1434] T e67a2affef634fd8bc96a64fa83cffdb P d13c58ad8ac441c4a9b396eb04b1a476: Time spent starting tablet: real 0.009s	user 0.007s	sys 0.000s
I20260709 12:27:01.541013 29792 tablet_bootstrap.cc:492] T 1d4e0ad0462e4f74a44155aa3ffaf87f P 584dac86183a41d9b3bea1c2085e62fa: No bootstrap required, opened a new log
I20260709 12:27:01.541111 29743 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "425777eed34441cd93e0902bdc126516" candidate_uuid: "584dac86183a41d9b3bea1c2085e62fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" is_pre_election: true
I20260709 12:27:01.541683 29792 ts_tablet_manager.cc:1403] T 1d4e0ad0462e4f74a44155aa3ffaf87f P 584dac86183a41d9b3bea1c2085e62fa: Time spent bootstrapping tablet: real 0.105s	user 0.008s	sys 0.011s
I20260709 12:27:01.541889 29743 raft_consensus.cc:2468] T 425777eed34441cd93e0902bdc126516 P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 584dac86183a41d9b3bea1c2085e62fa in term 0.
I20260709 12:27:01.543269 30046 consensus_queue.cc:1048] T c1c8411289a74b5685700229d245ff75 P c6610c00ffb948ce9c34f41dec3d49e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:27:01.544534 29792 raft_consensus.cc:359] T 1d4e0ad0462e4f74a44155aa3ffaf87f P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:01.545223 29792 raft_consensus.cc:385] T 1d4e0ad0462e4f74a44155aa3ffaf87f P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:01.551705 29486 leader_election.cc:304] T 425777eed34441cd93e0902bdc126516 P 584dac86183a41d9b3bea1c2085e62fa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 584dac86183a41d9b3bea1c2085e62fa, c6610c00ffb948ce9c34f41dec3d49e8; no voters: 
I20260709 12:27:01.552598 30071 raft_consensus.cc:2804] T 425777eed34441cd93e0902bdc126516 P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:01.553030 30071 raft_consensus.cc:493] T 425777eed34441cd93e0902bdc126516 P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:01.553431 30071 raft_consensus.cc:3060] T 425777eed34441cd93e0902bdc126516 P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:01.555112 29794 tablet_bootstrap.cc:492] T cbf03426b6df43e5b2b9c56e2eb7687c P d13c58ad8ac441c4a9b396eb04b1a476: Bootstrap starting.
I20260709 12:27:01.545475 29792 raft_consensus.cc:740] T 1d4e0ad0462e4f74a44155aa3ffaf87f P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 584dac86183a41d9b3bea1c2085e62fa, State: Initialized, Role: FOLLOWER
I20260709 12:27:01.560539 29792 consensus_queue.cc:260] T 1d4e0ad0462e4f74a44155aa3ffaf87f P 584dac86183a41d9b3bea1c2085e62fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:01.561724 29743 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e67a2affef634fd8bc96a64fa83cffdb" candidate_uuid: "584dac86183a41d9b3bea1c2085e62fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6610c00ffb948ce9c34f41dec3d49e8"
I20260709 12:27:01.562412 29743 raft_consensus.cc:3060] T e67a2affef634fd8bc96a64fa83cffdb P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:01.563254 29792 ts_tablet_manager.cc:1434] T 1d4e0ad0462e4f74a44155aa3ffaf87f P 584dac86183a41d9b3bea1c2085e62fa: Time spent starting tablet: real 0.021s	user 0.006s	sys 0.000s
I20260709 12:27:01.565735 29634 leader_election.cc:304] T 16649fd17d434e3c95e9d585d2404659 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d13c58ad8ac441c4a9b396eb04b1a476; no voters: 584dac86183a41d9b3bea1c2085e62fa, d73d92fb35234d16824c55689b5fce28
I20260709 12:27:01.566236 29792 tablet_bootstrap.cc:492] T 01788883c8ea42989f246a3486111a7e P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap starting.
I20260709 12:27:01.567186 30081 raft_consensus.cc:2749] T 16649fd17d434e3c95e9d585d2404659 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:01.572089 30068 leader_election.cc:290] T e67a2affef634fd8bc96a64fa83cffdb P 584dac86183a41d9b3bea1c2085e62fa [CANDIDATE]: Term 1 election: Requested vote from peers d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209), c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895)
I20260709 12:27:01.572656 29671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "425777eed34441cd93e0902bdc126516" candidate_uuid: "584dac86183a41d9b3bea1c2085e62fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" is_pre_election: true
I20260709 12:27:01.573678 29671 raft_consensus.cc:2468] T 425777eed34441cd93e0902bdc126516 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 584dac86183a41d9b3bea1c2085e62fa in term 0.
I20260709 12:27:01.580600 29743 raft_consensus.cc:2468] T e67a2affef634fd8bc96a64fa83cffdb P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 584dac86183a41d9b3bea1c2085e62fa in term 1.
I20260709 12:27:01.581573 29793 tablet_bootstrap.cc:492] T 1d4e0ad0462e4f74a44155aa3ffaf87f P d73d92fb35234d16824c55689b5fce28: No bootstrap required, opened a new log
I20260709 12:27:01.581626 29671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e67a2affef634fd8bc96a64fa83cffdb" candidate_uuid: "584dac86183a41d9b3bea1c2085e62fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d13c58ad8ac441c4a9b396eb04b1a476"
I20260709 12:27:01.582203 29793 ts_tablet_manager.cc:1403] T 1d4e0ad0462e4f74a44155aa3ffaf87f P d73d92fb35234d16824c55689b5fce28: Time spent bootstrapping tablet: real 0.053s	user 0.011s	sys 0.001s
I20260709 12:27:01.582442 29671 raft_consensus.cc:3060] T e67a2affef634fd8bc96a64fa83cffdb P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:01.582068 29486 leader_election.cc:304] T e67a2affef634fd8bc96a64fa83cffdb P 584dac86183a41d9b3bea1c2085e62fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 584dac86183a41d9b3bea1c2085e62fa, c6610c00ffb948ce9c34f41dec3d49e8; no voters: 
I20260709 12:27:01.584349 30038 raft_consensus.cc:2804] T e67a2affef634fd8bc96a64fa83cffdb P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:01.584888 30038 raft_consensus.cc:697] T e67a2affef634fd8bc96a64fa83cffdb P 584dac86183a41d9b3bea1c2085e62fa [term 1 LEADER]: Becoming Leader. State: Replica: 584dac86183a41d9b3bea1c2085e62fa, State: Running, Role: LEADER
I20260709 12:27:01.585783 30038 consensus_queue.cc:237] T e67a2affef634fd8bc96a64fa83cffdb P 584dac86183a41d9b3bea1c2085e62fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:27:01.587052 29793 raft_consensus.cc:359] T 1d4e0ad0462e4f74a44155aa3ffaf87f P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:01.587806 29793 raft_consensus.cc:385] T 1d4e0ad0462e4f74a44155aa3ffaf87f P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:01.588100 29793 raft_consensus.cc:740] T 1d4e0ad0462e4f74a44155aa3ffaf87f P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d73d92fb35234d16824c55689b5fce28, State: Initialized, Role: FOLLOWER
I20260709 12:27:01.588785 29793 consensus_queue.cc:260] T 1d4e0ad0462e4f74a44155aa3ffaf87f P d73d92fb35234d16824c55689b5fce28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:01.591583 29793 ts_tablet_manager.cc:1434] T 1d4e0ad0462e4f74a44155aa3ffaf87f P d73d92fb35234d16824c55689b5fce28: Time spent starting tablet: real 0.009s	user 0.003s	sys 0.000s
I20260709 12:27:01.593747 29792 tablet_bootstrap.cc:654] T 01788883c8ea42989f246a3486111a7e P 584dac86183a41d9b3bea1c2085e62fa: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:01.595355 29667 raft_consensus.cc:3060] T 16649fd17d434e3c95e9d585d2404659 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:01.584856 29794 tablet_bootstrap.cc:654] T cbf03426b6df43e5b2b9c56e2eb7687c P d13c58ad8ac441c4a9b396eb04b1a476: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:01.606607 29793 tablet_bootstrap.cc:492] T 5be58d64d45e40d7981018b84a01b1ed P d73d92fb35234d16824c55689b5fce28: Bootstrap starting.
I20260709 12:27:01.607124 30071 raft_consensus.cc:515] T 425777eed34441cd93e0902bdc126516 P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:27:01.610245 29669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "425777eed34441cd93e0902bdc126516" candidate_uuid: "584dac86183a41d9b3bea1c2085e62fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d13c58ad8ac441c4a9b396eb04b1a476"
I20260709 12:27:01.610884 29669 raft_consensus.cc:3060] T 425777eed34441cd93e0902bdc126516 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:01.615293 29793 tablet_bootstrap.cc:654] T 5be58d64d45e40d7981018b84a01b1ed P d73d92fb35234d16824c55689b5fce28: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:01.615149 30003 raft_consensus.cc:493] T 3617af2dee3f48cf85973b1c94cdc33a P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:01.620225 29669 raft_consensus.cc:2468] T 425777eed34441cd93e0902bdc126516 P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 584dac86183a41d9b3bea1c2085e62fa in term 1.
I20260709 12:27:01.621737 29486 leader_election.cc:304] T 425777eed34441cd93e0902bdc126516 P 584dac86183a41d9b3bea1c2085e62fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 584dac86183a41d9b3bea1c2085e62fa, d13c58ad8ac441c4a9b396eb04b1a476; no voters: 
I20260709 12:27:01.637655 29794 tablet_bootstrap.cc:492] T cbf03426b6df43e5b2b9c56e2eb7687c P d13c58ad8ac441c4a9b396eb04b1a476: No bootstrap required, opened a new log
I20260709 12:27:01.638391 29794 ts_tablet_manager.cc:1403] T cbf03426b6df43e5b2b9c56e2eb7687c P d13c58ad8ac441c4a9b396eb04b1a476: Time spent bootstrapping tablet: real 0.084s	user 0.014s	sys 0.003s
I20260709 12:27:01.641659 29792 tablet_bootstrap.cc:492] T 01788883c8ea42989f246a3486111a7e P 584dac86183a41d9b3bea1c2085e62fa: No bootstrap required, opened a new log
I20260709 12:27:01.642452 29792 ts_tablet_manager.cc:1403] T 01788883c8ea42989f246a3486111a7e P 584dac86183a41d9b3bea1c2085e62fa: Time spent bootstrapping tablet: real 0.076s	user 0.011s	sys 0.007s
I20260709 12:27:01.624804 30003 raft_consensus.cc:515] T 3617af2dee3f48cf85973b1c94cdc33a P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:27:01.632869 30068 raft_consensus.cc:2804] T 425777eed34441cd93e0902bdc126516 P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:01.656433 30068 raft_consensus.cc:697] T 425777eed34441cd93e0902bdc126516 P 584dac86183a41d9b3bea1c2085e62fa [term 1 LEADER]: Becoming Leader. State: Replica: 584dac86183a41d9b3bea1c2085e62fa, State: Running, Role: LEADER
I20260709 12:27:01.657364 30068 consensus_queue.cc:237] T 425777eed34441cd93e0902bdc126516 P 584dac86183a41d9b3bea1c2085e62fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:27:01.632643 29671 raft_consensus.cc:2468] T e67a2affef634fd8bc96a64fa83cffdb P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 584dac86183a41d9b3bea1c2085e62fa in term 1.
I20260709 12:27:01.658066 29417 catalog_manager.cc:5697] T e67a2affef634fd8bc96a64fa83cffdb P 584dac86183a41d9b3bea1c2085e62fa reported cstate change: term changed from 0 to 1, leader changed from <none> to 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65). New cstate: current_term: 1 leader_uuid: "584dac86183a41d9b3bea1c2085e62fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } health_report { overall_health: HEALTHY } } }
I20260709 12:27:01.660645 30046 consensus_queue.cc:1048] T c1c8411289a74b5685700229d245ff75 P c6610c00ffb948ce9c34f41dec3d49e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:27:01.662447 30003 leader_election.cc:290] T 3617af2dee3f48cf85973b1c94cdc33a P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209), c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895)
I20260709 12:27:01.663066 29743 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3617af2dee3f48cf85973b1c94cdc33a" candidate_uuid: "d73d92fb35234d16824c55689b5fce28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" is_pre_election: true
I20260709 12:27:01.663270 29671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3617af2dee3f48cf85973b1c94cdc33a" candidate_uuid: "d73d92fb35234d16824c55689b5fce28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" is_pre_election: true
W20260709 12:27:01.664846 29561 leader_election.cc:343] T 3617af2dee3f48cf85973b1c94cdc33a P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:01.672297 30046 raft_consensus.cc:493] T 1d4e0ad0462e4f74a44155aa3ffaf87f P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:01.672878 30046 raft_consensus.cc:515] T 1d4e0ad0462e4f74a44155aa3ffaf87f P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:01.673308 30071 leader_election.cc:290] T 425777eed34441cd93e0902bdc126516 P 584dac86183a41d9b3bea1c2085e62fa [CANDIDATE]: Term 1 election: Requested vote from peers d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209), c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895)
I20260709 12:27:01.667215 29670 raft_consensus.cc:2468] T d1918297b9eb41f5847a211e668ae748 P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d73d92fb35234d16824c55689b5fce28 in term 1.
I20260709 12:27:01.676137 29745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "425777eed34441cd93e0902bdc126516" candidate_uuid: "584dac86183a41d9b3bea1c2085e62fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6610c00ffb948ce9c34f41dec3d49e8"
I20260709 12:27:01.676839 29745 raft_consensus.cc:3060] T 425777eed34441cd93e0902bdc126516 P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:01.676237 29521 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d4e0ad0462e4f74a44155aa3ffaf87f" candidate_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "584dac86183a41d9b3bea1c2085e62fa" is_pre_election: true
I20260709 12:27:01.680202 29521 raft_consensus.cc:2468] T 1d4e0ad0462e4f74a44155aa3ffaf87f P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6610c00ffb948ce9c34f41dec3d49e8 in term 0.
I20260709 12:27:01.681701 29708 leader_election.cc:304] T 1d4e0ad0462e4f74a44155aa3ffaf87f P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 584dac86183a41d9b3bea1c2085e62fa, c6610c00ffb948ce9c34f41dec3d49e8; no voters: 
I20260709 12:27:01.681768 30046 leader_election.cc:290] T 1d4e0ad0462e4f74a44155aa3ffaf87f P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033), d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825)
I20260709 12:27:01.682858 29860 raft_consensus.cc:2804] T 1d4e0ad0462e4f74a44155aa3ffaf87f P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:01.683225 29860 raft_consensus.cc:493] T 1d4e0ad0462e4f74a44155aa3ffaf87f P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:01.683564 29860 raft_consensus.cc:3060] T 1d4e0ad0462e4f74a44155aa3ffaf87f P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:01.691679 29596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d4e0ad0462e4f74a44155aa3ffaf87f" candidate_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d73d92fb35234d16824c55689b5fce28" is_pre_election: true
I20260709 12:27:01.692502 29596 raft_consensus.cc:2468] T 1d4e0ad0462e4f74a44155aa3ffaf87f P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6610c00ffb948ce9c34f41dec3d49e8 in term 0.
W20260709 12:27:01.694702 29467 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:27:01.697116 29794 raft_consensus.cc:359] T cbf03426b6df43e5b2b9c56e2eb7687c P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
W20260709 12:27:01.697496 29561 leader_election.cc:343] T 3617af2dee3f48cf85973b1c94cdc33a P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:01.697909 29794 raft_consensus.cc:385] T cbf03426b6df43e5b2b9c56e2eb7687c P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:01.698019 29561 leader_election.cc:304] T 3617af2dee3f48cf85973b1c94cdc33a P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d73d92fb35234d16824c55689b5fce28; no voters: c6610c00ffb948ce9c34f41dec3d49e8, d13c58ad8ac441c4a9b396eb04b1a476
I20260709 12:27:01.699226 30003 raft_consensus.cc:2749] T 3617af2dee3f48cf85973b1c94cdc33a P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:01.698238 29794 raft_consensus.cc:740] T cbf03426b6df43e5b2b9c56e2eb7687c P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d13c58ad8ac441c4a9b396eb04b1a476, State: Initialized, Role: FOLLOWER
I20260709 12:27:01.701499 29794 consensus_queue.cc:260] T cbf03426b6df43e5b2b9c56e2eb7687c P d13c58ad8ac441c4a9b396eb04b1a476 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:27:01.709683 29794 ts_tablet_manager.cc:1434] T cbf03426b6df43e5b2b9c56e2eb7687c P d13c58ad8ac441c4a9b396eb04b1a476: Time spent starting tablet: real 0.071s	user 0.007s	sys 0.001s
I20260709 12:27:01.712634 29793 tablet_bootstrap.cc:492] T 5be58d64d45e40d7981018b84a01b1ed P d73d92fb35234d16824c55689b5fce28: No bootstrap required, opened a new log
I20260709 12:27:01.713310 29793 ts_tablet_manager.cc:1403] T 5be58d64d45e40d7981018b84a01b1ed P d73d92fb35234d16824c55689b5fce28: Time spent bootstrapping tablet: real 0.107s	user 0.019s	sys 0.016s
I20260709 12:27:01.718209 29793 raft_consensus.cc:359] T 5be58d64d45e40d7981018b84a01b1ed P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:01.719062 29793 raft_consensus.cc:385] T 5be58d64d45e40d7981018b84a01b1ed P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:01.719389 29793 raft_consensus.cc:740] T 5be58d64d45e40d7981018b84a01b1ed P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d73d92fb35234d16824c55689b5fce28, State: Initialized, Role: FOLLOWER
I20260709 12:27:01.720306 29793 consensus_queue.cc:260] T 5be58d64d45e40d7981018b84a01b1ed P d73d92fb35234d16824c55689b5fce28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:01.722514 29793 ts_tablet_manager.cc:1434] T 5be58d64d45e40d7981018b84a01b1ed P d73d92fb35234d16824c55689b5fce28: Time spent starting tablet: real 0.009s	user 0.004s	sys 0.004s
I20260709 12:27:01.723964 29793 tablet_bootstrap.cc:492] T 5ee924f859f54f64908499bfac48ee62 P d73d92fb35234d16824c55689b5fce28: Bootstrap starting.
I20260709 12:27:01.726300 29860 raft_consensus.cc:515] T 1d4e0ad0462e4f74a44155aa3ffaf87f P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:01.728785 29794 tablet_bootstrap.cc:492] T 6e37d72807e547898f4fce9e8b82b2b0 P d13c58ad8ac441c4a9b396eb04b1a476: Bootstrap starting.
W20260709 12:27:01.732251 29634 consensus_peers.cc:597] T 31a190e9f5ff407fab90502fb3197b70 P d13c58ad8ac441c4a9b396eb04b1a476 -> Peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Couldn't send request to peer 584dac86183a41d9b3bea1c2085e62fa. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:01.737571 29520 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d4e0ad0462e4f74a44155aa3ffaf87f" candidate_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "584dac86183a41d9b3bea1c2085e62fa"
I20260709 12:27:01.739610 29790 tablet_bootstrap.cc:492] T 1d1c1f58c4d340ac951dfa403d34138a P c6610c00ffb948ce9c34f41dec3d49e8: No bootstrap required, opened a new log
I20260709 12:27:01.740540 29790 ts_tablet_manager.cc:1403] T 1d1c1f58c4d340ac951dfa403d34138a P c6610c00ffb948ce9c34f41dec3d49e8: Time spent bootstrapping tablet: real 0.281s	user 0.009s	sys 0.009s
I20260709 12:27:01.739125 29520 raft_consensus.cc:3060] T 1d4e0ad0462e4f74a44155aa3ffaf87f P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:01.712941 29745 raft_consensus.cc:2468] T 425777eed34441cd93e0902bdc126516 P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 584dac86183a41d9b3bea1c2085e62fa in term 1.
I20260709 12:27:01.754354 29596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d4e0ad0462e4f74a44155aa3ffaf87f" candidate_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d73d92fb35234d16824c55689b5fce28"
I20260709 12:27:01.755097 29596 raft_consensus.cc:3060] T 1d4e0ad0462e4f74a44155aa3ffaf87f P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:01.764062 30003 raft_consensus.cc:493] T 1d1c1f58c4d340ac951dfa403d34138a P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:01.764791 30003 raft_consensus.cc:515] T 1d1c1f58c4d340ac951dfa403d34138a P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:01.767295 30003 leader_election.cc:290] T 1d1c1f58c4d340ac951dfa403d34138a P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895), 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033)
I20260709 12:27:01.760763 29860 leader_election.cc:290] T 1d4e0ad0462e4f74a44155aa3ffaf87f P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 1 election: Requested vote from peers 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033), d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825)
I20260709 12:27:01.767683 29745 raft_consensus.cc:3060] T 07a2b94c0823467695508292a9ccdcc5 P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:01.772832 29792 raft_consensus.cc:359] T 01788883c8ea42989f246a3486111a7e P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:01.773604 29792 raft_consensus.cc:385] T 01788883c8ea42989f246a3486111a7e P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:01.773885 29792 raft_consensus.cc:740] T 01788883c8ea42989f246a3486111a7e P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 584dac86183a41d9b3bea1c2085e62fa, State: Initialized, Role: FOLLOWER
I20260709 12:27:01.774622 29743 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d1c1f58c4d340ac951dfa403d34138a" candidate_uuid: "d73d92fb35234d16824c55689b5fce28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" is_pre_election: true
I20260709 12:27:01.780870 29521 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d1c1f58c4d340ac951dfa403d34138a" candidate_uuid: "d73d92fb35234d16824c55689b5fce28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "584dac86183a41d9b3bea1c2085e62fa" is_pre_election: true
I20260709 12:27:01.781670 29521 raft_consensus.cc:2468] T 1d1c1f58c4d340ac951dfa403d34138a P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d73d92fb35234d16824c55689b5fce28 in term 0.
I20260709 12:27:01.782914 29560 leader_election.cc:304] T 1d1c1f58c4d340ac951dfa403d34138a P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 584dac86183a41d9b3bea1c2085e62fa, d73d92fb35234d16824c55689b5fce28; no voters: 
I20260709 12:27:01.783993 30003 raft_consensus.cc:2804] T 1d1c1f58c4d340ac951dfa403d34138a P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:01.784375 30003 raft_consensus.cc:493] T 1d1c1f58c4d340ac951dfa403d34138a P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:01.784708 30003 raft_consensus.cc:3060] T 1d1c1f58c4d340ac951dfa403d34138a P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:01.786378 29794 tablet_bootstrap.cc:654] T 6e37d72807e547898f4fce9e8b82b2b0 P d13c58ad8ac441c4a9b396eb04b1a476: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:01.787914 29420 catalog_manager.cc:5697] T 425777eed34441cd93e0902bdc126516 P 584dac86183a41d9b3bea1c2085e62fa reported cstate change: term changed from 0 to 1, leader changed from <none> to 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65). New cstate: current_term: 1 leader_uuid: "584dac86183a41d9b3bea1c2085e62fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } health_report { overall_health: HEALTHY } } }
W20260709 12:27:01.794533 29561 leader_election.cc:343] T 1d1c1f58c4d340ac951dfa403d34138a P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:01.794518 29792 consensus_queue.cc:260] T 01788883c8ea42989f246a3486111a7e P 584dac86183a41d9b3bea1c2085e62fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:01.802474 29520 raft_consensus.cc:2468] T 1d4e0ad0462e4f74a44155aa3ffaf87f P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6610c00ffb948ce9c34f41dec3d49e8 in term 1.
I20260709 12:27:01.804255 29708 leader_election.cc:304] T 1d4e0ad0462e4f74a44155aa3ffaf87f P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 584dac86183a41d9b3bea1c2085e62fa, c6610c00ffb948ce9c34f41dec3d49e8; no voters: 
I20260709 12:27:01.805264 30046 raft_consensus.cc:2804] T 1d4e0ad0462e4f74a44155aa3ffaf87f P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:01.805832 30046 raft_consensus.cc:697] T 1d4e0ad0462e4f74a44155aa3ffaf87f P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 LEADER]: Becoming Leader. State: Replica: c6610c00ffb948ce9c34f41dec3d49e8, State: Running, Role: LEADER
I20260709 12:27:01.801374 29793 tablet_bootstrap.cc:654] T 5ee924f859f54f64908499bfac48ee62 P d73d92fb35234d16824c55689b5fce28: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:01.813573 30046 consensus_queue.cc:237] T 1d4e0ad0462e4f74a44155aa3ffaf87f P c6610c00ffb948ce9c34f41dec3d49e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:01.814927 29792 ts_tablet_manager.cc:1434] T 01788883c8ea42989f246a3486111a7e P 584dac86183a41d9b3bea1c2085e62fa: Time spent starting tablet: real 0.172s	user 0.006s	sys 0.001s
I20260709 12:27:01.816196 29792 tablet_bootstrap.cc:492] T 31a190e9f5ff407fab90502fb3197b70 P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap starting.
I20260709 12:27:01.813050 29790 raft_consensus.cc:359] T 1d1c1f58c4d340ac951dfa403d34138a P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:01.825035 29790 raft_consensus.cc:385] T 1d1c1f58c4d340ac951dfa403d34138a P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:01.830117 29792 tablet_bootstrap.cc:654] T 31a190e9f5ff407fab90502fb3197b70 P 584dac86183a41d9b3bea1c2085e62fa: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:01.839149 30043 raft_consensus.cc:493] T cbf03426b6df43e5b2b9c56e2eb7687c P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:01.839722 30043 raft_consensus.cc:515] T cbf03426b6df43e5b2b9c56e2eb7687c P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:27:01.825325 29790 raft_consensus.cc:740] T 1d1c1f58c4d340ac951dfa403d34138a P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6610c00ffb948ce9c34f41dec3d49e8, State: Initialized, Role: FOLLOWER
I20260709 12:27:01.843786 29790 consensus_queue.cc:260] T 1d1c1f58c4d340ac951dfa403d34138a P c6610c00ffb948ce9c34f41dec3d49e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:01.846712 30043 leader_election.cc:290] T cbf03426b6df43e5b2b9c56e2eb7687c P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033), d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825)
I20260709 12:27:01.848312 29520 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cbf03426b6df43e5b2b9c56e2eb7687c" candidate_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "584dac86183a41d9b3bea1c2085e62fa" is_pre_election: true
W20260709 12:27:01.852770 29634 leader_election.cc:343] T cbf03426b6df43e5b2b9c56e2eb7687c P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:01.854008 30003 raft_consensus.cc:515] T 1d1c1f58c4d340ac951dfa403d34138a P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:01.855254 29594 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cbf03426b6df43e5b2b9c56e2eb7687c" candidate_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d73d92fb35234d16824c55689b5fce28" is_pre_election: true
I20260709 12:27:01.858208 29794 tablet_bootstrap.cc:492] T 6e37d72807e547898f4fce9e8b82b2b0 P d13c58ad8ac441c4a9b396eb04b1a476: No bootstrap required, opened a new log
I20260709 12:27:01.858937 29794 ts_tablet_manager.cc:1403] T 6e37d72807e547898f4fce9e8b82b2b0 P d13c58ad8ac441c4a9b396eb04b1a476: Time spent bootstrapping tablet: real 0.131s	user 0.011s	sys 0.003s
I20260709 12:27:01.858667 29594 raft_consensus.cc:2468] T cbf03426b6df43e5b2b9c56e2eb7687c P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d13c58ad8ac441c4a9b396eb04b1a476 in term 0.
I20260709 12:27:01.860739 29634 leader_election.cc:304] T cbf03426b6df43e5b2b9c56e2eb7687c P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d13c58ad8ac441c4a9b396eb04b1a476, d73d92fb35234d16824c55689b5fce28; no voters: 584dac86183a41d9b3bea1c2085e62fa
I20260709 12:27:01.863958 30072 raft_consensus.cc:2804] T cbf03426b6df43e5b2b9c56e2eb7687c P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:01.864347 30072 raft_consensus.cc:493] T cbf03426b6df43e5b2b9c56e2eb7687c P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:01.864830 30072 raft_consensus.cc:3060] T cbf03426b6df43e5b2b9c56e2eb7687c P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:01.864349 29743 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d1c1f58c4d340ac951dfa403d34138a" candidate_uuid: "d73d92fb35234d16824c55689b5fce28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6610c00ffb948ce9c34f41dec3d49e8"
I20260709 12:27:01.888043 29743 raft_consensus.cc:3060] T 1d1c1f58c4d340ac951dfa403d34138a P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:01.893335 29790 ts_tablet_manager.cc:1434] T 1d1c1f58c4d340ac951dfa403d34138a P c6610c00ffb948ce9c34f41dec3d49e8: Time spent starting tablet: real 0.153s	user 0.005s	sys 0.003s
I20260709 12:27:01.894516 29790 tablet_bootstrap.cc:492] T 959940284eb24fdf83d78438d710318a P c6610c00ffb948ce9c34f41dec3d49e8: Bootstrap starting.
I20260709 12:27:01.898607 30081 consensus_queue.cc:1048] T 31a190e9f5ff407fab90502fb3197b70 P d13c58ad8ac441c4a9b396eb04b1a476 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:27:01.886093 29794 raft_consensus.cc:359] T 6e37d72807e547898f4fce9e8b82b2b0 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:27:01.904222 29794 raft_consensus.cc:385] T 6e37d72807e547898f4fce9e8b82b2b0 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:01.904592 29794 raft_consensus.cc:740] T 6e37d72807e547898f4fce9e8b82b2b0 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d13c58ad8ac441c4a9b396eb04b1a476, State: Initialized, Role: FOLLOWER
I20260709 12:27:01.914132 29794 consensus_queue.cc:260] T 6e37d72807e547898f4fce9e8b82b2b0 P d13c58ad8ac441c4a9b396eb04b1a476 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:27:01.922246 30003 leader_election.cc:290] T 1d1c1f58c4d340ac951dfa403d34138a P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 1 election: Requested vote from peers c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895), 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033)
I20260709 12:27:01.933354 29792 tablet_bootstrap.cc:492] T 31a190e9f5ff407fab90502fb3197b70 P 584dac86183a41d9b3bea1c2085e62fa: No bootstrap required, opened a new log
I20260709 12:27:01.934062 29792 ts_tablet_manager.cc:1403] T 31a190e9f5ff407fab90502fb3197b70 P 584dac86183a41d9b3bea1c2085e62fa: Time spent bootstrapping tablet: real 0.118s	user 0.039s	sys 0.023s
I20260709 12:27:01.936746 29596 raft_consensus.cc:2468] T 1d4e0ad0462e4f74a44155aa3ffaf87f P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6610c00ffb948ce9c34f41dec3d49e8 in term 1.
I20260709 12:27:01.938014 29521 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d1c1f58c4d340ac951dfa403d34138a" candidate_uuid: "d73d92fb35234d16824c55689b5fce28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "584dac86183a41d9b3bea1c2085e62fa"
I20260709 12:27:01.938793 29521 raft_consensus.cc:3060] T 1d1c1f58c4d340ac951dfa403d34138a P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:01.940196 29417 catalog_manager.cc:5697] T 1d4e0ad0462e4f74a44155aa3ffaf87f P c6610c00ffb948ce9c34f41dec3d49e8 reported cstate change: term changed from 0 to 1, leader changed from <none> to c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68). New cstate: current_term: 1 leader_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:01.937675 29790 tablet_bootstrap.cc:654] T 959940284eb24fdf83d78438d710318a P c6610c00ffb948ce9c34f41dec3d49e8: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:01.952610 29794 ts_tablet_manager.cc:1434] T 6e37d72807e547898f4fce9e8b82b2b0 P d13c58ad8ac441c4a9b396eb04b1a476: Time spent starting tablet: real 0.093s	user 0.009s	sys 0.001s
I20260709 12:27:01.959848 29743 raft_consensus.cc:2468] T 1d1c1f58c4d340ac951dfa403d34138a P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d73d92fb35234d16824c55689b5fce28 in term 1.
I20260709 12:27:01.960197 29793 tablet_bootstrap.cc:492] T 5ee924f859f54f64908499bfac48ee62 P d73d92fb35234d16824c55689b5fce28: No bootstrap required, opened a new log
I20260709 12:27:01.961529 29793 ts_tablet_manager.cc:1403] T 5ee924f859f54f64908499bfac48ee62 P d73d92fb35234d16824c55689b5fce28: Time spent bootstrapping tablet: real 0.238s	user 0.016s	sys 0.014s
I20260709 12:27:01.965505 29561 leader_election.cc:304] T 1d1c1f58c4d340ac951dfa403d34138a P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c6610c00ffb948ce9c34f41dec3d49e8, d73d92fb35234d16824c55689b5fce28; no voters: 
I20260709 12:27:01.966322 30006 raft_consensus.cc:2804] T 1d1c1f58c4d340ac951dfa403d34138a P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:01.966854 30006 raft_consensus.cc:697] T 1d1c1f58c4d340ac951dfa403d34138a P d73d92fb35234d16824c55689b5fce28 [term 1 LEADER]: Becoming Leader. State: Replica: d73d92fb35234d16824c55689b5fce28, State: Running, Role: LEADER
I20260709 12:27:01.967671 30006 consensus_queue.cc:237] T 1d1c1f58c4d340ac951dfa403d34138a P d73d92fb35234d16824c55689b5fce28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:01.969230 29793 raft_consensus.cc:359] T 5ee924f859f54f64908499bfac48ee62 P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:27:01.970122 29793 raft_consensus.cc:385] T 5ee924f859f54f64908499bfac48ee62 P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:01.970784 29793 raft_consensus.cc:740] T 5ee924f859f54f64908499bfac48ee62 P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d73d92fb35234d16824c55689b5fce28, State: Initialized, Role: FOLLOWER
W20260709 12:27:01.972147 29635 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:43895, user_credentials={real_user=slave}} blocked reactor thread for 49660.9us
I20260709 12:27:01.978243 30119 consensus_queue.cc:1048] T d1918297b9eb41f5847a211e668ae748 P d73d92fb35234d16824c55689b5fce28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:01.979449 29794 tablet_bootstrap.cc:492] T 5ee924f859f54f64908499bfac48ee62 P d13c58ad8ac441c4a9b396eb04b1a476: Bootstrap starting.
I20260709 12:27:01.966871 30072 raft_consensus.cc:515] T cbf03426b6df43e5b2b9c56e2eb7687c P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:27:01.986346 29520 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cbf03426b6df43e5b2b9c56e2eb7687c" candidate_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "584dac86183a41d9b3bea1c2085e62fa"
W20260709 12:27:01.988173 29634 leader_election.cc:343] T cbf03426b6df43e5b2b9c56e2eb7687c P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:02.002676 29793 consensus_queue.cc:260] T 5ee924f859f54f64908499bfac48ee62 P d73d92fb35234d16824c55689b5fce28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:27:02.010926 29792 raft_consensus.cc:359] T 31a190e9f5ff407fab90502fb3197b70 P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:27:02.025342 29792 raft_consensus.cc:385] T 31a190e9f5ff407fab90502fb3197b70 P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:02.025816 29792 raft_consensus.cc:740] T 31a190e9f5ff407fab90502fb3197b70 P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 584dac86183a41d9b3bea1c2085e62fa, State: Initialized, Role: FOLLOWER
I20260709 12:27:02.032024 29792 consensus_queue.cc:260] T 31a190e9f5ff407fab90502fb3197b70 P 584dac86183a41d9b3bea1c2085e62fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:27:02.034590 29792 ts_tablet_manager.cc:1434] T 31a190e9f5ff407fab90502fb3197b70 P 584dac86183a41d9b3bea1c2085e62fa: Time spent starting tablet: real 0.100s	user 0.008s	sys 0.002s
I20260709 12:27:02.035612 29792 tablet_bootstrap.cc:492] T 6e37d72807e547898f4fce9e8b82b2b0 P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap starting.
I20260709 12:27:02.038022 29793 ts_tablet_manager.cc:1434] T 5ee924f859f54f64908499bfac48ee62 P d73d92fb35234d16824c55689b5fce28: Time spent starting tablet: real 0.076s	user 0.009s	sys 0.000s
I20260709 12:27:01.999500 30003 consensus_queue.cc:1048] T d1918297b9eb41f5847a211e668ae748 P d73d92fb35234d16824c55689b5fce28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.026s
I20260709 12:27:02.047370 30072 leader_election.cc:290] T cbf03426b6df43e5b2b9c56e2eb7687c P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 1 election: Requested vote from peers 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033), d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825)
I20260709 12:27:02.050495 30120 raft_consensus.cc:493] T 6e37d72807e547898f4fce9e8b82b2b0 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:02.067267 29793 tablet_bootstrap.cc:492] T 6c8b2fa9f2c044ec9eec97c72b561042 P d73d92fb35234d16824c55689b5fce28: Bootstrap starting.
I20260709 12:27:02.079391 29790 tablet_bootstrap.cc:492] T 959940284eb24fdf83d78438d710318a P c6610c00ffb948ce9c34f41dec3d49e8: No bootstrap required, opened a new log
I20260709 12:27:02.080051 29596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cbf03426b6df43e5b2b9c56e2eb7687c" candidate_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d73d92fb35234d16824c55689b5fce28"
I20260709 12:27:02.080488 29790 ts_tablet_manager.cc:1403] T 959940284eb24fdf83d78438d710318a P c6610c00ffb948ce9c34f41dec3d49e8: Time spent bootstrapping tablet: real 0.186s	user 0.012s	sys 0.012s
I20260709 12:27:02.080698 29596 raft_consensus.cc:3060] T cbf03426b6df43e5b2b9c56e2eb7687c P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:02.069775 30120 raft_consensus.cc:515] T 6e37d72807e547898f4fce9e8b82b2b0 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:27:02.087373 30120 leader_election.cc:290] T 6e37d72807e547898f4fce9e8b82b2b0 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895), 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033)
I20260709 12:27:02.098367 29792 tablet_bootstrap.cc:654] T 6e37d72807e547898f4fce9e8b82b2b0 P 584dac86183a41d9b3bea1c2085e62fa: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:02.101466 29793 tablet_bootstrap.cc:654] T 6c8b2fa9f2c044ec9eec97c72b561042 P d73d92fb35234d16824c55689b5fce28: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:02.117606 30006 raft_consensus.cc:493] T 5be58d64d45e40d7981018b84a01b1ed P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:02.118182 30006 raft_consensus.cc:515] T 5be58d64d45e40d7981018b84a01b1ed P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:02.120448 30006 leader_election.cc:290] T 5be58d64d45e40d7981018b84a01b1ed P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895), 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033)
I20260709 12:27:02.117579 29794 tablet_bootstrap.cc:654] T 5ee924f859f54f64908499bfac48ee62 P d13c58ad8ac441c4a9b396eb04b1a476: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:02.120769 29417 catalog_manager.cc:5697] T 1d1c1f58c4d340ac951dfa403d34138a P d73d92fb35234d16824c55689b5fce28 reported cstate change: term changed from 0 to 1, leader changed from <none> to d73d92fb35234d16824c55689b5fce28 (127.16.29.66). New cstate: current_term: 1 leader_uuid: "d73d92fb35234d16824c55689b5fce28" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } health_report { overall_health: HEALTHY } } }
I20260709 12:27:02.121547 29520 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5be58d64d45e40d7981018b84a01b1ed" candidate_uuid: "d73d92fb35234d16824c55689b5fce28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "584dac86183a41d9b3bea1c2085e62fa" is_pre_election: true
W20260709 12:27:02.123198 29560 leader_election.cc:343] T 5be58d64d45e40d7981018b84a01b1ed P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:02.125780 29521 raft_consensus.cc:2468] T 1d1c1f58c4d340ac951dfa403d34138a P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d73d92fb35234d16824c55689b5fce28 in term 1.
I20260709 12:27:02.145889 29521 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e37d72807e547898f4fce9e8b82b2b0" candidate_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "584dac86183a41d9b3bea1c2085e62fa" is_pre_election: true
I20260709 12:27:02.146788 29596 raft_consensus.cc:2468] T cbf03426b6df43e5b2b9c56e2eb7687c P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d13c58ad8ac441c4a9b396eb04b1a476 in term 1.
I20260709 12:27:02.149055 29743 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5be58d64d45e40d7981018b84a01b1ed" candidate_uuid: "d73d92fb35234d16824c55689b5fce28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" is_pre_election: true
I20260709 12:27:02.149789 29743 raft_consensus.cc:2468] T 5be58d64d45e40d7981018b84a01b1ed P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d73d92fb35234d16824c55689b5fce28 in term 0.
I20260709 12:27:02.151507 29634 leader_election.cc:304] T cbf03426b6df43e5b2b9c56e2eb7687c P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d13c58ad8ac441c4a9b396eb04b1a476, d73d92fb35234d16824c55689b5fce28; no voters: 584dac86183a41d9b3bea1c2085e62fa
W20260709 12:27:02.153098 29634 leader_election.cc:343] T 6e37d72807e547898f4fce9e8b82b2b0 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:02.160061 29561 leader_election.cc:304] T 5be58d64d45e40d7981018b84a01b1ed P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c6610c00ffb948ce9c34f41dec3d49e8, d73d92fb35234d16824c55689b5fce28; no voters: 584dac86183a41d9b3bea1c2085e62fa
I20260709 12:27:02.156589 29790 raft_consensus.cc:359] T 959940284eb24fdf83d78438d710318a P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:27:02.162704 29790 raft_consensus.cc:385] T 959940284eb24fdf83d78438d710318a P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:02.163312 29790 raft_consensus.cc:740] T 959940284eb24fdf83d78438d710318a P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6610c00ffb948ce9c34f41dec3d49e8, State: Initialized, Role: FOLLOWER
I20260709 12:27:02.167507 29790 consensus_queue.cc:260] T 959940284eb24fdf83d78438d710318a P c6610c00ffb948ce9c34f41dec3d49e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:27:02.168864 29794 tablet_bootstrap.cc:492] T 5ee924f859f54f64908499bfac48ee62 P d13c58ad8ac441c4a9b396eb04b1a476: No bootstrap required, opened a new log
I20260709 12:27:02.169442 29794 ts_tablet_manager.cc:1403] T 5ee924f859f54f64908499bfac48ee62 P d13c58ad8ac441c4a9b396eb04b1a476: Time spent bootstrapping tablet: real 0.190s	user 0.009s	sys 0.013s
I20260709 12:27:02.164750 29742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e37d72807e547898f4fce9e8b82b2b0" candidate_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" is_pre_election: true
I20260709 12:27:02.173120 29742 raft_consensus.cc:2468] T 6e37d72807e547898f4fce9e8b82b2b0 P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d13c58ad8ac441c4a9b396eb04b1a476 in term 0.
I20260709 12:27:02.174245 29794 raft_consensus.cc:359] T 5ee924f859f54f64908499bfac48ee62 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:27:02.174940 29794 raft_consensus.cc:385] T 5ee924f859f54f64908499bfac48ee62 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:02.175189 29794 raft_consensus.cc:740] T 5ee924f859f54f64908499bfac48ee62 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d13c58ad8ac441c4a9b396eb04b1a476, State: Initialized, Role: FOLLOWER
I20260709 12:27:02.175940 29794 consensus_queue.cc:260] T 5ee924f859f54f64908499bfac48ee62 P d13c58ad8ac441c4a9b396eb04b1a476 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:27:02.157274 30043 raft_consensus.cc:2804] T cbf03426b6df43e5b2b9c56e2eb7687c P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:02.177202 30043 raft_consensus.cc:697] T cbf03426b6df43e5b2b9c56e2eb7687c P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 LEADER]: Becoming Leader. State: Replica: d13c58ad8ac441c4a9b396eb04b1a476, State: Running, Role: LEADER
I20260709 12:27:02.177748 29794 ts_tablet_manager.cc:1434] T 5ee924f859f54f64908499bfac48ee62 P d13c58ad8ac441c4a9b396eb04b1a476: Time spent starting tablet: real 0.008s	user 0.006s	sys 0.000s
I20260709 12:27:02.178864 29794 tablet_bootstrap.cc:492] T d299fb38094e47e8aa3890ca74f6b366 P d13c58ad8ac441c4a9b396eb04b1a476: Bootstrap starting.
I20260709 12:27:02.178369 30043 consensus_queue.cc:237] T cbf03426b6df43e5b2b9c56e2eb7687c P d13c58ad8ac441c4a9b396eb04b1a476 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:27:02.185230 29792 tablet_bootstrap.cc:492] T 6e37d72807e547898f4fce9e8b82b2b0 P 584dac86183a41d9b3bea1c2085e62fa: No bootstrap required, opened a new log
I20260709 12:27:02.186043 29792 ts_tablet_manager.cc:1403] T 6e37d72807e547898f4fce9e8b82b2b0 P 584dac86183a41d9b3bea1c2085e62fa: Time spent bootstrapping tablet: real 0.151s	user 0.020s	sys 0.024s
I20260709 12:27:02.187778 29635 leader_election.cc:304] T 6e37d72807e547898f4fce9e8b82b2b0 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c6610c00ffb948ce9c34f41dec3d49e8, d13c58ad8ac441c4a9b396eb04b1a476; no voters: 584dac86183a41d9b3bea1c2085e62fa
I20260709 12:27:02.190100 29792 raft_consensus.cc:359] T 6e37d72807e547898f4fce9e8b82b2b0 P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:27:02.195660 29792 raft_consensus.cc:385] T 6e37d72807e547898f4fce9e8b82b2b0 P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:02.166745 30003 raft_consensus.cc:2804] T 5be58d64d45e40d7981018b84a01b1ed P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:02.195993 29792 raft_consensus.cc:740] T 6e37d72807e547898f4fce9e8b82b2b0 P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 584dac86183a41d9b3bea1c2085e62fa, State: Initialized, Role: FOLLOWER
I20260709 12:27:02.196233 30003 raft_consensus.cc:493] T 5be58d64d45e40d7981018b84a01b1ed P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:02.196696 30003 raft_consensus.cc:3060] T 5be58d64d45e40d7981018b84a01b1ed P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:02.198179 29792 consensus_queue.cc:260] T 6e37d72807e547898f4fce9e8b82b2b0 P 584dac86183a41d9b3bea1c2085e62fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:27:02.190063 30120 raft_consensus.cc:2804] T 6e37d72807e547898f4fce9e8b82b2b0 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:02.210549 30120 raft_consensus.cc:493] T 6e37d72807e547898f4fce9e8b82b2b0 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:02.210966 30120 raft_consensus.cc:3060] T 6e37d72807e547898f4fce9e8b82b2b0 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:02.223708 29790 ts_tablet_manager.cc:1434] T 959940284eb24fdf83d78438d710318a P c6610c00ffb948ce9c34f41dec3d49e8: Time spent starting tablet: real 0.143s	user 0.008s	sys 0.001s
I20260709 12:27:02.229861 30113 raft_consensus.cc:493] T 959940284eb24fdf83d78438d710318a P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:02.224934 29790 tablet_bootstrap.cc:492] T d299fb38094e47e8aa3890ca74f6b366 P c6610c00ffb948ce9c34f41dec3d49e8: Bootstrap starting.
I20260709 12:27:02.232848 30038 consensus_queue.cc:1048] T e67a2affef634fd8bc96a64fa83cffdb P 584dac86183a41d9b3bea1c2085e62fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:27:02.230460 30113 raft_consensus.cc:515] T 959940284eb24fdf83d78438d710318a P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:27:02.239051 30113 leader_election.cc:290] T 959940284eb24fdf83d78438d710318a P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825), d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209)
I20260709 12:27:02.239542 29667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "959940284eb24fdf83d78438d710318a" candidate_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" is_pre_election: true
I20260709 12:27:02.241824 29595 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "959940284eb24fdf83d78438d710318a" candidate_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d73d92fb35234d16824c55689b5fce28" is_pre_election: true
I20260709 12:27:02.244747 30003 raft_consensus.cc:515] T 5be58d64d45e40d7981018b84a01b1ed P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:02.245400 29708 leader_election.cc:304] T 959940284eb24fdf83d78438d710318a P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c6610c00ffb948ce9c34f41dec3d49e8; no voters: d13c58ad8ac441c4a9b396eb04b1a476, d73d92fb35234d16824c55689b5fce28
I20260709 12:27:02.246181 29794 tablet_bootstrap.cc:654] T d299fb38094e47e8aa3890ca74f6b366 P d13c58ad8ac441c4a9b396eb04b1a476: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:02.246038 30068 raft_consensus.cc:493] T 01788883c8ea42989f246a3486111a7e P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:02.247112 29792 ts_tablet_manager.cc:1434] T 6e37d72807e547898f4fce9e8b82b2b0 P 584dac86183a41d9b3bea1c2085e62fa: Time spent starting tablet: real 0.061s	user 0.005s	sys 0.002s
I20260709 12:27:02.246958 30038 consensus_queue.cc:1048] T e67a2affef634fd8bc96a64fa83cffdb P 584dac86183a41d9b3bea1c2085e62fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:02.247491 30071 consensus_queue.cc:1048] T 425777eed34441cd93e0902bdc126516 P 584dac86183a41d9b3bea1c2085e62fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260709 12:27:02.251609 29793 tablet_bootstrap.cc:492] T 6c8b2fa9f2c044ec9eec97c72b561042 P d73d92fb35234d16824c55689b5fce28: No bootstrap required, opened a new log
I20260709 12:27:02.263792 29793 ts_tablet_manager.cc:1403] T 6c8b2fa9f2c044ec9eec97c72b561042 P d73d92fb35234d16824c55689b5fce28: Time spent bootstrapping tablet: real 0.201s	user 0.026s	sys 0.027s
I20260709 12:27:02.264022 29790 tablet_bootstrap.cc:654] T d299fb38094e47e8aa3890ca74f6b366 P c6610c00ffb948ce9c34f41dec3d49e8: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:02.243415 30120 raft_consensus.cc:515] T 6e37d72807e547898f4fce9e8b82b2b0 P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:27:02.271296 30039 consensus_queue.cc:1048] T 425777eed34441cd93e0902bdc126516 P 584dac86183a41d9b3bea1c2085e62fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:02.275708 30072 raft_consensus.cc:493] T 5ee924f859f54f64908499bfac48ee62 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:02.263134 30068 raft_consensus.cc:515] T 01788883c8ea42989f246a3486111a7e P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:02.276424 30072 raft_consensus.cc:515] T 5ee924f859f54f64908499bfac48ee62 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:27:02.267655 29793 raft_consensus.cc:359] T 6c8b2fa9f2c044ec9eec97c72b561042 P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:02.279268 29793 raft_consensus.cc:385] T 6c8b2fa9f2c044ec9eec97c72b561042 P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:02.279536 29793 raft_consensus.cc:740] T 6c8b2fa9f2c044ec9eec97c72b561042 P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d73d92fb35234d16824c55689b5fce28, State: Initialized, Role: FOLLOWER
I20260709 12:27:02.253417 30113 raft_consensus.cc:2749] T 959940284eb24fdf83d78438d710318a P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:02.289215 29792 tablet_bootstrap.cc:492] T cbf03426b6df43e5b2b9c56e2eb7687c P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap starting.
I20260709 12:27:02.292372 29793 consensus_queue.cc:260] T 6c8b2fa9f2c044ec9eec97c72b561042 P d73d92fb35234d16824c55689b5fce28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:02.312554 29521 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ee924f859f54f64908499bfac48ee62" candidate_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "584dac86183a41d9b3bea1c2085e62fa" is_pre_election: true
I20260709 12:27:02.304056 30068 leader_election.cc:290] T 01788883c8ea42989f246a3486111a7e P 584dac86183a41d9b3bea1c2085e62fa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895), d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825)
W20260709 12:27:02.315238 29634 leader_election.cc:343] T 5ee924f859f54f64908499bfac48ee62 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:02.316157 29596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01788883c8ea42989f246a3486111a7e" candidate_uuid: "584dac86183a41d9b3bea1c2085e62fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d73d92fb35234d16824c55689b5fce28" is_pre_election: true
I20260709 12:27:02.321779 29741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01788883c8ea42989f246a3486111a7e" candidate_uuid: "584dac86183a41d9b3bea1c2085e62fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" is_pre_election: true
I20260709 12:27:02.325558 29745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e37d72807e547898f4fce9e8b82b2b0" candidate_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6610c00ffb948ce9c34f41dec3d49e8"
I20260709 12:27:02.327858 30072 leader_election.cc:290] T 5ee924f859f54f64908499bfac48ee62 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033), d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825)
I20260709 12:27:02.331738 30003 leader_election.cc:290] T 5be58d64d45e40d7981018b84a01b1ed P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 1 election: Requested vote from peers c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895), 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033)
W20260709 12:27:02.323541 29485 leader_election.cc:343] T 01788883c8ea42989f246a3486111a7e P 584dac86183a41d9b3bea1c2085e62fa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:02.351147 29793 ts_tablet_manager.cc:1434] T 6c8b2fa9f2c044ec9eec97c72b561042 P d73d92fb35234d16824c55689b5fce28: Time spent starting tablet: real 0.087s	user 0.008s	sys 0.000s
I20260709 12:27:02.341584 29594 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ee924f859f54f64908499bfac48ee62" candidate_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d73d92fb35234d16824c55689b5fce28" is_pre_election: true
I20260709 12:27:02.352711 29594 raft_consensus.cc:2468] T 5ee924f859f54f64908499bfac48ee62 P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d13c58ad8ac441c4a9b396eb04b1a476 in term 0.
I20260709 12:27:02.363237 29790 tablet_bootstrap.cc:492] T d299fb38094e47e8aa3890ca74f6b366 P c6610c00ffb948ce9c34f41dec3d49e8: No bootstrap required, opened a new log
I20260709 12:27:02.363818 29790 ts_tablet_manager.cc:1403] T d299fb38094e47e8aa3890ca74f6b366 P c6610c00ffb948ce9c34f41dec3d49e8: Time spent bootstrapping tablet: real 0.139s	user 0.017s	sys 0.011s
I20260709 12:27:02.357549 29419 catalog_manager.cc:5697] T cbf03426b6df43e5b2b9c56e2eb7687c P d13c58ad8ac441c4a9b396eb04b1a476 reported cstate change: term changed from 0 to 1, leader changed from <none> to d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67). New cstate: current_term: 1 leader_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } health_report { overall_health: HEALTHY } } }
I20260709 12:27:02.368364 29744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5be58d64d45e40d7981018b84a01b1ed" candidate_uuid: "d73d92fb35234d16824c55689b5fce28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6610c00ffb948ce9c34f41dec3d49e8"
I20260709 12:27:02.369194 29744 raft_consensus.cc:3060] T 5be58d64d45e40d7981018b84a01b1ed P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:02.370857 29522 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5be58d64d45e40d7981018b84a01b1ed" candidate_uuid: "d73d92fb35234d16824c55689b5fce28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "584dac86183a41d9b3bea1c2085e62fa"
I20260709 12:27:02.364377 29858 consensus_queue.cc:1048] T 1d4e0ad0462e4f74a44155aa3ffaf87f P c6610c00ffb948ce9c34f41dec3d49e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.008s
I20260709 12:27:02.379438 29744 raft_consensus.cc:2468] T 5be58d64d45e40d7981018b84a01b1ed P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d73d92fb35234d16824c55689b5fce28 in term 1.
I20260709 12:27:02.379819 29792 tablet_bootstrap.cc:654] T cbf03426b6df43e5b2b9c56e2eb7687c P 584dac86183a41d9b3bea1c2085e62fa: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:02.350069 29745 raft_consensus.cc:3060] T 6e37d72807e547898f4fce9e8b82b2b0 P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:02.392529 29561 leader_election.cc:304] T 5be58d64d45e40d7981018b84a01b1ed P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c6610c00ffb948ce9c34f41dec3d49e8, d73d92fb35234d16824c55689b5fce28; no voters: 
I20260709 12:27:02.398233 29634 leader_election.cc:304] T 5ee924f859f54f64908499bfac48ee62 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d13c58ad8ac441c4a9b396eb04b1a476, d73d92fb35234d16824c55689b5fce28; no voters: 584dac86183a41d9b3bea1c2085e62fa
I20260709 12:27:02.408689 29517 raft_consensus.cc:3060] T 31a190e9f5ff407fab90502fb3197b70 P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:02.410226 29518 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e37d72807e547898f4fce9e8b82b2b0" candidate_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "584dac86183a41d9b3bea1c2085e62fa"
I20260709 12:27:02.410984 29518 raft_consensus.cc:3060] T 6e37d72807e547898f4fce9e8b82b2b0 P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:02.411281 29793 tablet_bootstrap.cc:492] T 01788883c8ea42989f246a3486111a7e P d73d92fb35234d16824c55689b5fce28: Bootstrap starting.
I20260709 12:27:02.412586 30138 raft_consensus.cc:2804] T 5ee924f859f54f64908499bfac48ee62 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:02.413064 30138 raft_consensus.cc:493] T 5ee924f859f54f64908499bfac48ee62 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:02.413381 30138 raft_consensus.cc:3060] T 5ee924f859f54f64908499bfac48ee62 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:02.399636 30003 raft_consensus.cc:2804] T 5be58d64d45e40d7981018b84a01b1ed P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:02.414139 30003 raft_consensus.cc:697] T 5be58d64d45e40d7981018b84a01b1ed P d73d92fb35234d16824c55689b5fce28 [term 1 LEADER]: Becoming Leader. State: Replica: d73d92fb35234d16824c55689b5fce28, State: Running, Role: LEADER
I20260709 12:27:02.415060 30003 consensus_queue.cc:237] T 5be58d64d45e40d7981018b84a01b1ed P d73d92fb35234d16824c55689b5fce28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:02.388623 30120 leader_election.cc:290] T 6e37d72807e547898f4fce9e8b82b2b0 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 1 election: Requested vote from peers c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895), 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033)
I20260709 12:27:02.418076 29741 raft_consensus.cc:3060] T 959940284eb24fdf83d78438d710318a P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:02.418725 29790 raft_consensus.cc:359] T d299fb38094e47e8aa3890ca74f6b366 P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:27:02.419493 29790 raft_consensus.cc:385] T d299fb38094e47e8aa3890ca74f6b366 P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:02.419759 29790 raft_consensus.cc:740] T d299fb38094e47e8aa3890ca74f6b366 P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6610c00ffb948ce9c34f41dec3d49e8, State: Initialized, Role: FOLLOWER
I20260709 12:27:02.395195 29858 consensus_queue.cc:1048] T 1d4e0ad0462e4f74a44155aa3ffaf87f P c6610c00ffb948ce9c34f41dec3d49e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.025s
I20260709 12:27:02.420508 29790 consensus_queue.cc:260] T d299fb38094e47e8aa3890ca74f6b366 P c6610c00ffb948ce9c34f41dec3d49e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:27:02.422158 29790 ts_tablet_manager.cc:1434] T d299fb38094e47e8aa3890ca74f6b366 P c6610c00ffb948ce9c34f41dec3d49e8: Time spent starting tablet: real 0.058s	user 0.007s	sys 0.000s
W20260709 12:27:02.422477 29560 leader_election.cc:343] T 5be58d64d45e40d7981018b84a01b1ed P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:02.423079 29790 tablet_bootstrap.cc:492] T fb7d78e534634e0f992fc39b578ebd8d P c6610c00ffb948ce9c34f41dec3d49e8: Bootstrap starting.
W20260709 12:27:02.425580 29486 leader_election.cc:343] T 01788883c8ea42989f246a3486111a7e P 584dac86183a41d9b3bea1c2085e62fa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:02.426060 29486 leader_election.cc:304] T 01788883c8ea42989f246a3486111a7e P 584dac86183a41d9b3bea1c2085e62fa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 584dac86183a41d9b3bea1c2085e62fa; no voters: c6610c00ffb948ce9c34f41dec3d49e8, d73d92fb35234d16824c55689b5fce28
I20260709 12:27:02.441962 30039 raft_consensus.cc:2749] T 01788883c8ea42989f246a3486111a7e P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:02.479995 29794 tablet_bootstrap.cc:492] T d299fb38094e47e8aa3890ca74f6b366 P d13c58ad8ac441c4a9b396eb04b1a476: No bootstrap required, opened a new log
I20260709 12:27:02.480878 29794 ts_tablet_manager.cc:1403] T d299fb38094e47e8aa3890ca74f6b366 P d13c58ad8ac441c4a9b396eb04b1a476: Time spent bootstrapping tablet: real 0.302s	user 0.017s	sys 0.004s
I20260709 12:27:02.491397 29790 tablet_bootstrap.cc:654] T fb7d78e534634e0f992fc39b578ebd8d P c6610c00ffb948ce9c34f41dec3d49e8: Neither blocks nor log segments found. Creating new log.
W20260709 12:27:02.493309 30136 log.cc:927] Time spent T 425777eed34441cd93e0902bdc126516 P c6610c00ffb948ce9c34f41dec3d49e8: Append to log took a long time: real 0.092s	user 0.000s	sys 0.000s
I20260709 12:27:02.502863 29793 tablet_bootstrap.cc:654] T 01788883c8ea42989f246a3486111a7e P d73d92fb35234d16824c55689b5fce28: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:02.508628 30126 consensus_queue.cc:1048] T 1d1c1f58c4d340ac951dfa403d34138a P d73d92fb35234d16824c55689b5fce28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:02.511685 29794 raft_consensus.cc:359] T d299fb38094e47e8aa3890ca74f6b366 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:27:02.512717 29794 raft_consensus.cc:385] T d299fb38094e47e8aa3890ca74f6b366 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:02.513025 29794 raft_consensus.cc:740] T d299fb38094e47e8aa3890ca74f6b366 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d13c58ad8ac441c4a9b396eb04b1a476, State: Initialized, Role: FOLLOWER
I20260709 12:27:02.517189 29518 raft_consensus.cc:2468] T 6e37d72807e547898f4fce9e8b82b2b0 P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d13c58ad8ac441c4a9b396eb04b1a476 in term 1.
I20260709 12:27:02.519026 29634 leader_election.cc:304] T 6e37d72807e547898f4fce9e8b82b2b0 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 584dac86183a41d9b3bea1c2085e62fa, d13c58ad8ac441c4a9b396eb04b1a476; no voters: 
I20260709 12:27:02.519831 30081 raft_consensus.cc:2804] T 6e37d72807e547898f4fce9e8b82b2b0 P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:02.520445 30081 raft_consensus.cc:697] T 6e37d72807e547898f4fce9e8b82b2b0 P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 LEADER]: Becoming Leader. State: Replica: d13c58ad8ac441c4a9b396eb04b1a476, State: Running, Role: LEADER
I20260709 12:27:02.521247 30081 consensus_queue.cc:237] T 6e37d72807e547898f4fce9e8b82b2b0 P d13c58ad8ac441c4a9b396eb04b1a476 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:27:02.525318 30113 raft_consensus.cc:493] T d299fb38094e47e8aa3890ca74f6b366 P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:02.528661 29794 consensus_queue.cc:260] T d299fb38094e47e8aa3890ca74f6b366 P d13c58ad8ac441c4a9b396eb04b1a476 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:27:02.525864 30113 raft_consensus.cc:515] T d299fb38094e47e8aa3890ca74f6b366 P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:27:02.544811 29794 ts_tablet_manager.cc:1434] T d299fb38094e47e8aa3890ca74f6b366 P d13c58ad8ac441c4a9b396eb04b1a476: Time spent starting tablet: real 0.063s	user 0.010s	sys 0.000s
I20260709 12:27:02.586691 30003 raft_consensus.cc:493] T 6c8b2fa9f2c044ec9eec97c72b561042 P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:02.587837 30003 raft_consensus.cc:515] T 6c8b2fa9f2c044ec9eec97c72b561042 P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
W20260709 12:27:02.620420 29635 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:43895, user_credentials={real_user=slave}} blocked reactor thread for 48517.8us
I20260709 12:27:02.622989 29792 tablet_bootstrap.cc:492] T cbf03426b6df43e5b2b9c56e2eb7687c P 584dac86183a41d9b3bea1c2085e62fa: No bootstrap required, opened a new log
I20260709 12:27:02.623726 29792 ts_tablet_manager.cc:1403] T cbf03426b6df43e5b2b9c56e2eb7687c P 584dac86183a41d9b3bea1c2085e62fa: Time spent bootstrapping tablet: real 0.343s	user 0.015s	sys 0.021s
I20260709 12:27:02.650008 29790 tablet_bootstrap.cc:492] T fb7d78e534634e0f992fc39b578ebd8d P c6610c00ffb948ce9c34f41dec3d49e8: No bootstrap required, opened a new log
I20260709 12:27:02.650635 29790 ts_tablet_manager.cc:1403] T fb7d78e534634e0f992fc39b578ebd8d P c6610c00ffb948ce9c34f41dec3d49e8: Time spent bootstrapping tablet: real 0.228s	user 0.022s	sys 0.011s
I20260709 12:27:02.653851 29793 tablet_bootstrap.cc:492] T 01788883c8ea42989f246a3486111a7e P d73d92fb35234d16824c55689b5fce28: No bootstrap required, opened a new log
I20260709 12:27:02.654381 29793 ts_tablet_manager.cc:1403] T 01788883c8ea42989f246a3486111a7e P d73d92fb35234d16824c55689b5fce28: Time spent bootstrapping tablet: real 0.243s	user 0.018s	sys 0.015s
I20260709 12:27:02.659220 29793 raft_consensus.cc:359] T 01788883c8ea42989f246a3486111a7e P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:02.662456 29793 raft_consensus.cc:385] T 01788883c8ea42989f246a3486111a7e P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:02.662799 29793 raft_consensus.cc:740] T 01788883c8ea42989f246a3486111a7e P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d73d92fb35234d16824c55689b5fce28, State: Initialized, Role: FOLLOWER
I20260709 12:27:02.663833 30003 leader_election.cc:290] T 6c8b2fa9f2c044ec9eec97c72b561042 P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895), 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033)
I20260709 12:27:02.665006 29521 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c8b2fa9f2c044ec9eec97c72b561042" candidate_uuid: "d73d92fb35234d16824c55689b5fce28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "584dac86183a41d9b3bea1c2085e62fa" is_pre_election: true
W20260709 12:27:02.666725 29560 leader_election.cc:343] T 6c8b2fa9f2c044ec9eec97c72b561042 P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:02.663637 29793 consensus_queue.cc:260] T 01788883c8ea42989f246a3486111a7e P d73d92fb35234d16824c55689b5fce28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:02.628597 30138 raft_consensus.cc:515] T 5ee924f859f54f64908499bfac48ee62 P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:27:02.701290 29793 ts_tablet_manager.cc:1434] T 01788883c8ea42989f246a3486111a7e P d73d92fb35234d16824c55689b5fce28: Time spent starting tablet: real 0.047s	user 0.008s	sys 0.000s
I20260709 12:27:02.702919 29793 tablet_bootstrap.cc:492] T abec627108a84083884e64e1eb9dcda6 P d73d92fb35234d16824c55689b5fce28: Bootstrap starting.
I20260709 12:27:02.715504 29792 raft_consensus.cc:359] T cbf03426b6df43e5b2b9c56e2eb7687c P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:27:02.716320 29792 raft_consensus.cc:385] T cbf03426b6df43e5b2b9c56e2eb7687c P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:02.717696 29793 tablet_bootstrap.cc:654] T abec627108a84083884e64e1eb9dcda6 P d73d92fb35234d16824c55689b5fce28: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:02.716637 29792 raft_consensus.cc:740] T cbf03426b6df43e5b2b9c56e2eb7687c P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 584dac86183a41d9b3bea1c2085e62fa, State: Initialized, Role: FOLLOWER
I20260709 12:27:02.722184 30138 leader_election.cc:290] T 5ee924f859f54f64908499bfac48ee62 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 1 election: Requested vote from peers 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033), d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825)
W20260709 12:27:02.722522 30144 log.cc:927] Time spent T 5be58d64d45e40d7981018b84a01b1ed P d73d92fb35234d16824c55689b5fce28: Append to log took a long time: real 0.184s	user 0.001s	sys 0.000s
I20260709 12:27:02.727141 29745 raft_consensus.cc:2468] T 6e37d72807e547898f4fce9e8b82b2b0 P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d13c58ad8ac441c4a9b396eb04b1a476 in term 1.
W20260709 12:27:02.732329 29634 consensus_peers.cc:597] T cbf03426b6df43e5b2b9c56e2eb7687c P d13c58ad8ac441c4a9b396eb04b1a476 -> Peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Couldn't send request to peer 584dac86183a41d9b3bea1c2085e62fa. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:02.737732 29792 consensus_queue.cc:260] T cbf03426b6df43e5b2b9c56e2eb7687c P 584dac86183a41d9b3bea1c2085e62fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:27:02.729619 29666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d299fb38094e47e8aa3890ca74f6b366" candidate_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" is_pre_election: true
I20260709 12:27:02.698688 30113 leader_election.cc:290] T d299fb38094e47e8aa3890ca74f6b366 P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825), d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209)
I20260709 12:27:02.750844 29792 ts_tablet_manager.cc:1434] T cbf03426b6df43e5b2b9c56e2eb7687c P 584dac86183a41d9b3bea1c2085e62fa: Time spent starting tablet: real 0.127s	user 0.008s	sys 0.000s
I20260709 12:27:02.754643 29790 raft_consensus.cc:359] T fb7d78e534634e0f992fc39b578ebd8d P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:27:02.755386 29790 raft_consensus.cc:385] T fb7d78e534634e0f992fc39b578ebd8d P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:02.755723 29790 raft_consensus.cc:740] T fb7d78e534634e0f992fc39b578ebd8d P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6610c00ffb948ce9c34f41dec3d49e8, State: Initialized, Role: FOLLOWER
I20260709 12:27:02.765282 29520 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ee924f859f54f64908499bfac48ee62" candidate_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "584dac86183a41d9b3bea1c2085e62fa"
I20260709 12:27:02.764932 29596 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ee924f859f54f64908499bfac48ee62" candidate_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d73d92fb35234d16824c55689b5fce28"
I20260709 12:27:02.768000 29596 raft_consensus.cc:3060] T 5ee924f859f54f64908499bfac48ee62 P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:27:02.769593 29634 leader_election.cc:343] T 5ee924f859f54f64908499bfac48ee62 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:02.750792 30153 raft_consensus.cc:493] T d299fb38094e47e8aa3890ca74f6b366 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:02.780411 29594 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d299fb38094e47e8aa3890ca74f6b366" candidate_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d73d92fb35234d16824c55689b5fce28" is_pre_election: true
I20260709 12:27:02.782286 29596 raft_consensus.cc:2468] T 5ee924f859f54f64908499bfac48ee62 P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d13c58ad8ac441c4a9b396eb04b1a476 in term 1.
I20260709 12:27:02.788913 29794 tablet_bootstrap.cc:492] T fb7d78e534634e0f992fc39b578ebd8d P d13c58ad8ac441c4a9b396eb04b1a476: Bootstrap starting.
I20260709 12:27:02.790333 29790 consensus_queue.cc:260] T fb7d78e534634e0f992fc39b578ebd8d P c6610c00ffb948ce9c34f41dec3d49e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:27:02.794663 29634 leader_election.cc:304] T 5ee924f859f54f64908499bfac48ee62 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d13c58ad8ac441c4a9b396eb04b1a476, d73d92fb35234d16824c55689b5fce28; no voters: 584dac86183a41d9b3bea1c2085e62fa
I20260709 12:27:02.795706 30043 raft_consensus.cc:2804] T 5ee924f859f54f64908499bfac48ee62 P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:02.796204 30043 raft_consensus.cc:697] T 5ee924f859f54f64908499bfac48ee62 P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 LEADER]: Becoming Leader. State: Replica: d13c58ad8ac441c4a9b396eb04b1a476, State: Running, Role: LEADER
W20260709 12:27:02.797287 29708 leader_election.cc:343] T d299fb38094e47e8aa3890ca74f6b366 P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:02.758541 30126 consensus_queue.cc:1048] T 1d1c1f58c4d340ac951dfa403d34138a P d73d92fb35234d16824c55689b5fce28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.036s
I20260709 12:27:02.800866 30153 raft_consensus.cc:515] T d299fb38094e47e8aa3890ca74f6b366 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:27:02.809366 29666 raft_consensus.cc:2468] T d299fb38094e47e8aa3890ca74f6b366 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6610c00ffb948ce9c34f41dec3d49e8 in term 0.
I20260709 12:27:02.811048 30153 leader_election.cc:290] T d299fb38094e47e8aa3890ca74f6b366 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825), c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895)
I20260709 12:27:02.812624 29792 tablet_bootstrap.cc:492] T 5be58d64d45e40d7981018b84a01b1ed P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap starting.
I20260709 12:27:02.815846 29597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d299fb38094e47e8aa3890ca74f6b366" candidate_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d73d92fb35234d16824c55689b5fce28" is_pre_election: true
I20260709 12:27:02.823154 29709 leader_election.cc:304] T d299fb38094e47e8aa3890ca74f6b366 P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c6610c00ffb948ce9c34f41dec3d49e8, d13c58ad8ac441c4a9b396eb04b1a476; no voters: d73d92fb35234d16824c55689b5fce28
I20260709 12:27:02.824385 29744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c8b2fa9f2c044ec9eec97c72b561042" candidate_uuid: "d73d92fb35234d16824c55689b5fce28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" is_pre_election: true
I20260709 12:27:02.828336 29790 ts_tablet_manager.cc:1434] T fb7d78e534634e0f992fc39b578ebd8d P c6610c00ffb948ce9c34f41dec3d49e8: Time spent starting tablet: real 0.177s	user 0.008s	sys 0.000s
I20260709 12:27:02.829668 29790 tablet_bootstrap.cc:492] T f76101eb30f14b98a9aa8c41e0d6f831 P c6610c00ffb948ce9c34f41dec3d49e8: Bootstrap starting.
I20260709 12:27:02.797837 30043 consensus_queue.cc:237] T 5ee924f859f54f64908499bfac48ee62 P d13c58ad8ac441c4a9b396eb04b1a476 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
W20260709 12:27:02.832913 29634 leader_election.cc:343] T d299fb38094e47e8aa3890ca74f6b366 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:02.846664 29794 tablet_bootstrap.cc:654] T fb7d78e534634e0f992fc39b578ebd8d P d13c58ad8ac441c4a9b396eb04b1a476: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:02.851157 29420 catalog_manager.cc:5697] T 5be58d64d45e40d7981018b84a01b1ed P d73d92fb35234d16824c55689b5fce28 reported cstate change: term changed from 0 to 1, leader changed from <none> to d73d92fb35234d16824c55689b5fce28 (127.16.29.66). New cstate: current_term: 1 leader_uuid: "d73d92fb35234d16824c55689b5fce28" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } health_report { overall_health: HEALTHY } } }
W20260709 12:27:02.900112 29709 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.67:38209, user_credentials={real_user=slave}} blocked reactor thread for 77245.6us
W20260709 12:27:02.902904 29561 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:38209, user_credentials={real_user=slave}} blocked reactor thread for 63869.6us
W20260709 12:27:02.879679 29467 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:27:02.912294 29790 tablet_bootstrap.cc:654] T f76101eb30f14b98a9aa8c41e0d6f831 P c6610c00ffb948ce9c34f41dec3d49e8: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:02.894711 29858 raft_consensus.cc:2804] T d299fb38094e47e8aa3890ca74f6b366 P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:02.913483 29858 raft_consensus.cc:493] T d299fb38094e47e8aa3890ca74f6b366 P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:02.913833 29858 raft_consensus.cc:3060] T d299fb38094e47e8aa3890ca74f6b366 P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:02.915154 29418 catalog_manager.cc:5697] T 6e37d72807e547898f4fce9e8b82b2b0 P d13c58ad8ac441c4a9b396eb04b1a476 reported cstate change: term changed from 0 to 1, leader changed from <none> to d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67). New cstate: current_term: 1 leader_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:02.881383 29745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d299fb38094e47e8aa3890ca74f6b366" candidate_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" is_pre_election: true
I20260709 12:27:02.921979 30072 consensus_queue.cc:1048] T cbf03426b6df43e5b2b9c56e2eb7687c P d13c58ad8ac441c4a9b396eb04b1a476 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:27:02.940407 29858 raft_consensus.cc:515] T d299fb38094e47e8aa3890ca74f6b366 P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:27:02.942018 29745 raft_consensus.cc:2393] T d299fb38094e47e8aa3890ca74f6b366 P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d13c58ad8ac441c4a9b396eb04b1a476 in current term 1: Already voted for candidate c6610c00ffb948ce9c34f41dec3d49e8 in this term.
I20260709 12:27:02.956329 29635 leader_election.cc:304] T d299fb38094e47e8aa3890ca74f6b366 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d13c58ad8ac441c4a9b396eb04b1a476; no voters: c6610c00ffb948ce9c34f41dec3d49e8, d73d92fb35234d16824c55689b5fce28
I20260709 12:27:02.959004 30043 raft_consensus.cc:3060] T d299fb38094e47e8aa3890ca74f6b366 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:27:02.965539 29560 consensus_peers.cc:597] T 5be58d64d45e40d7981018b84a01b1ed P d73d92fb35234d16824c55689b5fce28 -> Peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Couldn't send request to peer 584dac86183a41d9b3bea1c2085e62fa. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:02.971565 30043 raft_consensus.cc:2749] T d299fb38094e47e8aa3890ca74f6b366 P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:02.973381 29858 leader_election.cc:290] T d299fb38094e47e8aa3890ca74f6b366 P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 1 election: Requested vote from peers d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825), d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209)
I20260709 12:27:02.975286 29666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d299fb38094e47e8aa3890ca74f6b366" candidate_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d13c58ad8ac441c4a9b396eb04b1a476"
I20260709 12:27:03.001101 29792 tablet_bootstrap.cc:654] T 5be58d64d45e40d7981018b84a01b1ed P 584dac86183a41d9b3bea1c2085e62fa: Neither blocks nor log segments found. Creating new log.
W20260709 12:27:03.033857 29561 leader_election.cc:343] T 6c8b2fa9f2c044ec9eec97c72b561042 P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:03.034394 29561 leader_election.cc:304] T 6c8b2fa9f2c044ec9eec97c72b561042 P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d73d92fb35234d16824c55689b5fce28; no voters: 584dac86183a41d9b3bea1c2085e62fa, c6610c00ffb948ce9c34f41dec3d49e8
I20260709 12:27:03.036055 30126 raft_consensus.cc:2749] T 6c8b2fa9f2c044ec9eec97c72b561042 P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:03.037199 30003 consensus_queue.cc:1048] T 5be58d64d45e40d7981018b84a01b1ed P d73d92fb35234d16824c55689b5fce28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:27:03.038350 29790 tablet_bootstrap.cc:492] T f76101eb30f14b98a9aa8c41e0d6f831 P c6610c00ffb948ce9c34f41dec3d49e8: No bootstrap required, opened a new log
I20260709 12:27:03.038919 29790 ts_tablet_manager.cc:1403] T f76101eb30f14b98a9aa8c41e0d6f831 P c6610c00ffb948ce9c34f41dec3d49e8: Time spent bootstrapping tablet: real 0.210s	user 0.010s	sys 0.008s
I20260709 12:27:03.012082 29597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d299fb38094e47e8aa3890ca74f6b366" candidate_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d73d92fb35234d16824c55689b5fce28"
I20260709 12:27:03.062662 29793 tablet_bootstrap.cc:492] T abec627108a84083884e64e1eb9dcda6 P d73d92fb35234d16824c55689b5fce28: No bootstrap required, opened a new log
I20260709 12:27:03.063434 29793 ts_tablet_manager.cc:1403] T abec627108a84083884e64e1eb9dcda6 P d73d92fb35234d16824c55689b5fce28: Time spent bootstrapping tablet: real 0.361s	user 0.019s	sys 0.002s
I20260709 12:27:03.085232 30043 consensus_queue.cc:1048] T 6e37d72807e547898f4fce9e8b82b2b0 P d13c58ad8ac441c4a9b396eb04b1a476 [LEADER]: Connected to new peer: Peer: permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:03.094362 29666 raft_consensus.cc:2468] T d299fb38094e47e8aa3890ca74f6b366 P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6610c00ffb948ce9c34f41dec3d49e8 in term 1.
I20260709 12:27:03.104027 30003 raft_consensus.cc:493] T 01788883c8ea42989f246a3486111a7e P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:27:03.106252 29708 leader_election.cc:343] T d299fb38094e47e8aa3890ca74f6b366 P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:03.121769 29790 raft_consensus.cc:359] T f76101eb30f14b98a9aa8c41e0d6f831 P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:27:03.122752 29790 raft_consensus.cc:385] T f76101eb30f14b98a9aa8c41e0d6f831 P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:03.104784 30003 raft_consensus.cc:515] T 01788883c8ea42989f246a3486111a7e P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:03.132589 30148 raft_consensus.cc:493] T cbf03426b6df43e5b2b9c56e2eb7687c P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:03.133548 30148 raft_consensus.cc:515] T cbf03426b6df43e5b2b9c56e2eb7687c P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:27:03.138952 30148 leader_election.cc:290] T cbf03426b6df43e5b2b9c56e2eb7687c P 584dac86183a41d9b3bea1c2085e62fa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825), d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209)
I20260709 12:27:03.141929 29670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cbf03426b6df43e5b2b9c56e2eb7687c" candidate_uuid: "584dac86183a41d9b3bea1c2085e62fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" is_pre_election: true
I20260709 12:27:03.127736 30003 leader_election.cc:290] T 01788883c8ea42989f246a3486111a7e P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895), 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033)
I20260709 12:27:03.152861 29793 raft_consensus.cc:359] T abec627108a84083884e64e1eb9dcda6 P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:27:03.153815 29793 raft_consensus.cc:385] T abec627108a84083884e64e1eb9dcda6 P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:03.154166 29793 raft_consensus.cc:740] T abec627108a84083884e64e1eb9dcda6 P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d73d92fb35234d16824c55689b5fce28, State: Initialized, Role: FOLLOWER
W20260709 12:27:03.156383 29561 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:43895, user_credentials={real_user=slave}} blocked reactor thread for 54573.6us
I20260709 12:27:03.155067 29793 consensus_queue.cc:260] T abec627108a84083884e64e1eb9dcda6 P d73d92fb35234d16824c55689b5fce28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:27:03.123112 29790 raft_consensus.cc:740] T f76101eb30f14b98a9aa8c41e0d6f831 P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6610c00ffb948ce9c34f41dec3d49e8, State: Initialized, Role: FOLLOWER
I20260709 12:27:03.155687 30156 raft_consensus.cc:493] T fb7d78e534634e0f992fc39b578ebd8d P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:03.178804 29790 consensus_queue.cc:260] T f76101eb30f14b98a9aa8c41e0d6f831 P c6610c00ffb948ce9c34f41dec3d49e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:27:03.196061 29793 ts_tablet_manager.cc:1434] T abec627108a84083884e64e1eb9dcda6 P d73d92fb35234d16824c55689b5fce28: Time spent starting tablet: real 0.132s	user 0.010s	sys 0.000s
I20260709 12:27:03.197444 29518 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01788883c8ea42989f246a3486111a7e" candidate_uuid: "d73d92fb35234d16824c55689b5fce28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "584dac86183a41d9b3bea1c2085e62fa" is_pre_election: true
I20260709 12:27:03.198207 29518 raft_consensus.cc:2468] T 01788883c8ea42989f246a3486111a7e P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d73d92fb35234d16824c55689b5fce28 in term 0.
I20260709 12:27:03.183159 29709 leader_election.cc:304] T d299fb38094e47e8aa3890ca74f6b366 P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c6610c00ffb948ce9c34f41dec3d49e8, d13c58ad8ac441c4a9b396eb04b1a476; no voters: d73d92fb35234d16824c55689b5fce28
I20260709 12:27:03.189584 29594 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cbf03426b6df43e5b2b9c56e2eb7687c" candidate_uuid: "584dac86183a41d9b3bea1c2085e62fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d73d92fb35234d16824c55689b5fce28" is_pre_election: true
I20260709 12:27:03.183310 30156 raft_consensus.cc:515] T fb7d78e534634e0f992fc39b578ebd8d P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:27:03.202235 30151 raft_consensus.cc:2804] T d299fb38094e47e8aa3890ca74f6b366 P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:03.204272 30151 raft_consensus.cc:697] T d299fb38094e47e8aa3890ca74f6b366 P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 LEADER]: Becoming Leader. State: Replica: c6610c00ffb948ce9c34f41dec3d49e8, State: Running, Role: LEADER
I20260709 12:27:03.203279 29742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01788883c8ea42989f246a3486111a7e" candidate_uuid: "d73d92fb35234d16824c55689b5fce28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" is_pre_election: true
I20260709 12:27:03.205163 30151 consensus_queue.cc:237] T d299fb38094e47e8aa3890ca74f6b366 P c6610c00ffb948ce9c34f41dec3d49e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
W20260709 12:27:03.214126 29561 leader_election.cc:343] T 01788883c8ea42989f246a3486111a7e P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:03.216006 29485 leader_election.cc:304] T cbf03426b6df43e5b2b9c56e2eb7687c P 584dac86183a41d9b3bea1c2085e62fa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 584dac86183a41d9b3bea1c2085e62fa; no voters: d13c58ad8ac441c4a9b396eb04b1a476, d73d92fb35234d16824c55689b5fce28
I20260709 12:27:03.217875 30148 raft_consensus.cc:2749] T cbf03426b6df43e5b2b9c56e2eb7687c P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:03.220193 29790 ts_tablet_manager.cc:1434] T f76101eb30f14b98a9aa8c41e0d6f831 P c6610c00ffb948ce9c34f41dec3d49e8: Time spent starting tablet: real 0.181s	user 0.006s	sys 0.003s
I20260709 12:27:03.221068 29560 leader_election.cc:304] T 01788883c8ea42989f246a3486111a7e P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 584dac86183a41d9b3bea1c2085e62fa, d73d92fb35234d16824c55689b5fce28; no voters: c6610c00ffb948ce9c34f41dec3d49e8
I20260709 12:27:03.221491 29790 tablet_bootstrap.cc:492] T 394378d078194257b9e04fc659c1f5d4 P c6610c00ffb948ce9c34f41dec3d49e8: Bootstrap starting.
I20260709 12:27:03.228428 30119 raft_consensus.cc:2804] T 01788883c8ea42989f246a3486111a7e P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:03.237627 30119 raft_consensus.cc:493] T 01788883c8ea42989f246a3486111a7e P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:03.238034 30119 raft_consensus.cc:3060] T 01788883c8ea42989f246a3486111a7e P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:03.238436 30156 leader_election.cc:290] T fb7d78e534634e0f992fc39b578ebd8d P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209), 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033)
I20260709 12:27:03.239017 30043 consensus_queue.cc:1048] T 6e37d72807e547898f4fce9e8b82b2b0 P d13c58ad8ac441c4a9b396eb04b1a476 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:27:03.241688 29666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb7d78e534634e0f992fc39b578ebd8d" candidate_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" is_pre_election: true
W20260709 12:27:03.243525 29709 leader_election.cc:343] T fb7d78e534634e0f992fc39b578ebd8d P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:27:03.246996 29634 consensus_peers.cc:597] T 5ee924f859f54f64908499bfac48ee62 P d13c58ad8ac441c4a9b396eb04b1a476 -> Peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Couldn't send request to peer 584dac86183a41d9b3bea1c2085e62fa. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:03.248307 29521 raft_consensus.cc:3060] T cbf03426b6df43e5b2b9c56e2eb7687c P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:03.238699 29794 tablet_bootstrap.cc:492] T fb7d78e534634e0f992fc39b578ebd8d P d13c58ad8ac441c4a9b396eb04b1a476: No bootstrap required, opened a new log
I20260709 12:27:03.249666 29794 ts_tablet_manager.cc:1403] T fb7d78e534634e0f992fc39b578ebd8d P d13c58ad8ac441c4a9b396eb04b1a476: Time spent bootstrapping tablet: real 0.461s	user 0.058s	sys 0.053s
I20260709 12:27:03.252456 30119 raft_consensus.cc:515] T 01788883c8ea42989f246a3486111a7e P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:03.255709 29520 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01788883c8ea42989f246a3486111a7e" candidate_uuid: "d73d92fb35234d16824c55689b5fce28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "584dac86183a41d9b3bea1c2085e62fa"
I20260709 12:27:03.256341 29520 raft_consensus.cc:3060] T 01788883c8ea42989f246a3486111a7e P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:03.256443 29522 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb7d78e534634e0f992fc39b578ebd8d" candidate_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "584dac86183a41d9b3bea1c2085e62fa" is_pre_election: true
I20260709 12:27:03.257686 29741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01788883c8ea42989f246a3486111a7e" candidate_uuid: "d73d92fb35234d16824c55689b5fce28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6610c00ffb948ce9c34f41dec3d49e8"
I20260709 12:27:03.260499 30153 consensus_queue.cc:1048] T 5ee924f859f54f64908499bfac48ee62 P d13c58ad8ac441c4a9b396eb04b1a476 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260709 12:27:03.262061 29561 leader_election.cc:343] T 01788883c8ea42989f246a3486111a7e P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:03.263315 29794 raft_consensus.cc:359] T fb7d78e534634e0f992fc39b578ebd8d P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
W20260709 12:27:03.262072 29708 leader_election.cc:343] T fb7d78e534634e0f992fc39b578ebd8d P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:03.264464 29708 leader_election.cc:304] T fb7d78e534634e0f992fc39b578ebd8d P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c6610c00ffb948ce9c34f41dec3d49e8; no voters: 584dac86183a41d9b3bea1c2085e62fa, d13c58ad8ac441c4a9b396eb04b1a476
I20260709 12:27:03.265167 30046 raft_consensus.cc:2749] T fb7d78e534634e0f992fc39b578ebd8d P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:03.265341 29520 raft_consensus.cc:2468] T 01788883c8ea42989f246a3486111a7e P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d73d92fb35234d16824c55689b5fce28 in term 1.
I20260709 12:27:03.268548 29790 tablet_bootstrap.cc:654] T 394378d078194257b9e04fc659c1f5d4 P c6610c00ffb948ce9c34f41dec3d49e8: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:03.264079 29794 raft_consensus.cc:385] T fb7d78e534634e0f992fc39b578ebd8d P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:03.269483 29794 raft_consensus.cc:740] T fb7d78e534634e0f992fc39b578ebd8d P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d13c58ad8ac441c4a9b396eb04b1a476, State: Initialized, Role: FOLLOWER
I20260709 12:27:03.270170 29560 leader_election.cc:304] T 01788883c8ea42989f246a3486111a7e P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 584dac86183a41d9b3bea1c2085e62fa, d73d92fb35234d16824c55689b5fce28; no voters: c6610c00ffb948ce9c34f41dec3d49e8
I20260709 12:27:03.270256 29794 consensus_queue.cc:260] T fb7d78e534634e0f992fc39b578ebd8d P d13c58ad8ac441c4a9b396eb04b1a476 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:27:03.275032 29794 ts_tablet_manager.cc:1434] T fb7d78e534634e0f992fc39b578ebd8d P d13c58ad8ac441c4a9b396eb04b1a476: Time spent starting tablet: real 0.025s	user 0.007s	sys 0.000s
I20260709 12:27:03.272405 30003 raft_consensus.cc:2804] T 01788883c8ea42989f246a3486111a7e P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:03.272217 30119 leader_election.cc:290] T 01788883c8ea42989f246a3486111a7e P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 1 election: Requested vote from peers c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895), 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033)
I20260709 12:27:03.292604 30003 raft_consensus.cc:697] T 01788883c8ea42989f246a3486111a7e P d73d92fb35234d16824c55689b5fce28 [term 1 LEADER]: Becoming Leader. State: Replica: d73d92fb35234d16824c55689b5fce28, State: Running, Role: LEADER
I20260709 12:27:03.293738 29792 tablet_bootstrap.cc:492] T 5be58d64d45e40d7981018b84a01b1ed P 584dac86183a41d9b3bea1c2085e62fa: No bootstrap required, opened a new log
I20260709 12:27:03.293573 30003 consensus_queue.cc:237] T 01788883c8ea42989f246a3486111a7e P d73d92fb35234d16824c55689b5fce28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:03.294360 29792 ts_tablet_manager.cc:1403] T 5be58d64d45e40d7981018b84a01b1ed P 584dac86183a41d9b3bea1c2085e62fa: Time spent bootstrapping tablet: real 0.482s	user 0.021s	sys 0.010s
I20260709 12:27:03.334179 29792 raft_consensus.cc:359] T 5be58d64d45e40d7981018b84a01b1ed P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:03.334901 29792 raft_consensus.cc:385] T 5be58d64d45e40d7981018b84a01b1ed P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:03.335177 29792 raft_consensus.cc:740] T 5be58d64d45e40d7981018b84a01b1ed P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 584dac86183a41d9b3bea1c2085e62fa, State: Initialized, Role: FOLLOWER
I20260709 12:27:03.335832 29792 consensus_queue.cc:260] T 5be58d64d45e40d7981018b84a01b1ed P 584dac86183a41d9b3bea1c2085e62fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:03.345227 29792 ts_tablet_manager.cc:1434] T 5be58d64d45e40d7981018b84a01b1ed P 584dac86183a41d9b3bea1c2085e62fa: Time spent starting tablet: real 0.050s	user 0.006s	sys 0.004s
I20260709 12:27:03.346684 29792 tablet_bootstrap.cc:492] T 5ee924f859f54f64908499bfac48ee62 P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap starting.
I20260709 12:27:03.352116 29793 tablet_bootstrap.cc:492] T d299fb38094e47e8aa3890ca74f6b366 P d73d92fb35234d16824c55689b5fce28: Bootstrap starting.
I20260709 12:27:03.371097 29790 tablet_bootstrap.cc:492] T 394378d078194257b9e04fc659c1f5d4 P c6610c00ffb948ce9c34f41dec3d49e8: No bootstrap required, opened a new log
I20260709 12:27:03.371034 29420 catalog_manager.cc:5697] T 5ee924f859f54f64908499bfac48ee62 P d13c58ad8ac441c4a9b396eb04b1a476 reported cstate change: term changed from 0 to 1, leader changed from <none> to d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67). New cstate: current_term: 1 leader_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } health_report { overall_health: HEALTHY } } }
I20260709 12:27:03.389690 29790 ts_tablet_manager.cc:1403] T 394378d078194257b9e04fc659c1f5d4 P c6610c00ffb948ce9c34f41dec3d49e8: Time spent bootstrapping tablet: real 0.169s	user 0.008s	sys 0.014s
I20260709 12:27:03.394385 30068 raft_consensus.cc:493] T 5be58d64d45e40d7981018b84a01b1ed P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:03.395439 29794 tablet_bootstrap.cc:492] T f3e17f0a8fc14095850e8138fd79b86e P d13c58ad8ac441c4a9b396eb04b1a476: Bootstrap starting.
I20260709 12:27:03.395181 30068 raft_consensus.cc:515] T 5be58d64d45e40d7981018b84a01b1ed P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:03.401608 30166 raft_consensus.cc:493] T 3617af2dee3f48cf85973b1c94cdc33a P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:03.402179 30166 raft_consensus.cc:515] T 3617af2dee3f48cf85973b1c94cdc33a P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:27:03.407335 30068 leader_election.cc:290] T 5be58d64d45e40d7981018b84a01b1ed P 584dac86183a41d9b3bea1c2085e62fa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895), d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825)
I20260709 12:27:03.409193 29592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5be58d64d45e40d7981018b84a01b1ed" candidate_uuid: "584dac86183a41d9b3bea1c2085e62fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d73d92fb35234d16824c55689b5fce28" is_pre_election: true
I20260709 12:27:03.409999 30166 leader_election.cc:290] T 3617af2dee3f48cf85973b1c94cdc33a P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209), c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895)
I20260709 12:27:03.410804 29744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5be58d64d45e40d7981018b84a01b1ed" candidate_uuid: "584dac86183a41d9b3bea1c2085e62fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" is_pre_election: true
I20260709 12:27:03.412796 29518 raft_consensus.cc:3060] T 5be58d64d45e40d7981018b84a01b1ed P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:03.414430 29670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3617af2dee3f48cf85973b1c94cdc33a" candidate_uuid: "d73d92fb35234d16824c55689b5fce28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" is_pre_election: true
W20260709 12:27:03.415699 29561 leader_election.cc:343] T 3617af2dee3f48cf85973b1c94cdc33a P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:03.415795 29486 leader_election.cc:304] T 5be58d64d45e40d7981018b84a01b1ed P 584dac86183a41d9b3bea1c2085e62fa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 584dac86183a41d9b3bea1c2085e62fa; no voters: c6610c00ffb948ce9c34f41dec3d49e8, d73d92fb35234d16824c55689b5fce28
I20260709 12:27:03.417145 29793 tablet_bootstrap.cc:654] T d299fb38094e47e8aa3890ca74f6b366 P d73d92fb35234d16824c55689b5fce28: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:03.417403 29744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3617af2dee3f48cf85973b1c94cdc33a" candidate_uuid: "d73d92fb35234d16824c55689b5fce28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" is_pre_election: true
W20260709 12:27:03.419804 29561 leader_election.cc:343] T 3617af2dee3f48cf85973b1c94cdc33a P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:03.420284 29561 leader_election.cc:304] T 3617af2dee3f48cf85973b1c94cdc33a P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d73d92fb35234d16824c55689b5fce28; no voters: c6610c00ffb948ce9c34f41dec3d49e8, d13c58ad8ac441c4a9b396eb04b1a476
I20260709 12:27:03.421289 30166 raft_consensus.cc:2749] T 3617af2dee3f48cf85973b1c94cdc33a P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:03.422046 29794 tablet_bootstrap.cc:654] T f3e17f0a8fc14095850e8138fd79b86e P d13c58ad8ac441c4a9b396eb04b1a476: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:03.422219 29790 raft_consensus.cc:359] T 394378d078194257b9e04fc659c1f5d4 P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:27:03.422933 29790 raft_consensus.cc:385] T 394378d078194257b9e04fc659c1f5d4 P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:03.423174 29790 raft_consensus.cc:740] T 394378d078194257b9e04fc659c1f5d4 P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6610c00ffb948ce9c34f41dec3d49e8, State: Initialized, Role: FOLLOWER
I20260709 12:27:03.441883 29790 consensus_queue.cc:260] T 394378d078194257b9e04fc659c1f5d4 P c6610c00ffb948ce9c34f41dec3d49e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:27:03.444250 29792 tablet_bootstrap.cc:654] T 5ee924f859f54f64908499bfac48ee62 P 584dac86183a41d9b3bea1c2085e62fa: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:03.441087 29417 catalog_manager.cc:5697] T 01788883c8ea42989f246a3486111a7e P d73d92fb35234d16824c55689b5fce28 reported cstate change: term changed from 0 to 1, leader changed from <none> to d73d92fb35234d16824c55689b5fce28 (127.16.29.66). New cstate: current_term: 1 leader_uuid: "d73d92fb35234d16824c55689b5fce28" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } health_report { overall_health: HEALTHY } } }
I20260709 12:27:03.479393 29790 ts_tablet_manager.cc:1434] T 394378d078194257b9e04fc659c1f5d4 P c6610c00ffb948ce9c34f41dec3d49e8: Time spent starting tablet: real 0.089s	user 0.001s	sys 0.005s
I20260709 12:27:03.481766 29418 catalog_manager.cc:5697] T d299fb38094e47e8aa3890ca74f6b366 P c6610c00ffb948ce9c34f41dec3d49e8 reported cstate change: term changed from 0 to 1, leader changed from <none> to c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68). New cstate: current_term: 1 leader_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } health_report { overall_health: HEALTHY } } }
I20260709 12:27:03.484490 29790 tablet_bootstrap.cc:492] T 01788883c8ea42989f246a3486111a7e P c6610c00ffb948ce9c34f41dec3d49e8: Bootstrap starting.
I20260709 12:27:03.488199 29793 tablet_bootstrap.cc:492] T d299fb38094e47e8aa3890ca74f6b366 P d73d92fb35234d16824c55689b5fce28: No bootstrap required, opened a new log
I20260709 12:27:03.490062 29793 ts_tablet_manager.cc:1403] T d299fb38094e47e8aa3890ca74f6b366 P d73d92fb35234d16824c55689b5fce28: Time spent bootstrapping tablet: real 0.138s	user 0.012s	sys 0.008s
I20260709 12:27:03.499063 29793 raft_consensus.cc:359] T d299fb38094e47e8aa3890ca74f6b366 P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:27:03.506089 29793 raft_consensus.cc:385] T d299fb38094e47e8aa3890ca74f6b366 P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:03.506489 29793 raft_consensus.cc:740] T d299fb38094e47e8aa3890ca74f6b366 P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d73d92fb35234d16824c55689b5fce28, State: Initialized, Role: FOLLOWER
I20260709 12:27:03.508410 29792 tablet_bootstrap.cc:492] T 5ee924f859f54f64908499bfac48ee62 P 584dac86183a41d9b3bea1c2085e62fa: No bootstrap required, opened a new log
I20260709 12:27:03.508651 29790 tablet_bootstrap.cc:654] T 01788883c8ea42989f246a3486111a7e P c6610c00ffb948ce9c34f41dec3d49e8: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:03.506886 29794 tablet_bootstrap.cc:492] T f3e17f0a8fc14095850e8138fd79b86e P d13c58ad8ac441c4a9b396eb04b1a476: No bootstrap required, opened a new log
I20260709 12:27:03.510474 29794 ts_tablet_manager.cc:1403] T f3e17f0a8fc14095850e8138fd79b86e P d13c58ad8ac441c4a9b396eb04b1a476: Time spent bootstrapping tablet: real 0.115s	user 0.025s	sys 0.021s
I20260709 12:27:03.509074 29792 ts_tablet_manager.cc:1403] T 5ee924f859f54f64908499bfac48ee62 P 584dac86183a41d9b3bea1c2085e62fa: Time spent bootstrapping tablet: real 0.163s	user 0.012s	sys 0.000s
I20260709 12:27:03.509127 29793 consensus_queue.cc:260] T d299fb38094e47e8aa3890ca74f6b366 P d73d92fb35234d16824c55689b5fce28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:27:03.521291 29793 ts_tablet_manager.cc:1434] T d299fb38094e47e8aa3890ca74f6b366 P d73d92fb35234d16824c55689b5fce28: Time spent starting tablet: real 0.026s	user 0.005s	sys 0.004s
I20260709 12:27:03.521108 29794 raft_consensus.cc:359] T f3e17f0a8fc14095850e8138fd79b86e P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:27:03.521894 29794 raft_consensus.cc:385] T f3e17f0a8fc14095850e8138fd79b86e P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:03.522197 29794 raft_consensus.cc:740] T f3e17f0a8fc14095850e8138fd79b86e P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d13c58ad8ac441c4a9b396eb04b1a476, State: Initialized, Role: FOLLOWER
I20260709 12:27:03.522789 29793 tablet_bootstrap.cc:492] T f3e17f0a8fc14095850e8138fd79b86e P d73d92fb35234d16824c55689b5fce28: Bootstrap starting.
I20260709 12:27:03.522876 29794 consensus_queue.cc:260] T f3e17f0a8fc14095850e8138fd79b86e P d13c58ad8ac441c4a9b396eb04b1a476 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:27:03.514956 29792 raft_consensus.cc:359] T 5ee924f859f54f64908499bfac48ee62 P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:27:03.524634 29792 raft_consensus.cc:385] T 5ee924f859f54f64908499bfac48ee62 P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:03.526051 30071 raft_consensus.cc:2749] T 5be58d64d45e40d7981018b84a01b1ed P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:03.528112 29792 raft_consensus.cc:740] T 5ee924f859f54f64908499bfac48ee62 P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 584dac86183a41d9b3bea1c2085e62fa, State: Initialized, Role: FOLLOWER
I20260709 12:27:03.534461 29794 ts_tablet_manager.cc:1434] T f3e17f0a8fc14095850e8138fd79b86e P d13c58ad8ac441c4a9b396eb04b1a476: Time spent starting tablet: real 0.024s	user 0.008s	sys 0.002s
I20260709 12:27:03.535620 29794 tablet_bootstrap.cc:492] T 3617af2dee3f48cf85973b1c94cdc33a P d13c58ad8ac441c4a9b396eb04b1a476: Bootstrap starting.
I20260709 12:27:03.535192 29792 consensus_queue.cc:260] T 5ee924f859f54f64908499bfac48ee62 P 584dac86183a41d9b3bea1c2085e62fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:27:03.542229 29792 ts_tablet_manager.cc:1434] T 5ee924f859f54f64908499bfac48ee62 P 584dac86183a41d9b3bea1c2085e62fa: Time spent starting tablet: real 0.030s	user 0.003s	sys 0.005s
I20260709 12:27:03.544298 29792 tablet_bootstrap.cc:492] T abec627108a84083884e64e1eb9dcda6 P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap starting.
I20260709 12:27:03.543008 29793 tablet_bootstrap.cc:654] T f3e17f0a8fc14095850e8138fd79b86e P d73d92fb35234d16824c55689b5fce28: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:03.549714 29790 tablet_bootstrap.cc:492] T 01788883c8ea42989f246a3486111a7e P c6610c00ffb948ce9c34f41dec3d49e8: No bootstrap required, opened a new log
I20260709 12:27:03.551367 29790 ts_tablet_manager.cc:1403] T 01788883c8ea42989f246a3486111a7e P c6610c00ffb948ce9c34f41dec3d49e8: Time spent bootstrapping tablet: real 0.067s	user 0.015s	sys 0.001s
I20260709 12:27:03.553555 29794 tablet_bootstrap.cc:654] T 3617af2dee3f48cf85973b1c94cdc33a P d13c58ad8ac441c4a9b396eb04b1a476: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:03.556006 29790 raft_consensus.cc:359] T 01788883c8ea42989f246a3486111a7e P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:03.557482 29790 raft_consensus.cc:385] T 01788883c8ea42989f246a3486111a7e P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:03.557725 29790 raft_consensus.cc:740] T 01788883c8ea42989f246a3486111a7e P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6610c00ffb948ce9c34f41dec3d49e8, State: Initialized, Role: FOLLOWER
I20260709 12:27:03.561468 29792 tablet_bootstrap.cc:654] T abec627108a84083884e64e1eb9dcda6 P 584dac86183a41d9b3bea1c2085e62fa: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:03.562191 29790 consensus_queue.cc:260] T 01788883c8ea42989f246a3486111a7e P c6610c00ffb948ce9c34f41dec3d49e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:03.570750 29794 tablet_bootstrap.cc:492] T 3617af2dee3f48cf85973b1c94cdc33a P d13c58ad8ac441c4a9b396eb04b1a476: No bootstrap required, opened a new log
I20260709 12:27:03.571925 29794 ts_tablet_manager.cc:1403] T 3617af2dee3f48cf85973b1c94cdc33a P d13c58ad8ac441c4a9b396eb04b1a476: Time spent bootstrapping tablet: real 0.036s	user 0.010s	sys 0.003s
I20260709 12:27:03.573504 29790 ts_tablet_manager.cc:1434] T 01788883c8ea42989f246a3486111a7e P c6610c00ffb948ce9c34f41dec3d49e8: Time spent starting tablet: real 0.022s	user 0.009s	sys 0.000s
I20260709 12:27:03.576723 29790 tablet_bootstrap.cc:492] T 6c8b2fa9f2c044ec9eec97c72b561042 P c6610c00ffb948ce9c34f41dec3d49e8: Bootstrap starting.
I20260709 12:27:03.576784 29793 tablet_bootstrap.cc:492] T f3e17f0a8fc14095850e8138fd79b86e P d73d92fb35234d16824c55689b5fce28: No bootstrap required, opened a new log
I20260709 12:27:03.577291 29793 ts_tablet_manager.cc:1403] T f3e17f0a8fc14095850e8138fd79b86e P d73d92fb35234d16824c55689b5fce28: Time spent bootstrapping tablet: real 0.055s	user 0.012s	sys 0.007s
I20260709 12:27:03.574579 29794 raft_consensus.cc:359] T 3617af2dee3f48cf85973b1c94cdc33a P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:27:03.580235 29794 raft_consensus.cc:385] T 3617af2dee3f48cf85973b1c94cdc33a P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:03.580542 29794 raft_consensus.cc:740] T 3617af2dee3f48cf85973b1c94cdc33a P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d13c58ad8ac441c4a9b396eb04b1a476, State: Initialized, Role: FOLLOWER
I20260709 12:27:03.581209 29794 consensus_queue.cc:260] T 3617af2dee3f48cf85973b1c94cdc33a P d13c58ad8ac441c4a9b396eb04b1a476 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:27:03.582454 29793 raft_consensus.cc:359] T f3e17f0a8fc14095850e8138fd79b86e P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:27:03.582883 29792 tablet_bootstrap.cc:492] T abec627108a84083884e64e1eb9dcda6 P 584dac86183a41d9b3bea1c2085e62fa: No bootstrap required, opened a new log
I20260709 12:27:03.583158 29793 raft_consensus.cc:385] T f3e17f0a8fc14095850e8138fd79b86e P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:03.583494 29792 ts_tablet_manager.cc:1403] T abec627108a84083884e64e1eb9dcda6 P 584dac86183a41d9b3bea1c2085e62fa: Time spent bootstrapping tablet: real 0.039s	user 0.009s	sys 0.008s
I20260709 12:27:03.583487 29793 raft_consensus.cc:740] T f3e17f0a8fc14095850e8138fd79b86e P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d73d92fb35234d16824c55689b5fce28, State: Initialized, Role: FOLLOWER
I20260709 12:27:03.584594 29793 consensus_queue.cc:260] T f3e17f0a8fc14095850e8138fd79b86e P d73d92fb35234d16824c55689b5fce28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:27:03.586938 30151 raft_consensus.cc:493] T f76101eb30f14b98a9aa8c41e0d6f831 P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:03.587455 30151 raft_consensus.cc:515] T f76101eb30f14b98a9aa8c41e0d6f831 P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:27:03.589881 30151 leader_election.cc:290] T f76101eb30f14b98a9aa8c41e0d6f831 P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209), 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033)
I20260709 12:27:03.590308 29794 ts_tablet_manager.cc:1434] T 3617af2dee3f48cf85973b1c94cdc33a P d13c58ad8ac441c4a9b396eb04b1a476: Time spent starting tablet: real 0.018s	user 0.005s	sys 0.000s
I20260709 12:27:03.591970 29666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f76101eb30f14b98a9aa8c41e0d6f831" candidate_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" is_pre_election: true
W20260709 12:27:03.593312 29709 leader_election.cc:343] T f76101eb30f14b98a9aa8c41e0d6f831 P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:03.594163 29518 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f76101eb30f14b98a9aa8c41e0d6f831" candidate_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "584dac86183a41d9b3bea1c2085e62fa" is_pre_election: true
I20260709 12:27:03.597059 29794 tablet_bootstrap.cc:492] T ef20d7124e7a44aa97a219ee194f4b8b P d13c58ad8ac441c4a9b396eb04b1a476: Bootstrap starting.
W20260709 12:27:03.597190 29708 leader_election.cc:343] T f76101eb30f14b98a9aa8c41e0d6f831 P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:03.597693 29708 leader_election.cc:304] T f76101eb30f14b98a9aa8c41e0d6f831 P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c6610c00ffb948ce9c34f41dec3d49e8; no voters: 584dac86183a41d9b3bea1c2085e62fa, d13c58ad8ac441c4a9b396eb04b1a476
I20260709 12:27:03.602742 30112 raft_consensus.cc:2749] T f76101eb30f14b98a9aa8c41e0d6f831 P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:03.605782 29792 raft_consensus.cc:359] T abec627108a84083884e64e1eb9dcda6 P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:27:03.606053 29794 tablet_bootstrap.cc:654] T ef20d7124e7a44aa97a219ee194f4b8b P d13c58ad8ac441c4a9b396eb04b1a476: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:03.611701 29792 raft_consensus.cc:385] T abec627108a84083884e64e1eb9dcda6 P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:03.612171 29792 raft_consensus.cc:740] T abec627108a84083884e64e1eb9dcda6 P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 584dac86183a41d9b3bea1c2085e62fa, State: Initialized, Role: FOLLOWER
I20260709 12:27:03.612828 30119 raft_consensus.cc:493] T abec627108a84083884e64e1eb9dcda6 P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:03.613308 30119 raft_consensus.cc:515] T abec627108a84083884e64e1eb9dcda6 P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:27:03.613982 29790 tablet_bootstrap.cc:654] T 6c8b2fa9f2c044ec9eec97c72b561042 P c6610c00ffb948ce9c34f41dec3d49e8: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:03.615399 30119 leader_election.cc:290] T abec627108a84083884e64e1eb9dcda6 P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033), d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209)
I20260709 12:27:03.618793 29666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abec627108a84083884e64e1eb9dcda6" candidate_uuid: "d73d92fb35234d16824c55689b5fce28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" is_pre_election: true
I20260709 12:27:03.620615 29518 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abec627108a84083884e64e1eb9dcda6" candidate_uuid: "d73d92fb35234d16824c55689b5fce28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "584dac86183a41d9b3bea1c2085e62fa" is_pre_election: true
I20260709 12:27:03.606916 29793 ts_tablet_manager.cc:1434] T f3e17f0a8fc14095850e8138fd79b86e P d73d92fb35234d16824c55689b5fce28: Time spent starting tablet: real 0.029s	user 0.006s	sys 0.003s
W20260709 12:27:03.621353 29561 leader_election.cc:343] T abec627108a84083884e64e1eb9dcda6 P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:03.622349 29793 tablet_bootstrap.cc:492] T 394378d078194257b9e04fc659c1f5d4 P d73d92fb35234d16824c55689b5fce28: Bootstrap starting.
I20260709 12:27:03.628440 29793 tablet_bootstrap.cc:654] T 394378d078194257b9e04fc659c1f5d4 P d73d92fb35234d16824c55689b5fce28: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:03.640249 29793 tablet_bootstrap.cc:492] T 394378d078194257b9e04fc659c1f5d4 P d73d92fb35234d16824c55689b5fce28: No bootstrap required, opened a new log
I20260709 12:27:03.640849 29793 ts_tablet_manager.cc:1403] T 394378d078194257b9e04fc659c1f5d4 P d73d92fb35234d16824c55689b5fce28: Time spent bootstrapping tablet: real 0.019s	user 0.006s	sys 0.011s
I20260709 12:27:03.644101 29793 raft_consensus.cc:359] T 394378d078194257b9e04fc659c1f5d4 P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:27:03.645036 29793 raft_consensus.cc:385] T 394378d078194257b9e04fc659c1f5d4 P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:03.645378 29793 raft_consensus.cc:740] T 394378d078194257b9e04fc659c1f5d4 P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d73d92fb35234d16824c55689b5fce28, State: Initialized, Role: FOLLOWER
I20260709 12:27:03.646214 29793 consensus_queue.cc:260] T 394378d078194257b9e04fc659c1f5d4 P d73d92fb35234d16824c55689b5fce28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:27:03.646768 29790 tablet_bootstrap.cc:492] T 6c8b2fa9f2c044ec9eec97c72b561042 P c6610c00ffb948ce9c34f41dec3d49e8: No bootstrap required, opened a new log
I20260709 12:27:03.647294 29790 ts_tablet_manager.cc:1403] T 6c8b2fa9f2c044ec9eec97c72b561042 P c6610c00ffb948ce9c34f41dec3d49e8: Time spent bootstrapping tablet: real 0.071s	user 0.010s	sys 0.010s
I20260709 12:27:03.648427 29793 ts_tablet_manager.cc:1434] T 394378d078194257b9e04fc659c1f5d4 P d73d92fb35234d16824c55689b5fce28: Time spent starting tablet: real 0.007s	user 0.006s	sys 0.000s
I20260709 12:27:03.649863 29793 tablet_bootstrap.cc:492] T f8c01e05981747a395e5483b551297dd P d73d92fb35234d16824c55689b5fce28: Bootstrap starting.
I20260709 12:27:03.650661 29792 consensus_queue.cc:260] T abec627108a84083884e64e1eb9dcda6 P 584dac86183a41d9b3bea1c2085e62fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:27:03.651603 29518 raft_consensus.cc:2468] T abec627108a84083884e64e1eb9dcda6 P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d73d92fb35234d16824c55689b5fce28 in term 0.
I20260709 12:27:03.655367 29560 leader_election.cc:304] T abec627108a84083884e64e1eb9dcda6 P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 584dac86183a41d9b3bea1c2085e62fa, d73d92fb35234d16824c55689b5fce28; no voters: d13c58ad8ac441c4a9b396eb04b1a476
I20260709 12:27:03.656494 29793 tablet_bootstrap.cc:654] T f8c01e05981747a395e5483b551297dd P d73d92fb35234d16824c55689b5fce28: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:03.662935 29792 ts_tablet_manager.cc:1434] T abec627108a84083884e64e1eb9dcda6 P 584dac86183a41d9b3bea1c2085e62fa: Time spent starting tablet: real 0.079s	user 0.006s	sys 0.003s
I20260709 12:27:03.664659 29792 tablet_bootstrap.cc:492] T fb7d78e534634e0f992fc39b578ebd8d P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap starting.
I20260709 12:27:03.650013 29790 raft_consensus.cc:359] T 6c8b2fa9f2c044ec9eec97c72b561042 P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:03.665932 29790 raft_consensus.cc:385] T 6c8b2fa9f2c044ec9eec97c72b561042 P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:03.666196 29790 raft_consensus.cc:740] T 6c8b2fa9f2c044ec9eec97c72b561042 P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6610c00ffb948ce9c34f41dec3d49e8, State: Initialized, Role: FOLLOWER
I20260709 12:27:03.667900 30119 raft_consensus.cc:2804] T abec627108a84083884e64e1eb9dcda6 P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:03.668458 30119 raft_consensus.cc:493] T abec627108a84083884e64e1eb9dcda6 P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:03.668936 30119 raft_consensus.cc:3060] T abec627108a84083884e64e1eb9dcda6 P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:03.668479 29790 consensus_queue.cc:260] T 6c8b2fa9f2c044ec9eec97c72b561042 P c6610c00ffb948ce9c34f41dec3d49e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:03.671011 29790 ts_tablet_manager.cc:1434] T 6c8b2fa9f2c044ec9eec97c72b561042 P c6610c00ffb948ce9c34f41dec3d49e8: Time spent starting tablet: real 0.023s	user 0.006s	sys 0.001s
I20260709 12:27:03.674719 29790 tablet_bootstrap.cc:492] T f8c01e05981747a395e5483b551297dd P c6610c00ffb948ce9c34f41dec3d49e8: Bootstrap starting.
I20260709 12:27:03.682765 29793 tablet_bootstrap.cc:492] T f8c01e05981747a395e5483b551297dd P d73d92fb35234d16824c55689b5fce28: No bootstrap required, opened a new log
I20260709 12:27:03.688895 29790 tablet_bootstrap.cc:654] T f8c01e05981747a395e5483b551297dd P c6610c00ffb948ce9c34f41dec3d49e8: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:03.691255 29792 tablet_bootstrap.cc:654] T fb7d78e534634e0f992fc39b578ebd8d P 584dac86183a41d9b3bea1c2085e62fa: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:03.694437 29790 tablet_bootstrap.cc:492] T f8c01e05981747a395e5483b551297dd P c6610c00ffb948ce9c34f41dec3d49e8: No bootstrap required, opened a new log
I20260709 12:27:03.694967 29790 ts_tablet_manager.cc:1403] T f8c01e05981747a395e5483b551297dd P c6610c00ffb948ce9c34f41dec3d49e8: Time spent bootstrapping tablet: real 0.021s	user 0.007s	sys 0.003s
I20260709 12:27:03.695624 29793 ts_tablet_manager.cc:1403] T f8c01e05981747a395e5483b551297dd P d73d92fb35234d16824c55689b5fce28: Time spent bootstrapping tablet: real 0.046s	user 0.016s	sys 0.012s
I20260709 12:27:03.698472 30119 raft_consensus.cc:515] T abec627108a84083884e64e1eb9dcda6 P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:27:03.700115 29794 tablet_bootstrap.cc:492] T ef20d7124e7a44aa97a219ee194f4b8b P d13c58ad8ac441c4a9b396eb04b1a476: No bootstrap required, opened a new log
I20260709 12:27:03.700662 29794 ts_tablet_manager.cc:1403] T ef20d7124e7a44aa97a219ee194f4b8b P d13c58ad8ac441c4a9b396eb04b1a476: Time spent bootstrapping tablet: real 0.104s	user 0.025s	sys 0.017s
I20260709 12:27:03.703572 30166 consensus_queue.cc:1048] T 01788883c8ea42989f246a3486111a7e P d73d92fb35234d16824c55689b5fce28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:03.704632 29790 raft_consensus.cc:359] T f8c01e05981747a395e5483b551297dd P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:03.705600 29790 raft_consensus.cc:385] T f8c01e05981747a395e5483b551297dd P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:03.705945 29790 raft_consensus.cc:740] T f8c01e05981747a395e5483b551297dd P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6610c00ffb948ce9c34f41dec3d49e8, State: Initialized, Role: FOLLOWER
I20260709 12:27:03.706842 29790 consensus_queue.cc:260] T f8c01e05981747a395e5483b551297dd P c6610c00ffb948ce9c34f41dec3d49e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:03.709712 29793 raft_consensus.cc:359] T f8c01e05981747a395e5483b551297dd P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:03.710297 29794 raft_consensus.cc:359] T ef20d7124e7a44aa97a219ee194f4b8b P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:27:03.710685 29793 raft_consensus.cc:385] T f8c01e05981747a395e5483b551297dd P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:03.711071 29794 raft_consensus.cc:385] T ef20d7124e7a44aa97a219ee194f4b8b P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:03.711078 29793 raft_consensus.cc:740] T f8c01e05981747a395e5483b551297dd P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d73d92fb35234d16824c55689b5fce28, State: Initialized, Role: FOLLOWER
I20260709 12:27:03.711542 29794 raft_consensus.cc:740] T ef20d7124e7a44aa97a219ee194f4b8b P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d13c58ad8ac441c4a9b396eb04b1a476, State: Initialized, Role: FOLLOWER
I20260709 12:27:03.712278 29794 consensus_queue.cc:260] T ef20d7124e7a44aa97a219ee194f4b8b P d13c58ad8ac441c4a9b396eb04b1a476 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:27:03.712257 29793 consensus_queue.cc:260] T f8c01e05981747a395e5483b551297dd P d73d92fb35234d16824c55689b5fce28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:03.714725 29794 ts_tablet_manager.cc:1434] T ef20d7124e7a44aa97a219ee194f4b8b P d13c58ad8ac441c4a9b396eb04b1a476: Time spent starting tablet: real 0.014s	user 0.006s	sys 0.000s
I20260709 12:27:03.715462 29793 ts_tablet_manager.cc:1434] T f8c01e05981747a395e5483b551297dd P d73d92fb35234d16824c55689b5fce28: Time spent starting tablet: real 0.019s	user 0.006s	sys 0.000s
I20260709 12:27:03.718571 29790 ts_tablet_manager.cc:1434] T f8c01e05981747a395e5483b551297dd P c6610c00ffb948ce9c34f41dec3d49e8: Time spent starting tablet: real 0.023s	user 0.008s	sys 0.000s
I20260709 12:27:03.719635 29793 tablet_bootstrap.cc:492] T 137e03144e754f6e9da4f0544513d27c P d73d92fb35234d16824c55689b5fce28: Bootstrap starting.
I20260709 12:27:03.723260 30138 raft_consensus.cc:493] T f3e17f0a8fc14095850e8138fd79b86e P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:03.723961 30138 raft_consensus.cc:515] T f3e17f0a8fc14095850e8138fd79b86e P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:27:03.716459 29794 tablet_bootstrap.cc:492] T f76101eb30f14b98a9aa8c41e0d6f831 P d13c58ad8ac441c4a9b396eb04b1a476: Bootstrap starting.
I20260709 12:27:03.730392 30112 raft_consensus.cc:493] T 394378d078194257b9e04fc659c1f5d4 P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:03.727910 30003 raft_consensus.cc:493] T f3e17f0a8fc14095850e8138fd79b86e P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:03.741110 30112 raft_consensus.cc:515] T 394378d078194257b9e04fc659c1f5d4 P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:27:03.741268 30003 raft_consensus.cc:515] T f3e17f0a8fc14095850e8138fd79b86e P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:27:03.743746 30112 leader_election.cc:290] T 394378d078194257b9e04fc659c1f5d4 P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825), d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209)
I20260709 12:27:03.728214 29790 tablet_bootstrap.cc:492] T 3617af2dee3f48cf85973b1c94cdc33a P c6610c00ffb948ce9c34f41dec3d49e8: Bootstrap starting.
I20260709 12:27:03.753270 30043 raft_consensus.cc:493] T fb7d78e534634e0f992fc39b578ebd8d P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:03.762549 29520 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f3e17f0a8fc14095850e8138fd79b86e" candidate_uuid: "d73d92fb35234d16824c55689b5fce28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "584dac86183a41d9b3bea1c2085e62fa" is_pre_election: true
I20260709 12:27:03.756852 29595 raft_consensus.cc:3060] T d299fb38094e47e8aa3890ca74f6b366 P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:03.753028 29670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "394378d078194257b9e04fc659c1f5d4" candidate_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" is_pre_election: true
W20260709 12:27:03.766769 29560 leader_election.cc:343] T f3e17f0a8fc14095850e8138fd79b86e P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:03.750782 29521 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f3e17f0a8fc14095850e8138fd79b86e" candidate_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "584dac86183a41d9b3bea1c2085e62fa" is_pre_election: true
I20260709 12:27:03.764442 29668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f3e17f0a8fc14095850e8138fd79b86e" candidate_uuid: "d73d92fb35234d16824c55689b5fce28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" is_pre_election: true
W20260709 12:27:03.768239 29709 leader_election.cc:343] T 394378d078194257b9e04fc659c1f5d4 P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:03.768597 29668 raft_consensus.cc:2468] T f3e17f0a8fc14095850e8138fd79b86e P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d73d92fb35234d16824c55689b5fce28 in term 0.
I20260709 12:27:03.767689 30043 raft_consensus.cc:515] T fb7d78e534634e0f992fc39b578ebd8d P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
W20260709 12:27:03.770080 29634 leader_election.cc:343] T f3e17f0a8fc14095850e8138fd79b86e P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:03.762645 30003 leader_election.cc:290] T f3e17f0a8fc14095850e8138fd79b86e P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033), d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209)
I20260709 12:27:03.771028 30043 leader_election.cc:290] T fb7d78e534634e0f992fc39b578ebd8d P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895), 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033)
I20260709 12:27:03.772789 29742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb7d78e534634e0f992fc39b578ebd8d" candidate_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" is_pre_election: true
I20260709 12:27:03.752605 29590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "394378d078194257b9e04fc659c1f5d4" candidate_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d73d92fb35234d16824c55689b5fce28" is_pre_election: true
I20260709 12:27:03.773459 29742 raft_consensus.cc:2468] T fb7d78e534634e0f992fc39b578ebd8d P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d13c58ad8ac441c4a9b396eb04b1a476 in term 0.
I20260709 12:27:03.773809 29590 raft_consensus.cc:2468] T 394378d078194257b9e04fc659c1f5d4 P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6610c00ffb948ce9c34f41dec3d49e8 in term 0.
I20260709 12:27:03.726214 30138 leader_election.cc:290] T f3e17f0a8fc14095850e8138fd79b86e P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033), d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825)
I20260709 12:27:03.774791 29635 leader_election.cc:304] T fb7d78e534634e0f992fc39b578ebd8d P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c6610c00ffb948ce9c34f41dec3d49e8, d13c58ad8ac441c4a9b396eb04b1a476; no voters: 
I20260709 12:27:03.752218 29592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f3e17f0a8fc14095850e8138fd79b86e" candidate_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d73d92fb35234d16824c55689b5fce28" is_pre_election: true
I20260709 12:27:03.775768 29592 raft_consensus.cc:2468] T f3e17f0a8fc14095850e8138fd79b86e P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d13c58ad8ac441c4a9b396eb04b1a476 in term 0.
I20260709 12:27:03.776466 29521 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abec627108a84083884e64e1eb9dcda6" candidate_uuid: "d73d92fb35234d16824c55689b5fce28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "584dac86183a41d9b3bea1c2085e62fa"
I20260709 12:27:03.777797 29521 raft_consensus.cc:3060] T abec627108a84083884e64e1eb9dcda6 P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:03.779024 29520 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb7d78e534634e0f992fc39b578ebd8d" candidate_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "584dac86183a41d9b3bea1c2085e62fa" is_pre_election: true
I20260709 12:27:03.779579 30119 leader_election.cc:290] T abec627108a84083884e64e1eb9dcda6 P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 1 election: Requested vote from peers 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033), d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209)
I20260709 12:27:03.727293 29793 tablet_bootstrap.cc:654] T 137e03144e754f6e9da4f0544513d27c P d73d92fb35234d16824c55689b5fce28: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:03.780745 30046 consensus_queue.cc:1048] T d299fb38094e47e8aa3890ca74f6b366 P c6610c00ffb948ce9c34f41dec3d49e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:27:03.781659 29708 leader_election.cc:304] T 394378d078194257b9e04fc659c1f5d4 P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c6610c00ffb948ce9c34f41dec3d49e8, d73d92fb35234d16824c55689b5fce28; no voters: d13c58ad8ac441c4a9b396eb04b1a476
I20260709 12:27:03.782490 30046 raft_consensus.cc:2804] T 394378d078194257b9e04fc659c1f5d4 P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:03.782878 30046 raft_consensus.cc:493] T 394378d078194257b9e04fc659c1f5d4 P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:03.783823 30046 raft_consensus.cc:3060] T 394378d078194257b9e04fc659c1f5d4 P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:03.784713 30138 raft_consensus.cc:2804] T fb7d78e534634e0f992fc39b578ebd8d P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:03.785181 30138 raft_consensus.cc:493] T fb7d78e534634e0f992fc39b578ebd8d P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:03.785586 30138 raft_consensus.cc:3060] T fb7d78e534634e0f992fc39b578ebd8d P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:03.790902 29634 leader_election.cc:304] T f3e17f0a8fc14095850e8138fd79b86e P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d13c58ad8ac441c4a9b396eb04b1a476, d73d92fb35234d16824c55689b5fce28; no voters: 584dac86183a41d9b3bea1c2085e62fa
W20260709 12:27:03.792213 29634 leader_election.cc:343] T fb7d78e534634e0f992fc39b578ebd8d P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:03.793432 30072 raft_consensus.cc:2804] T f3e17f0a8fc14095850e8138fd79b86e P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:03.793640 29790 tablet_bootstrap.cc:654] T 3617af2dee3f48cf85973b1c94cdc33a P c6610c00ffb948ce9c34f41dec3d49e8: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:03.793923 30072 raft_consensus.cc:493] T f3e17f0a8fc14095850e8138fd79b86e P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:03.794344 30072 raft_consensus.cc:3060] T f3e17f0a8fc14095850e8138fd79b86e P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:03.797137 30153 raft_consensus.cc:493] T 3617af2dee3f48cf85973b1c94cdc33a P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:03.797632 30153 raft_consensus.cc:515] T 3617af2dee3f48cf85973b1c94cdc33a P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:27:03.801865 29521 raft_consensus.cc:2468] T abec627108a84083884e64e1eb9dcda6 P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d73d92fb35234d16824c55689b5fce28 in term 1.
I20260709 12:27:03.802856 29560 leader_election.cc:304] T abec627108a84083884e64e1eb9dcda6 P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 584dac86183a41d9b3bea1c2085e62fa, d73d92fb35234d16824c55689b5fce28; no voters: 
I20260709 12:27:03.804302 30003 raft_consensus.cc:2804] T abec627108a84083884e64e1eb9dcda6 P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:03.804761 30003 raft_consensus.cc:697] T abec627108a84083884e64e1eb9dcda6 P d73d92fb35234d16824c55689b5fce28 [term 1 LEADER]: Becoming Leader. State: Replica: d73d92fb35234d16824c55689b5fce28, State: Running, Role: LEADER
I20260709 12:27:03.804627 29794 tablet_bootstrap.cc:654] T f76101eb30f14b98a9aa8c41e0d6f831 P d13c58ad8ac441c4a9b396eb04b1a476: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:03.805605 30003 consensus_queue.cc:237] T abec627108a84083884e64e1eb9dcda6 P d73d92fb35234d16824c55689b5fce28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:27:03.809595 30072 raft_consensus.cc:515] T f3e17f0a8fc14095850e8138fd79b86e P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:27:03.799705 30153 leader_election.cc:290] T 3617af2dee3f48cf85973b1c94cdc33a P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825), c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895)
I20260709 12:27:03.811772 29744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3617af2dee3f48cf85973b1c94cdc33a" candidate_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" is_pre_election: true
I20260709 12:27:03.829386 29518 raft_consensus.cc:3060] T 5ee924f859f54f64908499bfac48ee62 P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:03.791766 30046 raft_consensus.cc:515] T 394378d078194257b9e04fc659c1f5d4 P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:27:03.814800 29561 leader_election.cc:304] T f3e17f0a8fc14095850e8138fd79b86e P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d13c58ad8ac441c4a9b396eb04b1a476, d73d92fb35234d16824c55689b5fce28; no voters: 584dac86183a41d9b3bea1c2085e62fa
I20260709 12:27:03.817970 30043 raft_consensus.cc:493] T ef20d7124e7a44aa97a219ee194f4b8b P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:03.820464 29794 tablet_bootstrap.cc:492] T f76101eb30f14b98a9aa8c41e0d6f831 P d13c58ad8ac441c4a9b396eb04b1a476: No bootstrap required, opened a new log
I20260709 12:27:03.831264 29794 ts_tablet_manager.cc:1403] T f76101eb30f14b98a9aa8c41e0d6f831 P d13c58ad8ac441c4a9b396eb04b1a476: Time spent bootstrapping tablet: real 0.115s	user 0.006s	sys 0.007s
I20260709 12:27:03.831512 30046 leader_election.cc:290] T 394378d078194257b9e04fc659c1f5d4 P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 1 election: Requested vote from peers d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825), d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209)
I20260709 12:27:03.832604 29670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "394378d078194257b9e04fc659c1f5d4" candidate_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d13c58ad8ac441c4a9b396eb04b1a476"
W20260709 12:27:03.832892 29635 leader_election.cc:343] T 3617af2dee3f48cf85973b1c94cdc33a P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:27:03.833820 29709 leader_election.cc:343] T 394378d078194257b9e04fc659c1f5d4 P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:03.831084 30043 raft_consensus.cc:515] T ef20d7124e7a44aa97a219ee194f4b8b P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:27:03.834025 29794 raft_consensus.cc:359] T f76101eb30f14b98a9aa8c41e0d6f831 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:27:03.822644 29666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abec627108a84083884e64e1eb9dcda6" candidate_uuid: "d73d92fb35234d16824c55689b5fce28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d13c58ad8ac441c4a9b396eb04b1a476"
I20260709 12:27:03.835727 29794 raft_consensus.cc:385] T f76101eb30f14b98a9aa8c41e0d6f831 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:03.836066 29794 raft_consensus.cc:740] T f76101eb30f14b98a9aa8c41e0d6f831 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d13c58ad8ac441c4a9b396eb04b1a476, State: Initialized, Role: FOLLOWER
I20260709 12:27:03.836205 30043 leader_election.cc:290] T ef20d7124e7a44aa97a219ee194f4b8b P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895), 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033)
I20260709 12:27:03.836728 29794 consensus_queue.cc:260] T f76101eb30f14b98a9aa8c41e0d6f831 P d13c58ad8ac441c4a9b396eb04b1a476 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:27:03.837354 30126 raft_consensus.cc:2804] T f3e17f0a8fc14095850e8138fd79b86e P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:03.837762 30126 raft_consensus.cc:493] T f3e17f0a8fc14095850e8138fd79b86e P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:03.837931 29744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef20d7124e7a44aa97a219ee194f4b8b" candidate_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" is_pre_election: true
I20260709 12:27:03.838172 30126 raft_consensus.cc:3060] T f3e17f0a8fc14095850e8138fd79b86e P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:03.839365 29794 ts_tablet_manager.cc:1434] T f76101eb30f14b98a9aa8c41e0d6f831 P d13c58ad8ac441c4a9b396eb04b1a476: Time spent starting tablet: real 0.008s	user 0.005s	sys 0.001s
W20260709 12:27:03.839502 29635 leader_election.cc:343] T ef20d7124e7a44aa97a219ee194f4b8b P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:03.840441 29794 tablet_bootstrap.cc:492] T abec627108a84083884e64e1eb9dcda6 P d13c58ad8ac441c4a9b396eb04b1a476: Bootstrap starting.
I20260709 12:27:03.841529 29521 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef20d7124e7a44aa97a219ee194f4b8b" candidate_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "584dac86183a41d9b3bea1c2085e62fa" is_pre_election: true
I20260709 12:27:03.842629 29417 catalog_manager.cc:5697] T abec627108a84083884e64e1eb9dcda6 P d73d92fb35234d16824c55689b5fce28 reported cstate change: term changed from 0 to 1, leader changed from <none> to d73d92fb35234d16824c55689b5fce28 (127.16.29.66). New cstate: current_term: 1 leader_uuid: "d73d92fb35234d16824c55689b5fce28" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } health_report { overall_health: UNKNOWN } } }
W20260709 12:27:03.844449 29634 leader_election.cc:343] T ef20d7124e7a44aa97a219ee194f4b8b P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:03.852182 30046 consensus_queue.cc:1048] T d299fb38094e47e8aa3890ca74f6b366 P c6610c00ffb948ce9c34f41dec3d49e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:27:03.851037 29634 leader_election.cc:304] T ef20d7124e7a44aa97a219ee194f4b8b P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d13c58ad8ac441c4a9b396eb04b1a476; no voters: 584dac86183a41d9b3bea1c2085e62fa, c6610c00ffb948ce9c34f41dec3d49e8
I20260709 12:27:03.854898 30153 raft_consensus.cc:2749] T ef20d7124e7a44aa97a219ee194f4b8b P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:03.854904 29595 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "394378d078194257b9e04fc659c1f5d4" candidate_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d73d92fb35234d16824c55689b5fce28"
I20260709 12:27:03.856299 29595 raft_consensus.cc:3060] T 394378d078194257b9e04fc659c1f5d4 P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:03.865069 30166 raft_consensus.cc:493] T f8c01e05981747a395e5483b551297dd P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:03.865651 30166 raft_consensus.cc:515] T f8c01e05981747a395e5483b551297dd P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:03.866175 30046 raft_consensus.cc:493] T 6c8b2fa9f2c044ec9eec97c72b561042 P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:03.870764 30166 leader_election.cc:290] T f8c01e05981747a395e5483b551297dd P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895), 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033)
I20260709 12:27:03.875372 29590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3617af2dee3f48cf85973b1c94cdc33a" candidate_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d73d92fb35234d16824c55689b5fce28" is_pre_election: true
I20260709 12:27:03.876192 29590 raft_consensus.cc:2468] T 3617af2dee3f48cf85973b1c94cdc33a P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d13c58ad8ac441c4a9b396eb04b1a476 in term 0.
W20260709 12:27:03.876614 29561 leader_election.cc:343] T abec627108a84083884e64e1eb9dcda6 P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:03.878520 29742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8c01e05981747a395e5483b551297dd" candidate_uuid: "d73d92fb35234d16824c55689b5fce28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" is_pre_election: true
I20260709 12:27:03.879310 29742 raft_consensus.cc:2468] T f8c01e05981747a395e5483b551297dd P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d73d92fb35234d16824c55689b5fce28 in term 0.
I20260709 12:27:03.881156 30046 raft_consensus.cc:515] T 6c8b2fa9f2c044ec9eec97c72b561042 P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:03.884719 29520 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c8b2fa9f2c044ec9eec97c72b561042" candidate_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "584dac86183a41d9b3bea1c2085e62fa" is_pre_election: true
I20260709 12:27:03.886500 30046 leader_election.cc:290] T 6c8b2fa9f2c044ec9eec97c72b561042 P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033), d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825)
W20260709 12:27:03.888849 29708 leader_election.cc:343] T 6c8b2fa9f2c044ec9eec97c72b561042 P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:03.845975 30126 raft_consensus.cc:515] T f3e17f0a8fc14095850e8138fd79b86e P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:27:03.847141 29794 tablet_bootstrap.cc:654] T abec627108a84083884e64e1eb9dcda6 P d13c58ad8ac441c4a9b396eb04b1a476: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:03.896191 29595 raft_consensus.cc:2468] T 394378d078194257b9e04fc659c1f5d4 P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6610c00ffb948ce9c34f41dec3d49e8 in term 1.
I20260709 12:27:03.848853 29744 raft_consensus.cc:3060] T 01788883c8ea42989f246a3486111a7e P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:03.849257 29790 tablet_bootstrap.cc:492] T 3617af2dee3f48cf85973b1c94cdc33a P c6610c00ffb948ce9c34f41dec3d49e8: No bootstrap required, opened a new log
I20260709 12:27:03.900259 29790 ts_tablet_manager.cc:1403] T 3617af2dee3f48cf85973b1c94cdc33a P c6610c00ffb948ce9c34f41dec3d49e8: Time spent bootstrapping tablet: real 0.172s	user 0.015s	sys 0.002s
I20260709 12:27:03.899691 29521 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8c01e05981747a395e5483b551297dd" candidate_uuid: "d73d92fb35234d16824c55689b5fce28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "584dac86183a41d9b3bea1c2085e62fa" is_pre_election: true
W20260709 12:27:03.902180 29560 leader_election.cc:343] T f8c01e05981747a395e5483b551297dd P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:27:03.903234 29634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:35033, user_credentials={real_user=slave}} blocked reactor thread for 58811.5us
I20260709 12:27:03.906193 29521 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f3e17f0a8fc14095850e8138fd79b86e" candidate_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "584dac86183a41d9b3bea1c2085e62fa"
I20260709 12:27:03.908490 30138 raft_consensus.cc:515] T fb7d78e534634e0f992fc39b578ebd8d P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
W20260709 12:27:03.916302 29634 leader_election.cc:343] T f3e17f0a8fc14095850e8138fd79b86e P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:03.921372 29793 tablet_bootstrap.cc:492] T 137e03144e754f6e9da4f0544513d27c P d73d92fb35234d16824c55689b5fce28: No bootstrap required, opened a new log
I20260709 12:27:03.922204 29793 ts_tablet_manager.cc:1403] T 137e03144e754f6e9da4f0544513d27c P d73d92fb35234d16824c55689b5fce28: Time spent bootstrapping tablet: real 0.203s	user 0.025s	sys 0.022s
I20260709 12:27:03.903303 29790 raft_consensus.cc:359] T 3617af2dee3f48cf85973b1c94cdc33a P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:27:03.923022 29790 raft_consensus.cc:385] T 3617af2dee3f48cf85973b1c94cdc33a P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:03.923372 29790 raft_consensus.cc:740] T 3617af2dee3f48cf85973b1c94cdc33a P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6610c00ffb948ce9c34f41dec3d49e8, State: Initialized, Role: FOLLOWER
I20260709 12:27:03.924440 29790 consensus_queue.cc:260] T 3617af2dee3f48cf85973b1c94cdc33a P c6610c00ffb948ce9c34f41dec3d49e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:27:03.932250 29793 raft_consensus.cc:359] T 137e03144e754f6e9da4f0544513d27c P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:27:03.933378 29793 raft_consensus.cc:385] T 137e03144e754f6e9da4f0544513d27c P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:03.933670 29793 raft_consensus.cc:740] T 137e03144e754f6e9da4f0544513d27c P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d73d92fb35234d16824c55689b5fce28, State: Initialized, Role: FOLLOWER
I20260709 12:27:03.934326 29742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb7d78e534634e0f992fc39b578ebd8d" candidate_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6610c00ffb948ce9c34f41dec3d49e8"
I20260709 12:27:03.935132 29742 raft_consensus.cc:3060] T fb7d78e534634e0f992fc39b578ebd8d P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:03.935492 29793 consensus_queue.cc:260] T 137e03144e754f6e9da4f0544513d27c P d73d92fb35234d16824c55689b5fce28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:27:03.939183 29793 ts_tablet_manager.cc:1434] T 137e03144e754f6e9da4f0544513d27c P d73d92fb35234d16824c55689b5fce28: Time spent starting tablet: real 0.017s	user 0.008s	sys 0.000s
I20260709 12:27:03.939268 29790 ts_tablet_manager.cc:1434] T 3617af2dee3f48cf85973b1c94cdc33a P c6610c00ffb948ce9c34f41dec3d49e8: Time spent starting tablet: real 0.039s	user 0.006s	sys 0.000s
I20260709 12:27:03.849733 29792 tablet_bootstrap.cc:492] T fb7d78e534634e0f992fc39b578ebd8d P 584dac86183a41d9b3bea1c2085e62fa: No bootstrap required, opened a new log
I20260709 12:27:03.850543 30072 leader_election.cc:290] T f3e17f0a8fc14095850e8138fd79b86e P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 1 election: Requested vote from peers 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033), d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825)
I20260709 12:27:03.947152 29792 ts_tablet_manager.cc:1403] T fb7d78e534634e0f992fc39b578ebd8d P 584dac86183a41d9b3bea1c2085e62fa: Time spent bootstrapping tablet: real 0.283s	user 0.015s	sys 0.010s
I20260709 12:27:03.947232 30138 leader_election.cc:290] T fb7d78e534634e0f992fc39b578ebd8d P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 1 election: Requested vote from peers c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895), 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033)
I20260709 12:27:03.950413 29742 raft_consensus.cc:2468] T fb7d78e534634e0f992fc39b578ebd8d P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d13c58ad8ac441c4a9b396eb04b1a476 in term 1.
I20260709 12:27:03.951930 29635 leader_election.cc:304] T fb7d78e534634e0f992fc39b578ebd8d P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c6610c00ffb948ce9c34f41dec3d49e8, d13c58ad8ac441c4a9b396eb04b1a476; no voters: 
I20260709 12:27:03.954481 29793 tablet_bootstrap.cc:492] T a3dcbb9c64bf4b5bb6680e8cb81af93e P d73d92fb35234d16824c55689b5fce28: Bootstrap starting.
I20260709 12:27:03.959254 29521 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f3e17f0a8fc14095850e8138fd79b86e" candidate_uuid: "d73d92fb35234d16824c55689b5fce28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "584dac86183a41d9b3bea1c2085e62fa"
I20260709 12:27:03.959427 29792 raft_consensus.cc:359] T fb7d78e534634e0f992fc39b578ebd8d P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:27:03.960172 29792 raft_consensus.cc:385] T fb7d78e534634e0f992fc39b578ebd8d P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:03.960417 30138 raft_consensus.cc:2804] T fb7d78e534634e0f992fc39b578ebd8d P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:03.967808 29597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c8b2fa9f2c044ec9eec97c72b561042" candidate_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d73d92fb35234d16824c55689b5fce28" is_pre_election: true
W20260709 12:27:03.961020 29560 leader_election.cc:343] T f3e17f0a8fc14095850e8138fd79b86e P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:03.974232 29597 raft_consensus.cc:2468] T 6c8b2fa9f2c044ec9eec97c72b561042 P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6610c00ffb948ce9c34f41dec3d49e8 in term 0.
I20260709 12:27:03.974745 29790 tablet_bootstrap.cc:492] T 0c428ff656ae4b9d99d5b5073f98725d P c6610c00ffb948ce9c34f41dec3d49e8: Bootstrap starting.
I20260709 12:27:03.977221 29793 tablet_bootstrap.cc:654] T a3dcbb9c64bf4b5bb6680e8cb81af93e P d73d92fb35234d16824c55689b5fce28: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:03.980648 29520 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb7d78e534634e0f992fc39b578ebd8d" candidate_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "584dac86183a41d9b3bea1c2085e62fa"
W20260709 12:27:03.981714 29467 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:27:03.994566 30138 raft_consensus.cc:697] T fb7d78e534634e0f992fc39b578ebd8d P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 LEADER]: Becoming Leader. State: Replica: d13c58ad8ac441c4a9b396eb04b1a476, State: Running, Role: LEADER
I20260709 12:27:03.992476 30126 leader_election.cc:290] T f3e17f0a8fc14095850e8138fd79b86e P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 1 election: Requested vote from peers 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033), d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209)
I20260709 12:27:03.995558 30138 consensus_queue.cc:237] T fb7d78e534634e0f992fc39b578ebd8d P d13c58ad8ac441c4a9b396eb04b1a476 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:27:04.001335 29793 tablet_bootstrap.cc:492] T a3dcbb9c64bf4b5bb6680e8cb81af93e P d73d92fb35234d16824c55689b5fce28: No bootstrap required, opened a new log
I20260709 12:27:04.001873 29793 ts_tablet_manager.cc:1403] T a3dcbb9c64bf4b5bb6680e8cb81af93e P d73d92fb35234d16824c55689b5fce28: Time spent bootstrapping tablet: real 0.048s	user 0.021s	sys 0.000s
I20260709 12:27:04.004266 29634 leader_election.cc:304] T 3617af2dee3f48cf85973b1c94cdc33a P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d13c58ad8ac441c4a9b396eb04b1a476, d73d92fb35234d16824c55689b5fce28; no voters: c6610c00ffb948ce9c34f41dec3d49e8
I20260709 12:27:04.004400 29708 leader_election.cc:304] T 394378d078194257b9e04fc659c1f5d4 P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c6610c00ffb948ce9c34f41dec3d49e8, d73d92fb35234d16824c55689b5fce28; no voters: d13c58ad8ac441c4a9b396eb04b1a476
I20260709 12:27:04.005249 30046 raft_consensus.cc:2804] T 394378d078194257b9e04fc659c1f5d4 P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:04.005730 30046 raft_consensus.cc:697] T 394378d078194257b9e04fc659c1f5d4 P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 LEADER]: Becoming Leader. State: Replica: c6610c00ffb948ce9c34f41dec3d49e8, State: Running, Role: LEADER
I20260709 12:27:04.006551 30046 consensus_queue.cc:237] T 394378d078194257b9e04fc659c1f5d4 P c6610c00ffb948ce9c34f41dec3d49e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:27:04.008701 29793 raft_consensus.cc:359] T a3dcbb9c64bf4b5bb6680e8cb81af93e P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:04.009343 29794 tablet_bootstrap.cc:492] T abec627108a84083884e64e1eb9dcda6 P d13c58ad8ac441c4a9b396eb04b1a476: No bootstrap required, opened a new log
I20260709 12:27:04.009449 29793 raft_consensus.cc:385] T a3dcbb9c64bf4b5bb6680e8cb81af93e P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:04.009904 29793 raft_consensus.cc:740] T a3dcbb9c64bf4b5bb6680e8cb81af93e P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d73d92fb35234d16824c55689b5fce28, State: Initialized, Role: FOLLOWER
I20260709 12:27:04.010004 29794 ts_tablet_manager.cc:1403] T abec627108a84083884e64e1eb9dcda6 P d13c58ad8ac441c4a9b396eb04b1a476: Time spent bootstrapping tablet: real 0.170s	user 0.014s	sys 0.003s
I20260709 12:27:04.010609 29793 consensus_queue.cc:260] T a3dcbb9c64bf4b5bb6680e8cb81af93e P d73d92fb35234d16824c55689b5fce28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:03.960484 29792 raft_consensus.cc:740] T fb7d78e534634e0f992fc39b578ebd8d P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 584dac86183a41d9b3bea1c2085e62fa, State: Initialized, Role: FOLLOWER
I20260709 12:27:04.013489 29792 consensus_queue.cc:260] T fb7d78e534634e0f992fc39b578ebd8d P 584dac86183a41d9b3bea1c2085e62fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:27:04.014840 29793 ts_tablet_manager.cc:1434] T a3dcbb9c64bf4b5bb6680e8cb81af93e P d73d92fb35234d16824c55689b5fce28: Time spent starting tablet: real 0.013s	user 0.009s	sys 0.000s
I20260709 12:27:04.015746 29520 raft_consensus.cc:3060] T fb7d78e534634e0f992fc39b578ebd8d P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:04.015017 29790 tablet_bootstrap.cc:654] T 0c428ff656ae4b9d99d5b5073f98725d P c6610c00ffb948ce9c34f41dec3d49e8: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:04.018649 30112 raft_consensus.cc:493] T 01788883c8ea42989f246a3486111a7e P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:04.019375 29793 tablet_bootstrap.cc:492] T d5fcd1f030614bbe82cf1dc261dfc1c3 P d73d92fb35234d16824c55689b5fce28: Bootstrap starting.
I20260709 12:27:04.019219 30112 raft_consensus.cc:515] T 01788883c8ea42989f246a3486111a7e P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:04.022095 29561 leader_election.cc:304] T f8c01e05981747a395e5483b551297dd P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c6610c00ffb948ce9c34f41dec3d49e8, d73d92fb35234d16824c55689b5fce28; no voters: 584dac86183a41d9b3bea1c2085e62fa
I20260709 12:27:04.022897 30126 raft_consensus.cc:2804] T f8c01e05981747a395e5483b551297dd P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:04.022766 29518 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01788883c8ea42989f246a3486111a7e" candidate_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "584dac86183a41d9b3bea1c2085e62fa" is_pre_election: true
I20260709 12:27:04.023829 30112 leader_election.cc:290] T 01788883c8ea42989f246a3486111a7e P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033), d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825)
I20260709 12:27:04.024421 30126 raft_consensus.cc:493] T f8c01e05981747a395e5483b551297dd P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:04.024847 30126 raft_consensus.cc:3060] T f8c01e05981747a395e5483b551297dd P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:04.028007 30153 raft_consensus.cc:2804] T 3617af2dee3f48cf85973b1c94cdc33a P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:04.028447 30153 raft_consensus.cc:493] T 3617af2dee3f48cf85973b1c94cdc33a P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:04.028774 30153 raft_consensus.cc:3060] T 3617af2dee3f48cf85973b1c94cdc33a P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:04.029096 29793 tablet_bootstrap.cc:654] T d5fcd1f030614bbe82cf1dc261dfc1c3 P d73d92fb35234d16824c55689b5fce28: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:04.032712 30166 consensus_queue.cc:1048] T 01788883c8ea42989f246a3486111a7e P d73d92fb35234d16824c55689b5fce28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:27:04.030766 29418 catalog_manager.cc:5697] T 394378d078194257b9e04fc659c1f5d4 P c6610c00ffb948ce9c34f41dec3d49e8 reported cstate change: term changed from 0 to 1, leader changed from <none> to c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68). New cstate: current_term: 1 leader_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } health_report { overall_health: HEALTHY } } }
I20260709 12:27:04.036877 29597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f3e17f0a8fc14095850e8138fd79b86e" candidate_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d73d92fb35234d16824c55689b5fce28"
I20260709 12:27:04.037348 29794 raft_consensus.cc:359] T abec627108a84083884e64e1eb9dcda6 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:27:04.037801 29597 raft_consensus.cc:2393] T f3e17f0a8fc14095850e8138fd79b86e P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate d13c58ad8ac441c4a9b396eb04b1a476 in current term 1: Already voted for candidate d73d92fb35234d16824c55689b5fce28 in this term.
I20260709 12:27:04.038051 29794 raft_consensus.cc:385] T abec627108a84083884e64e1eb9dcda6 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:04.038395 29794 raft_consensus.cc:740] T abec627108a84083884e64e1eb9dcda6 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d13c58ad8ac441c4a9b396eb04b1a476, State: Initialized, Role: FOLLOWER
I20260709 12:27:04.039081 29794 consensus_queue.cc:260] T abec627108a84083884e64e1eb9dcda6 P d13c58ad8ac441c4a9b396eb04b1a476 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:27:04.043355 29794 ts_tablet_manager.cc:1434] T abec627108a84083884e64e1eb9dcda6 P d13c58ad8ac441c4a9b396eb04b1a476: Time spent starting tablet: real 0.033s	user 0.005s	sys 0.002s
I20260709 12:27:04.051313 29794 tablet_bootstrap.cc:492] T 137e03144e754f6e9da4f0544513d27c P d13c58ad8ac441c4a9b396eb04b1a476: Bootstrap starting.
I20260709 12:27:04.052712 29670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f3e17f0a8fc14095850e8138fd79b86e" candidate_uuid: "d73d92fb35234d16824c55689b5fce28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d13c58ad8ac441c4a9b396eb04b1a476"
I20260709 12:27:04.053755 29670 raft_consensus.cc:2393] T f3e17f0a8fc14095850e8138fd79b86e P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate d73d92fb35234d16824c55689b5fce28 in current term 1: Already voted for candidate d13c58ad8ac441c4a9b396eb04b1a476 in this term.
I20260709 12:27:04.059386 30126 raft_consensus.cc:515] T f8c01e05981747a395e5483b551297dd P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:04.063426 29790 tablet_bootstrap.cc:492] T 0c428ff656ae4b9d99d5b5073f98725d P c6610c00ffb948ce9c34f41dec3d49e8: No bootstrap required, opened a new log
I20260709 12:27:04.064039 29790 ts_tablet_manager.cc:1403] T 0c428ff656ae4b9d99d5b5073f98725d P c6610c00ffb948ce9c34f41dec3d49e8: Time spent bootstrapping tablet: real 0.090s	user 0.011s	sys 0.003s
W20260709 12:27:04.064970 29634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:36825, user_credentials={real_user=slave}} blocked reactor thread for 60786.9us
I20260709 12:27:04.069865 29520 raft_consensus.cc:2468] T fb7d78e534634e0f992fc39b578ebd8d P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d13c58ad8ac441c4a9b396eb04b1a476 in term 1.
I20260709 12:27:04.070749 30153 raft_consensus.cc:515] T 3617af2dee3f48cf85973b1c94cdc33a P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:27:04.073287 30153 leader_election.cc:290] T 3617af2dee3f48cf85973b1c94cdc33a P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 1 election: Requested vote from peers d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825), c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895)
I20260709 12:27:04.074620 29792 ts_tablet_manager.cc:1434] T fb7d78e534634e0f992fc39b578ebd8d P 584dac86183a41d9b3bea1c2085e62fa: Time spent starting tablet: real 0.127s	user 0.006s	sys 0.003s
I20260709 12:27:04.074791 29561 leader_election.cc:304] T f3e17f0a8fc14095850e8138fd79b86e P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d73d92fb35234d16824c55689b5fce28; no voters: 584dac86183a41d9b3bea1c2085e62fa, d13c58ad8ac441c4a9b396eb04b1a476
I20260709 12:27:04.081424 29794 tablet_bootstrap.cc:654] T 137e03144e754f6e9da4f0544513d27c P d13c58ad8ac441c4a9b396eb04b1a476: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:04.081084 29790 raft_consensus.cc:359] T 0c428ff656ae4b9d99d5b5073f98725d P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:27:04.081907 29790 raft_consensus.cc:385] T 0c428ff656ae4b9d99d5b5073f98725d P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:04.082250 29790 raft_consensus.cc:740] T 0c428ff656ae4b9d99d5b5073f98725d P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6610c00ffb948ce9c34f41dec3d49e8, State: Initialized, Role: FOLLOWER
I20260709 12:27:04.085530 29792 tablet_bootstrap.cc:492] T a3dcbb9c64bf4b5bb6680e8cb81af93e P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap starting.
I20260709 12:27:04.086086 29708 leader_election.cc:304] T 6c8b2fa9f2c044ec9eec97c72b561042 P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c6610c00ffb948ce9c34f41dec3d49e8, d73d92fb35234d16824c55689b5fce28; no voters: 584dac86183a41d9b3bea1c2085e62fa
I20260709 12:27:04.086793 29744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3617af2dee3f48cf85973b1c94cdc33a" candidate_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6610c00ffb948ce9c34f41dec3d49e8"
I20260709 12:27:04.087399 29744 raft_consensus.cc:3060] T 3617af2dee3f48cf85973b1c94cdc33a P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:04.088055 30166 raft_consensus.cc:2749] T f3e17f0a8fc14095850e8138fd79b86e P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 12:27:04.088055 30046 raft_consensus.cc:2804] T 6c8b2fa9f2c044ec9eec97c72b561042 P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:04.089932 30046 raft_consensus.cc:493] T 6c8b2fa9f2c044ec9eec97c72b561042 P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:04.090339 30046 raft_consensus.cc:3060] T 6c8b2fa9f2c044ec9eec97c72b561042 P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:04.092396 29520 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8c01e05981747a395e5483b551297dd" candidate_uuid: "d73d92fb35234d16824c55689b5fce28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "584dac86183a41d9b3bea1c2085e62fa"
W20260709 12:27:04.094169 29560 leader_election.cc:343] T f8c01e05981747a395e5483b551297dd P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:04.093042 30126 leader_election.cc:290] T f8c01e05981747a395e5483b551297dd P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 1 election: Requested vote from peers c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895), 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033)
I20260709 12:27:04.097362 29792 tablet_bootstrap.cc:654] T a3dcbb9c64bf4b5bb6680e8cb81af93e P 584dac86183a41d9b3bea1c2085e62fa: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:04.099840 29595 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01788883c8ea42989f246a3486111a7e" candidate_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d73d92fb35234d16824c55689b5fce28" is_pre_election: true
I20260709 12:27:04.101809 29744 raft_consensus.cc:2468] T 3617af2dee3f48cf85973b1c94cdc33a P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d13c58ad8ac441c4a9b396eb04b1a476 in term 1.
I20260709 12:27:04.106352 29744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8c01e05981747a395e5483b551297dd" candidate_uuid: "d73d92fb35234d16824c55689b5fce28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6610c00ffb948ce9c34f41dec3d49e8"
I20260709 12:27:04.107374 29744 raft_consensus.cc:3060] T f8c01e05981747a395e5483b551297dd P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:04.110762 29792 tablet_bootstrap.cc:492] T a3dcbb9c64bf4b5bb6680e8cb81af93e P 584dac86183a41d9b3bea1c2085e62fa: No bootstrap required, opened a new log
I20260709 12:27:04.111310 29792 ts_tablet_manager.cc:1403] T a3dcbb9c64bf4b5bb6680e8cb81af93e P 584dac86183a41d9b3bea1c2085e62fa: Time spent bootstrapping tablet: real 0.026s	user 0.016s	sys 0.001s
I20260709 12:27:04.111549 29793 tablet_bootstrap.cc:492] T d5fcd1f030614bbe82cf1dc261dfc1c3 P d73d92fb35234d16824c55689b5fce28: No bootstrap required, opened a new log
I20260709 12:27:04.112063 29793 ts_tablet_manager.cc:1403] T d5fcd1f030614bbe82cf1dc261dfc1c3 P d73d92fb35234d16824c55689b5fce28: Time spent bootstrapping tablet: real 0.093s	user 0.029s	sys 0.004s
I20260709 12:27:04.114451 29792 raft_consensus.cc:359] T a3dcbb9c64bf4b5bb6680e8cb81af93e P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:04.115396 29792 raft_consensus.cc:385] T a3dcbb9c64bf4b5bb6680e8cb81af93e P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:04.115798 29792 raft_consensus.cc:740] T a3dcbb9c64bf4b5bb6680e8cb81af93e P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 584dac86183a41d9b3bea1c2085e62fa, State: Initialized, Role: FOLLOWER
I20260709 12:27:04.116705 29792 consensus_queue.cc:260] T a3dcbb9c64bf4b5bb6680e8cb81af93e P 584dac86183a41d9b3bea1c2085e62fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:04.119273 29793 raft_consensus.cc:359] T d5fcd1f030614bbe82cf1dc261dfc1c3 P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:27:04.121618 29790 consensus_queue.cc:260] T 0c428ff656ae4b9d99d5b5073f98725d P c6610c00ffb948ce9c34f41dec3d49e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:27:04.124039 29635 leader_election.cc:304] T 3617af2dee3f48cf85973b1c94cdc33a P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c6610c00ffb948ce9c34f41dec3d49e8, d13c58ad8ac441c4a9b396eb04b1a476; no voters: 
I20260709 12:27:04.131175 30138 raft_consensus.cc:2804] T 3617af2dee3f48cf85973b1c94cdc33a P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:04.131852 30138 raft_consensus.cc:697] T 3617af2dee3f48cf85973b1c94cdc33a P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 LEADER]: Becoming Leader. State: Replica: d13c58ad8ac441c4a9b396eb04b1a476, State: Running, Role: LEADER
I20260709 12:27:04.132812 29792 ts_tablet_manager.cc:1434] T a3dcbb9c64bf4b5bb6680e8cb81af93e P 584dac86183a41d9b3bea1c2085e62fa: Time spent starting tablet: real 0.021s	user 0.004s	sys 0.004s
I20260709 12:27:04.120318 29793 raft_consensus.cc:385] T d5fcd1f030614bbe82cf1dc261dfc1c3 P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:04.133641 29793 raft_consensus.cc:740] T d5fcd1f030614bbe82cf1dc261dfc1c3 P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d73d92fb35234d16824c55689b5fce28, State: Initialized, Role: FOLLOWER
I20260709 12:27:04.134167 29792 tablet_bootstrap.cc:492] T f8c01e05981747a395e5483b551297dd P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap starting.
I20260709 12:27:04.134418 29793 consensus_queue.cc:260] T d5fcd1f030614bbe82cf1dc261dfc1c3 P d73d92fb35234d16824c55689b5fce28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:27:04.133116 30138 consensus_queue.cc:237] T 3617af2dee3f48cf85973b1c94cdc33a P d13c58ad8ac441c4a9b396eb04b1a476 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:27:04.126171 29708 leader_election.cc:304] T 01788883c8ea42989f246a3486111a7e P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c6610c00ffb948ce9c34f41dec3d49e8; no voters: 584dac86183a41d9b3bea1c2085e62fa, d73d92fb35234d16824c55689b5fce28
I20260709 12:27:04.139909 29793 ts_tablet_manager.cc:1434] T d5fcd1f030614bbe82cf1dc261dfc1c3 P d73d92fb35234d16824c55689b5fce28: Time spent starting tablet: real 0.028s	user 0.007s	sys 0.000s
I20260709 12:27:04.140367 29744 raft_consensus.cc:2468] T f8c01e05981747a395e5483b551297dd P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d73d92fb35234d16824c55689b5fce28 in term 1.
I20260709 12:27:04.140862 30213 raft_consensus.cc:2749] T 01788883c8ea42989f246a3486111a7e P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:27:04.140862 29634 leader_election.cc:304] T f3e17f0a8fc14095850e8138fd79b86e P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d13c58ad8ac441c4a9b396eb04b1a476; no voters: 584dac86183a41d9b3bea1c2085e62fa, d73d92fb35234d16824c55689b5fce28
I20260709 12:27:04.142406 30072 raft_consensus.cc:2749] T f3e17f0a8fc14095850e8138fd79b86e P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 12:27:04.152750 29790 ts_tablet_manager.cc:1434] T 0c428ff656ae4b9d99d5b5073f98725d P c6610c00ffb948ce9c34f41dec3d49e8: Time spent starting tablet: real 0.075s	user 0.007s	sys 0.001s
I20260709 12:27:04.154835 29790 tablet_bootstrap.cc:492] T ef20d7124e7a44aa97a219ee194f4b8b P c6610c00ffb948ce9c34f41dec3d49e8: Bootstrap starting.
I20260709 12:27:04.161406 29561 leader_election.cc:304] T f8c01e05981747a395e5483b551297dd P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c6610c00ffb948ce9c34f41dec3d49e8, d73d92fb35234d16824c55689b5fce28; no voters: 584dac86183a41d9b3bea1c2085e62fa
I20260709 12:27:04.162400 30126 raft_consensus.cc:2804] T f8c01e05981747a395e5483b551297dd P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:04.162945 30126 raft_consensus.cc:697] T f8c01e05981747a395e5483b551297dd P d73d92fb35234d16824c55689b5fce28 [term 1 LEADER]: Becoming Leader. State: Replica: d73d92fb35234d16824c55689b5fce28, State: Running, Role: LEADER
I20260709 12:27:04.163905 30126 consensus_queue.cc:237] T f8c01e05981747a395e5483b551297dd P d73d92fb35234d16824c55689b5fce28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:04.172271 29790 tablet_bootstrap.cc:654] T ef20d7124e7a44aa97a219ee194f4b8b P c6610c00ffb948ce9c34f41dec3d49e8: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:04.183420 29792 tablet_bootstrap.cc:654] T f8c01e05981747a395e5483b551297dd P 584dac86183a41d9b3bea1c2085e62fa: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:04.190423 29595 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3617af2dee3f48cf85973b1c94cdc33a" candidate_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d73d92fb35234d16824c55689b5fce28"
I20260709 12:27:04.190269 30046 raft_consensus.cc:515] T 6c8b2fa9f2c044ec9eec97c72b561042 P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:04.191102 29595 raft_consensus.cc:3060] T 3617af2dee3f48cf85973b1c94cdc33a P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:04.192711 30046 leader_election.cc:290] T 6c8b2fa9f2c044ec9eec97c72b561042 P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 1 election: Requested vote from peers 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033), d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825)
I20260709 12:27:04.195065 30166 raft_consensus.cc:493] T a3dcbb9c64bf4b5bb6680e8cb81af93e P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:04.195456 29518 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c8b2fa9f2c044ec9eec97c72b561042" candidate_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "584dac86183a41d9b3bea1c2085e62fa"
I20260709 12:27:04.195657 29597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c8b2fa9f2c044ec9eec97c72b561042" candidate_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d73d92fb35234d16824c55689b5fce28"
I20260709 12:27:04.196419 29597 raft_consensus.cc:3060] T 6c8b2fa9f2c044ec9eec97c72b561042 P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:27:04.196943 29708 leader_election.cc:343] T 6c8b2fa9f2c044ec9eec97c72b561042 P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:04.195645 30166 raft_consensus.cc:515] T a3dcbb9c64bf4b5bb6680e8cb81af93e P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:04.203397 30166 leader_election.cc:290] T a3dcbb9c64bf4b5bb6680e8cb81af93e P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895), 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033)
I20260709 12:27:04.203284 29419 catalog_manager.cc:5697] T fb7d78e534634e0f992fc39b578ebd8d P d13c58ad8ac441c4a9b396eb04b1a476 reported cstate change: term changed from 0 to 1, leader changed from <none> to d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67). New cstate: current_term: 1 leader_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:04.210083 29744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3dcbb9c64bf4b5bb6680e8cb81af93e" candidate_uuid: "d73d92fb35234d16824c55689b5fce28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" is_pre_election: true
W20260709 12:27:04.211503 29561 leader_election.cc:343] T a3dcbb9c64bf4b5bb6680e8cb81af93e P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:04.212118 29597 raft_consensus.cc:2468] T 6c8b2fa9f2c044ec9eec97c72b561042 P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6610c00ffb948ce9c34f41dec3d49e8 in term 1.
I20260709 12:27:04.214874 29708 leader_election.cc:304] T 6c8b2fa9f2c044ec9eec97c72b561042 P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c6610c00ffb948ce9c34f41dec3d49e8, d73d92fb35234d16824c55689b5fce28; no voters: 584dac86183a41d9b3bea1c2085e62fa
I20260709 12:27:04.216228 30072 raft_consensus.cc:493] T abec627108a84083884e64e1eb9dcda6 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:04.216017 29518 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3dcbb9c64bf4b5bb6680e8cb81af93e" candidate_uuid: "d73d92fb35234d16824c55689b5fce28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "584dac86183a41d9b3bea1c2085e62fa" is_pre_election: true
I20260709 12:27:04.216887 30072 raft_consensus.cc:515] T abec627108a84083884e64e1eb9dcda6 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:27:04.214892 29595 raft_consensus.cc:2468] T 3617af2dee3f48cf85973b1c94cdc33a P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d13c58ad8ac441c4a9b396eb04b1a476 in term 1.
I20260709 12:27:04.219311 30072 leader_election.cc:290] T abec627108a84083884e64e1eb9dcda6 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033), d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825)
I20260709 12:27:04.220080 30213 raft_consensus.cc:2804] T 6c8b2fa9f2c044ec9eec97c72b561042 P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:04.217165 29518 raft_consensus.cc:2468] T a3dcbb9c64bf4b5bb6680e8cb81af93e P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d73d92fb35234d16824c55689b5fce28 in term 0.
I20260709 12:27:04.221966 30213 raft_consensus.cc:697] T 6c8b2fa9f2c044ec9eec97c72b561042 P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 LEADER]: Becoming Leader. State: Replica: c6610c00ffb948ce9c34f41dec3d49e8, State: Running, Role: LEADER
I20260709 12:27:04.223028 29560 leader_election.cc:304] T a3dcbb9c64bf4b5bb6680e8cb81af93e P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 584dac86183a41d9b3bea1c2085e62fa, d73d92fb35234d16824c55689b5fce28; no voters: c6610c00ffb948ce9c34f41dec3d49e8
I20260709 12:27:04.222939 30213 consensus_queue.cc:237] T 6c8b2fa9f2c044ec9eec97c72b561042 P c6610c00ffb948ce9c34f41dec3d49e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:04.224498 30003 raft_consensus.cc:2804] T a3dcbb9c64bf4b5bb6680e8cb81af93e P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:04.225000 30003 raft_consensus.cc:493] T a3dcbb9c64bf4b5bb6680e8cb81af93e P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:04.225760 30003 raft_consensus.cc:3060] T a3dcbb9c64bf4b5bb6680e8cb81af93e P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:04.230322 29518 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abec627108a84083884e64e1eb9dcda6" candidate_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "584dac86183a41d9b3bea1c2085e62fa" is_pre_election: true
I20260709 12:27:04.235069 29518 raft_consensus.cc:2393] T abec627108a84083884e64e1eb9dcda6 P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d13c58ad8ac441c4a9b396eb04b1a476 in current term 1: Already voted for candidate d73d92fb35234d16824c55689b5fce28 in this term.
I20260709 12:27:04.241681 29794 tablet_bootstrap.cc:492] T 137e03144e754f6e9da4f0544513d27c P d13c58ad8ac441c4a9b396eb04b1a476: No bootstrap required, opened a new log
I20260709 12:27:04.242254 29794 ts_tablet_manager.cc:1403] T 137e03144e754f6e9da4f0544513d27c P d13c58ad8ac441c4a9b396eb04b1a476: Time spent bootstrapping tablet: real 0.191s	user 0.016s	sys 0.000s
I20260709 12:27:04.244413 29595 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abec627108a84083884e64e1eb9dcda6" candidate_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d73d92fb35234d16824c55689b5fce28" is_pre_election: true
I20260709 12:27:04.264981 30203 raft_consensus.cc:493] T f76101eb30f14b98a9aa8c41e0d6f831 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:04.265519 30203 raft_consensus.cc:515] T f76101eb30f14b98a9aa8c41e0d6f831 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:27:04.266943 29790 tablet_bootstrap.cc:492] T ef20d7124e7a44aa97a219ee194f4b8b P c6610c00ffb948ce9c34f41dec3d49e8: No bootstrap required, opened a new log
I20260709 12:27:04.267539 29790 ts_tablet_manager.cc:1403] T ef20d7124e7a44aa97a219ee194f4b8b P c6610c00ffb948ce9c34f41dec3d49e8: Time spent bootstrapping tablet: real 0.113s	user 0.011s	sys 0.007s
I20260709 12:27:04.267611 30203 leader_election.cc:290] T f76101eb30f14b98a9aa8c41e0d6f831 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895), 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033)
I20260709 12:27:04.269212 29744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f76101eb30f14b98a9aa8c41e0d6f831" candidate_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" is_pre_election: true
I20260709 12:27:04.269833 29744 raft_consensus.cc:2468] T f76101eb30f14b98a9aa8c41e0d6f831 P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d13c58ad8ac441c4a9b396eb04b1a476 in term 0.
I20260709 12:27:04.269918 29670 raft_consensus.cc:3060] T abec627108a84083884e64e1eb9dcda6 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:04.271273 29635 leader_election.cc:304] T f76101eb30f14b98a9aa8c41e0d6f831 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c6610c00ffb948ce9c34f41dec3d49e8, d13c58ad8ac441c4a9b396eb04b1a476; no voters: 
I20260709 12:27:04.272424 30203 raft_consensus.cc:2804] T f76101eb30f14b98a9aa8c41e0d6f831 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:04.272778 30203 raft_consensus.cc:493] T f76101eb30f14b98a9aa8c41e0d6f831 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:04.273111 30203 raft_consensus.cc:3060] T f76101eb30f14b98a9aa8c41e0d6f831 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:04.263816 29634 leader_election.cc:304] T abec627108a84083884e64e1eb9dcda6 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d13c58ad8ac441c4a9b396eb04b1a476; no voters: 584dac86183a41d9b3bea1c2085e62fa, d73d92fb35234d16824c55689b5fce28
I20260709 12:27:04.280681 30072 raft_consensus.cc:2749] T abec627108a84083884e64e1eb9dcda6 P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:04.257982 30003 raft_consensus.cc:515] T a3dcbb9c64bf4b5bb6680e8cb81af93e P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:04.292264 29790 raft_consensus.cc:359] T ef20d7124e7a44aa97a219ee194f4b8b P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:27:04.293002 29790 raft_consensus.cc:385] T ef20d7124e7a44aa97a219ee194f4b8b P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:04.293265 29790 raft_consensus.cc:740] T ef20d7124e7a44aa97a219ee194f4b8b P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6610c00ffb948ce9c34f41dec3d49e8, State: Initialized, Role: FOLLOWER
I20260709 12:27:04.289940 29520 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f76101eb30f14b98a9aa8c41e0d6f831" candidate_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "584dac86183a41d9b3bea1c2085e62fa" is_pre_election: true
I20260709 12:27:04.300051 29419 catalog_manager.cc:5697] T 6c8b2fa9f2c044ec9eec97c72b561042 P c6610c00ffb948ce9c34f41dec3d49e8 reported cstate change: term changed from 0 to 1, leader changed from <none> to c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68). New cstate: current_term: 1 leader_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:04.290689 30126 consensus_queue.cc:1048] T abec627108a84083884e64e1eb9dcda6 P d73d92fb35234d16824c55689b5fce28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.012s
I20260709 12:27:04.302664 29744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3dcbb9c64bf4b5bb6680e8cb81af93e" candidate_uuid: "d73d92fb35234d16824c55689b5fce28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6610c00ffb948ce9c34f41dec3d49e8"
I20260709 12:27:04.302961 29794 raft_consensus.cc:359] T 137e03144e754f6e9da4f0544513d27c P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:27:04.303725 29794 raft_consensus.cc:385] T 137e03144e754f6e9da4f0544513d27c P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:04.304009 29794 raft_consensus.cc:740] T 137e03144e754f6e9da4f0544513d27c P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d13c58ad8ac441c4a9b396eb04b1a476, State: Initialized, Role: FOLLOWER
I20260709 12:27:04.304978 30203 raft_consensus.cc:515] T f76101eb30f14b98a9aa8c41e0d6f831 P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:27:04.307092 29792 tablet_bootstrap.cc:492] T f8c01e05981747a395e5483b551297dd P 584dac86183a41d9b3bea1c2085e62fa: No bootstrap required, opened a new log
I20260709 12:27:04.307734 29792 ts_tablet_manager.cc:1403] T f8c01e05981747a395e5483b551297dd P 584dac86183a41d9b3bea1c2085e62fa: Time spent bootstrapping tablet: real 0.174s	user 0.034s	sys 0.019s
I20260709 12:27:04.309715 29744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f76101eb30f14b98a9aa8c41e0d6f831" candidate_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6610c00ffb948ce9c34f41dec3d49e8"
I20260709 12:27:04.310281 29744 raft_consensus.cc:3060] T f76101eb30f14b98a9aa8c41e0d6f831 P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:04.310497 29792 raft_consensus.cc:359] T f8c01e05981747a395e5483b551297dd P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:04.311148 29792 raft_consensus.cc:385] T f8c01e05981747a395e5483b551297dd P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:04.311389 29792 raft_consensus.cc:740] T f8c01e05981747a395e5483b551297dd P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 584dac86183a41d9b3bea1c2085e62fa, State: Initialized, Role: FOLLOWER
W20260709 12:27:04.312263 29561 leader_election.cc:343] T a3dcbb9c64bf4b5bb6680e8cb81af93e P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:04.312057 29794 consensus_queue.cc:260] T 137e03144e754f6e9da4f0544513d27c P d13c58ad8ac441c4a9b396eb04b1a476 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:27:04.316731 29794 ts_tablet_manager.cc:1434] T 137e03144e754f6e9da4f0544513d27c P d13c58ad8ac441c4a9b396eb04b1a476: Time spent starting tablet: real 0.074s	user 0.007s	sys 0.000s
I20260709 12:27:04.317355 30203 leader_election.cc:290] T f76101eb30f14b98a9aa8c41e0d6f831 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 1 election: Requested vote from peers c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895), 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033)
I20260709 12:27:04.314944 29417 catalog_manager.cc:5697] T 3617af2dee3f48cf85973b1c94cdc33a P d13c58ad8ac441c4a9b396eb04b1a476 reported cstate change: term changed from 0 to 1, leader changed from <none> to d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67). New cstate: current_term: 1 leader_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:04.317807 29794 tablet_bootstrap.cc:492] T d5fcd1f030614bbe82cf1dc261dfc1c3 P d13c58ad8ac441c4a9b396eb04b1a476: Bootstrap starting.
I20260709 12:27:04.318235 29792 consensus_queue.cc:260] T f8c01e05981747a395e5483b551297dd P 584dac86183a41d9b3bea1c2085e62fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:04.294027 29790 consensus_queue.cc:260] T ef20d7124e7a44aa97a219ee194f4b8b P c6610c00ffb948ce9c34f41dec3d49e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:27:04.321566 29792 ts_tablet_manager.cc:1434] T f8c01e05981747a395e5483b551297dd P 584dac86183a41d9b3bea1c2085e62fa: Time spent starting tablet: real 0.014s	user 0.005s	sys 0.000s
I20260709 12:27:04.301011 29418 catalog_manager.cc:5697] T f8c01e05981747a395e5483b551297dd P d73d92fb35234d16824c55689b5fce28 reported cstate change: term changed from 0 to 1, leader changed from <none> to d73d92fb35234d16824c55689b5fce28 (127.16.29.66). New cstate: current_term: 1 leader_uuid: "d73d92fb35234d16824c55689b5fce28" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } health_report { overall_health: HEALTHY } } }
I20260709 12:27:04.331113 29794 tablet_bootstrap.cc:654] T d5fcd1f030614bbe82cf1dc261dfc1c3 P d13c58ad8ac441c4a9b396eb04b1a476: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:04.334518 30119 consensus_queue.cc:1048] T abec627108a84083884e64e1eb9dcda6 P d73d92fb35234d16824c55689b5fce28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.056s
I20260709 12:27:04.335741 29518 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f76101eb30f14b98a9aa8c41e0d6f831" candidate_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "584dac86183a41d9b3bea1c2085e62fa"
W20260709 12:27:04.349371 29634 leader_election.cc:343] T f76101eb30f14b98a9aa8c41e0d6f831 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:04.482625 29792 tablet_bootstrap.cc:492] T ef20d7124e7a44aa97a219ee194f4b8b P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap starting.
I20260709 12:27:04.484325 30228 raft_consensus.cc:493] T f8c01e05981747a395e5483b551297dd P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:04.517161 30228 raft_consensus.cc:515] T f8c01e05981747a395e5483b551297dd P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:04.519322 30228 leader_election.cc:290] T f8c01e05981747a395e5483b551297dd P 584dac86183a41d9b3bea1c2085e62fa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895), d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825)
W20260709 12:27:04.520112 29709 consensus_peers.cc:597] T 394378d078194257b9e04fc659c1f5d4 P c6610c00ffb948ce9c34f41dec3d49e8 -> Peer d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209): Couldn't send request to peer d13c58ad8ac441c4a9b396eb04b1a476. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:04.515693 30003 leader_election.cc:290] T a3dcbb9c64bf4b5bb6680e8cb81af93e P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 1 election: Requested vote from peers c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895), 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033)
I20260709 12:27:04.524602 29792 tablet_bootstrap.cc:654] T ef20d7124e7a44aa97a219ee194f4b8b P 584dac86183a41d9b3bea1c2085e62fa: Neither blocks nor log segments found. Creating new log.
W20260709 12:27:04.524888 29634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:35033, user_credentials={real_user=slave}} blocked reactor thread for 175553us
W20260709 12:27:04.525876 29634 leader_election.cc:343] T f76101eb30f14b98a9aa8c41e0d6f831 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:04.535634 30138 raft_consensus.cc:493] T 137e03144e754f6e9da4f0544513d27c P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:04.536202 30138 raft_consensus.cc:515] T 137e03144e754f6e9da4f0544513d27c P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:27:04.484508 29744 raft_consensus.cc:2468] T f76101eb30f14b98a9aa8c41e0d6f831 P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d13c58ad8ac441c4a9b396eb04b1a476 in term 1.
I20260709 12:27:04.538345 30138 leader_election.cc:290] T 137e03144e754f6e9da4f0544513d27c P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825), c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895)
I20260709 12:27:04.539813 29744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8c01e05981747a395e5483b551297dd" candidate_uuid: "584dac86183a41d9b3bea1c2085e62fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" is_pre_election: true
I20260709 12:27:04.540582 29744 raft_consensus.cc:2393] T f8c01e05981747a395e5483b551297dd P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 584dac86183a41d9b3bea1c2085e62fa in current term 1: Already voted for candidate d73d92fb35234d16824c55689b5fce28 in this term.
I20260709 12:27:04.541755 29635 leader_election.cc:304] T f76101eb30f14b98a9aa8c41e0d6f831 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c6610c00ffb948ce9c34f41dec3d49e8, d13c58ad8ac441c4a9b396eb04b1a476; no voters: 584dac86183a41d9b3bea1c2085e62fa
I20260709 12:27:04.542836 30043 raft_consensus.cc:2804] T f76101eb30f14b98a9aa8c41e0d6f831 P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:04.543381 30043 raft_consensus.cc:697] T f76101eb30f14b98a9aa8c41e0d6f831 P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 LEADER]: Becoming Leader. State: Replica: d13c58ad8ac441c4a9b396eb04b1a476, State: Running, Role: LEADER
I20260709 12:27:04.542207 29744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "137e03144e754f6e9da4f0544513d27c" candidate_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" is_pre_election: true
W20260709 12:27:04.544745 29635 leader_election.cc:343] T 137e03144e754f6e9da4f0544513d27c P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:04.488348 30046 raft_consensus.cc:493] T 0c428ff656ae4b9d99d5b5073f98725d P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:04.545372 30046 raft_consensus.cc:515] T 0c428ff656ae4b9d99d5b5073f98725d P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:27:04.547353 30046 leader_election.cc:290] T 0c428ff656ae4b9d99d5b5073f98725d P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209), 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033)
I20260709 12:27:04.491644 30165 raft_consensus.cc:493] T d5fcd1f030614bbe82cf1dc261dfc1c3 P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:04.491814 29794 tablet_bootstrap.cc:492] T d5fcd1f030614bbe82cf1dc261dfc1c3 P d13c58ad8ac441c4a9b396eb04b1a476: No bootstrap required, opened a new log
W20260709 12:27:04.494956 29560 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:35033, user_credentials={real_user=slave}} blocked reactor thread for 120440us
I20260709 12:27:04.548305 29794 ts_tablet_manager.cc:1403] T d5fcd1f030614bbe82cf1dc261dfc1c3 P d13c58ad8ac441c4a9b396eb04b1a476: Time spent bootstrapping tablet: real 0.231s	user 0.022s	sys 0.015s
I20260709 12:27:04.547976 30165 raft_consensus.cc:515] T d5fcd1f030614bbe82cf1dc261dfc1c3 P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:27:04.550014 30165 leader_election.cc:290] T d5fcd1f030614bbe82cf1dc261dfc1c3 P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033), d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209)
I20260709 12:27:04.498275 30072 consensus_queue.cc:1048] T fb7d78e534634e0f992fc39b578ebd8d P d13c58ad8ac441c4a9b396eb04b1a476 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:27:04.484850 29790 ts_tablet_manager.cc:1434] T ef20d7124e7a44aa97a219ee194f4b8b P c6610c00ffb948ce9c34f41dec3d49e8: Time spent starting tablet: real 0.217s	user 0.006s	sys 0.003s
I20260709 12:27:04.544314 30043 consensus_queue.cc:237] T f76101eb30f14b98a9aa8c41e0d6f831 P d13c58ad8ac441c4a9b396eb04b1a476 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:27:04.586359 29667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c428ff656ae4b9d99d5b5073f98725d" candidate_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" is_pre_election: true
W20260709 12:27:04.589286 29709 leader_election.cc:343] T 0c428ff656ae4b9d99d5b5073f98725d P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:04.592866 29519 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c428ff656ae4b9d99d5b5073f98725d" candidate_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "584dac86183a41d9b3bea1c2085e62fa" is_pre_election: true
I20260709 12:27:04.551107 29794 raft_consensus.cc:359] T d5fcd1f030614bbe82cf1dc261dfc1c3 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:27:04.334789 30166 raft_consensus.cc:493] T 137e03144e754f6e9da4f0544513d27c P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:04.581873 30072 consensus_queue.cc:1048] T 3617af2dee3f48cf85973b1c94cdc33a P d13c58ad8ac441c4a9b396eb04b1a476 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:27:04.597024 29519 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3dcbb9c64bf4b5bb6680e8cb81af93e" candidate_uuid: "d73d92fb35234d16824c55689b5fce28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "584dac86183a41d9b3bea1c2085e62fa"
I20260709 12:27:04.597421 30166 raft_consensus.cc:515] T 137e03144e754f6e9da4f0544513d27c P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:27:04.597821 29519 raft_consensus.cc:3060] T a3dcbb9c64bf4b5bb6680e8cb81af93e P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:04.599538 30166 leader_election.cc:290] T 137e03144e754f6e9da4f0544513d27c P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209), c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895)
I20260709 12:27:04.602928 29794 raft_consensus.cc:385] T d5fcd1f030614bbe82cf1dc261dfc1c3 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:04.603293 29794 raft_consensus.cc:740] T d5fcd1f030614bbe82cf1dc261dfc1c3 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d13c58ad8ac441c4a9b396eb04b1a476, State: Initialized, Role: FOLLOWER
I20260709 12:27:04.604355 29790 tablet_bootstrap.cc:492] T a3dcbb9c64bf4b5bb6680e8cb81af93e P c6610c00ffb948ce9c34f41dec3d49e8: Bootstrap starting.
I20260709 12:27:04.604140 29794 consensus_queue.cc:260] T d5fcd1f030614bbe82cf1dc261dfc1c3 P d13c58ad8ac441c4a9b396eb04b1a476 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:27:04.607911 29794 ts_tablet_manager.cc:1434] T d5fcd1f030614bbe82cf1dc261dfc1c3 P d13c58ad8ac441c4a9b396eb04b1a476: Time spent starting tablet: real 0.059s	user 0.006s	sys 0.004s
I20260709 12:27:04.609066 29794 tablet_bootstrap.cc:492] T 0c428ff656ae4b9d99d5b5073f98725d P d13c58ad8ac441c4a9b396eb04b1a476: Bootstrap starting.
W20260709 12:27:04.610146 29708 leader_election.cc:343] T 0c428ff656ae4b9d99d5b5073f98725d P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:04.610797 29708 leader_election.cc:304] T 0c428ff656ae4b9d99d5b5073f98725d P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c6610c00ffb948ce9c34f41dec3d49e8; no voters: 584dac86183a41d9b3bea1c2085e62fa, d13c58ad8ac441c4a9b396eb04b1a476
W20260709 12:27:04.529274 29561 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:43895, user_credentials={real_user=slave}} blocked reactor thread for 122054us
I20260709 12:27:04.615293 30216 raft_consensus.cc:2749] T 0c428ff656ae4b9d99d5b5073f98725d P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:04.616113 30138 consensus_queue.cc:1048] T fb7d78e534634e0f992fc39b578ebd8d P d13c58ad8ac441c4a9b396eb04b1a476 [LEADER]: Connected to new peer: Peer: permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:27:04.620841 29794 tablet_bootstrap.cc:654] T 0c428ff656ae4b9d99d5b5073f98725d P d13c58ad8ac441c4a9b396eb04b1a476: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:04.628262 29519 raft_consensus.cc:2468] T a3dcbb9c64bf4b5bb6680e8cb81af93e P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d73d92fb35234d16824c55689b5fce28 in term 1.
I20260709 12:27:04.629901 29667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d5fcd1f030614bbe82cf1dc261dfc1c3" candidate_uuid: "d73d92fb35234d16824c55689b5fce28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" is_pre_election: true
I20260709 12:27:04.630600 29667 raft_consensus.cc:2468] T d5fcd1f030614bbe82cf1dc261dfc1c3 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d73d92fb35234d16824c55689b5fce28 in term 0.
I20260709 12:27:04.630788 29792 tablet_bootstrap.cc:492] T ef20d7124e7a44aa97a219ee194f4b8b P 584dac86183a41d9b3bea1c2085e62fa: No bootstrap required, opened a new log
I20260709 12:27:04.631386 29792 ts_tablet_manager.cc:1403] T ef20d7124e7a44aa97a219ee194f4b8b P 584dac86183a41d9b3bea1c2085e62fa: Time spent bootstrapping tablet: real 0.149s	user 0.013s	sys 0.050s
I20260709 12:27:04.640908 29560 leader_election.cc:304] T a3dcbb9c64bf4b5bb6680e8cb81af93e P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 584dac86183a41d9b3bea1c2085e62fa, d73d92fb35234d16824c55689b5fce28; no voters: c6610c00ffb948ce9c34f41dec3d49e8
I20260709 12:27:04.643464 30166 raft_consensus.cc:2804] T a3dcbb9c64bf4b5bb6680e8cb81af93e P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:04.644223 30166 raft_consensus.cc:697] T a3dcbb9c64bf4b5bb6680e8cb81af93e P d73d92fb35234d16824c55689b5fce28 [term 1 LEADER]: Becoming Leader. State: Replica: d73d92fb35234d16824c55689b5fce28, State: Running, Role: LEADER
I20260709 12:27:04.645344 30166 consensus_queue.cc:237] T a3dcbb9c64bf4b5bb6680e8cb81af93e P d73d92fb35234d16824c55689b5fce28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:04.649894 29741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "137e03144e754f6e9da4f0544513d27c" candidate_uuid: "d73d92fb35234d16824c55689b5fce28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" is_pre_election: true
I20260709 12:27:04.652343 29668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "137e03144e754f6e9da4f0544513d27c" candidate_uuid: "d73d92fb35234d16824c55689b5fce28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" is_pre_election: true
I20260709 12:27:04.653021 29668 raft_consensus.cc:2468] T 137e03144e754f6e9da4f0544513d27c P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d73d92fb35234d16824c55689b5fce28 in term 0.
I20260709 12:27:04.654642 30216 raft_consensus.cc:493] T ef20d7124e7a44aa97a219ee194f4b8b P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:04.662003 30216 raft_consensus.cc:515] T ef20d7124e7a44aa97a219ee194f4b8b P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:27:04.665035 30216 leader_election.cc:290] T ef20d7124e7a44aa97a219ee194f4b8b P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209), 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033)
I20260709 12:27:04.666399 29668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef20d7124e7a44aa97a219ee194f4b8b" candidate_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" is_pre_election: true
I20260709 12:27:04.667092 29668 raft_consensus.cc:2468] T ef20d7124e7a44aa97a219ee194f4b8b P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6610c00ffb948ce9c34f41dec3d49e8 in term 0.
I20260709 12:27:04.668443 29709 leader_election.cc:304] T ef20d7124e7a44aa97a219ee194f4b8b P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c6610c00ffb948ce9c34f41dec3d49e8, d13c58ad8ac441c4a9b396eb04b1a476; no voters: 
I20260709 12:27:04.669449 30216 raft_consensus.cc:2804] T ef20d7124e7a44aa97a219ee194f4b8b P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:04.669870 30216 raft_consensus.cc:493] T ef20d7124e7a44aa97a219ee194f4b8b P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:04.670251 30216 raft_consensus.cc:3060] T ef20d7124e7a44aa97a219ee194f4b8b P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:04.671103 29518 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef20d7124e7a44aa97a219ee194f4b8b" candidate_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "584dac86183a41d9b3bea1c2085e62fa" is_pre_election: true
I20260709 12:27:04.671630 29790 tablet_bootstrap.cc:654] T a3dcbb9c64bf4b5bb6680e8cb81af93e P c6610c00ffb948ce9c34f41dec3d49e8: Neither blocks nor log segments found. Creating new log.
W20260709 12:27:04.672657 29708 leader_election.cc:343] T ef20d7124e7a44aa97a219ee194f4b8b P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:04.674311 29418 catalog_manager.cc:5697] T f76101eb30f14b98a9aa8c41e0d6f831 P d13c58ad8ac441c4a9b396eb04b1a476 reported cstate change: term changed from 0 to 1, leader changed from <none> to d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67). New cstate: current_term: 1 leader_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:04.676283 29595 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "137e03144e754f6e9da4f0544513d27c" candidate_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d73d92fb35234d16824c55689b5fce28" is_pre_election: true
I20260709 12:27:04.677008 29595 raft_consensus.cc:2468] T 137e03144e754f6e9da4f0544513d27c P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d13c58ad8ac441c4a9b396eb04b1a476 in term 0.
I20260709 12:27:04.680898 30216 raft_consensus.cc:515] T ef20d7124e7a44aa97a219ee194f4b8b P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:27:04.683975 29668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef20d7124e7a44aa97a219ee194f4b8b" candidate_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d13c58ad8ac441c4a9b396eb04b1a476"
I20260709 12:27:04.684633 29668 raft_consensus.cc:3060] T ef20d7124e7a44aa97a219ee194f4b8b P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:04.685815 29792 raft_consensus.cc:359] T ef20d7124e7a44aa97a219ee194f4b8b P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:27:04.686726 29792 raft_consensus.cc:385] T ef20d7124e7a44aa97a219ee194f4b8b P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:04.686991 29792 raft_consensus.cc:740] T ef20d7124e7a44aa97a219ee194f4b8b P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 584dac86183a41d9b3bea1c2085e62fa, State: Initialized, Role: FOLLOWER
I20260709 12:27:04.687667 30216 leader_election.cc:290] T ef20d7124e7a44aa97a219ee194f4b8b P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 1 election: Requested vote from peers d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209), 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033)
I20260709 12:27:04.687609 29792 consensus_queue.cc:260] T ef20d7124e7a44aa97a219ee194f4b8b P 584dac86183a41d9b3bea1c2085e62fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:27:04.689708 29597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8c01e05981747a395e5483b551297dd" candidate_uuid: "584dac86183a41d9b3bea1c2085e62fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d73d92fb35234d16824c55689b5fce28" is_pre_election: true
I20260709 12:27:04.691116 29792 ts_tablet_manager.cc:1434] T ef20d7124e7a44aa97a219ee194f4b8b P 584dac86183a41d9b3bea1c2085e62fa: Time spent starting tablet: real 0.059s	user 0.009s	sys 0.000s
I20260709 12:27:04.692410 29792 tablet_bootstrap.cc:492] T f3e17f0a8fc14095850e8138fd79b86e P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap starting.
I20260709 12:27:04.692824 29794 tablet_bootstrap.cc:492] T 0c428ff656ae4b9d99d5b5073f98725d P d13c58ad8ac441c4a9b396eb04b1a476: No bootstrap required, opened a new log
I20260709 12:27:04.693346 29794 ts_tablet_manager.cc:1403] T 0c428ff656ae4b9d99d5b5073f98725d P d13c58ad8ac441c4a9b396eb04b1a476: Time spent bootstrapping tablet: real 0.085s	user 0.007s	sys 0.003s
I20260709 12:27:04.702234 29668 raft_consensus.cc:2468] T ef20d7124e7a44aa97a219ee194f4b8b P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6610c00ffb948ce9c34f41dec3d49e8 in term 1.
I20260709 12:27:04.703558 29709 leader_election.cc:304] T ef20d7124e7a44aa97a219ee194f4b8b P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c6610c00ffb948ce9c34f41dec3d49e8, d13c58ad8ac441c4a9b396eb04b1a476; no voters: 
I20260709 12:27:04.704461 30046 raft_consensus.cc:2804] T ef20d7124e7a44aa97a219ee194f4b8b P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Leader election won for term 1
W20260709 12:27:04.704658 29560 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:35033, user_credentials={real_user=slave}} blocked reactor thread for 63901.7us
I20260709 12:27:04.705035 30046 raft_consensus.cc:697] T ef20d7124e7a44aa97a219ee194f4b8b P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 LEADER]: Becoming Leader. State: Replica: c6610c00ffb948ce9c34f41dec3d49e8, State: Running, Role: LEADER
I20260709 12:27:04.734710 30046 consensus_queue.cc:237] T ef20d7124e7a44aa97a219ee194f4b8b P c6610c00ffb948ce9c34f41dec3d49e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:27:04.706619 29520 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef20d7124e7a44aa97a219ee194f4b8b" candidate_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "584dac86183a41d9b3bea1c2085e62fa"
I20260709 12:27:04.709522 29794 raft_consensus.cc:359] T 0c428ff656ae4b9d99d5b5073f98725d P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:27:04.710525 29518 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d5fcd1f030614bbe82cf1dc261dfc1c3" candidate_uuid: "d73d92fb35234d16824c55689b5fce28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "584dac86183a41d9b3bea1c2085e62fa" is_pre_election: true
I20260709 12:27:04.718372 30213 consensus_queue.cc:1048] T 394378d078194257b9e04fc659c1f5d4 P c6610c00ffb948ce9c34f41dec3d49e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:27:04.738840 29794 raft_consensus.cc:385] T 0c428ff656ae4b9d99d5b5073f98725d P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:04.741323 29794 raft_consensus.cc:740] T 0c428ff656ae4b9d99d5b5073f98725d P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d13c58ad8ac441c4a9b396eb04b1a476, State: Initialized, Role: FOLLOWER
I20260709 12:27:04.742188 29794 consensus_queue.cc:260] T 0c428ff656ae4b9d99d5b5073f98725d P d13c58ad8ac441c4a9b396eb04b1a476 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:27:04.746443 29794 ts_tablet_manager.cc:1434] T 0c428ff656ae4b9d99d5b5073f98725d P d13c58ad8ac441c4a9b396eb04b1a476: Time spent starting tablet: real 0.053s	user 0.009s	sys 0.000s
I20260709 12:27:04.747550 29634 leader_election.cc:304] T 137e03144e754f6e9da4f0544513d27c P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d13c58ad8ac441c4a9b396eb04b1a476, d73d92fb35234d16824c55689b5fce28; no voters: c6610c00ffb948ce9c34f41dec3d49e8
I20260709 12:27:04.748034 29794 tablet_bootstrap.cc:492] T 394378d078194257b9e04fc659c1f5d4 P d13c58ad8ac441c4a9b396eb04b1a476: Bootstrap starting.
I20260709 12:27:04.748404 30203 raft_consensus.cc:2804] T 137e03144e754f6e9da4f0544513d27c P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:04.748847 30203 raft_consensus.cc:493] T 137e03144e754f6e9da4f0544513d27c P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:04.749284 30203 raft_consensus.cc:3060] T 137e03144e754f6e9da4f0544513d27c P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:04.758039 30203 raft_consensus.cc:515] T 137e03144e754f6e9da4f0544513d27c P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
W20260709 12:27:04.759099 29561 leader_election.cc:343] T 137e03144e754f6e9da4f0544513d27c P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:27:04.760609 29560 leader_election.cc:343] T d5fcd1f030614bbe82cf1dc261dfc1c3 P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:04.763197 30043 consensus_queue.cc:1048] T 3617af2dee3f48cf85973b1c94cdc33a P d13c58ad8ac441c4a9b396eb04b1a476 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:04.764114 29561 leader_election.cc:304] T d5fcd1f030614bbe82cf1dc261dfc1c3 P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d13c58ad8ac441c4a9b396eb04b1a476, d73d92fb35234d16824c55689b5fce28; no voters: 584dac86183a41d9b3bea1c2085e62fa
I20260709 12:27:04.729220 29485 leader_election.cc:304] T f8c01e05981747a395e5483b551297dd P 584dac86183a41d9b3bea1c2085e62fa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 584dac86183a41d9b3bea1c2085e62fa; no voters: c6610c00ffb948ce9c34f41dec3d49e8, d73d92fb35234d16824c55689b5fce28
I20260709 12:27:04.770908 29794 tablet_bootstrap.cc:654] T 394378d078194257b9e04fc659c1f5d4 P d13c58ad8ac441c4a9b396eb04b1a476: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:04.772158 30231 raft_consensus.cc:2804] T d5fcd1f030614bbe82cf1dc261dfc1c3 P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:04.772660 30231 raft_consensus.cc:493] T d5fcd1f030614bbe82cf1dc261dfc1c3 P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:04.772749 30068 raft_consensus.cc:3060] T f8c01e05981747a395e5483b551297dd P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:04.773058 30231 raft_consensus.cc:3060] T d5fcd1f030614bbe82cf1dc261dfc1c3 P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:04.776552 29419 catalog_manager.cc:5697] T a3dcbb9c64bf4b5bb6680e8cb81af93e P d73d92fb35234d16824c55689b5fce28 reported cstate change: term changed from 0 to 1, leader changed from <none> to d73d92fb35234d16824c55689b5fce28 (127.16.29.66). New cstate: current_term: 1 leader_uuid: "d73d92fb35234d16824c55689b5fce28" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } health_report { overall_health: HEALTHY } } }
I20260709 12:27:04.781739 30231 raft_consensus.cc:515] T d5fcd1f030614bbe82cf1dc261dfc1c3 P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:27:04.738687 29520 raft_consensus.cc:3060] T ef20d7124e7a44aa97a219ee194f4b8b P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:27:04.784027 29708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:36825, user_credentials={real_user=slave}} blocked reactor thread for 66379.9us
I20260709 12:27:04.795336 30068 raft_consensus.cc:2749] T f8c01e05981747a395e5483b551297dd P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260709 12:27:04.797812 29708 consensus_peers.cc:597] T 6c8b2fa9f2c044ec9eec97c72b561042 P c6610c00ffb948ce9c34f41dec3d49e8 -> Peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Couldn't send request to peer 584dac86183a41d9b3bea1c2085e62fa. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:04.791589 29520 raft_consensus.cc:2468] T ef20d7124e7a44aa97a219ee194f4b8b P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6610c00ffb948ce9c34f41dec3d49e8 in term 1.
I20260709 12:27:04.804531 29561 leader_election.cc:304] T 137e03144e754f6e9da4f0544513d27c P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d13c58ad8ac441c4a9b396eb04b1a476, d73d92fb35234d16824c55689b5fce28; no voters: c6610c00ffb948ce9c34f41dec3d49e8
I20260709 12:27:04.806161 30119 raft_consensus.cc:2804] T 137e03144e754f6e9da4f0544513d27c P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:04.806613 30119 raft_consensus.cc:493] T 137e03144e754f6e9da4f0544513d27c P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:04.807010 30119 raft_consensus.cc:3060] T 137e03144e754f6e9da4f0544513d27c P d73d92fb35234d16824c55689b5fce28 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:04.809010 30231 leader_election.cc:290] T d5fcd1f030614bbe82cf1dc261dfc1c3 P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 1 election: Requested vote from peers 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033), d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209)
I20260709 12:27:04.813632 29671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d5fcd1f030614bbe82cf1dc261dfc1c3" candidate_uuid: "d73d92fb35234d16824c55689b5fce28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d13c58ad8ac441c4a9b396eb04b1a476"
W20260709 12:27:04.800257 29485 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:36825, user_credentials={real_user=slave}} blocked reactor thread for 71138.7us
I20260709 12:27:04.814371 29671 raft_consensus.cc:3060] T d5fcd1f030614bbe82cf1dc261dfc1c3 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:04.814671 29744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "137e03144e754f6e9da4f0544513d27c" candidate_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6610c00ffb948ce9c34f41dec3d49e8"
W20260709 12:27:04.818676 29635 leader_election.cc:343] T 137e03144e754f6e9da4f0544513d27c P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:04.810525 30203 leader_election.cc:290] T 137e03144e754f6e9da4f0544513d27c P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 1 election: Requested vote from peers d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825), c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895)
I20260709 12:27:04.824608 29671 raft_consensus.cc:2468] T d5fcd1f030614bbe82cf1dc261dfc1c3 P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d73d92fb35234d16824c55689b5fce28 in term 1.
I20260709 12:27:04.733222 29792 tablet_bootstrap.cc:654] T f3e17f0a8fc14095850e8138fd79b86e P 584dac86183a41d9b3bea1c2085e62fa: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:04.829856 29790 tablet_bootstrap.cc:492] T a3dcbb9c64bf4b5bb6680e8cb81af93e P c6610c00ffb948ce9c34f41dec3d49e8: No bootstrap required, opened a new log
I20260709 12:27:04.830466 29790 ts_tablet_manager.cc:1403] T a3dcbb9c64bf4b5bb6680e8cb81af93e P c6610c00ffb948ce9c34f41dec3d49e8: Time spent bootstrapping tablet: real 0.227s	user 0.030s	sys 0.042s
I20260709 12:27:04.831557 29419 catalog_manager.cc:5697] T ef20d7124e7a44aa97a219ee194f4b8b P c6610c00ffb948ce9c34f41dec3d49e8 reported cstate change: term changed from 0 to 1, leader changed from <none> to c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68). New cstate: current_term: 1 leader_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:04.835500 29790 raft_consensus.cc:359] T a3dcbb9c64bf4b5bb6680e8cb81af93e P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:04.836558 29790 raft_consensus.cc:385] T a3dcbb9c64bf4b5bb6680e8cb81af93e P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:04.837024 29790 raft_consensus.cc:740] T a3dcbb9c64bf4b5bb6680e8cb81af93e P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6610c00ffb948ce9c34f41dec3d49e8, State: Initialized, Role: FOLLOWER
I20260709 12:27:04.840178 29592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "137e03144e754f6e9da4f0544513d27c" candidate_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d73d92fb35234d16824c55689b5fce28"
I20260709 12:27:04.846742 30119 raft_consensus.cc:515] T 137e03144e754f6e9da4f0544513d27c P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:27:04.848155 29592 raft_consensus.cc:2393] T 137e03144e754f6e9da4f0544513d27c P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate d13c58ad8ac441c4a9b396eb04b1a476 in current term 1: Already voted for candidate d73d92fb35234d16824c55689b5fce28 in this term.
I20260709 12:27:04.848753 30242 consensus_queue.cc:1048] T 6c8b2fa9f2c044ec9eec97c72b561042 P c6610c00ffb948ce9c34f41dec3d49e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:27:04.850062 29520 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d5fcd1f030614bbe82cf1dc261dfc1c3" candidate_uuid: "d73d92fb35234d16824c55689b5fce28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "584dac86183a41d9b3bea1c2085e62fa"
I20260709 12:27:04.838150 29790 consensus_queue.cc:260] T a3dcbb9c64bf4b5bb6680e8cb81af93e P c6610c00ffb948ce9c34f41dec3d49e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:04.854665 30119 leader_election.cc:290] T 137e03144e754f6e9da4f0544513d27c P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 1 election: Requested vote from peers d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209), c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895)
I20260709 12:27:04.856370 29790 ts_tablet_manager.cc:1434] T a3dcbb9c64bf4b5bb6680e8cb81af93e P c6610c00ffb948ce9c34f41dec3d49e8: Time spent starting tablet: real 0.026s	user 0.007s	sys 0.000s
I20260709 12:27:04.856719 29561 leader_election.cc:304] T d5fcd1f030614bbe82cf1dc261dfc1c3 P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d13c58ad8ac441c4a9b396eb04b1a476, d73d92fb35234d16824c55689b5fce28; no voters: 
I20260709 12:27:04.857291 30231 consensus_queue.cc:1048] T f8c01e05981747a395e5483b551297dd P d73d92fb35234d16824c55689b5fce28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:27:04.857944 30003 raft_consensus.cc:2804] T d5fcd1f030614bbe82cf1dc261dfc1c3 P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:04.882174 30003 raft_consensus.cc:697] T d5fcd1f030614bbe82cf1dc261dfc1c3 P d73d92fb35234d16824c55689b5fce28 [term 1 LEADER]: Becoming Leader. State: Replica: d73d92fb35234d16824c55689b5fce28, State: Running, Role: LEADER
I20260709 12:27:04.883071 30003 consensus_queue.cc:237] T d5fcd1f030614bbe82cf1dc261dfc1c3 P d73d92fb35234d16824c55689b5fce28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:27:04.884366 29634 leader_election.cc:304] T 137e03144e754f6e9da4f0544513d27c P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d13c58ad8ac441c4a9b396eb04b1a476; no voters: c6610c00ffb948ce9c34f41dec3d49e8, d73d92fb35234d16824c55689b5fce28
W20260709 12:27:04.864423 29560 leader_election.cc:343] T d5fcd1f030614bbe82cf1dc261dfc1c3 P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:04.885134 29668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "137e03144e754f6e9da4f0544513d27c" candidate_uuid: "d73d92fb35234d16824c55689b5fce28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d13c58ad8ac441c4a9b396eb04b1a476"
I20260709 12:27:04.885504 30203 raft_consensus.cc:2749] T 137e03144e754f6e9da4f0544513d27c P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 12:27:04.886462 29668 raft_consensus.cc:2393] T 137e03144e754f6e9da4f0544513d27c P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate d73d92fb35234d16824c55689b5fce28 in current term 1: Already voted for candidate d13c58ad8ac441c4a9b396eb04b1a476 in this term.
I20260709 12:27:04.864435 30231 consensus_queue.cc:1048] T f8c01e05981747a395e5483b551297dd P d73d92fb35234d16824c55689b5fce28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.022s
I20260709 12:27:04.886996 29744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "137e03144e754f6e9da4f0544513d27c" candidate_uuid: "d73d92fb35234d16824c55689b5fce28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6610c00ffb948ce9c34f41dec3d49e8"
I20260709 12:27:04.892218 29792 tablet_bootstrap.cc:492] T f3e17f0a8fc14095850e8138fd79b86e P 584dac86183a41d9b3bea1c2085e62fa: No bootstrap required, opened a new log
I20260709 12:27:04.892947 29792 ts_tablet_manager.cc:1403] T f3e17f0a8fc14095850e8138fd79b86e P 584dac86183a41d9b3bea1c2085e62fa: Time spent bootstrapping tablet: real 0.201s	user 0.021s	sys 0.003s
I20260709 12:27:04.879900 29794 tablet_bootstrap.cc:492] T 394378d078194257b9e04fc659c1f5d4 P d13c58ad8ac441c4a9b396eb04b1a476: No bootstrap required, opened a new log
I20260709 12:27:04.894989 29794 ts_tablet_manager.cc:1403] T 394378d078194257b9e04fc659c1f5d4 P d13c58ad8ac441c4a9b396eb04b1a476: Time spent bootstrapping tablet: real 0.147s	user 0.012s	sys 0.012s
I20260709 12:27:04.857431 29790 tablet_bootstrap.cc:492] T 137e03144e754f6e9da4f0544513d27c P c6610c00ffb948ce9c34f41dec3d49e8: Bootstrap starting.
I20260709 12:27:04.897882 29794 raft_consensus.cc:359] T 394378d078194257b9e04fc659c1f5d4 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:27:04.898675 29794 raft_consensus.cc:385] T 394378d078194257b9e04fc659c1f5d4 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:04.898923 29794 raft_consensus.cc:740] T 394378d078194257b9e04fc659c1f5d4 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d13c58ad8ac441c4a9b396eb04b1a476, State: Initialized, Role: FOLLOWER
I20260709 12:27:04.906277 29790 tablet_bootstrap.cc:654] T 137e03144e754f6e9da4f0544513d27c P c6610c00ffb948ce9c34f41dec3d49e8: Neither blocks nor log segments found. Creating new log.
W20260709 12:27:04.917629 29561 leader_election.cc:343] T 137e03144e754f6e9da4f0544513d27c P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:04.918668 29561 leader_election.cc:304] T 137e03144e754f6e9da4f0544513d27c P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d73d92fb35234d16824c55689b5fce28; no voters: c6610c00ffb948ce9c34f41dec3d49e8, d13c58ad8ac441c4a9b396eb04b1a476
I20260709 12:27:04.922341 30046 raft_consensus.cc:493] T a3dcbb9c64bf4b5bb6680e8cb81af93e P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:04.922911 30046 raft_consensus.cc:515] T a3dcbb9c64bf4b5bb6680e8cb81af93e P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:04.924295 30231 raft_consensus.cc:2749] T 137e03144e754f6e9da4f0544513d27c P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 12:27:04.925149 30046 leader_election.cc:290] T a3dcbb9c64bf4b5bb6680e8cb81af93e P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033), d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825)
I20260709 12:27:04.926812 29794 consensus_queue.cc:260] T 394378d078194257b9e04fc659c1f5d4 P d13c58ad8ac441c4a9b396eb04b1a476 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:27:04.939047 29520 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3dcbb9c64bf4b5bb6680e8cb81af93e" candidate_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "584dac86183a41d9b3bea1c2085e62fa" is_pre_election: true
I20260709 12:27:04.941829 29520 raft_consensus.cc:2393] T a3dcbb9c64bf4b5bb6680e8cb81af93e P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c6610c00ffb948ce9c34f41dec3d49e8 in current term 1: Already voted for candidate d73d92fb35234d16824c55689b5fce28 in this term.
I20260709 12:27:04.915295 29792 raft_consensus.cc:359] T f3e17f0a8fc14095850e8138fd79b86e P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:27:04.947202 29792 raft_consensus.cc:385] T f3e17f0a8fc14095850e8138fd79b86e P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:04.947562 29792 raft_consensus.cc:740] T f3e17f0a8fc14095850e8138fd79b86e P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 584dac86183a41d9b3bea1c2085e62fa, State: Initialized, Role: FOLLOWER
I20260709 12:27:04.948426 30203 raft_consensus.cc:493] T 394378d078194257b9e04fc659c1f5d4 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:04.949010 30203 raft_consensus.cc:515] T 394378d078194257b9e04fc659c1f5d4 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:27:04.954589 30203 leader_election.cc:290] T 394378d078194257b9e04fc659c1f5d4 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825), c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895)
I20260709 12:27:04.956585 29741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "394378d078194257b9e04fc659c1f5d4" candidate_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" is_pre_election: true
I20260709 12:27:04.964893 29590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3dcbb9c64bf4b5bb6680e8cb81af93e" candidate_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d73d92fb35234d16824c55689b5fce28" is_pre_election: true
I20260709 12:27:04.960709 29792 consensus_queue.cc:260] T f3e17f0a8fc14095850e8138fd79b86e P 584dac86183a41d9b3bea1c2085e62fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:27:04.974112 29792 ts_tablet_manager.cc:1434] T f3e17f0a8fc14095850e8138fd79b86e P 584dac86183a41d9b3bea1c2085e62fa: Time spent starting tablet: real 0.081s	user 0.004s	sys 0.002s
I20260709 12:27:04.975200 29792 tablet_bootstrap.cc:492] T d5fcd1f030614bbe82cf1dc261dfc1c3 P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap starting.
I20260709 12:27:04.977006 29708 leader_election.cc:304] T a3dcbb9c64bf4b5bb6680e8cb81af93e P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c6610c00ffb948ce9c34f41dec3d49e8; no voters: 584dac86183a41d9b3bea1c2085e62fa, d73d92fb35234d16824c55689b5fce28
I20260709 12:27:04.977727 29590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "394378d078194257b9e04fc659c1f5d4" candidate_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d73d92fb35234d16824c55689b5fce28" is_pre_election: true
I20260709 12:27:04.975647 29419 catalog_manager.cc:5697] T d5fcd1f030614bbe82cf1dc261dfc1c3 P d73d92fb35234d16824c55689b5fce28 reported cstate change: term changed from 0 to 1, leader changed from <none> to d73d92fb35234d16824c55689b5fce28 (127.16.29.66). New cstate: current_term: 1 leader_uuid: "d73d92fb35234d16824c55689b5fce28" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:04.980762 30242 raft_consensus.cc:3060] T a3dcbb9c64bf4b5bb6680e8cb81af93e P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:04.981982 29634 leader_election.cc:304] T 394378d078194257b9e04fc659c1f5d4 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d13c58ad8ac441c4a9b396eb04b1a476; no voters: c6610c00ffb948ce9c34f41dec3d49e8, d73d92fb35234d16824c55689b5fce28
I20260709 12:27:04.982452 29792 tablet_bootstrap.cc:654] T d5fcd1f030614bbe82cf1dc261dfc1c3 P 584dac86183a41d9b3bea1c2085e62fa: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:04.983147 30203 raft_consensus.cc:2749] T 394378d078194257b9e04fc659c1f5d4 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:04.990705 29790 tablet_bootstrap.cc:492] T 137e03144e754f6e9da4f0544513d27c P c6610c00ffb948ce9c34f41dec3d49e8: No bootstrap required, opened a new log
I20260709 12:27:04.991506 29790 ts_tablet_manager.cc:1403] T 137e03144e754f6e9da4f0544513d27c P c6610c00ffb948ce9c34f41dec3d49e8: Time spent bootstrapping tablet: real 0.134s	user 0.027s	sys 0.029s
I20260709 12:27:05.002180 29794 ts_tablet_manager.cc:1434] T 394378d078194257b9e04fc659c1f5d4 P d13c58ad8ac441c4a9b396eb04b1a476: Time spent starting tablet: real 0.107s	user 0.010s	sys 0.000s
I20260709 12:27:05.003750 29790 raft_consensus.cc:359] T 137e03144e754f6e9da4f0544513d27c P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:27:05.004608 29790 raft_consensus.cc:385] T 137e03144e754f6e9da4f0544513d27c P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:05.004904 29790 raft_consensus.cc:740] T 137e03144e754f6e9da4f0544513d27c P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6610c00ffb948ce9c34f41dec3d49e8, State: Initialized, Role: FOLLOWER
I20260709 12:27:05.005651 29790 consensus_queue.cc:260] T 137e03144e754f6e9da4f0544513d27c P c6610c00ffb948ce9c34f41dec3d49e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:27:05.009660 29668 raft_consensus.cc:3060] T 394378d078194257b9e04fc659c1f5d4 P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:05.008287 29790 ts_tablet_manager.cc:1434] T 137e03144e754f6e9da4f0544513d27c P c6610c00ffb948ce9c34f41dec3d49e8: Time spent starting tablet: real 0.016s	user 0.017s	sys 0.000s
W20260709 12:27:05.024658 29708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:36825, user_credentials={real_user=slave}} blocked reactor thread for 44112.2us
I20260709 12:27:05.026024 29792 tablet_bootstrap.cc:492] T d5fcd1f030614bbe82cf1dc261dfc1c3 P 584dac86183a41d9b3bea1c2085e62fa: No bootstrap required, opened a new log
I20260709 12:27:05.026603 29792 ts_tablet_manager.cc:1403] T d5fcd1f030614bbe82cf1dc261dfc1c3 P 584dac86183a41d9b3bea1c2085e62fa: Time spent bootstrapping tablet: real 0.052s	user 0.015s	sys 0.018s
I20260709 12:27:05.029747 29792 raft_consensus.cc:359] T d5fcd1f030614bbe82cf1dc261dfc1c3 P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:27:05.039381 30242 raft_consensus.cc:2749] T a3dcbb9c64bf4b5bb6680e8cb81af93e P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:05.039940 29792 raft_consensus.cc:385] T d5fcd1f030614bbe82cf1dc261dfc1c3 P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:05.040336 29792 raft_consensus.cc:740] T d5fcd1f030614bbe82cf1dc261dfc1c3 P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 584dac86183a41d9b3bea1c2085e62fa, State: Initialized, Role: FOLLOWER
I20260709 12:27:05.041072 29792 consensus_queue.cc:260] T d5fcd1f030614bbe82cf1dc261dfc1c3 P 584dac86183a41d9b3bea1c2085e62fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:27:05.044543 29792 ts_tablet_manager.cc:1434] T d5fcd1f030614bbe82cf1dc261dfc1c3 P 584dac86183a41d9b3bea1c2085e62fa: Time spent starting tablet: real 0.018s	user 0.009s	sys 0.001s
I20260709 12:27:05.045675 29792 tablet_bootstrap.cc:492] T 6c8b2fa9f2c044ec9eec97c72b561042 P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap starting.
I20260709 12:27:05.060190 30203 raft_consensus.cc:493] T 0c428ff656ae4b9d99d5b5073f98725d P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:05.060762 30203 raft_consensus.cc:515] T 0c428ff656ae4b9d99d5b5073f98725d P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:27:05.062774 30203 leader_election.cc:290] T 0c428ff656ae4b9d99d5b5073f98725d P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895), 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033)
I20260709 12:27:05.065948 29519 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c428ff656ae4b9d99d5b5073f98725d" candidate_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "584dac86183a41d9b3bea1c2085e62fa" is_pre_election: true
W20260709 12:27:05.068673 29634 leader_election.cc:343] T 0c428ff656ae4b9d99d5b5073f98725d P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:27:05.069371 29634 consensus_peers.cc:597] T f76101eb30f14b98a9aa8c41e0d6f831 P d13c58ad8ac441c4a9b396eb04b1a476 -> Peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Couldn't send request to peer 584dac86183a41d9b3bea1c2085e62fa. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:05.072875 29741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c428ff656ae4b9d99d5b5073f98725d" candidate_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" is_pre_election: true
I20260709 12:27:05.073629 29741 raft_consensus.cc:2468] T 0c428ff656ae4b9d99d5b5073f98725d P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d13c58ad8ac441c4a9b396eb04b1a476 in term 0.
I20260709 12:27:05.075868 29635 leader_election.cc:304] T 0c428ff656ae4b9d99d5b5073f98725d P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c6610c00ffb948ce9c34f41dec3d49e8, d13c58ad8ac441c4a9b396eb04b1a476; no voters: 584dac86183a41d9b3bea1c2085e62fa
I20260709 12:27:05.077214 30203 raft_consensus.cc:2804] T 0c428ff656ae4b9d99d5b5073f98725d P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:05.077791 30203 raft_consensus.cc:493] T 0c428ff656ae4b9d99d5b5073f98725d P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:05.078286 30203 raft_consensus.cc:3060] T 0c428ff656ae4b9d99d5b5073f98725d P d13c58ad8ac441c4a9b396eb04b1a476 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:05.079404 29792 tablet_bootstrap.cc:654] T 6c8b2fa9f2c044ec9eec97c72b561042 P 584dac86183a41d9b3bea1c2085e62fa: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:05.117955 30203 raft_consensus.cc:515] T 0c428ff656ae4b9d99d5b5073f98725d P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
W20260709 12:27:05.119081 29467 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:27:05.143462 29741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c428ff656ae4b9d99d5b5073f98725d" candidate_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6610c00ffb948ce9c34f41dec3d49e8"
I20260709 12:27:05.144140 29741 raft_consensus.cc:3060] T 0c428ff656ae4b9d99d5b5073f98725d P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:05.144488 29520 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c428ff656ae4b9d99d5b5073f98725d" candidate_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "584dac86183a41d9b3bea1c2085e62fa"
W20260709 12:27:05.148466 29634 leader_election.cc:343] T 0c428ff656ae4b9d99d5b5073f98725d P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:05.152256 29741 raft_consensus.cc:2468] T 0c428ff656ae4b9d99d5b5073f98725d P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d13c58ad8ac441c4a9b396eb04b1a476 in term 1.
I20260709 12:27:05.153364 29635 leader_election.cc:304] T 0c428ff656ae4b9d99d5b5073f98725d P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c6610c00ffb948ce9c34f41dec3d49e8, d13c58ad8ac441c4a9b396eb04b1a476; no voters: 584dac86183a41d9b3bea1c2085e62fa
I20260709 12:27:05.142827 30203 leader_election.cc:290] T 0c428ff656ae4b9d99d5b5073f98725d P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 1 election: Requested vote from peers c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895), 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033)
I20260709 12:27:05.154263 30072 raft_consensus.cc:2804] T 0c428ff656ae4b9d99d5b5073f98725d P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:05.154863 30072 raft_consensus.cc:697] T 0c428ff656ae4b9d99d5b5073f98725d P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 LEADER]: Becoming Leader. State: Replica: d13c58ad8ac441c4a9b396eb04b1a476, State: Running, Role: LEADER
I20260709 12:27:05.155828 29792 tablet_bootstrap.cc:492] T 6c8b2fa9f2c044ec9eec97c72b561042 P 584dac86183a41d9b3bea1c2085e62fa: No bootstrap required, opened a new log
I20260709 12:27:05.156435 29792 ts_tablet_manager.cc:1403] T 6c8b2fa9f2c044ec9eec97c72b561042 P 584dac86183a41d9b3bea1c2085e62fa: Time spent bootstrapping tablet: real 0.111s	user 0.036s	sys 0.024s
I20260709 12:27:05.155762 30072 consensus_queue.cc:237] T 0c428ff656ae4b9d99d5b5073f98725d P d13c58ad8ac441c4a9b396eb04b1a476 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:27:05.160876 30231 consensus_queue.cc:1048] T a3dcbb9c64bf4b5bb6680e8cb81af93e P d73d92fb35234d16824c55689b5fce28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:05.162420 29792 raft_consensus.cc:359] T 6c8b2fa9f2c044ec9eec97c72b561042 P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:05.163259 29792 raft_consensus.cc:385] T 6c8b2fa9f2c044ec9eec97c72b561042 P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:05.163625 29792 raft_consensus.cc:740] T 6c8b2fa9f2c044ec9eec97c72b561042 P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 584dac86183a41d9b3bea1c2085e62fa, State: Initialized, Role: FOLLOWER
I20260709 12:27:05.164467 29792 consensus_queue.cc:260] T 6c8b2fa9f2c044ec9eec97c72b561042 P 584dac86183a41d9b3bea1c2085e62fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:05.166980 29792 ts_tablet_manager.cc:1434] T 6c8b2fa9f2c044ec9eec97c72b561042 P 584dac86183a41d9b3bea1c2085e62fa: Time spent starting tablet: real 0.010s	user 0.006s	sys 0.000s
I20260709 12:27:05.175549 29419 catalog_manager.cc:5697] T 0c428ff656ae4b9d99d5b5073f98725d P d13c58ad8ac441c4a9b396eb04b1a476 reported cstate change: term changed from 0 to 1, leader changed from <none> to d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67). New cstate: current_term: 1 leader_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:05.176115 30203 consensus_queue.cc:1048] T f76101eb30f14b98a9aa8c41e0d6f831 P d13c58ad8ac441c4a9b396eb04b1a476 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:05.181376 30038 raft_consensus.cc:493] T d5fcd1f030614bbe82cf1dc261dfc1c3 P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:05.182086 30038 raft_consensus.cc:515] T d5fcd1f030614bbe82cf1dc261dfc1c3 P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:27:05.183346 29792 tablet_bootstrap.cc:492] T 0c428ff656ae4b9d99d5b5073f98725d P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap starting.
I20260709 12:27:05.184697 30038 leader_election.cc:290] T d5fcd1f030614bbe82cf1dc261dfc1c3 P 584dac86183a41d9b3bea1c2085e62fa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825), d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209)
I20260709 12:27:05.187851 29671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d5fcd1f030614bbe82cf1dc261dfc1c3" candidate_uuid: "584dac86183a41d9b3bea1c2085e62fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" is_pre_election: true
I20260709 12:27:05.188647 29671 raft_consensus.cc:2393] T d5fcd1f030614bbe82cf1dc261dfc1c3 P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 584dac86183a41d9b3bea1c2085e62fa in current term 1: Already voted for candidate d73d92fb35234d16824c55689b5fce28 in this term.
I20260709 12:27:05.191051 29792 tablet_bootstrap.cc:654] T 0c428ff656ae4b9d99d5b5073f98725d P 584dac86183a41d9b3bea1c2085e62fa: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:05.217921 29590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d5fcd1f030614bbe82cf1dc261dfc1c3" candidate_uuid: "584dac86183a41d9b3bea1c2085e62fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d73d92fb35234d16824c55689b5fce28" is_pre_election: true
I20260709 12:27:05.233151 29485 leader_election.cc:304] T d5fcd1f030614bbe82cf1dc261dfc1c3 P 584dac86183a41d9b3bea1c2085e62fa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 584dac86183a41d9b3bea1c2085e62fa; no voters: d13c58ad8ac441c4a9b396eb04b1a476, d73d92fb35234d16824c55689b5fce28
I20260709 12:27:05.234325 30071 raft_consensus.cc:493] T 6c8b2fa9f2c044ec9eec97c72b561042 P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:05.234674 29792 tablet_bootstrap.cc:492] T 0c428ff656ae4b9d99d5b5073f98725d P 584dac86183a41d9b3bea1c2085e62fa: No bootstrap required, opened a new log
I20260709 12:27:05.235241 29792 ts_tablet_manager.cc:1403] T 0c428ff656ae4b9d99d5b5073f98725d P 584dac86183a41d9b3bea1c2085e62fa: Time spent bootstrapping tablet: real 0.063s	user 0.012s	sys 0.012s
I20260709 12:27:05.234875 30071 raft_consensus.cc:515] T 6c8b2fa9f2c044ec9eec97c72b561042 P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:05.235405 30038 raft_consensus.cc:3060] T d5fcd1f030614bbe82cf1dc261dfc1c3 P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:05.237185 30071 leader_election.cc:290] T 6c8b2fa9f2c044ec9eec97c72b561042 P 584dac86183a41d9b3bea1c2085e62fa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895), d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825)
I20260709 12:27:05.237866 29741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c8b2fa9f2c044ec9eec97c72b561042" candidate_uuid: "584dac86183a41d9b3bea1c2085e62fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" is_pre_election: true
I20260709 12:27:05.240971 30231 consensus_queue.cc:1048] T a3dcbb9c64bf4b5bb6680e8cb81af93e P d73d92fb35234d16824c55689b5fce28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:05.241209 29792 raft_consensus.cc:359] T 0c428ff656ae4b9d99d5b5073f98725d P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:27:05.241539 29590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c8b2fa9f2c044ec9eec97c72b561042" candidate_uuid: "584dac86183a41d9b3bea1c2085e62fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d73d92fb35234d16824c55689b5fce28" is_pre_election: true
I20260709 12:27:05.242167 29792 raft_consensus.cc:385] T 0c428ff656ae4b9d99d5b5073f98725d P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:05.242493 29792 raft_consensus.cc:740] T 0c428ff656ae4b9d99d5b5073f98725d P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 584dac86183a41d9b3bea1c2085e62fa, State: Initialized, Role: FOLLOWER
I20260709 12:27:05.243228 29792 consensus_queue.cc:260] T 0c428ff656ae4b9d99d5b5073f98725d P 584dac86183a41d9b3bea1c2085e62fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:27:05.245455 29792 ts_tablet_manager.cc:1434] T 0c428ff656ae4b9d99d5b5073f98725d P 584dac86183a41d9b3bea1c2085e62fa: Time spent starting tablet: real 0.010s	user 0.003s	sys 0.003s
I20260709 12:27:05.245779 30038 raft_consensus.cc:2749] T d5fcd1f030614bbe82cf1dc261dfc1c3 P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:05.247026 29792 tablet_bootstrap.cc:492] T f76101eb30f14b98a9aa8c41e0d6f831 P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap starting.
I20260709 12:27:05.254370 29792 tablet_bootstrap.cc:654] T f76101eb30f14b98a9aa8c41e0d6f831 P 584dac86183a41d9b3bea1c2085e62fa: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:05.255149 29485 leader_election.cc:304] T 6c8b2fa9f2c044ec9eec97c72b561042 P 584dac86183a41d9b3bea1c2085e62fa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 584dac86183a41d9b3bea1c2085e62fa; no voters: c6610c00ffb948ce9c34f41dec3d49e8, d73d92fb35234d16824c55689b5fce28
I20260709 12:27:05.257929 30071 raft_consensus.cc:2749] T 6c8b2fa9f2c044ec9eec97c72b561042 P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:05.296624 29792 tablet_bootstrap.cc:492] T f76101eb30f14b98a9aa8c41e0d6f831 P 584dac86183a41d9b3bea1c2085e62fa: No bootstrap required, opened a new log
I20260709 12:27:05.297480 29792 ts_tablet_manager.cc:1403] T f76101eb30f14b98a9aa8c41e0d6f831 P 584dac86183a41d9b3bea1c2085e62fa: Time spent bootstrapping tablet: real 0.051s	user 0.032s	sys 0.014s
I20260709 12:27:05.296630 30166 consensus_queue.cc:1048] T d5fcd1f030614bbe82cf1dc261dfc1c3 P d73d92fb35234d16824c55689b5fce28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:05.299326 30046 consensus_queue.cc:1048] T ef20d7124e7a44aa97a219ee194f4b8b P c6610c00ffb948ce9c34f41dec3d49e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:05.300151 29520 raft_consensus.cc:3060] T 6c8b2fa9f2c044ec9eec97c72b561042 P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:05.303455 29792 raft_consensus.cc:359] T f76101eb30f14b98a9aa8c41e0d6f831 P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:27:05.304301 29792 raft_consensus.cc:385] T f76101eb30f14b98a9aa8c41e0d6f831 P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:05.304607 29792 raft_consensus.cc:740] T f76101eb30f14b98a9aa8c41e0d6f831 P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 584dac86183a41d9b3bea1c2085e62fa, State: Initialized, Role: FOLLOWER
I20260709 12:27:05.305336 29792 consensus_queue.cc:260] T f76101eb30f14b98a9aa8c41e0d6f831 P 584dac86183a41d9b3bea1c2085e62fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:27:05.307523 29792 ts_tablet_manager.cc:1434] T f76101eb30f14b98a9aa8c41e0d6f831 P 584dac86183a41d9b3bea1c2085e62fa: Time spent starting tablet: real 0.010s	user 0.006s	sys 0.000s
I20260709 12:27:05.322500 30046 consensus_queue.cc:1048] T ef20d7124e7a44aa97a219ee194f4b8b P c6610c00ffb948ce9c34f41dec3d49e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:27:05.328387 30231 consensus_queue.cc:1048] T d5fcd1f030614bbe82cf1dc261dfc1c3 P d73d92fb35234d16824c55689b5fce28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:27:05.398120 30038 raft_consensus.cc:493] T f3e17f0a8fc14095850e8138fd79b86e P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:05.398710 30038 raft_consensus.cc:515] T f3e17f0a8fc14095850e8138fd79b86e P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:27:05.402276 30038 leader_election.cc:290] T f3e17f0a8fc14095850e8138fd79b86e P 584dac86183a41d9b3bea1c2085e62fa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825), d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209)
I20260709 12:27:05.402446 29671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f3e17f0a8fc14095850e8138fd79b86e" candidate_uuid: "584dac86183a41d9b3bea1c2085e62fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" is_pre_election: true
I20260709 12:27:05.403228 29671 raft_consensus.cc:2393] T f3e17f0a8fc14095850e8138fd79b86e P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 584dac86183a41d9b3bea1c2085e62fa in current term 1: Already voted for candidate d13c58ad8ac441c4a9b396eb04b1a476 in this term.
I20260709 12:27:05.404628 29597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f3e17f0a8fc14095850e8138fd79b86e" candidate_uuid: "584dac86183a41d9b3bea1c2085e62fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d73d92fb35234d16824c55689b5fce28" is_pre_election: true
I20260709 12:27:05.405750 29597 raft_consensus.cc:2393] T f3e17f0a8fc14095850e8138fd79b86e P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 584dac86183a41d9b3bea1c2085e62fa in current term 1: Already voted for candidate d73d92fb35234d16824c55689b5fce28 in this term.
I20260709 12:27:05.409607 29485 leader_election.cc:304] T f3e17f0a8fc14095850e8138fd79b86e P 584dac86183a41d9b3bea1c2085e62fa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 584dac86183a41d9b3bea1c2085e62fa; no voters: d13c58ad8ac441c4a9b396eb04b1a476, d73d92fb35234d16824c55689b5fce28
I20260709 12:27:05.410473 30038 raft_consensus.cc:3060] T f3e17f0a8fc14095850e8138fd79b86e P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:05.415273 30213 raft_consensus.cc:493] T 137e03144e754f6e9da4f0544513d27c P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:05.416018 30213 raft_consensus.cc:515] T 137e03144e754f6e9da4f0544513d27c P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:27:05.418030 30038 raft_consensus.cc:2749] T f3e17f0a8fc14095850e8138fd79b86e P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:05.418150 30213 leader_election.cc:290] T 137e03144e754f6e9da4f0544513d27c P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825), d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209)
I20260709 12:27:05.419193 29671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "137e03144e754f6e9da4f0544513d27c" candidate_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" is_pre_election: true
I20260709 12:27:05.419554 29590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "137e03144e754f6e9da4f0544513d27c" candidate_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d73d92fb35234d16824c55689b5fce28" is_pre_election: true
I20260709 12:27:05.419762 30071 raft_consensus.cc:493] T f76101eb30f14b98a9aa8c41e0d6f831 P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:05.420512 29590 raft_consensus.cc:2393] T 137e03144e754f6e9da4f0544513d27c P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c6610c00ffb948ce9c34f41dec3d49e8 in current term 1: Already voted for candidate d73d92fb35234d16824c55689b5fce28 in this term.
I20260709 12:27:05.420356 30071 raft_consensus.cc:515] T f76101eb30f14b98a9aa8c41e0d6f831 P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:27:05.420312 29671 raft_consensus.cc:2393] T 137e03144e754f6e9da4f0544513d27c P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c6610c00ffb948ce9c34f41dec3d49e8 in current term 1: Already voted for candidate d13c58ad8ac441c4a9b396eb04b1a476 in this term.
I20260709 12:27:05.422480 30071 leader_election.cc:290] T f76101eb30f14b98a9aa8c41e0d6f831 P 584dac86183a41d9b3bea1c2085e62fa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209), c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895)
I20260709 12:27:05.423249 29668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f76101eb30f14b98a9aa8c41e0d6f831" candidate_uuid: "584dac86183a41d9b3bea1c2085e62fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" is_pre_election: true
I20260709 12:27:05.424016 29709 leader_election.cc:304] T 137e03144e754f6e9da4f0544513d27c P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c6610c00ffb948ce9c34f41dec3d49e8; no voters: d13c58ad8ac441c4a9b396eb04b1a476, d73d92fb35234d16824c55689b5fce28
I20260709 12:27:05.424420 29741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f76101eb30f14b98a9aa8c41e0d6f831" candidate_uuid: "584dac86183a41d9b3bea1c2085e62fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" is_pre_election: true
I20260709 12:27:05.425328 30213 raft_consensus.cc:3060] T 137e03144e754f6e9da4f0544513d27c P c6610c00ffb948ce9c34f41dec3d49e8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:05.425839 29486 leader_election.cc:304] T f76101eb30f14b98a9aa8c41e0d6f831 P 584dac86183a41d9b3bea1c2085e62fa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 584dac86183a41d9b3bea1c2085e62fa; no voters: c6610c00ffb948ce9c34f41dec3d49e8, d13c58ad8ac441c4a9b396eb04b1a476
I20260709 12:27:05.426513 30071 raft_consensus.cc:2749] T f76101eb30f14b98a9aa8c41e0d6f831 P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:05.432091 30213 raft_consensus.cc:2749] T 137e03144e754f6e9da4f0544513d27c P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:05.517580 30038 raft_consensus.cc:493] T 0c428ff656ae4b9d99d5b5073f98725d P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:05.518149 30038 raft_consensus.cc:515] T 0c428ff656ae4b9d99d5b5073f98725d P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:27:05.519804 30038 leader_election.cc:290] T 0c428ff656ae4b9d99d5b5073f98725d P 584dac86183a41d9b3bea1c2085e62fa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209), c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895)
I20260709 12:27:05.520866 29744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c428ff656ae4b9d99d5b5073f98725d" candidate_uuid: "584dac86183a41d9b3bea1c2085e62fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" is_pre_election: true
I20260709 12:27:05.521031 29668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c428ff656ae4b9d99d5b5073f98725d" candidate_uuid: "584dac86183a41d9b3bea1c2085e62fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" is_pre_election: true
I20260709 12:27:05.521657 29744 raft_consensus.cc:2393] T 0c428ff656ae4b9d99d5b5073f98725d P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 584dac86183a41d9b3bea1c2085e62fa in current term 1: Already voted for candidate d13c58ad8ac441c4a9b396eb04b1a476 in this term.
I20260709 12:27:05.522572 29486 leader_election.cc:304] T 0c428ff656ae4b9d99d5b5073f98725d P 584dac86183a41d9b3bea1c2085e62fa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 584dac86183a41d9b3bea1c2085e62fa; no voters: c6610c00ffb948ce9c34f41dec3d49e8, d13c58ad8ac441c4a9b396eb04b1a476
I20260709 12:27:05.523223 30038 raft_consensus.cc:3060] T 0c428ff656ae4b9d99d5b5073f98725d P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:05.530896 30038 raft_consensus.cc:2749] T 0c428ff656ae4b9d99d5b5073f98725d P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:05.558118 30203 consensus_queue.cc:1048] T 0c428ff656ae4b9d99d5b5073f98725d P d13c58ad8ac441c4a9b396eb04b1a476 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:27:05.572260 30203 consensus_queue.cc:1048] T 0c428ff656ae4b9d99d5b5073f98725d P d13c58ad8ac441c4a9b396eb04b1a476 [LEADER]: Connected to new peer: Peer: permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260709 12:27:05.595180 29520 raft_consensus.cc:3060] T f76101eb30f14b98a9aa8c41e0d6f831 P 584dac86183a41d9b3bea1c2085e62fa [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:05.852551 30138 raft_consensus.cc:493] T f3e17f0a8fc14095850e8138fd79b86e P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:05.853077 30138 raft_consensus.cc:515] T f3e17f0a8fc14095850e8138fd79b86e P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:27:05.855341 30138 leader_election.cc:290] T f3e17f0a8fc14095850e8138fd79b86e P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033), d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825)
I20260709 12:27:05.856118 29520 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f3e17f0a8fc14095850e8138fd79b86e" candidate_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "584dac86183a41d9b3bea1c2085e62fa" is_pre_election: true
I20260709 12:27:05.856449 29590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f3e17f0a8fc14095850e8138fd79b86e" candidate_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d73d92fb35234d16824c55689b5fce28" is_pre_election: true
I20260709 12:27:05.856760 29520 raft_consensus.cc:2468] T f3e17f0a8fc14095850e8138fd79b86e P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d13c58ad8ac441c4a9b396eb04b1a476 in term 1.
I20260709 12:27:05.857091 29590 raft_consensus.cc:2468] T f3e17f0a8fc14095850e8138fd79b86e P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d13c58ad8ac441c4a9b396eb04b1a476 in term 1.
I20260709 12:27:05.857967 29634 leader_election.cc:304] T f3e17f0a8fc14095850e8138fd79b86e P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 584dac86183a41d9b3bea1c2085e62fa, d13c58ad8ac441c4a9b396eb04b1a476; no voters: 
I20260709 12:27:05.858749 30138 raft_consensus.cc:2804] T f3e17f0a8fc14095850e8138fd79b86e P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:27:05.859081 30138 raft_consensus.cc:493] T f3e17f0a8fc14095850e8138fd79b86e P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:05.859336 30138 raft_consensus.cc:3060] T f3e17f0a8fc14095850e8138fd79b86e P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:05.867194 30138 raft_consensus.cc:515] T f3e17f0a8fc14095850e8138fd79b86e P d13c58ad8ac441c4a9b396eb04b1a476 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:27:05.869505 30138 leader_election.cc:290] T f3e17f0a8fc14095850e8138fd79b86e P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 2 election: Requested vote from peers 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033), d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825)
I20260709 12:27:05.870488 29590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f3e17f0a8fc14095850e8138fd79b86e" candidate_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d73d92fb35234d16824c55689b5fce28"
I20260709 12:27:05.870474 29520 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f3e17f0a8fc14095850e8138fd79b86e" candidate_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "584dac86183a41d9b3bea1c2085e62fa"
I20260709 12:27:05.871305 29590 raft_consensus.cc:3060] T f3e17f0a8fc14095850e8138fd79b86e P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:05.871331 29520 raft_consensus.cc:3060] T f3e17f0a8fc14095850e8138fd79b86e P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:05.878364 29520 raft_consensus.cc:2468] T f3e17f0a8fc14095850e8138fd79b86e P 584dac86183a41d9b3bea1c2085e62fa [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d13c58ad8ac441c4a9b396eb04b1a476 in term 2.
I20260709 12:27:05.879446 29634 leader_election.cc:304] T f3e17f0a8fc14095850e8138fd79b86e P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 584dac86183a41d9b3bea1c2085e62fa, d13c58ad8ac441c4a9b396eb04b1a476; no voters: 
I20260709 12:27:05.880342 29590 raft_consensus.cc:2468] T f3e17f0a8fc14095850e8138fd79b86e P d73d92fb35234d16824c55689b5fce28 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d13c58ad8ac441c4a9b396eb04b1a476 in term 2.
I20260709 12:27:05.880364 30138 raft_consensus.cc:2804] T f3e17f0a8fc14095850e8138fd79b86e P d13c58ad8ac441c4a9b396eb04b1a476 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:27:05.882315 30138 raft_consensus.cc:697] T f3e17f0a8fc14095850e8138fd79b86e P d13c58ad8ac441c4a9b396eb04b1a476 [term 2 LEADER]: Becoming Leader. State: Replica: d13c58ad8ac441c4a9b396eb04b1a476, State: Running, Role: LEADER
I20260709 12:27:05.883235 30138 consensus_queue.cc:237] T f3e17f0a8fc14095850e8138fd79b86e P d13c58ad8ac441c4a9b396eb04b1a476 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:27:05.896411 29419 catalog_manager.cc:5697] T f3e17f0a8fc14095850e8138fd79b86e P d13c58ad8ac441c4a9b396eb04b1a476 reported cstate change: term changed from 0 to 2, leader changed from <none> to d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67). New cstate: current_term: 2 leader_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } health_report { overall_health: HEALTHY } } }
W20260709 12:27:06.153374 29467 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:27:06.416337 29597 raft_consensus.cc:1275] T f3e17f0a8fc14095850e8138fd79b86e P d73d92fb35234d16824c55689b5fce28 [term 2 FOLLOWER]: Refusing update from remote peer d13c58ad8ac441c4a9b396eb04b1a476: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260709 12:27:06.417594 30138 consensus_queue.cc:1048] T f3e17f0a8fc14095850e8138fd79b86e P d13c58ad8ac441c4a9b396eb04b1a476 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:27:06.433246 29520 raft_consensus.cc:1275] T f3e17f0a8fc14095850e8138fd79b86e P 584dac86183a41d9b3bea1c2085e62fa [term 2 FOLLOWER]: Refusing update from remote peer d13c58ad8ac441c4a9b396eb04b1a476: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260709 12:27:06.434968 30072 consensus_queue.cc:1048] T f3e17f0a8fc14095850e8138fd79b86e P d13c58ad8ac441c4a9b396eb04b1a476 [LEADER]: Connected to new peer: Peer: permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:27:07.050026 30166 raft_consensus.cc:493] T 137e03144e754f6e9da4f0544513d27c P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:07.050614 30166 raft_consensus.cc:515] T 137e03144e754f6e9da4f0544513d27c P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:27:07.052830 30166 leader_election.cc:290] T 137e03144e754f6e9da4f0544513d27c P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209), c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895)
I20260709 12:27:07.053978 29671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "137e03144e754f6e9da4f0544513d27c" candidate_uuid: "d73d92fb35234d16824c55689b5fce28" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" is_pre_election: true
I20260709 12:27:07.054325 29741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "137e03144e754f6e9da4f0544513d27c" candidate_uuid: "d73d92fb35234d16824c55689b5fce28" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" is_pre_election: true
I20260709 12:27:07.054698 29671 raft_consensus.cc:2468] T 137e03144e754f6e9da4f0544513d27c P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d73d92fb35234d16824c55689b5fce28 in term 1.
I20260709 12:27:07.054908 29741 raft_consensus.cc:2468] T 137e03144e754f6e9da4f0544513d27c P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d73d92fb35234d16824c55689b5fce28 in term 1.
I20260709 12:27:07.055837 29561 leader_election.cc:304] T 137e03144e754f6e9da4f0544513d27c P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d13c58ad8ac441c4a9b396eb04b1a476, d73d92fb35234d16824c55689b5fce28; no voters: 
I20260709 12:27:07.056495 30166 raft_consensus.cc:2804] T 137e03144e754f6e9da4f0544513d27c P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:27:07.056862 30166 raft_consensus.cc:493] T 137e03144e754f6e9da4f0544513d27c P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:07.057183 30166 raft_consensus.cc:3060] T 137e03144e754f6e9da4f0544513d27c P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:07.063603 30166 raft_consensus.cc:515] T 137e03144e754f6e9da4f0544513d27c P d73d92fb35234d16824c55689b5fce28 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:27:07.065455 30166 leader_election.cc:290] T 137e03144e754f6e9da4f0544513d27c P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 2 election: Requested vote from peers d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209), c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895)
I20260709 12:27:07.066599 29668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "137e03144e754f6e9da4f0544513d27c" candidate_uuid: "d73d92fb35234d16824c55689b5fce28" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d13c58ad8ac441c4a9b396eb04b1a476"
I20260709 12:27:07.067237 29668 raft_consensus.cc:3060] T 137e03144e754f6e9da4f0544513d27c P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:07.067266 29744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "137e03144e754f6e9da4f0544513d27c" candidate_uuid: "d73d92fb35234d16824c55689b5fce28" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6610c00ffb948ce9c34f41dec3d49e8"
I20260709 12:27:07.067940 29744 raft_consensus.cc:3060] T 137e03144e754f6e9da4f0544513d27c P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:07.074649 29744 raft_consensus.cc:2468] T 137e03144e754f6e9da4f0544513d27c P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d73d92fb35234d16824c55689b5fce28 in term 2.
I20260709 12:27:07.075641 29668 raft_consensus.cc:2468] T 137e03144e754f6e9da4f0544513d27c P d13c58ad8ac441c4a9b396eb04b1a476 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d73d92fb35234d16824c55689b5fce28 in term 2.
I20260709 12:27:07.076292 29561 leader_election.cc:304] T 137e03144e754f6e9da4f0544513d27c P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c6610c00ffb948ce9c34f41dec3d49e8, d73d92fb35234d16824c55689b5fce28; no voters: 
I20260709 12:27:07.077811 30003 raft_consensus.cc:2804] T 137e03144e754f6e9da4f0544513d27c P d73d92fb35234d16824c55689b5fce28 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:27:07.080789 30003 raft_consensus.cc:697] T 137e03144e754f6e9da4f0544513d27c P d73d92fb35234d16824c55689b5fce28 [term 2 LEADER]: Becoming Leader. State: Replica: d73d92fb35234d16824c55689b5fce28, State: Running, Role: LEADER
I20260709 12:27:07.081660 30003 consensus_queue.cc:237] T 137e03144e754f6e9da4f0544513d27c P d73d92fb35234d16824c55689b5fce28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:27:07.091041 29417 catalog_manager.cc:5697] T 137e03144e754f6e9da4f0544513d27c P d73d92fb35234d16824c55689b5fce28 reported cstate change: term changed from 0 to 2, leader changed from <none> to d73d92fb35234d16824c55689b5fce28 (127.16.29.66). New cstate: current_term: 2 leader_uuid: "d73d92fb35234d16824c55689b5fce28" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:07.560328 29670 raft_consensus.cc:1275] T 137e03144e754f6e9da4f0544513d27c P d13c58ad8ac441c4a9b396eb04b1a476 [term 2 FOLLOWER]: Refusing update from remote peer d73d92fb35234d16824c55689b5fce28: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260709 12:27:07.561447 30003 consensus_queue.cc:1048] T 137e03144e754f6e9da4f0544513d27c P d73d92fb35234d16824c55689b5fce28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:27:07.584863 29744 raft_consensus.cc:1275] T 137e03144e754f6e9da4f0544513d27c P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Refusing update from remote peer d73d92fb35234d16824c55689b5fce28: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260709 12:27:07.588325 30166 consensus_queue.cc:1048] T 137e03144e754f6e9da4f0544513d27c P d73d92fb35234d16824c55689b5fce28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:27:08.886729 16501 tablet_server.cc:179] TabletServer@127.16.29.65:0 shutting down...
W20260709 12:27:08.893682 29708 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:27:08.908561 29708 consensus_peers.cc:597] T ef20d7124e7a44aa97a219ee194f4b8b P c6610c00ffb948ce9c34f41dec3d49e8 -> Peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Couldn't send request to peer 584dac86183a41d9b3bea1c2085e62fa. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:08.909251 29708 consensus_peers.cc:597] T 66f652ba2c224885a2aea974bfe8f88a P c6610c00ffb948ce9c34f41dec3d49e8 -> Peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Couldn't send request to peer 584dac86183a41d9b3bea1c2085e62fa. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:08.918913 29634 consensus_peers.cc:597] T 31a190e9f5ff407fab90502fb3197b70 P d13c58ad8ac441c4a9b396eb04b1a476 -> Peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Couldn't send request to peer 584dac86183a41d9b3bea1c2085e62fa. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:08.943718 29634 consensus_peers.cc:597] T f76101eb30f14b98a9aa8c41e0d6f831 P d13c58ad8ac441c4a9b396eb04b1a476 -> Peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Couldn't send request to peer 584dac86183a41d9b3bea1c2085e62fa. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:08.950171 16501 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:27:08.951359 16501 tablet_replica.cc:333] T f76101eb30f14b98a9aa8c41e0d6f831 P 584dac86183a41d9b3bea1c2085e62fa: stopping tablet replica
I20260709 12:27:08.952885 16501 raft_consensus.cc:2243] T f76101eb30f14b98a9aa8c41e0d6f831 P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:08.953552 16501 raft_consensus.cc:2272] T f76101eb30f14b98a9aa8c41e0d6f831 P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:08.955695 16501 tablet_replica.cc:333] T a3dcbb9c64bf4b5bb6680e8cb81af93e P 584dac86183a41d9b3bea1c2085e62fa: stopping tablet replica
I20260709 12:27:08.956457 16501 raft_consensus.cc:2243] T a3dcbb9c64bf4b5bb6680e8cb81af93e P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:08.956969 16501 raft_consensus.cc:2272] T a3dcbb9c64bf4b5bb6680e8cb81af93e P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:08.959384 16501 tablet_replica.cc:333] T abec627108a84083884e64e1eb9dcda6 P 584dac86183a41d9b3bea1c2085e62fa: stopping tablet replica
I20260709 12:27:08.960065 16501 raft_consensus.cc:2243] T abec627108a84083884e64e1eb9dcda6 P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:08.960536 16501 raft_consensus.cc:2272] T abec627108a84083884e64e1eb9dcda6 P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:08.962581 16501 tablet_replica.cc:333] T 31a190e9f5ff407fab90502fb3197b70 P 584dac86183a41d9b3bea1c2085e62fa: stopping tablet replica
I20260709 12:27:08.963168 16501 raft_consensus.cc:2243] T 31a190e9f5ff407fab90502fb3197b70 P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:08.963579 16501 raft_consensus.cc:2272] T 31a190e9f5ff407fab90502fb3197b70 P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:08.966118 16501 tablet_replica.cc:333] T 01788883c8ea42989f246a3486111a7e P 584dac86183a41d9b3bea1c2085e62fa: stopping tablet replica
I20260709 12:27:08.967093 16501 raft_consensus.cc:2243] T 01788883c8ea42989f246a3486111a7e P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:08.967661 16501 raft_consensus.cc:2272] T 01788883c8ea42989f246a3486111a7e P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:08.969632 16501 tablet_replica.cc:333] T cbf03426b6df43e5b2b9c56e2eb7687c P 584dac86183a41d9b3bea1c2085e62fa: stopping tablet replica
W20260709 12:27:08.969965 29560 consensus_peers.cc:597] T d1918297b9eb41f5847a211e668ae748 P d73d92fb35234d16824c55689b5fce28 -> Peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Couldn't send request to peer 584dac86183a41d9b3bea1c2085e62fa. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:08.970466 16501 raft_consensus.cc:2243] T cbf03426b6df43e5b2b9c56e2eb7687c P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:08.970968 16501 raft_consensus.cc:2272] T cbf03426b6df43e5b2b9c56e2eb7687c P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:08.975260 16501 tablet_replica.cc:333] T 1d4e0ad0462e4f74a44155aa3ffaf87f P 584dac86183a41d9b3bea1c2085e62fa: stopping tablet replica
W20260709 12:27:08.976105 29634 consensus_peers.cc:597] T 0c428ff656ae4b9d99d5b5073f98725d P d13c58ad8ac441c4a9b396eb04b1a476 -> Peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Couldn't send request to peer 584dac86183a41d9b3bea1c2085e62fa. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:08.976215 16501 raft_consensus.cc:2243] T 1d4e0ad0462e4f74a44155aa3ffaf87f P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:08.977138 16501 raft_consensus.cc:2272] T 1d4e0ad0462e4f74a44155aa3ffaf87f P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:08.980851 16501 tablet_replica.cc:333] T ef20d7124e7a44aa97a219ee194f4b8b P 584dac86183a41d9b3bea1c2085e62fa: stopping tablet replica
I20260709 12:27:08.981997 16501 raft_consensus.cc:2243] T ef20d7124e7a44aa97a219ee194f4b8b P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:08.982620 16501 raft_consensus.cc:2272] T ef20d7124e7a44aa97a219ee194f4b8b P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:08.985306 16501 tablet_replica.cc:333] T e67a2affef634fd8bc96a64fa83cffdb P 584dac86183a41d9b3bea1c2085e62fa: stopping tablet replica
I20260709 12:27:08.986395 16501 raft_consensus.cc:2243] T e67a2affef634fd8bc96a64fa83cffdb P 584dac86183a41d9b3bea1c2085e62fa [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:27:08.987447 16501 raft_consensus.cc:2272] T e67a2affef634fd8bc96a64fa83cffdb P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:08.990327 16501 tablet_replica.cc:333] T d1918297b9eb41f5847a211e668ae748 P 584dac86183a41d9b3bea1c2085e62fa: stopping tablet replica
I20260709 12:27:08.991560 16501 raft_consensus.cc:2243] T d1918297b9eb41f5847a211e668ae748 P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:08.992175 16501 raft_consensus.cc:2272] T d1918297b9eb41f5847a211e668ae748 P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:08.994364 16501 tablet_replica.cc:333] T 6a30685caefb4d0e80fa4ea87d70760c P 584dac86183a41d9b3bea1c2085e62fa: stopping tablet replica
I20260709 12:27:08.995141 16501 raft_consensus.cc:2243] T 6a30685caefb4d0e80fa4ea87d70760c P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:08.995549 16501 raft_consensus.cc:2272] T 6a30685caefb4d0e80fa4ea87d70760c P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:08.997941 16501 tablet_replica.cc:333] T f3e17f0a8fc14095850e8138fd79b86e P 584dac86183a41d9b3bea1c2085e62fa: stopping tablet replica
W20260709 12:27:08.997931 29708 consensus_peers.cc:597] T 8627b58b4f42491597125eb45f8d3b61 P c6610c00ffb948ce9c34f41dec3d49e8 -> Peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Couldn't send request to peer 584dac86183a41d9b3bea1c2085e62fa. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:08.998883 16501 raft_consensus.cc:2243] T f3e17f0a8fc14095850e8138fd79b86e P 584dac86183a41d9b3bea1c2085e62fa [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:08.999481 16501 raft_consensus.cc:2272] T f3e17f0a8fc14095850e8138fd79b86e P 584dac86183a41d9b3bea1c2085e62fa [term 2 FOLLOWER]: Raft consensus is shut down!
W20260709 12:27:09.000731 29560 consensus_peers.cc:597] T d5fcd1f030614bbe82cf1dc261dfc1c3 P d73d92fb35234d16824c55689b5fce28 -> Peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Couldn't send request to peer 584dac86183a41d9b3bea1c2085e62fa. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:09.001792 16501 tablet_replica.cc:333] T 425777eed34441cd93e0902bdc126516 P 584dac86183a41d9b3bea1c2085e62fa: stopping tablet replica
I20260709 12:27:09.002411 16501 raft_consensus.cc:2243] T 425777eed34441cd93e0902bdc126516 P 584dac86183a41d9b3bea1c2085e62fa [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:27:09.003070 16501 raft_consensus.cc:2272] T 425777eed34441cd93e0902bdc126516 P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:09.005069 16501 tablet_replica.cc:333] T 3e9451c3797c4496a66dfdb3e9fb5e38 P 584dac86183a41d9b3bea1c2085e62fa: stopping tablet replica
I20260709 12:27:09.005591 16501 raft_consensus.cc:2243] T 3e9451c3797c4496a66dfdb3e9fb5e38 P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:09.005959 16501 raft_consensus.cc:2272] T 3e9451c3797c4496a66dfdb3e9fb5e38 P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:09.008195 16501 tablet_replica.cc:333] T c6e621aa77d5474580dfc5949801ccc9 P 584dac86183a41d9b3bea1c2085e62fa: stopping tablet replica
I20260709 12:27:09.008807 16501 raft_consensus.cc:2243] T c6e621aa77d5474580dfc5949801ccc9 P 584dac86183a41d9b3bea1c2085e62fa [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:27:09.009562 16501 raft_consensus.cc:2272] T c6e621aa77d5474580dfc5949801ccc9 P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:09.012396 16501 tablet_replica.cc:333] T 6c8b2fa9f2c044ec9eec97c72b561042 P 584dac86183a41d9b3bea1c2085e62fa: stopping tablet replica
I20260709 12:27:09.013322 16501 raft_consensus.cc:2243] T 6c8b2fa9f2c044ec9eec97c72b561042 P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:09.013926 16501 raft_consensus.cc:2272] T 6c8b2fa9f2c044ec9eec97c72b561042 P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:09.016443 16501 tablet_replica.cc:333] T 66f652ba2c224885a2aea974bfe8f88a P 584dac86183a41d9b3bea1c2085e62fa: stopping tablet replica
I20260709 12:27:09.017174 16501 raft_consensus.cc:2243] T 66f652ba2c224885a2aea974bfe8f88a P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:09.017598 16501 raft_consensus.cc:2272] T 66f652ba2c224885a2aea974bfe8f88a P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:09.019599 16501 tablet_replica.cc:333] T 3f41ce441fb047c8bbb741b1b9cbc0ec P 584dac86183a41d9b3bea1c2085e62fa: stopping tablet replica
I20260709 12:27:09.020249 16501 raft_consensus.cc:2243] T 3f41ce441fb047c8bbb741b1b9cbc0ec P 584dac86183a41d9b3bea1c2085e62fa [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:27:09.021049 16501 raft_consensus.cc:2272] T 3f41ce441fb047c8bbb741b1b9cbc0ec P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:09.023142 16501 tablet_replica.cc:333] T f8c01e05981747a395e5483b551297dd P 584dac86183a41d9b3bea1c2085e62fa: stopping tablet replica
I20260709 12:27:09.023698 16501 raft_consensus.cc:2243] T f8c01e05981747a395e5483b551297dd P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:09.024221 16501 raft_consensus.cc:2272] T f8c01e05981747a395e5483b551297dd P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:09.026125 16501 tablet_replica.cc:333] T 6e37d72807e547898f4fce9e8b82b2b0 P 584dac86183a41d9b3bea1c2085e62fa: stopping tablet replica
I20260709 12:27:09.026746 16501 raft_consensus.cc:2243] T 6e37d72807e547898f4fce9e8b82b2b0 P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:09.027200 16501 raft_consensus.cc:2272] T 6e37d72807e547898f4fce9e8b82b2b0 P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:09.029528 16501 tablet_replica.cc:333] T f4516786e5564e9eac08ca472f79a9f0 P 584dac86183a41d9b3bea1c2085e62fa: stopping tablet replica
I20260709 12:27:09.030186 16501 raft_consensus.cc:2243] T f4516786e5564e9eac08ca472f79a9f0 P 584dac86183a41d9b3bea1c2085e62fa [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:27:09.031010 16501 raft_consensus.cc:2272] T f4516786e5564e9eac08ca472f79a9f0 P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:09.033805 16501 tablet_replica.cc:333] T 71e2de235adf46d9a2407659080d42ef P 584dac86183a41d9b3bea1c2085e62fa: stopping tablet replica
I20260709 12:27:09.034761 16501 raft_consensus.cc:2243] T 71e2de235adf46d9a2407659080d42ef P 584dac86183a41d9b3bea1c2085e62fa [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:27:09.035933 16501 raft_consensus.cc:2272] T 71e2de235adf46d9a2407659080d42ef P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:09.039150 16501 tablet_replica.cc:333] T 8017a44219404faa8db8ff049c65205e P 584dac86183a41d9b3bea1c2085e62fa: stopping tablet replica
W20260709 12:27:09.039296 29708 consensus_peers.cc:597] T 4685c2e824c14fc1bcd45c16169798cc P c6610c00ffb948ce9c34f41dec3d49e8 -> Peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Couldn't send request to peer 584dac86183a41d9b3bea1c2085e62fa. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:09.040099 16501 raft_consensus.cc:2243] T 8017a44219404faa8db8ff049c65205e P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:09.040994 16501 raft_consensus.cc:2272] T 8017a44219404faa8db8ff049c65205e P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:09.043269 16501 tablet_replica.cc:333] T fb6e78ccc01c4707af34cddf81619eab P 584dac86183a41d9b3bea1c2085e62fa: stopping tablet replica
I20260709 12:27:09.044009 16501 raft_consensus.cc:2243] T fb6e78ccc01c4707af34cddf81619eab P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:09.044492 16501 raft_consensus.cc:2272] T fb6e78ccc01c4707af34cddf81619eab P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:09.047451 16501 tablet_replica.cc:333] T 8627b58b4f42491597125eb45f8d3b61 P 584dac86183a41d9b3bea1c2085e62fa: stopping tablet replica
I20260709 12:27:09.048290 16501 raft_consensus.cc:2243] T 8627b58b4f42491597125eb45f8d3b61 P 584dac86183a41d9b3bea1c2085e62fa [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:09.048947 16501 raft_consensus.cc:2272] T 8627b58b4f42491597125eb45f8d3b61 P 584dac86183a41d9b3bea1c2085e62fa [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:09.051066 16501 tablet_replica.cc:333] T 2707271fd46f456aa765026642a3efa9 P 584dac86183a41d9b3bea1c2085e62fa: stopping tablet replica
I20260709 12:27:09.051708 16501 raft_consensus.cc:2243] T 2707271fd46f456aa765026642a3efa9 P 584dac86183a41d9b3bea1c2085e62fa [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:27:09.052616 16501 raft_consensus.cc:2272] T 2707271fd46f456aa765026642a3efa9 P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:09.054557 16501 tablet_replica.cc:333] T 9633e3b1d78f403097e4a00b7d4b869b P 584dac86183a41d9b3bea1c2085e62fa: stopping tablet replica
I20260709 12:27:09.055231 16501 raft_consensus.cc:2243] T 9633e3b1d78f403097e4a00b7d4b869b P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:09.055792 16501 raft_consensus.cc:2272] T 9633e3b1d78f403097e4a00b7d4b869b P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 12:27:09.058279 29560 consensus_peers.cc:597] T 5be58d64d45e40d7981018b84a01b1ed P d73d92fb35234d16824c55689b5fce28 -> Peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Couldn't send request to peer 584dac86183a41d9b3bea1c2085e62fa. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:09.058288 16501 tablet_replica.cc:333] T 5be58d64d45e40d7981018b84a01b1ed P 584dac86183a41d9b3bea1c2085e62fa: stopping tablet replica
I20260709 12:27:09.059257 16501 raft_consensus.cc:2243] T 5be58d64d45e40d7981018b84a01b1ed P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:09.059739 16501 raft_consensus.cc:2272] T 5be58d64d45e40d7981018b84a01b1ed P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:09.062295 16501 tablet_replica.cc:333] T b3c02c910d2b43cc9265bb1d98491cd3 P 584dac86183a41d9b3bea1c2085e62fa: stopping tablet replica
I20260709 12:27:09.063031 16501 raft_consensus.cc:2243] T b3c02c910d2b43cc9265bb1d98491cd3 P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:09.063530 16501 raft_consensus.cc:2272] T b3c02c910d2b43cc9265bb1d98491cd3 P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:09.066155 16501 tablet_replica.cc:333] T 3615115233c34461b85763151751e475 P 584dac86183a41d9b3bea1c2085e62fa: stopping tablet replica
I20260709 12:27:09.066900 16501 raft_consensus.cc:2243] T 3615115233c34461b85763151751e475 P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:09.067473 16501 raft_consensus.cc:2272] T 3615115233c34461b85763151751e475 P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:09.070729 16501 tablet_replica.cc:333] T 1d1c1f58c4d340ac951dfa403d34138a P 584dac86183a41d9b3bea1c2085e62fa: stopping tablet replica
W20260709 12:27:09.070876 29634 consensus_peers.cc:597] T 6e37d72807e547898f4fce9e8b82b2b0 P d13c58ad8ac441c4a9b396eb04b1a476 -> Peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Couldn't send request to peer 584dac86183a41d9b3bea1c2085e62fa. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:09.071663 16501 raft_consensus.cc:2243] T 1d1c1f58c4d340ac951dfa403d34138a P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:09.072281 16501 raft_consensus.cc:2272] T 1d1c1f58c4d340ac951dfa403d34138a P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:09.074406 16501 tablet_replica.cc:333] T a46289e8fc824dd5a818e30df6e7ec3e P 584dac86183a41d9b3bea1c2085e62fa: stopping tablet replica
I20260709 12:27:09.074977 16501 raft_consensus.cc:2243] T a46289e8fc824dd5a818e30df6e7ec3e P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:09.075457 16501 raft_consensus.cc:2272] T a46289e8fc824dd5a818e30df6e7ec3e P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:09.077544 16501 tablet_replica.cc:333] T 62f8c9472b214576b9002e02536a3ace P 584dac86183a41d9b3bea1c2085e62fa: stopping tablet replica
I20260709 12:27:09.078137 16501 raft_consensus.cc:2243] T 62f8c9472b214576b9002e02536a3ace P 584dac86183a41d9b3bea1c2085e62fa [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:27:09.079102 16501 raft_consensus.cc:2272] T 62f8c9472b214576b9002e02536a3ace P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:09.081914 16501 tablet_replica.cc:333] T d5fcd1f030614bbe82cf1dc261dfc1c3 P 584dac86183a41d9b3bea1c2085e62fa: stopping tablet replica
I20260709 12:27:09.082679 16501 raft_consensus.cc:2243] T d5fcd1f030614bbe82cf1dc261dfc1c3 P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Raft consensus shutting down.
W20260709 12:27:09.082731 29708 consensus_peers.cc:597] T 9633e3b1d78f403097e4a00b7d4b869b P c6610c00ffb948ce9c34f41dec3d49e8 -> Peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Couldn't send request to peer 584dac86183a41d9b3bea1c2085e62fa. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:09.083220 16501 raft_consensus.cc:2272] T d5fcd1f030614bbe82cf1dc261dfc1c3 P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:09.085613 16501 tablet_replica.cc:333] T d42259e436d241b29fb4ded16d9b6455 P 584dac86183a41d9b3bea1c2085e62fa: stopping tablet replica
I20260709 12:27:09.086354 16501 raft_consensus.cc:2243] T d42259e436d241b29fb4ded16d9b6455 P 584dac86183a41d9b3bea1c2085e62fa [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:27:09.087320 16501 raft_consensus.cc:2272] T d42259e436d241b29fb4ded16d9b6455 P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:09.089814 16501 tablet_replica.cc:333] T 592801934cad4789b9ce8dc09f1470bd P 584dac86183a41d9b3bea1c2085e62fa: stopping tablet replica
I20260709 12:27:09.090529 16501 raft_consensus.cc:2243] T 592801934cad4789b9ce8dc09f1470bd P 584dac86183a41d9b3bea1c2085e62fa [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:27:09.091549 16501 raft_consensus.cc:2272] T 592801934cad4789b9ce8dc09f1470bd P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 12:27:09.092077 29560 consensus_peers.cc:597] T a3dcbb9c64bf4b5bb6680e8cb81af93e P d73d92fb35234d16824c55689b5fce28 -> Peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Couldn't send request to peer 584dac86183a41d9b3bea1c2085e62fa. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:09.094179 16501 tablet_replica.cc:333] T fb7d78e534634e0f992fc39b578ebd8d P 584dac86183a41d9b3bea1c2085e62fa: stopping tablet replica
I20260709 12:27:09.095006 16501 raft_consensus.cc:2243] T fb7d78e534634e0f992fc39b578ebd8d P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:09.095505 16501 raft_consensus.cc:2272] T fb7d78e534634e0f992fc39b578ebd8d P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:09.097937 16501 tablet_replica.cc:333] T 4685c2e824c14fc1bcd45c16169798cc P 584dac86183a41d9b3bea1c2085e62fa: stopping tablet replica
I20260709 12:27:09.098688 16501 raft_consensus.cc:2243] T 4685c2e824c14fc1bcd45c16169798cc P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:09.099267 16501 raft_consensus.cc:2272] T 4685c2e824c14fc1bcd45c16169798cc P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:09.101819 16501 tablet_replica.cc:333] T fcb6a7f720b14dc98e821f663357a0bb P 584dac86183a41d9b3bea1c2085e62fa: stopping tablet replica
I20260709 12:27:09.102607 16501 raft_consensus.cc:2243] T fcb6a7f720b14dc98e821f663357a0bb P 584dac86183a41d9b3bea1c2085e62fa [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:27:09.103655 16501 raft_consensus.cc:2272] T fcb6a7f720b14dc98e821f663357a0bb P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 12:27:09.105377 29560 consensus_peers.cc:597] T b3c02c910d2b43cc9265bb1d98491cd3 P d73d92fb35234d16824c55689b5fce28 -> Peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Couldn't send request to peer 584dac86183a41d9b3bea1c2085e62fa. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:09.105937 16501 tablet_replica.cc:333] T d21ab1641501475b829863275dc8182a P 584dac86183a41d9b3bea1c2085e62fa: stopping tablet replica
I20260709 12:27:09.106709 16501 raft_consensus.cc:2243] T d21ab1641501475b829863275dc8182a P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:09.107197 16501 raft_consensus.cc:2272] T d21ab1641501475b829863275dc8182a P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:09.109234 16501 tablet_replica.cc:333] T 16649fd17d434e3c95e9d585d2404659 P 584dac86183a41d9b3bea1c2085e62fa: stopping tablet replica
I20260709 12:27:09.109719 16501 raft_consensus.cc:2243] T 16649fd17d434e3c95e9d585d2404659 P 584dac86183a41d9b3bea1c2085e62fa [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:27:09.110612 16501 raft_consensus.cc:2272] T 16649fd17d434e3c95e9d585d2404659 P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:09.113085 16501 tablet_replica.cc:333] T 0c428ff656ae4b9d99d5b5073f98725d P 584dac86183a41d9b3bea1c2085e62fa: stopping tablet replica
I20260709 12:27:09.113853 16501 raft_consensus.cc:2243] T 0c428ff656ae4b9d99d5b5073f98725d P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:09.114326 16501 raft_consensus.cc:2272] T 0c428ff656ae4b9d99d5b5073f98725d P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:09.116578 16501 tablet_replica.cc:333] T 742beffede4f407d8a72909d2b147e6d P 584dac86183a41d9b3bea1c2085e62fa: stopping tablet replica
I20260709 12:27:09.117204 16501 raft_consensus.cc:2243] T 742beffede4f407d8a72909d2b147e6d P 584dac86183a41d9b3bea1c2085e62fa [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:27:09.118120 16501 raft_consensus.cc:2272] T 742beffede4f407d8a72909d2b147e6d P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:09.120175 16501 tablet_replica.cc:333] T 5ee924f859f54f64908499bfac48ee62 P 584dac86183a41d9b3bea1c2085e62fa: stopping tablet replica
I20260709 12:27:09.120657 16501 raft_consensus.cc:2243] T 5ee924f859f54f64908499bfac48ee62 P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:09.121049 16501 raft_consensus.cc:2272] T 5ee924f859f54f64908499bfac48ee62 P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:09.123284 16501 tablet_replica.cc:333] T 7ae3b542012d4f35a9c363eee22976ec P 584dac86183a41d9b3bea1c2085e62fa: stopping tablet replica
I20260709 12:27:09.123907 16501 raft_consensus.cc:2243] T 7ae3b542012d4f35a9c363eee22976ec P 584dac86183a41d9b3bea1c2085e62fa [term 1 LEADER]: Raft consensus shutting down.
W20260709 12:27:09.124361 30038 consensus_queue.cc:1175] T 7ae3b542012d4f35a9c363eee22976ec P 584dac86183a41d9b3bea1c2085e62fa [LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" responder_term: 1 status { last_received { term: 1 index: 1 } last_committed_idx: 1 last_received_current_leader { term: 1 index: 1 } }
I20260709 12:27:09.125106 16501 raft_consensus.cc:2272] T 7ae3b542012d4f35a9c363eee22976ec P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 12:27:09.129513 29560 consensus_peers.cc:597] T d21ab1641501475b829863275dc8182a P d73d92fb35234d16824c55689b5fce28 -> Peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Couldn't send request to peer 584dac86183a41d9b3bea1c2085e62fa. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:09.143080 29708 consensus_peers.cc:597] T 6c8b2fa9f2c044ec9eec97c72b561042 P c6610c00ffb948ce9c34f41dec3d49e8 -> Peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Couldn't send request to peer 584dac86183a41d9b3bea1c2085e62fa. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:09.174665 29560 consensus_peers.cc:597] T f8c01e05981747a395e5483b551297dd P d73d92fb35234d16824c55689b5fce28 -> Peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Couldn't send request to peer 584dac86183a41d9b3bea1c2085e62fa. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:09.174722 29634 consensus_peers.cc:597] T f3e17f0a8fc14095850e8138fd79b86e P d13c58ad8ac441c4a9b396eb04b1a476 -> Peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Couldn't send request to peer 584dac86183a41d9b3bea1c2085e62fa. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:09.186520 29560 consensus_peers.cc:597] T 3615115233c34461b85763151751e475 P d73d92fb35234d16824c55689b5fce28 -> Peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Couldn't send request to peer 584dac86183a41d9b3bea1c2085e62fa. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:09.197067 29634 consensus_peers.cc:597] T 5ee924f859f54f64908499bfac48ee62 P d13c58ad8ac441c4a9b396eb04b1a476 -> Peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Couldn't send request to peer 584dac86183a41d9b3bea1c2085e62fa. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:09.214238 16501 tablet_server.cc:196] TabletServer@127.16.29.65:0 shutdown complete.
W20260709 12:27:09.215134 29708 consensus_peers.cc:597] T fb6e78ccc01c4707af34cddf81619eab P c6610c00ffb948ce9c34f41dec3d49e8 -> Peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Couldn't send request to peer 584dac86183a41d9b3bea1c2085e62fa. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:09.215665 29708 consensus_peers.cc:597] T 6a30685caefb4d0e80fa4ea87d70760c P c6610c00ffb948ce9c34f41dec3d49e8 -> Peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Couldn't send request to peer 584dac86183a41d9b3bea1c2085e62fa. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:09.229425 29708 consensus_peers.cc:597] T 1d4e0ad0462e4f74a44155aa3ffaf87f P c6610c00ffb948ce9c34f41dec3d49e8 -> Peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Couldn't send request to peer 584dac86183a41d9b3bea1c2085e62fa. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:09.231215 29634 consensus_peers.cc:597] T 3e9451c3797c4496a66dfdb3e9fb5e38 P d13c58ad8ac441c4a9b396eb04b1a476 -> Peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Couldn't send request to peer 584dac86183a41d9b3bea1c2085e62fa. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:09.282159 29634 consensus_peers.cc:597] T fb7d78e534634e0f992fc39b578ebd8d P d13c58ad8ac441c4a9b396eb04b1a476 -> Peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Couldn't send request to peer 584dac86183a41d9b3bea1c2085e62fa. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:09.288188 16501 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:27:09.297446 30303 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:27:09.297485 30302 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:27:09.304113 29708 consensus_peers.cc:597] T 8017a44219404faa8db8ff049c65205e P c6610c00ffb948ce9c34f41dec3d49e8 -> Peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Couldn't send request to peer 584dac86183a41d9b3bea1c2085e62fa. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:09.308379 16501 server_base.cc:1061] running on GCE node
W20260709 12:27:09.312470 30305 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:27:09.316798 16501 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:27:09.317180 16501 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:27:09.317418 16501 hybrid_clock.cc:648] HybridClock initialized: now 1783600029317398 us; error 0 us; skew 500 ppm
I20260709 12:27:09.318197 16501 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:27:09.336620 16501 webserver.cc:533] Webserver started at http://127.16.29.65:33857/ using document root <none> and password file <none>
I20260709 12:27:09.337579 16501 fs_manager.cc:362] Metadata directory not provided
I20260709 12:27:09.337848 16501 fs_manager.cc:365] Using existing metadata directory in first data directory
I20260709 12:27:09.359593 16501 fs_manager.cc:714] Time spent opening directory manager: real 0.020s	user 0.017s	sys 0.004s
W20260709 12:27:09.361342 29634 consensus_peers.cc:597] T cbf03426b6df43e5b2b9c56e2eb7687c P d13c58ad8ac441c4a9b396eb04b1a476 -> Peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Couldn't send request to peer 584dac86183a41d9b3bea1c2085e62fa. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:09.369550 29560 consensus_peers.cc:597] T abec627108a84083884e64e1eb9dcda6 P d73d92fb35234d16824c55689b5fce28 -> Peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Couldn't send request to peer 584dac86183a41d9b3bea1c2085e62fa. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:09.370215 29560 consensus_peers.cc:597] T 01788883c8ea42989f246a3486111a7e P d73d92fb35234d16824c55689b5fce28 -> Peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Couldn't send request to peer 584dac86183a41d9b3bea1c2085e62fa. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:09.372509 30312 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:27:09.373507 16501 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20260709 12:27:09.373956 16501 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599551912677-16501-0/minicluster-data/ts-0-root
uuid: "584dac86183a41d9b3bea1c2085e62fa"
format_stamp: "Formatted at 2026-07-09 12:26:53 on dist-test-slave-5n9v"
I20260709 12:27:09.375574 16501 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599551912677-16501-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599551912677-16501-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599551912677-16501-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
W20260709 12:27:09.390648 29708 consensus_peers.cc:597] T a46289e8fc824dd5a818e30df6e7ec3e P c6610c00ffb948ce9c34f41dec3d49e8 -> Peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Couldn't send request to peer 584dac86183a41d9b3bea1c2085e62fa. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:09.393201 16501 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:27:09.394550 16501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:27:09.398721 16501 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
W20260709 12:27:09.401672 29560 consensus_peers.cc:597] T 1d1c1f58c4d340ac951dfa403d34138a P d73d92fb35234d16824c55689b5fce28 -> Peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Couldn't send request to peer 584dac86183a41d9b3bea1c2085e62fa. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:09.410236 30319 ts_tablet_manager.cc:548] Loading tablet metadata (0/45 complete)
I20260709 12:27:09.684746 16501 ts_tablet_manager.cc:585] Loaded tablet metadata (45 total tablets, 45 live tablets)
I20260709 12:27:09.685107 16501 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.276s	user 0.008s	sys 0.000s
I20260709 12:27:09.685459 16501 ts_tablet_manager.cc:600] Registering tablets (0/45 complete)
I20260709 12:27:09.693392 30319 tablet_bootstrap.cc:492] T 5be58d64d45e40d7981018b84a01b1ed P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap starting.
I20260709 12:27:09.712622 30319 tablet_bootstrap.cc:492] T 5be58d64d45e40d7981018b84a01b1ed P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:27:09.713693 30319 tablet_bootstrap.cc:492] T 5be58d64d45e40d7981018b84a01b1ed P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap complete.
I20260709 12:27:09.714354 30319 ts_tablet_manager.cc:1403] T 5be58d64d45e40d7981018b84a01b1ed P 584dac86183a41d9b3bea1c2085e62fa: Time spent bootstrapping tablet: real 0.021s	user 0.012s	sys 0.004s
I20260709 12:27:09.718103 30319 raft_consensus.cc:359] T 5be58d64d45e40d7981018b84a01b1ed P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:09.718870 30319 raft_consensus.cc:740] T 5be58d64d45e40d7981018b84a01b1ed P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 584dac86183a41d9b3bea1c2085e62fa, State: Initialized, Role: FOLLOWER
I20260709 12:27:09.719612 30319 consensus_queue.cc:260] T 5be58d64d45e40d7981018b84a01b1ed P 584dac86183a41d9b3bea1c2085e62fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:09.742147 30319 ts_tablet_manager.cc:1434] T 5be58d64d45e40d7981018b84a01b1ed P 584dac86183a41d9b3bea1c2085e62fa: Time spent starting tablet: real 0.027s	user 0.014s	sys 0.011s
I20260709 12:27:09.743000 30319 tablet_bootstrap.cc:492] T fb6e78ccc01c4707af34cddf81619eab P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap starting.
I20260709 12:27:09.763585 30319 tablet_bootstrap.cc:492] T fb6e78ccc01c4707af34cddf81619eab P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:27:09.764731 30319 tablet_bootstrap.cc:492] T fb6e78ccc01c4707af34cddf81619eab P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap complete.
I20260709 12:27:09.765359 30319 ts_tablet_manager.cc:1403] T fb6e78ccc01c4707af34cddf81619eab P 584dac86183a41d9b3bea1c2085e62fa: Time spent bootstrapping tablet: real 0.023s	user 0.014s	sys 0.001s
I20260709 12:27:09.767735 30319 raft_consensus.cc:359] T fb6e78ccc01c4707af34cddf81619eab P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:09.768429 30319 raft_consensus.cc:740] T fb6e78ccc01c4707af34cddf81619eab P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 584dac86183a41d9b3bea1c2085e62fa, State: Initialized, Role: FOLLOWER
I20260709 12:27:09.769227 30319 consensus_queue.cc:260] T fb6e78ccc01c4707af34cddf81619eab P 584dac86183a41d9b3bea1c2085e62fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:09.771025 30319 ts_tablet_manager.cc:1434] T fb6e78ccc01c4707af34cddf81619eab P 584dac86183a41d9b3bea1c2085e62fa: Time spent starting tablet: real 0.005s	user 0.004s	sys 0.000s
I20260709 12:27:09.771939 30319 tablet_bootstrap.cc:492] T 3f41ce441fb047c8bbb741b1b9cbc0ec P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap starting.
I20260709 12:27:09.788368 30319 tablet_bootstrap.cc:492] T 3f41ce441fb047c8bbb741b1b9cbc0ec P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:27:09.789314 30319 tablet_bootstrap.cc:492] T 3f41ce441fb047c8bbb741b1b9cbc0ec P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap complete.
I20260709 12:27:09.789891 30319 ts_tablet_manager.cc:1403] T 3f41ce441fb047c8bbb741b1b9cbc0ec P 584dac86183a41d9b3bea1c2085e62fa: Time spent bootstrapping tablet: real 0.018s	user 0.012s	sys 0.003s
I20260709 12:27:09.791788 30319 raft_consensus.cc:359] T 3f41ce441fb047c8bbb741b1b9cbc0ec P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:27:09.792337 30319 raft_consensus.cc:740] T 3f41ce441fb047c8bbb741b1b9cbc0ec P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 584dac86183a41d9b3bea1c2085e62fa, State: Initialized, Role: FOLLOWER
I20260709 12:27:09.792929 30319 consensus_queue.cc:260] T 3f41ce441fb047c8bbb741b1b9cbc0ec P 584dac86183a41d9b3bea1c2085e62fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:27:09.794310 30319 ts_tablet_manager.cc:1434] T 3f41ce441fb047c8bbb741b1b9cbc0ec P 584dac86183a41d9b3bea1c2085e62fa: Time spent starting tablet: real 0.004s	user 0.004s	sys 0.000s
I20260709 12:27:09.794960 30319 tablet_bootstrap.cc:492] T 6c8b2fa9f2c044ec9eec97c72b561042 P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap starting.
I20260709 12:27:09.808504 30319 tablet_bootstrap.cc:492] T 6c8b2fa9f2c044ec9eec97c72b561042 P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:27:09.809432 30319 tablet_bootstrap.cc:492] T 6c8b2fa9f2c044ec9eec97c72b561042 P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap complete.
I20260709 12:27:09.810002 30319 ts_tablet_manager.cc:1403] T 6c8b2fa9f2c044ec9eec97c72b561042 P 584dac86183a41d9b3bea1c2085e62fa: Time spent bootstrapping tablet: real 0.015s	user 0.005s	sys 0.008s
I20260709 12:27:09.812321 30319 raft_consensus.cc:359] T 6c8b2fa9f2c044ec9eec97c72b561042 P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:09.812837 30319 raft_consensus.cc:740] T 6c8b2fa9f2c044ec9eec97c72b561042 P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 584dac86183a41d9b3bea1c2085e62fa, State: Initialized, Role: FOLLOWER
I20260709 12:27:09.813437 30319 consensus_queue.cc:260] T 6c8b2fa9f2c044ec9eec97c72b561042 P 584dac86183a41d9b3bea1c2085e62fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:09.815030 30319 ts_tablet_manager.cc:1434] T 6c8b2fa9f2c044ec9eec97c72b561042 P 584dac86183a41d9b3bea1c2085e62fa: Time spent starting tablet: real 0.005s	user 0.004s	sys 0.000s
I20260709 12:27:09.815848 30319 tablet_bootstrap.cc:492] T c6e621aa77d5474580dfc5949801ccc9 P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap starting.
I20260709 12:27:09.828863 30319 tablet_bootstrap.cc:492] T c6e621aa77d5474580dfc5949801ccc9 P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:27:09.829852 30319 tablet_bootstrap.cc:492] T c6e621aa77d5474580dfc5949801ccc9 P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap complete.
I20260709 12:27:09.830410 30319 ts_tablet_manager.cc:1403] T c6e621aa77d5474580dfc5949801ccc9 P 584dac86183a41d9b3bea1c2085e62fa: Time spent bootstrapping tablet: real 0.015s	user 0.008s	sys 0.004s
I20260709 12:27:09.832803 30319 raft_consensus.cc:359] T c6e621aa77d5474580dfc5949801ccc9 P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:27:09.833468 30319 raft_consensus.cc:740] T c6e621aa77d5474580dfc5949801ccc9 P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 584dac86183a41d9b3bea1c2085e62fa, State: Initialized, Role: FOLLOWER
I20260709 12:27:09.834252 30319 consensus_queue.cc:260] T c6e621aa77d5474580dfc5949801ccc9 P 584dac86183a41d9b3bea1c2085e62fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:27:09.836199 30319 ts_tablet_manager.cc:1434] T c6e621aa77d5474580dfc5949801ccc9 P 584dac86183a41d9b3bea1c2085e62fa: Time spent starting tablet: real 0.005s	user 0.008s	sys 0.000s
I20260709 12:27:09.836946 30319 tablet_bootstrap.cc:492] T d42259e436d241b29fb4ded16d9b6455 P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap starting.
I20260709 12:27:09.855541 30319 tablet_bootstrap.cc:492] T d42259e436d241b29fb4ded16d9b6455 P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:27:09.856475 30319 tablet_bootstrap.cc:492] T d42259e436d241b29fb4ded16d9b6455 P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap complete.
I20260709 12:27:09.857070 30319 ts_tablet_manager.cc:1403] T d42259e436d241b29fb4ded16d9b6455 P 584dac86183a41d9b3bea1c2085e62fa: Time spent bootstrapping tablet: real 0.020s	user 0.013s	sys 0.004s
I20260709 12:27:09.859345 30319 raft_consensus.cc:359] T d42259e436d241b29fb4ded16d9b6455 P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:27:09.860097 30319 raft_consensus.cc:740] T d42259e436d241b29fb4ded16d9b6455 P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 584dac86183a41d9b3bea1c2085e62fa, State: Initialized, Role: FOLLOWER
I20260709 12:27:09.860841 30319 consensus_queue.cc:260] T d42259e436d241b29fb4ded16d9b6455 P 584dac86183a41d9b3bea1c2085e62fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:27:09.862986 30319 ts_tablet_manager.cc:1434] T d42259e436d241b29fb4ded16d9b6455 P 584dac86183a41d9b3bea1c2085e62fa: Time spent starting tablet: real 0.006s	user 0.006s	sys 0.000s
I20260709 12:27:09.863798 30319 tablet_bootstrap.cc:492] T 1d1c1f58c4d340ac951dfa403d34138a P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap starting.
I20260709 12:27:09.934669 30319 tablet_bootstrap.cc:492] T 1d1c1f58c4d340ac951dfa403d34138a P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:27:09.936060 30319 tablet_bootstrap.cc:492] T 1d1c1f58c4d340ac951dfa403d34138a P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap complete.
I20260709 12:27:09.936756 30319 ts_tablet_manager.cc:1403] T 1d1c1f58c4d340ac951dfa403d34138a P 584dac86183a41d9b3bea1c2085e62fa: Time spent bootstrapping tablet: real 0.073s	user 0.034s	sys 0.034s
I20260709 12:27:09.939422 30319 raft_consensus.cc:359] T 1d1c1f58c4d340ac951dfa403d34138a P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:09.940117 30319 raft_consensus.cc:740] T 1d1c1f58c4d340ac951dfa403d34138a P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 584dac86183a41d9b3bea1c2085e62fa, State: Initialized, Role: FOLLOWER
I20260709 12:27:09.940915 30319 consensus_queue.cc:260] T 1d1c1f58c4d340ac951dfa403d34138a P 584dac86183a41d9b3bea1c2085e62fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:09.943022 30319 ts_tablet_manager.cc:1434] T 1d1c1f58c4d340ac951dfa403d34138a P 584dac86183a41d9b3bea1c2085e62fa: Time spent starting tablet: real 0.006s	user 0.005s	sys 0.000s
I20260709 12:27:09.943766 30319 tablet_bootstrap.cc:492] T 7ae3b542012d4f35a9c363eee22976ec P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap starting.
I20260709 12:27:09.957048 30319 tablet_bootstrap.cc:492] T 7ae3b542012d4f35a9c363eee22976ec P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:27:09.957952 30319 tablet_bootstrap.cc:492] T 7ae3b542012d4f35a9c363eee22976ec P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap complete.
I20260709 12:27:09.958601 30319 ts_tablet_manager.cc:1403] T 7ae3b542012d4f35a9c363eee22976ec P 584dac86183a41d9b3bea1c2085e62fa: Time spent bootstrapping tablet: real 0.015s	user 0.006s	sys 0.006s
I20260709 12:27:09.961374 30319 raft_consensus.cc:359] T 7ae3b542012d4f35a9c363eee22976ec P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:27:09.962040 30319 raft_consensus.cc:740] T 7ae3b542012d4f35a9c363eee22976ec P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 584dac86183a41d9b3bea1c2085e62fa, State: Initialized, Role: FOLLOWER
I20260709 12:27:09.962670 30319 consensus_queue.cc:260] T 7ae3b542012d4f35a9c363eee22976ec P 584dac86183a41d9b3bea1c2085e62fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:27:09.964604 30319 ts_tablet_manager.cc:1434] T 7ae3b542012d4f35a9c363eee22976ec P 584dac86183a41d9b3bea1c2085e62fa: Time spent starting tablet: real 0.006s	user 0.006s	sys 0.002s
I20260709 12:27:09.965327 30319 tablet_bootstrap.cc:492] T fb7d78e534634e0f992fc39b578ebd8d P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap starting.
I20260709 12:27:09.980074 30319 tablet_bootstrap.cc:492] T fb7d78e534634e0f992fc39b578ebd8d P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:27:09.981086 30319 tablet_bootstrap.cc:492] T fb7d78e534634e0f992fc39b578ebd8d P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap complete.
I20260709 12:27:09.981654 30319 ts_tablet_manager.cc:1403] T fb7d78e534634e0f992fc39b578ebd8d P 584dac86183a41d9b3bea1c2085e62fa: Time spent bootstrapping tablet: real 0.017s	user 0.010s	sys 0.004s
I20260709 12:27:09.983713 30319 raft_consensus.cc:359] T fb7d78e534634e0f992fc39b578ebd8d P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:27:09.984411 30319 raft_consensus.cc:740] T fb7d78e534634e0f992fc39b578ebd8d P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 584dac86183a41d9b3bea1c2085e62fa, State: Initialized, Role: FOLLOWER
I20260709 12:27:09.985082 30319 consensus_queue.cc:260] T fb7d78e534634e0f992fc39b578ebd8d P 584dac86183a41d9b3bea1c2085e62fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:27:09.987063 30319 ts_tablet_manager.cc:1434] T fb7d78e534634e0f992fc39b578ebd8d P 584dac86183a41d9b3bea1c2085e62fa: Time spent starting tablet: real 0.005s	user 0.004s	sys 0.000s
I20260709 12:27:09.987784 30319 tablet_bootstrap.cc:492] T abec627108a84083884e64e1eb9dcda6 P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap starting.
I20260709 12:27:10.000661 16501 ts_tablet_manager.cc:616] Registered 45 tablets
I20260709 12:27:10.001070 16501 ts_tablet_manager.cc:595] Time spent register tablets: real 0.316s	user 0.256s	sys 0.004s
I20260709 12:27:10.004627 30319 tablet_bootstrap.cc:492] T abec627108a84083884e64e1eb9dcda6 P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:27:10.005721 30319 tablet_bootstrap.cc:492] T abec627108a84083884e64e1eb9dcda6 P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap complete.
I20260709 12:27:10.006351 30319 ts_tablet_manager.cc:1403] T abec627108a84083884e64e1eb9dcda6 P 584dac86183a41d9b3bea1c2085e62fa: Time spent bootstrapping tablet: real 0.019s	user 0.007s	sys 0.008s
I20260709 12:27:10.008886 30319 raft_consensus.cc:359] T abec627108a84083884e64e1eb9dcda6 P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:27:10.009625 30319 raft_consensus.cc:740] T abec627108a84083884e64e1eb9dcda6 P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 584dac86183a41d9b3bea1c2085e62fa, State: Initialized, Role: FOLLOWER
I20260709 12:27:10.010260 30319 consensus_queue.cc:260] T abec627108a84083884e64e1eb9dcda6 P 584dac86183a41d9b3bea1c2085e62fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:27:10.011999 30319 ts_tablet_manager.cc:1434] T abec627108a84083884e64e1eb9dcda6 P 584dac86183a41d9b3bea1c2085e62fa: Time spent starting tablet: real 0.005s	user 0.007s	sys 0.000s
I20260709 12:27:10.012612 30319 tablet_bootstrap.cc:492] T 6a30685caefb4d0e80fa4ea87d70760c P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap starting.
I20260709 12:27:10.020614 30320 raft_consensus.cc:493] T c6e621aa77d5474580dfc5949801ccc9 P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 584dac86183a41d9b3bea1c2085e62fa)
I20260709 12:27:10.021466 30320 raft_consensus.cc:515] T c6e621aa77d5474580dfc5949801ccc9 P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:27:10.024386 30320 leader_election.cc:290] T c6e621aa77d5474580dfc5949801ccc9 P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033), d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209)
I20260709 12:27:10.026655 29668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6e621aa77d5474580dfc5949801ccc9" candidate_uuid: "d73d92fb35234d16824c55689b5fce28" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" is_pre_election: true
I20260709 12:27:10.027184 30324 raft_consensus.cc:493] T e67a2affef634fd8bc96a64fa83cffdb P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 584dac86183a41d9b3bea1c2085e62fa)
I20260709 12:27:10.027761 30324 raft_consensus.cc:515] T e67a2affef634fd8bc96a64fa83cffdb P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
W20260709 12:27:10.029771 29560 leader_election.cc:336] T c6e621aa77d5474580dfc5949801ccc9 P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111)
I20260709 12:27:10.030252 29560 leader_election.cc:304] T c6e621aa77d5474580dfc5949801ccc9 P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d73d92fb35234d16824c55689b5fce28; no voters: 584dac86183a41d9b3bea1c2085e62fa, d13c58ad8ac441c4a9b396eb04b1a476
I20260709 12:27:10.030907 30320 raft_consensus.cc:2749] T c6e621aa77d5474580dfc5949801ccc9 P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:27:10.033491 30319 tablet_bootstrap.cc:492] T 6a30685caefb4d0e80fa4ea87d70760c P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:27:10.034158 29668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e67a2affef634fd8bc96a64fa83cffdb" candidate_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" is_pre_election: true
I20260709 12:27:10.034832 30324 leader_election.cc:290] T e67a2affef634fd8bc96a64fa83cffdb P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209), 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033)
I20260709 12:27:10.035976 30319 tablet_bootstrap.cc:492] T 6a30685caefb4d0e80fa4ea87d70760c P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap complete.
I20260709 12:27:10.036904 30319 ts_tablet_manager.cc:1403] T 6a30685caefb4d0e80fa4ea87d70760c P 584dac86183a41d9b3bea1c2085e62fa: Time spent bootstrapping tablet: real 0.024s	user 0.018s	sys 0.000s
W20260709 12:27:10.037053 29708 leader_election.cc:336] T e67a2affef634fd8bc96a64fa83cffdb P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111)
I20260709 12:27:10.040207 30319 raft_consensus.cc:359] T 6a30685caefb4d0e80fa4ea87d70760c P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:10.041244 30319 raft_consensus.cc:740] T 6a30685caefb4d0e80fa4ea87d70760c P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 584dac86183a41d9b3bea1c2085e62fa, State: Initialized, Role: FOLLOWER
I20260709 12:27:10.040352 29708 leader_election.cc:304] T e67a2affef634fd8bc96a64fa83cffdb P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c6610c00ffb948ce9c34f41dec3d49e8; no voters: 584dac86183a41d9b3bea1c2085e62fa, d13c58ad8ac441c4a9b396eb04b1a476
I20260709 12:27:10.042126 30319 consensus_queue.cc:260] T 6a30685caefb4d0e80fa4ea87d70760c P 584dac86183a41d9b3bea1c2085e62fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:10.043542 30324 raft_consensus.cc:2749] T e67a2affef634fd8bc96a64fa83cffdb P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:27:10.045660 30319 ts_tablet_manager.cc:1434] T 6a30685caefb4d0e80fa4ea87d70760c P 584dac86183a41d9b3bea1c2085e62fa: Time spent starting tablet: real 0.008s	user 0.007s	sys 0.001s
I20260709 12:27:10.046842 30319 tablet_bootstrap.cc:492] T 8627b58b4f42491597125eb45f8d3b61 P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap starting.
I20260709 12:27:10.067322 30319 tablet_bootstrap.cc:492] T 8627b58b4f42491597125eb45f8d3b61 P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:27:10.068416 30319 tablet_bootstrap.cc:492] T 8627b58b4f42491597125eb45f8d3b61 P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap complete.
I20260709 12:27:10.069111 30319 ts_tablet_manager.cc:1403] T 8627b58b4f42491597125eb45f8d3b61 P 584dac86183a41d9b3bea1c2085e62fa: Time spent bootstrapping tablet: real 0.023s	user 0.017s	sys 0.003s
I20260709 12:27:10.071740 30319 raft_consensus.cc:359] T 8627b58b4f42491597125eb45f8d3b61 P 584dac86183a41d9b3bea1c2085e62fa [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:27:10.072415 30319 raft_consensus.cc:740] T 8627b58b4f42491597125eb45f8d3b61 P 584dac86183a41d9b3bea1c2085e62fa [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: 584dac86183a41d9b3bea1c2085e62fa, State: Initialized, Role: FOLLOWER
I20260709 12:27:10.073177 30319 consensus_queue.cc:260] T 8627b58b4f42491597125eb45f8d3b61 P 584dac86183a41d9b3bea1c2085e62fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:27:10.075161 30319 ts_tablet_manager.cc:1434] T 8627b58b4f42491597125eb45f8d3b61 P 584dac86183a41d9b3bea1c2085e62fa: Time spent starting tablet: real 0.006s	user 0.004s	sys 0.001s
I20260709 12:27:10.076026 30319 tablet_bootstrap.cc:492] T d21ab1641501475b829863275dc8182a P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap starting.
I20260709 12:27:10.091778 30319 tablet_bootstrap.cc:492] T d21ab1641501475b829863275dc8182a P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:27:10.093442 30319 tablet_bootstrap.cc:492] T d21ab1641501475b829863275dc8182a P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap complete.
I20260709 12:27:10.094203 30319 ts_tablet_manager.cc:1403] T d21ab1641501475b829863275dc8182a P 584dac86183a41d9b3bea1c2085e62fa: Time spent bootstrapping tablet: real 0.018s	user 0.017s	sys 0.000s
I20260709 12:27:10.099200 30319 raft_consensus.cc:359] T d21ab1641501475b829863275dc8182a P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:27:10.100034 30319 raft_consensus.cc:740] T d21ab1641501475b829863275dc8182a P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 584dac86183a41d9b3bea1c2085e62fa, State: Initialized, Role: FOLLOWER
I20260709 12:27:10.101028 30319 consensus_queue.cc:260] T d21ab1641501475b829863275dc8182a P 584dac86183a41d9b3bea1c2085e62fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:27:10.103559 30319 ts_tablet_manager.cc:1434] T d21ab1641501475b829863275dc8182a P 584dac86183a41d9b3bea1c2085e62fa: Time spent starting tablet: real 0.007s	user 0.004s	sys 0.001s
I20260709 12:27:10.104857 30319 tablet_bootstrap.cc:492] T d5fcd1f030614bbe82cf1dc261dfc1c3 P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap starting.
I20260709 12:27:10.106710 30282 raft_consensus.cc:493] T e67a2affef634fd8bc96a64fa83cffdb P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 584dac86183a41d9b3bea1c2085e62fa)
I20260709 12:27:10.107462 30282 raft_consensus.cc:515] T e67a2affef634fd8bc96a64fa83cffdb P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:27:10.111164 29741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e67a2affef634fd8bc96a64fa83cffdb" candidate_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" is_pre_election: true
I20260709 12:27:10.111788 29741 raft_consensus.cc:2468] T e67a2affef634fd8bc96a64fa83cffdb P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d13c58ad8ac441c4a9b396eb04b1a476 in term 1.
I20260709 12:27:10.113111 29635 leader_election.cc:304] T e67a2affef634fd8bc96a64fa83cffdb P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c6610c00ffb948ce9c34f41dec3d49e8, d13c58ad8ac441c4a9b396eb04b1a476; no voters: 
I20260709 12:27:10.114087 30138 raft_consensus.cc:2804] T e67a2affef634fd8bc96a64fa83cffdb P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:27:10.114494 30138 raft_consensus.cc:493] T e67a2affef634fd8bc96a64fa83cffdb P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 584dac86183a41d9b3bea1c2085e62fa)
I20260709 12:27:10.114842 30138 raft_consensus.cc:3060] T e67a2affef634fd8bc96a64fa83cffdb P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:10.118546 30282 leader_election.cc:290] T e67a2affef634fd8bc96a64fa83cffdb P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895), 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033)
I20260709 12:27:10.126621 30282 raft_consensus.cc:493] T fcb6a7f720b14dc98e821f663357a0bb P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 584dac86183a41d9b3bea1c2085e62fa)
I20260709 12:27:10.127362 30282 raft_consensus.cc:515] T fcb6a7f720b14dc98e821f663357a0bb P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
W20260709 12:27:10.134363 29634 leader_election.cc:336] T e67a2affef634fd8bc96a64fa83cffdb P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111)
I20260709 12:27:10.143321 29590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fcb6a7f720b14dc98e821f663357a0bb" candidate_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d73d92fb35234d16824c55689b5fce28" is_pre_election: true
I20260709 12:27:10.137141 30282 leader_election.cc:290] T fcb6a7f720b14dc98e821f663357a0bb P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033), d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825)
W20260709 12:27:10.153690 29634 leader_election.cc:336] T fcb6a7f720b14dc98e821f663357a0bb P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111)
I20260709 12:27:10.154304 29634 leader_election.cc:304] T fcb6a7f720b14dc98e821f663357a0bb P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d13c58ad8ac441c4a9b396eb04b1a476; no voters: 584dac86183a41d9b3bea1c2085e62fa, d73d92fb35234d16824c55689b5fce28
I20260709 12:27:10.157078 30321 raft_consensus.cc:2749] T fcb6a7f720b14dc98e821f663357a0bb P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:27:10.173600 30166 raft_consensus.cc:493] T 592801934cad4789b9ce8dc09f1470bd P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 584dac86183a41d9b3bea1c2085e62fa)
I20260709 12:27:10.176365 30321 raft_consensus.cc:493] T 2707271fd46f456aa765026642a3efa9 P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 584dac86183a41d9b3bea1c2085e62fa)
I20260709 12:27:10.176041 30166 raft_consensus.cc:515] T 592801934cad4789b9ce8dc09f1470bd P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:10.177052 30321 raft_consensus.cc:515] T 2707271fd46f456aa765026642a3efa9 P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:27:10.180248 30282 raft_consensus.cc:493] T f4516786e5564e9eac08ca472f79a9f0 P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 584dac86183a41d9b3bea1c2085e62fa)
I20260709 12:27:10.180832 30321 leader_election.cc:290] T 2707271fd46f456aa765026642a3efa9 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033), d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825)
I20260709 12:27:10.181082 30282 raft_consensus.cc:515] T f4516786e5564e9eac08ca472f79a9f0 P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:27:10.184679 29741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4516786e5564e9eac08ca472f79a9f0" candidate_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" is_pre_election: true
I20260709 12:27:10.178769 30166 leader_election.cc:290] T 592801934cad4789b9ce8dc09f1470bd P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033), c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895)
I20260709 12:27:10.185743 30282 leader_election.cc:290] T f4516786e5564e9eac08ca472f79a9f0 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895), 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033)
I20260709 12:27:10.188329 30138 raft_consensus.cc:515] T e67a2affef634fd8bc96a64fa83cffdb P d13c58ad8ac441c4a9b396eb04b1a476 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:27:10.195003 29741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e67a2affef634fd8bc96a64fa83cffdb" candidate_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c6610c00ffb948ce9c34f41dec3d49e8"
I20260709 12:27:10.195843 29741 raft_consensus.cc:3060] T e67a2affef634fd8bc96a64fa83cffdb P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:10.196939 30282 raft_consensus.cc:493] T 3f41ce441fb047c8bbb741b1b9cbc0ec P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 584dac86183a41d9b3bea1c2085e62fa)
I20260709 12:27:10.199070 30282 raft_consensus.cc:515] T 3f41ce441fb047c8bbb741b1b9cbc0ec P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:27:10.202922 30321 raft_consensus.cc:493] T 425777eed34441cd93e0902bdc126516 P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 584dac86183a41d9b3bea1c2085e62fa)
I20260709 12:27:10.232151 30321 raft_consensus.cc:515] T 425777eed34441cd93e0902bdc126516 P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:27:10.235500 30321 leader_election.cc:290] T 425777eed34441cd93e0902bdc126516 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895), 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033)
W20260709 12:27:10.221895 29634 leader_election.cc:336] T 2707271fd46f456aa765026642a3efa9 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111)
I20260709 12:27:10.223802 30308 raft_consensus.cc:493] T 742beffede4f407d8a72909d2b147e6d P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 584dac86183a41d9b3bea1c2085e62fa)
I20260709 12:27:10.237224 30308 raft_consensus.cc:515] T 742beffede4f407d8a72909d2b147e6d P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:27:10.238866 30324 raft_consensus.cc:493] T 425777eed34441cd93e0902bdc126516 P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 584dac86183a41d9b3bea1c2085e62fa)
I20260709 12:27:10.225723 29744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592801934cad4789b9ce8dc09f1470bd" candidate_uuid: "d73d92fb35234d16824c55689b5fce28" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" is_pre_election: true
I20260709 12:27:10.239775 30324 raft_consensus.cc:515] T 425777eed34441cd93e0902bdc126516 P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:27:10.244311 29670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "742beffede4f407d8a72909d2b147e6d" candidate_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" is_pre_election: true
I20260709 12:27:10.246091 30308 leader_election.cc:290] T 742beffede4f407d8a72909d2b147e6d P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033), d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209)
I20260709 12:27:10.229266 30386 raft_consensus.cc:493] T 742beffede4f407d8a72909d2b147e6d P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 584dac86183a41d9b3bea1c2085e62fa)
I20260709 12:27:10.255533 29668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "425777eed34441cd93e0902bdc126516" candidate_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" is_pre_election: true
I20260709 12:27:10.255399 30386 raft_consensus.cc:515] T 742beffede4f407d8a72909d2b147e6d P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:27:10.256428 29668 raft_consensus.cc:2468] T 425777eed34441cd93e0902bdc126516 P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6610c00ffb948ce9c34f41dec3d49e8 in term 1.
I20260709 12:27:10.258090 30386 leader_election.cc:290] T 742beffede4f407d8a72909d2b147e6d P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033), c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895)
I20260709 12:27:10.258438 29709 leader_election.cc:304] T 425777eed34441cd93e0902bdc126516 P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c6610c00ffb948ce9c34f41dec3d49e8, d13c58ad8ac441c4a9b396eb04b1a476; no voters: 
I20260709 12:27:10.259778 29670 raft_consensus.cc:2468] T 742beffede4f407d8a72909d2b147e6d P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6610c00ffb948ce9c34f41dec3d49e8 in term 1.
I20260709 12:27:10.262198 29709 leader_election.cc:304] T 742beffede4f407d8a72909d2b147e6d P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c6610c00ffb948ce9c34f41dec3d49e8, d13c58ad8ac441c4a9b396eb04b1a476; no voters: 
I20260709 12:27:10.263038 30271 raft_consensus.cc:2804] T 425777eed34441cd93e0902bdc126516 P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:27:10.263221 30308 raft_consensus.cc:2804] T 742beffede4f407d8a72909d2b147e6d P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:27:10.263718 30271 raft_consensus.cc:493] T 425777eed34441cd93e0902bdc126516 P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 584dac86183a41d9b3bea1c2085e62fa)
I20260709 12:27:10.264509 30271 raft_consensus.cc:3060] T 425777eed34441cd93e0902bdc126516 P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:10.210539 29590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2707271fd46f456aa765026642a3efa9" candidate_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d73d92fb35234d16824c55689b5fce28" is_pre_election: true
I20260709 12:27:10.263078 30320 raft_consensus.cc:493] T fcb6a7f720b14dc98e821f663357a0bb P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 584dac86183a41d9b3bea1c2085e62fa)
I20260709 12:27:10.267469 30320 raft_consensus.cc:515] T fcb6a7f720b14dc98e821f663357a0bb P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:27:10.270620 30320 leader_election.cc:290] T fcb6a7f720b14dc98e821f663357a0bb P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033), d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209)
I20260709 12:27:10.263857 30308 raft_consensus.cc:493] T 742beffede4f407d8a72909d2b147e6d P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 584dac86183a41d9b3bea1c2085e62fa)
I20260709 12:27:10.282920 30308 raft_consensus.cc:3060] T 742beffede4f407d8a72909d2b147e6d P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:10.254494 30324 leader_election.cc:290] T 425777eed34441cd93e0902bdc126516 P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209), 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033)
I20260709 12:27:10.272640 30282 leader_election.cc:290] T 3f41ce441fb047c8bbb741b1b9cbc0ec P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895), 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033)
I20260709 12:27:10.277704 30166 raft_consensus.cc:493] T d42259e436d241b29fb4ded16d9b6455 P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 584dac86183a41d9b3bea1c2085e62fa)
I20260709 12:27:10.288520 30166 raft_consensus.cc:515] T d42259e436d241b29fb4ded16d9b6455 P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:27:10.290422 29741 raft_consensus.cc:2468] T e67a2affef634fd8bc96a64fa83cffdb P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d13c58ad8ac441c4a9b396eb04b1a476 in term 2.
I20260709 12:27:10.290705 30166 leader_election.cc:290] T d42259e436d241b29fb4ded16d9b6455 P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033), d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209)
I20260709 12:27:10.227129 30138 leader_election.cc:290] T e67a2affef634fd8bc96a64fa83cffdb P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 2 election: Requested vote from peers c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895), 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033)
I20260709 12:27:10.291352 30271 raft_consensus.cc:515] T 425777eed34441cd93e0902bdc126516 P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
W20260709 12:27:10.231094 29560 leader_election.cc:336] T 592801934cad4789b9ce8dc09f1470bd P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111)
W20260709 12:27:10.332180 29560 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:35033, user_credentials={real_user=slave}} blocked reactor thread for 100899us
W20260709 12:27:10.338085 29561 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:43895, user_credentials={real_user=slave}} blocked reactor thread for 56764.7us
I20260709 12:27:10.339632 29561 leader_election.cc:304] T 592801934cad4789b9ce8dc09f1470bd P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d73d92fb35234d16824c55689b5fce28; no voters: 584dac86183a41d9b3bea1c2085e62fa, c6610c00ffb948ce9c34f41dec3d49e8
I20260709 12:27:10.341734 30397 raft_consensus.cc:493] T 62f8c9472b214576b9002e02536a3ace P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 584dac86183a41d9b3bea1c2085e62fa)
I20260709 12:27:10.343715 30397 raft_consensus.cc:515] T 62f8c9472b214576b9002e02536a3ace P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:10.346879 30399 raft_consensus.cc:493] T 16649fd17d434e3c95e9d585d2404659 P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 584dac86183a41d9b3bea1c2085e62fa)
I20260709 12:27:10.348573 30399 raft_consensus.cc:515] T 16649fd17d434e3c95e9d585d2404659 P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:27:10.356053 30166 raft_consensus.cc:2749] T 592801934cad4789b9ce8dc09f1470bd P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:27:10.356667 30271 leader_election.cc:290] T 425777eed34441cd93e0902bdc126516 P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 2 election: Requested vote from peers d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209), 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033)
I20260709 12:27:10.361434 30271 raft_consensus.cc:493] T f4516786e5564e9eac08ca472f79a9f0 P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 584dac86183a41d9b3bea1c2085e62fa)
I20260709 12:27:10.362826 30271 raft_consensus.cc:515] T f4516786e5564e9eac08ca472f79a9f0 P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:27:10.368036 29634 leader_election.cc:304] T 2707271fd46f456aa765026642a3efa9 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d13c58ad8ac441c4a9b396eb04b1a476; no voters: 584dac86183a41d9b3bea1c2085e62fa, d73d92fb35234d16824c55689b5fce28
I20260709 12:27:10.368253 30271 leader_election.cc:290] T f4516786e5564e9eac08ca472f79a9f0 P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209), 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033)
I20260709 12:27:10.370404 29668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "425777eed34441cd93e0902bdc126516" candidate_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d13c58ad8ac441c4a9b396eb04b1a476"
I20260709 12:27:10.372599 29671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4516786e5564e9eac08ca472f79a9f0" candidate_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" is_pre_election: true
I20260709 12:27:10.372555 30138 raft_consensus.cc:2749] T 2707271fd46f456aa765026642a3efa9 P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:27:10.373623 29741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "425777eed34441cd93e0902bdc126516" candidate_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" is_pre_election: true
I20260709 12:27:10.376480 29741 raft_consensus.cc:2393] T 425777eed34441cd93e0902bdc126516 P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d13c58ad8ac441c4a9b396eb04b1a476 in current term 2: Already voted for candidate c6610c00ffb948ce9c34f41dec3d49e8 in this term.
I20260709 12:27:10.371738 29668 raft_consensus.cc:3060] T 425777eed34441cd93e0902bdc126516 P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:10.377315 29745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "742beffede4f407d8a72909d2b147e6d" candidate_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" is_pre_election: true
I20260709 12:27:10.373625 29671 raft_consensus.cc:2468] T f4516786e5564e9eac08ca472f79a9f0 P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6610c00ffb948ce9c34f41dec3d49e8 in term 1.
W20260709 12:27:10.380199 29634 leader_election.cc:336] T f4516786e5564e9eac08ca472f79a9f0 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111)
I20260709 12:27:10.380399 29709 leader_election.cc:304] T f4516786e5564e9eac08ca472f79a9f0 P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c6610c00ffb948ce9c34f41dec3d49e8, d13c58ad8ac441c4a9b396eb04b1a476; no voters: 
I20260709 12:27:10.380936 29634 leader_election.cc:304] T f4516786e5564e9eac08ca472f79a9f0 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d13c58ad8ac441c4a9b396eb04b1a476; no voters: 584dac86183a41d9b3bea1c2085e62fa, c6610c00ffb948ce9c34f41dec3d49e8
I20260709 12:27:10.382515 30271 raft_consensus.cc:493] T 62f8c9472b214576b9002e02536a3ace P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 584dac86183a41d9b3bea1c2085e62fa)
I20260709 12:27:10.386899 30397 leader_election.cc:290] T 62f8c9472b214576b9002e02536a3ace P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895), 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033)
I20260709 12:27:10.387790 29667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d42259e436d241b29fb4ded16d9b6455" candidate_uuid: "d73d92fb35234d16824c55689b5fce28" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" is_pre_election: true
I20260709 12:27:10.412106 30321 raft_consensus.cc:493] T c6e621aa77d5474580dfc5949801ccc9 P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 584dac86183a41d9b3bea1c2085e62fa)
I20260709 12:27:10.413326 30321 raft_consensus.cc:515] T c6e621aa77d5474580dfc5949801ccc9 P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:27:10.396709 29670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fcb6a7f720b14dc98e821f663357a0bb" candidate_uuid: "d73d92fb35234d16824c55689b5fce28" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" is_pre_election: true
I20260709 12:27:10.403690 30388 raft_consensus.cc:2804] T f4516786e5564e9eac08ca472f79a9f0 P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:27:10.389890 29742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3f41ce441fb047c8bbb741b1b9cbc0ec" candidate_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" is_pre_election: true
I20260709 12:27:10.416158 30388 raft_consensus.cc:493] T f4516786e5564e9eac08ca472f79a9f0 P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 584dac86183a41d9b3bea1c2085e62fa)
W20260709 12:27:10.404848 29634 leader_election.cc:336] T 742beffede4f407d8a72909d2b147e6d P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111)
I20260709 12:27:10.416689 30388 raft_consensus.cc:3060] T f4516786e5564e9eac08ca472f79a9f0 P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:10.406247 29635 leader_election.cc:304] T e67a2affef634fd8bc96a64fa83cffdb P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c6610c00ffb948ce9c34f41dec3d49e8, d13c58ad8ac441c4a9b396eb04b1a476; no voters: 
I20260709 12:27:10.407316 30319 tablet_bootstrap.cc:492] T d5fcd1f030614bbe82cf1dc261dfc1c3 P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:27:10.392783 30282 raft_consensus.cc:2749] T f4516786e5564e9eac08ca472f79a9f0 P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:27:10.420182 30385 raft_consensus.cc:2804] T e67a2affef634fd8bc96a64fa83cffdb P d13c58ad8ac441c4a9b396eb04b1a476 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:27:10.421411 30319 tablet_bootstrap.cc:492] T d5fcd1f030614bbe82cf1dc261dfc1c3 P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap complete.
I20260709 12:27:10.422585 30385 raft_consensus.cc:697] T e67a2affef634fd8bc96a64fa83cffdb P d13c58ad8ac441c4a9b396eb04b1a476 [term 2 LEADER]: Becoming Leader. State: Replica: d13c58ad8ac441c4a9b396eb04b1a476, State: Running, Role: LEADER
I20260709 12:27:10.423708 30319 ts_tablet_manager.cc:1403] T d5fcd1f030614bbe82cf1dc261dfc1c3 P 584dac86183a41d9b3bea1c2085e62fa: Time spent bootstrapping tablet: real 0.319s	user 0.059s	sys 0.065s
I20260709 12:27:10.424974 30321 leader_election.cc:290] T c6e621aa77d5474580dfc5949801ccc9 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033), d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825)
I20260709 12:27:10.424487 30385 consensus_queue.cc:237] T e67a2affef634fd8bc96a64fa83cffdb P d13c58ad8ac441c4a9b396eb04b1a476 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
W20260709 12:27:10.419013 29560 leader_election.cc:336] T fcb6a7f720b14dc98e821f663357a0bb P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111)
I20260709 12:27:10.432143 29670 raft_consensus.cc:2468] T fcb6a7f720b14dc98e821f663357a0bb P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d73d92fb35234d16824c55689b5fce28 in term 1.
I20260709 12:27:10.434619 30319 raft_consensus.cc:359] T d5fcd1f030614bbe82cf1dc261dfc1c3 P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:27:10.435818 30319 raft_consensus.cc:740] T d5fcd1f030614bbe82cf1dc261dfc1c3 P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 584dac86183a41d9b3bea1c2085e62fa, State: Initialized, Role: FOLLOWER
I20260709 12:27:10.468175 30271 raft_consensus.cc:515] T 62f8c9472b214576b9002e02536a3ace P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:10.471927 30271 leader_election.cc:290] T 62f8c9472b214576b9002e02536a3ace P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033), d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825)
I20260709 12:27:10.473989 30399 leader_election.cc:290] T 16649fd17d434e3c95e9d585d2404659 P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033), d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209)
I20260709 12:27:10.460690 30319 consensus_queue.cc:260] T d5fcd1f030614bbe82cf1dc261dfc1c3 P 584dac86183a41d9b3bea1c2085e62fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:27:10.477562 29668 raft_consensus.cc:2468] T 425777eed34441cd93e0902bdc126516 P d13c58ad8ac441c4a9b396eb04b1a476 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6610c00ffb948ce9c34f41dec3d49e8 in term 2.
I20260709 12:27:10.478891 29590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6e621aa77d5474580dfc5949801ccc9" candidate_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d73d92fb35234d16824c55689b5fce28" is_pre_election: true
I20260709 12:27:10.488888 29590 raft_consensus.cc:2468] T c6e621aa77d5474580dfc5949801ccc9 P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d13c58ad8ac441c4a9b396eb04b1a476 in term 1.
W20260709 12:27:10.482100 29634 leader_election.cc:336] T 3f41ce441fb047c8bbb741b1b9cbc0ec P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111)
W20260709 12:27:10.459235 29560 leader_election.cc:336] T d42259e436d241b29fb4ded16d9b6455 P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111)
I20260709 12:27:10.484988 30397 raft_consensus.cc:493] T 7ae3b542012d4f35a9c363eee22976ec P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 584dac86183a41d9b3bea1c2085e62fa)
W20260709 12:27:10.485530 29708 leader_election.cc:336] T 742beffede4f407d8a72909d2b147e6d P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111)
I20260709 12:27:10.487246 30283 raft_consensus.cc:493] T 71e2de235adf46d9a2407659080d42ef P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 584dac86183a41d9b3bea1c2085e62fa)
I20260709 12:27:10.486624 30402 raft_consensus.cc:493] T 592801934cad4789b9ce8dc09f1470bd P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 584dac86183a41d9b3bea1c2085e62fa)
I20260709 12:27:10.493613 30397 raft_consensus.cc:515] T 7ae3b542012d4f35a9c363eee22976ec P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
W20260709 12:27:10.493928 29708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:35033, user_credentials={real_user=slave}} blocked reactor thread for 62806.8us
W20260709 12:27:10.494799 29708 leader_election.cc:336] T 425777eed34441cd93e0902bdc126516 P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111)
I20260709 12:27:10.494522 30402 raft_consensus.cc:515] T 592801934cad4789b9ce8dc09f1470bd P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:10.493952 30283 raft_consensus.cc:515] T 71e2de235adf46d9a2407659080d42ef P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:27:10.496240 29561 leader_election.cc:304] T d42259e436d241b29fb4ded16d9b6455 P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d73d92fb35234d16824c55689b5fce28; no voters: 584dac86183a41d9b3bea1c2085e62fa, d13c58ad8ac441c4a9b396eb04b1a476
I20260709 12:27:10.497149 30402 leader_election.cc:290] T 592801934cad4789b9ce8dc09f1470bd P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033), d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825)
I20260709 12:27:10.497690 30283 leader_election.cc:290] T 71e2de235adf46d9a2407659080d42ef P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033), d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209)
I20260709 12:27:10.496403 30397 leader_election.cc:290] T 7ae3b542012d4f35a9c363eee22976ec P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033), d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209)
I20260709 12:27:10.499900 29709 leader_election.cc:304] T 425777eed34441cd93e0902bdc126516 P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c6610c00ffb948ce9c34f41dec3d49e8, d13c58ad8ac441c4a9b396eb04b1a476; no voters: 
W20260709 12:27:10.500496 29634 leader_election.cc:336] T e67a2affef634fd8bc96a64fa83cffdb P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111)
W20260709 12:27:10.492731 29560 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:35033, user_credentials={real_user=slave}} blocked reactor thread for 45011.4us
W20260709 12:27:10.502036 29560 leader_election.cc:336] T 62f8c9472b214576b9002e02536a3ace P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111)
I20260709 12:27:10.503957 30320 raft_consensus.cc:2749] T d42259e436d241b29fb4ded16d9b6455 P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:27:10.507154 30401 raft_consensus.cc:2804] T 425777eed34441cd93e0902bdc126516 P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:27:10.509588 30401 raft_consensus.cc:697] T 425777eed34441cd93e0902bdc126516 P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 LEADER]: Becoming Leader. State: Replica: c6610c00ffb948ce9c34f41dec3d49e8, State: Running, Role: LEADER
W20260709 12:27:10.507675 29634 leader_election.cc:336] T 425777eed34441cd93e0902bdc126516 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111)
I20260709 12:27:10.511734 29634 leader_election.cc:304] T 425777eed34441cd93e0902bdc126516 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d13c58ad8ac441c4a9b396eb04b1a476; no voters: 584dac86183a41d9b3bea1c2085e62fa, c6610c00ffb948ce9c34f41dec3d49e8
I20260709 12:27:10.512619 30282 raft_consensus.cc:2749] T 425777eed34441cd93e0902bdc126516 P d13c58ad8ac441c4a9b396eb04b1a476 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:27:10.508045 30308 raft_consensus.cc:515] T 742beffede4f407d8a72909d2b147e6d P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:27:10.508745 29742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f8c9472b214576b9002e02536a3ace" candidate_uuid: "d73d92fb35234d16824c55689b5fce28" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" is_pre_election: true
I20260709 12:27:10.521710 29742 raft_consensus.cc:2468] T 62f8c9472b214576b9002e02536a3ace P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d73d92fb35234d16824c55689b5fce28 in term 1.
I20260709 12:27:10.510442 30320 raft_consensus.cc:493] T 2707271fd46f456aa765026642a3efa9 P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 584dac86183a41d9b3bea1c2085e62fa)
I20260709 12:27:10.522122 29745 raft_consensus.cc:2393] T 742beffede4f407d8a72909d2b147e6d P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d13c58ad8ac441c4a9b396eb04b1a476 in current term 2: Already voted for candidate c6610c00ffb948ce9c34f41dec3d49e8 in this term.
I20260709 12:27:10.522542 30320 raft_consensus.cc:515] T 2707271fd46f456aa765026642a3efa9 P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:27:10.524421 30282 raft_consensus.cc:493] T d42259e436d241b29fb4ded16d9b6455 P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 584dac86183a41d9b3bea1c2085e62fa)
I20260709 12:27:10.525271 30320 leader_election.cc:290] T 2707271fd46f456aa765026642a3efa9 P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033), d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209)
I20260709 12:27:10.525102 30282 raft_consensus.cc:515] T d42259e436d241b29fb4ded16d9b6455 P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:27:10.527686 30282 leader_election.cc:290] T d42259e436d241b29fb4ded16d9b6455 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033), d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825)
I20260709 12:27:10.528482 30308 leader_election.cc:290] T 742beffede4f407d8a72909d2b147e6d P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 2 election: Requested vote from peers 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033), d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209)
I20260709 12:27:10.529743 30282 raft_consensus.cc:493] T 71e2de235adf46d9a2407659080d42ef P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 584dac86183a41d9b3bea1c2085e62fa)
I20260709 12:27:10.530668 29635 leader_election.cc:304] T 3f41ce441fb047c8bbb741b1b9cbc0ec P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d13c58ad8ac441c4a9b396eb04b1a476; no voters: 584dac86183a41d9b3bea1c2085e62fa, c6610c00ffb948ce9c34f41dec3d49e8
I20260709 12:27:10.520756 29561 leader_election.cc:304] T fcb6a7f720b14dc98e821f663357a0bb P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d13c58ad8ac441c4a9b396eb04b1a476, d73d92fb35234d16824c55689b5fce28; no voters: 584dac86183a41d9b3bea1c2085e62fa
I20260709 12:27:10.507177 30319 ts_tablet_manager.cc:1434] T d5fcd1f030614bbe82cf1dc261dfc1c3 P 584dac86183a41d9b3bea1c2085e62fa: Time spent starting tablet: real 0.083s	user 0.011s	sys 0.000s
I20260709 12:27:10.534263 30386 raft_consensus.cc:493] T 16649fd17d434e3c95e9d585d2404659 P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 584dac86183a41d9b3bea1c2085e62fa)
I20260709 12:27:10.544936 30386 raft_consensus.cc:515] T 16649fd17d434e3c95e9d585d2404659 P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:27:10.547176 30386 leader_election.cc:290] T 16649fd17d434e3c95e9d585d2404659 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033), d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825)
I20260709 12:27:10.531720 30401 consensus_queue.cc:237] T 425777eed34441cd93e0902bdc126516 P c6610c00ffb948ce9c34f41dec3d49e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:27:10.544128 30320 raft_consensus.cc:2804] T fcb6a7f720b14dc98e821f663357a0bb P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:27:10.549654 30320 raft_consensus.cc:493] T fcb6a7f720b14dc98e821f663357a0bb P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 584dac86183a41d9b3bea1c2085e62fa)
I20260709 12:27:10.550081 30320 raft_consensus.cc:3060] T fcb6a7f720b14dc98e821f663357a0bb P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:10.551172 29670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "16649fd17d434e3c95e9d585d2404659" candidate_uuid: "d73d92fb35234d16824c55689b5fce28" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" is_pre_election: true
I20260709 12:27:10.551860 29670 raft_consensus.cc:2468] T 16649fd17d434e3c95e9d585d2404659 P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d73d92fb35234d16824c55689b5fce28 in term 1.
I20260709 12:27:10.536067 29668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "742beffede4f407d8a72909d2b147e6d" candidate_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d13c58ad8ac441c4a9b396eb04b1a476"
I20260709 12:27:10.553442 29668 raft_consensus.cc:3060] T 742beffede4f407d8a72909d2b147e6d P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:10.541064 29419 catalog_manager.cc:5697] T e67a2affef634fd8bc96a64fa83cffdb P d13c58ad8ac441c4a9b396eb04b1a476 reported cstate change: term changed from 1 to 2, leader changed from 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65) to d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67). New cstate: current_term: 2 leader_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:10.533026 30321 raft_consensus.cc:2749] T 3f41ce441fb047c8bbb741b1b9cbc0ec P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:27:10.542937 30319 tablet_bootstrap.cc:492] T 592801934cad4789b9ce8dc09f1470bd P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap starting.
W20260709 12:27:10.530620 29560 leader_election.cc:336] T 16649fd17d434e3c95e9d585d2404659 P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111)
W20260709 12:27:10.561482 29560 leader_election.cc:336] T 7ae3b542012d4f35a9c363eee22976ec P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111)
W20260709 12:27:10.562126 29560 leader_election.cc:336] T 71e2de235adf46d9a2407659080d42ef P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111)
I20260709 12:27:10.533308 30282 raft_consensus.cc:515] T 71e2de235adf46d9a2407659080d42ef P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:27:10.566207 30282 leader_election.cc:290] T 71e2de235adf46d9a2407659080d42ef P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033), d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825)
I20260709 12:27:10.529769 30388 raft_consensus.cc:515] T f4516786e5564e9eac08ca472f79a9f0 P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:27:10.568925 30388 leader_election.cc:290] T f4516786e5564e9eac08ca472f79a9f0 P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 2 election: Requested vote from peers d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209), 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033)
I20260709 12:27:10.559609 30320 raft_consensus.cc:515] T fcb6a7f720b14dc98e821f663357a0bb P d73d92fb35234d16824c55689b5fce28 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:27:10.569976 29668 raft_consensus.cc:2468] T 742beffede4f407d8a72909d2b147e6d P d13c58ad8ac441c4a9b396eb04b1a476 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6610c00ffb948ce9c34f41dec3d49e8 in term 2.
I20260709 12:27:10.571254 29709 leader_election.cc:304] T 742beffede4f407d8a72909d2b147e6d P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c6610c00ffb948ce9c34f41dec3d49e8, d13c58ad8ac441c4a9b396eb04b1a476; no voters: 
I20260709 12:27:10.571722 30320 leader_election.cc:290] T fcb6a7f720b14dc98e821f663357a0bb P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 2 election: Requested vote from peers 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033), d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209)
W20260709 12:27:10.572093 29708 leader_election.cc:336] T 425777eed34441cd93e0902bdc126516 P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111)
I20260709 12:27:10.573065 29670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4516786e5564e9eac08ca472f79a9f0" candidate_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d13c58ad8ac441c4a9b396eb04b1a476"
W20260709 12:27:10.573419 29708 leader_election.cc:336] T f4516786e5564e9eac08ca472f79a9f0 P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111)
I20260709 12:27:10.573808 29670 raft_consensus.cc:3060] T f4516786e5564e9eac08ca472f79a9f0 P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:10.572196 30308 raft_consensus.cc:2804] T 742beffede4f407d8a72909d2b147e6d P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:27:10.576265 30308 raft_consensus.cc:697] T 742beffede4f407d8a72909d2b147e6d P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 LEADER]: Becoming Leader. State: Replica: c6610c00ffb948ce9c34f41dec3d49e8, State: Running, Role: LEADER
W20260709 12:27:10.577888 29634 leader_election.cc:336] T c6e621aa77d5474580dfc5949801ccc9 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111)
I20260709 12:27:10.577769 30308 consensus_queue.cc:237] T 742beffede4f407d8a72909d2b147e6d P c6610c00ffb948ce9c34f41dec3d49e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:27:10.587507 29667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ae3b542012d4f35a9c363eee22976ec" candidate_uuid: "d73d92fb35234d16824c55689b5fce28" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" is_pre_election: true
I20260709 12:27:10.588903 29670 raft_consensus.cc:2468] T f4516786e5564e9eac08ca472f79a9f0 P d13c58ad8ac441c4a9b396eb04b1a476 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6610c00ffb948ce9c34f41dec3d49e8 in term 2.
I20260709 12:27:10.587920 29671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "71e2de235adf46d9a2407659080d42ef" candidate_uuid: "d73d92fb35234d16824c55689b5fce28" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" is_pre_election: true
I20260709 12:27:10.589628 29671 raft_consensus.cc:2468] T 71e2de235adf46d9a2407659080d42ef P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d73d92fb35234d16824c55689b5fce28 in term 1.
I20260709 12:27:10.591967 29635 leader_election.cc:304] T 742beffede4f407d8a72909d2b147e6d P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d13c58ad8ac441c4a9b396eb04b1a476; no voters: 584dac86183a41d9b3bea1c2085e62fa, c6610c00ffb948ce9c34f41dec3d49e8
I20260709 12:27:10.592811 30282 raft_consensus.cc:2749] T 742beffede4f407d8a72909d2b147e6d P d13c58ad8ac441c4a9b396eb04b1a476 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:27:10.597599 29709 leader_election.cc:304] T f4516786e5564e9eac08ca472f79a9f0 P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c6610c00ffb948ce9c34f41dec3d49e8, d13c58ad8ac441c4a9b396eb04b1a476; no voters: 
I20260709 12:27:10.598933 29668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2707271fd46f456aa765026642a3efa9" candidate_uuid: "d73d92fb35234d16824c55689b5fce28" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" is_pre_election: true
I20260709 12:27:10.599742 29668 raft_consensus.cc:2468] T 2707271fd46f456aa765026642a3efa9 P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d73d92fb35234d16824c55689b5fce28 in term 1.
I20260709 12:27:10.602236 30388 raft_consensus.cc:2804] T f4516786e5564e9eac08ca472f79a9f0 P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:27:10.602825 30388 raft_consensus.cc:697] T f4516786e5564e9eac08ca472f79a9f0 P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 LEADER]: Becoming Leader. State: Replica: c6610c00ffb948ce9c34f41dec3d49e8, State: Running, Role: LEADER
I20260709 12:27:10.612150 30390 raft_consensus.cc:493] T 3f41ce441fb047c8bbb741b1b9cbc0ec P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 584dac86183a41d9b3bea1c2085e62fa)
I20260709 12:27:10.612967 30390 raft_consensus.cc:515] T 3f41ce441fb047c8bbb741b1b9cbc0ec P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:27:10.615301 30390 leader_election.cc:290] T 3f41ce441fb047c8bbb741b1b9cbc0ec P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209), 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033)
I20260709 12:27:10.604321 30388 consensus_queue.cc:237] T f4516786e5564e9eac08ca472f79a9f0 P c6610c00ffb948ce9c34f41dec3d49e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:27:10.619470 30406 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.29.65:35033 every 8 connection(s)
I20260709 12:27:10.620007 16501 rpc_server.cc:307] RPC server started. Bound to: 127.16.29.65:35033
I20260709 12:27:10.624619 29561 leader_election.cc:304] T 62f8c9472b214576b9002e02536a3ace P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c6610c00ffb948ce9c34f41dec3d49e8, d73d92fb35234d16824c55689b5fce28; no voters: 584dac86183a41d9b3bea1c2085e62fa
I20260709 12:27:10.634971 29670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3f41ce441fb047c8bbb741b1b9cbc0ec" candidate_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" is_pre_election: true
I20260709 12:27:10.646154 29670 raft_consensus.cc:2468] T 3f41ce441fb047c8bbb741b1b9cbc0ec P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6610c00ffb948ce9c34f41dec3d49e8 in term 1.
I20260709 12:27:10.636777 30320 raft_consensus.cc:2804] T 62f8c9472b214576b9002e02536a3ace P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:27:10.653358 29709 leader_election.cc:304] T 3f41ce441fb047c8bbb741b1b9cbc0ec P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c6610c00ffb948ce9c34f41dec3d49e8, d13c58ad8ac441c4a9b396eb04b1a476; no voters: 
I20260709 12:27:10.648371 30320 raft_consensus.cc:493] T 62f8c9472b214576b9002e02536a3ace P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 584dac86183a41d9b3bea1c2085e62fa)
I20260709 12:27:10.673766 30320 raft_consensus.cc:3060] T 62f8c9472b214576b9002e02536a3ace P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:10.683207 30390 raft_consensus.cc:2804] T 3f41ce441fb047c8bbb741b1b9cbc0ec P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:27:10.686362 30390 raft_consensus.cc:493] T 3f41ce441fb047c8bbb741b1b9cbc0ec P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 584dac86183a41d9b3bea1c2085e62fa)
I20260709 12:27:10.686897 30390 raft_consensus.cc:3060] T 3f41ce441fb047c8bbb741b1b9cbc0ec P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:10.703663 29561 leader_election.cc:304] T 16649fd17d434e3c95e9d585d2404659 P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d13c58ad8ac441c4a9b396eb04b1a476, d73d92fb35234d16824c55689b5fce28; no voters: 584dac86183a41d9b3bea1c2085e62fa
I20260709 12:27:10.705637 30166 raft_consensus.cc:2804] T 16649fd17d434e3c95e9d585d2404659 P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:27:10.706162 30166 raft_consensus.cc:493] T 16649fd17d434e3c95e9d585d2404659 P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 584dac86183a41d9b3bea1c2085e62fa)
I20260709 12:27:10.706651 30166 raft_consensus.cc:3060] T 16649fd17d434e3c95e9d585d2404659 P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Advancing to term 2
W20260709 12:27:10.710976 29634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:36825, user_credentials={real_user=slave}} blocked reactor thread for 75962.3us
I20260709 12:27:10.712080 29634 leader_election.cc:304] T c6e621aa77d5474580dfc5949801ccc9 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d13c58ad8ac441c4a9b396eb04b1a476, d73d92fb35234d16824c55689b5fce28; no voters: 584dac86183a41d9b3bea1c2085e62fa
I20260709 12:27:10.719620 29671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fcb6a7f720b14dc98e821f663357a0bb" candidate_uuid: "d73d92fb35234d16824c55689b5fce28" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d13c58ad8ac441c4a9b396eb04b1a476"
I20260709 12:27:10.721964 30386 raft_consensus.cc:2804] T c6e621aa77d5474580dfc5949801ccc9 P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:27:10.722560 29561 leader_election.cc:304] T 7ae3b542012d4f35a9c363eee22976ec P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d73d92fb35234d16824c55689b5fce28; no voters: 584dac86183a41d9b3bea1c2085e62fa, d13c58ad8ac441c4a9b396eb04b1a476
I20260709 12:27:10.737560 30386 raft_consensus.cc:493] T c6e621aa77d5474580dfc5949801ccc9 P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 584dac86183a41d9b3bea1c2085e62fa)
I20260709 12:27:10.738131 30386 raft_consensus.cc:3060] T c6e621aa77d5474580dfc5949801ccc9 P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:10.736147 30321 raft_consensus.cc:493] T 7ae3b542012d4f35a9c363eee22976ec P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 584dac86183a41d9b3bea1c2085e62fa)
I20260709 12:27:10.738528 29561 leader_election.cc:304] T 71e2de235adf46d9a2407659080d42ef P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d13c58ad8ac441c4a9b396eb04b1a476, d73d92fb35234d16824c55689b5fce28; no voters: 584dac86183a41d9b3bea1c2085e62fa
I20260709 12:27:10.739149 30321 raft_consensus.cc:515] T 7ae3b542012d4f35a9c363eee22976ec P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:27:10.739782 30397 raft_consensus.cc:2804] T 71e2de235adf46d9a2407659080d42ef P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:27:10.740271 30397 raft_consensus.cc:493] T 71e2de235adf46d9a2407659080d42ef P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 584dac86183a41d9b3bea1c2085e62fa)
I20260709 12:27:10.740701 30397 raft_consensus.cc:3060] T 71e2de235adf46d9a2407659080d42ef P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:10.741544 30321 leader_election.cc:290] T 7ae3b542012d4f35a9c363eee22976ec P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033), d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825)
I20260709 12:27:10.726500 30166 raft_consensus.cc:515] T 16649fd17d434e3c95e9d585d2404659 P d73d92fb35234d16824c55689b5fce28 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:27:10.734022 30320 raft_consensus.cc:515] T 62f8c9472b214576b9002e02536a3ace P d73d92fb35234d16824c55689b5fce28 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:10.735860 29671 raft_consensus.cc:3060] T fcb6a7f720b14dc98e821f663357a0bb P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:10.751117 30368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2707271fd46f456aa765026642a3efa9" candidate_uuid: "d73d92fb35234d16824c55689b5fce28" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "584dac86183a41d9b3bea1c2085e62fa" is_pre_election: true
I20260709 12:27:10.753741 30366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fcb6a7f720b14dc98e821f663357a0bb" candidate_uuid: "d73d92fb35234d16824c55689b5fce28" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "584dac86183a41d9b3bea1c2085e62fa"
I20260709 12:27:10.753502 30390 raft_consensus.cc:515] T 3f41ce441fb047c8bbb741b1b9cbc0ec P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:27:10.758427 30363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d42259e436d241b29fb4ded16d9b6455" candidate_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "584dac86183a41d9b3bea1c2085e62fa" is_pre_election: true
I20260709 12:27:10.759382 30363 raft_consensus.cc:2468] T d42259e436d241b29fb4ded16d9b6455 P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d13c58ad8ac441c4a9b396eb04b1a476 in term 1.
I20260709 12:27:10.760531 30390 leader_election.cc:290] T 3f41ce441fb047c8bbb741b1b9cbc0ec P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 2 election: Requested vote from peers d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209), 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033)
I20260709 12:27:10.761233 30367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "71e2de235adf46d9a2407659080d42ef" candidate_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "584dac86183a41d9b3bea1c2085e62fa" is_pre_election: true
I20260709 12:27:10.761580 30360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ae3b542012d4f35a9c363eee22976ec" candidate_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "584dac86183a41d9b3bea1c2085e62fa" is_pre_election: true
I20260709 12:27:10.769500 30360 raft_consensus.cc:2468] T 7ae3b542012d4f35a9c363eee22976ec P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d13c58ad8ac441c4a9b396eb04b1a476 in term 1.
I20260709 12:27:10.763010 30366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "16649fd17d434e3c95e9d585d2404659" candidate_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "584dac86183a41d9b3bea1c2085e62fa" is_pre_election: true
I20260709 12:27:10.770676 29561 leader_election.cc:304] T 2707271fd46f456aa765026642a3efa9 P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d13c58ad8ac441c4a9b396eb04b1a476, d73d92fb35234d16824c55689b5fce28; no voters: 
I20260709 12:27:10.738744 30283 raft_consensus.cc:2749] T 7ae3b542012d4f35a9c363eee22976ec P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:27:10.764043 30166 leader_election.cc:290] T 16649fd17d434e3c95e9d585d2404659 P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 2 election: Requested vote from peers 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033), d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209)
I20260709 12:27:10.768397 29670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3f41ce441fb047c8bbb741b1b9cbc0ec" candidate_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d13c58ad8ac441c4a9b396eb04b1a476"
I20260709 12:27:10.776561 29670 raft_consensus.cc:3060] T 3f41ce441fb047c8bbb741b1b9cbc0ec P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:10.750777 29418 catalog_manager.cc:5697] T 425777eed34441cd93e0902bdc126516 P c6610c00ffb948ce9c34f41dec3d49e8 reported cstate change: term changed from 1 to 2, leader changed from 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65) to c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68). New cstate: current_term: 2 leader_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:10.774096 30362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "16649fd17d434e3c95e9d585d2404659" candidate_uuid: "d73d92fb35234d16824c55689b5fce28" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "584dac86183a41d9b3bea1c2085e62fa"
I20260709 12:27:10.784356 30320 leader_election.cc:290] T 62f8c9472b214576b9002e02536a3ace P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 2 election: Requested vote from peers c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895), 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033)
I20260709 12:27:10.786204 29590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f8c9472b214576b9002e02536a3ace" candidate_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d73d92fb35234d16824c55689b5fce28" is_pre_election: true
I20260709 12:27:10.787186 29590 raft_consensus.cc:2393] T 62f8c9472b214576b9002e02536a3ace P d73d92fb35234d16824c55689b5fce28 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c6610c00ffb948ce9c34f41dec3d49e8 in current term 2: Already voted for candidate d73d92fb35234d16824c55689b5fce28 in this term.
I20260709 12:27:10.788486 29595 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592801934cad4789b9ce8dc09f1470bd" candidate_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d73d92fb35234d16824c55689b5fce28" is_pre_election: true
I20260709 12:27:10.789601 29595 raft_consensus.cc:2468] T 592801934cad4789b9ce8dc09f1470bd P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6610c00ffb948ce9c34f41dec3d49e8 in term 1.
I20260709 12:27:10.798813 30319 tablet_bootstrap.cc:492] T 592801934cad4789b9ce8dc09f1470bd P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
W20260709 12:27:10.801528 29560 leader_election.cc:343] T fcb6a7f720b14dc98e821f663357a0bb P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:27:10.802901 29560 leader_election.cc:343] T 2707271fd46f456aa765026642a3efa9 P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:10.808498 29634 leader_election.cc:304] T d42259e436d241b29fb4ded16d9b6455 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 584dac86183a41d9b3bea1c2085e62fa, d13c58ad8ac441c4a9b396eb04b1a476; no voters: 
I20260709 12:27:10.810611 29634 leader_election.cc:304] T 7ae3b542012d4f35a9c363eee22976ec P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 584dac86183a41d9b3bea1c2085e62fa, d13c58ad8ac441c4a9b396eb04b1a476; no voters: 
I20260709 12:27:10.810727 29671 raft_consensus.cc:2468] T fcb6a7f720b14dc98e821f663357a0bb P d13c58ad8ac441c4a9b396eb04b1a476 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d73d92fb35234d16824c55689b5fce28 in term 2.
I20260709 12:27:10.811650 30319 tablet_bootstrap.cc:492] T 592801934cad4789b9ce8dc09f1470bd P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap complete.
I20260709 12:27:10.813023 30415 raft_consensus.cc:2804] T d42259e436d241b29fb4ded16d9b6455 P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:27:10.813426 30319 ts_tablet_manager.cc:1403] T 592801934cad4789b9ce8dc09f1470bd P 584dac86183a41d9b3bea1c2085e62fa: Time spent bootstrapping tablet: real 0.271s	user 0.020s	sys 0.005s
I20260709 12:27:10.813588 30415 raft_consensus.cc:493] T d42259e436d241b29fb4ded16d9b6455 P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 584dac86183a41d9b3bea1c2085e62fa)
I20260709 12:27:10.813990 30415 raft_consensus.cc:3060] T d42259e436d241b29fb4ded16d9b6455 P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:10.814569 30364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f8c9472b214576b9002e02536a3ace" candidate_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "584dac86183a41d9b3bea1c2085e62fa" is_pre_election: true
I20260709 12:27:10.816582 30367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "742beffede4f407d8a72909d2b147e6d" candidate_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "584dac86183a41d9b3bea1c2085e62fa"
I20260709 12:27:10.817068 30389 raft_consensus.cc:2804] T 2707271fd46f456aa765026642a3efa9 P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:27:10.817575 30389 raft_consensus.cc:493] T 2707271fd46f456aa765026642a3efa9 P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 584dac86183a41d9b3bea1c2085e62fa)
I20260709 12:27:10.817987 30389 raft_consensus.cc:3060] T 2707271fd46f456aa765026642a3efa9 P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:10.821525 30365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4516786e5564e9eac08ca472f79a9f0" candidate_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "584dac86183a41d9b3bea1c2085e62fa"
I20260709 12:27:10.822151 30414 raft_consensus.cc:2804] T 7ae3b542012d4f35a9c363eee22976ec P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:27:10.822613 30414 raft_consensus.cc:493] T 7ae3b542012d4f35a9c363eee22976ec P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 584dac86183a41d9b3bea1c2085e62fa)
I20260709 12:27:10.822991 30414 raft_consensus.cc:3060] T 7ae3b542012d4f35a9c363eee22976ec P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Advancing to term 2
W20260709 12:27:10.829857 29560 leader_election.cc:343] T 16649fd17d434e3c95e9d585d2404659 P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:10.807050 29589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d42259e436d241b29fb4ded16d9b6455" candidate_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d73d92fb35234d16824c55689b5fce28" is_pre_election: true
W20260709 12:27:10.811623 29634 leader_election.cc:343] T 16649fd17d434e3c95e9d585d2404659 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:27:10.833503 29634 leader_election.cc:343] T 71e2de235adf46d9a2407659080d42ef P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:10.865242 30363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592801934cad4789b9ce8dc09f1470bd" candidate_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "584dac86183a41d9b3bea1c2085e62fa" is_pre_election: true
I20260709 12:27:10.865857 29670 raft_consensus.cc:2468] T 3f41ce441fb047c8bbb741b1b9cbc0ec P d13c58ad8ac441c4a9b396eb04b1a476 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6610c00ffb948ce9c34f41dec3d49e8 in term 2.
I20260709 12:27:10.866230 29671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "16649fd17d434e3c95e9d585d2404659" candidate_uuid: "d73d92fb35234d16824c55689b5fce28" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d13c58ad8ac441c4a9b396eb04b1a476"
I20260709 12:27:10.866834 29671 raft_consensus.cc:3060] T 16649fd17d434e3c95e9d585d2404659 P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:10.867003 29709 leader_election.cc:304] T 3f41ce441fb047c8bbb741b1b9cbc0ec P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c6610c00ffb948ce9c34f41dec3d49e8, d13c58ad8ac441c4a9b396eb04b1a476; no voters: 
I20260709 12:27:10.867892 30271 raft_consensus.cc:2804] T 3f41ce441fb047c8bbb741b1b9cbc0ec P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:27:10.868438 30271 raft_consensus.cc:697] T 3f41ce441fb047c8bbb741b1b9cbc0ec P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 LEADER]: Becoming Leader. State: Replica: c6610c00ffb948ce9c34f41dec3d49e8, State: Running, Role: LEADER
I20260709 12:27:10.869508 30271 consensus_queue.cc:237] T 3f41ce441fb047c8bbb741b1b9cbc0ec P c6610c00ffb948ce9c34f41dec3d49e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:27:10.872161 29708 leader_election.cc:304] T 592801934cad4789b9ce8dc09f1470bd P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c6610c00ffb948ce9c34f41dec3d49e8, d73d92fb35234d16824c55689b5fce28; no voters: 
I20260709 12:27:10.875211 30388 raft_consensus.cc:2804] T 592801934cad4789b9ce8dc09f1470bd P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:27:10.875624 30388 raft_consensus.cc:493] T 592801934cad4789b9ce8dc09f1470bd P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 584dac86183a41d9b3bea1c2085e62fa)
I20260709 12:27:10.876036 30388 raft_consensus.cc:3060] T 592801934cad4789b9ce8dc09f1470bd P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:10.879300 30359 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3f41ce441fb047c8bbb741b1b9cbc0ec" candidate_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "584dac86183a41d9b3bea1c2085e62fa"
I20260709 12:27:10.880115 30359 raft_consensus.cc:3060] T 3f41ce441fb047c8bbb741b1b9cbc0ec P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:10.881018 29590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ae3b542012d4f35a9c363eee22976ec" candidate_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d73d92fb35234d16824c55689b5fce28" is_pre_election: true
I20260709 12:27:10.881795 29590 raft_consensus.cc:2468] T 7ae3b542012d4f35a9c363eee22976ec P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d13c58ad8ac441c4a9b396eb04b1a476 in term 1.
I20260709 12:27:10.883001 29594 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "16649fd17d434e3c95e9d585d2404659" candidate_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d73d92fb35234d16824c55689b5fce28" is_pre_election: true
I20260709 12:27:10.884027 29594 raft_consensus.cc:2393] T 16649fd17d434e3c95e9d585d2404659 P d73d92fb35234d16824c55689b5fce28 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d13c58ad8ac441c4a9b396eb04b1a476 in current term 2: Already voted for candidate d73d92fb35234d16824c55689b5fce28 in this term.
I20260709 12:27:10.885344 29741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f8c9472b214576b9002e02536a3ace" candidate_uuid: "d73d92fb35234d16824c55689b5fce28" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c6610c00ffb948ce9c34f41dec3d49e8"
I20260709 12:27:10.886008 29741 raft_consensus.cc:3060] T 62f8c9472b214576b9002e02536a3ace P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:10.857456 29589 raft_consensus.cc:2468] T d42259e436d241b29fb4ded16d9b6455 P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d13c58ad8ac441c4a9b396eb04b1a476 in term 1.
I20260709 12:27:10.800423 30397 raft_consensus.cc:515] T 71e2de235adf46d9a2407659080d42ef P d73d92fb35234d16824c55689b5fce28 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:27:10.898259 29671 raft_consensus.cc:2468] T 16649fd17d434e3c95e9d585d2404659 P d13c58ad8ac441c4a9b396eb04b1a476 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d73d92fb35234d16824c55689b5fce28 in term 2.
I20260709 12:27:10.904522 30319 raft_consensus.cc:359] T 592801934cad4789b9ce8dc09f1470bd P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:10.905256 30319 raft_consensus.cc:740] T 592801934cad4789b9ce8dc09f1470bd P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 584dac86183a41d9b3bea1c2085e62fa, State: Initialized, Role: FOLLOWER
W20260709 12:27:10.903913 29708 leader_election.cc:343] T 62f8c9472b214576b9002e02536a3ace P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:10.909080 29708 leader_election.cc:304] T 62f8c9472b214576b9002e02536a3ace P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c6610c00ffb948ce9c34f41dec3d49e8; no voters: 584dac86183a41d9b3bea1c2085e62fa, d73d92fb35234d16824c55689b5fce28
W20260709 12:27:10.909987 29708 leader_election.cc:343] T 742beffede4f407d8a72909d2b147e6d P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:27:10.910719 29708 leader_election.cc:343] T f4516786e5564e9eac08ca472f79a9f0 P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:10.906550 29561 leader_election.cc:304] T fcb6a7f720b14dc98e821f663357a0bb P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d13c58ad8ac441c4a9b396eb04b1a476, d73d92fb35234d16824c55689b5fce28; no voters: 584dac86183a41d9b3bea1c2085e62fa
I20260709 12:27:10.893713 29592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "71e2de235adf46d9a2407659080d42ef" candidate_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d73d92fb35234d16824c55689b5fce28" is_pre_election: true
I20260709 12:27:10.916004 29592 raft_consensus.cc:2393] T 71e2de235adf46d9a2407659080d42ef P d73d92fb35234d16824c55689b5fce28 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d13c58ad8ac441c4a9b396eb04b1a476 in current term 2: Already voted for candidate d73d92fb35234d16824c55689b5fce28 in this term.
I20260709 12:27:10.925297 30320 raft_consensus.cc:2804] T fcb6a7f720b14dc98e821f663357a0bb P d73d92fb35234d16824c55689b5fce28 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:27:10.926306 30320 raft_consensus.cc:697] T fcb6a7f720b14dc98e821f663357a0bb P d73d92fb35234d16824c55689b5fce28 [term 2 LEADER]: Becoming Leader. State: Replica: d73d92fb35234d16824c55689b5fce28, State: Running, Role: LEADER
I20260709 12:27:10.919512 30360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f8c9472b214576b9002e02536a3ace" candidate_uuid: "d73d92fb35234d16824c55689b5fce28" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "584dac86183a41d9b3bea1c2085e62fa"
I20260709 12:27:10.912951 30363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3f41ce441fb047c8bbb741b1b9cbc0ec" candidate_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "584dac86183a41d9b3bea1c2085e62fa" is_pre_election: true
W20260709 12:27:10.937567 29708 leader_election.cc:343] T 592801934cad4789b9ce8dc09f1470bd P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:10.938966 29561 leader_election.cc:304] T 16649fd17d434e3c95e9d585d2404659 P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d13c58ad8ac441c4a9b396eb04b1a476, d73d92fb35234d16824c55689b5fce28; no voters: 584dac86183a41d9b3bea1c2085e62fa
I20260709 12:27:10.943742 30166 raft_consensus.cc:2804] T 16649fd17d434e3c95e9d585d2404659 P d73d92fb35234d16824c55689b5fce28 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:27:10.944811 30166 raft_consensus.cc:697] T 16649fd17d434e3c95e9d585d2404659 P d73d92fb35234d16824c55689b5fce28 [term 2 LEADER]: Becoming Leader. State: Replica: d73d92fb35234d16824c55689b5fce28, State: Running, Role: LEADER
I20260709 12:27:10.927412 30320 consensus_queue.cc:237] T fcb6a7f720b14dc98e821f663357a0bb P d73d92fb35234d16824c55689b5fce28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:27:10.961004 29741 raft_consensus.cc:2468] T 62f8c9472b214576b9002e02536a3ace P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d73d92fb35234d16824c55689b5fce28 in term 2.
I20260709 12:27:10.914651 30319 consensus_queue.cc:260] T 592801934cad4789b9ce8dc09f1470bd P 584dac86183a41d9b3bea1c2085e62fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:10.975463 29634 leader_election.cc:304] T 16649fd17d434e3c95e9d585d2404659 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d13c58ad8ac441c4a9b396eb04b1a476; no voters: 584dac86183a41d9b3bea1c2085e62fa, d73d92fb35234d16824c55689b5fce28
I20260709 12:27:10.978885 30385 raft_consensus.cc:2749] T 16649fd17d434e3c95e9d585d2404659 P d13c58ad8ac441c4a9b396eb04b1a476 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:27:10.976817 30324 raft_consensus.cc:2749] T 62f8c9472b214576b9002e02536a3ace P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260709 12:27:10.994598 29560 leader_election.cc:343] T 62f8c9472b214576b9002e02536a3ace P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:27:10.999775 29709 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:38209, user_credentials={real_user=slave}} blocked reactor thread for 106171us
I20260709 12:27:10.995833 30319 ts_tablet_manager.cc:1434] T 592801934cad4789b9ce8dc09f1470bd P 584dac86183a41d9b3bea1c2085e62fa: Time spent starting tablet: real 0.182s	user 0.003s	sys 0.009s
I20260709 12:27:10.965287 30386 raft_consensus.cc:515] T c6e621aa77d5474580dfc5949801ccc9 P d13c58ad8ac441c4a9b396eb04b1a476 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:27:11.010844 30166 consensus_queue.cc:237] T 16649fd17d434e3c95e9d585d2404659 P d73d92fb35234d16824c55689b5fce28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:27:11.015820 30319 tablet_bootstrap.cc:492] T a3dcbb9c64bf4b5bb6680e8cb81af93e P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap starting.
I20260709 12:27:11.017590 30386 leader_election.cc:290] T c6e621aa77d5474580dfc5949801ccc9 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 2 election: Requested vote from peers 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033), d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825)
W20260709 12:27:11.025648 29634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:36825, user_credentials={real_user=slave}} blocked reactor thread for 49000.2us
I20260709 12:27:11.027114 29634 leader_election.cc:304] T 71e2de235adf46d9a2407659080d42ef P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d13c58ad8ac441c4a9b396eb04b1a476; no voters: 584dac86183a41d9b3bea1c2085e62fa, d73d92fb35234d16824c55689b5fce28
W20260709 12:27:11.032858 29634 consensus_peers.cc:597] T e67a2affef634fd8bc96a64fa83cffdb P d13c58ad8ac441c4a9b396eb04b1a476 -> Peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Couldn't send request to peer 584dac86183a41d9b3bea1c2085e62fa. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:11.021369 30397 leader_election.cc:290] T 71e2de235adf46d9a2407659080d42ef P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 2 election: Requested vote from peers 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033), d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209)
I20260709 12:27:11.040268 30388 raft_consensus.cc:515] T 592801934cad4789b9ce8dc09f1470bd P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:11.041702 30389 raft_consensus.cc:515] T 2707271fd46f456aa765026642a3efa9 P d73d92fb35234d16824c55689b5fce28 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:27:11.043073 30388 leader_election.cc:290] T 592801934cad4789b9ce8dc09f1470bd P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 2 election: Requested vote from peers 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033), d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825)
I20260709 12:27:11.031996 30321 raft_consensus.cc:3060] T 71e2de235adf46d9a2407659080d42ef P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:11.045960 30360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "71e2de235adf46d9a2407659080d42ef" candidate_uuid: "d73d92fb35234d16824c55689b5fce28" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "584dac86183a41d9b3bea1c2085e62fa"
I20260709 12:27:11.058372 30365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592801934cad4789b9ce8dc09f1470bd" candidate_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "584dac86183a41d9b3bea1c2085e62fa"
I20260709 12:27:11.059032 30365 raft_consensus.cc:3060] T 592801934cad4789b9ce8dc09f1470bd P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:11.059582 30363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6e621aa77d5474580dfc5949801ccc9" candidate_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "584dac86183a41d9b3bea1c2085e62fa"
I20260709 12:27:11.060372 30363 raft_consensus.cc:3060] T c6e621aa77d5474580dfc5949801ccc9 P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:11.062022 29743 raft_consensus.cc:1275] T e67a2affef634fd8bc96a64fa83cffdb P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Refusing update from remote peer d13c58ad8ac441c4a9b396eb04b1a476: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:27:11.068343 30385 consensus_queue.cc:1048] T e67a2affef634fd8bc96a64fa83cffdb P d13c58ad8ac441c4a9b396eb04b1a476 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.006s
I20260709 12:27:11.029726 30414 raft_consensus.cc:515] T 7ae3b542012d4f35a9c363eee22976ec P d13c58ad8ac441c4a9b396eb04b1a476 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:27:11.080879 30414 leader_election.cc:290] T 7ae3b542012d4f35a9c363eee22976ec P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 2 election: Requested vote from peers 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033), d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825)
I20260709 12:27:11.085958 30363 raft_consensus.cc:2468] T c6e621aa77d5474580dfc5949801ccc9 P 584dac86183a41d9b3bea1c2085e62fa [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d13c58ad8ac441c4a9b396eb04b1a476 in term 2.
I20260709 12:27:11.086589 30359 raft_consensus.cc:2468] T 3f41ce441fb047c8bbb741b1b9cbc0ec P 584dac86183a41d9b3bea1c2085e62fa [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6610c00ffb948ce9c34f41dec3d49e8 in term 2.
I20260709 12:27:11.111462 30413 heartbeater.cc:344] Connected to a master server at 127.16.29.126:41639
I20260709 12:27:11.112890 29671 raft_consensus.cc:1275] T f4516786e5564e9eac08ca472f79a9f0 P d13c58ad8ac441c4a9b396eb04b1a476 [term 2 FOLLOWER]: Refusing update from remote peer c6610c00ffb948ce9c34f41dec3d49e8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:27:11.116856 30389 leader_election.cc:290] T 2707271fd46f456aa765026642a3efa9 P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 2 election: Requested vote from peers 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033), d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209)
I20260709 12:27:11.091975 30415 raft_consensus.cc:515] T d42259e436d241b29fb4ded16d9b6455 P d13c58ad8ac441c4a9b396eb04b1a476 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:27:11.120314 30415 leader_election.cc:290] T d42259e436d241b29fb4ded16d9b6455 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 2 election: Requested vote from peers 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033), d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825)
I20260709 12:27:11.123047 30426 consensus_queue.cc:1048] T f4516786e5564e9eac08ca472f79a9f0 P c6610c00ffb948ce9c34f41dec3d49e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:27:11.124221 29634 leader_election.cc:304] T c6e621aa77d5474580dfc5949801ccc9 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 584dac86183a41d9b3bea1c2085e62fa, d13c58ad8ac441c4a9b396eb04b1a476; no voters: 
I20260709 12:27:11.125062 30359 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ae3b542012d4f35a9c363eee22976ec" candidate_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "584dac86183a41d9b3bea1c2085e62fa"
I20260709 12:27:11.125793 30359 raft_consensus.cc:3060] T 7ae3b542012d4f35a9c363eee22976ec P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:11.126582 30431 raft_consensus.cc:2804] T c6e621aa77d5474580dfc5949801ccc9 P d13c58ad8ac441c4a9b396eb04b1a476 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:27:11.127123 30431 raft_consensus.cc:697] T c6e621aa77d5474580dfc5949801ccc9 P d13c58ad8ac441c4a9b396eb04b1a476 [term 2 LEADER]: Becoming Leader. State: Replica: d13c58ad8ac441c4a9b396eb04b1a476, State: Running, Role: LEADER
W20260709 12:27:11.135855 29561 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:43895, user_credentials={real_user=slave}} blocked reactor thread for 140666us
W20260709 12:27:11.141192 29560 leader_election.cc:343] T 71e2de235adf46d9a2407659080d42ef P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:11.136917 29561 leader_election.cc:304] T 62f8c9472b214576b9002e02536a3ace P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c6610c00ffb948ce9c34f41dec3d49e8, d73d92fb35234d16824c55689b5fce28; no voters: 584dac86183a41d9b3bea1c2085e62fa
I20260709 12:27:11.138513 30413 heartbeater.cc:461] Registering TS with master...
I20260709 12:27:11.147804 30368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d42259e436d241b29fb4ded16d9b6455" candidate_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "584dac86183a41d9b3bea1c2085e62fa"
I20260709 12:27:11.150468 30368 raft_consensus.cc:3060] T d42259e436d241b29fb4ded16d9b6455 P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:11.158938 30361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2707271fd46f456aa765026642a3efa9" candidate_uuid: "d73d92fb35234d16824c55689b5fce28" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "584dac86183a41d9b3bea1c2085e62fa"
I20260709 12:27:11.161801 30431 consensus_queue.cc:237] T c6e621aa77d5474580dfc5949801ccc9 P d13c58ad8ac441c4a9b396eb04b1a476 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:27:11.162607 30413 heartbeater.cc:507] Master 127.16.29.126:41639 requested a full tablet report, sending...
I20260709 12:27:11.163267 30321 raft_consensus.cc:2749] T 71e2de235adf46d9a2407659080d42ef P d13c58ad8ac441c4a9b396eb04b1a476 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:27:11.164974 29417 catalog_manager.cc:5697] T fcb6a7f720b14dc98e821f663357a0bb P d73d92fb35234d16824c55689b5fce28 reported cstate change: term changed from 1 to 2, leader changed from 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65) to d73d92fb35234d16824c55689b5fce28 (127.16.29.66). New cstate: current_term: 2 leader_uuid: "d73d92fb35234d16824c55689b5fce28" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:11.170990 29671 raft_consensus.cc:1275] T 425777eed34441cd93e0902bdc126516 P d13c58ad8ac441c4a9b396eb04b1a476 [term 2 FOLLOWER]: Refusing update from remote peer c6610c00ffb948ce9c34f41dec3d49e8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20260709 12:27:11.175448 29708 consensus_peers.cc:597] T 425777eed34441cd93e0902bdc126516 P c6610c00ffb948ce9c34f41dec3d49e8 -> Peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Couldn't send request to peer 584dac86183a41d9b3bea1c2085e62fa. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:11.178568 29560 leader_election.cc:343] T 2707271fd46f456aa765026642a3efa9 P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:11.156201 30365 raft_consensus.cc:2468] T 592801934cad4789b9ce8dc09f1470bd P 584dac86183a41d9b3bea1c2085e62fa [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6610c00ffb948ce9c34f41dec3d49e8 in term 2.
I20260709 12:27:11.178304 30388 consensus_queue.cc:1048] T 425777eed34441cd93e0902bdc126516 P c6610c00ffb948ce9c34f41dec3d49e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.016s
I20260709 12:27:11.201386 30435 raft_consensus.cc:2804] T 62f8c9472b214576b9002e02536a3ace P d73d92fb35234d16824c55689b5fce28 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:27:11.202023 30435 raft_consensus.cc:697] T 62f8c9472b214576b9002e02536a3ace P d73d92fb35234d16824c55689b5fce28 [term 2 LEADER]: Becoming Leader. State: Replica: d73d92fb35234d16824c55689b5fce28, State: Running, Role: LEADER
I20260709 12:27:11.202978 30435 consensus_queue.cc:237] T 62f8c9472b214576b9002e02536a3ace P d73d92fb35234d16824c55689b5fce28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:11.203902 29671 raft_consensus.cc:1275] T 742beffede4f407d8a72909d2b147e6d P d13c58ad8ac441c4a9b396eb04b1a476 [term 2 FOLLOWER]: Refusing update from remote peer c6610c00ffb948ce9c34f41dec3d49e8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:27:11.205855 30426 consensus_queue.cc:1048] T 742beffede4f407d8a72909d2b147e6d P c6610c00ffb948ce9c34f41dec3d49e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:27:11.207979 30368 raft_consensus.cc:2468] T d42259e436d241b29fb4ded16d9b6455 P 584dac86183a41d9b3bea1c2085e62fa [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d13c58ad8ac441c4a9b396eb04b1a476 in term 2.
I20260709 12:27:11.200042 29708 leader_election.cc:304] T 592801934cad4789b9ce8dc09f1470bd P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 584dac86183a41d9b3bea1c2085e62fa, c6610c00ffb948ce9c34f41dec3d49e8; no voters: 
I20260709 12:27:11.216125 30271 raft_consensus.cc:2804] T 592801934cad4789b9ce8dc09f1470bd P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Leader election won for term 2
W20260709 12:27:11.224869 29708 consensus_peers.cc:597] T 742beffede4f407d8a72909d2b147e6d P c6610c00ffb948ce9c34f41dec3d49e8 -> Peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Couldn't send request to peer 584dac86183a41d9b3bea1c2085e62fa. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:11.216679 30271 raft_consensus.cc:697] T 592801934cad4789b9ce8dc09f1470bd P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 LEADER]: Becoming Leader. State: Replica: c6610c00ffb948ce9c34f41dec3d49e8, State: Running, Role: LEADER
I20260709 12:27:11.232220 29634 leader_election.cc:304] T d42259e436d241b29fb4ded16d9b6455 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 584dac86183a41d9b3bea1c2085e62fa, d13c58ad8ac441c4a9b396eb04b1a476; no voters: 
I20260709 12:27:11.234062 30431 raft_consensus.cc:2804] T d42259e436d241b29fb4ded16d9b6455 P d13c58ad8ac441c4a9b396eb04b1a476 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:27:11.234596 30431 raft_consensus.cc:697] T d42259e436d241b29fb4ded16d9b6455 P d13c58ad8ac441c4a9b396eb04b1a476 [term 2 LEADER]: Becoming Leader. State: Replica: d13c58ad8ac441c4a9b396eb04b1a476, State: Running, Role: LEADER
I20260709 12:27:11.235476 30431 consensus_queue.cc:237] T d42259e436d241b29fb4ded16d9b6455 P d13c58ad8ac441c4a9b396eb04b1a476 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:27:11.235841 30271 consensus_queue.cc:237] T 592801934cad4789b9ce8dc09f1470bd P c6610c00ffb948ce9c34f41dec3d49e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
W20260709 12:27:11.247289 29561 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:38209, user_credentials={real_user=slave}} blocked reactor thread for 91562.7us
I20260709 12:27:11.262022 29668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "71e2de235adf46d9a2407659080d42ef" candidate_uuid: "d73d92fb35234d16824c55689b5fce28" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d13c58ad8ac441c4a9b396eb04b1a476"
I20260709 12:27:11.270445 29595 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ae3b542012d4f35a9c363eee22976ec" candidate_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d73d92fb35234d16824c55689b5fce28"
I20260709 12:27:11.271173 29595 raft_consensus.cc:3060] T 7ae3b542012d4f35a9c363eee22976ec P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:11.272387 29597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592801934cad4789b9ce8dc09f1470bd" candidate_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d73d92fb35234d16824c55689b5fce28"
I20260709 12:27:11.273180 29597 raft_consensus.cc:3060] T 592801934cad4789b9ce8dc09f1470bd P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:11.277173 29590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d42259e436d241b29fb4ded16d9b6455" candidate_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d73d92fb35234d16824c55689b5fce28"
I20260709 12:27:11.277808 29590 raft_consensus.cc:3060] T d42259e436d241b29fb4ded16d9b6455 P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Advancing to term 2
W20260709 12:27:11.280437 29708 consensus_peers.cc:597] T f4516786e5564e9eac08ca472f79a9f0 P c6610c00ffb948ce9c34f41dec3d49e8 -> Peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Couldn't send request to peer 584dac86183a41d9b3bea1c2085e62fa. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:11.297751 29708 consensus_peers.cc:597] T ef20d7124e7a44aa97a219ee194f4b8b P c6610c00ffb948ce9c34f41dec3d49e8 -> Peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Couldn't send request to peer 584dac86183a41d9b3bea1c2085e62fa. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:27:11.290637 29419 catalog_manager.cc:5697] T c6e621aa77d5474580dfc5949801ccc9 P d13c58ad8ac441c4a9b396eb04b1a476 reported cstate change: term changed from 1 to 2, leader changed from 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65) to d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67). New cstate: current_term: 2 leader_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } health_report { overall_health: HEALTHY } } }
W20260709 12:27:11.312754 30437 log.cc:927] Time spent T f4516786e5564e9eac08ca472f79a9f0 P d13c58ad8ac441c4a9b396eb04b1a476: Append to log took a long time: real 0.083s	user 0.001s	sys 0.000s
I20260709 12:27:11.283962 29589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6e621aa77d5474580dfc5949801ccc9" candidate_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d73d92fb35234d16824c55689b5fce28"
I20260709 12:27:11.315579 29589 raft_consensus.cc:3060] T c6e621aa77d5474580dfc5949801ccc9 P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:11.302040 29419 catalog_manager.cc:5697] T f4516786e5564e9eac08ca472f79a9f0 P d13c58ad8ac441c4a9b396eb04b1a476 reported cstate change: term changed from 1 to 2, leader changed from 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65) to c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68). New cstate: current_term: 2 leader_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } }
I20260709 12:27:11.327898 29670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2707271fd46f456aa765026642a3efa9" candidate_uuid: "d73d92fb35234d16824c55689b5fce28" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d13c58ad8ac441c4a9b396eb04b1a476"
I20260709 12:27:11.328577 29670 raft_consensus.cc:3060] T 2707271fd46f456aa765026642a3efa9 P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:11.338902 29595 raft_consensus.cc:2468] T 7ae3b542012d4f35a9c363eee22976ec P d73d92fb35234d16824c55689b5fce28 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d13c58ad8ac441c4a9b396eb04b1a476 in term 2.
I20260709 12:27:11.372979 30359 raft_consensus.cc:2468] T 7ae3b542012d4f35a9c363eee22976ec P 584dac86183a41d9b3bea1c2085e62fa [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d13c58ad8ac441c4a9b396eb04b1a476 in term 2.
W20260709 12:27:11.376477 30440 log.cc:927] Time spent T 425777eed34441cd93e0902bdc126516 P d13c58ad8ac441c4a9b396eb04b1a476: Append to log took a long time: real 0.084s	user 0.001s	sys 0.000s
I20260709 12:27:11.382093 29670 raft_consensus.cc:2468] T 2707271fd46f456aa765026642a3efa9 P d13c58ad8ac441c4a9b396eb04b1a476 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d73d92fb35234d16824c55689b5fce28 in term 2.
I20260709 12:27:11.393647 29417 catalog_manager.cc:5697] T 742beffede4f407d8a72909d2b147e6d P c6610c00ffb948ce9c34f41dec3d49e8 reported cstate change: term changed from 1 to 2, leader changed from 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65) to c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68). New cstate: current_term: 2 leader_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } health_report { overall_health: HEALTHY } } }
I20260709 12:27:11.403213 29668 raft_consensus.cc:2468] T 71e2de235adf46d9a2407659080d42ef P d13c58ad8ac441c4a9b396eb04b1a476 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d73d92fb35234d16824c55689b5fce28 in term 2.
I20260709 12:27:11.408061 29634 leader_election.cc:304] T 7ae3b542012d4f35a9c363eee22976ec P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d13c58ad8ac441c4a9b396eb04b1a476, d73d92fb35234d16824c55689b5fce28; no voters: 
I20260709 12:27:11.409363 29597 raft_consensus.cc:2468] T 592801934cad4789b9ce8dc09f1470bd P d73d92fb35234d16824c55689b5fce28 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6610c00ffb948ce9c34f41dec3d49e8 in term 2.
I20260709 12:27:11.410881 29561 leader_election.cc:304] T 2707271fd46f456aa765026642a3efa9 P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d13c58ad8ac441c4a9b396eb04b1a476, d73d92fb35234d16824c55689b5fce28; no voters: 584dac86183a41d9b3bea1c2085e62fa
I20260709 12:27:11.410521 29417 catalog_manager.cc:5697] T 3f41ce441fb047c8bbb741b1b9cbc0ec P c6610c00ffb948ce9c34f41dec3d49e8 reported cstate change: term changed from 1 to 2, leader changed from 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65) to c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68). New cstate: current_term: 2 leader_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:11.411689 30389 raft_consensus.cc:2804] T 2707271fd46f456aa765026642a3efa9 P d73d92fb35234d16824c55689b5fce28 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:27:11.412237 30389 raft_consensus.cc:697] T 2707271fd46f456aa765026642a3efa9 P d73d92fb35234d16824c55689b5fce28 [term 2 LEADER]: Becoming Leader. State: Replica: d73d92fb35234d16824c55689b5fce28, State: Running, Role: LEADER
I20260709 12:27:11.415022 30319 tablet_bootstrap.cc:492] T a3dcbb9c64bf4b5bb6680e8cb81af93e P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:27:11.413144 30389 consensus_queue.cc:237] T 2707271fd46f456aa765026642a3efa9 P d73d92fb35234d16824c55689b5fce28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:27:11.418067 29589 raft_consensus.cc:2468] T c6e621aa77d5474580dfc5949801ccc9 P d73d92fb35234d16824c55689b5fce28 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d13c58ad8ac441c4a9b396eb04b1a476 in term 2.
I20260709 12:27:11.419189 29590 raft_consensus.cc:2468] T d42259e436d241b29fb4ded16d9b6455 P d73d92fb35234d16824c55689b5fce28 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d13c58ad8ac441c4a9b396eb04b1a476 in term 2.
I20260709 12:27:11.420614 30319 tablet_bootstrap.cc:492] T a3dcbb9c64bf4b5bb6680e8cb81af93e P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap complete.
I20260709 12:27:11.421381 30319 ts_tablet_manager.cc:1403] T a3dcbb9c64bf4b5bb6680e8cb81af93e P 584dac86183a41d9b3bea1c2085e62fa: Time spent bootstrapping tablet: real 0.407s	user 0.030s	sys 0.003s
I20260709 12:27:11.422518 30429 raft_consensus.cc:2804] T 7ae3b542012d4f35a9c363eee22976ec P d13c58ad8ac441c4a9b396eb04b1a476 [term 2 FOLLOWER]: Leader election won for term 2
W20260709 12:27:11.430614 29634 consensus_peers.cc:597] T 31a190e9f5ff407fab90502fb3197b70 P d13c58ad8ac441c4a9b396eb04b1a476 -> Peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Couldn't send request to peer 584dac86183a41d9b3bea1c2085e62fa. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:27:11.432132 30319 raft_consensus.cc:359] T a3dcbb9c64bf4b5bb6680e8cb81af93e P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:11.433327 30319 raft_consensus.cc:740] T a3dcbb9c64bf4b5bb6680e8cb81af93e P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 584dac86183a41d9b3bea1c2085e62fa, State: Initialized, Role: FOLLOWER
I20260709 12:27:11.436620 30319 consensus_queue.cc:260] T a3dcbb9c64bf4b5bb6680e8cb81af93e P 584dac86183a41d9b3bea1c2085e62fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
W20260709 12:27:11.431589 29634 consensus_peers.cc:597] T f3e17f0a8fc14095850e8138fd79b86e P d13c58ad8ac441c4a9b396eb04b1a476 -> Peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Couldn't send request to peer 584dac86183a41d9b3bea1c2085e62fa. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:27:11.435572 30429 raft_consensus.cc:697] T 7ae3b542012d4f35a9c363eee22976ec P d13c58ad8ac441c4a9b396eb04b1a476 [term 2 LEADER]: Becoming Leader. State: Replica: d13c58ad8ac441c4a9b396eb04b1a476, State: Running, Role: LEADER
I20260709 12:27:11.446391 29561 leader_election.cc:304] T 71e2de235adf46d9a2407659080d42ef P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d13c58ad8ac441c4a9b396eb04b1a476, d73d92fb35234d16824c55689b5fce28; no voters: 584dac86183a41d9b3bea1c2085e62fa
I20260709 12:27:11.446242 30429 consensus_queue.cc:237] T 7ae3b542012d4f35a9c363eee22976ec P d13c58ad8ac441c4a9b396eb04b1a476 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:27:11.448861 30399 raft_consensus.cc:2804] T 71e2de235adf46d9a2407659080d42ef P d73d92fb35234d16824c55689b5fce28 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:27:11.449458 30399 raft_consensus.cc:697] T 71e2de235adf46d9a2407659080d42ef P d73d92fb35234d16824c55689b5fce28 [term 2 LEADER]: Becoming Leader. State: Replica: d73d92fb35234d16824c55689b5fce28, State: Running, Role: LEADER
I20260709 12:27:11.460292 29667 raft_consensus.cc:1275] T 3f41ce441fb047c8bbb741b1b9cbc0ec P d13c58ad8ac441c4a9b396eb04b1a476 [term 2 FOLLOWER]: Refusing update from remote peer c6610c00ffb948ce9c34f41dec3d49e8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:27:11.458875 30399 consensus_queue.cc:237] T 71e2de235adf46d9a2407659080d42ef P d73d92fb35234d16824c55689b5fce28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:27:11.468956 30319 ts_tablet_manager.cc:1434] T a3dcbb9c64bf4b5bb6680e8cb81af93e P 584dac86183a41d9b3bea1c2085e62fa: Time spent starting tablet: real 0.047s	user 0.007s	sys 0.000s
I20260709 12:27:11.470870 30319 tablet_bootstrap.cc:492] T f4516786e5564e9eac08ca472f79a9f0 P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap starting.
I20260709 12:27:11.477638 30324 consensus_queue.cc:1048] T 3f41ce441fb047c8bbb741b1b9cbc0ec P c6610c00ffb948ce9c34f41dec3d49e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:27:11.481981 29419 catalog_manager.cc:5697] T d42259e436d241b29fb4ded16d9b6455 P d13c58ad8ac441c4a9b396eb04b1a476 reported cstate change: term changed from 1 to 2, leader changed from 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65) to d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67). New cstate: current_term: 2 leader_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } health_report { overall_health: HEALTHY } } }
I20260709 12:27:11.482518 29418 catalog_manager.cc:5697] T 16649fd17d434e3c95e9d585d2404659 P d73d92fb35234d16824c55689b5fce28 reported cstate change: term changed from 1 to 2, leader changed from 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65) to d73d92fb35234d16824c55689b5fce28 (127.16.29.66). New cstate: current_term: 2 leader_uuid: "d73d92fb35234d16824c55689b5fce28" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } health_report { overall_health: UNKNOWN } } }
W20260709 12:27:11.496712 29634 consensus_peers.cc:597] T f76101eb30f14b98a9aa8c41e0d6f831 P d13c58ad8ac441c4a9b396eb04b1a476 -> Peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Couldn't send request to peer 584dac86183a41d9b3bea1c2085e62fa. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:27:11.502207 30366 raft_consensus.cc:1275] T 3f41ce441fb047c8bbb741b1b9cbc0ec P 584dac86183a41d9b3bea1c2085e62fa [term 2 FOLLOWER]: Refusing update from remote peer c6610c00ffb948ce9c34f41dec3d49e8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20260709 12:27:11.512022 29560 consensus_peers.cc:597] T f8c01e05981747a395e5483b551297dd P d73d92fb35234d16824c55689b5fce28 -> Peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Couldn't send request to peer 584dac86183a41d9b3bea1c2085e62fa. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:27:11.495160 29418 catalog_manager.cc:5697] T 62f8c9472b214576b9002e02536a3ace P d73d92fb35234d16824c55689b5fce28 reported cstate change: term changed from 1 to 2, leader changed from 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65) to d73d92fb35234d16824c55689b5fce28 (127.16.29.66). New cstate: current_term: 2 leader_uuid: "d73d92fb35234d16824c55689b5fce28" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } health_report { overall_health: HEALTHY } } }
W20260709 12:27:11.527835 29560 consensus_peers.cc:597] T fcb6a7f720b14dc98e821f663357a0bb P d73d92fb35234d16824c55689b5fce28 -> Peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Couldn't send request to peer 584dac86183a41d9b3bea1c2085e62fa. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:11.531987 29634 consensus_peers.cc:597] T 6e37d72807e547898f4fce9e8b82b2b0 P d13c58ad8ac441c4a9b396eb04b1a476 -> Peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Couldn't send request to peer 584dac86183a41d9b3bea1c2085e62fa. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 6: this message will repeat every 5th retry.
W20260709 12:27:11.538828 29560 consensus_peers.cc:597] T 16649fd17d434e3c95e9d585d2404659 P d73d92fb35234d16824c55689b5fce28 -> Peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Couldn't send request to peer 584dac86183a41d9b3bea1c2085e62fa. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:11.540998 29667 raft_consensus.cc:1275] T 16649fd17d434e3c95e9d585d2404659 P d13c58ad8ac441c4a9b396eb04b1a476 [term 2 FOLLOWER]: Refusing update from remote peer d73d92fb35234d16824c55689b5fce28: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:27:11.543916 29420 ts_manager.cc:194] Re-registered known tserver with Master: 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033)
I20260709 12:27:11.546041 29668 raft_consensus.cc:1275] T fcb6a7f720b14dc98e821f663357a0bb P d13c58ad8ac441c4a9b396eb04b1a476 [term 2 FOLLOWER]: Refusing update from remote peer d73d92fb35234d16824c55689b5fce28: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20260709 12:27:11.553926 29708 consensus_peers.cc:597] T 66f652ba2c224885a2aea974bfe8f88a P c6610c00ffb948ce9c34f41dec3d49e8 -> Peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Couldn't send request to peer 584dac86183a41d9b3bea1c2085e62fa. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:27:11.554903 30388 consensus_queue.cc:1048] T 3f41ce441fb047c8bbb741b1b9cbc0ec P c6610c00ffb948ce9c34f41dec3d49e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:27:11.555006 30389 consensus_queue.cc:1048] T 16649fd17d434e3c95e9d585d2404659 P d73d92fb35234d16824c55689b5fce28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:27:11.561443 30166 consensus_queue.cc:1048] T fcb6a7f720b14dc98e821f663357a0bb P d73d92fb35234d16824c55689b5fce28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260709 12:27:11.564925 29708 consensus_peers.cc:597] T 9633e3b1d78f403097e4a00b7d4b869b P c6610c00ffb948ce9c34f41dec3d49e8 -> Peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Couldn't send request to peer 584dac86183a41d9b3bea1c2085e62fa. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 6: this message will repeat every 5th retry.
W20260709 12:27:11.568806 29634 consensus_peers.cc:597] T 0c428ff656ae4b9d99d5b5073f98725d P d13c58ad8ac441c4a9b396eb04b1a476 -> Peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Couldn't send request to peer 584dac86183a41d9b3bea1c2085e62fa. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:27:11.587167 29417 catalog_manager.cc:5697] T 592801934cad4789b9ce8dc09f1470bd P c6610c00ffb948ce9c34f41dec3d49e8 reported cstate change: term changed from 1 to 2, leader changed from 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65) to c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68). New cstate: current_term: 2 leader_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:11.616147 30319 tablet_bootstrap.cc:492] T f4516786e5564e9eac08ca472f79a9f0 P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:27:11.617283 30319 tablet_bootstrap.cc:492] T f4516786e5564e9eac08ca472f79a9f0 P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap complete.
I20260709 12:27:11.617973 30319 ts_tablet_manager.cc:1403] T f4516786e5564e9eac08ca472f79a9f0 P 584dac86183a41d9b3bea1c2085e62fa: Time spent bootstrapping tablet: real 0.147s	user 0.014s	sys 0.003s
W20260709 12:27:11.619910 29560 consensus_peers.cc:597] T d1918297b9eb41f5847a211e668ae748 P d73d92fb35234d16824c55689b5fce28 -> Peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Couldn't send request to peer 584dac86183a41d9b3bea1c2085e62fa. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 6: this message will repeat every 5th retry.
W20260709 12:27:11.619298 29708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:35033, user_credentials={real_user=slave}} blocked reactor thread for 53609.2us
I20260709 12:27:11.620816 30319 raft_consensus.cc:359] T f4516786e5564e9eac08ca472f79a9f0 P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:27:11.630134 30319 raft_consensus.cc:740] T f4516786e5564e9eac08ca472f79a9f0 P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 584dac86183a41d9b3bea1c2085e62fa, State: Initialized, Role: FOLLOWER
I20260709 12:27:11.631065 30319 consensus_queue.cc:260] T f4516786e5564e9eac08ca472f79a9f0 P 584dac86183a41d9b3bea1c2085e62fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:27:11.634235 30319 ts_tablet_manager.cc:1434] T f4516786e5564e9eac08ca472f79a9f0 P 584dac86183a41d9b3bea1c2085e62fa: Time spent starting tablet: real 0.016s	user 0.007s	sys 0.000s
I20260709 12:27:11.635121 30319 tablet_bootstrap.cc:492] T 8017a44219404faa8db8ff049c65205e P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap starting.
W20260709 12:27:11.642324 29634 consensus_peers.cc:597] T cbf03426b6df43e5b2b9c56e2eb7687c P d13c58ad8ac441c4a9b396eb04b1a476 -> Peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Couldn't send request to peer 584dac86183a41d9b3bea1c2085e62fa. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:27:11.653952 30366 raft_consensus.cc:1275] T d42259e436d241b29fb4ded16d9b6455 P 584dac86183a41d9b3bea1c2085e62fa [term 2 FOLLOWER]: Refusing update from remote peer d13c58ad8ac441c4a9b396eb04b1a476: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:27:11.656415 30429 consensus_queue.cc:1048] T d42259e436d241b29fb4ded16d9b6455 P d13c58ad8ac441c4a9b396eb04b1a476 [LEADER]: Connected to new peer: Peer: permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260709 12:27:11.690866 29708 consensus_peers.cc:597] T 1d4e0ad0462e4f74a44155aa3ffaf87f P c6610c00ffb948ce9c34f41dec3d49e8 -> Peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Couldn't send request to peer 584dac86183a41d9b3bea1c2085e62fa. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:27:11.697044 29419 catalog_manager.cc:5697] T 71e2de235adf46d9a2407659080d42ef P d73d92fb35234d16824c55689b5fce28 reported cstate change: term changed from 1 to 2, leader changed from 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65) to d73d92fb35234d16824c55689b5fce28 (127.16.29.66). New cstate: current_term: 2 leader_uuid: "d73d92fb35234d16824c55689b5fce28" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:11.704814 30319 tablet_bootstrap.cc:492] T 8017a44219404faa8db8ff049c65205e P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:27:11.704389 29419 catalog_manager.cc:5697] T 2707271fd46f456aa765026642a3efa9 P d73d92fb35234d16824c55689b5fce28 reported cstate change: term changed from 1 to 2, leader changed from 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65) to d73d92fb35234d16824c55689b5fce28 (127.16.29.66). New cstate: current_term: 2 leader_uuid: "d73d92fb35234d16824c55689b5fce28" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:11.709630 29597 raft_consensus.cc:1275] T 592801934cad4789b9ce8dc09f1470bd P d73d92fb35234d16824c55689b5fce28 [term 2 FOLLOWER]: Refusing update from remote peer c6610c00ffb948ce9c34f41dec3d49e8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:27:11.716419 30368 raft_consensus.cc:3060] T f4516786e5564e9eac08ca472f79a9f0 P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:11.720213 30363 raft_consensus.cc:1275] T c6e621aa77d5474580dfc5949801ccc9 P 584dac86183a41d9b3bea1c2085e62fa [term 2 FOLLOWER]: Refusing update from remote peer d13c58ad8ac441c4a9b396eb04b1a476: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:27:11.721772 30388 consensus_queue.cc:1048] T 592801934cad4789b9ce8dc09f1470bd P c6610c00ffb948ce9c34f41dec3d49e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:27:11.724400 29597 raft_consensus.cc:1275] T d42259e436d241b29fb4ded16d9b6455 P d73d92fb35234d16824c55689b5fce28 [term 2 FOLLOWER]: Refusing update from remote peer d13c58ad8ac441c4a9b396eb04b1a476: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:27:11.725069 30429 consensus_queue.cc:1048] T c6e621aa77d5474580dfc5949801ccc9 P d13c58ad8ac441c4a9b396eb04b1a476 [LEADER]: Connected to new peer: Peer: permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:27:11.727664 30416 consensus_queue.cc:1048] T d42259e436d241b29fb4ded16d9b6455 P d13c58ad8ac441c4a9b396eb04b1a476 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:27:11.736336 30319 tablet_bootstrap.cc:492] T 8017a44219404faa8db8ff049c65205e P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap complete.
I20260709 12:27:11.737365 30319 ts_tablet_manager.cc:1403] T 8017a44219404faa8db8ff049c65205e P 584dac86183a41d9b3bea1c2085e62fa: Time spent bootstrapping tablet: real 0.102s	user 0.013s	sys 0.020s
W20260709 12:27:11.755257 29634 consensus_peers.cc:597] T 5ee924f859f54f64908499bfac48ee62 P d13c58ad8ac441c4a9b396eb04b1a476 -> Peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Couldn't send request to peer 584dac86183a41d9b3bea1c2085e62fa. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:27:11.759093 30319 raft_consensus.cc:359] T 8017a44219404faa8db8ff049c65205e P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:11.760316 30319 raft_consensus.cc:740] T 8017a44219404faa8db8ff049c65205e P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 584dac86183a41d9b3bea1c2085e62fa, State: Initialized, Role: FOLLOWER
I20260709 12:27:11.761607 30319 consensus_queue.cc:260] T 8017a44219404faa8db8ff049c65205e P 584dac86183a41d9b3bea1c2085e62fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:11.781456 30319 ts_tablet_manager.cc:1434] T 8017a44219404faa8db8ff049c65205e P 584dac86183a41d9b3bea1c2085e62fa: Time spent starting tablet: real 0.044s	user 0.007s	sys 0.000s
I20260709 12:27:11.782414 30319 tablet_bootstrap.cc:492] T 66f652ba2c224885a2aea974bfe8f88a P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap starting.
I20260709 12:27:11.795524 29743 raft_consensus.cc:1275] T 62f8c9472b214576b9002e02536a3ace P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Refusing update from remote peer d73d92fb35234d16824c55689b5fce28: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:27:11.797413 29417 catalog_manager.cc:5697] T 7ae3b542012d4f35a9c363eee22976ec P d13c58ad8ac441c4a9b396eb04b1a476 reported cstate change: term changed from 1 to 2, leader changed from 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65) to d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67). New cstate: current_term: 2 leader_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } health_report { overall_health: HEALTHY } } }
W20260709 12:27:11.802193 29560 consensus_peers.cc:597] T b3c02c910d2b43cc9265bb1d98491cd3 P d73d92fb35234d16824c55689b5fce28 -> Peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Couldn't send request to peer 584dac86183a41d9b3bea1c2085e62fa. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:27:11.802584 30430 consensus_queue.cc:1048] T 62f8c9472b214576b9002e02536a3ace P d73d92fb35234d16824c55689b5fce28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.005s
W20260709 12:27:11.814570 29560 consensus_peers.cc:597] T 3615115233c34461b85763151751e475 P d73d92fb35234d16824c55689b5fce28 -> Peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Couldn't send request to peer 584dac86183a41d9b3bea1c2085e62fa. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:27:11.807119 29589 raft_consensus.cc:1275] T c6e621aa77d5474580dfc5949801ccc9 P d73d92fb35234d16824c55689b5fce28 [term 2 FOLLOWER]: Refusing update from remote peer d13c58ad8ac441c4a9b396eb04b1a476: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:27:11.821493 30282 consensus_queue.cc:1048] T c6e621aa77d5474580dfc5949801ccc9 P d13c58ad8ac441c4a9b396eb04b1a476 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:27:11.830497 29420 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:55362
W20260709 12:27:11.848567 29708 consensus_peers.cc:597] T 4685c2e824c14fc1bcd45c16169798cc P c6610c00ffb948ce9c34f41dec3d49e8 -> Peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Couldn't send request to peer 584dac86183a41d9b3bea1c2085e62fa. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:27:11.858270 30359 raft_consensus.cc:1275] T 592801934cad4789b9ce8dc09f1470bd P 584dac86183a41d9b3bea1c2085e62fa [term 2 FOLLOWER]: Refusing update from remote peer c6610c00ffb948ce9c34f41dec3d49e8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20260709 12:27:11.868872 29560 consensus_peers.cc:597] T 62f8c9472b214576b9002e02536a3ace P d73d92fb35234d16824c55689b5fce28 -> Peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Couldn't send request to peer 584dac86183a41d9b3bea1c2085e62fa. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:11.870584 30413 heartbeater.cc:499] Master 127.16.29.126:41639 was elected leader, sending a full tablet report...
W20260709 12:27:11.872430 29560 consensus_peers.cc:597] T 2707271fd46f456aa765026642a3efa9 P d73d92fb35234d16824c55689b5fce28 -> Peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Couldn't send request to peer 584dac86183a41d9b3bea1c2085e62fa. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:11.864418 30388 consensus_queue.cc:1048] T 592801934cad4789b9ce8dc09f1470bd P c6610c00ffb948ce9c34f41dec3d49e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.011s
I20260709 12:27:11.892740 29591 raft_consensus.cc:1275] T 7ae3b542012d4f35a9c363eee22976ec P d73d92fb35234d16824c55689b5fce28 [term 2 FOLLOWER]: Refusing update from remote peer d13c58ad8ac441c4a9b396eb04b1a476: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:27:11.897109 30431 consensus_queue.cc:1048] T 7ae3b542012d4f35a9c363eee22976ec P d13c58ad8ac441c4a9b396eb04b1a476 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:27:11.914366 30363 raft_consensus.cc:1275] T 7ae3b542012d4f35a9c363eee22976ec P 584dac86183a41d9b3bea1c2085e62fa [term 2 FOLLOWER]: Refusing update from remote peer d13c58ad8ac441c4a9b396eb04b1a476: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:27:11.916185 30321 consensus_queue.cc:1048] T 7ae3b542012d4f35a9c363eee22976ec P d13c58ad8ac441c4a9b396eb04b1a476 [LEADER]: Connected to new peer: Peer: permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:27:11.948963 30319 tablet_bootstrap.cc:492] T 66f652ba2c224885a2aea974bfe8f88a P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:27:11.950109 30319 tablet_bootstrap.cc:492] T 66f652ba2c224885a2aea974bfe8f88a P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap complete.
I20260709 12:27:11.950939 30319 ts_tablet_manager.cc:1403] T 66f652ba2c224885a2aea974bfe8f88a P 584dac86183a41d9b3bea1c2085e62fa: Time spent bootstrapping tablet: real 0.169s	user 0.017s	sys 0.008s
W20260709 12:27:11.953329 29634 consensus_peers.cc:597] T 3e9451c3797c4496a66dfdb3e9fb5e38 P d13c58ad8ac441c4a9b396eb04b1a476 -> Peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Couldn't send request to peer 584dac86183a41d9b3bea1c2085e62fa. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:27:11.953743 30319 raft_consensus.cc:359] T 66f652ba2c224885a2aea974bfe8f88a P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:11.954629 30319 raft_consensus.cc:740] T 66f652ba2c224885a2aea974bfe8f88a P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 584dac86183a41d9b3bea1c2085e62fa, State: Initialized, Role: FOLLOWER
I20260709 12:27:11.955539 30319 consensus_queue.cc:260] T 66f652ba2c224885a2aea974bfe8f88a P 584dac86183a41d9b3bea1c2085e62fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:11.958014 30319 ts_tablet_manager.cc:1434] T 66f652ba2c224885a2aea974bfe8f88a P 584dac86183a41d9b3bea1c2085e62fa: Time spent starting tablet: real 0.007s	user 0.005s	sys 0.000s
I20260709 12:27:11.959206 30319 tablet_bootstrap.cc:492] T 62f8c9472b214576b9002e02536a3ace P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap starting.
I20260709 12:27:11.979616 30319 tablet_bootstrap.cc:492] T 62f8c9472b214576b9002e02536a3ace P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:27:11.980907 30319 tablet_bootstrap.cc:492] T 62f8c9472b214576b9002e02536a3ace P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap complete.
I20260709 12:27:11.982322 30319 ts_tablet_manager.cc:1403] T 62f8c9472b214576b9002e02536a3ace P 584dac86183a41d9b3bea1c2085e62fa: Time spent bootstrapping tablet: real 0.023s	user 0.017s	sys 0.004s
I20260709 12:27:11.985598 30319 raft_consensus.cc:359] T 62f8c9472b214576b9002e02536a3ace P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:11.986584 30319 raft_consensus.cc:740] T 62f8c9472b214576b9002e02536a3ace P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 584dac86183a41d9b3bea1c2085e62fa, State: Initialized, Role: FOLLOWER
I20260709 12:27:11.987465 30319 consensus_queue.cc:260] T 62f8c9472b214576b9002e02536a3ace P 584dac86183a41d9b3bea1c2085e62fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:11.989907 30319 ts_tablet_manager.cc:1434] T 62f8c9472b214576b9002e02536a3ace P 584dac86183a41d9b3bea1c2085e62fa: Time spent starting tablet: real 0.007s	user 0.005s	sys 0.000s
I20260709 12:27:11.990748 30319 tablet_bootstrap.cc:492] T 31a190e9f5ff407fab90502fb3197b70 P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap starting.
W20260709 12:27:11.991576 29560 consensus_peers.cc:597] T 71e2de235adf46d9a2407659080d42ef P d73d92fb35234d16824c55689b5fce28 -> Peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Couldn't send request to peer 584dac86183a41d9b3bea1c2085e62fa. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:11.997936 29708 consensus_peers.cc:597] T a46289e8fc824dd5a818e30df6e7ec3e P c6610c00ffb948ce9c34f41dec3d49e8 -> Peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Couldn't send request to peer 584dac86183a41d9b3bea1c2085e62fa. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:27:12.016608 30319 tablet_bootstrap.cc:492] T 31a190e9f5ff407fab90502fb3197b70 P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:27:12.017717 30319 tablet_bootstrap.cc:492] T 31a190e9f5ff407fab90502fb3197b70 P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap complete.
I20260709 12:27:12.018445 30319 ts_tablet_manager.cc:1403] T 31a190e9f5ff407fab90502fb3197b70 P 584dac86183a41d9b3bea1c2085e62fa: Time spent bootstrapping tablet: real 0.028s	user 0.017s	sys 0.004s
I20260709 12:27:12.021675 30319 raft_consensus.cc:359] T 31a190e9f5ff407fab90502fb3197b70 P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:27:12.022423 30319 raft_consensus.cc:740] T 31a190e9f5ff407fab90502fb3197b70 P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 584dac86183a41d9b3bea1c2085e62fa, State: Initialized, Role: FOLLOWER
I20260709 12:27:12.023216 30319 consensus_queue.cc:260] T 31a190e9f5ff407fab90502fb3197b70 P 584dac86183a41d9b3bea1c2085e62fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:27:12.025514 30319 ts_tablet_manager.cc:1434] T 31a190e9f5ff407fab90502fb3197b70 P 584dac86183a41d9b3bea1c2085e62fa: Time spent starting tablet: real 0.007s	user 0.006s	sys 0.001s
I20260709 12:27:12.026307 30319 tablet_bootstrap.cc:492] T ef20d7124e7a44aa97a219ee194f4b8b P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap starting.
I20260709 12:27:12.043179 29668 raft_consensus.cc:1275] T 2707271fd46f456aa765026642a3efa9 P d13c58ad8ac441c4a9b396eb04b1a476 [term 2 FOLLOWER]: Refusing update from remote peer d73d92fb35234d16824c55689b5fce28: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:27:12.045771 30430 consensus_queue.cc:1048] T 2707271fd46f456aa765026642a3efa9 P d73d92fb35234d16824c55689b5fce28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:27:12.050505 30319 tablet_bootstrap.cc:492] T ef20d7124e7a44aa97a219ee194f4b8b P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:27:12.051676 30319 tablet_bootstrap.cc:492] T ef20d7124e7a44aa97a219ee194f4b8b P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap complete.
I20260709 12:27:12.052409 30319 ts_tablet_manager.cc:1403] T ef20d7124e7a44aa97a219ee194f4b8b P 584dac86183a41d9b3bea1c2085e62fa: Time spent bootstrapping tablet: real 0.026s	user 0.019s	sys 0.005s
I20260709 12:27:12.055326 30319 raft_consensus.cc:359] T ef20d7124e7a44aa97a219ee194f4b8b P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:27:12.056543 30319 raft_consensus.cc:740] T ef20d7124e7a44aa97a219ee194f4b8b P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 584dac86183a41d9b3bea1c2085e62fa, State: Initialized, Role: FOLLOWER
I20260709 12:27:12.057365 30319 consensus_queue.cc:260] T ef20d7124e7a44aa97a219ee194f4b8b P 584dac86183a41d9b3bea1c2085e62fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:27:12.059921 30319 ts_tablet_manager.cc:1434] T ef20d7124e7a44aa97a219ee194f4b8b P 584dac86183a41d9b3bea1c2085e62fa: Time spent starting tablet: real 0.007s	user 0.005s	sys 0.000s
I20260709 12:27:12.060813 30319 tablet_bootstrap.cc:492] T cbf03426b6df43e5b2b9c56e2eb7687c P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap starting.
I20260709 12:27:12.066161 29666 raft_consensus.cc:1275] T 71e2de235adf46d9a2407659080d42ef P d13c58ad8ac441c4a9b396eb04b1a476 [term 2 FOLLOWER]: Refusing update from remote peer d73d92fb35234d16824c55689b5fce28: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:27:12.068073 30166 consensus_queue.cc:1048] T 71e2de235adf46d9a2407659080d42ef P d73d92fb35234d16824c55689b5fce28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:27:12.093919 30319 tablet_bootstrap.cc:492] T cbf03426b6df43e5b2b9c56e2eb7687c P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:27:12.095220 30319 tablet_bootstrap.cc:492] T cbf03426b6df43e5b2b9c56e2eb7687c P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap complete.
I20260709 12:27:12.095964 30319 ts_tablet_manager.cc:1403] T cbf03426b6df43e5b2b9c56e2eb7687c P 584dac86183a41d9b3bea1c2085e62fa: Time spent bootstrapping tablet: real 0.035s	user 0.018s	sys 0.009s
I20260709 12:27:12.098758 30319 raft_consensus.cc:359] T cbf03426b6df43e5b2b9c56e2eb7687c P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:27:12.100088 30319 raft_consensus.cc:740] T cbf03426b6df43e5b2b9c56e2eb7687c P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 584dac86183a41d9b3bea1c2085e62fa, State: Initialized, Role: FOLLOWER
I20260709 12:27:12.100960 30319 consensus_queue.cc:260] T cbf03426b6df43e5b2b9c56e2eb7687c P 584dac86183a41d9b3bea1c2085e62fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:27:12.104889 30319 ts_tablet_manager.cc:1434] T cbf03426b6df43e5b2b9c56e2eb7687c P 584dac86183a41d9b3bea1c2085e62fa: Time spent starting tablet: real 0.009s	user 0.005s	sys 0.001s
I20260709 12:27:12.105881 30319 tablet_bootstrap.cc:492] T fcb6a7f720b14dc98e821f663357a0bb P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap starting.
I20260709 12:27:12.132165 30319 tablet_bootstrap.cc:492] T fcb6a7f720b14dc98e821f663357a0bb P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:27:12.133200 30319 tablet_bootstrap.cc:492] T fcb6a7f720b14dc98e821f663357a0bb P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap complete.
I20260709 12:27:12.134157 30319 ts_tablet_manager.cc:1403] T fcb6a7f720b14dc98e821f663357a0bb P 584dac86183a41d9b3bea1c2085e62fa: Time spent bootstrapping tablet: real 0.029s	user 0.020s	sys 0.005s
I20260709 12:27:12.137008 30319 raft_consensus.cc:359] T fcb6a7f720b14dc98e821f663357a0bb P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:27:12.137775 30319 raft_consensus.cc:740] T fcb6a7f720b14dc98e821f663357a0bb P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 584dac86183a41d9b3bea1c2085e62fa, State: Initialized, Role: FOLLOWER
I20260709 12:27:12.138881 30319 consensus_queue.cc:260] T fcb6a7f720b14dc98e821f663357a0bb P 584dac86183a41d9b3bea1c2085e62fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:27:12.141300 30319 ts_tablet_manager.cc:1434] T fcb6a7f720b14dc98e821f663357a0bb P 584dac86183a41d9b3bea1c2085e62fa: Time spent starting tablet: real 0.007s	user 0.005s	sys 0.000s
I20260709 12:27:12.142222 30319 tablet_bootstrap.cc:492] T b3c02c910d2b43cc9265bb1d98491cd3 P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap starting.
I20260709 12:27:12.187067 30319 tablet_bootstrap.cc:492] T b3c02c910d2b43cc9265bb1d98491cd3 P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:27:12.187934 30319 tablet_bootstrap.cc:492] T b3c02c910d2b43cc9265bb1d98491cd3 P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap complete.
I20260709 12:27:12.188657 30319 ts_tablet_manager.cc:1403] T b3c02c910d2b43cc9265bb1d98491cd3 P 584dac86183a41d9b3bea1c2085e62fa: Time spent bootstrapping tablet: real 0.047s	user 0.018s	sys 0.005s
I20260709 12:27:12.195940 30319 raft_consensus.cc:359] T b3c02c910d2b43cc9265bb1d98491cd3 P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:12.196866 30319 raft_consensus.cc:740] T b3c02c910d2b43cc9265bb1d98491cd3 P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 584dac86183a41d9b3bea1c2085e62fa, State: Initialized, Role: FOLLOWER
I20260709 12:27:12.197590 30319 consensus_queue.cc:260] T b3c02c910d2b43cc9265bb1d98491cd3 P 584dac86183a41d9b3bea1c2085e62fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:12.207755 30319 ts_tablet_manager.cc:1434] T b3c02c910d2b43cc9265bb1d98491cd3 P 584dac86183a41d9b3bea1c2085e62fa: Time spent starting tablet: real 0.019s	user 0.005s	sys 0.000s
I20260709 12:27:12.208968 30319 tablet_bootstrap.cc:492] T f3e17f0a8fc14095850e8138fd79b86e P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap starting.
I20260709 12:27:12.242944 30319 tablet_bootstrap.cc:492] T f3e17f0a8fc14095850e8138fd79b86e P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:27:12.244134 30319 tablet_bootstrap.cc:492] T f3e17f0a8fc14095850e8138fd79b86e P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap complete.
I20260709 12:27:12.244884 30319 ts_tablet_manager.cc:1403] T f3e17f0a8fc14095850e8138fd79b86e P 584dac86183a41d9b3bea1c2085e62fa: Time spent bootstrapping tablet: real 0.036s	user 0.024s	sys 0.003s
I20260709 12:27:12.247742 30319 raft_consensus.cc:359] T f3e17f0a8fc14095850e8138fd79b86e P 584dac86183a41d9b3bea1c2085e62fa [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:27:12.248524 30319 raft_consensus.cc:740] T f3e17f0a8fc14095850e8138fd79b86e P 584dac86183a41d9b3bea1c2085e62fa [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 584dac86183a41d9b3bea1c2085e62fa, State: Initialized, Role: FOLLOWER
I20260709 12:27:12.249315 30319 consensus_queue.cc:260] T f3e17f0a8fc14095850e8138fd79b86e P 584dac86183a41d9b3bea1c2085e62fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:27:12.252076 30319 ts_tablet_manager.cc:1434] T f3e17f0a8fc14095850e8138fd79b86e P 584dac86183a41d9b3bea1c2085e62fa: Time spent starting tablet: real 0.007s	user 0.008s	sys 0.000s
I20260709 12:27:12.253360 30319 tablet_bootstrap.cc:492] T f8c01e05981747a395e5483b551297dd P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap starting.
I20260709 12:27:12.270273 30319 tablet_bootstrap.cc:492] T f8c01e05981747a395e5483b551297dd P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:27:12.271054 30319 tablet_bootstrap.cc:492] T f8c01e05981747a395e5483b551297dd P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap complete.
I20260709 12:27:12.271622 30319 ts_tablet_manager.cc:1403] T f8c01e05981747a395e5483b551297dd P 584dac86183a41d9b3bea1c2085e62fa: Time spent bootstrapping tablet: real 0.018s	user 0.010s	sys 0.004s
I20260709 12:27:12.273699 30319 raft_consensus.cc:359] T f8c01e05981747a395e5483b551297dd P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:12.274302 30319 raft_consensus.cc:740] T f8c01e05981747a395e5483b551297dd P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 584dac86183a41d9b3bea1c2085e62fa, State: Initialized, Role: FOLLOWER
I20260709 12:27:12.275039 30319 consensus_queue.cc:260] T f8c01e05981747a395e5483b551297dd P 584dac86183a41d9b3bea1c2085e62fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:12.277148 30319 ts_tablet_manager.cc:1434] T f8c01e05981747a395e5483b551297dd P 584dac86183a41d9b3bea1c2085e62fa: Time spent starting tablet: real 0.005s	user 0.005s	sys 0.003s
I20260709 12:27:12.280519 30319 tablet_bootstrap.cc:492] T d1918297b9eb41f5847a211e668ae748 P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap starting.
I20260709 12:27:12.307657 30319 tablet_bootstrap.cc:492] T d1918297b9eb41f5847a211e668ae748 P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:27:12.308450 30319 tablet_bootstrap.cc:492] T d1918297b9eb41f5847a211e668ae748 P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap complete.
I20260709 12:27:12.309238 30319 ts_tablet_manager.cc:1403] T d1918297b9eb41f5847a211e668ae748 P 584dac86183a41d9b3bea1c2085e62fa: Time spent bootstrapping tablet: real 0.029s	user 0.013s	sys 0.011s
I20260709 12:27:12.312086 30319 raft_consensus.cc:359] T d1918297b9eb41f5847a211e668ae748 P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:27:12.312891 30319 raft_consensus.cc:740] T d1918297b9eb41f5847a211e668ae748 P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 584dac86183a41d9b3bea1c2085e62fa, State: Initialized, Role: FOLLOWER
I20260709 12:27:12.313771 30319 consensus_queue.cc:260] T d1918297b9eb41f5847a211e668ae748 P 584dac86183a41d9b3bea1c2085e62fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:27:12.315667 30319 ts_tablet_manager.cc:1434] T d1918297b9eb41f5847a211e668ae748 P 584dac86183a41d9b3bea1c2085e62fa: Time spent starting tablet: real 0.006s	user 0.005s	sys 0.000s
I20260709 12:27:12.317193 30319 tablet_bootstrap.cc:492] T 5ee924f859f54f64908499bfac48ee62 P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap starting.
I20260709 12:27:12.336323 30319 tablet_bootstrap.cc:492] T 5ee924f859f54f64908499bfac48ee62 P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:27:12.337848 30319 tablet_bootstrap.cc:492] T 5ee924f859f54f64908499bfac48ee62 P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap complete.
I20260709 12:27:12.338528 30319 ts_tablet_manager.cc:1403] T 5ee924f859f54f64908499bfac48ee62 P 584dac86183a41d9b3bea1c2085e62fa: Time spent bootstrapping tablet: real 0.022s	user 0.018s	sys 0.000s
I20260709 12:27:12.342031 30319 raft_consensus.cc:359] T 5ee924f859f54f64908499bfac48ee62 P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:27:12.342747 30319 raft_consensus.cc:740] T 5ee924f859f54f64908499bfac48ee62 P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 584dac86183a41d9b3bea1c2085e62fa, State: Initialized, Role: FOLLOWER
I20260709 12:27:12.343444 30319 consensus_queue.cc:260] T 5ee924f859f54f64908499bfac48ee62 P 584dac86183a41d9b3bea1c2085e62fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:27:12.346155 30319 ts_tablet_manager.cc:1434] T 5ee924f859f54f64908499bfac48ee62 P 584dac86183a41d9b3bea1c2085e62fa: Time spent starting tablet: real 0.007s	user 0.007s	sys 0.000s
I20260709 12:27:12.346967 30319 tablet_bootstrap.cc:492] T 0c428ff656ae4b9d99d5b5073f98725d P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap starting.
W20260709 12:27:12.354741 29560 consensus_peers.cc:597] T 01788883c8ea42989f246a3486111a7e P d73d92fb35234d16824c55689b5fce28 -> Peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Couldn't send request to peer 584dac86183a41d9b3bea1c2085e62fa. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:27:12.367722 30319 tablet_bootstrap.cc:492] T 0c428ff656ae4b9d99d5b5073f98725d P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:27:12.368685 30319 tablet_bootstrap.cc:492] T 0c428ff656ae4b9d99d5b5073f98725d P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap complete.
I20260709 12:27:12.369232 30319 ts_tablet_manager.cc:1403] T 0c428ff656ae4b9d99d5b5073f98725d P 584dac86183a41d9b3bea1c2085e62fa: Time spent bootstrapping tablet: real 0.022s	user 0.016s	sys 0.004s
I20260709 12:27:12.371228 30319 raft_consensus.cc:359] T 0c428ff656ae4b9d99d5b5073f98725d P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:27:12.371809 30319 raft_consensus.cc:740] T 0c428ff656ae4b9d99d5b5073f98725d P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 584dac86183a41d9b3bea1c2085e62fa, State: Initialized, Role: FOLLOWER
I20260709 12:27:12.372427 30319 consensus_queue.cc:260] T 0c428ff656ae4b9d99d5b5073f98725d P 584dac86183a41d9b3bea1c2085e62fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:27:12.374025 30319 ts_tablet_manager.cc:1434] T 0c428ff656ae4b9d99d5b5073f98725d P 584dac86183a41d9b3bea1c2085e62fa: Time spent starting tablet: real 0.005s	user 0.004s	sys 0.000s
I20260709 12:27:12.375217 30319 tablet_bootstrap.cc:492] T a46289e8fc824dd5a818e30df6e7ec3e P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap starting.
I20260709 12:27:12.388639 30319 tablet_bootstrap.cc:492] T a46289e8fc824dd5a818e30df6e7ec3e P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:27:12.389629 30319 tablet_bootstrap.cc:492] T a46289e8fc824dd5a818e30df6e7ec3e P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap complete.
I20260709 12:27:12.390197 30319 ts_tablet_manager.cc:1403] T a46289e8fc824dd5a818e30df6e7ec3e P 584dac86183a41d9b3bea1c2085e62fa: Time spent bootstrapping tablet: real 0.015s	user 0.012s	sys 0.000s
I20260709 12:27:12.392102 30319 raft_consensus.cc:359] T a46289e8fc824dd5a818e30df6e7ec3e P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:27:12.392593 30319 raft_consensus.cc:740] T a46289e8fc824dd5a818e30df6e7ec3e P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 584dac86183a41d9b3bea1c2085e62fa, State: Initialized, Role: FOLLOWER
I20260709 12:27:12.393172 30319 consensus_queue.cc:260] T a46289e8fc824dd5a818e30df6e7ec3e P 584dac86183a41d9b3bea1c2085e62fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:27:12.394784 30319 ts_tablet_manager.cc:1434] T a46289e8fc824dd5a818e30df6e7ec3e P 584dac86183a41d9b3bea1c2085e62fa: Time spent starting tablet: real 0.004s	user 0.004s	sys 0.000s
I20260709 12:27:12.395540 30319 tablet_bootstrap.cc:492] T 3e9451c3797c4496a66dfdb3e9fb5e38 P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap starting.
I20260709 12:27:12.418227 30319 tablet_bootstrap.cc:492] T 3e9451c3797c4496a66dfdb3e9fb5e38 P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:27:12.419312 30319 tablet_bootstrap.cc:492] T 3e9451c3797c4496a66dfdb3e9fb5e38 P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap complete.
I20260709 12:27:12.420075 30319 ts_tablet_manager.cc:1403] T 3e9451c3797c4496a66dfdb3e9fb5e38 P 584dac86183a41d9b3bea1c2085e62fa: Time spent bootstrapping tablet: real 0.025s	user 0.018s	sys 0.004s
I20260709 12:27:12.422789 30319 raft_consensus.cc:359] T 3e9451c3797c4496a66dfdb3e9fb5e38 P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:27:12.423336 30319 raft_consensus.cc:740] T 3e9451c3797c4496a66dfdb3e9fb5e38 P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 584dac86183a41d9b3bea1c2085e62fa, State: Initialized, Role: FOLLOWER
I20260709 12:27:12.424137 30359 raft_consensus.cc:3060] T 62f8c9472b214576b9002e02536a3ace P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:12.424137 30319 consensus_queue.cc:260] T 3e9451c3797c4496a66dfdb3e9fb5e38 P 584dac86183a41d9b3bea1c2085e62fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:27:12.427044 30319 ts_tablet_manager.cc:1434] T 3e9451c3797c4496a66dfdb3e9fb5e38 P 584dac86183a41d9b3bea1c2085e62fa: Time spent starting tablet: real 0.006s	user 0.005s	sys 0.000s
I20260709 12:27:12.428057 30319 tablet_bootstrap.cc:492] T 3615115233c34461b85763151751e475 P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap starting.
I20260709 12:27:12.449433 30319 tablet_bootstrap.cc:492] T 3615115233c34461b85763151751e475 P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:27:12.450320 30319 tablet_bootstrap.cc:492] T 3615115233c34461b85763151751e475 P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap complete.
I20260709 12:27:12.450961 30319 ts_tablet_manager.cc:1403] T 3615115233c34461b85763151751e475 P 584dac86183a41d9b3bea1c2085e62fa: Time spent bootstrapping tablet: real 0.023s	user 0.016s	sys 0.003s
I20260709 12:27:12.454208 30319 raft_consensus.cc:359] T 3615115233c34461b85763151751e475 P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:27:12.454931 30319 raft_consensus.cc:740] T 3615115233c34461b85763151751e475 P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 584dac86183a41d9b3bea1c2085e62fa, State: Initialized, Role: FOLLOWER
I20260709 12:27:12.455744 30319 consensus_queue.cc:260] T 3615115233c34461b85763151751e475 P 584dac86183a41d9b3bea1c2085e62fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:27:12.458108 30319 ts_tablet_manager.cc:1434] T 3615115233c34461b85763151751e475 P 584dac86183a41d9b3bea1c2085e62fa: Time spent starting tablet: real 0.007s	user 0.008s	sys 0.000s
I20260709 12:27:12.458935 30319 tablet_bootstrap.cc:492] T 16649fd17d434e3c95e9d585d2404659 P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap starting.
I20260709 12:27:12.532363 30319 tablet_bootstrap.cc:492] T 16649fd17d434e3c95e9d585d2404659 P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:27:12.533290 30319 tablet_bootstrap.cc:492] T 16649fd17d434e3c95e9d585d2404659 P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap complete.
I20260709 12:27:12.533941 30319 ts_tablet_manager.cc:1403] T 16649fd17d434e3c95e9d585d2404659 P 584dac86183a41d9b3bea1c2085e62fa: Time spent bootstrapping tablet: real 0.075s	user 0.033s	sys 0.033s
I20260709 12:27:12.536932 30319 raft_consensus.cc:359] T 16649fd17d434e3c95e9d585d2404659 P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:27:12.537602 30319 raft_consensus.cc:740] T 16649fd17d434e3c95e9d585d2404659 P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 584dac86183a41d9b3bea1c2085e62fa, State: Initialized, Role: FOLLOWER
I20260709 12:27:12.538486 30319 consensus_queue.cc:260] T 16649fd17d434e3c95e9d585d2404659 P 584dac86183a41d9b3bea1c2085e62fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:27:12.540674 30319 ts_tablet_manager.cc:1434] T 16649fd17d434e3c95e9d585d2404659 P 584dac86183a41d9b3bea1c2085e62fa: Time spent starting tablet: real 0.006s	user 0.005s	sys 0.000s
I20260709 12:27:12.541461 30319 tablet_bootstrap.cc:492] T 1d4e0ad0462e4f74a44155aa3ffaf87f P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap starting.
I20260709 12:27:12.558996 30366 raft_consensus.cc:3060] T 16649fd17d434e3c95e9d585d2404659 P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:12.566547 30319 tablet_bootstrap.cc:492] T 1d4e0ad0462e4f74a44155aa3ffaf87f P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:27:12.567592 30319 tablet_bootstrap.cc:492] T 1d4e0ad0462e4f74a44155aa3ffaf87f P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap complete.
I20260709 12:27:12.568300 30319 ts_tablet_manager.cc:1403] T 1d4e0ad0462e4f74a44155aa3ffaf87f P 584dac86183a41d9b3bea1c2085e62fa: Time spent bootstrapping tablet: real 0.027s	user 0.016s	sys 0.008s
I20260709 12:27:12.592141 30319 raft_consensus.cc:359] T 1d4e0ad0462e4f74a44155aa3ffaf87f P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:12.592861 30319 raft_consensus.cc:740] T 1d4e0ad0462e4f74a44155aa3ffaf87f P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 584dac86183a41d9b3bea1c2085e62fa, State: Initialized, Role: FOLLOWER
I20260709 12:27:12.593521 30319 consensus_queue.cc:260] T 1d4e0ad0462e4f74a44155aa3ffaf87f P 584dac86183a41d9b3bea1c2085e62fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:12.597642 30319 ts_tablet_manager.cc:1434] T 1d4e0ad0462e4f74a44155aa3ffaf87f P 584dac86183a41d9b3bea1c2085e62fa: Time spent starting tablet: real 0.029s	user 0.007s	sys 0.000s
I20260709 12:27:12.597738 30363 raft_consensus.cc:3060] T fcb6a7f720b14dc98e821f663357a0bb P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:12.598510 30319 tablet_bootstrap.cc:492] T 6e37d72807e547898f4fce9e8b82b2b0 P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap starting.
I20260709 12:27:12.650965 30319 tablet_bootstrap.cc:492] T 6e37d72807e547898f4fce9e8b82b2b0 P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:27:12.652288 30319 tablet_bootstrap.cc:492] T 6e37d72807e547898f4fce9e8b82b2b0 P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap complete.
I20260709 12:27:12.653307 30319 ts_tablet_manager.cc:1403] T 6e37d72807e547898f4fce9e8b82b2b0 P 584dac86183a41d9b3bea1c2085e62fa: Time spent bootstrapping tablet: real 0.055s	user 0.028s	sys 0.023s
I20260709 12:27:12.655800 30319 raft_consensus.cc:359] T 6e37d72807e547898f4fce9e8b82b2b0 P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:27:12.656634 30319 raft_consensus.cc:740] T 6e37d72807e547898f4fce9e8b82b2b0 P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 584dac86183a41d9b3bea1c2085e62fa, State: Initialized, Role: FOLLOWER
I20260709 12:27:12.657344 30319 consensus_queue.cc:260] T 6e37d72807e547898f4fce9e8b82b2b0 P 584dac86183a41d9b3bea1c2085e62fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:27:12.659003 30319 ts_tablet_manager.cc:1434] T 6e37d72807e547898f4fce9e8b82b2b0 P 584dac86183a41d9b3bea1c2085e62fa: Time spent starting tablet: real 0.005s	user 0.004s	sys 0.000s
I20260709 12:27:12.659687 30319 tablet_bootstrap.cc:492] T 425777eed34441cd93e0902bdc126516 P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap starting.
I20260709 12:27:12.672539 30319 tablet_bootstrap.cc:492] T 425777eed34441cd93e0902bdc126516 P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:27:12.673375 30319 tablet_bootstrap.cc:492] T 425777eed34441cd93e0902bdc126516 P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap complete.
I20260709 12:27:12.673923 30319 ts_tablet_manager.cc:1403] T 425777eed34441cd93e0902bdc126516 P 584dac86183a41d9b3bea1c2085e62fa: Time spent bootstrapping tablet: real 0.014s	user 0.010s	sys 0.002s
I20260709 12:27:12.676126 30319 raft_consensus.cc:359] T 425777eed34441cd93e0902bdc126516 P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:27:12.676743 30319 raft_consensus.cc:740] T 425777eed34441cd93e0902bdc126516 P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 584dac86183a41d9b3bea1c2085e62fa, State: Initialized, Role: FOLLOWER
I20260709 12:27:12.677460 30319 consensus_queue.cc:260] T 425777eed34441cd93e0902bdc126516 P 584dac86183a41d9b3bea1c2085e62fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:27:12.679463 30319 ts_tablet_manager.cc:1434] T 425777eed34441cd93e0902bdc126516 P 584dac86183a41d9b3bea1c2085e62fa: Time spent starting tablet: real 0.005s	user 0.003s	sys 0.001s
I20260709 12:27:12.680812 30319 tablet_bootstrap.cc:492] T 4685c2e824c14fc1bcd45c16169798cc P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap starting.
I20260709 12:27:12.696700 30319 tablet_bootstrap.cc:492] T 4685c2e824c14fc1bcd45c16169798cc P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:27:12.697625 30319 tablet_bootstrap.cc:492] T 4685c2e824c14fc1bcd45c16169798cc P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap complete.
I20260709 12:27:12.698275 30319 ts_tablet_manager.cc:1403] T 4685c2e824c14fc1bcd45c16169798cc P 584dac86183a41d9b3bea1c2085e62fa: Time spent bootstrapping tablet: real 0.018s	user 0.010s	sys 0.004s
I20260709 12:27:12.700719 30319 raft_consensus.cc:359] T 4685c2e824c14fc1bcd45c16169798cc P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:12.701382 30319 raft_consensus.cc:740] T 4685c2e824c14fc1bcd45c16169798cc P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 584dac86183a41d9b3bea1c2085e62fa, State: Initialized, Role: FOLLOWER
I20260709 12:27:12.702020 30319 consensus_queue.cc:260] T 4685c2e824c14fc1bcd45c16169798cc P 584dac86183a41d9b3bea1c2085e62fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:12.704145 30319 ts_tablet_manager.cc:1434] T 4685c2e824c14fc1bcd45c16169798cc P 584dac86183a41d9b3bea1c2085e62fa: Time spent starting tablet: real 0.006s	user 0.008s	sys 0.000s
I20260709 12:27:12.704933 30319 tablet_bootstrap.cc:492] T f76101eb30f14b98a9aa8c41e0d6f831 P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap starting.
I20260709 12:27:12.719956 30319 tablet_bootstrap.cc:492] T f76101eb30f14b98a9aa8c41e0d6f831 P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:27:12.721195 30319 tablet_bootstrap.cc:492] T f76101eb30f14b98a9aa8c41e0d6f831 P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap complete.
I20260709 12:27:12.721849 30319 ts_tablet_manager.cc:1403] T f76101eb30f14b98a9aa8c41e0d6f831 P 584dac86183a41d9b3bea1c2085e62fa: Time spent bootstrapping tablet: real 0.017s	user 0.010s	sys 0.005s
I20260709 12:27:12.724380 30319 raft_consensus.cc:359] T f76101eb30f14b98a9aa8c41e0d6f831 P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:27:12.724951 30319 raft_consensus.cc:740] T f76101eb30f14b98a9aa8c41e0d6f831 P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 584dac86183a41d9b3bea1c2085e62fa, State: Initialized, Role: FOLLOWER
I20260709 12:27:12.725718 30319 consensus_queue.cc:260] T f76101eb30f14b98a9aa8c41e0d6f831 P 584dac86183a41d9b3bea1c2085e62fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:27:12.727795 30319 ts_tablet_manager.cc:1434] T f76101eb30f14b98a9aa8c41e0d6f831 P 584dac86183a41d9b3bea1c2085e62fa: Time spent starting tablet: real 0.006s	user 0.003s	sys 0.000s
I20260709 12:27:12.728551 30319 tablet_bootstrap.cc:492] T 01788883c8ea42989f246a3486111a7e P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap starting.
I20260709 12:27:12.744966 30319 tablet_bootstrap.cc:492] T 01788883c8ea42989f246a3486111a7e P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:27:12.745767 30319 tablet_bootstrap.cc:492] T 01788883c8ea42989f246a3486111a7e P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap complete.
I20260709 12:27:12.746379 30319 ts_tablet_manager.cc:1403] T 01788883c8ea42989f246a3486111a7e P 584dac86183a41d9b3bea1c2085e62fa: Time spent bootstrapping tablet: real 0.018s	user 0.014s	sys 0.000s
I20260709 12:27:12.748731 30319 raft_consensus.cc:359] T 01788883c8ea42989f246a3486111a7e P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:12.749219 30319 raft_consensus.cc:740] T 01788883c8ea42989f246a3486111a7e P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 584dac86183a41d9b3bea1c2085e62fa, State: Initialized, Role: FOLLOWER
I20260709 12:27:12.749835 30319 consensus_queue.cc:260] T 01788883c8ea42989f246a3486111a7e P 584dac86183a41d9b3bea1c2085e62fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:12.751412 30319 ts_tablet_manager.cc:1434] T 01788883c8ea42989f246a3486111a7e P 584dac86183a41d9b3bea1c2085e62fa: Time spent starting tablet: real 0.005s	user 0.004s	sys 0.000s
I20260709 12:27:12.752610 30319 tablet_bootstrap.cc:492] T 2707271fd46f456aa765026642a3efa9 P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap starting.
I20260709 12:27:12.772315 30319 tablet_bootstrap.cc:492] T 2707271fd46f456aa765026642a3efa9 P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:27:12.773103 30319 tablet_bootstrap.cc:492] T 2707271fd46f456aa765026642a3efa9 P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap complete.
I20260709 12:27:12.773765 30319 ts_tablet_manager.cc:1403] T 2707271fd46f456aa765026642a3efa9 P 584dac86183a41d9b3bea1c2085e62fa: Time spent bootstrapping tablet: real 0.021s	user 0.012s	sys 0.005s
I20260709 12:27:12.776084 30319 raft_consensus.cc:359] T 2707271fd46f456aa765026642a3efa9 P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:27:12.776744 30319 raft_consensus.cc:740] T 2707271fd46f456aa765026642a3efa9 P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 584dac86183a41d9b3bea1c2085e62fa, State: Initialized, Role: FOLLOWER
I20260709 12:27:12.777351 30319 consensus_queue.cc:260] T 2707271fd46f456aa765026642a3efa9 P 584dac86183a41d9b3bea1c2085e62fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:27:12.778893 30319 ts_tablet_manager.cc:1434] T 2707271fd46f456aa765026642a3efa9 P 584dac86183a41d9b3bea1c2085e62fa: Time spent starting tablet: real 0.005s	user 0.001s	sys 0.003s
I20260709 12:27:12.779626 30319 tablet_bootstrap.cc:492] T 742beffede4f407d8a72909d2b147e6d P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap starting.
I20260709 12:27:12.799697 30319 tablet_bootstrap.cc:492] T 742beffede4f407d8a72909d2b147e6d P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:27:12.800681 30319 tablet_bootstrap.cc:492] T 742beffede4f407d8a72909d2b147e6d P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap complete.
I20260709 12:27:12.801298 30319 ts_tablet_manager.cc:1403] T 742beffede4f407d8a72909d2b147e6d P 584dac86183a41d9b3bea1c2085e62fa: Time spent bootstrapping tablet: real 0.022s	user 0.015s	sys 0.001s
I20260709 12:27:12.803201 30319 raft_consensus.cc:359] T 742beffede4f407d8a72909d2b147e6d P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:27:12.803992 30319 raft_consensus.cc:740] T 742beffede4f407d8a72909d2b147e6d P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 584dac86183a41d9b3bea1c2085e62fa, State: Initialized, Role: FOLLOWER
I20260709 12:27:12.804819 30319 consensus_queue.cc:260] T 742beffede4f407d8a72909d2b147e6d P 584dac86183a41d9b3bea1c2085e62fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:27:12.806598 30319 ts_tablet_manager.cc:1434] T 742beffede4f407d8a72909d2b147e6d P 584dac86183a41d9b3bea1c2085e62fa: Time spent starting tablet: real 0.005s	user 0.005s	sys 0.000s
I20260709 12:27:12.807379 30319 tablet_bootstrap.cc:492] T 9633e3b1d78f403097e4a00b7d4b869b P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap starting.
I20260709 12:27:12.824242 30319 tablet_bootstrap.cc:492] T 9633e3b1d78f403097e4a00b7d4b869b P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:27:12.825273 30319 tablet_bootstrap.cc:492] T 9633e3b1d78f403097e4a00b7d4b869b P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap complete.
I20260709 12:27:12.825961 30319 ts_tablet_manager.cc:1403] T 9633e3b1d78f403097e4a00b7d4b869b P 584dac86183a41d9b3bea1c2085e62fa: Time spent bootstrapping tablet: real 0.019s	user 0.013s	sys 0.003s
I20260709 12:27:12.828449 30319 raft_consensus.cc:359] T 9633e3b1d78f403097e4a00b7d4b869b P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:27:12.828989 30319 raft_consensus.cc:740] T 9633e3b1d78f403097e4a00b7d4b869b P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 584dac86183a41d9b3bea1c2085e62fa, State: Initialized, Role: FOLLOWER
I20260709 12:27:12.829543 30319 consensus_queue.cc:260] T 9633e3b1d78f403097e4a00b7d4b869b P 584dac86183a41d9b3bea1c2085e62fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:27:12.831079 30319 ts_tablet_manager.cc:1434] T 9633e3b1d78f403097e4a00b7d4b869b P 584dac86183a41d9b3bea1c2085e62fa: Time spent starting tablet: real 0.005s	user 0.004s	sys 0.000s
I20260709 12:27:12.832370 30319 tablet_bootstrap.cc:492] T e67a2affef634fd8bc96a64fa83cffdb P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap starting.
I20260709 12:27:12.849933 30319 tablet_bootstrap.cc:492] T e67a2affef634fd8bc96a64fa83cffdb P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:27:12.850751 30319 tablet_bootstrap.cc:492] T e67a2affef634fd8bc96a64fa83cffdb P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap complete.
I20260709 12:27:12.851362 30319 ts_tablet_manager.cc:1403] T e67a2affef634fd8bc96a64fa83cffdb P 584dac86183a41d9b3bea1c2085e62fa: Time spent bootstrapping tablet: real 0.019s	user 0.015s	sys 0.000s
I20260709 12:27:12.853603 30319 raft_consensus.cc:359] T e67a2affef634fd8bc96a64fa83cffdb P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:27:12.854127 30319 raft_consensus.cc:740] T e67a2affef634fd8bc96a64fa83cffdb P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 584dac86183a41d9b3bea1c2085e62fa, State: Initialized, Role: FOLLOWER
I20260709 12:27:12.854861 30319 consensus_queue.cc:260] T e67a2affef634fd8bc96a64fa83cffdb P 584dac86183a41d9b3bea1c2085e62fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:27:12.856969 30319 ts_tablet_manager.cc:1434] T e67a2affef634fd8bc96a64fa83cffdb P 584dac86183a41d9b3bea1c2085e62fa: Time spent starting tablet: real 0.005s	user 0.008s	sys 0.000s
I20260709 12:27:12.858712 30319 tablet_bootstrap.cc:492] T 71e2de235adf46d9a2407659080d42ef P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap starting.
I20260709 12:27:12.874512 30319 tablet_bootstrap.cc:492] T 71e2de235adf46d9a2407659080d42ef P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:27:12.875560 30319 tablet_bootstrap.cc:492] T 71e2de235adf46d9a2407659080d42ef P 584dac86183a41d9b3bea1c2085e62fa: Bootstrap complete.
I20260709 12:27:12.876183 30319 ts_tablet_manager.cc:1403] T 71e2de235adf46d9a2407659080d42ef P 584dac86183a41d9b3bea1c2085e62fa: Time spent bootstrapping tablet: real 0.018s	user 0.017s	sys 0.000s
I20260709 12:27:12.878417 30319 raft_consensus.cc:359] T 71e2de235adf46d9a2407659080d42ef P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:27:12.878937 30319 raft_consensus.cc:740] T 71e2de235adf46d9a2407659080d42ef P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 584dac86183a41d9b3bea1c2085e62fa, State: Initialized, Role: FOLLOWER
I20260709 12:27:12.879583 30319 consensus_queue.cc:260] T 71e2de235adf46d9a2407659080d42ef P 584dac86183a41d9b3bea1c2085e62fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:27:12.881148 30319 ts_tablet_manager.cc:1434] T 71e2de235adf46d9a2407659080d42ef P 584dac86183a41d9b3bea1c2085e62fa: Time spent starting tablet: real 0.005s	user 0.004s	sys 0.000s
I20260709 12:27:12.967113 30366 raft_consensus.cc:3060] T 2707271fd46f456aa765026642a3efa9 P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:13.076900 30366 raft_consensus.cc:3060] T 425777eed34441cd93e0902bdc126516 P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:13.158154 30363 raft_consensus.cc:3060] T 71e2de235adf46d9a2407659080d42ef P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:13.255239 30366 raft_consensus.cc:3060] T 742beffede4f407d8a72909d2b147e6d P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:13.275151 30366 raft_consensus.cc:3060] T e67a2affef634fd8bc96a64fa83cffdb P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:20.758028 16501 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:27:20.852960 29591 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "1d1c1f58c4d340ac951dfa403d34138a"
dest_uuid: "d73d92fb35234d16824c55689b5fce28"
mode: GRACEFUL
new_leader_uuid: "584dac86183a41d9b3bea1c2085e62fa"
 from {username='slave'} at 127.0.0.1:43236
I20260709 12:27:20.853826 29591 raft_consensus.cc:606] T 1d1c1f58c4d340ac951dfa403d34138a P d73d92fb35234d16824c55689b5fce28 [term 1 LEADER]: Received request to transfer leadership to TS 584dac86183a41d9b3bea1c2085e62fa
I20260709 12:27:20.872215 29743 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "3f41ce441fb047c8bbb741b1b9cbc0ec"
dest_uuid: "c6610c00ffb948ce9c34f41dec3d49e8"
mode: GRACEFUL
new_leader_uuid: "584dac86183a41d9b3bea1c2085e62fa"
 from {username='slave'} at 127.0.0.1:58802
I20260709 12:27:20.872766 29743 raft_consensus.cc:606] T 3f41ce441fb047c8bbb741b1b9cbc0ec P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 LEADER]: Received request to transfer leadership to TS 584dac86183a41d9b3bea1c2085e62fa
I20260709 12:27:20.875658 29743 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "592801934cad4789b9ce8dc09f1470bd"
dest_uuid: "c6610c00ffb948ce9c34f41dec3d49e8"
mode: GRACEFUL
new_leader_uuid: "584dac86183a41d9b3bea1c2085e62fa"
 from {username='slave'} at 127.0.0.1:58802
I20260709 12:27:20.876283 29743 raft_consensus.cc:606] T 592801934cad4789b9ce8dc09f1470bd P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 LEADER]: Received request to transfer leadership to TS 584dac86183a41d9b3bea1c2085e62fa
I20260709 12:27:20.879050 29743 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "6a30685caefb4d0e80fa4ea87d70760c"
dest_uuid: "c6610c00ffb948ce9c34f41dec3d49e8"
mode: GRACEFUL
new_leader_uuid: "584dac86183a41d9b3bea1c2085e62fa"
 from {username='slave'} at 127.0.0.1:58802
I20260709 12:27:20.879523 29743 raft_consensus.cc:606] T 6a30685caefb4d0e80fa4ea87d70760c P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 LEADER]: Received request to transfer leadership to TS 584dac86183a41d9b3bea1c2085e62fa
I20260709 12:27:20.882572 29743 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "9633e3b1d78f403097e4a00b7d4b869b"
dest_uuid: "c6610c00ffb948ce9c34f41dec3d49e8"
mode: GRACEFUL
new_leader_uuid: "584dac86183a41d9b3bea1c2085e62fa"
 from {username='slave'} at 127.0.0.1:58802
I20260709 12:27:20.883225 29743 raft_consensus.cc:606] T 9633e3b1d78f403097e4a00b7d4b869b P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 LEADER]: Received request to transfer leadership to TS 584dac86183a41d9b3bea1c2085e62fa
I20260709 12:27:20.897274 29668 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "f76101eb30f14b98a9aa8c41e0d6f831"
dest_uuid: "d13c58ad8ac441c4a9b396eb04b1a476"
mode: GRACEFUL
new_leader_uuid: "584dac86183a41d9b3bea1c2085e62fa"
 from {username='slave'} at 127.0.0.1:46668
I20260709 12:27:20.897743 29668 raft_consensus.cc:606] T f76101eb30f14b98a9aa8c41e0d6f831 P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 LEADER]: Received request to transfer leadership to TS 584dac86183a41d9b3bea1c2085e62fa
I20260709 12:27:20.900722 29743 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "fb6e78ccc01c4707af34cddf81619eab"
dest_uuid: "c6610c00ffb948ce9c34f41dec3d49e8"
mode: GRACEFUL
new_leader_uuid: "584dac86183a41d9b3bea1c2085e62fa"
 from {username='slave'} at 127.0.0.1:58802
I20260709 12:27:20.901403 29743 raft_consensus.cc:606] T fb6e78ccc01c4707af34cddf81619eab P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 LEADER]: Received request to transfer leadership to TS 584dac86183a41d9b3bea1c2085e62fa
I20260709 12:27:20.902876 16501 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 7
I20260709 12:27:20.904510 16501 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:27:20.905310 16501 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:27:20.931170 30521 raft_consensus.cc:993] T 3f41ce441fb047c8bbb741b1b9cbc0ec P c6610c00ffb948ce9c34f41dec3d49e8: : Instructing follower 584dac86183a41d9b3bea1c2085e62fa to start an election
I20260709 12:27:20.932183 30522 raft_consensus.cc:1081] T 3f41ce441fb047c8bbb741b1b9cbc0ec P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 LEADER]: Signalling peer 584dac86183a41d9b3bea1c2085e62fa to start an election
I20260709 12:27:20.934418 30366 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "3f41ce441fb047c8bbb741b1b9cbc0ec"
dest_uuid: "584dac86183a41d9b3bea1c2085e62fa"
 from {username='slave'} at 127.0.0.1:38910
I20260709 12:27:20.935108 30366 raft_consensus.cc:493] T 3f41ce441fb047c8bbb741b1b9cbc0ec P 584dac86183a41d9b3bea1c2085e62fa [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:27:20.935603 30366 raft_consensus.cc:3060] T 3f41ce441fb047c8bbb741b1b9cbc0ec P 584dac86183a41d9b3bea1c2085e62fa [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:27:20.942696 30531 raft_consensus.cc:993] T fb6e78ccc01c4707af34cddf81619eab P c6610c00ffb948ce9c34f41dec3d49e8: : Instructing follower 584dac86183a41d9b3bea1c2085e62fa to start an election
I20260709 12:27:20.943312 30522 raft_consensus.cc:1081] T fb6e78ccc01c4707af34cddf81619eab P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 LEADER]: Signalling peer 584dac86183a41d9b3bea1c2085e62fa to start an election
I20260709 12:27:20.944949 30363 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "fb6e78ccc01c4707af34cddf81619eab"
dest_uuid: "584dac86183a41d9b3bea1c2085e62fa"
 from {username='slave'} at 127.0.0.1:38910
I20260709 12:27:20.945405 30363 raft_consensus.cc:493] T fb6e78ccc01c4707af34cddf81619eab P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:27:20.945746 30363 raft_consensus.cc:3060] T fb6e78ccc01c4707af34cddf81619eab P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:20.947943 30366 raft_consensus.cc:515] T 3f41ce441fb047c8bbb741b1b9cbc0ec P 584dac86183a41d9b3bea1c2085e62fa [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:27:20.954985 30363 raft_consensus.cc:515] T fb6e78ccc01c4707af34cddf81619eab P 584dac86183a41d9b3bea1c2085e62fa [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:20.956631 30366 leader_election.cc:290] T 3f41ce441fb047c8bbb741b1b9cbc0ec P 584dac86183a41d9b3bea1c2085e62fa [CANDIDATE]: Term 3 election: Requested vote from peers d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209), c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895)
I20260709 12:27:20.959757 30363 leader_election.cc:290] T fb6e78ccc01c4707af34cddf81619eab P 584dac86183a41d9b3bea1c2085e62fa [CANDIDATE]: Term 2 election: Requested vote from peers c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895), d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825)
I20260709 12:27:20.978606 29668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3f41ce441fb047c8bbb741b1b9cbc0ec" candidate_uuid: "584dac86183a41d9b3bea1c2085e62fa" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "d13c58ad8ac441c4a9b396eb04b1a476"
I20260709 12:27:20.979619 29668 raft_consensus.cc:3060] T 3f41ce441fb047c8bbb741b1b9cbc0ec P d13c58ad8ac441c4a9b396eb04b1a476 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:27:20.982442 29591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb6e78ccc01c4707af34cddf81619eab" candidate_uuid: "584dac86183a41d9b3bea1c2085e62fa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d73d92fb35234d16824c55689b5fce28"
I20260709 12:27:20.983108 29591 raft_consensus.cc:3060] T fb6e78ccc01c4707af34cddf81619eab P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:20.984254 29743 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3f41ce441fb047c8bbb741b1b9cbc0ec" candidate_uuid: "584dac86183a41d9b3bea1c2085e62fa" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "c6610c00ffb948ce9c34f41dec3d49e8"
I20260709 12:27:20.984962 29743 raft_consensus.cc:3055] T 3f41ce441fb047c8bbb741b1b9cbc0ec P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:27:20.985219 29743 raft_consensus.cc:740] T 3f41ce441fb047c8bbb741b1b9cbc0ec P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: c6610c00ffb948ce9c34f41dec3d49e8, State: Running, Role: LEADER
I20260709 12:27:20.986059 29743 consensus_queue.cc:260] T 3f41ce441fb047c8bbb741b1b9cbc0ec P c6610c00ffb948ce9c34f41dec3d49e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:27:20.987529 29743 raft_consensus.cc:3060] T 3f41ce441fb047c8bbb741b1b9cbc0ec P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:27:20.991016 29668 raft_consensus.cc:2468] T 3f41ce441fb047c8bbb741b1b9cbc0ec P d13c58ad8ac441c4a9b396eb04b1a476 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 584dac86183a41d9b3bea1c2085e62fa in term 3.
I20260709 12:27:20.992413 29744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb6e78ccc01c4707af34cddf81619eab" candidate_uuid: "584dac86183a41d9b3bea1c2085e62fa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "c6610c00ffb948ce9c34f41dec3d49e8"
I20260709 12:27:20.993040 29744 raft_consensus.cc:3055] T fb6e78ccc01c4707af34cddf81619eab P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:27:20.993372 29744 raft_consensus.cc:740] T fb6e78ccc01c4707af34cddf81619eab P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c6610c00ffb948ce9c34f41dec3d49e8, State: Running, Role: LEADER
I20260709 12:27:20.993149 30314 leader_election.cc:304] T 3f41ce441fb047c8bbb741b1b9cbc0ec P 584dac86183a41d9b3bea1c2085e62fa [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 584dac86183a41d9b3bea1c2085e62fa, d13c58ad8ac441c4a9b396eb04b1a476; no voters: 
I20260709 12:27:20.994112 29591 raft_consensus.cc:2468] T fb6e78ccc01c4707af34cddf81619eab P d73d92fb35234d16824c55689b5fce28 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 584dac86183a41d9b3bea1c2085e62fa in term 2.
I20260709 12:27:20.994369 29744 consensus_queue.cc:260] T fb6e78ccc01c4707af34cddf81619eab P c6610c00ffb948ce9c34f41dec3d49e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:20.995828 29744 raft_consensus.cc:3060] T fb6e78ccc01c4707af34cddf81619eab P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:20.995997 30313 leader_election.cc:304] T fb6e78ccc01c4707af34cddf81619eab P 584dac86183a41d9b3bea1c2085e62fa [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 584dac86183a41d9b3bea1c2085e62fa, d73d92fb35234d16824c55689b5fce28; no voters: 
I20260709 12:27:20.999771 29743 raft_consensus.cc:2468] T 3f41ce441fb047c8bbb741b1b9cbc0ec P c6610c00ffb948ce9c34f41dec3d49e8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 584dac86183a41d9b3bea1c2085e62fa in term 3.
I20260709 12:27:21.004281 30540 raft_consensus.cc:2804] T 3f41ce441fb047c8bbb741b1b9cbc0ec P 584dac86183a41d9b3bea1c2085e62fa [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:27:21.018628 29744 raft_consensus.cc:2468] T fb6e78ccc01c4707af34cddf81619eab P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 584dac86183a41d9b3bea1c2085e62fa in term 2.
I20260709 12:27:21.035485 30540 raft_consensus.cc:697] T 3f41ce441fb047c8bbb741b1b9cbc0ec P 584dac86183a41d9b3bea1c2085e62fa [term 3 LEADER]: Becoming Leader. State: Replica: 584dac86183a41d9b3bea1c2085e62fa, State: Running, Role: LEADER
I20260709 12:27:21.050747 30540 consensus_queue.cc:237] T 3f41ce441fb047c8bbb741b1b9cbc0ec P 584dac86183a41d9b3bea1c2085e62fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:27:21.054535 30542 raft_consensus.cc:2804] T fb6e78ccc01c4707af34cddf81619eab P 584dac86183a41d9b3bea1c2085e62fa [term 2 FOLLOWER]: Leader election won for term 2
W20260709 12:27:21.055934 30313 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.66:36825, user_credentials={real_user=slave}} blocked reactor thread for 60189.7us
I20260709 12:27:21.076061 30542 raft_consensus.cc:697] T fb6e78ccc01c4707af34cddf81619eab P 584dac86183a41d9b3bea1c2085e62fa [term 2 LEADER]: Becoming Leader. State: Replica: 584dac86183a41d9b3bea1c2085e62fa, State: Running, Role: LEADER
I20260709 12:27:21.077802 30542 consensus_queue.cc:237] T fb6e78ccc01c4707af34cddf81619eab P 584dac86183a41d9b3bea1c2085e62fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:21.302423 30521 raft_consensus.cc:993] T 9633e3b1d78f403097e4a00b7d4b869b P c6610c00ffb948ce9c34f41dec3d49e8: : Instructing follower 584dac86183a41d9b3bea1c2085e62fa to start an election
I20260709 12:27:21.303345 30521 raft_consensus.cc:1081] T 9633e3b1d78f403097e4a00b7d4b869b P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 LEADER]: Signalling peer 584dac86183a41d9b3bea1c2085e62fa to start an election
I20260709 12:27:21.344857 30522 raft_consensus.cc:993] T 6a30685caefb4d0e80fa4ea87d70760c P c6610c00ffb948ce9c34f41dec3d49e8: : Instructing follower 584dac86183a41d9b3bea1c2085e62fa to start an election
I20260709 12:27:21.345357 30521 raft_consensus.cc:1081] T 6a30685caefb4d0e80fa4ea87d70760c P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 LEADER]: Signalling peer 584dac86183a41d9b3bea1c2085e62fa to start an election
I20260709 12:27:21.348996 29419 catalog_manager.cc:5697] T 3f41ce441fb047c8bbb741b1b9cbc0ec P 584dac86183a41d9b3bea1c2085e62fa reported cstate change: term changed from 2 to 3, leader changed from c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68) to 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65). New cstate: current_term: 3 leader_uuid: "584dac86183a41d9b3bea1c2085e62fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } health_report { overall_health: HEALTHY } } }
I20260709 12:27:21.350152 30367 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "9633e3b1d78f403097e4a00b7d4b869b"
dest_uuid: "584dac86183a41d9b3bea1c2085e62fa"
 from {username='slave'} at 127.0.0.1:38910
I20260709 12:27:21.350782 30367 raft_consensus.cc:493] T 9633e3b1d78f403097e4a00b7d4b869b P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:27:21.351166 30367 raft_consensus.cc:3060] T 9633e3b1d78f403097e4a00b7d4b869b P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:21.350871 29419 catalog_manager.cc:5697] T fb6e78ccc01c4707af34cddf81619eab P 584dac86183a41d9b3bea1c2085e62fa reported cstate change: term changed from 1 to 2, leader changed from c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68) to 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65). New cstate: current_term: 2 leader_uuid: "584dac86183a41d9b3bea1c2085e62fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:21.359321 30522 raft_consensus.cc:993] T 592801934cad4789b9ce8dc09f1470bd P c6610c00ffb948ce9c34f41dec3d49e8: : Instructing follower 584dac86183a41d9b3bea1c2085e62fa to start an election
I20260709 12:27:21.360059 30548 raft_consensus.cc:1081] T 592801934cad4789b9ce8dc09f1470bd P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 LEADER]: Signalling peer 584dac86183a41d9b3bea1c2085e62fa to start an election
I20260709 12:27:21.361642 30363 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "592801934cad4789b9ce8dc09f1470bd"
dest_uuid: "584dac86183a41d9b3bea1c2085e62fa"
 from {username='slave'} at 127.0.0.1:38910
I20260709 12:27:21.362206 30363 raft_consensus.cc:493] T 592801934cad4789b9ce8dc09f1470bd P 584dac86183a41d9b3bea1c2085e62fa [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:27:21.362545 30363 raft_consensus.cc:3060] T 592801934cad4789b9ce8dc09f1470bd P 584dac86183a41d9b3bea1c2085e62fa [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:27:21.363554 30551 raft_consensus.cc:993] T 1d1c1f58c4d340ac951dfa403d34138a P d73d92fb35234d16824c55689b5fce28: : Instructing follower 584dac86183a41d9b3bea1c2085e62fa to start an election
I20260709 12:27:21.361642 30365 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "6a30685caefb4d0e80fa4ea87d70760c"
dest_uuid: "584dac86183a41d9b3bea1c2085e62fa"
 from {username='slave'} at 127.0.0.1:38910
I20260709 12:27:21.364241 30547 raft_consensus.cc:1081] T 1d1c1f58c4d340ac951dfa403d34138a P d73d92fb35234d16824c55689b5fce28 [term 1 LEADER]: Signalling peer 584dac86183a41d9b3bea1c2085e62fa to start an election
I20260709 12:27:21.364694 30365 raft_consensus.cc:493] T 6a30685caefb4d0e80fa4ea87d70760c P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:27:21.365104 30365 raft_consensus.cc:3060] T 6a30685caefb4d0e80fa4ea87d70760c P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:21.363332 30367 raft_consensus.cc:515] T 9633e3b1d78f403097e4a00b7d4b869b P 584dac86183a41d9b3bea1c2085e62fa [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:27:21.371596 30364 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "1d1c1f58c4d340ac951dfa403d34138a"
dest_uuid: "584dac86183a41d9b3bea1c2085e62fa"
 from {username='slave'} at 127.0.0.1:38894
I20260709 12:27:21.372207 30527 raft_consensus.cc:993] T f76101eb30f14b98a9aa8c41e0d6f831 P d13c58ad8ac441c4a9b396eb04b1a476: : Instructing follower 584dac86183a41d9b3bea1c2085e62fa to start an election
I20260709 12:27:21.372830 30527 raft_consensus.cc:1081] T f76101eb30f14b98a9aa8c41e0d6f831 P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 LEADER]: Signalling peer 584dac86183a41d9b3bea1c2085e62fa to start an election
I20260709 12:27:21.372332 30364 raft_consensus.cc:493] T 1d1c1f58c4d340ac951dfa403d34138a P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:27:21.374009 30364 raft_consensus.cc:3060] T 1d1c1f58c4d340ac951dfa403d34138a P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:21.376096 30359 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "f76101eb30f14b98a9aa8c41e0d6f831"
dest_uuid: "584dac86183a41d9b3bea1c2085e62fa"
 from {username='slave'} at 127.0.0.1:38916
I20260709 12:27:21.376696 30359 raft_consensus.cc:493] T f76101eb30f14b98a9aa8c41e0d6f831 P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:27:21.377687 30359 raft_consensus.cc:3060] T f76101eb30f14b98a9aa8c41e0d6f831 P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:21.384131 29668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9633e3b1d78f403097e4a00b7d4b869b" candidate_uuid: "584dac86183a41d9b3bea1c2085e62fa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d13c58ad8ac441c4a9b396eb04b1a476"
I20260709 12:27:21.384825 29668 raft_consensus.cc:3060] T 9633e3b1d78f403097e4a00b7d4b869b P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:21.385854 30364 raft_consensus.cc:515] T 1d1c1f58c4d340ac951dfa403d34138a P 584dac86183a41d9b3bea1c2085e62fa [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:21.386566 30367 leader_election.cc:290] T 9633e3b1d78f403097e4a00b7d4b869b P 584dac86183a41d9b3bea1c2085e62fa [CANDIDATE]: Term 2 election: Requested vote from peers d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209), c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895)
I20260709 12:27:21.386759 29743 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9633e3b1d78f403097e4a00b7d4b869b" candidate_uuid: "584dac86183a41d9b3bea1c2085e62fa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "c6610c00ffb948ce9c34f41dec3d49e8"
I20260709 12:27:21.387477 29743 raft_consensus.cc:3055] T 9633e3b1d78f403097e4a00b7d4b869b P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:27:21.387785 29743 raft_consensus.cc:740] T 9633e3b1d78f403097e4a00b7d4b869b P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c6610c00ffb948ce9c34f41dec3d49e8, State: Running, Role: LEADER
I20260709 12:27:21.388803 29741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d1c1f58c4d340ac951dfa403d34138a" candidate_uuid: "584dac86183a41d9b3bea1c2085e62fa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "c6610c00ffb948ce9c34f41dec3d49e8"
I20260709 12:27:21.388648 29743 consensus_queue.cc:260] T 9633e3b1d78f403097e4a00b7d4b869b P c6610c00ffb948ce9c34f41dec3d49e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:27:21.389082 30364 leader_election.cc:290] T 1d1c1f58c4d340ac951dfa403d34138a P 584dac86183a41d9b3bea1c2085e62fa [CANDIDATE]: Term 2 election: Requested vote from peers c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895), d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825)
I20260709 12:27:21.389415 29741 raft_consensus.cc:3060] T 1d1c1f58c4d340ac951dfa403d34138a P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:21.389827 29743 raft_consensus.cc:3060] T 9633e3b1d78f403097e4a00b7d4b869b P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:21.400743 29589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d1c1f58c4d340ac951dfa403d34138a" candidate_uuid: "584dac86183a41d9b3bea1c2085e62fa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d73d92fb35234d16824c55689b5fce28"
I20260709 12:27:21.401396 29589 raft_consensus.cc:3055] T 1d1c1f58c4d340ac951dfa403d34138a P d73d92fb35234d16824c55689b5fce28 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:27:21.401695 29589 raft_consensus.cc:740] T 1d1c1f58c4d340ac951dfa403d34138a P d73d92fb35234d16824c55689b5fce28 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d73d92fb35234d16824c55689b5fce28, State: Running, Role: LEADER
I20260709 12:27:21.402534 29589 consensus_queue.cc:260] T 1d1c1f58c4d340ac951dfa403d34138a P d73d92fb35234d16824c55689b5fce28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:21.403447 29741 raft_consensus.cc:2468] T 1d1c1f58c4d340ac951dfa403d34138a P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 584dac86183a41d9b3bea1c2085e62fa in term 2.
I20260709 12:27:21.405546 30314 leader_election.cc:304] T 1d1c1f58c4d340ac951dfa403d34138a P 584dac86183a41d9b3bea1c2085e62fa [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 584dac86183a41d9b3bea1c2085e62fa, c6610c00ffb948ce9c34f41dec3d49e8; no voters: 
I20260709 12:27:21.405750 30365 raft_consensus.cc:515] T 6a30685caefb4d0e80fa4ea87d70760c P 584dac86183a41d9b3bea1c2085e62fa [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:21.407243 30542 raft_consensus.cc:2804] T 1d1c1f58c4d340ac951dfa403d34138a P 584dac86183a41d9b3bea1c2085e62fa [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:27:21.407732 30542 raft_consensus.cc:697] T 1d1c1f58c4d340ac951dfa403d34138a P 584dac86183a41d9b3bea1c2085e62fa [term 2 LEADER]: Becoming Leader. State: Replica: 584dac86183a41d9b3bea1c2085e62fa, State: Running, Role: LEADER
I20260709 12:27:21.408432 29741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6a30685caefb4d0e80fa4ea87d70760c" candidate_uuid: "584dac86183a41d9b3bea1c2085e62fa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "c6610c00ffb948ce9c34f41dec3d49e8"
I20260709 12:27:21.409014 29741 raft_consensus.cc:3055] T 6a30685caefb4d0e80fa4ea87d70760c P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:27:21.408699 30542 consensus_queue.cc:237] T 1d1c1f58c4d340ac951dfa403d34138a P 584dac86183a41d9b3bea1c2085e62fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:21.409358 29741 raft_consensus.cc:740] T 6a30685caefb4d0e80fa4ea87d70760c P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c6610c00ffb948ce9c34f41dec3d49e8, State: Running, Role: LEADER
I20260709 12:27:21.410215 29741 consensus_queue.cc:260] T 6a30685caefb4d0e80fa4ea87d70760c P c6610c00ffb948ce9c34f41dec3d49e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:21.411269 29741 raft_consensus.cc:3060] T 6a30685caefb4d0e80fa4ea87d70760c P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:21.403800 29589 raft_consensus.cc:3060] T 1d1c1f58c4d340ac951dfa403d34138a P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:21.414604 29743 raft_consensus.cc:2468] T 9633e3b1d78f403097e4a00b7d4b869b P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 584dac86183a41d9b3bea1c2085e62fa in term 2.
I20260709 12:27:21.415649 30314 leader_election.cc:304] T 9633e3b1d78f403097e4a00b7d4b869b P 584dac86183a41d9b3bea1c2085e62fa [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 584dac86183a41d9b3bea1c2085e62fa, c6610c00ffb948ce9c34f41dec3d49e8; no voters: 
I20260709 12:27:21.416649 30540 raft_consensus.cc:2804] T 9633e3b1d78f403097e4a00b7d4b869b P 584dac86183a41d9b3bea1c2085e62fa [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:27:21.417263 30540 raft_consensus.cc:697] T 9633e3b1d78f403097e4a00b7d4b869b P 584dac86183a41d9b3bea1c2085e62fa [term 2 LEADER]: Becoming Leader. State: Replica: 584dac86183a41d9b3bea1c2085e62fa, State: Running, Role: LEADER
I20260709 12:27:21.418102 29668 raft_consensus.cc:2468] T 9633e3b1d78f403097e4a00b7d4b869b P d13c58ad8ac441c4a9b396eb04b1a476 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 584dac86183a41d9b3bea1c2085e62fa in term 2.
I20260709 12:27:21.418252 30540 consensus_queue.cc:237] T 9633e3b1d78f403097e4a00b7d4b869b P 584dac86183a41d9b3bea1c2085e62fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:27:21.419210 30365 leader_election.cc:290] T 6a30685caefb4d0e80fa4ea87d70760c P 584dac86183a41d9b3bea1c2085e62fa [CANDIDATE]: Term 2 election: Requested vote from peers c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895), d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825)
I20260709 12:27:21.419381 29591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6a30685caefb4d0e80fa4ea87d70760c" candidate_uuid: "584dac86183a41d9b3bea1c2085e62fa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d73d92fb35234d16824c55689b5fce28"
I20260709 12:27:21.420112 29591 raft_consensus.cc:3060] T 6a30685caefb4d0e80fa4ea87d70760c P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:21.376682 30363 raft_consensus.cc:515] T 592801934cad4789b9ce8dc09f1470bd P 584dac86183a41d9b3bea1c2085e62fa [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:21.425396 29741 raft_consensus.cc:2468] T 6a30685caefb4d0e80fa4ea87d70760c P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 584dac86183a41d9b3bea1c2085e62fa in term 2.
I20260709 12:27:21.426736 29591 raft_consensus.cc:2468] T 6a30685caefb4d0e80fa4ea87d70760c P d73d92fb35234d16824c55689b5fce28 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 584dac86183a41d9b3bea1c2085e62fa in term 2.
I20260709 12:27:21.437121 30314 leader_election.cc:304] T 6a30685caefb4d0e80fa4ea87d70760c P 584dac86183a41d9b3bea1c2085e62fa [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 584dac86183a41d9b3bea1c2085e62fa, c6610c00ffb948ce9c34f41dec3d49e8; no voters: 
I20260709 12:27:21.438555 30540 raft_consensus.cc:2804] T 6a30685caefb4d0e80fa4ea87d70760c P 584dac86183a41d9b3bea1c2085e62fa [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:27:21.439050 30540 raft_consensus.cc:697] T 6a30685caefb4d0e80fa4ea87d70760c P 584dac86183a41d9b3bea1c2085e62fa [term 2 LEADER]: Becoming Leader. State: Replica: 584dac86183a41d9b3bea1c2085e62fa, State: Running, Role: LEADER
I20260709 12:27:21.440071 30363 leader_election.cc:290] T 592801934cad4789b9ce8dc09f1470bd P 584dac86183a41d9b3bea1c2085e62fa [CANDIDATE]: Term 3 election: Requested vote from peers c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895), d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825)
I20260709 12:27:21.445160 29591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592801934cad4789b9ce8dc09f1470bd" candidate_uuid: "584dac86183a41d9b3bea1c2085e62fa" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "d73d92fb35234d16824c55689b5fce28"
I20260709 12:27:21.445767 29591 raft_consensus.cc:3060] T 592801934cad4789b9ce8dc09f1470bd P d73d92fb35234d16824c55689b5fce28 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:27:21.445773 29741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "592801934cad4789b9ce8dc09f1470bd" candidate_uuid: "584dac86183a41d9b3bea1c2085e62fa" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "c6610c00ffb948ce9c34f41dec3d49e8"
I20260709 12:27:21.446447 29741 raft_consensus.cc:3055] T 592801934cad4789b9ce8dc09f1470bd P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:27:21.446763 29741 raft_consensus.cc:740] T 592801934cad4789b9ce8dc09f1470bd P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: c6610c00ffb948ce9c34f41dec3d49e8, State: Running, Role: LEADER
I20260709 12:27:21.447563 29741 consensus_queue.cc:260] T 592801934cad4789b9ce8dc09f1470bd P c6610c00ffb948ce9c34f41dec3d49e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:21.448812 29741 raft_consensus.cc:3060] T 592801934cad4789b9ce8dc09f1470bd P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:27:21.452813 29591 raft_consensus.cc:2468] T 592801934cad4789b9ce8dc09f1470bd P d73d92fb35234d16824c55689b5fce28 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 584dac86183a41d9b3bea1c2085e62fa in term 3.
I20260709 12:27:21.440069 30540 consensus_queue.cc:237] T 6a30685caefb4d0e80fa4ea87d70760c P 584dac86183a41d9b3bea1c2085e62fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:21.453393 29743 raft_consensus.cc:1275] T 3f41ce441fb047c8bbb741b1b9cbc0ec P c6610c00ffb948ce9c34f41dec3d49e8 [term 3 FOLLOWER]: Refusing update from remote peer 584dac86183a41d9b3bea1c2085e62fa: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:27:21.455084 30542 consensus_queue.cc:1048] T 3f41ce441fb047c8bbb741b1b9cbc0ec P 584dac86183a41d9b3bea1c2085e62fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:27:21.456597 30313 leader_election.cc:304] T 592801934cad4789b9ce8dc09f1470bd P 584dac86183a41d9b3bea1c2085e62fa [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 584dac86183a41d9b3bea1c2085e62fa, d73d92fb35234d16824c55689b5fce28; no voters: 
I20260709 12:27:21.460377 30542 raft_consensus.cc:2804] T 592801934cad4789b9ce8dc09f1470bd P 584dac86183a41d9b3bea1c2085e62fa [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:27:21.461369 30542 raft_consensus.cc:697] T 592801934cad4789b9ce8dc09f1470bd P 584dac86183a41d9b3bea1c2085e62fa [term 3 LEADER]: Becoming Leader. State: Replica: 584dac86183a41d9b3bea1c2085e62fa, State: Running, Role: LEADER
I20260709 12:27:21.462574 30542 consensus_queue.cc:237] T 592801934cad4789b9ce8dc09f1470bd P 584dac86183a41d9b3bea1c2085e62fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:21.443321 30359 raft_consensus.cc:515] T f76101eb30f14b98a9aa8c41e0d6f831 P 584dac86183a41d9b3bea1c2085e62fa [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:27:21.468236 29667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f76101eb30f14b98a9aa8c41e0d6f831" candidate_uuid: "584dac86183a41d9b3bea1c2085e62fa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d13c58ad8ac441c4a9b396eb04b1a476"
I20260709 12:27:21.469007 29667 raft_consensus.cc:3055] T f76101eb30f14b98a9aa8c41e0d6f831 P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:27:21.469377 29667 raft_consensus.cc:740] T f76101eb30f14b98a9aa8c41e0d6f831 P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d13c58ad8ac441c4a9b396eb04b1a476, State: Running, Role: LEADER
I20260709 12:27:21.470341 29667 consensus_queue.cc:260] T f76101eb30f14b98a9aa8c41e0d6f831 P d13c58ad8ac441c4a9b396eb04b1a476 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:27:21.471488 29667 raft_consensus.cc:3060] T f76101eb30f14b98a9aa8c41e0d6f831 P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:21.452598 29589 raft_consensus.cc:2468] T 1d1c1f58c4d340ac951dfa403d34138a P d73d92fb35234d16824c55689b5fce28 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 584dac86183a41d9b3bea1c2085e62fa in term 2.
I20260709 12:27:21.485914 29667 raft_consensus.cc:2468] T f76101eb30f14b98a9aa8c41e0d6f831 P d13c58ad8ac441c4a9b396eb04b1a476 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 584dac86183a41d9b3bea1c2085e62fa in term 2.
I20260709 12:27:21.486800 29742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f76101eb30f14b98a9aa8c41e0d6f831" candidate_uuid: "584dac86183a41d9b3bea1c2085e62fa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "c6610c00ffb948ce9c34f41dec3d49e8"
I20260709 12:27:21.487464 29742 raft_consensus.cc:3060] T f76101eb30f14b98a9aa8c41e0d6f831 P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:21.489149 30359 leader_election.cc:290] T f76101eb30f14b98a9aa8c41e0d6f831 P 584dac86183a41d9b3bea1c2085e62fa [CANDIDATE]: Term 2 election: Requested vote from peers d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209), c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895)
I20260709 12:27:21.500119 29741 raft_consensus.cc:2468] T 592801934cad4789b9ce8dc09f1470bd P c6610c00ffb948ce9c34f41dec3d49e8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 584dac86183a41d9b3bea1c2085e62fa in term 3.
I20260709 12:27:21.502475 30314 leader_election.cc:304] T f76101eb30f14b98a9aa8c41e0d6f831 P 584dac86183a41d9b3bea1c2085e62fa [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 584dac86183a41d9b3bea1c2085e62fa, d13c58ad8ac441c4a9b396eb04b1a476; no voters: 
I20260709 12:27:21.503340 30543 raft_consensus.cc:2804] T f76101eb30f14b98a9aa8c41e0d6f831 P 584dac86183a41d9b3bea1c2085e62fa [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:27:21.503859 30543 raft_consensus.cc:697] T f76101eb30f14b98a9aa8c41e0d6f831 P 584dac86183a41d9b3bea1c2085e62fa [term 2 LEADER]: Becoming Leader. State: Replica: 584dac86183a41d9b3bea1c2085e62fa, State: Running, Role: LEADER
I20260709 12:27:21.505100 30543 consensus_queue.cc:237] T f76101eb30f14b98a9aa8c41e0d6f831 P 584dac86183a41d9b3bea1c2085e62fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:27:21.506211 29742 raft_consensus.cc:2468] T f76101eb30f14b98a9aa8c41e0d6f831 P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 584dac86183a41d9b3bea1c2085e62fa in term 2.
I20260709 12:27:21.529028 29589 raft_consensus.cc:1275] T fb6e78ccc01c4707af34cddf81619eab P d73d92fb35234d16824c55689b5fce28 [term 2 FOLLOWER]: Refusing update from remote peer 584dac86183a41d9b3bea1c2085e62fa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:27:21.530747 30540 consensus_queue.cc:1048] T fb6e78ccc01c4707af34cddf81619eab P 584dac86183a41d9b3bea1c2085e62fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:27:21.536494 29667 raft_consensus.cc:1275] T 3f41ce441fb047c8bbb741b1b9cbc0ec P d13c58ad8ac441c4a9b396eb04b1a476 [term 3 FOLLOWER]: Refusing update from remote peer 584dac86183a41d9b3bea1c2085e62fa: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:27:21.537689 30543 consensus_queue.cc:1048] T 3f41ce441fb047c8bbb741b1b9cbc0ec P 584dac86183a41d9b3bea1c2085e62fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:27:21.566303 29743 raft_consensus.cc:1275] T fb6e78ccc01c4707af34cddf81619eab P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Refusing update from remote peer 584dac86183a41d9b3bea1c2085e62fa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:27:21.567963 30542 consensus_queue.cc:1048] T fb6e78ccc01c4707af34cddf81619eab P 584dac86183a41d9b3bea1c2085e62fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:27:21.591833 29419 catalog_manager.cc:5697] T 1d1c1f58c4d340ac951dfa403d34138a P 584dac86183a41d9b3bea1c2085e62fa reported cstate change: term changed from 1 to 2, leader changed from d73d92fb35234d16824c55689b5fce28 (127.16.29.66) to 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65). New cstate: current_term: 2 leader_uuid: "584dac86183a41d9b3bea1c2085e62fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:21.593932 29419 catalog_manager.cc:5697] T 6a30685caefb4d0e80fa4ea87d70760c P 584dac86183a41d9b3bea1c2085e62fa reported cstate change: term changed from 1 to 2, leader changed from c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68) to 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65). New cstate: current_term: 2 leader_uuid: "584dac86183a41d9b3bea1c2085e62fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:21.595219 29419 catalog_manager.cc:5697] T f76101eb30f14b98a9aa8c41e0d6f831 P 584dac86183a41d9b3bea1c2085e62fa reported cstate change: term changed from 1 to 2, leader changed from d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67) to 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65). New cstate: current_term: 2 leader_uuid: "584dac86183a41d9b3bea1c2085e62fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } health_report { overall_health: HEALTHY } } }
I20260709 12:27:21.596714 29419 catalog_manager.cc:5697] T 9633e3b1d78f403097e4a00b7d4b869b P 584dac86183a41d9b3bea1c2085e62fa reported cstate change: term changed from 1 to 2, leader changed from c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68) to 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65). New cstate: current_term: 2 leader_uuid: "584dac86183a41d9b3bea1c2085e62fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } health_report { overall_health: HEALTHY } } }
I20260709 12:27:21.598229 29419 catalog_manager.cc:5697] T 592801934cad4789b9ce8dc09f1470bd P 584dac86183a41d9b3bea1c2085e62fa reported cstate change: term changed from 2 to 3, leader changed from c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68) to 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65). New cstate: current_term: 3 leader_uuid: "584dac86183a41d9b3bea1c2085e62fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:21.861784 29589 raft_consensus.cc:1275] T 1d1c1f58c4d340ac951dfa403d34138a P d73d92fb35234d16824c55689b5fce28 [term 2 FOLLOWER]: Refusing update from remote peer 584dac86183a41d9b3bea1c2085e62fa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:27:21.862340 29743 raft_consensus.cc:1275] T 9633e3b1d78f403097e4a00b7d4b869b P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Refusing update from remote peer 584dac86183a41d9b3bea1c2085e62fa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:27:21.863348 30542 consensus_queue.cc:1048] T 1d1c1f58c4d340ac951dfa403d34138a P 584dac86183a41d9b3bea1c2085e62fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:27:21.864176 30541 consensus_queue.cc:1048] T 9633e3b1d78f403097e4a00b7d4b869b P 584dac86183a41d9b3bea1c2085e62fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:27:21.877537 29668 raft_consensus.cc:1275] T 9633e3b1d78f403097e4a00b7d4b869b P d13c58ad8ac441c4a9b396eb04b1a476 [term 2 FOLLOWER]: Refusing update from remote peer 584dac86183a41d9b3bea1c2085e62fa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:27:21.879900 30541 consensus_queue.cc:1048] T 9633e3b1d78f403097e4a00b7d4b869b P 584dac86183a41d9b3bea1c2085e62fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:27:21.890254 29742 raft_consensus.cc:1275] T 1d1c1f58c4d340ac951dfa403d34138a P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Refusing update from remote peer 584dac86183a41d9b3bea1c2085e62fa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:27:21.895795 30542 consensus_queue.cc:1048] T 1d1c1f58c4d340ac951dfa403d34138a P 584dac86183a41d9b3bea1c2085e62fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:27:21.930629 29742 raft_consensus.cc:1275] T 592801934cad4789b9ce8dc09f1470bd P c6610c00ffb948ce9c34f41dec3d49e8 [term 3 FOLLOWER]: Refusing update from remote peer 584dac86183a41d9b3bea1c2085e62fa: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:27:21.932149 30542 consensus_queue.cc:1048] T 592801934cad4789b9ce8dc09f1470bd P 584dac86183a41d9b3bea1c2085e62fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:27:21.944643 29589 raft_consensus.cc:1275] T 592801934cad4789b9ce8dc09f1470bd P d73d92fb35234d16824c55689b5fce28 [term 3 FOLLOWER]: Refusing update from remote peer 584dac86183a41d9b3bea1c2085e62fa: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:27:21.946058 30540 consensus_queue.cc:1048] T 592801934cad4789b9ce8dc09f1470bd P 584dac86183a41d9b3bea1c2085e62fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:27:21.958195 29742 raft_consensus.cc:1275] T 6a30685caefb4d0e80fa4ea87d70760c P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Refusing update from remote peer 584dac86183a41d9b3bea1c2085e62fa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:27:21.959703 30541 consensus_queue.cc:1048] T 6a30685caefb4d0e80fa4ea87d70760c P 584dac86183a41d9b3bea1c2085e62fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:27:21.981684 29589 raft_consensus.cc:1275] T 6a30685caefb4d0e80fa4ea87d70760c P d73d92fb35234d16824c55689b5fce28 [term 2 FOLLOWER]: Refusing update from remote peer 584dac86183a41d9b3bea1c2085e62fa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:27:21.983127 30540 consensus_queue.cc:1048] T 6a30685caefb4d0e80fa4ea87d70760c P 584dac86183a41d9b3bea1c2085e62fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:27:22.031770 29668 raft_consensus.cc:1275] T f76101eb30f14b98a9aa8c41e0d6f831 P d13c58ad8ac441c4a9b396eb04b1a476 [term 2 FOLLOWER]: Refusing update from remote peer 584dac86183a41d9b3bea1c2085e62fa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:27:22.032992 30540 consensus_queue.cc:1048] T f76101eb30f14b98a9aa8c41e0d6f831 P 584dac86183a41d9b3bea1c2085e62fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:27:22.041661 29742 raft_consensus.cc:1275] T f76101eb30f14b98a9aa8c41e0d6f831 P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Refusing update from remote peer 584dac86183a41d9b3bea1c2085e62fa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:27:22.043332 30542 consensus_queue.cc:1048] T f76101eb30f14b98a9aa8c41e0d6f831 P 584dac86183a41d9b3bea1c2085e62fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:27:22.170668 16501 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:27:22.212565 29589 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "01788883c8ea42989f246a3486111a7e"
dest_uuid: "d73d92fb35234d16824c55689b5fce28"
mode: GRACEFUL
new_leader_uuid: "584dac86183a41d9b3bea1c2085e62fa"
 from {username='slave'} at 127.0.0.1:43236
I20260709 12:27:22.213186 29589 raft_consensus.cc:606] T 01788883c8ea42989f246a3486111a7e P d73d92fb35234d16824c55689b5fce28 [term 1 LEADER]: Received request to transfer leadership to TS 584dac86183a41d9b3bea1c2085e62fa
I20260709 12:27:22.216064 29667 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "43c7bfb33af74d4f912e5153e8a54b3e"
dest_uuid: "d13c58ad8ac441c4a9b396eb04b1a476"
mode: GRACEFUL
new_leader_uuid: "c6610c00ffb948ce9c34f41dec3d49e8"
 from {username='slave'} at 127.0.0.1:46668
I20260709 12:27:22.216571 29667 raft_consensus.cc:606] T 43c7bfb33af74d4f912e5153e8a54b3e P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 LEADER]: Received request to transfer leadership to TS c6610c00ffb948ce9c34f41dec3d49e8
I20260709 12:27:22.219154 29589 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "71e2de235adf46d9a2407659080d42ef"
dest_uuid: "d73d92fb35234d16824c55689b5fce28"
mode: GRACEFUL
new_leader_uuid: "584dac86183a41d9b3bea1c2085e62fa"
 from {username='slave'} at 127.0.0.1:43236
I20260709 12:27:22.219640 29589 raft_consensus.cc:606] T 71e2de235adf46d9a2407659080d42ef P d73d92fb35234d16824c55689b5fce28 [term 2 LEADER]: Received request to transfer leadership to TS 584dac86183a41d9b3bea1c2085e62fa
I20260709 12:27:22.221863 29667 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "959940284eb24fdf83d78438d710318a"
dest_uuid: "d13c58ad8ac441c4a9b396eb04b1a476"
mode: GRACEFUL
new_leader_uuid: "c6610c00ffb948ce9c34f41dec3d49e8"
 from {username='slave'} at 127.0.0.1:46668
I20260709 12:27:22.222479 29667 raft_consensus.cc:606] T 959940284eb24fdf83d78438d710318a P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 LEADER]: Received request to transfer leadership to TS c6610c00ffb948ce9c34f41dec3d49e8
I20260709 12:27:22.225365 29589 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "d1918297b9eb41f5847a211e668ae748"
dest_uuid: "d73d92fb35234d16824c55689b5fce28"
mode: GRACEFUL
new_leader_uuid: "584dac86183a41d9b3bea1c2085e62fa"
 from {username='slave'} at 127.0.0.1:43236
I20260709 12:27:22.225893 29589 raft_consensus.cc:606] T d1918297b9eb41f5847a211e668ae748 P d73d92fb35234d16824c55689b5fce28 [term 1 LEADER]: Received request to transfer leadership to TS 584dac86183a41d9b3bea1c2085e62fa
I20260709 12:27:22.227075 16501 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260709 12:27:22.228312 16501 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:27:22.228991 16501 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
W20260709 12:27:22.255661 29766 debug-util.cc:398] Leaking SignalData structure 0x7b08001ca980 after lost signal to thread 16504
W20260709 12:27:22.256928 29766 debug-util.cc:398] Leaking SignalData structure 0x7b08001caf60 after lost signal to thread 29454
W20260709 12:27:22.258208 29766 debug-util.cc:398] Leaking SignalData structure 0x7b08000c0680 after lost signal to thread 29621
W20260709 12:27:22.259379 29766 debug-util.cc:398] Leaking SignalData structure 0x7b0800326e80 after lost signal to thread 29695
W20260709 12:27:22.260594 29766 debug-util.cc:398] Leaking SignalData structure 0x7b080003de40 after lost signal to thread 29769
W20260709 12:27:22.261840 29766 debug-util.cc:398] Leaking SignalData structure 0x7b0800359c20 after lost signal to thread 30406
I20260709 12:27:22.301609 30549 raft_consensus.cc:993] T 43c7bfb33af74d4f912e5153e8a54b3e P d13c58ad8ac441c4a9b396eb04b1a476: : Instructing follower c6610c00ffb948ce9c34f41dec3d49e8 to start an election
I20260709 12:27:22.302326 30549 raft_consensus.cc:1081] T 43c7bfb33af74d4f912e5153e8a54b3e P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 LEADER]: Signalling peer c6610c00ffb948ce9c34f41dec3d49e8 to start an election
I20260709 12:27:22.313719 29743 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "43c7bfb33af74d4f912e5153e8a54b3e"
dest_uuid: "c6610c00ffb948ce9c34f41dec3d49e8"
 from {username='slave'} at 127.0.0.1:33134
I20260709 12:27:22.314294 29743 raft_consensus.cc:493] T 43c7bfb33af74d4f912e5153e8a54b3e P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:27:22.314638 29743 raft_consensus.cc:3060] T 43c7bfb33af74d4f912e5153e8a54b3e P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:22.322508 29743 raft_consensus.cc:515] T 43c7bfb33af74d4f912e5153e8a54b3e P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:27:22.324717 29743 leader_election.cc:290] T 43c7bfb33af74d4f912e5153e8a54b3e P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 2 election: Requested vote from peers d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825), d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209)
I20260709 12:27:22.326444 29589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43c7bfb33af74d4f912e5153e8a54b3e" candidate_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d73d92fb35234d16824c55689b5fce28"
I20260709 12:27:22.327131 29589 raft_consensus.cc:3060] T 43c7bfb33af74d4f912e5153e8a54b3e P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:22.331622 29668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43c7bfb33af74d4f912e5153e8a54b3e" candidate_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d13c58ad8ac441c4a9b396eb04b1a476"
I20260709 12:27:22.332249 29668 raft_consensus.cc:3055] T 43c7bfb33af74d4f912e5153e8a54b3e P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:27:22.332535 29668 raft_consensus.cc:740] T 43c7bfb33af74d4f912e5153e8a54b3e P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d13c58ad8ac441c4a9b396eb04b1a476, State: Running, Role: LEADER
I20260709 12:27:22.333288 29668 consensus_queue.cc:260] T 43c7bfb33af74d4f912e5153e8a54b3e P d13c58ad8ac441c4a9b396eb04b1a476 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:27:22.334453 29668 raft_consensus.cc:3060] T 43c7bfb33af74d4f912e5153e8a54b3e P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:22.334743 29589 raft_consensus.cc:2468] T 43c7bfb33af74d4f912e5153e8a54b3e P d73d92fb35234d16824c55689b5fce28 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6610c00ffb948ce9c34f41dec3d49e8 in term 2.
I20260709 12:27:22.339952 29708 leader_election.cc:304] T 43c7bfb33af74d4f912e5153e8a54b3e P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c6610c00ffb948ce9c34f41dec3d49e8, d73d92fb35234d16824c55689b5fce28; no voters: 
I20260709 12:27:22.340986 30586 raft_consensus.cc:2804] T 43c7bfb33af74d4f912e5153e8a54b3e P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:27:22.341521 30586 raft_consensus.cc:697] T 43c7bfb33af74d4f912e5153e8a54b3e P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 LEADER]: Becoming Leader. State: Replica: c6610c00ffb948ce9c34f41dec3d49e8, State: Running, Role: LEADER
I20260709 12:27:22.342460 30586 consensus_queue.cc:237] T 43c7bfb33af74d4f912e5153e8a54b3e P c6610c00ffb948ce9c34f41dec3d49e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:27:22.344483 29668 raft_consensus.cc:2468] T 43c7bfb33af74d4f912e5153e8a54b3e P d13c58ad8ac441c4a9b396eb04b1a476 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6610c00ffb948ce9c34f41dec3d49e8 in term 2.
I20260709 12:27:22.354698 29419 catalog_manager.cc:5697] T 43c7bfb33af74d4f912e5153e8a54b3e P c6610c00ffb948ce9c34f41dec3d49e8 reported cstate change: term changed from 1 to 2, leader changed from d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67) to c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68). New cstate: current_term: 2 leader_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } health_report { overall_health: HEALTHY } } }
I20260709 12:27:22.456266 30532 raft_consensus.cc:993] T 71e2de235adf46d9a2407659080d42ef P d73d92fb35234d16824c55689b5fce28: : Instructing follower 584dac86183a41d9b3bea1c2085e62fa to start an election
I20260709 12:27:22.457006 30498 raft_consensus.cc:1081] T 71e2de235adf46d9a2407659080d42ef P d73d92fb35234d16824c55689b5fce28 [term 2 LEADER]: Signalling peer 584dac86183a41d9b3bea1c2085e62fa to start an election
I20260709 12:27:22.458659 30587 raft_consensus.cc:993] T 959940284eb24fdf83d78438d710318a P d13c58ad8ac441c4a9b396eb04b1a476: : Instructing follower c6610c00ffb948ce9c34f41dec3d49e8 to start an election
I20260709 12:27:22.458678 30359 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "71e2de235adf46d9a2407659080d42ef"
dest_uuid: "584dac86183a41d9b3bea1c2085e62fa"
 from {username='slave'} at 127.0.0.1:38894
I20260709 12:27:22.459254 30587 raft_consensus.cc:1081] T 959940284eb24fdf83d78438d710318a P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 LEADER]: Signalling peer c6610c00ffb948ce9c34f41dec3d49e8 to start an election
I20260709 12:27:22.459728 30359 raft_consensus.cc:493] T 71e2de235adf46d9a2407659080d42ef P 584dac86183a41d9b3bea1c2085e62fa [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:27:22.461092 30359 raft_consensus.cc:3060] T 71e2de235adf46d9a2407659080d42ef P 584dac86183a41d9b3bea1c2085e62fa [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:27:22.465365 29743 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "959940284eb24fdf83d78438d710318a"
dest_uuid: "c6610c00ffb948ce9c34f41dec3d49e8"
 from {username='slave'} at 127.0.0.1:33134
I20260709 12:27:22.465976 29743 raft_consensus.cc:493] T 959940284eb24fdf83d78438d710318a P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:27:22.466320 29743 raft_consensus.cc:3060] T 959940284eb24fdf83d78438d710318a P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:22.468220 30359 raft_consensus.cc:515] T 71e2de235adf46d9a2407659080d42ef P 584dac86183a41d9b3bea1c2085e62fa [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:27:22.472926 29668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "71e2de235adf46d9a2407659080d42ef" candidate_uuid: "584dac86183a41d9b3bea1c2085e62fa" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "d13c58ad8ac441c4a9b396eb04b1a476"
I20260709 12:27:22.473464 29668 raft_consensus.cc:3060] T 71e2de235adf46d9a2407659080d42ef P d13c58ad8ac441c4a9b396eb04b1a476 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:27:22.473819 29743 raft_consensus.cc:515] T 959940284eb24fdf83d78438d710318a P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:27:22.476647 29743 leader_election.cc:290] T 959940284eb24fdf83d78438d710318a P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 2 election: Requested vote from peers d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825), d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209)
I20260709 12:27:22.477955 29667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "959940284eb24fdf83d78438d710318a" candidate_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d13c58ad8ac441c4a9b396eb04b1a476"
I20260709 12:27:22.478529 29667 raft_consensus.cc:3055] T 959940284eb24fdf83d78438d710318a P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:27:22.478595 30359 leader_election.cc:290] T 71e2de235adf46d9a2407659080d42ef P 584dac86183a41d9b3bea1c2085e62fa [CANDIDATE]: Term 3 election: Requested vote from peers d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825), d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209)
I20260709 12:27:22.478881 29667 raft_consensus.cc:740] T 959940284eb24fdf83d78438d710318a P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d13c58ad8ac441c4a9b396eb04b1a476, State: Running, Role: LEADER
I20260709 12:27:22.480072 29667 consensus_queue.cc:260] T 959940284eb24fdf83d78438d710318a P d13c58ad8ac441c4a9b396eb04b1a476 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:27:22.481271 29667 raft_consensus.cc:3060] T 959940284eb24fdf83d78438d710318a P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:22.481370 29668 raft_consensus.cc:2468] T 71e2de235adf46d9a2407659080d42ef P d13c58ad8ac441c4a9b396eb04b1a476 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 584dac86183a41d9b3bea1c2085e62fa in term 3.
I20260709 12:27:22.481658 29589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "959940284eb24fdf83d78438d710318a" candidate_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d73d92fb35234d16824c55689b5fce28"
I20260709 12:27:22.482293 29589 raft_consensus.cc:3060] T 959940284eb24fdf83d78438d710318a P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:22.482561 30314 leader_election.cc:304] T 71e2de235adf46d9a2407659080d42ef P 584dac86183a41d9b3bea1c2085e62fa [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 584dac86183a41d9b3bea1c2085e62fa, d13c58ad8ac441c4a9b396eb04b1a476; no voters: 
I20260709 12:27:22.485658 29597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "71e2de235adf46d9a2407659080d42ef" candidate_uuid: "584dac86183a41d9b3bea1c2085e62fa" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "d73d92fb35234d16824c55689b5fce28"
I20260709 12:27:22.486090 30541 raft_consensus.cc:2804] T 71e2de235adf46d9a2407659080d42ef P 584dac86183a41d9b3bea1c2085e62fa [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:27:22.486263 29597 raft_consensus.cc:3055] T 71e2de235adf46d9a2407659080d42ef P d73d92fb35234d16824c55689b5fce28 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:27:22.486649 30541 raft_consensus.cc:697] T 71e2de235adf46d9a2407659080d42ef P 584dac86183a41d9b3bea1c2085e62fa [term 3 LEADER]: Becoming Leader. State: Replica: 584dac86183a41d9b3bea1c2085e62fa, State: Running, Role: LEADER
I20260709 12:27:22.486752 29597 raft_consensus.cc:740] T 71e2de235adf46d9a2407659080d42ef P d73d92fb35234d16824c55689b5fce28 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d73d92fb35234d16824c55689b5fce28, State: Running, Role: LEADER
I20260709 12:27:22.487637 30541 consensus_queue.cc:237] T 71e2de235adf46d9a2407659080d42ef P 584dac86183a41d9b3bea1c2085e62fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:27:22.487717 29597 consensus_queue.cc:260] T 71e2de235adf46d9a2407659080d42ef P d73d92fb35234d16824c55689b5fce28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:27:22.489116 29597 raft_consensus.cc:3060] T 71e2de235adf46d9a2407659080d42ef P d73d92fb35234d16824c55689b5fce28 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:27:22.494369 29667 raft_consensus.cc:2468] T 959940284eb24fdf83d78438d710318a P d13c58ad8ac441c4a9b396eb04b1a476 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6610c00ffb948ce9c34f41dec3d49e8 in term 2.
I20260709 12:27:22.499039 29709 leader_election.cc:304] T 959940284eb24fdf83d78438d710318a P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c6610c00ffb948ce9c34f41dec3d49e8, d13c58ad8ac441c4a9b396eb04b1a476; no voters: 
I20260709 12:27:22.500921 30553 raft_consensus.cc:2804] T 959940284eb24fdf83d78438d710318a P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:27:22.501698 30553 raft_consensus.cc:697] T 959940284eb24fdf83d78438d710318a P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 LEADER]: Becoming Leader. State: Replica: c6610c00ffb948ce9c34f41dec3d49e8, State: Running, Role: LEADER
I20260709 12:27:22.501864 29589 raft_consensus.cc:2468] T 959940284eb24fdf83d78438d710318a P d73d92fb35234d16824c55689b5fce28 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6610c00ffb948ce9c34f41dec3d49e8 in term 2.
I20260709 12:27:22.505901 30553 consensus_queue.cc:237] T 959940284eb24fdf83d78438d710318a P c6610c00ffb948ce9c34f41dec3d49e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:27:22.513916 29597 raft_consensus.cc:2468] T 71e2de235adf46d9a2407659080d42ef P d73d92fb35234d16824c55689b5fce28 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 584dac86183a41d9b3bea1c2085e62fa in term 3.
I20260709 12:27:22.519814 29419 catalog_manager.cc:5697] T 71e2de235adf46d9a2407659080d42ef P 584dac86183a41d9b3bea1c2085e62fa reported cstate change: term changed from 2 to 3, leader changed from d73d92fb35234d16824c55689b5fce28 (127.16.29.66) to 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65). New cstate: current_term: 3 leader_uuid: "584dac86183a41d9b3bea1c2085e62fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:22.525842 29417 catalog_manager.cc:5697] T 959940284eb24fdf83d78438d710318a P c6610c00ffb948ce9c34f41dec3d49e8 reported cstate change: term changed from 1 to 2, leader changed from d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67) to c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68). New cstate: current_term: 2 leader_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } health_report { overall_health: HEALTHY } } }
I20260709 12:27:22.659395 30532 raft_consensus.cc:993] T d1918297b9eb41f5847a211e668ae748 P d73d92fb35234d16824c55689b5fce28: : Instructing follower 584dac86183a41d9b3bea1c2085e62fa to start an election
I20260709 12:27:22.660147 30532 raft_consensus.cc:1081] T d1918297b9eb41f5847a211e668ae748 P d73d92fb35234d16824c55689b5fce28 [term 1 LEADER]: Signalling peer 584dac86183a41d9b3bea1c2085e62fa to start an election
I20260709 12:27:22.664443 30365 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "d1918297b9eb41f5847a211e668ae748"
dest_uuid: "584dac86183a41d9b3bea1c2085e62fa"
 from {username='slave'} at 127.0.0.1:38894
I20260709 12:27:22.665107 30365 raft_consensus.cc:493] T d1918297b9eb41f5847a211e668ae748 P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:27:22.665570 30365 raft_consensus.cc:3060] T d1918297b9eb41f5847a211e668ae748 P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:22.682224 30365 raft_consensus.cc:515] T d1918297b9eb41f5847a211e668ae748 P 584dac86183a41d9b3bea1c2085e62fa [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:27:22.688357 30365 leader_election.cc:290] T d1918297b9eb41f5847a211e668ae748 P 584dac86183a41d9b3bea1c2085e62fa [CANDIDATE]: Term 2 election: Requested vote from peers d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825), d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209)
I20260709 12:27:22.690704 29667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1918297b9eb41f5847a211e668ae748" candidate_uuid: "584dac86183a41d9b3bea1c2085e62fa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d13c58ad8ac441c4a9b396eb04b1a476"
I20260709 12:27:22.691432 29667 raft_consensus.cc:3060] T d1918297b9eb41f5847a211e668ae748 P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:22.698989 29667 raft_consensus.cc:2468] T d1918297b9eb41f5847a211e668ae748 P d13c58ad8ac441c4a9b396eb04b1a476 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 584dac86183a41d9b3bea1c2085e62fa in term 2.
I20260709 12:27:22.703307 30314 leader_election.cc:304] T d1918297b9eb41f5847a211e668ae748 P 584dac86183a41d9b3bea1c2085e62fa [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 584dac86183a41d9b3bea1c2085e62fa, d13c58ad8ac441c4a9b396eb04b1a476; no voters: 
I20260709 12:27:22.719379 30543 raft_consensus.cc:2804] T d1918297b9eb41f5847a211e668ae748 P 584dac86183a41d9b3bea1c2085e62fa [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:27:22.719962 29597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1918297b9eb41f5847a211e668ae748" candidate_uuid: "584dac86183a41d9b3bea1c2085e62fa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d73d92fb35234d16824c55689b5fce28"
I20260709 12:27:22.720332 30543 raft_consensus.cc:697] T d1918297b9eb41f5847a211e668ae748 P 584dac86183a41d9b3bea1c2085e62fa [term 2 LEADER]: Becoming Leader. State: Replica: 584dac86183a41d9b3bea1c2085e62fa, State: Running, Role: LEADER
I20260709 12:27:22.720690 29597 raft_consensus.cc:3055] T d1918297b9eb41f5847a211e668ae748 P d73d92fb35234d16824c55689b5fce28 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:27:22.721509 30543 consensus_queue.cc:237] T d1918297b9eb41f5847a211e668ae748 P 584dac86183a41d9b3bea1c2085e62fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:27:22.721062 29597 raft_consensus.cc:740] T d1918297b9eb41f5847a211e668ae748 P d73d92fb35234d16824c55689b5fce28 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d73d92fb35234d16824c55689b5fce28, State: Running, Role: LEADER
I20260709 12:27:22.731848 29597 consensus_queue.cc:260] T d1918297b9eb41f5847a211e668ae748 P d73d92fb35234d16824c55689b5fce28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:27:22.733326 29597 raft_consensus.cc:3060] T d1918297b9eb41f5847a211e668ae748 P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:22.742419 29597 raft_consensus.cc:2468] T d1918297b9eb41f5847a211e668ae748 P d73d92fb35234d16824c55689b5fce28 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 584dac86183a41d9b3bea1c2085e62fa in term 2.
W20260709 12:27:23.175197 29561 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:27:23.226539 30313 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:36825, user_credentials={real_user=slave}} blocked reactor thread for 510406us
I20260709 12:27:23.241554 29591 raft_consensus.cc:1275] T 71e2de235adf46d9a2407659080d42ef P d73d92fb35234d16824c55689b5fce28 [term 3 FOLLOWER]: Refusing update from remote peer 584dac86183a41d9b3bea1c2085e62fa: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:27:23.245199 30599 consensus_queue.cc:1048] T 71e2de235adf46d9a2407659080d42ef P 584dac86183a41d9b3bea1c2085e62fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:27:23.297353 16501 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:27:23.457414 29667 raft_consensus.cc:1275] T 43c7bfb33af74d4f912e5153e8a54b3e P d13c58ad8ac441c4a9b396eb04b1a476 [term 2 FOLLOWER]: Refusing update from remote peer c6610c00ffb948ce9c34f41dec3d49e8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:27:23.459551 30532 raft_consensus.cc:993] T 01788883c8ea42989f246a3486111a7e P d73d92fb35234d16824c55689b5fce28: : Instructing follower 584dac86183a41d9b3bea1c2085e62fa to start an election
I20260709 12:27:23.460136 30532 raft_consensus.cc:1081] T 01788883c8ea42989f246a3486111a7e P d73d92fb35234d16824c55689b5fce28 [term 1 LEADER]: Signalling peer 584dac86183a41d9b3bea1c2085e62fa to start an election
I20260709 12:27:23.461802 30364 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "01788883c8ea42989f246a3486111a7e"
dest_uuid: "584dac86183a41d9b3bea1c2085e62fa"
 from {username='slave'} at 127.0.0.1:38894
I20260709 12:27:23.462337 30364 raft_consensus.cc:493] T 01788883c8ea42989f246a3486111a7e P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:27:23.462702 30364 raft_consensus.cc:3060] T 01788883c8ea42989f246a3486111a7e P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:23.468134 30611 consensus_queue.cc:1048] T 43c7bfb33af74d4f912e5153e8a54b3e P c6610c00ffb948ce9c34f41dec3d49e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.002s
I20260709 12:27:23.470636 30527 raft_consensus.cc:493] T 2707271fd46f456aa765026642a3efa9 P d13c58ad8ac441c4a9b396eb04b1a476 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d73d92fb35234d16824c55689b5fce28)
I20260709 12:27:23.471215 30527 raft_consensus.cc:515] T 2707271fd46f456aa765026642a3efa9 P d13c58ad8ac441c4a9b396eb04b1a476 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:27:23.472167 30364 raft_consensus.cc:515] T 01788883c8ea42989f246a3486111a7e P 584dac86183a41d9b3bea1c2085e62fa [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:23.473537 30527 leader_election.cc:290] T 2707271fd46f456aa765026642a3efa9 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033), d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825)
I20260709 12:27:23.475998 29590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2707271fd46f456aa765026642a3efa9" candidate_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d73d92fb35234d16824c55689b5fce28" is_pre_election: true
I20260709 12:27:23.492692 29419 catalog_manager.cc:5697] T d1918297b9eb41f5847a211e668ae748 P 584dac86183a41d9b3bea1c2085e62fa reported cstate change: term changed from 1 to 2, leader changed from d73d92fb35234d16824c55689b5fce28 (127.16.29.66) to 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65). New cstate: current_term: 2 leader_uuid: "584dac86183a41d9b3bea1c2085e62fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:23.499508 29591 raft_consensus.cc:1275] T 959940284eb24fdf83d78438d710318a P d73d92fb35234d16824c55689b5fce28 [term 2 FOLLOWER]: Refusing update from remote peer c6610c00ffb948ce9c34f41dec3d49e8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:27:23.495781 30527 raft_consensus.cc:493] T 137e03144e754f6e9da4f0544513d27c P d13c58ad8ac441c4a9b396eb04b1a476 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d73d92fb35234d16824c55689b5fce28)
I20260709 12:27:23.506572 30527 raft_consensus.cc:515] T 137e03144e754f6e9da4f0544513d27c P d13c58ad8ac441c4a9b396eb04b1a476 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:27:23.495985 30366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2707271fd46f456aa765026642a3efa9" candidate_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "584dac86183a41d9b3bea1c2085e62fa" is_pre_election: true
I20260709 12:27:23.496304 29597 raft_consensus.cc:1275] T 43c7bfb33af74d4f912e5153e8a54b3e P d73d92fb35234d16824c55689b5fce28 [term 2 FOLLOWER]: Refusing update from remote peer c6610c00ffb948ce9c34f41dec3d49e8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:27:23.511145 29634 leader_election.cc:304] T 2707271fd46f456aa765026642a3efa9 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d13c58ad8ac441c4a9b396eb04b1a476; no voters: 584dac86183a41d9b3bea1c2085e62fa, d73d92fb35234d16824c55689b5fce28
I20260709 12:27:23.512223 30549 raft_consensus.cc:2749] T 2707271fd46f456aa765026642a3efa9 P d13c58ad8ac441c4a9b396eb04b1a476 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:27:23.515344 29597 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "137e03144e754f6e9da4f0544513d27c" candidate_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "d73d92fb35234d16824c55689b5fce28" is_pre_election: true
I20260709 12:27:23.516528 30527 leader_election.cc:290] T 137e03144e754f6e9da4f0544513d27c P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825), c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895)
I20260709 12:27:23.520529 30585 consensus_queue.cc:1048] T 959940284eb24fdf83d78438d710318a P c6610c00ffb948ce9c34f41dec3d49e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:27:23.546052 30364 leader_election.cc:290] T 01788883c8ea42989f246a3486111a7e P 584dac86183a41d9b3bea1c2085e62fa [CANDIDATE]: Term 2 election: Requested vote from peers c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895), d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825)
I20260709 12:27:23.569376 29590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01788883c8ea42989f246a3486111a7e" candidate_uuid: "584dac86183a41d9b3bea1c2085e62fa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d73d92fb35234d16824c55689b5fce28"
I20260709 12:27:23.573529 29590 raft_consensus.cc:3055] T 01788883c8ea42989f246a3486111a7e P d73d92fb35234d16824c55689b5fce28 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:27:23.578670 29590 raft_consensus.cc:740] T 01788883c8ea42989f246a3486111a7e P d73d92fb35234d16824c55689b5fce28 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d73d92fb35234d16824c55689b5fce28, State: Running, Role: LEADER
I20260709 12:27:23.583819 29590 consensus_queue.cc:260] T 01788883c8ea42989f246a3486111a7e P d73d92fb35234d16824c55689b5fce28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:23.590220 29590 raft_consensus.cc:3060] T 01788883c8ea42989f246a3486111a7e P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Advancing to term 2
W20260709 12:27:23.599242 29708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:36825, user_credentials={real_user=slave}} blocked reactor thread for 79964.1us
I20260709 12:27:23.600517 30617 consensus_queue.cc:1048] T 43c7bfb33af74d4f912e5153e8a54b3e P c6610c00ffb948ce9c34f41dec3d49e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:27:23.607825 29662 raft_consensus.cc:1275] T 71e2de235adf46d9a2407659080d42ef P d13c58ad8ac441c4a9b396eb04b1a476 [term 3 FOLLOWER]: Refusing update from remote peer 584dac86183a41d9b3bea1c2085e62fa: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:27:23.609719 30599 consensus_queue.cc:1048] T 71e2de235adf46d9a2407659080d42ef P 584dac86183a41d9b3bea1c2085e62fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:27:23.613108 29666 raft_consensus.cc:1275] T 959940284eb24fdf83d78438d710318a P d13c58ad8ac441c4a9b396eb04b1a476 [term 2 FOLLOWER]: Refusing update from remote peer c6610c00ffb948ce9c34f41dec3d49e8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:27:23.620306 30617 consensus_queue.cc:1048] T 959940284eb24fdf83d78438d710318a P c6610c00ffb948ce9c34f41dec3d49e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:27:23.621037 29671 raft_consensus.cc:1240] T d1918297b9eb41f5847a211e668ae748 P d13c58ad8ac441c4a9b396eb04b1a476 [term 2 FOLLOWER]: Rejecting Update request from peer d73d92fb35234d16824c55689b5fce28 for earlier term 1. Current term is 2. Ops: []
I20260709 12:27:23.644213 29742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01788883c8ea42989f246a3486111a7e" candidate_uuid: "584dac86183a41d9b3bea1c2085e62fa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "c6610c00ffb948ce9c34f41dec3d49e8"
I20260709 12:27:23.645256 29742 raft_consensus.cc:3060] T 01788883c8ea42989f246a3486111a7e P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:23.665331 30622 raft_consensus.cc:493] T fb7d78e534634e0f992fc39b578ebd8d P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d13c58ad8ac441c4a9b396eb04b1a476)
I20260709 12:27:23.665983 30622 raft_consensus.cc:515] T fb7d78e534634e0f992fc39b578ebd8d P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:27:23.669461 29666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb7d78e534634e0f992fc39b578ebd8d" candidate_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" is_pre_election: true
I20260709 12:27:23.671653 30622 leader_election.cc:290] T fb7d78e534634e0f992fc39b578ebd8d P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209), 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033)
I20260709 12:27:23.685266 29742 raft_consensus.cc:2468] T 01788883c8ea42989f246a3486111a7e P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 584dac86183a41d9b3bea1c2085e62fa in term 2.
I20260709 12:27:23.686527 30314 leader_election.cc:304] T 01788883c8ea42989f246a3486111a7e P 584dac86183a41d9b3bea1c2085e62fa [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 584dac86183a41d9b3bea1c2085e62fa, c6610c00ffb948ce9c34f41dec3d49e8; no voters: 
I20260709 12:27:23.687543 30599 raft_consensus.cc:2804] T 01788883c8ea42989f246a3486111a7e P 584dac86183a41d9b3bea1c2085e62fa [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:27:23.688946 30599 raft_consensus.cc:697] T 01788883c8ea42989f246a3486111a7e P 584dac86183a41d9b3bea1c2085e62fa [term 2 LEADER]: Becoming Leader. State: Replica: 584dac86183a41d9b3bea1c2085e62fa, State: Running, Role: LEADER
I20260709 12:27:23.690053 30599 consensus_queue.cc:237] T 01788883c8ea42989f246a3486111a7e P 584dac86183a41d9b3bea1c2085e62fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:23.705199 29742 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "4685c2e824c14fc1bcd45c16169798cc"
dest_uuid: "c6610c00ffb948ce9c34f41dec3d49e8"
mode: GRACEFUL
new_leader_uuid: "584dac86183a41d9b3bea1c2085e62fa"
 from {username='slave'} at 127.0.0.1:58802
I20260709 12:27:23.705947 29742 raft_consensus.cc:606] T 4685c2e824c14fc1bcd45c16169798cc P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 LEADER]: Received request to transfer leadership to TS 584dac86183a41d9b3bea1c2085e62fa
W20260709 12:27:23.722631 29561 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:23.720690 29419 catalog_manager.cc:5697] T 01788883c8ea42989f246a3486111a7e P 584dac86183a41d9b3bea1c2085e62fa reported cstate change: term changed from 1 to 2, leader changed from d73d92fb35234d16824c55689b5fce28 (127.16.29.66) to 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65). New cstate: current_term: 2 leader_uuid: "584dac86183a41d9b3bea1c2085e62fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:23.726267 29589 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "58154d49c1a344f2b51366ede3c87de5"
dest_uuid: "d73d92fb35234d16824c55689b5fce28"
mode: GRACEFUL
new_leader_uuid: "c6610c00ffb948ce9c34f41dec3d49e8"
 from {username='slave'} at 127.0.0.1:43236
I20260709 12:27:23.727114 29589 raft_consensus.cc:606] T 58154d49c1a344f2b51366ede3c87de5 P d73d92fb35234d16824c55689b5fce28 [term 1 LEADER]: Received request to transfer leadership to TS c6610c00ffb948ce9c34f41dec3d49e8
W20260709 12:27:23.729818 29618 debug-util.cc:398] Leaking SignalData structure 0x7b08001496a0 after lost signal to thread 16504
W20260709 12:27:23.731076 29618 debug-util.cc:398] Leaking SignalData structure 0x7b080024a0c0 after lost signal to thread 29454
W20260709 12:27:23.732575 29618 debug-util.cc:398] Leaking SignalData structure 0x7b0800430c40 after lost signal to thread 29621
W20260709 12:27:23.733903 29618 debug-util.cc:398] Leaking SignalData structure 0x7b08001f4860 after lost signal to thread 29695
W20260709 12:27:23.737094 29618 debug-util.cc:398] Leaking SignalData structure 0x7b0800375440 after lost signal to thread 29769
W20260709 12:27:23.738667 29618 debug-util.cc:398] Leaking SignalData structure 0x7b08002956c0 after lost signal to thread 30406
I20260709 12:27:23.742815 30613 raft_consensus.cc:993] T 4685c2e824c14fc1bcd45c16169798cc P c6610c00ffb948ce9c34f41dec3d49e8: : Instructing follower 584dac86183a41d9b3bea1c2085e62fa to start an election
I20260709 12:27:23.744813 29590 raft_consensus.cc:2468] T 01788883c8ea42989f246a3486111a7e P d73d92fb35234d16824c55689b5fce28 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 584dac86183a41d9b3bea1c2085e62fa in term 2.
I20260709 12:27:23.745187 30622 raft_consensus.cc:1081] T 4685c2e824c14fc1bcd45c16169798cc P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 LEADER]: Signalling peer 584dac86183a41d9b3bea1c2085e62fa to start an election
I20260709 12:27:23.769542 29743 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "6c8b2fa9f2c044ec9eec97c72b561042"
dest_uuid: "c6610c00ffb948ce9c34f41dec3d49e8"
mode: GRACEFUL
new_leader_uuid: "584dac86183a41d9b3bea1c2085e62fa"
 from {username='slave'} at 127.0.0.1:58802
I20260709 12:27:23.770316 29743 raft_consensus.cc:606] T 6c8b2fa9f2c044ec9eec97c72b561042 P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 LEADER]: Received request to transfer leadership to TS 584dac86183a41d9b3bea1c2085e62fa
I20260709 12:27:23.779822 29589 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "b3c02c910d2b43cc9265bb1d98491cd3"
dest_uuid: "d73d92fb35234d16824c55689b5fce28"
mode: GRACEFUL
new_leader_uuid: "584dac86183a41d9b3bea1c2085e62fa"
 from {username='slave'} at 127.0.0.1:43236
I20260709 12:27:23.780525 29589 raft_consensus.cc:606] T b3c02c910d2b43cc9265bb1d98491cd3 P d73d92fb35234d16824c55689b5fce28 [term 1 LEADER]: Received request to transfer leadership to TS 584dac86183a41d9b3bea1c2085e62fa
I20260709 12:27:23.784593 30366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb7d78e534634e0f992fc39b578ebd8d" candidate_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "584dac86183a41d9b3bea1c2085e62fa" is_pre_election: true
I20260709 12:27:23.786616 30366 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "4685c2e824c14fc1bcd45c16169798cc"
dest_uuid: "584dac86183a41d9b3bea1c2085e62fa"
 from {username='slave'} at 127.0.0.1:38910
I20260709 12:27:23.787170 30366 raft_consensus.cc:493] T 4685c2e824c14fc1bcd45c16169798cc P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:27:23.787550 30366 raft_consensus.cc:3060] T 4685c2e824c14fc1bcd45c16169798cc P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:23.794750 29589 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "fcb6a7f720b14dc98e821f663357a0bb"
dest_uuid: "d73d92fb35234d16824c55689b5fce28"
mode: GRACEFUL
new_leader_uuid: "584dac86183a41d9b3bea1c2085e62fa"
 from {username='slave'} at 127.0.0.1:43236
I20260709 12:27:23.795398 29589 raft_consensus.cc:606] T fcb6a7f720b14dc98e821f663357a0bb P d73d92fb35234d16824c55689b5fce28 [term 2 LEADER]: Received request to transfer leadership to TS 584dac86183a41d9b3bea1c2085e62fa
I20260709 12:27:23.796896 16501 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260709 12:27:23.798406 16501 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:27:23.799217 16501 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:27:23.800397 30366 raft_consensus.cc:515] T 4685c2e824c14fc1bcd45c16169798cc P 584dac86183a41d9b3bea1c2085e62fa [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:23.803078 29742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4685c2e824c14fc1bcd45c16169798cc" candidate_uuid: "584dac86183a41d9b3bea1c2085e62fa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "c6610c00ffb948ce9c34f41dec3d49e8"
I20260709 12:27:23.803236 30366 leader_election.cc:290] T 4685c2e824c14fc1bcd45c16169798cc P 584dac86183a41d9b3bea1c2085e62fa [CANDIDATE]: Term 2 election: Requested vote from peers c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895), d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825)
I20260709 12:27:23.803635 29742 raft_consensus.cc:3055] T 4685c2e824c14fc1bcd45c16169798cc P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:27:23.803970 29742 raft_consensus.cc:740] T 4685c2e824c14fc1bcd45c16169798cc P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c6610c00ffb948ce9c34f41dec3d49e8, State: Running, Role: LEADER
I20260709 12:27:23.804829 29590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4685c2e824c14fc1bcd45c16169798cc" candidate_uuid: "584dac86183a41d9b3bea1c2085e62fa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d73d92fb35234d16824c55689b5fce28"
I20260709 12:27:23.804682 29742 consensus_queue.cc:260] T 4685c2e824c14fc1bcd45c16169798cc P c6610c00ffb948ce9c34f41dec3d49e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:23.805373 29590 raft_consensus.cc:3060] T 4685c2e824c14fc1bcd45c16169798cc P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:23.805773 29742 raft_consensus.cc:3060] T 4685c2e824c14fc1bcd45c16169798cc P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:23.812181 29590 raft_consensus.cc:2468] T 4685c2e824c14fc1bcd45c16169798cc P d73d92fb35234d16824c55689b5fce28 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 584dac86183a41d9b3bea1c2085e62fa in term 2.
I20260709 12:27:23.812390 29742 raft_consensus.cc:2468] T 4685c2e824c14fc1bcd45c16169798cc P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 584dac86183a41d9b3bea1c2085e62fa in term 2.
I20260709 12:27:23.813452 30313 leader_election.cc:304] T 4685c2e824c14fc1bcd45c16169798cc P 584dac86183a41d9b3bea1c2085e62fa [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 584dac86183a41d9b3bea1c2085e62fa, d73d92fb35234d16824c55689b5fce28; no voters: 
I20260709 12:27:23.814843 30599 raft_consensus.cc:2804] T 4685c2e824c14fc1bcd45c16169798cc P 584dac86183a41d9b3bea1c2085e62fa [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:27:23.815380 30599 raft_consensus.cc:697] T 4685c2e824c14fc1bcd45c16169798cc P 584dac86183a41d9b3bea1c2085e62fa [term 2 LEADER]: Becoming Leader. State: Replica: 584dac86183a41d9b3bea1c2085e62fa, State: Running, Role: LEADER
I20260709 12:27:23.816365 30599 consensus_queue.cc:237] T 4685c2e824c14fc1bcd45c16169798cc P 584dac86183a41d9b3bea1c2085e62fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:23.822999 29590 raft_consensus.cc:1275] T d1918297b9eb41f5847a211e668ae748 P d73d92fb35234d16824c55689b5fce28 [term 2 FOLLOWER]: Refusing update from remote peer 584dac86183a41d9b3bea1c2085e62fa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:27:23.825604 30543 consensus_queue.cc:1048] T d1918297b9eb41f5847a211e668ae748 P 584dac86183a41d9b3bea1c2085e62fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:27:23.834457 29419 catalog_manager.cc:5697] T 4685c2e824c14fc1bcd45c16169798cc P 584dac86183a41d9b3bea1c2085e62fa reported cstate change: term changed from 1 to 2, leader changed from c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68) to 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65). New cstate: current_term: 2 leader_uuid: "584dac86183a41d9b3bea1c2085e62fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:23.836131 29741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "137e03144e754f6e9da4f0544513d27c" candidate_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" is_pre_election: true
W20260709 12:27:23.859974 29709 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:23.863104 30592 raft_consensus.cc:993] T 58154d49c1a344f2b51366ede3c87de5 P d73d92fb35234d16824c55689b5fce28: : Instructing follower c6610c00ffb948ce9c34f41dec3d49e8 to start an election
I20260709 12:27:23.863811 30592 raft_consensus.cc:1081] T 58154d49c1a344f2b51366ede3c87de5 P d73d92fb35234d16824c55689b5fce28 [term 1 LEADER]: Signalling peer c6610c00ffb948ce9c34f41dec3d49e8 to start an election
I20260709 12:27:23.863168 29671 raft_consensus.cc:1275] T d1918297b9eb41f5847a211e668ae748 P d13c58ad8ac441c4a9b396eb04b1a476 [term 2 FOLLOWER]: Refusing update from remote peer 584dac86183a41d9b3bea1c2085e62fa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:27:23.869531 29708 leader_election.cc:304] T fb7d78e534634e0f992fc39b578ebd8d P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c6610c00ffb948ce9c34f41dec3d49e8; no voters: 584dac86183a41d9b3bea1c2085e62fa, d13c58ad8ac441c4a9b396eb04b1a476
I20260709 12:27:23.869693 29635 leader_election.cc:304] T 137e03144e754f6e9da4f0544513d27c P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d13c58ad8ac441c4a9b396eb04b1a476; no voters: c6610c00ffb948ce9c34f41dec3d49e8, d73d92fb35234d16824c55689b5fce28
I20260709 12:27:23.872124 30609 raft_consensus.cc:2749] T 137e03144e754f6e9da4f0544513d27c P d13c58ad8ac441c4a9b396eb04b1a476 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:27:23.872887 30611 raft_consensus.cc:2749] T fb7d78e534634e0f992fc39b578ebd8d P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:27:23.876684 30543 consensus_queue.cc:1048] T d1918297b9eb41f5847a211e668ae748 P 584dac86183a41d9b3bea1c2085e62fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:27:23.883893 29741 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "58154d49c1a344f2b51366ede3c87de5"
dest_uuid: "c6610c00ffb948ce9c34f41dec3d49e8"
 from {username='slave'} at 127.0.0.1:33144
I20260709 12:27:23.884640 29741 raft_consensus.cc:493] T 58154d49c1a344f2b51366ede3c87de5 P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:27:23.885113 29741 raft_consensus.cc:3060] T 58154d49c1a344f2b51366ede3c87de5 P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:24.001185 30498 raft_consensus.cc:993] T fcb6a7f720b14dc98e821f663357a0bb P d73d92fb35234d16824c55689b5fce28: : Instructing follower 584dac86183a41d9b3bea1c2085e62fa to start an election
I20260709 12:27:24.001848 30498 raft_consensus.cc:1081] T fcb6a7f720b14dc98e821f663357a0bb P d73d92fb35234d16824c55689b5fce28 [term 2 LEADER]: Signalling peer 584dac86183a41d9b3bea1c2085e62fa to start an election
I20260709 12:27:24.010281 30366 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "fcb6a7f720b14dc98e821f663357a0bb"
dest_uuid: "584dac86183a41d9b3bea1c2085e62fa"
 from {username='slave'} at 127.0.0.1:38894
I20260709 12:27:24.011034 30366 raft_consensus.cc:493] T fcb6a7f720b14dc98e821f663357a0bb P 584dac86183a41d9b3bea1c2085e62fa [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:27:24.011471 30366 raft_consensus.cc:3060] T fcb6a7f720b14dc98e821f663357a0bb P 584dac86183a41d9b3bea1c2085e62fa [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:27:24.045819 29741 raft_consensus.cc:515] T 58154d49c1a344f2b51366ede3c87de5 P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:27:24.074299 29671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58154d49c1a344f2b51366ede3c87de5" candidate_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d13c58ad8ac441c4a9b396eb04b1a476"
I20260709 12:27:24.075122 29671 raft_consensus.cc:3060] T 58154d49c1a344f2b51366ede3c87de5 P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:24.084821 30596 raft_consensus.cc:993] T b3c02c910d2b43cc9265bb1d98491cd3 P d73d92fb35234d16824c55689b5fce28: : Instructing follower 584dac86183a41d9b3bea1c2085e62fa to start an election
I20260709 12:27:24.085418 30498 raft_consensus.cc:1081] T b3c02c910d2b43cc9265bb1d98491cd3 P d73d92fb35234d16824c55689b5fce28 [term 1 LEADER]: Signalling peer 584dac86183a41d9b3bea1c2085e62fa to start an election
I20260709 12:27:24.087491 30368 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "b3c02c910d2b43cc9265bb1d98491cd3"
dest_uuid: "584dac86183a41d9b3bea1c2085e62fa"
 from {username='slave'} at 127.0.0.1:38894
I20260709 12:27:24.089123 30368 raft_consensus.cc:493] T b3c02c910d2b43cc9265bb1d98491cd3 P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:27:24.089689 30368 raft_consensus.cc:3060] T b3c02c910d2b43cc9265bb1d98491cd3 P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:24.107152 29671 raft_consensus.cc:2468] T 58154d49c1a344f2b51366ede3c87de5 P d13c58ad8ac441c4a9b396eb04b1a476 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6610c00ffb948ce9c34f41dec3d49e8 in term 2.
I20260709 12:27:24.123381 29741 leader_election.cc:290] T 58154d49c1a344f2b51366ede3c87de5 P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 2 election: Requested vote from peers d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825), d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209)
I20260709 12:27:24.141884 29709 leader_election.cc:304] T 58154d49c1a344f2b51366ede3c87de5 P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c6610c00ffb948ce9c34f41dec3d49e8, d13c58ad8ac441c4a9b396eb04b1a476; no voters: 
I20260709 12:27:24.142735 30610 raft_consensus.cc:2804] T 58154d49c1a344f2b51366ede3c87de5 P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:27:24.143224 30610 raft_consensus.cc:697] T 58154d49c1a344f2b51366ede3c87de5 P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 LEADER]: Becoming Leader. State: Replica: c6610c00ffb948ce9c34f41dec3d49e8, State: Running, Role: LEADER
I20260709 12:27:24.144130 30610 consensus_queue.cc:237] T 58154d49c1a344f2b51366ede3c87de5 P c6610c00ffb948ce9c34f41dec3d49e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:27:24.125746 30366 raft_consensus.cc:515] T fcb6a7f720b14dc98e821f663357a0bb P 584dac86183a41d9b3bea1c2085e62fa [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:27:24.146883 29590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fcb6a7f720b14dc98e821f663357a0bb" candidate_uuid: "584dac86183a41d9b3bea1c2085e62fa" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "d73d92fb35234d16824c55689b5fce28"
I20260709 12:27:24.147570 29590 raft_consensus.cc:3055] T fcb6a7f720b14dc98e821f663357a0bb P d73d92fb35234d16824c55689b5fce28 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:27:24.147919 29590 raft_consensus.cc:740] T fcb6a7f720b14dc98e821f663357a0bb P d73d92fb35234d16824c55689b5fce28 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d73d92fb35234d16824c55689b5fce28, State: Running, Role: LEADER
I20260709 12:27:24.148836 29590 consensus_queue.cc:260] T fcb6a7f720b14dc98e821f663357a0bb P d73d92fb35234d16824c55689b5fce28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:27:24.149946 29590 raft_consensus.cc:3060] T fcb6a7f720b14dc98e821f663357a0bb P d73d92fb35234d16824c55689b5fce28 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:27:24.152637 29589 raft_consensus.cc:1275] T 01788883c8ea42989f246a3486111a7e P d73d92fb35234d16824c55689b5fce28 [term 2 FOLLOWER]: Refusing update from remote peer 584dac86183a41d9b3bea1c2085e62fa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:27:24.154170 30543 consensus_queue.cc:1048] T 01788883c8ea42989f246a3486111a7e P 584dac86183a41d9b3bea1c2085e62fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:27:24.182649 29591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58154d49c1a344f2b51366ede3c87de5" candidate_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d73d92fb35234d16824c55689b5fce28"
I20260709 12:27:24.183326 29591 raft_consensus.cc:3055] T 58154d49c1a344f2b51366ede3c87de5 P d73d92fb35234d16824c55689b5fce28 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:27:24.183614 29591 raft_consensus.cc:740] T 58154d49c1a344f2b51366ede3c87de5 P d73d92fb35234d16824c55689b5fce28 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d73d92fb35234d16824c55689b5fce28, State: Running, Role: LEADER
I20260709 12:27:24.187225 29591 consensus_queue.cc:260] T 58154d49c1a344f2b51366ede3c87de5 P d73d92fb35234d16824c55689b5fce28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:27:24.188862 29591 raft_consensus.cc:3060] T 58154d49c1a344f2b51366ede3c87de5 P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:24.190826 30366 leader_election.cc:290] T fcb6a7f720b14dc98e821f663357a0bb P 584dac86183a41d9b3bea1c2085e62fa [CANDIDATE]: Term 3 election: Requested vote from peers d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825), d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209)
I20260709 12:27:24.191746 29666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fcb6a7f720b14dc98e821f663357a0bb" candidate_uuid: "584dac86183a41d9b3bea1c2085e62fa" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "d13c58ad8ac441c4a9b396eb04b1a476"
I20260709 12:27:24.192317 29666 raft_consensus.cc:3060] T fcb6a7f720b14dc98e821f663357a0bb P d13c58ad8ac441c4a9b396eb04b1a476 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:27:24.133662 30368 raft_consensus.cc:515] T b3c02c910d2b43cc9265bb1d98491cd3 P 584dac86183a41d9b3bea1c2085e62fa [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:24.138803 29742 raft_consensus.cc:1275] T 01788883c8ea42989f246a3486111a7e P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Refusing update from remote peer 584dac86183a41d9b3bea1c2085e62fa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:27:24.195852 29590 raft_consensus.cc:2468] T fcb6a7f720b14dc98e821f663357a0bb P d73d92fb35234d16824c55689b5fce28 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 584dac86183a41d9b3bea1c2085e62fa in term 3.
I20260709 12:27:24.197532 30368 leader_election.cc:290] T b3c02c910d2b43cc9265bb1d98491cd3 P 584dac86183a41d9b3bea1c2085e62fa [CANDIDATE]: Term 2 election: Requested vote from peers c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895), d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825)
I20260709 12:27:24.197808 30542 consensus_queue.cc:1048] T 01788883c8ea42989f246a3486111a7e P 584dac86183a41d9b3bea1c2085e62fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.004s
I20260709 12:27:24.198591 29742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3c02c910d2b43cc9265bb1d98491cd3" candidate_uuid: "584dac86183a41d9b3bea1c2085e62fa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "c6610c00ffb948ce9c34f41dec3d49e8"
I20260709 12:27:24.203842 29742 raft_consensus.cc:3060] T b3c02c910d2b43cc9265bb1d98491cd3 P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:24.203923 29591 raft_consensus.cc:2468] T 58154d49c1a344f2b51366ede3c87de5 P d73d92fb35234d16824c55689b5fce28 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6610c00ffb948ce9c34f41dec3d49e8 in term 2.
I20260709 12:27:24.215194 29666 raft_consensus.cc:2468] T fcb6a7f720b14dc98e821f663357a0bb P d13c58ad8ac441c4a9b396eb04b1a476 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 584dac86183a41d9b3bea1c2085e62fa in term 3.
I20260709 12:27:24.223855 29591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3c02c910d2b43cc9265bb1d98491cd3" candidate_uuid: "584dac86183a41d9b3bea1c2085e62fa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d73d92fb35234d16824c55689b5fce28"
I20260709 12:27:24.241065 29591 raft_consensus.cc:3055] T b3c02c910d2b43cc9265bb1d98491cd3 P d73d92fb35234d16824c55689b5fce28 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:27:24.225407 30313 leader_election.cc:304] T fcb6a7f720b14dc98e821f663357a0bb P 584dac86183a41d9b3bea1c2085e62fa [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 584dac86183a41d9b3bea1c2085e62fa, d73d92fb35234d16824c55689b5fce28; no voters: 
I20260709 12:27:24.232931 29742 raft_consensus.cc:2468] T b3c02c910d2b43cc9265bb1d98491cd3 P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 584dac86183a41d9b3bea1c2085e62fa in term 2.
I20260709 12:27:24.241453 29591 raft_consensus.cc:740] T b3c02c910d2b43cc9265bb1d98491cd3 P d73d92fb35234d16824c55689b5fce28 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d73d92fb35234d16824c55689b5fce28, State: Running, Role: LEADER
I20260709 12:27:24.242082 29744 raft_consensus.cc:1240] T b3c02c910d2b43cc9265bb1d98491cd3 P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Rejecting Update request from peer d73d92fb35234d16824c55689b5fce28 for earlier term 1. Current term is 2. Ops: []
I20260709 12:27:24.242357 29591 consensus_queue.cc:260] T b3c02c910d2b43cc9265bb1d98491cd3 P d73d92fb35234d16824c55689b5fce28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:24.243721 29591 raft_consensus.cc:3060] T b3c02c910d2b43cc9265bb1d98491cd3 P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:24.244314 30314 leader_election.cc:304] T b3c02c910d2b43cc9265bb1d98491cd3 P 584dac86183a41d9b3bea1c2085e62fa [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 584dac86183a41d9b3bea1c2085e62fa, c6610c00ffb948ce9c34f41dec3d49e8; no voters: 
I20260709 12:27:24.245182 30542 raft_consensus.cc:2804] T b3c02c910d2b43cc9265bb1d98491cd3 P 584dac86183a41d9b3bea1c2085e62fa [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:27:24.246023 30542 raft_consensus.cc:697] T b3c02c910d2b43cc9265bb1d98491cd3 P 584dac86183a41d9b3bea1c2085e62fa [term 2 LEADER]: Becoming Leader. State: Replica: 584dac86183a41d9b3bea1c2085e62fa, State: Running, Role: LEADER
I20260709 12:27:24.249662 30542 consensus_queue.cc:237] T b3c02c910d2b43cc9265bb1d98491cd3 P 584dac86183a41d9b3bea1c2085e62fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:24.253605 30600 raft_consensus.cc:2804] T fcb6a7f720b14dc98e821f663357a0bb P 584dac86183a41d9b3bea1c2085e62fa [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:27:24.266527 30600 raft_consensus.cc:697] T fcb6a7f720b14dc98e821f663357a0bb P 584dac86183a41d9b3bea1c2085e62fa [term 3 LEADER]: Becoming Leader. State: Replica: 584dac86183a41d9b3bea1c2085e62fa, State: Running, Role: LEADER
I20260709 12:27:24.267470 30600 consensus_queue.cc:237] T fcb6a7f720b14dc98e821f663357a0bb P 584dac86183a41d9b3bea1c2085e62fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:27:24.253245 29591 raft_consensus.cc:2468] T b3c02c910d2b43cc9265bb1d98491cd3 P d73d92fb35234d16824c55689b5fce28 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 584dac86183a41d9b3bea1c2085e62fa in term 2.
W20260709 12:27:24.308715 29708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:36825, user_credentials={real_user=slave}} blocked reactor thread for 48532.9us
I20260709 12:27:24.334837 29419 catalog_manager.cc:5697] T 58154d49c1a344f2b51366ede3c87de5 P c6610c00ffb948ce9c34f41dec3d49e8 reported cstate change: term changed from 1 to 2, leader changed from d73d92fb35234d16824c55689b5fce28 (127.16.29.66) to c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68). New cstate: current_term: 2 leader_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } health_report { overall_health: HEALTHY } } }
I20260709 12:27:24.342914 30611 raft_consensus.cc:993] T 6c8b2fa9f2c044ec9eec97c72b561042 P c6610c00ffb948ce9c34f41dec3d49e8: : Instructing follower 584dac86183a41d9b3bea1c2085e62fa to start an election
I20260709 12:27:24.344745 30611 raft_consensus.cc:1081] T 6c8b2fa9f2c044ec9eec97c72b561042 P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 LEADER]: Signalling peer 584dac86183a41d9b3bea1c2085e62fa to start an election
I20260709 12:27:24.359772 30366 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "6c8b2fa9f2c044ec9eec97c72b561042"
dest_uuid: "584dac86183a41d9b3bea1c2085e62fa"
 from {username='slave'} at 127.0.0.1:38910
I20260709 12:27:24.360512 30366 raft_consensus.cc:493] T 6c8b2fa9f2c044ec9eec97c72b561042 P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:27:24.360903 30366 raft_consensus.cc:3060] T 6c8b2fa9f2c044ec9eec97c72b561042 P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:24.364866 29744 raft_consensus.cc:1275] T 4685c2e824c14fc1bcd45c16169798cc P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Refusing update from remote peer 584dac86183a41d9b3bea1c2085e62fa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:27:24.366811 30542 consensus_queue.cc:1048] T 4685c2e824c14fc1bcd45c16169798cc P 584dac86183a41d9b3bea1c2085e62fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.007s
I20260709 12:27:24.374341 30366 raft_consensus.cc:515] T 6c8b2fa9f2c044ec9eec97c72b561042 P 584dac86183a41d9b3bea1c2085e62fa [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:24.376590 30366 leader_election.cc:290] T 6c8b2fa9f2c044ec9eec97c72b561042 P 584dac86183a41d9b3bea1c2085e62fa [CANDIDATE]: Term 2 election: Requested vote from peers c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895), d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825)
I20260709 12:27:24.377247 29744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c8b2fa9f2c044ec9eec97c72b561042" candidate_uuid: "584dac86183a41d9b3bea1c2085e62fa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "c6610c00ffb948ce9c34f41dec3d49e8"
I20260709 12:27:24.377842 29744 raft_consensus.cc:3055] T 6c8b2fa9f2c044ec9eec97c72b561042 P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:27:24.378127 29744 raft_consensus.cc:740] T 6c8b2fa9f2c044ec9eec97c72b561042 P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c6610c00ffb948ce9c34f41dec3d49e8, State: Running, Role: LEADER
I20260709 12:27:24.378153 29590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c8b2fa9f2c044ec9eec97c72b561042" candidate_uuid: "584dac86183a41d9b3bea1c2085e62fa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d73d92fb35234d16824c55689b5fce28"
I20260709 12:27:24.378895 29590 raft_consensus.cc:3060] T 6c8b2fa9f2c044ec9eec97c72b561042 P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:24.378880 29744 consensus_queue.cc:260] T 6c8b2fa9f2c044ec9eec97c72b561042 P c6610c00ffb948ce9c34f41dec3d49e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:24.380151 29744 raft_consensus.cc:3060] T 6c8b2fa9f2c044ec9eec97c72b561042 P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:24.387336 29744 raft_consensus.cc:2468] T 6c8b2fa9f2c044ec9eec97c72b561042 P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 584dac86183a41d9b3bea1c2085e62fa in term 2.
I20260709 12:27:24.389106 30314 leader_election.cc:304] T 6c8b2fa9f2c044ec9eec97c72b561042 P 584dac86183a41d9b3bea1c2085e62fa [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 584dac86183a41d9b3bea1c2085e62fa, c6610c00ffb948ce9c34f41dec3d49e8; no voters: 
I20260709 12:27:24.390749 30542 raft_consensus.cc:2804] T 6c8b2fa9f2c044ec9eec97c72b561042 P 584dac86183a41d9b3bea1c2085e62fa [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:27:24.391291 30542 raft_consensus.cc:697] T 6c8b2fa9f2c044ec9eec97c72b561042 P 584dac86183a41d9b3bea1c2085e62fa [term 2 LEADER]: Becoming Leader. State: Replica: 584dac86183a41d9b3bea1c2085e62fa, State: Running, Role: LEADER
I20260709 12:27:24.392277 30542 consensus_queue.cc:237] T 6c8b2fa9f2c044ec9eec97c72b561042 P 584dac86183a41d9b3bea1c2085e62fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:24.393649 29590 raft_consensus.cc:2468] T 6c8b2fa9f2c044ec9eec97c72b561042 P d73d92fb35234d16824c55689b5fce28 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 584dac86183a41d9b3bea1c2085e62fa in term 2.
I20260709 12:27:24.404081 29590 raft_consensus.cc:1275] T 4685c2e824c14fc1bcd45c16169798cc P d73d92fb35234d16824c55689b5fce28 [term 2 FOLLOWER]: Refusing update from remote peer 584dac86183a41d9b3bea1c2085e62fa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:27:24.405467 30600 consensus_queue.cc:1048] T 4685c2e824c14fc1bcd45c16169798cc P 584dac86183a41d9b3bea1c2085e62fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:27:24.519604 29417 catalog_manager.cc:5697] T b3c02c910d2b43cc9265bb1d98491cd3 P 584dac86183a41d9b3bea1c2085e62fa reported cstate change: term changed from 1 to 2, leader changed from d73d92fb35234d16824c55689b5fce28 (127.16.29.66) to 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65). New cstate: current_term: 2 leader_uuid: "584dac86183a41d9b3bea1c2085e62fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:24.521243 29417 catalog_manager.cc:5697] T fcb6a7f720b14dc98e821f663357a0bb P 584dac86183a41d9b3bea1c2085e62fa reported cstate change: term changed from 2 to 3, leader changed from d73d92fb35234d16824c55689b5fce28 (127.16.29.66) to 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65). New cstate: current_term: 3 leader_uuid: "584dac86183a41d9b3bea1c2085e62fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:24.558228 29417 catalog_manager.cc:5697] T 6c8b2fa9f2c044ec9eec97c72b561042 P 584dac86183a41d9b3bea1c2085e62fa reported cstate change: term changed from 1 to 2, leader changed from c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68) to 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65). New cstate: current_term: 2 leader_uuid: "584dac86183a41d9b3bea1c2085e62fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:24.676469 29666 raft_consensus.cc:1275] T fcb6a7f720b14dc98e821f663357a0bb P d13c58ad8ac441c4a9b396eb04b1a476 [term 3 FOLLOWER]: Refusing update from remote peer 584dac86183a41d9b3bea1c2085e62fa: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:27:24.678079 30542 consensus_queue.cc:1048] T fcb6a7f720b14dc98e821f663357a0bb P 584dac86183a41d9b3bea1c2085e62fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:27:24.683562 29671 raft_consensus.cc:1275] T 58154d49c1a344f2b51366ede3c87de5 P d13c58ad8ac441c4a9b396eb04b1a476 [term 2 FOLLOWER]: Refusing update from remote peer c6610c00ffb948ce9c34f41dec3d49e8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:27:24.685158 30610 consensus_queue.cc:1048] T 58154d49c1a344f2b51366ede3c87de5 P c6610c00ffb948ce9c34f41dec3d49e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:27:24.697170 29590 raft_consensus.cc:1275] T 58154d49c1a344f2b51366ede3c87de5 P d73d92fb35234d16824c55689b5fce28 [term 2 FOLLOWER]: Refusing update from remote peer c6610c00ffb948ce9c34f41dec3d49e8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:27:24.702858 30631 consensus_queue.cc:1048] T 58154d49c1a344f2b51366ede3c87de5 P c6610c00ffb948ce9c34f41dec3d49e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:27:24.780921 29591 raft_consensus.cc:1275] T b3c02c910d2b43cc9265bb1d98491cd3 P d73d92fb35234d16824c55689b5fce28 [term 2 FOLLOWER]: Refusing update from remote peer 584dac86183a41d9b3bea1c2085e62fa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:27:24.782678 30542 consensus_queue.cc:1048] T b3c02c910d2b43cc9265bb1d98491cd3 P 584dac86183a41d9b3bea1c2085e62fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:27:24.783546 29742 raft_consensus.cc:1275] T b3c02c910d2b43cc9265bb1d98491cd3 P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Refusing update from remote peer 584dac86183a41d9b3bea1c2085e62fa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:27:24.795274 30542 consensus_queue.cc:1048] T b3c02c910d2b43cc9265bb1d98491cd3 P 584dac86183a41d9b3bea1c2085e62fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:27:24.834656 29590 raft_consensus.cc:1275] T fcb6a7f720b14dc98e821f663357a0bb P d73d92fb35234d16824c55689b5fce28 [term 3 FOLLOWER]: Refusing update from remote peer 584dac86183a41d9b3bea1c2085e62fa: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:27:24.836786 30601 consensus_queue.cc:1048] T fcb6a7f720b14dc98e821f663357a0bb P 584dac86183a41d9b3bea1c2085e62fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:27:24.844937 29744 raft_consensus.cc:1275] T 6c8b2fa9f2c044ec9eec97c72b561042 P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Refusing update from remote peer 584dac86183a41d9b3bea1c2085e62fa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:27:24.850256 16501 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:27:24.879798 30601 consensus_queue.cc:1048] T 6c8b2fa9f2c044ec9eec97c72b561042 P 584dac86183a41d9b3bea1c2085e62fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260709 12:27:24.970594 29560 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:35033, user_credentials={real_user=slave}} blocked reactor thread for 48691.4us
I20260709 12:27:24.978510 29597 raft_consensus.cc:1275] T 6c8b2fa9f2c044ec9eec97c72b561042 P d73d92fb35234d16824c55689b5fce28 [term 2 FOLLOWER]: Refusing update from remote peer 584dac86183a41d9b3bea1c2085e62fa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:27:24.985719 30542 consensus_queue.cc:1048] T 6c8b2fa9f2c044ec9eec97c72b561042 P 584dac86183a41d9b3bea1c2085e62fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260709 12:27:24.989850 30653 log.cc:927] Time spent T b3c02c910d2b43cc9265bb1d98491cd3 P c6610c00ffb948ce9c34f41dec3d49e8: Append to log took a long time: real 0.145s	user 0.001s	sys 0.000s
I20260709 12:27:25.172582 29742 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "c1c8411289a74b5685700229d245ff75"
dest_uuid: "c6610c00ffb948ce9c34f41dec3d49e8"
mode: GRACEFUL
new_leader_uuid: "d73d92fb35234d16824c55689b5fce28"
 from {username='slave'} at 127.0.0.1:58802
I20260709 12:27:25.173192 29742 raft_consensus.cc:606] T c1c8411289a74b5685700229d245ff75 P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 LEADER]: Received request to transfer leadership to TS d73d92fb35234d16824c55689b5fce28
I20260709 12:27:25.175648 16501 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 1
I20260709 12:27:25.177292 16501 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:27:25.177982 16501 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:27:25.353725 30632 raft_consensus.cc:993] T c1c8411289a74b5685700229d245ff75 P c6610c00ffb948ce9c34f41dec3d49e8: : Instructing follower d73d92fb35234d16824c55689b5fce28 to start an election
I20260709 12:27:25.354429 30632 raft_consensus.cc:1081] T c1c8411289a74b5685700229d245ff75 P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 LEADER]: Signalling peer d73d92fb35234d16824c55689b5fce28 to start an election
I20260709 12:27:25.368404 29597 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "c1c8411289a74b5685700229d245ff75"
dest_uuid: "d73d92fb35234d16824c55689b5fce28"
 from {username='slave'} at 127.0.0.1:50724
I20260709 12:27:25.369079 29597 raft_consensus.cc:493] T c1c8411289a74b5685700229d245ff75 P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:27:25.369496 29597 raft_consensus.cc:3060] T c1c8411289a74b5685700229d245ff75 P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:25.377538 29597 raft_consensus.cc:515] T c1c8411289a74b5685700229d245ff75 P d73d92fb35234d16824c55689b5fce28 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:27:25.380016 29597 leader_election.cc:290] T c1c8411289a74b5685700229d245ff75 P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 2 election: Requested vote from peers d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209), c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895)
I20260709 12:27:25.383424 29666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c1c8411289a74b5685700229d245ff75" candidate_uuid: "d73d92fb35234d16824c55689b5fce28" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d13c58ad8ac441c4a9b396eb04b1a476"
I20260709 12:27:25.384166 29666 raft_consensus.cc:3060] T c1c8411289a74b5685700229d245ff75 P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:25.387123 29742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c1c8411289a74b5685700229d245ff75" candidate_uuid: "d73d92fb35234d16824c55689b5fce28" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "c6610c00ffb948ce9c34f41dec3d49e8"
I20260709 12:27:25.387773 29742 raft_consensus.cc:3055] T c1c8411289a74b5685700229d245ff75 P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:27:25.388110 29742 raft_consensus.cc:740] T c1c8411289a74b5685700229d245ff75 P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c6610c00ffb948ce9c34f41dec3d49e8, State: Running, Role: LEADER
I20260709 12:27:25.388967 29742 consensus_queue.cc:260] T c1c8411289a74b5685700229d245ff75 P c6610c00ffb948ce9c34f41dec3d49e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:27:25.391479 29666 raft_consensus.cc:2468] T c1c8411289a74b5685700229d245ff75 P d13c58ad8ac441c4a9b396eb04b1a476 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d73d92fb35234d16824c55689b5fce28 in term 2.
I20260709 12:27:25.390012 29742 raft_consensus.cc:3060] T c1c8411289a74b5685700229d245ff75 P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:25.396531 29561 leader_election.cc:304] T c1c8411289a74b5685700229d245ff75 P d73d92fb35234d16824c55689b5fce28 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d13c58ad8ac441c4a9b396eb04b1a476, d73d92fb35234d16824c55689b5fce28; no voters: 
I20260709 12:27:25.398383 30657 raft_consensus.cc:2804] T c1c8411289a74b5685700229d245ff75 P d73d92fb35234d16824c55689b5fce28 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:27:25.398931 30657 raft_consensus.cc:697] T c1c8411289a74b5685700229d245ff75 P d73d92fb35234d16824c55689b5fce28 [term 2 LEADER]: Becoming Leader. State: Replica: d73d92fb35234d16824c55689b5fce28, State: Running, Role: LEADER
I20260709 12:27:25.399953 30657 consensus_queue.cc:237] T c1c8411289a74b5685700229d245ff75 P d73d92fb35234d16824c55689b5fce28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:27:25.492586 29742 raft_consensus.cc:2468] T c1c8411289a74b5685700229d245ff75 P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d73d92fb35234d16824c55689b5fce28 in term 2.
I20260709 12:27:25.498291 29418 catalog_manager.cc:5697] T c1c8411289a74b5685700229d245ff75 P d73d92fb35234d16824c55689b5fce28 reported cstate change: term changed from 1 to 2, leader changed from c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68) to d73d92fb35234d16824c55689b5fce28 (127.16.29.66). New cstate: current_term: 2 leader_uuid: "d73d92fb35234d16824c55689b5fce28" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:25.782931 29743 raft_consensus.cc:1275] T c1c8411289a74b5685700229d245ff75 P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Refusing update from remote peer d73d92fb35234d16824c55689b5fce28: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:27:25.786041 30657 consensus_queue.cc:1048] T c1c8411289a74b5685700229d245ff75 P d73d92fb35234d16824c55689b5fce28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:27:25.812494 29666 raft_consensus.cc:1275] T c1c8411289a74b5685700229d245ff75 P d13c58ad8ac441c4a9b396eb04b1a476 [term 2 FOLLOWER]: Refusing update from remote peer d73d92fb35234d16824c55689b5fce28: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:27:25.814703 30657 consensus_queue.cc:1048] T c1c8411289a74b5685700229d245ff75 P d73d92fb35234d16824c55689b5fce28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:27:26.263770 16501 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:27:26.337867 16501 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:27:26.340070 16501 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:27:26.340868 16501 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
W20260709 12:27:26.380267 30314 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:38209, user_credentials={real_user=slave}} blocked reactor thread for 58623.7us
I20260709 12:27:26.441495 16501 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:27:26.525395 16501 tablet_server.cc:179] TabletServer@127.16.29.65:35033 shutting down...
W20260709 12:27:26.539912 29634 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 125 similar messages]
W20260709 12:27:26.592054 29560 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:35033, user_credentials={real_user=slave}} blocked reactor thread for 44157us
W20260709 12:27:26.596539 29634 consensus_peers.cc:597] T 7ae3b542012d4f35a9c363eee22976ec P d13c58ad8ac441c4a9b396eb04b1a476 -> Peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Couldn't send request to peer 584dac86183a41d9b3bea1c2085e62fa. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:26.598089 29634 consensus_peers.cc:597] T 6e37d72807e547898f4fce9e8b82b2b0 P d13c58ad8ac441c4a9b396eb04b1a476 -> Peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Couldn't send request to peer 584dac86183a41d9b3bea1c2085e62fa. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:26.599383 29634 consensus_peers.cc:597] T c6e621aa77d5474580dfc5949801ccc9 P d13c58ad8ac441c4a9b396eb04b1a476 -> Peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Couldn't send request to peer 584dac86183a41d9b3bea1c2085e62fa. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:26.602803 29708 consensus_peers.cc:597] T 66f652ba2c224885a2aea974bfe8f88a P c6610c00ffb948ce9c34f41dec3d49e8 -> Peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Couldn't send request to peer 584dac86183a41d9b3bea1c2085e62fa. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:26.618348 29560 consensus_peers.cc:597] T d21ab1641501475b829863275dc8182a P d73d92fb35234d16824c55689b5fce28 -> Peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Couldn't send request to peer 584dac86183a41d9b3bea1c2085e62fa. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:26.619350 29634 consensus_peers.cc:597] T fb7d78e534634e0f992fc39b578ebd8d P d13c58ad8ac441c4a9b396eb04b1a476 -> Peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Couldn't send request to peer 584dac86183a41d9b3bea1c2085e62fa. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:26.620076 29560 consensus_peers.cc:597] T abec627108a84083884e64e1eb9dcda6 P d73d92fb35234d16824c55689b5fce28 -> Peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Couldn't send request to peer 584dac86183a41d9b3bea1c2085e62fa. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:26.624183 29560 consensus_peers.cc:597] T d5fcd1f030614bbe82cf1dc261dfc1c3 P d73d92fb35234d16824c55689b5fce28 -> Peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Couldn't send request to peer 584dac86183a41d9b3bea1c2085e62fa. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:26.629312 29708 consensus_peers.cc:597] T a46289e8fc824dd5a818e30df6e7ec3e P c6610c00ffb948ce9c34f41dec3d49e8 -> Peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Couldn't send request to peer 584dac86183a41d9b3bea1c2085e62fa. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:26.635453 29560 consensus_peers.cc:597] T a3dcbb9c64bf4b5bb6680e8cb81af93e P d73d92fb35234d16824c55689b5fce28 -> Peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Couldn't send request to peer 584dac86183a41d9b3bea1c2085e62fa. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:26.652868 29634 consensus_peers.cc:597] T d42259e436d241b29fb4ded16d9b6455 P d13c58ad8ac441c4a9b396eb04b1a476 -> Peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Couldn't send request to peer 584dac86183a41d9b3bea1c2085e62fa. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:26.659423 29708 consensus_peers.cc:597] T 8627b58b4f42491597125eb45f8d3b61 P c6610c00ffb948ce9c34f41dec3d49e8 -> Peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Couldn't send request to peer 584dac86183a41d9b3bea1c2085e62fa. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:26.661554 29634 consensus_peers.cc:597] T 3e9451c3797c4496a66dfdb3e9fb5e38 P d13c58ad8ac441c4a9b396eb04b1a476 -> Peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Couldn't send request to peer 584dac86183a41d9b3bea1c2085e62fa. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:26.670387 29634 consensus_peers.cc:597] T f3e17f0a8fc14095850e8138fd79b86e P d13c58ad8ac441c4a9b396eb04b1a476 -> Peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Couldn't send request to peer 584dac86183a41d9b3bea1c2085e62fa. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:26.671043 29634 consensus_peers.cc:597] T 31a190e9f5ff407fab90502fb3197b70 P d13c58ad8ac441c4a9b396eb04b1a476 -> Peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Couldn't send request to peer 584dac86183a41d9b3bea1c2085e62fa. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:26.672171 16501 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:27:26.676719 16501 tablet_replica.cc:333] stopping tablet replica
W20260709 12:27:26.676761 29560 consensus_peers.cc:597] T 16649fd17d434e3c95e9d585d2404659 P d73d92fb35234d16824c55689b5fce28 -> Peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Couldn't send request to peer 584dac86183a41d9b3bea1c2085e62fa. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:26.678081 16501 raft_consensus.cc:2243] T 9633e3b1d78f403097e4a00b7d4b869b P 584dac86183a41d9b3bea1c2085e62fa [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:27:26.680770 16501 raft_consensus.cc:2272] T 9633e3b1d78f403097e4a00b7d4b869b P 584dac86183a41d9b3bea1c2085e62fa [term 2 FOLLOWER]: Raft consensus is shut down!
W20260709 12:27:26.684219 29708 consensus_peers.cc:597] T 742beffede4f407d8a72909d2b147e6d P c6610c00ffb948ce9c34f41dec3d49e8 -> Peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Couldn't send request to peer 584dac86183a41d9b3bea1c2085e62fa. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:26.684688 16501 tablet_replica.cc:333] stopping tablet replica
I20260709 12:27:26.686064 16501 raft_consensus.cc:2243] T 01788883c8ea42989f246a3486111a7e P 584dac86183a41d9b3bea1c2085e62fa [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:27:26.687285 16501 raft_consensus.cc:2272] T 01788883c8ea42989f246a3486111a7e P 584dac86183a41d9b3bea1c2085e62fa [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:26.691035 16501 tablet_replica.cc:333] stopping tablet replica
I20260709 12:27:26.692180 16501 raft_consensus.cc:2243] T f76101eb30f14b98a9aa8c41e0d6f831 P 584dac86183a41d9b3bea1c2085e62fa [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:27:26.693271 16501 raft_consensus.cc:2272] T f76101eb30f14b98a9aa8c41e0d6f831 P 584dac86183a41d9b3bea1c2085e62fa [term 2 FOLLOWER]: Raft consensus is shut down!
W20260709 12:27:26.694299 29708 consensus_peers.cc:597] T ef20d7124e7a44aa97a219ee194f4b8b P c6610c00ffb948ce9c34f41dec3d49e8 -> Peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Couldn't send request to peer 584dac86183a41d9b3bea1c2085e62fa. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:26.696467 16501 tablet_replica.cc:333] stopping tablet replica
I20260709 12:27:26.697510 16501 raft_consensus.cc:2243] T 16649fd17d434e3c95e9d585d2404659 P 584dac86183a41d9b3bea1c2085e62fa [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:26.698221 16501 raft_consensus.cc:2272] T 16649fd17d434e3c95e9d585d2404659 P 584dac86183a41d9b3bea1c2085e62fa [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:26.701884 16501 tablet_replica.cc:333] stopping tablet replica
I20260709 12:27:26.702921 16501 raft_consensus.cc:2243] T 742beffede4f407d8a72909d2b147e6d P 584dac86183a41d9b3bea1c2085e62fa [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:26.703559 16501 raft_consensus.cc:2272] T 742beffede4f407d8a72909d2b147e6d P 584dac86183a41d9b3bea1c2085e62fa [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:26.707008 16501 tablet_replica.cc:333] stopping tablet replica
I20260709 12:27:26.708007 16501 raft_consensus.cc:2243] T 0c428ff656ae4b9d99d5b5073f98725d P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:26.708727 16501 raft_consensus.cc:2272] T 0c428ff656ae4b9d99d5b5073f98725d P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:26.713797 16501 tablet_replica.cc:333] stopping tablet replica
I20260709 12:27:26.714846 16501 raft_consensus.cc:2243] T d1918297b9eb41f5847a211e668ae748 P 584dac86183a41d9b3bea1c2085e62fa [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:27:26.716001 16501 raft_consensus.cc:2272] T d1918297b9eb41f5847a211e668ae748 P 584dac86183a41d9b3bea1c2085e62fa [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:26.719621 16501 tablet_replica.cc:333] stopping tablet replica
I20260709 12:27:26.720676 16501 raft_consensus.cc:2243] T fcb6a7f720b14dc98e821f663357a0bb P 584dac86183a41d9b3bea1c2085e62fa [term 3 LEADER]: Raft consensus shutting down.
I20260709 12:27:26.721772 16501 raft_consensus.cc:2272] T fcb6a7f720b14dc98e821f663357a0bb P 584dac86183a41d9b3bea1c2085e62fa [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:26.725023 16501 tablet_replica.cc:333] stopping tablet replica
I20260709 12:27:26.726017 16501 raft_consensus.cc:2243] T 1d4e0ad0462e4f74a44155aa3ffaf87f P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:26.726573 16501 raft_consensus.cc:2272] T 1d4e0ad0462e4f74a44155aa3ffaf87f P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:26.729990 16501 tablet_replica.cc:333] stopping tablet replica
I20260709 12:27:26.730927 16501 raft_consensus.cc:2243] T cbf03426b6df43e5b2b9c56e2eb7687c P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:26.731539 16501 raft_consensus.cc:2272] T cbf03426b6df43e5b2b9c56e2eb7687c P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:26.734933 16501 tablet_replica.cc:333] stopping tablet replica
I20260709 12:27:26.736066 16501 raft_consensus.cc:2243] T e67a2affef634fd8bc96a64fa83cffdb P 584dac86183a41d9b3bea1c2085e62fa [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:26.736706 16501 raft_consensus.cc:2272] T e67a2affef634fd8bc96a64fa83cffdb P 584dac86183a41d9b3bea1c2085e62fa [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:26.740036 16501 tablet_replica.cc:333] stopping tablet replica
I20260709 12:27:26.740950 16501 raft_consensus.cc:2243] T ef20d7124e7a44aa97a219ee194f4b8b P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:26.741535 16501 raft_consensus.cc:2272] T ef20d7124e7a44aa97a219ee194f4b8b P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:26.745039 16501 tablet_replica.cc:333] stopping tablet replica
I20260709 12:27:26.746007 16501 raft_consensus.cc:2243] T 31a190e9f5ff407fab90502fb3197b70 P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:26.746644 16501 raft_consensus.cc:2272] T 31a190e9f5ff407fab90502fb3197b70 P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:26.750082 16501 tablet_replica.cc:333] stopping tablet replica
I20260709 12:27:26.751073 16501 raft_consensus.cc:2243] T 6c8b2fa9f2c044ec9eec97c72b561042 P 584dac86183a41d9b3bea1c2085e62fa [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:27:26.752239 16501 raft_consensus.cc:2272] T 6c8b2fa9f2c044ec9eec97c72b561042 P 584dac86183a41d9b3bea1c2085e62fa [term 2 FOLLOWER]: Raft consensus is shut down!
W20260709 12:27:26.753756 29560 consensus_peers.cc:597] T 2707271fd46f456aa765026642a3efa9 P d73d92fb35234d16824c55689b5fce28 -> Peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Couldn't send request to peer 584dac86183a41d9b3bea1c2085e62fa. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:26.756953 29634 consensus_peers.cc:597] T 0c428ff656ae4b9d99d5b5073f98725d P d13c58ad8ac441c4a9b396eb04b1a476 -> Peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Couldn't send request to peer 584dac86183a41d9b3bea1c2085e62fa. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:26.760185 16501 tablet_replica.cc:333] stopping tablet replica
I20260709 12:27:26.761349 16501 raft_consensus.cc:2243] T 66f652ba2c224885a2aea974bfe8f88a P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:26.761909 16501 raft_consensus.cc:2272] T 66f652ba2c224885a2aea974bfe8f88a P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:26.765509 16501 tablet_replica.cc:333] stopping tablet replica
I20260709 12:27:26.766454 16501 raft_consensus.cc:2243] T 8017a44219404faa8db8ff049c65205e P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:26.767014 16501 raft_consensus.cc:2272] T 8017a44219404faa8db8ff049c65205e P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:26.770309 16501 tablet_replica.cc:333] stopping tablet replica
I20260709 12:27:26.771266 16501 raft_consensus.cc:2243] T 62f8c9472b214576b9002e02536a3ace P 584dac86183a41d9b3bea1c2085e62fa [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:26.771937 16501 raft_consensus.cc:2272] T 62f8c9472b214576b9002e02536a3ace P 584dac86183a41d9b3bea1c2085e62fa [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:26.775322 16501 tablet_replica.cc:333] stopping tablet replica
I20260709 12:27:26.776324 16501 raft_consensus.cc:2243] T 2707271fd46f456aa765026642a3efa9 P 584dac86183a41d9b3bea1c2085e62fa [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:26.776975 16501 raft_consensus.cc:2272] T 2707271fd46f456aa765026642a3efa9 P 584dac86183a41d9b3bea1c2085e62fa [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:26.780400 16501 tablet_replica.cc:333] stopping tablet replica
I20260709 12:27:26.781471 16501 raft_consensus.cc:2243] T fb6e78ccc01c4707af34cddf81619eab P 584dac86183a41d9b3bea1c2085e62fa [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:27:26.782740 16501 raft_consensus.cc:2272] T fb6e78ccc01c4707af34cddf81619eab P 584dac86183a41d9b3bea1c2085e62fa [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:26.786311 16501 tablet_replica.cc:333] stopping tablet replica
I20260709 12:27:26.787352 16501 raft_consensus.cc:2243] T 8627b58b4f42491597125eb45f8d3b61 P 584dac86183a41d9b3bea1c2085e62fa [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:26.787961 16501 raft_consensus.cc:2272] T 8627b58b4f42491597125eb45f8d3b61 P 584dac86183a41d9b3bea1c2085e62fa [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:26.791621 16501 tablet_replica.cc:333] stopping tablet replica
I20260709 12:27:26.792680 16501 raft_consensus.cc:2243] T a46289e8fc824dd5a818e30df6e7ec3e P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:26.793419 16501 raft_consensus.cc:2272] T a46289e8fc824dd5a818e30df6e7ec3e P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:26.797151 16501 tablet_replica.cc:333] stopping tablet replica
I20260709 12:27:26.798048 16501 raft_consensus.cc:2243] T 1d1c1f58c4d340ac951dfa403d34138a P 584dac86183a41d9b3bea1c2085e62fa [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:27:26.799157 16501 raft_consensus.cc:2272] T 1d1c1f58c4d340ac951dfa403d34138a P 584dac86183a41d9b3bea1c2085e62fa [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:26.802549 16501 tablet_replica.cc:333] stopping tablet replica
I20260709 12:27:26.803481 16501 raft_consensus.cc:2243] T 71e2de235adf46d9a2407659080d42ef P 584dac86183a41d9b3bea1c2085e62fa [term 3 LEADER]: Raft consensus shutting down.
I20260709 12:27:26.804531 16501 raft_consensus.cc:2272] T 71e2de235adf46d9a2407659080d42ef P 584dac86183a41d9b3bea1c2085e62fa [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:26.807626 16501 tablet_replica.cc:333] stopping tablet replica
I20260709 12:27:26.810015 16501 raft_consensus.cc:2243] T 6e37d72807e547898f4fce9e8b82b2b0 P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:26.810648 16501 raft_consensus.cc:2272] T 6e37d72807e547898f4fce9e8b82b2b0 P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:26.814123 16501 tablet_replica.cc:333] stopping tablet replica
I20260709 12:27:26.814939 16501 raft_consensus.cc:2243] T f8c01e05981747a395e5483b551297dd P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:26.815434 16501 raft_consensus.cc:2272] T f8c01e05981747a395e5483b551297dd P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:26.818527 16501 tablet_replica.cc:333] stopping tablet replica
W20260709 12:27:26.820729 29634 consensus_peers.cc:597] T e67a2affef634fd8bc96a64fa83cffdb P d13c58ad8ac441c4a9b396eb04b1a476 -> Peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Couldn't send request to peer 584dac86183a41d9b3bea1c2085e62fa. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:26.832104 16501 raft_consensus.cc:2243] T f4516786e5564e9eac08ca472f79a9f0 P 584dac86183a41d9b3bea1c2085e62fa [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:26.832978 16501 raft_consensus.cc:2272] T f4516786e5564e9eac08ca472f79a9f0 P 584dac86183a41d9b3bea1c2085e62fa [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:26.836475 16501 tablet_replica.cc:333] stopping tablet replica
I20260709 12:27:26.837371 16501 raft_consensus.cc:2243] T 4685c2e824c14fc1bcd45c16169798cc P 584dac86183a41d9b3bea1c2085e62fa [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:27:26.838426 16501 raft_consensus.cc:2272] T 4685c2e824c14fc1bcd45c16169798cc P 584dac86183a41d9b3bea1c2085e62fa [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:26.846192 16501 tablet_replica.cc:333] stopping tablet replica
I20260709 12:27:26.847046 16501 raft_consensus.cc:2243] T fb7d78e534634e0f992fc39b578ebd8d P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:26.847577 16501 raft_consensus.cc:2272] T fb7d78e534634e0f992fc39b578ebd8d P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:26.850605 16501 tablet_replica.cc:333] stopping tablet replica
I20260709 12:27:26.851450 16501 raft_consensus.cc:2243] T d5fcd1f030614bbe82cf1dc261dfc1c3 P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:26.852113 16501 raft_consensus.cc:2272] T d5fcd1f030614bbe82cf1dc261dfc1c3 P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:26.855324 16501 tablet_replica.cc:333] stopping tablet replica
I20260709 12:27:26.856221 16501 raft_consensus.cc:2243] T d42259e436d241b29fb4ded16d9b6455 P 584dac86183a41d9b3bea1c2085e62fa [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:26.856818 16501 raft_consensus.cc:2272] T d42259e436d241b29fb4ded16d9b6455 P 584dac86183a41d9b3bea1c2085e62fa [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:26.860107 16501 tablet_replica.cc:333] stopping tablet replica
I20260709 12:27:26.861042 16501 raft_consensus.cc:2243] T 425777eed34441cd93e0902bdc126516 P 584dac86183a41d9b3bea1c2085e62fa [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:26.861683 16501 raft_consensus.cc:2272] T 425777eed34441cd93e0902bdc126516 P 584dac86183a41d9b3bea1c2085e62fa [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:26.864882 16501 tablet_replica.cc:333] stopping tablet replica
I20260709 12:27:26.865743 16501 raft_consensus.cc:2243] T 3e9451c3797c4496a66dfdb3e9fb5e38 P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:26.866286 16501 raft_consensus.cc:2272] T 3e9451c3797c4496a66dfdb3e9fb5e38 P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 12:27:26.869567 29634 consensus_peers.cc:597] T 5ee924f859f54f64908499bfac48ee62 P d13c58ad8ac441c4a9b396eb04b1a476 -> Peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Couldn't send request to peer 584dac86183a41d9b3bea1c2085e62fa. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:26.869602 16501 tablet_replica.cc:333] stopping tablet replica
I20260709 12:27:26.877175 16501 raft_consensus.cc:2243] T f3e17f0a8fc14095850e8138fd79b86e P 584dac86183a41d9b3bea1c2085e62fa [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:26.877833 16501 raft_consensus.cc:2272] T f3e17f0a8fc14095850e8138fd79b86e P 584dac86183a41d9b3bea1c2085e62fa [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:26.881568 16501 tablet_replica.cc:333] stopping tablet replica
I20260709 12:27:26.882923 16501 raft_consensus.cc:2243] T c6e621aa77d5474580dfc5949801ccc9 P 584dac86183a41d9b3bea1c2085e62fa [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:26.883641 16501 raft_consensus.cc:2272] T c6e621aa77d5474580dfc5949801ccc9 P 584dac86183a41d9b3bea1c2085e62fa [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:26.888522 16501 tablet_replica.cc:333] stopping tablet replica
I20260709 12:27:26.889354 16501 raft_consensus.cc:2243] T a3dcbb9c64bf4b5bb6680e8cb81af93e P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:26.889994 16501 raft_consensus.cc:2272] T a3dcbb9c64bf4b5bb6680e8cb81af93e P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:26.893923 16501 tablet_replica.cc:333] stopping tablet replica
I20260709 12:27:26.894843 16501 raft_consensus.cc:2243] T abec627108a84083884e64e1eb9dcda6 P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:26.895416 16501 raft_consensus.cc:2272] T abec627108a84083884e64e1eb9dcda6 P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:26.898334 16501 tablet_replica.cc:333] stopping tablet replica
W20260709 12:27:26.898370 29560 consensus_peers.cc:597] T f8c01e05981747a395e5483b551297dd P d73d92fb35234d16824c55689b5fce28 -> Peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Couldn't send request to peer 584dac86183a41d9b3bea1c2085e62fa. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:26.899435 16501 raft_consensus.cc:2243] T 3f41ce441fb047c8bbb741b1b9cbc0ec P 584dac86183a41d9b3bea1c2085e62fa [term 3 LEADER]: Raft consensus shutting down.
I20260709 12:27:26.900642 16501 raft_consensus.cc:2272] T 3f41ce441fb047c8bbb741b1b9cbc0ec P 584dac86183a41d9b3bea1c2085e62fa [term 3 FOLLOWER]: Raft consensus is shut down!
W20260709 12:27:26.903812 29708 consensus_peers.cc:597] T 1d4e0ad0462e4f74a44155aa3ffaf87f P c6610c00ffb948ce9c34f41dec3d49e8 -> Peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Couldn't send request to peer 584dac86183a41d9b3bea1c2085e62fa. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:26.909281 16501 tablet_replica.cc:333] stopping tablet replica
I20260709 12:27:26.910351 16501 raft_consensus.cc:2243] T 5ee924f859f54f64908499bfac48ee62 P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:26.910992 16501 raft_consensus.cc:2272] T 5ee924f859f54f64908499bfac48ee62 P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:26.914094 16501 tablet_replica.cc:333] stopping tablet replica
I20260709 12:27:26.915046 16501 raft_consensus.cc:2243] T 7ae3b542012d4f35a9c363eee22976ec P 584dac86183a41d9b3bea1c2085e62fa [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:26.915714 16501 raft_consensus.cc:2272] T 7ae3b542012d4f35a9c363eee22976ec P 584dac86183a41d9b3bea1c2085e62fa [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:26.918928 16501 tablet_replica.cc:333] stopping tablet replica
I20260709 12:27:26.919804 16501 raft_consensus.cc:2243] T 592801934cad4789b9ce8dc09f1470bd P 584dac86183a41d9b3bea1c2085e62fa [term 3 LEADER]: Raft consensus shutting down.
I20260709 12:27:26.920858 16501 raft_consensus.cc:2272] T 592801934cad4789b9ce8dc09f1470bd P 584dac86183a41d9b3bea1c2085e62fa [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:26.924014 16501 tablet_replica.cc:333] stopping tablet replica
I20260709 12:27:26.928544 16501 raft_consensus.cc:2243] T 6a30685caefb4d0e80fa4ea87d70760c P 584dac86183a41d9b3bea1c2085e62fa [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:27:26.930444 16501 raft_consensus.cc:2272] T 6a30685caefb4d0e80fa4ea87d70760c P 584dac86183a41d9b3bea1c2085e62fa [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:26.935038 16501 tablet_replica.cc:333] stopping tablet replica
I20260709 12:27:26.935863 16501 raft_consensus.cc:2243] T d21ab1641501475b829863275dc8182a P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:26.936506 16501 raft_consensus.cc:2272] T d21ab1641501475b829863275dc8182a P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:26.939815 16501 tablet_replica.cc:333] stopping tablet replica
I20260709 12:27:26.940665 16501 raft_consensus.cc:2243] T 3615115233c34461b85763151751e475 P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Raft consensus shutting down.
W20260709 12:27:26.941009 29708 consensus_peers.cc:597] T 8017a44219404faa8db8ff049c65205e P c6610c00ffb948ce9c34f41dec3d49e8 -> Peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Couldn't send request to peer 584dac86183a41d9b3bea1c2085e62fa. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:26.941259 16501 raft_consensus.cc:2272] T 3615115233c34461b85763151751e475 P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:26.945353 16501 tablet_replica.cc:333] stopping tablet replica
I20260709 12:27:26.946187 16501 raft_consensus.cc:2243] T b3c02c910d2b43cc9265bb1d98491cd3 P 584dac86183a41d9b3bea1c2085e62fa [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:27:26.947324 16501 raft_consensus.cc:2272] T b3c02c910d2b43cc9265bb1d98491cd3 P 584dac86183a41d9b3bea1c2085e62fa [term 2 FOLLOWER]: Raft consensus is shut down!
W20260709 12:27:26.950167 29560 consensus_peers.cc:597] T 62f8c9472b214576b9002e02536a3ace P d73d92fb35234d16824c55689b5fce28 -> Peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Couldn't send request to peer 584dac86183a41d9b3bea1c2085e62fa. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:26.950220 16501 tablet_replica.cc:333] stopping tablet replica
I20260709 12:27:26.951498 16501 raft_consensus.cc:2243] T 5be58d64d45e40d7981018b84a01b1ed P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:26.953446 16501 raft_consensus.cc:2272] T 5be58d64d45e40d7981018b84a01b1ed P 584dac86183a41d9b3bea1c2085e62fa [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 12:27:26.990235 29560 consensus_peers.cc:597] T 3615115233c34461b85763151751e475 P d73d92fb35234d16824c55689b5fce28 -> Peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Couldn't send request to peer 584dac86183a41d9b3bea1c2085e62fa. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:27.012705 29560 consensus_peers.cc:597] T 5be58d64d45e40d7981018b84a01b1ed P d73d92fb35234d16824c55689b5fce28 -> Peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Couldn't send request to peer 584dac86183a41d9b3bea1c2085e62fa. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:27.119480 16501 tablet_server.cc:196] TabletServer@127.16.29.65:35033 shutdown complete.
W20260709 12:27:27.125669 29635 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.68:43895, user_credentials={real_user=slave}} blocked reactor thread for 51241.6us
W20260709 12:27:27.137177 29709 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:38209, user_credentials={real_user=slave}} blocked reactor thread for 55413.1us
W20260709 12:27:27.143350 29708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:36825, user_credentials={real_user=slave}} blocked reactor thread for 60974.3us
W20260709 12:27:27.152555 29708 connection.cc:769] client connection to 127.16.29.65:35033 send error: Network error: sendmsg error: Broken pipe (error 32)
W20260709 12:27:27.177889 29634 consensus_peers.cc:597] T cbf03426b6df43e5b2b9c56e2eb7687c P d13c58ad8ac441c4a9b396eb04b1a476 -> Peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Couldn't send request to peer 584dac86183a41d9b3bea1c2085e62fa. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:27.208351 29708 consensus_peers.cc:597] T 425777eed34441cd93e0902bdc126516 P c6610c00ffb948ce9c34f41dec3d49e8 -> Peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Couldn't send request to peer 584dac86183a41d9b3bea1c2085e62fa. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:27.209249 29708 consensus_peers.cc:597] T f4516786e5564e9eac08ca472f79a9f0 P c6610c00ffb948ce9c34f41dec3d49e8 -> Peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Couldn't send request to peer 584dac86183a41d9b3bea1c2085e62fa. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:27.396930 16501 tablet_server.cc:179] TabletServer@127.16.29.66:0 shutting down...
W20260709 12:27:27.477351 29634 consensus_peers.cc:597] T 76140ad6258543118a9bfeee7cf0d5fc P d13c58ad8ac441c4a9b396eb04b1a476 -> Peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Couldn't send request to peer d73d92fb35234d16824c55689b5fce28. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:27.487298 29634 consensus_peers.cc:597] T 3617af2dee3f48cf85973b1c94cdc33a P d13c58ad8ac441c4a9b396eb04b1a476 -> Peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Couldn't send request to peer d73d92fb35234d16824c55689b5fce28. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:27.491806 29634 consensus_peers.cc:597] T 07a2b94c0823467695508292a9ccdcc5 P d13c58ad8ac441c4a9b396eb04b1a476 -> Peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Couldn't send request to peer d73d92fb35234d16824c55689b5fce28. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:27.512903 29634 consensus_peers.cc:597] T d42259e436d241b29fb4ded16d9b6455 P d13c58ad8ac441c4a9b396eb04b1a476 -> Peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Couldn't send request to peer d73d92fb35234d16824c55689b5fce28. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:27.519145 29708 consensus_peers.cc:597] T 394378d078194257b9e04fc659c1f5d4 P c6610c00ffb948ce9c34f41dec3d49e8 -> Peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Couldn't send request to peer d73d92fb35234d16824c55689b5fce28. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:27.521755 29708 consensus_peers.cc:597] T 8017a44219404faa8db8ff049c65205e P c6610c00ffb948ce9c34f41dec3d49e8 -> Peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Couldn't send request to peer d73d92fb35234d16824c55689b5fce28. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:27.523483 29634 consensus_peers.cc:597] T cbf03426b6df43e5b2b9c56e2eb7687c P d13c58ad8ac441c4a9b396eb04b1a476 -> Peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Couldn't send request to peer d73d92fb35234d16824c55689b5fce28. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:27.531298 16501 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:27:27.534021 16501 tablet_replica.cc:333] T a3dcbb9c64bf4b5bb6680e8cb81af93e P d73d92fb35234d16824c55689b5fce28: stopping tablet replica
I20260709 12:27:27.535130 16501 raft_consensus.cc:2243] T a3dcbb9c64bf4b5bb6680e8cb81af93e P d73d92fb35234d16824c55689b5fce28 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:27:27.536469 16501 raft_consensus.cc:2272] T a3dcbb9c64bf4b5bb6680e8cb81af93e P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:27.539942 16501 tablet_replica.cc:333] T 137e03144e754f6e9da4f0544513d27c P d73d92fb35234d16824c55689b5fce28: stopping tablet replica
W20260709 12:27:27.546666 29708 consensus_peers.cc:597] T 1d4e0ad0462e4f74a44155aa3ffaf87f P c6610c00ffb948ce9c34f41dec3d49e8 -> Peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Couldn't send request to peer d73d92fb35234d16824c55689b5fce28. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:27.547536 16501 raft_consensus.cc:2243] T 137e03144e754f6e9da4f0544513d27c P d73d92fb35234d16824c55689b5fce28 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:27:27.548765 16501 raft_consensus.cc:2272] T 137e03144e754f6e9da4f0544513d27c P d73d92fb35234d16824c55689b5fce28 [term 2 FOLLOWER]: Raft consensus is shut down!
W20260709 12:27:27.549296 29634 consensus_peers.cc:597] T f3e17f0a8fc14095850e8138fd79b86e P d13c58ad8ac441c4a9b396eb04b1a476 -> Peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Couldn't send request to peer d73d92fb35234d16824c55689b5fce28. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:27.551932 16501 tablet_replica.cc:333] T 394378d078194257b9e04fc659c1f5d4 P d73d92fb35234d16824c55689b5fce28: stopping tablet replica
I20260709 12:27:27.552989 16501 raft_consensus.cc:2243] T 394378d078194257b9e04fc659c1f5d4 P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:27.553606 16501 raft_consensus.cc:2272] T 394378d078194257b9e04fc659c1f5d4 P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:27.557089 16501 tablet_replica.cc:333] T abec627108a84083884e64e1eb9dcda6 P d73d92fb35234d16824c55689b5fce28: stopping tablet replica
I20260709 12:27:27.558080 16501 raft_consensus.cc:2243] T abec627108a84083884e64e1eb9dcda6 P d73d92fb35234d16824c55689b5fce28 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:27:27.560261 16501 raft_consensus.cc:2272] T abec627108a84083884e64e1eb9dcda6 P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:27.563393 16501 tablet_replica.cc:333] T 01788883c8ea42989f246a3486111a7e P d73d92fb35234d16824c55689b5fce28: stopping tablet replica
I20260709 12:27:27.564440 16501 raft_consensus.cc:2243] T 01788883c8ea42989f246a3486111a7e P d73d92fb35234d16824c55689b5fce28 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:27.565075 16501 raft_consensus.cc:2272] T 01788883c8ea42989f246a3486111a7e P d73d92fb35234d16824c55689b5fce28 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:27.568270 16501 tablet_replica.cc:333] T 1d4e0ad0462e4f74a44155aa3ffaf87f P d73d92fb35234d16824c55689b5fce28: stopping tablet replica
I20260709 12:27:27.569273 16501 raft_consensus.cc:2243] T 1d4e0ad0462e4f74a44155aa3ffaf87f P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:27.569933 16501 raft_consensus.cc:2272] T 1d4e0ad0462e4f74a44155aa3ffaf87f P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:27.573235 16501 tablet_replica.cc:333] T cbf03426b6df43e5b2b9c56e2eb7687c P d73d92fb35234d16824c55689b5fce28: stopping tablet replica
I20260709 12:27:27.574218 16501 raft_consensus.cc:2243] T cbf03426b6df43e5b2b9c56e2eb7687c P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:27.578356 16501 raft_consensus.cc:2272] T cbf03426b6df43e5b2b9c56e2eb7687c P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:27.581833 16501 tablet_replica.cc:333] T f3e17f0a8fc14095850e8138fd79b86e P d73d92fb35234d16824c55689b5fce28: stopping tablet replica
I20260709 12:27:27.582886 16501 raft_consensus.cc:2243] T f3e17f0a8fc14095850e8138fd79b86e P d73d92fb35234d16824c55689b5fce28 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:27.583564 16501 raft_consensus.cc:2272] T f3e17f0a8fc14095850e8138fd79b86e P d73d92fb35234d16824c55689b5fce28 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:27.586795 16501 tablet_replica.cc:333] T c6e621aa77d5474580dfc5949801ccc9 P d73d92fb35234d16824c55689b5fce28: stopping tablet replica
I20260709 12:27:27.587684 16501 raft_consensus.cc:2243] T c6e621aa77d5474580dfc5949801ccc9 P d73d92fb35234d16824c55689b5fce28 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:27.588337 16501 raft_consensus.cc:2272] T c6e621aa77d5474580dfc5949801ccc9 P d73d92fb35234d16824c55689b5fce28 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:27.591966 16501 tablet_replica.cc:333] T 16649fd17d434e3c95e9d585d2404659 P d73d92fb35234d16824c55689b5fce28: stopping tablet replica
I20260709 12:27:27.592916 16501 raft_consensus.cc:2243] T 16649fd17d434e3c95e9d585d2404659 P d73d92fb35234d16824c55689b5fce28 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:27:27.595445 16501 raft_consensus.cc:2272] T 16649fd17d434e3c95e9d585d2404659 P d73d92fb35234d16824c55689b5fce28 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:27.598687 16501 tablet_replica.cc:333] T f8c01e05981747a395e5483b551297dd P d73d92fb35234d16824c55689b5fce28: stopping tablet replica
I20260709 12:27:27.599607 16501 raft_consensus.cc:2243] T f8c01e05981747a395e5483b551297dd P d73d92fb35234d16824c55689b5fce28 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:27:27.600711 16501 raft_consensus.cc:2272] T f8c01e05981747a395e5483b551297dd P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:27.603938 16501 tablet_replica.cc:333] T 71e2de235adf46d9a2407659080d42ef P d73d92fb35234d16824c55689b5fce28: stopping tablet replica
I20260709 12:27:27.604825 16501 raft_consensus.cc:2243] T 71e2de235adf46d9a2407659080d42ef P d73d92fb35234d16824c55689b5fce28 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:27.605614 16501 raft_consensus.cc:2272] T 71e2de235adf46d9a2407659080d42ef P d73d92fb35234d16824c55689b5fce28 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:27.608770 16501 tablet_replica.cc:333] T 07a2b94c0823467695508292a9ccdcc5 P d73d92fb35234d16824c55689b5fce28: stopping tablet replica
I20260709 12:27:27.609630 16501 raft_consensus.cc:2243] T 07a2b94c0823467695508292a9ccdcc5 P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:27.610258 16501 raft_consensus.cc:2272] T 07a2b94c0823467695508292a9ccdcc5 P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:27.613675 16501 tablet_replica.cc:333] T 959940284eb24fdf83d78438d710318a P d73d92fb35234d16824c55689b5fce28: stopping tablet replica
I20260709 12:27:27.614753 16501 raft_consensus.cc:2243] T 959940284eb24fdf83d78438d710318a P d73d92fb35234d16824c55689b5fce28 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:27.615351 16501 raft_consensus.cc:2272] T 959940284eb24fdf83d78438d710318a P d73d92fb35234d16824c55689b5fce28 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:27.618378 16501 tablet_replica.cc:333] T 8017a44219404faa8db8ff049c65205e P d73d92fb35234d16824c55689b5fce28: stopping tablet replica
I20260709 12:27:27.619259 16501 raft_consensus.cc:2243] T 8017a44219404faa8db8ff049c65205e P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:27.619848 16501 raft_consensus.cc:2272] T 8017a44219404faa8db8ff049c65205e P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:27.622736 16501 tablet_replica.cc:333] T 1d1c1f58c4d340ac951dfa403d34138a P d73d92fb35234d16824c55689b5fce28: stopping tablet replica
W20260709 12:27:27.623272 29708 consensus_peers.cc:597] T d299fb38094e47e8aa3890ca74f6b366 P c6610c00ffb948ce9c34f41dec3d49e8 -> Peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Couldn't send request to peer d73d92fb35234d16824c55689b5fce28. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:27.623663 16501 raft_consensus.cc:2243] T 1d1c1f58c4d340ac951dfa403d34138a P d73d92fb35234d16824c55689b5fce28 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:27.624583 16501 raft_consensus.cc:2272] T 1d1c1f58c4d340ac951dfa403d34138a P d73d92fb35234d16824c55689b5fce28 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:27.627954 16501 tablet_replica.cc:333] T ed057e06dec148a0b9d7062beeb9d005 P d73d92fb35234d16824c55689b5fce28: stopping tablet replica
I20260709 12:27:27.628942 16501 raft_consensus.cc:2243] T ed057e06dec148a0b9d7062beeb9d005 P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:27.629737 16501 raft_consensus.cc:2272] T ed057e06dec148a0b9d7062beeb9d005 P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:27.633152 16501 tablet_replica.cc:333] T fb6e78ccc01c4707af34cddf81619eab P d73d92fb35234d16824c55689b5fce28: stopping tablet replica
I20260709 12:27:27.639354 16501 raft_consensus.cc:2243] T fb6e78ccc01c4707af34cddf81619eab P d73d92fb35234d16824c55689b5fce28 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:27.640121 16501 raft_consensus.cc:2272] T fb6e78ccc01c4707af34cddf81619eab P d73d92fb35234d16824c55689b5fce28 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:27.652964 16501 tablet_replica.cc:333] T 2707271fd46f456aa765026642a3efa9 P d73d92fb35234d16824c55689b5fce28: stopping tablet replica
I20260709 12:27:27.654069 16501 raft_consensus.cc:2243] T 2707271fd46f456aa765026642a3efa9 P d73d92fb35234d16824c55689b5fce28 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:27:27.655332 16501 raft_consensus.cc:2272] T 2707271fd46f456aa765026642a3efa9 P d73d92fb35234d16824c55689b5fce28 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:27.658533 16501 tablet_replica.cc:333] T 6c8b2fa9f2c044ec9eec97c72b561042 P d73d92fb35234d16824c55689b5fce28: stopping tablet replica
I20260709 12:27:27.659708 16501 raft_consensus.cc:2243] T 6c8b2fa9f2c044ec9eec97c72b561042 P d73d92fb35234d16824c55689b5fce28 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:27.660488 16501 raft_consensus.cc:2272] T 6c8b2fa9f2c044ec9eec97c72b561042 P d73d92fb35234d16824c55689b5fce28 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:27.663697 16501 tablet_replica.cc:333] T 66f652ba2c224885a2aea974bfe8f88a P d73d92fb35234d16824c55689b5fce28: stopping tablet replica
I20260709 12:27:27.664822 16501 raft_consensus.cc:2243] T 66f652ba2c224885a2aea974bfe8f88a P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:27.665518 16501 raft_consensus.cc:2272] T 66f652ba2c224885a2aea974bfe8f88a P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 12:27:27.666520 29634 consensus_peers.cc:597] T c6e621aa77d5474580dfc5949801ccc9 P d13c58ad8ac441c4a9b396eb04b1a476 -> Peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Couldn't send request to peer d73d92fb35234d16824c55689b5fce28. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:27.669608 16501 tablet_replica.cc:333] T 5be58d64d45e40d7981018b84a01b1ed P d73d92fb35234d16824c55689b5fce28: stopping tablet replica
I20260709 12:27:27.670646 16501 raft_consensus.cc:2243] T 5be58d64d45e40d7981018b84a01b1ed P d73d92fb35234d16824c55689b5fce28 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:27:27.671834 16501 raft_consensus.cc:2272] T 5be58d64d45e40d7981018b84a01b1ed P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:27.675755 16501 tablet_replica.cc:333] T 3615115233c34461b85763151751e475 P d73d92fb35234d16824c55689b5fce28: stopping tablet replica
I20260709 12:27:27.676687 16501 raft_consensus.cc:2243] T 3615115233c34461b85763151751e475 P d73d92fb35234d16824c55689b5fce28 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:27:27.677775 16501 raft_consensus.cc:2272] T 3615115233c34461b85763151751e475 P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:27.680934 16501 tablet_replica.cc:333] T b3c02c910d2b43cc9265bb1d98491cd3 P d73d92fb35234d16824c55689b5fce28: stopping tablet replica
I20260709 12:27:27.681787 16501 raft_consensus.cc:2243] T b3c02c910d2b43cc9265bb1d98491cd3 P d73d92fb35234d16824c55689b5fce28 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:27.682335 16501 raft_consensus.cc:2272] T b3c02c910d2b43cc9265bb1d98491cd3 P d73d92fb35234d16824c55689b5fce28 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:27.685220 16501 tablet_replica.cc:333] T 76140ad6258543118a9bfeee7cf0d5fc P d73d92fb35234d16824c55689b5fce28: stopping tablet replica
I20260709 12:27:27.686384 16501 raft_consensus.cc:2243] T 76140ad6258543118a9bfeee7cf0d5fc P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:27.687377 16501 raft_consensus.cc:2272] T 76140ad6258543118a9bfeee7cf0d5fc P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:27.690415 16501 tablet_replica.cc:333] T d299fb38094e47e8aa3890ca74f6b366 P d73d92fb35234d16824c55689b5fce28: stopping tablet replica
I20260709 12:27:27.691296 16501 raft_consensus.cc:2243] T d299fb38094e47e8aa3890ca74f6b366 P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:27.691861 16501 raft_consensus.cc:2272] T d299fb38094e47e8aa3890ca74f6b366 P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:27.695824 16501 tablet_replica.cc:333] T 6a30685caefb4d0e80fa4ea87d70760c P d73d92fb35234d16824c55689b5fce28: stopping tablet replica
I20260709 12:27:27.698439 16501 raft_consensus.cc:2243] T 6a30685caefb4d0e80fa4ea87d70760c P d73d92fb35234d16824c55689b5fce28 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:27.699918 16501 raft_consensus.cc:2272] T 6a30685caefb4d0e80fa4ea87d70760c P d73d92fb35234d16824c55689b5fce28 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:27.702729 16501 tablet_replica.cc:333] T 592801934cad4789b9ce8dc09f1470bd P d73d92fb35234d16824c55689b5fce28: stopping tablet replica
I20260709 12:27:27.703596 16501 raft_consensus.cc:2243] T 592801934cad4789b9ce8dc09f1470bd P d73d92fb35234d16824c55689b5fce28 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:27.704255 16501 raft_consensus.cc:2272] T 592801934cad4789b9ce8dc09f1470bd P d73d92fb35234d16824c55689b5fce28 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:27.707032 16501 tablet_replica.cc:333] T 00c6cf271c91400aa834467def9af9b1 P d73d92fb35234d16824c55689b5fce28: stopping tablet replica
I20260709 12:27:27.707855 16501 raft_consensus.cc:2243] T 00c6cf271c91400aa834467def9af9b1 P d73d92fb35234d16824c55689b5fce28 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:27:27.709007 16501 raft_consensus.cc:2272] T 00c6cf271c91400aa834467def9af9b1 P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:27.711802 16501 tablet_replica.cc:333] T c1c8411289a74b5685700229d245ff75 P d73d92fb35234d16824c55689b5fce28: stopping tablet replica
I20260709 12:27:27.712579 16501 raft_consensus.cc:2243] T c1c8411289a74b5685700229d245ff75 P d73d92fb35234d16824c55689b5fce28 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:27:27.713524 16501 raft_consensus.cc:2272] T c1c8411289a74b5685700229d245ff75 P d73d92fb35234d16824c55689b5fce28 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:27.715997 16501 tablet_replica.cc:333] T 62f8c9472b214576b9002e02536a3ace P d73d92fb35234d16824c55689b5fce28: stopping tablet replica
I20260709 12:27:27.716686 16501 raft_consensus.cc:2243] T 62f8c9472b214576b9002e02536a3ace P d73d92fb35234d16824c55689b5fce28 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:27:27.717522 16501 raft_consensus.cc:2272] T 62f8c9472b214576b9002e02536a3ace P d73d92fb35234d16824c55689b5fce28 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:27.719954 16501 tablet_replica.cc:333] T d5fcd1f030614bbe82cf1dc261dfc1c3 P d73d92fb35234d16824c55689b5fce28: stopping tablet replica
I20260709 12:27:27.720657 16501 raft_consensus.cc:2243] T d5fcd1f030614bbe82cf1dc261dfc1c3 P d73d92fb35234d16824c55689b5fce28 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:27:27.721660 16501 raft_consensus.cc:2272] T d5fcd1f030614bbe82cf1dc261dfc1c3 P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:27.724722 16501 tablet_replica.cc:333] T d42259e436d241b29fb4ded16d9b6455 P d73d92fb35234d16824c55689b5fce28: stopping tablet replica
I20260709 12:27:27.725467 16501 raft_consensus.cc:2243] T d42259e436d241b29fb4ded16d9b6455 P d73d92fb35234d16824c55689b5fce28 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:27.726015 16501 raft_consensus.cc:2272] T d42259e436d241b29fb4ded16d9b6455 P d73d92fb35234d16824c55689b5fce28 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:27.729264 16501 tablet_replica.cc:333] T 55061e62184b45f0bb1726adc1a4af81 P d73d92fb35234d16824c55689b5fce28: stopping tablet replica
I20260709 12:27:27.730094 16501 raft_consensus.cc:2243] T 55061e62184b45f0bb1726adc1a4af81 P d73d92fb35234d16824c55689b5fce28 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:27:27.731699 16501 raft_consensus.cc:2272] T 55061e62184b45f0bb1726adc1a4af81 P d73d92fb35234d16824c55689b5fce28 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:27.734846 16501 tablet_replica.cc:333] T cdf706ffabc34902ad3d1bf2cee7889e P d73d92fb35234d16824c55689b5fce28: stopping tablet replica
I20260709 12:27:27.735638 16501 raft_consensus.cc:2243] T cdf706ffabc34902ad3d1bf2cee7889e P d73d92fb35234d16824c55689b5fce28 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:27:27.736896 16501 raft_consensus.cc:2272] T cdf706ffabc34902ad3d1bf2cee7889e P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:27.739985 16501 tablet_replica.cc:333] T d1918297b9eb41f5847a211e668ae748 P d73d92fb35234d16824c55689b5fce28: stopping tablet replica
I20260709 12:27:27.740823 16501 raft_consensus.cc:2243] T d1918297b9eb41f5847a211e668ae748 P d73d92fb35234d16824c55689b5fce28 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:27.741343 16501 raft_consensus.cc:2272] T d1918297b9eb41f5847a211e668ae748 P d73d92fb35234d16824c55689b5fce28 [term 2 FOLLOWER]: Raft consensus is shut down!
W20260709 12:27:27.741986 29708 consensus_peers.cc:597] T 66f652ba2c224885a2aea974bfe8f88a P c6610c00ffb948ce9c34f41dec3d49e8 -> Peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Couldn't send request to peer d73d92fb35234d16824c55689b5fce28. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:27.744167 16501 tablet_replica.cc:333] T 43c7bfb33af74d4f912e5153e8a54b3e P d73d92fb35234d16824c55689b5fce28: stopping tablet replica
I20260709 12:27:27.745038 16501 raft_consensus.cc:2243] T 43c7bfb33af74d4f912e5153e8a54b3e P d73d92fb35234d16824c55689b5fce28 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:27.745563 16501 raft_consensus.cc:2272] T 43c7bfb33af74d4f912e5153e8a54b3e P d73d92fb35234d16824c55689b5fce28 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:27.748365 16501 tablet_replica.cc:333] T 4685c2e824c14fc1bcd45c16169798cc P d73d92fb35234d16824c55689b5fce28: stopping tablet replica
I20260709 12:27:27.749673 16501 raft_consensus.cc:2243] T 4685c2e824c14fc1bcd45c16169798cc P d73d92fb35234d16824c55689b5fce28 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:27.750336 16501 raft_consensus.cc:2272] T 4685c2e824c14fc1bcd45c16169798cc P d73d92fb35234d16824c55689b5fce28 [term 2 FOLLOWER]: Raft consensus is shut down!
W20260709 12:27:27.752877 29708 consensus_peers.cc:597] T 58154d49c1a344f2b51366ede3c87de5 P c6610c00ffb948ce9c34f41dec3d49e8 -> Peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Couldn't send request to peer d73d92fb35234d16824c55689b5fce28. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:27.753399 16501 tablet_replica.cc:333] T 58154d49c1a344f2b51366ede3c87de5 P d73d92fb35234d16824c55689b5fce28: stopping tablet replica
W20260709 12:27:27.756485 29634 consensus_peers.cc:597] T 5ee924f859f54f64908499bfac48ee62 P d13c58ad8ac441c4a9b396eb04b1a476 -> Peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Couldn't send request to peer d73d92fb35234d16824c55689b5fce28. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:27.758903 16501 raft_consensus.cc:2243] T 58154d49c1a344f2b51366ede3c87de5 P d73d92fb35234d16824c55689b5fce28 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:27.759516 16501 raft_consensus.cc:2272] T 58154d49c1a344f2b51366ede3c87de5 P d73d92fb35234d16824c55689b5fce28 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:27.763083 16501 tablet_replica.cc:333] T fcb6a7f720b14dc98e821f663357a0bb P d73d92fb35234d16824c55689b5fce28: stopping tablet replica
I20260709 12:27:27.763902 16501 raft_consensus.cc:2243] T fcb6a7f720b14dc98e821f663357a0bb P d73d92fb35234d16824c55689b5fce28 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:27.765828 16501 raft_consensus.cc:2272] T fcb6a7f720b14dc98e821f663357a0bb P d73d92fb35234d16824c55689b5fce28 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:27.768903 16501 tablet_replica.cc:333] T 3617af2dee3f48cf85973b1c94cdc33a P d73d92fb35234d16824c55689b5fce28: stopping tablet replica
I20260709 12:27:27.769729 16501 raft_consensus.cc:2243] T 3617af2dee3f48cf85973b1c94cdc33a P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:27.770387 16501 raft_consensus.cc:2272] T 3617af2dee3f48cf85973b1c94cdc33a P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:27.773475 16501 tablet_replica.cc:333] T d21ab1641501475b829863275dc8182a P d73d92fb35234d16824c55689b5fce28: stopping tablet replica
I20260709 12:27:27.774210 16501 raft_consensus.cc:2243] T d21ab1641501475b829863275dc8182a P d73d92fb35234d16824c55689b5fce28 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:27:27.775240 16501 raft_consensus.cc:2272] T d21ab1641501475b829863275dc8182a P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:27.778097 16501 tablet_replica.cc:333] T 5ee924f859f54f64908499bfac48ee62 P d73d92fb35234d16824c55689b5fce28: stopping tablet replica
I20260709 12:27:27.778807 16501 raft_consensus.cc:2243] T 5ee924f859f54f64908499bfac48ee62 P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:27.779320 16501 raft_consensus.cc:2272] T 5ee924f859f54f64908499bfac48ee62 P d73d92fb35234d16824c55689b5fce28 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:27.782620 16501 tablet_replica.cc:333] T 7ae3b542012d4f35a9c363eee22976ec P d73d92fb35234d16824c55689b5fce28: stopping tablet replica
I20260709 12:27:27.783326 16501 raft_consensus.cc:2243] T 7ae3b542012d4f35a9c363eee22976ec P d73d92fb35234d16824c55689b5fce28 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:27.783938 16501 raft_consensus.cc:2272] T 7ae3b542012d4f35a9c363eee22976ec P d73d92fb35234d16824c55689b5fce28 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:27.787981 30661 raft_consensus.cc:493] T 01788883c8ea42989f246a3486111a7e P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 584dac86183a41d9b3bea1c2085e62fa)
I20260709 12:27:27.788569 30661 raft_consensus.cc:515] T 01788883c8ea42989f246a3486111a7e P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:27.790874 30661 leader_election.cc:290] T 01788883c8ea42989f246a3486111a7e P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033), d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825)
I20260709 12:27:27.796672 30682 raft_consensus.cc:493] T f76101eb30f14b98a9aa8c41e0d6f831 P d13c58ad8ac441c4a9b396eb04b1a476 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 584dac86183a41d9b3bea1c2085e62fa)
I20260709 12:27:27.797415 30682 raft_consensus.cc:515] T f76101eb30f14b98a9aa8c41e0d6f831 P d13c58ad8ac441c4a9b396eb04b1a476 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
W20260709 12:27:27.806631 29634 leader_election.cc:336] T f76101eb30f14b98a9aa8c41e0d6f831 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111)
I20260709 12:27:27.807580 30682 leader_election.cc:290] T f76101eb30f14b98a9aa8c41e0d6f831 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895), 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033)
I20260709 12:27:27.813176 29743 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f76101eb30f14b98a9aa8c41e0d6f831" candidate_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" is_pre_election: true
W20260709 12:27:27.815282 29708 leader_election.cc:336] T 01788883c8ea42989f246a3486111a7e P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111)
I20260709 12:27:27.817945 30682 raft_consensus.cc:493] T d1918297b9eb41f5847a211e668ae748 P d13c58ad8ac441c4a9b396eb04b1a476 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 584dac86183a41d9b3bea1c2085e62fa)
I20260709 12:27:27.818593 30682 raft_consensus.cc:515] T d1918297b9eb41f5847a211e668ae748 P d13c58ad8ac441c4a9b396eb04b1a476 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
W20260709 12:27:27.827111 29634 leader_election.cc:336] T d1918297b9eb41f5847a211e668ae748 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111)
I20260709 12:27:27.829028 30682 leader_election.cc:290] T d1918297b9eb41f5847a211e668ae748 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033), d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825)
W20260709 12:27:27.835443 29634 leader_election.cc:336] T d1918297b9eb41f5847a211e668ae748 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:27:27.837375 29634 leader_election.cc:304] T d1918297b9eb41f5847a211e668ae748 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d13c58ad8ac441c4a9b396eb04b1a476; no voters: 584dac86183a41d9b3bea1c2085e62fa, d73d92fb35234d16824c55689b5fce28
I20260709 12:27:27.841526 30682 raft_consensus.cc:2749] T d1918297b9eb41f5847a211e668ae748 P d13c58ad8ac441c4a9b396eb04b1a476 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:27:27.853073 29635 leader_election.cc:304] T f76101eb30f14b98a9aa8c41e0d6f831 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d13c58ad8ac441c4a9b396eb04b1a476; no voters: 584dac86183a41d9b3bea1c2085e62fa, c6610c00ffb948ce9c34f41dec3d49e8
I20260709 12:27:27.854605 30682 raft_consensus.cc:2749] T f76101eb30f14b98a9aa8c41e0d6f831 P d13c58ad8ac441c4a9b396eb04b1a476 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260709 12:27:27.862352 29708 leader_election.cc:336] T 01788883c8ea42989f246a3486111a7e P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:27:27.863058 29708 leader_election.cc:304] T 01788883c8ea42989f246a3486111a7e P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c6610c00ffb948ce9c34f41dec3d49e8; no voters: 584dac86183a41d9b3bea1c2085e62fa, d73d92fb35234d16824c55689b5fce28
I20260709 12:27:27.864104 30683 raft_consensus.cc:2749] T 01788883c8ea42989f246a3486111a7e P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260709 12:27:27.868853 29708 consensus_peers.cc:597] T ed057e06dec148a0b9d7062beeb9d005 P c6610c00ffb948ce9c34f41dec3d49e8 -> Peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Couldn't send request to peer d73d92fb35234d16824c55689b5fce28. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:27.875427 29708 consensus_peers.cc:597] T 959940284eb24fdf83d78438d710318a P c6610c00ffb948ce9c34f41dec3d49e8 -> Peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Couldn't send request to peer d73d92fb35234d16824c55689b5fce28. Status: Network error: recv got EOF from unknown peer (error 108). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:27.891042 30682 raft_consensus.cc:493] T fcb6a7f720b14dc98e821f663357a0bb P d13c58ad8ac441c4a9b396eb04b1a476 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 584dac86183a41d9b3bea1c2085e62fa)
I20260709 12:27:27.891774 30682 raft_consensus.cc:515] T fcb6a7f720b14dc98e821f663357a0bb P d13c58ad8ac441c4a9b396eb04b1a476 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:27:27.894654 30682 leader_election.cc:290] T fcb6a7f720b14dc98e821f663357a0bb P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033), d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825)
W20260709 12:27:27.902861 29634 leader_election.cc:336] T fcb6a7f720b14dc98e821f663357a0bb P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111)
W20260709 12:27:27.903662 29634 leader_election.cc:336] T fcb6a7f720b14dc98e821f663357a0bb P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Network error: Client connection negotiation failed: client connection to 127.16.29.66:36825: connect: Connection refused (error 111)
I20260709 12:27:27.904100 29634 leader_election.cc:304] T fcb6a7f720b14dc98e821f663357a0bb P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d13c58ad8ac441c4a9b396eb04b1a476; no voters: 584dac86183a41d9b3bea1c2085e62fa, d73d92fb35234d16824c55689b5fce28
I20260709 12:27:27.905531 30682 raft_consensus.cc:2749] T fcb6a7f720b14dc98e821f663357a0bb P d13c58ad8ac441c4a9b396eb04b1a476 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:27:27.925294 30683 raft_consensus.cc:493] T 4685c2e824c14fc1bcd45c16169798cc P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 584dac86183a41d9b3bea1c2085e62fa)
I20260709 12:27:27.925976 30683 raft_consensus.cc:515] T 4685c2e824c14fc1bcd45c16169798cc P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:27.928334 30661 raft_consensus.cc:493] T 9633e3b1d78f403097e4a00b7d4b869b P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 584dac86183a41d9b3bea1c2085e62fa)
I20260709 12:27:27.928951 30661 raft_consensus.cc:515] T 9633e3b1d78f403097e4a00b7d4b869b P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:27:27.931237 30661 leader_election.cc:290] T 9633e3b1d78f403097e4a00b7d4b869b P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209), 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033)
I20260709 12:27:27.933156 30683 leader_election.cc:290] T 4685c2e824c14fc1bcd45c16169798cc P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033), d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825)
I20260709 12:27:27.934716 29662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9633e3b1d78f403097e4a00b7d4b869b" candidate_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" is_pre_election: true
W20260709 12:27:27.962710 29708 leader_election.cc:336] T 4685c2e824c14fc1bcd45c16169798cc P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111)
W20260709 12:27:27.963232 29708 leader_election.cc:336] T 9633e3b1d78f403097e4a00b7d4b869b P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111)
I20260709 12:27:27.963573 29708 leader_election.cc:304] T 9633e3b1d78f403097e4a00b7d4b869b P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c6610c00ffb948ce9c34f41dec3d49e8; no voters: 584dac86183a41d9b3bea1c2085e62fa, d13c58ad8ac441c4a9b396eb04b1a476
I20260709 12:27:27.966996 30632 raft_consensus.cc:2749] T 9633e3b1d78f403097e4a00b7d4b869b P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260709 12:27:27.967058 29708 consensus_peers.cc:597] T 43c7bfb33af74d4f912e5153e8a54b3e P c6610c00ffb948ce9c34f41dec3d49e8 -> Peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Couldn't send request to peer d73d92fb35234d16824c55689b5fce28. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.66:36825: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:27.967018 29634 consensus_peers.cc:597] T 7ae3b542012d4f35a9c363eee22976ec P d13c58ad8ac441c4a9b396eb04b1a476 -> Peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Couldn't send request to peer d73d92fb35234d16824c55689b5fce28. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.66:36825: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:27.983553 29708 leader_election.cc:336] T 4685c2e824c14fc1bcd45c16169798cc P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Network error: Client connection negotiation failed: client connection to 127.16.29.66:36825: connect: Connection refused (error 111)
I20260709 12:27:27.984427 29708 leader_election.cc:304] T 4685c2e824c14fc1bcd45c16169798cc P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c6610c00ffb948ce9c34f41dec3d49e8; no voters: 584dac86183a41d9b3bea1c2085e62fa, d73d92fb35234d16824c55689b5fce28
I20260709 12:27:27.985219 30681 raft_consensus.cc:2749] T 4685c2e824c14fc1bcd45c16169798cc P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:27:28.050357 30681 raft_consensus.cc:493] T 6a30685caefb4d0e80fa4ea87d70760c P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 584dac86183a41d9b3bea1c2085e62fa)
I20260709 12:27:28.050956 30681 raft_consensus.cc:515] T 6a30685caefb4d0e80fa4ea87d70760c P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:28.053653 30681 leader_election.cc:290] T 6a30685caefb4d0e80fa4ea87d70760c P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033), d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825)
I20260709 12:27:28.059906 30681 raft_consensus.cc:493] T 592801934cad4789b9ce8dc09f1470bd P c6610c00ffb948ce9c34f41dec3d49e8 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 584dac86183a41d9b3bea1c2085e62fa)
I20260709 12:27:28.060541 30681 raft_consensus.cc:515] T 592801934cad4789b9ce8dc09f1470bd P c6610c00ffb948ce9c34f41dec3d49e8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:28.063000 30684 raft_consensus.cc:493] T 9633e3b1d78f403097e4a00b7d4b869b P d13c58ad8ac441c4a9b396eb04b1a476 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 584dac86183a41d9b3bea1c2085e62fa)
I20260709 12:27:28.063639 30684 raft_consensus.cc:515] T 9633e3b1d78f403097e4a00b7d4b869b P d13c58ad8ac441c4a9b396eb04b1a476 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:27:28.064260 30632 raft_consensus.cc:493] T 1d1c1f58c4d340ac951dfa403d34138a P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 584dac86183a41d9b3bea1c2085e62fa)
I20260709 12:27:28.064821 30632 raft_consensus.cc:515] T 1d1c1f58c4d340ac951dfa403d34138a P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:28.066437 30684 leader_election.cc:290] T 9633e3b1d78f403097e4a00b7d4b869b P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895), 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033)
I20260709 12:27:28.067341 30632 leader_election.cc:290] T 1d1c1f58c4d340ac951dfa403d34138a P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033), d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825)
W20260709 12:27:28.068471 29708 leader_election.cc:336] T 6a30685caefb4d0e80fa4ea87d70760c P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111)
I20260709 12:27:28.071516 30632 raft_consensus.cc:493] T b3c02c910d2b43cc9265bb1d98491cd3 P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 584dac86183a41d9b3bea1c2085e62fa)
W20260709 12:27:28.072189 29634 leader_election.cc:336] T 9633e3b1d78f403097e4a00b7d4b869b P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111)
I20260709 12:27:28.072966 30681 leader_election.cc:290] T 592801934cad4789b9ce8dc09f1470bd P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033), d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825)
I20260709 12:27:28.072216 30632 raft_consensus.cc:515] T b3c02c910d2b43cc9265bb1d98491cd3 P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:28.075595 30632 leader_election.cc:290] T b3c02c910d2b43cc9265bb1d98491cd3 P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033), d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825)
W20260709 12:27:28.077536 29708 leader_election.cc:336] T 6a30685caefb4d0e80fa4ea87d70760c P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Network error: Client connection negotiation failed: client connection to 127.16.29.66:36825: connect: Connection refused (error 111)
I20260709 12:27:28.078045 29708 leader_election.cc:304] T 6a30685caefb4d0e80fa4ea87d70760c P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c6610c00ffb948ce9c34f41dec3d49e8; no voters: 584dac86183a41d9b3bea1c2085e62fa, d73d92fb35234d16824c55689b5fce28
I20260709 12:27:28.079279 30681 raft_consensus.cc:2749] T 6a30685caefb4d0e80fa4ea87d70760c P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:27:28.085673 29743 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9633e3b1d78f403097e4a00b7d4b869b" candidate_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" is_pre_election: true
I20260709 12:27:28.086472 29743 raft_consensus.cc:2468] T 9633e3b1d78f403097e4a00b7d4b869b P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d13c58ad8ac441c4a9b396eb04b1a476 in term 2.
I20260709 12:27:28.087632 30661 raft_consensus.cc:493] T 6c8b2fa9f2c044ec9eec97c72b561042 P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 584dac86183a41d9b3bea1c2085e62fa)
I20260709 12:27:28.088356 30661 raft_consensus.cc:515] T 6c8b2fa9f2c044ec9eec97c72b561042 P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:28.091471 29635 leader_election.cc:304] T 9633e3b1d78f403097e4a00b7d4b869b P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c6610c00ffb948ce9c34f41dec3d49e8, d13c58ad8ac441c4a9b396eb04b1a476; no voters: 584dac86183a41d9b3bea1c2085e62fa
I20260709 12:27:28.092763 30684 raft_consensus.cc:2804] T 9633e3b1d78f403097e4a00b7d4b869b P d13c58ad8ac441c4a9b396eb04b1a476 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:27:28.093343 30684 raft_consensus.cc:493] T 9633e3b1d78f403097e4a00b7d4b869b P d13c58ad8ac441c4a9b396eb04b1a476 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 584dac86183a41d9b3bea1c2085e62fa)
I20260709 12:27:28.093726 30684 raft_consensus.cc:3060] T 9633e3b1d78f403097e4a00b7d4b869b P d13c58ad8ac441c4a9b396eb04b1a476 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:27:28.090831 30661 leader_election.cc:290] T 6c8b2fa9f2c044ec9eec97c72b561042 P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033), d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825)
W20260709 12:27:28.105403 29708 leader_election.cc:336] T 1d1c1f58c4d340ac951dfa403d34138a P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Network error: Client connection negotiation failed: client connection to 127.16.29.66:36825: connect: Connection refused (error 111)
I20260709 12:27:28.105259 30684 raft_consensus.cc:515] T 9633e3b1d78f403097e4a00b7d4b869b P d13c58ad8ac441c4a9b396eb04b1a476 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
W20260709 12:27:28.106266 29708 leader_election.cc:336] T 592801934cad4789b9ce8dc09f1470bd P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Network error: Client connection negotiation failed: client connection to 127.16.29.66:36825: connect: Connection refused (error 111)
W20260709 12:27:28.106765 29708 leader_election.cc:336] T b3c02c910d2b43cc9265bb1d98491cd3 P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Network error: Client connection negotiation failed: client connection to 127.16.29.66:36825: connect: Connection refused (error 111)
W20260709 12:27:28.109416 29708 leader_election.cc:336] T 592801934cad4789b9ce8dc09f1470bd P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111)
I20260709 12:27:28.109905 29708 leader_election.cc:304] T 592801934cad4789b9ce8dc09f1470bd P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c6610c00ffb948ce9c34f41dec3d49e8; no voters: 584dac86183a41d9b3bea1c2085e62fa, d73d92fb35234d16824c55689b5fce28
W20260709 12:27:28.110966 29708 leader_election.cc:336] T 1d1c1f58c4d340ac951dfa403d34138a P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111)
I20260709 12:27:28.111410 30661 raft_consensus.cc:2749] T 592801934cad4789b9ce8dc09f1470bd P c6610c00ffb948ce9c34f41dec3d49e8 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:27:28.112907 30684 leader_election.cc:290] T 9633e3b1d78f403097e4a00b7d4b869b P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 3 election: Requested vote from peers c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895), 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033)
I20260709 12:27:28.114360 30661 raft_consensus.cc:493] T f76101eb30f14b98a9aa8c41e0d6f831 P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 584dac86183a41d9b3bea1c2085e62fa)
I20260709 12:27:28.114950 30661 raft_consensus.cc:515] T f76101eb30f14b98a9aa8c41e0d6f831 P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
W20260709 12:27:28.116117 29634 leader_election.cc:336] T 9633e3b1d78f403097e4a00b7d4b869b P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111)
I20260709 12:27:28.111436 29708 leader_election.cc:304] T 1d1c1f58c4d340ac951dfa403d34138a P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c6610c00ffb948ce9c34f41dec3d49e8; no voters: 584dac86183a41d9b3bea1c2085e62fa, d73d92fb35234d16824c55689b5fce28
I20260709 12:27:28.118273 30681 raft_consensus.cc:2749] T 1d1c1f58c4d340ac951dfa403d34138a P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:27:28.126055 29662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f76101eb30f14b98a9aa8c41e0d6f831" candidate_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" is_pre_election: true
I20260709 12:27:28.126843 29662 raft_consensus.cc:2468] T f76101eb30f14b98a9aa8c41e0d6f831 P d13c58ad8ac441c4a9b396eb04b1a476 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6610c00ffb948ce9c34f41dec3d49e8 in term 2.
I20260709 12:27:28.129097 29709 leader_election.cc:304] T f76101eb30f14b98a9aa8c41e0d6f831 P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c6610c00ffb948ce9c34f41dec3d49e8, d13c58ad8ac441c4a9b396eb04b1a476; no voters: 
I20260709 12:27:28.130533 30681 raft_consensus.cc:2804] T f76101eb30f14b98a9aa8c41e0d6f831 P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:27:28.130995 30681 raft_consensus.cc:493] T f76101eb30f14b98a9aa8c41e0d6f831 P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 584dac86183a41d9b3bea1c2085e62fa)
I20260709 12:27:28.133970 30681 raft_consensus.cc:3060] T f76101eb30f14b98a9aa8c41e0d6f831 P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:27:28.139575 29708 leader_election.cc:336] T 6c8b2fa9f2c044ec9eec97c72b561042 P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Network error: Client connection negotiation failed: client connection to 127.16.29.66:36825: connect: Connection refused (error 111)
I20260709 12:27:28.141656 30632 raft_consensus.cc:493] T fb6e78ccc01c4707af34cddf81619eab P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 584dac86183a41d9b3bea1c2085e62fa)
I20260709 12:27:28.142309 29743 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9633e3b1d78f403097e4a00b7d4b869b" candidate_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c6610c00ffb948ce9c34f41dec3d49e8"
I20260709 12:27:28.142200 30632 raft_consensus.cc:515] T fb6e78ccc01c4707af34cddf81619eab P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:28.142998 29743 raft_consensus.cc:3060] T 9633e3b1d78f403097e4a00b7d4b869b P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:27:28.144800 30632 leader_election.cc:290] T fb6e78ccc01c4707af34cddf81619eab P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033), d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825)
I20260709 12:27:28.147264 30661 leader_election.cc:290] T f76101eb30f14b98a9aa8c41e0d6f831 P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209), 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033)
I20260709 12:27:28.154093 29743 raft_consensus.cc:2468] T 9633e3b1d78f403097e4a00b7d4b869b P c6610c00ffb948ce9c34f41dec3d49e8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d13c58ad8ac441c4a9b396eb04b1a476 in term 3.
W20260709 12:27:28.155254 29708 leader_election.cc:336] T b3c02c910d2b43cc9265bb1d98491cd3 P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111)
I20260709 12:27:28.155764 29708 leader_election.cc:304] T b3c02c910d2b43cc9265bb1d98491cd3 P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c6610c00ffb948ce9c34f41dec3d49e8; no voters: 584dac86183a41d9b3bea1c2085e62fa, d73d92fb35234d16824c55689b5fce28
I20260709 12:27:28.157301 30684 raft_consensus.cc:493] T 71e2de235adf46d9a2407659080d42ef P d13c58ad8ac441c4a9b396eb04b1a476 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 584dac86183a41d9b3bea1c2085e62fa)
I20260709 12:27:28.157941 30684 raft_consensus.cc:515] T 71e2de235adf46d9a2407659080d42ef P d13c58ad8ac441c4a9b396eb04b1a476 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:27:28.161872 30681 raft_consensus.cc:515] T f76101eb30f14b98a9aa8c41e0d6f831 P c6610c00ffb948ce9c34f41dec3d49e8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:27:28.164970 30661 raft_consensus.cc:2749] T b3c02c910d2b43cc9265bb1d98491cd3 P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:27:28.165174 29662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f76101eb30f14b98a9aa8c41e0d6f831" candidate_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d13c58ad8ac441c4a9b396eb04b1a476"
I20260709 12:27:28.166074 29662 raft_consensus.cc:3060] T f76101eb30f14b98a9aa8c41e0d6f831 P d13c58ad8ac441c4a9b396eb04b1a476 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:27:28.166270 30684 leader_election.cc:290] T 71e2de235adf46d9a2407659080d42ef P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033), d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825)
W20260709 12:27:28.156841 29708 leader_election.cc:336] T 6c8b2fa9f2c044ec9eec97c72b561042 P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111)
I20260709 12:27:28.167457 29708 leader_election.cc:304] T 6c8b2fa9f2c044ec9eec97c72b561042 P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c6610c00ffb948ce9c34f41dec3d49e8; no voters: 584dac86183a41d9b3bea1c2085e62fa, d73d92fb35234d16824c55689b5fce28
I20260709 12:27:28.168182 30661 raft_consensus.cc:2749] T 6c8b2fa9f2c044ec9eec97c72b561042 P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:27:28.177971 29635 leader_election.cc:304] T 9633e3b1d78f403097e4a00b7d4b869b P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c6610c00ffb948ce9c34f41dec3d49e8, d13c58ad8ac441c4a9b396eb04b1a476; no voters: 584dac86183a41d9b3bea1c2085e62fa
I20260709 12:27:28.179217 30684 raft_consensus.cc:2804] T 9633e3b1d78f403097e4a00b7d4b869b P d13c58ad8ac441c4a9b396eb04b1a476 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:27:28.179983 30684 raft_consensus.cc:697] T 9633e3b1d78f403097e4a00b7d4b869b P d13c58ad8ac441c4a9b396eb04b1a476 [term 3 LEADER]: Becoming Leader. State: Replica: d13c58ad8ac441c4a9b396eb04b1a476, State: Running, Role: LEADER
I20260709 12:27:28.180907 30684 consensus_queue.cc:237] T 9633e3b1d78f403097e4a00b7d4b869b P d13c58ad8ac441c4a9b396eb04b1a476 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:27:28.189730 30681 leader_election.cc:290] T f76101eb30f14b98a9aa8c41e0d6f831 P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 3 election: Requested vote from peers d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209), 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033)
I20260709 12:27:28.185974 29662 raft_consensus.cc:2468] T f76101eb30f14b98a9aa8c41e0d6f831 P d13c58ad8ac441c4a9b396eb04b1a476 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6610c00ffb948ce9c34f41dec3d49e8 in term 3.
I20260709 12:27:28.201376 29709 leader_election.cc:304] T f76101eb30f14b98a9aa8c41e0d6f831 P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c6610c00ffb948ce9c34f41dec3d49e8, d13c58ad8ac441c4a9b396eb04b1a476; no voters: 
I20260709 12:27:28.202246 30681 raft_consensus.cc:2804] T f76101eb30f14b98a9aa8c41e0d6f831 P c6610c00ffb948ce9c34f41dec3d49e8 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:27:28.202765 30681 raft_consensus.cc:697] T f76101eb30f14b98a9aa8c41e0d6f831 P c6610c00ffb948ce9c34f41dec3d49e8 [term 3 LEADER]: Becoming Leader. State: Replica: c6610c00ffb948ce9c34f41dec3d49e8, State: Running, Role: LEADER
I20260709 12:27:28.203652 30681 consensus_queue.cc:237] T f76101eb30f14b98a9aa8c41e0d6f831 P c6610c00ffb948ce9c34f41dec3d49e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
W20260709 12:27:28.205075 29708 leader_election.cc:336] T fb6e78ccc01c4707af34cddf81619eab P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Network error: Client connection negotiation failed: client connection to 127.16.29.66:36825: connect: Connection refused (error 111)
W20260709 12:27:28.211464 29708 leader_election.cc:336] T fb6e78ccc01c4707af34cddf81619eab P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111)
I20260709 12:27:28.213611 29708 leader_election.cc:304] T fb6e78ccc01c4707af34cddf81619eab P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c6610c00ffb948ce9c34f41dec3d49e8; no voters: 584dac86183a41d9b3bea1c2085e62fa, d73d92fb35234d16824c55689b5fce28
I20260709 12:27:28.215283 30661 raft_consensus.cc:2749] T fb6e78ccc01c4707af34cddf81619eab P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260709 12:27:28.216454 29708 leader_election.cc:336] T f76101eb30f14b98a9aa8c41e0d6f831 P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111)
W20260709 12:27:28.217387 29634 leader_election.cc:336] T 71e2de235adf46d9a2407659080d42ef P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111)
W20260709 12:27:28.220448 29634 leader_election.cc:336] T 71e2de235adf46d9a2407659080d42ef P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Network error: Client connection negotiation failed: client connection to 127.16.29.66:36825: connect: Connection refused (error 111)
I20260709 12:27:28.230002 29634 leader_election.cc:304] T 71e2de235adf46d9a2407659080d42ef P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d13c58ad8ac441c4a9b396eb04b1a476; no voters: 584dac86183a41d9b3bea1c2085e62fa, d73d92fb35234d16824c55689b5fce28
W20260709 12:27:28.227643 29708 leader_election.cc:336] T f76101eb30f14b98a9aa8c41e0d6f831 P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111)
I20260709 12:27:28.231843 30684 raft_consensus.cc:2749] T 71e2de235adf46d9a2407659080d42ef P d13c58ad8ac441c4a9b396eb04b1a476 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:27:28.239385 29418 catalog_manager.cc:5697] T 9633e3b1d78f403097e4a00b7d4b869b P d13c58ad8ac441c4a9b396eb04b1a476 reported cstate change: term changed from 2 to 3, leader changed from 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65) to d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67). New cstate: current_term: 3 leader_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:28.270162 29419 catalog_manager.cc:5697] T f76101eb30f14b98a9aa8c41e0d6f831 P c6610c00ffb948ce9c34f41dec3d49e8 reported cstate change: term changed from 2 to 3, leader changed from 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65) to c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68). New cstate: current_term: 3 leader_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:28.308468 30661 raft_consensus.cc:493] T 3f41ce441fb047c8bbb741b1b9cbc0ec P c6610c00ffb948ce9c34f41dec3d49e8 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 584dac86183a41d9b3bea1c2085e62fa)
I20260709 12:27:28.309091 30661 raft_consensus.cc:515] T 3f41ce441fb047c8bbb741b1b9cbc0ec P c6610c00ffb948ce9c34f41dec3d49e8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:27:28.311590 30661 leader_election.cc:290] T 3f41ce441fb047c8bbb741b1b9cbc0ec P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209), 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033)
I20260709 12:27:28.313406 29662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3f41ce441fb047c8bbb741b1b9cbc0ec" candidate_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" is_pre_election: true
W20260709 12:27:28.326011 29708 leader_election.cc:336] T 3f41ce441fb047c8bbb741b1b9cbc0ec P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111)
I20260709 12:27:28.326541 29708 leader_election.cc:304] T 3f41ce441fb047c8bbb741b1b9cbc0ec P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c6610c00ffb948ce9c34f41dec3d49e8; no voters: 584dac86183a41d9b3bea1c2085e62fa, d13c58ad8ac441c4a9b396eb04b1a476
I20260709 12:27:28.327311 30661 raft_consensus.cc:2749] T 3f41ce441fb047c8bbb741b1b9cbc0ec P c6610c00ffb948ce9c34f41dec3d49e8 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:27:28.356235 30682 raft_consensus.cc:493] T 3f41ce441fb047c8bbb741b1b9cbc0ec P d13c58ad8ac441c4a9b396eb04b1a476 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 584dac86183a41d9b3bea1c2085e62fa)
I20260709 12:27:28.356806 30682 raft_consensus.cc:515] T 3f41ce441fb047c8bbb741b1b9cbc0ec P d13c58ad8ac441c4a9b396eb04b1a476 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:27:28.359025 30682 leader_election.cc:290] T 3f41ce441fb047c8bbb741b1b9cbc0ec P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895), 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033)
W20260709 12:27:28.366182 29634 leader_election.cc:336] T 3f41ce441fb047c8bbb741b1b9cbc0ec P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111)
I20260709 12:27:28.368523 29743 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3f41ce441fb047c8bbb741b1b9cbc0ec" candidate_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" is_pre_election: true
I20260709 12:27:28.369194 29743 raft_consensus.cc:2468] T 3f41ce441fb047c8bbb741b1b9cbc0ec P c6610c00ffb948ce9c34f41dec3d49e8 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d13c58ad8ac441c4a9b396eb04b1a476 in term 3.
I20260709 12:27:28.370663 29635 leader_election.cc:304] T 3f41ce441fb047c8bbb741b1b9cbc0ec P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c6610c00ffb948ce9c34f41dec3d49e8, d13c58ad8ac441c4a9b396eb04b1a476; no voters: 584dac86183a41d9b3bea1c2085e62fa
I20260709 12:27:28.374132 30682 raft_consensus.cc:2804] T 3f41ce441fb047c8bbb741b1b9cbc0ec P d13c58ad8ac441c4a9b396eb04b1a476 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:27:28.374711 30682 raft_consensus.cc:493] T 3f41ce441fb047c8bbb741b1b9cbc0ec P d13c58ad8ac441c4a9b396eb04b1a476 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 584dac86183a41d9b3bea1c2085e62fa)
I20260709 12:27:28.375101 30682 raft_consensus.cc:3060] T 3f41ce441fb047c8bbb741b1b9cbc0ec P d13c58ad8ac441c4a9b396eb04b1a476 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:27:28.393652 30682 raft_consensus.cc:515] T 3f41ce441fb047c8bbb741b1b9cbc0ec P d13c58ad8ac441c4a9b396eb04b1a476 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:27:28.401281 29743 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3f41ce441fb047c8bbb741b1b9cbc0ec" candidate_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "c6610c00ffb948ce9c34f41dec3d49e8"
I20260709 12:27:28.401945 29743 raft_consensus.cc:3060] T 3f41ce441fb047c8bbb741b1b9cbc0ec P c6610c00ffb948ce9c34f41dec3d49e8 [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:27:28.419471 29634 leader_election.cc:336] T 3f41ce441fb047c8bbb741b1b9cbc0ec P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111)
I20260709 12:27:28.420464 30682 leader_election.cc:290] T 3f41ce441fb047c8bbb741b1b9cbc0ec P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 4 election: Requested vote from peers c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895), 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033)
I20260709 12:27:28.421811 29743 raft_consensus.cc:2468] T 3f41ce441fb047c8bbb741b1b9cbc0ec P c6610c00ffb948ce9c34f41dec3d49e8 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d13c58ad8ac441c4a9b396eb04b1a476 in term 4.
I20260709 12:27:28.423518 29635 leader_election.cc:304] T 3f41ce441fb047c8bbb741b1b9cbc0ec P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c6610c00ffb948ce9c34f41dec3d49e8, d13c58ad8ac441c4a9b396eb04b1a476; no voters: 584dac86183a41d9b3bea1c2085e62fa
I20260709 12:27:28.425863 30682 raft_consensus.cc:2804] T 3f41ce441fb047c8bbb741b1b9cbc0ec P d13c58ad8ac441c4a9b396eb04b1a476 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:27:28.426457 30682 raft_consensus.cc:697] T 3f41ce441fb047c8bbb741b1b9cbc0ec P d13c58ad8ac441c4a9b396eb04b1a476 [term 4 LEADER]: Becoming Leader. State: Replica: d13c58ad8ac441c4a9b396eb04b1a476, State: Running, Role: LEADER
I20260709 12:27:28.427574 30682 consensus_queue.cc:237] T 3f41ce441fb047c8bbb741b1b9cbc0ec P d13c58ad8ac441c4a9b396eb04b1a476 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:27:28.501791 29419 catalog_manager.cc:5697] T 3f41ce441fb047c8bbb741b1b9cbc0ec P d13c58ad8ac441c4a9b396eb04b1a476 reported cstate change: term changed from 3 to 4, leader changed from 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65) to d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67). New cstate: current_term: 4 leader_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:28.673827 30682 raft_consensus.cc:493] T 16649fd17d434e3c95e9d585d2404659 P d13c58ad8ac441c4a9b396eb04b1a476 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d73d92fb35234d16824c55689b5fce28)
I20260709 12:27:28.674517 30682 raft_consensus.cc:515] T 16649fd17d434e3c95e9d585d2404659 P d13c58ad8ac441c4a9b396eb04b1a476 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:27:28.676959 30682 leader_election.cc:290] T 16649fd17d434e3c95e9d585d2404659 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033), d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825)
W20260709 12:27:28.684692 29634 leader_election.cc:336] T 16649fd17d434e3c95e9d585d2404659 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Network error: Client connection negotiation failed: client connection to 127.16.29.66:36825: connect: Connection refused (error 111)
W20260709 12:27:28.685396 29634 leader_election.cc:336] T 16649fd17d434e3c95e9d585d2404659 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111)
I20260709 12:27:28.685792 29634 leader_election.cc:304] T 16649fd17d434e3c95e9d585d2404659 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d13c58ad8ac441c4a9b396eb04b1a476; no voters: 584dac86183a41d9b3bea1c2085e62fa, d73d92fb35234d16824c55689b5fce28
I20260709 12:27:28.686547 30682 raft_consensus.cc:2749] T 16649fd17d434e3c95e9d585d2404659 P d13c58ad8ac441c4a9b396eb04b1a476 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:27:28.707137 29743 raft_consensus.cc:1275] T 9633e3b1d78f403097e4a00b7d4b869b P c6610c00ffb948ce9c34f41dec3d49e8 [term 3 FOLLOWER]: Refusing update from remote peer d13c58ad8ac441c4a9b396eb04b1a476: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:27:28.708698 30682 consensus_queue.cc:1048] T 9633e3b1d78f403097e4a00b7d4b869b P d13c58ad8ac441c4a9b396eb04b1a476 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:27:28.714159 30682 raft_consensus.cc:493] T c1c8411289a74b5685700229d245ff75 P d13c58ad8ac441c4a9b396eb04b1a476 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d73d92fb35234d16824c55689b5fce28)
I20260709 12:27:28.714720 30682 raft_consensus.cc:515] T c1c8411289a74b5685700229d245ff75 P d13c58ad8ac441c4a9b396eb04b1a476 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:27:28.718111 30682 leader_election.cc:290] T c1c8411289a74b5685700229d245ff75 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825), c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895)
I20260709 12:27:28.718822 29742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c1c8411289a74b5685700229d245ff75" candidate_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" is_pre_election: true
W20260709 12:27:28.723500 29634 leader_election.cc:336] T c1c8411289a74b5685700229d245ff75 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Network error: Client connection negotiation failed: client connection to 127.16.29.66:36825: connect: Connection refused (error 111)
I20260709 12:27:28.723974 29634 leader_election.cc:304] T c1c8411289a74b5685700229d245ff75 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d13c58ad8ac441c4a9b396eb04b1a476; no voters: c6610c00ffb948ce9c34f41dec3d49e8, d73d92fb35234d16824c55689b5fce28
I20260709 12:27:28.725435 30684 raft_consensus.cc:2749] T c1c8411289a74b5685700229d245ff75 P d13c58ad8ac441c4a9b396eb04b1a476 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260709 12:27:28.731453 29708 consensus_peers.cc:597] T f76101eb30f14b98a9aa8c41e0d6f831 P c6610c00ffb948ce9c34f41dec3d49e8 -> Peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Couldn't send request to peer 584dac86183a41d9b3bea1c2085e62fa. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:28.746227 30632 raft_consensus.cc:493] T 00c6cf271c91400aa834467def9af9b1 P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d73d92fb35234d16824c55689b5fce28)
I20260709 12:27:28.746837 30632 raft_consensus.cc:515] T 00c6cf271c91400aa834467def9af9b1 P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:27:28.748901 30632 leader_election.cc:290] T 00c6cf271c91400aa834467def9af9b1 P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825), d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209)
I20260709 12:27:28.751847 29662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00c6cf271c91400aa834467def9af9b1" candidate_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" is_pre_election: true
I20260709 12:27:28.770486 29666 raft_consensus.cc:1275] T f76101eb30f14b98a9aa8c41e0d6f831 P d13c58ad8ac441c4a9b396eb04b1a476 [term 3 FOLLOWER]: Refusing update from remote peer c6610c00ffb948ce9c34f41dec3d49e8: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:27:28.773064 30661 consensus_queue.cc:1048] T f76101eb30f14b98a9aa8c41e0d6f831 P c6610c00ffb948ce9c34f41dec3d49e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20260709 12:27:28.785565 29708 leader_election.cc:336] T 00c6cf271c91400aa834467def9af9b1 P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Network error: Client connection negotiation failed: client connection to 127.16.29.66:36825: connect: Connection refused (error 111)
I20260709 12:27:28.786599 29708 leader_election.cc:304] T 00c6cf271c91400aa834467def9af9b1 P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c6610c00ffb948ce9c34f41dec3d49e8; no voters: d13c58ad8ac441c4a9b396eb04b1a476, d73d92fb35234d16824c55689b5fce28
I20260709 12:27:28.789143 30661 raft_consensus.cc:2749] T 00c6cf271c91400aa834467def9af9b1 P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:27:28.797770 30661 raft_consensus.cc:493] T f8c01e05981747a395e5483b551297dd P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d73d92fb35234d16824c55689b5fce28)
I20260709 12:27:28.798365 30661 raft_consensus.cc:515] T f8c01e05981747a395e5483b551297dd P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:28.800412 30661 leader_election.cc:290] T f8c01e05981747a395e5483b551297dd P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033), d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825)
I20260709 12:27:28.845333 30632 raft_consensus.cc:493] T 5be58d64d45e40d7981018b84a01b1ed P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d73d92fb35234d16824c55689b5fce28)
I20260709 12:27:28.845837 30632 raft_consensus.cc:515] T 5be58d64d45e40d7981018b84a01b1ed P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
W20260709 12:27:28.873413 29708 leader_election.cc:336] T f8c01e05981747a395e5483b551297dd P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111)
W20260709 12:27:28.885058 16501 thread.cc:527] Waited for 1000ms trying to join with diag-logger (tid 29618)
W20260709 12:27:28.902611 29708 leader_election.cc:336] T 5be58d64d45e40d7981018b84a01b1ed P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111)
I20260709 12:27:28.906575 30681 raft_consensus.cc:493] T c1c8411289a74b5685700229d245ff75 P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d73d92fb35234d16824c55689b5fce28)
W20260709 12:27:28.906670 29708 leader_election.cc:336] T f8c01e05981747a395e5483b551297dd P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Network error: Client connection negotiation failed: client connection to 127.16.29.66:36825: connect: Connection refused (error 111)
I20260709 12:27:28.912504 30632 leader_election.cc:290] T 5be58d64d45e40d7981018b84a01b1ed P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033), d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825)
I20260709 12:27:28.912441 29708 leader_election.cc:304] T f8c01e05981747a395e5483b551297dd P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c6610c00ffb948ce9c34f41dec3d49e8; no voters: 584dac86183a41d9b3bea1c2085e62fa, d73d92fb35234d16824c55689b5fce28
I20260709 12:27:28.916090 30683 raft_consensus.cc:493] T 55061e62184b45f0bb1726adc1a4af81 P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d73d92fb35234d16824c55689b5fce28)
I20260709 12:27:28.916941 30683 raft_consensus.cc:515] T 55061e62184b45f0bb1726adc1a4af81 P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:27:28.907320 30681 raft_consensus.cc:515] T c1c8411289a74b5685700229d245ff75 P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:27:28.925419 30681 leader_election.cc:290] T c1c8411289a74b5685700229d245ff75 P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825), d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209)
I20260709 12:27:28.926569 30681 raft_consensus.cc:2749] T f8c01e05981747a395e5483b551297dd P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:27:28.973317 30632 raft_consensus.cc:493] T cdf706ffabc34902ad3d1bf2cee7889e P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d73d92fb35234d16824c55689b5fce28)
I20260709 12:27:28.974083 30632 raft_consensus.cc:515] T cdf706ffabc34902ad3d1bf2cee7889e P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
W20260709 12:27:28.991216 29708 leader_election.cc:336] T 5be58d64d45e40d7981018b84a01b1ed P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Network error: Client connection negotiation failed: client connection to 127.16.29.66:36825: connect: Connection refused (error 111)
I20260709 12:27:28.997862 29708 leader_election.cc:304] T 5be58d64d45e40d7981018b84a01b1ed P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c6610c00ffb948ce9c34f41dec3d49e8; no voters: 584dac86183a41d9b3bea1c2085e62fa, d73d92fb35234d16824c55689b5fce28
W20260709 12:27:29.003090 29708 leader_election.cc:336] T 55061e62184b45f0bb1726adc1a4af81 P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Network error: Client connection negotiation failed: client connection to 127.16.29.66:36825: connect: Connection refused (error 111)
W20260709 12:27:29.012787 29708 leader_election.cc:336] T c1c8411289a74b5685700229d245ff75 P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Network error: Client connection negotiation failed: client connection to 127.16.29.66:36825: connect: Connection refused (error 111)
I20260709 12:27:29.000808 30661 raft_consensus.cc:493] T a3dcbb9c64bf4b5bb6680e8cb81af93e P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d73d92fb35234d16824c55689b5fce28)
I20260709 12:27:29.017653 30661 raft_consensus.cc:515] T a3dcbb9c64bf4b5bb6680e8cb81af93e P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:29.008380 30703 raft_consensus.cc:2749] T 5be58d64d45e40d7981018b84a01b1ed P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:27:29.014873 30683 leader_election.cc:290] T 55061e62184b45f0bb1726adc1a4af81 P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825), d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209)
I20260709 12:27:28.927474 30681 raft_consensus.cc:493] T 137e03144e754f6e9da4f0544513d27c P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d73d92fb35234d16824c55689b5fce28)
I20260709 12:27:29.043815 30681 raft_consensus.cc:515] T 137e03144e754f6e9da4f0544513d27c P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:27:29.049926 30661 leader_election.cc:290] T a3dcbb9c64bf4b5bb6680e8cb81af93e P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033), d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825)
W20260709 12:27:29.050267 29708 leader_election.cc:336] T cdf706ffabc34902ad3d1bf2cee7889e P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Network error: Client connection negotiation failed: client connection to 127.16.29.66:36825: connect: Connection refused (error 111)
I20260709 12:27:29.050616 30632 leader_election.cc:290] T cdf706ffabc34902ad3d1bf2cee7889e P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825), d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209)
I20260709 12:27:29.046074 30681 leader_election.cc:290] T 137e03144e754f6e9da4f0544513d27c P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825), d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209)
W20260709 12:27:29.056201 29708 leader_election.cc:336] T a3dcbb9c64bf4b5bb6680e8cb81af93e P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111)
W20260709 12:27:29.061160 29708 leader_election.cc:336] T 137e03144e754f6e9da4f0544513d27c P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Network error: Client connection negotiation failed: client connection to 127.16.29.66:36825: connect: Connection refused (error 111)
W20260709 12:27:29.067562 29708 leader_election.cc:336] T a3dcbb9c64bf4b5bb6680e8cb81af93e P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Network error: Client connection negotiation failed: client connection to 127.16.29.66:36825: connect: Connection refused (error 111)
I20260709 12:27:29.072444 29708 leader_election.cc:304] T a3dcbb9c64bf4b5bb6680e8cb81af93e P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c6610c00ffb948ce9c34f41dec3d49e8; no voters: 584dac86183a41d9b3bea1c2085e62fa, d73d92fb35234d16824c55689b5fce28
I20260709 12:27:29.073458 30702 raft_consensus.cc:2749] T a3dcbb9c64bf4b5bb6680e8cb81af93e P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:27:29.075240 30684 raft_consensus.cc:493] T 00c6cf271c91400aa834467def9af9b1 P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d73d92fb35234d16824c55689b5fce28)
I20260709 12:27:29.075855 30684 raft_consensus.cc:515] T 00c6cf271c91400aa834467def9af9b1 P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:27:29.078316 30684 leader_election.cc:290] T 00c6cf271c91400aa834467def9af9b1 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825), c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895)
I20260709 12:27:29.079046 30677 raft_consensus.cc:493] T 2707271fd46f456aa765026642a3efa9 P d13c58ad8ac441c4a9b396eb04b1a476 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d73d92fb35234d16824c55689b5fce28)
I20260709 12:27:29.079535 30677 raft_consensus.cc:515] T 2707271fd46f456aa765026642a3efa9 P d13c58ad8ac441c4a9b396eb04b1a476 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:27:29.083098 30702 raft_consensus.cc:493] T 62f8c9472b214576b9002e02536a3ace P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d73d92fb35234d16824c55689b5fce28)
I20260709 12:27:29.081707 30677 leader_election.cc:290] T 2707271fd46f456aa765026642a3efa9 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033), d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825)
I20260709 12:27:29.083828 30702 raft_consensus.cc:515] T 62f8c9472b214576b9002e02536a3ace P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:29.089898 30702 leader_election.cc:290] T 62f8c9472b214576b9002e02536a3ace P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033), d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825)
W20260709 12:27:29.099727 29634 consensus_peers.cc:597] T 9633e3b1d78f403097e4a00b7d4b869b P d13c58ad8ac441c4a9b396eb04b1a476 -> Peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Couldn't send request to peer 584dac86183a41d9b3bea1c2085e62fa. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:29.103173 30684 raft_consensus.cc:493] T abec627108a84083884e64e1eb9dcda6 P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d73d92fb35234d16824c55689b5fce28)
I20260709 12:27:29.103806 30684 raft_consensus.cc:515] T abec627108a84083884e64e1eb9dcda6 P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:27:29.106698 30684 leader_election.cc:290] T abec627108a84083884e64e1eb9dcda6 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033), d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825)
I20260709 12:27:29.108033 30682 raft_consensus.cc:493] T 137e03144e754f6e9da4f0544513d27c P d13c58ad8ac441c4a9b396eb04b1a476 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d73d92fb35234d16824c55689b5fce28)
I20260709 12:27:29.108574 30682 raft_consensus.cc:515] T 137e03144e754f6e9da4f0544513d27c P d13c58ad8ac441c4a9b396eb04b1a476 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:27:29.110451 30682 leader_election.cc:290] T 137e03144e754f6e9da4f0544513d27c P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825), c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895)
I20260709 12:27:29.159354 29666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55061e62184b45f0bb1726adc1a4af81" candidate_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" is_pre_election: true
I20260709 12:27:29.160161 29666 raft_consensus.cc:2468] T 55061e62184b45f0bb1726adc1a4af81 P d13c58ad8ac441c4a9b396eb04b1a476 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6610c00ffb948ce9c34f41dec3d49e8 in term 2.
I20260709 12:27:29.162434 29666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c1c8411289a74b5685700229d245ff75" candidate_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" is_pre_election: true
I20260709 12:27:29.163127 29666 raft_consensus.cc:2468] T c1c8411289a74b5685700229d245ff75 P d13c58ad8ac441c4a9b396eb04b1a476 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6610c00ffb948ce9c34f41dec3d49e8 in term 2.
I20260709 12:27:29.164947 29666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cdf706ffabc34902ad3d1bf2cee7889e" candidate_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" is_pre_election: true
I20260709 12:27:29.166671 29662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "137e03144e754f6e9da4f0544513d27c" candidate_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" is_pre_election: true
I20260709 12:27:29.167281 29662 raft_consensus.cc:2468] T 137e03144e754f6e9da4f0544513d27c P d13c58ad8ac441c4a9b396eb04b1a476 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6610c00ffb948ce9c34f41dec3d49e8 in term 2.
I20260709 12:27:29.170641 30706 raft_consensus.cc:493] T d5fcd1f030614bbe82cf1dc261dfc1c3 P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d73d92fb35234d16824c55689b5fce28)
I20260709 12:27:29.171293 30706 raft_consensus.cc:515] T d5fcd1f030614bbe82cf1dc261dfc1c3 P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:27:29.173424 30706 leader_election.cc:290] T d5fcd1f030614bbe82cf1dc261dfc1c3 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033), d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825)
W20260709 12:27:29.173791 29634 leader_election.cc:336] T 00c6cf271c91400aa834467def9af9b1 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Network error: Client connection negotiation failed: client connection to 127.16.29.66:36825: connect: Connection refused (error 111)
W20260709 12:27:29.177479 29634 leader_election.cc:336] T 2707271fd46f456aa765026642a3efa9 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Network error: Client connection negotiation failed: client connection to 127.16.29.66:36825: connect: Connection refused (error 111)
W20260709 12:27:29.180269 29634 leader_election.cc:336] T 2707271fd46f456aa765026642a3efa9 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111)
I20260709 12:27:29.181372 29634 leader_election.cc:304] T 2707271fd46f456aa765026642a3efa9 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d13c58ad8ac441c4a9b396eb04b1a476; no voters: 584dac86183a41d9b3bea1c2085e62fa, d73d92fb35234d16824c55689b5fce28
I20260709 12:27:29.182567 30709 raft_consensus.cc:2749] T 2707271fd46f456aa765026642a3efa9 P d13c58ad8ac441c4a9b396eb04b1a476 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:27:29.188017 30708 raft_consensus.cc:493] T d21ab1641501475b829863275dc8182a P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d73d92fb35234d16824c55689b5fce28)
I20260709 12:27:29.188592 30708 raft_consensus.cc:515] T d21ab1641501475b829863275dc8182a P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:27:29.191146 30708 leader_election.cc:290] T d21ab1641501475b829863275dc8182a P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033), d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825)
I20260709 12:27:29.192348 30684 raft_consensus.cc:493] T 3615115233c34461b85763151751e475 P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d73d92fb35234d16824c55689b5fce28)
I20260709 12:27:29.192848 30684 raft_consensus.cc:515] T 3615115233c34461b85763151751e475 P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:27:29.195159 30684 leader_election.cc:290] T 3615115233c34461b85763151751e475 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033), d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825)
I20260709 12:27:29.208269 29709 leader_election.cc:304] T 55061e62184b45f0bb1726adc1a4af81 P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c6610c00ffb948ce9c34f41dec3d49e8, d13c58ad8ac441c4a9b396eb04b1a476; no voters: d73d92fb35234d16824c55689b5fce28
I20260709 12:27:29.209484 30702 raft_consensus.cc:2804] T 55061e62184b45f0bb1726adc1a4af81 P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:27:29.209903 30702 raft_consensus.cc:493] T 55061e62184b45f0bb1726adc1a4af81 P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Starting leader election (detected failure of leader d73d92fb35234d16824c55689b5fce28)
I20260709 12:27:29.210301 30702 raft_consensus.cc:3060] T 55061e62184b45f0bb1726adc1a4af81 P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:27:29.219357 30702 raft_consensus.cc:515] T 55061e62184b45f0bb1726adc1a4af81 P c6610c00ffb948ce9c34f41dec3d49e8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:27:29.222347 29709 leader_election.cc:304] T c1c8411289a74b5685700229d245ff75 P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c6610c00ffb948ce9c34f41dec3d49e8, d13c58ad8ac441c4a9b396eb04b1a476; no voters: d73d92fb35234d16824c55689b5fce28
I20260709 12:27:29.223181 30632 raft_consensus.cc:2804] T c1c8411289a74b5685700229d245ff75 P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:27:29.223555 30632 raft_consensus.cc:493] T c1c8411289a74b5685700229d245ff75 P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Starting leader election (detected failure of leader d73d92fb35234d16824c55689b5fce28)
I20260709 12:27:29.223941 30632 raft_consensus.cc:3060] T c1c8411289a74b5685700229d245ff75 P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:27:29.233263 30632 raft_consensus.cc:515] T c1c8411289a74b5685700229d245ff75 P c6610c00ffb948ce9c34f41dec3d49e8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
W20260709 12:27:29.233630 29634 leader_election.cc:336] T d21ab1641501475b829863275dc8182a P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Network error: Client connection negotiation failed: client connection to 127.16.29.66:36825: connect: Connection refused (error 111)
W20260709 12:27:29.234321 29634 leader_election.cc:336] T abec627108a84083884e64e1eb9dcda6 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Network error: Client connection negotiation failed: client connection to 127.16.29.66:36825: connect: Connection refused (error 111)
W20260709 12:27:29.234761 29634 leader_election.cc:336] T 137e03144e754f6e9da4f0544513d27c P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Network error: Client connection negotiation failed: client connection to 127.16.29.66:36825: connect: Connection refused (error 111)
I20260709 12:27:29.235185 30632 leader_election.cc:290] T c1c8411289a74b5685700229d245ff75 P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 3 election: Requested vote from peers d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825), d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209)
W20260709 12:27:29.235522 29634 consensus_peers.cc:597] T 6e37d72807e547898f4fce9e8b82b2b0 P d13c58ad8ac441c4a9b396eb04b1a476 -> Peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Couldn't send request to peer 584dac86183a41d9b3bea1c2085e62fa. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:27:29.236049 29634 consensus_peers.cc:597] T c6e621aa77d5474580dfc5949801ccc9 P d13c58ad8ac441c4a9b396eb04b1a476 -> Peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Couldn't send request to peer 584dac86183a41d9b3bea1c2085e62fa. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:27:29.236287 29709 leader_election.cc:304] T cdf706ffabc34902ad3d1bf2cee7889e P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c6610c00ffb948ce9c34f41dec3d49e8; no voters: d13c58ad8ac441c4a9b396eb04b1a476, d73d92fb35234d16824c55689b5fce28
I20260709 12:27:29.237433 30632 raft_consensus.cc:2749] T cdf706ffabc34902ad3d1bf2cee7889e P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:27:29.238932 29709 leader_election.cc:304] T 137e03144e754f6e9da4f0544513d27c P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c6610c00ffb948ce9c34f41dec3d49e8, d13c58ad8ac441c4a9b396eb04b1a476; no voters: d73d92fb35234d16824c55689b5fce28
W20260709 12:27:29.236433 29634 leader_election.cc:336] T d5fcd1f030614bbe82cf1dc261dfc1c3 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111)
W20260709 12:27:29.240159 29634 consensus_peers.cc:597] T 7ae3b542012d4f35a9c363eee22976ec P d13c58ad8ac441c4a9b396eb04b1a476 -> Peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Couldn't send request to peer 584dac86183a41d9b3bea1c2085e62fa. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:27:29.240270 30704 raft_consensus.cc:2804] T 137e03144e754f6e9da4f0544513d27c P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260709 12:27:29.240692 29634 leader_election.cc:336] T d21ab1641501475b829863275dc8182a P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111)
I20260709 12:27:29.240749 30704 raft_consensus.cc:493] T 137e03144e754f6e9da4f0544513d27c P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Starting leader election (detected failure of leader d73d92fb35234d16824c55689b5fce28)
I20260709 12:27:29.241163 29634 leader_election.cc:304] T d21ab1641501475b829863275dc8182a P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d13c58ad8ac441c4a9b396eb04b1a476; no voters: 584dac86183a41d9b3bea1c2085e62fa, d73d92fb35234d16824c55689b5fce28
I20260709 12:27:29.241258 30704 raft_consensus.cc:3060] T 137e03144e754f6e9da4f0544513d27c P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:27:29.242977 30677 raft_consensus.cc:493] T cdf706ffabc34902ad3d1bf2cee7889e P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d73d92fb35234d16824c55689b5fce28)
I20260709 12:27:29.243582 30677 raft_consensus.cc:515] T cdf706ffabc34902ad3d1bf2cee7889e P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:27:29.246351 30677 leader_election.cc:290] T cdf706ffabc34902ad3d1bf2cee7889e P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825), c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895)
I20260709 12:27:29.247013 30677 raft_consensus.cc:2749] T d21ab1641501475b829863275dc8182a P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260709 12:27:29.248018 29634 consensus_peers.cc:597] T 3f41ce441fb047c8bbb741b1b9cbc0ec P d13c58ad8ac441c4a9b396eb04b1a476 -> Peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Couldn't send request to peer 584dac86183a41d9b3bea1c2085e62fa. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:29.248699 29634 consensus_peers.cc:597] T 31a190e9f5ff407fab90502fb3197b70 P d13c58ad8ac441c4a9b396eb04b1a476 -> Peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Couldn't send request to peer 584dac86183a41d9b3bea1c2085e62fa. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:27:29.249125 29634 leader_election.cc:336] T abec627108a84083884e64e1eb9dcda6 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111)
I20260709 12:27:29.249591 29634 leader_election.cc:304] T abec627108a84083884e64e1eb9dcda6 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d13c58ad8ac441c4a9b396eb04b1a476; no voters: 584dac86183a41d9b3bea1c2085e62fa, d73d92fb35234d16824c55689b5fce28
I20260709 12:27:29.250502 30702 leader_election.cc:290] T 55061e62184b45f0bb1726adc1a4af81 P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 3 election: Requested vote from peers d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825), d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209)
I20260709 12:27:29.251255 30677 raft_consensus.cc:2749] T abec627108a84083884e64e1eb9dcda6 P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:27:29.254041 29662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55061e62184b45f0bb1726adc1a4af81" candidate_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "d13c58ad8ac441c4a9b396eb04b1a476"
I20260709 12:27:29.254689 29662 raft_consensus.cc:3060] T 55061e62184b45f0bb1726adc1a4af81 P d13c58ad8ac441c4a9b396eb04b1a476 [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:27:29.257061 29708 leader_election.cc:336] T 62f8c9472b214576b9002e02536a3ace P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Network error: Client connection negotiation failed: client connection to 127.16.29.66:36825: connect: Connection refused (error 111)
I20260709 12:27:29.259053 30704 raft_consensus.cc:515] T 137e03144e754f6e9da4f0544513d27c P c6610c00ffb948ce9c34f41dec3d49e8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:27:29.261900 29666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c1c8411289a74b5685700229d245ff75" candidate_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d13c58ad8ac441c4a9b396eb04b1a476"
I20260709 12:27:29.264089 29666 raft_consensus.cc:3060] T c1c8411289a74b5685700229d245ff75 P d13c58ad8ac441c4a9b396eb04b1a476 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:27:29.269237 29662 raft_consensus.cc:2468] T 55061e62184b45f0bb1726adc1a4af81 P d13c58ad8ac441c4a9b396eb04b1a476 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6610c00ffb948ce9c34f41dec3d49e8 in term 3.
I20260709 12:27:29.269698 29743 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00c6cf271c91400aa834467def9af9b1" candidate_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" is_pre_election: true
I20260709 12:27:29.270370 29743 raft_consensus.cc:2468] T 00c6cf271c91400aa834467def9af9b1 P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d13c58ad8ac441c4a9b396eb04b1a476 in term 1.
I20260709 12:27:29.272886 29666 raft_consensus.cc:2468] T c1c8411289a74b5685700229d245ff75 P d13c58ad8ac441c4a9b396eb04b1a476 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6610c00ffb948ce9c34f41dec3d49e8 in term 3.
I20260709 12:27:29.273126 29743 raft_consensus.cc:1275] T 3f41ce441fb047c8bbb741b1b9cbc0ec P c6610c00ffb948ce9c34f41dec3d49e8 [term 4 FOLLOWER]: Refusing update from remote peer d13c58ad8ac441c4a9b396eb04b1a476: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
W20260709 12:27:29.274379 29634 leader_election.cc:336] T 3615115233c34461b85763151751e475 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Network error: Client connection negotiation failed: client connection to 127.16.29.66:36825: connect: Connection refused (error 111)
W20260709 12:27:29.275207 29634 leader_election.cc:336] T d5fcd1f030614bbe82cf1dc261dfc1c3 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Network error: Client connection negotiation failed: client connection to 127.16.29.66:36825: connect: Connection refused (error 111)
I20260709 12:27:29.275609 29634 leader_election.cc:304] T d5fcd1f030614bbe82cf1dc261dfc1c3 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d13c58ad8ac441c4a9b396eb04b1a476; no voters: 584dac86183a41d9b3bea1c2085e62fa, d73d92fb35234d16824c55689b5fce28
I20260709 12:27:29.276391 30704 leader_election.cc:290] T 137e03144e754f6e9da4f0544513d27c P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 3 election: Requested vote from peers d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825), d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209)
I20260709 12:27:29.276883 30677 raft_consensus.cc:2749] T d5fcd1f030614bbe82cf1dc261dfc1c3 P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:27:29.279016 29666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "137e03144e754f6e9da4f0544513d27c" candidate_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "d13c58ad8ac441c4a9b396eb04b1a476"
I20260709 12:27:29.279734 29666 raft_consensus.cc:3060] T 137e03144e754f6e9da4f0544513d27c P d13c58ad8ac441c4a9b396eb04b1a476 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:27:29.280629 29742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "137e03144e754f6e9da4f0544513d27c" candidate_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" is_pre_election: true
I20260709 12:27:29.281364 29742 raft_consensus.cc:2393] T 137e03144e754f6e9da4f0544513d27c P c6610c00ffb948ce9c34f41dec3d49e8 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d13c58ad8ac441c4a9b396eb04b1a476 in current term 3: Already voted for candidate c6610c00ffb948ce9c34f41dec3d49e8 in this term.
I20260709 12:27:29.279137 29709 leader_election.cc:304] T 55061e62184b45f0bb1726adc1a4af81 P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c6610c00ffb948ce9c34f41dec3d49e8, d13c58ad8ac441c4a9b396eb04b1a476; no voters: 
I20260709 12:27:29.283298 30632 raft_consensus.cc:2804] T 55061e62184b45f0bb1726adc1a4af81 P c6610c00ffb948ce9c34f41dec3d49e8 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:27:29.283785 30632 raft_consensus.cc:697] T 55061e62184b45f0bb1726adc1a4af81 P c6610c00ffb948ce9c34f41dec3d49e8 [term 3 LEADER]: Becoming Leader. State: Replica: c6610c00ffb948ce9c34f41dec3d49e8, State: Running, Role: LEADER
W20260709 12:27:29.284665 29634 leader_election.cc:336] T 3615115233c34461b85763151751e475 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111)
I20260709 12:27:29.284588 30632 consensus_queue.cc:237] T 55061e62184b45f0bb1726adc1a4af81 P c6610c00ffb948ce9c34f41dec3d49e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:27:29.285173 29634 leader_election.cc:304] T 3615115233c34461b85763151751e475 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d13c58ad8ac441c4a9b396eb04b1a476; no voters: 584dac86183a41d9b3bea1c2085e62fa, d73d92fb35234d16824c55689b5fce28
I20260709 12:27:29.285964 30677 raft_consensus.cc:2749] T 3615115233c34461b85763151751e475 P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:27:29.288311 29709 leader_election.cc:304] T c1c8411289a74b5685700229d245ff75 P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c6610c00ffb948ce9c34f41dec3d49e8, d13c58ad8ac441c4a9b396eb04b1a476; no voters: 
I20260709 12:27:29.288976 29743 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cdf706ffabc34902ad3d1bf2cee7889e" candidate_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" is_pre_election: true
I20260709 12:27:29.289284 30702 raft_consensus.cc:2804] T c1c8411289a74b5685700229d245ff75 P c6610c00ffb948ce9c34f41dec3d49e8 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:27:29.289731 29743 raft_consensus.cc:2468] T cdf706ffabc34902ad3d1bf2cee7889e P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d13c58ad8ac441c4a9b396eb04b1a476 in term 1.
I20260709 12:27:29.289788 30702 raft_consensus.cc:697] T c1c8411289a74b5685700229d245ff75 P c6610c00ffb948ce9c34f41dec3d49e8 [term 3 LEADER]: Becoming Leader. State: Replica: c6610c00ffb948ce9c34f41dec3d49e8, State: Running, Role: LEADER
W20260709 12:27:29.290663 29708 leader_election.cc:336] T 62f8c9472b214576b9002e02536a3ace P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111)
I20260709 12:27:29.291164 29708 leader_election.cc:304] T 62f8c9472b214576b9002e02536a3ace P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c6610c00ffb948ce9c34f41dec3d49e8; no voters: 584dac86183a41d9b3bea1c2085e62fa, d73d92fb35234d16824c55689b5fce28
I20260709 12:27:29.292012 30681 raft_consensus.cc:2749] T 62f8c9472b214576b9002e02536a3ace P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:27:29.290973 30702 consensus_queue.cc:237] T c1c8411289a74b5685700229d245ff75 P c6610c00ffb948ce9c34f41dec3d49e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:27:29.294277 29666 raft_consensus.cc:2468] T 137e03144e754f6e9da4f0544513d27c P d13c58ad8ac441c4a9b396eb04b1a476 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6610c00ffb948ce9c34f41dec3d49e8 in term 3.
I20260709 12:27:29.296383 29635 leader_election.cc:304] T 00c6cf271c91400aa834467def9af9b1 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c6610c00ffb948ce9c34f41dec3d49e8, d13c58ad8ac441c4a9b396eb04b1a476; no voters: d73d92fb35234d16824c55689b5fce28
W20260709 12:27:29.296576 29634 leader_election.cc:336] T cdf706ffabc34902ad3d1bf2cee7889e P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Network error: Client connection negotiation failed: client connection to 127.16.29.66:36825: connect: Connection refused (error 111)
I20260709 12:27:29.297572 30677 raft_consensus.cc:2804] T 00c6cf271c91400aa834467def9af9b1 P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:27:29.297834 29709 leader_election.cc:304] T 137e03144e754f6e9da4f0544513d27c P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c6610c00ffb948ce9c34f41dec3d49e8, d13c58ad8ac441c4a9b396eb04b1a476; no voters: 
I20260709 12:27:29.297950 30677 raft_consensus.cc:493] T 00c6cf271c91400aa834467def9af9b1 P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Starting leader election (detected failure of leader d73d92fb35234d16824c55689b5fce28)
I20260709 12:27:29.298393 30677 raft_consensus.cc:3060] T 00c6cf271c91400aa834467def9af9b1 P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:29.300302 30707 consensus_queue.cc:1048] T 3f41ce441fb047c8bbb741b1b9cbc0ec P d13c58ad8ac441c4a9b396eb04b1a476 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:27:29.307040 29635 leader_election.cc:304] T 137e03144e754f6e9da4f0544513d27c P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d13c58ad8ac441c4a9b396eb04b1a476; no voters: c6610c00ffb948ce9c34f41dec3d49e8, d73d92fb35234d16824c55689b5fce28
I20260709 12:27:29.308372 30707 raft_consensus.cc:2749] T 137e03144e754f6e9da4f0544513d27c P d13c58ad8ac441c4a9b396eb04b1a476 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:27:29.308902 30681 raft_consensus.cc:2804] T 137e03144e754f6e9da4f0544513d27c P c6610c00ffb948ce9c34f41dec3d49e8 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:27:29.311549 30681 raft_consensus.cc:697] T 137e03144e754f6e9da4f0544513d27c P c6610c00ffb948ce9c34f41dec3d49e8 [term 3 LEADER]: Becoming Leader. State: Replica: c6610c00ffb948ce9c34f41dec3d49e8, State: Running, Role: LEADER
I20260709 12:27:29.312079 30677 raft_consensus.cc:515] T 00c6cf271c91400aa834467def9af9b1 P d13c58ad8ac441c4a9b396eb04b1a476 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:27:29.314442 29635 leader_election.cc:304] T cdf706ffabc34902ad3d1bf2cee7889e P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c6610c00ffb948ce9c34f41dec3d49e8, d13c58ad8ac441c4a9b396eb04b1a476; no voters: d73d92fb35234d16824c55689b5fce28
I20260709 12:27:29.312808 30681 consensus_queue.cc:237] T 137e03144e754f6e9da4f0544513d27c P c6610c00ffb948ce9c34f41dec3d49e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:27:29.315634 30707 raft_consensus.cc:2804] T cdf706ffabc34902ad3d1bf2cee7889e P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:27:29.316151 30707 raft_consensus.cc:493] T cdf706ffabc34902ad3d1bf2cee7889e P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Starting leader election (detected failure of leader d73d92fb35234d16824c55689b5fce28)
I20260709 12:27:29.316574 30707 raft_consensus.cc:3060] T cdf706ffabc34902ad3d1bf2cee7889e P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Advancing to term 2
W20260709 12:27:29.320083 29634 leader_election.cc:336] T 00c6cf271c91400aa834467def9af9b1 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Network error: Client connection negotiation failed: client connection to 127.16.29.66:36825: connect: Connection refused (error 111)
I20260709 12:27:29.321266 30677 leader_election.cc:290] T 00c6cf271c91400aa834467def9af9b1 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 2 election: Requested vote from peers d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825), c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895)
W20260709 12:27:29.324697 29708 leader_election.cc:336] T 55061e62184b45f0bb1726adc1a4af81 P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Network error: Client connection negotiation failed: client connection to 127.16.29.66:36825: connect: Connection refused (error 111)
W20260709 12:27:29.325524 29708 leader_election.cc:336] T c1c8411289a74b5685700229d245ff75 P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Network error: Client connection negotiation failed: client connection to 127.16.29.66:36825: connect: Connection refused (error 111)
I20260709 12:27:29.329181 29742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00c6cf271c91400aa834467def9af9b1" candidate_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c6610c00ffb948ce9c34f41dec3d49e8"
I20260709 12:27:29.329722 29742 raft_consensus.cc:3060] T 00c6cf271c91400aa834467def9af9b1 P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:29.336566 29742 raft_consensus.cc:2468] T 00c6cf271c91400aa834467def9af9b1 P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d13c58ad8ac441c4a9b396eb04b1a476 in term 2.
I20260709 12:27:29.336614 30707 raft_consensus.cc:515] T cdf706ffabc34902ad3d1bf2cee7889e P d13c58ad8ac441c4a9b396eb04b1a476 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
W20260709 12:27:29.339555 29708 leader_election.cc:336] T 137e03144e754f6e9da4f0544513d27c P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Network error: Client connection negotiation failed: client connection to 127.16.29.66:36825: connect: Connection refused (error 111)
I20260709 12:27:29.341468 29635 leader_election.cc:304] T 00c6cf271c91400aa834467def9af9b1 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c6610c00ffb948ce9c34f41dec3d49e8, d13c58ad8ac441c4a9b396eb04b1a476; no voters: d73d92fb35234d16824c55689b5fce28
I20260709 12:27:29.342517 30677 raft_consensus.cc:2804] T 00c6cf271c91400aa834467def9af9b1 P d13c58ad8ac441c4a9b396eb04b1a476 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:27:29.343015 30677 raft_consensus.cc:697] T 00c6cf271c91400aa834467def9af9b1 P d13c58ad8ac441c4a9b396eb04b1a476 [term 2 LEADER]: Becoming Leader. State: Replica: d13c58ad8ac441c4a9b396eb04b1a476, State: Running, Role: LEADER
W20260709 12:27:29.343605 29708 consensus_peers.cc:597] T 8627b58b4f42491597125eb45f8d3b61 P c6610c00ffb948ce9c34f41dec3d49e8 -> Peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Couldn't send request to peer 584dac86183a41d9b3bea1c2085e62fa. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:27:29.344169 29708 consensus_peers.cc:597] T 8017a44219404faa8db8ff049c65205e P c6610c00ffb948ce9c34f41dec3d49e8 -> Peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Couldn't send request to peer 584dac86183a41d9b3bea1c2085e62fa. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:27:29.343911 30677 consensus_queue.cc:237] T 00c6cf271c91400aa834467def9af9b1 P d13c58ad8ac441c4a9b396eb04b1a476 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
W20260709 12:27:29.344554 29708 consensus_peers.cc:597] T 742beffede4f407d8a72909d2b147e6d P c6610c00ffb948ce9c34f41dec3d49e8 -> Peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Couldn't send request to peer 584dac86183a41d9b3bea1c2085e62fa. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:27:29.357443 29634 leader_election.cc:336] T cdf706ffabc34902ad3d1bf2cee7889e P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Network error: Client connection negotiation failed: client connection to 127.16.29.66:36825: connect: Connection refused (error 111)
I20260709 12:27:29.357288 29419 catalog_manager.cc:5697] T 55061e62184b45f0bb1726adc1a4af81 P c6610c00ffb948ce9c34f41dec3d49e8 reported cstate change: term changed from 2 to 3, leader changed from d73d92fb35234d16824c55689b5fce28 (127.16.29.66) to c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68). New cstate: current_term: 3 leader_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:29.359550 29419 catalog_manager.cc:5697] T c1c8411289a74b5685700229d245ff75 P c6610c00ffb948ce9c34f41dec3d49e8 reported cstate change: term changed from 2 to 3, leader changed from d73d92fb35234d16824c55689b5fce28 (127.16.29.66) to c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68). New cstate: current_term: 3 leader_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } health_report { overall_health: HEALTHY } } }
I20260709 12:27:29.364568 30707 leader_election.cc:290] T cdf706ffabc34902ad3d1bf2cee7889e P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 2 election: Requested vote from peers d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825), c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68:43895)
I20260709 12:27:29.361667 29419 catalog_manager.cc:5697] T 137e03144e754f6e9da4f0544513d27c P c6610c00ffb948ce9c34f41dec3d49e8 reported cstate change: term changed from 2 to 3, leader changed from d73d92fb35234d16824c55689b5fce28 (127.16.29.66) to c6610c00ffb948ce9c34f41dec3d49e8 (127.16.29.68). New cstate: current_term: 3 leader_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } health_report { overall_health: HEALTHY } } }
I20260709 12:27:29.373343 29743 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cdf706ffabc34902ad3d1bf2cee7889e" candidate_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c6610c00ffb948ce9c34f41dec3d49e8"
I20260709 12:27:29.373967 29743 raft_consensus.cc:3060] T cdf706ffabc34902ad3d1bf2cee7889e P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:29.373231 29418 catalog_manager.cc:5697] T 00c6cf271c91400aa834467def9af9b1 P d13c58ad8ac441c4a9b396eb04b1a476 reported cstate change: term changed from 1 to 2, leader changed from d73d92fb35234d16824c55689b5fce28 (127.16.29.66) to d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67). New cstate: current_term: 2 leader_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:29.381381 29743 raft_consensus.cc:2468] T cdf706ffabc34902ad3d1bf2cee7889e P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d13c58ad8ac441c4a9b396eb04b1a476 in term 2.
W20260709 12:27:29.382094 29708 consensus_peers.cc:597] T 66f652ba2c224885a2aea974bfe8f88a P c6610c00ffb948ce9c34f41dec3d49e8 -> Peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Couldn't send request to peer 584dac86183a41d9b3bea1c2085e62fa. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:27:29.387698 29635 leader_election.cc:304] T cdf706ffabc34902ad3d1bf2cee7889e P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c6610c00ffb948ce9c34f41dec3d49e8, d13c58ad8ac441c4a9b396eb04b1a476; no voters: d73d92fb35234d16824c55689b5fce28
I20260709 12:27:29.388569 30677 raft_consensus.cc:2804] T cdf706ffabc34902ad3d1bf2cee7889e P d13c58ad8ac441c4a9b396eb04b1a476 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:27:29.389072 30677 raft_consensus.cc:697] T cdf706ffabc34902ad3d1bf2cee7889e P d13c58ad8ac441c4a9b396eb04b1a476 [term 2 LEADER]: Becoming Leader. State: Replica: d13c58ad8ac441c4a9b396eb04b1a476, State: Running, Role: LEADER
I20260709 12:27:29.390018 30677 consensus_queue.cc:237] T cdf706ffabc34902ad3d1bf2cee7889e P d13c58ad8ac441c4a9b396eb04b1a476 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
W20260709 12:27:29.392679 29708 consensus_peers.cc:597] T ef20d7124e7a44aa97a219ee194f4b8b P c6610c00ffb948ce9c34f41dec3d49e8 -> Peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Couldn't send request to peer 584dac86183a41d9b3bea1c2085e62fa. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:27:29.424530 29708 consensus_peers.cc:597] T a46289e8fc824dd5a818e30df6e7ec3e P c6610c00ffb948ce9c34f41dec3d49e8 -> Peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Couldn't send request to peer 584dac86183a41d9b3bea1c2085e62fa. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:27:29.436133 29419 catalog_manager.cc:5697] T cdf706ffabc34902ad3d1bf2cee7889e P d13c58ad8ac441c4a9b396eb04b1a476 reported cstate change: term changed from 1 to 2, leader changed from d73d92fb35234d16824c55689b5fce28 (127.16.29.66) to d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67). New cstate: current_term: 2 leader_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } health_report { overall_health: UNKNOWN } } }
W20260709 12:27:29.485918 29708 consensus_peers.cc:597] T 425777eed34441cd93e0902bdc126516 P c6610c00ffb948ce9c34f41dec3d49e8 -> Peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Couldn't send request to peer 584dac86183a41d9b3bea1c2085e62fa. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:27:29.511744 30677 raft_consensus.cc:493] T d1918297b9eb41f5847a211e668ae748 P d13c58ad8ac441c4a9b396eb04b1a476 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 584dac86183a41d9b3bea1c2085e62fa)
I20260709 12:27:29.512307 30677 raft_consensus.cc:515] T d1918297b9eb41f5847a211e668ae748 P d13c58ad8ac441c4a9b396eb04b1a476 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:27:29.516692 30677 leader_election.cc:290] T d1918297b9eb41f5847a211e668ae748 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033), d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825)
W20260709 12:27:29.519316 29634 leader_election.cc:336] T d1918297b9eb41f5847a211e668ae748 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111)
W20260709 12:27:29.521164 29634 leader_election.cc:336] T d1918297b9eb41f5847a211e668ae748 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Network error: Client connection negotiation failed: client connection to 127.16.29.66:36825: connect: Connection refused (error 111)
I20260709 12:27:29.521512 29634 leader_election.cc:304] T d1918297b9eb41f5847a211e668ae748 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d13c58ad8ac441c4a9b396eb04b1a476; no voters: 584dac86183a41d9b3bea1c2085e62fa, d73d92fb35234d16824c55689b5fce28
I20260709 12:27:29.522091 30677 raft_consensus.cc:2749] T d1918297b9eb41f5847a211e668ae748 P d13c58ad8ac441c4a9b396eb04b1a476 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260709 12:27:29.550733 29634 consensus_peers.cc:597] T fb7d78e534634e0f992fc39b578ebd8d P d13c58ad8ac441c4a9b396eb04b1a476 -> Peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Couldn't send request to peer 584dac86183a41d9b3bea1c2085e62fa. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:27:29.551298 29634 consensus_peers.cc:597] T f3e17f0a8fc14095850e8138fd79b86e P d13c58ad8ac441c4a9b396eb04b1a476 -> Peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Couldn't send request to peer 584dac86183a41d9b3bea1c2085e62fa. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:27:29.552296 30661 raft_consensus.cc:493] T 4685c2e824c14fc1bcd45c16169798cc P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 584dac86183a41d9b3bea1c2085e62fa)
I20260709 12:27:29.552850 30661 raft_consensus.cc:515] T 4685c2e824c14fc1bcd45c16169798cc P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:29.555109 30661 leader_election.cc:290] T 4685c2e824c14fc1bcd45c16169798cc P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033), d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825)
W20260709 12:27:29.563061 29708 leader_election.cc:336] T 4685c2e824c14fc1bcd45c16169798cc P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111)
W20260709 12:27:29.563755 29708 leader_election.cc:336] T 4685c2e824c14fc1bcd45c16169798cc P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Network error: Client connection negotiation failed: client connection to 127.16.29.66:36825: connect: Connection refused (error 111)
I20260709 12:27:29.564177 29708 leader_election.cc:304] T 4685c2e824c14fc1bcd45c16169798cc P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c6610c00ffb948ce9c34f41dec3d49e8; no voters: 584dac86183a41d9b3bea1c2085e62fa, d73d92fb35234d16824c55689b5fce28
I20260709 12:27:29.564885 30661 raft_consensus.cc:2749] T 4685c2e824c14fc1bcd45c16169798cc P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260709 12:27:29.579448 29708 consensus_peers.cc:597] T 1d4e0ad0462e4f74a44155aa3ffaf87f P c6610c00ffb948ce9c34f41dec3d49e8 -> Peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Couldn't send request to peer 584dac86183a41d9b3bea1c2085e62fa. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:27:29.580021 29708 consensus_peers.cc:597] T f4516786e5564e9eac08ca472f79a9f0 P c6610c00ffb948ce9c34f41dec3d49e8 -> Peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Couldn't send request to peer 584dac86183a41d9b3bea1c2085e62fa. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:27:29.608110 29634 consensus_peers.cc:597] T e67a2affef634fd8bc96a64fa83cffdb P d13c58ad8ac441c4a9b396eb04b1a476 -> Peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Couldn't send request to peer 584dac86183a41d9b3bea1c2085e62fa. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:27:29.640357 29634 consensus_peers.cc:597] T cbf03426b6df43e5b2b9c56e2eb7687c P d13c58ad8ac441c4a9b396eb04b1a476 -> Peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Couldn't send request to peer 584dac86183a41d9b3bea1c2085e62fa. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:27:29.702703 29666 raft_consensus.cc:1275] T 55061e62184b45f0bb1726adc1a4af81 P d13c58ad8ac441c4a9b396eb04b1a476 [term 3 FOLLOWER]: Refusing update from remote peer c6610c00ffb948ce9c34f41dec3d49e8: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 12:27:29.703913 30681 consensus_queue.cc:1048] T 55061e62184b45f0bb1726adc1a4af81 P c6610c00ffb948ce9c34f41dec3d49e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260709 12:27:29.716917 29634 consensus_peers.cc:597] T 5ee924f859f54f64908499bfac48ee62 P d13c58ad8ac441c4a9b396eb04b1a476 -> Peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Couldn't send request to peer 584dac86183a41d9b3bea1c2085e62fa. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:27:29.720981 29708 consensus_peers.cc:597] T d299fb38094e47e8aa3890ca74f6b366 P c6610c00ffb948ce9c34f41dec3d49e8 -> Peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Couldn't send request to peer d73d92fb35234d16824c55689b5fce28. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.66:36825: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:27:29.739450 29743 raft_consensus.cc:1275] T 00c6cf271c91400aa834467def9af9b1 P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Refusing update from remote peer d13c58ad8ac441c4a9b396eb04b1a476: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:27:29.741895 30677 consensus_queue.cc:1048] T 00c6cf271c91400aa834467def9af9b1 P d13c58ad8ac441c4a9b396eb04b1a476 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:27:29.747572 29662 raft_consensus.cc:1275] T 137e03144e754f6e9da4f0544513d27c P d13c58ad8ac441c4a9b396eb04b1a476 [term 3 FOLLOWER]: Refusing update from remote peer c6610c00ffb948ce9c34f41dec3d49e8: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 12:27:29.749876 30661 consensus_queue.cc:1048] T 137e03144e754f6e9da4f0544513d27c P c6610c00ffb948ce9c34f41dec3d49e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260709 12:27:29.780615 29708 consensus_peers.cc:597] T 55061e62184b45f0bb1726adc1a4af81 P c6610c00ffb948ce9c34f41dec3d49e8 -> Peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Couldn't send request to peer d73d92fb35234d16824c55689b5fce28. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.66:36825: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:29.803458 29634 consensus_peers.cc:597] T 00c6cf271c91400aa834467def9af9b1 P d13c58ad8ac441c4a9b396eb04b1a476 -> Peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Couldn't send request to peer d73d92fb35234d16824c55689b5fce28. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.66:36825: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:29.804277 29634 consensus_peers.cc:597] T 3e9451c3797c4496a66dfdb3e9fb5e38 P d13c58ad8ac441c4a9b396eb04b1a476 -> Peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Couldn't send request to peer 584dac86183a41d9b3bea1c2085e62fa. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:27:29.804824 29634 consensus_peers.cc:597] T 0c428ff656ae4b9d99d5b5073f98725d P d13c58ad8ac441c4a9b396eb04b1a476 -> Peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Couldn't send request to peer 584dac86183a41d9b3bea1c2085e62fa. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:27:29.816187 29634 consensus_peers.cc:597] T d42259e436d241b29fb4ded16d9b6455 P d13c58ad8ac441c4a9b396eb04b1a476 -> Peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Couldn't send request to peer 584dac86183a41d9b3bea1c2085e62fa. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:27:29.825201 30706 raft_consensus.cc:493] T fcb6a7f720b14dc98e821f663357a0bb P d13c58ad8ac441c4a9b396eb04b1a476 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 584dac86183a41d9b3bea1c2085e62fa)
I20260709 12:27:29.825714 30706 raft_consensus.cc:515] T fcb6a7f720b14dc98e821f663357a0bb P d13c58ad8ac441c4a9b396eb04b1a476 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:27:29.827633 30706 leader_election.cc:290] T fcb6a7f720b14dc98e821f663357a0bb P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033), d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825)
W20260709 12:27:29.857818 29708 consensus_peers.cc:597] T 137e03144e754f6e9da4f0544513d27c P c6610c00ffb948ce9c34f41dec3d49e8 -> Peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Couldn't send request to peer d73d92fb35234d16824c55689b5fce28. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.66:36825: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:29.861178 29634 leader_election.cc:336] T fcb6a7f720b14dc98e821f663357a0bb P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111)
I20260709 12:27:29.864181 29662 raft_consensus.cc:1275] T c1c8411289a74b5685700229d245ff75 P d13c58ad8ac441c4a9b396eb04b1a476 [term 3 FOLLOWER]: Refusing update from remote peer c6610c00ffb948ce9c34f41dec3d49e8: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
W20260709 12:27:29.867965 29634 leader_election.cc:336] T fcb6a7f720b14dc98e821f663357a0bb P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Network error: Client connection negotiation failed: client connection to 127.16.29.66:36825: connect: Connection refused (error 111)
I20260709 12:27:29.869568 29634 leader_election.cc:304] T fcb6a7f720b14dc98e821f663357a0bb P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d13c58ad8ac441c4a9b396eb04b1a476; no voters: 584dac86183a41d9b3bea1c2085e62fa, d73d92fb35234d16824c55689b5fce28
I20260709 12:27:29.870466 30702 consensus_queue.cc:1048] T c1c8411289a74b5685700229d245ff75 P c6610c00ffb948ce9c34f41dec3d49e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:27:29.872452 30709 raft_consensus.cc:2749] T fcb6a7f720b14dc98e821f663357a0bb P d13c58ad8ac441c4a9b396eb04b1a476 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:27:29.875762 30702 raft_consensus.cc:493] T 01788883c8ea42989f246a3486111a7e P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 584dac86183a41d9b3bea1c2085e62fa)
I20260709 12:27:29.876397 30702 raft_consensus.cc:515] T 01788883c8ea42989f246a3486111a7e P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:29.878419 30702 leader_election.cc:290] T 01788883c8ea42989f246a3486111a7e P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033), d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825)
W20260709 12:27:29.885725 16501 thread.cc:527] Waited for 2000ms trying to join with diag-logger (tid 29618)
I20260709 12:27:29.896385 29743 raft_consensus.cc:1275] T cdf706ffabc34902ad3d1bf2cee7889e P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Refusing update from remote peer d13c58ad8ac441c4a9b396eb04b1a476: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:27:29.899405 30706 consensus_queue.cc:1048] T cdf706ffabc34902ad3d1bf2cee7889e P d13c58ad8ac441c4a9b396eb04b1a476 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:27:29.902567 30702 raft_consensus.cc:493] T b3c02c910d2b43cc9265bb1d98491cd3 P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 584dac86183a41d9b3bea1c2085e62fa)
I20260709 12:27:29.903132 30702 raft_consensus.cc:515] T b3c02c910d2b43cc9265bb1d98491cd3 P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:29.905084 30702 leader_election.cc:290] T b3c02c910d2b43cc9265bb1d98491cd3 P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033), d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825)
W20260709 12:27:29.915115 29708 leader_election.cc:336] T 01788883c8ea42989f246a3486111a7e P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111)
W20260709 12:27:29.929539 29708 leader_election.cc:336] T 01788883c8ea42989f246a3486111a7e P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Network error: Client connection negotiation failed: client connection to 127.16.29.66:36825: connect: Connection refused (error 111)
I20260709 12:27:29.930099 29708 leader_election.cc:304] T 01788883c8ea42989f246a3486111a7e P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c6610c00ffb948ce9c34f41dec3d49e8; no voters: 584dac86183a41d9b3bea1c2085e62fa, d73d92fb35234d16824c55689b5fce28
I20260709 12:27:29.931306 30702 raft_consensus.cc:2749] T 01788883c8ea42989f246a3486111a7e P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260709 12:27:29.947829 29708 leader_election.cc:336] T b3c02c910d2b43cc9265bb1d98491cd3 P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Network error: Client connection negotiation failed: client connection to 127.16.29.66:36825: connect: Connection refused (error 111)
W20260709 12:27:29.952134 29708 leader_election.cc:336] T b3c02c910d2b43cc9265bb1d98491cd3 P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111)
I20260709 12:27:29.952698 29708 leader_election.cc:304] T b3c02c910d2b43cc9265bb1d98491cd3 P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c6610c00ffb948ce9c34f41dec3d49e8; no voters: 584dac86183a41d9b3bea1c2085e62fa, d73d92fb35234d16824c55689b5fce28
I20260709 12:27:29.953686 30702 raft_consensus.cc:2749] T b3c02c910d2b43cc9265bb1d98491cd3 P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260709 12:27:29.970412 29634 consensus_peers.cc:597] T cdf706ffabc34902ad3d1bf2cee7889e P d13c58ad8ac441c4a9b396eb04b1a476 -> Peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Couldn't send request to peer d73d92fb35234d16824c55689b5fce28. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.66:36825: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:29.979571 29708 consensus_peers.cc:597] T c1c8411289a74b5685700229d245ff75 P c6610c00ffb948ce9c34f41dec3d49e8 -> Peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Couldn't send request to peer d73d92fb35234d16824c55689b5fce28. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.66:36825: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:29.994122 30702 raft_consensus.cc:493] T 1d1c1f58c4d340ac951dfa403d34138a P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 584dac86183a41d9b3bea1c2085e62fa)
I20260709 12:27:29.994648 30702 raft_consensus.cc:515] T 1d1c1f58c4d340ac951dfa403d34138a P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:29.996480 30702 leader_election.cc:290] T 1d1c1f58c4d340ac951dfa403d34138a P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033), d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825)
W20260709 12:27:30.008317 29708 leader_election.cc:336] T 1d1c1f58c4d340ac951dfa403d34138a P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Network error: Client connection negotiation failed: client connection to 127.16.29.66:36825: connect: Connection refused (error 111)
W20260709 12:27:30.009021 29708 leader_election.cc:336] T 1d1c1f58c4d340ac951dfa403d34138a P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111)
I20260709 12:27:30.009428 29708 leader_election.cc:304] T 1d1c1f58c4d340ac951dfa403d34138a P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c6610c00ffb948ce9c34f41dec3d49e8; no voters: 584dac86183a41d9b3bea1c2085e62fa, d73d92fb35234d16824c55689b5fce28
I20260709 12:27:30.010179 30702 raft_consensus.cc:2749] T 1d1c1f58c4d340ac951dfa403d34138a P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260709 12:27:30.010854 29634 consensus_peers.cc:597] T cbf03426b6df43e5b2b9c56e2eb7687c P d13c58ad8ac441c4a9b396eb04b1a476 -> Peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Couldn't send request to peer d73d92fb35234d16824c55689b5fce28. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.66:36825: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:27:30.065277 30702 raft_consensus.cc:493] T 6c8b2fa9f2c044ec9eec97c72b561042 P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 584dac86183a41d9b3bea1c2085e62fa)
I20260709 12:27:30.065914 30702 raft_consensus.cc:515] T 6c8b2fa9f2c044ec9eec97c72b561042 P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:30.068341 30702 leader_election.cc:290] T 6c8b2fa9f2c044ec9eec97c72b561042 P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033), d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825)
W20260709 12:27:30.077625 29634 consensus_peers.cc:597] T 07a2b94c0823467695508292a9ccdcc5 P d13c58ad8ac441c4a9b396eb04b1a476 -> Peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Couldn't send request to peer d73d92fb35234d16824c55689b5fce28. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.66:36825: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:27:30.079926 29708 consensus_peers.cc:597] T 394378d078194257b9e04fc659c1f5d4 P c6610c00ffb948ce9c34f41dec3d49e8 -> Peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Couldn't send request to peer d73d92fb35234d16824c55689b5fce28. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.66:36825: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:27:30.080569 29708 leader_election.cc:336] T 6c8b2fa9f2c044ec9eec97c72b561042 P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Network error: Client connection negotiation failed: client connection to 127.16.29.66:36825: connect: Connection refused (error 111)
W20260709 12:27:30.081571 29708 leader_election.cc:336] T 6c8b2fa9f2c044ec9eec97c72b561042 P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111)
I20260709 12:27:30.082067 29708 leader_election.cc:304] T 6c8b2fa9f2c044ec9eec97c72b561042 P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c6610c00ffb948ce9c34f41dec3d49e8; no voters: 584dac86183a41d9b3bea1c2085e62fa, d73d92fb35234d16824c55689b5fce28
I20260709 12:27:30.082883 30702 raft_consensus.cc:2749] T 6c8b2fa9f2c044ec9eec97c72b561042 P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:27:30.086267 30702 raft_consensus.cc:493] T 6a30685caefb4d0e80fa4ea87d70760c P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 584dac86183a41d9b3bea1c2085e62fa)
I20260709 12:27:30.086824 30702 raft_consensus.cc:515] T 6a30685caefb4d0e80fa4ea87d70760c P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:30.089041 30702 leader_election.cc:290] T 6a30685caefb4d0e80fa4ea87d70760c P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033), d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825)
I20260709 12:27:30.094329 30702 raft_consensus.cc:493] T 592801934cad4789b9ce8dc09f1470bd P c6610c00ffb948ce9c34f41dec3d49e8 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 584dac86183a41d9b3bea1c2085e62fa)
I20260709 12:27:30.094899 30702 raft_consensus.cc:515] T 592801934cad4789b9ce8dc09f1470bd P c6610c00ffb948ce9c34f41dec3d49e8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:30.096985 30702 leader_election.cc:290] T 592801934cad4789b9ce8dc09f1470bd P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033), d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825)
W20260709 12:27:30.103403 29708 leader_election.cc:336] T 6a30685caefb4d0e80fa4ea87d70760c P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111)
W20260709 12:27:30.104708 29708 leader_election.cc:336] T 6a30685caefb4d0e80fa4ea87d70760c P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Network error: Client connection negotiation failed: client connection to 127.16.29.66:36825: connect: Connection refused (error 111)
I20260709 12:27:30.105350 29708 leader_election.cc:304] T 6a30685caefb4d0e80fa4ea87d70760c P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c6610c00ffb948ce9c34f41dec3d49e8; no voters: 584dac86183a41d9b3bea1c2085e62fa, d73d92fb35234d16824c55689b5fce28
I20260709 12:27:30.106108 30702 raft_consensus.cc:2749] T 6a30685caefb4d0e80fa4ea87d70760c P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260709 12:27:30.106983 29634 consensus_peers.cc:597] T d42259e436d241b29fb4ded16d9b6455 P d13c58ad8ac441c4a9b396eb04b1a476 -> Peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Couldn't send request to peer d73d92fb35234d16824c55689b5fce28. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.66:36825: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:27:30.112300 29708 leader_election.cc:336] T 592801934cad4789b9ce8dc09f1470bd P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111)
W20260709 12:27:30.113121 29708 leader_election.cc:336] T 592801934cad4789b9ce8dc09f1470bd P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Network error: Client connection negotiation failed: client connection to 127.16.29.66:36825: connect: Connection refused (error 111)
I20260709 12:27:30.113550 29708 leader_election.cc:304] T 592801934cad4789b9ce8dc09f1470bd P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c6610c00ffb948ce9c34f41dec3d49e8; no voters: 584dac86183a41d9b3bea1c2085e62fa, d73d92fb35234d16824c55689b5fce28
I20260709 12:27:30.114791 30632 raft_consensus.cc:2749] T 592801934cad4789b9ce8dc09f1470bd P c6610c00ffb948ce9c34f41dec3d49e8 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:27:30.122490 30681 raft_consensus.cc:493] T fb6e78ccc01c4707af34cddf81619eab P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 584dac86183a41d9b3bea1c2085e62fa)
I20260709 12:27:30.123003 30681 raft_consensus.cc:515] T fb6e78ccc01c4707af34cddf81619eab P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:30.124766 30681 leader_election.cc:290] T fb6e78ccc01c4707af34cddf81619eab P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033), d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825)
W20260709 12:27:30.128628 29708 leader_election.cc:336] T fb6e78ccc01c4707af34cddf81619eab P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111)
W20260709 12:27:30.129483 29708 leader_election.cc:336] T fb6e78ccc01c4707af34cddf81619eab P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Network error: Client connection negotiation failed: client connection to 127.16.29.66:36825: connect: Connection refused (error 111)
I20260709 12:27:30.129884 29708 leader_election.cc:304] T fb6e78ccc01c4707af34cddf81619eab P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c6610c00ffb948ce9c34f41dec3d49e8; no voters: 584dac86183a41d9b3bea1c2085e62fa, d73d92fb35234d16824c55689b5fce28
I20260709 12:27:30.131367 30702 raft_consensus.cc:2749] T fb6e78ccc01c4707af34cddf81619eab P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260709 12:27:30.136667 29634 consensus_peers.cc:597] T 3617af2dee3f48cf85973b1c94cdc33a P d13c58ad8ac441c4a9b396eb04b1a476 -> Peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Couldn't send request to peer d73d92fb35234d16824c55689b5fce28. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.66:36825: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:27:30.137193 29708 consensus_peers.cc:597] T 1d4e0ad0462e4f74a44155aa3ffaf87f P c6610c00ffb948ce9c34f41dec3d49e8 -> Peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Couldn't send request to peer d73d92fb35234d16824c55689b5fce28. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.66:36825: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:27:30.137611 29708 consensus_peers.cc:597] T 58154d49c1a344f2b51366ede3c87de5 P c6610c00ffb948ce9c34f41dec3d49e8 -> Peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Couldn't send request to peer d73d92fb35234d16824c55689b5fce28. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.66:36825: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:27:30.162325 29692 debug-util.cc:398] Leaking SignalData structure 0x7b08000d5160 after lost signal to thread 16504
W20260709 12:27:30.163519 29692 debug-util.cc:398] Leaking SignalData structure 0x7b0800340b80 after lost signal to thread 29454
W20260709 12:27:30.164984 29692 debug-util.cc:398] Leaking SignalData structure 0x7b080027de40 after lost signal to thread 29695
W20260709 12:27:30.167232 29692 debug-util.cc:398] Leaking SignalData structure 0x7b08002c41c0 after lost signal to thread 29769
W20260709 12:27:30.210633 29634 consensus_peers.cc:597] T 76140ad6258543118a9bfeee7cf0d5fc P d13c58ad8ac441c4a9b396eb04b1a476 -> Peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Couldn't send request to peer d73d92fb35234d16824c55689b5fce28. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.66:36825: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:27:30.257597 29708 consensus_peers.cc:597] T 8017a44219404faa8db8ff049c65205e P c6610c00ffb948ce9c34f41dec3d49e8 -> Peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Couldn't send request to peer d73d92fb35234d16824c55689b5fce28. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.66:36825: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:27:30.280066 29634 consensus_peers.cc:597] T f3e17f0a8fc14095850e8138fd79b86e P d13c58ad8ac441c4a9b396eb04b1a476 -> Peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Couldn't send request to peer d73d92fb35234d16824c55689b5fce28. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.66:36825: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:27:30.299448 29634 consensus_peers.cc:597] T c6e621aa77d5474580dfc5949801ccc9 P d13c58ad8ac441c4a9b396eb04b1a476 -> Peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Couldn't send request to peer d73d92fb35234d16824c55689b5fce28. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.66:36825: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:27:30.305549 29708 consensus_peers.cc:597] T 959940284eb24fdf83d78438d710318a P c6610c00ffb948ce9c34f41dec3d49e8 -> Peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Couldn't send request to peer d73d92fb35234d16824c55689b5fce28. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.66:36825: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:27:30.332574 29708 consensus_peers.cc:597] T 66f652ba2c224885a2aea974bfe8f88a P c6610c00ffb948ce9c34f41dec3d49e8 -> Peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Couldn't send request to peer d73d92fb35234d16824c55689b5fce28. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.66:36825: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:27:30.338449 30709 raft_consensus.cc:493] T 16649fd17d434e3c95e9d585d2404659 P d13c58ad8ac441c4a9b396eb04b1a476 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d73d92fb35234d16824c55689b5fce28)
I20260709 12:27:30.339046 30709 raft_consensus.cc:515] T 16649fd17d434e3c95e9d585d2404659 P d13c58ad8ac441c4a9b396eb04b1a476 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:27:30.340895 30709 leader_election.cc:290] T 16649fd17d434e3c95e9d585d2404659 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033), d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825)
W20260709 12:27:30.347010 29634 leader_election.cc:336] T 16649fd17d434e3c95e9d585d2404659 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111)
W20260709 12:27:30.347714 29634 leader_election.cc:336] T 16649fd17d434e3c95e9d585d2404659 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Network error: Client connection negotiation failed: client connection to 127.16.29.66:36825: connect: Connection refused (error 111)
I20260709 12:27:30.348167 29634 leader_election.cc:304] T 16649fd17d434e3c95e9d585d2404659 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d13c58ad8ac441c4a9b396eb04b1a476; no voters: 584dac86183a41d9b3bea1c2085e62fa, d73d92fb35234d16824c55689b5fce28
I20260709 12:27:30.348778 30709 raft_consensus.cc:2749] T 16649fd17d434e3c95e9d585d2404659 P d13c58ad8ac441c4a9b396eb04b1a476 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:27:30.351137 30709 raft_consensus.cc:493] T 71e2de235adf46d9a2407659080d42ef P d13c58ad8ac441c4a9b396eb04b1a476 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 584dac86183a41d9b3bea1c2085e62fa)
I20260709 12:27:30.351583 30709 raft_consensus.cc:515] T 71e2de235adf46d9a2407659080d42ef P d13c58ad8ac441c4a9b396eb04b1a476 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:27:30.355918 30709 leader_election.cc:290] T 71e2de235adf46d9a2407659080d42ef P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033), d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825)
W20260709 12:27:30.362072 29634 leader_election.cc:336] T 71e2de235adf46d9a2407659080d42ef P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111)
W20260709 12:27:30.362668 29634 leader_election.cc:336] T 71e2de235adf46d9a2407659080d42ef P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Network error: Client connection negotiation failed: client connection to 127.16.29.66:36825: connect: Connection refused (error 111)
I20260709 12:27:30.363030 29634 leader_election.cc:304] T 71e2de235adf46d9a2407659080d42ef P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d13c58ad8ac441c4a9b396eb04b1a476; no voters: 584dac86183a41d9b3bea1c2085e62fa, d73d92fb35234d16824c55689b5fce28
I20260709 12:27:30.363682 30709 raft_consensus.cc:2749] T 71e2de235adf46d9a2407659080d42ef P d13c58ad8ac441c4a9b396eb04b1a476 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260709 12:27:30.368685 29708 consensus_peers.cc:597] T ed057e06dec148a0b9d7062beeb9d005 P c6610c00ffb948ce9c34f41dec3d49e8 -> Peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Couldn't send request to peer d73d92fb35234d16824c55689b5fce28. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.66:36825: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:27:30.373376 29634 consensus_peers.cc:597] T 5ee924f859f54f64908499bfac48ee62 P d13c58ad8ac441c4a9b396eb04b1a476 -> Peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Couldn't send request to peer d73d92fb35234d16824c55689b5fce28. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.66:36825: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:27:30.631269 29708 consensus_peers.cc:597] T 43c7bfb33af74d4f912e5153e8a54b3e P c6610c00ffb948ce9c34f41dec3d49e8 -> Peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Couldn't send request to peer d73d92fb35234d16824c55689b5fce28. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.66:36825: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:27:30.713795 29634 consensus_peers.cc:597] T 7ae3b542012d4f35a9c363eee22976ec P d13c58ad8ac441c4a9b396eb04b1a476 -> Peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Couldn't send request to peer d73d92fb35234d16824c55689b5fce28. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.66:36825: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:27:30.887019 16501 thread.cc:527] Waited for 3000ms trying to join with diag-logger (tid 29618)
I20260709 12:27:30.897156 30677 raft_consensus.cc:493] T d21ab1641501475b829863275dc8182a P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d73d92fb35234d16824c55689b5fce28)
I20260709 12:27:30.898190 30677 raft_consensus.cc:515] T d21ab1641501475b829863275dc8182a P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:27:30.900868 30681 raft_consensus.cc:493] T a3dcbb9c64bf4b5bb6680e8cb81af93e P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d73d92fb35234d16824c55689b5fce28)
I20260709 12:27:30.901911 30681 raft_consensus.cc:515] T a3dcbb9c64bf4b5bb6680e8cb81af93e P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:30.905862 30681 leader_election.cc:290] T a3dcbb9c64bf4b5bb6680e8cb81af93e P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033), d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825)
I20260709 12:27:30.910290 30709 raft_consensus.cc:493] T abec627108a84083884e64e1eb9dcda6 P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d73d92fb35234d16824c55689b5fce28)
I20260709 12:27:30.911538 30709 raft_consensus.cc:515] T abec627108a84083884e64e1eb9dcda6 P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
W20260709 12:27:30.914899 29708 leader_election.cc:336] T a3dcbb9c64bf4b5bb6680e8cb81af93e P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Network error: Client connection negotiation failed: client connection to 127.16.29.66:36825: connect: Connection refused (error 111)
W20260709 12:27:30.915913 29708 leader_election.cc:336] T a3dcbb9c64bf4b5bb6680e8cb81af93e P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111)
I20260709 12:27:30.914342 30707 raft_consensus.cc:493] T d5fcd1f030614bbe82cf1dc261dfc1c3 P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d73d92fb35234d16824c55689b5fce28)
I20260709 12:27:30.916942 30707 raft_consensus.cc:515] T d5fcd1f030614bbe82cf1dc261dfc1c3 P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:27:30.916376 29708 leader_election.cc:304] T a3dcbb9c64bf4b5bb6680e8cb81af93e P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c6610c00ffb948ce9c34f41dec3d49e8; no voters: 584dac86183a41d9b3bea1c2085e62fa, d73d92fb35234d16824c55689b5fce28
W20260709 12:27:30.918687 29634 leader_election.cc:336] T d21ab1641501475b829863275dc8182a P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111)
I20260709 12:27:30.919402 30707 leader_election.cc:290] T d5fcd1f030614bbe82cf1dc261dfc1c3 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033), d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825)
I20260709 12:27:30.920176 30702 raft_consensus.cc:2749] T a3dcbb9c64bf4b5bb6680e8cb81af93e P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:27:30.921475 30709 leader_election.cc:290] T abec627108a84083884e64e1eb9dcda6 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033), d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825)
I20260709 12:27:30.926611 30677 leader_election.cc:290] T d21ab1641501475b829863275dc8182a P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033), d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825)
W20260709 12:27:30.928561 29634 leader_election.cc:336] T abec627108a84083884e64e1eb9dcda6 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111)
W20260709 12:27:30.929775 29634 leader_election.cc:336] T d5fcd1f030614bbe82cf1dc261dfc1c3 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111)
W20260709 12:27:30.931766 29634 leader_election.cc:336] T d5fcd1f030614bbe82cf1dc261dfc1c3 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Network error: Client connection negotiation failed: client connection to 127.16.29.66:36825: connect: Connection refused (error 111)
I20260709 12:27:30.932304 29634 leader_election.cc:304] T d5fcd1f030614bbe82cf1dc261dfc1c3 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d13c58ad8ac441c4a9b396eb04b1a476; no voters: 584dac86183a41d9b3bea1c2085e62fa, d73d92fb35234d16824c55689b5fce28
W20260709 12:27:30.933336 29634 leader_election.cc:336] T abec627108a84083884e64e1eb9dcda6 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Network error: Client connection negotiation failed: client connection to 127.16.29.66:36825: connect: Connection refused (error 111)
I20260709 12:27:30.933894 29634 leader_election.cc:304] T abec627108a84083884e64e1eb9dcda6 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d13c58ad8ac441c4a9b396eb04b1a476; no voters: 584dac86183a41d9b3bea1c2085e62fa, d73d92fb35234d16824c55689b5fce28
I20260709 12:27:30.936071 30709 raft_consensus.cc:2749] T d5fcd1f030614bbe82cf1dc261dfc1c3 P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:27:30.936273 30707 raft_consensus.cc:2749] T abec627108a84083884e64e1eb9dcda6 P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260709 12:27:30.940011 29634 leader_election.cc:336] T d21ab1641501475b829863275dc8182a P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Network error: Client connection negotiation failed: client connection to 127.16.29.66:36825: connect: Connection refused (error 111)
I20260709 12:27:30.940573 29634 leader_election.cc:304] T d21ab1641501475b829863275dc8182a P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d13c58ad8ac441c4a9b396eb04b1a476; no voters: 584dac86183a41d9b3bea1c2085e62fa, d73d92fb35234d16824c55689b5fce28
I20260709 12:27:30.941779 30677 raft_consensus.cc:2749] T d21ab1641501475b829863275dc8182a P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:27:31.109993 30681 raft_consensus.cc:493] T f8c01e05981747a395e5483b551297dd P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d73d92fb35234d16824c55689b5fce28)
I20260709 12:27:31.111071 30681 raft_consensus.cc:515] T f8c01e05981747a395e5483b551297dd P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:31.113392 30681 leader_election.cc:290] T f8c01e05981747a395e5483b551297dd P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033), d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825)
W20260709 12:27:31.125800 29708 leader_election.cc:336] T f8c01e05981747a395e5483b551297dd P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Network error: Client connection negotiation failed: client connection to 127.16.29.66:36825: connect: Connection refused (error 111)
W20260709 12:27:31.126645 29708 leader_election.cc:336] T f8c01e05981747a395e5483b551297dd P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111)
I20260709 12:27:31.127079 29708 leader_election.cc:304] T f8c01e05981747a395e5483b551297dd P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c6610c00ffb948ce9c34f41dec3d49e8; no voters: 584dac86183a41d9b3bea1c2085e62fa, d73d92fb35234d16824c55689b5fce28
I20260709 12:27:31.128007 30727 raft_consensus.cc:2749] T f8c01e05981747a395e5483b551297dd P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:27:31.180492 30681 raft_consensus.cc:493] T 5be58d64d45e40d7981018b84a01b1ed P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d73d92fb35234d16824c55689b5fce28)
I20260709 12:27:31.183106 30681 raft_consensus.cc:515] T 5be58d64d45e40d7981018b84a01b1ed P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:31.185389 30727 raft_consensus.cc:493] T 62f8c9472b214576b9002e02536a3ace P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d73d92fb35234d16824c55689b5fce28)
I20260709 12:27:31.186146 30681 leader_election.cc:290] T 5be58d64d45e40d7981018b84a01b1ed P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033), d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825)
I20260709 12:27:31.186137 30727 raft_consensus.cc:515] T 62f8c9472b214576b9002e02536a3ace P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:31.189499 30727 leader_election.cc:290] T 62f8c9472b214576b9002e02536a3ace P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033), d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825)
W20260709 12:27:31.196019 29708 leader_election.cc:336] T 5be58d64d45e40d7981018b84a01b1ed P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Network error: Client connection negotiation failed: client connection to 127.16.29.66:36825: connect: Connection refused (error 111)
W20260709 12:27:31.197139 29708 leader_election.cc:336] T 62f8c9472b214576b9002e02536a3ace P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Network error: Client connection negotiation failed: client connection to 127.16.29.66:36825: connect: Connection refused (error 111)
W20260709 12:27:31.197854 29708 leader_election.cc:336] T 5be58d64d45e40d7981018b84a01b1ed P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111)
I20260709 12:27:31.198287 29708 leader_election.cc:304] T 5be58d64d45e40d7981018b84a01b1ed P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c6610c00ffb948ce9c34f41dec3d49e8; no voters: 584dac86183a41d9b3bea1c2085e62fa, d73d92fb35234d16824c55689b5fce28
W20260709 12:27:31.199191 29708 leader_election.cc:336] T 62f8c9472b214576b9002e02536a3ace P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111)
I20260709 12:27:31.199606 30727 raft_consensus.cc:2749] T 5be58d64d45e40d7981018b84a01b1ed P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:27:31.199636 29708 leader_election.cc:304] T 62f8c9472b214576b9002e02536a3ace P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c6610c00ffb948ce9c34f41dec3d49e8; no voters: 584dac86183a41d9b3bea1c2085e62fa, d73d92fb35234d16824c55689b5fce28
I20260709 12:27:31.202451 30727 raft_consensus.cc:2749] T 62f8c9472b214576b9002e02536a3ace P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260709 12:27:31.239759 29634 consensus_peers.cc:597] T 9633e3b1d78f403097e4a00b7d4b869b P d13c58ad8ac441c4a9b396eb04b1a476 -> Peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Couldn't send request to peer 584dac86183a41d9b3bea1c2085e62fa. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:27:31.260241 30677 raft_consensus.cc:493] T 3615115233c34461b85763151751e475 P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d73d92fb35234d16824c55689b5fce28)
I20260709 12:27:31.260993 30677 raft_consensus.cc:515] T 3615115233c34461b85763151751e475 P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:27:31.264315 30677 leader_election.cc:290] T 3615115233c34461b85763151751e475 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033), d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825)
W20260709 12:27:31.270326 29634 leader_election.cc:336] T 3615115233c34461b85763151751e475 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Network error: Client connection negotiation failed: client connection to 127.16.29.66:36825: connect: Connection refused (error 111)
W20260709 12:27:31.271023 29634 leader_election.cc:336] T 3615115233c34461b85763151751e475 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111)
I20260709 12:27:31.271347 29634 leader_election.cc:304] T 3615115233c34461b85763151751e475 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d13c58ad8ac441c4a9b396eb04b1a476; no voters: 584dac86183a41d9b3bea1c2085e62fa, d73d92fb35234d16824c55689b5fce28
I20260709 12:27:31.271989 30677 raft_consensus.cc:2749] T 3615115233c34461b85763151751e475 P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:27:31.396971 30677 raft_consensus.cc:493] T 2707271fd46f456aa765026642a3efa9 P d13c58ad8ac441c4a9b396eb04b1a476 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d73d92fb35234d16824c55689b5fce28)
I20260709 12:27:31.397789 30677 raft_consensus.cc:515] T 2707271fd46f456aa765026642a3efa9 P d13c58ad8ac441c4a9b396eb04b1a476 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:27:31.399664 30677 leader_election.cc:290] T 2707271fd46f456aa765026642a3efa9 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033), d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825)
W20260709 12:27:31.407351 29634 leader_election.cc:336] T 2707271fd46f456aa765026642a3efa9 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Network error: Client connection negotiation failed: client connection to 127.16.29.66:36825: connect: Connection refused (error 111)
W20260709 12:27:31.413641 29634 leader_election.cc:336] T 2707271fd46f456aa765026642a3efa9 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111)
I20260709 12:27:31.414033 29634 leader_election.cc:304] T 2707271fd46f456aa765026642a3efa9 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d13c58ad8ac441c4a9b396eb04b1a476; no voters: 584dac86183a41d9b3bea1c2085e62fa, d73d92fb35234d16824c55689b5fce28
I20260709 12:27:31.415547 30677 raft_consensus.cc:2749] T 2707271fd46f456aa765026642a3efa9 P d13c58ad8ac441c4a9b396eb04b1a476 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260709 12:27:31.487998 29634 consensus_peers.cc:597] T 31a190e9f5ff407fab90502fb3197b70 P d13c58ad8ac441c4a9b396eb04b1a476 -> Peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Couldn't send request to peer 584dac86183a41d9b3bea1c2085e62fa. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:27:31.500366 29708 consensus_peers.cc:597] T f76101eb30f14b98a9aa8c41e0d6f831 P c6610c00ffb948ce9c34f41dec3d49e8 -> Peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Couldn't send request to peer 584dac86183a41d9b3bea1c2085e62fa. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:27:31.527223 29634 consensus_peers.cc:597] T 7ae3b542012d4f35a9c363eee22976ec P d13c58ad8ac441c4a9b396eb04b1a476 -> Peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Couldn't send request to peer 584dac86183a41d9b3bea1c2085e62fa. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:27:31.545667 29634 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111) [suppressed 514 similar messages]
I20260709 12:27:31.599120 30727 raft_consensus.cc:493] T 1d1c1f58c4d340ac951dfa403d34138a P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 584dac86183a41d9b3bea1c2085e62fa)
I20260709 12:27:31.599820 30727 raft_consensus.cc:515] T 1d1c1f58c4d340ac951dfa403d34138a P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:31.602137 30727 leader_election.cc:290] T 1d1c1f58c4d340ac951dfa403d34138a P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033), d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825)
W20260709 12:27:31.608901 29708 leader_election.cc:336] T 1d1c1f58c4d340ac951dfa403d34138a P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111)
W20260709 12:27:31.609704 29708 leader_election.cc:336] T 1d1c1f58c4d340ac951dfa403d34138a P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Network error: Client connection negotiation failed: client connection to 127.16.29.66:36825: connect: Connection refused (error 111)
I20260709 12:27:31.610095 29708 leader_election.cc:304] T 1d1c1f58c4d340ac951dfa403d34138a P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c6610c00ffb948ce9c34f41dec3d49e8; no voters: 584dac86183a41d9b3bea1c2085e62fa, d73d92fb35234d16824c55689b5fce28
I20260709 12:27:31.610920 30727 raft_consensus.cc:2749] T 1d1c1f58c4d340ac951dfa403d34138a P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260709 12:27:31.613979 29708 consensus_peers.cc:597] T 66f652ba2c224885a2aea974bfe8f88a P c6610c00ffb948ce9c34f41dec3d49e8 -> Peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Couldn't send request to peer 584dac86183a41d9b3bea1c2085e62fa. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:27:31.642849 29634 consensus_peers.cc:597] T 3f41ce441fb047c8bbb741b1b9cbc0ec P d13c58ad8ac441c4a9b396eb04b1a476 -> Peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Couldn't send request to peer 584dac86183a41d9b3bea1c2085e62fa. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:27:31.668715 29634 consensus_peers.cc:597] T 6e37d72807e547898f4fce9e8b82b2b0 P d13c58ad8ac441c4a9b396eb04b1a476 -> Peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Couldn't send request to peer 584dac86183a41d9b3bea1c2085e62fa. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:27:31.708774 29708 consensus_peers.cc:597] T 8627b58b4f42491597125eb45f8d3b61 P c6610c00ffb948ce9c34f41dec3d49e8 -> Peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Couldn't send request to peer 584dac86183a41d9b3bea1c2085e62fa. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:27:31.781577 29708 consensus_peers.cc:597] T 742beffede4f407d8a72909d2b147e6d P c6610c00ffb948ce9c34f41dec3d49e8 -> Peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Couldn't send request to peer 584dac86183a41d9b3bea1c2085e62fa. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 12:27:31.802549 30681 raft_consensus.cc:493] T b3c02c910d2b43cc9265bb1d98491cd3 P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 584dac86183a41d9b3bea1c2085e62fa)
I20260709 12:27:31.803216 30681 raft_consensus.cc:515] T b3c02c910d2b43cc9265bb1d98491cd3 P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:31.811127 30681 leader_election.cc:290] T b3c02c910d2b43cc9265bb1d98491cd3 P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033), d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825)
W20260709 12:27:31.820147 29708 leader_election.cc:336] T b3c02c910d2b43cc9265bb1d98491cd3 P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111)
W20260709 12:27:31.822329 29708 leader_election.cc:336] T b3c02c910d2b43cc9265bb1d98491cd3 P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Network error: Client connection negotiation failed: client connection to 127.16.29.66:36825: connect: Connection refused (error 111)
I20260709 12:27:31.823730 29708 leader_election.cc:304] T b3c02c910d2b43cc9265bb1d98491cd3 P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c6610c00ffb948ce9c34f41dec3d49e8; no voters: 584dac86183a41d9b3bea1c2085e62fa, d73d92fb35234d16824c55689b5fce28
I20260709 12:27:31.825464 30681 raft_consensus.cc:2749] T b3c02c910d2b43cc9265bb1d98491cd3 P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260709 12:27:31.828910 29708 consensus_peers.cc:597] T a46289e8fc824dd5a818e30df6e7ec3e P c6610c00ffb948ce9c34f41dec3d49e8 -> Peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Couldn't send request to peer 584dac86183a41d9b3bea1c2085e62fa. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:27:31.843098 29708 consensus_peers.cc:597] T 8017a44219404faa8db8ff049c65205e P c6610c00ffb948ce9c34f41dec3d49e8 -> Peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Couldn't send request to peer 584dac86183a41d9b3bea1c2085e62fa. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 12:27:31.875108 30727 raft_consensus.cc:493] T 6a30685caefb4d0e80fa4ea87d70760c P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 584dac86183a41d9b3bea1c2085e62fa)
I20260709 12:27:31.875805 30727 raft_consensus.cc:515] T 6a30685caefb4d0e80fa4ea87d70760c P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
W20260709 12:27:31.877893 29634 consensus_peers.cc:597] T f3e17f0a8fc14095850e8138fd79b86e P d13c58ad8ac441c4a9b396eb04b1a476 -> Peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Couldn't send request to peer 584dac86183a41d9b3bea1c2085e62fa. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 12:27:31.877893 30727 leader_election.cc:290] T 6a30685caefb4d0e80fa4ea87d70760c P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033), d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825)
W20260709 12:27:31.882869 29708 leader_election.cc:336] T 6a30685caefb4d0e80fa4ea87d70760c P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Network error: Client connection negotiation failed: client connection to 127.16.29.66:36825: connect: Connection refused (error 111)
W20260709 12:27:31.883908 29708 leader_election.cc:336] T 6a30685caefb4d0e80fa4ea87d70760c P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111)
I20260709 12:27:31.884320 29708 leader_election.cc:304] T 6a30685caefb4d0e80fa4ea87d70760c P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c6610c00ffb948ce9c34f41dec3d49e8; no voters: 584dac86183a41d9b3bea1c2085e62fa, d73d92fb35234d16824c55689b5fce28
I20260709 12:27:31.885057 30727 raft_consensus.cc:2749] T 6a30685caefb4d0e80fa4ea87d70760c P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260709 12:27:31.887509 16501 thread.cc:527] Waited for 4000ms trying to join with diag-logger (tid 29618)
W20260709 12:27:31.925308 29634 consensus_peers.cc:597] T fb7d78e534634e0f992fc39b578ebd8d P d13c58ad8ac441c4a9b396eb04b1a476 -> Peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Couldn't send request to peer 584dac86183a41d9b3bea1c2085e62fa. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 12:27:31.931766 30677 raft_consensus.cc:493] T d1918297b9eb41f5847a211e668ae748 P d13c58ad8ac441c4a9b396eb04b1a476 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 584dac86183a41d9b3bea1c2085e62fa)
I20260709 12:27:31.932636 30677 raft_consensus.cc:515] T d1918297b9eb41f5847a211e668ae748 P d13c58ad8ac441c4a9b396eb04b1a476 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
W20260709 12:27:31.934285 29634 consensus_peers.cc:597] T c6e621aa77d5474580dfc5949801ccc9 P d13c58ad8ac441c4a9b396eb04b1a476 -> Peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Couldn't send request to peer 584dac86183a41d9b3bea1c2085e62fa. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:27:31.934085 29708 consensus_peers.cc:597] T 425777eed34441cd93e0902bdc126516 P c6610c00ffb948ce9c34f41dec3d49e8 -> Peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Couldn't send request to peer 584dac86183a41d9b3bea1c2085e62fa. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 12:27:31.935946 30677 leader_election.cc:290] T d1918297b9eb41f5847a211e668ae748 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033), d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825)
W20260709 12:27:31.940766 29634 leader_election.cc:336] T d1918297b9eb41f5847a211e668ae748 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111)
W20260709 12:27:31.941676 29634 leader_election.cc:336] T d1918297b9eb41f5847a211e668ae748 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Network error: Client connection negotiation failed: client connection to 127.16.29.66:36825: connect: Connection refused (error 111)
I20260709 12:27:31.942139 29634 leader_election.cc:304] T d1918297b9eb41f5847a211e668ae748 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d13c58ad8ac441c4a9b396eb04b1a476; no voters: 584dac86183a41d9b3bea1c2085e62fa, d73d92fb35234d16824c55689b5fce28
I20260709 12:27:31.943002 30677 raft_consensus.cc:2749] T d1918297b9eb41f5847a211e668ae748 P d13c58ad8ac441c4a9b396eb04b1a476 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260709 12:27:31.977382 29634 consensus_peers.cc:597] T 0c428ff656ae4b9d99d5b5073f98725d P d13c58ad8ac441c4a9b396eb04b1a476 -> Peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Couldn't send request to peer 584dac86183a41d9b3bea1c2085e62fa. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:27:32.013571 29634 consensus_peers.cc:597] T e67a2affef634fd8bc96a64fa83cffdb P d13c58ad8ac441c4a9b396eb04b1a476 -> Peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Couldn't send request to peer 584dac86183a41d9b3bea1c2085e62fa. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:27:32.028297 29634 consensus_peers.cc:597] T 3e9451c3797c4496a66dfdb3e9fb5e38 P d13c58ad8ac441c4a9b396eb04b1a476 -> Peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Couldn't send request to peer 584dac86183a41d9b3bea1c2085e62fa. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:27:32.065284 29708 consensus_peers.cc:597] T d299fb38094e47e8aa3890ca74f6b366 P c6610c00ffb948ce9c34f41dec3d49e8 -> Peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Couldn't send request to peer d73d92fb35234d16824c55689b5fce28. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.66:36825: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:27:32.068240 29708 consensus_peers.cc:597] T f4516786e5564e9eac08ca472f79a9f0 P c6610c00ffb948ce9c34f41dec3d49e8 -> Peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Couldn't send request to peer 584dac86183a41d9b3bea1c2085e62fa. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:27:32.117352 29708 consensus_peers.cc:597] T ef20d7124e7a44aa97a219ee194f4b8b P c6610c00ffb948ce9c34f41dec3d49e8 -> Peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Couldn't send request to peer 584dac86183a41d9b3bea1c2085e62fa. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:27:32.139354 29708 consensus_peers.cc:597] T 137e03144e754f6e9da4f0544513d27c P c6610c00ffb948ce9c34f41dec3d49e8 -> Peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Couldn't send request to peer d73d92fb35234d16824c55689b5fce28. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.66:36825: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:27:32.146813 29634 consensus_peers.cc:597] T 00c6cf271c91400aa834467def9af9b1 P d13c58ad8ac441c4a9b396eb04b1a476 -> Peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Couldn't send request to peer d73d92fb35234d16824c55689b5fce28. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.66:36825: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:27:32.165038 29708 consensus_peers.cc:597] T 1d4e0ad0462e4f74a44155aa3ffaf87f P c6610c00ffb948ce9c34f41dec3d49e8 -> Peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Couldn't send request to peer 584dac86183a41d9b3bea1c2085e62fa. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 12:27:32.191758 30727 raft_consensus.cc:493] T 01788883c8ea42989f246a3486111a7e P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 584dac86183a41d9b3bea1c2085e62fa)
I20260709 12:27:32.192512 30727 raft_consensus.cc:515] T 01788883c8ea42989f246a3486111a7e P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
W20260709 12:27:32.197961 29634 consensus_peers.cc:597] T cbf03426b6df43e5b2b9c56e2eb7687c P d13c58ad8ac441c4a9b396eb04b1a476 -> Peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Couldn't send request to peer 584dac86183a41d9b3bea1c2085e62fa. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 12:27:32.198010 30727 leader_election.cc:290] T 01788883c8ea42989f246a3486111a7e P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033), d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825)
W20260709 12:27:32.205458 29708 leader_election.cc:336] T 01788883c8ea42989f246a3486111a7e P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111)
W20260709 12:27:32.206496 29708 leader_election.cc:336] T 01788883c8ea42989f246a3486111a7e P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Network error: Client connection negotiation failed: client connection to 127.16.29.66:36825: connect: Connection refused (error 111)
I20260709 12:27:32.206947 29708 leader_election.cc:304] T 01788883c8ea42989f246a3486111a7e P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c6610c00ffb948ce9c34f41dec3d49e8; no voters: 584dac86183a41d9b3bea1c2085e62fa, d73d92fb35234d16824c55689b5fce28
I20260709 12:27:32.207607 30727 raft_consensus.cc:2749] T 01788883c8ea42989f246a3486111a7e P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:27:32.234359 30727 raft_consensus.cc:493] T 4685c2e824c14fc1bcd45c16169798cc P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 584dac86183a41d9b3bea1c2085e62fa)
I20260709 12:27:32.235035 30727 raft_consensus.cc:515] T 4685c2e824c14fc1bcd45c16169798cc P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:32.237156 30727 leader_election.cc:290] T 4685c2e824c14fc1bcd45c16169798cc P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033), d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825)
W20260709 12:27:32.245074 29708 leader_election.cc:336] T 4685c2e824c14fc1bcd45c16169798cc P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111)
W20260709 12:27:32.245847 29708 leader_election.cc:336] T 4685c2e824c14fc1bcd45c16169798cc P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Network error: Client connection negotiation failed: client connection to 127.16.29.66:36825: connect: Connection refused (error 111)
I20260709 12:27:32.246345 29708 leader_election.cc:304] T 4685c2e824c14fc1bcd45c16169798cc P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c6610c00ffb948ce9c34f41dec3d49e8; no voters: 584dac86183a41d9b3bea1c2085e62fa, d73d92fb35234d16824c55689b5fce28
I20260709 12:27:32.247841 30727 raft_consensus.cc:2749] T 4685c2e824c14fc1bcd45c16169798cc P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260709 12:27:32.330611 29634 consensus_peers.cc:597] T 5ee924f859f54f64908499bfac48ee62 P d13c58ad8ac441c4a9b396eb04b1a476 -> Peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Couldn't send request to peer 584dac86183a41d9b3bea1c2085e62fa. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:27:32.338514 29634 consensus_peers.cc:597] T d42259e436d241b29fb4ded16d9b6455 P d13c58ad8ac441c4a9b396eb04b1a476 -> Peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Couldn't send request to peer 584dac86183a41d9b3bea1c2085e62fa. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 12:27:32.365336 30727 raft_consensus.cc:493] T fb6e78ccc01c4707af34cddf81619eab P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 584dac86183a41d9b3bea1c2085e62fa)
I20260709 12:27:32.366142 30727 raft_consensus.cc:515] T fb6e78ccc01c4707af34cddf81619eab P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:32.368582 30727 leader_election.cc:290] T fb6e78ccc01c4707af34cddf81619eab P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033), d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825)
W20260709 12:27:32.375804 29708 leader_election.cc:336] T fb6e78ccc01c4707af34cddf81619eab P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111)
W20260709 12:27:32.376745 29708 leader_election.cc:336] T fb6e78ccc01c4707af34cddf81619eab P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Network error: Client connection negotiation failed: client connection to 127.16.29.66:36825: connect: Connection refused (error 111)
I20260709 12:27:32.377210 29708 leader_election.cc:304] T fb6e78ccc01c4707af34cddf81619eab P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c6610c00ffb948ce9c34f41dec3d49e8; no voters: 584dac86183a41d9b3bea1c2085e62fa, d73d92fb35234d16824c55689b5fce28
I20260709 12:27:32.378043 30727 raft_consensus.cc:2749] T fb6e78ccc01c4707af34cddf81619eab P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260709 12:27:32.386219 29708 consensus_peers.cc:597] T 55061e62184b45f0bb1726adc1a4af81 P c6610c00ffb948ce9c34f41dec3d49e8 -> Peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Couldn't send request to peer d73d92fb35234d16824c55689b5fce28. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.66:36825: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:27:32.463992 29634 consensus_peers.cc:597] T cbf03426b6df43e5b2b9c56e2eb7687c P d13c58ad8ac441c4a9b396eb04b1a476 -> Peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Couldn't send request to peer d73d92fb35234d16824c55689b5fce28. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.66:36825: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 12:27:32.473091 30707 raft_consensus.cc:493] T 16649fd17d434e3c95e9d585d2404659 P d13c58ad8ac441c4a9b396eb04b1a476 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d73d92fb35234d16824c55689b5fce28)
I20260709 12:27:32.473905 30707 raft_consensus.cc:515] T 16649fd17d434e3c95e9d585d2404659 P d13c58ad8ac441c4a9b396eb04b1a476 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:27:32.476090 30707 leader_election.cc:290] T 16649fd17d434e3c95e9d585d2404659 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033), d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825)
W20260709 12:27:32.488996 29634 leader_election.cc:336] T 16649fd17d434e3c95e9d585d2404659 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111)
W20260709 12:27:32.490235 29634 leader_election.cc:336] T 16649fd17d434e3c95e9d585d2404659 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Network error: Client connection negotiation failed: client connection to 127.16.29.66:36825: connect: Connection refused (error 111)
I20260709 12:27:32.490984 29634 leader_election.cc:304] T 16649fd17d434e3c95e9d585d2404659 P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d13c58ad8ac441c4a9b396eb04b1a476; no voters: 584dac86183a41d9b3bea1c2085e62fa, d73d92fb35234d16824c55689b5fce28
I20260709 12:27:32.492260 30707 raft_consensus.cc:2749] T 16649fd17d434e3c95e9d585d2404659 P d13c58ad8ac441c4a9b396eb04b1a476 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260709 12:27:32.504277 29634 consensus_peers.cc:597] T 07a2b94c0823467695508292a9ccdcc5 P d13c58ad8ac441c4a9b396eb04b1a476 -> Peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Couldn't send request to peer d73d92fb35234d16824c55689b5fce28. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.66:36825: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:27:32.574858 29708 consensus_peers.cc:597] T 8017a44219404faa8db8ff049c65205e P c6610c00ffb948ce9c34f41dec3d49e8 -> Peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Couldn't send request to peer d73d92fb35234d16824c55689b5fce28. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.66:36825: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:27:32.584558 29708 consensus_peers.cc:597] T c1c8411289a74b5685700229d245ff75 P c6610c00ffb948ce9c34f41dec3d49e8 -> Peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Couldn't send request to peer d73d92fb35234d16824c55689b5fce28. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.66:36825: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:27:32.622200 16501 tablet_server.cc:196] TabletServer@127.16.29.66:0 shutdown complete.
W20260709 12:27:32.623346 29708 consensus_peers.cc:597] T 1d4e0ad0462e4f74a44155aa3ffaf87f P c6610c00ffb948ce9c34f41dec3d49e8 -> Peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Couldn't send request to peer d73d92fb35234d16824c55689b5fce28. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.66:36825: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:27:32.631460 29708 consensus_peers.cc:597] T 394378d078194257b9e04fc659c1f5d4 P c6610c00ffb948ce9c34f41dec3d49e8 -> Peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Couldn't send request to peer d73d92fb35234d16824c55689b5fce28. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.66:36825: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:27:32.631783 29634 consensus_peers.cc:597] T cdf706ffabc34902ad3d1bf2cee7889e P d13c58ad8ac441c4a9b396eb04b1a476 -> Peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Couldn't send request to peer d73d92fb35234d16824c55689b5fce28. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.66:36825: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:27:32.652042 30730 raft_consensus.cc:493] T d21ab1641501475b829863275dc8182a P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d73d92fb35234d16824c55689b5fce28)
I20260709 12:27:32.653318 30730 raft_consensus.cc:515] T d21ab1641501475b829863275dc8182a P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:27:32.660912 30730 leader_election.cc:290] T d21ab1641501475b829863275dc8182a P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033), d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825)
W20260709 12:27:32.675347 29634 leader_election.cc:336] T d21ab1641501475b829863275dc8182a P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111)
W20260709 12:27:32.676823 29634 consensus_peers.cc:597] T d42259e436d241b29fb4ded16d9b6455 P d13c58ad8ac441c4a9b396eb04b1a476 -> Peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Couldn't send request to peer d73d92fb35234d16824c55689b5fce28. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.66:36825: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:27:32.677508 29634 leader_election.cc:336] T d21ab1641501475b829863275dc8182a P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Network error: Client connection negotiation failed: client connection to 127.16.29.66:36825: connect: Connection refused (error 111)
I20260709 12:27:32.678098 29634 leader_election.cc:304] T d21ab1641501475b829863275dc8182a P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d13c58ad8ac441c4a9b396eb04b1a476; no voters: 584dac86183a41d9b3bea1c2085e62fa, d73d92fb35234d16824c55689b5fce28
I20260709 12:27:32.679047 30730 raft_consensus.cc:2749] T d21ab1641501475b829863275dc8182a P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260709 12:27:32.680320 29634 consensus_peers.cc:597] T 5ee924f859f54f64908499bfac48ee62 P d13c58ad8ac441c4a9b396eb04b1a476 -> Peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Couldn't send request to peer d73d92fb35234d16824c55689b5fce28. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.66:36825: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:27:32.743284 29708 consensus_peers.cc:597] T 959940284eb24fdf83d78438d710318a P c6610c00ffb948ce9c34f41dec3d49e8 -> Peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Couldn't send request to peer d73d92fb35234d16824c55689b5fce28. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.66:36825: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:27:32.759761 29708 consensus_peers.cc:597] T 58154d49c1a344f2b51366ede3c87de5 P c6610c00ffb948ce9c34f41dec3d49e8 -> Peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Couldn't send request to peer d73d92fb35234d16824c55689b5fce28. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.66:36825: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 12:27:32.778409 16501 tablet_server.cc:179] TabletServer@127.16.29.67:0 shutting down...
I20260709 12:27:32.780174 30730 raft_consensus.cc:493] T fcb6a7f720b14dc98e821f663357a0bb P d13c58ad8ac441c4a9b396eb04b1a476 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 584dac86183a41d9b3bea1c2085e62fa)
I20260709 12:27:32.783233 30730 raft_consensus.cc:515] T fcb6a7f720b14dc98e821f663357a0bb P d13c58ad8ac441c4a9b396eb04b1a476 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
I20260709 12:27:32.785624 30730 leader_election.cc:290] T fcb6a7f720b14dc98e821f663357a0bb P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033), d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825)
W20260709 12:27:32.787420 29709 consensus_peers.cc:597] T 425777eed34441cd93e0902bdc126516 P c6610c00ffb948ce9c34f41dec3d49e8 -> Peer d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209): Couldn't send request to peer d13c58ad8ac441c4a9b396eb04b1a476. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:32.793015 29634 leader_election.cc:336] T fcb6a7f720b14dc98e821f663357a0bb P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111)
W20260709 12:27:32.793699 29634 leader_election.cc:336] T fcb6a7f720b14dc98e821f663357a0bb P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Network error: Client connection negotiation failed: client connection to 127.16.29.66:36825: connect: Connection refused (error 111)
I20260709 12:27:32.794142 29634 leader_election.cc:304] T fcb6a7f720b14dc98e821f663357a0bb P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d13c58ad8ac441c4a9b396eb04b1a476; no voters: 584dac86183a41d9b3bea1c2085e62fa, d73d92fb35234d16824c55689b5fce28
I20260709 12:27:32.797138 30730 raft_consensus.cc:2749] T fcb6a7f720b14dc98e821f663357a0bb P d13c58ad8ac441c4a9b396eb04b1a476 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260709 12:27:32.808079 29708 consensus_peers.cc:597] T ed057e06dec148a0b9d7062beeb9d005 P c6610c00ffb948ce9c34f41dec3d49e8 -> Peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Couldn't send request to peer d73d92fb35234d16824c55689b5fce28. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.66:36825: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 12:27:32.817826 30677 raft_consensus.cc:493] T 71e2de235adf46d9a2407659080d42ef P d13c58ad8ac441c4a9b396eb04b1a476 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 584dac86183a41d9b3bea1c2085e62fa)
I20260709 12:27:32.819418 30677 raft_consensus.cc:515] T 71e2de235adf46d9a2407659080d42ef P d13c58ad8ac441c4a9b396eb04b1a476 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } }
W20260709 12:27:32.824337 29708 consensus_peers.cc:597] T 43c7bfb33af74d4f912e5153e8a54b3e P c6610c00ffb948ce9c34f41dec3d49e8 -> Peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Couldn't send request to peer d73d92fb35234d16824c55689b5fce28. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.66:36825: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 12:27:32.826365 30728 raft_consensus.cc:493] T 62f8c9472b214576b9002e02536a3ace P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d73d92fb35234d16824c55689b5fce28)
I20260709 12:27:32.827437 30677 leader_election.cc:290] T 71e2de235adf46d9a2407659080d42ef P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033), d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825)
I20260709 12:27:32.827180 30728 raft_consensus.cc:515] T 62f8c9472b214576b9002e02536a3ace P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:32.829437 30728 leader_election.cc:290] T 62f8c9472b214576b9002e02536a3ace P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033), d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825)
W20260709 12:27:32.825800 29634 consensus_peers.cc:597] T 3617af2dee3f48cf85973b1c94cdc33a P d13c58ad8ac441c4a9b396eb04b1a476 -> Peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Couldn't send request to peer d73d92fb35234d16824c55689b5fce28. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.66:36825: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:27:32.839820 29634 leader_election.cc:336] T 71e2de235adf46d9a2407659080d42ef P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111)
W20260709 12:27:32.841463 29708 leader_election.cc:336] T 62f8c9472b214576b9002e02536a3ace P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Network error: Client connection negotiation failed: client connection to 127.16.29.66:36825: connect: Connection refused (error 111)
W20260709 12:27:32.842610 29708 leader_election.cc:336] T 62f8c9472b214576b9002e02536a3ace P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111)
I20260709 12:27:32.843149 29708 leader_election.cc:304] T 62f8c9472b214576b9002e02536a3ace P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c6610c00ffb948ce9c34f41dec3d49e8; no voters: 584dac86183a41d9b3bea1c2085e62fa, d73d92fb35234d16824c55689b5fce28
I20260709 12:27:32.844169 30729 raft_consensus.cc:2749] T 62f8c9472b214576b9002e02536a3ace P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260709 12:27:32.847397 29709 consensus_peers.cc:597] T 137e03144e754f6e9da4f0544513d27c P c6610c00ffb948ce9c34f41dec3d49e8 -> Peer d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209): Couldn't send request to peer d13c58ad8ac441c4a9b396eb04b1a476. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:32.854290 29708 consensus_peers.cc:597] T 66f652ba2c224885a2aea974bfe8f88a P c6610c00ffb948ce9c34f41dec3d49e8 -> Peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Couldn't send request to peer d73d92fb35234d16824c55689b5fce28. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.66:36825: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:27:32.861078 29709 consensus_peers.cc:597] T ed057e06dec148a0b9d7062beeb9d005 P c6610c00ffb948ce9c34f41dec3d49e8 -> Peer d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209): Couldn't send request to peer d13c58ad8ac441c4a9b396eb04b1a476. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:32.875290 16501 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:27:32.878746 16501 tablet_replica.cc:333] T 394378d078194257b9e04fc659c1f5d4 P d13c58ad8ac441c4a9b396eb04b1a476: stopping tablet replica
I20260709 12:27:32.880118 16501 raft_consensus.cc:2243] T 394378d078194257b9e04fc659c1f5d4 P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:32.880810 16501 raft_consensus.cc:2272] T 394378d078194257b9e04fc659c1f5d4 P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:32.884572 16501 tablet_replica.cc:333] T 137e03144e754f6e9da4f0544513d27c P d13c58ad8ac441c4a9b396eb04b1a476: stopping tablet replica
I20260709 12:27:32.885699 16501 raft_consensus.cc:2243] T 137e03144e754f6e9da4f0544513d27c P d13c58ad8ac441c4a9b396eb04b1a476 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:32.886425 16501 raft_consensus.cc:2272] T 137e03144e754f6e9da4f0544513d27c P d13c58ad8ac441c4a9b396eb04b1a476 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:32.890108 16501 tablet_replica.cc:333] T abec627108a84083884e64e1eb9dcda6 P d13c58ad8ac441c4a9b396eb04b1a476: stopping tablet replica
I20260709 12:27:32.891312 16501 raft_consensus.cc:2243] T abec627108a84083884e64e1eb9dcda6 P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:32.891958 16501 raft_consensus.cc:2272] T abec627108a84083884e64e1eb9dcda6 P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 12:27:32.894387 29709 consensus_peers.cc:597] T 58154d49c1a344f2b51366ede3c87de5 P c6610c00ffb948ce9c34f41dec3d49e8 -> Peer d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209): Couldn't send request to peer d13c58ad8ac441c4a9b396eb04b1a476. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:32.895500 16501 tablet_replica.cc:333] T f76101eb30f14b98a9aa8c41e0d6f831 P d13c58ad8ac441c4a9b396eb04b1a476: stopping tablet replica
W20260709 12:27:32.895839 29709 consensus_peers.cc:597] T 55061e62184b45f0bb1726adc1a4af81 P c6610c00ffb948ce9c34f41dec3d49e8 -> Peer d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209): Couldn't send request to peer d13c58ad8ac441c4a9b396eb04b1a476. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:32.896647 16501 raft_consensus.cc:2243] T f76101eb30f14b98a9aa8c41e0d6f831 P d13c58ad8ac441c4a9b396eb04b1a476 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:32.897357 16501 raft_consensus.cc:2272] T f76101eb30f14b98a9aa8c41e0d6f831 P d13c58ad8ac441c4a9b396eb04b1a476 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:32.900930 16501 tablet_replica.cc:333] T fb7d78e534634e0f992fc39b578ebd8d P d13c58ad8ac441c4a9b396eb04b1a476: stopping tablet replica
I20260709 12:27:32.901827 16501 raft_consensus.cc:2243] T fb7d78e534634e0f992fc39b578ebd8d P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:27:32.903136 16501 raft_consensus.cc:2272] T fb7d78e534634e0f992fc39b578ebd8d P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:32.906492 16501 tablet_replica.cc:333] T d299fb38094e47e8aa3890ca74f6b366 P d13c58ad8ac441c4a9b396eb04b1a476: stopping tablet replica
I20260709 12:27:32.907498 16501 raft_consensus.cc:2243] T d299fb38094e47e8aa3890ca74f6b366 P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:32.908190 16501 raft_consensus.cc:2272] T d299fb38094e47e8aa3890ca74f6b366 P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:32.911754 16501 tablet_replica.cc:333] T cbf03426b6df43e5b2b9c56e2eb7687c P d13c58ad8ac441c4a9b396eb04b1a476: stopping tablet replica
I20260709 12:27:32.912791 16501 raft_consensus.cc:2243] T cbf03426b6df43e5b2b9c56e2eb7687c P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:27:32.914091 16501 raft_consensus.cc:2272] T cbf03426b6df43e5b2b9c56e2eb7687c P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:32.917711 16501 tablet_replica.cc:333] T c1c8411289a74b5685700229d245ff75 P d13c58ad8ac441c4a9b396eb04b1a476: stopping tablet replica
I20260709 12:27:32.918643 16501 raft_consensus.cc:2243] T c1c8411289a74b5685700229d245ff75 P d13c58ad8ac441c4a9b396eb04b1a476 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:32.919335 16501 raft_consensus.cc:2272] T c1c8411289a74b5685700229d245ff75 P d13c58ad8ac441c4a9b396eb04b1a476 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:32.922645 16501 tablet_replica.cc:333] T 16649fd17d434e3c95e9d585d2404659 P d13c58ad8ac441c4a9b396eb04b1a476: stopping tablet replica
I20260709 12:27:32.923671 16501 raft_consensus.cc:2243] T 16649fd17d434e3c95e9d585d2404659 P d13c58ad8ac441c4a9b396eb04b1a476 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:32.924558 16501 raft_consensus.cc:2272] T 16649fd17d434e3c95e9d585d2404659 P d13c58ad8ac441c4a9b396eb04b1a476 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:32.927935 16501 tablet_replica.cc:333] T 959940284eb24fdf83d78438d710318a P d13c58ad8ac441c4a9b396eb04b1a476: stopping tablet replica
I20260709 12:27:32.928931 16501 raft_consensus.cc:2243] T 959940284eb24fdf83d78438d710318a P d13c58ad8ac441c4a9b396eb04b1a476 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:32.929603 16501 raft_consensus.cc:2272] T 959940284eb24fdf83d78438d710318a P d13c58ad8ac441c4a9b396eb04b1a476 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:32.933187 16501 tablet_replica.cc:333] T 31a190e9f5ff407fab90502fb3197b70 P d13c58ad8ac441c4a9b396eb04b1a476: stopping tablet replica
I20260709 12:27:32.934016 16501 raft_consensus.cc:2243] T 31a190e9f5ff407fab90502fb3197b70 P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:27:32.935266 16501 raft_consensus.cc:2272] T 31a190e9f5ff407fab90502fb3197b70 P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:32.938686 16501 tablet_replica.cc:333] T 3f41ce441fb047c8bbb741b1b9cbc0ec P d13c58ad8ac441c4a9b396eb04b1a476: stopping tablet replica
I20260709 12:27:32.939604 16501 raft_consensus.cc:2243] T 3f41ce441fb047c8bbb741b1b9cbc0ec P d13c58ad8ac441c4a9b396eb04b1a476 [term 4 LEADER]: Raft consensus shutting down.
I20260709 12:27:32.941190 16501 raft_consensus.cc:2272] T 3f41ce441fb047c8bbb741b1b9cbc0ec P d13c58ad8ac441c4a9b396eb04b1a476 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:32.945032 16501 tablet_replica.cc:333] T 6e37d72807e547898f4fce9e8b82b2b0 P d13c58ad8ac441c4a9b396eb04b1a476: stopping tablet replica
I20260709 12:27:32.946237 16501 raft_consensus.cc:2243] T 6e37d72807e547898f4fce9e8b82b2b0 P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:27:32.947729 16501 raft_consensus.cc:2272] T 6e37d72807e547898f4fce9e8b82b2b0 P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:32.949935 30729 raft_consensus.cc:493] T f8c01e05981747a395e5483b551297dd P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d73d92fb35234d16824c55689b5fce28)
I20260709 12:27:32.950634 30729 raft_consensus.cc:515] T f8c01e05981747a395e5483b551297dd P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:32.951546 16501 tablet_replica.cc:333] T 07a2b94c0823467695508292a9ccdcc5 P d13c58ad8ac441c4a9b396eb04b1a476: stopping tablet replica
I20260709 12:27:32.952525 16501 raft_consensus.cc:2243] T 07a2b94c0823467695508292a9ccdcc5 P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:27:32.953862 16501 raft_consensus.cc:2272] T 07a2b94c0823467695508292a9ccdcc5 P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 12:27:32.955343 29709 consensus_peers.cc:597] T f4516786e5564e9eac08ca472f79a9f0 P c6610c00ffb948ce9c34f41dec3d49e8 -> Peer d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209): Couldn't send request to peer d13c58ad8ac441c4a9b396eb04b1a476. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:32.955552 30729 leader_election.cc:290] T f8c01e05981747a395e5483b551297dd P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033), d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825)
W20260709 12:27:32.956903 29709 consensus_peers.cc:597] T a46289e8fc824dd5a818e30df6e7ec3e P c6610c00ffb948ce9c34f41dec3d49e8 -> Peer d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209): Couldn't send request to peer d13c58ad8ac441c4a9b396eb04b1a476. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:32.957229 16501 tablet_replica.cc:333] T f4516786e5564e9eac08ca472f79a9f0 P d13c58ad8ac441c4a9b396eb04b1a476: stopping tablet replica
I20260709 12:27:32.958184 16501 raft_consensus.cc:2243] T f4516786e5564e9eac08ca472f79a9f0 P d13c58ad8ac441c4a9b396eb04b1a476 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:32.958745 16501 raft_consensus.cc:2272] T f4516786e5564e9eac08ca472f79a9f0 P d13c58ad8ac441c4a9b396eb04b1a476 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:32.962095 16501 tablet_replica.cc:333] T 71e2de235adf46d9a2407659080d42ef P d13c58ad8ac441c4a9b396eb04b1a476: stopping tablet replica
I20260709 12:27:32.962819 16501 raft_consensus.cc:2243] T 71e2de235adf46d9a2407659080d42ef P d13c58ad8ac441c4a9b396eb04b1a476 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:32.963404 16501 raft_consensus.cc:2272] T 71e2de235adf46d9a2407659080d42ef P d13c58ad8ac441c4a9b396eb04b1a476 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:32.966149 16501 tablet_replica.cc:333] T f3e17f0a8fc14095850e8138fd79b86e P d13c58ad8ac441c4a9b396eb04b1a476: stopping tablet replica
I20260709 12:27:32.967051 16501 raft_consensus.cc:2243] T f3e17f0a8fc14095850e8138fd79b86e P d13c58ad8ac441c4a9b396eb04b1a476 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:27:32.968235 16501 raft_consensus.cc:2272] T f3e17f0a8fc14095850e8138fd79b86e P d13c58ad8ac441c4a9b396eb04b1a476 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:32.971403 16501 tablet_replica.cc:333] T 425777eed34441cd93e0902bdc126516 P d13c58ad8ac441c4a9b396eb04b1a476: stopping tablet replica
I20260709 12:27:32.972348 16501 raft_consensus.cc:2243] T 425777eed34441cd93e0902bdc126516 P d13c58ad8ac441c4a9b396eb04b1a476 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:32.973055 16501 raft_consensus.cc:2272] T 425777eed34441cd93e0902bdc126516 P d13c58ad8ac441c4a9b396eb04b1a476 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:32.976073 16501 tablet_replica.cc:333] T 3e9451c3797c4496a66dfdb3e9fb5e38 P d13c58ad8ac441c4a9b396eb04b1a476: stopping tablet replica
I20260709 12:27:32.976907 16501 raft_consensus.cc:2243] T 3e9451c3797c4496a66dfdb3e9fb5e38 P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 LEADER]: Raft consensus shutting down.
W20260709 12:27:32.976940 29709 consensus_peers.cc:597] T c1c8411289a74b5685700229d245ff75 P c6610c00ffb948ce9c34f41dec3d49e8 -> Peer d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209): Couldn't send request to peer d13c58ad8ac441c4a9b396eb04b1a476. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:32.978152 16501 raft_consensus.cc:2272] T 3e9451c3797c4496a66dfdb3e9fb5e38 P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 12:27:32.978588 29708 leader_election.cc:336] T f8c01e05981747a395e5483b551297dd P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111)
W20260709 12:27:32.979676 29708 leader_election.cc:336] T f8c01e05981747a395e5483b551297dd P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Network error: Client connection negotiation failed: client connection to 127.16.29.66:36825: connect: Connection refused (error 111)
I20260709 12:27:32.980185 29708 leader_election.cc:304] T f8c01e05981747a395e5483b551297dd P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c6610c00ffb948ce9c34f41dec3d49e8; no voters: 584dac86183a41d9b3bea1c2085e62fa, d73d92fb35234d16824c55689b5fce28
I20260709 12:27:32.981055 16501 tablet_replica.cc:333] T c6e621aa77d5474580dfc5949801ccc9 P d13c58ad8ac441c4a9b396eb04b1a476: stopping tablet replica
I20260709 12:27:32.980997 30729 raft_consensus.cc:2749] T f8c01e05981747a395e5483b551297dd P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:27:32.981948 16501 raft_consensus.cc:2243] T c6e621aa77d5474580dfc5949801ccc9 P d13c58ad8ac441c4a9b396eb04b1a476 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:27:32.983103 16501 raft_consensus.cc:2272] T c6e621aa77d5474580dfc5949801ccc9 P d13c58ad8ac441c4a9b396eb04b1a476 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:32.986189 16501 tablet_replica.cc:333] T 3615115233c34461b85763151751e475 P d13c58ad8ac441c4a9b396eb04b1a476: stopping tablet replica
I20260709 12:27:32.986979 16501 raft_consensus.cc:2243] T 3615115233c34461b85763151751e475 P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:32.987653 16501 raft_consensus.cc:2272] T 3615115233c34461b85763151751e475 P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:32.990510 16501 tablet_replica.cc:333] T ed057e06dec148a0b9d7062beeb9d005 P d13c58ad8ac441c4a9b396eb04b1a476: stopping tablet replica
I20260709 12:27:32.991510 16501 raft_consensus.cc:2243] T ed057e06dec148a0b9d7062beeb9d005 P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:32.992082 16501 raft_consensus.cc:2272] T ed057e06dec148a0b9d7062beeb9d005 P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:32.995240 16501 tablet_replica.cc:333] T a46289e8fc824dd5a818e30df6e7ec3e P d13c58ad8ac441c4a9b396eb04b1a476: stopping tablet replica
W20260709 12:27:32.995257 29709 consensus_peers.cc:597] T 959940284eb24fdf83d78438d710318a P c6610c00ffb948ce9c34f41dec3d49e8 -> Peer d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209): Couldn't send request to peer d13c58ad8ac441c4a9b396eb04b1a476. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:32.996042 16501 raft_consensus.cc:2243] T a46289e8fc824dd5a818e30df6e7ec3e P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:32.996584 16501 raft_consensus.cc:2272] T a46289e8fc824dd5a818e30df6e7ec3e P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:32.999680 16501 tablet_replica.cc:333] T 76140ad6258543118a9bfeee7cf0d5fc P d13c58ad8ac441c4a9b396eb04b1a476: stopping tablet replica
I20260709 12:27:33.000478 16501 raft_consensus.cc:2243] T 76140ad6258543118a9bfeee7cf0d5fc P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 LEADER]: Raft consensus shutting down.
W20260709 12:27:33.001339 29709 consensus_peers.cc:597] T 742beffede4f407d8a72909d2b147e6d P c6610c00ffb948ce9c34f41dec3d49e8 -> Peer d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209): Couldn't send request to peer d13c58ad8ac441c4a9b396eb04b1a476. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:33.001405 16501 raft_consensus.cc:2272] T 76140ad6258543118a9bfeee7cf0d5fc P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:33.004122 16501 tablet_replica.cc:333] T 9633e3b1d78f403097e4a00b7d4b869b P d13c58ad8ac441c4a9b396eb04b1a476: stopping tablet replica
I20260709 12:27:33.004742 16501 raft_consensus.cc:2243] T 9633e3b1d78f403097e4a00b7d4b869b P d13c58ad8ac441c4a9b396eb04b1a476 [term 3 LEADER]: Raft consensus shutting down.
I20260709 12:27:33.005659 16501 raft_consensus.cc:2272] T 9633e3b1d78f403097e4a00b7d4b869b P d13c58ad8ac441c4a9b396eb04b1a476 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:33.007807 16501 tablet_replica.cc:333] T 8627b58b4f42491597125eb45f8d3b61 P d13c58ad8ac441c4a9b396eb04b1a476: stopping tablet replica
I20260709 12:27:33.008411 16501 raft_consensus.cc:2243] T 8627b58b4f42491597125eb45f8d3b61 P d13c58ad8ac441c4a9b396eb04b1a476 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:33.008874 16501 raft_consensus.cc:2272] T 8627b58b4f42491597125eb45f8d3b61 P d13c58ad8ac441c4a9b396eb04b1a476 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:33.011212 16501 tablet_replica.cc:333] T 2707271fd46f456aa765026642a3efa9 P d13c58ad8ac441c4a9b396eb04b1a476: stopping tablet replica
I20260709 12:27:33.011755 16501 raft_consensus.cc:2243] T 2707271fd46f456aa765026642a3efa9 P d13c58ad8ac441c4a9b396eb04b1a476 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:33.012274 16501 raft_consensus.cc:2272] T 2707271fd46f456aa765026642a3efa9 P d13c58ad8ac441c4a9b396eb04b1a476 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:33.015154 16501 tablet_replica.cc:333] T 3617af2dee3f48cf85973b1c94cdc33a P d13c58ad8ac441c4a9b396eb04b1a476: stopping tablet replica
I20260709 12:27:33.016031 16501 raft_consensus.cc:2243] T 3617af2dee3f48cf85973b1c94cdc33a P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:27:33.017275 16501 raft_consensus.cc:2272] T 3617af2dee3f48cf85973b1c94cdc33a P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:33.020187 16501 tablet_replica.cc:333] T d21ab1641501475b829863275dc8182a P d13c58ad8ac441c4a9b396eb04b1a476: stopping tablet replica
I20260709 12:27:33.020823 16501 raft_consensus.cc:2243] T d21ab1641501475b829863275dc8182a P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:33.021416 16501 raft_consensus.cc:2272] T d21ab1641501475b829863275dc8182a P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:33.024499 16501 tablet_replica.cc:333] T 43c7bfb33af74d4f912e5153e8a54b3e P d13c58ad8ac441c4a9b396eb04b1a476: stopping tablet replica
I20260709 12:27:33.025403 16501 raft_consensus.cc:2243] T 43c7bfb33af74d4f912e5153e8a54b3e P d13c58ad8ac441c4a9b396eb04b1a476 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:33.026038 16501 raft_consensus.cc:2272] T 43c7bfb33af74d4f912e5153e8a54b3e P d13c58ad8ac441c4a9b396eb04b1a476 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:33.029677 16501 tablet_replica.cc:333] T d1918297b9eb41f5847a211e668ae748 P d13c58ad8ac441c4a9b396eb04b1a476: stopping tablet replica
I20260709 12:27:33.030519 16501 raft_consensus.cc:2243] T d1918297b9eb41f5847a211e668ae748 P d13c58ad8ac441c4a9b396eb04b1a476 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:33.031167 16501 raft_consensus.cc:2272] T d1918297b9eb41f5847a211e668ae748 P d13c58ad8ac441c4a9b396eb04b1a476 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:33.034080 16501 tablet_replica.cc:333] T 0c428ff656ae4b9d99d5b5073f98725d P d13c58ad8ac441c4a9b396eb04b1a476: stopping tablet replica
I20260709 12:27:33.034883 16501 raft_consensus.cc:2243] T 0c428ff656ae4b9d99d5b5073f98725d P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 LEADER]: Raft consensus shutting down.
W20260709 12:27:33.035169 29709 consensus_peers.cc:597] T 8627b58b4f42491597125eb45f8d3b61 P c6610c00ffb948ce9c34f41dec3d49e8 -> Peer d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209): Couldn't send request to peer d13c58ad8ac441c4a9b396eb04b1a476. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:33.035995 16501 raft_consensus.cc:2272] T 0c428ff656ae4b9d99d5b5073f98725d P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:33.039471 16501 tablet_replica.cc:333] T 742beffede4f407d8a72909d2b147e6d P d13c58ad8ac441c4a9b396eb04b1a476: stopping tablet replica
I20260709 12:27:33.040330 16501 raft_consensus.cc:2243] T 742beffede4f407d8a72909d2b147e6d P d13c58ad8ac441c4a9b396eb04b1a476 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:33.040562 30729 raft_consensus.cc:493] T 592801934cad4789b9ce8dc09f1470bd P c6610c00ffb948ce9c34f41dec3d49e8 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 584dac86183a41d9b3bea1c2085e62fa)
I20260709 12:27:33.041049 16501 raft_consensus.cc:2272] T 742beffede4f407d8a72909d2b147e6d P d13c58ad8ac441c4a9b396eb04b1a476 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:33.041339 30729 raft_consensus.cc:515] T 592801934cad4789b9ce8dc09f1470bd P c6610c00ffb948ce9c34f41dec3d49e8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:33.045109 16501 tablet_replica.cc:333] T 00c6cf271c91400aa834467def9af9b1 P d13c58ad8ac441c4a9b396eb04b1a476: stopping tablet replica
I20260709 12:27:33.046065 16501 raft_consensus.cc:2243] T 00c6cf271c91400aa834467def9af9b1 P d13c58ad8ac441c4a9b396eb04b1a476 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:27:33.046275 30729 leader_election.cc:290] T 592801934cad4789b9ce8dc09f1470bd P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033), d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825)
I20260709 12:27:33.047521 16501 raft_consensus.cc:2272] T 00c6cf271c91400aa834467def9af9b1 P d13c58ad8ac441c4a9b396eb04b1a476 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:33.051137 16501 tablet_replica.cc:333] T ef20d7124e7a44aa97a219ee194f4b8b P d13c58ad8ac441c4a9b396eb04b1a476: stopping tablet replica
W20260709 12:27:33.051016 29709 consensus_peers.cc:597] T d299fb38094e47e8aa3890ca74f6b366 P c6610c00ffb948ce9c34f41dec3d49e8 -> Peer d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209): Couldn't send request to peer d13c58ad8ac441c4a9b396eb04b1a476. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:33.052148 16501 raft_consensus.cc:2243] T ef20d7124e7a44aa97a219ee194f4b8b P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:33.052872 16501 raft_consensus.cc:2272] T ef20d7124e7a44aa97a219ee194f4b8b P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 12:27:33.055037 29708 leader_election.cc:336] T 592801934cad4789b9ce8dc09f1470bd P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Network error: Client connection negotiation failed: client connection to 127.16.29.66:36825: connect: Connection refused (error 111)
W20260709 12:27:33.055922 29708 leader_election.cc:336] T 592801934cad4789b9ce8dc09f1470bd P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111)
I20260709 12:27:33.056084 16501 tablet_replica.cc:333] T e67a2affef634fd8bc96a64fa83cffdb P d13c58ad8ac441c4a9b396eb04b1a476: stopping tablet replica
I20260709 12:27:33.056363 29708 leader_election.cc:304] T 592801934cad4789b9ce8dc09f1470bd P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c6610c00ffb948ce9c34f41dec3d49e8; no voters: 584dac86183a41d9b3bea1c2085e62fa, d73d92fb35234d16824c55689b5fce28
I20260709 12:27:33.056962 16501 raft_consensus.cc:2243] T e67a2affef634fd8bc96a64fa83cffdb P d13c58ad8ac441c4a9b396eb04b1a476 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:27:33.057343 30729 raft_consensus.cc:2749] T 592801934cad4789b9ce8dc09f1470bd P c6610c00ffb948ce9c34f41dec3d49e8 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:27:33.058192 16501 raft_consensus.cc:2272] T e67a2affef634fd8bc96a64fa83cffdb P d13c58ad8ac441c4a9b396eb04b1a476 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:33.060585 16501 tablet_replica.cc:333] T 55061e62184b45f0bb1726adc1a4af81 P d13c58ad8ac441c4a9b396eb04b1a476: stopping tablet replica
I20260709 12:27:33.061139 16501 raft_consensus.cc:2243] T 55061e62184b45f0bb1726adc1a4af81 P d13c58ad8ac441c4a9b396eb04b1a476 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:33.061638 16501 raft_consensus.cc:2272] T 55061e62184b45f0bb1726adc1a4af81 P d13c58ad8ac441c4a9b396eb04b1a476 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:33.063768 16501 tablet_replica.cc:333] T cdf706ffabc34902ad3d1bf2cee7889e P d13c58ad8ac441c4a9b396eb04b1a476: stopping tablet replica
I20260709 12:27:33.064347 16501 raft_consensus.cc:2243] T cdf706ffabc34902ad3d1bf2cee7889e P d13c58ad8ac441c4a9b396eb04b1a476 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:27:33.065263 16501 raft_consensus.cc:2272] T cdf706ffabc34902ad3d1bf2cee7889e P d13c58ad8ac441c4a9b396eb04b1a476 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:33.067451 16501 tablet_replica.cc:333] T 58154d49c1a344f2b51366ede3c87de5 P d13c58ad8ac441c4a9b396eb04b1a476: stopping tablet replica
I20260709 12:27:33.068192 16501 raft_consensus.cc:2243] T 58154d49c1a344f2b51366ede3c87de5 P d13c58ad8ac441c4a9b396eb04b1a476 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:33.068884 16501 raft_consensus.cc:2272] T 58154d49c1a344f2b51366ede3c87de5 P d13c58ad8ac441c4a9b396eb04b1a476 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:33.072671 16501 tablet_replica.cc:333] T fcb6a7f720b14dc98e821f663357a0bb P d13c58ad8ac441c4a9b396eb04b1a476: stopping tablet replica
I20260709 12:27:33.073566 16501 raft_consensus.cc:2243] T fcb6a7f720b14dc98e821f663357a0bb P d13c58ad8ac441c4a9b396eb04b1a476 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:33.074292 16501 raft_consensus.cc:2272] T fcb6a7f720b14dc98e821f663357a0bb P d13c58ad8ac441c4a9b396eb04b1a476 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:33.076694 16501 tablet_replica.cc:333] T 5ee924f859f54f64908499bfac48ee62 P d13c58ad8ac441c4a9b396eb04b1a476: stopping tablet replica
I20260709 12:27:33.077236 16501 raft_consensus.cc:2243] T 5ee924f859f54f64908499bfac48ee62 P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:27:33.078086 16501 raft_consensus.cc:2272] T 5ee924f859f54f64908499bfac48ee62 P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:33.080431 16501 tablet_replica.cc:333] T 7ae3b542012d4f35a9c363eee22976ec P d13c58ad8ac441c4a9b396eb04b1a476: stopping tablet replica
I20260709 12:27:33.080947 16501 raft_consensus.cc:2243] T 7ae3b542012d4f35a9c363eee22976ec P d13c58ad8ac441c4a9b396eb04b1a476 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:27:33.081804 16501 raft_consensus.cc:2272] T 7ae3b542012d4f35a9c363eee22976ec P d13c58ad8ac441c4a9b396eb04b1a476 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:33.084159 16501 tablet_replica.cc:333] T d5fcd1f030614bbe82cf1dc261dfc1c3 P d13c58ad8ac441c4a9b396eb04b1a476: stopping tablet replica
I20260709 12:27:33.084648 16501 raft_consensus.cc:2243] T d5fcd1f030614bbe82cf1dc261dfc1c3 P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:33.085103 16501 raft_consensus.cc:2272] T d5fcd1f030614bbe82cf1dc261dfc1c3 P d13c58ad8ac441c4a9b396eb04b1a476 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:33.087723 16501 tablet_replica.cc:333] T d42259e436d241b29fb4ded16d9b6455 P d13c58ad8ac441c4a9b396eb04b1a476: stopping tablet replica
I20260709 12:27:33.088300 16501 raft_consensus.cc:2243] T d42259e436d241b29fb4ded16d9b6455 P d13c58ad8ac441c4a9b396eb04b1a476 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:27:33.089311 16501 raft_consensus.cc:2272] T d42259e436d241b29fb4ded16d9b6455 P d13c58ad8ac441c4a9b396eb04b1a476 [term 2 FOLLOWER]: Raft consensus is shut down!
W20260709 12:27:33.160620 29709 consensus_peers.cc:597] T ef20d7124e7a44aa97a219ee194f4b8b P c6610c00ffb948ce9c34f41dec3d49e8 -> Peer d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209): Couldn't send request to peer d13c58ad8ac441c4a9b396eb04b1a476. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:33.176231 30729 raft_consensus.cc:493] T 6c8b2fa9f2c044ec9eec97c72b561042 P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 584dac86183a41d9b3bea1c2085e62fa)
I20260709 12:27:33.177081 30729 raft_consensus.cc:515] T 6c8b2fa9f2c044ec9eec97c72b561042 P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:33.179373 30729 leader_election.cc:290] T 6c8b2fa9f2c044ec9eec97c72b561042 P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033), d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825)
W20260709 12:27:33.183267 29708 leader_election.cc:336] T 6c8b2fa9f2c044ec9eec97c72b561042 P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Network error: Client connection negotiation failed: client connection to 127.16.29.66:36825: connect: Connection refused (error 111)
W20260709 12:27:33.184000 29708 leader_election.cc:336] T 6c8b2fa9f2c044ec9eec97c72b561042 P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111)
I20260709 12:27:33.184309 29708 leader_election.cc:304] T 6c8b2fa9f2c044ec9eec97c72b561042 P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c6610c00ffb948ce9c34f41dec3d49e8; no voters: 584dac86183a41d9b3bea1c2085e62fa, d73d92fb35234d16824c55689b5fce28
I20260709 12:27:33.184878 30728 raft_consensus.cc:2749] T 6c8b2fa9f2c044ec9eec97c72b561042 P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260709 12:27:33.204136 29709 consensus_peers.cc:597] T 394378d078194257b9e04fc659c1f5d4 P c6610c00ffb948ce9c34f41dec3d49e8 -> Peer d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209): Couldn't send request to peer d13c58ad8ac441c4a9b396eb04b1a476. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:33.207333 29709 consensus_peers.cc:597] T 43c7bfb33af74d4f912e5153e8a54b3e P c6610c00ffb948ce9c34f41dec3d49e8 -> Peer d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209): Couldn't send request to peer d13c58ad8ac441c4a9b396eb04b1a476. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:33.248811 29635 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:27:33.260665 29635 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:27:33.324612 29709 consensus_peers.cc:597] T f76101eb30f14b98a9aa8c41e0d6f831 P c6610c00ffb948ce9c34f41dec3d49e8 -> Peer d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209): Couldn't send request to peer d13c58ad8ac441c4a9b396eb04b1a476. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:33.430255 29636 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:27:33.444448 29635 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:27:33.729403 29635 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:27:33.754658 29708 consensus_peers.cc:597] T f76101eb30f14b98a9aa8c41e0d6f831 P c6610c00ffb948ce9c34f41dec3d49e8 -> Peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Couldn't send request to peer 584dac86183a41d9b3bea1c2085e62fa. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:27:33.755957 29635 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:27:33.780087 29708 consensus_peers.cc:597] T 66f652ba2c224885a2aea974bfe8f88a P c6610c00ffb948ce9c34f41dec3d49e8 -> Peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Couldn't send request to peer 584dac86183a41d9b3bea1c2085e62fa. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:27:33.841074 29636 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:27:33.850071 29634 leader_election.cc:336] T 71e2de235adf46d9a2407659080d42ef P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Network error: Client connection negotiation failed: client connection to 127.16.29.66:36825: connect: Connection refused (error 111)
I20260709 12:27:33.850543 29634 leader_election.cc:304] T 71e2de235adf46d9a2407659080d42ef P d13c58ad8ac441c4a9b396eb04b1a476 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d13c58ad8ac441c4a9b396eb04b1a476; no voters: 584dac86183a41d9b3bea1c2085e62fa, d73d92fb35234d16824c55689b5fce28
W20260709 12:27:33.851159 29634 raft_consensus.cc:2686] T 71e2de235adf46d9a2407659080d42ef P d13c58ad8ac441c4a9b396eb04b1a476: unable to run election callback: Service unavailable: Thread pool token was shut down
I20260709 12:27:33.861528 30732 raft_consensus.cc:493] T 5be58d64d45e40d7981018b84a01b1ed P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d73d92fb35234d16824c55689b5fce28)
I20260709 12:27:33.862274 30732 raft_consensus.cc:515] T 5be58d64d45e40d7981018b84a01b1ed P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:33.863507 30728 raft_consensus.cc:493] T a3dcbb9c64bf4b5bb6680e8cb81af93e P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d73d92fb35234d16824c55689b5fce28)
I20260709 12:27:33.864120 30728 raft_consensus.cc:515] T a3dcbb9c64bf4b5bb6680e8cb81af93e P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:33.864538 30732 leader_election.cc:290] T 5be58d64d45e40d7981018b84a01b1ed P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033), d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825)
I20260709 12:27:33.866431 30728 leader_election.cc:290] T a3dcbb9c64bf4b5bb6680e8cb81af93e P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033), d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825)
W20260709 12:27:33.873518 29708 leader_election.cc:336] T 5be58d64d45e40d7981018b84a01b1ed P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111)
W20260709 12:27:33.874112 29708 leader_election.cc:336] T a3dcbb9c64bf4b5bb6680e8cb81af93e P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111)
W20260709 12:27:33.876026 29708 leader_election.cc:336] T 5be58d64d45e40d7981018b84a01b1ed P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Network error: Client connection negotiation failed: client connection to 127.16.29.66:36825: connect: Connection refused (error 111)
I20260709 12:27:33.876437 29708 leader_election.cc:304] T 5be58d64d45e40d7981018b84a01b1ed P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c6610c00ffb948ce9c34f41dec3d49e8; no voters: 584dac86183a41d9b3bea1c2085e62fa, d73d92fb35234d16824c55689b5fce28
I20260709 12:27:33.877209 30728 raft_consensus.cc:2749] T 5be58d64d45e40d7981018b84a01b1ed P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260709 12:27:33.877369 29708 leader_election.cc:336] T a3dcbb9c64bf4b5bb6680e8cb81af93e P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Network error: Client connection negotiation failed: client connection to 127.16.29.66:36825: connect: Connection refused (error 111)
I20260709 12:27:33.877882 29708 leader_election.cc:304] T a3dcbb9c64bf4b5bb6680e8cb81af93e P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c6610c00ffb948ce9c34f41dec3d49e8; no voters: 584dac86183a41d9b3bea1c2085e62fa, d73d92fb35234d16824c55689b5fce28
I20260709 12:27:33.878538 30728 raft_consensus.cc:2749] T a3dcbb9c64bf4b5bb6680e8cb81af93e P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:27:33.931867 30728 raft_consensus.cc:493] T 3f41ce441fb047c8bbb741b1b9cbc0ec P c6610c00ffb948ce9c34f41dec3d49e8 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d13c58ad8ac441c4a9b396eb04b1a476)
I20260709 12:27:33.932490 30728 raft_consensus.cc:515] T 3f41ce441fb047c8bbb741b1b9cbc0ec P c6610c00ffb948ce9c34f41dec3d49e8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:27:33.934228 30728 leader_election.cc:290] T 3f41ce441fb047c8bbb741b1b9cbc0ec P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209), 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033)
W20260709 12:27:33.938722 29709 leader_election.cc:336] T 3f41ce441fb047c8bbb741b1b9cbc0ec P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209): Network error: Client connection negotiation failed: client connection to 127.16.29.67:38209: connect: Connection refused (error 111)
W20260709 12:27:33.940131 29708 leader_election.cc:336] T 3f41ce441fb047c8bbb741b1b9cbc0ec P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111)
I20260709 12:27:33.940575 29708 leader_election.cc:304] T 3f41ce441fb047c8bbb741b1b9cbc0ec P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c6610c00ffb948ce9c34f41dec3d49e8; no voters: 584dac86183a41d9b3bea1c2085e62fa, d13c58ad8ac441c4a9b396eb04b1a476
I20260709 12:27:33.941274 30728 raft_consensus.cc:2749] T 3f41ce441fb047c8bbb741b1b9cbc0ec P c6610c00ffb948ce9c34f41dec3d49e8 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:27:34.010433 30732 raft_consensus.cc:493] T 3617af2dee3f48cf85973b1c94cdc33a P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d13c58ad8ac441c4a9b396eb04b1a476)
I20260709 12:27:34.010928 30732 raft_consensus.cc:515] T 3617af2dee3f48cf85973b1c94cdc33a P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:27:34.013033 30732 leader_election.cc:290] T 3617af2dee3f48cf85973b1c94cdc33a P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825), d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209)
W20260709 12:27:34.017937 29709 leader_election.cc:336] T 3617af2dee3f48cf85973b1c94cdc33a P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209): Network error: Client connection negotiation failed: client connection to 127.16.29.67:38209: connect: Connection refused (error 111)
W20260709 12:27:34.018524 29708 leader_election.cc:336] T 3617af2dee3f48cf85973b1c94cdc33a P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Network error: Client connection negotiation failed: client connection to 127.16.29.66:36825: connect: Connection refused (error 111)
I20260709 12:27:34.018992 29708 leader_election.cc:304] T 3617af2dee3f48cf85973b1c94cdc33a P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c6610c00ffb948ce9c34f41dec3d49e8; no voters: d13c58ad8ac441c4a9b396eb04b1a476, d73d92fb35234d16824c55689b5fce28
I20260709 12:27:34.020176 30732 raft_consensus.cc:2749] T 3617af2dee3f48cf85973b1c94cdc33a P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:27:34.206373 30732 raft_consensus.cc:493] T 3e9451c3797c4496a66dfdb3e9fb5e38 P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d13c58ad8ac441c4a9b396eb04b1a476)
I20260709 12:27:34.206899 30732 raft_consensus.cc:515] T 3e9451c3797c4496a66dfdb3e9fb5e38 P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:27:34.208796 30732 leader_election.cc:290] T 3e9451c3797c4496a66dfdb3e9fb5e38 P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209), 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033)
W20260709 12:27:34.214188 29708 leader_election.cc:336] T 3e9451c3797c4496a66dfdb3e9fb5e38 P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111)
W20260709 12:27:34.215190 29709 leader_election.cc:336] T 3e9451c3797c4496a66dfdb3e9fb5e38 P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209): Network error: Client connection negotiation failed: client connection to 127.16.29.67:38209: connect: Connection refused (error 111)
I20260709 12:27:34.215823 29709 leader_election.cc:304] T 3e9451c3797c4496a66dfdb3e9fb5e38 P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c6610c00ffb948ce9c34f41dec3d49e8; no voters: 584dac86183a41d9b3bea1c2085e62fa, d13c58ad8ac441c4a9b396eb04b1a476
W20260709 12:27:34.216913 29708 consensus_peers.cc:597] T 742beffede4f407d8a72909d2b147e6d P c6610c00ffb948ce9c34f41dec3d49e8 -> Peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Couldn't send request to peer 584dac86183a41d9b3bea1c2085e62fa. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 12:27:34.217799 30732 raft_consensus.cc:2749] T 3e9451c3797c4496a66dfdb3e9fb5e38 P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:27:34.233371 30732 raft_consensus.cc:493] T 9633e3b1d78f403097e4a00b7d4b869b P c6610c00ffb948ce9c34f41dec3d49e8 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d13c58ad8ac441c4a9b396eb04b1a476)
I20260709 12:27:34.234233 30732 raft_consensus.cc:515] T 9633e3b1d78f403097e4a00b7d4b869b P c6610c00ffb948ce9c34f41dec3d49e8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:27:34.235080 30728 raft_consensus.cc:493] T 31a190e9f5ff407fab90502fb3197b70 P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d13c58ad8ac441c4a9b396eb04b1a476)
I20260709 12:27:34.235778 30728 raft_consensus.cc:515] T 31a190e9f5ff407fab90502fb3197b70 P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:27:34.236469 30732 leader_election.cc:290] T 9633e3b1d78f403097e4a00b7d4b869b P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209), 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033)
I20260709 12:27:34.238830 30728 leader_election.cc:290] T 31a190e9f5ff407fab90502fb3197b70 P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209), 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033)
W20260709 12:27:34.240703 29709 leader_election.cc:336] T 9633e3b1d78f403097e4a00b7d4b869b P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209): Network error: Client connection negotiation failed: client connection to 127.16.29.67:38209: connect: Connection refused (error 111)
W20260709 12:27:34.241869 29708 leader_election.cc:336] T 9633e3b1d78f403097e4a00b7d4b869b P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111)
I20260709 12:27:34.242331 29708 leader_election.cc:304] T 9633e3b1d78f403097e4a00b7d4b869b P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c6610c00ffb948ce9c34f41dec3d49e8; no voters: 584dac86183a41d9b3bea1c2085e62fa, d13c58ad8ac441c4a9b396eb04b1a476
I20260709 12:27:34.243093 30728 raft_consensus.cc:2749] T 9633e3b1d78f403097e4a00b7d4b869b P c6610c00ffb948ce9c34f41dec3d49e8 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260709 12:27:34.243600 29709 leader_election.cc:336] T 31a190e9f5ff407fab90502fb3197b70 P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209): Network error: Client connection negotiation failed: client connection to 127.16.29.67:38209: connect: Connection refused (error 111)
W20260709 12:27:34.245779 29708 leader_election.cc:336] T 31a190e9f5ff407fab90502fb3197b70 P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111)
I20260709 12:27:34.246162 29708 leader_election.cc:304] T 31a190e9f5ff407fab90502fb3197b70 P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c6610c00ffb948ce9c34f41dec3d49e8; no voters: 584dac86183a41d9b3bea1c2085e62fa, d13c58ad8ac441c4a9b396eb04b1a476
I20260709 12:27:34.246706 30728 raft_consensus.cc:2749] T 31a190e9f5ff407fab90502fb3197b70 P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:27:34.254848 30728 raft_consensus.cc:493] T 0c428ff656ae4b9d99d5b5073f98725d P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d13c58ad8ac441c4a9b396eb04b1a476)
I20260709 12:27:34.255358 30728 raft_consensus.cc:515] T 0c428ff656ae4b9d99d5b5073f98725d P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:27:34.257112 30728 leader_election.cc:290] T 0c428ff656ae4b9d99d5b5073f98725d P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209), 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033)
W20260709 12:27:34.260885 29708 leader_election.cc:336] T 0c428ff656ae4b9d99d5b5073f98725d P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111)
W20260709 12:27:34.261523 29709 leader_election.cc:336] T 0c428ff656ae4b9d99d5b5073f98725d P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209): Network error: Client connection negotiation failed: client connection to 127.16.29.67:38209: connect: Connection refused (error 111)
I20260709 12:27:34.261997 29709 leader_election.cc:304] T 0c428ff656ae4b9d99d5b5073f98725d P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c6610c00ffb948ce9c34f41dec3d49e8; no voters: 584dac86183a41d9b3bea1c2085e62fa, d13c58ad8ac441c4a9b396eb04b1a476
I20260709 12:27:34.262856 30728 raft_consensus.cc:2749] T 0c428ff656ae4b9d99d5b5073f98725d P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260709 12:27:34.298182 29708 consensus_peers.cc:597] T 8627b58b4f42491597125eb45f8d3b61 P c6610c00ffb948ce9c34f41dec3d49e8 -> Peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Couldn't send request to peer 584dac86183a41d9b3bea1c2085e62fa. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 12:27:34.303790 30728 raft_consensus.cc:493] T cdf706ffabc34902ad3d1bf2cee7889e P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d13c58ad8ac441c4a9b396eb04b1a476)
I20260709 12:27:34.304435 30728 raft_consensus.cc:515] T cdf706ffabc34902ad3d1bf2cee7889e P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:27:34.306273 30728 leader_election.cc:290] T cdf706ffabc34902ad3d1bf2cee7889e P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825), d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209)
W20260709 12:27:34.309913 29708 leader_election.cc:336] T cdf706ffabc34902ad3d1bf2cee7889e P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Network error: Client connection negotiation failed: client connection to 127.16.29.66:36825: connect: Connection refused (error 111)
W20260709 12:27:34.310469 29709 leader_election.cc:336] T cdf706ffabc34902ad3d1bf2cee7889e P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209): Network error: Client connection negotiation failed: client connection to 127.16.29.67:38209: connect: Connection refused (error 111)
I20260709 12:27:34.310829 29709 leader_election.cc:304] T cdf706ffabc34902ad3d1bf2cee7889e P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c6610c00ffb948ce9c34f41dec3d49e8; no voters: d13c58ad8ac441c4a9b396eb04b1a476, d73d92fb35234d16824c55689b5fce28
I20260709 12:27:34.311422 30728 raft_consensus.cc:2749] T cdf706ffabc34902ad3d1bf2cee7889e P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:27:34.312278 30728 raft_consensus.cc:493] T fb7d78e534634e0f992fc39b578ebd8d P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d13c58ad8ac441c4a9b396eb04b1a476)
I20260709 12:27:34.312676 30728 raft_consensus.cc:515] T fb7d78e534634e0f992fc39b578ebd8d P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:27:34.314419 30728 leader_election.cc:290] T fb7d78e534634e0f992fc39b578ebd8d P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209), 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033)
W20260709 12:27:34.318158 29709 leader_election.cc:336] T fb7d78e534634e0f992fc39b578ebd8d P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209): Network error: Client connection negotiation failed: client connection to 127.16.29.67:38209: connect: Connection refused (error 111)
W20260709 12:27:34.318923 29708 leader_election.cc:336] T fb7d78e534634e0f992fc39b578ebd8d P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111)
I20260709 12:27:34.319329 29708 leader_election.cc:304] T fb7d78e534634e0f992fc39b578ebd8d P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c6610c00ffb948ce9c34f41dec3d49e8; no voters: 584dac86183a41d9b3bea1c2085e62fa, d13c58ad8ac441c4a9b396eb04b1a476
I20260709 12:27:34.320055 30732 raft_consensus.cc:2749] T fb7d78e534634e0f992fc39b578ebd8d P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:27:34.326614 30732 raft_consensus.cc:493] T 00c6cf271c91400aa834467def9af9b1 P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d13c58ad8ac441c4a9b396eb04b1a476)
I20260709 12:27:34.327204 30732 raft_consensus.cc:515] T 00c6cf271c91400aa834467def9af9b1 P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:27:34.329716 30732 leader_election.cc:290] T 00c6cf271c91400aa834467def9af9b1 P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825), d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209)
W20260709 12:27:34.333636 29708 leader_election.cc:336] T 00c6cf271c91400aa834467def9af9b1 P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Network error: Client connection negotiation failed: client connection to 127.16.29.66:36825: connect: Connection refused (error 111)
W20260709 12:27:34.335067 29709 leader_election.cc:336] T 00c6cf271c91400aa834467def9af9b1 P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209): Network error: Client connection negotiation failed: client connection to 127.16.29.67:38209: connect: Connection refused (error 111)
I20260709 12:27:34.335431 29709 leader_election.cc:304] T 00c6cf271c91400aa834467def9af9b1 P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c6610c00ffb948ce9c34f41dec3d49e8; no voters: d13c58ad8ac441c4a9b396eb04b1a476, d73d92fb35234d16824c55689b5fce28
I20260709 12:27:34.336102 30732 raft_consensus.cc:2749] T 00c6cf271c91400aa834467def9af9b1 P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:27:34.337750 30732 raft_consensus.cc:493] T 6e37d72807e547898f4fce9e8b82b2b0 P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d13c58ad8ac441c4a9b396eb04b1a476)
I20260709 12:27:34.338356 30732 raft_consensus.cc:515] T 6e37d72807e547898f4fce9e8b82b2b0 P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:27:34.340904 30732 leader_election.cc:290] T 6e37d72807e547898f4fce9e8b82b2b0 P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209), 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033)
W20260709 12:27:34.343953 29709 leader_election.cc:336] T 6e37d72807e547898f4fce9e8b82b2b0 P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209): Network error: Client connection negotiation failed: client connection to 127.16.29.67:38209: connect: Connection refused (error 111)
W20260709 12:27:34.345340 29708 leader_election.cc:336] T 6e37d72807e547898f4fce9e8b82b2b0 P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111)
I20260709 12:27:34.345815 29708 leader_election.cc:304] T 6e37d72807e547898f4fce9e8b82b2b0 P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c6610c00ffb948ce9c34f41dec3d49e8; no voters: 584dac86183a41d9b3bea1c2085e62fa, d13c58ad8ac441c4a9b396eb04b1a476
I20260709 12:27:34.346590 30732 raft_consensus.cc:2749] T 6e37d72807e547898f4fce9e8b82b2b0 P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:27:34.374068 30732 raft_consensus.cc:493] T 07a2b94c0823467695508292a9ccdcc5 P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d13c58ad8ac441c4a9b396eb04b1a476)
I20260709 12:27:34.374752 30732 raft_consensus.cc:515] T 07a2b94c0823467695508292a9ccdcc5 P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:27:34.377466 30732 leader_election.cc:290] T 07a2b94c0823467695508292a9ccdcc5 P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825), d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209)
W20260709 12:27:34.381624 29708 leader_election.cc:336] T 07a2b94c0823467695508292a9ccdcc5 P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Network error: Client connection negotiation failed: client connection to 127.16.29.66:36825: connect: Connection refused (error 111)
W20260709 12:27:34.382198 29709 leader_election.cc:336] T 07a2b94c0823467695508292a9ccdcc5 P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209): Network error: Client connection negotiation failed: client connection to 127.16.29.67:38209: connect: Connection refused (error 111)
I20260709 12:27:34.382647 29709 leader_election.cc:304] T 07a2b94c0823467695508292a9ccdcc5 P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c6610c00ffb948ce9c34f41dec3d49e8; no voters: d13c58ad8ac441c4a9b396eb04b1a476, d73d92fb35234d16824c55689b5fce28
I20260709 12:27:34.383464 30732 raft_consensus.cc:2749] T 07a2b94c0823467695508292a9ccdcc5 P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260709 12:27:34.397047 29708 consensus_peers.cc:597] T 425777eed34441cd93e0902bdc126516 P c6610c00ffb948ce9c34f41dec3d49e8 -> Peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Couldn't send request to peer 584dac86183a41d9b3bea1c2085e62fa. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:27:34.416368 29708 consensus_peers.cc:597] T a46289e8fc824dd5a818e30df6e7ec3e P c6610c00ffb948ce9c34f41dec3d49e8 -> Peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Couldn't send request to peer 584dac86183a41d9b3bea1c2085e62fa. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:27:34.439795 29708 consensus_peers.cc:597] T 8017a44219404faa8db8ff049c65205e P c6610c00ffb948ce9c34f41dec3d49e8 -> Peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Couldn't send request to peer 584dac86183a41d9b3bea1c2085e62fa. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 12:27:34.472012 30735 raft_consensus.cc:493] T 76140ad6258543118a9bfeee7cf0d5fc P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d13c58ad8ac441c4a9b396eb04b1a476)
I20260709 12:27:34.472579 30735 raft_consensus.cc:515] T 76140ad6258543118a9bfeee7cf0d5fc P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } } peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } }
I20260709 12:27:34.474642 30735 leader_election.cc:290] T 76140ad6258543118a9bfeee7cf0d5fc P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825), d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209)
W20260709 12:27:34.476907 29708 consensus_peers.cc:597] T ef20d7124e7a44aa97a219ee194f4b8b P c6610c00ffb948ce9c34f41dec3d49e8 -> Peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Couldn't send request to peer 584dac86183a41d9b3bea1c2085e62fa. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:27:34.478528 29709 leader_election.cc:336] T 76140ad6258543118a9bfeee7cf0d5fc P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209): Network error: Client connection negotiation failed: client connection to 127.16.29.67:38209: connect: Connection refused (error 111)
W20260709 12:27:34.479144 29708 leader_election.cc:336] T 76140ad6258543118a9bfeee7cf0d5fc P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Network error: Client connection negotiation failed: client connection to 127.16.29.66:36825: connect: Connection refused (error 111)
I20260709 12:27:34.479476 29708 leader_election.cc:304] T 76140ad6258543118a9bfeee7cf0d5fc P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c6610c00ffb948ce9c34f41dec3d49e8; no voters: d13c58ad8ac441c4a9b396eb04b1a476, d73d92fb35234d16824c55689b5fce28
I20260709 12:27:34.480089 30735 raft_consensus.cc:2749] T 76140ad6258543118a9bfeee7cf0d5fc P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:27:34.530184 30735 raft_consensus.cc:493] T e67a2affef634fd8bc96a64fa83cffdb P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d13c58ad8ac441c4a9b396eb04b1a476)
I20260709 12:27:34.530826 30735 raft_consensus.cc:515] T e67a2affef634fd8bc96a64fa83cffdb P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13c58ad8ac441c4a9b396eb04b1a476" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 38209 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } }
I20260709 12:27:34.532908 30735 leader_election.cc:290] T e67a2affef634fd8bc96a64fa83cffdb P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209), 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033)
W20260709 12:27:34.539322 29708 leader_election.cc:336] T e67a2affef634fd8bc96a64fa83cffdb P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111)
W20260709 12:27:34.540649 29709 leader_election.cc:336] T e67a2affef634fd8bc96a64fa83cffdb P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209): Network error: Client connection negotiation failed: client connection to 127.16.29.67:38209: connect: Connection refused (error 111)
I20260709 12:27:34.541065 29709 leader_election.cc:304] T e67a2affef634fd8bc96a64fa83cffdb P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c6610c00ffb948ce9c34f41dec3d49e8; no voters: 584dac86183a41d9b3bea1c2085e62fa, d13c58ad8ac441c4a9b396eb04b1a476
W20260709 12:27:34.541226 29708 consensus_peers.cc:597] T d299fb38094e47e8aa3890ca74f6b366 P c6610c00ffb948ce9c34f41dec3d49e8 -> Peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Couldn't send request to peer d73d92fb35234d16824c55689b5fce28. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.66:36825: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 12:27:34.541755 30735 raft_consensus.cc:2749] T e67a2affef634fd8bc96a64fa83cffdb P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:27:34.733229 30735 raft_consensus.cc:493] T 62f8c9472b214576b9002e02536a3ace P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d73d92fb35234d16824c55689b5fce28)
I20260709 12:27:34.733692 30735 raft_consensus.cc:515] T 62f8c9472b214576b9002e02536a3ace P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:34.735242 30735 leader_election.cc:290] T 62f8c9472b214576b9002e02536a3ace P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033), d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825)
W20260709 12:27:34.740317 29708 leader_election.cc:336] T 62f8c9472b214576b9002e02536a3ace P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111)
W20260709 12:27:34.741089 29708 leader_election.cc:336] T 62f8c9472b214576b9002e02536a3ace P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Network error: Client connection negotiation failed: client connection to 127.16.29.66:36825: connect: Connection refused (error 111)
I20260709 12:27:34.741546 29708 leader_election.cc:304] T 62f8c9472b214576b9002e02536a3ace P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c6610c00ffb948ce9c34f41dec3d49e8; no voters: 584dac86183a41d9b3bea1c2085e62fa, d73d92fb35234d16824c55689b5fce28
I20260709 12:27:34.742832 30735 raft_consensus.cc:2749] T 62f8c9472b214576b9002e02536a3ace P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260709 12:27:34.795934 29708 consensus_peers.cc:597] T f4516786e5564e9eac08ca472f79a9f0 P c6610c00ffb948ce9c34f41dec3d49e8 -> Peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Couldn't send request to peer 584dac86183a41d9b3bea1c2085e62fa. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 12:27:34.812778 30735 raft_consensus.cc:493] T f8c01e05981747a395e5483b551297dd P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d73d92fb35234d16824c55689b5fce28)
I20260709 12:27:34.813452 30735 raft_consensus.cc:515] T f8c01e05981747a395e5483b551297dd P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:34.815351 30735 leader_election.cc:290] T f8c01e05981747a395e5483b551297dd P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033), d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825)
W20260709 12:27:34.820282 29708 leader_election.cc:336] T f8c01e05981747a395e5483b551297dd P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111)
W20260709 12:27:34.821450 29708 leader_election.cc:336] T f8c01e05981747a395e5483b551297dd P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Network error: Client connection negotiation failed: client connection to 127.16.29.66:36825: connect: Connection refused (error 111)
I20260709 12:27:34.821792 29708 leader_election.cc:304] T f8c01e05981747a395e5483b551297dd P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c6610c00ffb948ce9c34f41dec3d49e8; no voters: 584dac86183a41d9b3bea1c2085e62fa, d73d92fb35234d16824c55689b5fce28
I20260709 12:27:34.822369 30735 raft_consensus.cc:2749] T f8c01e05981747a395e5483b551297dd P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260709 12:27:34.869396 16501 thread.cc:527] Waited for 1000ms trying to join with diag-logger (tid 29692)
W20260709 12:27:34.874563 29708 consensus_peers.cc:597] T 1d4e0ad0462e4f74a44155aa3ffaf87f P c6610c00ffb948ce9c34f41dec3d49e8 -> Peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Couldn't send request to peer 584dac86183a41d9b3bea1c2085e62fa. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:27:34.901410 29708 consensus_peers.cc:597] T 137e03144e754f6e9da4f0544513d27c P c6610c00ffb948ce9c34f41dec3d49e8 -> Peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Couldn't send request to peer d73d92fb35234d16824c55689b5fce28. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.66:36825: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:27:34.987666 29708 consensus_peers.cc:597] T c1c8411289a74b5685700229d245ff75 P c6610c00ffb948ce9c34f41dec3d49e8 -> Peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Couldn't send request to peer d73d92fb35234d16824c55689b5fce28. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.66:36825: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 12:27:35.059453 30735 raft_consensus.cc:493] T 592801934cad4789b9ce8dc09f1470bd P c6610c00ffb948ce9c34f41dec3d49e8 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 584dac86183a41d9b3bea1c2085e62fa)
I20260709 12:27:35.060097 30735 raft_consensus.cc:515] T 592801934cad4789b9ce8dc09f1470bd P c6610c00ffb948ce9c34f41dec3d49e8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584dac86183a41d9b3bea1c2085e62fa" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 35033 } } peers { permanent_uuid: "c6610c00ffb948ce9c34f41dec3d49e8" member_type: VOTER last_known_addr { host: "127.16.29.68" port: 43895 } } peers { permanent_uuid: "d73d92fb35234d16824c55689b5fce28" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 36825 } }
I20260709 12:27:35.062610 30735 leader_election.cc:290] T 592801934cad4789b9ce8dc09f1470bd P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033), d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825)
W20260709 12:27:35.066694 29708 leader_election.cc:336] T 592801934cad4789b9ce8dc09f1470bd P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 584dac86183a41d9b3bea1c2085e62fa (127.16.29.65:35033): Network error: Client connection negotiation failed: client connection to 127.16.29.65:35033: connect: Connection refused (error 111)
W20260709 12:27:35.067369 29708 leader_election.cc:336] T 592801934cad4789b9ce8dc09f1470bd P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Network error: Client connection negotiation failed: client connection to 127.16.29.66:36825: connect: Connection refused (error 111)
I20260709 12:27:35.067734 29708 leader_election.cc:304] T 592801934cad4789b9ce8dc09f1470bd P c6610c00ffb948ce9c34f41dec3d49e8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c6610c00ffb948ce9c34f41dec3d49e8; no voters: 584dac86183a41d9b3bea1c2085e62fa, d73d92fb35234d16824c55689b5fce28
I20260709 12:27:35.068434 30735 raft_consensus.cc:2749] T 592801934cad4789b9ce8dc09f1470bd P c6610c00ffb948ce9c34f41dec3d49e8 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260709 12:27:35.079556 29708 consensus_peers.cc:597] T 394378d078194257b9e04fc659c1f5d4 P c6610c00ffb948ce9c34f41dec3d49e8 -> Peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Couldn't send request to peer d73d92fb35234d16824c55689b5fce28. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.66:36825: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:27:35.089596 29708 consensus_peers.cc:597] T 55061e62184b45f0bb1726adc1a4af81 P c6610c00ffb948ce9c34f41dec3d49e8 -> Peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Couldn't send request to peer d73d92fb35234d16824c55689b5fce28. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.66:36825: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 12:27:35.112289 16501 tablet_server.cc:196] TabletServer@127.16.29.67:0 shutdown complete.
W20260709 12:27:35.135941 29709 consensus_peers.cc:597] T 55061e62184b45f0bb1726adc1a4af81 P c6610c00ffb948ce9c34f41dec3d49e8 -> Peer d13c58ad8ac441c4a9b396eb04b1a476 (127.16.29.67:38209): Couldn't send request to peer d13c58ad8ac441c4a9b396eb04b1a476. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.67:38209: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:27:35.175803 29708 consensus_peers.cc:597] T ed057e06dec148a0b9d7062beeb9d005 P c6610c00ffb948ce9c34f41dec3d49e8 -> Peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Couldn't send request to peer d73d92fb35234d16824c55689b5fce28. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.66:36825: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:27:35.180212 29708 consensus_peers.cc:597] T 1d4e0ad0462e4f74a44155aa3ffaf87f P c6610c00ffb948ce9c34f41dec3d49e8 -> Peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Couldn't send request to peer d73d92fb35234d16824c55689b5fce28. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.66:36825: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:27:35.182596 29708 consensus_peers.cc:597] T 8017a44219404faa8db8ff049c65205e P c6610c00ffb948ce9c34f41dec3d49e8 -> Peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Couldn't send request to peer d73d92fb35234d16824c55689b5fce28. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.66:36825: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 12:27:35.192332 16501 tablet_server.cc:179] TabletServer@127.16.29.68:0 shutting down...
W20260709 12:27:35.220371 29708 consensus_peers.cc:597] T 959940284eb24fdf83d78438d710318a P c6610c00ffb948ce9c34f41dec3d49e8 -> Peer d73d92fb35234d16824c55689b5fce28 (127.16.29.66:36825): Couldn't send request to peer d73d92fb35234d16824c55689b5fce28. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.66:36825: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 12:27:35.233644 16501 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:27:35.234676 16501 tablet_replica.cc:333] T 137e03144e754f6e9da4f0544513d27c P c6610c00ffb948ce9c34f41dec3d49e8: stopping tablet replica
I20260709 12:27:35.235831 16501 raft_consensus.cc:2243] T 137e03144e754f6e9da4f0544513d27c P c6610c00ffb948ce9c34f41dec3d49e8 [term 3 LEADER]: Raft consensus shutting down.
I20260709 12:27:35.236871 16501 raft_consensus.cc:2272] T 137e03144e754f6e9da4f0544513d27c P c6610c00ffb948ce9c34f41dec3d49e8 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:35.238832 16501 tablet_replica.cc:333] T a3dcbb9c64bf4b5bb6680e8cb81af93e P c6610c00ffb948ce9c34f41dec3d49e8: stopping tablet replica
I20260709 12:27:35.239378 16501 raft_consensus.cc:2243] T a3dcbb9c64bf4b5bb6680e8cb81af93e P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:35.239739 16501 raft_consensus.cc:2272] T a3dcbb9c64bf4b5bb6680e8cb81af93e P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:35.241729 16501 tablet_replica.cc:333] T 3617af2dee3f48cf85973b1c94cdc33a P c6610c00ffb948ce9c34f41dec3d49e8: stopping tablet replica
I20260709 12:27:35.242254 16501 raft_consensus.cc:2243] T 3617af2dee3f48cf85973b1c94cdc33a P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:35.242604 16501 raft_consensus.cc:2272] T 3617af2dee3f48cf85973b1c94cdc33a P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:35.244984 16501 tablet_replica.cc:333] T 01788883c8ea42989f246a3486111a7e P c6610c00ffb948ce9c34f41dec3d49e8: stopping tablet replica
I20260709 12:27:35.245648 16501 raft_consensus.cc:2243] T 01788883c8ea42989f246a3486111a7e P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:35.246063 16501 raft_consensus.cc:2272] T 01788883c8ea42989f246a3486111a7e P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:35.247973 16501 tablet_replica.cc:333] T 394378d078194257b9e04fc659c1f5d4 P c6610c00ffb948ce9c34f41dec3d49e8: stopping tablet replica
I20260709 12:27:35.248517 16501 raft_consensus.cc:2243] T 394378d078194257b9e04fc659c1f5d4 P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:27:35.249606 16501 raft_consensus.cc:2272] T 394378d078194257b9e04fc659c1f5d4 P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:35.252447 16501 tablet_replica.cc:333] T f76101eb30f14b98a9aa8c41e0d6f831 P c6610c00ffb948ce9c34f41dec3d49e8: stopping tablet replica
I20260709 12:27:35.253350 16501 raft_consensus.cc:2243] T f76101eb30f14b98a9aa8c41e0d6f831 P c6610c00ffb948ce9c34f41dec3d49e8 [term 3 LEADER]: Raft consensus shutting down.
I20260709 12:27:35.254518 16501 raft_consensus.cc:2272] T f76101eb30f14b98a9aa8c41e0d6f831 P c6610c00ffb948ce9c34f41dec3d49e8 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:35.257341 16501 tablet_replica.cc:333] T 1d4e0ad0462e4f74a44155aa3ffaf87f P c6610c00ffb948ce9c34f41dec3d49e8: stopping tablet replica
I20260709 12:27:35.258095 16501 raft_consensus.cc:2243] T 1d4e0ad0462e4f74a44155aa3ffaf87f P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:27:35.258906 16501 raft_consensus.cc:2272] T 1d4e0ad0462e4f74a44155aa3ffaf87f P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:35.260938 16501 tablet_replica.cc:333] T f8c01e05981747a395e5483b551297dd P c6610c00ffb948ce9c34f41dec3d49e8: stopping tablet replica
I20260709 12:27:35.261533 16501 raft_consensus.cc:2243] T f8c01e05981747a395e5483b551297dd P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:35.262051 16501 raft_consensus.cc:2272] T f8c01e05981747a395e5483b551297dd P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:35.264103 16501 tablet_replica.cc:333] T 07a2b94c0823467695508292a9ccdcc5 P c6610c00ffb948ce9c34f41dec3d49e8: stopping tablet replica
I20260709 12:27:35.264642 16501 raft_consensus.cc:2243] T 07a2b94c0823467695508292a9ccdcc5 P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:35.265066 16501 raft_consensus.cc:2272] T 07a2b94c0823467695508292a9ccdcc5 P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:35.267305 16501 tablet_replica.cc:333] T 6e37d72807e547898f4fce9e8b82b2b0 P c6610c00ffb948ce9c34f41dec3d49e8: stopping tablet replica
I20260709 12:27:35.268092 16501 raft_consensus.cc:2243] T 6e37d72807e547898f4fce9e8b82b2b0 P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:35.268463 16501 raft_consensus.cc:2272] T 6e37d72807e547898f4fce9e8b82b2b0 P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:35.270303 16501 tablet_replica.cc:333] T f4516786e5564e9eac08ca472f79a9f0 P c6610c00ffb948ce9c34f41dec3d49e8: stopping tablet replica
I20260709 12:27:35.270790 16501 raft_consensus.cc:2243] T f4516786e5564e9eac08ca472f79a9f0 P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:27:35.271495 16501 raft_consensus.cc:2272] T f4516786e5564e9eac08ca472f79a9f0 P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:35.273556 16501 tablet_replica.cc:333] T 959940284eb24fdf83d78438d710318a P c6610c00ffb948ce9c34f41dec3d49e8: stopping tablet replica
I20260709 12:27:35.274114 16501 raft_consensus.cc:2243] T 959940284eb24fdf83d78438d710318a P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:27:35.274827 16501 raft_consensus.cc:2272] T 959940284eb24fdf83d78438d710318a P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:35.277014 16501 tablet_replica.cc:333] T 8017a44219404faa8db8ff049c65205e P c6610c00ffb948ce9c34f41dec3d49e8: stopping tablet replica
I20260709 12:27:35.277518 16501 raft_consensus.cc:2243] T 8017a44219404faa8db8ff049c65205e P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:27:35.278283 16501 raft_consensus.cc:2272] T 8017a44219404faa8db8ff049c65205e P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:35.280337 16501 tablet_replica.cc:333] T 6c8b2fa9f2c044ec9eec97c72b561042 P c6610c00ffb948ce9c34f41dec3d49e8: stopping tablet replica
I20260709 12:27:35.280849 16501 raft_consensus.cc:2243] T 6c8b2fa9f2c044ec9eec97c72b561042 P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:35.281220 16501 raft_consensus.cc:2272] T 6c8b2fa9f2c044ec9eec97c72b561042 P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:35.283034 16501 tablet_replica.cc:333] T 66f652ba2c224885a2aea974bfe8f88a P c6610c00ffb948ce9c34f41dec3d49e8: stopping tablet replica
I20260709 12:27:35.283500 16501 raft_consensus.cc:2243] T 66f652ba2c224885a2aea974bfe8f88a P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:27:35.284291 16501 raft_consensus.cc:2272] T 66f652ba2c224885a2aea974bfe8f88a P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:35.286167 16501 tablet_replica.cc:333] T 76140ad6258543118a9bfeee7cf0d5fc P c6610c00ffb948ce9c34f41dec3d49e8: stopping tablet replica
I20260709 12:27:35.286638 16501 raft_consensus.cc:2243] T 76140ad6258543118a9bfeee7cf0d5fc P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:35.287053 16501 raft_consensus.cc:2272] T 76140ad6258543118a9bfeee7cf0d5fc P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:35.289244 16501 tablet_replica.cc:333] T 5be58d64d45e40d7981018b84a01b1ed P c6610c00ffb948ce9c34f41dec3d49e8: stopping tablet replica
I20260709 12:27:35.289794 16501 raft_consensus.cc:2243] T 5be58d64d45e40d7981018b84a01b1ed P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:35.290228 16501 raft_consensus.cc:2272] T 5be58d64d45e40d7981018b84a01b1ed P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:35.292366 16501 tablet_replica.cc:333] T b3c02c910d2b43cc9265bb1d98491cd3 P c6610c00ffb948ce9c34f41dec3d49e8: stopping tablet replica
I20260709 12:27:35.292909 16501 raft_consensus.cc:2243] T b3c02c910d2b43cc9265bb1d98491cd3 P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:35.293382 16501 raft_consensus.cc:2272] T b3c02c910d2b43cc9265bb1d98491cd3 P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:35.295699 16501 tablet_replica.cc:333] T 1d1c1f58c4d340ac951dfa403d34138a P c6610c00ffb948ce9c34f41dec3d49e8: stopping tablet replica
I20260709 12:27:35.296316 16501 raft_consensus.cc:2243] T 1d1c1f58c4d340ac951dfa403d34138a P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:35.296846 16501 raft_consensus.cc:2272] T 1d1c1f58c4d340ac951dfa403d34138a P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:35.299063 16501 tablet_replica.cc:333] T ed057e06dec148a0b9d7062beeb9d005 P c6610c00ffb948ce9c34f41dec3d49e8: stopping tablet replica
I20260709 12:27:35.299607 16501 raft_consensus.cc:2243] T ed057e06dec148a0b9d7062beeb9d005 P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:27:35.300432 16501 raft_consensus.cc:2272] T ed057e06dec148a0b9d7062beeb9d005 P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:35.302443 16501 tablet_replica.cc:333] T a46289e8fc824dd5a818e30df6e7ec3e P c6610c00ffb948ce9c34f41dec3d49e8: stopping tablet replica
I20260709 12:27:35.302979 16501 raft_consensus.cc:2243] T a46289e8fc824dd5a818e30df6e7ec3e P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:27:35.303843 16501 raft_consensus.cc:2272] T a46289e8fc824dd5a818e30df6e7ec3e P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:35.305835 16501 tablet_replica.cc:333] T 58154d49c1a344f2b51366ede3c87de5 P c6610c00ffb948ce9c34f41dec3d49e8: stopping tablet replica
I20260709 12:27:35.306371 16501 raft_consensus.cc:2243] T 58154d49c1a344f2b51366ede3c87de5 P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:27:35.307117 16501 raft_consensus.cc:2272] T 58154d49c1a344f2b51366ede3c87de5 P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:35.309104 16501 tablet_replica.cc:333] T 3f41ce441fb047c8bbb741b1b9cbc0ec P c6610c00ffb948ce9c34f41dec3d49e8: stopping tablet replica
I20260709 12:27:35.309624 16501 raft_consensus.cc:2243] T 3f41ce441fb047c8bbb741b1b9cbc0ec P c6610c00ffb948ce9c34f41dec3d49e8 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:35.310077 16501 raft_consensus.cc:2272] T 3f41ce441fb047c8bbb741b1b9cbc0ec P c6610c00ffb948ce9c34f41dec3d49e8 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:35.311893 16501 tablet_replica.cc:333] T 0c428ff656ae4b9d99d5b5073f98725d P c6610c00ffb948ce9c34f41dec3d49e8: stopping tablet replica
I20260709 12:27:35.312417 16501 raft_consensus.cc:2243] T 0c428ff656ae4b9d99d5b5073f98725d P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:35.312814 16501 raft_consensus.cc:2272] T 0c428ff656ae4b9d99d5b5073f98725d P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:35.314793 16501 tablet_replica.cc:333] T 742beffede4f407d8a72909d2b147e6d P c6610c00ffb948ce9c34f41dec3d49e8: stopping tablet replica
I20260709 12:27:35.315311 16501 raft_consensus.cc:2243] T 742beffede4f407d8a72909d2b147e6d P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:27:35.316201 16501 raft_consensus.cc:2272] T 742beffede4f407d8a72909d2b147e6d P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:35.317961 16501 tablet_replica.cc:333] T ef20d7124e7a44aa97a219ee194f4b8b P c6610c00ffb948ce9c34f41dec3d49e8: stopping tablet replica
I20260709 12:27:35.318473 16501 raft_consensus.cc:2243] T ef20d7124e7a44aa97a219ee194f4b8b P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:27:35.319214 16501 raft_consensus.cc:2272] T ef20d7124e7a44aa97a219ee194f4b8b P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:35.321175 16501 tablet_replica.cc:333] T e67a2affef634fd8bc96a64fa83cffdb P c6610c00ffb948ce9c34f41dec3d49e8: stopping tablet replica
I20260709 12:27:35.321694 16501 raft_consensus.cc:2243] T e67a2affef634fd8bc96a64fa83cffdb P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:35.322125 16501 raft_consensus.cc:2272] T e67a2affef634fd8bc96a64fa83cffdb P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:35.324074 16501 tablet_replica.cc:333] T 55061e62184b45f0bb1726adc1a4af81 P c6610c00ffb948ce9c34f41dec3d49e8: stopping tablet replica
I20260709 12:27:35.324617 16501 raft_consensus.cc:2243] T 55061e62184b45f0bb1726adc1a4af81 P c6610c00ffb948ce9c34f41dec3d49e8 [term 3 LEADER]: Raft consensus shutting down.
I20260709 12:27:35.325451 16501 raft_consensus.cc:2272] T 55061e62184b45f0bb1726adc1a4af81 P c6610c00ffb948ce9c34f41dec3d49e8 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:35.327380 16501 tablet_replica.cc:333] T cdf706ffabc34902ad3d1bf2cee7889e P c6610c00ffb948ce9c34f41dec3d49e8: stopping tablet replica
I20260709 12:27:35.327934 16501 raft_consensus.cc:2243] T cdf706ffabc34902ad3d1bf2cee7889e P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:35.328346 16501 raft_consensus.cc:2272] T cdf706ffabc34902ad3d1bf2cee7889e P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:35.330350 16501 tablet_replica.cc:333] T 43c7bfb33af74d4f912e5153e8a54b3e P c6610c00ffb948ce9c34f41dec3d49e8: stopping tablet replica
I20260709 12:27:35.330961 16501 raft_consensus.cc:2243] T 43c7bfb33af74d4f912e5153e8a54b3e P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:27:35.331808 16501 raft_consensus.cc:2272] T 43c7bfb33af74d4f912e5153e8a54b3e P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:35.333755 16501 tablet_replica.cc:333] T 31a190e9f5ff407fab90502fb3197b70 P c6610c00ffb948ce9c34f41dec3d49e8: stopping tablet replica
I20260709 12:27:35.334276 16501 raft_consensus.cc:2243] T 31a190e9f5ff407fab90502fb3197b70 P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:35.334708 16501 raft_consensus.cc:2272] T 31a190e9f5ff407fab90502fb3197b70 P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:35.336719 16501 tablet_replica.cc:333] T fb7d78e534634e0f992fc39b578ebd8d P c6610c00ffb948ce9c34f41dec3d49e8: stopping tablet replica
I20260709 12:27:35.337258 16501 raft_consensus.cc:2243] T fb7d78e534634e0f992fc39b578ebd8d P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:35.337674 16501 raft_consensus.cc:2272] T fb7d78e534634e0f992fc39b578ebd8d P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:35.339704 16501 tablet_replica.cc:333] T 4685c2e824c14fc1bcd45c16169798cc P c6610c00ffb948ce9c34f41dec3d49e8: stopping tablet replica
I20260709 12:27:35.340245 16501 raft_consensus.cc:2243] T 4685c2e824c14fc1bcd45c16169798cc P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:35.340641 16501 raft_consensus.cc:2272] T 4685c2e824c14fc1bcd45c16169798cc P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:35.342478 16501 tablet_replica.cc:333] T 8627b58b4f42491597125eb45f8d3b61 P c6610c00ffb948ce9c34f41dec3d49e8: stopping tablet replica
I20260709 12:27:35.342980 16501 raft_consensus.cc:2243] T 8627b58b4f42491597125eb45f8d3b61 P c6610c00ffb948ce9c34f41dec3d49e8 [term 3 LEADER]: Raft consensus shutting down.
I20260709 12:27:35.343760 16501 raft_consensus.cc:2272] T 8627b58b4f42491597125eb45f8d3b61 P c6610c00ffb948ce9c34f41dec3d49e8 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:35.345850 16501 tablet_replica.cc:333] T fb6e78ccc01c4707af34cddf81619eab P c6610c00ffb948ce9c34f41dec3d49e8: stopping tablet replica
I20260709 12:27:35.346410 16501 raft_consensus.cc:2243] T fb6e78ccc01c4707af34cddf81619eab P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:35.346813 16501 raft_consensus.cc:2272] T fb6e78ccc01c4707af34cddf81619eab P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:35.348692 16501 tablet_replica.cc:333] T 9633e3b1d78f403097e4a00b7d4b869b P c6610c00ffb948ce9c34f41dec3d49e8: stopping tablet replica
I20260709 12:27:35.349233 16501 raft_consensus.cc:2243] T 9633e3b1d78f403097e4a00b7d4b869b P c6610c00ffb948ce9c34f41dec3d49e8 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:35.349682 16501 raft_consensus.cc:2272] T 9633e3b1d78f403097e4a00b7d4b869b P c6610c00ffb948ce9c34f41dec3d49e8 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:35.351531 16501 tablet_replica.cc:333] T 425777eed34441cd93e0902bdc126516 P c6610c00ffb948ce9c34f41dec3d49e8: stopping tablet replica
I20260709 12:27:35.352090 16501 raft_consensus.cc:2243] T 425777eed34441cd93e0902bdc126516 P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:27:35.352855 16501 raft_consensus.cc:2272] T 425777eed34441cd93e0902bdc126516 P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:35.354727 16501 tablet_replica.cc:333] T 3e9451c3797c4496a66dfdb3e9fb5e38 P c6610c00ffb948ce9c34f41dec3d49e8: stopping tablet replica
I20260709 12:27:35.355260 16501 raft_consensus.cc:2243] T 3e9451c3797c4496a66dfdb3e9fb5e38 P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:35.355654 16501 raft_consensus.cc:2272] T 3e9451c3797c4496a66dfdb3e9fb5e38 P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:35.357646 16501 tablet_replica.cc:333] T d299fb38094e47e8aa3890ca74f6b366 P c6610c00ffb948ce9c34f41dec3d49e8: stopping tablet replica
I20260709 12:27:35.358199 16501 raft_consensus.cc:2243] T d299fb38094e47e8aa3890ca74f6b366 P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:27:35.358997 16501 raft_consensus.cc:2272] T d299fb38094e47e8aa3890ca74f6b366 P c6610c00ffb948ce9c34f41dec3d49e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:35.361301 16501 tablet_replica.cc:333] T 6a30685caefb4d0e80fa4ea87d70760c P c6610c00ffb948ce9c34f41dec3d49e8: stopping tablet replica
I20260709 12:27:35.361819 16501 raft_consensus.cc:2243] T 6a30685caefb4d0e80fa4ea87d70760c P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:35.362228 16501 raft_consensus.cc:2272] T 6a30685caefb4d0e80fa4ea87d70760c P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:35.364181 16501 tablet_replica.cc:333] T 592801934cad4789b9ce8dc09f1470bd P c6610c00ffb948ce9c34f41dec3d49e8: stopping tablet replica
I20260709 12:27:35.364641 16501 raft_consensus.cc:2243] T 592801934cad4789b9ce8dc09f1470bd P c6610c00ffb948ce9c34f41dec3d49e8 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:35.365058 16501 raft_consensus.cc:2272] T 592801934cad4789b9ce8dc09f1470bd P c6610c00ffb948ce9c34f41dec3d49e8 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:35.366946 16501 tablet_replica.cc:333] T 00c6cf271c91400aa834467def9af9b1 P c6610c00ffb948ce9c34f41dec3d49e8: stopping tablet replica
I20260709 12:27:35.367671 16501 raft_consensus.cc:2243] T 00c6cf271c91400aa834467def9af9b1 P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:35.368181 16501 raft_consensus.cc:2272] T 00c6cf271c91400aa834467def9af9b1 P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:35.370198 16501 tablet_replica.cc:333] T c1c8411289a74b5685700229d245ff75 P c6610c00ffb948ce9c34f41dec3d49e8: stopping tablet replica
I20260709 12:27:35.370673 16501 raft_consensus.cc:2243] T c1c8411289a74b5685700229d245ff75 P c6610c00ffb948ce9c34f41dec3d49e8 [term 3 LEADER]: Raft consensus shutting down.
I20260709 12:27:35.371546 16501 raft_consensus.cc:2272] T c1c8411289a74b5685700229d245ff75 P c6610c00ffb948ce9c34f41dec3d49e8 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:35.373871 16501 tablet_replica.cc:333] T 62f8c9472b214576b9002e02536a3ace P c6610c00ffb948ce9c34f41dec3d49e8: stopping tablet replica
I20260709 12:27:35.374347 16501 raft_consensus.cc:2243] T 62f8c9472b214576b9002e02536a3ace P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:35.374792 16501 raft_consensus.cc:2272] T 62f8c9472b214576b9002e02536a3ace P c6610c00ffb948ce9c34f41dec3d49e8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:35.473935 16501 tablet_server.cc:196] TabletServer@127.16.29.68:0 shutdown complete.
I20260709 12:27:35.544234 16501 master.cc:562] Master@127.16.29.126:41639 shutting down...
I20260709 12:27:35.573753 16501 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 8cb294248c0f442d9c16f6b66cd0648e [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:27:35.574427 16501 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 8cb294248c0f442d9c16f6b66cd0648e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:35.574800 16501 tablet_replica.cc:333] T 00000000000000000000000000000000 P 8cb294248c0f442d9c16f6b66cd0648e: stopping tablet replica
I20260709 12:27:35.597926 16501 master.cc:584] Master@127.16.29.126:41639 shutdown complete.
[       OK ] LeaderRebalancerTest.RestartTserver (41990 ms)
[ RUN      ] LeaderRebalancerTest.TestMaintenanceMode
I20260709 12:27:35.695070 16501 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.16.29.126:44357
I20260709 12:27:35.696228 16501 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:27:35.702354 30736 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:27:35.702370 30737 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:27:35.703662 30739 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:27:35.706704 16501 server_base.cc:1061] running on GCE node
I20260709 12:27:35.708177 16501 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:27:35.708359 16501 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:27:35.708559 16501 hybrid_clock.cc:648] HybridClock initialized: now 1783600055708524 us; error 0 us; skew 500 ppm
I20260709 12:27:35.709153 16501 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:27:35.711627 16501 webserver.cc:533] Webserver started at http://127.16.29.126:32977/ using document root <none> and password file <none>
I20260709 12:27:35.712322 16501 fs_manager.cc:362] Metadata directory not provided
I20260709 12:27:35.712545 16501 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:27:35.712888 16501 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:27:35.714259 16501 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783599551912677-16501-0/minicluster-data/master-0-root/instance:
uuid: "43c3827cbc0344b09cc6a5e8215bdb3e"
format_stamp: "Formatted at 2026-07-09 12:27:35 on dist-test-slave-5n9v"
I20260709 12:27:35.719213 16501 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.007s	sys 0.000s
I20260709 12:27:35.723295 30744 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:27:35.724301 16501 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20260709 12:27:35.724674 16501 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783599551912677-16501-0/minicluster-data/master-0-root
uuid: "43c3827cbc0344b09cc6a5e8215bdb3e"
format_stamp: "Formatted at 2026-07-09 12:27:35 on dist-test-slave-5n9v"
I20260709 12:27:35.725015 16501 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783599551912677-16501-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783599551912677-16501-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783599551912677-16501-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:27:35.747969 16501 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:27:35.749229 16501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:27:35.793467 16501 rpc_server.cc:307] RPC server started. Bound to: 127.16.29.126:44357
I20260709 12:27:35.793566 30795 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.29.126:44357 every 8 connection(s)
I20260709 12:27:35.797791 30796 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:35.811107 30796 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43c3827cbc0344b09cc6a5e8215bdb3e: Bootstrap starting.
I20260709 12:27:35.816035 30796 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 43c3827cbc0344b09cc6a5e8215bdb3e: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:35.820422 30796 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43c3827cbc0344b09cc6a5e8215bdb3e: No bootstrap required, opened a new log
I20260709 12:27:35.822744 30796 raft_consensus.cc:359] T 00000000000000000000000000000000 P 43c3827cbc0344b09cc6a5e8215bdb3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43c3827cbc0344b09cc6a5e8215bdb3e" member_type: VOTER }
I20260709 12:27:35.823115 30796 raft_consensus.cc:385] T 00000000000000000000000000000000 P 43c3827cbc0344b09cc6a5e8215bdb3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:35.823366 30796 raft_consensus.cc:740] T 00000000000000000000000000000000 P 43c3827cbc0344b09cc6a5e8215bdb3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43c3827cbc0344b09cc6a5e8215bdb3e, State: Initialized, Role: FOLLOWER
I20260709 12:27:35.824110 30796 consensus_queue.cc:260] T 00000000000000000000000000000000 P 43c3827cbc0344b09cc6a5e8215bdb3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43c3827cbc0344b09cc6a5e8215bdb3e" member_type: VOTER }
I20260709 12:27:35.824568 30796 raft_consensus.cc:399] T 00000000000000000000000000000000 P 43c3827cbc0344b09cc6a5e8215bdb3e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260709 12:27:35.824779 30796 raft_consensus.cc:493] T 00000000000000000000000000000000 P 43c3827cbc0344b09cc6a5e8215bdb3e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260709 12:27:35.825071 30796 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 43c3827cbc0344b09cc6a5e8215bdb3e [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:35.830029 30796 raft_consensus.cc:515] T 00000000000000000000000000000000 P 43c3827cbc0344b09cc6a5e8215bdb3e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43c3827cbc0344b09cc6a5e8215bdb3e" member_type: VOTER }
I20260709 12:27:35.830557 30796 leader_election.cc:304] T 00000000000000000000000000000000 P 43c3827cbc0344b09cc6a5e8215bdb3e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 43c3827cbc0344b09cc6a5e8215bdb3e; no voters: 
I20260709 12:27:35.831703 30796 leader_election.cc:290] T 00000000000000000000000000000000 P 43c3827cbc0344b09cc6a5e8215bdb3e [CANDIDATE]: Term 1 election: Requested vote from peers 
I20260709 12:27:35.832104 30799 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 43c3827cbc0344b09cc6a5e8215bdb3e [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:35.833453 30799 raft_consensus.cc:697] T 00000000000000000000000000000000 P 43c3827cbc0344b09cc6a5e8215bdb3e [term 1 LEADER]: Becoming Leader. State: Replica: 43c3827cbc0344b09cc6a5e8215bdb3e, State: Running, Role: LEADER
I20260709 12:27:35.834270 30799 consensus_queue.cc:237] T 00000000000000000000000000000000 P 43c3827cbc0344b09cc6a5e8215bdb3e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43c3827cbc0344b09cc6a5e8215bdb3e" member_type: VOTER }
I20260709 12:27:35.834700 30796 sys_catalog.cc:565] T 00000000000000000000000000000000 P 43c3827cbc0344b09cc6a5e8215bdb3e [sys.catalog]: configured and running, proceeding with master startup.
I20260709 12:27:35.839020 30800 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43c3827cbc0344b09cc6a5e8215bdb3e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "43c3827cbc0344b09cc6a5e8215bdb3e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43c3827cbc0344b09cc6a5e8215bdb3e" member_type: VOTER } }
I20260709 12:27:35.839942 30800 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43c3827cbc0344b09cc6a5e8215bdb3e [sys.catalog]: This master's current role is: LEADER
I20260709 12:27:35.839646 30801 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43c3827cbc0344b09cc6a5e8215bdb3e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 43c3827cbc0344b09cc6a5e8215bdb3e. Latest consensus state: current_term: 1 leader_uuid: "43c3827cbc0344b09cc6a5e8215bdb3e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43c3827cbc0344b09cc6a5e8215bdb3e" member_type: VOTER } }
I20260709 12:27:35.840703 30801 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43c3827cbc0344b09cc6a5e8215bdb3e [sys.catalog]: This master's current role is: LEADER
I20260709 12:27:35.847172 30806 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260709 12:27:35.855597 30806 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260709 12:27:35.856657 16501 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260709 12:27:35.866257 30806 catalog_manager.cc:1383] Generated new cluster ID: 0a714040d9114c72a35046d6d18af17f
I20260709 12:27:35.866514 30806 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260709 12:27:35.882551 30806 catalog_manager.cc:1406] Generated new certificate authority record
I20260709 12:27:35.884042 30806 catalog_manager.cc:1540] Loading token signing keys...
I20260709 12:27:35.902680 30806 catalog_manager.cc:6070] T 00000000000000000000000000000000 P 43c3827cbc0344b09cc6a5e8215bdb3e: Generated new TSK 0
I20260709 12:27:35.903452 30806 catalog_manager.cc:1550] Initializing in-progress tserver states...
I20260709 12:27:35.923823 16501 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:27:35.931016 30817 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:27:35.932145 30818 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:27:35.934844 30820 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:27:35.935470 16501 server_base.cc:1061] running on GCE node
I20260709 12:27:35.942221 16501 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:27:35.942474 16501 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:27:35.942699 16501 hybrid_clock.cc:648] HybridClock initialized: now 1783600055942683 us; error 0 us; skew 500 ppm
I20260709 12:27:35.943372 16501 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:27:35.946137 16501 webserver.cc:533] Webserver started at http://127.16.29.65:34811/ using document root <none> and password file <none>
I20260709 12:27:35.946785 16501 fs_manager.cc:362] Metadata directory not provided
I20260709 12:27:35.946974 16501 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:27:35.947216 16501 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:27:35.948532 16501 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783599551912677-16501-0/minicluster-data/ts-0-root/instance:
uuid: "75f56c1a7f86421da9247e9b35990e34"
format_stamp: "Formatted at 2026-07-09 12:27:35 on dist-test-slave-5n9v"
I20260709 12:27:35.954236 16501 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.005s	sys 0.001s
I20260709 12:27:35.957890 30825 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:27:35.958849 16501 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.000s
I20260709 12:27:35.959189 16501 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783599551912677-16501-0/minicluster-data/ts-0-root
uuid: "75f56c1a7f86421da9247e9b35990e34"
format_stamp: "Formatted at 2026-07-09 12:27:35 on dist-test-slave-5n9v"
I20260709 12:27:35.959498 16501 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783599551912677-16501-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783599551912677-16501-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783599551912677-16501-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:27:35.975095 16501 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:27:35.976706 16501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:27:35.978410 16501 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:27:35.981149 16501 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:27:35.981415 16501 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:27:35.981696 16501 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:27:35.981940 16501 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:27:36.030505 16501 rpc_server.cc:307] RPC server started. Bound to: 127.16.29.65:45249
I20260709 12:27:36.030632 30887 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.29.65:45249 every 8 connection(s)
I20260709 12:27:36.036008 16501 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:27:36.043936 30892 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:27:36.048792 30893 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:27:36.050704 16501 server_base.cc:1061] running on GCE node
W20260709 12:27:36.050748 30895 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:27:36.054780 16501 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:27:36.055114 16501 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:27:36.055335 16501 hybrid_clock.cc:648] HybridClock initialized: now 1783600056055316 us; error 0 us; skew 500 ppm
I20260709 12:27:36.056090 16501 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:27:36.056658 30888 heartbeater.cc:344] Connected to a master server at 127.16.29.126:44357
I20260709 12:27:36.057009 30888 heartbeater.cc:461] Registering TS with master...
I20260709 12:27:36.057898 30888 heartbeater.cc:507] Master 127.16.29.126:44357 requested a full tablet report, sending...
I20260709 12:27:36.058665 16501 webserver.cc:533] Webserver started at http://127.16.29.66:35141/ using document root <none> and password file <none>
I20260709 12:27:36.059244 16501 fs_manager.cc:362] Metadata directory not provided
I20260709 12:27:36.059454 16501 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:27:36.059728 16501 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:27:36.060284 30761 ts_manager.cc:194] Registered new tserver with Master: 75f56c1a7f86421da9247e9b35990e34 (127.16.29.65:45249)
I20260709 12:27:36.061028 16501 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783599551912677-16501-0/minicluster-data/ts-1-root/instance:
uuid: "e77b2d60f57d4dceb865e440eab35126"
format_stamp: "Formatted at 2026-07-09 12:27:36 on dist-test-slave-5n9v"
I20260709 12:27:36.062237 30761 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:38960
I20260709 12:27:36.066497 16501 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.006s	sys 0.000s
I20260709 12:27:36.071141 30900 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:27:36.072218 16501 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.001s	sys 0.002s
I20260709 12:27:36.072561 16501 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783599551912677-16501-0/minicluster-data/ts-1-root
uuid: "e77b2d60f57d4dceb865e440eab35126"
format_stamp: "Formatted at 2026-07-09 12:27:36 on dist-test-slave-5n9v"
I20260709 12:27:36.072875 16501 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783599551912677-16501-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783599551912677-16501-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783599551912677-16501-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:27:36.097676 16501 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:27:36.099004 16501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:27:36.100698 16501 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:27:36.103140 16501 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:27:36.103397 16501 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:27:36.103680 16501 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:27:36.103919 16501 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:27:36.151801 16501 rpc_server.cc:307] RPC server started. Bound to: 127.16.29.66:46695
I20260709 12:27:36.152021 30962 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.29.66:46695 every 8 connection(s)
I20260709 12:27:36.157227 16501 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:27:36.165710 30966 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:27:36.167376 30967 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:27:36.169425 16501 server_base.cc:1061] running on GCE node
W20260709 12:27:36.170231 30969 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:27:36.171743 16501 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:27:36.172072 16501 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:27:36.172327 16501 hybrid_clock.cc:648] HybridClock initialized: now 1783600056172306 us; error 0 us; skew 500 ppm
I20260709 12:27:36.172387 30963 heartbeater.cc:344] Connected to a master server at 127.16.29.126:44357
I20260709 12:27:36.172842 30963 heartbeater.cc:461] Registering TS with master...
I20260709 12:27:36.172983 16501 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:27:36.173614 30963 heartbeater.cc:507] Master 127.16.29.126:44357 requested a full tablet report, sending...
I20260709 12:27:36.176000 30761 ts_manager.cc:194] Registered new tserver with Master: e77b2d60f57d4dceb865e440eab35126 (127.16.29.66:46695)
I20260709 12:27:36.176091 16501 webserver.cc:533] Webserver started at http://127.16.29.67:35303/ using document root <none> and password file <none>
I20260709 12:27:36.176704 16501 fs_manager.cc:362] Metadata directory not provided
I20260709 12:27:36.176923 16501 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:27:36.177201 16501 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:27:36.177772 30761 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:38968
I20260709 12:27:36.178341 16501 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783599551912677-16501-0/minicluster-data/ts-2-root/instance:
uuid: "2fbc4f1ad97a4f2492306d77a614527a"
format_stamp: "Formatted at 2026-07-09 12:27:36 on dist-test-slave-5n9v"
I20260709 12:27:36.183065 16501 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.004s	sys 0.000s
I20260709 12:27:36.186543 30974 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:27:36.187352 16501 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.000s
I20260709 12:27:36.187678 16501 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783599551912677-16501-0/minicluster-data/ts-2-root
uuid: "2fbc4f1ad97a4f2492306d77a614527a"
format_stamp: "Formatted at 2026-07-09 12:27:36 on dist-test-slave-5n9v"
I20260709 12:27:36.187978 16501 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783599551912677-16501-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783599551912677-16501-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783599551912677-16501-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:27:36.201738 16501 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:27:36.202960 16501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:27:36.204656 16501 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:27:36.207087 16501 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:27:36.207273 16501 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:27:36.207561 16501 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:27:36.207711 16501 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:27:36.255206 16501 rpc_server.cc:307] RPC server started. Bound to: 127.16.29.67:44761
I20260709 12:27:36.255295 31036 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.29.67:44761 every 8 connection(s)
I20260709 12:27:36.269830 31037 heartbeater.cc:344] Connected to a master server at 127.16.29.126:44357
I20260709 12:27:36.270390 31037 heartbeater.cc:461] Registering TS with master...
I20260709 12:27:36.271283 31037 heartbeater.cc:507] Master 127.16.29.126:44357 requested a full tablet report, sending...
I20260709 12:27:36.273424 30761 ts_manager.cc:194] Registered new tserver with Master: 2fbc4f1ad97a4f2492306d77a614527a (127.16.29.67:44761)
I20260709 12:27:36.273778 16501 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.014946492s
I20260709 12:27:36.275166 30761 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:38984
I20260709 12:27:36.304272 30761 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38994:
name: "test-workload"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
  rows: "\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s""\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s"
  indirect_data: """"
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W20260709 12:27:36.307577 30761 catalog_manager.cc:7077] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260709 12:27:36.396122 30928 tablet_service.cc:1511] Processing CreateTablet for tablet b555d1cdbcec4c278fd300d628a8b9ab (DEFAULT_TABLE table=test-workload [id=74f856c0155f4b3684de143988ba8613]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260709 12:27:36.396912 30852 tablet_service.cc:1511] Processing CreateTablet for tablet 2c458afcbd8848d2aa9893f4e74c8561 (DEFAULT_TABLE table=test-workload [id=74f856c0155f4b3684de143988ba8613]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260709 12:27:36.397890 30928 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b555d1cdbcec4c278fd300d628a8b9ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:36.397492 30853 tablet_service.cc:1511] Processing CreateTablet for tablet b555d1cdbcec4c278fd300d628a8b9ab (DEFAULT_TABLE table=test-workload [id=74f856c0155f4b3684de143988ba8613]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260709 12:27:36.398536 30852 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c458afcbd8848d2aa9893f4e74c8561. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:36.399318 30853 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b555d1cdbcec4c278fd300d628a8b9ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:36.401021 30851 tablet_service.cc:1511] Processing CreateTablet for tablet 8e560587c5c24e4587295d2d1b6d2be6 (DEFAULT_TABLE table=test-workload [id=74f856c0155f4b3684de143988ba8613]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260709 12:27:36.402603 30851 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e560587c5c24e4587295d2d1b6d2be6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:36.409705 30927 tablet_service.cc:1511] Processing CreateTablet for tablet 2c458afcbd8848d2aa9893f4e74c8561 (DEFAULT_TABLE table=test-workload [id=74f856c0155f4b3684de143988ba8613]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260709 12:27:36.411331 30927 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c458afcbd8848d2aa9893f4e74c8561. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:36.411762 31002 tablet_service.cc:1511] Processing CreateTablet for tablet b555d1cdbcec4c278fd300d628a8b9ab (DEFAULT_TABLE table=test-workload [id=74f856c0155f4b3684de143988ba8613]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260709 12:27:36.412549 31001 tablet_service.cc:1511] Processing CreateTablet for tablet 2c458afcbd8848d2aa9893f4e74c8561 (DEFAULT_TABLE table=test-workload [id=74f856c0155f4b3684de143988ba8613]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260709 12:27:36.413372 31002 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b555d1cdbcec4c278fd300d628a8b9ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:36.414163 31001 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c458afcbd8848d2aa9893f4e74c8561. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:36.418979 30926 tablet_service.cc:1511] Processing CreateTablet for tablet 8e560587c5c24e4587295d2d1b6d2be6 (DEFAULT_TABLE table=test-workload [id=74f856c0155f4b3684de143988ba8613]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260709 12:27:36.420621 30926 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e560587c5c24e4587295d2d1b6d2be6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:36.421078 30850 tablet_service.cc:1511] Processing CreateTablet for tablet f64692cd6a0347979834f29861bde5d4 (DEFAULT_TABLE table=test-workload [id=74f856c0155f4b3684de143988ba8613]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260709 12:27:36.422572 30850 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f64692cd6a0347979834f29861bde5d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:36.437520 30925 tablet_service.cc:1511] Processing CreateTablet for tablet f64692cd6a0347979834f29861bde5d4 (DEFAULT_TABLE table=test-workload [id=74f856c0155f4b3684de143988ba8613]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260709 12:27:36.439167 30925 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f64692cd6a0347979834f29861bde5d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:36.448688 30852 tablet_service.cc:1511] Processing CreateTablet for tablet 0be78a9b3c5d47df8efea08d91f58111 (DEFAULT_TABLE table=test-workload [id=74f856c0155f4b3684de143988ba8613]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260709 12:27:36.450268 30852 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0be78a9b3c5d47df8efea08d91f58111. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:36.451380 30924 tablet_service.cc:1511] Processing CreateTablet for tablet 0be78a9b3c5d47df8efea08d91f58111 (DEFAULT_TABLE table=test-workload [id=74f856c0155f4b3684de143988ba8613]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260709 12:27:36.452860 30924 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0be78a9b3c5d47df8efea08d91f58111. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:36.457178 30849 tablet_service.cc:1511] Processing CreateTablet for tablet de8ccd191af64c50ac32e3342ca995b0 (DEFAULT_TABLE table=test-workload [id=74f856c0155f4b3684de143988ba8613]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260709 12:27:36.458633 30849 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de8ccd191af64c50ac32e3342ca995b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:36.464911 30923 tablet_service.cc:1511] Processing CreateTablet for tablet de8ccd191af64c50ac32e3342ca995b0 (DEFAULT_TABLE table=test-workload [id=74f856c0155f4b3684de143988ba8613]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260709 12:27:36.466481 30923 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de8ccd191af64c50ac32e3342ca995b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:36.470371 31000 tablet_service.cc:1511] Processing CreateTablet for tablet 8e560587c5c24e4587295d2d1b6d2be6 (DEFAULT_TABLE table=test-workload [id=74f856c0155f4b3684de143988ba8613]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260709 12:27:36.471037 30999 tablet_service.cc:1511] Processing CreateTablet for tablet f64692cd6a0347979834f29861bde5d4 (DEFAULT_TABLE table=test-workload [id=74f856c0155f4b3684de143988ba8613]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260709 12:27:36.472028 31000 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e560587c5c24e4587295d2d1b6d2be6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:36.473004 30997 tablet_service.cc:1511] Processing CreateTablet for tablet de8ccd191af64c50ac32e3342ca995b0 (DEFAULT_TABLE table=test-workload [id=74f856c0155f4b3684de143988ba8613]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260709 12:27:36.473667 30999 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f64692cd6a0347979834f29861bde5d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:36.474597 30997 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de8ccd191af64c50ac32e3342ca995b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:36.477288 30998 tablet_service.cc:1511] Processing CreateTablet for tablet 0be78a9b3c5d47df8efea08d91f58111 (DEFAULT_TABLE table=test-workload [id=74f856c0155f4b3684de143988ba8613]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260709 12:27:36.478884 30998 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0be78a9b3c5d47df8efea08d91f58111. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:36.481042 30848 tablet_service.cc:1511] Processing CreateTablet for tablet 009d38185f3e49afbbd9faf57f968b2c (DEFAULT_TABLE table=test-workload [id=74f856c0155f4b3684de143988ba8613]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260709 12:27:36.483006 30848 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 009d38185f3e49afbbd9faf57f968b2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:36.490945 31052 tablet_bootstrap.cc:492] T b555d1cdbcec4c278fd300d628a8b9ab P 75f56c1a7f86421da9247e9b35990e34: Bootstrap starting.
I20260709 12:27:36.511117 30922 tablet_service.cc:1511] Processing CreateTablet for tablet 009d38185f3e49afbbd9faf57f968b2c (DEFAULT_TABLE table=test-workload [id=74f856c0155f4b3684de143988ba8613]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260709 12:27:36.512758 30922 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 009d38185f3e49afbbd9faf57f968b2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:36.514281 30921 tablet_service.cc:1511] Processing CreateTablet for tablet 3ed242e2c2b843978a7af7cca5694ae2 (DEFAULT_TABLE table=test-workload [id=74f856c0155f4b3684de143988ba8613]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260709 12:27:36.515756 30921 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3ed242e2c2b843978a7af7cca5694ae2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:36.512782 30850 tablet_service.cc:1511] Processing CreateTablet for tablet 3ed242e2c2b843978a7af7cca5694ae2 (DEFAULT_TABLE table=test-workload [id=74f856c0155f4b3684de143988ba8613]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260709 12:27:36.520663 30850 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3ed242e2c2b843978a7af7cca5694ae2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:36.523460 30996 tablet_service.cc:1511] Processing CreateTablet for tablet 3ed242e2c2b843978a7af7cca5694ae2 (DEFAULT_TABLE table=test-workload [id=74f856c0155f4b3684de143988ba8613]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260709 12:27:36.525241 30996 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3ed242e2c2b843978a7af7cca5694ae2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:36.529228 31002 tablet_service.cc:1511] Processing CreateTablet for tablet 009d38185f3e49afbbd9faf57f968b2c (DEFAULT_TABLE table=test-workload [id=74f856c0155f4b3684de143988ba8613]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260709 12:27:36.529964 30853 tablet_service.cc:1511] Processing CreateTablet for tablet adedadb9002047ba87a533a33374f4e7 (DEFAULT_TABLE table=test-workload [id=74f856c0155f4b3684de143988ba8613]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260709 12:27:36.537863 31053 tablet_bootstrap.cc:492] T 2c458afcbd8848d2aa9893f4e74c8561 P 2fbc4f1ad97a4f2492306d77a614527a: Bootstrap starting.
I20260709 12:27:36.538282 31052 tablet_bootstrap.cc:654] T b555d1cdbcec4c278fd300d628a8b9ab P 75f56c1a7f86421da9247e9b35990e34: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:36.538853 30853 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet adedadb9002047ba87a533a33374f4e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:36.538295 31002 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 009d38185f3e49afbbd9faf57f968b2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:36.548642 30998 tablet_service.cc:1511] Processing CreateTablet for tablet adedadb9002047ba87a533a33374f4e7 (DEFAULT_TABLE table=test-workload [id=74f856c0155f4b3684de143988ba8613]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260709 12:27:36.554908 30851 tablet_service.cc:1511] Processing CreateTablet for tablet 881773ceed164ddf8ebc390254b1eb01 (DEFAULT_TABLE table=test-workload [id=74f856c0155f4b3684de143988ba8613]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260709 12:27:36.558007 30851 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 881773ceed164ddf8ebc390254b1eb01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:36.558084 30926 tablet_service.cc:1511] Processing CreateTablet for tablet adedadb9002047ba87a533a33374f4e7 (DEFAULT_TABLE table=test-workload [id=74f856c0155f4b3684de143988ba8613]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260709 12:27:36.559463 30926 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet adedadb9002047ba87a533a33374f4e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:36.567349 30998 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet adedadb9002047ba87a533a33374f4e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:36.583858 31053 tablet_bootstrap.cc:654] T 2c458afcbd8848d2aa9893f4e74c8561 P 2fbc4f1ad97a4f2492306d77a614527a: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:36.590597 30927 tablet_service.cc:1511] Processing CreateTablet for tablet 881773ceed164ddf8ebc390254b1eb01 (DEFAULT_TABLE table=test-workload [id=74f856c0155f4b3684de143988ba8613]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260709 12:27:36.592120 30927 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 881773ceed164ddf8ebc390254b1eb01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:36.616364 31053 tablet_bootstrap.cc:492] T 2c458afcbd8848d2aa9893f4e74c8561 P 2fbc4f1ad97a4f2492306d77a614527a: No bootstrap required, opened a new log
I20260709 12:27:36.617084 31054 tablet_bootstrap.cc:492] T 2c458afcbd8848d2aa9893f4e74c8561 P e77b2d60f57d4dceb865e440eab35126: Bootstrap starting.
I20260709 12:27:36.617226 31053 ts_tablet_manager.cc:1403] T 2c458afcbd8848d2aa9893f4e74c8561 P 2fbc4f1ad97a4f2492306d77a614527a: Time spent bootstrapping tablet: real 0.080s	user 0.011s	sys 0.013s
I20260709 12:27:36.621335 31053 raft_consensus.cc:359] T 2c458afcbd8848d2aa9893f4e74c8561 P 2fbc4f1ad97a4f2492306d77a614527a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } }
I20260709 12:27:36.622115 31053 raft_consensus.cc:385] T 2c458afcbd8848d2aa9893f4e74c8561 P 2fbc4f1ad97a4f2492306d77a614527a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:36.622476 31053 raft_consensus.cc:740] T 2c458afcbd8848d2aa9893f4e74c8561 P 2fbc4f1ad97a4f2492306d77a614527a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fbc4f1ad97a4f2492306d77a614527a, State: Initialized, Role: FOLLOWER
I20260709 12:27:36.623566 31053 consensus_queue.cc:260] T 2c458afcbd8848d2aa9893f4e74c8561 P 2fbc4f1ad97a4f2492306d77a614527a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } }
I20260709 12:27:36.625671 30999 tablet_service.cc:1511] Processing CreateTablet for tablet 881773ceed164ddf8ebc390254b1eb01 (DEFAULT_TABLE table=test-workload [id=74f856c0155f4b3684de143988ba8613]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260709 12:27:36.627014 30999 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 881773ceed164ddf8ebc390254b1eb01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:36.630251 31052 tablet_bootstrap.cc:492] T b555d1cdbcec4c278fd300d628a8b9ab P 75f56c1a7f86421da9247e9b35990e34: No bootstrap required, opened a new log
I20260709 12:27:36.631032 31052 ts_tablet_manager.cc:1403] T b555d1cdbcec4c278fd300d628a8b9ab P 75f56c1a7f86421da9247e9b35990e34: Time spent bootstrapping tablet: real 0.141s	user 0.023s	sys 0.029s
I20260709 12:27:36.636929 31054 tablet_bootstrap.cc:654] T 2c458afcbd8848d2aa9893f4e74c8561 P e77b2d60f57d4dceb865e440eab35126: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:36.635294 31052 raft_consensus.cc:359] T b555d1cdbcec4c278fd300d628a8b9ab P 75f56c1a7f86421da9247e9b35990e34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } }
I20260709 12:27:36.637976 31052 raft_consensus.cc:385] T b555d1cdbcec4c278fd300d628a8b9ab P 75f56c1a7f86421da9247e9b35990e34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:36.638397 31052 raft_consensus.cc:740] T b555d1cdbcec4c278fd300d628a8b9ab P 75f56c1a7f86421da9247e9b35990e34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75f56c1a7f86421da9247e9b35990e34, State: Initialized, Role: FOLLOWER
I20260709 12:27:36.644111 31052 consensus_queue.cc:260] T b555d1cdbcec4c278fd300d628a8b9ab P 75f56c1a7f86421da9247e9b35990e34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } }
I20260709 12:27:36.647775 31037 heartbeater.cc:499] Master 127.16.29.126:44357 was elected leader, sending a full tablet report...
I20260709 12:27:36.648636 31053 ts_tablet_manager.cc:1434] T 2c458afcbd8848d2aa9893f4e74c8561 P 2fbc4f1ad97a4f2492306d77a614527a: Time spent starting tablet: real 0.030s	user 0.008s	sys 0.018s
I20260709 12:27:36.649338 31054 tablet_bootstrap.cc:492] T 2c458afcbd8848d2aa9893f4e74c8561 P e77b2d60f57d4dceb865e440eab35126: No bootstrap required, opened a new log
I20260709 12:27:36.649667 31053 tablet_bootstrap.cc:492] T b555d1cdbcec4c278fd300d628a8b9ab P 2fbc4f1ad97a4f2492306d77a614527a: Bootstrap starting.
I20260709 12:27:36.649868 31054 ts_tablet_manager.cc:1403] T 2c458afcbd8848d2aa9893f4e74c8561 P e77b2d60f57d4dceb865e440eab35126: Time spent bootstrapping tablet: real 0.033s	user 0.009s	sys 0.005s
I20260709 12:27:36.653375 31054 raft_consensus.cc:359] T 2c458afcbd8848d2aa9893f4e74c8561 P e77b2d60f57d4dceb865e440eab35126 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } }
I20260709 12:27:36.654772 31054 raft_consensus.cc:385] T 2c458afcbd8848d2aa9893f4e74c8561 P e77b2d60f57d4dceb865e440eab35126 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:36.655128 31054 raft_consensus.cc:740] T 2c458afcbd8848d2aa9893f4e74c8561 P e77b2d60f57d4dceb865e440eab35126 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e77b2d60f57d4dceb865e440eab35126, State: Initialized, Role: FOLLOWER
I20260709 12:27:36.657640 31054 consensus_queue.cc:260] T 2c458afcbd8848d2aa9893f4e74c8561 P e77b2d60f57d4dceb865e440eab35126 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } }
I20260709 12:27:36.660393 31053 tablet_bootstrap.cc:654] T b555d1cdbcec4c278fd300d628a8b9ab P 2fbc4f1ad97a4f2492306d77a614527a: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:36.662319 31052 ts_tablet_manager.cc:1434] T b555d1cdbcec4c278fd300d628a8b9ab P 75f56c1a7f86421da9247e9b35990e34: Time spent starting tablet: real 0.031s	user 0.011s	sys 0.002s
I20260709 12:27:36.663146 30888 heartbeater.cc:499] Master 127.16.29.126:44357 was elected leader, sending a full tablet report...
I20260709 12:27:36.675292 31052 tablet_bootstrap.cc:492] T 2c458afcbd8848d2aa9893f4e74c8561 P 75f56c1a7f86421da9247e9b35990e34: Bootstrap starting.
I20260709 12:27:36.684716 31052 tablet_bootstrap.cc:654] T 2c458afcbd8848d2aa9893f4e74c8561 P 75f56c1a7f86421da9247e9b35990e34: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:36.691671 30963 heartbeater.cc:499] Master 127.16.29.126:44357 was elected leader, sending a full tablet report...
I20260709 12:27:36.692281 31054 ts_tablet_manager.cc:1434] T 2c458afcbd8848d2aa9893f4e74c8561 P e77b2d60f57d4dceb865e440eab35126: Time spent starting tablet: real 0.042s	user 0.005s	sys 0.033s
I20260709 12:27:36.693344 31054 tablet_bootstrap.cc:492] T 8e560587c5c24e4587295d2d1b6d2be6 P e77b2d60f57d4dceb865e440eab35126: Bootstrap starting.
I20260709 12:27:36.700505 31054 tablet_bootstrap.cc:654] T 8e560587c5c24e4587295d2d1b6d2be6 P e77b2d60f57d4dceb865e440eab35126: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:36.702643 31053 tablet_bootstrap.cc:492] T b555d1cdbcec4c278fd300d628a8b9ab P 2fbc4f1ad97a4f2492306d77a614527a: No bootstrap required, opened a new log
I20260709 12:27:36.703367 31053 ts_tablet_manager.cc:1403] T b555d1cdbcec4c278fd300d628a8b9ab P 2fbc4f1ad97a4f2492306d77a614527a: Time spent bootstrapping tablet: real 0.054s	user 0.026s	sys 0.014s
I20260709 12:27:36.707931 31053 raft_consensus.cc:359] T b555d1cdbcec4c278fd300d628a8b9ab P 2fbc4f1ad97a4f2492306d77a614527a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } }
I20260709 12:27:36.709075 31053 raft_consensus.cc:385] T b555d1cdbcec4c278fd300d628a8b9ab P 2fbc4f1ad97a4f2492306d77a614527a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:36.709564 31053 raft_consensus.cc:740] T b555d1cdbcec4c278fd300d628a8b9ab P 2fbc4f1ad97a4f2492306d77a614527a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fbc4f1ad97a4f2492306d77a614527a, State: Initialized, Role: FOLLOWER
I20260709 12:27:36.710997 31053 consensus_queue.cc:260] T b555d1cdbcec4c278fd300d628a8b9ab P 2fbc4f1ad97a4f2492306d77a614527a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } }
I20260709 12:27:36.714298 31053 ts_tablet_manager.cc:1434] T b555d1cdbcec4c278fd300d628a8b9ab P 2fbc4f1ad97a4f2492306d77a614527a: Time spent starting tablet: real 0.010s	user 0.006s	sys 0.002s
I20260709 12:27:36.716045 31053 tablet_bootstrap.cc:492] T 0be78a9b3c5d47df8efea08d91f58111 P 2fbc4f1ad97a4f2492306d77a614527a: Bootstrap starting.
I20260709 12:27:36.720151 31057 raft_consensus.cc:493] T b555d1cdbcec4c278fd300d628a8b9ab P 2fbc4f1ad97a4f2492306d77a614527a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:36.720929 31057 raft_consensus.cc:515] T b555d1cdbcec4c278fd300d628a8b9ab P 2fbc4f1ad97a4f2492306d77a614527a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } }
I20260709 12:27:36.726176 31053 tablet_bootstrap.cc:654] T 0be78a9b3c5d47df8efea08d91f58111 P 2fbc4f1ad97a4f2492306d77a614527a: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:36.735144 31057 leader_election.cc:290] T b555d1cdbcec4c278fd300d628a8b9ab P 2fbc4f1ad97a4f2492306d77a614527a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 75f56c1a7f86421da9247e9b35990e34 (127.16.29.65:45249), e77b2d60f57d4dceb865e440eab35126 (127.16.29.66:46695)
I20260709 12:27:36.742872 30863 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b555d1cdbcec4c278fd300d628a8b9ab" candidate_uuid: "2fbc4f1ad97a4f2492306d77a614527a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75f56c1a7f86421da9247e9b35990e34" is_pre_election: true
I20260709 12:27:36.743932 30863 raft_consensus.cc:2468] T b555d1cdbcec4c278fd300d628a8b9ab P 75f56c1a7f86421da9247e9b35990e34 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2fbc4f1ad97a4f2492306d77a614527a in term 0.
I20260709 12:27:36.745956 30977 leader_election.cc:304] T b555d1cdbcec4c278fd300d628a8b9ab P 2fbc4f1ad97a4f2492306d77a614527a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2fbc4f1ad97a4f2492306d77a614527a, 75f56c1a7f86421da9247e9b35990e34; no voters: 
I20260709 12:27:36.753557 31057 raft_consensus.cc:2804] T b555d1cdbcec4c278fd300d628a8b9ab P 2fbc4f1ad97a4f2492306d77a614527a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:36.754050 31057 raft_consensus.cc:493] T b555d1cdbcec4c278fd300d628a8b9ab P 2fbc4f1ad97a4f2492306d77a614527a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:36.754441 31057 raft_consensus.cc:3060] T b555d1cdbcec4c278fd300d628a8b9ab P 2fbc4f1ad97a4f2492306d77a614527a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:36.757145 31052 tablet_bootstrap.cc:492] T 2c458afcbd8848d2aa9893f4e74c8561 P 75f56c1a7f86421da9247e9b35990e34: No bootstrap required, opened a new log
I20260709 12:27:36.757810 31052 ts_tablet_manager.cc:1403] T 2c458afcbd8848d2aa9893f4e74c8561 P 75f56c1a7f86421da9247e9b35990e34: Time spent bootstrapping tablet: real 0.083s	user 0.024s	sys 0.017s
I20260709 12:27:36.758162 31053 tablet_bootstrap.cc:492] T 0be78a9b3c5d47df8efea08d91f58111 P 2fbc4f1ad97a4f2492306d77a614527a: No bootstrap required, opened a new log
I20260709 12:27:36.758694 31053 ts_tablet_manager.cc:1403] T 0be78a9b3c5d47df8efea08d91f58111 P 2fbc4f1ad97a4f2492306d77a614527a: Time spent bootstrapping tablet: real 0.043s	user 0.019s	sys 0.019s
W20260709 12:27:36.759783 31038 tablet.cc:2406] T 2c458afcbd8848d2aa9893f4e74c8561 P 2fbc4f1ad97a4f2492306d77a614527a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260709 12:27:36.761343 31052 raft_consensus.cc:359] T 2c458afcbd8848d2aa9893f4e74c8561 P 75f56c1a7f86421da9247e9b35990e34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } }
I20260709 12:27:36.761634 31053 raft_consensus.cc:359] T 0be78a9b3c5d47df8efea08d91f58111 P 2fbc4f1ad97a4f2492306d77a614527a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } }
I20260709 12:27:36.762225 31052 raft_consensus.cc:385] T 2c458afcbd8848d2aa9893f4e74c8561 P 75f56c1a7f86421da9247e9b35990e34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:36.762403 31053 raft_consensus.cc:385] T 0be78a9b3c5d47df8efea08d91f58111 P 2fbc4f1ad97a4f2492306d77a614527a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:36.762593 31057 raft_consensus.cc:515] T b555d1cdbcec4c278fd300d628a8b9ab P 2fbc4f1ad97a4f2492306d77a614527a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } }
I20260709 12:27:36.762609 31052 raft_consensus.cc:740] T 2c458afcbd8848d2aa9893f4e74c8561 P 75f56c1a7f86421da9247e9b35990e34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75f56c1a7f86421da9247e9b35990e34, State: Initialized, Role: FOLLOWER
I20260709 12:27:36.764504 30938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b555d1cdbcec4c278fd300d628a8b9ab" candidate_uuid: "2fbc4f1ad97a4f2492306d77a614527a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e77b2d60f57d4dceb865e440eab35126" is_pre_election: true
I20260709 12:27:36.764596 31052 consensus_queue.cc:260] T 2c458afcbd8848d2aa9893f4e74c8561 P 75f56c1a7f86421da9247e9b35990e34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } }
I20260709 12:27:36.762814 31053 raft_consensus.cc:740] T 0be78a9b3c5d47df8efea08d91f58111 P 2fbc4f1ad97a4f2492306d77a614527a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fbc4f1ad97a4f2492306d77a614527a, State: Initialized, Role: FOLLOWER
I20260709 12:27:36.765926 31054 tablet_bootstrap.cc:492] T 8e560587c5c24e4587295d2d1b6d2be6 P e77b2d60f57d4dceb865e440eab35126: No bootstrap required, opened a new log
W20260709 12:27:36.766000 30976 leader_election.cc:343] T b555d1cdbcec4c278fd300d628a8b9ab P 2fbc4f1ad97a4f2492306d77a614527a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e77b2d60f57d4dceb865e440eab35126 (127.16.29.66:46695): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:36.766726 31054 ts_tablet_manager.cc:1403] T 8e560587c5c24e4587295d2d1b6d2be6 P e77b2d60f57d4dceb865e440eab35126: Time spent bootstrapping tablet: real 0.074s	user 0.030s	sys 0.022s
I20260709 12:27:36.766464 31053 consensus_queue.cc:260] T 0be78a9b3c5d47df8efea08d91f58111 P 2fbc4f1ad97a4f2492306d77a614527a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } }
I20260709 12:27:36.767460 30863 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b555d1cdbcec4c278fd300d628a8b9ab" candidate_uuid: "2fbc4f1ad97a4f2492306d77a614527a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75f56c1a7f86421da9247e9b35990e34"
I20260709 12:27:36.768031 30863 raft_consensus.cc:3060] T b555d1cdbcec4c278fd300d628a8b9ab P 75f56c1a7f86421da9247e9b35990e34 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:36.771181 31052 ts_tablet_manager.cc:1434] T 2c458afcbd8848d2aa9893f4e74c8561 P 75f56c1a7f86421da9247e9b35990e34: Time spent starting tablet: real 0.013s	user 0.008s	sys 0.000s
I20260709 12:27:36.771163 31054 raft_consensus.cc:359] T 8e560587c5c24e4587295d2d1b6d2be6 P e77b2d60f57d4dceb865e440eab35126 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } }
I20260709 12:27:36.772590 31052 tablet_bootstrap.cc:492] T 8e560587c5c24e4587295d2d1b6d2be6 P 75f56c1a7f86421da9247e9b35990e34: Bootstrap starting.
I20260709 12:27:36.772636 31054 raft_consensus.cc:385] T 8e560587c5c24e4587295d2d1b6d2be6 P e77b2d60f57d4dceb865e440eab35126 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:36.773373 31054 raft_consensus.cc:740] T 8e560587c5c24e4587295d2d1b6d2be6 P e77b2d60f57d4dceb865e440eab35126 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e77b2d60f57d4dceb865e440eab35126, State: Initialized, Role: FOLLOWER
I20260709 12:27:36.775010 31054 consensus_queue.cc:260] T 8e560587c5c24e4587295d2d1b6d2be6 P e77b2d60f57d4dceb865e440eab35126 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } }
I20260709 12:27:36.780144 31052 tablet_bootstrap.cc:654] T 8e560587c5c24e4587295d2d1b6d2be6 P 75f56c1a7f86421da9247e9b35990e34: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:36.789441 31060 raft_consensus.cc:493] T 2c458afcbd8848d2aa9893f4e74c8561 P e77b2d60f57d4dceb865e440eab35126 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:36.789541 31054 ts_tablet_manager.cc:1434] T 8e560587c5c24e4587295d2d1b6d2be6 P e77b2d60f57d4dceb865e440eab35126: Time spent starting tablet: real 0.022s	user 0.009s	sys 0.002s
I20260709 12:27:36.790089 31060 raft_consensus.cc:515] T 2c458afcbd8848d2aa9893f4e74c8561 P e77b2d60f57d4dceb865e440eab35126 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } }
I20260709 12:27:36.792163 31052 tablet_bootstrap.cc:492] T 8e560587c5c24e4587295d2d1b6d2be6 P 75f56c1a7f86421da9247e9b35990e34: No bootstrap required, opened a new log
I20260709 12:27:36.792729 31052 ts_tablet_manager.cc:1403] T 8e560587c5c24e4587295d2d1b6d2be6 P 75f56c1a7f86421da9247e9b35990e34: Time spent bootstrapping tablet: real 0.020s	user 0.008s	sys 0.010s
I20260709 12:27:36.796365 31052 raft_consensus.cc:359] T 8e560587c5c24e4587295d2d1b6d2be6 P 75f56c1a7f86421da9247e9b35990e34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } }
I20260709 12:27:36.797330 31052 raft_consensus.cc:385] T 8e560587c5c24e4587295d2d1b6d2be6 P 75f56c1a7f86421da9247e9b35990e34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:36.797700 31052 raft_consensus.cc:740] T 8e560587c5c24e4587295d2d1b6d2be6 P 75f56c1a7f86421da9247e9b35990e34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75f56c1a7f86421da9247e9b35990e34, State: Initialized, Role: FOLLOWER
I20260709 12:27:36.798583 31052 consensus_queue.cc:260] T 8e560587c5c24e4587295d2d1b6d2be6 P 75f56c1a7f86421da9247e9b35990e34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } }
I20260709 12:27:36.801676 31052 ts_tablet_manager.cc:1434] T 8e560587c5c24e4587295d2d1b6d2be6 P 75f56c1a7f86421da9247e9b35990e34: Time spent starting tablet: real 0.009s	user 0.007s	sys 0.001s
I20260709 12:27:36.802825 31052 tablet_bootstrap.cc:492] T f64692cd6a0347979834f29861bde5d4 P 75f56c1a7f86421da9247e9b35990e34: Bootstrap starting.
I20260709 12:27:36.803761 31054 tablet_bootstrap.cc:492] T b555d1cdbcec4c278fd300d628a8b9ab P e77b2d60f57d4dceb865e440eab35126: Bootstrap starting.
I20260709 12:27:36.809793 31052 tablet_bootstrap.cc:654] T f64692cd6a0347979834f29861bde5d4 P 75f56c1a7f86421da9247e9b35990e34: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:36.810369 31060 leader_election.cc:290] T 2c458afcbd8848d2aa9893f4e74c8561 P e77b2d60f57d4dceb865e440eab35126 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 75f56c1a7f86421da9247e9b35990e34 (127.16.29.65:45249), 2fbc4f1ad97a4f2492306d77a614527a (127.16.29.67:44761)
I20260709 12:27:36.816432 31054 tablet_bootstrap.cc:654] T b555d1cdbcec4c278fd300d628a8b9ab P e77b2d60f57d4dceb865e440eab35126: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:36.822049 31052 tablet_bootstrap.cc:492] T f64692cd6a0347979834f29861bde5d4 P 75f56c1a7f86421da9247e9b35990e34: No bootstrap required, opened a new log
I20260709 12:27:36.825870 31052 ts_tablet_manager.cc:1403] T f64692cd6a0347979834f29861bde5d4 P 75f56c1a7f86421da9247e9b35990e34: Time spent bootstrapping tablet: real 0.023s	user 0.009s	sys 0.009s
I20260709 12:27:36.829098 31052 raft_consensus.cc:359] T f64692cd6a0347979834f29861bde5d4 P 75f56c1a7f86421da9247e9b35990e34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } }
I20260709 12:27:36.830228 31052 raft_consensus.cc:385] T f64692cd6a0347979834f29861bde5d4 P 75f56c1a7f86421da9247e9b35990e34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:36.830871 31052 raft_consensus.cc:740] T f64692cd6a0347979834f29861bde5d4 P 75f56c1a7f86421da9247e9b35990e34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75f56c1a7f86421da9247e9b35990e34, State: Initialized, Role: FOLLOWER
I20260709 12:27:36.821887 30863 raft_consensus.cc:2468] T b555d1cdbcec4c278fd300d628a8b9ab P 75f56c1a7f86421da9247e9b35990e34 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2fbc4f1ad97a4f2492306d77a614527a in term 1.
I20260709 12:27:36.832196 31052 consensus_queue.cc:260] T f64692cd6a0347979834f29861bde5d4 P 75f56c1a7f86421da9247e9b35990e34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } }
I20260709 12:27:36.832823 30977 leader_election.cc:304] T b555d1cdbcec4c278fd300d628a8b9ab P 2fbc4f1ad97a4f2492306d77a614527a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2fbc4f1ad97a4f2492306d77a614527a, 75f56c1a7f86421da9247e9b35990e34; no voters: 
I20260709 12:27:36.837751 31052 ts_tablet_manager.cc:1434] T f64692cd6a0347979834f29861bde5d4 P 75f56c1a7f86421da9247e9b35990e34: Time spent starting tablet: real 0.011s	user 0.002s	sys 0.005s
I20260709 12:27:36.839000 31052 tablet_bootstrap.cc:492] T 009d38185f3e49afbbd9faf57f968b2c P 75f56c1a7f86421da9247e9b35990e34: Bootstrap starting.
I20260709 12:27:36.839428 31054 tablet_bootstrap.cc:492] T b555d1cdbcec4c278fd300d628a8b9ab P e77b2d60f57d4dceb865e440eab35126: No bootstrap required, opened a new log
I20260709 12:27:36.840265 31054 ts_tablet_manager.cc:1403] T b555d1cdbcec4c278fd300d628a8b9ab P e77b2d60f57d4dceb865e440eab35126: Time spent bootstrapping tablet: real 0.037s	user 0.012s	sys 0.013s
I20260709 12:27:36.823028 31057 leader_election.cc:290] T b555d1cdbcec4c278fd300d628a8b9ab P 2fbc4f1ad97a4f2492306d77a614527a [CANDIDATE]: Term 1 election: Requested vote from peers 75f56c1a7f86421da9247e9b35990e34 (127.16.29.65:45249), e77b2d60f57d4dceb865e440eab35126 (127.16.29.66:46695)
I20260709 12:27:36.824584 30938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b555d1cdbcec4c278fd300d628a8b9ab" candidate_uuid: "2fbc4f1ad97a4f2492306d77a614527a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e77b2d60f57d4dceb865e440eab35126"
I20260709 12:27:36.844965 31057 raft_consensus.cc:2804] T b555d1cdbcec4c278fd300d628a8b9ab P 2fbc4f1ad97a4f2492306d77a614527a [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:36.846558 31057 raft_consensus.cc:697] T b555d1cdbcec4c278fd300d628a8b9ab P 2fbc4f1ad97a4f2492306d77a614527a [term 1 LEADER]: Becoming Leader. State: Replica: 2fbc4f1ad97a4f2492306d77a614527a, State: Running, Role: LEADER
W20260709 12:27:36.853088 30976 leader_election.cc:343] T b555d1cdbcec4c278fd300d628a8b9ab P 2fbc4f1ad97a4f2492306d77a614527a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer e77b2d60f57d4dceb865e440eab35126 (127.16.29.66:46695): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:36.853675 31057 consensus_queue.cc:237] T b555d1cdbcec4c278fd300d628a8b9ab P 2fbc4f1ad97a4f2492306d77a614527a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } }
I20260709 12:27:36.847811 31052 tablet_bootstrap.cc:654] T 009d38185f3e49afbbd9faf57f968b2c P 75f56c1a7f86421da9247e9b35990e34: Neither blocks nor log segments found. Creating new log.
W20260709 12:27:36.848227 30809 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:27:36.856911 31054 raft_consensus.cc:359] T b555d1cdbcec4c278fd300d628a8b9ab P e77b2d60f57d4dceb865e440eab35126 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } }
I20260709 12:27:36.858095 31054 raft_consensus.cc:385] T b555d1cdbcec4c278fd300d628a8b9ab P e77b2d60f57d4dceb865e440eab35126 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:36.858573 31054 raft_consensus.cc:740] T b555d1cdbcec4c278fd300d628a8b9ab P e77b2d60f57d4dceb865e440eab35126 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e77b2d60f57d4dceb865e440eab35126, State: Initialized, Role: FOLLOWER
I20260709 12:27:36.860208 31054 consensus_queue.cc:260] T b555d1cdbcec4c278fd300d628a8b9ab P e77b2d60f57d4dceb865e440eab35126 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } }
I20260709 12:27:36.864041 31054 ts_tablet_manager.cc:1434] T b555d1cdbcec4c278fd300d628a8b9ab P e77b2d60f57d4dceb865e440eab35126: Time spent starting tablet: real 0.023s	user 0.010s	sys 0.000s
I20260709 12:27:36.870740 31054 tablet_bootstrap.cc:492] T 0be78a9b3c5d47df8efea08d91f58111 P e77b2d60f57d4dceb865e440eab35126: Bootstrap starting.
I20260709 12:27:36.879557 31059 raft_consensus.cc:493] T 2c458afcbd8848d2aa9893f4e74c8561 P 75f56c1a7f86421da9247e9b35990e34 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:36.880285 31059 raft_consensus.cc:515] T 2c458afcbd8848d2aa9893f4e74c8561 P 75f56c1a7f86421da9247e9b35990e34 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } }
I20260709 12:27:36.883069 31059 leader_election.cc:290] T 2c458afcbd8848d2aa9893f4e74c8561 P 75f56c1a7f86421da9247e9b35990e34 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e77b2d60f57d4dceb865e440eab35126 (127.16.29.66:46695), 2fbc4f1ad97a4f2492306d77a614527a (127.16.29.67:44761)
W20260709 12:27:36.883194 30977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:45249, user_credentials={real_user=slave}} blocked reactor thread for 50477.5us
I20260709 12:27:36.902570 31054 tablet_bootstrap.cc:654] T 0be78a9b3c5d47df8efea08d91f58111 P e77b2d60f57d4dceb865e440eab35126: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:36.905896 31052 tablet_bootstrap.cc:492] T 009d38185f3e49afbbd9faf57f968b2c P 75f56c1a7f86421da9247e9b35990e34: No bootstrap required, opened a new log
I20260709 12:27:36.906549 31052 ts_tablet_manager.cc:1403] T 009d38185f3e49afbbd9faf57f968b2c P 75f56c1a7f86421da9247e9b35990e34: Time spent bootstrapping tablet: real 0.068s	user 0.024s	sys 0.010s
I20260709 12:27:36.908525 31053 ts_tablet_manager.cc:1434] T 0be78a9b3c5d47df8efea08d91f58111 P 2fbc4f1ad97a4f2492306d77a614527a: Time spent starting tablet: real 0.150s	user 0.066s	sys 0.081s
I20260709 12:27:36.910234 31053 tablet_bootstrap.cc:492] T de8ccd191af64c50ac32e3342ca995b0 P 2fbc4f1ad97a4f2492306d77a614527a: Bootstrap starting.
I20260709 12:27:36.910310 31052 raft_consensus.cc:359] T 009d38185f3e49afbbd9faf57f968b2c P 75f56c1a7f86421da9247e9b35990e34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } }
I20260709 12:27:36.915599 31012 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c458afcbd8848d2aa9893f4e74c8561" candidate_uuid: "e77b2d60f57d4dceb865e440eab35126" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2fbc4f1ad97a4f2492306d77a614527a" is_pre_election: true
I20260709 12:27:36.917374 31012 raft_consensus.cc:2468] T 2c458afcbd8848d2aa9893f4e74c8561 P 2fbc4f1ad97a4f2492306d77a614527a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e77b2d60f57d4dceb865e440eab35126 in term 0.
I20260709 12:27:36.918512 31053 tablet_bootstrap.cc:654] T de8ccd191af64c50ac32e3342ca995b0 P 2fbc4f1ad97a4f2492306d77a614527a: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:36.918936 30904 leader_election.cc:304] T 2c458afcbd8848d2aa9893f4e74c8561 P e77b2d60f57d4dceb865e440eab35126 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2fbc4f1ad97a4f2492306d77a614527a, e77b2d60f57d4dceb865e440eab35126; no voters: 
I20260709 12:27:36.919893 31060 raft_consensus.cc:2804] T 2c458afcbd8848d2aa9893f4e74c8561 P e77b2d60f57d4dceb865e440eab35126 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:36.920249 31060 raft_consensus.cc:493] T 2c458afcbd8848d2aa9893f4e74c8561 P e77b2d60f57d4dceb865e440eab35126 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:36.920600 31060 raft_consensus.cc:3060] T 2c458afcbd8848d2aa9893f4e74c8561 P e77b2d60f57d4dceb865e440eab35126 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:36.922232 30759 catalog_manager.cc:5697] T b555d1cdbcec4c278fd300d628a8b9ab P 2fbc4f1ad97a4f2492306d77a614527a reported cstate change: term changed from 0 to 1, leader changed from <none> to 2fbc4f1ad97a4f2492306d77a614527a (127.16.29.67). New cstate: current_term: 1 leader_uuid: "2fbc4f1ad97a4f2492306d77a614527a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } health_report { overall_health: HEALTHY } } }
I20260709 12:27:36.926669 31054 tablet_bootstrap.cc:492] T 0be78a9b3c5d47df8efea08d91f58111 P e77b2d60f57d4dceb865e440eab35126: No bootstrap required, opened a new log
I20260709 12:27:36.932720 31054 ts_tablet_manager.cc:1403] T 0be78a9b3c5d47df8efea08d91f58111 P e77b2d60f57d4dceb865e440eab35126: Time spent bootstrapping tablet: real 0.062s	user 0.011s	sys 0.005s
I20260709 12:27:36.929569 31060 raft_consensus.cc:515] T 2c458afcbd8848d2aa9893f4e74c8561 P e77b2d60f57d4dceb865e440eab35126 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } }
I20260709 12:27:36.930389 31053 tablet_bootstrap.cc:492] T de8ccd191af64c50ac32e3342ca995b0 P 2fbc4f1ad97a4f2492306d77a614527a: No bootstrap required, opened a new log
I20260709 12:27:36.934024 31053 ts_tablet_manager.cc:1403] T de8ccd191af64c50ac32e3342ca995b0 P 2fbc4f1ad97a4f2492306d77a614527a: Time spent bootstrapping tablet: real 0.024s	user 0.014s	sys 0.005s
I20260709 12:27:36.936003 30862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c458afcbd8848d2aa9893f4e74c8561" candidate_uuid: "e77b2d60f57d4dceb865e440eab35126" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75f56c1a7f86421da9247e9b35990e34"
I20260709 12:27:36.936139 31012 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c458afcbd8848d2aa9893f4e74c8561" candidate_uuid: "e77b2d60f57d4dceb865e440eab35126" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2fbc4f1ad97a4f2492306d77a614527a"
I20260709 12:27:36.936682 30862 raft_consensus.cc:3060] T 2c458afcbd8848d2aa9893f4e74c8561 P 75f56c1a7f86421da9247e9b35990e34 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:36.936800 31012 raft_consensus.cc:3060] T 2c458afcbd8848d2aa9893f4e74c8561 P 2fbc4f1ad97a4f2492306d77a614527a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:36.940191 31060 leader_election.cc:290] T 2c458afcbd8848d2aa9893f4e74c8561 P e77b2d60f57d4dceb865e440eab35126 [CANDIDATE]: Term 1 election: Requested vote from peers 75f56c1a7f86421da9247e9b35990e34 (127.16.29.65:45249), 2fbc4f1ad97a4f2492306d77a614527a (127.16.29.67:44761)
I20260709 12:27:36.931958 30863 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c458afcbd8848d2aa9893f4e74c8561" candidate_uuid: "e77b2d60f57d4dceb865e440eab35126" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75f56c1a7f86421da9247e9b35990e34" is_pre_election: true
I20260709 12:27:36.941228 31057 raft_consensus.cc:493] T 0be78a9b3c5d47df8efea08d91f58111 P 2fbc4f1ad97a4f2492306d77a614527a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:36.942421 31057 raft_consensus.cc:515] T 0be78a9b3c5d47df8efea08d91f58111 P 2fbc4f1ad97a4f2492306d77a614527a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } }
I20260709 12:27:36.944406 30862 raft_consensus.cc:2468] T 2c458afcbd8848d2aa9893f4e74c8561 P 75f56c1a7f86421da9247e9b35990e34 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e77b2d60f57d4dceb865e440eab35126 in term 1.
I20260709 12:27:36.945039 31053 raft_consensus.cc:359] T de8ccd191af64c50ac32e3342ca995b0 P 2fbc4f1ad97a4f2492306d77a614527a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } }
I20260709 12:27:36.945819 31053 raft_consensus.cc:385] T de8ccd191af64c50ac32e3342ca995b0 P 2fbc4f1ad97a4f2492306d77a614527a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:36.945832 30903 leader_election.cc:304] T 2c458afcbd8848d2aa9893f4e74c8561 P e77b2d60f57d4dceb865e440eab35126 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 75f56c1a7f86421da9247e9b35990e34, e77b2d60f57d4dceb865e440eab35126; no voters: 
I20260709 12:27:36.946233 31053 raft_consensus.cc:740] T de8ccd191af64c50ac32e3342ca995b0 P 2fbc4f1ad97a4f2492306d77a614527a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fbc4f1ad97a4f2492306d77a614527a, State: Initialized, Role: FOLLOWER
I20260709 12:27:36.947077 31053 consensus_queue.cc:260] T de8ccd191af64c50ac32e3342ca995b0 P 2fbc4f1ad97a4f2492306d77a614527a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } }
I20260709 12:27:36.947588 30863 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0be78a9b3c5d47df8efea08d91f58111" candidate_uuid: "2fbc4f1ad97a4f2492306d77a614527a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75f56c1a7f86421da9247e9b35990e34" is_pre_election: true
I20260709 12:27:36.948415 31057 leader_election.cc:290] T 0be78a9b3c5d47df8efea08d91f58111 P 2fbc4f1ad97a4f2492306d77a614527a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 75f56c1a7f86421da9247e9b35990e34 (127.16.29.65:45249), e77b2d60f57d4dceb865e440eab35126 (127.16.29.66:46695)
W20260709 12:27:36.949402 30977 leader_election.cc:343] T 0be78a9b3c5d47df8efea08d91f58111 P 2fbc4f1ad97a4f2492306d77a614527a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 75f56c1a7f86421da9247e9b35990e34 (127.16.29.65:45249): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:36.947781 30938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0be78a9b3c5d47df8efea08d91f58111" candidate_uuid: "2fbc4f1ad97a4f2492306d77a614527a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e77b2d60f57d4dceb865e440eab35126" is_pre_election: true
I20260709 12:27:36.946009 31011 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c458afcbd8848d2aa9893f4e74c8561" candidate_uuid: "75f56c1a7f86421da9247e9b35990e34" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2fbc4f1ad97a4f2492306d77a614527a" is_pre_election: true
I20260709 12:27:36.949410 31053 ts_tablet_manager.cc:1434] T de8ccd191af64c50ac32e3342ca995b0 P 2fbc4f1ad97a4f2492306d77a614527a: Time spent starting tablet: real 0.015s	user 0.006s	sys 0.000s
I20260709 12:27:36.949893 31060 raft_consensus.cc:2804] T 2c458afcbd8848d2aa9893f4e74c8561 P e77b2d60f57d4dceb865e440eab35126 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:36.943570 31054 raft_consensus.cc:359] T 0be78a9b3c5d47df8efea08d91f58111 P e77b2d60f57d4dceb865e440eab35126 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } }
I20260709 12:27:36.953053 31054 raft_consensus.cc:385] T 0be78a9b3c5d47df8efea08d91f58111 P e77b2d60f57d4dceb865e440eab35126 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:36.953425 31054 raft_consensus.cc:740] T 0be78a9b3c5d47df8efea08d91f58111 P e77b2d60f57d4dceb865e440eab35126 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e77b2d60f57d4dceb865e440eab35126, State: Initialized, Role: FOLLOWER
I20260709 12:27:36.931715 31052 raft_consensus.cc:385] T 009d38185f3e49afbbd9faf57f968b2c P 75f56c1a7f86421da9247e9b35990e34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:36.954232 31054 consensus_queue.cc:260] T 0be78a9b3c5d47df8efea08d91f58111 P e77b2d60f57d4dceb865e440eab35126 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } }
I20260709 12:27:36.954782 31052 raft_consensus.cc:740] T 009d38185f3e49afbbd9faf57f968b2c P 75f56c1a7f86421da9247e9b35990e34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75f56c1a7f86421da9247e9b35990e34, State: Initialized, Role: FOLLOWER
I20260709 12:27:36.955749 31052 consensus_queue.cc:260] T 009d38185f3e49afbbd9faf57f968b2c P 75f56c1a7f86421da9247e9b35990e34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } }
I20260709 12:27:36.958392 31012 raft_consensus.cc:2468] T 2c458afcbd8848d2aa9893f4e74c8561 P 2fbc4f1ad97a4f2492306d77a614527a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e77b2d60f57d4dceb865e440eab35126 in term 1.
I20260709 12:27:36.960866 31052 ts_tablet_manager.cc:1434] T 009d38185f3e49afbbd9faf57f968b2c P 75f56c1a7f86421da9247e9b35990e34: Time spent starting tablet: real 0.053s	user 0.006s	sys 0.001s
I20260709 12:27:36.962042 31052 tablet_bootstrap.cc:492] T de8ccd191af64c50ac32e3342ca995b0 P 75f56c1a7f86421da9247e9b35990e34: Bootstrap starting.
I20260709 12:27:36.960530 30937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c458afcbd8848d2aa9893f4e74c8561" candidate_uuid: "75f56c1a7f86421da9247e9b35990e34" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e77b2d60f57d4dceb865e440eab35126" is_pre_election: true
I20260709 12:27:36.955204 30938 raft_consensus.cc:2468] T 0be78a9b3c5d47df8efea08d91f58111 P e77b2d60f57d4dceb865e440eab35126 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2fbc4f1ad97a4f2492306d77a614527a in term 0.
I20260709 12:27:36.970775 31052 tablet_bootstrap.cc:654] T de8ccd191af64c50ac32e3342ca995b0 P 75f56c1a7f86421da9247e9b35990e34: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:36.970639 30976 leader_election.cc:304] T 0be78a9b3c5d47df8efea08d91f58111 P 2fbc4f1ad97a4f2492306d77a614527a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2fbc4f1ad97a4f2492306d77a614527a, e77b2d60f57d4dceb865e440eab35126; no voters: 75f56c1a7f86421da9247e9b35990e34
I20260709 12:27:36.972615 31057 raft_consensus.cc:2804] T 0be78a9b3c5d47df8efea08d91f58111 P 2fbc4f1ad97a4f2492306d77a614527a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:36.973021 31057 raft_consensus.cc:493] T 0be78a9b3c5d47df8efea08d91f58111 P 2fbc4f1ad97a4f2492306d77a614527a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:36.973373 31057 raft_consensus.cc:3060] T 0be78a9b3c5d47df8efea08d91f58111 P 2fbc4f1ad97a4f2492306d77a614527a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:36.975114 31060 raft_consensus.cc:697] T 2c458afcbd8848d2aa9893f4e74c8561 P e77b2d60f57d4dceb865e440eab35126 [term 1 LEADER]: Becoming Leader. State: Replica: e77b2d60f57d4dceb865e440eab35126, State: Running, Role: LEADER
I20260709 12:27:36.976078 31060 consensus_queue.cc:237] T 2c458afcbd8848d2aa9893f4e74c8561 P e77b2d60f57d4dceb865e440eab35126 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } }
I20260709 12:27:36.981283 31057 raft_consensus.cc:515] T 0be78a9b3c5d47df8efea08d91f58111 P 2fbc4f1ad97a4f2492306d77a614527a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } }
I20260709 12:27:36.981947 31052 tablet_bootstrap.cc:492] T de8ccd191af64c50ac32e3342ca995b0 P 75f56c1a7f86421da9247e9b35990e34: No bootstrap required, opened a new log
I20260709 12:27:36.982517 31052 ts_tablet_manager.cc:1403] T de8ccd191af64c50ac32e3342ca995b0 P 75f56c1a7f86421da9247e9b35990e34: Time spent bootstrapping tablet: real 0.021s	user 0.009s	sys 0.003s
I20260709 12:27:36.985653 31052 raft_consensus.cc:359] T de8ccd191af64c50ac32e3342ca995b0 P 75f56c1a7f86421da9247e9b35990e34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } }
I20260709 12:27:36.986423 31052 raft_consensus.cc:385] T de8ccd191af64c50ac32e3342ca995b0 P 75f56c1a7f86421da9247e9b35990e34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:36.987591 31053 tablet_bootstrap.cc:492] T 8e560587c5c24e4587295d2d1b6d2be6 P 2fbc4f1ad97a4f2492306d77a614527a: Bootstrap starting.
I20260709 12:27:36.989039 30827 leader_election.cc:304] T 2c458afcbd8848d2aa9893f4e74c8561 P 75f56c1a7f86421da9247e9b35990e34 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 75f56c1a7f86421da9247e9b35990e34; no voters: 2fbc4f1ad97a4f2492306d77a614527a, e77b2d60f57d4dceb865e440eab35126
I20260709 12:27:36.986737 31052 raft_consensus.cc:740] T de8ccd191af64c50ac32e3342ca995b0 P 75f56c1a7f86421da9247e9b35990e34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75f56c1a7f86421da9247e9b35990e34, State: Initialized, Role: FOLLOWER
I20260709 12:27:36.990432 31059 raft_consensus.cc:2749] T 2c458afcbd8848d2aa9893f4e74c8561 P 75f56c1a7f86421da9247e9b35990e34 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:36.991110 31052 consensus_queue.cc:260] T de8ccd191af64c50ac32e3342ca995b0 P 75f56c1a7f86421da9247e9b35990e34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } }
I20260709 12:27:36.993317 31052 ts_tablet_manager.cc:1434] T de8ccd191af64c50ac32e3342ca995b0 P 75f56c1a7f86421da9247e9b35990e34: Time spent starting tablet: real 0.010s	user 0.006s	sys 0.000s
I20260709 12:27:36.994302 31052 tablet_bootstrap.cc:492] T 3ed242e2c2b843978a7af7cca5694ae2 P 75f56c1a7f86421da9247e9b35990e34: Bootstrap starting.
I20260709 12:27:36.994690 31053 tablet_bootstrap.cc:654] T 8e560587c5c24e4587295d2d1b6d2be6 P 2fbc4f1ad97a4f2492306d77a614527a: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:36.986682 31054 ts_tablet_manager.cc:1434] T 0be78a9b3c5d47df8efea08d91f58111 P e77b2d60f57d4dceb865e440eab35126: Time spent starting tablet: real 0.054s	user 0.005s	sys 0.001s
I20260709 12:27:36.996996 31054 tablet_bootstrap.cc:492] T de8ccd191af64c50ac32e3342ca995b0 P e77b2d60f57d4dceb865e440eab35126: Bootstrap starting.
I20260709 12:27:37.001233 31057 leader_election.cc:290] T 0be78a9b3c5d47df8efea08d91f58111 P 2fbc4f1ad97a4f2492306d77a614527a [CANDIDATE]: Term 1 election: Requested vote from peers 75f56c1a7f86421da9247e9b35990e34 (127.16.29.65:45249), e77b2d60f57d4dceb865e440eab35126 (127.16.29.66:46695)
I20260709 12:27:37.002794 31052 tablet_bootstrap.cc:654] T 3ed242e2c2b843978a7af7cca5694ae2 P 75f56c1a7f86421da9247e9b35990e34: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:37.002879 30937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0be78a9b3c5d47df8efea08d91f58111" candidate_uuid: "2fbc4f1ad97a4f2492306d77a614527a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e77b2d60f57d4dceb865e440eab35126"
I20260709 12:27:37.003702 31054 tablet_bootstrap.cc:654] T de8ccd191af64c50ac32e3342ca995b0 P e77b2d60f57d4dceb865e440eab35126: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:37.003827 30937 raft_consensus.cc:3060] T 0be78a9b3c5d47df8efea08d91f58111 P e77b2d60f57d4dceb865e440eab35126 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:37.005054 30862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0be78a9b3c5d47df8efea08d91f58111" candidate_uuid: "2fbc4f1ad97a4f2492306d77a614527a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75f56c1a7f86421da9247e9b35990e34"
I20260709 12:27:36.991698 30760 catalog_manager.cc:5697] T 2c458afcbd8848d2aa9893f4e74c8561 P e77b2d60f57d4dceb865e440eab35126 reported cstate change: term changed from 0 to 1, leader changed from <none> to e77b2d60f57d4dceb865e440eab35126 (127.16.29.66). New cstate: current_term: 1 leader_uuid: "e77b2d60f57d4dceb865e440eab35126" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } health_report { overall_health: UNKNOWN } } }
W20260709 12:27:37.006497 30977 leader_election.cc:343] T 0be78a9b3c5d47df8efea08d91f58111 P 2fbc4f1ad97a4f2492306d77a614527a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 75f56c1a7f86421da9247e9b35990e34 (127.16.29.65:45249): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:37.009371 31060 raft_consensus.cc:493] T b555d1cdbcec4c278fd300d628a8b9ab P e77b2d60f57d4dceb865e440eab35126 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:37.012969 31060 raft_consensus.cc:515] T b555d1cdbcec4c278fd300d628a8b9ab P e77b2d60f57d4dceb865e440eab35126 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } }
I20260709 12:27:37.016932 30937 raft_consensus.cc:2468] T 0be78a9b3c5d47df8efea08d91f58111 P e77b2d60f57d4dceb865e440eab35126 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2fbc4f1ad97a4f2492306d77a614527a in term 1.
I20260709 12:27:37.018859 30862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b555d1cdbcec4c278fd300d628a8b9ab" candidate_uuid: "e77b2d60f57d4dceb865e440eab35126" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75f56c1a7f86421da9247e9b35990e34" is_pre_election: true
I20260709 12:27:37.019786 30862 raft_consensus.cc:2393] T b555d1cdbcec4c278fd300d628a8b9ab P 75f56c1a7f86421da9247e9b35990e34 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e77b2d60f57d4dceb865e440eab35126 in current term 1: Already voted for candidate 2fbc4f1ad97a4f2492306d77a614527a in this term.
I20260709 12:27:37.023283 31012 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b555d1cdbcec4c278fd300d628a8b9ab" candidate_uuid: "e77b2d60f57d4dceb865e440eab35126" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2fbc4f1ad97a4f2492306d77a614527a" is_pre_election: true
I20260709 12:27:37.024737 31053 tablet_bootstrap.cc:492] T 8e560587c5c24e4587295d2d1b6d2be6 P 2fbc4f1ad97a4f2492306d77a614527a: No bootstrap required, opened a new log
I20260709 12:27:37.024998 30976 leader_election.cc:304] T 0be78a9b3c5d47df8efea08d91f58111 P 2fbc4f1ad97a4f2492306d77a614527a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2fbc4f1ad97a4f2492306d77a614527a, e77b2d60f57d4dceb865e440eab35126; no voters: 75f56c1a7f86421da9247e9b35990e34
I20260709 12:27:37.025307 31053 ts_tablet_manager.cc:1403] T 8e560587c5c24e4587295d2d1b6d2be6 P 2fbc4f1ad97a4f2492306d77a614527a: Time spent bootstrapping tablet: real 0.038s	user 0.010s	sys 0.005s
I20260709 12:27:37.025082 30904 leader_election.cc:304] T b555d1cdbcec4c278fd300d628a8b9ab P e77b2d60f57d4dceb865e440eab35126 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e77b2d60f57d4dceb865e440eab35126; no voters: 2fbc4f1ad97a4f2492306d77a614527a, 75f56c1a7f86421da9247e9b35990e34
I20260709 12:27:37.025679 31060 leader_election.cc:290] T b555d1cdbcec4c278fd300d628a8b9ab P e77b2d60f57d4dceb865e440eab35126 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 75f56c1a7f86421da9247e9b35990e34 (127.16.29.65:45249), 2fbc4f1ad97a4f2492306d77a614527a (127.16.29.67:44761)
I20260709 12:27:37.026392 31083 raft_consensus.cc:3060] T b555d1cdbcec4c278fd300d628a8b9ab P e77b2d60f57d4dceb865e440eab35126 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:37.026502 31057 raft_consensus.cc:2804] T 0be78a9b3c5d47df8efea08d91f58111 P 2fbc4f1ad97a4f2492306d77a614527a [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:37.027186 31057 raft_consensus.cc:697] T 0be78a9b3c5d47df8efea08d91f58111 P 2fbc4f1ad97a4f2492306d77a614527a [term 1 LEADER]: Becoming Leader. State: Replica: 2fbc4f1ad97a4f2492306d77a614527a, State: Running, Role: LEADER
I20260709 12:27:37.028151 31057 consensus_queue.cc:237] T 0be78a9b3c5d47df8efea08d91f58111 P 2fbc4f1ad97a4f2492306d77a614527a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } }
I20260709 12:27:37.028945 31054 tablet_bootstrap.cc:492] T de8ccd191af64c50ac32e3342ca995b0 P e77b2d60f57d4dceb865e440eab35126: No bootstrap required, opened a new log
I20260709 12:27:37.029497 31054 ts_tablet_manager.cc:1403] T de8ccd191af64c50ac32e3342ca995b0 P e77b2d60f57d4dceb865e440eab35126: Time spent bootstrapping tablet: real 0.033s	user 0.005s	sys 0.009s
I20260709 12:27:37.029363 31053 raft_consensus.cc:359] T 8e560587c5c24e4587295d2d1b6d2be6 P 2fbc4f1ad97a4f2492306d77a614527a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } }
I20260709 12:27:37.030314 31053 raft_consensus.cc:385] T 8e560587c5c24e4587295d2d1b6d2be6 P 2fbc4f1ad97a4f2492306d77a614527a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:37.030678 31053 raft_consensus.cc:740] T 8e560587c5c24e4587295d2d1b6d2be6 P 2fbc4f1ad97a4f2492306d77a614527a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fbc4f1ad97a4f2492306d77a614527a, State: Initialized, Role: FOLLOWER
I20260709 12:27:37.031459 31053 consensus_queue.cc:260] T 8e560587c5c24e4587295d2d1b6d2be6 P 2fbc4f1ad97a4f2492306d77a614527a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } }
I20260709 12:27:37.032720 31054 raft_consensus.cc:359] T de8ccd191af64c50ac32e3342ca995b0 P e77b2d60f57d4dceb865e440eab35126 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } }
I20260709 12:27:37.033514 31054 raft_consensus.cc:385] T de8ccd191af64c50ac32e3342ca995b0 P e77b2d60f57d4dceb865e440eab35126 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:37.033867 31054 raft_consensus.cc:740] T de8ccd191af64c50ac32e3342ca995b0 P e77b2d60f57d4dceb865e440eab35126 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e77b2d60f57d4dceb865e440eab35126, State: Initialized, Role: FOLLOWER
I20260709 12:27:37.034039 31053 ts_tablet_manager.cc:1434] T 8e560587c5c24e4587295d2d1b6d2be6 P 2fbc4f1ad97a4f2492306d77a614527a: Time spent starting tablet: real 0.008s	user 0.006s	sys 0.000s
I20260709 12:27:37.034754 31083 raft_consensus.cc:2749] T b555d1cdbcec4c278fd300d628a8b9ab P e77b2d60f57d4dceb865e440eab35126 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:37.037040 31052 tablet_bootstrap.cc:492] T 3ed242e2c2b843978a7af7cca5694ae2 P 75f56c1a7f86421da9247e9b35990e34: No bootstrap required, opened a new log
I20260709 12:27:37.038170 31052 ts_tablet_manager.cc:1403] T 3ed242e2c2b843978a7af7cca5694ae2 P 75f56c1a7f86421da9247e9b35990e34: Time spent bootstrapping tablet: real 0.044s	user 0.018s	sys 0.007s
I20260709 12:27:37.038745 31054 consensus_queue.cc:260] T de8ccd191af64c50ac32e3342ca995b0 P e77b2d60f57d4dceb865e440eab35126 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } }
I20260709 12:27:37.038221 31053 tablet_bootstrap.cc:492] T 009d38185f3e49afbbd9faf57f968b2c P 2fbc4f1ad97a4f2492306d77a614527a: Bootstrap starting.
I20260709 12:27:37.041282 31054 ts_tablet_manager.cc:1434] T de8ccd191af64c50ac32e3342ca995b0 P e77b2d60f57d4dceb865e440eab35126: Time spent starting tablet: real 0.011s	user 0.006s	sys 0.002s
I20260709 12:27:37.042474 31054 tablet_bootstrap.cc:492] T 3ed242e2c2b843978a7af7cca5694ae2 P e77b2d60f57d4dceb865e440eab35126: Bootstrap starting.
I20260709 12:27:37.042261 30759 catalog_manager.cc:5697] T 0be78a9b3c5d47df8efea08d91f58111 P 2fbc4f1ad97a4f2492306d77a614527a reported cstate change: term changed from 0 to 1, leader changed from <none> to 2fbc4f1ad97a4f2492306d77a614527a (127.16.29.67). New cstate: current_term: 1 leader_uuid: "2fbc4f1ad97a4f2492306d77a614527a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } health_report { overall_health: HEALTHY } } }
I20260709 12:27:37.046748 31083 raft_consensus.cc:493] T de8ccd191af64c50ac32e3342ca995b0 P e77b2d60f57d4dceb865e440eab35126 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:37.047365 31083 raft_consensus.cc:515] T de8ccd191af64c50ac32e3342ca995b0 P e77b2d60f57d4dceb865e440eab35126 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } }
I20260709 12:27:37.048192 31052 raft_consensus.cc:359] T 3ed242e2c2b843978a7af7cca5694ae2 P 75f56c1a7f86421da9247e9b35990e34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } }
I20260709 12:27:37.049026 31052 raft_consensus.cc:385] T 3ed242e2c2b843978a7af7cca5694ae2 P 75f56c1a7f86421da9247e9b35990e34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:37.049484 31052 raft_consensus.cc:740] T 3ed242e2c2b843978a7af7cca5694ae2 P 75f56c1a7f86421da9247e9b35990e34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75f56c1a7f86421da9247e9b35990e34, State: Initialized, Role: FOLLOWER
I20260709 12:27:37.050356 30862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de8ccd191af64c50ac32e3342ca995b0" candidate_uuid: "e77b2d60f57d4dceb865e440eab35126" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75f56c1a7f86421da9247e9b35990e34" is_pre_election: true
I20260709 12:27:37.051198 31054 tablet_bootstrap.cc:654] T 3ed242e2c2b843978a7af7cca5694ae2 P e77b2d60f57d4dceb865e440eab35126: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:37.051157 30862 raft_consensus.cc:2468] T de8ccd191af64c50ac32e3342ca995b0 P 75f56c1a7f86421da9247e9b35990e34 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e77b2d60f57d4dceb865e440eab35126 in term 0.
I20260709 12:27:37.050475 31052 consensus_queue.cc:260] T 3ed242e2c2b843978a7af7cca5694ae2 P 75f56c1a7f86421da9247e9b35990e34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } }
I20260709 12:27:37.052771 30903 leader_election.cc:304] T de8ccd191af64c50ac32e3342ca995b0 P e77b2d60f57d4dceb865e440eab35126 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 75f56c1a7f86421da9247e9b35990e34, e77b2d60f57d4dceb865e440eab35126; no voters: 
I20260709 12:27:37.054651 31052 ts_tablet_manager.cc:1434] T 3ed242e2c2b843978a7af7cca5694ae2 P 75f56c1a7f86421da9247e9b35990e34: Time spent starting tablet: real 0.016s	user 0.006s	sys 0.000s
I20260709 12:27:37.051821 31053 tablet_bootstrap.cc:654] T 009d38185f3e49afbbd9faf57f968b2c P 2fbc4f1ad97a4f2492306d77a614527a: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:37.056066 31060 raft_consensus.cc:2804] T de8ccd191af64c50ac32e3342ca995b0 P e77b2d60f57d4dceb865e440eab35126 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:37.056478 31060 raft_consensus.cc:493] T de8ccd191af64c50ac32e3342ca995b0 P e77b2d60f57d4dceb865e440eab35126 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:37.056813 31060 raft_consensus.cc:3060] T de8ccd191af64c50ac32e3342ca995b0 P e77b2d60f57d4dceb865e440eab35126 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:37.060182 31052 tablet_bootstrap.cc:492] T 0be78a9b3c5d47df8efea08d91f58111 P 75f56c1a7f86421da9247e9b35990e34: Bootstrap starting.
I20260709 12:27:37.060241 31012 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de8ccd191af64c50ac32e3342ca995b0" candidate_uuid: "e77b2d60f57d4dceb865e440eab35126" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2fbc4f1ad97a4f2492306d77a614527a" is_pre_election: true
I20260709 12:27:37.061141 31012 raft_consensus.cc:2468] T de8ccd191af64c50ac32e3342ca995b0 P 2fbc4f1ad97a4f2492306d77a614527a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e77b2d60f57d4dceb865e440eab35126 in term 0.
I20260709 12:27:37.068090 31054 tablet_bootstrap.cc:492] T 3ed242e2c2b843978a7af7cca5694ae2 P e77b2d60f57d4dceb865e440eab35126: No bootstrap required, opened a new log
I20260709 12:27:37.068177 31052 tablet_bootstrap.cc:654] T 0be78a9b3c5d47df8efea08d91f58111 P 75f56c1a7f86421da9247e9b35990e34: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:37.068926 31054 ts_tablet_manager.cc:1403] T 3ed242e2c2b843978a7af7cca5694ae2 P e77b2d60f57d4dceb865e440eab35126: Time spent bootstrapping tablet: real 0.027s	user 0.010s	sys 0.010s
I20260709 12:27:37.069294 31053 tablet_bootstrap.cc:492] T 009d38185f3e49afbbd9faf57f968b2c P 2fbc4f1ad97a4f2492306d77a614527a: No bootstrap required, opened a new log
I20260709 12:27:37.069962 31053 ts_tablet_manager.cc:1403] T 009d38185f3e49afbbd9faf57f968b2c P 2fbc4f1ad97a4f2492306d77a614527a: Time spent bootstrapping tablet: real 0.032s	user 0.004s	sys 0.011s
I20260709 12:27:37.072171 31054 raft_consensus.cc:359] T 3ed242e2c2b843978a7af7cca5694ae2 P e77b2d60f57d4dceb865e440eab35126 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } }
I20260709 12:27:37.073007 31054 raft_consensus.cc:385] T 3ed242e2c2b843978a7af7cca5694ae2 P e77b2d60f57d4dceb865e440eab35126 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:37.073347 31054 raft_consensus.cc:740] T 3ed242e2c2b843978a7af7cca5694ae2 P e77b2d60f57d4dceb865e440eab35126 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e77b2d60f57d4dceb865e440eab35126, State: Initialized, Role: FOLLOWER
I20260709 12:27:37.073560 31053 raft_consensus.cc:359] T 009d38185f3e49afbbd9faf57f968b2c P 2fbc4f1ad97a4f2492306d77a614527a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } }
I20260709 12:27:37.074172 31054 consensus_queue.cc:260] T 3ed242e2c2b843978a7af7cca5694ae2 P e77b2d60f57d4dceb865e440eab35126 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } }
I20260709 12:27:37.074679 31053 raft_consensus.cc:385] T 009d38185f3e49afbbd9faf57f968b2c P 2fbc4f1ad97a4f2492306d77a614527a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:37.077428 31053 raft_consensus.cc:740] T 009d38185f3e49afbbd9faf57f968b2c P 2fbc4f1ad97a4f2492306d77a614527a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fbc4f1ad97a4f2492306d77a614527a, State: Initialized, Role: FOLLOWER
I20260709 12:27:37.077831 31052 tablet_bootstrap.cc:492] T 0be78a9b3c5d47df8efea08d91f58111 P 75f56c1a7f86421da9247e9b35990e34: No bootstrap required, opened a new log
I20260709 12:27:37.078518 31052 ts_tablet_manager.cc:1403] T 0be78a9b3c5d47df8efea08d91f58111 P 75f56c1a7f86421da9247e9b35990e34: Time spent bootstrapping tablet: real 0.019s	user 0.013s	sys 0.004s
I20260709 12:27:37.078343 31053 consensus_queue.cc:260] T 009d38185f3e49afbbd9faf57f968b2c P 2fbc4f1ad97a4f2492306d77a614527a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } }
I20260709 12:27:37.080093 31054 ts_tablet_manager.cc:1434] T 3ed242e2c2b843978a7af7cca5694ae2 P e77b2d60f57d4dceb865e440eab35126: Time spent starting tablet: real 0.011s	user 0.007s	sys 0.000s
I20260709 12:27:37.080684 31083 leader_election.cc:290] T de8ccd191af64c50ac32e3342ca995b0 P e77b2d60f57d4dceb865e440eab35126 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 75f56c1a7f86421da9247e9b35990e34 (127.16.29.65:45249), 2fbc4f1ad97a4f2492306d77a614527a (127.16.29.67:44761)
I20260709 12:27:37.081175 31054 tablet_bootstrap.cc:492] T f64692cd6a0347979834f29861bde5d4 P e77b2d60f57d4dceb865e440eab35126: Bootstrap starting.
I20260709 12:27:37.076766 31060 raft_consensus.cc:515] T de8ccd191af64c50ac32e3342ca995b0 P e77b2d60f57d4dceb865e440eab35126 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } }
I20260709 12:27:37.081600 31052 raft_consensus.cc:359] T 0be78a9b3c5d47df8efea08d91f58111 P 75f56c1a7f86421da9247e9b35990e34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } }
I20260709 12:27:37.082353 31052 raft_consensus.cc:385] T 0be78a9b3c5d47df8efea08d91f58111 P 75f56c1a7f86421da9247e9b35990e34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:37.082671 31052 raft_consensus.cc:740] T 0be78a9b3c5d47df8efea08d91f58111 P 75f56c1a7f86421da9247e9b35990e34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75f56c1a7f86421da9247e9b35990e34, State: Initialized, Role: FOLLOWER
I20260709 12:27:37.083531 31052 consensus_queue.cc:260] T 0be78a9b3c5d47df8efea08d91f58111 P 75f56c1a7f86421da9247e9b35990e34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } }
I20260709 12:27:37.084630 30862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de8ccd191af64c50ac32e3342ca995b0" candidate_uuid: "e77b2d60f57d4dceb865e440eab35126" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75f56c1a7f86421da9247e9b35990e34"
I20260709 12:27:37.085304 30862 raft_consensus.cc:3060] T de8ccd191af64c50ac32e3342ca995b0 P 75f56c1a7f86421da9247e9b35990e34 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:37.085644 31052 ts_tablet_manager.cc:1434] T 0be78a9b3c5d47df8efea08d91f58111 P 75f56c1a7f86421da9247e9b35990e34: Time spent starting tablet: real 0.007s	user 0.005s	sys 0.000s
I20260709 12:27:37.086668 31052 tablet_bootstrap.cc:492] T adedadb9002047ba87a533a33374f4e7 P 75f56c1a7f86421da9247e9b35990e34: Bootstrap starting.
I20260709 12:27:37.088234 31054 tablet_bootstrap.cc:654] T f64692cd6a0347979834f29861bde5d4 P e77b2d60f57d4dceb865e440eab35126: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:37.092471 31052 tablet_bootstrap.cc:654] T adedadb9002047ba87a533a33374f4e7 P 75f56c1a7f86421da9247e9b35990e34: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:37.101329 30862 raft_consensus.cc:2468] T de8ccd191af64c50ac32e3342ca995b0 P 75f56c1a7f86421da9247e9b35990e34 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e77b2d60f57d4dceb865e440eab35126 in term 1.
I20260709 12:27:37.103408 30903 leader_election.cc:304] T de8ccd191af64c50ac32e3342ca995b0 P e77b2d60f57d4dceb865e440eab35126 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 75f56c1a7f86421da9247e9b35990e34, e77b2d60f57d4dceb865e440eab35126; no voters: 
I20260709 12:27:37.104257 31052 tablet_bootstrap.cc:492] T adedadb9002047ba87a533a33374f4e7 P 75f56c1a7f86421da9247e9b35990e34: No bootstrap required, opened a new log
I20260709 12:27:37.104887 31052 ts_tablet_manager.cc:1403] T adedadb9002047ba87a533a33374f4e7 P 75f56c1a7f86421da9247e9b35990e34: Time spent bootstrapping tablet: real 0.018s	user 0.014s	sys 0.003s
I20260709 12:27:37.107856 31052 raft_consensus.cc:359] T adedadb9002047ba87a533a33374f4e7 P 75f56c1a7f86421da9247e9b35990e34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } }
I20260709 12:27:37.108119 31012 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de8ccd191af64c50ac32e3342ca995b0" candidate_uuid: "e77b2d60f57d4dceb865e440eab35126" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2fbc4f1ad97a4f2492306d77a614527a"
I20260709 12:27:37.108723 31052 raft_consensus.cc:385] T adedadb9002047ba87a533a33374f4e7 P 75f56c1a7f86421da9247e9b35990e34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:37.109027 31012 raft_consensus.cc:3060] T de8ccd191af64c50ac32e3342ca995b0 P 2fbc4f1ad97a4f2492306d77a614527a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:37.104681 31083 raft_consensus.cc:2804] T de8ccd191af64c50ac32e3342ca995b0 P e77b2d60f57d4dceb865e440eab35126 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:37.109061 31052 raft_consensus.cc:740] T adedadb9002047ba87a533a33374f4e7 P 75f56c1a7f86421da9247e9b35990e34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75f56c1a7f86421da9247e9b35990e34, State: Initialized, Role: FOLLOWER
I20260709 12:27:37.115933 31060 leader_election.cc:290] T de8ccd191af64c50ac32e3342ca995b0 P e77b2d60f57d4dceb865e440eab35126 [CANDIDATE]: Term 1 election: Requested vote from peers 75f56c1a7f86421da9247e9b35990e34 (127.16.29.65:45249), 2fbc4f1ad97a4f2492306d77a614527a (127.16.29.67:44761)
I20260709 12:27:37.116750 31052 consensus_queue.cc:260] T adedadb9002047ba87a533a33374f4e7 P 75f56c1a7f86421da9247e9b35990e34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } }
I20260709 12:27:37.116735 31083 raft_consensus.cc:697] T de8ccd191af64c50ac32e3342ca995b0 P e77b2d60f57d4dceb865e440eab35126 [term 1 LEADER]: Becoming Leader. State: Replica: e77b2d60f57d4dceb865e440eab35126, State: Running, Role: LEADER
I20260709 12:27:37.117964 31012 raft_consensus.cc:2468] T de8ccd191af64c50ac32e3342ca995b0 P 2fbc4f1ad97a4f2492306d77a614527a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e77b2d60f57d4dceb865e440eab35126 in term 1.
I20260709 12:27:37.118922 31052 ts_tablet_manager.cc:1434] T adedadb9002047ba87a533a33374f4e7 P 75f56c1a7f86421da9247e9b35990e34: Time spent starting tablet: real 0.014s	user 0.002s	sys 0.002s
I20260709 12:27:37.120004 31052 tablet_bootstrap.cc:492] T 881773ceed164ddf8ebc390254b1eb01 P 75f56c1a7f86421da9247e9b35990e34: Bootstrap starting.
I20260709 12:27:37.120056 31053 ts_tablet_manager.cc:1434] T 009d38185f3e49afbbd9faf57f968b2c P 2fbc4f1ad97a4f2492306d77a614527a: Time spent starting tablet: real 0.050s	user 0.002s	sys 0.007s
I20260709 12:27:37.121101 31053 tablet_bootstrap.cc:492] T f64692cd6a0347979834f29861bde5d4 P 2fbc4f1ad97a4f2492306d77a614527a: Bootstrap starting.
I20260709 12:27:37.118389 31083 consensus_queue.cc:237] T de8ccd191af64c50ac32e3342ca995b0 P e77b2d60f57d4dceb865e440eab35126 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } }
I20260709 12:27:37.126783 31059 raft_consensus.cc:493] T f64692cd6a0347979834f29861bde5d4 P 75f56c1a7f86421da9247e9b35990e34 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:37.127347 31059 raft_consensus.cc:515] T f64692cd6a0347979834f29861bde5d4 P 75f56c1a7f86421da9247e9b35990e34 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } }
I20260709 12:27:37.128553 31053 tablet_bootstrap.cc:654] T f64692cd6a0347979834f29861bde5d4 P 2fbc4f1ad97a4f2492306d77a614527a: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:37.129709 31059 leader_election.cc:290] T f64692cd6a0347979834f29861bde5d4 P 75f56c1a7f86421da9247e9b35990e34 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e77b2d60f57d4dceb865e440eab35126 (127.16.29.66:46695), 2fbc4f1ad97a4f2492306d77a614527a (127.16.29.67:44761)
I20260709 12:27:37.130287 31052 tablet_bootstrap.cc:654] T 881773ceed164ddf8ebc390254b1eb01 P 75f56c1a7f86421da9247e9b35990e34: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:37.131690 31054 tablet_bootstrap.cc:492] T f64692cd6a0347979834f29861bde5d4 P e77b2d60f57d4dceb865e440eab35126: No bootstrap required, opened a new log
I20260709 12:27:37.132205 31054 ts_tablet_manager.cc:1403] T f64692cd6a0347979834f29861bde5d4 P e77b2d60f57d4dceb865e440eab35126: Time spent bootstrapping tablet: real 0.051s	user 0.029s	sys 0.006s
I20260709 12:27:37.132280 30937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f64692cd6a0347979834f29861bde5d4" candidate_uuid: "75f56c1a7f86421da9247e9b35990e34" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e77b2d60f57d4dceb865e440eab35126" is_pre_election: true
W20260709 12:27:37.134689 30827 leader_election.cc:343] T f64692cd6a0347979834f29861bde5d4 P 75f56c1a7f86421da9247e9b35990e34 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e77b2d60f57d4dceb865e440eab35126 (127.16.29.66:46695): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:37.135289 31054 raft_consensus.cc:359] T f64692cd6a0347979834f29861bde5d4 P e77b2d60f57d4dceb865e440eab35126 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } }
I20260709 12:27:37.136071 31054 raft_consensus.cc:385] T f64692cd6a0347979834f29861bde5d4 P e77b2d60f57d4dceb865e440eab35126 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:37.136363 31054 raft_consensus.cc:740] T f64692cd6a0347979834f29861bde5d4 P e77b2d60f57d4dceb865e440eab35126 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e77b2d60f57d4dceb865e440eab35126, State: Initialized, Role: FOLLOWER
I20260709 12:27:37.137125 31054 consensus_queue.cc:260] T f64692cd6a0347979834f29861bde5d4 P e77b2d60f57d4dceb865e440eab35126 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } }
I20260709 12:27:37.139436 31012 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f64692cd6a0347979834f29861bde5d4" candidate_uuid: "75f56c1a7f86421da9247e9b35990e34" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2fbc4f1ad97a4f2492306d77a614527a" is_pre_election: true
I20260709 12:27:37.139436 31054 ts_tablet_manager.cc:1434] T f64692cd6a0347979834f29861bde5d4 P e77b2d60f57d4dceb865e440eab35126: Time spent starting tablet: real 0.007s	user 0.004s	sys 0.000s
W20260709 12:27:37.141076 30829 leader_election.cc:343] T f64692cd6a0347979834f29861bde5d4 P 75f56c1a7f86421da9247e9b35990e34 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2fbc4f1ad97a4f2492306d77a614527a (127.16.29.67:44761): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:37.141710 31053 tablet_bootstrap.cc:492] T f64692cd6a0347979834f29861bde5d4 P 2fbc4f1ad97a4f2492306d77a614527a: No bootstrap required, opened a new log
I20260709 12:27:37.141613 30829 leader_election.cc:304] T f64692cd6a0347979834f29861bde5d4 P 75f56c1a7f86421da9247e9b35990e34 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 75f56c1a7f86421da9247e9b35990e34; no voters: 2fbc4f1ad97a4f2492306d77a614527a, e77b2d60f57d4dceb865e440eab35126
I20260709 12:27:37.142401 31053 ts_tablet_manager.cc:1403] T f64692cd6a0347979834f29861bde5d4 P 2fbc4f1ad97a4f2492306d77a614527a: Time spent bootstrapping tablet: real 0.022s	user 0.009s	sys 0.010s
I20260709 12:27:37.141592 31052 tablet_bootstrap.cc:492] T 881773ceed164ddf8ebc390254b1eb01 P 75f56c1a7f86421da9247e9b35990e34: No bootstrap required, opened a new log
I20260709 12:27:37.142692 31059 raft_consensus.cc:2749] T f64692cd6a0347979834f29861bde5d4 P 75f56c1a7f86421da9247e9b35990e34 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:37.143411 31052 ts_tablet_manager.cc:1403] T 881773ceed164ddf8ebc390254b1eb01 P 75f56c1a7f86421da9247e9b35990e34: Time spent bootstrapping tablet: real 0.024s	user 0.009s	sys 0.005s
I20260709 12:27:37.142061 31054 tablet_bootstrap.cc:492] T adedadb9002047ba87a533a33374f4e7 P e77b2d60f57d4dceb865e440eab35126: Bootstrap starting.
I20260709 12:27:37.145439 31053 raft_consensus.cc:359] T f64692cd6a0347979834f29861bde5d4 P 2fbc4f1ad97a4f2492306d77a614527a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } }
I20260709 12:27:37.146317 31053 raft_consensus.cc:385] T f64692cd6a0347979834f29861bde5d4 P 2fbc4f1ad97a4f2492306d77a614527a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:37.146148 31052 raft_consensus.cc:359] T 881773ceed164ddf8ebc390254b1eb01 P 75f56c1a7f86421da9247e9b35990e34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } }
I20260709 12:27:37.146704 31053 raft_consensus.cc:740] T f64692cd6a0347979834f29861bde5d4 P 2fbc4f1ad97a4f2492306d77a614527a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fbc4f1ad97a4f2492306d77a614527a, State: Initialized, Role: FOLLOWER
I20260709 12:27:37.146991 31052 raft_consensus.cc:385] T 881773ceed164ddf8ebc390254b1eb01 P 75f56c1a7f86421da9247e9b35990e34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:37.147336 31052 raft_consensus.cc:740] T 881773ceed164ddf8ebc390254b1eb01 P 75f56c1a7f86421da9247e9b35990e34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75f56c1a7f86421da9247e9b35990e34, State: Initialized, Role: FOLLOWER
I20260709 12:27:37.148739 31052 consensus_queue.cc:260] T 881773ceed164ddf8ebc390254b1eb01 P 75f56c1a7f86421da9247e9b35990e34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } }
I20260709 12:27:37.148952 31053 consensus_queue.cc:260] T f64692cd6a0347979834f29861bde5d4 P 2fbc4f1ad97a4f2492306d77a614527a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } }
I20260709 12:27:37.153055 31053 ts_tablet_manager.cc:1434] T f64692cd6a0347979834f29861bde5d4 P 2fbc4f1ad97a4f2492306d77a614527a: Time spent starting tablet: real 0.010s	user 0.005s	sys 0.001s
I20260709 12:27:37.153156 31054 tablet_bootstrap.cc:654] T adedadb9002047ba87a533a33374f4e7 P e77b2d60f57d4dceb865e440eab35126: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:37.156163 31052 ts_tablet_manager.cc:1434] T 881773ceed164ddf8ebc390254b1eb01 P 75f56c1a7f86421da9247e9b35990e34: Time spent starting tablet: real 0.012s	user 0.005s	sys 0.001s
I20260709 12:27:37.153842 30759 catalog_manager.cc:5697] T de8ccd191af64c50ac32e3342ca995b0 P e77b2d60f57d4dceb865e440eab35126 reported cstate change: term changed from 0 to 1, leader changed from <none> to e77b2d60f57d4dceb865e440eab35126 (127.16.29.66). New cstate: current_term: 1 leader_uuid: "e77b2d60f57d4dceb865e440eab35126" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:37.164439 31053 tablet_bootstrap.cc:492] T 3ed242e2c2b843978a7af7cca5694ae2 P 2fbc4f1ad97a4f2492306d77a614527a: Bootstrap starting.
I20260709 12:27:37.174485 31053 tablet_bootstrap.cc:654] T 3ed242e2c2b843978a7af7cca5694ae2 P 2fbc4f1ad97a4f2492306d77a614527a: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:37.174423 31054 tablet_bootstrap.cc:492] T adedadb9002047ba87a533a33374f4e7 P e77b2d60f57d4dceb865e440eab35126: No bootstrap required, opened a new log
I20260709 12:27:37.175341 31054 ts_tablet_manager.cc:1403] T adedadb9002047ba87a533a33374f4e7 P e77b2d60f57d4dceb865e440eab35126: Time spent bootstrapping tablet: real 0.033s	user 0.020s	sys 0.007s
I20260709 12:27:37.178268 31054 raft_consensus.cc:359] T adedadb9002047ba87a533a33374f4e7 P e77b2d60f57d4dceb865e440eab35126 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } }
I20260709 12:27:37.178915 31054 raft_consensus.cc:385] T adedadb9002047ba87a533a33374f4e7 P e77b2d60f57d4dceb865e440eab35126 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:37.179185 31054 raft_consensus.cc:740] T adedadb9002047ba87a533a33374f4e7 P e77b2d60f57d4dceb865e440eab35126 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e77b2d60f57d4dceb865e440eab35126, State: Initialized, Role: FOLLOWER
I20260709 12:27:37.179917 31054 consensus_queue.cc:260] T adedadb9002047ba87a533a33374f4e7 P e77b2d60f57d4dceb865e440eab35126 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } }
I20260709 12:27:37.180203 31053 tablet_bootstrap.cc:492] T 3ed242e2c2b843978a7af7cca5694ae2 P 2fbc4f1ad97a4f2492306d77a614527a: No bootstrap required, opened a new log
I20260709 12:27:37.180757 31053 ts_tablet_manager.cc:1403] T 3ed242e2c2b843978a7af7cca5694ae2 P 2fbc4f1ad97a4f2492306d77a614527a: Time spent bootstrapping tablet: real 0.017s	user 0.006s	sys 0.008s
I20260709 12:27:37.181530 31054 ts_tablet_manager.cc:1434] T adedadb9002047ba87a533a33374f4e7 P e77b2d60f57d4dceb865e440eab35126: Time spent starting tablet: real 0.006s	user 0.005s	sys 0.000s
I20260709 12:27:37.182317 31054 tablet_bootstrap.cc:492] T 009d38185f3e49afbbd9faf57f968b2c P e77b2d60f57d4dceb865e440eab35126: Bootstrap starting.
I20260709 12:27:37.183422 31053 raft_consensus.cc:359] T 3ed242e2c2b843978a7af7cca5694ae2 P 2fbc4f1ad97a4f2492306d77a614527a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } }
I20260709 12:27:37.184142 31053 raft_consensus.cc:385] T 3ed242e2c2b843978a7af7cca5694ae2 P 2fbc4f1ad97a4f2492306d77a614527a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:37.184412 31053 raft_consensus.cc:740] T 3ed242e2c2b843978a7af7cca5694ae2 P 2fbc4f1ad97a4f2492306d77a614527a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fbc4f1ad97a4f2492306d77a614527a, State: Initialized, Role: FOLLOWER
I20260709 12:27:37.185099 31053 consensus_queue.cc:260] T 3ed242e2c2b843978a7af7cca5694ae2 P 2fbc4f1ad97a4f2492306d77a614527a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } }
I20260709 12:27:37.187052 31053 ts_tablet_manager.cc:1434] T 3ed242e2c2b843978a7af7cca5694ae2 P 2fbc4f1ad97a4f2492306d77a614527a: Time spent starting tablet: real 0.006s	user 0.004s	sys 0.000s
I20260709 12:27:37.187572 31054 tablet_bootstrap.cc:654] T 009d38185f3e49afbbd9faf57f968b2c P e77b2d60f57d4dceb865e440eab35126: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:37.188081 31053 tablet_bootstrap.cc:492] T adedadb9002047ba87a533a33374f4e7 P 2fbc4f1ad97a4f2492306d77a614527a: Bootstrap starting.
I20260709 12:27:37.192843 31054 tablet_bootstrap.cc:492] T 009d38185f3e49afbbd9faf57f968b2c P e77b2d60f57d4dceb865e440eab35126: No bootstrap required, opened a new log
I20260709 12:27:37.193279 31054 ts_tablet_manager.cc:1403] T 009d38185f3e49afbbd9faf57f968b2c P e77b2d60f57d4dceb865e440eab35126: Time spent bootstrapping tablet: real 0.011s	user 0.009s	sys 0.000s
I20260709 12:27:37.194407 31053 tablet_bootstrap.cc:654] T adedadb9002047ba87a533a33374f4e7 P 2fbc4f1ad97a4f2492306d77a614527a: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:37.195179 31054 raft_consensus.cc:359] T 009d38185f3e49afbbd9faf57f968b2c P e77b2d60f57d4dceb865e440eab35126 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } }
I20260709 12:27:37.195807 31054 raft_consensus.cc:385] T 009d38185f3e49afbbd9faf57f968b2c P e77b2d60f57d4dceb865e440eab35126 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:37.196089 31054 raft_consensus.cc:740] T 009d38185f3e49afbbd9faf57f968b2c P e77b2d60f57d4dceb865e440eab35126 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e77b2d60f57d4dceb865e440eab35126, State: Initialized, Role: FOLLOWER
I20260709 12:27:37.196795 31054 consensus_queue.cc:260] T 009d38185f3e49afbbd9faf57f968b2c P e77b2d60f57d4dceb865e440eab35126 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } }
I20260709 12:27:37.198844 31054 ts_tablet_manager.cc:1434] T 009d38185f3e49afbbd9faf57f968b2c P e77b2d60f57d4dceb865e440eab35126: Time spent starting tablet: real 0.005s	user 0.005s	sys 0.000s
I20260709 12:27:37.199621 31053 tablet_bootstrap.cc:492] T adedadb9002047ba87a533a33374f4e7 P 2fbc4f1ad97a4f2492306d77a614527a: No bootstrap required, opened a new log
I20260709 12:27:37.200006 31054 tablet_bootstrap.cc:492] T 881773ceed164ddf8ebc390254b1eb01 P e77b2d60f57d4dceb865e440eab35126: Bootstrap starting.
I20260709 12:27:37.200138 31053 ts_tablet_manager.cc:1403] T adedadb9002047ba87a533a33374f4e7 P 2fbc4f1ad97a4f2492306d77a614527a: Time spent bootstrapping tablet: real 0.012s	user 0.011s	sys 0.000s
I20260709 12:27:37.202935 31053 raft_consensus.cc:359] T adedadb9002047ba87a533a33374f4e7 P 2fbc4f1ad97a4f2492306d77a614527a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } }
I20260709 12:27:37.203599 31053 raft_consensus.cc:385] T adedadb9002047ba87a533a33374f4e7 P 2fbc4f1ad97a4f2492306d77a614527a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:37.203943 31053 raft_consensus.cc:740] T adedadb9002047ba87a533a33374f4e7 P 2fbc4f1ad97a4f2492306d77a614527a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fbc4f1ad97a4f2492306d77a614527a, State: Initialized, Role: FOLLOWER
I20260709 12:27:37.204556 31053 consensus_queue.cc:260] T adedadb9002047ba87a533a33374f4e7 P 2fbc4f1ad97a4f2492306d77a614527a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } }
I20260709 12:27:37.204905 31054 tablet_bootstrap.cc:654] T 881773ceed164ddf8ebc390254b1eb01 P e77b2d60f57d4dceb865e440eab35126: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:37.206432 31053 ts_tablet_manager.cc:1434] T adedadb9002047ba87a533a33374f4e7 P 2fbc4f1ad97a4f2492306d77a614527a: Time spent starting tablet: real 0.006s	user 0.005s	sys 0.000s
I20260709 12:27:37.207456 31053 tablet_bootstrap.cc:492] T 881773ceed164ddf8ebc390254b1eb01 P 2fbc4f1ad97a4f2492306d77a614527a: Bootstrap starting.
I20260709 12:27:37.214370 31053 tablet_bootstrap.cc:654] T 881773ceed164ddf8ebc390254b1eb01 P 2fbc4f1ad97a4f2492306d77a614527a: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:37.214653 31054 tablet_bootstrap.cc:492] T 881773ceed164ddf8ebc390254b1eb01 P e77b2d60f57d4dceb865e440eab35126: No bootstrap required, opened a new log
I20260709 12:27:37.215086 31054 ts_tablet_manager.cc:1403] T 881773ceed164ddf8ebc390254b1eb01 P e77b2d60f57d4dceb865e440eab35126: Time spent bootstrapping tablet: real 0.015s	user 0.005s	sys 0.008s
I20260709 12:27:37.217139 31054 raft_consensus.cc:359] T 881773ceed164ddf8ebc390254b1eb01 P e77b2d60f57d4dceb865e440eab35126 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } }
I20260709 12:27:37.217629 31054 raft_consensus.cc:385] T 881773ceed164ddf8ebc390254b1eb01 P e77b2d60f57d4dceb865e440eab35126 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:37.217847 31054 raft_consensus.cc:740] T 881773ceed164ddf8ebc390254b1eb01 P e77b2d60f57d4dceb865e440eab35126 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e77b2d60f57d4dceb865e440eab35126, State: Initialized, Role: FOLLOWER
I20260709 12:27:37.218431 31054 consensus_queue.cc:260] T 881773ceed164ddf8ebc390254b1eb01 P e77b2d60f57d4dceb865e440eab35126 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } }
I20260709 12:27:37.220247 31054 ts_tablet_manager.cc:1434] T 881773ceed164ddf8ebc390254b1eb01 P e77b2d60f57d4dceb865e440eab35126: Time spent starting tablet: real 0.005s	user 0.006s	sys 0.000s
I20260709 12:27:37.222885 31053 tablet_bootstrap.cc:492] T 881773ceed164ddf8ebc390254b1eb01 P 2fbc4f1ad97a4f2492306d77a614527a: No bootstrap required, opened a new log
I20260709 12:27:37.223348 31053 ts_tablet_manager.cc:1403] T 881773ceed164ddf8ebc390254b1eb01 P 2fbc4f1ad97a4f2492306d77a614527a: Time spent bootstrapping tablet: real 0.016s	user 0.003s	sys 0.010s
I20260709 12:27:37.225517 31053 raft_consensus.cc:359] T 881773ceed164ddf8ebc390254b1eb01 P 2fbc4f1ad97a4f2492306d77a614527a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } }
I20260709 12:27:37.225948 31053 raft_consensus.cc:385] T 881773ceed164ddf8ebc390254b1eb01 P 2fbc4f1ad97a4f2492306d77a614527a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:37.226267 31053 raft_consensus.cc:740] T 881773ceed164ddf8ebc390254b1eb01 P 2fbc4f1ad97a4f2492306d77a614527a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fbc4f1ad97a4f2492306d77a614527a, State: Initialized, Role: FOLLOWER
I20260709 12:27:37.226853 31053 consensus_queue.cc:260] T 881773ceed164ddf8ebc390254b1eb01 P 2fbc4f1ad97a4f2492306d77a614527a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } }
I20260709 12:27:37.228658 31053 ts_tablet_manager.cc:1434] T 881773ceed164ddf8ebc390254b1eb01 P 2fbc4f1ad97a4f2492306d77a614527a: Time spent starting tablet: real 0.005s	user 0.003s	sys 0.003s
I20260709 12:27:37.250079 31083 raft_consensus.cc:493] T 8e560587c5c24e4587295d2d1b6d2be6 P e77b2d60f57d4dceb865e440eab35126 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:37.250692 31083 raft_consensus.cc:515] T 8e560587c5c24e4587295d2d1b6d2be6 P e77b2d60f57d4dceb865e440eab35126 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } }
I20260709 12:27:37.252629 31083 leader_election.cc:290] T 8e560587c5c24e4587295d2d1b6d2be6 P e77b2d60f57d4dceb865e440eab35126 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 75f56c1a7f86421da9247e9b35990e34 (127.16.29.65:45249), 2fbc4f1ad97a4f2492306d77a614527a (127.16.29.67:44761)
I20260709 12:27:37.253756 30862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e560587c5c24e4587295d2d1b6d2be6" candidate_uuid: "e77b2d60f57d4dceb865e440eab35126" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75f56c1a7f86421da9247e9b35990e34" is_pre_election: true
I20260709 12:27:37.253923 31012 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e560587c5c24e4587295d2d1b6d2be6" candidate_uuid: "e77b2d60f57d4dceb865e440eab35126" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2fbc4f1ad97a4f2492306d77a614527a" is_pre_election: true
I20260709 12:27:37.254419 30862 raft_consensus.cc:2468] T 8e560587c5c24e4587295d2d1b6d2be6 P 75f56c1a7f86421da9247e9b35990e34 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e77b2d60f57d4dceb865e440eab35126 in term 0.
I20260709 12:27:37.254617 31012 raft_consensus.cc:2468] T 8e560587c5c24e4587295d2d1b6d2be6 P 2fbc4f1ad97a4f2492306d77a614527a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e77b2d60f57d4dceb865e440eab35126 in term 0.
I20260709 12:27:37.255806 30903 leader_election.cc:304] T 8e560587c5c24e4587295d2d1b6d2be6 P e77b2d60f57d4dceb865e440eab35126 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 75f56c1a7f86421da9247e9b35990e34, e77b2d60f57d4dceb865e440eab35126; no voters: 
I20260709 12:27:37.256739 31083 raft_consensus.cc:2804] T 8e560587c5c24e4587295d2d1b6d2be6 P e77b2d60f57d4dceb865e440eab35126 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:37.257002 31083 raft_consensus.cc:493] T 8e560587c5c24e4587295d2d1b6d2be6 P e77b2d60f57d4dceb865e440eab35126 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:37.257287 31083 raft_consensus.cc:3060] T 8e560587c5c24e4587295d2d1b6d2be6 P e77b2d60f57d4dceb865e440eab35126 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:37.259702 31060 raft_consensus.cc:493] T 881773ceed164ddf8ebc390254b1eb01 P e77b2d60f57d4dceb865e440eab35126 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:37.260376 31060 raft_consensus.cc:515] T 881773ceed164ddf8ebc390254b1eb01 P e77b2d60f57d4dceb865e440eab35126 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } }
I20260709 12:27:37.262329 31060 leader_election.cc:290] T 881773ceed164ddf8ebc390254b1eb01 P e77b2d60f57d4dceb865e440eab35126 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 75f56c1a7f86421da9247e9b35990e34 (127.16.29.65:45249), 2fbc4f1ad97a4f2492306d77a614527a (127.16.29.67:44761)
I20260709 12:27:37.263257 30862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "881773ceed164ddf8ebc390254b1eb01" candidate_uuid: "e77b2d60f57d4dceb865e440eab35126" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75f56c1a7f86421da9247e9b35990e34" is_pre_election: true
I20260709 12:27:37.263516 31012 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "881773ceed164ddf8ebc390254b1eb01" candidate_uuid: "e77b2d60f57d4dceb865e440eab35126" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2fbc4f1ad97a4f2492306d77a614527a" is_pre_election: true
I20260709 12:27:37.263384 31083 raft_consensus.cc:515] T 8e560587c5c24e4587295d2d1b6d2be6 P e77b2d60f57d4dceb865e440eab35126 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } }
I20260709 12:27:37.264212 31012 raft_consensus.cc:2468] T 881773ceed164ddf8ebc390254b1eb01 P 2fbc4f1ad97a4f2492306d77a614527a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e77b2d60f57d4dceb865e440eab35126 in term 0.
I20260709 12:27:37.264034 30862 raft_consensus.cc:2468] T 881773ceed164ddf8ebc390254b1eb01 P 75f56c1a7f86421da9247e9b35990e34 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e77b2d60f57d4dceb865e440eab35126 in term 0.
I20260709 12:27:37.265866 30904 leader_election.cc:304] T 881773ceed164ddf8ebc390254b1eb01 P e77b2d60f57d4dceb865e440eab35126 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2fbc4f1ad97a4f2492306d77a614527a, e77b2d60f57d4dceb865e440eab35126; no voters: 
I20260709 12:27:37.266069 31083 leader_election.cc:290] T 8e560587c5c24e4587295d2d1b6d2be6 P e77b2d60f57d4dceb865e440eab35126 [CANDIDATE]: Term 1 election: Requested vote from peers 75f56c1a7f86421da9247e9b35990e34 (127.16.29.65:45249), 2fbc4f1ad97a4f2492306d77a614527a (127.16.29.67:44761)
I20260709 12:27:37.266893 31083 raft_consensus.cc:2804] T 881773ceed164ddf8ebc390254b1eb01 P e77b2d60f57d4dceb865e440eab35126 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:37.267283 31083 raft_consensus.cc:493] T 881773ceed164ddf8ebc390254b1eb01 P e77b2d60f57d4dceb865e440eab35126 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:37.267611 31083 raft_consensus.cc:3060] T 881773ceed164ddf8ebc390254b1eb01 P e77b2d60f57d4dceb865e440eab35126 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:37.267860 30862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e560587c5c24e4587295d2d1b6d2be6" candidate_uuid: "e77b2d60f57d4dceb865e440eab35126" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75f56c1a7f86421da9247e9b35990e34"
I20260709 12:27:37.268013 31012 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e560587c5c24e4587295d2d1b6d2be6" candidate_uuid: "e77b2d60f57d4dceb865e440eab35126" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2fbc4f1ad97a4f2492306d77a614527a"
I20260709 12:27:37.268589 30862 raft_consensus.cc:3060] T 8e560587c5c24e4587295d2d1b6d2be6 P 75f56c1a7f86421da9247e9b35990e34 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:37.268683 31012 raft_consensus.cc:3060] T 8e560587c5c24e4587295d2d1b6d2be6 P 2fbc4f1ad97a4f2492306d77a614527a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:37.273617 30862 raft_consensus.cc:2468] T 8e560587c5c24e4587295d2d1b6d2be6 P 75f56c1a7f86421da9247e9b35990e34 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e77b2d60f57d4dceb865e440eab35126 in term 1.
I20260709 12:27:37.274756 30903 leader_election.cc:304] T 8e560587c5c24e4587295d2d1b6d2be6 P e77b2d60f57d4dceb865e440eab35126 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 75f56c1a7f86421da9247e9b35990e34, e77b2d60f57d4dceb865e440eab35126; no voters: 
I20260709 12:27:37.274967 31083 raft_consensus.cc:515] T 881773ceed164ddf8ebc390254b1eb01 P e77b2d60f57d4dceb865e440eab35126 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } }
I20260709 12:27:37.275529 31060 raft_consensus.cc:2804] T 8e560587c5c24e4587295d2d1b6d2be6 P e77b2d60f57d4dceb865e440eab35126 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:37.276090 31012 raft_consensus.cc:2468] T 8e560587c5c24e4587295d2d1b6d2be6 P 2fbc4f1ad97a4f2492306d77a614527a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e77b2d60f57d4dceb865e440eab35126 in term 1.
I20260709 12:27:37.276160 31060 raft_consensus.cc:697] T 8e560587c5c24e4587295d2d1b6d2be6 P e77b2d60f57d4dceb865e440eab35126 [term 1 LEADER]: Becoming Leader. State: Replica: e77b2d60f57d4dceb865e440eab35126, State: Running, Role: LEADER
I20260709 12:27:37.277395 31083 leader_election.cc:290] T 881773ceed164ddf8ebc390254b1eb01 P e77b2d60f57d4dceb865e440eab35126 [CANDIDATE]: Term 1 election: Requested vote from peers 75f56c1a7f86421da9247e9b35990e34 (127.16.29.65:45249), 2fbc4f1ad97a4f2492306d77a614527a (127.16.29.67:44761)
I20260709 12:27:37.278215 30862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "881773ceed164ddf8ebc390254b1eb01" candidate_uuid: "e77b2d60f57d4dceb865e440eab35126" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75f56c1a7f86421da9247e9b35990e34"
I20260709 12:27:37.278467 31011 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "881773ceed164ddf8ebc390254b1eb01" candidate_uuid: "e77b2d60f57d4dceb865e440eab35126" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2fbc4f1ad97a4f2492306d77a614527a"
I20260709 12:27:37.278801 30862 raft_consensus.cc:3060] T 881773ceed164ddf8ebc390254b1eb01 P 75f56c1a7f86421da9247e9b35990e34 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:37.278393 31060 consensus_queue.cc:237] T 8e560587c5c24e4587295d2d1b6d2be6 P e77b2d60f57d4dceb865e440eab35126 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } }
I20260709 12:27:37.279255 31011 raft_consensus.cc:3060] T 881773ceed164ddf8ebc390254b1eb01 P 2fbc4f1ad97a4f2492306d77a614527a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:37.285352 30862 raft_consensus.cc:2468] T 881773ceed164ddf8ebc390254b1eb01 P 75f56c1a7f86421da9247e9b35990e34 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e77b2d60f57d4dceb865e440eab35126 in term 1.
I20260709 12:27:37.286501 30903 leader_election.cc:304] T 881773ceed164ddf8ebc390254b1eb01 P e77b2d60f57d4dceb865e440eab35126 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 75f56c1a7f86421da9247e9b35990e34, e77b2d60f57d4dceb865e440eab35126; no voters: 
I20260709 12:27:37.287031 31011 raft_consensus.cc:2468] T 881773ceed164ddf8ebc390254b1eb01 P 2fbc4f1ad97a4f2492306d77a614527a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e77b2d60f57d4dceb865e440eab35126 in term 1.
I20260709 12:27:37.287391 31060 raft_consensus.cc:2804] T 881773ceed164ddf8ebc390254b1eb01 P e77b2d60f57d4dceb865e440eab35126 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:37.288007 31060 raft_consensus.cc:697] T 881773ceed164ddf8ebc390254b1eb01 P e77b2d60f57d4dceb865e440eab35126 [term 1 LEADER]: Becoming Leader. State: Replica: e77b2d60f57d4dceb865e440eab35126, State: Running, Role: LEADER
I20260709 12:27:37.288853 31060 consensus_queue.cc:237] T 881773ceed164ddf8ebc390254b1eb01 P e77b2d60f57d4dceb865e440eab35126 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } }
I20260709 12:27:37.289196 30761 catalog_manager.cc:5697] T 8e560587c5c24e4587295d2d1b6d2be6 P e77b2d60f57d4dceb865e440eab35126 reported cstate change: term changed from 0 to 1, leader changed from <none> to e77b2d60f57d4dceb865e440eab35126 (127.16.29.66). New cstate: current_term: 1 leader_uuid: "e77b2d60f57d4dceb865e440eab35126" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:37.302910 30761 catalog_manager.cc:5697] T 881773ceed164ddf8ebc390254b1eb01 P e77b2d60f57d4dceb865e440eab35126 reported cstate change: term changed from 0 to 1, leader changed from <none> to e77b2d60f57d4dceb865e440eab35126 (127.16.29.66). New cstate: current_term: 1 leader_uuid: "e77b2d60f57d4dceb865e440eab35126" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:37.306596 31060 raft_consensus.cc:493] T adedadb9002047ba87a533a33374f4e7 P e77b2d60f57d4dceb865e440eab35126 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:37.307240 31060 raft_consensus.cc:515] T adedadb9002047ba87a533a33374f4e7 P e77b2d60f57d4dceb865e440eab35126 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } }
I20260709 12:27:37.309540 31060 leader_election.cc:290] T adedadb9002047ba87a533a33374f4e7 P e77b2d60f57d4dceb865e440eab35126 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 75f56c1a7f86421da9247e9b35990e34 (127.16.29.65:45249), 2fbc4f1ad97a4f2492306d77a614527a (127.16.29.67:44761)
I20260709 12:27:37.310313 30862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adedadb9002047ba87a533a33374f4e7" candidate_uuid: "e77b2d60f57d4dceb865e440eab35126" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75f56c1a7f86421da9247e9b35990e34" is_pre_election: true
I20260709 12:27:37.310653 31011 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adedadb9002047ba87a533a33374f4e7" candidate_uuid: "e77b2d60f57d4dceb865e440eab35126" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2fbc4f1ad97a4f2492306d77a614527a" is_pre_election: true
I20260709 12:27:37.311160 30862 raft_consensus.cc:2468] T adedadb9002047ba87a533a33374f4e7 P 75f56c1a7f86421da9247e9b35990e34 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e77b2d60f57d4dceb865e440eab35126 in term 0.
I20260709 12:27:37.311348 31011 raft_consensus.cc:2468] T adedadb9002047ba87a533a33374f4e7 P 2fbc4f1ad97a4f2492306d77a614527a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e77b2d60f57d4dceb865e440eab35126 in term 0.
I20260709 12:27:37.312575 30904 leader_election.cc:304] T adedadb9002047ba87a533a33374f4e7 P e77b2d60f57d4dceb865e440eab35126 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2fbc4f1ad97a4f2492306d77a614527a, e77b2d60f57d4dceb865e440eab35126; no voters: 
I20260709 12:27:37.313428 31060 raft_consensus.cc:2804] T adedadb9002047ba87a533a33374f4e7 P e77b2d60f57d4dceb865e440eab35126 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:37.313853 31060 raft_consensus.cc:493] T adedadb9002047ba87a533a33374f4e7 P e77b2d60f57d4dceb865e440eab35126 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:37.314172 31060 raft_consensus.cc:3060] T adedadb9002047ba87a533a33374f4e7 P e77b2d60f57d4dceb865e440eab35126 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:37.319641 31060 raft_consensus.cc:515] T adedadb9002047ba87a533a33374f4e7 P e77b2d60f57d4dceb865e440eab35126 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } }
I20260709 12:27:37.321442 31060 leader_election.cc:290] T adedadb9002047ba87a533a33374f4e7 P e77b2d60f57d4dceb865e440eab35126 [CANDIDATE]: Term 1 election: Requested vote from peers 75f56c1a7f86421da9247e9b35990e34 (127.16.29.65:45249), 2fbc4f1ad97a4f2492306d77a614527a (127.16.29.67:44761)
I20260709 12:27:37.322316 30862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adedadb9002047ba87a533a33374f4e7" candidate_uuid: "e77b2d60f57d4dceb865e440eab35126" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75f56c1a7f86421da9247e9b35990e34"
I20260709 12:27:37.322535 31011 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adedadb9002047ba87a533a33374f4e7" candidate_uuid: "e77b2d60f57d4dceb865e440eab35126" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2fbc4f1ad97a4f2492306d77a614527a"
I20260709 12:27:37.323102 30862 raft_consensus.cc:3060] T adedadb9002047ba87a533a33374f4e7 P 75f56c1a7f86421da9247e9b35990e34 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:37.323161 31011 raft_consensus.cc:3060] T adedadb9002047ba87a533a33374f4e7 P 2fbc4f1ad97a4f2492306d77a614527a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:37.331131 30862 raft_consensus.cc:2468] T adedadb9002047ba87a533a33374f4e7 P 75f56c1a7f86421da9247e9b35990e34 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e77b2d60f57d4dceb865e440eab35126 in term 1.
I20260709 12:27:37.331131 31011 raft_consensus.cc:2468] T adedadb9002047ba87a533a33374f4e7 P 2fbc4f1ad97a4f2492306d77a614527a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e77b2d60f57d4dceb865e440eab35126 in term 1.
I20260709 12:27:37.332741 31057 raft_consensus.cc:493] T f64692cd6a0347979834f29861bde5d4 P 2fbc4f1ad97a4f2492306d77a614527a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:37.332796 30904 leader_election.cc:304] T adedadb9002047ba87a533a33374f4e7 P e77b2d60f57d4dceb865e440eab35126 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2fbc4f1ad97a4f2492306d77a614527a, e77b2d60f57d4dceb865e440eab35126; no voters: 
I20260709 12:27:37.333475 31057 raft_consensus.cc:515] T f64692cd6a0347979834f29861bde5d4 P 2fbc4f1ad97a4f2492306d77a614527a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } }
I20260709 12:27:37.333873 31060 raft_consensus.cc:2804] T adedadb9002047ba87a533a33374f4e7 P e77b2d60f57d4dceb865e440eab35126 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:37.334465 31060 raft_consensus.cc:697] T adedadb9002047ba87a533a33374f4e7 P e77b2d60f57d4dceb865e440eab35126 [term 1 LEADER]: Becoming Leader. State: Replica: e77b2d60f57d4dceb865e440eab35126, State: Running, Role: LEADER
I20260709 12:27:37.335567 31057 leader_election.cc:290] T f64692cd6a0347979834f29861bde5d4 P 2fbc4f1ad97a4f2492306d77a614527a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 75f56c1a7f86421da9247e9b35990e34 (127.16.29.65:45249), e77b2d60f57d4dceb865e440eab35126 (127.16.29.66:46695)
I20260709 12:27:37.335250 31060 consensus_queue.cc:237] T adedadb9002047ba87a533a33374f4e7 P e77b2d60f57d4dceb865e440eab35126 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } }
I20260709 12:27:37.336485 30862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f64692cd6a0347979834f29861bde5d4" candidate_uuid: "2fbc4f1ad97a4f2492306d77a614527a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75f56c1a7f86421da9247e9b35990e34" is_pre_election: true
I20260709 12:27:37.336668 30937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f64692cd6a0347979834f29861bde5d4" candidate_uuid: "2fbc4f1ad97a4f2492306d77a614527a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e77b2d60f57d4dceb865e440eab35126" is_pre_election: true
I20260709 12:27:37.337236 30862 raft_consensus.cc:2468] T f64692cd6a0347979834f29861bde5d4 P 75f56c1a7f86421da9247e9b35990e34 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2fbc4f1ad97a4f2492306d77a614527a in term 0.
I20260709 12:27:37.337332 30937 raft_consensus.cc:2468] T f64692cd6a0347979834f29861bde5d4 P e77b2d60f57d4dceb865e440eab35126 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2fbc4f1ad97a4f2492306d77a614527a in term 0.
I20260709 12:27:37.338519 30977 leader_election.cc:304] T f64692cd6a0347979834f29861bde5d4 P 2fbc4f1ad97a4f2492306d77a614527a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2fbc4f1ad97a4f2492306d77a614527a, 75f56c1a7f86421da9247e9b35990e34; no voters: 
I20260709 12:27:37.339771 31057 raft_consensus.cc:2804] T f64692cd6a0347979834f29861bde5d4 P 2fbc4f1ad97a4f2492306d77a614527a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:37.340173 31057 raft_consensus.cc:493] T f64692cd6a0347979834f29861bde5d4 P 2fbc4f1ad97a4f2492306d77a614527a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:37.340531 31057 raft_consensus.cc:3060] T f64692cd6a0347979834f29861bde5d4 P 2fbc4f1ad97a4f2492306d77a614527a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:37.343818 30761 catalog_manager.cc:5697] T adedadb9002047ba87a533a33374f4e7 P e77b2d60f57d4dceb865e440eab35126 reported cstate change: term changed from 0 to 1, leader changed from <none> to e77b2d60f57d4dceb865e440eab35126 (127.16.29.66). New cstate: current_term: 1 leader_uuid: "e77b2d60f57d4dceb865e440eab35126" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:37.346082 31057 raft_consensus.cc:515] T f64692cd6a0347979834f29861bde5d4 P 2fbc4f1ad97a4f2492306d77a614527a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } }
I20260709 12:27:37.348124 31059 raft_consensus.cc:493] T 0be78a9b3c5d47df8efea08d91f58111 P 75f56c1a7f86421da9247e9b35990e34 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:37.348636 31059 raft_consensus.cc:515] T 0be78a9b3c5d47df8efea08d91f58111 P 75f56c1a7f86421da9247e9b35990e34 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } }
I20260709 12:27:37.348897 30937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f64692cd6a0347979834f29861bde5d4" candidate_uuid: "2fbc4f1ad97a4f2492306d77a614527a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e77b2d60f57d4dceb865e440eab35126"
I20260709 12:27:37.349511 30937 raft_consensus.cc:3060] T f64692cd6a0347979834f29861bde5d4 P e77b2d60f57d4dceb865e440eab35126 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:37.347713 31057 leader_election.cc:290] T f64692cd6a0347979834f29861bde5d4 P 2fbc4f1ad97a4f2492306d77a614527a [CANDIDATE]: Term 1 election: Requested vote from peers 75f56c1a7f86421da9247e9b35990e34 (127.16.29.65:45249), e77b2d60f57d4dceb865e440eab35126 (127.16.29.66:46695)
I20260709 12:27:37.350800 31059 leader_election.cc:290] T 0be78a9b3c5d47df8efea08d91f58111 P 75f56c1a7f86421da9247e9b35990e34 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e77b2d60f57d4dceb865e440eab35126 (127.16.29.66:46695), 2fbc4f1ad97a4f2492306d77a614527a (127.16.29.67:44761)
I20260709 12:27:37.350811 30862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f64692cd6a0347979834f29861bde5d4" candidate_uuid: "2fbc4f1ad97a4f2492306d77a614527a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75f56c1a7f86421da9247e9b35990e34"
I20260709 12:27:37.353220 30938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0be78a9b3c5d47df8efea08d91f58111" candidate_uuid: "75f56c1a7f86421da9247e9b35990e34" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e77b2d60f57d4dceb865e440eab35126" is_pre_election: true
I20260709 12:27:37.353521 30862 raft_consensus.cc:3060] T f64692cd6a0347979834f29861bde5d4 P 75f56c1a7f86421da9247e9b35990e34 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:37.354079 30938 raft_consensus.cc:2393] T 0be78a9b3c5d47df8efea08d91f58111 P e77b2d60f57d4dceb865e440eab35126 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 75f56c1a7f86421da9247e9b35990e34 in current term 1: Already voted for candidate 2fbc4f1ad97a4f2492306d77a614527a in this term.
I20260709 12:27:37.352576 31011 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0be78a9b3c5d47df8efea08d91f58111" candidate_uuid: "75f56c1a7f86421da9247e9b35990e34" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2fbc4f1ad97a4f2492306d77a614527a" is_pre_election: true
I20260709 12:27:37.356101 30829 leader_election.cc:304] T 0be78a9b3c5d47df8efea08d91f58111 P 75f56c1a7f86421da9247e9b35990e34 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 75f56c1a7f86421da9247e9b35990e34; no voters: 2fbc4f1ad97a4f2492306d77a614527a, e77b2d60f57d4dceb865e440eab35126
I20260709 12:27:37.356169 31060 raft_consensus.cc:493] T 009d38185f3e49afbbd9faf57f968b2c P e77b2d60f57d4dceb865e440eab35126 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:37.357064 31059 raft_consensus.cc:3060] T 0be78a9b3c5d47df8efea08d91f58111 P 75f56c1a7f86421da9247e9b35990e34 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:37.357468 31060 raft_consensus.cc:515] T 009d38185f3e49afbbd9faf57f968b2c P e77b2d60f57d4dceb865e440eab35126 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } }
I20260709 12:27:37.358805 30937 raft_consensus.cc:2468] T f64692cd6a0347979834f29861bde5d4 P e77b2d60f57d4dceb865e440eab35126 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2fbc4f1ad97a4f2492306d77a614527a in term 1.
I20260709 12:27:37.360111 31060 leader_election.cc:290] T 009d38185f3e49afbbd9faf57f968b2c P e77b2d60f57d4dceb865e440eab35126 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 75f56c1a7f86421da9247e9b35990e34 (127.16.29.65:45249), 2fbc4f1ad97a4f2492306d77a614527a (127.16.29.67:44761)
I20260709 12:27:37.360481 30863 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "009d38185f3e49afbbd9faf57f968b2c" candidate_uuid: "e77b2d60f57d4dceb865e440eab35126" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75f56c1a7f86421da9247e9b35990e34" is_pre_election: true
I20260709 12:27:37.360152 30976 leader_election.cc:304] T f64692cd6a0347979834f29861bde5d4 P 2fbc4f1ad97a4f2492306d77a614527a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2fbc4f1ad97a4f2492306d77a614527a, e77b2d60f57d4dceb865e440eab35126; no voters: 
I20260709 12:27:37.361259 30863 raft_consensus.cc:2468] T 009d38185f3e49afbbd9faf57f968b2c P 75f56c1a7f86421da9247e9b35990e34 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e77b2d60f57d4dceb865e440eab35126 in term 0.
I20260709 12:27:37.361387 31011 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "009d38185f3e49afbbd9faf57f968b2c" candidate_uuid: "e77b2d60f57d4dceb865e440eab35126" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2fbc4f1ad97a4f2492306d77a614527a" is_pre_election: true
I20260709 12:27:37.361768 31057 raft_consensus.cc:2804] T f64692cd6a0347979834f29861bde5d4 P 2fbc4f1ad97a4f2492306d77a614527a [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:37.362082 31011 raft_consensus.cc:2468] T 009d38185f3e49afbbd9faf57f968b2c P 2fbc4f1ad97a4f2492306d77a614527a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e77b2d60f57d4dceb865e440eab35126 in term 0.
I20260709 12:27:37.363066 30862 raft_consensus.cc:2468] T f64692cd6a0347979834f29861bde5d4 P 75f56c1a7f86421da9247e9b35990e34 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2fbc4f1ad97a4f2492306d77a614527a in term 1.
I20260709 12:27:37.365656 31057 raft_consensus.cc:697] T f64692cd6a0347979834f29861bde5d4 P 2fbc4f1ad97a4f2492306d77a614527a [term 1 LEADER]: Becoming Leader. State: Replica: 2fbc4f1ad97a4f2492306d77a614527a, State: Running, Role: LEADER
I20260709 12:27:37.362766 30903 leader_election.cc:304] T 009d38185f3e49afbbd9faf57f968b2c P e77b2d60f57d4dceb865e440eab35126 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 75f56c1a7f86421da9247e9b35990e34, e77b2d60f57d4dceb865e440eab35126; no voters: 
I20260709 12:27:37.367251 31060 raft_consensus.cc:2804] T 009d38185f3e49afbbd9faf57f968b2c P e77b2d60f57d4dceb865e440eab35126 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:37.367020 31057 consensus_queue.cc:237] T f64692cd6a0347979834f29861bde5d4 P 2fbc4f1ad97a4f2492306d77a614527a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } }
I20260709 12:27:37.367678 31060 raft_consensus.cc:493] T 009d38185f3e49afbbd9faf57f968b2c P e77b2d60f57d4dceb865e440eab35126 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:37.368098 31060 raft_consensus.cc:3060] T 009d38185f3e49afbbd9faf57f968b2c P e77b2d60f57d4dceb865e440eab35126 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:37.371703 31059 raft_consensus.cc:2749] T 0be78a9b3c5d47df8efea08d91f58111 P 75f56c1a7f86421da9247e9b35990e34 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:37.374854 31060 raft_consensus.cc:515] T 009d38185f3e49afbbd9faf57f968b2c P e77b2d60f57d4dceb865e440eab35126 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } }
I20260709 12:27:37.376622 30761 catalog_manager.cc:5697] T f64692cd6a0347979834f29861bde5d4 P 2fbc4f1ad97a4f2492306d77a614527a reported cstate change: term changed from 0 to 1, leader changed from <none> to 2fbc4f1ad97a4f2492306d77a614527a (127.16.29.67). New cstate: current_term: 1 leader_uuid: "2fbc4f1ad97a4f2492306d77a614527a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } health_report { overall_health: HEALTHY } } }
I20260709 12:27:37.377455 31060 leader_election.cc:290] T 009d38185f3e49afbbd9faf57f968b2c P e77b2d60f57d4dceb865e440eab35126 [CANDIDATE]: Term 1 election: Requested vote from peers 75f56c1a7f86421da9247e9b35990e34 (127.16.29.65:45249), 2fbc4f1ad97a4f2492306d77a614527a (127.16.29.67:44761)
I20260709 12:27:37.377818 30862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "009d38185f3e49afbbd9faf57f968b2c" candidate_uuid: "e77b2d60f57d4dceb865e440eab35126" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75f56c1a7f86421da9247e9b35990e34"
I20260709 12:27:37.378116 31011 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "009d38185f3e49afbbd9faf57f968b2c" candidate_uuid: "e77b2d60f57d4dceb865e440eab35126" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2fbc4f1ad97a4f2492306d77a614527a"
I20260709 12:27:37.378530 30862 raft_consensus.cc:3060] T 009d38185f3e49afbbd9faf57f968b2c P 75f56c1a7f86421da9247e9b35990e34 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:37.378707 31011 raft_consensus.cc:3060] T 009d38185f3e49afbbd9faf57f968b2c P 2fbc4f1ad97a4f2492306d77a614527a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:37.386070 30862 raft_consensus.cc:2468] T 009d38185f3e49afbbd9faf57f968b2c P 75f56c1a7f86421da9247e9b35990e34 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e77b2d60f57d4dceb865e440eab35126 in term 1.
I20260709 12:27:37.386070 31011 raft_consensus.cc:2468] T 009d38185f3e49afbbd9faf57f968b2c P 2fbc4f1ad97a4f2492306d77a614527a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e77b2d60f57d4dceb865e440eab35126 in term 1.
I20260709 12:27:37.387506 30903 leader_election.cc:304] T 009d38185f3e49afbbd9faf57f968b2c P e77b2d60f57d4dceb865e440eab35126 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 75f56c1a7f86421da9247e9b35990e34, e77b2d60f57d4dceb865e440eab35126; no voters: 
I20260709 12:27:37.388393 31060 raft_consensus.cc:2804] T 009d38185f3e49afbbd9faf57f968b2c P e77b2d60f57d4dceb865e440eab35126 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:37.388803 31060 raft_consensus.cc:697] T 009d38185f3e49afbbd9faf57f968b2c P e77b2d60f57d4dceb865e440eab35126 [term 1 LEADER]: Becoming Leader. State: Replica: e77b2d60f57d4dceb865e440eab35126, State: Running, Role: LEADER
I20260709 12:27:37.389978 31060 consensus_queue.cc:237] T 009d38185f3e49afbbd9faf57f968b2c P e77b2d60f57d4dceb865e440eab35126 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } }
I20260709 12:27:37.396591 30761 catalog_manager.cc:5697] T 009d38185f3e49afbbd9faf57f968b2c P e77b2d60f57d4dceb865e440eab35126 reported cstate change: term changed from 0 to 1, leader changed from <none> to e77b2d60f57d4dceb865e440eab35126 (127.16.29.66). New cstate: current_term: 1 leader_uuid: "e77b2d60f57d4dceb865e440eab35126" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:37.403597 31059 raft_consensus.cc:493] T 3ed242e2c2b843978a7af7cca5694ae2 P 75f56c1a7f86421da9247e9b35990e34 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:37.404238 31059 raft_consensus.cc:515] T 3ed242e2c2b843978a7af7cca5694ae2 P 75f56c1a7f86421da9247e9b35990e34 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } }
I20260709 12:27:37.405827 31059 leader_election.cc:290] T 3ed242e2c2b843978a7af7cca5694ae2 P 75f56c1a7f86421da9247e9b35990e34 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e77b2d60f57d4dceb865e440eab35126 (127.16.29.66:46695), 2fbc4f1ad97a4f2492306d77a614527a (127.16.29.67:44761)
I20260709 12:27:37.406677 30937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed242e2c2b843978a7af7cca5694ae2" candidate_uuid: "75f56c1a7f86421da9247e9b35990e34" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e77b2d60f57d4dceb865e440eab35126" is_pre_election: true
I20260709 12:27:37.406865 31011 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed242e2c2b843978a7af7cca5694ae2" candidate_uuid: "75f56c1a7f86421da9247e9b35990e34" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2fbc4f1ad97a4f2492306d77a614527a" is_pre_election: true
I20260709 12:27:37.407275 30937 raft_consensus.cc:2468] T 3ed242e2c2b843978a7af7cca5694ae2 P e77b2d60f57d4dceb865e440eab35126 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 75f56c1a7f86421da9247e9b35990e34 in term 0.
I20260709 12:27:37.407675 31011 raft_consensus.cc:2468] T 3ed242e2c2b843978a7af7cca5694ae2 P 2fbc4f1ad97a4f2492306d77a614527a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 75f56c1a7f86421da9247e9b35990e34 in term 0.
I20260709 12:27:37.408393 30827 leader_election.cc:304] T 3ed242e2c2b843978a7af7cca5694ae2 P 75f56c1a7f86421da9247e9b35990e34 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 75f56c1a7f86421da9247e9b35990e34, e77b2d60f57d4dceb865e440eab35126; no voters: 
I20260709 12:27:37.409224 31059 raft_consensus.cc:2804] T 3ed242e2c2b843978a7af7cca5694ae2 P 75f56c1a7f86421da9247e9b35990e34 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:37.409570 31059 raft_consensus.cc:493] T 3ed242e2c2b843978a7af7cca5694ae2 P 75f56c1a7f86421da9247e9b35990e34 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:37.409792 31059 raft_consensus.cc:3060] T 3ed242e2c2b843978a7af7cca5694ae2 P 75f56c1a7f86421da9247e9b35990e34 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:37.415339 31059 raft_consensus.cc:515] T 3ed242e2c2b843978a7af7cca5694ae2 P 75f56c1a7f86421da9247e9b35990e34 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } }
I20260709 12:27:37.417331 31059 leader_election.cc:290] T 3ed242e2c2b843978a7af7cca5694ae2 P 75f56c1a7f86421da9247e9b35990e34 [CANDIDATE]: Term 1 election: Requested vote from peers e77b2d60f57d4dceb865e440eab35126 (127.16.29.66:46695), 2fbc4f1ad97a4f2492306d77a614527a (127.16.29.67:44761)
I20260709 12:27:37.418098 30937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed242e2c2b843978a7af7cca5694ae2" candidate_uuid: "75f56c1a7f86421da9247e9b35990e34" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e77b2d60f57d4dceb865e440eab35126"
I20260709 12:27:37.418226 31011 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed242e2c2b843978a7af7cca5694ae2" candidate_uuid: "75f56c1a7f86421da9247e9b35990e34" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2fbc4f1ad97a4f2492306d77a614527a"
I20260709 12:27:37.418833 30937 raft_consensus.cc:3060] T 3ed242e2c2b843978a7af7cca5694ae2 P e77b2d60f57d4dceb865e440eab35126 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:37.418833 31011 raft_consensus.cc:3060] T 3ed242e2c2b843978a7af7cca5694ae2 P 2fbc4f1ad97a4f2492306d77a614527a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:37.426156 30937 raft_consensus.cc:2468] T 3ed242e2c2b843978a7af7cca5694ae2 P e77b2d60f57d4dceb865e440eab35126 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 75f56c1a7f86421da9247e9b35990e34 in term 1.
I20260709 12:27:37.426173 31011 raft_consensus.cc:2468] T 3ed242e2c2b843978a7af7cca5694ae2 P 2fbc4f1ad97a4f2492306d77a614527a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 75f56c1a7f86421da9247e9b35990e34 in term 1.
I20260709 12:27:37.427569 30827 leader_election.cc:304] T 3ed242e2c2b843978a7af7cca5694ae2 P 75f56c1a7f86421da9247e9b35990e34 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 75f56c1a7f86421da9247e9b35990e34, e77b2d60f57d4dceb865e440eab35126; no voters: 
I20260709 12:27:37.428395 31059 raft_consensus.cc:2804] T 3ed242e2c2b843978a7af7cca5694ae2 P 75f56c1a7f86421da9247e9b35990e34 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:37.429327 31059 raft_consensus.cc:697] T 3ed242e2c2b843978a7af7cca5694ae2 P 75f56c1a7f86421da9247e9b35990e34 [term 1 LEADER]: Becoming Leader. State: Replica: 75f56c1a7f86421da9247e9b35990e34, State: Running, Role: LEADER
I20260709 12:27:37.430138 31059 consensus_queue.cc:237] T 3ed242e2c2b843978a7af7cca5694ae2 P 75f56c1a7f86421da9247e9b35990e34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } }
I20260709 12:27:37.437772 30761 catalog_manager.cc:5697] T 3ed242e2c2b843978a7af7cca5694ae2 P 75f56c1a7f86421da9247e9b35990e34 reported cstate change: term changed from 0 to 1, leader changed from <none> to 75f56c1a7f86421da9247e9b35990e34 (127.16.29.65). New cstate: current_term: 1 leader_uuid: "75f56c1a7f86421da9247e9b35990e34" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:37.458002 31057 consensus_queue.cc:1048] T b555d1cdbcec4c278fd300d628a8b9ab P 2fbc4f1ad97a4f2492306d77a614527a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:27:37.468627 31113 consensus_queue.cc:1048] T b555d1cdbcec4c278fd300d628a8b9ab P 2fbc4f1ad97a4f2492306d77a614527a [LEADER]: Connected to new peer: Peer: permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:37.524080 31113 consensus_queue.cc:1048] T 0be78a9b3c5d47df8efea08d91f58111 P 2fbc4f1ad97a4f2492306d77a614527a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:27:37.535853 31066 consensus_queue.cc:1048] T 0be78a9b3c5d47df8efea08d91f58111 P 2fbc4f1ad97a4f2492306d77a614527a [LEADER]: Connected to new peer: Peer: permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:37.536993 31060 consensus_queue.cc:1048] T 2c458afcbd8848d2aa9893f4e74c8561 P e77b2d60f57d4dceb865e440eab35126 [LEADER]: Connected to new peer: Peer: permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:37.543512 31060 consensus_queue.cc:1048] T de8ccd191af64c50ac32e3342ca995b0 P e77b2d60f57d4dceb865e440eab35126 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:37.557184 31060 consensus_queue.cc:1048] T 2c458afcbd8848d2aa9893f4e74c8561 P e77b2d60f57d4dceb865e440eab35126 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:27:37.567747 31083 consensus_queue.cc:1048] T de8ccd191af64c50ac32e3342ca995b0 P e77b2d60f57d4dceb865e440eab35126 [LEADER]: Connected to new peer: Peer: permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:37.638513 16501 ts_manager.cc:295] Set tserver state for 75f56c1a7f86421da9247e9b35990e34 to MAINTENANCE_MODE
I20260709 12:27:37.639348 16501 tablet_server.cc:179] TabletServer@127.16.29.65:0 shutting down...
I20260709 12:27:37.669885 16501 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:27:37.670879 16501 tablet_replica.cc:333] T adedadb9002047ba87a533a33374f4e7 P 75f56c1a7f86421da9247e9b35990e34: stopping tablet replica
I20260709 12:27:37.671463 16501 raft_consensus.cc:2243] T adedadb9002047ba87a533a33374f4e7 P 75f56c1a7f86421da9247e9b35990e34 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:37.671931 16501 raft_consensus.cc:2272] T adedadb9002047ba87a533a33374f4e7 P 75f56c1a7f86421da9247e9b35990e34 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:37.674307 16501 tablet_replica.cc:333] T 3ed242e2c2b843978a7af7cca5694ae2 P 75f56c1a7f86421da9247e9b35990e34: stopping tablet replica
I20260709 12:27:37.674942 16501 raft_consensus.cc:2243] T 3ed242e2c2b843978a7af7cca5694ae2 P 75f56c1a7f86421da9247e9b35990e34 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:27:37.675614 16501 pending_rounds.cc:70] T 3ed242e2c2b843978a7af7cca5694ae2 P 75f56c1a7f86421da9247e9b35990e34: Trying to abort 1 pending ops.
I20260709 12:27:37.675815 16501 pending_rounds.cc:77] T 3ed242e2c2b843978a7af7cca5694ae2 P 75f56c1a7f86421da9247e9b35990e34: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7305625835241832448 op_type: NO_OP noop_request { }
I20260709 12:27:37.676183 16501 raft_consensus.cc:2889] T 3ed242e2c2b843978a7af7cca5694ae2 P 75f56c1a7f86421da9247e9b35990e34 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260709 12:27:37.676457 16501 raft_consensus.cc:2272] T 3ed242e2c2b843978a7af7cca5694ae2 P 75f56c1a7f86421da9247e9b35990e34 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:37.678879 16501 tablet_replica.cc:333] T de8ccd191af64c50ac32e3342ca995b0 P 75f56c1a7f86421da9247e9b35990e34: stopping tablet replica
I20260709 12:27:37.679625 16501 raft_consensus.cc:2243] T de8ccd191af64c50ac32e3342ca995b0 P 75f56c1a7f86421da9247e9b35990e34 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:37.680137 16501 raft_consensus.cc:2272] T de8ccd191af64c50ac32e3342ca995b0 P 75f56c1a7f86421da9247e9b35990e34 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:37.682447 31083 consensus_queue.cc:1048] T 881773ceed164ddf8ebc390254b1eb01 P e77b2d60f57d4dceb865e440eab35126 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:27:37.683171 16501 tablet_replica.cc:333] T 8e560587c5c24e4587295d2d1b6d2be6 P 75f56c1a7f86421da9247e9b35990e34: stopping tablet replica
I20260709 12:27:37.684010 16501 raft_consensus.cc:2243] T 8e560587c5c24e4587295d2d1b6d2be6 P 75f56c1a7f86421da9247e9b35990e34 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:37.684450 16501 raft_consensus.cc:2272] T 8e560587c5c24e4587295d2d1b6d2be6 P 75f56c1a7f86421da9247e9b35990e34 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:37.686822 16501 tablet_replica.cc:333] T 009d38185f3e49afbbd9faf57f968b2c P 75f56c1a7f86421da9247e9b35990e34: stopping tablet replica
I20260709 12:27:37.687422 16501 raft_consensus.cc:2243] T 009d38185f3e49afbbd9faf57f968b2c P 75f56c1a7f86421da9247e9b35990e34 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:37.687808 16501 raft_consensus.cc:2272] T 009d38185f3e49afbbd9faf57f968b2c P 75f56c1a7f86421da9247e9b35990e34 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:37.690331 16501 tablet_replica.cc:333] T f64692cd6a0347979834f29861bde5d4 P 75f56c1a7f86421da9247e9b35990e34: stopping tablet replica
I20260709 12:27:37.692772 16501 raft_consensus.cc:2243] T f64692cd6a0347979834f29861bde5d4 P 75f56c1a7f86421da9247e9b35990e34 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:37.693215 16501 raft_consensus.cc:2272] T f64692cd6a0347979834f29861bde5d4 P 75f56c1a7f86421da9247e9b35990e34 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 12:27:37.695510 30903 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 387 similar messages]
I20260709 12:27:37.695735 16501 tablet_replica.cc:333] T 2c458afcbd8848d2aa9893f4e74c8561 P 75f56c1a7f86421da9247e9b35990e34: stopping tablet replica
I20260709 12:27:37.696708 16501 raft_consensus.cc:2243] T 2c458afcbd8848d2aa9893f4e74c8561 P 75f56c1a7f86421da9247e9b35990e34 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:37.697219 16501 raft_consensus.cc:2272] T 2c458afcbd8848d2aa9893f4e74c8561 P 75f56c1a7f86421da9247e9b35990e34 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:37.700141 16501 tablet_replica.cc:333] T 881773ceed164ddf8ebc390254b1eb01 P 75f56c1a7f86421da9247e9b35990e34: stopping tablet replica
W20260709 12:27:37.700392 30903 consensus_peers.cc:597] T 881773ceed164ddf8ebc390254b1eb01 P e77b2d60f57d4dceb865e440eab35126 -> Peer 75f56c1a7f86421da9247e9b35990e34 (127.16.29.65:45249): Couldn't send request to peer 75f56c1a7f86421da9247e9b35990e34. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:37.700861 16501 raft_consensus.cc:2243] T 881773ceed164ddf8ebc390254b1eb01 P 75f56c1a7f86421da9247e9b35990e34 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260709 12:27:37.701098 30903 consensus_peers.cc:597] T 8e560587c5c24e4587295d2d1b6d2be6 P e77b2d60f57d4dceb865e440eab35126 -> Peer 75f56c1a7f86421da9247e9b35990e34 (127.16.29.65:45249): Couldn't send request to peer 75f56c1a7f86421da9247e9b35990e34. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:37.701411 16501 raft_consensus.cc:2272] T 881773ceed164ddf8ebc390254b1eb01 P 75f56c1a7f86421da9247e9b35990e34 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:37.703857 16501 tablet_replica.cc:333] T 0be78a9b3c5d47df8efea08d91f58111 P 75f56c1a7f86421da9247e9b35990e34: stopping tablet replica
I20260709 12:27:37.704586 16501 raft_consensus.cc:2243] T 0be78a9b3c5d47df8efea08d91f58111 P 75f56c1a7f86421da9247e9b35990e34 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:37.705080 16501 raft_consensus.cc:2272] T 0be78a9b3c5d47df8efea08d91f58111 P 75f56c1a7f86421da9247e9b35990e34 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:37.707476 16501 tablet_replica.cc:333] T b555d1cdbcec4c278fd300d628a8b9ab P 75f56c1a7f86421da9247e9b35990e34: stopping tablet replica
I20260709 12:27:37.707976 16501 raft_consensus.cc:2243] T b555d1cdbcec4c278fd300d628a8b9ab P 75f56c1a7f86421da9247e9b35990e34 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:37.708344 16501 raft_consensus.cc:2272] T b555d1cdbcec4c278fd300d628a8b9ab P 75f56c1a7f86421da9247e9b35990e34 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:37.739180 16501 tablet_server.cc:196] TabletServer@127.16.29.65:0 shutdown complete.
I20260709 12:27:37.855792 31066 consensus_queue.cc:1048] T f64692cd6a0347979834f29861bde5d4 P 2fbc4f1ad97a4f2492306d77a614527a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260709 12:27:37.856829 30809 auto_rebalancer.cc:288] Could not retrieve cluster info: Illegal state: not all tservers available for tablet placement
I20260709 12:27:37.869865 31122 consensus_queue.cc:1048] T 8e560587c5c24e4587295d2d1b6d2be6 P e77b2d60f57d4dceb865e440eab35126 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260709 12:27:37.879253 30977 consensus_peers.cc:597] T f64692cd6a0347979834f29861bde5d4 P 2fbc4f1ad97a4f2492306d77a614527a -> Peer 75f56c1a7f86421da9247e9b35990e34 (127.16.29.65:45249): Couldn't send request to peer 75f56c1a7f86421da9247e9b35990e34. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:45249: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:37.906805 31083 consensus_queue.cc:1048] T 009d38185f3e49afbbd9faf57f968b2c P e77b2d60f57d4dceb865e440eab35126 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:27:37.916383 31084 consensus_queue.cc:1048] T adedadb9002047ba87a533a33374f4e7 P e77b2d60f57d4dceb865e440eab35126 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260709 12:27:37.927726 30903 consensus_peers.cc:597] T 009d38185f3e49afbbd9faf57f968b2c P e77b2d60f57d4dceb865e440eab35126 -> Peer 75f56c1a7f86421da9247e9b35990e34 (127.16.29.65:45249): Couldn't send request to peer 75f56c1a7f86421da9247e9b35990e34. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:45249: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:37.939357 30903 consensus_peers.cc:597] T adedadb9002047ba87a533a33374f4e7 P e77b2d60f57d4dceb865e440eab35126 -> Peer 75f56c1a7f86421da9247e9b35990e34 (127.16.29.65:45249): Couldn't send request to peer 75f56c1a7f86421da9247e9b35990e34. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:45249: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:37.941895 30903 consensus_peers.cc:597] T 2c458afcbd8848d2aa9893f4e74c8561 P e77b2d60f57d4dceb865e440eab35126 -> Peer 75f56c1a7f86421da9247e9b35990e34 (127.16.29.65:45249): Couldn't send request to peer 75f56c1a7f86421da9247e9b35990e34. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:45249: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:37.973029 30977 consensus_peers.cc:597] T b555d1cdbcec4c278fd300d628a8b9ab P 2fbc4f1ad97a4f2492306d77a614527a -> Peer 75f56c1a7f86421da9247e9b35990e34 (127.16.29.65:45249): Couldn't send request to peer 75f56c1a7f86421da9247e9b35990e34. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:45249: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:38.019057 30903 consensus_peers.cc:597] T de8ccd191af64c50ac32e3342ca995b0 P e77b2d60f57d4dceb865e440eab35126 -> Peer 75f56c1a7f86421da9247e9b35990e34 (127.16.29.65:45249): Couldn't send request to peer 75f56c1a7f86421da9247e9b35990e34. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:45249: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:38.166641 30977 consensus_peers.cc:597] T 0be78a9b3c5d47df8efea08d91f58111 P 2fbc4f1ad97a4f2492306d77a614527a -> Peer 75f56c1a7f86421da9247e9b35990e34 (127.16.29.65:45249): Couldn't send request to peer 75f56c1a7f86421da9247e9b35990e34. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:45249: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:38.858028 30809 auto_rebalancer.cc:288] Could not retrieve cluster info: Illegal state: not all tservers available for tablet placement
I20260709 12:27:39.216142 31113 raft_consensus.cc:493] T 3ed242e2c2b843978a7af7cca5694ae2 P 2fbc4f1ad97a4f2492306d77a614527a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:39.216770 31113 raft_consensus.cc:515] T 3ed242e2c2b843978a7af7cca5694ae2 P 2fbc4f1ad97a4f2492306d77a614527a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } }
I20260709 12:27:39.218474 31113 leader_election.cc:290] T 3ed242e2c2b843978a7af7cca5694ae2 P 2fbc4f1ad97a4f2492306d77a614527a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 75f56c1a7f86421da9247e9b35990e34 (127.16.29.65:45249), e77b2d60f57d4dceb865e440eab35126 (127.16.29.66:46695)
I20260709 12:27:39.219663 30937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed242e2c2b843978a7af7cca5694ae2" candidate_uuid: "2fbc4f1ad97a4f2492306d77a614527a" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e77b2d60f57d4dceb865e440eab35126" is_pre_election: true
I20260709 12:27:39.220402 30937 raft_consensus.cc:2468] T 3ed242e2c2b843978a7af7cca5694ae2 P e77b2d60f57d4dceb865e440eab35126 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2fbc4f1ad97a4f2492306d77a614527a in term 1.
I20260709 12:27:39.221498 30976 leader_election.cc:304] T 3ed242e2c2b843978a7af7cca5694ae2 P 2fbc4f1ad97a4f2492306d77a614527a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2fbc4f1ad97a4f2492306d77a614527a, e77b2d60f57d4dceb865e440eab35126; no voters: 
I20260709 12:27:39.222205 31113 raft_consensus.cc:2804] T 3ed242e2c2b843978a7af7cca5694ae2 P 2fbc4f1ad97a4f2492306d77a614527a [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260709 12:27:39.222321 30977 leader_election.cc:336] T 3ed242e2c2b843978a7af7cca5694ae2 P 2fbc4f1ad97a4f2492306d77a614527a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 75f56c1a7f86421da9247e9b35990e34 (127.16.29.65:45249): Network error: Client connection negotiation failed: client connection to 127.16.29.65:45249: connect: Connection refused (error 111)
I20260709 12:27:39.222527 31113 raft_consensus.cc:493] T 3ed242e2c2b843978a7af7cca5694ae2 P 2fbc4f1ad97a4f2492306d77a614527a [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:39.222957 31113 raft_consensus.cc:3060] T 3ed242e2c2b843978a7af7cca5694ae2 P 2fbc4f1ad97a4f2492306d77a614527a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:39.227993 31113 raft_consensus.cc:515] T 3ed242e2c2b843978a7af7cca5694ae2 P 2fbc4f1ad97a4f2492306d77a614527a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } }
I20260709 12:27:39.229563 31113 leader_election.cc:290] T 3ed242e2c2b843978a7af7cca5694ae2 P 2fbc4f1ad97a4f2492306d77a614527a [CANDIDATE]: Term 2 election: Requested vote from peers 75f56c1a7f86421da9247e9b35990e34 (127.16.29.65:45249), e77b2d60f57d4dceb865e440eab35126 (127.16.29.66:46695)
I20260709 12:27:39.230644 30937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ed242e2c2b843978a7af7cca5694ae2" candidate_uuid: "2fbc4f1ad97a4f2492306d77a614527a" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e77b2d60f57d4dceb865e440eab35126"
I20260709 12:27:39.231212 30937 raft_consensus.cc:3060] T 3ed242e2c2b843978a7af7cca5694ae2 P e77b2d60f57d4dceb865e440eab35126 [term 1 FOLLOWER]: Advancing to term 2
W20260709 12:27:39.233919 30977 leader_election.cc:336] T 3ed242e2c2b843978a7af7cca5694ae2 P 2fbc4f1ad97a4f2492306d77a614527a [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 75f56c1a7f86421da9247e9b35990e34 (127.16.29.65:45249): Network error: Client connection negotiation failed: client connection to 127.16.29.65:45249: connect: Connection refused (error 111)
I20260709 12:27:39.236310 30937 raft_consensus.cc:2468] T 3ed242e2c2b843978a7af7cca5694ae2 P e77b2d60f57d4dceb865e440eab35126 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2fbc4f1ad97a4f2492306d77a614527a in term 2.
I20260709 12:27:39.237360 30976 leader_election.cc:304] T 3ed242e2c2b843978a7af7cca5694ae2 P 2fbc4f1ad97a4f2492306d77a614527a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2fbc4f1ad97a4f2492306d77a614527a, e77b2d60f57d4dceb865e440eab35126; no voters: 75f56c1a7f86421da9247e9b35990e34
I20260709 12:27:39.238142 31113 raft_consensus.cc:2804] T 3ed242e2c2b843978a7af7cca5694ae2 P 2fbc4f1ad97a4f2492306d77a614527a [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:27:39.239030 31113 raft_consensus.cc:697] T 3ed242e2c2b843978a7af7cca5694ae2 P 2fbc4f1ad97a4f2492306d77a614527a [term 2 LEADER]: Becoming Leader. State: Replica: 2fbc4f1ad97a4f2492306d77a614527a, State: Running, Role: LEADER
I20260709 12:27:39.239820 31113 consensus_queue.cc:237] T 3ed242e2c2b843978a7af7cca5694ae2 P 2fbc4f1ad97a4f2492306d77a614527a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } }
I20260709 12:27:39.246518 30760 catalog_manager.cc:5697] T 3ed242e2c2b843978a7af7cca5694ae2 P 2fbc4f1ad97a4f2492306d77a614527a reported cstate change: term changed from 1 to 2, leader changed from 75f56c1a7f86421da9247e9b35990e34 (127.16.29.65) to 2fbc4f1ad97a4f2492306d77a614527a (127.16.29.67). New cstate: current_term: 2 leader_uuid: "2fbc4f1ad97a4f2492306d77a614527a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } health_report { overall_health: HEALTHY } } }
W20260709 12:27:39.683792 30977 consensus_peers.cc:597] T 3ed242e2c2b843978a7af7cca5694ae2 P 2fbc4f1ad97a4f2492306d77a614527a -> Peer 75f56c1a7f86421da9247e9b35990e34 (127.16.29.65:45249): Couldn't send request to peer 75f56c1a7f86421da9247e9b35990e34. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:45249: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:39.802520 30937 raft_consensus.cc:1275] T 3ed242e2c2b843978a7af7cca5694ae2 P e77b2d60f57d4dceb865e440eab35126 [term 2 FOLLOWER]: Refusing update from remote peer 2fbc4f1ad97a4f2492306d77a614527a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260709 12:27:39.804095 31113 consensus_queue.cc:1048] T 3ed242e2c2b843978a7af7cca5694ae2 P 2fbc4f1ad97a4f2492306d77a614527a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260709 12:27:39.859136 30809 auto_rebalancer.cc:288] Could not retrieve cluster info: Illegal state: not all tservers available for tablet placement
W20260709 12:27:40.075052 30977 consensus_peers.cc:597] T f64692cd6a0347979834f29861bde5d4 P 2fbc4f1ad97a4f2492306d77a614527a -> Peer 75f56c1a7f86421da9247e9b35990e34 (127.16.29.65:45249): Couldn't send request to peer 75f56c1a7f86421da9247e9b35990e34. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:45249: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:27:40.208191 30903 consensus_peers.cc:597] T 8e560587c5c24e4587295d2d1b6d2be6 P e77b2d60f57d4dceb865e440eab35126 -> Peer 75f56c1a7f86421da9247e9b35990e34 (127.16.29.65:45249): Couldn't send request to peer 75f56c1a7f86421da9247e9b35990e34. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:45249: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:27:40.230039 30977 consensus_peers.cc:597] T b555d1cdbcec4c278fd300d628a8b9ab P 2fbc4f1ad97a4f2492306d77a614527a -> Peer 75f56c1a7f86421da9247e9b35990e34 (127.16.29.65:45249): Couldn't send request to peer 75f56c1a7f86421da9247e9b35990e34. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:45249: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:27:40.300789 30903 consensus_peers.cc:597] T adedadb9002047ba87a533a33374f4e7 P e77b2d60f57d4dceb865e440eab35126 -> Peer 75f56c1a7f86421da9247e9b35990e34 (127.16.29.65:45249): Couldn't send request to peer 75f56c1a7f86421da9247e9b35990e34. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:45249: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:27:40.306289 30903 consensus_peers.cc:597] T 2c458afcbd8848d2aa9893f4e74c8561 P e77b2d60f57d4dceb865e440eab35126 -> Peer 75f56c1a7f86421da9247e9b35990e34 (127.16.29.65:45249): Couldn't send request to peer 75f56c1a7f86421da9247e9b35990e34. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:45249: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:27:40.322746 30903 consensus_peers.cc:597] T 881773ceed164ddf8ebc390254b1eb01 P e77b2d60f57d4dceb865e440eab35126 -> Peer 75f56c1a7f86421da9247e9b35990e34 (127.16.29.65:45249): Couldn't send request to peer 75f56c1a7f86421da9247e9b35990e34. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:45249: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:27:40.424906 30903 consensus_peers.cc:597] T 009d38185f3e49afbbd9faf57f968b2c P e77b2d60f57d4dceb865e440eab35126 -> Peer 75f56c1a7f86421da9247e9b35990e34 (127.16.29.65:45249): Couldn't send request to peer 75f56c1a7f86421da9247e9b35990e34. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:45249: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:27:40.622602 30977 consensus_peers.cc:597] T 0be78a9b3c5d47df8efea08d91f58111 P 2fbc4f1ad97a4f2492306d77a614527a -> Peer 75f56c1a7f86421da9247e9b35990e34 (127.16.29.65:45249): Couldn't send request to peer 75f56c1a7f86421da9247e9b35990e34. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:45249: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:27:40.712663 30903 consensus_peers.cc:597] T de8ccd191af64c50ac32e3342ca995b0 P e77b2d60f57d4dceb865e440eab35126 -> Peer 75f56c1a7f86421da9247e9b35990e34 (127.16.29.65:45249): Couldn't send request to peer 75f56c1a7f86421da9247e9b35990e34. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:45249: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:27:40.860488 30809 auto_rebalancer.cc:288] Could not retrieve cluster info: Illegal state: not all tservers available for tablet placement
W20260709 12:27:41.861583 30809 auto_rebalancer.cc:288] Could not retrieve cluster info: Illegal state: not all tservers available for tablet placement
W20260709 12:27:42.282404 30977 consensus_peers.cc:597] T 3ed242e2c2b843978a7af7cca5694ae2 P 2fbc4f1ad97a4f2492306d77a614527a -> Peer 75f56c1a7f86421da9247e9b35990e34 (127.16.29.65:45249): Couldn't send request to peer 75f56c1a7f86421da9247e9b35990e34. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:45249: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:27:42.569895 30903 consensus_peers.cc:597] T 881773ceed164ddf8ebc390254b1eb01 P e77b2d60f57d4dceb865e440eab35126 -> Peer 75f56c1a7f86421da9247e9b35990e34 (127.16.29.65:45249): Couldn't send request to peer 75f56c1a7f86421da9247e9b35990e34. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:45249: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:27:42.572768 30977 consensus_peers.cc:597] T b555d1cdbcec4c278fd300d628a8b9ab P 2fbc4f1ad97a4f2492306d77a614527a -> Peer 75f56c1a7f86421da9247e9b35990e34 (127.16.29.65:45249): Couldn't send request to peer 75f56c1a7f86421da9247e9b35990e34. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:45249: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:27:42.628415 30903 consensus_peers.cc:597] T 009d38185f3e49afbbd9faf57f968b2c P e77b2d60f57d4dceb865e440eab35126 -> Peer 75f56c1a7f86421da9247e9b35990e34 (127.16.29.65:45249): Couldn't send request to peer 75f56c1a7f86421da9247e9b35990e34. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:45249: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:27:42.649003 30903 consensus_peers.cc:597] T 8e560587c5c24e4587295d2d1b6d2be6 P e77b2d60f57d4dceb865e440eab35126 -> Peer 75f56c1a7f86421da9247e9b35990e34 (127.16.29.65:45249): Couldn't send request to peer 75f56c1a7f86421da9247e9b35990e34. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:45249: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:27:42.696223 30903 consensus_peers.cc:597] T 2c458afcbd8848d2aa9893f4e74c8561 P e77b2d60f57d4dceb865e440eab35126 -> Peer 75f56c1a7f86421da9247e9b35990e34 (127.16.29.65:45249): Couldn't send request to peer 75f56c1a7f86421da9247e9b35990e34. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:45249: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:27:42.715943 30977 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.29.65:45249: connect: Connection refused (error 111) [suppressed 101 similar messages]
W20260709 12:27:42.757226 30977 consensus_peers.cc:597] T f64692cd6a0347979834f29861bde5d4 P 2fbc4f1ad97a4f2492306d77a614527a -> Peer 75f56c1a7f86421da9247e9b35990e34 (127.16.29.65:45249): Couldn't send request to peer 75f56c1a7f86421da9247e9b35990e34. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:45249: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 12:27:42.759234 16501 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:27:42.767493 31147 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:27:42.768327 31148 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:27:42.772867 31150 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:27:42.776589 16501 server_base.cc:1061] running on GCE node
I20260709 12:27:42.779469 16501 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:27:42.779757 16501 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:27:42.779987 16501 hybrid_clock.cc:648] HybridClock initialized: now 1783600062779964 us; error 0 us; skew 500 ppm
I20260709 12:27:42.780577 16501 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:27:42.783416 16501 webserver.cc:533] Webserver started at http://127.16.29.65:34811/ using document root <none> and password file <none>
I20260709 12:27:42.784071 16501 fs_manager.cc:362] Metadata directory not provided
I20260709 12:27:42.784283 16501 fs_manager.cc:365] Using existing metadata directory in first data directory
I20260709 12:27:42.788444 16501 fs_manager.cc:714] Time spent opening directory manager: real 0.003s	user 0.003s	sys 0.000s
I20260709 12:27:42.792754 31155 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:27:42.793735 16501 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.000s	sys 0.004s
I20260709 12:27:42.794082 16501 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783599551912677-16501-0/minicluster-data/ts-0-root
uuid: "75f56c1a7f86421da9247e9b35990e34"
format_stamp: "Formatted at 2026-07-09 12:27:35 on dist-test-slave-5n9v"
I20260709 12:27:42.794524 16501 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783599551912677-16501-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783599551912677-16501-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783599551912677-16501-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:27:42.824066 16501 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:27:42.825363 16501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:27:42.827181 16501 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:27:42.831090 31162 ts_tablet_manager.cc:548] Loading tablet metadata (0/10 complete)
W20260709 12:27:42.862448 30809 auto_rebalancer.cc:288] Could not retrieve cluster info: Illegal state: not all tservers available for tablet placement
I20260709 12:27:42.884893 16501 ts_tablet_manager.cc:585] Loaded tablet metadata (10 total tablets, 10 live tablets)
I20260709 12:27:42.885326 16501 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.055s	user 0.003s	sys 0.000s
I20260709 12:27:42.885733 16501 ts_tablet_manager.cc:600] Registering tablets (0/10 complete)
I20260709 12:27:42.891446 31162 tablet_bootstrap.cc:492] T 009d38185f3e49afbbd9faf57f968b2c P 75f56c1a7f86421da9247e9b35990e34: Bootstrap starting.
I20260709 12:27:42.904553 31162 tablet_bootstrap.cc:492] T 009d38185f3e49afbbd9faf57f968b2c P 75f56c1a7f86421da9247e9b35990e34: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:27:42.905308 31162 tablet_bootstrap.cc:492] T 009d38185f3e49afbbd9faf57f968b2c P 75f56c1a7f86421da9247e9b35990e34: Bootstrap complete.
I20260709 12:27:42.905891 31162 ts_tablet_manager.cc:1403] T 009d38185f3e49afbbd9faf57f968b2c P 75f56c1a7f86421da9247e9b35990e34: Time spent bootstrapping tablet: real 0.015s	user 0.012s	sys 0.000s
I20260709 12:27:42.908288 31162 raft_consensus.cc:359] T 009d38185f3e49afbbd9faf57f968b2c P 75f56c1a7f86421da9247e9b35990e34 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } }
I20260709 12:27:42.909104 31162 raft_consensus.cc:740] T 009d38185f3e49afbbd9faf57f968b2c P 75f56c1a7f86421da9247e9b35990e34 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75f56c1a7f86421da9247e9b35990e34, State: Initialized, Role: FOLLOWER
I20260709 12:27:42.909816 31162 consensus_queue.cc:260] T 009d38185f3e49afbbd9faf57f968b2c P 75f56c1a7f86421da9247e9b35990e34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } }
I20260709 12:27:42.913704 31162 ts_tablet_manager.cc:1434] T 009d38185f3e49afbbd9faf57f968b2c P 75f56c1a7f86421da9247e9b35990e34: Time spent starting tablet: real 0.007s	user 0.008s	sys 0.000s
I20260709 12:27:42.914584 31162 tablet_bootstrap.cc:492] T 3ed242e2c2b843978a7af7cca5694ae2 P 75f56c1a7f86421da9247e9b35990e34: Bootstrap starting.
W20260709 12:27:42.914693 30903 consensus_peers.cc:597] T adedadb9002047ba87a533a33374f4e7 P e77b2d60f57d4dceb865e440eab35126 -> Peer 75f56c1a7f86421da9247e9b35990e34 (127.16.29.65:45249): Couldn't send request to peer 75f56c1a7f86421da9247e9b35990e34. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:45249: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 12:27:42.931299 31162 tablet_bootstrap.cc:492] T 3ed242e2c2b843978a7af7cca5694ae2 P 75f56c1a7f86421da9247e9b35990e34: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20260709 12:27:42.932377 31162 tablet_bootstrap.cc:492] T 3ed242e2c2b843978a7af7cca5694ae2 P 75f56c1a7f86421da9247e9b35990e34: Bootstrap complete.
I20260709 12:27:42.933055 31162 ts_tablet_manager.cc:1403] T 3ed242e2c2b843978a7af7cca5694ae2 P 75f56c1a7f86421da9247e9b35990e34: Time spent bootstrapping tablet: real 0.019s	user 0.014s	sys 0.004s
I20260709 12:27:42.935916 31162 raft_consensus.cc:359] T 3ed242e2c2b843978a7af7cca5694ae2 P 75f56c1a7f86421da9247e9b35990e34 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } }
I20260709 12:27:42.936826 31162 raft_consensus.cc:740] T 3ed242e2c2b843978a7af7cca5694ae2 P 75f56c1a7f86421da9247e9b35990e34 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75f56c1a7f86421da9247e9b35990e34, State: Initialized, Role: FOLLOWER
I20260709 12:27:42.937551 31162 consensus_queue.cc:260] T 3ed242e2c2b843978a7af7cca5694ae2 P 75f56c1a7f86421da9247e9b35990e34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } }
I20260709 12:27:42.939507 31162 ts_tablet_manager.cc:1434] T 3ed242e2c2b843978a7af7cca5694ae2 P 75f56c1a7f86421da9247e9b35990e34: Time spent starting tablet: real 0.006s	user 0.004s	sys 0.000s
I20260709 12:27:42.940042 16501 ts_tablet_manager.cc:616] Registered 10 tablets
I20260709 12:27:42.940330 16501 ts_tablet_manager.cc:595] Time spent register tablets: real 0.055s	user 0.046s	sys 0.009s
I20260709 12:27:42.940385 31162 tablet_bootstrap.cc:492] T 2c458afcbd8848d2aa9893f4e74c8561 P 75f56c1a7f86421da9247e9b35990e34: Bootstrap starting.
I20260709 12:27:42.961722 31162 tablet_bootstrap.cc:492] T 2c458afcbd8848d2aa9893f4e74c8561 P 75f56c1a7f86421da9247e9b35990e34: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:27:42.962927 31162 tablet_bootstrap.cc:492] T 2c458afcbd8848d2aa9893f4e74c8561 P 75f56c1a7f86421da9247e9b35990e34: Bootstrap complete.
I20260709 12:27:42.963749 31162 ts_tablet_manager.cc:1403] T 2c458afcbd8848d2aa9893f4e74c8561 P 75f56c1a7f86421da9247e9b35990e34: Time spent bootstrapping tablet: real 0.024s	user 0.011s	sys 0.008s
I20260709 12:27:42.967195 31162 raft_consensus.cc:359] T 2c458afcbd8848d2aa9893f4e74c8561 P 75f56c1a7f86421da9247e9b35990e34 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } }
I20260709 12:27:42.968098 31162 raft_consensus.cc:740] T 2c458afcbd8848d2aa9893f4e74c8561 P 75f56c1a7f86421da9247e9b35990e34 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75f56c1a7f86421da9247e9b35990e34, State: Initialized, Role: FOLLOWER
I20260709 12:27:42.968931 31162 consensus_queue.cc:260] T 2c458afcbd8848d2aa9893f4e74c8561 P 75f56c1a7f86421da9247e9b35990e34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } }
I20260709 12:27:42.971597 31162 ts_tablet_manager.cc:1434] T 2c458afcbd8848d2aa9893f4e74c8561 P 75f56c1a7f86421da9247e9b35990e34: Time spent starting tablet: real 0.007s	user 0.001s	sys 0.004s
I20260709 12:27:42.972594 31162 tablet_bootstrap.cc:492] T 0be78a9b3c5d47df8efea08d91f58111 P 75f56c1a7f86421da9247e9b35990e34: Bootstrap starting.
I20260709 12:27:42.990991 31162 tablet_bootstrap.cc:492] T 0be78a9b3c5d47df8efea08d91f58111 P 75f56c1a7f86421da9247e9b35990e34: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:27:42.992151 31162 tablet_bootstrap.cc:492] T 0be78a9b3c5d47df8efea08d91f58111 P 75f56c1a7f86421da9247e9b35990e34: Bootstrap complete.
I20260709 12:27:42.992928 31162 ts_tablet_manager.cc:1403] T 0be78a9b3c5d47df8efea08d91f58111 P 75f56c1a7f86421da9247e9b35990e34: Time spent bootstrapping tablet: real 0.021s	user 0.015s	sys 0.004s
I20260709 12:27:42.996080 31162 raft_consensus.cc:359] T 0be78a9b3c5d47df8efea08d91f58111 P 75f56c1a7f86421da9247e9b35990e34 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } }
I20260709 12:27:42.996980 31162 raft_consensus.cc:740] T 0be78a9b3c5d47df8efea08d91f58111 P 75f56c1a7f86421da9247e9b35990e34 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75f56c1a7f86421da9247e9b35990e34, State: Initialized, Role: FOLLOWER
I20260709 12:27:42.997738 31162 consensus_queue.cc:260] T 0be78a9b3c5d47df8efea08d91f58111 P 75f56c1a7f86421da9247e9b35990e34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } }
I20260709 12:27:43.000351 31162 ts_tablet_manager.cc:1434] T 0be78a9b3c5d47df8efea08d91f58111 P 75f56c1a7f86421da9247e9b35990e34: Time spent starting tablet: real 0.007s	user 0.008s	sys 0.000s
I20260709 12:27:43.001310 31162 tablet_bootstrap.cc:492] T adedadb9002047ba87a533a33374f4e7 P 75f56c1a7f86421da9247e9b35990e34: Bootstrap starting.
I20260709 12:27:43.001487 16501 rpc_server.cc:307] RPC server started. Bound to: 127.16.29.65:45249
I20260709 12:27:43.001721 31224 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.29.65:45249 every 8 connection(s)
I20260709 12:27:43.007661 16501 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:27:43.020608 31162 tablet_bootstrap.cc:492] T adedadb9002047ba87a533a33374f4e7 P 75f56c1a7f86421da9247e9b35990e34: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:27:43.022235 31162 tablet_bootstrap.cc:492] T adedadb9002047ba87a533a33374f4e7 P 75f56c1a7f86421da9247e9b35990e34: Bootstrap complete.
I20260709 12:27:43.023298 31162 ts_tablet_manager.cc:1403] T adedadb9002047ba87a533a33374f4e7 P 75f56c1a7f86421da9247e9b35990e34: Time spent bootstrapping tablet: real 0.022s	user 0.015s	sys 0.004s
I20260709 12:27:43.027401 31162 raft_consensus.cc:359] T adedadb9002047ba87a533a33374f4e7 P 75f56c1a7f86421da9247e9b35990e34 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } }
I20260709 12:27:43.029325 31162 raft_consensus.cc:740] T adedadb9002047ba87a533a33374f4e7 P 75f56c1a7f86421da9247e9b35990e34 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75f56c1a7f86421da9247e9b35990e34, State: Initialized, Role: FOLLOWER
I20260709 12:27:43.031991 31162 consensus_queue.cc:260] T adedadb9002047ba87a533a33374f4e7 P 75f56c1a7f86421da9247e9b35990e34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } }
I20260709 12:27:43.033911 31162 ts_tablet_manager.cc:1434] T adedadb9002047ba87a533a33374f4e7 P 75f56c1a7f86421da9247e9b35990e34: Time spent starting tablet: real 0.010s	user 0.005s	sys 0.000s
I20260709 12:27:43.034893 31162 tablet_bootstrap.cc:492] T 881773ceed164ddf8ebc390254b1eb01 P 75f56c1a7f86421da9247e9b35990e34: Bootstrap starting.
I20260709 12:27:43.043466 30937 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "009d38185f3e49afbbd9faf57f968b2c"
dest_uuid: "e77b2d60f57d4dceb865e440eab35126"
mode: GRACEFUL
new_leader_uuid: "2fbc4f1ad97a4f2492306d77a614527a"
 from {username='slave'} at 127.0.0.1:33634
I20260709 12:27:43.044461 30937 raft_consensus.cc:606] T 009d38185f3e49afbbd9faf57f968b2c P e77b2d60f57d4dceb865e440eab35126 [term 1 LEADER]: Received request to transfer leadership to TS 2fbc4f1ad97a4f2492306d77a614527a
I20260709 12:27:43.044739 31225 heartbeater.cc:344] Connected to a master server at 127.16.29.126:44357
I20260709 12:27:43.045250 31225 heartbeater.cc:461] Registering TS with master...
I20260709 12:27:43.046263 31225 heartbeater.cc:507] Master 127.16.29.126:44357 requested a full tablet report, sending...
I20260709 12:27:43.047449 30937 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "8e560587c5c24e4587295d2d1b6d2be6"
dest_uuid: "e77b2d60f57d4dceb865e440eab35126"
mode: GRACEFUL
new_leader_uuid: "2fbc4f1ad97a4f2492306d77a614527a"
 from {username='slave'} at 127.0.0.1:33634
I20260709 12:27:43.048069 30937 raft_consensus.cc:606] T 8e560587c5c24e4587295d2d1b6d2be6 P e77b2d60f57d4dceb865e440eab35126 [term 1 LEADER]: Received request to transfer leadership to TS 2fbc4f1ad97a4f2492306d77a614527a
I20260709 12:27:43.048622 31162 tablet_bootstrap.cc:492] T 881773ceed164ddf8ebc390254b1eb01 P 75f56c1a7f86421da9247e9b35990e34: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:27:43.049479 16501 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260709 12:27:43.049592 31162 tablet_bootstrap.cc:492] T 881773ceed164ddf8ebc390254b1eb01 P 75f56c1a7f86421da9247e9b35990e34: Bootstrap complete.
I20260709 12:27:43.050241 16501 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:27:43.050446 31162 ts_tablet_manager.cc:1403] T 881773ceed164ddf8ebc390254b1eb01 P 75f56c1a7f86421da9247e9b35990e34: Time spent bootstrapping tablet: real 0.016s	user 0.014s	sys 0.000s
I20260709 12:27:43.051102 16501 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:27:43.052670 30759 ts_manager.cc:194] Re-registered known tserver with Master: 75f56c1a7f86421da9247e9b35990e34 (127.16.29.65:45249)
I20260709 12:27:43.053473 31162 raft_consensus.cc:359] T 881773ceed164ddf8ebc390254b1eb01 P 75f56c1a7f86421da9247e9b35990e34 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } }
I20260709 12:27:43.054210 31162 raft_consensus.cc:740] T 881773ceed164ddf8ebc390254b1eb01 P 75f56c1a7f86421da9247e9b35990e34 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75f56c1a7f86421da9247e9b35990e34, State: Initialized, Role: FOLLOWER
I20260709 12:27:43.054968 31162 consensus_queue.cc:260] T 881773ceed164ddf8ebc390254b1eb01 P 75f56c1a7f86421da9247e9b35990e34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } }
I20260709 12:27:43.057041 31162 ts_tablet_manager.cc:1434] T 881773ceed164ddf8ebc390254b1eb01 P 75f56c1a7f86421da9247e9b35990e34: Time spent starting tablet: real 0.006s	user 0.006s	sys 0.000s
I20260709 12:27:43.057853 31162 tablet_bootstrap.cc:492] T f64692cd6a0347979834f29861bde5d4 P 75f56c1a7f86421da9247e9b35990e34: Bootstrap starting.
I20260709 12:27:43.063365 30759 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:37880
I20260709 12:27:43.066534 31225 heartbeater.cc:499] Master 127.16.29.126:44357 was elected leader, sending a full tablet report...
I20260709 12:27:43.069996 31162 tablet_bootstrap.cc:492] T f64692cd6a0347979834f29861bde5d4 P 75f56c1a7f86421da9247e9b35990e34: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:27:43.070791 31162 tablet_bootstrap.cc:492] T f64692cd6a0347979834f29861bde5d4 P 75f56c1a7f86421da9247e9b35990e34: Bootstrap complete.
I20260709 12:27:43.071573 31162 ts_tablet_manager.cc:1403] T f64692cd6a0347979834f29861bde5d4 P 75f56c1a7f86421da9247e9b35990e34: Time spent bootstrapping tablet: real 0.014s	user 0.012s	sys 0.000s
I20260709 12:27:43.074077 31162 raft_consensus.cc:359] T f64692cd6a0347979834f29861bde5d4 P 75f56c1a7f86421da9247e9b35990e34 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } }
I20260709 12:27:43.074785 31162 raft_consensus.cc:740] T f64692cd6a0347979834f29861bde5d4 P 75f56c1a7f86421da9247e9b35990e34 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75f56c1a7f86421da9247e9b35990e34, State: Initialized, Role: FOLLOWER
I20260709 12:27:43.075515 31162 consensus_queue.cc:260] T f64692cd6a0347979834f29861bde5d4 P 75f56c1a7f86421da9247e9b35990e34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } }
I20260709 12:27:43.077170 31162 ts_tablet_manager.cc:1434] T f64692cd6a0347979834f29861bde5d4 P 75f56c1a7f86421da9247e9b35990e34: Time spent starting tablet: real 0.005s	user 0.004s	sys 0.000s
I20260709 12:27:43.078058 31162 tablet_bootstrap.cc:492] T b555d1cdbcec4c278fd300d628a8b9ab P 75f56c1a7f86421da9247e9b35990e34: Bootstrap starting.
I20260709 12:27:43.097819 31162 tablet_bootstrap.cc:492] T b555d1cdbcec4c278fd300d628a8b9ab P 75f56c1a7f86421da9247e9b35990e34: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:27:43.098913 31162 tablet_bootstrap.cc:492] T b555d1cdbcec4c278fd300d628a8b9ab P 75f56c1a7f86421da9247e9b35990e34: Bootstrap complete.
I20260709 12:27:43.099782 31162 ts_tablet_manager.cc:1403] T b555d1cdbcec4c278fd300d628a8b9ab P 75f56c1a7f86421da9247e9b35990e34: Time spent bootstrapping tablet: real 0.022s	user 0.014s	sys 0.004s
I20260709 12:27:43.102828 31162 raft_consensus.cc:359] T b555d1cdbcec4c278fd300d628a8b9ab P 75f56c1a7f86421da9247e9b35990e34 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } }
I20260709 12:27:43.103590 31162 raft_consensus.cc:740] T b555d1cdbcec4c278fd300d628a8b9ab P 75f56c1a7f86421da9247e9b35990e34 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75f56c1a7f86421da9247e9b35990e34, State: Initialized, Role: FOLLOWER
I20260709 12:27:43.104355 31162 consensus_queue.cc:260] T b555d1cdbcec4c278fd300d628a8b9ab P 75f56c1a7f86421da9247e9b35990e34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } }
I20260709 12:27:43.109627 31162 ts_tablet_manager.cc:1434] T b555d1cdbcec4c278fd300d628a8b9ab P 75f56c1a7f86421da9247e9b35990e34: Time spent starting tablet: real 0.010s	user 0.001s	sys 0.004s
I20260709 12:27:43.110611 31162 tablet_bootstrap.cc:492] T de8ccd191af64c50ac32e3342ca995b0 P 75f56c1a7f86421da9247e9b35990e34: Bootstrap starting.
I20260709 12:27:43.125515 31162 tablet_bootstrap.cc:492] T de8ccd191af64c50ac32e3342ca995b0 P 75f56c1a7f86421da9247e9b35990e34: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:27:43.126315 31162 tablet_bootstrap.cc:492] T de8ccd191af64c50ac32e3342ca995b0 P 75f56c1a7f86421da9247e9b35990e34: Bootstrap complete.
I20260709 12:27:43.126948 31162 ts_tablet_manager.cc:1403] T de8ccd191af64c50ac32e3342ca995b0 P 75f56c1a7f86421da9247e9b35990e34: Time spent bootstrapping tablet: real 0.017s	user 0.005s	sys 0.008s
I20260709 12:27:43.129200 31162 raft_consensus.cc:359] T de8ccd191af64c50ac32e3342ca995b0 P 75f56c1a7f86421da9247e9b35990e34 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } }
I20260709 12:27:43.129698 31162 raft_consensus.cc:740] T de8ccd191af64c50ac32e3342ca995b0 P 75f56c1a7f86421da9247e9b35990e34 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75f56c1a7f86421da9247e9b35990e34, State: Initialized, Role: FOLLOWER
I20260709 12:27:43.130318 31162 consensus_queue.cc:260] T de8ccd191af64c50ac32e3342ca995b0 P 75f56c1a7f86421da9247e9b35990e34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } }
I20260709 12:27:43.132040 31162 ts_tablet_manager.cc:1434] T de8ccd191af64c50ac32e3342ca995b0 P 75f56c1a7f86421da9247e9b35990e34: Time spent starting tablet: real 0.005s	user 0.008s	sys 0.000s
I20260709 12:27:43.132876 31162 tablet_bootstrap.cc:492] T 8e560587c5c24e4587295d2d1b6d2be6 P 75f56c1a7f86421da9247e9b35990e34: Bootstrap starting.
I20260709 12:27:43.146062 31162 tablet_bootstrap.cc:492] T 8e560587c5c24e4587295d2d1b6d2be6 P 75f56c1a7f86421da9247e9b35990e34: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:27:43.146890 31162 tablet_bootstrap.cc:492] T 8e560587c5c24e4587295d2d1b6d2be6 P 75f56c1a7f86421da9247e9b35990e34: Bootstrap complete.
I20260709 12:27:43.147462 31162 ts_tablet_manager.cc:1403] T 8e560587c5c24e4587295d2d1b6d2be6 P 75f56c1a7f86421da9247e9b35990e34: Time spent bootstrapping tablet: real 0.015s	user 0.008s	sys 0.004s
I20260709 12:27:43.149808 31162 raft_consensus.cc:359] T 8e560587c5c24e4587295d2d1b6d2be6 P 75f56c1a7f86421da9247e9b35990e34 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } }
I20260709 12:27:43.150465 31162 raft_consensus.cc:740] T 8e560587c5c24e4587295d2d1b6d2be6 P 75f56c1a7f86421da9247e9b35990e34 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75f56c1a7f86421da9247e9b35990e34, State: Initialized, Role: FOLLOWER
I20260709 12:27:43.151131 31162 consensus_queue.cc:260] T 8e560587c5c24e4587295d2d1b6d2be6 P 75f56c1a7f86421da9247e9b35990e34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } }
I20260709 12:27:43.153048 31162 ts_tablet_manager.cc:1434] T 8e560587c5c24e4587295d2d1b6d2be6 P 75f56c1a7f86421da9247e9b35990e34: Time spent starting tablet: real 0.005s	user 0.007s	sys 0.000s
I20260709 12:27:43.213047 31145 raft_consensus.cc:993] T 8e560587c5c24e4587295d2d1b6d2be6 P e77b2d60f57d4dceb865e440eab35126: : Instructing follower 2fbc4f1ad97a4f2492306d77a614527a to start an election
I20260709 12:27:43.213694 31167 raft_consensus.cc:1081] T 8e560587c5c24e4587295d2d1b6d2be6 P e77b2d60f57d4dceb865e440eab35126 [term 1 LEADER]: Signalling peer 2fbc4f1ad97a4f2492306d77a614527a to start an election
I20260709 12:27:43.215366 31011 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "8e560587c5c24e4587295d2d1b6d2be6"
dest_uuid: "2fbc4f1ad97a4f2492306d77a614527a"
 from {username='slave'} at 127.0.0.1:39866
I20260709 12:27:43.215922 31011 raft_consensus.cc:493] T 8e560587c5c24e4587295d2d1b6d2be6 P 2fbc4f1ad97a4f2492306d77a614527a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:27:43.216298 31011 raft_consensus.cc:3060] T 8e560587c5c24e4587295d2d1b6d2be6 P 2fbc4f1ad97a4f2492306d77a614527a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:43.222944 31011 raft_consensus.cc:515] T 8e560587c5c24e4587295d2d1b6d2be6 P 2fbc4f1ad97a4f2492306d77a614527a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } }
I20260709 12:27:43.224728 31011 leader_election.cc:290] T 8e560587c5c24e4587295d2d1b6d2be6 P 2fbc4f1ad97a4f2492306d77a614527a [CANDIDATE]: Term 2 election: Requested vote from peers 75f56c1a7f86421da9247e9b35990e34 (127.16.29.65:45249), e77b2d60f57d4dceb865e440eab35126 (127.16.29.66:46695)
I20260709 12:27:43.225940 31198 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e560587c5c24e4587295d2d1b6d2be6" candidate_uuid: "2fbc4f1ad97a4f2492306d77a614527a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "75f56c1a7f86421da9247e9b35990e34"
I20260709 12:27:43.226290 30937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e560587c5c24e4587295d2d1b6d2be6" candidate_uuid: "2fbc4f1ad97a4f2492306d77a614527a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e77b2d60f57d4dceb865e440eab35126"
I20260709 12:27:43.226775 31198 raft_consensus.cc:3060] T 8e560587c5c24e4587295d2d1b6d2be6 P 75f56c1a7f86421da9247e9b35990e34 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:43.226899 30937 raft_consensus.cc:3055] T 8e560587c5c24e4587295d2d1b6d2be6 P e77b2d60f57d4dceb865e440eab35126 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:27:43.227332 30937 raft_consensus.cc:740] T 8e560587c5c24e4587295d2d1b6d2be6 P e77b2d60f57d4dceb865e440eab35126 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e77b2d60f57d4dceb865e440eab35126, State: Running, Role: LEADER
I20260709 12:27:43.228160 30937 consensus_queue.cc:260] T 8e560587c5c24e4587295d2d1b6d2be6 P e77b2d60f57d4dceb865e440eab35126 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } }
I20260709 12:27:43.229214 30937 raft_consensus.cc:3060] T 8e560587c5c24e4587295d2d1b6d2be6 P e77b2d60f57d4dceb865e440eab35126 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:43.234218 31198 raft_consensus.cc:2468] T 8e560587c5c24e4587295d2d1b6d2be6 P 75f56c1a7f86421da9247e9b35990e34 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2fbc4f1ad97a4f2492306d77a614527a in term 2.
I20260709 12:27:43.235963 30977 leader_election.cc:304] T 8e560587c5c24e4587295d2d1b6d2be6 P 2fbc4f1ad97a4f2492306d77a614527a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2fbc4f1ad97a4f2492306d77a614527a, 75f56c1a7f86421da9247e9b35990e34; no voters: 
I20260709 12:27:43.237157 31141 raft_consensus.cc:2804] T 8e560587c5c24e4587295d2d1b6d2be6 P 2fbc4f1ad97a4f2492306d77a614527a [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:27:43.237382 30937 raft_consensus.cc:2468] T 8e560587c5c24e4587295d2d1b6d2be6 P e77b2d60f57d4dceb865e440eab35126 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2fbc4f1ad97a4f2492306d77a614527a in term 2.
I20260709 12:27:43.238744 31141 raft_consensus.cc:697] T 8e560587c5c24e4587295d2d1b6d2be6 P 2fbc4f1ad97a4f2492306d77a614527a [term 2 LEADER]: Becoming Leader. State: Replica: 2fbc4f1ad97a4f2492306d77a614527a, State: Running, Role: LEADER
I20260709 12:27:43.239985 31141 consensus_queue.cc:237] T 8e560587c5c24e4587295d2d1b6d2be6 P 2fbc4f1ad97a4f2492306d77a614527a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } }
I20260709 12:27:43.248562 30759 catalog_manager.cc:5697] T 8e560587c5c24e4587295d2d1b6d2be6 P 2fbc4f1ad97a4f2492306d77a614527a reported cstate change: term changed from 1 to 2, leader changed from e77b2d60f57d4dceb865e440eab35126 (127.16.29.66) to 2fbc4f1ad97a4f2492306d77a614527a (127.16.29.67). New cstate: current_term: 2 leader_uuid: "2fbc4f1ad97a4f2492306d77a614527a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } health_report { overall_health: HEALTHY } } }
I20260709 12:27:43.300776 31145 raft_consensus.cc:993] T 009d38185f3e49afbbd9faf57f968b2c P e77b2d60f57d4dceb865e440eab35126: : Instructing follower 2fbc4f1ad97a4f2492306d77a614527a to start an election
I20260709 12:27:43.301328 31145 raft_consensus.cc:1081] T 009d38185f3e49afbbd9faf57f968b2c P e77b2d60f57d4dceb865e440eab35126 [term 1 LEADER]: Signalling peer 2fbc4f1ad97a4f2492306d77a614527a to start an election
I20260709 12:27:43.302994 31011 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "009d38185f3e49afbbd9faf57f968b2c"
dest_uuid: "2fbc4f1ad97a4f2492306d77a614527a"
 from {username='slave'} at 127.0.0.1:39866
I20260709 12:27:43.303572 31011 raft_consensus.cc:493] T 009d38185f3e49afbbd9faf57f968b2c P 2fbc4f1ad97a4f2492306d77a614527a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:27:43.303946 31011 raft_consensus.cc:3060] T 009d38185f3e49afbbd9faf57f968b2c P 2fbc4f1ad97a4f2492306d77a614527a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:43.310011 31011 raft_consensus.cc:515] T 009d38185f3e49afbbd9faf57f968b2c P 2fbc4f1ad97a4f2492306d77a614527a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } }
I20260709 12:27:43.311887 31011 leader_election.cc:290] T 009d38185f3e49afbbd9faf57f968b2c P 2fbc4f1ad97a4f2492306d77a614527a [CANDIDATE]: Term 2 election: Requested vote from peers 75f56c1a7f86421da9247e9b35990e34 (127.16.29.65:45249), e77b2d60f57d4dceb865e440eab35126 (127.16.29.66:46695)
I20260709 12:27:43.312969 31198 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "009d38185f3e49afbbd9faf57f968b2c" candidate_uuid: "2fbc4f1ad97a4f2492306d77a614527a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "75f56c1a7f86421da9247e9b35990e34"
I20260709 12:27:43.313047 30937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "009d38185f3e49afbbd9faf57f968b2c" candidate_uuid: "2fbc4f1ad97a4f2492306d77a614527a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e77b2d60f57d4dceb865e440eab35126"
I20260709 12:27:43.313669 31198 raft_consensus.cc:3060] T 009d38185f3e49afbbd9faf57f968b2c P 75f56c1a7f86421da9247e9b35990e34 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:43.313920 30937 raft_consensus.cc:3055] T 009d38185f3e49afbbd9faf57f968b2c P e77b2d60f57d4dceb865e440eab35126 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:27:43.314332 30937 raft_consensus.cc:740] T 009d38185f3e49afbbd9faf57f968b2c P e77b2d60f57d4dceb865e440eab35126 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e77b2d60f57d4dceb865e440eab35126, State: Running, Role: LEADER
I20260709 12:27:43.315158 30937 consensus_queue.cc:260] T 009d38185f3e49afbbd9faf57f968b2c P e77b2d60f57d4dceb865e440eab35126 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } }
I20260709 12:27:43.316320 30937 raft_consensus.cc:3060] T 009d38185f3e49afbbd9faf57f968b2c P e77b2d60f57d4dceb865e440eab35126 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:43.319686 31198 raft_consensus.cc:2468] T 009d38185f3e49afbbd9faf57f968b2c P 75f56c1a7f86421da9247e9b35990e34 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2fbc4f1ad97a4f2492306d77a614527a in term 2.
I20260709 12:27:43.320937 30977 leader_election.cc:304] T 009d38185f3e49afbbd9faf57f968b2c P 2fbc4f1ad97a4f2492306d77a614527a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2fbc4f1ad97a4f2492306d77a614527a, 75f56c1a7f86421da9247e9b35990e34; no voters: 
I20260709 12:27:43.321753 31141 raft_consensus.cc:2804] T 009d38185f3e49afbbd9faf57f968b2c P 2fbc4f1ad97a4f2492306d77a614527a [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:27:43.322242 30937 raft_consensus.cc:2468] T 009d38185f3e49afbbd9faf57f968b2c P e77b2d60f57d4dceb865e440eab35126 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2fbc4f1ad97a4f2492306d77a614527a in term 2.
I20260709 12:27:43.322317 31141 raft_consensus.cc:697] T 009d38185f3e49afbbd9faf57f968b2c P 2fbc4f1ad97a4f2492306d77a614527a [term 2 LEADER]: Becoming Leader. State: Replica: 2fbc4f1ad97a4f2492306d77a614527a, State: Running, Role: LEADER
I20260709 12:27:43.323922 31141 consensus_queue.cc:237] T 009d38185f3e49afbbd9faf57f968b2c P 2fbc4f1ad97a4f2492306d77a614527a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } }
I20260709 12:27:43.326784 31198 raft_consensus.cc:3060] T 3ed242e2c2b843978a7af7cca5694ae2 P 75f56c1a7f86421da9247e9b35990e34 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:43.334365 31198 pending_rounds.cc:85] T 3ed242e2c2b843978a7af7cca5694ae2 P 75f56c1a7f86421da9247e9b35990e34: Aborting all ops after (but not including) 0
I20260709 12:27:43.334740 31198 pending_rounds.cc:107] T 3ed242e2c2b843978a7af7cca5694ae2 P 75f56c1a7f86421da9247e9b35990e34: Aborting uncommitted NO_OP operation due to leader change: 1.1
I20260709 12:27:43.333935 30759 catalog_manager.cc:5697] T 009d38185f3e49afbbd9faf57f968b2c P 2fbc4f1ad97a4f2492306d77a614527a reported cstate change: term changed from 1 to 2, leader changed from e77b2d60f57d4dceb865e440eab35126 (127.16.29.66) to 2fbc4f1ad97a4f2492306d77a614527a (127.16.29.67). New cstate: current_term: 2 leader_uuid: "2fbc4f1ad97a4f2492306d77a614527a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } health_report { overall_health: HEALTHY } } }
I20260709 12:27:43.335057 31198 raft_consensus.cc:2889] T 3ed242e2c2b843978a7af7cca5694ae2 P 75f56c1a7f86421da9247e9b35990e34 [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:27:43.658263 31198 raft_consensus.cc:1275] T 8e560587c5c24e4587295d2d1b6d2be6 P 75f56c1a7f86421da9247e9b35990e34 [term 2 FOLLOWER]: Refusing update from remote peer 2fbc4f1ad97a4f2492306d77a614527a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:27:43.660161 31242 consensus_queue.cc:1048] T 8e560587c5c24e4587295d2d1b6d2be6 P 2fbc4f1ad97a4f2492306d77a614527a [LEADER]: Connected to new peer: Peer: permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:43.673734 30937 raft_consensus.cc:1275] T 8e560587c5c24e4587295d2d1b6d2be6 P e77b2d60f57d4dceb865e440eab35126 [term 2 FOLLOWER]: Refusing update from remote peer 2fbc4f1ad97a4f2492306d77a614527a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:27:43.675371 31253 consensus_queue.cc:1048] T 8e560587c5c24e4587295d2d1b6d2be6 P 2fbc4f1ad97a4f2492306d77a614527a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:27:43.804630 31198 raft_consensus.cc:1275] T 009d38185f3e49afbbd9faf57f968b2c P 75f56c1a7f86421da9247e9b35990e34 [term 2 FOLLOWER]: Refusing update from remote peer 2fbc4f1ad97a4f2492306d77a614527a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:27:43.806006 31242 consensus_queue.cc:1048] T 009d38185f3e49afbbd9faf57f968b2c P 2fbc4f1ad97a4f2492306d77a614527a [LEADER]: Connected to new peer: Peer: permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:27:43.813853 30937 raft_consensus.cc:1275] T 009d38185f3e49afbbd9faf57f968b2c P e77b2d60f57d4dceb865e440eab35126 [term 2 FOLLOWER]: Refusing update from remote peer 2fbc4f1ad97a4f2492306d77a614527a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:27:43.815390 31242 consensus_queue.cc:1048] T 009d38185f3e49afbbd9faf57f968b2c P 2fbc4f1ad97a4f2492306d77a614527a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260709 12:27:43.863617 30809 auto_rebalancer.cc:288] Could not retrieve cluster info: Illegal state: not all tservers available for tablet placement
I20260709 12:27:44.062655 16501 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:27:44.081629 31011 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "0be78a9b3c5d47df8efea08d91f58111"
dest_uuid: "2fbc4f1ad97a4f2492306d77a614527a"
mode: GRACEFUL
new_leader_uuid: "e77b2d60f57d4dceb865e440eab35126"
 from {username='slave'} at 127.0.0.1:53526
I20260709 12:27:44.082213 31011 raft_consensus.cc:606] T 0be78a9b3c5d47df8efea08d91f58111 P 2fbc4f1ad97a4f2492306d77a614527a [term 1 LEADER]: Received request to transfer leadership to TS e77b2d60f57d4dceb865e440eab35126
I20260709 12:27:44.085366 30937 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "881773ceed164ddf8ebc390254b1eb01"
dest_uuid: "e77b2d60f57d4dceb865e440eab35126"
mode: GRACEFUL
new_leader_uuid: "2fbc4f1ad97a4f2492306d77a614527a"
 from {username='slave'} at 127.0.0.1:33634
I20260709 12:27:44.085906 30937 raft_consensus.cc:606] T 881773ceed164ddf8ebc390254b1eb01 P e77b2d60f57d4dceb865e440eab35126 [term 1 LEADER]: Received request to transfer leadership to TS 2fbc4f1ad97a4f2492306d77a614527a
I20260709 12:27:44.088359 31011 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "8e560587c5c24e4587295d2d1b6d2be6"
dest_uuid: "2fbc4f1ad97a4f2492306d77a614527a"
mode: GRACEFUL
new_leader_uuid: "e77b2d60f57d4dceb865e440eab35126"
 from {username='slave'} at 127.0.0.1:53526
I20260709 12:27:44.088845 31011 raft_consensus.cc:606] T 8e560587c5c24e4587295d2d1b6d2be6 P 2fbc4f1ad97a4f2492306d77a614527a [term 2 LEADER]: Received request to transfer leadership to TS e77b2d60f57d4dceb865e440eab35126
I20260709 12:27:44.090059 16501 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 3
I20260709 12:27:44.090531 16501 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:27:44.091099 16501 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:27:44.106745 31254 raft_consensus.cc:993] T 8e560587c5c24e4587295d2d1b6d2be6 P 2fbc4f1ad97a4f2492306d77a614527a: : Instructing follower e77b2d60f57d4dceb865e440eab35126 to start an election
I20260709 12:27:44.107424 31242 raft_consensus.cc:1081] T 8e560587c5c24e4587295d2d1b6d2be6 P 2fbc4f1ad97a4f2492306d77a614527a [term 2 LEADER]: Signalling peer e77b2d60f57d4dceb865e440eab35126 to start an election
I20260709 12:27:44.109040 30937 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "8e560587c5c24e4587295d2d1b6d2be6"
dest_uuid: "e77b2d60f57d4dceb865e440eab35126"
 from {username='slave'} at 127.0.0.1:42606
I20260709 12:27:44.109583 30937 raft_consensus.cc:493] T 8e560587c5c24e4587295d2d1b6d2be6 P e77b2d60f57d4dceb865e440eab35126 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:27:44.109939 30937 raft_consensus.cc:3060] T 8e560587c5c24e4587295d2d1b6d2be6 P e77b2d60f57d4dceb865e440eab35126 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:27:44.115685 30937 raft_consensus.cc:515] T 8e560587c5c24e4587295d2d1b6d2be6 P e77b2d60f57d4dceb865e440eab35126 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } }
I20260709 12:27:44.117286 30937 leader_election.cc:290] T 8e560587c5c24e4587295d2d1b6d2be6 P e77b2d60f57d4dceb865e440eab35126 [CANDIDATE]: Term 3 election: Requested vote from peers 75f56c1a7f86421da9247e9b35990e34 (127.16.29.65:45249), 2fbc4f1ad97a4f2492306d77a614527a (127.16.29.67:44761)
I20260709 12:27:44.118283 31198 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e560587c5c24e4587295d2d1b6d2be6" candidate_uuid: "e77b2d60f57d4dceb865e440eab35126" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "75f56c1a7f86421da9247e9b35990e34"
I20260709 12:27:44.118559 31011 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e560587c5c24e4587295d2d1b6d2be6" candidate_uuid: "e77b2d60f57d4dceb865e440eab35126" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "2fbc4f1ad97a4f2492306d77a614527a"
I20260709 12:27:44.118855 31198 raft_consensus.cc:3060] T 8e560587c5c24e4587295d2d1b6d2be6 P 75f56c1a7f86421da9247e9b35990e34 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:27:44.119305 31011 raft_consensus.cc:3055] T 8e560587c5c24e4587295d2d1b6d2be6 P 2fbc4f1ad97a4f2492306d77a614527a [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:27:44.119692 31011 raft_consensus.cc:740] T 8e560587c5c24e4587295d2d1b6d2be6 P 2fbc4f1ad97a4f2492306d77a614527a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 2fbc4f1ad97a4f2492306d77a614527a, State: Running, Role: LEADER
I20260709 12:27:44.120599 31011 consensus_queue.cc:260] T 8e560587c5c24e4587295d2d1b6d2be6 P 2fbc4f1ad97a4f2492306d77a614527a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } }
I20260709 12:27:44.122007 31011 raft_consensus.cc:3060] T 8e560587c5c24e4587295d2d1b6d2be6 P 2fbc4f1ad97a4f2492306d77a614527a [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:27:44.125303 31198 raft_consensus.cc:2468] T 8e560587c5c24e4587295d2d1b6d2be6 P 75f56c1a7f86421da9247e9b35990e34 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e77b2d60f57d4dceb865e440eab35126 in term 3.
I20260709 12:27:44.126415 30903 leader_election.cc:304] T 8e560587c5c24e4587295d2d1b6d2be6 P e77b2d60f57d4dceb865e440eab35126 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 75f56c1a7f86421da9247e9b35990e34, e77b2d60f57d4dceb865e440eab35126; no voters: 
I20260709 12:27:44.127625 31252 raft_consensus.cc:2804] T 8e560587c5c24e4587295d2d1b6d2be6 P e77b2d60f57d4dceb865e440eab35126 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:27:44.128057 31252 raft_consensus.cc:697] T 8e560587c5c24e4587295d2d1b6d2be6 P e77b2d60f57d4dceb865e440eab35126 [term 3 LEADER]: Becoming Leader. State: Replica: e77b2d60f57d4dceb865e440eab35126, State: Running, Role: LEADER
I20260709 12:27:44.128571 31011 raft_consensus.cc:2468] T 8e560587c5c24e4587295d2d1b6d2be6 P 2fbc4f1ad97a4f2492306d77a614527a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e77b2d60f57d4dceb865e440eab35126 in term 3.
I20260709 12:27:44.128800 31252 consensus_queue.cc:237] T 8e560587c5c24e4587295d2d1b6d2be6 P e77b2d60f57d4dceb865e440eab35126 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } }
I20260709 12:27:44.135017 30759 catalog_manager.cc:5697] T 8e560587c5c24e4587295d2d1b6d2be6 P e77b2d60f57d4dceb865e440eab35126 reported cstate change: term changed from 2 to 3, leader changed from 2fbc4f1ad97a4f2492306d77a614527a (127.16.29.67) to e77b2d60f57d4dceb865e440eab35126 (127.16.29.66). New cstate: current_term: 3 leader_uuid: "e77b2d60f57d4dceb865e440eab35126" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:44.346383 31253 raft_consensus.cc:993] T 0be78a9b3c5d47df8efea08d91f58111 P 2fbc4f1ad97a4f2492306d77a614527a: : Instructing follower e77b2d60f57d4dceb865e440eab35126 to start an election
I20260709 12:27:44.346909 31242 raft_consensus.cc:1081] T 0be78a9b3c5d47df8efea08d91f58111 P 2fbc4f1ad97a4f2492306d77a614527a [term 1 LEADER]: Signalling peer e77b2d60f57d4dceb865e440eab35126 to start an election
I20260709 12:27:44.348384 30937 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "0be78a9b3c5d47df8efea08d91f58111"
dest_uuid: "e77b2d60f57d4dceb865e440eab35126"
 from {username='slave'} at 127.0.0.1:42606
I20260709 12:27:44.348834 30937 raft_consensus.cc:493] T 0be78a9b3c5d47df8efea08d91f58111 P e77b2d60f57d4dceb865e440eab35126 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:27:44.349066 30937 raft_consensus.cc:3060] T 0be78a9b3c5d47df8efea08d91f58111 P e77b2d60f57d4dceb865e440eab35126 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:44.353965 30937 raft_consensus.cc:515] T 0be78a9b3c5d47df8efea08d91f58111 P e77b2d60f57d4dceb865e440eab35126 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } }
I20260709 12:27:44.355369 30937 leader_election.cc:290] T 0be78a9b3c5d47df8efea08d91f58111 P e77b2d60f57d4dceb865e440eab35126 [CANDIDATE]: Term 2 election: Requested vote from peers 75f56c1a7f86421da9247e9b35990e34 (127.16.29.65:45249), 2fbc4f1ad97a4f2492306d77a614527a (127.16.29.67:44761)
I20260709 12:27:44.356282 31198 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0be78a9b3c5d47df8efea08d91f58111" candidate_uuid: "e77b2d60f57d4dceb865e440eab35126" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "75f56c1a7f86421da9247e9b35990e34"
I20260709 12:27:44.356508 31011 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0be78a9b3c5d47df8efea08d91f58111" candidate_uuid: "e77b2d60f57d4dceb865e440eab35126" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2fbc4f1ad97a4f2492306d77a614527a"
I20260709 12:27:44.356874 31198 raft_consensus.cc:3060] T 0be78a9b3c5d47df8efea08d91f58111 P 75f56c1a7f86421da9247e9b35990e34 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:44.357259 31011 raft_consensus.cc:3055] T 0be78a9b3c5d47df8efea08d91f58111 P 2fbc4f1ad97a4f2492306d77a614527a [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:27:44.357579 31011 raft_consensus.cc:740] T 0be78a9b3c5d47df8efea08d91f58111 P 2fbc4f1ad97a4f2492306d77a614527a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2fbc4f1ad97a4f2492306d77a614527a, State: Running, Role: LEADER
I20260709 12:27:44.358284 31011 consensus_queue.cc:260] T 0be78a9b3c5d47df8efea08d91f58111 P 2fbc4f1ad97a4f2492306d77a614527a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } }
I20260709 12:27:44.359205 31011 raft_consensus.cc:3060] T 0be78a9b3c5d47df8efea08d91f58111 P 2fbc4f1ad97a4f2492306d77a614527a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:44.362123 31198 raft_consensus.cc:2468] T 0be78a9b3c5d47df8efea08d91f58111 P 75f56c1a7f86421da9247e9b35990e34 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e77b2d60f57d4dceb865e440eab35126 in term 2.
I20260709 12:27:44.363288 30903 leader_election.cc:304] T 0be78a9b3c5d47df8efea08d91f58111 P e77b2d60f57d4dceb865e440eab35126 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 75f56c1a7f86421da9247e9b35990e34, e77b2d60f57d4dceb865e440eab35126; no voters: 
I20260709 12:27:44.363960 31011 raft_consensus.cc:2468] T 0be78a9b3c5d47df8efea08d91f58111 P 2fbc4f1ad97a4f2492306d77a614527a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e77b2d60f57d4dceb865e440eab35126 in term 2.
I20260709 12:27:44.363986 31252 raft_consensus.cc:2804] T 0be78a9b3c5d47df8efea08d91f58111 P e77b2d60f57d4dceb865e440eab35126 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:27:44.364691 31252 raft_consensus.cc:697] T 0be78a9b3c5d47df8efea08d91f58111 P e77b2d60f57d4dceb865e440eab35126 [term 2 LEADER]: Becoming Leader. State: Replica: e77b2d60f57d4dceb865e440eab35126, State: Running, Role: LEADER
I20260709 12:27:44.365550 31252 consensus_queue.cc:237] T 0be78a9b3c5d47df8efea08d91f58111 P e77b2d60f57d4dceb865e440eab35126 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } }
I20260709 12:27:44.372759 30759 catalog_manager.cc:5697] T 0be78a9b3c5d47df8efea08d91f58111 P e77b2d60f57d4dceb865e440eab35126 reported cstate change: term changed from 1 to 2, leader changed from 2fbc4f1ad97a4f2492306d77a614527a (127.16.29.67) to e77b2d60f57d4dceb865e440eab35126 (127.16.29.66). New cstate: current_term: 2 leader_uuid: "e77b2d60f57d4dceb865e440eab35126" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:44.602272 31198 raft_consensus.cc:1275] T 8e560587c5c24e4587295d2d1b6d2be6 P 75f56c1a7f86421da9247e9b35990e34 [term 3 FOLLOWER]: Refusing update from remote peer e77b2d60f57d4dceb865e440eab35126: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:27:44.603690 31252 consensus_queue.cc:1048] T 8e560587c5c24e4587295d2d1b6d2be6 P e77b2d60f57d4dceb865e440eab35126 [LEADER]: Connected to new peer: Peer: permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:27:44.618587 31011 raft_consensus.cc:1275] T 8e560587c5c24e4587295d2d1b6d2be6 P 2fbc4f1ad97a4f2492306d77a614527a [term 3 FOLLOWER]: Refusing update from remote peer e77b2d60f57d4dceb865e440eab35126: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:27:44.620221 31167 consensus_queue.cc:1048] T 8e560587c5c24e4587295d2d1b6d2be6 P e77b2d60f57d4dceb865e440eab35126 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:27:44.659853 31268 raft_consensus.cc:993] T 881773ceed164ddf8ebc390254b1eb01 P e77b2d60f57d4dceb865e440eab35126: : Instructing follower 2fbc4f1ad97a4f2492306d77a614527a to start an election
I20260709 12:27:44.660663 31167 raft_consensus.cc:1081] T 881773ceed164ddf8ebc390254b1eb01 P e77b2d60f57d4dceb865e440eab35126 [term 1 LEADER]: Signalling peer 2fbc4f1ad97a4f2492306d77a614527a to start an election
I20260709 12:27:44.662252 31011 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "881773ceed164ddf8ebc390254b1eb01"
dest_uuid: "2fbc4f1ad97a4f2492306d77a614527a"
 from {username='slave'} at 127.0.0.1:39866
I20260709 12:27:44.662671 31011 raft_consensus.cc:493] T 881773ceed164ddf8ebc390254b1eb01 P 2fbc4f1ad97a4f2492306d77a614527a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:27:44.662899 31011 raft_consensus.cc:3060] T 881773ceed164ddf8ebc390254b1eb01 P 2fbc4f1ad97a4f2492306d77a614527a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:44.667730 31011 raft_consensus.cc:515] T 881773ceed164ddf8ebc390254b1eb01 P 2fbc4f1ad97a4f2492306d77a614527a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } }
I20260709 12:27:44.669153 31011 leader_election.cc:290] T 881773ceed164ddf8ebc390254b1eb01 P 2fbc4f1ad97a4f2492306d77a614527a [CANDIDATE]: Term 2 election: Requested vote from peers 75f56c1a7f86421da9247e9b35990e34 (127.16.29.65:45249), e77b2d60f57d4dceb865e440eab35126 (127.16.29.66:46695)
I20260709 12:27:44.670332 30937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "881773ceed164ddf8ebc390254b1eb01" candidate_uuid: "2fbc4f1ad97a4f2492306d77a614527a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e77b2d60f57d4dceb865e440eab35126"
I20260709 12:27:44.670351 31198 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "881773ceed164ddf8ebc390254b1eb01" candidate_uuid: "2fbc4f1ad97a4f2492306d77a614527a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "75f56c1a7f86421da9247e9b35990e34"
I20260709 12:27:44.670946 30937 raft_consensus.cc:3055] T 881773ceed164ddf8ebc390254b1eb01 P e77b2d60f57d4dceb865e440eab35126 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:27:44.671078 31198 raft_consensus.cc:3060] T 881773ceed164ddf8ebc390254b1eb01 P 75f56c1a7f86421da9247e9b35990e34 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:44.671241 30937 raft_consensus.cc:740] T 881773ceed164ddf8ebc390254b1eb01 P e77b2d60f57d4dceb865e440eab35126 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e77b2d60f57d4dceb865e440eab35126, State: Running, Role: LEADER
I20260709 12:27:44.672093 30937 consensus_queue.cc:260] T 881773ceed164ddf8ebc390254b1eb01 P e77b2d60f57d4dceb865e440eab35126 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } }
I20260709 12:27:44.673002 30937 raft_consensus.cc:3060] T 881773ceed164ddf8ebc390254b1eb01 P e77b2d60f57d4dceb865e440eab35126 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:44.676592 31198 raft_consensus.cc:2468] T 881773ceed164ddf8ebc390254b1eb01 P 75f56c1a7f86421da9247e9b35990e34 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2fbc4f1ad97a4f2492306d77a614527a in term 2.
I20260709 12:27:44.677641 30977 leader_election.cc:304] T 881773ceed164ddf8ebc390254b1eb01 P 2fbc4f1ad97a4f2492306d77a614527a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2fbc4f1ad97a4f2492306d77a614527a, 75f56c1a7f86421da9247e9b35990e34; no voters: 
I20260709 12:27:44.678045 30937 raft_consensus.cc:2468] T 881773ceed164ddf8ebc390254b1eb01 P e77b2d60f57d4dceb865e440eab35126 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2fbc4f1ad97a4f2492306d77a614527a in term 2.
I20260709 12:27:44.678421 31242 raft_consensus.cc:2804] T 881773ceed164ddf8ebc390254b1eb01 P 2fbc4f1ad97a4f2492306d77a614527a [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:27:44.678969 31242 raft_consensus.cc:697] T 881773ceed164ddf8ebc390254b1eb01 P 2fbc4f1ad97a4f2492306d77a614527a [term 2 LEADER]: Becoming Leader. State: Replica: 2fbc4f1ad97a4f2492306d77a614527a, State: Running, Role: LEADER
I20260709 12:27:44.679971 31242 consensus_queue.cc:237] T 881773ceed164ddf8ebc390254b1eb01 P 2fbc4f1ad97a4f2492306d77a614527a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } }
I20260709 12:27:44.687613 30760 catalog_manager.cc:5697] T 881773ceed164ddf8ebc390254b1eb01 P 2fbc4f1ad97a4f2492306d77a614527a reported cstate change: term changed from 1 to 2, leader changed from e77b2d60f57d4dceb865e440eab35126 (127.16.29.66) to 2fbc4f1ad97a4f2492306d77a614527a (127.16.29.67). New cstate: current_term: 2 leader_uuid: "2fbc4f1ad97a4f2492306d77a614527a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } health_report { overall_health: HEALTHY } } }
I20260709 12:27:44.768432 31011 raft_consensus.cc:1275] T 0be78a9b3c5d47df8efea08d91f58111 P 2fbc4f1ad97a4f2492306d77a614527a [term 2 FOLLOWER]: Refusing update from remote peer e77b2d60f57d4dceb865e440eab35126: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:27:44.769688 31167 consensus_queue.cc:1048] T 0be78a9b3c5d47df8efea08d91f58111 P e77b2d60f57d4dceb865e440eab35126 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:27:44.779505 31198 raft_consensus.cc:1275] T 0be78a9b3c5d47df8efea08d91f58111 P 75f56c1a7f86421da9247e9b35990e34 [term 2 FOLLOWER]: Refusing update from remote peer e77b2d60f57d4dceb865e440eab35126: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:27:44.781137 31167 consensus_queue.cc:1048] T 0be78a9b3c5d47df8efea08d91f58111 P e77b2d60f57d4dceb865e440eab35126 [LEADER]: Connected to new peer: Peer: permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260709 12:27:44.864676 30809 auto_rebalancer.cc:288] Could not retrieve cluster info: Illegal state: not all tservers available for tablet placement
I20260709 12:27:45.099280 16501 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:27:45.107832 30937 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "0be78a9b3c5d47df8efea08d91f58111"
dest_uuid: "e77b2d60f57d4dceb865e440eab35126"
mode: GRACEFUL
new_leader_uuid: "2fbc4f1ad97a4f2492306d77a614527a"
 from {username='slave'} at 127.0.0.1:33634
I20260709 12:27:45.108400 30937 raft_consensus.cc:606] T 0be78a9b3c5d47df8efea08d91f58111 P e77b2d60f57d4dceb865e440eab35126 [term 2 LEADER]: Received request to transfer leadership to TS 2fbc4f1ad97a4f2492306d77a614527a
I20260709 12:27:45.110996 30937 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "8e560587c5c24e4587295d2d1b6d2be6"
dest_uuid: "e77b2d60f57d4dceb865e440eab35126"
mode: GRACEFUL
new_leader_uuid: "2fbc4f1ad97a4f2492306d77a614527a"
 from {username='slave'} at 127.0.0.1:33634
I20260709 12:27:45.111433 30937 raft_consensus.cc:606] T 8e560587c5c24e4587295d2d1b6d2be6 P e77b2d60f57d4dceb865e440eab35126 [term 3 LEADER]: Received request to transfer leadership to TS 2fbc4f1ad97a4f2492306d77a614527a
I20260709 12:27:45.114295 31011 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "b555d1cdbcec4c278fd300d628a8b9ab"
dest_uuid: "2fbc4f1ad97a4f2492306d77a614527a"
mode: GRACEFUL
new_leader_uuid: "e77b2d60f57d4dceb865e440eab35126"
 from {username='slave'} at 127.0.0.1:53526
I20260709 12:27:45.114863 31011 raft_consensus.cc:606] T b555d1cdbcec4c278fd300d628a8b9ab P 2fbc4f1ad97a4f2492306d77a614527a [term 1 LEADER]: Received request to transfer leadership to TS e77b2d60f57d4dceb865e440eab35126
I20260709 12:27:45.116156 16501 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 3
I20260709 12:27:45.116667 16501 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:27:45.117228 16501 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:27:45.168507 30937 raft_consensus.cc:1275] T 881773ceed164ddf8ebc390254b1eb01 P e77b2d60f57d4dceb865e440eab35126 [term 2 FOLLOWER]: Refusing update from remote peer 2fbc4f1ad97a4f2492306d77a614527a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:27:45.170253 31242 consensus_queue.cc:1048] T 881773ceed164ddf8ebc390254b1eb01 P 2fbc4f1ad97a4f2492306d77a614527a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:27:45.179479 31198 raft_consensus.cc:1275] T 881773ceed164ddf8ebc390254b1eb01 P 75f56c1a7f86421da9247e9b35990e34 [term 2 FOLLOWER]: Refusing update from remote peer 2fbc4f1ad97a4f2492306d77a614527a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:27:45.181155 31253 consensus_queue.cc:1048] T 881773ceed164ddf8ebc390254b1eb01 P 2fbc4f1ad97a4f2492306d77a614527a [LEADER]: Connected to new peer: Peer: permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:27:45.199273 31252 raft_consensus.cc:993] T 8e560587c5c24e4587295d2d1b6d2be6 P e77b2d60f57d4dceb865e440eab35126: : Instructing follower 2fbc4f1ad97a4f2492306d77a614527a to start an election
I20260709 12:27:45.199826 31167 raft_consensus.cc:1081] T 8e560587c5c24e4587295d2d1b6d2be6 P e77b2d60f57d4dceb865e440eab35126 [term 3 LEADER]: Signalling peer 2fbc4f1ad97a4f2492306d77a614527a to start an election
I20260709 12:27:45.201469 31011 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "8e560587c5c24e4587295d2d1b6d2be6"
dest_uuid: "2fbc4f1ad97a4f2492306d77a614527a"
 from {username='slave'} at 127.0.0.1:39866
I20260709 12:27:45.202018 31011 raft_consensus.cc:493] T 8e560587c5c24e4587295d2d1b6d2be6 P 2fbc4f1ad97a4f2492306d77a614527a [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:27:45.202277 31011 raft_consensus.cc:3060] T 8e560587c5c24e4587295d2d1b6d2be6 P 2fbc4f1ad97a4f2492306d77a614527a [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:27:45.207166 31011 raft_consensus.cc:515] T 8e560587c5c24e4587295d2d1b6d2be6 P 2fbc4f1ad97a4f2492306d77a614527a [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } }
I20260709 12:27:45.208760 31011 leader_election.cc:290] T 8e560587c5c24e4587295d2d1b6d2be6 P 2fbc4f1ad97a4f2492306d77a614527a [CANDIDATE]: Term 4 election: Requested vote from peers 75f56c1a7f86421da9247e9b35990e34 (127.16.29.65:45249), e77b2d60f57d4dceb865e440eab35126 (127.16.29.66:46695)
I20260709 12:27:45.209739 31198 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e560587c5c24e4587295d2d1b6d2be6" candidate_uuid: "2fbc4f1ad97a4f2492306d77a614527a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "75f56c1a7f86421da9247e9b35990e34"
I20260709 12:27:45.209784 30937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e560587c5c24e4587295d2d1b6d2be6" candidate_uuid: "2fbc4f1ad97a4f2492306d77a614527a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "e77b2d60f57d4dceb865e440eab35126"
I20260709 12:27:45.210582 31198 raft_consensus.cc:3060] T 8e560587c5c24e4587295d2d1b6d2be6 P 75f56c1a7f86421da9247e9b35990e34 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:27:45.210670 30937 raft_consensus.cc:3055] T 8e560587c5c24e4587295d2d1b6d2be6 P e77b2d60f57d4dceb865e440eab35126 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:27:45.211134 30937 raft_consensus.cc:740] T 8e560587c5c24e4587295d2d1b6d2be6 P e77b2d60f57d4dceb865e440eab35126 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: e77b2d60f57d4dceb865e440eab35126, State: Running, Role: LEADER
I20260709 12:27:45.211812 30937 consensus_queue.cc:260] T 8e560587c5c24e4587295d2d1b6d2be6 P e77b2d60f57d4dceb865e440eab35126 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } }
I20260709 12:27:45.212833 30937 raft_consensus.cc:3060] T 8e560587c5c24e4587295d2d1b6d2be6 P e77b2d60f57d4dceb865e440eab35126 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:27:45.216161 31198 raft_consensus.cc:2468] T 8e560587c5c24e4587295d2d1b6d2be6 P 75f56c1a7f86421da9247e9b35990e34 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2fbc4f1ad97a4f2492306d77a614527a in term 4.
I20260709 12:27:45.217350 30977 leader_election.cc:304] T 8e560587c5c24e4587295d2d1b6d2be6 P 2fbc4f1ad97a4f2492306d77a614527a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2fbc4f1ad97a4f2492306d77a614527a, 75f56c1a7f86421da9247e9b35990e34; no voters: 
I20260709 12:27:45.218205 30937 raft_consensus.cc:2468] T 8e560587c5c24e4587295d2d1b6d2be6 P e77b2d60f57d4dceb865e440eab35126 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2fbc4f1ad97a4f2492306d77a614527a in term 4.
I20260709 12:27:45.218214 31253 raft_consensus.cc:2804] T 8e560587c5c24e4587295d2d1b6d2be6 P 2fbc4f1ad97a4f2492306d77a614527a [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:27:45.219107 31253 raft_consensus.cc:697] T 8e560587c5c24e4587295d2d1b6d2be6 P 2fbc4f1ad97a4f2492306d77a614527a [term 4 LEADER]: Becoming Leader. State: Replica: 2fbc4f1ad97a4f2492306d77a614527a, State: Running, Role: LEADER
I20260709 12:27:45.219851 31253 consensus_queue.cc:237] T 8e560587c5c24e4587295d2d1b6d2be6 P 2fbc4f1ad97a4f2492306d77a614527a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } }
I20260709 12:27:45.226398 30760 catalog_manager.cc:5697] T 8e560587c5c24e4587295d2d1b6d2be6 P 2fbc4f1ad97a4f2492306d77a614527a reported cstate change: term changed from 3 to 4, leader changed from e77b2d60f57d4dceb865e440eab35126 (127.16.29.66) to 2fbc4f1ad97a4f2492306d77a614527a (127.16.29.67). New cstate: current_term: 4 leader_uuid: "2fbc4f1ad97a4f2492306d77a614527a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } health_report { overall_health: HEALTHY } } }
I20260709 12:27:45.260697 31252 raft_consensus.cc:993] T 0be78a9b3c5d47df8efea08d91f58111 P e77b2d60f57d4dceb865e440eab35126: : Instructing follower 2fbc4f1ad97a4f2492306d77a614527a to start an election
I20260709 12:27:45.261166 31167 raft_consensus.cc:1081] T 0be78a9b3c5d47df8efea08d91f58111 P e77b2d60f57d4dceb865e440eab35126 [term 2 LEADER]: Signalling peer 2fbc4f1ad97a4f2492306d77a614527a to start an election
I20260709 12:27:45.262542 31011 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "0be78a9b3c5d47df8efea08d91f58111"
dest_uuid: "2fbc4f1ad97a4f2492306d77a614527a"
 from {username='slave'} at 127.0.0.1:39866
I20260709 12:27:45.262966 31011 raft_consensus.cc:493] T 0be78a9b3c5d47df8efea08d91f58111 P 2fbc4f1ad97a4f2492306d77a614527a [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:27:45.263200 31011 raft_consensus.cc:3060] T 0be78a9b3c5d47df8efea08d91f58111 P 2fbc4f1ad97a4f2492306d77a614527a [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:27:45.267786 31011 raft_consensus.cc:515] T 0be78a9b3c5d47df8efea08d91f58111 P 2fbc4f1ad97a4f2492306d77a614527a [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } }
I20260709 12:27:45.269825 31011 leader_election.cc:290] T 0be78a9b3c5d47df8efea08d91f58111 P 2fbc4f1ad97a4f2492306d77a614527a [CANDIDATE]: Term 3 election: Requested vote from peers 75f56c1a7f86421da9247e9b35990e34 (127.16.29.65:45249), e77b2d60f57d4dceb865e440eab35126 (127.16.29.66:46695)
I20260709 12:27:45.270788 31198 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0be78a9b3c5d47df8efea08d91f58111" candidate_uuid: "2fbc4f1ad97a4f2492306d77a614527a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "75f56c1a7f86421da9247e9b35990e34"
I20260709 12:27:45.271452 31198 raft_consensus.cc:3060] T 0be78a9b3c5d47df8efea08d91f58111 P 75f56c1a7f86421da9247e9b35990e34 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:27:45.271961 30937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0be78a9b3c5d47df8efea08d91f58111" candidate_uuid: "2fbc4f1ad97a4f2492306d77a614527a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "e77b2d60f57d4dceb865e440eab35126"
I20260709 12:27:45.272632 30937 raft_consensus.cc:3055] T 0be78a9b3c5d47df8efea08d91f58111 P e77b2d60f57d4dceb865e440eab35126 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:27:45.273036 30937 raft_consensus.cc:740] T 0be78a9b3c5d47df8efea08d91f58111 P e77b2d60f57d4dceb865e440eab35126 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: e77b2d60f57d4dceb865e440eab35126, State: Running, Role: LEADER
I20260709 12:27:45.274147 30937 consensus_queue.cc:260] T 0be78a9b3c5d47df8efea08d91f58111 P e77b2d60f57d4dceb865e440eab35126 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } }
I20260709 12:27:45.275624 30937 raft_consensus.cc:3060] T 0be78a9b3c5d47df8efea08d91f58111 P e77b2d60f57d4dceb865e440eab35126 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:27:45.278589 31198 raft_consensus.cc:2468] T 0be78a9b3c5d47df8efea08d91f58111 P 75f56c1a7f86421da9247e9b35990e34 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2fbc4f1ad97a4f2492306d77a614527a in term 3.
I20260709 12:27:45.279767 30977 leader_election.cc:304] T 0be78a9b3c5d47df8efea08d91f58111 P 2fbc4f1ad97a4f2492306d77a614527a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2fbc4f1ad97a4f2492306d77a614527a, 75f56c1a7f86421da9247e9b35990e34; no voters: 
I20260709 12:27:45.280584 31253 raft_consensus.cc:2804] T 0be78a9b3c5d47df8efea08d91f58111 P 2fbc4f1ad97a4f2492306d77a614527a [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:27:45.281038 31253 raft_consensus.cc:697] T 0be78a9b3c5d47df8efea08d91f58111 P 2fbc4f1ad97a4f2492306d77a614527a [term 3 LEADER]: Becoming Leader. State: Replica: 2fbc4f1ad97a4f2492306d77a614527a, State: Running, Role: LEADER
I20260709 12:27:45.281790 31253 consensus_queue.cc:237] T 0be78a9b3c5d47df8efea08d91f58111 P 2fbc4f1ad97a4f2492306d77a614527a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } }
I20260709 12:27:45.282720 30937 raft_consensus.cc:2468] T 0be78a9b3c5d47df8efea08d91f58111 P e77b2d60f57d4dceb865e440eab35126 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2fbc4f1ad97a4f2492306d77a614527a in term 3.
I20260709 12:27:45.289144 30760 catalog_manager.cc:5697] T 0be78a9b3c5d47df8efea08d91f58111 P 2fbc4f1ad97a4f2492306d77a614527a reported cstate change: term changed from 2 to 3, leader changed from e77b2d60f57d4dceb865e440eab35126 (127.16.29.66) to 2fbc4f1ad97a4f2492306d77a614527a (127.16.29.67). New cstate: current_term: 3 leader_uuid: "2fbc4f1ad97a4f2492306d77a614527a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } health_report { overall_health: HEALTHY } } }
I20260709 12:27:45.336727 31242 raft_consensus.cc:993] T b555d1cdbcec4c278fd300d628a8b9ab P 2fbc4f1ad97a4f2492306d77a614527a: : Instructing follower e77b2d60f57d4dceb865e440eab35126 to start an election
I20260709 12:27:45.337352 31253 raft_consensus.cc:1081] T b555d1cdbcec4c278fd300d628a8b9ab P 2fbc4f1ad97a4f2492306d77a614527a [term 1 LEADER]: Signalling peer e77b2d60f57d4dceb865e440eab35126 to start an election
I20260709 12:27:45.339044 30937 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "b555d1cdbcec4c278fd300d628a8b9ab"
dest_uuid: "e77b2d60f57d4dceb865e440eab35126"
 from {username='slave'} at 127.0.0.1:42606
I20260709 12:27:45.339534 30937 raft_consensus.cc:493] T b555d1cdbcec4c278fd300d628a8b9ab P e77b2d60f57d4dceb865e440eab35126 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:27:45.339859 30937 raft_consensus.cc:3060] T b555d1cdbcec4c278fd300d628a8b9ab P e77b2d60f57d4dceb865e440eab35126 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:45.344918 30937 raft_consensus.cc:515] T b555d1cdbcec4c278fd300d628a8b9ab P e77b2d60f57d4dceb865e440eab35126 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } }
I20260709 12:27:45.346491 30937 leader_election.cc:290] T b555d1cdbcec4c278fd300d628a8b9ab P e77b2d60f57d4dceb865e440eab35126 [CANDIDATE]: Term 2 election: Requested vote from peers 75f56c1a7f86421da9247e9b35990e34 (127.16.29.65:45249), 2fbc4f1ad97a4f2492306d77a614527a (127.16.29.67:44761)
I20260709 12:27:45.347573 31198 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b555d1cdbcec4c278fd300d628a8b9ab" candidate_uuid: "e77b2d60f57d4dceb865e440eab35126" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "75f56c1a7f86421da9247e9b35990e34"
I20260709 12:27:45.347577 31011 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b555d1cdbcec4c278fd300d628a8b9ab" candidate_uuid: "e77b2d60f57d4dceb865e440eab35126" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2fbc4f1ad97a4f2492306d77a614527a"
I20260709 12:27:45.348292 31198 raft_consensus.cc:3060] T b555d1cdbcec4c278fd300d628a8b9ab P 75f56c1a7f86421da9247e9b35990e34 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:45.348389 31011 raft_consensus.cc:3055] T b555d1cdbcec4c278fd300d628a8b9ab P 2fbc4f1ad97a4f2492306d77a614527a [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:27:45.348826 31011 raft_consensus.cc:740] T b555d1cdbcec4c278fd300d628a8b9ab P 2fbc4f1ad97a4f2492306d77a614527a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2fbc4f1ad97a4f2492306d77a614527a, State: Running, Role: LEADER
I20260709 12:27:45.349676 31011 consensus_queue.cc:260] T b555d1cdbcec4c278fd300d628a8b9ab P 2fbc4f1ad97a4f2492306d77a614527a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } }
I20260709 12:27:45.350811 31011 raft_consensus.cc:3060] T b555d1cdbcec4c278fd300d628a8b9ab P 2fbc4f1ad97a4f2492306d77a614527a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:45.354969 31198 raft_consensus.cc:2468] T b555d1cdbcec4c278fd300d628a8b9ab P 75f56c1a7f86421da9247e9b35990e34 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e77b2d60f57d4dceb865e440eab35126 in term 2.
I20260709 12:27:45.356163 31011 raft_consensus.cc:2468] T b555d1cdbcec4c278fd300d628a8b9ab P 2fbc4f1ad97a4f2492306d77a614527a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e77b2d60f57d4dceb865e440eab35126 in term 2.
I20260709 12:27:45.356200 30903 leader_election.cc:304] T b555d1cdbcec4c278fd300d628a8b9ab P e77b2d60f57d4dceb865e440eab35126 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 75f56c1a7f86421da9247e9b35990e34, e77b2d60f57d4dceb865e440eab35126; no voters: 
I20260709 12:27:45.357213 31167 raft_consensus.cc:2804] T b555d1cdbcec4c278fd300d628a8b9ab P e77b2d60f57d4dceb865e440eab35126 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:27:45.357703 31167 raft_consensus.cc:697] T b555d1cdbcec4c278fd300d628a8b9ab P e77b2d60f57d4dceb865e440eab35126 [term 2 LEADER]: Becoming Leader. State: Replica: e77b2d60f57d4dceb865e440eab35126, State: Running, Role: LEADER
I20260709 12:27:45.358600 31167 consensus_queue.cc:237] T b555d1cdbcec4c278fd300d628a8b9ab P e77b2d60f57d4dceb865e440eab35126 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } }
I20260709 12:27:45.365477 30760 catalog_manager.cc:5697] T b555d1cdbcec4c278fd300d628a8b9ab P e77b2d60f57d4dceb865e440eab35126 reported cstate change: term changed from 1 to 2, leader changed from 2fbc4f1ad97a4f2492306d77a614527a (127.16.29.67) to e77b2d60f57d4dceb865e440eab35126 (127.16.29.66). New cstate: current_term: 2 leader_uuid: "e77b2d60f57d4dceb865e440eab35126" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:45.604645 30937 raft_consensus.cc:1275] T 8e560587c5c24e4587295d2d1b6d2be6 P e77b2d60f57d4dceb865e440eab35126 [term 4 FOLLOWER]: Refusing update from remote peer 2fbc4f1ad97a4f2492306d77a614527a: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:27:45.605926 31253 consensus_queue.cc:1048] T 8e560587c5c24e4587295d2d1b6d2be6 P 2fbc4f1ad97a4f2492306d77a614527a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:27:45.613286 31199 raft_consensus.cc:1275] T 8e560587c5c24e4587295d2d1b6d2be6 P 75f56c1a7f86421da9247e9b35990e34 [term 4 FOLLOWER]: Refusing update from remote peer 2fbc4f1ad97a4f2492306d77a614527a: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:27:45.614923 31275 consensus_queue.cc:1048] T 8e560587c5c24e4587295d2d1b6d2be6 P 2fbc4f1ad97a4f2492306d77a614527a [LEADER]: Connected to new peer: Peer: permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:27:45.670324 31199 raft_consensus.cc:1275] T 0be78a9b3c5d47df8efea08d91f58111 P 75f56c1a7f86421da9247e9b35990e34 [term 3 FOLLOWER]: Refusing update from remote peer 2fbc4f1ad97a4f2492306d77a614527a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:27:45.671746 31275 consensus_queue.cc:1048] T 0be78a9b3c5d47df8efea08d91f58111 P 2fbc4f1ad97a4f2492306d77a614527a [LEADER]: Connected to new peer: Peer: permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:27:45.679284 30937 raft_consensus.cc:1275] T 0be78a9b3c5d47df8efea08d91f58111 P e77b2d60f57d4dceb865e440eab35126 [term 3 FOLLOWER]: Refusing update from remote peer 2fbc4f1ad97a4f2492306d77a614527a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:27:45.681733 31275 consensus_queue.cc:1048] T 0be78a9b3c5d47df8efea08d91f58111 P 2fbc4f1ad97a4f2492306d77a614527a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:27:45.811297 31011 raft_consensus.cc:1275] T b555d1cdbcec4c278fd300d628a8b9ab P 2fbc4f1ad97a4f2492306d77a614527a [term 2 FOLLOWER]: Refusing update from remote peer e77b2d60f57d4dceb865e440eab35126: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:27:45.812752 31167 consensus_queue.cc:1048] T b555d1cdbcec4c278fd300d628a8b9ab P e77b2d60f57d4dceb865e440eab35126 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:27:45.824735 31199 raft_consensus.cc:1275] T b555d1cdbcec4c278fd300d628a8b9ab P 75f56c1a7f86421da9247e9b35990e34 [term 2 FOLLOWER]: Refusing update from remote peer e77b2d60f57d4dceb865e440eab35126: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:27:45.826068 31167 consensus_queue.cc:1048] T b555d1cdbcec4c278fd300d628a8b9ab P e77b2d60f57d4dceb865e440eab35126 [LEADER]: Connected to new peer: Peer: permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260709 12:27:45.865885 30809 auto_rebalancer.cc:288] Could not retrieve cluster info: Illegal state: not all tservers available for tablet placement
I20260709 12:27:46.124413 16501 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:27:46.132849 31011 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "0be78a9b3c5d47df8efea08d91f58111"
dest_uuid: "2fbc4f1ad97a4f2492306d77a614527a"
mode: GRACEFUL
new_leader_uuid: "e77b2d60f57d4dceb865e440eab35126"
 from {username='slave'} at 127.0.0.1:53526
I20260709 12:27:46.133457 31011 raft_consensus.cc:606] T 0be78a9b3c5d47df8efea08d91f58111 P 2fbc4f1ad97a4f2492306d77a614527a [term 3 LEADER]: Received request to transfer leadership to TS e77b2d60f57d4dceb865e440eab35126
I20260709 12:27:46.135829 31011 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "881773ceed164ddf8ebc390254b1eb01"
dest_uuid: "2fbc4f1ad97a4f2492306d77a614527a"
mode: GRACEFUL
new_leader_uuid: "e77b2d60f57d4dceb865e440eab35126"
 from {username='slave'} at 127.0.0.1:53526
I20260709 12:27:46.136334 31011 raft_consensus.cc:606] T 881773ceed164ddf8ebc390254b1eb01 P 2fbc4f1ad97a4f2492306d77a614527a [term 2 LEADER]: Received request to transfer leadership to TS e77b2d60f57d4dceb865e440eab35126
I20260709 12:27:46.138553 30937 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "b555d1cdbcec4c278fd300d628a8b9ab"
dest_uuid: "e77b2d60f57d4dceb865e440eab35126"
mode: GRACEFUL
new_leader_uuid: "2fbc4f1ad97a4f2492306d77a614527a"
 from {username='slave'} at 127.0.0.1:33634
I20260709 12:27:46.139034 30937 raft_consensus.cc:606] T b555d1cdbcec4c278fd300d628a8b9ab P e77b2d60f57d4dceb865e440eab35126 [term 2 LEADER]: Received request to transfer leadership to TS 2fbc4f1ad97a4f2492306d77a614527a
I20260709 12:27:46.140224 16501 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 3
I20260709 12:27:46.140759 16501 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:27:46.141335 16501 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:27:46.291148 31253 raft_consensus.cc:993] T 881773ceed164ddf8ebc390254b1eb01 P 2fbc4f1ad97a4f2492306d77a614527a: : Instructing follower e77b2d60f57d4dceb865e440eab35126 to start an election
I20260709 12:27:46.291796 31275 raft_consensus.cc:1081] T 881773ceed164ddf8ebc390254b1eb01 P 2fbc4f1ad97a4f2492306d77a614527a [term 2 LEADER]: Signalling peer e77b2d60f57d4dceb865e440eab35126 to start an election
I20260709 12:27:46.293677 30937 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "881773ceed164ddf8ebc390254b1eb01"
dest_uuid: "e77b2d60f57d4dceb865e440eab35126"
 from {username='slave'} at 127.0.0.1:42606
I20260709 12:27:46.294241 30937 raft_consensus.cc:493] T 881773ceed164ddf8ebc390254b1eb01 P e77b2d60f57d4dceb865e440eab35126 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:27:46.294562 30937 raft_consensus.cc:3060] T 881773ceed164ddf8ebc390254b1eb01 P e77b2d60f57d4dceb865e440eab35126 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:27:46.300446 30937 raft_consensus.cc:515] T 881773ceed164ddf8ebc390254b1eb01 P e77b2d60f57d4dceb865e440eab35126 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } }
I20260709 12:27:46.302625 30937 leader_election.cc:290] T 881773ceed164ddf8ebc390254b1eb01 P e77b2d60f57d4dceb865e440eab35126 [CANDIDATE]: Term 3 election: Requested vote from peers 75f56c1a7f86421da9247e9b35990e34 (127.16.29.65:45249), 2fbc4f1ad97a4f2492306d77a614527a (127.16.29.67:44761)
I20260709 12:27:46.303516 31199 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "881773ceed164ddf8ebc390254b1eb01" candidate_uuid: "e77b2d60f57d4dceb865e440eab35126" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "75f56c1a7f86421da9247e9b35990e34"
I20260709 12:27:46.303795 31011 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "881773ceed164ddf8ebc390254b1eb01" candidate_uuid: "e77b2d60f57d4dceb865e440eab35126" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "2fbc4f1ad97a4f2492306d77a614527a"
I20260709 12:27:46.304327 31199 raft_consensus.cc:3060] T 881773ceed164ddf8ebc390254b1eb01 P 75f56c1a7f86421da9247e9b35990e34 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:27:46.304497 31011 raft_consensus.cc:3055] T 881773ceed164ddf8ebc390254b1eb01 P 2fbc4f1ad97a4f2492306d77a614527a [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:27:46.305051 31011 raft_consensus.cc:740] T 881773ceed164ddf8ebc390254b1eb01 P 2fbc4f1ad97a4f2492306d77a614527a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 2fbc4f1ad97a4f2492306d77a614527a, State: Running, Role: LEADER
I20260709 12:27:46.305884 31011 consensus_queue.cc:260] T 881773ceed164ddf8ebc390254b1eb01 P 2fbc4f1ad97a4f2492306d77a614527a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } }
I20260709 12:27:46.306959 31011 raft_consensus.cc:3060] T 881773ceed164ddf8ebc390254b1eb01 P 2fbc4f1ad97a4f2492306d77a614527a [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:27:46.311902 31199 raft_consensus.cc:2468] T 881773ceed164ddf8ebc390254b1eb01 P 75f56c1a7f86421da9247e9b35990e34 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e77b2d60f57d4dceb865e440eab35126 in term 3.
I20260709 12:27:46.313030 31011 raft_consensus.cc:2468] T 881773ceed164ddf8ebc390254b1eb01 P 2fbc4f1ad97a4f2492306d77a614527a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e77b2d60f57d4dceb865e440eab35126 in term 3.
I20260709 12:27:46.313244 30903 leader_election.cc:304] T 881773ceed164ddf8ebc390254b1eb01 P e77b2d60f57d4dceb865e440eab35126 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 75f56c1a7f86421da9247e9b35990e34, e77b2d60f57d4dceb865e440eab35126; no voters: 
I20260709 12:27:46.314195 31167 raft_consensus.cc:2804] T 881773ceed164ddf8ebc390254b1eb01 P e77b2d60f57d4dceb865e440eab35126 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:27:46.314680 31167 raft_consensus.cc:697] T 881773ceed164ddf8ebc390254b1eb01 P e77b2d60f57d4dceb865e440eab35126 [term 3 LEADER]: Becoming Leader. State: Replica: e77b2d60f57d4dceb865e440eab35126, State: Running, Role: LEADER
I20260709 12:27:46.315353 31167 consensus_queue.cc:237] T 881773ceed164ddf8ebc390254b1eb01 P e77b2d60f57d4dceb865e440eab35126 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } }
I20260709 12:27:46.322105 30759 catalog_manager.cc:5697] T 881773ceed164ddf8ebc390254b1eb01 P e77b2d60f57d4dceb865e440eab35126 reported cstate change: term changed from 2 to 3, leader changed from 2fbc4f1ad97a4f2492306d77a614527a (127.16.29.67) to e77b2d60f57d4dceb865e440eab35126 (127.16.29.66). New cstate: current_term: 3 leader_uuid: "e77b2d60f57d4dceb865e440eab35126" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:46.377379 31284 raft_consensus.cc:993] T b555d1cdbcec4c278fd300d628a8b9ab P e77b2d60f57d4dceb865e440eab35126: : Instructing follower 2fbc4f1ad97a4f2492306d77a614527a to start an election
I20260709 12:27:46.377902 31167 raft_consensus.cc:1081] T b555d1cdbcec4c278fd300d628a8b9ab P e77b2d60f57d4dceb865e440eab35126 [term 2 LEADER]: Signalling peer 2fbc4f1ad97a4f2492306d77a614527a to start an election
I20260709 12:27:46.379614 31011 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "b555d1cdbcec4c278fd300d628a8b9ab"
dest_uuid: "2fbc4f1ad97a4f2492306d77a614527a"
 from {username='slave'} at 127.0.0.1:39866
I20260709 12:27:46.380188 31011 raft_consensus.cc:493] T b555d1cdbcec4c278fd300d628a8b9ab P 2fbc4f1ad97a4f2492306d77a614527a [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:27:46.380457 31011 raft_consensus.cc:3060] T b555d1cdbcec4c278fd300d628a8b9ab P 2fbc4f1ad97a4f2492306d77a614527a [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:27:46.387111 31011 raft_consensus.cc:515] T b555d1cdbcec4c278fd300d628a8b9ab P 2fbc4f1ad97a4f2492306d77a614527a [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } }
I20260709 12:27:46.389659 31199 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b555d1cdbcec4c278fd300d628a8b9ab" candidate_uuid: "2fbc4f1ad97a4f2492306d77a614527a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "75f56c1a7f86421da9247e9b35990e34"
I20260709 12:27:46.390326 30937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b555d1cdbcec4c278fd300d628a8b9ab" candidate_uuid: "2fbc4f1ad97a4f2492306d77a614527a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "e77b2d60f57d4dceb865e440eab35126"
I20260709 12:27:46.388947 31011 leader_election.cc:290] T b555d1cdbcec4c278fd300d628a8b9ab P 2fbc4f1ad97a4f2492306d77a614527a [CANDIDATE]: Term 3 election: Requested vote from peers 75f56c1a7f86421da9247e9b35990e34 (127.16.29.65:45249), e77b2d60f57d4dceb865e440eab35126 (127.16.29.66:46695)
I20260709 12:27:46.390863 31199 raft_consensus.cc:3060] T b555d1cdbcec4c278fd300d628a8b9ab P 75f56c1a7f86421da9247e9b35990e34 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:27:46.390985 30937 raft_consensus.cc:3055] T b555d1cdbcec4c278fd300d628a8b9ab P e77b2d60f57d4dceb865e440eab35126 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:27:46.391669 30937 raft_consensus.cc:740] T b555d1cdbcec4c278fd300d628a8b9ab P e77b2d60f57d4dceb865e440eab35126 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: e77b2d60f57d4dceb865e440eab35126, State: Running, Role: LEADER
I20260709 12:27:46.392457 30937 consensus_queue.cc:260] T b555d1cdbcec4c278fd300d628a8b9ab P e77b2d60f57d4dceb865e440eab35126 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } }
I20260709 12:27:46.393622 30937 raft_consensus.cc:3060] T b555d1cdbcec4c278fd300d628a8b9ab P e77b2d60f57d4dceb865e440eab35126 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:27:46.398587 31199 raft_consensus.cc:2468] T b555d1cdbcec4c278fd300d628a8b9ab P 75f56c1a7f86421da9247e9b35990e34 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2fbc4f1ad97a4f2492306d77a614527a in term 3.
I20260709 12:27:46.400429 30977 leader_election.cc:304] T b555d1cdbcec4c278fd300d628a8b9ab P 2fbc4f1ad97a4f2492306d77a614527a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2fbc4f1ad97a4f2492306d77a614527a, 75f56c1a7f86421da9247e9b35990e34; no voters: 
I20260709 12:27:46.400743 30937 raft_consensus.cc:2468] T b555d1cdbcec4c278fd300d628a8b9ab P e77b2d60f57d4dceb865e440eab35126 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2fbc4f1ad97a4f2492306d77a614527a in term 3.
I20260709 12:27:46.401343 31275 raft_consensus.cc:2804] T b555d1cdbcec4c278fd300d628a8b9ab P 2fbc4f1ad97a4f2492306d77a614527a [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:27:46.401888 31275 raft_consensus.cc:697] T b555d1cdbcec4c278fd300d628a8b9ab P 2fbc4f1ad97a4f2492306d77a614527a [term 3 LEADER]: Becoming Leader. State: Replica: 2fbc4f1ad97a4f2492306d77a614527a, State: Running, Role: LEADER
I20260709 12:27:46.402706 31275 consensus_queue.cc:237] T b555d1cdbcec4c278fd300d628a8b9ab P 2fbc4f1ad97a4f2492306d77a614527a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } }
I20260709 12:27:46.409826 30759 catalog_manager.cc:5697] T b555d1cdbcec4c278fd300d628a8b9ab P 2fbc4f1ad97a4f2492306d77a614527a reported cstate change: term changed from 2 to 3, leader changed from e77b2d60f57d4dceb865e440eab35126 (127.16.29.66) to 2fbc4f1ad97a4f2492306d77a614527a (127.16.29.67). New cstate: current_term: 3 leader_uuid: "2fbc4f1ad97a4f2492306d77a614527a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } health_report { overall_health: HEALTHY } } }
I20260709 12:27:46.713054 31253 raft_consensus.cc:993] T 0be78a9b3c5d47df8efea08d91f58111 P 2fbc4f1ad97a4f2492306d77a614527a: : Instructing follower e77b2d60f57d4dceb865e440eab35126 to start an election
I20260709 12:27:46.713586 31275 raft_consensus.cc:1081] T 0be78a9b3c5d47df8efea08d91f58111 P 2fbc4f1ad97a4f2492306d77a614527a [term 3 LEADER]: Signalling peer e77b2d60f57d4dceb865e440eab35126 to start an election
I20260709 12:27:46.715199 30937 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "0be78a9b3c5d47df8efea08d91f58111"
dest_uuid: "e77b2d60f57d4dceb865e440eab35126"
 from {username='slave'} at 127.0.0.1:42606
I20260709 12:27:46.715752 30937 raft_consensus.cc:493] T 0be78a9b3c5d47df8efea08d91f58111 P e77b2d60f57d4dceb865e440eab35126 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:27:46.716149 30937 raft_consensus.cc:3060] T 0be78a9b3c5d47df8efea08d91f58111 P e77b2d60f57d4dceb865e440eab35126 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:27:46.721266 30937 raft_consensus.cc:515] T 0be78a9b3c5d47df8efea08d91f58111 P e77b2d60f57d4dceb865e440eab35126 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } }
I20260709 12:27:46.722886 30937 leader_election.cc:290] T 0be78a9b3c5d47df8efea08d91f58111 P e77b2d60f57d4dceb865e440eab35126 [CANDIDATE]: Term 4 election: Requested vote from peers 75f56c1a7f86421da9247e9b35990e34 (127.16.29.65:45249), 2fbc4f1ad97a4f2492306d77a614527a (127.16.29.67:44761)
I20260709 12:27:46.723774 31199 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0be78a9b3c5d47df8efea08d91f58111" candidate_uuid: "e77b2d60f57d4dceb865e440eab35126" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "75f56c1a7f86421da9247e9b35990e34"
I20260709 12:27:46.723951 31011 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0be78a9b3c5d47df8efea08d91f58111" candidate_uuid: "e77b2d60f57d4dceb865e440eab35126" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "2fbc4f1ad97a4f2492306d77a614527a"
I20260709 12:27:46.724464 31199 raft_consensus.cc:3060] T 0be78a9b3c5d47df8efea08d91f58111 P 75f56c1a7f86421da9247e9b35990e34 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:27:46.724570 31011 raft_consensus.cc:3055] T 0be78a9b3c5d47df8efea08d91f58111 P 2fbc4f1ad97a4f2492306d77a614527a [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:27:46.724970 31011 raft_consensus.cc:740] T 0be78a9b3c5d47df8efea08d91f58111 P 2fbc4f1ad97a4f2492306d77a614527a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 2fbc4f1ad97a4f2492306d77a614527a, State: Running, Role: LEADER
I20260709 12:27:46.725636 31011 consensus_queue.cc:260] T 0be78a9b3c5d47df8efea08d91f58111 P 2fbc4f1ad97a4f2492306d77a614527a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } }
I20260709 12:27:46.726487 31011 raft_consensus.cc:3060] T 0be78a9b3c5d47df8efea08d91f58111 P 2fbc4f1ad97a4f2492306d77a614527a [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:27:46.730746 31199 raft_consensus.cc:2468] T 0be78a9b3c5d47df8efea08d91f58111 P 75f56c1a7f86421da9247e9b35990e34 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e77b2d60f57d4dceb865e440eab35126 in term 4.
I20260709 12:27:46.730998 31011 raft_consensus.cc:2468] T 0be78a9b3c5d47df8efea08d91f58111 P 2fbc4f1ad97a4f2492306d77a614527a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e77b2d60f57d4dceb865e440eab35126 in term 4.
I20260709 12:27:46.732116 30903 leader_election.cc:304] T 0be78a9b3c5d47df8efea08d91f58111 P e77b2d60f57d4dceb865e440eab35126 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 75f56c1a7f86421da9247e9b35990e34, e77b2d60f57d4dceb865e440eab35126; no voters: 
I20260709 12:27:46.732985 31284 raft_consensus.cc:2804] T 0be78a9b3c5d47df8efea08d91f58111 P e77b2d60f57d4dceb865e440eab35126 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:27:46.733399 31284 raft_consensus.cc:697] T 0be78a9b3c5d47df8efea08d91f58111 P e77b2d60f57d4dceb865e440eab35126 [term 4 LEADER]: Becoming Leader. State: Replica: e77b2d60f57d4dceb865e440eab35126, State: Running, Role: LEADER
I20260709 12:27:46.734107 31284 consensus_queue.cc:237] T 0be78a9b3c5d47df8efea08d91f58111 P e77b2d60f57d4dceb865e440eab35126 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } }
I20260709 12:27:46.740958 30759 catalog_manager.cc:5697] T 0be78a9b3c5d47df8efea08d91f58111 P e77b2d60f57d4dceb865e440eab35126 reported cstate change: term changed from 3 to 4, leader changed from 2fbc4f1ad97a4f2492306d77a614527a (127.16.29.67) to e77b2d60f57d4dceb865e440eab35126 (127.16.29.66). New cstate: current_term: 4 leader_uuid: "e77b2d60f57d4dceb865e440eab35126" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:46.834826 31199 raft_consensus.cc:1275] T b555d1cdbcec4c278fd300d628a8b9ab P 75f56c1a7f86421da9247e9b35990e34 [term 3 FOLLOWER]: Refusing update from remote peer 2fbc4f1ad97a4f2492306d77a614527a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:27:46.836486 31275 consensus_queue.cc:1048] T b555d1cdbcec4c278fd300d628a8b9ab P 2fbc4f1ad97a4f2492306d77a614527a [LEADER]: Connected to new peer: Peer: permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:27:46.848425 31011 raft_consensus.cc:1275] T 881773ceed164ddf8ebc390254b1eb01 P 2fbc4f1ad97a4f2492306d77a614527a [term 3 FOLLOWER]: Refusing update from remote peer e77b2d60f57d4dceb865e440eab35126: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:27:46.848428 30937 raft_consensus.cc:1275] T b555d1cdbcec4c278fd300d628a8b9ab P e77b2d60f57d4dceb865e440eab35126 [term 3 FOLLOWER]: Refusing update from remote peer 2fbc4f1ad97a4f2492306d77a614527a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:27:46.849821 31284 consensus_queue.cc:1048] T 881773ceed164ddf8ebc390254b1eb01 P e77b2d60f57d4dceb865e440eab35126 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:27:46.851624 31253 consensus_queue.cc:1048] T b555d1cdbcec4c278fd300d628a8b9ab P 2fbc4f1ad97a4f2492306d77a614527a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20260709 12:27:46.867029 30809 auto_rebalancer.cc:288] Could not retrieve cluster info: Illegal state: not all tservers available for tablet placement
I20260709 12:27:46.873137 31198 raft_consensus.cc:1275] T 881773ceed164ddf8ebc390254b1eb01 P 75f56c1a7f86421da9247e9b35990e34 [term 3 FOLLOWER]: Refusing update from remote peer e77b2d60f57d4dceb865e440eab35126: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:27:46.875236 31167 consensus_queue.cc:1048] T 881773ceed164ddf8ebc390254b1eb01 P e77b2d60f57d4dceb865e440eab35126 [LEADER]: Connected to new peer: Peer: permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:27:47.127094 31011 raft_consensus.cc:1275] T 0be78a9b3c5d47df8efea08d91f58111 P 2fbc4f1ad97a4f2492306d77a614527a [term 4 FOLLOWER]: Refusing update from remote peer e77b2d60f57d4dceb865e440eab35126: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:27:47.128501 31167 consensus_queue.cc:1048] T 0be78a9b3c5d47df8efea08d91f58111 P e77b2d60f57d4dceb865e440eab35126 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:27:47.137960 31198 raft_consensus.cc:1275] T 0be78a9b3c5d47df8efea08d91f58111 P 75f56c1a7f86421da9247e9b35990e34 [term 4 FOLLOWER]: Refusing update from remote peer e77b2d60f57d4dceb865e440eab35126: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:27:47.139557 31290 consensus_queue.cc:1048] T 0be78a9b3c5d47df8efea08d91f58111 P e77b2d60f57d4dceb865e440eab35126 [LEADER]: Connected to new peer: Peer: permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:27:47.148085 16501 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:27:47.158450 30937 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "881773ceed164ddf8ebc390254b1eb01"
dest_uuid: "e77b2d60f57d4dceb865e440eab35126"
mode: GRACEFUL
new_leader_uuid: "2fbc4f1ad97a4f2492306d77a614527a"
 from {username='slave'} at 127.0.0.1:33634
I20260709 12:27:47.159149 30937 raft_consensus.cc:606] T 881773ceed164ddf8ebc390254b1eb01 P e77b2d60f57d4dceb865e440eab35126 [term 3 LEADER]: Received request to transfer leadership to TS 2fbc4f1ad97a4f2492306d77a614527a
I20260709 12:27:47.162060 30937 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "adedadb9002047ba87a533a33374f4e7"
dest_uuid: "e77b2d60f57d4dceb865e440eab35126"
mode: GRACEFUL
new_leader_uuid: "2fbc4f1ad97a4f2492306d77a614527a"
 from {username='slave'} at 127.0.0.1:33634
I20260709 12:27:47.162614 30937 raft_consensus.cc:606] T adedadb9002047ba87a533a33374f4e7 P e77b2d60f57d4dceb865e440eab35126 [term 1 LEADER]: Received request to transfer leadership to TS 2fbc4f1ad97a4f2492306d77a614527a
I20260709 12:27:47.165575 31011 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "f64692cd6a0347979834f29861bde5d4"
dest_uuid: "2fbc4f1ad97a4f2492306d77a614527a"
mode: GRACEFUL
new_leader_uuid: "e77b2d60f57d4dceb865e440eab35126"
 from {username='slave'} at 127.0.0.1:53526
I20260709 12:27:47.166121 31011 raft_consensus.cc:606] T f64692cd6a0347979834f29861bde5d4 P 2fbc4f1ad97a4f2492306d77a614527a [term 1 LEADER]: Received request to transfer leadership to TS e77b2d60f57d4dceb865e440eab35126
I20260709 12:27:47.167815 16501 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 3
I20260709 12:27:47.168370 16501 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:27:47.168987 16501 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:27:47.369269 31167 raft_consensus.cc:993] T adedadb9002047ba87a533a33374f4e7 P e77b2d60f57d4dceb865e440eab35126: : Instructing follower 2fbc4f1ad97a4f2492306d77a614527a to start an election
I20260709 12:27:47.369815 31284 raft_consensus.cc:1081] T adedadb9002047ba87a533a33374f4e7 P e77b2d60f57d4dceb865e440eab35126 [term 1 LEADER]: Signalling peer 2fbc4f1ad97a4f2492306d77a614527a to start an election
I20260709 12:27:47.371975 31012 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "adedadb9002047ba87a533a33374f4e7"
dest_uuid: "2fbc4f1ad97a4f2492306d77a614527a"
 from {username='slave'} at 127.0.0.1:39866
I20260709 12:27:47.372596 31012 raft_consensus.cc:493] T adedadb9002047ba87a533a33374f4e7 P 2fbc4f1ad97a4f2492306d77a614527a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:27:47.372859 31167 raft_consensus.cc:993] T 881773ceed164ddf8ebc390254b1eb01 P e77b2d60f57d4dceb865e440eab35126: : Instructing follower 2fbc4f1ad97a4f2492306d77a614527a to start an election
I20260709 12:27:47.372982 31012 raft_consensus.cc:3060] T adedadb9002047ba87a533a33374f4e7 P 2fbc4f1ad97a4f2492306d77a614527a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:47.373519 31167 raft_consensus.cc:1081] T 881773ceed164ddf8ebc390254b1eb01 P e77b2d60f57d4dceb865e440eab35126 [term 3 LEADER]: Signalling peer 2fbc4f1ad97a4f2492306d77a614527a to start an election
I20260709 12:27:47.375203 31011 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "881773ceed164ddf8ebc390254b1eb01"
dest_uuid: "2fbc4f1ad97a4f2492306d77a614527a"
 from {username='slave'} at 127.0.0.1:39866
I20260709 12:27:47.375775 31011 raft_consensus.cc:493] T 881773ceed164ddf8ebc390254b1eb01 P 2fbc4f1ad97a4f2492306d77a614527a [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:27:47.376156 31011 raft_consensus.cc:3060] T 881773ceed164ddf8ebc390254b1eb01 P 2fbc4f1ad97a4f2492306d77a614527a [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:27:47.380281 31012 raft_consensus.cc:515] T adedadb9002047ba87a533a33374f4e7 P 2fbc4f1ad97a4f2492306d77a614527a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } }
I20260709 12:27:47.381168 31011 raft_consensus.cc:515] T 881773ceed164ddf8ebc390254b1eb01 P 2fbc4f1ad97a4f2492306d77a614527a [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } }
I20260709 12:27:47.382254 31012 leader_election.cc:290] T adedadb9002047ba87a533a33374f4e7 P 2fbc4f1ad97a4f2492306d77a614527a [CANDIDATE]: Term 2 election: Requested vote from peers 75f56c1a7f86421da9247e9b35990e34 (127.16.29.65:45249), e77b2d60f57d4dceb865e440eab35126 (127.16.29.66:46695)
I20260709 12:27:47.383113 31198 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adedadb9002047ba87a533a33374f4e7" candidate_uuid: "2fbc4f1ad97a4f2492306d77a614527a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "75f56c1a7f86421da9247e9b35990e34"
I20260709 12:27:47.383504 30937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "adedadb9002047ba87a533a33374f4e7" candidate_uuid: "2fbc4f1ad97a4f2492306d77a614527a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e77b2d60f57d4dceb865e440eab35126"
I20260709 12:27:47.383778 31198 raft_consensus.cc:3060] T adedadb9002047ba87a533a33374f4e7 P 75f56c1a7f86421da9247e9b35990e34 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:47.383867 31199 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "881773ceed164ddf8ebc390254b1eb01" candidate_uuid: "2fbc4f1ad97a4f2492306d77a614527a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "75f56c1a7f86421da9247e9b35990e34"
I20260709 12:27:47.384382 30937 raft_consensus.cc:3055] T adedadb9002047ba87a533a33374f4e7 P e77b2d60f57d4dceb865e440eab35126 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:27:47.384655 31199 raft_consensus.cc:3060] T 881773ceed164ddf8ebc390254b1eb01 P 75f56c1a7f86421da9247e9b35990e34 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:27:47.384806 30937 raft_consensus.cc:740] T adedadb9002047ba87a533a33374f4e7 P e77b2d60f57d4dceb865e440eab35126 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e77b2d60f57d4dceb865e440eab35126, State: Running, Role: LEADER
I20260709 12:27:47.384404 31011 leader_election.cc:290] T 881773ceed164ddf8ebc390254b1eb01 P 2fbc4f1ad97a4f2492306d77a614527a [CANDIDATE]: Term 4 election: Requested vote from peers 75f56c1a7f86421da9247e9b35990e34 (127.16.29.65:45249), e77b2d60f57d4dceb865e440eab35126 (127.16.29.66:46695)
I20260709 12:27:47.385805 30938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "881773ceed164ddf8ebc390254b1eb01" candidate_uuid: "2fbc4f1ad97a4f2492306d77a614527a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "e77b2d60f57d4dceb865e440eab35126"
I20260709 12:27:47.385937 30937 consensus_queue.cc:260] T adedadb9002047ba87a533a33374f4e7 P e77b2d60f57d4dceb865e440eab35126 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } }
I20260709 12:27:47.386615 30938 raft_consensus.cc:3055] T 881773ceed164ddf8ebc390254b1eb01 P e77b2d60f57d4dceb865e440eab35126 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:27:47.387027 30938 raft_consensus.cc:740] T 881773ceed164ddf8ebc390254b1eb01 P e77b2d60f57d4dceb865e440eab35126 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: e77b2d60f57d4dceb865e440eab35126, State: Running, Role: LEADER
I20260709 12:27:47.387470 30937 raft_consensus.cc:3060] T adedadb9002047ba87a533a33374f4e7 P e77b2d60f57d4dceb865e440eab35126 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:47.388188 30938 consensus_queue.cc:260] T 881773ceed164ddf8ebc390254b1eb01 P e77b2d60f57d4dceb865e440eab35126 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } }
I20260709 12:27:47.389851 30938 raft_consensus.cc:3060] T 881773ceed164ddf8ebc390254b1eb01 P e77b2d60f57d4dceb865e440eab35126 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:27:47.393733 31198 raft_consensus.cc:2468] T adedadb9002047ba87a533a33374f4e7 P 75f56c1a7f86421da9247e9b35990e34 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2fbc4f1ad97a4f2492306d77a614527a in term 2.
I20260709 12:27:47.393990 31199 raft_consensus.cc:2468] T 881773ceed164ddf8ebc390254b1eb01 P 75f56c1a7f86421da9247e9b35990e34 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2fbc4f1ad97a4f2492306d77a614527a in term 4.
I20260709 12:27:47.395381 30977 leader_election.cc:304] T adedadb9002047ba87a533a33374f4e7 P 2fbc4f1ad97a4f2492306d77a614527a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2fbc4f1ad97a4f2492306d77a614527a, 75f56c1a7f86421da9247e9b35990e34; no voters: 
I20260709 12:27:47.395315 30937 raft_consensus.cc:2468] T adedadb9002047ba87a533a33374f4e7 P e77b2d60f57d4dceb865e440eab35126 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2fbc4f1ad97a4f2492306d77a614527a in term 2.
I20260709 12:27:47.396735 31302 raft_consensus.cc:2804] T adedadb9002047ba87a533a33374f4e7 P 2fbc4f1ad97a4f2492306d77a614527a [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:27:47.396735 30977 leader_election.cc:304] T 881773ceed164ddf8ebc390254b1eb01 P 2fbc4f1ad97a4f2492306d77a614527a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2fbc4f1ad97a4f2492306d77a614527a, 75f56c1a7f86421da9247e9b35990e34; no voters: 
I20260709 12:27:47.397363 31302 raft_consensus.cc:697] T adedadb9002047ba87a533a33374f4e7 P 2fbc4f1ad97a4f2492306d77a614527a [term 2 LEADER]: Becoming Leader. State: Replica: 2fbc4f1ad97a4f2492306d77a614527a, State: Running, Role: LEADER
I20260709 12:27:47.398244 31275 raft_consensus.cc:2804] T 881773ceed164ddf8ebc390254b1eb01 P 2fbc4f1ad97a4f2492306d77a614527a [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:27:47.398831 31275 raft_consensus.cc:697] T 881773ceed164ddf8ebc390254b1eb01 P 2fbc4f1ad97a4f2492306d77a614527a [term 4 LEADER]: Becoming Leader. State: Replica: 2fbc4f1ad97a4f2492306d77a614527a, State: Running, Role: LEADER
I20260709 12:27:47.398414 31302 consensus_queue.cc:237] T adedadb9002047ba87a533a33374f4e7 P 2fbc4f1ad97a4f2492306d77a614527a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } }
I20260709 12:27:47.399477 30938 raft_consensus.cc:2468] T 881773ceed164ddf8ebc390254b1eb01 P e77b2d60f57d4dceb865e440eab35126 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2fbc4f1ad97a4f2492306d77a614527a in term 4.
I20260709 12:27:47.399783 31275 consensus_queue.cc:237] T 881773ceed164ddf8ebc390254b1eb01 P 2fbc4f1ad97a4f2492306d77a614527a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } }
I20260709 12:27:47.407771 30760 catalog_manager.cc:5697] T adedadb9002047ba87a533a33374f4e7 P 2fbc4f1ad97a4f2492306d77a614527a reported cstate change: term changed from 1 to 2, leader changed from e77b2d60f57d4dceb865e440eab35126 (127.16.29.66) to 2fbc4f1ad97a4f2492306d77a614527a (127.16.29.67). New cstate: current_term: 2 leader_uuid: "2fbc4f1ad97a4f2492306d77a614527a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } health_report { overall_health: HEALTHY } } }
I20260709 12:27:47.424459 30760 catalog_manager.cc:5697] T 881773ceed164ddf8ebc390254b1eb01 P 2fbc4f1ad97a4f2492306d77a614527a reported cstate change: term changed from 3 to 4, leader changed from e77b2d60f57d4dceb865e440eab35126 (127.16.29.66) to 2fbc4f1ad97a4f2492306d77a614527a (127.16.29.67). New cstate: current_term: 4 leader_uuid: "2fbc4f1ad97a4f2492306d77a614527a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } health_report { overall_health: HEALTHY } } }
I20260709 12:27:47.490607 31275 raft_consensus.cc:993] T f64692cd6a0347979834f29861bde5d4 P 2fbc4f1ad97a4f2492306d77a614527a: : Instructing follower e77b2d60f57d4dceb865e440eab35126 to start an election
I20260709 12:27:47.491103 31302 raft_consensus.cc:1081] T f64692cd6a0347979834f29861bde5d4 P 2fbc4f1ad97a4f2492306d77a614527a [term 1 LEADER]: Signalling peer e77b2d60f57d4dceb865e440eab35126 to start an election
I20260709 12:27:47.493247 30938 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "f64692cd6a0347979834f29861bde5d4"
dest_uuid: "e77b2d60f57d4dceb865e440eab35126"
 from {username='slave'} at 127.0.0.1:42606
I20260709 12:27:47.493737 30938 raft_consensus.cc:493] T f64692cd6a0347979834f29861bde5d4 P e77b2d60f57d4dceb865e440eab35126 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:27:47.494005 30938 raft_consensus.cc:3060] T f64692cd6a0347979834f29861bde5d4 P e77b2d60f57d4dceb865e440eab35126 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:47.500011 30938 raft_consensus.cc:515] T f64692cd6a0347979834f29861bde5d4 P e77b2d60f57d4dceb865e440eab35126 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } }
I20260709 12:27:47.502120 30938 leader_election.cc:290] T f64692cd6a0347979834f29861bde5d4 P e77b2d60f57d4dceb865e440eab35126 [CANDIDATE]: Term 2 election: Requested vote from peers 75f56c1a7f86421da9247e9b35990e34 (127.16.29.65:45249), 2fbc4f1ad97a4f2492306d77a614527a (127.16.29.67:44761)
I20260709 12:27:47.503098 31199 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f64692cd6a0347979834f29861bde5d4" candidate_uuid: "e77b2d60f57d4dceb865e440eab35126" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "75f56c1a7f86421da9247e9b35990e34"
I20260709 12:27:47.503185 31011 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f64692cd6a0347979834f29861bde5d4" candidate_uuid: "e77b2d60f57d4dceb865e440eab35126" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2fbc4f1ad97a4f2492306d77a614527a"
I20260709 12:27:47.503755 31199 raft_consensus.cc:3060] T f64692cd6a0347979834f29861bde5d4 P 75f56c1a7f86421da9247e9b35990e34 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:47.503954 31011 raft_consensus.cc:3055] T f64692cd6a0347979834f29861bde5d4 P 2fbc4f1ad97a4f2492306d77a614527a [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:27:47.504401 31011 raft_consensus.cc:740] T f64692cd6a0347979834f29861bde5d4 P 2fbc4f1ad97a4f2492306d77a614527a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2fbc4f1ad97a4f2492306d77a614527a, State: Running, Role: LEADER
I20260709 12:27:47.505501 31011 consensus_queue.cc:260] T f64692cd6a0347979834f29861bde5d4 P 2fbc4f1ad97a4f2492306d77a614527a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } }
I20260709 12:27:47.506628 31011 raft_consensus.cc:3060] T f64692cd6a0347979834f29861bde5d4 P 2fbc4f1ad97a4f2492306d77a614527a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:47.509927 31199 raft_consensus.cc:2468] T f64692cd6a0347979834f29861bde5d4 P 75f56c1a7f86421da9247e9b35990e34 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e77b2d60f57d4dceb865e440eab35126 in term 2.
I20260709 12:27:47.511071 30903 leader_election.cc:304] T f64692cd6a0347979834f29861bde5d4 P e77b2d60f57d4dceb865e440eab35126 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 75f56c1a7f86421da9247e9b35990e34, e77b2d60f57d4dceb865e440eab35126; no voters: 
I20260709 12:27:47.511840 31167 raft_consensus.cc:2804] T f64692cd6a0347979834f29861bde5d4 P e77b2d60f57d4dceb865e440eab35126 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:27:47.512416 31167 raft_consensus.cc:697] T f64692cd6a0347979834f29861bde5d4 P e77b2d60f57d4dceb865e440eab35126 [term 2 LEADER]: Becoming Leader. State: Replica: e77b2d60f57d4dceb865e440eab35126, State: Running, Role: LEADER
I20260709 12:27:47.512629 31011 raft_consensus.cc:2468] T f64692cd6a0347979834f29861bde5d4 P 2fbc4f1ad97a4f2492306d77a614527a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e77b2d60f57d4dceb865e440eab35126 in term 2.
I20260709 12:27:47.513465 31167 consensus_queue.cc:237] T f64692cd6a0347979834f29861bde5d4 P e77b2d60f57d4dceb865e440eab35126 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } }
I20260709 12:27:47.522334 30760 catalog_manager.cc:5697] T f64692cd6a0347979834f29861bde5d4 P e77b2d60f57d4dceb865e440eab35126 reported cstate change: term changed from 1 to 2, leader changed from 2fbc4f1ad97a4f2492306d77a614527a (127.16.29.67) to e77b2d60f57d4dceb865e440eab35126 (127.16.29.66). New cstate: current_term: 2 leader_uuid: "e77b2d60f57d4dceb865e440eab35126" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:47.824613 31199 raft_consensus.cc:1275] T adedadb9002047ba87a533a33374f4e7 P 75f56c1a7f86421da9247e9b35990e34 [term 2 FOLLOWER]: Refusing update from remote peer 2fbc4f1ad97a4f2492306d77a614527a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:27:47.826086 31302 consensus_queue.cc:1048] T adedadb9002047ba87a533a33374f4e7 P 2fbc4f1ad97a4f2492306d77a614527a [LEADER]: Connected to new peer: Peer: permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:27:47.832868 30938 raft_consensus.cc:1275] T adedadb9002047ba87a533a33374f4e7 P e77b2d60f57d4dceb865e440eab35126 [term 2 FOLLOWER]: Refusing update from remote peer 2fbc4f1ad97a4f2492306d77a614527a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:27:47.834976 31275 consensus_queue.cc:1048] T adedadb9002047ba87a533a33374f4e7 P 2fbc4f1ad97a4f2492306d77a614527a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260709 12:27:47.868076 30809 auto_rebalancer.cc:288] Could not retrieve cluster info: Illegal state: not all tservers available for tablet placement
I20260709 12:27:47.870718 30938 raft_consensus.cc:1275] T 881773ceed164ddf8ebc390254b1eb01 P e77b2d60f57d4dceb865e440eab35126 [term 4 FOLLOWER]: Refusing update from remote peer 2fbc4f1ad97a4f2492306d77a614527a: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:27:47.872761 31253 consensus_queue.cc:1048] T 881773ceed164ddf8ebc390254b1eb01 P 2fbc4f1ad97a4f2492306d77a614527a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e77b2d60f57d4dceb865e440eab35126" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:27:47.885526 31199 raft_consensus.cc:1275] T 881773ceed164ddf8ebc390254b1eb01 P 75f56c1a7f86421da9247e9b35990e34 [term 4 FOLLOWER]: Refusing update from remote peer 2fbc4f1ad97a4f2492306d77a614527a: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:27:47.887825 31253 consensus_queue.cc:1048] T 881773ceed164ddf8ebc390254b1eb01 P 2fbc4f1ad97a4f2492306d77a614527a [LEADER]: Connected to new peer: Peer: permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:27:47.897822 31011 raft_consensus.cc:1275] T f64692cd6a0347979834f29861bde5d4 P 2fbc4f1ad97a4f2492306d77a614527a [term 2 FOLLOWER]: Refusing update from remote peer e77b2d60f57d4dceb865e440eab35126: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:27:47.899219 31284 consensus_queue.cc:1048] T f64692cd6a0347979834f29861bde5d4 P e77b2d60f57d4dceb865e440eab35126 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2fbc4f1ad97a4f2492306d77a614527a" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 44761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:27:47.911535 31199 raft_consensus.cc:1275] T f64692cd6a0347979834f29861bde5d4 P 75f56c1a7f86421da9247e9b35990e34 [term 2 FOLLOWER]: Refusing update from remote peer e77b2d60f57d4dceb865e440eab35126: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:27:47.913947 31167 consensus_queue.cc:1048] T f64692cd6a0347979834f29861bde5d4 P e77b2d60f57d4dceb865e440eab35126 [LEADER]: Connected to new peer: Peer: permanent_uuid: "75f56c1a7f86421da9247e9b35990e34" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 45249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:27:48.177974 16501 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:27:48.213457 16501 tablet_server.cc:179] TabletServer@127.16.29.65:45249 shutting down...
I20260709 12:27:48.233831 16501 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:27:48.234488 16501 tablet_replica.cc:333] stopping tablet replica
I20260709 12:27:48.235217 16501 raft_consensus.cc:2243] T 8e560587c5c24e4587295d2d1b6d2be6 P 75f56c1a7f86421da9247e9b35990e34 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:48.235781 16501 raft_consensus.cc:2272] T 8e560587c5c24e4587295d2d1b6d2be6 P 75f56c1a7f86421da9247e9b35990e34 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:48.238384 16501 tablet_replica.cc:333] stopping tablet replica
I20260709 12:27:48.239073 16501 raft_consensus.cc:2243] T de8ccd191af64c50ac32e3342ca995b0 P 75f56c1a7f86421da9247e9b35990e34 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:48.239508 16501 raft_consensus.cc:2272] T de8ccd191af64c50ac32e3342ca995b0 P 75f56c1a7f86421da9247e9b35990e34 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:48.241681 16501 tablet_replica.cc:333] stopping tablet replica
I20260709 12:27:48.242348 16501 raft_consensus.cc:2243] T b555d1cdbcec4c278fd300d628a8b9ab P 75f56c1a7f86421da9247e9b35990e34 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:48.242825 16501 raft_consensus.cc:2272] T b555d1cdbcec4c278fd300d628a8b9ab P 75f56c1a7f86421da9247e9b35990e34 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:48.244814 16501 tablet_replica.cc:333] stopping tablet replica
I20260709 12:27:48.245363 16501 raft_consensus.cc:2243] T 881773ceed164ddf8ebc390254b1eb01 P 75f56c1a7f86421da9247e9b35990e34 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:48.245826 16501 raft_consensus.cc:2272] T 881773ceed164ddf8ebc390254b1eb01 P 75f56c1a7f86421da9247e9b35990e34 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:48.248365 16501 tablet_replica.cc:333] stopping tablet replica
I20260709 12:27:48.248957 16501 raft_consensus.cc:2243] T 0be78a9b3c5d47df8efea08d91f58111 P 75f56c1a7f86421da9247e9b35990e34 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:48.249418 16501 raft_consensus.cc:2272] T 0be78a9b3c5d47df8efea08d91f58111 P 75f56c1a7f86421da9247e9b35990e34 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:48.251224 16501 tablet_replica.cc:333] stopping tablet replica
I20260709 12:27:48.251773 16501 raft_consensus.cc:2243] T adedadb9002047ba87a533a33374f4e7 P 75f56c1a7f86421da9247e9b35990e34 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:48.252271 16501 raft_consensus.cc:2272] T adedadb9002047ba87a533a33374f4e7 P 75f56c1a7f86421da9247e9b35990e34 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:48.254586 16501 tablet_replica.cc:333] stopping tablet replica
I20260709 12:27:48.255158 16501 raft_consensus.cc:2243] T 3ed242e2c2b843978a7af7cca5694ae2 P 75f56c1a7f86421da9247e9b35990e34 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:48.255635 16501 raft_consensus.cc:2272] T 3ed242e2c2b843978a7af7cca5694ae2 P 75f56c1a7f86421da9247e9b35990e34 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:48.257628 16501 tablet_replica.cc:333] stopping tablet replica
I20260709 12:27:48.258183 16501 raft_consensus.cc:2243] T f64692cd6a0347979834f29861bde5d4 P 75f56c1a7f86421da9247e9b35990e34 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:48.258627 16501 raft_consensus.cc:2272] T f64692cd6a0347979834f29861bde5d4 P 75f56c1a7f86421da9247e9b35990e34 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:48.261296 16501 tablet_replica.cc:333] stopping tablet replica
I20260709 12:27:48.261912 16501 raft_consensus.cc:2243] T 2c458afcbd8848d2aa9893f4e74c8561 P 75f56c1a7f86421da9247e9b35990e34 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:48.262336 16501 raft_consensus.cc:2272] T 2c458afcbd8848d2aa9893f4e74c8561 P 75f56c1a7f86421da9247e9b35990e34 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:48.264300 16501 tablet_replica.cc:333] stopping tablet replica
I20260709 12:27:48.264820 16501 raft_consensus.cc:2243] T 009d38185f3e49afbbd9faf57f968b2c P 75f56c1a7f86421da9247e9b35990e34 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:48.265285 16501 raft_consensus.cc:2272] T 009d38185f3e49afbbd9faf57f968b2c P 75f56c1a7f86421da9247e9b35990e34 [term 2 FOLLOWER]: Raft consensus is shut down!
W20260709 12:27:48.273902 30903 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 3 similar messages]
W20260709 12:27:48.277981 30903 consensus_peers.cc:597] T 0be78a9b3c5d47df8efea08d91f58111 P e77b2d60f57d4dceb865e440eab35126 -> Peer 75f56c1a7f86421da9247e9b35990e34 (127.16.29.65:45249): Couldn't send request to peer 75f56c1a7f86421da9247e9b35990e34. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:48.299340 16501 tablet_server.cc:196] TabletServer@127.16.29.65:45249 shutdown complete.
W20260709 12:27:48.304157 30977 consensus_peers.cc:597] T 3ed242e2c2b843978a7af7cca5694ae2 P 2fbc4f1ad97a4f2492306d77a614527a -> Peer 75f56c1a7f86421da9247e9b35990e34 (127.16.29.65:45249): Couldn't send request to peer 75f56c1a7f86421da9247e9b35990e34. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:45249: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:48.312479 30977 consensus_peers.cc:597] T 009d38185f3e49afbbd9faf57f968b2c P 2fbc4f1ad97a4f2492306d77a614527a -> Peer 75f56c1a7f86421da9247e9b35990e34 (127.16.29.65:45249): Couldn't send request to peer 75f56c1a7f86421da9247e9b35990e34. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:45249: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:48.327297 16501 tablet_server.cc:179] TabletServer@127.16.29.66:0 shutting down...
W20260709 12:27:48.353721 30977 consensus_peers.cc:597] T 881773ceed164ddf8ebc390254b1eb01 P 2fbc4f1ad97a4f2492306d77a614527a -> Peer 75f56c1a7f86421da9247e9b35990e34 (127.16.29.65:45249): Couldn't send request to peer 75f56c1a7f86421da9247e9b35990e34. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:45249: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:48.356950 16501 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:27:48.357828 16501 tablet_replica.cc:333] T 009d38185f3e49afbbd9faf57f968b2c P e77b2d60f57d4dceb865e440eab35126: stopping tablet replica
I20260709 12:27:48.358511 16501 raft_consensus.cc:2243] T 009d38185f3e49afbbd9faf57f968b2c P e77b2d60f57d4dceb865e440eab35126 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:48.358978 16501 raft_consensus.cc:2272] T 009d38185f3e49afbbd9faf57f968b2c P e77b2d60f57d4dceb865e440eab35126 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:48.361617 16501 tablet_replica.cc:333] T adedadb9002047ba87a533a33374f4e7 P e77b2d60f57d4dceb865e440eab35126: stopping tablet replica
I20260709 12:27:48.362278 16501 raft_consensus.cc:2243] T adedadb9002047ba87a533a33374f4e7 P e77b2d60f57d4dceb865e440eab35126 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:48.362704 16501 raft_consensus.cc:2272] T adedadb9002047ba87a533a33374f4e7 P e77b2d60f57d4dceb865e440eab35126 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:48.366191 16501 tablet_replica.cc:333] T 3ed242e2c2b843978a7af7cca5694ae2 P e77b2d60f57d4dceb865e440eab35126: stopping tablet replica
I20260709 12:27:48.366817 16501 raft_consensus.cc:2243] T 3ed242e2c2b843978a7af7cca5694ae2 P e77b2d60f57d4dceb865e440eab35126 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:48.367297 16501 raft_consensus.cc:2272] T 3ed242e2c2b843978a7af7cca5694ae2 P e77b2d60f57d4dceb865e440eab35126 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:48.370342 16501 tablet_replica.cc:333] T de8ccd191af64c50ac32e3342ca995b0 P e77b2d60f57d4dceb865e440eab35126: stopping tablet replica
I20260709 12:27:48.370975 16501 raft_consensus.cc:2243] T de8ccd191af64c50ac32e3342ca995b0 P e77b2d60f57d4dceb865e440eab35126 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:27:48.372141 16501 raft_consensus.cc:2272] T de8ccd191af64c50ac32e3342ca995b0 P e77b2d60f57d4dceb865e440eab35126 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 12:27:48.373569 30977 consensus_peers.cc:597] T adedadb9002047ba87a533a33374f4e7 P 2fbc4f1ad97a4f2492306d77a614527a -> Peer 75f56c1a7f86421da9247e9b35990e34 (127.16.29.65:45249): Couldn't send request to peer 75f56c1a7f86421da9247e9b35990e34. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:45249: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:48.375829 16501 tablet_replica.cc:333] T 881773ceed164ddf8ebc390254b1eb01 P e77b2d60f57d4dceb865e440eab35126: stopping tablet replica
I20260709 12:27:48.376858 16501 raft_consensus.cc:2243] T 881773ceed164ddf8ebc390254b1eb01 P e77b2d60f57d4dceb865e440eab35126 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:48.377394 16501 raft_consensus.cc:2272] T 881773ceed164ddf8ebc390254b1eb01 P e77b2d60f57d4dceb865e440eab35126 [term 4 FOLLOWER]: Raft consensus is shut down!
W20260709 12:27:48.379990 30903 consensus_peers.cc:597] T f64692cd6a0347979834f29861bde5d4 P e77b2d60f57d4dceb865e440eab35126 -> Peer 75f56c1a7f86421da9247e9b35990e34 (127.16.29.65:45249): Couldn't send request to peer 75f56c1a7f86421da9247e9b35990e34. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:45249: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:48.380843 16501 tablet_replica.cc:333] T b555d1cdbcec4c278fd300d628a8b9ab P e77b2d60f57d4dceb865e440eab35126: stopping tablet replica
I20260709 12:27:48.381516 16501 raft_consensus.cc:2243] T b555d1cdbcec4c278fd300d628a8b9ab P e77b2d60f57d4dceb865e440eab35126 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:48.381932 16501 raft_consensus.cc:2272] T b555d1cdbcec4c278fd300d628a8b9ab P e77b2d60f57d4dceb865e440eab35126 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:48.384332 16501 tablet_replica.cc:333] T 0be78a9b3c5d47df8efea08d91f58111 P e77b2d60f57d4dceb865e440eab35126: stopping tablet replica
I20260709 12:27:48.384922 16501 raft_consensus.cc:2243] T 0be78a9b3c5d47df8efea08d91f58111 P e77b2d60f57d4dceb865e440eab35126 [term 4 LEADER]: Raft consensus shutting down.
I20260709 12:27:48.385695 16501 raft_consensus.cc:2272] T 0be78a9b3c5d47df8efea08d91f58111 P e77b2d60f57d4dceb865e440eab35126 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:48.388094 16501 tablet_replica.cc:333] T 8e560587c5c24e4587295d2d1b6d2be6 P e77b2d60f57d4dceb865e440eab35126: stopping tablet replica
I20260709 12:27:48.388650 16501 raft_consensus.cc:2243] T 8e560587c5c24e4587295d2d1b6d2be6 P e77b2d60f57d4dceb865e440eab35126 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:48.389147 16501 raft_consensus.cc:2272] T 8e560587c5c24e4587295d2d1b6d2be6 P e77b2d60f57d4dceb865e440eab35126 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:48.392086 16501 tablet_replica.cc:333] T f64692cd6a0347979834f29861bde5d4 P e77b2d60f57d4dceb865e440eab35126: stopping tablet replica
I20260709 12:27:48.392650 16501 raft_consensus.cc:2243] T f64692cd6a0347979834f29861bde5d4 P e77b2d60f57d4dceb865e440eab35126 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:27:48.394210 16501 raft_consensus.cc:2272] T f64692cd6a0347979834f29861bde5d4 P e77b2d60f57d4dceb865e440eab35126 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:48.397691 16501 tablet_replica.cc:333] T 2c458afcbd8848d2aa9893f4e74c8561 P e77b2d60f57d4dceb865e440eab35126: stopping tablet replica
W20260709 12:27:48.397711 30976 consensus_peers.cc:597] T 881773ceed164ddf8ebc390254b1eb01 P 2fbc4f1ad97a4f2492306d77a614527a -> Peer e77b2d60f57d4dceb865e440eab35126 (127.16.29.66:46695): Couldn't send request to peer e77b2d60f57d4dceb865e440eab35126. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:48.398531 16501 raft_consensus.cc:2243] T 2c458afcbd8848d2aa9893f4e74c8561 P e77b2d60f57d4dceb865e440eab35126 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:27:48.399471 16501 raft_consensus.cc:2272] T 2c458afcbd8848d2aa9893f4e74c8561 P e77b2d60f57d4dceb865e440eab35126 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:48.434628 16501 tablet_server.cc:196] TabletServer@127.16.29.66:0 shutdown complete.
I20260709 12:27:48.461015 16501 tablet_server.cc:179] TabletServer@127.16.29.67:0 shutting down...
W20260709 12:27:48.474606 30976 consensus_peers.cc:597] T 3ed242e2c2b843978a7af7cca5694ae2 P 2fbc4f1ad97a4f2492306d77a614527a -> Peer e77b2d60f57d4dceb865e440eab35126 (127.16.29.66:46695): Couldn't send request to peer e77b2d60f57d4dceb865e440eab35126. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.66:46695: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:48.478747 30976 consensus_peers.cc:597] T 009d38185f3e49afbbd9faf57f968b2c P 2fbc4f1ad97a4f2492306d77a614527a -> Peer e77b2d60f57d4dceb865e440eab35126 (127.16.29.66:46695): Couldn't send request to peer e77b2d60f57d4dceb865e440eab35126. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.66:46695: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:48.481565 30976 consensus_peers.cc:597] T b555d1cdbcec4c278fd300d628a8b9ab P 2fbc4f1ad97a4f2492306d77a614527a -> Peer e77b2d60f57d4dceb865e440eab35126 (127.16.29.66:46695): Couldn't send request to peer e77b2d60f57d4dceb865e440eab35126. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.66:46695: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:48.492753 16501 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:27:48.493577 16501 tablet_replica.cc:333] T adedadb9002047ba87a533a33374f4e7 P 2fbc4f1ad97a4f2492306d77a614527a: stopping tablet replica
I20260709 12:27:48.494325 16501 raft_consensus.cc:2243] T adedadb9002047ba87a533a33374f4e7 P 2fbc4f1ad97a4f2492306d77a614527a [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:27:48.495206 16501 raft_consensus.cc:2272] T adedadb9002047ba87a533a33374f4e7 P 2fbc4f1ad97a4f2492306d77a614527a [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:48.497959 16501 tablet_replica.cc:333] T 3ed242e2c2b843978a7af7cca5694ae2 P 2fbc4f1ad97a4f2492306d77a614527a: stopping tablet replica
I20260709 12:27:48.498574 16501 raft_consensus.cc:2243] T 3ed242e2c2b843978a7af7cca5694ae2 P 2fbc4f1ad97a4f2492306d77a614527a [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:27:48.499409 16501 raft_consensus.cc:2272] T 3ed242e2c2b843978a7af7cca5694ae2 P 2fbc4f1ad97a4f2492306d77a614527a [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:48.501649 16501 tablet_replica.cc:333] T 009d38185f3e49afbbd9faf57f968b2c P 2fbc4f1ad97a4f2492306d77a614527a: stopping tablet replica
I20260709 12:27:48.502279 16501 raft_consensus.cc:2243] T 009d38185f3e49afbbd9faf57f968b2c P 2fbc4f1ad97a4f2492306d77a614527a [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:27:48.503088 16501 raft_consensus.cc:2272] T 009d38185f3e49afbbd9faf57f968b2c P 2fbc4f1ad97a4f2492306d77a614527a [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:48.505349 16501 tablet_replica.cc:333] T 8e560587c5c24e4587295d2d1b6d2be6 P 2fbc4f1ad97a4f2492306d77a614527a: stopping tablet replica
I20260709 12:27:48.505967 16501 raft_consensus.cc:2243] T 8e560587c5c24e4587295d2d1b6d2be6 P 2fbc4f1ad97a4f2492306d77a614527a [term 4 LEADER]: Raft consensus shutting down.
I20260709 12:27:48.506745 16501 raft_consensus.cc:2272] T 8e560587c5c24e4587295d2d1b6d2be6 P 2fbc4f1ad97a4f2492306d77a614527a [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:48.509120 16501 tablet_replica.cc:333] T de8ccd191af64c50ac32e3342ca995b0 P 2fbc4f1ad97a4f2492306d77a614527a: stopping tablet replica
I20260709 12:27:48.509699 16501 raft_consensus.cc:2243] T de8ccd191af64c50ac32e3342ca995b0 P 2fbc4f1ad97a4f2492306d77a614527a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:48.510187 16501 raft_consensus.cc:2272] T de8ccd191af64c50ac32e3342ca995b0 P 2fbc4f1ad97a4f2492306d77a614527a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:48.512563 16501 tablet_replica.cc:333] T 881773ceed164ddf8ebc390254b1eb01 P 2fbc4f1ad97a4f2492306d77a614527a: stopping tablet replica
I20260709 12:27:48.513180 16501 raft_consensus.cc:2243] T 881773ceed164ddf8ebc390254b1eb01 P 2fbc4f1ad97a4f2492306d77a614527a [term 4 LEADER]: Raft consensus shutting down.
I20260709 12:27:48.514045 16501 raft_consensus.cc:2272] T 881773ceed164ddf8ebc390254b1eb01 P 2fbc4f1ad97a4f2492306d77a614527a [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:48.517163 16501 tablet_replica.cc:333] T 0be78a9b3c5d47df8efea08d91f58111 P 2fbc4f1ad97a4f2492306d77a614527a: stopping tablet replica
I20260709 12:27:48.517813 16501 raft_consensus.cc:2243] T 0be78a9b3c5d47df8efea08d91f58111 P 2fbc4f1ad97a4f2492306d77a614527a [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:48.518253 16501 raft_consensus.cc:2272] T 0be78a9b3c5d47df8efea08d91f58111 P 2fbc4f1ad97a4f2492306d77a614527a [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:48.520450 16501 tablet_replica.cc:333] T b555d1cdbcec4c278fd300d628a8b9ab P 2fbc4f1ad97a4f2492306d77a614527a: stopping tablet replica
I20260709 12:27:48.520982 16501 raft_consensus.cc:2243] T b555d1cdbcec4c278fd300d628a8b9ab P 2fbc4f1ad97a4f2492306d77a614527a [term 3 LEADER]: Raft consensus shutting down.
I20260709 12:27:48.521749 16501 raft_consensus.cc:2272] T b555d1cdbcec4c278fd300d628a8b9ab P 2fbc4f1ad97a4f2492306d77a614527a [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:48.524047 16501 tablet_replica.cc:333] T f64692cd6a0347979834f29861bde5d4 P 2fbc4f1ad97a4f2492306d77a614527a: stopping tablet replica
I20260709 12:27:48.524612 16501 raft_consensus.cc:2243] T f64692cd6a0347979834f29861bde5d4 P 2fbc4f1ad97a4f2492306d77a614527a [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:48.525043 16501 raft_consensus.cc:2272] T f64692cd6a0347979834f29861bde5d4 P 2fbc4f1ad97a4f2492306d77a614527a [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:48.527599 16501 tablet_replica.cc:333] T 2c458afcbd8848d2aa9893f4e74c8561 P 2fbc4f1ad97a4f2492306d77a614527a: stopping tablet replica
I20260709 12:27:48.528151 16501 raft_consensus.cc:2243] T 2c458afcbd8848d2aa9893f4e74c8561 P 2fbc4f1ad97a4f2492306d77a614527a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:48.528594 16501 raft_consensus.cc:2272] T 2c458afcbd8848d2aa9893f4e74c8561 P 2fbc4f1ad97a4f2492306d77a614527a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:48.555028 16501 tablet_server.cc:196] TabletServer@127.16.29.67:0 shutdown complete.
I20260709 12:27:48.582031 16501 master.cc:562] Master@127.16.29.126:44357 shutting down...
I20260709 12:27:48.609099 16501 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 43c3827cbc0344b09cc6a5e8215bdb3e [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:27:48.609723 16501 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 43c3827cbc0344b09cc6a5e8215bdb3e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:48.610049 16501 tablet_replica.cc:333] T 00000000000000000000000000000000 P 43c3827cbc0344b09cc6a5e8215bdb3e: stopping tablet replica
I20260709 12:27:48.633041 16501 master.cc:584] Master@127.16.29.126:44357 shutdown complete.
[       OK ] LeaderRebalancerTest.TestMaintenanceMode (12986 ms)
[ RUN      ] LeaderRebalancerTest.MultiTableLeaderBalance
I20260709 12:27:48.682540 16501 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.16.29.126:43837
I20260709 12:27:48.683837 16501 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:27:48.689532 31324 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:27:48.690024 31325 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:27:48.691207 31327 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:27:48.692265 16501 server_base.cc:1061] running on GCE node
I20260709 12:27:48.693231 16501 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:27:48.693426 16501 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:27:48.693612 16501 hybrid_clock.cc:648] HybridClock initialized: now 1783600068693597 us; error 0 us; skew 500 ppm
I20260709 12:27:48.694209 16501 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:27:48.696648 16501 webserver.cc:533] Webserver started at http://127.16.29.126:41619/ using document root <none> and password file <none>
I20260709 12:27:48.697160 16501 fs_manager.cc:362] Metadata directory not provided
I20260709 12:27:48.697345 16501 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:27:48.697641 16501 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:27:48.698784 16501 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783599551912677-16501-0/minicluster-data/master-0-root/instance:
uuid: "48143a6fed1a4cceb7a1ecc2b35f83de"
format_stamp: "Formatted at 2026-07-09 12:27:48 on dist-test-slave-5n9v"
I20260709 12:27:48.703343 16501 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.006s	sys 0.000s
I20260709 12:27:48.707111 31332 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:27:48.708038 16501 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.000s	sys 0.003s
I20260709 12:27:48.708289 16501 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783599551912677-16501-0/minicluster-data/master-0-root
uuid: "48143a6fed1a4cceb7a1ecc2b35f83de"
format_stamp: "Formatted at 2026-07-09 12:27:48 on dist-test-slave-5n9v"
I20260709 12:27:48.708539 16501 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783599551912677-16501-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783599551912677-16501-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783599551912677-16501-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:27:48.748317 16501 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:27:48.749470 16501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:27:48.789777 16501 rpc_server.cc:307] RPC server started. Bound to: 127.16.29.126:43837
I20260709 12:27:48.789816 31383 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.29.126:43837 every 8 connection(s)
I20260709 12:27:48.793771 31384 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:48.804838 31384 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48143a6fed1a4cceb7a1ecc2b35f83de: Bootstrap starting.
I20260709 12:27:48.809849 31384 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 48143a6fed1a4cceb7a1ecc2b35f83de: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:48.814402 31384 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48143a6fed1a4cceb7a1ecc2b35f83de: No bootstrap required, opened a new log
I20260709 12:27:48.816758 31384 raft_consensus.cc:359] T 00000000000000000000000000000000 P 48143a6fed1a4cceb7a1ecc2b35f83de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48143a6fed1a4cceb7a1ecc2b35f83de" member_type: VOTER }
I20260709 12:27:48.817226 31384 raft_consensus.cc:385] T 00000000000000000000000000000000 P 48143a6fed1a4cceb7a1ecc2b35f83de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:48.817461 31384 raft_consensus.cc:740] T 00000000000000000000000000000000 P 48143a6fed1a4cceb7a1ecc2b35f83de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48143a6fed1a4cceb7a1ecc2b35f83de, State: Initialized, Role: FOLLOWER
I20260709 12:27:48.818024 31384 consensus_queue.cc:260] T 00000000000000000000000000000000 P 48143a6fed1a4cceb7a1ecc2b35f83de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48143a6fed1a4cceb7a1ecc2b35f83de" member_type: VOTER }
I20260709 12:27:48.818495 31384 raft_consensus.cc:399] T 00000000000000000000000000000000 P 48143a6fed1a4cceb7a1ecc2b35f83de [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260709 12:27:48.818740 31384 raft_consensus.cc:493] T 00000000000000000000000000000000 P 48143a6fed1a4cceb7a1ecc2b35f83de [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260709 12:27:48.819007 31384 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 48143a6fed1a4cceb7a1ecc2b35f83de [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:48.824613 31384 raft_consensus.cc:515] T 00000000000000000000000000000000 P 48143a6fed1a4cceb7a1ecc2b35f83de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48143a6fed1a4cceb7a1ecc2b35f83de" member_type: VOTER }
I20260709 12:27:48.825461 31384 leader_election.cc:304] T 00000000000000000000000000000000 P 48143a6fed1a4cceb7a1ecc2b35f83de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 48143a6fed1a4cceb7a1ecc2b35f83de; no voters: 
I20260709 12:27:48.826717 31384 leader_election.cc:290] T 00000000000000000000000000000000 P 48143a6fed1a4cceb7a1ecc2b35f83de [CANDIDATE]: Term 1 election: Requested vote from peers 
I20260709 12:27:48.827188 31387 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 48143a6fed1a4cceb7a1ecc2b35f83de [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:48.828634 31387 raft_consensus.cc:697] T 00000000000000000000000000000000 P 48143a6fed1a4cceb7a1ecc2b35f83de [term 1 LEADER]: Becoming Leader. State: Replica: 48143a6fed1a4cceb7a1ecc2b35f83de, State: Running, Role: LEADER
I20260709 12:27:48.829336 31387 consensus_queue.cc:237] T 00000000000000000000000000000000 P 48143a6fed1a4cceb7a1ecc2b35f83de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48143a6fed1a4cceb7a1ecc2b35f83de" member_type: VOTER }
I20260709 12:27:48.829583 31384 sys_catalog.cc:565] T 00000000000000000000000000000000 P 48143a6fed1a4cceb7a1ecc2b35f83de [sys.catalog]: configured and running, proceeding with master startup.
I20260709 12:27:48.834592 31389 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48143a6fed1a4cceb7a1ecc2b35f83de [sys.catalog]: SysCatalogTable state changed. Reason: New leader 48143a6fed1a4cceb7a1ecc2b35f83de. Latest consensus state: current_term: 1 leader_uuid: "48143a6fed1a4cceb7a1ecc2b35f83de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48143a6fed1a4cceb7a1ecc2b35f83de" member_type: VOTER } }
I20260709 12:27:48.834714 31388 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48143a6fed1a4cceb7a1ecc2b35f83de [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "48143a6fed1a4cceb7a1ecc2b35f83de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48143a6fed1a4cceb7a1ecc2b35f83de" member_type: VOTER } }
I20260709 12:27:48.835489 31389 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48143a6fed1a4cceb7a1ecc2b35f83de [sys.catalog]: This master's current role is: LEADER
I20260709 12:27:48.835706 31388 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48143a6fed1a4cceb7a1ecc2b35f83de [sys.catalog]: This master's current role is: LEADER
I20260709 12:27:48.838037 31395 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260709 12:27:48.843657 31395 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260709 12:27:48.847141 16501 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260709 12:27:48.853364 31395 catalog_manager.cc:1383] Generated new cluster ID: d176780cf6aa489eb218671234c88474
I20260709 12:27:48.853714 31395 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260709 12:27:48.869472 31395 catalog_manager.cc:1406] Generated new certificate authority record
I20260709 12:27:48.870950 31395 catalog_manager.cc:1540] Loading token signing keys...
I20260709 12:27:48.887694 31395 catalog_manager.cc:6070] T 00000000000000000000000000000000 P 48143a6fed1a4cceb7a1ecc2b35f83de: Generated new TSK 0
I20260709 12:27:48.888444 31395 catalog_manager.cc:1550] Initializing in-progress tserver states...
I20260709 12:27:48.914760 16501 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:27:48.921089 31405 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:27:48.922662 31406 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:27:48.924422 31408 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:27:48.924765 16501 server_base.cc:1061] running on GCE node
I20260709 12:27:48.925812 16501 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:27:48.926057 16501 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:27:48.926231 16501 hybrid_clock.cc:648] HybridClock initialized: now 1783600068926207 us; error 0 us; skew 500 ppm
I20260709 12:27:48.926774 16501 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:27:48.929340 16501 webserver.cc:533] Webserver started at http://127.16.29.65:45579/ using document root <none> and password file <none>
I20260709 12:27:48.929857 16501 fs_manager.cc:362] Metadata directory not provided
I20260709 12:27:48.930084 16501 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:27:48.930351 16501 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:27:48.931507 16501 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783599551912677-16501-0/minicluster-data/ts-0-root/instance:
uuid: "412e6ff66efa4d2ab996be122bfa1ed5"
format_stamp: "Formatted at 2026-07-09 12:27:48 on dist-test-slave-5n9v"
I20260709 12:27:48.936265 16501 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.000s	sys 0.003s
I20260709 12:27:48.940465 31413 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:27:48.941437 16501 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.004s	sys 0.000s
I20260709 12:27:48.941788 16501 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783599551912677-16501-0/minicluster-data/ts-0-root
uuid: "412e6ff66efa4d2ab996be122bfa1ed5"
format_stamp: "Formatted at 2026-07-09 12:27:48 on dist-test-slave-5n9v"
I20260709 12:27:48.942103 16501 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783599551912677-16501-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783599551912677-16501-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783599551912677-16501-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:27:48.974707 16501 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:27:48.976423 16501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:27:48.978129 16501 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:27:48.980585 16501 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:27:48.980827 16501 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:27:48.981093 16501 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:27:48.981271 16501 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:27:49.026350 16501 rpc_server.cc:307] RPC server started. Bound to: 127.16.29.65:43129
I20260709 12:27:49.026453 31475 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.29.65:43129 every 8 connection(s)
I20260709 12:27:49.031821 16501 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:27:49.040333 31480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:27:49.041188 31481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:27:49.044471 16501 server_base.cc:1061] running on GCE node
W20260709 12:27:49.045503 31483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:27:49.046947 16501 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:27:49.047259 16501 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:27:49.047495 16501 hybrid_clock.cc:648] HybridClock initialized: now 1783600069047476 us; error 0 us; skew 500 ppm
I20260709 12:27:49.048291 16501 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:27:49.049368 31476 heartbeater.cc:344] Connected to a master server at 127.16.29.126:43837
I20260709 12:27:49.049813 31476 heartbeater.cc:461] Registering TS with master...
I20260709 12:27:49.050686 31476 heartbeater.cc:507] Master 127.16.29.126:43837 requested a full tablet report, sending...
I20260709 12:27:49.051246 16501 webserver.cc:533] Webserver started at http://127.16.29.66:41075/ using document root <none> and password file <none>
I20260709 12:27:49.052006 16501 fs_manager.cc:362] Metadata directory not provided
I20260709 12:27:49.052232 16501 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:27:49.052536 16501 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:27:49.053412 31349 ts_manager.cc:194] Registered new tserver with Master: 412e6ff66efa4d2ab996be122bfa1ed5 (127.16.29.65:43129)
I20260709 12:27:49.054180 16501 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783599551912677-16501-0/minicluster-data/ts-1-root/instance:
uuid: "3e6943c7aa1f4133953e746fdeee2d5c"
format_stamp: "Formatted at 2026-07-09 12:27:49 on dist-test-slave-5n9v"
I20260709 12:27:49.056010 31349 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:33326
I20260709 12:27:49.060503 16501 fs_manager.cc:696] Time spent creating directory manager: real 0.006s	user 0.007s	sys 0.000s
I20260709 12:27:49.064275 31488 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:27:49.065280 16501 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.002s
I20260709 12:27:49.065609 16501 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783599551912677-16501-0/minicluster-data/ts-1-root
uuid: "3e6943c7aa1f4133953e746fdeee2d5c"
format_stamp: "Formatted at 2026-07-09 12:27:49 on dist-test-slave-5n9v"
I20260709 12:27:49.065879 16501 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783599551912677-16501-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783599551912677-16501-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783599551912677-16501-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:27:49.097353 16501 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:27:49.099433 16501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:27:49.101922 16501 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:27:49.105953 16501 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:27:49.106288 16501 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:27:49.106601 16501 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:27:49.107036 16501 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:27:49.155365 16501 rpc_server.cc:307] RPC server started. Bound to: 127.16.29.66:46089
I20260709 12:27:49.155474 31550 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.29.66:46089 every 8 connection(s)
I20260709 12:27:49.160190 16501 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:27:49.167600 31555 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:27:49.167709 31554 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:27:49.171399 31551 heartbeater.cc:344] Connected to a master server at 127.16.29.126:43837
I20260709 12:27:49.171497 16501 server_base.cc:1061] running on GCE node
W20260709 12:27:49.171631 31558 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:27:49.172001 31551 heartbeater.cc:461] Registering TS with master...
I20260709 12:27:49.173060 16501 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20260709 12:27:49.173091 31551 heartbeater.cc:507] Master 127.16.29.126:43837 requested a full tablet report, sending...
W20260709 12:27:49.173458 16501 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:27:49.173707 16501 hybrid_clock.cc:648] HybridClock initialized: now 1783600069173687 us; error 0 us; skew 500 ppm
I20260709 12:27:49.174422 16501 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:27:49.175503 31349 ts_manager.cc:194] Registered new tserver with Master: 3e6943c7aa1f4133953e746fdeee2d5c (127.16.29.66:46089)
I20260709 12:27:49.177117 16501 webserver.cc:533] Webserver started at http://127.16.29.67:39413/ using document root <none> and password file <none>
I20260709 12:27:49.177295 31349 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:33342
I20260709 12:27:49.177743 16501 fs_manager.cc:362] Metadata directory not provided
I20260709 12:27:49.177990 16501 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:27:49.178267 16501 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:27:49.179824 16501 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783599551912677-16501-0/minicluster-data/ts-2-root/instance:
uuid: "cbb731225b44446c86d71a20ed7d1a32"
format_stamp: "Formatted at 2026-07-09 12:27:49 on dist-test-slave-5n9v"
I20260709 12:27:49.184381 16501 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.004s	sys 0.000s
I20260709 12:27:49.187635 31562 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:27:49.188551 16501 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.001s
I20260709 12:27:49.188957 16501 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783599551912677-16501-0/minicluster-data/ts-2-root
uuid: "cbb731225b44446c86d71a20ed7d1a32"
format_stamp: "Formatted at 2026-07-09 12:27:49 on dist-test-slave-5n9v"
I20260709 12:27:49.189333 16501 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783599551912677-16501-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783599551912677-16501-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783599551912677-16501-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:27:49.211589 16501 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:27:49.212811 16501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:27:49.214100 16501 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:27:49.216271 16501 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:27:49.216514 16501 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:27:49.216760 16501 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:27:49.216957 16501 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:27:49.260977 16501 rpc_server.cc:307] RPC server started. Bound to: 127.16.29.67:34677
I20260709 12:27:49.261089 31624 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.29.67:34677 every 8 connection(s)
I20260709 12:27:49.276446 31625 heartbeater.cc:344] Connected to a master server at 127.16.29.126:43837
I20260709 12:27:49.276906 31625 heartbeater.cc:461] Registering TS with master...
I20260709 12:27:49.277657 31625 heartbeater.cc:507] Master 127.16.29.126:43837 requested a full tablet report, sending...
I20260709 12:27:49.279906 31349 ts_manager.cc:194] Registered new tserver with Master: cbb731225b44446c86d71a20ed7d1a32 (127.16.29.67:34677)
I20260709 12:27:49.280649 16501 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.016343922s
I20260709 12:27:49.281441 31349 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:33354
I20260709 12:27:49.306574 31349 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33368:
name: "multi_table_leader_balance_table1"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
  rows: "\004\001\000\377\377\377?""\004\001\000\377\377\377?"
  indirect_data: """"
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W20260709 12:27:49.308751 31349 catalog_manager.cc:7077] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table multi_table_leader_balance_table1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260709 12:27:49.363137 31441 tablet_service.cc:1511] Processing CreateTablet for tablet 229edfbe40a242629bb93e5c30b90c86 (DEFAULT_TABLE table=multi_table_leader_balance_table1 [id=2cd309a730d14d9aaaad3e0c9ff38d88]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20260709 12:27:49.363459 31440 tablet_service.cc:1511] Processing CreateTablet for tablet a8ceaa7770964d06a68441336935dd0b (DEFAULT_TABLE table=multi_table_leader_balance_table1 [id=2cd309a730d14d9aaaad3e0c9ff38d88]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20260709 12:27:49.363966 31589 tablet_service.cc:1511] Processing CreateTablet for tablet a8ceaa7770964d06a68441336935dd0b (DEFAULT_TABLE table=multi_table_leader_balance_table1 [id=2cd309a730d14d9aaaad3e0c9ff38d88]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20260709 12:27:49.364660 31441 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 229edfbe40a242629bb93e5c30b90c86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:49.365485 31589 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8ceaa7770964d06a68441336935dd0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:49.366102 31590 tablet_service.cc:1511] Processing CreateTablet for tablet 229edfbe40a242629bb93e5c30b90c86 (DEFAULT_TABLE table=multi_table_leader_balance_table1 [id=2cd309a730d14d9aaaad3e0c9ff38d88]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20260709 12:27:49.367365 31590 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 229edfbe40a242629bb93e5c30b90c86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:49.370174 31440 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8ceaa7770964d06a68441336935dd0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:49.373636 31516 tablet_service.cc:1511] Processing CreateTablet for tablet 229edfbe40a242629bb93e5c30b90c86 (DEFAULT_TABLE table=multi_table_leader_balance_table1 [id=2cd309a730d14d9aaaad3e0c9ff38d88]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20260709 12:27:49.374956 31516 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 229edfbe40a242629bb93e5c30b90c86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:49.381508 31515 tablet_service.cc:1511] Processing CreateTablet for tablet a8ceaa7770964d06a68441336935dd0b (DEFAULT_TABLE table=multi_table_leader_balance_table1 [id=2cd309a730d14d9aaaad3e0c9ff38d88]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20260709 12:27:49.382835 31515 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8ceaa7770964d06a68441336935dd0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:49.394251 31641 tablet_bootstrap.cc:492] T a8ceaa7770964d06a68441336935dd0b P 412e6ff66efa4d2ab996be122bfa1ed5: Bootstrap starting.
I20260709 12:27:49.401394 31641 tablet_bootstrap.cc:654] T a8ceaa7770964d06a68441336935dd0b P 412e6ff66efa4d2ab996be122bfa1ed5: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:49.411245 31642 tablet_bootstrap.cc:492] T 229edfbe40a242629bb93e5c30b90c86 P 3e6943c7aa1f4133953e746fdeee2d5c: Bootstrap starting.
I20260709 12:27:49.413149 31641 tablet_bootstrap.cc:492] T a8ceaa7770964d06a68441336935dd0b P 412e6ff66efa4d2ab996be122bfa1ed5: No bootstrap required, opened a new log
I20260709 12:27:49.413733 31641 ts_tablet_manager.cc:1403] T a8ceaa7770964d06a68441336935dd0b P 412e6ff66efa4d2ab996be122bfa1ed5: Time spent bootstrapping tablet: real 0.020s	user 0.008s	sys 0.010s
I20260709 12:27:49.416849 31641 raft_consensus.cc:359] T a8ceaa7770964d06a68441336935dd0b P 412e6ff66efa4d2ab996be122bfa1ed5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412e6ff66efa4d2ab996be122bfa1ed5" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 43129 } } peers { permanent_uuid: "3e6943c7aa1f4133953e746fdeee2d5c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46089 } } peers { permanent_uuid: "cbb731225b44446c86d71a20ed7d1a32" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34677 } }
I20260709 12:27:49.417728 31641 raft_consensus.cc:385] T a8ceaa7770964d06a68441336935dd0b P 412e6ff66efa4d2ab996be122bfa1ed5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:49.418077 31641 raft_consensus.cc:740] T a8ceaa7770964d06a68441336935dd0b P 412e6ff66efa4d2ab996be122bfa1ed5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 412e6ff66efa4d2ab996be122bfa1ed5, State: Initialized, Role: FOLLOWER
I20260709 12:27:49.417728 31642 tablet_bootstrap.cc:654] T 229edfbe40a242629bb93e5c30b90c86 P 3e6943c7aa1f4133953e746fdeee2d5c: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:49.418979 31641 consensus_queue.cc:260] T a8ceaa7770964d06a68441336935dd0b P 412e6ff66efa4d2ab996be122bfa1ed5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412e6ff66efa4d2ab996be122bfa1ed5" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 43129 } } peers { permanent_uuid: "3e6943c7aa1f4133953e746fdeee2d5c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46089 } } peers { permanent_uuid: "cbb731225b44446c86d71a20ed7d1a32" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34677 } }
I20260709 12:27:49.429136 31641 ts_tablet_manager.cc:1434] T a8ceaa7770964d06a68441336935dd0b P 412e6ff66efa4d2ab996be122bfa1ed5: Time spent starting tablet: real 0.015s	user 0.008s	sys 0.005s
I20260709 12:27:49.429569 31640 tablet_bootstrap.cc:492] T 229edfbe40a242629bb93e5c30b90c86 P cbb731225b44446c86d71a20ed7d1a32: Bootstrap starting.
I20260709 12:27:49.430140 31476 heartbeater.cc:499] Master 127.16.29.126:43837 was elected leader, sending a full tablet report...
I20260709 12:27:49.433667 31641 tablet_bootstrap.cc:492] T 229edfbe40a242629bb93e5c30b90c86 P 412e6ff66efa4d2ab996be122bfa1ed5: Bootstrap starting.
I20260709 12:27:49.438133 31640 tablet_bootstrap.cc:654] T 229edfbe40a242629bb93e5c30b90c86 P cbb731225b44446c86d71a20ed7d1a32: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:49.441334 31641 tablet_bootstrap.cc:654] T 229edfbe40a242629bb93e5c30b90c86 P 412e6ff66efa4d2ab996be122bfa1ed5: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:49.446918 31642 tablet_bootstrap.cc:492] T 229edfbe40a242629bb93e5c30b90c86 P 3e6943c7aa1f4133953e746fdeee2d5c: No bootstrap required, opened a new log
I20260709 12:27:49.447654 31642 ts_tablet_manager.cc:1403] T 229edfbe40a242629bb93e5c30b90c86 P 3e6943c7aa1f4133953e746fdeee2d5c: Time spent bootstrapping tablet: real 0.037s	user 0.019s	sys 0.012s
I20260709 12:27:49.448643 31641 tablet_bootstrap.cc:492] T 229edfbe40a242629bb93e5c30b90c86 P 412e6ff66efa4d2ab996be122bfa1ed5: No bootstrap required, opened a new log
I20260709 12:27:49.449142 31641 ts_tablet_manager.cc:1403] T 229edfbe40a242629bb93e5c30b90c86 P 412e6ff66efa4d2ab996be122bfa1ed5: Time spent bootstrapping tablet: real 0.016s	user 0.013s	sys 0.000s
I20260709 12:27:49.449780 31640 tablet_bootstrap.cc:492] T 229edfbe40a242629bb93e5c30b90c86 P cbb731225b44446c86d71a20ed7d1a32: No bootstrap required, opened a new log
I20260709 12:27:49.450289 31640 ts_tablet_manager.cc:1403] T 229edfbe40a242629bb93e5c30b90c86 P cbb731225b44446c86d71a20ed7d1a32: Time spent bootstrapping tablet: real 0.021s	user 0.016s	sys 0.000s
I20260709 12:27:49.450696 31642 raft_consensus.cc:359] T 229edfbe40a242629bb93e5c30b90c86 P 3e6943c7aa1f4133953e746fdeee2d5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbb731225b44446c86d71a20ed7d1a32" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34677 } } peers { permanent_uuid: "412e6ff66efa4d2ab996be122bfa1ed5" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 43129 } } peers { permanent_uuid: "3e6943c7aa1f4133953e746fdeee2d5c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46089 } }
I20260709 12:27:49.451546 31642 raft_consensus.cc:385] T 229edfbe40a242629bb93e5c30b90c86 P 3e6943c7aa1f4133953e746fdeee2d5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:49.451279 31641 raft_consensus.cc:359] T 229edfbe40a242629bb93e5c30b90c86 P 412e6ff66efa4d2ab996be122bfa1ed5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbb731225b44446c86d71a20ed7d1a32" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34677 } } peers { permanent_uuid: "412e6ff66efa4d2ab996be122bfa1ed5" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 43129 } } peers { permanent_uuid: "3e6943c7aa1f4133953e746fdeee2d5c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46089 } }
I20260709 12:27:49.451929 31642 raft_consensus.cc:740] T 229edfbe40a242629bb93e5c30b90c86 P 3e6943c7aa1f4133953e746fdeee2d5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e6943c7aa1f4133953e746fdeee2d5c, State: Initialized, Role: FOLLOWER
I20260709 12:27:49.452075 31641 raft_consensus.cc:385] T 229edfbe40a242629bb93e5c30b90c86 P 412e6ff66efa4d2ab996be122bfa1ed5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:49.452478 31641 raft_consensus.cc:740] T 229edfbe40a242629bb93e5c30b90c86 P 412e6ff66efa4d2ab996be122bfa1ed5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 412e6ff66efa4d2ab996be122bfa1ed5, State: Initialized, Role: FOLLOWER
I20260709 12:27:49.452836 31642 consensus_queue.cc:260] T 229edfbe40a242629bb93e5c30b90c86 P 3e6943c7aa1f4133953e746fdeee2d5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbb731225b44446c86d71a20ed7d1a32" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34677 } } peers { permanent_uuid: "412e6ff66efa4d2ab996be122bfa1ed5" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 43129 } } peers { permanent_uuid: "3e6943c7aa1f4133953e746fdeee2d5c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46089 } }
I20260709 12:27:49.453234 31641 consensus_queue.cc:260] T 229edfbe40a242629bb93e5c30b90c86 P 412e6ff66efa4d2ab996be122bfa1ed5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbb731225b44446c86d71a20ed7d1a32" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34677 } } peers { permanent_uuid: "412e6ff66efa4d2ab996be122bfa1ed5" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 43129 } } peers { permanent_uuid: "3e6943c7aa1f4133953e746fdeee2d5c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46089 } }
I20260709 12:27:49.453361 31640 raft_consensus.cc:359] T 229edfbe40a242629bb93e5c30b90c86 P cbb731225b44446c86d71a20ed7d1a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbb731225b44446c86d71a20ed7d1a32" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34677 } } peers { permanent_uuid: "412e6ff66efa4d2ab996be122bfa1ed5" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 43129 } } peers { permanent_uuid: "3e6943c7aa1f4133953e746fdeee2d5c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46089 } }
I20260709 12:27:49.454216 31640 raft_consensus.cc:385] T 229edfbe40a242629bb93e5c30b90c86 P cbb731225b44446c86d71a20ed7d1a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:49.454687 31640 raft_consensus.cc:740] T 229edfbe40a242629bb93e5c30b90c86 P cbb731225b44446c86d71a20ed7d1a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbb731225b44446c86d71a20ed7d1a32, State: Initialized, Role: FOLLOWER
I20260709 12:27:49.455330 31641 ts_tablet_manager.cc:1434] T 229edfbe40a242629bb93e5c30b90c86 P 412e6ff66efa4d2ab996be122bfa1ed5: Time spent starting tablet: real 0.006s	user 0.004s	sys 0.000s
I20260709 12:27:49.455598 31640 consensus_queue.cc:260] T 229edfbe40a242629bb93e5c30b90c86 P cbb731225b44446c86d71a20ed7d1a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbb731225b44446c86d71a20ed7d1a32" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34677 } } peers { permanent_uuid: "412e6ff66efa4d2ab996be122bfa1ed5" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 43129 } } peers { permanent_uuid: "3e6943c7aa1f4133953e746fdeee2d5c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46089 } }
I20260709 12:27:49.456960 31644 raft_consensus.cc:493] T 229edfbe40a242629bb93e5c30b90c86 P 412e6ff66efa4d2ab996be122bfa1ed5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:49.457491 31644 raft_consensus.cc:515] T 229edfbe40a242629bb93e5c30b90c86 P 412e6ff66efa4d2ab996be122bfa1ed5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbb731225b44446c86d71a20ed7d1a32" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34677 } } peers { permanent_uuid: "412e6ff66efa4d2ab996be122bfa1ed5" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 43129 } } peers { permanent_uuid: "3e6943c7aa1f4133953e746fdeee2d5c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46089 } }
I20260709 12:27:49.468118 31644 leader_election.cc:290] T 229edfbe40a242629bb93e5c30b90c86 P 412e6ff66efa4d2ab996be122bfa1ed5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cbb731225b44446c86d71a20ed7d1a32 (127.16.29.67:34677), 3e6943c7aa1f4133953e746fdeee2d5c (127.16.29.66:46089)
I20260709 12:27:49.475062 31600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "229edfbe40a242629bb93e5c30b90c86" candidate_uuid: "412e6ff66efa4d2ab996be122bfa1ed5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cbb731225b44446c86d71a20ed7d1a32" is_pre_election: true
I20260709 12:27:49.476038 31600 raft_consensus.cc:2468] T 229edfbe40a242629bb93e5c30b90c86 P cbb731225b44446c86d71a20ed7d1a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 412e6ff66efa4d2ab996be122bfa1ed5 in term 0.
I20260709 12:27:49.477851 31414 leader_election.cc:304] T 229edfbe40a242629bb93e5c30b90c86 P 412e6ff66efa4d2ab996be122bfa1ed5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 412e6ff66efa4d2ab996be122bfa1ed5, cbb731225b44446c86d71a20ed7d1a32; no voters: 
I20260709 12:27:49.478858 31644 raft_consensus.cc:2804] T 229edfbe40a242629bb93e5c30b90c86 P 412e6ff66efa4d2ab996be122bfa1ed5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:49.479272 31644 raft_consensus.cc:493] T 229edfbe40a242629bb93e5c30b90c86 P 412e6ff66efa4d2ab996be122bfa1ed5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:49.479662 31644 raft_consensus.cc:3060] T 229edfbe40a242629bb93e5c30b90c86 P 412e6ff66efa4d2ab996be122bfa1ed5 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:49.487706 31644 raft_consensus.cc:515] T 229edfbe40a242629bb93e5c30b90c86 P 412e6ff66efa4d2ab996be122bfa1ed5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbb731225b44446c86d71a20ed7d1a32" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34677 } } peers { permanent_uuid: "412e6ff66efa4d2ab996be122bfa1ed5" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 43129 } } peers { permanent_uuid: "3e6943c7aa1f4133953e746fdeee2d5c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46089 } }
I20260709 12:27:49.491400 31526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "229edfbe40a242629bb93e5c30b90c86" candidate_uuid: "412e6ff66efa4d2ab996be122bfa1ed5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e6943c7aa1f4133953e746fdeee2d5c" is_pre_election: true
I20260709 12:27:49.492247 31526 raft_consensus.cc:2468] T 229edfbe40a242629bb93e5c30b90c86 P 3e6943c7aa1f4133953e746fdeee2d5c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 412e6ff66efa4d2ab996be122bfa1ed5 in term 0.
I20260709 12:27:49.493813 31644 leader_election.cc:290] T 229edfbe40a242629bb93e5c30b90c86 P 412e6ff66efa4d2ab996be122bfa1ed5 [CANDIDATE]: Term 1 election: Requested vote from peers cbb731225b44446c86d71a20ed7d1a32 (127.16.29.67:34677), 3e6943c7aa1f4133953e746fdeee2d5c (127.16.29.66:46089)
I20260709 12:27:49.493956 31526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "229edfbe40a242629bb93e5c30b90c86" candidate_uuid: "412e6ff66efa4d2ab996be122bfa1ed5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e6943c7aa1f4133953e746fdeee2d5c"
I20260709 12:27:49.494549 31526 raft_consensus.cc:3060] T 229edfbe40a242629bb93e5c30b90c86 P 3e6943c7aa1f4133953e746fdeee2d5c [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:49.495916 31625 heartbeater.cc:499] Master 127.16.29.126:43837 was elected leader, sending a full tablet report...
I20260709 12:27:49.496387 31640 ts_tablet_manager.cc:1434] T 229edfbe40a242629bb93e5c30b90c86 P cbb731225b44446c86d71a20ed7d1a32: Time spent starting tablet: real 0.046s	user 0.013s	sys 0.034s
I20260709 12:27:49.496443 31551 heartbeater.cc:499] Master 127.16.29.126:43837 was elected leader, sending a full tablet report...
I20260709 12:27:49.496883 31642 ts_tablet_manager.cc:1434] T 229edfbe40a242629bb93e5c30b90c86 P 3e6943c7aa1f4133953e746fdeee2d5c: Time spent starting tablet: real 0.049s	user 0.013s	sys 0.030s
I20260709 12:27:49.497812 31640 tablet_bootstrap.cc:492] T a8ceaa7770964d06a68441336935dd0b P cbb731225b44446c86d71a20ed7d1a32: Bootstrap starting.
I20260709 12:27:49.498471 31642 tablet_bootstrap.cc:492] T a8ceaa7770964d06a68441336935dd0b P 3e6943c7aa1f4133953e746fdeee2d5c: Bootstrap starting.
I20260709 12:27:49.498675 31600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "229edfbe40a242629bb93e5c30b90c86" candidate_uuid: "412e6ff66efa4d2ab996be122bfa1ed5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cbb731225b44446c86d71a20ed7d1a32"
I20260709 12:27:49.499264 31600 raft_consensus.cc:3060] T 229edfbe40a242629bb93e5c30b90c86 P cbb731225b44446c86d71a20ed7d1a32 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:49.505165 31640 tablet_bootstrap.cc:654] T a8ceaa7770964d06a68441336935dd0b P cbb731225b44446c86d71a20ed7d1a32: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:49.506459 31642 tablet_bootstrap.cc:654] T a8ceaa7770964d06a68441336935dd0b P 3e6943c7aa1f4133953e746fdeee2d5c: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:49.508283 31600 raft_consensus.cc:2468] T 229edfbe40a242629bb93e5c30b90c86 P cbb731225b44446c86d71a20ed7d1a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 412e6ff66efa4d2ab996be122bfa1ed5 in term 1.
I20260709 12:27:49.511945 31414 leader_election.cc:304] T 229edfbe40a242629bb93e5c30b90c86 P 412e6ff66efa4d2ab996be122bfa1ed5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 412e6ff66efa4d2ab996be122bfa1ed5, cbb731225b44446c86d71a20ed7d1a32; no voters: 
I20260709 12:27:49.512646 31642 tablet_bootstrap.cc:492] T a8ceaa7770964d06a68441336935dd0b P 3e6943c7aa1f4133953e746fdeee2d5c: No bootstrap required, opened a new log
I20260709 12:27:49.512917 31640 tablet_bootstrap.cc:492] T a8ceaa7770964d06a68441336935dd0b P cbb731225b44446c86d71a20ed7d1a32: No bootstrap required, opened a new log
I20260709 12:27:49.512756 31644 raft_consensus.cc:2804] T 229edfbe40a242629bb93e5c30b90c86 P 412e6ff66efa4d2ab996be122bfa1ed5 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:49.513307 31642 ts_tablet_manager.cc:1403] T a8ceaa7770964d06a68441336935dd0b P 3e6943c7aa1f4133953e746fdeee2d5c: Time spent bootstrapping tablet: real 0.015s	user 0.011s	sys 0.003s
I20260709 12:27:49.516287 31644 raft_consensus.cc:697] T 229edfbe40a242629bb93e5c30b90c86 P 412e6ff66efa4d2ab996be122bfa1ed5 [term 1 LEADER]: Becoming Leader. State: Replica: 412e6ff66efa4d2ab996be122bfa1ed5, State: Running, Role: LEADER
I20260709 12:27:49.516592 31526 raft_consensus.cc:2468] T 229edfbe40a242629bb93e5c30b90c86 P 3e6943c7aa1f4133953e746fdeee2d5c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 412e6ff66efa4d2ab996be122bfa1ed5 in term 1.
I20260709 12:27:49.517160 31642 raft_consensus.cc:359] T a8ceaa7770964d06a68441336935dd0b P 3e6943c7aa1f4133953e746fdeee2d5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412e6ff66efa4d2ab996be122bfa1ed5" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 43129 } } peers { permanent_uuid: "3e6943c7aa1f4133953e746fdeee2d5c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46089 } } peers { permanent_uuid: "cbb731225b44446c86d71a20ed7d1a32" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34677 } }
I20260709 12:27:49.517268 31644 consensus_queue.cc:237] T 229edfbe40a242629bb93e5c30b90c86 P 412e6ff66efa4d2ab996be122bfa1ed5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbb731225b44446c86d71a20ed7d1a32" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34677 } } peers { permanent_uuid: "412e6ff66efa4d2ab996be122bfa1ed5" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 43129 } } peers { permanent_uuid: "3e6943c7aa1f4133953e746fdeee2d5c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46089 } }
I20260709 12:27:49.518054 31642 raft_consensus.cc:385] T a8ceaa7770964d06a68441336935dd0b P 3e6943c7aa1f4133953e746fdeee2d5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:49.518361 31642 raft_consensus.cc:740] T a8ceaa7770964d06a68441336935dd0b P 3e6943c7aa1f4133953e746fdeee2d5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e6943c7aa1f4133953e746fdeee2d5c, State: Initialized, Role: FOLLOWER
I20260709 12:27:49.519116 31642 consensus_queue.cc:260] T a8ceaa7770964d06a68441336935dd0b P 3e6943c7aa1f4133953e746fdeee2d5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412e6ff66efa4d2ab996be122bfa1ed5" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 43129 } } peers { permanent_uuid: "3e6943c7aa1f4133953e746fdeee2d5c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46089 } } peers { permanent_uuid: "cbb731225b44446c86d71a20ed7d1a32" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34677 } }
I20260709 12:27:49.521327 31642 ts_tablet_manager.cc:1434] T a8ceaa7770964d06a68441336935dd0b P 3e6943c7aa1f4133953e746fdeee2d5c: Time spent starting tablet: real 0.008s	user 0.007s	sys 0.000s
I20260709 12:27:49.521358 31640 ts_tablet_manager.cc:1403] T a8ceaa7770964d06a68441336935dd0b P cbb731225b44446c86d71a20ed7d1a32: Time spent bootstrapping tablet: real 0.024s	user 0.010s	sys 0.005s
I20260709 12:27:49.524241 31640 raft_consensus.cc:359] T a8ceaa7770964d06a68441336935dd0b P cbb731225b44446c86d71a20ed7d1a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412e6ff66efa4d2ab996be122bfa1ed5" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 43129 } } peers { permanent_uuid: "3e6943c7aa1f4133953e746fdeee2d5c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46089 } } peers { permanent_uuid: "cbb731225b44446c86d71a20ed7d1a32" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34677 } }
I20260709 12:27:49.524914 31640 raft_consensus.cc:385] T a8ceaa7770964d06a68441336935dd0b P cbb731225b44446c86d71a20ed7d1a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:49.525221 31640 raft_consensus.cc:740] T a8ceaa7770964d06a68441336935dd0b P cbb731225b44446c86d71a20ed7d1a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbb731225b44446c86d71a20ed7d1a32, State: Initialized, Role: FOLLOWER
I20260709 12:27:49.525828 31640 consensus_queue.cc:260] T a8ceaa7770964d06a68441336935dd0b P cbb731225b44446c86d71a20ed7d1a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412e6ff66efa4d2ab996be122bfa1ed5" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 43129 } } peers { permanent_uuid: "3e6943c7aa1f4133953e746fdeee2d5c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46089 } } peers { permanent_uuid: "cbb731225b44446c86d71a20ed7d1a32" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34677 } }
I20260709 12:27:49.527427 31640 ts_tablet_manager.cc:1434] T a8ceaa7770964d06a68441336935dd0b P cbb731225b44446c86d71a20ed7d1a32: Time spent starting tablet: real 0.005s	user 0.000s	sys 0.002s
I20260709 12:27:49.531872 31349 catalog_manager.cc:5697] T 229edfbe40a242629bb93e5c30b90c86 P 412e6ff66efa4d2ab996be122bfa1ed5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 412e6ff66efa4d2ab996be122bfa1ed5 (127.16.29.65). New cstate: current_term: 1 leader_uuid: "412e6ff66efa4d2ab996be122bfa1ed5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbb731225b44446c86d71a20ed7d1a32" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34677 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "412e6ff66efa4d2ab996be122bfa1ed5" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 43129 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3e6943c7aa1f4133953e746fdeee2d5c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46089 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:49.802556 31648 raft_consensus.cc:493] T a8ceaa7770964d06a68441336935dd0b P 3e6943c7aa1f4133953e746fdeee2d5c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:49.803107 31648 raft_consensus.cc:515] T a8ceaa7770964d06a68441336935dd0b P 3e6943c7aa1f4133953e746fdeee2d5c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412e6ff66efa4d2ab996be122bfa1ed5" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 43129 } } peers { permanent_uuid: "3e6943c7aa1f4133953e746fdeee2d5c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46089 } } peers { permanent_uuid: "cbb731225b44446c86d71a20ed7d1a32" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34677 } }
I20260709 12:27:49.805433 31648 leader_election.cc:290] T a8ceaa7770964d06a68441336935dd0b P 3e6943c7aa1f4133953e746fdeee2d5c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 412e6ff66efa4d2ab996be122bfa1ed5 (127.16.29.65:43129), cbb731225b44446c86d71a20ed7d1a32 (127.16.29.67:34677)
I20260709 12:27:49.815507 31451 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8ceaa7770964d06a68441336935dd0b" candidate_uuid: "3e6943c7aa1f4133953e746fdeee2d5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "412e6ff66efa4d2ab996be122bfa1ed5" is_pre_election: true
I20260709 12:27:49.815820 31600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8ceaa7770964d06a68441336935dd0b" candidate_uuid: "3e6943c7aa1f4133953e746fdeee2d5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cbb731225b44446c86d71a20ed7d1a32" is_pre_election: true
I20260709 12:27:49.816386 31451 raft_consensus.cc:2468] T a8ceaa7770964d06a68441336935dd0b P 412e6ff66efa4d2ab996be122bfa1ed5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e6943c7aa1f4133953e746fdeee2d5c in term 0.
I20260709 12:27:49.816479 31600 raft_consensus.cc:2468] T a8ceaa7770964d06a68441336935dd0b P cbb731225b44446c86d71a20ed7d1a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e6943c7aa1f4133953e746fdeee2d5c in term 0.
I20260709 12:27:49.817641 31489 leader_election.cc:304] T a8ceaa7770964d06a68441336935dd0b P 3e6943c7aa1f4133953e746fdeee2d5c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e6943c7aa1f4133953e746fdeee2d5c, cbb731225b44446c86d71a20ed7d1a32; no voters: 
I20260709 12:27:49.818454 31648 raft_consensus.cc:2804] T a8ceaa7770964d06a68441336935dd0b P 3e6943c7aa1f4133953e746fdeee2d5c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:49.818818 31648 raft_consensus.cc:493] T a8ceaa7770964d06a68441336935dd0b P 3e6943c7aa1f4133953e746fdeee2d5c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:49.819097 31648 raft_consensus.cc:3060] T a8ceaa7770964d06a68441336935dd0b P 3e6943c7aa1f4133953e746fdeee2d5c [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:49.824246 31648 raft_consensus.cc:515] T a8ceaa7770964d06a68441336935dd0b P 3e6943c7aa1f4133953e746fdeee2d5c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412e6ff66efa4d2ab996be122bfa1ed5" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 43129 } } peers { permanent_uuid: "3e6943c7aa1f4133953e746fdeee2d5c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46089 } } peers { permanent_uuid: "cbb731225b44446c86d71a20ed7d1a32" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34677 } }
I20260709 12:27:49.826292 31648 leader_election.cc:290] T a8ceaa7770964d06a68441336935dd0b P 3e6943c7aa1f4133953e746fdeee2d5c [CANDIDATE]: Term 1 election: Requested vote from peers 412e6ff66efa4d2ab996be122bfa1ed5 (127.16.29.65:43129), cbb731225b44446c86d71a20ed7d1a32 (127.16.29.67:34677)
I20260709 12:27:49.827118 31451 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8ceaa7770964d06a68441336935dd0b" candidate_uuid: "3e6943c7aa1f4133953e746fdeee2d5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "412e6ff66efa4d2ab996be122bfa1ed5"
I20260709 12:27:49.827315 31600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8ceaa7770964d06a68441336935dd0b" candidate_uuid: "3e6943c7aa1f4133953e746fdeee2d5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cbb731225b44446c86d71a20ed7d1a32"
I20260709 12:27:49.827669 31451 raft_consensus.cc:3060] T a8ceaa7770964d06a68441336935dd0b P 412e6ff66efa4d2ab996be122bfa1ed5 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:49.827956 31600 raft_consensus.cc:3060] T a8ceaa7770964d06a68441336935dd0b P cbb731225b44446c86d71a20ed7d1a32 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:49.833089 31600 raft_consensus.cc:2468] T a8ceaa7770964d06a68441336935dd0b P cbb731225b44446c86d71a20ed7d1a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e6943c7aa1f4133953e746fdeee2d5c in term 1.
I20260709 12:27:49.833129 31451 raft_consensus.cc:2468] T a8ceaa7770964d06a68441336935dd0b P 412e6ff66efa4d2ab996be122bfa1ed5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e6943c7aa1f4133953e746fdeee2d5c in term 1.
I20260709 12:27:49.834210 31489 leader_election.cc:304] T a8ceaa7770964d06a68441336935dd0b P 3e6943c7aa1f4133953e746fdeee2d5c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e6943c7aa1f4133953e746fdeee2d5c, cbb731225b44446c86d71a20ed7d1a32; no voters: 
I20260709 12:27:49.834940 31648 raft_consensus.cc:2804] T a8ceaa7770964d06a68441336935dd0b P 3e6943c7aa1f4133953e746fdeee2d5c [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:49.835870 31648 raft_consensus.cc:697] T a8ceaa7770964d06a68441336935dd0b P 3e6943c7aa1f4133953e746fdeee2d5c [term 1 LEADER]: Becoming Leader. State: Replica: 3e6943c7aa1f4133953e746fdeee2d5c, State: Running, Role: LEADER
I20260709 12:27:49.836674 31648 consensus_queue.cc:237] T a8ceaa7770964d06a68441336935dd0b P 3e6943c7aa1f4133953e746fdeee2d5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412e6ff66efa4d2ab996be122bfa1ed5" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 43129 } } peers { permanent_uuid: "3e6943c7aa1f4133953e746fdeee2d5c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46089 } } peers { permanent_uuid: "cbb731225b44446c86d71a20ed7d1a32" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34677 } }
W20260709 12:27:49.842708 31397 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:27:49.843837 31349 catalog_manager.cc:5697] T a8ceaa7770964d06a68441336935dd0b P 3e6943c7aa1f4133953e746fdeee2d5c reported cstate change: term changed from 0 to 1, leader changed from <none> to 3e6943c7aa1f4133953e746fdeee2d5c (127.16.29.66). New cstate: current_term: 1 leader_uuid: "3e6943c7aa1f4133953e746fdeee2d5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412e6ff66efa4d2ab996be122bfa1ed5" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 43129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e6943c7aa1f4133953e746fdeee2d5c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cbb731225b44446c86d71a20ed7d1a32" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34677 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:49.964948 31349 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33384:
name: "multi_table_leader_balance_table2"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
  rows: "\004\001\000\377\377\377?""\004\001\000\377\377\377?"
  indirect_data: """"
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W20260709 12:27:49.966950 31349 catalog_manager.cc:7077] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table multi_table_leader_balance_table2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260709 12:27:49.990944 31441 tablet_service.cc:1511] Processing CreateTablet for tablet 39e95550b0a04a898b1939d488003a52 (DEFAULT_TABLE table=multi_table_leader_balance_table2 [id=57b1be4fa4cd4b60bd1b961d1af3ca64]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20260709 12:27:49.992163 31515 tablet_service.cc:1511] Processing CreateTablet for tablet 39e95550b0a04a898b1939d488003a52 (DEFAULT_TABLE table=multi_table_leader_balance_table2 [id=57b1be4fa4cd4b60bd1b961d1af3ca64]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20260709 12:27:49.992552 31441 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 39e95550b0a04a898b1939d488003a52. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:49.993541 31515 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 39e95550b0a04a898b1939d488003a52. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:49.994441 31590 tablet_service.cc:1511] Processing CreateTablet for tablet 39e95550b0a04a898b1939d488003a52 (DEFAULT_TABLE table=multi_table_leader_balance_table2 [id=57b1be4fa4cd4b60bd1b961d1af3ca64]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20260709 12:27:49.995708 31590 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 39e95550b0a04a898b1939d488003a52. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:49.999056 31516 tablet_service.cc:1511] Processing CreateTablet for tablet 79ffa3b5fb8b48c285e8f4e5c192a2ef (DEFAULT_TABLE table=multi_table_leader_balance_table2 [id=57b1be4fa4cd4b60bd1b961d1af3ca64]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20260709 12:27:49.999686 31440 tablet_service.cc:1511] Processing CreateTablet for tablet 79ffa3b5fb8b48c285e8f4e5c192a2ef (DEFAULT_TABLE table=multi_table_leader_balance_table2 [id=57b1be4fa4cd4b60bd1b961d1af3ca64]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20260709 12:27:50.000947 31440 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 79ffa3b5fb8b48c285e8f4e5c192a2ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:50.001247 31589 tablet_service.cc:1511] Processing CreateTablet for tablet 79ffa3b5fb8b48c285e8f4e5c192a2ef (DEFAULT_TABLE table=multi_table_leader_balance_table2 [id=57b1be4fa4cd4b60bd1b961d1af3ca64]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20260709 12:27:50.002480 31589 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 79ffa3b5fb8b48c285e8f4e5c192a2ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:50.005148 31516 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 79ffa3b5fb8b48c285e8f4e5c192a2ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:50.012660 31642 tablet_bootstrap.cc:492] T 39e95550b0a04a898b1939d488003a52 P 3e6943c7aa1f4133953e746fdeee2d5c: Bootstrap starting.
I20260709 12:27:50.013749 31665 tablet_bootstrap.cc:492] T 39e95550b0a04a898b1939d488003a52 P 412e6ff66efa4d2ab996be122bfa1ed5: Bootstrap starting.
I20260709 12:27:50.019347 31642 tablet_bootstrap.cc:654] T 39e95550b0a04a898b1939d488003a52 P 3e6943c7aa1f4133953e746fdeee2d5c: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:50.020028 31665 tablet_bootstrap.cc:654] T 39e95550b0a04a898b1939d488003a52 P 412e6ff66efa4d2ab996be122bfa1ed5: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:50.045461 31669 tablet_bootstrap.cc:492] T 79ffa3b5fb8b48c285e8f4e5c192a2ef P cbb731225b44446c86d71a20ed7d1a32: Bootstrap starting.
I20260709 12:27:50.048146 31642 tablet_bootstrap.cc:492] T 39e95550b0a04a898b1939d488003a52 P 3e6943c7aa1f4133953e746fdeee2d5c: No bootstrap required, opened a new log
I20260709 12:27:50.048857 31642 ts_tablet_manager.cc:1403] T 39e95550b0a04a898b1939d488003a52 P 3e6943c7aa1f4133953e746fdeee2d5c: Time spent bootstrapping tablet: real 0.036s	user 0.009s	sys 0.006s
I20260709 12:27:50.049427 31665 tablet_bootstrap.cc:492] T 39e95550b0a04a898b1939d488003a52 P 412e6ff66efa4d2ab996be122bfa1ed5: No bootstrap required, opened a new log
I20260709 12:27:50.049969 31665 ts_tablet_manager.cc:1403] T 39e95550b0a04a898b1939d488003a52 P 412e6ff66efa4d2ab996be122bfa1ed5: Time spent bootstrapping tablet: real 0.036s	user 0.008s	sys 0.020s
I20260709 12:27:50.053439 31665 raft_consensus.cc:359] T 39e95550b0a04a898b1939d488003a52 P 412e6ff66efa4d2ab996be122bfa1ed5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412e6ff66efa4d2ab996be122bfa1ed5" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 43129 } } peers { permanent_uuid: "3e6943c7aa1f4133953e746fdeee2d5c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46089 } } peers { permanent_uuid: "cbb731225b44446c86d71a20ed7d1a32" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34677 } }
I20260709 12:27:50.054234 31665 raft_consensus.cc:385] T 39e95550b0a04a898b1939d488003a52 P 412e6ff66efa4d2ab996be122bfa1ed5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:50.053956 31642 raft_consensus.cc:359] T 39e95550b0a04a898b1939d488003a52 P 3e6943c7aa1f4133953e746fdeee2d5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412e6ff66efa4d2ab996be122bfa1ed5" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 43129 } } peers { permanent_uuid: "3e6943c7aa1f4133953e746fdeee2d5c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46089 } } peers { permanent_uuid: "cbb731225b44446c86d71a20ed7d1a32" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34677 } }
I20260709 12:27:50.054595 31665 raft_consensus.cc:740] T 39e95550b0a04a898b1939d488003a52 P 412e6ff66efa4d2ab996be122bfa1ed5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 412e6ff66efa4d2ab996be122bfa1ed5, State: Initialized, Role: FOLLOWER
I20260709 12:27:50.055640 31669 tablet_bootstrap.cc:654] T 79ffa3b5fb8b48c285e8f4e5c192a2ef P cbb731225b44446c86d71a20ed7d1a32: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:50.057253 31665 consensus_queue.cc:260] T 39e95550b0a04a898b1939d488003a52 P 412e6ff66efa4d2ab996be122bfa1ed5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412e6ff66efa4d2ab996be122bfa1ed5" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 43129 } } peers { permanent_uuid: "3e6943c7aa1f4133953e746fdeee2d5c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46089 } } peers { permanent_uuid: "cbb731225b44446c86d71a20ed7d1a32" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34677 } }
I20260709 12:27:50.058049 31668 consensus_queue.cc:1048] T 229edfbe40a242629bb93e5c30b90c86 P 412e6ff66efa4d2ab996be122bfa1ed5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e6943c7aa1f4133953e746fdeee2d5c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:27:50.059836 31668 consensus_queue.cc:1048] T 229edfbe40a242629bb93e5c30b90c86 P 412e6ff66efa4d2ab996be122bfa1ed5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbb731225b44446c86d71a20ed7d1a32" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:50.054955 31642 raft_consensus.cc:385] T 39e95550b0a04a898b1939d488003a52 P 3e6943c7aa1f4133953e746fdeee2d5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:50.067503 31642 raft_consensus.cc:740] T 39e95550b0a04a898b1939d488003a52 P 3e6943c7aa1f4133953e746fdeee2d5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e6943c7aa1f4133953e746fdeee2d5c, State: Initialized, Role: FOLLOWER
I20260709 12:27:50.068701 31642 consensus_queue.cc:260] T 39e95550b0a04a898b1939d488003a52 P 3e6943c7aa1f4133953e746fdeee2d5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412e6ff66efa4d2ab996be122bfa1ed5" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 43129 } } peers { permanent_uuid: "3e6943c7aa1f4133953e746fdeee2d5c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46089 } } peers { permanent_uuid: "cbb731225b44446c86d71a20ed7d1a32" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34677 } }
I20260709 12:27:50.095068 31665 ts_tablet_manager.cc:1434] T 39e95550b0a04a898b1939d488003a52 P 412e6ff66efa4d2ab996be122bfa1ed5: Time spent starting tablet: real 0.045s	user 0.016s	sys 0.001s
I20260709 12:27:50.096204 31642 ts_tablet_manager.cc:1434] T 39e95550b0a04a898b1939d488003a52 P 3e6943c7aa1f4133953e746fdeee2d5c: Time spent starting tablet: real 0.047s	user 0.009s	sys 0.002s
I20260709 12:27:50.097852 31642 tablet_bootstrap.cc:492] T 79ffa3b5fb8b48c285e8f4e5c192a2ef P 3e6943c7aa1f4133953e746fdeee2d5c: Bootstrap starting.
I20260709 12:27:50.111027 31665 tablet_bootstrap.cc:492] T 79ffa3b5fb8b48c285e8f4e5c192a2ef P 412e6ff66efa4d2ab996be122bfa1ed5: Bootstrap starting.
I20260709 12:27:50.122882 31669 tablet_bootstrap.cc:492] T 79ffa3b5fb8b48c285e8f4e5c192a2ef P cbb731225b44446c86d71a20ed7d1a32: No bootstrap required, opened a new log
I20260709 12:27:50.123478 31669 ts_tablet_manager.cc:1403] T 79ffa3b5fb8b48c285e8f4e5c192a2ef P cbb731225b44446c86d71a20ed7d1a32: Time spent bootstrapping tablet: real 0.079s	user 0.032s	sys 0.041s
I20260709 12:27:50.126865 31669 raft_consensus.cc:359] T 79ffa3b5fb8b48c285e8f4e5c192a2ef P cbb731225b44446c86d71a20ed7d1a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412e6ff66efa4d2ab996be122bfa1ed5" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 43129 } } peers { permanent_uuid: "3e6943c7aa1f4133953e746fdeee2d5c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46089 } } peers { permanent_uuid: "cbb731225b44446c86d71a20ed7d1a32" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34677 } }
I20260709 12:27:50.127671 31669 raft_consensus.cc:385] T 79ffa3b5fb8b48c285e8f4e5c192a2ef P cbb731225b44446c86d71a20ed7d1a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:50.127993 31669 raft_consensus.cc:740] T 79ffa3b5fb8b48c285e8f4e5c192a2ef P cbb731225b44446c86d71a20ed7d1a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbb731225b44446c86d71a20ed7d1a32, State: Initialized, Role: FOLLOWER
I20260709 12:27:50.128803 31669 consensus_queue.cc:260] T 79ffa3b5fb8b48c285e8f4e5c192a2ef P cbb731225b44446c86d71a20ed7d1a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412e6ff66efa4d2ab996be122bfa1ed5" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 43129 } } peers { permanent_uuid: "3e6943c7aa1f4133953e746fdeee2d5c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46089 } } peers { permanent_uuid: "cbb731225b44446c86d71a20ed7d1a32" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34677 } }
I20260709 12:27:50.130820 31665 tablet_bootstrap.cc:654] T 79ffa3b5fb8b48c285e8f4e5c192a2ef P 412e6ff66efa4d2ab996be122bfa1ed5: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:50.131101 31669 ts_tablet_manager.cc:1434] T 79ffa3b5fb8b48c285e8f4e5c192a2ef P cbb731225b44446c86d71a20ed7d1a32: Time spent starting tablet: real 0.007s	user 0.004s	sys 0.000s
I20260709 12:27:50.135043 31642 tablet_bootstrap.cc:654] T 79ffa3b5fb8b48c285e8f4e5c192a2ef P 3e6943c7aa1f4133953e746fdeee2d5c: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:50.140033 31665 tablet_bootstrap.cc:492] T 79ffa3b5fb8b48c285e8f4e5c192a2ef P 412e6ff66efa4d2ab996be122bfa1ed5: No bootstrap required, opened a new log
I20260709 12:27:50.140605 31665 ts_tablet_manager.cc:1403] T 79ffa3b5fb8b48c285e8f4e5c192a2ef P 412e6ff66efa4d2ab996be122bfa1ed5: Time spent bootstrapping tablet: real 0.030s	user 0.006s	sys 0.012s
I20260709 12:27:50.142093 31669 tablet_bootstrap.cc:492] T 39e95550b0a04a898b1939d488003a52 P cbb731225b44446c86d71a20ed7d1a32: Bootstrap starting.
I20260709 12:27:50.144131 31642 tablet_bootstrap.cc:492] T 79ffa3b5fb8b48c285e8f4e5c192a2ef P 3e6943c7aa1f4133953e746fdeee2d5c: No bootstrap required, opened a new log
I20260709 12:27:50.144670 31642 ts_tablet_manager.cc:1403] T 79ffa3b5fb8b48c285e8f4e5c192a2ef P 3e6943c7aa1f4133953e746fdeee2d5c: Time spent bootstrapping tablet: real 0.047s	user 0.007s	sys 0.008s
I20260709 12:27:50.147706 31642 raft_consensus.cc:359] T 79ffa3b5fb8b48c285e8f4e5c192a2ef P 3e6943c7aa1f4133953e746fdeee2d5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412e6ff66efa4d2ab996be122bfa1ed5" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 43129 } } peers { permanent_uuid: "3e6943c7aa1f4133953e746fdeee2d5c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46089 } } peers { permanent_uuid: "cbb731225b44446c86d71a20ed7d1a32" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34677 } }
I20260709 12:27:50.148442 31642 raft_consensus.cc:385] T 79ffa3b5fb8b48c285e8f4e5c192a2ef P 3e6943c7aa1f4133953e746fdeee2d5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:50.148730 31642 raft_consensus.cc:740] T 79ffa3b5fb8b48c285e8f4e5c192a2ef P 3e6943c7aa1f4133953e746fdeee2d5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e6943c7aa1f4133953e746fdeee2d5c, State: Initialized, Role: FOLLOWER
I20260709 12:27:50.149600 31642 consensus_queue.cc:260] T 79ffa3b5fb8b48c285e8f4e5c192a2ef P 3e6943c7aa1f4133953e746fdeee2d5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412e6ff66efa4d2ab996be122bfa1ed5" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 43129 } } peers { permanent_uuid: "3e6943c7aa1f4133953e746fdeee2d5c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46089 } } peers { permanent_uuid: "cbb731225b44446c86d71a20ed7d1a32" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34677 } }
I20260709 12:27:50.152594 31669 tablet_bootstrap.cc:654] T 39e95550b0a04a898b1939d488003a52 P cbb731225b44446c86d71a20ed7d1a32: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:50.153321 31671 raft_consensus.cc:493] T 39e95550b0a04a898b1939d488003a52 P 412e6ff66efa4d2ab996be122bfa1ed5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:50.156997 31671 raft_consensus.cc:515] T 39e95550b0a04a898b1939d488003a52 P 412e6ff66efa4d2ab996be122bfa1ed5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412e6ff66efa4d2ab996be122bfa1ed5" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 43129 } } peers { permanent_uuid: "3e6943c7aa1f4133953e746fdeee2d5c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46089 } } peers { permanent_uuid: "cbb731225b44446c86d71a20ed7d1a32" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34677 } }
I20260709 12:27:50.159054 31671 leader_election.cc:290] T 39e95550b0a04a898b1939d488003a52 P 412e6ff66efa4d2ab996be122bfa1ed5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3e6943c7aa1f4133953e746fdeee2d5c (127.16.29.66:46089), cbb731225b44446c86d71a20ed7d1a32 (127.16.29.67:34677)
I20260709 12:27:50.160261 31526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39e95550b0a04a898b1939d488003a52" candidate_uuid: "412e6ff66efa4d2ab996be122bfa1ed5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e6943c7aa1f4133953e746fdeee2d5c" is_pre_election: true
I20260709 12:27:50.160954 31526 raft_consensus.cc:2468] T 39e95550b0a04a898b1939d488003a52 P 3e6943c7aa1f4133953e746fdeee2d5c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 412e6ff66efa4d2ab996be122bfa1ed5 in term 0.
I20260709 12:27:50.162540 31599 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39e95550b0a04a898b1939d488003a52" candidate_uuid: "412e6ff66efa4d2ab996be122bfa1ed5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cbb731225b44446c86d71a20ed7d1a32" is_pre_election: true
I20260709 12:27:50.156002 31665 raft_consensus.cc:359] T 79ffa3b5fb8b48c285e8f4e5c192a2ef P 412e6ff66efa4d2ab996be122bfa1ed5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412e6ff66efa4d2ab996be122bfa1ed5" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 43129 } } peers { permanent_uuid: "3e6943c7aa1f4133953e746fdeee2d5c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46089 } } peers { permanent_uuid: "cbb731225b44446c86d71a20ed7d1a32" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34677 } }
I20260709 12:27:50.165143 31416 leader_election.cc:304] T 39e95550b0a04a898b1939d488003a52 P 412e6ff66efa4d2ab996be122bfa1ed5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e6943c7aa1f4133953e746fdeee2d5c, 412e6ff66efa4d2ab996be122bfa1ed5; no voters: 
I20260709 12:27:50.165465 31665 raft_consensus.cc:385] T 79ffa3b5fb8b48c285e8f4e5c192a2ef P 412e6ff66efa4d2ab996be122bfa1ed5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:50.165779 31665 raft_consensus.cc:740] T 79ffa3b5fb8b48c285e8f4e5c192a2ef P 412e6ff66efa4d2ab996be122bfa1ed5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 412e6ff66efa4d2ab996be122bfa1ed5, State: Initialized, Role: FOLLOWER
W20260709 12:27:50.166321 31414 leader_election.cc:343] T 39e95550b0a04a898b1939d488003a52 P 412e6ff66efa4d2ab996be122bfa1ed5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cbb731225b44446c86d71a20ed7d1a32 (127.16.29.67:34677): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:50.166580 31665 consensus_queue.cc:260] T 79ffa3b5fb8b48c285e8f4e5c192a2ef P 412e6ff66efa4d2ab996be122bfa1ed5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412e6ff66efa4d2ab996be122bfa1ed5" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 43129 } } peers { permanent_uuid: "3e6943c7aa1f4133953e746fdeee2d5c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46089 } } peers { permanent_uuid: "cbb731225b44446c86d71a20ed7d1a32" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34677 } }
I20260709 12:27:50.167816 31668 raft_consensus.cc:2804] T 39e95550b0a04a898b1939d488003a52 P 412e6ff66efa4d2ab996be122bfa1ed5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:50.168372 31668 raft_consensus.cc:493] T 39e95550b0a04a898b1939d488003a52 P 412e6ff66efa4d2ab996be122bfa1ed5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:50.168805 31668 raft_consensus.cc:3060] T 39e95550b0a04a898b1939d488003a52 P 412e6ff66efa4d2ab996be122bfa1ed5 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:50.169171 31665 ts_tablet_manager.cc:1434] T 79ffa3b5fb8b48c285e8f4e5c192a2ef P 412e6ff66efa4d2ab996be122bfa1ed5: Time spent starting tablet: real 0.027s	user 0.005s	sys 0.002s
I20260709 12:27:50.154664 31642 ts_tablet_manager.cc:1434] T 79ffa3b5fb8b48c285e8f4e5c192a2ef P 3e6943c7aa1f4133953e746fdeee2d5c: Time spent starting tablet: real 0.010s	user 0.005s	sys 0.001s
I20260709 12:27:50.178748 31669 tablet_bootstrap.cc:492] T 39e95550b0a04a898b1939d488003a52 P cbb731225b44446c86d71a20ed7d1a32: No bootstrap required, opened a new log
I20260709 12:27:50.179409 31669 ts_tablet_manager.cc:1403] T 39e95550b0a04a898b1939d488003a52 P cbb731225b44446c86d71a20ed7d1a32: Time spent bootstrapping tablet: real 0.038s	user 0.010s	sys 0.014s
I20260709 12:27:50.179633 31668 raft_consensus.cc:515] T 39e95550b0a04a898b1939d488003a52 P 412e6ff66efa4d2ab996be122bfa1ed5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412e6ff66efa4d2ab996be122bfa1ed5" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 43129 } } peers { permanent_uuid: "3e6943c7aa1f4133953e746fdeee2d5c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46089 } } peers { permanent_uuid: "cbb731225b44446c86d71a20ed7d1a32" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34677 } }
I20260709 12:27:50.183017 31526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39e95550b0a04a898b1939d488003a52" candidate_uuid: "412e6ff66efa4d2ab996be122bfa1ed5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e6943c7aa1f4133953e746fdeee2d5c"
I20260709 12:27:50.183658 31526 raft_consensus.cc:3060] T 39e95550b0a04a898b1939d488003a52 P 3e6943c7aa1f4133953e746fdeee2d5c [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:50.183768 31669 raft_consensus.cc:359] T 39e95550b0a04a898b1939d488003a52 P cbb731225b44446c86d71a20ed7d1a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412e6ff66efa4d2ab996be122bfa1ed5" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 43129 } } peers { permanent_uuid: "3e6943c7aa1f4133953e746fdeee2d5c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46089 } } peers { permanent_uuid: "cbb731225b44446c86d71a20ed7d1a32" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34677 } }
I20260709 12:27:50.184760 31669 raft_consensus.cc:385] T 39e95550b0a04a898b1939d488003a52 P cbb731225b44446c86d71a20ed7d1a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:50.185111 31669 raft_consensus.cc:740] T 39e95550b0a04a898b1939d488003a52 P cbb731225b44446c86d71a20ed7d1a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbb731225b44446c86d71a20ed7d1a32, State: Initialized, Role: FOLLOWER
I20260709 12:27:50.185917 31669 consensus_queue.cc:260] T 39e95550b0a04a898b1939d488003a52 P cbb731225b44446c86d71a20ed7d1a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412e6ff66efa4d2ab996be122bfa1ed5" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 43129 } } peers { permanent_uuid: "3e6943c7aa1f4133953e746fdeee2d5c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46089 } } peers { permanent_uuid: "cbb731225b44446c86d71a20ed7d1a32" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34677 } }
I20260709 12:27:50.188503 31669 ts_tablet_manager.cc:1434] T 39e95550b0a04a898b1939d488003a52 P cbb731225b44446c86d71a20ed7d1a32: Time spent starting tablet: real 0.009s	user 0.007s	sys 0.000s
I20260709 12:27:50.186625 31668 leader_election.cc:290] T 39e95550b0a04a898b1939d488003a52 P 412e6ff66efa4d2ab996be122bfa1ed5 [CANDIDATE]: Term 1 election: Requested vote from peers 3e6943c7aa1f4133953e746fdeee2d5c (127.16.29.66:46089), cbb731225b44446c86d71a20ed7d1a32 (127.16.29.67:34677)
I20260709 12:27:50.189378 31600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39e95550b0a04a898b1939d488003a52" candidate_uuid: "412e6ff66efa4d2ab996be122bfa1ed5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cbb731225b44446c86d71a20ed7d1a32"
I20260709 12:27:50.190056 31600 raft_consensus.cc:3060] T 39e95550b0a04a898b1939d488003a52 P cbb731225b44446c86d71a20ed7d1a32 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:50.191392 31526 raft_consensus.cc:2468] T 39e95550b0a04a898b1939d488003a52 P 3e6943c7aa1f4133953e746fdeee2d5c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 412e6ff66efa4d2ab996be122bfa1ed5 in term 1.
I20260709 12:27:50.192619 31416 leader_election.cc:304] T 39e95550b0a04a898b1939d488003a52 P 412e6ff66efa4d2ab996be122bfa1ed5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e6943c7aa1f4133953e746fdeee2d5c, 412e6ff66efa4d2ab996be122bfa1ed5; no voters: 
I20260709 12:27:50.193487 31668 raft_consensus.cc:2804] T 39e95550b0a04a898b1939d488003a52 P 412e6ff66efa4d2ab996be122bfa1ed5 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:50.193935 31668 raft_consensus.cc:697] T 39e95550b0a04a898b1939d488003a52 P 412e6ff66efa4d2ab996be122bfa1ed5 [term 1 LEADER]: Becoming Leader. State: Replica: 412e6ff66efa4d2ab996be122bfa1ed5, State: Running, Role: LEADER
I20260709 12:27:50.194729 31668 consensus_queue.cc:237] T 39e95550b0a04a898b1939d488003a52 P 412e6ff66efa4d2ab996be122bfa1ed5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412e6ff66efa4d2ab996be122bfa1ed5" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 43129 } } peers { permanent_uuid: "3e6943c7aa1f4133953e746fdeee2d5c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46089 } } peers { permanent_uuid: "cbb731225b44446c86d71a20ed7d1a32" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34677 } }
I20260709 12:27:50.196229 31600 raft_consensus.cc:2468] T 39e95550b0a04a898b1939d488003a52 P cbb731225b44446c86d71a20ed7d1a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 412e6ff66efa4d2ab996be122bfa1ed5 in term 1.
I20260709 12:27:50.203855 31349 catalog_manager.cc:5697] T 39e95550b0a04a898b1939d488003a52 P 412e6ff66efa4d2ab996be122bfa1ed5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 412e6ff66efa4d2ab996be122bfa1ed5 (127.16.29.65). New cstate: current_term: 1 leader_uuid: "412e6ff66efa4d2ab996be122bfa1ed5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412e6ff66efa4d2ab996be122bfa1ed5" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 43129 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3e6943c7aa1f4133953e746fdeee2d5c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cbb731225b44446c86d71a20ed7d1a32" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34677 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:50.205646 31648 raft_consensus.cc:493] T 79ffa3b5fb8b48c285e8f4e5c192a2ef P 3e6943c7aa1f4133953e746fdeee2d5c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:50.206215 31648 raft_consensus.cc:515] T 79ffa3b5fb8b48c285e8f4e5c192a2ef P 3e6943c7aa1f4133953e746fdeee2d5c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412e6ff66efa4d2ab996be122bfa1ed5" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 43129 } } peers { permanent_uuid: "3e6943c7aa1f4133953e746fdeee2d5c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46089 } } peers { permanent_uuid: "cbb731225b44446c86d71a20ed7d1a32" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34677 } }
I20260709 12:27:50.208129 31648 leader_election.cc:290] T 79ffa3b5fb8b48c285e8f4e5c192a2ef P 3e6943c7aa1f4133953e746fdeee2d5c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 412e6ff66efa4d2ab996be122bfa1ed5 (127.16.29.65:43129), cbb731225b44446c86d71a20ed7d1a32 (127.16.29.67:34677)
I20260709 12:27:50.208976 31451 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ffa3b5fb8b48c285e8f4e5c192a2ef" candidate_uuid: "3e6943c7aa1f4133953e746fdeee2d5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "412e6ff66efa4d2ab996be122bfa1ed5" is_pre_election: true
I20260709 12:27:50.209406 31600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ffa3b5fb8b48c285e8f4e5c192a2ef" candidate_uuid: "3e6943c7aa1f4133953e746fdeee2d5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cbb731225b44446c86d71a20ed7d1a32" is_pre_election: true
I20260709 12:27:50.209642 31451 raft_consensus.cc:2468] T 79ffa3b5fb8b48c285e8f4e5c192a2ef P 412e6ff66efa4d2ab996be122bfa1ed5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e6943c7aa1f4133953e746fdeee2d5c in term 0.
I20260709 12:27:50.210103 31600 raft_consensus.cc:2468] T 79ffa3b5fb8b48c285e8f4e5c192a2ef P cbb731225b44446c86d71a20ed7d1a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e6943c7aa1f4133953e746fdeee2d5c in term 0.
I20260709 12:27:50.211017 31489 leader_election.cc:304] T 79ffa3b5fb8b48c285e8f4e5c192a2ef P 3e6943c7aa1f4133953e746fdeee2d5c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e6943c7aa1f4133953e746fdeee2d5c, 412e6ff66efa4d2ab996be122bfa1ed5; no voters: 
I20260709 12:27:50.212322 31648 raft_consensus.cc:2804] T 79ffa3b5fb8b48c285e8f4e5c192a2ef P 3e6943c7aa1f4133953e746fdeee2d5c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:50.212673 31648 raft_consensus.cc:493] T 79ffa3b5fb8b48c285e8f4e5c192a2ef P 3e6943c7aa1f4133953e746fdeee2d5c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:50.212998 31648 raft_consensus.cc:3060] T 79ffa3b5fb8b48c285e8f4e5c192a2ef P 3e6943c7aa1f4133953e746fdeee2d5c [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:50.218488 31648 raft_consensus.cc:515] T 79ffa3b5fb8b48c285e8f4e5c192a2ef P 3e6943c7aa1f4133953e746fdeee2d5c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412e6ff66efa4d2ab996be122bfa1ed5" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 43129 } } peers { permanent_uuid: "3e6943c7aa1f4133953e746fdeee2d5c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46089 } } peers { permanent_uuid: "cbb731225b44446c86d71a20ed7d1a32" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34677 } }
I20260709 12:27:50.220109 31648 leader_election.cc:290] T 79ffa3b5fb8b48c285e8f4e5c192a2ef P 3e6943c7aa1f4133953e746fdeee2d5c [CANDIDATE]: Term 1 election: Requested vote from peers 412e6ff66efa4d2ab996be122bfa1ed5 (127.16.29.65:43129), cbb731225b44446c86d71a20ed7d1a32 (127.16.29.67:34677)
I20260709 12:27:50.221136 31451 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ffa3b5fb8b48c285e8f4e5c192a2ef" candidate_uuid: "3e6943c7aa1f4133953e746fdeee2d5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "412e6ff66efa4d2ab996be122bfa1ed5"
I20260709 12:27:50.221393 31600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ffa3b5fb8b48c285e8f4e5c192a2ef" candidate_uuid: "3e6943c7aa1f4133953e746fdeee2d5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cbb731225b44446c86d71a20ed7d1a32"
I20260709 12:27:50.221760 31451 raft_consensus.cc:3060] T 79ffa3b5fb8b48c285e8f4e5c192a2ef P 412e6ff66efa4d2ab996be122bfa1ed5 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:50.222079 31600 raft_consensus.cc:3060] T 79ffa3b5fb8b48c285e8f4e5c192a2ef P cbb731225b44446c86d71a20ed7d1a32 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:50.227385 31600 raft_consensus.cc:2468] T 79ffa3b5fb8b48c285e8f4e5c192a2ef P cbb731225b44446c86d71a20ed7d1a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e6943c7aa1f4133953e746fdeee2d5c in term 1.
I20260709 12:27:50.227397 31451 raft_consensus.cc:2468] T 79ffa3b5fb8b48c285e8f4e5c192a2ef P 412e6ff66efa4d2ab996be122bfa1ed5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e6943c7aa1f4133953e746fdeee2d5c in term 1.
I20260709 12:27:50.228588 31489 leader_election.cc:304] T 79ffa3b5fb8b48c285e8f4e5c192a2ef P 3e6943c7aa1f4133953e746fdeee2d5c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e6943c7aa1f4133953e746fdeee2d5c, cbb731225b44446c86d71a20ed7d1a32; no voters: 
I20260709 12:27:50.229316 31648 raft_consensus.cc:2804] T 79ffa3b5fb8b48c285e8f4e5c192a2ef P 3e6943c7aa1f4133953e746fdeee2d5c [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:50.229809 31648 raft_consensus.cc:697] T 79ffa3b5fb8b48c285e8f4e5c192a2ef P 3e6943c7aa1f4133953e746fdeee2d5c [term 1 LEADER]: Becoming Leader. State: Replica: 3e6943c7aa1f4133953e746fdeee2d5c, State: Running, Role: LEADER
I20260709 12:27:50.230670 31648 consensus_queue.cc:237] T 79ffa3b5fb8b48c285e8f4e5c192a2ef P 3e6943c7aa1f4133953e746fdeee2d5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412e6ff66efa4d2ab996be122bfa1ed5" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 43129 } } peers { permanent_uuid: "3e6943c7aa1f4133953e746fdeee2d5c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46089 } } peers { permanent_uuid: "cbb731225b44446c86d71a20ed7d1a32" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34677 } }
I20260709 12:27:50.239240 31347 catalog_manager.cc:5697] T 79ffa3b5fb8b48c285e8f4e5c192a2ef P 3e6943c7aa1f4133953e746fdeee2d5c reported cstate change: term changed from 0 to 1, leader changed from <none> to 3e6943c7aa1f4133953e746fdeee2d5c (127.16.29.66). New cstate: current_term: 1 leader_uuid: "3e6943c7aa1f4133953e746fdeee2d5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412e6ff66efa4d2ab996be122bfa1ed5" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 43129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e6943c7aa1f4133953e746fdeee2d5c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cbb731225b44446c86d71a20ed7d1a32" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34677 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:50.310592 31526 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "79ffa3b5fb8b48c285e8f4e5c192a2ef"
dest_uuid: "3e6943c7aa1f4133953e746fdeee2d5c"
mode: GRACEFUL
new_leader_uuid: "412e6ff66efa4d2ab996be122bfa1ed5"
 from {username='slave'} at 127.0.0.1:44104
I20260709 12:27:50.311188 31526 raft_consensus.cc:606] T 79ffa3b5fb8b48c285e8f4e5c192a2ef P 3e6943c7aa1f4133953e746fdeee2d5c [term 1 LEADER]: Received request to transfer leadership to TS 412e6ff66efa4d2ab996be122bfa1ed5
I20260709 12:27:50.370527 31648 consensus_queue.cc:1048] T a8ceaa7770964d06a68441336935dd0b P 3e6943c7aa1f4133953e746fdeee2d5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbb731225b44446c86d71a20ed7d1a32" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:27:50.383569 31648 consensus_queue.cc:1048] T a8ceaa7770964d06a68441336935dd0b P 3e6943c7aa1f4133953e746fdeee2d5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "412e6ff66efa4d2ab996be122bfa1ed5" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 43129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:50.688585 31671 consensus_queue.cc:1048] T 39e95550b0a04a898b1939d488003a52 P 412e6ff66efa4d2ab996be122bfa1ed5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e6943c7aa1f4133953e746fdeee2d5c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:50.696890 31687 consensus_queue.cc:1048] T 79ffa3b5fb8b48c285e8f4e5c192a2ef P 3e6943c7aa1f4133953e746fdeee2d5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbb731225b44446c86d71a20ed7d1a32" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:50.710182 31668 consensus_queue.cc:1048] T 39e95550b0a04a898b1939d488003a52 P 412e6ff66efa4d2ab996be122bfa1ed5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbb731225b44446c86d71a20ed7d1a32" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260709 12:27:50.731194 31648 consensus_queue.cc:1048] T 79ffa3b5fb8b48c285e8f4e5c192a2ef P 3e6943c7aa1f4133953e746fdeee2d5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "412e6ff66efa4d2ab996be122bfa1ed5" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 43129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:50.745750 31648 raft_consensus.cc:993] T 79ffa3b5fb8b48c285e8f4e5c192a2ef P 3e6943c7aa1f4133953e746fdeee2d5c: : Instructing follower 412e6ff66efa4d2ab996be122bfa1ed5 to start an election
I20260709 12:27:50.746490 31657 raft_consensus.cc:1081] T 79ffa3b5fb8b48c285e8f4e5c192a2ef P 3e6943c7aa1f4133953e746fdeee2d5c [term 1 LEADER]: Signalling peer 412e6ff66efa4d2ab996be122bfa1ed5 to start an election
I20260709 12:27:50.749197 31451 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "79ffa3b5fb8b48c285e8f4e5c192a2ef"
dest_uuid: "412e6ff66efa4d2ab996be122bfa1ed5"
 from {username='slave'} at 127.0.0.1:42494
I20260709 12:27:50.750090 31451 raft_consensus.cc:493] T 79ffa3b5fb8b48c285e8f4e5c192a2ef P 412e6ff66efa4d2ab996be122bfa1ed5 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:27:50.750519 31451 raft_consensus.cc:3060] T 79ffa3b5fb8b48c285e8f4e5c192a2ef P 412e6ff66efa4d2ab996be122bfa1ed5 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:50.759545 31451 raft_consensus.cc:515] T 79ffa3b5fb8b48c285e8f4e5c192a2ef P 412e6ff66efa4d2ab996be122bfa1ed5 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412e6ff66efa4d2ab996be122bfa1ed5" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 43129 } } peers { permanent_uuid: "3e6943c7aa1f4133953e746fdeee2d5c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46089 } } peers { permanent_uuid: "cbb731225b44446c86d71a20ed7d1a32" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34677 } }
I20260709 12:27:50.761843 31451 leader_election.cc:290] T 79ffa3b5fb8b48c285e8f4e5c192a2ef P 412e6ff66efa4d2ab996be122bfa1ed5 [CANDIDATE]: Term 2 election: Requested vote from peers 3e6943c7aa1f4133953e746fdeee2d5c (127.16.29.66:46089), cbb731225b44446c86d71a20ed7d1a32 (127.16.29.67:34677)
I20260709 12:27:50.762991 31525 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ffa3b5fb8b48c285e8f4e5c192a2ef" candidate_uuid: "412e6ff66efa4d2ab996be122bfa1ed5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "3e6943c7aa1f4133953e746fdeee2d5c"
I20260709 12:27:50.763233 31600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ffa3b5fb8b48c285e8f4e5c192a2ef" candidate_uuid: "412e6ff66efa4d2ab996be122bfa1ed5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "cbb731225b44446c86d71a20ed7d1a32"
I20260709 12:27:50.763918 31525 raft_consensus.cc:3055] T 79ffa3b5fb8b48c285e8f4e5c192a2ef P 3e6943c7aa1f4133953e746fdeee2d5c [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:27:50.763962 31600 raft_consensus.cc:3060] T 79ffa3b5fb8b48c285e8f4e5c192a2ef P cbb731225b44446c86d71a20ed7d1a32 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:50.764456 31525 raft_consensus.cc:740] T 79ffa3b5fb8b48c285e8f4e5c192a2ef P 3e6943c7aa1f4133953e746fdeee2d5c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3e6943c7aa1f4133953e746fdeee2d5c, State: Running, Role: LEADER
I20260709 12:27:50.765570 31525 consensus_queue.cc:260] T 79ffa3b5fb8b48c285e8f4e5c192a2ef P 3e6943c7aa1f4133953e746fdeee2d5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412e6ff66efa4d2ab996be122bfa1ed5" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 43129 } } peers { permanent_uuid: "3e6943c7aa1f4133953e746fdeee2d5c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46089 } } peers { permanent_uuid: "cbb731225b44446c86d71a20ed7d1a32" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34677 } }
I20260709 12:27:50.766835 31525 raft_consensus.cc:3060] T 79ffa3b5fb8b48c285e8f4e5c192a2ef P 3e6943c7aa1f4133953e746fdeee2d5c [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:50.773481 31600 raft_consensus.cc:2468] T 79ffa3b5fb8b48c285e8f4e5c192a2ef P cbb731225b44446c86d71a20ed7d1a32 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 412e6ff66efa4d2ab996be122bfa1ed5 in term 2.
I20260709 12:27:50.775620 31414 leader_election.cc:304] T 79ffa3b5fb8b48c285e8f4e5c192a2ef P 412e6ff66efa4d2ab996be122bfa1ed5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 412e6ff66efa4d2ab996be122bfa1ed5, cbb731225b44446c86d71a20ed7d1a32; no voters: 
I20260709 12:27:50.776310 31525 raft_consensus.cc:2468] T 79ffa3b5fb8b48c285e8f4e5c192a2ef P 3e6943c7aa1f4133953e746fdeee2d5c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 412e6ff66efa4d2ab996be122bfa1ed5 in term 2.
I20260709 12:27:50.776927 31694 raft_consensus.cc:2804] T 79ffa3b5fb8b48c285e8f4e5c192a2ef P 412e6ff66efa4d2ab996be122bfa1ed5 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:27:50.777508 31694 raft_consensus.cc:697] T 79ffa3b5fb8b48c285e8f4e5c192a2ef P 412e6ff66efa4d2ab996be122bfa1ed5 [term 2 LEADER]: Becoming Leader. State: Replica: 412e6ff66efa4d2ab996be122bfa1ed5, State: Running, Role: LEADER
I20260709 12:27:50.778594 31694 consensus_queue.cc:237] T 79ffa3b5fb8b48c285e8f4e5c192a2ef P 412e6ff66efa4d2ab996be122bfa1ed5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412e6ff66efa4d2ab996be122bfa1ed5" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 43129 } } peers { permanent_uuid: "3e6943c7aa1f4133953e746fdeee2d5c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46089 } } peers { permanent_uuid: "cbb731225b44446c86d71a20ed7d1a32" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34677 } }
I20260709 12:27:50.785735 31349 catalog_manager.cc:5697] T 79ffa3b5fb8b48c285e8f4e5c192a2ef P 412e6ff66efa4d2ab996be122bfa1ed5 reported cstate change: term changed from 1 to 2, leader changed from 3e6943c7aa1f4133953e746fdeee2d5c (127.16.29.66) to 412e6ff66efa4d2ab996be122bfa1ed5 (127.16.29.65). New cstate: current_term: 2 leader_uuid: "412e6ff66efa4d2ab996be122bfa1ed5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412e6ff66efa4d2ab996be122bfa1ed5" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 43129 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3e6943c7aa1f4133953e746fdeee2d5c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cbb731225b44446c86d71a20ed7d1a32" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34677 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:50.818264 16501 auto_leader_rebalancer.cc:141] leader rebalance for table multi_table_leader_balance_table1
I20260709 12:27:50.820043 16501 auto_leader_rebalancer.cc:440] table: multi_table_leader_balance_table1, leader rebalance finish, leader transfer count: 0
I20260709 12:27:50.820497 16501 auto_leader_rebalancer.cc:141] leader rebalance for table multi_table_leader_balance_table2
I20260709 12:27:50.835691 31451 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "79ffa3b5fb8b48c285e8f4e5c192a2ef"
dest_uuid: "412e6ff66efa4d2ab996be122bfa1ed5"
mode: GRACEFUL
new_leader_uuid: "cbb731225b44446c86d71a20ed7d1a32"
 from {username='slave'} at 127.0.0.1:35134
I20260709 12:27:50.836288 31451 raft_consensus.cc:606] T 79ffa3b5fb8b48c285e8f4e5c192a2ef P 412e6ff66efa4d2ab996be122bfa1ed5 [term 2 LEADER]: Received request to transfer leadership to TS cbb731225b44446c86d71a20ed7d1a32
I20260709 12:27:50.837531 16501 auto_leader_rebalancer.cc:440] table: multi_table_leader_balance_table2, leader rebalance finish, leader transfer count: 1
I20260709 12:27:51.172658 31600 raft_consensus.cc:1275] T 79ffa3b5fb8b48c285e8f4e5c192a2ef P cbb731225b44446c86d71a20ed7d1a32 [term 2 FOLLOWER]: Refusing update from remote peer 412e6ff66efa4d2ab996be122bfa1ed5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:27:51.174511 31694 consensus_queue.cc:1048] T 79ffa3b5fb8b48c285e8f4e5c192a2ef P 412e6ff66efa4d2ab996be122bfa1ed5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbb731225b44446c86d71a20ed7d1a32" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:27:51.180394 31668 raft_consensus.cc:993] T 79ffa3b5fb8b48c285e8f4e5c192a2ef P 412e6ff66efa4d2ab996be122bfa1ed5: : Instructing follower cbb731225b44446c86d71a20ed7d1a32 to start an election
I20260709 12:27:51.180979 31668 raft_consensus.cc:1081] T 79ffa3b5fb8b48c285e8f4e5c192a2ef P 412e6ff66efa4d2ab996be122bfa1ed5 [term 2 LEADER]: Signalling peer cbb731225b44446c86d71a20ed7d1a32 to start an election
I20260709 12:27:51.182938 31599 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "79ffa3b5fb8b48c285e8f4e5c192a2ef"
dest_uuid: "cbb731225b44446c86d71a20ed7d1a32"
 from {username='slave'} at 127.0.0.1:41060
I20260709 12:27:51.184613 31525 raft_consensus.cc:1275] T 79ffa3b5fb8b48c285e8f4e5c192a2ef P 3e6943c7aa1f4133953e746fdeee2d5c [term 2 FOLLOWER]: Refusing update from remote peer 412e6ff66efa4d2ab996be122bfa1ed5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:27:51.184732 31599 raft_consensus.cc:493] T 79ffa3b5fb8b48c285e8f4e5c192a2ef P cbb731225b44446c86d71a20ed7d1a32 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:27:51.185880 31599 raft_consensus.cc:3060] T 79ffa3b5fb8b48c285e8f4e5c192a2ef P cbb731225b44446c86d71a20ed7d1a32 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:27:51.186909 31694 consensus_queue.cc:1048] T 79ffa3b5fb8b48c285e8f4e5c192a2ef P 412e6ff66efa4d2ab996be122bfa1ed5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e6943c7aa1f4133953e746fdeee2d5c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:27:51.196803 31599 raft_consensus.cc:515] T 79ffa3b5fb8b48c285e8f4e5c192a2ef P cbb731225b44446c86d71a20ed7d1a32 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412e6ff66efa4d2ab996be122bfa1ed5" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 43129 } } peers { permanent_uuid: "3e6943c7aa1f4133953e746fdeee2d5c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46089 } } peers { permanent_uuid: "cbb731225b44446c86d71a20ed7d1a32" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34677 } }
I20260709 12:27:51.202791 31599 leader_election.cc:290] T 79ffa3b5fb8b48c285e8f4e5c192a2ef P cbb731225b44446c86d71a20ed7d1a32 [CANDIDATE]: Term 3 election: Requested vote from peers 412e6ff66efa4d2ab996be122bfa1ed5 (127.16.29.65:43129), 3e6943c7aa1f4133953e746fdeee2d5c (127.16.29.66:46089)
I20260709 12:27:51.219417 31451 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ffa3b5fb8b48c285e8f4e5c192a2ef" candidate_uuid: "cbb731225b44446c86d71a20ed7d1a32" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "412e6ff66efa4d2ab996be122bfa1ed5"
I20260709 12:27:51.220136 31451 raft_consensus.cc:3055] T 79ffa3b5fb8b48c285e8f4e5c192a2ef P 412e6ff66efa4d2ab996be122bfa1ed5 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:27:51.220407 31451 raft_consensus.cc:740] T 79ffa3b5fb8b48c285e8f4e5c192a2ef P 412e6ff66efa4d2ab996be122bfa1ed5 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 412e6ff66efa4d2ab996be122bfa1ed5, State: Running, Role: LEADER
I20260709 12:27:51.221100 31451 consensus_queue.cc:260] T 79ffa3b5fb8b48c285e8f4e5c192a2ef P 412e6ff66efa4d2ab996be122bfa1ed5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412e6ff66efa4d2ab996be122bfa1ed5" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 43129 } } peers { permanent_uuid: "3e6943c7aa1f4133953e746fdeee2d5c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46089 } } peers { permanent_uuid: "cbb731225b44446c86d71a20ed7d1a32" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34677 } }
I20260709 12:27:51.222211 31451 raft_consensus.cc:3060] T 79ffa3b5fb8b48c285e8f4e5c192a2ef P 412e6ff66efa4d2ab996be122bfa1ed5 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:27:51.225243 31525 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ffa3b5fb8b48c285e8f4e5c192a2ef" candidate_uuid: "cbb731225b44446c86d71a20ed7d1a32" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "3e6943c7aa1f4133953e746fdeee2d5c"
I20260709 12:27:51.225937 31525 raft_consensus.cc:3060] T 79ffa3b5fb8b48c285e8f4e5c192a2ef P 3e6943c7aa1f4133953e746fdeee2d5c [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:27:51.230381 31451 raft_consensus.cc:2468] T 79ffa3b5fb8b48c285e8f4e5c192a2ef P 412e6ff66efa4d2ab996be122bfa1ed5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbb731225b44446c86d71a20ed7d1a32 in term 3.
I20260709 12:27:51.231827 31563 leader_election.cc:304] T 79ffa3b5fb8b48c285e8f4e5c192a2ef P cbb731225b44446c86d71a20ed7d1a32 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 412e6ff66efa4d2ab996be122bfa1ed5, cbb731225b44446c86d71a20ed7d1a32; no voters: 
I20260709 12:27:51.232841 31675 raft_consensus.cc:2804] T 79ffa3b5fb8b48c285e8f4e5c192a2ef P cbb731225b44446c86d71a20ed7d1a32 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:27:51.233676 31525 raft_consensus.cc:2468] T 79ffa3b5fb8b48c285e8f4e5c192a2ef P 3e6943c7aa1f4133953e746fdeee2d5c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbb731225b44446c86d71a20ed7d1a32 in term 3.
I20260709 12:27:51.234069 31675 raft_consensus.cc:697] T 79ffa3b5fb8b48c285e8f4e5c192a2ef P cbb731225b44446c86d71a20ed7d1a32 [term 3 LEADER]: Becoming Leader. State: Replica: cbb731225b44446c86d71a20ed7d1a32, State: Running, Role: LEADER
I20260709 12:27:51.235235 31675 consensus_queue.cc:237] T 79ffa3b5fb8b48c285e8f4e5c192a2ef P cbb731225b44446c86d71a20ed7d1a32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412e6ff66efa4d2ab996be122bfa1ed5" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 43129 } } peers { permanent_uuid: "3e6943c7aa1f4133953e746fdeee2d5c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46089 } } peers { permanent_uuid: "cbb731225b44446c86d71a20ed7d1a32" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34677 } }
I20260709 12:27:51.241575 31349 catalog_manager.cc:5697] T 79ffa3b5fb8b48c285e8f4e5c192a2ef P cbb731225b44446c86d71a20ed7d1a32 reported cstate change: term changed from 2 to 3, leader changed from 412e6ff66efa4d2ab996be122bfa1ed5 (127.16.29.65) to cbb731225b44446c86d71a20ed7d1a32 (127.16.29.67). New cstate: current_term: 3 leader_uuid: "cbb731225b44446c86d71a20ed7d1a32" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412e6ff66efa4d2ab996be122bfa1ed5" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 43129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e6943c7aa1f4133953e746fdeee2d5c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 46089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cbb731225b44446c86d71a20ed7d1a32" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34677 } health_report { overall_health: HEALTHY } } }
I20260709 12:27:51.468681 16501 tablet_server.cc:179] TabletServer@127.16.29.65:0 shutting down...
I20260709 12:27:51.493420 16501 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:27:51.494300 16501 tablet_replica.cc:333] T 79ffa3b5fb8b48c285e8f4e5c192a2ef P 412e6ff66efa4d2ab996be122bfa1ed5: stopping tablet replica
I20260709 12:27:51.495051 16501 raft_consensus.cc:2243] T 79ffa3b5fb8b48c285e8f4e5c192a2ef P 412e6ff66efa4d2ab996be122bfa1ed5 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:51.495563 16501 raft_consensus.cc:2272] T 79ffa3b5fb8b48c285e8f4e5c192a2ef P 412e6ff66efa4d2ab996be122bfa1ed5 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:51.498389 16501 tablet_replica.cc:333] T 39e95550b0a04a898b1939d488003a52 P 412e6ff66efa4d2ab996be122bfa1ed5: stopping tablet replica
I20260709 12:27:51.498979 16501 raft_consensus.cc:2243] T 39e95550b0a04a898b1939d488003a52 P 412e6ff66efa4d2ab996be122bfa1ed5 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:27:51.499783 16501 raft_consensus.cc:2272] T 39e95550b0a04a898b1939d488003a52 P 412e6ff66efa4d2ab996be122bfa1ed5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:51.503450 16501 tablet_replica.cc:333] T 229edfbe40a242629bb93e5c30b90c86 P 412e6ff66efa4d2ab996be122bfa1ed5: stopping tablet replica
I20260709 12:27:51.504182 16501 raft_consensus.cc:2243] T 229edfbe40a242629bb93e5c30b90c86 P 412e6ff66efa4d2ab996be122bfa1ed5 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:27:51.505162 16501 raft_consensus.cc:2272] T 229edfbe40a242629bb93e5c30b90c86 P 412e6ff66efa4d2ab996be122bfa1ed5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:51.507685 16501 tablet_replica.cc:333] T a8ceaa7770964d06a68441336935dd0b P 412e6ff66efa4d2ab996be122bfa1ed5: stopping tablet replica
I20260709 12:27:51.508235 16501 raft_consensus.cc:2243] T a8ceaa7770964d06a68441336935dd0b P 412e6ff66efa4d2ab996be122bfa1ed5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:51.508715 16501 raft_consensus.cc:2272] T a8ceaa7770964d06a68441336935dd0b P 412e6ff66efa4d2ab996be122bfa1ed5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:51.536952 16501 tablet_server.cc:196] TabletServer@127.16.29.65:0 shutdown complete.
I20260709 12:27:51.558647 16501 tablet_server.cc:179] TabletServer@127.16.29.66:0 shutting down...
I20260709 12:27:51.588892 16501 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:27:51.589742 16501 tablet_replica.cc:333] T 79ffa3b5fb8b48c285e8f4e5c192a2ef P 3e6943c7aa1f4133953e746fdeee2d5c: stopping tablet replica
I20260709 12:27:51.590478 16501 raft_consensus.cc:2243] T 79ffa3b5fb8b48c285e8f4e5c192a2ef P 3e6943c7aa1f4133953e746fdeee2d5c [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:51.591020 16501 raft_consensus.cc:2272] T 79ffa3b5fb8b48c285e8f4e5c192a2ef P 3e6943c7aa1f4133953e746fdeee2d5c [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:51.594492 16501 tablet_replica.cc:333] T 39e95550b0a04a898b1939d488003a52 P 3e6943c7aa1f4133953e746fdeee2d5c: stopping tablet replica
I20260709 12:27:51.595227 16501 raft_consensus.cc:2243] T 39e95550b0a04a898b1939d488003a52 P 3e6943c7aa1f4133953e746fdeee2d5c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:51.595672 16501 raft_consensus.cc:2272] T 39e95550b0a04a898b1939d488003a52 P 3e6943c7aa1f4133953e746fdeee2d5c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:51.599203 16501 tablet_replica.cc:333] T a8ceaa7770964d06a68441336935dd0b P 3e6943c7aa1f4133953e746fdeee2d5c: stopping tablet replica
I20260709 12:27:51.599839 16501 raft_consensus.cc:2243] T a8ceaa7770964d06a68441336935dd0b P 3e6943c7aa1f4133953e746fdeee2d5c [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:27:51.601169 16501 raft_consensus.cc:2272] T a8ceaa7770964d06a68441336935dd0b P 3e6943c7aa1f4133953e746fdeee2d5c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:51.603946 16501 tablet_replica.cc:333] T 229edfbe40a242629bb93e5c30b90c86 P 3e6943c7aa1f4133953e746fdeee2d5c: stopping tablet replica
I20260709 12:27:51.604514 16501 raft_consensus.cc:2243] T 229edfbe40a242629bb93e5c30b90c86 P 3e6943c7aa1f4133953e746fdeee2d5c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:51.605026 16501 raft_consensus.cc:2272] T 229edfbe40a242629bb93e5c30b90c86 P 3e6943c7aa1f4133953e746fdeee2d5c [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 12:27:51.619524 31565 consensus_peers.cc:597] T 79ffa3b5fb8b48c285e8f4e5c192a2ef P cbb731225b44446c86d71a20ed7d1a32 -> Peer 3e6943c7aa1f4133953e746fdeee2d5c (127.16.29.66:46089): Couldn't send request to peer 3e6943c7aa1f4133953e746fdeee2d5c. Status: Network error: recv got EOF from unknown peer (error 108). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:51.634341 16501 tablet_server.cc:196] TabletServer@127.16.29.66:0 shutdown complete.
I20260709 12:27:51.655726 16501 tablet_server.cc:179] TabletServer@127.16.29.67:0 shutting down...
I20260709 12:27:51.685793 16501 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:27:51.686682 16501 tablet_replica.cc:333] T 39e95550b0a04a898b1939d488003a52 P cbb731225b44446c86d71a20ed7d1a32: stopping tablet replica
I20260709 12:27:51.687630 16501 raft_consensus.cc:2243] T 39e95550b0a04a898b1939d488003a52 P cbb731225b44446c86d71a20ed7d1a32 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:51.688246 16501 raft_consensus.cc:2272] T 39e95550b0a04a898b1939d488003a52 P cbb731225b44446c86d71a20ed7d1a32 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:51.691620 16501 tablet_replica.cc:333] T 79ffa3b5fb8b48c285e8f4e5c192a2ef P cbb731225b44446c86d71a20ed7d1a32: stopping tablet replica
I20260709 12:27:51.692253 16501 raft_consensus.cc:2243] T 79ffa3b5fb8b48c285e8f4e5c192a2ef P cbb731225b44446c86d71a20ed7d1a32 [term 3 LEADER]: Raft consensus shutting down.
I20260709 12:27:51.692986 16501 pending_rounds.cc:70] T 79ffa3b5fb8b48c285e8f4e5c192a2ef P cbb731225b44446c86d71a20ed7d1a32: Trying to abort 1 pending ops.
I20260709 12:27:51.693214 16501 pending_rounds.cc:77] T 79ffa3b5fb8b48c285e8f4e5c192a2ef P cbb731225b44446c86d71a20ed7d1a32: Aborting op as it isn't in flight: id { term: 3 index: 3 } timestamp: 7305625891787313152 op_type: NO_OP noop_request { }
I20260709 12:27:51.693555 16501 raft_consensus.cc:2889] T 79ffa3b5fb8b48c285e8f4e5c192a2ef P cbb731225b44446c86d71a20ed7d1a32 [term 3 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260709 12:27:51.693835 16501 raft_consensus.cc:2272] T 79ffa3b5fb8b48c285e8f4e5c192a2ef P cbb731225b44446c86d71a20ed7d1a32 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:51.697301 16501 tablet_replica.cc:333] T a8ceaa7770964d06a68441336935dd0b P cbb731225b44446c86d71a20ed7d1a32: stopping tablet replica
I20260709 12:27:51.697965 16501 raft_consensus.cc:2243] T a8ceaa7770964d06a68441336935dd0b P cbb731225b44446c86d71a20ed7d1a32 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:51.698547 16501 raft_consensus.cc:2272] T a8ceaa7770964d06a68441336935dd0b P cbb731225b44446c86d71a20ed7d1a32 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:51.701527 16501 tablet_replica.cc:333] T 229edfbe40a242629bb93e5c30b90c86 P cbb731225b44446c86d71a20ed7d1a32: stopping tablet replica
I20260709 12:27:51.702116 16501 raft_consensus.cc:2243] T 229edfbe40a242629bb93e5c30b90c86 P cbb731225b44446c86d71a20ed7d1a32 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:51.702669 16501 raft_consensus.cc:2272] T 229edfbe40a242629bb93e5c30b90c86 P cbb731225b44446c86d71a20ed7d1a32 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:51.730559 16501 tablet_server.cc:196] TabletServer@127.16.29.67:0 shutdown complete.
I20260709 12:27:51.750046 16501 master.cc:562] Master@127.16.29.126:43837 shutting down...
I20260709 12:27:51.772364 16501 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 48143a6fed1a4cceb7a1ecc2b35f83de [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:27:51.772994 16501 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 48143a6fed1a4cceb7a1ecc2b35f83de [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:51.773350 16501 tablet_replica.cc:333] T 00000000000000000000000000000000 P 48143a6fed1a4cceb7a1ecc2b35f83de: stopping tablet replica
I20260709 12:27:51.794874 16501 master.cc:584] Master@127.16.29.126:43837 shutdown complete.
[       OK ] LeaderRebalancerTest.MultiTableLeaderBalance (3154 ms)
[----------] 6 tests from LeaderRebalancerTest (519720 ms total)

[----------] 2 tests from FilterSoftDeletedTableTest
[ RUN      ] FilterSoftDeletedTableTest.TestFilterSofteDeletedTable/0
I20260709 12:27:51.837631 16501 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.16.29.126:37739
I20260709 12:27:51.839143 16501 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:27:51.846483 31709 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:27:51.846590 31708 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:27:51.847729 31711 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:27:51.849280 16501 server_base.cc:1061] running on GCE node
I20260709 12:27:51.850298 16501 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:27:51.850490 16501 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:27:51.850821 16501 hybrid_clock.cc:648] HybridClock initialized: now 1783600071850655 us; error 0 us; skew 500 ppm
I20260709 12:27:51.851641 16501 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:27:51.854620 16501 webserver.cc:533] Webserver started at http://127.16.29.126:42277/ using document root <none> and password file <none>
I20260709 12:27:51.855190 16501 fs_manager.cc:362] Metadata directory not provided
I20260709 12:27:51.855398 16501 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:27:51.855746 16501 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:27:51.857085 16501 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783599551912677-16501-0/minicluster-data/master-0-root/instance:
uuid: "c569449549d1476cb2d243fd3c5e4804"
format_stamp: "Formatted at 2026-07-09 12:27:51 on dist-test-slave-5n9v"
I20260709 12:27:51.862427 16501 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.005s	sys 0.000s
I20260709 12:27:51.866308 31716 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:27:51.867260 16501 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.000s
I20260709 12:27:51.867612 16501 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783599551912677-16501-0/minicluster-data/master-0-root
uuid: "c569449549d1476cb2d243fd3c5e4804"
format_stamp: "Formatted at 2026-07-09 12:27:51 on dist-test-slave-5n9v"
I20260709 12:27:51.867949 16501 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783599551912677-16501-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783599551912677-16501-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783599551912677-16501-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:27:51.880975 16501 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:27:51.882339 16501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:27:51.931397 16501 rpc_server.cc:307] RPC server started. Bound to: 127.16.29.126:37739
I20260709 12:27:51.931502 31767 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.29.126:37739 every 8 connection(s)
I20260709 12:27:51.935545 31768 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:51.947570 31768 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c569449549d1476cb2d243fd3c5e4804: Bootstrap starting.
I20260709 12:27:51.952472 31768 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c569449549d1476cb2d243fd3c5e4804: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:51.957288 31768 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c569449549d1476cb2d243fd3c5e4804: No bootstrap required, opened a new log
I20260709 12:27:51.959663 31768 raft_consensus.cc:359] T 00000000000000000000000000000000 P c569449549d1476cb2d243fd3c5e4804 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c569449549d1476cb2d243fd3c5e4804" member_type: VOTER }
I20260709 12:27:51.960124 31768 raft_consensus.cc:385] T 00000000000000000000000000000000 P c569449549d1476cb2d243fd3c5e4804 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:51.960316 31768 raft_consensus.cc:740] T 00000000000000000000000000000000 P c569449549d1476cb2d243fd3c5e4804 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c569449549d1476cb2d243fd3c5e4804, State: Initialized, Role: FOLLOWER
I20260709 12:27:51.960924 31768 consensus_queue.cc:260] T 00000000000000000000000000000000 P c569449549d1476cb2d243fd3c5e4804 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c569449549d1476cb2d243fd3c5e4804" member_type: VOTER }
I20260709 12:27:51.961349 31768 raft_consensus.cc:399] T 00000000000000000000000000000000 P c569449549d1476cb2d243fd3c5e4804 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260709 12:27:51.961539 31768 raft_consensus.cc:493] T 00000000000000000000000000000000 P c569449549d1476cb2d243fd3c5e4804 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260709 12:27:51.961745 31768 raft_consensus.cc:3060] T 00000000000000000000000000000000 P c569449549d1476cb2d243fd3c5e4804 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:51.966589 31768 raft_consensus.cc:515] T 00000000000000000000000000000000 P c569449549d1476cb2d243fd3c5e4804 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c569449549d1476cb2d243fd3c5e4804" member_type: VOTER }
I20260709 12:27:51.967080 31768 leader_election.cc:304] T 00000000000000000000000000000000 P c569449549d1476cb2d243fd3c5e4804 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c569449549d1476cb2d243fd3c5e4804; no voters: 
I20260709 12:27:51.968262 31768 leader_election.cc:290] T 00000000000000000000000000000000 P c569449549d1476cb2d243fd3c5e4804 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20260709 12:27:51.968705 31771 raft_consensus.cc:2804] T 00000000000000000000000000000000 P c569449549d1476cb2d243fd3c5e4804 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:51.970216 31771 raft_consensus.cc:697] T 00000000000000000000000000000000 P c569449549d1476cb2d243fd3c5e4804 [term 1 LEADER]: Becoming Leader. State: Replica: c569449549d1476cb2d243fd3c5e4804, State: Running, Role: LEADER
I20260709 12:27:51.970829 31771 consensus_queue.cc:237] T 00000000000000000000000000000000 P c569449549d1476cb2d243fd3c5e4804 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c569449549d1476cb2d243fd3c5e4804" member_type: VOTER }
I20260709 12:27:51.971400 31768 sys_catalog.cc:565] T 00000000000000000000000000000000 P c569449549d1476cb2d243fd3c5e4804 [sys.catalog]: configured and running, proceeding with master startup.
I20260709 12:27:51.973891 31772 sys_catalog.cc:455] T 00000000000000000000000000000000 P c569449549d1476cb2d243fd3c5e4804 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c569449549d1476cb2d243fd3c5e4804" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c569449549d1476cb2d243fd3c5e4804" member_type: VOTER } }
I20260709 12:27:51.973997 31773 sys_catalog.cc:455] T 00000000000000000000000000000000 P c569449549d1476cb2d243fd3c5e4804 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c569449549d1476cb2d243fd3c5e4804. Latest consensus state: current_term: 1 leader_uuid: "c569449549d1476cb2d243fd3c5e4804" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c569449549d1476cb2d243fd3c5e4804" member_type: VOTER } }
I20260709 12:27:51.974665 31773 sys_catalog.cc:458] T 00000000000000000000000000000000 P c569449549d1476cb2d243fd3c5e4804 [sys.catalog]: This master's current role is: LEADER
I20260709 12:27:51.974630 31772 sys_catalog.cc:458] T 00000000000000000000000000000000 P c569449549d1476cb2d243fd3c5e4804 [sys.catalog]: This master's current role is: LEADER
I20260709 12:27:51.978089 31777 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260709 12:27:51.983134 31777 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260709 12:27:51.987417 16501 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260709 12:27:51.993256 31777 catalog_manager.cc:1383] Generated new cluster ID: 2bcadb571c6443c984208c4a4857256d
I20260709 12:27:51.993631 31777 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260709 12:27:52.011688 31777 catalog_manager.cc:1406] Generated new certificate authority record
I20260709 12:27:52.013300 31777 catalog_manager.cc:1540] Loading token signing keys...
I20260709 12:27:52.036823 31777 catalog_manager.cc:6070] T 00000000000000000000000000000000 P c569449549d1476cb2d243fd3c5e4804: Generated new TSK 0
I20260709 12:27:52.037559 31777 catalog_manager.cc:1550] Initializing in-progress tserver states...
I20260709 12:27:52.054834 16501 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:27:52.062971 31789 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:27:52.064124 31790 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:27:52.067140 16501 server_base.cc:1061] running on GCE node
W20260709 12:27:52.067209 31792 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:27:52.069000 16501 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:27:52.069293 16501 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:27:52.069500 16501 hybrid_clock.cc:648] HybridClock initialized: now 1783600072069481 us; error 0 us; skew 500 ppm
I20260709 12:27:52.070418 16501 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:27:52.073661 16501 webserver.cc:533] Webserver started at http://127.16.29.65:35365/ using document root <none> and password file <none>
I20260709 12:27:52.074204 16501 fs_manager.cc:362] Metadata directory not provided
I20260709 12:27:52.074419 16501 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:27:52.074715 16501 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:27:52.076421 16501 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783599551912677-16501-0/minicluster-data/ts-0-root/instance:
uuid: "2ae3a9e2293442e291b1262a717a5e50"
format_stamp: "Formatted at 2026-07-09 12:27:52 on dist-test-slave-5n9v"
I20260709 12:27:52.081777 16501 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.005s	sys 0.000s
I20260709 12:27:52.086529 31797 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:27:52.087468 16501 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.000s
I20260709 12:27:52.088008 16501 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783599551912677-16501-0/minicluster-data/ts-0-root
uuid: "2ae3a9e2293442e291b1262a717a5e50"
format_stamp: "Formatted at 2026-07-09 12:27:52 on dist-test-slave-5n9v"
I20260709 12:27:52.088347 16501 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783599551912677-16501-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783599551912677-16501-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783599551912677-16501-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:27:52.100162 16501 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:27:52.101540 16501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:27:52.103371 16501 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:27:52.106562 16501 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:27:52.106832 16501 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:27:52.107108 16501 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:27:52.107297 16501 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:27:52.152968 16501 rpc_server.cc:307] RPC server started. Bound to: 127.16.29.65:44289
I20260709 12:27:52.153086 31859 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.29.65:44289 every 8 connection(s)
I20260709 12:27:52.158504 16501 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:27:52.166605 31864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:27:52.168941 31865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:27:52.171559 31867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:27:52.172928 16501 server_base.cc:1061] running on GCE node
I20260709 12:27:52.173972 16501 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20260709 12:27:52.174180 31860 heartbeater.cc:344] Connected to a master server at 127.16.29.126:37739
W20260709 12:27:52.174288 16501 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:27:52.174623 16501 hybrid_clock.cc:648] HybridClock initialized: now 1783600072174603 us; error 0 us; skew 500 ppm
I20260709 12:27:52.174681 31860 heartbeater.cc:461] Registering TS with master...
I20260709 12:27:52.175427 16501 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:27:52.175655 31860 heartbeater.cc:507] Master 127.16.29.126:37739 requested a full tablet report, sending...
I20260709 12:27:52.178649 16501 webserver.cc:533] Webserver started at http://127.16.29.66:42125/ using document root <none> and password file <none>
I20260709 12:27:52.179129 31733 ts_manager.cc:194] Registered new tserver with Master: 2ae3a9e2293442e291b1262a717a5e50 (127.16.29.65:44289)
I20260709 12:27:52.179315 16501 fs_manager.cc:362] Metadata directory not provided
I20260709 12:27:52.179666 16501 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:27:52.179993 16501 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:27:52.181175 16501 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783599551912677-16501-0/minicluster-data/ts-1-root/instance:
uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039"
format_stamp: "Formatted at 2026-07-09 12:27:52 on dist-test-slave-5n9v"
I20260709 12:27:52.181221 31733 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:43830
I20260709 12:27:52.186262 16501 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.005s	sys 0.001s
I20260709 12:27:52.190196 31872 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:27:52.191139 16501 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.000s	sys 0.002s
I20260709 12:27:52.191402 16501 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783599551912677-16501-0/minicluster-data/ts-1-root
uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039"
format_stamp: "Formatted at 2026-07-09 12:27:52 on dist-test-slave-5n9v"
I20260709 12:27:52.191663 16501 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783599551912677-16501-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783599551912677-16501-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783599551912677-16501-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:27:52.243604 16501 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:27:52.244874 16501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:27:52.246320 16501 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:27:52.248756 16501 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:27:52.248955 16501 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:27:52.249260 16501 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:27:52.249404 16501 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:27:52.294473 16501 rpc_server.cc:307] RPC server started. Bound to: 127.16.29.66:42163
I20260709 12:27:52.294581 31934 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.29.66:42163 every 8 connection(s)
I20260709 12:27:52.300037 16501 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:27:52.308387 31938 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:27:52.309048 31939 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:27:52.312170 31941 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:27:52.312708 16501 server_base.cc:1061] running on GCE node
I20260709 12:27:52.313838 16501 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:27:52.314150 16501 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:27:52.314266 31935 heartbeater.cc:344] Connected to a master server at 127.16.29.126:37739
I20260709 12:27:52.314373 16501 hybrid_clock.cc:648] HybridClock initialized: now 1783600072314338 us; error 0 us; skew 500 ppm
I20260709 12:27:52.314795 31935 heartbeater.cc:461] Registering TS with master...
I20260709 12:27:52.315209 16501 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:27:52.315615 31935 heartbeater.cc:507] Master 127.16.29.126:37739 requested a full tablet report, sending...
I20260709 12:27:52.318084 31733 ts_manager.cc:194] Registered new tserver with Master: 64fd1d8a3cd14e66ac94c4fbcceaf039 (127.16.29.66:42163)
I20260709 12:27:52.318176 16501 webserver.cc:533] Webserver started at http://127.16.29.67:34017/ using document root <none> and password file <none>
I20260709 12:27:52.318850 16501 fs_manager.cc:362] Metadata directory not provided
I20260709 12:27:52.319101 16501 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:27:52.319411 16501 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:27:52.319995 31733 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:43834
I20260709 12:27:52.321017 16501 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783599551912677-16501-0/minicluster-data/ts-2-root/instance:
uuid: "07b97783285d4b22b6fd0f1d816ef930"
format_stamp: "Formatted at 2026-07-09 12:27:52 on dist-test-slave-5n9v"
I20260709 12:27:52.325990 16501 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.005s	sys 0.000s
I20260709 12:27:52.329380 31946 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:27:52.330305 16501 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.001s	sys 0.000s
I20260709 12:27:52.330605 16501 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783599551912677-16501-0/minicluster-data/ts-2-root
uuid: "07b97783285d4b22b6fd0f1d816ef930"
format_stamp: "Formatted at 2026-07-09 12:27:52 on dist-test-slave-5n9v"
I20260709 12:27:52.330855 16501 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783599551912677-16501-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783599551912677-16501-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783599551912677-16501-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:27:52.340728 16501 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:27:52.341948 16501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:27:52.343401 16501 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:27:52.345816 16501 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:27:52.346040 16501 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:27:52.346306 16501 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:27:52.346454 16501 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:27:52.392975 16501 rpc_server.cc:307] RPC server started. Bound to: 127.16.29.67:34419
I20260709 12:27:52.393092 32008 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.29.67:34419 every 8 connection(s)
I20260709 12:27:52.410351 32009 heartbeater.cc:344] Connected to a master server at 127.16.29.126:37739
I20260709 12:27:52.410816 32009 heartbeater.cc:461] Registering TS with master...
I20260709 12:27:52.411847 32009 heartbeater.cc:507] Master 127.16.29.126:37739 requested a full tablet report, sending...
I20260709 12:27:52.414707 31733 ts_manager.cc:194] Registered new tserver with Master: 07b97783285d4b22b6fd0f1d816ef930 (127.16.29.67:34419)
I20260709 12:27:52.414947 16501 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.018032806s
I20260709 12:27:52.416467 31733 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:43842
I20260709 12:27:52.443253 31733 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43856:
name: "test-workload"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
  rows: "\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q""\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q"
  indirect_data: """"
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W20260709 12:27:52.446116 31733 catalog_manager.cc:7077] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260709 12:27:52.514880 31974 tablet_service.cc:1511] Processing CreateTablet for tablet 89784f886fb545d2871174c1e169fb32 (DEFAULT_TABLE table=test-workload [id=108e8cf6859142debe8f012d0a9b704d]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260709 12:27:52.516400 31974 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 89784f886fb545d2871174c1e169fb32. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:52.514881 31973 tablet_service.cc:1511] Processing CreateTablet for tablet fad0b7ef0f804aa494f44e9ec456e11e (DEFAULT_TABLE table=test-workload [id=108e8cf6859142debe8f012d0a9b704d]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260709 12:27:52.522433 31972 tablet_service.cc:1511] Processing CreateTablet for tablet d00d4a3dbb1c435cb565cf6e5d723bf2 (DEFAULT_TABLE table=test-workload [id=108e8cf6859142debe8f012d0a9b704d]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260709 12:27:52.523226 31899 tablet_service.cc:1511] Processing CreateTablet for tablet fad0b7ef0f804aa494f44e9ec456e11e (DEFAULT_TABLE table=test-workload [id=108e8cf6859142debe8f012d0a9b704d]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260709 12:27:52.524020 31972 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d00d4a3dbb1c435cb565cf6e5d723bf2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:52.524658 31899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fad0b7ef0f804aa494f44e9ec456e11e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:52.525076 31898 tablet_service.cc:1511] Processing CreateTablet for tablet d00d4a3dbb1c435cb565cf6e5d723bf2 (DEFAULT_TABLE table=test-workload [id=108e8cf6859142debe8f012d0a9b704d]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260709 12:27:52.526446 31898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d00d4a3dbb1c435cb565cf6e5d723bf2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:52.528152 31973 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fad0b7ef0f804aa494f44e9ec456e11e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:52.529070 31900 tablet_service.cc:1511] Processing CreateTablet for tablet 89784f886fb545d2871174c1e169fb32 (DEFAULT_TABLE table=test-workload [id=108e8cf6859142debe8f012d0a9b704d]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260709 12:27:52.530380 31900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 89784f886fb545d2871174c1e169fb32. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:52.534811 31971 tablet_service.cc:1511] Processing CreateTablet for tablet 1638e215a9ee4068a999e92023fff415 (DEFAULT_TABLE table=test-workload [id=108e8cf6859142debe8f012d0a9b704d]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260709 12:27:52.536242 31971 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1638e215a9ee4068a999e92023fff415. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:52.536126 31895 tablet_service.cc:1511] Processing CreateTablet for tablet c61f10da93eb4af5a80a204e679f1ee3 (DEFAULT_TABLE table=test-workload [id=108e8cf6859142debe8f012d0a9b704d]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260709 12:27:52.537556 31895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c61f10da93eb4af5a80a204e679f1ee3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:52.540146 31896 tablet_service.cc:1511] Processing CreateTablet for tablet 378f56c32c1e46f99a874bd1adcbcf25 (DEFAULT_TABLE table=test-workload [id=108e8cf6859142debe8f012d0a9b704d]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260709 12:27:52.541769 31970 tablet_service.cc:1511] Processing CreateTablet for tablet 378f56c32c1e46f99a874bd1adcbcf25 (DEFAULT_TABLE table=test-workload [id=108e8cf6859142debe8f012d0a9b704d]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260709 12:27:52.542590 31897 tablet_service.cc:1511] Processing CreateTablet for tablet 1638e215a9ee4068a999e92023fff415 (DEFAULT_TABLE table=test-workload [id=108e8cf6859142debe8f012d0a9b704d]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260709 12:27:52.542762 31969 tablet_service.cc:1511] Processing CreateTablet for tablet c61f10da93eb4af5a80a204e679f1ee3 (DEFAULT_TABLE table=test-workload [id=108e8cf6859142debe8f012d0a9b704d]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260709 12:27:52.543269 31970 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 378f56c32c1e46f99a874bd1adcbcf25. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:52.544235 31969 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c61f10da93eb4af5a80a204e679f1ee3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:52.545482 31894 tablet_service.cc:1511] Processing CreateTablet for tablet 52322cf19f734f19a4ccfc8389f32b41 (DEFAULT_TABLE table=test-workload [id=108e8cf6859142debe8f012d0a9b704d]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260709 12:27:52.547497 31893 tablet_service.cc:1511] Processing CreateTablet for tablet 6a5720f014ed48769eb641989edef3e8 (DEFAULT_TABLE table=test-workload [id=108e8cf6859142debe8f012d0a9b704d]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260709 12:27:52.549168 31896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 378f56c32c1e46f99a874bd1adcbcf25. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:52.550155 31968 tablet_service.cc:1511] Processing CreateTablet for tablet 52322cf19f734f19a4ccfc8389f32b41 (DEFAULT_TABLE table=test-workload [id=108e8cf6859142debe8f012d0a9b704d]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260709 12:27:52.551492 31968 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 52322cf19f734f19a4ccfc8389f32b41. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:52.552783 31897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1638e215a9ee4068a999e92023fff415. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:52.556609 31894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 52322cf19f734f19a4ccfc8389f32b41. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:52.557284 31893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a5720f014ed48769eb641989edef3e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:52.558863 31967 tablet_service.cc:1511] Processing CreateTablet for tablet 6a5720f014ed48769eb641989edef3e8 (DEFAULT_TABLE table=test-workload [id=108e8cf6859142debe8f012d0a9b704d]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260709 12:27:52.560321 31967 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a5720f014ed48769eb641989edef3e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:52.582674 31825 tablet_service.cc:1511] Processing CreateTablet for tablet 89784f886fb545d2871174c1e169fb32 (DEFAULT_TABLE table=test-workload [id=108e8cf6859142debe8f012d0a9b704d]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260709 12:27:52.584784 31825 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 89784f886fb545d2871174c1e169fb32. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:52.589707 31824 tablet_service.cc:1511] Processing CreateTablet for tablet fad0b7ef0f804aa494f44e9ec456e11e (DEFAULT_TABLE table=test-workload [id=108e8cf6859142debe8f012d0a9b704d]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260709 12:27:52.591331 31824 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fad0b7ef0f804aa494f44e9ec456e11e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:52.593776 31823 tablet_service.cc:1511] Processing CreateTablet for tablet d00d4a3dbb1c435cb565cf6e5d723bf2 (DEFAULT_TABLE table=test-workload [id=108e8cf6859142debe8f012d0a9b704d]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260709 12:27:52.598942 31823 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d00d4a3dbb1c435cb565cf6e5d723bf2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:52.598609 31966 tablet_service.cc:1511] Processing CreateTablet for tablet 0fb83fa22e13487f9d56facc914278ba (DEFAULT_TABLE table=test-workload [id=108e8cf6859142debe8f012d0a9b704d]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260709 12:27:52.596506 31822 tablet_service.cc:1511] Processing CreateTablet for tablet 1638e215a9ee4068a999e92023fff415 (DEFAULT_TABLE table=test-workload [id=108e8cf6859142debe8f012d0a9b704d]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260709 12:27:52.600021 31966 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0fb83fa22e13487f9d56facc914278ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:52.600692 31822 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1638e215a9ee4068a999e92023fff415. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:52.604508 31892 tablet_service.cc:1511] Processing CreateTablet for tablet 0fb83fa22e13487f9d56facc914278ba (DEFAULT_TABLE table=test-workload [id=108e8cf6859142debe8f012d0a9b704d]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260709 12:27:52.605108 31821 tablet_service.cc:1511] Processing CreateTablet for tablet 378f56c32c1e46f99a874bd1adcbcf25 (DEFAULT_TABLE table=test-workload [id=108e8cf6859142debe8f012d0a9b704d]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260709 12:27:52.606076 31892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0fb83fa22e13487f9d56facc914278ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:52.606698 31821 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 378f56c32c1e46f99a874bd1adcbcf25. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:52.609617 31820 tablet_service.cc:1511] Processing CreateTablet for tablet c61f10da93eb4af5a80a204e679f1ee3 (DEFAULT_TABLE table=test-workload [id=108e8cf6859142debe8f012d0a9b704d]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260709 12:27:52.611079 31820 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c61f10da93eb4af5a80a204e679f1ee3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:52.613540 31819 tablet_service.cc:1511] Processing CreateTablet for tablet 52322cf19f734f19a4ccfc8389f32b41 (DEFAULT_TABLE table=test-workload [id=108e8cf6859142debe8f012d0a9b704d]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260709 12:27:52.614933 31819 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 52322cf19f734f19a4ccfc8389f32b41. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:52.620249 31818 tablet_service.cc:1511] Processing CreateTablet for tablet 6a5720f014ed48769eb641989edef3e8 (DEFAULT_TABLE table=test-workload [id=108e8cf6859142debe8f012d0a9b704d]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260709 12:27:52.621701 31818 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a5720f014ed48769eb641989edef3e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:52.646221 32025 tablet_bootstrap.cc:492] T 6a5720f014ed48769eb641989edef3e8 P 64fd1d8a3cd14e66ac94c4fbcceaf039: Bootstrap starting.
I20260709 12:27:52.658711 31819 tablet_service.cc:1511] Processing CreateTablet for tablet 0fb83fa22e13487f9d56facc914278ba (DEFAULT_TABLE table=test-workload [id=108e8cf6859142debe8f012d0a9b704d]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260709 12:27:52.661916 31819 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0fb83fa22e13487f9d56facc914278ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:52.680581 32024 tablet_bootstrap.cc:492] T d00d4a3dbb1c435cb565cf6e5d723bf2 P 07b97783285d4b22b6fd0f1d816ef930: Bootstrap starting.
I20260709 12:27:52.698805 32026 tablet_bootstrap.cc:492] T 89784f886fb545d2871174c1e169fb32 P 2ae3a9e2293442e291b1262a717a5e50: Bootstrap starting.
I20260709 12:27:52.705704 32026 tablet_bootstrap.cc:654] T 89784f886fb545d2871174c1e169fb32 P 2ae3a9e2293442e291b1262a717a5e50: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:52.706583 32025 tablet_bootstrap.cc:654] T 6a5720f014ed48769eb641989edef3e8 P 64fd1d8a3cd14e66ac94c4fbcceaf039: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:52.707296 32024 tablet_bootstrap.cc:654] T d00d4a3dbb1c435cb565cf6e5d723bf2 P 07b97783285d4b22b6fd0f1d816ef930: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:52.714561 32026 tablet_bootstrap.cc:492] T 89784f886fb545d2871174c1e169fb32 P 2ae3a9e2293442e291b1262a717a5e50: No bootstrap required, opened a new log
I20260709 12:27:52.715165 32026 ts_tablet_manager.cc:1403] T 89784f886fb545d2871174c1e169fb32 P 2ae3a9e2293442e291b1262a717a5e50: Time spent bootstrapping tablet: real 0.017s	user 0.014s	sys 0.000s
I20260709 12:27:52.716364 32025 tablet_bootstrap.cc:492] T 6a5720f014ed48769eb641989edef3e8 P 64fd1d8a3cd14e66ac94c4fbcceaf039: No bootstrap required, opened a new log
I20260709 12:27:52.716961 32025 ts_tablet_manager.cc:1403] T 6a5720f014ed48769eb641989edef3e8 P 64fd1d8a3cd14e66ac94c4fbcceaf039: Time spent bootstrapping tablet: real 0.071s	user 0.008s	sys 0.008s
I20260709 12:27:52.718042 32026 raft_consensus.cc:359] T 89784f886fb545d2871174c1e169fb32 P 2ae3a9e2293442e291b1262a717a5e50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } } peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } }
I20260709 12:27:52.718775 32026 raft_consensus.cc:385] T 89784f886fb545d2871174c1e169fb32 P 2ae3a9e2293442e291b1262a717a5e50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:52.718852 32024 tablet_bootstrap.cc:492] T d00d4a3dbb1c435cb565cf6e5d723bf2 P 07b97783285d4b22b6fd0f1d816ef930: No bootstrap required, opened a new log
I20260709 12:27:52.719106 32026 raft_consensus.cc:740] T 89784f886fb545d2871174c1e169fb32 P 2ae3a9e2293442e291b1262a717a5e50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ae3a9e2293442e291b1262a717a5e50, State: Initialized, Role: FOLLOWER
I20260709 12:27:52.719491 32024 ts_tablet_manager.cc:1403] T d00d4a3dbb1c435cb565cf6e5d723bf2 P 07b97783285d4b22b6fd0f1d816ef930: Time spent bootstrapping tablet: real 0.039s	user 0.003s	sys 0.010s
I20260709 12:27:52.719980 32026 consensus_queue.cc:260] T 89784f886fb545d2871174c1e169fb32 P 2ae3a9e2293442e291b1262a717a5e50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } } peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } }
I20260709 12:27:52.720315 32025 raft_consensus.cc:359] T 6a5720f014ed48769eb641989edef3e8 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:52.721249 32025 raft_consensus.cc:385] T 6a5720f014ed48769eb641989edef3e8 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:52.721596 32025 raft_consensus.cc:740] T 6a5720f014ed48769eb641989edef3e8 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64fd1d8a3cd14e66ac94c4fbcceaf039, State: Initialized, Role: FOLLOWER
I20260709 12:27:52.722574 32025 consensus_queue.cc:260] T 6a5720f014ed48769eb641989edef3e8 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:52.725085 31860 heartbeater.cc:499] Master 127.16.29.126:37739 was elected leader, sending a full tablet report...
I20260709 12:27:52.725559 32024 raft_consensus.cc:359] T d00d4a3dbb1c435cb565cf6e5d723bf2 P 07b97783285d4b22b6fd0f1d816ef930 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:52.726786 32024 raft_consensus.cc:385] T d00d4a3dbb1c435cb565cf6e5d723bf2 P 07b97783285d4b22b6fd0f1d816ef930 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:52.727206 32024 raft_consensus.cc:740] T d00d4a3dbb1c435cb565cf6e5d723bf2 P 07b97783285d4b22b6fd0f1d816ef930 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07b97783285d4b22b6fd0f1d816ef930, State: Initialized, Role: FOLLOWER
I20260709 12:27:52.728883 32026 ts_tablet_manager.cc:1434] T 89784f886fb545d2871174c1e169fb32 P 2ae3a9e2293442e291b1262a717a5e50: Time spent starting tablet: real 0.013s	user 0.010s	sys 0.000s
I20260709 12:27:52.728530 32024 consensus_queue.cc:260] T d00d4a3dbb1c435cb565cf6e5d723bf2 P 07b97783285d4b22b6fd0f1d816ef930 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:52.730051 32026 tablet_bootstrap.cc:492] T 52322cf19f734f19a4ccfc8389f32b41 P 2ae3a9e2293442e291b1262a717a5e50: Bootstrap starting.
I20260709 12:27:52.732945 32025 ts_tablet_manager.cc:1434] T 6a5720f014ed48769eb641989edef3e8 P 64fd1d8a3cd14e66ac94c4fbcceaf039: Time spent starting tablet: real 0.016s	user 0.002s	sys 0.012s
I20260709 12:27:52.734133 31935 heartbeater.cc:499] Master 127.16.29.126:37739 was elected leader, sending a full tablet report...
I20260709 12:27:52.738023 32026 tablet_bootstrap.cc:654] T 52322cf19f734f19a4ccfc8389f32b41 P 2ae3a9e2293442e291b1262a717a5e50: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:52.737903 32009 heartbeater.cc:499] Master 127.16.29.126:37739 was elected leader, sending a full tablet report...
I20260709 12:27:52.739377 32025 tablet_bootstrap.cc:492] T d00d4a3dbb1c435cb565cf6e5d723bf2 P 64fd1d8a3cd14e66ac94c4fbcceaf039: Bootstrap starting.
I20260709 12:27:52.740159 32024 ts_tablet_manager.cc:1434] T d00d4a3dbb1c435cb565cf6e5d723bf2 P 07b97783285d4b22b6fd0f1d816ef930: Time spent starting tablet: real 0.020s	user 0.012s	sys 0.003s
I20260709 12:27:52.744139 32024 tablet_bootstrap.cc:492] T 52322cf19f734f19a4ccfc8389f32b41 P 07b97783285d4b22b6fd0f1d816ef930: Bootstrap starting.
I20260709 12:27:52.747020 32025 tablet_bootstrap.cc:654] T d00d4a3dbb1c435cb565cf6e5d723bf2 P 64fd1d8a3cd14e66ac94c4fbcceaf039: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:52.751201 32024 tablet_bootstrap.cc:654] T 52322cf19f734f19a4ccfc8389f32b41 P 07b97783285d4b22b6fd0f1d816ef930: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:52.753443 32026 tablet_bootstrap.cc:492] T 52322cf19f734f19a4ccfc8389f32b41 P 2ae3a9e2293442e291b1262a717a5e50: No bootstrap required, opened a new log
I20260709 12:27:52.754027 32026 ts_tablet_manager.cc:1403] T 52322cf19f734f19a4ccfc8389f32b41 P 2ae3a9e2293442e291b1262a717a5e50: Time spent bootstrapping tablet: real 0.024s	user 0.012s	sys 0.009s
I20260709 12:27:52.757491 32026 raft_consensus.cc:359] T 52322cf19f734f19a4ccfc8389f32b41 P 2ae3a9e2293442e291b1262a717a5e50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:52.758229 32026 raft_consensus.cc:385] T 52322cf19f734f19a4ccfc8389f32b41 P 2ae3a9e2293442e291b1262a717a5e50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:52.758525 32026 raft_consensus.cc:740] T 52322cf19f734f19a4ccfc8389f32b41 P 2ae3a9e2293442e291b1262a717a5e50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ae3a9e2293442e291b1262a717a5e50, State: Initialized, Role: FOLLOWER
I20260709 12:27:52.758725 32024 tablet_bootstrap.cc:492] T 52322cf19f734f19a4ccfc8389f32b41 P 07b97783285d4b22b6fd0f1d816ef930: No bootstrap required, opened a new log
I20260709 12:27:52.759385 32024 ts_tablet_manager.cc:1403] T 52322cf19f734f19a4ccfc8389f32b41 P 07b97783285d4b22b6fd0f1d816ef930: Time spent bootstrapping tablet: real 0.016s	user 0.006s	sys 0.008s
I20260709 12:27:52.763259 32024 raft_consensus.cc:359] T 52322cf19f734f19a4ccfc8389f32b41 P 07b97783285d4b22b6fd0f1d816ef930 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:52.764074 32024 raft_consensus.cc:385] T 52322cf19f734f19a4ccfc8389f32b41 P 07b97783285d4b22b6fd0f1d816ef930 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:52.764439 32024 raft_consensus.cc:740] T 52322cf19f734f19a4ccfc8389f32b41 P 07b97783285d4b22b6fd0f1d816ef930 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07b97783285d4b22b6fd0f1d816ef930, State: Initialized, Role: FOLLOWER
I20260709 12:27:52.764825 32025 tablet_bootstrap.cc:492] T d00d4a3dbb1c435cb565cf6e5d723bf2 P 64fd1d8a3cd14e66ac94c4fbcceaf039: No bootstrap required, opened a new log
I20260709 12:27:52.765395 32025 ts_tablet_manager.cc:1403] T d00d4a3dbb1c435cb565cf6e5d723bf2 P 64fd1d8a3cd14e66ac94c4fbcceaf039: Time spent bootstrapping tablet: real 0.026s	user 0.009s	sys 0.005s
I20260709 12:27:52.765475 32024 consensus_queue.cc:260] T 52322cf19f734f19a4ccfc8389f32b41 P 07b97783285d4b22b6fd0f1d816ef930 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:52.768390 32025 raft_consensus.cc:359] T d00d4a3dbb1c435cb565cf6e5d723bf2 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:52.769371 32025 raft_consensus.cc:385] T d00d4a3dbb1c435cb565cf6e5d723bf2 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:52.769526 32024 ts_tablet_manager.cc:1434] T 52322cf19f734f19a4ccfc8389f32b41 P 07b97783285d4b22b6fd0f1d816ef930: Time spent starting tablet: real 0.010s	user 0.008s	sys 0.000s
I20260709 12:27:52.759337 32026 consensus_queue.cc:260] T 52322cf19f734f19a4ccfc8389f32b41 P 2ae3a9e2293442e291b1262a717a5e50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:52.771795 32026 ts_tablet_manager.cc:1434] T 52322cf19f734f19a4ccfc8389f32b41 P 2ae3a9e2293442e291b1262a717a5e50: Time spent starting tablet: real 0.017s	user 0.005s	sys 0.003s
I20260709 12:27:52.772615 32024 tablet_bootstrap.cc:492] T 89784f886fb545d2871174c1e169fb32 P 07b97783285d4b22b6fd0f1d816ef930: Bootstrap starting.
I20260709 12:27:52.772902 32026 tablet_bootstrap.cc:492] T 378f56c32c1e46f99a874bd1adcbcf25 P 2ae3a9e2293442e291b1262a717a5e50: Bootstrap starting.
I20260709 12:27:52.773483 32025 raft_consensus.cc:740] T d00d4a3dbb1c435cb565cf6e5d723bf2 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64fd1d8a3cd14e66ac94c4fbcceaf039, State: Initialized, Role: FOLLOWER
I20260709 12:27:52.774451 32025 consensus_queue.cc:260] T d00d4a3dbb1c435cb565cf6e5d723bf2 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:52.779577 32025 ts_tablet_manager.cc:1434] T d00d4a3dbb1c435cb565cf6e5d723bf2 P 64fd1d8a3cd14e66ac94c4fbcceaf039: Time spent starting tablet: real 0.014s	user 0.005s	sys 0.004s
I20260709 12:27:52.780733 32025 tablet_bootstrap.cc:492] T c61f10da93eb4af5a80a204e679f1ee3 P 64fd1d8a3cd14e66ac94c4fbcceaf039: Bootstrap starting.
I20260709 12:27:52.783142 32024 tablet_bootstrap.cc:654] T 89784f886fb545d2871174c1e169fb32 P 07b97783285d4b22b6fd0f1d816ef930: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:52.788532 32026 tablet_bootstrap.cc:654] T 378f56c32c1e46f99a874bd1adcbcf25 P 2ae3a9e2293442e291b1262a717a5e50: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:52.791204 32024 tablet_bootstrap.cc:492] T 89784f886fb545d2871174c1e169fb32 P 07b97783285d4b22b6fd0f1d816ef930: No bootstrap required, opened a new log
I20260709 12:27:52.791838 32024 ts_tablet_manager.cc:1403] T 89784f886fb545d2871174c1e169fb32 P 07b97783285d4b22b6fd0f1d816ef930: Time spent bootstrapping tablet: real 0.020s	user 0.011s	sys 0.006s
I20260709 12:27:52.794993 32024 raft_consensus.cc:359] T 89784f886fb545d2871174c1e169fb32 P 07b97783285d4b22b6fd0f1d816ef930 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } } peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } }
I20260709 12:27:52.795778 32024 raft_consensus.cc:385] T 89784f886fb545d2871174c1e169fb32 P 07b97783285d4b22b6fd0f1d816ef930 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:52.796115 32024 raft_consensus.cc:740] T 89784f886fb545d2871174c1e169fb32 P 07b97783285d4b22b6fd0f1d816ef930 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07b97783285d4b22b6fd0f1d816ef930, State: Initialized, Role: FOLLOWER
I20260709 12:27:52.796846 32024 consensus_queue.cc:260] T 89784f886fb545d2871174c1e169fb32 P 07b97783285d4b22b6fd0f1d816ef930 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } } peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } }
I20260709 12:27:52.797477 32025 tablet_bootstrap.cc:654] T c61f10da93eb4af5a80a204e679f1ee3 P 64fd1d8a3cd14e66ac94c4fbcceaf039: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:52.799235 32024 ts_tablet_manager.cc:1434] T 89784f886fb545d2871174c1e169fb32 P 07b97783285d4b22b6fd0f1d816ef930: Time spent starting tablet: real 0.007s	user 0.004s	sys 0.000s
I20260709 12:27:52.800276 32024 tablet_bootstrap.cc:492] T c61f10da93eb4af5a80a204e679f1ee3 P 07b97783285d4b22b6fd0f1d816ef930: Bootstrap starting.
I20260709 12:27:52.803285 32025 tablet_bootstrap.cc:492] T c61f10da93eb4af5a80a204e679f1ee3 P 64fd1d8a3cd14e66ac94c4fbcceaf039: No bootstrap required, opened a new log
I20260709 12:27:52.803953 32025 ts_tablet_manager.cc:1403] T c61f10da93eb4af5a80a204e679f1ee3 P 64fd1d8a3cd14e66ac94c4fbcceaf039: Time spent bootstrapping tablet: real 0.024s	user 0.010s	sys 0.003s
I20260709 12:27:52.806672 32026 tablet_bootstrap.cc:492] T 378f56c32c1e46f99a874bd1adcbcf25 P 2ae3a9e2293442e291b1262a717a5e50: No bootstrap required, opened a new log
I20260709 12:27:52.807212 32026 ts_tablet_manager.cc:1403] T 378f56c32c1e46f99a874bd1adcbcf25 P 2ae3a9e2293442e291b1262a717a5e50: Time spent bootstrapping tablet: real 0.035s	user 0.007s	sys 0.005s
I20260709 12:27:52.807852 32025 raft_consensus.cc:359] T c61f10da93eb4af5a80a204e679f1ee3 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:52.808511 32024 tablet_bootstrap.cc:654] T c61f10da93eb4af5a80a204e679f1ee3 P 07b97783285d4b22b6fd0f1d816ef930: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:52.808599 32025 raft_consensus.cc:385] T c61f10da93eb4af5a80a204e679f1ee3 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:52.809141 32025 raft_consensus.cc:740] T c61f10da93eb4af5a80a204e679f1ee3 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64fd1d8a3cd14e66ac94c4fbcceaf039, State: Initialized, Role: FOLLOWER
I20260709 12:27:52.809878 32025 consensus_queue.cc:260] T c61f10da93eb4af5a80a204e679f1ee3 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:52.810393 32026 raft_consensus.cc:359] T 378f56c32c1e46f99a874bd1adcbcf25 P 2ae3a9e2293442e291b1262a717a5e50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:52.811167 32026 raft_consensus.cc:385] T 378f56c32c1e46f99a874bd1adcbcf25 P 2ae3a9e2293442e291b1262a717a5e50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:52.811496 32026 raft_consensus.cc:740] T 378f56c32c1e46f99a874bd1adcbcf25 P 2ae3a9e2293442e291b1262a717a5e50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ae3a9e2293442e291b1262a717a5e50, State: Initialized, Role: FOLLOWER
I20260709 12:27:52.812323 32025 ts_tablet_manager.cc:1434] T c61f10da93eb4af5a80a204e679f1ee3 P 64fd1d8a3cd14e66ac94c4fbcceaf039: Time spent starting tablet: real 0.008s	user 0.007s	sys 0.000s
I20260709 12:27:52.812654 32026 consensus_queue.cc:260] T 378f56c32c1e46f99a874bd1adcbcf25 P 2ae3a9e2293442e291b1262a717a5e50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:52.813934 32025 tablet_bootstrap.cc:492] T 1638e215a9ee4068a999e92023fff415 P 64fd1d8a3cd14e66ac94c4fbcceaf039: Bootstrap starting.
I20260709 12:27:52.822023 32025 tablet_bootstrap.cc:654] T 1638e215a9ee4068a999e92023fff415 P 64fd1d8a3cd14e66ac94c4fbcceaf039: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:52.824997 32026 ts_tablet_manager.cc:1434] T 378f56c32c1e46f99a874bd1adcbcf25 P 2ae3a9e2293442e291b1262a717a5e50: Time spent starting tablet: real 0.017s	user 0.005s	sys 0.004s
I20260709 12:27:52.829290 32026 tablet_bootstrap.cc:492] T d00d4a3dbb1c435cb565cf6e5d723bf2 P 2ae3a9e2293442e291b1262a717a5e50: Bootstrap starting.
I20260709 12:27:52.831090 32025 tablet_bootstrap.cc:492] T 1638e215a9ee4068a999e92023fff415 P 64fd1d8a3cd14e66ac94c4fbcceaf039: No bootstrap required, opened a new log
I20260709 12:27:52.831818 32025 ts_tablet_manager.cc:1403] T 1638e215a9ee4068a999e92023fff415 P 64fd1d8a3cd14e66ac94c4fbcceaf039: Time spent bootstrapping tablet: real 0.018s	user 0.011s	sys 0.005s
I20260709 12:27:52.836019 32024 tablet_bootstrap.cc:492] T c61f10da93eb4af5a80a204e679f1ee3 P 07b97783285d4b22b6fd0f1d816ef930: No bootstrap required, opened a new log
I20260709 12:27:52.835572 32025 raft_consensus.cc:359] T 1638e215a9ee4068a999e92023fff415 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:52.836484 32024 ts_tablet_manager.cc:1403] T c61f10da93eb4af5a80a204e679f1ee3 P 07b97783285d4b22b6fd0f1d816ef930: Time spent bootstrapping tablet: real 0.036s	user 0.011s	sys 0.023s
I20260709 12:27:52.836616 32025 raft_consensus.cc:385] T 1638e215a9ee4068a999e92023fff415 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:52.837128 32025 raft_consensus.cc:740] T 1638e215a9ee4068a999e92023fff415 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64fd1d8a3cd14e66ac94c4fbcceaf039, State: Initialized, Role: FOLLOWER
I20260709 12:27:52.837954 32026 tablet_bootstrap.cc:654] T d00d4a3dbb1c435cb565cf6e5d723bf2 P 2ae3a9e2293442e291b1262a717a5e50: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:52.838069 32025 consensus_queue.cc:260] T 1638e215a9ee4068a999e92023fff415 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:52.838802 32024 raft_consensus.cc:359] T c61f10da93eb4af5a80a204e679f1ee3 P 07b97783285d4b22b6fd0f1d816ef930 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:52.839359 32024 raft_consensus.cc:385] T c61f10da93eb4af5a80a204e679f1ee3 P 07b97783285d4b22b6fd0f1d816ef930 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:52.839556 32024 raft_consensus.cc:740] T c61f10da93eb4af5a80a204e679f1ee3 P 07b97783285d4b22b6fd0f1d816ef930 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07b97783285d4b22b6fd0f1d816ef930, State: Initialized, Role: FOLLOWER
I20260709 12:27:52.840309 32024 consensus_queue.cc:260] T c61f10da93eb4af5a80a204e679f1ee3 P 07b97783285d4b22b6fd0f1d816ef930 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:52.842434 32024 ts_tablet_manager.cc:1434] T c61f10da93eb4af5a80a204e679f1ee3 P 07b97783285d4b22b6fd0f1d816ef930: Time spent starting tablet: real 0.006s	user 0.004s	sys 0.000s
I20260709 12:27:52.843436 32024 tablet_bootstrap.cc:492] T 378f56c32c1e46f99a874bd1adcbcf25 P 07b97783285d4b22b6fd0f1d816ef930: Bootstrap starting.
I20260709 12:27:52.845506 32025 ts_tablet_manager.cc:1434] T 1638e215a9ee4068a999e92023fff415 P 64fd1d8a3cd14e66ac94c4fbcceaf039: Time spent starting tablet: real 0.013s	user 0.007s	sys 0.000s
I20260709 12:27:52.847187 32025 tablet_bootstrap.cc:492] T fad0b7ef0f804aa494f44e9ec456e11e P 64fd1d8a3cd14e66ac94c4fbcceaf039: Bootstrap starting.
I20260709 12:27:52.850445 32024 tablet_bootstrap.cc:654] T 378f56c32c1e46f99a874bd1adcbcf25 P 07b97783285d4b22b6fd0f1d816ef930: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:52.850999 32032 raft_consensus.cc:493] T 89784f886fb545d2871174c1e169fb32 P 07b97783285d4b22b6fd0f1d816ef930 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:52.851534 32032 raft_consensus.cc:515] T 89784f886fb545d2871174c1e169fb32 P 07b97783285d4b22b6fd0f1d816ef930 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } } peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } }
I20260709 12:27:52.854488 32032 leader_election.cc:290] T 89784f886fb545d2871174c1e169fb32 P 07b97783285d4b22b6fd0f1d816ef930 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2ae3a9e2293442e291b1262a717a5e50 (127.16.29.65:44289), 64fd1d8a3cd14e66ac94c4fbcceaf039 (127.16.29.66:42163)
I20260709 12:27:52.856822 32025 tablet_bootstrap.cc:654] T fad0b7ef0f804aa494f44e9ec456e11e P 64fd1d8a3cd14e66ac94c4fbcceaf039: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:52.858422 32030 raft_consensus.cc:493] T 378f56c32c1e46f99a874bd1adcbcf25 P 2ae3a9e2293442e291b1262a717a5e50 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:52.859035 32030 raft_consensus.cc:515] T 378f56c32c1e46f99a874bd1adcbcf25 P 2ae3a9e2293442e291b1262a717a5e50 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:52.873863 32030 leader_election.cc:290] T 378f56c32c1e46f99a874bd1adcbcf25 P 2ae3a9e2293442e291b1262a717a5e50 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 64fd1d8a3cd14e66ac94c4fbcceaf039 (127.16.29.66:42163), 07b97783285d4b22b6fd0f1d816ef930 (127.16.29.67:34419)
I20260709 12:27:52.874575 32030 raft_consensus.cc:493] T 52322cf19f734f19a4ccfc8389f32b41 P 2ae3a9e2293442e291b1262a717a5e50 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:52.875504 32030 raft_consensus.cc:515] T 52322cf19f734f19a4ccfc8389f32b41 P 2ae3a9e2293442e291b1262a717a5e50 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:52.877960 32026 tablet_bootstrap.cc:492] T d00d4a3dbb1c435cb565cf6e5d723bf2 P 2ae3a9e2293442e291b1262a717a5e50: No bootstrap required, opened a new log
I20260709 12:27:52.878778 32026 ts_tablet_manager.cc:1403] T d00d4a3dbb1c435cb565cf6e5d723bf2 P 2ae3a9e2293442e291b1262a717a5e50: Time spent bootstrapping tablet: real 0.050s	user 0.021s	sys 0.024s
I20260709 12:27:52.878340 32030 leader_election.cc:290] T 52322cf19f734f19a4ccfc8389f32b41 P 2ae3a9e2293442e291b1262a717a5e50 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 64fd1d8a3cd14e66ac94c4fbcceaf039 (127.16.29.66:42163), 07b97783285d4b22b6fd0f1d816ef930 (127.16.29.67:34419)
I20260709 12:27:52.882208 32026 raft_consensus.cc:359] T d00d4a3dbb1c435cb565cf6e5d723bf2 P 2ae3a9e2293442e291b1262a717a5e50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:52.883140 32026 raft_consensus.cc:385] T d00d4a3dbb1c435cb565cf6e5d723bf2 P 2ae3a9e2293442e291b1262a717a5e50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:52.883456 32026 raft_consensus.cc:740] T d00d4a3dbb1c435cb565cf6e5d723bf2 P 2ae3a9e2293442e291b1262a717a5e50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ae3a9e2293442e291b1262a717a5e50, State: Initialized, Role: FOLLOWER
I20260709 12:27:52.884251 32026 consensus_queue.cc:260] T d00d4a3dbb1c435cb565cf6e5d723bf2 P 2ae3a9e2293442e291b1262a717a5e50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:52.886529 32026 ts_tablet_manager.cc:1434] T d00d4a3dbb1c435cb565cf6e5d723bf2 P 2ae3a9e2293442e291b1262a717a5e50: Time spent starting tablet: real 0.007s	user 0.009s	sys 0.000s
I20260709 12:27:52.887533 32026 tablet_bootstrap.cc:492] T c61f10da93eb4af5a80a204e679f1ee3 P 2ae3a9e2293442e291b1262a717a5e50: Bootstrap starting.
I20260709 12:27:52.897799 32031 raft_consensus.cc:493] T c61f10da93eb4af5a80a204e679f1ee3 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:52.897957 32026 tablet_bootstrap.cc:654] T c61f10da93eb4af5a80a204e679f1ee3 P 2ae3a9e2293442e291b1262a717a5e50: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:52.898397 32031 raft_consensus.cc:515] T c61f10da93eb4af5a80a204e679f1ee3 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:52.899816 32025 tablet_bootstrap.cc:492] T fad0b7ef0f804aa494f44e9ec456e11e P 64fd1d8a3cd14e66ac94c4fbcceaf039: No bootstrap required, opened a new log
I20260709 12:27:52.906764 32025 ts_tablet_manager.cc:1403] T fad0b7ef0f804aa494f44e9ec456e11e P 64fd1d8a3cd14e66ac94c4fbcceaf039: Time spent bootstrapping tablet: real 0.060s	user 0.022s	sys 0.003s
I20260709 12:27:52.919179 32024 tablet_bootstrap.cc:492] T 378f56c32c1e46f99a874bd1adcbcf25 P 07b97783285d4b22b6fd0f1d816ef930: No bootstrap required, opened a new log
I20260709 12:27:52.919800 32024 ts_tablet_manager.cc:1403] T 378f56c32c1e46f99a874bd1adcbcf25 P 07b97783285d4b22b6fd0f1d816ef930: Time spent bootstrapping tablet: real 0.077s	user 0.009s	sys 0.009s
I20260709 12:27:52.923377 32024 raft_consensus.cc:359] T 378f56c32c1e46f99a874bd1adcbcf25 P 07b97783285d4b22b6fd0f1d816ef930 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:52.924180 32024 raft_consensus.cc:385] T 378f56c32c1e46f99a874bd1adcbcf25 P 07b97783285d4b22b6fd0f1d816ef930 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:52.924472 32024 raft_consensus.cc:740] T 378f56c32c1e46f99a874bd1adcbcf25 P 07b97783285d4b22b6fd0f1d816ef930 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07b97783285d4b22b6fd0f1d816ef930, State: Initialized, Role: FOLLOWER
I20260709 12:27:52.925308 32024 consensus_queue.cc:260] T 378f56c32c1e46f99a874bd1adcbcf25 P 07b97783285d4b22b6fd0f1d816ef930 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:52.926553 31835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "89784f886fb545d2871174c1e169fb32" candidate_uuid: "07b97783285d4b22b6fd0f1d816ef930" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ae3a9e2293442e291b1262a717a5e50" is_pre_election: true
I20260709 12:27:52.927335 31835 raft_consensus.cc:2468] T 89784f886fb545d2871174c1e169fb32 P 2ae3a9e2293442e291b1262a717a5e50 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 07b97783285d4b22b6fd0f1d816ef930 in term 0.
I20260709 12:27:52.904713 32031 leader_election.cc:290] T c61f10da93eb4af5a80a204e679f1ee3 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2ae3a9e2293442e291b1262a717a5e50 (127.16.29.65:44289), 07b97783285d4b22b6fd0f1d816ef930 (127.16.29.67:34419)
I20260709 12:27:52.915871 32026 tablet_bootstrap.cc:492] T c61f10da93eb4af5a80a204e679f1ee3 P 2ae3a9e2293442e291b1262a717a5e50: No bootstrap required, opened a new log
I20260709 12:27:52.927737 32024 ts_tablet_manager.cc:1434] T 378f56c32c1e46f99a874bd1adcbcf25 P 07b97783285d4b22b6fd0f1d816ef930: Time spent starting tablet: real 0.008s	user 0.004s	sys 0.000s
I20260709 12:27:52.930902 32026 ts_tablet_manager.cc:1403] T c61f10da93eb4af5a80a204e679f1ee3 P 2ae3a9e2293442e291b1262a717a5e50: Time spent bootstrapping tablet: real 0.044s	user 0.015s	sys 0.008s
I20260709 12:27:52.938169 32024 tablet_bootstrap.cc:492] T 1638e215a9ee4068a999e92023fff415 P 07b97783285d4b22b6fd0f1d816ef930: Bootstrap starting.
I20260709 12:27:52.937134 31949 leader_election.cc:304] T 89784f886fb545d2871174c1e169fb32 P 07b97783285d4b22b6fd0f1d816ef930 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07b97783285d4b22b6fd0f1d816ef930, 2ae3a9e2293442e291b1262a717a5e50; no voters: 
I20260709 12:27:52.939275 32032 raft_consensus.cc:2804] T 89784f886fb545d2871174c1e169fb32 P 07b97783285d4b22b6fd0f1d816ef930 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:52.956533 31910 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "89784f886fb545d2871174c1e169fb32" candidate_uuid: "07b97783285d4b22b6fd0f1d816ef930" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" is_pre_election: true
I20260709 12:27:52.963639 32025 raft_consensus.cc:359] T fad0b7ef0f804aa494f44e9ec456e11e P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:52.964681 32025 raft_consensus.cc:385] T fad0b7ef0f804aa494f44e9ec456e11e P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:52.965080 32025 raft_consensus.cc:740] T fad0b7ef0f804aa494f44e9ec456e11e P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64fd1d8a3cd14e66ac94c4fbcceaf039, State: Initialized, Role: FOLLOWER
I20260709 12:27:52.966244 32025 consensus_queue.cc:260] T fad0b7ef0f804aa494f44e9ec456e11e P 64fd1d8a3cd14e66ac94c4fbcceaf039 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:52.945565 32032 raft_consensus.cc:493] T 89784f886fb545d2871174c1e169fb32 P 07b97783285d4b22b6fd0f1d816ef930 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:52.968282 32032 raft_consensus.cc:3060] T 89784f886fb545d2871174c1e169fb32 P 07b97783285d4b22b6fd0f1d816ef930 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:52.966127 32026 raft_consensus.cc:359] T c61f10da93eb4af5a80a204e679f1ee3 P 2ae3a9e2293442e291b1262a717a5e50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:52.986971 32052 raft_consensus.cc:493] T 378f56c32c1e46f99a874bd1adcbcf25 P 07b97783285d4b22b6fd0f1d816ef930 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:52.980631 32031 raft_consensus.cc:493] T d00d4a3dbb1c435cb565cf6e5d723bf2 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:52.987565 32052 raft_consensus.cc:515] T 378f56c32c1e46f99a874bd1adcbcf25 P 07b97783285d4b22b6fd0f1d816ef930 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:52.984119 32026 raft_consensus.cc:385] T c61f10da93eb4af5a80a204e679f1ee3 P 2ae3a9e2293442e291b1262a717a5e50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:52.991393 32026 raft_consensus.cc:740] T c61f10da93eb4af5a80a204e679f1ee3 P 2ae3a9e2293442e291b1262a717a5e50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ae3a9e2293442e291b1262a717a5e50, State: Initialized, Role: FOLLOWER
W20260709 12:27:52.994355 31947 leader_election.cc:343] T 89784f886fb545d2871174c1e169fb32 P 07b97783285d4b22b6fd0f1d816ef930 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 64fd1d8a3cd14e66ac94c4fbcceaf039 (127.16.29.66:42163): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:52.989081 32031 raft_consensus.cc:515] T d00d4a3dbb1c435cb565cf6e5d723bf2 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
W20260709 12:27:52.999375 31781 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:27:53.003113 32024 tablet_bootstrap.cc:654] T 1638e215a9ee4068a999e92023fff415 P 07b97783285d4b22b6fd0f1d816ef930: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:53.003808 32031 leader_election.cc:290] T d00d4a3dbb1c435cb565cf6e5d723bf2 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2ae3a9e2293442e291b1262a717a5e50 (127.16.29.65:44289), 07b97783285d4b22b6fd0f1d816ef930 (127.16.29.67:34419)
I20260709 12:27:53.004757 32025 ts_tablet_manager.cc:1434] T fad0b7ef0f804aa494f44e9ec456e11e P 64fd1d8a3cd14e66ac94c4fbcceaf039: Time spent starting tablet: real 0.092s	user 0.010s	sys 0.001s
I20260709 12:27:53.005033 31835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378f56c32c1e46f99a874bd1adcbcf25" candidate_uuid: "07b97783285d4b22b6fd0f1d816ef930" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ae3a9e2293442e291b1262a717a5e50" is_pre_election: true
I20260709 12:27:53.007985 31835 raft_consensus.cc:2468] T 378f56c32c1e46f99a874bd1adcbcf25 P 2ae3a9e2293442e291b1262a717a5e50 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 07b97783285d4b22b6fd0f1d816ef930 in term 0.
I20260709 12:27:53.009474 31949 leader_election.cc:304] T 378f56c32c1e46f99a874bd1adcbcf25 P 07b97783285d4b22b6fd0f1d816ef930 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07b97783285d4b22b6fd0f1d816ef930, 2ae3a9e2293442e291b1262a717a5e50; no voters: 
I20260709 12:27:53.005335 32052 leader_election.cc:290] T 378f56c32c1e46f99a874bd1adcbcf25 P 07b97783285d4b22b6fd0f1d816ef930 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2ae3a9e2293442e291b1262a717a5e50 (127.16.29.65:44289), 64fd1d8a3cd14e66ac94c4fbcceaf039 (127.16.29.66:42163)
I20260709 12:27:53.006992 31910 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378f56c32c1e46f99a874bd1adcbcf25" candidate_uuid: "07b97783285d4b22b6fd0f1d816ef930" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" is_pre_election: true
I20260709 12:27:53.012645 32025 tablet_bootstrap.cc:492] T 378f56c32c1e46f99a874bd1adcbcf25 P 64fd1d8a3cd14e66ac94c4fbcceaf039: Bootstrap starting.
I20260709 12:27:53.017822 31835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c61f10da93eb4af5a80a204e679f1ee3" candidate_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ae3a9e2293442e291b1262a717a5e50" is_pre_election: true
I20260709 12:27:53.018479 31834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d00d4a3dbb1c435cb565cf6e5d723bf2" candidate_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ae3a9e2293442e291b1262a717a5e50" is_pre_election: true
I20260709 12:27:53.019450 31834 raft_consensus.cc:2468] T d00d4a3dbb1c435cb565cf6e5d723bf2 P 2ae3a9e2293442e291b1262a717a5e50 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64fd1d8a3cd14e66ac94c4fbcceaf039 in term 0.
I20260709 12:27:52.998909 32026 consensus_queue.cc:260] T c61f10da93eb4af5a80a204e679f1ee3 P 2ae3a9e2293442e291b1262a717a5e50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:53.023203 32031 raft_consensus.cc:493] T 6a5720f014ed48769eb641989edef3e8 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:53.023923 32031 raft_consensus.cc:515] T 6a5720f014ed48769eb641989edef3e8 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:53.027498 31875 leader_election.cc:304] T d00d4a3dbb1c435cb565cf6e5d723bf2 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ae3a9e2293442e291b1262a717a5e50, 64fd1d8a3cd14e66ac94c4fbcceaf039; no voters: 
I20260709 12:27:53.029119 31835 raft_consensus.cc:2468] T c61f10da93eb4af5a80a204e679f1ee3 P 2ae3a9e2293442e291b1262a717a5e50 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64fd1d8a3cd14e66ac94c4fbcceaf039 in term 0.
W20260709 12:27:53.020260 31947 leader_election.cc:343] T 378f56c32c1e46f99a874bd1adcbcf25 P 07b97783285d4b22b6fd0f1d816ef930 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 64fd1d8a3cd14e66ac94c4fbcceaf039 (127.16.29.66:42163): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:53.034197 31875 leader_election.cc:304] T c61f10da93eb4af5a80a204e679f1ee3 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ae3a9e2293442e291b1262a717a5e50, 64fd1d8a3cd14e66ac94c4fbcceaf039; no voters: 
I20260709 12:27:53.034245 32032 raft_consensus.cc:515] T 89784f886fb545d2871174c1e169fb32 P 07b97783285d4b22b6fd0f1d816ef930 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } } peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } }
I20260709 12:27:53.037757 32031 leader_election.cc:290] T 6a5720f014ed48769eb641989edef3e8 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2ae3a9e2293442e291b1262a717a5e50 (127.16.29.65:44289), 07b97783285d4b22b6fd0f1d816ef930 (127.16.29.67:34419)
I20260709 12:27:53.038623 32031 raft_consensus.cc:2804] T d00d4a3dbb1c435cb565cf6e5d723bf2 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:53.039095 32031 raft_consensus.cc:493] T d00d4a3dbb1c435cb565cf6e5d723bf2 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:53.039501 32031 raft_consensus.cc:3060] T d00d4a3dbb1c435cb565cf6e5d723bf2 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:53.020733 32052 raft_consensus.cc:2804] T 378f56c32c1e46f99a874bd1adcbcf25 P 07b97783285d4b22b6fd0f1d816ef930 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:53.040928 32052 raft_consensus.cc:493] T 378f56c32c1e46f99a874bd1adcbcf25 P 07b97783285d4b22b6fd0f1d816ef930 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:53.041288 32052 raft_consensus.cc:3060] T 378f56c32c1e46f99a874bd1adcbcf25 P 07b97783285d4b22b6fd0f1d816ef930 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:53.037806 31910 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "89784f886fb545d2871174c1e169fb32" candidate_uuid: "07b97783285d4b22b6fd0f1d816ef930" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039"
I20260709 12:27:53.037791 32026 ts_tablet_manager.cc:1434] T c61f10da93eb4af5a80a204e679f1ee3 P 2ae3a9e2293442e291b1262a717a5e50: Time spent starting tablet: real 0.100s	user 0.010s	sys 0.000s
I20260709 12:27:53.053057 32054 raft_consensus.cc:2804] T c61f10da93eb4af5a80a204e679f1ee3 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:53.053656 32054 raft_consensus.cc:493] T c61f10da93eb4af5a80a204e679f1ee3 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:53.054015 32054 raft_consensus.cc:3060] T c61f10da93eb4af5a80a204e679f1ee3 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:53.054066 31909 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52322cf19f734f19a4ccfc8389f32b41" candidate_uuid: "2ae3a9e2293442e291b1262a717a5e50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" is_pre_election: true
I20260709 12:27:53.054457 31910 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378f56c32c1e46f99a874bd1adcbcf25" candidate_uuid: "2ae3a9e2293442e291b1262a717a5e50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" is_pre_election: true
W20260709 12:27:53.052878 31949 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:44289, user_credentials={real_user=slave}} blocked reactor thread for 43544.8us
W20260709 12:27:53.055673 31798 leader_election.cc:343] T 52322cf19f734f19a4ccfc8389f32b41 P 2ae3a9e2293442e291b1262a717a5e50 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 64fd1d8a3cd14e66ac94c4fbcceaf039 (127.16.29.66:42163): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:27:53.056897 31798 leader_election.cc:343] T 378f56c32c1e46f99a874bd1adcbcf25 P 2ae3a9e2293442e291b1262a717a5e50 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 64fd1d8a3cd14e66ac94c4fbcceaf039 (127.16.29.66:42163): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:53.057018 31835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "89784f886fb545d2871174c1e169fb32" candidate_uuid: "07b97783285d4b22b6fd0f1d816ef930" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ae3a9e2293442e291b1262a717a5e50"
I20260709 12:27:53.057857 31835 raft_consensus.cc:3060] T 89784f886fb545d2871174c1e169fb32 P 2ae3a9e2293442e291b1262a717a5e50 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:53.058202 32026 tablet_bootstrap.cc:492] T 6a5720f014ed48769eb641989edef3e8 P 2ae3a9e2293442e291b1262a717a5e50: Bootstrap starting.
I20260709 12:27:53.066274 31983 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c61f10da93eb4af5a80a204e679f1ee3" candidate_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07b97783285d4b22b6fd0f1d816ef930" is_pre_election: true
I20260709 12:27:53.067096 31983 raft_consensus.cc:2468] T c61f10da93eb4af5a80a204e679f1ee3 P 07b97783285d4b22b6fd0f1d816ef930 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64fd1d8a3cd14e66ac94c4fbcceaf039 in term 0.
I20260709 12:27:53.068423 31984 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378f56c32c1e46f99a874bd1adcbcf25" candidate_uuid: "2ae3a9e2293442e291b1262a717a5e50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07b97783285d4b22b6fd0f1d816ef930" is_pre_election: true
I20260709 12:27:53.069482 31982 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d00d4a3dbb1c435cb565cf6e5d723bf2" candidate_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07b97783285d4b22b6fd0f1d816ef930" is_pre_election: true
I20260709 12:27:53.069844 32024 tablet_bootstrap.cc:492] T 1638e215a9ee4068a999e92023fff415 P 07b97783285d4b22b6fd0f1d816ef930: No bootstrap required, opened a new log
I20260709 12:27:53.070250 31982 raft_consensus.cc:2468] T d00d4a3dbb1c435cb565cf6e5d723bf2 P 07b97783285d4b22b6fd0f1d816ef930 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64fd1d8a3cd14e66ac94c4fbcceaf039 in term 0.
I20260709 12:27:53.070647 32024 ts_tablet_manager.cc:1403] T 1638e215a9ee4068a999e92023fff415 P 07b97783285d4b22b6fd0f1d816ef930: Time spent bootstrapping tablet: real 0.133s	user 0.017s	sys 0.012s
I20260709 12:27:53.070222 31981 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52322cf19f734f19a4ccfc8389f32b41" candidate_uuid: "2ae3a9e2293442e291b1262a717a5e50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07b97783285d4b22b6fd0f1d816ef930" is_pre_election: true
I20260709 12:27:53.071580 31981 raft_consensus.cc:2468] T 52322cf19f734f19a4ccfc8389f32b41 P 07b97783285d4b22b6fd0f1d816ef930 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ae3a9e2293442e291b1262a717a5e50 in term 0.
I20260709 12:27:53.072121 32032 leader_election.cc:290] T 89784f886fb545d2871174c1e169fb32 P 07b97783285d4b22b6fd0f1d816ef930 [CANDIDATE]: Term 1 election: Requested vote from peers 2ae3a9e2293442e291b1262a717a5e50 (127.16.29.65:44289), 64fd1d8a3cd14e66ac94c4fbcceaf039 (127.16.29.66:42163)
I20260709 12:27:53.074131 31981 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6a5720f014ed48769eb641989edef3e8" candidate_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07b97783285d4b22b6fd0f1d816ef930" is_pre_election: true
W20260709 12:27:53.076122 31876 leader_election.cc:343] T 6a5720f014ed48769eb641989edef3e8 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 07b97783285d4b22b6fd0f1d816ef930 (127.16.29.67:34419): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:53.089025 31835 raft_consensus.cc:2468] T 89784f886fb545d2871174c1e169fb32 P 2ae3a9e2293442e291b1262a717a5e50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07b97783285d4b22b6fd0f1d816ef930 in term 1.
I20260709 12:27:53.076740 31801 leader_election.cc:304] T 52322cf19f734f19a4ccfc8389f32b41 P 2ae3a9e2293442e291b1262a717a5e50 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 07b97783285d4b22b6fd0f1d816ef930, 2ae3a9e2293442e291b1262a717a5e50; no voters: 64fd1d8a3cd14e66ac94c4fbcceaf039
I20260709 12:27:53.089048 32031 raft_consensus.cc:515] T d00d4a3dbb1c435cb565cf6e5d723bf2 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:53.090758 32046 raft_consensus.cc:2804] T 52322cf19f734f19a4ccfc8389f32b41 P 2ae3a9e2293442e291b1262a717a5e50 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:53.091295 32046 raft_consensus.cc:493] T 52322cf19f734f19a4ccfc8389f32b41 P 2ae3a9e2293442e291b1262a717a5e50 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:53.091753 32046 raft_consensus.cc:3060] T 52322cf19f734f19a4ccfc8389f32b41 P 2ae3a9e2293442e291b1262a717a5e50 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:53.094832 32058 raft_consensus.cc:493] T fad0b7ef0f804aa494f44e9ec456e11e P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:53.095453 32058 raft_consensus.cc:515] T fad0b7ef0f804aa494f44e9ec456e11e P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:53.099181 31981 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fad0b7ef0f804aa494f44e9ec456e11e" candidate_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07b97783285d4b22b6fd0f1d816ef930" is_pre_election: true
W20260709 12:27:53.100693 31876 leader_election.cc:343] T fad0b7ef0f804aa494f44e9ec456e11e P 64fd1d8a3cd14e66ac94c4fbcceaf039 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 07b97783285d4b22b6fd0f1d816ef930 (127.16.29.67:34419): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:27:53.102416 31875 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:44289, user_credentials={real_user=slave}} blocked reactor thread for 68307.9us
I20260709 12:27:53.102669 32057 raft_consensus.cc:493] T 1638e215a9ee4068a999e92023fff415 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:53.092517 31949 leader_election.cc:304] T 89784f886fb545d2871174c1e169fb32 P 07b97783285d4b22b6fd0f1d816ef930 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07b97783285d4b22b6fd0f1d816ef930, 2ae3a9e2293442e291b1262a717a5e50; no voters: 
I20260709 12:27:53.103546 32057 raft_consensus.cc:515] T 1638e215a9ee4068a999e92023fff415 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
W20260709 12:27:53.104514 31947 leader_election.cc:343] T 89784f886fb545d2871174c1e169fb32 P 07b97783285d4b22b6fd0f1d816ef930 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 64fd1d8a3cd14e66ac94c4fbcceaf039 (127.16.29.66:42163): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:53.104626 32032 raft_consensus.cc:2804] T 89784f886fb545d2871174c1e169fb32 P 07b97783285d4b22b6fd0f1d816ef930 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:53.105355 32032 raft_consensus.cc:697] T 89784f886fb545d2871174c1e169fb32 P 07b97783285d4b22b6fd0f1d816ef930 [term 1 LEADER]: Becoming Leader. State: Replica: 07b97783285d4b22b6fd0f1d816ef930, State: Running, Role: LEADER
I20260709 12:27:53.106730 32032 consensus_queue.cc:237] T 89784f886fb545d2871174c1e169fb32 P 07b97783285d4b22b6fd0f1d816ef930 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } } peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } }
I20260709 12:27:53.108726 31981 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1638e215a9ee4068a999e92023fff415" candidate_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07b97783285d4b22b6fd0f1d816ef930" is_pre_election: true
I20260709 12:27:53.108475 32046 raft_consensus.cc:515] T 52322cf19f734f19a4ccfc8389f32b41 P 2ae3a9e2293442e291b1262a717a5e50 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:53.089149 32024 raft_consensus.cc:359] T 1638e215a9ee4068a999e92023fff415 P 07b97783285d4b22b6fd0f1d816ef930 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:53.110520 32025 tablet_bootstrap.cc:654] T 378f56c32c1e46f99a874bd1adcbcf25 P 64fd1d8a3cd14e66ac94c4fbcceaf039: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:53.114976 32026 tablet_bootstrap.cc:654] T 6a5720f014ed48769eb641989edef3e8 P 2ae3a9e2293442e291b1262a717a5e50: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:53.112349 32052 raft_consensus.cc:515] T 378f56c32c1e46f99a874bd1adcbcf25 P 07b97783285d4b22b6fd0f1d816ef930 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:53.117398 31984 raft_consensus.cc:2393] T 378f56c32c1e46f99a874bd1adcbcf25 P 07b97783285d4b22b6fd0f1d816ef930 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2ae3a9e2293442e291b1262a717a5e50 in current term 1: Already voted for candidate 07b97783285d4b22b6fd0f1d816ef930 in this term.
I20260709 12:27:53.114019 31833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1638e215a9ee4068a999e92023fff415" candidate_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ae3a9e2293442e291b1262a717a5e50" is_pre_election: true
I20260709 12:27:53.118569 32052 leader_election.cc:290] T 378f56c32c1e46f99a874bd1adcbcf25 P 07b97783285d4b22b6fd0f1d816ef930 [CANDIDATE]: Term 1 election: Requested vote from peers 2ae3a9e2293442e291b1262a717a5e50 (127.16.29.65:44289), 64fd1d8a3cd14e66ac94c4fbcceaf039 (127.16.29.66:42163)
I20260709 12:27:53.114971 31910 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52322cf19f734f19a4ccfc8389f32b41" candidate_uuid: "2ae3a9e2293442e291b1262a717a5e50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039"
I20260709 12:27:53.111181 31834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fad0b7ef0f804aa494f44e9ec456e11e" candidate_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ae3a9e2293442e291b1262a717a5e50" is_pre_election: true
I20260709 12:27:53.113153 32024 raft_consensus.cc:385] T 1638e215a9ee4068a999e92023fff415 P 07b97783285d4b22b6fd0f1d816ef930 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:53.120412 31801 leader_election.cc:304] T 378f56c32c1e46f99a874bd1adcbcf25 P 2ae3a9e2293442e291b1262a717a5e50 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2ae3a9e2293442e291b1262a717a5e50; no voters: 07b97783285d4b22b6fd0f1d816ef930, 64fd1d8a3cd14e66ac94c4fbcceaf039
I20260709 12:27:53.120565 31909 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378f56c32c1e46f99a874bd1adcbcf25" candidate_uuid: "07b97783285d4b22b6fd0f1d816ef930" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039"
I20260709 12:27:53.120627 32024 raft_consensus.cc:740] T 1638e215a9ee4068a999e92023fff415 P 07b97783285d4b22b6fd0f1d816ef930 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07b97783285d4b22b6fd0f1d816ef930, State: Initialized, Role: FOLLOWER
W20260709 12:27:53.120924 31798 leader_election.cc:343] T 52322cf19f734f19a4ccfc8389f32b41 P 2ae3a9e2293442e291b1262a717a5e50 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 64fd1d8a3cd14e66ac94c4fbcceaf039 (127.16.29.66:42163): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:53.121512 32030 raft_consensus.cc:3060] T 378f56c32c1e46f99a874bd1adcbcf25 P 2ae3a9e2293442e291b1262a717a5e50 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:53.110944 31835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6a5720f014ed48769eb641989edef3e8" candidate_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ae3a9e2293442e291b1262a717a5e50" is_pre_election: true
W20260709 12:27:53.122184 31947 leader_election.cc:343] T 378f56c32c1e46f99a874bd1adcbcf25 P 07b97783285d4b22b6fd0f1d816ef930 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 64fd1d8a3cd14e66ac94c4fbcceaf039 (127.16.29.66:42163): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:53.121909 32024 consensus_queue.cc:260] T 1638e215a9ee4068a999e92023fff415 P 07b97783285d4b22b6fd0f1d816ef930 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:53.109553 32058 leader_election.cc:290] T fad0b7ef0f804aa494f44e9ec456e11e P 64fd1d8a3cd14e66ac94c4fbcceaf039 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2ae3a9e2293442e291b1262a717a5e50 (127.16.29.65:44289), 07b97783285d4b22b6fd0f1d816ef930 (127.16.29.67:34419)
I20260709 12:27:53.123625 31981 raft_consensus.cc:2468] T 1638e215a9ee4068a999e92023fff415 P 07b97783285d4b22b6fd0f1d816ef930 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64fd1d8a3cd14e66ac94c4fbcceaf039 in term 0.
I20260709 12:27:53.111560 32057 leader_election.cc:290] T 1638e215a9ee4068a999e92023fff415 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2ae3a9e2293442e291b1262a717a5e50 (127.16.29.65:44289), 07b97783285d4b22b6fd0f1d816ef930 (127.16.29.67:34419)
I20260709 12:27:53.126266 32046 leader_election.cc:290] T 52322cf19f734f19a4ccfc8389f32b41 P 2ae3a9e2293442e291b1262a717a5e50 [CANDIDATE]: Term 1 election: Requested vote from peers 64fd1d8a3cd14e66ac94c4fbcceaf039 (127.16.29.66:42163), 07b97783285d4b22b6fd0f1d816ef930 (127.16.29.67:34419)
I20260709 12:27:53.126060 31876 leader_election.cc:304] T 1638e215a9ee4068a999e92023fff415 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07b97783285d4b22b6fd0f1d816ef930, 64fd1d8a3cd14e66ac94c4fbcceaf039; no voters: 
I20260709 12:27:53.127271 32057 raft_consensus.cc:2804] T 1638e215a9ee4068a999e92023fff415 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:53.127645 32057 raft_consensus.cc:493] T 1638e215a9ee4068a999e92023fff415 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:53.128027 32057 raft_consensus.cc:3060] T 1638e215a9ee4068a999e92023fff415 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:27:53.128392 31875 leader_election.cc:343] T 1638e215a9ee4068a999e92023fff415 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2ae3a9e2293442e291b1262a717a5e50 (127.16.29.65:44289): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:53.133172 32024 ts_tablet_manager.cc:1434] T 1638e215a9ee4068a999e92023fff415 P 07b97783285d4b22b6fd0f1d816ef930: Time spent starting tablet: real 0.062s	user 0.013s	sys 0.003s
I20260709 12:27:53.134624 32024 tablet_bootstrap.cc:492] T 0fb83fa22e13487f9d56facc914278ba P 07b97783285d4b22b6fd0f1d816ef930: Bootstrap starting.
I20260709 12:27:53.138806 32030 raft_consensus.cc:2749] T 378f56c32c1e46f99a874bd1adcbcf25 P 2ae3a9e2293442e291b1262a717a5e50 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:53.111460 32054 raft_consensus.cc:515] T c61f10da93eb4af5a80a204e679f1ee3 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
W20260709 12:27:53.145931 31875 leader_election.cc:343] T fad0b7ef0f804aa494f44e9ec456e11e P 64fd1d8a3cd14e66ac94c4fbcceaf039 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2ae3a9e2293442e291b1262a717a5e50 (127.16.29.65:44289): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:53.146528 31875 leader_election.cc:304] T fad0b7ef0f804aa494f44e9ec456e11e P 64fd1d8a3cd14e66ac94c4fbcceaf039 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 64fd1d8a3cd14e66ac94c4fbcceaf039; no voters: 07b97783285d4b22b6fd0f1d816ef930, 2ae3a9e2293442e291b1262a717a5e50
W20260709 12:27:53.147787 31875 leader_election.cc:343] T 6a5720f014ed48769eb641989edef3e8 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2ae3a9e2293442e291b1262a717a5e50 (127.16.29.65:44289): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:53.148240 31981 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d00d4a3dbb1c435cb565cf6e5d723bf2" candidate_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07b97783285d4b22b6fd0f1d816ef930"
I20260709 12:27:53.148269 31875 leader_election.cc:304] T 6a5720f014ed48769eb641989edef3e8 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 64fd1d8a3cd14e66ac94c4fbcceaf039; no voters: 07b97783285d4b22b6fd0f1d816ef930, 2ae3a9e2293442e291b1262a717a5e50
I20260709 12:27:53.148950 31981 raft_consensus.cc:3060] T d00d4a3dbb1c435cb565cf6e5d723bf2 P 07b97783285d4b22b6fd0f1d816ef930 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:53.150522 32058 raft_consensus.cc:2749] T fad0b7ef0f804aa494f44e9ec456e11e P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:53.152451 32058 raft_consensus.cc:2749] T 6a5720f014ed48769eb641989edef3e8 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:53.154963 31984 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c61f10da93eb4af5a80a204e679f1ee3" candidate_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07b97783285d4b22b6fd0f1d816ef930"
I20260709 12:27:53.155751 31984 raft_consensus.cc:3060] T c61f10da93eb4af5a80a204e679f1ee3 P 07b97783285d4b22b6fd0f1d816ef930 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:53.156641 31731 catalog_manager.cc:5697] T 89784f886fb545d2871174c1e169fb32 P 07b97783285d4b22b6fd0f1d816ef930 reported cstate change: term changed from 0 to 1, leader changed from <none> to 07b97783285d4b22b6fd0f1d816ef930 (127.16.29.67). New cstate: current_term: 1 leader_uuid: "07b97783285d4b22b6fd0f1d816ef930" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:53.162508 32026 tablet_bootstrap.cc:492] T 6a5720f014ed48769eb641989edef3e8 P 2ae3a9e2293442e291b1262a717a5e50: No bootstrap required, opened a new log
I20260709 12:27:53.159292 32054 leader_election.cc:290] T c61f10da93eb4af5a80a204e679f1ee3 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [CANDIDATE]: Term 1 election: Requested vote from peers 2ae3a9e2293442e291b1262a717a5e50 (127.16.29.65:44289), 07b97783285d4b22b6fd0f1d816ef930 (127.16.29.67:34419)
I20260709 12:27:53.163134 32026 ts_tablet_manager.cc:1403] T 6a5720f014ed48769eb641989edef3e8 P 2ae3a9e2293442e291b1262a717a5e50: Time spent bootstrapping tablet: real 0.116s	user 0.014s	sys 0.017s
I20260709 12:27:53.166603 32026 raft_consensus.cc:359] T 6a5720f014ed48769eb641989edef3e8 P 2ae3a9e2293442e291b1262a717a5e50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:53.150614 32031 leader_election.cc:290] T d00d4a3dbb1c435cb565cf6e5d723bf2 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [CANDIDATE]: Term 1 election: Requested vote from peers 2ae3a9e2293442e291b1262a717a5e50 (127.16.29.65:44289), 07b97783285d4b22b6fd0f1d816ef930 (127.16.29.67:34419)
I20260709 12:27:53.167392 32026 raft_consensus.cc:385] T 6a5720f014ed48769eb641989edef3e8 P 2ae3a9e2293442e291b1262a717a5e50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:53.166991 32024 tablet_bootstrap.cc:654] T 0fb83fa22e13487f9d56facc914278ba P 07b97783285d4b22b6fd0f1d816ef930: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:53.167827 31981 raft_consensus.cc:2468] T d00d4a3dbb1c435cb565cf6e5d723bf2 P 07b97783285d4b22b6fd0f1d816ef930 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64fd1d8a3cd14e66ac94c4fbcceaf039 in term 1.
I20260709 12:27:53.170531 31876 leader_election.cc:304] T d00d4a3dbb1c435cb565cf6e5d723bf2 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07b97783285d4b22b6fd0f1d816ef930, 64fd1d8a3cd14e66ac94c4fbcceaf039; no voters: 
I20260709 12:27:53.168692 32026 raft_consensus.cc:740] T 6a5720f014ed48769eb641989edef3e8 P 2ae3a9e2293442e291b1262a717a5e50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ae3a9e2293442e291b1262a717a5e50, State: Initialized, Role: FOLLOWER
I20260709 12:27:53.172451 31835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378f56c32c1e46f99a874bd1adcbcf25" candidate_uuid: "07b97783285d4b22b6fd0f1d816ef930" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ae3a9e2293442e291b1262a717a5e50"
I20260709 12:27:53.172667 32026 consensus_queue.cc:260] T 6a5720f014ed48769eb641989edef3e8 P 2ae3a9e2293442e291b1262a717a5e50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:53.174561 32025 tablet_bootstrap.cc:492] T 378f56c32c1e46f99a874bd1adcbcf25 P 64fd1d8a3cd14e66ac94c4fbcceaf039: No bootstrap required, opened a new log
I20260709 12:27:53.175199 32025 ts_tablet_manager.cc:1403] T 378f56c32c1e46f99a874bd1adcbcf25 P 64fd1d8a3cd14e66ac94c4fbcceaf039: Time spent bootstrapping tablet: real 0.163s	user 0.020s	sys 0.012s
I20260709 12:27:53.175453 32026 ts_tablet_manager.cc:1434] T 6a5720f014ed48769eb641989edef3e8 P 2ae3a9e2293442e291b1262a717a5e50: Time spent starting tablet: real 0.012s	user 0.009s	sys 0.000s
I20260709 12:27:53.175976 32031 raft_consensus.cc:2804] T d00d4a3dbb1c435cb565cf6e5d723bf2 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:53.176597 32031 raft_consensus.cc:697] T d00d4a3dbb1c435cb565cf6e5d723bf2 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 1 LEADER]: Becoming Leader. State: Replica: 64fd1d8a3cd14e66ac94c4fbcceaf039, State: Running, Role: LEADER
I20260709 12:27:53.177327 32026 tablet_bootstrap.cc:492] T fad0b7ef0f804aa494f44e9ec456e11e P 2ae3a9e2293442e291b1262a717a5e50: Bootstrap starting.
I20260709 12:27:53.177541 32031 consensus_queue.cc:237] T d00d4a3dbb1c435cb565cf6e5d723bf2 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:53.178542 31984 raft_consensus.cc:2468] T c61f10da93eb4af5a80a204e679f1ee3 P 07b97783285d4b22b6fd0f1d816ef930 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64fd1d8a3cd14e66ac94c4fbcceaf039 in term 1.
I20260709 12:27:53.180116 31876 leader_election.cc:304] T c61f10da93eb4af5a80a204e679f1ee3 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07b97783285d4b22b6fd0f1d816ef930, 64fd1d8a3cd14e66ac94c4fbcceaf039; no voters: 
I20260709 12:27:53.181001 32054 raft_consensus.cc:2804] T c61f10da93eb4af5a80a204e679f1ee3 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:53.181500 32054 raft_consensus.cc:697] T c61f10da93eb4af5a80a204e679f1ee3 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 1 LEADER]: Becoming Leader. State: Replica: 64fd1d8a3cd14e66ac94c4fbcceaf039, State: Running, Role: LEADER
I20260709 12:27:53.145144 32057 raft_consensus.cc:515] T 1638e215a9ee4068a999e92023fff415 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:53.182402 32054 consensus_queue.cc:237] T c61f10da93eb4af5a80a204e679f1ee3 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:53.186666 31981 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52322cf19f734f19a4ccfc8389f32b41" candidate_uuid: "2ae3a9e2293442e291b1262a717a5e50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07b97783285d4b22b6fd0f1d816ef930"
I20260709 12:27:53.187448 31981 raft_consensus.cc:3060] T 52322cf19f734f19a4ccfc8389f32b41 P 07b97783285d4b22b6fd0f1d816ef930 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:53.190261 31834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d00d4a3dbb1c435cb565cf6e5d723bf2" candidate_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ae3a9e2293442e291b1262a717a5e50"
I20260709 12:27:53.198443 32026 tablet_bootstrap.cc:654] T fad0b7ef0f804aa494f44e9ec456e11e P 2ae3a9e2293442e291b1262a717a5e50: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:53.198724 31834 raft_consensus.cc:3060] T d00d4a3dbb1c435cb565cf6e5d723bf2 P 2ae3a9e2293442e291b1262a717a5e50 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:53.192088 31984 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1638e215a9ee4068a999e92023fff415" candidate_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07b97783285d4b22b6fd0f1d816ef930"
I20260709 12:27:53.206475 31984 raft_consensus.cc:3060] T 1638e215a9ee4068a999e92023fff415 P 07b97783285d4b22b6fd0f1d816ef930 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:53.191274 31832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1638e215a9ee4068a999e92023fff415" candidate_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ae3a9e2293442e291b1262a717a5e50"
I20260709 12:27:53.190631 31833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c61f10da93eb4af5a80a204e679f1ee3" candidate_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ae3a9e2293442e291b1262a717a5e50"
I20260709 12:27:53.213018 31833 raft_consensus.cc:3060] T c61f10da93eb4af5a80a204e679f1ee3 P 2ae3a9e2293442e291b1262a717a5e50 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:53.209512 32024 tablet_bootstrap.cc:492] T 0fb83fa22e13487f9d56facc914278ba P 07b97783285d4b22b6fd0f1d816ef930: No bootstrap required, opened a new log
I20260709 12:27:53.211476 32025 raft_consensus.cc:359] T 378f56c32c1e46f99a874bd1adcbcf25 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:53.212392 31835 raft_consensus.cc:2468] T 378f56c32c1e46f99a874bd1adcbcf25 P 2ae3a9e2293442e291b1262a717a5e50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07b97783285d4b22b6fd0f1d816ef930 in term 1.
I20260709 12:27:53.215910 32024 ts_tablet_manager.cc:1403] T 0fb83fa22e13487f9d56facc914278ba P 07b97783285d4b22b6fd0f1d816ef930: Time spent bootstrapping tablet: real 0.082s	user 0.003s	sys 0.023s
W20260709 12:27:53.214902 31875 leader_election.cc:343] T 1638e215a9ee4068a999e92023fff415 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 2ae3a9e2293442e291b1262a717a5e50 (127.16.29.65:44289): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:53.216581 32025 raft_consensus.cc:385] T 378f56c32c1e46f99a874bd1adcbcf25 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:53.216958 32025 raft_consensus.cc:740] T 378f56c32c1e46f99a874bd1adcbcf25 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64fd1d8a3cd14e66ac94c4fbcceaf039, State: Initialized, Role: FOLLOWER
I20260709 12:27:53.216806 31949 leader_election.cc:304] T 378f56c32c1e46f99a874bd1adcbcf25 P 07b97783285d4b22b6fd0f1d816ef930 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 07b97783285d4b22b6fd0f1d816ef930, 2ae3a9e2293442e291b1262a717a5e50; no voters: 64fd1d8a3cd14e66ac94c4fbcceaf039
I20260709 12:27:53.218334 32032 raft_consensus.cc:2804] T 378f56c32c1e46f99a874bd1adcbcf25 P 07b97783285d4b22b6fd0f1d816ef930 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:53.218091 32025 consensus_queue.cc:260] T 378f56c32c1e46f99a874bd1adcbcf25 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:53.218911 32032 raft_consensus.cc:697] T 378f56c32c1e46f99a874bd1adcbcf25 P 07b97783285d4b22b6fd0f1d816ef930 [term 1 LEADER]: Becoming Leader. State: Replica: 07b97783285d4b22b6fd0f1d816ef930, State: Running, Role: LEADER
I20260709 12:27:53.191656 32057 leader_election.cc:290] T 1638e215a9ee4068a999e92023fff415 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [CANDIDATE]: Term 1 election: Requested vote from peers 2ae3a9e2293442e291b1262a717a5e50 (127.16.29.65:44289), 07b97783285d4b22b6fd0f1d816ef930 (127.16.29.67:34419)
I20260709 12:27:53.219851 32032 consensus_queue.cc:237] T 378f56c32c1e46f99a874bd1adcbcf25 P 07b97783285d4b22b6fd0f1d816ef930 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:53.224164 32025 ts_tablet_manager.cc:1434] T 378f56c32c1e46f99a874bd1adcbcf25 P 64fd1d8a3cd14e66ac94c4fbcceaf039: Time spent starting tablet: real 0.049s	user 0.011s	sys 0.000s
I20260709 12:27:53.225381 32025 tablet_bootstrap.cc:492] T 52322cf19f734f19a4ccfc8389f32b41 P 64fd1d8a3cd14e66ac94c4fbcceaf039: Bootstrap starting.
I20260709 12:27:53.226974 31834 raft_consensus.cc:2468] T d00d4a3dbb1c435cb565cf6e5d723bf2 P 2ae3a9e2293442e291b1262a717a5e50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64fd1d8a3cd14e66ac94c4fbcceaf039 in term 1.
I20260709 12:27:53.227042 32024 raft_consensus.cc:359] T 0fb83fa22e13487f9d56facc914278ba P 07b97783285d4b22b6fd0f1d816ef930 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:53.228564 32024 raft_consensus.cc:385] T 0fb83fa22e13487f9d56facc914278ba P 07b97783285d4b22b6fd0f1d816ef930 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:53.228927 32024 raft_consensus.cc:740] T 0fb83fa22e13487f9d56facc914278ba P 07b97783285d4b22b6fd0f1d816ef930 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07b97783285d4b22b6fd0f1d816ef930, State: Initialized, Role: FOLLOWER
I20260709 12:27:53.230099 32024 consensus_queue.cc:260] T 0fb83fa22e13487f9d56facc914278ba P 07b97783285d4b22b6fd0f1d816ef930 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:53.237505 32024 ts_tablet_manager.cc:1434] T 0fb83fa22e13487f9d56facc914278ba P 07b97783285d4b22b6fd0f1d816ef930: Time spent starting tablet: real 0.021s	user 0.008s	sys 0.000s
I20260709 12:27:53.239228 31733 catalog_manager.cc:5697] T d00d4a3dbb1c435cb565cf6e5d723bf2 P 64fd1d8a3cd14e66ac94c4fbcceaf039 reported cstate change: term changed from 0 to 1, leader changed from <none> to 64fd1d8a3cd14e66ac94c4fbcceaf039 (127.16.29.66). New cstate: current_term: 1 leader_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:53.240355 31833 raft_consensus.cc:2468] T c61f10da93eb4af5a80a204e679f1ee3 P 2ae3a9e2293442e291b1262a717a5e50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64fd1d8a3cd14e66ac94c4fbcceaf039 in term 1.
I20260709 12:27:53.242517 32024 tablet_bootstrap.cc:492] T 6a5720f014ed48769eb641989edef3e8 P 07b97783285d4b22b6fd0f1d816ef930: Bootstrap starting.
I20260709 12:27:53.245157 32025 tablet_bootstrap.cc:654] T 52322cf19f734f19a4ccfc8389f32b41 P 64fd1d8a3cd14e66ac94c4fbcceaf039: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:53.249987 31981 raft_consensus.cc:2468] T 52322cf19f734f19a4ccfc8389f32b41 P 07b97783285d4b22b6fd0f1d816ef930 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ae3a9e2293442e291b1262a717a5e50 in term 1.
I20260709 12:27:53.250429 32024 tablet_bootstrap.cc:654] T 6a5720f014ed48769eb641989edef3e8 P 07b97783285d4b22b6fd0f1d816ef930: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:53.251331 32026 tablet_bootstrap.cc:492] T fad0b7ef0f804aa494f44e9ec456e11e P 2ae3a9e2293442e291b1262a717a5e50: No bootstrap required, opened a new log
I20260709 12:27:53.251952 32026 ts_tablet_manager.cc:1403] T fad0b7ef0f804aa494f44e9ec456e11e P 2ae3a9e2293442e291b1262a717a5e50: Time spent bootstrapping tablet: real 0.076s	user 0.017s	sys 0.004s
I20260709 12:27:53.251655 31801 leader_election.cc:304] T 52322cf19f734f19a4ccfc8389f32b41 P 2ae3a9e2293442e291b1262a717a5e50 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 07b97783285d4b22b6fd0f1d816ef930, 2ae3a9e2293442e291b1262a717a5e50; no voters: 64fd1d8a3cd14e66ac94c4fbcceaf039
I20260709 12:27:53.253264 32046 raft_consensus.cc:2804] T 52322cf19f734f19a4ccfc8389f32b41 P 2ae3a9e2293442e291b1262a717a5e50 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:53.253876 32046 raft_consensus.cc:697] T 52322cf19f734f19a4ccfc8389f32b41 P 2ae3a9e2293442e291b1262a717a5e50 [term 1 LEADER]: Becoming Leader. State: Replica: 2ae3a9e2293442e291b1262a717a5e50, State: Running, Role: LEADER
I20260709 12:27:53.255002 32046 consensus_queue.cc:237] T 52322cf19f734f19a4ccfc8389f32b41 P 2ae3a9e2293442e291b1262a717a5e50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:53.256340 32026 raft_consensus.cc:359] T fad0b7ef0f804aa494f44e9ec456e11e P 2ae3a9e2293442e291b1262a717a5e50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:53.257035 32026 raft_consensus.cc:385] T fad0b7ef0f804aa494f44e9ec456e11e P 2ae3a9e2293442e291b1262a717a5e50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:53.257400 32026 raft_consensus.cc:740] T fad0b7ef0f804aa494f44e9ec456e11e P 2ae3a9e2293442e291b1262a717a5e50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ae3a9e2293442e291b1262a717a5e50, State: Initialized, Role: FOLLOWER
I20260709 12:27:53.258163 32026 consensus_queue.cc:260] T fad0b7ef0f804aa494f44e9ec456e11e P 2ae3a9e2293442e291b1262a717a5e50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:53.260694 32026 ts_tablet_manager.cc:1434] T fad0b7ef0f804aa494f44e9ec456e11e P 2ae3a9e2293442e291b1262a717a5e50: Time spent starting tablet: real 0.008s	user 0.002s	sys 0.005s
I20260709 12:27:53.261981 32026 tablet_bootstrap.cc:492] T 1638e215a9ee4068a999e92023fff415 P 2ae3a9e2293442e291b1262a717a5e50: Bootstrap starting.
I20260709 12:27:53.262442 32025 tablet_bootstrap.cc:492] T 52322cf19f734f19a4ccfc8389f32b41 P 64fd1d8a3cd14e66ac94c4fbcceaf039: No bootstrap required, opened a new log
I20260709 12:27:53.263042 32025 ts_tablet_manager.cc:1403] T 52322cf19f734f19a4ccfc8389f32b41 P 64fd1d8a3cd14e66ac94c4fbcceaf039: Time spent bootstrapping tablet: real 0.038s	user 0.010s	sys 0.000s
I20260709 12:27:53.267915 32024 tablet_bootstrap.cc:492] T 6a5720f014ed48769eb641989edef3e8 P 07b97783285d4b22b6fd0f1d816ef930: No bootstrap required, opened a new log
I20260709 12:27:53.268509 32024 ts_tablet_manager.cc:1403] T 6a5720f014ed48769eb641989edef3e8 P 07b97783285d4b22b6fd0f1d816ef930: Time spent bootstrapping tablet: real 0.026s	user 0.014s	sys 0.000s
I20260709 12:27:53.268314 32025 raft_consensus.cc:359] T 52322cf19f734f19a4ccfc8389f32b41 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:53.269121 32025 raft_consensus.cc:385] T 52322cf19f734f19a4ccfc8389f32b41 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:53.269464 32025 raft_consensus.cc:740] T 52322cf19f734f19a4ccfc8389f32b41 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64fd1d8a3cd14e66ac94c4fbcceaf039, State: Initialized, Role: FOLLOWER
I20260709 12:27:53.270524 32026 tablet_bootstrap.cc:654] T 1638e215a9ee4068a999e92023fff415 P 2ae3a9e2293442e291b1262a717a5e50: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:53.269330 31732 catalog_manager.cc:5697] T 52322cf19f734f19a4ccfc8389f32b41 P 2ae3a9e2293442e291b1262a717a5e50 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2ae3a9e2293442e291b1262a717a5e50 (127.16.29.65). New cstate: current_term: 1 leader_uuid: "2ae3a9e2293442e291b1262a717a5e50" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:53.270529 32025 consensus_queue.cc:260] T 52322cf19f734f19a4ccfc8389f32b41 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:53.271597 32024 raft_consensus.cc:359] T 6a5720f014ed48769eb641989edef3e8 P 07b97783285d4b22b6fd0f1d816ef930 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:53.272305 32024 raft_consensus.cc:385] T 6a5720f014ed48769eb641989edef3e8 P 07b97783285d4b22b6fd0f1d816ef930 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:53.272616 32024 raft_consensus.cc:740] T 6a5720f014ed48769eb641989edef3e8 P 07b97783285d4b22b6fd0f1d816ef930 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07b97783285d4b22b6fd0f1d816ef930, State: Initialized, Role: FOLLOWER
I20260709 12:27:53.273569 32025 ts_tablet_manager.cc:1434] T 52322cf19f734f19a4ccfc8389f32b41 P 64fd1d8a3cd14e66ac94c4fbcceaf039: Time spent starting tablet: real 0.010s	user 0.009s	sys 0.000s
I20260709 12:27:53.273569 32024 consensus_queue.cc:260] T 6a5720f014ed48769eb641989edef3e8 P 07b97783285d4b22b6fd0f1d816ef930 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:53.275163 32025 tablet_bootstrap.cc:492] T 0fb83fa22e13487f9d56facc914278ba P 64fd1d8a3cd14e66ac94c4fbcceaf039: Bootstrap starting.
I20260709 12:27:53.276451 32024 ts_tablet_manager.cc:1434] T 6a5720f014ed48769eb641989edef3e8 P 07b97783285d4b22b6fd0f1d816ef930: Time spent starting tablet: real 0.008s	user 0.008s	sys 0.000s
I20260709 12:27:53.277501 32024 tablet_bootstrap.cc:492] T fad0b7ef0f804aa494f44e9ec456e11e P 07b97783285d4b22b6fd0f1d816ef930: Bootstrap starting.
I20260709 12:27:53.282804 32025 tablet_bootstrap.cc:654] T 0fb83fa22e13487f9d56facc914278ba P 64fd1d8a3cd14e66ac94c4fbcceaf039: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:53.282893 31984 raft_consensus.cc:2468] T 1638e215a9ee4068a999e92023fff415 P 07b97783285d4b22b6fd0f1d816ef930 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64fd1d8a3cd14e66ac94c4fbcceaf039 in term 1.
I20260709 12:27:53.283550 32024 tablet_bootstrap.cc:654] T fad0b7ef0f804aa494f44e9ec456e11e P 07b97783285d4b22b6fd0f1d816ef930: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:53.283843 32026 tablet_bootstrap.cc:492] T 1638e215a9ee4068a999e92023fff415 P 2ae3a9e2293442e291b1262a717a5e50: No bootstrap required, opened a new log
I20260709 12:27:53.284715 32026 ts_tablet_manager.cc:1403] T 1638e215a9ee4068a999e92023fff415 P 2ae3a9e2293442e291b1262a717a5e50: Time spent bootstrapping tablet: real 0.023s	user 0.017s	sys 0.003s
I20260709 12:27:53.285168 31876 leader_election.cc:304] T 1638e215a9ee4068a999e92023fff415 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 07b97783285d4b22b6fd0f1d816ef930, 64fd1d8a3cd14e66ac94c4fbcceaf039; no voters: 2ae3a9e2293442e291b1262a717a5e50
I20260709 12:27:53.286463 32031 raft_consensus.cc:2804] T 1638e215a9ee4068a999e92023fff415 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:53.286975 32031 raft_consensus.cc:697] T 1638e215a9ee4068a999e92023fff415 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 1 LEADER]: Becoming Leader. State: Replica: 64fd1d8a3cd14e66ac94c4fbcceaf039, State: Running, Role: LEADER
I20260709 12:27:53.287966 32026 raft_consensus.cc:359] T 1638e215a9ee4068a999e92023fff415 P 2ae3a9e2293442e291b1262a717a5e50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:53.287815 32031 consensus_queue.cc:237] T 1638e215a9ee4068a999e92023fff415 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:53.288854 32026 raft_consensus.cc:385] T 1638e215a9ee4068a999e92023fff415 P 2ae3a9e2293442e291b1262a717a5e50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:53.289206 32026 raft_consensus.cc:740] T 1638e215a9ee4068a999e92023fff415 P 2ae3a9e2293442e291b1262a717a5e50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ae3a9e2293442e291b1262a717a5e50, State: Initialized, Role: FOLLOWER
I20260709 12:27:53.290150 32026 consensus_queue.cc:260] T 1638e215a9ee4068a999e92023fff415 P 2ae3a9e2293442e291b1262a717a5e50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:53.293869 32026 ts_tablet_manager.cc:1434] T 1638e215a9ee4068a999e92023fff415 P 2ae3a9e2293442e291b1262a717a5e50: Time spent starting tablet: real 0.009s	user 0.006s	sys 0.000s
I20260709 12:27:53.295363 32026 tablet_bootstrap.cc:492] T 0fb83fa22e13487f9d56facc914278ba P 2ae3a9e2293442e291b1262a717a5e50: Bootstrap starting.
I20260709 12:27:53.296295 32024 tablet_bootstrap.cc:492] T fad0b7ef0f804aa494f44e9ec456e11e P 07b97783285d4b22b6fd0f1d816ef930: No bootstrap required, opened a new log
I20260709 12:27:53.296555 32025 tablet_bootstrap.cc:492] T 0fb83fa22e13487f9d56facc914278ba P 64fd1d8a3cd14e66ac94c4fbcceaf039: No bootstrap required, opened a new log
I20260709 12:27:53.296835 32024 ts_tablet_manager.cc:1403] T fad0b7ef0f804aa494f44e9ec456e11e P 07b97783285d4b22b6fd0f1d816ef930: Time spent bootstrapping tablet: real 0.020s	user 0.014s	sys 0.004s
I20260709 12:27:53.296986 32025 ts_tablet_manager.cc:1403] T 0fb83fa22e13487f9d56facc914278ba P 64fd1d8a3cd14e66ac94c4fbcceaf039: Time spent bootstrapping tablet: real 0.022s	user 0.009s	sys 0.009s
I20260709 12:27:53.299829 32024 raft_consensus.cc:359] T fad0b7ef0f804aa494f44e9ec456e11e P 07b97783285d4b22b6fd0f1d816ef930 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:53.300173 32025 raft_consensus.cc:359] T 0fb83fa22e13487f9d56facc914278ba P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:53.300758 32024 raft_consensus.cc:385] T fad0b7ef0f804aa494f44e9ec456e11e P 07b97783285d4b22b6fd0f1d816ef930 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:53.300889 32025 raft_consensus.cc:385] T 0fb83fa22e13487f9d56facc914278ba P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:53.301175 32024 raft_consensus.cc:740] T fad0b7ef0f804aa494f44e9ec456e11e P 07b97783285d4b22b6fd0f1d816ef930 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07b97783285d4b22b6fd0f1d816ef930, State: Initialized, Role: FOLLOWER
I20260709 12:27:53.301224 32025 raft_consensus.cc:740] T 0fb83fa22e13487f9d56facc914278ba P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64fd1d8a3cd14e66ac94c4fbcceaf039, State: Initialized, Role: FOLLOWER
I20260709 12:27:53.302220 32024 consensus_queue.cc:260] T fad0b7ef0f804aa494f44e9ec456e11e P 07b97783285d4b22b6fd0f1d816ef930 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:53.302506 32025 consensus_queue.cc:260] T 0fb83fa22e13487f9d56facc914278ba P 64fd1d8a3cd14e66ac94c4fbcceaf039 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:53.304360 32026 tablet_bootstrap.cc:654] T 0fb83fa22e13487f9d56facc914278ba P 2ae3a9e2293442e291b1262a717a5e50: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:53.304718 32024 ts_tablet_manager.cc:1434] T fad0b7ef0f804aa494f44e9ec456e11e P 07b97783285d4b22b6fd0f1d816ef930: Time spent starting tablet: real 0.008s	user 0.008s	sys 0.000s
I20260709 12:27:53.307569 32025 ts_tablet_manager.cc:1434] T 0fb83fa22e13487f9d56facc914278ba P 64fd1d8a3cd14e66ac94c4fbcceaf039: Time spent starting tablet: real 0.010s	user 0.007s	sys 0.000s
I20260709 12:27:53.308669 32025 tablet_bootstrap.cc:492] T 89784f886fb545d2871174c1e169fb32 P 64fd1d8a3cd14e66ac94c4fbcceaf039: Bootstrap starting.
I20260709 12:27:53.310254 32026 tablet_bootstrap.cc:492] T 0fb83fa22e13487f9d56facc914278ba P 2ae3a9e2293442e291b1262a717a5e50: No bootstrap required, opened a new log
I20260709 12:27:53.310811 32026 ts_tablet_manager.cc:1403] T 0fb83fa22e13487f9d56facc914278ba P 2ae3a9e2293442e291b1262a717a5e50: Time spent bootstrapping tablet: real 0.016s	user 0.011s	sys 0.003s
I20260709 12:27:53.313812 32026 raft_consensus.cc:359] T 0fb83fa22e13487f9d56facc914278ba P 2ae3a9e2293442e291b1262a717a5e50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:53.314610 32026 raft_consensus.cc:385] T 0fb83fa22e13487f9d56facc914278ba P 2ae3a9e2293442e291b1262a717a5e50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:53.315522 32026 raft_consensus.cc:740] T 0fb83fa22e13487f9d56facc914278ba P 2ae3a9e2293442e291b1262a717a5e50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ae3a9e2293442e291b1262a717a5e50, State: Initialized, Role: FOLLOWER
I20260709 12:27:53.316390 32026 consensus_queue.cc:260] T 0fb83fa22e13487f9d56facc914278ba P 2ae3a9e2293442e291b1262a717a5e50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:53.318537 32026 ts_tablet_manager.cc:1434] T 0fb83fa22e13487f9d56facc914278ba P 2ae3a9e2293442e291b1262a717a5e50: Time spent starting tablet: real 0.007s	user 0.002s	sys 0.003s
I20260709 12:27:53.320674 32031 raft_consensus.cc:493] T 52322cf19f734f19a4ccfc8389f32b41 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:53.321249 32031 raft_consensus.cc:515] T 52322cf19f734f19a4ccfc8389f32b41 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:53.322398 32025 tablet_bootstrap.cc:654] T 89784f886fb545d2871174c1e169fb32 P 64fd1d8a3cd14e66ac94c4fbcceaf039: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:53.324285 31833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52322cf19f734f19a4ccfc8389f32b41" candidate_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ae3a9e2293442e291b1262a717a5e50" is_pre_election: true
I20260709 12:27:53.325042 31733 catalog_manager.cc:5697] T 378f56c32c1e46f99a874bd1adcbcf25 P 07b97783285d4b22b6fd0f1d816ef930 reported cstate change: term changed from 0 to 1, leader changed from <none> to 07b97783285d4b22b6fd0f1d816ef930 (127.16.29.67). New cstate: current_term: 1 leader_uuid: "07b97783285d4b22b6fd0f1d816ef930" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } health_report { overall_health: HEALTHY } } }
I20260709 12:27:53.328461 32025 tablet_bootstrap.cc:492] T 89784f886fb545d2871174c1e169fb32 P 64fd1d8a3cd14e66ac94c4fbcceaf039: No bootstrap required, opened a new log
I20260709 12:27:53.329003 32025 ts_tablet_manager.cc:1403] T 89784f886fb545d2871174c1e169fb32 P 64fd1d8a3cd14e66ac94c4fbcceaf039: Time spent bootstrapping tablet: real 0.021s	user 0.010s	sys 0.001s
I20260709 12:27:53.329298 32031 leader_election.cc:290] T 52322cf19f734f19a4ccfc8389f32b41 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2ae3a9e2293442e291b1262a717a5e50 (127.16.29.65:44289), 07b97783285d4b22b6fd0f1d816ef930 (127.16.29.67:34419)
I20260709 12:27:53.330461 31984 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52322cf19f734f19a4ccfc8389f32b41" candidate_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07b97783285d4b22b6fd0f1d816ef930" is_pre_election: true
I20260709 12:27:53.331154 31984 raft_consensus.cc:2393] T 52322cf19f734f19a4ccfc8389f32b41 P 07b97783285d4b22b6fd0f1d816ef930 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 64fd1d8a3cd14e66ac94c4fbcceaf039 in current term 1: Already voted for candidate 2ae3a9e2293442e291b1262a717a5e50 in this term.
I20260709 12:27:53.332064 31876 leader_election.cc:304] T 52322cf19f734f19a4ccfc8389f32b41 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 64fd1d8a3cd14e66ac94c4fbcceaf039; no voters: 07b97783285d4b22b6fd0f1d816ef930, 2ae3a9e2293442e291b1262a717a5e50
I20260709 12:27:53.331911 32025 raft_consensus.cc:359] T 89784f886fb545d2871174c1e169fb32 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } } peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } }
I20260709 12:27:53.332769 32025 raft_consensus.cc:385] T 89784f886fb545d2871174c1e169fb32 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:53.332976 32031 raft_consensus.cc:3060] T 52322cf19f734f19a4ccfc8389f32b41 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:53.333088 32025 raft_consensus.cc:740] T 89784f886fb545d2871174c1e169fb32 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64fd1d8a3cd14e66ac94c4fbcceaf039, State: Initialized, Role: FOLLOWER
I20260709 12:27:53.333904 32025 consensus_queue.cc:260] T 89784f886fb545d2871174c1e169fb32 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } } peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } }
I20260709 12:27:53.336021 32025 ts_tablet_manager.cc:1434] T 89784f886fb545d2871174c1e169fb32 P 64fd1d8a3cd14e66ac94c4fbcceaf039: Time spent starting tablet: real 0.007s	user 0.008s	sys 0.000s
I20260709 12:27:53.339659 31731 catalog_manager.cc:5697] T c61f10da93eb4af5a80a204e679f1ee3 P 64fd1d8a3cd14e66ac94c4fbcceaf039 reported cstate change: term changed from 0 to 1, leader changed from <none> to 64fd1d8a3cd14e66ac94c4fbcceaf039 (127.16.29.66). New cstate: current_term: 1 leader_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:53.341915 32031 raft_consensus.cc:2749] T 52322cf19f734f19a4ccfc8389f32b41 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:53.341774 31731 catalog_manager.cc:5697] T 1638e215a9ee4068a999e92023fff415 P 64fd1d8a3cd14e66ac94c4fbcceaf039 reported cstate change: term changed from 0 to 1, leader changed from <none> to 64fd1d8a3cd14e66ac94c4fbcceaf039 (127.16.29.66). New cstate: current_term: 1 leader_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:53.442055 32046 raft_consensus.cc:493] T fad0b7ef0f804aa494f44e9ec456e11e P 2ae3a9e2293442e291b1262a717a5e50 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:53.442754 32046 raft_consensus.cc:515] T fad0b7ef0f804aa494f44e9ec456e11e P 2ae3a9e2293442e291b1262a717a5e50 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:53.444700 32046 leader_election.cc:290] T fad0b7ef0f804aa494f44e9ec456e11e P 2ae3a9e2293442e291b1262a717a5e50 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 64fd1d8a3cd14e66ac94c4fbcceaf039 (127.16.29.66:42163), 07b97783285d4b22b6fd0f1d816ef930 (127.16.29.67:34419)
I20260709 12:27:53.445679 31909 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fad0b7ef0f804aa494f44e9ec456e11e" candidate_uuid: "2ae3a9e2293442e291b1262a717a5e50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" is_pre_election: true
I20260709 12:27:53.445839 31984 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fad0b7ef0f804aa494f44e9ec456e11e" candidate_uuid: "2ae3a9e2293442e291b1262a717a5e50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07b97783285d4b22b6fd0f1d816ef930" is_pre_election: true
I20260709 12:27:53.446326 31909 raft_consensus.cc:2468] T fad0b7ef0f804aa494f44e9ec456e11e P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ae3a9e2293442e291b1262a717a5e50 in term 0.
I20260709 12:27:53.446628 31984 raft_consensus.cc:2468] T fad0b7ef0f804aa494f44e9ec456e11e P 07b97783285d4b22b6fd0f1d816ef930 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ae3a9e2293442e291b1262a717a5e50 in term 0.
I20260709 12:27:53.447382 31798 leader_election.cc:304] T fad0b7ef0f804aa494f44e9ec456e11e P 2ae3a9e2293442e291b1262a717a5e50 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ae3a9e2293442e291b1262a717a5e50, 64fd1d8a3cd14e66ac94c4fbcceaf039; no voters: 
I20260709 12:27:53.448262 32046 raft_consensus.cc:2804] T fad0b7ef0f804aa494f44e9ec456e11e P 2ae3a9e2293442e291b1262a717a5e50 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:53.448567 32046 raft_consensus.cc:493] T fad0b7ef0f804aa494f44e9ec456e11e P 2ae3a9e2293442e291b1262a717a5e50 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:53.448801 32046 raft_consensus.cc:3060] T fad0b7ef0f804aa494f44e9ec456e11e P 2ae3a9e2293442e291b1262a717a5e50 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:53.454358 32046 raft_consensus.cc:515] T fad0b7ef0f804aa494f44e9ec456e11e P 2ae3a9e2293442e291b1262a717a5e50 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:53.456095 32046 leader_election.cc:290] T fad0b7ef0f804aa494f44e9ec456e11e P 2ae3a9e2293442e291b1262a717a5e50 [CANDIDATE]: Term 1 election: Requested vote from peers 64fd1d8a3cd14e66ac94c4fbcceaf039 (127.16.29.66:42163), 07b97783285d4b22b6fd0f1d816ef930 (127.16.29.67:34419)
I20260709 12:27:53.456964 31909 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fad0b7ef0f804aa494f44e9ec456e11e" candidate_uuid: "2ae3a9e2293442e291b1262a717a5e50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039"
I20260709 12:27:53.457157 31984 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fad0b7ef0f804aa494f44e9ec456e11e" candidate_uuid: "2ae3a9e2293442e291b1262a717a5e50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07b97783285d4b22b6fd0f1d816ef930"
I20260709 12:27:53.457749 31909 raft_consensus.cc:3060] T fad0b7ef0f804aa494f44e9ec456e11e P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:53.457772 31984 raft_consensus.cc:3060] T fad0b7ef0f804aa494f44e9ec456e11e P 07b97783285d4b22b6fd0f1d816ef930 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:53.465806 31909 raft_consensus.cc:2468] T fad0b7ef0f804aa494f44e9ec456e11e P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ae3a9e2293442e291b1262a717a5e50 in term 1.
I20260709 12:27:53.465801 31984 raft_consensus.cc:2468] T fad0b7ef0f804aa494f44e9ec456e11e P 07b97783285d4b22b6fd0f1d816ef930 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ae3a9e2293442e291b1262a717a5e50 in term 1.
I20260709 12:27:53.467458 31801 leader_election.cc:304] T fad0b7ef0f804aa494f44e9ec456e11e P 2ae3a9e2293442e291b1262a717a5e50 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 07b97783285d4b22b6fd0f1d816ef930, 2ae3a9e2293442e291b1262a717a5e50, 64fd1d8a3cd14e66ac94c4fbcceaf039; no voters: 
I20260709 12:27:53.468361 32046 raft_consensus.cc:2804] T fad0b7ef0f804aa494f44e9ec456e11e P 2ae3a9e2293442e291b1262a717a5e50 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:53.468863 32046 raft_consensus.cc:697] T fad0b7ef0f804aa494f44e9ec456e11e P 2ae3a9e2293442e291b1262a717a5e50 [term 1 LEADER]: Becoming Leader. State: Replica: 2ae3a9e2293442e291b1262a717a5e50, State: Running, Role: LEADER
I20260709 12:27:53.469749 32046 consensus_queue.cc:237] T fad0b7ef0f804aa494f44e9ec456e11e P 2ae3a9e2293442e291b1262a717a5e50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:53.477526 31732 catalog_manager.cc:5697] T fad0b7ef0f804aa494f44e9ec456e11e P 2ae3a9e2293442e291b1262a717a5e50 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2ae3a9e2293442e291b1262a717a5e50 (127.16.29.65). New cstate: current_term: 1 leader_uuid: "2ae3a9e2293442e291b1262a717a5e50" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:53.510269 31909 raft_consensus.cc:3060] T 89784f886fb545d2871174c1e169fb32 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:53.517251 32032 consensus_queue.cc:1048] T 89784f886fb545d2871174c1e169fb32 P 07b97783285d4b22b6fd0f1d816ef930 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:27:53.522162 32046 raft_consensus.cc:493] T 6a5720f014ed48769eb641989edef3e8 P 2ae3a9e2293442e291b1262a717a5e50 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:53.522812 32046 raft_consensus.cc:515] T 6a5720f014ed48769eb641989edef3e8 P 2ae3a9e2293442e291b1262a717a5e50 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:53.525686 31909 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6a5720f014ed48769eb641989edef3e8" candidate_uuid: "2ae3a9e2293442e291b1262a717a5e50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" is_pre_election: true
I20260709 12:27:53.526054 31984 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6a5720f014ed48769eb641989edef3e8" candidate_uuid: "2ae3a9e2293442e291b1262a717a5e50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07b97783285d4b22b6fd0f1d816ef930" is_pre_election: true
I20260709 12:27:53.526494 31909 raft_consensus.cc:2468] T 6a5720f014ed48769eb641989edef3e8 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ae3a9e2293442e291b1262a717a5e50 in term 0.
I20260709 12:27:53.526844 31984 raft_consensus.cc:2468] T 6a5720f014ed48769eb641989edef3e8 P 07b97783285d4b22b6fd0f1d816ef930 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ae3a9e2293442e291b1262a717a5e50 in term 0.
I20260709 12:27:53.528167 31798 leader_election.cc:304] T 6a5720f014ed48769eb641989edef3e8 P 2ae3a9e2293442e291b1262a717a5e50 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ae3a9e2293442e291b1262a717a5e50, 64fd1d8a3cd14e66ac94c4fbcceaf039; no voters: 
I20260709 12:27:53.531399 32030 raft_consensus.cc:2804] T 6a5720f014ed48769eb641989edef3e8 P 2ae3a9e2293442e291b1262a717a5e50 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:53.529708 32046 leader_election.cc:290] T 6a5720f014ed48769eb641989edef3e8 P 2ae3a9e2293442e291b1262a717a5e50 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 64fd1d8a3cd14e66ac94c4fbcceaf039 (127.16.29.66:42163), 07b97783285d4b22b6fd0f1d816ef930 (127.16.29.67:34419)
I20260709 12:27:53.531926 32030 raft_consensus.cc:493] T 6a5720f014ed48769eb641989edef3e8 P 2ae3a9e2293442e291b1262a717a5e50 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:53.532337 32030 raft_consensus.cc:3060] T 6a5720f014ed48769eb641989edef3e8 P 2ae3a9e2293442e291b1262a717a5e50 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:53.536851 32032 consensus_queue.cc:1048] T 89784f886fb545d2871174c1e169fb32 P 07b97783285d4b22b6fd0f1d816ef930 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:27:53.537567 32052 raft_consensus.cc:493] T 0fb83fa22e13487f9d56facc914278ba P 07b97783285d4b22b6fd0f1d816ef930 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:53.538061 32052 raft_consensus.cc:515] T 0fb83fa22e13487f9d56facc914278ba P 07b97783285d4b22b6fd0f1d816ef930 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:53.539677 32030 raft_consensus.cc:515] T 6a5720f014ed48769eb641989edef3e8 P 2ae3a9e2293442e291b1262a717a5e50 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:53.540182 32052 leader_election.cc:290] T 0fb83fa22e13487f9d56facc914278ba P 07b97783285d4b22b6fd0f1d816ef930 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2ae3a9e2293442e291b1262a717a5e50 (127.16.29.65:44289), 64fd1d8a3cd14e66ac94c4fbcceaf039 (127.16.29.66:42163)
I20260709 12:27:53.541080 31834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0fb83fa22e13487f9d56facc914278ba" candidate_uuid: "07b97783285d4b22b6fd0f1d816ef930" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ae3a9e2293442e291b1262a717a5e50" is_pre_election: true
I20260709 12:27:53.541805 31834 raft_consensus.cc:2468] T 0fb83fa22e13487f9d56facc914278ba P 2ae3a9e2293442e291b1262a717a5e50 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 07b97783285d4b22b6fd0f1d816ef930 in term 0.
I20260709 12:27:53.542160 31910 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0fb83fa22e13487f9d56facc914278ba" candidate_uuid: "07b97783285d4b22b6fd0f1d816ef930" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" is_pre_election: true
I20260709 12:27:53.542758 31909 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6a5720f014ed48769eb641989edef3e8" candidate_uuid: "2ae3a9e2293442e291b1262a717a5e50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039"
I20260709 12:27:53.542867 31910 raft_consensus.cc:2468] T 0fb83fa22e13487f9d56facc914278ba P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 07b97783285d4b22b6fd0f1d816ef930 in term 0.
I20260709 12:27:53.543404 31949 leader_election.cc:304] T 0fb83fa22e13487f9d56facc914278ba P 07b97783285d4b22b6fd0f1d816ef930 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07b97783285d4b22b6fd0f1d816ef930, 2ae3a9e2293442e291b1262a717a5e50; no voters: 
I20260709 12:27:53.544061 32030 leader_election.cc:290] T 6a5720f014ed48769eb641989edef3e8 P 2ae3a9e2293442e291b1262a717a5e50 [CANDIDATE]: Term 1 election: Requested vote from peers 64fd1d8a3cd14e66ac94c4fbcceaf039 (127.16.29.66:42163), 07b97783285d4b22b6fd0f1d816ef930 (127.16.29.67:34419)
I20260709 12:27:53.544605 32032 raft_consensus.cc:2804] T 0fb83fa22e13487f9d56facc914278ba P 07b97783285d4b22b6fd0f1d816ef930 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:53.543452 31909 raft_consensus.cc:3060] T 6a5720f014ed48769eb641989edef3e8 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:53.545063 32032 raft_consensus.cc:493] T 0fb83fa22e13487f9d56facc914278ba P 07b97783285d4b22b6fd0f1d816ef930 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:53.545491 32032 raft_consensus.cc:3060] T 0fb83fa22e13487f9d56facc914278ba P 07b97783285d4b22b6fd0f1d816ef930 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:53.548768 31984 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6a5720f014ed48769eb641989edef3e8" candidate_uuid: "2ae3a9e2293442e291b1262a717a5e50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07b97783285d4b22b6fd0f1d816ef930"
I20260709 12:27:53.549365 31984 raft_consensus.cc:3060] T 6a5720f014ed48769eb641989edef3e8 P 07b97783285d4b22b6fd0f1d816ef930 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:53.552299 32031 raft_consensus.cc:493] T 378f56c32c1e46f99a874bd1adcbcf25 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:53.552851 32031 raft_consensus.cc:515] T 378f56c32c1e46f99a874bd1adcbcf25 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:53.555109 32031 leader_election.cc:290] T 378f56c32c1e46f99a874bd1adcbcf25 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2ae3a9e2293442e291b1262a717a5e50 (127.16.29.65:44289), 07b97783285d4b22b6fd0f1d816ef930 (127.16.29.67:34419)
I20260709 12:27:53.557389 31984 raft_consensus.cc:2468] T 6a5720f014ed48769eb641989edef3e8 P 07b97783285d4b22b6fd0f1d816ef930 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ae3a9e2293442e291b1262a717a5e50 in term 1.
I20260709 12:27:53.558036 32032 raft_consensus.cc:515] T 0fb83fa22e13487f9d56facc914278ba P 07b97783285d4b22b6fd0f1d816ef930 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:53.558495 31801 leader_election.cc:304] T 6a5720f014ed48769eb641989edef3e8 P 2ae3a9e2293442e291b1262a717a5e50 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07b97783285d4b22b6fd0f1d816ef930, 2ae3a9e2293442e291b1262a717a5e50; no voters: 
I20260709 12:27:53.558631 31981 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378f56c32c1e46f99a874bd1adcbcf25" candidate_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07b97783285d4b22b6fd0f1d816ef930" is_pre_election: true
I20260709 12:27:53.560317 32032 leader_election.cc:290] T 0fb83fa22e13487f9d56facc914278ba P 07b97783285d4b22b6fd0f1d816ef930 [CANDIDATE]: Term 1 election: Requested vote from peers 2ae3a9e2293442e291b1262a717a5e50 (127.16.29.65:44289), 64fd1d8a3cd14e66ac94c4fbcceaf039 (127.16.29.66:42163)
I20260709 12:27:53.560921 31834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0fb83fa22e13487f9d56facc914278ba" candidate_uuid: "07b97783285d4b22b6fd0f1d816ef930" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ae3a9e2293442e291b1262a717a5e50"
I20260709 12:27:53.561525 31834 raft_consensus.cc:3060] T 0fb83fa22e13487f9d56facc914278ba P 2ae3a9e2293442e291b1262a717a5e50 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:53.561611 31910 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0fb83fa22e13487f9d56facc914278ba" candidate_uuid: "07b97783285d4b22b6fd0f1d816ef930" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039"
I20260709 12:27:53.561898 31833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378f56c32c1e46f99a874bd1adcbcf25" candidate_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ae3a9e2293442e291b1262a717a5e50" is_pre_election: true
I20260709 12:27:53.562325 31910 raft_consensus.cc:3060] T 0fb83fa22e13487f9d56facc914278ba P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:53.562726 31833 raft_consensus.cc:2393] T 378f56c32c1e46f99a874bd1adcbcf25 P 2ae3a9e2293442e291b1262a717a5e50 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 64fd1d8a3cd14e66ac94c4fbcceaf039 in current term 1: Already voted for candidate 07b97783285d4b22b6fd0f1d816ef930 in this term.
I20260709 12:27:53.559634 32046 raft_consensus.cc:2804] T 6a5720f014ed48769eb641989edef3e8 P 2ae3a9e2293442e291b1262a717a5e50 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:53.564023 32046 raft_consensus.cc:697] T 6a5720f014ed48769eb641989edef3e8 P 2ae3a9e2293442e291b1262a717a5e50 [term 1 LEADER]: Becoming Leader. State: Replica: 2ae3a9e2293442e291b1262a717a5e50, State: Running, Role: LEADER
I20260709 12:27:53.564131 31875 leader_election.cc:304] T 378f56c32c1e46f99a874bd1adcbcf25 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 64fd1d8a3cd14e66ac94c4fbcceaf039; no voters: 07b97783285d4b22b6fd0f1d816ef930, 2ae3a9e2293442e291b1262a717a5e50
I20260709 12:27:53.564929 32046 consensus_queue.cc:237] T 6a5720f014ed48769eb641989edef3e8 P 2ae3a9e2293442e291b1262a717a5e50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:53.561623 31909 raft_consensus.cc:2468] T 6a5720f014ed48769eb641989edef3e8 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ae3a9e2293442e291b1262a717a5e50 in term 1.
I20260709 12:27:53.565321 32031 raft_consensus.cc:3060] T 378f56c32c1e46f99a874bd1adcbcf25 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:53.570915 31910 raft_consensus.cc:2468] T 0fb83fa22e13487f9d56facc914278ba P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07b97783285d4b22b6fd0f1d816ef930 in term 1.
I20260709 12:27:53.572695 31947 leader_election.cc:304] T 0fb83fa22e13487f9d56facc914278ba P 07b97783285d4b22b6fd0f1d816ef930 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07b97783285d4b22b6fd0f1d816ef930, 64fd1d8a3cd14e66ac94c4fbcceaf039; no voters: 
I20260709 12:27:53.573818 32032 raft_consensus.cc:2804] T 0fb83fa22e13487f9d56facc914278ba P 07b97783285d4b22b6fd0f1d816ef930 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:53.575143 32032 raft_consensus.cc:697] T 0fb83fa22e13487f9d56facc914278ba P 07b97783285d4b22b6fd0f1d816ef930 [term 1 LEADER]: Becoming Leader. State: Replica: 07b97783285d4b22b6fd0f1d816ef930, State: Running, Role: LEADER
I20260709 12:27:53.576045 32032 consensus_queue.cc:237] T 0fb83fa22e13487f9d56facc914278ba P 07b97783285d4b22b6fd0f1d816ef930 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:53.579421 32031 raft_consensus.cc:2749] T 378f56c32c1e46f99a874bd1adcbcf25 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:53.582700 31834 raft_consensus.cc:2468] T 0fb83fa22e13487f9d56facc914278ba P 2ae3a9e2293442e291b1262a717a5e50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07b97783285d4b22b6fd0f1d816ef930 in term 1.
I20260709 12:27:53.586658 31732 catalog_manager.cc:5697] T 0fb83fa22e13487f9d56facc914278ba P 07b97783285d4b22b6fd0f1d816ef930 reported cstate change: term changed from 0 to 1, leader changed from <none> to 07b97783285d4b22b6fd0f1d816ef930 (127.16.29.67). New cstate: current_term: 1 leader_uuid: "07b97783285d4b22b6fd0f1d816ef930" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } health_report { overall_health: HEALTHY } } }
I20260709 12:27:53.586810 31731 catalog_manager.cc:5697] T 6a5720f014ed48769eb641989edef3e8 P 2ae3a9e2293442e291b1262a717a5e50 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2ae3a9e2293442e291b1262a717a5e50 (127.16.29.65). New cstate: current_term: 1 leader_uuid: "2ae3a9e2293442e291b1262a717a5e50" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:53.634147 32031 consensus_queue.cc:1048] T d00d4a3dbb1c435cb565cf6e5d723bf2 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:27:53.646024 32031 consensus_queue.cc:1048] T d00d4a3dbb1c435cb565cf6e5d723bf2 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:53.691495 32031 consensus_queue.cc:1048] T 1638e215a9ee4068a999e92023fff415 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:27:53.702744 31834 raft_consensus.cc:3060] T 1638e215a9ee4068a999e92023fff415 P 2ae3a9e2293442e291b1262a717a5e50 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:53.708307 32046 consensus_queue.cc:1048] T 52322cf19f734f19a4ccfc8389f32b41 P 2ae3a9e2293442e291b1262a717a5e50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:53.712026 32031 consensus_queue.cc:1048] T 1638e215a9ee4068a999e92023fff415 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:27:53.721567 32093 consensus_queue.cc:1048] T 52322cf19f734f19a4ccfc8389f32b41 P 2ae3a9e2293442e291b1262a717a5e50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:53.742048 32031 consensus_queue.cc:1048] T c61f10da93eb4af5a80a204e679f1ee3 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:27:53.752481 32062 consensus_queue.cc:1048] T c61f10da93eb4af5a80a204e679f1ee3 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:53.756567 31981 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "89784f886fb545d2871174c1e169fb32"
dest_uuid: "07b97783285d4b22b6fd0f1d816ef930"
mode: GRACEFUL
new_leader_uuid: "2ae3a9e2293442e291b1262a717a5e50"
 from {username='slave'} at 127.0.0.1:38900
I20260709 12:27:53.757314 31981 raft_consensus.cc:606] T 89784f886fb545d2871174c1e169fb32 P 07b97783285d4b22b6fd0f1d816ef930 [term 1 LEADER]: Received request to transfer leadership to TS 2ae3a9e2293442e291b1262a717a5e50
I20260709 12:27:53.766444 32032 consensus_queue.cc:1048] T 378f56c32c1e46f99a874bd1adcbcf25 P 07b97783285d4b22b6fd0f1d816ef930 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:27:53.781127 32056 consensus_queue.cc:1048] T 378f56c32c1e46f99a874bd1adcbcf25 P 07b97783285d4b22b6fd0f1d816ef930 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:53.797369 31910 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "d00d4a3dbb1c435cb565cf6e5d723bf2"
dest_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039"
mode: GRACEFUL
new_leader_uuid: "2ae3a9e2293442e291b1262a717a5e50"
 from {username='slave'} at 127.0.0.1:33772
I20260709 12:27:53.797966 31910 raft_consensus.cc:606] T d00d4a3dbb1c435cb565cf6e5d723bf2 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 1 LEADER]: Received request to transfer leadership to TS 2ae3a9e2293442e291b1262a717a5e50
I20260709 12:27:53.804438 31910 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "1638e215a9ee4068a999e92023fff415"
dest_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039"
mode: GRACEFUL
new_leader_uuid: "2ae3a9e2293442e291b1262a717a5e50"
 from {username='slave'} at 127.0.0.1:33772
I20260709 12:27:53.805095 31910 raft_consensus.cc:606] T 1638e215a9ee4068a999e92023fff415 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 1 LEADER]: Received request to transfer leadership to TS 2ae3a9e2293442e291b1262a717a5e50
I20260709 12:27:53.812359 31984 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "378f56c32c1e46f99a874bd1adcbcf25"
dest_uuid: "07b97783285d4b22b6fd0f1d816ef930"
mode: GRACEFUL
new_leader_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039"
 from {username='slave'} at 127.0.0.1:38900
I20260709 12:27:53.812945 31984 raft_consensus.cc:606] T 378f56c32c1e46f99a874bd1adcbcf25 P 07b97783285d4b22b6fd0f1d816ef930 [term 1 LEADER]: Received request to transfer leadership to TS 64fd1d8a3cd14e66ac94c4fbcceaf039
I20260709 12:27:53.831768 31834 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "52322cf19f734f19a4ccfc8389f32b41"
dest_uuid: "2ae3a9e2293442e291b1262a717a5e50"
mode: GRACEFUL
new_leader_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039"
 from {username='slave'} at 127.0.0.1:40002
I20260709 12:27:53.832491 31834 raft_consensus.cc:606] T 52322cf19f734f19a4ccfc8389f32b41 P 2ae3a9e2293442e291b1262a717a5e50 [term 1 LEADER]: Received request to transfer leadership to TS 64fd1d8a3cd14e66ac94c4fbcceaf039
I20260709 12:27:53.838434 31834 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "6a5720f014ed48769eb641989edef3e8"
dest_uuid: "2ae3a9e2293442e291b1262a717a5e50"
mode: GRACEFUL
new_leader_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039"
 from {username='slave'} at 127.0.0.1:40002
I20260709 12:27:53.839025 31834 raft_consensus.cc:606] T 6a5720f014ed48769eb641989edef3e8 P 2ae3a9e2293442e291b1262a717a5e50 [term 1 LEADER]: Received request to transfer leadership to TS 64fd1d8a3cd14e66ac94c4fbcceaf039
I20260709 12:27:53.992312 32052 raft_consensus.cc:993] T 89784f886fb545d2871174c1e169fb32 P 07b97783285d4b22b6fd0f1d816ef930: : Instructing follower 2ae3a9e2293442e291b1262a717a5e50 to start an election
I20260709 12:27:53.992794 32056 raft_consensus.cc:1081] T 89784f886fb545d2871174c1e169fb32 P 07b97783285d4b22b6fd0f1d816ef930 [term 1 LEADER]: Signalling peer 2ae3a9e2293442e291b1262a717a5e50 to start an election
I20260709 12:27:53.994551 31834 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "89784f886fb545d2871174c1e169fb32"
dest_uuid: "2ae3a9e2293442e291b1262a717a5e50"
 from {username='slave'} at 127.0.0.1:39986
I20260709 12:27:53.995121 31834 raft_consensus.cc:493] T 89784f886fb545d2871174c1e169fb32 P 2ae3a9e2293442e291b1262a717a5e50 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:27:53.995462 31834 raft_consensus.cc:3060] T 89784f886fb545d2871174c1e169fb32 P 2ae3a9e2293442e291b1262a717a5e50 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:54.001801 31834 raft_consensus.cc:515] T 89784f886fb545d2871174c1e169fb32 P 2ae3a9e2293442e291b1262a717a5e50 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } } peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } }
I20260709 12:27:54.003973 31834 leader_election.cc:290] T 89784f886fb545d2871174c1e169fb32 P 2ae3a9e2293442e291b1262a717a5e50 [CANDIDATE]: Term 2 election: Requested vote from peers 07b97783285d4b22b6fd0f1d816ef930 (127.16.29.67:34419), 64fd1d8a3cd14e66ac94c4fbcceaf039 (127.16.29.66:42163)
I20260709 12:27:54.004418 31984 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "89784f886fb545d2871174c1e169fb32" candidate_uuid: "2ae3a9e2293442e291b1262a717a5e50" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "07b97783285d4b22b6fd0f1d816ef930"
I20260709 12:27:54.004629 31910 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "89784f886fb545d2871174c1e169fb32" candidate_uuid: "2ae3a9e2293442e291b1262a717a5e50" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039"
I20260709 12:27:54.005151 31910 raft_consensus.cc:3060] T 89784f886fb545d2871174c1e169fb32 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:54.005193 31984 raft_consensus.cc:3055] T 89784f886fb545d2871174c1e169fb32 P 07b97783285d4b22b6fd0f1d816ef930 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:27:54.005710 31984 raft_consensus.cc:740] T 89784f886fb545d2871174c1e169fb32 P 07b97783285d4b22b6fd0f1d816ef930 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 07b97783285d4b22b6fd0f1d816ef930, State: Running, Role: LEADER
I20260709 12:27:54.006654 31984 consensus_queue.cc:260] T 89784f886fb545d2871174c1e169fb32 P 07b97783285d4b22b6fd0f1d816ef930 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } } peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } }
I20260709 12:27:54.007951 31984 raft_consensus.cc:3060] T 89784f886fb545d2871174c1e169fb32 P 07b97783285d4b22b6fd0f1d816ef930 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:54.011266 31910 raft_consensus.cc:2468] T 89784f886fb545d2871174c1e169fb32 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ae3a9e2293442e291b1262a717a5e50 in term 2.
I20260709 12:27:54.012604 31798 leader_election.cc:304] T 89784f886fb545d2871174c1e169fb32 P 2ae3a9e2293442e291b1262a717a5e50 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ae3a9e2293442e291b1262a717a5e50, 64fd1d8a3cd14e66ac94c4fbcceaf039; no voters: 
I20260709 12:27:54.013366 32093 raft_consensus.cc:2804] T 89784f886fb545d2871174c1e169fb32 P 2ae3a9e2293442e291b1262a717a5e50 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:27:54.013825 32093 raft_consensus.cc:697] T 89784f886fb545d2871174c1e169fb32 P 2ae3a9e2293442e291b1262a717a5e50 [term 2 LEADER]: Becoming Leader. State: Replica: 2ae3a9e2293442e291b1262a717a5e50, State: Running, Role: LEADER
I20260709 12:27:54.014562 32093 consensus_queue.cc:237] T 89784f886fb545d2871174c1e169fb32 P 2ae3a9e2293442e291b1262a717a5e50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } } peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } }
I20260709 12:27:54.016696 31984 raft_consensus.cc:2468] T 89784f886fb545d2871174c1e169fb32 P 07b97783285d4b22b6fd0f1d816ef930 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ae3a9e2293442e291b1262a717a5e50 in term 2.
I20260709 12:27:54.022218 31733 catalog_manager.cc:5697] T 89784f886fb545d2871174c1e169fb32 P 2ae3a9e2293442e291b1262a717a5e50 reported cstate change: term changed from 1 to 2, leader changed from 07b97783285d4b22b6fd0f1d816ef930 (127.16.29.67) to 2ae3a9e2293442e291b1262a717a5e50 (127.16.29.65). New cstate: current_term: 2 leader_uuid: "2ae3a9e2293442e291b1262a717a5e50" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:54.045765 32093 consensus_queue.cc:1048] T fad0b7ef0f804aa494f44e9ec456e11e P 2ae3a9e2293442e291b1262a717a5e50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:27:54.057253 32046 consensus_queue.cc:1048] T fad0b7ef0f804aa494f44e9ec456e11e P 2ae3a9e2293442e291b1262a717a5e50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:27:54.071534 32056 consensus_queue.cc:1048] T 0fb83fa22e13487f9d56facc914278ba P 07b97783285d4b22b6fd0f1d816ef930 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:27:54.071712 32104 consensus_queue.cc:1048] T 6a5720f014ed48769eb641989edef3e8 P 2ae3a9e2293442e291b1262a717a5e50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:27:54.075982 32056 consensus_queue.cc:1048] T 0fb83fa22e13487f9d56facc914278ba P 07b97783285d4b22b6fd0f1d816ef930 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:27:54.095939 32046 raft_consensus.cc:993] T 6a5720f014ed48769eb641989edef3e8 P 2ae3a9e2293442e291b1262a717a5e50: : Instructing follower 64fd1d8a3cd14e66ac94c4fbcceaf039 to start an election
I20260709 12:27:54.096426 32046 raft_consensus.cc:1081] T 6a5720f014ed48769eb641989edef3e8 P 2ae3a9e2293442e291b1262a717a5e50 [term 1 LEADER]: Signalling peer 64fd1d8a3cd14e66ac94c4fbcceaf039 to start an election
I20260709 12:27:54.098780 31909 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "6a5720f014ed48769eb641989edef3e8"
dest_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039"
 from {username='slave'} at 127.0.0.1:33764
I20260709 12:27:54.099363 31909 raft_consensus.cc:493] T 6a5720f014ed48769eb641989edef3e8 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:27:54.099745 31909 raft_consensus.cc:3060] T 6a5720f014ed48769eb641989edef3e8 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:54.104456 32046 consensus_queue.cc:1048] T 6a5720f014ed48769eb641989edef3e8 P 2ae3a9e2293442e291b1262a717a5e50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:54.108664 31909 raft_consensus.cc:515] T 6a5720f014ed48769eb641989edef3e8 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:54.109853 31908 raft_consensus.cc:1240] T 6a5720f014ed48769eb641989edef3e8 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 2 FOLLOWER]: Rejecting Update request from peer 2ae3a9e2293442e291b1262a717a5e50 for earlier term 1. Current term is 2. Ops: []
I20260709 12:27:54.112900 31834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6a5720f014ed48769eb641989edef3e8" candidate_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2ae3a9e2293442e291b1262a717a5e50"
I20260709 12:27:54.113366 31981 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6a5720f014ed48769eb641989edef3e8" candidate_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "07b97783285d4b22b6fd0f1d816ef930"
I20260709 12:27:54.114053 31981 raft_consensus.cc:3060] T 6a5720f014ed48769eb641989edef3e8 P 07b97783285d4b22b6fd0f1d816ef930 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:54.118322 32046 consensus_queue.cc:1059] T 6a5720f014ed48769eb641989edef3e8 P 2ae3a9e2293442e291b1262a717a5e50 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:54.113523 31834 raft_consensus.cc:3055] T 6a5720f014ed48769eb641989edef3e8 P 2ae3a9e2293442e291b1262a717a5e50 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:27:54.119939 31834 raft_consensus.cc:740] T 6a5720f014ed48769eb641989edef3e8 P 2ae3a9e2293442e291b1262a717a5e50 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2ae3a9e2293442e291b1262a717a5e50, State: Running, Role: LEADER
I20260709 12:27:54.120003 31909 leader_election.cc:290] T 6a5720f014ed48769eb641989edef3e8 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [CANDIDATE]: Term 2 election: Requested vote from peers 2ae3a9e2293442e291b1262a717a5e50 (127.16.29.65:44289), 07b97783285d4b22b6fd0f1d816ef930 (127.16.29.67:34419)
I20260709 12:27:54.120958 31834 consensus_queue.cc:260] T 6a5720f014ed48769eb641989edef3e8 P 2ae3a9e2293442e291b1262a717a5e50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:54.122100 31981 raft_consensus.cc:2468] T 6a5720f014ed48769eb641989edef3e8 P 07b97783285d4b22b6fd0f1d816ef930 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64fd1d8a3cd14e66ac94c4fbcceaf039 in term 2.
I20260709 12:27:54.122123 31834 raft_consensus.cc:3060] T 6a5720f014ed48769eb641989edef3e8 P 2ae3a9e2293442e291b1262a717a5e50 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:54.123324 31876 leader_election.cc:304] T 6a5720f014ed48769eb641989edef3e8 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07b97783285d4b22b6fd0f1d816ef930, 64fd1d8a3cd14e66ac94c4fbcceaf039; no voters: 
I20260709 12:27:54.124104 32031 raft_consensus.cc:2804] T 6a5720f014ed48769eb641989edef3e8 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:27:54.124621 32031 raft_consensus.cc:697] T 6a5720f014ed48769eb641989edef3e8 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 2 LEADER]: Becoming Leader. State: Replica: 64fd1d8a3cd14e66ac94c4fbcceaf039, State: Running, Role: LEADER
I20260709 12:27:54.125396 32031 consensus_queue.cc:237] T 6a5720f014ed48769eb641989edef3e8 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:54.127825 31834 raft_consensus.cc:2468] T 6a5720f014ed48769eb641989edef3e8 P 2ae3a9e2293442e291b1262a717a5e50 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64fd1d8a3cd14e66ac94c4fbcceaf039 in term 2.
W20260709 12:27:54.128464 32093 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 2 current term: 2 is higher.
I20260709 12:27:54.135696 31730 catalog_manager.cc:5697] T 6a5720f014ed48769eb641989edef3e8 P 64fd1d8a3cd14e66ac94c4fbcceaf039 reported cstate change: term changed from 1 to 2, leader changed from 2ae3a9e2293442e291b1262a717a5e50 (127.16.29.65) to 64fd1d8a3cd14e66ac94c4fbcceaf039 (127.16.29.66). New cstate: current_term: 2 leader_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:54.147784 32054 raft_consensus.cc:993] T d00d4a3dbb1c435cb565cf6e5d723bf2 P 64fd1d8a3cd14e66ac94c4fbcceaf039: : Instructing follower 2ae3a9e2293442e291b1262a717a5e50 to start an election
I20260709 12:27:54.148397 32054 raft_consensus.cc:1081] T d00d4a3dbb1c435cb565cf6e5d723bf2 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 1 LEADER]: Signalling peer 2ae3a9e2293442e291b1262a717a5e50 to start an election
I20260709 12:27:54.150247 31834 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "d00d4a3dbb1c435cb565cf6e5d723bf2"
dest_uuid: "2ae3a9e2293442e291b1262a717a5e50"
 from {username='slave'} at 127.0.0.1:39996
I20260709 12:27:54.150837 31834 raft_consensus.cc:493] T d00d4a3dbb1c435cb565cf6e5d723bf2 P 2ae3a9e2293442e291b1262a717a5e50 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:27:54.151242 31834 raft_consensus.cc:3060] T d00d4a3dbb1c435cb565cf6e5d723bf2 P 2ae3a9e2293442e291b1262a717a5e50 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:54.156699 31834 raft_consensus.cc:515] T d00d4a3dbb1c435cb565cf6e5d723bf2 P 2ae3a9e2293442e291b1262a717a5e50 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:54.158334 31834 leader_election.cc:290] T d00d4a3dbb1c435cb565cf6e5d723bf2 P 2ae3a9e2293442e291b1262a717a5e50 [CANDIDATE]: Term 2 election: Requested vote from peers 64fd1d8a3cd14e66ac94c4fbcceaf039 (127.16.29.66:42163), 07b97783285d4b22b6fd0f1d816ef930 (127.16.29.67:34419)
I20260709 12:27:54.159206 31909 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d00d4a3dbb1c435cb565cf6e5d723bf2" candidate_uuid: "2ae3a9e2293442e291b1262a717a5e50" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039"
I20260709 12:27:54.159554 31981 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d00d4a3dbb1c435cb565cf6e5d723bf2" candidate_uuid: "2ae3a9e2293442e291b1262a717a5e50" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "07b97783285d4b22b6fd0f1d816ef930"
I20260709 12:27:54.159854 31909 raft_consensus.cc:3055] T d00d4a3dbb1c435cb565cf6e5d723bf2 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:27:54.160262 31909 raft_consensus.cc:740] T d00d4a3dbb1c435cb565cf6e5d723bf2 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 64fd1d8a3cd14e66ac94c4fbcceaf039, State: Running, Role: LEADER
I20260709 12:27:54.160260 31981 raft_consensus.cc:3060] T d00d4a3dbb1c435cb565cf6e5d723bf2 P 07b97783285d4b22b6fd0f1d816ef930 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:54.161010 31909 consensus_queue.cc:260] T d00d4a3dbb1c435cb565cf6e5d723bf2 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:54.161985 31909 raft_consensus.cc:3060] T d00d4a3dbb1c435cb565cf6e5d723bf2 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:54.166745 31981 raft_consensus.cc:2468] T d00d4a3dbb1c435cb565cf6e5d723bf2 P 07b97783285d4b22b6fd0f1d816ef930 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ae3a9e2293442e291b1262a717a5e50 in term 2.
I20260709 12:27:54.168659 31909 raft_consensus.cc:2468] T d00d4a3dbb1c435cb565cf6e5d723bf2 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ae3a9e2293442e291b1262a717a5e50 in term 2.
I20260709 12:27:54.168814 31801 leader_election.cc:304] T d00d4a3dbb1c435cb565cf6e5d723bf2 P 2ae3a9e2293442e291b1262a717a5e50 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07b97783285d4b22b6fd0f1d816ef930, 2ae3a9e2293442e291b1262a717a5e50; no voters: 
I20260709 12:27:54.169839 32093 raft_consensus.cc:2804] T d00d4a3dbb1c435cb565cf6e5d723bf2 P 2ae3a9e2293442e291b1262a717a5e50 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:27:54.170327 32093 raft_consensus.cc:697] T d00d4a3dbb1c435cb565cf6e5d723bf2 P 2ae3a9e2293442e291b1262a717a5e50 [term 2 LEADER]: Becoming Leader. State: Replica: 2ae3a9e2293442e291b1262a717a5e50, State: Running, Role: LEADER
I20260709 12:27:54.171186 32093 consensus_queue.cc:237] T d00d4a3dbb1c435cb565cf6e5d723bf2 P 2ae3a9e2293442e291b1262a717a5e50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:54.178750 31730 catalog_manager.cc:5697] T d00d4a3dbb1c435cb565cf6e5d723bf2 P 2ae3a9e2293442e291b1262a717a5e50 reported cstate change: term changed from 1 to 2, leader changed from 64fd1d8a3cd14e66ac94c4fbcceaf039 (127.16.29.66) to 2ae3a9e2293442e291b1262a717a5e50 (127.16.29.65). New cstate: current_term: 2 leader_uuid: "2ae3a9e2293442e291b1262a717a5e50" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:54.223531 32032 raft_consensus.cc:993] T 378f56c32c1e46f99a874bd1adcbcf25 P 07b97783285d4b22b6fd0f1d816ef930: : Instructing follower 64fd1d8a3cd14e66ac94c4fbcceaf039 to start an election
I20260709 12:27:54.224020 32052 raft_consensus.cc:1081] T 378f56c32c1e46f99a874bd1adcbcf25 P 07b97783285d4b22b6fd0f1d816ef930 [term 1 LEADER]: Signalling peer 64fd1d8a3cd14e66ac94c4fbcceaf039 to start an election
I20260709 12:27:54.225796 31909 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "378f56c32c1e46f99a874bd1adcbcf25"
dest_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039"
 from {username='slave'} at 127.0.0.1:33758
I20260709 12:27:54.226378 31909 raft_consensus.cc:493] T 378f56c32c1e46f99a874bd1adcbcf25 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:27:54.226714 31909 raft_consensus.cc:3060] T 378f56c32c1e46f99a874bd1adcbcf25 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:54.231930 31909 raft_consensus.cc:515] T 378f56c32c1e46f99a874bd1adcbcf25 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:54.233989 31909 leader_election.cc:290] T 378f56c32c1e46f99a874bd1adcbcf25 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [CANDIDATE]: Term 2 election: Requested vote from peers 2ae3a9e2293442e291b1262a717a5e50 (127.16.29.65:44289), 07b97783285d4b22b6fd0f1d816ef930 (127.16.29.67:34419)
I20260709 12:27:54.234869 31834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378f56c32c1e46f99a874bd1adcbcf25" candidate_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2ae3a9e2293442e291b1262a717a5e50"
I20260709 12:27:54.235435 31981 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378f56c32c1e46f99a874bd1adcbcf25" candidate_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "07b97783285d4b22b6fd0f1d816ef930"
I20260709 12:27:54.235658 31834 raft_consensus.cc:3060] T 378f56c32c1e46f99a874bd1adcbcf25 P 2ae3a9e2293442e291b1262a717a5e50 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:54.236115 31981 raft_consensus.cc:3055] T 378f56c32c1e46f99a874bd1adcbcf25 P 07b97783285d4b22b6fd0f1d816ef930 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:27:54.236490 31981 raft_consensus.cc:740] T 378f56c32c1e46f99a874bd1adcbcf25 P 07b97783285d4b22b6fd0f1d816ef930 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 07b97783285d4b22b6fd0f1d816ef930, State: Running, Role: LEADER
I20260709 12:27:54.237448 31981 consensus_queue.cc:260] T 378f56c32c1e46f99a874bd1adcbcf25 P 07b97783285d4b22b6fd0f1d816ef930 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:54.238744 31981 raft_consensus.cc:3060] T 378f56c32c1e46f99a874bd1adcbcf25 P 07b97783285d4b22b6fd0f1d816ef930 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:54.242429 31834 raft_consensus.cc:2468] T 378f56c32c1e46f99a874bd1adcbcf25 P 2ae3a9e2293442e291b1262a717a5e50 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64fd1d8a3cd14e66ac94c4fbcceaf039 in term 2.
I20260709 12:27:54.243672 31875 leader_election.cc:304] T 378f56c32c1e46f99a874bd1adcbcf25 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ae3a9e2293442e291b1262a717a5e50, 64fd1d8a3cd14e66ac94c4fbcceaf039; no voters: 
I20260709 12:27:54.244501 32054 raft_consensus.cc:2804] T 378f56c32c1e46f99a874bd1adcbcf25 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:27:54.244575 31981 raft_consensus.cc:2468] T 378f56c32c1e46f99a874bd1adcbcf25 P 07b97783285d4b22b6fd0f1d816ef930 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64fd1d8a3cd14e66ac94c4fbcceaf039 in term 2.
I20260709 12:27:54.245158 32054 raft_consensus.cc:697] T 378f56c32c1e46f99a874bd1adcbcf25 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 2 LEADER]: Becoming Leader. State: Replica: 64fd1d8a3cd14e66ac94c4fbcceaf039, State: Running, Role: LEADER
I20260709 12:27:54.246078 32054 consensus_queue.cc:237] T 378f56c32c1e46f99a874bd1adcbcf25 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:54.252885 31730 catalog_manager.cc:5697] T 378f56c32c1e46f99a874bd1adcbcf25 P 64fd1d8a3cd14e66ac94c4fbcceaf039 reported cstate change: term changed from 1 to 2, leader changed from 07b97783285d4b22b6fd0f1d816ef930 (127.16.29.67) to 64fd1d8a3cd14e66ac94c4fbcceaf039 (127.16.29.66). New cstate: current_term: 2 leader_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:54.272709 32031 raft_consensus.cc:993] T 1638e215a9ee4068a999e92023fff415 P 64fd1d8a3cd14e66ac94c4fbcceaf039: : Instructing follower 2ae3a9e2293442e291b1262a717a5e50 to start an election
I20260709 12:27:54.273226 32054 raft_consensus.cc:1081] T 1638e215a9ee4068a999e92023fff415 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 1 LEADER]: Signalling peer 2ae3a9e2293442e291b1262a717a5e50 to start an election
I20260709 12:27:54.274901 31834 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "1638e215a9ee4068a999e92023fff415"
dest_uuid: "2ae3a9e2293442e291b1262a717a5e50"
 from {username='slave'} at 127.0.0.1:39996
I20260709 12:27:54.275465 31834 raft_consensus.cc:493] T 1638e215a9ee4068a999e92023fff415 P 2ae3a9e2293442e291b1262a717a5e50 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:27:54.275913 31834 raft_consensus.cc:3060] T 1638e215a9ee4068a999e92023fff415 P 2ae3a9e2293442e291b1262a717a5e50 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:54.281495 31834 raft_consensus.cc:515] T 1638e215a9ee4068a999e92023fff415 P 2ae3a9e2293442e291b1262a717a5e50 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:54.283201 31834 leader_election.cc:290] T 1638e215a9ee4068a999e92023fff415 P 2ae3a9e2293442e291b1262a717a5e50 [CANDIDATE]: Term 2 election: Requested vote from peers 64fd1d8a3cd14e66ac94c4fbcceaf039 (127.16.29.66:42163), 07b97783285d4b22b6fd0f1d816ef930 (127.16.29.67:34419)
I20260709 12:27:54.284439 31909 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1638e215a9ee4068a999e92023fff415" candidate_uuid: "2ae3a9e2293442e291b1262a717a5e50" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039"
I20260709 12:27:54.284776 31981 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1638e215a9ee4068a999e92023fff415" candidate_uuid: "2ae3a9e2293442e291b1262a717a5e50" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "07b97783285d4b22b6fd0f1d816ef930"
I20260709 12:27:54.285173 31909 raft_consensus.cc:3055] T 1638e215a9ee4068a999e92023fff415 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:27:54.285315 31981 raft_consensus.cc:3060] T 1638e215a9ee4068a999e92023fff415 P 07b97783285d4b22b6fd0f1d816ef930 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:54.285578 31909 raft_consensus.cc:740] T 1638e215a9ee4068a999e92023fff415 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 64fd1d8a3cd14e66ac94c4fbcceaf039, State: Running, Role: LEADER
I20260709 12:27:54.286485 31909 consensus_queue.cc:260] T 1638e215a9ee4068a999e92023fff415 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:54.287772 31909 raft_consensus.cc:3060] T 1638e215a9ee4068a999e92023fff415 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:54.293422 31981 raft_consensus.cc:2468] T 1638e215a9ee4068a999e92023fff415 P 07b97783285d4b22b6fd0f1d816ef930 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ae3a9e2293442e291b1262a717a5e50 in term 2.
I20260709 12:27:54.294996 31801 leader_election.cc:304] T 1638e215a9ee4068a999e92023fff415 P 2ae3a9e2293442e291b1262a717a5e50 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07b97783285d4b22b6fd0f1d816ef930, 2ae3a9e2293442e291b1262a717a5e50; no voters: 
I20260709 12:27:54.295845 31909 raft_consensus.cc:2468] T 1638e215a9ee4068a999e92023fff415 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ae3a9e2293442e291b1262a717a5e50 in term 2.
I20260709 12:27:54.295948 32093 raft_consensus.cc:2804] T 1638e215a9ee4068a999e92023fff415 P 2ae3a9e2293442e291b1262a717a5e50 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:27:54.296693 32093 raft_consensus.cc:697] T 1638e215a9ee4068a999e92023fff415 P 2ae3a9e2293442e291b1262a717a5e50 [term 2 LEADER]: Becoming Leader. State: Replica: 2ae3a9e2293442e291b1262a717a5e50, State: Running, Role: LEADER
I20260709 12:27:54.297556 32093 consensus_queue.cc:237] T 1638e215a9ee4068a999e92023fff415 P 2ae3a9e2293442e291b1262a717a5e50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:54.305796 31730 catalog_manager.cc:5697] T 1638e215a9ee4068a999e92023fff415 P 2ae3a9e2293442e291b1262a717a5e50 reported cstate change: term changed from 1 to 2, leader changed from 64fd1d8a3cd14e66ac94c4fbcceaf039 (127.16.29.66) to 2ae3a9e2293442e291b1262a717a5e50 (127.16.29.65). New cstate: current_term: 2 leader_uuid: "2ae3a9e2293442e291b1262a717a5e50" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:54.343204 32046 raft_consensus.cc:993] T 52322cf19f734f19a4ccfc8389f32b41 P 2ae3a9e2293442e291b1262a717a5e50: : Instructing follower 64fd1d8a3cd14e66ac94c4fbcceaf039 to start an election
I20260709 12:27:54.343695 32093 raft_consensus.cc:1081] T 52322cf19f734f19a4ccfc8389f32b41 P 2ae3a9e2293442e291b1262a717a5e50 [term 1 LEADER]: Signalling peer 64fd1d8a3cd14e66ac94c4fbcceaf039 to start an election
I20260709 12:27:54.345433 31909 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "52322cf19f734f19a4ccfc8389f32b41"
dest_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039"
 from {username='slave'} at 127.0.0.1:33764
I20260709 12:27:54.345947 31909 raft_consensus.cc:493] T 52322cf19f734f19a4ccfc8389f32b41 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:27:54.346341 31909 raft_consensus.cc:3060] T 52322cf19f734f19a4ccfc8389f32b41 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:54.351552 31909 raft_consensus.cc:515] T 52322cf19f734f19a4ccfc8389f32b41 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:54.353542 31909 leader_election.cc:290] T 52322cf19f734f19a4ccfc8389f32b41 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [CANDIDATE]: Term 2 election: Requested vote from peers 2ae3a9e2293442e291b1262a717a5e50 (127.16.29.65:44289), 07b97783285d4b22b6fd0f1d816ef930 (127.16.29.67:34419)
I20260709 12:27:54.354823 31981 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52322cf19f734f19a4ccfc8389f32b41" candidate_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "07b97783285d4b22b6fd0f1d816ef930"
I20260709 12:27:54.354717 31834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52322cf19f734f19a4ccfc8389f32b41" candidate_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2ae3a9e2293442e291b1262a717a5e50"
I20260709 12:27:54.355412 31981 raft_consensus.cc:3060] T 52322cf19f734f19a4ccfc8389f32b41 P 07b97783285d4b22b6fd0f1d816ef930 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:54.355690 31834 raft_consensus.cc:3055] T 52322cf19f734f19a4ccfc8389f32b41 P 2ae3a9e2293442e291b1262a717a5e50 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:27:54.356119 31834 raft_consensus.cc:740] T 52322cf19f734f19a4ccfc8389f32b41 P 2ae3a9e2293442e291b1262a717a5e50 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2ae3a9e2293442e291b1262a717a5e50, State: Running, Role: LEADER
I20260709 12:27:54.356951 31834 consensus_queue.cc:260] T 52322cf19f734f19a4ccfc8389f32b41 P 2ae3a9e2293442e291b1262a717a5e50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:54.357999 31834 raft_consensus.cc:3060] T 52322cf19f734f19a4ccfc8389f32b41 P 2ae3a9e2293442e291b1262a717a5e50 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:54.360838 31981 raft_consensus.cc:2468] T 52322cf19f734f19a4ccfc8389f32b41 P 07b97783285d4b22b6fd0f1d816ef930 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64fd1d8a3cd14e66ac94c4fbcceaf039 in term 2.
I20260709 12:27:54.362003 31876 leader_election.cc:304] T 52322cf19f734f19a4ccfc8389f32b41 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07b97783285d4b22b6fd0f1d816ef930, 64fd1d8a3cd14e66ac94c4fbcceaf039; no voters: 
I20260709 12:27:54.362818 32054 raft_consensus.cc:2804] T 52322cf19f734f19a4ccfc8389f32b41 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:27:54.363354 32054 raft_consensus.cc:697] T 52322cf19f734f19a4ccfc8389f32b41 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 2 LEADER]: Becoming Leader. State: Replica: 64fd1d8a3cd14e66ac94c4fbcceaf039, State: Running, Role: LEADER
I20260709 12:27:54.364447 31834 raft_consensus.cc:2468] T 52322cf19f734f19a4ccfc8389f32b41 P 2ae3a9e2293442e291b1262a717a5e50 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64fd1d8a3cd14e66ac94c4fbcceaf039 in term 2.
I20260709 12:27:54.364331 32054 consensus_queue.cc:237] T 52322cf19f734f19a4ccfc8389f32b41 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:54.372258 31730 catalog_manager.cc:5697] T 52322cf19f734f19a4ccfc8389f32b41 P 64fd1d8a3cd14e66ac94c4fbcceaf039 reported cstate change: term changed from 1 to 2, leader changed from 2ae3a9e2293442e291b1262a717a5e50 (127.16.29.65) to 64fd1d8a3cd14e66ac94c4fbcceaf039 (127.16.29.66). New cstate: current_term: 2 leader_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:54.580672 31909 raft_consensus.cc:1275] T 89784f886fb545d2871174c1e169fb32 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 2 FOLLOWER]: Refusing update from remote peer 2ae3a9e2293442e291b1262a717a5e50: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:27:54.582134 32093 consensus_queue.cc:1048] T 89784f886fb545d2871174c1e169fb32 P 2ae3a9e2293442e291b1262a717a5e50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:27:54.591413 31981 raft_consensus.cc:1275] T 89784f886fb545d2871174c1e169fb32 P 07b97783285d4b22b6fd0f1d816ef930 [term 2 FOLLOWER]: Refusing update from remote peer 2ae3a9e2293442e291b1262a717a5e50: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:27:54.593029 32104 consensus_queue.cc:1048] T 89784f886fb545d2871174c1e169fb32 P 2ae3a9e2293442e291b1262a717a5e50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:27:54.707013 31981 raft_consensus.cc:1275] T 6a5720f014ed48769eb641989edef3e8 P 07b97783285d4b22b6fd0f1d816ef930 [term 2 FOLLOWER]: Refusing update from remote peer 64fd1d8a3cd14e66ac94c4fbcceaf039: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:27:54.708498 32054 consensus_queue.cc:1048] T 6a5720f014ed48769eb641989edef3e8 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:27:54.717242 31834 raft_consensus.cc:1275] T 6a5720f014ed48769eb641989edef3e8 P 2ae3a9e2293442e291b1262a717a5e50 [term 2 FOLLOWER]: Refusing update from remote peer 64fd1d8a3cd14e66ac94c4fbcceaf039: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:27:54.717775 31984 raft_consensus.cc:1275] T d00d4a3dbb1c435cb565cf6e5d723bf2 P 07b97783285d4b22b6fd0f1d816ef930 [term 2 FOLLOWER]: Refusing update from remote peer 2ae3a9e2293442e291b1262a717a5e50: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:27:54.719741 32062 consensus_queue.cc:1048] T 6a5720f014ed48769eb641989edef3e8 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:27:54.719906 32046 consensus_queue.cc:1048] T d00d4a3dbb1c435cb565cf6e5d723bf2 P 2ae3a9e2293442e291b1262a717a5e50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:27:54.720140 31909 raft_consensus.cc:1275] T d00d4a3dbb1c435cb565cf6e5d723bf2 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 2 FOLLOWER]: Refusing update from remote peer 2ae3a9e2293442e291b1262a717a5e50: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:27:54.722677 32046 consensus_queue.cc:1048] T d00d4a3dbb1c435cb565cf6e5d723bf2 P 2ae3a9e2293442e291b1262a717a5e50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:27:54.756742 31981 raft_consensus.cc:1275] T 378f56c32c1e46f99a874bd1adcbcf25 P 07b97783285d4b22b6fd0f1d816ef930 [term 2 FOLLOWER]: Refusing update from remote peer 64fd1d8a3cd14e66ac94c4fbcceaf039: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:27:54.758113 32062 consensus_queue.cc:1048] T 378f56c32c1e46f99a874bd1adcbcf25 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:27:54.765965 31834 raft_consensus.cc:1275] T 378f56c32c1e46f99a874bd1adcbcf25 P 2ae3a9e2293442e291b1262a717a5e50 [term 2 FOLLOWER]: Refusing update from remote peer 64fd1d8a3cd14e66ac94c4fbcceaf039: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:27:54.769109 32062 consensus_queue.cc:1048] T 378f56c32c1e46f99a874bd1adcbcf25 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:27:54.818280 31909 raft_consensus.cc:1275] T 1638e215a9ee4068a999e92023fff415 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 2 FOLLOWER]: Refusing update from remote peer 2ae3a9e2293442e291b1262a717a5e50: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:27:54.819551 32046 consensus_queue.cc:1048] T 1638e215a9ee4068a999e92023fff415 P 2ae3a9e2293442e291b1262a717a5e50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:27:54.828374 31981 raft_consensus.cc:1275] T 1638e215a9ee4068a999e92023fff415 P 07b97783285d4b22b6fd0f1d816ef930 [term 2 FOLLOWER]: Refusing update from remote peer 2ae3a9e2293442e291b1262a717a5e50: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:27:54.829913 32093 consensus_queue.cc:1048] T 1638e215a9ee4068a999e92023fff415 P 2ae3a9e2293442e291b1262a717a5e50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:27:54.855283 31981 raft_consensus.cc:1275] T 52322cf19f734f19a4ccfc8389f32b41 P 07b97783285d4b22b6fd0f1d816ef930 [term 2 FOLLOWER]: Refusing update from remote peer 64fd1d8a3cd14e66ac94c4fbcceaf039: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:27:54.856727 32062 consensus_queue.cc:1048] T 52322cf19f734f19a4ccfc8389f32b41 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:27:54.866158 31834 raft_consensus.cc:1275] T 52322cf19f734f19a4ccfc8389f32b41 P 2ae3a9e2293442e291b1262a717a5e50 [term 2 FOLLOWER]: Refusing update from remote peer 64fd1d8a3cd14e66ac94c4fbcceaf039: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:27:54.868135 32112 consensus_queue.cc:1048] T 52322cf19f734f19a4ccfc8389f32b41 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:27:56.872989 31732 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43858:
name: "soft_deleted_table"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
  rows: "\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q""\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q"
  indirect_data: """"
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W20260709 12:27:56.875265 31732 catalog_manager.cc:7077] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table soft_deleted_table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260709 12:27:56.930269 31819 tablet_service.cc:1511] Processing CreateTablet for tablet c11f552ceb904522856ef259018a8442 (DEFAULT_TABLE table=soft_deleted_table [id=945de90ed607462385a524cf47b507d2]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260709 12:27:56.931702 31819 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c11f552ceb904522856ef259018a8442. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:56.931702 31894 tablet_service.cc:1511] Processing CreateTablet for tablet c11f552ceb904522856ef259018a8442 (DEFAULT_TABLE table=soft_deleted_table [id=945de90ed607462385a524cf47b507d2]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260709 12:27:56.934064 31894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c11f552ceb904522856ef259018a8442. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:56.935418 31822 tablet_service.cc:1511] Processing CreateTablet for tablet 0d5a572ce9954c72b6d74a24da7de624 (DEFAULT_TABLE table=soft_deleted_table [id=945de90ed607462385a524cf47b507d2]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260709 12:27:56.933760 31973 tablet_service.cc:1511] Processing CreateTablet for tablet c11f552ceb904522856ef259018a8442 (DEFAULT_TABLE table=soft_deleted_table [id=945de90ed607462385a524cf47b507d2]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260709 12:27:56.936908 31973 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c11f552ceb904522856ef259018a8442. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:56.937424 31900 tablet_service.cc:1511] Processing CreateTablet for tablet 0d5a572ce9954c72b6d74a24da7de624 (DEFAULT_TABLE table=soft_deleted_table [id=945de90ed607462385a524cf47b507d2]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260709 12:27:56.938759 31900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d5a572ce9954c72b6d74a24da7de624. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:56.936908 31822 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d5a572ce9954c72b6d74a24da7de624. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:56.956089 31967 tablet_service.cc:1511] Processing CreateTablet for tablet 0d5a572ce9954c72b6d74a24da7de624 (DEFAULT_TABLE table=soft_deleted_table [id=945de90ed607462385a524cf47b507d2]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260709 12:27:56.958132 31967 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d5a572ce9954c72b6d74a24da7de624. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:56.966176 32131 tablet_bootstrap.cc:492] T c11f552ceb904522856ef259018a8442 P 2ae3a9e2293442e291b1262a717a5e50: Bootstrap starting.
I20260709 12:27:56.970999 31818 tablet_service.cc:1511] Processing CreateTablet for tablet 8eda1839ee054ee59ec8629c9139dd2e (DEFAULT_TABLE table=soft_deleted_table [id=945de90ed607462385a524cf47b507d2]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260709 12:27:56.972651 31818 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8eda1839ee054ee59ec8629c9139dd2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:56.974025 31892 tablet_service.cc:1511] Processing CreateTablet for tablet 8eda1839ee054ee59ec8629c9139dd2e (DEFAULT_TABLE table=soft_deleted_table [id=945de90ed607462385a524cf47b507d2]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260709 12:27:56.975430 31892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8eda1839ee054ee59ec8629c9139dd2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:56.975744 31966 tablet_service.cc:1511] Processing CreateTablet for tablet 8eda1839ee054ee59ec8629c9139dd2e (DEFAULT_TABLE table=soft_deleted_table [id=945de90ed607462385a524cf47b507d2]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260709 12:27:56.977226 31966 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8eda1839ee054ee59ec8629c9139dd2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:56.981513 31819 tablet_service.cc:1511] Processing CreateTablet for tablet 9fe7f1d426c94232a1ec785665ef1416 (DEFAULT_TABLE table=soft_deleted_table [id=945de90ed607462385a524cf47b507d2]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260709 12:27:56.982928 31819 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9fe7f1d426c94232a1ec785665ef1416. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:56.983186 31896 tablet_service.cc:1511] Processing CreateTablet for tablet 9fe7f1d426c94232a1ec785665ef1416 (DEFAULT_TABLE table=soft_deleted_table [id=945de90ed607462385a524cf47b507d2]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260709 12:27:56.984500 31896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9fe7f1d426c94232a1ec785665ef1416. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:57.002838 31971 tablet_service.cc:1511] Processing CreateTablet for tablet 9fe7f1d426c94232a1ec785665ef1416 (DEFAULT_TABLE table=soft_deleted_table [id=945de90ed607462385a524cf47b507d2]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260709 12:27:57.004343 31971 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9fe7f1d426c94232a1ec785665ef1416. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:57.005439 32132 tablet_bootstrap.cc:492] T 0d5a572ce9954c72b6d74a24da7de624 P 07b97783285d4b22b6fd0f1d816ef930: Bootstrap starting.
I20260709 12:27:57.006296 32131 tablet_bootstrap.cc:654] T c11f552ceb904522856ef259018a8442 P 2ae3a9e2293442e291b1262a717a5e50: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:57.012913 32132 tablet_bootstrap.cc:654] T 0d5a572ce9954c72b6d74a24da7de624 P 07b97783285d4b22b6fd0f1d816ef930: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:57.029021 31892 tablet_service.cc:1511] Processing CreateTablet for tablet b659027528154c4c86becb863718c732 (DEFAULT_TABLE table=soft_deleted_table [id=945de90ed607462385a524cf47b507d2]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260709 12:27:57.030470 31892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b659027528154c4c86becb863718c732. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:57.032474 32131 tablet_bootstrap.cc:492] T c11f552ceb904522856ef259018a8442 P 2ae3a9e2293442e291b1262a717a5e50: No bootstrap required, opened a new log
I20260709 12:27:57.033135 32131 ts_tablet_manager.cc:1403] T c11f552ceb904522856ef259018a8442 P 2ae3a9e2293442e291b1262a717a5e50: Time spent bootstrapping tablet: real 0.067s	user 0.013s	sys 0.000s
I20260709 12:27:57.036573 32131 raft_consensus.cc:359] T c11f552ceb904522856ef259018a8442 P 2ae3a9e2293442e291b1262a717a5e50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:57.037379 32131 raft_consensus.cc:385] T c11f552ceb904522856ef259018a8442 P 2ae3a9e2293442e291b1262a717a5e50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:57.037696 32131 raft_consensus.cc:740] T c11f552ceb904522856ef259018a8442 P 2ae3a9e2293442e291b1262a717a5e50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ae3a9e2293442e291b1262a717a5e50, State: Initialized, Role: FOLLOWER
I20260709 12:27:57.038522 32131 consensus_queue.cc:260] T c11f552ceb904522856ef259018a8442 P 2ae3a9e2293442e291b1262a717a5e50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:57.039338 31973 tablet_service.cc:1511] Processing CreateTablet for tablet b659027528154c4c86becb863718c732 (DEFAULT_TABLE table=soft_deleted_table [id=945de90ed607462385a524cf47b507d2]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260709 12:27:57.041074 32134 tablet_bootstrap.cc:492] T 0d5a572ce9954c72b6d74a24da7de624 P 64fd1d8a3cd14e66ac94c4fbcceaf039: Bootstrap starting.
I20260709 12:27:57.041445 31819 tablet_service.cc:1511] Processing CreateTablet for tablet 0ca01749cbc64fd987e4174047baac00 (DEFAULT_TABLE table=soft_deleted_table [id=945de90ed607462385a524cf47b507d2]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260709 12:27:57.043784 31819 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0ca01749cbc64fd987e4174047baac00. 1 dirs total, 0 dirs full, 0 dirs failed
W20260709 12:27:57.046267 31781 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:27:57.047866 32132 tablet_bootstrap.cc:492] T 0d5a572ce9954c72b6d74a24da7de624 P 07b97783285d4b22b6fd0f1d816ef930: No bootstrap required, opened a new log
I20260709 12:27:57.048519 32132 ts_tablet_manager.cc:1403] T 0d5a572ce9954c72b6d74a24da7de624 P 07b97783285d4b22b6fd0f1d816ef930: Time spent bootstrapping tablet: real 0.043s	user 0.016s	sys 0.005s
I20260709 12:27:57.051010 31822 tablet_service.cc:1511] Processing CreateTablet for tablet b659027528154c4c86becb863718c732 (DEFAULT_TABLE table=soft_deleted_table [id=945de90ed607462385a524cf47b507d2]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260709 12:27:57.052240 32132 raft_consensus.cc:359] T 0d5a572ce9954c72b6d74a24da7de624 P 07b97783285d4b22b6fd0f1d816ef930 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:57.052922 32131 ts_tablet_manager.cc:1434] T c11f552ceb904522856ef259018a8442 P 2ae3a9e2293442e291b1262a717a5e50: Time spent starting tablet: real 0.019s	user 0.007s	sys 0.000s
I20260709 12:27:57.053155 32132 raft_consensus.cc:385] T 0d5a572ce9954c72b6d74a24da7de624 P 07b97783285d4b22b6fd0f1d816ef930 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:57.053614 32132 raft_consensus.cc:740] T 0d5a572ce9954c72b6d74a24da7de624 P 07b97783285d4b22b6fd0f1d816ef930 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07b97783285d4b22b6fd0f1d816ef930, State: Initialized, Role: FOLLOWER
I20260709 12:27:57.054476 32132 consensus_queue.cc:260] T 0d5a572ce9954c72b6d74a24da7de624 P 07b97783285d4b22b6fd0f1d816ef930 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:57.052999 31822 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b659027528154c4c86becb863718c732. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:57.041244 31973 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b659027528154c4c86becb863718c732. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:57.058264 31896 tablet_service.cc:1511] Processing CreateTablet for tablet 0ca01749cbc64fd987e4174047baac00 (DEFAULT_TABLE table=soft_deleted_table [id=945de90ed607462385a524cf47b507d2]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260709 12:27:57.059701 31896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0ca01749cbc64fd987e4174047baac00. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:57.063038 32131 tablet_bootstrap.cc:492] T 0d5a572ce9954c72b6d74a24da7de624 P 2ae3a9e2293442e291b1262a717a5e50: Bootstrap starting.
I20260709 12:27:57.072973 32134 tablet_bootstrap.cc:654] T 0d5a572ce9954c72b6d74a24da7de624 P 64fd1d8a3cd14e66ac94c4fbcceaf039: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:57.075557 32131 tablet_bootstrap.cc:654] T 0d5a572ce9954c72b6d74a24da7de624 P 2ae3a9e2293442e291b1262a717a5e50: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:57.080031 32132 ts_tablet_manager.cc:1434] T 0d5a572ce9954c72b6d74a24da7de624 P 07b97783285d4b22b6fd0f1d816ef930: Time spent starting tablet: real 0.031s	user 0.009s	sys 0.000s
I20260709 12:27:57.090679 31818 tablet_service.cc:1511] Processing CreateTablet for tablet c4f237debd4942b198e077f8e7091088 (DEFAULT_TABLE table=soft_deleted_table [id=945de90ed607462385a524cf47b507d2]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260709 12:27:57.082226 31971 tablet_service.cc:1511] Processing CreateTablet for tablet 0ca01749cbc64fd987e4174047baac00 (DEFAULT_TABLE table=soft_deleted_table [id=945de90ed607462385a524cf47b507d2]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260709 12:27:57.093634 32132 tablet_bootstrap.cc:492] T 8eda1839ee054ee59ec8629c9139dd2e P 07b97783285d4b22b6fd0f1d816ef930: Bootstrap starting.
I20260709 12:27:57.096117 31892 tablet_service.cc:1511] Processing CreateTablet for tablet c4f237debd4942b198e077f8e7091088 (DEFAULT_TABLE table=soft_deleted_table [id=945de90ed607462385a524cf47b507d2]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260709 12:27:57.096824 31971 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0ca01749cbc64fd987e4174047baac00. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:57.097728 31892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c4f237debd4942b198e077f8e7091088. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:57.108065 31818 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c4f237debd4942b198e077f8e7091088. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:57.121172 31973 tablet_service.cc:1511] Processing CreateTablet for tablet c4f237debd4942b198e077f8e7091088 (DEFAULT_TABLE table=soft_deleted_table [id=945de90ed607462385a524cf47b507d2]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260709 12:27:57.123361 31973 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c4f237debd4942b198e077f8e7091088. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:57.127018 32132 tablet_bootstrap.cc:654] T 8eda1839ee054ee59ec8629c9139dd2e P 07b97783285d4b22b6fd0f1d816ef930: Neither blocks nor log segments found. Creating new log.
W20260709 12:27:57.133932 31801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:34419, user_credentials={real_user=slave}} blocked reactor thread for 49930.6us
I20260709 12:27:57.137856 31966 tablet_service.cc:1511] Processing CreateTablet for tablet ff138d12849e4f63a9b257f4395179de (DEFAULT_TABLE table=soft_deleted_table [id=945de90ed607462385a524cf47b507d2]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260709 12:27:57.139065 32134 tablet_bootstrap.cc:492] T 0d5a572ce9954c72b6d74a24da7de624 P 64fd1d8a3cd14e66ac94c4fbcceaf039: No bootstrap required, opened a new log
I20260709 12:27:57.139376 31966 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff138d12849e4f63a9b257f4395179de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:57.139674 32134 ts_tablet_manager.cc:1403] T 0d5a572ce9954c72b6d74a24da7de624 P 64fd1d8a3cd14e66ac94c4fbcceaf039: Time spent bootstrapping tablet: real 0.099s	user 0.007s	sys 0.015s
I20260709 12:27:57.141650 31822 tablet_service.cc:1511] Processing CreateTablet for tablet ff138d12849e4f63a9b257f4395179de (DEFAULT_TABLE table=soft_deleted_table [id=945de90ed607462385a524cf47b507d2]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260709 12:27:57.142987 31822 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff138d12849e4f63a9b257f4395179de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:57.145776 31900 tablet_service.cc:1511] Processing CreateTablet for tablet 703b2df19bf24c31a5acbe85178314ec (DEFAULT_TABLE table=soft_deleted_table [id=945de90ed607462385a524cf47b507d2]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260709 12:27:57.147502 31900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 703b2df19bf24c31a5acbe85178314ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:57.135432 31896 tablet_service.cc:1511] Processing CreateTablet for tablet ff138d12849e4f63a9b257f4395179de (DEFAULT_TABLE table=soft_deleted_table [id=945de90ed607462385a524cf47b507d2]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260709 12:27:57.150739 31896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff138d12849e4f63a9b257f4395179de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:57.154371 32134 raft_consensus.cc:359] T 0d5a572ce9954c72b6d74a24da7de624 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:57.155231 32134 raft_consensus.cc:385] T 0d5a572ce9954c72b6d74a24da7de624 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:57.155519 32134 raft_consensus.cc:740] T 0d5a572ce9954c72b6d74a24da7de624 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64fd1d8a3cd14e66ac94c4fbcceaf039, State: Initialized, Role: FOLLOWER
I20260709 12:27:57.156288 32134 consensus_queue.cc:260] T 0d5a572ce9954c72b6d74a24da7de624 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:57.161505 32134 ts_tablet_manager.cc:1434] T 0d5a572ce9954c72b6d74a24da7de624 P 64fd1d8a3cd14e66ac94c4fbcceaf039: Time spent starting tablet: real 0.021s	user 0.005s	sys 0.001s
I20260709 12:27:57.161731 31971 tablet_service.cc:1511] Processing CreateTablet for tablet 703b2df19bf24c31a5acbe85178314ec (DEFAULT_TABLE table=soft_deleted_table [id=945de90ed607462385a524cf47b507d2]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260709 12:27:57.164734 31971 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 703b2df19bf24c31a5acbe85178314ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:57.137907 31819 tablet_service.cc:1511] Processing CreateTablet for tablet 703b2df19bf24c31a5acbe85178314ec (DEFAULT_TABLE table=soft_deleted_table [id=945de90ed607462385a524cf47b507d2]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260709 12:27:57.168363 31819 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 703b2df19bf24c31a5acbe85178314ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:57.173393 32134 tablet_bootstrap.cc:492] T c11f552ceb904522856ef259018a8442 P 64fd1d8a3cd14e66ac94c4fbcceaf039: Bootstrap starting.
I20260709 12:27:57.173697 32132 tablet_bootstrap.cc:492] T 8eda1839ee054ee59ec8629c9139dd2e P 07b97783285d4b22b6fd0f1d816ef930: No bootstrap required, opened a new log
I20260709 12:27:57.174335 32132 ts_tablet_manager.cc:1403] T 8eda1839ee054ee59ec8629c9139dd2e P 07b97783285d4b22b6fd0f1d816ef930: Time spent bootstrapping tablet: real 0.082s	user 0.024s	sys 0.008s
I20260709 12:27:57.177764 32132 raft_consensus.cc:359] T 8eda1839ee054ee59ec8629c9139dd2e P 07b97783285d4b22b6fd0f1d816ef930 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:57.178608 32132 raft_consensus.cc:385] T 8eda1839ee054ee59ec8629c9139dd2e P 07b97783285d4b22b6fd0f1d816ef930 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:57.178960 32132 raft_consensus.cc:740] T 8eda1839ee054ee59ec8629c9139dd2e P 07b97783285d4b22b6fd0f1d816ef930 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07b97783285d4b22b6fd0f1d816ef930, State: Initialized, Role: FOLLOWER
I20260709 12:27:57.179960 32132 consensus_queue.cc:260] T 8eda1839ee054ee59ec8629c9139dd2e P 07b97783285d4b22b6fd0f1d816ef930 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:57.180581 32134 tablet_bootstrap.cc:654] T c11f552ceb904522856ef259018a8442 P 64fd1d8a3cd14e66ac94c4fbcceaf039: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:57.182873 32132 ts_tablet_manager.cc:1434] T 8eda1839ee054ee59ec8629c9139dd2e P 07b97783285d4b22b6fd0f1d816ef930: Time spent starting tablet: real 0.008s	user 0.006s	sys 0.000s
I20260709 12:27:57.184139 32132 tablet_bootstrap.cc:492] T c11f552ceb904522856ef259018a8442 P 07b97783285d4b22b6fd0f1d816ef930: Bootstrap starting.
I20260709 12:27:57.136418 32131 tablet_bootstrap.cc:492] T 0d5a572ce9954c72b6d74a24da7de624 P 2ae3a9e2293442e291b1262a717a5e50: No bootstrap required, opened a new log
I20260709 12:27:57.188303 32131 ts_tablet_manager.cc:1403] T 0d5a572ce9954c72b6d74a24da7de624 P 2ae3a9e2293442e291b1262a717a5e50: Time spent bootstrapping tablet: real 0.125s	user 0.020s	sys 0.002s
I20260709 12:27:57.188812 32052 raft_consensus.cc:493] T 0d5a572ce9954c72b6d74a24da7de624 P 07b97783285d4b22b6fd0f1d816ef930 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:57.189388 32052 raft_consensus.cc:515] T 0d5a572ce9954c72b6d74a24da7de624 P 07b97783285d4b22b6fd0f1d816ef930 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:57.191614 32052 leader_election.cc:290] T 0d5a572ce9954c72b6d74a24da7de624 P 07b97783285d4b22b6fd0f1d816ef930 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2ae3a9e2293442e291b1262a717a5e50 (127.16.29.65:44289), 64fd1d8a3cd14e66ac94c4fbcceaf039 (127.16.29.66:42163)
I20260709 12:27:57.193492 32132 tablet_bootstrap.cc:654] T c11f552ceb904522856ef259018a8442 P 07b97783285d4b22b6fd0f1d816ef930: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:57.197048 32134 tablet_bootstrap.cc:492] T c11f552ceb904522856ef259018a8442 P 64fd1d8a3cd14e66ac94c4fbcceaf039: No bootstrap required, opened a new log
I20260709 12:27:57.202955 32134 ts_tablet_manager.cc:1403] T c11f552ceb904522856ef259018a8442 P 64fd1d8a3cd14e66ac94c4fbcceaf039: Time spent bootstrapping tablet: real 0.030s	user 0.019s	sys 0.007s
I20260709 12:27:57.196045 31909 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d5a572ce9954c72b6d74a24da7de624" candidate_uuid: "07b97783285d4b22b6fd0f1d816ef930" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" is_pre_election: true
I20260709 12:27:57.203841 31909 raft_consensus.cc:2468] T 0d5a572ce9954c72b6d74a24da7de624 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 07b97783285d4b22b6fd0f1d816ef930 in term 0.
I20260709 12:27:57.194808 31834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d5a572ce9954c72b6d74a24da7de624" candidate_uuid: "07b97783285d4b22b6fd0f1d816ef930" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ae3a9e2293442e291b1262a717a5e50" is_pre_election: true
I20260709 12:27:57.206557 31947 leader_election.cc:304] T 0d5a572ce9954c72b6d74a24da7de624 P 07b97783285d4b22b6fd0f1d816ef930 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07b97783285d4b22b6fd0f1d816ef930, 64fd1d8a3cd14e66ac94c4fbcceaf039; no voters: 
I20260709 12:27:57.207532 32052 raft_consensus.cc:2804] T 0d5a572ce9954c72b6d74a24da7de624 P 07b97783285d4b22b6fd0f1d816ef930 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:57.208024 32052 raft_consensus.cc:493] T 0d5a572ce9954c72b6d74a24da7de624 P 07b97783285d4b22b6fd0f1d816ef930 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:27:57.208209 31949 leader_election.cc:343] T 0d5a572ce9954c72b6d74a24da7de624 P 07b97783285d4b22b6fd0f1d816ef930 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2ae3a9e2293442e291b1262a717a5e50 (127.16.29.65:44289): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:57.208442 32052 raft_consensus.cc:3060] T 0d5a572ce9954c72b6d74a24da7de624 P 07b97783285d4b22b6fd0f1d816ef930 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:57.212358 32131 raft_consensus.cc:359] T 0d5a572ce9954c72b6d74a24da7de624 P 2ae3a9e2293442e291b1262a717a5e50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:57.216077 32052 raft_consensus.cc:515] T 0d5a572ce9954c72b6d74a24da7de624 P 07b97783285d4b22b6fd0f1d816ef930 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:57.211522 32134 raft_consensus.cc:359] T c11f552ceb904522856ef259018a8442 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:57.224767 32131 raft_consensus.cc:385] T 0d5a572ce9954c72b6d74a24da7de624 P 2ae3a9e2293442e291b1262a717a5e50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:57.225142 32134 raft_consensus.cc:385] T c11f552ceb904522856ef259018a8442 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:57.225183 32131 raft_consensus.cc:740] T 0d5a572ce9954c72b6d74a24da7de624 P 2ae3a9e2293442e291b1262a717a5e50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ae3a9e2293442e291b1262a717a5e50, State: Initialized, Role: FOLLOWER
I20260709 12:27:57.225775 32134 raft_consensus.cc:740] T c11f552ceb904522856ef259018a8442 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64fd1d8a3cd14e66ac94c4fbcceaf039, State: Initialized, Role: FOLLOWER
I20260709 12:27:57.226459 32131 consensus_queue.cc:260] T 0d5a572ce9954c72b6d74a24da7de624 P 2ae3a9e2293442e291b1262a717a5e50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:57.226950 32132 tablet_bootstrap.cc:492] T c11f552ceb904522856ef259018a8442 P 07b97783285d4b22b6fd0f1d816ef930: No bootstrap required, opened a new log
I20260709 12:27:57.227489 31834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d5a572ce9954c72b6d74a24da7de624" candidate_uuid: "07b97783285d4b22b6fd0f1d816ef930" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ae3a9e2293442e291b1262a717a5e50"
I20260709 12:27:57.228076 31834 raft_consensus.cc:3060] T 0d5a572ce9954c72b6d74a24da7de624 P 2ae3a9e2293442e291b1262a717a5e50 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:57.228559 32132 ts_tablet_manager.cc:1403] T c11f552ceb904522856ef259018a8442 P 07b97783285d4b22b6fd0f1d816ef930: Time spent bootstrapping tablet: real 0.045s	user 0.022s	sys 0.014s
I20260709 12:27:57.228425 31909 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d5a572ce9954c72b6d74a24da7de624" candidate_uuid: "07b97783285d4b22b6fd0f1d816ef930" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039"
I20260709 12:27:57.229076 32131 ts_tablet_manager.cc:1434] T 0d5a572ce9954c72b6d74a24da7de624 P 2ae3a9e2293442e291b1262a717a5e50: Time spent starting tablet: real 0.040s	user 0.004s	sys 0.003s
I20260709 12:27:57.229097 31909 raft_consensus.cc:3060] T 0d5a572ce9954c72b6d74a24da7de624 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:57.230366 32131 tablet_bootstrap.cc:492] T 9fe7f1d426c94232a1ec785665ef1416 P 2ae3a9e2293442e291b1262a717a5e50: Bootstrap starting.
I20260709 12:27:57.231832 32132 raft_consensus.cc:359] T c11f552ceb904522856ef259018a8442 P 07b97783285d4b22b6fd0f1d816ef930 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:57.232697 32132 raft_consensus.cc:385] T c11f552ceb904522856ef259018a8442 P 07b97783285d4b22b6fd0f1d816ef930 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:57.232990 32132 raft_consensus.cc:740] T c11f552ceb904522856ef259018a8442 P 07b97783285d4b22b6fd0f1d816ef930 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07b97783285d4b22b6fd0f1d816ef930, State: Initialized, Role: FOLLOWER
I20260709 12:27:57.233512 32134 consensus_queue.cc:260] T c11f552ceb904522856ef259018a8442 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:57.238545 32134 ts_tablet_manager.cc:1434] T c11f552ceb904522856ef259018a8442 P 64fd1d8a3cd14e66ac94c4fbcceaf039: Time spent starting tablet: real 0.035s	user 0.003s	sys 0.003s
I20260709 12:27:57.236420 31834 raft_consensus.cc:2468] T 0d5a572ce9954c72b6d74a24da7de624 P 2ae3a9e2293442e291b1262a717a5e50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07b97783285d4b22b6fd0f1d816ef930 in term 1.
I20260709 12:27:57.239709 31909 raft_consensus.cc:2468] T 0d5a572ce9954c72b6d74a24da7de624 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07b97783285d4b22b6fd0f1d816ef930 in term 1.
I20260709 12:27:57.240723 32134 tablet_bootstrap.cc:492] T 8eda1839ee054ee59ec8629c9139dd2e P 64fd1d8a3cd14e66ac94c4fbcceaf039: Bootstrap starting.
I20260709 12:27:57.240226 32052 leader_election.cc:290] T 0d5a572ce9954c72b6d74a24da7de624 P 07b97783285d4b22b6fd0f1d816ef930 [CANDIDATE]: Term 1 election: Requested vote from peers 2ae3a9e2293442e291b1262a717a5e50 (127.16.29.65:44289), 64fd1d8a3cd14e66ac94c4fbcceaf039 (127.16.29.66:42163)
I20260709 12:27:57.242355 31949 leader_election.cc:304] T 0d5a572ce9954c72b6d74a24da7de624 P 07b97783285d4b22b6fd0f1d816ef930 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07b97783285d4b22b6fd0f1d816ef930, 2ae3a9e2293442e291b1262a717a5e50; no voters: 
I20260709 12:27:57.245502 32052 raft_consensus.cc:2804] T 0d5a572ce9954c72b6d74a24da7de624 P 07b97783285d4b22b6fd0f1d816ef930 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:57.238579 32131 tablet_bootstrap.cc:654] T 9fe7f1d426c94232a1ec785665ef1416 P 2ae3a9e2293442e291b1262a717a5e50: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:57.253823 32134 tablet_bootstrap.cc:654] T 8eda1839ee054ee59ec8629c9139dd2e P 64fd1d8a3cd14e66ac94c4fbcceaf039: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:57.233768 32132 consensus_queue.cc:260] T c11f552ceb904522856ef259018a8442 P 07b97783285d4b22b6fd0f1d816ef930 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:57.260840 32132 ts_tablet_manager.cc:1434] T c11f552ceb904522856ef259018a8442 P 07b97783285d4b22b6fd0f1d816ef930: Time spent starting tablet: real 0.032s	user 0.007s	sys 0.003s
I20260709 12:27:57.262112 32132 tablet_bootstrap.cc:492] T 9fe7f1d426c94232a1ec785665ef1416 P 07b97783285d4b22b6fd0f1d816ef930: Bootstrap starting.
I20260709 12:27:57.266535 32131 tablet_bootstrap.cc:492] T 9fe7f1d426c94232a1ec785665ef1416 P 2ae3a9e2293442e291b1262a717a5e50: No bootstrap required, opened a new log
I20260709 12:27:57.267153 32131 ts_tablet_manager.cc:1403] T 9fe7f1d426c94232a1ec785665ef1416 P 2ae3a9e2293442e291b1262a717a5e50: Time spent bootstrapping tablet: real 0.037s	user 0.010s	sys 0.005s
I20260709 12:27:57.269335 32132 tablet_bootstrap.cc:654] T 9fe7f1d426c94232a1ec785665ef1416 P 07b97783285d4b22b6fd0f1d816ef930: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:57.270725 32131 raft_consensus.cc:359] T 9fe7f1d426c94232a1ec785665ef1416 P 2ae3a9e2293442e291b1262a717a5e50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:57.271555 32131 raft_consensus.cc:385] T 9fe7f1d426c94232a1ec785665ef1416 P 2ae3a9e2293442e291b1262a717a5e50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:57.271863 32131 raft_consensus.cc:740] T 9fe7f1d426c94232a1ec785665ef1416 P 2ae3a9e2293442e291b1262a717a5e50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ae3a9e2293442e291b1262a717a5e50, State: Initialized, Role: FOLLOWER
I20260709 12:27:57.275272 32131 consensus_queue.cc:260] T 9fe7f1d426c94232a1ec785665ef1416 P 2ae3a9e2293442e291b1262a717a5e50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:57.277974 32131 ts_tablet_manager.cc:1434] T 9fe7f1d426c94232a1ec785665ef1416 P 2ae3a9e2293442e291b1262a717a5e50: Time spent starting tablet: real 0.010s	user 0.009s	sys 0.000s
I20260709 12:27:57.279058 32131 tablet_bootstrap.cc:492] T 8eda1839ee054ee59ec8629c9139dd2e P 2ae3a9e2293442e291b1262a717a5e50: Bootstrap starting.
I20260709 12:27:57.285178 32120 raft_consensus.cc:493] T c11f552ceb904522856ef259018a8442 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:57.285970 32120 raft_consensus.cc:515] T c11f552ceb904522856ef259018a8442 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:57.286907 32131 tablet_bootstrap.cc:654] T 8eda1839ee054ee59ec8629c9139dd2e P 2ae3a9e2293442e291b1262a717a5e50: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:57.297122 32120 leader_election.cc:290] T c11f552ceb904522856ef259018a8442 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2ae3a9e2293442e291b1262a717a5e50 (127.16.29.65:44289), 07b97783285d4b22b6fd0f1d816ef930 (127.16.29.67:34419)
I20260709 12:27:57.296984 31834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c11f552ceb904522856ef259018a8442" candidate_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ae3a9e2293442e291b1262a717a5e50" is_pre_election: true
I20260709 12:27:57.298763 31834 raft_consensus.cc:2468] T c11f552ceb904522856ef259018a8442 P 2ae3a9e2293442e291b1262a717a5e50 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64fd1d8a3cd14e66ac94c4fbcceaf039 in term 0.
I20260709 12:27:57.297004 31981 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c11f552ceb904522856ef259018a8442" candidate_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07b97783285d4b22b6fd0f1d816ef930" is_pre_election: true
I20260709 12:27:57.300179 31981 raft_consensus.cc:2468] T c11f552ceb904522856ef259018a8442 P 07b97783285d4b22b6fd0f1d816ef930 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64fd1d8a3cd14e66ac94c4fbcceaf039 in term 0.
I20260709 12:27:57.301604 31876 leader_election.cc:304] T c11f552ceb904522856ef259018a8442 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07b97783285d4b22b6fd0f1d816ef930, 64fd1d8a3cd14e66ac94c4fbcceaf039; no voters: 
I20260709 12:27:57.301827 32052 raft_consensus.cc:697] T 0d5a572ce9954c72b6d74a24da7de624 P 07b97783285d4b22b6fd0f1d816ef930 [term 1 LEADER]: Becoming Leader. State: Replica: 07b97783285d4b22b6fd0f1d816ef930, State: Running, Role: LEADER
I20260709 12:27:57.302441 32147 raft_consensus.cc:2804] T c11f552ceb904522856ef259018a8442 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:57.303103 32134 tablet_bootstrap.cc:492] T 8eda1839ee054ee59ec8629c9139dd2e P 64fd1d8a3cd14e66ac94c4fbcceaf039: No bootstrap required, opened a new log
I20260709 12:27:57.302884 32147 raft_consensus.cc:493] T c11f552ceb904522856ef259018a8442 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:57.302815 32052 consensus_queue.cc:237] T 0d5a572ce9954c72b6d74a24da7de624 P 07b97783285d4b22b6fd0f1d816ef930 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:57.303606 32147 raft_consensus.cc:3060] T c11f552ceb904522856ef259018a8442 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:57.303731 32134 ts_tablet_manager.cc:1403] T 8eda1839ee054ee59ec8629c9139dd2e P 64fd1d8a3cd14e66ac94c4fbcceaf039: Time spent bootstrapping tablet: real 0.063s	user 0.005s	sys 0.006s
I20260709 12:27:57.308059 32134 raft_consensus.cc:359] T 8eda1839ee054ee59ec8629c9139dd2e P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:57.309108 32134 raft_consensus.cc:385] T 8eda1839ee054ee59ec8629c9139dd2e P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:57.309549 32134 raft_consensus.cc:740] T 8eda1839ee054ee59ec8629c9139dd2e P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64fd1d8a3cd14e66ac94c4fbcceaf039, State: Initialized, Role: FOLLOWER
I20260709 12:27:57.310802 32134 consensus_queue.cc:260] T 8eda1839ee054ee59ec8629c9139dd2e P 64fd1d8a3cd14e66ac94c4fbcceaf039 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:57.318512 32134 ts_tablet_manager.cc:1434] T 8eda1839ee054ee59ec8629c9139dd2e P 64fd1d8a3cd14e66ac94c4fbcceaf039: Time spent starting tablet: real 0.014s	user 0.006s	sys 0.002s
I20260709 12:27:57.319149 32131 tablet_bootstrap.cc:492] T 8eda1839ee054ee59ec8629c9139dd2e P 2ae3a9e2293442e291b1262a717a5e50: No bootstrap required, opened a new log
I20260709 12:27:57.319751 32131 ts_tablet_manager.cc:1403] T 8eda1839ee054ee59ec8629c9139dd2e P 2ae3a9e2293442e291b1262a717a5e50: Time spent bootstrapping tablet: real 0.041s	user 0.006s	sys 0.020s
I20260709 12:27:57.319903 32132 tablet_bootstrap.cc:492] T 9fe7f1d426c94232a1ec785665ef1416 P 07b97783285d4b22b6fd0f1d816ef930: No bootstrap required, opened a new log
I20260709 12:27:57.320711 32132 ts_tablet_manager.cc:1403] T 9fe7f1d426c94232a1ec785665ef1416 P 07b97783285d4b22b6fd0f1d816ef930: Time spent bootstrapping tablet: real 0.059s	user 0.024s	sys 0.011s
I20260709 12:27:57.319780 31730 catalog_manager.cc:5697] T 0d5a572ce9954c72b6d74a24da7de624 P 07b97783285d4b22b6fd0f1d816ef930 reported cstate change: term changed from 0 to 1, leader changed from <none> to 07b97783285d4b22b6fd0f1d816ef930 (127.16.29.67). New cstate: current_term: 1 leader_uuid: "07b97783285d4b22b6fd0f1d816ef930" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } health_report { overall_health: HEALTHY } } }
I20260709 12:27:57.319892 32134 tablet_bootstrap.cc:492] T 9fe7f1d426c94232a1ec785665ef1416 P 64fd1d8a3cd14e66ac94c4fbcceaf039: Bootstrap starting.
I20260709 12:27:57.323427 32131 raft_consensus.cc:359] T 8eda1839ee054ee59ec8629c9139dd2e P 2ae3a9e2293442e291b1262a717a5e50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:57.324440 32131 raft_consensus.cc:385] T 8eda1839ee054ee59ec8629c9139dd2e P 2ae3a9e2293442e291b1262a717a5e50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:57.324793 32131 raft_consensus.cc:740] T 8eda1839ee054ee59ec8629c9139dd2e P 2ae3a9e2293442e291b1262a717a5e50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ae3a9e2293442e291b1262a717a5e50, State: Initialized, Role: FOLLOWER
I20260709 12:27:57.325465 32147 raft_consensus.cc:515] T c11f552ceb904522856ef259018a8442 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:57.328289 32134 tablet_bootstrap.cc:654] T 9fe7f1d426c94232a1ec785665ef1416 P 64fd1d8a3cd14e66ac94c4fbcceaf039: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:57.328682 31833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c11f552ceb904522856ef259018a8442" candidate_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ae3a9e2293442e291b1262a717a5e50"
I20260709 12:27:57.329339 31833 raft_consensus.cc:3060] T c11f552ceb904522856ef259018a8442 P 2ae3a9e2293442e291b1262a717a5e50 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:57.325739 32131 consensus_queue.cc:260] T 8eda1839ee054ee59ec8629c9139dd2e P 2ae3a9e2293442e291b1262a717a5e50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:57.337208 31833 raft_consensus.cc:2468] T c11f552ceb904522856ef259018a8442 P 2ae3a9e2293442e291b1262a717a5e50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64fd1d8a3cd14e66ac94c4fbcceaf039 in term 1.
I20260709 12:27:57.338433 31875 leader_election.cc:304] T c11f552ceb904522856ef259018a8442 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ae3a9e2293442e291b1262a717a5e50, 64fd1d8a3cd14e66ac94c4fbcceaf039; no voters: 
I20260709 12:27:57.339241 32120 raft_consensus.cc:2804] T c11f552ceb904522856ef259018a8442 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:57.339752 32120 raft_consensus.cc:697] T c11f552ceb904522856ef259018a8442 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 1 LEADER]: Becoming Leader. State: Replica: 64fd1d8a3cd14e66ac94c4fbcceaf039, State: Running, Role: LEADER
I20260709 12:27:57.339710 32147 leader_election.cc:290] T c11f552ceb904522856ef259018a8442 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [CANDIDATE]: Term 1 election: Requested vote from peers 2ae3a9e2293442e291b1262a717a5e50 (127.16.29.65:44289), 07b97783285d4b22b6fd0f1d816ef930 (127.16.29.67:34419)
I20260709 12:27:57.341029 31981 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c11f552ceb904522856ef259018a8442" candidate_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07b97783285d4b22b6fd0f1d816ef930"
I20260709 12:27:57.340788 32120 consensus_queue.cc:237] T c11f552ceb904522856ef259018a8442 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:57.341667 31981 raft_consensus.cc:3060] T c11f552ceb904522856ef259018a8442 P 07b97783285d4b22b6fd0f1d816ef930 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:57.338724 32132 raft_consensus.cc:359] T 9fe7f1d426c94232a1ec785665ef1416 P 07b97783285d4b22b6fd0f1d816ef930 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:57.346055 32132 raft_consensus.cc:385] T 9fe7f1d426c94232a1ec785665ef1416 P 07b97783285d4b22b6fd0f1d816ef930 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:57.346871 32132 raft_consensus.cc:740] T 9fe7f1d426c94232a1ec785665ef1416 P 07b97783285d4b22b6fd0f1d816ef930 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07b97783285d4b22b6fd0f1d816ef930, State: Initialized, Role: FOLLOWER
I20260709 12:27:57.348755 32132 consensus_queue.cc:260] T 9fe7f1d426c94232a1ec785665ef1416 P 07b97783285d4b22b6fd0f1d816ef930 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:57.349325 32131 ts_tablet_manager.cc:1434] T 8eda1839ee054ee59ec8629c9139dd2e P 2ae3a9e2293442e291b1262a717a5e50: Time spent starting tablet: real 0.029s	user 0.007s	sys 0.001s
I20260709 12:27:57.354522 32149 raft_consensus.cc:493] T 8eda1839ee054ee59ec8629c9139dd2e P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:57.355173 32149 raft_consensus.cc:515] T 8eda1839ee054ee59ec8629c9139dd2e P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:57.357947 32149 leader_election.cc:290] T 8eda1839ee054ee59ec8629c9139dd2e P 64fd1d8a3cd14e66ac94c4fbcceaf039 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2ae3a9e2293442e291b1262a717a5e50 (127.16.29.65:44289), 07b97783285d4b22b6fd0f1d816ef930 (127.16.29.67:34419)
I20260709 12:27:57.359802 31834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8eda1839ee054ee59ec8629c9139dd2e" candidate_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ae3a9e2293442e291b1262a717a5e50" is_pre_election: true
I20260709 12:27:57.360515 31834 raft_consensus.cc:2468] T 8eda1839ee054ee59ec8629c9139dd2e P 2ae3a9e2293442e291b1262a717a5e50 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64fd1d8a3cd14e66ac94c4fbcceaf039 in term 0.
I20260709 12:27:57.362689 31984 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8eda1839ee054ee59ec8629c9139dd2e" candidate_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07b97783285d4b22b6fd0f1d816ef930" is_pre_election: true
I20260709 12:27:57.363616 31984 raft_consensus.cc:2468] T 8eda1839ee054ee59ec8629c9139dd2e P 07b97783285d4b22b6fd0f1d816ef930 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64fd1d8a3cd14e66ac94c4fbcceaf039 in term 0.
I20260709 12:27:57.365949 31875 leader_election.cc:304] T 8eda1839ee054ee59ec8629c9139dd2e P 64fd1d8a3cd14e66ac94c4fbcceaf039 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ae3a9e2293442e291b1262a717a5e50, 64fd1d8a3cd14e66ac94c4fbcceaf039; no voters: 
I20260709 12:27:57.367110 32132 ts_tablet_manager.cc:1434] T 9fe7f1d426c94232a1ec785665ef1416 P 07b97783285d4b22b6fd0f1d816ef930: Time spent starting tablet: real 0.031s	user 0.000s	sys 0.006s
I20260709 12:27:57.367623 32131 tablet_bootstrap.cc:492] T b659027528154c4c86becb863718c732 P 2ae3a9e2293442e291b1262a717a5e50: Bootstrap starting.
I20260709 12:27:57.368183 32149 raft_consensus.cc:2804] T 8eda1839ee054ee59ec8629c9139dd2e P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:57.370283 32149 raft_consensus.cc:493] T 8eda1839ee054ee59ec8629c9139dd2e P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:57.372532 32149 raft_consensus.cc:3060] T 8eda1839ee054ee59ec8629c9139dd2e P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:57.374289 32132 tablet_bootstrap.cc:492] T b659027528154c4c86becb863718c732 P 07b97783285d4b22b6fd0f1d816ef930: Bootstrap starting.
I20260709 12:27:57.379464 31981 raft_consensus.cc:2468] T c11f552ceb904522856ef259018a8442 P 07b97783285d4b22b6fd0f1d816ef930 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64fd1d8a3cd14e66ac94c4fbcceaf039 in term 1.
I20260709 12:27:57.381356 32149 raft_consensus.cc:515] T 8eda1839ee054ee59ec8629c9139dd2e P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:57.384377 31833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8eda1839ee054ee59ec8629c9139dd2e" candidate_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ae3a9e2293442e291b1262a717a5e50"
I20260709 12:27:57.385042 31833 raft_consensus.cc:3060] T 8eda1839ee054ee59ec8629c9139dd2e P 2ae3a9e2293442e291b1262a717a5e50 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:57.386137 31732 catalog_manager.cc:5697] T c11f552ceb904522856ef259018a8442 P 64fd1d8a3cd14e66ac94c4fbcceaf039 reported cstate change: term changed from 0 to 1, leader changed from <none> to 64fd1d8a3cd14e66ac94c4fbcceaf039 (127.16.29.66). New cstate: current_term: 1 leader_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:57.387272 32134 tablet_bootstrap.cc:492] T 9fe7f1d426c94232a1ec785665ef1416 P 64fd1d8a3cd14e66ac94c4fbcceaf039: No bootstrap required, opened a new log
I20260709 12:27:57.387815 32134 ts_tablet_manager.cc:1403] T 9fe7f1d426c94232a1ec785665ef1416 P 64fd1d8a3cd14e66ac94c4fbcceaf039: Time spent bootstrapping tablet: real 0.068s	user 0.022s	sys 0.011s
I20260709 12:27:57.388247 32132 tablet_bootstrap.cc:654] T b659027528154c4c86becb863718c732 P 07b97783285d4b22b6fd0f1d816ef930: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:57.378432 32131 tablet_bootstrap.cc:654] T b659027528154c4c86becb863718c732 P 2ae3a9e2293442e291b1262a717a5e50: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:57.391513 32134 raft_consensus.cc:359] T 9fe7f1d426c94232a1ec785665ef1416 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:57.392365 32134 raft_consensus.cc:385] T 9fe7f1d426c94232a1ec785665ef1416 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:57.392712 32134 raft_consensus.cc:740] T 9fe7f1d426c94232a1ec785665ef1416 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64fd1d8a3cd14e66ac94c4fbcceaf039, State: Initialized, Role: FOLLOWER
I20260709 12:27:57.393913 32134 consensus_queue.cc:260] T 9fe7f1d426c94232a1ec785665ef1416 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:57.393429 31833 raft_consensus.cc:2468] T 8eda1839ee054ee59ec8629c9139dd2e P 2ae3a9e2293442e291b1262a717a5e50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64fd1d8a3cd14e66ac94c4fbcceaf039 in term 1.
I20260709 12:27:57.396268 31875 leader_election.cc:304] T 8eda1839ee054ee59ec8629c9139dd2e P 64fd1d8a3cd14e66ac94c4fbcceaf039 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ae3a9e2293442e291b1262a717a5e50, 64fd1d8a3cd14e66ac94c4fbcceaf039; no voters: 
I20260709 12:27:57.397022 32134 ts_tablet_manager.cc:1434] T 9fe7f1d426c94232a1ec785665ef1416 P 64fd1d8a3cd14e66ac94c4fbcceaf039: Time spent starting tablet: real 0.009s	user 0.007s	sys 0.000s
I20260709 12:27:57.397744 32120 raft_consensus.cc:2804] T 8eda1839ee054ee59ec8629c9139dd2e P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:57.398254 32134 tablet_bootstrap.cc:492] T b659027528154c4c86becb863718c732 P 64fd1d8a3cd14e66ac94c4fbcceaf039: Bootstrap starting.
I20260709 12:27:57.398342 31981 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8eda1839ee054ee59ec8629c9139dd2e" candidate_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07b97783285d4b22b6fd0f1d816ef930"
I20260709 12:27:57.398452 32120 raft_consensus.cc:697] T 8eda1839ee054ee59ec8629c9139dd2e P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 1 LEADER]: Becoming Leader. State: Replica: 64fd1d8a3cd14e66ac94c4fbcceaf039, State: Running, Role: LEADER
I20260709 12:27:57.399108 31981 raft_consensus.cc:3060] T 8eda1839ee054ee59ec8629c9139dd2e P 07b97783285d4b22b6fd0f1d816ef930 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:57.400310 32120 consensus_queue.cc:237] T 8eda1839ee054ee59ec8629c9139dd2e P 64fd1d8a3cd14e66ac94c4fbcceaf039 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:57.394447 32149 leader_election.cc:290] T 8eda1839ee054ee59ec8629c9139dd2e P 64fd1d8a3cd14e66ac94c4fbcceaf039 [CANDIDATE]: Term 1 election: Requested vote from peers 2ae3a9e2293442e291b1262a717a5e50 (127.16.29.65:44289), 07b97783285d4b22b6fd0f1d816ef930 (127.16.29.67:34419)
I20260709 12:27:57.406415 32134 tablet_bootstrap.cc:654] T b659027528154c4c86becb863718c732 P 64fd1d8a3cd14e66ac94c4fbcceaf039: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:57.411348 31981 raft_consensus.cc:2468] T 8eda1839ee054ee59ec8629c9139dd2e P 07b97783285d4b22b6fd0f1d816ef930 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64fd1d8a3cd14e66ac94c4fbcceaf039 in term 1.
I20260709 12:27:57.422760 32131 tablet_bootstrap.cc:492] T b659027528154c4c86becb863718c732 P 2ae3a9e2293442e291b1262a717a5e50: No bootstrap required, opened a new log
I20260709 12:27:57.423484 32131 ts_tablet_manager.cc:1403] T b659027528154c4c86becb863718c732 P 2ae3a9e2293442e291b1262a717a5e50: Time spent bootstrapping tablet: real 0.056s	user 0.018s	sys 0.009s
I20260709 12:27:57.423714 32134 tablet_bootstrap.cc:492] T b659027528154c4c86becb863718c732 P 64fd1d8a3cd14e66ac94c4fbcceaf039: No bootstrap required, opened a new log
I20260709 12:27:57.424412 32134 ts_tablet_manager.cc:1403] T b659027528154c4c86becb863718c732 P 64fd1d8a3cd14e66ac94c4fbcceaf039: Time spent bootstrapping tablet: real 0.026s	user 0.014s	sys 0.011s
I20260709 12:27:57.424708 32132 tablet_bootstrap.cc:492] T b659027528154c4c86becb863718c732 P 07b97783285d4b22b6fd0f1d816ef930: No bootstrap required, opened a new log
I20260709 12:27:57.425345 32132 ts_tablet_manager.cc:1403] T b659027528154c4c86becb863718c732 P 07b97783285d4b22b6fd0f1d816ef930: Time spent bootstrapping tablet: real 0.051s	user 0.012s	sys 0.014s
I20260709 12:27:57.427171 32131 raft_consensus.cc:359] T b659027528154c4c86becb863718c732 P 2ae3a9e2293442e291b1262a717a5e50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:57.424299 31732 catalog_manager.cc:5697] T 8eda1839ee054ee59ec8629c9139dd2e P 64fd1d8a3cd14e66ac94c4fbcceaf039 reported cstate change: term changed from 0 to 1, leader changed from <none> to 64fd1d8a3cd14e66ac94c4fbcceaf039 (127.16.29.66). New cstate: current_term: 1 leader_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:57.427824 32134 raft_consensus.cc:359] T b659027528154c4c86becb863718c732 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:57.428685 32134 raft_consensus.cc:385] T b659027528154c4c86becb863718c732 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:57.429005 32134 raft_consensus.cc:740] T b659027528154c4c86becb863718c732 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64fd1d8a3cd14e66ac94c4fbcceaf039, State: Initialized, Role: FOLLOWER
I20260709 12:27:57.429811 32134 consensus_queue.cc:260] T b659027528154c4c86becb863718c732 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:57.428113 32131 raft_consensus.cc:385] T b659027528154c4c86becb863718c732 P 2ae3a9e2293442e291b1262a717a5e50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:57.430967 32131 raft_consensus.cc:740] T b659027528154c4c86becb863718c732 P 2ae3a9e2293442e291b1262a717a5e50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ae3a9e2293442e291b1262a717a5e50, State: Initialized, Role: FOLLOWER
I20260709 12:27:57.430665 32132 raft_consensus.cc:359] T b659027528154c4c86becb863718c732 P 07b97783285d4b22b6fd0f1d816ef930 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:57.431661 32132 raft_consensus.cc:385] T b659027528154c4c86becb863718c732 P 07b97783285d4b22b6fd0f1d816ef930 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:57.432148 32132 raft_consensus.cc:740] T b659027528154c4c86becb863718c732 P 07b97783285d4b22b6fd0f1d816ef930 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07b97783285d4b22b6fd0f1d816ef930, State: Initialized, Role: FOLLOWER
I20260709 12:27:57.432572 32134 ts_tablet_manager.cc:1434] T b659027528154c4c86becb863718c732 P 64fd1d8a3cd14e66ac94c4fbcceaf039: Time spent starting tablet: real 0.008s	user 0.003s	sys 0.004s
I20260709 12:27:57.433651 32134 tablet_bootstrap.cc:492] T 0ca01749cbc64fd987e4174047baac00 P 64fd1d8a3cd14e66ac94c4fbcceaf039: Bootstrap starting.
I20260709 12:27:57.433300 32132 consensus_queue.cc:260] T b659027528154c4c86becb863718c732 P 07b97783285d4b22b6fd0f1d816ef930 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:57.431907 32131 consensus_queue.cc:260] T b659027528154c4c86becb863718c732 P 2ae3a9e2293442e291b1262a717a5e50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:57.440937 32134 tablet_bootstrap.cc:654] T 0ca01749cbc64fd987e4174047baac00 P 64fd1d8a3cd14e66ac94c4fbcceaf039: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:57.446059 32132 ts_tablet_manager.cc:1434] T b659027528154c4c86becb863718c732 P 07b97783285d4b22b6fd0f1d816ef930: Time spent starting tablet: real 0.020s	user 0.006s	sys 0.001s
I20260709 12:27:57.446789 32134 tablet_bootstrap.cc:492] T 0ca01749cbc64fd987e4174047baac00 P 64fd1d8a3cd14e66ac94c4fbcceaf039: No bootstrap required, opened a new log
I20260709 12:27:57.447417 32132 tablet_bootstrap.cc:492] T 0ca01749cbc64fd987e4174047baac00 P 07b97783285d4b22b6fd0f1d816ef930: Bootstrap starting.
I20260709 12:27:57.447573 32134 ts_tablet_manager.cc:1403] T 0ca01749cbc64fd987e4174047baac00 P 64fd1d8a3cd14e66ac94c4fbcceaf039: Time spent bootstrapping tablet: real 0.014s	user 0.009s	sys 0.004s
I20260709 12:27:57.448581 32136 raft_consensus.cc:493] T 9fe7f1d426c94232a1ec785665ef1416 P 2ae3a9e2293442e291b1262a717a5e50 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:57.449388 32131 ts_tablet_manager.cc:1434] T b659027528154c4c86becb863718c732 P 2ae3a9e2293442e291b1262a717a5e50: Time spent starting tablet: real 0.025s	user 0.008s	sys 0.000s
I20260709 12:27:57.449132 32136 raft_consensus.cc:515] T 9fe7f1d426c94232a1ec785665ef1416 P 2ae3a9e2293442e291b1262a717a5e50 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:57.451488 32134 raft_consensus.cc:359] T 0ca01749cbc64fd987e4174047baac00 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:57.452297 32134 raft_consensus.cc:385] T 0ca01749cbc64fd987e4174047baac00 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:57.453142 32131 tablet_bootstrap.cc:492] T 0ca01749cbc64fd987e4174047baac00 P 2ae3a9e2293442e291b1262a717a5e50: Bootstrap starting.
I20260709 12:27:57.454893 31981 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe7f1d426c94232a1ec785665ef1416" candidate_uuid: "2ae3a9e2293442e291b1262a717a5e50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07b97783285d4b22b6fd0f1d816ef930" is_pre_election: true
I20260709 12:27:57.455235 31909 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe7f1d426c94232a1ec785665ef1416" candidate_uuid: "2ae3a9e2293442e291b1262a717a5e50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" is_pre_election: true
I20260709 12:27:57.455569 31981 raft_consensus.cc:2468] T 9fe7f1d426c94232a1ec785665ef1416 P 07b97783285d4b22b6fd0f1d816ef930 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ae3a9e2293442e291b1262a717a5e50 in term 0.
I20260709 12:27:57.455977 32132 tablet_bootstrap.cc:654] T 0ca01749cbc64fd987e4174047baac00 P 07b97783285d4b22b6fd0f1d816ef930: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:57.457123 31801 leader_election.cc:304] T 9fe7f1d426c94232a1ec785665ef1416 P 2ae3a9e2293442e291b1262a717a5e50 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07b97783285d4b22b6fd0f1d816ef930, 2ae3a9e2293442e291b1262a717a5e50; no voters: 
I20260709 12:27:57.452607 32134 raft_consensus.cc:740] T 0ca01749cbc64fd987e4174047baac00 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64fd1d8a3cd14e66ac94c4fbcceaf039, State: Initialized, Role: FOLLOWER
I20260709 12:27:57.458559 32093 raft_consensus.cc:2804] T 9fe7f1d426c94232a1ec785665ef1416 P 2ae3a9e2293442e291b1262a717a5e50 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:57.457197 31909 raft_consensus.cc:2468] T 9fe7f1d426c94232a1ec785665ef1416 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ae3a9e2293442e291b1262a717a5e50 in term 0.
I20260709 12:27:57.459262 32093 raft_consensus.cc:493] T 9fe7f1d426c94232a1ec785665ef1416 P 2ae3a9e2293442e291b1262a717a5e50 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:57.458388 32134 consensus_queue.cc:260] T 0ca01749cbc64fd987e4174047baac00 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:57.459672 32093 raft_consensus.cc:3060] T 9fe7f1d426c94232a1ec785665ef1416 P 2ae3a9e2293442e291b1262a717a5e50 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:57.461584 32134 ts_tablet_manager.cc:1434] T 0ca01749cbc64fd987e4174047baac00 P 64fd1d8a3cd14e66ac94c4fbcceaf039: Time spent starting tablet: real 0.013s	user 0.006s	sys 0.000s
I20260709 12:27:57.463079 32134 tablet_bootstrap.cc:492] T c4f237debd4942b198e077f8e7091088 P 64fd1d8a3cd14e66ac94c4fbcceaf039: Bootstrap starting.
I20260709 12:27:57.470125 32131 tablet_bootstrap.cc:654] T 0ca01749cbc64fd987e4174047baac00 P 2ae3a9e2293442e291b1262a717a5e50: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:57.472005 32093 raft_consensus.cc:515] T 9fe7f1d426c94232a1ec785665ef1416 P 2ae3a9e2293442e291b1262a717a5e50 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:57.459077 32136 leader_election.cc:290] T 9fe7f1d426c94232a1ec785665ef1416 P 2ae3a9e2293442e291b1262a717a5e50 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 64fd1d8a3cd14e66ac94c4fbcceaf039 (127.16.29.66:42163), 07b97783285d4b22b6fd0f1d816ef930 (127.16.29.67:34419)
I20260709 12:27:57.483047 31909 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe7f1d426c94232a1ec785665ef1416" candidate_uuid: "2ae3a9e2293442e291b1262a717a5e50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039"
I20260709 12:27:57.484018 31909 raft_consensus.cc:3060] T 9fe7f1d426c94232a1ec785665ef1416 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:57.490267 32134 tablet_bootstrap.cc:654] T c4f237debd4942b198e077f8e7091088 P 64fd1d8a3cd14e66ac94c4fbcceaf039: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:57.497113 32160 raft_consensus.cc:493] T b659027528154c4c86becb863718c732 P 2ae3a9e2293442e291b1262a717a5e50 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:57.497720 32160 raft_consensus.cc:515] T b659027528154c4c86becb863718c732 P 2ae3a9e2293442e291b1262a717a5e50 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:57.499953 32160 leader_election.cc:290] T b659027528154c4c86becb863718c732 P 2ae3a9e2293442e291b1262a717a5e50 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 64fd1d8a3cd14e66ac94c4fbcceaf039 (127.16.29.66:42163), 07b97783285d4b22b6fd0f1d816ef930 (127.16.29.67:34419)
I20260709 12:27:57.500998 31908 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b659027528154c4c86becb863718c732" candidate_uuid: "2ae3a9e2293442e291b1262a717a5e50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" is_pre_election: true
I20260709 12:27:57.501716 31908 raft_consensus.cc:2468] T b659027528154c4c86becb863718c732 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ae3a9e2293442e291b1262a717a5e50 in term 0.
I20260709 12:27:57.502660 32132 tablet_bootstrap.cc:492] T 0ca01749cbc64fd987e4174047baac00 P 07b97783285d4b22b6fd0f1d816ef930: No bootstrap required, opened a new log
I20260709 12:27:57.503237 32132 ts_tablet_manager.cc:1403] T 0ca01749cbc64fd987e4174047baac00 P 07b97783285d4b22b6fd0f1d816ef930: Time spent bootstrapping tablet: real 0.056s	user 0.024s	sys 0.017s
I20260709 12:27:57.503456 32093 leader_election.cc:290] T 9fe7f1d426c94232a1ec785665ef1416 P 2ae3a9e2293442e291b1262a717a5e50 [CANDIDATE]: Term 1 election: Requested vote from peers 64fd1d8a3cd14e66ac94c4fbcceaf039 (127.16.29.66:42163), 07b97783285d4b22b6fd0f1d816ef930 (127.16.29.67:34419)
I20260709 12:27:57.504971 31984 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe7f1d426c94232a1ec785665ef1416" candidate_uuid: "2ae3a9e2293442e291b1262a717a5e50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07b97783285d4b22b6fd0f1d816ef930"
I20260709 12:27:57.505865 31984 raft_consensus.cc:3060] T 9fe7f1d426c94232a1ec785665ef1416 P 07b97783285d4b22b6fd0f1d816ef930 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:57.507100 32132 raft_consensus.cc:359] T 0ca01749cbc64fd987e4174047baac00 P 07b97783285d4b22b6fd0f1d816ef930 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:57.507793 32132 raft_consensus.cc:385] T 0ca01749cbc64fd987e4174047baac00 P 07b97783285d4b22b6fd0f1d816ef930 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:57.508145 32132 raft_consensus.cc:740] T 0ca01749cbc64fd987e4174047baac00 P 07b97783285d4b22b6fd0f1d816ef930 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07b97783285d4b22b6fd0f1d816ef930, State: Initialized, Role: FOLLOWER
I20260709 12:27:57.508193 31909 raft_consensus.cc:2468] T 9fe7f1d426c94232a1ec785665ef1416 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ae3a9e2293442e291b1262a717a5e50 in term 1.
I20260709 12:27:57.508970 32132 consensus_queue.cc:260] T 0ca01749cbc64fd987e4174047baac00 P 07b97783285d4b22b6fd0f1d816ef930 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:57.509310 31798 leader_election.cc:304] T b659027528154c4c86becb863718c732 P 2ae3a9e2293442e291b1262a717a5e50 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ae3a9e2293442e291b1262a717a5e50, 64fd1d8a3cd14e66ac94c4fbcceaf039; no voters: 
I20260709 12:27:57.514771 32161 raft_consensus.cc:2804] T b659027528154c4c86becb863718c732 P 2ae3a9e2293442e291b1262a717a5e50 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:57.515407 32161 raft_consensus.cc:493] T b659027528154c4c86becb863718c732 P 2ae3a9e2293442e291b1262a717a5e50 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:57.511763 31981 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b659027528154c4c86becb863718c732" candidate_uuid: "2ae3a9e2293442e291b1262a717a5e50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07b97783285d4b22b6fd0f1d816ef930" is_pre_election: true
I20260709 12:27:57.515902 32161 raft_consensus.cc:3060] T b659027528154c4c86becb863718c732 P 2ae3a9e2293442e291b1262a717a5e50 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:57.516496 31981 raft_consensus.cc:2468] T b659027528154c4c86becb863718c732 P 07b97783285d4b22b6fd0f1d816ef930 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ae3a9e2293442e291b1262a717a5e50 in term 0.
I20260709 12:27:57.517342 32052 raft_consensus.cc:493] T 0ca01749cbc64fd987e4174047baac00 P 07b97783285d4b22b6fd0f1d816ef930 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:57.518215 32052 raft_consensus.cc:515] T 0ca01749cbc64fd987e4174047baac00 P 07b97783285d4b22b6fd0f1d816ef930 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:57.519942 32132 ts_tablet_manager.cc:1434] T 0ca01749cbc64fd987e4174047baac00 P 07b97783285d4b22b6fd0f1d816ef930: Time spent starting tablet: real 0.016s	user 0.011s	sys 0.000s
I20260709 12:27:57.521559 32132 tablet_bootstrap.cc:492] T c4f237debd4942b198e077f8e7091088 P 07b97783285d4b22b6fd0f1d816ef930: Bootstrap starting.
I20260709 12:27:57.524828 32052 leader_election.cc:290] T 0ca01749cbc64fd987e4174047baac00 P 07b97783285d4b22b6fd0f1d816ef930 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2ae3a9e2293442e291b1262a717a5e50 (127.16.29.65:44289), 64fd1d8a3cd14e66ac94c4fbcceaf039 (127.16.29.66:42163)
I20260709 12:27:57.527485 31833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ca01749cbc64fd987e4174047baac00" candidate_uuid: "07b97783285d4b22b6fd0f1d816ef930" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ae3a9e2293442e291b1262a717a5e50" is_pre_election: true
I20260709 12:27:57.529659 31798 leader_election.cc:304] T 9fe7f1d426c94232a1ec785665ef1416 P 2ae3a9e2293442e291b1262a717a5e50 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ae3a9e2293442e291b1262a717a5e50, 64fd1d8a3cd14e66ac94c4fbcceaf039; no voters: 
I20260709 12:27:57.531566 32160 raft_consensus.cc:2804] T 9fe7f1d426c94232a1ec785665ef1416 P 2ae3a9e2293442e291b1262a717a5e50 [term 1 FOLLOWER]: Leader election won for term 1
W20260709 12:27:57.531628 31949 leader_election.cc:343] T 0ca01749cbc64fd987e4174047baac00 P 07b97783285d4b22b6fd0f1d816ef930 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2ae3a9e2293442e291b1262a717a5e50 (127.16.29.65:44289): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:57.532174 32160 raft_consensus.cc:697] T 9fe7f1d426c94232a1ec785665ef1416 P 2ae3a9e2293442e291b1262a717a5e50 [term 1 LEADER]: Becoming Leader. State: Replica: 2ae3a9e2293442e291b1262a717a5e50, State: Running, Role: LEADER
I20260709 12:27:57.533263 31909 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ca01749cbc64fd987e4174047baac00" candidate_uuid: "07b97783285d4b22b6fd0f1d816ef930" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" is_pre_election: true
I20260709 12:27:57.533411 32160 consensus_queue.cc:237] T 9fe7f1d426c94232a1ec785665ef1416 P 2ae3a9e2293442e291b1262a717a5e50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:57.534425 31909 raft_consensus.cc:2468] T 0ca01749cbc64fd987e4174047baac00 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 07b97783285d4b22b6fd0f1d816ef930 in term 0.
I20260709 12:27:57.536052 31947 leader_election.cc:304] T 0ca01749cbc64fd987e4174047baac00 P 07b97783285d4b22b6fd0f1d816ef930 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 07b97783285d4b22b6fd0f1d816ef930, 64fd1d8a3cd14e66ac94c4fbcceaf039; no voters: 2ae3a9e2293442e291b1262a717a5e50
I20260709 12:27:57.537990 32052 raft_consensus.cc:2804] T 0ca01749cbc64fd987e4174047baac00 P 07b97783285d4b22b6fd0f1d816ef930 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:57.538568 32052 raft_consensus.cc:493] T 0ca01749cbc64fd987e4174047baac00 P 07b97783285d4b22b6fd0f1d816ef930 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:57.539072 32052 raft_consensus.cc:3060] T 0ca01749cbc64fd987e4174047baac00 P 07b97783285d4b22b6fd0f1d816ef930 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:57.539505 32132 tablet_bootstrap.cc:654] T c4f237debd4942b198e077f8e7091088 P 07b97783285d4b22b6fd0f1d816ef930: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:57.545625 32161 raft_consensus.cc:515] T b659027528154c4c86becb863718c732 P 2ae3a9e2293442e291b1262a717a5e50 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:57.558818 31984 raft_consensus.cc:2468] T 9fe7f1d426c94232a1ec785665ef1416 P 07b97783285d4b22b6fd0f1d816ef930 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ae3a9e2293442e291b1262a717a5e50 in term 1.
I20260709 12:27:57.563125 32132 tablet_bootstrap.cc:492] T c4f237debd4942b198e077f8e7091088 P 07b97783285d4b22b6fd0f1d816ef930: No bootstrap required, opened a new log
I20260709 12:27:57.563675 32132 ts_tablet_manager.cc:1403] T c4f237debd4942b198e077f8e7091088 P 07b97783285d4b22b6fd0f1d816ef930: Time spent bootstrapping tablet: real 0.043s	user 0.016s	sys 0.000s
I20260709 12:27:57.564035 31981 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b659027528154c4c86becb863718c732" candidate_uuid: "2ae3a9e2293442e291b1262a717a5e50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07b97783285d4b22b6fd0f1d816ef930"
I20260709 12:27:57.564807 31981 raft_consensus.cc:3060] T b659027528154c4c86becb863718c732 P 07b97783285d4b22b6fd0f1d816ef930 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:57.566363 32132 raft_consensus.cc:359] T c4f237debd4942b198e077f8e7091088 P 07b97783285d4b22b6fd0f1d816ef930 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:57.567034 32132 raft_consensus.cc:385] T c4f237debd4942b198e077f8e7091088 P 07b97783285d4b22b6fd0f1d816ef930 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:57.567394 32132 raft_consensus.cc:740] T c4f237debd4942b198e077f8e7091088 P 07b97783285d4b22b6fd0f1d816ef930 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07b97783285d4b22b6fd0f1d816ef930, State: Initialized, Role: FOLLOWER
I20260709 12:27:57.568296 32132 consensus_queue.cc:260] T c4f237debd4942b198e077f8e7091088 P 07b97783285d4b22b6fd0f1d816ef930 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:57.560952 31909 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b659027528154c4c86becb863718c732" candidate_uuid: "2ae3a9e2293442e291b1262a717a5e50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039"
I20260709 12:27:57.570683 31909 raft_consensus.cc:3060] T b659027528154c4c86becb863718c732 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:57.571053 32132 ts_tablet_manager.cc:1434] T c4f237debd4942b198e077f8e7091088 P 07b97783285d4b22b6fd0f1d816ef930: Time spent starting tablet: real 0.007s	user 0.002s	sys 0.002s
I20260709 12:27:57.561931 32161 leader_election.cc:290] T b659027528154c4c86becb863718c732 P 2ae3a9e2293442e291b1262a717a5e50 [CANDIDATE]: Term 1 election: Requested vote from peers 64fd1d8a3cd14e66ac94c4fbcceaf039 (127.16.29.66:42163), 07b97783285d4b22b6fd0f1d816ef930 (127.16.29.67:34419)
I20260709 12:27:57.572161 32131 tablet_bootstrap.cc:492] T 0ca01749cbc64fd987e4174047baac00 P 2ae3a9e2293442e291b1262a717a5e50: No bootstrap required, opened a new log
I20260709 12:27:57.573137 32131 ts_tablet_manager.cc:1403] T 0ca01749cbc64fd987e4174047baac00 P 2ae3a9e2293442e291b1262a717a5e50: Time spent bootstrapping tablet: real 0.120s	user 0.023s	sys 0.009s
I20260709 12:27:57.572301 32132 tablet_bootstrap.cc:492] T 703b2df19bf24c31a5acbe85178314ec P 07b97783285d4b22b6fd0f1d816ef930: Bootstrap starting.
I20260709 12:27:57.576893 32131 raft_consensus.cc:359] T 0ca01749cbc64fd987e4174047baac00 P 2ae3a9e2293442e291b1262a717a5e50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:57.577970 32131 raft_consensus.cc:385] T 0ca01749cbc64fd987e4174047baac00 P 2ae3a9e2293442e291b1262a717a5e50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:57.578402 32131 raft_consensus.cc:740] T 0ca01749cbc64fd987e4174047baac00 P 2ae3a9e2293442e291b1262a717a5e50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ae3a9e2293442e291b1262a717a5e50, State: Initialized, Role: FOLLOWER
I20260709 12:27:57.579326 32131 consensus_queue.cc:260] T 0ca01749cbc64fd987e4174047baac00 P 2ae3a9e2293442e291b1262a717a5e50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:57.581687 32132 tablet_bootstrap.cc:654] T 703b2df19bf24c31a5acbe85178314ec P 07b97783285d4b22b6fd0f1d816ef930: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:57.582156 32131 ts_tablet_manager.cc:1434] T 0ca01749cbc64fd987e4174047baac00 P 2ae3a9e2293442e291b1262a717a5e50: Time spent starting tablet: real 0.009s	user 0.008s	sys 0.000s
I20260709 12:27:57.583421 32131 tablet_bootstrap.cc:492] T c4f237debd4942b198e077f8e7091088 P 2ae3a9e2293442e291b1262a717a5e50: Bootstrap starting.
I20260709 12:27:57.584424 31732 catalog_manager.cc:5697] T 9fe7f1d426c94232a1ec785665ef1416 P 2ae3a9e2293442e291b1262a717a5e50 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2ae3a9e2293442e291b1262a717a5e50 (127.16.29.65). New cstate: current_term: 1 leader_uuid: "2ae3a9e2293442e291b1262a717a5e50" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:57.593128 32134 tablet_bootstrap.cc:492] T c4f237debd4942b198e077f8e7091088 P 64fd1d8a3cd14e66ac94c4fbcceaf039: No bootstrap required, opened a new log
I20260709 12:27:57.555531 32052 raft_consensus.cc:515] T 0ca01749cbc64fd987e4174047baac00 P 07b97783285d4b22b6fd0f1d816ef930 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:57.594117 32134 ts_tablet_manager.cc:1403] T c4f237debd4942b198e077f8e7091088 P 64fd1d8a3cd14e66ac94c4fbcceaf039: Time spent bootstrapping tablet: real 0.131s	user 0.023s	sys 0.011s
I20260709 12:27:57.593037 32131 tablet_bootstrap.cc:654] T c4f237debd4942b198e077f8e7091088 P 2ae3a9e2293442e291b1262a717a5e50: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:57.597381 31909 raft_consensus.cc:2468] T b659027528154c4c86becb863718c732 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ae3a9e2293442e291b1262a717a5e50 in term 1.
I20260709 12:27:57.598834 32052 leader_election.cc:290] T 0ca01749cbc64fd987e4174047baac00 P 07b97783285d4b22b6fd0f1d816ef930 [CANDIDATE]: Term 1 election: Requested vote from peers 2ae3a9e2293442e291b1262a717a5e50 (127.16.29.65:44289), 64fd1d8a3cd14e66ac94c4fbcceaf039 (127.16.29.66:42163)
I20260709 12:27:57.601939 31910 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ca01749cbc64fd987e4174047baac00" candidate_uuid: "07b97783285d4b22b6fd0f1d816ef930" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039"
I20260709 12:27:57.602851 31910 raft_consensus.cc:3060] T 0ca01749cbc64fd987e4174047baac00 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:57.600201 31798 leader_election.cc:304] T b659027528154c4c86becb863718c732 P 2ae3a9e2293442e291b1262a717a5e50 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ae3a9e2293442e291b1262a717a5e50, 64fd1d8a3cd14e66ac94c4fbcceaf039; no voters: 
I20260709 12:27:57.607801 32132 tablet_bootstrap.cc:492] T 703b2df19bf24c31a5acbe85178314ec P 07b97783285d4b22b6fd0f1d816ef930: No bootstrap required, opened a new log
I20260709 12:27:57.608458 32132 ts_tablet_manager.cc:1403] T 703b2df19bf24c31a5acbe85178314ec P 07b97783285d4b22b6fd0f1d816ef930: Time spent bootstrapping tablet: real 0.036s	user 0.015s	sys 0.005s
I20260709 12:27:57.608556 32131 tablet_bootstrap.cc:492] T c4f237debd4942b198e077f8e7091088 P 2ae3a9e2293442e291b1262a717a5e50: No bootstrap required, opened a new log
I20260709 12:27:57.609115 32131 ts_tablet_manager.cc:1403] T c4f237debd4942b198e077f8e7091088 P 2ae3a9e2293442e291b1262a717a5e50: Time spent bootstrapping tablet: real 0.026s	user 0.007s	sys 0.011s
I20260709 12:27:57.609300 32161 raft_consensus.cc:2804] T b659027528154c4c86becb863718c732 P 2ae3a9e2293442e291b1262a717a5e50 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:57.609141 31833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ca01749cbc64fd987e4174047baac00" candidate_uuid: "07b97783285d4b22b6fd0f1d816ef930" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ae3a9e2293442e291b1262a717a5e50"
I20260709 12:27:57.610443 31833 raft_consensus.cc:3060] T 0ca01749cbc64fd987e4174047baac00 P 2ae3a9e2293442e291b1262a717a5e50 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:57.611625 32132 raft_consensus.cc:359] T 703b2df19bf24c31a5acbe85178314ec P 07b97783285d4b22b6fd0f1d816ef930 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:57.612435 32132 raft_consensus.cc:385] T 703b2df19bf24c31a5acbe85178314ec P 07b97783285d4b22b6fd0f1d816ef930 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:57.612712 32132 raft_consensus.cc:740] T 703b2df19bf24c31a5acbe85178314ec P 07b97783285d4b22b6fd0f1d816ef930 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07b97783285d4b22b6fd0f1d816ef930, State: Initialized, Role: FOLLOWER
I20260709 12:27:57.613714 32132 consensus_queue.cc:260] T 703b2df19bf24c31a5acbe85178314ec P 07b97783285d4b22b6fd0f1d816ef930 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:57.616786 32132 ts_tablet_manager.cc:1434] T 703b2df19bf24c31a5acbe85178314ec P 07b97783285d4b22b6fd0f1d816ef930: Time spent starting tablet: real 0.008s	user 0.007s	sys 0.000s
I20260709 12:27:57.618039 32132 tablet_bootstrap.cc:492] T ff138d12849e4f63a9b257f4395179de P 07b97783285d4b22b6fd0f1d816ef930: Bootstrap starting.
I20260709 12:27:57.624564 32132 tablet_bootstrap.cc:654] T ff138d12849e4f63a9b257f4395179de P 07b97783285d4b22b6fd0f1d816ef930: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:57.615442 32134 raft_consensus.cc:359] T c4f237debd4942b198e077f8e7091088 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:57.626464 32134 raft_consensus.cc:385] T c4f237debd4942b198e077f8e7091088 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:57.626893 32134 raft_consensus.cc:740] T c4f237debd4942b198e077f8e7091088 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64fd1d8a3cd14e66ac94c4fbcceaf039, State: Initialized, Role: FOLLOWER
I20260709 12:27:57.628450 32134 consensus_queue.cc:260] T c4f237debd4942b198e077f8e7091088 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:57.629957 32131 raft_consensus.cc:359] T c4f237debd4942b198e077f8e7091088 P 2ae3a9e2293442e291b1262a717a5e50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:57.630734 32131 raft_consensus.cc:385] T c4f237debd4942b198e077f8e7091088 P 2ae3a9e2293442e291b1262a717a5e50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:57.631033 32131 raft_consensus.cc:740] T c4f237debd4942b198e077f8e7091088 P 2ae3a9e2293442e291b1262a717a5e50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ae3a9e2293442e291b1262a717a5e50, State: Initialized, Role: FOLLOWER
I20260709 12:27:57.631153 32134 ts_tablet_manager.cc:1434] T c4f237debd4942b198e077f8e7091088 P 64fd1d8a3cd14e66ac94c4fbcceaf039: Time spent starting tablet: real 0.037s	user 0.004s	sys 0.003s
I20260709 12:27:57.631803 32131 consensus_queue.cc:260] T c4f237debd4942b198e077f8e7091088 P 2ae3a9e2293442e291b1262a717a5e50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:57.631937 32161 raft_consensus.cc:697] T b659027528154c4c86becb863718c732 P 2ae3a9e2293442e291b1262a717a5e50 [term 1 LEADER]: Becoming Leader. State: Replica: 2ae3a9e2293442e291b1262a717a5e50, State: Running, Role: LEADER
I20260709 12:27:57.635841 32131 ts_tablet_manager.cc:1434] T c4f237debd4942b198e077f8e7091088 P 2ae3a9e2293442e291b1262a717a5e50: Time spent starting tablet: real 0.026s	user 0.002s	sys 0.006s
I20260709 12:27:57.636941 32131 tablet_bootstrap.cc:492] T ff138d12849e4f63a9b257f4395179de P 2ae3a9e2293442e291b1262a717a5e50: Bootstrap starting.
I20260709 12:27:57.637527 31910 raft_consensus.cc:2468] T 0ca01749cbc64fd987e4174047baac00 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07b97783285d4b22b6fd0f1d816ef930 in term 1.
I20260709 12:27:57.638933 31947 leader_election.cc:304] T 0ca01749cbc64fd987e4174047baac00 P 07b97783285d4b22b6fd0f1d816ef930 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07b97783285d4b22b6fd0f1d816ef930, 64fd1d8a3cd14e66ac94c4fbcceaf039; no voters: 
I20260709 12:27:57.639828 32052 raft_consensus.cc:2804] T 0ca01749cbc64fd987e4174047baac00 P 07b97783285d4b22b6fd0f1d816ef930 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:57.640322 32052 raft_consensus.cc:697] T 0ca01749cbc64fd987e4174047baac00 P 07b97783285d4b22b6fd0f1d816ef930 [term 1 LEADER]: Becoming Leader. State: Replica: 07b97783285d4b22b6fd0f1d816ef930, State: Running, Role: LEADER
I20260709 12:27:57.640797 31981 raft_consensus.cc:2468] T b659027528154c4c86becb863718c732 P 07b97783285d4b22b6fd0f1d816ef930 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ae3a9e2293442e291b1262a717a5e50 in term 1.
I20260709 12:27:57.641206 32052 consensus_queue.cc:237] T 0ca01749cbc64fd987e4174047baac00 P 07b97783285d4b22b6fd0f1d816ef930 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:57.644001 31833 raft_consensus.cc:2468] T 0ca01749cbc64fd987e4174047baac00 P 2ae3a9e2293442e291b1262a717a5e50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07b97783285d4b22b6fd0f1d816ef930 in term 1.
I20260709 12:27:57.644125 32134 tablet_bootstrap.cc:492] T ff138d12849e4f63a9b257f4395179de P 64fd1d8a3cd14e66ac94c4fbcceaf039: Bootstrap starting.
I20260709 12:27:57.650750 32161 consensus_queue.cc:237] T b659027528154c4c86becb863718c732 P 2ae3a9e2293442e291b1262a717a5e50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:57.665441 32131 tablet_bootstrap.cc:654] T ff138d12849e4f63a9b257f4395179de P 2ae3a9e2293442e291b1262a717a5e50: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:57.668730 32132 tablet_bootstrap.cc:492] T ff138d12849e4f63a9b257f4395179de P 07b97783285d4b22b6fd0f1d816ef930: No bootstrap required, opened a new log
I20260709 12:27:57.669590 32132 ts_tablet_manager.cc:1403] T ff138d12849e4f63a9b257f4395179de P 07b97783285d4b22b6fd0f1d816ef930: Time spent bootstrapping tablet: real 0.052s	user 0.023s	sys 0.023s
I20260709 12:27:57.670094 31732 catalog_manager.cc:5697] T b659027528154c4c86becb863718c732 P 2ae3a9e2293442e291b1262a717a5e50 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2ae3a9e2293442e291b1262a717a5e50 (127.16.29.65). New cstate: current_term: 1 leader_uuid: "2ae3a9e2293442e291b1262a717a5e50" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:57.667392 32134 tablet_bootstrap.cc:654] T ff138d12849e4f63a9b257f4395179de P 64fd1d8a3cd14e66ac94c4fbcceaf039: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:57.675796 32132 raft_consensus.cc:359] T ff138d12849e4f63a9b257f4395179de P 07b97783285d4b22b6fd0f1d816ef930 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:57.676612 32132 raft_consensus.cc:385] T ff138d12849e4f63a9b257f4395179de P 07b97783285d4b22b6fd0f1d816ef930 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:57.676822 32131 tablet_bootstrap.cc:492] T ff138d12849e4f63a9b257f4395179de P 2ae3a9e2293442e291b1262a717a5e50: No bootstrap required, opened a new log
I20260709 12:27:57.677676 32131 ts_tablet_manager.cc:1403] T ff138d12849e4f63a9b257f4395179de P 2ae3a9e2293442e291b1262a717a5e50: Time spent bootstrapping tablet: real 0.041s	user 0.018s	sys 0.000s
I20260709 12:27:57.678215 32162 raft_consensus.cc:493] T c4f237debd4942b198e077f8e7091088 P 2ae3a9e2293442e291b1262a717a5e50 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:57.678905 32162 raft_consensus.cc:515] T c4f237debd4942b198e077f8e7091088 P 2ae3a9e2293442e291b1262a717a5e50 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:57.676911 32132 raft_consensus.cc:740] T ff138d12849e4f63a9b257f4395179de P 07b97783285d4b22b6fd0f1d816ef930 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07b97783285d4b22b6fd0f1d816ef930, State: Initialized, Role: FOLLOWER
I20260709 12:27:57.684036 32131 raft_consensus.cc:359] T ff138d12849e4f63a9b257f4395179de P 2ae3a9e2293442e291b1262a717a5e50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:57.685169 31909 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c4f237debd4942b198e077f8e7091088" candidate_uuid: "2ae3a9e2293442e291b1262a717a5e50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" is_pre_election: true
I20260709 12:27:57.681079 32132 consensus_queue.cc:260] T ff138d12849e4f63a9b257f4395179de P 07b97783285d4b22b6fd0f1d816ef930 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:57.688375 31909 raft_consensus.cc:2468] T c4f237debd4942b198e077f8e7091088 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ae3a9e2293442e291b1262a717a5e50 in term 0.
I20260709 12:27:57.687484 31981 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c4f237debd4942b198e077f8e7091088" candidate_uuid: "2ae3a9e2293442e291b1262a717a5e50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07b97783285d4b22b6fd0f1d816ef930" is_pre_election: true
I20260709 12:27:57.689846 31798 leader_election.cc:304] T c4f237debd4942b198e077f8e7091088 P 2ae3a9e2293442e291b1262a717a5e50 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ae3a9e2293442e291b1262a717a5e50, 64fd1d8a3cd14e66ac94c4fbcceaf039; no voters: 
I20260709 12:27:57.689980 31981 raft_consensus.cc:2468] T c4f237debd4942b198e077f8e7091088 P 07b97783285d4b22b6fd0f1d816ef930 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ae3a9e2293442e291b1262a717a5e50 in term 0.
I20260709 12:27:57.686399 32162 leader_election.cc:290] T c4f237debd4942b198e077f8e7091088 P 2ae3a9e2293442e291b1262a717a5e50 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 64fd1d8a3cd14e66ac94c4fbcceaf039 (127.16.29.66:42163), 07b97783285d4b22b6fd0f1d816ef930 (127.16.29.67:34419)
I20260709 12:27:57.690706 32132 ts_tablet_manager.cc:1434] T ff138d12849e4f63a9b257f4395179de P 07b97783285d4b22b6fd0f1d816ef930: Time spent starting tablet: real 0.021s	user 0.008s	sys 0.000s
I20260709 12:27:57.690737 32161 raft_consensus.cc:2804] T c4f237debd4942b198e077f8e7091088 P 2ae3a9e2293442e291b1262a717a5e50 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:57.691529 32161 raft_consensus.cc:493] T c4f237debd4942b198e077f8e7091088 P 2ae3a9e2293442e291b1262a717a5e50 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:57.691702 32131 raft_consensus.cc:385] T ff138d12849e4f63a9b257f4395179de P 2ae3a9e2293442e291b1262a717a5e50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:57.692595 32161 raft_consensus.cc:3060] T c4f237debd4942b198e077f8e7091088 P 2ae3a9e2293442e291b1262a717a5e50 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:57.692852 32131 raft_consensus.cc:740] T ff138d12849e4f63a9b257f4395179de P 2ae3a9e2293442e291b1262a717a5e50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ae3a9e2293442e291b1262a717a5e50, State: Initialized, Role: FOLLOWER
I20260709 12:27:57.693368 32134 tablet_bootstrap.cc:492] T ff138d12849e4f63a9b257f4395179de P 64fd1d8a3cd14e66ac94c4fbcceaf039: No bootstrap required, opened a new log
I20260709 12:27:57.693905 32134 ts_tablet_manager.cc:1403] T ff138d12849e4f63a9b257f4395179de P 64fd1d8a3cd14e66ac94c4fbcceaf039: Time spent bootstrapping tablet: real 0.061s	user 0.013s	sys 0.010s
I20260709 12:27:57.693840 32131 consensus_queue.cc:260] T ff138d12849e4f63a9b257f4395179de P 2ae3a9e2293442e291b1262a717a5e50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:57.696424 32131 ts_tablet_manager.cc:1434] T ff138d12849e4f63a9b257f4395179de P 2ae3a9e2293442e291b1262a717a5e50: Time spent starting tablet: real 0.018s	user 0.009s	sys 0.000s
I20260709 12:27:57.695847 31729 catalog_manager.cc:5697] T 0ca01749cbc64fd987e4174047baac00 P 07b97783285d4b22b6fd0f1d816ef930 reported cstate change: term changed from 0 to 1, leader changed from <none> to 07b97783285d4b22b6fd0f1d816ef930 (127.16.29.67). New cstate: current_term: 1 leader_uuid: "07b97783285d4b22b6fd0f1d816ef930" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } health_report { overall_health: HEALTHY } } }
I20260709 12:27:57.697543 32134 raft_consensus.cc:359] T ff138d12849e4f63a9b257f4395179de P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:57.698359 32134 raft_consensus.cc:385] T ff138d12849e4f63a9b257f4395179de P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:57.698671 32134 raft_consensus.cc:740] T ff138d12849e4f63a9b257f4395179de P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64fd1d8a3cd14e66ac94c4fbcceaf039, State: Initialized, Role: FOLLOWER
I20260709 12:27:57.699450 32134 consensus_queue.cc:260] T ff138d12849e4f63a9b257f4395179de P 64fd1d8a3cd14e66ac94c4fbcceaf039 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:57.697597 32131 tablet_bootstrap.cc:492] T 703b2df19bf24c31a5acbe85178314ec P 2ae3a9e2293442e291b1262a717a5e50: Bootstrap starting.
I20260709 12:27:57.704942 32134 ts_tablet_manager.cc:1434] T ff138d12849e4f63a9b257f4395179de P 64fd1d8a3cd14e66ac94c4fbcceaf039: Time spent starting tablet: real 0.010s	user 0.006s	sys 0.001s
I20260709 12:27:57.706063 32134 tablet_bootstrap.cc:492] T 703b2df19bf24c31a5acbe85178314ec P 64fd1d8a3cd14e66ac94c4fbcceaf039: Bootstrap starting.
I20260709 12:27:57.707394 32131 tablet_bootstrap.cc:654] T 703b2df19bf24c31a5acbe85178314ec P 2ae3a9e2293442e291b1262a717a5e50: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:57.709460 32052 raft_consensus.cc:493] T ff138d12849e4f63a9b257f4395179de P 07b97783285d4b22b6fd0f1d816ef930 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:57.710023 32052 raft_consensus.cc:515] T ff138d12849e4f63a9b257f4395179de P 07b97783285d4b22b6fd0f1d816ef930 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:57.713121 32134 tablet_bootstrap.cc:654] T 703b2df19bf24c31a5acbe85178314ec P 64fd1d8a3cd14e66ac94c4fbcceaf039: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:57.713279 32052 leader_election.cc:290] T ff138d12849e4f63a9b257f4395179de P 07b97783285d4b22b6fd0f1d816ef930 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2ae3a9e2293442e291b1262a717a5e50 (127.16.29.65:44289), 64fd1d8a3cd14e66ac94c4fbcceaf039 (127.16.29.66:42163)
I20260709 12:27:57.716630 32142 raft_consensus.cc:493] T 703b2df19bf24c31a5acbe85178314ec P 07b97783285d4b22b6fd0f1d816ef930 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:57.717182 32142 raft_consensus.cc:515] T 703b2df19bf24c31a5acbe85178314ec P 07b97783285d4b22b6fd0f1d816ef930 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:57.721405 31910 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "703b2df19bf24c31a5acbe85178314ec" candidate_uuid: "07b97783285d4b22b6fd0f1d816ef930" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" is_pre_election: true
W20260709 12:27:57.723067 31947 leader_election.cc:343] T 703b2df19bf24c31a5acbe85178314ec P 07b97783285d4b22b6fd0f1d816ef930 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 64fd1d8a3cd14e66ac94c4fbcceaf039 (127.16.29.66:42163): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:57.730768 31909 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff138d12849e4f63a9b257f4395179de" candidate_uuid: "07b97783285d4b22b6fd0f1d816ef930" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" is_pre_election: true
I20260709 12:27:57.731490 31909 raft_consensus.cc:2468] T ff138d12849e4f63a9b257f4395179de P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 07b97783285d4b22b6fd0f1d816ef930 in term 0.
I20260709 12:27:57.732831 31947 leader_election.cc:304] T ff138d12849e4f63a9b257f4395179de P 07b97783285d4b22b6fd0f1d816ef930 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07b97783285d4b22b6fd0f1d816ef930, 64fd1d8a3cd14e66ac94c4fbcceaf039; no voters: 
I20260709 12:27:57.733829 32144 consensus_queue.cc:1048] T 0d5a572ce9954c72b6d74a24da7de624 P 07b97783285d4b22b6fd0f1d816ef930 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:57.734953 31834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "703b2df19bf24c31a5acbe85178314ec" candidate_uuid: "07b97783285d4b22b6fd0f1d816ef930" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ae3a9e2293442e291b1262a717a5e50" is_pre_election: true
I20260709 12:27:57.735728 32144 raft_consensus.cc:2804] T ff138d12849e4f63a9b257f4395179de P 07b97783285d4b22b6fd0f1d816ef930 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:57.736339 32144 raft_consensus.cc:493] T ff138d12849e4f63a9b257f4395179de P 07b97783285d4b22b6fd0f1d816ef930 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:57.736806 32144 raft_consensus.cc:3060] T ff138d12849e4f63a9b257f4395179de P 07b97783285d4b22b6fd0f1d816ef930 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:57.738337 31833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff138d12849e4f63a9b257f4395179de" candidate_uuid: "07b97783285d4b22b6fd0f1d816ef930" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ae3a9e2293442e291b1262a717a5e50" is_pre_election: true
I20260709 12:27:57.739058 31833 raft_consensus.cc:2468] T ff138d12849e4f63a9b257f4395179de P 2ae3a9e2293442e291b1262a717a5e50 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 07b97783285d4b22b6fd0f1d816ef930 in term 0.
W20260709 12:27:57.741789 31949 leader_election.cc:343] T 703b2df19bf24c31a5acbe85178314ec P 07b97783285d4b22b6fd0f1d816ef930 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2ae3a9e2293442e291b1262a717a5e50 (127.16.29.65:44289): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:57.742249 31949 leader_election.cc:304] T 703b2df19bf24c31a5acbe85178314ec P 07b97783285d4b22b6fd0f1d816ef930 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 07b97783285d4b22b6fd0f1d816ef930; no voters: 2ae3a9e2293442e291b1262a717a5e50, 64fd1d8a3cd14e66ac94c4fbcceaf039
I20260709 12:27:57.719641 32142 leader_election.cc:290] T 703b2df19bf24c31a5acbe85178314ec P 07b97783285d4b22b6fd0f1d816ef930 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2ae3a9e2293442e291b1262a717a5e50 (127.16.29.65:44289), 64fd1d8a3cd14e66ac94c4fbcceaf039 (127.16.29.66:42163)
I20260709 12:27:57.744048 32142 raft_consensus.cc:2749] T 703b2df19bf24c31a5acbe85178314ec P 07b97783285d4b22b6fd0f1d816ef930 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:57.701787 32161 raft_consensus.cc:515] T c4f237debd4942b198e077f8e7091088 P 2ae3a9e2293442e291b1262a717a5e50 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:57.747267 32161 leader_election.cc:290] T c4f237debd4942b198e077f8e7091088 P 2ae3a9e2293442e291b1262a717a5e50 [CANDIDATE]: Term 1 election: Requested vote from peers 64fd1d8a3cd14e66ac94c4fbcceaf039 (127.16.29.66:42163), 07b97783285d4b22b6fd0f1d816ef930 (127.16.29.67:34419)
I20260709 12:27:57.749008 31981 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c4f237debd4942b198e077f8e7091088" candidate_uuid: "2ae3a9e2293442e291b1262a717a5e50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07b97783285d4b22b6fd0f1d816ef930"
I20260709 12:27:57.749706 31981 raft_consensus.cc:3060] T c4f237debd4942b198e077f8e7091088 P 07b97783285d4b22b6fd0f1d816ef930 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:57.750171 31909 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c4f237debd4942b198e077f8e7091088" candidate_uuid: "2ae3a9e2293442e291b1262a717a5e50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039"
I20260709 12:27:57.750705 31909 raft_consensus.cc:3060] T c4f237debd4942b198e077f8e7091088 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:57.751360 32131 tablet_bootstrap.cc:492] T 703b2df19bf24c31a5acbe85178314ec P 2ae3a9e2293442e291b1262a717a5e50: No bootstrap required, opened a new log
I20260709 12:27:57.751379 32134 tablet_bootstrap.cc:492] T 703b2df19bf24c31a5acbe85178314ec P 64fd1d8a3cd14e66ac94c4fbcceaf039: No bootstrap required, opened a new log
I20260709 12:27:57.752010 32131 ts_tablet_manager.cc:1403] T 703b2df19bf24c31a5acbe85178314ec P 2ae3a9e2293442e291b1262a717a5e50: Time spent bootstrapping tablet: real 0.055s	user 0.026s	sys 0.023s
I20260709 12:27:57.752143 32134 ts_tablet_manager.cc:1403] T 703b2df19bf24c31a5acbe85178314ec P 64fd1d8a3cd14e66ac94c4fbcceaf039: Time spent bootstrapping tablet: real 0.046s	user 0.022s	sys 0.008s
I20260709 12:27:57.754812 32131 raft_consensus.cc:359] T 703b2df19bf24c31a5acbe85178314ec P 2ae3a9e2293442e291b1262a717a5e50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:57.755496 32131 raft_consensus.cc:385] T 703b2df19bf24c31a5acbe85178314ec P 2ae3a9e2293442e291b1262a717a5e50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:57.755182 32134 raft_consensus.cc:359] T 703b2df19bf24c31a5acbe85178314ec P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:57.756093 32134 raft_consensus.cc:385] T 703b2df19bf24c31a5acbe85178314ec P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:57.755780 32131 raft_consensus.cc:740] T 703b2df19bf24c31a5acbe85178314ec P 2ae3a9e2293442e291b1262a717a5e50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ae3a9e2293442e291b1262a717a5e50, State: Initialized, Role: FOLLOWER
I20260709 12:27:57.756457 32134 raft_consensus.cc:740] T 703b2df19bf24c31a5acbe85178314ec P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64fd1d8a3cd14e66ac94c4fbcceaf039, State: Initialized, Role: FOLLOWER
I20260709 12:27:57.757179 31981 raft_consensus.cc:2468] T c4f237debd4942b198e077f8e7091088 P 07b97783285d4b22b6fd0f1d816ef930 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ae3a9e2293442e291b1262a717a5e50 in term 1.
I20260709 12:27:57.757102 32131 consensus_queue.cc:260] T 703b2df19bf24c31a5acbe85178314ec P 2ae3a9e2293442e291b1262a717a5e50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:57.757378 32134 consensus_queue.cc:260] T 703b2df19bf24c31a5acbe85178314ec P 64fd1d8a3cd14e66ac94c4fbcceaf039 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:57.758958 31801 leader_election.cc:304] T c4f237debd4942b198e077f8e7091088 P 2ae3a9e2293442e291b1262a717a5e50 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07b97783285d4b22b6fd0f1d816ef930, 2ae3a9e2293442e291b1262a717a5e50; no voters: 
I20260709 12:27:57.759893 32161 raft_consensus.cc:2804] T c4f237debd4942b198e077f8e7091088 P 2ae3a9e2293442e291b1262a717a5e50 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:57.760010 32120 consensus_queue.cc:1048] T c11f552ceb904522856ef259018a8442 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:27:57.760396 32131 ts_tablet_manager.cc:1434] T 703b2df19bf24c31a5acbe85178314ec P 2ae3a9e2293442e291b1262a717a5e50: Time spent starting tablet: real 0.008s	user 0.007s	sys 0.000s
I20260709 12:27:57.760492 32161 raft_consensus.cc:697] T c4f237debd4942b198e077f8e7091088 P 2ae3a9e2293442e291b1262a717a5e50 [term 1 LEADER]: Becoming Leader. State: Replica: 2ae3a9e2293442e291b1262a717a5e50, State: Running, Role: LEADER
I20260709 12:27:57.749667 32144 raft_consensus.cc:515] T ff138d12849e4f63a9b257f4395179de P 07b97783285d4b22b6fd0f1d816ef930 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:57.761179 32134 ts_tablet_manager.cc:1434] T 703b2df19bf24c31a5acbe85178314ec P 64fd1d8a3cd14e66ac94c4fbcceaf039: Time spent starting tablet: real 0.009s	user 0.007s	sys 0.001s
I20260709 12:27:57.761863 32161 consensus_queue.cc:237] T c4f237debd4942b198e077f8e7091088 P 2ae3a9e2293442e291b1262a717a5e50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:57.765373 32144 leader_election.cc:290] T ff138d12849e4f63a9b257f4395179de P 07b97783285d4b22b6fd0f1d816ef930 [CANDIDATE]: Term 1 election: Requested vote from peers 2ae3a9e2293442e291b1262a717a5e50 (127.16.29.65:44289), 64fd1d8a3cd14e66ac94c4fbcceaf039 (127.16.29.66:42163)
I20260709 12:27:57.766800 31834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff138d12849e4f63a9b257f4395179de" candidate_uuid: "07b97783285d4b22b6fd0f1d816ef930" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ae3a9e2293442e291b1262a717a5e50"
I20260709 12:27:57.767428 31834 raft_consensus.cc:3060] T ff138d12849e4f63a9b257f4395179de P 2ae3a9e2293442e291b1262a717a5e50 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:57.771463 31909 raft_consensus.cc:2468] T c4f237debd4942b198e077f8e7091088 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ae3a9e2293442e291b1262a717a5e50 in term 1.
I20260709 12:27:57.777777 31909 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff138d12849e4f63a9b257f4395179de" candidate_uuid: "07b97783285d4b22b6fd0f1d816ef930" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039"
I20260709 12:27:57.778352 31909 raft_consensus.cc:3060] T ff138d12849e4f63a9b257f4395179de P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:57.790587 32149 consensus_queue.cc:1048] T c11f552ceb904522856ef259018a8442 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:57.795831 31909 raft_consensus.cc:2468] T ff138d12849e4f63a9b257f4395179de P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07b97783285d4b22b6fd0f1d816ef930 in term 1.
I20260709 12:27:57.797372 31947 leader_election.cc:304] T ff138d12849e4f63a9b257f4395179de P 07b97783285d4b22b6fd0f1d816ef930 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07b97783285d4b22b6fd0f1d816ef930, 64fd1d8a3cd14e66ac94c4fbcceaf039; no voters: 
I20260709 12:27:57.804710 32144 raft_consensus.cc:2804] T ff138d12849e4f63a9b257f4395179de P 07b97783285d4b22b6fd0f1d816ef930 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:57.804857 31834 raft_consensus.cc:2468] T ff138d12849e4f63a9b257f4395179de P 2ae3a9e2293442e291b1262a717a5e50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07b97783285d4b22b6fd0f1d816ef930 in term 1.
I20260709 12:27:57.806393 32144 raft_consensus.cc:697] T ff138d12849e4f63a9b257f4395179de P 07b97783285d4b22b6fd0f1d816ef930 [term 1 LEADER]: Becoming Leader. State: Replica: 07b97783285d4b22b6fd0f1d816ef930, State: Running, Role: LEADER
I20260709 12:27:57.807624 32144 consensus_queue.cc:237] T ff138d12849e4f63a9b257f4395179de P 07b97783285d4b22b6fd0f1d816ef930 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:57.818113 32142 consensus_queue.cc:1048] T 0d5a572ce9954c72b6d74a24da7de624 P 07b97783285d4b22b6fd0f1d816ef930 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:27:57.840674 31732 catalog_manager.cc:5697] T c4f237debd4942b198e077f8e7091088 P 2ae3a9e2293442e291b1262a717a5e50 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2ae3a9e2293442e291b1262a717a5e50 (127.16.29.65). New cstate: current_term: 1 leader_uuid: "2ae3a9e2293442e291b1262a717a5e50" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:57.843539 32149 consensus_queue.cc:1048] T 8eda1839ee054ee59ec8629c9139dd2e P 64fd1d8a3cd14e66ac94c4fbcceaf039 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:57.848807 31730 catalog_manager.cc:5697] T ff138d12849e4f63a9b257f4395179de P 07b97783285d4b22b6fd0f1d816ef930 reported cstate change: term changed from 0 to 1, leader changed from <none> to 07b97783285d4b22b6fd0f1d816ef930 (127.16.29.67). New cstate: current_term: 1 leader_uuid: "07b97783285d4b22b6fd0f1d816ef930" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } health_report { overall_health: HEALTHY } } }
I20260709 12:27:57.857138 32149 consensus_queue.cc:1048] T 8eda1839ee054ee59ec8629c9139dd2e P 64fd1d8a3cd14e66ac94c4fbcceaf039 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:27:58.024350 32161 consensus_queue.cc:1048] T 9fe7f1d426c94232a1ec785665ef1416 P 2ae3a9e2293442e291b1262a717a5e50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:27:58.034317 32160 consensus_queue.cc:1048] T 9fe7f1d426c94232a1ec785665ef1416 P 2ae3a9e2293442e291b1262a717a5e50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:27:58.040732 32161 consensus_queue.cc:1048] T b659027528154c4c86becb863718c732 P 2ae3a9e2293442e291b1262a717a5e50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:58.043202 32142 consensus_queue.cc:1048] T 0ca01749cbc64fd987e4174047baac00 P 07b97783285d4b22b6fd0f1d816ef930 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260709 12:27:58.061075 31781 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:27:58.067631 32052 consensus_queue.cc:1048] T 0ca01749cbc64fd987e4174047baac00 P 07b97783285d4b22b6fd0f1d816ef930 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:58.071637 32161 consensus_queue.cc:1048] T b659027528154c4c86becb863718c732 P 2ae3a9e2293442e291b1262a717a5e50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
I20260709 12:27:58.105701 32120 raft_consensus.cc:493] T 703b2df19bf24c31a5acbe85178314ec P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:58.106330 32120 raft_consensus.cc:515] T 703b2df19bf24c31a5acbe85178314ec P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:58.108382 32120 leader_election.cc:290] T 703b2df19bf24c31a5acbe85178314ec P 64fd1d8a3cd14e66ac94c4fbcceaf039 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2ae3a9e2293442e291b1262a717a5e50 (127.16.29.65:44289), 07b97783285d4b22b6fd0f1d816ef930 (127.16.29.67:34419)
I20260709 12:27:58.109454 32161 raft_consensus.cc:493] T 703b2df19bf24c31a5acbe85178314ec P 2ae3a9e2293442e291b1262a717a5e50 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:58.109534 31833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "703b2df19bf24c31a5acbe85178314ec" candidate_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ae3a9e2293442e291b1262a717a5e50" is_pre_election: true
I20260709 12:27:58.109768 31984 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "703b2df19bf24c31a5acbe85178314ec" candidate_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07b97783285d4b22b6fd0f1d816ef930" is_pre_election: true
I20260709 12:27:58.110085 32161 raft_consensus.cc:515] T 703b2df19bf24c31a5acbe85178314ec P 2ae3a9e2293442e291b1262a717a5e50 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:58.110757 31984 raft_consensus.cc:2468] T 703b2df19bf24c31a5acbe85178314ec P 07b97783285d4b22b6fd0f1d816ef930 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64fd1d8a3cd14e66ac94c4fbcceaf039 in term 0.
I20260709 12:27:58.111356 31833 raft_consensus.cc:2468] T 703b2df19bf24c31a5acbe85178314ec P 2ae3a9e2293442e291b1262a717a5e50 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64fd1d8a3cd14e66ac94c4fbcceaf039 in term 0.
I20260709 12:27:58.112217 31876 leader_election.cc:304] T 703b2df19bf24c31a5acbe85178314ec P 64fd1d8a3cd14e66ac94c4fbcceaf039 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07b97783285d4b22b6fd0f1d816ef930, 64fd1d8a3cd14e66ac94c4fbcceaf039; no voters: 
I20260709 12:27:58.112429 32161 leader_election.cc:290] T 703b2df19bf24c31a5acbe85178314ec P 2ae3a9e2293442e291b1262a717a5e50 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 64fd1d8a3cd14e66ac94c4fbcceaf039 (127.16.29.66:42163), 07b97783285d4b22b6fd0f1d816ef930 (127.16.29.67:34419)
I20260709 12:27:58.113215 32120 raft_consensus.cc:2804] T 703b2df19bf24c31a5acbe85178314ec P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:58.113628 31909 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "703b2df19bf24c31a5acbe85178314ec" candidate_uuid: "2ae3a9e2293442e291b1262a717a5e50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" is_pre_election: true
I20260709 12:27:58.113623 32120 raft_consensus.cc:493] T 703b2df19bf24c31a5acbe85178314ec P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:58.113615 31984 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "703b2df19bf24c31a5acbe85178314ec" candidate_uuid: "2ae3a9e2293442e291b1262a717a5e50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07b97783285d4b22b6fd0f1d816ef930" is_pre_election: true
I20260709 12:27:58.114408 32120 raft_consensus.cc:3060] T 703b2df19bf24c31a5acbe85178314ec P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:58.114835 31984 raft_consensus.cc:2468] T 703b2df19bf24c31a5acbe85178314ec P 07b97783285d4b22b6fd0f1d816ef930 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ae3a9e2293442e291b1262a717a5e50 in term 0.
I20260709 12:27:58.116108 31801 leader_election.cc:304] T 703b2df19bf24c31a5acbe85178314ec P 2ae3a9e2293442e291b1262a717a5e50 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07b97783285d4b22b6fd0f1d816ef930, 2ae3a9e2293442e291b1262a717a5e50; no voters: 
I20260709 12:27:58.117075 32162 raft_consensus.cc:2804] T 703b2df19bf24c31a5acbe85178314ec P 2ae3a9e2293442e291b1262a717a5e50 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:58.117479 32162 raft_consensus.cc:493] T 703b2df19bf24c31a5acbe85178314ec P 2ae3a9e2293442e291b1262a717a5e50 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:58.117817 32162 raft_consensus.cc:3060] T 703b2df19bf24c31a5acbe85178314ec P 2ae3a9e2293442e291b1262a717a5e50 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:58.122713 32120 raft_consensus.cc:515] T 703b2df19bf24c31a5acbe85178314ec P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:58.124166 31909 raft_consensus.cc:2393] T 703b2df19bf24c31a5acbe85178314ec P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2ae3a9e2293442e291b1262a717a5e50 in current term 1: Already voted for candidate 64fd1d8a3cd14e66ac94c4fbcceaf039 in this term.
I20260709 12:27:58.126191 31984 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "703b2df19bf24c31a5acbe85178314ec" candidate_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07b97783285d4b22b6fd0f1d816ef930"
I20260709 12:27:58.125982 31833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "703b2df19bf24c31a5acbe85178314ec" candidate_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ae3a9e2293442e291b1262a717a5e50"
I20260709 12:27:58.125577 32162 raft_consensus.cc:515] T 703b2df19bf24c31a5acbe85178314ec P 2ae3a9e2293442e291b1262a717a5e50 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:58.125757 32120 leader_election.cc:290] T 703b2df19bf24c31a5acbe85178314ec P 64fd1d8a3cd14e66ac94c4fbcceaf039 [CANDIDATE]: Term 1 election: Requested vote from peers 2ae3a9e2293442e291b1262a717a5e50 (127.16.29.65:44289), 07b97783285d4b22b6fd0f1d816ef930 (127.16.29.67:34419)
I20260709 12:27:58.127480 31984 raft_consensus.cc:3060] T 703b2df19bf24c31a5acbe85178314ec P 07b97783285d4b22b6fd0f1d816ef930 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:58.127763 31833 raft_consensus.cc:2393] T 703b2df19bf24c31a5acbe85178314ec P 2ae3a9e2293442e291b1262a717a5e50 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 64fd1d8a3cd14e66ac94c4fbcceaf039 in current term 1: Already voted for candidate 2ae3a9e2293442e291b1262a717a5e50 in this term.
I20260709 12:27:58.129684 32162 leader_election.cc:290] T 703b2df19bf24c31a5acbe85178314ec P 2ae3a9e2293442e291b1262a717a5e50 [CANDIDATE]: Term 1 election: Requested vote from peers 64fd1d8a3cd14e66ac94c4fbcceaf039 (127.16.29.66:42163), 07b97783285d4b22b6fd0f1d816ef930 (127.16.29.67:34419)
I20260709 12:27:58.130425 31909 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "703b2df19bf24c31a5acbe85178314ec" candidate_uuid: "2ae3a9e2293442e291b1262a717a5e50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039"
I20260709 12:27:58.131176 31909 raft_consensus.cc:2393] T 703b2df19bf24c31a5acbe85178314ec P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 2ae3a9e2293442e291b1262a717a5e50 in current term 1: Already voted for candidate 64fd1d8a3cd14e66ac94c4fbcceaf039 in this term.
I20260709 12:27:58.132975 31981 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "703b2df19bf24c31a5acbe85178314ec" candidate_uuid: "2ae3a9e2293442e291b1262a717a5e50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07b97783285d4b22b6fd0f1d816ef930"
I20260709 12:27:58.134135 31801 leader_election.cc:304] T 703b2df19bf24c31a5acbe85178314ec P 2ae3a9e2293442e291b1262a717a5e50 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2ae3a9e2293442e291b1262a717a5e50; no voters: 07b97783285d4b22b6fd0f1d816ef930, 64fd1d8a3cd14e66ac94c4fbcceaf039
I20260709 12:27:58.134709 31984 raft_consensus.cc:2468] T 703b2df19bf24c31a5acbe85178314ec P 07b97783285d4b22b6fd0f1d816ef930 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64fd1d8a3cd14e66ac94c4fbcceaf039 in term 1.
I20260709 12:27:58.134904 32162 raft_consensus.cc:2749] T 703b2df19bf24c31a5acbe85178314ec P 2ae3a9e2293442e291b1262a717a5e50 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 12:27:58.135758 31876 leader_election.cc:304] T 703b2df19bf24c31a5acbe85178314ec P 64fd1d8a3cd14e66ac94c4fbcceaf039 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 07b97783285d4b22b6fd0f1d816ef930, 64fd1d8a3cd14e66ac94c4fbcceaf039; no voters: 2ae3a9e2293442e291b1262a717a5e50
I20260709 12:27:58.136466 32120 raft_consensus.cc:2804] T 703b2df19bf24c31a5acbe85178314ec P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:58.136868 32120 raft_consensus.cc:697] T 703b2df19bf24c31a5acbe85178314ec P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 1 LEADER]: Becoming Leader. State: Replica: 64fd1d8a3cd14e66ac94c4fbcceaf039, State: Running, Role: LEADER
I20260709 12:27:58.137624 32120 consensus_queue.cc:237] T 703b2df19bf24c31a5acbe85178314ec P 64fd1d8a3cd14e66ac94c4fbcceaf039 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:58.146294 31733 catalog_manager.cc:5697] T 703b2df19bf24c31a5acbe85178314ec P 64fd1d8a3cd14e66ac94c4fbcceaf039 reported cstate change: term changed from 0 to 1, leader changed from <none> to 64fd1d8a3cd14e66ac94c4fbcceaf039 (127.16.29.66). New cstate: current_term: 1 leader_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:58.164808 32161 consensus_queue.cc:1048] T c4f237debd4942b198e077f8e7091088 P 2ae3a9e2293442e291b1262a717a5e50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:27:58.176060 32093 consensus_queue.cc:1048] T c4f237debd4942b198e077f8e7091088 P 2ae3a9e2293442e291b1262a717a5e50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:27:58.354082 31910 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "c11f552ceb904522856ef259018a8442"
dest_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039"
mode: GRACEFUL
new_leader_uuid: "2ae3a9e2293442e291b1262a717a5e50"
 from {username='slave'} at 127.0.0.1:33772
I20260709 12:27:58.354686 31910 raft_consensus.cc:606] T c11f552ceb904522856ef259018a8442 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 1 LEADER]: Received request to transfer leadership to TS 2ae3a9e2293442e291b1262a717a5e50
I20260709 12:27:58.360035 31984 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "0d5a572ce9954c72b6d74a24da7de624"
dest_uuid: "07b97783285d4b22b6fd0f1d816ef930"
mode: GRACEFUL
new_leader_uuid: "2ae3a9e2293442e291b1262a717a5e50"
 from {username='slave'} at 127.0.0.1:38900
I20260709 12:27:58.360536 31984 raft_consensus.cc:606] T 0d5a572ce9954c72b6d74a24da7de624 P 07b97783285d4b22b6fd0f1d816ef930 [term 1 LEADER]: Received request to transfer leadership to TS 2ae3a9e2293442e291b1262a717a5e50
I20260709 12:27:58.365484 31910 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "8eda1839ee054ee59ec8629c9139dd2e"
dest_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039"
mode: GRACEFUL
new_leader_uuid: "2ae3a9e2293442e291b1262a717a5e50"
 from {username='slave'} at 127.0.0.1:33772
I20260709 12:27:58.366132 31910 raft_consensus.cc:606] T 8eda1839ee054ee59ec8629c9139dd2e P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 1 LEADER]: Received request to transfer leadership to TS 2ae3a9e2293442e291b1262a717a5e50
I20260709 12:27:58.366220 32142 consensus_queue.cc:1048] T ff138d12849e4f63a9b257f4395179de P 07b97783285d4b22b6fd0f1d816ef930 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:27:58.370797 32149 raft_consensus.cc:993] T 8eda1839ee054ee59ec8629c9139dd2e P 64fd1d8a3cd14e66ac94c4fbcceaf039: : Instructing follower 2ae3a9e2293442e291b1262a717a5e50 to start an election
I20260709 12:27:58.371296 32149 raft_consensus.cc:1081] T 8eda1839ee054ee59ec8629c9139dd2e P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 1 LEADER]: Signalling peer 2ae3a9e2293442e291b1262a717a5e50 to start an election
I20260709 12:27:58.373150 31835 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "8eda1839ee054ee59ec8629c9139dd2e"
dest_uuid: "2ae3a9e2293442e291b1262a717a5e50"
 from {username='slave'} at 127.0.0.1:39996
I20260709 12:27:58.373728 31835 raft_consensus.cc:493] T 8eda1839ee054ee59ec8629c9139dd2e P 2ae3a9e2293442e291b1262a717a5e50 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:27:58.374068 31835 raft_consensus.cc:3060] T 8eda1839ee054ee59ec8629c9139dd2e P 2ae3a9e2293442e291b1262a717a5e50 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:58.380786 31834 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "b659027528154c4c86becb863718c732"
dest_uuid: "2ae3a9e2293442e291b1262a717a5e50"
mode: GRACEFUL
new_leader_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039"
 from {username='slave'} at 127.0.0.1:40002
I20260709 12:27:58.381467 31834 raft_consensus.cc:606] T b659027528154c4c86becb863718c732 P 2ae3a9e2293442e291b1262a717a5e50 [term 1 LEADER]: Received request to transfer leadership to TS 64fd1d8a3cd14e66ac94c4fbcceaf039
I20260709 12:27:58.381392 31835 raft_consensus.cc:515] T 8eda1839ee054ee59ec8629c9139dd2e P 2ae3a9e2293442e291b1262a717a5e50 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:58.382114 32052 consensus_queue.cc:1048] T ff138d12849e4f63a9b257f4395179de P 07b97783285d4b22b6fd0f1d816ef930 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:27:58.384358 31910 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8eda1839ee054ee59ec8629c9139dd2e" candidate_uuid: "2ae3a9e2293442e291b1262a717a5e50" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039"
I20260709 12:27:58.384979 31910 raft_consensus.cc:3055] T 8eda1839ee054ee59ec8629c9139dd2e P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:27:58.385277 31910 raft_consensus.cc:740] T 8eda1839ee054ee59ec8629c9139dd2e P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 64fd1d8a3cd14e66ac94c4fbcceaf039, State: Running, Role: LEADER
I20260709 12:27:58.386349 31835 leader_election.cc:290] T 8eda1839ee054ee59ec8629c9139dd2e P 2ae3a9e2293442e291b1262a717a5e50 [CANDIDATE]: Term 2 election: Requested vote from peers 64fd1d8a3cd14e66ac94c4fbcceaf039 (127.16.29.66:42163), 07b97783285d4b22b6fd0f1d816ef930 (127.16.29.67:34419)
I20260709 12:27:58.387480 31910 consensus_queue.cc:260] T 8eda1839ee054ee59ec8629c9139dd2e P 64fd1d8a3cd14e66ac94c4fbcceaf039 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:58.388743 31910 raft_consensus.cc:3060] T 8eda1839ee054ee59ec8629c9139dd2e P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:58.388808 31983 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "0ca01749cbc64fd987e4174047baac00"
dest_uuid: "07b97783285d4b22b6fd0f1d816ef930"
mode: GRACEFUL
new_leader_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039"
 from {username='slave'} at 127.0.0.1:38900
I20260709 12:27:58.389600 31983 raft_consensus.cc:606] T 0ca01749cbc64fd987e4174047baac00 P 07b97783285d4b22b6fd0f1d816ef930 [term 1 LEADER]: Received request to transfer leadership to TS 64fd1d8a3cd14e66ac94c4fbcceaf039
I20260709 12:27:58.392627 31984 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8eda1839ee054ee59ec8629c9139dd2e" candidate_uuid: "2ae3a9e2293442e291b1262a717a5e50" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "07b97783285d4b22b6fd0f1d816ef930"
I20260709 12:27:58.395793 31984 raft_consensus.cc:3060] T 8eda1839ee054ee59ec8629c9139dd2e P 07b97783285d4b22b6fd0f1d816ef930 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:58.396552 31835 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "c4f237debd4942b198e077f8e7091088"
dest_uuid: "2ae3a9e2293442e291b1262a717a5e50"
mode: GRACEFUL
new_leader_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039"
 from {username='slave'} at 127.0.0.1:40002
I20260709 12:27:58.397257 31835 raft_consensus.cc:606] T c4f237debd4942b198e077f8e7091088 P 2ae3a9e2293442e291b1262a717a5e50 [term 1 LEADER]: Received request to transfer leadership to TS 64fd1d8a3cd14e66ac94c4fbcceaf039
I20260709 12:27:58.399123 31910 raft_consensus.cc:2468] T 8eda1839ee054ee59ec8629c9139dd2e P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ae3a9e2293442e291b1262a717a5e50 in term 2.
I20260709 12:27:58.400589 31798 leader_election.cc:304] T 8eda1839ee054ee59ec8629c9139dd2e P 2ae3a9e2293442e291b1262a717a5e50 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ae3a9e2293442e291b1262a717a5e50, 64fd1d8a3cd14e66ac94c4fbcceaf039; no voters: 
I20260709 12:27:58.402956 32093 raft_consensus.cc:2804] T 8eda1839ee054ee59ec8629c9139dd2e P 2ae3a9e2293442e291b1262a717a5e50 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:27:58.403421 32093 raft_consensus.cc:697] T 8eda1839ee054ee59ec8629c9139dd2e P 2ae3a9e2293442e291b1262a717a5e50 [term 2 LEADER]: Becoming Leader. State: Replica: 2ae3a9e2293442e291b1262a717a5e50, State: Running, Role: LEADER
I20260709 12:27:58.404249 32093 consensus_queue.cc:237] T 8eda1839ee054ee59ec8629c9139dd2e P 2ae3a9e2293442e291b1262a717a5e50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:58.413251 32149 raft_consensus.cc:993] T c11f552ceb904522856ef259018a8442 P 64fd1d8a3cd14e66ac94c4fbcceaf039: : Instructing follower 2ae3a9e2293442e291b1262a717a5e50 to start an election
I20260709 12:27:58.414937 32149 raft_consensus.cc:1081] T c11f552ceb904522856ef259018a8442 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 1 LEADER]: Signalling peer 2ae3a9e2293442e291b1262a717a5e50 to start an election
I20260709 12:27:58.418357 31983 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "ff138d12849e4f63a9b257f4395179de"
dest_uuid: "07b97783285d4b22b6fd0f1d816ef930"
mode: GRACEFUL
new_leader_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039"
 from {username='slave'} at 127.0.0.1:38900
I20260709 12:27:58.419283 31983 raft_consensus.cc:606] T ff138d12849e4f63a9b257f4395179de P 07b97783285d4b22b6fd0f1d816ef930 [term 1 LEADER]: Received request to transfer leadership to TS 64fd1d8a3cd14e66ac94c4fbcceaf039
I20260709 12:27:58.419694 31984 raft_consensus.cc:2468] T 8eda1839ee054ee59ec8629c9139dd2e P 07b97783285d4b22b6fd0f1d816ef930 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ae3a9e2293442e291b1262a717a5e50 in term 2.
I20260709 12:27:58.425315 31910 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "703b2df19bf24c31a5acbe85178314ec"
dest_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039"
mode: GRACEFUL
new_leader_uuid: "07b97783285d4b22b6fd0f1d816ef930"
 from {username='slave'} at 127.0.0.1:33772
I20260709 12:27:58.425940 31910 raft_consensus.cc:606] T 703b2df19bf24c31a5acbe85178314ec P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 1 LEADER]: Received request to transfer leadership to TS 07b97783285d4b22b6fd0f1d816ef930
I20260709 12:27:58.425454 31733 catalog_manager.cc:5697] T 8eda1839ee054ee59ec8629c9139dd2e P 2ae3a9e2293442e291b1262a717a5e50 reported cstate change: term changed from 1 to 2, leader changed from 64fd1d8a3cd14e66ac94c4fbcceaf039 (127.16.29.66) to 2ae3a9e2293442e291b1262a717a5e50 (127.16.29.65). New cstate: current_term: 2 leader_uuid: "2ae3a9e2293442e291b1262a717a5e50" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:58.428640 31835 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "c11f552ceb904522856ef259018a8442"
dest_uuid: "2ae3a9e2293442e291b1262a717a5e50"
 from {username='slave'} at 127.0.0.1:39996
I20260709 12:27:58.429339 31835 raft_consensus.cc:493] T c11f552ceb904522856ef259018a8442 P 2ae3a9e2293442e291b1262a717a5e50 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:27:58.429744 31835 raft_consensus.cc:3060] T c11f552ceb904522856ef259018a8442 P 2ae3a9e2293442e291b1262a717a5e50 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:58.437098 31835 raft_consensus.cc:515] T c11f552ceb904522856ef259018a8442 P 2ae3a9e2293442e291b1262a717a5e50 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:58.439535 31835 leader_election.cc:290] T c11f552ceb904522856ef259018a8442 P 2ae3a9e2293442e291b1262a717a5e50 [CANDIDATE]: Term 2 election: Requested vote from peers 64fd1d8a3cd14e66ac94c4fbcceaf039 (127.16.29.66:42163), 07b97783285d4b22b6fd0f1d816ef930 (127.16.29.67:34419)
I20260709 12:27:58.439733 31910 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c11f552ceb904522856ef259018a8442" candidate_uuid: "2ae3a9e2293442e291b1262a717a5e50" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039"
I20260709 12:27:58.440213 31984 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c11f552ceb904522856ef259018a8442" candidate_uuid: "2ae3a9e2293442e291b1262a717a5e50" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "07b97783285d4b22b6fd0f1d816ef930"
I20260709 12:27:58.440491 31910 raft_consensus.cc:3055] T c11f552ceb904522856ef259018a8442 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:27:58.441002 31984 raft_consensus.cc:3060] T c11f552ceb904522856ef259018a8442 P 07b97783285d4b22b6fd0f1d816ef930 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:58.441048 31910 raft_consensus.cc:740] T c11f552ceb904522856ef259018a8442 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 64fd1d8a3cd14e66ac94c4fbcceaf039, State: Running, Role: LEADER
I20260709 12:27:58.442447 31910 consensus_queue.cc:260] T c11f552ceb904522856ef259018a8442 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:58.443395 31910 raft_consensus.cc:3060] T c11f552ceb904522856ef259018a8442 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:58.446985 31984 raft_consensus.cc:2468] T c11f552ceb904522856ef259018a8442 P 07b97783285d4b22b6fd0f1d816ef930 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ae3a9e2293442e291b1262a717a5e50 in term 2.
I20260709 12:27:58.448213 31801 leader_election.cc:304] T c11f552ceb904522856ef259018a8442 P 2ae3a9e2293442e291b1262a717a5e50 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07b97783285d4b22b6fd0f1d816ef930, 2ae3a9e2293442e291b1262a717a5e50; no voters: 
I20260709 12:27:58.448576 31910 raft_consensus.cc:2468] T c11f552ceb904522856ef259018a8442 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ae3a9e2293442e291b1262a717a5e50 in term 2.
I20260709 12:27:58.449115 32093 raft_consensus.cc:2804] T c11f552ceb904522856ef259018a8442 P 2ae3a9e2293442e291b1262a717a5e50 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:27:58.449649 32093 raft_consensus.cc:697] T c11f552ceb904522856ef259018a8442 P 2ae3a9e2293442e291b1262a717a5e50 [term 2 LEADER]: Becoming Leader. State: Replica: 2ae3a9e2293442e291b1262a717a5e50, State: Running, Role: LEADER
I20260709 12:27:58.450508 32093 consensus_queue.cc:237] T c11f552ceb904522856ef259018a8442 P 2ae3a9e2293442e291b1262a717a5e50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:58.456939 31733 catalog_manager.cc:5697] T c11f552ceb904522856ef259018a8442 P 2ae3a9e2293442e291b1262a717a5e50 reported cstate change: term changed from 1 to 2, leader changed from 64fd1d8a3cd14e66ac94c4fbcceaf039 (127.16.29.66) to 2ae3a9e2293442e291b1262a717a5e50 (127.16.29.65). New cstate: current_term: 2 leader_uuid: "2ae3a9e2293442e291b1262a717a5e50" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:58.472286 32162 raft_consensus.cc:993] T b659027528154c4c86becb863718c732 P 2ae3a9e2293442e291b1262a717a5e50: : Instructing follower 64fd1d8a3cd14e66ac94c4fbcceaf039 to start an election
I20260709 12:27:58.472811 32093 raft_consensus.cc:1081] T b659027528154c4c86becb863718c732 P 2ae3a9e2293442e291b1262a717a5e50 [term 1 LEADER]: Signalling peer 64fd1d8a3cd14e66ac94c4fbcceaf039 to start an election
I20260709 12:27:58.474370 31910 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "b659027528154c4c86becb863718c732"
dest_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039"
 from {username='slave'} at 127.0.0.1:33764
I20260709 12:27:58.474907 31910 raft_consensus.cc:493] T b659027528154c4c86becb863718c732 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:27:58.475265 31910 raft_consensus.cc:3060] T b659027528154c4c86becb863718c732 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:58.480353 31910 raft_consensus.cc:515] T b659027528154c4c86becb863718c732 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:58.482353 31910 leader_election.cc:290] T b659027528154c4c86becb863718c732 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [CANDIDATE]: Term 2 election: Requested vote from peers 2ae3a9e2293442e291b1262a717a5e50 (127.16.29.65:44289), 07b97783285d4b22b6fd0f1d816ef930 (127.16.29.67:34419)
I20260709 12:27:58.483258 31835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b659027528154c4c86becb863718c732" candidate_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2ae3a9e2293442e291b1262a717a5e50"
I20260709 12:27:58.483502 31984 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b659027528154c4c86becb863718c732" candidate_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "07b97783285d4b22b6fd0f1d816ef930"
I20260709 12:27:58.484165 31835 raft_consensus.cc:3055] T b659027528154c4c86becb863718c732 P 2ae3a9e2293442e291b1262a717a5e50 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:27:58.484280 31984 raft_consensus.cc:3060] T b659027528154c4c86becb863718c732 P 07b97783285d4b22b6fd0f1d816ef930 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:58.484623 31835 raft_consensus.cc:740] T b659027528154c4c86becb863718c732 P 2ae3a9e2293442e291b1262a717a5e50 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2ae3a9e2293442e291b1262a717a5e50, State: Running, Role: LEADER
I20260709 12:27:58.485594 31835 consensus_queue.cc:260] T b659027528154c4c86becb863718c732 P 2ae3a9e2293442e291b1262a717a5e50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:58.487000 31835 raft_consensus.cc:3060] T b659027528154c4c86becb863718c732 P 2ae3a9e2293442e291b1262a717a5e50 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:58.492149 31984 raft_consensus.cc:2468] T b659027528154c4c86becb863718c732 P 07b97783285d4b22b6fd0f1d816ef930 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64fd1d8a3cd14e66ac94c4fbcceaf039 in term 2.
I20260709 12:27:58.493432 31876 leader_election.cc:304] T b659027528154c4c86becb863718c732 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07b97783285d4b22b6fd0f1d816ef930, 64fd1d8a3cd14e66ac94c4fbcceaf039; no voters: 
I20260709 12:27:58.494287 32120 raft_consensus.cc:2804] T b659027528154c4c86becb863718c732 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:27:58.494517 31835 raft_consensus.cc:2468] T b659027528154c4c86becb863718c732 P 2ae3a9e2293442e291b1262a717a5e50 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64fd1d8a3cd14e66ac94c4fbcceaf039 in term 2.
I20260709 12:27:58.494854 32120 raft_consensus.cc:697] T b659027528154c4c86becb863718c732 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 2 LEADER]: Becoming Leader. State: Replica: 64fd1d8a3cd14e66ac94c4fbcceaf039, State: Running, Role: LEADER
I20260709 12:27:58.495884 32120 consensus_queue.cc:237] T b659027528154c4c86becb863718c732 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:58.503019 31733 catalog_manager.cc:5697] T b659027528154c4c86becb863718c732 P 64fd1d8a3cd14e66ac94c4fbcceaf039 reported cstate change: term changed from 1 to 2, leader changed from 2ae3a9e2293442e291b1262a717a5e50 (127.16.29.65) to 64fd1d8a3cd14e66ac94c4fbcceaf039 (127.16.29.66). New cstate: current_term: 2 leader_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:58.539420 32142 raft_consensus.cc:993] T 0ca01749cbc64fd987e4174047baac00 P 07b97783285d4b22b6fd0f1d816ef930: : Instructing follower 64fd1d8a3cd14e66ac94c4fbcceaf039 to start an election
I20260709 12:27:58.540024 32052 raft_consensus.cc:1081] T 0ca01749cbc64fd987e4174047baac00 P 07b97783285d4b22b6fd0f1d816ef930 [term 1 LEADER]: Signalling peer 64fd1d8a3cd14e66ac94c4fbcceaf039 to start an election
I20260709 12:27:58.541656 31910 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "0ca01749cbc64fd987e4174047baac00"
dest_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039"
 from {username='slave'} at 127.0.0.1:33758
I20260709 12:27:58.542194 31910 raft_consensus.cc:493] T 0ca01749cbc64fd987e4174047baac00 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:27:58.542520 31910 raft_consensus.cc:3060] T 0ca01749cbc64fd987e4174047baac00 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:58.547822 31910 raft_consensus.cc:515] T 0ca01749cbc64fd987e4174047baac00 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:58.549445 31910 leader_election.cc:290] T 0ca01749cbc64fd987e4174047baac00 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [CANDIDATE]: Term 2 election: Requested vote from peers 2ae3a9e2293442e291b1262a717a5e50 (127.16.29.65:44289), 07b97783285d4b22b6fd0f1d816ef930 (127.16.29.67:34419)
I20260709 12:27:58.550639 31984 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ca01749cbc64fd987e4174047baac00" candidate_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "07b97783285d4b22b6fd0f1d816ef930"
I20260709 12:27:58.550817 31835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ca01749cbc64fd987e4174047baac00" candidate_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2ae3a9e2293442e291b1262a717a5e50"
I20260709 12:27:58.551290 31984 raft_consensus.cc:3055] T 0ca01749cbc64fd987e4174047baac00 P 07b97783285d4b22b6fd0f1d816ef930 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:27:58.551471 31835 raft_consensus.cc:3060] T 0ca01749cbc64fd987e4174047baac00 P 2ae3a9e2293442e291b1262a717a5e50 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:58.551653 31984 raft_consensus.cc:740] T 0ca01749cbc64fd987e4174047baac00 P 07b97783285d4b22b6fd0f1d816ef930 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 07b97783285d4b22b6fd0f1d816ef930, State: Running, Role: LEADER
I20260709 12:27:58.552444 31984 consensus_queue.cc:260] T 0ca01749cbc64fd987e4174047baac00 P 07b97783285d4b22b6fd0f1d816ef930 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:58.553587 31984 raft_consensus.cc:3060] T 0ca01749cbc64fd987e4174047baac00 P 07b97783285d4b22b6fd0f1d816ef930 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:58.557220 31835 raft_consensus.cc:2468] T 0ca01749cbc64fd987e4174047baac00 P 2ae3a9e2293442e291b1262a717a5e50 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64fd1d8a3cd14e66ac94c4fbcceaf039 in term 2.
I20260709 12:27:58.558576 31875 leader_election.cc:304] T 0ca01749cbc64fd987e4174047baac00 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ae3a9e2293442e291b1262a717a5e50, 64fd1d8a3cd14e66ac94c4fbcceaf039; no voters: 
I20260709 12:27:58.558810 31984 raft_consensus.cc:2468] T 0ca01749cbc64fd987e4174047baac00 P 07b97783285d4b22b6fd0f1d816ef930 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64fd1d8a3cd14e66ac94c4fbcceaf039 in term 2.
I20260709 12:27:58.559376 32200 raft_consensus.cc:2804] T 0ca01749cbc64fd987e4174047baac00 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:27:58.559839 32200 raft_consensus.cc:697] T 0ca01749cbc64fd987e4174047baac00 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 2 LEADER]: Becoming Leader. State: Replica: 64fd1d8a3cd14e66ac94c4fbcceaf039, State: Running, Role: LEADER
I20260709 12:27:58.560600 32200 consensus_queue.cc:237] T 0ca01749cbc64fd987e4174047baac00 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:58.567615 31733 catalog_manager.cc:5697] T 0ca01749cbc64fd987e4174047baac00 P 64fd1d8a3cd14e66ac94c4fbcceaf039 reported cstate change: term changed from 1 to 2, leader changed from 07b97783285d4b22b6fd0f1d816ef930 (127.16.29.67) to 64fd1d8a3cd14e66ac94c4fbcceaf039 (127.16.29.66). New cstate: current_term: 2 leader_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:58.581079 32162 raft_consensus.cc:993] T c4f237debd4942b198e077f8e7091088 P 2ae3a9e2293442e291b1262a717a5e50: : Instructing follower 64fd1d8a3cd14e66ac94c4fbcceaf039 to start an election
I20260709 12:27:58.581552 32093 raft_consensus.cc:1081] T c4f237debd4942b198e077f8e7091088 P 2ae3a9e2293442e291b1262a717a5e50 [term 1 LEADER]: Signalling peer 64fd1d8a3cd14e66ac94c4fbcceaf039 to start an election
I20260709 12:27:58.583038 31910 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "c4f237debd4942b198e077f8e7091088"
dest_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039"
 from {username='slave'} at 127.0.0.1:33764
I20260709 12:27:58.583529 31910 raft_consensus.cc:493] T c4f237debd4942b198e077f8e7091088 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:27:58.583811 31910 raft_consensus.cc:3060] T c4f237debd4942b198e077f8e7091088 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:58.588856 31910 raft_consensus.cc:515] T c4f237debd4942b198e077f8e7091088 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:58.590618 31910 leader_election.cc:290] T c4f237debd4942b198e077f8e7091088 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [CANDIDATE]: Term 2 election: Requested vote from peers 2ae3a9e2293442e291b1262a717a5e50 (127.16.29.65:44289), 07b97783285d4b22b6fd0f1d816ef930 (127.16.29.67:34419)
I20260709 12:27:58.591454 31835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c4f237debd4942b198e077f8e7091088" candidate_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2ae3a9e2293442e291b1262a717a5e50"
I20260709 12:27:58.591748 31984 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c4f237debd4942b198e077f8e7091088" candidate_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "07b97783285d4b22b6fd0f1d816ef930"
I20260709 12:27:58.592116 31835 raft_consensus.cc:3055] T c4f237debd4942b198e077f8e7091088 P 2ae3a9e2293442e291b1262a717a5e50 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:27:58.592350 31984 raft_consensus.cc:3060] T c4f237debd4942b198e077f8e7091088 P 07b97783285d4b22b6fd0f1d816ef930 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:58.592505 31835 raft_consensus.cc:740] T c4f237debd4942b198e077f8e7091088 P 2ae3a9e2293442e291b1262a717a5e50 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2ae3a9e2293442e291b1262a717a5e50, State: Running, Role: LEADER
I20260709 12:27:58.593307 31835 consensus_queue.cc:260] T c4f237debd4942b198e077f8e7091088 P 2ae3a9e2293442e291b1262a717a5e50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:58.594429 31835 raft_consensus.cc:3060] T c4f237debd4942b198e077f8e7091088 P 2ae3a9e2293442e291b1262a717a5e50 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:58.597530 31984 raft_consensus.cc:2468] T c4f237debd4942b198e077f8e7091088 P 07b97783285d4b22b6fd0f1d816ef930 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64fd1d8a3cd14e66ac94c4fbcceaf039 in term 2.
I20260709 12:27:58.598666 31876 leader_election.cc:304] T c4f237debd4942b198e077f8e7091088 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07b97783285d4b22b6fd0f1d816ef930, 64fd1d8a3cd14e66ac94c4fbcceaf039; no voters: 
I20260709 12:27:58.599494 32200 raft_consensus.cc:2804] T c4f237debd4942b198e077f8e7091088 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:27:58.600065 32200 raft_consensus.cc:697] T c4f237debd4942b198e077f8e7091088 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 2 LEADER]: Becoming Leader. State: Replica: 64fd1d8a3cd14e66ac94c4fbcceaf039, State: Running, Role: LEADER
I20260709 12:27:58.600870 31835 raft_consensus.cc:2468] T c4f237debd4942b198e077f8e7091088 P 2ae3a9e2293442e291b1262a717a5e50 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64fd1d8a3cd14e66ac94c4fbcceaf039 in term 2.
I20260709 12:27:58.600838 32200 consensus_queue.cc:237] T c4f237debd4942b198e077f8e7091088 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:58.607514 31733 catalog_manager.cc:5697] T c4f237debd4942b198e077f8e7091088 P 64fd1d8a3cd14e66ac94c4fbcceaf039 reported cstate change: term changed from 1 to 2, leader changed from 2ae3a9e2293442e291b1262a717a5e50 (127.16.29.65) to 64fd1d8a3cd14e66ac94c4fbcceaf039 (127.16.29.66). New cstate: current_term: 2 leader_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:58.670468 32200 consensus_queue.cc:1048] T 703b2df19bf24c31a5acbe85178314ec P 64fd1d8a3cd14e66ac94c4fbcceaf039 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:27:58.686985 32120 consensus_queue.cc:1048] T 703b2df19bf24c31a5acbe85178314ec P 64fd1d8a3cd14e66ac94c4fbcceaf039 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:58.699354 32120 raft_consensus.cc:993] T 703b2df19bf24c31a5acbe85178314ec P 64fd1d8a3cd14e66ac94c4fbcceaf039: : Instructing follower 07b97783285d4b22b6fd0f1d816ef930 to start an election
I20260709 12:27:58.699810 32120 raft_consensus.cc:1081] T 703b2df19bf24c31a5acbe85178314ec P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 1 LEADER]: Signalling peer 07b97783285d4b22b6fd0f1d816ef930 to start an election
I20260709 12:27:58.701510 31983 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "703b2df19bf24c31a5acbe85178314ec"
dest_uuid: "07b97783285d4b22b6fd0f1d816ef930"
 from {username='slave'} at 127.0.0.1:38872
I20260709 12:27:58.702116 31983 raft_consensus.cc:493] T 703b2df19bf24c31a5acbe85178314ec P 07b97783285d4b22b6fd0f1d816ef930 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:27:58.703267 31983 raft_consensus.cc:3060] T 703b2df19bf24c31a5acbe85178314ec P 07b97783285d4b22b6fd0f1d816ef930 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:58.704247 32142 raft_consensus.cc:993] T 0d5a572ce9954c72b6d74a24da7de624 P 07b97783285d4b22b6fd0f1d816ef930: : Instructing follower 2ae3a9e2293442e291b1262a717a5e50 to start an election
I20260709 12:27:58.704774 32052 raft_consensus.cc:1081] T 0d5a572ce9954c72b6d74a24da7de624 P 07b97783285d4b22b6fd0f1d816ef930 [term 1 LEADER]: Signalling peer 2ae3a9e2293442e291b1262a717a5e50 to start an election
I20260709 12:27:58.706373 31835 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "0d5a572ce9954c72b6d74a24da7de624"
dest_uuid: "2ae3a9e2293442e291b1262a717a5e50"
 from {username='slave'} at 127.0.0.1:39986
I20260709 12:27:58.706938 31835 raft_consensus.cc:493] T 0d5a572ce9954c72b6d74a24da7de624 P 2ae3a9e2293442e291b1262a717a5e50 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:27:58.707319 31835 raft_consensus.cc:3060] T 0d5a572ce9954c72b6d74a24da7de624 P 2ae3a9e2293442e291b1262a717a5e50 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:58.710830 31983 raft_consensus.cc:515] T 703b2df19bf24c31a5acbe85178314ec P 07b97783285d4b22b6fd0f1d816ef930 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:58.715065 31835 raft_consensus.cc:515] T 0d5a572ce9954c72b6d74a24da7de624 P 2ae3a9e2293442e291b1262a717a5e50 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:58.717329 31835 leader_election.cc:290] T 0d5a572ce9954c72b6d74a24da7de624 P 2ae3a9e2293442e291b1262a717a5e50 [CANDIDATE]: Term 2 election: Requested vote from peers 64fd1d8a3cd14e66ac94c4fbcceaf039 (127.16.29.66:42163), 07b97783285d4b22b6fd0f1d816ef930 (127.16.29.67:34419)
I20260709 12:27:58.717784 31833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "703b2df19bf24c31a5acbe85178314ec" candidate_uuid: "07b97783285d4b22b6fd0f1d816ef930" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2ae3a9e2293442e291b1262a717a5e50"
I20260709 12:27:58.718447 31833 raft_consensus.cc:3060] T 703b2df19bf24c31a5acbe85178314ec P 2ae3a9e2293442e291b1262a717a5e50 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:58.718513 31984 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d5a572ce9954c72b6d74a24da7de624" candidate_uuid: "2ae3a9e2293442e291b1262a717a5e50" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "07b97783285d4b22b6fd0f1d816ef930"
I20260709 12:27:58.719172 31984 raft_consensus.cc:3055] T 0d5a572ce9954c72b6d74a24da7de624 P 07b97783285d4b22b6fd0f1d816ef930 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:27:58.719525 31984 raft_consensus.cc:740] T 0d5a572ce9954c72b6d74a24da7de624 P 07b97783285d4b22b6fd0f1d816ef930 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 07b97783285d4b22b6fd0f1d816ef930, State: Running, Role: LEADER
I20260709 12:27:58.720497 31984 consensus_queue.cc:260] T 0d5a572ce9954c72b6d74a24da7de624 P 07b97783285d4b22b6fd0f1d816ef930 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:58.721838 31984 raft_consensus.cc:3060] T 0d5a572ce9954c72b6d74a24da7de624 P 07b97783285d4b22b6fd0f1d816ef930 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:58.725970 31910 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d5a572ce9954c72b6d74a24da7de624" candidate_uuid: "2ae3a9e2293442e291b1262a717a5e50" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039"
I20260709 12:27:58.726868 31910 raft_consensus.cc:3060] T 0d5a572ce9954c72b6d74a24da7de624 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:58.727988 31833 raft_consensus.cc:2468] T 703b2df19bf24c31a5acbe85178314ec P 2ae3a9e2293442e291b1262a717a5e50 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07b97783285d4b22b6fd0f1d816ef930 in term 2.
I20260709 12:27:58.729039 31983 leader_election.cc:290] T 703b2df19bf24c31a5acbe85178314ec P 07b97783285d4b22b6fd0f1d816ef930 [CANDIDATE]: Term 2 election: Requested vote from peers 2ae3a9e2293442e291b1262a717a5e50 (127.16.29.65:44289), 64fd1d8a3cd14e66ac94c4fbcceaf039 (127.16.29.66:42163)
I20260709 12:27:58.729907 31984 raft_consensus.cc:2468] T 0d5a572ce9954c72b6d74a24da7de624 P 07b97783285d4b22b6fd0f1d816ef930 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ae3a9e2293442e291b1262a717a5e50 in term 2.
I20260709 12:27:58.732262 31909 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "703b2df19bf24c31a5acbe85178314ec" candidate_uuid: "07b97783285d4b22b6fd0f1d816ef930" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039"
I20260709 12:27:58.732925 31909 raft_consensus.cc:3055] T 703b2df19bf24c31a5acbe85178314ec P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:27:58.733210 31909 raft_consensus.cc:740] T 703b2df19bf24c31a5acbe85178314ec P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 64fd1d8a3cd14e66ac94c4fbcceaf039, State: Running, Role: LEADER
I20260709 12:27:58.733973 31909 consensus_queue.cc:260] T 703b2df19bf24c31a5acbe85178314ec P 64fd1d8a3cd14e66ac94c4fbcceaf039 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:58.735369 31909 raft_consensus.cc:3060] T 703b2df19bf24c31a5acbe85178314ec P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:58.737394 31949 leader_election.cc:304] T 703b2df19bf24c31a5acbe85178314ec P 07b97783285d4b22b6fd0f1d816ef930 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07b97783285d4b22b6fd0f1d816ef930, 2ae3a9e2293442e291b1262a717a5e50; no voters: 
I20260709 12:27:58.737526 31801 leader_election.cc:304] T 0d5a572ce9954c72b6d74a24da7de624 P 2ae3a9e2293442e291b1262a717a5e50 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07b97783285d4b22b6fd0f1d816ef930, 2ae3a9e2293442e291b1262a717a5e50; no voters: 
I20260709 12:27:58.738485 32093 raft_consensus.cc:2804] T 0d5a572ce9954c72b6d74a24da7de624 P 2ae3a9e2293442e291b1262a717a5e50 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:27:58.739056 32093 raft_consensus.cc:697] T 0d5a572ce9954c72b6d74a24da7de624 P 2ae3a9e2293442e291b1262a717a5e50 [term 2 LEADER]: Becoming Leader. State: Replica: 2ae3a9e2293442e291b1262a717a5e50, State: Running, Role: LEADER
I20260709 12:27:58.740413 32093 consensus_queue.cc:237] T 0d5a572ce9954c72b6d74a24da7de624 P 2ae3a9e2293442e291b1262a717a5e50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:58.742178 31909 raft_consensus.cc:2468] T 703b2df19bf24c31a5acbe85178314ec P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07b97783285d4b22b6fd0f1d816ef930 in term 2.
I20260709 12:27:58.743160 32052 raft_consensus.cc:2804] T 703b2df19bf24c31a5acbe85178314ec P 07b97783285d4b22b6fd0f1d816ef930 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:27:58.743793 32052 raft_consensus.cc:697] T 703b2df19bf24c31a5acbe85178314ec P 07b97783285d4b22b6fd0f1d816ef930 [term 2 LEADER]: Becoming Leader. State: Replica: 07b97783285d4b22b6fd0f1d816ef930, State: Running, Role: LEADER
I20260709 12:27:58.744724 32052 consensus_queue.cc:237] T 703b2df19bf24c31a5acbe85178314ec P 07b97783285d4b22b6fd0f1d816ef930 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:58.750460 31910 raft_consensus.cc:2468] T 0d5a572ce9954c72b6d74a24da7de624 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ae3a9e2293442e291b1262a717a5e50 in term 2.
I20260709 12:27:58.753509 31733 catalog_manager.cc:5697] T 703b2df19bf24c31a5acbe85178314ec P 07b97783285d4b22b6fd0f1d816ef930 reported cstate change: term changed from 1 to 2, leader changed from 64fd1d8a3cd14e66ac94c4fbcceaf039 (127.16.29.66) to 07b97783285d4b22b6fd0f1d816ef930 (127.16.29.67). New cstate: current_term: 2 leader_uuid: "07b97783285d4b22b6fd0f1d816ef930" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } health_report { overall_health: HEALTHY } } }
I20260709 12:27:58.755422 31730 catalog_manager.cc:5697] T 0d5a572ce9954c72b6d74a24da7de624 P 2ae3a9e2293442e291b1262a717a5e50 reported cstate change: term changed from 1 to 2, leader changed from 07b97783285d4b22b6fd0f1d816ef930 (127.16.29.67) to 2ae3a9e2293442e291b1262a717a5e50 (127.16.29.65). New cstate: current_term: 2 leader_uuid: "2ae3a9e2293442e291b1262a717a5e50" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:58.892797 31833 raft_consensus.cc:1275] T b659027528154c4c86becb863718c732 P 2ae3a9e2293442e291b1262a717a5e50 [term 2 FOLLOWER]: Refusing update from remote peer 64fd1d8a3cd14e66ac94c4fbcceaf039: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:27:58.894215 32149 consensus_queue.cc:1048] T b659027528154c4c86becb863718c732 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:27:58.901541 31984 raft_consensus.cc:1275] T 8eda1839ee054ee59ec8629c9139dd2e P 07b97783285d4b22b6fd0f1d816ef930 [term 2 FOLLOWER]: Refusing update from remote peer 2ae3a9e2293442e291b1262a717a5e50: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:27:58.903153 31983 raft_consensus.cc:1275] T b659027528154c4c86becb863718c732 P 07b97783285d4b22b6fd0f1d816ef930 [term 2 FOLLOWER]: Refusing update from remote peer 64fd1d8a3cd14e66ac94c4fbcceaf039: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:27:58.904778 32162 consensus_queue.cc:1048] T 8eda1839ee054ee59ec8629c9139dd2e P 2ae3a9e2293442e291b1262a717a5e50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:27:58.905511 32149 consensus_queue.cc:1048] T b659027528154c4c86becb863718c732 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:27:58.918222 31910 raft_consensus.cc:1275] T 8eda1839ee054ee59ec8629c9139dd2e P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 2 FOLLOWER]: Refusing update from remote peer 2ae3a9e2293442e291b1262a717a5e50: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:27:58.919806 32204 consensus_queue.cc:1048] T 8eda1839ee054ee59ec8629c9139dd2e P 2ae3a9e2293442e291b1262a717a5e50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:27:58.970150 31984 raft_consensus.cc:1275] T c11f552ceb904522856ef259018a8442 P 07b97783285d4b22b6fd0f1d816ef930 [term 2 FOLLOWER]: Refusing update from remote peer 2ae3a9e2293442e291b1262a717a5e50: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:27:58.971498 32162 consensus_queue.cc:1048] T c11f552ceb904522856ef259018a8442 P 2ae3a9e2293442e291b1262a717a5e50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:27:58.980463 31910 raft_consensus.cc:1275] T c11f552ceb904522856ef259018a8442 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 2 FOLLOWER]: Refusing update from remote peer 2ae3a9e2293442e291b1262a717a5e50: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:27:58.982003 32093 consensus_queue.cc:1048] T c11f552ceb904522856ef259018a8442 P 2ae3a9e2293442e291b1262a717a5e50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:27:59.002100 31984 raft_consensus.cc:1275] T 0ca01749cbc64fd987e4174047baac00 P 07b97783285d4b22b6fd0f1d816ef930 [term 2 FOLLOWER]: Refusing update from remote peer 64fd1d8a3cd14e66ac94c4fbcceaf039: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:27:59.003669 32200 consensus_queue.cc:1048] T 0ca01749cbc64fd987e4174047baac00 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:27:59.005720 32142 raft_consensus.cc:993] T ff138d12849e4f63a9b257f4395179de P 07b97783285d4b22b6fd0f1d816ef930: : Instructing follower 64fd1d8a3cd14e66ac94c4fbcceaf039 to start an election
I20260709 12:27:59.006275 32142 raft_consensus.cc:1081] T ff138d12849e4f63a9b257f4395179de P 07b97783285d4b22b6fd0f1d816ef930 [term 1 LEADER]: Signalling peer 64fd1d8a3cd14e66ac94c4fbcceaf039 to start an election
I20260709 12:27:59.009760 31910 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "ff138d12849e4f63a9b257f4395179de"
dest_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039"
 from {username='slave'} at 127.0.0.1:33758
I20260709 12:27:59.010398 31910 raft_consensus.cc:493] T ff138d12849e4f63a9b257f4395179de P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:27:59.010733 31910 raft_consensus.cc:3060] T ff138d12849e4f63a9b257f4395179de P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:59.019030 31910 raft_consensus.cc:515] T ff138d12849e4f63a9b257f4395179de P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:59.020924 31910 leader_election.cc:290] T ff138d12849e4f63a9b257f4395179de P 64fd1d8a3cd14e66ac94c4fbcceaf039 [CANDIDATE]: Term 2 election: Requested vote from peers 2ae3a9e2293442e291b1262a717a5e50 (127.16.29.65:44289), 07b97783285d4b22b6fd0f1d816ef930 (127.16.29.67:34419)
I20260709 12:27:59.022476 31983 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff138d12849e4f63a9b257f4395179de" candidate_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "07b97783285d4b22b6fd0f1d816ef930"
I20260709 12:27:59.023207 31983 raft_consensus.cc:3055] T ff138d12849e4f63a9b257f4395179de P 07b97783285d4b22b6fd0f1d816ef930 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:27:59.023648 31983 raft_consensus.cc:740] T ff138d12849e4f63a9b257f4395179de P 07b97783285d4b22b6fd0f1d816ef930 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 07b97783285d4b22b6fd0f1d816ef930, State: Running, Role: LEADER
I20260709 12:27:59.024760 31833 raft_consensus.cc:1275] T 0ca01749cbc64fd987e4174047baac00 P 2ae3a9e2293442e291b1262a717a5e50 [term 2 FOLLOWER]: Refusing update from remote peer 64fd1d8a3cd14e66ac94c4fbcceaf039: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:27:59.024583 31983 consensus_queue.cc:260] T ff138d12849e4f63a9b257f4395179de P 07b97783285d4b22b6fd0f1d816ef930 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:59.025197 31835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff138d12849e4f63a9b257f4395179de" candidate_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2ae3a9e2293442e291b1262a717a5e50"
I20260709 12:27:59.025873 31835 raft_consensus.cc:3060] T ff138d12849e4f63a9b257f4395179de P 2ae3a9e2293442e291b1262a717a5e50 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:59.026438 32199 consensus_queue.cc:1048] T 0ca01749cbc64fd987e4174047baac00 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:27:59.026106 31983 raft_consensus.cc:3060] T ff138d12849e4f63a9b257f4395179de P 07b97783285d4b22b6fd0f1d816ef930 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:59.037730 31835 raft_consensus.cc:2468] T ff138d12849e4f63a9b257f4395179de P 2ae3a9e2293442e291b1262a717a5e50 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64fd1d8a3cd14e66ac94c4fbcceaf039 in term 2.
I20260709 12:27:59.038014 31983 raft_consensus.cc:2468] T ff138d12849e4f63a9b257f4395179de P 07b97783285d4b22b6fd0f1d816ef930 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64fd1d8a3cd14e66ac94c4fbcceaf039 in term 2.
I20260709 12:27:59.039039 31875 leader_election.cc:304] T ff138d12849e4f63a9b257f4395179de P 64fd1d8a3cd14e66ac94c4fbcceaf039 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ae3a9e2293442e291b1262a717a5e50, 64fd1d8a3cd14e66ac94c4fbcceaf039; no voters: 
I20260709 12:27:59.039861 32199 raft_consensus.cc:2804] T ff138d12849e4f63a9b257f4395179de P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:27:59.040503 32199 raft_consensus.cc:697] T ff138d12849e4f63a9b257f4395179de P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 2 LEADER]: Becoming Leader. State: Replica: 64fd1d8a3cd14e66ac94c4fbcceaf039, State: Running, Role: LEADER
I20260709 12:27:59.041255 32199 consensus_queue.cc:237] T ff138d12849e4f63a9b257f4395179de P 64fd1d8a3cd14e66ac94c4fbcceaf039 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:27:59.048720 31730 catalog_manager.cc:5697] T ff138d12849e4f63a9b257f4395179de P 64fd1d8a3cd14e66ac94c4fbcceaf039 reported cstate change: term changed from 1 to 2, leader changed from 07b97783285d4b22b6fd0f1d816ef930 (127.16.29.67) to 64fd1d8a3cd14e66ac94c4fbcceaf039 (127.16.29.66). New cstate: current_term: 2 leader_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:59.100096 31983 raft_consensus.cc:1275] T c4f237debd4942b198e077f8e7091088 P 07b97783285d4b22b6fd0f1d816ef930 [term 2 FOLLOWER]: Refusing update from remote peer 64fd1d8a3cd14e66ac94c4fbcceaf039: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:27:59.101397 32199 consensus_queue.cc:1048] T c4f237debd4942b198e077f8e7091088 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:27:59.112818 31835 raft_consensus.cc:1275] T c4f237debd4942b198e077f8e7091088 P 2ae3a9e2293442e291b1262a717a5e50 [term 2 FOLLOWER]: Refusing update from remote peer 64fd1d8a3cd14e66ac94c4fbcceaf039: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:27:59.114712 32149 consensus_queue.cc:1048] T c4f237debd4942b198e077f8e7091088 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:27:59.163872 31983 raft_consensus.cc:1275] T 0d5a572ce9954c72b6d74a24da7de624 P 07b97783285d4b22b6fd0f1d816ef930 [term 2 FOLLOWER]: Refusing update from remote peer 2ae3a9e2293442e291b1262a717a5e50: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:27:59.165611 32162 consensus_queue.cc:1048] T 0d5a572ce9954c72b6d74a24da7de624 P 2ae3a9e2293442e291b1262a717a5e50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:27:59.175673 31910 raft_consensus.cc:1275] T 0d5a572ce9954c72b6d74a24da7de624 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 2 FOLLOWER]: Refusing update from remote peer 2ae3a9e2293442e291b1262a717a5e50: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:27:59.177695 32204 consensus_queue.cc:1048] T 0d5a572ce9954c72b6d74a24da7de624 P 2ae3a9e2293442e291b1262a717a5e50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:27:59.194571 31835 raft_consensus.cc:1275] T 703b2df19bf24c31a5acbe85178314ec P 2ae3a9e2293442e291b1262a717a5e50 [term 2 FOLLOWER]: Refusing update from remote peer 07b97783285d4b22b6fd0f1d816ef930: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:27:59.196058 32144 consensus_queue.cc:1048] T 703b2df19bf24c31a5acbe85178314ec P 07b97783285d4b22b6fd0f1d816ef930 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:27:59.204847 31910 raft_consensus.cc:1275] T 703b2df19bf24c31a5acbe85178314ec P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 2 FOLLOWER]: Refusing update from remote peer 07b97783285d4b22b6fd0f1d816ef930: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:27:59.206204 32210 consensus_queue.cc:1048] T 703b2df19bf24c31a5acbe85178314ec P 07b97783285d4b22b6fd0f1d816ef930 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:27:59.420491 31835 raft_consensus.cc:1275] T ff138d12849e4f63a9b257f4395179de P 2ae3a9e2293442e291b1262a717a5e50 [term 2 FOLLOWER]: Refusing update from remote peer 64fd1d8a3cd14e66ac94c4fbcceaf039: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:27:59.421773 32200 consensus_queue.cc:1048] T ff138d12849e4f63a9b257f4395179de P 64fd1d8a3cd14e66ac94c4fbcceaf039 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:27:59.433116 31983 raft_consensus.cc:1275] T ff138d12849e4f63a9b257f4395179de P 07b97783285d4b22b6fd0f1d816ef930 [term 2 FOLLOWER]: Refusing update from remote peer 64fd1d8a3cd14e66ac94c4fbcceaf039: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:27:59.435562 32200 consensus_queue.cc:1048] T ff138d12849e4f63a9b257f4395179de P 64fd1d8a3cd14e66ac94c4fbcceaf039 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:28:01.430015 31729 catalog_manager.cc:2533] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43858:
table { table_name: "soft_deleted_table" } modify_external_catalogs: true reserve_seconds: 3600
I20260709 12:28:01.454165 31729 catalog_manager.cc:3570] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43858:
table { table_name: "soft_deleted_table" } new_extra_configs { key: "kudu.table.disable_compaction" value: "true" }
I20260709 12:28:01.505658 16501 auto_leader_rebalancer.cc:141] leader rebalance for table soft_deleted_table
W20260709 12:28:01.682585 32245 log.cc:927] Time spent T 703b2df19bf24c31a5acbe85178314ec P 64fd1d8a3cd14e66ac94c4fbcceaf039: Append to log took a long time: real 0.052s	user 0.004s	sys 0.000s
W20260709 12:28:01.705739 32240 log.cc:927] Time spent T 9fe7f1d426c94232a1ec785665ef1416 P 2ae3a9e2293442e291b1262a717a5e50: Append to log took a long time: real 0.096s	user 0.000s	sys 0.002s
I20260709 12:28:01.781548 32263 tablet.cc:1724] T 0d5a572ce9954c72b6d74a24da7de624 P 2ae3a9e2293442e291b1262a717a5e50: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
W20260709 12:28:01.803109 31949 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:44289, user_credentials={real_user=slave}} blocked reactor thread for 70682.9us
W20260709 12:28:01.804401 32249 log.cc:927] Time spent T ff138d12849e4f63a9b257f4395179de P 64fd1d8a3cd14e66ac94c4fbcceaf039: Append to log took a long time: real 0.104s	user 0.000s	sys 0.001s
W20260709 12:28:01.805732 32241 log.cc:927] Time spent T 0d5a572ce9954c72b6d74a24da7de624 P 07b97783285d4b22b6fd0f1d816ef930: Append to log took a long time: real 0.124s	user 0.001s	sys 0.000s
W20260709 12:28:01.829991 32253 log.cc:927] Time spent T b659027528154c4c86becb863718c732 P 2ae3a9e2293442e291b1262a717a5e50: Append to log took a long time: real 0.056s	user 0.000s	sys 0.000s
I20260709 12:28:01.877779 32270 tablet.cc:1724] T 0ca01749cbc64fd987e4174047baac00 P 64fd1d8a3cd14e66ac94c4fbcceaf039: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
W20260709 12:28:01.915272 31861 tablet_mm_ops.cc:139] Rowset compaction is disabled (check --enable_rowset_compaction and disable_compaction in extra_config for tablet:0d5a572ce9954c72b6d74a24da7de624)
I20260709 12:28:01.920737 32277 tablet.cc:1724] T 8eda1839ee054ee59ec8629c9139dd2e P 2ae3a9e2293442e291b1262a717a5e50: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:28:01.857162 32268 tablet.cc:1724] T 0d5a572ce9954c72b6d74a24da7de624 P 64fd1d8a3cd14e66ac94c4fbcceaf039: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:28:01.948163 32274 tablet.cc:1724] T 703b2df19bf24c31a5acbe85178314ec P 07b97783285d4b22b6fd0f1d816ef930: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:28:01.953701 32271 tablet.cc:1724] T b659027528154c4c86becb863718c732 P 64fd1d8a3cd14e66ac94c4fbcceaf039: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
W20260709 12:28:01.998736 32264 log.cc:927] Time spent T c4f237debd4942b198e077f8e7091088 P 2ae3a9e2293442e291b1262a717a5e50: Append to log took a long time: real 0.116s	user 0.008s	sys 0.000s
I20260709 12:28:01.998454 32276 tablet.cc:1724] T 0ca01749cbc64fd987e4174047baac00 P 07b97783285d4b22b6fd0f1d816ef930: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
W20260709 12:28:02.003450 32273 log.cc:927] Time spent T 8eda1839ee054ee59ec8629c9139dd2e P 07b97783285d4b22b6fd0f1d816ef930: Append to log took a long time: real 0.074s	user 0.001s	sys 0.000s
W20260709 12:28:02.009789 31861 tablet_mm_ops.cc:304] Major delta compaction is disabled (check --enable_major_delta_compaction and disable_compaction in extra_config for tablet:0d5a572ce9954c72b6d74a24da7de624)
W20260709 12:28:02.016394 32272 log.cc:927] Time spent T c4f237debd4942b198e077f8e7091088 P 07b97783285d4b22b6fd0f1d816ef930: Append to log took a long time: real 0.141s	user 0.006s	sys 0.005s
W20260709 12:28:02.019035 32267 log.cc:927] Time spent T 9fe7f1d426c94232a1ec785665ef1416 P 07b97783285d4b22b6fd0f1d816ef930: Append to log took a long time: real 0.090s	user 0.001s	sys 0.000s
I20260709 12:28:02.029189 31834 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "9fe7f1d426c94232a1ec785665ef1416"
dest_uuid: "2ae3a9e2293442e291b1262a717a5e50"
mode: GRACEFUL
new_leader_uuid: "07b97783285d4b22b6fd0f1d816ef930"
 from {username='slave'} at 127.0.0.1:49414
I20260709 12:28:02.030221 31834 raft_consensus.cc:606] T 9fe7f1d426c94232a1ec785665ef1416 P 2ae3a9e2293442e291b1262a717a5e50 [term 1 LEADER]: Received request to transfer leadership to TS 07b97783285d4b22b6fd0f1d816ef930
W20260709 12:28:02.021049 31861 tablet_mm_ops.cc:220] Minor delta compaction is disabled (check --enable_minor_delta_compaction and disable_compaction in extra_config for tablet:0d5a572ce9954c72b6d74a24da7de624)
W20260709 12:28:01.989907 31875 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.65:44289, user_credentials={real_user=slave}} blocked reactor thread for 84486.4us
I20260709 12:28:02.044082 32269 tablet.cc:1724] T 703b2df19bf24c31a5acbe85178314ec P 64fd1d8a3cd14e66ac94c4fbcceaf039: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
W20260709 12:28:02.048081 32275 log.cc:927] Time spent T ff138d12849e4f63a9b257f4395179de P 2ae3a9e2293442e291b1262a717a5e50: Append to log took a long time: real 0.171s	user 0.007s	sys 0.004s
I20260709 12:28:02.049218 32217 raft_consensus.cc:993] T 9fe7f1d426c94232a1ec785665ef1416 P 2ae3a9e2293442e291b1262a717a5e50: : Instructing follower 07b97783285d4b22b6fd0f1d816ef930 to start an election
I20260709 12:28:02.049957 32217 raft_consensus.cc:1081] T 9fe7f1d426c94232a1ec785665ef1416 P 2ae3a9e2293442e291b1262a717a5e50 [term 1 LEADER]: Signalling peer 07b97783285d4b22b6fd0f1d816ef930 to start an election
I20260709 12:28:02.056110 32280 tablet.cc:1724] T b659027528154c4c86becb863718c732 P 07b97783285d4b22b6fd0f1d816ef930: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:28:02.056457 32282 tablet.cc:1724] T 703b2df19bf24c31a5acbe85178314ec P 2ae3a9e2293442e291b1262a717a5e50: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:28:02.059139 31978 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "9fe7f1d426c94232a1ec785665ef1416"
dest_uuid: "07b97783285d4b22b6fd0f1d816ef930"
 from {username='slave'} at 127.0.0.1:38886
I20260709 12:28:02.059985 31978 raft_consensus.cc:493] T 9fe7f1d426c94232a1ec785665ef1416 P 07b97783285d4b22b6fd0f1d816ef930 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:28:02.060449 31978 raft_consensus.cc:3060] T 9fe7f1d426c94232a1ec785665ef1416 P 07b97783285d4b22b6fd0f1d816ef930 [term 1 FOLLOWER]: Advancing to term 2
W20260709 12:28:02.046411 31876 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.67:34419, user_credentials={real_user=slave}} blocked reactor thread for 44416.6us
I20260709 12:28:02.072584 32283 tablet.cc:1724] T c11f552ceb904522856ef259018a8442 P 2ae3a9e2293442e291b1262a717a5e50: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:28:02.073494 31978 raft_consensus.cc:515] T 9fe7f1d426c94232a1ec785665ef1416 P 07b97783285d4b22b6fd0f1d816ef930 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:28:02.087651 32269 tablet.cc:1724] T 8eda1839ee054ee59ec8629c9139dd2e P 64fd1d8a3cd14e66ac94c4fbcceaf039: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:28:02.104727 32263 tablet.cc:1724] T 9fe7f1d426c94232a1ec785665ef1416 P 2ae3a9e2293442e291b1262a717a5e50: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:28:02.119159 31832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe7f1d426c94232a1ec785665ef1416" candidate_uuid: "07b97783285d4b22b6fd0f1d816ef930" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "2ae3a9e2293442e291b1262a717a5e50"
I20260709 12:28:02.120016 31832 raft_consensus.cc:3055] T 9fe7f1d426c94232a1ec785665ef1416 P 2ae3a9e2293442e291b1262a717a5e50 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:28:02.120450 31832 raft_consensus.cc:740] T 9fe7f1d426c94232a1ec785665ef1416 P 2ae3a9e2293442e291b1262a717a5e50 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2ae3a9e2293442e291b1262a717a5e50, State: Running, Role: LEADER
I20260709 12:28:02.121342 32270 tablet.cc:1724] T c11f552ceb904522856ef259018a8442 P 64fd1d8a3cd14e66ac94c4fbcceaf039: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:28:02.104673 32274 tablet.cc:1724] T 0d5a572ce9954c72b6d74a24da7de624 P 07b97783285d4b22b6fd0f1d816ef930: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:28:02.125844 32268 tablet.cc:1724] T ff138d12849e4f63a9b257f4395179de P 64fd1d8a3cd14e66ac94c4fbcceaf039: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:28:02.122270 32277 tablet.cc:1724] T b659027528154c4c86becb863718c732 P 2ae3a9e2293442e291b1262a717a5e50: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:28:02.144418 31907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe7f1d426c94232a1ec785665ef1416" candidate_uuid: "07b97783285d4b22b6fd0f1d816ef930" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039"
I20260709 12:28:02.145157 31907 raft_consensus.cc:3060] T 9fe7f1d426c94232a1ec785665ef1416 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:02.145418 32285 tablet.cc:1724] T c11f552ceb904522856ef259018a8442 P 07b97783285d4b22b6fd0f1d816ef930: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:28:02.145856 32283 tablet.cc:1724] T c4f237debd4942b198e077f8e7091088 P 2ae3a9e2293442e291b1262a717a5e50: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:28:02.148860 31832 consensus_queue.cc:260] T 9fe7f1d426c94232a1ec785665ef1416 P 2ae3a9e2293442e291b1262a717a5e50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:28:02.150302 31832 raft_consensus.cc:3060] T 9fe7f1d426c94232a1ec785665ef1416 P 2ae3a9e2293442e291b1262a717a5e50 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:02.155375 32280 tablet.cc:1724] T 8eda1839ee054ee59ec8629c9139dd2e P 07b97783285d4b22b6fd0f1d816ef930: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:28:02.141170 31978 leader_election.cc:290] T 9fe7f1d426c94232a1ec785665ef1416 P 07b97783285d4b22b6fd0f1d816ef930 [CANDIDATE]: Term 2 election: Requested vote from peers 2ae3a9e2293442e291b1262a717a5e50 (127.16.29.65:44289), 64fd1d8a3cd14e66ac94c4fbcceaf039 (127.16.29.66:42163)
I20260709 12:28:02.176497 31907 raft_consensus.cc:2468] T 9fe7f1d426c94232a1ec785665ef1416 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07b97783285d4b22b6fd0f1d816ef930 in term 2.
I20260709 12:28:02.181911 32283 tablet.cc:1724] T ff138d12849e4f63a9b257f4395179de P 2ae3a9e2293442e291b1262a717a5e50: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:28:02.189798 32274 tablet.cc:1724] T ff138d12849e4f63a9b257f4395179de P 07b97783285d4b22b6fd0f1d816ef930: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:28:02.194172 31947 leader_election.cc:304] T 9fe7f1d426c94232a1ec785665ef1416 P 07b97783285d4b22b6fd0f1d816ef930 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07b97783285d4b22b6fd0f1d816ef930, 64fd1d8a3cd14e66ac94c4fbcceaf039; no voters: 
I20260709 12:28:02.196256 32216 raft_consensus.cc:2804] T 9fe7f1d426c94232a1ec785665ef1416 P 07b97783285d4b22b6fd0f1d816ef930 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:28:02.198017 32216 raft_consensus.cc:697] T 9fe7f1d426c94232a1ec785665ef1416 P 07b97783285d4b22b6fd0f1d816ef930 [term 2 LEADER]: Becoming Leader. State: Replica: 07b97783285d4b22b6fd0f1d816ef930, State: Running, Role: LEADER
I20260709 12:28:02.199076 32216 consensus_queue.cc:237] T 9fe7f1d426c94232a1ec785665ef1416 P 07b97783285d4b22b6fd0f1d816ef930 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:28:02.163653 32269 tablet.cc:1724] T 9fe7f1d426c94232a1ec785665ef1416 P 64fd1d8a3cd14e66ac94c4fbcceaf039: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:28:02.221654 32285 tablet.cc:1724] T c4f237debd4942b198e077f8e7091088 P 07b97783285d4b22b6fd0f1d816ef930: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:28:02.228973 31905 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "b659027528154c4c86becb863718c732"
dest_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039"
mode: GRACEFUL
new_leader_uuid: "07b97783285d4b22b6fd0f1d816ef930"
 from {username='slave'} at 127.0.0.1:53802
I20260709 12:28:02.229768 31905 raft_consensus.cc:606] T b659027528154c4c86becb863718c732 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 2 LEADER]: Received request to transfer leadership to TS 07b97783285d4b22b6fd0f1d816ef930
I20260709 12:28:02.231456 16501 auto_leader_rebalancer.cc:440] table: soft_deleted_table, leader rebalance finish, leader transfer count: 2
I20260709 12:28:02.232340 16501 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:28:02.160128 32271 tablet.cc:1724] T c4f237debd4942b198e077f8e7091088 P 64fd1d8a3cd14e66ac94c4fbcceaf039: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:28:02.234339 32263 tablet.cc:1724] T 0ca01749cbc64fd987e4174047baac00 P 2ae3a9e2293442e291b1262a717a5e50: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:28:02.231442 31832 raft_consensus.cc:2468] T 9fe7f1d426c94232a1ec785665ef1416 P 2ae3a9e2293442e291b1262a717a5e50 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07b97783285d4b22b6fd0f1d816ef930 in term 2.
I20260709 12:28:02.165848 32276 tablet.cc:1724] T 9fe7f1d426c94232a1ec785665ef1416 P 07b97783285d4b22b6fd0f1d816ef930: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:28:02.244822 31905 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "378f56c32c1e46f99a874bd1adcbcf25"
dest_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039"
mode: GRACEFUL
new_leader_uuid: "07b97783285d4b22b6fd0f1d816ef930"
 from {username='slave'} at 127.0.0.1:53802
I20260709 12:28:02.245462 31905 raft_consensus.cc:606] T 378f56c32c1e46f99a874bd1adcbcf25 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 2 LEADER]: Received request to transfer leadership to TS 07b97783285d4b22b6fd0f1d816ef930
I20260709 12:28:02.248510 31832 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "89784f886fb545d2871174c1e169fb32"
dest_uuid: "2ae3a9e2293442e291b1262a717a5e50"
mode: GRACEFUL
new_leader_uuid: "07b97783285d4b22b6fd0f1d816ef930"
 from {username='slave'} at 127.0.0.1:49414
I20260709 12:28:02.249303 31832 raft_consensus.cc:606] T 89784f886fb545d2871174c1e169fb32 P 2ae3a9e2293442e291b1262a717a5e50 [term 2 LEADER]: Received request to transfer leadership to TS 07b97783285d4b22b6fd0f1d816ef930
I20260709 12:28:02.250974 16501 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260709 12:28:02.251698 16501 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:28:02.253563 31733 catalog_manager.cc:5697] T 9fe7f1d426c94232a1ec785665ef1416 P 07b97783285d4b22b6fd0f1d816ef930 reported cstate change: term changed from 1 to 2, leader changed from 2ae3a9e2293442e291b1262a717a5e50 (127.16.29.65) to 07b97783285d4b22b6fd0f1d816ef930 (127.16.29.67). New cstate: current_term: 2 leader_uuid: "07b97783285d4b22b6fd0f1d816ef930" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } health_report { overall_health: HEALTHY } } }
I20260709 12:28:02.258527 31733 catalog_manager.cc:5789] TS 07b97783285d4b22b6fd0f1d816ef930 (127.16.29.67:34419) does not have the latest schema for tablet 9fe7f1d426c94232a1ec785665ef1416 (table soft_deleted_table [id=945de90ed607462385a524cf47b507d2]). Expected version 1 got 0
I20260709 12:28:02.274230 32238 raft_consensus.cc:993] T 89784f886fb545d2871174c1e169fb32 P 2ae3a9e2293442e291b1262a717a5e50: : Instructing follower 07b97783285d4b22b6fd0f1d816ef930 to start an election
I20260709 12:28:02.274928 32232 raft_consensus.cc:1081] T 89784f886fb545d2871174c1e169fb32 P 2ae3a9e2293442e291b1262a717a5e50 [term 2 LEADER]: Signalling peer 07b97783285d4b22b6fd0f1d816ef930 to start an election
I20260709 12:28:02.276922 31978 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "89784f886fb545d2871174c1e169fb32"
dest_uuid: "07b97783285d4b22b6fd0f1d816ef930"
 from {username='slave'} at 127.0.0.1:38886
I20260709 12:28:02.277518 31978 raft_consensus.cc:493] T 89784f886fb545d2871174c1e169fb32 P 07b97783285d4b22b6fd0f1d816ef930 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:28:02.277890 31978 raft_consensus.cc:3060] T 89784f886fb545d2871174c1e169fb32 P 07b97783285d4b22b6fd0f1d816ef930 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:28:02.285547 31978 raft_consensus.cc:515] T 89784f886fb545d2871174c1e169fb32 P 07b97783285d4b22b6fd0f1d816ef930 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } } peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } }
I20260709 12:28:02.287532 31978 leader_election.cc:290] T 89784f886fb545d2871174c1e169fb32 P 07b97783285d4b22b6fd0f1d816ef930 [CANDIDATE]: Term 3 election: Requested vote from peers 2ae3a9e2293442e291b1262a717a5e50 (127.16.29.65:44289), 64fd1d8a3cd14e66ac94c4fbcceaf039 (127.16.29.66:42163)
I20260709 12:28:02.288671 31832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "89784f886fb545d2871174c1e169fb32" candidate_uuid: "07b97783285d4b22b6fd0f1d816ef930" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "2ae3a9e2293442e291b1262a717a5e50"
I20260709 12:28:02.288883 31905 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "89784f886fb545d2871174c1e169fb32" candidate_uuid: "07b97783285d4b22b6fd0f1d816ef930" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039"
I20260709 12:28:02.296068 31832 raft_consensus.cc:3055] T 89784f886fb545d2871174c1e169fb32 P 2ae3a9e2293442e291b1262a717a5e50 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:28:02.297672 31832 raft_consensus.cc:740] T 89784f886fb545d2871174c1e169fb32 P 2ae3a9e2293442e291b1262a717a5e50 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 2ae3a9e2293442e291b1262a717a5e50, State: Running, Role: LEADER
I20260709 12:28:02.289574 31905 raft_consensus.cc:3060] T 89784f886fb545d2871174c1e169fb32 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:28:02.299019 31832 consensus_queue.cc:260] T 89784f886fb545d2871174c1e169fb32 P 2ae3a9e2293442e291b1262a717a5e50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } } peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } }
I20260709 12:28:02.300675 31832 raft_consensus.cc:3060] T 89784f886fb545d2871174c1e169fb32 P 2ae3a9e2293442e291b1262a717a5e50 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:28:02.306838 31905 raft_consensus.cc:2468] T 89784f886fb545d2871174c1e169fb32 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07b97783285d4b22b6fd0f1d816ef930 in term 3.
I20260709 12:28:02.308175 31947 leader_election.cc:304] T 89784f886fb545d2871174c1e169fb32 P 07b97783285d4b22b6fd0f1d816ef930 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07b97783285d4b22b6fd0f1d816ef930, 64fd1d8a3cd14e66ac94c4fbcceaf039; no voters: 
I20260709 12:28:02.309254 32262 raft_consensus.cc:2804] T 89784f886fb545d2871174c1e169fb32 P 07b97783285d4b22b6fd0f1d816ef930 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:28:02.309732 32262 raft_consensus.cc:697] T 89784f886fb545d2871174c1e169fb32 P 07b97783285d4b22b6fd0f1d816ef930 [term 3 LEADER]: Becoming Leader. State: Replica: 07b97783285d4b22b6fd0f1d816ef930, State: Running, Role: LEADER
I20260709 12:28:02.310674 32262 consensus_queue.cc:237] T 89784f886fb545d2871174c1e169fb32 P 07b97783285d4b22b6fd0f1d816ef930 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } } peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } }
I20260709 12:28:02.314539 31730 catalog_manager.cc:6242] T 00000000000000000000000000000000 P c569449549d1476cb2d243fd3c5e4804: soft_deleted_table [id=945de90ed607462385a524cf47b507d2] alter complete (version 1)
I20260709 12:28:02.320201 31832 raft_consensus.cc:2468] T 89784f886fb545d2871174c1e169fb32 P 2ae3a9e2293442e291b1262a717a5e50 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07b97783285d4b22b6fd0f1d816ef930 in term 3.
I20260709 12:28:02.332593 32278 raft_consensus.cc:993] T b659027528154c4c86becb863718c732 P 64fd1d8a3cd14e66ac94c4fbcceaf039: : Instructing follower 07b97783285d4b22b6fd0f1d816ef930 to start an election
I20260709 12:28:02.333109 32278 raft_consensus.cc:1081] T b659027528154c4c86becb863718c732 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 2 LEADER]: Signalling peer 07b97783285d4b22b6fd0f1d816ef930 to start an election
I20260709 12:28:02.335229 31978 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "b659027528154c4c86becb863718c732"
dest_uuid: "07b97783285d4b22b6fd0f1d816ef930"
 from {username='slave'} at 127.0.0.1:38872
I20260709 12:28:02.335798 31978 raft_consensus.cc:493] T b659027528154c4c86becb863718c732 P 07b97783285d4b22b6fd0f1d816ef930 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:28:02.336174 31978 raft_consensus.cc:3060] T b659027528154c4c86becb863718c732 P 07b97783285d4b22b6fd0f1d816ef930 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:28:02.344327 31978 raft_consensus.cc:515] T b659027528154c4c86becb863718c732 P 07b97783285d4b22b6fd0f1d816ef930 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:28:02.343870 31729 catalog_manager.cc:5697] T 89784f886fb545d2871174c1e169fb32 P 07b97783285d4b22b6fd0f1d816ef930 reported cstate change: term changed from 2 to 3, leader changed from 2ae3a9e2293442e291b1262a717a5e50 (127.16.29.65) to 07b97783285d4b22b6fd0f1d816ef930 (127.16.29.67). New cstate: current_term: 3 leader_uuid: "07b97783285d4b22b6fd0f1d816ef930" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:02.346629 31978 leader_election.cc:290] T b659027528154c4c86becb863718c732 P 07b97783285d4b22b6fd0f1d816ef930 [CANDIDATE]: Term 3 election: Requested vote from peers 2ae3a9e2293442e291b1262a717a5e50 (127.16.29.65:44289), 64fd1d8a3cd14e66ac94c4fbcceaf039 (127.16.29.66:42163)
I20260709 12:28:02.349110 31905 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b659027528154c4c86becb863718c732" candidate_uuid: "07b97783285d4b22b6fd0f1d816ef930" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: true dest_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039"
I20260709 12:28:02.349855 31905 raft_consensus.cc:3055] T b659027528154c4c86becb863718c732 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:28:02.350198 31905 raft_consensus.cc:740] T b659027528154c4c86becb863718c732 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 64fd1d8a3cd14e66ac94c4fbcceaf039, State: Running, Role: LEADER
I20260709 12:28:02.351116 31905 consensus_queue.cc:260] T b659027528154c4c86becb863718c732 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:28:02.351954 31832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b659027528154c4c86becb863718c732" candidate_uuid: "07b97783285d4b22b6fd0f1d816ef930" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: true dest_uuid: "2ae3a9e2293442e291b1262a717a5e50"
I20260709 12:28:02.352610 31832 raft_consensus.cc:3060] T b659027528154c4c86becb863718c732 P 2ae3a9e2293442e291b1262a717a5e50 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:28:02.352651 31905 raft_consensus.cc:3060] T b659027528154c4c86becb863718c732 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:28:02.360716 31832 raft_consensus.cc:2468] T b659027528154c4c86becb863718c732 P 2ae3a9e2293442e291b1262a717a5e50 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07b97783285d4b22b6fd0f1d816ef930 in term 3.
I20260709 12:28:02.361935 31949 leader_election.cc:304] T b659027528154c4c86becb863718c732 P 07b97783285d4b22b6fd0f1d816ef930 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07b97783285d4b22b6fd0f1d816ef930, 2ae3a9e2293442e291b1262a717a5e50; no voters: 
I20260709 12:28:02.362802 32262 raft_consensus.cc:2804] T b659027528154c4c86becb863718c732 P 07b97783285d4b22b6fd0f1d816ef930 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:28:02.363322 32262 raft_consensus.cc:697] T b659027528154c4c86becb863718c732 P 07b97783285d4b22b6fd0f1d816ef930 [term 3 LEADER]: Becoming Leader. State: Replica: 07b97783285d4b22b6fd0f1d816ef930, State: Running, Role: LEADER
I20260709 12:28:02.365586 31905 raft_consensus.cc:2468] T b659027528154c4c86becb863718c732 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07b97783285d4b22b6fd0f1d816ef930 in term 3.
I20260709 12:28:02.364382 32262 consensus_queue.cc:237] T b659027528154c4c86becb863718c732 P 07b97783285d4b22b6fd0f1d816ef930 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:28:02.369117 32199 raft_consensus.cc:993] T 378f56c32c1e46f99a874bd1adcbcf25 P 64fd1d8a3cd14e66ac94c4fbcceaf039: : Instructing follower 07b97783285d4b22b6fd0f1d816ef930 to start an election
I20260709 12:28:02.369542 32199 raft_consensus.cc:1081] T 378f56c32c1e46f99a874bd1adcbcf25 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 2 LEADER]: Signalling peer 07b97783285d4b22b6fd0f1d816ef930 to start an election
I20260709 12:28:02.371194 31978 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "378f56c32c1e46f99a874bd1adcbcf25"
dest_uuid: "07b97783285d4b22b6fd0f1d816ef930"
 from {username='slave'} at 127.0.0.1:38872
I20260709 12:28:02.371809 31978 raft_consensus.cc:493] T 378f56c32c1e46f99a874bd1adcbcf25 P 07b97783285d4b22b6fd0f1d816ef930 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:28:02.372291 31978 raft_consensus.cc:3060] T 378f56c32c1e46f99a874bd1adcbcf25 P 07b97783285d4b22b6fd0f1d816ef930 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:28:02.374627 31729 catalog_manager.cc:5697] T b659027528154c4c86becb863718c732 P 07b97783285d4b22b6fd0f1d816ef930 reported cstate change: term changed from 2 to 3, leader changed from 64fd1d8a3cd14e66ac94c4fbcceaf039 (127.16.29.66) to 07b97783285d4b22b6fd0f1d816ef930 (127.16.29.67). New cstate: current_term: 3 leader_uuid: "07b97783285d4b22b6fd0f1d816ef930" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } health_report { overall_health: HEALTHY } } }
I20260709 12:28:02.380484 31978 raft_consensus.cc:515] T 378f56c32c1e46f99a874bd1adcbcf25 P 07b97783285d4b22b6fd0f1d816ef930 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:28:02.383723 31832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378f56c32c1e46f99a874bd1adcbcf25" candidate_uuid: "07b97783285d4b22b6fd0f1d816ef930" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "2ae3a9e2293442e291b1262a717a5e50"
I20260709 12:28:02.384413 31832 raft_consensus.cc:3060] T 378f56c32c1e46f99a874bd1adcbcf25 P 2ae3a9e2293442e291b1262a717a5e50 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:28:02.391959 31905 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378f56c32c1e46f99a874bd1adcbcf25" candidate_uuid: "07b97783285d4b22b6fd0f1d816ef930" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039"
I20260709 12:28:02.392025 31832 raft_consensus.cc:2468] T 378f56c32c1e46f99a874bd1adcbcf25 P 2ae3a9e2293442e291b1262a717a5e50 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07b97783285d4b22b6fd0f1d816ef930 in term 3.
I20260709 12:28:02.392824 31905 raft_consensus.cc:3055] T 378f56c32c1e46f99a874bd1adcbcf25 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:28:02.393219 31905 raft_consensus.cc:740] T 378f56c32c1e46f99a874bd1adcbcf25 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 64fd1d8a3cd14e66ac94c4fbcceaf039, State: Running, Role: LEADER
I20260709 12:28:02.393662 31949 leader_election.cc:304] T 378f56c32c1e46f99a874bd1adcbcf25 P 07b97783285d4b22b6fd0f1d816ef930 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07b97783285d4b22b6fd0f1d816ef930, 2ae3a9e2293442e291b1262a717a5e50; no voters: 
I20260709 12:28:02.393858 31978 leader_election.cc:290] T 378f56c32c1e46f99a874bd1adcbcf25 P 07b97783285d4b22b6fd0f1d816ef930 [CANDIDATE]: Term 3 election: Requested vote from peers 2ae3a9e2293442e291b1262a717a5e50 (127.16.29.65:44289), 64fd1d8a3cd14e66ac94c4fbcceaf039 (127.16.29.66:42163)
I20260709 12:28:02.394698 32262 raft_consensus.cc:2804] T 378f56c32c1e46f99a874bd1adcbcf25 P 07b97783285d4b22b6fd0f1d816ef930 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:28:02.395380 32262 raft_consensus.cc:697] T 378f56c32c1e46f99a874bd1adcbcf25 P 07b97783285d4b22b6fd0f1d816ef930 [term 3 LEADER]: Becoming Leader. State: Replica: 07b97783285d4b22b6fd0f1d816ef930, State: Running, Role: LEADER
I20260709 12:28:02.394692 31905 consensus_queue.cc:260] T 378f56c32c1e46f99a874bd1adcbcf25 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:28:02.396349 32262 consensus_queue.cc:237] T 378f56c32c1e46f99a874bd1adcbcf25 P 07b97783285d4b22b6fd0f1d816ef930 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } }
I20260709 12:28:02.396714 31905 raft_consensus.cc:3060] T 378f56c32c1e46f99a874bd1adcbcf25 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:28:02.405277 31729 catalog_manager.cc:5697] T 378f56c32c1e46f99a874bd1adcbcf25 P 07b97783285d4b22b6fd0f1d816ef930 reported cstate change: term changed from 2 to 3, leader changed from 64fd1d8a3cd14e66ac94c4fbcceaf039 (127.16.29.66) to 07b97783285d4b22b6fd0f1d816ef930 (127.16.29.67). New cstate: current_term: 3 leader_uuid: "07b97783285d4b22b6fd0f1d816ef930" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07b97783285d4b22b6fd0f1d816ef930" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 34419 } health_report { overall_health: HEALTHY } } }
I20260709 12:28:02.409523 31905 raft_consensus.cc:2468] T 378f56c32c1e46f99a874bd1adcbcf25 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07b97783285d4b22b6fd0f1d816ef930 in term 3.
I20260709 12:28:02.631394 31905 raft_consensus.cc:1275] T 9fe7f1d426c94232a1ec785665ef1416 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 2 FOLLOWER]: Refusing update from remote peer 07b97783285d4b22b6fd0f1d816ef930: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260709 12:28:02.632855 32262 consensus_queue.cc:1048] T 9fe7f1d426c94232a1ec785665ef1416 P 07b97783285d4b22b6fd0f1d816ef930 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:28:02.642443 31832 raft_consensus.cc:1275] T 9fe7f1d426c94232a1ec785665ef1416 P 2ae3a9e2293442e291b1262a717a5e50 [term 2 FOLLOWER]: Refusing update from remote peer 07b97783285d4b22b6fd0f1d816ef930: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260709 12:28:02.644377 32262 consensus_queue.cc:1048] T 9fe7f1d426c94232a1ec785665ef1416 P 07b97783285d4b22b6fd0f1d816ef930 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:28:02.825047 31905 raft_consensus.cc:1275] T b659027528154c4c86becb863718c732 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 3 FOLLOWER]: Refusing update from remote peer 07b97783285d4b22b6fd0f1d816ef930: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260709 12:28:02.826277 32262 consensus_queue.cc:1048] T b659027528154c4c86becb863718c732 P 07b97783285d4b22b6fd0f1d816ef930 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:28:02.834213 31832 raft_consensus.cc:1275] T b659027528154c4c86becb863718c732 P 2ae3a9e2293442e291b1262a717a5e50 [term 3 FOLLOWER]: Refusing update from remote peer 07b97783285d4b22b6fd0f1d816ef930: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260709 12:28:02.835850 32295 consensus_queue.cc:1048] T b659027528154c4c86becb863718c732 P 07b97783285d4b22b6fd0f1d816ef930 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:28:02.872157 31832 raft_consensus.cc:1275] T 378f56c32c1e46f99a874bd1adcbcf25 P 2ae3a9e2293442e291b1262a717a5e50 [term 3 FOLLOWER]: Refusing update from remote peer 07b97783285d4b22b6fd0f1d816ef930: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:28:02.873543 32295 consensus_queue.cc:1048] T 378f56c32c1e46f99a874bd1adcbcf25 P 07b97783285d4b22b6fd0f1d816ef930 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:28:02.882329 31905 raft_consensus.cc:1275] T 378f56c32c1e46f99a874bd1adcbcf25 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 3 FOLLOWER]: Refusing update from remote peer 07b97783285d4b22b6fd0f1d816ef930: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:28:02.884073 32297 consensus_queue.cc:1048] T 378f56c32c1e46f99a874bd1adcbcf25 P 07b97783285d4b22b6fd0f1d816ef930 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:28:02.885553 31832 raft_consensus.cc:1275] T 89784f886fb545d2871174c1e169fb32 P 2ae3a9e2293442e291b1262a717a5e50 [term 3 FOLLOWER]: Refusing update from remote peer 07b97783285d4b22b6fd0f1d816ef930: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:28:02.887002 32262 consensus_queue.cc:1048] T 89784f886fb545d2871174c1e169fb32 P 07b97783285d4b22b6fd0f1d816ef930 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ae3a9e2293442e291b1262a717a5e50" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 44289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:28:02.908963 31907 raft_consensus.cc:1275] T 89784f886fb545d2871174c1e169fb32 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 3 FOLLOWER]: Refusing update from remote peer 07b97783285d4b22b6fd0f1d816ef930: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:28:02.910832 32295 consensus_queue.cc:1048] T 89784f886fb545d2871174c1e169fb32 P 07b97783285d4b22b6fd0f1d816ef930 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64fd1d8a3cd14e66ac94c4fbcceaf039" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 42163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:28:03.253225 16501 auto_leader_rebalancer.cc:141] leader rebalance for table soft_deleted_table
I20260709 12:28:03.262946 16501 auto_leader_rebalancer.cc:440] table: soft_deleted_table, leader rebalance finish, leader transfer count: 0
I20260709 12:28:03.263558 16501 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:28:03.272075 16501 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:28:03.272797 16501 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:28:04.273878 16501 auto_leader_rebalancer.cc:141] leader rebalance for table soft_deleted_table
I20260709 12:28:04.282644 16501 auto_leader_rebalancer.cc:440] table: soft_deleted_table, leader rebalance finish, leader transfer count: 0
I20260709 12:28:04.283339 16501 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:28:04.291551 16501 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:28:04.292155 16501 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:28:05.293360 16501 auto_leader_rebalancer.cc:141] leader rebalance for table soft_deleted_table
I20260709 12:28:05.300508 16501 auto_leader_rebalancer.cc:440] table: soft_deleted_table, leader rebalance finish, leader transfer count: 0
I20260709 12:28:05.301010 16501 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:28:05.310344 16501 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:28:05.310995 16501 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:28:06.312422 16501 auto_leader_rebalancer.cc:141] leader rebalance for table soft_deleted_table
I20260709 12:28:06.318548 16501 auto_leader_rebalancer.cc:440] table: soft_deleted_table, leader rebalance finish, leader transfer count: 0
I20260709 12:28:06.319097 16501 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:28:06.324986 16501 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:28:06.325465 16501 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:28:07.326800 16501 auto_leader_rebalancer.cc:141] leader rebalance for table soft_deleted_table
I20260709 12:28:07.332597 16501 auto_leader_rebalancer.cc:440] table: soft_deleted_table, leader rebalance finish, leader transfer count: 0
I20260709 12:28:07.333065 16501 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:28:07.338246 16501 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:28:07.338673 16501 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:28:08.339764 16501 auto_leader_rebalancer.cc:141] leader rebalance for table soft_deleted_table
I20260709 12:28:08.346101 16501 auto_leader_rebalancer.cc:440] table: soft_deleted_table, leader rebalance finish, leader transfer count: 0
I20260709 12:28:08.346546 16501 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:28:08.352126 16501 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:28:08.352634 16501 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:28:09.353688 16501 auto_leader_rebalancer.cc:141] leader rebalance for table soft_deleted_table
I20260709 12:28:09.358639 16501 auto_leader_rebalancer.cc:440] table: soft_deleted_table, leader rebalance finish, leader transfer count: 0
I20260709 12:28:09.359089 16501 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:28:09.364486 16501 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:28:09.364988 16501 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:28:10.366015 16501 auto_leader_rebalancer.cc:141] leader rebalance for table soft_deleted_table
I20260709 12:28:10.372231 16501 auto_leader_rebalancer.cc:440] table: soft_deleted_table, leader rebalance finish, leader transfer count: 0
I20260709 12:28:10.372646 16501 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:28:10.377317 16501 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:28:10.377676 16501 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:28:11.379000 16501 auto_leader_rebalancer.cc:141] leader rebalance for table soft_deleted_table
I20260709 12:28:11.388680 16501 auto_leader_rebalancer.cc:440] table: soft_deleted_table, leader rebalance finish, leader transfer count: 0
I20260709 12:28:11.389235 16501 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:28:11.395826 16501 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:28:11.396337 16501 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:28:12.457090 16501 tablet_server.cc:179] TabletServer@127.16.29.65:0 shutting down...
W20260709 12:28:12.492341 31949 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 10 similar messages]
W20260709 12:28:12.506983 31949 consensus_peers.cc:597] T 703b2df19bf24c31a5acbe85178314ec P 07b97783285d4b22b6fd0f1d816ef930 -> Peer 2ae3a9e2293442e291b1262a717a5e50 (127.16.29.65:44289): Couldn't send request to peer 2ae3a9e2293442e291b1262a717a5e50. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:28:12.507777 16501 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:28:12.508832 16501 tablet_replica.cc:333] T ff138d12849e4f63a9b257f4395179de P 2ae3a9e2293442e291b1262a717a5e50: stopping tablet replica
I20260709 12:28:12.510114 16501 raft_consensus.cc:2243] T ff138d12849e4f63a9b257f4395179de P 2ae3a9e2293442e291b1262a717a5e50 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:12.511243 16501 raft_consensus.cc:2272] T ff138d12849e4f63a9b257f4395179de P 2ae3a9e2293442e291b1262a717a5e50 [term 2 FOLLOWER]: Raft consensus is shut down!
W20260709 12:28:12.513267 31875 consensus_peers.cc:597] T c4f237debd4942b198e077f8e7091088 P 64fd1d8a3cd14e66ac94c4fbcceaf039 -> Peer 2ae3a9e2293442e291b1262a717a5e50 (127.16.29.65:44289): Couldn't send request to peer 2ae3a9e2293442e291b1262a717a5e50. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:28:12.514818 16501 tablet_replica.cc:333] T 0ca01749cbc64fd987e4174047baac00 P 2ae3a9e2293442e291b1262a717a5e50: stopping tablet replica
I20260709 12:28:12.515496 16501 raft_consensus.cc:2243] T 0ca01749cbc64fd987e4174047baac00 P 2ae3a9e2293442e291b1262a717a5e50 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:12.516078 16501 raft_consensus.cc:2272] T 0ca01749cbc64fd987e4174047baac00 P 2ae3a9e2293442e291b1262a717a5e50 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:12.518630 16501 tablet_replica.cc:333] T b659027528154c4c86becb863718c732 P 2ae3a9e2293442e291b1262a717a5e50: stopping tablet replica
I20260709 12:28:12.519315 16501 raft_consensus.cc:2243] T b659027528154c4c86becb863718c732 P 2ae3a9e2293442e291b1262a717a5e50 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:12.519989 16501 raft_consensus.cc:2272] T b659027528154c4c86becb863718c732 P 2ae3a9e2293442e291b1262a717a5e50 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:12.522457 16501 tablet_replica.cc:333] T 8eda1839ee054ee59ec8629c9139dd2e P 2ae3a9e2293442e291b1262a717a5e50: stopping tablet replica
I20260709 12:28:12.523058 16501 raft_consensus.cc:2243] T 8eda1839ee054ee59ec8629c9139dd2e P 2ae3a9e2293442e291b1262a717a5e50 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:28:12.524032 16501 raft_consensus.cc:2272] T 8eda1839ee054ee59ec8629c9139dd2e P 2ae3a9e2293442e291b1262a717a5e50 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:12.526432 16501 tablet_replica.cc:333] T fad0b7ef0f804aa494f44e9ec456e11e P 2ae3a9e2293442e291b1262a717a5e50: stopping tablet replica
I20260709 12:28:12.527081 16501 raft_consensus.cc:2243] T fad0b7ef0f804aa494f44e9ec456e11e P 2ae3a9e2293442e291b1262a717a5e50 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:28:12.528034 16501 raft_consensus.cc:2272] T fad0b7ef0f804aa494f44e9ec456e11e P 2ae3a9e2293442e291b1262a717a5e50 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:12.530392 16501 tablet_replica.cc:333] T 0d5a572ce9954c72b6d74a24da7de624 P 2ae3a9e2293442e291b1262a717a5e50: stopping tablet replica
I20260709 12:28:12.530905 16501 raft_consensus.cc:2243] T 0d5a572ce9954c72b6d74a24da7de624 P 2ae3a9e2293442e291b1262a717a5e50 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:28:12.532063 16501 raft_consensus.cc:2272] T 0d5a572ce9954c72b6d74a24da7de624 P 2ae3a9e2293442e291b1262a717a5e50 [term 2 FOLLOWER]: Raft consensus is shut down!
W20260709 12:28:12.535640 31949 consensus_peers.cc:597] T 378f56c32c1e46f99a874bd1adcbcf25 P 07b97783285d4b22b6fd0f1d816ef930 -> Peer 2ae3a9e2293442e291b1262a717a5e50 (127.16.29.65:44289): Couldn't send request to peer 2ae3a9e2293442e291b1262a717a5e50. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:28:12.535979 16501 tablet_replica.cc:333] T c11f552ceb904522856ef259018a8442 P 2ae3a9e2293442e291b1262a717a5e50: stopping tablet replica
I20260709 12:28:12.536687 16501 raft_consensus.cc:2243] T c11f552ceb904522856ef259018a8442 P 2ae3a9e2293442e291b1262a717a5e50 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:28:12.537565 16501 raft_consensus.cc:2272] T c11f552ceb904522856ef259018a8442 P 2ae3a9e2293442e291b1262a717a5e50 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:12.540108 16501 tablet_replica.cc:333] T 9fe7f1d426c94232a1ec785665ef1416 P 2ae3a9e2293442e291b1262a717a5e50: stopping tablet replica
I20260709 12:28:12.540717 16501 raft_consensus.cc:2243] T 9fe7f1d426c94232a1ec785665ef1416 P 2ae3a9e2293442e291b1262a717a5e50 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:12.541437 16501 raft_consensus.cc:2272] T 9fe7f1d426c94232a1ec785665ef1416 P 2ae3a9e2293442e291b1262a717a5e50 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:12.543625 16501 tablet_replica.cc:333] T 1638e215a9ee4068a999e92023fff415 P 2ae3a9e2293442e291b1262a717a5e50: stopping tablet replica
I20260709 12:28:12.544274 16501 raft_consensus.cc:2243] T 1638e215a9ee4068a999e92023fff415 P 2ae3a9e2293442e291b1262a717a5e50 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:28:12.545223 16501 raft_consensus.cc:2272] T 1638e215a9ee4068a999e92023fff415 P 2ae3a9e2293442e291b1262a717a5e50 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:12.547660 16501 tablet_replica.cc:333] T 6a5720f014ed48769eb641989edef3e8 P 2ae3a9e2293442e291b1262a717a5e50: stopping tablet replica
I20260709 12:28:12.548332 16501 raft_consensus.cc:2243] T 6a5720f014ed48769eb641989edef3e8 P 2ae3a9e2293442e291b1262a717a5e50 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:12.548878 16501 raft_consensus.cc:2272] T 6a5720f014ed48769eb641989edef3e8 P 2ae3a9e2293442e291b1262a717a5e50 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:12.551221 16501 tablet_replica.cc:333] T 703b2df19bf24c31a5acbe85178314ec P 2ae3a9e2293442e291b1262a717a5e50: stopping tablet replica
I20260709 12:28:12.551817 16501 raft_consensus.cc:2243] T 703b2df19bf24c31a5acbe85178314ec P 2ae3a9e2293442e291b1262a717a5e50 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:12.552419 16501 raft_consensus.cc:2272] T 703b2df19bf24c31a5acbe85178314ec P 2ae3a9e2293442e291b1262a717a5e50 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:12.554608 16501 tablet_replica.cc:333] T d00d4a3dbb1c435cb565cf6e5d723bf2 P 2ae3a9e2293442e291b1262a717a5e50: stopping tablet replica
I20260709 12:28:12.555176 16501 raft_consensus.cc:2243] T d00d4a3dbb1c435cb565cf6e5d723bf2 P 2ae3a9e2293442e291b1262a717a5e50 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:28:12.556110 16501 raft_consensus.cc:2272] T d00d4a3dbb1c435cb565cf6e5d723bf2 P 2ae3a9e2293442e291b1262a717a5e50 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:12.558496 16501 tablet_replica.cc:333] T 0fb83fa22e13487f9d56facc914278ba P 2ae3a9e2293442e291b1262a717a5e50: stopping tablet replica
I20260709 12:28:12.559046 16501 raft_consensus.cc:2243] T 0fb83fa22e13487f9d56facc914278ba P 2ae3a9e2293442e291b1262a717a5e50 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:12.559507 16501 raft_consensus.cc:2272] T 0fb83fa22e13487f9d56facc914278ba P 2ae3a9e2293442e291b1262a717a5e50 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:12.561780 16501 tablet_replica.cc:333] T 89784f886fb545d2871174c1e169fb32 P 2ae3a9e2293442e291b1262a717a5e50: stopping tablet replica
I20260709 12:28:12.562301 16501 raft_consensus.cc:2243] T 89784f886fb545d2871174c1e169fb32 P 2ae3a9e2293442e291b1262a717a5e50 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:12.562781 16501 raft_consensus.cc:2272] T 89784f886fb545d2871174c1e169fb32 P 2ae3a9e2293442e291b1262a717a5e50 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:12.565146 16501 tablet_replica.cc:333] T 52322cf19f734f19a4ccfc8389f32b41 P 2ae3a9e2293442e291b1262a717a5e50: stopping tablet replica
I20260709 12:28:12.565678 16501 raft_consensus.cc:2243] T 52322cf19f734f19a4ccfc8389f32b41 P 2ae3a9e2293442e291b1262a717a5e50 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:12.566181 16501 raft_consensus.cc:2272] T 52322cf19f734f19a4ccfc8389f32b41 P 2ae3a9e2293442e291b1262a717a5e50 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:12.568538 16501 tablet_replica.cc:333] T c4f237debd4942b198e077f8e7091088 P 2ae3a9e2293442e291b1262a717a5e50: stopping tablet replica
I20260709 12:28:12.569223 16501 raft_consensus.cc:2243] T c4f237debd4942b198e077f8e7091088 P 2ae3a9e2293442e291b1262a717a5e50 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:12.570058 16501 raft_consensus.cc:2272] T c4f237debd4942b198e077f8e7091088 P 2ae3a9e2293442e291b1262a717a5e50 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:12.573668 16501 tablet_replica.cc:333] T c61f10da93eb4af5a80a204e679f1ee3 P 2ae3a9e2293442e291b1262a717a5e50: stopping tablet replica
I20260709 12:28:12.574328 16501 raft_consensus.cc:2243] T c61f10da93eb4af5a80a204e679f1ee3 P 2ae3a9e2293442e291b1262a717a5e50 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:12.574923 16501 raft_consensus.cc:2272] T c61f10da93eb4af5a80a204e679f1ee3 P 2ae3a9e2293442e291b1262a717a5e50 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:12.577231 16501 tablet_replica.cc:333] T 378f56c32c1e46f99a874bd1adcbcf25 P 2ae3a9e2293442e291b1262a717a5e50: stopping tablet replica
I20260709 12:28:12.577688 16501 raft_consensus.cc:2243] T 378f56c32c1e46f99a874bd1adcbcf25 P 2ae3a9e2293442e291b1262a717a5e50 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:12.578276 16501 raft_consensus.cc:2272] T 378f56c32c1e46f99a874bd1adcbcf25 P 2ae3a9e2293442e291b1262a717a5e50 [term 3 FOLLOWER]: Raft consensus is shut down!
W20260709 12:28:12.627514 31875 consensus_peers.cc:597] T 6a5720f014ed48769eb641989edef3e8 P 64fd1d8a3cd14e66ac94c4fbcceaf039 -> Peer 2ae3a9e2293442e291b1262a717a5e50 (127.16.29.65:44289): Couldn't send request to peer 2ae3a9e2293442e291b1262a717a5e50. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:44289: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:28:12.628831 16501 tablet_server.cc:196] TabletServer@127.16.29.65:0 shutdown complete.
I20260709 12:28:12.677434 16501 tablet_server.cc:179] TabletServer@127.16.29.66:0 shutting down...
W20260709 12:28:12.700253 31949 consensus_peers.cc:597] T 89784f886fb545d2871174c1e169fb32 P 07b97783285d4b22b6fd0f1d816ef930 -> Peer 2ae3a9e2293442e291b1262a717a5e50 (127.16.29.65:44289): Couldn't send request to peer 2ae3a9e2293442e291b1262a717a5e50. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:44289: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:28:12.701032 31949 consensus_peers.cc:597] T 0fb83fa22e13487f9d56facc914278ba P 07b97783285d4b22b6fd0f1d816ef930 -> Peer 2ae3a9e2293442e291b1262a717a5e50 (127.16.29.65:44289): Couldn't send request to peer 2ae3a9e2293442e291b1262a717a5e50. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:44289: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:28:12.715627 16501 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:28:12.716634 16501 tablet_replica.cc:333] T ff138d12849e4f63a9b257f4395179de P 64fd1d8a3cd14e66ac94c4fbcceaf039: stopping tablet replica
I20260709 12:28:12.717576 16501 raft_consensus.cc:2243] T ff138d12849e4f63a9b257f4395179de P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:28:12.718640 16501 raft_consensus.cc:2272] T ff138d12849e4f63a9b257f4395179de P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:12.721508 16501 tablet_replica.cc:333] T 0ca01749cbc64fd987e4174047baac00 P 64fd1d8a3cd14e66ac94c4fbcceaf039: stopping tablet replica
I20260709 12:28:12.722218 16501 raft_consensus.cc:2243] T 0ca01749cbc64fd987e4174047baac00 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:28:12.723237 16501 raft_consensus.cc:2272] T 0ca01749cbc64fd987e4174047baac00 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:12.725950 16501 tablet_replica.cc:333] T b659027528154c4c86becb863718c732 P 64fd1d8a3cd14e66ac94c4fbcceaf039: stopping tablet replica
I20260709 12:28:12.726826 16501 raft_consensus.cc:2243] T b659027528154c4c86becb863718c732 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:12.727380 16501 raft_consensus.cc:2272] T b659027528154c4c86becb863718c732 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:12.729983 16501 tablet_replica.cc:333] T 8eda1839ee054ee59ec8629c9139dd2e P 64fd1d8a3cd14e66ac94c4fbcceaf039: stopping tablet replica
I20260709 12:28:12.730744 16501 raft_consensus.cc:2243] T 8eda1839ee054ee59ec8629c9139dd2e P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:12.731379 16501 raft_consensus.cc:2272] T 8eda1839ee054ee59ec8629c9139dd2e P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:12.734035 16501 tablet_replica.cc:333] T c11f552ceb904522856ef259018a8442 P 64fd1d8a3cd14e66ac94c4fbcceaf039: stopping tablet replica
I20260709 12:28:12.734652 16501 raft_consensus.cc:2243] T c11f552ceb904522856ef259018a8442 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:12.735240 16501 raft_consensus.cc:2272] T c11f552ceb904522856ef259018a8442 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:12.737867 16501 tablet_replica.cc:333] T fad0b7ef0f804aa494f44e9ec456e11e P 64fd1d8a3cd14e66ac94c4fbcceaf039: stopping tablet replica
I20260709 12:28:12.738530 16501 raft_consensus.cc:2243] T fad0b7ef0f804aa494f44e9ec456e11e P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:12.739034 16501 raft_consensus.cc:2272] T fad0b7ef0f804aa494f44e9ec456e11e P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:12.741616 16501 tablet_replica.cc:333] T 0d5a572ce9954c72b6d74a24da7de624 P 64fd1d8a3cd14e66ac94c4fbcceaf039: stopping tablet replica
I20260709 12:28:12.742256 16501 raft_consensus.cc:2243] T 0d5a572ce9954c72b6d74a24da7de624 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:12.742833 16501 raft_consensus.cc:2272] T 0d5a572ce9954c72b6d74a24da7de624 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:12.745441 16501 tablet_replica.cc:333] T 89784f886fb545d2871174c1e169fb32 P 64fd1d8a3cd14e66ac94c4fbcceaf039: stopping tablet replica
I20260709 12:28:12.746124 16501 raft_consensus.cc:2243] T 89784f886fb545d2871174c1e169fb32 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:12.746698 16501 raft_consensus.cc:2272] T 89784f886fb545d2871174c1e169fb32 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:12.749224 16501 tablet_replica.cc:333] T 0fb83fa22e13487f9d56facc914278ba P 64fd1d8a3cd14e66ac94c4fbcceaf039: stopping tablet replica
I20260709 12:28:12.749814 16501 raft_consensus.cc:2243] T 0fb83fa22e13487f9d56facc914278ba P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:12.750329 16501 raft_consensus.cc:2272] T 0fb83fa22e13487f9d56facc914278ba P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:12.752666 16501 tablet_replica.cc:333] T 52322cf19f734f19a4ccfc8389f32b41 P 64fd1d8a3cd14e66ac94c4fbcceaf039: stopping tablet replica
I20260709 12:28:12.753336 16501 raft_consensus.cc:2243] T 52322cf19f734f19a4ccfc8389f32b41 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:28:12.754309 16501 raft_consensus.cc:2272] T 52322cf19f734f19a4ccfc8389f32b41 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:12.756744 16501 tablet_replica.cc:333] T 378f56c32c1e46f99a874bd1adcbcf25 P 64fd1d8a3cd14e66ac94c4fbcceaf039: stopping tablet replica
I20260709 12:28:12.757369 16501 raft_consensus.cc:2243] T 378f56c32c1e46f99a874bd1adcbcf25 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:12.757906 16501 raft_consensus.cc:2272] T 378f56c32c1e46f99a874bd1adcbcf25 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:12.760135 16501 tablet_replica.cc:333] T c4f237debd4942b198e077f8e7091088 P 64fd1d8a3cd14e66ac94c4fbcceaf039: stopping tablet replica
I20260709 12:28:12.760715 16501 raft_consensus.cc:2243] T c4f237debd4942b198e077f8e7091088 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:28:12.761668 16501 raft_consensus.cc:2272] T c4f237debd4942b198e077f8e7091088 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:12.764184 16501 tablet_replica.cc:333] T c61f10da93eb4af5a80a204e679f1ee3 P 64fd1d8a3cd14e66ac94c4fbcceaf039: stopping tablet replica
I20260709 12:28:12.764884 16501 raft_consensus.cc:2243] T c61f10da93eb4af5a80a204e679f1ee3 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:28:12.765842 16501 raft_consensus.cc:2272] T c61f10da93eb4af5a80a204e679f1ee3 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:12.768199 16501 tablet_replica.cc:333] T 9fe7f1d426c94232a1ec785665ef1416 P 64fd1d8a3cd14e66ac94c4fbcceaf039: stopping tablet replica
I20260709 12:28:12.768802 16501 raft_consensus.cc:2243] T 9fe7f1d426c94232a1ec785665ef1416 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:12.769459 16501 raft_consensus.cc:2272] T 9fe7f1d426c94232a1ec785665ef1416 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:12.771708 16501 tablet_replica.cc:333] T 1638e215a9ee4068a999e92023fff415 P 64fd1d8a3cd14e66ac94c4fbcceaf039: stopping tablet replica
I20260709 12:28:12.772313 16501 raft_consensus.cc:2243] T 1638e215a9ee4068a999e92023fff415 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:12.772888 16501 raft_consensus.cc:2272] T 1638e215a9ee4068a999e92023fff415 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:12.775280 16501 tablet_replica.cc:333] T 703b2df19bf24c31a5acbe85178314ec P 64fd1d8a3cd14e66ac94c4fbcceaf039: stopping tablet replica
I20260709 12:28:12.775848 16501 raft_consensus.cc:2243] T 703b2df19bf24c31a5acbe85178314ec P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:12.776538 16501 raft_consensus.cc:2272] T 703b2df19bf24c31a5acbe85178314ec P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:12.778977 16501 tablet_replica.cc:333] T d00d4a3dbb1c435cb565cf6e5d723bf2 P 64fd1d8a3cd14e66ac94c4fbcceaf039: stopping tablet replica
I20260709 12:28:12.779497 16501 raft_consensus.cc:2243] T d00d4a3dbb1c435cb565cf6e5d723bf2 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:12.780077 16501 raft_consensus.cc:2272] T d00d4a3dbb1c435cb565cf6e5d723bf2 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:12.782389 16501 tablet_replica.cc:333] T 6a5720f014ed48769eb641989edef3e8 P 64fd1d8a3cd14e66ac94c4fbcceaf039: stopping tablet replica
I20260709 12:28:12.782907 16501 raft_consensus.cc:2243] T 6a5720f014ed48769eb641989edef3e8 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:28:12.783849 16501 raft_consensus.cc:2272] T 6a5720f014ed48769eb641989edef3e8 P 64fd1d8a3cd14e66ac94c4fbcceaf039 [term 2 FOLLOWER]: Raft consensus is shut down!
W20260709 12:28:12.788930 31947 consensus_peers.cc:597] T 89784f886fb545d2871174c1e169fb32 P 07b97783285d4b22b6fd0f1d816ef930 -> Peer 64fd1d8a3cd14e66ac94c4fbcceaf039 (127.16.29.66:42163): Couldn't send request to peer 64fd1d8a3cd14e66ac94c4fbcceaf039. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:28:12.830935 16501 tablet_server.cc:196] TabletServer@127.16.29.66:0 shutdown complete.
W20260709 12:28:12.866264 31949 consensus_peers.cc:597] T 9fe7f1d426c94232a1ec785665ef1416 P 07b97783285d4b22b6fd0f1d816ef930 -> Peer 2ae3a9e2293442e291b1262a717a5e50 (127.16.29.65:44289): Couldn't send request to peer 2ae3a9e2293442e291b1262a717a5e50. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:44289: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:28:12.879717 16501 tablet_server.cc:179] TabletServer@127.16.29.67:0 shutting down...
W20260709 12:28:12.907860 31947 consensus_peers.cc:597] T 378f56c32c1e46f99a874bd1adcbcf25 P 07b97783285d4b22b6fd0f1d816ef930 -> Peer 64fd1d8a3cd14e66ac94c4fbcceaf039 (127.16.29.66:42163): Couldn't send request to peer 64fd1d8a3cd14e66ac94c4fbcceaf039. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.66:42163: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:28:12.909914 31949 consensus_peers.cc:597] T b659027528154c4c86becb863718c732 P 07b97783285d4b22b6fd0f1d816ef930 -> Peer 2ae3a9e2293442e291b1262a717a5e50 (127.16.29.65:44289): Couldn't send request to peer 2ae3a9e2293442e291b1262a717a5e50. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:44289: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:28:12.916731 16501 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:28:12.917419 16501 tablet_replica.cc:333] T ff138d12849e4f63a9b257f4395179de P 07b97783285d4b22b6fd0f1d816ef930: stopping tablet replica
I20260709 12:28:12.918277 16501 raft_consensus.cc:2243] T ff138d12849e4f63a9b257f4395179de P 07b97783285d4b22b6fd0f1d816ef930 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:12.918814 16501 raft_consensus.cc:2272] T ff138d12849e4f63a9b257f4395179de P 07b97783285d4b22b6fd0f1d816ef930 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:12.921180 16501 tablet_replica.cc:333] T 0ca01749cbc64fd987e4174047baac00 P 07b97783285d4b22b6fd0f1d816ef930: stopping tablet replica
I20260709 12:28:12.921700 16501 raft_consensus.cc:2243] T 0ca01749cbc64fd987e4174047baac00 P 07b97783285d4b22b6fd0f1d816ef930 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:12.922161 16501 raft_consensus.cc:2272] T 0ca01749cbc64fd987e4174047baac00 P 07b97783285d4b22b6fd0f1d816ef930 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:12.924341 16501 tablet_replica.cc:333] T c11f552ceb904522856ef259018a8442 P 07b97783285d4b22b6fd0f1d816ef930: stopping tablet replica
I20260709 12:28:12.924901 16501 raft_consensus.cc:2243] T c11f552ceb904522856ef259018a8442 P 07b97783285d4b22b6fd0f1d816ef930 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:12.925365 16501 raft_consensus.cc:2272] T c11f552ceb904522856ef259018a8442 P 07b97783285d4b22b6fd0f1d816ef930 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:12.927546 16501 tablet_replica.cc:333] T b659027528154c4c86becb863718c732 P 07b97783285d4b22b6fd0f1d816ef930: stopping tablet replica
I20260709 12:28:12.928135 16501 raft_consensus.cc:2243] T b659027528154c4c86becb863718c732 P 07b97783285d4b22b6fd0f1d816ef930 [term 3 LEADER]: Raft consensus shutting down.
I20260709 12:28:12.928926 16501 raft_consensus.cc:2272] T b659027528154c4c86becb863718c732 P 07b97783285d4b22b6fd0f1d816ef930 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:12.931118 16501 tablet_replica.cc:333] T 8eda1839ee054ee59ec8629c9139dd2e P 07b97783285d4b22b6fd0f1d816ef930: stopping tablet replica
I20260709 12:28:12.931672 16501 raft_consensus.cc:2243] T 8eda1839ee054ee59ec8629c9139dd2e P 07b97783285d4b22b6fd0f1d816ef930 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:12.932216 16501 raft_consensus.cc:2272] T 8eda1839ee054ee59ec8629c9139dd2e P 07b97783285d4b22b6fd0f1d816ef930 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:12.934513 16501 tablet_replica.cc:333] T fad0b7ef0f804aa494f44e9ec456e11e P 07b97783285d4b22b6fd0f1d816ef930: stopping tablet replica
I20260709 12:28:12.935087 16501 raft_consensus.cc:2243] T fad0b7ef0f804aa494f44e9ec456e11e P 07b97783285d4b22b6fd0f1d816ef930 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:12.935542 16501 raft_consensus.cc:2272] T fad0b7ef0f804aa494f44e9ec456e11e P 07b97783285d4b22b6fd0f1d816ef930 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:12.937912 16501 tablet_replica.cc:333] T 0d5a572ce9954c72b6d74a24da7de624 P 07b97783285d4b22b6fd0f1d816ef930: stopping tablet replica
I20260709 12:28:12.938506 16501 raft_consensus.cc:2243] T 0d5a572ce9954c72b6d74a24da7de624 P 07b97783285d4b22b6fd0f1d816ef930 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:12.939086 16501 raft_consensus.cc:2272] T 0d5a572ce9954c72b6d74a24da7de624 P 07b97783285d4b22b6fd0f1d816ef930 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:12.941535 16501 tablet_replica.cc:333] T 6a5720f014ed48769eb641989edef3e8 P 07b97783285d4b22b6fd0f1d816ef930: stopping tablet replica
I20260709 12:28:12.942107 16501 raft_consensus.cc:2243] T 6a5720f014ed48769eb641989edef3e8 P 07b97783285d4b22b6fd0f1d816ef930 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:12.942574 16501 raft_consensus.cc:2272] T 6a5720f014ed48769eb641989edef3e8 P 07b97783285d4b22b6fd0f1d816ef930 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:12.944687 16501 tablet_replica.cc:333] T 9fe7f1d426c94232a1ec785665ef1416 P 07b97783285d4b22b6fd0f1d816ef930: stopping tablet replica
I20260709 12:28:12.945210 16501 raft_consensus.cc:2243] T 9fe7f1d426c94232a1ec785665ef1416 P 07b97783285d4b22b6fd0f1d816ef930 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:28:12.946048 16501 raft_consensus.cc:2272] T 9fe7f1d426c94232a1ec785665ef1416 P 07b97783285d4b22b6fd0f1d816ef930 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:12.948238 16501 tablet_replica.cc:333] T 1638e215a9ee4068a999e92023fff415 P 07b97783285d4b22b6fd0f1d816ef930: stopping tablet replica
I20260709 12:28:12.948766 16501 raft_consensus.cc:2243] T 1638e215a9ee4068a999e92023fff415 P 07b97783285d4b22b6fd0f1d816ef930 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:12.949239 16501 raft_consensus.cc:2272] T 1638e215a9ee4068a999e92023fff415 P 07b97783285d4b22b6fd0f1d816ef930 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:12.951303 16501 tablet_replica.cc:333] T 378f56c32c1e46f99a874bd1adcbcf25 P 07b97783285d4b22b6fd0f1d816ef930: stopping tablet replica
I20260709 12:28:12.951807 16501 raft_consensus.cc:2243] T 378f56c32c1e46f99a874bd1adcbcf25 P 07b97783285d4b22b6fd0f1d816ef930 [term 3 LEADER]: Raft consensus shutting down.
I20260709 12:28:12.952653 16501 raft_consensus.cc:2272] T 378f56c32c1e46f99a874bd1adcbcf25 P 07b97783285d4b22b6fd0f1d816ef930 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:12.954891 16501 tablet_replica.cc:333] T c4f237debd4942b198e077f8e7091088 P 07b97783285d4b22b6fd0f1d816ef930: stopping tablet replica
I20260709 12:28:12.955396 16501 raft_consensus.cc:2243] T c4f237debd4942b198e077f8e7091088 P 07b97783285d4b22b6fd0f1d816ef930 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:12.955864 16501 raft_consensus.cc:2272] T c4f237debd4942b198e077f8e7091088 P 07b97783285d4b22b6fd0f1d816ef930 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:12.957888 16501 tablet_replica.cc:333] T c61f10da93eb4af5a80a204e679f1ee3 P 07b97783285d4b22b6fd0f1d816ef930: stopping tablet replica
I20260709 12:28:12.958369 16501 raft_consensus.cc:2243] T c61f10da93eb4af5a80a204e679f1ee3 P 07b97783285d4b22b6fd0f1d816ef930 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:12.958729 16501 raft_consensus.cc:2272] T c61f10da93eb4af5a80a204e679f1ee3 P 07b97783285d4b22b6fd0f1d816ef930 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:12.960810 16501 tablet_replica.cc:333] T 0fb83fa22e13487f9d56facc914278ba P 07b97783285d4b22b6fd0f1d816ef930: stopping tablet replica
I20260709 12:28:12.961282 16501 raft_consensus.cc:2243] T 0fb83fa22e13487f9d56facc914278ba P 07b97783285d4b22b6fd0f1d816ef930 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:28:12.962049 16501 raft_consensus.cc:2272] T 0fb83fa22e13487f9d56facc914278ba P 07b97783285d4b22b6fd0f1d816ef930 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:12.964028 16501 tablet_replica.cc:333] T 89784f886fb545d2871174c1e169fb32 P 07b97783285d4b22b6fd0f1d816ef930: stopping tablet replica
I20260709 12:28:12.964468 16501 raft_consensus.cc:2243] T 89784f886fb545d2871174c1e169fb32 P 07b97783285d4b22b6fd0f1d816ef930 [term 3 LEADER]: Raft consensus shutting down.
I20260709 12:28:12.965171 16501 raft_consensus.cc:2272] T 89784f886fb545d2871174c1e169fb32 P 07b97783285d4b22b6fd0f1d816ef930 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:12.967152 16501 tablet_replica.cc:333] T 52322cf19f734f19a4ccfc8389f32b41 P 07b97783285d4b22b6fd0f1d816ef930: stopping tablet replica
I20260709 12:28:12.967569 16501 raft_consensus.cc:2243] T 52322cf19f734f19a4ccfc8389f32b41 P 07b97783285d4b22b6fd0f1d816ef930 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:12.968034 16501 raft_consensus.cc:2272] T 52322cf19f734f19a4ccfc8389f32b41 P 07b97783285d4b22b6fd0f1d816ef930 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:12.970247 16501 tablet_replica.cc:333] T 703b2df19bf24c31a5acbe85178314ec P 07b97783285d4b22b6fd0f1d816ef930: stopping tablet replica
I20260709 12:28:12.970681 16501 raft_consensus.cc:2243] T 703b2df19bf24c31a5acbe85178314ec P 07b97783285d4b22b6fd0f1d816ef930 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:28:12.971462 16501 raft_consensus.cc:2272] T 703b2df19bf24c31a5acbe85178314ec P 07b97783285d4b22b6fd0f1d816ef930 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:12.973668 16501 tablet_replica.cc:333] T d00d4a3dbb1c435cb565cf6e5d723bf2 P 07b97783285d4b22b6fd0f1d816ef930: stopping tablet replica
I20260709 12:28:12.974083 16501 raft_consensus.cc:2243] T d00d4a3dbb1c435cb565cf6e5d723bf2 P 07b97783285d4b22b6fd0f1d816ef930 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:12.974484 16501 raft_consensus.cc:2272] T d00d4a3dbb1c435cb565cf6e5d723bf2 P 07b97783285d4b22b6fd0f1d816ef930 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:13.018728 16501 tablet_server.cc:196] TabletServer@127.16.29.67:0 shutdown complete.
I20260709 12:28:13.056738 16501 master.cc:562] Master@127.16.29.126:37739 shutting down...
I20260709 12:28:13.081753 16501 raft_consensus.cc:2243] T 00000000000000000000000000000000 P c569449549d1476cb2d243fd3c5e4804 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:28:13.082501 16501 raft_consensus.cc:2272] T 00000000000000000000000000000000 P c569449549d1476cb2d243fd3c5e4804 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:13.082854 16501 tablet_replica.cc:333] T 00000000000000000000000000000000 P c569449549d1476cb2d243fd3c5e4804: stopping tablet replica
I20260709 12:28:13.104893 16501 master.cc:584] Master@127.16.29.126:37739 shutdown complete.
[       OK ] FilterSoftDeletedTableTest.TestFilterSofteDeletedTable/0 (21330 ms)
[ RUN      ] FilterSoftDeletedTableTest.TestFilterSofteDeletedTable/1
I20260709 12:28:13.168341 16501 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.16.29.126:45119
I20260709 12:28:13.169760 16501 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:28:13.176601 32327 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:28:13.176608 32326 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:28:13.176894 32329 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:28:13.178037 16501 server_base.cc:1061] running on GCE node
I20260709 12:28:13.179064 16501 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:28:13.179260 16501 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:28:13.179386 16501 hybrid_clock.cc:648] HybridClock initialized: now 1783600093179375 us; error 0 us; skew 500 ppm
I20260709 12:28:13.180616 16501 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:28:13.183276 16501 webserver.cc:533] Webserver started at http://127.16.29.126:35475/ using document root <none> and password file <none>
I20260709 12:28:13.183779 16501 fs_manager.cc:362] Metadata directory not provided
I20260709 12:28:13.184026 16501 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:28:13.184276 16501 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:28:13.185480 16501 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783599551912677-16501-0/minicluster-data/master-0-root/instance:
uuid: "66f4042e77394c5a8e850bb63a8a7ac5"
format_stamp: "Formatted at 2026-07-09 12:28:13 on dist-test-slave-5n9v"
I20260709 12:28:13.190201 16501 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.004s	sys 0.002s
I20260709 12:28:13.194116 32334 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:28:13.194957 16501 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.000s
I20260709 12:28:13.195241 16501 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783599551912677-16501-0/minicluster-data/master-0-root
uuid: "66f4042e77394c5a8e850bb63a8a7ac5"
format_stamp: "Formatted at 2026-07-09 12:28:13 on dist-test-slave-5n9v"
I20260709 12:28:13.195494 16501 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783599551912677-16501-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783599551912677-16501-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783599551912677-16501-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:28:13.221880 16501 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:28:13.223068 16501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:28:13.263257 16501 rpc_server.cc:307] RPC server started. Bound to: 127.16.29.126:45119
I20260709 12:28:13.263358 32385 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.29.126:45119 every 8 connection(s)
I20260709 12:28:13.267304 32386 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:13.278980 32386 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66f4042e77394c5a8e850bb63a8a7ac5: Bootstrap starting.
I20260709 12:28:13.283833 32386 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 66f4042e77394c5a8e850bb63a8a7ac5: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:13.288126 32386 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66f4042e77394c5a8e850bb63a8a7ac5: No bootstrap required, opened a new log
I20260709 12:28:13.290371 32386 raft_consensus.cc:359] T 00000000000000000000000000000000 P 66f4042e77394c5a8e850bb63a8a7ac5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66f4042e77394c5a8e850bb63a8a7ac5" member_type: VOTER }
I20260709 12:28:13.290797 32386 raft_consensus.cc:385] T 00000000000000000000000000000000 P 66f4042e77394c5a8e850bb63a8a7ac5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:13.290990 32386 raft_consensus.cc:740] T 00000000000000000000000000000000 P 66f4042e77394c5a8e850bb63a8a7ac5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66f4042e77394c5a8e850bb63a8a7ac5, State: Initialized, Role: FOLLOWER
I20260709 12:28:13.291486 32386 consensus_queue.cc:260] T 00000000000000000000000000000000 P 66f4042e77394c5a8e850bb63a8a7ac5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66f4042e77394c5a8e850bb63a8a7ac5" member_type: VOTER }
I20260709 12:28:13.291940 32386 raft_consensus.cc:399] T 00000000000000000000000000000000 P 66f4042e77394c5a8e850bb63a8a7ac5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260709 12:28:13.292193 32386 raft_consensus.cc:493] T 00000000000000000000000000000000 P 66f4042e77394c5a8e850bb63a8a7ac5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260709 12:28:13.292414 32386 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 66f4042e77394c5a8e850bb63a8a7ac5 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:13.297238 32386 raft_consensus.cc:515] T 00000000000000000000000000000000 P 66f4042e77394c5a8e850bb63a8a7ac5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66f4042e77394c5a8e850bb63a8a7ac5" member_type: VOTER }
I20260709 12:28:13.297842 32386 leader_election.cc:304] T 00000000000000000000000000000000 P 66f4042e77394c5a8e850bb63a8a7ac5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 66f4042e77394c5a8e850bb63a8a7ac5; no voters: 
I20260709 12:28:13.299512 32386 leader_election.cc:290] T 00000000000000000000000000000000 P 66f4042e77394c5a8e850bb63a8a7ac5 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20260709 12:28:13.300072 32389 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 66f4042e77394c5a8e850bb63a8a7ac5 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:28:13.301925 32389 raft_consensus.cc:697] T 00000000000000000000000000000000 P 66f4042e77394c5a8e850bb63a8a7ac5 [term 1 LEADER]: Becoming Leader. State: Replica: 66f4042e77394c5a8e850bb63a8a7ac5, State: Running, Role: LEADER
I20260709 12:28:13.302701 32389 consensus_queue.cc:237] T 00000000000000000000000000000000 P 66f4042e77394c5a8e850bb63a8a7ac5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66f4042e77394c5a8e850bb63a8a7ac5" member_type: VOTER }
I20260709 12:28:13.303246 32386 sys_catalog.cc:565] T 00000000000000000000000000000000 P 66f4042e77394c5a8e850bb63a8a7ac5 [sys.catalog]: configured and running, proceeding with master startup.
I20260709 12:28:13.307565 32390 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66f4042e77394c5a8e850bb63a8a7ac5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "66f4042e77394c5a8e850bb63a8a7ac5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66f4042e77394c5a8e850bb63a8a7ac5" member_type: VOTER } }
I20260709 12:28:13.308337 32390 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66f4042e77394c5a8e850bb63a8a7ac5 [sys.catalog]: This master's current role is: LEADER
I20260709 12:28:13.309628 32391 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66f4042e77394c5a8e850bb63a8a7ac5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 66f4042e77394c5a8e850bb63a8a7ac5. Latest consensus state: current_term: 1 leader_uuid: "66f4042e77394c5a8e850bb63a8a7ac5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66f4042e77394c5a8e850bb63a8a7ac5" member_type: VOTER } }
I20260709 12:28:13.310240 32391 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66f4042e77394c5a8e850bb63a8a7ac5 [sys.catalog]: This master's current role is: LEADER
I20260709 12:28:13.311035 32397 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260709 12:28:13.316195 32397 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260709 12:28:13.319427 16501 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260709 12:28:13.325846 32397 catalog_manager.cc:1383] Generated new cluster ID: 7c04547500f74822965c5104ce610a0c
I20260709 12:28:13.326196 32397 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260709 12:28:13.361092 32397 catalog_manager.cc:1406] Generated new certificate authority record
I20260709 12:28:13.362591 32397 catalog_manager.cc:1540] Loading token signing keys...
I20260709 12:28:13.378556 32397 catalog_manager.cc:6070] T 00000000000000000000000000000000 P 66f4042e77394c5a8e850bb63a8a7ac5: Generated new TSK 0
I20260709 12:28:13.379253 32397 catalog_manager.cc:1550] Initializing in-progress tserver states...
I20260709 12:28:13.386610 16501 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:28:13.392869 32407 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:28:13.393764 32408 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:28:13.395308 32410 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:28:13.395917 16501 server_base.cc:1061] running on GCE node
I20260709 12:28:13.396811 16501 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:28:13.396991 16501 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:28:13.397185 16501 hybrid_clock.cc:648] HybridClock initialized: now 1783600093397169 us; error 0 us; skew 500 ppm
I20260709 12:28:13.397809 16501 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:28:13.400327 16501 webserver.cc:533] Webserver started at http://127.16.29.65:40849/ using document root <none> and password file <none>
I20260709 12:28:13.400841 16501 fs_manager.cc:362] Metadata directory not provided
I20260709 12:28:13.401026 16501 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:28:13.401304 16501 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:28:13.402421 16501 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783599551912677-16501-0/minicluster-data/ts-0-root/instance:
uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d"
format_stamp: "Formatted at 2026-07-09 12:28:13 on dist-test-slave-5n9v"
I20260709 12:28:13.406916 16501 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.003s	sys 0.003s
I20260709 12:28:13.410358 32415 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:28:13.411043 16501 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.000s
I20260709 12:28:13.411360 16501 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783599551912677-16501-0/minicluster-data/ts-0-root
uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d"
format_stamp: "Formatted at 2026-07-09 12:28:13 on dist-test-slave-5n9v"
I20260709 12:28:13.411641 16501 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783599551912677-16501-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783599551912677-16501-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783599551912677-16501-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:28:13.434340 16501 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:28:13.435542 16501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:28:13.437027 16501 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:28:13.439307 16501 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:28:13.439524 16501 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:28:13.439762 16501 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:28:13.439970 16501 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.001s
I20260709 12:28:13.479991 16501 rpc_server.cc:307] RPC server started. Bound to: 127.16.29.65:41229
I20260709 12:28:13.480188 32477 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.29.65:41229 every 8 connection(s)
I20260709 12:28:13.485085 16501 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:28:13.492694 32482 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:28:13.495421 32483 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:28:13.498689 16501 server_base.cc:1061] running on GCE node
I20260709 12:28:13.499222 32478 heartbeater.cc:344] Connected to a master server at 127.16.29.126:45119
I20260709 12:28:13.499778 32478 heartbeater.cc:461] Registering TS with master...
W20260709 12:28:13.499738 32485 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:28:13.500840 32478 heartbeater.cc:507] Master 127.16.29.126:45119 requested a full tablet report, sending...
I20260709 12:28:13.500941 16501 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:28:13.501302 16501 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:28:13.501480 16501 hybrid_clock.cc:648] HybridClock initialized: now 1783600093501460 us; error 0 us; skew 500 ppm
I20260709 12:28:13.502247 16501 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:28:13.502974 32351 ts_manager.cc:194] Registered new tserver with Master: cc8d9fdc73b14a3cb716b47c24bafc1d (127.16.29.65:41229)
I20260709 12:28:13.504879 16501 webserver.cc:533] Webserver started at http://127.16.29.66:40787/ using document root <none> and password file <none>
I20260709 12:28:13.505309 32351 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:34546
I20260709 12:28:13.505450 16501 fs_manager.cc:362] Metadata directory not provided
I20260709 12:28:13.505777 16501 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:28:13.506069 16501 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:28:13.507362 16501 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783599551912677-16501-0/minicluster-data/ts-1-root/instance:
uuid: "a8adcce962244bc688dc4ecc1f69612c"
format_stamp: "Formatted at 2026-07-09 12:28:13 on dist-test-slave-5n9v"
I20260709 12:28:13.511971 16501 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.003s	sys 0.000s
I20260709 12:28:13.515667 32490 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:28:13.516443 16501 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.003s	sys 0.000s
I20260709 12:28:13.516747 16501 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783599551912677-16501-0/minicluster-data/ts-1-root
uuid: "a8adcce962244bc688dc4ecc1f69612c"
format_stamp: "Formatted at 2026-07-09 12:28:13 on dist-test-slave-5n9v"
I20260709 12:28:13.517092 16501 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783599551912677-16501-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783599551912677-16501-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783599551912677-16501-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:28:13.532541 16501 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:28:13.533955 16501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:28:13.535488 16501 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:28:13.537709 16501 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:28:13.537920 16501 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:28:13.538170 16501 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:28:13.538339 16501 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:28:13.577584 16501 rpc_server.cc:307] RPC server started. Bound to: 127.16.29.66:33369
I20260709 12:28:13.577703 32552 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.29.66:33369 every 8 connection(s)
I20260709 12:28:13.582527 16501 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:28:13.590157 32556 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:28:13.592324 32557 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:28:13.593537 16501 server_base.cc:1061] running on GCE node
W20260709 12:28:13.595464 32559 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:28:13.596036 32553 heartbeater.cc:344] Connected to a master server at 127.16.29.126:45119
I20260709 12:28:13.596443 32553 heartbeater.cc:461] Registering TS with master...
I20260709 12:28:13.596547 16501 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:28:13.596868 16501 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:28:13.597085 16501 hybrid_clock.cc:648] HybridClock initialized: now 1783600093597070 us; error 0 us; skew 500 ppm
I20260709 12:28:13.597406 32553 heartbeater.cc:507] Master 127.16.29.126:45119 requested a full tablet report, sending...
I20260709 12:28:13.597800 16501 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:28:13.599707 32351 ts_manager.cc:194] Registered new tserver with Master: a8adcce962244bc688dc4ecc1f69612c (127.16.29.66:33369)
I20260709 12:28:13.600777 16501 webserver.cc:533] Webserver started at http://127.16.29.67:40627/ using document root <none> and password file <none>
I20260709 12:28:13.601181 32351 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:34554
I20260709 12:28:13.601337 16501 fs_manager.cc:362] Metadata directory not provided
I20260709 12:28:13.601671 16501 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:28:13.601954 16501 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:28:13.603219 16501 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783599551912677-16501-0/minicluster-data/ts-2-root/instance:
uuid: "f8257d37af0941c2a978546ef2276b30"
format_stamp: "Formatted at 2026-07-09 12:28:13 on dist-test-slave-5n9v"
I20260709 12:28:13.607988 16501 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.007s	sys 0.000s
I20260709 12:28:13.611238 32564 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:28:13.612207 16501 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.003s	sys 0.000s
I20260709 12:28:13.612517 16501 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783599551912677-16501-0/minicluster-data/ts-2-root
uuid: "f8257d37af0941c2a978546ef2276b30"
format_stamp: "Formatted at 2026-07-09 12:28:13 on dist-test-slave-5n9v"
I20260709 12:28:13.612826 16501 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783599551912677-16501-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783599551912677-16501-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskx_YJnz/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783599551912677-16501-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:28:13.646314 16501 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:28:13.647819 16501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:28:13.649636 16501 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:28:13.652387 16501 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:28:13.652633 16501 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:28:13.652876 16501 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:28:13.653059 16501 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:28:13.691515 16501 rpc_server.cc:307] RPC server started. Bound to: 127.16.29.67:37155
I20260709 12:28:13.691627 32626 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.29.67:37155 every 8 connection(s)
I20260709 12:28:13.705865 32627 heartbeater.cc:344] Connected to a master server at 127.16.29.126:45119
I20260709 12:28:13.706310 32627 heartbeater.cc:461] Registering TS with master...
I20260709 12:28:13.707079 32627 heartbeater.cc:507] Master 127.16.29.126:45119 requested a full tablet report, sending...
I20260709 12:28:13.709110 32351 ts_manager.cc:194] Registered new tserver with Master: f8257d37af0941c2a978546ef2276b30 (127.16.29.67:37155)
I20260709 12:28:13.709321 16501 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.014720572s
I20260709 12:28:13.710606 32351 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:34568
I20260709 12:28:13.734711 32351 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34572:
name: "test-workload"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
  rows: "\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q""\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q"
  indirect_data: """"
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W20260709 12:28:13.737515 32351 catalog_manager.cc:7077] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260709 12:28:13.810278 32592 tablet_service.cc:1511] Processing CreateTablet for tablet 50fe031340a94b6590d29d44cd57ca55 (DEFAULT_TABLE table=test-workload [id=5ab6bea9d0404058bfa6172ef03109fe]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260709 12:28:13.811153 32591 tablet_service.cc:1511] Processing CreateTablet for tablet f5cb9ea668324bfb883f09876ddd56f7 (DEFAULT_TABLE table=test-workload [id=5ab6bea9d0404058bfa6172ef03109fe]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260709 12:28:13.811672 32592 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 50fe031340a94b6590d29d44cd57ca55. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:13.812606 32591 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f5cb9ea668324bfb883f09876ddd56f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:13.821115 32590 tablet_service.cc:1511] Processing CreateTablet for tablet aa861e9a505d425792eb54a2a29cab97 (DEFAULT_TABLE table=test-workload [id=5ab6bea9d0404058bfa6172ef03109fe]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260709 12:28:13.822623 32590 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa861e9a505d425792eb54a2a29cab97. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:13.828267 32443 tablet_service.cc:1511] Processing CreateTablet for tablet 50fe031340a94b6590d29d44cd57ca55 (DEFAULT_TABLE table=test-workload [id=5ab6bea9d0404058bfa6172ef03109fe]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260709 12:28:13.829731 32443 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 50fe031340a94b6590d29d44cd57ca55. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:13.832252 32589 tablet_service.cc:1511] Processing CreateTablet for tablet f22ce6f243334c2281f0543e605f614a (DEFAULT_TABLE table=test-workload [id=5ab6bea9d0404058bfa6172ef03109fe]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260709 12:28:13.833722 32589 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f22ce6f243334c2281f0543e605f614a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:13.833194 32442 tablet_service.cc:1511] Processing CreateTablet for tablet f5cb9ea668324bfb883f09876ddd56f7 (DEFAULT_TABLE table=test-workload [id=5ab6bea9d0404058bfa6172ef03109fe]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260709 12:28:13.835011 32442 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f5cb9ea668324bfb883f09876ddd56f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:13.837563 32642 tablet_bootstrap.cc:492] T f5cb9ea668324bfb883f09876ddd56f7 P f8257d37af0941c2a978546ef2276b30: Bootstrap starting.
I20260709 12:28:13.841625 32441 tablet_service.cc:1511] Processing CreateTablet for tablet aa861e9a505d425792eb54a2a29cab97 (DEFAULT_TABLE table=test-workload [id=5ab6bea9d0404058bfa6172ef03109fe]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260709 12:28:13.843132 32441 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa861e9a505d425792eb54a2a29cab97. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:13.845064 32642 tablet_bootstrap.cc:654] T f5cb9ea668324bfb883f09876ddd56f7 P f8257d37af0941c2a978546ef2276b30: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:13.845661 32518 tablet_service.cc:1511] Processing CreateTablet for tablet 50fe031340a94b6590d29d44cd57ca55 (DEFAULT_TABLE table=test-workload [id=5ab6bea9d0404058bfa6172ef03109fe]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260709 12:28:13.847040 32518 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 50fe031340a94b6590d29d44cd57ca55. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:13.850214 32440 tablet_service.cc:1511] Processing CreateTablet for tablet f22ce6f243334c2281f0543e605f614a (DEFAULT_TABLE table=test-workload [id=5ab6bea9d0404058bfa6172ef03109fe]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260709 12:28:13.852093 32440 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f22ce6f243334c2281f0543e605f614a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:13.862041 32439 tablet_service.cc:1511] Processing CreateTablet for tablet 8e7d5cee9cf64a3e87372bde6431883e (DEFAULT_TABLE table=test-workload [id=5ab6bea9d0404058bfa6172ef03109fe]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260709 12:28:13.864507 32439 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e7d5cee9cf64a3e87372bde6431883e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:13.865722 32517 tablet_service.cc:1511] Processing CreateTablet for tablet f5cb9ea668324bfb883f09876ddd56f7 (DEFAULT_TABLE table=test-workload [id=5ab6bea9d0404058bfa6172ef03109fe]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260709 12:28:13.867128 32517 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f5cb9ea668324bfb883f09876ddd56f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:13.868875 32590 tablet_service.cc:1511] Processing CreateTablet for tablet 8e7d5cee9cf64a3e87372bde6431883e (DEFAULT_TABLE table=test-workload [id=5ab6bea9d0404058bfa6172ef03109fe]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260709 12:28:13.870241 32590 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e7d5cee9cf64a3e87372bde6431883e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:13.873786 32438 tablet_service.cc:1511] Processing CreateTablet for tablet 9396e8d0ed7b44f58bbc5b421f492f50 (DEFAULT_TABLE table=test-workload [id=5ab6bea9d0404058bfa6172ef03109fe]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260709 12:28:13.874708 32516 tablet_service.cc:1511] Processing CreateTablet for tablet aa861e9a505d425792eb54a2a29cab97 (DEFAULT_TABLE table=test-workload [id=5ab6bea9d0404058bfa6172ef03109fe]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260709 12:28:13.875293 32438 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9396e8d0ed7b44f58bbc5b421f492f50. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:13.876107 32516 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa861e9a505d425792eb54a2a29cab97. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:13.877159 32437 tablet_service.cc:1511] Processing CreateTablet for tablet 4b2251c329154ac49d6147b1cfd6981b (DEFAULT_TABLE table=test-workload [id=5ab6bea9d0404058bfa6172ef03109fe]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260709 12:28:13.878499 32437 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b2251c329154ac49d6147b1cfd6981b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:13.881001 32591 tablet_service.cc:1511] Processing CreateTablet for tablet 9396e8d0ed7b44f58bbc5b421f492f50 (DEFAULT_TABLE table=test-workload [id=5ab6bea9d0404058bfa6172ef03109fe]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260709 12:28:13.882544 32591 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9396e8d0ed7b44f58bbc5b421f492f50. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:13.882159 32514 tablet_service.cc:1511] Processing CreateTablet for tablet 8e7d5cee9cf64a3e87372bde6431883e (DEFAULT_TABLE table=test-workload [id=5ab6bea9d0404058bfa6172ef03109fe]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260709 12:28:13.890931 32587 tablet_service.cc:1511] Processing CreateTablet for tablet d768701c29784c52a2ffb8303021347d (DEFAULT_TABLE table=test-workload [id=5ab6bea9d0404058bfa6172ef03109fe]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260709 12:28:13.895977 32513 tablet_service.cc:1511] Processing CreateTablet for tablet 9396e8d0ed7b44f58bbc5b421f492f50 (DEFAULT_TABLE table=test-workload [id=5ab6bea9d0404058bfa6172ef03109fe]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260709 12:28:13.898083 32435 tablet_service.cc:1511] Processing CreateTablet for tablet c127bfae907d4960956f04576bf0c6c7 (DEFAULT_TABLE table=test-workload [id=5ab6bea9d0404058bfa6172ef03109fe]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260709 12:28:13.899447 32435 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c127bfae907d4960956f04576bf0c6c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:13.884963 32586 tablet_service.cc:1511] Processing CreateTablet for tablet c127bfae907d4960956f04576bf0c6c7 (DEFAULT_TABLE table=test-workload [id=5ab6bea9d0404058bfa6172ef03109fe]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260709 12:28:13.885001 32588 tablet_service.cc:1511] Processing CreateTablet for tablet 4b2251c329154ac49d6147b1cfd6981b (DEFAULT_TABLE table=test-workload [id=5ab6bea9d0404058bfa6172ef03109fe]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260709 12:28:13.904727 32513 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9396e8d0ed7b44f58bbc5b421f492f50. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:13.906419 32514 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e7d5cee9cf64a3e87372bde6431883e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:13.881040 32515 tablet_service.cc:1511] Processing CreateTablet for tablet f22ce6f243334c2281f0543e605f614a (DEFAULT_TABLE table=test-workload [id=5ab6bea9d0404058bfa6172ef03109fe]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260709 12:28:13.909484 32515 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f22ce6f243334c2281f0543e605f614a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:13.921461 32586 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c127bfae907d4960956f04576bf0c6c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:13.885200 32436 tablet_service.cc:1511] Processing CreateTablet for tablet d768701c29784c52a2ffb8303021347d (DEFAULT_TABLE table=test-workload [id=5ab6bea9d0404058bfa6172ef03109fe]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260709 12:28:13.925486 32436 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d768701c29784c52a2ffb8303021347d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:13.927299 32588 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b2251c329154ac49d6147b1cfd6981b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:13.928921 32587 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d768701c29784c52a2ffb8303021347d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:13.932219 32512 tablet_service.cc:1511] Processing CreateTablet for tablet 4b2251c329154ac49d6147b1cfd6981b (DEFAULT_TABLE table=test-workload [id=5ab6bea9d0404058bfa6172ef03109fe]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260709 12:28:13.934155 32512 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b2251c329154ac49d6147b1cfd6981b. 1 dirs total, 0 dirs full, 0 dirs failed
W20260709 12:28:13.948045 32337 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.16.29.67:37155, user_credentials={real_user=slave}} blocked reactor thread for 58718.2us
I20260709 12:28:13.967558 32511 tablet_service.cc:1511] Processing CreateTablet for tablet d768701c29784c52a2ffb8303021347d (DEFAULT_TABLE table=test-workload [id=5ab6bea9d0404058bfa6172ef03109fe]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260709 12:28:13.970099 32510 tablet_service.cc:1511] Processing CreateTablet for tablet c127bfae907d4960956f04576bf0c6c7 (DEFAULT_TABLE table=test-workload [id=5ab6bea9d0404058bfa6172ef03109fe]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260709 12:28:13.971015 32511 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d768701c29784c52a2ffb8303021347d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:13.971686 32510 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c127bfae907d4960956f04576bf0c6c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:14.005810 32642 tablet_bootstrap.cc:492] T f5cb9ea668324bfb883f09876ddd56f7 P f8257d37af0941c2a978546ef2276b30: No bootstrap required, opened a new log
I20260709 12:28:14.006400 32642 ts_tablet_manager.cc:1403] T f5cb9ea668324bfb883f09876ddd56f7 P f8257d37af0941c2a978546ef2276b30: Time spent bootstrapping tablet: real 0.169s	user 0.042s	sys 0.070s
I20260709 12:28:14.028537 32644 tablet_bootstrap.cc:492] T 8e7d5cee9cf64a3e87372bde6431883e P cc8d9fdc73b14a3cb716b47c24bafc1d: Bootstrap starting.
I20260709 12:28:14.036088 32644 tablet_bootstrap.cc:654] T 8e7d5cee9cf64a3e87372bde6431883e P cc8d9fdc73b14a3cb716b47c24bafc1d: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:14.053330 32642 raft_consensus.cc:359] T f5cb9ea668324bfb883f09876ddd56f7 P f8257d37af0941c2a978546ef2276b30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:14.054270 32642 raft_consensus.cc:385] T f5cb9ea668324bfb883f09876ddd56f7 P f8257d37af0941c2a978546ef2276b30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:14.054641 32642 raft_consensus.cc:740] T f5cb9ea668324bfb883f09876ddd56f7 P f8257d37af0941c2a978546ef2276b30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8257d37af0941c2a978546ef2276b30, State: Initialized, Role: FOLLOWER
I20260709 12:28:14.055509 32642 consensus_queue.cc:260] T f5cb9ea668324bfb883f09876ddd56f7 P f8257d37af0941c2a978546ef2276b30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:14.065424 32645 tablet_bootstrap.cc:492] T 50fe031340a94b6590d29d44cd57ca55 P a8adcce962244bc688dc4ecc1f69612c: Bootstrap starting.
I20260709 12:28:14.078140 32645 tablet_bootstrap.cc:654] T 50fe031340a94b6590d29d44cd57ca55 P a8adcce962244bc688dc4ecc1f69612c: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:14.083541 32627 heartbeater.cc:499] Master 127.16.29.126:45119 was elected leader, sending a full tablet report...
I20260709 12:28:14.083635 32642 ts_tablet_manager.cc:1434] T f5cb9ea668324bfb883f09876ddd56f7 P f8257d37af0941c2a978546ef2276b30: Time spent starting tablet: real 0.077s	user 0.012s	sys 0.005s
I20260709 12:28:14.083917 32644 tablet_bootstrap.cc:492] T 8e7d5cee9cf64a3e87372bde6431883e P cc8d9fdc73b14a3cb716b47c24bafc1d: No bootstrap required, opened a new log
I20260709 12:28:14.084621 32644 ts_tablet_manager.cc:1403] T 8e7d5cee9cf64a3e87372bde6431883e P cc8d9fdc73b14a3cb716b47c24bafc1d: Time spent bootstrapping tablet: real 0.056s	user 0.018s	sys 0.027s
I20260709 12:28:14.085112 32642 tablet_bootstrap.cc:492] T aa861e9a505d425792eb54a2a29cab97 P f8257d37af0941c2a978546ef2276b30: Bootstrap starting.
I20260709 12:28:14.087918 32644 raft_consensus.cc:359] T 8e7d5cee9cf64a3e87372bde6431883e P cc8d9fdc73b14a3cb716b47c24bafc1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:14.088701 32644 raft_consensus.cc:385] T 8e7d5cee9cf64a3e87372bde6431883e P cc8d9fdc73b14a3cb716b47c24bafc1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:14.088980 32644 raft_consensus.cc:740] T 8e7d5cee9cf64a3e87372bde6431883e P cc8d9fdc73b14a3cb716b47c24bafc1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc8d9fdc73b14a3cb716b47c24bafc1d, State: Initialized, Role: FOLLOWER
I20260709 12:28:14.089732 32644 consensus_queue.cc:260] T 8e7d5cee9cf64a3e87372bde6431883e P cc8d9fdc73b14a3cb716b47c24bafc1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:14.091568 32642 tablet_bootstrap.cc:654] T aa861e9a505d425792eb54a2a29cab97 P f8257d37af0941c2a978546ef2276b30: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:14.092096 32478 heartbeater.cc:499] Master 127.16.29.126:45119 was elected leader, sending a full tablet report...
I20260709 12:28:14.092614 32644 ts_tablet_manager.cc:1434] T 8e7d5cee9cf64a3e87372bde6431883e P cc8d9fdc73b14a3cb716b47c24bafc1d: Time spent starting tablet: real 0.008s	user 0.007s	sys 0.000s
I20260709 12:28:14.093624 32644 tablet_bootstrap.cc:492] T 50fe031340a94b6590d29d44cd57ca55 P cc8d9fdc73b14a3cb716b47c24bafc1d: Bootstrap starting.
I20260709 12:28:14.100665 32644 tablet_bootstrap.cc:654] T 50fe031340a94b6590d29d44cd57ca55 P cc8d9fdc73b14a3cb716b47c24bafc1d: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:14.105672 32645 tablet_bootstrap.cc:492] T 50fe031340a94b6590d29d44cd57ca55 P a8adcce962244bc688dc4ecc1f69612c: No bootstrap required, opened a new log
I20260709 12:28:14.106325 32645 ts_tablet_manager.cc:1403] T 50fe031340a94b6590d29d44cd57ca55 P a8adcce962244bc688dc4ecc1f69612c: Time spent bootstrapping tablet: real 0.041s	user 0.014s	sys 0.000s
I20260709 12:28:14.108547 32642 tablet_bootstrap.cc:492] T aa861e9a505d425792eb54a2a29cab97 P f8257d37af0941c2a978546ef2276b30: No bootstrap required, opened a new log
I20260709 12:28:14.109222 32642 ts_tablet_manager.cc:1403] T aa861e9a505d425792eb54a2a29cab97 P f8257d37af0941c2a978546ef2276b30: Time spent bootstrapping tablet: real 0.024s	user 0.019s	sys 0.003s
I20260709 12:28:14.109279 32645 raft_consensus.cc:359] T 50fe031340a94b6590d29d44cd57ca55 P a8adcce962244bc688dc4ecc1f69612c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } } peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } }
I20260709 12:28:14.110172 32645 raft_consensus.cc:385] T 50fe031340a94b6590d29d44cd57ca55 P a8adcce962244bc688dc4ecc1f69612c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:14.110484 32645 raft_consensus.cc:740] T 50fe031340a94b6590d29d44cd57ca55 P a8adcce962244bc688dc4ecc1f69612c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8adcce962244bc688dc4ecc1f69612c, State: Initialized, Role: FOLLOWER
I20260709 12:28:14.111451 32645 consensus_queue.cc:260] T 50fe031340a94b6590d29d44cd57ca55 P a8adcce962244bc688dc4ecc1f69612c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } } peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } }
I20260709 12:28:14.105672 32644 tablet_bootstrap.cc:492] T 50fe031340a94b6590d29d44cd57ca55 P cc8d9fdc73b14a3cb716b47c24bafc1d: No bootstrap required, opened a new log
I20260709 12:28:14.113229 32644 ts_tablet_manager.cc:1403] T 50fe031340a94b6590d29d44cd57ca55 P cc8d9fdc73b14a3cb716b47c24bafc1d: Time spent bootstrapping tablet: real 0.020s	user 0.009s	sys 0.002s
I20260709 12:28:14.112978 32642 raft_consensus.cc:359] T aa861e9a505d425792eb54a2a29cab97 P f8257d37af0941c2a978546ef2276b30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:14.114233 32553 heartbeater.cc:499] Master 127.16.29.126:45119 was elected leader, sending a full tablet report...
I20260709 12:28:14.114272 32642 raft_consensus.cc:385] T aa861e9a505d425792eb54a2a29cab97 P f8257d37af0941c2a978546ef2276b30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:14.114866 32645 ts_tablet_manager.cc:1434] T 50fe031340a94b6590d29d44cd57ca55 P a8adcce962244bc688dc4ecc1f69612c: Time spent starting tablet: real 0.008s	user 0.001s	sys 0.006s
I20260709 12:28:14.115056 32642 raft_consensus.cc:740] T aa861e9a505d425792eb54a2a29cab97 P f8257d37af0941c2a978546ef2276b30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8257d37af0941c2a978546ef2276b30, State: Initialized, Role: FOLLOWER
I20260709 12:28:14.115998 32645 tablet_bootstrap.cc:492] T 4b2251c329154ac49d6147b1cfd6981b P a8adcce962244bc688dc4ecc1f69612c: Bootstrap starting.
I20260709 12:28:14.116287 32642 consensus_queue.cc:260] T aa861e9a505d425792eb54a2a29cab97 P f8257d37af0941c2a978546ef2276b30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:14.119344 32642 ts_tablet_manager.cc:1434] T aa861e9a505d425792eb54a2a29cab97 P f8257d37af0941c2a978546ef2276b30: Time spent starting tablet: real 0.010s	user 0.008s	sys 0.000s
I20260709 12:28:14.120584 32642 tablet_bootstrap.cc:492] T f22ce6f243334c2281f0543e605f614a P f8257d37af0941c2a978546ef2276b30: Bootstrap starting.
I20260709 12:28:14.122099 32645 tablet_bootstrap.cc:654] T 4b2251c329154ac49d6147b1cfd6981b P a8adcce962244bc688dc4ecc1f69612c: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:14.126240 32647 raft_consensus.cc:493] T aa861e9a505d425792eb54a2a29cab97 P f8257d37af0941c2a978546ef2276b30 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:14.126780 32647 raft_consensus.cc:515] T aa861e9a505d425792eb54a2a29cab97 P f8257d37af0941c2a978546ef2276b30 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:14.126849 32644 raft_consensus.cc:359] T 50fe031340a94b6590d29d44cd57ca55 P cc8d9fdc73b14a3cb716b47c24bafc1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } } peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } }
I20260709 12:28:14.127692 32644 raft_consensus.cc:385] T 50fe031340a94b6590d29d44cd57ca55 P cc8d9fdc73b14a3cb716b47c24bafc1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:14.128043 32644 raft_consensus.cc:740] T 50fe031340a94b6590d29d44cd57ca55 P cc8d9fdc73b14a3cb716b47c24bafc1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc8d9fdc73b14a3cb716b47c24bafc1d, State: Initialized, Role: FOLLOWER
I20260709 12:28:14.128154 32642 tablet_bootstrap.cc:654] T f22ce6f243334c2281f0543e605f614a P f8257d37af0941c2a978546ef2276b30: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:14.128953 32644 consensus_queue.cc:260] T 50fe031340a94b6590d29d44cd57ca55 P cc8d9fdc73b14a3cb716b47c24bafc1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } } peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } }
I20260709 12:28:14.131217 32644 ts_tablet_manager.cc:1434] T 50fe031340a94b6590d29d44cd57ca55 P cc8d9fdc73b14a3cb716b47c24bafc1d: Time spent starting tablet: real 0.007s	user 0.005s	sys 0.001s
I20260709 12:28:14.132280 32644 tablet_bootstrap.cc:492] T aa861e9a505d425792eb54a2a29cab97 P cc8d9fdc73b14a3cb716b47c24bafc1d: Bootstrap starting.
I20260709 12:28:14.138464 32644 tablet_bootstrap.cc:654] T aa861e9a505d425792eb54a2a29cab97 P cc8d9fdc73b14a3cb716b47c24bafc1d: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:14.139310 32649 raft_consensus.cc:493] T 50fe031340a94b6590d29d44cd57ca55 P cc8d9fdc73b14a3cb716b47c24bafc1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:14.140040 32649 raft_consensus.cc:515] T 50fe031340a94b6590d29d44cd57ca55 P cc8d9fdc73b14a3cb716b47c24bafc1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } } peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } }
I20260709 12:28:14.143425 32649 leader_election.cc:290] T 50fe031340a94b6590d29d44cd57ca55 P cc8d9fdc73b14a3cb716b47c24bafc1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f8257d37af0941c2a978546ef2276b30 (127.16.29.67:37155), a8adcce962244bc688dc4ecc1f69612c (127.16.29.66:33369)
I20260709 12:28:14.147807 32453 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aa861e9a505d425792eb54a2a29cab97" candidate_uuid: "f8257d37af0941c2a978546ef2276b30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" is_pre_election: true
I20260709 12:28:14.149156 32642 tablet_bootstrap.cc:492] T f22ce6f243334c2281f0543e605f614a P f8257d37af0941c2a978546ef2276b30: No bootstrap required, opened a new log
I20260709 12:28:14.149652 32642 ts_tablet_manager.cc:1403] T f22ce6f243334c2281f0543e605f614a P f8257d37af0941c2a978546ef2276b30: Time spent bootstrapping tablet: real 0.029s	user 0.009s	sys 0.017s
I20260709 12:28:14.150890 32647 leader_election.cc:290] T aa861e9a505d425792eb54a2a29cab97 P f8257d37af0941c2a978546ef2276b30 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc8d9fdc73b14a3cb716b47c24bafc1d (127.16.29.65:41229), a8adcce962244bc688dc4ecc1f69612c (127.16.29.66:33369)
I20260709 12:28:14.152788 32642 raft_consensus.cc:359] T f22ce6f243334c2281f0543e605f614a P f8257d37af0941c2a978546ef2276b30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:14.153563 32642 raft_consensus.cc:385] T f22ce6f243334c2281f0543e605f614a P f8257d37af0941c2a978546ef2276b30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:14.153837 32642 raft_consensus.cc:740] T f22ce6f243334c2281f0543e605f614a P f8257d37af0941c2a978546ef2276b30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8257d37af0941c2a978546ef2276b30, State: Initialized, Role: FOLLOWER
I20260709 12:28:14.154719 32642 consensus_queue.cc:260] T f22ce6f243334c2281f0543e605f614a P f8257d37af0941c2a978546ef2276b30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:14.161372 32642 ts_tablet_manager.cc:1434] T f22ce6f243334c2281f0543e605f614a P f8257d37af0941c2a978546ef2276b30: Time spent starting tablet: real 0.011s	user 0.002s	sys 0.006s
I20260709 12:28:14.164084 32642 tablet_bootstrap.cc:492] T d768701c29784c52a2ffb8303021347d P f8257d37af0941c2a978546ef2276b30: Bootstrap starting.
W20260709 12:28:14.168864 32568 leader_election.cc:343] T aa861e9a505d425792eb54a2a29cab97 P f8257d37af0941c2a978546ef2276b30 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cc8d9fdc73b14a3cb716b47c24bafc1d (127.16.29.65:41229): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:28:14.170929 32602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fe031340a94b6590d29d44cd57ca55" candidate_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8257d37af0941c2a978546ef2276b30" is_pre_election: true
W20260709 12:28:14.174347 32418 leader_election.cc:343] T 50fe031340a94b6590d29d44cd57ca55 P cc8d9fdc73b14a3cb716b47c24bafc1d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f8257d37af0941c2a978546ef2276b30 (127.16.29.67:37155): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:28:14.175998 32642 tablet_bootstrap.cc:654] T d768701c29784c52a2ffb8303021347d P f8257d37af0941c2a978546ef2276b30: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:14.177484 32528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fe031340a94b6590d29d44cd57ca55" candidate_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8adcce962244bc688dc4ecc1f69612c" is_pre_election: true
I20260709 12:28:14.178473 32528 raft_consensus.cc:2468] T 50fe031340a94b6590d29d44cd57ca55 P a8adcce962244bc688dc4ecc1f69612c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc8d9fdc73b14a3cb716b47c24bafc1d in term 0.
I20260709 12:28:14.180081 32418 leader_election.cc:304] T 50fe031340a94b6590d29d44cd57ca55 P cc8d9fdc73b14a3cb716b47c24bafc1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a8adcce962244bc688dc4ecc1f69612c, cc8d9fdc73b14a3cb716b47c24bafc1d; no voters: f8257d37af0941c2a978546ef2276b30
I20260709 12:28:14.181156 32649 raft_consensus.cc:2804] T 50fe031340a94b6590d29d44cd57ca55 P cc8d9fdc73b14a3cb716b47c24bafc1d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:28:14.181594 32649 raft_consensus.cc:493] T 50fe031340a94b6590d29d44cd57ca55 P cc8d9fdc73b14a3cb716b47c24bafc1d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:14.181982 32649 raft_consensus.cc:3060] T 50fe031340a94b6590d29d44cd57ca55 P cc8d9fdc73b14a3cb716b47c24bafc1d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:14.192119 32528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aa861e9a505d425792eb54a2a29cab97" candidate_uuid: "f8257d37af0941c2a978546ef2276b30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8adcce962244bc688dc4ecc1f69612c" is_pre_election: true
W20260709 12:28:14.194195 32567 leader_election.cc:343] T aa861e9a505d425792eb54a2a29cab97 P f8257d37af0941c2a978546ef2276b30 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a8adcce962244bc688dc4ecc1f69612c (127.16.29.66:33369): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:28:14.194799 32567 leader_election.cc:304] T aa861e9a505d425792eb54a2a29cab97 P f8257d37af0941c2a978546ef2276b30 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8257d37af0941c2a978546ef2276b30; no voters: a8adcce962244bc688dc4ecc1f69612c, cc8d9fdc73b14a3cb716b47c24bafc1d
I20260709 12:28:14.196388 32647 raft_consensus.cc:2749] T aa861e9a505d425792eb54a2a29cab97 P f8257d37af0941c2a978546ef2276b30 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:28:14.198899 32649 raft_consensus.cc:515] T 50fe031340a94b6590d29d44cd57ca55 P cc8d9fdc73b14a3cb716b47c24bafc1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } } peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } }
W20260709 12:28:14.199862 32628 tablet.cc:2406] T aa861e9a505d425792eb54a2a29cab97 P f8257d37af0941c2a978546ef2276b30: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260709 12:28:14.203049 32602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fe031340a94b6590d29d44cd57ca55" candidate_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8257d37af0941c2a978546ef2276b30"
W20260709 12:28:14.204974 32418 leader_election.cc:343] T 50fe031340a94b6590d29d44cd57ca55 P cc8d9fdc73b14a3cb716b47c24bafc1d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f8257d37af0941c2a978546ef2276b30 (127.16.29.67:37155): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:28:14.207553 32528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fe031340a94b6590d29d44cd57ca55" candidate_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8adcce962244bc688dc4ecc1f69612c"
I20260709 12:28:14.208376 32528 raft_consensus.cc:3060] T 50fe031340a94b6590d29d44cd57ca55 P a8adcce962244bc688dc4ecc1f69612c [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:14.218389 32649 leader_election.cc:290] T 50fe031340a94b6590d29d44cd57ca55 P cc8d9fdc73b14a3cb716b47c24bafc1d [CANDIDATE]: Term 1 election: Requested vote from peers f8257d37af0941c2a978546ef2276b30 (127.16.29.67:37155), a8adcce962244bc688dc4ecc1f69612c (127.16.29.66:33369)
I20260709 12:28:14.219264 32644 tablet_bootstrap.cc:492] T aa861e9a505d425792eb54a2a29cab97 P cc8d9fdc73b14a3cb716b47c24bafc1d: No bootstrap required, opened a new log
I20260709 12:28:14.219815 32644 ts_tablet_manager.cc:1403] T aa861e9a505d425792eb54a2a29cab97 P cc8d9fdc73b14a3cb716b47c24bafc1d: Time spent bootstrapping tablet: real 0.088s	user 0.020s	sys 0.020s
I20260709 12:28:14.220609 32528 raft_consensus.cc:2468] T 50fe031340a94b6590d29d44cd57ca55 P a8adcce962244bc688dc4ecc1f69612c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc8d9fdc73b14a3cb716b47c24bafc1d in term 1.
I20260709 12:28:14.222002 32418 leader_election.cc:304] T 50fe031340a94b6590d29d44cd57ca55 P cc8d9fdc73b14a3cb716b47c24bafc1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a8adcce962244bc688dc4ecc1f69612c, cc8d9fdc73b14a3cb716b47c24bafc1d; no voters: f8257d37af0941c2a978546ef2276b30
I20260709 12:28:14.223210 32649 raft_consensus.cc:2804] T 50fe031340a94b6590d29d44cd57ca55 P cc8d9fdc73b14a3cb716b47c24bafc1d [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:28:14.223214 32644 raft_consensus.cc:359] T aa861e9a505d425792eb54a2a29cab97 P cc8d9fdc73b14a3cb716b47c24bafc1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:14.224229 32644 raft_consensus.cc:385] T aa861e9a505d425792eb54a2a29cab97 P cc8d9fdc73b14a3cb716b47c24bafc1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:14.224578 32644 raft_consensus.cc:740] T aa861e9a505d425792eb54a2a29cab97 P cc8d9fdc73b14a3cb716b47c24bafc1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc8d9fdc73b14a3cb716b47c24bafc1d, State: Initialized, Role: FOLLOWER
I20260709 12:28:14.228173 32645 tablet_bootstrap.cc:492] T 4b2251c329154ac49d6147b1cfd6981b P a8adcce962244bc688dc4ecc1f69612c: No bootstrap required, opened a new log
I20260709 12:28:14.228960 32645 ts_tablet_manager.cc:1403] T 4b2251c329154ac49d6147b1cfd6981b P a8adcce962244bc688dc4ecc1f69612c: Time spent bootstrapping tablet: real 0.113s	user 0.039s	sys 0.008s
I20260709 12:28:14.230615 32642 tablet_bootstrap.cc:492] T d768701c29784c52a2ffb8303021347d P f8257d37af0941c2a978546ef2276b30: No bootstrap required, opened a new log
I20260709 12:28:14.231079 32642 ts_tablet_manager.cc:1403] T d768701c29784c52a2ffb8303021347d P f8257d37af0941c2a978546ef2276b30: Time spent bootstrapping tablet: real 0.067s	user 0.018s	sys 0.038s
I20260709 12:28:14.225368 32644 consensus_queue.cc:260] T aa861e9a505d425792eb54a2a29cab97 P cc8d9fdc73b14a3cb716b47c24bafc1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:14.233280 32649 raft_consensus.cc:697] T 50fe031340a94b6590d29d44cd57ca55 P cc8d9fdc73b14a3cb716b47c24bafc1d [term 1 LEADER]: Becoming Leader. State: Replica: cc8d9fdc73b14a3cb716b47c24bafc1d, State: Running, Role: LEADER
I20260709 12:28:14.233390 32645 raft_consensus.cc:359] T 4b2251c329154ac49d6147b1cfd6981b P a8adcce962244bc688dc4ecc1f69612c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:14.233999 32642 raft_consensus.cc:359] T d768701c29784c52a2ffb8303021347d P f8257d37af0941c2a978546ef2276b30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:14.234347 32649 consensus_queue.cc:237] T 50fe031340a94b6590d29d44cd57ca55 P cc8d9fdc73b14a3cb716b47c24bafc1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } } peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } }
I20260709 12:28:14.234200 32645 raft_consensus.cc:385] T 4b2251c329154ac49d6147b1cfd6981b P a8adcce962244bc688dc4ecc1f69612c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:14.234879 32642 raft_consensus.cc:385] T d768701c29784c52a2ffb8303021347d P f8257d37af0941c2a978546ef2276b30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:14.236078 32645 raft_consensus.cc:740] T 4b2251c329154ac49d6147b1cfd6981b P a8adcce962244bc688dc4ecc1f69612c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8adcce962244bc688dc4ecc1f69612c, State: Initialized, Role: FOLLOWER
I20260709 12:28:14.236229 32642 raft_consensus.cc:740] T d768701c29784c52a2ffb8303021347d P f8257d37af0941c2a978546ef2276b30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8257d37af0941c2a978546ef2276b30, State: Initialized, Role: FOLLOWER
I20260709 12:28:14.237053 32645 consensus_queue.cc:260] T 4b2251c329154ac49d6147b1cfd6981b P a8adcce962244bc688dc4ecc1f69612c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:14.237655 32642 consensus_queue.cc:260] T d768701c29784c52a2ffb8303021347d P f8257d37af0941c2a978546ef2276b30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:14.239308 32645 ts_tablet_manager.cc:1434] T 4b2251c329154ac49d6147b1cfd6981b P a8adcce962244bc688dc4ecc1f69612c: Time spent starting tablet: real 0.010s	user 0.004s	sys 0.003s
I20260709 12:28:14.239025 32661 raft_consensus.cc:493] T 8e7d5cee9cf64a3e87372bde6431883e P cc8d9fdc73b14a3cb716b47c24bafc1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:14.240290 32661 raft_consensus.cc:515] T 8e7d5cee9cf64a3e87372bde6431883e P cc8d9fdc73b14a3cb716b47c24bafc1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:14.241014 32645 tablet_bootstrap.cc:492] T 9396e8d0ed7b44f58bbc5b421f492f50 P a8adcce962244bc688dc4ecc1f69612c: Bootstrap starting.
I20260709 12:28:14.241822 32644 ts_tablet_manager.cc:1434] T aa861e9a505d425792eb54a2a29cab97 P cc8d9fdc73b14a3cb716b47c24bafc1d: Time spent starting tablet: real 0.022s	user 0.006s	sys 0.007s
I20260709 12:28:14.242913 32644 tablet_bootstrap.cc:492] T c127bfae907d4960956f04576bf0c6c7 P cc8d9fdc73b14a3cb716b47c24bafc1d: Bootstrap starting.
I20260709 12:28:14.243227 32528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e7d5cee9cf64a3e87372bde6431883e" candidate_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8adcce962244bc688dc4ecc1f69612c" is_pre_election: true
W20260709 12:28:14.244735 32418 leader_election.cc:343] T 8e7d5cee9cf64a3e87372bde6431883e P cc8d9fdc73b14a3cb716b47c24bafc1d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a8adcce962244bc688dc4ecc1f69612c (127.16.29.66:33369): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:28:14.246181 32661 leader_election.cc:290] T 8e7d5cee9cf64a3e87372bde6431883e P cc8d9fdc73b14a3cb716b47c24bafc1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a8adcce962244bc688dc4ecc1f69612c (127.16.29.66:33369), f8257d37af0941c2a978546ef2276b30 (127.16.29.67:37155)
I20260709 12:28:14.248027 32642 ts_tablet_manager.cc:1434] T d768701c29784c52a2ffb8303021347d P f8257d37af0941c2a978546ef2276b30: Time spent starting tablet: real 0.017s	user 0.007s	sys 0.004s
I20260709 12:28:14.248183 32645 tablet_bootstrap.cc:654] T 9396e8d0ed7b44f58bbc5b421f492f50 P a8adcce962244bc688dc4ecc1f69612c: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:14.248477 32647 raft_consensus.cc:493] T f22ce6f243334c2281f0543e605f614a P f8257d37af0941c2a978546ef2276b30 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:14.249150 32642 tablet_bootstrap.cc:492] T 8e7d5cee9cf64a3e87372bde6431883e P f8257d37af0941c2a978546ef2276b30: Bootstrap starting.
I20260709 12:28:14.248989 32647 raft_consensus.cc:515] T f22ce6f243334c2281f0543e605f614a P f8257d37af0941c2a978546ef2276b30 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:14.250746 32644 tablet_bootstrap.cc:654] T c127bfae907d4960956f04576bf0c6c7 P cc8d9fdc73b14a3cb716b47c24bafc1d: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:14.252930 32528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f22ce6f243334c2281f0543e605f614a" candidate_uuid: "f8257d37af0941c2a978546ef2276b30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8adcce962244bc688dc4ecc1f69612c" is_pre_election: true
I20260709 12:28:14.254292 32647 leader_election.cc:290] T f22ce6f243334c2281f0543e605f614a P f8257d37af0941c2a978546ef2276b30 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc8d9fdc73b14a3cb716b47c24bafc1d (127.16.29.65:41229), a8adcce962244bc688dc4ecc1f69612c (127.16.29.66:33369)
W20260709 12:28:14.256740 32567 leader_election.cc:343] T f22ce6f243334c2281f0543e605f614a P f8257d37af0941c2a978546ef2276b30 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a8adcce962244bc688dc4ecc1f69612c (127.16.29.66:33369): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:28:14.256978 32453 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f22ce6f243334c2281f0543e605f614a" candidate_uuid: "f8257d37af0941c2a978546ef2276b30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" is_pre_election: true
I20260709 12:28:14.247540 32602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e7d5cee9cf64a3e87372bde6431883e" candidate_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8257d37af0941c2a978546ef2276b30" is_pre_election: true
W20260709 12:28:14.258669 32568 leader_election.cc:343] T f22ce6f243334c2281f0543e605f614a P f8257d37af0941c2a978546ef2276b30 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cc8d9fdc73b14a3cb716b47c24bafc1d (127.16.29.65:41229): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:28:14.259114 32568 leader_election.cc:304] T f22ce6f243334c2281f0543e605f614a P f8257d37af0941c2a978546ef2276b30 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8257d37af0941c2a978546ef2276b30; no voters: a8adcce962244bc688dc4ecc1f69612c, cc8d9fdc73b14a3cb716b47c24bafc1d
I20260709 12:28:14.260411 32645 tablet_bootstrap.cc:492] T 9396e8d0ed7b44f58bbc5b421f492f50 P a8adcce962244bc688dc4ecc1f69612c: No bootstrap required, opened a new log
W20260709 12:28:14.260648 32418 leader_election.cc:343] T 8e7d5cee9cf64a3e87372bde6431883e P cc8d9fdc73b14a3cb716b47c24bafc1d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f8257d37af0941c2a978546ef2276b30 (127.16.29.67:37155): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:28:14.260984 32645 ts_tablet_manager.cc:1403] T 9396e8d0ed7b44f58bbc5b421f492f50 P a8adcce962244bc688dc4ecc1f69612c: Time spent bootstrapping tablet: real 0.020s	user 0.008s	sys 0.010s
I20260709 12:28:14.261200 32418 leader_election.cc:304] T 8e7d5cee9cf64a3e87372bde6431883e P cc8d9fdc73b14a3cb716b47c24bafc1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cc8d9fdc73b14a3cb716b47c24bafc1d; no voters: a8adcce962244bc688dc4ecc1f69612c, f8257d37af0941c2a978546ef2276b30
I20260709 12:28:14.262363 32661 raft_consensus.cc:2749] T 8e7d5cee9cf64a3e87372bde6431883e P cc8d9fdc73b14a3cb716b47c24bafc1d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:28:14.263167 32647 raft_consensus.cc:2749] T f22ce6f243334c2281f0543e605f614a P f8257d37af0941c2a978546ef2276b30 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:28:14.264828 32642 tablet_bootstrap.cc:654] T 8e7d5cee9cf64a3e87372bde6431883e P f8257d37af0941c2a978546ef2276b30: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:14.264673 32645 raft_consensus.cc:359] T 9396e8d0ed7b44f58bbc5b421f492f50 P a8adcce962244bc688dc4ecc1f69612c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:14.265532 32645 raft_consensus.cc:385] T 9396e8d0ed7b44f58bbc5b421f492f50 P a8adcce962244bc688dc4ecc1f69612c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:14.265875 32645 raft_consensus.cc:740] T 9396e8d0ed7b44f58bbc5b421f492f50 P a8adcce962244bc688dc4ecc1f69612c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8adcce962244bc688dc4ecc1f69612c, State: Initialized, Role: FOLLOWER
I20260709 12:28:14.267156 32645 consensus_queue.cc:260] T 9396e8d0ed7b44f58bbc5b421f492f50 P a8adcce962244bc688dc4ecc1f69612c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:14.271744 32645 ts_tablet_manager.cc:1434] T 9396e8d0ed7b44f58bbc5b421f492f50 P a8adcce962244bc688dc4ecc1f69612c: Time spent starting tablet: real 0.010s	user 0.001s	sys 0.007s
I20260709 12:28:14.272833 32645 tablet_bootstrap.cc:492] T c127bfae907d4960956f04576bf0c6c7 P a8adcce962244bc688dc4ecc1f69612c: Bootstrap starting.
I20260709 12:28:14.275074 32644 tablet_bootstrap.cc:492] T c127bfae907d4960956f04576bf0c6c7 P cc8d9fdc73b14a3cb716b47c24bafc1d: No bootstrap required, opened a new log
I20260709 12:28:14.275969 32644 ts_tablet_manager.cc:1403] T c127bfae907d4960956f04576bf0c6c7 P cc8d9fdc73b14a3cb716b47c24bafc1d: Time spent bootstrapping tablet: real 0.033s	user 0.018s	sys 0.001s
I20260709 12:28:14.280289 32644 raft_consensus.cc:359] T c127bfae907d4960956f04576bf0c6c7 P cc8d9fdc73b14a3cb716b47c24bafc1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:14.281550 32644 raft_consensus.cc:385] T c127bfae907d4960956f04576bf0c6c7 P cc8d9fdc73b14a3cb716b47c24bafc1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:14.282044 32644 raft_consensus.cc:740] T c127bfae907d4960956f04576bf0c6c7 P cc8d9fdc73b14a3cb716b47c24bafc1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc8d9fdc73b14a3cb716b47c24bafc1d, State: Initialized, Role: FOLLOWER
I20260709 12:28:14.283272 32644 consensus_queue.cc:260] T c127bfae907d4960956f04576bf0c6c7 P cc8d9fdc73b14a3cb716b47c24bafc1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:14.288231 32645 tablet_bootstrap.cc:654] T c127bfae907d4960956f04576bf0c6c7 P a8adcce962244bc688dc4ecc1f69612c: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:14.289661 32350 catalog_manager.cc:5697] T 50fe031340a94b6590d29d44cd57ca55 P cc8d9fdc73b14a3cb716b47c24bafc1d reported cstate change: term changed from 0 to 1, leader changed from <none> to cc8d9fdc73b14a3cb716b47c24bafc1d (127.16.29.65). New cstate: current_term: 1 leader_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:14.290759 32642 tablet_bootstrap.cc:492] T 8e7d5cee9cf64a3e87372bde6431883e P f8257d37af0941c2a978546ef2276b30: No bootstrap required, opened a new log
I20260709 12:28:14.291563 32642 ts_tablet_manager.cc:1403] T 8e7d5cee9cf64a3e87372bde6431883e P f8257d37af0941c2a978546ef2276b30: Time spent bootstrapping tablet: real 0.043s	user 0.016s	sys 0.017s
I20260709 12:28:14.292946 32645 tablet_bootstrap.cc:492] T c127bfae907d4960956f04576bf0c6c7 P a8adcce962244bc688dc4ecc1f69612c: No bootstrap required, opened a new log
I20260709 12:28:14.293475 32645 ts_tablet_manager.cc:1403] T c127bfae907d4960956f04576bf0c6c7 P a8adcce962244bc688dc4ecc1f69612c: Time spent bootstrapping tablet: real 0.021s	user 0.010s	sys 0.000s
I20260709 12:28:14.293543 32644 ts_tablet_manager.cc:1434] T c127bfae907d4960956f04576bf0c6c7 P cc8d9fdc73b14a3cb716b47c24bafc1d: Time spent starting tablet: real 0.017s	user 0.005s	sys 0.004s
I20260709 12:28:14.294831 32644 tablet_bootstrap.cc:492] T f22ce6f243334c2281f0543e605f614a P cc8d9fdc73b14a3cb716b47c24bafc1d: Bootstrap starting.
I20260709 12:28:14.294585 32642 raft_consensus.cc:359] T 8e7d5cee9cf64a3e87372bde6431883e P f8257d37af0941c2a978546ef2276b30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:14.295389 32642 raft_consensus.cc:385] T 8e7d5cee9cf64a3e87372bde6431883e P f8257d37af0941c2a978546ef2276b30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:14.295696 32642 raft_consensus.cc:740] T 8e7d5cee9cf64a3e87372bde6431883e P f8257d37af0941c2a978546ef2276b30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8257d37af0941c2a978546ef2276b30, State: Initialized, Role: FOLLOWER
I20260709 12:28:14.296657 32642 consensus_queue.cc:260] T 8e7d5cee9cf64a3e87372bde6431883e P f8257d37af0941c2a978546ef2276b30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:14.296669 32645 raft_consensus.cc:359] T c127bfae907d4960956f04576bf0c6c7 P a8adcce962244bc688dc4ecc1f69612c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:14.297573 32645 raft_consensus.cc:385] T c127bfae907d4960956f04576bf0c6c7 P a8adcce962244bc688dc4ecc1f69612c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:14.297986 32645 raft_consensus.cc:740] T c127bfae907d4960956f04576bf0c6c7 P a8adcce962244bc688dc4ecc1f69612c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8adcce962244bc688dc4ecc1f69612c, State: Initialized, Role: FOLLOWER
I20260709 12:28:14.299108 32645 consensus_queue.cc:260] T c127bfae907d4960956f04576bf0c6c7 P a8adcce962244bc688dc4ecc1f69612c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:14.301928 32642 ts_tablet_manager.cc:1434] T 8e7d5cee9cf64a3e87372bde6431883e P f8257d37af0941c2a978546ef2276b30: Time spent starting tablet: real 0.010s	user 0.007s	sys 0.000s
I20260709 12:28:14.302367 32644 tablet_bootstrap.cc:654] T f22ce6f243334c2281f0543e605f614a P cc8d9fdc73b14a3cb716b47c24bafc1d: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:14.303076 32642 tablet_bootstrap.cc:492] T 50fe031340a94b6590d29d44cd57ca55 P f8257d37af0941c2a978546ef2276b30: Bootstrap starting.
I20260709 12:28:14.303897 32647 raft_consensus.cc:493] T f5cb9ea668324bfb883f09876ddd56f7 P f8257d37af0941c2a978546ef2276b30 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:14.304410 32647 raft_consensus.cc:515] T f5cb9ea668324bfb883f09876ddd56f7 P f8257d37af0941c2a978546ef2276b30 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:14.305914 32645 ts_tablet_manager.cc:1434] T c127bfae907d4960956f04576bf0c6c7 P a8adcce962244bc688dc4ecc1f69612c: Time spent starting tablet: real 0.012s	user 0.006s	sys 0.002s
I20260709 12:28:14.306681 32647 leader_election.cc:290] T f5cb9ea668324bfb883f09876ddd56f7 P f8257d37af0941c2a978546ef2276b30 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc8d9fdc73b14a3cb716b47c24bafc1d (127.16.29.65:41229), a8adcce962244bc688dc4ecc1f69612c (127.16.29.66:33369)
I20260709 12:28:14.307044 32645 tablet_bootstrap.cc:492] T aa861e9a505d425792eb54a2a29cab97 P a8adcce962244bc688dc4ecc1f69612c: Bootstrap starting.
I20260709 12:28:14.309991 32528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5cb9ea668324bfb883f09876ddd56f7" candidate_uuid: "f8257d37af0941c2a978546ef2276b30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8adcce962244bc688dc4ecc1f69612c" is_pre_election: true
I20260709 12:28:14.309933 32453 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5cb9ea668324bfb883f09876ddd56f7" candidate_uuid: "f8257d37af0941c2a978546ef2276b30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" is_pre_election: true
W20260709 12:28:14.312319 32567 leader_election.cc:343] T f5cb9ea668324bfb883f09876ddd56f7 P f8257d37af0941c2a978546ef2276b30 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a8adcce962244bc688dc4ecc1f69612c (127.16.29.66:33369): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:28:14.313174 32647 raft_consensus.cc:493] T d768701c29784c52a2ffb8303021347d P f8257d37af0941c2a978546ef2276b30 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:28:14.313851 32568 leader_election.cc:343] T f5cb9ea668324bfb883f09876ddd56f7 P f8257d37af0941c2a978546ef2276b30 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cc8d9fdc73b14a3cb716b47c24bafc1d (127.16.29.65:41229): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:28:14.313984 32647 raft_consensus.cc:515] T d768701c29784c52a2ffb8303021347d P f8257d37af0941c2a978546ef2276b30 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:14.315442 32644 tablet_bootstrap.cc:492] T f22ce6f243334c2281f0543e605f614a P cc8d9fdc73b14a3cb716b47c24bafc1d: No bootstrap required, opened a new log
I20260709 12:28:14.315615 32645 tablet_bootstrap.cc:654] T aa861e9a505d425792eb54a2a29cab97 P a8adcce962244bc688dc4ecc1f69612c: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:14.316077 32644 ts_tablet_manager.cc:1403] T f22ce6f243334c2281f0543e605f614a P cc8d9fdc73b14a3cb716b47c24bafc1d: Time spent bootstrapping tablet: real 0.021s	user 0.016s	sys 0.003s
I20260709 12:28:14.316735 32568 leader_election.cc:304] T f5cb9ea668324bfb883f09876ddd56f7 P f8257d37af0941c2a978546ef2276b30 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8257d37af0941c2a978546ef2276b30; no voters: a8adcce962244bc688dc4ecc1f69612c, cc8d9fdc73b14a3cb716b47c24bafc1d
I20260709 12:28:14.316816 32647 leader_election.cc:290] T d768701c29784c52a2ffb8303021347d P f8257d37af0941c2a978546ef2276b30 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc8d9fdc73b14a3cb716b47c24bafc1d (127.16.29.65:41229), a8adcce962244bc688dc4ecc1f69612c (127.16.29.66:33369)
I20260709 12:28:14.317952 32647 raft_consensus.cc:2749] T f5cb9ea668324bfb883f09876ddd56f7 P f8257d37af0941c2a978546ef2276b30 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:28:14.319965 32528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d768701c29784c52a2ffb8303021347d" candidate_uuid: "f8257d37af0941c2a978546ef2276b30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8adcce962244bc688dc4ecc1f69612c" is_pre_election: true
I20260709 12:28:14.319796 32644 raft_consensus.cc:359] T f22ce6f243334c2281f0543e605f614a P cc8d9fdc73b14a3cb716b47c24bafc1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:14.320868 32644 raft_consensus.cc:385] T f22ce6f243334c2281f0543e605f614a P cc8d9fdc73b14a3cb716b47c24bafc1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:14.321177 32644 raft_consensus.cc:740] T f22ce6f243334c2281f0543e605f614a P cc8d9fdc73b14a3cb716b47c24bafc1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc8d9fdc73b14a3cb716b47c24bafc1d, State: Initialized, Role: FOLLOWER
I20260709 12:28:14.321643 32642 tablet_bootstrap.cc:654] T 50fe031340a94b6590d29d44cd57ca55 P f8257d37af0941c2a978546ef2276b30: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:14.322103 32644 consensus_queue.cc:260] T f22ce6f243334c2281f0543e605f614a P cc8d9fdc73b14a3cb716b47c24bafc1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
W20260709 12:28:14.322564 32567 leader_election.cc:343] T d768701c29784c52a2ffb8303021347d P f8257d37af0941c2a978546ef2276b30 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a8adcce962244bc688dc4ecc1f69612c (127.16.29.66:33369): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:28:14.324553 32644 ts_tablet_manager.cc:1434] T f22ce6f243334c2281f0543e605f614a P cc8d9fdc73b14a3cb716b47c24bafc1d: Time spent starting tablet: real 0.008s	user 0.008s	sys 0.000s
I20260709 12:28:14.325884 32644 tablet_bootstrap.cc:492] T f5cb9ea668324bfb883f09876ddd56f7 P cc8d9fdc73b14a3cb716b47c24bafc1d: Bootstrap starting.
I20260709 12:28:14.328052 32645 tablet_bootstrap.cc:492] T aa861e9a505d425792eb54a2a29cab97 P a8adcce962244bc688dc4ecc1f69612c: No bootstrap required, opened a new log
I20260709 12:28:14.328823 32645 ts_tablet_manager.cc:1403] T aa861e9a505d425792eb54a2a29cab97 P a8adcce962244bc688dc4ecc1f69612c: Time spent bootstrapping tablet: real 0.022s	user 0.016s	sys 0.004s
I20260709 12:28:14.332412 32645 raft_consensus.cc:359] T aa861e9a505d425792eb54a2a29cab97 P a8adcce962244bc688dc4ecc1f69612c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:14.333240 32645 raft_consensus.cc:385] T aa861e9a505d425792eb54a2a29cab97 P a8adcce962244bc688dc4ecc1f69612c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:14.333552 32645 raft_consensus.cc:740] T aa861e9a505d425792eb54a2a29cab97 P a8adcce962244bc688dc4ecc1f69612c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8adcce962244bc688dc4ecc1f69612c, State: Initialized, Role: FOLLOWER
I20260709 12:28:14.333781 32644 tablet_bootstrap.cc:654] T f5cb9ea668324bfb883f09876ddd56f7 P cc8d9fdc73b14a3cb716b47c24bafc1d: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:14.334564 32645 consensus_queue.cc:260] T aa861e9a505d425792eb54a2a29cab97 P a8adcce962244bc688dc4ecc1f69612c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:14.337270 32645 ts_tablet_manager.cc:1434] T aa861e9a505d425792eb54a2a29cab97 P a8adcce962244bc688dc4ecc1f69612c: Time spent starting tablet: real 0.008s	user 0.005s	sys 0.003s
I20260709 12:28:14.338562 32645 tablet_bootstrap.cc:492] T 8e7d5cee9cf64a3e87372bde6431883e P a8adcce962244bc688dc4ecc1f69612c: Bootstrap starting.
I20260709 12:28:14.342046 32642 tablet_bootstrap.cc:492] T 50fe031340a94b6590d29d44cd57ca55 P f8257d37af0941c2a978546ef2276b30: No bootstrap required, opened a new log
I20260709 12:28:14.342841 32642 ts_tablet_manager.cc:1403] T 50fe031340a94b6590d29d44cd57ca55 P f8257d37af0941c2a978546ef2276b30: Time spent bootstrapping tablet: real 0.040s	user 0.015s	sys 0.003s
I20260709 12:28:14.345645 32645 tablet_bootstrap.cc:654] T 8e7d5cee9cf64a3e87372bde6431883e P a8adcce962244bc688dc4ecc1f69612c: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:14.347226 32642 raft_consensus.cc:359] T 50fe031340a94b6590d29d44cd57ca55 P f8257d37af0941c2a978546ef2276b30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } } peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } }
I20260709 12:28:14.348644 32642 raft_consensus.cc:385] T 50fe031340a94b6590d29d44cd57ca55 P f8257d37af0941c2a978546ef2276b30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:14.349121 32642 raft_consensus.cc:740] T 50fe031340a94b6590d29d44cd57ca55 P f8257d37af0941c2a978546ef2276b30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8257d37af0941c2a978546ef2276b30, State: Initialized, Role: FOLLOWER
I20260709 12:28:14.350394 32642 consensus_queue.cc:260] T 50fe031340a94b6590d29d44cd57ca55 P f8257d37af0941c2a978546ef2276b30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } } peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } }
I20260709 12:28:14.354199 32642 ts_tablet_manager.cc:1434] T 50fe031340a94b6590d29d44cd57ca55 P f8257d37af0941c2a978546ef2276b30: Time spent starting tablet: real 0.011s	user 0.011s	sys 0.000s
I20260709 12:28:14.355700 32642 tablet_bootstrap.cc:492] T c127bfae907d4960956f04576bf0c6c7 P f8257d37af0941c2a978546ef2276b30: Bootstrap starting.
I20260709 12:28:14.358013 32645 tablet_bootstrap.cc:492] T 8e7d5cee9cf64a3e87372bde6431883e P a8adcce962244bc688dc4ecc1f69612c: No bootstrap required, opened a new log
I20260709 12:28:14.358580 32645 ts_tablet_manager.cc:1403] T 8e7d5cee9cf64a3e87372bde6431883e P a8adcce962244bc688dc4ecc1f69612c: Time spent bootstrapping tablet: real 0.020s	user 0.013s	sys 0.004s
W20260709 12:28:14.360493 32399 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:28:14.364265 32649 raft_consensus.cc:493] T f22ce6f243334c2281f0543e605f614a P cc8d9fdc73b14a3cb716b47c24bafc1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:14.364913 32649 raft_consensus.cc:515] T f22ce6f243334c2281f0543e605f614a P cc8d9fdc73b14a3cb716b47c24bafc1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:14.366717 32642 tablet_bootstrap.cc:654] T c127bfae907d4960956f04576bf0c6c7 P f8257d37af0941c2a978546ef2276b30: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:14.367337 32649 leader_election.cc:290] T f22ce6f243334c2281f0543e605f614a P cc8d9fdc73b14a3cb716b47c24bafc1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a8adcce962244bc688dc4ecc1f69612c (127.16.29.66:33369), f8257d37af0941c2a978546ef2276b30 (127.16.29.67:37155)
W20260709 12:28:14.368217 32568 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.29.65:41229, user_credentials={real_user=slave}} blocked reactor thread for 55460.9us
I20260709 12:28:14.369925 32453 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d768701c29784c52a2ffb8303021347d" candidate_uuid: "f8257d37af0941c2a978546ef2276b30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" is_pre_election: true
I20260709 12:28:14.371457 32528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f22ce6f243334c2281f0543e605f614a" candidate_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8adcce962244bc688dc4ecc1f69612c" is_pre_election: true
I20260709 12:28:14.372165 32645 raft_consensus.cc:359] T 8e7d5cee9cf64a3e87372bde6431883e P a8adcce962244bc688dc4ecc1f69612c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:14.372928 32645 raft_consensus.cc:385] T 8e7d5cee9cf64a3e87372bde6431883e P a8adcce962244bc688dc4ecc1f69612c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260709 12:28:14.372960 32418 leader_election.cc:343] T f22ce6f243334c2281f0543e605f614a P cc8d9fdc73b14a3cb716b47c24bafc1d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a8adcce962244bc688dc4ecc1f69612c (127.16.29.66:33369): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:28:14.373308 32645 raft_consensus.cc:740] T 8e7d5cee9cf64a3e87372bde6431883e P a8adcce962244bc688dc4ecc1f69612c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8adcce962244bc688dc4ecc1f69612c, State: Initialized, Role: FOLLOWER
I20260709 12:28:14.374132 32645 consensus_queue.cc:260] T 8e7d5cee9cf64a3e87372bde6431883e P a8adcce962244bc688dc4ecc1f69612c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:14.376397 32602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f22ce6f243334c2281f0543e605f614a" candidate_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8257d37af0941c2a978546ef2276b30" is_pre_election: true
I20260709 12:28:14.377082 32602 raft_consensus.cc:2468] T f22ce6f243334c2281f0543e605f614a P f8257d37af0941c2a978546ef2276b30 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc8d9fdc73b14a3cb716b47c24bafc1d in term 0.
I20260709 12:28:14.378180 32418 leader_election.cc:304] T f22ce6f243334c2281f0543e605f614a P cc8d9fdc73b14a3cb716b47c24bafc1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cc8d9fdc73b14a3cb716b47c24bafc1d, f8257d37af0941c2a978546ef2276b30; no voters: a8adcce962244bc688dc4ecc1f69612c
I20260709 12:28:14.378940 32649 raft_consensus.cc:2804] T f22ce6f243334c2281f0543e605f614a P cc8d9fdc73b14a3cb716b47c24bafc1d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:28:14.379323 32649 raft_consensus.cc:493] T f22ce6f243334c2281f0543e605f614a P cc8d9fdc73b14a3cb716b47c24bafc1d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:14.379642 32649 raft_consensus.cc:3060] T f22ce6f243334c2281f0543e605f614a P cc8d9fdc73b14a3cb716b47c24bafc1d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:14.379724 32642 tablet_bootstrap.cc:492] T c127bfae907d4960956f04576bf0c6c7 P f8257d37af0941c2a978546ef2276b30: No bootstrap required, opened a new log
I20260709 12:28:14.380404 32652 raft_consensus.cc:493] T c127bfae907d4960956f04576bf0c6c7 P a8adcce962244bc688dc4ecc1f69612c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:14.380571 32642 ts_tablet_manager.cc:1403] T c127bfae907d4960956f04576bf0c6c7 P f8257d37af0941c2a978546ef2276b30: Time spent bootstrapping tablet: real 0.025s	user 0.020s	sys 0.002s
I20260709 12:28:14.380934 32652 raft_consensus.cc:515] T c127bfae907d4960956f04576bf0c6c7 P a8adcce962244bc688dc4ecc1f69612c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:14.384809 32642 raft_consensus.cc:359] T c127bfae907d4960956f04576bf0c6c7 P f8257d37af0941c2a978546ef2276b30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:14.385789 32642 raft_consensus.cc:385] T c127bfae907d4960956f04576bf0c6c7 P f8257d37af0941c2a978546ef2276b30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:14.386344 32642 raft_consensus.cc:740] T c127bfae907d4960956f04576bf0c6c7 P f8257d37af0941c2a978546ef2276b30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8257d37af0941c2a978546ef2276b30, State: Initialized, Role: FOLLOWER
I20260709 12:28:14.387517 32642 consensus_queue.cc:260] T c127bfae907d4960956f04576bf0c6c7 P f8257d37af0941c2a978546ef2276b30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:14.392211 32642 ts_tablet_manager.cc:1434] T c127bfae907d4960956f04576bf0c6c7 P f8257d37af0941c2a978546ef2276b30: Time spent starting tablet: real 0.011s	user 0.005s	sys 0.005s
I20260709 12:28:14.392716 32652 leader_election.cc:290] T c127bfae907d4960956f04576bf0c6c7 P a8adcce962244bc688dc4ecc1f69612c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc8d9fdc73b14a3cb716b47c24bafc1d (127.16.29.65:41229), f8257d37af0941c2a978546ef2276b30 (127.16.29.67:37155)
I20260709 12:28:14.393482 32642 tablet_bootstrap.cc:492] T 9396e8d0ed7b44f58bbc5b421f492f50 P f8257d37af0941c2a978546ef2276b30: Bootstrap starting.
I20260709 12:28:14.393278 32649 raft_consensus.cc:515] T f22ce6f243334c2281f0543e605f614a P cc8d9fdc73b14a3cb716b47c24bafc1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:14.396900 32528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f22ce6f243334c2281f0543e605f614a" candidate_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8adcce962244bc688dc4ecc1f69612c"
I20260709 12:28:14.379750 32645 ts_tablet_manager.cc:1434] T 8e7d5cee9cf64a3e87372bde6431883e P a8adcce962244bc688dc4ecc1f69612c: Time spent starting tablet: real 0.021s	user 0.008s	sys 0.000s
W20260709 12:28:14.399003 32418 leader_election.cc:343] T f22ce6f243334c2281f0543e605f614a P cc8d9fdc73b14a3cb716b47c24bafc1d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a8adcce962244bc688dc4ecc1f69612c (127.16.29.66:33369): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:28:14.400005 32568 leader_election.cc:343] T d768701c29784c52a2ffb8303021347d P f8257d37af0941c2a978546ef2276b30 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cc8d9fdc73b14a3cb716b47c24bafc1d (127.16.29.65:41229): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:28:14.401371 32568 leader_election.cc:304] T d768701c29784c52a2ffb8303021347d P f8257d37af0941c2a978546ef2276b30 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8257d37af0941c2a978546ef2276b30; no voters: a8adcce962244bc688dc4ecc1f69612c, cc8d9fdc73b14a3cb716b47c24bafc1d
I20260709 12:28:14.404608 32671 raft_consensus.cc:2749] T d768701c29784c52a2ffb8303021347d P f8257d37af0941c2a978546ef2276b30 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:28:14.405150 32649 leader_election.cc:290] T f22ce6f243334c2281f0543e605f614a P cc8d9fdc73b14a3cb716b47c24bafc1d [CANDIDATE]: Term 1 election: Requested vote from peers a8adcce962244bc688dc4ecc1f69612c (127.16.29.66:33369), f8257d37af0941c2a978546ef2276b30 (127.16.29.67:37155)
I20260709 12:28:14.405717 32645 tablet_bootstrap.cc:492] T f22ce6f243334c2281f0543e605f614a P a8adcce962244bc688dc4ecc1f69612c: Bootstrap starting.
I20260709 12:28:14.406539 32602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f22ce6f243334c2281f0543e605f614a" candidate_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8257d37af0941c2a978546ef2276b30"
I20260709 12:28:14.407176 32602 raft_consensus.cc:3060] T f22ce6f243334c2281f0543e605f614a P f8257d37af0941c2a978546ef2276b30 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:14.417999 32453 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c127bfae907d4960956f04576bf0c6c7" candidate_uuid: "a8adcce962244bc688dc4ecc1f69612c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" is_pre_election: true
I20260709 12:28:14.418901 32453 raft_consensus.cc:2468] T c127bfae907d4960956f04576bf0c6c7 P cc8d9fdc73b14a3cb716b47c24bafc1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a8adcce962244bc688dc4ecc1f69612c in term 0.
I20260709 12:28:14.420496 32494 leader_election.cc:304] T c127bfae907d4960956f04576bf0c6c7 P a8adcce962244bc688dc4ecc1f69612c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a8adcce962244bc688dc4ecc1f69612c, cc8d9fdc73b14a3cb716b47c24bafc1d; no voters: 
I20260709 12:28:14.422019 32652 raft_consensus.cc:2804] T c127bfae907d4960956f04576bf0c6c7 P a8adcce962244bc688dc4ecc1f69612c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:28:14.422397 32652 raft_consensus.cc:493] T c127bfae907d4960956f04576bf0c6c7 P a8adcce962244bc688dc4ecc1f69612c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:14.422744 32652 raft_consensus.cc:3060] T c127bfae907d4960956f04576bf0c6c7 P a8adcce962244bc688dc4ecc1f69612c [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:14.424458 32642 tablet_bootstrap.cc:654] T 9396e8d0ed7b44f58bbc5b421f492f50 P f8257d37af0941c2a978546ef2276b30: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:14.426942 32645 tablet_bootstrap.cc:654] T f22ce6f243334c2281f0543e605f614a P a8adcce962244bc688dc4ecc1f69612c: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:14.431521 32652 raft_consensus.cc:515] T c127bfae907d4960956f04576bf0c6c7 P a8adcce962244bc688dc4ecc1f69612c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:14.435128 32453 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c127bfae907d4960956f04576bf0c6c7" candidate_uuid: "a8adcce962244bc688dc4ecc1f69612c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d"
I20260709 12:28:14.435820 32453 raft_consensus.cc:3060] T c127bfae907d4960956f04576bf0c6c7 P cc8d9fdc73b14a3cb716b47c24bafc1d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:14.436275 32601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c127bfae907d4960956f04576bf0c6c7" candidate_uuid: "a8adcce962244bc688dc4ecc1f69612c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8257d37af0941c2a978546ef2276b30" is_pre_election: true
I20260709 12:28:14.437069 32601 raft_consensus.cc:2468] T c127bfae907d4960956f04576bf0c6c7 P f8257d37af0941c2a978546ef2276b30 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a8adcce962244bc688dc4ecc1f69612c in term 0.
I20260709 12:28:14.442221 32645 tablet_bootstrap.cc:492] T f22ce6f243334c2281f0543e605f614a P a8adcce962244bc688dc4ecc1f69612c: No bootstrap required, opened a new log
I20260709 12:28:14.442848 32645 ts_tablet_manager.cc:1403] T f22ce6f243334c2281f0543e605f614a P a8adcce962244bc688dc4ecc1f69612c: Time spent bootstrapping tablet: real 0.037s	user 0.008s	sys 0.003s
I20260709 12:28:14.445045 32453 raft_consensus.cc:2468] T c127bfae907d4960956f04576bf0c6c7 P cc8d9fdc73b14a3cb716b47c24bafc1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a8adcce962244bc688dc4ecc1f69612c in term 1.
I20260709 12:28:14.417436 32602 raft_consensus.cc:2468] T f22ce6f243334c2281f0543e605f614a P f8257d37af0941c2a978546ef2276b30 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc8d9fdc73b14a3cb716b47c24bafc1d in term 1.
I20260709 12:28:14.446602 32652 leader_election.cc:290] T c127bfae907d4960956f04576bf0c6c7 P a8adcce962244bc688dc4ecc1f69612c [CANDIDATE]: Term 1 election: Requested vote from peers cc8d9fdc73b14a3cb716b47c24bafc1d (127.16.29.65:41229), f8257d37af0941c2a978546ef2276b30 (127.16.29.67:37155)
I20260709 12:28:14.446631 32642 tablet_bootstrap.cc:492] T 9396e8d0ed7b44f58bbc5b421f492f50 P f8257d37af0941c2a978546ef2276b30: No bootstrap required, opened a new log
I20260709 12:28:14.453222 32418 leader_election.cc:304] T f22ce6f243334c2281f0543e605f614a P cc8d9fdc73b14a3cb716b47c24bafc1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cc8d9fdc73b14a3cb716b47c24bafc1d, f8257d37af0941c2a978546ef2276b30; no voters: a8adcce962244bc688dc4ecc1f69612c
I20260709 12:28:14.453893 32642 ts_tablet_manager.cc:1403] T 9396e8d0ed7b44f58bbc5b421f492f50 P f8257d37af0941c2a978546ef2276b30: Time spent bootstrapping tablet: real 0.061s	user 0.012s	sys 0.019s
I20260709 12:28:14.447911 32601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c127bfae907d4960956f04576bf0c6c7" candidate_uuid: "a8adcce962244bc688dc4ecc1f69612c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8257d37af0941c2a978546ef2276b30"
I20260709 12:28:14.454674 32601 raft_consensus.cc:3060] T c127bfae907d4960956f04576bf0c6c7 P f8257d37af0941c2a978546ef2276b30 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:14.455775 32649 raft_consensus.cc:2804] T f22ce6f243334c2281f0543e605f614a P cc8d9fdc73b14a3cb716b47c24bafc1d [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:28:14.456513 32649 raft_consensus.cc:697] T f22ce6f243334c2281f0543e605f614a P cc8d9fdc73b14a3cb716b47c24bafc1d [term 1 LEADER]: Becoming Leader. State: Replica: cc8d9fdc73b14a3cb716b47c24bafc1d, State: Running, Role: LEADER
I20260709 12:28:14.457392 32642 raft_consensus.cc:359] T 9396e8d0ed7b44f58bbc5b421f492f50 P f8257d37af0941c2a978546ef2276b30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:14.446199 32645 raft_consensus.cc:359] T f22ce6f243334c2281f0543e605f614a P a8adcce962244bc688dc4ecc1f69612c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:14.457361 32649 consensus_queue.cc:237] T f22ce6f243334c2281f0543e605f614a P cc8d9fdc73b14a3cb716b47c24bafc1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:14.458370 32645 raft_consensus.cc:385] T f22ce6f243334c2281f0543e605f614a P a8adcce962244bc688dc4ecc1f69612c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:14.458756 32645 raft_consensus.cc:740] T f22ce6f243334c2281f0543e605f614a P a8adcce962244bc688dc4ecc1f69612c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8adcce962244bc688dc4ecc1f69612c, State: Initialized, Role: FOLLOWER
I20260709 12:28:14.459688 32645 consensus_queue.cc:260] T f22ce6f243334c2281f0543e605f614a P a8adcce962244bc688dc4ecc1f69612c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:14.460223 32494 leader_election.cc:304] T c127bfae907d4960956f04576bf0c6c7 P a8adcce962244bc688dc4ecc1f69612c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a8adcce962244bc688dc4ecc1f69612c, cc8d9fdc73b14a3cb716b47c24bafc1d; no voters: 
I20260709 12:28:14.449373 32644 tablet_bootstrap.cc:492] T f5cb9ea668324bfb883f09876ddd56f7 P cc8d9fdc73b14a3cb716b47c24bafc1d: No bootstrap required, opened a new log
I20260709 12:28:14.461496 32644 ts_tablet_manager.cc:1403] T f5cb9ea668324bfb883f09876ddd56f7 P cc8d9fdc73b14a3cb716b47c24bafc1d: Time spent bootstrapping tablet: real 0.136s	user 0.033s	sys 0.063s
I20260709 12:28:14.463241 32661 raft_consensus.cc:493] T aa861e9a505d425792eb54a2a29cab97 P cc8d9fdc73b14a3cb716b47c24bafc1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:14.463389 32601 raft_consensus.cc:2468] T c127bfae907d4960956f04576bf0c6c7 P f8257d37af0941c2a978546ef2276b30 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a8adcce962244bc688dc4ecc1f69612c in term 1.
I20260709 12:28:14.463788 32661 raft_consensus.cc:515] T aa861e9a505d425792eb54a2a29cab97 P cc8d9fdc73b14a3cb716b47c24bafc1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:14.464406 32652 raft_consensus.cc:2804] T c127bfae907d4960956f04576bf0c6c7 P a8adcce962244bc688dc4ecc1f69612c [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:28:14.465050 32652 raft_consensus.cc:697] T c127bfae907d4960956f04576bf0c6c7 P a8adcce962244bc688dc4ecc1f69612c [term 1 LEADER]: Becoming Leader. State: Replica: a8adcce962244bc688dc4ecc1f69612c, State: Running, Role: LEADER
I20260709 12:28:14.465996 32652 consensus_queue.cc:237] T c127bfae907d4960956f04576bf0c6c7 P a8adcce962244bc688dc4ecc1f69612c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:14.466338 32661 leader_election.cc:290] T aa861e9a505d425792eb54a2a29cab97 P cc8d9fdc73b14a3cb716b47c24bafc1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a8adcce962244bc688dc4ecc1f69612c (127.16.29.66:33369), f8257d37af0941c2a978546ef2276b30 (127.16.29.67:37155)
I20260709 12:28:14.468917 32528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aa861e9a505d425792eb54a2a29cab97" candidate_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8adcce962244bc688dc4ecc1f69612c" is_pre_election: true
I20260709 12:28:14.469219 32644 raft_consensus.cc:359] T f5cb9ea668324bfb883f09876ddd56f7 P cc8d9fdc73b14a3cb716b47c24bafc1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:14.469828 32528 raft_consensus.cc:2468] T aa861e9a505d425792eb54a2a29cab97 P a8adcce962244bc688dc4ecc1f69612c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc8d9fdc73b14a3cb716b47c24bafc1d in term 0.
I20260709 12:28:14.470156 32644 raft_consensus.cc:385] T f5cb9ea668324bfb883f09876ddd56f7 P cc8d9fdc73b14a3cb716b47c24bafc1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:14.470563 32644 raft_consensus.cc:740] T f5cb9ea668324bfb883f09876ddd56f7 P cc8d9fdc73b14a3cb716b47c24bafc1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc8d9fdc73b14a3cb716b47c24bafc1d, State: Initialized, Role: FOLLOWER
I20260709 12:28:14.471386 32645 ts_tablet_manager.cc:1434] T f22ce6f243334c2281f0543e605f614a P a8adcce962244bc688dc4ecc1f69612c: Time spent starting tablet: real 0.028s	user 0.003s	sys 0.007s
I20260709 12:28:14.471648 32418 leader_election.cc:304] T aa861e9a505d425792eb54a2a29cab97 P cc8d9fdc73b14a3cb716b47c24bafc1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a8adcce962244bc688dc4ecc1f69612c, cc8d9fdc73b14a3cb716b47c24bafc1d; no voters: 
I20260709 12:28:14.472440 32642 raft_consensus.cc:385] T 9396e8d0ed7b44f58bbc5b421f492f50 P f8257d37af0941c2a978546ef2276b30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:14.472801 32642 raft_consensus.cc:740] T 9396e8d0ed7b44f58bbc5b421f492f50 P f8257d37af0941c2a978546ef2276b30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8257d37af0941c2a978546ef2276b30, State: Initialized, Role: FOLLOWER
I20260709 12:28:14.472851 32661 raft_consensus.cc:2804] T aa861e9a505d425792eb54a2a29cab97 P cc8d9fdc73b14a3cb716b47c24bafc1d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:28:14.474210 32661 raft_consensus.cc:493] T aa861e9a505d425792eb54a2a29cab97 P cc8d9fdc73b14a3cb716b47c24bafc1d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:14.474251 32601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aa861e9a505d425792eb54a2a29cab97" candidate_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8257d37af0941c2a978546ef2276b30" is_pre_election: true
I20260709 12:28:14.474748 32661 raft_consensus.cc:3060] T aa861e9a505d425792eb54a2a29cab97 P cc8d9fdc73b14a3cb716b47c24bafc1d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:14.475164 32601 raft_consensus.cc:2468] T aa861e9a505d425792eb54a2a29cab97 P f8257d37af0941c2a978546ef2276b30 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc8d9fdc73b14a3cb716b47c24bafc1d in term 0.
I20260709 12:28:14.473722 32645 tablet_bootstrap.cc:492] T d768701c29784c52a2ffb8303021347d P a8adcce962244bc688dc4ecc1f69612c: Bootstrap starting.
I20260709 12:28:14.481300 32642 consensus_queue.cc:260] T 9396e8d0ed7b44f58bbc5b421f492f50 P f8257d37af0941c2a978546ef2276b30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:14.483871 32661 raft_consensus.cc:515] T aa861e9a505d425792eb54a2a29cab97 P cc8d9fdc73b14a3cb716b47c24bafc1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:14.485191 32350 catalog_manager.cc:5697] T c127bfae907d4960956f04576bf0c6c7 P a8adcce962244bc688dc4ecc1f69612c reported cstate change: term changed from 0 to 1, leader changed from <none> to a8adcce962244bc688dc4ecc1f69612c (127.16.29.66). New cstate: current_term: 1 leader_uuid: "a8adcce962244bc688dc4ecc1f69612c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:14.487356 32645 tablet_bootstrap.cc:654] T d768701c29784c52a2ffb8303021347d P a8adcce962244bc688dc4ecc1f69612c: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:14.487771 32642 ts_tablet_manager.cc:1434] T 9396e8d0ed7b44f58bbc5b421f492f50 P f8257d37af0941c2a978546ef2276b30: Time spent starting tablet: real 0.033s	user 0.001s	sys 0.005s
I20260709 12:28:14.489924 32642 tablet_bootstrap.cc:492] T 4b2251c329154ac49d6147b1cfd6981b P f8257d37af0941c2a978546ef2276b30: Bootstrap starting.
I20260709 12:28:14.493566 32644 consensus_queue.cc:260] T f5cb9ea668324bfb883f09876ddd56f7 P cc8d9fdc73b14a3cb716b47c24bafc1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:14.496290 32642 tablet_bootstrap.cc:654] T 4b2251c329154ac49d6147b1cfd6981b P f8257d37af0941c2a978546ef2276b30: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:14.496517 32644 ts_tablet_manager.cc:1434] T f5cb9ea668324bfb883f09876ddd56f7 P cc8d9fdc73b14a3cb716b47c24bafc1d: Time spent starting tablet: real 0.035s	user 0.006s	sys 0.001s
I20260709 12:28:14.498759 32348 catalog_manager.cc:5697] T f22ce6f243334c2281f0543e605f614a P cc8d9fdc73b14a3cb716b47c24bafc1d reported cstate change: term changed from 0 to 1, leader changed from <none> to cc8d9fdc73b14a3cb716b47c24bafc1d (127.16.29.65). New cstate: current_term: 1 leader_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:14.500612 32644 tablet_bootstrap.cc:492] T 9396e8d0ed7b44f58bbc5b421f492f50 P cc8d9fdc73b14a3cb716b47c24bafc1d: Bootstrap starting.
I20260709 12:28:14.508407 32528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aa861e9a505d425792eb54a2a29cab97" candidate_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8adcce962244bc688dc4ecc1f69612c"
I20260709 12:28:14.509124 32528 raft_consensus.cc:3060] T aa861e9a505d425792eb54a2a29cab97 P a8adcce962244bc688dc4ecc1f69612c [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:14.516955 32671 raft_consensus.cc:493] T 50fe031340a94b6590d29d44cd57ca55 P f8257d37af0941c2a978546ef2276b30 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:14.517817 32671 raft_consensus.cc:515] T 50fe031340a94b6590d29d44cd57ca55 P f8257d37af0941c2a978546ef2276b30 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } } peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } }
I20260709 12:28:14.518478 32644 tablet_bootstrap.cc:654] T 9396e8d0ed7b44f58bbc5b421f492f50 P cc8d9fdc73b14a3cb716b47c24bafc1d: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:14.521723 32453 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fe031340a94b6590d29d44cd57ca55" candidate_uuid: "f8257d37af0941c2a978546ef2276b30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" is_pre_election: true
I20260709 12:28:14.521978 32671 leader_election.cc:290] T 50fe031340a94b6590d29d44cd57ca55 P f8257d37af0941c2a978546ef2276b30 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc8d9fdc73b14a3cb716b47c24bafc1d (127.16.29.65:41229), a8adcce962244bc688dc4ecc1f69612c (127.16.29.66:33369)
I20260709 12:28:14.522387 32527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fe031340a94b6590d29d44cd57ca55" candidate_uuid: "f8257d37af0941c2a978546ef2276b30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8adcce962244bc688dc4ecc1f69612c" is_pre_election: true
I20260709 12:28:14.523252 32527 raft_consensus.cc:2393] T 50fe031340a94b6590d29d44cd57ca55 P a8adcce962244bc688dc4ecc1f69612c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f8257d37af0941c2a978546ef2276b30 in current term 1: Already voted for candidate cc8d9fdc73b14a3cb716b47c24bafc1d in this term.
I20260709 12:28:14.524462 32567 leader_election.cc:304] T 50fe031340a94b6590d29d44cd57ca55 P f8257d37af0941c2a978546ef2276b30 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8257d37af0941c2a978546ef2276b30; no voters: a8adcce962244bc688dc4ecc1f69612c, cc8d9fdc73b14a3cb716b47c24bafc1d
I20260709 12:28:14.525683 32671 raft_consensus.cc:3060] T 50fe031340a94b6590d29d44cd57ca55 P f8257d37af0941c2a978546ef2276b30 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:14.527444 32601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aa861e9a505d425792eb54a2a29cab97" candidate_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8257d37af0941c2a978546ef2276b30"
I20260709 12:28:14.527853 32528 raft_consensus.cc:2468] T aa861e9a505d425792eb54a2a29cab97 P a8adcce962244bc688dc4ecc1f69612c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc8d9fdc73b14a3cb716b47c24bafc1d in term 1.
I20260709 12:28:14.528342 32601 raft_consensus.cc:3060] T aa861e9a505d425792eb54a2a29cab97 P f8257d37af0941c2a978546ef2276b30 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:14.529151 32418 leader_election.cc:304] T aa861e9a505d425792eb54a2a29cab97 P cc8d9fdc73b14a3cb716b47c24bafc1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a8adcce962244bc688dc4ecc1f69612c, cc8d9fdc73b14a3cb716b47c24bafc1d; no voters: 
I20260709 12:28:14.530117 32649 raft_consensus.cc:2804] T aa861e9a505d425792eb54a2a29cab97 P cc8d9fdc73b14a3cb716b47c24bafc1d [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:28:14.530582 32649 raft_consensus.cc:697] T aa861e9a505d425792eb54a2a29cab97 P cc8d9fdc73b14a3cb716b47c24bafc1d [term 1 LEADER]: Becoming Leader. State: Replica: cc8d9fdc73b14a3cb716b47c24bafc1d, State: Running, Role: LEADER
I20260709 12:28:14.531425 32649 consensus_queue.cc:237] T aa861e9a505d425792eb54a2a29cab97 P cc8d9fdc73b14a3cb716b47c24bafc1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:14.533746 32642 tablet_bootstrap.cc:492] T 4b2251c329154ac49d6147b1cfd6981b P f8257d37af0941c2a978546ef2276b30: No bootstrap required, opened a new log
I20260709 12:28:14.534257 32642 ts_tablet_manager.cc:1403] T 4b2251c329154ac49d6147b1cfd6981b P f8257d37af0941c2a978546ef2276b30: Time spent bootstrapping tablet: real 0.045s	user 0.016s	sys 0.011s
I20260709 12:28:14.537475 32642 raft_consensus.cc:359] T 4b2251c329154ac49d6147b1cfd6981b P f8257d37af0941c2a978546ef2276b30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:14.538335 32642 raft_consensus.cc:385] T 4b2251c329154ac49d6147b1cfd6981b P f8257d37af0941c2a978546ef2276b30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:14.538637 32642 raft_consensus.cc:740] T 4b2251c329154ac49d6147b1cfd6981b P f8257d37af0941c2a978546ef2276b30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8257d37af0941c2a978546ef2276b30, State: Initialized, Role: FOLLOWER
I20260709 12:28:14.539500 32661 leader_election.cc:290] T aa861e9a505d425792eb54a2a29cab97 P cc8d9fdc73b14a3cb716b47c24bafc1d [CANDIDATE]: Term 1 election: Requested vote from peers a8adcce962244bc688dc4ecc1f69612c (127.16.29.66:33369), f8257d37af0941c2a978546ef2276b30 (127.16.29.67:37155)
I20260709 12:28:14.539546 32642 consensus_queue.cc:260] T 4b2251c329154ac49d6147b1cfd6981b P f8257d37af0941c2a978546ef2276b30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:14.542093 32671 raft_consensus.cc:2749] T 50fe031340a94b6590d29d44cd57ca55 P f8257d37af0941c2a978546ef2276b30 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:28:14.543411 32642 ts_tablet_manager.cc:1434] T 4b2251c329154ac49d6147b1cfd6981b P f8257d37af0941c2a978546ef2276b30: Time spent starting tablet: real 0.009s	user 0.006s	sys 0.002s
I20260709 12:28:14.539065 32601 raft_consensus.cc:2468] T aa861e9a505d425792eb54a2a29cab97 P f8257d37af0941c2a978546ef2276b30 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc8d9fdc73b14a3cb716b47c24bafc1d in term 1.
I20260709 12:28:14.548353 32645 tablet_bootstrap.cc:492] T d768701c29784c52a2ffb8303021347d P a8adcce962244bc688dc4ecc1f69612c: No bootstrap required, opened a new log
I20260709 12:28:14.548960 32645 ts_tablet_manager.cc:1403] T d768701c29784c52a2ffb8303021347d P a8adcce962244bc688dc4ecc1f69612c: Time spent bootstrapping tablet: real 0.075s	user 0.017s	sys 0.043s
I20260709 12:28:14.549465 32644 tablet_bootstrap.cc:492] T 9396e8d0ed7b44f58bbc5b421f492f50 P cc8d9fdc73b14a3cb716b47c24bafc1d: No bootstrap required, opened a new log
I20260709 12:28:14.550033 32644 ts_tablet_manager.cc:1403] T 9396e8d0ed7b44f58bbc5b421f492f50 P cc8d9fdc73b14a3cb716b47c24bafc1d: Time spent bootstrapping tablet: real 0.050s	user 0.016s	sys 0.001s
I20260709 12:28:14.551859 32645 raft_consensus.cc:359] T d768701c29784c52a2ffb8303021347d P a8adcce962244bc688dc4ecc1f69612c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:14.552691 32645 raft_consensus.cc:385] T d768701c29784c52a2ffb8303021347d P a8adcce962244bc688dc4ecc1f69612c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:14.553030 32645 raft_consensus.cc:740] T d768701c29784c52a2ffb8303021347d P a8adcce962244bc688dc4ecc1f69612c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8adcce962244bc688dc4ecc1f69612c, State: Initialized, Role: FOLLOWER
I20260709 12:28:14.552984 32644 raft_consensus.cc:359] T 9396e8d0ed7b44f58bbc5b421f492f50 P cc8d9fdc73b14a3cb716b47c24bafc1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:14.552816 32347 catalog_manager.cc:5697] T aa861e9a505d425792eb54a2a29cab97 P cc8d9fdc73b14a3cb716b47c24bafc1d reported cstate change: term changed from 0 to 1, leader changed from <none> to cc8d9fdc73b14a3cb716b47c24bafc1d (127.16.29.65). New cstate: current_term: 1 leader_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:14.554021 32644 raft_consensus.cc:385] T 9396e8d0ed7b44f58bbc5b421f492f50 P cc8d9fdc73b14a3cb716b47c24bafc1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:14.554426 32644 raft_consensus.cc:740] T 9396e8d0ed7b44f58bbc5b421f492f50 P cc8d9fdc73b14a3cb716b47c24bafc1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc8d9fdc73b14a3cb716b47c24bafc1d, State: Initialized, Role: FOLLOWER
I20260709 12:28:14.554173 32645 consensus_queue.cc:260] T d768701c29784c52a2ffb8303021347d P a8adcce962244bc688dc4ecc1f69612c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:14.555258 32644 consensus_queue.cc:260] T 9396e8d0ed7b44f58bbc5b421f492f50 P cc8d9fdc73b14a3cb716b47c24bafc1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:14.557860 32644 ts_tablet_manager.cc:1434] T 9396e8d0ed7b44f58bbc5b421f492f50 P cc8d9fdc73b14a3cb716b47c24bafc1d: Time spent starting tablet: real 0.007s	user 0.006s	sys 0.000s
I20260709 12:28:14.558898 32644 tablet_bootstrap.cc:492] T 4b2251c329154ac49d6147b1cfd6981b P cc8d9fdc73b14a3cb716b47c24bafc1d: Bootstrap starting.
I20260709 12:28:14.559612 32645 ts_tablet_manager.cc:1434] T d768701c29784c52a2ffb8303021347d P a8adcce962244bc688dc4ecc1f69612c: Time spent starting tablet: real 0.010s	user 0.006s	sys 0.000s
I20260709 12:28:14.560655 32645 tablet_bootstrap.cc:492] T f5cb9ea668324bfb883f09876ddd56f7 P a8adcce962244bc688dc4ecc1f69612c: Bootstrap starting.
I20260709 12:28:14.565459 32644 tablet_bootstrap.cc:654] T 4b2251c329154ac49d6147b1cfd6981b P cc8d9fdc73b14a3cb716b47c24bafc1d: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:14.566967 32645 tablet_bootstrap.cc:654] T f5cb9ea668324bfb883f09876ddd56f7 P a8adcce962244bc688dc4ecc1f69612c: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:14.572816 32645 tablet_bootstrap.cc:492] T f5cb9ea668324bfb883f09876ddd56f7 P a8adcce962244bc688dc4ecc1f69612c: No bootstrap required, opened a new log
I20260709 12:28:14.573316 32645 ts_tablet_manager.cc:1403] T f5cb9ea668324bfb883f09876ddd56f7 P a8adcce962244bc688dc4ecc1f69612c: Time spent bootstrapping tablet: real 0.013s	user 0.004s	sys 0.006s
I20260709 12:28:14.573653 32644 tablet_bootstrap.cc:492] T 4b2251c329154ac49d6147b1cfd6981b P cc8d9fdc73b14a3cb716b47c24bafc1d: No bootstrap required, opened a new log
I20260709 12:28:14.574142 32644 ts_tablet_manager.cc:1403] T 4b2251c329154ac49d6147b1cfd6981b P cc8d9fdc73b14a3cb716b47c24bafc1d: Time spent bootstrapping tablet: real 0.015s	user 0.009s	sys 0.005s
I20260709 12:28:14.575424 32645 raft_consensus.cc:359] T f5cb9ea668324bfb883f09876ddd56f7 P a8adcce962244bc688dc4ecc1f69612c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:14.575987 32645 raft_consensus.cc:385] T f5cb9ea668324bfb883f09876ddd56f7 P a8adcce962244bc688dc4ecc1f69612c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:14.576174 32645 raft_consensus.cc:740] T f5cb9ea668324bfb883f09876ddd56f7 P a8adcce962244bc688dc4ecc1f69612c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8adcce962244bc688dc4ecc1f69612c, State: Initialized, Role: FOLLOWER
I20260709 12:28:14.576557 32644 raft_consensus.cc:359] T 4b2251c329154ac49d6147b1cfd6981b P cc8d9fdc73b14a3cb716b47c24bafc1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:14.576731 32645 consensus_queue.cc:260] T f5cb9ea668324bfb883f09876ddd56f7 P a8adcce962244bc688dc4ecc1f69612c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:14.577219 32644 raft_consensus.cc:385] T 4b2251c329154ac49d6147b1cfd6981b P cc8d9fdc73b14a3cb716b47c24bafc1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:14.577525 32644 raft_consensus.cc:740] T 4b2251c329154ac49d6147b1cfd6981b P cc8d9fdc73b14a3cb716b47c24bafc1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc8d9fdc73b14a3cb716b47c24bafc1d, State: Initialized, Role: FOLLOWER
I20260709 12:28:14.578130 32644 consensus_queue.cc:260] T 4b2251c329154ac49d6147b1cfd6981b P cc8d9fdc73b14a3cb716b47c24bafc1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:14.578557 32645 ts_tablet_manager.cc:1434] T f5cb9ea668324bfb883f09876ddd56f7 P a8adcce962244bc688dc4ecc1f69612c: Time spent starting tablet: real 0.005s	user 0.002s	sys 0.000s
I20260709 12:28:14.580263 32644 ts_tablet_manager.cc:1434] T 4b2251c329154ac49d6147b1cfd6981b P cc8d9fdc73b14a3cb716b47c24bafc1d: Time spent starting tablet: real 0.006s	user 0.008s	sys 0.000s
I20260709 12:28:14.581212 32644 tablet_bootstrap.cc:492] T d768701c29784c52a2ffb8303021347d P cc8d9fdc73b14a3cb716b47c24bafc1d: Bootstrap starting.
I20260709 12:28:14.587097 32644 tablet_bootstrap.cc:654] T d768701c29784c52a2ffb8303021347d P cc8d9fdc73b14a3cb716b47c24bafc1d: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:14.588199 32652 raft_consensus.cc:493] T f5cb9ea668324bfb883f09876ddd56f7 P a8adcce962244bc688dc4ecc1f69612c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:14.588757 32652 raft_consensus.cc:515] T f5cb9ea668324bfb883f09876ddd56f7 P a8adcce962244bc688dc4ecc1f69612c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:14.590827 32652 leader_election.cc:290] T f5cb9ea668324bfb883f09876ddd56f7 P a8adcce962244bc688dc4ecc1f69612c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc8d9fdc73b14a3cb716b47c24bafc1d (127.16.29.65:41229), f8257d37af0941c2a978546ef2276b30 (127.16.29.67:37155)
I20260709 12:28:14.592068 32601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5cb9ea668324bfb883f09876ddd56f7" candidate_uuid: "a8adcce962244bc688dc4ecc1f69612c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8257d37af0941c2a978546ef2276b30" is_pre_election: true
I20260709 12:28:14.592475 32453 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5cb9ea668324bfb883f09876ddd56f7" candidate_uuid: "a8adcce962244bc688dc4ecc1f69612c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" is_pre_election: true
I20260709 12:28:14.592794 32601 raft_consensus.cc:2468] T f5cb9ea668324bfb883f09876ddd56f7 P f8257d37af0941c2a978546ef2276b30 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a8adcce962244bc688dc4ecc1f69612c in term 0.
I20260709 12:28:14.593116 32453 raft_consensus.cc:2468] T f5cb9ea668324bfb883f09876ddd56f7 P cc8d9fdc73b14a3cb716b47c24bafc1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a8adcce962244bc688dc4ecc1f69612c in term 0.
I20260709 12:28:14.594127 32652 raft_consensus.cc:493] T 4b2251c329154ac49d6147b1cfd6981b P a8adcce962244bc688dc4ecc1f69612c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:14.594110 32493 leader_election.cc:304] T f5cb9ea668324bfb883f09876ddd56f7 P a8adcce962244bc688dc4ecc1f69612c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a8adcce962244bc688dc4ecc1f69612c, f8257d37af0941c2a978546ef2276b30; no voters: 
I20260709 12:28:14.595327 32683 raft_consensus.cc:2804] T f5cb9ea668324bfb883f09876ddd56f7 P a8adcce962244bc688dc4ecc1f69612c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:28:14.594862 32652 raft_consensus.cc:515] T 4b2251c329154ac49d6147b1cfd6981b P a8adcce962244bc688dc4ecc1f69612c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:14.595762 32683 raft_consensus.cc:493] T f5cb9ea668324bfb883f09876ddd56f7 P a8adcce962244bc688dc4ecc1f69612c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:14.596210 32683 raft_consensus.cc:3060] T f5cb9ea668324bfb883f09876ddd56f7 P a8adcce962244bc688dc4ecc1f69612c [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:14.596654 32644 tablet_bootstrap.cc:492] T d768701c29784c52a2ffb8303021347d P cc8d9fdc73b14a3cb716b47c24bafc1d: No bootstrap required, opened a new log
I20260709 12:28:14.597148 32644 ts_tablet_manager.cc:1403] T d768701c29784c52a2ffb8303021347d P cc8d9fdc73b14a3cb716b47c24bafc1d: Time spent bootstrapping tablet: real 0.016s	user 0.005s	sys 0.004s
I20260709 12:28:14.597527 32652 leader_election.cc:290] T 4b2251c329154ac49d6147b1cfd6981b P a8adcce962244bc688dc4ecc1f69612c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc8d9fdc73b14a3cb716b47c24bafc1d (127.16.29.65:41229), f8257d37af0941c2a978546ef2276b30 (127.16.29.67:37155)
I20260709 12:28:14.598726 32601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b2251c329154ac49d6147b1cfd6981b" candidate_uuid: "a8adcce962244bc688dc4ecc1f69612c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8257d37af0941c2a978546ef2276b30" is_pre_election: true
I20260709 12:28:14.599381 32601 raft_consensus.cc:2468] T 4b2251c329154ac49d6147b1cfd6981b P f8257d37af0941c2a978546ef2276b30 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a8adcce962244bc688dc4ecc1f69612c in term 0.
I20260709 12:28:14.599938 32644 raft_consensus.cc:359] T d768701c29784c52a2ffb8303021347d P cc8d9fdc73b14a3cb716b47c24bafc1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:14.600539 32644 raft_consensus.cc:385] T d768701c29784c52a2ffb8303021347d P cc8d9fdc73b14a3cb716b47c24bafc1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:14.600831 32644 raft_consensus.cc:740] T d768701c29784c52a2ffb8303021347d P cc8d9fdc73b14a3cb716b47c24bafc1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc8d9fdc73b14a3cb716b47c24bafc1d, State: Initialized, Role: FOLLOWER
I20260709 12:28:14.600665 32493 leader_election.cc:304] T 4b2251c329154ac49d6147b1cfd6981b P a8adcce962244bc688dc4ecc1f69612c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a8adcce962244bc688dc4ecc1f69612c, f8257d37af0941c2a978546ef2276b30; no voters: 
I20260709 12:28:14.601630 32453 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b2251c329154ac49d6147b1cfd6981b" candidate_uuid: "a8adcce962244bc688dc4ecc1f69612c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" is_pre_election: true
I20260709 12:28:14.601610 32644 consensus_queue.cc:260] T d768701c29784c52a2ffb8303021347d P cc8d9fdc73b14a3cb716b47c24bafc1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:14.601931 32652 raft_consensus.cc:2804] T 4b2251c329154ac49d6147b1cfd6981b P a8adcce962244bc688dc4ecc1f69612c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:28:14.602429 32453 raft_consensus.cc:2468] T 4b2251c329154ac49d6147b1cfd6981b P cc8d9fdc73b14a3cb716b47c24bafc1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a8adcce962244bc688dc4ecc1f69612c in term 0.
I20260709 12:28:14.602542 32652 raft_consensus.cc:493] T 4b2251c329154ac49d6147b1cfd6981b P a8adcce962244bc688dc4ecc1f69612c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:14.603312 32652 raft_consensus.cc:3060] T 4b2251c329154ac49d6147b1cfd6981b P a8adcce962244bc688dc4ecc1f69612c [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:14.603914 32644 ts_tablet_manager.cc:1434] T d768701c29784c52a2ffb8303021347d P cc8d9fdc73b14a3cb716b47c24bafc1d: Time spent starting tablet: real 0.006s	user 0.004s	sys 0.000s
I20260709 12:28:14.604976 32683 raft_consensus.cc:515] T f5cb9ea668324bfb883f09876ddd56f7 P a8adcce962244bc688dc4ecc1f69612c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:14.608361 32453 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5cb9ea668324bfb883f09876ddd56f7" candidate_uuid: "a8adcce962244bc688dc4ecc1f69612c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d"
I20260709 12:28:14.608471 32683 leader_election.cc:290] T f5cb9ea668324bfb883f09876ddd56f7 P a8adcce962244bc688dc4ecc1f69612c [CANDIDATE]: Term 1 election: Requested vote from peers cc8d9fdc73b14a3cb716b47c24bafc1d (127.16.29.65:41229), f8257d37af0941c2a978546ef2276b30 (127.16.29.67:37155)
I20260709 12:28:14.609007 32453 raft_consensus.cc:3060] T f5cb9ea668324bfb883f09876ddd56f7 P cc8d9fdc73b14a3cb716b47c24bafc1d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:14.610690 32652 raft_consensus.cc:515] T 4b2251c329154ac49d6147b1cfd6981b P a8adcce962244bc688dc4ecc1f69612c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:14.608390 32601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5cb9ea668324bfb883f09876ddd56f7" candidate_uuid: "a8adcce962244bc688dc4ecc1f69612c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8257d37af0941c2a978546ef2276b30"
I20260709 12:28:14.612568 32601 raft_consensus.cc:3060] T f5cb9ea668324bfb883f09876ddd56f7 P f8257d37af0941c2a978546ef2276b30 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:14.613698 32652 leader_election.cc:290] T 4b2251c329154ac49d6147b1cfd6981b P a8adcce962244bc688dc4ecc1f69612c [CANDIDATE]: Term 1 election: Requested vote from peers cc8d9fdc73b14a3cb716b47c24bafc1d (127.16.29.65:41229), f8257d37af0941c2a978546ef2276b30 (127.16.29.67:37155)
I20260709 12:28:14.613870 32452 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b2251c329154ac49d6147b1cfd6981b" candidate_uuid: "a8adcce962244bc688dc4ecc1f69612c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d"
I20260709 12:28:14.614616 32452 raft_consensus.cc:3060] T 4b2251c329154ac49d6147b1cfd6981b P cc8d9fdc73b14a3cb716b47c24bafc1d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:14.613926 32602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b2251c329154ac49d6147b1cfd6981b" candidate_uuid: "a8adcce962244bc688dc4ecc1f69612c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8257d37af0941c2a978546ef2276b30"
I20260709 12:28:14.616145 32602 raft_consensus.cc:3060] T 4b2251c329154ac49d6147b1cfd6981b P f8257d37af0941c2a978546ef2276b30 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:14.620339 32601 raft_consensus.cc:2468] T f5cb9ea668324bfb883f09876ddd56f7 P f8257d37af0941c2a978546ef2276b30 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a8adcce962244bc688dc4ecc1f69612c in term 1.
I20260709 12:28:14.621542 32493 leader_election.cc:304] T f5cb9ea668324bfb883f09876ddd56f7 P a8adcce962244bc688dc4ecc1f69612c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a8adcce962244bc688dc4ecc1f69612c, f8257d37af0941c2a978546ef2276b30; no voters: 
I20260709 12:28:14.622373 32652 raft_consensus.cc:2804] T f5cb9ea668324bfb883f09876ddd56f7 P a8adcce962244bc688dc4ecc1f69612c [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:28:14.622388 32452 raft_consensus.cc:2468] T 4b2251c329154ac49d6147b1cfd6981b P cc8d9fdc73b14a3cb716b47c24bafc1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a8adcce962244bc688dc4ecc1f69612c in term 1.
I20260709 12:28:14.622903 32602 raft_consensus.cc:2468] T 4b2251c329154ac49d6147b1cfd6981b P f8257d37af0941c2a978546ef2276b30 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a8adcce962244bc688dc4ecc1f69612c in term 1.
I20260709 12:28:14.622973 32652 raft_consensus.cc:697] T f5cb9ea668324bfb883f09876ddd56f7 P a8adcce962244bc688dc4ecc1f69612c [term 1 LEADER]: Becoming Leader. State: Replica: a8adcce962244bc688dc4ecc1f69612c, State: Running, Role: LEADER
I20260709 12:28:14.624368 32493 leader_election.cc:304] T 4b2251c329154ac49d6147b1cfd6981b P a8adcce962244bc688dc4ecc1f69612c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a8adcce962244bc688dc4ecc1f69612c, f8257d37af0941c2a978546ef2276b30; no voters: 
I20260709 12:28:14.624145 32652 consensus_queue.cc:237] T f5cb9ea668324bfb883f09876ddd56f7 P a8adcce962244bc688dc4ecc1f69612c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:14.625244 32683 raft_consensus.cc:2804] T 4b2251c329154ac49d6147b1cfd6981b P a8adcce962244bc688dc4ecc1f69612c [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:28:14.626468 32671 raft_consensus.cc:493] T 8e7d5cee9cf64a3e87372bde6431883e P f8257d37af0941c2a978546ef2276b30 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:14.627038 32671 raft_consensus.cc:515] T 8e7d5cee9cf64a3e87372bde6431883e P f8257d37af0941c2a978546ef2276b30 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:14.628706 32453 raft_consensus.cc:2468] T f5cb9ea668324bfb883f09876ddd56f7 P cc8d9fdc73b14a3cb716b47c24bafc1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a8adcce962244bc688dc4ecc1f69612c in term 1.
I20260709 12:28:14.629359 32671 leader_election.cc:290] T 8e7d5cee9cf64a3e87372bde6431883e P f8257d37af0941c2a978546ef2276b30 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc8d9fdc73b14a3cb716b47c24bafc1d (127.16.29.65:41229), a8adcce962244bc688dc4ecc1f69612c (127.16.29.66:33369)
I20260709 12:28:14.630229 32453 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e7d5cee9cf64a3e87372bde6431883e" candidate_uuid: "f8257d37af0941c2a978546ef2276b30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" is_pre_election: true
I20260709 12:28:14.630996 32453 raft_consensus.cc:2468] T 8e7d5cee9cf64a3e87372bde6431883e P cc8d9fdc73b14a3cb716b47c24bafc1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8257d37af0941c2a978546ef2276b30 in term 0.
I20260709 12:28:14.632107 32568 leader_election.cc:304] T 8e7d5cee9cf64a3e87372bde6431883e P f8257d37af0941c2a978546ef2276b30 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cc8d9fdc73b14a3cb716b47c24bafc1d, f8257d37af0941c2a978546ef2276b30; no voters: 
I20260709 12:28:14.632942 32671 raft_consensus.cc:2804] T 8e7d5cee9cf64a3e87372bde6431883e P f8257d37af0941c2a978546ef2276b30 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:28:14.632416 32683 raft_consensus.cc:697] T 4b2251c329154ac49d6147b1cfd6981b P a8adcce962244bc688dc4ecc1f69612c [term 1 LEADER]: Becoming Leader. State: Replica: a8adcce962244bc688dc4ecc1f69612c, State: Running, Role: LEADER
I20260709 12:28:14.633014 32528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e7d5cee9cf64a3e87372bde6431883e" candidate_uuid: "f8257d37af0941c2a978546ef2276b30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8adcce962244bc688dc4ecc1f69612c" is_pre_election: true
I20260709 12:28:14.633338 32671 raft_consensus.cc:493] T 8e7d5cee9cf64a3e87372bde6431883e P f8257d37af0941c2a978546ef2276b30 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:14.633878 32671 raft_consensus.cc:3060] T 8e7d5cee9cf64a3e87372bde6431883e P f8257d37af0941c2a978546ef2276b30 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:14.633932 32528 raft_consensus.cc:2468] T 8e7d5cee9cf64a3e87372bde6431883e P a8adcce962244bc688dc4ecc1f69612c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8257d37af0941c2a978546ef2276b30 in term 0.
I20260709 12:28:14.634194 32683 consensus_queue.cc:237] T 4b2251c329154ac49d6147b1cfd6981b P a8adcce962244bc688dc4ecc1f69612c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:14.635699 32349 catalog_manager.cc:5697] T f5cb9ea668324bfb883f09876ddd56f7 P a8adcce962244bc688dc4ecc1f69612c reported cstate change: term changed from 0 to 1, leader changed from <none> to a8adcce962244bc688dc4ecc1f69612c (127.16.29.66). New cstate: current_term: 1 leader_uuid: "a8adcce962244bc688dc4ecc1f69612c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:14.641112 32671 raft_consensus.cc:515] T 8e7d5cee9cf64a3e87372bde6431883e P f8257d37af0941c2a978546ef2276b30 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:14.644098 32453 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e7d5cee9cf64a3e87372bde6431883e" candidate_uuid: "f8257d37af0941c2a978546ef2276b30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d"
I20260709 12:28:14.644281 32528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e7d5cee9cf64a3e87372bde6431883e" candidate_uuid: "f8257d37af0941c2a978546ef2276b30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8adcce962244bc688dc4ecc1f69612c"
I20260709 12:28:14.644737 32453 raft_consensus.cc:3060] T 8e7d5cee9cf64a3e87372bde6431883e P cc8d9fdc73b14a3cb716b47c24bafc1d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:14.644894 32528 raft_consensus.cc:3060] T 8e7d5cee9cf64a3e87372bde6431883e P a8adcce962244bc688dc4ecc1f69612c [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:14.647934 32671 leader_election.cc:290] T 8e7d5cee9cf64a3e87372bde6431883e P f8257d37af0941c2a978546ef2276b30 [CANDIDATE]: Term 1 election: Requested vote from peers cc8d9fdc73b14a3cb716b47c24bafc1d (127.16.29.65:41229), a8adcce962244bc688dc4ecc1f69612c (127.16.29.66:33369)
I20260709 12:28:14.650666 32528 raft_consensus.cc:2468] T 8e7d5cee9cf64a3e87372bde6431883e P a8adcce962244bc688dc4ecc1f69612c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8257d37af0941c2a978546ef2276b30 in term 1.
I20260709 12:28:14.650776 32349 catalog_manager.cc:5697] T 4b2251c329154ac49d6147b1cfd6981b P a8adcce962244bc688dc4ecc1f69612c reported cstate change: term changed from 0 to 1, leader changed from <none> to a8adcce962244bc688dc4ecc1f69612c (127.16.29.66). New cstate: current_term: 1 leader_uuid: "a8adcce962244bc688dc4ecc1f69612c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:14.651770 32567 leader_election.cc:304] T 8e7d5cee9cf64a3e87372bde6431883e P f8257d37af0941c2a978546ef2276b30 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a8adcce962244bc688dc4ecc1f69612c, f8257d37af0941c2a978546ef2276b30; no voters: 
I20260709 12:28:14.652598 32671 raft_consensus.cc:2804] T 8e7d5cee9cf64a3e87372bde6431883e P f8257d37af0941c2a978546ef2276b30 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:28:14.653301 32671 raft_consensus.cc:697] T 8e7d5cee9cf64a3e87372bde6431883e P f8257d37af0941c2a978546ef2276b30 [term 1 LEADER]: Becoming Leader. State: Replica: f8257d37af0941c2a978546ef2276b30, State: Running, Role: LEADER
I20260709 12:28:14.654062 32671 consensus_queue.cc:237] T 8e7d5cee9cf64a3e87372bde6431883e P f8257d37af0941c2a978546ef2276b30 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:14.654541 32453 raft_consensus.cc:2468] T 8e7d5cee9cf64a3e87372bde6431883e P cc8d9fdc73b14a3cb716b47c24bafc1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8257d37af0941c2a978546ef2276b30 in term 1.
I20260709 12:28:14.662458 32349 catalog_manager.cc:5697] T 8e7d5cee9cf64a3e87372bde6431883e P f8257d37af0941c2a978546ef2276b30 reported cstate change: term changed from 0 to 1, leader changed from <none> to f8257d37af0941c2a978546ef2276b30 (127.16.29.67). New cstate: current_term: 1 leader_uuid: "f8257d37af0941c2a978546ef2276b30" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } health_report { overall_health: HEALTHY } } }
I20260709 12:28:14.677279 32649 consensus_queue.cc:1048] T 50fe031340a94b6590d29d44cd57ca55 P cc8d9fdc73b14a3cb716b47c24bafc1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:28:14.681433 32683 raft_consensus.cc:493] T 9396e8d0ed7b44f58bbc5b421f492f50 P a8adcce962244bc688dc4ecc1f69612c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:14.681890 32683 raft_consensus.cc:515] T 9396e8d0ed7b44f58bbc5b421f492f50 P a8adcce962244bc688dc4ecc1f69612c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:14.683763 32683 leader_election.cc:290] T 9396e8d0ed7b44f58bbc5b421f492f50 P a8adcce962244bc688dc4ecc1f69612c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc8d9fdc73b14a3cb716b47c24bafc1d (127.16.29.65:41229), f8257d37af0941c2a978546ef2276b30 (127.16.29.67:37155)
I20260709 12:28:14.684661 32453 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9396e8d0ed7b44f58bbc5b421f492f50" candidate_uuid: "a8adcce962244bc688dc4ecc1f69612c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" is_pre_election: true
I20260709 12:28:14.685396 32602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9396e8d0ed7b44f58bbc5b421f492f50" candidate_uuid: "a8adcce962244bc688dc4ecc1f69612c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8257d37af0941c2a978546ef2276b30" is_pre_election: true
I20260709 12:28:14.686225 32602 raft_consensus.cc:2468] T 9396e8d0ed7b44f58bbc5b421f492f50 P f8257d37af0941c2a978546ef2276b30 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a8adcce962244bc688dc4ecc1f69612c in term 0.
I20260709 12:28:14.687484 32493 leader_election.cc:304] T 9396e8d0ed7b44f58bbc5b421f492f50 P a8adcce962244bc688dc4ecc1f69612c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a8adcce962244bc688dc4ecc1f69612c, f8257d37af0941c2a978546ef2276b30; no voters: 
I20260709 12:28:14.688623 32683 raft_consensus.cc:2804] T 9396e8d0ed7b44f58bbc5b421f492f50 P a8adcce962244bc688dc4ecc1f69612c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:28:14.689100 32683 raft_consensus.cc:493] T 9396e8d0ed7b44f58bbc5b421f492f50 P a8adcce962244bc688dc4ecc1f69612c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:14.689563 32683 raft_consensus.cc:3060] T 9396e8d0ed7b44f58bbc5b421f492f50 P a8adcce962244bc688dc4ecc1f69612c [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:14.685503 32453 raft_consensus.cc:2468] T 9396e8d0ed7b44f58bbc5b421f492f50 P cc8d9fdc73b14a3cb716b47c24bafc1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a8adcce962244bc688dc4ecc1f69612c in term 0.
I20260709 12:28:14.693593 32662 consensus_queue.cc:1048] T 50fe031340a94b6590d29d44cd57ca55 P cc8d9fdc73b14a3cb716b47c24bafc1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:28:14.699131 32683 raft_consensus.cc:515] T 9396e8d0ed7b44f58bbc5b421f492f50 P a8adcce962244bc688dc4ecc1f69612c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:14.702656 32453 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9396e8d0ed7b44f58bbc5b421f492f50" candidate_uuid: "a8adcce962244bc688dc4ecc1f69612c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d"
I20260709 12:28:14.703512 32453 raft_consensus.cc:3060] T 9396e8d0ed7b44f58bbc5b421f492f50 P cc8d9fdc73b14a3cb716b47c24bafc1d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:14.706576 32683 leader_election.cc:290] T 9396e8d0ed7b44f58bbc5b421f492f50 P a8adcce962244bc688dc4ecc1f69612c [CANDIDATE]: Term 1 election: Requested vote from peers cc8d9fdc73b14a3cb716b47c24bafc1d (127.16.29.65:41229), f8257d37af0941c2a978546ef2276b30 (127.16.29.67:37155)
I20260709 12:28:14.707646 32602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9396e8d0ed7b44f58bbc5b421f492f50" candidate_uuid: "a8adcce962244bc688dc4ecc1f69612c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8257d37af0941c2a978546ef2276b30"
I20260709 12:28:14.708235 32602 raft_consensus.cc:3060] T 9396e8d0ed7b44f58bbc5b421f492f50 P f8257d37af0941c2a978546ef2276b30 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:14.713013 32453 raft_consensus.cc:2468] T 9396e8d0ed7b44f58bbc5b421f492f50 P cc8d9fdc73b14a3cb716b47c24bafc1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a8adcce962244bc688dc4ecc1f69612c in term 1.
I20260709 12:28:14.714525 32494 leader_election.cc:304] T 9396e8d0ed7b44f58bbc5b421f492f50 P a8adcce962244bc688dc4ecc1f69612c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a8adcce962244bc688dc4ecc1f69612c, cc8d9fdc73b14a3cb716b47c24bafc1d; no voters: 
I20260709 12:28:14.715534 32683 raft_consensus.cc:2804] T 9396e8d0ed7b44f58bbc5b421f492f50 P a8adcce962244bc688dc4ecc1f69612c [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:28:14.716324 32683 raft_consensus.cc:697] T 9396e8d0ed7b44f58bbc5b421f492f50 P a8adcce962244bc688dc4ecc1f69612c [term 1 LEADER]: Becoming Leader. State: Replica: a8adcce962244bc688dc4ecc1f69612c, State: Running, Role: LEADER
I20260709 12:28:14.716562 32602 raft_consensus.cc:2468] T 9396e8d0ed7b44f58bbc5b421f492f50 P f8257d37af0941c2a978546ef2276b30 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a8adcce962244bc688dc4ecc1f69612c in term 1.
I20260709 12:28:14.717188 32683 consensus_queue.cc:237] T 9396e8d0ed7b44f58bbc5b421f492f50 P a8adcce962244bc688dc4ecc1f69612c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:14.725458 32348 catalog_manager.cc:5697] T 9396e8d0ed7b44f58bbc5b421f492f50 P a8adcce962244bc688dc4ecc1f69612c reported cstate change: term changed from 0 to 1, leader changed from <none> to a8adcce962244bc688dc4ecc1f69612c (127.16.29.66). New cstate: current_term: 1 leader_uuid: "a8adcce962244bc688dc4ecc1f69612c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:14.773547 32683 raft_consensus.cc:493] T f22ce6f243334c2281f0543e605f614a P a8adcce962244bc688dc4ecc1f69612c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:14.774008 32683 raft_consensus.cc:515] T f22ce6f243334c2281f0543e605f614a P a8adcce962244bc688dc4ecc1f69612c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:14.775743 32683 leader_election.cc:290] T f22ce6f243334c2281f0543e605f614a P a8adcce962244bc688dc4ecc1f69612c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc8d9fdc73b14a3cb716b47c24bafc1d (127.16.29.65:41229), f8257d37af0941c2a978546ef2276b30 (127.16.29.67:37155)
I20260709 12:28:14.776803 32602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f22ce6f243334c2281f0543e605f614a" candidate_uuid: "a8adcce962244bc688dc4ecc1f69612c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8257d37af0941c2a978546ef2276b30" is_pre_election: true
I20260709 12:28:14.776747 32453 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f22ce6f243334c2281f0543e605f614a" candidate_uuid: "a8adcce962244bc688dc4ecc1f69612c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" is_pre_election: true
I20260709 12:28:14.777812 32602 raft_consensus.cc:2393] T f22ce6f243334c2281f0543e605f614a P f8257d37af0941c2a978546ef2276b30 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a8adcce962244bc688dc4ecc1f69612c in current term 1: Already voted for candidate cc8d9fdc73b14a3cb716b47c24bafc1d in this term.
I20260709 12:28:14.778774 32493 leader_election.cc:304] T f22ce6f243334c2281f0543e605f614a P a8adcce962244bc688dc4ecc1f69612c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a8adcce962244bc688dc4ecc1f69612c; no voters: cc8d9fdc73b14a3cb716b47c24bafc1d, f8257d37af0941c2a978546ef2276b30
I20260709 12:28:14.779395 32683 raft_consensus.cc:3060] T f22ce6f243334c2281f0543e605f614a P a8adcce962244bc688dc4ecc1f69612c [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:14.784911 32683 raft_consensus.cc:2749] T f22ce6f243334c2281f0543e605f614a P a8adcce962244bc688dc4ecc1f69612c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:28:14.830372 32683 raft_consensus.cc:493] T d768701c29784c52a2ffb8303021347d P a8adcce962244bc688dc4ecc1f69612c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:14.831040 32683 raft_consensus.cc:515] T d768701c29784c52a2ffb8303021347d P a8adcce962244bc688dc4ecc1f69612c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:14.833047 32683 leader_election.cc:290] T d768701c29784c52a2ffb8303021347d P a8adcce962244bc688dc4ecc1f69612c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc8d9fdc73b14a3cb716b47c24bafc1d (127.16.29.65:41229), f8257d37af0941c2a978546ef2276b30 (127.16.29.67:37155)
I20260709 12:28:14.834131 32602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d768701c29784c52a2ffb8303021347d" candidate_uuid: "a8adcce962244bc688dc4ecc1f69612c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8257d37af0941c2a978546ef2276b30" is_pre_election: true
I20260709 12:28:14.834162 32453 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d768701c29784c52a2ffb8303021347d" candidate_uuid: "a8adcce962244bc688dc4ecc1f69612c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" is_pre_election: true
I20260709 12:28:14.835193 32602 raft_consensus.cc:2468] T d768701c29784c52a2ffb8303021347d P f8257d37af0941c2a978546ef2276b30 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a8adcce962244bc688dc4ecc1f69612c in term 0.
I20260709 12:28:14.835261 32453 raft_consensus.cc:2468] T d768701c29784c52a2ffb8303021347d P cc8d9fdc73b14a3cb716b47c24bafc1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a8adcce962244bc688dc4ecc1f69612c in term 0.
I20260709 12:28:14.836297 32493 leader_election.cc:304] T d768701c29784c52a2ffb8303021347d P a8adcce962244bc688dc4ecc1f69612c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a8adcce962244bc688dc4ecc1f69612c, f8257d37af0941c2a978546ef2276b30; no voters: 
I20260709 12:28:14.837179 32683 raft_consensus.cc:2804] T d768701c29784c52a2ffb8303021347d P a8adcce962244bc688dc4ecc1f69612c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:28:14.837487 32683 raft_consensus.cc:493] T d768701c29784c52a2ffb8303021347d P a8adcce962244bc688dc4ecc1f69612c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:14.837711 32683 raft_consensus.cc:3060] T d768701c29784c52a2ffb8303021347d P a8adcce962244bc688dc4ecc1f69612c [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:14.843092 32683 raft_consensus.cc:515] T d768701c29784c52a2ffb8303021347d P a8adcce962244bc688dc4ecc1f69612c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:14.844720 32683 leader_election.cc:290] T d768701c29784c52a2ffb8303021347d P a8adcce962244bc688dc4ecc1f69612c [CANDIDATE]: Term 1 election: Requested vote from peers cc8d9fdc73b14a3cb716b47c24bafc1d (127.16.29.65:41229), f8257d37af0941c2a978546ef2276b30 (127.16.29.67:37155)
I20260709 12:28:14.845614 32453 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d768701c29784c52a2ffb8303021347d" candidate_uuid: "a8adcce962244bc688dc4ecc1f69612c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d"
I20260709 12:28:14.845880 32602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d768701c29784c52a2ffb8303021347d" candidate_uuid: "a8adcce962244bc688dc4ecc1f69612c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8257d37af0941c2a978546ef2276b30"
I20260709 12:28:14.846316 32453 raft_consensus.cc:3060] T d768701c29784c52a2ffb8303021347d P cc8d9fdc73b14a3cb716b47c24bafc1d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:14.846424 32602 raft_consensus.cc:3060] T d768701c29784c52a2ffb8303021347d P f8257d37af0941c2a978546ef2276b30 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:14.850713 32662 consensus_queue.cc:1048] T f22ce6f243334c2281f0543e605f614a P cc8d9fdc73b14a3cb716b47c24bafc1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:28:14.854640 32453 raft_consensus.cc:2468] T d768701c29784c52a2ffb8303021347d P cc8d9fdc73b14a3cb716b47c24bafc1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a8adcce962244bc688dc4ecc1f69612c in term 1.
I20260709 12:28:14.855175 32602 raft_consensus.cc:2468] T d768701c29784c52a2ffb8303021347d P f8257d37af0941c2a978546ef2276b30 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a8adcce962244bc688dc4ecc1f69612c in term 1.
I20260709 12:28:14.856069 32494 leader_election.cc:304] T d768701c29784c52a2ffb8303021347d P a8adcce962244bc688dc4ecc1f69612c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a8adcce962244bc688dc4ecc1f69612c, cc8d9fdc73b14a3cb716b47c24bafc1d; no voters: 
I20260709 12:28:14.856918 32683 raft_consensus.cc:2804] T d768701c29784c52a2ffb8303021347d P a8adcce962244bc688dc4ecc1f69612c [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:28:14.857394 32683 raft_consensus.cc:697] T d768701c29784c52a2ffb8303021347d P a8adcce962244bc688dc4ecc1f69612c [term 1 LEADER]: Becoming Leader. State: Replica: a8adcce962244bc688dc4ecc1f69612c, State: Running, Role: LEADER
I20260709 12:28:14.858206 32683 consensus_queue.cc:237] T d768701c29784c52a2ffb8303021347d P a8adcce962244bc688dc4ecc1f69612c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:14.864462 32661 consensus_queue.cc:1048] T f22ce6f243334c2281f0543e605f614a P cc8d9fdc73b14a3cb716b47c24bafc1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:28:14.880259 32349 catalog_manager.cc:5697] T d768701c29784c52a2ffb8303021347d P a8adcce962244bc688dc4ecc1f69612c reported cstate change: term changed from 0 to 1, leader changed from <none> to a8adcce962244bc688dc4ecc1f69612c (127.16.29.66). New cstate: current_term: 1 leader_uuid: "a8adcce962244bc688dc4ecc1f69612c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:14.907497 32683 consensus_queue.cc:1048] T c127bfae907d4960956f04576bf0c6c7 P a8adcce962244bc688dc4ecc1f69612c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:28:14.920606 32652 consensus_queue.cc:1048] T c127bfae907d4960956f04576bf0c6c7 P a8adcce962244bc688dc4ecc1f69612c [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:28:14.948690 32528 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "f5cb9ea668324bfb883f09876ddd56f7"
dest_uuid: "a8adcce962244bc688dc4ecc1f69612c"
mode: GRACEFUL
new_leader_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d"
 from {username='slave'} at 127.0.0.1:56052
I20260709 12:28:14.949410 32528 raft_consensus.cc:606] T f5cb9ea668324bfb883f09876ddd56f7 P a8adcce962244bc688dc4ecc1f69612c [term 1 LEADER]: Received request to transfer leadership to TS cc8d9fdc73b14a3cb716b47c24bafc1d
I20260709 12:28:14.949872 32649 consensus_queue.cc:1048] T aa861e9a505d425792eb54a2a29cab97 P cc8d9fdc73b14a3cb716b47c24bafc1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:28:14.962968 32649 consensus_queue.cc:1048] T aa861e9a505d425792eb54a2a29cab97 P cc8d9fdc73b14a3cb716b47c24bafc1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.006s
I20260709 12:28:14.980579 32602 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "8e7d5cee9cf64a3e87372bde6431883e"
dest_uuid: "f8257d37af0941c2a978546ef2276b30"
mode: GRACEFUL
new_leader_uuid: "a8adcce962244bc688dc4ecc1f69612c"
 from {username='slave'} at 127.0.0.1:42876
I20260709 12:28:14.981225 32602 raft_consensus.cc:606] T 8e7d5cee9cf64a3e87372bde6431883e P f8257d37af0941c2a978546ef2276b30 [term 1 LEADER]: Received request to transfer leadership to TS a8adcce962244bc688dc4ecc1f69612c
I20260709 12:28:14.997711 32528 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "c127bfae907d4960956f04576bf0c6c7"
dest_uuid: "a8adcce962244bc688dc4ecc1f69612c"
mode: GRACEFUL
new_leader_uuid: "f8257d37af0941c2a978546ef2276b30"
 from {username='slave'} at 127.0.0.1:56052
I20260709 12:28:14.998338 32528 raft_consensus.cc:606] T c127bfae907d4960956f04576bf0c6c7 P a8adcce962244bc688dc4ecc1f69612c [term 1 LEADER]: Received request to transfer leadership to TS f8257d37af0941c2a978546ef2276b30
I20260709 12:28:15.034540 32707 consensus_queue.cc:1048] T 4b2251c329154ac49d6147b1cfd6981b P a8adcce962244bc688dc4ecc1f69612c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:28:15.046006 32707 consensus_queue.cc:1048] T 4b2251c329154ac49d6147b1cfd6981b P a8adcce962244bc688dc4ecc1f69612c [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:28:15.054342 32671 consensus_queue.cc:1048] T 8e7d5cee9cf64a3e87372bde6431883e P f8257d37af0941c2a978546ef2276b30 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:28:15.065433 32671 consensus_queue.cc:1048] T 8e7d5cee9cf64a3e87372bde6431883e P f8257d37af0941c2a978546ef2276b30 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:28:15.073300 32671 raft_consensus.cc:993] T 8e7d5cee9cf64a3e87372bde6431883e P f8257d37af0941c2a978546ef2276b30: : Instructing follower a8adcce962244bc688dc4ecc1f69612c to start an election
I20260709 12:28:15.073818 32717 raft_consensus.cc:1081] T 8e7d5cee9cf64a3e87372bde6431883e P f8257d37af0941c2a978546ef2276b30 [term 1 LEADER]: Signalling peer a8adcce962244bc688dc4ecc1f69612c to start an election
I20260709 12:28:15.075361 32528 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "8e7d5cee9cf64a3e87372bde6431883e"
dest_uuid: "a8adcce962244bc688dc4ecc1f69612c"
 from {username='slave'} at 127.0.0.1:56044
I20260709 12:28:15.076010 32528 raft_consensus.cc:493] T 8e7d5cee9cf64a3e87372bde6431883e P a8adcce962244bc688dc4ecc1f69612c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:28:15.076474 32528 raft_consensus.cc:3060] T 8e7d5cee9cf64a3e87372bde6431883e P a8adcce962244bc688dc4ecc1f69612c [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:15.083127 32528 raft_consensus.cc:515] T 8e7d5cee9cf64a3e87372bde6431883e P a8adcce962244bc688dc4ecc1f69612c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:15.085522 32528 leader_election.cc:290] T 8e7d5cee9cf64a3e87372bde6431883e P a8adcce962244bc688dc4ecc1f69612c [CANDIDATE]: Term 2 election: Requested vote from peers cc8d9fdc73b14a3cb716b47c24bafc1d (127.16.29.65:41229), f8257d37af0941c2a978546ef2276b30 (127.16.29.67:37155)
I20260709 12:28:15.085845 32453 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e7d5cee9cf64a3e87372bde6431883e" candidate_uuid: "a8adcce962244bc688dc4ecc1f69612c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d"
I20260709 12:28:15.086174 32602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e7d5cee9cf64a3e87372bde6431883e" candidate_uuid: "a8adcce962244bc688dc4ecc1f69612c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f8257d37af0941c2a978546ef2276b30"
I20260709 12:28:15.086542 32453 raft_consensus.cc:3060] T 8e7d5cee9cf64a3e87372bde6431883e P cc8d9fdc73b14a3cb716b47c24bafc1d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:15.086913 32602 raft_consensus.cc:3055] T 8e7d5cee9cf64a3e87372bde6431883e P f8257d37af0941c2a978546ef2276b30 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:28:15.087218 32602 raft_consensus.cc:740] T 8e7d5cee9cf64a3e87372bde6431883e P f8257d37af0941c2a978546ef2276b30 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f8257d37af0941c2a978546ef2276b30, State: Running, Role: LEADER
I20260709 12:28:15.087977 32602 consensus_queue.cc:260] T 8e7d5cee9cf64a3e87372bde6431883e P f8257d37af0941c2a978546ef2276b30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:15.089030 32602 raft_consensus.cc:3060] T 8e7d5cee9cf64a3e87372bde6431883e P f8257d37af0941c2a978546ef2276b30 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:15.092902 32453 raft_consensus.cc:2468] T 8e7d5cee9cf64a3e87372bde6431883e P cc8d9fdc73b14a3cb716b47c24bafc1d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a8adcce962244bc688dc4ecc1f69612c in term 2.
I20260709 12:28:15.094278 32494 leader_election.cc:304] T 8e7d5cee9cf64a3e87372bde6431883e P a8adcce962244bc688dc4ecc1f69612c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a8adcce962244bc688dc4ecc1f69612c, cc8d9fdc73b14a3cb716b47c24bafc1d; no voters: 
I20260709 12:28:15.095503 32707 raft_consensus.cc:2804] T 8e7d5cee9cf64a3e87372bde6431883e P a8adcce962244bc688dc4ecc1f69612c [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:28:15.096060 32707 raft_consensus.cc:697] T 8e7d5cee9cf64a3e87372bde6431883e P a8adcce962244bc688dc4ecc1f69612c [term 2 LEADER]: Becoming Leader. State: Replica: a8adcce962244bc688dc4ecc1f69612c, State: Running, Role: LEADER
I20260709 12:28:15.096637 32602 raft_consensus.cc:2468] T 8e7d5cee9cf64a3e87372bde6431883e P f8257d37af0941c2a978546ef2276b30 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a8adcce962244bc688dc4ecc1f69612c in term 2.
I20260709 12:28:15.097050 32707 consensus_queue.cc:237] T 8e7d5cee9cf64a3e87372bde6431883e P a8adcce962244bc688dc4ecc1f69612c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:15.105964 32348 catalog_manager.cc:5697] T 8e7d5cee9cf64a3e87372bde6431883e P a8adcce962244bc688dc4ecc1f69612c reported cstate change: term changed from 1 to 2, leader changed from f8257d37af0941c2a978546ef2276b30 (127.16.29.67) to a8adcce962244bc688dc4ecc1f69612c (127.16.29.66). New cstate: current_term: 2 leader_uuid: "a8adcce962244bc688dc4ecc1f69612c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:15.119829 32707 consensus_queue.cc:1048] T 9396e8d0ed7b44f58bbc5b421f492f50 P a8adcce962244bc688dc4ecc1f69612c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:28:15.131263 32652 consensus_queue.cc:1048] T 9396e8d0ed7b44f58bbc5b421f492f50 P a8adcce962244bc688dc4ecc1f69612c [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:28:15.191313 32652 consensus_queue.cc:1048] T f5cb9ea668324bfb883f09876ddd56f7 P a8adcce962244bc688dc4ecc1f69612c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:28:15.201623 32683 consensus_queue.cc:1048] T f5cb9ea668324bfb883f09876ddd56f7 P a8adcce962244bc688dc4ecc1f69612c [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:28:15.209687 32707 raft_consensus.cc:993] T f5cb9ea668324bfb883f09876ddd56f7 P a8adcce962244bc688dc4ecc1f69612c: : Instructing follower cc8d9fdc73b14a3cb716b47c24bafc1d to start an election
I20260709 12:28:15.210268 32683 raft_consensus.cc:1081] T f5cb9ea668324bfb883f09876ddd56f7 P a8adcce962244bc688dc4ecc1f69612c [term 1 LEADER]: Signalling peer cc8d9fdc73b14a3cb716b47c24bafc1d to start an election
I20260709 12:28:15.211876 32453 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "f5cb9ea668324bfb883f09876ddd56f7"
dest_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d"
 from {username='slave'} at 127.0.0.1:45590
I20260709 12:28:15.212532 32453 raft_consensus.cc:493] T f5cb9ea668324bfb883f09876ddd56f7 P cc8d9fdc73b14a3cb716b47c24bafc1d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:28:15.212905 32453 raft_consensus.cc:3060] T f5cb9ea668324bfb883f09876ddd56f7 P cc8d9fdc73b14a3cb716b47c24bafc1d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:15.218370 32453 raft_consensus.cc:515] T f5cb9ea668324bfb883f09876ddd56f7 P cc8d9fdc73b14a3cb716b47c24bafc1d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:15.220475 32453 leader_election.cc:290] T f5cb9ea668324bfb883f09876ddd56f7 P cc8d9fdc73b14a3cb716b47c24bafc1d [CANDIDATE]: Term 2 election: Requested vote from peers a8adcce962244bc688dc4ecc1f69612c (127.16.29.66:33369), f8257d37af0941c2a978546ef2276b30 (127.16.29.67:37155)
I20260709 12:28:15.221199 32528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5cb9ea668324bfb883f09876ddd56f7" candidate_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a8adcce962244bc688dc4ecc1f69612c"
I20260709 12:28:15.221686 32601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5cb9ea668324bfb883f09876ddd56f7" candidate_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f8257d37af0941c2a978546ef2276b30"
I20260709 12:28:15.221865 32528 raft_consensus.cc:3055] T f5cb9ea668324bfb883f09876ddd56f7 P a8adcce962244bc688dc4ecc1f69612c [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:28:15.222283 32601 raft_consensus.cc:3060] T f5cb9ea668324bfb883f09876ddd56f7 P f8257d37af0941c2a978546ef2276b30 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:15.222379 32528 raft_consensus.cc:740] T f5cb9ea668324bfb883f09876ddd56f7 P a8adcce962244bc688dc4ecc1f69612c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a8adcce962244bc688dc4ecc1f69612c, State: Running, Role: LEADER
I20260709 12:28:15.223456 32528 consensus_queue.cc:260] T f5cb9ea668324bfb883f09876ddd56f7 P a8adcce962244bc688dc4ecc1f69612c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:15.224846 32528 raft_consensus.cc:3060] T f5cb9ea668324bfb883f09876ddd56f7 P a8adcce962244bc688dc4ecc1f69612c [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:15.228494 32601 raft_consensus.cc:2468] T f5cb9ea668324bfb883f09876ddd56f7 P f8257d37af0941c2a978546ef2276b30 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc8d9fdc73b14a3cb716b47c24bafc1d in term 2.
I20260709 12:28:15.229802 32418 leader_election.cc:304] T f5cb9ea668324bfb883f09876ddd56f7 P cc8d9fdc73b14a3cb716b47c24bafc1d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cc8d9fdc73b14a3cb716b47c24bafc1d, f8257d37af0941c2a978546ef2276b30; no voters: 
I20260709 12:28:15.230322 32528 raft_consensus.cc:2468] T f5cb9ea668324bfb883f09876ddd56f7 P a8adcce962244bc688dc4ecc1f69612c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc8d9fdc73b14a3cb716b47c24bafc1d in term 2.
I20260709 12:28:15.230644 32649 raft_consensus.cc:2804] T f5cb9ea668324bfb883f09876ddd56f7 P cc8d9fdc73b14a3cb716b47c24bafc1d [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:28:15.231233 32649 raft_consensus.cc:697] T f5cb9ea668324bfb883f09876ddd56f7 P cc8d9fdc73b14a3cb716b47c24bafc1d [term 2 LEADER]: Becoming Leader. State: Replica: cc8d9fdc73b14a3cb716b47c24bafc1d, State: Running, Role: LEADER
I20260709 12:28:15.232120 32649 consensus_queue.cc:237] T f5cb9ea668324bfb883f09876ddd56f7 P cc8d9fdc73b14a3cb716b47c24bafc1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:15.238904 32349 catalog_manager.cc:5697] T f5cb9ea668324bfb883f09876ddd56f7 P cc8d9fdc73b14a3cb716b47c24bafc1d reported cstate change: term changed from 1 to 2, leader changed from a8adcce962244bc688dc4ecc1f69612c (127.16.29.66) to cc8d9fdc73b14a3cb716b47c24bafc1d (127.16.29.65). New cstate: current_term: 2 leader_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:15.252648 32683 consensus_queue.cc:1048] T d768701c29784c52a2ffb8303021347d P a8adcce962244bc688dc4ecc1f69612c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:28:15.263316 32707 consensus_queue.cc:1048] T d768701c29784c52a2ffb8303021347d P a8adcce962244bc688dc4ecc1f69612c [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:28:15.435678 32652 raft_consensus.cc:993] T c127bfae907d4960956f04576bf0c6c7 P a8adcce962244bc688dc4ecc1f69612c: : Instructing follower f8257d37af0941c2a978546ef2276b30 to start an election
I20260709 12:28:15.436214 32707 raft_consensus.cc:1081] T c127bfae907d4960956f04576bf0c6c7 P a8adcce962244bc688dc4ecc1f69612c [term 1 LEADER]: Signalling peer f8257d37af0941c2a978546ef2276b30 to start an election
I20260709 12:28:15.437660 32601 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "c127bfae907d4960956f04576bf0c6c7"
dest_uuid: "f8257d37af0941c2a978546ef2276b30"
 from {username='slave'} at 127.0.0.1:42862
I20260709 12:28:15.438145 32601 raft_consensus.cc:493] T c127bfae907d4960956f04576bf0c6c7 P f8257d37af0941c2a978546ef2276b30 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:28:15.438444 32601 raft_consensus.cc:3060] T c127bfae907d4960956f04576bf0c6c7 P f8257d37af0941c2a978546ef2276b30 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:15.443567 32601 raft_consensus.cc:515] T c127bfae907d4960956f04576bf0c6c7 P f8257d37af0941c2a978546ef2276b30 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:15.445305 32601 leader_election.cc:290] T c127bfae907d4960956f04576bf0c6c7 P f8257d37af0941c2a978546ef2276b30 [CANDIDATE]: Term 2 election: Requested vote from peers cc8d9fdc73b14a3cb716b47c24bafc1d (127.16.29.65:41229), a8adcce962244bc688dc4ecc1f69612c (127.16.29.66:33369)
I20260709 12:28:15.446286 32528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c127bfae907d4960956f04576bf0c6c7" candidate_uuid: "f8257d37af0941c2a978546ef2276b30" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a8adcce962244bc688dc4ecc1f69612c"
I20260709 12:28:15.446817 32528 raft_consensus.cc:3055] T c127bfae907d4960956f04576bf0c6c7 P a8adcce962244bc688dc4ecc1f69612c [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:28:15.447090 32528 raft_consensus.cc:740] T c127bfae907d4960956f04576bf0c6c7 P a8adcce962244bc688dc4ecc1f69612c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a8adcce962244bc688dc4ecc1f69612c, State: Running, Role: LEADER
I20260709 12:28:15.446905 32453 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c127bfae907d4960956f04576bf0c6c7" candidate_uuid: "f8257d37af0941c2a978546ef2276b30" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d"
I20260709 12:28:15.447693 32453 raft_consensus.cc:3060] T c127bfae907d4960956f04576bf0c6c7 P cc8d9fdc73b14a3cb716b47c24bafc1d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:15.448062 32528 consensus_queue.cc:260] T c127bfae907d4960956f04576bf0c6c7 P a8adcce962244bc688dc4ecc1f69612c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:15.449486 32528 raft_consensus.cc:3060] T c127bfae907d4960956f04576bf0c6c7 P a8adcce962244bc688dc4ecc1f69612c [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:15.453809 32453 raft_consensus.cc:2468] T c127bfae907d4960956f04576bf0c6c7 P cc8d9fdc73b14a3cb716b47c24bafc1d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8257d37af0941c2a978546ef2276b30 in term 2.
I20260709 12:28:15.454857 32528 raft_consensus.cc:2468] T c127bfae907d4960956f04576bf0c6c7 P a8adcce962244bc688dc4ecc1f69612c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8257d37af0941c2a978546ef2276b30 in term 2.
I20260709 12:28:15.454985 32568 leader_election.cc:304] T c127bfae907d4960956f04576bf0c6c7 P f8257d37af0941c2a978546ef2276b30 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cc8d9fdc73b14a3cb716b47c24bafc1d, f8257d37af0941c2a978546ef2276b30; no voters: 
I20260709 12:28:15.455813 32717 raft_consensus.cc:2804] T c127bfae907d4960956f04576bf0c6c7 P f8257d37af0941c2a978546ef2276b30 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:28:15.456296 32717 raft_consensus.cc:697] T c127bfae907d4960956f04576bf0c6c7 P f8257d37af0941c2a978546ef2276b30 [term 2 LEADER]: Becoming Leader. State: Replica: f8257d37af0941c2a978546ef2276b30, State: Running, Role: LEADER
I20260709 12:28:15.457024 32717 consensus_queue.cc:237] T c127bfae907d4960956f04576bf0c6c7 P f8257d37af0941c2a978546ef2276b30 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:15.463080 32348 catalog_manager.cc:5697] T c127bfae907d4960956f04576bf0c6c7 P f8257d37af0941c2a978546ef2276b30 reported cstate change: term changed from 1 to 2, leader changed from a8adcce962244bc688dc4ecc1f69612c (127.16.29.66) to f8257d37af0941c2a978546ef2276b30 (127.16.29.67). New cstate: current_term: 2 leader_uuid: "f8257d37af0941c2a978546ef2276b30" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } health_report { overall_health: HEALTHY } } }
I20260709 12:28:15.527064 32453 raft_consensus.cc:1275] T 8e7d5cee9cf64a3e87372bde6431883e P cc8d9fdc73b14a3cb716b47c24bafc1d [term 2 FOLLOWER]: Refusing update from remote peer a8adcce962244bc688dc4ecc1f69612c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:15.528456 32692 consensus_queue.cc:1048] T 8e7d5cee9cf64a3e87372bde6431883e P a8adcce962244bc688dc4ecc1f69612c [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:28:15.536219 32601 raft_consensus.cc:1275] T 8e7d5cee9cf64a3e87372bde6431883e P f8257d37af0941c2a978546ef2276b30 [term 2 FOLLOWER]: Refusing update from remote peer a8adcce962244bc688dc4ecc1f69612c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:15.537797 32692 consensus_queue.cc:1048] T 8e7d5cee9cf64a3e87372bde6431883e P a8adcce962244bc688dc4ecc1f69612c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:28:15.638327 32601 raft_consensus.cc:1275] T f5cb9ea668324bfb883f09876ddd56f7 P f8257d37af0941c2a978546ef2276b30 [term 2 FOLLOWER]: Refusing update from remote peer cc8d9fdc73b14a3cb716b47c24bafc1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:15.639693 32649 consensus_queue.cc:1048] T f5cb9ea668324bfb883f09876ddd56f7 P cc8d9fdc73b14a3cb716b47c24bafc1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:28:15.647370 32528 raft_consensus.cc:1275] T f5cb9ea668324bfb883f09876ddd56f7 P a8adcce962244bc688dc4ecc1f69612c [term 2 FOLLOWER]: Refusing update from remote peer cc8d9fdc73b14a3cb716b47c24bafc1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:15.648945 32725 consensus_queue.cc:1048] T f5cb9ea668324bfb883f09876ddd56f7 P cc8d9fdc73b14a3cb716b47c24bafc1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:28:15.889662 32453 raft_consensus.cc:1275] T c127bfae907d4960956f04576bf0c6c7 P cc8d9fdc73b14a3cb716b47c24bafc1d [term 2 FOLLOWER]: Refusing update from remote peer f8257d37af0941c2a978546ef2276b30: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:15.891263 32717 consensus_queue.cc:1048] T c127bfae907d4960956f04576bf0c6c7 P f8257d37af0941c2a978546ef2276b30 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:28:15.902583 32528 raft_consensus.cc:1275] T c127bfae907d4960956f04576bf0c6c7 P a8adcce962244bc688dc4ecc1f69612c [term 2 FOLLOWER]: Refusing update from remote peer f8257d37af0941c2a978546ef2276b30: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:15.904752 32717 consensus_queue.cc:1048] T c127bfae907d4960956f04576bf0c6c7 P f8257d37af0941c2a978546ef2276b30 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:28:18.031778 32348 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34582:
name: "soft_deleted_table"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
  rows: "\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q""\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q"
  indirect_data: """"
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W20260709 12:28:18.034206 32348 catalog_manager.cc:7077] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table soft_deleted_table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260709 12:28:18.084227 32436 tablet_service.cc:1511] Processing CreateTablet for tablet 51c3eb697520458093debb61eebb75a7 (DEFAULT_TABLE table=soft_deleted_table [id=06201339da074254986a87154a02a188]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260709 12:28:18.085511 32436 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 51c3eb697520458093debb61eebb75a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:18.085297 32517 tablet_service.cc:1511] Processing CreateTablet for tablet 51c3eb697520458093debb61eebb75a7 (DEFAULT_TABLE table=soft_deleted_table [id=06201339da074254986a87154a02a188]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260709 12:28:18.086700 32517 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 51c3eb697520458093debb61eebb75a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:18.087112 32588 tablet_service.cc:1511] Processing CreateTablet for tablet 51c3eb697520458093debb61eebb75a7 (DEFAULT_TABLE table=soft_deleted_table [id=06201339da074254986a87154a02a188]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260709 12:28:18.089334 32588 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 51c3eb697520458093debb61eebb75a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:18.089949 32437 tablet_service.cc:1511] Processing CreateTablet for tablet db65db1003af45d4b6315902498f3f2b (DEFAULT_TABLE table=soft_deleted_table [id=06201339da074254986a87154a02a188]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260709 12:28:18.091317 32437 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet db65db1003af45d4b6315902498f3f2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:18.092976 32591 tablet_service.cc:1511] Processing CreateTablet for tablet db65db1003af45d4b6315902498f3f2b (DEFAULT_TABLE table=soft_deleted_table [id=06201339da074254986a87154a02a188]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260709 12:28:18.094326 32591 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet db65db1003af45d4b6315902498f3f2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:18.094702 32511 tablet_service.cc:1511] Processing CreateTablet for tablet db65db1003af45d4b6315902498f3f2b (DEFAULT_TABLE table=soft_deleted_table [id=06201339da074254986a87154a02a188]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260709 12:28:18.096083 32511 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet db65db1003af45d4b6315902498f3f2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:18.095938 32586 tablet_service.cc:1511] Processing CreateTablet for tablet 92a7506e828b47a28f4e2c080d05419b (DEFAULT_TABLE table=soft_deleted_table [id=06201339da074254986a87154a02a188]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260709 12:28:18.097281 32586 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 92a7506e828b47a28f4e2c080d05419b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:18.101056 32515 tablet_service.cc:1511] Processing CreateTablet for tablet 92a7506e828b47a28f4e2c080d05419b (DEFAULT_TABLE table=soft_deleted_table [id=06201339da074254986a87154a02a188]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260709 12:28:18.102519 32515 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 92a7506e828b47a28f4e2c080d05419b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:18.107012 32438 tablet_service.cc:1511] Processing CreateTablet for tablet 92a7506e828b47a28f4e2c080d05419b (DEFAULT_TABLE table=soft_deleted_table [id=06201339da074254986a87154a02a188]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260709 12:28:18.108314 32440 tablet_service.cc:1511] Processing CreateTablet for tablet 276dee5db2d34451b642ed9359137ad4 (DEFAULT_TABLE table=soft_deleted_table [id=06201339da074254986a87154a02a188]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260709 12:28:18.109298 32438 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 92a7506e828b47a28f4e2c080d05419b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:18.110460 32514 tablet_service.cc:1511] Processing CreateTablet for tablet 276dee5db2d34451b642ed9359137ad4 (DEFAULT_TABLE table=soft_deleted_table [id=06201339da074254986a87154a02a188]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260709 12:28:18.112003 32514 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 276dee5db2d34451b642ed9359137ad4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:18.112270 32440 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 276dee5db2d34451b642ed9359137ad4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:18.116192 32592 tablet_service.cc:1511] Processing CreateTablet for tablet 276dee5db2d34451b642ed9359137ad4 (DEFAULT_TABLE table=soft_deleted_table [id=06201339da074254986a87154a02a188]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260709 12:28:18.118346 32592 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 276dee5db2d34451b642ed9359137ad4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:18.122495 32442 tablet_service.cc:1511] Processing CreateTablet for tablet 7035e41df60347b188a3b8079aa7e551 (DEFAULT_TABLE table=soft_deleted_table [id=06201339da074254986a87154a02a188]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260709 12:28:18.124125 32442 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7035e41df60347b188a3b8079aa7e551. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:18.127063 32516 tablet_service.cc:1511] Processing CreateTablet for tablet 7035e41df60347b188a3b8079aa7e551 (DEFAULT_TABLE table=soft_deleted_table [id=06201339da074254986a87154a02a188]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260709 12:28:18.128556 32516 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7035e41df60347b188a3b8079aa7e551. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:18.130162 32590 tablet_service.cc:1511] Processing CreateTablet for tablet 7035e41df60347b188a3b8079aa7e551 (DEFAULT_TABLE table=soft_deleted_table [id=06201339da074254986a87154a02a188]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260709 12:28:18.131922 32590 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7035e41df60347b188a3b8079aa7e551. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:18.140381 32435 tablet_service.cc:1511] Processing CreateTablet for tablet 894a3f7744bd44ad814f22f90682c678 (DEFAULT_TABLE table=soft_deleted_table [id=06201339da074254986a87154a02a188]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260709 12:28:18.142099 32435 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 894a3f7744bd44ad814f22f90682c678. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:18.145282 32510 tablet_service.cc:1511] Processing CreateTablet for tablet 894a3f7744bd44ad814f22f90682c678 (DEFAULT_TABLE table=soft_deleted_table [id=06201339da074254986a87154a02a188]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260709 12:28:18.146771 32510 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 894a3f7744bd44ad814f22f90682c678. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:18.155254 32589 tablet_service.cc:1511] Processing CreateTablet for tablet 894a3f7744bd44ad814f22f90682c678 (DEFAULT_TABLE table=soft_deleted_table [id=06201339da074254986a87154a02a188]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260709 12:28:18.156805 32589 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 894a3f7744bd44ad814f22f90682c678. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:18.155283 32743 tablet_bootstrap.cc:492] T 92a7506e828b47a28f4e2c080d05419b P f8257d37af0941c2a978546ef2276b30: Bootstrap starting.
I20260709 12:28:18.169970 32518 tablet_service.cc:1511] Processing CreateTablet for tablet 11a335d230b94355ad8859ba469b7989 (DEFAULT_TABLE table=soft_deleted_table [id=06201339da074254986a87154a02a188]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260709 12:28:18.171504 32518 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11a335d230b94355ad8859ba469b7989. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:18.172108 32438 tablet_service.cc:1511] Processing CreateTablet for tablet 11a335d230b94355ad8859ba469b7989 (DEFAULT_TABLE table=soft_deleted_table [id=06201339da074254986a87154a02a188]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260709 12:28:18.173457 32438 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11a335d230b94355ad8859ba469b7989. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:18.180306 32586 tablet_service.cc:1511] Processing CreateTablet for tablet 11a335d230b94355ad8859ba469b7989 (DEFAULT_TABLE table=soft_deleted_table [id=06201339da074254986a87154a02a188]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260709 12:28:18.182286 32586 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11a335d230b94355ad8859ba469b7989. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:18.196888 32437 tablet_service.cc:1511] Processing CreateTablet for tablet f9cc8a9e92e64d2083b47e3619f88e32 (DEFAULT_TABLE table=soft_deleted_table [id=06201339da074254986a87154a02a188]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260709 12:28:18.198340 32437 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f9cc8a9e92e64d2083b47e3619f88e32. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:18.198778 32513 tablet_service.cc:1511] Processing CreateTablet for tablet f9cc8a9e92e64d2083b47e3619f88e32 (DEFAULT_TABLE table=soft_deleted_table [id=06201339da074254986a87154a02a188]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260709 12:28:18.200285 32513 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f9cc8a9e92e64d2083b47e3619f88e32. 1 dirs total, 0 dirs full, 0 dirs failed
W20260709 12:28:18.200577 32418 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:33369, user_credentials={real_user=slave}} blocked reactor thread for 70586.6us
I20260709 12:28:18.209439 32743 tablet_bootstrap.cc:654] T 92a7506e828b47a28f4e2c080d05419b P f8257d37af0941c2a978546ef2276b30: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:18.218232 32587 tablet_service.cc:1511] Processing CreateTablet for tablet f9cc8a9e92e64d2083b47e3619f88e32 (DEFAULT_TABLE table=soft_deleted_table [id=06201339da074254986a87154a02a188]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260709 12:28:18.220475 32587 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f9cc8a9e92e64d2083b47e3619f88e32. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:18.231962 32441 tablet_service.cc:1511] Processing CreateTablet for tablet f998d4ae7a1e419c9e457994500ce4d1 (DEFAULT_TABLE table=soft_deleted_table [id=06201339da074254986a87154a02a188]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260709 12:28:18.233527 32441 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f998d4ae7a1e419c9e457994500ce4d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:18.233956 32512 tablet_service.cc:1511] Processing CreateTablet for tablet f998d4ae7a1e419c9e457994500ce4d1 (DEFAULT_TABLE table=soft_deleted_table [id=06201339da074254986a87154a02a188]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260709 12:28:18.235352 32512 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f998d4ae7a1e419c9e457994500ce4d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:18.244189 32590 tablet_service.cc:1511] Processing CreateTablet for tablet f998d4ae7a1e419c9e457994500ce4d1 (DEFAULT_TABLE table=soft_deleted_table [id=06201339da074254986a87154a02a188]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260709 12:28:18.246088 32590 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f998d4ae7a1e419c9e457994500ce4d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:18.249253 32744 tablet_bootstrap.cc:492] T 51c3eb697520458093debb61eebb75a7 P cc8d9fdc73b14a3cb716b47c24bafc1d: Bootstrap starting.
I20260709 12:28:18.275271 32744 tablet_bootstrap.cc:654] T 51c3eb697520458093debb61eebb75a7 P cc8d9fdc73b14a3cb716b47c24bafc1d: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:18.292055 32743 tablet_bootstrap.cc:492] T 92a7506e828b47a28f4e2c080d05419b P f8257d37af0941c2a978546ef2276b30: No bootstrap required, opened a new log
I20260709 12:28:18.292869 32743 ts_tablet_manager.cc:1403] T 92a7506e828b47a28f4e2c080d05419b P f8257d37af0941c2a978546ef2276b30: Time spent bootstrapping tablet: real 0.138s	user 0.014s	sys 0.012s
I20260709 12:28:18.312856 32743 raft_consensus.cc:359] T 92a7506e828b47a28f4e2c080d05419b P f8257d37af0941c2a978546ef2276b30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:18.323756 32746 tablet_bootstrap.cc:492] T f9cc8a9e92e64d2083b47e3619f88e32 P a8adcce962244bc688dc4ecc1f69612c: Bootstrap starting.
I20260709 12:28:18.324165 32743 raft_consensus.cc:385] T 92a7506e828b47a28f4e2c080d05419b P f8257d37af0941c2a978546ef2276b30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:18.324591 32743 raft_consensus.cc:740] T 92a7506e828b47a28f4e2c080d05419b P f8257d37af0941c2a978546ef2276b30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8257d37af0941c2a978546ef2276b30, State: Initialized, Role: FOLLOWER
I20260709 12:28:18.333325 32746 tablet_bootstrap.cc:654] T f9cc8a9e92e64d2083b47e3619f88e32 P a8adcce962244bc688dc4ecc1f69612c: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:18.337204 32743 consensus_queue.cc:260] T 92a7506e828b47a28f4e2c080d05419b P f8257d37af0941c2a978546ef2276b30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:18.338498 32744 tablet_bootstrap.cc:492] T 51c3eb697520458093debb61eebb75a7 P cc8d9fdc73b14a3cb716b47c24bafc1d: No bootstrap required, opened a new log
I20260709 12:28:18.339037 32744 ts_tablet_manager.cc:1403] T 51c3eb697520458093debb61eebb75a7 P cc8d9fdc73b14a3cb716b47c24bafc1d: Time spent bootstrapping tablet: real 0.090s	user 0.021s	sys 0.013s
I20260709 12:28:18.340258 32743 ts_tablet_manager.cc:1434] T 92a7506e828b47a28f4e2c080d05419b P f8257d37af0941c2a978546ef2276b30: Time spent starting tablet: real 0.044s	user 0.009s	sys 0.001s
I20260709 12:28:18.342262 32744 raft_consensus.cc:359] T 51c3eb697520458093debb61eebb75a7 P cc8d9fdc73b14a3cb716b47c24bafc1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:18.343005 32744 raft_consensus.cc:385] T 51c3eb697520458093debb61eebb75a7 P cc8d9fdc73b14a3cb716b47c24bafc1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:18.343307 32744 raft_consensus.cc:740] T 51c3eb697520458093debb61eebb75a7 P cc8d9fdc73b14a3cb716b47c24bafc1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc8d9fdc73b14a3cb716b47c24bafc1d, State: Initialized, Role: FOLLOWER
I20260709 12:28:18.344192 32744 consensus_queue.cc:260] T 51c3eb697520458093debb61eebb75a7 P cc8d9fdc73b14a3cb716b47c24bafc1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:18.345500 32746 tablet_bootstrap.cc:492] T f9cc8a9e92e64d2083b47e3619f88e32 P a8adcce962244bc688dc4ecc1f69612c: No bootstrap required, opened a new log
I20260709 12:28:18.346105 32746 ts_tablet_manager.cc:1403] T f9cc8a9e92e64d2083b47e3619f88e32 P a8adcce962244bc688dc4ecc1f69612c: Time spent bootstrapping tablet: real 0.023s	user 0.013s	sys 0.000s
I20260709 12:28:18.346701 32744 ts_tablet_manager.cc:1434] T 51c3eb697520458093debb61eebb75a7 P cc8d9fdc73b14a3cb716b47c24bafc1d: Time spent starting tablet: real 0.007s	user 0.006s	sys 0.001s
I20260709 12:28:18.348026 32744 tablet_bootstrap.cc:492] T db65db1003af45d4b6315902498f3f2b P cc8d9fdc73b14a3cb716b47c24bafc1d: Bootstrap starting.
I20260709 12:28:18.348910 32743 tablet_bootstrap.cc:492] T 7035e41df60347b188a3b8079aa7e551 P f8257d37af0941c2a978546ef2276b30: Bootstrap starting.
I20260709 12:28:18.354597 32746 raft_consensus.cc:359] T f9cc8a9e92e64d2083b47e3619f88e32 P a8adcce962244bc688dc4ecc1f69612c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:18.355361 32744 tablet_bootstrap.cc:654] T db65db1003af45d4b6315902498f3f2b P cc8d9fdc73b14a3cb716b47c24bafc1d: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:18.355434 32746 raft_consensus.cc:385] T f9cc8a9e92e64d2083b47e3619f88e32 P a8adcce962244bc688dc4ecc1f69612c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:18.357154 32746 raft_consensus.cc:740] T f9cc8a9e92e64d2083b47e3619f88e32 P a8adcce962244bc688dc4ecc1f69612c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8adcce962244bc688dc4ecc1f69612c, State: Initialized, Role: FOLLOWER
I20260709 12:28:18.358234 32746 consensus_queue.cc:260] T f9cc8a9e92e64d2083b47e3619f88e32 P a8adcce962244bc688dc4ecc1f69612c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:18.360095 32743 tablet_bootstrap.cc:654] T 7035e41df60347b188a3b8079aa7e551 P f8257d37af0941c2a978546ef2276b30: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:18.360944 32746 ts_tablet_manager.cc:1434] T f9cc8a9e92e64d2083b47e3619f88e32 P a8adcce962244bc688dc4ecc1f69612c: Time spent starting tablet: real 0.015s	user 0.005s	sys 0.004s
I20260709 12:28:18.363162 32746 tablet_bootstrap.cc:492] T 276dee5db2d34451b642ed9359137ad4 P a8adcce962244bc688dc4ecc1f69612c: Bootstrap starting.
I20260709 12:28:18.372820 32746 tablet_bootstrap.cc:654] T 276dee5db2d34451b642ed9359137ad4 P a8adcce962244bc688dc4ecc1f69612c: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:18.376463 32744 tablet_bootstrap.cc:492] T db65db1003af45d4b6315902498f3f2b P cc8d9fdc73b14a3cb716b47c24bafc1d: No bootstrap required, opened a new log
I20260709 12:28:18.377135 32744 ts_tablet_manager.cc:1403] T db65db1003af45d4b6315902498f3f2b P cc8d9fdc73b14a3cb716b47c24bafc1d: Time spent bootstrapping tablet: real 0.029s	user 0.010s	sys 0.017s
I20260709 12:28:18.380367 32744 raft_consensus.cc:359] T db65db1003af45d4b6315902498f3f2b P cc8d9fdc73b14a3cb716b47c24bafc1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:18.381167 32744 raft_consensus.cc:385] T db65db1003af45d4b6315902498f3f2b P cc8d9fdc73b14a3cb716b47c24bafc1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:18.381505 32744 raft_consensus.cc:740] T db65db1003af45d4b6315902498f3f2b P cc8d9fdc73b14a3cb716b47c24bafc1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc8d9fdc73b14a3cb716b47c24bafc1d, State: Initialized, Role: FOLLOWER
I20260709 12:28:18.388473 32746 tablet_bootstrap.cc:492] T 276dee5db2d34451b642ed9359137ad4 P a8adcce962244bc688dc4ecc1f69612c: No bootstrap required, opened a new log
I20260709 12:28:18.388250 32744 consensus_queue.cc:260] T db65db1003af45d4b6315902498f3f2b P cc8d9fdc73b14a3cb716b47c24bafc1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:18.389209 32746 ts_tablet_manager.cc:1403] T 276dee5db2d34451b642ed9359137ad4 P a8adcce962244bc688dc4ecc1f69612c: Time spent bootstrapping tablet: real 0.026s	user 0.009s	sys 0.013s
I20260709 12:28:18.390710 32743 tablet_bootstrap.cc:492] T 7035e41df60347b188a3b8079aa7e551 P f8257d37af0941c2a978546ef2276b30: No bootstrap required, opened a new log
I20260709 12:28:18.391319 32743 ts_tablet_manager.cc:1403] T 7035e41df60347b188a3b8079aa7e551 P f8257d37af0941c2a978546ef2276b30: Time spent bootstrapping tablet: real 0.043s	user 0.023s	sys 0.007s
I20260709 12:28:18.392745 32746 raft_consensus.cc:359] T 276dee5db2d34451b642ed9359137ad4 P a8adcce962244bc688dc4ecc1f69612c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:18.393599 32744 ts_tablet_manager.cc:1434] T db65db1003af45d4b6315902498f3f2b P cc8d9fdc73b14a3cb716b47c24bafc1d: Time spent starting tablet: real 0.016s	user 0.000s	sys 0.008s
I20260709 12:28:18.393601 32746 raft_consensus.cc:385] T 276dee5db2d34451b642ed9359137ad4 P a8adcce962244bc688dc4ecc1f69612c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:18.400611 32746 raft_consensus.cc:740] T 276dee5db2d34451b642ed9359137ad4 P a8adcce962244bc688dc4ecc1f69612c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8adcce962244bc688dc4ecc1f69612c, State: Initialized, Role: FOLLOWER
I20260709 12:28:18.394702 32743 raft_consensus.cc:359] T 7035e41df60347b188a3b8079aa7e551 P f8257d37af0941c2a978546ef2276b30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:18.401149 32744 tablet_bootstrap.cc:492] T 92a7506e828b47a28f4e2c080d05419b P cc8d9fdc73b14a3cb716b47c24bafc1d: Bootstrap starting.
I20260709 12:28:18.401352 32743 raft_consensus.cc:385] T 7035e41df60347b188a3b8079aa7e551 P f8257d37af0941c2a978546ef2276b30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260709 12:28:18.398252 32399 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:28:18.401889 32743 raft_consensus.cc:740] T 7035e41df60347b188a3b8079aa7e551 P f8257d37af0941c2a978546ef2276b30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8257d37af0941c2a978546ef2276b30, State: Initialized, Role: FOLLOWER
I20260709 12:28:18.401737 32746 consensus_queue.cc:260] T 276dee5db2d34451b642ed9359137ad4 P a8adcce962244bc688dc4ecc1f69612c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:18.403142 32743 consensus_queue.cc:260] T 7035e41df60347b188a3b8079aa7e551 P f8257d37af0941c2a978546ef2276b30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:18.406294 32743 ts_tablet_manager.cc:1434] T 7035e41df60347b188a3b8079aa7e551 P f8257d37af0941c2a978546ef2276b30: Time spent starting tablet: real 0.015s	user 0.011s	sys 0.004s
I20260709 12:28:18.407931 32743 tablet_bootstrap.cc:492] T 11a335d230b94355ad8859ba469b7989 P f8257d37af0941c2a978546ef2276b30: Bootstrap starting.
I20260709 12:28:18.411857 32744 tablet_bootstrap.cc:654] T 92a7506e828b47a28f4e2c080d05419b P cc8d9fdc73b14a3cb716b47c24bafc1d: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:18.413964 32746 ts_tablet_manager.cc:1434] T 276dee5db2d34451b642ed9359137ad4 P a8adcce962244bc688dc4ecc1f69612c: Time spent starting tablet: real 0.024s	user 0.006s	sys 0.002s
I20260709 12:28:18.415441 32746 tablet_bootstrap.cc:492] T 894a3f7744bd44ad814f22f90682c678 P a8adcce962244bc688dc4ecc1f69612c: Bootstrap starting.
I20260709 12:28:18.419579 32743 tablet_bootstrap.cc:654] T 11a335d230b94355ad8859ba469b7989 P f8257d37af0941c2a978546ef2276b30: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:18.425510 32744 tablet_bootstrap.cc:492] T 92a7506e828b47a28f4e2c080d05419b P cc8d9fdc73b14a3cb716b47c24bafc1d: No bootstrap required, opened a new log
I20260709 12:28:18.425925 32746 tablet_bootstrap.cc:654] T 894a3f7744bd44ad814f22f90682c678 P a8adcce962244bc688dc4ecc1f69612c: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:18.426251 32744 ts_tablet_manager.cc:1403] T 92a7506e828b47a28f4e2c080d05419b P cc8d9fdc73b14a3cb716b47c24bafc1d: Time spent bootstrapping tablet: real 0.025s	user 0.012s	sys 0.007s
I20260709 12:28:18.430090 32744 raft_consensus.cc:359] T 92a7506e828b47a28f4e2c080d05419b P cc8d9fdc73b14a3cb716b47c24bafc1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:18.430801 32744 raft_consensus.cc:385] T 92a7506e828b47a28f4e2c080d05419b P cc8d9fdc73b14a3cb716b47c24bafc1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:18.431129 32744 raft_consensus.cc:740] T 92a7506e828b47a28f4e2c080d05419b P cc8d9fdc73b14a3cb716b47c24bafc1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc8d9fdc73b14a3cb716b47c24bafc1d, State: Initialized, Role: FOLLOWER
I20260709 12:28:18.431820 32744 consensus_queue.cc:260] T 92a7506e828b47a28f4e2c080d05419b P cc8d9fdc73b14a3cb716b47c24bafc1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:18.432883 32743 tablet_bootstrap.cc:492] T 11a335d230b94355ad8859ba469b7989 P f8257d37af0941c2a978546ef2276b30: No bootstrap required, opened a new log
I20260709 12:28:18.433449 32743 ts_tablet_manager.cc:1403] T 11a335d230b94355ad8859ba469b7989 P f8257d37af0941c2a978546ef2276b30: Time spent bootstrapping tablet: real 0.026s	user 0.011s	sys 0.000s
I20260709 12:28:18.433735 32746 tablet_bootstrap.cc:492] T 894a3f7744bd44ad814f22f90682c678 P a8adcce962244bc688dc4ecc1f69612c: No bootstrap required, opened a new log
I20260709 12:28:18.434242 32746 ts_tablet_manager.cc:1403] T 894a3f7744bd44ad814f22f90682c678 P a8adcce962244bc688dc4ecc1f69612c: Time spent bootstrapping tablet: real 0.019s	user 0.010s	sys 0.004s
I20260709 12:28:18.434410 32744 ts_tablet_manager.cc:1434] T 92a7506e828b47a28f4e2c080d05419b P cc8d9fdc73b14a3cb716b47c24bafc1d: Time spent starting tablet: real 0.008s	user 0.007s	sys 0.000s
I20260709 12:28:18.435748 32744 tablet_bootstrap.cc:492] T 894a3f7744bd44ad814f22f90682c678 P cc8d9fdc73b14a3cb716b47c24bafc1d: Bootstrap starting.
I20260709 12:28:18.436753 32743 raft_consensus.cc:359] T 11a335d230b94355ad8859ba469b7989 P f8257d37af0941c2a978546ef2276b30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:18.437523 32743 raft_consensus.cc:385] T 11a335d230b94355ad8859ba469b7989 P f8257d37af0941c2a978546ef2276b30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:18.437433 32746 raft_consensus.cc:359] T 894a3f7744bd44ad814f22f90682c678 P a8adcce962244bc688dc4ecc1f69612c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:18.438050 32743 raft_consensus.cc:740] T 11a335d230b94355ad8859ba469b7989 P f8257d37af0941c2a978546ef2276b30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8257d37af0941c2a978546ef2276b30, State: Initialized, Role: FOLLOWER
I20260709 12:28:18.438367 32746 raft_consensus.cc:385] T 894a3f7744bd44ad814f22f90682c678 P a8adcce962244bc688dc4ecc1f69612c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:18.438742 32746 raft_consensus.cc:740] T 894a3f7744bd44ad814f22f90682c678 P a8adcce962244bc688dc4ecc1f69612c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8adcce962244bc688dc4ecc1f69612c, State: Initialized, Role: FOLLOWER
I20260709 12:28:18.438990 32743 consensus_queue.cc:260] T 11a335d230b94355ad8859ba469b7989 P f8257d37af0941c2a978546ef2276b30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:18.439512 32746 consensus_queue.cc:260] T 894a3f7744bd44ad814f22f90682c678 P a8adcce962244bc688dc4ecc1f69612c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:18.441521 32743 ts_tablet_manager.cc:1434] T 11a335d230b94355ad8859ba469b7989 P f8257d37af0941c2a978546ef2276b30: Time spent starting tablet: real 0.008s	user 0.005s	sys 0.003s
I20260709 12:28:18.442044 32746 ts_tablet_manager.cc:1434] T 894a3f7744bd44ad814f22f90682c678 P a8adcce962244bc688dc4ecc1f69612c: Time spent starting tablet: real 0.007s	user 0.008s	sys 0.000s
I20260709 12:28:18.443113 32744 tablet_bootstrap.cc:654] T 894a3f7744bd44ad814f22f90682c678 P cc8d9fdc73b14a3cb716b47c24bafc1d: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:18.443123 32746 tablet_bootstrap.cc:492] T 7035e41df60347b188a3b8079aa7e551 P a8adcce962244bc688dc4ecc1f69612c: Bootstrap starting.
I20260709 12:28:18.447206 32743 tablet_bootstrap.cc:492] T 276dee5db2d34451b642ed9359137ad4 P f8257d37af0941c2a978546ef2276b30: Bootstrap starting.
I20260709 12:28:18.451498 32746 tablet_bootstrap.cc:654] T 7035e41df60347b188a3b8079aa7e551 P a8adcce962244bc688dc4ecc1f69612c: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:18.461598 32743 tablet_bootstrap.cc:654] T 276dee5db2d34451b642ed9359137ad4 P f8257d37af0941c2a978546ef2276b30: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:18.464648 32744 tablet_bootstrap.cc:492] T 894a3f7744bd44ad814f22f90682c678 P cc8d9fdc73b14a3cb716b47c24bafc1d: No bootstrap required, opened a new log
I20260709 12:28:18.465183 32744 ts_tablet_manager.cc:1403] T 894a3f7744bd44ad814f22f90682c678 P cc8d9fdc73b14a3cb716b47c24bafc1d: Time spent bootstrapping tablet: real 0.030s	user 0.009s	sys 0.019s
I20260709 12:28:18.465569 32746 tablet_bootstrap.cc:492] T 7035e41df60347b188a3b8079aa7e551 P a8adcce962244bc688dc4ecc1f69612c: No bootstrap required, opened a new log
I20260709 12:28:18.466094 32746 ts_tablet_manager.cc:1403] T 7035e41df60347b188a3b8079aa7e551 P a8adcce962244bc688dc4ecc1f69612c: Time spent bootstrapping tablet: real 0.023s	user 0.014s	sys 0.005s
I20260709 12:28:18.468489 32743 tablet_bootstrap.cc:492] T 276dee5db2d34451b642ed9359137ad4 P f8257d37af0941c2a978546ef2276b30: No bootstrap required, opened a new log
I20260709 12:28:18.468179 32744 raft_consensus.cc:359] T 894a3f7744bd44ad814f22f90682c678 P cc8d9fdc73b14a3cb716b47c24bafc1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:18.468959 32744 raft_consensus.cc:385] T 894a3f7744bd44ad814f22f90682c678 P cc8d9fdc73b14a3cb716b47c24bafc1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:18.469014 32743 ts_tablet_manager.cc:1403] T 276dee5db2d34451b642ed9359137ad4 P f8257d37af0941c2a978546ef2276b30: Time spent bootstrapping tablet: real 0.022s	user 0.012s	sys 0.001s
I20260709 12:28:18.469422 32744 raft_consensus.cc:740] T 894a3f7744bd44ad814f22f90682c678 P cc8d9fdc73b14a3cb716b47c24bafc1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc8d9fdc73b14a3cb716b47c24bafc1d, State: Initialized, Role: FOLLOWER
I20260709 12:28:18.469040 32746 raft_consensus.cc:359] T 7035e41df60347b188a3b8079aa7e551 P a8adcce962244bc688dc4ecc1f69612c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:18.470026 32746 raft_consensus.cc:385] T 7035e41df60347b188a3b8079aa7e551 P a8adcce962244bc688dc4ecc1f69612c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:18.470340 32746 raft_consensus.cc:740] T 7035e41df60347b188a3b8079aa7e551 P a8adcce962244bc688dc4ecc1f69612c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8adcce962244bc688dc4ecc1f69612c, State: Initialized, Role: FOLLOWER
I20260709 12:28:18.470257 32744 consensus_queue.cc:260] T 894a3f7744bd44ad814f22f90682c678 P cc8d9fdc73b14a3cb716b47c24bafc1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:18.471242 32746 consensus_queue.cc:260] T 7035e41df60347b188a3b8079aa7e551 P a8adcce962244bc688dc4ecc1f69612c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:18.472282 32743 raft_consensus.cc:359] T 276dee5db2d34451b642ed9359137ad4 P f8257d37af0941c2a978546ef2276b30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:18.473140 32743 raft_consensus.cc:385] T 276dee5db2d34451b642ed9359137ad4 P f8257d37af0941c2a978546ef2276b30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:18.473157 32744 ts_tablet_manager.cc:1434] T 894a3f7744bd44ad814f22f90682c678 P cc8d9fdc73b14a3cb716b47c24bafc1d: Time spent starting tablet: real 0.008s	user 0.008s	sys 0.000s
I20260709 12:28:18.473593 32746 ts_tablet_manager.cc:1434] T 7035e41df60347b188a3b8079aa7e551 P a8adcce962244bc688dc4ecc1f69612c: Time spent starting tablet: real 0.007s	user 0.004s	sys 0.004s
I20260709 12:28:18.474393 32744 tablet_bootstrap.cc:492] T f9cc8a9e92e64d2083b47e3619f88e32 P cc8d9fdc73b14a3cb716b47c24bafc1d: Bootstrap starting.
I20260709 12:28:18.474784 32746 tablet_bootstrap.cc:492] T 51c3eb697520458093debb61eebb75a7 P a8adcce962244bc688dc4ecc1f69612c: Bootstrap starting.
I20260709 12:28:18.473582 32743 raft_consensus.cc:740] T 276dee5db2d34451b642ed9359137ad4 P f8257d37af0941c2a978546ef2276b30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8257d37af0941c2a978546ef2276b30, State: Initialized, Role: FOLLOWER
I20260709 12:28:18.476969 32743 consensus_queue.cc:260] T 276dee5db2d34451b642ed9359137ad4 P f8257d37af0941c2a978546ef2276b30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:18.479451 32743 ts_tablet_manager.cc:1434] T 276dee5db2d34451b642ed9359137ad4 P f8257d37af0941c2a978546ef2276b30: Time spent starting tablet: real 0.010s	user 0.002s	sys 0.004s
I20260709 12:28:18.481118 32746 tablet_bootstrap.cc:654] T 51c3eb697520458093debb61eebb75a7 P a8adcce962244bc688dc4ecc1f69612c: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:18.481297 32744 tablet_bootstrap.cc:654] T f9cc8a9e92e64d2083b47e3619f88e32 P cc8d9fdc73b14a3cb716b47c24bafc1d: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:18.481609 32743 tablet_bootstrap.cc:492] T db65db1003af45d4b6315902498f3f2b P f8257d37af0941c2a978546ef2276b30: Bootstrap starting.
I20260709 12:28:18.488721 32743 tablet_bootstrap.cc:654] T db65db1003af45d4b6315902498f3f2b P f8257d37af0941c2a978546ef2276b30: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:18.497807 32746 tablet_bootstrap.cc:492] T 51c3eb697520458093debb61eebb75a7 P a8adcce962244bc688dc4ecc1f69612c: No bootstrap required, opened a new log
I20260709 12:28:18.498253 32746 ts_tablet_manager.cc:1403] T 51c3eb697520458093debb61eebb75a7 P a8adcce962244bc688dc4ecc1f69612c: Time spent bootstrapping tablet: real 0.024s	user 0.012s	sys 0.004s
I20260709 12:28:18.500770 32746 raft_consensus.cc:359] T 51c3eb697520458093debb61eebb75a7 P a8adcce962244bc688dc4ecc1f69612c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:18.501514 32746 raft_consensus.cc:385] T 51c3eb697520458093debb61eebb75a7 P a8adcce962244bc688dc4ecc1f69612c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:18.501756 32744 tablet_bootstrap.cc:492] T f9cc8a9e92e64d2083b47e3619f88e32 P cc8d9fdc73b14a3cb716b47c24bafc1d: No bootstrap required, opened a new log
I20260709 12:28:18.501832 32746 raft_consensus.cc:740] T 51c3eb697520458093debb61eebb75a7 P a8adcce962244bc688dc4ecc1f69612c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8adcce962244bc688dc4ecc1f69612c, State: Initialized, Role: FOLLOWER
I20260709 12:28:18.502573 32744 ts_tablet_manager.cc:1403] T f9cc8a9e92e64d2083b47e3619f88e32 P cc8d9fdc73b14a3cb716b47c24bafc1d: Time spent bootstrapping tablet: real 0.028s	user 0.019s	sys 0.004s
I20260709 12:28:18.503153 32746 consensus_queue.cc:260] T 51c3eb697520458093debb61eebb75a7 P a8adcce962244bc688dc4ecc1f69612c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:18.505769 32746 ts_tablet_manager.cc:1434] T 51c3eb697520458093debb61eebb75a7 P a8adcce962244bc688dc4ecc1f69612c: Time spent starting tablet: real 0.007s	user 0.009s	sys 0.000s
I20260709 12:28:18.506090 32744 raft_consensus.cc:359] T f9cc8a9e92e64d2083b47e3619f88e32 P cc8d9fdc73b14a3cb716b47c24bafc1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:18.506772 32746 tablet_bootstrap.cc:492] T f998d4ae7a1e419c9e457994500ce4d1 P a8adcce962244bc688dc4ecc1f69612c: Bootstrap starting.
I20260709 12:28:18.506824 32744 raft_consensus.cc:385] T f9cc8a9e92e64d2083b47e3619f88e32 P cc8d9fdc73b14a3cb716b47c24bafc1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:18.508683 32731 raft_consensus.cc:493] T 894a3f7744bd44ad814f22f90682c678 P a8adcce962244bc688dc4ecc1f69612c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:18.513274 32731 raft_consensus.cc:515] T 894a3f7744bd44ad814f22f90682c678 P a8adcce962244bc688dc4ecc1f69612c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:18.514158 32746 tablet_bootstrap.cc:654] T f998d4ae7a1e419c9e457994500ce4d1 P a8adcce962244bc688dc4ecc1f69612c: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:18.517819 32601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "894a3f7744bd44ad814f22f90682c678" candidate_uuid: "a8adcce962244bc688dc4ecc1f69612c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8257d37af0941c2a978546ef2276b30" is_pre_election: true
I20260709 12:28:18.520210 32452 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "894a3f7744bd44ad814f22f90682c678" candidate_uuid: "a8adcce962244bc688dc4ecc1f69612c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" is_pre_election: true
I20260709 12:28:18.521137 32452 raft_consensus.cc:2468] T 894a3f7744bd44ad814f22f90682c678 P cc8d9fdc73b14a3cb716b47c24bafc1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a8adcce962244bc688dc4ecc1f69612c in term 0.
W20260709 12:28:18.523248 32493 leader_election.cc:343] T 894a3f7744bd44ad814f22f90682c678 P a8adcce962244bc688dc4ecc1f69612c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f8257d37af0941c2a978546ef2276b30 (127.16.29.67:37155): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:28:18.524643 32494 leader_election.cc:304] T 894a3f7744bd44ad814f22f90682c678 P a8adcce962244bc688dc4ecc1f69612c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a8adcce962244bc688dc4ecc1f69612c, cc8d9fdc73b14a3cb716b47c24bafc1d; no voters: f8257d37af0941c2a978546ef2276b30
I20260709 12:28:18.515650 32731 leader_election.cc:290] T 894a3f7744bd44ad814f22f90682c678 P a8adcce962244bc688dc4ecc1f69612c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc8d9fdc73b14a3cb716b47c24bafc1d (127.16.29.65:41229), f8257d37af0941c2a978546ef2276b30 (127.16.29.67:37155)
I20260709 12:28:18.527508 32731 raft_consensus.cc:2804] T 894a3f7744bd44ad814f22f90682c678 P a8adcce962244bc688dc4ecc1f69612c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:28:18.528076 32731 raft_consensus.cc:493] T 894a3f7744bd44ad814f22f90682c678 P a8adcce962244bc688dc4ecc1f69612c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:18.528635 32731 raft_consensus.cc:3060] T 894a3f7744bd44ad814f22f90682c678 P a8adcce962244bc688dc4ecc1f69612c [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:18.533965 32746 tablet_bootstrap.cc:492] T f998d4ae7a1e419c9e457994500ce4d1 P a8adcce962244bc688dc4ecc1f69612c: No bootstrap required, opened a new log
I20260709 12:28:18.534605 32746 ts_tablet_manager.cc:1403] T f998d4ae7a1e419c9e457994500ce4d1 P a8adcce962244bc688dc4ecc1f69612c: Time spent bootstrapping tablet: real 0.028s	user 0.004s	sys 0.013s
I20260709 12:28:18.538079 32746 raft_consensus.cc:359] T f998d4ae7a1e419c9e457994500ce4d1 P a8adcce962244bc688dc4ecc1f69612c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:18.538859 32746 raft_consensus.cc:385] T f998d4ae7a1e419c9e457994500ce4d1 P a8adcce962244bc688dc4ecc1f69612c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:18.539194 32746 raft_consensus.cc:740] T f998d4ae7a1e419c9e457994500ce4d1 P a8adcce962244bc688dc4ecc1f69612c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8adcce962244bc688dc4ecc1f69612c, State: Initialized, Role: FOLLOWER
I20260709 12:28:18.540086 32746 consensus_queue.cc:260] T f998d4ae7a1e419c9e457994500ce4d1 P a8adcce962244bc688dc4ecc1f69612c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:18.540633 32744 raft_consensus.cc:740] T f9cc8a9e92e64d2083b47e3619f88e32 P cc8d9fdc73b14a3cb716b47c24bafc1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc8d9fdc73b14a3cb716b47c24bafc1d, State: Initialized, Role: FOLLOWER
I20260709 12:28:18.546917 32746 ts_tablet_manager.cc:1434] T f998d4ae7a1e419c9e457994500ce4d1 P a8adcce962244bc688dc4ecc1f69612c: Time spent starting tablet: real 0.012s	user 0.009s	sys 0.000s
I20260709 12:28:18.548084 32746 tablet_bootstrap.cc:492] T 11a335d230b94355ad8859ba469b7989 P a8adcce962244bc688dc4ecc1f69612c: Bootstrap starting.
I20260709 12:28:18.541620 32744 consensus_queue.cc:260] T f9cc8a9e92e64d2083b47e3619f88e32 P cc8d9fdc73b14a3cb716b47c24bafc1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:18.550524 32744 ts_tablet_manager.cc:1434] T f9cc8a9e92e64d2083b47e3619f88e32 P cc8d9fdc73b14a3cb716b47c24bafc1d: Time spent starting tablet: real 0.048s	user 0.005s	sys 0.001s
I20260709 12:28:18.551956 32743 tablet_bootstrap.cc:492] T db65db1003af45d4b6315902498f3f2b P f8257d37af0941c2a978546ef2276b30: No bootstrap required, opened a new log
I20260709 12:28:18.552165 32762 raft_consensus.cc:493] T 7035e41df60347b188a3b8079aa7e551 P a8adcce962244bc688dc4ecc1f69612c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:18.552731 32743 ts_tablet_manager.cc:1403] T db65db1003af45d4b6315902498f3f2b P f8257d37af0941c2a978546ef2276b30: Time spent bootstrapping tablet: real 0.071s	user 0.029s	sys 0.027s
I20260709 12:28:18.553148 32744 tablet_bootstrap.cc:492] T 7035e41df60347b188a3b8079aa7e551 P cc8d9fdc73b14a3cb716b47c24bafc1d: Bootstrap starting.
I20260709 12:28:18.552815 32762 raft_consensus.cc:515] T 7035e41df60347b188a3b8079aa7e551 P a8adcce962244bc688dc4ecc1f69612c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:18.555765 32452 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7035e41df60347b188a3b8079aa7e551" candidate_uuid: "a8adcce962244bc688dc4ecc1f69612c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" is_pre_election: true
I20260709 12:28:18.555995 32743 raft_consensus.cc:359] T db65db1003af45d4b6315902498f3f2b P f8257d37af0941c2a978546ef2276b30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:18.556910 32743 raft_consensus.cc:385] T db65db1003af45d4b6315902498f3f2b P f8257d37af0941c2a978546ef2276b30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:18.557253 32743 raft_consensus.cc:740] T db65db1003af45d4b6315902498f3f2b P f8257d37af0941c2a978546ef2276b30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8257d37af0941c2a978546ef2276b30, State: Initialized, Role: FOLLOWER
W20260709 12:28:18.557313 32494 leader_election.cc:343] T 7035e41df60347b188a3b8079aa7e551 P a8adcce962244bc688dc4ecc1f69612c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cc8d9fdc73b14a3cb716b47c24bafc1d (127.16.29.65:41229): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:28:18.558693 32743 consensus_queue.cc:260] T db65db1003af45d4b6315902498f3f2b P f8257d37af0941c2a978546ef2276b30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:18.560333 32744 tablet_bootstrap.cc:654] T 7035e41df60347b188a3b8079aa7e551 P cc8d9fdc73b14a3cb716b47c24bafc1d: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:18.559832 32601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7035e41df60347b188a3b8079aa7e551" candidate_uuid: "a8adcce962244bc688dc4ecc1f69612c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8257d37af0941c2a978546ef2276b30" is_pre_election: true
I20260709 12:28:18.561362 32743 ts_tablet_manager.cc:1434] T db65db1003af45d4b6315902498f3f2b P f8257d37af0941c2a978546ef2276b30: Time spent starting tablet: real 0.008s	user 0.006s	sys 0.002s
I20260709 12:28:18.561496 32601 raft_consensus.cc:2468] T 7035e41df60347b188a3b8079aa7e551 P f8257d37af0941c2a978546ef2276b30 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a8adcce962244bc688dc4ecc1f69612c in term 0.
I20260709 12:28:18.562577 32743 tablet_bootstrap.cc:492] T 894a3f7744bd44ad814f22f90682c678 P f8257d37af0941c2a978546ef2276b30: Bootstrap starting.
I20260709 12:28:18.562959 32493 leader_election.cc:304] T 7035e41df60347b188a3b8079aa7e551 P a8adcce962244bc688dc4ecc1f69612c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a8adcce962244bc688dc4ecc1f69612c, f8257d37af0941c2a978546ef2276b30; no voters: cc8d9fdc73b14a3cb716b47c24bafc1d
I20260709 12:28:18.552139 32731 raft_consensus.cc:515] T 894a3f7744bd44ad814f22f90682c678 P a8adcce962244bc688dc4ecc1f69612c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:18.564443 32762 leader_election.cc:290] T 7035e41df60347b188a3b8079aa7e551 P a8adcce962244bc688dc4ecc1f69612c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc8d9fdc73b14a3cb716b47c24bafc1d (127.16.29.65:41229), f8257d37af0941c2a978546ef2276b30 (127.16.29.67:37155)
I20260709 12:28:18.564991 32762 raft_consensus.cc:2804] T 7035e41df60347b188a3b8079aa7e551 P a8adcce962244bc688dc4ecc1f69612c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:28:18.565044 32729 raft_consensus.cc:493] T db65db1003af45d4b6315902498f3f2b P f8257d37af0941c2a978546ef2276b30 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:18.565508 32726 raft_consensus.cc:493] T db65db1003af45d4b6315902498f3f2b P cc8d9fdc73b14a3cb716b47c24bafc1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:18.565701 32729 raft_consensus.cc:515] T db65db1003af45d4b6315902498f3f2b P f8257d37af0941c2a978546ef2276b30 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:18.566236 32726 raft_consensus.cc:515] T db65db1003af45d4b6315902498f3f2b P cc8d9fdc73b14a3cb716b47c24bafc1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:18.565459 32762 raft_consensus.cc:493] T 7035e41df60347b188a3b8079aa7e551 P a8adcce962244bc688dc4ecc1f69612c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:18.566953 32731 leader_election.cc:290] T 894a3f7744bd44ad814f22f90682c678 P a8adcce962244bc688dc4ecc1f69612c [CANDIDATE]: Term 1 election: Requested vote from peers cc8d9fdc73b14a3cb716b47c24bafc1d (127.16.29.65:41229), f8257d37af0941c2a978546ef2276b30 (127.16.29.67:37155)
I20260709 12:28:18.567221 32762 raft_consensus.cc:3060] T 7035e41df60347b188a3b8079aa7e551 P a8adcce962244bc688dc4ecc1f69612c [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:18.569008 32729 leader_election.cc:290] T db65db1003af45d4b6315902498f3f2b P f8257d37af0941c2a978546ef2276b30 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc8d9fdc73b14a3cb716b47c24bafc1d (127.16.29.65:41229), a8adcce962244bc688dc4ecc1f69612c (127.16.29.66:33369)
I20260709 12:28:18.569758 32528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "db65db1003af45d4b6315902498f3f2b" candidate_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8adcce962244bc688dc4ecc1f69612c" is_pre_election: true
I20260709 12:28:18.570725 32527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "db65db1003af45d4b6315902498f3f2b" candidate_uuid: "f8257d37af0941c2a978546ef2276b30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8adcce962244bc688dc4ecc1f69612c" is_pre_election: true
I20260709 12:28:18.570861 32726 leader_election.cc:290] T db65db1003af45d4b6315902498f3f2b P cc8d9fdc73b14a3cb716b47c24bafc1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a8adcce962244bc688dc4ecc1f69612c (127.16.29.66:33369), f8257d37af0941c2a978546ef2276b30 (127.16.29.67:37155)
W20260709 12:28:18.572173 32567 leader_election.cc:343] T db65db1003af45d4b6315902498f3f2b P f8257d37af0941c2a978546ef2276b30 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a8adcce962244bc688dc4ecc1f69612c (127.16.29.66:33369): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:28:18.572553 32418 leader_election.cc:343] T db65db1003af45d4b6315902498f3f2b P cc8d9fdc73b14a3cb716b47c24bafc1d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a8adcce962244bc688dc4ecc1f69612c (127.16.29.66:33369): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:28:18.573021 32601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "db65db1003af45d4b6315902498f3f2b" candidate_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8257d37af0941c2a978546ef2276b30" is_pre_election: true
I20260709 12:28:18.566255 32452 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "894a3f7744bd44ad814f22f90682c678" candidate_uuid: "a8adcce962244bc688dc4ecc1f69612c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d"
I20260709 12:28:18.573731 32453 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "db65db1003af45d4b6315902498f3f2b" candidate_uuid: "f8257d37af0941c2a978546ef2276b30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" is_pre_election: true
I20260709 12:28:18.573787 32601 raft_consensus.cc:2468] T db65db1003af45d4b6315902498f3f2b P f8257d37af0941c2a978546ef2276b30 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc8d9fdc73b14a3cb716b47c24bafc1d in term 0.
I20260709 12:28:18.574602 32453 raft_consensus.cc:2468] T db65db1003af45d4b6315902498f3f2b P cc8d9fdc73b14a3cb716b47c24bafc1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8257d37af0941c2a978546ef2276b30 in term 0.
I20260709 12:28:18.575371 32418 leader_election.cc:304] T db65db1003af45d4b6315902498f3f2b P cc8d9fdc73b14a3cb716b47c24bafc1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cc8d9fdc73b14a3cb716b47c24bafc1d, f8257d37af0941c2a978546ef2276b30; no voters: a8adcce962244bc688dc4ecc1f69612c
I20260709 12:28:18.574002 32452 raft_consensus.cc:3060] T 894a3f7744bd44ad814f22f90682c678 P cc8d9fdc73b14a3cb716b47c24bafc1d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:18.577121 32568 leader_election.cc:304] T db65db1003af45d4b6315902498f3f2b P f8257d37af0941c2a978546ef2276b30 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cc8d9fdc73b14a3cb716b47c24bafc1d, f8257d37af0941c2a978546ef2276b30; no voters: a8adcce962244bc688dc4ecc1f69612c
I20260709 12:28:18.576481 32726 raft_consensus.cc:2804] T db65db1003af45d4b6315902498f3f2b P cc8d9fdc73b14a3cb716b47c24bafc1d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:28:18.579977 32729 raft_consensus.cc:2804] T db65db1003af45d4b6315902498f3f2b P f8257d37af0941c2a978546ef2276b30 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:28:18.581106 32726 raft_consensus.cc:493] T db65db1003af45d4b6315902498f3f2b P cc8d9fdc73b14a3cb716b47c24bafc1d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:18.582134 32729 raft_consensus.cc:493] T db65db1003af45d4b6315902498f3f2b P f8257d37af0941c2a978546ef2276b30 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:18.587930 32746 tablet_bootstrap.cc:654] T 11a335d230b94355ad8859ba469b7989 P a8adcce962244bc688dc4ecc1f69612c: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:18.588182 32729 raft_consensus.cc:3060] T db65db1003af45d4b6315902498f3f2b P f8257d37af0941c2a978546ef2276b30 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:18.588220 32726 raft_consensus.cc:3060] T db65db1003af45d4b6315902498f3f2b P cc8d9fdc73b14a3cb716b47c24bafc1d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:18.587283 32601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "894a3f7744bd44ad814f22f90682c678" candidate_uuid: "a8adcce962244bc688dc4ecc1f69612c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8257d37af0941c2a978546ef2276b30"
I20260709 12:28:18.594919 32764 raft_consensus.cc:493] T 276dee5db2d34451b642ed9359137ad4 P a8adcce962244bc688dc4ecc1f69612c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:18.595690 32764 raft_consensus.cc:515] T 276dee5db2d34451b642ed9359137ad4 P a8adcce962244bc688dc4ecc1f69612c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:18.597041 32726 raft_consensus.cc:515] T db65db1003af45d4b6315902498f3f2b P cc8d9fdc73b14a3cb716b47c24bafc1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:18.599644 32453 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "276dee5db2d34451b642ed9359137ad4" candidate_uuid: "a8adcce962244bc688dc4ecc1f69612c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" is_pre_election: true
I20260709 12:28:18.602137 32601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "db65db1003af45d4b6315902498f3f2b" candidate_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8257d37af0941c2a978546ef2276b30"
W20260709 12:28:18.602217 32494 leader_election.cc:343] T 276dee5db2d34451b642ed9359137ad4 P a8adcce962244bc688dc4ecc1f69612c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cc8d9fdc73b14a3cb716b47c24bafc1d (127.16.29.65:41229): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:28:18.599928 32527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "db65db1003af45d4b6315902498f3f2b" candidate_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8adcce962244bc688dc4ecc1f69612c"
W20260709 12:28:18.603840 32418 leader_election.cc:343] T db65db1003af45d4b6315902498f3f2b P cc8d9fdc73b14a3cb716b47c24bafc1d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a8adcce962244bc688dc4ecc1f69612c (127.16.29.66:33369): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:28:18.602939 32726 leader_election.cc:290] T db65db1003af45d4b6315902498f3f2b P cc8d9fdc73b14a3cb716b47c24bafc1d [CANDIDATE]: Term 1 election: Requested vote from peers a8adcce962244bc688dc4ecc1f69612c (127.16.29.66:33369), f8257d37af0941c2a978546ef2276b30 (127.16.29.67:37155)
W20260709 12:28:18.605340 32493 leader_election.cc:343] T 894a3f7744bd44ad814f22f90682c678 P a8adcce962244bc688dc4ecc1f69612c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f8257d37af0941c2a978546ef2276b30 (127.16.29.67:37155): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:28:18.599788 32744 tablet_bootstrap.cc:492] T 7035e41df60347b188a3b8079aa7e551 P cc8d9fdc73b14a3cb716b47c24bafc1d: No bootstrap required, opened a new log
I20260709 12:28:18.606449 32744 ts_tablet_manager.cc:1403] T 7035e41df60347b188a3b8079aa7e551 P cc8d9fdc73b14a3cb716b47c24bafc1d: Time spent bootstrapping tablet: real 0.054s	user 0.012s	sys 0.001s
I20260709 12:28:18.587950 32743 tablet_bootstrap.cc:654] T 894a3f7744bd44ad814f22f90682c678 P f8257d37af0941c2a978546ef2276b30: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:18.609414 32744 raft_consensus.cc:359] T 7035e41df60347b188a3b8079aa7e551 P cc8d9fdc73b14a3cb716b47c24bafc1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:18.610121 32744 raft_consensus.cc:385] T 7035e41df60347b188a3b8079aa7e551 P cc8d9fdc73b14a3cb716b47c24bafc1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:18.610370 32744 raft_consensus.cc:740] T 7035e41df60347b188a3b8079aa7e551 P cc8d9fdc73b14a3cb716b47c24bafc1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc8d9fdc73b14a3cb716b47c24bafc1d, State: Initialized, Role: FOLLOWER
I20260709 12:28:18.611088 32744 consensus_queue.cc:260] T 7035e41df60347b188a3b8079aa7e551 P cc8d9fdc73b14a3cb716b47c24bafc1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:18.614269 32762 raft_consensus.cc:515] T 7035e41df60347b188a3b8079aa7e551 P a8adcce962244bc688dc4ecc1f69612c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:18.617283 32453 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7035e41df60347b188a3b8079aa7e551" candidate_uuid: "a8adcce962244bc688dc4ecc1f69612c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d"
I20260709 12:28:18.617341 32602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "276dee5db2d34451b642ed9359137ad4" candidate_uuid: "a8adcce962244bc688dc4ecc1f69612c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8257d37af0941c2a978546ef2276b30" is_pre_election: true
I20260709 12:28:18.618162 32453 raft_consensus.cc:3060] T 7035e41df60347b188a3b8079aa7e551 P cc8d9fdc73b14a3cb716b47c24bafc1d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:18.618438 32602 raft_consensus.cc:2468] T 276dee5db2d34451b642ed9359137ad4 P f8257d37af0941c2a978546ef2276b30 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a8adcce962244bc688dc4ecc1f69612c in term 0.
I20260709 12:28:18.620446 32493 leader_election.cc:304] T 276dee5db2d34451b642ed9359137ad4 P a8adcce962244bc688dc4ecc1f69612c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a8adcce962244bc688dc4ecc1f69612c, f8257d37af0941c2a978546ef2276b30; no voters: cc8d9fdc73b14a3cb716b47c24bafc1d
I20260709 12:28:18.621606 32731 raft_consensus.cc:2804] T 276dee5db2d34451b642ed9359137ad4 P a8adcce962244bc688dc4ecc1f69612c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:28:18.622334 32731 raft_consensus.cc:493] T 276dee5db2d34451b642ed9359137ad4 P a8adcce962244bc688dc4ecc1f69612c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:18.622938 32731 raft_consensus.cc:3060] T 276dee5db2d34451b642ed9359137ad4 P a8adcce962244bc688dc4ecc1f69612c [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:18.622239 32729 raft_consensus.cc:515] T db65db1003af45d4b6315902498f3f2b P f8257d37af0941c2a978546ef2276b30 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:18.624064 32601 raft_consensus.cc:2393] T db65db1003af45d4b6315902498f3f2b P f8257d37af0941c2a978546ef2276b30 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate cc8d9fdc73b14a3cb716b47c24bafc1d in current term 1: Already voted for candidate f8257d37af0941c2a978546ef2276b30 in this term.
I20260709 12:28:18.626163 32453 raft_consensus.cc:2468] T 7035e41df60347b188a3b8079aa7e551 P cc8d9fdc73b14a3cb716b47c24bafc1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a8adcce962244bc688dc4ecc1f69612c in term 1.
I20260709 12:28:18.612716 32764 leader_election.cc:290] T 276dee5db2d34451b642ed9359137ad4 P a8adcce962244bc688dc4ecc1f69612c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc8d9fdc73b14a3cb716b47c24bafc1d (127.16.29.65:41229), f8257d37af0941c2a978546ef2276b30 (127.16.29.67:37155)
I20260709 12:28:18.632793 32762 leader_election.cc:290] T 7035e41df60347b188a3b8079aa7e551 P a8adcce962244bc688dc4ecc1f69612c [CANDIDATE]: Term 1 election: Requested vote from peers cc8d9fdc73b14a3cb716b47c24bafc1d (127.16.29.65:41229), f8257d37af0941c2a978546ef2276b30 (127.16.29.67:37155)
I20260709 12:28:18.634321 32602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7035e41df60347b188a3b8079aa7e551" candidate_uuid: "a8adcce962244bc688dc4ecc1f69612c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8257d37af0941c2a978546ef2276b30"
I20260709 12:28:18.635088 32744 ts_tablet_manager.cc:1434] T 7035e41df60347b188a3b8079aa7e551 P cc8d9fdc73b14a3cb716b47c24bafc1d: Time spent starting tablet: real 0.028s	user 0.006s	sys 0.000s
I20260709 12:28:18.635327 32602 raft_consensus.cc:3060] T 7035e41df60347b188a3b8079aa7e551 P f8257d37af0941c2a978546ef2276b30 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:18.637842 32744 tablet_bootstrap.cc:492] T 276dee5db2d34451b642ed9359137ad4 P cc8d9fdc73b14a3cb716b47c24bafc1d: Bootstrap starting.
I20260709 12:28:18.639739 32731 raft_consensus.cc:515] T 276dee5db2d34451b642ed9359137ad4 P a8adcce962244bc688dc4ecc1f69612c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:18.643339 32494 leader_election.cc:304] T 7035e41df60347b188a3b8079aa7e551 P a8adcce962244bc688dc4ecc1f69612c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a8adcce962244bc688dc4ecc1f69612c, cc8d9fdc73b14a3cb716b47c24bafc1d; no voters: 
I20260709 12:28:18.644117 32746 tablet_bootstrap.cc:492] T 11a335d230b94355ad8859ba469b7989 P a8adcce962244bc688dc4ecc1f69612c: No bootstrap required, opened a new log
I20260709 12:28:18.602275 32452 raft_consensus.cc:2468] T 894a3f7744bd44ad814f22f90682c678 P cc8d9fdc73b14a3cb716b47c24bafc1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a8adcce962244bc688dc4ecc1f69612c in term 1.
I20260709 12:28:18.648069 32766 raft_consensus.cc:2804] T 7035e41df60347b188a3b8079aa7e551 P a8adcce962244bc688dc4ecc1f69612c [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:28:18.648526 32746 ts_tablet_manager.cc:1403] T 11a335d230b94355ad8859ba469b7989 P a8adcce962244bc688dc4ecc1f69612c: Time spent bootstrapping tablet: real 0.101s	user 0.032s	sys 0.030s
I20260709 12:28:18.648717 32766 raft_consensus.cc:697] T 7035e41df60347b188a3b8079aa7e551 P a8adcce962244bc688dc4ecc1f69612c [term 1 LEADER]: Becoming Leader. State: Replica: a8adcce962244bc688dc4ecc1f69612c, State: Running, Role: LEADER
I20260709 12:28:18.649322 32494 leader_election.cc:304] T 894a3f7744bd44ad814f22f90682c678 P a8adcce962244bc688dc4ecc1f69612c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a8adcce962244bc688dc4ecc1f69612c, cc8d9fdc73b14a3cb716b47c24bafc1d; no voters: f8257d37af0941c2a978546ef2276b30
I20260709 12:28:18.649778 32766 consensus_queue.cc:237] T 7035e41df60347b188a3b8079aa7e551 P a8adcce962244bc688dc4ecc1f69612c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:18.651219 32731 leader_election.cc:290] T 276dee5db2d34451b642ed9359137ad4 P a8adcce962244bc688dc4ecc1f69612c [CANDIDATE]: Term 1 election: Requested vote from peers cc8d9fdc73b14a3cb716b47c24bafc1d (127.16.29.65:41229), f8257d37af0941c2a978546ef2276b30 (127.16.29.67:37155)
I20260709 12:28:18.651386 32453 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "276dee5db2d34451b642ed9359137ad4" candidate_uuid: "a8adcce962244bc688dc4ecc1f69612c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d"
I20260709 12:28:18.652329 32762 raft_consensus.cc:2804] T 894a3f7744bd44ad814f22f90682c678 P a8adcce962244bc688dc4ecc1f69612c [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:28:18.651768 32746 raft_consensus.cc:359] T 11a335d230b94355ad8859ba469b7989 P a8adcce962244bc688dc4ecc1f69612c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:18.652848 32762 raft_consensus.cc:697] T 894a3f7744bd44ad814f22f90682c678 P a8adcce962244bc688dc4ecc1f69612c [term 1 LEADER]: Becoming Leader. State: Replica: a8adcce962244bc688dc4ecc1f69612c, State: Running, Role: LEADER
I20260709 12:28:18.653010 32746 raft_consensus.cc:385] T 11a335d230b94355ad8859ba469b7989 P a8adcce962244bc688dc4ecc1f69612c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:18.653407 32746 raft_consensus.cc:740] T 11a335d230b94355ad8859ba469b7989 P a8adcce962244bc688dc4ecc1f69612c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8adcce962244bc688dc4ecc1f69612c, State: Initialized, Role: FOLLOWER
I20260709 12:28:18.653467 32602 raft_consensus.cc:2468] T 7035e41df60347b188a3b8079aa7e551 P f8257d37af0941c2a978546ef2276b30 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a8adcce962244bc688dc4ecc1f69612c in term 1.
W20260709 12:28:18.653447 32494 leader_election.cc:343] T 276dee5db2d34451b642ed9359137ad4 P a8adcce962244bc688dc4ecc1f69612c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cc8d9fdc73b14a3cb716b47c24bafc1d (127.16.29.65:41229): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:28:18.654524 32746 consensus_queue.cc:260] T 11a335d230b94355ad8859ba469b7989 P a8adcce962244bc688dc4ecc1f69612c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:18.653764 32762 consensus_queue.cc:237] T 894a3f7744bd44ad814f22f90682c678 P a8adcce962244bc688dc4ecc1f69612c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:18.656770 32746 ts_tablet_manager.cc:1434] T 11a335d230b94355ad8859ba469b7989 P a8adcce962244bc688dc4ecc1f69612c: Time spent starting tablet: real 0.008s	user 0.007s	sys 0.000s
I20260709 12:28:18.657766 32746 tablet_bootstrap.cc:492] T 92a7506e828b47a28f4e2c080d05419b P a8adcce962244bc688dc4ecc1f69612c: Bootstrap starting.
I20260709 12:28:18.657663 32418 leader_election.cc:304] T db65db1003af45d4b6315902498f3f2b P cc8d9fdc73b14a3cb716b47c24bafc1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cc8d9fdc73b14a3cb716b47c24bafc1d; no voters: a8adcce962244bc688dc4ecc1f69612c, f8257d37af0941c2a978546ef2276b30
I20260709 12:28:18.659008 32726 raft_consensus.cc:2749] T db65db1003af45d4b6315902498f3f2b P cc8d9fdc73b14a3cb716b47c24bafc1d [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 12:28:18.662006 32744 tablet_bootstrap.cc:654] T 276dee5db2d34451b642ed9359137ad4 P cc8d9fdc73b14a3cb716b47c24bafc1d: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:18.662722 32602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "276dee5db2d34451b642ed9359137ad4" candidate_uuid: "a8adcce962244bc688dc4ecc1f69612c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8257d37af0941c2a978546ef2276b30"
I20260709 12:28:18.663388 32602 raft_consensus.cc:3060] T 276dee5db2d34451b642ed9359137ad4 P f8257d37af0941c2a978546ef2276b30 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:18.670636 32746 tablet_bootstrap.cc:654] T 92a7506e828b47a28f4e2c080d05419b P a8adcce962244bc688dc4ecc1f69612c: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:18.673403 32453 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "db65db1003af45d4b6315902498f3f2b" candidate_uuid: "f8257d37af0941c2a978546ef2276b30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d"
I20260709 12:28:18.673667 32729 leader_election.cc:290] T db65db1003af45d4b6315902498f3f2b P f8257d37af0941c2a978546ef2276b30 [CANDIDATE]: Term 1 election: Requested vote from peers cc8d9fdc73b14a3cb716b47c24bafc1d (127.16.29.65:41229), a8adcce962244bc688dc4ecc1f69612c (127.16.29.66:33369)
I20260709 12:28:18.674360 32453 raft_consensus.cc:2393] T db65db1003af45d4b6315902498f3f2b P cc8d9fdc73b14a3cb716b47c24bafc1d [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate f8257d37af0941c2a978546ef2276b30 in current term 1: Already voted for candidate cc8d9fdc73b14a3cb716b47c24bafc1d in this term.
I20260709 12:28:18.675606 32527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "db65db1003af45d4b6315902498f3f2b" candidate_uuid: "f8257d37af0941c2a978546ef2276b30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8adcce962244bc688dc4ecc1f69612c"
W20260709 12:28:18.677577 32567 leader_election.cc:343] T db65db1003af45d4b6315902498f3f2b P f8257d37af0941c2a978546ef2276b30 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a8adcce962244bc688dc4ecc1f69612c (127.16.29.66:33369): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:28:18.678229 32567 leader_election.cc:304] T db65db1003af45d4b6315902498f3f2b P f8257d37af0941c2a978546ef2276b30 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8257d37af0941c2a978546ef2276b30; no voters: a8adcce962244bc688dc4ecc1f69612c, cc8d9fdc73b14a3cb716b47c24bafc1d
I20260709 12:28:18.680117 32744 tablet_bootstrap.cc:492] T 276dee5db2d34451b642ed9359137ad4 P cc8d9fdc73b14a3cb716b47c24bafc1d: No bootstrap required, opened a new log
I20260709 12:28:18.681263 32744 ts_tablet_manager.cc:1403] T 276dee5db2d34451b642ed9359137ad4 P cc8d9fdc73b14a3cb716b47c24bafc1d: Time spent bootstrapping tablet: real 0.044s	user 0.019s	sys 0.001s
I20260709 12:28:18.679507 32729 raft_consensus.cc:2749] T db65db1003af45d4b6315902498f3f2b P f8257d37af0941c2a978546ef2276b30 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 12:28:18.685359 32746 tablet_bootstrap.cc:492] T 92a7506e828b47a28f4e2c080d05419b P a8adcce962244bc688dc4ecc1f69612c: No bootstrap required, opened a new log
I20260709 12:28:18.685809 32746 ts_tablet_manager.cc:1403] T 92a7506e828b47a28f4e2c080d05419b P a8adcce962244bc688dc4ecc1f69612c: Time spent bootstrapping tablet: real 0.028s	user 0.015s	sys 0.004s
I20260709 12:28:18.689277 32602 raft_consensus.cc:2468] T 276dee5db2d34451b642ed9359137ad4 P f8257d37af0941c2a978546ef2276b30 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a8adcce962244bc688dc4ecc1f69612c in term 1.
I20260709 12:28:18.690876 32744 raft_consensus.cc:359] T 276dee5db2d34451b642ed9359137ad4 P cc8d9fdc73b14a3cb716b47c24bafc1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:18.690971 32493 leader_election.cc:304] T 276dee5db2d34451b642ed9359137ad4 P a8adcce962244bc688dc4ecc1f69612c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a8adcce962244bc688dc4ecc1f69612c, f8257d37af0941c2a978546ef2276b30; no voters: cc8d9fdc73b14a3cb716b47c24bafc1d
I20260709 12:28:18.691749 32744 raft_consensus.cc:385] T 276dee5db2d34451b642ed9359137ad4 P cc8d9fdc73b14a3cb716b47c24bafc1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:18.692159 32744 raft_consensus.cc:740] T 276dee5db2d34451b642ed9359137ad4 P cc8d9fdc73b14a3cb716b47c24bafc1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc8d9fdc73b14a3cb716b47c24bafc1d, State: Initialized, Role: FOLLOWER
I20260709 12:28:18.692524 32766 raft_consensus.cc:2804] T 276dee5db2d34451b642ed9359137ad4 P a8adcce962244bc688dc4ecc1f69612c [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:28:18.693056 32766 raft_consensus.cc:697] T 276dee5db2d34451b642ed9359137ad4 P a8adcce962244bc688dc4ecc1f69612c [term 1 LEADER]: Becoming Leader. State: Replica: a8adcce962244bc688dc4ecc1f69612c, State: Running, Role: LEADER
I20260709 12:28:18.693440 32744 consensus_queue.cc:260] T 276dee5db2d34451b642ed9359137ad4 P cc8d9fdc73b14a3cb716b47c24bafc1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:18.693884 32766 consensus_queue.cc:237] T 276dee5db2d34451b642ed9359137ad4 P a8adcce962244bc688dc4ecc1f69612c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:18.695109 32746 raft_consensus.cc:359] T 92a7506e828b47a28f4e2c080d05419b P a8adcce962244bc688dc4ecc1f69612c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:18.696646 32744 ts_tablet_manager.cc:1434] T 276dee5db2d34451b642ed9359137ad4 P cc8d9fdc73b14a3cb716b47c24bafc1d: Time spent starting tablet: real 0.014s	user 0.008s	sys 0.000s
I20260709 12:28:18.697261 32746 raft_consensus.cc:385] T 92a7506e828b47a28f4e2c080d05419b P a8adcce962244bc688dc4ecc1f69612c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:18.697768 32746 raft_consensus.cc:740] T 92a7506e828b47a28f4e2c080d05419b P a8adcce962244bc688dc4ecc1f69612c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8adcce962244bc688dc4ecc1f69612c, State: Initialized, Role: FOLLOWER
I20260709 12:28:18.698765 32746 consensus_queue.cc:260] T 92a7506e828b47a28f4e2c080d05419b P a8adcce962244bc688dc4ecc1f69612c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:18.700986 32746 ts_tablet_manager.cc:1434] T 92a7506e828b47a28f4e2c080d05419b P a8adcce962244bc688dc4ecc1f69612c: Time spent starting tablet: real 0.015s	user 0.005s	sys 0.003s
I20260709 12:28:18.702003 32746 tablet_bootstrap.cc:492] T db65db1003af45d4b6315902498f3f2b P a8adcce962244bc688dc4ecc1f69612c: Bootstrap starting.
I20260709 12:28:18.704290 32764 raft_consensus.cc:493] T 11a335d230b94355ad8859ba469b7989 P a8adcce962244bc688dc4ecc1f69612c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:18.704847 32764 raft_consensus.cc:515] T 11a335d230b94355ad8859ba469b7989 P a8adcce962244bc688dc4ecc1f69612c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:18.707154 32764 leader_election.cc:290] T 11a335d230b94355ad8859ba469b7989 P a8adcce962244bc688dc4ecc1f69612c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc8d9fdc73b14a3cb716b47c24bafc1d (127.16.29.65:41229), f8257d37af0941c2a978546ef2276b30 (127.16.29.67:37155)
I20260709 12:28:18.708587 32746 tablet_bootstrap.cc:654] T db65db1003af45d4b6315902498f3f2b P a8adcce962244bc688dc4ecc1f69612c: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:18.710057 32602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11a335d230b94355ad8859ba469b7989" candidate_uuid: "a8adcce962244bc688dc4ecc1f69612c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8257d37af0941c2a978546ef2276b30" is_pre_election: true
I20260709 12:28:18.710783 32602 raft_consensus.cc:2468] T 11a335d230b94355ad8859ba469b7989 P f8257d37af0941c2a978546ef2276b30 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a8adcce962244bc688dc4ecc1f69612c in term 0.
I20260709 12:28:18.711715 32453 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11a335d230b94355ad8859ba469b7989" candidate_uuid: "a8adcce962244bc688dc4ecc1f69612c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" is_pre_election: true
I20260709 12:28:18.714031 32493 leader_election.cc:304] T 11a335d230b94355ad8859ba469b7989 P a8adcce962244bc688dc4ecc1f69612c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a8adcce962244bc688dc4ecc1f69612c, f8257d37af0941c2a978546ef2276b30; no voters: 
W20260709 12:28:18.715994 32494 leader_election.cc:343] T 11a335d230b94355ad8859ba469b7989 P a8adcce962244bc688dc4ecc1f69612c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cc8d9fdc73b14a3cb716b47c24bafc1d (127.16.29.65:41229): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:28:18.717232 32743 tablet_bootstrap.cc:492] T 894a3f7744bd44ad814f22f90682c678 P f8257d37af0941c2a978546ef2276b30: No bootstrap required, opened a new log
I20260709 12:28:18.717378 32764 raft_consensus.cc:2804] T 11a335d230b94355ad8859ba469b7989 P a8adcce962244bc688dc4ecc1f69612c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:28:18.718108 32743 ts_tablet_manager.cc:1403] T 894a3f7744bd44ad814f22f90682c678 P f8257d37af0941c2a978546ef2276b30: Time spent bootstrapping tablet: real 0.156s	user 0.017s	sys 0.000s
I20260709 12:28:18.718453 32764 raft_consensus.cc:493] T 11a335d230b94355ad8859ba469b7989 P a8adcce962244bc688dc4ecc1f69612c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:18.719551 32764 raft_consensus.cc:3060] T 11a335d230b94355ad8859ba469b7989 P a8adcce962244bc688dc4ecc1f69612c [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:18.728192 32726 raft_consensus.cc:493] T f9cc8a9e92e64d2083b47e3619f88e32 P cc8d9fdc73b14a3cb716b47c24bafc1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:18.728391 32743 raft_consensus.cc:359] T 894a3f7744bd44ad814f22f90682c678 P f8257d37af0941c2a978546ef2276b30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:18.728893 32729 raft_consensus.cc:493] T 92a7506e828b47a28f4e2c080d05419b P f8257d37af0941c2a978546ef2276b30 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:18.728808 32726 raft_consensus.cc:515] T f9cc8a9e92e64d2083b47e3619f88e32 P cc8d9fdc73b14a3cb716b47c24bafc1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:18.729293 32743 raft_consensus.cc:385] T 894a3f7744bd44ad814f22f90682c678 P f8257d37af0941c2a978546ef2276b30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:18.729676 32743 raft_consensus.cc:740] T 894a3f7744bd44ad814f22f90682c678 P f8257d37af0941c2a978546ef2276b30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8257d37af0941c2a978546ef2276b30, State: Initialized, Role: FOLLOWER
I20260709 12:28:18.729426 32729 raft_consensus.cc:515] T 92a7506e828b47a28f4e2c080d05419b P f8257d37af0941c2a978546ef2276b30 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:18.730422 32743 consensus_queue.cc:260] T 894a3f7744bd44ad814f22f90682c678 P f8257d37af0941c2a978546ef2276b30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:18.731456 32726 leader_election.cc:290] T f9cc8a9e92e64d2083b47e3619f88e32 P cc8d9fdc73b14a3cb716b47c24bafc1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a8adcce962244bc688dc4ecc1f69612c (127.16.29.66:33369), f8257d37af0941c2a978546ef2276b30 (127.16.29.67:37155)
I20260709 12:28:18.732949 32743 ts_tablet_manager.cc:1434] T 894a3f7744bd44ad814f22f90682c678 P f8257d37af0941c2a978546ef2276b30: Time spent starting tablet: real 0.014s	user 0.011s	sys 0.000s
I20260709 12:28:18.738909 32743 tablet_bootstrap.cc:492] T f9cc8a9e92e64d2083b47e3619f88e32 P f8257d37af0941c2a978546ef2276b30: Bootstrap starting.
I20260709 12:28:18.739233 32527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92a7506e828b47a28f4e2c080d05419b" candidate_uuid: "f8257d37af0941c2a978546ef2276b30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8adcce962244bc688dc4ecc1f69612c" is_pre_election: true
I20260709 12:28:18.740178 32527 raft_consensus.cc:2468] T 92a7506e828b47a28f4e2c080d05419b P a8adcce962244bc688dc4ecc1f69612c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8257d37af0941c2a978546ef2276b30 in term 0.
I20260709 12:28:18.744573 32744 tablet_bootstrap.cc:492] T 11a335d230b94355ad8859ba469b7989 P cc8d9fdc73b14a3cb716b47c24bafc1d: Bootstrap starting.
I20260709 12:28:18.744894 32567 leader_election.cc:304] T 92a7506e828b47a28f4e2c080d05419b P f8257d37af0941c2a978546ef2276b30 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a8adcce962244bc688dc4ecc1f69612c, f8257d37af0941c2a978546ef2276b30; no voters: 
I20260709 12:28:18.746127 32767 raft_consensus.cc:2804] T 92a7506e828b47a28f4e2c080d05419b P f8257d37af0941c2a978546ef2276b30 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:28:18.746518 32767 raft_consensus.cc:493] T 92a7506e828b47a28f4e2c080d05419b P f8257d37af0941c2a978546ef2276b30 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:18.746847 32767 raft_consensus.cc:3060] T 92a7506e828b47a28f4e2c080d05419b P f8257d37af0941c2a978546ef2276b30 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:18.747678 32743 tablet_bootstrap.cc:654] T f9cc8a9e92e64d2083b47e3619f88e32 P f8257d37af0941c2a978546ef2276b30: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:18.731563 32729 leader_election.cc:290] T 92a7506e828b47a28f4e2c080d05419b P f8257d37af0941c2a978546ef2276b30 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc8d9fdc73b14a3cb716b47c24bafc1d (127.16.29.65:41229), a8adcce962244bc688dc4ecc1f69612c (127.16.29.66:33369)
I20260709 12:28:18.757386 32347 catalog_manager.cc:5697] T 7035e41df60347b188a3b8079aa7e551 P a8adcce962244bc688dc4ecc1f69612c reported cstate change: term changed from 0 to 1, leader changed from <none> to a8adcce962244bc688dc4ecc1f69612c (127.16.29.66). New cstate: current_term: 1 leader_uuid: "a8adcce962244bc688dc4ecc1f69612c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:18.758507 32767 raft_consensus.cc:515] T 92a7506e828b47a28f4e2c080d05419b P f8257d37af0941c2a978546ef2276b30 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:18.759181 32764 raft_consensus.cc:515] T 11a335d230b94355ad8859ba469b7989 P a8adcce962244bc688dc4ecc1f69612c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:18.763640 32527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9cc8a9e92e64d2083b47e3619f88e32" candidate_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8adcce962244bc688dc4ecc1f69612c" is_pre_election: true
I20260709 12:28:18.763657 32767 leader_election.cc:290] T 92a7506e828b47a28f4e2c080d05419b P f8257d37af0941c2a978546ef2276b30 [CANDIDATE]: Term 1 election: Requested vote from peers cc8d9fdc73b14a3cb716b47c24bafc1d (127.16.29.65:41229), a8adcce962244bc688dc4ecc1f69612c (127.16.29.66:33369)
I20260709 12:28:18.765637 32528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92a7506e828b47a28f4e2c080d05419b" candidate_uuid: "f8257d37af0941c2a978546ef2276b30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8adcce962244bc688dc4ecc1f69612c"
I20260709 12:28:18.766276 32528 raft_consensus.cc:3060] T 92a7506e828b47a28f4e2c080d05419b P a8adcce962244bc688dc4ecc1f69612c [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:18.764496 32527 raft_consensus.cc:2468] T f9cc8a9e92e64d2083b47e3619f88e32 P a8adcce962244bc688dc4ecc1f69612c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc8d9fdc73b14a3cb716b47c24bafc1d in term 0.
I20260709 12:28:18.772282 32746 tablet_bootstrap.cc:492] T db65db1003af45d4b6315902498f3f2b P a8adcce962244bc688dc4ecc1f69612c: No bootstrap required, opened a new log
I20260709 12:28:18.773002 32746 ts_tablet_manager.cc:1403] T db65db1003af45d4b6315902498f3f2b P a8adcce962244bc688dc4ecc1f69612c: Time spent bootstrapping tablet: real 0.071s	user 0.031s	sys 0.016s
I20260709 12:28:18.773828 32452 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92a7506e828b47a28f4e2c080d05419b" candidate_uuid: "f8257d37af0941c2a978546ef2276b30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d"
I20260709 12:28:18.773756 32453 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92a7506e828b47a28f4e2c080d05419b" candidate_uuid: "f8257d37af0941c2a978546ef2276b30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" is_pre_election: true
I20260709 12:28:18.774528 32452 raft_consensus.cc:3060] T 92a7506e828b47a28f4e2c080d05419b P cc8d9fdc73b14a3cb716b47c24bafc1d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:18.772914 32602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9cc8a9e92e64d2083b47e3619f88e32" candidate_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8257d37af0941c2a978546ef2276b30" is_pre_election: true
I20260709 12:28:18.777273 32746 raft_consensus.cc:359] T db65db1003af45d4b6315902498f3f2b P a8adcce962244bc688dc4ecc1f69612c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:18.778446 32746 raft_consensus.cc:385] T db65db1003af45d4b6315902498f3f2b P a8adcce962244bc688dc4ecc1f69612c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:18.778803 32746 raft_consensus.cc:740] T db65db1003af45d4b6315902498f3f2b P a8adcce962244bc688dc4ecc1f69612c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8adcce962244bc688dc4ecc1f69612c, State: Initialized, Role: FOLLOWER
I20260709 12:28:18.779623 32746 consensus_queue.cc:260] T db65db1003af45d4b6315902498f3f2b P a8adcce962244bc688dc4ecc1f69612c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:18.781689 32528 raft_consensus.cc:2468] T 92a7506e828b47a28f4e2c080d05419b P a8adcce962244bc688dc4ecc1f69612c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8257d37af0941c2a978546ef2276b30 in term 1.
I20260709 12:28:18.783231 32746 ts_tablet_manager.cc:1434] T db65db1003af45d4b6315902498f3f2b P a8adcce962244bc688dc4ecc1f69612c: Time spent starting tablet: real 0.010s	user 0.009s	sys 0.000s
W20260709 12:28:18.784184 32418 leader_election.cc:343] T f9cc8a9e92e64d2083b47e3619f88e32 P cc8d9fdc73b14a3cb716b47c24bafc1d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f8257d37af0941c2a978546ef2276b30 (127.16.29.67:37155): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:28:18.784638 32567 leader_election.cc:304] T 92a7506e828b47a28f4e2c080d05419b P f8257d37af0941c2a978546ef2276b30 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a8adcce962244bc688dc4ecc1f69612c, f8257d37af0941c2a978546ef2276b30; no voters: 
I20260709 12:28:18.786125   309 raft_consensus.cc:493] T 51c3eb697520458093debb61eebb75a7 P cc8d9fdc73b14a3cb716b47c24bafc1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:18.786527 32767 raft_consensus.cc:2804] T 92a7506e828b47a28f4e2c080d05419b P f8257d37af0941c2a978546ef2276b30 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:28:18.786751 32418 leader_election.cc:304] T f9cc8a9e92e64d2083b47e3619f88e32 P cc8d9fdc73b14a3cb716b47c24bafc1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a8adcce962244bc688dc4ecc1f69612c, cc8d9fdc73b14a3cb716b47c24bafc1d; no voters: f8257d37af0941c2a978546ef2276b30
I20260709 12:28:18.787047 32767 raft_consensus.cc:697] T 92a7506e828b47a28f4e2c080d05419b P f8257d37af0941c2a978546ef2276b30 [term 1 LEADER]: Becoming Leader. State: Replica: f8257d37af0941c2a978546ef2276b30, State: Running, Role: LEADER
I20260709 12:28:18.787992 32767 consensus_queue.cc:237] T 92a7506e828b47a28f4e2c080d05419b P f8257d37af0941c2a978546ef2276b30 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:18.788133   309 raft_consensus.cc:515] T 51c3eb697520458093debb61eebb75a7 P cc8d9fdc73b14a3cb716b47c24bafc1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:18.789290 32453 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11a335d230b94355ad8859ba469b7989" candidate_uuid: "a8adcce962244bc688dc4ecc1f69612c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d"
I20260709 12:28:18.792189 32743 tablet_bootstrap.cc:492] T f9cc8a9e92e64d2083b47e3619f88e32 P f8257d37af0941c2a978546ef2276b30: No bootstrap required, opened a new log
I20260709 12:28:18.795897 32602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "51c3eb697520458093debb61eebb75a7" candidate_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8257d37af0941c2a978546ef2276b30" is_pre_election: true
I20260709 12:28:18.798928 32743 ts_tablet_manager.cc:1403] T f9cc8a9e92e64d2083b47e3619f88e32 P f8257d37af0941c2a978546ef2276b30: Time spent bootstrapping tablet: real 0.060s	user 0.025s	sys 0.021s
W20260709 12:28:18.790535 32494 leader_election.cc:343] T 11a335d230b94355ad8859ba469b7989 P a8adcce962244bc688dc4ecc1f69612c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cc8d9fdc73b14a3cb716b47c24bafc1d (127.16.29.65:41229): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:28:18.800760 32729 raft_consensus.cc:493] T 894a3f7744bd44ad814f22f90682c678 P f8257d37af0941c2a978546ef2276b30 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:18.801311 32729 raft_consensus.cc:515] T 894a3f7744bd44ad814f22f90682c678 P f8257d37af0941c2a978546ef2276b30 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:18.795696 32744 tablet_bootstrap.cc:654] T 11a335d230b94355ad8859ba469b7989 P cc8d9fdc73b14a3cb716b47c24bafc1d: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:18.803496 32729 leader_election.cc:290] T 894a3f7744bd44ad814f22f90682c678 P f8257d37af0941c2a978546ef2276b30 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc8d9fdc73b14a3cb716b47c24bafc1d (127.16.29.65:41229), a8adcce962244bc688dc4ecc1f69612c (127.16.29.66:33369)
I20260709 12:28:18.804517 32602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11a335d230b94355ad8859ba469b7989" candidate_uuid: "a8adcce962244bc688dc4ecc1f69612c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8257d37af0941c2a978546ef2276b30"
I20260709 12:28:18.805205 32602 raft_consensus.cc:3060] T 11a335d230b94355ad8859ba469b7989 P f8257d37af0941c2a978546ef2276b30 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:18.805773 32527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "894a3f7744bd44ad814f22f90682c678" candidate_uuid: "f8257d37af0941c2a978546ef2276b30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8adcce962244bc688dc4ecc1f69612c" is_pre_election: true
I20260709 12:28:18.802621 32743 raft_consensus.cc:359] T f9cc8a9e92e64d2083b47e3619f88e32 P f8257d37af0941c2a978546ef2276b30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:18.807292 32743 raft_consensus.cc:385] T f9cc8a9e92e64d2083b47e3619f88e32 P f8257d37af0941c2a978546ef2276b30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:18.807773 32743 raft_consensus.cc:740] T f9cc8a9e92e64d2083b47e3619f88e32 P f8257d37af0941c2a978546ef2276b30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8257d37af0941c2a978546ef2276b30, State: Initialized, Role: FOLLOWER
I20260709 12:28:18.790453   309 leader_election.cc:290] T 51c3eb697520458093debb61eebb75a7 P cc8d9fdc73b14a3cb716b47c24bafc1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a8adcce962244bc688dc4ecc1f69612c (127.16.29.66:33369), f8257d37af0941c2a978546ef2276b30 (127.16.29.67:37155)
I20260709 12:28:18.790781 32726 raft_consensus.cc:2804] T f9cc8a9e92e64d2083b47e3619f88e32 P cc8d9fdc73b14a3cb716b47c24bafc1d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:28:18.809509 32726 raft_consensus.cc:493] T f9cc8a9e92e64d2083b47e3619f88e32 P cc8d9fdc73b14a3cb716b47c24bafc1d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:18.808866 32743 consensus_queue.cc:260] T f9cc8a9e92e64d2083b47e3619f88e32 P f8257d37af0941c2a978546ef2276b30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:18.809883 32726 raft_consensus.cc:3060] T f9cc8a9e92e64d2083b47e3619f88e32 P cc8d9fdc73b14a3cb716b47c24bafc1d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:18.812001 32764 leader_election.cc:290] T 11a335d230b94355ad8859ba469b7989 P a8adcce962244bc688dc4ecc1f69612c [CANDIDATE]: Term 1 election: Requested vote from peers cc8d9fdc73b14a3cb716b47c24bafc1d (127.16.29.65:41229), f8257d37af0941c2a978546ef2276b30 (127.16.29.67:37155)
I20260709 12:28:18.789249 32452 raft_consensus.cc:2468] T 92a7506e828b47a28f4e2c080d05419b P cc8d9fdc73b14a3cb716b47c24bafc1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8257d37af0941c2a978546ef2276b30 in term 1.
I20260709 12:28:18.785935 32347 catalog_manager.cc:5697] T 894a3f7744bd44ad814f22f90682c678 P a8adcce962244bc688dc4ecc1f69612c reported cstate change: term changed from 0 to 1, leader changed from <none> to a8adcce962244bc688dc4ecc1f69612c (127.16.29.66). New cstate: current_term: 1 leader_uuid: "a8adcce962244bc688dc4ecc1f69612c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:18.812381 32743 ts_tablet_manager.cc:1434] T f9cc8a9e92e64d2083b47e3619f88e32 P f8257d37af0941c2a978546ef2276b30: Time spent starting tablet: real 0.013s	user 0.006s	sys 0.003s
I20260709 12:28:18.814268 32528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "51c3eb697520458093debb61eebb75a7" candidate_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8adcce962244bc688dc4ecc1f69612c" is_pre_election: true
I20260709 12:28:18.815029 32602 raft_consensus.cc:2468] T 11a335d230b94355ad8859ba469b7989 P f8257d37af0941c2a978546ef2276b30 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a8adcce962244bc688dc4ecc1f69612c in term 1.
I20260709 12:28:18.823485 32493 leader_election.cc:304] T 11a335d230b94355ad8859ba469b7989 P a8adcce962244bc688dc4ecc1f69612c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a8adcce962244bc688dc4ecc1f69612c, f8257d37af0941c2a978546ef2276b30; no voters: cc8d9fdc73b14a3cb716b47c24bafc1d
I20260709 12:28:18.824407 32766 raft_consensus.cc:2804] T 11a335d230b94355ad8859ba469b7989 P a8adcce962244bc688dc4ecc1f69612c [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:28:18.824942 32766 raft_consensus.cc:697] T 11a335d230b94355ad8859ba469b7989 P a8adcce962244bc688dc4ecc1f69612c [term 1 LEADER]: Becoming Leader. State: Replica: a8adcce962244bc688dc4ecc1f69612c, State: Running, Role: LEADER
I20260709 12:28:18.825280 32726 raft_consensus.cc:515] T f9cc8a9e92e64d2083b47e3619f88e32 P cc8d9fdc73b14a3cb716b47c24bafc1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:18.825910 32766 consensus_queue.cc:237] T 11a335d230b94355ad8859ba469b7989 P a8adcce962244bc688dc4ecc1f69612c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:18.819305 32453 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "894a3f7744bd44ad814f22f90682c678" candidate_uuid: "f8257d37af0941c2a978546ef2276b30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" is_pre_election: true
I20260709 12:28:18.828843 32453 raft_consensus.cc:2393] T 894a3f7744bd44ad814f22f90682c678 P cc8d9fdc73b14a3cb716b47c24bafc1d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f8257d37af0941c2a978546ef2276b30 in current term 1: Already voted for candidate a8adcce962244bc688dc4ecc1f69612c in this term.
W20260709 12:28:18.819164 32418 leader_election.cc:343] T 51c3eb697520458093debb61eebb75a7 P cc8d9fdc73b14a3cb716b47c24bafc1d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f8257d37af0941c2a978546ef2276b30 (127.16.29.67:37155): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:28:18.832257 32568 leader_election.cc:304] T 894a3f7744bd44ad814f22f90682c678 P f8257d37af0941c2a978546ef2276b30 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8257d37af0941c2a978546ef2276b30; no voters: a8adcce962244bc688dc4ecc1f69612c, cc8d9fdc73b14a3cb716b47c24bafc1d
I20260709 12:28:18.832443 32527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9cc8a9e92e64d2083b47e3619f88e32" candidate_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8adcce962244bc688dc4ecc1f69612c"
I20260709 12:28:18.833210 32527 raft_consensus.cc:3060] T f9cc8a9e92e64d2083b47e3619f88e32 P a8adcce962244bc688dc4ecc1f69612c [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:18.834075 32602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9cc8a9e92e64d2083b47e3619f88e32" candidate_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8257d37af0941c2a978546ef2276b30"
I20260709 12:28:18.834897 32602 raft_consensus.cc:3060] T f9cc8a9e92e64d2083b47e3619f88e32 P f8257d37af0941c2a978546ef2276b30 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:18.819865 32528 raft_consensus.cc:2468] T 51c3eb697520458093debb61eebb75a7 P a8adcce962244bc688dc4ecc1f69612c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc8d9fdc73b14a3cb716b47c24bafc1d in term 0.
I20260709 12:28:18.841751 32418 leader_election.cc:304] T 51c3eb697520458093debb61eebb75a7 P cc8d9fdc73b14a3cb716b47c24bafc1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a8adcce962244bc688dc4ecc1f69612c, cc8d9fdc73b14a3cb716b47c24bafc1d; no voters: f8257d37af0941c2a978546ef2276b30
I20260709 12:28:18.844069 32767 raft_consensus.cc:3060] T 894a3f7744bd44ad814f22f90682c678 P f8257d37af0941c2a978546ef2276b30 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:18.817305 32743 tablet_bootstrap.cc:492] T 51c3eb697520458093debb61eebb75a7 P f8257d37af0941c2a978546ef2276b30: Bootstrap starting.
I20260709 12:28:18.852061 32602 raft_consensus.cc:2468] T f9cc8a9e92e64d2083b47e3619f88e32 P f8257d37af0941c2a978546ef2276b30 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc8d9fdc73b14a3cb716b47c24bafc1d in term 1.
I20260709 12:28:18.852218   309 raft_consensus.cc:2804] T 51c3eb697520458093debb61eebb75a7 P cc8d9fdc73b14a3cb716b47c24bafc1d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:28:18.852888   309 raft_consensus.cc:493] T 51c3eb697520458093debb61eebb75a7 P cc8d9fdc73b14a3cb716b47c24bafc1d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:18.853336   309 raft_consensus.cc:3060] T 51c3eb697520458093debb61eebb75a7 P cc8d9fdc73b14a3cb716b47c24bafc1d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:18.854660 32418 leader_election.cc:304] T f9cc8a9e92e64d2083b47e3619f88e32 P cc8d9fdc73b14a3cb716b47c24bafc1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cc8d9fdc73b14a3cb716b47c24bafc1d, f8257d37af0941c2a978546ef2276b30; no voters: 
I20260709 12:28:18.856159   308 raft_consensus.cc:2804] T f9cc8a9e92e64d2083b47e3619f88e32 P cc8d9fdc73b14a3cb716b47c24bafc1d [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:28:18.861788 32767 raft_consensus.cc:2749] T 894a3f7744bd44ad814f22f90682c678 P f8257d37af0941c2a978546ef2276b30 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:28:18.844722 32527 raft_consensus.cc:2468] T f9cc8a9e92e64d2083b47e3619f88e32 P a8adcce962244bc688dc4ecc1f69612c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc8d9fdc73b14a3cb716b47c24bafc1d in term 1.
I20260709 12:28:18.868821 32743 tablet_bootstrap.cc:654] T 51c3eb697520458093debb61eebb75a7 P f8257d37af0941c2a978546ef2276b30: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:18.827576 32726 leader_election.cc:290] T f9cc8a9e92e64d2083b47e3619f88e32 P cc8d9fdc73b14a3cb716b47c24bafc1d [CANDIDATE]: Term 1 election: Requested vote from peers a8adcce962244bc688dc4ecc1f69612c (127.16.29.66:33369), f8257d37af0941c2a978546ef2276b30 (127.16.29.67:37155)
I20260709 12:28:18.869269   308 raft_consensus.cc:697] T f9cc8a9e92e64d2083b47e3619f88e32 P cc8d9fdc73b14a3cb716b47c24bafc1d [term 1 LEADER]: Becoming Leader. State: Replica: cc8d9fdc73b14a3cb716b47c24bafc1d, State: Running, Role: LEADER
I20260709 12:28:18.870702 32744 tablet_bootstrap.cc:492] T 11a335d230b94355ad8859ba469b7989 P cc8d9fdc73b14a3cb716b47c24bafc1d: No bootstrap required, opened a new log
I20260709 12:28:18.870308   308 consensus_queue.cc:237] T f9cc8a9e92e64d2083b47e3619f88e32 P cc8d9fdc73b14a3cb716b47c24bafc1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:18.871300 32744 ts_tablet_manager.cc:1403] T 11a335d230b94355ad8859ba469b7989 P cc8d9fdc73b14a3cb716b47c24bafc1d: Time spent bootstrapping tablet: real 0.167s	user 0.032s	sys 0.024s
I20260709 12:28:18.874689 32744 raft_consensus.cc:359] T 11a335d230b94355ad8859ba469b7989 P cc8d9fdc73b14a3cb716b47c24bafc1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:18.875499 32744 raft_consensus.cc:385] T 11a335d230b94355ad8859ba469b7989 P cc8d9fdc73b14a3cb716b47c24bafc1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:18.875810 32744 raft_consensus.cc:740] T 11a335d230b94355ad8859ba469b7989 P cc8d9fdc73b14a3cb716b47c24bafc1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc8d9fdc73b14a3cb716b47c24bafc1d, State: Initialized, Role: FOLLOWER
I20260709 12:28:18.876603 32744 consensus_queue.cc:260] T 11a335d230b94355ad8859ba469b7989 P cc8d9fdc73b14a3cb716b47c24bafc1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:18.878842 32744 ts_tablet_manager.cc:1434] T 11a335d230b94355ad8859ba469b7989 P cc8d9fdc73b14a3cb716b47c24bafc1d: Time spent starting tablet: real 0.007s	user 0.007s	sys 0.000s
I20260709 12:28:18.880329 32744 tablet_bootstrap.cc:492] T f998d4ae7a1e419c9e457994500ce4d1 P cc8d9fdc73b14a3cb716b47c24bafc1d: Bootstrap starting.
I20260709 12:28:18.887097   309 raft_consensus.cc:515] T 51c3eb697520458093debb61eebb75a7 P cc8d9fdc73b14a3cb716b47c24bafc1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:18.888427 32744 tablet_bootstrap.cc:654] T f998d4ae7a1e419c9e457994500ce4d1 P cc8d9fdc73b14a3cb716b47c24bafc1d: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:18.890348   309 leader_election.cc:290] T 51c3eb697520458093debb61eebb75a7 P cc8d9fdc73b14a3cb716b47c24bafc1d [CANDIDATE]: Term 1 election: Requested vote from peers a8adcce962244bc688dc4ecc1f69612c (127.16.29.66:33369), f8257d37af0941c2a978546ef2276b30 (127.16.29.67:37155)
I20260709 12:28:18.893216 32743 tablet_bootstrap.cc:492] T 51c3eb697520458093debb61eebb75a7 P f8257d37af0941c2a978546ef2276b30: No bootstrap required, opened a new log
I20260709 12:28:18.893801 32743 ts_tablet_manager.cc:1403] T 51c3eb697520458093debb61eebb75a7 P f8257d37af0941c2a978546ef2276b30: Time spent bootstrapping tablet: real 0.077s	user 0.014s	sys 0.009s
I20260709 12:28:18.907567 32766 raft_consensus.cc:493] T f998d4ae7a1e419c9e457994500ce4d1 P a8adcce962244bc688dc4ecc1f69612c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:18.908227 32743 raft_consensus.cc:359] T 51c3eb697520458093debb61eebb75a7 P f8257d37af0941c2a978546ef2276b30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:18.908524 32766 raft_consensus.cc:515] T f998d4ae7a1e419c9e457994500ce4d1 P a8adcce962244bc688dc4ecc1f69612c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:18.908993 32743 raft_consensus.cc:385] T 51c3eb697520458093debb61eebb75a7 P f8257d37af0941c2a978546ef2276b30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:18.909283 32743 raft_consensus.cc:740] T 51c3eb697520458093debb61eebb75a7 P f8257d37af0941c2a978546ef2276b30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8257d37af0941c2a978546ef2276b30, State: Initialized, Role: FOLLOWER
I20260709 12:28:18.910091 32743 consensus_queue.cc:260] T 51c3eb697520458093debb61eebb75a7 P f8257d37af0941c2a978546ef2276b30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:18.912097 32600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f998d4ae7a1e419c9e457994500ce4d1" candidate_uuid: "a8adcce962244bc688dc4ecc1f69612c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8257d37af0941c2a978546ef2276b30" is_pre_election: true
W20260709 12:28:18.914256 32493 leader_election.cc:343] T f998d4ae7a1e419c9e457994500ce4d1 P a8adcce962244bc688dc4ecc1f69612c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f8257d37af0941c2a978546ef2276b30 (127.16.29.67:37155): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:28:18.916230 32453 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f998d4ae7a1e419c9e457994500ce4d1" candidate_uuid: "a8adcce962244bc688dc4ecc1f69612c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" is_pre_election: true
W20260709 12:28:18.917862 32494 leader_election.cc:343] T f998d4ae7a1e419c9e457994500ce4d1 P a8adcce962244bc688dc4ecc1f69612c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cc8d9fdc73b14a3cb716b47c24bafc1d (127.16.29.65:41229): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:28:18.918298 32494 leader_election.cc:304] T f998d4ae7a1e419c9e457994500ce4d1 P a8adcce962244bc688dc4ecc1f69612c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a8adcce962244bc688dc4ecc1f69612c; no voters: cc8d9fdc73b14a3cb716b47c24bafc1d, f8257d37af0941c2a978546ef2276b30
I20260709 12:28:18.919169 32731 raft_consensus.cc:2749] T f998d4ae7a1e419c9e457994500ce4d1 P a8adcce962244bc688dc4ecc1f69612c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:28:18.919972 32743 ts_tablet_manager.cc:1434] T 51c3eb697520458093debb61eebb75a7 P f8257d37af0941c2a978546ef2276b30: Time spent starting tablet: real 0.026s	user 0.010s	sys 0.000s
I20260709 12:28:18.921094 32743 tablet_bootstrap.cc:492] T f998d4ae7a1e419c9e457994500ce4d1 P f8257d37af0941c2a978546ef2276b30: Bootstrap starting.
I20260709 12:28:18.927088 32743 tablet_bootstrap.cc:654] T f998d4ae7a1e419c9e457994500ce4d1 P f8257d37af0941c2a978546ef2276b30: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:18.927085 32766 leader_election.cc:290] T f998d4ae7a1e419c9e457994500ce4d1 P a8adcce962244bc688dc4ecc1f69612c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc8d9fdc73b14a3cb716b47c24bafc1d (127.16.29.65:41229), f8257d37af0941c2a978546ef2276b30 (127.16.29.67:37155)
I20260709 12:28:18.932313 32527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "51c3eb697520458093debb61eebb75a7" candidate_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8adcce962244bc688dc4ecc1f69612c"
I20260709 12:28:18.931546 32348 catalog_manager.cc:5697] T 11a335d230b94355ad8859ba469b7989 P a8adcce962244bc688dc4ecc1f69612c reported cstate change: term changed from 0 to 1, leader changed from <none> to a8adcce962244bc688dc4ecc1f69612c (127.16.29.66). New cstate: current_term: 1 leader_uuid: "a8adcce962244bc688dc4ecc1f69612c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:18.933220 32527 raft_consensus.cc:3060] T 51c3eb697520458093debb61eebb75a7 P a8adcce962244bc688dc4ecc1f69612c [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:18.934284 32348 catalog_manager.cc:5697] T 276dee5db2d34451b642ed9359137ad4 P a8adcce962244bc688dc4ecc1f69612c reported cstate change: term changed from 0 to 1, leader changed from <none> to a8adcce962244bc688dc4ecc1f69612c (127.16.29.66). New cstate: current_term: 1 leader_uuid: "a8adcce962244bc688dc4ecc1f69612c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:18.945034 32347 catalog_manager.cc:5697] T f9cc8a9e92e64d2083b47e3619f88e32 P cc8d9fdc73b14a3cb716b47c24bafc1d reported cstate change: term changed from 0 to 1, leader changed from <none> to cc8d9fdc73b14a3cb716b47c24bafc1d (127.16.29.65). New cstate: current_term: 1 leader_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:18.951551 32600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "51c3eb697520458093debb61eebb75a7" candidate_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8257d37af0941c2a978546ef2276b30"
I20260709 12:28:18.952574 32600 raft_consensus.cc:3060] T 51c3eb697520458093debb61eebb75a7 P f8257d37af0941c2a978546ef2276b30 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:18.952633 32527 raft_consensus.cc:2468] T 51c3eb697520458093debb61eebb75a7 P a8adcce962244bc688dc4ecc1f69612c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc8d9fdc73b14a3cb716b47c24bafc1d in term 1.
I20260709 12:28:18.954903 32418 leader_election.cc:304] T 51c3eb697520458093debb61eebb75a7 P cc8d9fdc73b14a3cb716b47c24bafc1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a8adcce962244bc688dc4ecc1f69612c, cc8d9fdc73b14a3cb716b47c24bafc1d; no voters: 
I20260709 12:28:18.956285   308 raft_consensus.cc:2804] T 51c3eb697520458093debb61eebb75a7 P cc8d9fdc73b14a3cb716b47c24bafc1d [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:28:18.956969   308 raft_consensus.cc:697] T 51c3eb697520458093debb61eebb75a7 P cc8d9fdc73b14a3cb716b47c24bafc1d [term 1 LEADER]: Becoming Leader. State: Replica: cc8d9fdc73b14a3cb716b47c24bafc1d, State: Running, Role: LEADER
I20260709 12:28:18.958201   308 consensus_queue.cc:237] T 51c3eb697520458093debb61eebb75a7 P cc8d9fdc73b14a3cb716b47c24bafc1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:18.979861 32744 tablet_bootstrap.cc:492] T f998d4ae7a1e419c9e457994500ce4d1 P cc8d9fdc73b14a3cb716b47c24bafc1d: No bootstrap required, opened a new log
I20260709 12:28:18.980470 32744 ts_tablet_manager.cc:1403] T f998d4ae7a1e419c9e457994500ce4d1 P cc8d9fdc73b14a3cb716b47c24bafc1d: Time spent bootstrapping tablet: real 0.101s	user 0.051s	sys 0.042s
I20260709 12:28:18.981362 32743 tablet_bootstrap.cc:492] T f998d4ae7a1e419c9e457994500ce4d1 P f8257d37af0941c2a978546ef2276b30: No bootstrap required, opened a new log
I20260709 12:28:18.981938 32743 ts_tablet_manager.cc:1403] T f998d4ae7a1e419c9e457994500ce4d1 P f8257d37af0941c2a978546ef2276b30: Time spent bootstrapping tablet: real 0.061s	user 0.031s	sys 0.026s
I20260709 12:28:18.983415 32744 raft_consensus.cc:359] T f998d4ae7a1e419c9e457994500ce4d1 P cc8d9fdc73b14a3cb716b47c24bafc1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:18.984225 32744 raft_consensus.cc:385] T f998d4ae7a1e419c9e457994500ce4d1 P cc8d9fdc73b14a3cb716b47c24bafc1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:18.984200 32743 raft_consensus.cc:359] T f998d4ae7a1e419c9e457994500ce4d1 P f8257d37af0941c2a978546ef2276b30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:18.984700 32744 raft_consensus.cc:740] T f998d4ae7a1e419c9e457994500ce4d1 P cc8d9fdc73b14a3cb716b47c24bafc1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc8d9fdc73b14a3cb716b47c24bafc1d, State: Initialized, Role: FOLLOWER
I20260709 12:28:18.984920 32743 raft_consensus.cc:385] T f998d4ae7a1e419c9e457994500ce4d1 P f8257d37af0941c2a978546ef2276b30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:18.985280 32743 raft_consensus.cc:740] T f998d4ae7a1e419c9e457994500ce4d1 P f8257d37af0941c2a978546ef2276b30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8257d37af0941c2a978546ef2276b30, State: Initialized, Role: FOLLOWER
I20260709 12:28:18.985476 32744 consensus_queue.cc:260] T f998d4ae7a1e419c9e457994500ce4d1 P cc8d9fdc73b14a3cb716b47c24bafc1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:18.985962 32743 consensus_queue.cc:260] T f998d4ae7a1e419c9e457994500ce4d1 P f8257d37af0941c2a978546ef2276b30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:18.987632 32744 ts_tablet_manager.cc:1434] T f998d4ae7a1e419c9e457994500ce4d1 P cc8d9fdc73b14a3cb716b47c24bafc1d: Time spent starting tablet: real 0.007s	user 0.004s	sys 0.000s
I20260709 12:28:18.988111 32743 ts_tablet_manager.cc:1434] T f998d4ae7a1e419c9e457994500ce4d1 P f8257d37af0941c2a978546ef2276b30: Time spent starting tablet: real 0.006s	user 0.008s	sys 0.000s
I20260709 12:28:18.990490 32600 raft_consensus.cc:2468] T 51c3eb697520458093debb61eebb75a7 P f8257d37af0941c2a978546ef2276b30 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc8d9fdc73b14a3cb716b47c24bafc1d in term 1.
I20260709 12:28:18.992532 32349 catalog_manager.cc:5697] T 92a7506e828b47a28f4e2c080d05419b P f8257d37af0941c2a978546ef2276b30 reported cstate change: term changed from 0 to 1, leader changed from <none> to f8257d37af0941c2a978546ef2276b30 (127.16.29.67). New cstate: current_term: 1 leader_uuid: "f8257d37af0941c2a978546ef2276b30" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } health_report { overall_health: HEALTHY } } }
I20260709 12:28:18.994406 32348 catalog_manager.cc:5697] T 51c3eb697520458093debb61eebb75a7 P cc8d9fdc73b14a3cb716b47c24bafc1d reported cstate change: term changed from 0 to 1, leader changed from <none> to cc8d9fdc73b14a3cb716b47c24bafc1d (127.16.29.65). New cstate: current_term: 1 leader_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:19.050194   308 raft_consensus.cc:493] T 11a335d230b94355ad8859ba469b7989 P cc8d9fdc73b14a3cb716b47c24bafc1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:19.050926   308 raft_consensus.cc:515] T 11a335d230b94355ad8859ba469b7989 P cc8d9fdc73b14a3cb716b47c24bafc1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:19.052999   308 leader_election.cc:290] T 11a335d230b94355ad8859ba469b7989 P cc8d9fdc73b14a3cb716b47c24bafc1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a8adcce962244bc688dc4ecc1f69612c (127.16.29.66:33369), f8257d37af0941c2a978546ef2276b30 (127.16.29.67:37155)
I20260709 12:28:19.053912 32527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11a335d230b94355ad8859ba469b7989" candidate_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8adcce962244bc688dc4ecc1f69612c" is_pre_election: true
I20260709 12:28:19.054605 32600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11a335d230b94355ad8859ba469b7989" candidate_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8257d37af0941c2a978546ef2276b30" is_pre_election: true
I20260709 12:28:19.055558 32600 raft_consensus.cc:2393] T 11a335d230b94355ad8859ba469b7989 P f8257d37af0941c2a978546ef2276b30 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cc8d9fdc73b14a3cb716b47c24bafc1d in current term 1: Already voted for candidate a8adcce962244bc688dc4ecc1f69612c in this term.
I20260709 12:28:19.056563 32418 leader_election.cc:304] T 11a335d230b94355ad8859ba469b7989 P cc8d9fdc73b14a3cb716b47c24bafc1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cc8d9fdc73b14a3cb716b47c24bafc1d; no voters: a8adcce962244bc688dc4ecc1f69612c, f8257d37af0941c2a978546ef2276b30
I20260709 12:28:19.057178   308 raft_consensus.cc:3060] T 11a335d230b94355ad8859ba469b7989 P cc8d9fdc73b14a3cb716b47c24bafc1d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:19.061803 32766 consensus_queue.cc:1048] T 7035e41df60347b188a3b8079aa7e551 P a8adcce962244bc688dc4ecc1f69612c [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:28:19.064000   308 raft_consensus.cc:2749] T 11a335d230b94355ad8859ba469b7989 P cc8d9fdc73b14a3cb716b47c24bafc1d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:28:19.075359 32764 consensus_queue.cc:1048] T 7035e41df60347b188a3b8079aa7e551 P a8adcce962244bc688dc4ecc1f69612c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:28:19.125573 32764 consensus_queue.cc:1048] T 276dee5db2d34451b642ed9359137ad4 P a8adcce962244bc688dc4ecc1f69612c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:28:19.135423 32453 raft_consensus.cc:3060] T 276dee5db2d34451b642ed9359137ad4 P cc8d9fdc73b14a3cb716b47c24bafc1d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:19.143666 32766 consensus_queue.cc:1048] T 276dee5db2d34451b642ed9359137ad4 P a8adcce962244bc688dc4ecc1f69612c [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:28:19.144093 32731 consensus_queue.cc:1048] T 894a3f7744bd44ad814f22f90682c678 P a8adcce962244bc688dc4ecc1f69612c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:28:19.157097 32764 consensus_queue.cc:1048] T 894a3f7744bd44ad814f22f90682c678 P a8adcce962244bc688dc4ecc1f69612c [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:28:19.178335   309 raft_consensus.cc:493] T f998d4ae7a1e419c9e457994500ce4d1 P cc8d9fdc73b14a3cb716b47c24bafc1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:19.178833   309 raft_consensus.cc:515] T f998d4ae7a1e419c9e457994500ce4d1 P cc8d9fdc73b14a3cb716b47c24bafc1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:19.180585   309 leader_election.cc:290] T f998d4ae7a1e419c9e457994500ce4d1 P cc8d9fdc73b14a3cb716b47c24bafc1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a8adcce962244bc688dc4ecc1f69612c (127.16.29.66:33369), f8257d37af0941c2a978546ef2276b30 (127.16.29.67:37155)
I20260709 12:28:19.181432 32528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f998d4ae7a1e419c9e457994500ce4d1" candidate_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8adcce962244bc688dc4ecc1f69612c" is_pre_election: true
I20260709 12:28:19.181704 32602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f998d4ae7a1e419c9e457994500ce4d1" candidate_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8257d37af0941c2a978546ef2276b30" is_pre_election: true
I20260709 12:28:19.182047 32528 raft_consensus.cc:2468] T f998d4ae7a1e419c9e457994500ce4d1 P a8adcce962244bc688dc4ecc1f69612c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc8d9fdc73b14a3cb716b47c24bafc1d in term 0.
I20260709 12:28:19.182494 32602 raft_consensus.cc:2468] T f998d4ae7a1e419c9e457994500ce4d1 P f8257d37af0941c2a978546ef2276b30 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc8d9fdc73b14a3cb716b47c24bafc1d in term 0.
I20260709 12:28:19.183209 32418 leader_election.cc:304] T f998d4ae7a1e419c9e457994500ce4d1 P cc8d9fdc73b14a3cb716b47c24bafc1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a8adcce962244bc688dc4ecc1f69612c, cc8d9fdc73b14a3cb716b47c24bafc1d; no voters: 
I20260709 12:28:19.184017   309 raft_consensus.cc:2804] T f998d4ae7a1e419c9e457994500ce4d1 P cc8d9fdc73b14a3cb716b47c24bafc1d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:28:19.184321   309 raft_consensus.cc:493] T f998d4ae7a1e419c9e457994500ce4d1 P cc8d9fdc73b14a3cb716b47c24bafc1d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:19.184653   309 raft_consensus.cc:3060] T f998d4ae7a1e419c9e457994500ce4d1 P cc8d9fdc73b14a3cb716b47c24bafc1d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:19.189829   309 raft_consensus.cc:515] T f998d4ae7a1e419c9e457994500ce4d1 P cc8d9fdc73b14a3cb716b47c24bafc1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:19.191452   309 leader_election.cc:290] T f998d4ae7a1e419c9e457994500ce4d1 P cc8d9fdc73b14a3cb716b47c24bafc1d [CANDIDATE]: Term 1 election: Requested vote from peers a8adcce962244bc688dc4ecc1f69612c (127.16.29.66:33369), f8257d37af0941c2a978546ef2276b30 (127.16.29.67:37155)
I20260709 12:28:19.192261 32528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f998d4ae7a1e419c9e457994500ce4d1" candidate_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8adcce962244bc688dc4ecc1f69612c"
I20260709 12:28:19.192795 32602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f998d4ae7a1e419c9e457994500ce4d1" candidate_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8257d37af0941c2a978546ef2276b30"
I20260709 12:28:19.192950 32528 raft_consensus.cc:3060] T f998d4ae7a1e419c9e457994500ce4d1 P a8adcce962244bc688dc4ecc1f69612c [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:19.193418 32602 raft_consensus.cc:3060] T f998d4ae7a1e419c9e457994500ce4d1 P f8257d37af0941c2a978546ef2276b30 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:19.200563 32602 raft_consensus.cc:2468] T f998d4ae7a1e419c9e457994500ce4d1 P f8257d37af0941c2a978546ef2276b30 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc8d9fdc73b14a3cb716b47c24bafc1d in term 1.
I20260709 12:28:19.200563 32528 raft_consensus.cc:2468] T f998d4ae7a1e419c9e457994500ce4d1 P a8adcce962244bc688dc4ecc1f69612c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc8d9fdc73b14a3cb716b47c24bafc1d in term 1.
I20260709 12:28:19.202196 32418 leader_election.cc:304] T f998d4ae7a1e419c9e457994500ce4d1 P cc8d9fdc73b14a3cb716b47c24bafc1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a8adcce962244bc688dc4ecc1f69612c, cc8d9fdc73b14a3cb716b47c24bafc1d; no voters: 
I20260709 12:28:19.202935   309 raft_consensus.cc:2804] T f998d4ae7a1e419c9e457994500ce4d1 P cc8d9fdc73b14a3cb716b47c24bafc1d [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:28:19.203466   309 raft_consensus.cc:697] T f998d4ae7a1e419c9e457994500ce4d1 P cc8d9fdc73b14a3cb716b47c24bafc1d [term 1 LEADER]: Becoming Leader. State: Replica: cc8d9fdc73b14a3cb716b47c24bafc1d, State: Running, Role: LEADER
I20260709 12:28:19.204192   309 consensus_queue.cc:237] T f998d4ae7a1e419c9e457994500ce4d1 P cc8d9fdc73b14a3cb716b47c24bafc1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:19.212297 32348 catalog_manager.cc:5697] T f998d4ae7a1e419c9e457994500ce4d1 P cc8d9fdc73b14a3cb716b47c24bafc1d reported cstate change: term changed from 0 to 1, leader changed from <none> to cc8d9fdc73b14a3cb716b47c24bafc1d (127.16.29.65). New cstate: current_term: 1 leader_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:19.221467 32764 raft_consensus.cc:493] T db65db1003af45d4b6315902498f3f2b P a8adcce962244bc688dc4ecc1f69612c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:19.222028 32764 raft_consensus.cc:515] T db65db1003af45d4b6315902498f3f2b P a8adcce962244bc688dc4ecc1f69612c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:19.223647 32764 leader_election.cc:290] T db65db1003af45d4b6315902498f3f2b P a8adcce962244bc688dc4ecc1f69612c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc8d9fdc73b14a3cb716b47c24bafc1d (127.16.29.65:41229), f8257d37af0941c2a978546ef2276b30 (127.16.29.67:37155)
I20260709 12:28:19.224390 32453 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "db65db1003af45d4b6315902498f3f2b" candidate_uuid: "a8adcce962244bc688dc4ecc1f69612c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" is_pre_election: true
I20260709 12:28:19.224670 32602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "db65db1003af45d4b6315902498f3f2b" candidate_uuid: "a8adcce962244bc688dc4ecc1f69612c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8257d37af0941c2a978546ef2276b30" is_pre_election: true
I20260709 12:28:19.225195 32453 raft_consensus.cc:2393] T db65db1003af45d4b6315902498f3f2b P cc8d9fdc73b14a3cb716b47c24bafc1d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a8adcce962244bc688dc4ecc1f69612c in current term 1: Already voted for candidate cc8d9fdc73b14a3cb716b47c24bafc1d in this term.
I20260709 12:28:19.225412 32602 raft_consensus.cc:2393] T db65db1003af45d4b6315902498f3f2b P f8257d37af0941c2a978546ef2276b30 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a8adcce962244bc688dc4ecc1f69612c in current term 1: Already voted for candidate f8257d37af0941c2a978546ef2276b30 in this term.
I20260709 12:28:19.226397 32493 leader_election.cc:304] T db65db1003af45d4b6315902498f3f2b P a8adcce962244bc688dc4ecc1f69612c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a8adcce962244bc688dc4ecc1f69612c; no voters: cc8d9fdc73b14a3cb716b47c24bafc1d, f8257d37af0941c2a978546ef2276b30
I20260709 12:28:19.226969 32764 raft_consensus.cc:3060] T db65db1003af45d4b6315902498f3f2b P a8adcce962244bc688dc4ecc1f69612c [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:19.232007 32764 raft_consensus.cc:2749] T db65db1003af45d4b6315902498f3f2b P a8adcce962244bc688dc4ecc1f69612c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:28:19.239063 32764 consensus_queue.cc:1048] T 11a335d230b94355ad8859ba469b7989 P a8adcce962244bc688dc4ecc1f69612c [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:28:19.250047 32766 consensus_queue.cc:1048] T 11a335d230b94355ad8859ba469b7989 P a8adcce962244bc688dc4ecc1f69612c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:28:19.263706 32767 consensus_queue.cc:1048] T 92a7506e828b47a28f4e2c080d05419b P f8257d37af0941c2a978546ef2276b30 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:28:19.274855 32729 consensus_queue.cc:1048] T 92a7506e828b47a28f4e2c080d05419b P f8257d37af0941c2a978546ef2276b30 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:28:19.354969   309 consensus_queue.cc:1048] T f9cc8a9e92e64d2083b47e3619f88e32 P cc8d9fdc73b14a3cb716b47c24bafc1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:28:19.365311   317 consensus_queue.cc:1048] T f9cc8a9e92e64d2083b47e3619f88e32 P cc8d9fdc73b14a3cb716b47c24bafc1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260709 12:28:19.404407 32399 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:28:19.413296   317 consensus_queue.cc:1048] T 51c3eb697520458093debb61eebb75a7 P cc8d9fdc73b14a3cb716b47c24bafc1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:28:19.424880   309 consensus_queue.cc:1048] T 51c3eb697520458093debb61eebb75a7 P cc8d9fdc73b14a3cb716b47c24bafc1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:28:19.655949   309 consensus_queue.cc:1048] T f998d4ae7a1e419c9e457994500ce4d1 P cc8d9fdc73b14a3cb716b47c24bafc1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:28:19.667558   308 consensus_queue.cc:1048] T f998d4ae7a1e419c9e457994500ce4d1 P cc8d9fdc73b14a3cb716b47c24bafc1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260709 12:28:20.407073 32399 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:28:20.598366   345 raft_consensus.cc:493] T db65db1003af45d4b6315902498f3f2b P f8257d37af0941c2a978546ef2276b30 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:20.598980   345 raft_consensus.cc:515] T db65db1003af45d4b6315902498f3f2b P f8257d37af0941c2a978546ef2276b30 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:20.601007   345 leader_election.cc:290] T db65db1003af45d4b6315902498f3f2b P f8257d37af0941c2a978546ef2276b30 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cc8d9fdc73b14a3cb716b47c24bafc1d (127.16.29.65:41229), a8adcce962244bc688dc4ecc1f69612c (127.16.29.66:33369)
I20260709 12:28:20.601665 32453 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "db65db1003af45d4b6315902498f3f2b" candidate_uuid: "f8257d37af0941c2a978546ef2276b30" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" is_pre_election: true
I20260709 12:28:20.602084 32527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "db65db1003af45d4b6315902498f3f2b" candidate_uuid: "f8257d37af0941c2a978546ef2276b30" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8adcce962244bc688dc4ecc1f69612c" is_pre_election: true
I20260709 12:28:20.602331 32453 raft_consensus.cc:2468] T db65db1003af45d4b6315902498f3f2b P cc8d9fdc73b14a3cb716b47c24bafc1d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8257d37af0941c2a978546ef2276b30 in term 1.
I20260709 12:28:20.602883 32527 raft_consensus.cc:2468] T db65db1003af45d4b6315902498f3f2b P a8adcce962244bc688dc4ecc1f69612c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8257d37af0941c2a978546ef2276b30 in term 1.
I20260709 12:28:20.603535 32568 leader_election.cc:304] T db65db1003af45d4b6315902498f3f2b P f8257d37af0941c2a978546ef2276b30 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cc8d9fdc73b14a3cb716b47c24bafc1d, f8257d37af0941c2a978546ef2276b30; no voters: 
I20260709 12:28:20.604267   345 raft_consensus.cc:2804] T db65db1003af45d4b6315902498f3f2b P f8257d37af0941c2a978546ef2276b30 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:28:20.604609   345 raft_consensus.cc:493] T db65db1003af45d4b6315902498f3f2b P f8257d37af0941c2a978546ef2276b30 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:20.604837   345 raft_consensus.cc:3060] T db65db1003af45d4b6315902498f3f2b P f8257d37af0941c2a978546ef2276b30 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:20.609961   345 raft_consensus.cc:515] T db65db1003af45d4b6315902498f3f2b P f8257d37af0941c2a978546ef2276b30 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:20.611763   345 leader_election.cc:290] T db65db1003af45d4b6315902498f3f2b P f8257d37af0941c2a978546ef2276b30 [CANDIDATE]: Term 2 election: Requested vote from peers cc8d9fdc73b14a3cb716b47c24bafc1d (127.16.29.65:41229), a8adcce962244bc688dc4ecc1f69612c (127.16.29.66:33369)
I20260709 12:28:20.612452 32453 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "db65db1003af45d4b6315902498f3f2b" candidate_uuid: "f8257d37af0941c2a978546ef2276b30" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d"
I20260709 12:28:20.613112 32453 raft_consensus.cc:3060] T db65db1003af45d4b6315902498f3f2b P cc8d9fdc73b14a3cb716b47c24bafc1d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:20.612983 32527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "db65db1003af45d4b6315902498f3f2b" candidate_uuid: "f8257d37af0941c2a978546ef2276b30" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8adcce962244bc688dc4ecc1f69612c"
I20260709 12:28:20.613745 32527 raft_consensus.cc:3060] T db65db1003af45d4b6315902498f3f2b P a8adcce962244bc688dc4ecc1f69612c [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:20.619421 32453 raft_consensus.cc:2468] T db65db1003af45d4b6315902498f3f2b P cc8d9fdc73b14a3cb716b47c24bafc1d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8257d37af0941c2a978546ef2276b30 in term 2.
I20260709 12:28:20.620996 32527 raft_consensus.cc:2468] T db65db1003af45d4b6315902498f3f2b P a8adcce962244bc688dc4ecc1f69612c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8257d37af0941c2a978546ef2276b30 in term 2.
I20260709 12:28:20.621433 32568 leader_election.cc:304] T db65db1003af45d4b6315902498f3f2b P f8257d37af0941c2a978546ef2276b30 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cc8d9fdc73b14a3cb716b47c24bafc1d, f8257d37af0941c2a978546ef2276b30; no voters: 
I20260709 12:28:20.622211   345 raft_consensus.cc:2804] T db65db1003af45d4b6315902498f3f2b P f8257d37af0941c2a978546ef2276b30 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:28:20.623253   345 raft_consensus.cc:697] T db65db1003af45d4b6315902498f3f2b P f8257d37af0941c2a978546ef2276b30 [term 2 LEADER]: Becoming Leader. State: Replica: f8257d37af0941c2a978546ef2276b30, State: Running, Role: LEADER
I20260709 12:28:20.624070   345 consensus_queue.cc:237] T db65db1003af45d4b6315902498f3f2b P f8257d37af0941c2a978546ef2276b30 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:20.631994 32348 catalog_manager.cc:5697] T db65db1003af45d4b6315902498f3f2b P f8257d37af0941c2a978546ef2276b30 reported cstate change: term changed from 0 to 2, leader changed from <none> to f8257d37af0941c2a978546ef2276b30 (127.16.29.67). New cstate: current_term: 2 leader_uuid: "f8257d37af0941c2a978546ef2276b30" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } health_report { overall_health: HEALTHY } } }
I20260709 12:28:20.872220 32602 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "db65db1003af45d4b6315902498f3f2b"
dest_uuid: "f8257d37af0941c2a978546ef2276b30"
mode: GRACEFUL
new_leader_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d"
 from {username='slave'} at 127.0.0.1:42876
I20260709 12:28:20.873530 32602 raft_consensus.cc:606] T db65db1003af45d4b6315902498f3f2b P f8257d37af0941c2a978546ef2276b30 [term 2 LEADER]: Received request to transfer leadership to TS cc8d9fdc73b14a3cb716b47c24bafc1d
I20260709 12:28:20.883020 32600 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "92a7506e828b47a28f4e2c080d05419b"
dest_uuid: "f8257d37af0941c2a978546ef2276b30"
mode: GRACEFUL
new_leader_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d"
 from {username='slave'} at 127.0.0.1:42876
I20260709 12:28:20.883860 32600 raft_consensus.cc:606] T 92a7506e828b47a28f4e2c080d05419b P f8257d37af0941c2a978546ef2276b30 [term 1 LEADER]: Received request to transfer leadership to TS cc8d9fdc73b14a3cb716b47c24bafc1d
I20260709 12:28:20.890905 32527 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "276dee5db2d34451b642ed9359137ad4"
dest_uuid: "a8adcce962244bc688dc4ecc1f69612c"
mode: GRACEFUL
new_leader_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d"
 from {username='slave'} at 127.0.0.1:56052
I20260709 12:28:20.891479 32527 raft_consensus.cc:606] T 276dee5db2d34451b642ed9359137ad4 P a8adcce962244bc688dc4ecc1f69612c [term 1 LEADER]: Received request to transfer leadership to TS cc8d9fdc73b14a3cb716b47c24bafc1d
I20260709 12:28:20.916852 32762 raft_consensus.cc:993] T 276dee5db2d34451b642ed9359137ad4 P a8adcce962244bc688dc4ecc1f69612c: : Instructing follower cc8d9fdc73b14a3cb716b47c24bafc1d to start an election
I20260709 12:28:20.918130 32766 raft_consensus.cc:1081] T 276dee5db2d34451b642ed9359137ad4 P a8adcce962244bc688dc4ecc1f69612c [term 1 LEADER]: Signalling peer cc8d9fdc73b14a3cb716b47c24bafc1d to start an election
I20260709 12:28:20.933331 32452 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "276dee5db2d34451b642ed9359137ad4"
dest_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d"
 from {username='slave'} at 127.0.0.1:45590
I20260709 12:28:20.934100 32452 raft_consensus.cc:493] T 276dee5db2d34451b642ed9359137ad4 P cc8d9fdc73b14a3cb716b47c24bafc1d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:28:20.934536 32452 raft_consensus.cc:3060] T 276dee5db2d34451b642ed9359137ad4 P cc8d9fdc73b14a3cb716b47c24bafc1d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:20.944576 32452 raft_consensus.cc:515] T 276dee5db2d34451b642ed9359137ad4 P cc8d9fdc73b14a3cb716b47c24bafc1d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:20.947059 32453 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "f9cc8a9e92e64d2083b47e3619f88e32"
dest_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d"
mode: GRACEFUL
new_leader_uuid: "a8adcce962244bc688dc4ecc1f69612c"
 from {username='slave'} at 127.0.0.1:55040
I20260709 12:28:20.947456 32452 leader_election.cc:290] T 276dee5db2d34451b642ed9359137ad4 P cc8d9fdc73b14a3cb716b47c24bafc1d [CANDIDATE]: Term 2 election: Requested vote from peers a8adcce962244bc688dc4ecc1f69612c (127.16.29.66:33369), f8257d37af0941c2a978546ef2276b30 (127.16.29.67:37155)
I20260709 12:28:20.947973 32453 raft_consensus.cc:606] T f9cc8a9e92e64d2083b47e3619f88e32 P cc8d9fdc73b14a3cb716b47c24bafc1d [term 1 LEADER]: Received request to transfer leadership to TS a8adcce962244bc688dc4ecc1f69612c
I20260709 12:28:20.948457 32528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "276dee5db2d34451b642ed9359137ad4" candidate_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a8adcce962244bc688dc4ecc1f69612c"
I20260709 12:28:20.948716 32600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "276dee5db2d34451b642ed9359137ad4" candidate_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f8257d37af0941c2a978546ef2276b30"
I20260709 12:28:20.949416 32528 raft_consensus.cc:3055] T 276dee5db2d34451b642ed9359137ad4 P a8adcce962244bc688dc4ecc1f69612c [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:28:20.949719 32600 raft_consensus.cc:3060] T 276dee5db2d34451b642ed9359137ad4 P f8257d37af0941c2a978546ef2276b30 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:20.949738 32528 raft_consensus.cc:740] T 276dee5db2d34451b642ed9359137ad4 P a8adcce962244bc688dc4ecc1f69612c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a8adcce962244bc688dc4ecc1f69612c, State: Running, Role: LEADER
I20260709 12:28:20.951447 32528 consensus_queue.cc:260] T 276dee5db2d34451b642ed9359137ad4 P a8adcce962244bc688dc4ecc1f69612c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:20.953495 32528 raft_consensus.cc:3060] T 276dee5db2d34451b642ed9359137ad4 P a8adcce962244bc688dc4ecc1f69612c [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:20.955724 32453 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "f998d4ae7a1e419c9e457994500ce4d1"
dest_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d"
mode: GRACEFUL
new_leader_uuid: "f8257d37af0941c2a978546ef2276b30"
 from {username='slave'} at 127.0.0.1:55040
I20260709 12:28:20.956794 32453 raft_consensus.cc:606] T f998d4ae7a1e419c9e457994500ce4d1 P cc8d9fdc73b14a3cb716b47c24bafc1d [term 1 LEADER]: Received request to transfer leadership to TS f8257d37af0941c2a978546ef2276b30
I20260709 12:28:20.960644 32600 raft_consensus.cc:2468] T 276dee5db2d34451b642ed9359137ad4 P f8257d37af0941c2a978546ef2276b30 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc8d9fdc73b14a3cb716b47c24bafc1d in term 2.
I20260709 12:28:20.962684 32418 leader_election.cc:304] T 276dee5db2d34451b642ed9359137ad4 P cc8d9fdc73b14a3cb716b47c24bafc1d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cc8d9fdc73b14a3cb716b47c24bafc1d, f8257d37af0941c2a978546ef2276b30; no voters: 
I20260709 12:28:20.963853   346 raft_consensus.cc:2804] T 276dee5db2d34451b642ed9359137ad4 P cc8d9fdc73b14a3cb716b47c24bafc1d [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:28:20.964051 32528 raft_consensus.cc:2468] T 276dee5db2d34451b642ed9359137ad4 P a8adcce962244bc688dc4ecc1f69612c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc8d9fdc73b14a3cb716b47c24bafc1d in term 2.
I20260709 12:28:20.964515   346 raft_consensus.cc:697] T 276dee5db2d34451b642ed9359137ad4 P cc8d9fdc73b14a3cb716b47c24bafc1d [term 2 LEADER]: Becoming Leader. State: Replica: cc8d9fdc73b14a3cb716b47c24bafc1d, State: Running, Role: LEADER
I20260709 12:28:20.966002   346 consensus_queue.cc:237] T 276dee5db2d34451b642ed9359137ad4 P cc8d9fdc73b14a3cb716b47c24bafc1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:20.979049 32349 catalog_manager.cc:5697] T 276dee5db2d34451b642ed9359137ad4 P cc8d9fdc73b14a3cb716b47c24bafc1d reported cstate change: term changed from 1 to 2, leader changed from a8adcce962244bc688dc4ecc1f69612c (127.16.29.66) to cc8d9fdc73b14a3cb716b47c24bafc1d (127.16.29.65). New cstate: current_term: 2 leader_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:21.024284 32528 raft_consensus.cc:1275] T db65db1003af45d4b6315902498f3f2b P a8adcce962244bc688dc4ecc1f69612c [term 2 FOLLOWER]: Refusing update from remote peer f8257d37af0941c2a978546ef2276b30: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260709 12:28:21.025758   345 consensus_queue.cc:1048] T db65db1003af45d4b6315902498f3f2b P f8257d37af0941c2a978546ef2276b30 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:28:21.035826 32453 raft_consensus.cc:1275] T db65db1003af45d4b6315902498f3f2b P cc8d9fdc73b14a3cb716b47c24bafc1d [term 2 FOLLOWER]: Refusing update from remote peer f8257d37af0941c2a978546ef2276b30: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260709 12:28:21.038513   360 consensus_queue.cc:1048] T db65db1003af45d4b6315902498f3f2b P f8257d37af0941c2a978546ef2276b30 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:28:21.051438   360 raft_consensus.cc:993] T db65db1003af45d4b6315902498f3f2b P f8257d37af0941c2a978546ef2276b30: : Instructing follower cc8d9fdc73b14a3cb716b47c24bafc1d to start an election
I20260709 12:28:21.052103   347 raft_consensus.cc:1081] T db65db1003af45d4b6315902498f3f2b P f8257d37af0941c2a978546ef2276b30 [term 2 LEADER]: Signalling peer cc8d9fdc73b14a3cb716b47c24bafc1d to start an election
I20260709 12:28:21.053843 32453 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "db65db1003af45d4b6315902498f3f2b"
dest_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d"
 from {username='slave'} at 127.0.0.1:45580
I20260709 12:28:21.054366 32453 raft_consensus.cc:493] T db65db1003af45d4b6315902498f3f2b P cc8d9fdc73b14a3cb716b47c24bafc1d [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:28:21.054713 32453 raft_consensus.cc:3060] T db65db1003af45d4b6315902498f3f2b P cc8d9fdc73b14a3cb716b47c24bafc1d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:28:21.061826 32453 raft_consensus.cc:515] T db65db1003af45d4b6315902498f3f2b P cc8d9fdc73b14a3cb716b47c24bafc1d [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:21.063517 32453 leader_election.cc:290] T db65db1003af45d4b6315902498f3f2b P cc8d9fdc73b14a3cb716b47c24bafc1d [CANDIDATE]: Term 3 election: Requested vote from peers a8adcce962244bc688dc4ecc1f69612c (127.16.29.66:33369), f8257d37af0941c2a978546ef2276b30 (127.16.29.67:37155)
I20260709 12:28:21.064378 32528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "db65db1003af45d4b6315902498f3f2b" candidate_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: true dest_uuid: "a8adcce962244bc688dc4ecc1f69612c"
I20260709 12:28:21.064714 32600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "db65db1003af45d4b6315902498f3f2b" candidate_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: true dest_uuid: "f8257d37af0941c2a978546ef2276b30"
I20260709 12:28:21.065044 32528 raft_consensus.cc:3060] T db65db1003af45d4b6315902498f3f2b P a8adcce962244bc688dc4ecc1f69612c [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:28:21.065452 32600 raft_consensus.cc:3055] T db65db1003af45d4b6315902498f3f2b P f8257d37af0941c2a978546ef2276b30 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:28:21.065802 32600 raft_consensus.cc:740] T db65db1003af45d4b6315902498f3f2b P f8257d37af0941c2a978546ef2276b30 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: f8257d37af0941c2a978546ef2276b30, State: Running, Role: LEADER
I20260709 12:28:21.066768 32600 consensus_queue.cc:260] T db65db1003af45d4b6315902498f3f2b P f8257d37af0941c2a978546ef2276b30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:21.068279 32600 raft_consensus.cc:3060] T db65db1003af45d4b6315902498f3f2b P f8257d37af0941c2a978546ef2276b30 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:28:21.071306 32528 raft_consensus.cc:2468] T db65db1003af45d4b6315902498f3f2b P a8adcce962244bc688dc4ecc1f69612c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc8d9fdc73b14a3cb716b47c24bafc1d in term 3.
I20260709 12:28:21.072369 32418 leader_election.cc:304] T db65db1003af45d4b6315902498f3f2b P cc8d9fdc73b14a3cb716b47c24bafc1d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a8adcce962244bc688dc4ecc1f69612c, cc8d9fdc73b14a3cb716b47c24bafc1d; no voters: 
I20260709 12:28:21.073114   308 raft_consensus.cc:2804] T db65db1003af45d4b6315902498f3f2b P cc8d9fdc73b14a3cb716b47c24bafc1d [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:28:21.073518   308 raft_consensus.cc:697] T db65db1003af45d4b6315902498f3f2b P cc8d9fdc73b14a3cb716b47c24bafc1d [term 3 LEADER]: Becoming Leader. State: Replica: cc8d9fdc73b14a3cb716b47c24bafc1d, State: Running, Role: LEADER
I20260709 12:28:21.074527 32600 raft_consensus.cc:2468] T db65db1003af45d4b6315902498f3f2b P f8257d37af0941c2a978546ef2276b30 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc8d9fdc73b14a3cb716b47c24bafc1d in term 3.
I20260709 12:28:21.074518   308 consensus_queue.cc:237] T db65db1003af45d4b6315902498f3f2b P cc8d9fdc73b14a3cb716b47c24bafc1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:21.082262 32348 catalog_manager.cc:5697] T db65db1003af45d4b6315902498f3f2b P cc8d9fdc73b14a3cb716b47c24bafc1d reported cstate change: term changed from 2 to 3, leader changed from f8257d37af0941c2a978546ef2276b30 (127.16.29.67) to cc8d9fdc73b14a3cb716b47c24bafc1d (127.16.29.65). New cstate: current_term: 3 leader_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:21.181741   346 raft_consensus.cc:993] T f9cc8a9e92e64d2083b47e3619f88e32 P cc8d9fdc73b14a3cb716b47c24bafc1d: : Instructing follower a8adcce962244bc688dc4ecc1f69612c to start an election
I20260709 12:28:21.182264   308 raft_consensus.cc:1081] T f9cc8a9e92e64d2083b47e3619f88e32 P cc8d9fdc73b14a3cb716b47c24bafc1d [term 1 LEADER]: Signalling peer a8adcce962244bc688dc4ecc1f69612c to start an election
I20260709 12:28:21.183993 32528 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "f9cc8a9e92e64d2083b47e3619f88e32"
dest_uuid: "a8adcce962244bc688dc4ecc1f69612c"
 from {username='slave'} at 127.0.0.1:56046
I20260709 12:28:21.184623 32528 raft_consensus.cc:493] T f9cc8a9e92e64d2083b47e3619f88e32 P a8adcce962244bc688dc4ecc1f69612c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:28:21.184876 32528 raft_consensus.cc:3060] T f9cc8a9e92e64d2083b47e3619f88e32 P a8adcce962244bc688dc4ecc1f69612c [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:21.189857 32528 raft_consensus.cc:515] T f9cc8a9e92e64d2083b47e3619f88e32 P a8adcce962244bc688dc4ecc1f69612c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:21.191684 32528 leader_election.cc:290] T f9cc8a9e92e64d2083b47e3619f88e32 P a8adcce962244bc688dc4ecc1f69612c [CANDIDATE]: Term 2 election: Requested vote from peers cc8d9fdc73b14a3cb716b47c24bafc1d (127.16.29.65:41229), f8257d37af0941c2a978546ef2276b30 (127.16.29.67:37155)
I20260709 12:28:21.192741 32453 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9cc8a9e92e64d2083b47e3619f88e32" candidate_uuid: "a8adcce962244bc688dc4ecc1f69612c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d"
I20260709 12:28:21.192948 32600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9cc8a9e92e64d2083b47e3619f88e32" candidate_uuid: "a8adcce962244bc688dc4ecc1f69612c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f8257d37af0941c2a978546ef2276b30"
I20260709 12:28:21.193370 32453 raft_consensus.cc:3055] T f9cc8a9e92e64d2083b47e3619f88e32 P cc8d9fdc73b14a3cb716b47c24bafc1d [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:28:21.193789 32453 raft_consensus.cc:740] T f9cc8a9e92e64d2083b47e3619f88e32 P cc8d9fdc73b14a3cb716b47c24bafc1d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: cc8d9fdc73b14a3cb716b47c24bafc1d, State: Running, Role: LEADER
I20260709 12:28:21.194977 32453 consensus_queue.cc:260] T f9cc8a9e92e64d2083b47e3619f88e32 P cc8d9fdc73b14a3cb716b47c24bafc1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:21.196516 32453 raft_consensus.cc:3060] T f9cc8a9e92e64d2083b47e3619f88e32 P cc8d9fdc73b14a3cb716b47c24bafc1d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:21.201946 32453 raft_consensus.cc:2468] T f9cc8a9e92e64d2083b47e3619f88e32 P cc8d9fdc73b14a3cb716b47c24bafc1d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a8adcce962244bc688dc4ecc1f69612c in term 2.
I20260709 12:28:21.203254 32494 leader_election.cc:304] T f9cc8a9e92e64d2083b47e3619f88e32 P a8adcce962244bc688dc4ecc1f69612c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a8adcce962244bc688dc4ecc1f69612c, cc8d9fdc73b14a3cb716b47c24bafc1d; no voters: f8257d37af0941c2a978546ef2276b30
I20260709 12:28:21.204185   357 raft_consensus.cc:2804] T f9cc8a9e92e64d2083b47e3619f88e32 P a8adcce962244bc688dc4ecc1f69612c [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:28:21.204670   357 raft_consensus.cc:697] T f9cc8a9e92e64d2083b47e3619f88e32 P a8adcce962244bc688dc4ecc1f69612c [term 2 LEADER]: Becoming Leader. State: Replica: a8adcce962244bc688dc4ecc1f69612c, State: Running, Role: LEADER
I20260709 12:28:21.205885   357 consensus_queue.cc:237] T f9cc8a9e92e64d2083b47e3619f88e32 P a8adcce962244bc688dc4ecc1f69612c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:21.213770 32348 catalog_manager.cc:5697] T f9cc8a9e92e64d2083b47e3619f88e32 P a8adcce962244bc688dc4ecc1f69612c reported cstate change: term changed from 1 to 2, leader changed from cc8d9fdc73b14a3cb716b47c24bafc1d (127.16.29.65) to a8adcce962244bc688dc4ecc1f69612c (127.16.29.66). New cstate: current_term: 2 leader_uuid: "a8adcce962244bc688dc4ecc1f69612c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:21.353253   360 raft_consensus.cc:993] T 92a7506e828b47a28f4e2c080d05419b P f8257d37af0941c2a978546ef2276b30: : Instructing follower cc8d9fdc73b14a3cb716b47c24bafc1d to start an election
I20260709 12:28:21.353752   347 raft_consensus.cc:1081] T 92a7506e828b47a28f4e2c080d05419b P f8257d37af0941c2a978546ef2276b30 [term 1 LEADER]: Signalling peer cc8d9fdc73b14a3cb716b47c24bafc1d to start an election
I20260709 12:28:21.355417 32453 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "92a7506e828b47a28f4e2c080d05419b"
dest_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d"
 from {username='slave'} at 127.0.0.1:45580
I20260709 12:28:21.355960 32453 raft_consensus.cc:493] T 92a7506e828b47a28f4e2c080d05419b P cc8d9fdc73b14a3cb716b47c24bafc1d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:28:21.356248 32453 raft_consensus.cc:3060] T 92a7506e828b47a28f4e2c080d05419b P cc8d9fdc73b14a3cb716b47c24bafc1d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:21.357038 32528 raft_consensus.cc:1275] T 276dee5db2d34451b642ed9359137ad4 P a8adcce962244bc688dc4ecc1f69612c [term 2 FOLLOWER]: Refusing update from remote peer cc8d9fdc73b14a3cb716b47c24bafc1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:21.358479   308 consensus_queue.cc:1048] T 276dee5db2d34451b642ed9359137ad4 P cc8d9fdc73b14a3cb716b47c24bafc1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:28:21.362468 32453 raft_consensus.cc:515] T 92a7506e828b47a28f4e2c080d05419b P cc8d9fdc73b14a3cb716b47c24bafc1d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:21.364423 32453 leader_election.cc:290] T 92a7506e828b47a28f4e2c080d05419b P cc8d9fdc73b14a3cb716b47c24bafc1d [CANDIDATE]: Term 2 election: Requested vote from peers a8adcce962244bc688dc4ecc1f69612c (127.16.29.66:33369), f8257d37af0941c2a978546ef2276b30 (127.16.29.67:37155)
I20260709 12:28:21.365458 32528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92a7506e828b47a28f4e2c080d05419b" candidate_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a8adcce962244bc688dc4ecc1f69612c"
I20260709 12:28:21.365686 32600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92a7506e828b47a28f4e2c080d05419b" candidate_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f8257d37af0941c2a978546ef2276b30"
I20260709 12:28:21.366070 32528 raft_consensus.cc:3060] T 92a7506e828b47a28f4e2c080d05419b P a8adcce962244bc688dc4ecc1f69612c [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:21.366240 32600 raft_consensus.cc:3055] T 92a7506e828b47a28f4e2c080d05419b P f8257d37af0941c2a978546ef2276b30 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:28:21.366628 32600 raft_consensus.cc:740] T 92a7506e828b47a28f4e2c080d05419b P f8257d37af0941c2a978546ef2276b30 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f8257d37af0941c2a978546ef2276b30, State: Running, Role: LEADER
I20260709 12:28:21.367673 32600 consensus_queue.cc:260] T 92a7506e828b47a28f4e2c080d05419b P f8257d37af0941c2a978546ef2276b30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:21.369225 32600 raft_consensus.cc:3060] T 92a7506e828b47a28f4e2c080d05419b P f8257d37af0941c2a978546ef2276b30 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:21.373764 32528 raft_consensus.cc:2468] T 92a7506e828b47a28f4e2c080d05419b P a8adcce962244bc688dc4ecc1f69612c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc8d9fdc73b14a3cb716b47c24bafc1d in term 2.
I20260709 12:28:21.375950 32418 leader_election.cc:304] T 92a7506e828b47a28f4e2c080d05419b P cc8d9fdc73b14a3cb716b47c24bafc1d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a8adcce962244bc688dc4ecc1f69612c, cc8d9fdc73b14a3cb716b47c24bafc1d; no voters: 
I20260709 12:28:21.376529 32602 raft_consensus.cc:1275] T 276dee5db2d34451b642ed9359137ad4 P f8257d37af0941c2a978546ef2276b30 [term 2 FOLLOWER]: Refusing update from remote peer cc8d9fdc73b14a3cb716b47c24bafc1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:21.377121 32600 raft_consensus.cc:2468] T 92a7506e828b47a28f4e2c080d05419b P f8257d37af0941c2a978546ef2276b30 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc8d9fdc73b14a3cb716b47c24bafc1d in term 2.
I20260709 12:28:21.378136   308 raft_consensus.cc:2804] T 92a7506e828b47a28f4e2c080d05419b P cc8d9fdc73b14a3cb716b47c24bafc1d [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:28:21.378665   308 raft_consensus.cc:697] T 92a7506e828b47a28f4e2c080d05419b P cc8d9fdc73b14a3cb716b47c24bafc1d [term 2 LEADER]: Becoming Leader. State: Replica: cc8d9fdc73b14a3cb716b47c24bafc1d, State: Running, Role: LEADER
I20260709 12:28:21.379654   308 consensus_queue.cc:237] T 92a7506e828b47a28f4e2c080d05419b P cc8d9fdc73b14a3cb716b47c24bafc1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:21.382457   367 consensus_queue.cc:1048] T 276dee5db2d34451b642ed9359137ad4 P cc8d9fdc73b14a3cb716b47c24bafc1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:28:21.397611 32348 catalog_manager.cc:5697] T 92a7506e828b47a28f4e2c080d05419b P cc8d9fdc73b14a3cb716b47c24bafc1d reported cstate change: term changed from 1 to 2, leader changed from f8257d37af0941c2a978546ef2276b30 (127.16.29.67) to cc8d9fdc73b14a3cb716b47c24bafc1d (127.16.29.65). New cstate: current_term: 2 leader_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:21.461967 32600 raft_consensus.cc:1275] T db65db1003af45d4b6315902498f3f2b P f8257d37af0941c2a978546ef2276b30 [term 3 FOLLOWER]: Refusing update from remote peer cc8d9fdc73b14a3cb716b47c24bafc1d: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 12:28:21.463176   308 consensus_queue.cc:1048] T db65db1003af45d4b6315902498f3f2b P cc8d9fdc73b14a3cb716b47c24bafc1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:28:21.472965 32528 raft_consensus.cc:1275] T db65db1003af45d4b6315902498f3f2b P a8adcce962244bc688dc4ecc1f69612c [term 3 FOLLOWER]: Refusing update from remote peer cc8d9fdc73b14a3cb716b47c24bafc1d: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 12:28:21.474782   346 consensus_queue.cc:1048] T db65db1003af45d4b6315902498f3f2b P cc8d9fdc73b14a3cb716b47c24bafc1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:28:21.506564   308 raft_consensus.cc:993] T f998d4ae7a1e419c9e457994500ce4d1 P cc8d9fdc73b14a3cb716b47c24bafc1d: : Instructing follower f8257d37af0941c2a978546ef2276b30 to start an election
I20260709 12:28:21.507112   346 raft_consensus.cc:1081] T f998d4ae7a1e419c9e457994500ce4d1 P cc8d9fdc73b14a3cb716b47c24bafc1d [term 1 LEADER]: Signalling peer f8257d37af0941c2a978546ef2276b30 to start an election
I20260709 12:28:21.508694 32602 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "f998d4ae7a1e419c9e457994500ce4d1"
dest_uuid: "f8257d37af0941c2a978546ef2276b30"
 from {username='slave'} at 127.0.0.1:42860
I20260709 12:28:21.509217 32602 raft_consensus.cc:493] T f998d4ae7a1e419c9e457994500ce4d1 P f8257d37af0941c2a978546ef2276b30 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:28:21.509449 32602 raft_consensus.cc:3060] T f998d4ae7a1e419c9e457994500ce4d1 P f8257d37af0941c2a978546ef2276b30 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:21.516479 32602 raft_consensus.cc:515] T f998d4ae7a1e419c9e457994500ce4d1 P f8257d37af0941c2a978546ef2276b30 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:21.518163 32602 leader_election.cc:290] T f998d4ae7a1e419c9e457994500ce4d1 P f8257d37af0941c2a978546ef2276b30 [CANDIDATE]: Term 2 election: Requested vote from peers cc8d9fdc73b14a3cb716b47c24bafc1d (127.16.29.65:41229), a8adcce962244bc688dc4ecc1f69612c (127.16.29.66:33369)
I20260709 12:28:21.519022 32452 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f998d4ae7a1e419c9e457994500ce4d1" candidate_uuid: "f8257d37af0941c2a978546ef2276b30" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d"
I20260709 12:28:21.519177 32528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f998d4ae7a1e419c9e457994500ce4d1" candidate_uuid: "f8257d37af0941c2a978546ef2276b30" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a8adcce962244bc688dc4ecc1f69612c"
I20260709 12:28:21.519764 32452 raft_consensus.cc:3055] T f998d4ae7a1e419c9e457994500ce4d1 P cc8d9fdc73b14a3cb716b47c24bafc1d [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:28:21.519824 32528 raft_consensus.cc:3060] T f998d4ae7a1e419c9e457994500ce4d1 P a8adcce962244bc688dc4ecc1f69612c [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:21.520162 32452 raft_consensus.cc:740] T f998d4ae7a1e419c9e457994500ce4d1 P cc8d9fdc73b14a3cb716b47c24bafc1d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: cc8d9fdc73b14a3cb716b47c24bafc1d, State: Running, Role: LEADER
I20260709 12:28:21.521346 32452 consensus_queue.cc:260] T f998d4ae7a1e419c9e457994500ce4d1 P cc8d9fdc73b14a3cb716b47c24bafc1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:21.522557 32452 raft_consensus.cc:3060] T f998d4ae7a1e419c9e457994500ce4d1 P cc8d9fdc73b14a3cb716b47c24bafc1d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:21.526345 32528 raft_consensus.cc:2468] T f998d4ae7a1e419c9e457994500ce4d1 P a8adcce962244bc688dc4ecc1f69612c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8257d37af0941c2a978546ef2276b30 in term 2.
I20260709 12:28:21.527410 32567 leader_election.cc:304] T f998d4ae7a1e419c9e457994500ce4d1 P f8257d37af0941c2a978546ef2276b30 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a8adcce962244bc688dc4ecc1f69612c, f8257d37af0941c2a978546ef2276b30; no voters: 
I20260709 12:28:21.528220   347 raft_consensus.cc:2804] T f998d4ae7a1e419c9e457994500ce4d1 P f8257d37af0941c2a978546ef2276b30 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:28:21.528645   347 raft_consensus.cc:697] T f998d4ae7a1e419c9e457994500ce4d1 P f8257d37af0941c2a978546ef2276b30 [term 2 LEADER]: Becoming Leader. State: Replica: f8257d37af0941c2a978546ef2276b30, State: Running, Role: LEADER
I20260709 12:28:21.528960 32452 raft_consensus.cc:2468] T f998d4ae7a1e419c9e457994500ce4d1 P cc8d9fdc73b14a3cb716b47c24bafc1d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8257d37af0941c2a978546ef2276b30 in term 2.
I20260709 12:28:21.529583   347 consensus_queue.cc:237] T f998d4ae7a1e419c9e457994500ce4d1 P f8257d37af0941c2a978546ef2276b30 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:21.537904 32349 catalog_manager.cc:5697] T f998d4ae7a1e419c9e457994500ce4d1 P f8257d37af0941c2a978546ef2276b30 reported cstate change: term changed from 1 to 2, leader changed from cc8d9fdc73b14a3cb716b47c24bafc1d (127.16.29.65) to f8257d37af0941c2a978546ef2276b30 (127.16.29.67). New cstate: current_term: 2 leader_uuid: "f8257d37af0941c2a978546ef2276b30" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } health_report { overall_health: HEALTHY } } }
I20260709 12:28:21.691046 32602 raft_consensus.cc:3060] T f9cc8a9e92e64d2083b47e3619f88e32 P f8257d37af0941c2a978546ef2276b30 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:21.696451 32602 raft_consensus.cc:1275] T f9cc8a9e92e64d2083b47e3619f88e32 P f8257d37af0941c2a978546ef2276b30 [term 2 FOLLOWER]: Refusing update from remote peer a8adcce962244bc688dc4ecc1f69612c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:21.698046 32762 consensus_queue.cc:1048] T f9cc8a9e92e64d2083b47e3619f88e32 P a8adcce962244bc688dc4ecc1f69612c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:28:21.709733 32452 raft_consensus.cc:1275] T f9cc8a9e92e64d2083b47e3619f88e32 P cc8d9fdc73b14a3cb716b47c24bafc1d [term 2 FOLLOWER]: Refusing update from remote peer a8adcce962244bc688dc4ecc1f69612c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:21.712412   373 consensus_queue.cc:1048] T f9cc8a9e92e64d2083b47e3619f88e32 P a8adcce962244bc688dc4ecc1f69612c [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:28:21.780346 32528 raft_consensus.cc:1275] T 92a7506e828b47a28f4e2c080d05419b P a8adcce962244bc688dc4ecc1f69612c [term 2 FOLLOWER]: Refusing update from remote peer cc8d9fdc73b14a3cb716b47c24bafc1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:21.781679   346 consensus_queue.cc:1048] T 92a7506e828b47a28f4e2c080d05419b P cc8d9fdc73b14a3cb716b47c24bafc1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:28:21.791288 32602 raft_consensus.cc:1275] T 92a7506e828b47a28f4e2c080d05419b P f8257d37af0941c2a978546ef2276b30 [term 2 FOLLOWER]: Refusing update from remote peer cc8d9fdc73b14a3cb716b47c24bafc1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:21.793018   346 consensus_queue.cc:1048] T 92a7506e828b47a28f4e2c080d05419b P cc8d9fdc73b14a3cb716b47c24bafc1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:28:21.971369 32452 raft_consensus.cc:1275] T f998d4ae7a1e419c9e457994500ce4d1 P cc8d9fdc73b14a3cb716b47c24bafc1d [term 2 FOLLOWER]: Refusing update from remote peer f8257d37af0941c2a978546ef2276b30: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:21.972867   347 consensus_queue.cc:1048] T f998d4ae7a1e419c9e457994500ce4d1 P f8257d37af0941c2a978546ef2276b30 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:28:21.982947 32528 raft_consensus.cc:1275] T f998d4ae7a1e419c9e457994500ce4d1 P a8adcce962244bc688dc4ecc1f69612c [term 2 FOLLOWER]: Refusing update from remote peer f8257d37af0941c2a978546ef2276b30: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:21.984513   360 consensus_queue.cc:1048] T f998d4ae7a1e419c9e457994500ce4d1 P f8257d37af0941c2a978546ef2276b30 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:28:23.964126 32349 catalog_manager.cc:2533] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34582:
table { table_name: "soft_deleted_table" } modify_external_catalogs: true reserve_seconds: 3600
I20260709 12:28:23.984894 32349 catalog_manager.cc:3570] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34582:
table { table_name: "soft_deleted_table" } new_extra_configs { key: "kudu.table.disable_compaction" value: "true" }
I20260709 12:28:24.018600 16501 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:28:24.278695   429 tablet.cc:1724] T 51c3eb697520458093debb61eebb75a7 P cc8d9fdc73b14a3cb716b47c24bafc1d: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
W20260709 12:28:24.282928   420 log.cc:927] Time spent T 51c3eb697520458093debb61eebb75a7 P a8adcce962244bc688dc4ecc1f69612c: Append to log took a long time: real 0.109s	user 0.005s	sys 0.000s
W20260709 12:28:24.301647   431 log.cc:927] Time spent T 7035e41df60347b188a3b8079aa7e551 P cc8d9fdc73b14a3cb716b47c24bafc1d: Append to log took a long time: real 0.067s	user 0.000s	sys 0.000s
I20260709 12:28:24.327690   437 tablet.cc:1724] T f9cc8a9e92e64d2083b47e3619f88e32 P a8adcce962244bc688dc4ecc1f69612c: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:28:24.340225   438 tablet.cc:1724] T f998d4ae7a1e419c9e457994500ce4d1 P f8257d37af0941c2a978546ef2276b30: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:28:24.362179   439 tablet.cc:1724] T f998d4ae7a1e419c9e457994500ce4d1 P a8adcce962244bc688dc4ecc1f69612c: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:28:24.393196   448 tablet.cc:1724] T 7035e41df60347b188a3b8079aa7e551 P cc8d9fdc73b14a3cb716b47c24bafc1d: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:28:24.428828   447 tablet.cc:1724] T f9cc8a9e92e64d2083b47e3619f88e32 P cc8d9fdc73b14a3cb716b47c24bafc1d: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:28:24.407469   445 tablet.cc:1724] T 894a3f7744bd44ad814f22f90682c678 P a8adcce962244bc688dc4ecc1f69612c: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:28:24.446151 32524 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "9396e8d0ed7b44f58bbc5b421f492f50"
dest_uuid: "a8adcce962244bc688dc4ecc1f69612c"
mode: GRACEFUL
new_leader_uuid: "f8257d37af0941c2a978546ef2276b30"
 from {username='slave'} at 127.0.0.1:57458
I20260709 12:28:24.447423 32524 raft_consensus.cc:606] T 9396e8d0ed7b44f58bbc5b421f492f50 P a8adcce962244bc688dc4ecc1f69612c [term 1 LEADER]: Received request to transfer leadership to TS f8257d37af0941c2a978546ef2276b30
I20260709 12:28:24.379105   443 tablet.cc:1724] T f9cc8a9e92e64d2083b47e3619f88e32 P f8257d37af0941c2a978546ef2276b30: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:28:24.459260   438 tablet.cc:1724] T 51c3eb697520458093debb61eebb75a7 P f8257d37af0941c2a978546ef2276b30: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:28:24.466996   451 tablet.cc:1724] T db65db1003af45d4b6315902498f3f2b P f8257d37af0941c2a978546ef2276b30: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:28:24.468688   446 tablet.cc:1724] T db65db1003af45d4b6315902498f3f2b P cc8d9fdc73b14a3cb716b47c24bafc1d: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:28:24.479430   450 tablet.cc:1724] T 7035e41df60347b188a3b8079aa7e551 P a8adcce962244bc688dc4ecc1f69612c: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
W20260709 12:28:24.481863   449 log.cc:927] Time spent T 11a335d230b94355ad8859ba469b7989 P cc8d9fdc73b14a3cb716b47c24bafc1d: Append to log took a long time: real 0.050s	user 0.000s	sys 0.000s
I20260709 12:28:24.508755   429 tablet.cc:1724] T f998d4ae7a1e419c9e457994500ce4d1 P cc8d9fdc73b14a3cb716b47c24bafc1d: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:28:24.505798 32349 catalog_manager.cc:5789] TS cc8d9fdc73b14a3cb716b47c24bafc1d (127.16.29.65:41229) does not have the latest schema for tablet 276dee5db2d34451b642ed9359137ad4 (table soft_deleted_table [id=06201339da074254986a87154a02a188]). Expected version 1 got 0
I20260709 12:28:24.515588 32349 catalog_manager.cc:5789] TS cc8d9fdc73b14a3cb716b47c24bafc1d (127.16.29.65:41229) does not have the latest schema for tablet 92a7506e828b47a28f4e2c080d05419b (table soft_deleted_table [id=06201339da074254986a87154a02a188]). Expected version 1 got 0
I20260709 12:28:24.516185 32348 catalog_manager.cc:5789] TS a8adcce962244bc688dc4ecc1f69612c (127.16.29.66:33369) does not have the latest schema for tablet 11a335d230b94355ad8859ba469b7989 (table soft_deleted_table [id=06201339da074254986a87154a02a188]). Expected version 1 got 0
I20260709 12:28:24.528359   453 tablet.cc:1724] T 92a7506e828b47a28f4e2c080d05419b P f8257d37af0941c2a978546ef2276b30: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:28:24.530014   438 tablet.cc:1724] T 894a3f7744bd44ad814f22f90682c678 P f8257d37af0941c2a978546ef2276b30: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:28:24.536900   447 tablet.cc:1724] T 894a3f7744bd44ad814f22f90682c678 P cc8d9fdc73b14a3cb716b47c24bafc1d: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:28:24.538944   437 tablet.cc:1724] T 51c3eb697520458093debb61eebb75a7 P a8adcce962244bc688dc4ecc1f69612c: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:28:24.546769   448 tablet.cc:1724] T 276dee5db2d34451b642ed9359137ad4 P cc8d9fdc73b14a3cb716b47c24bafc1d: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:28:24.543975   438 tablet.cc:1724] T 7035e41df60347b188a3b8079aa7e551 P f8257d37af0941c2a978546ef2276b30: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:28:24.542349   445 tablet.cc:1724] T 11a335d230b94355ad8859ba469b7989 P a8adcce962244bc688dc4ecc1f69612c: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:28:24.542956   439 tablet.cc:1724] T 92a7506e828b47a28f4e2c080d05419b P a8adcce962244bc688dc4ecc1f69612c: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:28:24.563508   447 tablet.cc:1724] T 92a7506e828b47a28f4e2c080d05419b P cc8d9fdc73b14a3cb716b47c24bafc1d: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:28:24.555751   416 raft_consensus.cc:993] T 9396e8d0ed7b44f58bbc5b421f492f50 P a8adcce962244bc688dc4ecc1f69612c: : Instructing follower f8257d37af0941c2a978546ef2276b30 to start an election
I20260709 12:28:24.566674   416 raft_consensus.cc:1081] T 9396e8d0ed7b44f58bbc5b421f492f50 P a8adcce962244bc688dc4ecc1f69612c [term 1 LEADER]: Signalling peer f8257d37af0941c2a978546ef2276b30 to start an election
I20260709 12:28:24.566890 32453 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "f22ce6f243334c2281f0543e605f614a"
dest_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d"
mode: GRACEFUL
new_leader_uuid: "f8257d37af0941c2a978546ef2276b30"
 from {username='slave'} at 127.0.0.1:55048
I20260709 12:28:24.567543 32453 raft_consensus.cc:606] T f22ce6f243334c2281f0543e605f614a P cc8d9fdc73b14a3cb716b47c24bafc1d [term 1 LEADER]: Received request to transfer leadership to TS f8257d37af0941c2a978546ef2276b30
I20260709 12:28:24.551270   451 tablet.cc:1724] T 11a335d230b94355ad8859ba469b7989 P f8257d37af0941c2a978546ef2276b30: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:28:24.569116 16501 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260709 12:28:24.569832 16501 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:28:24.568651 32599 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "9396e8d0ed7b44f58bbc5b421f492f50"
dest_uuid: "f8257d37af0941c2a978546ef2276b30"
 from {username='slave'} at 127.0.0.1:42862
I20260709 12:28:24.571033 32599 raft_consensus.cc:493] T 9396e8d0ed7b44f58bbc5b421f492f50 P f8257d37af0941c2a978546ef2276b30 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:28:24.571501 32599 raft_consensus.cc:3060] T 9396e8d0ed7b44f58bbc5b421f492f50 P f8257d37af0941c2a978546ef2276b30 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:24.588856   445 tablet.cc:1724] T 276dee5db2d34451b642ed9359137ad4 P a8adcce962244bc688dc4ecc1f69612c: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:28:24.593860   446 tablet.cc:1724] T 11a335d230b94355ad8859ba469b7989 P cc8d9fdc73b14a3cb716b47c24bafc1d: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:28:24.590808   437 tablet.cc:1724] T db65db1003af45d4b6315902498f3f2b P a8adcce962244bc688dc4ecc1f69612c: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:28:24.592588   443 tablet.cc:1724] T 276dee5db2d34451b642ed9359137ad4 P f8257d37af0941c2a978546ef2276b30: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:28:24.626713 32599 raft_consensus.cc:515] T 9396e8d0ed7b44f58bbc5b421f492f50 P f8257d37af0941c2a978546ef2276b30 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:24.631448 32599 leader_election.cc:290] T 9396e8d0ed7b44f58bbc5b421f492f50 P f8257d37af0941c2a978546ef2276b30 [CANDIDATE]: Term 2 election: Requested vote from peers cc8d9fdc73b14a3cb716b47c24bafc1d (127.16.29.65:41229), a8adcce962244bc688dc4ecc1f69612c (127.16.29.66:33369)
I20260709 12:28:24.632686 32527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9396e8d0ed7b44f58bbc5b421f492f50" candidate_uuid: "f8257d37af0941c2a978546ef2276b30" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a8adcce962244bc688dc4ecc1f69612c"
I20260709 12:28:24.633268 32527 raft_consensus.cc:3055] T 9396e8d0ed7b44f58bbc5b421f492f50 P a8adcce962244bc688dc4ecc1f69612c [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:28:24.633492 32527 raft_consensus.cc:740] T 9396e8d0ed7b44f58bbc5b421f492f50 P a8adcce962244bc688dc4ecc1f69612c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a8adcce962244bc688dc4ecc1f69612c, State: Running, Role: LEADER
I20260709 12:28:24.634423 32453 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9396e8d0ed7b44f58bbc5b421f492f50" candidate_uuid: "f8257d37af0941c2a978546ef2276b30" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d"
I20260709 12:28:24.635066 32453 raft_consensus.cc:3060] T 9396e8d0ed7b44f58bbc5b421f492f50 P cc8d9fdc73b14a3cb716b47c24bafc1d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:24.634995 32527 consensus_queue.cc:260] T 9396e8d0ed7b44f58bbc5b421f492f50 P a8adcce962244bc688dc4ecc1f69612c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:24.636603 32527 raft_consensus.cc:3060] T 9396e8d0ed7b44f58bbc5b421f492f50 P a8adcce962244bc688dc4ecc1f69612c [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:24.643121 32453 raft_consensus.cc:2468] T 9396e8d0ed7b44f58bbc5b421f492f50 P cc8d9fdc73b14a3cb716b47c24bafc1d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8257d37af0941c2a978546ef2276b30 in term 2.
I20260709 12:28:24.643914 32527 raft_consensus.cc:2468] T 9396e8d0ed7b44f58bbc5b421f492f50 P a8adcce962244bc688dc4ecc1f69612c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8257d37af0941c2a978546ef2276b30 in term 2.
I20260709 12:28:24.644822 32568 leader_election.cc:304] T 9396e8d0ed7b44f58bbc5b421f492f50 P f8257d37af0941c2a978546ef2276b30 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cc8d9fdc73b14a3cb716b47c24bafc1d, f8257d37af0941c2a978546ef2276b30; no voters: 
I20260709 12:28:24.646028   382 raft_consensus.cc:2804] T 9396e8d0ed7b44f58bbc5b421f492f50 P f8257d37af0941c2a978546ef2276b30 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:28:24.646517   382 raft_consensus.cc:697] T 9396e8d0ed7b44f58bbc5b421f492f50 P f8257d37af0941c2a978546ef2276b30 [term 2 LEADER]: Becoming Leader. State: Replica: f8257d37af0941c2a978546ef2276b30, State: Running, Role: LEADER
I20260709 12:28:24.647372   382 consensus_queue.cc:237] T 9396e8d0ed7b44f58bbc5b421f492f50 P f8257d37af0941c2a978546ef2276b30 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:24.656350 32349 catalog_manager.cc:6242] T 00000000000000000000000000000000 P 66f4042e77394c5a8e850bb63a8a7ac5: soft_deleted_table [id=06201339da074254986a87154a02a188] alter complete (version 1)
I20260709 12:28:24.663908 32348 catalog_manager.cc:5697] T 9396e8d0ed7b44f58bbc5b421f492f50 P f8257d37af0941c2a978546ef2276b30 reported cstate change: term changed from 1 to 2, leader changed from a8adcce962244bc688dc4ecc1f69612c (127.16.29.66) to f8257d37af0941c2a978546ef2276b30 (127.16.29.67). New cstate: current_term: 2 leader_uuid: "f8257d37af0941c2a978546ef2276b30" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } health_report { overall_health: HEALTHY } } }
I20260709 12:28:24.805847   422 raft_consensus.cc:993] T f22ce6f243334c2281f0543e605f614a P cc8d9fdc73b14a3cb716b47c24bafc1d: : Instructing follower f8257d37af0941c2a978546ef2276b30 to start an election
I20260709 12:28:24.806402   417 raft_consensus.cc:1081] T f22ce6f243334c2281f0543e605f614a P cc8d9fdc73b14a3cb716b47c24bafc1d [term 1 LEADER]: Signalling peer f8257d37af0941c2a978546ef2276b30 to start an election
I20260709 12:28:24.808096 32599 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "f22ce6f243334c2281f0543e605f614a"
dest_uuid: "f8257d37af0941c2a978546ef2276b30"
 from {username='slave'} at 127.0.0.1:42860
I20260709 12:28:24.808606 32599 raft_consensus.cc:493] T f22ce6f243334c2281f0543e605f614a P f8257d37af0941c2a978546ef2276b30 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:28:24.808918 32599 raft_consensus.cc:3060] T f22ce6f243334c2281f0543e605f614a P f8257d37af0941c2a978546ef2276b30 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:24.814352 32599 raft_consensus.cc:515] T f22ce6f243334c2281f0543e605f614a P f8257d37af0941c2a978546ef2276b30 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:24.816180 32599 leader_election.cc:290] T f22ce6f243334c2281f0543e605f614a P f8257d37af0941c2a978546ef2276b30 [CANDIDATE]: Term 2 election: Requested vote from peers cc8d9fdc73b14a3cb716b47c24bafc1d (127.16.29.65:41229), a8adcce962244bc688dc4ecc1f69612c (127.16.29.66:33369)
I20260709 12:28:24.817278 32527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f22ce6f243334c2281f0543e605f614a" candidate_uuid: "f8257d37af0941c2a978546ef2276b30" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a8adcce962244bc688dc4ecc1f69612c"
I20260709 12:28:24.817273 32453 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f22ce6f243334c2281f0543e605f614a" candidate_uuid: "f8257d37af0941c2a978546ef2276b30" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d"
I20260709 12:28:24.817998 32527 raft_consensus.cc:3060] T f22ce6f243334c2281f0543e605f614a P a8adcce962244bc688dc4ecc1f69612c [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:24.818092 32453 raft_consensus.cc:3055] T f22ce6f243334c2281f0543e605f614a P cc8d9fdc73b14a3cb716b47c24bafc1d [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:28:24.818535 32453 raft_consensus.cc:740] T f22ce6f243334c2281f0543e605f614a P cc8d9fdc73b14a3cb716b47c24bafc1d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: cc8d9fdc73b14a3cb716b47c24bafc1d, State: Running, Role: LEADER
I20260709 12:28:24.819432 32453 consensus_queue.cc:260] T f22ce6f243334c2281f0543e605f614a P cc8d9fdc73b14a3cb716b47c24bafc1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:24.820595 32453 raft_consensus.cc:3060] T f22ce6f243334c2281f0543e605f614a P cc8d9fdc73b14a3cb716b47c24bafc1d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:24.823758 32527 raft_consensus.cc:2468] T f22ce6f243334c2281f0543e605f614a P a8adcce962244bc688dc4ecc1f69612c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8257d37af0941c2a978546ef2276b30 in term 2.
I20260709 12:28:24.825066 32567 leader_election.cc:304] T f22ce6f243334c2281f0543e605f614a P f8257d37af0941c2a978546ef2276b30 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a8adcce962244bc688dc4ecc1f69612c, f8257d37af0941c2a978546ef2276b30; no voters: 
I20260709 12:28:24.825803   382 raft_consensus.cc:2804] T f22ce6f243334c2281f0543e605f614a P f8257d37af0941c2a978546ef2276b30 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:28:24.825904 32453 raft_consensus.cc:2468] T f22ce6f243334c2281f0543e605f614a P cc8d9fdc73b14a3cb716b47c24bafc1d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8257d37af0941c2a978546ef2276b30 in term 2.
I20260709 12:28:24.826411   382 raft_consensus.cc:697] T f22ce6f243334c2281f0543e605f614a P f8257d37af0941c2a978546ef2276b30 [term 2 LEADER]: Becoming Leader. State: Replica: f8257d37af0941c2a978546ef2276b30, State: Running, Role: LEADER
I20260709 12:28:24.827443   382 consensus_queue.cc:237] T f22ce6f243334c2281f0543e605f614a P f8257d37af0941c2a978546ef2276b30 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } }
I20260709 12:28:24.836815 32348 catalog_manager.cc:5697] T f22ce6f243334c2281f0543e605f614a P f8257d37af0941c2a978546ef2276b30 reported cstate change: term changed from 1 to 2, leader changed from cc8d9fdc73b14a3cb716b47c24bafc1d (127.16.29.65) to f8257d37af0941c2a978546ef2276b30 (127.16.29.67). New cstate: current_term: 2 leader_uuid: "f8257d37af0941c2a978546ef2276b30" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f8257d37af0941c2a978546ef2276b30" member_type: VOTER last_known_addr { host: "127.16.29.67" port: 37155 } health_report { overall_health: HEALTHY } } }
I20260709 12:28:25.172335 32453 raft_consensus.cc:1275] T 9396e8d0ed7b44f58bbc5b421f492f50 P cc8d9fdc73b14a3cb716b47c24bafc1d [term 2 FOLLOWER]: Refusing update from remote peer f8257d37af0941c2a978546ef2276b30: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:25.173631   382 consensus_queue.cc:1048] T 9396e8d0ed7b44f58bbc5b421f492f50 P f8257d37af0941c2a978546ef2276b30 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:28:25.183354 32527 raft_consensus.cc:1275] T 9396e8d0ed7b44f58bbc5b421f492f50 P a8adcce962244bc688dc4ecc1f69612c [term 2 FOLLOWER]: Refusing update from remote peer f8257d37af0941c2a978546ef2276b30: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:25.184794   382 consensus_queue.cc:1048] T 9396e8d0ed7b44f58bbc5b421f492f50 P f8257d37af0941c2a978546ef2276b30 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:28:25.330469 32453 raft_consensus.cc:1275] T f22ce6f243334c2281f0543e605f614a P cc8d9fdc73b14a3cb716b47c24bafc1d [term 2 FOLLOWER]: Refusing update from remote peer f8257d37af0941c2a978546ef2276b30: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:25.331580   463 consensus_queue.cc:1048] T f22ce6f243334c2281f0543e605f614a P f8257d37af0941c2a978546ef2276b30 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc8d9fdc73b14a3cb716b47c24bafc1d" member_type: VOTER last_known_addr { host: "127.16.29.65" port: 41229 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:28:25.340607 32527 raft_consensus.cc:1275] T f22ce6f243334c2281f0543e605f614a P a8adcce962244bc688dc4ecc1f69612c [term 2 FOLLOWER]: Refusing update from remote peer f8257d37af0941c2a978546ef2276b30: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:25.342360   382 consensus_queue.cc:1048] T f22ce6f243334c2281f0543e605f614a P f8257d37af0941c2a978546ef2276b30 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8adcce962244bc688dc4ecc1f69612c" member_type: VOTER last_known_addr { host: "127.16.29.66" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:28:25.571039 16501 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:28:25.576313 16501 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:28:25.576871 16501 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:28:26.578003 16501 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:28:26.583663 16501 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:28:26.584182 16501 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:28:27.585198 16501 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:28:27.591519 16501 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:28:27.591996 16501 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:28:28.592919 16501 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:28:28.598238 16501 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:28:28.598690 16501 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:28:29.599745 16501 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:28:29.606449 16501 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:28:29.607209 16501 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:28:30.608359 16501 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:28:30.614341 16501 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:28:30.615062 16501 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:28:31.616447 16501 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:28:31.624656 16501 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:28:31.625216 16501 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:28:32.626493 16501 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:28:32.631271 16501 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:28:32.631700 16501 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:28:33.632817 16501 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:28:33.638455 16501 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:28:33.638929 16501 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
W20260709 12:28:34.570019 32418 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.29.66:33369, user_credentials={real_user=slave}} blocked reactor thread for 404917us
I20260709 12:28:34.763861 16501 tablet_server.cc:179] TabletServer@127.16.29.65:0 shutting down...
W20260709 12:28:34.794310 32494 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 9 similar messages]
W20260709 12:28:34.810029 32494 consensus_peers.cc:597] T f9cc8a9e92e64d2083b47e3619f88e32 P a8adcce962244bc688dc4ecc1f69612c -> Peer cc8d9fdc73b14a3cb716b47c24bafc1d (127.16.29.65:41229): Couldn't send request to peer cc8d9fdc73b14a3cb716b47c24bafc1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:28:34.832854 16501 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:28:34.833945 16501 tablet_replica.cc:333] T 11a335d230b94355ad8859ba469b7989 P cc8d9fdc73b14a3cb716b47c24bafc1d: stopping tablet replica
I20260709 12:28:34.835392 16501 raft_consensus.cc:2243] T 11a335d230b94355ad8859ba469b7989 P cc8d9fdc73b14a3cb716b47c24bafc1d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:34.836709 16501 raft_consensus.cc:2272] T 11a335d230b94355ad8859ba469b7989 P cc8d9fdc73b14a3cb716b47c24bafc1d [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 12:28:34.839025 32494 consensus_peers.cc:597] T 8e7d5cee9cf64a3e87372bde6431883e P a8adcce962244bc688dc4ecc1f69612c -> Peer cc8d9fdc73b14a3cb716b47c24bafc1d (127.16.29.65:41229): Couldn't send request to peer cc8d9fdc73b14a3cb716b47c24bafc1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:28:34.841343 16501 tablet_replica.cc:333] T 276dee5db2d34451b642ed9359137ad4 P cc8d9fdc73b14a3cb716b47c24bafc1d: stopping tablet replica
I20260709 12:28:34.842468 16501 raft_consensus.cc:2243] T 276dee5db2d34451b642ed9359137ad4 P cc8d9fdc73b14a3cb716b47c24bafc1d [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:28:34.844092 16501 raft_consensus.cc:2272] T 276dee5db2d34451b642ed9359137ad4 P cc8d9fdc73b14a3cb716b47c24bafc1d [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:34.848801 16501 tablet_replica.cc:333] T 7035e41df60347b188a3b8079aa7e551 P cc8d9fdc73b14a3cb716b47c24bafc1d: stopping tablet replica
I20260709 12:28:34.850122 16501 raft_consensus.cc:2243] T 7035e41df60347b188a3b8079aa7e551 P cc8d9fdc73b14a3cb716b47c24bafc1d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:34.851138 16501 raft_consensus.cc:2272] T 7035e41df60347b188a3b8079aa7e551 P cc8d9fdc73b14a3cb716b47c24bafc1d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:34.854851 16501 tablet_replica.cc:333] T f9cc8a9e92e64d2083b47e3619f88e32 P cc8d9fdc73b14a3cb716b47c24bafc1d: stopping tablet replica
I20260709 12:28:34.855926 16501 raft_consensus.cc:2243] T f9cc8a9e92e64d2083b47e3619f88e32 P cc8d9fdc73b14a3cb716b47c24bafc1d [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:34.856784 16501 raft_consensus.cc:2272] T f9cc8a9e92e64d2083b47e3619f88e32 P cc8d9fdc73b14a3cb716b47c24bafc1d [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:34.861073 16501 tablet_replica.cc:333] T 894a3f7744bd44ad814f22f90682c678 P cc8d9fdc73b14a3cb716b47c24bafc1d: stopping tablet replica
I20260709 12:28:34.862089 16501 raft_consensus.cc:2243] T 894a3f7744bd44ad814f22f90682c678 P cc8d9fdc73b14a3cb716b47c24bafc1d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:34.863046 16501 raft_consensus.cc:2272] T 894a3f7744bd44ad814f22f90682c678 P cc8d9fdc73b14a3cb716b47c24bafc1d [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 12:28:34.867254 32568 consensus_peers.cc:597] T f998d4ae7a1e419c9e457994500ce4d1 P f8257d37af0941c2a978546ef2276b30 -> Peer cc8d9fdc73b14a3cb716b47c24bafc1d (127.16.29.65:41229): Couldn't send request to peer cc8d9fdc73b14a3cb716b47c24bafc1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:28:34.867686 16501 tablet_replica.cc:333] T 92a7506e828b47a28f4e2c080d05419b P cc8d9fdc73b14a3cb716b47c24bafc1d: stopping tablet replica
I20260709 12:28:34.868882 16501 raft_consensus.cc:2243] T 92a7506e828b47a28f4e2c080d05419b P cc8d9fdc73b14a3cb716b47c24bafc1d [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:28:34.870316 16501 raft_consensus.cc:2272] T 92a7506e828b47a28f4e2c080d05419b P cc8d9fdc73b14a3cb716b47c24bafc1d [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:34.874017 16501 tablet_replica.cc:333] T d768701c29784c52a2ffb8303021347d P cc8d9fdc73b14a3cb716b47c24bafc1d: stopping tablet replica
I20260709 12:28:34.875053 16501 raft_consensus.cc:2243] T d768701c29784c52a2ffb8303021347d P cc8d9fdc73b14a3cb716b47c24bafc1d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:34.875771 16501 raft_consensus.cc:2272] T d768701c29784c52a2ffb8303021347d P cc8d9fdc73b14a3cb716b47c24bafc1d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:34.879727 16501 tablet_replica.cc:333] T f22ce6f243334c2281f0543e605f614a P cc8d9fdc73b14a3cb716b47c24bafc1d: stopping tablet replica
I20260709 12:28:34.880770 16501 raft_consensus.cc:2243] T f22ce6f243334c2281f0543e605f614a P cc8d9fdc73b14a3cb716b47c24bafc1d [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:34.881482 16501 raft_consensus.cc:2272] T f22ce6f243334c2281f0543e605f614a P cc8d9fdc73b14a3cb716b47c24bafc1d [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:34.885496 16501 tablet_replica.cc:333] T 51c3eb697520458093debb61eebb75a7 P cc8d9fdc73b14a3cb716b47c24bafc1d: stopping tablet replica
I20260709 12:28:34.886396 16501 raft_consensus.cc:2243] T 51c3eb697520458093debb61eebb75a7 P cc8d9fdc73b14a3cb716b47c24bafc1d [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:28:34.887765 16501 raft_consensus.cc:2272] T 51c3eb697520458093debb61eebb75a7 P cc8d9fdc73b14a3cb716b47c24bafc1d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:34.891556 16501 tablet_replica.cc:333] T 4b2251c329154ac49d6147b1cfd6981b P cc8d9fdc73b14a3cb716b47c24bafc1d: stopping tablet replica
I20260709 12:28:34.892539 16501 raft_consensus.cc:2243] T 4b2251c329154ac49d6147b1cfd6981b P cc8d9fdc73b14a3cb716b47c24bafc1d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:34.893296 16501 raft_consensus.cc:2272] T 4b2251c329154ac49d6147b1cfd6981b P cc8d9fdc73b14a3cb716b47c24bafc1d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:34.898176 16501 tablet_replica.cc:333] T f5cb9ea668324bfb883f09876ddd56f7 P cc8d9fdc73b14a3cb716b47c24bafc1d: stopping tablet replica
I20260709 12:28:34.899111 16501 raft_consensus.cc:2243] T f5cb9ea668324bfb883f09876ddd56f7 P cc8d9fdc73b14a3cb716b47c24bafc1d [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:28:34.900473 16501 raft_consensus.cc:2272] T f5cb9ea668324bfb883f09876ddd56f7 P cc8d9fdc73b14a3cb716b47c24bafc1d [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:34.904503 16501 tablet_replica.cc:333] T db65db1003af45d4b6315902498f3f2b P cc8d9fdc73b14a3cb716b47c24bafc1d: stopping tablet replica
I20260709 12:28:34.905515 16501 raft_consensus.cc:2243] T db65db1003af45d4b6315902498f3f2b P cc8d9fdc73b14a3cb716b47c24bafc1d [term 3 LEADER]: Raft consensus shutting down.
I20260709 12:28:34.906828 16501 raft_consensus.cc:2272] T db65db1003af45d4b6315902498f3f2b P cc8d9fdc73b14a3cb716b47c24bafc1d [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:34.910755 16501 tablet_replica.cc:333] T 9396e8d0ed7b44f58bbc5b421f492f50 P cc8d9fdc73b14a3cb716b47c24bafc1d: stopping tablet replica
I20260709 12:28:34.911623 16501 raft_consensus.cc:2243] T 9396e8d0ed7b44f58bbc5b421f492f50 P cc8d9fdc73b14a3cb716b47c24bafc1d [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:34.912395 16501 raft_consensus.cc:2272] T 9396e8d0ed7b44f58bbc5b421f492f50 P cc8d9fdc73b14a3cb716b47c24bafc1d [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:34.916126 16501 tablet_replica.cc:333] T c127bfae907d4960956f04576bf0c6c7 P cc8d9fdc73b14a3cb716b47c24bafc1d: stopping tablet replica
I20260709 12:28:34.916952 16501 raft_consensus.cc:2243] T c127bfae907d4960956f04576bf0c6c7 P cc8d9fdc73b14a3cb716b47c24bafc1d [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:34.917647 16501 raft_consensus.cc:2272] T c127bfae907d4960956f04576bf0c6c7 P cc8d9fdc73b14a3cb716b47c24bafc1d [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:34.921303 16501 tablet_replica.cc:333] T aa861e9a505d425792eb54a2a29cab97 P cc8d9fdc73b14a3cb716b47c24bafc1d: stopping tablet replica
I20260709 12:28:34.922151 16501 raft_consensus.cc:2243] T aa861e9a505d425792eb54a2a29cab97 P cc8d9fdc73b14a3cb716b47c24bafc1d [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:28:34.923460 16501 raft_consensus.cc:2272] T aa861e9a505d425792eb54a2a29cab97 P cc8d9fdc73b14a3cb716b47c24bafc1d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:34.927096 16501 tablet_replica.cc:333] T 50fe031340a94b6590d29d44cd57ca55 P cc8d9fdc73b14a3cb716b47c24bafc1d: stopping tablet replica
I20260709 12:28:34.927994 16501 raft_consensus.cc:2243] T 50fe031340a94b6590d29d44cd57ca55 P cc8d9fdc73b14a3cb716b47c24bafc1d [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:28:34.929378 16501 raft_consensus.cc:2272] T 50fe031340a94b6590d29d44cd57ca55 P cc8d9fdc73b14a3cb716b47c24bafc1d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:34.933010 16501 tablet_replica.cc:333] T f998d4ae7a1e419c9e457994500ce4d1 P cc8d9fdc73b14a3cb716b47c24bafc1d: stopping tablet replica
I20260709 12:28:34.933835 16501 raft_consensus.cc:2243] T f998d4ae7a1e419c9e457994500ce4d1 P cc8d9fdc73b14a3cb716b47c24bafc1d [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:34.934901 16501 raft_consensus.cc:2272] T f998d4ae7a1e419c9e457994500ce4d1 P cc8d9fdc73b14a3cb716b47c24bafc1d [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:34.939124 16501 tablet_replica.cc:333] T 8e7d5cee9cf64a3e87372bde6431883e P cc8d9fdc73b14a3cb716b47c24bafc1d: stopping tablet replica
I20260709 12:28:34.940022 16501 raft_consensus.cc:2243] T 8e7d5cee9cf64a3e87372bde6431883e P cc8d9fdc73b14a3cb716b47c24bafc1d [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:34.940788 16501 raft_consensus.cc:2272] T 8e7d5cee9cf64a3e87372bde6431883e P cc8d9fdc73b14a3cb716b47c24bafc1d [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:35.003424 16501 tablet_server.cc:196] TabletServer@127.16.29.65:0 shutdown complete.
W20260709 12:28:35.028044 32494 consensus_peers.cc:597] T d768701c29784c52a2ffb8303021347d P a8adcce962244bc688dc4ecc1f69612c -> Peer cc8d9fdc73b14a3cb716b47c24bafc1d (127.16.29.65:41229): Couldn't send request to peer cc8d9fdc73b14a3cb716b47c24bafc1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:41229: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:28:35.059067 16501 tablet_server.cc:179] TabletServer@127.16.29.66:0 shutting down...
W20260709 12:28:35.092388 32494 consensus_peers.cc:597] T 11a335d230b94355ad8859ba469b7989 P a8adcce962244bc688dc4ecc1f69612c -> Peer cc8d9fdc73b14a3cb716b47c24bafc1d (127.16.29.65:41229): Couldn't send request to peer cc8d9fdc73b14a3cb716b47c24bafc1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:41229: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:28:35.096501 32568 consensus_peers.cc:597] T f22ce6f243334c2281f0543e605f614a P f8257d37af0941c2a978546ef2276b30 -> Peer cc8d9fdc73b14a3cb716b47c24bafc1d (127.16.29.65:41229): Couldn't send request to peer cc8d9fdc73b14a3cb716b47c24bafc1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:41229: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:28:35.106773 16501 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:28:35.107972 16501 tablet_replica.cc:333] T 11a335d230b94355ad8859ba469b7989 P a8adcce962244bc688dc4ecc1f69612c: stopping tablet replica
I20260709 12:28:35.109154 16501 raft_consensus.cc:2243] T 11a335d230b94355ad8859ba469b7989 P a8adcce962244bc688dc4ecc1f69612c [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:28:35.110523 16501 raft_consensus.cc:2272] T 11a335d230b94355ad8859ba469b7989 P a8adcce962244bc688dc4ecc1f69612c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:35.113641 16501 tablet_replica.cc:333] T 7035e41df60347b188a3b8079aa7e551 P a8adcce962244bc688dc4ecc1f69612c: stopping tablet replica
I20260709 12:28:35.114367 16501 raft_consensus.cc:2243] T 7035e41df60347b188a3b8079aa7e551 P a8adcce962244bc688dc4ecc1f69612c [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:28:35.115319 16501 raft_consensus.cc:2272] T 7035e41df60347b188a3b8079aa7e551 P a8adcce962244bc688dc4ecc1f69612c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:35.117784 16501 tablet_replica.cc:333] T 894a3f7744bd44ad814f22f90682c678 P a8adcce962244bc688dc4ecc1f69612c: stopping tablet replica
I20260709 12:28:35.118335 16501 raft_consensus.cc:2243] T 894a3f7744bd44ad814f22f90682c678 P a8adcce962244bc688dc4ecc1f69612c [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:28:35.119295 16501 raft_consensus.cc:2272] T 894a3f7744bd44ad814f22f90682c678 P a8adcce962244bc688dc4ecc1f69612c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:35.122153 16501 tablet_replica.cc:333] T 276dee5db2d34451b642ed9359137ad4 P a8adcce962244bc688dc4ecc1f69612c: stopping tablet replica
I20260709 12:28:35.122854 16501 raft_consensus.cc:2243] T 276dee5db2d34451b642ed9359137ad4 P a8adcce962244bc688dc4ecc1f69612c [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:35.123546 16501 raft_consensus.cc:2272] T 276dee5db2d34451b642ed9359137ad4 P a8adcce962244bc688dc4ecc1f69612c [term 2 FOLLOWER]: Raft consensus is shut down!
W20260709 12:28:35.123476 32567 consensus_peers.cc:597] T f998d4ae7a1e419c9e457994500ce4d1 P f8257d37af0941c2a978546ef2276b30 -> Peer a8adcce962244bc688dc4ecc1f69612c (127.16.29.66:33369): Couldn't send request to peer a8adcce962244bc688dc4ecc1f69612c. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:28:35.127264 16501 tablet_replica.cc:333] T 92a7506e828b47a28f4e2c080d05419b P a8adcce962244bc688dc4ecc1f69612c: stopping tablet replica
I20260709 12:28:35.127983 16501 raft_consensus.cc:2243] T 92a7506e828b47a28f4e2c080d05419b P a8adcce962244bc688dc4ecc1f69612c [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:35.128796 16501 raft_consensus.cc:2272] T 92a7506e828b47a28f4e2c080d05419b P a8adcce962244bc688dc4ecc1f69612c [term 2 FOLLOWER]: Raft consensus is shut down!
W20260709 12:28:35.129607 32568 consensus_peers.cc:597] T c127bfae907d4960956f04576bf0c6c7 P f8257d37af0941c2a978546ef2276b30 -> Peer cc8d9fdc73b14a3cb716b47c24bafc1d (127.16.29.65:41229): Couldn't send request to peer cc8d9fdc73b14a3cb716b47c24bafc1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:41229: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:28:35.131794 16501 tablet_replica.cc:333] T d768701c29784c52a2ffb8303021347d P a8adcce962244bc688dc4ecc1f69612c: stopping tablet replica
I20260709 12:28:35.132721 16501 raft_consensus.cc:2243] T d768701c29784c52a2ffb8303021347d P a8adcce962244bc688dc4ecc1f69612c [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:28:35.133754 16501 raft_consensus.cc:2272] T d768701c29784c52a2ffb8303021347d P a8adcce962244bc688dc4ecc1f69612c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:35.136893 16501 tablet_replica.cc:333] T 51c3eb697520458093debb61eebb75a7 P a8adcce962244bc688dc4ecc1f69612c: stopping tablet replica
I20260709 12:28:35.137637 16501 raft_consensus.cc:2243] T 51c3eb697520458093debb61eebb75a7 P a8adcce962244bc688dc4ecc1f69612c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:35.138319 16501 raft_consensus.cc:2272] T 51c3eb697520458093debb61eebb75a7 P a8adcce962244bc688dc4ecc1f69612c [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 12:28:35.139389 32494 consensus_peers.cc:597] T 4b2251c329154ac49d6147b1cfd6981b P a8adcce962244bc688dc4ecc1f69612c -> Peer cc8d9fdc73b14a3cb716b47c24bafc1d (127.16.29.65:41229): Couldn't send request to peer cc8d9fdc73b14a3cb716b47c24bafc1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:41229: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:28:35.141103 16501 tablet_replica.cc:333] T f22ce6f243334c2281f0543e605f614a P a8adcce962244bc688dc4ecc1f69612c: stopping tablet replica
I20260709 12:28:35.141855 16501 raft_consensus.cc:2243] T f22ce6f243334c2281f0543e605f614a P a8adcce962244bc688dc4ecc1f69612c [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:35.142386 16501 raft_consensus.cc:2272] T f22ce6f243334c2281f0543e605f614a P a8adcce962244bc688dc4ecc1f69612c [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:35.144851 16501 tablet_replica.cc:333] T f998d4ae7a1e419c9e457994500ce4d1 P a8adcce962244bc688dc4ecc1f69612c: stopping tablet replica
I20260709 12:28:35.145531 16501 raft_consensus.cc:2243] T f998d4ae7a1e419c9e457994500ce4d1 P a8adcce962244bc688dc4ecc1f69612c [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:35.146147 16501 raft_consensus.cc:2272] T f998d4ae7a1e419c9e457994500ce4d1 P a8adcce962244bc688dc4ecc1f69612c [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:35.148586 16501 tablet_replica.cc:333] T 8e7d5cee9cf64a3e87372bde6431883e P a8adcce962244bc688dc4ecc1f69612c: stopping tablet replica
I20260709 12:28:35.149199 16501 raft_consensus.cc:2243] T 8e7d5cee9cf64a3e87372bde6431883e P a8adcce962244bc688dc4ecc1f69612c [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:28:35.150231 16501 raft_consensus.cc:2272] T 8e7d5cee9cf64a3e87372bde6431883e P a8adcce962244bc688dc4ecc1f69612c [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:35.152848 16501 tablet_replica.cc:333] T db65db1003af45d4b6315902498f3f2b P a8adcce962244bc688dc4ecc1f69612c: stopping tablet replica
I20260709 12:28:35.153600 16501 raft_consensus.cc:2243] T db65db1003af45d4b6315902498f3f2b P a8adcce962244bc688dc4ecc1f69612c [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:35.154192 16501 raft_consensus.cc:2272] T db65db1003af45d4b6315902498f3f2b P a8adcce962244bc688dc4ecc1f69612c [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:35.156678 16501 tablet_replica.cc:333] T f5cb9ea668324bfb883f09876ddd56f7 P a8adcce962244bc688dc4ecc1f69612c: stopping tablet replica
I20260709 12:28:35.157300 16501 raft_consensus.cc:2243] T f5cb9ea668324bfb883f09876ddd56f7 P a8adcce962244bc688dc4ecc1f69612c [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:35.157868 16501 raft_consensus.cc:2272] T f5cb9ea668324bfb883f09876ddd56f7 P a8adcce962244bc688dc4ecc1f69612c [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:35.160485 16501 tablet_replica.cc:333] T c127bfae907d4960956f04576bf0c6c7 P a8adcce962244bc688dc4ecc1f69612c: stopping tablet replica
I20260709 12:28:35.161096 16501 raft_consensus.cc:2243] T c127bfae907d4960956f04576bf0c6c7 P a8adcce962244bc688dc4ecc1f69612c [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:35.161671 16501 raft_consensus.cc:2272] T c127bfae907d4960956f04576bf0c6c7 P a8adcce962244bc688dc4ecc1f69612c [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:35.164430 16501 tablet_replica.cc:333] T aa861e9a505d425792eb54a2a29cab97 P a8adcce962244bc688dc4ecc1f69612c: stopping tablet replica
I20260709 12:28:35.165143 16501 raft_consensus.cc:2243] T aa861e9a505d425792eb54a2a29cab97 P a8adcce962244bc688dc4ecc1f69612c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:35.165671 16501 raft_consensus.cc:2272] T aa861e9a505d425792eb54a2a29cab97 P a8adcce962244bc688dc4ecc1f69612c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:35.168294 16501 tablet_replica.cc:333] T 9396e8d0ed7b44f58bbc5b421f492f50 P a8adcce962244bc688dc4ecc1f69612c: stopping tablet replica
I20260709 12:28:35.168905 16501 raft_consensus.cc:2243] T 9396e8d0ed7b44f58bbc5b421f492f50 P a8adcce962244bc688dc4ecc1f69612c [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:35.169464 16501 raft_consensus.cc:2272] T 9396e8d0ed7b44f58bbc5b421f492f50 P a8adcce962244bc688dc4ecc1f69612c [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:35.171972 16501 tablet_replica.cc:333] T 4b2251c329154ac49d6147b1cfd6981b P a8adcce962244bc688dc4ecc1f69612c: stopping tablet replica
I20260709 12:28:35.172580 16501 raft_consensus.cc:2243] T 4b2251c329154ac49d6147b1cfd6981b P a8adcce962244bc688dc4ecc1f69612c [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:28:35.173570 16501 raft_consensus.cc:2272] T 4b2251c329154ac49d6147b1cfd6981b P a8adcce962244bc688dc4ecc1f69612c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:35.177263 16501 tablet_replica.cc:333] T f9cc8a9e92e64d2083b47e3619f88e32 P a8adcce962244bc688dc4ecc1f69612c: stopping tablet replica
I20260709 12:28:35.178153 16501 raft_consensus.cc:2243] T f9cc8a9e92e64d2083b47e3619f88e32 P a8adcce962244bc688dc4ecc1f69612c [term 2 LEADER]: Raft consensus shutting down.
W20260709 12:28:35.178228 32567 consensus_peers.cc:597] T c127bfae907d4960956f04576bf0c6c7 P f8257d37af0941c2a978546ef2276b30 -> Peer a8adcce962244bc688dc4ecc1f69612c (127.16.29.66:33369): Couldn't send request to peer a8adcce962244bc688dc4ecc1f69612c. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:28:35.179222 16501 raft_consensus.cc:2272] T f9cc8a9e92e64d2083b47e3619f88e32 P a8adcce962244bc688dc4ecc1f69612c [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:35.181766 16501 tablet_replica.cc:333] T 50fe031340a94b6590d29d44cd57ca55 P a8adcce962244bc688dc4ecc1f69612c: stopping tablet replica
I20260709 12:28:35.182343 16501 raft_consensus.cc:2243] T 50fe031340a94b6590d29d44cd57ca55 P a8adcce962244bc688dc4ecc1f69612c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:35.182876 16501 raft_consensus.cc:2272] T 50fe031340a94b6590d29d44cd57ca55 P a8adcce962244bc688dc4ecc1f69612c [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 12:28:35.212667 32382 debug-util.cc:398] Leaking SignalData structure 0x7b08002d8340 after lost signal to thread 16504
W20260709 12:28:35.214229 32382 debug-util.cc:398] Leaking SignalData structure 0x7b08002acda0 after lost signal to thread 32385
W20260709 12:28:35.217273 32382 debug-util.cc:398] Leaking SignalData structure 0x7b0800152be0 after lost signal to thread 32626
I20260709 12:28:35.230257 16501 tablet_server.cc:196] TabletServer@127.16.29.66:0 shutdown complete.
W20260709 12:28:35.230418 32568 consensus_peers.cc:597] T 9396e8d0ed7b44f58bbc5b421f492f50 P f8257d37af0941c2a978546ef2276b30 -> Peer cc8d9fdc73b14a3cb716b47c24bafc1d (127.16.29.65:41229): Couldn't send request to peer cc8d9fdc73b14a3cb716b47c24bafc1d. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.65:41229: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:28:35.287343 16501 tablet_server.cc:179] TabletServer@127.16.29.67:0 shutting down...
I20260709 12:28:35.331429 16501 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:28:35.332628 16501 tablet_replica.cc:333] T f9cc8a9e92e64d2083b47e3619f88e32 P f8257d37af0941c2a978546ef2276b30: stopping tablet replica
I20260709 12:28:35.333549 16501 raft_consensus.cc:2243] T f9cc8a9e92e64d2083b47e3619f88e32 P f8257d37af0941c2a978546ef2276b30 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:35.334218 16501 raft_consensus.cc:2272] T f9cc8a9e92e64d2083b47e3619f88e32 P f8257d37af0941c2a978546ef2276b30 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:35.337141 16501 tablet_replica.cc:333] T 894a3f7744bd44ad814f22f90682c678 P f8257d37af0941c2a978546ef2276b30: stopping tablet replica
I20260709 12:28:35.337900 16501 raft_consensus.cc:2243] T 894a3f7744bd44ad814f22f90682c678 P f8257d37af0941c2a978546ef2276b30 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:35.338548 16501 raft_consensus.cc:2272] T 894a3f7744bd44ad814f22f90682c678 P f8257d37af0941c2a978546ef2276b30 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:35.341166 16501 tablet_replica.cc:333] T 276dee5db2d34451b642ed9359137ad4 P f8257d37af0941c2a978546ef2276b30: stopping tablet replica
I20260709 12:28:35.342015 16501 raft_consensus.cc:2243] T 276dee5db2d34451b642ed9359137ad4 P f8257d37af0941c2a978546ef2276b30 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:35.342648 16501 raft_consensus.cc:2272] T 276dee5db2d34451b642ed9359137ad4 P f8257d37af0941c2a978546ef2276b30 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:35.345394 16501 tablet_replica.cc:333] T 11a335d230b94355ad8859ba469b7989 P f8257d37af0941c2a978546ef2276b30: stopping tablet replica
I20260709 12:28:35.346134 16501 raft_consensus.cc:2243] T 11a335d230b94355ad8859ba469b7989 P f8257d37af0941c2a978546ef2276b30 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:35.346742 16501 raft_consensus.cc:2272] T 11a335d230b94355ad8859ba469b7989 P f8257d37af0941c2a978546ef2276b30 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:35.349246 16501 tablet_replica.cc:333] T d768701c29784c52a2ffb8303021347d P f8257d37af0941c2a978546ef2276b30: stopping tablet replica
I20260709 12:28:35.350071 16501 raft_consensus.cc:2243] T d768701c29784c52a2ffb8303021347d P f8257d37af0941c2a978546ef2276b30 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:35.350561 16501 raft_consensus.cc:2272] T d768701c29784c52a2ffb8303021347d P f8257d37af0941c2a978546ef2276b30 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:35.353462 16501 tablet_replica.cc:333] T 92a7506e828b47a28f4e2c080d05419b P f8257d37af0941c2a978546ef2276b30: stopping tablet replica
I20260709 12:28:35.354367 16501 raft_consensus.cc:2243] T 92a7506e828b47a28f4e2c080d05419b P f8257d37af0941c2a978546ef2276b30 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:35.355000 16501 raft_consensus.cc:2272] T 92a7506e828b47a28f4e2c080d05419b P f8257d37af0941c2a978546ef2276b30 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:35.357676 16501 tablet_replica.cc:333] T 4b2251c329154ac49d6147b1cfd6981b P f8257d37af0941c2a978546ef2276b30: stopping tablet replica
I20260709 12:28:35.358289 16501 raft_consensus.cc:2243] T 4b2251c329154ac49d6147b1cfd6981b P f8257d37af0941c2a978546ef2276b30 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:35.358883 16501 raft_consensus.cc:2272] T 4b2251c329154ac49d6147b1cfd6981b P f8257d37af0941c2a978546ef2276b30 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:35.361792 16501 tablet_replica.cc:333] T 9396e8d0ed7b44f58bbc5b421f492f50 P f8257d37af0941c2a978546ef2276b30: stopping tablet replica
I20260709 12:28:35.362444 16501 raft_consensus.cc:2243] T 9396e8d0ed7b44f58bbc5b421f492f50 P f8257d37af0941c2a978546ef2276b30 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:28:35.363480 16501 raft_consensus.cc:2272] T 9396e8d0ed7b44f58bbc5b421f492f50 P f8257d37af0941c2a978546ef2276b30 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:35.366346 16501 tablet_replica.cc:333] T 50fe031340a94b6590d29d44cd57ca55 P f8257d37af0941c2a978546ef2276b30: stopping tablet replica
I20260709 12:28:35.367165 16501 raft_consensus.cc:2243] T 50fe031340a94b6590d29d44cd57ca55 P f8257d37af0941c2a978546ef2276b30 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:35.367838 16501 raft_consensus.cc:2272] T 50fe031340a94b6590d29d44cd57ca55 P f8257d37af0941c2a978546ef2276b30 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:35.371151 16501 tablet_replica.cc:333] T f998d4ae7a1e419c9e457994500ce4d1 P f8257d37af0941c2a978546ef2276b30: stopping tablet replica
I20260709 12:28:35.371713 16501 raft_consensus.cc:2243] T f998d4ae7a1e419c9e457994500ce4d1 P f8257d37af0941c2a978546ef2276b30 [term 2 LEADER]: Raft consensus shutting down.
W20260709 12:28:35.371971 32567 consensus_peers.cc:597] T f22ce6f243334c2281f0543e605f614a P f8257d37af0941c2a978546ef2276b30 -> Peer a8adcce962244bc688dc4ecc1f69612c (127.16.29.66:33369): Couldn't send request to peer a8adcce962244bc688dc4ecc1f69612c. Status: Network error: Client connection negotiation failed: client connection to 127.16.29.66:33369: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:28:35.372850 16501 raft_consensus.cc:2272] T f998d4ae7a1e419c9e457994500ce4d1 P f8257d37af0941c2a978546ef2276b30 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:35.375494 16501 tablet_replica.cc:333] T 8e7d5cee9cf64a3e87372bde6431883e P f8257d37af0941c2a978546ef2276b30: stopping tablet replica
I20260709 12:28:35.376175 16501 raft_consensus.cc:2243] T 8e7d5cee9cf64a3e87372bde6431883e P f8257d37af0941c2a978546ef2276b30 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:35.377058 16501 raft_consensus.cc:2272] T 8e7d5cee9cf64a3e87372bde6431883e P f8257d37af0941c2a978546ef2276b30 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:35.380944 16501 tablet_replica.cc:333] T 51c3eb697520458093debb61eebb75a7 P f8257d37af0941c2a978546ef2276b30: stopping tablet replica
I20260709 12:28:35.381860 16501 raft_consensus.cc:2243] T 51c3eb697520458093debb61eebb75a7 P f8257d37af0941c2a978546ef2276b30 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:35.382721 16501 raft_consensus.cc:2272] T 51c3eb697520458093debb61eebb75a7 P f8257d37af0941c2a978546ef2276b30 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:35.386387 16501 tablet_replica.cc:333] T f22ce6f243334c2281f0543e605f614a P f8257d37af0941c2a978546ef2276b30: stopping tablet replica
I20260709 12:28:35.387084 16501 raft_consensus.cc:2243] T f22ce6f243334c2281f0543e605f614a P f8257d37af0941c2a978546ef2276b30 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:28:35.388178 16501 raft_consensus.cc:2272] T f22ce6f243334c2281f0543e605f614a P f8257d37af0941c2a978546ef2276b30 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:35.390923 16501 tablet_replica.cc:333] T 7035e41df60347b188a3b8079aa7e551 P f8257d37af0941c2a978546ef2276b30: stopping tablet replica
I20260709 12:28:35.391583 16501 raft_consensus.cc:2243] T 7035e41df60347b188a3b8079aa7e551 P f8257d37af0941c2a978546ef2276b30 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:35.392316 16501 raft_consensus.cc:2272] T 7035e41df60347b188a3b8079aa7e551 P f8257d37af0941c2a978546ef2276b30 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:35.394965 16501 tablet_replica.cc:333] T aa861e9a505d425792eb54a2a29cab97 P f8257d37af0941c2a978546ef2276b30: stopping tablet replica
I20260709 12:28:35.395768 16501 raft_consensus.cc:2243] T aa861e9a505d425792eb54a2a29cab97 P f8257d37af0941c2a978546ef2276b30 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:35.396430 16501 raft_consensus.cc:2272] T aa861e9a505d425792eb54a2a29cab97 P f8257d37af0941c2a978546ef2276b30 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:35.399147 16501 tablet_replica.cc:333] T c127bfae907d4960956f04576bf0c6c7 P f8257d37af0941c2a978546ef2276b30: stopping tablet replica
I20260709 12:28:35.399797 16501 raft_consensus.cc:2243] T c127bfae907d4960956f04576bf0c6c7 P f8257d37af0941c2a978546ef2276b30 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:28:35.400889 16501 raft_consensus.cc:2272] T c127bfae907d4960956f04576bf0c6c7 P f8257d37af0941c2a978546ef2276b30 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:35.403465 16501 tablet_replica.cc:333] T db65db1003af45d4b6315902498f3f2b P f8257d37af0941c2a978546ef2276b30: stopping tablet replica
I20260709 12:28:35.404111 16501 raft_consensus.cc:2243] T db65db1003af45d4b6315902498f3f2b P f8257d37af0941c2a978546ef2276b30 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:35.404716 16501 raft_consensus.cc:2272] T db65db1003af45d4b6315902498f3f2b P f8257d37af0941c2a978546ef2276b30 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:35.407024 16501 tablet_replica.cc:333] T f5cb9ea668324bfb883f09876ddd56f7 P f8257d37af0941c2a978546ef2276b30: stopping tablet replica
I20260709 12:28:35.407567 16501 raft_consensus.cc:2243] T f5cb9ea668324bfb883f09876ddd56f7 P f8257d37af0941c2a978546ef2276b30 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:35.408141 16501 raft_consensus.cc:2272] T f5cb9ea668324bfb883f09876ddd56f7 P f8257d37af0941c2a978546ef2276b30 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:35.460572 16501 tablet_server.cc:196] TabletServer@127.16.29.67:0 shutdown complete.
I20260709 12:28:35.523061 16501 master.cc:562] Master@127.16.29.126:45119 shutting down...
I20260709 12:28:35.555599 16501 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 66f4042e77394c5a8e850bb63a8a7ac5 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:28:35.556581 16501 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 66f4042e77394c5a8e850bb63a8a7ac5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:35.557001 16501 tablet_replica.cc:333] T 00000000000000000000000000000000 P 66f4042e77394c5a8e850bb63a8a7ac5: stopping tablet replica
W20260709 12:28:36.576026 16501 thread.cc:527] Waited for 1000ms trying to join with diag-logger (tid 32382)
W20260709 12:28:37.576622 16501 thread.cc:527] Waited for 2000ms trying to join with diag-logger (tid 32382)
I20260709 12:28:38.329202 16501 master.cc:584] Master@127.16.29.126:45119 shutdown complete.
[       OK ] FilterSoftDeletedTableTest.TestFilterSofteDeletedTable/1 (25216 ms)
[----------] 2 tests from FilterSoftDeletedTableTest (46547 ms total)

[----------] Global test environment tear-down
[==========] 8 tests from 2 test suites ran. (566269 ms total)
[  PASSED  ] 7 tests.
[  FAILED  ] 1 test, listed below:
[  FAILED  ] LeaderRebalancerTest.AddTserver

 1 FAILED TEST
I20260709 12:28:38.387837 16501 logging.cc:436] LogThrottler /apache/dev/git/kudu/src/kudu/tablet/tablet_mm_ops.cc:220: suppressed but not reported on 2240 messages since previous log ~36 seconds ago
I20260709 12:28:38.388159 16501 logging.cc:436] LogThrottler /apache/dev/git/kudu/src/kudu/tablet/tablet_mm_ops.cc:304: suppressed but not reported on 2240 messages since previous log ~36 seconds ago
I20260709 12:28:38.388370 16501 logging.cc:436] LogThrottler /apache/dev/git/kudu/src/kudu/tablet/tablet_mm_ops.cc:139: suppressed but not reported on 2238 messages since previous log ~36 seconds ago
I20260709 12:28:38.388618 16501 logging.cc:436] LogThrottler /apache/dev/git/kudu/src/kudu/tserver/ts_tablet_manager.cc:600: suppressed but not reported on 9 messages since previous log ~55 seconds ago
I20260709 12:28:38.388833 16501 logging.cc:436] LogThrottler /apache/dev/git/kudu/src/kudu/tserver/ts_tablet_manager.cc:548: suppressed but not reported on 9 messages since previous log ~55 seconds ago
I20260709 12:28:38.389021 16501 logging.cc:436] LogThrottler /apache/dev/git/kudu/src/kudu/consensus/raft_consensus.cc:670: suppressed but not reported on 1 messages since previous log ~235 seconds ago
I20260709 12:28:38.389273 16501 logging.cc:436] LogThrottler /apache/dev/git/kudu/src/kudu/tserver/heartbeater.cc:646: suppressed but not reported on 7 messages since previous log ~220 seconds ago
I20260709 12:28:38.389572 16501 logging.cc:436] LogThrottler /apache/dev/git/kudu/src/kudu/rpc/proxy.cc:239: suppressed but not reported on 12 messages since previous log ~3 seconds ago
I20260709 12:28:38.389739 16501 logging.cc:436] LogThrottler /apache/dev/git/kudu/src/kudu/tablet/tablet.cc:2406: suppressed but not reported on 53 messages since previous log ~24 seconds ago
